Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf
Pablo Neira Ayuso says:
====================
Netfilter/IPVS/OVS fixes for net
The following patchset contains a rather large batch of Netfilter, IPVS
and OVS fixes for your net tree. This includes fixes for ctnetlink, the
userspace conntrack helper infrastructure, conntrack OVS support,
ebtables DNAT target, several leaks in error path among other. More
specifically, they are:
1) Fix reference count leak in the CT target error path, from Gao Feng.
2) Remove conntrack entry clashing with a matching expectation, patch
from Jarno Rajahalme.
3) Fix bogus EEXIST when registering two different userspace helpers,
from Liping Zhang.
4) Don't leak dummy elements in the new bitmap set type in nf_tables,
from Liping Zhang.
5) Get rid of module autoload from conntrack update path in ctnetlink,
we don't need autoload at this late stage and it is happening with
rcu read lock held which is not good. From Liping Zhang.
6) Fix deadlock due to double-acquire of the expect_lock from conntrack
update path, this fixes a bug that was introduced when the central
spinlock got removed. Again from Liping Zhang.
7) Safe ct->status update from ctnetlink path, from Liping. The expect_lock
protection that was selected when the central spinlock was removed was
not really protecting anything at all.
8) Protect sequence adjustment under ct->lock.
9) Missing socket match with IPv6, from Peter Tirsek.
10) Adjust skb->pkt_type of DNAT'ed frames from ebtables, from
Linus Luessing.
11) Don't give up on evaluating the expression on new entries added via
dynset expression in nf_tables, from Liping Zhang.
12) Use skb_checksum() when mangling icmpv6 in IPv6 NAT as this deals
with non-linear skbuffs.
13) Don't allow IPv6 service in IPVS if no IPv6 support is available,
from Paolo Abeni.
14) Missing mutex release in error path of xt_find_table_lock(), from
Dan Carpenter.
15) Update maintainers files, Netfilter section. Add Florian to the
file, refer to nftables.org and change project status from Supported
to Maintained.
16) Bail out on mismatching extensions in element updates in nf_tables.
====================
Signed-off-by: David S. Miller <davem@davemloft.net>
diff --git a/CREDITS b/CREDITS
index c5626bf..5d09c26 100644
--- a/CREDITS
+++ b/CREDITS
@@ -1034,6 +1034,10 @@
S: Boulder, Colorado 80302
S: USA
+N: Hans-Christian Noren Egtvedt
+E: egtvedt@samfundet.no
+D: AVR32 architecture maintainer.
+
N: Heiko Eißfeldt
E: heiko@colossus.escape.de heiko@unifix.de
D: verify_area stuff, generic SCSI fixes
@@ -3398,6 +3402,10 @@
S: Markham, Ontario L3R 2Z6
S: Canada
+N: Haavard Skinnemoen
+M: Haavard Skinnemoen <hskinnemoen@gmail.com>
+D: AVR32 architecture port to Linux and maintainer.
+
N: Rick Sladkey
E: jrs@world.std.com
D: utility hacker: Emacs, NFS server, mount, kmem-ps, UPS debugger, strace, GDB
diff --git a/Documentation/ABI/obsolete/sysfs-firmware-acpi b/Documentation/ABI/obsolete/sysfs-firmware-acpi
new file mode 100644
index 0000000..6715a71
--- /dev/null
+++ b/Documentation/ABI/obsolete/sysfs-firmware-acpi
@@ -0,0 +1,8 @@
+What: /sys/firmware/acpi/hotplug/force_remove
+Date: Mar 2017
+Contact: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
+Description:
+ Since the force_remove is inherently broken and dangerous to
+ use for some hotplugable resources like memory (because ignoring
+ the offline failure might lead to memory corruption and crashes)
+ enabling this knob is not safe and thus unsupported.
diff --git a/Documentation/ABI/stable/sysfs-bus-usb b/Documentation/ABI/stable/sysfs-bus-usb
index 831f15d..b832eef 100644
--- a/Documentation/ABI/stable/sysfs-bus-usb
+++ b/Documentation/ABI/stable/sysfs-bus-usb
@@ -9,7 +9,7 @@
hubs this facility is always enabled and their device
directories will not contain this file.
- For more information, see Documentation/usb/persist.txt.
+ For more information, see Documentation/driver-api/usb/persist.rst.
What: /sys/bus/usb/devices/.../power/autosuspend
Date: March 2007
diff --git a/Documentation/ABI/stable/vdso b/Documentation/ABI/stable/vdso
index 7cdfc28..55406ec 100644
--- a/Documentation/ABI/stable/vdso
+++ b/Documentation/ABI/stable/vdso
@@ -16,7 +16,8 @@
vDSO, specify the version you are expecting.
Programs that dynamically link to glibc will use the vDSO automatically.
-Otherwise, you can use the reference parser in Documentation/vDSO/parse_vdso.c.
+Otherwise, you can use the reference parser in
+tools/testing/selftests/vDSO/parse_vdso.c.
Unless otherwise noted, the set of symbols with any given version and the
ABI of those symbols is considered stable. It may vary across architectures,
diff --git a/Documentation/ABI/testing/sysfs-block b/Documentation/ABI/testing/sysfs-block
index 2da04ce..dea212d 100644
--- a/Documentation/ABI/testing/sysfs-block
+++ b/Documentation/ABI/testing/sysfs-block
@@ -213,14 +213,8 @@
Date: May 2011
Contact: Martin K. Petersen <martin.petersen@oracle.com>
Description:
- Devices that support discard functionality may return
- stale or random data when a previously discarded block
- is read back. This can cause problems if the filesystem
- expects discarded blocks to be explicitly cleared. If a
- device reports that it deterministically returns zeroes
- when a discarded area is read the discard_zeroes_data
- parameter will be set to one. Otherwise it will be 0 and
- the result of reading a discarded area is undefined.
+ Will always return 0. Don't rely on any specific behavior
+ for discards, and don't read this file.
What: /sys/block/<disk>/queue/write_same_max_bytes
Date: January 2012
diff --git a/Documentation/ABI/testing/sysfs-bus-pci b/Documentation/ABI/testing/sysfs-bus-pci
index 5a1732b..e4e9010 100644
--- a/Documentation/ABI/testing/sysfs-bus-pci
+++ b/Documentation/ABI/testing/sysfs-bus-pci
@@ -299,5 +299,5 @@
Date: November 2016
Contact: Emil Velikov <emil.l.velikov@gmail.com>
Description:
- This file contains the revision field of the the PCI device.
+ This file contains the revision field of the PCI device.
The value comes from device config space. The file is read only.
diff --git a/Documentation/ABI/testing/sysfs-class-net-qmi b/Documentation/ABI/testing/sysfs-class-net-qmi
index fa5a00b..7122d62 100644
--- a/Documentation/ABI/testing/sysfs-class-net-qmi
+++ b/Documentation/ABI/testing/sysfs-class-net-qmi
@@ -21,3 +21,30 @@
is responsible for coordination of driver and firmware
link framing mode, changing this setting to 'Y' if the
firmware is configured for 'raw-ip' mode.
+
+What: /sys/class/net/<iface>/qmi/add_mux
+Date: March 2017
+KernelVersion: 4.11
+Contact: Bjørn Mork <bjorn@mork.no>
+Description:
+ Unsigned integer.
+
+ Write a number ranging from 1 to 127 to add a qmap mux
+ based network device, supported by recent Qualcomm based
+ modems.
+
+ The network device will be called qmimux.
+
+ Userspace is in charge of managing the qmux network device
+ activation and data stream setup on the modem side by
+ using the proper QMI protocol requests.
+
+What: /sys/class/net/<iface>/qmi/del_mux
+Date: March 2017
+KernelVersion: 4.11
+Contact: Bjørn Mork <bjorn@mork.no>
+Description:
+ Unsigned integer.
+
+ Write a number ranging from 1 to 127 to delete a previously
+ created qmap mux based network device.
diff --git a/Documentation/ABI/testing/sysfs-firmware-acpi b/Documentation/ABI/testing/sysfs-firmware-acpi
index c7fc72d..613f42a9 100644
--- a/Documentation/ABI/testing/sysfs-firmware-acpi
+++ b/Documentation/ABI/testing/sysfs-firmware-acpi
@@ -44,16 +44,6 @@
or 0 (unset). Attempts to write any other values to it will
cause -EINVAL to be returned.
-What: /sys/firmware/acpi/hotplug/force_remove
-Date: May 2013
-Contact: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
-Description:
- The number in this file (0 or 1) determines whether (1) or not
- (0) the ACPI subsystem will allow devices to be hot-removed even
- if they cannot be put offline gracefully (from the kernel's
- viewpoint). That number can be changed by writing a boolean
- value to this file.
-
What: /sys/firmware/acpi/interrupts/
Date: February 2008
Contact: Len Brown <lenb@kernel.org>
diff --git a/Documentation/ABI/testing/sysfs-kernel-livepatch b/Documentation/ABI/testing/sysfs-kernel-livepatch
index da87f43..d5d3974 100644
--- a/Documentation/ABI/testing/sysfs-kernel-livepatch
+++ b/Documentation/ABI/testing/sysfs-kernel-livepatch
@@ -25,6 +25,14 @@
code is currently applied. Writing 0 will disable the patch
while writing 1 will re-enable the patch.
+What: /sys/kernel/livepatch/<patch>/transition
+Date: Feb 2017
+KernelVersion: 4.12.0
+Contact: live-patching@vger.kernel.org
+Description:
+ An attribute which indicates whether the patch is currently in
+ transition.
+
What: /sys/kernel/livepatch/<patch>/<object>
Date: Nov 2014
KernelVersion: 3.19.0
diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile
index 164c1c7..85916f1 100644
--- a/Documentation/DocBook/Makefile
+++ b/Documentation/DocBook/Makefile
@@ -8,12 +8,11 @@
DOCBOOKS := z8530book.xml \
kernel-hacking.xml kernel-locking.xml \
- writing_usb_driver.xml networking.xml \
- kernel-api.xml filesystems.xml lsm.xml kgdb.xml \
- gadget.xml libata.xml mtdnand.xml librs.xml rapidio.xml \
- genericirq.xml s390-drivers.xml scsi.xml \
- sh.xml w1.xml \
- writing_musb_glue_layer.xml
+ networking.xml \
+ filesystems.xml lsm.xml kgdb.xml \
+ libata.xml mtdnand.xml librs.xml rapidio.xml \
+ s390-drivers.xml scsi.xml \
+ sh.xml w1.xml
ifeq ($(DOCBOOKS),)
@@ -62,11 +61,14 @@
mandocs: $(MAN)
find $(obj)/man -name '*.9' | xargs gzip -nf
+# Default location for installed man pages
+export INSTALL_MAN_PATH = $(objtree)/usr
+
installmandocs: mandocs
- mkdir -p /usr/local/man/man9/
+ mkdir -p $(INSTALL_MAN_PATH)/man/man9/
find $(obj)/man -name '*.9.gz' -printf '%h %f\n' | \
sort -k 2 -k 1 | uniq -f 1 | sed -e 's: :/:' | \
- xargs install -m 644 -t /usr/local/man/man9/
+ xargs install -m 644 -t $(INSTALL_MAN_PATH)/man/man9/
# no-op for the DocBook toolchain
epubdocs:
@@ -238,7 +240,9 @@
@echo ' psdocs - Postscript'
@echo ' xmldocs - XML DocBook'
@echo ' mandocs - man pages'
- @echo ' installmandocs - install man pages generated by mandocs'
+ @echo ' installmandocs - install man pages generated by mandocs to INSTALL_MAN_PATH'; \
+ echo ' (default: $(INSTALL_MAN_PATH))'; \
+ echo ''
@echo ' cleandocs - clean all generated DocBook files'
@echo
@echo ' make DOCBOOKS="s1.xml s2.xml" [target] Generate only docs s1.xml s2.xml'
diff --git a/Documentation/DocBook/gadget.tmpl b/Documentation/DocBook/gadget.tmpl
deleted file mode 100644
index 6416292..0000000
--- a/Documentation/DocBook/gadget.tmpl
+++ /dev/null
@@ -1,793 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="USB-Gadget-API">
- <bookinfo>
- <title>USB Gadget API for Linux</title>
- <date>20 August 2004</date>
- <edition>20 August 2004</edition>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License as published by the Free Software Foundation; either
- version 2 of the License, or (at your option) any later
- version.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- <copyright>
- <year>2003-2004</year>
- <holder>David Brownell</holder>
- </copyright>
-
- <author>
- <firstname>David</firstname>
- <surname>Brownell</surname>
- <affiliation>
- <address><email>dbrownell@users.sourceforge.net</email></address>
- </affiliation>
- </author>
- </bookinfo>
-
-<toc></toc>
-
-<chapter id="intro"><title>Introduction</title>
-
-<para>This document presents a Linux-USB "Gadget"
-kernel mode
-API, for use within peripherals and other USB devices
-that embed Linux.
-It provides an overview of the API structure,
-and shows how that fits into a system development project.
-This is the first such API released on Linux to address
-a number of important problems, including: </para>
-
-<itemizedlist>
- <listitem><para>Supports USB 2.0, for high speed devices which
- can stream data at several dozen megabytes per second.
- </para></listitem>
- <listitem><para>Handles devices with dozens of endpoints just as
- well as ones with just two fixed-function ones. Gadget drivers
- can be written so they're easy to port to new hardware.
- </para></listitem>
- <listitem><para>Flexible enough to expose more complex USB device
- capabilities such as multiple configurations, multiple interfaces,
- composite devices,
- and alternate interface settings.
- </para></listitem>
- <listitem><para>USB "On-The-Go" (OTG) support, in conjunction
- with updates to the Linux-USB host side.
- </para></listitem>
- <listitem><para>Sharing data structures and API models with the
- Linux-USB host side API. This helps the OTG support, and
- looks forward to more-symmetric frameworks (where the same
- I/O model is used by both host and device side drivers).
- </para></listitem>
- <listitem><para>Minimalist, so it's easier to support new device
- controller hardware. I/O processing doesn't imply large
- demands for memory or CPU resources.
- </para></listitem>
-</itemizedlist>
-
-
-<para>Most Linux developers will not be able to use this API, since they
-have USB "host" hardware in a PC, workstation, or server.
-Linux users with embedded systems are more likely to
-have USB peripheral hardware.
-To distinguish drivers running inside such hardware from the
-more familiar Linux "USB device drivers",
-which are host side proxies for the real USB devices,
-a different term is used:
-the drivers inside the peripherals are "USB gadget drivers".
-In USB protocol interactions, the device driver is the master
-(or "client driver")
-and the gadget driver is the slave (or "function driver").
-</para>
-
-<para>The gadget API resembles the host side Linux-USB API in that both
-use queues of request objects to package I/O buffers, and those requests
-may be submitted or canceled.
-They share common definitions for the standard USB
-<emphasis>Chapter 9</emphasis> messages, structures, and constants.
-Also, both APIs bind and unbind drivers to devices.
-The APIs differ in detail, since the host side's current
-URB framework exposes a number of implementation details
-and assumptions that are inappropriate for a gadget API.
-While the model for control transfers and configuration
-management is necessarily different (one side is a hardware-neutral master,
-the other is a hardware-aware slave), the endpoint I/0 API used here
-should also be usable for an overhead-reduced host side API.
-</para>
-
-</chapter>
-
-<chapter id="structure"><title>Structure of Gadget Drivers</title>
-
-<para>A system running inside a USB peripheral
-normally has at least three layers inside the kernel to handle
-USB protocol processing, and may have additional layers in
-user space code.
-The "gadget" API is used by the middle layer to interact
-with the lowest level (which directly handles hardware).
-</para>
-
-<para>In Linux, from the bottom up, these layers are:
-</para>
-
-<variablelist>
-
- <varlistentry>
- <term><emphasis>USB Controller Driver</emphasis></term>
-
- <listitem>
- <para>This is the lowest software level.
- It is the only layer that talks to hardware,
- through registers, fifos, dma, irqs, and the like.
- The <filename><linux/usb/gadget.h></filename> API abstracts
- the peripheral controller endpoint hardware.
- That hardware is exposed through endpoint objects, which accept
- streams of IN/OUT buffers, and through callbacks that interact
- with gadget drivers.
- Since normal USB devices only have one upstream
- port, they only have one of these drivers.
- The controller driver can support any number of different
- gadget drivers, but only one of them can be used at a time.
- </para>
-
- <para>Examples of such controller hardware include
- the PCI-based NetChip 2280 USB 2.0 high speed controller,
- the SA-11x0 or PXA-25x UDC (found within many PDAs),
- and a variety of other products.
- </para>
-
- </listitem></varlistentry>
-
- <varlistentry>
- <term><emphasis>Gadget Driver</emphasis></term>
-
- <listitem>
- <para>The lower boundary of this driver implements hardware-neutral
- USB functions, using calls to the controller driver.
- Because such hardware varies widely in capabilities and restrictions,
- and is used in embedded environments where space is at a premium,
- the gadget driver is often configured at compile time
- to work with endpoints supported by one particular controller.
- Gadget drivers may be portable to several different controllers,
- using conditional compilation.
- (Recent kernels substantially simplify the work involved in
- supporting new hardware, by <emphasis>autoconfiguring</emphasis>
- endpoints automatically for many bulk-oriented drivers.)
- Gadget driver responsibilities include:
- </para>
- <itemizedlist>
- <listitem><para>handling setup requests (ep0 protocol responses)
- possibly including class-specific functionality
- </para></listitem>
- <listitem><para>returning configuration and string descriptors
- </para></listitem>
- <listitem><para>(re)setting configurations and interface
- altsettings, including enabling and configuring endpoints
- </para></listitem>
- <listitem><para>handling life cycle events, such as managing
- bindings to hardware,
- USB suspend/resume, remote wakeup,
- and disconnection from the USB host.
- </para></listitem>
- <listitem><para>managing IN and OUT transfers on all currently
- enabled endpoints
- </para></listitem>
- </itemizedlist>
-
- <para>
- Such drivers may be modules of proprietary code, although
- that approach is discouraged in the Linux community.
- </para>
- </listitem></varlistentry>
-
- <varlistentry>
- <term><emphasis>Upper Level</emphasis></term>
-
- <listitem>
- <para>Most gadget drivers have an upper boundary that connects
- to some Linux driver or framework in Linux.
- Through that boundary flows the data which the gadget driver
- produces and/or consumes through protocol transfers over USB.
- Examples include:
- </para>
- <itemizedlist>
- <listitem><para>user mode code, using generic (gadgetfs)
- or application specific files in
- <filename>/dev</filename>
- </para></listitem>
- <listitem><para>networking subsystem (for network gadgets,
- like the CDC Ethernet Model gadget driver)
- </para></listitem>
- <listitem><para>data capture drivers, perhaps video4Linux or
- a scanner driver; or test and measurement hardware.
- </para></listitem>
- <listitem><para>input subsystem (for HID gadgets)
- </para></listitem>
- <listitem><para>sound subsystem (for audio gadgets)
- </para></listitem>
- <listitem><para>file system (for PTP gadgets)
- </para></listitem>
- <listitem><para>block i/o subsystem (for usb-storage gadgets)
- </para></listitem>
- <listitem><para>... and more </para></listitem>
- </itemizedlist>
- </listitem></varlistentry>
-
- <varlistentry>
- <term><emphasis>Additional Layers</emphasis></term>
-
- <listitem>
- <para>Other layers may exist.
- These could include kernel layers, such as network protocol stacks,
- as well as user mode applications building on standard POSIX
- system call APIs such as
- <emphasis>open()</emphasis>, <emphasis>close()</emphasis>,
- <emphasis>read()</emphasis> and <emphasis>write()</emphasis>.
- On newer systems, POSIX Async I/O calls may be an option.
- Such user mode code will not necessarily be subject to
- the GNU General Public License (GPL).
- </para>
- </listitem></varlistentry>
-
-
-</variablelist>
-
-<para>OTG-capable systems will also need to include a standard Linux-USB
-host side stack,
-with <emphasis>usbcore</emphasis>,
-one or more <emphasis>Host Controller Drivers</emphasis> (HCDs),
-<emphasis>USB Device Drivers</emphasis> to support
-the OTG "Targeted Peripheral List",
-and so forth.
-There will also be an <emphasis>OTG Controller Driver</emphasis>,
-which is visible to gadget and device driver developers only indirectly.
-That helps the host and device side USB controllers implement the
-two new OTG protocols (HNP and SRP).
-Roles switch (host to peripheral, or vice versa) using HNP
-during USB suspend processing, and SRP can be viewed as a
-more battery-friendly kind of device wakeup protocol.
-</para>
-
-<para>Over time, reusable utilities are evolving to help make some
-gadget driver tasks simpler.
-For example, building configuration descriptors from vectors of
-descriptors for the configurations interfaces and endpoints is
-now automated, and many drivers now use autoconfiguration to
-choose hardware endpoints and initialize their descriptors.
-
-A potential example of particular interest
-is code implementing standard USB-IF protocols for
-HID, networking, storage, or audio classes.
-Some developers are interested in KDB or KGDB hooks, to let
-target hardware be remotely debugged.
-Most such USB protocol code doesn't need to be hardware-specific,
-any more than network protocols like X11, HTTP, or NFS are.
-Such gadget-side interface drivers should eventually be combined,
-to implement composite devices.
-</para>
-
-</chapter>
-
-
-<chapter id="api"><title>Kernel Mode Gadget API</title>
-
-<para>Gadget drivers declare themselves through a
-<emphasis>struct usb_gadget_driver</emphasis>, which is responsible for
-most parts of enumeration for a <emphasis>struct usb_gadget</emphasis>.
-The response to a set_configuration usually involves
-enabling one or more of the <emphasis>struct usb_ep</emphasis> objects
-exposed by the gadget, and submitting one or more
-<emphasis>struct usb_request</emphasis> buffers to transfer data.
-Understand those four data types, and their operations, and
-you will understand how this API works.
-</para>
-
-<note><title>Incomplete Data Type Descriptions</title>
-
-<para>This documentation was prepared using the standard Linux
-kernel <filename>docproc</filename> tool, which turns text
-and in-code comments into SGML DocBook and then into usable
-formats such as HTML or PDF.
-Other than the "Chapter 9" data types, most of the significant
-data types and functions are described here.
-</para>
-
-<para>However, docproc does not understand all the C constructs
-that are used, so some relevant information is likely omitted from
-what you are reading.
-One example of such information is endpoint autoconfiguration.
-You'll have to read the header file, and use example source
-code (such as that for "Gadget Zero"), to fully understand the API.
-</para>
-
-<para>The part of the API implementing some basic
-driver capabilities is specific to the version of the
-Linux kernel that's in use.
-The 2.6 kernel includes a <emphasis>driver model</emphasis>
-framework that has no analogue on earlier kernels;
-so those parts of the gadget API are not fully portable.
-(They are implemented on 2.4 kernels, but in a different way.)
-The driver model state is another part of this API that is
-ignored by the kerneldoc tools.
-</para>
-</note>
-
-<para>The core API does not expose
-every possible hardware feature, only the most widely available ones.
-There are significant hardware features, such as device-to-device DMA
-(without temporary storage in a memory buffer)
-that would be added using hardware-specific APIs.
-</para>
-
-<para>This API allows drivers to use conditional compilation to handle
-endpoint capabilities of different hardware, but doesn't require that.
-Hardware tends to have arbitrary restrictions, relating to
-transfer types, addressing, packet sizes, buffering, and availability.
-As a rule, such differences only matter for "endpoint zero" logic
-that handles device configuration and management.
-The API supports limited run-time
-detection of capabilities, through naming conventions for endpoints.
-Many drivers will be able to at least partially autoconfigure
-themselves.
-In particular, driver init sections will often have endpoint
-autoconfiguration logic that scans the hardware's list of endpoints
-to find ones matching the driver requirements
-(relying on those conventions), to eliminate some of the most
-common reasons for conditional compilation.
-</para>
-
-<para>Like the Linux-USB host side API, this API exposes
-the "chunky" nature of USB messages: I/O requests are in terms
-of one or more "packets", and packet boundaries are visible to drivers.
-Compared to RS-232 serial protocols, USB resembles
-synchronous protocols like HDLC
-(N bytes per frame, multipoint addressing, host as the primary
-station and devices as secondary stations)
-more than asynchronous ones
-(tty style: 8 data bits per frame, no parity, one stop bit).
-So for example the controller drivers won't buffer
-two single byte writes into a single two-byte USB IN packet,
-although gadget drivers may do so when they implement
-protocols where packet boundaries (and "short packets")
-are not significant.
-</para>
-
-<sect1 id="lifecycle"><title>Driver Life Cycle</title>
-
-<para>Gadget drivers make endpoint I/O requests to hardware without
-needing to know many details of the hardware, but driver
-setup/configuration code needs to handle some differences.
-Use the API like this:
-</para>
-
-<orderedlist numeration='arabic'>
-
-<listitem><para>Register a driver for the particular device side
-usb controller hardware,
-such as the net2280 on PCI (USB 2.0),
-sa11x0 or pxa25x as found in Linux PDAs,
-and so on.
-At this point the device is logically in the USB ch9 initial state
-("attached"), drawing no power and not usable
-(since it does not yet support enumeration).
-Any host should not see the device, since it's not
-activated the data line pullup used by the host to
-detect a device, even if VBUS power is available.
-</para></listitem>
-
-<listitem><para>Register a gadget driver that implements some higher level
-device function. That will then bind() to a usb_gadget, which
-activates the data line pullup sometime after detecting VBUS.
-</para></listitem>
-
-<listitem><para>The hardware driver can now start enumerating.
-The steps it handles are to accept USB power and set_address requests.
-Other steps are handled by the gadget driver.
-If the gadget driver module is unloaded before the host starts to
-enumerate, steps before step 7 are skipped.
-</para></listitem>
-
-<listitem><para>The gadget driver's setup() call returns usb descriptors,
-based both on what the bus interface hardware provides and on the
-functionality being implemented.
-That can involve alternate settings or configurations,
-unless the hardware prevents such operation.
-For OTG devices, each configuration descriptor includes
-an OTG descriptor.
-</para></listitem>
-
-<listitem><para>The gadget driver handles the last step of enumeration,
-when the USB host issues a set_configuration call.
-It enables all endpoints used in that configuration,
-with all interfaces in their default settings.
-That involves using a list of the hardware's endpoints, enabling each
-endpoint according to its descriptor.
-It may also involve using <function>usb_gadget_vbus_draw</function>
-to let more power be drawn from VBUS, as allowed by that configuration.
-For OTG devices, setting a configuration may also involve reporting
-HNP capabilities through a user interface.
-</para></listitem>
-
-<listitem><para>Do real work and perform data transfers, possibly involving
-changes to interface settings or switching to new configurations, until the
-device is disconnect()ed from the host.
-Queue any number of transfer requests to each endpoint.
-It may be suspended and resumed several times before being disconnected.
-On disconnect, the drivers go back to step 3 (above).
-</para></listitem>
-
-<listitem><para>When the gadget driver module is being unloaded,
-the driver unbind() callback is issued. That lets the controller
-driver be unloaded.
-</para></listitem>
-
-</orderedlist>
-
-<para>Drivers will normally be arranged so that just loading the
-gadget driver module (or statically linking it into a Linux kernel)
-allows the peripheral device to be enumerated, but some drivers
-will defer enumeration until some higher level component (like
-a user mode daemon) enables it.
-Note that at this lowest level there are no policies about how
-ep0 configuration logic is implemented,
-except that it should obey USB specifications.
-Such issues are in the domain of gadget drivers,
-including knowing about implementation constraints
-imposed by some USB controllers
-or understanding that composite devices might happen to
-be built by integrating reusable components.
-</para>
-
-<para>Note that the lifecycle above can be slightly different
-for OTG devices.
-Other than providing an additional OTG descriptor in each
-configuration, only the HNP-related differences are particularly
-visible to driver code.
-They involve reporting requirements during the SET_CONFIGURATION
-request, and the option to invoke HNP during some suspend callbacks.
-Also, SRP changes the semantics of
-<function>usb_gadget_wakeup</function>
-slightly.
-</para>
-
-</sect1>
-
-<sect1 id="ch9"><title>USB 2.0 Chapter 9 Types and Constants</title>
-
-<para>Gadget drivers
-rely on common USB structures and constants
-defined in the
-<filename><linux/usb/ch9.h></filename>
-header file, which is standard in Linux 2.6 kernels.
-These are the same types and constants used by host
-side drivers (and usbcore).
-</para>
-
-!Iinclude/linux/usb/ch9.h
-</sect1>
-
-<sect1 id="core"><title>Core Objects and Methods</title>
-
-<para>These are declared in
-<filename><linux/usb/gadget.h></filename>,
-and are used by gadget drivers to interact with
-USB peripheral controller drivers.
-</para>
-
- <!-- yeech, this is ugly in nsgmls PDF output.
-
- the PDF bookmark and refentry output nesting is wrong,
- and the member/argument documentation indents ugly.
-
- plus something (docproc?) adds whitespace before the
- descriptive paragraph text, so it can't line up right
- unless the explanations are trivial.
- -->
-
-!Iinclude/linux/usb/gadget.h
-</sect1>
-
-<sect1 id="utils"><title>Optional Utilities</title>
-
-<para>The core API is sufficient for writing a USB Gadget Driver,
-but some optional utilities are provided to simplify common tasks.
-These utilities include endpoint autoconfiguration.
-</para>
-
-!Edrivers/usb/gadget/usbstring.c
-!Edrivers/usb/gadget/config.c
-<!-- !Edrivers/usb/gadget/epautoconf.c -->
-</sect1>
-
-<sect1 id="composite"><title>Composite Device Framework</title>
-
-<para>The core API is sufficient for writing drivers for composite
-USB devices (with more than one function in a given configuration),
-and also multi-configuration devices (also more than one function,
-but not necessarily sharing a given configuration).
-There is however an optional framework which makes it easier to
-reuse and combine functions.
-</para>
-
-<para>Devices using this framework provide a <emphasis>struct
-usb_composite_driver</emphasis>, which in turn provides one or
-more <emphasis>struct usb_configuration</emphasis> instances.
-Each such configuration includes at least one
-<emphasis>struct usb_function</emphasis>, which packages a user
-visible role such as "network link" or "mass storage device".
-Management functions may also exist, such as "Device Firmware
-Upgrade".
-</para>
-
-!Iinclude/linux/usb/composite.h
-!Edrivers/usb/gadget/composite.c
-
-</sect1>
-
-<sect1 id="functions"><title>Composite Device Functions</title>
-
-<para>At this writing, a few of the current gadget drivers have
-been converted to this framework.
-Near-term plans include converting all of them, except for "gadgetfs".
-</para>
-
-!Edrivers/usb/gadget/function/f_acm.c
-!Edrivers/usb/gadget/function/f_ecm.c
-!Edrivers/usb/gadget/function/f_subset.c
-!Edrivers/usb/gadget/function/f_obex.c
-!Edrivers/usb/gadget/function/f_serial.c
-
-</sect1>
-
-
-</chapter>
-
-<chapter id="controllers"><title>Peripheral Controller Drivers</title>
-
-<para>The first hardware supporting this API was the NetChip 2280
-controller, which supports USB 2.0 high speed and is based on PCI.
-This is the <filename>net2280</filename> driver module.
-The driver supports Linux kernel versions 2.4 and 2.6;
-contact NetChip Technologies for development boards and product
-information.
-</para>
-
-<para>Other hardware working in the "gadget" framework includes:
-Intel's PXA 25x and IXP42x series processors
-(<filename>pxa2xx_udc</filename>),
-Toshiba TC86c001 "Goku-S" (<filename>goku_udc</filename>),
-Renesas SH7705/7727 (<filename>sh_udc</filename>),
-MediaQ 11xx (<filename>mq11xx_udc</filename>),
-Hynix HMS30C7202 (<filename>h7202_udc</filename>),
-National 9303/4 (<filename>n9604_udc</filename>),
-Texas Instruments OMAP (<filename>omap_udc</filename>),
-Sharp LH7A40x (<filename>lh7a40x_udc</filename>),
-and more.
-Most of those are full speed controllers.
-</para>
-
-<para>At this writing, there are people at work on drivers in
-this framework for several other USB device controllers,
-with plans to make many of them be widely available.
-</para>
-
-<!-- !Edrivers/usb/gadget/net2280.c -->
-
-<para>A partial USB simulator,
-the <filename>dummy_hcd</filename> driver, is available.
-It can act like a net2280, a pxa25x, or an sa11x0 in terms
-of available endpoints and device speeds; and it simulates
-control, bulk, and to some extent interrupt transfers.
-That lets you develop some parts of a gadget driver on a normal PC,
-without any special hardware, and perhaps with the assistance
-of tools such as GDB running with User Mode Linux.
-At least one person has expressed interest in adapting that
-approach, hooking it up to a simulator for a microcontroller.
-Such simulators can help debug subsystems where the runtime hardware
-is unfriendly to software development, or is not yet available.
-</para>
-
-<para>Support for other controllers is expected to be developed
-and contributed
-over time, as this driver framework evolves.
-</para>
-
-</chapter>
-
-<chapter id="gadget"><title>Gadget Drivers</title>
-
-<para>In addition to <emphasis>Gadget Zero</emphasis>
-(used primarily for testing and development with drivers
-for usb controller hardware), other gadget drivers exist.
-</para>
-
-<para>There's an <emphasis>ethernet</emphasis> gadget
-driver, which implements one of the most useful
-<emphasis>Communications Device Class</emphasis> (CDC) models.
-One of the standards for cable modem interoperability even
-specifies the use of this ethernet model as one of two
-mandatory options.
-Gadgets using this code look to a USB host as if they're
-an Ethernet adapter.
-It provides access to a network where the gadget's CPU is one host,
-which could easily be bridging, routing, or firewalling
-access to other networks.
-Since some hardware can't fully implement the CDC Ethernet
-requirements, this driver also implements a "good parts only"
-subset of CDC Ethernet.
-(That subset doesn't advertise itself as CDC Ethernet,
-to avoid creating problems.)
-</para>
-
-<para>Support for Microsoft's <emphasis>RNDIS</emphasis>
-protocol has been contributed by Pengutronix and Auerswald GmbH.
-This is like CDC Ethernet, but it runs on more slightly USB hardware
-(but less than the CDC subset).
-However, its main claim to fame is being able to connect directly to
-recent versions of Windows, using drivers that Microsoft bundles
-and supports, making it much simpler to network with Windows.
-</para>
-
-<para>There is also support for user mode gadget drivers,
-using <emphasis>gadgetfs</emphasis>.
-This provides a <emphasis>User Mode API</emphasis> that presents
-each endpoint as a single file descriptor. I/O is done using
-normal <emphasis>read()</emphasis> and <emphasis>read()</emphasis> calls.
-Familiar tools like GDB and pthreads can be used to
-develop and debug user mode drivers, so that once a robust
-controller driver is available many applications for it
-won't require new kernel mode software.
-Linux 2.6 <emphasis>Async I/O (AIO)</emphasis>
-support is available, so that user mode software
-can stream data with only slightly more overhead
-than a kernel driver.
-</para>
-
-<para>There's a USB Mass Storage class driver, which provides
-a different solution for interoperability with systems such
-as MS-Windows and MacOS.
-That <emphasis>Mass Storage</emphasis> driver uses a
-file or block device as backing store for a drive,
-like the <filename>loop</filename> driver.
-The USB host uses the BBB, CB, or CBI versions of the mass
-storage class specification, using transparent SCSI commands
-to access the data from the backing store.
-</para>
-
-<para>There's a "serial line" driver, useful for TTY style
-operation over USB.
-The latest version of that driver supports CDC ACM style
-operation, like a USB modem, and so on most hardware it can
-interoperate easily with MS-Windows.
-One interesting use of that driver is in boot firmware (like a BIOS),
-which can sometimes use that model with very small systems without
-real serial lines.
-</para>
-
-<para>Support for other kinds of gadget is expected to
-be developed and contributed
-over time, as this driver framework evolves.
-</para>
-
-</chapter>
-
-<chapter id="otg"><title>USB On-The-GO (OTG)</title>
-
-<para>USB OTG support on Linux 2.6 was initially developed
-by Texas Instruments for
-<ulink url="http://www.omap.com">OMAP</ulink> 16xx and 17xx
-series processors.
-Other OTG systems should work in similar ways, but the
-hardware level details could be very different.
-</para>
-
-<para>Systems need specialized hardware support to implement OTG,
-notably including a special <emphasis>Mini-AB</emphasis> jack
-and associated transceiver to support <emphasis>Dual-Role</emphasis>
-operation:
-they can act either as a host, using the standard
-Linux-USB host side driver stack,
-or as a peripheral, using this "gadget" framework.
-To do that, the system software relies on small additions
-to those programming interfaces,
-and on a new internal component (here called an "OTG Controller")
-affecting which driver stack connects to the OTG port.
-In each role, the system can re-use the existing pool of
-hardware-neutral drivers, layered on top of the controller
-driver interfaces (<emphasis>usb_bus</emphasis> or
-<emphasis>usb_gadget</emphasis>).
-Such drivers need at most minor changes, and most of the calls
-added to support OTG can also benefit non-OTG products.
-</para>
-
-<itemizedlist>
- <listitem><para>Gadget drivers test the <emphasis>is_otg</emphasis>
- flag, and use it to determine whether or not to include
- an OTG descriptor in each of their configurations.
- </para></listitem>
- <listitem><para>Gadget drivers may need changes to support the
- two new OTG protocols, exposed in new gadget attributes
- such as <emphasis>b_hnp_enable</emphasis> flag.
- HNP support should be reported through a user interface
- (two LEDs could suffice), and is triggered in some cases
- when the host suspends the peripheral.
- SRP support can be user-initiated just like remote wakeup,
- probably by pressing the same button.
- </para></listitem>
- <listitem><para>On the host side, USB device drivers need
- to be taught to trigger HNP at appropriate moments, using
- <function>usb_suspend_device()</function>.
- That also conserves battery power, which is useful even
- for non-OTG configurations.
- </para></listitem>
- <listitem><para>Also on the host side, a driver must support the
- OTG "Targeted Peripheral List". That's just a whitelist,
- used to reject peripherals not supported with a given
- Linux OTG host.
- <emphasis>This whitelist is product-specific;
- each product must modify <filename>otg_whitelist.h</filename>
- to match its interoperability specification.
- </emphasis>
- </para>
- <para>Non-OTG Linux hosts, like PCs and workstations,
- normally have some solution for adding drivers, so that
- peripherals that aren't recognized can eventually be supported.
- That approach is unreasonable for consumer products that may
- never have their firmware upgraded, and where it's usually
- unrealistic to expect traditional PC/workstation/server kinds
- of support model to work.
- For example, it's often impractical to change device firmware
- once the product has been distributed, so driver bugs can't
- normally be fixed if they're found after shipment.
- </para></listitem>
-</itemizedlist>
-
-<para>
-Additional changes are needed below those hardware-neutral
-<emphasis>usb_bus</emphasis> and <emphasis>usb_gadget</emphasis>
-driver interfaces; those aren't discussed here in any detail.
-Those affect the hardware-specific code for each USB Host or Peripheral
-controller, and how the HCD initializes (since OTG can be active only
-on a single port).
-They also involve what may be called an <emphasis>OTG Controller
-Driver</emphasis>, managing the OTG transceiver and the OTG state
-machine logic as well as much of the root hub behavior for the
-OTG port.
-The OTG controller driver needs to activate and deactivate USB
-controllers depending on the relevant device role.
-Some related changes were needed inside usbcore, so that it
-can identify OTG-capable devices and respond appropriately
-to HNP or SRP protocols.
-</para>
-
-</chapter>
-
-</book>
-<!--
- vim:syntax=sgml:sw=4
--->
diff --git a/Documentation/DocBook/genericirq.tmpl b/Documentation/DocBook/genericirq.tmpl
deleted file mode 100644
index 59fb5c0..0000000
--- a/Documentation/DocBook/genericirq.tmpl
+++ /dev/null
@@ -1,520 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="Generic-IRQ-Guide">
- <bookinfo>
- <title>Linux generic IRQ handling</title>
-
- <authorgroup>
- <author>
- <firstname>Thomas</firstname>
- <surname>Gleixner</surname>
- <affiliation>
- <address>
- <email>tglx@linutronix.de</email>
- </address>
- </affiliation>
- </author>
- <author>
- <firstname>Ingo</firstname>
- <surname>Molnar</surname>
- <affiliation>
- <address>
- <email>mingo@elte.hu</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
-
- <copyright>
- <year>2005-2010</year>
- <holder>Thomas Gleixner</holder>
- </copyright>
- <copyright>
- <year>2005-2006</year>
- <holder>Ingo Molnar</holder>
- </copyright>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License version 2 as published by the Free Software Foundation.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
-
- <chapter id="intro">
- <title>Introduction</title>
- <para>
- The generic interrupt handling layer is designed to provide a
- complete abstraction of interrupt handling for device drivers.
- It is able to handle all the different types of interrupt controller
- hardware. Device drivers use generic API functions to request, enable,
- disable and free interrupts. The drivers do not have to know anything
- about interrupt hardware details, so they can be used on different
- platforms without code changes.
- </para>
- <para>
- This documentation is provided to developers who want to implement
- an interrupt subsystem based for their architecture, with the help
- of the generic IRQ handling layer.
- </para>
- </chapter>
-
- <chapter id="rationale">
- <title>Rationale</title>
- <para>
- The original implementation of interrupt handling in Linux uses
- the __do_IRQ() super-handler, which is able to deal with every
- type of interrupt logic.
- </para>
- <para>
- Originally, Russell King identified different types of handlers to
- build a quite universal set for the ARM interrupt handler
- implementation in Linux 2.5/2.6. He distinguished between:
- <itemizedlist>
- <listitem><para>Level type</para></listitem>
- <listitem><para>Edge type</para></listitem>
- <listitem><para>Simple type</para></listitem>
- </itemizedlist>
- During the implementation we identified another type:
- <itemizedlist>
- <listitem><para>Fast EOI type</para></listitem>
- </itemizedlist>
- In the SMP world of the __do_IRQ() super-handler another type
- was identified:
- <itemizedlist>
- <listitem><para>Per CPU type</para></listitem>
- </itemizedlist>
- </para>
- <para>
- This split implementation of high-level IRQ handlers allows us to
- optimize the flow of the interrupt handling for each specific
- interrupt type. This reduces complexity in that particular code path
- and allows the optimized handling of a given type.
- </para>
- <para>
- The original general IRQ implementation used hw_interrupt_type
- structures and their ->ack(), ->end() [etc.] callbacks to
- differentiate the flow control in the super-handler. This leads to
- a mix of flow logic and low-level hardware logic, and it also leads
- to unnecessary code duplication: for example in i386, there is an
- ioapic_level_irq and an ioapic_edge_irq IRQ-type which share many
- of the low-level details but have different flow handling.
- </para>
- <para>
- A more natural abstraction is the clean separation of the
- 'irq flow' and the 'chip details'.
- </para>
- <para>
- Analysing a couple of architecture's IRQ subsystem implementations
- reveals that most of them can use a generic set of 'irq flow'
- methods and only need to add the chip-level specific code.
- The separation is also valuable for (sub)architectures
- which need specific quirks in the IRQ flow itself but not in the
- chip details - and thus provides a more transparent IRQ subsystem
- design.
- </para>
- <para>
- Each interrupt descriptor is assigned its own high-level flow
- handler, which is normally one of the generic
- implementations. (This high-level flow handler implementation also
- makes it simple to provide demultiplexing handlers which can be
- found in embedded platforms on various architectures.)
- </para>
- <para>
- The separation makes the generic interrupt handling layer more
- flexible and extensible. For example, an (sub)architecture can
- use a generic IRQ-flow implementation for 'level type' interrupts
- and add a (sub)architecture specific 'edge type' implementation.
- </para>
- <para>
- To make the transition to the new model easier and prevent the
- breakage of existing implementations, the __do_IRQ() super-handler
- is still available. This leads to a kind of duality for the time
- being. Over time the new model should be used in more and more
- architectures, as it enables smaller and cleaner IRQ subsystems.
- It's deprecated for three years now and about to be removed.
- </para>
- </chapter>
- <chapter id="bugs">
- <title>Known Bugs And Assumptions</title>
- <para>
- None (knock on wood).
- </para>
- </chapter>
-
- <chapter id="Abstraction">
- <title>Abstraction layers</title>
- <para>
- There are three main levels of abstraction in the interrupt code:
- <orderedlist>
- <listitem><para>High-level driver API</para></listitem>
- <listitem><para>High-level IRQ flow handlers</para></listitem>
- <listitem><para>Chip-level hardware encapsulation</para></listitem>
- </orderedlist>
- </para>
- <sect1 id="Interrupt_control_flow">
- <title>Interrupt control flow</title>
- <para>
- Each interrupt is described by an interrupt descriptor structure
- irq_desc. The interrupt is referenced by an 'unsigned int' numeric
- value which selects the corresponding interrupt description structure
- in the descriptor structures array.
- The descriptor structure contains status information and pointers
- to the interrupt flow method and the interrupt chip structure
- which are assigned to this interrupt.
- </para>
- <para>
- Whenever an interrupt triggers, the low-level architecture code calls
- into the generic interrupt code by calling desc->handle_irq().
- This high-level IRQ handling function only uses desc->irq_data.chip
- primitives referenced by the assigned chip descriptor structure.
- </para>
- </sect1>
- <sect1 id="Highlevel_Driver_API">
- <title>High-level Driver API</title>
- <para>
- The high-level Driver API consists of following functions:
- <itemizedlist>
- <listitem><para>request_irq()</para></listitem>
- <listitem><para>free_irq()</para></listitem>
- <listitem><para>disable_irq()</para></listitem>
- <listitem><para>enable_irq()</para></listitem>
- <listitem><para>disable_irq_nosync() (SMP only)</para></listitem>
- <listitem><para>synchronize_irq() (SMP only)</para></listitem>
- <listitem><para>irq_set_irq_type()</para></listitem>
- <listitem><para>irq_set_irq_wake()</para></listitem>
- <listitem><para>irq_set_handler_data()</para></listitem>
- <listitem><para>irq_set_chip()</para></listitem>
- <listitem><para>irq_set_chip_data()</para></listitem>
- </itemizedlist>
- See the autogenerated function documentation for details.
- </para>
- </sect1>
- <sect1 id="Highlevel_IRQ_flow_handlers">
- <title>High-level IRQ flow handlers</title>
- <para>
- The generic layer provides a set of pre-defined irq-flow methods:
- <itemizedlist>
- <listitem><para>handle_level_irq</para></listitem>
- <listitem><para>handle_edge_irq</para></listitem>
- <listitem><para>handle_fasteoi_irq</para></listitem>
- <listitem><para>handle_simple_irq</para></listitem>
- <listitem><para>handle_percpu_irq</para></listitem>
- <listitem><para>handle_edge_eoi_irq</para></listitem>
- <listitem><para>handle_bad_irq</para></listitem>
- </itemizedlist>
- The interrupt flow handlers (either pre-defined or architecture
- specific) are assigned to specific interrupts by the architecture
- either during bootup or during device initialization.
- </para>
- <sect2 id="Default_flow_implementations">
- <title>Default flow implementations</title>
- <sect3 id="Helper_functions">
- <title>Helper functions</title>
- <para>
- The helper functions call the chip primitives and
- are used by the default flow implementations.
- The following helper functions are implemented (simplified excerpt):
- <programlisting>
-default_enable(struct irq_data *data)
-{
- desc->irq_data.chip->irq_unmask(data);
-}
-
-default_disable(struct irq_data *data)
-{
- if (!delay_disable(data))
- desc->irq_data.chip->irq_mask(data);
-}
-
-default_ack(struct irq_data *data)
-{
- chip->irq_ack(data);
-}
-
-default_mask_ack(struct irq_data *data)
-{
- if (chip->irq_mask_ack) {
- chip->irq_mask_ack(data);
- } else {
- chip->irq_mask(data);
- chip->irq_ack(data);
- }
-}
-
-noop(struct irq_data *data))
-{
-}
-
- </programlisting>
- </para>
- </sect3>
- </sect2>
- <sect2 id="Default_flow_handler_implementations">
- <title>Default flow handler implementations</title>
- <sect3 id="Default_Level_IRQ_flow_handler">
- <title>Default Level IRQ flow handler</title>
- <para>
- handle_level_irq provides a generic implementation
- for level-triggered interrupts.
- </para>
- <para>
- The following control flow is implemented (simplified excerpt):
- <programlisting>
-desc->irq_data.chip->irq_mask_ack();
-handle_irq_event(desc->action);
-desc->irq_data.chip->irq_unmask();
- </programlisting>
- </para>
- </sect3>
- <sect3 id="Default_FASTEOI_IRQ_flow_handler">
- <title>Default Fast EOI IRQ flow handler</title>
- <para>
- handle_fasteoi_irq provides a generic implementation
- for interrupts, which only need an EOI at the end of
- the handler.
- </para>
- <para>
- The following control flow is implemented (simplified excerpt):
- <programlisting>
-handle_irq_event(desc->action);
-desc->irq_data.chip->irq_eoi();
- </programlisting>
- </para>
- </sect3>
- <sect3 id="Default_Edge_IRQ_flow_handler">
- <title>Default Edge IRQ flow handler</title>
- <para>
- handle_edge_irq provides a generic implementation
- for edge-triggered interrupts.
- </para>
- <para>
- The following control flow is implemented (simplified excerpt):
- <programlisting>
-if (desc->status & running) {
- desc->irq_data.chip->irq_mask_ack();
- desc->status |= pending | masked;
- return;
-}
-desc->irq_data.chip->irq_ack();
-desc->status |= running;
-do {
- if (desc->status & masked)
- desc->irq_data.chip->irq_unmask();
- desc->status &= ~pending;
- handle_irq_event(desc->action);
-} while (status & pending);
-desc->status &= ~running;
- </programlisting>
- </para>
- </sect3>
- <sect3 id="Default_simple_IRQ_flow_handler">
- <title>Default simple IRQ flow handler</title>
- <para>
- handle_simple_irq provides a generic implementation
- for simple interrupts.
- </para>
- <para>
- Note: The simple flow handler does not call any
- handler/chip primitives.
- </para>
- <para>
- The following control flow is implemented (simplified excerpt):
- <programlisting>
-handle_irq_event(desc->action);
- </programlisting>
- </para>
- </sect3>
- <sect3 id="Default_per_CPU_flow_handler">
- <title>Default per CPU flow handler</title>
- <para>
- handle_percpu_irq provides a generic implementation
- for per CPU interrupts.
- </para>
- <para>
- Per CPU interrupts are only available on SMP and
- the handler provides a simplified version without
- locking.
- </para>
- <para>
- The following control flow is implemented (simplified excerpt):
- <programlisting>
-if (desc->irq_data.chip->irq_ack)
- desc->irq_data.chip->irq_ack();
-handle_irq_event(desc->action);
-if (desc->irq_data.chip->irq_eoi)
- desc->irq_data.chip->irq_eoi();
- </programlisting>
- </para>
- </sect3>
- <sect3 id="EOI_Edge_IRQ_flow_handler">
- <title>EOI Edge IRQ flow handler</title>
- <para>
- handle_edge_eoi_irq provides an abnomination of the edge
- handler which is solely used to tame a badly wreckaged
- irq controller on powerpc/cell.
- </para>
- </sect3>
- <sect3 id="BAD_IRQ_flow_handler">
- <title>Bad IRQ flow handler</title>
- <para>
- handle_bad_irq is used for spurious interrupts which
- have no real handler assigned..
- </para>
- </sect3>
- </sect2>
- <sect2 id="Quirks_and_optimizations">
- <title>Quirks and optimizations</title>
- <para>
- The generic functions are intended for 'clean' architectures and chips,
- which have no platform-specific IRQ handling quirks. If an architecture
- needs to implement quirks on the 'flow' level then it can do so by
- overriding the high-level irq-flow handler.
- </para>
- </sect2>
- <sect2 id="Delayed_interrupt_disable">
- <title>Delayed interrupt disable</title>
- <para>
- This per interrupt selectable feature, which was introduced by Russell
- King in the ARM interrupt implementation, does not mask an interrupt
- at the hardware level when disable_irq() is called. The interrupt is
- kept enabled and is masked in the flow handler when an interrupt event
- happens. This prevents losing edge interrupts on hardware which does
- not store an edge interrupt event while the interrupt is disabled at
- the hardware level. When an interrupt arrives while the IRQ_DISABLED
- flag is set, then the interrupt is masked at the hardware level and
- the IRQ_PENDING bit is set. When the interrupt is re-enabled by
- enable_irq() the pending bit is checked and if it is set, the
- interrupt is resent either via hardware or by a software resend
- mechanism. (It's necessary to enable CONFIG_HARDIRQS_SW_RESEND when
- you want to use the delayed interrupt disable feature and your
- hardware is not capable of retriggering an interrupt.)
- The delayed interrupt disable is not configurable.
- </para>
- </sect2>
- </sect1>
- <sect1 id="Chiplevel_hardware_encapsulation">
- <title>Chip-level hardware encapsulation</title>
- <para>
- The chip-level hardware descriptor structure irq_chip
- contains all the direct chip relevant functions, which
- can be utilized by the irq flow implementations.
- <itemizedlist>
- <listitem><para>irq_ack()</para></listitem>
- <listitem><para>irq_mask_ack() - Optional, recommended for performance</para></listitem>
- <listitem><para>irq_mask()</para></listitem>
- <listitem><para>irq_unmask()</para></listitem>
- <listitem><para>irq_eoi() - Optional, required for EOI flow handlers</para></listitem>
- <listitem><para>irq_retrigger() - Optional</para></listitem>
- <listitem><para>irq_set_type() - Optional</para></listitem>
- <listitem><para>irq_set_wake() - Optional</para></listitem>
- </itemizedlist>
- These primitives are strictly intended to mean what they say: ack means
- ACK, masking means masking of an IRQ line, etc. It is up to the flow
- handler(s) to use these basic units of low-level functionality.
- </para>
- </sect1>
- </chapter>
-
- <chapter id="doirq">
- <title>__do_IRQ entry point</title>
- <para>
- The original implementation __do_IRQ() was an alternative entry
- point for all types of interrupts. It no longer exists.
- </para>
- <para>
- This handler turned out to be not suitable for all
- interrupt hardware and was therefore reimplemented with split
- functionality for edge/level/simple/percpu interrupts. This is not
- only a functional optimization. It also shortens code paths for
- interrupts.
- </para>
- </chapter>
-
- <chapter id="locking">
- <title>Locking on SMP</title>
- <para>
- The locking of chip registers is up to the architecture that
- defines the chip primitives. The per-irq structure is
- protected via desc->lock, by the generic layer.
- </para>
- </chapter>
-
- <chapter id="genericchip">
- <title>Generic interrupt chip</title>
- <para>
- To avoid copies of identical implementations of IRQ chips the
- core provides a configurable generic interrupt chip
- implementation. Developers should check carefully whether the
- generic chip fits their needs before implementing the same
- functionality slightly differently themselves.
- </para>
-!Ekernel/irq/generic-chip.c
- </chapter>
-
- <chapter id="structs">
- <title>Structures</title>
- <para>
- This chapter contains the autogenerated documentation of the structures which are
- used in the generic IRQ layer.
- </para>
-!Iinclude/linux/irq.h
-!Iinclude/linux/interrupt.h
- </chapter>
-
- <chapter id="pubfunctions">
- <title>Public Functions Provided</title>
- <para>
- This chapter contains the autogenerated documentation of the kernel API functions
- which are exported.
- </para>
-!Ekernel/irq/manage.c
-!Ekernel/irq/chip.c
- </chapter>
-
- <chapter id="intfunctions">
- <title>Internal Functions Provided</title>
- <para>
- This chapter contains the autogenerated documentation of the internal functions.
- </para>
-!Ikernel/irq/irqdesc.c
-!Ikernel/irq/handle.c
-!Ikernel/irq/chip.c
- </chapter>
-
- <chapter id="credits">
- <title>Credits</title>
- <para>
- The following people have contributed to this document:
- <orderedlist>
- <listitem><para>Thomas Gleixner<email>tglx@linutronix.de</email></para></listitem>
- <listitem><para>Ingo Molnar<email>mingo@elte.hu</email></para></listitem>
- </orderedlist>
- </para>
- </chapter>
-</book>
diff --git a/Documentation/DocBook/kernel-api.tmpl b/Documentation/DocBook/kernel-api.tmpl
deleted file mode 100644
index ecfd0ea..0000000
--- a/Documentation/DocBook/kernel-api.tmpl
+++ /dev/null
@@ -1,331 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="LinuxKernelAPI">
- <bookinfo>
- <title>The Linux Kernel API</title>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License as published by the Free Software Foundation; either
- version 2 of the License, or (at your option) any later
- version.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
-
- <chapter id="adt">
- <title>Data Types</title>
- <sect1><title>Doubly Linked Lists</title>
-!Iinclude/linux/list.h
- </sect1>
- </chapter>
-
- <chapter id="libc">
- <title>Basic C Library Functions</title>
-
- <para>
- When writing drivers, you cannot in general use routines which are
- from the C Library. Some of the functions have been found generally
- useful and they are listed below. The behaviour of these functions
- may vary slightly from those defined by ANSI, and these deviations
- are noted in the text.
- </para>
-
- <sect1><title>String Conversions</title>
-!Elib/vsprintf.c
-!Finclude/linux/kernel.h kstrtol
-!Finclude/linux/kernel.h kstrtoul
-!Elib/kstrtox.c
- </sect1>
- <sect1><title>String Manipulation</title>
-<!-- All functions are exported at now
-X!Ilib/string.c
- -->
-!Elib/string.c
- </sect1>
- <sect1><title>Bit Operations</title>
-!Iarch/x86/include/asm/bitops.h
- </sect1>
- </chapter>
-
- <chapter id="kernel-lib">
- <title>Basic Kernel Library Functions</title>
-
- <para>
- The Linux kernel provides more basic utility functions.
- </para>
-
- <sect1><title>Bitmap Operations</title>
-!Elib/bitmap.c
-!Ilib/bitmap.c
- </sect1>
-
- <sect1><title>Command-line Parsing</title>
-!Elib/cmdline.c
- </sect1>
-
- <sect1 id="crc"><title>CRC Functions</title>
-!Elib/crc7.c
-!Elib/crc16.c
-!Elib/crc-itu-t.c
-!Elib/crc32.c
-!Elib/crc-ccitt.c
- </sect1>
-
- <sect1 id="idr"><title>idr/ida Functions</title>
-!Pinclude/linux/idr.h idr sync
-!Plib/idr.c IDA description
-!Elib/idr.c
- </sect1>
- </chapter>
-
- <chapter id="mm">
- <title>Memory Management in Linux</title>
- <sect1><title>The Slab Cache</title>
-!Iinclude/linux/slab.h
-!Emm/slab.c
-!Emm/util.c
- </sect1>
- <sect1><title>User Space Memory Access</title>
-!Iarch/x86/include/asm/uaccess_32.h
-!Earch/x86/lib/usercopy_32.c
- </sect1>
- <sect1><title>More Memory Management Functions</title>
-!Emm/readahead.c
-!Emm/filemap.c
-!Emm/memory.c
-!Emm/vmalloc.c
-!Imm/page_alloc.c
-!Emm/mempool.c
-!Emm/dmapool.c
-!Emm/page-writeback.c
-!Emm/truncate.c
- </sect1>
- </chapter>
-
-
- <chapter id="ipc">
- <title>Kernel IPC facilities</title>
-
- <sect1><title>IPC utilities</title>
-!Iipc/util.c
- </sect1>
- </chapter>
-
- <chapter id="kfifo">
- <title>FIFO Buffer</title>
- <sect1><title>kfifo interface</title>
-!Iinclude/linux/kfifo.h
- </sect1>
- </chapter>
-
- <chapter id="relayfs">
- <title>relay interface support</title>
-
- <para>
- Relay interface support
- is designed to provide an efficient mechanism for tools and
- facilities to relay large amounts of data from kernel space to
- user space.
- </para>
-
- <sect1><title>relay interface</title>
-!Ekernel/relay.c
-!Ikernel/relay.c
- </sect1>
- </chapter>
-
- <chapter id="modload">
- <title>Module Support</title>
- <sect1><title>Module Loading</title>
-!Ekernel/kmod.c
- </sect1>
- <sect1><title>Inter Module support</title>
- <para>
- Refer to the file kernel/module.c for more information.
- </para>
-<!-- FIXME: Removed for now since no structured comments in source
-X!Ekernel/module.c
--->
- </sect1>
- </chapter>
-
- <chapter id="hardware">
- <title>Hardware Interfaces</title>
- <sect1><title>Interrupt Handling</title>
-!Ekernel/irq/manage.c
- </sect1>
-
- <sect1><title>DMA Channels</title>
-!Ekernel/dma.c
- </sect1>
-
- <sect1><title>Resources Management</title>
-!Ikernel/resource.c
-!Ekernel/resource.c
- </sect1>
-
- <sect1><title>MTRR Handling</title>
-!Earch/x86/kernel/cpu/mtrr/main.c
- </sect1>
-
- <sect1><title>PCI Support Library</title>
-!Edrivers/pci/pci.c
-!Edrivers/pci/pci-driver.c
-!Edrivers/pci/remove.c
-!Edrivers/pci/search.c
-!Edrivers/pci/msi.c
-!Edrivers/pci/bus.c
-!Edrivers/pci/access.c
-!Edrivers/pci/irq.c
-!Edrivers/pci/htirq.c
-<!-- FIXME: Removed for now since no structured comments in source
-X!Edrivers/pci/hotplug.c
--->
-!Edrivers/pci/probe.c
-!Edrivers/pci/slot.c
-!Edrivers/pci/rom.c
-!Edrivers/pci/iov.c
-!Idrivers/pci/pci-sysfs.c
- </sect1>
- <sect1><title>PCI Hotplug Support Library</title>
-!Edrivers/pci/hotplug/pci_hotplug_core.c
- </sect1>
- </chapter>
-
- <chapter id="firmware">
- <title>Firmware Interfaces</title>
- <sect1><title>DMI Interfaces</title>
-!Edrivers/firmware/dmi_scan.c
- </sect1>
- <sect1><title>EDD Interfaces</title>
-!Idrivers/firmware/edd.c
- </sect1>
- </chapter>
-
- <chapter id="security">
- <title>Security Framework</title>
-!Isecurity/security.c
-!Esecurity/inode.c
- </chapter>
-
- <chapter id="audit">
- <title>Audit Interfaces</title>
-!Ekernel/audit.c
-!Ikernel/auditsc.c
-!Ikernel/auditfilter.c
- </chapter>
-
- <chapter id="accounting">
- <title>Accounting Framework</title>
-!Ikernel/acct.c
- </chapter>
-
- <chapter id="blkdev">
- <title>Block Devices</title>
-!Eblock/blk-core.c
-!Iblock/blk-core.c
-!Eblock/blk-map.c
-!Iblock/blk-sysfs.c
-!Eblock/blk-settings.c
-!Eblock/blk-exec.c
-!Eblock/blk-flush.c
-!Eblock/blk-lib.c
-!Eblock/blk-tag.c
-!Iblock/blk-tag.c
-!Eblock/blk-integrity.c
-!Ikernel/trace/blktrace.c
-!Iblock/genhd.c
-!Eblock/genhd.c
- </chapter>
-
- <chapter id="chrdev">
- <title>Char devices</title>
-!Efs/char_dev.c
- </chapter>
-
- <chapter id="miscdev">
- <title>Miscellaneous Devices</title>
-!Edrivers/char/misc.c
- </chapter>
-
- <chapter id="clk">
- <title>Clock Framework</title>
-
- <para>
- The clock framework defines programming interfaces to support
- software management of the system clock tree.
- This framework is widely used with System-On-Chip (SOC) platforms
- to support power management and various devices which may need
- custom clock rates.
- Note that these "clocks" don't relate to timekeeping or real
- time clocks (RTCs), each of which have separate frameworks.
- These <structname>struct clk</structname> instances may be used
- to manage for example a 96 MHz signal that is used to shift bits
- into and out of peripherals or busses, or otherwise trigger
- synchronous state machine transitions in system hardware.
- </para>
-
- <para>
- Power management is supported by explicit software clock gating:
- unused clocks are disabled, so the system doesn't waste power
- changing the state of transistors that aren't in active use.
- On some systems this may be backed by hardware clock gating,
- where clocks are gated without being disabled in software.
- Sections of chips that are powered but not clocked may be able
- to retain their last state.
- This low power state is often called a <emphasis>retention
- mode</emphasis>.
- This mode still incurs leakage currents, especially with finer
- circuit geometries, but for CMOS circuits power is mostly used
- by clocked state changes.
- </para>
-
- <para>
- Power-aware drivers only enable their clocks when the device
- they manage is in active use. Also, system sleep states often
- differ according to which clock domains are active: while a
- "standby" state may allow wakeup from several active domains, a
- "mem" (suspend-to-RAM) state may require a more wholesale shutdown
- of clocks derived from higher speed PLLs and oscillators, limiting
- the number of possible wakeup event sources. A driver's suspend
- method may need to be aware of system-specific clock constraints
- on the target sleep state.
- </para>
-
- <para>
- Some platforms support programmable clock generators. These
- can be used by external chips of various kinds, such as other
- CPUs, multimedia codecs, and devices with strict requirements
- for interface clocking.
- </para>
-
-!Iinclude/linux/clk.h
- </chapter>
-
-</book>
diff --git a/Documentation/DocBook/writing_musb_glue_layer.tmpl b/Documentation/DocBook/writing_musb_glue_layer.tmpl
deleted file mode 100644
index 837eca7..0000000
--- a/Documentation/DocBook/writing_musb_glue_layer.tmpl
+++ /dev/null
@@ -1,873 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="Writing-MUSB-Glue-Layer">
- <bookinfo>
- <title>Writing an MUSB Glue Layer</title>
-
- <authorgroup>
- <author>
- <firstname>Apelete</firstname>
- <surname>Seketeli</surname>
- <affiliation>
- <address>
- <email>apelete at seketeli.net</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
-
- <copyright>
- <year>2014</year>
- <holder>Apelete Seketeli</holder>
- </copyright>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute it
- and/or modify it under the terms of the GNU General Public
- License as published by the Free Software Foundation; either
- version 2 of the License, or (at your option) any later version.
- </para>
-
- <para>
- This documentation is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public License
- along with this documentation; if not, write to the Free Software
- Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
- 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the Linux kernel source
- tree.
- </para>
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
-
- <chapter id="introduction">
- <title>Introduction</title>
- <para>
- The Linux MUSB subsystem is part of the larger Linux USB
- subsystem. It provides support for embedded USB Device Controllers
- (UDC) that do not use Universal Host Controller Interface (UHCI)
- or Open Host Controller Interface (OHCI).
- </para>
- <para>
- Instead, these embedded UDC rely on the USB On-the-Go (OTG)
- specification which they implement at least partially. The silicon
- reference design used in most cases is the Multipoint USB
- Highspeed Dual-Role Controller (MUSB HDRC) found in the Mentor
- Graphics Inventra™ design.
- </para>
- <para>
- As a self-taught exercise I have written an MUSB glue layer for
- the Ingenic JZ4740 SoC, modelled after the many MUSB glue layers
- in the kernel source tree. This layer can be found at
- drivers/usb/musb/jz4740.c. In this documentation I will walk
- through the basics of the jz4740.c glue layer, explaining the
- different pieces and what needs to be done in order to write your
- own device glue layer.
- </para>
- </chapter>
-
- <chapter id="linux-musb-basics">
- <title>Linux MUSB Basics</title>
- <para>
- To get started on the topic, please read USB On-the-Go Basics (see
- Resources) which provides an introduction of USB OTG operation at
- the hardware level. A couple of wiki pages by Texas Instruments
- and Analog Devices also provide an overview of the Linux kernel
- MUSB configuration, albeit focused on some specific devices
- provided by these companies. Finally, getting acquainted with the
- USB specification at USB home page may come in handy, with
- practical instance provided through the Writing USB Device Drivers
- documentation (again, see Resources).
- </para>
- <para>
- Linux USB stack is a layered architecture in which the MUSB
- controller hardware sits at the lowest. The MUSB controller driver
- abstract the MUSB controller hardware to the Linux USB stack.
- </para>
- <programlisting>
- ------------------------
- | | <------- drivers/usb/gadget
- | Linux USB Core Stack | <------- drivers/usb/host
- | | <------- drivers/usb/core
- ------------------------
- ⬍
- --------------------------
- | | <------ drivers/usb/musb/musb_gadget.c
- | MUSB Controller driver | <------ drivers/usb/musb/musb_host.c
- | | <------ drivers/usb/musb/musb_core.c
- --------------------------
- ⬍
- ---------------------------------
- | MUSB Platform Specific Driver |
- | | <-- drivers/usb/musb/jz4740.c
- | aka "Glue Layer" |
- ---------------------------------
- ⬍
- ---------------------------------
- | MUSB Controller Hardware |
- ---------------------------------
- </programlisting>
- <para>
- As outlined above, the glue layer is actually the platform
- specific code sitting in between the controller driver and the
- controller hardware.
- </para>
- <para>
- Just like a Linux USB driver needs to register itself with the
- Linux USB subsystem, the MUSB glue layer needs first to register
- itself with the MUSB controller driver. This will allow the
- controller driver to know about which device the glue layer
- supports and which functions to call when a supported device is
- detected or released; remember we are talking about an embedded
- controller chip here, so no insertion or removal at run-time.
- </para>
- <para>
- All of this information is passed to the MUSB controller driver
- through a platform_driver structure defined in the glue layer as:
- </para>
- <programlisting linenumbering="numbered">
-static struct platform_driver jz4740_driver = {
- .probe = jz4740_probe,
- .remove = jz4740_remove,
- .driver = {
- .name = "musb-jz4740",
- },
-};
- </programlisting>
- <para>
- The probe and remove function pointers are called when a matching
- device is detected and, respectively, released. The name string
- describes the device supported by this glue layer. In the current
- case it matches a platform_device structure declared in
- arch/mips/jz4740/platform.c. Note that we are not using device
- tree bindings here.
- </para>
- <para>
- In order to register itself to the controller driver, the glue
- layer goes through a few steps, basically allocating the
- controller hardware resources and initialising a couple of
- circuits. To do so, it needs to keep track of the information used
- throughout these steps. This is done by defining a private
- jz4740_glue structure:
- </para>
- <programlisting linenumbering="numbered">
-struct jz4740_glue {
- struct device *dev;
- struct platform_device *musb;
- struct clk *clk;
-};
- </programlisting>
- <para>
- The dev and musb members are both device structure variables. The
- first one holds generic information about the device, since it's
- the basic device structure, and the latter holds information more
- closely related to the subsystem the device is registered to. The
- clk variable keeps information related to the device clock
- operation.
- </para>
- <para>
- Let's go through the steps of the probe function that leads the
- glue layer to register itself to the controller driver.
- </para>
- <para>
- N.B.: For the sake of readability each function will be split in
- logical parts, each part being shown as if it was independent from
- the others.
- </para>
- <programlisting linenumbering="numbered">
-static int jz4740_probe(struct platform_device *pdev)
-{
- struct platform_device *musb;
- struct jz4740_glue *glue;
- struct clk *clk;
- int ret;
-
- glue = devm_kzalloc(&pdev->dev, sizeof(*glue), GFP_KERNEL);
- if (!glue)
- return -ENOMEM;
-
- musb = platform_device_alloc("musb-hdrc", PLATFORM_DEVID_AUTO);
- if (!musb) {
- dev_err(&pdev->dev, "failed to allocate musb device\n");
- return -ENOMEM;
- }
-
- clk = devm_clk_get(&pdev->dev, "udc");
- if (IS_ERR(clk)) {
- dev_err(&pdev->dev, "failed to get clock\n");
- ret = PTR_ERR(clk);
- goto err_platform_device_put;
- }
-
- ret = clk_prepare_enable(clk);
- if (ret) {
- dev_err(&pdev->dev, "failed to enable clock\n");
- goto err_platform_device_put;
- }
-
- musb->dev.parent = &pdev->dev;
-
- glue->dev = &pdev->dev;
- glue->musb = musb;
- glue->clk = clk;
-
- return 0;
-
-err_platform_device_put:
- platform_device_put(musb);
- return ret;
-}
- </programlisting>
- <para>
- The first few lines of the probe function allocate and assign the
- glue, musb and clk variables. The GFP_KERNEL flag (line 8) allows
- the allocation process to sleep and wait for memory, thus being
- usable in a blocking situation. The PLATFORM_DEVID_AUTO flag (line
- 12) allows automatic allocation and management of device IDs in
- order to avoid device namespace collisions with explicit IDs. With
- devm_clk_get() (line 18) the glue layer allocates the clock -- the
- <literal>devm_</literal> prefix indicates that clk_get() is
- managed: it automatically frees the allocated clock resource data
- when the device is released -- and enable it.
- </para>
- <para>
- Then comes the registration steps:
- </para>
- <programlisting linenumbering="numbered">
-static int jz4740_probe(struct platform_device *pdev)
-{
- struct musb_hdrc_platform_data *pdata = &jz4740_musb_platform_data;
-
- pdata->platform_ops = &jz4740_musb_ops;
-
- platform_set_drvdata(pdev, glue);
-
- ret = platform_device_add_resources(musb, pdev->resource,
- pdev->num_resources);
- if (ret) {
- dev_err(&pdev->dev, "failed to add resources\n");
- goto err_clk_disable;
- }
-
- ret = platform_device_add_data(musb, pdata, sizeof(*pdata));
- if (ret) {
- dev_err(&pdev->dev, "failed to add platform_data\n");
- goto err_clk_disable;
- }
-
- return 0;
-
-err_clk_disable:
- clk_disable_unprepare(clk);
-err_platform_device_put:
- platform_device_put(musb);
- return ret;
-}
- </programlisting>
- <para>
- The first step is to pass the device data privately held by the
- glue layer on to the controller driver through
- platform_set_drvdata() (line 7). Next is passing on the device
- resources information, also privately held at that point, through
- platform_device_add_resources() (line 9).
- </para>
- <para>
- Finally comes passing on the platform specific data to the
- controller driver (line 16). Platform data will be discussed in
- <link linkend="device-platform-data">Chapter 4</link>, but here
- we are looking at the platform_ops function pointer (line 5) in
- musb_hdrc_platform_data structure (line 3). This function
- pointer allows the MUSB controller driver to know which function
- to call for device operation:
- </para>
- <programlisting linenumbering="numbered">
-static const struct musb_platform_ops jz4740_musb_ops = {
- .init = jz4740_musb_init,
- .exit = jz4740_musb_exit,
-};
- </programlisting>
- <para>
- Here we have the minimal case where only init and exit functions
- are called by the controller driver when needed. Fact is the
- JZ4740 MUSB controller is a basic controller, lacking some
- features found in other controllers, otherwise we may also have
- pointers to a few other functions like a power management function
- or a function to switch between OTG and non-OTG modes, for
- instance.
- </para>
- <para>
- At that point of the registration process, the controller driver
- actually calls the init function:
- </para>
- <programlisting linenumbering="numbered">
-static int jz4740_musb_init(struct musb *musb)
-{
- musb->xceiv = usb_get_phy(USB_PHY_TYPE_USB2);
- if (!musb->xceiv) {
- pr_err("HS UDC: no transceiver configured\n");
- return -ENODEV;
- }
-
- /* Silicon does not implement ConfigData register.
- * Set dyn_fifo to avoid reading EP config from hardware.
- */
- musb->dyn_fifo = true;
-
- musb->isr = jz4740_musb_interrupt;
-
- return 0;
-}
- </programlisting>
- <para>
- The goal of jz4740_musb_init() is to get hold of the transceiver
- driver data of the MUSB controller hardware and pass it on to the
- MUSB controller driver, as usual. The transceiver is the circuitry
- inside the controller hardware responsible for sending/receiving
- the USB data. Since it is an implementation of the physical layer
- of the OSI model, the transceiver is also referred to as PHY.
- </para>
- <para>
- Getting hold of the MUSB PHY driver data is done with
- usb_get_phy() which returns a pointer to the structure
- containing the driver instance data. The next couple of
- instructions (line 12 and 14) are used as a quirk and to setup
- IRQ handling respectively. Quirks and IRQ handling will be
- discussed later in <link linkend="device-quirks">Chapter
- 5</link> and <link linkend="handling-irqs">Chapter 3</link>.
- </para>
- <programlisting linenumbering="numbered">
-static int jz4740_musb_exit(struct musb *musb)
-{
- usb_put_phy(musb->xceiv);
-
- return 0;
-}
- </programlisting>
- <para>
- Acting as the counterpart of init, the exit function releases the
- MUSB PHY driver when the controller hardware itself is about to be
- released.
- </para>
- <para>
- Again, note that init and exit are fairly simple in this case due
- to the basic set of features of the JZ4740 controller hardware.
- When writing an musb glue layer for a more complex controller
- hardware, you might need to take care of more processing in those
- two functions.
- </para>
- <para>
- Returning from the init function, the MUSB controller driver jumps
- back into the probe function:
- </para>
- <programlisting linenumbering="numbered">
-static int jz4740_probe(struct platform_device *pdev)
-{
- ret = platform_device_add(musb);
- if (ret) {
- dev_err(&pdev->dev, "failed to register musb device\n");
- goto err_clk_disable;
- }
-
- return 0;
-
-err_clk_disable:
- clk_disable_unprepare(clk);
-err_platform_device_put:
- platform_device_put(musb);
- return ret;
-}
- </programlisting>
- <para>
- This is the last part of the device registration process where the
- glue layer adds the controller hardware device to Linux kernel
- device hierarchy: at this stage, all known information about the
- device is passed on to the Linux USB core stack.
- </para>
- <programlisting linenumbering="numbered">
-static int jz4740_remove(struct platform_device *pdev)
-{
- struct jz4740_glue *glue = platform_get_drvdata(pdev);
-
- platform_device_unregister(glue->musb);
- clk_disable_unprepare(glue->clk);
-
- return 0;
-}
- </programlisting>
- <para>
- Acting as the counterpart of probe, the remove function unregister
- the MUSB controller hardware (line 5) and disable the clock (line
- 6), allowing it to be gated.
- </para>
- </chapter>
-
- <chapter id="handling-irqs">
- <title>Handling IRQs</title>
- <para>
- Additionally to the MUSB controller hardware basic setup and
- registration, the glue layer is also responsible for handling the
- IRQs:
- </para>
- <programlisting linenumbering="numbered">
-static irqreturn_t jz4740_musb_interrupt(int irq, void *__hci)
-{
- unsigned long flags;
- irqreturn_t retval = IRQ_NONE;
- struct musb *musb = __hci;
-
- spin_lock_irqsave(&musb->lock, flags);
-
- musb->int_usb = musb_readb(musb->mregs, MUSB_INTRUSB);
- musb->int_tx = musb_readw(musb->mregs, MUSB_INTRTX);
- musb->int_rx = musb_readw(musb->mregs, MUSB_INTRRX);
-
- /*
- * The controller is gadget only, the state of the host mode IRQ bits is
- * undefined. Mask them to make sure that the musb driver core will
- * never see them set
- */
- musb->int_usb &= MUSB_INTR_SUSPEND | MUSB_INTR_RESUME |
- MUSB_INTR_RESET | MUSB_INTR_SOF;
-
- if (musb->int_usb || musb->int_tx || musb->int_rx)
- retval = musb_interrupt(musb);
-
- spin_unlock_irqrestore(&musb->lock, flags);
-
- return retval;
-}
- </programlisting>
- <para>
- Here the glue layer mostly has to read the relevant hardware
- registers and pass their values on to the controller driver which
- will handle the actual event that triggered the IRQ.
- </para>
- <para>
- The interrupt handler critical section is protected by the
- spin_lock_irqsave() and counterpart spin_unlock_irqrestore()
- functions (line 7 and 24 respectively), which prevent the
- interrupt handler code to be run by two different threads at the
- same time.
- </para>
- <para>
- Then the relevant interrupt registers are read (line 9 to 11):
- </para>
- <itemizedlist>
- <listitem>
- <para>
- MUSB_INTRUSB: indicates which USB interrupts are currently
- active,
- </para>
- </listitem>
- <listitem>
- <para>
- MUSB_INTRTX: indicates which of the interrupts for TX
- endpoints are currently active,
- </para>
- </listitem>
- <listitem>
- <para>
- MUSB_INTRRX: indicates which of the interrupts for TX
- endpoints are currently active.
- </para>
- </listitem>
- </itemizedlist>
- <para>
- Note that musb_readb() is used to read 8-bit registers at most,
- while musb_readw() allows us to read at most 16-bit registers.
- There are other functions that can be used depending on the size
- of your device registers. See musb_io.h for more information.
- </para>
- <para>
- Instruction on line 18 is another quirk specific to the JZ4740
- USB device controller, which will be discussed later in <link
- linkend="device-quirks">Chapter 5</link>.
- </para>
- <para>
- The glue layer still needs to register the IRQ handler though.
- Remember the instruction on line 14 of the init function:
- </para>
- <programlisting linenumbering="numbered">
-static int jz4740_musb_init(struct musb *musb)
-{
- musb->isr = jz4740_musb_interrupt;
-
- return 0;
-}
- </programlisting>
- <para>
- This instruction sets a pointer to the glue layer IRQ handler
- function, in order for the controller hardware to call the handler
- back when an IRQ comes from the controller hardware. The interrupt
- handler is now implemented and registered.
- </para>
- </chapter>
-
- <chapter id="device-platform-data">
- <title>Device Platform Data</title>
- <para>
- In order to write an MUSB glue layer, you need to have some data
- describing the hardware capabilities of your controller hardware,
- which is called the platform data.
- </para>
- <para>
- Platform data is specific to your hardware, though it may cover a
- broad range of devices, and is generally found somewhere in the
- arch/ directory, depending on your device architecture.
- </para>
- <para>
- For instance, platform data for the JZ4740 SoC is found in
- arch/mips/jz4740/platform.c. In the platform.c file each device of
- the JZ4740 SoC is described through a set of structures.
- </para>
- <para>
- Here is the part of arch/mips/jz4740/platform.c that covers the
- USB Device Controller (UDC):
- </para>
- <programlisting linenumbering="numbered">
-/* USB Device Controller */
-struct platform_device jz4740_udc_xceiv_device = {
- .name = "usb_phy_gen_xceiv",
- .id = 0,
-};
-
-static struct resource jz4740_udc_resources[] = {
- [0] = {
- .start = JZ4740_UDC_BASE_ADDR,
- .end = JZ4740_UDC_BASE_ADDR + 0x10000 - 1,
- .flags = IORESOURCE_MEM,
- },
- [1] = {
- .start = JZ4740_IRQ_UDC,
- .end = JZ4740_IRQ_UDC,
- .flags = IORESOURCE_IRQ,
- .name = "mc",
- },
-};
-
-struct platform_device jz4740_udc_device = {
- .name = "musb-jz4740",
- .id = -1,
- .dev = {
- .dma_mask = &jz4740_udc_device.dev.coherent_dma_mask,
- .coherent_dma_mask = DMA_BIT_MASK(32),
- },
- .num_resources = ARRAY_SIZE(jz4740_udc_resources),
- .resource = jz4740_udc_resources,
-};
- </programlisting>
- <para>
- The jz4740_udc_xceiv_device platform device structure (line 2)
- describes the UDC transceiver with a name and id number.
- </para>
- <para>
- At the time of this writing, note that
- "usb_phy_gen_xceiv" is the specific name to be used for
- all transceivers that are either built-in with reference USB IP or
- autonomous and doesn't require any PHY programming. You will need
- to set CONFIG_NOP_USB_XCEIV=y in the kernel configuration to make
- use of the corresponding transceiver driver. The id field could be
- set to -1 (equivalent to PLATFORM_DEVID_NONE), -2 (equivalent to
- PLATFORM_DEVID_AUTO) or start with 0 for the first device of this
- kind if we want a specific id number.
- </para>
- <para>
- The jz4740_udc_resources resource structure (line 7) defines the
- UDC registers base addresses.
- </para>
- <para>
- The first array (line 9 to 11) defines the UDC registers base
- memory addresses: start points to the first register memory
- address, end points to the last register memory address and the
- flags member defines the type of resource we are dealing with. So
- IORESOURCE_MEM is used to define the registers memory addresses.
- The second array (line 14 to 17) defines the UDC IRQ registers
- addresses. Since there is only one IRQ register available for the
- JZ4740 UDC, start and end point at the same address. The
- IORESOURCE_IRQ flag tells that we are dealing with IRQ resources,
- and the name "mc" is in fact hard-coded in the MUSB core
- in order for the controller driver to retrieve this IRQ resource
- by querying it by its name.
- </para>
- <para>
- Finally, the jz4740_udc_device platform device structure (line 21)
- describes the UDC itself.
- </para>
- <para>
- The "musb-jz4740" name (line 22) defines the MUSB
- driver that is used for this device; remember this is in fact
- the name that we used in the jz4740_driver platform driver
- structure in <link linkend="linux-musb-basics">Chapter
- 2</link>. The id field (line 23) is set to -1 (equivalent to
- PLATFORM_DEVID_NONE) since we do not need an id for the device:
- the MUSB controller driver was already set to allocate an
- automatic id in <link linkend="linux-musb-basics">Chapter
- 2</link>. In the dev field we care for DMA related information
- here. The dma_mask field (line 25) defines the width of the DMA
- mask that is going to be used, and coherent_dma_mask (line 26)
- has the same purpose but for the alloc_coherent DMA mappings: in
- both cases we are using a 32 bits mask. Then the resource field
- (line 29) is simply a pointer to the resource structure defined
- before, while the num_resources field (line 28) keeps track of
- the number of arrays defined in the resource structure (in this
- case there were two resource arrays defined before).
- </para>
- <para>
- With this quick overview of the UDC platform data at the arch/
- level now done, let's get back to the MUSB glue layer specific
- platform data in drivers/usb/musb/jz4740.c:
- </para>
- <programlisting linenumbering="numbered">
-static struct musb_hdrc_config jz4740_musb_config = {
- /* Silicon does not implement USB OTG. */
- .multipoint = 0,
- /* Max EPs scanned, driver will decide which EP can be used. */
- .num_eps = 4,
- /* RAMbits needed to configure EPs from table */
- .ram_bits = 9,
- .fifo_cfg = jz4740_musb_fifo_cfg,
- .fifo_cfg_size = ARRAY_SIZE(jz4740_musb_fifo_cfg),
-};
-
-static struct musb_hdrc_platform_data jz4740_musb_platform_data = {
- .mode = MUSB_PERIPHERAL,
- .config = &jz4740_musb_config,
-};
- </programlisting>
- <para>
- First the glue layer configures some aspects of the controller
- driver operation related to the controller hardware specifics.
- This is done through the jz4740_musb_config musb_hdrc_config
- structure.
- </para>
- <para>
- Defining the OTG capability of the controller hardware, the
- multipoint member (line 3) is set to 0 (equivalent to false)
- since the JZ4740 UDC is not OTG compatible. Then num_eps (line
- 5) defines the number of USB endpoints of the controller
- hardware, including endpoint 0: here we have 3 endpoints +
- endpoint 0. Next is ram_bits (line 7) which is the width of the
- RAM address bus for the MUSB controller hardware. This
- information is needed when the controller driver cannot
- automatically configure endpoints by reading the relevant
- controller hardware registers. This issue will be discussed when
- we get to device quirks in <link linkend="device-quirks">Chapter
- 5</link>. Last two fields (line 8 and 9) are also about device
- quirks: fifo_cfg points to the USB endpoints configuration table
- and fifo_cfg_size keeps track of the size of the number of
- entries in that configuration table. More on that later in <link
- linkend="device-quirks">Chapter 5</link>.
- </para>
- <para>
- Then this configuration is embedded inside
- jz4740_musb_platform_data musb_hdrc_platform_data structure (line
- 11): config is a pointer to the configuration structure itself,
- and mode tells the controller driver if the controller hardware
- may be used as MUSB_HOST only, MUSB_PERIPHERAL only or MUSB_OTG
- which is a dual mode.
- </para>
- <para>
- Remember that jz4740_musb_platform_data is then used to convey
- platform data information as we have seen in the probe function
- in <link linkend="linux-musb-basics">Chapter 2</link>
- </para>
- </chapter>
-
- <chapter id="device-quirks">
- <title>Device Quirks</title>
- <para>
- Completing the platform data specific to your device, you may also
- need to write some code in the glue layer to work around some
- device specific limitations. These quirks may be due to some
- hardware bugs, or simply be the result of an incomplete
- implementation of the USB On-the-Go specification.
- </para>
- <para>
- The JZ4740 UDC exhibits such quirks, some of which we will discuss
- here for the sake of insight even though these might not be found
- in the controller hardware you are working on.
- </para>
- <para>
- Let's get back to the init function first:
- </para>
- <programlisting linenumbering="numbered">
-static int jz4740_musb_init(struct musb *musb)
-{
- musb->xceiv = usb_get_phy(USB_PHY_TYPE_USB2);
- if (!musb->xceiv) {
- pr_err("HS UDC: no transceiver configured\n");
- return -ENODEV;
- }
-
- /* Silicon does not implement ConfigData register.
- * Set dyn_fifo to avoid reading EP config from hardware.
- */
- musb->dyn_fifo = true;
-
- musb->isr = jz4740_musb_interrupt;
-
- return 0;
-}
- </programlisting>
- <para>
- Instruction on line 12 helps the MUSB controller driver to work
- around the fact that the controller hardware is missing registers
- that are used for USB endpoints configuration.
- </para>
- <para>
- Without these registers, the controller driver is unable to read
- the endpoints configuration from the hardware, so we use line 12
- instruction to bypass reading the configuration from silicon, and
- rely on a hard-coded table that describes the endpoints
- configuration instead:
- </para>
- <programlisting linenumbering="numbered">
-static struct musb_fifo_cfg jz4740_musb_fifo_cfg[] = {
-{ .hw_ep_num = 1, .style = FIFO_TX, .maxpacket = 512, },
-{ .hw_ep_num = 1, .style = FIFO_RX, .maxpacket = 512, },
-{ .hw_ep_num = 2, .style = FIFO_TX, .maxpacket = 64, },
-};
- </programlisting>
- <para>
- Looking at the configuration table above, we see that each
- endpoints is described by three fields: hw_ep_num is the endpoint
- number, style is its direction (either FIFO_TX for the controller
- driver to send packets in the controller hardware, or FIFO_RX to
- receive packets from hardware), and maxpacket defines the maximum
- size of each data packet that can be transmitted over that
- endpoint. Reading from the table, the controller driver knows that
- endpoint 1 can be used to send and receive USB data packets of 512
- bytes at once (this is in fact a bulk in/out endpoint), and
- endpoint 2 can be used to send data packets of 64 bytes at once
- (this is in fact an interrupt endpoint).
- </para>
- <para>
- Note that there is no information about endpoint 0 here: that one
- is implemented by default in every silicon design, with a
- predefined configuration according to the USB specification. For
- more examples of endpoint configuration tables, see musb_core.c.
- </para>
- <para>
- Let's now get back to the interrupt handler function:
- </para>
- <programlisting linenumbering="numbered">
-static irqreturn_t jz4740_musb_interrupt(int irq, void *__hci)
-{
- unsigned long flags;
- irqreturn_t retval = IRQ_NONE;
- struct musb *musb = __hci;
-
- spin_lock_irqsave(&musb->lock, flags);
-
- musb->int_usb = musb_readb(musb->mregs, MUSB_INTRUSB);
- musb->int_tx = musb_readw(musb->mregs, MUSB_INTRTX);
- musb->int_rx = musb_readw(musb->mregs, MUSB_INTRRX);
-
- /*
- * The controller is gadget only, the state of the host mode IRQ bits is
- * undefined. Mask them to make sure that the musb driver core will
- * never see them set
- */
- musb->int_usb &= MUSB_INTR_SUSPEND | MUSB_INTR_RESUME |
- MUSB_INTR_RESET | MUSB_INTR_SOF;
-
- if (musb->int_usb || musb->int_tx || musb->int_rx)
- retval = musb_interrupt(musb);
-
- spin_unlock_irqrestore(&musb->lock, flags);
-
- return retval;
-}
- </programlisting>
- <para>
- Instruction on line 18 above is a way for the controller driver to
- work around the fact that some interrupt bits used for USB host
- mode operation are missing in the MUSB_INTRUSB register, thus left
- in an undefined hardware state, since this MUSB controller
- hardware is used in peripheral mode only. As a consequence, the
- glue layer masks these missing bits out to avoid parasite
- interrupts by doing a logical AND operation between the value read
- from MUSB_INTRUSB and the bits that are actually implemented in
- the register.
- </para>
- <para>
- These are only a couple of the quirks found in the JZ4740 USB
- device controller. Some others were directly addressed in the MUSB
- core since the fixes were generic enough to provide a better
- handling of the issues for others controller hardware eventually.
- </para>
- </chapter>
-
- <chapter id="conclusion">
- <title>Conclusion</title>
- <para>
- Writing a Linux MUSB glue layer should be a more accessible task,
- as this documentation tries to show the ins and outs of this
- exercise.
- </para>
- <para>
- The JZ4740 USB device controller being fairly simple, I hope its
- glue layer serves as a good example for the curious mind. Used
- with the current MUSB glue layers, this documentation should
- provide enough guidance to get started; should anything gets out
- of hand, the linux-usb mailing list archive is another helpful
- resource to browse through.
- </para>
- </chapter>
-
- <chapter id="acknowledgements">
- <title>Acknowledgements</title>
- <para>
- Many thanks to Lars-Peter Clausen and Maarten ter Huurne for
- answering my questions while I was writing the JZ4740 glue layer
- and for helping me out getting the code in good shape.
- </para>
- <para>
- I would also like to thank the Qi-Hardware community at large for
- its cheerful guidance and support.
- </para>
- </chapter>
-
- <chapter id="resources">
- <title>Resources</title>
- <para>
- USB Home Page:
- <ulink url="http://www.usb.org">http://www.usb.org</ulink>
- </para>
- <para>
- linux-usb Mailing List Archives:
- <ulink url="http://marc.info/?l=linux-usb">http://marc.info/?l=linux-usb</ulink>
- </para>
- <para>
- USB On-the-Go Basics:
- <ulink url="http://www.maximintegrated.com/app-notes/index.mvp/id/1822">http://www.maximintegrated.com/app-notes/index.mvp/id/1822</ulink>
- </para>
- <para>
- Writing USB Device Drivers:
- <ulink url="https://www.kernel.org/doc/htmldocs/writing_usb_driver/index.html">https://www.kernel.org/doc/htmldocs/writing_usb_driver/index.html</ulink>
- </para>
- <para>
- Texas Instruments USB Configuration Wiki Page:
- <ulink url="http://processors.wiki.ti.com/index.php/Usbgeneralpage">http://processors.wiki.ti.com/index.php/Usbgeneralpage</ulink>
- </para>
- <para>
- Analog Devices Blackfin MUSB Configuration:
- <ulink url="http://docs.blackfin.uclinux.org/doku.php?id=linux-kernel:drivers:musb">http://docs.blackfin.uclinux.org/doku.php?id=linux-kernel:drivers:musb</ulink>
- </para>
- </chapter>
-
-</book>
diff --git a/Documentation/DocBook/writing_usb_driver.tmpl b/Documentation/DocBook/writing_usb_driver.tmpl
deleted file mode 100644
index 3210dcf..0000000
--- a/Documentation/DocBook/writing_usb_driver.tmpl
+++ /dev/null
@@ -1,412 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
- "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
-
-<book id="USBDeviceDriver">
- <bookinfo>
- <title>Writing USB Device Drivers</title>
-
- <authorgroup>
- <author>
- <firstname>Greg</firstname>
- <surname>Kroah-Hartman</surname>
- <affiliation>
- <address>
- <email>greg@kroah.com</email>
- </address>
- </affiliation>
- </author>
- </authorgroup>
-
- <copyright>
- <year>2001-2002</year>
- <holder>Greg Kroah-Hartman</holder>
- </copyright>
-
- <legalnotice>
- <para>
- This documentation is free software; you can redistribute
- it and/or modify it under the terms of the GNU General Public
- License as published by the Free Software Foundation; either
- version 2 of the License, or (at your option) any later
- version.
- </para>
-
- <para>
- This program is distributed in the hope that it will be
- useful, but WITHOUT ANY WARRANTY; without even the implied
- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- See the GNU General Public License for more details.
- </para>
-
- <para>
- You should have received a copy of the GNU General Public
- License along with this program; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
- MA 02111-1307 USA
- </para>
-
- <para>
- For more details see the file COPYING in the source
- distribution of Linux.
- </para>
-
- <para>
- This documentation is based on an article published in
- Linux Journal Magazine, October 2001, Issue 90.
- </para>
- </legalnotice>
- </bookinfo>
-
-<toc></toc>
-
- <chapter id="intro">
- <title>Introduction</title>
- <para>
- The Linux USB subsystem has grown from supporting only two different
- types of devices in the 2.2.7 kernel (mice and keyboards), to over 20
- different types of devices in the 2.4 kernel. Linux currently supports
- almost all USB class devices (standard types of devices like keyboards,
- mice, modems, printers and speakers) and an ever-growing number of
- vendor-specific devices (such as USB to serial converters, digital
- cameras, Ethernet devices and MP3 players). For a full list of the
- different USB devices currently supported, see Resources.
- </para>
- <para>
- The remaining kinds of USB devices that do not have support on Linux are
- almost all vendor-specific devices. Each vendor decides to implement a
- custom protocol to talk to their device, so a custom driver usually needs
- to be created. Some vendors are open with their USB protocols and help
- with the creation of Linux drivers, while others do not publish them, and
- developers are forced to reverse-engineer. See Resources for some links
- to handy reverse-engineering tools.
- </para>
- <para>
- Because each different protocol causes a new driver to be created, I have
- written a generic USB driver skeleton, modelled after the pci-skeleton.c
- file in the kernel source tree upon which many PCI network drivers have
- been based. This USB skeleton can be found at drivers/usb/usb-skeleton.c
- in the kernel source tree. In this article I will walk through the basics
- of the skeleton driver, explaining the different pieces and what needs to
- be done to customize it to your specific device.
- </para>
- </chapter>
-
- <chapter id="basics">
- <title>Linux USB Basics</title>
- <para>
- If you are going to write a Linux USB driver, please become familiar with
- the USB protocol specification. It can be found, along with many other
- useful documents, at the USB home page (see Resources). An excellent
- introduction to the Linux USB subsystem can be found at the USB Working
- Devices List (see Resources). It explains how the Linux USB subsystem is
- structured and introduces the reader to the concept of USB urbs
- (USB Request Blocks), which are essential to USB drivers.
- </para>
- <para>
- The first thing a Linux USB driver needs to do is register itself with
- the Linux USB subsystem, giving it some information about which devices
- the driver supports and which functions to call when a device supported
- by the driver is inserted or removed from the system. All of this
- information is passed to the USB subsystem in the usb_driver structure.
- The skeleton driver declares a usb_driver as:
- </para>
- <programlisting>
-static struct usb_driver skel_driver = {
- .name = "skeleton",
- .probe = skel_probe,
- .disconnect = skel_disconnect,
- .fops = &skel_fops,
- .minor = USB_SKEL_MINOR_BASE,
- .id_table = skel_table,
-};
- </programlisting>
- <para>
- The variable name is a string that describes the driver. It is used in
- informational messages printed to the system log. The probe and
- disconnect function pointers are called when a device that matches the
- information provided in the id_table variable is either seen or removed.
- </para>
- <para>
- The fops and minor variables are optional. Most USB drivers hook into
- another kernel subsystem, such as the SCSI, network or TTY subsystem.
- These types of drivers register themselves with the other kernel
- subsystem, and any user-space interactions are provided through that
- interface. But for drivers that do not have a matching kernel subsystem,
- such as MP3 players or scanners, a method of interacting with user space
- is needed. The USB subsystem provides a way to register a minor device
- number and a set of file_operations function pointers that enable this
- user-space interaction. The skeleton driver needs this kind of interface,
- so it provides a minor starting number and a pointer to its
- file_operations functions.
- </para>
- <para>
- The USB driver is then registered with a call to usb_register, usually in
- the driver's init function, as shown here:
- </para>
- <programlisting>
-static int __init usb_skel_init(void)
-{
- int result;
-
- /* register this driver with the USB subsystem */
- result = usb_register(&skel_driver);
- if (result < 0) {
- err("usb_register failed for the "__FILE__ "driver."
- "Error number %d", result);
- return -1;
- }
-
- return 0;
-}
-module_init(usb_skel_init);
- </programlisting>
- <para>
- When the driver is unloaded from the system, it needs to deregister
- itself with the USB subsystem. This is done with the usb_deregister
- function:
- </para>
- <programlisting>
-static void __exit usb_skel_exit(void)
-{
- /* deregister this driver with the USB subsystem */
- usb_deregister(&skel_driver);
-}
-module_exit(usb_skel_exit);
- </programlisting>
- <para>
- To enable the linux-hotplug system to load the driver automatically when
- the device is plugged in, you need to create a MODULE_DEVICE_TABLE. The
- following code tells the hotplug scripts that this module supports a
- single device with a specific vendor and product ID:
- </para>
- <programlisting>
-/* table of devices that work with this driver */
-static struct usb_device_id skel_table [] = {
- { USB_DEVICE(USB_SKEL_VENDOR_ID, USB_SKEL_PRODUCT_ID) },
- { } /* Terminating entry */
-};
-MODULE_DEVICE_TABLE (usb, skel_table);
- </programlisting>
- <para>
- There are other macros that can be used in describing a usb_device_id for
- drivers that support a whole class of USB drivers. See usb.h for more
- information on this.
- </para>
- </chapter>
-
- <chapter id="device">
- <title>Device operation</title>
- <para>
- When a device is plugged into the USB bus that matches the device ID
- pattern that your driver registered with the USB core, the probe function
- is called. The usb_device structure, interface number and the interface ID
- are passed to the function:
- </para>
- <programlisting>
-static int skel_probe(struct usb_interface *interface,
- const struct usb_device_id *id)
- </programlisting>
- <para>
- The driver now needs to verify that this device is actually one that it
- can accept. If so, it returns 0.
- If not, or if any error occurs during initialization, an errorcode
- (such as <literal>-ENOMEM</literal> or <literal>-ENODEV</literal>)
- is returned from the probe function.
- </para>
- <para>
- In the skeleton driver, we determine what end points are marked as bulk-in
- and bulk-out. We create buffers to hold the data that will be sent and
- received from the device, and a USB urb to write data to the device is
- initialized.
- </para>
- <para>
- Conversely, when the device is removed from the USB bus, the disconnect
- function is called with the device pointer. The driver needs to clean any
- private data that has been allocated at this time and to shut down any
- pending urbs that are in the USB system.
- </para>
- <para>
- Now that the device is plugged into the system and the driver is bound to
- the device, any of the functions in the file_operations structure that
- were passed to the USB subsystem will be called from a user program trying
- to talk to the device. The first function called will be open, as the
- program tries to open the device for I/O. We increment our private usage
- count and save a pointer to our internal structure in the file
- structure. This is done so that future calls to file operations will
- enable the driver to determine which device the user is addressing. All
- of this is done with the following code:
- </para>
- <programlisting>
-/* increment our usage count for the module */
-++skel->open_count;
-
-/* save our object in the file's private structure */
-file->private_data = dev;
- </programlisting>
- <para>
- After the open function is called, the read and write functions are called
- to receive and send data to the device. In the skel_write function, we
- receive a pointer to some data that the user wants to send to the device
- and the size of the data. The function determines how much data it can
- send to the device based on the size of the write urb it has created (this
- size depends on the size of the bulk out end point that the device has).
- Then it copies the data from user space to kernel space, points the urb to
- the data and submits the urb to the USB subsystem. This can be seen in
- the following code:
- </para>
- <programlisting>
-/* we can only write as much as 1 urb will hold */
-bytes_written = (count > skel->bulk_out_size) ? skel->bulk_out_size : count;
-
-/* copy the data from user space into our urb */
-copy_from_user(skel->write_urb->transfer_buffer, buffer, bytes_written);
-
-/* set up our urb */
-usb_fill_bulk_urb(skel->write_urb,
- skel->dev,
- usb_sndbulkpipe(skel->dev, skel->bulk_out_endpointAddr),
- skel->write_urb->transfer_buffer,
- bytes_written,
- skel_write_bulk_callback,
- skel);
-
-/* send the data out the bulk port */
-result = usb_submit_urb(skel->write_urb);
-if (result) {
- err("Failed submitting write urb, error %d", result);
-}
- </programlisting>
- <para>
- When the write urb is filled up with the proper information using the
- usb_fill_bulk_urb function, we point the urb's completion callback to call our
- own skel_write_bulk_callback function. This function is called when the
- urb is finished by the USB subsystem. The callback function is called in
- interrupt context, so caution must be taken not to do very much processing
- at that time. Our implementation of skel_write_bulk_callback merely
- reports if the urb was completed successfully or not and then returns.
- </para>
- <para>
- The read function works a bit differently from the write function in that
- we do not use an urb to transfer data from the device to the driver.
- Instead we call the usb_bulk_msg function, which can be used to send or
- receive data from a device without having to create urbs and handle
- urb completion callback functions. We call the usb_bulk_msg function,
- giving it a buffer into which to place any data received from the device
- and a timeout value. If the timeout period expires without receiving any
- data from the device, the function will fail and return an error message.
- This can be shown with the following code:
- </para>
- <programlisting>
-/* do an immediate bulk read to get data from the device */
-retval = usb_bulk_msg (skel->dev,
- usb_rcvbulkpipe (skel->dev,
- skel->bulk_in_endpointAddr),
- skel->bulk_in_buffer,
- skel->bulk_in_size,
- &count, HZ*10);
-/* if the read was successful, copy the data to user space */
-if (!retval) {
- if (copy_to_user (buffer, skel->bulk_in_buffer, count))
- retval = -EFAULT;
- else
- retval = count;
-}
- </programlisting>
- <para>
- The usb_bulk_msg function can be very useful for doing single reads or
- writes to a device; however, if you need to read or write constantly to a
- device, it is recommended to set up your own urbs and submit them to the
- USB subsystem.
- </para>
- <para>
- When the user program releases the file handle that it has been using to
- talk to the device, the release function in the driver is called. In this
- function we decrement our private usage count and wait for possible
- pending writes:
- </para>
- <programlisting>
-/* decrement our usage count for the device */
---skel->open_count;
- </programlisting>
- <para>
- One of the more difficult problems that USB drivers must be able to handle
- smoothly is the fact that the USB device may be removed from the system at
- any point in time, even if a program is currently talking to it. It needs
- to be able to shut down any current reads and writes and notify the
- user-space programs that the device is no longer there. The following
- code (function <function>skel_delete</function>)
- is an example of how to do this: </para>
- <programlisting>
-static inline void skel_delete (struct usb_skel *dev)
-{
- kfree (dev->bulk_in_buffer);
- if (dev->bulk_out_buffer != NULL)
- usb_free_coherent (dev->udev, dev->bulk_out_size,
- dev->bulk_out_buffer,
- dev->write_urb->transfer_dma);
- usb_free_urb (dev->write_urb);
- kfree (dev);
-}
- </programlisting>
- <para>
- If a program currently has an open handle to the device, we reset the flag
- <literal>device_present</literal>. For
- every read, write, release and other functions that expect a device to be
- present, the driver first checks this flag to see if the device is
- still present. If not, it releases that the device has disappeared, and a
- -ENODEV error is returned to the user-space program. When the release
- function is eventually called, it determines if there is no device
- and if not, it does the cleanup that the skel_disconnect
- function normally does if there are no open files on the device (see
- Listing 5).
- </para>
- </chapter>
-
- <chapter id="iso">
- <title>Isochronous Data</title>
- <para>
- This usb-skeleton driver does not have any examples of interrupt or
- isochronous data being sent to or from the device. Interrupt data is sent
- almost exactly as bulk data is, with a few minor exceptions. Isochronous
- data works differently with continuous streams of data being sent to or
- from the device. The audio and video camera drivers are very good examples
- of drivers that handle isochronous data and will be useful if you also
- need to do this.
- </para>
- </chapter>
-
- <chapter id="Conclusion">
- <title>Conclusion</title>
- <para>
- Writing Linux USB device drivers is not a difficult task as the
- usb-skeleton driver shows. This driver, combined with the other current
- USB drivers, should provide enough examples to help a beginning author
- create a working driver in a minimal amount of time. The linux-usb-devel
- mailing list archives also contain a lot of helpful information.
- </para>
- </chapter>
-
- <chapter id="resources">
- <title>Resources</title>
- <para>
- The Linux USB Project: <ulink url="http://www.linux-usb.org">http://www.linux-usb.org/</ulink>
- </para>
- <para>
- Linux Hotplug Project: <ulink url="http://linux-hotplug.sourceforge.net">http://linux-hotplug.sourceforge.net/</ulink>
- </para>
- <para>
- Linux USB Working Devices List: <ulink url="http://www.qbik.ch/usb/devices">http://www.qbik.ch/usb/devices/</ulink>
- </para>
- <para>
- linux-usb-devel Mailing List Archives: <ulink url="http://marc.theaimsgroup.com/?l=linux-usb-devel">http://marc.theaimsgroup.com/?l=linux-usb-devel</ulink>
- </para>
- <para>
- Programming Guide for Linux USB Device Drivers: <ulink url="http://usb.cs.tum.edu/usbdoc">http://usb.cs.tum.edu/usbdoc</ulink>
- </para>
- <para>
- USB Home Page: <ulink url="http://www.usb.org">http://www.usb.org</ulink>
- </para>
- </chapter>
-
-</book>
diff --git a/Documentation/PCI/pci-error-recovery.txt b/Documentation/PCI/pci-error-recovery.txt
index da3b217..0b6bb3e 100644
--- a/Documentation/PCI/pci-error-recovery.txt
+++ b/Documentation/PCI/pci-error-recovery.txt
@@ -11,7 +11,7 @@
Many PCI bus controllers are able to detect a variety of hardware
PCI errors on the bus, such as parity errors on the data and address
-busses, as well as SERR and PERR errors. Some of the more advanced
+buses, as well as SERR and PERR errors. Some of the more advanced
chipsets are able to deal with these errors; these include PCI-E chipsets,
and the PCI-host bridges found on IBM Power4, Power5 and Power6-based
pSeries boxes. A typical action taken is to disconnect the affected device,
@@ -173,7 +173,7 @@
>>> a value of 0xff on read, and writes will be dropped. If more than
>>> EEH_MAX_FAILS I/O's are attempted to a frozen adapter, EEH
>>> assumes that the device driver has gone into an infinite loop
->>> and prints an error to syslog. A reboot is then required to
+>>> and prints an error to syslog. A reboot is then required to
>>> get the device working again.
STEP 2: MMIO Enabled
@@ -231,14 +231,14 @@
STEP 3: Link Reset
------------------
The platform resets the link. This is a PCI-Express specific step
-and is done whenever a non-fatal error has been detected that can be
+and is done whenever a fatal error has been detected that can be
"solved" by resetting the link.
STEP 4: Slot Reset
------------------
In response to a return value of PCI_ERS_RESULT_NEED_RESET, the
-the platform will perform a slot reset on the requesting PCI device(s).
+the platform will perform a slot reset on the requesting PCI device(s).
The actual steps taken by a platform to perform a slot reset
will be platform-dependent. Upon completion of slot reset, the
platform will call the device slot_reset() callback.
@@ -258,7 +258,7 @@
For most PCI devices, a soft reset will be sufficient for recovery.
Optional fundamental reset is provided to support a limited number
-of PCI Express PCI devices for which a soft reset is not sufficient
+of PCI Express devices for which a soft reset is not sufficient
for recovery.
If the platform supports PCI hotplug, then the reset might be
@@ -303,7 +303,7 @@
Same as above.
Drivers for PCI Express cards that require a fundamental reset must
-set the needs_freset bit in the pci_dev structure in their probe function.
+set the needs_freset bit in the pci_dev structure in their probe function.
For example, the QLogic qla2xxx driver sets the needs_freset bit for certain
PCI card types:
diff --git a/Documentation/acpi/aml-debugger.txt b/Documentation/acpi/aml-debugger.txt
index 5f62aa4..e851cc5 100644
--- a/Documentation/acpi/aml-debugger.txt
+++ b/Documentation/acpi/aml-debugger.txt
@@ -15,7 +15,7 @@
CONFIG_ACPI_DEBUGGER=y
CONFIG_ACPI_DEBUGGER_USER=m
- The userspace utlities can be built from the kernel source tree using
+ The userspace utilities can be built from the kernel source tree using
the following commands:
$ cd tools
diff --git a/Documentation/acpi/dsd/graph.txt b/Documentation/acpi/dsd/graph.txt
new file mode 100644
index 0000000..ac09e31
--- /dev/null
+++ b/Documentation/acpi/dsd/graph.txt
@@ -0,0 +1,162 @@
+Graphs
+
+
+_DSD
+----
+
+_DSD (Device Specific Data) [7] is a predefined ACPI device
+configuration object that can be used to convey information on
+hardware features which are not specifically covered by the ACPI
+specification [1][6]. There are two _DSD extensions that are relevant
+for graphs: property [4] and hierarchical data extensions [5]. The
+property extension provides generic key-value pairs whereas the
+hierarchical data extension supports nodes with references to other
+nodes, forming a tree. The nodes in the tree may contain properties as
+defined by the property extension. The two extensions together provide
+a tree-like structure with zero or more properties (key-value pairs)
+in each node of the tree.
+
+The data structure may be accessed at runtime by using the device_*
+and fwnode_* functions defined in include/linux/fwnode.h .
+
+Fwnode represents a generic firmware node object. It is independent on
+the firmware type. In ACPI, fwnodes are _DSD hierarchical data
+extensions objects. A device's _DSD object is represented by an
+fwnode.
+
+The data structure may be referenced to elsewhere in the ACPI tables
+by using a hard reference to the device itself and an index to the
+hierarchical data extension array on each depth.
+
+
+Ports and endpoints
+-------------------
+
+The port and endpoint concepts are very similar to those in Devicetree
+[3]. A port represents an interface in a device, and an endpoint
+represents a connection to that interface.
+
+All port nodes are located under the device's "_DSD" node in the
+hierarchical data extension tree. The property extension related to
+each port node must contain the key "port" and an integer value which
+is the number of the port. The object it refers to should be called "PRTX",
+where "X" is the number of the port.
+
+Further on, endpoints are located under the individual port nodes. The
+first hierarchical data extension package list entry of the endpoint
+nodes must begin with "endpoint" and must be followed by the number
+of the endpoint. The object it refers to should be called "EPXY", where
+"X" is the number of the port and "Y" is the number of the endpoint.
+
+Each port node contains a property extension key "port", the value of
+which is the number of the port node. The each endpoint is similarly numbered
+with a property extension key "endpoint". Port numbers must be unique within a
+device and endpoint numbers must be unique within a port.
+
+The endpoint reference uses property extension with "remote-endpoint" property
+name followed by a reference in the same package. Such references consist of the
+the remote device reference, number of the port in the device and finally the
+number of the endpoint in that port. Individual references thus appear as:
+
+ Package() { device, port_number, endpoint_number }
+
+The references to endpoints must be always done both ways, to the
+remote endpoint and back from the referred remote endpoint node.
+
+A simple example of this is show below:
+
+ Scope (\_SB.PCI0.I2C2)
+ {
+ Device (CAM0)
+ {
+ Name (_DSD, Package () {
+ ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
+ Package () {
+ Package () { "compatible", Package () { "nokia,smia" } },
+ },
+ ToUUID("dbb8e3e6-5886-4ba6-8795-1319f52a966b"),
+ Package () {
+ Package () { "port0", "PRT0" },
+ }
+ })
+ Name (PRT0, Package() {
+ ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
+ Package () {
+ Package () { "port", 0 },
+ },
+ ToUUID("dbb8e3e6-5886-4ba6-8795-1319f52a966b"),
+ Package () {
+ Package () { "endpoint0", "EP00" },
+ }
+ })
+ Name (EP00, Package() {
+ ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
+ Package () {
+ Package () { "endpoint", 0 },
+ Package () { "remote-endpoint", Package() { \_SB.PCI0.ISP, 4, 0 } },
+ }
+ })
+ }
+ }
+
+ Scope (\_SB.PCI0)
+ {
+ Device (ISP)
+ {
+ Name (_DSD, Package () {
+ ToUUID("dbb8e3e6-5886-4ba6-8795-1319f52a966b"),
+ Package () {
+ Package () { "port4", "PRT4" },
+ }
+ })
+
+ Name (PRT4, Package() {
+ ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
+ Package () {
+ Package () { "port", 4 }, /* CSI-2 port number */
+ },
+ ToUUID("dbb8e3e6-5886-4ba6-8795-1319f52a966b"),
+ Package () {
+ Package () { "endpoint0", "EP40" },
+ }
+ })
+
+ Name (EP40, Package() {
+ ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
+ Package () {
+ Package () { "endpoint", 0 },
+ Package () { "remote-endpoint", Package () { \_SB.PCI0.I2C2.CAM0, 0, 0 } },
+ }
+ })
+ }
+ }
+
+Here, the port 0 of the "CAM0" device is connected to the port 4 of
+the "ISP" device and vice versa.
+
+
+References
+----------
+
+[1] _DSD (Device Specific Data) Implementation Guide.
+ <URL:http://www.uefi.org/sites/default/files/resources/_DSD-implementation-guide-toplevel-1_1.htm>,
+ referenced 2016-10-03.
+
+[2] Devicetree. <URL:http://www.devicetree.org>, referenced 2016-10-03.
+
+[3] Documentation/devicetree/bindings/graph.txt
+
+[4] Device Properties UUID For _DSD.
+ <URL:http://www.uefi.org/sites/default/files/resources/_DSD-device-properties-UUID.pdf>,
+ referenced 2016-10-04.
+
+[5] Hierarchical Data Extension UUID For _DSD.
+ <URL:http://www.uefi.org/sites/default/files/resources/_DSD-hierarchical-data-extension-UUID-v1.pdf>,
+ referenced 2016-10-04.
+
+[6] Advanced Configuration and Power Interface Specification.
+ <URL:http://www.uefi.org/sites/default/files/resources/ACPI_6_1.pdf>,
+ referenced 2016-10-04.
+
+[7] _DSD Device Properties Usage Rules.
+ Documentation/acpi/DSD-properties-rules.txt
diff --git a/Documentation/acpi/enumeration.txt b/Documentation/acpi/enumeration.txt
index 209a5eb..7bcf9c3 100644
--- a/Documentation/acpi/enumeration.txt
+++ b/Documentation/acpi/enumeration.txt
@@ -367,10 +367,10 @@
Device Tree namespace link device ID
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The Device Tree protocol uses device indentification based on the "compatible"
+The Device Tree protocol uses device identification based on the "compatible"
property whose value is a string or an array of strings recognized as device
identifiers by drivers and the driver core. The set of all those strings may be
-regarded as a device indentification namespace analogous to the ACPI/PNP device
+regarded as a device identification namespace analogous to the ACPI/PNP device
ID namespace. Consequently, in principle it should not be necessary to allocate
a new (and arguably redundant) ACPI/PNP device ID for a devices with an existing
identification string in the Device Tree (DT) namespace, especially if that ID
@@ -381,7 +381,7 @@
list the IDs of devices the given one is compatible with, but those IDs must
belong to one of the namespaces prescribed by the ACPI specification (see
Section 6.1.2 of ACPI 6.0 for details) and the DT namespace is not one of them.
-Moreover, the specification mandates that either a _HID or an _ADR identificaion
+Moreover, the specification mandates that either a _HID or an _ADR identification
object be present for all ACPI objects representing devices (Section 6.1 of ACPI
6.0). For non-enumerable bus types that object must be _HID and its value must
be a device ID from one of the namespaces prescribed by the specification too.
diff --git a/Documentation/acpi/linuxized-acpica.txt b/Documentation/acpi/linuxized-acpica.txt
index defe2ee..3ad7b0d 100644
--- a/Documentation/acpi/linuxized-acpica.txt
+++ b/Documentation/acpi/linuxized-acpica.txt
@@ -24,7 +24,7 @@
The homepage of ACPICA project is: www.acpica.org, it is maintained and
supported by Intel Corporation.
- The following figure depicts the Linux ACPI subystem where the ACPICA
+ The following figure depicts the Linux ACPI subsystem where the ACPICA
adaptation is included:
+---------------------------------------------------------+
@@ -110,7 +110,7 @@
Linux patches. The patches generated by this process are referred to as
"linuxized ACPICA patches". The release process is carried out on a local
copy the ACPICA git repository. Each commit in the monthly release is
- converted into a linuxized ACPICA patch. Together, they form the montly
+ converted into a linuxized ACPICA patch. Together, they form the monthly
ACPICA release patchset for the Linux ACPI community. This process is
illustrated in the following figure:
@@ -165,7 +165,7 @@
<http://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git>.
Before the linuxized ACPICA patches are sent to the Linux ACPI community
- for review, there is a quality ensurance build test process to reduce
+ for review, there is a quality assurance build test process to reduce
porting issues. Currently this build process only takes care of the
following kernel configuration options:
CONFIG_ACPI/CONFIG_ACPI_DEBUG/CONFIG_ACPI_DEBUGGER
@@ -195,12 +195,12 @@
release utilities (please refer to Section 4 below for the details).
3. Linux specific features - Sometimes it's impossible to use the
current ACPICA APIs to implement features required by the Linux kernel,
- so Linux developers occasionaly have to change ACPICA code directly.
+ so Linux developers occasionally have to change ACPICA code directly.
Those changes may not be acceptable by ACPICA upstream and in such cases
they are left as committed ACPICA divergences unless the ACPICA side can
implement new mechanisms as replacements for them.
4. ACPICA release fixups - ACPICA only tests commits using a set of the
- user space simulation utilies, thus the linuxized ACPICA patches may
+ user space simulation utilities, thus the linuxized ACPICA patches may
break the Linux kernel, leaving us build/boot failures. In order to
avoid breaking Linux bisection, fixes are applied directly to the
linuxized ACPICA patches during the release process. When the release
diff --git a/Documentation/admin-guide/README.rst b/Documentation/admin-guide/README.rst
index 697a00c..02f639a 100644
--- a/Documentation/admin-guide/README.rst
+++ b/Documentation/admin-guide/README.rst
@@ -27,7 +27,7 @@
today Linux also runs on (at least) the Compaq Alpha AXP, Sun SPARC and
UltraSPARC, Motorola 68000, PowerPC, PowerPC64, ARM, Hitachi SuperH, Cell,
IBM S/390, MIPS, HP PA-RISC, Intel IA-64, DEC VAX, AMD x86-64, AXIS CRIS,
- Xtensa, Tilera TILE, AVR32, ARC and Renesas M32R architectures.
+ Xtensa, Tilera TILE, ARC and Renesas M32R architectures.
Linux is easily portable to most general-purpose 32- or 64-bit architectures
as long as they have a paged memory management unit (PMMU) and a port of the
diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst
index 8ddae4e..8c60a8a 100644
--- a/Documentation/admin-guide/index.rst
+++ b/Documentation/admin-guide/index.rst
@@ -60,6 +60,7 @@
mono
java
ras
+ pm/index
.. only:: subproject and html
diff --git a/Documentation/admin-guide/kernel-parameters.rst b/Documentation/admin-guide/kernel-parameters.rst
index b516164..d76ab39 100644
--- a/Documentation/admin-guide/kernel-parameters.rst
+++ b/Documentation/admin-guide/kernel-parameters.rst
@@ -1,3 +1,5 @@
+.. _kernelparameters:
+
The kernel's command-line parameters
====================================
@@ -86,7 +88,6 @@
APIC APIC support is enabled.
APM Advanced Power Management support is enabled.
ARM ARM architecture is enabled.
- AVR32 AVR32 architecture is enabled.
AX25 Appropriate AX.25 support is enabled.
BLACKFIN Blackfin architecture is enabled.
CLK Common clock infrastructure is enabled.
@@ -197,7 +198,7 @@
Finally, the [KMG] suffix is commonly described after a number of kernel
parameter values. These 'K', 'M', and 'G' letters represent the _binary_
-multipliers 'Kilo', 'Mega', and 'Giga', equalling 2^10, 2^20, and 2^30
+multipliers 'Kilo', 'Mega', and 'Giga', equaling 2^10, 2^20, and 2^30
bytes respectively. Such letter suffixes can also be entirely omitted:
.. include:: kernel-parameters.txt
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index facc20a..e4c9e0e 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -531,7 +531,6 @@
[ACPI] acpi_pm
[ARM] imx_timer1,OSTS,netx_timer,mpu_timer2,
pxa_timer,timer3,32k_counter,timer0_1
- [AVR32] avr32
[X86-32] pit,hpet,tsc;
scx200_hrt on Geode; cyclone on IBM x440
[MIPS] MIPS
@@ -989,6 +988,7 @@
earlyprintk=ttySn[,baudrate]
earlyprintk=dbgp[debugController#]
earlyprintk=pciserial,bus:device.function[,baudrate]
+ earlyprintk=xdbc[xhciController#]
earlyprintk is useful when the kernel crashes before
the normal console is initialized. It is not enabled by
@@ -2425,7 +2425,7 @@
osd-targets. Please see:
Documentation/filesystems/pnfs.txt for more explanations
- nmi_debug= [KNL,AVR32,SH] Specify one or more actions to take
+ nmi_debug= [KNL,SH] Specify one or more actions to take
when a NMI is triggered.
Format: [state][,regs][,debounce][,die]
@@ -3178,6 +3178,12 @@
ramdisk_size= [RAM] Sizes of RAM disks in kilobytes
See Documentation/blockdev/ramdisk.txt.
+ ras=option[,option,...] [KNL] RAS-specific options
+
+ cec_disable [X86]
+ Disable the Correctable Errors Collector,
+ see CONFIG_RAS_CEC help text.
+
rcu_nocbs= [KNL]
The argument is a cpu list, as described above.
@@ -4121,6 +4127,9 @@
usbhid.mousepoll=
[USBHID] The interval which mice are to be polled at.
+ usbhid.jspoll=
+ [USBHID] The interval which joysticks are to be polled at.
+
usb-storage.delay_use=
[UMS] The delay in seconds before a new device is
scanned for Logical Units (default 1).
diff --git a/Documentation/admin-guide/pm/cpufreq.rst b/Documentation/admin-guide/pm/cpufreq.rst
new file mode 100644
index 0000000..289c80f
--- /dev/null
+++ b/Documentation/admin-guide/pm/cpufreq.rst
@@ -0,0 +1,700 @@
+.. |struct cpufreq_policy| replace:: :c:type:`struct cpufreq_policy <cpufreq_policy>`
+
+=======================
+CPU Performance Scaling
+=======================
+
+::
+
+ Copyright (c) 2017 Intel Corp., Rafael J. Wysocki <rafael.j.wysocki@intel.com>
+
+The Concept of CPU Performance Scaling
+======================================
+
+The majority of modern processors are capable of operating in a number of
+different clock frequency and voltage configurations, often referred to as
+Operating Performance Points or P-states (in ACPI terminology). As a rule,
+the higher the clock frequency and the higher the voltage, the more instructions
+can be retired by the CPU over a unit of time, but also the higher the clock
+frequency and the higher the voltage, the more energy is consumed over a unit of
+time (or the more power is drawn) by the CPU in the given P-state. Therefore
+there is a natural tradeoff between the CPU capacity (the number of instructions
+that can be executed over a unit of time) and the power drawn by the CPU.
+
+In some situations it is desirable or even necessary to run the program as fast
+as possible and then there is no reason to use any P-states different from the
+highest one (i.e. the highest-performance frequency/voltage configuration
+available). In some other cases, however, it may not be necessary to execute
+instructions so quickly and maintaining the highest available CPU capacity for a
+relatively long time without utilizing it entirely may be regarded as wasteful.
+It also may not be physically possible to maintain maximum CPU capacity for too
+long for thermal or power supply capacity reasons or similar. To cover those
+cases, there are hardware interfaces allowing CPUs to be switched between
+different frequency/voltage configurations or (in the ACPI terminology) to be
+put into different P-states.
+
+Typically, they are used along with algorithms to estimate the required CPU
+capacity, so as to decide which P-states to put the CPUs into. Of course, since
+the utilization of the system generally changes over time, that has to be done
+repeatedly on a regular basis. The activity by which this happens is referred
+to as CPU performance scaling or CPU frequency scaling (because it involves
+adjusting the CPU clock frequency).
+
+
+CPU Performance Scaling in Linux
+================================
+
+The Linux kernel supports CPU performance scaling by means of the ``CPUFreq``
+(CPU Frequency scaling) subsystem that consists of three layers of code: the
+core, scaling governors and scaling drivers.
+
+The ``CPUFreq`` core provides the common code infrastructure and user space
+interfaces for all platforms that support CPU performance scaling. It defines
+the basic framework in which the other components operate.
+
+Scaling governors implement algorithms to estimate the required CPU capacity.
+As a rule, each governor implements one, possibly parametrized, scaling
+algorithm.
+
+Scaling drivers talk to the hardware. They provide scaling governors with
+information on the available P-states (or P-state ranges in some cases) and
+access platform-specific hardware interfaces to change CPU P-states as requested
+by scaling governors.
+
+In principle, all available scaling governors can be used with every scaling
+driver. That design is based on the observation that the information used by
+performance scaling algorithms for P-state selection can be represented in a
+platform-independent form in the majority of cases, so it should be possible
+to use the same performance scaling algorithm implemented in exactly the same
+way regardless of which scaling driver is used. Consequently, the same set of
+scaling governors should be suitable for every supported platform.
+
+However, that observation may not hold for performance scaling algorithms
+based on information provided by the hardware itself, for example through
+feedback registers, as that information is typically specific to the hardware
+interface it comes from and may not be easily represented in an abstract,
+platform-independent way. For this reason, ``CPUFreq`` allows scaling drivers
+to bypass the governor layer and implement their own performance scaling
+algorithms. That is done by the ``intel_pstate`` scaling driver.
+
+
+``CPUFreq`` Policy Objects
+==========================
+
+In some cases the hardware interface for P-state control is shared by multiple
+CPUs. That is, for example, the same register (or set of registers) is used to
+control the P-state of multiple CPUs at the same time and writing to it affects
+all of those CPUs simultaneously.
+
+Sets of CPUs sharing hardware P-state control interfaces are represented by
+``CPUFreq`` as |struct cpufreq_policy| objects. For consistency,
+|struct cpufreq_policy| is also used when there is only one CPU in the given
+set.
+
+The ``CPUFreq`` core maintains a pointer to a |struct cpufreq_policy| object for
+every CPU in the system, including CPUs that are currently offline. If multiple
+CPUs share the same hardware P-state control interface, all of the pointers
+corresponding to them point to the same |struct cpufreq_policy| object.
+
+``CPUFreq`` uses |struct cpufreq_policy| as its basic data type and the design
+of its user space interface is based on the policy concept.
+
+
+CPU Initialization
+==================
+
+First of all, a scaling driver has to be registered for ``CPUFreq`` to work.
+It is only possible to register one scaling driver at a time, so the scaling
+driver is expected to be able to handle all CPUs in the system.
+
+The scaling driver may be registered before or after CPU registration. If
+CPUs are registered earlier, the driver core invokes the ``CPUFreq`` core to
+take a note of all of the already registered CPUs during the registration of the
+scaling driver. In turn, if any CPUs are registered after the registration of
+the scaling driver, the ``CPUFreq`` core will be invoked to take note of them
+at their registration time.
+
+In any case, the ``CPUFreq`` core is invoked to take note of any logical CPU it
+has not seen so far as soon as it is ready to handle that CPU. [Note that the
+logical CPU may be a physical single-core processor, or a single core in a
+multicore processor, or a hardware thread in a physical processor or processor
+core. In what follows "CPU" always means "logical CPU" unless explicitly stated
+otherwise and the word "processor" is used to refer to the physical part
+possibly including multiple logical CPUs.]
+
+Once invoked, the ``CPUFreq`` core checks if the policy pointer is already set
+for the given CPU and if so, it skips the policy object creation. Otherwise,
+a new policy object is created and initialized, which involves the creation of
+a new policy directory in ``sysfs``, and the policy pointer corresponding to
+the given CPU is set to the new policy object's address in memory.
+
+Next, the scaling driver's ``->init()`` callback is invoked with the policy
+pointer of the new CPU passed to it as the argument. That callback is expected
+to initialize the performance scaling hardware interface for the given CPU (or,
+more precisely, for the set of CPUs sharing the hardware interface it belongs
+to, represented by its policy object) and, if the policy object it has been
+called for is new, to set parameters of the policy, like the minimum and maximum
+frequencies supported by the hardware, the table of available frequencies (if
+the set of supported P-states is not a continuous range), and the mask of CPUs
+that belong to the same policy (including both online and offline CPUs). That
+mask is then used by the core to populate the policy pointers for all of the
+CPUs in it.
+
+The next major initialization step for a new policy object is to attach a
+scaling governor to it (to begin with, that is the default scaling governor
+determined by the kernel configuration, but it may be changed later
+via ``sysfs``). First, a pointer to the new policy object is passed to the
+governor's ``->init()`` callback which is expected to initialize all of the
+data structures necessary to handle the given policy and, possibly, to add
+a governor ``sysfs`` interface to it. Next, the governor is started by
+invoking its ``->start()`` callback.
+
+That callback it expected to register per-CPU utilization update callbacks for
+all of the online CPUs belonging to the given policy with the CPU scheduler.
+The utilization update callbacks will be invoked by the CPU scheduler on
+important events, like task enqueue and dequeue, on every iteration of the
+scheduler tick or generally whenever the CPU utilization may change (from the
+scheduler's perspective). They are expected to carry out computations needed
+to determine the P-state to use for the given policy going forward and to
+invoke the scaling driver to make changes to the hardware in accordance with
+the P-state selection. The scaling driver may be invoked directly from
+scheduler context or asynchronously, via a kernel thread or workqueue, depending
+on the configuration and capabilities of the scaling driver and the governor.
+
+Similar steps are taken for policy objects that are not new, but were "inactive"
+previously, meaning that all of the CPUs belonging to them were offline. The
+only practical difference in that case is that the ``CPUFreq`` core will attempt
+to use the scaling governor previously used with the policy that became
+"inactive" (and is re-initialized now) instead of the default governor.
+
+In turn, if a previously offline CPU is being brought back online, but some
+other CPUs sharing the policy object with it are online already, there is no
+need to re-initialize the policy object at all. In that case, it only is
+necessary to restart the scaling governor so that it can take the new online CPU
+into account. That is achieved by invoking the governor's ``->stop`` and
+``->start()`` callbacks, in this order, for the entire policy.
+
+As mentioned before, the ``intel_pstate`` scaling driver bypasses the scaling
+governor layer of ``CPUFreq`` and provides its own P-state selection algorithms.
+Consequently, if ``intel_pstate`` is used, scaling governors are not attached to
+new policy objects. Instead, the driver's ``->setpolicy()`` callback is invoked
+to register per-CPU utilization update callbacks for each policy. These
+callbacks are invoked by the CPU scheduler in the same way as for scaling
+governors, but in the ``intel_pstate`` case they both determine the P-state to
+use and change the hardware configuration accordingly in one go from scheduler
+context.
+
+The policy objects created during CPU initialization and other data structures
+associated with them are torn down when the scaling driver is unregistered
+(which happens when the kernel module containing it is unloaded, for example) or
+when the last CPU belonging to the given policy in unregistered.
+
+
+Policy Interface in ``sysfs``
+=============================
+
+During the initialization of the kernel, the ``CPUFreq`` core creates a
+``sysfs`` directory (kobject) called ``cpufreq`` under
+:file:`/sys/devices/system/cpu/`.
+
+That directory contains a ``policyX`` subdirectory (where ``X`` represents an
+integer number) for every policy object maintained by the ``CPUFreq`` core.
+Each ``policyX`` directory is pointed to by ``cpufreq`` symbolic links
+under :file:`/sys/devices/system/cpu/cpuY/` (where ``Y`` represents an integer
+that may be different from the one represented by ``X``) for all of the CPUs
+associated with (or belonging to) the given policy. The ``policyX`` directories
+in :file:`/sys/devices/system/cpu/cpufreq` each contain policy-specific
+attributes (files) to control ``CPUFreq`` behavior for the corresponding policy
+objects (that is, for all of the CPUs associated with them).
+
+Some of those attributes are generic. They are created by the ``CPUFreq`` core
+and their behavior generally does not depend on what scaling driver is in use
+and what scaling governor is attached to the given policy. Some scaling drivers
+also add driver-specific attributes to the policy directories in ``sysfs`` to
+control policy-specific aspects of driver behavior.
+
+The generic attributes under :file:`/sys/devices/system/cpu/cpufreq/policyX/`
+are the following:
+
+``affected_cpus``
+ List of online CPUs belonging to this policy (i.e. sharing the hardware
+ performance scaling interface represented by the ``policyX`` policy
+ object).
+
+``bios_limit``
+ If the platform firmware (BIOS) tells the OS to apply an upper limit to
+ CPU frequencies, that limit will be reported through this attribute (if
+ present).
+
+ The existence of the limit may be a result of some (often unintentional)
+ BIOS settings, restrictions coming from a service processor or another
+ BIOS/HW-based mechanisms.
+
+ This does not cover ACPI thermal limitations which can be discovered
+ through a generic thermal driver.
+
+ This attribute is not present if the scaling driver in use does not
+ support it.
+
+``cpuinfo_max_freq``
+ Maximum possible operating frequency the CPUs belonging to this policy
+ can run at (in kHz).
+
+``cpuinfo_min_freq``
+ Minimum possible operating frequency the CPUs belonging to this policy
+ can run at (in kHz).
+
+``cpuinfo_transition_latency``
+ The time it takes to switch the CPUs belonging to this policy from one
+ P-state to another, in nanoseconds.
+
+ If unknown or if known to be so high that the scaling driver does not
+ work with the `ondemand`_ governor, -1 (:c:macro:`CPUFREQ_ETERNAL`)
+ will be returned by reads from this attribute.
+
+``related_cpus``
+ List of all (online and offline) CPUs belonging to this policy.
+
+``scaling_available_governors``
+ List of ``CPUFreq`` scaling governors present in the kernel that can
+ be attached to this policy or (if the ``intel_pstate`` scaling driver is
+ in use) list of scaling algorithms provided by the driver that can be
+ applied to this policy.
+
+ [Note that some governors are modular and it may be necessary to load a
+ kernel module for the governor held by it to become available and be
+ listed by this attribute.]
+
+``scaling_cur_freq``
+ Current frequency of all of the CPUs belonging to this policy (in kHz).
+
+ For the majority of scaling drivers, this is the frequency of the last
+ P-state requested by the driver from the hardware using the scaling
+ interface provided by it, which may or may not reflect the frequency
+ the CPU is actually running at (due to hardware design and other
+ limitations).
+
+ Some scaling drivers (e.g. ``intel_pstate``) attempt to provide
+ information more precisely reflecting the current CPU frequency through
+ this attribute, but that still may not be the exact current CPU
+ frequency as seen by the hardware at the moment.
+
+``scaling_driver``
+ The scaling driver currently in use.
+
+``scaling_governor``
+ The scaling governor currently attached to this policy or (if the
+ ``intel_pstate`` scaling driver is in use) the scaling algorithm
+ provided by the driver that is currently applied to this policy.
+
+ This attribute is read-write and writing to it will cause a new scaling
+ governor to be attached to this policy or a new scaling algorithm
+ provided by the scaling driver to be applied to it (in the
+ ``intel_pstate`` case), as indicated by the string written to this
+ attribute (which must be one of the names listed by the
+ ``scaling_available_governors`` attribute described above).
+
+``scaling_max_freq``
+ Maximum frequency the CPUs belonging to this policy are allowed to be
+ running at (in kHz).
+
+ This attribute is read-write and writing a string representing an
+ integer to it will cause a new limit to be set (it must not be lower
+ than the value of the ``scaling_min_freq`` attribute).
+
+``scaling_min_freq``
+ Minimum frequency the CPUs belonging to this policy are allowed to be
+ running at (in kHz).
+
+ This attribute is read-write and writing a string representing a
+ non-negative integer to it will cause a new limit to be set (it must not
+ be higher than the value of the ``scaling_max_freq`` attribute).
+
+``scaling_setspeed``
+ This attribute is functional only if the `userspace`_ scaling governor
+ is attached to the given policy.
+
+ It returns the last frequency requested by the governor (in kHz) or can
+ be written to in order to set a new frequency for the policy.
+
+
+Generic Scaling Governors
+=========================
+
+``CPUFreq`` provides generic scaling governors that can be used with all
+scaling drivers. As stated before, each of them implements a single, possibly
+parametrized, performance scaling algorithm.
+
+Scaling governors are attached to policy objects and different policy objects
+can be handled by different scaling governors at the same time (although that
+may lead to suboptimal results in some cases).
+
+The scaling governor for a given policy object can be changed at any time with
+the help of the ``scaling_governor`` policy attribute in ``sysfs``.
+
+Some governors expose ``sysfs`` attributes to control or fine-tune the scaling
+algorithms implemented by them. Those attributes, referred to as governor
+tunables, can be either global (system-wide) or per-policy, depending on the
+scaling driver in use. If the driver requires governor tunables to be
+per-policy, they are located in a subdirectory of each policy directory.
+Otherwise, they are located in a subdirectory under
+:file:`/sys/devices/system/cpu/cpufreq/`. In either case the name of the
+subdirectory containing the governor tunables is the name of the governor
+providing them.
+
+``performance``
+---------------
+
+When attached to a policy object, this governor causes the highest frequency,
+within the ``scaling_max_freq`` policy limit, to be requested for that policy.
+
+The request is made once at that time the governor for the policy is set to
+``performance`` and whenever the ``scaling_max_freq`` or ``scaling_min_freq``
+policy limits change after that.
+
+``powersave``
+-------------
+
+When attached to a policy object, this governor causes the lowest frequency,
+within the ``scaling_min_freq`` policy limit, to be requested for that policy.
+
+The request is made once at that time the governor for the policy is set to
+``powersave`` and whenever the ``scaling_max_freq`` or ``scaling_min_freq``
+policy limits change after that.
+
+``userspace``
+-------------
+
+This governor does not do anything by itself. Instead, it allows user space
+to set the CPU frequency for the policy it is attached to by writing to the
+``scaling_setspeed`` attribute of that policy.
+
+``schedutil``
+-------------
+
+This governor uses CPU utilization data available from the CPU scheduler. It
+generally is regarded as a part of the CPU scheduler, so it can access the
+scheduler's internal data structures directly.
+
+It runs entirely in scheduler context, although in some cases it may need to
+invoke the scaling driver asynchronously when it decides that the CPU frequency
+should be changed for a given policy (that depends on whether or not the driver
+is capable of changing the CPU frequency from scheduler context).
+
+The actions of this governor for a particular CPU depend on the scheduling class
+invoking its utilization update callback for that CPU. If it is invoked by the
+RT or deadline scheduling classes, the governor will increase the frequency to
+the allowed maximum (that is, the ``scaling_max_freq`` policy limit). In turn,
+if it is invoked by the CFS scheduling class, the governor will use the
+Per-Entity Load Tracking (PELT) metric for the root control group of the
+given CPU as the CPU utilization estimate (see the `Per-entity load tracking`_
+LWN.net article for a description of the PELT mechanism). Then, the new
+CPU frequency to apply is computed in accordance with the formula
+
+ f = 1.25 * ``f_0`` * ``util`` / ``max``
+
+where ``util`` is the PELT number, ``max`` is the theoretical maximum of
+``util``, and ``f_0`` is either the maximum possible CPU frequency for the given
+policy (if the PELT number is frequency-invariant), or the current CPU frequency
+(otherwise).
+
+This governor also employs a mechanism allowing it to temporarily bump up the
+CPU frequency for tasks that have been waiting on I/O most recently, called
+"IO-wait boosting". That happens when the :c:macro:`SCHED_CPUFREQ_IOWAIT` flag
+is passed by the scheduler to the governor callback which causes the frequency
+to go up to the allowed maximum immediately and then draw back to the value
+returned by the above formula over time.
+
+This governor exposes only one tunable:
+
+``rate_limit_us``
+ Minimum time (in microseconds) that has to pass between two consecutive
+ runs of governor computations (default: 1000 times the scaling driver's
+ transition latency).
+
+ The purpose of this tunable is to reduce the scheduler context overhead
+ of the governor which might be excessive without it.
+
+This governor generally is regarded as a replacement for the older `ondemand`_
+and `conservative`_ governors (described below), as it is simpler and more
+tightly integrated with the CPU scheduler, its overhead in terms of CPU context
+switches and similar is less significant, and it uses the scheduler's own CPU
+utilization metric, so in principle its decisions should not contradict the
+decisions made by the other parts of the scheduler.
+
+``ondemand``
+------------
+
+This governor uses CPU load as a CPU frequency selection metric.
+
+In order to estimate the current CPU load, it measures the time elapsed between
+consecutive invocations of its worker routine and computes the fraction of that
+time in which the given CPU was not idle. The ratio of the non-idle (active)
+time to the total CPU time is taken as an estimate of the load.
+
+If this governor is attached to a policy shared by multiple CPUs, the load is
+estimated for all of them and the greatest result is taken as the load estimate
+for the entire policy.
+
+The worker routine of this governor has to run in process context, so it is
+invoked asynchronously (via a workqueue) and CPU P-states are updated from
+there if necessary. As a result, the scheduler context overhead from this
+governor is minimum, but it causes additional CPU context switches to happen
+relatively often and the CPU P-state updates triggered by it can be relatively
+irregular. Also, it affects its own CPU load metric by running code that
+reduces the CPU idle time (even though the CPU idle time is only reduced very
+slightly by it).
+
+It generally selects CPU frequencies proportional to the estimated load, so that
+the value of the ``cpuinfo_max_freq`` policy attribute corresponds to the load of
+1 (or 100%), and the value of the ``cpuinfo_min_freq`` policy attribute
+corresponds to the load of 0, unless when the load exceeds a (configurable)
+speedup threshold, in which case it will go straight for the highest frequency
+it is allowed to use (the ``scaling_max_freq`` policy limit).
+
+This governor exposes the following tunables:
+
+``sampling_rate``
+ This is how often the governor's worker routine should run, in
+ microseconds.
+
+ Typically, it is set to values of the order of 10000 (10 ms). Its
+ default value is equal to the value of ``cpuinfo_transition_latency``
+ for each policy this governor is attached to (but since the unit here
+ is greater by 1000, this means that the time represented by
+ ``sampling_rate`` is 1000 times greater than the transition latency by
+ default).
+
+ If this tunable is per-policy, the following shell command sets the time
+ represented by it to be 750 times as high as the transition latency::
+
+ # echo `$(($(cat cpuinfo_transition_latency) * 750 / 1000)) > ondemand/sampling_rate
+
+
+``min_sampling_rate``
+ The minimum value of ``sampling_rate``.
+
+ Equal to 10000 (10 ms) if :c:macro:`CONFIG_NO_HZ_COMMON` and
+ :c:data:`tick_nohz_active` are both set or to 20 times the value of
+ :c:data:`jiffies` in microseconds otherwise.
+
+``up_threshold``
+ If the estimated CPU load is above this value (in percent), the governor
+ will set the frequency to the maximum value allowed for the policy.
+ Otherwise, the selected frequency will be proportional to the estimated
+ CPU load.
+
+``ignore_nice_load``
+ If set to 1 (default 0), it will cause the CPU load estimation code to
+ treat the CPU time spent on executing tasks with "nice" levels greater
+ than 0 as CPU idle time.
+
+ This may be useful if there are tasks in the system that should not be
+ taken into account when deciding what frequency to run the CPUs at.
+ Then, to make that happen it is sufficient to increase the "nice" level
+ of those tasks above 0 and set this attribute to 1.
+
+``sampling_down_factor``
+ Temporary multiplier, between 1 (default) and 100 inclusive, to apply to
+ the ``sampling_rate`` value if the CPU load goes above ``up_threshold``.
+
+ This causes the next execution of the governor's worker routine (after
+ setting the frequency to the allowed maximum) to be delayed, so the
+ frequency stays at the maximum level for a longer time.
+
+ Frequency fluctuations in some bursty workloads may be avoided this way
+ at the cost of additional energy spent on maintaining the maximum CPU
+ capacity.
+
+``powersave_bias``
+ Reduction factor to apply to the original frequency target of the
+ governor (including the maximum value used when the ``up_threshold``
+ value is exceeded by the estimated CPU load) or sensitivity threshold
+ for the AMD frequency sensitivity powersave bias driver
+ (:file:`drivers/cpufreq/amd_freq_sensitivity.c`), between 0 and 1000
+ inclusive.
+
+ If the AMD frequency sensitivity powersave bias driver is not loaded,
+ the effective frequency to apply is given by
+
+ f * (1 - ``powersave_bias`` / 1000)
+
+ where f is the governor's original frequency target. The default value
+ of this attribute is 0 in that case.
+
+ If the AMD frequency sensitivity powersave bias driver is loaded, the
+ value of this attribute is 400 by default and it is used in a different
+ way.
+
+ On Family 16h (and later) AMD processors there is a mechanism to get a
+ measured workload sensitivity, between 0 and 100% inclusive, from the
+ hardware. That value can be used to estimate how the performance of the
+ workload running on a CPU will change in response to frequency changes.
+
+ The performance of a workload with the sensitivity of 0 (memory-bound or
+ IO-bound) is not expected to increase at all as a result of increasing
+ the CPU frequency, whereas workloads with the sensitivity of 100%
+ (CPU-bound) are expected to perform much better if the CPU frequency is
+ increased.
+
+ If the workload sensitivity is less than the threshold represented by
+ the ``powersave_bias`` value, the sensitivity powersave bias driver
+ will cause the governor to select a frequency lower than its original
+ target, so as to avoid over-provisioning workloads that will not benefit
+ from running at higher CPU frequencies.
+
+``conservative``
+----------------
+
+This governor uses CPU load as a CPU frequency selection metric.
+
+It estimates the CPU load in the same way as the `ondemand`_ governor described
+above, but the CPU frequency selection algorithm implemented by it is different.
+
+Namely, it avoids changing the frequency significantly over short time intervals
+which may not be suitable for systems with limited power supply capacity (e.g.
+battery-powered). To achieve that, it changes the frequency in relatively
+small steps, one step at a time, up or down - depending on whether or not a
+(configurable) threshold has been exceeded by the estimated CPU load.
+
+This governor exposes the following tunables:
+
+``freq_step``
+ Frequency step in percent of the maximum frequency the governor is
+ allowed to set (the ``scaling_max_freq`` policy limit), between 0 and
+ 100 (5 by default).
+
+ This is how much the frequency is allowed to change in one go. Setting
+ it to 0 will cause the default frequency step (5 percent) to be used
+ and setting it to 100 effectively causes the governor to periodically
+ switch the frequency between the ``scaling_min_freq`` and
+ ``scaling_max_freq`` policy limits.
+
+``down_threshold``
+ Threshold value (in percent, 20 by default) used to determine the
+ frequency change direction.
+
+ If the estimated CPU load is greater than this value, the frequency will
+ go up (by ``freq_step``). If the load is less than this value (and the
+ ``sampling_down_factor`` mechanism is not in effect), the frequency will
+ go down. Otherwise, the frequency will not be changed.
+
+``sampling_down_factor``
+ Frequency decrease deferral factor, between 1 (default) and 10
+ inclusive.
+
+ It effectively causes the frequency to go down ``sampling_down_factor``
+ times slower than it ramps up.
+
+
+Frequency Boost Support
+=======================
+
+Background
+----------
+
+Some processors support a mechanism to raise the operating frequency of some
+cores in a multicore package temporarily (and above the sustainable frequency
+threshold for the whole package) under certain conditions, for example if the
+whole chip is not fully utilized and below its intended thermal or power budget.
+
+Different names are used by different vendors to refer to this functionality.
+For Intel processors it is referred to as "Turbo Boost", AMD calls it
+"Turbo-Core" or (in technical documentation) "Core Performance Boost" and so on.
+As a rule, it also is implemented differently by different vendors. The simple
+term "frequency boost" is used here for brevity to refer to all of those
+implementations.
+
+The frequency boost mechanism may be either hardware-based or software-based.
+If it is hardware-based (e.g. on x86), the decision to trigger the boosting is
+made by the hardware (although in general it requires the hardware to be put
+into a special state in which it can control the CPU frequency within certain
+limits). If it is software-based (e.g. on ARM), the scaling driver decides
+whether or not to trigger boosting and when to do that.
+
+The ``boost`` File in ``sysfs``
+-------------------------------
+
+This file is located under :file:`/sys/devices/system/cpu/cpufreq/` and controls
+the "boost" setting for the whole system. It is not present if the underlying
+scaling driver does not support the frequency boost mechanism (or supports it,
+but provides a driver-specific interface for controlling it, like
+``intel_pstate``).
+
+If the value in this file is 1, the frequency boost mechanism is enabled. This
+means that either the hardware can be put into states in which it is able to
+trigger boosting (in the hardware-based case), or the software is allowed to
+trigger boosting (in the software-based case). It does not mean that boosting
+is actually in use at the moment on any CPUs in the system. It only means a
+permission to use the frequency boost mechanism (which still may never be used
+for other reasons).
+
+If the value in this file is 0, the frequency boost mechanism is disabled and
+cannot be used at all.
+
+The only values that can be written to this file are 0 and 1.
+
+Rationale for Boost Control Knob
+--------------------------------
+
+The frequency boost mechanism is generally intended to help to achieve optimum
+CPU performance on time scales below software resolution (e.g. below the
+scheduler tick interval) and it is demonstrably suitable for many workloads, but
+it may lead to problems in certain situations.
+
+For this reason, many systems make it possible to disable the frequency boost
+mechanism in the platform firmware (BIOS) setup, but that requires the system to
+be restarted for the setting to be adjusted as desired, which may not be
+practical at least in some cases. For example:
+
+ 1. Boosting means overclocking the processor, although under controlled
+ conditions. Generally, the processor's energy consumption increases
+ as a result of increasing its frequency and voltage, even temporarily.
+ That may not be desirable on systems that switch to power sources of
+ limited capacity, such as batteries, so the ability to disable the boost
+ mechanism while the system is running may help there (but that depends on
+ the workload too).
+
+ 2. In some situations deterministic behavior is more important than
+ performance or energy consumption (or both) and the ability to disable
+ boosting while the system is running may be useful then.
+
+ 3. To examine the impact of the frequency boost mechanism itself, it is useful
+ to be able to run tests with and without boosting, preferably without
+ restarting the system in the meantime.
+
+ 4. Reproducible results are important when running benchmarks. Since
+ the boosting functionality depends on the load of the whole package,
+ single-thread performance may vary because of it which may lead to
+ unreproducible results sometimes. That can be avoided by disabling the
+ frequency boost mechanism before running benchmarks sensitive to that
+ issue.
+
+Legacy AMD ``cpb`` Knob
+-----------------------
+
+The AMD powernow-k8 scaling driver supports a ``sysfs`` knob very similar to
+the global ``boost`` one. It is used for disabling/enabling the "Core
+Performance Boost" feature of some AMD processors.
+
+If present, that knob is located in every ``CPUFreq`` policy directory in
+``sysfs`` (:file:`/sys/devices/system/cpu/cpufreq/policyX/`) and is called
+``cpb``, which indicates a more fine grained control interface. The actual
+implementation, however, works on the system-wide basis and setting that knob
+for one policy causes the same value of it to be set for all of the other
+policies at the same time.
+
+That knob is still supported on AMD processors that support its underlying
+hardware feature, but it may be configured out of the kernel (via the
+:c:macro:`CONFIG_X86_ACPI_CPUFREQ_CPB` configuration option) and the global
+``boost`` knob is present regardless. Thus it is always possible use the
+``boost`` knob instead of the ``cpb`` one which is highly recommended, as that
+is more consistent with what all of the other systems do (and the ``cpb`` knob
+may not be supported any more in the future).
+
+The ``cpb`` knob is never present for any processors without the underlying
+hardware feature (e.g. all Intel ones), even if the
+:c:macro:`CONFIG_X86_ACPI_CPUFREQ_CPB` configuration option is set.
+
+
+.. _Per-entity load tracking: https://lwn.net/Articles/531853/
diff --git a/Documentation/admin-guide/pm/index.rst b/Documentation/admin-guide/pm/index.rst
new file mode 100644
index 0000000..c80f087
--- /dev/null
+++ b/Documentation/admin-guide/pm/index.rst
@@ -0,0 +1,15 @@
+================
+Power Management
+================
+
+.. toctree::
+ :maxdepth: 2
+
+ cpufreq
+
+.. only:: subproject and html
+
+ Indices
+ =======
+
+ * :ref:`genindex`
diff --git a/Documentation/admin-guide/ras.rst b/Documentation/admin-guide/ras.rst
index 1b90c6f..8c7bbf2 100644
--- a/Documentation/admin-guide/ras.rst
+++ b/Documentation/admin-guide/ras.rst
@@ -8,7 +8,7 @@
************
Reliability, Availability and Serviceability (RAS) is a concept used on
-servers meant to measure their robusteness.
+servers meant to measure their robustness.
Reliability
is the probability that a system will produce correct outputs.
@@ -42,13 +42,13 @@
* CPU – detect errors at instruction execution and at L1/L2/L3 caches;
* Memory – add error correction logic (ECC) to detect and correct errors;
-* I/O – add CRC checksums for tranfered data;
+* I/O – add CRC checksums for transferred data;
* Storage – RAID, journal file systems, checksums,
Self-Monitoring, Analysis and Reporting Technology (SMART).
By monitoring the number of occurrences of error detections, it is possible
to identify if the probability of hardware errors is increasing, and, on such
-case, do a preventive maintainance to replace a degrated component while
+case, do a preventive maintenance to replace a degraded component while
those errors are correctable.
Types of errors
@@ -121,7 +121,7 @@
On the above example, a DDR4 SO-DIMM memory module is located at the
system's memory labeled as "BANK 0", as given by the *bank locator* field.
Please notice that, on such system, the *total width* is equal to the
-*data witdh*. It means that such memory module doesn't have error
+*data width*. It means that such memory module doesn't have error
detection/correction mechanisms.
Unfortunately, not all systems use the same field to specify the memory
@@ -145,7 +145,7 @@
There, the DDR3 RDIMM memory module is located at the system's memory labeled
as "DIMM_A1", as given by the *locator* field. Please notice that this
-memory module has 64 bits of *data witdh* and 72 bits of *total width*. So,
+memory module has 64 bits of *data width* and 72 bits of *total width*. So,
it has 8 extra bits to be used by error detection and correction mechanisms.
Such kind of memory is called Error-correcting code memory (ECC memory).
@@ -186,7 +186,7 @@
.. [#f1] Please notice that several memory controllers allow operation on a
mode called "Lock-Step", where it groups two memory modules together,
doing 128-bit reads/writes. That gives 16 bits for error correction, with
- significatively improves the error correction mechanism, at the expense
+ significantly improves the error correction mechanism, at the expense
that, when an error happens, there's no way to know what memory module is
to blame. So, it has to blame both memory modules.
diff --git a/Documentation/admin-guide/security-bugs.rst b/Documentation/admin-guide/security-bugs.rst
index 4f7414c..47574b3 100644
--- a/Documentation/admin-guide/security-bugs.rst
+++ b/Documentation/admin-guide/security-bugs.rst
@@ -14,14 +14,17 @@
The Linux kernel security team can be contacted by email at
<security@kernel.org>. This is a private list of security officers
who will help verify the bug report and develop and release a fix.
-It is possible that the security team will bring in extra help from
-area maintainers to understand and fix the security vulnerability.
+If you already have a fix, please include it with your report, as
+that can speed up the process considerably. It is possible that the
+security team will bring in extra help from area maintainers to
+understand and fix the security vulnerability.
As it is with any bug, the more information provided the easier it
will be to diagnose and fix. Please review the procedure outlined in
-admin-guide/reporting-bugs.rst if you are unclear about what information is helpful.
-Any exploit code is very helpful and will not be released without
-consent from the reporter unless it has already been made public.
+admin-guide/reporting-bugs.rst if you are unclear about what
+information is helpful. Any exploit code is very helpful and will not
+be released without consent from the reporter unless it has already been
+made public.
Disclosure
----------
@@ -39,6 +42,32 @@
to a few weeks. As a basic default policy, we expect report date to
disclosure date to be on the order of 7 days.
+Coordination
+------------
+
+Fixes for sensitive bugs, such as those that might lead to privilege
+escalations, may need to be coordinated with the private
+<linux-distros@vs.openwall.org> mailing list so that distribution vendors
+are well prepared to issue a fixed kernel upon public disclosure of the
+upstream fix. Distros will need some time to test the proposed patch and
+will generally request at least a few days of embargo, and vendor update
+publication prefers to happen Tuesday through Thursday. When appropriate,
+the security team can assist with this coordination, or the reporter can
+include linux-distros from the start. In this case, remember to prefix
+the email Subject line with "[vs]" as described in the linux-distros wiki:
+<http://oss-security.openwall.org/wiki/mailing-lists/distros#how-to-use-the-lists>
+
+CVE assignment
+--------------
+
+The security team does not normally assign CVEs, nor do we require them
+for reports or fixes, as this can needlessly complicate the process and
+may delay the bug handling. If a reporter wishes to have a CVE identifier
+assigned ahead of public disclosure, they will need to contact the private
+linux-distros list, described above. When such a CVE identifier is known
+before a patch is provided, it is desirable to mention it in the commit
+message, though.
+
Non-disclosure agreements
-------------------------
diff --git a/Documentation/admin-guide/sysrq.rst b/Documentation/admin-guide/sysrq.rst
index d1712ea..7b9035c 100644
--- a/Documentation/admin-guide/sysrq.rst
+++ b/Documentation/admin-guide/sysrq.rst
@@ -212,7 +212,8 @@
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There are some keyboards that produce a different keycode for SysRq than the
-pre-defined value of 99 (see ``KEY_SYSRQ`` in ``include/linux/input.h``), or
+pre-defined value of 99
+(see ``KEY_SYSRQ`` in ``include/uapi/linux/input-event-codes.h``), or
which don't have a SysRq key at all. In these cases, run ``showkey -s`` to find
an appropriate scancode sequence, and use ``setkeycodes <sequence> 99`` to map
this sequence to the usual SysRq code (e.g., ``setkeycodes e05b 99``). It's
diff --git a/Documentation/arm/mem_alignment b/Documentation/arm/mem_alignment
index c7c7a11..6335fca 100644
--- a/Documentation/arm/mem_alignment
+++ b/Documentation/arm/mem_alignment
@@ -48,7 +48,7 @@
For example, the following will turn on the warnings, but without
fixing up or sending SIGBUS signals:
- echo 1 > /proc/sys/debug/alignment
+ echo 1 > /proc/cpu/alignment
You can also read the content of the same file to get statistical
information on unaligned access occurrences plus the current mode of
diff --git a/Documentation/arm64/silicon-errata.txt b/Documentation/arm64/silicon-errata.txt
index 2f66683..10f2ddd 100644
--- a/Documentation/arm64/silicon-errata.txt
+++ b/Documentation/arm64/silicon-errata.txt
@@ -54,6 +54,7 @@
| ARM | Cortex-A57 | #852523 | N/A |
| ARM | Cortex-A57 | #834220 | ARM64_ERRATUM_834220 |
| ARM | Cortex-A72 | #853709 | N/A |
+| ARM | Cortex-A73 | #858921 | ARM64_ERRATUM_858921 |
| ARM | MMU-500 | #841119,#826419 | N/A |
| | | | |
| Cavium | ThunderX ITS | #22375, #24313 | CAVIUM_ERRATUM_22375 |
diff --git a/Documentation/block/00-INDEX b/Documentation/block/00-INDEX
index e55103a..8d55b4b 100644
--- a/Documentation/block/00-INDEX
+++ b/Documentation/block/00-INDEX
@@ -1,5 +1,7 @@
00-INDEX
- This file
+bfq-iosched.txt
+ - BFQ IO scheduler and its tunables
biodoc.txt
- Notes on the Generic Block Layer Rewrite in Linux 2.5
biovecs.txt
diff --git a/Documentation/block/bfq-iosched.txt b/Documentation/block/bfq-iosched.txt
new file mode 100644
index 0000000..1b87df6
--- /dev/null
+++ b/Documentation/block/bfq-iosched.txt
@@ -0,0 +1,531 @@
+BFQ (Budget Fair Queueing)
+==========================
+
+BFQ is a proportional-share I/O scheduler, with some extra
+low-latency capabilities. In addition to cgroups support (blkio or io
+controllers), BFQ's main features are:
+- BFQ guarantees a high system and application responsiveness, and a
+ low latency for time-sensitive applications, such as audio or video
+ players;
+- BFQ distributes bandwidth, and not just time, among processes or
+ groups (switching back to time distribution when needed to keep
+ throughput high).
+
+On average CPUs, the current version of BFQ can handle devices
+performing at most ~30K IOPS; at most ~50 KIOPS on faster CPUs. As a
+reference, 30-50 KIOPS correspond to very high bandwidths with
+sequential I/O (e.g., 8-12 GB/s if I/O requests are 256 KB large), and
+to 120-200 MB/s with 4KB random I/O. BFQ has not yet been tested on
+multi-queue devices.
+
+The table of contents follow. Impatients can just jump to Section 3.
+
+CONTENTS
+
+1. When may BFQ be useful?
+ 1-1 Personal systems
+ 1-2 Server systems
+2. How does BFQ work?
+3. What are BFQ's tunable?
+4. BFQ group scheduling
+ 4-1 Service guarantees provided
+ 4-2 Interface
+
+1. When may BFQ be useful?
+==========================
+
+BFQ provides the following benefits on personal and server systems.
+
+1-1 Personal systems
+--------------------
+
+Low latency for interactive applications
+
+Regardless of the actual background workload, BFQ guarantees that, for
+interactive tasks, the storage device is virtually as responsive as if
+it was idle. For example, even if one or more of the following
+background workloads are being executed:
+- one or more large files are being read, written or copied,
+- a tree of source files is being compiled,
+- one or more virtual machines are performing I/O,
+- a software update is in progress,
+- indexing daemons are scanning filesystems and updating their
+ databases,
+starting an application or loading a file from within an application
+takes about the same time as if the storage device was idle. As a
+comparison, with CFQ, NOOP or DEADLINE, and in the same conditions,
+applications experience high latencies, or even become unresponsive
+until the background workload terminates (also on SSDs).
+
+Low latency for soft real-time applications
+
+Also soft real-time applications, such as audio and video
+players/streamers, enjoy a low latency and a low drop rate, regardless
+of the background I/O workload. As a consequence, these applications
+do not suffer from almost any glitch due to the background workload.
+
+Higher speed for code-development tasks
+
+If some additional workload happens to be executed in parallel, then
+BFQ executes the I/O-related components of typical code-development
+tasks (compilation, checkout, merge, ...) much more quickly than CFQ,
+NOOP or DEADLINE.
+
+High throughput
+
+On hard disks, BFQ achieves up to 30% higher throughput than CFQ, and
+up to 150% higher throughput than DEADLINE and NOOP, with all the
+sequential workloads considered in our tests. With random workloads,
+and with all the workloads on flash-based devices, BFQ achieves,
+instead, about the same throughput as the other schedulers.
+
+Strong fairness, bandwidth and delay guarantees
+
+BFQ distributes the device throughput, and not just the device time,
+among I/O-bound applications in proportion their weights, with any
+workload and regardless of the device parameters. From these bandwidth
+guarantees, it is possible to compute tight per-I/O-request delay
+guarantees by a simple formula. If not configured for strict service
+guarantees, BFQ switches to time-based resource sharing (only) for
+applications that would otherwise cause a throughput loss.
+
+1-2 Server systems
+------------------
+
+Most benefits for server systems follow from the same service
+properties as above. In particular, regardless of whether additional,
+possibly heavy workloads are being served, BFQ guarantees:
+
+. audio and video-streaming with zero or very low jitter and drop
+ rate;
+
+. fast retrieval of WEB pages and embedded objects;
+
+. real-time recording of data in live-dumping applications (e.g.,
+ packet logging);
+
+. responsiveness in local and remote access to a server.
+
+
+2. How does BFQ work?
+=====================
+
+BFQ is a proportional-share I/O scheduler, whose general structure,
+plus a lot of code, are borrowed from CFQ.
+
+- Each process doing I/O on a device is associated with a weight and a
+ (bfq_)queue.
+
+- BFQ grants exclusive access to the device, for a while, to one queue
+ (process) at a time, and implements this service model by
+ associating every queue with a budget, measured in number of
+ sectors.
+
+ - After a queue is granted access to the device, the budget of the
+ queue is decremented, on each request dispatch, by the size of the
+ request.
+
+ - The in-service queue is expired, i.e., its service is suspended,
+ only if one of the following events occurs: 1) the queue finishes
+ its budget, 2) the queue empties, 3) a "budget timeout" fires.
+
+ - The budget timeout prevents processes doing random I/O from
+ holding the device for too long and dramatically reducing
+ throughput.
+
+ - Actually, as in CFQ, a queue associated with a process issuing
+ sync requests may not be expired immediately when it empties. In
+ contrast, BFQ may idle the device for a short time interval,
+ giving the process the chance to go on being served if it issues
+ a new request in time. Device idling typically boosts the
+ throughput on rotational devices, if processes do synchronous
+ and sequential I/O. In addition, under BFQ, device idling is
+ also instrumental in guaranteeing the desired throughput
+ fraction to processes issuing sync requests (see the description
+ of the slice_idle tunable in this document, or [1, 2], for more
+ details).
+
+ - With respect to idling for service guarantees, if several
+ processes are competing for the device at the same time, but
+ all processes (and groups, after the following commit) have
+ the same weight, then BFQ guarantees the expected throughput
+ distribution without ever idling the device. Throughput is
+ thus as high as possible in this common scenario.
+
+ - If low-latency mode is enabled (default configuration), BFQ
+ executes some special heuristics to detect interactive and soft
+ real-time applications (e.g., video or audio players/streamers),
+ and to reduce their latency. The most important action taken to
+ achieve this goal is to give to the queues associated with these
+ applications more than their fair share of the device
+ throughput. For brevity, we call just "weight-raising" the whole
+ sets of actions taken by BFQ to privilege these queues. In
+ particular, BFQ provides a milder form of weight-raising for
+ interactive applications, and a stronger form for soft real-time
+ applications.
+
+ - BFQ automatically deactivates idling for queues born in a burst of
+ queue creations. In fact, these queues are usually associated with
+ the processes of applications and services that benefit mostly
+ from a high throughput. Examples are systemd during boot, or git
+ grep.
+
+ - As CFQ, BFQ merges queues performing interleaved I/O, i.e.,
+ performing random I/O that becomes mostly sequential if
+ merged. Differently from CFQ, BFQ achieves this goal with a more
+ reactive mechanism, called Early Queue Merge (EQM). EQM is so
+ responsive in detecting interleaved I/O (cooperating processes),
+ that it enables BFQ to achieve a high throughput, by queue
+ merging, even for queues for which CFQ needs a different
+ mechanism, preemption, to get a high throughput. As such EQM is a
+ unified mechanism to achieve a high throughput with interleaved
+ I/O.
+
+ - Queues are scheduled according to a variant of WF2Q+, named
+ B-WF2Q+, and implemented using an augmented rb-tree to preserve an
+ O(log N) overall complexity. See [2] for more details. B-WF2Q+ is
+ also ready for hierarchical scheduling. However, for a cleaner
+ logical breakdown, the code that enables and completes
+ hierarchical support is provided in the next commit, which focuses
+ exactly on this feature.
+
+ - B-WF2Q+ guarantees a tight deviation with respect to an ideal,
+ perfectly fair, and smooth service. In particular, B-WF2Q+
+ guarantees that each queue receives a fraction of the device
+ throughput proportional to its weight, even if the throughput
+ fluctuates, and regardless of: the device parameters, the current
+ workload and the budgets assigned to the queue.
+
+ - The last, budget-independence, property (although probably
+ counterintuitive in the first place) is definitely beneficial, for
+ the following reasons:
+
+ - First, with any proportional-share scheduler, the maximum
+ deviation with respect to an ideal service is proportional to
+ the maximum budget (slice) assigned to queues. As a consequence,
+ BFQ can keep this deviation tight not only because of the
+ accurate service of B-WF2Q+, but also because BFQ *does not*
+ need to assign a larger budget to a queue to let the queue
+ receive a higher fraction of the device throughput.
+
+ - Second, BFQ is free to choose, for every process (queue), the
+ budget that best fits the needs of the process, or best
+ leverages the I/O pattern of the process. In particular, BFQ
+ updates queue budgets with a simple feedback-loop algorithm that
+ allows a high throughput to be achieved, while still providing
+ tight latency guarantees to time-sensitive applications. When
+ the in-service queue expires, this algorithm computes the next
+ budget of the queue so as to:
+
+ - Let large budgets be eventually assigned to the queues
+ associated with I/O-bound applications performing sequential
+ I/O: in fact, the longer these applications are served once
+ got access to the device, the higher the throughput is.
+
+ - Let small budgets be eventually assigned to the queues
+ associated with time-sensitive applications (which typically
+ perform sporadic and short I/O), because, the smaller the
+ budget assigned to a queue waiting for service is, the sooner
+ B-WF2Q+ will serve that queue (Subsec 3.3 in [2]).
+
+- If several processes are competing for the device at the same time,
+ but all processes and groups have the same weight, then BFQ
+ guarantees the expected throughput distribution without ever idling
+ the device. It uses preemption instead. Throughput is then much
+ higher in this common scenario.
+
+- ioprio classes are served in strict priority order, i.e.,
+ lower-priority queues are not served as long as there are
+ higher-priority queues. Among queues in the same class, the
+ bandwidth is distributed in proportion to the weight of each
+ queue. A very thin extra bandwidth is however guaranteed to
+ the Idle class, to prevent it from starving.
+
+
+3. What are BFQ's tunable?
+==========================
+
+The tunables back_seek-max, back_seek_penalty, fifo_expire_async and
+fifo_expire_sync below are the same as in CFQ. Their description is
+just copied from that for CFQ. Some considerations in the description
+of slice_idle are copied from CFQ too.
+
+per-process ioprio and weight
+-----------------------------
+
+Unless the cgroups interface is used (see "4. BFQ group scheduling"),
+weights can be assigned to processes only indirectly, through I/O
+priorities, and according to the relation:
+weight = (IOPRIO_BE_NR - ioprio) * 10.
+
+Beware that, if low-latency is set, then BFQ automatically raises the
+weight of the queues associated with interactive and soft real-time
+applications. Unset this tunable if you need/want to control weights.
+
+slice_idle
+----------
+
+This parameter specifies how long BFQ should idle for next I/O
+request, when certain sync BFQ queues become empty. By default
+slice_idle is a non-zero value. Idling has a double purpose: boosting
+throughput and making sure that the desired throughput distribution is
+respected (see the description of how BFQ works, and, if needed, the
+papers referred there).
+
+As for throughput, idling can be very helpful on highly seeky media
+like single spindle SATA/SAS disks where we can cut down on overall
+number of seeks and see improved throughput.
+
+Setting slice_idle to 0 will remove all the idling on queues and one
+should see an overall improved throughput on faster storage devices
+like multiple SATA/SAS disks in hardware RAID configuration.
+
+So depending on storage and workload, it might be useful to set
+slice_idle=0. In general for SATA/SAS disks and software RAID of
+SATA/SAS disks keeping slice_idle enabled should be useful. For any
+configurations where there are multiple spindles behind single LUN
+(Host based hardware RAID controller or for storage arrays), setting
+slice_idle=0 might end up in better throughput and acceptable
+latencies.
+
+Idling is however necessary to have service guarantees enforced in
+case of differentiated weights or differentiated I/O-request lengths.
+To see why, suppose that a given BFQ queue A must get several I/O
+requests served for each request served for another queue B. Idling
+ensures that, if A makes a new I/O request slightly after becoming
+empty, then no request of B is dispatched in the middle, and thus A
+does not lose the possibility to get more than one request dispatched
+before the next request of B is dispatched. Note that idling
+guarantees the desired differentiated treatment of queues only in
+terms of I/O-request dispatches. To guarantee that the actual service
+order then corresponds to the dispatch order, the strict_guarantees
+tunable must be set too.
+
+There is an important flipside for idling: apart from the above cases
+where it is beneficial also for throughput, idling can severely impact
+throughput. One important case is random workload. Because of this
+issue, BFQ tends to avoid idling as much as possible, when it is not
+beneficial also for throughput. As a consequence of this behavior, and
+of further issues described for the strict_guarantees tunable,
+short-term service guarantees may be occasionally violated. And, in
+some cases, these guarantees may be more important than guaranteeing
+maximum throughput. For example, in video playing/streaming, a very
+low drop rate may be more important than maximum throughput. In these
+cases, consider setting the strict_guarantees parameter.
+
+strict_guarantees
+-----------------
+
+If this parameter is set (default: unset), then BFQ
+
+- always performs idling when the in-service queue becomes empty;
+
+- forces the device to serve one I/O request at a time, by dispatching a
+ new request only if there is no outstanding request.
+
+In the presence of differentiated weights or I/O-request sizes, both
+the above conditions are needed to guarantee that every BFQ queue
+receives its allotted share of the bandwidth. The first condition is
+needed for the reasons explained in the description of the slice_idle
+tunable. The second condition is needed because all modern storage
+devices reorder internally-queued requests, which may trivially break
+the service guarantees enforced by the I/O scheduler.
+
+Setting strict_guarantees may evidently affect throughput.
+
+back_seek_max
+-------------
+
+This specifies, given in Kbytes, the maximum "distance" for backward seeking.
+The distance is the amount of space from the current head location to the
+sectors that are backward in terms of distance.
+
+This parameter allows the scheduler to anticipate requests in the "backward"
+direction and consider them as being the "next" if they are within this
+distance from the current head location.
+
+back_seek_penalty
+-----------------
+
+This parameter is used to compute the cost of backward seeking. If the
+backward distance of request is just 1/back_seek_penalty from a "front"
+request, then the seeking cost of two requests is considered equivalent.
+
+So scheduler will not bias toward one or the other request (otherwise scheduler
+will bias toward front request). Default value of back_seek_penalty is 2.
+
+fifo_expire_async
+-----------------
+
+This parameter is used to set the timeout of asynchronous requests. Default
+value of this is 248ms.
+
+fifo_expire_sync
+----------------
+
+This parameter is used to set the timeout of synchronous requests. Default
+value of this is 124ms. In case to favor synchronous requests over asynchronous
+one, this value should be decreased relative to fifo_expire_async.
+
+low_latency
+-----------
+
+This parameter is used to enable/disable BFQ's low latency mode. By
+default, low latency mode is enabled. If enabled, interactive and soft
+real-time applications are privileged and experience a lower latency,
+as explained in more detail in the description of how BFQ works.
+
+DO NOT enable this mode if you need full control on bandwidth
+distribution. In fact, if it is enabled, then BFQ automatically
+increases the bandwidth share of privileged applications, as the main
+means to guarantee a lower latency to them.
+
+timeout_sync
+------------
+
+Maximum amount of device time that can be given to a task (queue) once
+it has been selected for service. On devices with costly seeks,
+increasing this time usually increases maximum throughput. On the
+opposite end, increasing this time coarsens the granularity of the
+short-term bandwidth and latency guarantees, especially if the
+following parameter is set to zero.
+
+max_budget
+----------
+
+Maximum amount of service, measured in sectors, that can be provided
+to a BFQ queue once it is set in service (of course within the limits
+of the above timeout). According to what said in the description of
+the algorithm, larger values increase the throughput in proportion to
+the percentage of sequential I/O requests issued. The price of larger
+values is that they coarsen the granularity of short-term bandwidth
+and latency guarantees.
+
+The default value is 0, which enables auto-tuning: BFQ sets max_budget
+to the maximum number of sectors that can be served during
+timeout_sync, according to the estimated peak rate.
+
+weights
+-------
+
+Read-only parameter, used to show the weights of the currently active
+BFQ queues.
+
+
+wr_ tunables
+------------
+
+BFQ exports a few parameters to control/tune the behavior of
+low-latency heuristics.
+
+wr_coeff
+
+Factor by which the weight of a weight-raised queue is multiplied. If
+the queue is deemed soft real-time, then the weight is further
+multiplied by an additional, constant factor.
+
+wr_max_time
+
+Maximum duration of a weight-raising period for an interactive task
+(ms). If set to zero (default value), then this value is computed
+automatically, as a function of the peak rate of the device. In any
+case, when the value of this parameter is read, it always reports the
+current duration, regardless of whether it has been set manually or
+computed automatically.
+
+wr_max_softrt_rate
+
+Maximum service rate below which a queue is deemed to be associated
+with a soft real-time application, and is then weight-raised
+accordingly (sectors/sec).
+
+wr_min_idle_time
+
+Minimum idle period after which interactive weight-raising may be
+reactivated for a queue (in ms).
+
+wr_rt_max_time
+
+Maximum weight-raising duration for soft real-time queues (in ms). The
+start time from which this duration is considered is automatically
+moved forward if the queue is detected to be still soft real-time
+before the current soft real-time weight-raising period finishes.
+
+wr_min_inter_arr_async
+
+Minimum period between I/O request arrivals after which weight-raising
+may be reactivated for an already busy async queue (in ms).
+
+
+4. Group scheduling with BFQ
+============================
+
+BFQ supports both cgroups-v1 and cgroups-v2 io controllers, namely
+blkio and io. In particular, BFQ supports weight-based proportional
+share. To activate cgroups support, set BFQ_GROUP_IOSCHED.
+
+4-1 Service guarantees provided
+-------------------------------
+
+With BFQ, proportional share means true proportional share of the
+device bandwidth, according to group weights. For example, a group
+with weight 200 gets twice the bandwidth, and not just twice the time,
+of a group with weight 100.
+
+BFQ supports hierarchies (group trees) of any depth. Bandwidth is
+distributed among groups and processes in the expected way: for each
+group, the children of the group share the whole bandwidth of the
+group in proportion to their weights. In particular, this implies
+that, for each leaf group, every process of the group receives the
+same share of the whole group bandwidth, unless the ioprio of the
+process is modified.
+
+The resource-sharing guarantee for a group may partially or totally
+switch from bandwidth to time, if providing bandwidth guarantees to
+the group lowers the throughput too much. This switch occurs on a
+per-process basis: if a process of a leaf group causes throughput loss
+if served in such a way to receive its share of the bandwidth, then
+BFQ switches back to just time-based proportional share for that
+process.
+
+4-2 Interface
+-------------
+
+To get proportional sharing of bandwidth with BFQ for a given device,
+BFQ must of course be the active scheduler for that device.
+
+Within each group directory, the names of the files associated with
+BFQ-specific cgroup parameters and stats begin with the "bfq."
+prefix. So, with cgroups-v1 or cgroups-v2, the full prefix for
+BFQ-specific files is "blkio.bfq." or "io.bfq." For example, the group
+parameter to set the weight of a group with BFQ is blkio.bfq.weight
+or io.bfq.weight.
+
+Parameters to set
+-----------------
+
+For each group, there is only the following parameter to set.
+
+weight (namely blkio.bfq.weight or io.bfq-weight): the weight of the
+group inside its parent. Available values: 1..10000 (default 100). The
+linear mapping between ioprio and weights, described at the beginning
+of the tunable section, is still valid, but all weights higher than
+IOPRIO_BE_NR*10 are mapped to ioprio 0.
+
+Recall that, if low-latency is set, then BFQ automatically raises the
+weight of the queues associated with interactive and soft real-time
+applications. Unset this tunable if you need/want to control weights.
+
+
+[1] P. Valente, A. Avanzini, "Evolution of the BFQ Storage I/O
+ Scheduler", Proceedings of the First Workshop on Mobile System
+ Technologies (MST-2015), May 2015.
+ http://algogroup.unimore.it/people/paolo/disk_sched/mst-2015.pdf
+
+[2] P. Valente and M. Andreolini, "Improving Application
+ Responsiveness with the BFQ Disk I/O Scheduler", Proceedings of
+ the 5th Annual International Systems and Storage Conference
+ (SYSTOR '12), June 2012.
+ Slightly extended version:
+ http://algogroup.unimore.it/people/paolo/disk_sched/bfq-v1-suite-
+ results.pdf
diff --git a/Documentation/block/kyber-iosched.txt b/Documentation/block/kyber-iosched.txt
new file mode 100644
index 0000000..e94feac
--- /dev/null
+++ b/Documentation/block/kyber-iosched.txt
@@ -0,0 +1,14 @@
+Kyber I/O scheduler tunables
+===========================
+
+The only two tunables for the Kyber scheduler are the target latencies for
+reads and synchronous writes. Kyber will throttle requests in order to meet
+these target latencies.
+
+read_lat_nsec
+-------------
+Target latency for reads (in nanoseconds).
+
+write_lat_nsec
+--------------
+Target latency for synchronous writes (in nanoseconds).
diff --git a/Documentation/block/queue-sysfs.txt b/Documentation/block/queue-sysfs.txt
index c0a3bb5..2c1e670 100644
--- a/Documentation/block/queue-sysfs.txt
+++ b/Documentation/block/queue-sysfs.txt
@@ -43,11 +43,6 @@
smaller discards and potentially help reduce latencies induced by large
discard operations.
-discard_zeroes_data (RO)
-------------------------
-When read, this file will show if the discarded block are zeroed by the
-device or not. If its value is '1' the blocks are zeroed otherwise not.
-
hw_sector_size (RO)
-------------------
This is the hardware sector size of the device, in bytes.
@@ -192,5 +187,11 @@
feature. Writing a value of '-1' to this file resets the value to the
default setting.
+throttle_sample_time (RW)
+-------------------------
+This is the time window that blk-throttle samples data, in millisecond.
+blk-throttle makes decision based on the samplings. Lower time means cgroups
+have more smooth throughput, but higher CPU overhead. This exists only when
+CONFIG_BLK_DEV_THROTTLING_LOW is enabled.
Jens Axboe <jens.axboe@oracle.com>, February 2009
diff --git a/Documentation/blockdev/mflash.txt b/Documentation/blockdev/mflash.txt
deleted file mode 100644
index f7e0505..0000000
--- a/Documentation/blockdev/mflash.txt
+++ /dev/null
@@ -1,84 +0,0 @@
-This document describes m[g]flash support in linux.
-
-Contents
- 1. Overview
- 2. Reserved area configuration
- 3. Example of mflash platform driver registration
-
-1. Overview
-
-Mflash and gflash are embedded flash drive. The only difference is mflash is
-MCP(Multi Chip Package) device. These two device operate exactly same way.
-So the rest mflash repersents mflash and gflash altogether.
-
-Internally, mflash has nand flash and other hardware logics and supports
-2 different operation (ATA, IO) modes. ATA mode doesn't need any new
-driver and currently works well under standard IDE subsystem. Actually it's
-one chip SSD. IO mode is ATA-like custom mode for the host that doesn't have
-IDE interface.
-
-Following are brief descriptions about IO mode.
-A. IO mode based on ATA protocol and uses some custom command. (read confirm,
-write confirm)
-B. IO mode uses SRAM bus interface.
-C. IO mode supports 4kB boot area, so host can boot from mflash.
-
-2. Reserved area configuration
-If host boot from mflash, usually needs raw area for boot loader image. All of
-the mflash's block device operation will be taken this value as start offset.
-Note that boot loader's size of reserved area and kernel configuration value
-must be same.
-
-3. Example of mflash platform driver registration
-Working mflash is very straight forward. Adding platform device stuff to board
-configuration file is all. Here is some pseudo example.
-
-static struct mg_drv_data mflash_drv_data = {
- /* If you want to polling driver set to 1 */
- .use_polling = 0,
- /* device attribution */
- .dev_attr = MG_BOOT_DEV
-};
-
-static struct resource mg_mflash_rsc[] = {
- /* Base address of mflash */
- [0] = {
- .start = 0x08000000,
- .end = 0x08000000 + SZ_64K - 1,
- .flags = IORESOURCE_MEM
- },
- /* mflash interrupt pin */
- [1] = {
- .start = IRQ_GPIO(84),
- .end = IRQ_GPIO(84),
- .flags = IORESOURCE_IRQ
- },
- /* mflash reset pin */
- [2] = {
- .start = 43,
- .end = 43,
- .name = MG_RST_PIN,
- .flags = IORESOURCE_IO
- },
- /* mflash reset-out pin
- * If you use mflash as storage device (i.e. other than MG_BOOT_DEV),
- * should assign this */
- [3] = {
- .start = 51,
- .end = 51,
- .name = MG_RSTOUT_PIN,
- .flags = IORESOURCE_IO
- }
-};
-
-static struct platform_device mflash_dev = {
- .name = MG_DEV_NAME,
- .id = -1,
- .dev = {
- .platform_data = &mflash_drv_data,
- },
- .num_resources = ARRAY_SIZE(mg_mflash_rsc),
- .resource = mg_mflash_rsc
-};
-
-platform_device_register(&mflash_dev);
diff --git a/Documentation/conf.py b/Documentation/conf.py
index 7fadb3b..08aef45 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -17,7 +17,7 @@
import sphinx
# Get Sphinx version
-major, minor, patch = map(int, sphinx.__version__.split("."))
+major, minor, patch = sphinx.version_info[:3]
# If extensions (or modules to document with autodoc) are in another directory,
@@ -29,12 +29,12 @@
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
-#needs_sphinx = '1.0'
+needs_sphinx = '1.2'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
-extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain']
+extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure']
# The name of the math extension changed on Sphinx 1.4
if major == 1 and minor > 3:
diff --git a/Documentation/core-api/flexible-arrays.rst b/Documentation/core-api/flexible-arrays.rst
new file mode 100644
index 0000000..b6b85a1
--- /dev/null
+++ b/Documentation/core-api/flexible-arrays.rst
@@ -0,0 +1,130 @@
+
+===================================
+Using flexible arrays in the kernel
+===================================
+
+Large contiguous memory allocations can be unreliable in the Linux kernel.
+Kernel programmers will sometimes respond to this problem by allocating
+pages with :c:func:`vmalloc()`. This solution not ideal, though. On 32-bit
+systems, memory from vmalloc() must be mapped into a relatively small address
+space; it's easy to run out. On SMP systems, the page table changes required
+by vmalloc() allocations can require expensive cross-processor interrupts on
+all CPUs. And, on all systems, use of space in the vmalloc() range increases
+pressure on the translation lookaside buffer (TLB), reducing the performance
+of the system.
+
+In many cases, the need for memory from vmalloc() can be eliminated by piecing
+together an array from smaller parts; the flexible array library exists to make
+this task easier.
+
+A flexible array holds an arbitrary (within limits) number of fixed-sized
+objects, accessed via an integer index. Sparse arrays are handled
+reasonably well. Only single-page allocations are made, so memory
+allocation failures should be relatively rare. The down sides are that the
+arrays cannot be indexed directly, individual object size cannot exceed the
+system page size, and putting data into a flexible array requires a copy
+operation. It's also worth noting that flexible arrays do no internal
+locking at all; if concurrent access to an array is possible, then the
+caller must arrange for appropriate mutual exclusion.
+
+The creation of a flexible array is done with :c:func:`flex_array_alloc()`::
+
+ #include <linux/flex_array.h>
+
+ struct flex_array *flex_array_alloc(int element_size,
+ unsigned int total,
+ gfp_t flags);
+
+The individual object size is provided by ``element_size``, while total is the
+maximum number of objects which can be stored in the array. The flags
+argument is passed directly to the internal memory allocation calls. With
+the current code, using flags to ask for high memory is likely to lead to
+notably unpleasant side effects.
+
+It is also possible to define flexible arrays at compile time with::
+
+ DEFINE_FLEX_ARRAY(name, element_size, total);
+
+This macro will result in a definition of an array with the given name; the
+element size and total will be checked for validity at compile time.
+
+Storing data into a flexible array is accomplished with a call to
+:c:func:`flex_array_put()`::
+
+ int flex_array_put(struct flex_array *array, unsigned int element_nr,
+ void *src, gfp_t flags);
+
+This call will copy the data from src into the array, in the position
+indicated by ``element_nr`` (which must be less than the maximum specified when
+the array was created). If any memory allocations must be performed, flags
+will be used. The return value is zero on success, a negative error code
+otherwise.
+
+There might possibly be a need to store data into a flexible array while
+running in some sort of atomic context; in this situation, sleeping in the
+memory allocator would be a bad thing. That can be avoided by using
+``GFP_ATOMIC`` for the flags value, but, often, there is a better way. The
+trick is to ensure that any needed memory allocations are done before
+entering atomic context, using :c:func:`flex_array_prealloc()`::
+
+ int flex_array_prealloc(struct flex_array *array, unsigned int start,
+ unsigned int nr_elements, gfp_t flags);
+
+This function will ensure that memory for the elements indexed in the range
+defined by ``start`` and ``nr_elements`` has been allocated. Thereafter, a
+``flex_array_put()`` call on an element in that range is guaranteed not to
+block.
+
+Getting data back out of the array is done with :c:func:`flex_array_get()`::
+
+ void *flex_array_get(struct flex_array *fa, unsigned int element_nr);
+
+The return value is a pointer to the data element, or NULL if that
+particular element has never been allocated.
+
+Note that it is possible to get back a valid pointer for an element which
+has never been stored in the array. Memory for array elements is allocated
+one page at a time; a single allocation could provide memory for several
+adjacent elements. Flexible array elements are normally initialized to the
+value ``FLEX_ARRAY_FREE`` (defined as 0x6c in <linux/poison.h>), so errors
+involving that number probably result from use of unstored array entries.
+Note that, if array elements are allocated with ``__GFP_ZERO``, they will be
+initialized to zero and this poisoning will not happen.
+
+Individual elements in the array can be cleared with
+:c:func:`flex_array_clear()`::
+
+ int flex_array_clear(struct flex_array *array, unsigned int element_nr);
+
+This function will set the given element to ``FLEX_ARRAY_FREE`` and return
+zero. If storage for the indicated element is not allocated for the array,
+``flex_array_clear()`` will return ``-EINVAL`` instead. Note that clearing an
+element does not release the storage associated with it; to reduce the
+allocated size of an array, call :c:func:`flex_array_shrink()`::
+
+ int flex_array_shrink(struct flex_array *array);
+
+The return value will be the number of pages of memory actually freed.
+This function works by scanning the array for pages containing nothing but
+``FLEX_ARRAY_FREE`` bytes, so (1) it can be expensive, and (2) it will not work
+if the array's pages are allocated with ``__GFP_ZERO``.
+
+It is possible to remove all elements of an array with a call to
+:c:func:`flex_array_free_parts()`::
+
+ void flex_array_free_parts(struct flex_array *array);
+
+This call frees all elements, but leaves the array itself in place.
+Freeing the entire array is done with :c:func:`flex_array_free()`::
+
+ void flex_array_free(struct flex_array *array);
+
+As of this writing, there are no users of flexible arrays in the mainline
+kernel. The functions described here are also not exported to modules;
+that will probably be fixed when somebody comes up with a need for it.
+
+
+Flexible array functions
+------------------------
+
+.. kernel-doc:: include/linux/flex_array.h
diff --git a/Documentation/core-api/genericirq.rst b/Documentation/core-api/genericirq.rst
new file mode 100644
index 0000000..0054bd4
--- /dev/null
+++ b/Documentation/core-api/genericirq.rst
@@ -0,0 +1,440 @@
+.. include:: <isonum.txt>
+
+==========================
+Linux generic IRQ handling
+==========================
+
+:Copyright: |copy| 2005-2010: Thomas Gleixner
+:Copyright: |copy| 2005-2006: Ingo Molnar
+
+Introduction
+============
+
+The generic interrupt handling layer is designed to provide a complete
+abstraction of interrupt handling for device drivers. It is able to
+handle all the different types of interrupt controller hardware. Device
+drivers use generic API functions to request, enable, disable and free
+interrupts. The drivers do not have to know anything about interrupt
+hardware details, so they can be used on different platforms without
+code changes.
+
+This documentation is provided to developers who want to implement an
+interrupt subsystem based for their architecture, with the help of the
+generic IRQ handling layer.
+
+Rationale
+=========
+
+The original implementation of interrupt handling in Linux uses the
+:c:func:`__do_IRQ` super-handler, which is able to deal with every type of
+interrupt logic.
+
+Originally, Russell King identified different types of handlers to build
+a quite universal set for the ARM interrupt handler implementation in
+Linux 2.5/2.6. He distinguished between:
+
+- Level type
+
+- Edge type
+
+- Simple type
+
+During the implementation we identified another type:
+
+- Fast EOI type
+
+In the SMP world of the :c:func:`__do_IRQ` super-handler another type was
+identified:
+
+- Per CPU type
+
+This split implementation of high-level IRQ handlers allows us to
+optimize the flow of the interrupt handling for each specific interrupt
+type. This reduces complexity in that particular code path and allows
+the optimized handling of a given type.
+
+The original general IRQ implementation used hw_interrupt_type
+structures and their ``->ack``, ``->end`` [etc.] callbacks to differentiate
+the flow control in the super-handler. This leads to a mix of flow logic
+and low-level hardware logic, and it also leads to unnecessary code
+duplication: for example in i386, there is an ``ioapic_level_irq`` and an
+``ioapic_edge_irq`` IRQ-type which share many of the low-level details but
+have different flow handling.
+
+A more natural abstraction is the clean separation of the 'irq flow' and
+the 'chip details'.
+
+Analysing a couple of architecture's IRQ subsystem implementations
+reveals that most of them can use a generic set of 'irq flow' methods
+and only need to add the chip-level specific code. The separation is
+also valuable for (sub)architectures which need specific quirks in the
+IRQ flow itself but not in the chip details - and thus provides a more
+transparent IRQ subsystem design.
+
+Each interrupt descriptor is assigned its own high-level flow handler,
+which is normally one of the generic implementations. (This high-level
+flow handler implementation also makes it simple to provide
+demultiplexing handlers which can be found in embedded platforms on
+various architectures.)
+
+The separation makes the generic interrupt handling layer more flexible
+and extensible. For example, an (sub)architecture can use a generic
+IRQ-flow implementation for 'level type' interrupts and add a
+(sub)architecture specific 'edge type' implementation.
+
+To make the transition to the new model easier and prevent the breakage
+of existing implementations, the :c:func:`__do_IRQ` super-handler is still
+available. This leads to a kind of duality for the time being. Over time
+the new model should be used in more and more architectures, as it
+enables smaller and cleaner IRQ subsystems. It's deprecated for three
+years now and about to be removed.
+
+Known Bugs And Assumptions
+==========================
+
+None (knock on wood).
+
+Abstraction layers
+==================
+
+There are three main levels of abstraction in the interrupt code:
+
+1. High-level driver API
+
+2. High-level IRQ flow handlers
+
+3. Chip-level hardware encapsulation
+
+Interrupt control flow
+----------------------
+
+Each interrupt is described by an interrupt descriptor structure
+irq_desc. The interrupt is referenced by an 'unsigned int' numeric
+value which selects the corresponding interrupt description structure in
+the descriptor structures array. The descriptor structure contains
+status information and pointers to the interrupt flow method and the
+interrupt chip structure which are assigned to this interrupt.
+
+Whenever an interrupt triggers, the low-level architecture code calls
+into the generic interrupt code by calling :c:func:`desc->handle_irq`. This
+high-level IRQ handling function only uses desc->irq_data.chip
+primitives referenced by the assigned chip descriptor structure.
+
+High-level Driver API
+---------------------
+
+The high-level Driver API consists of following functions:
+
+- :c:func:`request_irq`
+
+- :c:func:`free_irq`
+
+- :c:func:`disable_irq`
+
+- :c:func:`enable_irq`
+
+- :c:func:`disable_irq_nosync` (SMP only)
+
+- :c:func:`synchronize_irq` (SMP only)
+
+- :c:func:`irq_set_irq_type`
+
+- :c:func:`irq_set_irq_wake`
+
+- :c:func:`irq_set_handler_data`
+
+- :c:func:`irq_set_chip`
+
+- :c:func:`irq_set_chip_data`
+
+See the autogenerated function documentation for details.
+
+High-level IRQ flow handlers
+----------------------------
+
+The generic layer provides a set of pre-defined irq-flow methods:
+
+- :c:func:`handle_level_irq`
+
+- :c:func:`handle_edge_irq`
+
+- :c:func:`handle_fasteoi_irq`
+
+- :c:func:`handle_simple_irq`
+
+- :c:func:`handle_percpu_irq`
+
+- :c:func:`handle_edge_eoi_irq`
+
+- :c:func:`handle_bad_irq`
+
+The interrupt flow handlers (either pre-defined or architecture
+specific) are assigned to specific interrupts by the architecture either
+during bootup or during device initialization.
+
+Default flow implementations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Helper functions
+^^^^^^^^^^^^^^^^
+
+The helper functions call the chip primitives and are used by the
+default flow implementations. The following helper functions are
+implemented (simplified excerpt)::
+
+ default_enable(struct irq_data *data)
+ {
+ desc->irq_data.chip->irq_unmask(data);
+ }
+
+ default_disable(struct irq_data *data)
+ {
+ if (!delay_disable(data))
+ desc->irq_data.chip->irq_mask(data);
+ }
+
+ default_ack(struct irq_data *data)
+ {
+ chip->irq_ack(data);
+ }
+
+ default_mask_ack(struct irq_data *data)
+ {
+ if (chip->irq_mask_ack) {
+ chip->irq_mask_ack(data);
+ } else {
+ chip->irq_mask(data);
+ chip->irq_ack(data);
+ }
+ }
+
+ noop(struct irq_data *data))
+ {
+ }
+
+
+
+Default flow handler implementations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Default Level IRQ flow handler
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+handle_level_irq provides a generic implementation for level-triggered
+interrupts.
+
+The following control flow is implemented (simplified excerpt)::
+
+ :c:func:`desc->irq_data.chip->irq_mask_ack`;
+ handle_irq_event(desc->action);
+ :c:func:`desc->irq_data.chip->irq_unmask`;
+
+
+Default Fast EOI IRQ flow handler
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+handle_fasteoi_irq provides a generic implementation for interrupts,
+which only need an EOI at the end of the handler.
+
+The following control flow is implemented (simplified excerpt)::
+
+ handle_irq_event(desc->action);
+ :c:func:`desc->irq_data.chip->irq_eoi`;
+
+
+Default Edge IRQ flow handler
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+handle_edge_irq provides a generic implementation for edge-triggered
+interrupts.
+
+The following control flow is implemented (simplified excerpt)::
+
+ if (desc->status & running) {
+ :c:func:`desc->irq_data.chip->irq_mask_ack`;
+ desc->status |= pending | masked;
+ return;
+ }
+ :c:func:`desc->irq_data.chip->irq_ack`;
+ desc->status |= running;
+ do {
+ if (desc->status & masked)
+ :c:func:`desc->irq_data.chip->irq_unmask`;
+ desc->status &= ~pending;
+ handle_irq_event(desc->action);
+ } while (status & pending);
+ desc->status &= ~running;
+
+
+Default simple IRQ flow handler
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+handle_simple_irq provides a generic implementation for simple
+interrupts.
+
+.. note::
+
+ The simple flow handler does not call any handler/chip primitives.
+
+The following control flow is implemented (simplified excerpt)::
+
+ handle_irq_event(desc->action);
+
+
+Default per CPU flow handler
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+handle_percpu_irq provides a generic implementation for per CPU
+interrupts.
+
+Per CPU interrupts are only available on SMP and the handler provides a
+simplified version without locking.
+
+The following control flow is implemented (simplified excerpt)::
+
+ if (desc->irq_data.chip->irq_ack)
+ :c:func:`desc->irq_data.chip->irq_ack`;
+ handle_irq_event(desc->action);
+ if (desc->irq_data.chip->irq_eoi)
+ :c:func:`desc->irq_data.chip->irq_eoi`;
+
+
+EOI Edge IRQ flow handler
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+handle_edge_eoi_irq provides an abnomination of the edge handler
+which is solely used to tame a badly wreckaged irq controller on
+powerpc/cell.
+
+Bad IRQ flow handler
+^^^^^^^^^^^^^^^^^^^^
+
+handle_bad_irq is used for spurious interrupts which have no real
+handler assigned..
+
+Quirks and optimizations
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The generic functions are intended for 'clean' architectures and chips,
+which have no platform-specific IRQ handling quirks. If an architecture
+needs to implement quirks on the 'flow' level then it can do so by
+overriding the high-level irq-flow handler.
+
+Delayed interrupt disable
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This per interrupt selectable feature, which was introduced by Russell
+King in the ARM interrupt implementation, does not mask an interrupt at
+the hardware level when :c:func:`disable_irq` is called. The interrupt is kept
+enabled and is masked in the flow handler when an interrupt event
+happens. This prevents losing edge interrupts on hardware which does not
+store an edge interrupt event while the interrupt is disabled at the
+hardware level. When an interrupt arrives while the IRQ_DISABLED flag
+is set, then the interrupt is masked at the hardware level and the
+IRQ_PENDING bit is set. When the interrupt is re-enabled by
+:c:func:`enable_irq` the pending bit is checked and if it is set, the interrupt
+is resent either via hardware or by a software resend mechanism. (It's
+necessary to enable CONFIG_HARDIRQS_SW_RESEND when you want to use
+the delayed interrupt disable feature and your hardware is not capable
+of retriggering an interrupt.) The delayed interrupt disable is not
+configurable.
+
+Chip-level hardware encapsulation
+---------------------------------
+
+The chip-level hardware descriptor structure :c:type:`irq_chip` contains all
+the direct chip relevant functions, which can be utilized by the irq flow
+implementations.
+
+- ``irq_ack``
+
+- ``irq_mask_ack`` - Optional, recommended for performance
+
+- ``irq_mask``
+
+- ``irq_unmask``
+
+- ``irq_eoi`` - Optional, required for EOI flow handlers
+
+- ``irq_retrigger`` - Optional
+
+- ``irq_set_type`` - Optional
+
+- ``irq_set_wake`` - Optional
+
+These primitives are strictly intended to mean what they say: ack means
+ACK, masking means masking of an IRQ line, etc. It is up to the flow
+handler(s) to use these basic units of low-level functionality.
+
+__do_IRQ entry point
+====================
+
+The original implementation :c:func:`__do_IRQ` was an alternative entry point
+for all types of interrupts. It no longer exists.
+
+This handler turned out to be not suitable for all interrupt hardware
+and was therefore reimplemented with split functionality for
+edge/level/simple/percpu interrupts. This is not only a functional
+optimization. It also shortens code paths for interrupts.
+
+Locking on SMP
+==============
+
+The locking of chip registers is up to the architecture that defines the
+chip primitives. The per-irq structure is protected via desc->lock, by
+the generic layer.
+
+Generic interrupt chip
+======================
+
+To avoid copies of identical implementations of IRQ chips the core
+provides a configurable generic interrupt chip implementation.
+Developers should check carefully whether the generic chip fits their
+needs before implementing the same functionality slightly differently
+themselves.
+
+.. kernel-doc:: kernel/irq/generic-chip.c
+ :export:
+
+Structures
+==========
+
+This chapter contains the autogenerated documentation of the structures
+which are used in the generic IRQ layer.
+
+.. kernel-doc:: include/linux/irq.h
+ :internal:
+
+.. kernel-doc:: include/linux/interrupt.h
+ :internal:
+
+Public Functions Provided
+=========================
+
+This chapter contains the autogenerated documentation of the kernel API
+functions which are exported.
+
+.. kernel-doc:: kernel/irq/manage.c
+
+.. kernel-doc:: kernel/irq/chip.c
+
+Internal Functions Provided
+===========================
+
+This chapter contains the autogenerated documentation of the internal
+functions.
+
+.. kernel-doc:: kernel/irq/irqdesc.c
+
+.. kernel-doc:: kernel/irq/handle.c
+
+.. kernel-doc:: kernel/irq/chip.c
+
+Credits
+=======
+
+The following people have contributed to this document:
+
+1. Thomas Gleixner tglx@linutronix.de
+
+2. Ingo Molnar mingo@elte.hu
diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 0d93d80..62abd36 100644
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -11,11 +11,14 @@
.. toctree::
:maxdepth: 1
+ kernel-api
assoc_array
atomic_ops
cpu_hotplug
local_ops
workqueue
+ genericirq
+ flexible-arrays
Interfaces for kernel debugging
===============================
diff --git a/Documentation/core-api/kernel-api.rst b/Documentation/core-api/kernel-api.rst
new file mode 100644
index 0000000..9ec8488
--- /dev/null
+++ b/Documentation/core-api/kernel-api.rst
@@ -0,0 +1,346 @@
+====================
+The Linux Kernel API
+====================
+
+Data Types
+==========
+
+Doubly Linked Lists
+-------------------
+
+.. kernel-doc:: include/linux/list.h
+ :internal:
+
+Basic C Library Functions
+=========================
+
+When writing drivers, you cannot in general use routines which are from
+the C Library. Some of the functions have been found generally useful
+and they are listed below. The behaviour of these functions may vary
+slightly from those defined by ANSI, and these deviations are noted in
+the text.
+
+String Conversions
+------------------
+
+.. kernel-doc:: lib/vsprintf.c
+ :export:
+
+.. kernel-doc:: include/linux/kernel.h
+ :functions: kstrtol
+
+.. kernel-doc:: include/linux/kernel.h
+ :functions: kstrtoul
+
+.. kernel-doc:: lib/kstrtox.c
+ :export:
+
+String Manipulation
+-------------------
+
+.. kernel-doc:: lib/string.c
+ :export:
+
+Bit Operations
+--------------
+
+.. kernel-doc:: arch/x86/include/asm/bitops.h
+ :internal:
+
+Basic Kernel Library Functions
+==============================
+
+The Linux kernel provides more basic utility functions.
+
+Bitmap Operations
+-----------------
+
+.. kernel-doc:: lib/bitmap.c
+ :export:
+
+.. kernel-doc:: lib/bitmap.c
+ :internal:
+
+Command-line Parsing
+--------------------
+
+.. kernel-doc:: lib/cmdline.c
+ :export:
+
+CRC Functions
+-------------
+
+.. kernel-doc:: lib/crc7.c
+ :export:
+
+.. kernel-doc:: lib/crc16.c
+ :export:
+
+.. kernel-doc:: lib/crc-itu-t.c
+ :export:
+
+.. kernel-doc:: lib/crc32.c
+
+.. kernel-doc:: lib/crc-ccitt.c
+ :export:
+
+idr/ida Functions
+-----------------
+
+.. kernel-doc:: include/linux/idr.h
+ :doc: idr sync
+
+.. kernel-doc:: lib/idr.c
+ :doc: IDA description
+
+.. kernel-doc:: lib/idr.c
+ :export:
+
+Memory Management in Linux
+==========================
+
+The Slab Cache
+--------------
+
+.. kernel-doc:: include/linux/slab.h
+ :internal:
+
+.. kernel-doc:: mm/slab.c
+ :export:
+
+.. kernel-doc:: mm/util.c
+ :export:
+
+User Space Memory Access
+------------------------
+
+.. kernel-doc:: arch/x86/include/asm/uaccess_32.h
+ :internal:
+
+.. kernel-doc:: arch/x86/lib/usercopy_32.c
+ :export:
+
+More Memory Management Functions
+--------------------------------
+
+.. kernel-doc:: mm/readahead.c
+ :export:
+
+.. kernel-doc:: mm/filemap.c
+ :export:
+
+.. kernel-doc:: mm/memory.c
+ :export:
+
+.. kernel-doc:: mm/vmalloc.c
+ :export:
+
+.. kernel-doc:: mm/page_alloc.c
+ :internal:
+
+.. kernel-doc:: mm/mempool.c
+ :export:
+
+.. kernel-doc:: mm/dmapool.c
+ :export:
+
+.. kernel-doc:: mm/page-writeback.c
+ :export:
+
+.. kernel-doc:: mm/truncate.c
+ :export:
+
+Kernel IPC facilities
+=====================
+
+IPC utilities
+-------------
+
+.. kernel-doc:: ipc/util.c
+ :internal:
+
+FIFO Buffer
+===========
+
+kfifo interface
+---------------
+
+.. kernel-doc:: include/linux/kfifo.h
+ :internal:
+
+relay interface support
+=======================
+
+Relay interface support is designed to provide an efficient mechanism
+for tools and facilities to relay large amounts of data from kernel
+space to user space.
+
+relay interface
+---------------
+
+.. kernel-doc:: kernel/relay.c
+ :export:
+
+.. kernel-doc:: kernel/relay.c
+ :internal:
+
+Module Support
+==============
+
+Module Loading
+--------------
+
+.. kernel-doc:: kernel/kmod.c
+ :export:
+
+Inter Module support
+--------------------
+
+Refer to the file kernel/module.c for more information.
+
+Hardware Interfaces
+===================
+
+Interrupt Handling
+------------------
+
+.. kernel-doc:: kernel/irq/manage.c
+ :export:
+
+DMA Channels
+------------
+
+.. kernel-doc:: kernel/dma.c
+ :export:
+
+Resources Management
+--------------------
+
+.. kernel-doc:: kernel/resource.c
+ :internal:
+
+.. kernel-doc:: kernel/resource.c
+ :export:
+
+MTRR Handling
+-------------
+
+.. kernel-doc:: arch/x86/kernel/cpu/mtrr/main.c
+ :export:
+
+Security Framework
+==================
+
+.. kernel-doc:: security/security.c
+ :internal:
+
+.. kernel-doc:: security/inode.c
+ :export:
+
+Audit Interfaces
+================
+
+.. kernel-doc:: kernel/audit.c
+ :export:
+
+.. kernel-doc:: kernel/auditsc.c
+ :internal:
+
+.. kernel-doc:: kernel/auditfilter.c
+ :internal:
+
+Accounting Framework
+====================
+
+.. kernel-doc:: kernel/acct.c
+ :internal:
+
+Block Devices
+=============
+
+.. kernel-doc:: block/blk-core.c
+ :export:
+
+.. kernel-doc:: block/blk-core.c
+ :internal:
+
+.. kernel-doc:: block/blk-map.c
+ :export:
+
+.. kernel-doc:: block/blk-sysfs.c
+ :internal:
+
+.. kernel-doc:: block/blk-settings.c
+ :export:
+
+.. kernel-doc:: block/blk-exec.c
+ :export:
+
+.. kernel-doc:: block/blk-flush.c
+ :export:
+
+.. kernel-doc:: block/blk-lib.c
+ :export:
+
+.. kernel-doc:: block/blk-tag.c
+ :export:
+
+.. kernel-doc:: block/blk-tag.c
+ :internal:
+
+.. kernel-doc:: block/blk-integrity.c
+ :export:
+
+.. kernel-doc:: kernel/trace/blktrace.c
+ :internal:
+
+.. kernel-doc:: block/genhd.c
+ :internal:
+
+.. kernel-doc:: block/genhd.c
+ :export:
+
+Char devices
+============
+
+.. kernel-doc:: fs/char_dev.c
+ :export:
+
+Clock Framework
+===============
+
+The clock framework defines programming interfaces to support software
+management of the system clock tree. This framework is widely used with
+System-On-Chip (SOC) platforms to support power management and various
+devices which may need custom clock rates. Note that these "clocks"
+don't relate to timekeeping or real time clocks (RTCs), each of which
+have separate frameworks. These :c:type:`struct clk <clk>`
+instances may be used to manage for example a 96 MHz signal that is used
+to shift bits into and out of peripherals or busses, or otherwise
+trigger synchronous state machine transitions in system hardware.
+
+Power management is supported by explicit software clock gating: unused
+clocks are disabled, so the system doesn't waste power changing the
+state of transistors that aren't in active use. On some systems this may
+be backed by hardware clock gating, where clocks are gated without being
+disabled in software. Sections of chips that are powered but not clocked
+may be able to retain their last state. This low power state is often
+called a *retention mode*. This mode still incurs leakage currents,
+especially with finer circuit geometries, but for CMOS circuits power is
+mostly used by clocked state changes.
+
+Power-aware drivers only enable their clocks when the device they manage
+is in active use. Also, system sleep states often differ according to
+which clock domains are active: while a "standby" state may allow wakeup
+from several active domains, a "mem" (suspend-to-RAM) state may require
+a more wholesale shutdown of clocks derived from higher speed PLLs and
+oscillators, limiting the number of possible wakeup event sources. A
+driver's suspend method may need to be aware of system-specific clock
+constraints on the target sleep state.
+
+Some platforms support programmable clock generators. These can be used
+by external chips of various kinds, such as other CPUs, multimedia
+codecs, and devices with strict requirements for interface clocking.
+
+.. kernel-doc:: include/linux/clk.h
+ :internal:
diff --git a/Documentation/cpu-freq/boost.txt b/Documentation/cpu-freq/boost.txt
deleted file mode 100644
index dd62e13..0000000
--- a/Documentation/cpu-freq/boost.txt
+++ /dev/null
@@ -1,93 +0,0 @@
-Processor boosting control
-
- - information for users -
-
-Quick guide for the impatient:
---------------------
-/sys/devices/system/cpu/cpufreq/boost
-controls the boost setting for the whole system. You can read and write
-that file with either "0" (boosting disabled) or "1" (boosting allowed).
-Reading or writing 1 does not mean that the system is boosting at this
-very moment, but only that the CPU _may_ raise the frequency at it's
-discretion.
---------------------
-
-Introduction
--------------
-Some CPUs support a functionality to raise the operating frequency of
-some cores in a multi-core package if certain conditions apply, mostly
-if the whole chip is not fully utilized and below it's intended thermal
-budget. The decision about boost disable/enable is made either at hardware
-(e.g. x86) or software (e.g ARM).
-On Intel CPUs this is called "Turbo Boost", AMD calls it "Turbo-Core",
-in technical documentation "Core performance boost". In Linux we use
-the term "boost" for convenience.
-
-Rationale for disable switch
-----------------------------
-
-Though the idea is to just give better performance without any user
-intervention, sometimes the need arises to disable this functionality.
-Most systems offer a switch in the (BIOS) firmware to disable the
-functionality at all, but a more fine-grained and dynamic control would
-be desirable:
-1. While running benchmarks, reproducible results are important. Since
- the boosting functionality depends on the load of the whole package,
- single thread performance can vary. By explicitly disabling the boost
- functionality at least for the benchmark's run-time the system will run
- at a fixed frequency and results are reproducible again.
-2. To examine the impact of the boosting functionality it is helpful
- to do tests with and without boosting.
-3. Boosting means overclocking the processor, though under controlled
- conditions. By raising the frequency and the voltage the processor
- will consume more power than without the boosting, which may be
- undesirable for instance for mobile users. Disabling boosting may
- save power here, though this depends on the workload.
-
-
-User controlled switch
-----------------------
-
-To allow the user to toggle the boosting functionality, the cpufreq core
-driver exports a sysfs knob to enable or disable it. There is a file:
-/sys/devices/system/cpu/cpufreq/boost
-which can either read "0" (boosting disabled) or "1" (boosting enabled).
-The file is exported only when cpufreq driver supports boosting.
-Explicitly changing the permissions and writing to that file anyway will
-return EINVAL.
-
-On supported CPUs one can write either a "0" or a "1" into this file.
-This will either disable the boost functionality on all cores in the
-whole system (0) or will allow the software or hardware to boost at will
-(1).
-
-Writing a "1" does not explicitly boost the system, but just allows the
-CPU to boost at their discretion. Some implementations take external
-factors like the chip's temperature into account, so boosting once does
-not necessarily mean that it will occur every time even using the exact
-same software setup.
-
-
-AMD legacy cpb switch
----------------------
-The AMD powernow-k8 driver used to support a very similar switch to
-disable or enable the "Core Performance Boost" feature of some AMD CPUs.
-This switch was instantiated in each CPU's cpufreq directory
-(/sys/devices/system/cpu[0-9]*/cpufreq) and was called "cpb".
-Though the per CPU existence hints at a more fine grained control, the
-actual implementation only supported a system-global switch semantics,
-which was simply reflected into each CPU's file. Writing a 0 or 1 into it
-would pull the other CPUs to the same state.
-For compatibility reasons this file and its behavior is still supported
-on AMD CPUs, though it is now protected by a config switch
-(X86_ACPI_CPUFREQ_CPB). On Intel CPUs this file will never be created,
-even with the config option set.
-This functionality is considered legacy and will be removed in some future
-kernel version.
-
-More fine grained boosting control
-----------------------------------
-
-Technically it is possible to switch the boosting functionality at least
-on a per package basis, for some CPUs even per core. Currently the driver
-does not support it, but this may be implemented in the future.
diff --git a/Documentation/cpu-freq/cpu-drivers.txt b/Documentation/cpu-freq/cpu-drivers.txt
index f71e6be..434c49c 100644
--- a/Documentation/cpu-freq/cpu-drivers.txt
+++ b/Documentation/cpu-freq/cpu-drivers.txt
@@ -231,7 +231,7 @@
Only for drivers with target_index() and CPUFREQ_ASYNC_NOTIFICATION unset.
get_intermediate should return a stable intermediate frequency platform wants to
-switch to, and target_intermediate() should set CPU to to that frequency, before
+switch to, and target_intermediate() should set CPU to that frequency, before
jumping to the frequency corresponding to 'index'. Core will take care of
sending notifications and driver doesn't have to handle them in
target_intermediate() or target_index().
diff --git a/Documentation/cpu-freq/governors.txt b/Documentation/cpu-freq/governors.txt
deleted file mode 100644
index 61b3184..0000000
--- a/Documentation/cpu-freq/governors.txt
+++ /dev/null
@@ -1,301 +0,0 @@
- CPU frequency and voltage scaling code in the Linux(TM) kernel
-
-
- L i n u x C P U F r e q
-
- C P U F r e q G o v e r n o r s
-
- - information for users and developers -
-
-
- Dominik Brodowski <linux@brodo.de>
- some additions and corrections by Nico Golde <nico@ngolde.de>
- Rafael J. Wysocki <rafael.j.wysocki@intel.com>
- Viresh Kumar <viresh.kumar@linaro.org>
-
-
-
- Clock scaling allows you to change the clock speed of the CPUs on the
- fly. This is a nice method to save battery power, because the lower
- the clock speed, the less power the CPU consumes.
-
-
-Contents:
----------
-1. What is a CPUFreq Governor?
-
-2. Governors In the Linux Kernel
-2.1 Performance
-2.2 Powersave
-2.3 Userspace
-2.4 Ondemand
-2.5 Conservative
-2.6 Schedutil
-
-3. The Governor Interface in the CPUfreq Core
-
-4. References
-
-
-1. What Is A CPUFreq Governor?
-==============================
-
-Most cpufreq drivers (except the intel_pstate and longrun) or even most
-cpu frequency scaling algorithms only allow the CPU frequency to be set
-to predefined fixed values. In order to offer dynamic frequency
-scaling, the cpufreq core must be able to tell these drivers of a
-"target frequency". So these specific drivers will be transformed to
-offer a "->target/target_index/fast_switch()" call instead of the
-"->setpolicy()" call. For set_policy drivers, all stays the same,
-though.
-
-How to decide what frequency within the CPUfreq policy should be used?
-That's done using "cpufreq governors".
-
-Basically, it's the following flow graph:
-
-CPU can be set to switch independently | CPU can only be set
- within specific "limits" | to specific frequencies
-
- "CPUfreq policy"
- consists of frequency limits (policy->{min,max})
- and CPUfreq governor to be used
- / \
- / \
- / the cpufreq governor decides
- / (dynamically or statically)
- / what target_freq to set within
- / the limits of policy->{min,max}
- / \
- / \
- Using the ->setpolicy call, Using the ->target/target_index/fast_switch call,
- the limits and the the frequency closest
- "policy" is set. to target_freq is set.
- It is assured that it
- is within policy->{min,max}
-
-
-2. Governors In the Linux Kernel
-================================
-
-2.1 Performance
----------------
-
-The CPUfreq governor "performance" sets the CPU statically to the
-highest frequency within the borders of scaling_min_freq and
-scaling_max_freq.
-
-
-2.2 Powersave
--------------
-
-The CPUfreq governor "powersave" sets the CPU statically to the
-lowest frequency within the borders of scaling_min_freq and
-scaling_max_freq.
-
-
-2.3 Userspace
--------------
-
-The CPUfreq governor "userspace" allows the user, or any userspace
-program running with UID "root", to set the CPU to a specific frequency
-by making a sysfs file "scaling_setspeed" available in the CPU-device
-directory.
-
-
-2.4 Ondemand
-------------
-
-The CPUfreq governor "ondemand" sets the CPU frequency depending on the
-current system load. Load estimation is triggered by the scheduler
-through the update_util_data->func hook; when triggered, cpufreq checks
-the CPU-usage statistics over the last period and the governor sets the
-CPU accordingly. The CPU must have the capability to switch the
-frequency very quickly.
-
-Sysfs files:
-
-* sampling_rate:
-
- Measured in uS (10^-6 seconds), this is how often you want the kernel
- to look at the CPU usage and to make decisions on what to do about the
- frequency. Typically this is set to values of around '10000' or more.
- It's default value is (cmp. with users-guide.txt): transition_latency
- * 1000. Be aware that transition latency is in ns and sampling_rate
- is in us, so you get the same sysfs value by default. Sampling rate
- should always get adjusted considering the transition latency to set
- the sampling rate 750 times as high as the transition latency in the
- bash (as said, 1000 is default), do:
-
- $ echo `$(($(cat cpuinfo_transition_latency) * 750 / 1000)) > ondemand/sampling_rate
-
-* sampling_rate_min:
-
- The sampling rate is limited by the HW transition latency:
- transition_latency * 100
-
- Or by kernel restrictions:
- - If CONFIG_NO_HZ_COMMON is set, the limit is 10ms fixed.
- - If CONFIG_NO_HZ_COMMON is not set or nohz=off boot parameter is
- used, the limits depend on the CONFIG_HZ option:
- HZ=1000: min=20000us (20ms)
- HZ=250: min=80000us (80ms)
- HZ=100: min=200000us (200ms)
-
- The highest value of kernel and HW latency restrictions is shown and
- used as the minimum sampling rate.
-
-* up_threshold:
-
- This defines what the average CPU usage between the samplings of
- 'sampling_rate' needs to be for the kernel to make a decision on
- whether it should increase the frequency. For example when it is set
- to its default value of '95' it means that between the checking
- intervals the CPU needs to be on average more than 95% in use to then
- decide that the CPU frequency needs to be increased.
-
-* ignore_nice_load:
-
- This parameter takes a value of '0' or '1'. When set to '0' (its
- default), all processes are counted towards the 'cpu utilisation'
- value. When set to '1', the processes that are run with a 'nice'
- value will not count (and thus be ignored) in the overall usage
- calculation. This is useful if you are running a CPU intensive
- calculation on your laptop that you do not care how long it takes to
- complete as you can 'nice' it and prevent it from taking part in the
- deciding process of whether to increase your CPU frequency.
-
-* sampling_down_factor:
-
- This parameter controls the rate at which the kernel makes a decision
- on when to decrease the frequency while running at top speed. When set
- to 1 (the default) decisions to reevaluate load are made at the same
- interval regardless of current clock speed. But when set to greater
- than 1 (e.g. 100) it acts as a multiplier for the scheduling interval
- for reevaluating load when the CPU is at its top speed due to high
- load. This improves performance by reducing the overhead of load
- evaluation and helping the CPU stay at its top speed when truly busy,
- rather than shifting back and forth in speed. This tunable has no
- effect on behavior at lower speeds/lower CPU loads.
-
-* powersave_bias:
-
- This parameter takes a value between 0 to 1000. It defines the
- percentage (times 10) value of the target frequency that will be
- shaved off of the target. For example, when set to 100 -- 10%, when
- ondemand governor would have targeted 1000 MHz, it will target
- 1000 MHz - (10% of 1000 MHz) = 900 MHz instead. This is set to 0
- (disabled) by default.
-
- When AMD frequency sensitivity powersave bias driver --
- drivers/cpufreq/amd_freq_sensitivity.c is loaded, this parameter
- defines the workload frequency sensitivity threshold in which a lower
- frequency is chosen instead of ondemand governor's original target.
- The frequency sensitivity is a hardware reported (on AMD Family 16h
- Processors and above) value between 0 to 100% that tells software how
- the performance of the workload running on a CPU will change when
- frequency changes. A workload with sensitivity of 0% (memory/IO-bound)
- will not perform any better on higher core frequency, whereas a
- workload with sensitivity of 100% (CPU-bound) will perform better
- higher the frequency. When the driver is loaded, this is set to 400 by
- default -- for CPUs running workloads with sensitivity value below
- 40%, a lower frequency is chosen. Unloading the driver or writing 0
- will disable this feature.
-
-
-2.5 Conservative
-----------------
-
-The CPUfreq governor "conservative", much like the "ondemand"
-governor, sets the CPU frequency depending on the current usage. It
-differs in behaviour in that it gracefully increases and decreases the
-CPU speed rather than jumping to max speed the moment there is any load
-on the CPU. This behaviour is more suitable in a battery powered
-environment. The governor is tweaked in the same manner as the
-"ondemand" governor through sysfs with the addition of:
-
-* freq_step:
-
- This describes what percentage steps the cpu freq should be increased
- and decreased smoothly by. By default the cpu frequency will increase
- in 5% chunks of your maximum cpu frequency. You can change this value
- to anywhere between 0 and 100 where '0' will effectively lock your CPU
- at a speed regardless of its load whilst '100' will, in theory, make
- it behave identically to the "ondemand" governor.
-
-* down_threshold:
-
- Same as the 'up_threshold' found for the "ondemand" governor but for
- the opposite direction. For example when set to its default value of
- '20' it means that if the CPU usage needs to be below 20% between
- samples to have the frequency decreased.
-
-* sampling_down_factor:
-
- Similar functionality as in "ondemand" governor. But in
- "conservative", it controls the rate at which the kernel makes a
- decision on when to decrease the frequency while running in any speed.
- Load for frequency increase is still evaluated every sampling rate.
-
-
-2.6 Schedutil
--------------
-
-The "schedutil" governor aims at better integration with the Linux
-kernel scheduler. Load estimation is achieved through the scheduler's
-Per-Entity Load Tracking (PELT) mechanism, which also provides
-information about the recent load [1]. This governor currently does
-load based DVFS only for tasks managed by CFS. RT and DL scheduler tasks
-are always run at the highest frequency. Unlike all the other
-governors, the code is located under the kernel/sched/ directory.
-
-Sysfs files:
-
-* rate_limit_us:
-
- This contains a value in microseconds. The governor waits for
- rate_limit_us time before reevaluating the load again, after it has
- evaluated the load once.
-
-For an in-depth comparison with the other governors refer to [2].
-
-
-3. The Governor Interface in the CPUfreq Core
-=============================================
-
-A new governor must register itself with the CPUfreq core using
-"cpufreq_register_governor". The struct cpufreq_governor, which has to
-be passed to that function, must contain the following values:
-
-governor->name - A unique name for this governor.
-governor->owner - .THIS_MODULE for the governor module (if appropriate).
-
-plus a set of hooks to the functions implementing the governor's logic.
-
-The CPUfreq governor may call the CPU processor driver using one of
-these two functions:
-
-int cpufreq_driver_target(struct cpufreq_policy *policy,
- unsigned int target_freq,
- unsigned int relation);
-
-int __cpufreq_driver_target(struct cpufreq_policy *policy,
- unsigned int target_freq,
- unsigned int relation);
-
-target_freq must be within policy->min and policy->max, of course.
-What's the difference between these two functions? When your governor is
-in a direct code path of a call to governor callbacks, like
-governor->start(), the policy->rwsem is still held in the cpufreq core,
-and there's no need to lock it again (in fact, this would cause a
-deadlock). So use __cpufreq_driver_target only in these cases. In all
-other cases (for example, when there's a "daemonized" function that
-wakes up every second), use cpufreq_driver_target to take policy->rwsem
-before the command is passed to the cpufreq driver.
-
-4. References
-=============
-
-[1] Per-entity load tracking: https://lwn.net/Articles/531853/
-[2] Improvements in CPU frequency management: https://lwn.net/Articles/682391/
-
diff --git a/Documentation/cpu-freq/index.txt b/Documentation/cpu-freq/index.txt
index ef1d392..03a7cee 100644
--- a/Documentation/cpu-freq/index.txt
+++ b/Documentation/cpu-freq/index.txt
@@ -21,8 +21,6 @@
amd-powernow.txt - AMD powernow driver specific file.
-boost.txt - Frequency boosting support.
-
core.txt - General description of the CPUFreq core and
of CPUFreq notifiers.
@@ -32,17 +30,12 @@
cpufreq-stats.txt - General description of sysfs cpufreq stats.
-governors.txt - What are cpufreq governors and how to
- implement them?
-
index.txt - File index, Mailing list and Links (this document)
intel-pstate.txt - Intel pstate cpufreq driver specific file.
pcc-cpufreq.txt - PCC cpufreq driver specific file.
-user-guide.txt - User Guide to CPUFreq
-
Mailing List
------------
diff --git a/Documentation/cpu-freq/user-guide.txt b/Documentation/cpu-freq/user-guide.txt
deleted file mode 100644
index 391da64..0000000
--- a/Documentation/cpu-freq/user-guide.txt
+++ /dev/null
@@ -1,228 +0,0 @@
- CPU frequency and voltage scaling code in the Linux(TM) kernel
-
-
- L i n u x C P U F r e q
-
- U S E R G U I D E
-
-
- Dominik Brodowski <linux@brodo.de>
-
-
-
- Clock scaling allows you to change the clock speed of the CPUs on the
- fly. This is a nice method to save battery power, because the lower
- the clock speed, the less power the CPU consumes.
-
-
-Contents:
----------
-1. Supported Architectures and Processors
-1.1 ARM and ARM64
-1.2 x86
-1.3 sparc64
-1.4 ppc
-1.5 SuperH
-1.6 Blackfin
-
-2. "Policy" / "Governor"?
-2.1 Policy
-2.2 Governor
-
-3. How to change the CPU cpufreq policy and/or speed
-3.1 Preferred interface: sysfs
-
-
-
-1. Supported Architectures and Processors
-=========================================
-
-1.1 ARM and ARM64
------------------
-
-Almost all ARM and ARM64 platforms support CPU frequency scaling.
-
-1.2 x86
--------
-
-The following processors for the x86 architecture are supported by cpufreq:
-
-AMD Elan - SC400, SC410
-AMD mobile K6-2+
-AMD mobile K6-3+
-AMD mobile Duron
-AMD mobile Athlon
-AMD Opteron
-AMD Athlon 64
-Cyrix Media GXm
-Intel mobile PIII and Intel mobile PIII-M on certain chipsets
-Intel Pentium 4, Intel Xeon
-Intel Pentium M (Centrino)
-National Semiconductors Geode GX
-Transmeta Crusoe
-Transmeta Efficeon
-VIA Cyrix 3 / C3
-various processors on some ACPI 2.0-compatible systems [*]
-And many more
-
-[*] Only if "ACPI Processor Performance States" are available
-to the ACPI<->BIOS interface.
-
-
-1.3 sparc64
------------
-
-The following processors for the sparc64 architecture are supported by
-cpufreq:
-
-UltraSPARC-III
-
-
-1.4 ppc
--------
-
-Several "PowerBook" and "iBook2" notebooks are supported.
-The following POWER processors are supported in powernv mode:
-POWER8
-POWER9
-
-1.5 SuperH
-----------
-
-All SuperH processors supporting rate rounding through the clock
-framework are supported by cpufreq.
-
-1.6 Blackfin
-------------
-
-The following Blackfin processors are supported by cpufreq:
-
-BF522, BF523, BF524, BF525, BF526, BF527, Rev 0.1 or higher
-BF531, BF532, BF533, Rev 0.3 or higher
-BF534, BF536, BF537, Rev 0.2 or higher
-BF561, Rev 0.3 or higher
-BF542, BF544, BF547, BF548, BF549, Rev 0.1 or higher
-
-
-2. "Policy" / "Governor" ?
-==========================
-
-Some CPU frequency scaling-capable processor switch between various
-frequencies and operating voltages "on the fly" without any kernel or
-user involvement. This guarantees very fast switching to a frequency
-which is high enough to serve the user's needs, but low enough to save
-power.
-
-
-2.1 Policy
-----------
-
-On these systems, all you can do is select the lower and upper
-frequency limit as well as whether you want more aggressive
-power-saving or more instantly available processing power.
-
-
-2.2 Governor
-------------
-
-On all other cpufreq implementations, these boundaries still need to
-be set. Then, a "governor" must be selected. Such a "governor" decides
-what speed the processor shall run within the boundaries. One such
-"governor" is the "userspace" governor. This one allows the user - or
-a yet-to-implement userspace program - to decide what specific speed
-the processor shall run at.
-
-
-3. How to change the CPU cpufreq policy and/or speed
-====================================================
-
-3.1 Preferred Interface: sysfs
-------------------------------
-
-The preferred interface is located in the sysfs filesystem. If you
-mounted it at /sys, the cpufreq interface is located in a subdirectory
-"cpufreq" within the cpu-device directory
-(e.g. /sys/devices/system/cpu/cpu0/cpufreq/ for the first CPU).
-
-affected_cpus : List of Online CPUs that require software
- coordination of frequency.
-
-cpuinfo_cur_freq : Current frequency of the CPU as obtained from
- the hardware, in KHz. This is the frequency
- the CPU actually runs at.
-
-cpuinfo_min_freq : this file shows the minimum operating
- frequency the processor can run at(in kHz)
-
-cpuinfo_max_freq : this file shows the maximum operating
- frequency the processor can run at(in kHz)
-
-cpuinfo_transition_latency The time it takes on this CPU to
- switch between two frequencies in nano
- seconds. If unknown or known to be
- that high that the driver does not
- work with the ondemand governor, -1
- (CPUFREQ_ETERNAL) will be returned.
- Using this information can be useful
- to choose an appropriate polling
- frequency for a kernel governor or
- userspace daemon. Make sure to not
- switch the frequency too often
- resulting in performance loss.
-
-related_cpus : List of Online + Offline CPUs that need software
- coordination of frequency.
-
-scaling_available_frequencies : List of available frequencies, in KHz.
-
-scaling_available_governors : this file shows the CPUfreq governors
- available in this kernel. You can see the
- currently activated governor in
-
-scaling_cur_freq : Current frequency of the CPU as determined by
- the governor and cpufreq core, in KHz. This is
- the frequency the kernel thinks the CPU runs
- at.
-
-scaling_driver : this file shows what cpufreq driver is
- used to set the frequency on this CPU
-
-scaling_governor, and by "echoing" the name of another
- governor you can change it. Please note
- that some governors won't load - they only
- work on some specific architectures or
- processors.
-
-scaling_min_freq and
-scaling_max_freq show the current "policy limits" (in
- kHz). By echoing new values into these
- files, you can change these limits.
- NOTE: when setting a policy you need to
- first set scaling_max_freq, then
- scaling_min_freq.
-
-scaling_setspeed This can be read to get the currently programmed
- value by the governor. This can be written to
- change the current frequency for a group of
- CPUs, represented by a policy. This is supported
- currently only by the userspace governor.
-
-bios_limit : If the BIOS tells the OS to limit a CPU to
- lower frequencies, the user can read out the
- maximum available frequency from this file.
- This typically can happen through (often not
- intended) BIOS settings, restrictions
- triggered through a service processor or other
- BIOS/HW based implementations.
- This does not cover thermal ACPI limitations
- which can be detected through the generic
- thermal driver.
-
-If you have selected the "userspace" governor which allows you to
-set the CPU operating frequency to a specific value, you can read out
-the current frequency in
-
-scaling_setspeed. By "echoing" a new frequency into this
- you can change the speed of the CPU,
- but only within the limits of
- scaling_min_freq and scaling_max_freq.
diff --git a/Documentation/cputopology.txt b/Documentation/cputopology.txt
index f722f22..127c9d8 100644
--- a/Documentation/cputopology.txt
+++ b/Documentation/cputopology.txt
@@ -100,7 +100,7 @@
For architectures that don't support books (CONFIG_SCHED_BOOK) there are no
default definitions for topology_book_id() and topology_book_cpumask().
-For architectures that don't support drawes (CONFIG_SCHED_DRAWER) there are
+For architectures that don't support drawers (CONFIG_SCHED_DRAWER) there are
no default definitions for topology_drawer_id() and topology_drawer_cpumask().
Additionally, CPU topology information is provided under
diff --git a/Documentation/crypto/api-samples.rst b/Documentation/crypto/api-samples.rst
index 0a10819..d021fd9 100644
--- a/Documentation/crypto/api-samples.rst
+++ b/Documentation/crypto/api-samples.rst
@@ -155,9 +155,9 @@
char ctx[];
};
- static struct sdescinit_sdesc(struct crypto_shash *alg)
+ static struct sdesc init_sdesc(struct crypto_shash *alg)
{
- struct sdescsdesc;
+ struct sdesc sdesc;
int size;
size = sizeof(struct shash_desc) + crypto_shash_descsize(alg);
@@ -172,7 +172,7 @@
static int calc_hash(struct crypto_shashalg,
const unsigned chardata, unsigned int datalen,
unsigned chardigest) {
- struct sdescsdesc;
+ struct sdesc sdesc;
int ret;
sdesc = init_sdesc(alg);
diff --git a/Documentation/debugging-via-ohci1394.txt b/Documentation/debugging-via-ohci1394.txt
index 03703af..9ff026d 100644
--- a/Documentation/debugging-via-ohci1394.txt
+++ b/Documentation/debugging-via-ohci1394.txt
@@ -100,8 +100,8 @@
CardBus and even some Express cards which are fully compliant to OHCI-1394
specification are available. If it requires no driver for Windows operating
systems, it most likely is. Only specialized shops have cards which are not
- compliant, they are based on TI PCILynx chips and require drivers for Win-
- dows operating systems.
+ compliant, they are based on TI PCILynx chips and require drivers for Windows
+ operating systems.
The mentioned kernel log message contains the string "physUB" if the
controller implements a writable Physical Upper Bound register. This is
diff --git a/Documentation/device-mapper/cache.txt b/Documentation/device-mapper/cache.txt
index f228604..cdfd0fe 100644
--- a/Documentation/device-mapper/cache.txt
+++ b/Documentation/device-mapper/cache.txt
@@ -290,7 +290,7 @@
range's end value is "one past the end", meaning 5-10 expresses a range
of values from 5 to 9. Each cblock must be expressed as a decimal
value, in the future a variant message that takes cblock ranges
-expressed in hexidecimal may be needed to better support efficient
+expressed in hexadecimal may be needed to better support efficient
invalidation of larger caches. The cache must be in passthrough mode
when invalidate_cblocks is used.
diff --git a/Documentation/devicetree/bindings/ata/ahci-dm816.txt b/Documentation/devicetree/bindings/ata/ahci-dm816.txt
new file mode 100644
index 0000000..f8c535f
--- /dev/null
+++ b/Documentation/devicetree/bindings/ata/ahci-dm816.txt
@@ -0,0 +1,21 @@
+Device tree binding for the TI DM816 AHCI SATA Controller
+---------------------------------------------------------
+
+Required properties:
+ - compatible: must be "ti,dm816-ahci"
+ - reg: physical base address and size of the register region used by
+ the controller (as defined by the AHCI 1.1 standard)
+ - interrupts: interrupt specifier (refer to the interrupt binding)
+ - clocks: list of phandle and clock specifier pairs (or only
+ phandles for clock providers with '0' defined for
+ #clock-cells); two clocks must be specified: the functional
+ clock and an external reference clock
+
+Example:
+
+ sata: sata@4a140000 {
+ compatible = "ti,dm816-ahci";
+ reg = <0x4a140000 0x10000>;
+ interrupts = <16>;
+ clocks = <&sysclk5_ck>, <&sata_refclk>;
+ };
diff --git a/Documentation/devicetree/bindings/clock/armada3700-xtal-clock.txt b/Documentation/devicetree/bindings/clock/armada3700-xtal-clock.txt
index a88f1f0..4c0807f 100644
--- a/Documentation/devicetree/bindings/clock/armada3700-xtal-clock.txt
+++ b/Documentation/devicetree/bindings/clock/armada3700-xtal-clock.txt
@@ -5,6 +5,7 @@
This node must be a subnode of the node exposing the register address
of the GPIO block where the gpio latch is located.
+See Documentation/devicetree/bindings/pinctrl/marvell,armada-37xx-pinctrl.txt
Required properties:
- compatible : shall be one of the following:
@@ -16,9 +17,9 @@
output names ("xtal")
Example:
-gpio1: gpio@13800 {
- compatible = "marvell,armada-3700-gpio", "syscon", "simple-mfd";
- reg = <0x13800 0x1000>;
+pinctrl_nb: pinctrl-nb@13800 {
+ compatible = "armada3710-nb-pinctrl", "syscon", "simple-mfd";
+ reg = <0x13800 0x100>, <0x13C00 0x20>;
xtalclk: xtal-clk {
compatible = "marvell,armada-3700-xtal-clock";
diff --git a/Documentation/devicetree/bindings/crypto/st,stm32-crc.txt b/Documentation/devicetree/bindings/crypto/st,stm32-crc.txt
new file mode 100644
index 0000000..3ba92a5
--- /dev/null
+++ b/Documentation/devicetree/bindings/crypto/st,stm32-crc.txt
@@ -0,0 +1,16 @@
+* STMicroelectronics STM32 CRC
+
+Required properties:
+- compatible: Should be "st,stm32f7-crc".
+- reg: The address and length of the peripheral registers space
+- clocks: The input clock of the CRC instance
+
+Optional properties: none
+
+Example:
+
+crc: crc@40023000 {
+ compatible = "st,stm32f7-crc";
+ reg = <0x40023000 0x400>;
+ clocks = <&rcc 0 12>;
+};
diff --git a/Documentation/devicetree/bindings/hwmon/ads7828.txt b/Documentation/devicetree/bindings/hwmon/ads7828.txt
new file mode 100644
index 0000000..fe0cc4a
--- /dev/null
+++ b/Documentation/devicetree/bindings/hwmon/ads7828.txt
@@ -0,0 +1,25 @@
+ads7828 properties
+
+Required properties:
+- compatible: Should be one of
+ ti,ads7828
+ ti,ads7830
+- reg: I2C address
+
+Optional properties:
+
+- ti,differential-input
+ Set to use the device in differential mode.
+- vref-supply
+ The external reference on the device is set to this regulators output. If it
+ does not exists the internal reference will be used and output by the ads78xx
+ on the "external vref" pin.
+
+ Example ADS7828 node:
+
+ ads7828: ads@48 {
+ comatible = "ti,ads7828";
+ reg = <0x48>;
+ vref-supply = <&vref>;
+ ti,differential-input;
+ };
diff --git a/Documentation/devicetree/bindings/hwmon/aspeed-pwm-tacho.txt b/Documentation/devicetree/bindings/hwmon/aspeed-pwm-tacho.txt
new file mode 100644
index 0000000..cf44605
--- /dev/null
+++ b/Documentation/devicetree/bindings/hwmon/aspeed-pwm-tacho.txt
@@ -0,0 +1,68 @@
+ASPEED AST2400/AST2500 PWM and Fan Tacho controller device driver
+
+The ASPEED PWM controller can support upto 8 PWM outputs. The ASPEED Fan Tacho
+controller can support upto 16 Fan tachometer inputs.
+
+There can be upto 8 fans supported. Each fan can have one PWM output and
+one/two Fan tach inputs.
+
+Required properties for pwm-tacho node:
+- #address-cells : should be 1.
+
+- #size-cells : should be 1.
+
+- reg : address and length of the register set for the device.
+
+- pinctrl-names : a pinctrl state named "default" must be defined.
+
+- pinctrl-0 : phandle referencing pin configuration of the PWM ports.
+
+- compatible : should be "aspeed,ast2400-pwm-tacho" for AST2400 and
+ "aspeed,ast2500-pwm-tacho" for AST2500.
+
+- clocks : a fixed clock providing input clock frequency(PWM
+ and Fan Tach clock)
+
+fan subnode format:
+===================
+Under fan subnode there can upto 8 child nodes, with each child node
+representing a fan. If there are 8 fans each fan can have one PWM port and
+one/two Fan tach inputs.
+
+Required properties for each child node:
+- reg : should specify PWM source port.
+ integer value in the range 0 to 7 with 0 indicating PWM port A and
+ 7 indicating PWM port H.
+
+- aspeed,fan-tach-ch : should specify the Fan tach input channel.
+ integer value in the range 0 through 15, with 0 indicating
+ Fan tach channel 0 and 15 indicating Fan tach channel 15.
+ Atleast one Fan tach input channel is required.
+
+Examples:
+
+pwm_tacho_fixed_clk: fixedclk {
+ compatible = "fixed-clock";
+ #clock-cells = <0>;
+ clock-frequency = <24000000>;
+};
+
+pwm_tacho: pwmtachocontroller@1e786000 {
+ #address-cells = <1>;
+ #size-cells = <1>;
+ reg = <0x1E786000 0x1000>;
+ compatible = "aspeed,ast2500-pwm-tacho";
+ clocks = <&pwm_tacho_fixed_clk>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_pwm0_default &pinctrl_pwm1_default>;
+
+ fan@0 {
+ reg = <0x00>;
+ aspeed,fan-tach-ch = /bits/ 8 <0x00>;
+ };
+
+ fan@1 {
+ reg = <0x01>;
+ aspeed,fan-tach-ch = /bits/ 8 <0x01 0x02>;
+ };
+};
diff --git a/Documentation/devicetree/bindings/hwmon/lm87.txt b/Documentation/devicetree/bindings/hwmon/lm87.txt
new file mode 100644
index 0000000..e1b7990
--- /dev/null
+++ b/Documentation/devicetree/bindings/hwmon/lm87.txt
@@ -0,0 +1,30 @@
+*LM87 hwmon sensor.
+
+Required properties:
+- compatible: Should be
+ "ti,lm87"
+
+- reg: I2C address
+
+optional properties:
+- has-temp3: This configures pins 18 and 19 to be used as a second
+ remote temperature sensing channel. By default the pins
+ are configured as voltage input pins in0 and in5.
+
+- has-in6: When set, pin 5 is configured to be used as voltage input
+ in6. Otherwise the pin is set as FAN1 input.
+
+- has-in7: When set, pin 6 is configured to be used as voltage input
+ in7. Otherwise the pin is set as FAN2 input.
+
+- vcc-supply: a Phandle for the regulator supplying power, can be
+ cofigured to measure 5.0V power supply. Default is 3.3V.
+
+Example:
+
+lm87@2e {
+ compatible = "ti,lm87";
+ reg = <0x2e>;
+ has-temp3;
+ vcc-supply = <®_5v0>;
+};
diff --git a/Documentation/devicetree/bindings/input/hid-over-i2c.txt b/Documentation/devicetree/bindings/input/hid-over-i2c.txt
index 488edcb..28e8bd8 100644
--- a/Documentation/devicetree/bindings/input/hid-over-i2c.txt
+++ b/Documentation/devicetree/bindings/input/hid-over-i2c.txt
@@ -17,6 +17,22 @@
- interrupt-parent: the phandle for the interrupt controller
- interrupts: interrupt line
+Additional optional properties:
+
+Some devices may support additional optional properties to help with, e.g.,
+power sequencing. The following properties can be supported by one or more
+device-specific compatible properties, which should be used in addition to the
+"hid-over-i2c" string.
+
+- compatible:
+ * "wacom,w9013" (Wacom W9013 digitizer). Supports:
+ - vdd-supply
+ - post-power-on-delay-ms
+
+- vdd-supply: phandle of the regulator that provides the supply voltage.
+- post-power-on-delay-ms: time required by the device after enabling its regulators
+ before it is ready for communication. Must be used with 'vdd-supply'.
+
Example:
i2c-hid-dev@2c {
diff --git a/Documentation/devicetree/bindings/interrupt-controller/cortina,gemini-interrupt-controller.txt b/Documentation/devicetree/bindings/interrupt-controller/faraday,ftintc010.txt
similarity index 63%
rename from Documentation/devicetree/bindings/interrupt-controller/cortina,gemini-interrupt-controller.txt
rename to Documentation/devicetree/bindings/interrupt-controller/faraday,ftintc010.txt
index 97c1167..24428d4 100644
--- a/Documentation/devicetree/bindings/interrupt-controller/cortina,gemini-interrupt-controller.txt
+++ b/Documentation/devicetree/bindings/interrupt-controller/faraday,ftintc010.txt
@@ -1,9 +1,12 @@
-* Cortina Systems Gemini interrupt controller
+* Faraday Technologt FTINTC010 interrupt controller
-This interrupt controller is found on the Gemini SoCs.
+This interrupt controller is a stock IP block from Faraday Technology found
+in the Gemini SoCs and other designs.
Required properties:
-- compatible: must be "cortina,gemini-interrupt-controller"
+- compatible: must be one of
+ "faraday,ftintc010"
+ "cortina,gemini-interrupt-controller" (deprecated)
- reg: The register bank for the interrupt controller.
- interrupt-controller: Identifies the node as an interrupt controller
- #interrupt-cells: The number of cells to define the interrupts.
@@ -15,7 +18,7 @@
Example:
interrupt-controller@48000000 {
- compatible = "cortina,gemini-interrupt-controller";
+ compatible = "faraday,ftintc010"
reg = <0x48000000 0x1000>;
interrupt-controller;
#interrupt-cells = <2>;
diff --git a/Documentation/devicetree/bindings/interrupt-controller/mediatek,cirq.txt b/Documentation/devicetree/bindings/interrupt-controller/mediatek,cirq.txt
new file mode 100644
index 0000000..a7efdbc
--- /dev/null
+++ b/Documentation/devicetree/bindings/interrupt-controller/mediatek,cirq.txt
@@ -0,0 +1,35 @@
+* Mediatek 27xx cirq
+
+In Mediatek SOCs, the CIRQ is a low power interrupt controller designed to
+work outside MCUSYS which comprises with Cortex-Ax cores,CCI and GIC.
+The external interrupts (outside MCUSYS) will feed through CIRQ and connect
+to GIC in MCUSYS. When CIRQ is enabled, it will record the edge-sensitive
+interrupts and generate a pulse signal to parent interrupt controller when
+flush command is executed. With CIRQ, MCUSYS can be completely turned off
+to improve the system power consumption without losing interrupts.
+
+Required properties:
+- compatible: should be one of
+ - "mediatek,mt2701-cirq" for mt2701 CIRQ
+ - "mediatek,mt8135-cirq" for mt8135 CIRQ
+ - "mediatek,mt8173-cirq" for mt8173 CIRQ
+ and "mediatek,cirq" as a fallback.
+- interrupt-controller : Identifies the node as an interrupt controller.
+- #interrupt-cells : Use the same format as specified by GIC in arm,gic.txt.
+- interrupt-parent: phandle of irq parent for cirq. The parent must
+ use the same interrupt-cells format as GIC.
+- reg: Physical base address of the cirq registers and length of memory
+ mapped region.
+- mediatek,ext-irq-range: Identifies external irq number range in different
+ SOCs.
+
+Example:
+ cirq: interrupt-controller@10204000 {
+ compatible = "mediatek,mt2701-cirq",
+ "mediatek,mtk-cirq";
+ interrupt-controller;
+ #interrupt-cells = <3>;
+ interrupt-parent = <&sysirq>;
+ reg = <0 0x10204000 0 0x400>;
+ mediatek,ext-irq-start = <32 200>;
+ };
diff --git a/Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt b/Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt
index 9d1d72c..a89c03b 100644
--- a/Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt
+++ b/Documentation/devicetree/bindings/interrupt-controller/mediatek,sysirq.txt
@@ -21,13 +21,16 @@
- interrupt-parent: phandle of irq parent for sysirq. The parent must
use the same interrupt-cells format as GIC.
- reg: Physical base address of the intpol registers and length of memory
- mapped region.
+ mapped region. Could be multiple bases here. Ex: mt6797 needs 2 reg, others
+ need 1.
Example:
- sysirq: interrupt-controller@10200100 {
- compatible = "mediatek,mt6589-sysirq", "mediatek,mt6577-sysirq";
+ sysirq: intpol-controller@10200620 {
+ compatible = "mediatek,mt6797-sysirq",
+ "mediatek,mt6577-sysirq";
interrupt-controller;
#interrupt-cells = <3>;
interrupt-parent = <&gic>;
- reg = <0 0x10200100 0 0x1c>;
+ reg = <0 0x10220620 0 0x20>,
+ <0 0x10220690 0 0x10>;
};
diff --git a/Documentation/devicetree/bindings/ipmi/aspeed,ast2400-ibt-bmc.txt b/Documentation/devicetree/bindings/ipmi/aspeed,ast2400-ibt-bmc.txt
index 6f28969..028268fd 100644
--- a/Documentation/devicetree/bindings/ipmi/aspeed,ast2400-ibt-bmc.txt
+++ b/Documentation/devicetree/bindings/ipmi/aspeed,ast2400-ibt-bmc.txt
@@ -6,7 +6,9 @@
Required properties:
-- compatible : should be "aspeed,ast2400-ibt-bmc"
+- compatible : should be one of
+ "aspeed,ast2400-ibt-bmc"
+ "aspeed,ast2500-ibt-bmc"
- reg: physical address and size of the registers
Optional properties:
diff --git a/Documentation/devicetree/bindings/leds/leds-cpcap.txt b/Documentation/devicetree/bindings/leds/leds-cpcap.txt
new file mode 100644
index 0000000..ebf7cdc
--- /dev/null
+++ b/Documentation/devicetree/bindings/leds/leds-cpcap.txt
@@ -0,0 +1,29 @@
+Motorola CPCAP PMIC LEDs
+------------------------
+
+This module is part of the CPCAP. For more details about the whole
+chip see Documentation/devicetree/bindings/mfd/motorola-cpcap.txt.
+
+Requires node properties:
+- compatible: should be one of
+ * "motorola,cpcap-led-mdl" (Main Display Lighting)
+ * "motorola,cpcap-led-kl" (Keyboard Lighting)
+ * "motorola,cpcap-led-adl" (Aux Display Lighting)
+ * "motorola,cpcap-led-red" (Red Triode)
+ * "motorola,cpcap-led-green" (Green Triode)
+ * "motorola,cpcap-led-blue" (Blue Triode)
+ * "motorola,cpcap-led-cf" (Camera Flash)
+ * "motorola,cpcap-led-bt" (Bluetooth)
+ * "motorola,cpcap-led-cp" (Camera Privacy LED)
+- label: see Documentation/devicetree/bindings/leds/common.txt
+- vdd-supply: A phandle to the regulator powering the LED
+
+Example:
+
+&cpcap {
+ cpcap_led_red: red-led {
+ compatible = "motorola,cpcap-led-red";
+ label = "cpcap:red";
+ vdd-supply = <&sw5>;
+ };
+};
diff --git a/Documentation/devicetree/bindings/leds/leds-mt6323.txt b/Documentation/devicetree/bindings/leds/leds-mt6323.txt
new file mode 100644
index 0000000..45bf9f7
--- /dev/null
+++ b/Documentation/devicetree/bindings/leds/leds-mt6323.txt
@@ -0,0 +1,60 @@
+Device Tree Bindings for LED support on MT6323 PMIC
+
+MT6323 LED controller is subfunction provided by MT6323 PMIC, so the LED
+controllers are defined as the subnode of the function node provided by MT6323
+PMIC controller that is being defined as one kind of Muti-Function Device (MFD)
+using shared bus called PMIC wrapper for each subfunction to access remote
+MT6323 PMIC hardware.
+
+For MT6323 MFD bindings see:
+Documentation/devicetree/bindings/mfd/mt6397.txt
+For MediaTek PMIC wrapper bindings see:
+Documentation/devicetree/bindings/soc/mediatek/pwrap.txt
+
+Required properties:
+- compatible : Must be "mediatek,mt6323-led"
+- address-cells : Must be 1
+- size-cells : Must be 0
+
+Each led is represented as a child node of the mediatek,mt6323-led that
+describes the initial behavior for each LED physically and currently only four
+LED child nodes can be supported.
+
+Required properties for the LED child node:
+- reg : LED channel number (0..3)
+
+Optional properties for the LED child node:
+- label : See Documentation/devicetree/bindings/leds/common.txt
+- linux,default-trigger : See Documentation/devicetree/bindings/leds/common.txt
+- default-state: See Documentation/devicetree/bindings/leds/common.txt
+
+Example:
+
+ mt6323: pmic {
+ compatible = "mediatek,mt6323";
+
+ ...
+
+ mt6323led: leds {
+ compatible = "mediatek,mt6323-led";
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ led@0 {
+ reg = <0>;
+ label = "LED0";
+ linux,default-trigger = "timer";
+ default-state = "on";
+ };
+ led@1 {
+ reg = <1>;
+ label = "LED1";
+ default-state = "off";
+ };
+ led@2 {
+ reg = <2>;
+ label = "LED2";
+ default-state = "on";
+ };
+ };
+ };
diff --git a/Documentation/devicetree/bindings/leds/leds-pca9532.txt b/Documentation/devicetree/bindings/leds/leds-pca9532.txt
index 198f3ba..f769c52 100644
--- a/Documentation/devicetree/bindings/leds/leds-pca9532.txt
+++ b/Documentation/devicetree/bindings/leds/leds-pca9532.txt
@@ -17,6 +17,8 @@
- label: see Documentation/devicetree/bindings/leds/common.txt
- type: Output configuration, see dt-bindings/leds/leds-pca9532.h (default NONE)
- linux,default-trigger: see Documentation/devicetree/bindings/leds/common.txt
+ - default-state: see Documentation/devicetree/bindings/leds/common.txt
+ This property is only valid for sub-nodes of type <PCA9532_TYPE_LED>.
Example:
#include <dt-bindings/leds/leds-pca9532.h>
@@ -33,6 +35,14 @@
label = "pca:green:power";
type = <PCA9532_TYPE_LED>;
};
+ kernel-booting {
+ type = <PCA9532_TYPE_LED>;
+ default-state = "on";
+ };
+ sys-stat {
+ type = <PCA9532_TYPE_LED>;
+ default-state = "keep"; // don't touch, was set by U-Boot
+ };
};
For more product information please see the link below:
diff --git a/Documentation/devicetree/bindings/mailbox/brcm,iproc-flexrm-mbox.txt b/Documentation/devicetree/bindings/mailbox/brcm,iproc-flexrm-mbox.txt
new file mode 100644
index 0000000..752ae6b
--- /dev/null
+++ b/Documentation/devicetree/bindings/mailbox/brcm,iproc-flexrm-mbox.txt
@@ -0,0 +1,59 @@
+Broadcom FlexRM Ring Manager
+============================
+The Broadcom FlexRM ring manager provides a set of rings which can be
+used to submit work to offload engines. An SoC may have multiple FlexRM
+hardware blocks. There is one device tree entry per FlexRM block. The
+FlexRM driver will create a mailbox-controller instance for given FlexRM
+hardware block where each mailbox channel is a separate FlexRM ring.
+
+Required properties:
+--------------------
+- compatible: Should be "brcm,iproc-flexrm-mbox"
+- reg: Specifies base physical address and size of the FlexRM
+ ring registers
+- msi-parent: Phandles (and potential Device IDs) to MSI controllers
+ The FlexRM engine will send MSIs (instead of wired
+ interrupts) to CPU. There is one MSI for each FlexRM ring.
+ Refer devicetree/bindings/interrupt-controller/msi.txt
+- #mbox-cells: Specifies the number of cells needed to encode a mailbox
+ channel. This should be 3.
+
+ The 1st cell is the mailbox channel number.
+
+ The 2nd cell contains MSI completion threshold. This is the
+ number of completion messages for which FlexRM will inject
+ one MSI interrupt to CPU.
+
+ The 3nd cell contains MSI timer value representing time for
+ which FlexRM will wait to accumulate N completion messages
+ where N is the value specified by 2nd cell above. If FlexRM
+ does not get required number of completion messages in time
+ specified by this cell then it will inject one MSI interrupt
+ to CPU provided atleast one completion message is available.
+
+Optional properties:
+--------------------
+- dma-coherent: Present if DMA operations made by the FlexRM engine (such
+ as DMA descriptor access, access to buffers pointed by DMA
+ descriptors and read/write pointer updates to DDR) are
+ cache coherent with the CPU.
+
+Example:
+--------
+crypto_mbox: mbox@67000000 {
+ compatible = "brcm,iproc-flexrm-mbox";
+ reg = <0x67000000 0x200000>;
+ msi-parent = <&gic_its 0x7f00>;
+ #mbox-cells = <3>;
+};
+
+crypto@672c0000 {
+ compatible = "brcm,spu2-v2-crypto";
+ reg = <0x672c0000 0x1000>;
+ mboxes = <&crypto_mbox 0 0x1 0xffff>,
+ <&crypto_mbox 1 0x1 0xffff>,
+ <&crypto_mbox 16 0x1 0xffff>,
+ <&crypto_mbox 17 0x1 0xffff>,
+ <&crypto_mbox 30 0x1 0xffff>,
+ <&crypto_mbox 31 0x1 0xffff>;
+};
diff --git a/Documentation/devicetree/bindings/mailbox/brcm,iproc-pdc-mbox.txt b/Documentation/devicetree/bindings/mailbox/brcm,iproc-pdc-mbox.txt
index 411ccf4..0f3ee81 100644
--- a/Documentation/devicetree/bindings/mailbox/brcm,iproc-pdc-mbox.txt
+++ b/Documentation/devicetree/bindings/mailbox/brcm,iproc-pdc-mbox.txt
@@ -1,9 +1,11 @@
The PDC driver manages data transfer to and from various offload engines
on some Broadcom SoCs. An SoC may have multiple PDC hardware blocks. There is
-one device tree entry per block.
+one device tree entry per block. On some chips, the PDC functionality is
+handled by the FA2 (Northstar Plus).
Required properties:
-- compatible : Should be "brcm,iproc-pdc-mbox".
+- compatible : Should be "brcm,iproc-pdc-mbox" or "brcm,iproc-fa2-mbox" for
+ FA2/Northstar Plus.
- reg: Should contain PDC registers location and length.
- interrupts: Should contain the IRQ line for the PDC.
- #mbox-cells: 1
diff --git a/Documentation/devicetree/bindings/mmc/brcm,bcm2835-sdhost.txt b/Documentation/devicetree/bindings/mmc/brcm,bcm2835-sdhost.txt
new file mode 100644
index 0000000..d876580
--- /dev/null
+++ b/Documentation/devicetree/bindings/mmc/brcm,bcm2835-sdhost.txt
@@ -0,0 +1,23 @@
+Broadcom BCM2835 SDHOST controller
+
+This file documents differences between the core properties described
+by mmc.txt and the properties that represent the BCM2835 controller.
+
+Required properties:
+- compatible: Should be "brcm,bcm2835-sdhost".
+- clocks: The clock feeding the SDHOST controller.
+
+Optional properties:
+- dmas: DMA channel for read and write.
+ See Documentation/devicetree/bindings/dma/dma.txt for details
+
+Example:
+
+sdhost: mmc@7e202000 {
+ compatible = "brcm,bcm2835-sdhost";
+ reg = <0x7e202000 0x100>;
+ interrupts = <2 24>;
+ clocks = <&clocks BCM2835_CLOCK_VPU>;
+ dmas = <&dma 13>;
+ dma-names = "rx-tx";
+};
diff --git a/Documentation/devicetree/bindings/mmc/cavium-mmc.txt b/Documentation/devicetree/bindings/mmc/cavium-mmc.txt
new file mode 100644
index 0000000..1433e62
--- /dev/null
+++ b/Documentation/devicetree/bindings/mmc/cavium-mmc.txt
@@ -0,0 +1,57 @@
+* Cavium Octeon & ThunderX MMC controller
+
+The highspeed MMC host controller on Caviums SoCs provides an interface
+for MMC and SD types of memory cards.
+
+Supported maximum speeds are the ones of the eMMC standard 4.41 as well
+as the speed of SD standard 4.0. Only 3.3 Volt is supported.
+
+Required properties:
+ - compatible : should be one of:
+ cavium,octeon-6130-mmc
+ cavium,octeon-7890-mmc
+ cavium,thunder-8190-mmc
+ cavium,thunder-8390-mmc
+ mmc-slot
+ - reg : mmc controller base registers
+ - clocks : phandle
+
+Optional properties:
+ - for cd, bus-width and additional generic mmc parameters
+ please refer to mmc.txt within this directory
+ - cavium,cmd-clk-skew : number of coprocessor clocks before sampling command
+ - cavium,dat-clk-skew : number of coprocessor clocks before sampling data
+
+Deprecated properties:
+- spi-max-frequency : use max-frequency instead
+- cavium,bus-max-width : use bus-width instead
+- power-gpios : use vmmc-supply instead
+- cavium,octeon-6130-mmc-slot : use mmc-slot instead
+
+Examples:
+ mmc_1_4: mmc@1,4 {
+ compatible = "cavium,thunder-8390-mmc";
+ reg = <0x0c00 0 0 0 0>; /* DEVFN = 0x0c (1:4) */
+ #address-cells = <1>;
+ #size-cells = <0>;
+ clocks = <&sclk>;
+
+ mmc-slot@0 {
+ compatible = "mmc-slot";
+ reg = <0>;
+ vmmc-supply = <&mmc_supply_3v3>;
+ max-frequency = <42000000>;
+ bus-width = <4>;
+ cap-sd-highspeed;
+ };
+
+ mmc-slot@1 {
+ compatible = "mmc-slot";
+ reg = <1>;
+ vmmc-supply = <&mmc_supply_3v3>;
+ max-frequency = <42000000>;
+ bus-width = <8>;
+ cap-mmc-highspeed;
+ non-removable;
+ };
+ };
diff --git a/Documentation/devicetree/bindings/mmc/marvell,xenon-sdhci.txt b/Documentation/devicetree/bindings/mmc/marvell,xenon-sdhci.txt
new file mode 100644
index 0000000..b878a1e
--- /dev/null
+++ b/Documentation/devicetree/bindings/mmc/marvell,xenon-sdhci.txt
@@ -0,0 +1,170 @@
+Marvell Xenon SDHCI Controller device tree bindings
+This file documents differences between the core mmc properties
+described by mmc.txt and the properties used by the Xenon implementation.
+
+Multiple SDHCs might be put into a single Xenon IP, to save size and cost.
+Each SDHC is independent and owns independent resources, such as register sets,
+clock and PHY.
+Each SDHC should have an independent device tree node.
+
+Required Properties:
+- compatible: should be one of the following
+ - "marvell,armada-3700-sdhci": For controllers on Armada-3700 SoC.
+ Must provide a second register area and marvell,pad-type.
+ - "marvell,armada-ap806-sdhci": For controllers on Armada AP806.
+ - "marvell,armada-cp110-sdhci": For controllers on Armada CP110.
+
+- clocks:
+ Array of clocks required for SDHC.
+ Require at least input clock for Xenon IP core.
+
+- clock-names:
+ Array of names corresponding to clocks property.
+ The input clock for Xenon IP core should be named as "core".
+
+- reg:
+ * For "marvell,armada-3700-sdhci", two register areas.
+ The first one for Xenon IP register. The second one for the Armada 3700 SoC
+ PHY PAD Voltage Control register.
+ Please follow the examples with compatible "marvell,armada-3700-sdhci"
+ in below.
+ Please also check property marvell,pad-type in below.
+
+ * For other compatible strings, one register area for Xenon IP.
+
+Optional Properties:
+- marvell,xenon-sdhc-id:
+ Indicate the corresponding bit index of current SDHC in
+ SDHC System Operation Control Register Bit[7:0].
+ Set/clear the corresponding bit to enable/disable current SDHC.
+ If Xenon IP contains only one SDHC, this property is optional.
+
+- marvell,xenon-phy-type:
+ Xenon support multiple types of PHYs.
+ To select eMMC 5.1 PHY, set:
+ marvell,xenon-phy-type = "emmc 5.1 phy"
+ eMMC 5.1 PHY is the default choice if this property is not provided.
+ To select eMMC 5.0 PHY, set:
+ marvell,xenon-phy-type = "emmc 5.0 phy"
+
+ All those types of PHYs can support eMMC, SD and SDIO.
+ Please note that this property only presents the type of PHY.
+ It doesn't stand for the entire SDHC type or property.
+ For example, "emmc 5.1 phy" doesn't mean that this Xenon SDHC only
+ supports eMMC 5.1.
+
+- marvell,xenon-phy-znr:
+ Set PHY ZNR value.
+ Only available for eMMC PHY.
+ Valid range = [0:0x1F].
+ ZNR is set as 0xF by default if this property is not provided.
+
+- marvell,xenon-phy-zpr:
+ Set PHY ZPR value.
+ Only available for eMMC PHY.
+ Valid range = [0:0x1F].
+ ZPR is set as 0xF by default if this property is not provided.
+
+- marvell,xenon-phy-nr-success-tun:
+ Set the number of required consecutive successful sampling points
+ used to identify a valid sampling window, in tuning process.
+ Valid range = [1:7].
+ Set as 0x4 by default if this property is not provided.
+
+- marvell,xenon-phy-tun-step-divider:
+ Set the divider for calculating TUN_STEP.
+ Set as 64 by default if this property is not provided.
+
+- marvell,xenon-phy-slow-mode:
+ If this property is selected, transfers will bypass PHY.
+ Only available when bus frequency lower than 55MHz in SDR mode.
+ Disabled by default. Please only try this property if timing issues
+ always occur with PHY enabled in eMMC HS SDR, SD SDR12, SD SDR25,
+ SD Default Speed and HS mode and eMMC legacy speed mode.
+
+- marvell,xenon-tun-count:
+ Xenon SDHC SoC usually doesn't provide re-tuning counter in
+ Capabilities Register 3 Bit[11:8].
+ This property provides the re-tuning counter.
+ If this property is not set, default re-tuning counter will
+ be set as 0x9 in driver.
+
+- marvell,pad-type:
+ Type of Armada 3700 SoC PHY PAD Voltage Controller register.
+ Only valid when "marvell,armada-3700-sdhci" is selected.
+ Two types: "sd" and "fixed-1-8v".
+ If "sd" is selected, SoC PHY PAD is set as 3.3V at the beginning and is
+ switched to 1.8V when later in higher speed mode.
+ If "fixed-1-8v" is selected, SoC PHY PAD is fixed 1.8V, such as for eMMC.
+ Please follow the examples with compatible "marvell,armada-3700-sdhci"
+ in below.
+
+Example:
+- For eMMC:
+
+ sdhci@aa0000 {
+ compatible = "marvell,armada-ap806-sdhci";
+ reg = <0xaa0000 0x1000>;
+ interrupts = <GIC_SPI 13 IRQ_TYPE_LEVEL_HIGH>
+ clocks = <&emmc_clk>;
+ clock-names = "core";
+ bus-width = <4>;
+ marvell,xenon-phy-slow-mode;
+ marvell,xenon-tun-count = <11>;
+ non-removable;
+ no-sd;
+ no-sdio;
+
+ /* Vmmc and Vqmmc are both fixed */
+ };
+
+- For SD/SDIO:
+
+ sdhci@ab0000 {
+ compatible = "marvell,armada-cp110-sdhci";
+ reg = <0xab0000 0x1000>;
+ interrupts = <GIC_SPI 55 IRQ_TYPE_LEVEL_HIGH>
+ vqmmc-supply = <&sd_vqmmc_regulator>;
+ vmmc-supply = <&sd_vmmc_regulator>;
+ clocks = <&sdclk>;
+ clock-names = "core";
+ bus-width = <4>;
+ marvell,xenon-tun-count = <9>;
+ };
+
+- For eMMC with compatible "marvell,armada-3700-sdhci":
+
+ sdhci@aa0000 {
+ compatible = "marvell,armada-3700-sdhci";
+ reg = <0xaa0000 0x1000>,
+ <phy_addr 0x4>;
+ interrupts = <GIC_SPI 13 IRQ_TYPE_LEVEL_HIGH>
+ clocks = <&emmcclk>;
+ clock-names = "core";
+ bus-width = <8>;
+ mmc-ddr-1_8v;
+ mmc-hs400-1_8v;
+ non-removable;
+ no-sd;
+ no-sdio;
+
+ /* Vmmc and Vqmmc are both fixed */
+
+ marvell,pad-type = "fixed-1-8v";
+ };
+
+- For SD/SDIO with compatible "marvell,armada-3700-sdhci":
+
+ sdhci@ab0000 {
+ compatible = "marvell,armada-3700-sdhci";
+ reg = <0xab0000 0x1000>,
+ <phy_addr 0x4>;
+ interrupts = <GIC_SPI 55 IRQ_TYPE_LEVEL_HIGH>
+ vqmmc-supply = <&sd_regulator>;
+ /* Vmmc is fixed */
+ clocks = <&sdclk>;
+ clock-names = "core";
+ bus-width = <4>;
+
+ marvell,pad-type = "sd";
+ };
diff --git a/Documentation/devicetree/bindings/mmc/mtk-sd.txt b/Documentation/devicetree/bindings/mmc/mtk-sd.txt
index 0120c7f..4182ea3 100644
--- a/Documentation/devicetree/bindings/mmc/mtk-sd.txt
+++ b/Documentation/devicetree/bindings/mmc/mtk-sd.txt
@@ -21,6 +21,15 @@
- assigned-clocks: PLL of the source clock
- assigned-clock-parents: parent of source clock, used for HS400 mode to get 400Mhz source clock
- hs400-ds-delay: HS400 DS delay setting
+- mediatek,hs200-cmd-int-delay: HS200 command internal delay setting.
+ This field has total 32 stages.
+ The value is an integer from 0 to 31.
+- mediatek,hs400-cmd-int-delay: HS400 command internal delay setting
+ This field has total 32 stages.
+ The value is an integer from 0 to 31.
+- mediatek,hs400-cmd-resp-sel-rising: HS400 command response sample selection
+ If present,HS400 command responses are sampled on rising edges.
+ If not present,HS400 command responses are sampled on falling edges.
Examples:
mmc0: mmc@11230000 {
@@ -38,4 +47,7 @@
assigned-clocks = <&topckgen CLK_TOP_MSDC50_0_SEL>;
assigned-clock-parents = <&topckgen CLK_TOP_MSDCPLL_D2>;
hs400-ds-delay = <0x14015>;
+ mediatek,hs200-cmd-int-delay = <26>;
+ mediatek,hs400-cmd-int-delay = <14>;
+ mediatek,hs400-cmd-resp-sel-rising;
};
diff --git a/Documentation/devicetree/bindings/mmc/nvidia,tegra20-sdhci.txt b/Documentation/devicetree/bindings/mmc/nvidia,tegra20-sdhci.txt
index 15b8368..9bce578 100644
--- a/Documentation/devicetree/bindings/mmc/nvidia,tegra20-sdhci.txt
+++ b/Documentation/devicetree/bindings/mmc/nvidia,tegra20-sdhci.txt
@@ -7,11 +7,13 @@
by mmc.txt and the properties used by the sdhci-tegra driver.
Required properties:
-- compatible : For Tegra20, must contain "nvidia,tegra20-sdhci".
- For Tegra30, must contain "nvidia,tegra30-sdhci". For Tegra114,
- must contain "nvidia,tegra114-sdhci". For Tegra124, must contain
- "nvidia,tegra124-sdhci". Otherwise, must contain "nvidia,<chip>-sdhci",
- plus one of the above, where <chip> is tegra132 or tegra210.
+- compatible : should be one of:
+ - "nvidia,tegra20-sdhci": for Tegra20
+ - "nvidia,tegra30-sdhci": for Tegra30
+ - "nvidia,tegra114-sdhci": for Tegra114
+ - "nvidia,tegra124-sdhci": for Tegra124 and Tegra132
+ - "nvidia,tegra210-sdhci": for Tegra210
+ - "nvidia,tegra186-sdhci": for Tegra186
- clocks : Must contain one entry, for the module clock.
See ../clocks/clock-bindings.txt for details.
- resets : Must contain an entry for each entry in reset-names.
diff --git a/Documentation/devicetree/bindings/mmc/renesas,mmcif.txt b/Documentation/devicetree/bindings/mmc/renesas,mmcif.txt
index e4ba92aa..c32dc5a 100644
--- a/Documentation/devicetree/bindings/mmc/renesas,mmcif.txt
+++ b/Documentation/devicetree/bindings/mmc/renesas,mmcif.txt
@@ -8,6 +8,7 @@
- compatible: should be "renesas,mmcif-<soctype>", "renesas,sh-mmcif" as a
fallback. Examples with <soctype> are:
+ - "renesas,mmcif-r7s72100" for the MMCIF found in r7s72100 SoCs
- "renesas,mmcif-r8a73a4" for the MMCIF found in r8a73a4 SoCs
- "renesas,mmcif-r8a7740" for the MMCIF found in r8a7740 SoCs
- "renesas,mmcif-r8a7778" for the MMCIF found in r8a7778 SoCs
@@ -17,6 +18,13 @@
- "renesas,mmcif-r8a7794" for the MMCIF found in r8a7794 SoCs
- "renesas,mmcif-sh73a0" for the MMCIF found in sh73a0 SoCs
+- interrupts: Some SoCs have only 1 shared interrupt, while others have either
+ 2 or 3 individual interrupts (error, int, card detect). Below is the number
+ of interrupts for each SoC:
+ 1: r8a73a4, r8a7778, r8a7790, r8a7791, r8a7793, r8a7794
+ 2: r8a7740, sh73a0
+ 3: r7s72100
+
- clocks: reference to the functional clock
- dmas: reference to the DMA channels, one per channel name listed in the
diff --git a/Documentation/devicetree/bindings/mmc/samsung,s3cmci.txt b/Documentation/devicetree/bindings/mmc/samsung,s3cmci.txt
new file mode 100644
index 0000000..5f68feb
--- /dev/null
+++ b/Documentation/devicetree/bindings/mmc/samsung,s3cmci.txt
@@ -0,0 +1,42 @@
+* Samsung's S3C24XX MMC/SD/SDIO controller device tree bindings
+
+Samsung's S3C24XX MMC/SD/SDIO controller is used as a connectivity interface
+with external MMC, SD and SDIO storage mediums.
+
+This file documents differences between the core mmc properties described by
+mmc.txt and the properties used by the Samsung S3C24XX MMC/SD/SDIO controller
+implementation.
+
+Required SoC Specific Properties:
+- compatible: should be one of the following
+ - "samsung,s3c2410-sdi": for controllers compatible with s3c2410
+ - "samsung,s3c2412-sdi": for controllers compatible with s3c2412
+ - "samsung,s3c2440-sdi": for controllers compatible with s3c2440
+- reg: register location and length
+- interrupts: mmc controller interrupt
+- clocks: Should reference the controller clock
+- clock-names: Should contain "sdi"
+
+Required Board Specific Properties:
+- pinctrl-0: Should specify pin control groups used for this controller.
+- pinctrl-names: Should contain only one value - "default".
+
+Optional Properties:
+- bus-width: number of data lines (see mmc.txt)
+- cd-gpios: gpio for card detection (see mmc.txt)
+- wp-gpios: gpio for write protection (see mmc.txt)
+
+Example:
+
+ mmc0: mmc@5a000000 {
+ compatible = "samsung,s3c2440-sdi";
+ pinctrl-names = "default";
+ pinctrl-0 = <&sdi_pins>;
+ reg = <0x5a000000 0x100000>;
+ interrupts = <0 0 21 3>;
+ clocks = <&clocks PCLK_SDI>;
+ clock-names = "sdi";
+ bus-width = <4>;
+ cd-gpios = <&gpg 8 GPIO_ACTIVE_LOW>;
+ wp-gpios = <&gph 8 GPIO_ACTIVE_LOW>;
+ };
diff --git a/Documentation/devicetree/bindings/mmc/sdhci-cadence.txt b/Documentation/devicetree/bindings/mmc/sdhci-cadence.txt
index c0f37cb..fa423c2 100644
--- a/Documentation/devicetree/bindings/mmc/sdhci-cadence.txt
+++ b/Documentation/devicetree/bindings/mmc/sdhci-cadence.txt
@@ -19,6 +19,53 @@
- mmc-hs400-1_8v
- mmc-hs400-1_2v
+Some PHY delays can be configured by following properties.
+PHY DLL input delays:
+They are used to delay the data valid window, and align the window
+to sampling clock. The delay starts from 5ns (for delay parameter equal to 0)
+and it is increased by 2.5ns in each step.
+- cdns,phy-input-delay-sd-highspeed:
+ Value of the delay in the input path for SD high-speed timing
+ Valid range = [0:0x1F].
+- cdns,phy-input-delay-legacy:
+ Value of the delay in the input path for legacy timing
+ Valid range = [0:0x1F].
+- cdns,phy-input-delay-sd-uhs-sdr12:
+ Value of the delay in the input path for SD UHS SDR12 timing
+ Valid range = [0:0x1F].
+- cdns,phy-input-delay-sd-uhs-sdr25:
+ Value of the delay in the input path for SD UHS SDR25 timing
+ Valid range = [0:0x1F].
+- cdns,phy-input-delay-sd-uhs-sdr50:
+ Value of the delay in the input path for SD UHS SDR50 timing
+ Valid range = [0:0x1F].
+- cdns,phy-input-delay-sd-uhs-ddr50:
+ Value of the delay in the input path for SD UHS DDR50 timing
+ Valid range = [0:0x1F].
+- cdns,phy-input-delay-mmc-highspeed:
+ Value of the delay in the input path for MMC high-speed timing
+ Valid range = [0:0x1F].
+- cdns,phy-input-delay-mmc-ddr:
+ Value of the delay in the input path for eMMC high-speed DDR timing
+ Valid range = [0:0x1F].
+
+PHY DLL clock delays:
+Each delay property represents the fraction of the clock period.
+The approximate delay value will be
+(<delay property value>/128)*sdmclk_clock_period.
+- cdns,phy-dll-delay-sdclk:
+ Value of the delay introduced on the sdclk output
+ for all modes except HS200, HS400 and HS400_ES.
+ Valid range = [0:0x7F].
+- cdns,phy-dll-delay-sdclk-hsmmc:
+ Value of the delay introduced on the sdclk output
+ for HS200, HS400 and HS400_ES speed modes.
+ Valid range = [0:0x7F].
+- cdns,phy-dll-delay-strobe:
+ Value of the delay introduced on the dat_strobe input
+ used in HS400 / HS400_ES speed modes.
+ Valid range = [0:0x7F].
+
Example:
emmc: sdhci@5a000000 {
compatible = "socionext,uniphier-sd4hc", "cdns,sd4hc";
@@ -29,4 +76,5 @@
mmc-ddr-1_8v;
mmc-hs200-1_8v;
mmc-hs400-1_8v;
+ cdns,phy-dll-delay-sdclk = <0>;
};
diff --git a/Documentation/devicetree/bindings/net/brcm,bcmgenet.txt b/Documentation/devicetree/bindings/net/brcm,bcmgenet.txt
index 10587bd..26c77d9 100644
--- a/Documentation/devicetree/bindings/net/brcm,bcmgenet.txt
+++ b/Documentation/devicetree/bindings/net/brcm,bcmgenet.txt
@@ -2,11 +2,14 @@
Required properties:
- compatible: should contain one of "brcm,genet-v1", "brcm,genet-v2",
- "brcm,genet-v3", "brcm,genet-v4".
+ "brcm,genet-v3", "brcm,genet-v4", "brcm,genet-v5".
- reg: address and length of the register set for the device
-- interrupts: must be two cells, the first cell is the general purpose
- interrupt line, while the second cell is the interrupt for the ring
- RX and TX queues operating in ring mode
+- interrupts and/or interrupts-extended: must be two cells, the first cell
+ is the general purpose interrupt line, while the second cell is the
+ interrupt for the ring RX and TX queues operating in ring mode. An
+ optional third interrupt cell for Wake-on-LAN can be specified.
+ See Documentation/devicetree/bindings/interrupt-controller/interrupts.txt
+ for information on the property specifics.
- phy-mode: see ethernet.txt file in the same directory
- #address-cells: should be 1
- #size-cells: should be 1
@@ -29,15 +32,15 @@
Required child nodes:
-- mdio bus node: this node should always be present regarless of the PHY
+- mdio bus node: this node should always be present regardless of the PHY
configuration of the GENET instance
MDIO bus node required properties:
- compatible: should contain one of "brcm,genet-mdio-v1", "brcm,genet-mdio-v2"
- "brcm,genet-mdio-v3", "brcm,genet-mdio-v4", the version has to match the
- parent node compatible property (e.g: brcm,genet-v4 pairs with
- brcm,genet-mdio-v4)
+ "brcm,genet-mdio-v3", "brcm,genet-mdio-v4", "brcm,genet-mdio-v5", the version
+ has to match the parent node compatible property (e.g: brcm,genet-v4 pairs
+ with brcm,genet-mdio-v4)
- reg: address and length relative to the parent node base register address
- #address-cells: address cell for MDIO bus addressing, should be 1
- #size-cells: size of the cells for MDIO bus addressing, should be 0
diff --git a/Documentation/devicetree/bindings/net/brcm,unimac-mdio.txt b/Documentation/devicetree/bindings/net/brcm,unimac-mdio.txt
index ab0bb42..4648948 100644
--- a/Documentation/devicetree/bindings/net/brcm,unimac-mdio.txt
+++ b/Documentation/devicetree/bindings/net/brcm,unimac-mdio.txt
@@ -2,8 +2,9 @@
Required properties:
- compatible: should one from "brcm,genet-mdio-v1", "brcm,genet-mdio-v2",
- "brcm,genet-mdio-v3", "brcm,genet-mdio-v4" or "brcm,unimac-mdio"
-- reg: address and length of the regsiter set for the device, first one is the
+ "brcm,genet-mdio-v3", "brcm,genet-mdio-v4", "brcm,genet-mdio-v5" or
+ "brcm,unimac-mdio"
+- reg: address and length of the register set for the device, first one is the
base register, and the second one is optional and for indirect accesses to
larger than 16-bits MDIO transactions
- reg-names: name(s) of the register must be "mdio" and optional "mdio_indir_rw"
diff --git a/Documentation/devicetree/bindings/net/can/holt_hi311x.txt b/Documentation/devicetree/bindings/net/can/holt_hi311x.txt
new file mode 100644
index 0000000..23aa94e
--- /dev/null
+++ b/Documentation/devicetree/bindings/net/can/holt_hi311x.txt
@@ -0,0 +1,24 @@
+* Holt HI-311X stand-alone CAN controller device tree bindings
+
+Required properties:
+ - compatible: Should be one of the following:
+ - "holt,hi3110" for HI-3110
+ - reg: SPI chip select.
+ - clocks: The clock feeding the CAN controller.
+ - interrupt-parent: The parent interrupt controller.
+ - interrupts: Should contain IRQ line for the CAN controller.
+
+Optional properties:
+ - vdd-supply: Regulator that powers the CAN controller.
+ - xceiver-supply: Regulator that powers the CAN transceiver.
+
+Example:
+ can0: can@1 {
+ compatible = "holt,hi3110";
+ reg = <1>;
+ clocks = <&clk32m>;
+ interrupt-parent = <&gpio4>;
+ interrupts = <13 IRQ_TYPE_EDGE_RISING>;
+ vdd-supply = <®5v0>;
+ xceiver-supply = <®5v0>;
+ };
diff --git a/Documentation/devicetree/bindings/net/can/ti_hecc.txt b/Documentation/devicetree/bindings/net/can/ti_hecc.txt
new file mode 100644
index 0000000..e0f0a7c
--- /dev/null
+++ b/Documentation/devicetree/bindings/net/can/ti_hecc.txt
@@ -0,0 +1,32 @@
+Texas Instruments High End CAN Controller (HECC)
+================================================
+
+This file provides information, what the device node
+for the hecc interface contains.
+
+Required properties:
+- compatible: "ti,am3517-hecc"
+- reg: addresses and lengths of the register spaces for 'hecc', 'hecc-ram'
+ and 'mbx'
+- reg-names :"hecc", "hecc-ram", "mbx"
+- interrupts: interrupt mapping for the hecc interrupts sources
+- clocks: clock phandles (see clock bindings for details)
+
+Optional properties:
+- ti,use-hecc1int: if provided configures HECC to produce all interrupts
+ on HECC1INT interrupt line. By default HECC0INT interrupt
+ line will be used.
+- xceiver-supply: regulator that powers the CAN transceiver
+
+Example:
+
+For am3517evm board:
+ hecc: can@5c050000 {
+ compatible = "ti,am3517-hecc";
+ reg = <0x5c050000 0x80>,
+ <0x5c053000 0x180>,
+ <0x5c052000 0x200>;
+ reg-names = "hecc", "hecc-ram", "mbx";
+ interrupts = <24>;
+ clocks = <&hecc_ck>;
+ };
diff --git a/Documentation/devicetree/bindings/net/dsa/lan9303.txt b/Documentation/devicetree/bindings/net/dsa/lan9303.txt
new file mode 100644
index 0000000..04f2965
--- /dev/null
+++ b/Documentation/devicetree/bindings/net/dsa/lan9303.txt
@@ -0,0 +1,105 @@
+SMSC/MicroChip LAN9303 three port ethernet switch
+-------------------------------------------------
+
+Required properties:
+
+- compatible: should be
+ - "smsc,lan9303-i2c" for I2C managed mode
+ or
+ - "smsc,lan9303-mdio" for mdio managed mode
+
+Optional properties:
+
+- reset-gpios: GPIO to be used to reset the whole device
+- reset-duration: reset duration in milliseconds, defaults to 200 ms
+
+Subnodes:
+
+The integrated switch subnode should be specified according to the binding
+described in dsa/dsa.txt. The CPU port of this switch is always port 0.
+
+Note: always use 'reg = <0/1/2>;' for the three DSA ports, even if the device is
+configured to use 1/2/3 instead. This hardware configuration will be
+auto-detected and mapped accordingly.
+
+Example:
+
+I2C managed mode:
+
+ master: masterdevice@X {
+ status = "okay";
+
+ fixed-link { /* RMII fixed link to LAN9303 */
+ speed = <100>;
+ full-duplex;
+ };
+ };
+
+ switch: switch@a {
+ compatible = "smsc,lan9303-i2c";
+ reg = <0xa>;
+ status = "okay";
+ reset-gpios = <&gpio7 6 GPIO_ACTIVE_LOW>;
+ reset-duration = <200>;
+
+ ports {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ port@0 { /* RMII fixed link to master */
+ reg = <0>;
+ label = "cpu";
+ ethernet = <&master>;
+ };
+
+ port@1 { /* external port 1 */
+ reg = <1>;
+ label = "lan1;
+ };
+
+ port@2 { /* external port 2 */
+ reg = <2>;
+ label = "lan2";
+ };
+ };
+ };
+
+MDIO managed mode:
+
+ master: masterdevice@X {
+ status = "okay";
+ phy-handle = <&switch>;
+
+ mdio {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ switch: switch-phy@0 {
+ compatible = "smsc,lan9303-mdio";
+ reg = <0>;
+ reset-gpios = <&gpio7 6 GPIO_ACTIVE_LOW>;
+ reset-duration = <100>;
+
+ ports {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ port@0 {
+ reg = <0>;
+ label = "cpu";
+ ethernet = <&master>;
+ };
+
+ port@1 { /* external port 1 */
+ reg = <1>;
+ label = "lan1;
+ };
+
+ port@2 { /* external port 2 */
+ reg = <2>;
+ label = "lan2";
+ };
+ };
+ };
+ };
+ };
diff --git a/Documentation/devicetree/bindings/net/dsa/mt7530.txt b/Documentation/devicetree/bindings/net/dsa/mt7530.txt
new file mode 100644
index 0000000..a9bc27b
--- /dev/null
+++ b/Documentation/devicetree/bindings/net/dsa/mt7530.txt
@@ -0,0 +1,92 @@
+Mediatek MT7530 Ethernet switch
+================================
+
+Required properties:
+
+- compatible: Must be compatible = "mediatek,mt7530";
+- #address-cells: Must be 1.
+- #size-cells: Must be 0.
+- mediatek,mcm: Boolean; if defined, indicates that either MT7530 is the part
+ on multi-chip module belong to MT7623A has or the remotely standalone
+ chip as the function MT7623N reference board provided for.
+- core-supply: Phandle to the regulator node necessary for the core power.
+- io-supply: Phandle to the regulator node necessary for the I/O power.
+ See Documentation/devicetree/bindings/regulator/mt6323-regulator.txt
+ for details for the regulator setup on these boards.
+
+If the property mediatek,mcm isn't defined, following property is required
+
+- reset-gpios: Should be a gpio specifier for a reset line.
+
+Else, following properties are required
+
+- resets : Phandle pointing to the system reset controller with
+ line index for the ethsys.
+- reset-names : Should be set to "mcm".
+
+Required properties for the child nodes within ports container:
+
+- reg: Port address described must be 6 for CPU port and from 0 to 5 for
+ user ports.
+- phy-mode: String, must be either "trgmii" or "rgmii" for port labeled
+ "cpu".
+
+See Documentation/devicetree/bindings/dsa/dsa.txt for a list of additional
+required, optional properties and how the integrated switch subnodes must
+be specified.
+
+Example:
+
+ &mdio0 {
+ switch@0 {
+ compatible = "mediatek,mt7530";
+ #address-cells = <1>;
+ #size-cells = <0>;
+ reg = <0>;
+
+ core-supply = <&mt6323_vpa_reg>;
+ io-supply = <&mt6323_vemc3v3_reg>;
+ reset-gpios = <&pio 33 0>;
+
+ ports {
+ #address-cells = <1>;
+ #size-cells = <0>;
+ reg = <0>;
+ port@0 {
+ reg = <0>;
+ label = "lan0";
+ };
+
+ port@1 {
+ reg = <1>;
+ label = "lan1";
+ };
+
+ port@2 {
+ reg = <2>;
+ label = "lan2";
+ };
+
+ port@3 {
+ reg = <3>;
+ label = "lan3";
+ };
+
+ port@4 {
+ reg = <4>;
+ label = "wan";
+ };
+
+ port@6 {
+ reg = <6>;
+ label = "cpu";
+ ethernet = <&gmac0>;
+ phy-mode = "trgmii";
+ fixed-link {
+ speed = <1000>;
+ full-duplex;
+ };
+ };
+ };
+ };
+ };
diff --git a/Documentation/devicetree/bindings/net/ftgmac100.txt b/Documentation/devicetree/bindings/net/ftgmac100.txt
new file mode 100644
index 0000000..c1ce168
--- /dev/null
+++ b/Documentation/devicetree/bindings/net/ftgmac100.txt
@@ -0,0 +1,35 @@
+* Faraday Technology FTGMAC100 gigabit ethernet controller
+
+Required properties:
+- compatible: "faraday,ftgmac100"
+
+ Must also contain one of these if used as part of an Aspeed AST2400
+ or 2500 family SoC as they have some subtle tweaks to the
+ implementation:
+
+ - "aspeed,ast2400-mac"
+ - "aspeed,ast2500-mac"
+
+- reg: Address and length of the register set for the device
+- interrupts: Should contain ethernet controller interrupt
+
+Optional properties:
+- phy-mode: See ethernet.txt file in the same directory. If the property is
+ absent, "rgmii" is assumed. Supported values are "rgmii*" and "rmii" for
+ aspeed parts. Other (unknown) parts will accept any value.
+- use-ncsi: Use the NC-SI stack instead of an MDIO PHY. Currently assumes
+ rmii (100bT) but kept as a separate property in case NC-SI grows support
+ for a gigabit link.
+- no-hw-checksum: Used to disable HW checksum support. Here for backward
+ compatibility as the driver now should have correct defaults based on
+ the SoC.
+
+Example:
+
+ mac0: ethernet@1e660000 {
+ compatible = "aspeed,ast2500-mac", "faraday,ftgmac100";
+ reg = <0x1e660000 0x180>;
+ interrupts = <2>;
+ status = "okay";
+ use-ncsi;
+ };
diff --git a/Documentation/devicetree/bindings/net/ieee802154/ca8210.txt b/Documentation/devicetree/bindings/net/ieee802154/ca8210.txt
new file mode 100644
index 0000000..a1046e6
--- /dev/null
+++ b/Documentation/devicetree/bindings/net/ieee802154/ca8210.txt
@@ -0,0 +1,28 @@
+* CA8210 IEEE 802.15.4 *
+
+Required properties:
+ - compatible: Should be "cascoda,ca8210"
+ - reg: Controlling chip select
+ - spi-max-frequency: Maximum clock speed, should be *less than*
+ 4000000
+ - spi-cpol: Requires inverted clock polarity
+ - reset-gpio: GPIO attached to reset
+ - irq-gpio: GPIO attached to IRQ
+Optional properties:
+ - extclock-enable: Include for the ca8210 to route its 16MHz clock
+ to an output
+ - extclock-freq: Frequency in Hz of the external clock
+ - extclock-gpio: GPIO of the ca8210 to output the clock on
+
+Example:
+ ca8210@0 {
+ compatible = "cascoda,ca8210";
+ reg = <0>;
+ spi-max-frequency = <3000000>;
+ spi-cpol;
+ reset-gpio = <&gpio1 1 GPIO_ACTIVE_HIGH>;
+ irq-gpio = <&gpio1 2 GPIO_ACTIVE_HIGH>;
+ extclock-enable;
+ extclock-freq = 16000000;
+ extclock-gpio = 2;
+ };
diff --git a/Documentation/devicetree/bindings/net/marvell-orion-mdio.txt b/Documentation/devicetree/bindings/net/marvell-orion-mdio.txt
index 9417e54..ccdabdc 100644
--- a/Documentation/devicetree/bindings/net/marvell-orion-mdio.txt
+++ b/Documentation/devicetree/bindings/net/marvell-orion-mdio.txt
@@ -7,17 +7,20 @@
Required properties:
- compatible: "marvell,orion-mdio"
-- reg: address and length of the SMI register
+- reg: address and length of the MDIO registers. When an interrupt is
+ not present, the length is the size of the SMI register (4 bytes)
+ otherwise it must be 0x84 bytes to cover the interrupt control
+ registers.
Optional properties:
- interrupts: interrupt line number for the SMI error/done interrupt
-- clocks: Phandle to the clock control device and gate bit
+- clocks: phandle for up to three required clocks for the MDIO instance
The child nodes of the MDIO driver are the individual PHY devices
connected to this MDIO bus. They must have a "reg" property given the
PHY address on the MDIO bus.
-Example at the SoC level:
+Example at the SoC level without an interrupt property:
mdio {
#address-cells = <1>;
@@ -26,6 +29,16 @@
reg = <0xd0072004 0x4>;
};
+Example with an interrupt property:
+
+mdio {
+ #address-cells = <1>;
+ #size-cells = <0>;
+ compatible = "marvell,orion-mdio";
+ reg = <0xd0072004 0x84>;
+ interrupts = <30>;
+};
+
And at the board level:
mdio {
diff --git a/Documentation/devicetree/bindings/net/marvell-pp2.txt b/Documentation/devicetree/bindings/net/marvell-pp2.txt
index 4754364..6b4956b 100644
--- a/Documentation/devicetree/bindings/net/marvell-pp2.txt
+++ b/Documentation/devicetree/bindings/net/marvell-pp2.txt
@@ -1,17 +1,28 @@
-* Marvell Armada 375 Ethernet Controller (PPv2)
+* Marvell Armada 375 Ethernet Controller (PPv2.1)
+ Marvell Armada 7K/8K Ethernet Controller (PPv2.2)
Required properties:
-- compatible: should be "marvell,armada-375-pp2"
+- compatible: should be one of:
+ "marvell,armada-375-pp2"
+ "marvell,armada-7k-pp2"
- reg: addresses and length of the register sets for the device.
- Must contain the following register sets:
+ For "marvell,armada-375-pp2", must contain the following register
+ sets:
- common controller registers
- LMS registers
- In addition, at least one port register set is required.
-- clocks: a pointer to the reference clocks for this device, consequently:
- - main controller clock
- - GOP clock
-- clock-names: names of used clocks, must be "pp_clk" and "gop_clk".
+ - one register area per Ethernet port
+ For "marvell,armada-7k-pp2", must contain the following register
+ sets:
+ - packet processor registers
+ - networking interfaces registers
+
+- clocks: pointers to the reference clocks for this device, consequently:
+ - main controller clock (for both armada-375-pp2 and armada-7k-pp2)
+ - GOP clock (for both armada-375-pp2 and armada-7k-pp2)
+ - MG clock (only for armada-7k-pp2)
+- clock-names: names of used clocks, must be "pp_clk", "gop_clk" and
+ "mg_clk" (the latter only for armada-7k-pp2).
The ethernet ports are represented by subnodes. At least one port is
required.
@@ -19,8 +30,10 @@
Required properties (port):
- interrupts: interrupt for the port
-- port-id: should be '0' or '1' for ethernet ports, and '2' for the
- loopback port
+- port-id: ID of the port from the MAC point of view
+- gop-port-id: only for marvell,armada-7k-pp2, ID of the port from the
+ GOP (Group Of Ports) point of view. This ID is used to index the
+ per-port registers in the second register area.
- phy-mode: See ethernet.txt file in the same directory
Optional properties (port):
@@ -29,7 +42,7 @@
- phy: a phandle to a phy node defining the PHY address (as the reg
property, a single integer).
-Example:
+Example for marvell,armada-375-pp2:
ethernet@f0000 {
compatible = "marvell,armada-375-pp2";
@@ -57,3 +70,30 @@
phy-mode = "gmii";
};
};
+
+Example for marvell,armada-7k-pp2:
+
+cpm_ethernet: ethernet@0 {
+ compatible = "marvell,armada-7k-pp22";
+ reg = <0x0 0x100000>, <0x129000 0xb000>;
+ clocks = <&cpm_syscon0 1 3>, <&cpm_syscon0 1 9>, <&cpm_syscon0 1 5>;
+ clock-names = "pp_clk", "gop_clk", "gp_clk";
+
+ eth0: eth0 {
+ interrupts = <GIC_SPI 37 IRQ_TYPE_LEVEL_HIGH>;
+ port-id = <0>;
+ gop-port-id = <0>;
+ };
+
+ eth1: eth1 {
+ interrupts = <GIC_SPI 38 IRQ_TYPE_LEVEL_HIGH>;
+ port-id = <1>;
+ gop-port-id = <2>;
+ };
+
+ eth2: eth2 {
+ interrupts = <GIC_SPI 39 IRQ_TYPE_LEVEL_HIGH>;
+ port-id = <2>;
+ gop-port-id = <3>;
+ };
+};
diff --git a/Documentation/devicetree/bindings/net/mdio.txt b/Documentation/devicetree/bindings/net/mdio.txt
new file mode 100644
index 0000000..96a53f8
--- /dev/null
+++ b/Documentation/devicetree/bindings/net/mdio.txt
@@ -0,0 +1,37 @@
+Common MDIO bus properties.
+
+These are generic properties that can apply to any MDIO bus.
+
+Optional properties:
+- reset-gpios: One GPIO that control the RESET lines of all PHYs on that MDIO
+ bus.
+- reset-delay-us: RESET pulse width in microseconds.
+
+A list of child nodes, one per device on the bus is expected. These
+should follow the generic phy.txt, or a device specific binding document.
+
+The 'reset-delay-us' indicates the RESET signal pulse width in microseconds and
+applies to all PHY devices. It must therefore be appropriately determined based
+on all PHY requirements (maximum value of all per-PHY RESET pulse widths).
+
+Example :
+This example shows these optional properties, plus other properties
+required for the TI Davinci MDIO driver.
+
+ davinci_mdio: ethernet@0x5c030000 {
+ compatible = "ti,davinci_mdio";
+ reg = <0x5c030000 0x1000>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ reset-gpios = <&gpio2 5 GPIO_ACTIVE_LOW>;
+ reset-delay-us = <2>;
+
+ ethphy0: ethernet-phy@1 {
+ reg = <1>;
+ };
+
+ ethphy1: ethernet-phy@3 {
+ reg = <3>;
+ };
+ };
diff --git a/Documentation/devicetree/bindings/net/nfc/trf7970a.txt b/Documentation/devicetree/bindings/net/nfc/trf7970a.txt
index 32b35a0..c627bbb 100644
--- a/Documentation/devicetree/bindings/net/nfc/trf7970a.txt
+++ b/Documentation/devicetree/bindings/net/nfc/trf7970a.txt
@@ -5,8 +5,8 @@
- spi-max-frequency: Maximum SPI frequency (<= 2000000).
- interrupt-parent: phandle of parent interrupt handler.
- interrupts: A single interrupt specifier.
-- ti,enable-gpios: Two GPIO entries used for 'EN' and 'EN2' pins on the
- TRF7970A.
+- ti,enable-gpios: One or two GPIO entries used for 'EN' and 'EN2' pins on the
+ TRF7970A. EN2 is optional.
- vin-supply: Regulator for supply voltage to VIN pin
Optional SoC Specific Properties:
@@ -21,6 +21,8 @@
- t5t-rmb-extra-byte-quirk: Specify that the trf7970a has the erratum
where an extra byte is returned by Read Multiple Block commands issued
to Type 5 tags.
+- vdd-io-supply: Regulator specifying voltage for vdd-io
+- clock-frequency: Set to specify that the input frequency to the trf7970a is 13560000Hz or 27120000Hz
Example (for ARM-based BeagleBone with TRF7970A on SPI1):
@@ -39,10 +41,12 @@
<&gpio2 5 GPIO_ACTIVE_LOW>;
vin-supply = <&ldo3_reg>;
vin-voltage-override = <5000000>;
+ vdd-io-supply = <&ldo2_reg>;
autosuspend-delay = <30000>;
irq-status-read-quirk;
en2-rf-quirk;
t5t-rmb-extra-byte-quirk;
+ clock-frequency = <27120000>;
status = "okay";
};
};
diff --git a/Documentation/devicetree/bindings/net/nokia-bluetooth.txt b/Documentation/devicetree/bindings/net/nokia-bluetooth.txt
new file mode 100644
index 0000000..42be7dc
--- /dev/null
+++ b/Documentation/devicetree/bindings/net/nokia-bluetooth.txt
@@ -0,0 +1,51 @@
+Nokia Bluetooth Chips
+---------------------
+
+Nokia phones often come with UART connected bluetooth chips from different
+vendors and modified device API. Those devices speak a protocol named H4+
+(also known as h4p) by Nokia, which is similar to the H4 protocol from the
+Bluetooth standard. In addition to the H4 protocol it specifies two more
+UART status lines for wakeup of UART transceivers to improve power management
+and a few new packet types used to negotiate uart speed.
+
+Required properties:
+
+ - compatible: should contain "nokia,h4p-bluetooth" as well as one of the following:
+ * "brcm,bcm2048-nokia"
+ * "ti,wl1271-bluetooth-nokia"
+ - reset-gpios: GPIO specifier, used to reset the BT module (active low)
+ - bluetooth-wakeup-gpios: GPIO specifier, used to wakeup the BT module (active high)
+ - host-wakeup-gpios: GPIO specifier, used to wakeup the host processor (active high)
+ - clock-names: should be "sysclk"
+ - clocks: should contain a clock specifier for every name in clock-names
+
+Optional properties:
+
+ - None
+
+Example:
+
+/ {
+ /* controlled (enabled/disabled) directly by BT module */
+ bluetooth_clk: vctcxo {
+ compatible = "fixed-clock";
+ #clock-cells = <0>;
+ clock-frequency = <38400000>;
+ };
+};
+
+&uart2 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&uart2_pins>;
+
+ bluetooth {
+ compatible = "ti,wl1271-bluetooth-nokia", "nokia,h4p-bluetooth";
+
+ reset-gpios = <&gpio1 26 GPIO_ACTIVE_LOW>; /* gpio26 */
+ host-wakeup-gpios = <&gpio4 5 GPIO_ACTIVE_HIGH>; /* gpio101 */
+ bluetooth-wakeup-gpios = <&gpio2 5 GPIO_ACTIVE_HIGH>; /* gpio37 */
+
+ clocks = <&bluetooth_clk>;
+ clock-names = "sysclk";
+ };
+};
diff --git a/Documentation/devicetree/bindings/net/stmmac.txt b/Documentation/devicetree/bindings/net/stmmac.txt
index d3bfc2b..c3a7be6 100644
--- a/Documentation/devicetree/bindings/net/stmmac.txt
+++ b/Documentation/devicetree/bindings/net/stmmac.txt
@@ -7,9 +7,12 @@
- interrupt-parent: Should be the phandle for the interrupt controller
that services interrupts for this device
- interrupts: Should contain the STMMAC interrupts
-- interrupt-names: Should contain the interrupt names "macirq"
- "eth_wake_irq" if this interrupt is supported in the "interrupts"
- property
+- interrupt-names: Should contain a list of interrupt names corresponding to
+ the interrupts in the interrupts property, if available.
+ Valid interrupt names are:
+ - "macirq" (combined signal for various interrupt events)
+ - "eth_wake_irq" (the interrupt to manage the remote wake-up packet detection)
+ - "eth_lpi" (the interrupt that occurs when Tx or Rx enters/exits LPI state)
- phy-mode: See ethernet.txt file in the same directory.
- snps,reset-gpio gpio number for phy reset.
- snps,reset-active-low boolean flag to indicate if phy reset is active low.
@@ -28,9 +31,9 @@
clocks may be specified in derived bindings.
- clock-names: One name for each entry in the clocks property, the
first one should be "stmmaceth" and the second one should be "pclk".
-- clk_ptp_ref: this is the PTP reference clock; in case of the PTP is
- available this clock is used for programming the Timestamp Addend Register.
- If not passed then the system clock will be used and this is fine on some
+- ptp_ref: this is the PTP reference clock; in case of the PTP is available
+ this clock is used for programming the Timestamp Addend Register. If not
+ passed then the system clock will be used and this is fine on some
platforms.
- tx-fifo-depth: See ethernet.txt file in the same directory
- rx-fifo-depth: See ethernet.txt file in the same directory
@@ -72,7 +75,45 @@
- snps,mb: mixed-burst
- snps,rb: rebuild INCRx Burst
- mdio: with compatible = "snps,dwmac-mdio", create and register mdio bus.
-
+- Multiple RX Queues parameters: below the list of all the parameters to
+ configure the multiple RX queues:
+ - snps,rx-queues-to-use: number of RX queues to be used in the driver
+ - Choose one of these RX scheduling algorithms:
+ - snps,rx-sched-sp: Strict priority
+ - snps,rx-sched-wsp: Weighted Strict priority
+ - For each RX queue
+ - Choose one of these modes:
+ - snps,dcb-algorithm: Queue to be enabled as DCB
+ - snps,avb-algorithm: Queue to be enabled as AVB
+ - snps,map-to-dma-channel: Channel to map
+ - Specifiy specific packet routing:
+ - snps,route-avcp: AV Untagged Control packets
+ - snps,route-ptp: PTP Packets
+ - snps,route-dcbcp: DCB Control Packets
+ - snps,route-up: Untagged Packets
+ - snps,route-multi-broad: Multicast & Broadcast Packets
+ - snps,priority: RX queue priority (Range: 0x0 to 0xF)
+- Multiple TX Queues parameters: below the list of all the parameters to
+ configure the multiple TX queues:
+ - snps,tx-queues-to-use: number of TX queues to be used in the driver
+ - Choose one of these TX scheduling algorithms:
+ - snps,tx-sched-wrr: Weighted Round Robin
+ - snps,tx-sched-wfq: Weighted Fair Queuing
+ - snps,tx-sched-dwrr: Deficit Weighted Round Robin
+ - snps,tx-sched-sp: Strict priority
+ - For each TX queue
+ - snps,weight: TX queue weight (if using a DCB weight algorithm)
+ - Choose one of these modes:
+ - snps,dcb-algorithm: TX queue will be working in DCB
+ - snps,avb-algorithm: TX queue will be working in AVB
+ [Attention] Queue 0 is reserved for legacy traffic
+ and so no AVB is available in this queue.
+ - Configure Credit Base Shaper (if AVB Mode selected):
+ - snps,send_slope: enable Low Power Interface
+ - snps,idle_slope: unlock on WoL
+ - snps,high_credit: max write outstanding req. limit
+ - snps,low_credit: max read outstanding req. limit
+ - snps,priority: TX queue priority (Range: 0x0 to 0xF)
Examples:
stmmac_axi_setup: stmmac-axi-config {
@@ -81,12 +122,41 @@
snps,blen = <256 128 64 32 0 0 0>;
};
+ mtl_rx_setup: rx-queues-config {
+ snps,rx-queues-to-use = <1>;
+ snps,rx-sched-sp;
+ queue0 {
+ snps,dcb-algorithm;
+ snps,map-to-dma-channel = <0x0>;
+ snps,priority = <0x0>;
+ };
+ };
+
+ mtl_tx_setup: tx-queues-config {
+ snps,tx-queues-to-use = <2>;
+ snps,tx-sched-wrr;
+ queue0 {
+ snps,weight = <0x10>;
+ snps,dcb-algorithm;
+ snps,priority = <0x0>;
+ };
+
+ queue1 {
+ snps,avb-algorithm;
+ snps,send_slope = <0x1000>;
+ snps,idle_slope = <0x1000>;
+ snps,high_credit = <0x3E800>;
+ snps,low_credit = <0xFFC18000>;
+ snps,priority = <0x1>;
+ };
+ };
+
gmac0: ethernet@e0800000 {
compatible = "st,spear600-gmac";
reg = <0xe0800000 0x8000>;
interrupt-parent = <&vic1>;
- interrupts = <24 23>;
- interrupt-names = "macirq", "eth_wake_irq";
+ interrupts = <24 23 22>;
+ interrupt-names = "macirq", "eth_wake_irq", "eth_lpi";
mac-address = [000000000000]; /* Filled in by U-Boot */
max-frame-size = <3800>;
phy-mode = "gmii";
@@ -104,4 +174,6 @@
phy1: ethernet-phy@0 {
};
};
+ snps,mtl-rx-config = <&mtl_rx_setup>;
+ snps,mtl-tx-config = <&mtl_tx_setup>;
};
diff --git a/Documentation/devicetree/bindings/net/ti,wilink-st.txt b/Documentation/devicetree/bindings/net/ti,wilink-st.txt
new file mode 100644
index 0000000..cbad73a
--- /dev/null
+++ b/Documentation/devicetree/bindings/net/ti,wilink-st.txt
@@ -0,0 +1,35 @@
+TI WiLink 7/8 (wl12xx/wl18xx) Shared Transport BT/FM/GPS devices
+
+TI WiLink devices have a UART interface for providing Bluetooth, FM radio,
+and GPS over what's called "shared transport". The shared transport is
+standard BT HCI protocol with additional channels for the other functions.
+
+These devices also have a separate WiFi interface as described in
+wireless/ti,wlcore.txt.
+
+This bindings follows the UART slave device binding in
+../serial/slave-device.txt.
+
+Required properties:
+ - compatible: should be one of the following:
+ "ti,wl1271-st"
+ "ti,wl1273-st"
+ "ti,wl1831-st"
+ "ti,wl1835-st"
+ "ti,wl1837-st"
+
+Optional properties:
+ - enable-gpios : GPIO signal controlling enabling of BT. Active high.
+ - vio-supply : Vio input supply (1.8V)
+ - vbat-supply : Vbat input supply (2.9-4.8V)
+
+Example:
+
+&serial0 {
+ compatible = "ns16550a";
+ ...
+ bluetooth {
+ compatible = "ti,wl1835-st";
+ enable-gpios = <&gpio1 7 GPIO_ACTIVE_HIGH>;
+ };
+};
diff --git a/Documentation/devicetree/bindings/pinctrl/allwinner,sunxi-pinctrl.txt b/Documentation/devicetree/bindings/pinctrl/allwinner,sunxi-pinctrl.txt
index 2fd688c..b532244 100644
--- a/Documentation/devicetree/bindings/pinctrl/allwinner,sunxi-pinctrl.txt
+++ b/Documentation/devicetree/bindings/pinctrl/allwinner,sunxi-pinctrl.txt
@@ -23,7 +23,8 @@
"allwinner,sun8i-h3-pinctrl"
"allwinner,sun8i-h3-r-pinctrl"
"allwinner,sun50i-a64-pinctrl"
- "allwinner,sun50i-h5-r-pinctrl"
+ "allwinner,sun50i-a64-r-pinctrl"
+ "allwinner,sun50i-h5-pinctrl"
"nextthing,gr8-pinctrl"
- reg: Should contain the register physical address and length for the
diff --git a/Documentation/devicetree/bindings/pinctrl/atmel,at91-pinctrl.txt b/Documentation/devicetree/bindings/pinctrl/atmel,at91-pinctrl.txt
index 9a8a45d..590e603 100644
--- a/Documentation/devicetree/bindings/pinctrl/atmel,at91-pinctrl.txt
+++ b/Documentation/devicetree/bindings/pinctrl/atmel,at91-pinctrl.txt
@@ -4,7 +4,7 @@
to share one PAD to several functional blocks. The sharing is done by
multiplexing the PAD input/output signals. For each PAD there are up to
8 muxing options (called periph modes). Since different modules require
-different PAD settings (like pull up, keeper, etc) the contoller controls
+different PAD settings (like pull up, keeper, etc) the controller controls
also the PAD settings parameters.
Please refer to pinctrl-bindings.txt in this directory for details of the
diff --git a/Documentation/devicetree/bindings/pinctrl/axis,artpec6-pinctrl.txt b/Documentation/devicetree/bindings/pinctrl/axis,artpec6-pinctrl.txt
new file mode 100644
index 0000000..47284f8
--- /dev/null
+++ b/Documentation/devicetree/bindings/pinctrl/axis,artpec6-pinctrl.txt
@@ -0,0 +1,85 @@
+Axis ARTPEC-6 Pin Controller
+
+Required properties:
+- compatible: "axis,artpec6-pinctrl".
+- reg: Should contain the register physical address and length for the pin
+ controller.
+
+A pinctrl node should contain at least one subnode representing the pinctrl
+groups available on the machine. Each subnode will list the mux function
+required and what pin group it will use. Each subnode will also configure the
+drive strength and bias pullup of the pin group. If either of these options is
+not set, its actual value will be unspecified.
+
+
+Required subnode-properties:
+- function: Function to mux.
+- groups: Name of the pin group to use for the function above.
+
+ Available functions and groups (function: group0, group1...):
+ gpio: cpuclkoutgrp0, udlclkoutgrp0, i2c1grp0, i2c2grp0,
+ i2c3grp0, i2s0grp0, i2s1grp0, i2srefclkgrp0, spi0grp0,
+ spi1grp0, pciedebuggrp0, uart0grp0, uart0grp1, uart1grp0,
+ uart2grp0, uart2grp1, uart3grp0, uart4grp0, uart5grp0
+ cpuclkout: cpuclkoutgrp0
+ udlclkout: udlclkoutgrp0
+ i2c1: i2c1grp0
+ i2c2: i2c2grp0
+ i2c3: i2c3grp0
+ i2s0: i2s0grp0
+ i2s1: i2s1grp0
+ i2srefclk: i2srefclkgrp0
+ spi0: spi0grp0
+ spi1: spi1grp0
+ pciedebug: pciedebuggrp0
+ uart0: uart0grp0, uart0grp1
+ uart1: uart1grp0
+ uart2: uart2grp0, uart2grp1
+ uart3: uart3grp0
+ uart4: uart4grp0
+ uart5: uart5grp0
+ nand: nandgrp0
+ sdio0: sdio0grp0
+ sdio1: sdio1grp0
+ ethernet: ethernetgrp0
+
+
+Optional subnode-properties (see pinctrl-bindings.txt):
+- drive-strength: 4, 6, 8, 9 mA. For SD and NAND pins, this is for 3.3V VCCQ3.
+- bias-pull-up
+- bias-disable
+
+Examples:
+pinctrl@f801d000 {
+ compatible = "axis,artpec6-pinctrl";
+ reg = <0xf801d000 0x400>;
+
+ pinctrl_uart0: uart0grp {
+ function = "uart0";
+ groups = "uart0grp0";
+ drive-strength = <4>;
+ bias-pull-up;
+ };
+ pinctrl_uart3: uart3grp {
+ function = "uart3";
+ groups = "uart3grp0";
+ };
+};
+uart0: uart@f8036000 {
+ compatible = "arm,pl011", "arm,primecell";
+ reg = <0xf8036000 0x1000>;
+ interrupts = <0 104 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&pll2div24>, <&apb_pclk>;
+ clock-names = "uart_clk", "apb_pclk";
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_uart0>;
+};
+uart3: uart@f8039000 {
+ compatible = "arm,pl011", "arm,primecell";
+ reg = <0xf8039000 0x1000>;
+ interrupts = <0 128 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&pll2div24>, <&apb_pclk>;
+ clock-names = "uart_clk", "apb_pclk";
+ pinctrl-names = "default";
+ pinctrl-0 = <&pinctrl_uart3>;
+};
diff --git a/Documentation/devicetree/bindings/pinctrl/marvell,armada-37xx-pinctrl.txt b/Documentation/devicetree/bindings/pinctrl/marvell,armada-37xx-pinctrl.txt
new file mode 100644
index 0000000..f640609
--- /dev/null
+++ b/Documentation/devicetree/bindings/pinctrl/marvell,armada-37xx-pinctrl.txt
@@ -0,0 +1,183 @@
+* Marvell Armada 37xx SoC pin and gpio controller
+
+Each Armada 37xx SoC come with two pin and gpio controller one for the
+south bridge and the other for the north bridge.
+
+Inside this set of register the gpio latch allows exposing some
+configuration of the SoC and especially the clock frequency of the
+xtal. Hence, this node is a represent as syscon allowing sharing the
+register between multiple hardware block.
+
+GPIO and pin controller:
+------------------------
+
+Main node:
+
+Refer to pinctrl-bindings.txt in this directory for details of the
+common pinctrl bindings used by client devices, including the meaning
+of the phrase "pin configuration node".
+
+Required properties for pinctrl driver:
+
+- compatible: "marvell,armada3710-sb-pinctrl", "syscon, "simple-mfd"
+ for the south bridge
+ "marvell,armada3710-nb-pinctrl", "syscon, "simple-mfd"
+ for the north bridge
+- reg: The first set of register are for pinctrl/gpio and the second
+ set for the interrupt controller
+- interrupts: list of the interrupt use by the gpio
+
+Available groups and functions for the North bridge:
+
+group: jtag
+ - pins 20-24
+ - functions jtag, gpio
+
+group sdio0
+ - pins 8-10
+ - functions sdio, gpio
+
+group emmc_nb
+ - pins 27-35
+ - functions emmc, gpio
+
+group pwm0
+ - pin 11 (GPIO1-11)
+ - functions pwm, gpio
+
+group pwm1
+ - pin 12
+ - functions pwm, gpio
+
+group pwm2
+ - pin 13
+ - functions pwm, gpio
+
+group pwm3
+ - pin 14
+ - functions pwm, gpio
+
+group pmic1
+ - pin 17
+ - functions pmic, gpio
+
+group pmic0
+ - pin 16
+ - functions pmic, gpio
+
+group i2c2
+ - pins 2-3
+ - functions i2c, gpio
+
+group i2c1
+ - pins 0-1
+ - functions i2c, gpio
+
+group spi_cs1
+ - pin 17
+ - functions spi, gpio
+
+group spi_cs2
+ - pin 18
+ - functions spi, gpio
+
+group spi_cs3
+ - pin 19
+ - functions spi, gpio
+
+group onewire
+ - pin 4
+ - functions onewire, gpio
+
+group uart1
+ - pins 25-26
+ - functions uart, gpio
+
+group spi_quad
+ - pins 15-16
+ - functions spi, gpio
+
+group uart_2
+ - pins 9-10
+ - functions uart, gpio
+
+Available groups and functions for the South bridge:
+
+group usb32_drvvbus0
+ - pin 36
+ - functions drvbus, gpio
+
+group usb2_drvvbus1
+ - pin 37
+ - functions drvbus, gpio
+
+group sdio_sb
+ - pins 60-64
+ - functions sdio, gpio
+
+group rgmii
+ - pins 42-55
+ - functions mii, gpio
+
+group pcie1
+ - pins 39-40
+ - functions pcie, gpio
+
+group ptp
+ - pins 56-58
+ - functions ptp, gpio
+
+group ptp_clk
+ - pin 57
+ - functions ptp, mii
+
+group ptp_trig
+ - pin 58
+ - functions ptp, mii
+
+group mii_col
+ - pin 59
+ - functions mii, mii_err
+
+GPIO subnode:
+
+Please refer to gpio.txt in this directory for details of gpio-ranges property
+and the common GPIO bindings used by client devices.
+
+Required properties for gpio driver under the gpio subnode:
+- interrupts: List of interrupt specifier for the controllers interrupt.
+- gpio-controller: Marks the device node as a gpio controller.
+- #gpio-cells: Should be 2. The first cell is the GPIO number and the
+ second cell specifies GPIO flags, as defined in
+ <dt-bindings/gpio/gpio.h>. Only the GPIO_ACTIVE_HIGH and
+ GPIO_ACTIVE_LOW flags are supported.
+- gpio-ranges: Range of pins managed by the GPIO controller.
+
+Xtal Clock bindings for Marvell Armada 37xx SoCs
+------------------------------------------------
+
+see Documentation/devicetree/bindings/clock/armada3700-xtal-clock.txt
+
+
+Example:
+pinctrl_sb: pinctrl-sb@18800 {
+ compatible = "marvell,armada3710-sb-pinctrl", "syscon", "simple-mfd";
+ reg = <0x18800 0x100>, <0x18C00 0x20>;
+ gpio {
+ #gpio-cells = <2>;
+ gpio-ranges = <&pinctrl_sb 0 0 29>;
+ gpio-controller;
+ interrupts =
+ <GIC_SPI 160 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 159 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 158 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 157 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 156 IRQ_TYPE_LEVEL_HIGH>;
+ };
+
+ rgmii_pins: mii-pins {
+ groups = "rgmii";
+ function = "mii";
+ };
+
+};
diff --git a/Documentation/devicetree/bindings/pinctrl/pinctrl-aspeed.txt b/Documentation/devicetree/bindings/pinctrl/pinctrl-aspeed.txt
index b98e6f0..ca01710 100644
--- a/Documentation/devicetree/bindings/pinctrl/pinctrl-aspeed.txt
+++ b/Documentation/devicetree/bindings/pinctrl/pinctrl-aspeed.txt
@@ -34,13 +34,28 @@
Subnode Format
==============
-The required properties of child nodes are (as defined in pinctrl-bindings):
-- function
-- groups
+The required properties of pinmux child nodes are:
+- function: the mux function to select
+- groups : the list of groups to select with this function
-Each function has only one associated pin group. Each group is named by its
-function. The following values for the function and groups properties are
-supported:
+Required properties of pinconf child nodes are:
+- groups: A list of groups to select (either this or "pins" must be
+ specified)
+- pins : A list of ball names as strings, eg "D14" (either this or "groups"
+ must be specified)
+
+Optional properties of pinconf child nodes are:
+- bias-disable : disable any pin bias
+- bias-pull-down: pull down the pin
+- drive-strength: sink or source at most X mA
+
+Definitions are as specified in
+Documentation/devicetree/bindings/pinctrl/pinctrl-bindings.txt, with any
+further limitations as described above.
+
+For pinmux, each mux function has only one associated pin group. Each group is
+named by its function. The following values for the function and groups
+properties are supported:
aspeed,ast2400-pinctrl, aspeed,g4-pinctrl:
@@ -90,6 +105,11 @@
function = "I2C3";
groups = "I2C3";
};
+
+ pinctrl_gpioh0_unbiased_default: gpioh0 {
+ pins = "A8";
+ bias-disable;
+ };
};
};
@@ -110,6 +130,11 @@
function = "I2C3";
groups = "I2C3";
};
+
+ pinctrl_gpioh0_unbiased_default: gpioh0 {
+ pins = "A18";
+ bias-disable;
+ };
};
};
@@ -143,6 +168,3 @@
};
};
};
-
-Please refer to pinctrl-bindings.txt in this directory for details of the
-common pinctrl bindings used by client devices.
diff --git a/Documentation/devicetree/bindings/pinctrl/pinctrl-bindings.txt b/Documentation/devicetree/bindings/pinctrl/pinctrl-bindings.txt
index bf3f7b0..71a3c13 100644
--- a/Documentation/devicetree/bindings/pinctrl/pinctrl-bindings.txt
+++ b/Documentation/devicetree/bindings/pinctrl/pinctrl-bindings.txt
@@ -162,8 +162,8 @@
pins = "mfio29", "mfio30";
};
-Optionally an altenative binding can be used if more suitable depending on the
-pin controller hardware. For hardaware where there is a large number of identical
+Optionally an alternative binding can be used if more suitable depending on the
+pin controller hardware. For hardware where there is a large number of identical
pin controller instances, naming each pin and function can easily become
unmaintainable. This is especially the case if the same controller is used for
different pins and functions depending on the SoC revision and packaging.
@@ -198,6 +198,28 @@
this is to avoid mapping of the index in the dts files and the pin controller
driver as it can change.
+For hardware where pin multiplexing configurations have to be specified for
+each single pin the number of required sub-nodes containing "pin" and
+"function" properties can quickly escalate and become hard to write and
+maintain.
+
+For cases like this, the pin controller driver may use the pinmux helper
+property, where the pin identifier is packed with mux configuration settings
+in a single integer.
+
+The pinmux property accepts an array of integers, each of them describing
+a single pin multiplexing configuration.
+
+pincontroller {
+ state_0_node_a {
+ pinmux = <PIN_ID_AND_MUX>, <PIN_ID_AND_MUX>, ...;
+ };
+};
+
+Each individual pin controller driver bindings documentation shall specify
+how those values (pin IDs and pin multiplexing configuration) are defined and
+assembled together.
+
== Generic pin configuration node content ==
Many data items that are represented in a pin configuration node are common
@@ -210,18 +232,22 @@
Supported generic properties are:
pins - the list of pins that properties in the node
- apply to (either this or "group" has to be
+ apply to (either this, "group" or "pinmux" has to be
specified)
group - the group to apply the properties to, if the driver
supports configuration of whole groups rather than
- individual pins (either this or "pins" has to be
- specified)
+ individual pins (either this, "pins" or "pinmux" has
+ to be specified)
+pinmux - the list of numeric pin ids and their mux settings
+ that properties in the node apply to (either this,
+ "pins" or "groups" have to be specified)
bias-disable - disable any pin bias
bias-high-impedance - high impedance mode ("third-state", "floating")
bias-bus-hold - latch weakly
bias-pull-up - pull up the pin
bias-pull-down - pull down the pin
bias-pull-pin-default - use pin-default pull state
+bi-directional - pin supports simultaneous input/output operations
drive-push-pull - drive actively high and low
drive-open-drain - drive with open drain
drive-open-source - drive with open source
@@ -234,6 +260,7 @@
power-source - select between different power supplies
low-power-enable - enable low power mode
low-power-disable - disable low power mode
+output-enable - enable output on pin regardless of output value
output-low - set the pin to output mode with low level
output-high - set the pin to output mode with high level
slew-rate - set the slew rate
@@ -258,6 +285,12 @@
bias-pull-up;
};
};
+state_3_node_a {
+ mux {
+ pinmux = <GPIOx_PINm_MUXn>, <GPIOx_PINj_MUXk)>;
+ input-enable;
+ };
+};
Some of the generic properties take arguments. For those that do, the
arguments are described below.
@@ -266,6 +299,11 @@
binding for the hardware defines:
- Whether the entries are integers or strings, and their meaning.
+- pinmux takes a list of pin IDs and mux settings as required argument. The
+ specific bindings for the hardware defines:
+ - How pin IDs and mux settings are defined and assembled together in a single
+ integer.
+
- bias-pull-up, -down and -pin-default take as optional argument on hardware
supporting it the pull strength in Ohm. bias-disable will disable the pull.
diff --git a/Documentation/devicetree/bindings/pinctrl/rockchip,pinctrl.txt b/Documentation/devicetree/bindings/pinctrl/rockchip,pinctrl.txt
index 4722bc6..ee01ab5 100644
--- a/Documentation/devicetree/bindings/pinctrl/rockchip,pinctrl.txt
+++ b/Documentation/devicetree/bindings/pinctrl/rockchip,pinctrl.txt
@@ -19,11 +19,18 @@
defined as gpio sub-nodes of the pinmux controller.
Required properties for iomux controller:
- - compatible: one of "rockchip,rk1108-pinctrl", "rockchip,rk2928-pinctrl"
- "rockchip,rk3066a-pinctrl", "rockchip,rk3066b-pinctrl"
- "rockchip,rk3188-pinctrl", "rockchip,rk3228-pinctrl"
- "rockchip,rk3288-pinctrl", "rockchip,rk3368-pinctrl"
- "rockchip,rk3399-pinctrl"
+ - compatible: should be
+ "rockchip,rv1108-pinctrl": for Rockchip RV1108
+ "rockchip,rk2928-pinctrl": for Rockchip RK2928
+ "rockchip,rk3066a-pinctrl": for Rockchip RK3066a
+ "rockchip,rk3066b-pinctrl": for Rockchip RK3066b
+ "rockchip,rk3188-pinctrl": for Rockchip RK3188
+ "rockchip,rk3228-pinctrl": for Rockchip RK3228
+ "rockchip,rk3288-pinctrl": for Rockchip RK3288
+ "rockchip,rk3328-pinctrl": for Rockchip RK3328
+ "rockchip,rk3368-pinctrl": for Rockchip RK3368
+ "rockchip,rk3399-pinctrl": for Rockchip RK3399
+
- rockchip,grf: phandle referencing a syscon providing the
"general register files"
diff --git a/Documentation/devicetree/bindings/pinctrl/st,stm32-pinctrl.txt b/Documentation/devicetree/bindings/pinctrl/st,stm32-pinctrl.txt
index eac20aa..d907a74 100644
--- a/Documentation/devicetree/bindings/pinctrl/st,stm32-pinctrl.txt
+++ b/Documentation/devicetree/bindings/pinctrl/st,stm32-pinctrl.txt
@@ -9,6 +9,7 @@
Required properies:
- compatible: value should be one of the following:
"st,stm32f429-pinctrl"
+ "st,stm32f469-pinctrl"
"st,stm32f746-pinctrl"
"st,stm32h743-pinctrl"
- #address-cells: The value of this property must be 1
@@ -38,8 +39,6 @@
- st,syscfg: Should be phandle/offset pair. The phandle to the syscon node
which includes IRQ mux selection register, and the offset of the IRQ mux
selection register.
- - ngpios: Number of gpios in a bank (to use if bank gpio numbers is less
- than 16).
- gpio-ranges: Define a dedicated mapping between a pin-controller and
a gpio controller. Format is <&phandle a b c> with:
-(phandle): phandle of pin-controller.
diff --git a/Documentation/devicetree/bindings/power/power_domain.txt b/Documentation/devicetree/bindings/power/power_domain.txt
index 723e1ad..940707d 100644
--- a/Documentation/devicetree/bindings/power/power_domain.txt
+++ b/Documentation/devicetree/bindings/power/power_domain.txt
@@ -31,7 +31,9 @@
- domain-idle-states : A phandle of an idle-state that shall be soaked into a
generic domain power state. The idle state definitions are
- compatible with domain-idle-state specified in [1].
+ compatible with domain-idle-state specified in [1]. phandles
+ that are not compatible with domain-idle-state will be
+ ignored.
The domain-idle-state property reflects the idle state of this PM domain and
not the idle states of the devices or sub-domains in the PM domain. Devices
and sub-domains have their own idle-states independent of the parent
diff --git a/Documentation/devicetree/bindings/power/reset/gemini-poweroff.txt b/Documentation/devicetree/bindings/power/reset/gemini-poweroff.txt
new file mode 100644
index 0000000..7fec3e1
--- /dev/null
+++ b/Documentation/devicetree/bindings/power/reset/gemini-poweroff.txt
@@ -0,0 +1,17 @@
+* Device-Tree bindings for Cortina Systems Gemini Poweroff
+
+This is a special IP block in the Cortina Gemini SoC that only
+deals with different ways to power the system down.
+
+Required properties:
+- compatible: should be "cortina,gemini-power-controller"
+- reg: should contain the physical memory base and size
+- interrupts: should contain the power management interrupt
+
+Example:
+
+power-controller@4b000000 {
+ compatible = "cortina,gemini-power-controller";
+ reg = <0x4b000000 0x100>;
+ interrupts = <26 IRQ_TYPE_EDGE_FALLING>;
+};
diff --git a/Documentation/devicetree/bindings/power/reset/syscon-poweroff.txt b/Documentation/devicetree/bindings/power/reset/syscon-poweroff.txt
index 1e2546f..022ed1f 100644
--- a/Documentation/devicetree/bindings/power/reset/syscon-poweroff.txt
+++ b/Documentation/devicetree/bindings/power/reset/syscon-poweroff.txt
@@ -3,13 +3,20 @@
This is a generic poweroff driver using syscon to map the poweroff register.
The poweroff is generally performed with a write to the poweroff register
defined by the register map pointed by syscon reference plus the offset
-with the mask defined in the poweroff node.
+with the value and mask defined in the poweroff node.
Required properties:
- compatible: should contain "syscon-poweroff"
- regmap: this is phandle to the register map node
- offset: offset in the register map for the poweroff register (in bytes)
-- mask: the poweroff value written to the poweroff register (32 bit access)
+- value: the poweroff value written to the poweroff register (32 bit access)
+
+Optional properties:
+- mask: update only the register bits defined by the mask (32 bit)
+
+Legacy usage:
+If a node doesn't contain a value property but contains a mask property, the
+mask property is used as the value.
Default will be little endian mode, 32 bit access only.
diff --git a/Documentation/devicetree/bindings/power/rockchip-io-domain.txt b/Documentation/devicetree/bindings/power/rockchip-io-domain.txt
index d23dc00..d3a5a93 100644
--- a/Documentation/devicetree/bindings/power/rockchip-io-domain.txt
+++ b/Documentation/devicetree/bindings/power/rockchip-io-domain.txt
@@ -33,6 +33,7 @@
- compatible: should be one of:
- "rockchip,rk3188-io-voltage-domain" for rk3188
- "rockchip,rk3288-io-voltage-domain" for rk3288
+ - "rockchip,rk3328-io-voltage-domain" for rk3328
- "rockchip,rk3368-io-voltage-domain" for rk3368
- "rockchip,rk3368-pmu-io-voltage-domain" for rk3368 pmu-domains
- "rockchip,rk3399-io-voltage-domain" for rk3399
diff --git a/Documentation/devicetree/bindings/power/supply/cpcap-charger.txt b/Documentation/devicetree/bindings/power/supply/cpcap-charger.txt
new file mode 100644
index 0000000..80bd873
--- /dev/null
+++ b/Documentation/devicetree/bindings/power/supply/cpcap-charger.txt
@@ -0,0 +1,37 @@
+Motorola CPCAP PMIC battery charger binding
+
+Required properties:
+- compatible: Shall be "motorola,mapphone-cpcap-charger"
+- interrupts: Interrupt specifier for each name in interrupt-names
+- interrupt-names: Should contain the following entries:
+ "chrg_det", "rvrs_chrg", "chrg_se1b", "se0conn",
+ "rvrs_mode", "chrgcurr1", "vbusvld", "battdetb"
+- io-channels: IIO ADC channel specifier for each name in io-channel-names
+- io-channel-names: Should contain the following entries:
+ "battdetb", "battp", "vbus", "chg_isense", "batti"
+
+Optional properties:
+- mode-gpios: Optionally CPCAP charger can have a companion wireless
+ charge controller that is controlled with two GPIOs
+ that are active low.
+
+Example:
+
+cpcap_charger: charger {
+ compatible = "motorola,mapphone-cpcap-charger";
+ interrupts-extended = <
+ &cpcap 13 0 &cpcap 12 0 &cpcap 29 0 &cpcap 28 0
+ &cpcap 22 0 &cpcap 20 0 &cpcap 19 0 &cpcap 54 0
+ >;
+ interrupt-names =
+ "chrg_det", "rvrs_chrg", "chrg_se1b", "se0conn",
+ "rvrs_mode", "chrgcurr1", "vbusvld", "battdetb";
+ mode-gpios = <&gpio3 29 GPIO_ACTIVE_LOW
+ &gpio3 23 GPIO_ACTIVE_LOW>;
+ io-channels = <&cpcap_adc 0 &cpcap_adc 1
+ &cpcap_adc 2 &cpcap_adc 5
+ &cpcap_adc 6>;
+ io-channel-names = "battdetb", "battp",
+ "vbus", "chg_isense",
+ "batti";
+};
diff --git a/Documentation/devicetree/bindings/power/supply/lego_ev3_battery.txt b/Documentation/devicetree/bindings/power/supply/lego_ev3_battery.txt
new file mode 100644
index 0000000..5485633
--- /dev/null
+++ b/Documentation/devicetree/bindings/power/supply/lego_ev3_battery.txt
@@ -0,0 +1,21 @@
+LEGO MINDSTORMS EV3 Battery
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+LEGO MINDSTORMS EV3 has some built-in capability for monitoring the battery.
+It uses 6 AA batteries or a special Li-ion rechargeable battery pack that is
+detected by a key switch in the battery compartment.
+
+Required properties:
+ - compatible: Must be "lego,ev3-battery"
+ - io-channels: phandles to analog inputs for reading voltage and current
+ - io-channel-names: Must be "voltage", "current"
+ - rechargeable-gpios: phandle to the rechargeable battery indication gpio
+
+Example:
+
+ battery {
+ compatible = "lego,ev3-battery";
+ io-channels = <&adc 4>, <&adc 3>;
+ io-channel-names = "voltage", "current";
+ rechargeable-gpios = <&gpio 136 GPIO_ACTIVE_LOW>;
+ };
diff --git a/Documentation/devicetree/bindings/power/supply/ltc2941.txt b/Documentation/devicetree/bindings/power/supply/ltc2941.txt
index ea42ae1..a9d7aa6 100644
--- a/Documentation/devicetree/bindings/power/supply/ltc2941.txt
+++ b/Documentation/devicetree/bindings/power/supply/ltc2941.txt
@@ -6,8 +6,8 @@
formula for the charge counter.
Required properties:
-- compatible: Should contain "ltc2941" or "ltc2943" which also indicates the
- type of I2C chip attached.
+- compatible: Should contain "lltc,ltc2941" or "lltc,ltc2943" which also
+ indicates the type of I2C chip attached.
- reg: The 7-bit I2C address.
- lltc,resistor-sense: The sense resistor value in milli-ohms. Can be a 32-bit
negative value when the battery has been connected to the wrong end of the
@@ -20,7 +20,7 @@
Example from the Topic Miami Florida board:
fuelgauge: ltc2943@64 {
- compatible = "ltc2943";
+ compatible = "lltc,ltc2943";
reg = <0x64>;
lltc,resistor-sense = <15>;
lltc,prescaler-exponent = <5>; /* 2^(2*5) = 1024 */
diff --git a/Documentation/devicetree/bindings/power/supply/max8925_batter.txt b/Documentation/devicetree/bindings/power/supply/max8925_battery.txt
similarity index 100%
rename from Documentation/devicetree/bindings/power/supply/max8925_batter.txt
rename to Documentation/devicetree/bindings/power/supply/max8925_battery.txt
diff --git a/Documentation/devicetree/bindings/rng/amlogic,meson-rng.txt b/Documentation/devicetree/bindings/rng/amlogic,meson-rng.txt
index 202f2d0..4d40364 100644
--- a/Documentation/devicetree/bindings/rng/amlogic,meson-rng.txt
+++ b/Documentation/devicetree/bindings/rng/amlogic,meson-rng.txt
@@ -6,9 +6,16 @@
- compatible : should be "amlogic,meson-rng"
- reg : Specifies base physical address and size of the registers.
+Optional properties:
+
+- clocks : phandle to the following named clocks
+- clock-names: Name of core clock, must be "core"
+
Example:
rng {
- compatible = "amlogic,meson-rng";
- reg = <0x0 0xc8834000 0x0 0x4>;
+ compatible = "amlogic,meson-rng";
+ reg = <0x0 0xc8834000 0x0 0x4>;
+ clocks = <&clkc CLKID_RNG0>;
+ clock-names = "core";
};
diff --git a/Documentation/devicetree/bindings/rng/mtk-rng.txt b/Documentation/devicetree/bindings/rng/mtk-rng.txt
new file mode 100644
index 0000000..a6d62a2
--- /dev/null
+++ b/Documentation/devicetree/bindings/rng/mtk-rng.txt
@@ -0,0 +1,18 @@
+Device-Tree bindings for Mediatek random number generator
+found in Mediatek SoC family
+
+Required properties:
+- compatible : Should be "mediatek,mt7623-rng"
+- clocks : list of clock specifiers, corresponding to
+ entries in clock-names property;
+- clock-names : Should contain "rng" entries;
+- reg : Specifies base physical address and size of the registers
+
+Example:
+
+rng: rng@1020f000 {
+ compatible = "mediatek,mt7623-rng";
+ reg = <0 0x1020f000 0 0x1000>;
+ clocks = <&infracfg CLK_INFRA_TRNG>;
+ clock-names = "rng";
+};
diff --git a/Documentation/devicetree/bindings/timer/cortina,gemini-timer.txt b/Documentation/devicetree/bindings/timer/cortina,gemini-timer.txt
deleted file mode 100644
index 16ea1d3..0000000
--- a/Documentation/devicetree/bindings/timer/cortina,gemini-timer.txt
+++ /dev/null
@@ -1,22 +0,0 @@
-Cortina Systems Gemini timer
-
-This timer is embedded in the Cortina Systems Gemini SoCs.
-
-Required properties:
-
-- compatible : Must be "cortina,gemini-timer"
-- reg : Should contain registers location and length
-- interrupts : Should contain the three timer interrupts with
- flags for rising edge
-- syscon : a phandle to the global Gemini system controller
-
-Example:
-
-timer@43000000 {
- compatible = "cortina,gemini-timer";
- reg = <0x43000000 0x1000>;
- interrupts = <14 IRQ_TYPE_EDGE_RISING>, /* Timer 1 */
- <15 IRQ_TYPE_EDGE_RISING>, /* Timer 2 */
- <16 IRQ_TYPE_EDGE_RISING>; /* Timer 3 */
- syscon = <&syscon>;
-};
diff --git a/Documentation/devicetree/bindings/timer/faraday,fttmr010.txt b/Documentation/devicetree/bindings/timer/faraday,fttmr010.txt
new file mode 100644
index 0000000..b73ca6c
--- /dev/null
+++ b/Documentation/devicetree/bindings/timer/faraday,fttmr010.txt
@@ -0,0 +1,33 @@
+Faraday Technology timer
+
+This timer is a generic IP block from Faraday Technology, embedded in the
+Cortina Systems Gemini SoCs and other designs.
+
+Required properties:
+
+- compatible : Must be one of
+ "faraday,fttmr010"
+ "cortina,gemini-timer"
+- reg : Should contain registers location and length
+- interrupts : Should contain the three timer interrupts usually with
+ flags for falling edge
+
+Optionally required properties:
+
+- clocks : a clock to provide the tick rate for "faraday,fttmr010"
+- clock-names : should be "EXTCLK" and "PCLK" for the external tick timer
+ and peripheral clock respectively, for "faraday,fttmr010"
+- syscon : a phandle to the global Gemini system controller if the compatible
+ type is "cortina,gemini-timer"
+
+Example:
+
+timer@43000000 {
+ compatible = "faraday,fttmr010";
+ reg = <0x43000000 0x1000>;
+ interrupts = <14 IRQ_TYPE_EDGE_FALLING>, /* Timer 1 */
+ <15 IRQ_TYPE_EDGE_FALLING>, /* Timer 2 */
+ <16 IRQ_TYPE_EDGE_FALLING>; /* Timer 3 */
+ clocks = <&extclk>, <&pclk>;
+ clock-names = "EXTCLK", "PCLK";
+};
diff --git a/Documentation/devicetree/bindings/timer/rockchip,rk-timer.txt b/Documentation/devicetree/bindings/timer/rockchip,rk-timer.txt
index a41b184..16a5f45 100644
--- a/Documentation/devicetree/bindings/timer/rockchip,rk-timer.txt
+++ b/Documentation/devicetree/bindings/timer/rockchip,rk-timer.txt
@@ -1,9 +1,15 @@
Rockchip rk timer
Required properties:
-- compatible: shall be one of:
- "rockchip,rk3288-timer" - for rk3066, rk3036, rk3188, rk322x, rk3288, rk3368
- "rockchip,rk3399-timer" - for rk3399
+- compatible: should be:
+ "rockchip,rk3036-timer", "rockchip,rk3288-timer": for Rockchip RK3036
+ "rockchip,rk3066-timer", "rockchip,rk3288-timer": for Rockchip RK3066
+ "rockchip,rk3188-timer", "rockchip,rk3288-timer": for Rockchip RK3188
+ "rockchip,rk3228-timer", "rockchip,rk3288-timer": for Rockchip RK3228
+ "rockchip,rk3229-timer", "rockchip,rk3288-timer": for Rockchip RK3229
+ "rockchip,rk3288-timer": for Rockchip RK3288
+ "rockchip,rk3368-timer", "rockchip,rk3288-timer": for Rockchip RK3368
+ "rockchip,rk3399-timer": for Rockchip RK3399
- reg: base address of the timer register starting with TIMERS CONTROL register
- interrupts: should contain the interrupts for Timer0
- clocks : must contain an entry for each entry in clock-names
diff --git a/Documentation/devicetree/bindings/vendor-prefixes.txt b/Documentation/devicetree/bindings/vendor-prefixes.txt
index ec0bfb9..b685d13 100644
--- a/Documentation/devicetree/bindings/vendor-prefixes.txt
+++ b/Documentation/devicetree/bindings/vendor-prefixes.txt
@@ -51,6 +51,7 @@
buffalo Buffalo, Inc.
calxeda Calxeda
capella Capella Microsystems, Inc
+cascoda Cascoda, Ltd.
cavium Cavium, Inc.
cdns Cadence Design Systems Inc.
ceva Ceva, Inc.
@@ -265,6 +266,7 @@
schindler Schindler
seagate Seagate Technology PLC
semtech Semtech Corporation
+sensirion Sensirion AG
sgx SGX Sensortech
sharp Sharp Corporation
si-en Si-En Technology Ltd.
diff --git a/Documentation/doc-guide/hello.dot b/Documentation/doc-guide/hello.dot
new file mode 100644
index 0000000..504621df
--- /dev/null
+++ b/Documentation/doc-guide/hello.dot
@@ -0,0 +1,3 @@
+graph G {
+ Hello -- World
+}
diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/sphinx.rst
index 96fe7ccb..731334d 100644
--- a/Documentation/doc-guide/sphinx.rst
+++ b/Documentation/doc-guide/sphinx.rst
@@ -34,8 +34,9 @@
To generate documentation, Sphinx (``sphinx-build``) must obviously be
installed. For prettier HTML output, the Read the Docs Sphinx theme
-(``sphinx_rtd_theme``) is used if available. For PDF output, ``rst2pdf`` is also
-needed. All of these are widely available and packaged in distributions.
+(``sphinx_rtd_theme``) is used if available. For PDF output you'll also need
+``XeLaTeX`` and ``convert(1)`` from ImageMagick (https://www.imagemagick.org).
+All of these are widely available and packaged in distributions.
To pass extra options to Sphinx, you can use the ``SPHINXOPTS`` make
variable. For example, use ``make SPHINXOPTS=-v htmldocs`` to get more verbose
@@ -73,7 +74,16 @@
Here are some specific guidelines for the kernel documentation:
-* Please don't go overboard with reStructuredText markup. Keep it simple.
+* Please don't go overboard with reStructuredText markup. Keep it
+ simple. For the most part the documentation should be plain text with
+ just enough consistency in formatting that it can be converted to
+ other formats.
+
+* Please keep the formatting changes minimal when converting existing
+ documentation to reStructuredText.
+
+* Also update the content, not just the formatting, when converting
+ documentation.
* Please stick to this order of heading adornments:
@@ -103,6 +113,12 @@
the order as encountered."), having the higher levels the same overall makes
it easier to follow the documents.
+* For inserting fixed width text blocks (for code examples, use case
+ examples, etc.), use ``::`` for anything that doesn't really benefit
+ from syntax highlighting, especially short snippets. Use
+ ``.. code-block:: <language>`` for longer code blocks that benefit
+ from highlighting.
+
the C domain
------------
@@ -217,3 +233,96 @@
* .. _`last row`:
- column 3
+
+
+Figures & Images
+================
+
+If you want to add an image, you should use the ``kernel-figure`` and
+``kernel-image`` directives. E.g. to insert a figure with a scalable
+image format use SVG (:ref:`svg_image_example`)::
+
+ .. kernel-figure:: svg_image.svg
+ :alt: simple SVG image
+
+ SVG image example
+
+.. _svg_image_example:
+
+.. kernel-figure:: svg_image.svg
+ :alt: simple SVG image
+
+ SVG image example
+
+The kernel figure (and image) directive support **DOT** formated files, see
+
+* DOT: http://graphviz.org/pdf/dotguide.pdf
+* Graphviz: http://www.graphviz.org/content/dot-language
+
+A simple example (:ref:`hello_dot_file`)::
+
+ .. kernel-figure:: hello.dot
+ :alt: hello world
+
+ DOT's hello world example
+
+.. _hello_dot_file:
+
+.. kernel-figure:: hello.dot
+ :alt: hello world
+
+ DOT's hello world example
+
+Embed *render* markups (or languages) like Graphviz's **DOT** is provided by the
+``kernel-render`` directives.::
+
+ .. kernel-render:: DOT
+ :alt: foobar digraph
+ :caption: Embedded **DOT** (Graphviz) code
+
+ digraph foo {
+ "bar" -> "baz";
+ }
+
+How this will be rendered depends on the installed tools. If Graphviz is
+installed, you will see an vector image. If not the raw markup is inserted as
+*literal-block* (:ref:`hello_dot_render`).
+
+.. _hello_dot_render:
+
+.. kernel-render:: DOT
+ :alt: foobar digraph
+ :caption: Embedded **DOT** (Graphviz) code
+
+ digraph foo {
+ "bar" -> "baz";
+ }
+
+The *render* directive has all the options known from the *figure* directive,
+plus option ``caption``. If ``caption`` has a value, a *figure* node is
+inserted. If not, a *image* node is inserted. A ``caption`` is also needed, if
+you want to refer it (:ref:`hello_svg_render`).
+
+Embedded **SVG**::
+
+ .. kernel-render:: SVG
+ :caption: Embedded **SVG** markup
+ :alt: so-nw-arrow
+
+ <?xml version="1.0" encoding="UTF-8"?>
+ <svg xmlns="http://www.w3.org/2000/svg" version="1.1" ...>
+ ...
+ </svg>
+
+.. _hello_svg_render:
+
+.. kernel-render:: SVG
+ :caption: Embedded **SVG** markup
+ :alt: so-nw-arrow
+
+ <?xml version="1.0" encoding="UTF-8"?>
+ <svg xmlns="http://www.w3.org/2000/svg"
+ version="1.1" baseProfile="full" width="70px" height="40px" viewBox="0 0 700 400">
+ <line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/>
+ <polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/>
+ </svg>
diff --git a/Documentation/doc-guide/svg_image.svg b/Documentation/doc-guide/svg_image.svg
new file mode 100644
index 0000000..5405f85
--- /dev/null
+++ b/Documentation/doc-guide/svg_image.svg
@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- originate: https://commons.wikimedia.org/wiki/File:Variable_Resistor.svg -->
+<svg xmlns="http://www.w3.org/2000/svg"
+ version="1.1" baseProfile="full"
+ width="70px" height="40px" viewBox="0 0 700 400">
+ <line x1="0" y1="200" x2="700" y2="200" stroke="black" stroke-width="20px"/>
+ <rect x="100" y="100" width="500" height="200" fill="white" stroke="black" stroke-width="20px"/>
+ <line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/>
+ <polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/>
+</svg>
diff --git a/Documentation/driver-api/80211/cfg80211.rst b/Documentation/driver-api/80211/cfg80211.rst
index eca534a..8ffac57 100644
--- a/Documentation/driver-api/80211/cfg80211.rst
+++ b/Documentation/driver-api/80211/cfg80211.rst
@@ -2,6 +2,9 @@
cfg80211 subsystem
==================
+.. kernel-doc:: include/net/cfg80211.h
+ :doc: Introduction
+
Device registration
===================
@@ -180,6 +183,12 @@
:functions: cfg80211_ibss_joined
.. kernel-doc:: include/net/cfg80211.h
+ :functions: cfg80211_connect_resp_params
+
+.. kernel-doc:: include/net/cfg80211.h
+ :functions: cfg80211_connect_done
+
+.. kernel-doc:: include/net/cfg80211.h
:functions: cfg80211_connect_result
.. kernel-doc:: include/net/cfg80211.h
diff --git a/Documentation/driver-api/basics.rst b/Documentation/driver-api/basics.rst
index 935b9b8..472e7a6 100644
--- a/Documentation/driver-api/basics.rst
+++ b/Documentation/driver-api/basics.rst
@@ -7,6 +7,12 @@
.. kernel-doc:: include/linux/init.h
:internal:
+Driver device table
+-------------------
+
+.. kernel-doc:: include/linux/mod_devicetable.h
+ :internal:
+
Atomic and pointer manipulation
-------------------------------
diff --git a/Documentation/driver-api/firmware/index.rst b/Documentation/driver-api/firmware/index.rst
index 1abe017..29da39e 100644
--- a/Documentation/driver-api/firmware/index.rst
+++ b/Documentation/driver-api/firmware/index.rst
@@ -7,6 +7,7 @@
introduction
core
request_firmware
+ other_interfaces
.. only:: subproject and html
diff --git a/Documentation/driver-api/firmware/other_interfaces.rst b/Documentation/driver-api/firmware/other_interfaces.rst
new file mode 100644
index 0000000..36c47b1
--- /dev/null
+++ b/Documentation/driver-api/firmware/other_interfaces.rst
@@ -0,0 +1,15 @@
+Other Firmware Interfaces
+=========================
+
+DMI Interfaces
+--------------
+
+.. kernel-doc:: drivers/firmware/dmi_scan.c
+ :export:
+
+EDD Interfaces
+--------------
+
+.. kernel-doc:: drivers/firmware/edd.c
+ :internal:
+
diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst
index 60db00d..8058a87 100644
--- a/Documentation/driver-api/index.rst
+++ b/Documentation/driver-api/index.rst
@@ -26,7 +26,8 @@
regulator
iio/index
input
- usb
+ usb/index
+ pci
spi
i2c
hsi
@@ -36,6 +37,7 @@
80211/index
uio-howto
firmware/index
+ misc_devices
.. only:: subproject and html
diff --git a/Documentation/driver-api/misc_devices.rst b/Documentation/driver-api/misc_devices.rst
new file mode 100644
index 0000000..c7ee7b0
--- /dev/null
+++ b/Documentation/driver-api/misc_devices.rst
@@ -0,0 +1,5 @@
+Miscellaneous Devices
+=====================
+
+.. kernel-doc:: drivers/char/misc.c
+ :export:
diff --git a/Documentation/driver-api/pci.rst b/Documentation/driver-api/pci.rst
new file mode 100644
index 0000000..01a6c8b
--- /dev/null
+++ b/Documentation/driver-api/pci.rst
@@ -0,0 +1,50 @@
+PCI Support Library
+-------------------
+
+.. kernel-doc:: drivers/pci/pci.c
+ :export:
+
+.. kernel-doc:: drivers/pci/pci-driver.c
+ :export:
+
+.. kernel-doc:: drivers/pci/remove.c
+ :export:
+
+.. kernel-doc:: drivers/pci/search.c
+ :export:
+
+.. kernel-doc:: drivers/pci/msi.c
+ :export:
+
+.. kernel-doc:: drivers/pci/bus.c
+ :export:
+
+.. kernel-doc:: drivers/pci/access.c
+ :export:
+
+.. kernel-doc:: drivers/pci/irq.c
+ :export:
+
+.. kernel-doc:: drivers/pci/htirq.c
+ :export:
+
+.. kernel-doc:: drivers/pci/probe.c
+ :export:
+
+.. kernel-doc:: drivers/pci/slot.c
+ :export:
+
+.. kernel-doc:: drivers/pci/rom.c
+ :export:
+
+.. kernel-doc:: drivers/pci/iov.c
+ :export:
+
+.. kernel-doc:: drivers/pci/pci-sysfs.c
+ :internal:
+
+PCI Hotplug Support Library
+---------------------------
+
+.. kernel-doc:: drivers/pci/hotplug/pci_hotplug_core.c
+ :export:
diff --git a/Documentation/driver-api/usb.rst b/Documentation/driver-api/usb.rst
deleted file mode 100644
index 851cc40b..0000000
--- a/Documentation/driver-api/usb.rst
+++ /dev/null
@@ -1,748 +0,0 @@
-===========================
-The Linux-USB Host Side API
-===========================
-
-Introduction to USB on Linux
-============================
-
-A Universal Serial Bus (USB) is used to connect a host, such as a PC or
-workstation, to a number of peripheral devices. USB uses a tree
-structure, with the host as the root (the system's master), hubs as
-interior nodes, and peripherals as leaves (and slaves). Modern PCs
-support several such trees of USB devices, usually
-a few USB 3.0 (5 GBit/s) or USB 3.1 (10 GBit/s) and some legacy
-USB 2.0 (480 MBit/s) busses just in case.
-
-That master/slave asymmetry was designed-in for a number of reasons, one
-being ease of use. It is not physically possible to mistake upstream and
-downstream or it does not matter with a type C plug (or they are built into the
-peripheral). Also, the host software doesn't need to deal with
-distributed auto-configuration since the pre-designated master node
-manages all that.
-
-Kernel developers added USB support to Linux early in the 2.2 kernel
-series and have been developing it further since then. Besides support
-for each new generation of USB, various host controllers gained support,
-new drivers for peripherals have been added and advanced features for latency
-measurement and improved power management introduced.
-
-Linux can run inside USB devices as well as on the hosts that control
-the devices. But USB device drivers running inside those peripherals
-don't do the same things as the ones running inside hosts, so they've
-been given a different name: *gadget drivers*. This document does not
-cover gadget drivers.
-
-USB Host-Side API Model
-=======================
-
-Host-side drivers for USB devices talk to the "usbcore" APIs. There are
-two. One is intended for *general-purpose* drivers (exposed through
-driver frameworks), and the other is for drivers that are *part of the
-core*. Such core drivers include the *hub* driver (which manages trees
-of USB devices) and several different kinds of *host controller
-drivers*, which control individual busses.
-
-The device model seen by USB drivers is relatively complex.
-
-- USB supports four kinds of data transfers (control, bulk, interrupt,
- and isochronous). Two of them (control and bulk) use bandwidth as
- it's available, while the other two (interrupt and isochronous) are
- scheduled to provide guaranteed bandwidth.
-
-- The device description model includes one or more "configurations"
- per device, only one of which is active at a time. Devices are supposed
- to be capable of operating at lower than their top
- speeds and may provide a BOS descriptor showing the lowest speed they
- remain fully operational at.
-
-- From USB 3.0 on configurations have one or more "functions", which
- provide a common functionality and are grouped together for purposes
- of power management.
-
-- Configurations or functions have one or more "interfaces", each of which may have
- "alternate settings". Interfaces may be standardized by USB "Class"
- specifications, or may be specific to a vendor or device.
-
- USB device drivers actually bind to interfaces, not devices. Think of
- them as "interface drivers", though you may not see many devices
- where the distinction is important. *Most USB devices are simple,
- with only one function, one configuration, one interface, and one alternate
- setting.*
-
-- Interfaces have one or more "endpoints", each of which supports one
- type and direction of data transfer such as "bulk out" or "interrupt
- in". The entire configuration may have up to sixteen endpoints in
- each direction, allocated as needed among all the interfaces.
-
-- Data transfer on USB is packetized; each endpoint has a maximum
- packet size. Drivers must often be aware of conventions such as
- flagging the end of bulk transfers using "short" (including zero
- length) packets.
-
-- The Linux USB API supports synchronous calls for control and bulk
- messages. It also supports asynchronous calls for all kinds of data
- transfer, using request structures called "URBs" (USB Request
- Blocks).
-
-Accordingly, the USB Core API exposed to device drivers covers quite a
-lot of territory. You'll probably need to consult the USB 3.0
-specification, available online from www.usb.org at no cost, as well as
-class or device specifications.
-
-The only host-side drivers that actually touch hardware (reading/writing
-registers, handling IRQs, and so on) are the HCDs. In theory, all HCDs
-provide the same functionality through the same API. In practice, that's
-becoming more true, but there are still differences
-that crop up especially with fault handling on the less common controllers.
-Different controllers don't
-necessarily report the same aspects of failures, and recovery from
-faults (including software-induced ones like unlinking an URB) isn't yet
-fully consistent. Device driver authors should make a point of doing
-disconnect testing (while the device is active) with each different host
-controller driver, to make sure drivers don't have bugs of their own as
-well as to make sure they aren't relying on some HCD-specific behavior.
-
-USB-Standard Types
-==================
-
-In ``<linux/usb/ch9.h>`` you will find the USB data types defined in
-chapter 9 of the USB specification. These data types are used throughout
-USB, and in APIs including this host side API, gadget APIs, and usbfs.
-
-.. kernel-doc:: include/linux/usb/ch9.h
- :internal:
-
-Host-Side Data Types and Macros
-===============================
-
-The host side API exposes several layers to drivers, some of which are
-more necessary than others. These support lifecycle models for host side
-drivers and devices, and support passing buffers through usbcore to some
-HCD that performs the I/O for the device driver.
-
-.. kernel-doc:: include/linux/usb.h
- :internal:
-
-USB Core APIs
-=============
-
-There are two basic I/O models in the USB API. The most elemental one is
-asynchronous: drivers submit requests in the form of an URB, and the
-URB's completion callback handles the next step. All USB transfer types
-support that model, although there are special cases for control URBs
-(which always have setup and status stages, but may not have a data
-stage) and isochronous URBs (which allow large packets and include
-per-packet fault reports). Built on top of that is synchronous API
-support, where a driver calls a routine that allocates one or more URBs,
-submits them, and waits until they complete. There are synchronous
-wrappers for single-buffer control and bulk transfers (which are awkward
-to use in some driver disconnect scenarios), and for scatterlist based
-streaming i/o (bulk or interrupt).
-
-USB drivers need to provide buffers that can be used for DMA, although
-they don't necessarily need to provide the DMA mapping themselves. There
-are APIs to use used when allocating DMA buffers, which can prevent use
-of bounce buffers on some systems. In some cases, drivers may be able to
-rely on 64bit DMA to eliminate another kind of bounce buffer.
-
-.. kernel-doc:: drivers/usb/core/urb.c
- :export:
-
-.. kernel-doc:: drivers/usb/core/message.c
- :export:
-
-.. kernel-doc:: drivers/usb/core/file.c
- :export:
-
-.. kernel-doc:: drivers/usb/core/driver.c
- :export:
-
-.. kernel-doc:: drivers/usb/core/usb.c
- :export:
-
-.. kernel-doc:: drivers/usb/core/hub.c
- :export:
-
-Host Controller APIs
-====================
-
-These APIs are only for use by host controller drivers, most of which
-implement standard register interfaces such as XHCI, EHCI, OHCI, or UHCI. UHCI
-was one of the first interfaces, designed by Intel and also used by VIA;
-it doesn't do much in hardware. OHCI was designed later, to have the
-hardware do more work (bigger transfers, tracking protocol state, and so
-on). EHCI was designed with USB 2.0; its design has features that
-resemble OHCI (hardware does much more work) as well as UHCI (some parts
-of ISO support, TD list processing). XHCI was designed with USB 3.0. It
-continues to shift support for functionality into hardware.
-
-There are host controllers other than the "big three", although most PCI
-based controllers (and a few non-PCI based ones) use one of those
-interfaces. Not all host controllers use DMA; some use PIO, and there is
-also a simulator and a virtual host controller to pipe USB over the network.
-
-The same basic APIs are available to drivers for all those controllers.
-For historical reasons they are in two layers: :c:type:`struct
-usb_bus <usb_bus>` is a rather thin layer that became available
-in the 2.2 kernels, while :c:type:`struct usb_hcd <usb_hcd>`
-is a more featureful layer
-that lets HCDs share common code, to shrink driver size and
-significantly reduce hcd-specific behaviors.
-
-.. kernel-doc:: drivers/usb/core/hcd.c
- :export:
-
-.. kernel-doc:: drivers/usb/core/hcd-pci.c
- :export:
-
-.. kernel-doc:: drivers/usb/core/buffer.c
- :internal:
-
-The USB Filesystem (usbfs)
-==========================
-
-This chapter presents the Linux *usbfs*. You may prefer to avoid writing
-new kernel code for your USB driver; that's the problem that usbfs set
-out to solve. User mode device drivers are usually packaged as
-applications or libraries, and may use usbfs through some programming
-library that wraps it. Such libraries include
-`libusb <http://libusb.sourceforge.net>`__ for C/C++, and
-`jUSB <http://jUSB.sourceforge.net>`__ for Java.
-
- **Note**
-
- This particular documentation is incomplete, especially with respect
- to the asynchronous mode. As of kernel 2.5.66 the code and this
- (new) documentation need to be cross-reviewed.
-
-Configure usbfs into Linux kernels by enabling the *USB filesystem*
-option (CONFIG_USB_DEVICEFS), and you get basic support for user mode
-USB device drivers. Until relatively recently it was often (confusingly)
-called *usbdevfs* although it wasn't solving what *devfs* was. Every USB
-device will appear in usbfs, regardless of whether or not it has a
-kernel driver.
-
-What files are in "usbfs"?
---------------------------
-
-Conventionally mounted at ``/proc/bus/usb``, usbfs features include:
-
-- ``/proc/bus/usb/devices`` ... a text file showing each of the USB
- devices on known to the kernel, and their configuration descriptors.
- You can also poll() this to learn about new devices.
-
-- ``/proc/bus/usb/BBB/DDD`` ... magic files exposing the each device's
- configuration descriptors, and supporting a series of ioctls for
- making device requests, including I/O to devices. (Purely for access
- by programs.)
-
-Each bus is given a number (BBB) based on when it was enumerated; within
-each bus, each device is given a similar number (DDD). Those BBB/DDD
-paths are not "stable" identifiers; expect them to change even if you
-always leave the devices plugged in to the same hub port. *Don't even
-think of saving these in application configuration files.* Stable
-identifiers are available, for user mode applications that want to use
-them. HID and networking devices expose these stable IDs, so that for
-example you can be sure that you told the right UPS to power down its
-second server. "usbfs" doesn't (yet) expose those IDs.
-
-Mounting and Access Control
----------------------------
-
-There are a number of mount options for usbfs, which will be of most
-interest to you if you need to override the default access control
-policy. That policy is that only root may read or write device files
-(``/proc/bus/BBB/DDD``) although anyone may read the ``devices`` or
-``drivers`` files. I/O requests to the device also need the
-CAP_SYS_RAWIO capability,
-
-The significance of that is that by default, all user mode device
-drivers need super-user privileges. You can change modes or ownership in
-a driver setup when the device hotplugs, or maye just start the driver
-right then, as a privileged server (or some activity within one). That's
-the most secure approach for multi-user systems, but for single user
-systems ("trusted" by that user) it's more convenient just to grant
-everyone all access (using the *devmode=0666* option) so the driver can
-start whenever it's needed.
-
-The mount options for usbfs, usable in /etc/fstab or in command line
-invocations of *mount*, are:
-
-*busgid*\ =NNNNN
- Controls the GID used for the /proc/bus/usb/BBB directories.
- (Default: 0)
-
-*busmode*\ =MMM
- Controls the file mode used for the /proc/bus/usb/BBB directories.
- (Default: 0555)
-
-*busuid*\ =NNNNN
- Controls the UID used for the /proc/bus/usb/BBB directories.
- (Default: 0)
-
-*devgid*\ =NNNNN
- Controls the GID used for the /proc/bus/usb/BBB/DDD files. (Default:
- 0)
-
-*devmode*\ =MMM
- Controls the file mode used for the /proc/bus/usb/BBB/DDD files.
- (Default: 0644)
-
-*devuid*\ =NNNNN
- Controls the UID used for the /proc/bus/usb/BBB/DDD files. (Default:
- 0)
-
-*listgid*\ =NNNNN
- Controls the GID used for the /proc/bus/usb/devices and drivers
- files. (Default: 0)
-
-*listmode*\ =MMM
- Controls the file mode used for the /proc/bus/usb/devices and
- drivers files. (Default: 0444)
-
-*listuid*\ =NNNNN
- Controls the UID used for the /proc/bus/usb/devices and drivers
- files. (Default: 0)
-
-Note that many Linux distributions hard-wire the mount options for usbfs
-in their init scripts, such as ``/etc/rc.d/rc.sysinit``, rather than
-making it easy to set this per-system policy in ``/etc/fstab``.
-
-/proc/bus/usb/devices
----------------------
-
-This file is handy for status viewing tools in user mode, which can scan
-the text format and ignore most of it. More detailed device status
-(including class and vendor status) is available from device-specific
-files. For information about the current format of this file, see the
-``Documentation/usb/proc_usb_info.txt`` file in your Linux kernel
-sources.
-
-This file, in combination with the poll() system call, can also be used
-to detect when devices are added or removed:
-
-::
-
- int fd;
- struct pollfd pfd;
-
- fd = open("/proc/bus/usb/devices", O_RDONLY);
- pfd = { fd, POLLIN, 0 };
- for (;;) {
- /* The first time through, this call will return immediately. */
- poll(&pfd, 1, -1);
-
- /* To see what's changed, compare the file's previous and current
- contents or scan the filesystem. (Scanning is more precise.) */
- }
-
-Note that this behavior is intended to be used for informational and
-debug purposes. It would be more appropriate to use programs such as
-udev or HAL to initialize a device or start a user-mode helper program,
-for instance.
-
-/proc/bus/usb/BBB/DDD
----------------------
-
-Use these files in one of these basic ways:
-
-*They can be read,* producing first the device descriptor (18 bytes) and
-then the descriptors for the current configuration. See the USB 2.0 spec
-for details about those binary data formats. You'll need to convert most
-multibyte values from little endian format to your native host byte
-order, although a few of the fields in the device descriptor (both of
-the BCD-encoded fields, and the vendor and product IDs) will be
-byteswapped for you. Note that configuration descriptors include
-descriptors for interfaces, altsettings, endpoints, and maybe additional
-class descriptors.
-
-*Perform USB operations* using *ioctl()* requests to make endpoint I/O
-requests (synchronously or asynchronously) or manage the device. These
-requests need the CAP_SYS_RAWIO capability, as well as filesystem
-access permissions. Only one ioctl request can be made on one of these
-device files at a time. This means that if you are synchronously reading
-an endpoint from one thread, you won't be able to write to a different
-endpoint from another thread until the read completes. This works for
-*half duplex* protocols, but otherwise you'd use asynchronous i/o
-requests.
-
-Life Cycle of User Mode Drivers
--------------------------------
-
-Such a driver first needs to find a device file for a device it knows
-how to handle. Maybe it was told about it because a ``/sbin/hotplug``
-event handling agent chose that driver to handle the new device. Or
-maybe it's an application that scans all the /proc/bus/usb device files,
-and ignores most devices. In either case, it should :c:func:`read()`
-all the descriptors from the device file, and check them against what it
-knows how to handle. It might just reject everything except a particular
-vendor and product ID, or need a more complex policy.
-
-Never assume there will only be one such device on the system at a time!
-If your code can't handle more than one device at a time, at least
-detect when there's more than one, and have your users choose which
-device to use.
-
-Once your user mode driver knows what device to use, it interacts with
-it in either of two styles. The simple style is to make only control
-requests; some devices don't need more complex interactions than those.
-(An example might be software using vendor-specific control requests for
-some initialization or configuration tasks, with a kernel driver for the
-rest.)
-
-More likely, you need a more complex style driver: one using non-control
-endpoints, reading or writing data and claiming exclusive use of an
-interface. *Bulk* transfers are easiest to use, but only their sibling
-*interrupt* transfers work with low speed devices. Both interrupt and
-*isochronous* transfers offer service guarantees because their bandwidth
-is reserved. Such "periodic" transfers are awkward to use through usbfs,
-unless you're using the asynchronous calls. However, interrupt transfers
-can also be used in a synchronous "one shot" style.
-
-Your user-mode driver should never need to worry about cleaning up
-request state when the device is disconnected, although it should close
-its open file descriptors as soon as it starts seeing the ENODEV errors.
-
-The ioctl() Requests
---------------------
-
-To use these ioctls, you need to include the following headers in your
-userspace program:
-
-::
-
- #include <linux/usb.h>
- #include <linux/usbdevice_fs.h>
- #include <asm/byteorder.h>
-
-The standard USB device model requests, from "Chapter 9" of the USB 2.0
-specification, are automatically included from the ``<linux/usb/ch9.h>``
-header.
-
-Unless noted otherwise, the ioctl requests described here will update
-the modification time on the usbfs file to which they are applied
-(unless they fail). A return of zero indicates success; otherwise, a
-standard USB error code is returned. (These are documented in
-``Documentation/usb/error-codes.txt`` in your kernel sources.)
-
-Each of these files multiplexes access to several I/O streams, one per
-endpoint. Each device has one control endpoint (endpoint zero) which
-supports a limited RPC style RPC access. Devices are configured by
-hub_wq (in the kernel) setting a device-wide *configuration* that
-affects things like power consumption and basic functionality. The
-endpoints are part of USB *interfaces*, which may have *altsettings*
-affecting things like which endpoints are available. Many devices only
-have a single configuration and interface, so drivers for them will
-ignore configurations and altsettings.
-
-Management/Status Requests
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-A number of usbfs requests don't deal very directly with device I/O.
-They mostly relate to device management and status. These are all
-synchronous requests.
-
-USBDEVFS_CLAIMINTERFACE
- This is used to force usbfs to claim a specific interface, which has
- not previously been claimed by usbfs or any other kernel driver. The
- ioctl parameter is an integer holding the number of the interface
- (bInterfaceNumber from descriptor).
-
- Note that if your driver doesn't claim an interface before trying to
- use one of its endpoints, and no other driver has bound to it, then
- the interface is automatically claimed by usbfs.
-
- This claim will be released by a RELEASEINTERFACE ioctl, or by
- closing the file descriptor. File modification time is not updated
- by this request.
-
-USBDEVFS_CONNECTINFO
- Says whether the device is lowspeed. The ioctl parameter points to a
- structure like this:
-
- ::
-
- struct usbdevfs_connectinfo {
- unsigned int devnum;
- unsigned char slow;
- };
-
- File modification time is not updated by this request.
-
- *You can't tell whether a "not slow" device is connected at high
- speed (480 MBit/sec) or just full speed (12 MBit/sec).* You should
- know the devnum value already, it's the DDD value of the device file
- name.
-
-USBDEVFS_GETDRIVER
- Returns the name of the kernel driver bound to a given interface (a
- string). Parameter is a pointer to this structure, which is
- modified:
-
- ::
-
- struct usbdevfs_getdriver {
- unsigned int interface;
- char driver[USBDEVFS_MAXDRIVERNAME + 1];
- };
-
- File modification time is not updated by this request.
-
-USBDEVFS_IOCTL
- Passes a request from userspace through to a kernel driver that has
- an ioctl entry in the *struct usb_driver* it registered.
-
- ::
-
- struct usbdevfs_ioctl {
- int ifno;
- int ioctl_code;
- void *data;
- };
-
- /* user mode call looks like this.
- * 'request' becomes the driver->ioctl() 'code' parameter.
- * the size of 'param' is encoded in 'request', and that data
- * is copied to or from the driver->ioctl() 'buf' parameter.
- */
- static int
- usbdev_ioctl (int fd, int ifno, unsigned request, void *param)
- {
- struct usbdevfs_ioctl wrapper;
-
- wrapper.ifno = ifno;
- wrapper.ioctl_code = request;
- wrapper.data = param;
-
- return ioctl (fd, USBDEVFS_IOCTL, &wrapper);
- }
-
- File modification time is not updated by this request.
-
- This request lets kernel drivers talk to user mode code through
- filesystem operations even when they don't create a character or
- block special device. It's also been used to do things like ask
- devices what device special file should be used. Two pre-defined
- ioctls are used to disconnect and reconnect kernel drivers, so that
- user mode code can completely manage binding and configuration of
- devices.
-
-USBDEVFS_RELEASEINTERFACE
- This is used to release the claim usbfs made on interface, either
- implicitly or because of a USBDEVFS_CLAIMINTERFACE call, before the
- file descriptor is closed. The ioctl parameter is an integer holding
- the number of the interface (bInterfaceNumber from descriptor); File
- modification time is not updated by this request.
-
- **Warning**
-
- *No security check is made to ensure that the task which made
- the claim is the one which is releasing it. This means that user
- mode driver may interfere other ones.*
-
-USBDEVFS_RESETEP
- Resets the data toggle value for an endpoint (bulk or interrupt) to
- DATA0. The ioctl parameter is an integer endpoint number (1 to 15,
- as identified in the endpoint descriptor), with USB_DIR_IN added
- if the device's endpoint sends data to the host.
-
- **Warning**
-
- *Avoid using this request. It should probably be removed.* Using
- it typically means the device and driver will lose toggle
- synchronization. If you really lost synchronization, you likely
- need to completely handshake with the device, using a request
- like CLEAR_HALT or SET_INTERFACE.
-
-USBDEVFS_DROP_PRIVILEGES
- This is used to relinquish the ability to do certain operations
- which are considered to be privileged on a usbfs file descriptor.
- This includes claiming arbitrary interfaces, resetting a device on
- which there are currently claimed interfaces from other users, and
- issuing USBDEVFS_IOCTL calls. The ioctl parameter is a 32 bit mask
- of interfaces the user is allowed to claim on this file descriptor.
- You may issue this ioctl more than one time to narrow said mask.
-
-Synchronous I/O Support
-~~~~~~~~~~~~~~~~~~~~~~~
-
-Synchronous requests involve the kernel blocking until the user mode
-request completes, either by finishing successfully or by reporting an
-error. In most cases this is the simplest way to use usbfs, although as
-noted above it does prevent performing I/O to more than one endpoint at
-a time.
-
-USBDEVFS_BULK
- Issues a bulk read or write request to the device. The ioctl
- parameter is a pointer to this structure:
-
- ::
-
- struct usbdevfs_bulktransfer {
- unsigned int ep;
- unsigned int len;
- unsigned int timeout; /* in milliseconds */
- void *data;
- };
-
- The "ep" value identifies a bulk endpoint number (1 to 15, as
- identified in an endpoint descriptor), masked with USB_DIR_IN when
- referring to an endpoint which sends data to the host from the
- device. The length of the data buffer is identified by "len"; Recent
- kernels support requests up to about 128KBytes. *FIXME say how read
- length is returned, and how short reads are handled.*.
-
-USBDEVFS_CLEAR_HALT
- Clears endpoint halt (stall) and resets the endpoint toggle. This is
- only meaningful for bulk or interrupt endpoints. The ioctl parameter
- is an integer endpoint number (1 to 15, as identified in an endpoint
- descriptor), masked with USB_DIR_IN when referring to an endpoint
- which sends data to the host from the device.
-
- Use this on bulk or interrupt endpoints which have stalled,
- returning *-EPIPE* status to a data transfer request. Do not issue
- the control request directly, since that could invalidate the host's
- record of the data toggle.
-
-USBDEVFS_CONTROL
- Issues a control request to the device. The ioctl parameter points
- to a structure like this:
-
- ::
-
- struct usbdevfs_ctrltransfer {
- __u8 bRequestType;
- __u8 bRequest;
- __u16 wValue;
- __u16 wIndex;
- __u16 wLength;
- __u32 timeout; /* in milliseconds */
- void *data;
- };
-
- The first eight bytes of this structure are the contents of the
- SETUP packet to be sent to the device; see the USB 2.0 specification
- for details. The bRequestType value is composed by combining a
- USB_TYPE_\* value, a USB_DIR_\* value, and a USB_RECIP_\*
- value (from *<linux/usb.h>*). If wLength is nonzero, it describes
- the length of the data buffer, which is either written to the device
- (USB_DIR_OUT) or read from the device (USB_DIR_IN).
-
- At this writing, you can't transfer more than 4 KBytes of data to or
- from a device; usbfs has a limit, and some host controller drivers
- have a limit. (That's not usually a problem.) *Also* there's no way
- to say it's not OK to get a short read back from the device.
-
-USBDEVFS_RESET
- Does a USB level device reset. The ioctl parameter is ignored. After
- the reset, this rebinds all device interfaces. File modification
- time is not updated by this request.
-
- **Warning**
-
- *Avoid using this call* until some usbcore bugs get fixed, since
- it does not fully synchronize device, interface, and driver (not
- just usbfs) state.
-
-USBDEVFS_SETINTERFACE
- Sets the alternate setting for an interface. The ioctl parameter is
- a pointer to a structure like this:
-
- ::
-
- struct usbdevfs_setinterface {
- unsigned int interface;
- unsigned int altsetting;
- };
-
- File modification time is not updated by this request.
-
- Those struct members are from some interface descriptor applying to
- the current configuration. The interface number is the
- bInterfaceNumber value, and the altsetting number is the
- bAlternateSetting value. (This resets each endpoint in the
- interface.)
-
-USBDEVFS_SETCONFIGURATION
- Issues the :c:func:`usb_set_configuration()` call for the
- device. The parameter is an integer holding the number of a
- configuration (bConfigurationValue from descriptor). File
- modification time is not updated by this request.
-
- **Warning**
-
- *Avoid using this call* until some usbcore bugs get fixed, since
- it does not fully synchronize device, interface, and driver (not
- just usbfs) state.
-
-Asynchronous I/O Support
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-As mentioned above, there are situations where it may be important to
-initiate concurrent operations from user mode code. This is particularly
-important for periodic transfers (interrupt and isochronous), but it can
-be used for other kinds of USB requests too. In such cases, the
-asynchronous requests described here are essential. Rather than
-submitting one request and having the kernel block until it completes,
-the blocking is separate.
-
-These requests are packaged into a structure that resembles the URB used
-by kernel device drivers. (No POSIX Async I/O support here, sorry.) It
-identifies the endpoint type (USBDEVFS_URB_TYPE_\*), endpoint
-(number, masked with USB_DIR_IN as appropriate), buffer and length,
-and a user "context" value serving to uniquely identify each request.
-(It's usually a pointer to per-request data.) Flags can modify requests
-(not as many as supported for kernel drivers).
-
-Each request can specify a realtime signal number (between SIGRTMIN and
-SIGRTMAX, inclusive) to request a signal be sent when the request
-completes.
-
-When usbfs returns these urbs, the status value is updated, and the
-buffer may have been modified. Except for isochronous transfers, the
-actual_length is updated to say how many bytes were transferred; if the
-USBDEVFS_URB_DISABLE_SPD flag is set ("short packets are not OK"), if
-fewer bytes were read than were requested then you get an error report.
-
-::
-
- struct usbdevfs_iso_packet_desc {
- unsigned int length;
- unsigned int actual_length;
- unsigned int status;
- };
-
- struct usbdevfs_urb {
- unsigned char type;
- unsigned char endpoint;
- int status;
- unsigned int flags;
- void *buffer;
- int buffer_length;
- int actual_length;
- int start_frame;
- int number_of_packets;
- int error_count;
- unsigned int signr;
- void *usercontext;
- struct usbdevfs_iso_packet_desc iso_frame_desc[];
- };
-
-For these asynchronous requests, the file modification time reflects
-when the request was initiated. This contrasts with their use with the
-synchronous requests, where it reflects when requests complete.
-
-USBDEVFS_DISCARDURB
- *TBS* File modification time is not updated by this request.
-
-USBDEVFS_DISCSIGNAL
- *TBS* File modification time is not updated by this request.
-
-USBDEVFS_REAPURB
- *TBS* File modification time is not updated by this request.
-
-USBDEVFS_REAPURBNDELAY
- *TBS* File modification time is not updated by this request.
-
-USBDEVFS_SUBMITURB
- *TBS*
diff --git a/Documentation/driver-api/usb/URB.rst b/Documentation/driver-api/usb/URB.rst
new file mode 100644
index 0000000..61a54da
--- /dev/null
+++ b/Documentation/driver-api/usb/URB.rst
@@ -0,0 +1,290 @@
+.. _usb-urb:
+
+USB Request Block (URB)
+~~~~~~~~~~~~~~~~~~~~~~~
+
+:Revised: 2000-Dec-05
+:Again: 2002-Jul-06
+:Again: 2005-Sep-19
+:Again: 2017-Mar-29
+
+
+.. note::
+
+ The USB subsystem now has a substantial section at :ref:`usb-hostside-api`
+ section, generated from the current source code.
+ This particular documentation file isn't complete and may not be
+ updated to the last version; don't rely on it except for a quick
+ overview.
+
+Basic concept or 'What is an URB?'
+==================================
+
+The basic idea of the new driver is message passing, the message itself is
+called USB Request Block, or URB for short.
+
+- An URB consists of all relevant information to execute any USB transaction
+ and deliver the data and status back.
+
+- Execution of an URB is inherently an asynchronous operation, i.e. the
+ :c:func:`usb_submit_urb` call returns immediately after it has successfully
+ queued the requested action.
+
+- Transfers for one URB can be canceled with :c:func:`usb_unlink_urb`
+ at any time.
+
+- Each URB has a completion handler, which is called after the action
+ has been successfully completed or canceled. The URB also contains a
+ context-pointer for passing information to the completion handler.
+
+- Each endpoint for a device logically supports a queue of requests.
+ You can fill that queue, so that the USB hardware can still transfer
+ data to an endpoint while your driver handles completion of another.
+ This maximizes use of USB bandwidth, and supports seamless streaming
+ of data to (or from) devices when using periodic transfer modes.
+
+
+The URB structure
+=================
+
+Some of the fields in struct :c:type:`urb` are::
+
+ struct urb
+ {
+ // (IN) device and pipe specify the endpoint queue
+ struct usb_device *dev; // pointer to associated USB device
+ unsigned int pipe; // endpoint information
+
+ unsigned int transfer_flags; // URB_ISO_ASAP, URB_SHORT_NOT_OK, etc.
+
+ // (IN) all urbs need completion routines
+ void *context; // context for completion routine
+ usb_complete_t complete; // pointer to completion routine
+
+ // (OUT) status after each completion
+ int status; // returned status
+
+ // (IN) buffer used for data transfers
+ void *transfer_buffer; // associated data buffer
+ u32 transfer_buffer_length; // data buffer length
+ int number_of_packets; // size of iso_frame_desc
+
+ // (OUT) sometimes only part of CTRL/BULK/INTR transfer_buffer is used
+ u32 actual_length; // actual data buffer length
+
+ // (IN) setup stage for CTRL (pass a struct usb_ctrlrequest)
+ unsigned char *setup_packet; // setup packet (control only)
+
+ // Only for PERIODIC transfers (ISO, INTERRUPT)
+ // (IN/OUT) start_frame is set unless URB_ISO_ASAP isn't set
+ int start_frame; // start frame
+ int interval; // polling interval
+
+ // ISO only: packets are only "best effort"; each can have errors
+ int error_count; // number of errors
+ struct usb_iso_packet_descriptor iso_frame_desc[0];
+ };
+
+Your driver must create the "pipe" value using values from the appropriate
+endpoint descriptor in an interface that it's claimed.
+
+
+How to get an URB?
+==================
+
+URBs are allocated by calling :c:func:`usb_alloc_urb`::
+
+ struct urb *usb_alloc_urb(int isoframes, int mem_flags)
+
+Return value is a pointer to the allocated URB, 0 if allocation failed.
+The parameter isoframes specifies the number of isochronous transfer frames
+you want to schedule. For CTRL/BULK/INT, use 0. The mem_flags parameter
+holds standard memory allocation flags, letting you control (among other
+things) whether the underlying code may block or not.
+
+To free an URB, use :c:func:`usb_free_urb`::
+
+ void usb_free_urb(struct urb *urb)
+
+You may free an urb that you've submitted, but which hasn't yet been
+returned to you in a completion callback. It will automatically be
+deallocated when it is no longer in use.
+
+
+What has to be filled in?
+=========================
+
+Depending on the type of transaction, there are some inline functions
+defined in ``linux/usb.h`` to simplify the initialization, such as
+:c:func:`usb_fill_control_urb`, :c:func:`usb_fill_bulk_urb` and
+:c:func:`usb_fill_int_urb`. In general, they need the usb device pointer,
+the pipe (usual format from usb.h), the transfer buffer, the desired transfer
+length, the completion handler, and its context. Take a look at the some
+existing drivers to see how they're used.
+
+Flags:
+
+- For ISO there are two startup behaviors: Specified start_frame or ASAP.
+- For ASAP set ``URB_ISO_ASAP`` in transfer_flags.
+
+If short packets should NOT be tolerated, set ``URB_SHORT_NOT_OK`` in
+transfer_flags.
+
+
+How to submit an URB?
+=====================
+
+Just call :c:func:`usb_submit_urb`::
+
+ int usb_submit_urb(struct urb *urb, int mem_flags)
+
+The ``mem_flags`` parameter, such as ``GFP_ATOMIC``, controls memory
+allocation, such as whether the lower levels may block when memory is tight.
+
+It immediately returns, either with status 0 (request queued) or some
+error code, usually caused by the following:
+
+- Out of memory (``-ENOMEM``)
+- Unplugged device (``-ENODEV``)
+- Stalled endpoint (``-EPIPE``)
+- Too many queued ISO transfers (``-EAGAIN``)
+- Too many requested ISO frames (``-EFBIG``)
+- Invalid INT interval (``-EINVAL``)
+- More than one packet for INT (``-EINVAL``)
+
+After submission, ``urb->status`` is ``-EINPROGRESS``; however, you should
+never look at that value except in your completion callback.
+
+For isochronous endpoints, your completion handlers should (re)submit
+URBs to the same endpoint with the ``URB_ISO_ASAP`` flag, using
+multi-buffering, to get seamless ISO streaming.
+
+
+How to cancel an already running URB?
+=====================================
+
+There are two ways to cancel an URB you've submitted but which hasn't
+been returned to your driver yet. For an asynchronous cancel, call
+:c:func:`usb_unlink_urb`::
+
+ int usb_unlink_urb(struct urb *urb)
+
+It removes the urb from the internal list and frees all allocated
+HW descriptors. The status is changed to reflect unlinking. Note
+that the URB will not normally have finished when :c:func:`usb_unlink_urb`
+returns; you must still wait for the completion handler to be called.
+
+To cancel an URB synchronously, call :c:func:`usb_kill_urb`::
+
+ void usb_kill_urb(struct urb *urb)
+
+It does everything :c:func:`usb_unlink_urb` does, and in addition it waits
+until after the URB has been returned and the completion handler
+has finished. It also marks the URB as temporarily unusable, so
+that if the completion handler or anyone else tries to resubmit it
+they will get a ``-EPERM`` error. Thus you can be sure that when
+:c:func:`usb_kill_urb` returns, the URB is totally idle.
+
+There is a lifetime issue to consider. An URB may complete at any
+time, and the completion handler may free the URB. If this happens
+while :c:func:`usb_unlink_urb` or :c:func:`usb_kill_urb` is running, it will
+cause a memory-access violation. The driver is responsible for avoiding this,
+which often means some sort of lock will be needed to prevent the URB
+from being deallocated while it is still in use.
+
+On the other hand, since usb_unlink_urb may end up calling the
+completion handler, the handler must not take any lock that is held
+when usb_unlink_urb is invoked. The general solution to this problem
+is to increment the URB's reference count while holding the lock, then
+drop the lock and call usb_unlink_urb or usb_kill_urb, and then
+decrement the URB's reference count. You increment the reference
+count by calling :c:func`usb_get_urb`::
+
+ struct urb *usb_get_urb(struct urb *urb)
+
+(ignore the return value; it is the same as the argument) and
+decrement the reference count by calling :c:func:`usb_free_urb`. Of course,
+none of this is necessary if there's no danger of the URB being freed
+by the completion handler.
+
+
+What about the completion handler?
+==================================
+
+The handler is of the following type::
+
+ typedef void (*usb_complete_t)(struct urb *)
+
+I.e., it gets the URB that caused the completion call. In the completion
+handler, you should have a look at ``urb->status`` to detect any USB errors.
+Since the context parameter is included in the URB, you can pass
+information to the completion handler.
+
+Note that even when an error (or unlink) is reported, data may have been
+transferred. That's because USB transfers are packetized; it might take
+sixteen packets to transfer your 1KByte buffer, and ten of them might
+have transferred successfully before the completion was called.
+
+
+.. warning::
+
+ NEVER SLEEP IN A COMPLETION HANDLER.
+
+ These are often called in atomic context.
+
+In the current kernel, completion handlers run with local interrupts
+disabled, but in the future this will be changed, so don't assume that
+local IRQs are always disabled inside completion handlers.
+
+How to do isochronous (ISO) transfers?
+======================================
+
+Besides the fields present on a bulk transfer, for ISO, you also
+also have to set ``urb->interval`` to say how often to make transfers; it's
+often one per frame (which is once every microframe for highspeed devices).
+The actual interval used will be a power of two that's no bigger than what
+you specify. You can use the :c:func:`usb_fill_int_urb` macro to fill
+most ISO transfer fields.
+
+For ISO transfers you also have to fill a :c:type:`usb_iso_packet_descriptor`
+structure, allocated at the end of the URB by :c:func:`usb_alloc_urb`, for
+each packet you want to schedule.
+
+The :c:func:`usb_submit_urb` call modifies ``urb->interval`` to the implemented
+interval value that is less than or equal to the requested interval value. If
+``URB_ISO_ASAP`` scheduling is used, ``urb->start_frame`` is also updated.
+
+For each entry you have to specify the data offset for this frame (base is
+transfer_buffer), and the length you want to write/expect to read.
+After completion, actual_length contains the actual transferred length and
+status contains the resulting status for the ISO transfer for this frame.
+It is allowed to specify a varying length from frame to frame (e.g. for
+audio synchronisation/adaptive transfer rates). You can also use the length
+0 to omit one or more frames (striping).
+
+For scheduling you can choose your own start frame or ``URB_ISO_ASAP``. As
+explained earlier, if you always keep at least one URB queued and your
+completion keeps (re)submitting a later URB, you'll get smooth ISO streaming
+(if usb bandwidth utilization allows).
+
+If you specify your own start frame, make sure it's several frames in advance
+of the current frame. You might want this model if you're synchronizing
+ISO data with some other event stream.
+
+
+How to start interrupt (INT) transfers?
+=======================================
+
+Interrupt transfers, like isochronous transfers, are periodic, and happen
+in intervals that are powers of two (1, 2, 4 etc) units. Units are frames
+for full and low speed devices, and microframes for high speed ones.
+You can use the :c:func:`usb_fill_int_urb` macro to fill INT transfer fields.
+
+The :c:func:`usb_submit_urb` call modifies ``urb->interval`` to the implemented
+interval value that is less than or equal to the requested interval value.
+
+In Linux 2.6, unlike earlier versions, interrupt URBs are not automagically
+restarted when they complete. They end when the completion handler is
+called, just like other URBs. If you want an interrupt URB to be restarted,
+your completion handler must resubmit it.
+s
diff --git a/Documentation/usb/anchors.txt b/Documentation/driver-api/usb/anchors.rst
similarity index 74%
rename from Documentation/usb/anchors.txt
rename to Documentation/driver-api/usb/anchors.rst
index fe6a99a..4b248e6 100644
--- a/Documentation/usb/anchors.txt
+++ b/Documentation/driver-api/usb/anchors.rst
@@ -1,3 +1,6 @@
+USB Anchors
+~~~~~~~~~~~
+
What is anchor?
===============
@@ -13,7 +16,7 @@
=============================
There's no API to allocate an anchor. It is simply declared
-as struct usb_anchor. init_usb_anchor() must be called to
+as struct usb_anchor. :c:func:`init_usb_anchor` must be called to
initialise the data structure.
Deallocation
@@ -26,52 +29,53 @@
===================================================
An association of URBs to an anchor is made by an explicit
-call to usb_anchor_urb(). The association is maintained until
+call to :c:func:`usb_anchor_urb`. The association is maintained until
an URB is finished by (successful) completion. Thus disassociation
is automatic. A function is provided to forcibly finish (kill)
all URBs associated with an anchor.
-Furthermore, disassociation can be made with usb_unanchor_urb()
+Furthermore, disassociation can be made with :c:func:`usb_unanchor_urb`
Operations on multitudes of URBs
================================
-usb_kill_anchored_urbs()
-------------------------
+:c:func:`usb_kill_anchored_urbs`
+--------------------------------
This function kills all URBs associated with an anchor. The URBs
are called in the reverse temporal order they were submitted.
This way no data can be reordered.
-usb_unlink_anchored_urbs()
---------------------------
+:c:func:`usb_unlink_anchored_urbs`
+----------------------------------
+
This function unlinks all URBs associated with an anchor. The URBs
are processed in the reverse temporal order they were submitted.
-This is similar to usb_kill_anchored_urbs(), but it will not sleep.
+This is similar to :c:func:`usb_kill_anchored_urbs`, but it will not sleep.
Therefore no guarantee is made that the URBs have been unlinked when
the call returns. They may be unlinked later but will be unlinked in
finite time.
-usb_scuttle_anchored_urbs()
----------------------------
+:c:func:`usb_scuttle_anchored_urbs`
+-----------------------------------
All URBs of an anchor are unanchored en masse.
-usb_wait_anchor_empty_timeout()
--------------------------------
+:c:func:`usb_wait_anchor_empty_timeout`
+---------------------------------------
This function waits for all URBs associated with an anchor to finish
or a timeout, whichever comes first. Its return value will tell you
whether the timeout was reached.
-usb_anchor_empty()
-------------------
+:c:func:`usb_anchor_empty`
+--------------------------
Returns true if no URBs are associated with an anchor. Locking
is the caller's responsibility.
-usb_get_from_anchor()
----------------------
+:c:func:`usb_get_from_anchor`
+-----------------------------
Returns the oldest anchored URB of an anchor. The URB is unanchored
and returned with a reference. As you may mix URBs to several
diff --git a/Documentation/usb/bulk-streams.txt b/Documentation/driver-api/usb/bulk-streams.rst
similarity index 93%
rename from Documentation/usb/bulk-streams.txt
rename to Documentation/driver-api/usb/bulk-streams.rst
index ffc0202..99b515b 100644
--- a/Documentation/usb/bulk-streams.txt
+++ b/Documentation/driver-api/usb/bulk-streams.rst
@@ -1,3 +1,6 @@
+USB bulk streams
+~~~~~~~~~~~~~~~~
+
Background
==========
@@ -25,7 +28,9 @@
Driver implications
===================
-int usb_alloc_streams(struct usb_interface *interface,
+::
+
+ int usb_alloc_streams(struct usb_interface *interface,
struct usb_host_endpoint **eps, unsigned int num_eps,
unsigned int num_streams, gfp_t mem_flags);
@@ -53,7 +58,7 @@
Picking new Stream IDs to use
-============================
+=============================
Stream ID 0 is reserved, and should not be used to communicate with devices. If
usb_alloc_streams() returns with a value of N, you may use streams 1 though N.
@@ -68,9 +73,9 @@
========
If a driver wishes to stop using streams to communicate with the device, it
-should call
+should call::
-void usb_free_streams(struct usb_interface *interface,
+ void usb_free_streams(struct usb_interface *interface,
struct usb_host_endpoint **eps, unsigned int num_eps,
gfp_t mem_flags);
diff --git a/Documentation/usb/callbacks.txt b/Documentation/driver-api/usb/callbacks.rst
similarity index 75%
rename from Documentation/usb/callbacks.txt
rename to Documentation/driver-api/usb/callbacks.rst
index 9e85846..2b80cf5 100644
--- a/Documentation/usb/callbacks.txt
+++ b/Documentation/driver-api/usb/callbacks.rst
@@ -1,3 +1,6 @@
+USB core callbacks
+~~~~~~~~~~~~~~~~~~
+
What callbacks will usbcore do?
===============================
@@ -5,40 +8,52 @@
structure and through the completion handler of URBs a driver submits.
Only the former are in the scope of this document. These two kinds of
callbacks are completely independent of each other. Information on the
-completion callback can be found in Documentation/usb/URB.txt.
+completion callback can be found in :ref:`usb-urb`.
The callbacks defined in the driver structure are:
1. Hotplugging callbacks:
- * @probe: Called to see if the driver is willing to manage a particular
- * interface on a device.
- * @disconnect: Called when the interface is no longer accessible, usually
- * because its device has been (or is being) disconnected or the
- * driver module is being unloaded.
+ - @probe:
+ Called to see if the driver is willing to manage a particular
+ interface on a device.
+
+ - @disconnect:
+ Called when the interface is no longer accessible, usually
+ because its device has been (or is being) disconnected or the
+ driver module is being unloaded.
2. Odd backdoor through usbfs:
- * @ioctl: Used for drivers that want to talk to userspace through
- * the "usbfs" filesystem. This lets devices provide ways to
- * expose information to user space regardless of where they
- * do (or don't) show up otherwise in the filesystem.
+ - @ioctl:
+ Used for drivers that want to talk to userspace through
+ the "usbfs" filesystem. This lets devices provide ways to
+ expose information to user space regardless of where they
+ do (or don't) show up otherwise in the filesystem.
3. Power management (PM) callbacks:
- * @suspend: Called when the device is going to be suspended.
- * @resume: Called when the device is being resumed.
- * @reset_resume: Called when the suspended device has been reset instead
- * of being resumed.
+ - @suspend:
+ Called when the device is going to be suspended.
+
+ - @resume:
+ Called when the device is being resumed.
+
+ - @reset_resume:
+ Called when the suspended device has been reset instead
+ of being resumed.
4. Device level operations:
- * @pre_reset: Called when the device is about to be reset.
- * @post_reset: Called after the device has been reset
+ - @pre_reset:
+ Called when the device is about to be reset.
+
+ - @post_reset:
+ Called after the device has been reset
The ioctl interface (2) should be used only if you have a very good
reason. Sysfs is preferred these days. The PM callbacks are covered
-separately in Documentation/usb/power-management.txt.
+separately in :ref:`usb-power-management`.
Calling conventions
===================
@@ -58,7 +73,9 @@
The probe() callback
--------------------
-int (*probe) (struct usb_interface *intf,
+::
+
+ int (*probe) (struct usb_interface *intf,
const struct usb_device_id *id);
Accept or decline an interface. If you accept the device return 0,
@@ -75,7 +92,9 @@
The disconnect() callback
-------------------------
-void (*disconnect) (struct usb_interface *intf);
+::
+
+ void (*disconnect) (struct usb_interface *intf);
This callback is a signal to break any connection with an interface.
You are not allowed any IO to a device after returning from this
@@ -93,7 +112,9 @@
pre_reset
---------
-int (*pre_reset)(struct usb_interface *intf);
+::
+
+ int (*pre_reset)(struct usb_interface *intf);
A driver or user space is triggering a reset on the device which
contains the interface passed as an argument. Cease IO, wait for all
@@ -107,7 +128,9 @@
post_reset
----------
-int (*post_reset)(struct usb_interface *intf);
+::
+
+ int (*post_reset)(struct usb_interface *intf);
The reset has completed. Restore any saved device state and begin
using the device again.
diff --git a/Documentation/usb/dma.txt b/Documentation/driver-api/usb/dma.rst
similarity index 79%
rename from Documentation/usb/dma.txt
rename to Documentation/driver-api/usb/dma.rst
index 444651e..59d5aee 100644
--- a/Documentation/usb/dma.txt
+++ b/Documentation/driver-api/usb/dma.rst
@@ -1,16 +1,19 @@
+USB DMA
+~~~~~~~
+
In Linux 2.5 kernels (and later), USB device drivers have additional control
over how DMA may be used to perform I/O operations. The APIs are detailed
in the kernel usb programming guide (kerneldoc, from the source code).
-
-API OVERVIEW
+API overview
+============
The big picture is that USB drivers can continue to ignore most DMA issues,
though they still must provide DMA-ready buffers (see
-Documentation/DMA-API-HOWTO.txt). That's how they've worked through
-the 2.4 (and earlier) kernels.
+``Documentation/DMA-API-HOWTO.txt``). That's how they've worked through
+the 2.4 (and earlier) kernels, or they can now be DMA-aware.
-OR: they can now be DMA-aware.
+DMA-aware usb drivers:
- New calls enable DMA-aware drivers, letting them allocate dma buffers and
manage dma mappings for existing dma-ready buffers (see below).
@@ -20,15 +23,15 @@
drivers must not use it.)
- "usbcore" will map this DMA address, if a DMA-aware driver didn't do
- it first and set URB_NO_TRANSFER_DMA_MAP. HCDs
+ it first and set ``URB_NO_TRANSFER_DMA_MAP``. HCDs
don't manage dma mappings for URBs.
- There's a new "generic DMA API", parts of which are usable by USB device
drivers. Never use dma_set_mask() on any USB interface or device; that
would potentially break all devices sharing that bus.
-
-ELIMINATING COPIES
+Eliminating copies
+==================
It's good to avoid making CPUs copy data needlessly. The costs can add up,
and effects like cache-trashing can impose subtle penalties.
@@ -41,7 +44,7 @@
For those specific cases, USB has primitives to allocate less expensive
memory. They work like kmalloc and kfree versions that give you the right
kind of addresses to store in urb->transfer_buffer and urb->transfer_dma.
- You'd also set URB_NO_TRANSFER_DMA_MAP in urb->transfer_flags:
+ You'd also set ``URB_NO_TRANSFER_DMA_MAP`` in urb->transfer_flags::
void *usb_alloc_coherent (struct usb_device *dev, size_t size,
int mem_flags, dma_addr_t *dma);
@@ -49,15 +52,15 @@
void usb_free_coherent (struct usb_device *dev, size_t size,
void *addr, dma_addr_t dma);
- Most drivers should *NOT* be using these primitives; they don't need
+ Most drivers should **NOT** be using these primitives; they don't need
to use this type of memory ("dma-coherent"), and memory returned from
- kmalloc() will work just fine.
+ :c:func:`kmalloc` will work just fine.
The memory buffer returned is "dma-coherent"; sometimes you might need to
force a consistent memory access ordering by using memory barriers. It's
not using a streaming DMA mapping, so it's good for small transfers on
systems where the I/O would otherwise thrash an IOMMU mapping. (See
- Documentation/DMA-API-HOWTO.txt for definitions of "coherent" and
+ ``Documentation/DMA-API-HOWTO.txt`` for definitions of "coherent" and
"streaming" DMA mappings.)
Asking for 1/Nth of a page (as well as asking for N pages) is reasonably
@@ -75,15 +78,15 @@
way to expose these capabilities ... and in any case, HIGHMEM is mostly a
design wart specific to x86_32. So your best bet is to ensure you never
pass a highmem buffer into a USB driver. That's easy; it's the default
- behavior. Just don't override it; e.g. with NETIF_F_HIGHDMA.
+ behavior. Just don't override it; e.g. with ``NETIF_F_HIGHDMA``.
This may force your callers to do some bounce buffering, copying from
high memory to "normal" DMA memory. If you can come up with a good way
to fix this issue (for x86_32 machines with over 1 GByte of memory),
feel free to submit patches.
-
-WORKING WITH EXISTING BUFFERS
+Working with existing buffers
+=============================
Existing buffers aren't usable for DMA without first being mapped into the
DMA address space of the device. However, most buffers passed to your
@@ -92,7 +95,7 @@
- When you're using scatterlists, you can map everything at once. On some
systems, this kicks in an IOMMU and turns the scatterlists into single
- DMA transactions:
+ DMA transactions::
int usb_buffer_map_sg (struct usb_device *dev, unsigned pipe,
struct scatterlist *sg, int nents);
@@ -103,7 +106,7 @@
void usb_buffer_unmap_sg (struct usb_device *dev, unsigned pipe,
struct scatterlist *sg, int n_hw_ents);
- It's probably easier to use the new usb_sg_*() calls, which do the DMA
+ It's probably easier to use the new ``usb_sg_*()`` calls, which do the DMA
mapping and apply other tweaks to make scatterlist i/o be fast.
- Some drivers may prefer to work with the model that they're mapping large
@@ -112,10 +115,10 @@
here, since it's cheaper to just synchronize the buffer than to unmap it
each time an urb completes and then re-map it on during resubmission.
- These calls all work with initialized urbs: urb->dev, urb->pipe,
- urb->transfer_buffer, and urb->transfer_buffer_length must all be
- valid when these calls are used (urb->setup_packet must be valid too
- if urb is a control request):
+ These calls all work with initialized urbs: ``urb->dev``, ``urb->pipe``,
+ ``urb->transfer_buffer``, and ``urb->transfer_buffer_length`` must all be
+ valid when these calls are used (``urb->setup_packet`` must be valid too
+ if urb is a control request)::
struct urb *usb_buffer_map (struct urb *urb);
@@ -123,9 +126,9 @@
void usb_buffer_unmap (struct urb *urb);
- The calls manage urb->transfer_dma for you, and set URB_NO_TRANSFER_DMA_MAP
- so that usbcore won't map or unmap the buffer. They cannot be used for
- setup_packet buffers in control requests.
+ The calls manage ``urb->transfer_dma`` for you, and set
+ ``URB_NO_TRANSFER_DMA_MAP`` so that usbcore won't map or unmap the buffer.
+ They cannot be used for setup_packet buffers in control requests.
Note that several of those interfaces are currently commented out, since
they don't have current users. See the source code. Other than the dmasync
diff --git a/Documentation/driver-api/usb/error-codes.rst b/Documentation/driver-api/usb/error-codes.rst
new file mode 100644
index 0000000..a3e84bf
--- /dev/null
+++ b/Documentation/driver-api/usb/error-codes.rst
@@ -0,0 +1,207 @@
+.. _usb-error-codes:
+
+USB Error codes
+~~~~~~~~~~~~~~~
+
+:Revised: 2004-Oct-21
+
+This is the documentation of (hopefully) all possible error codes (and
+their interpretation) that can be returned from usbcore.
+
+Some of them are returned by the Host Controller Drivers (HCDs), which
+device drivers only see through usbcore. As a rule, all the HCDs should
+behave the same except for transfer speed dependent behaviors and the
+way certain faults are reported.
+
+
+Error codes returned by :c:func:`usb_submit_urb`
+================================================
+
+Non-USB-specific:
+
+
+=============== ===============================================
+0 URB submission went fine
+
+``-ENOMEM`` no memory for allocation of internal structures
+=============== ===============================================
+
+USB-specific:
+
+======================= =======================================================
+``-EBUSY`` The URB is already active.
+
+``-ENODEV`` specified USB-device or bus doesn't exist
+
+``-ENOENT`` specified interface or endpoint does not exist or
+ is not enabled
+
+``-ENXIO`` host controller driver does not support queuing of
+ this type of urb. (treat as a host controller bug.)
+
+``-EINVAL`` a) Invalid transfer type specified (or not supported)
+ b) Invalid or unsupported periodic transfer interval
+ c) ISO: attempted to change transfer interval
+ d) ISO: ``number_of_packets`` is < 0
+ e) various other cases
+
+``-EXDEV`` ISO: ``URB_ISO_ASAP`` wasn't specified and all the
+ frames the URB would be scheduled in have already
+ expired.
+
+``-EFBIG`` Host controller driver can't schedule that many ISO
+ frames.
+
+``-EPIPE`` The pipe type specified in the URB doesn't match the
+ endpoint's actual type.
+
+``-EMSGSIZE`` (a) endpoint maxpacket size is zero; it is not usable
+ in the current interface altsetting.
+ (b) ISO packet is larger than the endpoint maxpacket.
+ (c) requested data transfer length is invalid: negative
+ or too large for the host controller.
+
+``-ENOSPC`` This request would overcommit the usb bandwidth reserved
+ for periodic transfers (interrupt, isochronous).
+
+``-ESHUTDOWN`` The device or host controller has been disabled due to
+ some problem that could not be worked around.
+
+``-EPERM`` Submission failed because ``urb->reject`` was set.
+
+``-EHOSTUNREACH`` URB was rejected because the device is suspended.
+
+``-ENOEXEC`` A control URB doesn't contain a Setup packet.
+======================= =======================================================
+
+Error codes returned by ``in urb->status`` or in ``iso_frame_desc[n].status`` (for ISO)
+=======================================================================================
+
+USB device drivers may only test urb status values in completion handlers.
+This is because otherwise there would be a race between HCDs updating
+these values on one CPU, and device drivers testing them on another CPU.
+
+A transfer's actual_length may be positive even when an error has been
+reported. That's because transfers often involve several packets, so that
+one or more packets could finish before an error stops further endpoint I/O.
+
+For isochronous URBs, the urb status value is non-zero only if the URB is
+unlinked, the device is removed, the host controller is disabled, or the total
+transferred length is less than the requested length and the
+``URB_SHORT_NOT_OK`` flag is set. Completion handlers for isochronous URBs
+should only see ``urb->status`` set to zero, ``-ENOENT``, ``-ECONNRESET``,
+``-ESHUTDOWN``, or ``-EREMOTEIO``. Individual frame descriptor status fields
+may report more status codes.
+
+
+=============================== ===============================================
+0 Transfer completed successfully
+
+``-ENOENT`` URB was synchronously unlinked by
+ :c:func:`usb_unlink_urb`
+
+``-EINPROGRESS`` URB still pending, no results yet
+ (That is, if drivers see this it's a bug.)
+
+``-EPROTO`` [#f1]_, [#f2]_ a) bitstuff error
+ b) no response packet received within the
+ prescribed bus turn-around time
+ c) unknown USB error
+
+``-EILSEQ`` [#f1]_, [#f2]_ a) CRC mismatch
+ b) no response packet received within the
+ prescribed bus turn-around time
+ c) unknown USB error
+
+ Note that often the controller hardware does
+ not distinguish among cases a), b), and c), so
+ a driver cannot tell whether there was a
+ protocol error, a failure to respond (often
+ caused by device disconnect), or some other
+ fault.
+
+``-ETIME`` [#f2]_ No response packet received within the
+ prescribed bus turn-around time. This error
+ may instead be reported as
+ ``-EPROTO`` or ``-EILSEQ``.
+
+``-ETIMEDOUT`` Synchronous USB message functions use this code
+ to indicate timeout expired before the transfer
+ completed, and no other error was reported
+ by HC.
+
+``-EPIPE`` [#f2]_ Endpoint stalled. For non-control endpoints,
+ reset this status with
+ :c:func:`usb_clear_halt`.
+
+``-ECOMM`` During an IN transfer, the host controller
+ received data from an endpoint faster than it
+ could be written to system memory
+
+``-ENOSR`` During an OUT transfer, the host controller
+ could not retrieve data from system memory fast
+ enough to keep up with the USB data rate
+
+``-EOVERFLOW`` [#f1]_ The amount of data returned by the endpoint was
+ greater than either the max packet size of the
+ endpoint or the remaining buffer size.
+ "Babble".
+
+``-EREMOTEIO`` The data read from the endpoint did not fill
+ the specified buffer, and ``URB_SHORT_NOT_OK``
+ was set in ``urb->transfer_flags``.
+
+``-ENODEV`` Device was removed. Often preceded by a burst
+ of other errors, since the hub driver doesn't
+ detect device removal events immediately.
+
+``-EXDEV`` ISO transfer only partially completed
+ (only set in ``iso_frame_desc[n].status``,
+ not ``urb->status``)
+
+``-EINVAL`` ISO madness, if this happens: Log off and
+ go home
+
+``-ECONNRESET`` URB was asynchronously unlinked by
+ :c:func:`usb_unlink_urb`
+
+``-ESHUTDOWN`` The device or host controller has been
+ disabled due to some problem that could not
+ be worked around, such as a physical
+ disconnect.
+=============================== ===============================================
+
+
+.. [#f1]
+
+ Error codes like ``-EPROTO``, ``-EILSEQ`` and ``-EOVERFLOW`` normally
+ indicate hardware problems such as bad devices (including firmware)
+ or cables.
+
+.. [#f2]
+
+ This is also one of several codes that different kinds of host
+ controller use to indicate a transfer has failed because of device
+ disconnect. In the interval before the hub driver starts disconnect
+ processing, devices may receive such fault reports for every request.
+
+
+
+Error codes returned by usbcore-functions
+=========================================
+
+.. note:: expect also other submit and transfer status codes
+
+:c:func:`usb_register`:
+
+======================= ===================================
+``-EINVAL`` error during registering new driver
+======================= ===================================
+
+``usb_get_*/usb_set_*()``,
+:c:func:`usb_control_msg`,
+:c:func:`usb_bulk_msg()`:
+
+======================= ==============================================
+``-ETIMEDOUT`` Timeout expired before the transfer completed.
+======================= ==============================================
diff --git a/Documentation/driver-api/usb/gadget.rst b/Documentation/driver-api/usb/gadget.rst
new file mode 100644
index 0000000..3e8a380
--- /dev/null
+++ b/Documentation/driver-api/usb/gadget.rst
@@ -0,0 +1,510 @@
+========================
+USB Gadget API for Linux
+========================
+
+:Author: David Brownell
+:Date: 20 August 2004
+
+Introduction
+============
+
+This document presents a Linux-USB "Gadget" kernel mode API, for use
+within peripherals and other USB devices that embed Linux. It provides
+an overview of the API structure, and shows how that fits into a system
+development project. This is the first such API released on Linux to
+address a number of important problems, including:
+
+- Supports USB 2.0, for high speed devices which can stream data at
+ several dozen megabytes per second.
+
+- Handles devices with dozens of endpoints just as well as ones with
+ just two fixed-function ones. Gadget drivers can be written so
+ they're easy to port to new hardware.
+
+- Flexible enough to expose more complex USB device capabilities such
+ as multiple configurations, multiple interfaces, composite devices,
+ and alternate interface settings.
+
+- USB "On-The-Go" (OTG) support, in conjunction with updates to the
+ Linux-USB host side.
+
+- Sharing data structures and API models with the Linux-USB host side
+ API. This helps the OTG support, and looks forward to more-symmetric
+ frameworks (where the same I/O model is used by both host and device
+ side drivers).
+
+- Minimalist, so it's easier to support new device controller hardware.
+ I/O processing doesn't imply large demands for memory or CPU
+ resources.
+
+Most Linux developers will not be able to use this API, since they have
+USB ``host`` hardware in a PC, workstation, or server. Linux users with
+embedded systems are more likely to have USB peripheral hardware. To
+distinguish drivers running inside such hardware from the more familiar
+Linux "USB device drivers", which are host side proxies for the real USB
+devices, a different term is used: the drivers inside the peripherals
+are "USB gadget drivers". In USB protocol interactions, the device
+driver is the master (or "client driver") and the gadget driver is the
+slave (or "function driver").
+
+The gadget API resembles the host side Linux-USB API in that both use
+queues of request objects to package I/O buffers, and those requests may
+be submitted or canceled. They share common definitions for the standard
+USB *Chapter 9* messages, structures, and constants. Also, both APIs
+bind and unbind drivers to devices. The APIs differ in detail, since the
+host side's current URB framework exposes a number of implementation
+details and assumptions that are inappropriate for a gadget API. While
+the model for control transfers and configuration management is
+necessarily different (one side is a hardware-neutral master, the other
+is a hardware-aware slave), the endpoint I/0 API used here should also
+be usable for an overhead-reduced host side API.
+
+Structure of Gadget Drivers
+===========================
+
+A system running inside a USB peripheral normally has at least three
+layers inside the kernel to handle USB protocol processing, and may have
+additional layers in user space code. The ``gadget`` API is used by the
+middle layer to interact with the lowest level (which directly handles
+hardware).
+
+In Linux, from the bottom up, these layers are:
+
+*USB Controller Driver*
+ This is the lowest software level. It is the only layer that talks
+ to hardware, through registers, fifos, dma, irqs, and the like. The
+ ``<linux/usb/gadget.h>`` API abstracts the peripheral controller
+ endpoint hardware. That hardware is exposed through endpoint
+ objects, which accept streams of IN/OUT buffers, and through
+ callbacks that interact with gadget drivers. Since normal USB
+ devices only have one upstream port, they only have one of these
+ drivers. The controller driver can support any number of different
+ gadget drivers, but only one of them can be used at a time.
+
+ Examples of such controller hardware include the PCI-based NetChip
+ 2280 USB 2.0 high speed controller, the SA-11x0 or PXA-25x UDC
+ (found within many PDAs), and a variety of other products.
+
+*Gadget Driver*
+ The lower boundary of this driver implements hardware-neutral USB
+ functions, using calls to the controller driver. Because such
+ hardware varies widely in capabilities and restrictions, and is used
+ in embedded environments where space is at a premium, the gadget
+ driver is often configured at compile time to work with endpoints
+ supported by one particular controller. Gadget drivers may be
+ portable to several different controllers, using conditional
+ compilation. (Recent kernels substantially simplify the work
+ involved in supporting new hardware, by *autoconfiguring* endpoints
+ automatically for many bulk-oriented drivers.) Gadget driver
+ responsibilities include:
+
+ - handling setup requests (ep0 protocol responses) possibly
+ including class-specific functionality
+
+ - returning configuration and string descriptors
+
+ - (re)setting configurations and interface altsettings, including
+ enabling and configuring endpoints
+
+ - handling life cycle events, such as managing bindings to
+ hardware, USB suspend/resume, remote wakeup, and disconnection
+ from the USB host.
+
+ - managing IN and OUT transfers on all currently enabled endpoints
+
+ Such drivers may be modules of proprietary code, although that
+ approach is discouraged in the Linux community.
+
+*Upper Level*
+ Most gadget drivers have an upper boundary that connects to some
+ Linux driver or framework in Linux. Through that boundary flows the
+ data which the gadget driver produces and/or consumes through
+ protocol transfers over USB. Examples include:
+
+ - user mode code, using generic (gadgetfs) or application specific
+ files in ``/dev``
+
+ - networking subsystem (for network gadgets, like the CDC Ethernet
+ Model gadget driver)
+
+ - data capture drivers, perhaps video4Linux or a scanner driver; or
+ test and measurement hardware.
+
+ - input subsystem (for HID gadgets)
+
+ - sound subsystem (for audio gadgets)
+
+ - file system (for PTP gadgets)
+
+ - block i/o subsystem (for usb-storage gadgets)
+
+ - ... and more
+
+*Additional Layers*
+ Other layers may exist. These could include kernel layers, such as
+ network protocol stacks, as well as user mode applications building
+ on standard POSIX system call APIs such as ``open()``, ``close()``,
+ ``read()`` and ``write()``. On newer systems, POSIX Async I/O calls may
+ be an option. Such user mode code will not necessarily be subject to
+ the GNU General Public License (GPL).
+
+OTG-capable systems will also need to include a standard Linux-USB host
+side stack, with ``usbcore``, one or more *Host Controller Drivers*
+(HCDs), *USB Device Drivers* to support the OTG "Targeted Peripheral
+List", and so forth. There will also be an *OTG Controller Driver*,
+which is visible to gadget and device driver developers only indirectly.
+That helps the host and device side USB controllers implement the two
+new OTG protocols (HNP and SRP). Roles switch (host to peripheral, or
+vice versa) using HNP during USB suspend processing, and SRP can be
+viewed as a more battery-friendly kind of device wakeup protocol.
+
+Over time, reusable utilities are evolving to help make some gadget
+driver tasks simpler. For example, building configuration descriptors
+from vectors of descriptors for the configurations interfaces and
+endpoints is now automated, and many drivers now use autoconfiguration
+to choose hardware endpoints and initialize their descriptors. A
+potential example of particular interest is code implementing standard
+USB-IF protocols for HID, networking, storage, or audio classes. Some
+developers are interested in KDB or KGDB hooks, to let target hardware
+be remotely debugged. Most such USB protocol code doesn't need to be
+hardware-specific, any more than network protocols like X11, HTTP, or
+NFS are. Such gadget-side interface drivers should eventually be
+combined, to implement composite devices.
+
+Kernel Mode Gadget API
+======================
+
+Gadget drivers declare themselves through a struct
+:c:type:`usb_gadget_driver`, which is responsible for most parts of enumeration
+for a struct :c:type:`usb_gadget`. The response to a set_configuration usually
+involves enabling one or more of the struct :c:type:`usb_ep` objects exposed by
+the gadget, and submitting one or more struct :c:type:`usb_request` buffers to
+transfer data. Understand those four data types, and their operations,
+and you will understand how this API works.
+
+.. Note::
+
+ Other than the "Chapter 9" data types, most of the significant data
+ types and functions are described here.
+
+ However, some relevant information is likely omitted from what you
+ are reading. One example of such information is endpoint
+ autoconfiguration. You'll have to read the header file, and use
+ example source code (such as that for "Gadget Zero"), to fully
+ understand the API.
+
+ The part of the API implementing some basic driver capabilities is
+ specific to the version of the Linux kernel that's in use. The 2.6
+ and upper kernel versions include a *driver model* framework that has
+ no analogue on earlier kernels; so those parts of the gadget API are
+ not fully portable. (They are implemented on 2.4 kernels, but in a
+ different way.) The driver model state is another part of this API that is
+ ignored by the kerneldoc tools.
+
+The core API does not expose every possible hardware feature, only the
+most widely available ones. There are significant hardware features,
+such as device-to-device DMA (without temporary storage in a memory
+buffer) that would be added using hardware-specific APIs.
+
+This API allows drivers to use conditional compilation to handle
+endpoint capabilities of different hardware, but doesn't require that.
+Hardware tends to have arbitrary restrictions, relating to transfer
+types, addressing, packet sizes, buffering, and availability. As a rule,
+such differences only matter for "endpoint zero" logic that handles
+device configuration and management. The API supports limited run-time
+detection of capabilities, through naming conventions for endpoints.
+Many drivers will be able to at least partially autoconfigure
+themselves. In particular, driver init sections will often have endpoint
+autoconfiguration logic that scans the hardware's list of endpoints to
+find ones matching the driver requirements (relying on those
+conventions), to eliminate some of the most common reasons for
+conditional compilation.
+
+Like the Linux-USB host side API, this API exposes the "chunky" nature
+of USB messages: I/O requests are in terms of one or more "packets", and
+packet boundaries are visible to drivers. Compared to RS-232 serial
+protocols, USB resembles synchronous protocols like HDLC (N bytes per
+frame, multipoint addressing, host as the primary station and devices as
+secondary stations) more than asynchronous ones (tty style: 8 data bits
+per frame, no parity, one stop bit). So for example the controller
+drivers won't buffer two single byte writes into a single two-byte USB
+IN packet, although gadget drivers may do so when they implement
+protocols where packet boundaries (and "short packets") are not
+significant.
+
+Driver Life Cycle
+-----------------
+
+Gadget drivers make endpoint I/O requests to hardware without needing to
+know many details of the hardware, but driver setup/configuration code
+needs to handle some differences. Use the API like this:
+
+1. Register a driver for the particular device side usb controller
+ hardware, such as the net2280 on PCI (USB 2.0), sa11x0 or pxa25x as
+ found in Linux PDAs, and so on. At this point the device is logically
+ in the USB ch9 initial state (``attached``), drawing no power and not
+ usable (since it does not yet support enumeration). Any host should
+ not see the device, since it's not activated the data line pullup
+ used by the host to detect a device, even if VBUS power is available.
+
+2. Register a gadget driver that implements some higher level device
+ function. That will then bind() to a :c:type:`usb_gadget`, which activates
+ the data line pullup sometime after detecting VBUS.
+
+3. The hardware driver can now start enumerating. The steps it handles
+ are to accept USB ``power`` and ``set_address`` requests. Other steps are
+ handled by the gadget driver. If the gadget driver module is unloaded
+ before the host starts to enumerate, steps before step 7 are skipped.
+
+4. The gadget driver's ``setup()`` call returns usb descriptors, based both
+ on what the bus interface hardware provides and on the functionality
+ being implemented. That can involve alternate settings or
+ configurations, unless the hardware prevents such operation. For OTG
+ devices, each configuration descriptor includes an OTG descriptor.
+
+5. The gadget driver handles the last step of enumeration, when the USB
+ host issues a ``set_configuration`` call. It enables all endpoints used
+ in that configuration, with all interfaces in their default settings.
+ That involves using a list of the hardware's endpoints, enabling each
+ endpoint according to its descriptor. It may also involve using
+ ``usb_gadget_vbus_draw`` to let more power be drawn from VBUS, as
+ allowed by that configuration. For OTG devices, setting a
+ configuration may also involve reporting HNP capabilities through a
+ user interface.
+
+6. Do real work and perform data transfers, possibly involving changes
+ to interface settings or switching to new configurations, until the
+ device is disconnect()ed from the host. Queue any number of transfer
+ requests to each endpoint. It may be suspended and resumed several
+ times before being disconnected. On disconnect, the drivers go back
+ to step 3 (above).
+
+7. When the gadget driver module is being unloaded, the driver unbind()
+ callback is issued. That lets the controller driver be unloaded.
+
+Drivers will normally be arranged so that just loading the gadget driver
+module (or statically linking it into a Linux kernel) allows the
+peripheral device to be enumerated, but some drivers will defer
+enumeration until some higher level component (like a user mode daemon)
+enables it. Note that at this lowest level there are no policies about
+how ep0 configuration logic is implemented, except that it should obey
+USB specifications. Such issues are in the domain of gadget drivers,
+including knowing about implementation constraints imposed by some USB
+controllers or understanding that composite devices might happen to be
+built by integrating reusable components.
+
+Note that the lifecycle above can be slightly different for OTG devices.
+Other than providing an additional OTG descriptor in each configuration,
+only the HNP-related differences are particularly visible to driver
+code. They involve reporting requirements during the ``SET_CONFIGURATION``
+request, and the option to invoke HNP during some suspend callbacks.
+Also, SRP changes the semantics of ``usb_gadget_wakeup`` slightly.
+
+USB 2.0 Chapter 9 Types and Constants
+-------------------------------------
+
+Gadget drivers rely on common USB structures and constants defined in
+the :ref:`linux/usb/ch9.h <usb_chapter9>` header file, which is standard in
+Linux 2.6+ kernels. These are the same types and constants used by host side
+drivers (and usbcore).
+
+Core Objects and Methods
+------------------------
+
+These are declared in ``<linux/usb/gadget.h>``, and are used by gadget
+drivers to interact with USB peripheral controller drivers.
+
+.. kernel-doc:: include/linux/usb/gadget.h
+ :internal:
+
+Optional Utilities
+------------------
+
+The core API is sufficient for writing a USB Gadget Driver, but some
+optional utilities are provided to simplify common tasks. These
+utilities include endpoint autoconfiguration.
+
+.. kernel-doc:: drivers/usb/gadget/usbstring.c
+ :export:
+
+.. kernel-doc:: drivers/usb/gadget/config.c
+ :export:
+
+Composite Device Framework
+--------------------------
+
+The core API is sufficient for writing drivers for composite USB devices
+(with more than one function in a given configuration), and also
+multi-configuration devices (also more than one function, but not
+necessarily sharing a given configuration). There is however an optional
+framework which makes it easier to reuse and combine functions.
+
+Devices using this framework provide a struct :c:type:`usb_composite_driver`,
+which in turn provides one or more struct :c:type:`usb_configuration`
+instances. Each such configuration includes at least one struct
+:c:type:`usb_function`, which packages a user visible role such as "network
+link" or "mass storage device". Management functions may also exist,
+such as "Device Firmware Upgrade".
+
+.. kernel-doc:: include/linux/usb/composite.h
+ :internal:
+
+.. kernel-doc:: drivers/usb/gadget/composite.c
+ :export:
+
+Composite Device Functions
+--------------------------
+
+At this writing, a few of the current gadget drivers have been converted
+to this framework. Near-term plans include converting all of them,
+except for ``gadgetfs``.
+
+Peripheral Controller Drivers
+=============================
+
+The first hardware supporting this API was the NetChip 2280 controller,
+which supports USB 2.0 high speed and is based on PCI. This is the
+``net2280`` driver module. The driver supports Linux kernel versions 2.4
+and 2.6; contact NetChip Technologies for development boards and product
+information.
+
+Other hardware working in the ``gadget`` framework includes: Intel's PXA
+25x and IXP42x series processors (``pxa2xx_udc``), Toshiba TC86c001
+"Goku-S" (``goku_udc``), Renesas SH7705/7727 (``sh_udc``), MediaQ 11xx
+(``mq11xx_udc``), Hynix HMS30C7202 (``h7202_udc``), National 9303/4
+(``n9604_udc``), Texas Instruments OMAP (``omap_udc``), Sharp LH7A40x
+(``lh7a40x_udc``), and more. Most of those are full speed controllers.
+
+At this writing, there are people at work on drivers in this framework
+for several other USB device controllers, with plans to make many of
+them be widely available.
+
+A partial USB simulator, the ``dummy_hcd`` driver, is available. It can
+act like a net2280, a pxa25x, or an sa11x0 in terms of available
+endpoints and device speeds; and it simulates control, bulk, and to some
+extent interrupt transfers. That lets you develop some parts of a gadget
+driver on a normal PC, without any special hardware, and perhaps with
+the assistance of tools such as GDB running with User Mode Linux. At
+least one person has expressed interest in adapting that approach,
+hooking it up to a simulator for a microcontroller. Such simulators can
+help debug subsystems where the runtime hardware is unfriendly to
+software development, or is not yet available.
+
+Support for other controllers is expected to be developed and
+contributed over time, as this driver framework evolves.
+
+Gadget Drivers
+==============
+
+In addition to *Gadget Zero* (used primarily for testing and development
+with drivers for usb controller hardware), other gadget drivers exist.
+
+There's an ``ethernet`` gadget driver, which implements one of the most
+useful *Communications Device Class* (CDC) models. One of the standards
+for cable modem interoperability even specifies the use of this ethernet
+model as one of two mandatory options. Gadgets using this code look to a
+USB host as if they're an Ethernet adapter. It provides access to a
+network where the gadget's CPU is one host, which could easily be
+bridging, routing, or firewalling access to other networks. Since some
+hardware can't fully implement the CDC Ethernet requirements, this
+driver also implements a "good parts only" subset of CDC Ethernet. (That
+subset doesn't advertise itself as CDC Ethernet, to avoid creating
+problems.)
+
+Support for Microsoft's ``RNDIS`` protocol has been contributed by
+Pengutronix and Auerswald GmbH. This is like CDC Ethernet, but it runs
+on more slightly USB hardware (but less than the CDC subset). However,
+its main claim to fame is being able to connect directly to recent
+versions of Windows, using drivers that Microsoft bundles and supports,
+making it much simpler to network with Windows.
+
+There is also support for user mode gadget drivers, using ``gadgetfs``.
+This provides a *User Mode API* that presents each endpoint as a single
+file descriptor. I/O is done using normal ``read()`` and ``read()`` calls.
+Familiar tools like GDB and pthreads can be used to develop and debug
+user mode drivers, so that once a robust controller driver is available
+many applications for it won't require new kernel mode software. Linux
+2.6 *Async I/O (AIO)* support is available, so that user mode software
+can stream data with only slightly more overhead than a kernel driver.
+
+There's a USB Mass Storage class driver, which provides a different
+solution for interoperability with systems such as MS-Windows and MacOS.
+That *Mass Storage* driver uses a file or block device as backing store
+for a drive, like the ``loop`` driver. The USB host uses the BBB, CB, or
+CBI versions of the mass storage class specification, using transparent
+SCSI commands to access the data from the backing store.
+
+There's a "serial line" driver, useful for TTY style operation over USB.
+The latest version of that driver supports CDC ACM style operation, like
+a USB modem, and so on most hardware it can interoperate easily with
+MS-Windows. One interesting use of that driver is in boot firmware (like
+a BIOS), which can sometimes use that model with very small systems
+without real serial lines.
+
+Support for other kinds of gadget is expected to be developed and
+contributed over time, as this driver framework evolves.
+
+USB On-The-GO (OTG)
+===================
+
+USB OTG support on Linux 2.6 was initially developed by Texas
+Instruments for `OMAP <http://www.omap.com>`__ 16xx and 17xx series
+processors. Other OTG systems should work in similar ways, but the
+hardware level details could be very different.
+
+Systems need specialized hardware support to implement OTG, notably
+including a special *Mini-AB* jack and associated transceiver to support
+*Dual-Role* operation: they can act either as a host, using the standard
+Linux-USB host side driver stack, or as a peripheral, using this
+``gadget`` framework. To do that, the system software relies on small
+additions to those programming interfaces, and on a new internal
+component (here called an "OTG Controller") affecting which driver stack
+connects to the OTG port. In each role, the system can re-use the
+existing pool of hardware-neutral drivers, layered on top of the
+controller driver interfaces (:c:type:`usb_bus` or :c:type:`usb_gadget`).
+Such drivers need at most minor changes, and most of the calls added to
+support OTG can also benefit non-OTG products.
+
+- Gadget drivers test the ``is_otg`` flag, and use it to determine
+ whether or not to include an OTG descriptor in each of their
+ configurations.
+
+- Gadget drivers may need changes to support the two new OTG protocols,
+ exposed in new gadget attributes such as ``b_hnp_enable`` flag. HNP
+ support should be reported through a user interface (two LEDs could
+ suffice), and is triggered in some cases when the host suspends the
+ peripheral. SRP support can be user-initiated just like remote
+ wakeup, probably by pressing the same button.
+
+- On the host side, USB device drivers need to be taught to trigger HNP
+ at appropriate moments, using ``usb_suspend_device()``. That also
+ conserves battery power, which is useful even for non-OTG
+ configurations.
+
+- Also on the host side, a driver must support the OTG "Targeted
+ Peripheral List". That's just a whitelist, used to reject peripherals
+ not supported with a given Linux OTG host. *This whitelist is
+ product-specific; each product must modify* ``otg_whitelist.h`` *to
+ match its interoperability specification.*
+
+ Non-OTG Linux hosts, like PCs and workstations, normally have some
+ solution for adding drivers, so that peripherals that aren't
+ recognized can eventually be supported. That approach is unreasonable
+ for consumer products that may never have their firmware upgraded,
+ and where it's usually unrealistic to expect traditional
+ PC/workstation/server kinds of support model to work. For example,
+ it's often impractical to change device firmware once the product has
+ been distributed, so driver bugs can't normally be fixed if they're
+ found after shipment.
+
+Additional changes are needed below those hardware-neutral :c:type:`usb_bus`
+and :c:type:`usb_gadget` driver interfaces; those aren't discussed here in any
+detail. Those affect the hardware-specific code for each USB Host or
+Peripheral controller, and how the HCD initializes (since OTG can be
+active only on a single port). They also involve what may be called an
+*OTG Controller Driver*, managing the OTG transceiver and the OTG state
+machine logic as well as much of the root hub behavior for the OTG port.
+The OTG controller driver needs to activate and deactivate USB
+controllers depending on the relevant device role. Some related changes
+were needed inside usbcore, so that it can identify OTG-capable devices
+and respond appropriately to HNP or SRP protocols.
diff --git a/Documentation/usb/hotplug.txt b/Documentation/driver-api/usb/hotplug.rst
similarity index 75%
rename from Documentation/usb/hotplug.txt
rename to Documentation/driver-api/usb/hotplug.rst
index 5b243f3..79663e6 100644
--- a/Documentation/usb/hotplug.txt
+++ b/Documentation/driver-api/usb/hotplug.rst
@@ -1,4 +1,9 @@
-LINUX HOTPLUGGING
+USB hotplugging
+~~~~~~~~~~~~~~~
+
+Linux Hotplugging
+=================
+
In hotpluggable busses like USB (and Cardbus PCI), end-users plug devices
into the bus with power on. In most cases, users expect the devices to become
@@ -30,11 +35,11 @@
that is currently available only when the drivers are dynamically linked,
you get the best hotplugging when you configure a highly modular system.
+Kernel Hotplug Helper (``/sbin/hotplug``)
+=========================================
-KERNEL HOTPLUG HELPER (/sbin/hotplug)
-
-There is a kernel parameter: /proc/sys/kernel/hotplug, which normally
-holds the pathname "/sbin/hotplug". That parameter names a program
+There is a kernel parameter: ``/proc/sys/kernel/hotplug``, which normally
+holds the pathname ``/sbin/hotplug``. That parameter names a program
which the kernel may invoke at various times.
The /sbin/hotplug program can be invoked by any subsystem as part of its
@@ -51,26 +56,26 @@
Mailing list information is also available at that site.
---------------------------------------------------------------------------
+USB Policy Agent
+================
-
-USB POLICY AGENT
-
-The USB subsystem currently invokes /sbin/hotplug when USB devices
+The USB subsystem currently invokes ``/sbin/hotplug`` when USB devices
are added or removed from system. The invocation is done by the kernel
hub workqueue [hub_wq], or else as part of root hub initialization
(done by init, modprobe, kapmd, etc). Its single command line parameter
is the string "usb", and it passes these environment variables:
- ACTION ... "add", "remove"
- PRODUCT ... USB vendor, product, and version codes (hex)
- TYPE ... device class codes (decimal)
- INTERFACE ... interface 0 class codes (decimal)
+========== ============================================
+ACTION ``add``, ``remove``
+PRODUCT USB vendor, product, and version codes (hex)
+TYPE device class codes (decimal)
+INTERFACE interface 0 class codes (decimal)
+========== ============================================
If "usbdevfs" is configured, DEVICE and DEVFS are also passed. DEVICE is
the pathname of the device, and is useful for devices with multiple and/or
alternate interfaces that complicate driver selection. By design, USB
-hotplugging is independent of "usbdevfs": you can do most essential parts
+hotplugging is independent of ``usbdevfs``: you can do most essential parts
of USB device setup without using that filesystem, and without running a
user mode daemon to detect changes in system configuration.
@@ -79,19 +84,20 @@
leverage USB module-init-tools support. Later agents might unload drivers.
-USB MODUTILS SUPPORT
+USB Modutils Support
+====================
-Current versions of module-init-tools will create a "modules.usbmap" file
-which contains the entries from each driver's MODULE_DEVICE_TABLE. Such
+Current versions of module-init-tools will create a ``modules.usbmap`` file
+which contains the entries from each driver's ``MODULE_DEVICE_TABLE``. Such
files can be used by various user mode policy agents to make sure all the
right driver modules get loaded, either at boot time or later.
-See <linux/usb.h> for full information about such table entries; or look
+See ``linux/usb.h`` for full information about such table entries; or look
at existing drivers. Each table entry describes one or more criteria to
be used when matching a driver to a device or class of devices. The
specific criteria are identified by bits set in "match_flags", paired
with field values. You can construct the criteria directly, or with
-macros such as these, and use driver_info to store more information.
+macros such as these, and use driver_info to store more information::
USB_DEVICE (vendorId, productId)
... matching devices with specified vendor and product ids
@@ -103,7 +109,7 @@
... matching specified device class info
A short example, for a driver that supports several specific USB devices
-and their quirks, might have a MODULE_DEVICE_TABLE like this:
+and their quirks, might have a MODULE_DEVICE_TABLE like this::
static const struct usb_device_id mydriver_id_table[] = {
{ USB_DEVICE (0x9999, 0xaaaa), driver_info: QUIRK_X },
@@ -116,10 +122,10 @@
Most USB device drivers should pass these tables to the USB subsystem as
well as to the module management subsystem. Not all, though: some driver
frameworks connect using interfaces layered over USB, and so they won't
-need such a "struct usb_driver".
+need such a struct :c:type:`usb_driver`.
Drivers that connect directly to the USB subsystem should be declared
-something like this:
+something like this::
static struct usb_driver mydriver = {
.name = "mydriver",
@@ -138,11 +144,11 @@
When the USB subsystem knows about a driver's device ID table, it's used when
choosing drivers to probe(). The thread doing new device processing checks
-drivers' device ID entries from the MODULE_DEVICE_TABLE against interface and
-device descriptors for the device. It will only call probe() if there is a
-match, and the third argument to probe() will be the entry that matched.
+drivers' device ID entries from the ``MODULE_DEVICE_TABLE`` against interface
+and device descriptors for the device. It will only call ``probe()`` if there
+is a match, and the third argument to ``probe()`` will be the entry that
+matched.
-If you don't provide an id_table for your driver, then your driver may get
-probed for each new device; the third parameter to probe() will be null.
-
-
+If you don't provide an ``id_table`` for your driver, then your driver may get
+probed for each new device; the third parameter to ``probe()`` will be
+``NULL``.
diff --git a/Documentation/driver-api/usb/index.rst b/Documentation/driver-api/usb/index.rst
new file mode 100644
index 0000000..1bf64ed
--- /dev/null
+++ b/Documentation/driver-api/usb/index.rst
@@ -0,0 +1,26 @@
+=============
+Linux USB API
+=============
+
+.. toctree::
+
+ usb
+ gadget
+ anchors
+ bulk-streams
+ callbacks
+ dma
+ URB
+ power-management
+ hotplug
+ persist
+ error-codes
+ writing_usb_driver
+ writing_musb_glue_layer
+
+.. only:: subproject and html
+
+ Indices
+ =======
+
+ * :ref:`genindex`
diff --git a/Documentation/usb/persist.txt b/Documentation/driver-api/usb/persist.rst
similarity index 93%
rename from Documentation/usb/persist.txt
rename to Documentation/driver-api/usb/persist.rst
index 35d70ed..08cafc6 100644
--- a/Documentation/usb/persist.txt
+++ b/Documentation/driver-api/usb/persist.rst
@@ -1,11 +1,14 @@
- USB device persistence during system suspend
+.. _usb-persist:
- Alan Stern <stern@rowland.harvard.edu>
+USB device persistence during system suspend
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- September 2, 2006 (Updated February 25, 2008)
+:Author: Alan Stern <stern@rowland.harvard.edu>
+:Date: September 2, 2006 (Updated February 25, 2008)
- What is the problem?
+What is the problem?
+====================
According to the USB specification, when a USB bus is suspended the
bus must continue to supply suspend current (around 1-5 mA). This
@@ -63,7 +66,8 @@
much better.)
- What is the solution?
+What is the solution?
+=====================
The kernel includes a feature called USB-persist. It tries to work
around these issues by allowing the core USB device data structures to
@@ -99,7 +103,7 @@
Note that the "USB-persist" feature will be applied only to those
devices for which it is enabled. You can enable the feature by doing
-(as root):
+(as root)::
echo 1 >/sys/bus/usb/devices/.../power/persist
@@ -110,7 +114,8 @@
devices where it really matters.
- Is this the best solution?
+Is this the best solution?
+==========================
Perhaps not. Arguably, keeping track of mounted filesystems and
memory mappings across device disconnects should be handled by a
@@ -130,7 +135,8 @@
other device types, such as network interfaces.
- WARNING: USB-persist can be dangerous!!
+WARNING: USB-persist can be dangerous!!
+=======================================
When recovering an interrupted power session the kernel does its best
to make sure the USB device hasn't been changed; that is, the same
diff --git a/Documentation/usb/power-management.txt b/Documentation/driver-api/usb/power-management.rst
similarity index 69%
rename from Documentation/usb/power-management.txt
rename to Documentation/driver-api/usb/power-management.rst
index 00e7069..79beb80 100644
--- a/Documentation/usb/power-management.txt
+++ b/Documentation/driver-api/usb/power-management.rst
@@ -1,10 +1,12 @@
- Power Management for USB
+.. _usb-power-management:
- Alan Stern <stern@rowland.harvard.edu>
+Power Management for USB
+~~~~~~~~~~~~~~~~~~~~~~~~
- Last-updated: February 2014
+:Author: Alan Stern <stern@rowland.harvard.edu>
+:Date: Last-updated: February 2014
-
+..
Contents:
---------
* What is Power Management?
@@ -25,14 +27,14 @@
* Suggested Userspace Port Power Policy
- What is Power Management?
- -------------------------
+What is Power Management?
+-------------------------
Power Management (PM) is the practice of saving energy by suspending
parts of a computer system when they aren't being used. While a
-component is "suspended" it is in a nonfunctional low-power state; it
+component is ``suspended`` it is in a nonfunctional low-power state; it
might even be turned off completely. A suspended component can be
-"resumed" (returned to a functional full-power state) when the kernel
+``resumed`` (returned to a functional full-power state) when the kernel
needs to use it. (There also are forms of PM in which components are
placed in a less functional but still usable state instead of being
suspended; an example would be reducing the CPU's clock rate. This
@@ -44,22 +46,25 @@
call it a "dynamic suspend" (also known as a "runtime suspend" or
"selective suspend"). This document concentrates mostly on how
dynamic PM is implemented in the USB subsystem, although system PM is
-covered to some extent (see Documentation/power/*.txt for more
+covered to some extent (see ``Documentation/power/*.txt`` for more
information about system PM).
-System PM support is present only if the kernel was built with CONFIG_SUSPEND
-or CONFIG_HIBERNATION enabled. Dynamic PM support for USB is present whenever
-the kernel was built with CONFIG_PM enabled.
+System PM support is present only if the kernel was built with
+``CONFIG_SUSPEND`` or ``CONFIG_HIBERNATION`` enabled. Dynamic PM support
+
+for USB is present whenever
+the kernel was built with ``CONFIG_PM`` enabled.
[Historically, dynamic PM support for USB was present only if the
-kernel had been built with CONFIG_USB_SUSPEND enabled (which depended on
-CONFIG_PM_RUNTIME). Starting with the 3.10 kernel release, dynamic PM support
-for USB was present whenever the kernel was built with CONFIG_PM_RUNTIME
-enabled. The CONFIG_USB_SUSPEND option had been eliminated.]
+kernel had been built with ``CONFIG_USB_SUSPEND`` enabled (which depended on
+``CONFIG_PM_RUNTIME``). Starting with the 3.10 kernel release, dynamic PM
+support for USB was present whenever the kernel was built with
+``CONFIG_PM_RUNTIME`` enabled. The ``CONFIG_USB_SUSPEND`` option had been
+eliminated.]
- What is Remote Wakeup?
- ----------------------
+What is Remote Wakeup?
+----------------------
When a device has been suspended, it generally doesn't resume until
the computer tells it to. Likewise, if the entire computer has been
@@ -76,8 +81,8 @@
pressed, or a suspended USB hub resuming when a device is plugged in.
- When is a USB device idle?
- --------------------------
+When is a USB device idle?
+--------------------------
A device is idle whenever the kernel thinks it's not busy doing
anything important and thus is a candidate for being suspended. The
@@ -92,11 +97,11 @@
being accessed through sysfs, then it definitely is idle.
- Forms of dynamic PM
- -------------------
+Forms of dynamic PM
+-------------------
Dynamic suspends occur when the kernel decides to suspend an idle
-device. This is called "autosuspend" for short. In general, a device
+device. This is called ``autosuspend`` for short. In general, a device
won't be autosuspended unless it has been idle for some minimum period
of time, the so-called idle-delay time.
@@ -125,51 +130,51 @@
allowed to issue dynamic suspends.
- The user interface for dynamic PM
- ---------------------------------
+The user interface for dynamic PM
+---------------------------------
-The user interface for controlling dynamic PM is located in the power/
+The user interface for controlling dynamic PM is located in the ``power/``
subdirectory of each USB device's sysfs directory, that is, in
-/sys/bus/usb/devices/.../power/ where "..." is the device's ID. The
+``/sys/bus/usb/devices/.../power/`` where "..." is the device's ID. The
relevant attribute files are: wakeup, control, and
-autosuspend_delay_ms. (There may also be a file named "level"; this
+``autosuspend_delay_ms``. (There may also be a file named ``level``; this
file was deprecated as of the 2.6.35 kernel and replaced by the
-"control" file. In 2.6.38 the "autosuspend" file will be deprecated
-and replaced by the "autosuspend_delay_ms" file. The only difference
+``control`` file. In 2.6.38 the ``autosuspend`` file will be deprecated
+and replaced by the ``autosuspend_delay_ms`` file. The only difference
is that the newer file expresses the delay in milliseconds whereas the
older file uses seconds. Confusingly, both files are present in 2.6.37
-but only "autosuspend" works.)
+but only ``autosuspend`` works.)
- power/wakeup
+ ``power/wakeup``
This file is empty if the device does not support
remote wakeup. Otherwise the file contains either the
- word "enabled" or the word "disabled", and you can
+ word ``enabled`` or the word ``disabled``, and you can
write those words to the file. The setting determines
whether or not remote wakeup will be enabled when the
device is next suspended. (If the setting is changed
while the device is suspended, the change won't take
effect until the following suspend.)
- power/control
+ ``power/control``
- This file contains one of two words: "on" or "auto".
+ This file contains one of two words: ``on`` or ``auto``.
You can write those words to the file to change the
device's setting.
- "on" means that the device should be resumed and
- autosuspend is not allowed. (Of course, system
- suspends are still allowed.)
+ - ``on`` means that the device should be resumed and
+ autosuspend is not allowed. (Of course, system
+ suspends are still allowed.)
- "auto" is the normal state in which the kernel is
- allowed to autosuspend and autoresume the device.
+ - ``auto`` is the normal state in which the kernel is
+ allowed to autosuspend and autoresume the device.
(In kernels up to 2.6.32, you could also specify
- "suspend", meaning that the device should remain
+ ``suspend``, meaning that the device should remain
suspended and autoresume was not allowed. This
setting is no longer supported.)
- power/autosuspend_delay_ms
+ ``power/autosuspend_delay_ms``
This file contains an integer value, which is the
number of milliseconds the device should remain idle
@@ -180,31 +185,31 @@
number to the file to change the autosuspend
idle-delay time.
-Writing "-1" to power/autosuspend_delay_ms and writing "on" to
-power/control do essentially the same thing -- they both prevent the
+Writing ``-1`` to ``power/autosuspend_delay_ms`` and writing ``on`` to
+``power/control`` do essentially the same thing -- they both prevent the
device from being autosuspended. Yes, this is a redundancy in the
API.
-(In 2.6.21 writing "0" to power/autosuspend would prevent the device
+(In 2.6.21 writing ``0`` to ``power/autosuspend`` would prevent the device
from being autosuspended; the behavior was changed in 2.6.22. The
-power/autosuspend attribute did not exist prior to 2.6.21, and the
-power/level attribute did not exist prior to 2.6.22. power/control
-was added in 2.6.34, and power/autosuspend_delay_ms was added in
+``power/autosuspend`` attribute did not exist prior to 2.6.21, and the
+``power/level`` attribute did not exist prior to 2.6.22. ``power/control``
+was added in 2.6.34, and ``power/autosuspend_delay_ms`` was added in
2.6.37 but did not become functional until 2.6.38.)
- Changing the default idle-delay time
- ------------------------------------
+Changing the default idle-delay time
+------------------------------------
The default autosuspend idle-delay time (in seconds) is controlled by
a module parameter in usbcore. You can specify the value when usbcore
is loaded. For example, to set it to 5 seconds instead of 2 you would
-do:
+do::
modprobe usbcore autosuspend=5
Equivalently, you could add to a configuration file in /etc/modprobe.d
-a line saying:
+a line saying::
options usbcore autosuspend=5
@@ -214,14 +219,14 @@
image.
If usbcore is compiled into the kernel rather than built as a loadable
-module, you can add
+module, you can add::
usbcore.autosuspend=5
to the kernel's boot command line.
Finally, the parameter value can be changed while the system is
-running. If you do:
+running. If you do::
echo 5 >/sys/module/usbcore/parameters/autosuspend
@@ -234,8 +239,8 @@
then to enable autosuspend for selected devices.
- Warnings
- --------
+Warnings
+--------
The USB specification states that all USB devices must support power
management. Nevertheless, the sad fact is that many devices do not
@@ -246,7 +251,7 @@
the same deficiency.
For this reason, by default the kernel disables autosuspend (the
-power/control attribute is initialized to "on") for all devices other
+``power/control`` attribute is initialized to ``on``) for all devices other
than hubs. Hubs, at least, appear to be reasonably well-behaved in
this regard.
@@ -284,30 +289,30 @@
possible.) Take care.
- The driver interface for Power Management
- -----------------------------------------
+The driver interface for Power Management
+-----------------------------------------
The requirements for a USB driver to support external power management
-are pretty modest; the driver need only define
+are pretty modest; the driver need only define::
.suspend
.resume
.reset_resume
-methods in its usb_driver structure, and the reset_resume method is
-optional. The methods' jobs are quite simple:
+methods in its :c:type:`usb_driver` structure, and the ``reset_resume`` method
+is optional. The methods' jobs are quite simple:
- The suspend method is called to warn the driver that the
+ - The ``suspend`` method is called to warn the driver that the
device is going to be suspended. If the driver returns a
negative error code, the suspend will be aborted. Normally
the driver will return 0, in which case it must cancel all
- outstanding URBs (usb_kill_urb()) and not submit any more.
+ outstanding URBs (:c:func:`usb_kill_urb`) and not submit any more.
- The resume method is called to tell the driver that the
+ - The ``resume`` method is called to tell the driver that the
device has been resumed and the driver can return to normal
operation. URBs may once more be submitted.
- The reset_resume method is called to tell the driver that
+ - The ``reset_resume`` method is called to tell the driver that
the device has been resumed and it also has been reset.
The driver should redo any necessary device initialization,
since the device has probably lost most or all of its state
@@ -315,22 +320,22 @@
before the suspend).
If the device is disconnected or powered down while it is suspended,
-the disconnect method will be called instead of the resume or
-reset_resume method. This is also quite likely to happen when
+the ``disconnect`` method will be called instead of the ``resume`` or
+``reset_resume`` method. This is also quite likely to happen when
waking up from hibernation, as many systems do not maintain suspend
current to the USB host controllers during hibernation. (It's
possible to work around the hibernation-forces-disconnect problem by
using the USB Persist facility.)
-The reset_resume method is used by the USB Persist facility (see
-Documentation/usb/persist.txt) and it can also be used under certain
-circumstances when CONFIG_USB_PERSIST is not enabled. Currently, if a
+The ``reset_resume`` method is used by the USB Persist facility (see
+:ref:`usb-persist`) and it can also be used under certain
+circumstances when ``CONFIG_USB_PERSIST`` is not enabled. Currently, if a
device is reset during a resume and the driver does not have a
-reset_resume method, the driver won't receive any notification about
-the resume. Later kernels will call the driver's disconnect method;
+``reset_resume`` method, the driver won't receive any notification about
+the resume. Later kernels will call the driver's ``disconnect`` method;
2.6.23 doesn't do this.
-USB drivers are bound to interfaces, so their suspend and resume
+USB drivers are bound to interfaces, so their ``suspend`` and ``resume``
methods get called when the interfaces are suspended or resumed. In
principle one might want to suspend some interfaces on a device (i.e.,
force the drivers for those interface to stop all activity) without
@@ -341,15 +346,15 @@
closest you can come is to unbind the interfaces' drivers.
- The driver interface for autosuspend and autoresume
- ---------------------------------------------------
+The driver interface for autosuspend and autoresume
+---------------------------------------------------
To support autosuspend and autoresume, a driver should implement all
three of the methods listed above. In addition, a driver indicates
-that it supports autosuspend by setting the .supports_autosuspend flag
+that it supports autosuspend by setting the ``.supports_autosuspend`` flag
in its usb_driver structure. It is then responsible for informing the
USB core whenever one of its interfaces becomes busy or idle. The
-driver does so by calling these six functions:
+driver does so by calling these six functions::
int usb_autopm_get_interface(struct usb_interface *intf);
void usb_autopm_put_interface(struct usb_interface *intf);
@@ -368,41 +373,41 @@
Drivers need not be concerned about balancing changes to the usage
counter; the USB core will undo any remaining "get"s when a driver
is unbound from its interface. As a corollary, drivers must not call
-any of the usb_autopm_* functions after their disconnect() routine has
-returned.
+any of the ``usb_autopm_*`` functions after their ``disconnect``
+routine has returned.
Drivers using the async routines are responsible for their own
synchronization and mutual exclusion.
- usb_autopm_get_interface() increments the usage counter and
+ :c:func:`usb_autopm_get_interface` increments the usage counter and
does an autoresume if the device is suspended. If the
autoresume fails, the counter is decremented back.
- usb_autopm_put_interface() decrements the usage counter and
+ :c:func:`usb_autopm_put_interface` decrements the usage counter and
attempts an autosuspend if the new value is = 0.
- usb_autopm_get_interface_async() and
- usb_autopm_put_interface_async() do almost the same things as
+ :c:func:`usb_autopm_get_interface_async` and
+ :c:func:`usb_autopm_put_interface_async` do almost the same things as
their non-async counterparts. The big difference is that they
use a workqueue to do the resume or suspend part of their
jobs. As a result they can be called in an atomic context,
such as an URB's completion handler, but when they return the
device will generally not yet be in the desired state.
- usb_autopm_get_interface_no_resume() and
- usb_autopm_put_interface_no_suspend() merely increment or
+ :c:func:`usb_autopm_get_interface_no_resume` and
+ :c:func:`usb_autopm_put_interface_no_suspend` merely increment or
decrement the usage counter; they do not attempt to carry out
an autoresume or an autosuspend. Hence they can be called in
an atomic context.
The simplest usage pattern is that a driver calls
-usb_autopm_get_interface() in its open routine and
-usb_autopm_put_interface() in its close or release routine. But other
+:c:func:`usb_autopm_get_interface` in its open routine and
+:c:func:`usb_autopm_put_interface` in its close or release routine. But other
patterns are possible.
The autosuspend attempts mentioned above will often fail for one
-reason or another. For example, the power/control attribute might be
-set to "on", or another interface in the same device might not be
+reason or another. For example, the ``power/control`` attribute might be
+set to ``on``, or another interface in the same device might not be
idle. This is perfectly normal. If the reason for failure was that
the device hasn't been idle for long enough, a timer is scheduled to
carry out the operation automatically when the autosuspend idle-delay
@@ -413,37 +418,37 @@
autosuspend, there's no idle-delay for an autoresume.
- Other parts of the driver interface
- -----------------------------------
+Other parts of the driver interface
+-----------------------------------
-Drivers can enable autosuspend for their devices by calling
+Drivers can enable autosuspend for their devices by calling::
usb_enable_autosuspend(struct usb_device *udev);
-in their probe() routine, if they know that the device is capable of
+in their :c:func:`probe` routine, if they know that the device is capable of
suspending and resuming correctly. This is exactly equivalent to
-writing "auto" to the device's power/control attribute. Likewise,
-drivers can disable autosuspend by calling
+writing ``auto`` to the device's ``power/control`` attribute. Likewise,
+drivers can disable autosuspend by calling::
usb_disable_autosuspend(struct usb_device *udev);
-This is exactly the same as writing "on" to the power/control attribute.
+This is exactly the same as writing ``on`` to the ``power/control`` attribute.
Sometimes a driver needs to make sure that remote wakeup is enabled
during autosuspend. For example, there's not much point
autosuspending a keyboard if the user can't cause the keyboard to do a
remote wakeup by typing on it. If the driver sets
-intf->needs_remote_wakeup to 1, the kernel won't autosuspend the
+``intf->needs_remote_wakeup`` to 1, the kernel won't autosuspend the
device if remote wakeup isn't available. (If the device is already
autosuspended, though, setting this flag won't cause the kernel to
-autoresume it. Normally a driver would set this flag in its probe
+autoresume it. Normally a driver would set this flag in its ``probe``
method, at which time the device is guaranteed not to be
autosuspended.)
If a driver does its I/O asynchronously in interrupt context, it
-should call usb_autopm_get_interface_async() before starting output and
-usb_autopm_put_interface_async() when the output queue drains. When
-it receives an input event, it should call
+should call :c:func:`usb_autopm_get_interface_async` before starting output and
+:c:func:`usb_autopm_put_interface_async` when the output queue drains. When
+it receives an input event, it should call::
usb_mark_last_busy(struct usb_device *udev);
@@ -453,41 +458,41 @@
so drivers need to worry only when interrupt-driven input arrives.
Asynchronous operation is always subject to races. For example, a
-driver may call the usb_autopm_get_interface_async() routine at a time
+driver may call the :c:func:`usb_autopm_get_interface_async` routine at a time
when the core has just finished deciding the device has been idle for
-long enough but not yet gotten around to calling the driver's suspend
-method. The suspend method must be responsible for synchronizing with
+long enough but not yet gotten around to calling the driver's ``suspend``
+method. The ``suspend`` method must be responsible for synchronizing with
the I/O request routine and the URB completion handler; it should
cause autosuspends to fail with -EBUSY if the driver needs to use the
device.
External suspend calls should never be allowed to fail in this way,
only autosuspend calls. The driver can tell them apart by applying
-the PMSG_IS_AUTO() macro to the message argument to the suspend
+the :c:func:`PMSG_IS_AUTO` macro to the message argument to the ``suspend``
method; it will return True for internal PM events (autosuspend) and
False for external PM events.
- Mutual exclusion
- ----------------
+Mutual exclusion
+----------------
For external events -- but not necessarily for autosuspend or
autoresume -- the device semaphore (udev->dev.sem) will be held when a
-suspend or resume method is called. This implies that external
-suspend/resume events are mutually exclusive with calls to probe,
-disconnect, pre_reset, and post_reset; the USB core guarantees that
+``suspend`` or ``resume`` method is called. This implies that external
+suspend/resume events are mutually exclusive with calls to ``probe``,
+``disconnect``, ``pre_reset``, and ``post_reset``; the USB core guarantees that
this is true of autosuspend/autoresume events as well.
If a driver wants to block all suspend/resume calls during some
critical section, the best way is to lock the device and call
-usb_autopm_get_interface() (and do the reverse at the end of the
+:c:func:`usb_autopm_get_interface` (and do the reverse at the end of the
critical section). Holding the device semaphore will block all
-external PM calls, and the usb_autopm_get_interface() will prevent any
+external PM calls, and the :c:func:`usb_autopm_get_interface` will prevent any
internal PM calls, even if it fails. (Exercise: Why?)
- Interaction between dynamic PM and system PM
- --------------------------------------------
+Interaction between dynamic PM and system PM
+--------------------------------------------
Dynamic power management and system power management can interact in
a couple of ways.
@@ -512,8 +517,8 @@
and on the hardware and firmware design.
- xHCI hardware link PM
- ---------------------
+xHCI hardware link PM
+---------------------
xHCI host controller provides hardware link power management to usb2.0
(xHCI 1.0 feature) and usb3.0 devices which support link PM. By
@@ -522,11 +527,11 @@
which state device can enter and resume very quickly.
The user interface for controlling hardware LPM is located in the
-power/ subdirectory of each USB device's sysfs directory, that is, in
-/sys/bus/usb/devices/.../power/ where "..." is the device's ID. The
-relevant attribute files are usb2_hardware_lpm and usb3_hardware_lpm.
+``power/`` subdirectory of each USB device's sysfs directory, that is, in
+``/sys/bus/usb/devices/.../power/`` where "..." is the device's ID. The
+relevant attribute files are ``usb2_hardware_lpm`` and ``usb3_hardware_lpm``.
- power/usb2_hardware_lpm
+ ``power/usb2_hardware_lpm``
When a USB2 device which support LPM is plugged to a
xHCI host root hub which support software LPM, the
@@ -537,8 +542,8 @@
can write y/Y/1 or n/N/0 to the file to enable/disable
USB2 hardware LPM manually. This is for test purpose mainly.
- power/usb3_hardware_lpm_u1
- power/usb3_hardware_lpm_u2
+ ``power/usb3_hardware_lpm_u1``
+ ``power/usb3_hardware_lpm_u2``
When a USB 3.0 lpm-capable device is plugged in to a
xHCI host which supports link PM, it will check if U1
@@ -550,29 +555,31 @@
indicating whether or not USB3 hardware LPM U1 or U2
is enabled for the device.
- USB Port Power Control
- ----------------------
+USB Port Power Control
+----------------------
In addition to suspending endpoint devices and enabling hardware
controlled link power management, the USB subsystem also has the
capability to disable power to ports under some conditions. Power is
-controlled through Set/ClearPortFeature(PORT_POWER) requests to a hub.
+controlled through ``Set/ClearPortFeature(PORT_POWER)`` requests to a hub.
In the case of a root or platform-internal hub the host controller
-driver translates PORT_POWER requests into platform firmware (ACPI)
+driver translates ``PORT_POWER`` requests into platform firmware (ACPI)
method calls to set the port power state. For more background see the
-Linux Plumbers Conference 2012 slides [1] and video [2]:
+Linux Plumbers Conference 2012 slides [#f1]_ and video [#f2]_:
-Upon receiving a ClearPortFeature(PORT_POWER) request a USB port is
-logically off, and may trigger the actual loss of VBUS to the port [3].
+Upon receiving a ``ClearPortFeature(PORT_POWER)`` request a USB port is
+logically off, and may trigger the actual loss of VBUS to the port [#f3]_.
VBUS may be maintained in the case where a hub gangs multiple ports into
a shared power well causing power to remain until all ports in the gang
are turned off. VBUS may also be maintained by hub ports configured for
a charging application. In any event a logically off port will lose
connection with its device, not respond to hotplug events, and not
-respond to remote wakeup events*.
+respond to remote wakeup events.
-WARNING: turning off a port may result in the inability to hot add a device.
-Please see "User Interface for Port Power Control" for details.
+.. warning::
+
+ turning off a port may result in the inability to hot add a device.
+ Please see "User Interface for Port Power Control" for details.
As far as the effect on the device itself it is similar to what a device
goes through during system suspend, i.e. the power session is lost. Any
@@ -581,38 +588,49 @@
implementation shares the same device recovery path (and honors the same
quirks) as the system resume path for the hub.
-[1]: http://dl.dropbox.com/u/96820575/sarah-sharp-lpt-port-power-off2-mini.pdf
-[2]: http://linuxplumbers.ubicast.tv/videos/usb-port-power-off-kerneluserspace-api/
-[3]: USB 3.1 Section 10.12
-* wakeup note: if a device is configured to send wakeup events the port
+.. [#f1]
+
+ http://dl.dropbox.com/u/96820575/sarah-sharp-lpt-port-power-off2-mini.pdf
+
+.. [#f2]
+
+ http://linuxplumbers.ubicast.tv/videos/usb-port-power-off-kerneluserspace-api/
+
+.. [#f3]
+
+ USB 3.1 Section 10.12
+
+ wakeup note: if a device is configured to send wakeup events the port
power control implementation will block poweroff attempts on that
port.
- User Interface for Port Power Control
- -------------------------------------
+User Interface for Port Power Control
+-------------------------------------
The port power control mechanism uses the PM runtime system. Poweroff is
-requested by clearing the power/pm_qos_no_power_off flag of the port device
+requested by clearing the ``power/pm_qos_no_power_off`` flag of the port device
(defaults to 1). If the port is disconnected it will immediately receive a
-ClearPortFeature(PORT_POWER) request. Otherwise, it will honor the pm runtime
-rules and require the attached child device and all descendants to be suspended.
-This mechanism is dependent on the hub advertising port power switching in its
-hub descriptor (wHubCharacteristics logical power switching mode field).
+``ClearPortFeature(PORT_POWER)`` request. Otherwise, it will honor the pm
+runtime rules and require the attached child device and all descendants to be
+suspended. This mechanism is dependent on the hub advertising port power
+switching in its hub descriptor (wHubCharacteristics logical power switching
+mode field).
Note, some interface devices/drivers do not support autosuspend. Userspace may
-need to unbind the interface drivers before the usb_device will suspend. An
-unbound interface device is suspended by default. When unbinding, be careful
-to unbind interface drivers, not the driver of the parent usb device. Also,
-leave hub interface drivers bound. If the driver for the usb device (not
-interface) is unbound the kernel is no longer able to resume the device. If a
-hub interface driver is unbound, control of its child ports is lost and all
-attached child-devices will disconnect. A good rule of thumb is that if the
-'driver/module' link for a device points to /sys/module/usbcore then unbinding
-it will interfere with port power control.
+need to unbind the interface drivers before the :c:type:`usb_device` will
+suspend. An unbound interface device is suspended by default. When unbinding,
+be careful to unbind interface drivers, not the driver of the parent usb
+device. Also, leave hub interface drivers bound. If the driver for the usb
+device (not interface) is unbound the kernel is no longer able to resume the
+device. If a hub interface driver is unbound, control of its child ports is
+lost and all attached child-devices will disconnect. A good rule of thumb is
+that if the 'driver/module' link for a device points to
+``/sys/module/usbcore`` then unbinding it will interfere with port power
+control.
Example of the relevant files for port power control. Note, in this example
-these files are relative to a usb hub device (prefix).
+these files are relative to a usb hub device (prefix)::
prefix=/sys/devices/pci0000:00/0000:00:14.0/usb3/3-1
@@ -631,10 +649,10 @@
In addition to these files some ports may have a 'peer' link to a port on
another hub. The expectation is that all superspeed ports have a
-hi-speed peer.
+hi-speed peer::
-$prefix/3-1:1.0/3-1-port1/peer -> ../../../../usb2/2-1/2-1:1.0/2-1-port1
-../../../../usb2/2-1/2-1:1.0/2-1-port1/peer -> ../../../../usb3/3-1/3-1:1.0/3-1-port1
+ $prefix/3-1:1.0/3-1-port1/peer -> ../../../../usb2/2-1/2-1:1.0/2-1-port1
+ ../../../../usb2/2-1/2-1:1.0/2-1-port1/peer -> ../../../../usb3/3-1/3-1:1.0/3-1-port1
Distinct from 'companion ports', or 'ehci/xhci shared switchover ports'
peer ports are simply the hi-speed and superspeed interface pins that
@@ -645,24 +663,26 @@
connection and attempt to connect to the hi-speed pins. The
implementation takes steps to prevent this:
-1/ Port suspend is sequenced to guarantee that hi-speed ports are powered-off
+1. Port suspend is sequenced to guarantee that hi-speed ports are powered-off
before their superspeed peer is permitted to power-off. The implication is
- that the setting pm_qos_no_power_off to zero on a superspeed port may not cause
- the port to power-off until its highspeed peer has gone to its runtime suspend
- state. Userspace must take care to order the suspensions if it wants to
- guarantee that a superspeed port will power-off.
+ that the setting ``pm_qos_no_power_off`` to zero on a superspeed port may
+ not cause the port to power-off until its highspeed peer has gone to its
+ runtime suspend state. Userspace must take care to order the suspensions
+ if it wants to guarantee that a superspeed port will power-off.
-2/ Port resume is sequenced to force a superspeed port to power-on prior to its
+2. Port resume is sequenced to force a superspeed port to power-on prior to its
highspeed peer.
-3/ Port resume always triggers an attached child device to resume. After a
+3. Port resume always triggers an attached child device to resume. After a
power session is lost the device may have been removed, or need reset.
Resuming the child device when the parent port regains power resolves those
- states and clamps the maximum port power cycle frequency at the rate the child
- device can suspend (autosuspend-delay) and resume (reset-resume latency).
+ states and clamps the maximum port power cycle frequency at the rate the
+ child device can suspend (autosuspend-delay) and resume (reset-resume
+ latency).
Sysfs files relevant for port power control:
- <hubdev-portX>/power/pm_qos_no_power_off:
+
+ ``<hubdev-portX>/power/pm_qos_no_power_off``:
This writable flag controls the state of an idle port.
Once all children and descendants have suspended the
port may suspend/poweroff provided that
@@ -670,24 +690,24 @@
'1' the port will remain active/powered regardless of
the stats of descendants. Defaults to 1.
- <hubdev-portX>/power/runtime_status:
+ ``<hubdev-portX>/power/runtime_status``:
This file reflects whether the port is 'active' (power is on)
or 'suspended' (logically off). There is no indication to
userspace whether VBUS is still supplied.
- <hubdev-portX>/connect_type:
+ ``<hubdev-portX>/connect_type``:
An advisory read-only flag to userspace indicating the
location and connection type of the port. It returns
one of four values 'hotplug', 'hardwired', 'not used',
and 'unknown'. All values, besides unknown, are set by
platform firmware.
- "hotplug" indicates an externally connectable/visible
+ ``hotplug`` indicates an externally connectable/visible
port on the platform. Typically userspace would choose
to keep such a port powered to handle new device
connection events.
- "hardwired" refers to a port that is not visible but
+ ``hardwired`` refers to a port that is not visible but
connectable. Examples are internal ports for USB
bluetooth that can be disconnected via an external
switch or a port with a hardwired USB camera. It is
@@ -698,48 +718,50 @@
powering off, or to activate the port prior to enabling
connection via a switch.
- "not used" refers to an internal port that is expected
+ ``not used`` refers to an internal port that is expected
to never have a device connected to it. These may be
empty internal ports, or ports that are not physically
exposed on a platform. Considered safe to be
powered-off at all times.
- "unknown" means platform firmware does not provide
+ ``unknown`` means platform firmware does not provide
information for this port. Most commonly refers to
external hub ports which should be considered 'hotplug'
for policy decisions.
- NOTE1: since we are relying on the BIOS to get this ACPI
- information correct, the USB port descriptions may be
- missing or wrong.
+ .. note::
- NOTE2: Take care in clearing pm_qos_no_power_off. Once
- power is off this port will
- not respond to new connect events.
+ - since we are relying on the BIOS to get this ACPI
+ information correct, the USB port descriptions may
+ be missing or wrong.
+
+ - Take care in clearing ``pm_qos_no_power_off``. Once
+ power is off this port will
+ not respond to new connect events.
Once a child device is attached additional constraints are
applied before the port is allowed to poweroff.
- <child>/power/control:
- Must be 'auto', and the port will not
- power down until <child>/power/runtime_status
+ ``<child>/power/control``:
+ Must be ``auto``, and the port will not
+ power down until ``<child>/power/runtime_status``
reflects the 'suspended' state. Default
value is controlled by child device driver.
- <child>/power/persist:
- This defaults to '1' for most devices and indicates if
+ ``<child>/power/persist``:
+ This defaults to ``1`` for most devices and indicates if
kernel can persist the device's configuration across a
power session loss (suspend / port-power event). When
- this value is '0' (quirky devices), port poweroff is
+ this value is ``0`` (quirky devices), port poweroff is
disabled.
- <child>/driver/unbind:
+ ``<child>/driver/unbind``:
Wakeup capable devices will block port poweroff. At
this time the only mechanism to clear the usb-internal
wakeup-capability for an interface device is to unbind
its driver.
-Summary of poweroff pre-requisite settings relative to a port device:
+Summary of poweroff pre-requisite settings relative to a port device::
echo 0 > power/pm_qos_no_power_off
echo 0 > peer/power/pm_qos_no_power_off # if it exists
@@ -747,14 +769,14 @@
echo auto > <child>/power/control
echo 1 > <child>/power/persist # this is the default value
- Suggested Userspace Port Power Policy
- -------------------------------------
+Suggested Userspace Port Power Policy
+-------------------------------------
As noted above userspace needs to be careful and deliberate about what
ports are enabled for poweroff.
The default configuration is that all ports start with
-power/pm_qos_no_power_off set to '1' causing ports to always remain
+``power/pm_qos_no_power_off`` set to ``1`` causing ports to always remain
active.
Given confidence in the platform firmware's description of the ports
@@ -764,7 +786,7 @@
connection switch for the port.
A more aggressive userspace policy is to enable USB port power off for
-all ports (set <hubdev-portX>/power/pm_qos_no_power_off to '0') when
+all ports (set ``<hubdev-portX>/power/pm_qos_no_power_off`` to ``0``) when
some external factor indicates the user has stopped interacting with the
system. For example, a distro may want to enable power off all USB
ports when the screen blanks, and re-power them when the screen becomes
diff --git a/Documentation/driver-api/usb/usb.rst b/Documentation/driver-api/usb/usb.rst
new file mode 100644
index 0000000..dba0f87
--- /dev/null
+++ b/Documentation/driver-api/usb/usb.rst
@@ -0,0 +1,1047 @@
+.. _usb-hostside-api:
+
+===========================
+The Linux-USB Host Side API
+===========================
+
+Introduction to USB on Linux
+============================
+
+A Universal Serial Bus (USB) is used to connect a host, such as a PC or
+workstation, to a number of peripheral devices. USB uses a tree
+structure, with the host as the root (the system's master), hubs as
+interior nodes, and peripherals as leaves (and slaves). Modern PCs
+support several such trees of USB devices, usually
+a few USB 3.0 (5 GBit/s) or USB 3.1 (10 GBit/s) and some legacy
+USB 2.0 (480 MBit/s) busses just in case.
+
+That master/slave asymmetry was designed-in for a number of reasons, one
+being ease of use. It is not physically possible to mistake upstream and
+downstream or it does not matter with a type C plug (or they are built into the
+peripheral). Also, the host software doesn't need to deal with
+distributed auto-configuration since the pre-designated master node
+manages all that.
+
+Kernel developers added USB support to Linux early in the 2.2 kernel
+series and have been developing it further since then. Besides support
+for each new generation of USB, various host controllers gained support,
+new drivers for peripherals have been added and advanced features for latency
+measurement and improved power management introduced.
+
+Linux can run inside USB devices as well as on the hosts that control
+the devices. But USB device drivers running inside those peripherals
+don't do the same things as the ones running inside hosts, so they've
+been given a different name: *gadget drivers*. This document does not
+cover gadget drivers.
+
+USB Host-Side API Model
+=======================
+
+Host-side drivers for USB devices talk to the "usbcore" APIs. There are
+two. One is intended for *general-purpose* drivers (exposed through
+driver frameworks), and the other is for drivers that are *part of the
+core*. Such core drivers include the *hub* driver (which manages trees
+of USB devices) and several different kinds of *host controller
+drivers*, which control individual busses.
+
+The device model seen by USB drivers is relatively complex.
+
+- USB supports four kinds of data transfers (control, bulk, interrupt,
+ and isochronous). Two of them (control and bulk) use bandwidth as
+ it's available, while the other two (interrupt and isochronous) are
+ scheduled to provide guaranteed bandwidth.
+
+- The device description model includes one or more "configurations"
+ per device, only one of which is active at a time. Devices are supposed
+ to be capable of operating at lower than their top
+ speeds and may provide a BOS descriptor showing the lowest speed they
+ remain fully operational at.
+
+- From USB 3.0 on configurations have one or more "functions", which
+ provide a common functionality and are grouped together for purposes
+ of power management.
+
+- Configurations or functions have one or more "interfaces", each of which may have
+ "alternate settings". Interfaces may be standardized by USB "Class"
+ specifications, or may be specific to a vendor or device.
+
+ USB device drivers actually bind to interfaces, not devices. Think of
+ them as "interface drivers", though you may not see many devices
+ where the distinction is important. *Most USB devices are simple,
+ with only one function, one configuration, one interface, and one alternate
+ setting.*
+
+- Interfaces have one or more "endpoints", each of which supports one
+ type and direction of data transfer such as "bulk out" or "interrupt
+ in". The entire configuration may have up to sixteen endpoints in
+ each direction, allocated as needed among all the interfaces.
+
+- Data transfer on USB is packetized; each endpoint has a maximum
+ packet size. Drivers must often be aware of conventions such as
+ flagging the end of bulk transfers using "short" (including zero
+ length) packets.
+
+- The Linux USB API supports synchronous calls for control and bulk
+ messages. It also supports asynchronous calls for all kinds of data
+ transfer, using request structures called "URBs" (USB Request
+ Blocks).
+
+Accordingly, the USB Core API exposed to device drivers covers quite a
+lot of territory. You'll probably need to consult the USB 3.0
+specification, available online from www.usb.org at no cost, as well as
+class or device specifications.
+
+The only host-side drivers that actually touch hardware (reading/writing
+registers, handling IRQs, and so on) are the HCDs. In theory, all HCDs
+provide the same functionality through the same API. In practice, that's
+becoming more true, but there are still differences
+that crop up especially with fault handling on the less common controllers.
+Different controllers don't
+necessarily report the same aspects of failures, and recovery from
+faults (including software-induced ones like unlinking an URB) isn't yet
+fully consistent. Device driver authors should make a point of doing
+disconnect testing (while the device is active) with each different host
+controller driver, to make sure drivers don't have bugs of their own as
+well as to make sure they aren't relying on some HCD-specific behavior.
+
+.. _usb_chapter9:
+
+USB-Standard Types
+==================
+
+In ``<linux/usb/ch9.h>`` you will find the USB data types defined in
+chapter 9 of the USB specification. These data types are used throughout
+USB, and in APIs including this host side API, gadget APIs, usb character
+devices and debugfs interfaces.
+
+.. kernel-doc:: include/linux/usb/ch9.h
+ :internal:
+
+.. _usb_header:
+
+Host-Side Data Types and Macros
+===============================
+
+The host side API exposes several layers to drivers, some of which are
+more necessary than others. These support lifecycle models for host side
+drivers and devices, and support passing buffers through usbcore to some
+HCD that performs the I/O for the device driver.
+
+.. kernel-doc:: include/linux/usb.h
+ :internal:
+
+USB Core APIs
+=============
+
+There are two basic I/O models in the USB API. The most elemental one is
+asynchronous: drivers submit requests in the form of an URB, and the
+URB's completion callback handles the next step. All USB transfer types
+support that model, although there are special cases for control URBs
+(which always have setup and status stages, but may not have a data
+stage) and isochronous URBs (which allow large packets and include
+per-packet fault reports). Built on top of that is synchronous API
+support, where a driver calls a routine that allocates one or more URBs,
+submits them, and waits until they complete. There are synchronous
+wrappers for single-buffer control and bulk transfers (which are awkward
+to use in some driver disconnect scenarios), and for scatterlist based
+streaming i/o (bulk or interrupt).
+
+USB drivers need to provide buffers that can be used for DMA, although
+they don't necessarily need to provide the DMA mapping themselves. There
+are APIs to use used when allocating DMA buffers, which can prevent use
+of bounce buffers on some systems. In some cases, drivers may be able to
+rely on 64bit DMA to eliminate another kind of bounce buffer.
+
+.. kernel-doc:: drivers/usb/core/urb.c
+ :export:
+
+.. kernel-doc:: drivers/usb/core/message.c
+ :export:
+
+.. kernel-doc:: drivers/usb/core/file.c
+ :export:
+
+.. kernel-doc:: drivers/usb/core/driver.c
+ :export:
+
+.. kernel-doc:: drivers/usb/core/usb.c
+ :export:
+
+.. kernel-doc:: drivers/usb/core/hub.c
+ :export:
+
+Host Controller APIs
+====================
+
+These APIs are only for use by host controller drivers, most of which
+implement standard register interfaces such as XHCI, EHCI, OHCI, or UHCI. UHCI
+was one of the first interfaces, designed by Intel and also used by VIA;
+it doesn't do much in hardware. OHCI was designed later, to have the
+hardware do more work (bigger transfers, tracking protocol state, and so
+on). EHCI was designed with USB 2.0; its design has features that
+resemble OHCI (hardware does much more work) as well as UHCI (some parts
+of ISO support, TD list processing). XHCI was designed with USB 3.0. It
+continues to shift support for functionality into hardware.
+
+There are host controllers other than the "big three", although most PCI
+based controllers (and a few non-PCI based ones) use one of those
+interfaces. Not all host controllers use DMA; some use PIO, and there is
+also a simulator and a virtual host controller to pipe USB over the network.
+
+The same basic APIs are available to drivers for all those controllers.
+For historical reasons they are in two layers: :c:type:`struct
+usb_bus <usb_bus>` is a rather thin layer that became available
+in the 2.2 kernels, while :c:type:`struct usb_hcd <usb_hcd>`
+is a more featureful layer
+that lets HCDs share common code, to shrink driver size and
+significantly reduce hcd-specific behaviors.
+
+.. kernel-doc:: drivers/usb/core/hcd.c
+ :export:
+
+.. kernel-doc:: drivers/usb/core/hcd-pci.c
+ :export:
+
+.. kernel-doc:: drivers/usb/core/buffer.c
+ :internal:
+
+The USB character device nodes
+==============================
+
+This chapter presents the Linux character device nodes. You may prefer
+to avoid writing new kernel code for your USB driver. User mode device
+drivers are usually packaged as applications or libraries, and may use
+character devices through some programming library that wraps it.
+Such libraries include:
+
+ - `libusb <http://libusb.sourceforge.net>`__ for C/C++, and
+ - `jUSB <http://jUSB.sourceforge.net>`__ for Java.
+
+Some old information about it can be seen at the "USB Device Filesystem"
+section of the USB Guide. The latest copy of the USB Guide can be found
+at http://www.linux-usb.org/
+
+.. note::
+
+ - They were used to be implemented via *usbfs*, but this is not part of
+ the sysfs debug interface.
+
+ - This particular documentation is incomplete, especially with respect
+ to the asynchronous mode. As of kernel 2.5.66 the code and this
+ (new) documentation need to be cross-reviewed.
+
+What files are in "devtmpfs"?
+-----------------------------
+
+Conventionally mounted at ``/dev/bus/usb/``, usbfs features include:
+
+- ``/dev/bus/usb/BBB/DDD`` ... magic files exposing the each device's
+ configuration descriptors, and supporting a series of ioctls for
+ making device requests, including I/O to devices. (Purely for access
+ by programs.)
+
+Each bus is given a number (``BBB``) based on when it was enumerated; within
+each bus, each device is given a similar number (``DDD``). Those ``BBB/DDD``
+paths are not "stable" identifiers; expect them to change even if you
+always leave the devices plugged in to the same hub port. *Don't even
+think of saving these in application configuration files.* Stable
+identifiers are available, for user mode applications that want to use
+them. HID and networking devices expose these stable IDs, so that for
+example you can be sure that you told the right UPS to power down its
+second server. Pleast note that it doesn't (yet) expose those IDs.
+
+/dev/bus/usb/BBB/DDD
+--------------------
+
+Use these files in one of these basic ways:
+
+- *They can be read,* producing first the device descriptor (18 bytes) and
+ then the descriptors for the current configuration. See the USB 2.0 spec
+ for details about those binary data formats. You'll need to convert most
+ multibyte values from little endian format to your native host byte
+ order, although a few of the fields in the device descriptor (both of
+ the BCD-encoded fields, and the vendor and product IDs) will be
+ byteswapped for you. Note that configuration descriptors include
+ descriptors for interfaces, altsettings, endpoints, and maybe additional
+ class descriptors.
+
+- *Perform USB operations* using *ioctl()* requests to make endpoint I/O
+ requests (synchronously or asynchronously) or manage the device. These
+ requests need the ``CAP_SYS_RAWIO`` capability, as well as filesystem
+ access permissions. Only one ioctl request can be made on one of these
+ device files at a time. This means that if you are synchronously reading
+ an endpoint from one thread, you won't be able to write to a different
+ endpoint from another thread until the read completes. This works for
+ *half duplex* protocols, but otherwise you'd use asynchronous i/o
+ requests.
+
+Each connected USB device has one file. The ``BBB`` indicates the bus
+number. The ``DDD`` indicates the device address on that bus. Both
+of these numbers are assigned sequentially, and can be reused, so
+you can't rely on them for stable access to devices. For example,
+it's relatively common for devices to re-enumerate while they are
+still connected (perhaps someone jostled their power supply, hub,
+or USB cable), so a device might be ``002/027`` when you first connect
+it and ``002/048`` sometime later.
+
+These files can be read as binary data. The binary data consists
+of first the device descriptor, then the descriptors for each
+configuration of the device. Multi-byte fields in the device descriptor
+are converted to host endianness by the kernel. The configuration
+descriptors are in bus endian format! The configuration descriptor
+are wTotalLength bytes apart. If a device returns less configuration
+descriptor data than indicated by wTotalLength there will be a hole in
+the file for the missing bytes. This information is also shown
+in text form by the ``/sys/kernel/debug/usb/devices`` file, described later.
+
+These files may also be used to write user-level drivers for the USB
+devices. You would open the ``/dev/bus/usb/BBB/DDD`` file read/write,
+read its descriptors to make sure it's the device you expect, and then
+bind to an interface (or perhaps several) using an ioctl call. You
+would issue more ioctls to the device to communicate to it using
+control, bulk, or other kinds of USB transfers. The IOCTLs are
+listed in the ``<linux/usbdevice_fs.h>`` file, and at this writing the
+source code (``linux/drivers/usb/core/devio.c``) is the primary reference
+for how to access devices through those files.
+
+Note that since by default these ``BBB/DDD`` files are writable only by
+root, only root can write such user mode drivers. You can selectively
+grant read/write permissions to other users by using ``chmod``. Also,
+usbfs mount options such as ``devmode=0666`` may be helpful.
+
+
+Life Cycle of User Mode Drivers
+-------------------------------
+
+Such a driver first needs to find a device file for a device it knows
+how to handle. Maybe it was told about it because a ``/sbin/hotplug``
+event handling agent chose that driver to handle the new device. Or
+maybe it's an application that scans all the ``/dev/bus/usb`` device files,
+and ignores most devices. In either case, it should :c:func:`read()`
+all the descriptors from the device file, and check them against what it
+knows how to handle. It might just reject everything except a particular
+vendor and product ID, or need a more complex policy.
+
+Never assume there will only be one such device on the system at a time!
+If your code can't handle more than one device at a time, at least
+detect when there's more than one, and have your users choose which
+device to use.
+
+Once your user mode driver knows what device to use, it interacts with
+it in either of two styles. The simple style is to make only control
+requests; some devices don't need more complex interactions than those.
+(An example might be software using vendor-specific control requests for
+some initialization or configuration tasks, with a kernel driver for the
+rest.)
+
+More likely, you need a more complex style driver: one using non-control
+endpoints, reading or writing data and claiming exclusive use of an
+interface. *Bulk* transfers are easiest to use, but only their sibling
+*interrupt* transfers work with low speed devices. Both interrupt and
+*isochronous* transfers offer service guarantees because their bandwidth
+is reserved. Such "periodic" transfers are awkward to use through usbfs,
+unless you're using the asynchronous calls. However, interrupt transfers
+can also be used in a synchronous "one shot" style.
+
+Your user-mode driver should never need to worry about cleaning up
+request state when the device is disconnected, although it should close
+its open file descriptors as soon as it starts seeing the ENODEV errors.
+
+The ioctl() Requests
+--------------------
+
+To use these ioctls, you need to include the following headers in your
+userspace program::
+
+ #include <linux/usb.h>
+ #include <linux/usbdevice_fs.h>
+ #include <asm/byteorder.h>
+
+The standard USB device model requests, from "Chapter 9" of the USB 2.0
+specification, are automatically included from the ``<linux/usb/ch9.h>``
+header.
+
+Unless noted otherwise, the ioctl requests described here will update
+the modification time on the usbfs file to which they are applied
+(unless they fail). A return of zero indicates success; otherwise, a
+standard USB error code is returned (These are documented in
+:ref:`usb-error-codes`).
+
+Each of these files multiplexes access to several I/O streams, one per
+endpoint. Each device has one control endpoint (endpoint zero) which
+supports a limited RPC style RPC access. Devices are configured by
+hub_wq (in the kernel) setting a device-wide *configuration* that
+affects things like power consumption and basic functionality. The
+endpoints are part of USB *interfaces*, which may have *altsettings*
+affecting things like which endpoints are available. Many devices only
+have a single configuration and interface, so drivers for them will
+ignore configurations and altsettings.
+
+Management/Status Requests
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A number of usbfs requests don't deal very directly with device I/O.
+They mostly relate to device management and status. These are all
+synchronous requests.
+
+USBDEVFS_CLAIMINTERFACE
+ This is used to force usbfs to claim a specific interface, which has
+ not previously been claimed by usbfs or any other kernel driver. The
+ ioctl parameter is an integer holding the number of the interface
+ (bInterfaceNumber from descriptor).
+
+ Note that if your driver doesn't claim an interface before trying to
+ use one of its endpoints, and no other driver has bound to it, then
+ the interface is automatically claimed by usbfs.
+
+ This claim will be released by a RELEASEINTERFACE ioctl, or by
+ closing the file descriptor. File modification time is not updated
+ by this request.
+
+USBDEVFS_CONNECTINFO
+ Says whether the device is lowspeed. The ioctl parameter points to a
+ structure like this::
+
+ struct usbdevfs_connectinfo {
+ unsigned int devnum;
+ unsigned char slow;
+ };
+
+ File modification time is not updated by this request.
+
+ *You can't tell whether a "not slow" device is connected at high
+ speed (480 MBit/sec) or just full speed (12 MBit/sec).* You should
+ know the devnum value already, it's the DDD value of the device file
+ name.
+
+USBDEVFS_GETDRIVER
+ Returns the name of the kernel driver bound to a given interface (a
+ string). Parameter is a pointer to this structure, which is
+ modified::
+
+ struct usbdevfs_getdriver {
+ unsigned int interface;
+ char driver[USBDEVFS_MAXDRIVERNAME + 1];
+ };
+
+ File modification time is not updated by this request.
+
+USBDEVFS_IOCTL
+ Passes a request from userspace through to a kernel driver that has
+ an ioctl entry in the *struct usb_driver* it registered::
+
+ struct usbdevfs_ioctl {
+ int ifno;
+ int ioctl_code;
+ void *data;
+ };
+
+ /* user mode call looks like this.
+ * 'request' becomes the driver->ioctl() 'code' parameter.
+ * the size of 'param' is encoded in 'request', and that data
+ * is copied to or from the driver->ioctl() 'buf' parameter.
+ */
+ static int
+ usbdev_ioctl (int fd, int ifno, unsigned request, void *param)
+ {
+ struct usbdevfs_ioctl wrapper;
+
+ wrapper.ifno = ifno;
+ wrapper.ioctl_code = request;
+ wrapper.data = param;
+
+ return ioctl (fd, USBDEVFS_IOCTL, &wrapper);
+ }
+
+ File modification time is not updated by this request.
+
+ This request lets kernel drivers talk to user mode code through
+ filesystem operations even when they don't create a character or
+ block special device. It's also been used to do things like ask
+ devices what device special file should be used. Two pre-defined
+ ioctls are used to disconnect and reconnect kernel drivers, so that
+ user mode code can completely manage binding and configuration of
+ devices.
+
+USBDEVFS_RELEASEINTERFACE
+ This is used to release the claim usbfs made on interface, either
+ implicitly or because of a USBDEVFS_CLAIMINTERFACE call, before the
+ file descriptor is closed. The ioctl parameter is an integer holding
+ the number of the interface (bInterfaceNumber from descriptor); File
+ modification time is not updated by this request.
+
+ .. warning::
+
+ *No security check is made to ensure that the task which made
+ the claim is the one which is releasing it. This means that user
+ mode driver may interfere other ones.*
+
+USBDEVFS_RESETEP
+ Resets the data toggle value for an endpoint (bulk or interrupt) to
+ DATA0. The ioctl parameter is an integer endpoint number (1 to 15,
+ as identified in the endpoint descriptor), with USB_DIR_IN added
+ if the device's endpoint sends data to the host.
+
+ .. Warning::
+
+ *Avoid using this request. It should probably be removed.* Using
+ it typically means the device and driver will lose toggle
+ synchronization. If you really lost synchronization, you likely
+ need to completely handshake with the device, using a request
+ like CLEAR_HALT or SET_INTERFACE.
+
+USBDEVFS_DROP_PRIVILEGES
+ This is used to relinquish the ability to do certain operations
+ which are considered to be privileged on a usbfs file descriptor.
+ This includes claiming arbitrary interfaces, resetting a device on
+ which there are currently claimed interfaces from other users, and
+ issuing USBDEVFS_IOCTL calls. The ioctl parameter is a 32 bit mask
+ of interfaces the user is allowed to claim on this file descriptor.
+ You may issue this ioctl more than one time to narrow said mask.
+
+Synchronous I/O Support
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Synchronous requests involve the kernel blocking until the user mode
+request completes, either by finishing successfully or by reporting an
+error. In most cases this is the simplest way to use usbfs, although as
+noted above it does prevent performing I/O to more than one endpoint at
+a time.
+
+USBDEVFS_BULK
+ Issues a bulk read or write request to the device. The ioctl
+ parameter is a pointer to this structure::
+
+ struct usbdevfs_bulktransfer {
+ unsigned int ep;
+ unsigned int len;
+ unsigned int timeout; /* in milliseconds */
+ void *data;
+ };
+
+ The ``ep`` value identifies a bulk endpoint number (1 to 15, as
+ identified in an endpoint descriptor), masked with USB_DIR_IN when
+ referring to an endpoint which sends data to the host from the
+ device. The length of the data buffer is identified by ``len``; Recent
+ kernels support requests up to about 128KBytes. *FIXME say how read
+ length is returned, and how short reads are handled.*.
+
+USBDEVFS_CLEAR_HALT
+ Clears endpoint halt (stall) and resets the endpoint toggle. This is
+ only meaningful for bulk or interrupt endpoints. The ioctl parameter
+ is an integer endpoint number (1 to 15, as identified in an endpoint
+ descriptor), masked with USB_DIR_IN when referring to an endpoint
+ which sends data to the host from the device.
+
+ Use this on bulk or interrupt endpoints which have stalled,
+ returning ``-EPIPE`` status to a data transfer request. Do not issue
+ the control request directly, since that could invalidate the host's
+ record of the data toggle.
+
+USBDEVFS_CONTROL
+ Issues a control request to the device. The ioctl parameter points
+ to a structure like this::
+
+ struct usbdevfs_ctrltransfer {
+ __u8 bRequestType;
+ __u8 bRequest;
+ __u16 wValue;
+ __u16 wIndex;
+ __u16 wLength;
+ __u32 timeout; /* in milliseconds */
+ void *data;
+ };
+
+ The first eight bytes of this structure are the contents of the
+ SETUP packet to be sent to the device; see the USB 2.0 specification
+ for details. The bRequestType value is composed by combining a
+ ``USB_TYPE_*`` value, a ``USB_DIR_*`` value, and a ``USB_RECIP_*``
+ value (from ``linux/usb.h``). If wLength is nonzero, it describes
+ the length of the data buffer, which is either written to the device
+ (USB_DIR_OUT) or read from the device (USB_DIR_IN).
+
+ At this writing, you can't transfer more than 4 KBytes of data to or
+ from a device; usbfs has a limit, and some host controller drivers
+ have a limit. (That's not usually a problem.) *Also* there's no way
+ to say it's not OK to get a short read back from the device.
+
+USBDEVFS_RESET
+ Does a USB level device reset. The ioctl parameter is ignored. After
+ the reset, this rebinds all device interfaces. File modification
+ time is not updated by this request.
+
+.. warning::
+
+ *Avoid using this call* until some usbcore bugs get fixed, since
+ it does not fully synchronize device, interface, and driver (not
+ just usbfs) state.
+
+USBDEVFS_SETINTERFACE
+ Sets the alternate setting for an interface. The ioctl parameter is
+ a pointer to a structure like this::
+
+ struct usbdevfs_setinterface {
+ unsigned int interface;
+ unsigned int altsetting;
+ };
+
+ File modification time is not updated by this request.
+
+ Those struct members are from some interface descriptor applying to
+ the current configuration. The interface number is the
+ bInterfaceNumber value, and the altsetting number is the
+ bAlternateSetting value. (This resets each endpoint in the
+ interface.)
+
+USBDEVFS_SETCONFIGURATION
+ Issues the :c:func:`usb_set_configuration()` call for the
+ device. The parameter is an integer holding the number of a
+ configuration (bConfigurationValue from descriptor). File
+ modification time is not updated by this request.
+
+.. warning::
+
+ *Avoid using this call* until some usbcore bugs get fixed, since
+ it does not fully synchronize device, interface, and driver (not
+ just usbfs) state.
+
+Asynchronous I/O Support
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+As mentioned above, there are situations where it may be important to
+initiate concurrent operations from user mode code. This is particularly
+important for periodic transfers (interrupt and isochronous), but it can
+be used for other kinds of USB requests too. In such cases, the
+asynchronous requests described here are essential. Rather than
+submitting one request and having the kernel block until it completes,
+the blocking is separate.
+
+These requests are packaged into a structure that resembles the URB used
+by kernel device drivers. (No POSIX Async I/O support here, sorry.) It
+identifies the endpoint type (``USBDEVFS_URB_TYPE_*``), endpoint
+(number, masked with USB_DIR_IN as appropriate), buffer and length,
+and a user "context" value serving to uniquely identify each request.
+(It's usually a pointer to per-request data.) Flags can modify requests
+(not as many as supported for kernel drivers).
+
+Each request can specify a realtime signal number (between SIGRTMIN and
+SIGRTMAX, inclusive) to request a signal be sent when the request
+completes.
+
+When usbfs returns these urbs, the status value is updated, and the
+buffer may have been modified. Except for isochronous transfers, the
+actual_length is updated to say how many bytes were transferred; if the
+USBDEVFS_URB_DISABLE_SPD flag is set ("short packets are not OK"), if
+fewer bytes were read than were requested then you get an error report::
+
+ struct usbdevfs_iso_packet_desc {
+ unsigned int length;
+ unsigned int actual_length;
+ unsigned int status;
+ };
+
+ struct usbdevfs_urb {
+ unsigned char type;
+ unsigned char endpoint;
+ int status;
+ unsigned int flags;
+ void *buffer;
+ int buffer_length;
+ int actual_length;
+ int start_frame;
+ int number_of_packets;
+ int error_count;
+ unsigned int signr;
+ void *usercontext;
+ struct usbdevfs_iso_packet_desc iso_frame_desc[];
+ };
+
+For these asynchronous requests, the file modification time reflects
+when the request was initiated. This contrasts with their use with the
+synchronous requests, where it reflects when requests complete.
+
+USBDEVFS_DISCARDURB
+ *TBS* File modification time is not updated by this request.
+
+USBDEVFS_DISCSIGNAL
+ *TBS* File modification time is not updated by this request.
+
+USBDEVFS_REAPURB
+ *TBS* File modification time is not updated by this request.
+
+USBDEVFS_REAPURBNDELAY
+ *TBS* File modification time is not updated by this request.
+
+USBDEVFS_SUBMITURB
+ *TBS*
+
+The USB devices
+===============
+
+The USB devices are now exported via debugfs:
+
+- ``/sys/kernel/debug/usb/devices`` ... a text file showing each of the USB
+ devices on known to the kernel, and their configuration descriptors.
+ You can also poll() this to learn about new devices.
+
+/sys/kernel/debug/usb/devices
+-----------------------------
+
+This file is handy for status viewing tools in user mode, which can scan
+the text format and ignore most of it. More detailed device status
+(including class and vendor status) is available from device-specific
+files. For information about the current format of this file, see the
+``Documentation/usb/proc_usb_info.txt`` file in your Linux kernel
+sources.
+
+This file, in combination with the poll() system call, can also be used
+to detect when devices are added or removed::
+
+ int fd;
+ struct pollfd pfd;
+
+ fd = open("/sys/kernel/debug/usb/devices", O_RDONLY);
+ pfd = { fd, POLLIN, 0 };
+ for (;;) {
+ /* The first time through, this call will return immediately. */
+ poll(&pfd, 1, -1);
+
+ /* To see what's changed, compare the file's previous and current
+ contents or scan the filesystem. (Scanning is more precise.) */
+ }
+
+Note that this behavior is intended to be used for informational and
+debug purposes. It would be more appropriate to use programs such as
+udev or HAL to initialize a device or start a user-mode helper program,
+for instance.
+
+In this file, each device's output has multiple lines of ASCII output.
+
+I made it ASCII instead of binary on purpose, so that someone
+can obtain some useful data from it without the use of an
+auxiliary program. However, with an auxiliary program, the numbers
+in the first 4 columns of each ``T:`` line (topology info:
+Lev, Prnt, Port, Cnt) can be used to build a USB topology diagram.
+
+Each line is tagged with a one-character ID for that line::
+
+ T = Topology (etc.)
+ B = Bandwidth (applies only to USB host controllers, which are
+ virtualized as root hubs)
+ D = Device descriptor info.
+ P = Product ID info. (from Device descriptor, but they won't fit
+ together on one line)
+ S = String descriptors.
+ C = Configuration descriptor info. (* = active configuration)
+ I = Interface descriptor info.
+ E = Endpoint descriptor info.
+
+/sys/kernel/debug/usb/devices output format
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Legend::
+ d = decimal number (may have leading spaces or 0's)
+ x = hexadecimal number (may have leading spaces or 0's)
+ s = string
+
+
+
+Topology info
+^^^^^^^^^^^^^
+
+::
+
+ T: Bus=dd Lev=dd Prnt=dd Port=dd Cnt=dd Dev#=ddd Spd=dddd MxCh=dd
+ | | | | | | | | |__MaxChildren
+ | | | | | | | |__Device Speed in Mbps
+ | | | | | | |__DeviceNumber
+ | | | | | |__Count of devices at this level
+ | | | | |__Connector/Port on Parent for this device
+ | | | |__Parent DeviceNumber
+ | | |__Level in topology for this bus
+ | |__Bus number
+ |__Topology info tag
+
+Speed may be:
+
+ ======= ======================================================
+ 1.5 Mbit/s for low speed USB
+ 12 Mbit/s for full speed USB
+ 480 Mbit/s for high speed USB (added for USB 2.0);
+ also used for Wireless USB, which has no fixed speed
+ 5000 Mbit/s for SuperSpeed USB (added for USB 3.0)
+ ======= ======================================================
+
+For reasons lost in the mists of time, the Port number is always
+too low by 1. For example, a device plugged into port 4 will
+show up with ``Port=03``.
+
+Bandwidth info
+^^^^^^^^^^^^^^
+
+::
+
+ B: Alloc=ddd/ddd us (xx%), #Int=ddd, #Iso=ddd
+ | | | |__Number of isochronous requests
+ | | |__Number of interrupt requests
+ | |__Total Bandwidth allocated to this bus
+ |__Bandwidth info tag
+
+Bandwidth allocation is an approximation of how much of one frame
+(millisecond) is in use. It reflects only periodic transfers, which
+are the only transfers that reserve bandwidth. Control and bulk
+transfers use all other bandwidth, including reserved bandwidth that
+is not used for transfers (such as for short packets).
+
+The percentage is how much of the "reserved" bandwidth is scheduled by
+those transfers. For a low or full speed bus (loosely, "USB 1.1"),
+90% of the bus bandwidth is reserved. For a high speed bus (loosely,
+"USB 2.0") 80% is reserved.
+
+
+Device descriptor info & Product ID info
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+::
+
+ D: Ver=x.xx Cls=xx(s) Sub=xx Prot=xx MxPS=dd #Cfgs=dd
+ P: Vendor=xxxx ProdID=xxxx Rev=xx.xx
+
+where::
+
+ D: Ver=x.xx Cls=xx(sssss) Sub=xx Prot=xx MxPS=dd #Cfgs=dd
+ | | | | | | |__NumberConfigurations
+ | | | | | |__MaxPacketSize of Default Endpoint
+ | | | | |__DeviceProtocol
+ | | | |__DeviceSubClass
+ | | |__DeviceClass
+ | |__Device USB version
+ |__Device info tag #1
+
+where::
+
+ P: Vendor=xxxx ProdID=xxxx Rev=xx.xx
+ | | | |__Product revision number
+ | | |__Product ID code
+ | |__Vendor ID code
+ |__Device info tag #2
+
+
+String descriptor info
+^^^^^^^^^^^^^^^^^^^^^^
+::
+
+ S: Manufacturer=ssss
+ | |__Manufacturer of this device as read from the device.
+ | For USB host controller drivers (virtual root hubs) this may
+ | be omitted, or (for newer drivers) will identify the kernel
+ | version and the driver which provides this hub emulation.
+ |__String info tag
+
+ S: Product=ssss
+ | |__Product description of this device as read from the device.
+ | For older USB host controller drivers (virtual root hubs) this
+ | indicates the driver; for newer ones, it's a product (and vendor)
+ | description that often comes from the kernel's PCI ID database.
+ |__String info tag
+
+ S: SerialNumber=ssss
+ | |__Serial Number of this device as read from the device.
+ | For USB host controller drivers (virtual root hubs) this is
+ | some unique ID, normally a bus ID (address or slot name) that
+ | can't be shared with any other device.
+ |__String info tag
+
+
+
+Configuration descriptor info
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+::
+
+ C:* #Ifs=dd Cfg#=dd Atr=xx MPwr=dddmA
+ | | | | | |__MaxPower in mA
+ | | | | |__Attributes
+ | | | |__ConfiguratioNumber
+ | | |__NumberOfInterfaces
+ | |__ "*" indicates the active configuration (others are " ")
+ |__Config info tag
+
+USB devices may have multiple configurations, each of which act
+rather differently. For example, a bus-powered configuration
+might be much less capable than one that is self-powered. Only
+one device configuration can be active at a time; most devices
+have only one configuration.
+
+Each configuration consists of one or more interfaces. Each
+interface serves a distinct "function", which is typically bound
+to a different USB device driver. One common example is a USB
+speaker with an audio interface for playback, and a HID interface
+for use with software volume control.
+
+Interface descriptor info (can be multiple per Config)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+::
+
+ I:* If#=dd Alt=dd #EPs=dd Cls=xx(sssss) Sub=xx Prot=xx Driver=ssss
+ | | | | | | | | |__Driver name
+ | | | | | | | | or "(none)"
+ | | | | | | | |__InterfaceProtocol
+ | | | | | | |__InterfaceSubClass
+ | | | | | |__InterfaceClass
+ | | | | |__NumberOfEndpoints
+ | | | |__AlternateSettingNumber
+ | | |__InterfaceNumber
+ | |__ "*" indicates the active altsetting (others are " ")
+ |__Interface info tag
+
+A given interface may have one or more "alternate" settings.
+For example, default settings may not use more than a small
+amount of periodic bandwidth. To use significant fractions
+of bus bandwidth, drivers must select a non-default altsetting.
+
+Only one setting for an interface may be active at a time, and
+only one driver may bind to an interface at a time. Most devices
+have only one alternate setting per interface.
+
+
+Endpoint descriptor info (can be multiple per Interface)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+::
+
+ E: Ad=xx(s) Atr=xx(ssss) MxPS=dddd Ivl=dddss
+ | | | | |__Interval (max) between transfers
+ | | | |__EndpointMaxPacketSize
+ | | |__Attributes(EndpointType)
+ | |__EndpointAddress(I=In,O=Out)
+ |__Endpoint info tag
+
+The interval is nonzero for all periodic (interrupt or isochronous)
+endpoints. For high speed endpoints the transfer interval may be
+measured in microseconds rather than milliseconds.
+
+For high speed periodic endpoints, the ``EndpointMaxPacketSize`` reflects
+the per-microframe data transfer size. For "high bandwidth"
+endpoints, that can reflect two or three packets (for up to
+3KBytes every 125 usec) per endpoint.
+
+With the Linux-USB stack, periodic bandwidth reservations use the
+transfer intervals and sizes provided by URBs, which can be less
+than those found in endpoint descriptor.
+
+Usage examples
+~~~~~~~~~~~~~~
+
+If a user or script is interested only in Topology info, for
+example, use something like ``grep ^T: /sys/kernel/debug/usb/devices``
+for only the Topology lines. A command like
+``grep -i ^[tdp]: /sys/kernel/debug/usb/devices`` can be used to list
+only the lines that begin with the characters in square brackets,
+where the valid characters are TDPCIE. With a slightly more able
+script, it can display any selected lines (for example, only T, D,
+and P lines) and change their output format. (The ``procusb``
+Perl script is the beginning of this idea. It will list only
+selected lines [selected from TBDPSCIE] or "All" lines from
+``/sys/kernel/debug/usb/devices``.)
+
+The Topology lines can be used to generate a graphic/pictorial
+of the USB devices on a system's root hub. (See more below
+on how to do this.)
+
+The Interface lines can be used to determine what driver is
+being used for each device, and which altsetting it activated.
+
+The Configuration lines could be used to list maximum power
+(in milliamps) that a system's USB devices are using.
+For example, ``grep ^C: /sys/kernel/debug/usb/devices``.
+
+
+Here's an example, from a system which has a UHCI root hub,
+an external hub connected to the root hub, and a mouse and
+a serial converter connected to the external hub.
+
+::
+
+ T: Bus=00 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#= 1 Spd=12 MxCh= 2
+ B: Alloc= 28/900 us ( 3%), #Int= 2, #Iso= 0
+ D: Ver= 1.00 Cls=09(hub ) Sub=00 Prot=00 MxPS= 8 #Cfgs= 1
+ P: Vendor=0000 ProdID=0000 Rev= 0.00
+ S: Product=USB UHCI Root Hub
+ S: SerialNumber=dce0
+ C:* #Ifs= 1 Cfg#= 1 Atr=40 MxPwr= 0mA
+ I: If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub
+ E: Ad=81(I) Atr=03(Int.) MxPS= 8 Ivl=255ms
+
+ T: Bus=00 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#= 2 Spd=12 MxCh= 4
+ D: Ver= 1.00 Cls=09(hub ) Sub=00 Prot=00 MxPS= 8 #Cfgs= 1
+ P: Vendor=0451 ProdID=1446 Rev= 1.00
+ C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=100mA
+ I: If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub
+ E: Ad=81(I) Atr=03(Int.) MxPS= 1 Ivl=255ms
+
+ T: Bus=00 Lev=02 Prnt=02 Port=00 Cnt=01 Dev#= 3 Spd=1.5 MxCh= 0
+ D: Ver= 1.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS= 8 #Cfgs= 1
+ P: Vendor=04b4 ProdID=0001 Rev= 0.00
+ C:* #Ifs= 1 Cfg#= 1 Atr=80 MxPwr=100mA
+ I: If#= 0 Alt= 0 #EPs= 1 Cls=03(HID ) Sub=01 Prot=02 Driver=mouse
+ E: Ad=81(I) Atr=03(Int.) MxPS= 3 Ivl= 10ms
+
+ T: Bus=00 Lev=02 Prnt=02 Port=02 Cnt=02 Dev#= 4 Spd=12 MxCh= 0
+ D: Ver= 1.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS= 8 #Cfgs= 1
+ P: Vendor=0565 ProdID=0001 Rev= 1.08
+ S: Manufacturer=Peracom Networks, Inc.
+ S: Product=Peracom USB to Serial Converter
+ C:* #Ifs= 1 Cfg#= 1 Atr=a0 MxPwr=100mA
+ I: If#= 0 Alt= 0 #EPs= 3 Cls=00(>ifc ) Sub=00 Prot=00 Driver=serial
+ E: Ad=81(I) Atr=02(Bulk) MxPS= 64 Ivl= 16ms
+ E: Ad=01(O) Atr=02(Bulk) MxPS= 16 Ivl= 16ms
+ E: Ad=82(I) Atr=03(Int.) MxPS= 8 Ivl= 8ms
+
+
+Selecting only the ``T:`` and ``I:`` lines from this (for example, by using
+``procusb ti``), we have
+
+::
+
+ T: Bus=00 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#= 1 Spd=12 MxCh= 2
+ T: Bus=00 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#= 2 Spd=12 MxCh= 4
+ I: If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub
+ T: Bus=00 Lev=02 Prnt=02 Port=00 Cnt=01 Dev#= 3 Spd=1.5 MxCh= 0
+ I: If#= 0 Alt= 0 #EPs= 1 Cls=03(HID ) Sub=01 Prot=02 Driver=mouse
+ T: Bus=00 Lev=02 Prnt=02 Port=02 Cnt=02 Dev#= 4 Spd=12 MxCh= 0
+ I: If#= 0 Alt= 0 #EPs= 3 Cls=00(>ifc ) Sub=00 Prot=00 Driver=serial
+
+
+Physically this looks like (or could be converted to)::
+
+ +------------------+
+ | PC/root_hub (12)| Dev# = 1
+ +------------------+ (nn) is Mbps.
+ Level 0 | CN.0 | CN.1 | [CN = connector/port #]
+ +------------------+
+ /
+ /
+ +-----------------------+
+ Level 1 | Dev#2: 4-port hub (12)|
+ +-----------------------+
+ |CN.0 |CN.1 |CN.2 |CN.3 |
+ +-----------------------+
+ \ \____________________
+ \_____ \
+ \ \
+ +--------------------+ +--------------------+
+ Level 2 | Dev# 3: mouse (1.5)| | Dev# 4: serial (12)|
+ +--------------------+ +--------------------+
+
+
+
+Or, in a more tree-like structure (ports [Connectors] without
+connections could be omitted)::
+
+ PC: Dev# 1, root hub, 2 ports, 12 Mbps
+ |_ CN.0: Dev# 2, hub, 4 ports, 12 Mbps
+ |_ CN.0: Dev #3, mouse, 1.5 Mbps
+ |_ CN.1:
+ |_ CN.2: Dev #4, serial, 12 Mbps
+ |_ CN.3:
+ |_ CN.1:
diff --git a/Documentation/driver-api/usb/writing_musb_glue_layer.rst b/Documentation/driver-api/usb/writing_musb_glue_layer.rst
new file mode 100644
index 0000000..e90e8fa
--- /dev/null
+++ b/Documentation/driver-api/usb/writing_musb_glue_layer.rst
@@ -0,0 +1,723 @@
+=========================
+Writing a MUSB Glue Layer
+=========================
+
+:Author: Apelete Seketeli
+
+Introduction
+============
+
+The Linux MUSB subsystem is part of the larger Linux USB subsystem. It
+provides support for embedded USB Device Controllers (UDC) that do not
+use Universal Host Controller Interface (UHCI) or Open Host Controller
+Interface (OHCI).
+
+Instead, these embedded UDC rely on the USB On-the-Go (OTG)
+specification which they implement at least partially. The silicon
+reference design used in most cases is the Multipoint USB Highspeed
+Dual-Role Controller (MUSB HDRC) found in the Mentor Graphics Inventra™
+design.
+
+As a self-taught exercise I have written an MUSB glue layer for the
+Ingenic JZ4740 SoC, modelled after the many MUSB glue layers in the
+kernel source tree. This layer can be found at
+``drivers/usb/musb/jz4740.c``. In this documentation I will walk through the
+basics of the ``jz4740.c`` glue layer, explaining the different pieces and
+what needs to be done in order to write your own device glue layer.
+
+.. _musb-basics:
+
+Linux MUSB Basics
+=================
+
+To get started on the topic, please read USB On-the-Go Basics (see
+Resources) which provides an introduction of USB OTG operation at the
+hardware level. A couple of wiki pages by Texas Instruments and Analog
+Devices also provide an overview of the Linux kernel MUSB configuration,
+albeit focused on some specific devices provided by these companies.
+Finally, getting acquainted with the USB specification at USB home page
+may come in handy, with practical instance provided through the Writing
+USB Device Drivers documentation (again, see Resources).
+
+Linux USB stack is a layered architecture in which the MUSB controller
+hardware sits at the lowest. The MUSB controller driver abstract the
+MUSB controller hardware to the Linux USB stack::
+
+ ------------------------
+ | | <------- drivers/usb/gadget
+ | Linux USB Core Stack | <------- drivers/usb/host
+ | | <------- drivers/usb/core
+ ------------------------
+ ⬍
+ --------------------------
+ | | <------ drivers/usb/musb/musb_gadget.c
+ | MUSB Controller driver | <------ drivers/usb/musb/musb_host.c
+ | | <------ drivers/usb/musb/musb_core.c
+ --------------------------
+ ⬍
+ ---------------------------------
+ | MUSB Platform Specific Driver |
+ | | <-- drivers/usb/musb/jz4740.c
+ | aka "Glue Layer" |
+ ---------------------------------
+ ⬍
+ ---------------------------------
+ | MUSB Controller Hardware |
+ ---------------------------------
+
+As outlined above, the glue layer is actually the platform specific code
+sitting in between the controller driver and the controller hardware.
+
+Just like a Linux USB driver needs to register itself with the Linux USB
+subsystem, the MUSB glue layer needs first to register itself with the
+MUSB controller driver. This will allow the controller driver to know
+about which device the glue layer supports and which functions to call
+when a supported device is detected or released; remember we are talking
+about an embedded controller chip here, so no insertion or removal at
+run-time.
+
+All of this information is passed to the MUSB controller driver through
+a :c:type:`platform_driver` structure defined in the glue layer as::
+
+ static struct platform_driver jz4740_driver = {
+ .probe = jz4740_probe,
+ .remove = jz4740_remove,
+ .driver = {
+ .name = "musb-jz4740",
+ },
+ };
+
+The probe and remove function pointers are called when a matching device
+is detected and, respectively, released. The name string describes the
+device supported by this glue layer. In the current case it matches a
+platform_device structure declared in ``arch/mips/jz4740/platform.c``. Note
+that we are not using device tree bindings here.
+
+In order to register itself to the controller driver, the glue layer
+goes through a few steps, basically allocating the controller hardware
+resources and initialising a couple of circuits. To do so, it needs to
+keep track of the information used throughout these steps. This is done
+by defining a private ``jz4740_glue`` structure::
+
+ struct jz4740_glue {
+ struct device *dev;
+ struct platform_device *musb;
+ struct clk *clk;
+ };
+
+
+The dev and musb members are both device structure variables. The first
+one holds generic information about the device, since it's the basic
+device structure, and the latter holds information more closely related
+to the subsystem the device is registered to. The clk variable keeps
+information related to the device clock operation.
+
+Let's go through the steps of the probe function that leads the glue
+layer to register itself to the controller driver.
+
+.. note::
+
+ For the sake of readability each function will be split in logical
+ parts, each part being shown as if it was independent from the others.
+
+.. code-block:: c
+ :emphasize-lines: 8,12,18
+
+ static int jz4740_probe(struct platform_device *pdev)
+ {
+ struct platform_device *musb;
+ struct jz4740_glue *glue;
+ struct clk *clk;
+ int ret;
+
+ glue = devm_kzalloc(&pdev->dev, sizeof(*glue), GFP_KERNEL);
+ if (!glue)
+ return -ENOMEM;
+
+ musb = platform_device_alloc("musb-hdrc", PLATFORM_DEVID_AUTO);
+ if (!musb) {
+ dev_err(&pdev->dev, "failed to allocate musb device\n");
+ return -ENOMEM;
+ }
+
+ clk = devm_clk_get(&pdev->dev, "udc");
+ if (IS_ERR(clk)) {
+ dev_err(&pdev->dev, "failed to get clock\n");
+ ret = PTR_ERR(clk);
+ goto err_platform_device_put;
+ }
+
+ ret = clk_prepare_enable(clk);
+ if (ret) {
+ dev_err(&pdev->dev, "failed to enable clock\n");
+ goto err_platform_device_put;
+ }
+
+ musb->dev.parent = &pdev->dev;
+
+ glue->dev = &pdev->dev;
+ glue->musb = musb;
+ glue->clk = clk;
+
+ return 0;
+
+ err_platform_device_put:
+ platform_device_put(musb);
+ return ret;
+ }
+
+The first few lines of the probe function allocate and assign the glue,
+musb and clk variables. The ``GFP_KERNEL`` flag (line 8) allows the
+allocation process to sleep and wait for memory, thus being usable in a
+locking situation. The ``PLATFORM_DEVID_AUTO`` flag (line 12) allows
+automatic allocation and management of device IDs in order to avoid
+device namespace collisions with explicit IDs. With :c:func:`devm_clk_get`
+(line 18) the glue layer allocates the clock -- the ``devm_`` prefix
+indicates that :c:func:`clk_get` is managed: it automatically frees the
+allocated clock resource data when the device is released -- and enable
+it.
+
+
+
+Then comes the registration steps:
+
+.. code-block:: c
+ :emphasize-lines: 3,5,7,9,16
+
+ static int jz4740_probe(struct platform_device *pdev)
+ {
+ struct musb_hdrc_platform_data *pdata = &jz4740_musb_platform_data;
+
+ pdata->platform_ops = &jz4740_musb_ops;
+
+ platform_set_drvdata(pdev, glue);
+
+ ret = platform_device_add_resources(musb, pdev->resource,
+ pdev->num_resources);
+ if (ret) {
+ dev_err(&pdev->dev, "failed to add resources\n");
+ goto err_clk_disable;
+ }
+
+ ret = platform_device_add_data(musb, pdata, sizeof(*pdata));
+ if (ret) {
+ dev_err(&pdev->dev, "failed to add platform_data\n");
+ goto err_clk_disable;
+ }
+
+ return 0;
+
+ err_clk_disable:
+ clk_disable_unprepare(clk);
+ err_platform_device_put:
+ platform_device_put(musb);
+ return ret;
+ }
+
+The first step is to pass the device data privately held by the glue
+layer on to the controller driver through :c:func:`platform_set_drvdata`
+(line 7). Next is passing on the device resources information, also privately
+held at that point, through :c:func:`platform_device_add_resources` (line 9).
+
+Finally comes passing on the platform specific data to the controller
+driver (line 16). Platform data will be discussed in
+:ref:`musb-dev-platform-data`, but here we are looking at the
+``platform_ops`` function pointer (line 5) in ``musb_hdrc_platform_data``
+structure (line 3). This function pointer allows the MUSB controller
+driver to know which function to call for device operation::
+
+ static const struct musb_platform_ops jz4740_musb_ops = {
+ .init = jz4740_musb_init,
+ .exit = jz4740_musb_exit,
+ };
+
+Here we have the minimal case where only init and exit functions are
+called by the controller driver when needed. Fact is the JZ4740 MUSB
+controller is a basic controller, lacking some features found in other
+controllers, otherwise we may also have pointers to a few other
+functions like a power management function or a function to switch
+between OTG and non-OTG modes, for instance.
+
+At that point of the registration process, the controller driver
+actually calls the init function:
+
+ .. code-block:: c
+ :emphasize-lines: 12,14
+
+ static int jz4740_musb_init(struct musb *musb)
+ {
+ musb->xceiv = usb_get_phy(USB_PHY_TYPE_USB2);
+ if (!musb->xceiv) {
+ pr_err("HS UDC: no transceiver configured\n");
+ return -ENODEV;
+ }
+
+ /* Silicon does not implement ConfigData register.
+ * Set dyn_fifo to avoid reading EP config from hardware.
+ */
+ musb->dyn_fifo = true;
+
+ musb->isr = jz4740_musb_interrupt;
+
+ return 0;
+ }
+
+The goal of ``jz4740_musb_init()`` is to get hold of the transceiver
+driver data of the MUSB controller hardware and pass it on to the MUSB
+controller driver, as usual. The transceiver is the circuitry inside the
+controller hardware responsible for sending/receiving the USB data.
+Since it is an implementation of the physical layer of the OSI model,
+the transceiver is also referred to as PHY.
+
+Getting hold of the ``MUSB PHY`` driver data is done with ``usb_get_phy()``
+which returns a pointer to the structure containing the driver instance
+data. The next couple of instructions (line 12 and 14) are used as a
+quirk and to setup IRQ handling respectively. Quirks and IRQ handling
+will be discussed later in :ref:`musb-dev-quirks` and
+:ref:`musb-handling-irqs`\ ::
+
+ static int jz4740_musb_exit(struct musb *musb)
+ {
+ usb_put_phy(musb->xceiv);
+
+ return 0;
+ }
+
+Acting as the counterpart of init, the exit function releases the MUSB
+PHY driver when the controller hardware itself is about to be released.
+
+Again, note that init and exit are fairly simple in this case due to the
+basic set of features of the JZ4740 controller hardware. When writing an
+musb glue layer for a more complex controller hardware, you might need
+to take care of more processing in those two functions.
+
+Returning from the init function, the MUSB controller driver jumps back
+into the probe function::
+
+ static int jz4740_probe(struct platform_device *pdev)
+ {
+ ret = platform_device_add(musb);
+ if (ret) {
+ dev_err(&pdev->dev, "failed to register musb device\n");
+ goto err_clk_disable;
+ }
+
+ return 0;
+
+ err_clk_disable:
+ clk_disable_unprepare(clk);
+ err_platform_device_put:
+ platform_device_put(musb);
+ return ret;
+ }
+
+This is the last part of the device registration process where the glue
+layer adds the controller hardware device to Linux kernel device
+hierarchy: at this stage, all known information about the device is
+passed on to the Linux USB core stack:
+
+ .. code-block:: c
+ :emphasize-lines: 5,6
+
+ static int jz4740_remove(struct platform_device *pdev)
+ {
+ struct jz4740_glue *glue = platform_get_drvdata(pdev);
+
+ platform_device_unregister(glue->musb);
+ clk_disable_unprepare(glue->clk);
+
+ return 0;
+ }
+
+Acting as the counterpart of probe, the remove function unregister the
+MUSB controller hardware (line 5) and disable the clock (line 6),
+allowing it to be gated.
+
+.. _musb-handling-irqs:
+
+Handling IRQs
+=============
+
+Additionally to the MUSB controller hardware basic setup and
+registration, the glue layer is also responsible for handling the IRQs:
+
+ .. code-block:: c
+ :emphasize-lines: 7,9-11,14,24
+
+ static irqreturn_t jz4740_musb_interrupt(int irq, void *__hci)
+ {
+ unsigned long flags;
+ irqreturn_t retval = IRQ_NONE;
+ struct musb *musb = __hci;
+
+ spin_lock_irqsave(&musb->lock, flags);
+
+ musb->int_usb = musb_readb(musb->mregs, MUSB_INTRUSB);
+ musb->int_tx = musb_readw(musb->mregs, MUSB_INTRTX);
+ musb->int_rx = musb_readw(musb->mregs, MUSB_INTRRX);
+
+ /*
+ * The controller is gadget only, the state of the host mode IRQ bits is
+ * undefined. Mask them to make sure that the musb driver core will
+ * never see them set
+ */
+ musb->int_usb &= MUSB_INTR_SUSPEND | MUSB_INTR_RESUME |
+ MUSB_INTR_RESET | MUSB_INTR_SOF;
+
+ if (musb->int_usb || musb->int_tx || musb->int_rx)
+ retval = musb_interrupt(musb);
+
+ spin_unlock_irqrestore(&musb->lock, flags);
+
+ return retval;
+ }
+
+Here the glue layer mostly has to read the relevant hardware registers
+and pass their values on to the controller driver which will handle the
+actual event that triggered the IRQ.
+
+The interrupt handler critical section is protected by the
+:c:func:`spin_lock_irqsave` and counterpart :c:func:`spin_unlock_irqrestore`
+functions (line 7 and 24 respectively), which prevent the interrupt
+handler code to be run by two different threads at the same time.
+
+Then the relevant interrupt registers are read (line 9 to 11):
+
+- ``MUSB_INTRUSB``: indicates which USB interrupts are currently active,
+
+- ``MUSB_INTRTX``: indicates which of the interrupts for TX endpoints are
+ currently active,
+
+- ``MUSB_INTRRX``: indicates which of the interrupts for TX endpoints are
+ currently active.
+
+Note that :c:func:`musb_readb` is used to read 8-bit registers at most, while
+:c:func:`musb_readw` allows us to read at most 16-bit registers. There are
+other functions that can be used depending on the size of your device
+registers. See ``musb_io.h`` for more information.
+
+Instruction on line 18 is another quirk specific to the JZ4740 USB
+device controller, which will be discussed later in :ref:`musb-dev-quirks`.
+
+The glue layer still needs to register the IRQ handler though. Remember
+the instruction on line 14 of the init function::
+
+ static int jz4740_musb_init(struct musb *musb)
+ {
+ musb->isr = jz4740_musb_interrupt;
+
+ return 0;
+ }
+
+This instruction sets a pointer to the glue layer IRQ handler function,
+in order for the controller hardware to call the handler back when an
+IRQ comes from the controller hardware. The interrupt handler is now
+implemented and registered.
+
+.. _musb-dev-platform-data:
+
+Device Platform Data
+====================
+
+In order to write an MUSB glue layer, you need to have some data
+describing the hardware capabilities of your controller hardware, which
+is called the platform data.
+
+Platform data is specific to your hardware, though it may cover a broad
+range of devices, and is generally found somewhere in the ``arch/``
+directory, depending on your device architecture.
+
+For instance, platform data for the JZ4740 SoC is found in
+``arch/mips/jz4740/platform.c``. In the ``platform.c`` file each device of the
+JZ4740 SoC is described through a set of structures.
+
+Here is the part of ``arch/mips/jz4740/platform.c`` that covers the USB
+Device Controller (UDC):
+
+ .. code-block:: c
+ :emphasize-lines: 2,7,14-17,21,22,25,26,28,29
+
+ /* USB Device Controller */
+ struct platform_device jz4740_udc_xceiv_device = {
+ .name = "usb_phy_gen_xceiv",
+ .id = 0,
+ };
+
+ static struct resource jz4740_udc_resources[] = {
+ [0] = {
+ .start = JZ4740_UDC_BASE_ADDR,
+ .end = JZ4740_UDC_BASE_ADDR + 0x10000 - 1,
+ .flags = IORESOURCE_MEM,
+ },
+ [1] = {
+ .start = JZ4740_IRQ_UDC,
+ .end = JZ4740_IRQ_UDC,
+ .flags = IORESOURCE_IRQ,
+ .name = "mc",
+ },
+ };
+
+ struct platform_device jz4740_udc_device = {
+ .name = "musb-jz4740",
+ .id = -1,
+ .dev = {
+ .dma_mask = &jz4740_udc_device.dev.coherent_dma_mask,
+ .coherent_dma_mask = DMA_BIT_MASK(32),
+ },
+ .num_resources = ARRAY_SIZE(jz4740_udc_resources),
+ .resource = jz4740_udc_resources,
+ };
+
+The ``jz4740_udc_xceiv_device`` platform device structure (line 2)
+describes the UDC transceiver with a name and id number.
+
+At the time of this writing, note that ``usb_phy_gen_xceiv`` is the
+specific name to be used for all transceivers that are either built-in
+with reference USB IP or autonomous and doesn't require any PHY
+programming. You will need to set ``CONFIG_NOP_USB_XCEIV=y`` in the
+kernel configuration to make use of the corresponding transceiver
+driver. The id field could be set to -1 (equivalent to
+``PLATFORM_DEVID_NONE``), -2 (equivalent to ``PLATFORM_DEVID_AUTO``) or
+start with 0 for the first device of this kind if we want a specific id
+number.
+
+The ``jz4740_udc_resources`` resource structure (line 7) defines the UDC
+registers base addresses.
+
+The first array (line 9 to 11) defines the UDC registers base memory
+addresses: start points to the first register memory address, end points
+to the last register memory address and the flags member defines the
+type of resource we are dealing with. So ``IORESOURCE_MEM`` is used to
+define the registers memory addresses. The second array (line 14 to 17)
+defines the UDC IRQ registers addresses. Since there is only one IRQ
+register available for the JZ4740 UDC, start and end point at the same
+address. The ``IORESOURCE_IRQ`` flag tells that we are dealing with IRQ
+resources, and the name ``mc`` is in fact hard-coded in the MUSB core in
+order for the controller driver to retrieve this IRQ resource by
+querying it by its name.
+
+Finally, the ``jz4740_udc_device`` platform device structure (line 21)
+describes the UDC itself.
+
+The ``musb-jz4740`` name (line 22) defines the MUSB driver that is used
+for this device; remember this is in fact the name that we used in the
+``jz4740_driver`` platform driver structure in :ref:`musb-basics`.
+The id field (line 23) is set to -1 (equivalent to ``PLATFORM_DEVID_NONE``)
+since we do not need an id for the device: the MUSB controller driver was
+already set to allocate an automatic id in :ref:`musb-basics`. In the dev field
+we care for DMA related information here. The ``dma_mask`` field (line 25)
+defines the width of the DMA mask that is going to be used, and
+``coherent_dma_mask`` (line 26) has the same purpose but for the
+``alloc_coherent`` DMA mappings: in both cases we are using a 32 bits mask.
+Then the resource field (line 29) is simply a pointer to the resource
+structure defined before, while the ``num_resources`` field (line 28) keeps
+track of the number of arrays defined in the resource structure (in this
+case there were two resource arrays defined before).
+
+With this quick overview of the UDC platform data at the ``arch/`` level now
+done, let's get back to the MUSB glue layer specific platform data in
+``drivers/usb/musb/jz4740.c``:
+
+ .. code-block:: c
+ :emphasize-lines: 3,5,7-9,11
+
+ static struct musb_hdrc_config jz4740_musb_config = {
+ /* Silicon does not implement USB OTG. */
+ .multipoint = 0,
+ /* Max EPs scanned, driver will decide which EP can be used. */
+ .num_eps = 4,
+ /* RAMbits needed to configure EPs from table */
+ .ram_bits = 9,
+ .fifo_cfg = jz4740_musb_fifo_cfg,
+ .fifo_cfg_size = ARRAY_SIZE(jz4740_musb_fifo_cfg),
+ };
+
+ static struct musb_hdrc_platform_data jz4740_musb_platform_data = {
+ .mode = MUSB_PERIPHERAL,
+ .config = &jz4740_musb_config,
+ };
+
+First the glue layer configures some aspects of the controller driver
+operation related to the controller hardware specifics. This is done
+through the ``jz4740_musb_config`` :c:type:`musb_hdrc_config` structure.
+
+Defining the OTG capability of the controller hardware, the multipoint
+member (line 3) is set to 0 (equivalent to false) since the JZ4740 UDC
+is not OTG compatible. Then ``num_eps`` (line 5) defines the number of USB
+endpoints of the controller hardware, including endpoint 0: here we have
+3 endpoints + endpoint 0. Next is ``ram_bits`` (line 7) which is the width
+of the RAM address bus for the MUSB controller hardware. This
+information is needed when the controller driver cannot automatically
+configure endpoints by reading the relevant controller hardware
+registers. This issue will be discussed when we get to device quirks in
+:ref:`musb-dev-quirks`. Last two fields (line 8 and 9) are also
+about device quirks: ``fifo_cfg`` points to the USB endpoints configuration
+table and ``fifo_cfg_size`` keeps track of the size of the number of
+entries in that configuration table. More on that later in
+:ref:`musb-dev-quirks`.
+
+Then this configuration is embedded inside ``jz4740_musb_platform_data``
+:c:type:`musb_hdrc_platform_data` structure (line 11): config is a pointer to
+the configuration structure itself, and mode tells the controller driver
+if the controller hardware may be used as ``MUSB_HOST`` only,
+``MUSB_PERIPHERAL`` only or ``MUSB_OTG`` which is a dual mode.
+
+Remember that ``jz4740_musb_platform_data`` is then used to convey
+platform data information as we have seen in the probe function in
+:ref:`musb-basics`.
+
+.. _musb-dev-quirks:
+
+Device Quirks
+=============
+
+Completing the platform data specific to your device, you may also need
+to write some code in the glue layer to work around some device specific
+limitations. These quirks may be due to some hardware bugs, or simply be
+the result of an incomplete implementation of the USB On-the-Go
+specification.
+
+The JZ4740 UDC exhibits such quirks, some of which we will discuss here
+for the sake of insight even though these might not be found in the
+controller hardware you are working on.
+
+Let's get back to the init function first:
+
+ .. code-block:: c
+ :emphasize-lines: 12
+
+ static int jz4740_musb_init(struct musb *musb)
+ {
+ musb->xceiv = usb_get_phy(USB_PHY_TYPE_USB2);
+ if (!musb->xceiv) {
+ pr_err("HS UDC: no transceiver configured\n");
+ return -ENODEV;
+ }
+
+ /* Silicon does not implement ConfigData register.
+ * Set dyn_fifo to avoid reading EP config from hardware.
+ */
+ musb->dyn_fifo = true;
+
+ musb->isr = jz4740_musb_interrupt;
+
+ return 0;
+ }
+
+Instruction on line 12 helps the MUSB controller driver to work around
+the fact that the controller hardware is missing registers that are used
+for USB endpoints configuration.
+
+Without these registers, the controller driver is unable to read the
+endpoints configuration from the hardware, so we use line 12 instruction
+to bypass reading the configuration from silicon, and rely on a
+hard-coded table that describes the endpoints configuration instead::
+
+ static struct musb_fifo_cfg jz4740_musb_fifo_cfg[] = {
+ { .hw_ep_num = 1, .style = FIFO_TX, .maxpacket = 512, },
+ { .hw_ep_num = 1, .style = FIFO_RX, .maxpacket = 512, },
+ { .hw_ep_num = 2, .style = FIFO_TX, .maxpacket = 64, },
+ };
+
+Looking at the configuration table above, we see that each endpoints is
+described by three fields: ``hw_ep_num`` is the endpoint number, style is
+its direction (either ``FIFO_TX`` for the controller driver to send packets
+in the controller hardware, or ``FIFO_RX`` to receive packets from
+hardware), and maxpacket defines the maximum size of each data packet
+that can be transmitted over that endpoint. Reading from the table, the
+controller driver knows that endpoint 1 can be used to send and receive
+USB data packets of 512 bytes at once (this is in fact a bulk in/out
+endpoint), and endpoint 2 can be used to send data packets of 64 bytes
+at once (this is in fact an interrupt endpoint).
+
+Note that there is no information about endpoint 0 here: that one is
+implemented by default in every silicon design, with a predefined
+configuration according to the USB specification. For more examples of
+endpoint configuration tables, see ``musb_core.c``.
+
+Let's now get back to the interrupt handler function:
+
+ .. code-block:: c
+ :emphasize-lines: 18-19
+
+ static irqreturn_t jz4740_musb_interrupt(int irq, void *__hci)
+ {
+ unsigned long flags;
+ irqreturn_t retval = IRQ_NONE;
+ struct musb *musb = __hci;
+
+ spin_lock_irqsave(&musb->lock, flags);
+
+ musb->int_usb = musb_readb(musb->mregs, MUSB_INTRUSB);
+ musb->int_tx = musb_readw(musb->mregs, MUSB_INTRTX);
+ musb->int_rx = musb_readw(musb->mregs, MUSB_INTRRX);
+
+ /*
+ * The controller is gadget only, the state of the host mode IRQ bits is
+ * undefined. Mask them to make sure that the musb driver core will
+ * never see them set
+ */
+ musb->int_usb &= MUSB_INTR_SUSPEND | MUSB_INTR_RESUME |
+ MUSB_INTR_RESET | MUSB_INTR_SOF;
+
+ if (musb->int_usb || musb->int_tx || musb->int_rx)
+ retval = musb_interrupt(musb);
+
+ spin_unlock_irqrestore(&musb->lock, flags);
+
+ return retval;
+ }
+
+Instruction on line 18 above is a way for the controller driver to work
+around the fact that some interrupt bits used for USB host mode
+operation are missing in the ``MUSB_INTRUSB`` register, thus left in an
+undefined hardware state, since this MUSB controller hardware is used in
+peripheral mode only. As a consequence, the glue layer masks these
+missing bits out to avoid parasite interrupts by doing a logical AND
+operation between the value read from ``MUSB_INTRUSB`` and the bits that
+are actually implemented in the register.
+
+These are only a couple of the quirks found in the JZ4740 USB device
+controller. Some others were directly addressed in the MUSB core since
+the fixes were generic enough to provide a better handling of the issues
+for others controller hardware eventually.
+
+Conclusion
+==========
+
+Writing a Linux MUSB glue layer should be a more accessible task, as
+this documentation tries to show the ins and outs of this exercise.
+
+The JZ4740 USB device controller being fairly simple, I hope its glue
+layer serves as a good example for the curious mind. Used with the
+current MUSB glue layers, this documentation should provide enough
+guidance to get started; should anything gets out of hand, the linux-usb
+mailing list archive is another helpful resource to browse through.
+
+Acknowledgements
+================
+
+Many thanks to Lars-Peter Clausen and Maarten ter Huurne for answering
+my questions while I was writing the JZ4740 glue layer and for helping
+me out getting the code in good shape.
+
+I would also like to thank the Qi-Hardware community at large for its
+cheerful guidance and support.
+
+Resources
+=========
+
+USB Home Page: http://www.usb.org
+
+linux-usb Mailing List Archives: http://marc.info/?l=linux-usb
+
+USB On-the-Go Basics:
+http://www.maximintegrated.com/app-notes/index.mvp/id/1822
+
+:ref:`Writing USB Device Drivers <writing-usb-driver>`
+
+Texas Instruments USB Configuration Wiki Page:
+http://processors.wiki.ti.com/index.php/Usbgeneralpage
+
+Analog Devices Blackfin MUSB Configuration:
+http://docs.blackfin.uclinux.org/doku.php?id=linux-kernel:drivers:musb
diff --git a/Documentation/driver-api/usb/writing_usb_driver.rst b/Documentation/driver-api/usb/writing_usb_driver.rst
new file mode 100644
index 0000000..69f077d
--- /dev/null
+++ b/Documentation/driver-api/usb/writing_usb_driver.rst
@@ -0,0 +1,326 @@
+.. _writing-usb-driver:
+
+==========================
+Writing USB Device Drivers
+==========================
+
+:Author: Greg Kroah-Hartman
+
+Introduction
+============
+
+The Linux USB subsystem has grown from supporting only two different
+types of devices in the 2.2.7 kernel (mice and keyboards), to over 20
+different types of devices in the 2.4 kernel. Linux currently supports
+almost all USB class devices (standard types of devices like keyboards,
+mice, modems, printers and speakers) and an ever-growing number of
+vendor-specific devices (such as USB to serial converters, digital
+cameras, Ethernet devices and MP3 players). For a full list of the
+different USB devices currently supported, see Resources.
+
+The remaining kinds of USB devices that do not have support on Linux are
+almost all vendor-specific devices. Each vendor decides to implement a
+custom protocol to talk to their device, so a custom driver usually
+needs to be created. Some vendors are open with their USB protocols and
+help with the creation of Linux drivers, while others do not publish
+them, and developers are forced to reverse-engineer. See Resources for
+some links to handy reverse-engineering tools.
+
+Because each different protocol causes a new driver to be created, I
+have written a generic USB driver skeleton, modelled after the
+pci-skeleton.c file in the kernel source tree upon which many PCI
+network drivers have been based. This USB skeleton can be found at
+drivers/usb/usb-skeleton.c in the kernel source tree. In this article I
+will walk through the basics of the skeleton driver, explaining the
+different pieces and what needs to be done to customize it to your
+specific device.
+
+Linux USB Basics
+================
+
+If you are going to write a Linux USB driver, please become familiar
+with the USB protocol specification. It can be found, along with many
+other useful documents, at the USB home page (see Resources). An
+excellent introduction to the Linux USB subsystem can be found at the
+USB Working Devices List (see Resources). It explains how the Linux USB
+subsystem is structured and introduces the reader to the concept of USB
+urbs (USB Request Blocks), which are essential to USB drivers.
+
+The first thing a Linux USB driver needs to do is register itself with
+the Linux USB subsystem, giving it some information about which devices
+the driver supports and which functions to call when a device supported
+by the driver is inserted or removed from the system. All of this
+information is passed to the USB subsystem in the :c:type:`usb_driver`
+structure. The skeleton driver declares a :c:type:`usb_driver` as::
+
+ static struct usb_driver skel_driver = {
+ .name = "skeleton",
+ .probe = skel_probe,
+ .disconnect = skel_disconnect,
+ .fops = &skel_fops,
+ .minor = USB_SKEL_MINOR_BASE,
+ .id_table = skel_table,
+ };
+
+
+The variable name is a string that describes the driver. It is used in
+informational messages printed to the system log. The probe and
+disconnect function pointers are called when a device that matches the
+information provided in the ``id_table`` variable is either seen or
+removed.
+
+The fops and minor variables are optional. Most USB drivers hook into
+another kernel subsystem, such as the SCSI, network or TTY subsystem.
+These types of drivers register themselves with the other kernel
+subsystem, and any user-space interactions are provided through that
+interface. But for drivers that do not have a matching kernel subsystem,
+such as MP3 players or scanners, a method of interacting with user space
+is needed. The USB subsystem provides a way to register a minor device
+number and a set of :c:type:`file_operations` function pointers that enable
+this user-space interaction. The skeleton driver needs this kind of
+interface, so it provides a minor starting number and a pointer to its
+:c:type:`file_operations` functions.
+
+The USB driver is then registered with a call to :c:func:`usb_register`,
+usually in the driver's init function, as shown here::
+
+ static int __init usb_skel_init(void)
+ {
+ int result;
+
+ /* register this driver with the USB subsystem */
+ result = usb_register(&skel_driver);
+ if (result < 0) {
+ err("usb_register failed for the "__FILE__ "driver."
+ "Error number %d", result);
+ return -1;
+ }
+
+ return 0;
+ }
+ module_init(usb_skel_init);
+
+
+When the driver is unloaded from the system, it needs to deregister
+itself with the USB subsystem. This is done with the :c:func:`usb_deregister`
+function::
+
+ static void __exit usb_skel_exit(void)
+ {
+ /* deregister this driver with the USB subsystem */
+ usb_deregister(&skel_driver);
+ }
+ module_exit(usb_skel_exit);
+
+
+To enable the linux-hotplug system to load the driver automatically when
+the device is plugged in, you need to create a ``MODULE_DEVICE_TABLE``.
+The following code tells the hotplug scripts that this module supports a
+single device with a specific vendor and product ID::
+
+ /* table of devices that work with this driver */
+ static struct usb_device_id skel_table [] = {
+ { USB_DEVICE(USB_SKEL_VENDOR_ID, USB_SKEL_PRODUCT_ID) },
+ { } /* Terminating entry */
+ };
+ MODULE_DEVICE_TABLE (usb, skel_table);
+
+
+There are other macros that can be used in describing a struct
+:c:type:`usb_device_id` for drivers that support a whole class of USB
+drivers. See :ref:`usb.h <usb_header>` for more information on this.
+
+Device operation
+================
+
+When a device is plugged into the USB bus that matches the device ID
+pattern that your driver registered with the USB core, the probe
+function is called. The :c:type:`usb_device` structure, interface number and
+the interface ID are passed to the function::
+
+ static int skel_probe(struct usb_interface *interface,
+ const struct usb_device_id *id)
+
+
+The driver now needs to verify that this device is actually one that it
+can accept. If so, it returns 0. If not, or if any error occurs during
+initialization, an errorcode (such as ``-ENOMEM`` or ``-ENODEV``) is
+returned from the probe function.
+
+In the skeleton driver, we determine what end points are marked as
+bulk-in and bulk-out. We create buffers to hold the data that will be
+sent and received from the device, and a USB urb to write data to the
+device is initialized.
+
+Conversely, when the device is removed from the USB bus, the disconnect
+function is called with the device pointer. The driver needs to clean
+any private data that has been allocated at this time and to shut down
+any pending urbs that are in the USB system.
+
+Now that the device is plugged into the system and the driver is bound
+to the device, any of the functions in the :c:type:`file_operations` structure
+that were passed to the USB subsystem will be called from a user program
+trying to talk to the device. The first function called will be open, as
+the program tries to open the device for I/O. We increment our private
+usage count and save a pointer to our internal structure in the file
+structure. This is done so that future calls to file operations will
+enable the driver to determine which device the user is addressing. All
+of this is done with the following code::
+
+ /* increment our usage count for the module */
+ ++skel->open_count;
+
+ /* save our object in the file's private structure */
+ file->private_data = dev;
+
+
+After the open function is called, the read and write functions are
+called to receive and send data to the device. In the ``skel_write``
+function, we receive a pointer to some data that the user wants to send
+to the device and the size of the data. The function determines how much
+data it can send to the device based on the size of the write urb it has
+created (this size depends on the size of the bulk out end point that
+the device has). Then it copies the data from user space to kernel
+space, points the urb to the data and submits the urb to the USB
+subsystem. This can be seen in the following code::
+
+ /* we can only write as much as 1 urb will hold */
+ bytes_written = (count > skel->bulk_out_size) ? skel->bulk_out_size : count;
+
+ /* copy the data from user space into our urb */
+ copy_from_user(skel->write_urb->transfer_buffer, buffer, bytes_written);
+
+ /* set up our urb */
+ usb_fill_bulk_urb(skel->write_urb,
+ skel->dev,
+ usb_sndbulkpipe(skel->dev, skel->bulk_out_endpointAddr),
+ skel->write_urb->transfer_buffer,
+ bytes_written,
+ skel_write_bulk_callback,
+ skel);
+
+ /* send the data out the bulk port */
+ result = usb_submit_urb(skel->write_urb);
+ if (result) {
+ err("Failed submitting write urb, error %d", result);
+ }
+
+
+When the write urb is filled up with the proper information using the
+:c:func:`usb_fill_bulk_urb` function, we point the urb's completion callback
+to call our own ``skel_write_bulk_callback`` function. This function is
+called when the urb is finished by the USB subsystem. The callback
+function is called in interrupt context, so caution must be taken not to
+do very much processing at that time. Our implementation of
+``skel_write_bulk_callback`` merely reports if the urb was completed
+successfully or not and then returns.
+
+The read function works a bit differently from the write function in
+that we do not use an urb to transfer data from the device to the
+driver. Instead we call the :c:func:`usb_bulk_msg` function, which can be used
+to send or receive data from a device without having to create urbs and
+handle urb completion callback functions. We call the :c:func:`usb_bulk_msg`
+function, giving it a buffer into which to place any data received from
+the device and a timeout value. If the timeout period expires without
+receiving any data from the device, the function will fail and return an
+error message. This can be shown with the following code::
+
+ /* do an immediate bulk read to get data from the device */
+ retval = usb_bulk_msg (skel->dev,
+ usb_rcvbulkpipe (skel->dev,
+ skel->bulk_in_endpointAddr),
+ skel->bulk_in_buffer,
+ skel->bulk_in_size,
+ &count, HZ*10);
+ /* if the read was successful, copy the data to user space */
+ if (!retval) {
+ if (copy_to_user (buffer, skel->bulk_in_buffer, count))
+ retval = -EFAULT;
+ else
+ retval = count;
+ }
+
+
+The :c:func:`usb_bulk_msg` function can be very useful for doing single reads
+or writes to a device; however, if you need to read or write constantly to
+a device, it is recommended to set up your own urbs and submit them to
+the USB subsystem.
+
+When the user program releases the file handle that it has been using to
+talk to the device, the release function in the driver is called. In
+this function we decrement our private usage count and wait for possible
+pending writes::
+
+ /* decrement our usage count for the device */
+ --skel->open_count;
+
+
+One of the more difficult problems that USB drivers must be able to
+handle smoothly is the fact that the USB device may be removed from the
+system at any point in time, even if a program is currently talking to
+it. It needs to be able to shut down any current reads and writes and
+notify the user-space programs that the device is no longer there. The
+following code (function ``skel_delete``) is an example of how to do
+this::
+
+ static inline void skel_delete (struct usb_skel *dev)
+ {
+ kfree (dev->bulk_in_buffer);
+ if (dev->bulk_out_buffer != NULL)
+ usb_free_coherent (dev->udev, dev->bulk_out_size,
+ dev->bulk_out_buffer,
+ dev->write_urb->transfer_dma);
+ usb_free_urb (dev->write_urb);
+ kfree (dev);
+ }
+
+
+If a program currently has an open handle to the device, we reset the
+flag ``device_present``. For every read, write, release and other
+functions that expect a device to be present, the driver first checks
+this flag to see if the device is still present. If not, it releases
+that the device has disappeared, and a ``-ENODEV`` error is returned to the
+user-space program. When the release function is eventually called, it
+determines if there is no device and if not, it does the cleanup that
+the ``skel_disconnect`` function normally does if there are no open files
+on the device (see Listing 5).
+
+Isochronous Data
+================
+
+This usb-skeleton driver does not have any examples of interrupt or
+isochronous data being sent to or from the device. Interrupt data is
+sent almost exactly as bulk data is, with a few minor exceptions.
+Isochronous data works differently with continuous streams of data being
+sent to or from the device. The audio and video camera drivers are very
+good examples of drivers that handle isochronous data and will be useful
+if you also need to do this.
+
+Conclusion
+==========
+
+Writing Linux USB device drivers is not a difficult task as the
+usb-skeleton driver shows. This driver, combined with the other current
+USB drivers, should provide enough examples to help a beginning author
+create a working driver in a minimal amount of time. The linux-usb-devel
+mailing list archives also contain a lot of helpful information.
+
+Resources
+=========
+
+The Linux USB Project:
+http://www.linux-usb.org/
+
+Linux Hotplug Project:
+http://linux-hotplug.sourceforge.net/
+
+Linux USB Working Devices List:
+http://www.qbik.ch/usb/devices/
+
+linux-usb-devel Mailing List Archives:
+http://marc.theaimsgroup.com/?l=linux-usb-devel
+
+Programming Guide for Linux USB Device Drivers:
+http://usb.cs.tum.edu/usbdoc
+
+USB Home Page: http://www.usb.org
diff --git a/Documentation/early-userspace/README b/Documentation/early-userspace/README
index 93e63a9..2c00b07 100644
--- a/Documentation/early-userspace/README
+++ b/Documentation/early-userspace/README
@@ -86,7 +86,7 @@
maintained separately from the kernel.
You can obtain somewhat infrequent snapshots of klibc from
-ftp://ftp.kernel.org/pub/linux/libs/klibc/
+https://www.kernel.org/pub/linux/libs/klibc/
For active users, you are better off using the klibc git
repository, at http://git.kernel.org/?p=libs/klibc/klibc.git
diff --git a/Documentation/features/core/BPF-JIT/arch-support.txt b/Documentation/features/core/BPF-JIT/arch-support.txt
index c1b4f91..5575d2d 100644
--- a/Documentation/features/core/BPF-JIT/arch-support.txt
+++ b/Documentation/features/core/BPF-JIT/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/core/generic-idle-thread/arch-support.txt b/Documentation/features/core/generic-idle-thread/arch-support.txt
index 6d930fc..abb5f27 100644
--- a/Documentation/features/core/generic-idle-thread/arch-support.txt
+++ b/Documentation/features/core/generic-idle-thread/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | ok |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | ok |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/core/jump-labels/arch-support.txt b/Documentation/features/core/jump-labels/arch-support.txt
index 136868b..dbdaffc 100644
--- a/Documentation/features/core/jump-labels/arch-support.txt
+++ b/Documentation/features/core/jump-labels/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/core/tracehook/arch-support.txt b/Documentation/features/core/tracehook/arch-support.txt
index 728061d..5e97a89 100644
--- a/Documentation/features/core/tracehook/arch-support.txt
+++ b/Documentation/features/core/tracehook/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | ok |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | ok |
| c6x: | ok |
| cris: | TODO |
diff --git a/Documentation/features/debug/KASAN/arch-support.txt b/Documentation/features/debug/KASAN/arch-support.txt
index 703f5784..76bbd7f 100644
--- a/Documentation/features/debug/KASAN/arch-support.txt
+++ b/Documentation/features/debug/KASAN/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | TODO |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/debug/gcov-profile-all/arch-support.txt b/Documentation/features/debug/gcov-profile-all/arch-support.txt
index 38dea8e..830dbe8 100644
--- a/Documentation/features/debug/gcov-profile-all/arch-support.txt
+++ b/Documentation/features/debug/gcov-profile-all/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/debug/kgdb/arch-support.txt b/Documentation/features/debug/kgdb/arch-support.txt
index 862e15d..0217bf6 100644
--- a/Documentation/features/debug/kgdb/arch-support.txt
+++ b/Documentation/features/debug/kgdb/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | ok |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | ok |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/debug/kprobes-on-ftrace/arch-support.txt b/Documentation/features/debug/kprobes-on-ftrace/arch-support.txt
index 40f44d0..f9133a9 100644
--- a/Documentation/features/debug/kprobes-on-ftrace/arch-support.txt
+++ b/Documentation/features/debug/kprobes-on-ftrace/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | TODO |
| arm64: | TODO |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/debug/kprobes/arch-support.txt b/Documentation/features/debug/kprobes/arch-support.txt
index a44bfff..529f66e 100644
--- a/Documentation/features/debug/kprobes/arch-support.txt
+++ b/Documentation/features/debug/kprobes/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | ok |
| arm: | ok |
| arm64: | TODO |
- | avr32: | ok |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/debug/kretprobes/arch-support.txt b/Documentation/features/debug/kretprobes/arch-support.txt
index d87c1ce..4335324 100644
--- a/Documentation/features/debug/kretprobes/arch-support.txt
+++ b/Documentation/features/debug/kretprobes/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | ok |
| arm: | ok |
| arm64: | TODO |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/debug/optprobes/arch-support.txt b/Documentation/features/debug/optprobes/arch-support.txt
index b8999d8..f559f1b 100644
--- a/Documentation/features/debug/optprobes/arch-support.txt
+++ b/Documentation/features/debug/optprobes/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | TODO |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/debug/stackprotector/arch-support.txt b/Documentation/features/debug/stackprotector/arch-support.txt
index 0fa4233..d7acd7b 100644
--- a/Documentation/features/debug/stackprotector/arch-support.txt
+++ b/Documentation/features/debug/stackprotector/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/debug/uprobes/arch-support.txt b/Documentation/features/debug/uprobes/arch-support.txt
index d605c3f..53ed42b 100644
--- a/Documentation/features/debug/uprobes/arch-support.txt
+++ b/Documentation/features/debug/uprobes/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | TODO |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/debug/user-ret-profiler/arch-support.txt b/Documentation/features/debug/user-ret-profiler/arch-support.txt
index 44cc1ff..1494439 100644
--- a/Documentation/features/debug/user-ret-profiler/arch-support.txt
+++ b/Documentation/features/debug/user-ret-profiler/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | TODO |
| arm64: | TODO |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/io/dma-api-debug/arch-support.txt b/Documentation/features/io/dma-api-debug/arch-support.txt
index ffa522a..6be9206 100644
--- a/Documentation/features/io/dma-api-debug/arch-support.txt
+++ b/Documentation/features/io/dma-api-debug/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | ok |
| cris: | TODO |
diff --git a/Documentation/features/io/dma-contiguous/arch-support.txt b/Documentation/features/io/dma-contiguous/arch-support.txt
index 83d2cf9..0eb08e1 100644
--- a/Documentation/features/io/dma-contiguous/arch-support.txt
+++ b/Documentation/features/io/dma-contiguous/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/io/sg-chain/arch-support.txt b/Documentation/features/io/sg-chain/arch-support.txt
index 6ca98f9..514ad34 100644
--- a/Documentation/features/io/sg-chain/arch-support.txt
+++ b/Documentation/features/io/sg-chain/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | ok |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/lib/strncasecmp/arch-support.txt b/Documentation/features/lib/strncasecmp/arch-support.txt
index 12b1c93..532c6f0 100644
--- a/Documentation/features/lib/strncasecmp/arch-support.txt
+++ b/Documentation/features/lib/strncasecmp/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | TODO |
| arm64: | TODO |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/locking/cmpxchg-local/arch-support.txt b/Documentation/features/locking/cmpxchg-local/arch-support.txt
index d9c3108..f3eec26 100644
--- a/Documentation/features/locking/cmpxchg-local/arch-support.txt
+++ b/Documentation/features/locking/cmpxchg-local/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | TODO |
| arm64: | TODO |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/locking/lockdep/arch-support.txt b/Documentation/features/locking/lockdep/arch-support.txt
index cf90635..9756abc 100644
--- a/Documentation/features/locking/lockdep/arch-support.txt
+++ b/Documentation/features/locking/lockdep/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | ok |
| arm: | ok |
| arm64: | ok |
- | avr32: | ok |
| blackfin: | ok |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/locking/queued-rwlocks/arch-support.txt b/Documentation/features/locking/queued-rwlocks/arch-support.txt
index 68c3a5d..62f4ee5 100644
--- a/Documentation/features/locking/queued-rwlocks/arch-support.txt
+++ b/Documentation/features/locking/queued-rwlocks/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | TODO |
| arm64: | TODO |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/locking/queued-spinlocks/arch-support.txt b/Documentation/features/locking/queued-spinlocks/arch-support.txt
index e973b1a9..321b32f 100644
--- a/Documentation/features/locking/queued-spinlocks/arch-support.txt
+++ b/Documentation/features/locking/queued-spinlocks/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | TODO |
| arm64: | TODO |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/locking/rwsem-optimized/arch-support.txt b/Documentation/features/locking/rwsem-optimized/arch-support.txt
index ac93d7a..79bfa4d 100644
--- a/Documentation/features/locking/rwsem-optimized/arch-support.txt
+++ b/Documentation/features/locking/rwsem-optimized/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | TODO |
| arm64: | TODO |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/perf/kprobes-event/arch-support.txt b/Documentation/features/perf/kprobes-event/arch-support.txt
index 4660bf2..00f1606 100644
--- a/Documentation/features/perf/kprobes-event/arch-support.txt
+++ b/Documentation/features/perf/kprobes-event/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | TODO |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/perf/perf-regs/arch-support.txt b/Documentation/features/perf/perf-regs/arch-support.txt
index f179b1f..7d516ea 100644
--- a/Documentation/features/perf/perf-regs/arch-support.txt
+++ b/Documentation/features/perf/perf-regs/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/perf/perf-stackdump/arch-support.txt b/Documentation/features/perf/perf-stackdump/arch-support.txt
index 85777c5..f974b8d 100644
--- a/Documentation/features/perf/perf-stackdump/arch-support.txt
+++ b/Documentation/features/perf/perf-stackdump/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/sched/numa-balancing/arch-support.txt b/Documentation/features/sched/numa-balancing/arch-support.txt
index ac7cd6b..1d3c0f6 100644
--- a/Documentation/features/sched/numa-balancing/arch-support.txt
+++ b/Documentation/features/sched/numa-balancing/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | .. |
| arm: | .. |
| arm64: | .. |
- | avr32: | .. |
| blackfin: | .. |
| c6x: | .. |
| cris: | .. |
diff --git a/Documentation/features/seccomp/seccomp-filter/arch-support.txt b/Documentation/features/seccomp/seccomp-filter/arch-support.txt
index 4f66ec1..a32d5b2 100644
--- a/Documentation/features/seccomp/seccomp-filter/arch-support.txt
+++ b/Documentation/features/seccomp/seccomp-filter/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/time/arch-tick-broadcast/arch-support.txt b/Documentation/features/time/arch-tick-broadcast/arch-support.txt
index 8acb439..caee8f6 100644
--- a/Documentation/features/time/arch-tick-broadcast/arch-support.txt
+++ b/Documentation/features/time/arch-tick-broadcast/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/time/clockevents/arch-support.txt b/Documentation/features/time/clockevents/arch-support.txt
index ff670b2..1cd87f6 100644
--- a/Documentation/features/time/clockevents/arch-support.txt
+++ b/Documentation/features/time/clockevents/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | ok |
| arm: | ok |
| arm64: | ok |
- | avr32: | ok |
| blackfin: | ok |
| c6x: | ok |
| cris: | ok |
diff --git a/Documentation/features/time/context-tracking/arch-support.txt b/Documentation/features/time/context-tracking/arch-support.txt
index a1e3eea..e6d7c7b 100644
--- a/Documentation/features/time/context-tracking/arch-support.txt
+++ b/Documentation/features/time/context-tracking/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/time/irq-time-acct/arch-support.txt b/Documentation/features/time/irq-time-acct/arch-support.txt
index 4199ffec..15c6071 100644
--- a/Documentation/features/time/irq-time-acct/arch-support.txt
+++ b/Documentation/features/time/irq-time-acct/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/time/modern-timekeeping/arch-support.txt b/Documentation/features/time/modern-timekeeping/arch-support.txt
index 17f68a0..baee761 100644
--- a/Documentation/features/time/modern-timekeeping/arch-support.txt
+++ b/Documentation/features/time/modern-timekeeping/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | ok |
| arm: | TODO |
| arm64: | ok |
- | avr32: | ok |
| blackfin: | TODO |
| c6x: | ok |
| cris: | TODO |
diff --git a/Documentation/features/time/virt-cpuacct/arch-support.txt b/Documentation/features/time/virt-cpuacct/arch-support.txt
index cf3c3e3..9129530 100644
--- a/Documentation/features/time/virt-cpuacct/arch-support.txt
+++ b/Documentation/features/time/virt-cpuacct/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/vm/ELF-ASLR/arch-support.txt b/Documentation/features/vm/ELF-ASLR/arch-support.txt
index ec4dd28..f6829af 100644
--- a/Documentation/features/vm/ELF-ASLR/arch-support.txt
+++ b/Documentation/features/vm/ELF-ASLR/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/vm/PG_uncached/arch-support.txt b/Documentation/features/vm/PG_uncached/arch-support.txt
index 9919742..1a09ea9 100644
--- a/Documentation/features/vm/PG_uncached/arch-support.txt
+++ b/Documentation/features/vm/PG_uncached/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | TODO |
| arm64: | TODO |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/vm/THP/arch-support.txt b/Documentation/features/vm/THP/arch-support.txt
index 523f830..d170e62 100644
--- a/Documentation/features/vm/THP/arch-support.txt
+++ b/Documentation/features/vm/THP/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | ok |
| arm: | ok |
| arm64: | ok |
- | avr32: | .. |
| blackfin: | .. |
| c6x: | .. |
| cris: | .. |
diff --git a/Documentation/features/vm/TLB/arch-support.txt b/Documentation/features/vm/TLB/arch-support.txt
index 261b92e..abfab40 100644
--- a/Documentation/features/vm/TLB/arch-support.txt
+++ b/Documentation/features/vm/TLB/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | TODO |
| arm64: | TODO |
- | avr32: | .. |
| blackfin: | TODO |
| c6x: | .. |
| cris: | .. |
diff --git a/Documentation/features/vm/huge-vmap/arch-support.txt b/Documentation/features/vm/huge-vmap/arch-support.txt
index df1d1f3..f81f09b 100644
--- a/Documentation/features/vm/huge-vmap/arch-support.txt
+++ b/Documentation/features/vm/huge-vmap/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | TODO |
| arm: | TODO |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/vm/ioremap_prot/arch-support.txt b/Documentation/features/vm/ioremap_prot/arch-support.txt
index 90c5374..0cc3e11 100644
--- a/Documentation/features/vm/ioremap_prot/arch-support.txt
+++ b/Documentation/features/vm/ioremap_prot/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | ok |
| arm: | TODO |
| arm64: | TODO |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/features/vm/numa-memblock/arch-support.txt b/Documentation/features/vm/numa-memblock/arch-support.txt
index e7c252a..9a3fdac 100644
--- a/Documentation/features/vm/numa-memblock/arch-support.txt
+++ b/Documentation/features/vm/numa-memblock/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | .. |
| arm: | .. |
| arm64: | .. |
- | avr32: | .. |
| blackfin: | .. |
| c6x: | .. |
| cris: | .. |
diff --git a/Documentation/features/vm/pte_special/arch-support.txt b/Documentation/features/vm/pte_special/arch-support.txt
index 3de5434..dfaa39e 100644
--- a/Documentation/features/vm/pte_special/arch-support.txt
+++ b/Documentation/features/vm/pte_special/arch-support.txt
@@ -10,7 +10,6 @@
| arc: | ok |
| arm: | ok |
| arm64: | ok |
- | avr32: | TODO |
| blackfin: | TODO |
| c6x: | TODO |
| cris: | TODO |
diff --git a/Documentation/filesystems/ext4.txt b/Documentation/filesystems/ext4.txt
index 3698ed3..5a8f7f4 100644
--- a/Documentation/filesystems/ext4.txt
+++ b/Documentation/filesystems/ext4.txt
@@ -25,7 +25,7 @@
or
- ftp://ftp.kernel.org/pub/linux/kernel/people/tytso/e2fsprogs/
+ https://www.kernel.org/pub/linux/kernel/people/tytso/e2fsprogs/
or grab the latest git repository from:
diff --git a/Documentation/filesystems/nfs/nfs-rdma.txt b/Documentation/filesystems/nfs/nfs-rdma.txt
index 1e65645..22dc0dd 100644
--- a/Documentation/filesystems/nfs/nfs-rdma.txt
+++ b/Documentation/filesystems/nfs/nfs-rdma.txt
@@ -110,10 +110,10 @@
- Install a Linux kernel with NFS/RDMA
The NFS/RDMA client and server are both included in the mainline Linux
- kernel version 2.6.25 and later. This and other versions of the 2.6 Linux
+ kernel version 2.6.25 and later. This and other versions of the Linux
kernel can be found at:
- ftp://ftp.kernel.org/pub/linux/kernel/v2.6/
+ https://www.kernel.org/pub/linux/kernel/
Download the sources and place them in an appropriate location.
diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt
index c94b467..9036dbf 100644
--- a/Documentation/filesystems/proc.txt
+++ b/Documentation/filesystems/proc.txt
@@ -44,6 +44,7 @@
3.8 /proc/<pid>/fdinfo/<fd> - Information about opened file
3.9 /proc/<pid>/map_files - Information about memory mapped files
3.10 /proc/<pid>/timerslack_ns - Task timerslack value
+ 3.11 /proc/<pid>/patch_state - Livepatch patch operation state
4 Configuring procfs
4.1 Mount options
@@ -1887,6 +1888,23 @@
An application setting the value must have PTRACE_MODE_ATTACH_FSCREDS level
permissions on the task specified to change its timerslack_ns value.
+3.11 /proc/<pid>/patch_state - Livepatch patch operation state
+-----------------------------------------------------------------
+When CONFIG_LIVEPATCH is enabled, this file displays the value of the
+patch state for the task.
+
+A value of '-1' indicates that no patch is in transition.
+
+A value of '0' indicates that a patch is in transition and the task is
+unpatched. If the patch is being enabled, then the task hasn't been
+patched yet. If the patch is being disabled, then the task has already
+been unpatched.
+
+A value of '1' indicates that a patch is in transition and the task is
+patched. If the patch is being enabled, then the task has already been
+patched. If the patch is being disabled, then the task hasn't been
+unpatched yet.
+
------------------------------------------------------------------------------
Configuring procfs
diff --git a/Documentation/hid/hidraw.txt b/Documentation/hid/hidraw.txt
index 029e6cb..c8436e3 100644
--- a/Documentation/hid/hidraw.txt
+++ b/Documentation/hid/hidraw.txt
@@ -81,7 +81,7 @@
BUS_HIL
BUS_BLUETOOTH
BUS_VIRTUAL
-which are defined in linux/input.h.
+which are defined in uapi/linux/input.h.
HIDIOCGRAWNAME(len): Get Raw Name
This ioctl returns a string containing the vendor and product strings of
diff --git a/Documentation/hwmon/aspeed-pwm-tacho b/Documentation/hwmon/aspeed-pwm-tacho
new file mode 100644
index 0000000..7cfb349
--- /dev/null
+++ b/Documentation/hwmon/aspeed-pwm-tacho
@@ -0,0 +1,22 @@
+Kernel driver aspeed-pwm-tacho
+==============================
+
+Supported chips:
+ ASPEED AST2400/2500
+
+Authors:
+ <jaghu@google.com>
+
+Description:
+------------
+This driver implements support for ASPEED AST2400/2500 PWM and Fan Tacho
+controller. The PWM controller supports upto 8 PWM outputs. The Fan tacho
+controller supports up to 16 tachometer inputs.
+
+The driver provides the following sensor accesses in sysfs:
+
+fanX_input ro provide current fan rotation value in RPM as reported
+ by the fan to the device.
+
+pwmX rw get or set PWM fan control value. This is an integer
+ value between 0(off) and 255(full speed).
diff --git a/Documentation/hwmon/tc654 b/Documentation/hwmon/tc654
index 91a2843..47636a8 100644
--- a/Documentation/hwmon/tc654
+++ b/Documentation/hwmon/tc654
@@ -2,7 +2,7 @@
===================
Supported chips:
- * Microship TC654 and TC655
+ * Microchip TC654 and TC655
Prefix: 'tc654'
Datasheet: http://ww1.microchip.com/downloads/en/DeviceDoc/20001734C.pdf
diff --git a/Documentation/index.rst b/Documentation/index.rst
index f6e641a..61306a2 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -24,6 +24,18 @@
admin-guide/index
+Application-developer documentation
+-----------------------------------
+
+The user-space API manual gathers together documents describing aspects of
+the kernel interface as seen by application developers.
+
+.. toctree::
+ :maxdepth: 2
+
+ userspace-api/index
+
+
Introduction to kernel development
----------------------------------
@@ -76,6 +88,14 @@
translations/zh_CN/index
+Japanese translations
+---------------------
+
+.. toctree::
+ :maxdepth: 1
+
+ translations/ja_JP/index
+
Indices and tables
==================
diff --git a/Documentation/input/event-codes.txt b/Documentation/input/event-codes.txt
index 36ea940..575415f 100644
--- a/Documentation/input/event-codes.txt
+++ b/Documentation/input/event-codes.txt
@@ -301,7 +301,10 @@
INPUT_PROP_ACCELEROMETER
-------------------------
Directional axes on this device (absolute and/or relative x, y, z) represent
-accelerometer data. All other axes retain their meaning. A device must not mix
+accelerometer data. Some devices also report gyroscope data, which devices
+can report through the rotational axes (absolute and/or relative rx, ry, rz).
+
+All other axes retain their meaning. A device must not mix
regular directional axes and accelerometer axes on the same event node.
Guidelines:
diff --git a/Documentation/input/ff.txt b/Documentation/input/ff.txt
index b3867bf..be742ee 100644
--- a/Documentation/input/ff.txt
+++ b/Documentation/input/ff.txt
@@ -106,9 +106,9 @@
Effects are file descriptor specific.
-See <linux/input.h> for a description of the ff_effect struct. You should also
-find help in a few sketches, contained in files shape.fig and interactive.fig.
-You need xfig to visualize these files.
+See <uapi/linux/input.h> for a description of the ff_effect struct. You should
+also find help in a few sketches, contained in files shape.fig and
+interactive.fig. You need xfig to visualize these files.
3.3 Removing an effect from the device
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/Documentation/input/input-programming.txt b/Documentation/input/input-programming.txt
index 7f8b9d9..c3b940b 100644
--- a/Documentation/input/input-programming.txt
+++ b/Documentation/input/input-programming.txt
@@ -174,8 +174,8 @@
input_report_key(struct input_dev *dev, int code, int value)
-See linux/input.h for the allowable values of code (from 0 to KEY_MAX).
-Value is interpreted as a truth value, ie any nonzero value means key
+See uapi/linux/input-event-codes.h for the allowable values of code (from 0 to
+KEY_MAX). Value is interpreted as a truth value, ie any nonzero value means key
pressed, zero value means key released. The input code generates events only
in case the value is different from before.
diff --git a/Documentation/ioctl/ioctl-number.txt b/Documentation/ioctl/ioctl-number.txt
index 08244be..a77ead9 100644
--- a/Documentation/ioctl/ioctl-number.txt
+++ b/Documentation/ioctl/ioctl-number.txt
@@ -212,7 +212,7 @@
'c' 00-1F linux/chio.h conflict!
'c' 80-9F arch/s390/include/asm/chsc.h conflict!
'c' A0-AF arch/x86/include/asm/msr.h conflict!
-'d' 00-FF linux/char/drm/drm/h conflict!
+'d' 00-FF linux/char/drm/drm.h conflict!
'd' 02-40 pcmcia/ds.h conflict!
'd' F0-FF linux/digi1.h
'e' all linux/digi1.h conflict!
diff --git a/Documentation/leds/leds-lp55xx.txt b/Documentation/leds/leds-lp55xx.txt
index bcea12a..e23fa91 100644
--- a/Documentation/leds/leds-lp55xx.txt
+++ b/Documentation/leds/leds-lp55xx.txt
@@ -117,7 +117,7 @@
Inside the callback, the selected engine is loaded and memory is updated.
To run programmed pattern, 'run_engine' attribute should be enabled.
-The pattern sqeuence of LP8501 is similar to LP5523.
+The pattern sequence of LP8501 is similar to LP5523.
However pattern data is specific.
Ex 1) Engine 1 is used
echo 1 > /sys/bus/i2c/devices/xxxx/select_engine
diff --git a/Documentation/lightnvm/pblk.txt b/Documentation/lightnvm/pblk.txt
new file mode 100644
index 0000000..1040ed1
--- /dev/null
+++ b/Documentation/lightnvm/pblk.txt
@@ -0,0 +1,21 @@
+pblk: Physical Block Device Target
+==================================
+
+pblk implements a fully associative, host-based FTL that exposes a traditional
+block I/O interface. Its primary responsibilities are:
+
+ - Map logical addresses onto physical addresses (4KB granularity) in a
+ logical-to-physical (L2P) table.
+ - Maintain the integrity and consistency of the L2P table as well as its
+ recovery from normal tear down and power outage.
+ - Deal with controller- and media-specific constrains.
+ - Handle I/O errors.
+ - Implement garbage collection.
+ - Maintain consistency across the I/O stack during synchronization points.
+
+For more information please refer to:
+
+ http://lightnvm.io
+
+which maintains updated FAQs, manual pages, technical documentation, tools,
+contacts, etc.
diff --git a/Documentation/livepatch/livepatch.txt b/Documentation/livepatch/livepatch.txt
index 9d2096c..ecdb181 100644
--- a/Documentation/livepatch/livepatch.txt
+++ b/Documentation/livepatch/livepatch.txt
@@ -72,7 +72,8 @@
a missing memory barrier, or add some locking around a critical section.
Most of these changes are self contained and the function presents itself
the same way to the rest of the system. In this case, the functions might
-be updated independently one by one.
+be updated independently one by one. (This can be done by setting the
+'immediate' flag in the klp_patch struct.)
But there are more complex fixes. For example, a patch might change
ordering of locking in multiple functions at the same time. Or a patch
@@ -86,20 +87,141 @@
The theory about how to apply functions a safe way is rather complex.
The aim is to define a so-called consistency model. It attempts to define
conditions when the new implementation could be used so that the system
-stays consistent. The theory is not yet finished. See the discussion at
-https://lkml.kernel.org/r/20141107140458.GA21774@suse.cz
+stays consistent.
-The current consistency model is very simple. It guarantees that either
-the old or the new function is called. But various functions get redirected
-one by one without any synchronization.
+Livepatch has a consistency model which is a hybrid of kGraft and
+kpatch: it uses kGraft's per-task consistency and syscall barrier
+switching combined with kpatch's stack trace switching. There are also
+a number of fallback options which make it quite flexible.
-In other words, the current implementation _never_ modifies the behavior
-in the middle of the call. It is because it does _not_ rewrite the entire
-function in the memory. Instead, the function gets redirected at the
-very beginning. But this redirection is used immediately even when
-some other functions from the same patch have not been redirected yet.
+Patches are applied on a per-task basis, when the task is deemed safe to
+switch over. When a patch is enabled, livepatch enters into a
+transition state where tasks are converging to the patched state.
+Usually this transition state can complete in a few seconds. The same
+sequence occurs when a patch is disabled, except the tasks converge from
+the patched state to the unpatched state.
-See also the section "Limitations" below.
+An interrupt handler inherits the patched state of the task it
+interrupts. The same is true for forked tasks: the child inherits the
+patched state of the parent.
+
+Livepatch uses several complementary approaches to determine when it's
+safe to patch tasks:
+
+1. The first and most effective approach is stack checking of sleeping
+ tasks. If no affected functions are on the stack of a given task,
+ the task is patched. In most cases this will patch most or all of
+ the tasks on the first try. Otherwise it'll keep trying
+ periodically. This option is only available if the architecture has
+ reliable stacks (HAVE_RELIABLE_STACKTRACE).
+
+2. The second approach, if needed, is kernel exit switching. A
+ task is switched when it returns to user space from a system call, a
+ user space IRQ, or a signal. It's useful in the following cases:
+
+ a) Patching I/O-bound user tasks which are sleeping on an affected
+ function. In this case you have to send SIGSTOP and SIGCONT to
+ force it to exit the kernel and be patched.
+ b) Patching CPU-bound user tasks. If the task is highly CPU-bound
+ then it will get patched the next time it gets interrupted by an
+ IRQ.
+ c) In the future it could be useful for applying patches for
+ architectures which don't yet have HAVE_RELIABLE_STACKTRACE. In
+ this case you would have to signal most of the tasks on the
+ system. However this isn't supported yet because there's
+ currently no way to patch kthreads without
+ HAVE_RELIABLE_STACKTRACE.
+
+3. For idle "swapper" tasks, since they don't ever exit the kernel, they
+ instead have a klp_update_patch_state() call in the idle loop which
+ allows them to be patched before the CPU enters the idle state.
+
+ (Note there's not yet such an approach for kthreads.)
+
+All the above approaches may be skipped by setting the 'immediate' flag
+in the 'klp_patch' struct, which will disable per-task consistency and
+patch all tasks immediately. This can be useful if the patch doesn't
+change any function or data semantics. Note that, even with this flag
+set, it's possible that some tasks may still be running with an old
+version of the function, until that function returns.
+
+There's also an 'immediate' flag in the 'klp_func' struct which allows
+you to specify that certain functions in the patch can be applied
+without per-task consistency. This might be useful if you want to patch
+a common function like schedule(), and the function change doesn't need
+consistency but the rest of the patch does.
+
+For architectures which don't have HAVE_RELIABLE_STACKTRACE, the user
+must set patch->immediate which causes all tasks to be patched
+immediately. This option should be used with care, only when the patch
+doesn't change any function or data semantics.
+
+In the future, architectures which don't have HAVE_RELIABLE_STACKTRACE
+may be allowed to use per-task consistency if we can come up with
+another way to patch kthreads.
+
+The /sys/kernel/livepatch/<patch>/transition file shows whether a patch
+is in transition. Only a single patch (the topmost patch on the stack)
+can be in transition at a given time. A patch can remain in transition
+indefinitely, if any of the tasks are stuck in the initial patch state.
+
+A transition can be reversed and effectively canceled by writing the
+opposite value to the /sys/kernel/livepatch/<patch>/enabled file while
+the transition is in progress. Then all the tasks will attempt to
+converge back to the original patch state.
+
+There's also a /proc/<pid>/patch_state file which can be used to
+determine which tasks are blocking completion of a patching operation.
+If a patch is in transition, this file shows 0 to indicate the task is
+unpatched and 1 to indicate it's patched. Otherwise, if no patch is in
+transition, it shows -1. Any tasks which are blocking the transition
+can be signaled with SIGSTOP and SIGCONT to force them to change their
+patched state.
+
+
+3.1 Adding consistency model support to new architectures
+---------------------------------------------------------
+
+For adding consistency model support to new architectures, there are a
+few options:
+
+1) Add CONFIG_HAVE_RELIABLE_STACKTRACE. This means porting objtool, and
+ for non-DWARF unwinders, also making sure there's a way for the stack
+ tracing code to detect interrupts on the stack.
+
+2) Alternatively, ensure that every kthread has a call to
+ klp_update_patch_state() in a safe location. Kthreads are typically
+ in an infinite loop which does some action repeatedly. The safe
+ location to switch the kthread's patch state would be at a designated
+ point in the loop where there are no locks taken and all data
+ structures are in a well-defined state.
+
+ The location is clear when using workqueues or the kthread worker
+ API. These kthreads process independent actions in a generic loop.
+
+ It's much more complicated with kthreads which have a custom loop.
+ There the safe location must be carefully selected on a case-by-case
+ basis.
+
+ In that case, arches without HAVE_RELIABLE_STACKTRACE would still be
+ able to use the non-stack-checking parts of the consistency model:
+
+ a) patching user tasks when they cross the kernel/user space
+ boundary; and
+
+ b) patching kthreads and idle tasks at their designated patch points.
+
+ This option isn't as good as option 1 because it requires signaling
+ user tasks and waking kthreads to patch them. But it could still be
+ a good backup option for those architectures which don't have
+ reliable stack traces yet.
+
+In the meantime, patches for such architectures can bypass the
+consistency model by setting klp_patch.immediate to true. This option
+is perfectly fine for patches which don't change the semantics of the
+patched functions. In practice, this is usable for ~90% of security
+fixes. Use of this option also means the patch can't be unloaded after
+it has been disabled.
4. Livepatch module
@@ -134,7 +256,7 @@
4.2. Metadata
-------------
+-------------
The patch is described by several structures that split the information
into three levels:
@@ -156,6 +278,9 @@
only for a particular object ( vmlinux or a kernel module ). Note that
kallsyms allows for searching symbols according to the object name.
+ There's also an 'immediate' flag which, when set, patches the
+ function immediately, bypassing the consistency model safety checks.
+
+ struct klp_object defines an array of patched functions (struct
klp_func) in the same object. Where the object is either vmlinux
(NULL) or a module name.
@@ -172,10 +297,13 @@
This structure handles all patched functions consistently and eventually,
synchronously. The whole patch is applied only when all patched
symbols are found. The only exception are symbols from objects
- (kernel modules) that have not been loaded yet. Also if a more complex
- consistency model is supported then a selected unit (thread,
- kernel as a whole) will see the new code from the entire patch
- only when it is in a safe state.
+ (kernel modules) that have not been loaded yet.
+
+ Setting the 'immediate' flag applies the patch to all tasks
+ immediately, bypassing the consistency model safety checks.
+
+ For more details on how the patch is applied on a per-task basis,
+ see the "Consistency model" section.
4.3. Livepatch module handling
@@ -188,8 +316,15 @@
two operations.
Module removal is only safe when there are no users of the underlying
-functions. The immediate consistency model is not able to detect this;
-therefore livepatch modules cannot be removed. See "Limitations" below.
+functions. The immediate consistency model is not able to detect this. The
+code just redirects the functions at the very beginning and it does not
+check if the functions are in use. In other words, it knows when the
+functions get called but it does not know when the functions return.
+Therefore it cannot be decided when the livepatch module can be safely
+removed. This is solved by a hybrid consistency model. When the system is
+transitioned to a new patch state (patched/unpatched) it is guaranteed that
+no task sleeps or runs in the old code.
+
5. Livepatch life-cycle
=======================
@@ -239,9 +374,15 @@
by writing '1' to /sys/kernel/livepatch/<name>/enabled. The system will
start using the new implementation of the patched functions at this stage.
-In particular, if an original function is patched for the first time, a
-function specific struct klp_ops is created and an universal ftrace handler
-is registered.
+When a patch is enabled, livepatch enters into a transition state where
+tasks are converging to the patched state. This is indicated by a value
+of '1' in /sys/kernel/livepatch/<name>/transition. Once all tasks have
+been patched, the 'transition' value changes to '0'. For more
+information about this process, see the "Consistency model" section.
+
+If an original function is patched for the first time, a function
+specific struct klp_ops is created and an universal ftrace handler is
+registered.
Functions might be patched multiple times. The ftrace handler is registered
only once for the given function. Further patches just add an entry to the
@@ -261,6 +402,12 @@
either the code from the previously enabled patch or even the original
code gets used.
+When a patch is disabled, livepatch enters into a transition state where
+tasks are converging to the unpatched state. This is indicated by a
+value of '1' in /sys/kernel/livepatch/<name>/transition. Once all tasks
+have been unpatched, the 'transition' value changes to '0'. For more
+information about this process, see the "Consistency model" section.
+
Here all the functions (struct klp_func) associated with the to-be-disabled
patch are removed from the corresponding struct klp_ops. The ftrace handler
is unregistered and the struct klp_ops is freed when the func_stack list
@@ -329,23 +476,6 @@
by "notrace".
- + Livepatch modules can not be removed.
-
- The current implementation just redirects the functions at the very
- beginning. It does not check if the functions are in use. In other
- words, it knows when the functions get called but it does not
- know when the functions return. Therefore it can not decide when
- the livepatch module can be safely removed.
-
- This will get most likely solved once a more complex consistency model
- is supported. The idea is that a safe state for patching should also
- mean a safe state for removing the patch.
-
- Note that the patch itself might get disabled by writing zero
- to /sys/kernel/livepatch/<patch>/enabled. It causes that the new
- code will not longer get called. But it does not guarantee
- that anyone is not sleeping anywhere in the new code.
-
+ Livepatch works reliably only when the dynamic ftrace is located at
the very beginning of the function.
diff --git a/Documentation/md/md-cluster.txt b/Documentation/md/md-cluster.txt
index 3888327..d221039 100644
--- a/Documentation/md/md-cluster.txt
+++ b/Documentation/md/md-cluster.txt
@@ -77,7 +77,7 @@
3.1.2 RESYNCING: informs other nodes that a resync is initiated or
ended so that each node may suspend or resume the region. Each
RESYNCING message identifies a range of the devices that the
- sending node is about to resync. This over-rides any pervious
+ sending node is about to resync. This overrides any previous
notification from that node: only one ranged can be resynced at a
time per-node.
diff --git a/Documentation/media/Makefile b/Documentation/media/Makefile
index 9b3e70b..3616695 100644
--- a/Documentation/media/Makefile
+++ b/Documentation/media/Makefile
@@ -1,51 +1,6 @@
-# Rules to convert DOT and SVG to Sphinx images
-
-SRC_DIR=$(srctree)/Documentation/media
-
-DOTS = \
- uapi/v4l/pipeline.dot \
-
-IMAGES = \
- typical_media_device.svg \
- uapi/dvb/dvbstb.svg \
- uapi/v4l/bayer.svg \
- uapi/v4l/constraints.svg \
- uapi/v4l/crop.svg \
- uapi/v4l/fieldseq_bt.svg \
- uapi/v4l/fieldseq_tb.svg \
- uapi/v4l/nv12mt.svg \
- uapi/v4l/nv12mt_example.svg \
- uapi/v4l/pipeline.svg \
- uapi/v4l/selection.svg \
- uapi/v4l/subdev-image-processing-full.svg \
- uapi/v4l/subdev-image-processing-scaling-multi-source.svg \
- uapi/v4l/subdev-image-processing-crop.svg \
- uapi/v4l/vbi_525.svg \
- uapi/v4l/vbi_625.svg \
- uapi/v4l/vbi_hsync.svg \
-
-DOTTGT := $(patsubst %.dot,%.svg,$(DOTS))
-IMGDOT := $(patsubst %,$(SRC_DIR)/%,$(DOTTGT))
-
-IMGTGT := $(patsubst %.svg,%.pdf,$(IMAGES))
-IMGPDF := $(patsubst %,$(SRC_DIR)/%,$(IMGTGT))
-
-cmd = $(echo-cmd) $(cmd_$(1))
-
-quiet_cmd_genpdf = GENPDF $2
- cmd_genpdf = convert $2 $3
-
-quiet_cmd_gendot = DOT $2
- cmd_gendot = dot -Tsvg $2 > $3 || { rm -f $3; exit 1; }
-
-%.pdf: %.svg
- @$(call cmd,genpdf,$<,$@)
-
-%.svg: %.dot
- @$(call cmd,gendot,$<,$@)
-
# Rules to convert a .h file to inline RST documentation
+SRC_DIR=$(srctree)/Documentation/media
PARSER = $(srctree)/Documentation/sphinx/parse-headers.pl
UAPI = $(srctree)/include/uapi/linux
KAPI = $(srctree)/include/linux
diff --git a/Documentation/media/intro.rst b/Documentation/media/intro.rst
index 8f7490c..9ce2e23 100644
--- a/Documentation/media/intro.rst
+++ b/Documentation/media/intro.rst
@@ -13,9 +13,9 @@
.. _typical_media_device:
-.. figure:: typical_media_device.*
- :alt: typical_media_device.pdf / typical_media_device.svg
- :align: center
+.. kernel-figure:: typical_media_device.svg
+ :alt: typical_media_device.svg
+ :align: center
Typical Media Device
diff --git a/Documentation/media/uapi/dvb/intro.rst b/Documentation/media/uapi/dvb/intro.rst
index 2ed5c23..652c4aa 100644
--- a/Documentation/media/uapi/dvb/intro.rst
+++ b/Documentation/media/uapi/dvb/intro.rst
@@ -55,9 +55,9 @@
.. _stb_components:
-.. figure:: dvbstb.*
- :alt: dvbstb.pdf / dvbstb.svg
- :align: center
+.. kernel-figure:: dvbstb.svg
+ :alt: dvbstb.svg
+ :align: center
Components of a DVB card/STB
diff --git a/Documentation/media/uapi/v4l/crop.rst b/Documentation/media/uapi/v4l/crop.rst
index be58894..182565b 100644
--- a/Documentation/media/uapi/v4l/crop.rst
+++ b/Documentation/media/uapi/v4l/crop.rst
@@ -53,8 +53,8 @@
.. _crop-scale:
-.. figure:: crop.*
- :alt: crop.pdf / crop.svg
+.. kernel-figure:: crop.svg
+ :alt: crop.svg
:align: center
Image Cropping, Insertion and Scaling
diff --git a/Documentation/media/uapi/v4l/dev-raw-vbi.rst b/Documentation/media/uapi/v4l/dev-raw-vbi.rst
index baf5f24..2e6878b 100644
--- a/Documentation/media/uapi/v4l/dev-raw-vbi.rst
+++ b/Documentation/media/uapi/v4l/dev-raw-vbi.rst
@@ -221,33 +221,29 @@
.. _vbi-hsync:
-.. figure:: vbi_hsync.*
- :alt: vbi_hsync.pdf / vbi_hsync.svg
- :align: center
+.. kernel-figure:: vbi_hsync.svg
+ :alt: vbi_hsync.svg
+ :align: center
**Figure 4.1. Line synchronization**
.. _vbi-525:
-.. figure:: vbi_525.*
- :alt: vbi_525.pdf / vbi_525.svg
- :align: center
+.. kernel-figure:: vbi_525.svg
+ :alt: vbi_525.svg
+ :align: center
**Figure 4.2. ITU-R 525 line numbering (M/NTSC and M/PAL)**
-
-
.. _vbi-625:
-.. figure:: vbi_625.*
- :alt: vbi_625.pdf / vbi_625.svg
- :align: center
+.. kernel-figure:: vbi_625.svg
+ :alt: vbi_625.svg
+ :align: center
**Figure 4.3. ITU-R 625 line numbering**
-
-
Remember the VBI image format depends on the selected video standard,
therefore the application must choose a new standard or query the
current standard first. Attempts to read or write data ahead of format
diff --git a/Documentation/media/uapi/v4l/dev-subdev.rst b/Documentation/media/uapi/v4l/dev-subdev.rst
index cd28701..f0e7621 100644
--- a/Documentation/media/uapi/v4l/dev-subdev.rst
+++ b/Documentation/media/uapi/v4l/dev-subdev.rst
@@ -99,9 +99,9 @@
.. _pipeline-scaling:
-.. figure:: pipeline.*
- :alt: pipeline.pdf / pipeline.svg
- :align: center
+.. kernel-figure:: pipeline.dot
+ :alt: pipeline.dot
+ :align: center
Image Format Negotiation on Pipelines
@@ -404,9 +404,9 @@
.. _subdev-image-processing-crop:
-.. figure:: subdev-image-processing-crop.*
- :alt: subdev-image-processing-crop.pdf / subdev-image-processing-crop.svg
- :align: center
+.. kernel-figure:: subdev-image-processing-crop.svg
+ :alt: subdev-image-processing-crop.svg
+ :align: center
**Figure 4.5. Image processing in subdevs: simple crop example**
@@ -421,9 +421,9 @@
.. _subdev-image-processing-scaling-multi-source:
-.. figure:: subdev-image-processing-scaling-multi-source.*
- :alt: subdev-image-processing-scaling-multi-source.pdf / subdev-image-processing-scaling-multi-source.svg
- :align: center
+.. kernel-figure:: subdev-image-processing-scaling-multi-source.svg
+ :alt: subdev-image-processing-scaling-multi-source.svg
+ :align: center
**Figure 4.6. Image processing in subdevs: scaling with multiple sources**
@@ -437,8 +437,8 @@
.. _subdev-image-processing-full:
-.. figure:: subdev-image-processing-full.*
- :alt: subdev-image-processing-full.pdf / subdev-image-processing-full.svg
+.. kernel-figure:: subdev-image-processing-full.svg
+ :alt: subdev-image-processing-full.svg
:align: center
**Figure 4.7. Image processing in subdevs: scaling and composition with multiple sinks and sources**
diff --git a/Documentation/media/uapi/v4l/field-order.rst b/Documentation/media/uapi/v4l/field-order.rst
index e05fb10..5f3f82c 100644
--- a/Documentation/media/uapi/v4l/field-order.rst
+++ b/Documentation/media/uapi/v4l/field-order.rst
@@ -141,17 +141,20 @@
Field Order, Top Field First Transmitted
========================================
-.. figure:: fieldseq_tb.*
- :alt: fieldseq_tb.pdf / fieldseq_tb.svg
+.. kernel-figure:: fieldseq_tb.svg
+ :alt: fieldseq_tb.svg
:align: center
+ Field Order, Top Field First Transmitted
+
.. _fieldseq-bt:
Field Order, Bottom Field First Transmitted
===========================================
-.. figure:: fieldseq_bt.*
- :alt: fieldseq_bt.pdf / fieldseq_bt.svg
+.. kernel-figure:: fieldseq_bt.svg
+ :alt: fieldseq_bt.svg
:align: center
+ Field Order, Bottom Field First Transmitted
diff --git a/Documentation/media/uapi/v4l/pixfmt-nv12mt.rst b/Documentation/media/uapi/v4l/pixfmt-nv12mt.rst
index 32d0c87..172a382 100644
--- a/Documentation/media/uapi/v4l/pixfmt-nv12mt.rst
+++ b/Documentation/media/uapi/v4l/pixfmt-nv12mt.rst
@@ -33,8 +33,8 @@
.. _nv12mt:
-.. figure:: nv12mt.*
- :alt: nv12mt.pdf / nv12mt.svg
+.. kernel-figure:: nv12mt.svg
+ :alt: nv12mt.svg
:align: center
V4L2_PIX_FMT_NV12MT macroblock Z shape memory layout
@@ -50,8 +50,8 @@
.. _nv12mt_ex:
-.. figure:: nv12mt_example.*
- :alt: nv12mt_example.pdf / nv12mt_example.svg
+.. kernel-figure:: nv12mt_example.svg
+ :alt: nv12mt_example.svg
:align: center
Example V4L2_PIX_FMT_NV12MT memory layout of macroblocks
diff --git a/Documentation/media/uapi/v4l/selection-api-003.rst b/Documentation/media/uapi/v4l/selection-api-003.rst
index 21686f9..bf7e76d 100644
--- a/Documentation/media/uapi/v4l/selection-api-003.rst
+++ b/Documentation/media/uapi/v4l/selection-api-003.rst
@@ -7,9 +7,9 @@
.. _sel-targets-capture:
-.. figure:: selection.*
- :alt: selection.pdf / selection.svg
- :align: center
+.. kernel-figure:: selection.svg
+ :alt: selection.svg
+ :align: center
Cropping and composing targets
diff --git a/Documentation/media/uapi/v4l/subdev-formats.rst b/Documentation/media/uapi/v4l/subdev-formats.rst
index d6152c9..89a1fb9 100644
--- a/Documentation/media/uapi/v4l/subdev-formats.rst
+++ b/Documentation/media/uapi/v4l/subdev-formats.rst
@@ -1514,8 +1514,8 @@
.. _bayer-patterns:
-.. figure:: bayer.*
- :alt: bayer.pdf / bayer.svg
+.. kernel-figure:: bayer.svg
+ :alt: bayer.svg
:align: center
**Figure 4.8 Bayer Patterns**
diff --git a/Documentation/media/v4l-drivers/index.rst b/Documentation/media/v4l-drivers/index.rst
index a606d1c..90fe22a 100644
--- a/Documentation/media/v4l-drivers/index.rst
+++ b/Documentation/media/v4l-drivers/index.rst
@@ -45,6 +45,7 @@
meye
omap3isp
omap4_camera
+ philips
pvrusb2
pxa_camera
radiotrack
diff --git a/drivers/media/usb/pwc/philips.txt b/Documentation/media/v4l-drivers/philips.rst
similarity index 94%
rename from drivers/media/usb/pwc/philips.txt
rename to Documentation/media/v4l-drivers/philips.rst
index d38dd79..620bcfe 100644
--- a/drivers/media/usb/pwc/philips.txt
+++ b/Documentation/media/v4l-drivers/philips.rst
@@ -1,8 +1,12 @@
+Philips webcams (pwc driver)
+============================
+
This file contains some additional information for the Philips and OEM webcams.
E-mail: webcam@smcc.demon.nl Last updated: 2004-01-19
Site: http://www.smcc.demon.nl/webcam/
As of this moment, the following cameras are supported:
+
* Philips PCA645
* Philips PCA646
* Philips PCVC675
@@ -89,7 +93,8 @@
compression (only useful with the plugin)
With this option you can control the compression factor that the camera
uses to squeeze the image through the USB bus. You can set the
- parameter between 0 and 3:
+ parameter between 0 and 3::
+
0 = prefer uncompressed images; if the requested mode is not available
in an uncompressed format, the driver will silently switch to low
compression.
@@ -109,11 +114,11 @@
leds
This settings takes 2 integers, that define the on/off time for the LED
(in milliseconds). One of the interesting things that you can do with
- this is let the LED blink while the camera is in use. This:
+ this is let the LED blink while the camera is in use. This::
leds=500,500
- will blink the LED once every second. But with:
+ will blink the LED once every second. But with::
leds=0,0
@@ -141,7 +146,7 @@
A camera is specified by its type (the number from the camera model,
like PCA645, PCVC750VC, etc) and optionally the serial number (visible
in /proc/bus/usb/devices). A hint consists of a string with the following
- format:
+ format::
[type[.serialnumber]:]node
@@ -150,7 +155,7 @@
would be rather pointless). The serialnumber is separated from the type
by a '.'; the node number by a ':'.
- This somewhat cryptic syntax is best explained by a few examples:
+ This somewhat cryptic syntax is best explained by a few examples::
dev_hint=3,5 The first detected cam gets assigned
/dev/video3, the second /dev/video5. Any
@@ -170,6 +175,7 @@
through /dev/video6.
Some points worth knowing:
+
- Serialnumbers are case sensitive and must be written full, including
leading zeroes (it's treated as a string).
- If a device node is already occupied, registration will fail and
@@ -189,8 +195,10 @@
If you want to trace something, look up the bit value(s) in the table
below, add the values together and supply that to the trace variable.
+ ====== ======= ================================================ =======
Value Value Description Default
(dec) (hex)
+ ====== ======= ================================================ =======
1 0x1 Module initialization; this will log messages On
while loading and unloading the module
@@ -208,6 +216,7 @@
64 0x40 Show viewport and image sizes Off
128 0x80 PWCX debugging Off
+ ====== ======= ================================================ =======
For example, to trace the open() & read() functions, sum 8 + 4 = 12,
so you would supply trace=12 during insmod or modprobe. If
@@ -216,7 +225,7 @@
-Example:
+Example::
# modprobe pwc size=cif fps=15 power_save=1
diff --git a/Documentation/media/v4l-drivers/soc-camera.rst b/Documentation/media/v4l-drivers/soc-camera.rst
index ba0c15d..79d09e4 100644
--- a/Documentation/media/v4l-drivers/soc-camera.rst
+++ b/Documentation/media/v4l-drivers/soc-camera.rst
@@ -12,7 +12,7 @@
control and configuration, and a parallel or a serial bus for data.
- camera host - an interface, to which a camera is connected. Typically a
specialised interface, present on many SoCs, e.g. PXA27x and PXA3xx, SuperH,
- AVR32, i.MX27, i.MX31.
+ i.MX27, i.MX31.
- camera host bus - a connection between a camera host and a camera. Can be
parallel or serial, consists of data and control lines, e.g. clock, vertical
and horizontal synchronization signals.
diff --git a/Documentation/media/v4l-drivers/zr364xx.rst b/Documentation/media/v4l-drivers/zr364xx.rst
index f5280e3..3d193f0 100644
--- a/Documentation/media/v4l-drivers/zr364xx.rst
+++ b/Documentation/media/v4l-drivers/zr364xx.rst
@@ -30,7 +30,7 @@
at the source code).
You can get these values by looking at /var/log/messages when you plug
-your camera, or by typing : cat /proc/bus/usb/devices.
+your camera, or by typing : cat /sys/kernel/debug/usb/devices.
If you manage to use your cam with this code, you can send me a mail
(royale@zerezo.com) with the name of your cam and a patch if needed.
diff --git a/Documentation/mmc/mmc-dev-attrs.txt b/Documentation/mmc/mmc-dev-attrs.txt
index 404a0e9..4ad0bb1 100644
--- a/Documentation/mmc/mmc-dev-attrs.txt
+++ b/Documentation/mmc/mmc-dev-attrs.txt
@@ -13,7 +13,7 @@
All attributes are read-only.
- cid Card Identifaction Register
+ cid Card Identification Register
csd Card Specific Data Register
scr SD Card Configuration Register (SD only)
date Manufacturing Date (from CID Register)
@@ -30,6 +30,7 @@
rel_sectors Reliable write sector count
ocr Operation Conditions Register
dsr Driver Stage Register
+ cmdq_en Command Queue enabled: 1 => enabled, 0 => not enabled
Note on Erase Size and Preferred Erase Size:
@@ -71,6 +72,6 @@
"preferred_erase_size" is in bytes.
Note on raw_rpmb_size_mult:
- "raw_rpmb_size_mult" is a mutliple of 128kB block.
+ "raw_rpmb_size_mult" is a multiple of 128kB block.
RPMB size in byte is calculated by using the following equation:
RPMB partition size = 128kB x raw_rpmb_size_mult
diff --git a/Documentation/networking/cdc_mbim.txt b/Documentation/networking/cdc_mbim.txt
index b9482ca..e4c376a 100644
--- a/Documentation/networking/cdc_mbim.txt
+++ b/Documentation/networking/cdc_mbim.txt
@@ -332,7 +332,7 @@
[5] "MBIM (Mobile Broadband Interface Model) Registry"
- http://compliance.usb.org/mbim/
-[6] "/proc/bus/usb filesystem output"
+[6] "/dev/bus/usb filesystem output"
- Documentation/usb/proc_usb_info.txt
[7] "/sys/bus/usb/devices/.../descriptors"
diff --git a/Documentation/networking/e100.txt b/Documentation/networking/e100.txt
index 42ddbd4..54810b8 100644
--- a/Documentation/networking/e100.txt
+++ b/Documentation/networking/e100.txt
@@ -130,7 +130,7 @@
version 1.6 or later is required for this functionality.
The latest release of ethtool can be found from
- http://ftp.kernel.org/pub/software/network/ethtool/
+ https://www.kernel.org/pub/software/network/ethtool/
Enabling Wake on LAN* (WoL)
---------------------------
diff --git a/Documentation/networking/e1000.txt b/Documentation/networking/e1000.txt
index 437b209..1f6ed84 100644
--- a/Documentation/networking/e1000.txt
+++ b/Documentation/networking/e1000.txt
@@ -435,7 +435,7 @@
version 1.6 or later is required for this functionality.
The latest release of ethtool can be found from
- http://ftp.kernel.org/pub/software/network/ethtool/
+ https://www.kernel.org/pub/software/network/ethtool/
Enabling Wake on LAN* (WoL)
---------------------------
diff --git a/Documentation/networking/e1000e.txt b/Documentation/networking/e1000e.txt
index ad2d9f3..1208954 100644
--- a/Documentation/networking/e1000e.txt
+++ b/Documentation/networking/e1000e.txt
@@ -274,7 +274,7 @@
diagnostics, as well as displaying statistical information. We
strongly recommend downloading the latest version of ethtool at:
- http://ftp.kernel.org/pub/software/network/ethtool/
+ https://kernel.org/pub/software/network/ethtool/
NOTE: When validating enable/disable tests on some parts (82578, for example)
you need to add a few seconds between tests when working with ethtool.
diff --git a/Documentation/networking/filter.txt b/Documentation/networking/filter.txt
index 683ada5..b69b205 100644
--- a/Documentation/networking/filter.txt
+++ b/Documentation/networking/filter.txt
@@ -595,10 +595,9 @@
skb pointer). All constraints and restrictions from bpf_check_classic() apply
before a conversion to the new layout is being done behind the scenes!
-Currently, the classic BPF format is being used for JITing on most of the
-architectures. x86-64, aarch64 and s390x perform JIT compilation from eBPF
-instruction set, however, future work will migrate other JIT compilers as well,
-so that they will profit from the very same benefits.
+Currently, the classic BPF format is being used for JITing on most 32-bit
+architectures, whereas x86-64, aarch64, s390x, powerpc64, sparc64 perform JIT
+compilation from eBPF instruction set.
Some core changes of the new internal format:
diff --git a/Documentation/networking/i40e.txt b/Documentation/networking/i40e.txt
index a251bf4..57e616e 100644
--- a/Documentation/networking/i40e.txt
+++ b/Documentation/networking/i40e.txt
@@ -63,6 +63,78 @@
The latest release of ethtool can be found from
https://www.kernel.org/pub/software/network/ethtool
+
+ Flow Director n-ntuple traffic filters (FDir)
+ ---------------------------------------------
+ The driver utilizes the ethtool interface for configuring ntuple filters,
+ via "ethtool -N <device> <filter>".
+
+ The sctp4, ip4, udp4, and tcp4 flow types are supported with the standard
+ fields including src-ip, dst-ip, src-port and dst-port. The driver only
+ supports fully enabling or fully masking the fields, so use of the mask
+ fields for partial matches is not supported.
+
+ Additionally, the driver supports using the action to specify filters for a
+ Virtual Function. You can specify the action as a 64bit value, where the
+ lower 32 bits represents the queue number, while the next 8 bits represent
+ which VF. Note that 0 is the PF, so the VF identifier is offset by 1. For
+ example:
+
+ ... action 0x800000002 ...
+
+ Would indicate to direct traffic for Virtual Function 7 (8 minus 1) on queue
+ 2 of that VF.
+
+ The driver also supports using the user-defined field to specify 2 bytes of
+ arbitrary data to match within the packet payload in addition to the regular
+ fields. The data is specified in the lower 32bits of the user-def field in
+ the following way:
+
+ +----------------------------+---------------------------+
+ | 31 28 24 20 16 | 15 12 8 4 0|
+ +----------------------------+---------------------------+
+ | offset into packet payload | 2 bytes of flexible data |
+ +----------------------------+---------------------------+
+
+ As an example,
+
+ ... user-def 0x4FFFF ....
+
+ means to match the value 0xFFFF 4 bytes into the packet payload. Note that
+ the offset is based on the beginning of the payload, and not the beginning
+ of the packet. Thus
+
+ flow-type tcp4 ... user-def 0x8BEAF ....
+
+ would match TCP/IPv4 packets which have the value 0xBEAF 8bytes into the
+ TCP/IPv4 payload.
+
+ For ICMP, the hardware parses the ICMP header as 4 bytes of header and 4
+ bytes of payload, so if you want to match an ICMP frames payload you may need
+ to add 4 to the offset in order to match the data.
+
+ Furthermore, the offset can only be up to a value of 64, as the hardware
+ will only read up to 64 bytes of data from the payload. It must also be even
+ as the flexible data is 2 bytes long and must be aligned to byte 0 of the
+ packet payload.
+
+ When programming filters, the hardware is limited to using a single input
+ set for each flow type. This means that it is an error to program two
+ different filters with the same type that don't match on the same fields.
+ Thus the second of the following two commands will fail:
+
+ ethtool -N <device> flow-type tcp4 src-ip 192.168.0.7 action 5
+ ethtool -N <device> flow-type tcp4 dst-ip 192.168.15.18 action 1
+
+ This is because the first filter will be accepted and reprogram the input
+ set for TCPv4 filters, but the second filter will be unable to reprogram the
+ input set until all the conflicting TCPv4 filters are first removed.
+
+ Note that the user-defined flexible offset is also considered part of the
+ input set and cannot be programmed separately for multiple filters of the
+ same type. However, the flexible data is not part of the input set and
+ multiple filters may use the same offset but match against different data.
+
Data Center Bridging (DCB)
--------------------------
DCB configuration is not currently supported.
diff --git a/Documentation/networking/igb.txt b/Documentation/networking/igb.txt
index 15534fd..f90643e 100644
--- a/Documentation/networking/igb.txt
+++ b/Documentation/networking/igb.txt
@@ -63,7 +63,7 @@
diagnostics, as well as displaying statistical information. The latest
version of ethtool can be found at:
- http://ftp.kernel.org/pub/software/network/ethtool/
+ https://www.kernel.org/pub/software/network/ethtool/
Enabling Wake on LAN* (WoL)
---------------------------
diff --git a/Documentation/networking/igbvf.txt b/Documentation/networking/igbvf.txt
index 40db17a..bd40473 100644
--- a/Documentation/networking/igbvf.txt
+++ b/Documentation/networking/igbvf.txt
@@ -62,7 +62,7 @@
version 3.0 or later is required for this functionality, although we
strongly recommend downloading the latest version at:
- http://ftp.kernel.org/pub/software/network/ethtool/
+ https://www.kernel.org/pub/software/network/ethtool/
Support
=======
diff --git a/Documentation/networking/ip-sysctl.txt b/Documentation/networking/ip-sysctl.txt
index ab02304..974ab47 100644
--- a/Documentation/networking/ip-sysctl.txt
+++ b/Documentation/networking/ip-sysctl.txt
@@ -73,6 +73,14 @@
0 - disabled
1 - enabled
+fib_multipath_hash_policy - INTEGER
+ Controls which hash policy to use for multipath routes. Only valid
+ for kernels built with CONFIG_IP_ROUTE_MULTIPATH enabled.
+ Default: 0 (Layer 3)
+ Possible values:
+ 0 - Layer 3
+ 1 - Layer 4
+
route/max_size - INTEGER
Maximum number of routes allowed in the kernel. Increase
this when using large numbers of interfaces and/or routes.
@@ -594,6 +602,14 @@
Note that that additional client or server features are only
effective if the basic support (0x1 and 0x2) are enabled respectively.
+tcp_fastopen_blackhole_timeout_sec - INTEGER
+ Initial time period in second to disable Fastopen on active TCP sockets
+ when a TFO firewall blackhole issue happens.
+ This time period will grow exponentially when more blackhole issues
+ get detected right after Fastopen is re-enabled and will reset to
+ initial value when the blackhole issue goes away.
+ By default, it is set to 1hr.
+
tcp_syn_retries - INTEGER
Number of times initial SYNs for an active TCP connection attempt
will be retransmitted. Should not be higher than 127. Default value
@@ -640,11 +656,6 @@
building larger TSO frames.
Default: 3
-tcp_tw_recycle - BOOLEAN
- Enable fast recycling TIME-WAIT sockets. Default value is 0.
- It should not be changed without advice/request of technical
- experts.
-
tcp_tw_reuse - BOOLEAN
Allow to reuse TIME-WAIT sockets for new connections when it is
safe from protocol viewpoint. Default value is 0.
@@ -853,12 +864,21 @@
ip_early_demux - BOOLEAN
Optimize input packet processing down to one demux for
certain kinds of local sockets. Currently we only do this
- for established TCP sockets.
+ for established TCP and connected UDP sockets.
It may add an additional cost for pure routing workloads that
reduces overall throughput, in such case you should disable it.
Default: 1
+tcp_early_demux - BOOLEAN
+ Enable early demux for established TCP sockets.
+ Default: 1
+
+udp_early_demux - BOOLEAN
+ Enable early demux for connected UDP sockets. Disable this if
+ your system could experience more unconnected load.
+ Default: 1
+
icmp_echo_ignore_all - BOOLEAN
If set non-zero, then the kernel will ignore all ICMP ECHO
requests sent to it.
@@ -1458,11 +1478,20 @@
Functional default: enabled if accept_ra is enabled.
disabled if accept_ra is disabled.
+accept_ra_rt_info_min_plen - INTEGER
+ Minimum prefix length of Route Information in RA.
+
+ Route Information w/ prefix smaller than this variable shall
+ be ignored.
+
+ Functional default: 0 if accept_ra_rtr_pref is enabled.
+ -1 if accept_ra_rtr_pref is disabled.
+
accept_ra_rt_info_max_plen - INTEGER
Maximum prefix length of Route Information in RA.
- Route Information w/ prefix larger than or equal to this
- variable shall be ignored.
+ Route Information w/ prefix larger than this variable shall
+ be ignored.
Functional default: 0 if accept_ra_rtr_pref is enabled.
-1 if accept_ra_rtr_pref is disabled.
diff --git a/Documentation/networking/ipvs-sysctl.txt b/Documentation/networking/ipvs-sysctl.txt
index e6b1c02..0568986 100644
--- a/Documentation/networking/ipvs-sysctl.txt
+++ b/Documentation/networking/ipvs-sysctl.txt
@@ -175,6 +175,14 @@
for VS/NAT when the load balancer receives packets from real
servers but the connection entries don't exist.
+pmtu_disc - BOOLEAN
+ 0 - disabled
+ not 0 - enabled (default)
+
+ By default, reject with FRAG_NEEDED all DF packets that exceed
+ the PMTU, irrespective of the forwarding method. For TUN method
+ the flag can be disabled to fragment such packets.
+
secure_tcp - INTEGER
0 - disabled (default)
@@ -185,15 +193,59 @@
The value definition is the same as that of drop_entry and
drop_packet.
-sync_threshold - INTEGER
- default 3
+sync_threshold - vector of 2 INTEGERs: sync_threshold, sync_period
+ default 3 50
- It sets synchronization threshold, which is the minimum number
- of incoming packets that a connection needs to receive before
- the connection will be synchronized. A connection will be
- synchronized, every time the number of its incoming packets
- modulus 50 equals the threshold. The range of the threshold is
- from 0 to 49.
+ It sets synchronization threshold, which is the minimum number
+ of incoming packets that a connection needs to receive before
+ the connection will be synchronized. A connection will be
+ synchronized, every time the number of its incoming packets
+ modulus sync_period equals the threshold. The range of the
+ threshold is from 0 to sync_period.
+
+ When sync_period and sync_refresh_period are 0, send sync only
+ for state changes or only once when pkts matches sync_threshold
+
+sync_refresh_period - UNSIGNED INTEGER
+ default 0
+
+ In seconds, difference in reported connection timer that triggers
+ new sync message. It can be used to avoid sync messages for the
+ specified period (or half of the connection timeout if it is lower)
+ if connection state is not changed since last sync.
+
+ This is useful for normal connections with high traffic to reduce
+ sync rate. Additionally, retry sync_retries times with period of
+ sync_refresh_period/8.
+
+sync_retries - INTEGER
+ default 0
+
+ Defines sync retries with period of sync_refresh_period/8. Useful
+ to protect against loss of sync messages. The range of the
+ sync_retries is from 0 to 3.
+
+sync_qlen_max - UNSIGNED LONG
+
+ Hard limit for queued sync messages that are not sent yet. It
+ defaults to 1/32 of the memory pages but actually represents
+ number of messages. It will protect us from allocating large
+ parts of memory when the sending rate is lower than the queuing
+ rate.
+
+sync_sock_size - INTEGER
+ default 0
+
+ Configuration of SNDBUF (master) or RCVBUF (slave) socket limit.
+ Default value is 0 (preserve system defaults).
+
+sync_ports - INTEGER
+ default 1
+
+ The number of threads that master and backup servers can use for
+ sync traffic. Every thread will use single UDP port, thread 0 will
+ use the default port 8848 while last thread will use port
+ 8848+sync_ports-1.
snat_reroute - BOOLEAN
0 - disabled
diff --git a/Documentation/networking/ixgb.txt b/Documentation/networking/ixgb.txt
index 9b4a10a1..09f71d7 100644
--- a/Documentation/networking/ixgb.txt
+++ b/Documentation/networking/ixgb.txt
@@ -313,7 +313,7 @@
version 1.6 or later is required for this functionality.
The latest release of ethtool can be found from
- http://ftp.kernel.org/pub/software/network/ethtool/
+ https://www.kernel.org/pub/software/network/ethtool/
NOTE: The ethtool version 1.6 only supports a limited set of ethtool options.
Support for a more complete ethtool feature set can be enabled by
diff --git a/Documentation/networking/ixgbe.txt b/Documentation/networking/ixgbe.txt
index 6f0cb57..6878354 100644
--- a/Documentation/networking/ixgbe.txt
+++ b/Documentation/networking/ixgbe.txt
@@ -272,7 +272,7 @@
ethtool version is required for this functionality.
The latest release of ethtool can be found from
- http://ftp.kernel.org/pub/software/network/ethtool/
+ https://www.kernel.org/pub/software/network/ethtool/
FCoE
----
diff --git a/Documentation/networking/mpls-sysctl.txt b/Documentation/networking/mpls-sysctl.txt
index 15d8d16..2f24a19 100644
--- a/Documentation/networking/mpls-sysctl.txt
+++ b/Documentation/networking/mpls-sysctl.txt
@@ -19,6 +19,25 @@
Possible values: 0 - 1048575
Default: 0
+ip_ttl_propagate - BOOL
+ Control whether TTL is propagated from the IPv4/IPv6 header to
+ the MPLS header on imposing labels and propagated from the
+ MPLS header to the IPv4/IPv6 header on popping the last label.
+
+ If disabled, the MPLS transport network will appear as a
+ single hop to transit traffic.
+
+ 0 - disabled / RFC 3443 [Short] Pipe Model
+ 1 - enabled / RFC 3443 Uniform Model (default)
+
+default_ttl - BOOL
+ Default TTL value to use for MPLS packets where it cannot be
+ propagated from an IP header, either because one isn't present
+ or ip_ttl_propagate has been disabled.
+
+ Possible values: 1 - 255
+ Default: 255
+
conf/<interface>/input - BOOL
Control whether packets can be input on this interface.
diff --git a/Documentation/networking/switchdev.txt b/Documentation/networking/switchdev.txt
index 2bbac05..3e7b946 100644
--- a/Documentation/networking/switchdev.txt
+++ b/Documentation/networking/switchdev.txt
@@ -13,43 +13,43 @@
with SR-IOV or soft switches, such as OVS, are possible.
- User-space tools
-
- user space |
- +-------------------------------------------------------------------+
- kernel | Netlink
- |
- +--------------+-------------------------------+
- | Network stack |
- | (Linux) |
- | |
- +----------------------------------------------+
-
+ User-space tools
+
+ user space |
+ +-------------------------------------------------------------------+
+ kernel | Netlink
+ |
+ +--------------+-------------------------------+
+ | Network stack |
+ | (Linux) |
+ | |
+ +----------------------------------------------+
+
sw1p2 sw1p4 sw1p6
- sw1p1 + sw1p3 + sw1p5 + eth1
- + | + | + | +
- | | | | | | |
- +--+----+----+----+-+--+----+---+ +-----+-----+
- | Switch driver | | mgmt |
- | (this document) | | driver |
- | | | |
- +--------------+----------------+ +-----------+
- |
- kernel | HW bus (eg PCI)
- +-------------------------------------------------------------------+
- hardware |
- +--------------+---+------------+
- | Switch device (sw1) |
- | +----+ +--------+
- | | v offloaded data path | mgmt port
- | | | |
- +--|----|----+----+----+----+---+
- | | | | | |
- + + + + + +
+ sw1p1 + sw1p3 + sw1p5 + eth1
+ + | + | + | +
+ | | | | | | |
+ +--+----+----+----+-+--+----+---+ +-----+-----+
+ | Switch driver | | mgmt |
+ | (this document) | | driver |
+ | | | |
+ +--------------+----------------+ +-----------+
+ |
+ kernel | HW bus (eg PCI)
+ +-------------------------------------------------------------------+
+ hardware |
+ +--------------+---+------------+
+ | Switch device (sw1) |
+ | +----+ +--------+
+ | | v offloaded data path | mgmt port
+ | | | |
+ +--|----|----+----+----+----+---+
+ | | | | | |
+ + + + + + +
p1 p2 p3 p4 p5 p6
-
- front-panel ports
-
+
+ front-panel ports
+
Fig 1.
diff --git a/Documentation/phy.txt b/Documentation/phy.txt
index 0aa994b..383cdd8 100644
--- a/Documentation/phy.txt
+++ b/Documentation/phy.txt
@@ -97,7 +97,7 @@
non-dt boot, it should contain the label of the PHY. The two
devm_phy_get associates the device with the PHY using devres on
successful PHY get. On driver detach, release function is invoked on
-the the devres data and devres data is freed. phy_optional_get and
+the devres data and devres data is freed. phy_optional_get and
devm_phy_optional_get should be used when the phy is optional. These
two functions will never return -ENODEV, but instead returns NULL when
the phy cannot be found.Some generic drivers, such as ehci, may use multiple
diff --git a/Documentation/power/runtime_pm.txt b/Documentation/power/runtime_pm.txt
index 64546eb..ee69d753 100644
--- a/Documentation/power/runtime_pm.txt
+++ b/Documentation/power/runtime_pm.txt
@@ -478,15 +478,23 @@
- set the power.last_busy field to the current time
void pm_runtime_use_autosuspend(struct device *dev);
- - set the power.use_autosuspend flag, enabling autosuspend delays
+ - set the power.use_autosuspend flag, enabling autosuspend delays; call
+ pm_runtime_get_sync if the flag was previously cleared and
+ power.autosuspend_delay is negative
void pm_runtime_dont_use_autosuspend(struct device *dev);
- - clear the power.use_autosuspend flag, disabling autosuspend delays
+ - clear the power.use_autosuspend flag, disabling autosuspend delays;
+ decrement the device's usage counter if the flag was previously set and
+ power.autosuspend_delay is negative; call pm_runtime_idle
void pm_runtime_set_autosuspend_delay(struct device *dev, int delay);
- set the power.autosuspend_delay value to 'delay' (expressed in
milliseconds); if 'delay' is negative then runtime suspends are
- prevented
+ prevented; if power.use_autosuspend is set, pm_runtime_get_sync may be
+ called or the device's usage counter may be decremented and
+ pm_runtime_idle called depending on if power.autosuspend_delay is
+ changed to or from a negative value; if power.use_autosuspend is clear,
+ pm_runtime_idle is called
unsigned long pm_runtime_autosuspend_expiration(struct device *dev);
- calculate the time when the current autosuspend delay period will expire,
@@ -836,9 +844,8 @@
Instead of: pm_runtime_put_sync use: pm_runtime_put_sync_autosuspend.
Drivers may also continue to use the non-autosuspend helper functions; they
-will behave normally, not taking the autosuspend delay into account.
-Similarly, if the power.use_autosuspend field isn't set then the autosuspend
-helper functions will behave just like the non-autosuspend counterparts.
+will behave normally, which means sometimes taking the autosuspend delay into
+account (see pm_runtime_idle).
Under some circumstances a driver or subsystem may want to prevent a device
from autosuspending immediately, even though the usage counter is zero and the
diff --git a/Documentation/power/swsusp.txt b/Documentation/power/swsusp.txt
index 8cc17ca..9f2f942 100644
--- a/Documentation/power/swsusp.txt
+++ b/Documentation/power/swsusp.txt
@@ -406,7 +406,7 @@
before suspending; then remount them after resuming.
There is a work-around for this problem. For more information, see
-Documentation/usb/persist.txt.
+Documentation/driver-api/usb/persist.rst.
Q: Can I suspend-to-disk using a swap partition under LVM?
diff --git a/Documentation/process/4.Coding.rst b/Documentation/process/4.Coding.rst
index 2a728d8..6df1994 100644
--- a/Documentation/process/4.Coding.rst
+++ b/Documentation/process/4.Coding.rst
@@ -22,11 +22,11 @@
************
The kernel has long had a standard coding style, described in
-Documentation/process/coding-style.rst. For much of that time, the policies described
-in that file were taken as being, at most, advisory. As a result, there is
-a substantial amount of code in the kernel which does not meet the coding
-style guidelines. The presence of that code leads to two independent
-hazards for kernel developers.
+:ref:`Documentation/process/coding-style.rst <codingstyle>`. For much of
+that time, the policies described in that file were taken as being, at most,
+advisory. As a result, there is a substantial amount of code in the kernel
+which does not meet the coding style guidelines. The presence of that code
+leads to two independent hazards for kernel developers.
The first of these is to believe that the kernel coding standards do not
matter and are not enforced. The truth of the matter is that adding new
@@ -343,9 +343,10 @@
Documentation/ABI/README for a description of how this documentation should
be formatted and what information needs to be provided.
-The file Documentation/admin-guide/kernel-parameters.rst describes all of the kernel's
-boot-time parameters. Any patch which adds new parameters should add the
-appropriate entries to this file.
+The file :ref:`Documentation/admin-guide/kernel-parameters.rst
+<kernelparameters>` describes all of the kernel's boot-time parameters.
+Any patch which adds new parameters should add the appropriate entries to
+this file.
Any new configuration options must be accompanied by help text which
clearly explains the options and when the user might want to select them.
diff --git a/Documentation/process/applying-patches.rst b/Documentation/process/applying-patches.rst
index 87825cf..a0d058c 100644
--- a/Documentation/process/applying-patches.rst
+++ b/Documentation/process/applying-patches.rst
@@ -250,17 +250,11 @@
The 4.x.y (-stable) and 4.x patches live at
- ftp://ftp.kernel.org/pub/linux/kernel/v4.x/
+ https://www.kernel.org/pub/linux/kernel/v4.x/
The -rc patches live at
- ftp://ftp.kernel.org/pub/linux/kernel/v4.x/testing/
-
-In place of ``ftp.kernel.org`` you can use ``ftp.cc.kernel.org``, where cc is a
-country code. This way you'll be downloading from a mirror site that's most
-likely geographically closer to you, resulting in faster downloads for you,
-less bandwidth used globally and less load on the main kernel.org servers --
-these are good things, so do use mirrors when possible.
+ https://www.kernel.org/pub/linux/kernel/v4.x/testing/
The 4.x kernels
@@ -317,7 +311,7 @@
The -stable team usually do make incremental patches available as well
as patches against the latest mainline release, but I only cover the
non-incremental ones below. The incremental ones can be found at
- ftp://ftp.kernel.org/pub/linux/kernel/v4.x/incr/
+ https://www.kernel.org/pub/linux/kernel/v4.x/incr/
These patches are not incremental, meaning that for example the 4.7.3
patch does not apply on top of the 4.7.2 kernel source, but rather on top
diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
index 56ce661..01c5dbc 100644
--- a/Documentation/process/changes.rst
+++ b/Documentation/process/changes.rst
@@ -318,9 +318,10 @@
.. note::
Please notice that, for PDF and LaTeX output, you'll also need ``XeLaTeX``
- version 3.14159265. Depending on the distribution, you may also need
- to install a series of ``texlive`` packages that provide the minimal
- set of functionalities required for ``XeLaTex`` to work.
+ version 3.14159265. Depending on the distribution, you may also need to
+ install a series of ``texlive`` packages that provide the minimal set of
+ functionalities required for ``XeLaTex`` to work. For PDF output you'll also
+ need ``convert(1)`` from ImageMagick (https://www.imagemagick.org).
Other tools
-----------
@@ -348,7 +349,7 @@
Binutils
--------
-- <ftp://ftp.kernel.org/pub/linux/devel/binutils/>
+- <https://www.kernel.org/pub/linux/devel/binutils/>
OpenSSL
-------
@@ -361,17 +362,17 @@
Util-linux
----------
-- <ftp://ftp.kernel.org/pub/linux/utils/util-linux/>
+- <https://www.kernel.org/pub/linux/utils/util-linux/>
Ksymoops
--------
-- <ftp://ftp.kernel.org/pub/linux/utils/kernel/ksymoops/v2.4/>
+- <https://www.kernel.org/pub/linux/utils/kernel/ksymoops/v2.4/>
Module-Init-Tools
-----------------
-- <ftp://ftp.kernel.org/pub/linux/kernel/people/rusty/modules/>
+- <https://www.kernel.org/pub/linux/utils/kernel/module-init-tools/>
Mkinitrd
--------
@@ -401,7 +402,7 @@
Pcmciautils
-----------
-- <ftp://ftp.kernel.org/pub/linux/utils/kernel/pcmcia/>
+- <https://www.kernel.org/pub/linux/utils/kernel/pcmcia/>
Quota-tools
-----------
diff --git a/Documentation/s390/00-INDEX b/Documentation/s390/00-INDEX
index 9189535..317f037 100644
--- a/Documentation/s390/00-INDEX
+++ b/Documentation/s390/00-INDEX
@@ -22,5 +22,7 @@
- HiperSockets Bridge Port Support.
s390dbf.txt
- information on using the s390 debug feature.
+vfio-ccw.txt
+ information on the vfio-ccw I/O subchannel driver.
zfcpdump.txt
- information on the s390 SCSI dump tool.
diff --git a/Documentation/s390/vfio-ccw.txt b/Documentation/s390/vfio-ccw.txt
new file mode 100644
index 0000000..90b3dfe
--- /dev/null
+++ b/Documentation/s390/vfio-ccw.txt
@@ -0,0 +1,303 @@
+vfio-ccw: the basic infrastructure
+==================================
+
+Introduction
+------------
+
+Here we describe the vfio support for I/O subchannel devices for
+Linux/s390. Motivation for vfio-ccw is to passthrough subchannels to a
+virtual machine, while vfio is the means.
+
+Different than other hardware architectures, s390 has defined a unified
+I/O access method, which is so called Channel I/O. It has its own access
+patterns:
+- Channel programs run asynchronously on a separate (co)processor.
+- The channel subsystem will access any memory designated by the caller
+ in the channel program directly, i.e. there is no iommu involved.
+Thus when we introduce vfio support for these devices, we realize it
+with a mediated device (mdev) implementation. The vfio mdev will be
+added to an iommu group, so as to make itself able to be managed by the
+vfio framework. And we add read/write callbacks for special vfio I/O
+regions to pass the channel programs from the mdev to its parent device
+(the real I/O subchannel device) to do further address translation and
+to perform I/O instructions.
+
+This document does not intend to explain the s390 I/O architecture in
+every detail. More information/reference could be found here:
+- A good start to know Channel I/O in general:
+ https://en.wikipedia.org/wiki/Channel_I/O
+- s390 architecture:
+ s390 Principles of Operation manual (IBM Form. No. SA22-7832)
+- The existing Qemu code which implements a simple emulated channel
+ subsystem could also be a good reference. It makes it easier to follow
+ the flow.
+ qemu/hw/s390x/css.c
+
+For vfio mediated device framework:
+- Documentation/vfio-mediated-device.txt
+
+Motivation of vfio-ccw
+----------------------
+
+Currently, a guest virtualized via qemu/kvm on s390 only sees
+paravirtualized virtio devices via the "Virtio Over Channel I/O
+(virtio-ccw)" transport. This makes virtio devices discoverable via
+standard operating system algorithms for handling channel devices.
+
+However this is not enough. On s390 for the majority of devices, which
+use the standard Channel I/O based mechanism, we also need to provide
+the functionality of passing through them to a Qemu virtual machine.
+This includes devices that don't have a virtio counterpart (e.g. tape
+drives) or that have specific characteristics which guests want to
+exploit.
+
+For passing a device to a guest, we want to use the same interface as
+everybody else, namely vfio. Thus, we would like to introduce vfio
+support for channel devices. And we would like to name this new vfio
+device "vfio-ccw".
+
+Access patterns of CCW devices
+------------------------------
+
+s390 architecture has implemented a so called channel subsystem, that
+provides a unified view of the devices physically attached to the
+systems. Though the s390 hardware platform knows about a huge variety of
+different peripheral attachments like disk devices (aka. DASDs), tapes,
+communication controllers, etc. They can all be accessed by a well
+defined access method and they are presenting I/O completion a unified
+way: I/O interruptions.
+
+All I/O requires the use of channel command words (CCWs). A CCW is an
+instruction to a specialized I/O channel processor. A channel program is
+a sequence of CCWs which are executed by the I/O channel subsystem. To
+issue a channel program to the channel subsystem, it is required to
+build an operation request block (ORB), which can be used to point out
+the format of the CCW and other control information to the system. The
+operating system signals the I/O channel subsystem to begin executing
+the channel program with a SSCH (start sub-channel) instruction. The
+central processor is then free to proceed with non-I/O instructions
+until interrupted. The I/O completion result is received by the
+interrupt handler in the form of interrupt response block (IRB).
+
+Back to vfio-ccw, in short:
+- ORBs and channel programs are built in guest kernel (with guest
+ physical addresses).
+- ORBs and channel programs are passed to the host kernel.
+- Host kernel translates the guest physical addresses to real addresses
+ and starts the I/O with issuing a privileged Channel I/O instruction
+ (e.g SSCH).
+- channel programs run asynchronously on a separate processor.
+- I/O completion will be signaled to the host with I/O interruptions.
+ And it will be copied as IRB to user space to pass it back to the
+ guest.
+
+Physical vfio ccw device and its child mdev
+-------------------------------------------
+
+As mentioned above, we realize vfio-ccw with a mdev implementation.
+
+Channel I/O does not have IOMMU hardware support, so the physical
+vfio-ccw device does not have an IOMMU level translation or isolation.
+
+Sub-channel I/O instructions are all privileged instructions, When
+handling the I/O instruction interception, vfio-ccw has the software
+policing and translation how the channel program is programmed before
+it gets sent to hardware.
+
+Within this implementation, we have two drivers for two types of
+devices:
+- The vfio_ccw driver for the physical subchannel device.
+ This is an I/O subchannel driver for the real subchannel device. It
+ realizes a group of callbacks and registers to the mdev framework as a
+ parent (physical) device. As a consequence, mdev provides vfio_ccw a
+ generic interface (sysfs) to create mdev devices. A vfio mdev could be
+ created by vfio_ccw then and added to the mediated bus. It is the vfio
+ device that added to an IOMMU group and a vfio group.
+ vfio_ccw also provides an I/O region to accept channel program
+ request from user space and store I/O interrupt result for user
+ space to retrieve. To notify user space an I/O completion, it offers
+ an interface to setup an eventfd fd for asynchronous signaling.
+
+- The vfio_mdev driver for the mediated vfio ccw device.
+ This is provided by the mdev framework. It is a vfio device driver for
+ the mdev that created by vfio_ccw.
+ It realize a group of vfio device driver callbacks, adds itself to a
+ vfio group, and registers itself to the mdev framework as a mdev
+ driver.
+ It uses a vfio iommu backend that uses the existing map and unmap
+ ioctls, but rather than programming them into an IOMMU for a device,
+ it simply stores the translations for use by later requests. This
+ means that a device programmed in a VM with guest physical addresses
+ can have the vfio kernel convert that address to process virtual
+ address, pin the page and program the hardware with the host physical
+ address in one step.
+ For a mdev, the vfio iommu backend will not pin the pages during the
+ VFIO_IOMMU_MAP_DMA ioctl. Mdev framework will only maintain a database
+ of the iova<->vaddr mappings in this operation. And they export a
+ vfio_pin_pages and a vfio_unpin_pages interfaces from the vfio iommu
+ backend for the physical devices to pin and unpin pages by demand.
+
+Below is a high Level block diagram.
+
+ +-------------+
+ | |
+ | +---------+ | mdev_register_driver() +--------------+
+ | | Mdev | +<-----------------------+ |
+ | | bus | | | vfio_mdev.ko |
+ | | driver | +----------------------->+ |<-> VFIO user
+ | +---------+ | probe()/remove() +--------------+ APIs
+ | |
+ | MDEV CORE |
+ | MODULE |
+ | mdev.ko |
+ | +---------+ | mdev_register_device() +--------------+
+ | |Physical | +<-----------------------+ |
+ | | device | | | vfio_ccw.ko |<-> subchannel
+ | |interface| +----------------------->+ | device
+ | +---------+ | callback +--------------+
+ +-------------+
+
+The process of how these work together.
+1. vfio_ccw.ko drives the physical I/O subchannel, and registers the
+ physical device (with callbacks) to mdev framework.
+ When vfio_ccw probing the subchannel device, it registers device
+ pointer and callbacks to the mdev framework. Mdev related file nodes
+ under the device node in sysfs would be created for the subchannel
+ device, namely 'mdev_create', 'mdev_destroy' and
+ 'mdev_supported_types'.
+2. Create a mediated vfio ccw device.
+ Use the 'mdev_create' sysfs file, we need to manually create one (and
+ only one for our case) mediated device.
+3. vfio_mdev.ko drives the mediated ccw device.
+ vfio_mdev is also the vfio device drvier. It will probe the mdev and
+ add it to an iommu_group and a vfio_group. Then we could pass through
+ the mdev to a guest.
+
+vfio-ccw I/O region
+-------------------
+
+An I/O region is used to accept channel program request from user
+space and store I/O interrupt result for user space to retrieve. The
+defination of the region is:
+
+struct ccw_io_region {
+#define ORB_AREA_SIZE 12
+ __u8 orb_area[ORB_AREA_SIZE];
+#define SCSW_AREA_SIZE 12
+ __u8 scsw_area[SCSW_AREA_SIZE];
+#define IRB_AREA_SIZE 96
+ __u8 irb_area[IRB_AREA_SIZE];
+ __u32 ret_code;
+} __packed;
+
+While starting an I/O request, orb_area should be filled with the
+guest ORB, and scsw_area should be filled with the SCSW of the Virtual
+Subchannel.
+
+irb_area stores the I/O result.
+
+ret_code stores a return code for each access of the region.
+
+vfio-ccw patches overview
+-------------------------
+
+For now, our patches are rebased on the latest mdev implementation.
+vfio-ccw follows what vfio-pci did on the s390 paltform and uses
+vfio-iommu-type1 as the vfio iommu backend. It's a good start to launch
+the code review for vfio-ccw. Note that the implementation is far from
+complete yet; but we'd like to get feedback for the general
+architecture.
+
+* CCW translation APIs
+- Description:
+ These introduce a group of APIs (start with 'cp_') to do CCW
+ translation. The CCWs passed in by a user space program are
+ organized with their guest physical memory addresses. These APIs
+ will copy the CCWs into the kernel space, and assemble a runnable
+ kernel channel program by updating the guest physical addresses with
+ their corresponding host physical addresses.
+- Patches:
+ vfio: ccw: introduce channel program interfaces
+
+* vfio_ccw device driver
+- Description:
+ The following patches utilizes the CCW translation APIs and introduce
+ vfio_ccw, which is the driver for the I/O subchannel devices you want
+ to pass through.
+ vfio_ccw implements the following vfio ioctls:
+ VFIO_DEVICE_GET_INFO
+ VFIO_DEVICE_GET_IRQ_INFO
+ VFIO_DEVICE_GET_REGION_INFO
+ VFIO_DEVICE_RESET
+ VFIO_DEVICE_SET_IRQS
+ This provides an I/O region, so that the user space program can pass a
+ channel program to the kernel, to do further CCW translation before
+ issuing them to a real device.
+ This also provides the SET_IRQ ioctl to setup an event notifier to
+ notify the user space program the I/O completion in an asynchronous
+ way.
+- Patches:
+ vfio: ccw: basic implementation for vfio_ccw driver
+ vfio: ccw: introduce ccw_io_region
+ vfio: ccw: realize VFIO_DEVICE_GET_REGION_INFO ioctl
+ vfio: ccw: realize VFIO_DEVICE_RESET ioctl
+ vfio: ccw: realize VFIO_DEVICE_G(S)ET_IRQ_INFO ioctls
+
+The user of vfio-ccw is not limited to Qemu, while Qemu is definitely a
+good example to get understand how these patches work. Here is a little
+bit more detail how an I/O request triggered by the Qemu guest will be
+handled (without error handling).
+
+Explanation:
+Q1-Q7: Qemu side process.
+K1-K5: Kernel side process.
+
+Q1. Get I/O region info during initialization.
+Q2. Setup event notifier and handler to handle I/O completion.
+
+... ...
+
+Q3. Intercept a ssch instruction.
+Q4. Write the guest channel program and ORB to the I/O region.
+ K1. Copy from guest to kernel.
+ K2. Translate the guest channel program to a host kernel space
+ channel program, which becomes runnable for a real device.
+ K3. With the necessary information contained in the orb passed in
+ by Qemu, issue the ccwchain to the device.
+ K4. Return the ssch CC code.
+Q5. Return the CC code to the guest.
+
+... ...
+
+ K5. Interrupt handler gets the I/O result and write the result to
+ the I/O region.
+ K6. Signal Qemu to retrieve the result.
+Q6. Get the signal and event handler reads out the result from the I/O
+ region.
+Q7. Update the irb for the guest.
+
+Limitations
+-----------
+
+The current vfio-ccw implementation focuses on supporting basic commands
+needed to implement block device functionality (read/write) of DASD/ECKD
+device only. Some commands may need special handling in the future, for
+example, anything related to path grouping.
+
+DASD is a kind of storage device. While ECKD is a data recording format.
+More information for DASD and ECKD could be found here:
+https://en.wikipedia.org/wiki/Direct-access_storage_device
+https://en.wikipedia.org/wiki/Count_key_data
+
+Together with the corresponding work in Qemu, we can bring the passed
+through DASD/ECKD device online in a guest now and use it as a block
+device.
+
+Reference
+---------
+1. ESA/s390 Principles of Operation manual (IBM Form. No. SA22-7832)
+2. ESA/390 Common I/O Device Commands manual (IBM Form. No. SA22-7204)
+3. https://en.wikipedia.org/wiki/Channel_I/O
+4. Documentation/s390/cds.txt
+5. Documentation/vfio.txt
+6. Documentation/vfio-mediated-device.txt
diff --git a/Documentation/scheduler/sched-pelt.c b/Documentation/scheduler/sched-pelt.c
new file mode 100644
index 0000000..e421913
--- /dev/null
+++ b/Documentation/scheduler/sched-pelt.c
@@ -0,0 +1,108 @@
+/*
+ * The following program is used to generate the constants for
+ * computing sched averages.
+ *
+ * ==============================================================
+ * C program (compile with -lm)
+ * ==============================================================
+ */
+
+#include <math.h>
+#include <stdio.h>
+
+#define HALFLIFE 32
+#define SHIFT 32
+
+double y;
+
+void calc_runnable_avg_yN_inv(void)
+{
+ int i;
+ unsigned int x;
+
+ printf("static const u32 runnable_avg_yN_inv[] = {");
+ for (i = 0; i < HALFLIFE; i++) {
+ x = ((1UL<<32)-1)*pow(y, i);
+
+ if (i % 6 == 0) printf("\n\t");
+ printf("0x%8x, ", x);
+ }
+ printf("\n};\n\n");
+}
+
+int sum = 1024;
+
+void calc_runnable_avg_yN_sum(void)
+{
+ int i;
+
+ printf("static const u32 runnable_avg_yN_sum[] = {\n\t 0,");
+ for (i = 1; i <= HALFLIFE; i++) {
+ if (i == 1)
+ sum *= y;
+ else
+ sum = sum*y + 1024*y;
+
+ if (i % 11 == 0)
+ printf("\n\t");
+
+ printf("%5d,", sum);
+ }
+ printf("\n};\n\n");
+}
+
+int n = -1;
+/* first period */
+long max = 1024;
+
+void calc_converged_max(void)
+{
+ long last = 0, y_inv = ((1UL<<32)-1)*y;
+
+ for (; ; n++) {
+ if (n > -1)
+ max = ((max*y_inv)>>SHIFT) + 1024;
+ /*
+ * This is the same as:
+ * max = max*y + 1024;
+ */
+
+ if (last == max)
+ break;
+
+ last = max;
+ }
+ n--;
+ printf("#define LOAD_AVG_PERIOD %d\n", HALFLIFE);
+ printf("#define LOAD_AVG_MAX %ld\n", max);
+// printf("#define LOAD_AVG_MAX_N %d\n\n", n);
+}
+
+void calc_accumulated_sum_32(void)
+{
+ int i, x = sum;
+
+ printf("static const u32 __accumulated_sum_N32[] = {\n\t 0,");
+ for (i = 1; i <= n/HALFLIFE+1; i++) {
+ if (i > 1)
+ x = x/2 + sum;
+
+ if (i % 6 == 0)
+ printf("\n\t");
+
+ printf("%6d,", x);
+ }
+ printf("\n};\n\n");
+}
+
+void main(void)
+{
+ printf("/* Generated by Documentation/scheduler/sched-pelt; do not modify. */\n\n");
+
+ y = pow(0.5, 1/(double)HALFLIFE);
+
+ calc_runnable_avg_yN_inv();
+// calc_runnable_avg_yN_sum();
+ calc_converged_max();
+// calc_accumulated_sum_32();
+}
diff --git a/Documentation/sphinx/cdomain.py b/Documentation/sphinx/cdomain.py
index df0419c..cf13ff3 100644
--- a/Documentation/sphinx/cdomain.py
+++ b/Documentation/sphinx/cdomain.py
@@ -44,7 +44,7 @@
__version__ = '1.0'
# Get Sphinx version
-major, minor, patch = map(int, sphinx.__version__.split("."))
+major, minor, patch = sphinx.version_info[:3]
def setup(app):
diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py
new file mode 100644
index 0000000..cef4ad1
--- /dev/null
+++ b/Documentation/sphinx/kfigure.py
@@ -0,0 +1,551 @@
+# -*- coding: utf-8; mode: python -*-
+# pylint: disable=C0103, R0903, R0912, R0915
+u"""
+ scalable figure and image handling
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ Sphinx extension which implements scalable image handling.
+
+ :copyright: Copyright (C) 2016 Markus Heiser
+ :license: GPL Version 2, June 1991 see Linux/COPYING for details.
+
+ The build for image formats depend on image's source format and output's
+ destination format. This extension implement methods to simplify image
+ handling from the author's POV. Directives like ``kernel-figure`` implement
+ methods *to* always get the best output-format even if some tools are not
+ installed. For more details take a look at ``convert_image(...)`` which is
+ the core of all conversions.
+
+ * ``.. kernel-image``: for image handling / a ``.. image::`` replacement
+
+ * ``.. kernel-figure``: for figure handling / a ``.. figure::`` replacement
+
+ * ``.. kernel-render``: for render markup / a concept to embed *render*
+ markups (or languages). Supported markups (see ``RENDER_MARKUP_EXT``)
+
+ - ``DOT``: render embedded Graphviz's **DOC**
+ - ``SVG``: render embedded Scalable Vector Graphics (**SVG**)
+ - ... *developable*
+
+ Used tools:
+
+ * ``dot(1)``: Graphviz (http://www.graphviz.org). If Graphviz is not
+ available, the DOT language is inserted as literal-block.
+
+ * SVG to PDF: To generate PDF, you need at least one of this tools:
+
+ - ``convert(1)``: ImageMagick (https://www.imagemagick.org)
+
+ List of customizations:
+
+ * generate PDF from SVG / used by PDF (LaTeX) builder
+
+ * generate SVG (html-builder) and PDF (latex-builder) from DOT files.
+ DOT: see http://www.graphviz.org/content/dot-language
+
+ """
+
+import os
+from os import path
+import subprocess
+from hashlib import sha1
+import sys
+
+from docutils import nodes
+from docutils.statemachine import ViewList
+from docutils.parsers.rst import directives
+from docutils.parsers.rst.directives import images
+import sphinx
+
+from sphinx.util.nodes import clean_astext
+from six import iteritems
+
+PY3 = sys.version_info[0] == 3
+
+if PY3:
+ _unicode = str
+else:
+ _unicode = unicode
+
+# Get Sphinx version
+major, minor, patch = sphinx.version_info[:3]
+if major == 1 and minor > 3:
+ # patches.Figure only landed in Sphinx 1.4
+ from sphinx.directives.patches import Figure # pylint: disable=C0413
+else:
+ Figure = images.Figure
+
+__version__ = '1.0.0'
+
+# simple helper
+# -------------
+
+def which(cmd):
+ """Searches the ``cmd`` in the ``PATH`` enviroment.
+
+ This *which* searches the PATH for executable ``cmd`` . First match is
+ returned, if nothing is found, ``None` is returned.
+ """
+ envpath = os.environ.get('PATH', None) or os.defpath
+ for folder in envpath.split(os.pathsep):
+ fname = folder + os.sep + cmd
+ if path.isfile(fname):
+ return fname
+
+def mkdir(folder, mode=0o775):
+ if not path.isdir(folder):
+ os.makedirs(folder, mode)
+
+def file2literal(fname):
+ with open(fname, "r") as src:
+ data = src.read()
+ node = nodes.literal_block(data, data)
+ return node
+
+def isNewer(path1, path2):
+ """Returns True if ``path1`` is newer than ``path2``
+
+ If ``path1`` exists and is newer than ``path2`` the function returns
+ ``True`` is returned otherwise ``False``
+ """
+ return (path.exists(path1)
+ and os.stat(path1).st_ctime > os.stat(path2).st_ctime)
+
+def pass_handle(self, node): # pylint: disable=W0613
+ pass
+
+# setup conversion tools and sphinx extension
+# -------------------------------------------
+
+# Graphviz's dot(1) support
+dot_cmd = None
+
+# ImageMagick' convert(1) support
+convert_cmd = None
+
+
+def setup(app):
+ # check toolchain first
+ app.connect('builder-inited', setupTools)
+
+ # image handling
+ app.add_directive("kernel-image", KernelImage)
+ app.add_node(kernel_image,
+ html = (visit_kernel_image, pass_handle),
+ latex = (visit_kernel_image, pass_handle),
+ texinfo = (visit_kernel_image, pass_handle),
+ text = (visit_kernel_image, pass_handle),
+ man = (visit_kernel_image, pass_handle), )
+
+ # figure handling
+ app.add_directive("kernel-figure", KernelFigure)
+ app.add_node(kernel_figure,
+ html = (visit_kernel_figure, pass_handle),
+ latex = (visit_kernel_figure, pass_handle),
+ texinfo = (visit_kernel_figure, pass_handle),
+ text = (visit_kernel_figure, pass_handle),
+ man = (visit_kernel_figure, pass_handle), )
+
+ # render handling
+ app.add_directive('kernel-render', KernelRender)
+ app.add_node(kernel_render,
+ html = (visit_kernel_render, pass_handle),
+ latex = (visit_kernel_render, pass_handle),
+ texinfo = (visit_kernel_render, pass_handle),
+ text = (visit_kernel_render, pass_handle),
+ man = (visit_kernel_render, pass_handle), )
+
+ app.connect('doctree-read', add_kernel_figure_to_std_domain)
+
+ return dict(
+ version = __version__,
+ parallel_read_safe = True,
+ parallel_write_safe = True
+ )
+
+
+def setupTools(app):
+ u"""
+ Check available build tools and log some *verbose* messages.
+
+ This function is called once, when the builder is initiated.
+ """
+ global dot_cmd, convert_cmd # pylint: disable=W0603
+ app.verbose("kfigure: check installed tools ...")
+
+ dot_cmd = which('dot')
+ convert_cmd = which('convert')
+
+ if dot_cmd:
+ app.verbose("use dot(1) from: " + dot_cmd)
+ else:
+ app.warn("dot(1) not found, for better output quality install "
+ "graphviz from http://www.graphviz.org")
+ if convert_cmd:
+ app.verbose("use convert(1) from: " + convert_cmd)
+ else:
+ app.warn(
+ "convert(1) not found, for SVG to PDF conversion install "
+ "ImageMagick (https://www.imagemagick.org)")
+
+
+# integrate conversion tools
+# --------------------------
+
+RENDER_MARKUP_EXT = {
+ # The '.ext' must be handled by convert_image(..) function's *in_ext* input.
+ # <name> : <.ext>
+ 'DOT' : '.dot',
+ 'SVG' : '.svg'
+}
+
+def convert_image(img_node, translator, src_fname=None):
+ """Convert a image node for the builder.
+
+ Different builder prefer different image formats, e.g. *latex* builder
+ prefer PDF while *html* builder prefer SVG format for images.
+
+ This function handles output image formats in dependence of source the
+ format (of the image) and the translator's output format.
+ """
+ app = translator.builder.app
+
+ fname, in_ext = path.splitext(path.basename(img_node['uri']))
+ if src_fname is None:
+ src_fname = path.join(translator.builder.srcdir, img_node['uri'])
+ if not path.exists(src_fname):
+ src_fname = path.join(translator.builder.outdir, img_node['uri'])
+
+ dst_fname = None
+
+ # in kernel builds, use 'make SPHINXOPTS=-v' to see verbose messages
+
+ app.verbose('assert best format for: ' + img_node['uri'])
+
+ if in_ext == '.dot':
+
+ if not dot_cmd:
+ app.verbose("dot from graphviz not available / include DOT raw.")
+ img_node.replace_self(file2literal(src_fname))
+
+ elif translator.builder.format == 'latex':
+ dst_fname = path.join(translator.builder.outdir, fname + '.pdf')
+ img_node['uri'] = fname + '.pdf'
+ img_node['candidates'] = {'*': fname + '.pdf'}
+
+
+ elif translator.builder.format == 'html':
+ dst_fname = path.join(
+ translator.builder.outdir,
+ translator.builder.imagedir,
+ fname + '.svg')
+ img_node['uri'] = path.join(
+ translator.builder.imgpath, fname + '.svg')
+ img_node['candidates'] = {
+ '*': path.join(translator.builder.imgpath, fname + '.svg')}
+
+ else:
+ # all other builder formats will include DOT as raw
+ img_node.replace_self(file2literal(src_fname))
+
+ elif in_ext == '.svg':
+
+ if translator.builder.format == 'latex':
+ if convert_cmd is None:
+ app.verbose("no SVG to PDF conversion available / include SVG raw.")
+ img_node.replace_self(file2literal(src_fname))
+ else:
+ dst_fname = path.join(translator.builder.outdir, fname + '.pdf')
+ img_node['uri'] = fname + '.pdf'
+ img_node['candidates'] = {'*': fname + '.pdf'}
+
+ if dst_fname:
+ # the builder needs not to copy one more time, so pop it if exists.
+ translator.builder.images.pop(img_node['uri'], None)
+ _name = dst_fname[len(translator.builder.outdir) + 1:]
+
+ if isNewer(dst_fname, src_fname):
+ app.verbose("convert: {out}/%s already exists and is newer" % _name)
+
+ else:
+ ok = False
+ mkdir(path.dirname(dst_fname))
+
+ if in_ext == '.dot':
+ app.verbose('convert DOT to: {out}/' + _name)
+ ok = dot2format(app, src_fname, dst_fname)
+
+ elif in_ext == '.svg':
+ app.verbose('convert SVG to: {out}/' + _name)
+ ok = svg2pdf(app, src_fname, dst_fname)
+
+ if not ok:
+ img_node.replace_self(file2literal(src_fname))
+
+
+def dot2format(app, dot_fname, out_fname):
+ """Converts DOT file to ``out_fname`` using ``dot(1)``.
+
+ * ``dot_fname`` pathname of the input DOT file, including extension ``.dot``
+ * ``out_fname`` pathname of the output file, including format extension
+
+ The *format extension* depends on the ``dot`` command (see ``man dot``
+ option ``-Txxx``). Normally you will use one of the following extensions:
+
+ - ``.ps`` for PostScript,
+ - ``.svg`` or ``svgz`` for Structured Vector Graphics,
+ - ``.fig`` for XFIG graphics and
+ - ``.png`` or ``gif`` for common bitmap graphics.
+
+ """
+ out_format = path.splitext(out_fname)[1][1:]
+ cmd = [dot_cmd, '-T%s' % out_format, dot_fname]
+ exit_code = 42
+
+ with open(out_fname, "w") as out:
+ exit_code = subprocess.call(cmd, stdout = out)
+ if exit_code != 0:
+ app.warn("Error #%d when calling: %s" % (exit_code, " ".join(cmd)))
+ return bool(exit_code == 0)
+
+def svg2pdf(app, svg_fname, pdf_fname):
+ """Converts SVG to PDF with ``convert(1)`` command.
+
+ Uses ``convert(1)`` from ImageMagick (https://www.imagemagick.org) for
+ conversion. Returns ``True`` on success and ``False`` if an error occurred.
+
+ * ``svg_fname`` pathname of the input SVG file with extension (``.svg``)
+ * ``pdf_name`` pathname of the output PDF file with extension (``.pdf``)
+
+ """
+ cmd = [convert_cmd, svg_fname, pdf_fname]
+ # use stdout and stderr from parent
+ exit_code = subprocess.call(cmd)
+ if exit_code != 0:
+ app.warn("Error #%d when calling: %s" % (exit_code, " ".join(cmd)))
+ return bool(exit_code == 0)
+
+
+# image handling
+# ---------------------
+
+def visit_kernel_image(self, node): # pylint: disable=W0613
+ """Visitor of the ``kernel_image`` Node.
+
+ Handles the ``image`` child-node with the ``convert_image(...)``.
+ """
+ img_node = node[0]
+ convert_image(img_node, self)
+
+class kernel_image(nodes.image):
+ """Node for ``kernel-image`` directive."""
+ pass
+
+class KernelImage(images.Image):
+ u"""KernelImage directive
+
+ Earns everything from ``.. image::`` directive, except *remote URI* and
+ *glob* pattern. The KernelImage wraps a image node into a
+ kernel_image node. See ``visit_kernel_image``.
+ """
+
+ def run(self):
+ uri = self.arguments[0]
+ if uri.endswith('.*') or uri.find('://') != -1:
+ raise self.severe(
+ 'Error in "%s: %s": glob pattern and remote images are not allowed'
+ % (self.name, uri))
+ result = images.Image.run(self)
+ if len(result) == 2 or isinstance(result[0], nodes.system_message):
+ return result
+ (image_node,) = result
+ # wrap image node into a kernel_image node / see visitors
+ node = kernel_image('', image_node)
+ return [node]
+
+# figure handling
+# ---------------------
+
+def visit_kernel_figure(self, node): # pylint: disable=W0613
+ """Visitor of the ``kernel_figure`` Node.
+
+ Handles the ``image`` child-node with the ``convert_image(...)``.
+ """
+ img_node = node[0][0]
+ convert_image(img_node, self)
+
+class kernel_figure(nodes.figure):
+ """Node for ``kernel-figure`` directive."""
+
+class KernelFigure(Figure):
+ u"""KernelImage directive
+
+ Earns everything from ``.. figure::`` directive, except *remote URI* and
+ *glob* pattern. The KernelFigure wraps a figure node into a kernel_figure
+ node. See ``visit_kernel_figure``.
+ """
+
+ def run(self):
+ uri = self.arguments[0]
+ if uri.endswith('.*') or uri.find('://') != -1:
+ raise self.severe(
+ 'Error in "%s: %s":'
+ ' glob pattern and remote images are not allowed'
+ % (self.name, uri))
+ result = Figure.run(self)
+ if len(result) == 2 or isinstance(result[0], nodes.system_message):
+ return result
+ (figure_node,) = result
+ # wrap figure node into a kernel_figure node / see visitors
+ node = kernel_figure('', figure_node)
+ return [node]
+
+
+# render handling
+# ---------------------
+
+def visit_kernel_render(self, node):
+ """Visitor of the ``kernel_render`` Node.
+
+ If rendering tools available, save the markup of the ``literal_block`` child
+ node into a file and replace the ``literal_block`` node with a new created
+ ``image`` node, pointing to the saved markup file. Afterwards, handle the
+ image child-node with the ``convert_image(...)``.
+ """
+ app = self.builder.app
+ srclang = node.get('srclang')
+
+ app.verbose('visit kernel-render node lang: "%s"' % (srclang))
+
+ tmp_ext = RENDER_MARKUP_EXT.get(srclang, None)
+ if tmp_ext is None:
+ app.warn('kernel-render: "%s" unknow / include raw.' % (srclang))
+ return
+
+ if not dot_cmd and tmp_ext == '.dot':
+ app.verbose("dot from graphviz not available / include raw.")
+ return
+
+ literal_block = node[0]
+
+ code = literal_block.astext()
+ hashobj = code.encode('utf-8') # str(node.attributes)
+ fname = path.join('%s-%s' % (srclang, sha1(hashobj).hexdigest()))
+
+ tmp_fname = path.join(
+ self.builder.outdir, self.builder.imagedir, fname + tmp_ext)
+
+ if not path.isfile(tmp_fname):
+ mkdir(path.dirname(tmp_fname))
+ with open(tmp_fname, "w") as out:
+ out.write(code)
+
+ img_node = nodes.image(node.rawsource, **node.attributes)
+ img_node['uri'] = path.join(self.builder.imgpath, fname + tmp_ext)
+ img_node['candidates'] = {
+ '*': path.join(self.builder.imgpath, fname + tmp_ext)}
+
+ literal_block.replace_self(img_node)
+ convert_image(img_node, self, tmp_fname)
+
+
+class kernel_render(nodes.General, nodes.Inline, nodes.Element):
+ """Node for ``kernel-render`` directive."""
+ pass
+
+class KernelRender(Figure):
+ u"""KernelRender directive
+
+ Render content by external tool. Has all the options known from the
+ *figure* directive, plus option ``caption``. If ``caption`` has a
+ value, a figure node with the *caption* is inserted. If not, a image node is
+ inserted.
+
+ The KernelRender directive wraps the text of the directive into a
+ literal_block node and wraps it into a kernel_render node. See
+ ``visit_kernel_render``.
+ """
+ has_content = True
+ required_arguments = 1
+ optional_arguments = 0
+ final_argument_whitespace = False
+
+ # earn options from 'figure'
+ option_spec = Figure.option_spec.copy()
+ option_spec['caption'] = directives.unchanged
+
+ def run(self):
+ return [self.build_node()]
+
+ def build_node(self):
+
+ srclang = self.arguments[0].strip()
+ if srclang not in RENDER_MARKUP_EXT.keys():
+ return [self.state_machine.reporter.warning(
+ 'Unknow source language "%s", use one of: %s.' % (
+ srclang, ",".join(RENDER_MARKUP_EXT.keys())),
+ line=self.lineno)]
+
+ code = '\n'.join(self.content)
+ if not code.strip():
+ return [self.state_machine.reporter.warning(
+ 'Ignoring "%s" directive without content.' % (
+ self.name),
+ line=self.lineno)]
+
+ node = kernel_render()
+ node['alt'] = self.options.get('alt','')
+ node['srclang'] = srclang
+ literal_node = nodes.literal_block(code, code)
+ node += literal_node
+
+ caption = self.options.get('caption')
+ if caption:
+ # parse caption's content
+ parsed = nodes.Element()
+ self.state.nested_parse(
+ ViewList([caption], source=''), self.content_offset, parsed)
+ caption_node = nodes.caption(
+ parsed[0].rawsource, '', *parsed[0].children)
+ caption_node.source = parsed[0].source
+ caption_node.line = parsed[0].line
+
+ figure_node = nodes.figure('', node)
+ for k,v in self.options.items():
+ figure_node[k] = v
+ figure_node += caption_node
+
+ node = figure_node
+
+ return node
+
+def add_kernel_figure_to_std_domain(app, doctree):
+ """Add kernel-figure anchors to 'std' domain.
+
+ The ``StandardDomain.process_doc(..)`` method does not know how to resolve
+ the caption (label) of ``kernel-figure`` directive (it only knows about
+ standard nodes, e.g. table, figure etc.). Without any additional handling
+ this will result in a 'undefined label' for kernel-figures.
+
+ This handle adds labels of kernel-figure to the 'std' domain labels.
+ """
+
+ std = app.env.domains["std"]
+ docname = app.env.docname
+ labels = std.data["labels"]
+
+ for name, explicit in iteritems(doctree.nametypes):
+ if not explicit:
+ continue
+ labelid = doctree.nameids[name]
+ if labelid is None:
+ continue
+ node = doctree.ids[labelid]
+
+ if node.tagname == 'kernel_figure':
+ for n in node.next_node():
+ if n.tagname == 'caption':
+ sectname = clean_astext(n)
+ # add label to std domain
+ labels[name] = docname, labelid, sectname
+ break
diff --git a/Documentation/sphinx/tmplcvt b/Documentation/sphinx/tmplcvt
index 909a730..6848f0a 100755
--- a/Documentation/sphinx/tmplcvt
+++ b/Documentation/sphinx/tmplcvt
@@ -7,13 +7,22 @@
# fix \_
# title line?
#
+set -eu
+
+if [ "$#" != "2" ]; then
+ echo "$0 <docbook file> <rst file>"
+ exit
+fi
+
+DIR=$(dirname $0)
in=$1
rst=$2
tmp=$rst.tmp
cp $in $tmp
-sed --in-place -f convert_template.sed $tmp
+sed --in-place -f $DIR/convert_template.sed $tmp
pandoc -s -S -f docbook -t rst -o $rst $tmp
-sed --in-place -f post_convert.sed $rst
+sed --in-place -f $DIR/post_convert.sed $rst
rm $tmp
+echo "book writen to $rst"
diff --git a/Documentation/static-keys.txt b/Documentation/static-keys.txt
index 32a25fa..ef419fd 100644
--- a/Documentation/static-keys.txt
+++ b/Documentation/static-keys.txt
@@ -118,7 +118,7 @@
Keys defined via DEFINE_STATIC_KEY_TRUE(), or DEFINE_STATIC_KEY_FALSE, may
be used in either static_branch_likely() or static_branch_unlikely()
-statemnts.
+statements.
Branch(es) can be set true via:
diff --git a/Documentation/sync_file.txt b/Documentation/sync_file.txt
index 269681a..c3d033a 100644
--- a/Documentation/sync_file.txt
+++ b/Documentation/sync_file.txt
@@ -37,7 +37,7 @@
Out-fences are fences that the driver creates.
On the other hand if the driver receives fence(s) through a sync_file from
-userspace we call these fence(s) 'in-fences'. Receiveing in-fences means that
+userspace we call these fence(s) 'in-fences'. Receiving in-fences means that
we need to wait for the fence(s) to signal before using any buffer related to
the in-fences.
diff --git a/Documentation/sysctl/net.txt b/Documentation/sysctl/net.txt
index 2ebabc9..14db18c 100644
--- a/Documentation/sysctl/net.txt
+++ b/Documentation/sysctl/net.txt
@@ -188,7 +188,16 @@
Maximum number of packets taken from all interfaces in one polling cycle (NAPI
poll). In one polling cycle interfaces which are registered to polling are
-probed in a round-robin manner.
+probed in a round-robin manner. Also, a polling cycle may not exceed
+netdev_budget_usecs microseconds, even if netdev_budget has not been
+exhausted.
+
+netdev_budget_usecs
+---------------------
+
+Maximum number of microseconds in one NAPI polling cycle. Polling
+will exit when either netdev_budget_usecs have elapsed during the
+poll cycle or the number of packets processed reaches netdev_budget.
netdev_max_backlog
------------------
diff --git a/Documentation/thermal/intel_powerclamp.txt b/Documentation/thermal/intel_powerclamp.txt
index 60073dc..b5df211 100644
--- a/Documentation/thermal/intel_powerclamp.txt
+++ b/Documentation/thermal/intel_powerclamp.txt
@@ -268,6 +268,15 @@
max_state:50
type:intel_powerclamp
+cur_state allows user to set the desired idle percentage. Writing 0 to
+cur_state will stop idle injection. Writing a value between 1 and
+max_state will start the idle injection. Reading cur_state returns the
+actual and current idle percentage. This may not be the same value
+set by the user in that current idle percentage depends on workload
+and includes natural idle. When idle injection is disabled, reading
+cur_state returns value -1 instead of 0 which is to avoid confusing
+100% busy state with the disabled state.
+
Example usage:
- To inject 25% idle time
$ sudo sh -c "echo 25 > /sys/class/thermal/cooling_device80/cur_state
@@ -278,11 +287,12 @@
will not show idle injection kernel threads.
If the system is busy (spin test below) and has less than 25% natural
-idle time, powerclamp kernel threads will do idle injection, which
-appear running to the scheduler. But the overall system idle is still
-reflected. In this example, 24.1% idle is shown. This helps the
-system admin or user determine the cause of slowdown, when a
-powerclamp driver is in action.
+idle time, powerclamp kernel threads will do idle injection. Forced
+idle time is accounted as normal idle in that common code path is
+taken as the idle task.
+
+In this example, 24.1% idle is shown. This helps the system admin or
+user determine the cause of slowdown, when a powerclamp driver is in action.
Tasks: 197 total, 1 running, 196 sleeping, 0 stopped, 0 zombie
diff --git a/Documentation/trace/ftrace.txt b/Documentation/trace/ftrace.txt
index 006f47c..94a987b 100644
--- a/Documentation/trace/ftrace.txt
+++ b/Documentation/trace/ftrace.txt
@@ -1546,7 +1546,7 @@
<idle>-0 3d..3 5us : 0:120:R ==> [003] 2389: 94:R sleep
-The 0:120:R means idle was running with a nice priority of 0 (120 - 20)
+The 0:120:R means idle was running with a nice priority of 0 (120 - 120)
and in the running state 'R'. The sleep task was scheduled in with
2389: 94:R. That is the priority is the kernel rtprio (99 - 5 = 94)
and it too is in the running state.
diff --git a/Documentation/trace/kprobetrace.txt b/Documentation/trace/kprobetrace.txt
index 41ef9d8..5ea8505 100644
--- a/Documentation/trace/kprobetrace.txt
+++ b/Documentation/trace/kprobetrace.txt
@@ -8,8 +8,9 @@
--------
These events are similar to tracepoint based events. Instead of Tracepoint,
this is based on kprobes (kprobe and kretprobe). So it can probe wherever
-kprobes can probe (this means, all functions body except for __kprobes
-functions). Unlike the Tracepoint based event, this can be added and removed
+kprobes can probe (this means, all functions except those with
+__kprobes/nokprobe_inline annotation and those marked NOKPROBE_SYMBOL).
+Unlike the Tracepoint based event, this can be added and removed
dynamically, on the fly.
To enable this feature, build your kernel with CONFIG_KPROBE_EVENTS=y.
diff --git a/Documentation/translations/ja_JP/HOWTO b/Documentation/translations/ja_JP/HOWTO
deleted file mode 100644
index 4ebd207..0000000
--- a/Documentation/translations/ja_JP/HOWTO
+++ /dev/null
@@ -1,637 +0,0 @@
-NOTE:
-This is a version of Documentation/HOWTO translated into Japanese.
-This document is maintained by Tsugikazu Shibata <tshibata@ab.jp.nec.com>
-and the JF Project team <www.linux.or.jp/JF>.
-If you find any difference between this document and the original file
-or a problem with the translation,
-please contact the maintainer of this file or JF project.
-
-Please also note that the purpose of this file is to be easier to read
-for non English (read: Japanese) speakers and is not intended as a
-fork. So if you have any comments or updates for this file, please try
-to update the original English file first.
-
-Last Updated: 2013/07/19
-==================================
-これは、
-linux-3.10/Documentation/HOWTO
-の和訳です。
-
-翻訳団体： JF プロジェクト < http://linuxjf.sourceforge.jp/ >
-翻訳日： 2013/7/19
-翻訳者： Tsugikazu Shibata <tshibata at ab dot jp dot nec dot com>
-校正者： 松倉さん <nbh--mats at nifty dot com>
- 小林 雅典さん (Masanori Kobayasi) <zap03216 at nifty dot ne dot jp>
- 武井伸光さん、<takei at webmasters dot gr dot jp>
- かねこさん (Seiji Kaneko) <skaneko at a2 dot mbn dot or dot jp>
- 野口さん (Kenji Noguchi) <tokyo246 at gmail dot com>
- 河内さん (Takayoshi Kochi) <t-kochi at bq dot jp dot nec dot com>
- 岩本さん (iwamoto) <iwamoto.kn at ncos dot nec dot co dot jp>
- 内田さん (Satoshi Uchida) <s-uchida at ap dot jp dot nec dot com>
-==================================
-
-Linux カーネル開発のやり方
--------------------------------
-
-これは上のトピック( Linux カーネル開発のやり方)の重要な事柄を網羅した
-ドキュメントです。ここには Linux カーネル開発者になるための方法と
-Linux カーネル開発コミュニティと共に活動するやり方を学ぶ方法が含まれて
-います。カーネルプログラミングに関する技術的な項目に関することは何も含
-めないようにしていますが、カーネル開発者となるための正しい方向に向かう
-手助けになります。
-
-もし、このドキュメントのどこかが古くなっていた場合には、このドキュメン
-トの最後にリストしたメンテナにパッチを送ってください。
-
-はじめに
----------
-
-あなたは Linux カーネルの開発者になる方法を学びたいのでしょうか？ そ
-れともあなたは上司から「このデバイスの Linux ドライバを書くように」と
-言われているのでしょうか？
-この文書の目的は、あなたが踏むべき手順と、コミュニティと一緒にうまく働
-くヒントを書き下すことで、あなたが知るべき全てのことを教えることです。
-また、このコミュニティがなぜ今うまくまわっているのかという理由の一部も
-説明しようと試みています。
-
-
-カーネルは 少量のアーキテクチャ依存部分がアセンブリ言語で書かれている
-以外は大部分は C 言語で書かれています。C言語をよく理解していることはカー
-ネル開発者には必要です。アーキテクチャ向けの低レベル部分の開発をするの
-でなければ、(どんなアーキテクチャでも)アセンブリ(訳注: 言語)は必要あり
-ません。以下の本は、C 言語の十分な知識や何年もの経験に取って代わるもの
-ではありませんが、少なくともリファレンスとしては良い本です。
- - "The C Programming Language" by Kernighan and Ritchie [Prentice Hall]
- -『プログラミング言語Ｃ第2版』(B.W. カーニハン/D.M. リッチー著 石田晴久訳) [共立出版]
- - "Practical C Programming" by Steve Oualline [O'Reilly]
- - 『C実践プログラミング第3版』(Steve Oualline著 望月康司監訳 谷口功訳) [オライリージャパン]
- - "C: A Reference Manual" by Harbison and Steele [Prentice Hall]
- - 『新・詳説 C 言語 H&S リファレンス』
- (サミュエル P ハービソン/ガイ L スティール共著 斉藤 信男監訳)[ソフトバンク]
-
-カーネルは GNU C と GNU ツールチェインを使って書かれています。カーネル
-は ISO C89 仕様に準拠して書く一方で、標準には無い言語拡張を多く使って
-います。カーネルは標準 C ライブラリとは関係がないといった、C 言語フリー
-スタンディング環境です。そのため、C の標準で使えないものもあります。任
-意の long long の除算や浮動小数点は使えません。
-ときどき、カーネルがツールチェインや C 言語拡張に置いている前提がどう
-なっているのかわかりにくいことがあり、また、残念なことに決定的なリファ
-レンスは存在しません。情報を得るには、gcc の info ページ( info gcc )を
-見てください。
-
-あなたは既存の開発コミュニティと一緒に作業する方法を学ぼうとしているこ
-とに留意してください。そのコミュニティは、コーディング、スタイル、
-開発手順について高度な標準を持つ、多様な人の集まりです。
-地理的に分散した大規模なチームに対してもっともうまくいくとわかったこと
-をベースにしながら、これらの標準は長い時間をかけて築かれてきました。
-これらはきちんと文書化されていますから、事前にこれらの標準についてでき
-るだけたくさん学んでください。また皆があなたやあなたの会社のやり方に合わ
-せてくれると思わないでください。
-
-法的問題
-------------
-
-Linux カーネルのソースコードは GPL ライセンスの下でリリースされていま
-す。ライセンスの詳細については、ソースツリーのメインディレクトリに存在
-する、COPYING のファイルを見てください。もしライセンスについてさらに質
-問があれば、Linux Kernel メーリングリストに質問するのではなく、どうぞ
-法律家に相談してください。メーリングリストの人達は法律家ではなく、法的
-問題については彼らの声明はあてにするべきではありません。
-
-GPL に関する共通の質問や回答については、以下を参照してください。
- http://www.gnu.org/licenses/gpl-faq.html
-
-ドキュメント
-------------
-
-Linux カーネルソースツリーは幅広い範囲のドキュメントを含んでおり、それ
-らはカーネルコミュニティと会話する方法を学ぶのに非常に貴重なものです。
-新しい機能がカーネルに追加される場合、その機能の使い方について説明した
-新しいドキュメントファイルも追加することを勧めます。
-カーネルの変更が、カーネルがユーザ空間に公開しているインターフェイスの
-変更を引き起こす場合、その変更を説明するマニュアルページのパッチや情報
-をマニュアルページのメンテナ mtk.manpages@gmail.com に送り、CC を
-linux-api@vger.kernel.org に送ることを勧めます。
-
-以下はカーネルソースツリーに含まれている読んでおくべきファイルの一覧で
-す-
-
- README
- このファイルは Linuxカーネルの簡単な背景とカーネルを設定(訳注
- configure )し、生成(訳注 build )するために必要なことは何かが書かれ
- ています。カーネルに関して初めての人はここからスタートすると良いで
- しょう。
-
- Documentation/Changes
- このファイルはカーネルをうまく生成(訳注 build )し、走らせるのに最
- 小限のレベルで必要な数々のソフトウェアパッケージの一覧を示してい
- ます。
-
- Documentation/process/coding-style.rst
- これは Linux カーネルのコーディングスタイルと背景にある理由を記述
- しています。全ての新しいコードはこのドキュメントにあるガイドライン
- に従っていることを期待されています。大部分のメンテナはこれらのルー
- ルに従っているものだけを受け付け、多くの人は正しいスタイルのコード
- だけをレビューします。
-
- Documentation/process/submitting-patches.rst
- Documentation/process/submitting-drivers.rst
- これらのファイルには、どうやってうまくパッチを作って投稿するかに
- ついて非常に詳しく書かれており、以下を含みます(これだけに限らない
- けれども)
- - Email に含むこと
- - Email の形式
- - だれに送るか
- これらのルールに従えばうまくいくことを保証することではありません
- が (すべてのパッチは内容とスタイルについて精査を受けるので)、
- ルールに従わなければ間違いなくうまくいかないでしょう。
-
- この他にパッチを作る方法についてのよくできた記述は-
-
- "The Perfect Patch"
- http://www.ozlabs.org/~akpm/stuff/tpp.txt
- "Linux kernel patch submission format"
- http://linux.yyz.us/patch-format.html
-
- Documentation/process/stable-api-nonsense.rst
- このファイルはカーネルの中に不変のAPIを持たないことにした意識的な
- 決断の背景にある理由について書かれています。以下のようなことを含
- んでいます-
- - サブシステムとの間に層を作ること(コンパチビリティのため?)
- - オペレーティングシステム間のドライバの移植性
- - カーネルソースツリーの素早い変更を遅らせる(もしくは素早い変更
- を妨げる)
- このドキュメントは Linux 開発の思想を理解するのに非常に重要です。
- そして、他のOSでの開発者が Linux に移る時にとても重要です。
-
- Documentation/admin-guide/security-bugs.rst
- もし Linux カーネルでセキュリティ問題を発見したように思ったら、こ
- のドキュメントのステップに従ってカーネル開発者に連絡し、問題解決を
- 支援してください。
-
- Documentation/process/management-style.rst
- このドキュメントは Linux カーネルのメンテナ達がどう行動するか、
- 彼らの手法の背景にある共有されている精神について記述しています。こ
- れはカーネル開発の初心者なら（もしくは、単に興味があるだけの人でも）
- 重要です。なぜならこのドキュメントは、カーネルメンテナ達の独特な
- 行動についての多くの誤解や混乱を解消するからです。
-
- Documentation/process/stable-kernel-rules.rst
- このファイルはどのように stable カーネルのリリースが行われるかのルー
- ルが記述されています。そしてこれらのリリースの中のどこかで変更を取
- り入れてもらいたい場合に何をすれば良いかが示されています。
-
- Documentation/process/kernel-docs.rst
- カーネル開発に付随する外部ドキュメントのリストです。もしあなたが
- 探しているものがカーネル内のドキュメントでみつからなかった場合、
- このリストをあたってみてください。
-
- Documentation/process/applying-patches.rst
- パッチとはなにか、パッチをどうやって様々なカーネルの開発ブランチに
- 適用するのかについて正確に記述した良い入門書です。
-
-カーネルはソースコードから自動的に生成可能な多数のドキュメントを自分自
-身でもっています。これにはカーネル内 API のすべての記述や、どう正しく
-ロックをかけるかの規則が含まれます。このドキュメントは
-Documentation/DocBook/ ディレクトリに作られ、以下のように
- make pdfdocs
- make psdocs
- make htmldocs
- make mandocs
-コマンドを実行するとメインカーネルのソースディレクトリから
-それぞれ、PDF, Postscript, HTML, man page の形式で生成されます。
-
-カーネル開発者になるには
----------------------------
-
-もしあなたが、Linux カーネル開発について何も知らないならば、
-KernelNewbies プロジェクトを見るべきです
- http://kernelnewbies.org
-
-このサイトには役に立つメーリングリストがあり、基本的なカーネル開発に関
-するほとんどどんな種類の質問もできます (既に回答されているようなことを
-聞く前にまずはアーカイブを調べてください)。
-またここには、リアルタイムで質問を聞くことができる IRC チャネルや、Linux
-カーネルの開発に関して学ぶのに便利なたくさんの役に立つドキュメントがあ
-ります。
-
-web サイトには、コードの構成、サブシステム、現在存在するプロジェクト(ツ
-リーにあるもの無いものの両方)の基本的な管理情報があります。
-ここには、また、カーネルのコンパイルのやり方やパッチの当て方などの間接
-的な基本情報も記述されています。
-
-あなたがどこからスタートして良いかわからないが、Linux カーネル開発コミュ
-ニティに参加して何かすることをさがしている場合には、Linux kernel
-Janitor's プロジェクトにいけば良いでしょう -
- http://kernelnewbies.org/KernelJanitors
-ここはそのようなスタートをするのにうってつけの場所です。ここには、
-Linux カーネルソースツリーの中に含まれる、きれいにし、修正しなければな
-らない、単純な問題のリストが記述されています。このプロジェクトに関わる
-開発者と一緒に作業することで、あなたのパッチを Linuxカーネルツリーに入
-れるための基礎を学ぶことができ、そしてもしあなたがまだアイディアを持っ
-ていない場合には、次にやる仕事の方向性が見えてくるかもしれません。
-
-もしあなたが、すでにひとまとまりコードを書いていて、カーネルツリーに入
-れたいと思っていたり、それに関する適切な支援を求めたい場合、カーネル
-メンターズプロジェクトはそのような皆さんを助けるためにできました。
-ここにはメーリングリストがあり、以下から参照できます
- http://selenic.com/mailman/listinfo/kernel-mentors
-
-実際に Linux カーネルのコードについて修正を加える前に、どうやってその
-コードが動作するのかを理解することが必要です。そのためには、特別なツー
-ルの助けを借りてでも、それを直接よく読むことが最良の方法です(ほとんど
-のトリッキーな部分は十分にコメントしてありますから)。そういうツールで
-特におすすめなのは、Linux クロスリファレンスプロジェクトです。これは、
-自己参照方式で、索引がついた web 形式で、ソースコードを参照することが
-できます。この最新の素晴しいカーネルコードのリポジトリは以下で見つかり
-ます-
- http://lxr.free-electrons.com/
-
-開発プロセス
------------------------
-
-Linux カーネルの開発プロセスは現在幾つかの異なるメインカーネル「ブラン
-チ」と多数のサブシステム毎のカーネルブランチから構成されます。
-これらのブランチとは-
- - メインの 3.x カーネルツリー
- - 3.x.y -stable カーネルツリー
- - 3.x -git カーネルパッチ
- - サブシステム毎のカーネルツリーとパッチ
- - 統合テストのための 3.x -next カーネルツリー
-
-3.x カーネルツリー
------------------
-
-3.x カーネルは Linus Torvalds によってメンテナンスされ、kernel.org
-の pub/linux/kernel/v3.x/ ディレクトリに存在します。この開発プロセスは
-以下のとおり-
-
- - 新しいカーネルがリリースされた直後に、2週間の特別期間が設けられ、
- この期間中に、メンテナ達は Linus に大きな差分を送ることができます。
- このような差分は通常 -next カーネルに数週間含まれてきたパッチです。
- 大きな変更は git(カーネルのソース管理ツール、詳細は
- http://git-scm.com/ 参照) を使って送るのが好ましいやり方ですが、パッ
- チファイルの形式のまま送るのでも十分です。
-
- - 2週間後、-rc1 カーネルがリリースされ、この後にはカーネル全体の安定
- 性に影響をあたえるような新機能は含まない類のパッチしか取り込むこと
- はできません。新しいドライバ(もしくはファイルシステム)のパッチは
- -rc1 の後で受け付けられることもあることを覚えておいてください。な
- ぜなら、変更が独立していて、追加されたコードの外の領域に影響を与え
- ない限り、退行のリスクは無いからです。-rc1 がリリースされた後、
- Linus へパッチを送付するのに git を使うこともできますが、パッチは
- レビューのために、パブリックなメーリングリストへも同時に送る必要が
- あります。
-
- - 新しい -rc は Linus が、最新の git ツリーがテスト目的であれば十分
- に安定した状態にあると判断したときにリリースされます。目標は毎週新
- しい -rc カーネルをリリースすることです。
-
- - このプロセスはカーネルが 「準備ができた」と考えられるまで継続しま
- す。このプロセスはだいたい 6週間継続します。
-
-Andrew Morton が Linux-kernel メーリングリストにカーネルリリースについ
-て書いたことをここで言っておくことは価値があります-
- 「カーネルがいつリリースされるかは誰も知りません。なぜなら、これは現
- 実に認識されたバグの状況によりリリースされるのであり、前もって決めら
- れた計画によってリリースされるものではないからです。」
-
-3.x.y -stable カーネルツリー
----------------------------
-
-バージョン番号が3つの数字に分かれているカーネルは -stable カーネルです。
-これには、3.x カーネルで見つかったセキュリティ問題や重大な後戻りに対
-する比較的小さい重要な修正が含まれます。
-
-これは、開発/実験的バージョンのテストに協力することに興味が無く、
-最新の安定したカーネルを使いたいユーザに推奨するブランチです。
-
-もし、3.x.y カーネルが存在しない場合には、番号が一番大きい 3.x が
-最新の安定版カーネルです。
-
-3.x.y は "stable" チーム <stable@vger.kernel.org> でメンテされており、必
-要に応じてリリースされます。通常のリリース期間は 2週間毎ですが、差し迫っ
-た問題がなければもう少し長くなることもあります。セキュリティ関連の問題
-の場合はこれに対してだいたいの場合、すぐにリリースがされます。
-
-カーネルツリーに入っている、Documentation/process/stable-kernel-rules.rst ファ
-イルにはどのような種類の変更が -stable ツリーに受け入れ可能か、またリ
-リースプロセスがどう動くかが記述されています。
-
-3.x -git パッチ
-------------------
-
-git リポジトリで管理されているLinus のカーネルツリーの毎日のスナップ
-ショットがあります。(だから -git という名前がついています)。これらのパッ
-チはおおむね毎日リリースされており、Linus のツリーの現状を表します。こ
-れは -rc カーネルと比べて、パッチが大丈夫かどうかも確認しないで自動的
-に生成されるので、より実験的です。
-
-サブシステム毎のカーネルツリーとパッチ
--------------------------------------------
-
-それぞれのカーネルサブシステムのメンテナ達は --- そして多くのカーネル
-サブシステムの開発者達も --- 各自の最新の開発状況をソースリポジトリに
-公開しています。そのため、自分とは異なる領域のカーネルで何が起きている
-かを他の人が見られるようになっています。開発が早く進んでいる領域では、
-開発者は自身の投稿がどのサブシステムカーネルツリーを元にしているか質問
-されるので、その投稿とすでに進行中の他の作業との衝突が避けられます。
-
-大部分のこれらのリポジトリは git ツリーです。しかしその他の SCM や
-quilt シリーズとして公開されているパッチキューも使われています。これら
-のサブシステムリポジトリのアドレスは MAINTAINERS ファイルにリストされ
-ています。これらの多くは http://git.kernel.org/ で参照することができま
-す。
-
-提案されたパッチがこのようなサブシステムツリーにコミットされる前に、メー
-リングリストで事前にレビューにかけられます（以下の対応するセクションを
-参照）。いくつかのカーネルサブシステムでは、このレビューは patchwork
-というツールによって追跡されます。Patchwork は web インターフェイスに
-よってパッチ投稿の表示、パッチへのコメント付けや改訂などができ、そして
-メンテナはパッチに対して、レビュー中、受付済み、拒否というようなマーク
-をつけることができます。大部分のこれらの patchwork のサイトは
-http://patchwork.kernel.org/ でリストされています。
-
-統合テストのための 3.x -next カーネルツリー
----------------------------------------------
-
-サブシステムツリーの更新内容がメインラインの 3.x ツリーにマージされ
-る前に、それらは統合テストされる必要があります。この目的のため、実質的
-に全サブシステムツリーからほぼ毎日プルされてできる特別なテスト用のリ
-ポジトリが存在します-
- http://git.kernel.org/?p=linux/kernel/git/next/linux-next.git
-
-このやり方によって、-next カーネルは次のマージ機会でどんなものがメイン
-ラインカーネルにマージされるか、おおまかなの展望を提供します。-next
-カーネルの実行テストを行う冒険好きなテスターは大いに歓迎されます
-
-バグレポート
--------------
-
-bugzilla.kernel.org は Linux カーネル開発者がカーネルのバグを追跡する
-場所です。ユーザは見つけたバグの全てをこのツールで報告すべきです。
-どう kernel bugzilla を使うかの詳細は、以下を参照してください-
- http://bugzilla.kernel.org/page.cgi?id=faq.html
-メインカーネルソースディレクトリにあるファイル admin-guide/reporting-bugs.rst はカーネ
-ルバグらしいものについてどうレポートするかの良いテンプレートであり、問
-題の追跡を助けるためにカーネル開発者にとってどんな情報が必要なのかの詳
-細が書かれています。
-
-バグレポートの管理
--------------------
-
-あなたのハッキングのスキルを訓練する最高の方法のひとつに、他人がレポー
-トしたバグを修正することがあります。あなたがカーネルをより安定化させる
-こに寄与するということだけでなく、あなたは 現実の問題を修正することを
-学び、自分のスキルも強化でき、また他の開発者があなたの存在に気がつき
-ます。バグを修正することは、多くの開発者の中から自分が功績をあげる最善
-の道です、なぜなら多くの人は他人のバグの修正に時間を浪費することを好ま
-ないからです。
-
-すでにレポートされたバグのために仕事をするためには、
-http://bugzilla.kernel.org に行ってください。もし今後のバグレポートに
-ついてアドバイスを受けたいのであれば、bugme-new メーリングリスト(新し
-いバグレポートだけがここにメールされる) または bugme-janitor メーリン
-グリスト(bugzilla の変更毎にここにメールされる)を購読できます。
-
- http://lists.linux-foundation.org/mailman/listinfo/bugme-new
- http://lists.linux-foundation.org/mailman/listinfo/bugme-janitors
-
-メーリングリスト
--------------
-
-上のいくつかのドキュメントで述べていますが、コアカーネル開発者の大部分
-は Linux kernel メーリングリストに参加しています。このリストの登録/脱
-退の方法については以下を参照してください-
- http://vger.kernel.org/vger-lists.html#linux-kernel
-
-このメーリングリストのアーカイブは web 上の多数の場所に存在します。こ
-れらのアーカイブを探すにはサーチエンジンを使いましょう。例えば-
- http://dir.gmane.org/gmane.linux.kernel
-
-リストに投稿する前にすでにその話題がアーカイブに存在するかどうかを検索
-することを是非やってください。多数の事がすでに詳細に渡って議論されて
-おり、アーカイブにのみ記録されています。
-
-大部分のカーネルサブシステムも自分の個別の開発を実施するメーリングリス
-トを持っています。個々のグループがどんなリストを持っているかは、
-MAINTAINERS ファイルにリストがありますので参照してください。
-
-多くのリストは kernel.org でホストされています。これらの情報は以下にあ
-ります-
- http://vger.kernel.org/vger-lists.html
-
-メーリングリストを使う場合、良い行動習慣に従うようにしましょう。
-少し安っぽいが、以下の URL は上のリスト(や他のリスト)で会話する場合の
-シンプルなガイドラインを示しています-
- http://www.albion.com/netiquette/
-
-もし複数の人があなたのメールに返事をした場合、CC: で受ける人のリストは
-だいぶ多くなるでしょう。良い理由がない場合、CC: リストから誰かを削除を
-しないように、また、メーリングリストのアドレスだけにリプライすることの
-ないようにしましょう。1つは送信者から、もう1つはリストからのように、メー
-ルを2回受けることになってもそれに慣れ、しゃれたメールヘッダーを追加し
-てこの状態を変えようとしないように。人々はそのようなことは好みません。
-
-今までのメールでのやりとりとその間のあなたの発言はそのまま残し、
-"John Kernelhacker wrote ...:" の行をあなたのリプライの先頭行にして、
-メールの先頭でなく、各引用行の間にあなたの言いたいことを追加するべきで
-す。
-
-もしパッチをメールに付ける場合は、Documentation/process/submitting-patches.rst に提
-示されているように、それは プレーンな可読テキストにすることを忘れない
-ようにしましょう。カーネル開発者は 添付や圧縮したパッチを扱いたがりま
-せん-
-彼らはあなたのパッチの行毎にコメントを入れたいので、そのためにはそうす
-るしかありません。あなたのメールプログラムが空白やタブを圧縮しないよう
-に確認した方が良いです。最初の良いテストとしては、自分にメールを送って
-みて、そのパッチを自分で当ててみることです。もしそれがうまく行かないな
-ら、あなたのメールプログラムを直してもらうか、正しく動くように変えるべ
-きです。
-
-とりわけ、他の登録者に対する尊敬を表すようにすることを覚えておいてくだ
-さい。
-
-コミュニティと共に働くこと
---------------------------
-
-カーネルコミュニティのゴールは可能なかぎり最高のカーネルを提供すること
-です。あなたがパッチを受け入れてもらうために投稿した場合、それは、技術
-的メリットだけがレビューされます。その際、あなたは何を予想すべきでしょ
-うか?
- - 批判
- - コメント
- - 変更の要求
- - パッチの正当性の証明要求
- - 沈黙
-
-思い出してください、ここはあなたのパッチをカーネルに入れる話です。あ
-なたは、あなたのパッチに対する批判とコメントを受け入れるべきで、それら
-を技術的レベルで評価して、パッチを再作成するか、なぜそれらの変更をすべ
-きでないかを明確で簡潔な理由の説明を提供してください。
-もし、あなたのパッチに何も反応がない場合、たまにはメールの山に埋もれて
-見逃され、あなたの投稿が忘れられてしまうこともあるので、数日待って再度
-投稿してください。
-
-あなたがやるべきでないものは?
- - 質問なしにあなたのパッチが受け入れられると想像すること
- - 守りに入ること
- - コメントを無視すること
- - 要求された変更を何もしないでパッチを出し直すこと
-
-可能な限り最高の技術的解決を求めているコミュニティでは、パッチがどのく
-らい有益なのかについては常に異なる意見があります。あなたは協調的である
-べきですし、また、あなたのアイディアをカーネルに対してうまく合わせるよ
-うにすることが望まれています。もしくは、最低限あなたのアイディアがそれ
-だけの価値があるとすすんで証明するようにしなければなりません。
-正しい解決に向かって進もうという意志がある限り、間違うことがあっても許
-容されることを忘れないでください。
-
-あなたの最初のパッチに単に 1ダースもの修正を求めるリストの返答になるこ
-とも普通のことです。これはあなたのパッチが受け入れられないということで
-は *ありません*、そしてあなた自身に反対することを意味するのでも *ありま
-せん*。単に自分のパッチに対して指摘された問題を全て修正して再送すれば
-良いのです。
-
-
-カーネルコミュニティと企業組織のちがい
------------------------------------------------------------------
-
-カーネルコミュニティは大部分の伝統的な会社の開発環境とは異ったやり方で
-動いています。以下は問題を避けるためにできると良いことのリストです-
-
- あなたの提案する変更について言うときのうまい言い方：
-
- - "これは複数の問題を解決します"
- - "これは2000行のコードを削除します"
- - "以下のパッチは、私が言おうとしていることを説明するものです"
- - "私はこれを5つの異なるアーキテクチャでテストしたのですが..."
- - "以下は一連の小さなパッチ群ですが..."
- - "これは典型的なマシンでの性能を向上させます.."
-
- やめた方が良い悪い言い方：
-
- - このやり方で AIX/ptx/Solaris ではできたので、できるはずだ
- - 私はこれを20年もの間やってきた、だから
- - これは、私の会社が金儲けをするために必要だ
- - これは我々のエンタープライズ向け商品ラインのためである
- - これは 私が自分のアイディアを記述した、1000ページの設計資料である
- - 私はこれについて、6ケ月作業している。
- - 以下は ... に関する5000行のパッチです
- - 私は現在のぐちゃぐちゃを全部書き直した、それが以下です...
- - 私は〆切がある、そのためこのパッチは今すぐ適用される必要がある
-
-カーネルコミュニティが大部分の伝統的なソフトウェアエンジニアリングの労
-働環境と異なるもう一つの点は、やりとりに顔を合わせないということです。
-email と irc を第一のコミュニケーションの形とする一つの利点は、性別や
-民族の差別がないことです。Linux カーネルの職場環境は女性や少数民族を受
-容します。なぜなら、email アドレスによってのみあなたが認識されるからで
-す。
-国際的な側面からも活動領域を均等にするようにします。なぜならば、あなた
-は人の名前で性別を想像できないからです。ある男性が アンドレアという名
-前で、女性の名前は パット かもしれません (訳注 Andrea は米国では女性、
-それ以外(欧州など)では男性名として使われることが多い。同様に、Pat は
-Patricia (主に女性名)や Patrick (主に男性名)の略称)。
-Linux カーネルの活動をして、意見を表明したことがある大部分の女性は、前
-向きな経験をもっています。
-
-言葉の壁は英語が得意でない一部の人には問題になります。
-メーリングリストの中できちんとアイディアを交換するには、相当うまく英語
-を操れる必要があることもあります。そのため、あなたは自分のメール
-を送る前に英語で意味が通じているかをチェックすることをお薦めします。
-
-変更を分割する
----------------------
-
-Linux カーネルコミュニティは、一度に大量のコードの塊を喜んで受容するこ
-とはありません。変更は正確に説明される必要があり、議論され、小さい、個
-別の部分に分割する必要があります。これはこれまで多くの会社がやり慣れて
-きたことと全く正反対のことです。あなたのプロポーザルは、開発プロセスのと
-ても早い段階から紹介されるべきです。そうすれば あなたは自分のやってい
-ることにフィードバックを得られます。これは、コミュニティからみれば、あ
-なたが彼らと一緒にやっているように感じられ、単にあなたの提案する機能の
-ゴミ捨て場として使っているのではない、と感じられるでしょう。
-しかし、一度に 50 もの email をメーリングリストに送りつけるようなことは
-やってはいけません、あなたのパッチ群はいつもどんな時でもそれよりは小さ
-くなければなりません。
-
-パッチを分割する理由は以下です-
-
-1) 小さいパッチはあなたのパッチが適用される見込みを大きくします、カー
- ネルの人達はパッチが正しいかどうかを確認する時間や労力をかけないか
- らです。5行のパッチはメンテナがたった1秒見るだけで適用できます。
- しかし、500行のパッチは、正しいことをレビューするのに数時間かかるか
- もしれません(時間はパッチのサイズなどにより指数関数に比例してかかり
- ます)
-
- 小さいパッチは何かあったときにデバッグもとても簡単になります。パッ
- チを1個1個取り除くのは、とても大きなパッチを当てた後に(かつ、何かお
- かしくなった後で)解剖するのに比べればとても簡単です。
-
-2) 小さいパッチを送るだけでなく、送るまえに、書き直して、シンプルにす
- る(もしくは、単に順番を変えるだけでも)ことも、とても重要です。
-
-以下はカーネル開発者の Al Viro のたとえ話です：
-
- "生徒の数学の宿題を採点する先生のことを考えてみてください、先
- 生は生徒が解に到達するまでの試行錯誤を見たいとは思わないでしょ
- う。先生は簡潔な最高の解を見たいのです。良い生徒はこれを知って
- おり、そして最終解の前の中間作業を提出することは決してないので
- す"
-
- カーネル開発でもこれは同じです。メンテナ達とレビューア達は、
- 問題を解決する解の背後になる思考プロセスを見たいとは思いません。
- 彼らは単純であざやかな解決方法を見たいのです。
-
-あざやかな解を説明するのと、コミュニティと共に仕事をし、未解決の仕事を
-議論することのバランスをキープするのは難しいかもしれません。
-ですから、開発プロセスの早期段階で改善のためのフィードバックをもらうよ
-うにするのも良いですが、変更点を小さい部分に分割して全体ではまだ完成し
-ていない仕事を(部分的に)取り込んでもらえるようにすることも良いことです。
-
-また、でき上がっていないものや、"将来直す" ようなパッチを、本流に含め
-てもらうように送っても、それは受け付けられないことを理解してください。
-
-あなたの変更を正当化する
--------------------
-
-あなたのパッチを分割するのと同時に、なぜその変更を追加しなければならな
-いかを Linux コミュニティに知らせることはとても重要です。新機能は必要
-性と有用性で正当化されなければなりません。
-
-あなたの変更の説明
---------------------
-
-あなたのパッチを送付する場合には、メールの中のテキストで何を言うかにつ
-いて、特別に注意を払ってください。この情報はパッチの ChangeLog に使わ
-れ、いつも皆がみられるように保管されます。これは次のような項目を含め、
-パッチを完全に記述するべきです-
-
- - なぜ変更が必要か
- - パッチ全体の設計アプローチ
- - 実装の詳細
- - テスト結果
-
-これについて全てがどのようにあるべきかについての詳細は、以下のドキュメ
-ントの ChangeLog セクションを見てください-
- "The Perfect Patch"
- http://www.ozlabs.org/~akpm/stuff/tpp.txt
-
-これらのどれもが、時にはとても困難です。これらの慣例を完璧に実施するに
-は数年かかるかもしれません。これは継続的な改善のプロセスであり、そのた
-めには多数の忍耐と決意を必要とするものです。でも、諦めないで、これは可
-能なことです。多数の人がすでにできていますし、彼らも皆最初はあなたと同
-じところからスタートしたのですから。
-
-Paolo Ciarrocchi に感謝、彼は彼の書いた "Development Process"
-(http://lwn.net/Articles/94386/) セクションをこのテキストの原型にする
-ことを許可してくれました。Rundy Dunlap と Gerrit Huizenga はメーリング
-リストでやるべきこととやってはいけないことのリストを提供してくれました。
-以下の人々のレビュー、コメント、貢献に感謝。
-Pat Mochel, Hanna Linder, Randy Dunlap, Kay Sievers,
-Vojtech Pavlik, Jan Kara, Josh Boyer, Kees Cook, Andrew Morton, Andi
-Kleen, Vadim Lobanov, Jesper Juhl, Adrian Bunk, Keri Harris, Frans Pop,
-David A. Wheeler, Junio Hamano, Michael Kerrisk, と Alex Shepard
-彼らの支援なしでは、このドキュメントはできなかったでしょう。
-
-Maintainer: Greg Kroah-Hartman <greg@kroah.com>
diff --git a/Documentation/translations/ja_JP/howto.rst b/Documentation/translations/ja_JP/howto.rst
new file mode 100644
index 0000000..4511eed
--- /dev/null
+++ b/Documentation/translations/ja_JP/howto.rst
@@ -0,0 +1,661 @@
+NOTE:
+This is a version of Documentation/HOWTO translated into Japanese.
+This document is maintained by Tsugikazu Shibata <tshibata@ab.jp.nec.com>
+If you find any difference between this document and the original file or
+a problem with the translation, please contact the maintainer of this file.
+
+Please also note that the purpose of this file is to be easier to
+read for non English (read: Japanese) speakers and is not intended as
+a fork. So if you have any comments or updates for this file, please
+try to update the original English file first.
+
+----------------------------------
+
+この文書は、
+Documentation/process/howto.rst
+の和訳です。
+
+翻訳者： Tsugikazu Shibata <tshibata@ab.jp.nec.com>
+
+----------------------------------
+
+Linux カーネル開発のやり方
+==========================
+
+これは上のトピック( Linux カーネル開発のやり方)の重要な事柄を網羅した
+ドキュメントです。ここには Linux カーネル開発者になるための方法とLinux
+カーネル開発コミュニティと共に活動するやり方を学ぶ方法が含まれています。
+カーネルプログラミングに関する技術的な項目に関することは何も含めないよ
+うにしていますが、カーネル開発者となるための正しい方向に向かう手助けに
+なります。
+
+もし、このドキュメントのどこかが古くなっていた場合には、このドキュメント
+の最後にリストしたメンテナにパッチを送ってください。
+
+はじめに
+---------
+
+あなたは Linux カーネルの開発者になる方法を学びたいのでしょうか？ そ
+れとも上司から「このデバイスの Linux ドライバを書くように」と言われた
+のかもしれません。この文書の目的は、あなたが踏むべき手順と、コミュニティ
+と一緒にうまく働くヒントを書き下すことで、あなたが知るべき全てのことを
+教えることです。また、このコミュニティがなぜ今うまくまわっているのかと
+いう理由も説明しようと試みています。
+
+カーネルは少量のアーキテクチャ依存部分がアセンブリ言語で書かれている以
+外の大部分は C 言語で書かれています。C言語をよく理解していることはカー
+ネル開発に必要です。低レベルのアーキテクチャ開発をするのでなければ、
+(どんなアーキテクチャでも)アセンブリ(訳注: 言語)は必要ありません。以下
+の本は、C 言語の十分な知識や何年もの経験に取って代わるものではありませ
+んが、少なくともリファレンスとしては良い本です。
+
+ - "The C Programming Language" by Kernighan and Ritchie [Prentice Hall]
+ - 『プログラミング言語Ｃ第2版』(B.W. カーニハン/D.M. リッチー著 石田晴久訳) [共立出版]
+ - "Practical C Programming" by Steve Oualline [O'Reilly]
+ - 『C実践プログラミング第3版』(Steve Oualline著 望月康司監訳 谷口功訳) [オライリージャパン]
+ - "C: A Reference Manual" by Harbison and Steele [Prentice Hall]
+ - 『新・詳説 C 言語 H&S リファレンス』 (サミュエル P ハービソン/ガイ L スティール共著 斉藤 信男監訳)[ソフトバンク]
+
+カーネルは GNU C と GNU ツールチェインを使って書かれています。カーネル
+は ISO C89 仕様に準拠して書く一方で、標準には無い言語拡張を多く使って
+います。カーネルは標準 C ライブラリに依存しない、C 言語非依存環境です。
+そのため、C の標準の中で使えないものもあります。特に任意の long long
+の除算や浮動小数点は使えません。カーネルがツールチェインや C 言語拡張
+に置いている前提がどうなっているのかわかりにくいことが時々あり、また、
+残念なことに決定的なリファレンスは存在しません。情報を得るには、gcc の
+info ページ( info gcc )を見てください。
+
+あなたは既存の開発コミュニティと一緒に作業する方法を学ぼうとしているこ
+とに思い出してください。そのコミュニティは、コーディング、スタイル、開
+発手順について高度な標準を持つ、多様な人の集まりです。地理的に分散した
+大規模なチームに対してもっともうまくいくとわかったことをベースにしなが
+ら、これらの標準は長い時間をかけて築かれてきました。これらはきちんと文
+書化されていますから、事前にこれらの標準について事前にできるだけたくさ
+ん学んでください。また皆があなたやあなたの会社のやり方に合わせてくれる
+と思わないでください。
+
+法的問題
+--------
+
+Linux カーネルのソースコードは GPL ライセンスの下でリリースされていま
+す。ライセンスの詳細については、ソースツリーのメインディレクトリに存在
+する、COPYING のファイルを見てください。もしライセンスについてさらに質
+問があれば、Linux Kernel メーリングリストに質問するのではなく、どうぞ
+法律家に相談してください。メーリングリストの人達は法律家ではなく、法的
+問題については彼らの声明はあてにするべきではありません。
+
+GPL に関する共通の質問や回答については、以下を参照してください-
+
+ https://www.gnu.org/licenses/gpl-faq.html
+
+ドキュメント
+------------
+
+Linux カーネルソースツリーは幅広い範囲のドキュメントを含んでおり、それ
+らはカーネルコミュニティと会話する方法を学ぶのに非常に貴重なものです。
+新しい機能がカーネルに追加される場合、その機能の使い方について説明した
+新しいドキュメントファイルも追加することを勧めます。
+カーネルの変更が、カーネルがユーザ空間に公開しているインターフェイスの
+変更を引き起こす場合、その変更を説明するマニュアルページのパッチや情報
+をマニュアルページのメンテナ mtk.manpages@gmail.com に送り、CC を
+linux-api@vger.kernel.org に送ることを勧めます。
+
+以下はカーネルソースツリーに含まれている読んでおくべきファイルの一覧で
+す-
+
+ README
+ このファイルは Linuxカーネルの簡単な背景とカーネルを設定(訳注
+ configure )し、生成(訳注 build )するために必要なことは何かが書かれ
+ ています。 カーネルに関して初めての人はここからスタートすると良い
+ でしょう。
+
+ :ref:`Documentation/Process/changes.rst <changes>`
+ このファイルはカーネルをうまく生成(訳注 build )し、走らせるのに最
+ 小限のレベルで必要な数々のソフトウェアパッケージの一覧を示してい
+ ます。
+
+ :ref:`Documentation/process/coding-style.rst <codingstyle>`
+ これは Linux カーネルのコーディングスタイルと背景にある理由を記述
+ しています。全ての新しいコードはこのドキュメントにあるガイドライン
+ に従っていることを期待されています。大部分のメンテナはこれらのルー
+ ルに従っているものだけを受け付け、多くの人は正しいスタイルのコード
+ だけをレビューします。
+
+ :ref:`Documentation/process/submitting-patches.rst <codingstyle>` と :ref:`Documentation/process/submitting-drivers.rst <submittingdrivers>`
+ これらのファイルには、どうやってうまくパッチを作って投稿するかにつ
+ いて非常に詳しく書かれており、以下を含みます (これだけに限らない
+ けれども)
+
+ - Email に含むこと
+ - Email の形式
+ - だれに送るか
+
+ これらのルールに従えばうまくいくことを保証することではありません
+ が (すべてのパッチは内容とスタイルについて精査を受けるので)、
+ ルールに従わなければ間違いなくうまくいかないでしょう。
+
+ この他にパッチを作る方法についてのよくできた記述は-
+
+ "The Perfect Patch"
+ http://www.ozlabs.org/~akpm/stuff/tpp.txt
+ "Linux kernel patch submission format"
+ http://linux.yyz.us/patch-format.html
+
+ :ref:`Documentation/process/stable-api-nonsense.rst <stable_api_nonsense>`
+ このファイルはカーネルの中に不変の API を持たないことにした意識的
+ な決断の背景にある理由について書かれています。以下のようなことを含
+ んでいます-
+
+ - サブシステムとの間に層を作ること(コンパチビリティのため?)
+ - オペレーティングシステム間のドライバの移植性
+ - カーネルソースツリーの素早い変更を遅らせる(もしくは素早い変更を妨げる)
+
+ このドキュメントは Linux 開発の思想を理解するのに非常に重要です。
+ そして、他のOSでの開発者が Linux に移る時にとても重要です。
+
+ :ref:`Documentation/admin-guide/security-bugs.rst <securitybugs>`
+ もし Linux カーネルでセキュリティ問題を発見したように思ったら、こ
+ のドキュメントのステップに従ってカーネル開発者に連絡し、問題解決を
+ 支援してください。
+
+ :ref:`Documentation/process/management-style.rst <managementstyle>`
+ このドキュメントは Linux カーネルのメンテナ達がどう行動するか、
+ 彼らの手法の背景にある共有されている精神について記述しています。こ
+ れはカーネル開発の初心者なら（もしくは、単に興味があるだけの人でも）
+ 重要です。なぜならこのドキュメントは、カーネルメンテナ達の独特な
+ 行動についての多くの誤解や混乱を解消するからです。
+
+ :ref:`Documentation/process/stable-kernel-rules.rst <stable_kernel_rules>`
+ このファイルはどのように stable カーネルのリリースが行われるかのルー
+ ルが記述されています。そしてこれらのリリースの中のどこかで変更を取
+ り入れてもらいたい場合に何をすれば良いかが示されています。
+
+ :Ref:`Documentation/process/kernel-docs.rst <kernel_docs>`
+ カーネル開発に付随する外部ドキュメントのリストです。もしあなたが探
+ しているものがカーネル内のドキュメントでみつからなかった場合、この
+ リストをあたってみてください。
+
+ :ref:`Documentation/process/applying-patches.rst <applying_patches>`
+ パッチとはなにか、パッチをどうやって様々なカーネルの開発ブランチに
+ 適用するのかについて正確に記述した良い入門書です。
+
+カーネルはソースコードそのものや、このファイルのようなリストラクチャー
+ドテキストマークアップ(ReST)から自動的に生成可能な多数のドキュメントを
+もっています。これにはカーネル内APIの完全な記述や、正しくロックをかけ
+るための規則などが含まれます。
+
+これら全てのドキュメントを PDF や HTML で生成するには以下を実行します - ::
+
+ make pdfdocs
+ make htmldocs
+
+それぞれメインカーネルのソースディレクトリから実行します。
+
+ReSTマークアップを使ったドキュメントは Documentation/outputに生成され
+ます。Latex とePub 形式で生成するには - ::
+
+ make latexdocs
+ make epubdocs
+
+現在、幾つかの DocBook形式で書かれたドキュメントは ReST形式に転換中で
+す。それらのドキュメントはDocumentation/DocBook ディレクトリに生成され、
+Postscript または man ページの形式を生成するには以下のようにします - ::
+
+ make psdocs
+ make mandocs
+
+カーネル開発者になるには
+------------------------
+
+もしあなたが、Linux カーネル開発について何も知らないのならば、
+KernelNewbies プロジェクトを見るべきです
+
+ https://kernelnewbies.org
+
+このサイトには役に立つメーリングリストがあり、基本的なカーネル開発に関
+するほとんどどんな種類の質問もできます (既に回答されているようなことを
+聞く前にまずはアーカイブを調べてください)。またここには、リアルタイム
+で質問を聞くことができる IRC チャネルや、Linuxカーネルの開発に関して学
+ぶのに便利なたくさんの役に立つドキュメントがあります。
+
+Web サイトには、コードの構成、サブシステム、現在存在するプロジェクト
+(ツリーにあるもの無いものの両方)の基本的な管理情報があります。ここには、
+また、カーネルのコンパイルのやり方やパッチの当て方などの間接的な基本情
+報も記述されています。
+
+あなたがどこからスタートして良いかわからないが、Linux カーネル開発コミュ
+ニティに参加して何かすることをさがしているのであれば、Linux kernel
+Janitor's プロジェクトにいけば良いでしょう -
+
+ https://kernelnewbies.org/KernelJanitors
+
+ここはそのようなスタートをするのにうってつけの場所です。ここには、
+Linux カーネルソースツリーの中に含まれる、きれいにし、修正しなければな
+らない、単純な問題のリストが記述されています。このプロジェクトに関わる
+開発者と一緒に作業することで、あなたのパッチを Linuxカーネルツリーに入
+れるための基礎を学ぶことができ、そしてもしあなたがまだアイディアを持っ
+ていない場合には、次にやる仕事の方向性が見えてくるかもしれません。
+
+もしあなたが、すでにひとまとまりコードを書いていて、カーネルツリーに入
+れたいと思っていたり、それに関する適切な支援を求めたい場合、カーネルメ
+ンターズプロジェクトはそのような皆さんを助けるためにできました。ここに
+はメーリングリストがあり、以下から参照できます -
+
+ https://selenic.com/mailman/listinfo/kernel-mentors
+
+実際に Linux カーネルのコードについて修正を加える前に、どうやってその
+コードが動作するのかを理解することが必要です。そのためには、特別なツー
+ルの助けを借りてでも、それを直接よく読むことが最良の方法です(ほとんど
+のトリッキーな部分は十分にコメントしてありますから)。そういうツールで
+特におすすめなのは、Linux クロスリファレンスプロジェクトです。これは、
+自己参照方式で、索引がついた web 形式で、ソースコードを参照することが
+できます。この最新の素晴しいカーネルコードのリポジトリは以下で見つかり
+ます -
+
+ http://lxr.free-electrons.com/
+
+開発プロセス
+------------
+
+Linux カーネルの開発プロセスは現在幾つかの異なるメインカーネル「ブラン
+チ」と多数のサブシステム毎のカーネルブランチから構成されます。これらの
+ブランチとは -
+
+ - メインの 4.x カーネルツリー
+ - 4.x.y -stable カーネルツリー
+ - 4.x -git カーネルパッチ
+ - サブシステム毎のカーネルツリーとパッチ
+ - 統合テストのための 4.x -next カーネルツリー
+
+4.x カーネルツリー
+~~~~~~~~~~~~~~~~~~
+
+4.x カーネルは Linus Torvalds によってメンテナンスされ、
+https://kernel.org の pub/linux/kernel/v4.x/ ディレクトリに存在します。
+この開発プロセスは以下のとおり -
+
+ - 新しいカーネルがリリースされた直後に、2週間の特別期間が設けられ、
+ この期間中に、メンテナ達は Linus に大きな差分を送ることができます。
+ このような差分は通常 -next カーネルに数週間含まれてきたパッチです。
+ 大きな変更は git(カーネルのソース管理ツール、詳細は
+ http://git-scm.com/ 参照) を使って送るのが好ましいやり方ですが、パッ
+ チファイルの形式のまま送るのでも十分です。
+ - 2週間後、-rc1 カーネルがリリースされ、この後にはカーネル全体の安定
+ 性に影響をあたえるような新機能は含まない類のパッチしか取り込むこと
+ はできません。新しいドライバ(もしくはファイルシステム)のパッチは
+ -rc1 の後で受け付けられることもあることを覚えておいてください。な
+ ぜなら、変更が独立していて、追加されたコードの外の領域に影響を与え
+ ない限り、退行のリスクは無いからです。-rc1 がリリースされた後、
+ Linus へパッチを送付するのに git を使うこともできますが、パッチは
+ レビューのために、パブリックなメーリングリストへも同時に送る必要が
+ あります。
+ - 新しい -rc は Linus が、最新の git ツリーがテスト目的であれば十分
+ に安定した状態にあると判断したときにリリースされます。目標は毎週新
+ しい -rc カーネルをリリースすることです。
+ - このプロセスはカーネルが 「準備ができた」と考えられるまで継続しま
+ す。このプロセスはだいたい 6週間継続します。
+
+Andrew Morton が Linux-kernel メーリングリストにカーネルリリースについ
+て書いたことをここで言っておくことは価値があります -
+
+ *「カーネルがいつリリースされるかは誰も知りません。なぜなら、
+ これは現実に認識されたバグの状況によりリリースされるのであり、
+ 前もって決められた計画によってリリースされるものではないから
+ です。」*
+
+4.x.y -stable カーネルツリー
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+バージョン番号が3つの数字に分かれているカーネルは -stable カーネルです。
+これには、4.x カーネルで見つかったセキュリティ問題や重大な後戻りに対す
+る比較的小さい重要な修正が含まれます。
+
+これは、開発/実験的バージョンのテストに協力することに興味が無く、最新
+の安定したカーネルを使いたいユーザに推奨するブランチです。
+
+もし、4.x.y カーネルが存在しない場合には、番号が一番大きい 4.x が最新
+の安定版カーネルです。
+
+4.x.y は "stable" チーム <stable@vger.kernel.org> でメンテされており、
+必要に応じてリリースされます。通常のリリース期間は 2週間毎ですが、差
+し迫った問題がなければもう少し長くなることもあります。セキュリティ関
+連の問題の場合はこれに対してだいたいの場合、すぐにリリースがされます。
+
+カーネルツリーに入っている、
+Documentation/process/stable-kernel-rules.rst ファイルにはどのような種
+類の変更が -stable ツリーに受け入れ可能か、またリリースプロセスがどう
+動くかが記述されています。
+
+4.x -git パッチ
+~~~~~~~~~~~~~~~
+
+git リポジトリで管理されているLinus のカーネルツリーの毎日のスナップ
+ショットがあります。(だから -git という名前がついています)。これらのパッ
+チはおおむね毎日リリースされており、Linus のツリーの現状を表します。こ
+れは -rc カーネルと比べて、パッチが大丈夫かどうかも確認しないで自動的
+に生成されるので、より実験的です。
+
+サブシステム毎のカーネルツリーとパッチ
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+それぞれのカーネルサブシステムのメンテナ達は --- そして多くのカーネル
+サブシステムの開発者達も --- 各自の最新の開発状況をソースリポジトリに
+公開しています。そのため、自分とは異なる領域のカーネルで何が起きている
+かを他の人が見られるようになっています。開発が早く進んでいる領域では、
+開発者は自身の投稿がどのサブシステムカーネルツリーを元にしているか質問
+されるので、その投稿とすでに進行中の他の作業との衝突が避けられます。
+
+大部分のこれらのリポジトリは git ツリーです。しかしその他の SCM や
+quilt シリーズとして公開されているパッチキューも使われています。これら
+のサブシステムリポジトリのアドレスは MAINTAINERS ファイルにリストされ
+ています。これらの多くは https://git.kernel.org/ で参照することができま
+す。
+
+提案されたパッチがこのようなサブシステムツリーにコミットされる前に、メー
+リングリストで事前にレビューにかけられます（以下の対応するセクションを
+参照）。いくつかのカーネルサブシステムでは、このレビューは patchworkと
+いうツールによって追跡されます。Patchwork は web インターフェイスによっ
+てパッチ投稿の表示、パッチへのコメント付けや改訂などができ、そしてメン
+テナはパッチに対して、レビュー中、受付済み、拒否というようなマークをつ
+けることができます。大部分のこれらの patchwork のサイトは
+https://patchwork.kernel.org/ でリストされています。
+
+統合テストのための 4.x -next カーネルツリー
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+サブシステムツリーの更新内容がメインラインの 4.x ツリーにマージされる
+前に、それらは統合テストされる必要があります。この目的のため、実質的に
+全サブシステムツリーからほぼ毎日プルされてできる特別なテスト用のリポジ
+トリが存在します-
+
+ https://git.kernel.org/?p=linux/kernel/git/next/linux-next.git
+
+このやり方によって、-next カーネルは次のマージ機会でどんなものがメイン
+ラインカーネルにマージされるか、おおまかなの展望を提供します。-next カー
+ネルの実行テストを行う冒険好きなテスターは大いに歓迎されます。
+
+バグレポート
+-------------
+
+https://bugzilla.kernel.org は Linux カーネル開発者がカーネルのバグを追跡する
+場所です。ユーザは見つけたバグの全てをこのツールで報告すべきです。どう
+kernel bugzilla を使うかの詳細は、以下を参照してください -
+
+ https://bugzilla.kernel.org/page.cgi?id=faq.html
+
+メインカーネルソースディレクトリにあるファイル
+admin-guide/reporting-bugs.rstはカーネルバグらしいものについてどうレポー
+トするかの良いテンプレートであり、問題の追跡を助けるためにカーネル開発
+者にとってどんな情報が必要なのかの詳細が書かれています。
+
+バグレポートの管理
+-------------------
+
+あなたのハッキングのスキルを訓練する最高の方法のひとつに、他人がレポー
+トしたバグを修正することがあります。あなたがカーネルをより安定化させる
+こに寄与するということだけでなく、あなたは 現実の問題を修正することを
+学び、自分のスキルも強化でき、また他の開発者があなたの存在に気がつきま
+す。バグを修正することは、多くの開発者の中から自分が功績をあげる最善の
+道です、なぜなら多くの人は他人のバグの修正に時間を浪費することを好まな
+いからです。
+
+すでにレポートされたバグのために仕事をするためには、
+https://bugzilla.kernel.org に行ってください。もし今後のバグレポートに
+ついてアドバイスを受けたいのであれば、bugme-new メーリングリスト(新し
+いバグレポートだけがここにメールされる) または bugme-janitor メーリン
+グリスト(bugzilla の変更毎にここにメールされる)を購読できます。
+
+ https://lists.linux-foundation.org/mailman/listinfo/bugme-new
+
+ https://lists.linux-foundation.org/mailman/listinfo/bugme-janitors
+
+メーリングリスト
+----------------
+
+上のいくつかのドキュメントで述べていますが、コアカーネル開発者の大部分
+は Linux kernel メーリングリストに参加しています。このリストの登録/脱
+退の方法については以下を参照してください-
+
+ http://vger.kernel.org/vger-lists.html#linux-kernel
+
+このメーリングリストのアーカイブは web 上の多数の場所に存在します。こ
+れらのアーカイブを探すにはサーチエンジンを使いましょう。例えば-
+
+ http://dir.gmane.org/gmane.linux.kernel
+
+リストに投稿する前にすでにその話題がアーカイブに存在するかどうかを検索
+することを是非やってください。多数の事がすでに詳細に渡って議論されてお
+り、アーカイブにのみ記録されています。
+
+大部分のカーネルサブシステムも自分の個別の開発を実施するメーリングリス
+トを持っています。個々のグループがどんなリストを持っているかは、
+MAINTAINERS ファイルにリストがありますので参照してください。
+
+多くのリストは kernel.org でホストされています。これらの情報は以下にあ
+ります -
+
+ http://vger.kernel.org/vger-lists.html
+
+メーリングリストを使う場合、良い行動習慣に従うようにしましょう。少し安っ
+ぽいが、以下の URL は上のリスト(や他のリスト)で会話する場合のシンプル
+なガイドラインを示しています -
+
+ http://www.albion.com/netiquette/
+
+もし複数の人があなたのメールに返事をした場合、CC: で受ける人のリストは
+だいぶ多くなるでしょう。正当な理由がない限り、CC: リストから誰かを削除
+をしないように、また、メーリングリストのアドレスだけにリプライすること
+のないようにしましょう。1つは送信者から、もう1つはリストからのように、
+メールを2回受けることになってもそれに慣れ、しゃれたメールヘッダーを追
+加してこの状態を変えようとしないように。人々はそのようなことは好みませ
+ん。
+
+今までのメールでのやりとりとその間のあなたの発言はそのまま残し、
+"John Kernelhacker wrote ...:" の行をあなたのリプライの先頭行にして、
+メールの先頭でなく、各引用行の間にあなたの言いたいことを追加するべきで
+す。
+
+もしパッチをメールに付ける場合は、
+Documentation/process/submitting-patches.rst に提示されているように、そ
+れは プレーンな可読テキストにすることを忘れないようにしましょう。カー
+ネル開発者は 添付や圧縮したパッチを扱いたがりません。彼らはあなたのパッ
+チの行毎にコメントを入れたいので、そうするしかありません。あなたのメー
+ルプログラムが空白やタブを圧縮しないように確認しましょう。最初の良いテ
+ストとしては、自分にメールを送ってみて、そのパッチを自分で当ててみるこ
+とです。もしそれがうまく行かないなら、あなたのメールプログラムを直して
+もらうか、正しく動くように変えるべきです。
+
+何をおいても、他の購読者に対する敬意を表すことを忘れないでください。
+
+コミュニティと共に働くこと
+--------------------------
+
+カーネルコミュニティのゴールは可能なかぎり最高のカーネルを提供すること
+です。あなたがパッチを受け入れてもらうために投稿した場合、それは、技術
+的メリットだけがレビューされます。その際、あなたは何を予想すべきでしょ
+うか?
+
+ - 批判
+ - コメント
+ - 変更の要求
+ - パッチの正当性の証明要求
+ - 沈黙
+
+思い出してください、これはあなたのパッチをカーネルに入れる話です。あな
+たは、あなたのパッチに対する批判とコメントを受け入れるべきで、それらを
+技術的レベルで評価して、パッチを再作成するか、なぜそれらの変更をすべき
+でないかを明確で簡潔な理由の説明を提供してください。もし、あなたのパッ
+チに何も反応がない場合、たまにはメールの山に埋もれて見逃され、あなたの
+投稿が忘れられてしまうこともあるので、数日待って再度投稿してください。
+
+あなたがやるべきでないことは?
+
+ - 質問なしにあなたのパッチが受け入れられると想像すること
+ - 守りに入ること
+ - コメントを無視すること
+ - 要求された変更を何もしないでパッチを出し直すこと
+
+可能な限り最高の技術的解決を求めているコミュニティでは、パッチがどのく
+らい有益なのかについては常に異なる意見があります。あなたは協調的である
+べきですし、また、あなたのアイディアをカーネルに対してうまく合わせるよ
+うにすることが望まれています。もしくは、最低限あなたのアイディアがそれ
+だけの価値があるとすすんで証明するようにしなければなりません。
+正しい解決に向かって進もうという意志がある限り、間違うことがあっても許
+容されることを忘れないでください。
+
+あなたの最初のパッチに単に 1ダースもの修正を求めるリストの返答になるこ
+とも普通のことです。これはあなたのパッチが受け入れられないということで
+は **ありません**、そしてあなた自身に反対することを意味するのでも **あ
+りません**。単に自分のパッチに対して指摘された問題を全て修正して再送す
+れば良いのです。
+
+
+カーネルコミュニティと企業組織のちがい
+-----------------------------------------------------------------
+
+カーネルコミュニティは大部分の伝統的な会社の開発環境とは異ったやり方で
+動いています。以下は問題を避けるためにできると良いことのリストです。
+
+ あなたの提案する変更について言うときのうまい言い方 -
+
+ - "これは複数の問題を解決します"
+ - "これは2000行のコードを削除します"
+ - "以下のパッチは、私が言おうとしていることを説明するものです"
+ - "私はこれを5つの異なるアーキテクチャでテストしたのですが..."
+ - "以下は一連の小さなパッチ群ですが..."
+ - "これは典型的なマシンでの性能を向上させます..."
+
+ やめた方が良い悪い言い方 -
+
+ - "このやり方で AIX/ptx/Solaris ではできたので、できるはずだ..."
+ - "私はこれを20年もの間やってきた、だから..."
+ - "これは私の会社が金儲けをするために必要だ"
+ - "これは我々のエンタープライズ向け商品ラインのためである"
+ - "これは私が自分のアイディアを記述した、1000ページの設計資料である"
+ - "私はこれについて、6ケ月作業している..."
+ - "以下は ... に関する5000行のパッチです"
+ - "私は現在のぐちゃぐちゃを全部書き直した、それが以下です..."
+ - "私は〆切がある、そのためこのパッチは今すぐ適用される必要がある"
+
+カーネルコミュニティが大部分の伝統的なソフトウェアエンジニアリングの労
+働環境と異なるもう一つの点は、やりとりに顔を合わせないということです。
+email と irc を第一のコミュニケーションの形とする一つの利点は、性別や
+民族の差別がないことです。Linux カーネルの職場環境は女性や少数民族を受
+容します。なぜなら、email アドレスによってのみあなたが認識されるからで
+す。
+国際的な側面からも活動領域を均等にするようにします。なぜならば、あなた
+は人の名前で性別を想像できないからです。ある男性が アンドレアという名
+前で、女性の名前は パット かもしれません (訳注 Andrea は米国では女性、
+それ以外(欧州など)では男性名として使われることが多い。同様に、Pat は
+Patricia (主に女性名)や Patrick (主に男性名)の略称)。
+Linux カーネルの活動をして、意見を表明したことがある大部分の女性は、前
+向きな経験をもっています。
+
+言葉の壁は英語が得意でない一部の人には問題になります。メーリングリスト
+の中で、きちんとアイディアを交換するには、相当うまく英語を操れる必要が
+あることもあります。そのため、自分のメールを送る前に英語で意味が通じて
+いるかをチェックすることをお薦めします。
+
+変更を分割する
+--------------
+
+Linux カーネルコミュニティは、一度に大量のコードの塊を喜んで受容するこ
+とはありません。変更は正確に説明される必要があり、議論され、小さい、個
+別の部分に分割する必要があります。これはこれまで多くの会社がやり慣れて
+きたことと全く正反対のことです。あなたのプロポーザルは、開発プロセスのと
+ても早い段階から紹介されるべきです。そうすれば あなたは自分のやってい
+ることにフィードバックを得られます。これは、コミュニティからみれば、あ
+なたが彼らと一緒にやっているように感じられ、単にあなたの提案する機能の
+ゴミ捨て場として使っているのではない、と感じられるでしょう。
+しかし、一度に 50 もの email をメーリングリストに送りつけるようなことは
+やってはいけません、あなたのパッチ群はいつもどんな時でもそれよりは小さ
+くなければなりません。
+
+パッチを分割する理由は以下 -
+
+1) 小さいパッチはあなたのパッチが適用される見込みを大きくします、カー
+ ネルの人達はパッチが正しいかどうかを確認する時間や労力をかけないか
+ らです。5行のパッチはメンテナがたった1秒見るだけで適用できます。
+ しかし、500行のパッチは、正しいことをレビューするのに数時間かかるか
+ もしれません(時間はパッチのサイズなどにより指数関数に比例してかかり
+ ます)
+
+ 小さいパッチは何かあったときにデバッグもとても簡単になります。パッ
+ チを1個1個取り除くのは、とても大きなパッチを当てた後に(かつ、何かお
+ かしくなった後で)解剖するのに比べればとても簡単です。
+
+2) 小さいパッチを送るだけでなく、送るまえに、書き直して、シンプルにす
+ る(もしくは、単に順番を変えるだけでも)ことも、とても重要です。
+
+以下はカーネル開発者の Al Viro のたとえ話です -
+
+ *"生徒の数学の宿題を採点する先生のことを考えてみてください、
+ 先生は生徒が解に到達するまでの試行錯誤を見たいとは思わないでし
+ ょう。先生は簡潔な最高の解を見たいのです。良い生徒はこれを知っ
+ ており、そして最終解の前の中間作業を提出することは決してないの
+ です*
+
+ *カーネル開発でもこれは同じです。メンテナ達とレビューア達は、
+ 問題を解決する解の背後になる思考プロセスを見たいとは思いません。
+ 彼らは単純であざやかな解決方法を見たいのです。"*
+
+あざやかな解を説明するのと、コミュニティと共に仕事をし、未解決の仕事を
+議論することのバランスをキープするのは難しいかもしれません。ですから、
+開発プロセスの早期段階で改善のためのフィードバックをもらうようにするの
+も良いですが、変更点を小さい部分に分割して全体ではまだ完成していない仕
+事を(部分的に)取り込んでもらえるようにすることも良いことです。
+
+また、でき上がっていないものや、"将来直す" ようなパッチを、本流に含め
+てもらうように送っても、それは受け付けられないことを理解してください。
+
+あなたの変更を正当化する
+------------------------
+
+あなたのパッチを分割するのと同時に、なぜその変更を追加しなければならな
+いかを Linux コミュニティに知らせることはとても重要です。新機能は必要
+性と有用性で正当化されなければなりません。
+
+あなたの変更を説明する
+----------------------
+
+あなたのパッチを送付する場合には、メールの中のテキストで何を言うかにつ
+いて、特別に注意を払ってください。この情報はパッチの ChangeLog に使わ
+れ、いつも皆がみられるように保管されます。これは次のような項目を含め、
+パッチを完全に記述するべきです -
+
+ - なぜ変更が必要か
+ - パッチ全体の設計アプローチ
+ - 実装の詳細
+ - テスト結果
+
+これについて全てがどのようにあるべきかについての詳細は、以下のドキュメ
+ントの ChangeLog セクションを見てください -
+
+ "The Perfect Patch"
+ http://www.ozlabs.org/~akpm/stuff/tpp.txt
+
+これらはどれも、実行することが時にはとても困難です。これらの例を完璧に
+実施するには数年かかるかもしれません。これは継続的な改善のプロセスであ
+り、多くの忍耐と決意を必要とするものです。でも諦めないで、実現は可能で
+す。多数の人がすでにできていますし、彼らも最初はあなたと同じところから
+スタートしたのですから。
+
+
+
+
+----------
+
+Paolo Ciarrocchi に感謝、彼は彼の書いた "Development Process"
+(https://lwn.net/Articles/94386/) セクションをこのテキストの原型にする
+ことを許可してくれました。Rundy Dunlap と Gerrit Huizenga はメーリング
+リストでやるべきこととやってはいけないことのリストを提供してくれました。
+以下の人々のレビュー、コメント、貢献に感謝。
+Pat Mochel, Hanna Linder, Randy Dunlap, Kay Sievers,
+Vojtech Pavlik, Jan Kara, Josh Boyer, Kees Cook, Andrew Morton, Andi
+Kleen, Vadim Lobanov, Jesper Juhl, Adrian Bunk, Keri Harris, Frans Pop,
+David A. Wheeler, Junio Hamano, Michael Kerrisk, と Alex Shepard
+彼らの支援なしでは、このドキュメントはできなかったでしょう。
+
+
+
+Maintainer: Greg Kroah-Hartman <greg@kroah.com>
diff --git a/Documentation/translations/ja_JP/index.rst b/Documentation/translations/ja_JP/index.rst
new file mode 100644
index 0000000..2f91b89
--- /dev/null
+++ b/Documentation/translations/ja_JP/index.rst
@@ -0,0 +1,12 @@
+.. raw:: latex
+
+ \renewcommand\thesection*
+ \renewcommand\thesubsection*
+
+Japanese translations
+=====================
+
+.. toctree::
+ :maxdepth: 1
+
+ howto
diff --git a/Documentation/translations/zh_CN/sparse.txt b/Documentation/translations/zh_CN/sparse.txt
index e41dc94..2f72896 100644
--- a/Documentation/translations/zh_CN/sparse.txt
+++ b/Documentation/translations/zh_CN/sparse.txt
@@ -1,4 +1,4 @@
-Chinese translated version of Documentation/sparse.txt
+Chinese translated version of Documentation/dev-tools/sparse.rst
If you have any comment or update to the content, please contact the
original document maintainer directly. However, if you have a problem
@@ -8,7 +8,7 @@
Chinese maintainer: Li Yang <leo@zh-kernel.org>
---------------------------------------------------------------------
-Documentation/sparse.txt 的中文翻译
+Documentation/dev-tools/sparse.rst 的中文翻译
如果想评论或更新本文的内容，请直接联系原文档的维护者。如果你使用英文
交流有困难的话，也可以向中文版维护者求助。如果本翻译更新不及时或者翻
diff --git a/Documentation/usb/URB.txt b/Documentation/usb/URB.txt
deleted file mode 100644
index 50da0d4..0000000
--- a/Documentation/usb/URB.txt
+++ /dev/null
@@ -1,261 +0,0 @@
-Revised: 2000-Dec-05.
-Again: 2002-Jul-06
-Again: 2005-Sep-19
-
- NOTE:
-
- The USB subsystem now has a substantial section in "The Linux Kernel API"
- guide (in Documentation/DocBook), generated from the current source
- code. This particular documentation file isn't particularly current or
- complete; don't rely on it except for a quick overview.
-
-
-1.1. Basic concept or 'What is an URB?'
-
-The basic idea of the new driver is message passing, the message itself is
-called USB Request Block, or URB for short.
-
-- An URB consists of all relevant information to execute any USB transaction
- and deliver the data and status back.
-
-- Execution of an URB is inherently an asynchronous operation, i.e. the
- usb_submit_urb(urb) call returns immediately after it has successfully
- queued the requested action.
-
-- Transfers for one URB can be canceled with usb_unlink_urb(urb) at any time.
-
-- Each URB has a completion handler, which is called after the action
- has been successfully completed or canceled. The URB also contains a
- context-pointer for passing information to the completion handler.
-
-- Each endpoint for a device logically supports a queue of requests.
- You can fill that queue, so that the USB hardware can still transfer
- data to an endpoint while your driver handles completion of another.
- This maximizes use of USB bandwidth, and supports seamless streaming
- of data to (or from) devices when using periodic transfer modes.
-
-
-1.2. The URB structure
-
-Some of the fields in an URB are:
-
-struct urb
-{
-// (IN) device and pipe specify the endpoint queue
- struct usb_device *dev; // pointer to associated USB device
- unsigned int pipe; // endpoint information
-
- unsigned int transfer_flags; // ISO_ASAP, SHORT_NOT_OK, etc.
-
-// (IN) all urbs need completion routines
- void *context; // context for completion routine
- void (*complete)(struct urb *); // pointer to completion routine
-
-// (OUT) status after each completion
- int status; // returned status
-
-// (IN) buffer used for data transfers
- void *transfer_buffer; // associated data buffer
- int transfer_buffer_length; // data buffer length
- int number_of_packets; // size of iso_frame_desc
-
-// (OUT) sometimes only part of CTRL/BULK/INTR transfer_buffer is used
- int actual_length; // actual data buffer length
-
-// (IN) setup stage for CTRL (pass a struct usb_ctrlrequest)
- unsigned char* setup_packet; // setup packet (control only)
-
-// Only for PERIODIC transfers (ISO, INTERRUPT)
- // (IN/OUT) start_frame is set unless ISO_ASAP isn't set
- int start_frame; // start frame
- int interval; // polling interval
-
- // ISO only: packets are only "best effort"; each can have errors
- int error_count; // number of errors
- struct usb_iso_packet_descriptor iso_frame_desc[0];
-};
-
-Your driver must create the "pipe" value using values from the appropriate
-endpoint descriptor in an interface that it's claimed.
-
-
-1.3. How to get an URB?
-
-URBs are allocated with the following call
-
- struct urb *usb_alloc_urb(int isoframes, int mem_flags)
-
-Return value is a pointer to the allocated URB, 0 if allocation failed.
-The parameter isoframes specifies the number of isochronous transfer frames
-you want to schedule. For CTRL/BULK/INT, use 0. The mem_flags parameter
-holds standard memory allocation flags, letting you control (among other
-things) whether the underlying code may block or not.
-
-To free an URB, use
-
- void usb_free_urb(struct urb *urb)
-
-You may free an urb that you've submitted, but which hasn't yet been
-returned to you in a completion callback. It will automatically be
-deallocated when it is no longer in use.
-
-
-1.4. What has to be filled in?
-
-Depending on the type of transaction, there are some inline functions
-defined in <linux/usb.h> to simplify the initialization, such as
-fill_control_urb() and fill_bulk_urb(). In general, they need the usb
-device pointer, the pipe (usual format from usb.h), the transfer buffer,
-the desired transfer length, the completion handler, and its context.
-Take a look at the some existing drivers to see how they're used.
-
-Flags:
-For ISO there are two startup behaviors: Specified start_frame or ASAP.
-For ASAP set URB_ISO_ASAP in transfer_flags.
-
-If short packets should NOT be tolerated, set URB_SHORT_NOT_OK in
-transfer_flags.
-
-
-1.5. How to submit an URB?
-
-Just call
-
- int usb_submit_urb(struct urb *urb, int mem_flags)
-
-The mem_flags parameter, such as SLAB_ATOMIC, controls memory allocation,
-such as whether the lower levels may block when memory is tight.
-
-It immediately returns, either with status 0 (request queued) or some
-error code, usually caused by the following:
-
-- Out of memory (-ENOMEM)
-- Unplugged device (-ENODEV)
-- Stalled endpoint (-EPIPE)
-- Too many queued ISO transfers (-EAGAIN)
-- Too many requested ISO frames (-EFBIG)
-- Invalid INT interval (-EINVAL)
-- More than one packet for INT (-EINVAL)
-
-After submission, urb->status is -EINPROGRESS; however, you should never
-look at that value except in your completion callback.
-
-For isochronous endpoints, your completion handlers should (re)submit
-URBs to the same endpoint with the ISO_ASAP flag, using multi-buffering,
-to get seamless ISO streaming.
-
-
-1.6. How to cancel an already running URB?
-
-There are two ways to cancel an URB you've submitted but which hasn't
-been returned to your driver yet. For an asynchronous cancel, call
-
- int usb_unlink_urb(struct urb *urb)
-
-It removes the urb from the internal list and frees all allocated
-HW descriptors. The status is changed to reflect unlinking. Note
-that the URB will not normally have finished when usb_unlink_urb()
-returns; you must still wait for the completion handler to be called.
-
-To cancel an URB synchronously, call
-
- void usb_kill_urb(struct urb *urb)
-
-It does everything usb_unlink_urb does, and in addition it waits
-until after the URB has been returned and the completion handler
-has finished. It also marks the URB as temporarily unusable, so
-that if the completion handler or anyone else tries to resubmit it
-they will get a -EPERM error. Thus you can be sure that when
-usb_kill_urb() returns, the URB is totally idle.
-
-There is a lifetime issue to consider. An URB may complete at any
-time, and the completion handler may free the URB. If this happens
-while usb_unlink_urb or usb_kill_urb is running, it will cause a
-memory-access violation. The driver is responsible for avoiding this,
-which often means some sort of lock will be needed to prevent the URB
-from being deallocated while it is still in use.
-
-On the other hand, since usb_unlink_urb may end up calling the
-completion handler, the handler must not take any lock that is held
-when usb_unlink_urb is invoked. The general solution to this problem
-is to increment the URB's reference count while holding the lock, then
-drop the lock and call usb_unlink_urb or usb_kill_urb, and then
-decrement the URB's reference count. You increment the reference
-count by calling
-
- struct urb *usb_get_urb(struct urb *urb)
-
-(ignore the return value; it is the same as the argument) and
-decrement the reference count by calling usb_free_urb. Of course,
-none of this is necessary if there's no danger of the URB being freed
-by the completion handler.
-
-
-1.7. What about the completion handler?
-
-The handler is of the following type:
-
- typedef void (*usb_complete_t)(struct urb *)
-
-I.e., it gets the URB that caused the completion call. In the completion
-handler, you should have a look at urb->status to detect any USB errors.
-Since the context parameter is included in the URB, you can pass
-information to the completion handler.
-
-Note that even when an error (or unlink) is reported, data may have been
-transferred. That's because USB transfers are packetized; it might take
-sixteen packets to transfer your 1KByte buffer, and ten of them might
-have transferred successfully before the completion was called.
-
-
-NOTE: ***** WARNING *****
-NEVER SLEEP IN A COMPLETION HANDLER. These are often called in atomic
-context.
-
-In the current kernel, completion handlers run with local interrupts
-disabled, but in the future this will be changed, so don't assume that
-local IRQs are always disabled inside completion handlers.
-
-1.8. How to do isochronous (ISO) transfers?
-
-For ISO transfers you have to fill a usb_iso_packet_descriptor structure,
-allocated at the end of the URB by usb_alloc_urb(n,mem_flags), for each
-packet you want to schedule. You also have to set urb->interval to say
-how often to make transfers; it's often one per frame (which is once
-every microframe for highspeed devices). The actual interval used will
-be a power of two that's no bigger than what you specify.
-
-The usb_submit_urb() call modifies urb->interval to the implemented interval
-value that is less than or equal to the requested interval value. If
-ISO_ASAP scheduling is used, urb->start_frame is also updated.
-
-For each entry you have to specify the data offset for this frame (base is
-transfer_buffer), and the length you want to write/expect to read.
-After completion, actual_length contains the actual transferred length and
-status contains the resulting status for the ISO transfer for this frame.
-It is allowed to specify a varying length from frame to frame (e.g. for
-audio synchronisation/adaptive transfer rates). You can also use the length
-0 to omit one or more frames (striping).
-
-For scheduling you can choose your own start frame or ISO_ASAP. As explained
-earlier, if you always keep at least one URB queued and your completion
-keeps (re)submitting a later URB, you'll get smooth ISO streaming (if usb
-bandwidth utilization allows).
-
-If you specify your own start frame, make sure it's several frames in advance
-of the current frame. You might want this model if you're synchronizing
-ISO data with some other event stream.
-
-
-1.9. How to start interrupt (INT) transfers?
-
-Interrupt transfers, like isochronous transfers, are periodic, and happen
-in intervals that are powers of two (1, 2, 4 etc) units. Units are frames
-for full and low speed devices, and microframes for high speed ones.
-The usb_submit_urb() call modifies urb->interval to the implemented interval
-value that is less than or equal to the requested interval value.
-
-In Linux 2.6, unlike earlier versions, interrupt URBs are not automagically
-restarted when they complete. They end when the completion handler is
-called, just like other URBs. If you want an interrupt URB to be restarted,
-your completion handler must resubmit it.
diff --git a/Documentation/usb/acm.txt b/Documentation/usb/acm.txt
index 17f5c2e..903abca 100644
--- a/Documentation/usb/acm.txt
+++ b/Documentation/usb/acm.txt
@@ -64,7 +64,7 @@
2. Verifying that it works
~~~~~~~~~~~~~~~~~~~~~~~~~~
- The first step would be to check /proc/bus/usb/devices, it should look
+ The first step would be to check /sys/kernel/debug/usb/devices, it should look
like this:
T: Bus=01 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#= 1 Spd=12 MxCh= 2
diff --git a/Documentation/usb/error-codes.txt b/Documentation/usb/error-codes.txt
deleted file mode 100644
index 9c3eb84..0000000
--- a/Documentation/usb/error-codes.txt
+++ /dev/null
@@ -1,175 +0,0 @@
-Revised: 2004-Oct-21
-
-This is the documentation of (hopefully) all possible error codes (and
-their interpretation) that can be returned from usbcore.
-
-Some of them are returned by the Host Controller Drivers (HCDs), which
-device drivers only see through usbcore. As a rule, all the HCDs should
-behave the same except for transfer speed dependent behaviors and the
-way certain faults are reported.
-
-
-**************************************************************************
-* Error codes returned by usb_submit_urb *
-**************************************************************************
-
-Non-USB-specific:
-
-0 URB submission went fine
-
--ENOMEM no memory for allocation of internal structures
-
-USB-specific:
-
--EBUSY The URB is already active.
-
--ENODEV specified USB-device or bus doesn't exist
-
--ENOENT specified interface or endpoint does not exist or
- is not enabled
-
--ENXIO host controller driver does not support queuing of this type
- of urb. (treat as a host controller bug.)
-
--EINVAL a) Invalid transfer type specified (or not supported)
- b) Invalid or unsupported periodic transfer interval
- c) ISO: attempted to change transfer interval
- d) ISO: number_of_packets is < 0
- e) various other cases
-
--EXDEV ISO: URB_ISO_ASAP wasn't specified and all the frames
- the URB would be scheduled in have already expired.
-
--EFBIG Host controller driver can't schedule that many ISO frames.
-
--EPIPE The pipe type specified in the URB doesn't match the
- endpoint's actual type.
-
--EMSGSIZE (a) endpoint maxpacket size is zero; it is not usable
- in the current interface altsetting.
- (b) ISO packet is larger than the endpoint maxpacket.
- (c) requested data transfer length is invalid: negative
- or too large for the host controller.
-
--ENOSPC This request would overcommit the usb bandwidth reserved
- for periodic transfers (interrupt, isochronous).
-
--ESHUTDOWN The device or host controller has been disabled due to some
- problem that could not be worked around.
-
--EPERM Submission failed because urb->reject was set.
-
--EHOSTUNREACH URB was rejected because the device is suspended.
-
--ENOEXEC A control URB doesn't contain a Setup packet.
-
-
-**************************************************************************
-* Error codes returned by in urb->status *
-* or in iso_frame_desc[n].status (for ISO) *
-**************************************************************************
-
-USB device drivers may only test urb status values in completion handlers.
-This is because otherwise there would be a race between HCDs updating
-these values on one CPU, and device drivers testing them on another CPU.
-
-A transfer's actual_length may be positive even when an error has been
-reported. That's because transfers often involve several packets, so that
-one or more packets could finish before an error stops further endpoint I/O.
-
-For isochronous URBs, the urb status value is non-zero only if the URB is
-unlinked, the device is removed, the host controller is disabled, or the total
-transferred length is less than the requested length and the URB_SHORT_NOT_OK
-flag is set. Completion handlers for isochronous URBs should only see
-urb->status set to zero, -ENOENT, -ECONNRESET, -ESHUTDOWN, or -EREMOTEIO.
-Individual frame descriptor status fields may report more status codes.
-
-
-0 Transfer completed successfully
-
--ENOENT URB was synchronously unlinked by usb_unlink_urb
-
--EINPROGRESS URB still pending, no results yet
- (That is, if drivers see this it's a bug.)
-
--EPROTO (*, **) a) bitstuff error
- b) no response packet received within the
- prescribed bus turn-around time
- c) unknown USB error
-
--EILSEQ (*, **) a) CRC mismatch
- b) no response packet received within the
- prescribed bus turn-around time
- c) unknown USB error
-
- Note that often the controller hardware does not
- distinguish among cases a), b), and c), so a
- driver cannot tell whether there was a protocol
- error, a failure to respond (often caused by
- device disconnect), or some other fault.
-
--ETIME (**) No response packet received within the prescribed
- bus turn-around time. This error may instead be
- reported as -EPROTO or -EILSEQ.
-
--ETIMEDOUT Synchronous USB message functions use this code
- to indicate timeout expired before the transfer
- completed, and no other error was reported by HC.
-
--EPIPE (**) Endpoint stalled. For non-control endpoints,
- reset this status with usb_clear_halt().
-
--ECOMM During an IN transfer, the host controller
- received data from an endpoint faster than it
- could be written to system memory
-
--ENOSR During an OUT transfer, the host controller
- could not retrieve data from system memory fast
- enough to keep up with the USB data rate
-
--EOVERFLOW (*) The amount of data returned by the endpoint was
- greater than either the max packet size of the
- endpoint or the remaining buffer size. "Babble".
-
--EREMOTEIO The data read from the endpoint did not fill the
- specified buffer, and URB_SHORT_NOT_OK was set in
- urb->transfer_flags.
-
--ENODEV Device was removed. Often preceded by a burst of
- other errors, since the hub driver doesn't detect
- device removal events immediately.
-
--EXDEV ISO transfer only partially completed
- (only set in iso_frame_desc[n].status, not urb->status)
-
--EINVAL ISO madness, if this happens: Log off and go home
-
--ECONNRESET URB was asynchronously unlinked by usb_unlink_urb
-
--ESHUTDOWN The device or host controller has been disabled due
- to some problem that could not be worked around,
- such as a physical disconnect.
-
-
-(*) Error codes like -EPROTO, -EILSEQ and -EOVERFLOW normally indicate
-hardware problems such as bad devices (including firmware) or cables.
-
-(**) This is also one of several codes that different kinds of host
-controller use to indicate a transfer has failed because of device
-disconnect. In the interval before the hub driver starts disconnect
-processing, devices may receive such fault reports for every request.
-
-
-
-**************************************************************************
-* Error codes returned by usbcore-functions *
-* (expect also other submit and transfer status codes) *
-**************************************************************************
-
-usb_register():
--EINVAL error during registering new driver
-
-usb_get_*/usb_set_*():
-usb_control_msg():
-usb_bulk_msg():
--ETIMEDOUT Timeout expired before the transfer completed.
diff --git a/Documentation/usb/gadget_serial.txt b/Documentation/usb/gadget_serial.txt
index 6b4a88a..d1def31 100644
--- a/Documentation/usb/gadget_serial.txt
+++ b/Documentation/usb/gadget_serial.txt
@@ -189,7 +189,7 @@
to the Linux host with a USB cable, the host system should recognize
the gadget serial device. For example, the command
- cat /proc/bus/usb/devices
+ cat /sys/kernel/debug/usb/devices
should show something like this:
@@ -221,7 +221,7 @@
to the Linux host with a USB cable, the host system should recognize
the gadget serial device. For example, the command
- cat /proc/bus/usb/devices
+ cat /sys/kernel/debug/usb/devices
should show something like this:
diff --git a/Documentation/usb/proc_usb_info.txt b/Documentation/usb/proc_usb_info.txt
deleted file mode 100644
index 98be919..0000000
--- a/Documentation/usb/proc_usb_info.txt
+++ /dev/null
@@ -1,390 +0,0 @@
-/proc/bus/usb filesystem output
-===============================
-(version 2010.09.13)
-
-
-The usbfs filesystem for USB devices is traditionally mounted at
-/proc/bus/usb. It provides the /proc/bus/usb/devices file, as well as
-the /proc/bus/usb/BBB/DDD files.
-
-In many modern systems the usbfs filesystem isn't used at all. Instead
-USB device nodes are created under /dev/usb/ or someplace similar. The
-"devices" file is available in debugfs, typically as
-/sys/kernel/debug/usb/devices.
-
-
-**NOTE**: If /proc/bus/usb appears empty, and a host controller
- driver has been linked, then you need to mount the
- filesystem. Issue the command (as root):
-
- mount -t usbfs none /proc/bus/usb
-
- An alternative and more permanent method would be to add
-
- none /proc/bus/usb usbfs defaults 0 0
-
- to /etc/fstab. This will mount usbfs at each reboot.
- You can then issue `cat /proc/bus/usb/devices` to extract
- USB device information, and user mode drivers can use usbfs
- to interact with USB devices.
-
- There are a number of mount options supported by usbfs.
- Consult the source code (linux/drivers/usb/core/inode.c) for
- information about those options.
-
-**NOTE**: The filesystem has been renamed from "usbdevfs" to
- "usbfs", to reduce confusion with "devfs". You may
- still see references to the older "usbdevfs" name.
-
-For more information on mounting the usbfs file system, see the
-"USB Device Filesystem" section of the USB Guide. The latest copy
-of the USB Guide can be found at http://www.linux-usb.org/
-
-
-THE /proc/bus/usb/BBB/DDD FILES:
---------------------------------
-Each connected USB device has one file. The BBB indicates the bus
-number. The DDD indicates the device address on that bus. Both
-of these numbers are assigned sequentially, and can be reused, so
-you can't rely on them for stable access to devices. For example,
-it's relatively common for devices to re-enumerate while they are
-still connected (perhaps someone jostled their power supply, hub,
-or USB cable), so a device might be 002/027 when you first connect
-it and 002/048 sometime later.
-
-These files can be read as binary data. The binary data consists
-of first the device descriptor, then the descriptors for each
-configuration of the device. Multi-byte fields in the device descriptor
-are converted to host endianness by the kernel. The configuration
-descriptors are in bus endian format! The configuration descriptor
-are wTotalLength bytes apart. If a device returns less configuration
-descriptor data than indicated by wTotalLength there will be a hole in
-the file for the missing bytes. This information is also shown
-in text form by the /proc/bus/usb/devices file, described later.
-
-These files may also be used to write user-level drivers for the USB
-devices. You would open the /proc/bus/usb/BBB/DDD file read/write,
-read its descriptors to make sure it's the device you expect, and then
-bind to an interface (or perhaps several) using an ioctl call. You
-would issue more ioctls to the device to communicate to it using
-control, bulk, or other kinds of USB transfers. The IOCTLs are
-listed in the <linux/usbdevice_fs.h> file, and at this writing the
-source code (linux/drivers/usb/core/devio.c) is the primary reference
-for how to access devices through those files.
-
-Note that since by default these BBB/DDD files are writable only by
-root, only root can write such user mode drivers. You can selectively
-grant read/write permissions to other users by using "chmod". Also,
-usbfs mount options such as "devmode=0666" may be helpful.
-
-
-
-THE /proc/bus/usb/devices FILE:
--------------------------------
-In /proc/bus/usb/devices, each device's output has multiple
-lines of ASCII output.
-I made it ASCII instead of binary on purpose, so that someone
-can obtain some useful data from it without the use of an
-auxiliary program. However, with an auxiliary program, the numbers
-in the first 4 columns of each "T:" line (topology info:
-Lev, Prnt, Port, Cnt) can be used to build a USB topology diagram.
-
-Each line is tagged with a one-character ID for that line:
-
-T = Topology (etc.)
-B = Bandwidth (applies only to USB host controllers, which are
- virtualized as root hubs)
-D = Device descriptor info.
-P = Product ID info. (from Device descriptor, but they won't fit
- together on one line)
-S = String descriptors.
-C = Configuration descriptor info. (* = active configuration)
-I = Interface descriptor info.
-E = Endpoint descriptor info.
-
-=======================================================================
-
-/proc/bus/usb/devices output format:
-
-Legend:
- d = decimal number (may have leading spaces or 0's)
- x = hexadecimal number (may have leading spaces or 0's)
- s = string
-
-
-Topology info:
-
-T: Bus=dd Lev=dd Prnt=dd Port=dd Cnt=dd Dev#=ddd Spd=dddd MxCh=dd
-| | | | | | | | |__MaxChildren
-| | | | | | | |__Device Speed in Mbps
-| | | | | | |__DeviceNumber
-| | | | | |__Count of devices at this level
-| | | | |__Connector/Port on Parent for this device
-| | | |__Parent DeviceNumber
-| | |__Level in topology for this bus
-| |__Bus number
-|__Topology info tag
-
- Speed may be:
- 1.5 Mbit/s for low speed USB
- 12 Mbit/s for full speed USB
- 480 Mbit/s for high speed USB (added for USB 2.0);
- also used for Wireless USB, which has no fixed speed
- 5000 Mbit/s for SuperSpeed USB (added for USB 3.0)
-
- For reasons lost in the mists of time, the Port number is always
- too low by 1. For example, a device plugged into port 4 will
- show up with "Port=03".
-
-Bandwidth info:
-B: Alloc=ddd/ddd us (xx%), #Int=ddd, #Iso=ddd
-| | | |__Number of isochronous requests
-| | |__Number of interrupt requests
-| |__Total Bandwidth allocated to this bus
-|__Bandwidth info tag
-
- Bandwidth allocation is an approximation of how much of one frame
- (millisecond) is in use. It reflects only periodic transfers, which
- are the only transfers that reserve bandwidth. Control and bulk
- transfers use all other bandwidth, including reserved bandwidth that
- is not used for transfers (such as for short packets).
-
- The percentage is how much of the "reserved" bandwidth is scheduled by
- those transfers. For a low or full speed bus (loosely, "USB 1.1"),
- 90% of the bus bandwidth is reserved. For a high speed bus (loosely,
- "USB 2.0") 80% is reserved.
-
-
-Device descriptor info & Product ID info:
-
-D: Ver=x.xx Cls=xx(s) Sub=xx Prot=xx MxPS=dd #Cfgs=dd
-P: Vendor=xxxx ProdID=xxxx Rev=xx.xx
-
-where
-D: Ver=x.xx Cls=xx(sssss) Sub=xx Prot=xx MxPS=dd #Cfgs=dd
-| | | | | | |__NumberConfigurations
-| | | | | |__MaxPacketSize of Default Endpoint
-| | | | |__DeviceProtocol
-| | | |__DeviceSubClass
-| | |__DeviceClass
-| |__Device USB version
-|__Device info tag #1
-
-where
-P: Vendor=xxxx ProdID=xxxx Rev=xx.xx
-| | | |__Product revision number
-| | |__Product ID code
-| |__Vendor ID code
-|__Device info tag #2
-
-
-String descriptor info:
-
-S: Manufacturer=ssss
-| |__Manufacturer of this device as read from the device.
-| For USB host controller drivers (virtual root hubs) this may
-| be omitted, or (for newer drivers) will identify the kernel
-| version and the driver which provides this hub emulation.
-|__String info tag
-
-S: Product=ssss
-| |__Product description of this device as read from the device.
-| For older USB host controller drivers (virtual root hubs) this
-| indicates the driver; for newer ones, it's a product (and vendor)
-| description that often comes from the kernel's PCI ID database.
-|__String info tag
-
-S: SerialNumber=ssss
-| |__Serial Number of this device as read from the device.
-| For USB host controller drivers (virtual root hubs) this is
-| some unique ID, normally a bus ID (address or slot name) that
-| can't be shared with any other device.
-|__String info tag
-
-
-
-Configuration descriptor info:
-
-C:* #Ifs=dd Cfg#=dd Atr=xx MPwr=dddmA
-| | | | | |__MaxPower in mA
-| | | | |__Attributes
-| | | |__ConfiguratioNumber
-| | |__NumberOfInterfaces
-| |__ "*" indicates the active configuration (others are " ")
-|__Config info tag
-
- USB devices may have multiple configurations, each of which act
- rather differently. For example, a bus-powered configuration
- might be much less capable than one that is self-powered. Only
- one device configuration can be active at a time; most devices
- have only one configuration.
-
- Each configuration consists of one or more interfaces. Each
- interface serves a distinct "function", which is typically bound
- to a different USB device driver. One common example is a USB
- speaker with an audio interface for playback, and a HID interface
- for use with software volume control.
-
-
-Interface descriptor info (can be multiple per Config):
-
-I:* If#=dd Alt=dd #EPs=dd Cls=xx(sssss) Sub=xx Prot=xx Driver=ssss
-| | | | | | | | |__Driver name
-| | | | | | | | or "(none)"
-| | | | | | | |__InterfaceProtocol
-| | | | | | |__InterfaceSubClass
-| | | | | |__InterfaceClass
-| | | | |__NumberOfEndpoints
-| | | |__AlternateSettingNumber
-| | |__InterfaceNumber
-| |__ "*" indicates the active altsetting (others are " ")
-|__Interface info tag
-
- A given interface may have one or more "alternate" settings.
- For example, default settings may not use more than a small
- amount of periodic bandwidth. To use significant fractions
- of bus bandwidth, drivers must select a non-default altsetting.
-
- Only one setting for an interface may be active at a time, and
- only one driver may bind to an interface at a time. Most devices
- have only one alternate setting per interface.
-
-
-Endpoint descriptor info (can be multiple per Interface):
-
-E: Ad=xx(s) Atr=xx(ssss) MxPS=dddd Ivl=dddss
-| | | | |__Interval (max) between transfers
-| | | |__EndpointMaxPacketSize
-| | |__Attributes(EndpointType)
-| |__EndpointAddress(I=In,O=Out)
-|__Endpoint info tag
-
- The interval is nonzero for all periodic (interrupt or isochronous)
- endpoints. For high speed endpoints the transfer interval may be
- measured in microseconds rather than milliseconds.
-
- For high speed periodic endpoints, the "MaxPacketSize" reflects
- the per-microframe data transfer size. For "high bandwidth"
- endpoints, that can reflect two or three packets (for up to
- 3KBytes every 125 usec) per endpoint.
-
- With the Linux-USB stack, periodic bandwidth reservations use the
- transfer intervals and sizes provided by URBs, which can be less
- than those found in endpoint descriptor.
-
-
-=======================================================================
-
-
-If a user or script is interested only in Topology info, for
-example, use something like "grep ^T: /proc/bus/usb/devices"
-for only the Topology lines. A command like
-"grep -i ^[tdp]: /proc/bus/usb/devices" can be used to list
-only the lines that begin with the characters in square brackets,
-where the valid characters are TDPCIE. With a slightly more able
-script, it can display any selected lines (for example, only T, D,
-and P lines) and change their output format. (The "procusb"
-Perl script is the beginning of this idea. It will list only
-selected lines [selected from TBDPSCIE] or "All" lines from
-/proc/bus/usb/devices.)
-
-The Topology lines can be used to generate a graphic/pictorial
-of the USB devices on a system's root hub. (See more below
-on how to do this.)
-
-The Interface lines can be used to determine what driver is
-being used for each device, and which altsetting it activated.
-
-The Configuration lines could be used to list maximum power
-(in milliamps) that a system's USB devices are using.
-For example, "grep ^C: /proc/bus/usb/devices".
-
-
-Here's an example, from a system which has a UHCI root hub,
-an external hub connected to the root hub, and a mouse and
-a serial converter connected to the external hub.
-
-T: Bus=00 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#= 1 Spd=12 MxCh= 2
-B: Alloc= 28/900 us ( 3%), #Int= 2, #Iso= 0
-D: Ver= 1.00 Cls=09(hub ) Sub=00 Prot=00 MxPS= 8 #Cfgs= 1
-P: Vendor=0000 ProdID=0000 Rev= 0.00
-S: Product=USB UHCI Root Hub
-S: SerialNumber=dce0
-C:* #Ifs= 1 Cfg#= 1 Atr=40 MxPwr= 0mA
-I: If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub
-E: Ad=81(I) Atr=03(Int.) MxPS= 8 Ivl=255ms
-
-T: Bus=00 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#= 2 Spd=12 MxCh= 4
-D: Ver= 1.00 Cls=09(hub ) Sub=00 Prot=00 MxPS= 8 #Cfgs= 1
-P: Vendor=0451 ProdID=1446 Rev= 1.00
-C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=100mA
-I: If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub
-E: Ad=81(I) Atr=03(Int.) MxPS= 1 Ivl=255ms
-
-T: Bus=00 Lev=02 Prnt=02 Port=00 Cnt=01 Dev#= 3 Spd=1.5 MxCh= 0
-D: Ver= 1.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS= 8 #Cfgs= 1
-P: Vendor=04b4 ProdID=0001 Rev= 0.00
-C:* #Ifs= 1 Cfg#= 1 Atr=80 MxPwr=100mA
-I: If#= 0 Alt= 0 #EPs= 1 Cls=03(HID ) Sub=01 Prot=02 Driver=mouse
-E: Ad=81(I) Atr=03(Int.) MxPS= 3 Ivl= 10ms
-
-T: Bus=00 Lev=02 Prnt=02 Port=02 Cnt=02 Dev#= 4 Spd=12 MxCh= 0
-D: Ver= 1.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS= 8 #Cfgs= 1
-P: Vendor=0565 ProdID=0001 Rev= 1.08
-S: Manufacturer=Peracom Networks, Inc.
-S: Product=Peracom USB to Serial Converter
-C:* #Ifs= 1 Cfg#= 1 Atr=a0 MxPwr=100mA
-I: If#= 0 Alt= 0 #EPs= 3 Cls=00(>ifc ) Sub=00 Prot=00 Driver=serial
-E: Ad=81(I) Atr=02(Bulk) MxPS= 64 Ivl= 16ms
-E: Ad=01(O) Atr=02(Bulk) MxPS= 16 Ivl= 16ms
-E: Ad=82(I) Atr=03(Int.) MxPS= 8 Ivl= 8ms
-
-
-Selecting only the "T:" and "I:" lines from this (for example, by using
-"procusb ti"), we have:
-
-T: Bus=00 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#= 1 Spd=12 MxCh= 2
-T: Bus=00 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#= 2 Spd=12 MxCh= 4
-I: If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub
-T: Bus=00 Lev=02 Prnt=02 Port=00 Cnt=01 Dev#= 3 Spd=1.5 MxCh= 0
-I: If#= 0 Alt= 0 #EPs= 1 Cls=03(HID ) Sub=01 Prot=02 Driver=mouse
-T: Bus=00 Lev=02 Prnt=02 Port=02 Cnt=02 Dev#= 4 Spd=12 MxCh= 0
-I: If#= 0 Alt= 0 #EPs= 3 Cls=00(>ifc ) Sub=00 Prot=00 Driver=serial
-
-
-Physically this looks like (or could be converted to):
-
- +------------------+
- | PC/root_hub (12)| Dev# = 1
- +------------------+ (nn) is Mbps.
- Level 0 | CN.0 | CN.1 | [CN = connector/port #]
- +------------------+
- /
- /
- +-----------------------+
- Level 1 | Dev#2: 4-port hub (12)|
- +-----------------------+
- |CN.0 |CN.1 |CN.2 |CN.3 |
- +-----------------------+
- \ \____________________
- \_____ \
- \ \
- +--------------------+ +--------------------+
- Level 2 | Dev# 3: mouse (1.5)| | Dev# 4: serial (12)|
- +--------------------+ +--------------------+
-
-
-
-Or, in a more tree-like structure (ports [Connectors] without
-connections could be omitted):
-
-PC: Dev# 1, root hub, 2 ports, 12 Mbps
-|_ CN.0: Dev# 2, hub, 4 ports, 12 Mbps
- |_ CN.0: Dev #3, mouse, 1.5 Mbps
- |_ CN.1:
- |_ CN.2: Dev #4, serial, 12 Mbps
- |_ CN.3:
-|_ CN.1:
-
-
- ### END ###
diff --git a/Documentation/usb/usb3-debug-port.rst b/Documentation/usb/usb3-debug-port.rst
new file mode 100644
index 0000000..feb1a36
--- /dev/null
+++ b/Documentation/usb/usb3-debug-port.rst
@@ -0,0 +1,100 @@
+===============
+USB3 debug port
+===============
+
+:Author: Lu Baolu <baolu.lu@linux.intel.com>
+:Date: March 2017
+
+GENERAL
+=======
+
+This is a HOWTO for using the USB3 debug port on x86 systems.
+
+Before using any kernel debugging functionality based on USB3
+debug port, you need to::
+
+ 1) check whether any USB3 debug port is available in
+ your system;
+ 2) check which port is used for debugging purposes;
+ 3) have a USB 3.0 super-speed A-to-A debugging cable.
+
+INTRODUCTION
+============
+
+The xHCI debug capability (DbC) is an optional but standalone
+functionality provided by the xHCI host controller. The xHCI
+specification describes DbC in the section 7.6.
+
+When DbC is initialized and enabled, it will present a debug
+device through the debug port (normally the first USB3
+super-speed port). The debug device is fully compliant with
+the USB framework and provides the equivalent of a very high
+performance full-duplex serial link between the debug target
+(the system under debugging) and a debug host.
+
+EARLY PRINTK
+============
+
+DbC has been designed to log early printk messages. One use for
+this feature is kernel debugging. For example, when your machine
+crashes very early before the regular console code is initialized.
+Other uses include simpler, lockless logging instead of a full-
+blown printk console driver and klogd.
+
+On the debug target system, you need to customize a debugging
+kernel with CONFIG_EARLY_PRINTK_USB_XDBC enabled. And, add below
+kernel boot parameter::
+
+ "earlyprintk=xdbc"
+
+If there are multiple xHCI controllers in your system, you can
+append a host contoller index to this kernel parameter. This
+index starts from 0.
+
+Current design doesn't support DbC runtime suspend/resume. As
+the result, you'd better disable runtime power management for
+USB subsystem by adding below kernel boot parameter::
+
+ "usbcore.autosuspend=-1"
+
+Before starting the debug target, you should connect the debug
+port to a USB port (root port or port of any external hub) on
+the debug host. The cable used to connect these two ports
+should be a USB 3.0 super-speed A-to-A debugging cable.
+
+During early boot of the debug target, DbC will be detected and
+initialized. After initialization, the debug host should be able
+to enumerate the debug device in debug target. The debug host
+will then bind the debug device with the usb_debug driver module
+and create the /dev/ttyUSB device.
+
+If the debug device enumeration goes smoothly, you should be able
+to see below kernel messages on the debug host::
+
+ # tail -f /var/log/kern.log
+ [ 1815.983374] usb 4-3: new SuperSpeed USB device number 4 using xhci_hcd
+ [ 1815.999595] usb 4-3: LPM exit latency is zeroed, disabling LPM.
+ [ 1815.999899] usb 4-3: New USB device found, idVendor=1d6b, idProduct=0004
+ [ 1815.999902] usb 4-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
+ [ 1815.999903] usb 4-3: Product: Remote GDB
+ [ 1815.999904] usb 4-3: Manufacturer: Linux
+ [ 1815.999905] usb 4-3: SerialNumber: 0001
+ [ 1816.000240] usb_debug 4-3:1.0: xhci_dbc converter detected
+ [ 1816.000360] usb 4-3: xhci_dbc converter now attached to ttyUSB0
+
+You can use any communication program, for example minicom, to
+read and view the messages. Below simple bash scripts can help
+you to check the sanity of the setup.
+
+.. code-block:: sh
+
+ ===== start of bash scripts =============
+ #!/bin/bash
+
+ while true ; do
+ while [ ! -d /sys/class/tty/ttyUSB0 ] ; do
+ :
+ done
+ cat /dev/ttyUSB0
+ done
+ ===== end of bash scripts ===============
diff --git a/Documentation/userspace-api/conf.py b/Documentation/userspace-api/conf.py
new file mode 100644
index 0000000..2eaf59f
--- /dev/null
+++ b/Documentation/userspace-api/conf.py
@@ -0,0 +1,10 @@
+# -*- coding: utf-8; mode: python -*-
+
+project = "The Linux kernel user-space API guide"
+
+tags.add("subproject")
+
+latex_documents = [
+ ('index', 'userspace-api.tex', project,
+ 'The kernel development community', 'manual'),
+]
diff --git a/Documentation/userspace-api/index.rst b/Documentation/userspace-api/index.rst
new file mode 100644
index 0000000..a9d01b4
--- /dev/null
+++ b/Documentation/userspace-api/index.rst
@@ -0,0 +1,26 @@
+=====================================
+The Linux kernel user-space API guide
+=====================================
+
+.. _man-pages: https://www.kernel.org/doc/man-pages/
+
+While much of the kernel's user-space API is documented elsewhere
+(particularly in the man-pages_ project), some user-space information can
+also be found in the kernel tree itself. This manual is intended to be the
+place where this information is gathered.
+
+.. class:: toc-title
+
+ Table of contents
+
+.. toctree::
+ :maxdepth: 2
+
+ unshare
+
+.. only:: subproject and html
+
+ Indices
+ =======
+
+ * :ref:`genindex`
diff --git a/Documentation/unshare.txt b/Documentation/userspace-api/unshare.rst
similarity index 66%
rename from Documentation/unshare.txt
rename to Documentation/userspace-api/unshare.rst
index a864351..737c192 100644
--- a/Documentation/unshare.txt
+++ b/Documentation/userspace-api/unshare.rst
@@ -1,17 +1,17 @@
+unshare system call
+===================
-unshare system call:
---------------------
-This document describes the new system call, unshare. The document
+This document describes the new system call, unshare(). The document
provides an overview of the feature, why it is needed, how it can
be used, its interface specification, design, implementation and
how it can be tested.
-Change Log:
------------
+Change Log
+----------
version 0.1 Initial document, Janak Desai (janak@us.ibm.com), Jan 11, 2006
-Contents:
----------
+Contents
+--------
1) Overview
2) Benefits
3) Cost
@@ -24,6 +24,7 @@
1) Overview
-----------
+
Most legacy operating system kernels support an abstraction of threads
as multiple execution contexts within a process. These kernels provide
special resources and mechanisms to maintain these "threads". The Linux
@@ -38,33 +39,35 @@
call, applications can selectively choose which resources to share
between threads.
-unshare system call adds a primitive to the Linux thread model that
+unshare() system call adds a primitive to the Linux thread model that
allows threads to selectively 'unshare' any resources that were being
-shared at the time of their creation. unshare was conceptualized by
+shared at the time of their creation. unshare() was conceptualized by
Al Viro in the August of 2000, on the Linux-Kernel mailing list, as part
-of the discussion on POSIX threads on Linux. unshare augments the
+of the discussion on POSIX threads on Linux. unshare() augments the
usefulness of Linux threads for applications that would like to control
-shared resources without creating a new process. unshare is a natural
+shared resources without creating a new process. unshare() is a natural
addition to the set of available primitives on Linux that implement
the concept of process/thread as a virtual machine.
2) Benefits
-----------
-unshare would be useful to large application frameworks such as PAM
+
+unshare() would be useful to large application frameworks such as PAM
where creating a new process to control sharing/unsharing of process
resources is not possible. Since namespaces are shared by default
-when creating a new process using fork or clone, unshare can benefit
+when creating a new process using fork or clone, unshare() can benefit
even non-threaded applications if they have a need to disassociate
from default shared namespace. The following lists two use-cases
-where unshare can be used.
+where unshare() can be used.
2.1 Per-security context namespaces
------------------------------------
-unshare can be used to implement polyinstantiated directories using
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+unshare() can be used to implement polyinstantiated directories using
the kernel's per-process namespace mechanism. Polyinstantiated directories,
such as per-user and/or per-security context instance of /tmp, /var/tmp or
per-security context instance of a user's home directory, isolate user
-processes when working with these directories. Using unshare, a PAM
+processes when working with these directories. Using unshare(), a PAM
module can easily setup a private namespace for a user at login.
Polyinstantiated directories are required for Common Criteria certification
with Labeled System Protection Profile, however, with the availability
@@ -74,33 +77,36 @@
appropriate by system administrators.
2.2 unsharing of virtual memory and/or open files
--------------------------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
Consider a client/server application where the server is processing
client requests by creating processes that share resources such as
-virtual memory and open files. Without unshare, the server has to
+virtual memory and open files. Without unshare(), the server has to
decide what needs to be shared at the time of creating the process
-which services the request. unshare allows the server an ability to
+which services the request. unshare() allows the server an ability to
disassociate parts of the context during the servicing of the
request. For large and complex middleware application frameworks, this
-ability to unshare after the process was created can be very
+ability to unshare() after the process was created can be very
useful.
3) Cost
-------
-In order to not duplicate code and to handle the fact that unshare
+
+In order to not duplicate code and to handle the fact that unshare()
works on an active task (as opposed to clone/fork working on a newly
-allocated inactive task) unshare had to make minor reorganizational
+allocated inactive task) unshare() had to make minor reorganizational
changes to copy_* functions utilized by clone/fork system call.
There is a cost associated with altering existing, well tested and
stable code to implement a new feature that may not get exercised
extensively in the beginning. However, with proper design and code
-review of the changes and creation of an unshare test for the LTP
+review of the changes and creation of an unshare() test for the LTP
the benefits of this new feature can exceed its cost.
4) Requirements
---------------
-unshare reverses sharing that was done using clone(2) system call,
-so unshare should have a similar interface as clone(2). That is,
+
+unshare() reverses sharing that was done using clone(2) system call,
+so unshare() should have a similar interface as clone(2). That is,
since flags in clone(int flags, void *stack) specifies what should
be shared, similar flags in unshare(int flags) should specify
what should be unshared. Unfortunately, this may appear to invert
@@ -108,13 +114,14 @@
However, there was no easy solution that was less confusing and that
allowed incremental context unsharing in future without an ABI change.
-unshare interface should accommodate possible future addition of
+unshare() interface should accommodate possible future addition of
new context flags without requiring a rebuild of old applications.
-If and when new context flags are added, unshare design should allow
+If and when new context flags are added, unshare() design should allow
incremental unsharing of those resources on an as needed basis.
5) Functional Specification
---------------------------
+
NAME
unshare - disassociate parts of the process execution context
@@ -124,7 +131,7 @@
int unshare(int flags);
DESCRIPTION
- unshare allows a process to disassociate parts of its execution
+ unshare() allows a process to disassociate parts of its execution
context that are currently being shared with other processes. Part
of execution context, such as the namespace, is shared by default
when a new process is created using fork(2), while other parts,
@@ -132,7 +139,7 @@
shared by explicit request to share them when creating a process
using clone(2).
- The main use of unshare is to allow a process to control its
+ The main use of unshare() is to allow a process to control its
shared execution context without creating a new process.
The flags argument specifies one or bitwise-or'ed of several of
@@ -176,17 +183,20 @@
6) High Level Design
--------------------
-Depending on the flags argument, the unshare system call allocates
+
+Depending on the flags argument, the unshare() system call allocates
appropriate process context structures, populates it with values from
the current shared version, associates newly duplicated structures
with the current task structure and releases corresponding shared
versions. Helper functions of clone (copy_*) could not be used
-directly by unshare because of the following two reasons.
+directly by unshare() because of the following two reasons.
+
1) clone operates on a newly allocated not-yet-active task
- structure, where as unshare operates on the current active
- task. Therefore unshare has to take appropriate task_lock()
+ structure, where as unshare() operates on the current active
+ task. Therefore unshare() has to take appropriate task_lock()
before associating newly duplicated context structures
- 2) unshare has to allocate and duplicate all context structures
+
+ 2) unshare() has to allocate and duplicate all context structures
that are being unshared, before associating them with the
current task and releasing older shared structures. Failure
do so will create race conditions and/or oops when trying
@@ -202,94 +212,121 @@
current context structure was moved into new dup_* functions. Now,
copy_* functions call dup_* functions to allocate and duplicate
appropriate context structures and then associate them with the
-task structure that is being constructed. unshare system call on
+task structure that is being constructed. unshare() system call on
the other hand performs the following:
+
1) Check flags to force missing, but implied, flags
- 2) For each context structure, call the corresponding unshare
+
+ 2) For each context structure, call the corresponding unshare()
helper function to allocate and duplicate a new context
structure, if the appropriate bit is set in the flags argument.
+
3) If there is no error in allocation and duplication and there
are new context structures then lock the current task structure,
associate new context structures with the current task structure,
and release the lock on the current task structure.
+
4) Appropriately release older, shared, context structures.
7) Low Level Design
-------------------
-Implementation of unshare can be grouped in the following 4 different
+
+Implementation of unshare() can be grouped in the following 4 different
items:
+
a) Reorganization of existing copy_* functions
- b) unshare system call service function
- c) unshare helper functions for each different process context
+
+ b) unshare() system call service function
+
+ c) unshare() helper functions for each different process context
+
d) Registration of system call number for different architectures
- 7.1) Reorganization of copy_* functions
- Each copy function such as copy_mm, copy_namespace, copy_files,
- etc, had roughly two components. The first component allocated
- and duplicated the appropriate structure and the second component
- linked it to the task structure passed in as an argument to the copy
- function. The first component was split into its own function.
- These dup_* functions allocated and duplicated the appropriate
- context structure. The reorganized copy_* functions invoked
- their corresponding dup_* functions and then linked the newly
- duplicated structures to the task structure with which the
- copy function was called.
+7.1) Reorganization of copy_* functions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- 7.2) unshare system call service function
+Each copy function such as copy_mm, copy_namespace, copy_files,
+etc, had roughly two components. The first component allocated
+and duplicated the appropriate structure and the second component
+linked it to the task structure passed in as an argument to the copy
+function. The first component was split into its own function.
+These dup_* functions allocated and duplicated the appropriate
+context structure. The reorganized copy_* functions invoked
+their corresponding dup_* functions and then linked the newly
+duplicated structures to the task structure with which the
+copy function was called.
+
+7.2) unshare() system call service function
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
* Check flags
Force implied flags. If CLONE_THREAD is set force CLONE_VM.
If CLONE_VM is set, force CLONE_SIGHAND. If CLONE_SIGHAND is
set and signals are also being shared, force CLONE_THREAD. If
CLONE_NEWNS is set, force CLONE_FS.
+
* For each context flag, invoke the corresponding unshare_*
helper routine with flags passed into the system call and a
reference to pointer pointing the new unshared structure
+
* If any new structures are created by unshare_* helper
functions, take the task_lock() on the current task,
modify appropriate context pointers, and release the
task lock.
+
* For all newly unshared structures, release the corresponding
older, shared, structures.
- 7.3) unshare_* helper functions
- For unshare_* helpers corresponding to CLONE_SYSVSEM, CLONE_SIGHAND,
- and CLONE_THREAD, return -EINVAL since they are not implemented yet.
- For others, check the flag value to see if the unsharing is
- required for that structure. If it is, invoke the corresponding
- dup_* function to allocate and duplicate the structure and return
- a pointer to it.
+7.3) unshare_* helper functions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- 7.4) Appropriately modify architecture specific code to register the
- new system call.
+For unshare_* helpers corresponding to CLONE_SYSVSEM, CLONE_SIGHAND,
+and CLONE_THREAD, return -EINVAL since they are not implemented yet.
+For others, check the flag value to see if the unsharing is
+required for that structure. If it is, invoke the corresponding
+dup_* function to allocate and duplicate the structure and return
+a pointer to it.
+
+7.4) Finally
+~~~~~~~~~~~~
+
+Appropriately modify architecture specific code to register the
+new system call.
8) Test Specification
---------------------
-The test for unshare should test the following:
+
+The test for unshare() should test the following:
+
1) Valid flags: Test to check that clone flags for signal and
- signal handlers, for which unsharing is not implemented
- yet, return -EINVAL.
+ signal handlers, for which unsharing is not implemented
+ yet, return -EINVAL.
+
2) Missing/implied flags: Test to make sure that if unsharing
- namespace without specifying unsharing of filesystem, correctly
- unshares both namespace and filesystem information.
+ namespace without specifying unsharing of filesystem, correctly
+ unshares both namespace and filesystem information.
+
3) For each of the four (namespace, filesystem, files and vm)
- supported unsharing, verify that the system call correctly
- unshares the appropriate structure. Verify that unsharing
- them individually as well as in combination with each
- other works as expected.
+ supported unsharing, verify that the system call correctly
+ unshares the appropriate structure. Verify that unsharing
+ them individually as well as in combination with each
+ other works as expected.
+
4) Concurrent execution: Use shared memory segments and futex on
- an address in the shm segment to synchronize execution of
- about 10 threads. Have a couple of threads execute execve,
- a couple _exit and the rest unshare with different combination
- of flags. Verify that unsharing is performed as expected and
- that there are no oops or hangs.
+ an address in the shm segment to synchronize execution of
+ about 10 threads. Have a couple of threads execute execve,
+ a couple _exit and the rest unshare with different combination
+ of flags. Verify that unsharing is performed as expected and
+ that there are no oops or hangs.
9) Future Work
--------------
-The current implementation of unshare does not allow unsharing of
+
+The current implementation of unshare() does not allow unsharing of
signals and signal handlers. Signals are complex to begin with and
to unshare signals and/or signal handlers of a currently running
process is even more complex. If in the future there is a specific
need to allow unsharing of signals and/or signal handlers, it can
-be incrementally added to unshare without affecting legacy
-applications using unshare.
+be incrementally added to unshare() without affecting legacy
+applications using unshare().
diff --git a/Documentation/vfio-mediated-device.txt b/Documentation/vfio-mediated-device.txt
index d226c7a..e5e57b4 100644
--- a/Documentation/vfio-mediated-device.txt
+++ b/Documentation/vfio-mediated-device.txt
@@ -217,9 +217,9 @@
* [<type-id>]
- The [<type-id>] name is created by adding the the device driver string as a
- prefix to the string provided by the vendor driver. This format of this name
- is as follows:
+ The [<type-id>] name is created by adding the device driver string as a prefix
+ to the string provided by the vendor driver. This format of this name is as
+ follows:
sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);
@@ -380,7 +380,7 @@
/dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10
/dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10
-6. Using a minicom or any terminal enulation program, open port /dev/ttyS1 or
+6. Using minicom or any terminal emulation program, open port /dev/ttyS1 or
/dev/ttyS2 with hardware flow control disabled.
7. Type data on the minicom terminal or send data to the terminal emulation
diff --git a/Documentation/x86/intel_rdt_ui.txt b/Documentation/x86/intel_rdt_ui.txt
index 51cf6fa..0f6d847 100644
--- a/Documentation/x86/intel_rdt_ui.txt
+++ b/Documentation/x86/intel_rdt_ui.txt
@@ -4,6 +4,7 @@
Fenghua Yu <fenghua.yu@intel.com>
Tony Luck <tony.luck@intel.com>
+Vikas Shivappa <vikas.shivappa@intel.com>
This feature is enabled by the CONFIG_INTEL_RDT_A Kconfig and the
X86 /proc/cpuinfo flag bits "rdt", "cat_l3" and "cdp_l3".
@@ -22,19 +23,34 @@
The 'info' directory contains information about the enabled
resources. Each resource has its own subdirectory. The subdirectory
-names reflect the resource names. Each subdirectory contains the
-following files:
+names reflect the resource names.
+Cache resource(L3/L2) subdirectory contains the following files:
-"num_closids": The number of CLOSIDs which are valid for this
- resource. The kernel uses the smallest number of
- CLOSIDs of all enabled resources as limit.
+"num_closids": The number of CLOSIDs which are valid for this
+ resource. The kernel uses the smallest number of
+ CLOSIDs of all enabled resources as limit.
-"cbm_mask": The bitmask which is valid for this resource. This
- mask is equivalent to 100%.
+"cbm_mask": The bitmask which is valid for this resource.
+ This mask is equivalent to 100%.
-"min_cbm_bits": The minimum number of consecutive bits which must be
- set when writing a mask.
+"min_cbm_bits": The minimum number of consecutive bits which
+ must be set when writing a mask.
+Memory bandwitdh(MB) subdirectory contains the following files:
+
+"min_bandwidth": The minimum memory bandwidth percentage which
+ user can request.
+
+"bandwidth_gran": The granularity in which the memory bandwidth
+ percentage is allocated. The allocated
+ b/w percentage is rounded off to the next
+ control step available on the hardware. The
+ available bandwidth control steps are:
+ min_bandwidth + N * bandwidth_gran.
+
+"delay_linear": Indicates if the delay scale is linear or
+ non-linear. This field is purely informational
+ only.
Resource groups
---------------
@@ -59,6 +75,9 @@
given to the default (root) group. You cannot remove CPUs
from the default group.
+"cpus_list": One or more CPU ranges of logical CPUs assigned to this
+ group. Same rules apply like for the "cpus" file.
+
"schemata": A list of all the resources available to this group.
Each resource has its own line and format - see below for
details.
@@ -107,6 +126,22 @@
of the capacity of the cache. You could partition the cache into four
equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000.
+Memory bandwidth(b/w) percentage
+--------------------------------
+For Memory b/w resource, user controls the resource by indicating the
+percentage of total memory b/w.
+
+The minimum bandwidth percentage value for each cpu model is predefined
+and can be looked up through "info/MB/min_bandwidth". The bandwidth
+granularity that is allocated is also dependent on the cpu model and can
+be looked up at "info/MB/bandwidth_gran". The available bandwidth
+control steps are: min_bw + N * bw_gran. Intermediate values are rounded
+to the next control step available on the hardware.
+
+The bandwidth throttling is a core specific mechanism on some of Intel
+SKUs. Using a high bandwidth and a low bandwidth setting on two threads
+sharing a core will result in both threads being throttled to use the
+low bandwidth.
L3 details (code and data prioritization disabled)
--------------------------------------------------
@@ -129,16 +164,38 @@
L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
+Memory b/w Allocation details
+-----------------------------
+
+Memory b/w domain is L3 cache.
+
+ MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;...
+
+Reading/writing the schemata file
+---------------------------------
+Reading the schemata file will show the state of all resources
+on all domains. When writing you only need to specify those values
+which you wish to change. E.g.
+
+# cat schemata
+L3DATA:0=fffff;1=fffff;2=fffff;3=fffff
+L3CODE:0=fffff;1=fffff;2=fffff;3=fffff
+# echo "L3DATA:2=3c0;" > schemata
+# cat schemata
+L3DATA:0=fffff;1=fffff;2=3c0;3=fffff
+L3CODE:0=fffff;1=fffff;2=fffff;3=fffff
+
Example 1
---------
On a two socket machine (one L3 cache per socket) with just four bits
-for cache bit masks
+for cache bit masks, minimum b/w of 10% with a memory bandwidth
+granularity of 10%
# mount -t resctrl resctrl /sys/fs/resctrl
# cd /sys/fs/resctrl
# mkdir p0 p1
-# echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata
-# echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata
+# echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata
+# echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata
The default resource group is unmodified, so we have access to all parts
of all caches (its schemata file reads "L3:0=f;1=f").
@@ -147,6 +204,14 @@
"lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1.
Tasks in group "p1" use the "lower" 50% of cache on both sockets.
+Similarly, tasks that are under the control of group "p0" may use a
+maximum memory b/w of 50% on socket0 and 50% on socket 1.
+Tasks in group "p1" may also use 50% memory b/w on both sockets.
+Note that unlike cache masks, memory b/w cannot specify whether these
+allocations can overlap or not. The allocations specifies the maximum
+b/w that the group may be able to use and the system admin can configure
+the b/w accordingly.
+
Example 2
---------
Again two sockets, but this time with a more realistic 20-bit mask.
@@ -160,9 +225,10 @@
# cd /sys/fs/resctrl
First we reset the schemata for the default group so that the "upper"
-50% of the L3 cache on socket 0 cannot be used by ordinary tasks:
+50% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by
+ordinary tasks:
-# echo "L3:0=3ff;1=fffff" > schemata
+# echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata
Next we make a resource group for our first real time task and give
it access to the "top" 25% of the cache on socket 0.
@@ -185,6 +251,20 @@
# echo 5678 > p1/tasks
# taskset -cp 2 5678
+For the same 2 socket system with memory b/w resource and CAT L3 the
+schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is
+10):
+
+For our first real time task this would request 20% memory b/w on socket
+0.
+
+# echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata
+
+For our second real time task this would request an other 20% memory b/w
+on socket 0.
+
+# echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata
+
Example 3
---------
@@ -198,18 +278,22 @@
# cd /sys/fs/resctrl
First we reset the schemata for the default group so that the "upper"
-50% of the L3 cache on socket 0 cannot be used by ordinary tasks:
+50% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0
+cannot be used by ordinary tasks:
-# echo "L3:0=3ff" > schemata
+# echo "L3:0=3ff\nMB:0=50" > schemata
-Next we make a resource group for our real time cores and give
-it access to the "top" 50% of the cache on socket 0.
+Next we make a resource group for our real time cores and give it access
+to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on
+socket 0.
# mkdir p0
-# echo "L3:0=ffc00;" > p0/schemata
+# echo "L3:0=ffc00\nMB:0=50" > p0/schemata
Finally we move core 4-7 over to the new group and make sure that the
-kernel and the tasks running there get 50% of the cache.
+kernel and the tasks running there get 50% of the cache. They should
+also get 50% of memory bandwidth assuming that the cores 4-7 are SMT
+siblings and only the real time threads are scheduled on the cores 4-7.
# echo C0 > p0/cpus
diff --git a/Documentation/x86/x86_64/mm.txt b/Documentation/x86/x86_64/mm.txt
index 5724092..b0798e2 100644
--- a/Documentation/x86/x86_64/mm.txt
+++ b/Documentation/x86/x86_64/mm.txt
@@ -4,7 +4,7 @@
Virtual memory map with 4 level page tables:
0000000000000000 - 00007fffffffffff (=47 bits) user space, different per mm
-hole caused by [48:63] sign extension
+hole caused by [47:63] sign extension
ffff800000000000 - ffff87ffffffffff (=43 bits) guard hole, reserved for hypervisor
ffff880000000000 - ffffc7ffffffffff (=64 TB) direct mapping of all phys. memory
ffffc80000000000 - ffffc8ffffffffff (=40 bits) hole
@@ -19,16 +19,43 @@
ffffffef00000000 - fffffffeffffffff (=64 GB) EFI region mapping space
... unused hole ...
ffffffff80000000 - ffffffff9fffffff (=512 MB) kernel text mapping, from phys 0
+ffffffffa0000000 - ffffffffff5fffff (=1526 MB) module mapping space (variable)
+ffffffffff600000 - ffffffffffdfffff (=8 MB) vsyscalls
+ffffffffffe00000 - ffffffffffffffff (=2 MB) unused hole
+
+Virtual memory map with 5 level page tables:
+
+0000000000000000 - 00ffffffffffffff (=56 bits) user space, different per mm
+hole caused by [56:63] sign extension
+ff00000000000000 - ff0fffffffffffff (=52 bits) guard hole, reserved for hypervisor
+ff10000000000000 - ff8fffffffffffff (=55 bits) direct mapping of all phys. memory
+ff90000000000000 - ff91ffffffffffff (=49 bits) hole
+ff92000000000000 - ffd1ffffffffffff (=54 bits) vmalloc/ioremap space
+ffd2000000000000 - ffd3ffffffffffff (=49 bits) hole
+ffd4000000000000 - ffd5ffffffffffff (=49 bits) virtual memory map (512TB)
+... unused hole ...
+ffd8000000000000 - fff7ffffffffffff (=53 bits) kasan shadow memory (8PB)
+... unused hole ...
+ffffff0000000000 - ffffff7fffffffff (=39 bits) %esp fixup stacks
+... unused hole ...
+ffffffef00000000 - fffffffeffffffff (=64 GB) EFI region mapping space
+... unused hole ...
+ffffffff80000000 - ffffffff9fffffff (=512 MB) kernel text mapping, from phys 0
ffffffffa0000000 - ffffffffff5fffff (=1526 MB) module mapping space
ffffffffff600000 - ffffffffffdfffff (=8 MB) vsyscalls
ffffffffffe00000 - ffffffffffffffff (=2 MB) unused hole
+Architecture defines a 64-bit virtual address. Implementations can support
+less. Currently supported are 48- and 57-bit virtual addresses. Bits 63
+through to the most-significant implemented bit are set to either all ones
+or all zero. This causes hole between user space and kernel addresses.
+
The direct mapping covers all memory in the system up to the highest
memory address (this means in some cases it can also include PCI memory
holes).
-vmalloc space is lazily synchronized into the different PML4 pages of
-the processes using the page fault handler, with init_level4_pgt as
+vmalloc space is lazily synchronized into the different PML4/PML5 pages of
+the processes using the page fault handler, with init_top_pgt as
reference.
Current X86-64 implementations support up to 46 bits of address space (64 TB),
@@ -39,6 +66,9 @@
The mappings are not part of any other kernel PGD and are only available
during EFI runtime calls.
+The module mapping space size changes based on the CONFIG requirements for the
+following fixmap section.
+
Note that if CONFIG_RANDOMIZE_MEMORY is enabled, the direct mapping of all
physical memory, vmalloc/ioremap space and virtual memory map are randomized.
Their order is preserved but their base will be offset early at boot time.
diff --git a/Documentation/x86/zero-page.txt b/Documentation/x86/zero-page.txt
index b8527c6..97b7adb 100644
--- a/Documentation/x86/zero-page.txt
+++ b/Documentation/x86/zero-page.txt
@@ -27,7 +27,7 @@
1C0/020 ALL efi_info EFI 32 information (struct efi_info)
1E0/004 ALL alk_mem_k Alternative mem check, in KB
1E4/004 ALL scratch Scratch field for the kernel setup code
-1E8/001 ALL e820_entries Number of entries in e820_map (below)
+1E8/001 ALL e820_entries Number of entries in e820_table (below)
1E9/001 ALL eddbuf_entries Number of entries in eddbuf (below)
1EA/001 ALL edd_mbr_sig_buf_entries Number of entries in edd_mbr_sig_buffer
(below)
@@ -35,6 +35,6 @@
1EC/001 ALL secure_boot Secure boot is enabled in the firmware
1EF/001 ALL sentinel Used to detect broken bootloaders
290/040 ALL edd_mbr_sig_buffer EDD MBR signatures
-2D0/A00 ALL e820_map E820 memory map table
- (array of struct e820entry)
+2D0/A00 ALL e820_table E820 memory map table
+ (array of struct e820_entry)
D00/1EC ALL eddbuf EDD data (array of struct edd_info)
diff --git a/Documentation/zorro.txt b/Documentation/zorro.txt
index 90a64d5..d530971 100644
--- a/Documentation/zorro.txt
+++ b/Documentation/zorro.txt
@@ -11,7 +11,7 @@
The Zorro bus is the bus used in the Amiga family of computers. Thanks to
AutoConfig(tm), it's 100% Plug-and-Play.
-There are two types of Zorro busses, Zorro II and Zorro III:
+There are two types of Zorro buses, Zorro II and Zorro III:
- The Zorro II address space is 24-bit and lies within the first 16 MB of the
Amiga's address map.
diff --git a/MAINTAINERS b/MAINTAINERS
index fc95cb0..e5b802a 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -896,12 +896,19 @@
APPLIED MICRO (APM) X-GENE SOC ETHERNET DRIVER
M: Iyappan Subramanian <isubramanian@apm.com>
M: Keyur Chudgar <kchudgar@apm.com>
+M: Quan Nguyen <qnguyen@apm.com>
S: Supported
F: drivers/net/ethernet/apm/xgene/
F: drivers/net/phy/mdio-xgene.c
F: Documentation/devicetree/bindings/net/apm-xgene-enet.txt
F: Documentation/devicetree/bindings/net/apm-xgene-mdio.txt
+APPLIED MICRO (APM) X-GENE SOC ETHERNET (V2) DRIVER
+M: Iyappan Subramanian <isubramanian@apm.com>
+M: Keyur Chudgar <kchudgar@apm.com>
+S: Supported
+F: drivers/net/ethernet/apm/xgene-v2/
+
APPLIED MICRO (APM) X-GENE SOC PMU
M: Tai Nguyen <ttnguyen@apm.com>
S: Supported
@@ -1088,6 +1095,8 @@
F: arch/arm/mach-artpec
F: arch/arm/boot/dts/artpec6*
F: drivers/clk/axis
+F: drivers/pinctrl/pinctrl-artpec*
+F: Documentation/devicetree/bindings/pinctrl/axis,artpec6-pinctrl.txt
ARM/ASPEED MACHINE SUPPORT
M: Joel Stanley <joel@jms.id.au>
@@ -2327,21 +2336,6 @@
F: drivers/auxdisplay/
F: include/linux/cfag12864b.h
-AVR32 ARCHITECTURE
-M: Haavard Skinnemoen <hskinnemoen@gmail.com>
-M: Hans-Christian Egtvedt <egtvedt@samfundet.no>
-W: http://www.atmel.com/products/AVR32/
-W: http://mirror.egtvedt.no/avr32linux.org/
-W: http://avrfreaks.net/
-S: Maintained
-F: arch/avr32/
-
-AVR32/AT32AP MACHINE SUPPORT
-M: Haavard Skinnemoen <hskinnemoen@gmail.com>
-M: Hans-Christian Egtvedt <egtvedt@samfundet.no>
-S: Maintained
-F: arch/avr32/mach-at32ap/
-
AX.25 NETWORK LAYER
M: Ralf Baechle <ralf@linux-mips.org>
L: linux-hams@vger.kernel.org
@@ -2544,6 +2538,14 @@
F: kernel/trace/blktrace.c
F: lib/sbitmap.c
+BFQ I/O SCHEDULER
+M: Paolo Valente <paolo.valente@linaro.org>
+M: Jens Axboe <axboe@kernel.dk>
+L: linux-block@vger.kernel.org
+S: Maintained
+F: block/bfq-*
+F: Documentation/block/bfq-iosched.txt
+
BLOCK2MTD DRIVER
M: Joern Engel <joern@lazybastard.org>
L: linux-mtd@lists.infradead.org
@@ -2951,6 +2953,15 @@
S: Maintained
F: arch/c6x/
+CA8210 IEEE-802.15.4 RADIO DRIVER
+M: Harry Morris <h.morris@cascoda.com>
+M: linuxdev@cascoda.com
+L: linux-wpan@vger.kernel.org
+W: https://github.com/Cascoda/ca8210-linux.git
+S: Maintained
+F: drivers/net/ieee802154/ca8210.c
+F: Documentation/devicetree/bindings/net/ieee802154/ca8210.txt
+
CACHEFILES: FS-CACHE BACKEND FOR CACHING ON MOUNTED FILESYSTEMS
M: David Howells <dhowells@redhat.com>
L: linux-cachefs@redhat.com (moderated for non-subscribers)
@@ -3055,6 +3066,14 @@
F: drivers/i2c/busses/i2c-octeon*
F: drivers/i2c/busses/i2c-thunderx*
+CAVIUM MMC DRIVER
+M: Jan Glauber <jglauber@cavium.com>
+M: David Daney <david.daney@cavium.com>
+M: Steven J. Hill <Steven.Hill@cavium.com>
+W: http://www.cavium.com
+S: Supported
+F: drivers/mmc/host/cavium*
+
CAVIUM LIQUIDIO NETWORK DRIVER
M: Derek Chickles <derek.chickles@caviumnetworks.com>
M: Satanand Burla <satananda.burla@caviumnetworks.com>
@@ -3463,6 +3482,7 @@
T: git git://git.linaro.org/people/vireshk/linux.git (For ARM Updates)
B: https://bugzilla.kernel.org
F: Documentation/cpu-freq/
+F: Documentation/devicetree/bindings/cpufreq/
F: drivers/cpufreq/
F: include/linux/cpufreq.h
F: tools/testing/selftests/cpufreq/
@@ -4707,6 +4727,7 @@
L: linux-mips@linux-mips.org
S: Supported
F: drivers/edac/octeon_edac*
+F: drivers/edac/thunderx_edac*
EDAC-E752X
M: Mark Gross <mark.gross@intel.com>
@@ -5420,6 +5441,23 @@
F: include/uapi/linux/fuse.h
F: Documentation/filesystems/fuse.txt
+FUTEX SUBSYSTEM
+M: Thomas Gleixner <tglx@linutronix.de>
+M: Ingo Molnar <mingo@redhat.com>
+R: Peter Zijlstra <peterz@infradead.org>
+R: Darren Hart <dvhart@infradead.org>
+L: linux-kernel@vger.kernel.org
+T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git locking/core
+S: Maintained
+F: kernel/futex.c
+F: kernel/futex_compat.c
+F: include/asm-generic/futex.h
+F: include/linux/futex.h
+F: include/uapi/linux/futex.h
+F: tools/testing/selftests/futex/
+F: tools/perf/bench/futex*
+F: Documentation/*futex*
+
FUTURE DOMAIN TMC-16x0 SCSI DRIVER (16-bit)
M: Rik Faith <faith@cs.unc.edu>
L: linux-scsi@vger.kernel.org
@@ -6024,7 +6062,7 @@
T: git git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-hsi.git
S: Maintained
F: Documentation/ABI/testing/sysfs-bus-hsi
-F: Documentation/device-drivers/serial-interfaces.rst
+F: Documentation/driver-api/hsi.rst
F: drivers/hsi/
F: include/linux/hsi/
F: include/uapi/linux/hsi/
@@ -6230,7 +6268,7 @@
F: drivers/crypto/nx/nx_debugfs.h
IBM Power 842 compression accelerator
-M: Dan Streetman <ddstreet@ieee.org>
+M: Haren Myneni <haren@us.ibm.com>
S: Supported
F: drivers/crypto/nx/Makefile
F: drivers/crypto/nx/Kconfig
@@ -7189,6 +7227,7 @@
F: Documentation/s390/kvm.txt
F: arch/s390/include/asm/kvm*
F: arch/s390/kvm/
+F: arch/s390/mm/gmap.c
KERNEL VIRTUAL MACHINE (KVM) FOR ARM
M: Christoffer Dall <christoffer.dall@linaro.org>
@@ -7871,7 +7910,7 @@
F: drivers/net/ethernet/marvell/mvneta.*
MARVELL MWIFIEX WIRELESS DRIVER
-M: Amitkumar Karwar <akarwar@marvell.com>
+M: Amitkumar Karwar <amitkarwar@gmail.com>
M: Nishant Sarmukadam <nishants@marvell.com>
M: Ganapathi Bhat <gbhat@marvell.com>
M: Xinming Hu <huxm@marvell.com>
@@ -7890,6 +7929,13 @@
S: Odd Fixes
F: drivers/mmc/host/mvsdio.*
+MARVELL XENON MMC/SD/SDIO HOST CONTROLLER DRIVER
+M: Hu Ziji <huziji@marvell.com>
+L: linux-mmc@vger.kernel.org
+S: Supported
+F: drivers/mmc/host/sdhci-xenon*
+F: Documentation/devicetree/bindings/mmc/marvell,xenon-sdhci.txt
+
MATROX FRAMEBUFFER DRIVER
L: linux-fbdev@vger.kernel.org
S: Orphan
@@ -8818,12 +8864,12 @@
F: net/xfrm/
F: net/key/
F: net/ipv4/xfrm*
-F: net/ipv4/esp4.c
+F: net/ipv4/esp4*
F: net/ipv4/ah4.c
F: net/ipv4/ipcomp.c
F: net/ipv4/ip_vti.c
F: net/ipv6/xfrm*
-F: net/ipv6/esp6.c
+F: net/ipv6/esp6*
F: net/ipv6/ah6.c
F: net/ipv6/ipcomp6.c
F: net/ipv6/ip6_vti.c
@@ -8877,8 +8923,6 @@
F: drivers/net/ethernet/qlogic/netxen/
NFC SUBSYSTEM
-M: Lauro Ramos Venancio <lauro.venancio@openbossa.org>
-M: Aloisio Almeida Jr <aloisio.almeida@openbossa.org>
M: Samuel Ortiz <sameo@linux.intel.com>
L: linux-wireless@vger.kernel.org
L: linux-nfc@lists.01.org (subscribers-only)
@@ -9933,6 +9977,8 @@
M: Sylwester Nawrocki <s.nawrocki@samsung.com>
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
L: linux-samsung-soc@vger.kernel.org (moderated for non-subscribers)
+Q: https://patchwork.kernel.org/project/linux-samsung-soc/list/
+T: git git://git.kernel.org/pub/scm/linux/kernel/git/pinctrl/samsung.git
S: Maintained
F: drivers/pinctrl/samsung/
F: include/dt-bindings/pinctrl/samsung.h
@@ -10099,7 +10145,7 @@
PREEMPTIBLE KERNEL
M: Robert Love <rml@tech9.net>
L: kpreempt-tech@lists.sourceforge.net
-W: ftp://ftp.kernel.org/pub/linux/kernel/people/rml/preempt-kernel
+W: https://www.kernel.org/pub/linux/kernel/people/rml/preempt-kernel
S: Supported
F: Documentation/preempt-locking.txt
F: include/linux/preempt.h
@@ -10886,6 +10932,16 @@
S: Supported
F: drivers/iommu/s390-iommu.c
+S390 VFIO-CCW DRIVER
+M: Cornelia Huck <cornelia.huck@de.ibm.com>
+M: Dong Jia Shi <bjsdjshi@linux.vnet.ibm.com>
+L: linux-s390@vger.kernel.org
+L: kvm@vger.kernel.org
+S: Supported
+F: drivers/s390/cio/vfio_ccw*
+F: Documentation/s390/vfio-ccw.txt
+F: include/uapi/linux/vfio_ccw.h
+
S3C24XX SD/MMC Driver
M: Ben Dooks <ben-linux@fluff.org>
L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers)
@@ -10933,6 +10989,14 @@
S: Supported
F: sound/soc/samsung/
+SAMSUNG EXYNOS PSEUDO RANDOM NUMBER GENERATOR (RNG) DRIVER
+M: Krzysztof Kozlowski <krzk@kernel.org>
+L: linux-crypto@vger.kernel.org
+L: linux-samsung-soc@vger.kernel.org
+S: Maintained
+F: drivers/crypto/exynos-rng.c
+F: Documentation/devicetree/bindings/rng/samsung,exynos-rng4.txt
+
SAMSUNG FRAMEBUFFER DRIVER
M: Jingoo Han <jingoohan1@gmail.com>
L: linux-fbdev@vger.kernel.org
@@ -10957,6 +11021,14 @@
F: Documentation/devicetree/bindings/regulator/samsung,s5m*.txt
F: Documentation/devicetree/bindings/clock/samsung,s2mps11.txt
+SAMSUNG S5P Security SubSystem (SSS) DRIVER
+M: Krzysztof Kozlowski <krzk@kernel.org>
+M: Vladimir Zapolskiy <vz@mleia.com>
+L: linux-crypto@vger.kernel.org
+L: linux-samsung-soc@vger.kernel.org
+S: Maintained
+F: drivers/crypto/s5p-sss.c
+
SAMSUNG S5P/EXYNOS4 SOC SERIES CAMERA SUBSYSTEM DRIVERS
M: Kyungmin Park <kyungmin.park@samsung.com>
M: Sylwester Nawrocki <s.nawrocki@samsung.com>
@@ -11088,6 +11160,12 @@
F: include/linux/platform_data/dma-dw.h
F: drivers/dma/dw/
+SYNOPSYS DESIGNWARE ENTERPRISE ETHERNET DRIVER
+M: Jie Deng <jiedeng@synopsys.com>
+L: netdev@vger.kernel.org
+S: Supported
+F: drivers/net/ethernet/synopsys/
+
SYNOPSYS DESIGNWARE I2C DRIVER
M: Jarkko Nikula <jarkko.nikula@linux.intel.com>
R: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
@@ -11126,6 +11204,7 @@
TIMEKEEPING, CLOCKSOURCE CORE, NTP, ALARMTIMER
M: John Stultz <john.stultz@linaro.org>
M: Thomas Gleixner <tglx@linutronix.de>
+R: Stephen Boyd <sboyd@codeaurora.org>
L: linux-kernel@vger.kernel.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/core
S: Supported
@@ -12191,12 +12270,19 @@
F: include/linux/taskstats*
F: kernel/taskstats.c
-TC CLASSIFIER
+TC subsystem
M: Jamal Hadi Salim <jhs@mojatatu.com>
+M: Cong Wang <xiyou.wangcong@gmail.com>
+M: Jiri Pirko <jiri@resnulli.us>
L: netdev@vger.kernel.org
S: Maintained
F: include/net/pkt_cls.h
+F: include/net/pkt_sched.h
+F: include/net/tc_act/
F: include/uapi/linux/pkt_cls.h
+F: include/uapi/linux/pkt_sched.h
+F: include/uapi/linux/tc_act/
+F: include/uapi/linux/tc_ematch/
F: net/sched/
TCP LOW PRIORITY MODULE
@@ -13292,8 +13378,11 @@
S: Maintained
F: include/linux/virtio_vsock.h
F: include/uapi/linux/virtio_vsock.h
+F: include/uapi/linux/vsockmon.h
+F: net/vmw_vsock/af_vsock_tap.c
F: net/vmw_vsock/virtio_transport_common.c
F: net/vmw_vsock/virtio_transport.c
+F: drivers/net/vsockmon.c
F: drivers/vhost/vsock.c
F: drivers/vhost/vsock.h
diff --git a/Makefile b/Makefile
index 5039b91..43534cc 100644
--- a/Makefile
+++ b/Makefile
@@ -1,7 +1,7 @@
VERSION = 4
PATCHLEVEL = 11
SUBLEVEL = 0
-EXTRAVERSION = -rc7
+EXTRAVERSION =
NAME = Fearless Coyote
# *DOCUMENTATION*
@@ -172,8 +172,8 @@
# Use 'make C=2' to enable checking of *all* source files, regardless
# of whether they are re-compiled or not.
#
-# See the file "Documentation/sparse.txt" for more details, including
-# where to get the "sparse" utility.
+# See the file "Documentation/dev-tools/sparse.rst" for more details,
+# including where to get the "sparse" utility.
ifeq ("$(origin C)", "command line")
KBUILD_CHECKSRC = $(C)
diff --git a/arch/Kconfig b/arch/Kconfig
index cd211a1..6409994 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -700,6 +700,13 @@
This value can be changed after boot using the
/proc/sys/vm/mmap_rnd_compat_bits tunable
+config HAVE_ARCH_COMPAT_MMAP_BASES
+ bool
+ help
+ This allows 64bit applications to invoke 32-bit mmap() syscall
+ and vice-versa 32-bit applications to call 64-bit mmap().
+ Required for applications doing different bitness syscalls.
+
config HAVE_COPY_THREAD_TLS
bool
help
@@ -713,6 +720,12 @@
Architecture supports the 'objtool check' host tool command, which
performs compile-time stack metadata validation.
+config HAVE_RELIABLE_STACKTRACE
+ bool
+ help
+ Architecture has a save_stack_trace_tsk_reliable() function which
+ only returns a stack trace if it can guarantee the trace is reliable.
+
config HAVE_ARCH_HASH
bool
default n
diff --git a/arch/alpha/include/asm/extable.h b/arch/alpha/include/asm/extable.h
new file mode 100644
index 0000000..048e209
--- /dev/null
+++ b/arch/alpha/include/asm/extable.h
@@ -0,0 +1,55 @@
+#ifndef _ASM_EXTABLE_H
+#define _ASM_EXTABLE_H
+
+/*
+ * About the exception table:
+ *
+ * - insn is a 32-bit pc-relative offset from the faulting insn.
+ * - nextinsn is a 16-bit offset off of the faulting instruction
+ * (not off of the *next* instruction as branches are).
+ * - errreg is the register in which to place -EFAULT.
+ * - valreg is the final target register for the load sequence
+ * and will be zeroed.
+ *
+ * Either errreg or valreg may be $31, in which case nothing happens.
+ *
+ * The exception fixup information "just so happens" to be arranged
+ * as in a MEM format instruction. This lets us emit our three
+ * values like so:
+ *
+ * lda valreg, nextinsn(errreg)
+ *
+ */
+
+struct exception_table_entry
+{
+ signed int insn;
+ union exception_fixup {
+ unsigned unit;
+ struct {
+ signed int nextinsn : 16;
+ unsigned int errreg : 5;
+ unsigned int valreg : 5;
+ } bits;
+ } fixup;
+};
+
+/* Returns the new pc */
+#define fixup_exception(map_reg, _fixup, pc) \
+({ \
+ if ((_fixup)->fixup.bits.valreg != 31) \
+ map_reg((_fixup)->fixup.bits.valreg) = 0; \
+ if ((_fixup)->fixup.bits.errreg != 31) \
+ map_reg((_fixup)->fixup.bits.errreg) = -EFAULT; \
+ (pc) + (_fixup)->fixup.bits.nextinsn; \
+})
+
+#define ARCH_HAS_RELATIVE_EXTABLE
+
+#define swap_ex_entry_fixup(a, b, tmp, delta) \
+ do { \
+ (a)->fixup.unit = (b)->fixup.unit; \
+ (b)->fixup.unit = (tmp).fixup.unit; \
+ } while (0)
+
+#endif
diff --git a/arch/alpha/include/asm/futex.h b/arch/alpha/include/asm/futex.h
index f939794..fb01dfb 100644
--- a/arch/alpha/include/asm/futex.h
+++ b/arch/alpha/include/asm/futex.h
@@ -19,12 +19,8 @@
"3: .subsection 2\n" \
"4: br 1b\n" \
" .previous\n" \
- " .section __ex_table,\"a\"\n" \
- " .long 1b-.\n" \
- " lda $31,3b-1b(%1)\n" \
- " .long 2b-.\n" \
- " lda $31,3b-2b(%1)\n" \
- " .previous\n" \
+ EXC(1b,3b,%1,$31) \
+ EXC(2b,3b,%1,$31) \
: "=&r" (oldval), "=&r"(ret) \
: "r" (uaddr), "r"(oparg) \
: "memory")
@@ -101,12 +97,8 @@
"3: .subsection 2\n"
"4: br 1b\n"
" .previous\n"
- " .section __ex_table,\"a\"\n"
- " .long 1b-.\n"
- " lda $31,3b-1b(%0)\n"
- " .long 2b-.\n"
- " lda $31,3b-2b(%0)\n"
- " .previous\n"
+ EXC(1b,3b,%0,$31)
+ EXC(2b,3b,%0,$31)
: "+r"(ret), "=&r"(prev), "=&r"(cmp)
: "r"(uaddr), "r"((long)(int)oldval), "r"(newval)
: "memory");
diff --git a/arch/alpha/include/asm/uaccess.h b/arch/alpha/include/asm/uaccess.h
index 94f5875..7b82dc9 100644
--- a/arch/alpha/include/asm/uaccess.h
+++ b/arch/alpha/include/asm/uaccess.h
@@ -1,10 +1,6 @@
#ifndef __ALPHA_UACCESS_H
#define __ALPHA_UACCESS_H
-#include <linux/errno.h>
-#include <linux/sched.h>
-
-
/*
* The fs value determines whether argument validity checking should be
* performed or not. If get_fs() == USER_DS, checking is performed, with
@@ -20,9 +16,6 @@
#define KERNEL_DS ((mm_segment_t) { 0UL })
#define USER_DS ((mm_segment_t) { -0x40000000000UL })
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
#define get_fs() (current_thread_info()->addr_limit)
#define get_ds() (KERNEL_DS)
#define set_fs(x) (current_thread_info()->addr_limit = (x))
@@ -39,13 +32,13 @@
* - AND "addr+size" doesn't have any high-bits set
* - OR we are in kernel mode.
*/
-#define __access_ok(addr, size, segment) \
- (((segment).seg & (addr | size | (addr+size))) == 0)
+#define __access_ok(addr, size) \
+ ((get_fs().seg & (addr | size | (addr+size))) == 0)
-#define access_ok(type, addr, size) \
-({ \
- __chk_user_ptr(addr); \
- __access_ok(((unsigned long)(addr)), (size), get_fs()); \
+#define access_ok(type, addr, size) \
+({ \
+ __chk_user_ptr(addr); \
+ __access_ok(((unsigned long)(addr)), (size)); \
})
/*
@@ -61,9 +54,9 @@
* (b) require any knowledge of processes at this stage
*/
#define put_user(x, ptr) \
- __put_user_check((__typeof__(*(ptr)))(x), (ptr), sizeof(*(ptr)), get_fs())
+ __put_user_check((__typeof__(*(ptr)))(x), (ptr), sizeof(*(ptr)))
#define get_user(x, ptr) \
- __get_user_check((x), (ptr), sizeof(*(ptr)), get_fs())
+ __get_user_check((x), (ptr), sizeof(*(ptr)))
/*
* The "__xxx" versions do not do address space checking, useful when
@@ -81,6 +74,11 @@
* more extensive comments with fixup_inline_exception below for
* more information.
*/
+#define EXC(label,cont,res,err) \
+ ".section __ex_table,\"a\"\n" \
+ " .long "#label"-.\n" \
+ " lda "#res","#cont"-"#label"("#err")\n" \
+ ".previous\n"
extern void __get_user_unknown(void);
@@ -100,23 +98,23 @@
__gu_err; \
})
-#define __get_user_check(x, ptr, size, segment) \
-({ \
- long __gu_err = -EFAULT; \
- unsigned long __gu_val = 0; \
- const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \
- if (__access_ok((unsigned long)__gu_addr, size, segment)) { \
- __gu_err = 0; \
- switch (size) { \
- case 1: __get_user_8(__gu_addr); break; \
- case 2: __get_user_16(__gu_addr); break; \
- case 4: __get_user_32(__gu_addr); break; \
- case 8: __get_user_64(__gu_addr); break; \
- default: __get_user_unknown(); break; \
- } \
- } \
- (x) = (__force __typeof__(*(ptr))) __gu_val; \
- __gu_err; \
+#define __get_user_check(x, ptr, size) \
+({ \
+ long __gu_err = -EFAULT; \
+ unsigned long __gu_val = 0; \
+ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \
+ if (__access_ok((unsigned long)__gu_addr, size)) { \
+ __gu_err = 0; \
+ switch (size) { \
+ case 1: __get_user_8(__gu_addr); break; \
+ case 2: __get_user_16(__gu_addr); break; \
+ case 4: __get_user_32(__gu_addr); break; \
+ case 8: __get_user_64(__gu_addr); break; \
+ default: __get_user_unknown(); break; \
+ } \
+ } \
+ (x) = (__force __typeof__(*(ptr))) __gu_val; \
+ __gu_err; \
})
struct __large_struct { unsigned long buf[100]; };
@@ -125,20 +123,14 @@
#define __get_user_64(addr) \
__asm__("1: ldq %0,%2\n" \
"2:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - .\n" \
- " lda %0, 2b-1b(%1)\n" \
- ".previous" \
+ EXC(1b,2b,%0,%1) \
: "=r"(__gu_val), "=r"(__gu_err) \
: "m"(__m(addr)), "1"(__gu_err))
#define __get_user_32(addr) \
__asm__("1: ldl %0,%2\n" \
"2:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - .\n" \
- " lda %0, 2b-1b(%1)\n" \
- ".previous" \
+ EXC(1b,2b,%0,%1) \
: "=r"(__gu_val), "=r"(__gu_err) \
: "m"(__m(addr)), "1"(__gu_err))
@@ -148,20 +140,14 @@
#define __get_user_16(addr) \
__asm__("1: ldwu %0,%2\n" \
"2:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - .\n" \
- " lda %0, 2b-1b(%1)\n" \
- ".previous" \
+ EXC(1b,2b,%0,%1) \
: "=r"(__gu_val), "=r"(__gu_err) \
: "m"(__m(addr)), "1"(__gu_err))
#define __get_user_8(addr) \
__asm__("1: ldbu %0,%2\n" \
"2:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - .\n" \
- " lda %0, 2b-1b(%1)\n" \
- ".previous" \
+ EXC(1b,2b,%0,%1) \
: "=r"(__gu_val), "=r"(__gu_err) \
: "m"(__m(addr)), "1"(__gu_err))
#else
@@ -177,12 +163,8 @@
" extwh %1,%3,%1\n" \
" or %0,%1,%0\n" \
"3:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - .\n" \
- " lda %0, 3b-1b(%2)\n" \
- " .long 2b - .\n" \
- " lda %0, 3b-2b(%2)\n" \
- ".previous" \
+ EXC(1b,3b,%0,%2) \
+ EXC(2b,3b,%0,%2) \
: "=&r"(__gu_val), "=&r"(__gu_tmp), "=r"(__gu_err) \
: "r"(addr), "2"(__gu_err)); \
}
@@ -191,10 +173,7 @@
__asm__("1: ldq_u %0,0(%2)\n" \
" extbl %0,%2,%0\n" \
"2:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - .\n" \
- " lda %0, 2b-1b(%1)\n" \
- ".previous" \
+ EXC(1b,2b,%0,%1) \
: "=&r"(__gu_val), "=r"(__gu_err) \
: "r"(addr), "1"(__gu_err))
#endif
@@ -215,21 +194,21 @@
__pu_err; \
})
-#define __put_user_check(x, ptr, size, segment) \
-({ \
- long __pu_err = -EFAULT; \
- __typeof__(*(ptr)) __user *__pu_addr = (ptr); \
- if (__access_ok((unsigned long)__pu_addr, size, segment)) { \
- __pu_err = 0; \
- switch (size) { \
- case 1: __put_user_8(x, __pu_addr); break; \
- case 2: __put_user_16(x, __pu_addr); break; \
- case 4: __put_user_32(x, __pu_addr); break; \
- case 8: __put_user_64(x, __pu_addr); break; \
- default: __put_user_unknown(); break; \
- } \
- } \
- __pu_err; \
+#define __put_user_check(x, ptr, size) \
+({ \
+ long __pu_err = -EFAULT; \
+ __typeof__(*(ptr)) __user *__pu_addr = (ptr); \
+ if (__access_ok((unsigned long)__pu_addr, size)) { \
+ __pu_err = 0; \
+ switch (size) { \
+ case 1: __put_user_8(x, __pu_addr); break; \
+ case 2: __put_user_16(x, __pu_addr); break; \
+ case 4: __put_user_32(x, __pu_addr); break; \
+ case 8: __put_user_64(x, __pu_addr); break; \
+ default: __put_user_unknown(); break; \
+ } \
+ } \
+ __pu_err; \
})
/*
@@ -240,20 +219,14 @@
#define __put_user_64(x, addr) \
__asm__ __volatile__("1: stq %r2,%1\n" \
"2:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - .\n" \
- " lda $31,2b-1b(%0)\n" \
- ".previous" \
+ EXC(1b,2b,$31,%0) \
: "=r"(__pu_err) \
: "m" (__m(addr)), "rJ" (x), "0"(__pu_err))
#define __put_user_32(x, addr) \
__asm__ __volatile__("1: stl %r2,%1\n" \
"2:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - .\n" \
- " lda $31,2b-1b(%0)\n" \
- ".previous" \
+ EXC(1b,2b,$31,%0) \
: "=r"(__pu_err) \
: "m"(__m(addr)), "rJ"(x), "0"(__pu_err))
@@ -263,20 +236,14 @@
#define __put_user_16(x, addr) \
__asm__ __volatile__("1: stw %r2,%1\n" \
"2:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - .\n" \
- " lda $31,2b-1b(%0)\n" \
- ".previous" \
+ EXC(1b,2b,$31,%0) \
: "=r"(__pu_err) \
: "m"(__m(addr)), "rJ"(x), "0"(__pu_err))
#define __put_user_8(x, addr) \
__asm__ __volatile__("1: stb %r2,%1\n" \
"2:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - .\n" \
- " lda $31,2b-1b(%0)\n" \
- ".previous" \
+ EXC(1b,2b,$31,%0) \
: "=r"(__pu_err) \
: "m"(__m(addr)), "rJ"(x), "0"(__pu_err))
#else
@@ -298,16 +265,10 @@
"3: stq_u %2,1(%5)\n" \
"4: stq_u %1,0(%5)\n" \
"5:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - .\n" \
- " lda $31, 5b-1b(%0)\n" \
- " .long 2b - .\n" \
- " lda $31, 5b-2b(%0)\n" \
- " .long 3b - .\n" \
- " lda $31, 5b-3b(%0)\n" \
- " .long 4b - .\n" \
- " lda $31, 5b-4b(%0)\n" \
- ".previous" \
+ EXC(1b,5b,$31,%0) \
+ EXC(2b,5b,$31,%0) \
+ EXC(3b,5b,$31,%0) \
+ EXC(4b,5b,$31,%0) \
: "=r"(__pu_err), "=&r"(__pu_tmp1), \
"=&r"(__pu_tmp2), "=&r"(__pu_tmp3), \
"=&r"(__pu_tmp4) \
@@ -324,12 +285,8 @@
" or %1,%2,%1\n" \
"2: stq_u %1,0(%4)\n" \
"3:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - .\n" \
- " lda $31, 3b-1b(%0)\n" \
- " .long 2b - .\n" \
- " lda $31, 3b-2b(%0)\n" \
- ".previous" \
+ EXC(1b,3b,$31,%0) \
+ EXC(2b,3b,$31,%0) \
: "=r"(__pu_err), \
"=&r"(__pu_tmp1), "=&r"(__pu_tmp2) \
: "r"((unsigned long)(x)), "r"(addr), "0"(__pu_err)); \
@@ -341,153 +298,37 @@
* Complex access routines
*/
-/* This little bit of silliness is to get the GP loaded for a function
- that ordinarily wouldn't. Otherwise we could have it done by the macro
- directly, which can be optimized the linker. */
-#ifdef MODULE
-#define __module_address(sym) "r"(sym),
-#define __module_call(ra, arg, sym) "jsr $" #ra ",(%" #arg ")," #sym
-#else
-#define __module_address(sym)
-#define __module_call(ra, arg, sym) "bsr $" #ra "," #sym " !samegp"
-#endif
+extern long __copy_user(void *to, const void *from, long len);
-extern void __copy_user(void);
-
-extern inline long
-__copy_tofrom_user_nocheck(void *to, const void *from, long len)
+static inline unsigned long
+raw_copy_from_user(void *to, const void __user *from, unsigned long len)
{
- register void * __cu_to __asm__("$6") = to;
- register const void * __cu_from __asm__("$7") = from;
- register long __cu_len __asm__("$0") = len;
-
- __asm__ __volatile__(
- __module_call(28, 3, __copy_user)
- : "=r" (__cu_len), "=r" (__cu_from), "=r" (__cu_to)
- : __module_address(__copy_user)
- "0" (__cu_len), "1" (__cu_from), "2" (__cu_to)
- : "$1", "$2", "$3", "$4", "$5", "$28", "memory");
-
- return __cu_len;
+ return __copy_user(to, (__force const void *)from, len);
}
-#define __copy_to_user(to, from, n) \
-({ \
- __chk_user_ptr(to); \
- __copy_tofrom_user_nocheck((__force void *)(to), (from), (n)); \
-})
-#define __copy_from_user(to, from, n) \
-({ \
- __chk_user_ptr(from); \
- __copy_tofrom_user_nocheck((to), (__force void *)(from), (n)); \
-})
-
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
-
-extern inline long
-copy_to_user(void __user *to, const void *from, long n)
+static inline unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long len)
{
- if (likely(__access_ok((unsigned long)to, n, get_fs())))
- n = __copy_tofrom_user_nocheck((__force void *)to, from, n);
- return n;
+ return __copy_user((__force void *)to, from, len);
}
-extern inline long
-copy_from_user(void *to, const void __user *from, long n)
-{
- long res = n;
- if (likely(__access_ok((unsigned long)from, n, get_fs())))
- res = __copy_from_user_inatomic(to, from, n);
- if (unlikely(res))
- memset(to + (n - res), 0, res);
- return res;
-}
-
-extern void __do_clear_user(void);
-
-extern inline long
-__clear_user(void __user *to, long len)
-{
- register void __user * __cl_to __asm__("$6") = to;
- register long __cl_len __asm__("$0") = len;
- __asm__ __volatile__(
- __module_call(28, 2, __do_clear_user)
- : "=r"(__cl_len), "=r"(__cl_to)
- : __module_address(__do_clear_user)
- "0"(__cl_len), "1"(__cl_to)
- : "$1", "$2", "$3", "$4", "$5", "$28", "memory");
- return __cl_len;
-}
+extern long __clear_user(void __user *to, long len);
extern inline long
clear_user(void __user *to, long len)
{
- if (__access_ok((unsigned long)to, len, get_fs()))
+ if (__access_ok((unsigned long)to, len))
len = __clear_user(to, len);
return len;
}
-#undef __module_address
-#undef __module_call
-
#define user_addr_max() \
- (segment_eq(get_fs(), USER_DS) ? TASK_SIZE : ~0UL)
+ (uaccess_kernel() ? ~0UL : TASK_SIZE)
extern long strncpy_from_user(char *dest, const char __user *src, long count);
extern __must_check long strlen_user(const char __user *str);
extern __must_check long strnlen_user(const char __user *str, long n);
-/*
- * About the exception table:
- *
- * - insn is a 32-bit pc-relative offset from the faulting insn.
- * - nextinsn is a 16-bit offset off of the faulting instruction
- * (not off of the *next* instruction as branches are).
- * - errreg is the register in which to place -EFAULT.
- * - valreg is the final target register for the load sequence
- * and will be zeroed.
- *
- * Either errreg or valreg may be $31, in which case nothing happens.
- *
- * The exception fixup information "just so happens" to be arranged
- * as in a MEM format instruction. This lets us emit our three
- * values like so:
- *
- * lda valreg, nextinsn(errreg)
- *
- */
-
-struct exception_table_entry
-{
- signed int insn;
- union exception_fixup {
- unsigned unit;
- struct {
- signed int nextinsn : 16;
- unsigned int errreg : 5;
- unsigned int valreg : 5;
- } bits;
- } fixup;
-};
-
-/* Returns the new pc */
-#define fixup_exception(map_reg, _fixup, pc) \
-({ \
- if ((_fixup)->fixup.bits.valreg != 31) \
- map_reg((_fixup)->fixup.bits.valreg) = 0; \
- if ((_fixup)->fixup.bits.errreg != 31) \
- map_reg((_fixup)->fixup.bits.errreg) = -EFAULT; \
- (pc) + (_fixup)->fixup.bits.nextinsn; \
-})
-
-#define ARCH_HAS_RELATIVE_EXTABLE
-
-#define swap_ex_entry_fixup(a, b, tmp, delta) \
- do { \
- (a)->fixup.unit = (b)->fixup.unit; \
- (b)->fixup.unit = (tmp).fixup.unit; \
- } while (0)
-
+#include <asm/extable.h>
#endif /* __ALPHA_UACCESS_H */
diff --git a/arch/alpha/include/uapi/asm/socket.h b/arch/alpha/include/uapi/asm/socket.h
index afc901b..148d7a3 100644
--- a/arch/alpha/include/uapi/asm/socket.h
+++ b/arch/alpha/include/uapi/asm/socket.h
@@ -99,4 +99,10 @@
#define SCM_TIMESTAMPING_OPT_STATS 54
+#define SO_MEMINFO 55
+
+#define SO_INCOMING_NAPI_ID 56
+
+#define SO_COOKIE 57
+
#endif /* _UAPI_ASM_SOCKET_H */
diff --git a/arch/alpha/kernel/osf_sys.c b/arch/alpha/kernel/osf_sys.c
index 6d76e52..9ec56dc 100644
--- a/arch/alpha/kernel/osf_sys.c
+++ b/arch/alpha/kernel/osf_sys.c
@@ -1016,6 +1016,7 @@
SYSCALL_DEFINE2(osf_settimeofday, struct timeval32 __user *, tv,
struct timezone __user *, tz)
{
+ struct timespec64 kts64;
struct timespec kts;
struct timezone ktz;
@@ -1023,13 +1024,14 @@
if (get_tv32((struct timeval *)&kts, tv))
return -EFAULT;
kts.tv_nsec *= 1000;
+ kts64 = timespec_to_timespec64(kts);
}
if (tz) {
if (copy_from_user(&ktz, tz, sizeof(*tz)))
return -EFAULT;
}
- return do_sys_settimeofday(tv ? &kts : NULL, tz ? &ktz : NULL);
+ return do_sys_settimeofday64(tv ? &kts64 : NULL, tz ? &ktz : NULL);
}
asmlinkage long sys_ni_posix_timers(void);
diff --git a/arch/alpha/kernel/traps.c b/arch/alpha/kernel/traps.c
index b137390..65bb102 100644
--- a/arch/alpha/kernel/traps.c
+++ b/arch/alpha/kernel/traps.c
@@ -482,12 +482,8 @@
" extwl %1,%3,%1\n"
" extwh %2,%3,%2\n"
"3:\n"
- ".section __ex_table,\"a\"\n"
- " .long 1b - .\n"
- " lda %1,3b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %2,3b-2b(%0)\n"
- ".previous"
+ EXC(1b,3b,%1,%0)
+ EXC(2b,3b,%2,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2)
: "r"(va), "0"(0));
if (error)
@@ -502,12 +498,8 @@
" extll %1,%3,%1\n"
" extlh %2,%3,%2\n"
"3:\n"
- ".section __ex_table,\"a\"\n"
- " .long 1b - .\n"
- " lda %1,3b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %2,3b-2b(%0)\n"
- ".previous"
+ EXC(1b,3b,%1,%0)
+ EXC(2b,3b,%2,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2)
: "r"(va), "0"(0));
if (error)
@@ -522,12 +514,8 @@
" extql %1,%3,%1\n"
" extqh %2,%3,%2\n"
"3:\n"
- ".section __ex_table,\"a\"\n"
- " .long 1b - .\n"
- " lda %1,3b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %2,3b-2b(%0)\n"
- ".previous"
+ EXC(1b,3b,%1,%0)
+ EXC(2b,3b,%2,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2)
: "r"(va), "0"(0));
if (error)
@@ -551,16 +539,10 @@
"3: stq_u %2,1(%5)\n"
"4: stq_u %1,0(%5)\n"
"5:\n"
- ".section __ex_table,\"a\"\n"
- " .long 1b - .\n"
- " lda %2,5b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %1,5b-2b(%0)\n"
- " .long 3b - .\n"
- " lda $31,5b-3b(%0)\n"
- " .long 4b - .\n"
- " lda $31,5b-4b(%0)\n"
- ".previous"
+ EXC(1b,5b,%2,%0)
+ EXC(2b,5b,%1,%0)
+ EXC(3b,5b,$31,%0)
+ EXC(4b,5b,$31,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2),
"=&r"(tmp3), "=&r"(tmp4)
: "r"(va), "r"(una_reg(reg)), "0"(0));
@@ -581,16 +563,10 @@
"3: stq_u %2,3(%5)\n"
"4: stq_u %1,0(%5)\n"
"5:\n"
- ".section __ex_table,\"a\"\n"
- " .long 1b - .\n"
- " lda %2,5b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %1,5b-2b(%0)\n"
- " .long 3b - .\n"
- " lda $31,5b-3b(%0)\n"
- " .long 4b - .\n"
- " lda $31,5b-4b(%0)\n"
- ".previous"
+ EXC(1b,5b,%2,%0)
+ EXC(2b,5b,%1,%0)
+ EXC(3b,5b,$31,%0)
+ EXC(4b,5b,$31,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2),
"=&r"(tmp3), "=&r"(tmp4)
: "r"(va), "r"(una_reg(reg)), "0"(0));
@@ -611,16 +587,10 @@
"3: stq_u %2,7(%5)\n"
"4: stq_u %1,0(%5)\n"
"5:\n"
- ".section __ex_table,\"a\"\n\t"
- " .long 1b - .\n"
- " lda %2,5b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %1,5b-2b(%0)\n"
- " .long 3b - .\n"
- " lda $31,5b-3b(%0)\n"
- " .long 4b - .\n"
- " lda $31,5b-4b(%0)\n"
- ".previous"
+ EXC(1b,5b,%2,%0)
+ EXC(2b,5b,%1,%0)
+ EXC(3b,5b,$31,%0)
+ EXC(4b,5b,$31,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2),
"=&r"(tmp3), "=&r"(tmp4)
: "r"(va), "r"(una_reg(reg)), "0"(0));
@@ -802,7 +772,7 @@
/* Don't bother reading ds in the access check since we already
know that this came from the user. Also rely on the fact that
the page at TASK_SIZE is unmapped and so can't be touched anyway. */
- if (!__access_ok((unsigned long)va, 0, USER_DS))
+ if ((unsigned long)va >= TASK_SIZE)
goto give_sigsegv;
++unaligned[1].count;
@@ -835,12 +805,8 @@
" extwl %1,%3,%1\n"
" extwh %2,%3,%2\n"
"3:\n"
- ".section __ex_table,\"a\"\n"
- " .long 1b - .\n"
- " lda %1,3b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %2,3b-2b(%0)\n"
- ".previous"
+ EXC(1b,3b,%1,%0)
+ EXC(2b,3b,%2,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2)
: "r"(va), "0"(0));
if (error)
@@ -855,12 +821,8 @@
" extll %1,%3,%1\n"
" extlh %2,%3,%2\n"
"3:\n"
- ".section __ex_table,\"a\"\n"
- " .long 1b - .\n"
- " lda %1,3b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %2,3b-2b(%0)\n"
- ".previous"
+ EXC(1b,3b,%1,%0)
+ EXC(2b,3b,%2,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2)
: "r"(va), "0"(0));
if (error)
@@ -875,12 +837,8 @@
" extql %1,%3,%1\n"
" extqh %2,%3,%2\n"
"3:\n"
- ".section __ex_table,\"a\"\n"
- " .long 1b - .\n"
- " lda %1,3b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %2,3b-2b(%0)\n"
- ".previous"
+ EXC(1b,3b,%1,%0)
+ EXC(2b,3b,%2,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2)
: "r"(va), "0"(0));
if (error)
@@ -895,12 +853,8 @@
" extll %1,%3,%1\n"
" extlh %2,%3,%2\n"
"3:\n"
- ".section __ex_table,\"a\"\n"
- " .long 1b - .\n"
- " lda %1,3b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %2,3b-2b(%0)\n"
- ".previous"
+ EXC(1b,3b,%1,%0)
+ EXC(2b,3b,%2,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2)
: "r"(va), "0"(0));
if (error)
@@ -915,12 +869,8 @@
" extql %1,%3,%1\n"
" extqh %2,%3,%2\n"
"3:\n"
- ".section __ex_table,\"a\"\n"
- " .long 1b - .\n"
- " lda %1,3b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %2,3b-2b(%0)\n"
- ".previous"
+ EXC(1b,3b,%1,%0)
+ EXC(2b,3b,%2,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2)
: "r"(va), "0"(0));
if (error)
@@ -944,16 +894,10 @@
"3: stq_u %2,1(%5)\n"
"4: stq_u %1,0(%5)\n"
"5:\n"
- ".section __ex_table,\"a\"\n"
- " .long 1b - .\n"
- " lda %2,5b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %1,5b-2b(%0)\n"
- " .long 3b - .\n"
- " lda $31,5b-3b(%0)\n"
- " .long 4b - .\n"
- " lda $31,5b-4b(%0)\n"
- ".previous"
+ EXC(1b,5b,%2,%0)
+ EXC(2b,5b,%1,%0)
+ EXC(3b,5b,$31,%0)
+ EXC(4b,5b,$31,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2),
"=&r"(tmp3), "=&r"(tmp4)
: "r"(va), "r"(*reg_addr), "0"(0));
@@ -978,16 +922,10 @@
"3: stq_u %2,3(%5)\n"
"4: stq_u %1,0(%5)\n"
"5:\n"
- ".section __ex_table,\"a\"\n"
- " .long 1b - .\n"
- " lda %2,5b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %1,5b-2b(%0)\n"
- " .long 3b - .\n"
- " lda $31,5b-3b(%0)\n"
- " .long 4b - .\n"
- " lda $31,5b-4b(%0)\n"
- ".previous"
+ EXC(1b,5b,%2,%0)
+ EXC(2b,5b,%1,%0)
+ EXC(3b,5b,$31,%0)
+ EXC(4b,5b,$31,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2),
"=&r"(tmp3), "=&r"(tmp4)
: "r"(va), "r"(*reg_addr), "0"(0));
@@ -1012,16 +950,10 @@
"3: stq_u %2,7(%5)\n"
"4: stq_u %1,0(%5)\n"
"5:\n"
- ".section __ex_table,\"a\"\n\t"
- " .long 1b - .\n"
- " lda %2,5b-1b(%0)\n"
- " .long 2b - .\n"
- " lda %1,5b-2b(%0)\n"
- " .long 3b - .\n"
- " lda $31,5b-3b(%0)\n"
- " .long 4b - .\n"
- " lda $31,5b-4b(%0)\n"
- ".previous"
+ EXC(1b,5b,%2,%0)
+ EXC(2b,5b,%1,%0)
+ EXC(3b,5b,$31,%0)
+ EXC(4b,5b,$31,%0)
: "=r"(error), "=&r"(tmp1), "=&r"(tmp2),
"=&r"(tmp3), "=&r"(tmp4)
: "r"(va), "r"(*reg_addr), "0"(0));
@@ -1047,7 +979,7 @@
/* We need to replicate some of the logic in mm/fault.c,
since we don't have access to the fault code in the
exception handling return path. */
- if (!__access_ok((unsigned long)va, 0, USER_DS))
+ if ((unsigned long)va >= TASK_SIZE)
info.si_code = SEGV_ACCERR;
else {
struct mm_struct *mm = current->mm;
diff --git a/arch/alpha/lib/clear_user.S b/arch/alpha/lib/clear_user.S
index bf5b931..006f469 100644
--- a/arch/alpha/lib/clear_user.S
+++ b/arch/alpha/lib/clear_user.S
@@ -8,21 +8,6 @@
* right "bytes left to zero" value (and that it is updated only _after_
* a successful copy). There is also some rather minor exception setup
* stuff.
- *
- * NOTE! This is not directly C-callable, because the calling semantics
- * are different:
- *
- * Inputs:
- * length in $0
- * destination address in $6
- * exception pointer in $7
- * return address in $28 (exceptions expect it there)
- *
- * Outputs:
- * bytes left to copy in $0
- *
- * Clobbers:
- * $1,$2,$3,$4,$5,$6
*/
#include <asm/export.h>
@@ -38,62 +23,63 @@
.set noreorder
.align 4
- .globl __do_clear_user
- .ent __do_clear_user
- .frame $30, 0, $28
+ .globl __clear_user
+ .ent __clear_user
+ .frame $30, 0, $26
.prologue 0
$loop:
and $1, 3, $4 # e0 :
beq $4, 1f # .. e1 :
-0: EX( stq_u $31, 0($6) ) # e0 : zero one word
+0: EX( stq_u $31, 0($16) ) # e0 : zero one word
subq $0, 8, $0 # .. e1 :
subq $4, 1, $4 # e0 :
- addq $6, 8, $6 # .. e1 :
+ addq $16, 8, $16 # .. e1 :
bne $4, 0b # e1 :
unop # :
1: bic $1, 3, $1 # e0 :
beq $1, $tail # .. e1 :
-2: EX( stq_u $31, 0($6) ) # e0 : zero four words
+2: EX( stq_u $31, 0($16) ) # e0 : zero four words
subq $0, 8, $0 # .. e1 :
- EX( stq_u $31, 8($6) ) # e0 :
+ EX( stq_u $31, 8($16) ) # e0 :
subq $0, 8, $0 # .. e1 :
- EX( stq_u $31, 16($6) ) # e0 :
+ EX( stq_u $31, 16($16) ) # e0 :
subq $0, 8, $0 # .. e1 :
- EX( stq_u $31, 24($6) ) # e0 :
+ EX( stq_u $31, 24($16) ) # e0 :
subq $0, 8, $0 # .. e1 :
subq $1, 4, $1 # e0 :
- addq $6, 32, $6 # .. e1 :
+ addq $16, 32, $16 # .. e1 :
bne $1, 2b # e1 :
$tail:
bne $2, 1f # e1 : is there a tail to do?
- ret $31, ($28), 1 # .. e1 :
+ ret $31, ($26), 1 # .. e1 :
-1: EX( ldq_u $5, 0($6) ) # e0 :
+1: EX( ldq_u $5, 0($16) ) # e0 :
clr $0 # .. e1 :
nop # e1 :
mskqh $5, $0, $5 # e0 :
- EX( stq_u $5, 0($6) ) # e0 :
- ret $31, ($28), 1 # .. e1 :
+ EX( stq_u $5, 0($16) ) # e0 :
+ ret $31, ($26), 1 # .. e1 :
-__do_clear_user:
- and $6, 7, $4 # e0 : find dest misalignment
+__clear_user:
+ and $17, $17, $0
+ and $16, 7, $4 # e0 : find dest misalignment
beq $0, $zerolength # .. e1 :
addq $0, $4, $1 # e0 : bias counter
and $1, 7, $2 # e1 : number of bytes in tail
srl $1, 3, $1 # e0 :
beq $4, $loop # .. e1 :
- EX( ldq_u $5, 0($6) ) # e0 : load dst word to mask back in
+ EX( ldq_u $5, 0($16) ) # e0 : load dst word to mask back in
beq $1, $oneword # .. e1 : sub-word store?
- mskql $5, $6, $5 # e0 : take care of misaligned head
- addq $6, 8, $6 # .. e1 :
- EX( stq_u $5, -8($6) ) # e0 :
+ mskql $5, $16, $5 # e0 : take care of misaligned head
+ addq $16, 8, $16 # .. e1 :
+ EX( stq_u $5, -8($16) ) # e0 :
addq $0, $4, $0 # .. e1 : bytes left -= 8 - misalignment
subq $1, 1, $1 # e0 :
subq $0, 8, $0 # .. e1 :
@@ -101,15 +87,15 @@
unop # :
$oneword:
- mskql $5, $6, $4 # e0 :
+ mskql $5, $16, $4 # e0 :
mskqh $5, $2, $5 # e0 :
or $5, $4, $5 # e1 :
- EX( stq_u $5, 0($6) ) # e0 :
+ EX( stq_u $5, 0($16) ) # e0 :
clr $0 # .. e1 :
$zerolength:
$exception:
- ret $31, ($28), 1 # .. e1 :
+ ret $31, ($26), 1 # .. e1 :
- .end __do_clear_user
- EXPORT_SYMBOL(__do_clear_user)
+ .end __clear_user
+ EXPORT_SYMBOL(__clear_user)
diff --git a/arch/alpha/lib/copy_user.S b/arch/alpha/lib/copy_user.S
index 509f62b..159f1b7 100644
--- a/arch/alpha/lib/copy_user.S
+++ b/arch/alpha/lib/copy_user.S
@@ -9,21 +9,6 @@
* contains the right "bytes left to copy" value (and that it is updated
* only _after_ a successful copy). There is also some rather minor
* exception setup stuff..
- *
- * NOTE! This is not directly C-callable, because the calling semantics are
- * different:
- *
- * Inputs:
- * length in $0
- * destination address in $6
- * source address in $7
- * return address in $28
- *
- * Outputs:
- * bytes left to copy in $0
- *
- * Clobbers:
- * $1,$2,$3,$4,$5,$6,$7
*/
#include <asm/export.h>
@@ -49,58 +34,59 @@
.ent __copy_user
__copy_user:
.prologue 0
- and $6,7,$3
+ and $18,$18,$0
+ and $16,7,$3
beq $0,$35
beq $3,$36
subq $3,8,$3
.align 4
$37:
- EXI( ldq_u $1,0($7) )
- EXO( ldq_u $2,0($6) )
- extbl $1,$7,$1
- mskbl $2,$6,$2
- insbl $1,$6,$1
+ EXI( ldq_u $1,0($17) )
+ EXO( ldq_u $2,0($16) )
+ extbl $1,$17,$1
+ mskbl $2,$16,$2
+ insbl $1,$16,$1
addq $3,1,$3
bis $1,$2,$1
- EXO( stq_u $1,0($6) )
+ EXO( stq_u $1,0($16) )
subq $0,1,$0
- addq $6,1,$6
- addq $7,1,$7
+ addq $16,1,$16
+ addq $17,1,$17
beq $0,$41
bne $3,$37
$36:
- and $7,7,$1
+ and $17,7,$1
bic $0,7,$4
beq $1,$43
beq $4,$48
- EXI( ldq_u $3,0($7) )
+ EXI( ldq_u $3,0($17) )
.align 4
$50:
- EXI( ldq_u $2,8($7) )
+ EXI( ldq_u $2,8($17) )
subq $4,8,$4
- extql $3,$7,$3
- extqh $2,$7,$1
+ extql $3,$17,$3
+ extqh $2,$17,$1
bis $3,$1,$1
- EXO( stq $1,0($6) )
- addq $7,8,$7
+ EXO( stq $1,0($16) )
+ addq $17,8,$17
subq $0,8,$0
- addq $6,8,$6
+ addq $16,8,$16
bis $2,$2,$3
bne $4,$50
$48:
beq $0,$41
.align 4
$57:
- EXI( ldq_u $1,0($7) )
- EXO( ldq_u $2,0($6) )
- extbl $1,$7,$1
- mskbl $2,$6,$2
- insbl $1,$6,$1
+ EXI( ldq_u $1,0($17) )
+ EXO( ldq_u $2,0($16) )
+ extbl $1,$17,$1
+ mskbl $2,$16,$2
+ insbl $1,$16,$1
bis $1,$2,$1
- EXO( stq_u $1,0($6) )
+ EXO( stq_u $1,0($16) )
subq $0,1,$0
- addq $6,1,$6
- addq $7,1,$7
+ addq $16,1,$16
+ addq $17,1,$17
bne $0,$57
br $31,$41
.align 4
@@ -108,27 +94,27 @@
beq $4,$65
.align 4
$66:
- EXI( ldq $1,0($7) )
+ EXI( ldq $1,0($17) )
subq $4,8,$4
- EXO( stq $1,0($6) )
- addq $7,8,$7
+ EXO( stq $1,0($16) )
+ addq $17,8,$17
subq $0,8,$0
- addq $6,8,$6
+ addq $16,8,$16
bne $4,$66
$65:
beq $0,$41
- EXI( ldq $2,0($7) )
- EXO( ldq $1,0($6) )
+ EXI( ldq $2,0($17) )
+ EXO( ldq $1,0($16) )
mskql $2,$0,$2
mskqh $1,$0,$1
bis $2,$1,$2
- EXO( stq $2,0($6) )
+ EXO( stq $2,0($16) )
bis $31,$31,$0
$41:
$35:
$exitin:
$exitout:
- ret $31,($28),1
+ ret $31,($26),1
.end __copy_user
EXPORT_SYMBOL(__copy_user)
diff --git a/arch/alpha/lib/csum_partial_copy.c b/arch/alpha/lib/csum_partial_copy.c
index 5dfb797..ab42afb 100644
--- a/arch/alpha/lib/csum_partial_copy.c
+++ b/arch/alpha/lib/csum_partial_copy.c
@@ -45,10 +45,7 @@
__asm__ __volatile__( \
"1: ldq_u %0,%2\n" \
"2:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - .\n" \
- " lda %0,2b-1b(%1)\n" \
- ".previous" \
+ EXC(1b,2b,%0,%1) \
: "=r"(x), "=r"(__guu_err) \
: "m"(__m(ptr)), "1"(0)); \
__guu_err; \
@@ -60,10 +57,7 @@
__asm__ __volatile__( \
"1: stq_u %2,%1\n" \
"2:\n" \
- ".section __ex_table,\"a\"\n" \
- " .long 1b - ." \
- " lda $31,2b-1b(%0)\n" \
- ".previous" \
+ EXC(1b,2b,$31,%0) \
: "=r"(__puu_err) \
: "m"(__m(addr)), "rJ"(x), "0"(0)); \
__puu_err; \
diff --git a/arch/alpha/lib/ev6-clear_user.S b/arch/alpha/lib/ev6-clear_user.S
index 05bef6b..e179e47 100644
--- a/arch/alpha/lib/ev6-clear_user.S
+++ b/arch/alpha/lib/ev6-clear_user.S
@@ -9,21 +9,6 @@
* a successful copy). There is also some rather minor exception setup
* stuff.
*
- * NOTE! This is not directly C-callable, because the calling semantics
- * are different:
- *
- * Inputs:
- * length in $0
- * destination address in $6
- * exception pointer in $7
- * return address in $28 (exceptions expect it there)
- *
- * Outputs:
- * bytes left to copy in $0
- *
- * Clobbers:
- * $1,$2,$3,$4,$5,$6
- *
* Much of the information about 21264 scheduling/coding comes from:
* Compiler Writer's Guide for the Alpha 21264
* abbreviated as 'CWG' in other comments here
@@ -56,14 +41,15 @@
.set noreorder
.align 4
- .globl __do_clear_user
- .ent __do_clear_user
- .frame $30, 0, $28
+ .globl __clear_user
+ .ent __clear_user
+ .frame $30, 0, $26
.prologue 0
# Pipeline info : Slotting & Comments
-__do_clear_user:
- and $6, 7, $4 # .. E .. .. : find dest head misalignment
+__clear_user:
+ and $17, $17, $0
+ and $16, 7, $4 # .. E .. .. : find dest head misalignment
beq $0, $zerolength # U .. .. .. : U L U L
addq $0, $4, $1 # .. .. .. E : bias counter
@@ -75,14 +61,14 @@
/*
* Head is not aligned. Write (8 - $4) bytes to head of destination
- * This means $6 is known to be misaligned
+ * This means $16 is known to be misaligned
*/
- EX( ldq_u $5, 0($6) ) # .. .. .. L : load dst word to mask back in
+ EX( ldq_u $5, 0($16) ) # .. .. .. L : load dst word to mask back in
beq $1, $onebyte # .. .. U .. : sub-word store?
- mskql $5, $6, $5 # .. U .. .. : take care of misaligned head
- addq $6, 8, $6 # E .. .. .. : L U U L
+ mskql $5, $16, $5 # .. U .. .. : take care of misaligned head
+ addq $16, 8, $16 # E .. .. .. : L U U L
- EX( stq_u $5, -8($6) ) # .. .. .. L :
+ EX( stq_u $5, -8($16) ) # .. .. .. L :
subq $1, 1, $1 # .. .. E .. :
addq $0, $4, $0 # .. E .. .. : bytes left -= 8 - misalignment
subq $0, 8, $0 # E .. .. .. : U L U L
@@ -93,11 +79,11 @@
* values upon initial entry to the loop
* $1 is number of quadwords to clear (zero is a valid value)
* $2 is number of trailing bytes (0..7) ($2 never used...)
- * $6 is known to be aligned 0mod8
+ * $16 is known to be aligned 0mod8
*/
$headalign:
subq $1, 16, $4 # .. .. .. E : If < 16, we can not use the huge loop
- and $6, 0x3f, $2 # .. .. E .. : Forward work for huge loop
+ and $16, 0x3f, $2 # .. .. E .. : Forward work for huge loop
subq $2, 0x40, $3 # .. E .. .. : bias counter (huge loop)
blt $4, $trailquad # U .. .. .. : U L U L
@@ -114,21 +100,21 @@
beq $3, $bigalign # U .. .. .. : U L U L : Aligned 0mod64
$alignmod64:
- EX( stq_u $31, 0($6) ) # .. .. .. L
+ EX( stq_u $31, 0($16) ) # .. .. .. L
addq $3, 8, $3 # .. .. E ..
subq $0, 8, $0 # .. E .. ..
nop # E .. .. .. : U L U L
nop # .. .. .. E
subq $1, 1, $1 # .. .. E ..
- addq $6, 8, $6 # .. E .. ..
+ addq $16, 8, $16 # .. E .. ..
blt $3, $alignmod64 # U .. .. .. : U L U L
$bigalign:
/*
* $0 is the number of bytes left
* $1 is the number of quads left
- * $6 is aligned 0mod64
+ * $16 is aligned 0mod64
* we know that we'll be taking a minimum of one trip through
* CWG Section 3.7.6: do not expect a sustained store rate of > 1/cycle
* We are _not_ going to update $0 after every single store. That
@@ -145,39 +131,39 @@
nop # E :
nop # E :
nop # E :
- bis $6,$6,$3 # E : U L U L : Initial wh64 address is dest
+ bis $16,$16,$3 # E : U L U L : Initial wh64 address is dest
/* This might actually help for the current trip... */
$do_wh64:
wh64 ($3) # .. .. .. L1 : memory subsystem hint
subq $1, 16, $4 # .. .. E .. : Forward calculation - repeat the loop?
- EX( stq_u $31, 0($6) ) # .. L .. ..
+ EX( stq_u $31, 0($16) ) # .. L .. ..
subq $0, 8, $0 # E .. .. .. : U L U L
- addq $6, 128, $3 # E : Target address of wh64
- EX( stq_u $31, 8($6) ) # L :
- EX( stq_u $31, 16($6) ) # L :
+ addq $16, 128, $3 # E : Target address of wh64
+ EX( stq_u $31, 8($16) ) # L :
+ EX( stq_u $31, 16($16) ) # L :
subq $0, 16, $0 # E : U L L U
nop # E :
- EX( stq_u $31, 24($6) ) # L :
- EX( stq_u $31, 32($6) ) # L :
+ EX( stq_u $31, 24($16) ) # L :
+ EX( stq_u $31, 32($16) ) # L :
subq $0, 168, $5 # E : U L L U : two trips through the loop left?
/* 168 = 192 - 24, since we've already completed some stores */
subq $0, 16, $0 # E :
- EX( stq_u $31, 40($6) ) # L :
- EX( stq_u $31, 48($6) ) # L :
- cmovlt $5, $6, $3 # E : U L L U : Latency 2, extra mapping cycle
+ EX( stq_u $31, 40($16) ) # L :
+ EX( stq_u $31, 48($16) ) # L :
+ cmovlt $5, $16, $3 # E : U L L U : Latency 2, extra mapping cycle
subq $1, 8, $1 # E :
subq $0, 16, $0 # E :
- EX( stq_u $31, 56($6) ) # L :
+ EX( stq_u $31, 56($16) ) # L :
nop # E : U L U L
nop # E :
subq $0, 8, $0 # E :
- addq $6, 64, $6 # E :
+ addq $16, 64, $16 # E :
bge $4, $do_wh64 # U : U L U L
$trailquad:
@@ -190,14 +176,14 @@
beq $1, $trailbytes # U .. .. .. : U L U L : Only 0..7 bytes to go
$onequad:
- EX( stq_u $31, 0($6) ) # .. .. .. L
+ EX( stq_u $31, 0($16) ) # .. .. .. L
subq $1, 1, $1 # .. .. E ..
subq $0, 8, $0 # .. E .. ..
nop # E .. .. .. : U L U L
nop # .. .. .. E
nop # .. .. E ..
- addq $6, 8, $6 # .. E .. ..
+ addq $16, 8, $16 # .. E .. ..
bgt $1, $onequad # U .. .. .. : U L U L
# We have an unknown number of bytes left to go.
@@ -211,9 +197,9 @@
# so we will use $0 as the loop counter
# We know for a fact that $0 > 0 zero due to previous context
$onebyte:
- EX( stb $31, 0($6) ) # .. .. .. L
+ EX( stb $31, 0($16) ) # .. .. .. L
subq $0, 1, $0 # .. .. E .. :
- addq $6, 1, $6 # .. E .. .. :
+ addq $16, 1, $16 # .. E .. .. :
bgt $0, $onebyte # U .. .. .. : U L U L
$zerolength:
@@ -221,6 +207,6 @@
nop # .. .. .. E :
nop # .. .. E .. :
nop # .. E .. .. :
- ret $31, ($28), 1 # L0 .. .. .. : L U L U
- .end __do_clear_user
- EXPORT_SYMBOL(__do_clear_user)
+ ret $31, ($26), 1 # L0 .. .. .. : L U L U
+ .end __clear_user
+ EXPORT_SYMBOL(__clear_user)
diff --git a/arch/alpha/lib/ev6-copy_user.S b/arch/alpha/lib/ev6-copy_user.S
index be720b5..35e6710 100644
--- a/arch/alpha/lib/ev6-copy_user.S
+++ b/arch/alpha/lib/ev6-copy_user.S
@@ -12,21 +12,6 @@
* only _after_ a successful copy). There is also some rather minor
* exception setup stuff..
*
- * NOTE! This is not directly C-callable, because the calling semantics are
- * different:
- *
- * Inputs:
- * length in $0
- * destination address in $6
- * source address in $7
- * return address in $28
- *
- * Outputs:
- * bytes left to copy in $0
- *
- * Clobbers:
- * $1,$2,$3,$4,$5,$6,$7
- *
* Much of the information about 21264 scheduling/coding comes from:
* Compiler Writer's Guide for the Alpha 21264
* abbreviated as 'CWG' in other comments here
@@ -60,10 +45,11 @@
# Pipeline info: Slotting & Comments
__copy_user:
.prologue 0
- subq $0, 32, $1 # .. E .. .. : Is this going to be a small copy?
+ andq $18, $18, $0
+ subq $18, 32, $1 # .. E .. .. : Is this going to be a small copy?
beq $0, $zerolength # U .. .. .. : U L U L
- and $6,7,$3 # .. .. .. E : is leading dest misalignment
+ and $16,7,$3 # .. .. .. E : is leading dest misalignment
ble $1, $onebyteloop # .. .. U .. : 1st branch : small amount of data
beq $3, $destaligned # .. U .. .. : 2nd (one cycle fetcher stall)
subq $3, 8, $3 # E .. .. .. : L U U L : trip counter
@@ -73,17 +59,17 @@
* We know we have at least one trip through this loop
*/
$aligndest:
- EXI( ldbu $1,0($7) ) # .. .. .. L : Keep loads separate from stores
- addq $6,1,$6 # .. .. E .. : Section 3.8 in the CWG
+ EXI( ldbu $1,0($17) ) # .. .. .. L : Keep loads separate from stores
+ addq $16,1,$16 # .. .. E .. : Section 3.8 in the CWG
addq $3,1,$3 # .. E .. .. :
nop # E .. .. .. : U L U L
/*
- * the -1 is to compensate for the inc($6) done in a previous quadpack
+ * the -1 is to compensate for the inc($16) done in a previous quadpack
* which allows us zero dependencies within either quadpack in the loop
*/
- EXO( stb $1,-1($6) ) # .. .. .. L :
- addq $7,1,$7 # .. .. E .. : Section 3.8 in the CWG
+ EXO( stb $1,-1($16) ) # .. .. .. L :
+ addq $17,1,$17 # .. .. E .. : Section 3.8 in the CWG
subq $0,1,$0 # .. E .. .. :
bne $3, $aligndest # U .. .. .. : U L U L
@@ -92,29 +78,29 @@
* If we arrived via branch, we have a minimum of 32 bytes
*/
$destaligned:
- and $7,7,$1 # .. .. .. E : Check _current_ source alignment
+ and $17,7,$1 # .. .. .. E : Check _current_ source alignment
bic $0,7,$4 # .. .. E .. : number bytes as a quadword loop
- EXI( ldq_u $3,0($7) ) # .. L .. .. : Forward fetch for fallthrough code
+ EXI( ldq_u $3,0($17) ) # .. L .. .. : Forward fetch for fallthrough code
beq $1,$quadaligned # U .. .. .. : U L U L
/*
- * In the worst case, we've just executed an ldq_u here from 0($7)
+ * In the worst case, we've just executed an ldq_u here from 0($17)
* and we'll repeat it once if we take the branch
*/
/* Misaligned quadword loop - not unrolled. Leave it that way. */
$misquad:
- EXI( ldq_u $2,8($7) ) # .. .. .. L :
+ EXI( ldq_u $2,8($17) ) # .. .. .. L :
subq $4,8,$4 # .. .. E .. :
- extql $3,$7,$3 # .. U .. .. :
- extqh $2,$7,$1 # U .. .. .. : U U L L
+ extql $3,$17,$3 # .. U .. .. :
+ extqh $2,$17,$1 # U .. .. .. : U U L L
bis $3,$1,$1 # .. .. .. E :
- EXO( stq $1,0($6) ) # .. .. L .. :
- addq $7,8,$7 # .. E .. .. :
+ EXO( stq $1,0($16) ) # .. .. L .. :
+ addq $17,8,$17 # .. E .. .. :
subq $0,8,$0 # E .. .. .. : U L L U
- addq $6,8,$6 # .. .. .. E :
+ addq $16,8,$16 # .. .. .. E :
bis $2,$2,$3 # .. .. E .. :
nop # .. E .. .. :
bne $4,$misquad # U .. .. .. : U L U L
@@ -125,8 +111,8 @@
beq $0,$zerolength # U .. .. .. : U L U L
/* We know we have at least one trip through the byte loop */
- EXI ( ldbu $2,0($7) ) # .. .. .. L : No loads in the same quad
- addq $6,1,$6 # .. .. E .. : as the store (Section 3.8 in CWG)
+ EXI ( ldbu $2,0($17) ) # .. .. .. L : No loads in the same quad
+ addq $16,1,$16 # .. .. E .. : as the store (Section 3.8 in CWG)
nop # .. E .. .. :
br $31, $dirtyentry # L0 .. .. .. : L U U L
/* Do the trailing byte loop load, then hop into the store part of the loop */
@@ -136,8 +122,8 @@
* Based upon the usage context, it's worth the effort to unroll this loop
* $0 - number of bytes to be moved
* $4 - number of bytes to move as quadwords
- * $6 is current destination address
- * $7 is current source address
+ * $16 is current destination address
+ * $17 is current source address
*/
$quadaligned:
subq $4, 32, $2 # .. .. .. E : do not unroll for small stuff
@@ -155,29 +141,29 @@
* instruction memory hint instruction).
*/
$unroll4:
- EXI( ldq $1,0($7) ) # .. .. .. L
- EXI( ldq $2,8($7) ) # .. .. L ..
+ EXI( ldq $1,0($17) ) # .. .. .. L
+ EXI( ldq $2,8($17) ) # .. .. L ..
subq $4,32,$4 # .. E .. ..
nop # E .. .. .. : U U L L
- addq $7,16,$7 # .. .. .. E
- EXO( stq $1,0($6) ) # .. .. L ..
- EXO( stq $2,8($6) ) # .. L .. ..
+ addq $17,16,$17 # .. .. .. E
+ EXO( stq $1,0($16) ) # .. .. L ..
+ EXO( stq $2,8($16) ) # .. L .. ..
subq $0,16,$0 # E .. .. .. : U L L U
- addq $6,16,$6 # .. .. .. E
- EXI( ldq $1,0($7) ) # .. .. L ..
- EXI( ldq $2,8($7) ) # .. L .. ..
+ addq $16,16,$16 # .. .. .. E
+ EXI( ldq $1,0($17) ) # .. .. L ..
+ EXI( ldq $2,8($17) ) # .. L .. ..
subq $4, 32, $3 # E .. .. .. : U U L L : is there enough for another trip?
- EXO( stq $1,0($6) ) # .. .. .. L
- EXO( stq $2,8($6) ) # .. .. L ..
+ EXO( stq $1,0($16) ) # .. .. .. L
+ EXO( stq $2,8($16) ) # .. .. L ..
subq $0,16,$0 # .. E .. ..
- addq $7,16,$7 # E .. .. .. : U L L U
+ addq $17,16,$17 # E .. .. .. : U L L U
nop # .. .. .. E
nop # .. .. E ..
- addq $6,16,$6 # .. E .. ..
+ addq $16,16,$16 # .. E .. ..
bgt $3,$unroll4 # U .. .. .. : U L U L
nop
@@ -186,14 +172,14 @@
beq $4, $noquads
$onequad:
- EXI( ldq $1,0($7) )
+ EXI( ldq $1,0($17) )
subq $4,8,$4
- addq $7,8,$7
+ addq $17,8,$17
nop
- EXO( stq $1,0($6) )
+ EXO( stq $1,0($16) )
subq $0,8,$0
- addq $6,8,$6
+ addq $16,8,$16
bne $4,$onequad
$noquads:
@@ -207,23 +193,23 @@
* There's no point in doing a lot of complex alignment calculations to try to
* to quadword stuff for a small amount of data.
* $0 - remaining number of bytes left to copy
- * $6 - current dest addr
- * $7 - current source addr
+ * $16 - current dest addr
+ * $17 - current source addr
*/
$onebyteloop:
- EXI ( ldbu $2,0($7) ) # .. .. .. L : No loads in the same quad
- addq $6,1,$6 # .. .. E .. : as the store (Section 3.8 in CWG)
+ EXI ( ldbu $2,0($17) ) # .. .. .. L : No loads in the same quad
+ addq $16,1,$16 # .. .. E .. : as the store (Section 3.8 in CWG)
nop # .. E .. .. :
nop # E .. .. .. : U L U L
$dirtyentry:
/*
- * the -1 is to compensate for the inc($6) done in a previous quadpack
+ * the -1 is to compensate for the inc($16) done in a previous quadpack
* which allows us zero dependencies within either quadpack in the loop
*/
- EXO ( stb $2,-1($6) ) # .. .. .. L :
- addq $7,1,$7 # .. .. E .. : quadpack as the load
+ EXO ( stb $2,-1($16) ) # .. .. .. L :
+ addq $17,1,$17 # .. .. E .. : quadpack as the load
subq $0,1,$0 # .. E .. .. : change count _after_ copy
bgt $0,$onebyteloop # U .. .. .. : U L U L
@@ -233,7 +219,7 @@
nop # .. .. .. E
nop # .. .. E ..
nop # .. E .. ..
- ret $31,($28),1 # L0 .. .. .. : L U L U
+ ret $31,($26),1 # L0 .. .. .. : L U L U
.end __copy_user
EXPORT_SYMBOL(__copy_user)
diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig
index c9f30f4..5d7fb3e 100644
--- a/arch/arc/Kconfig
+++ b/arch/arc/Kconfig
@@ -406,6 +406,14 @@
bool "Insn: div, divu, rem, remu"
default y
+config ARC_HAS_ACCL_REGS
+ bool "Reg Pair ACCL:ACCH (FPU and/or MPY > 6)"
+ default n
+ help
+ Depending on the configuration, CPU can contain accumulator reg-pair
+ (also referred to as r58:r59). These can also be used by gcc as GPR so
+ kernel needs to save/restore per process
+
endif # ISA_ARCV2
endmenu # "ARC CPU Configuration"
diff --git a/arch/arc/include/asm/Kbuild b/arch/arc/include/asm/Kbuild
index 63a0401..7bee4e4 100644
--- a/arch/arc/include/asm/Kbuild
+++ b/arch/arc/include/asm/Kbuild
@@ -6,6 +6,7 @@
generic-y += div64.h
generic-y += emergency-restart.h
generic-y += errno.h
+generic-y += extable.h
generic-y += fb.h
generic-y += fcntl.h
generic-y += ftrace.h
diff --git a/arch/arc/include/asm/atomic.h b/arch/arc/include/asm/atomic.h
index b65930a..54b54da 100644
--- a/arch/arc/include/asm/atomic.h
+++ b/arch/arc/include/asm/atomic.h
@@ -17,10 +17,11 @@
#include <asm/barrier.h>
#include <asm/smp.h>
+#define ATOMIC_INIT(i) { (i) }
+
#ifndef CONFIG_ARC_PLAT_EZNPS
#define atomic_read(v) READ_ONCE((v)->counter)
-#define ATOMIC_INIT(i) { (i) }
#ifdef CONFIG_ARC_HAS_LLSC
diff --git a/arch/arc/include/asm/entry-arcv2.h b/arch/arc/include/asm/entry-arcv2.h
index aee1a77..ac85380 100644
--- a/arch/arc/include/asm/entry-arcv2.h
+++ b/arch/arc/include/asm/entry-arcv2.h
@@ -16,6 +16,11 @@
;
; Now manually save: r12, sp, fp, gp, r25
+#ifdef CONFIG_ARC_HAS_ACCL_REGS
+ PUSH r59
+ PUSH r58
+#endif
+
PUSH r30
PUSH r12
@@ -75,6 +80,11 @@
POP r12
POP r30
+#ifdef CONFIG_ARC_HAS_ACCL_REGS
+ POP r58
+ POP r59
+#endif
+
.endm
/*------------------------------------------------------------------------*/
diff --git a/arch/arc/include/asm/ptrace.h b/arch/arc/include/asm/ptrace.h
index 47111d5..5297faa 100644
--- a/arch/arc/include/asm/ptrace.h
+++ b/arch/arc/include/asm/ptrace.h
@@ -86,6 +86,10 @@
unsigned long r12, r30;
+#ifdef CONFIG_ARC_HAS_ACCL_REGS
+ unsigned long r58, r59; /* ACCL/ACCH used by FPU / DSP MPY */
+#endif
+
/*------- Below list auto saved by h/w -----------*/
unsigned long r0, r1, r2, r3, r4, r5, r6, r7, r8, r9, r10, r11;
diff --git a/arch/arc/include/asm/uaccess.h b/arch/arc/include/asm/uaccess.h
index 41faf17..f35974e 100644
--- a/arch/arc/include/asm/uaccess.h
+++ b/arch/arc/include/asm/uaccess.h
@@ -24,12 +24,10 @@
#ifndef _ASM_ARC_UACCESS_H
#define _ASM_ARC_UACCESS_H
-#include <linux/sched.h>
-#include <asm/errno.h>
#include <linux/string.h> /* for generic string functions */
-#define __kernel_ok (segment_eq(get_fs(), KERNEL_DS))
+#define __kernel_ok (uaccess_kernel())
/*
* Algorithmically, for __user_ok() we want do:
@@ -170,7 +168,7 @@
static inline unsigned long
-__arc_copy_from_user(void *to, const void __user *from, unsigned long n)
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
long res = 0;
char val;
@@ -396,11 +394,8 @@
return res;
}
-extern unsigned long slowpath_copy_to_user(void __user *to, const void *from,
- unsigned long n);
-
static inline unsigned long
-__arc_copy_to_user(void __user *to, const void *from, unsigned long n)
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
{
long res = 0;
char val;
@@ -726,24 +721,20 @@
}
#ifndef CONFIG_CC_OPTIMIZE_FOR_SIZE
-#define __copy_from_user(t, f, n) __arc_copy_from_user(t, f, n)
-#define __copy_to_user(t, f, n) __arc_copy_to_user(t, f, n)
+
+#define INLINE_COPY_TO_USER
+#define INLINE_COPY_FROM_USER
+
#define __clear_user(d, n) __arc_clear_user(d, n)
#define __strncpy_from_user(d, s, n) __arc_strncpy_from_user(d, s, n)
#define __strnlen_user(s, n) __arc_strnlen_user(s, n)
#else
-extern long arc_copy_from_user_noinline(void *to, const void __user * from,
- unsigned long n);
-extern long arc_copy_to_user_noinline(void __user *to, const void *from,
- unsigned long n);
extern unsigned long arc_clear_user_noinline(void __user *to,
unsigned long n);
extern long arc_strncpy_from_user_noinline (char *dst, const char __user *src,
long count);
extern long arc_strnlen_user_noinline(const char __user *src, long n);
-#define __copy_from_user(t, f, n) arc_copy_from_user_noinline(t, f, n)
-#define __copy_to_user(t, f, n) arc_copy_to_user_noinline(t, f, n)
#define __clear_user(d, n) arc_clear_user_noinline(d, n)
#define __strncpy_from_user(d, s, n) arc_strncpy_from_user_noinline(d, s, n)
#define __strnlen_user(s, n) arc_strnlen_user_noinline(s, n)
@@ -752,6 +743,4 @@
#include <asm-generic/uaccess.h>
-extern int fixup_exception(struct pt_regs *regs);
-
#endif
diff --git a/arch/arc/kernel/setup.c b/arch/arc/kernel/setup.c
index fa62404..fc8211f 100644
--- a/arch/arc/kernel/setup.c
+++ b/arch/arc/kernel/setup.c
@@ -319,7 +319,8 @@
static void arc_chk_core_config(void)
{
struct cpuinfo_arc *cpu = &cpuinfo_arc700[smp_processor_id()];
- int fpu_enabled;
+ int saved = 0, present = 0;
+ char *opt_nm = NULL;;
if (!cpu->extn.timer0)
panic("Timer0 is not present!\n");
@@ -346,17 +347,28 @@
/*
* FP hardware/software config sanity
- * -If hardware contains DPFP, kernel needs to save/restore FPU state
+ * -If hardware present, kernel needs to save/restore FPU state
* -If not, it will crash trying to save/restore the non-existant regs
- *
- * (only DPDP checked since SP has no arch visible regs)
*/
- fpu_enabled = IS_ENABLED(CONFIG_ARC_FPU_SAVE_RESTORE);
- if (cpu->extn.fpu_dp && !fpu_enabled)
- pr_warn("CONFIG_ARC_FPU_SAVE_RESTORE needed for working apps\n");
- else if (!cpu->extn.fpu_dp && fpu_enabled)
- panic("FPU non-existent, disable CONFIG_ARC_FPU_SAVE_RESTORE\n");
+ if (is_isa_arcompact()) {
+ opt_nm = "CONFIG_ARC_FPU_SAVE_RESTORE";
+ saved = IS_ENABLED(CONFIG_ARC_FPU_SAVE_RESTORE);
+
+ /* only DPDP checked since SP has no arch visible regs */
+ present = cpu->extn.fpu_dp;
+ } else {
+ opt_nm = "CONFIG_ARC_HAS_ACCL_REGS";
+ saved = IS_ENABLED(CONFIG_ARC_HAS_ACCL_REGS);
+
+ /* Accumulator Low:High pair (r58:59) present if DSP MPY or FPU */
+ present = cpu->extn_mpy.dsp | cpu->extn.fpu_sp | cpu->extn.fpu_dp;
+ }
+
+ if (present && !saved)
+ pr_warn("Enable %s for working apps\n", opt_nm);
+ else if (!present && saved)
+ panic("Disable %s, hardware NOT present\n", opt_nm);
}
/*
diff --git a/arch/arc/mm/extable.c b/arch/arc/mm/extable.c
index c86906b..72125a3 100644
--- a/arch/arc/mm/extable.c
+++ b/arch/arc/mm/extable.c
@@ -28,20 +28,6 @@
#ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE
-long arc_copy_from_user_noinline(void *to, const void __user *from,
- unsigned long n)
-{
- return __arc_copy_from_user(to, from, n);
-}
-EXPORT_SYMBOL(arc_copy_from_user_noinline);
-
-long arc_copy_to_user_noinline(void __user *to, const void *from,
- unsigned long n)
-{
- return __arc_copy_to_user(to, from, n);
-}
-EXPORT_SYMBOL(arc_copy_to_user_noinline);
-
unsigned long arc_clear_user_noinline(void __user *to,
unsigned long n)
{
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 0d4e71b..8a7ab5e 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -41,7 +41,6 @@
select HARDIRQS_SW_RESEND
select HAVE_ARCH_AUDITSYSCALL if (AEABI && !OABI_COMPAT)
select HAVE_ARCH_BITREVERSE if (CPU_32v7M || CPU_32v7) && !CPU_32v6
- select HAVE_ARCH_HARDENED_USERCOPY
select HAVE_ARCH_JUMP_LABEL if !XIP_KERNEL && !CPU_ENDIAN_BE32 && MMU
select HAVE_ARCH_KGDB if !CPU_ENDIAN_BE32 && MMU
select HAVE_ARCH_MMAP_RND_BITS if MMU
diff --git a/arch/arm/boot/dts/aspeed-g4.dtsi b/arch/arm/boot/dts/aspeed-g4.dtsi
index 0b4932c..c79c937 100644
--- a/arch/arm/boot/dts/aspeed-g4.dtsi
+++ b/arch/arm/boot/dts/aspeed-g4.dtsi
@@ -42,18 +42,16 @@
};
mac0: ethernet@1e660000 {
- compatible = "faraday,ftgmac100";
+ compatible = "aspeed,ast2400-mac", "faraday,ftgmac100";
reg = <0x1e660000 0x180>;
interrupts = <2>;
- no-hw-checksum;
status = "disabled";
};
mac1: ethernet@1e680000 {
- compatible = "faraday,ftgmac100";
+ compatible = "aspeed,ast2400-mac", "faraday,ftgmac100";
reg = <0x1e680000 0x180>;
interrupts = <3>;
- no-hw-checksum;
status = "disabled";
};
diff --git a/arch/arm/boot/dts/aspeed-g5.dtsi b/arch/arm/boot/dts/aspeed-g5.dtsi
index b664fe3..b659663 100644
--- a/arch/arm/boot/dts/aspeed-g5.dtsi
+++ b/arch/arm/boot/dts/aspeed-g5.dtsi
@@ -33,18 +33,16 @@
};
mac0: ethernet@1e660000 {
- compatible = "faraday,ftgmac100";
+ compatible = "aspeed,ast2500-mac", "faraday,ftgmac100";
reg = <0x1e660000 0x180>;
interrupts = <2>;
- no-hw-checksum;
status = "disabled";
};
mac1: ethernet@1e680000 {
- compatible = "faraday,ftgmac100";
+ compatible = "aspeed,ast2500-mac", "faraday,ftgmac100";
reg = <0x1e680000 0x180>;
interrupts = <3>;
- no-hw-checksum;
status = "disabled";
};
diff --git a/arch/arm/boot/dts/rk1108.dtsi b/arch/arm/boot/dts/rk1108.dtsi
index d6194bf..6c8fc19 100644
--- a/arch/arm/boot/dts/rk1108.dtsi
+++ b/arch/arm/boot/dts/rk1108.dtsi
@@ -222,7 +222,7 @@
};
pinctrl: pinctrl {
- compatible = "rockchip,rk1108-pinctrl";
+ compatible = "rockchip,rv1108-pinctrl";
rockchip,grf = <&grf>;
rockchip,pmu = <&pmugrf>;
#address-cells = <1>;
diff --git a/arch/arm/boot/dts/rk3188.dtsi b/arch/arm/boot/dts/rk3188.dtsi
index cf91254..1aff4ad2 100644
--- a/arch/arm/boot/dts/rk3188.dtsi
+++ b/arch/arm/boot/dts/rk3188.dtsi
@@ -106,6 +106,22 @@
};
};
+ timer3: timer@2000e000 {
+ compatible = "rockchip,rk3188-timer", "rockchip,rk3288-timer";
+ reg = <0x2000e000 0x20>;
+ interrupts = <GIC_SPI 46 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&cru SCLK_TIMER3>, <&cru PCLK_TIMER3>;
+ clock-names = "timer", "pclk";
+ };
+
+ timer6: timer@200380a0 {
+ compatible = "rockchip,rk3188-timer", "rockchip,rk3288-timer";
+ reg = <0x200380a0 0x20>;
+ interrupts = <GIC_SPI 64 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&cru SCLK_TIMER6>, <&cru PCLK_TIMER0>;
+ clock-names = "timer", "pclk";
+ };
+
i2s0: i2s@1011a000 {
compatible = "rockchip,rk3188-i2s", "rockchip,rk3066-i2s";
reg = <0x1011a000 0x2000>;
@@ -530,6 +546,7 @@
&global_timer {
interrupts = <GIC_PPI 11 0xf04>;
+ status = "disabled";
};
&local_timer {
diff --git a/arch/arm/boot/dts/rk322x.dtsi b/arch/arm/boot/dts/rk322x.dtsi
index 9dff822..641607d 100644
--- a/arch/arm/boot/dts/rk322x.dtsi
+++ b/arch/arm/boot/dts/rk322x.dtsi
@@ -325,7 +325,7 @@
};
timer: timer@110c0000 {
- compatible = "rockchip,rk3288-timer";
+ compatible = "rockchip,rk3228-timer", "rockchip,rk3288-timer";
reg = <0x110c0000 0x20>;
interrupts = <GIC_SPI 43 IRQ_TYPE_LEVEL_HIGH>;
clocks = <&xin24m>, <&cru PCLK_TIMER>;
diff --git a/arch/arm/boot/dts/ste-dbx5x0.dtsi b/arch/arm/boot/dts/ste-dbx5x0.dtsi
index 162e1eb..6c5affe 100644
--- a/arch/arm/boot/dts/ste-dbx5x0.dtsi
+++ b/arch/arm/boot/dts/ste-dbx5x0.dtsi
@@ -1189,11 +1189,6 @@
status = "disabled";
};
- cpufreq-cooling {
- compatible = "stericsson,db8500-cpufreq-cooling";
- status = "disabled";
- };
-
mcde@a0350000 {
compatible = "stericsson,mcde";
reg = <0xa0350000 0x1000>, /* MCDE */
diff --git a/arch/arm/boot/dts/stm32746g-eval.dts b/arch/arm/boot/dts/stm32746g-eval.dts
index aa03fac..0dc18a0 100644
--- a/arch/arm/boot/dts/stm32746g-eval.dts
+++ b/arch/arm/boot/dts/stm32746g-eval.dts
@@ -89,6 +89,10 @@
clock-frequency = <25000000>;
};
+&crc {
+ status = "okay";
+};
+
&usart1 {
pinctrl-0 = <&usart1_pins_a>;
pinctrl-names = "default";
diff --git a/arch/arm/boot/dts/stm32f746.dtsi b/arch/arm/boot/dts/stm32f746.dtsi
index f321ffe..755fb92 100644
--- a/arch/arm/boot/dts/stm32f746.dtsi
+++ b/arch/arm/boot/dts/stm32f746.dtsi
@@ -289,6 +289,13 @@
};
};
+ crc: crc@40023000 {
+ compatible = "st,stm32f7-crc";
+ reg = <0x40023000 0x400>;
+ clocks = <&rcc 0 12>;
+ status = "disabled";
+ };
+
rcc: rcc@40023800 {
#clock-cells = <2>;
compatible = "st,stm32f42xx-rcc", "st,stm32-rcc";
diff --git a/arch/arm/configs/multi_v7_defconfig b/arch/arm/configs/multi_v7_defconfig
index a94126f..6aa7be1 100644
--- a/arch/arm/configs/multi_v7_defconfig
+++ b/arch/arm/configs/multi_v7_defconfig
@@ -748,7 +748,6 @@
CONFIG_LEDS_TRIGGER_TRANSIENT=y
CONFIG_LEDS_TRIGGER_CAMERA=y
CONFIG_EDAC=y
-CONFIG_EDAC_MM_EDAC=y
CONFIG_EDAC_HIGHBANK_MC=y
CONFIG_EDAC_HIGHBANK_L2=y
CONFIG_RTC_CLASS=y
diff --git a/arch/arm/configs/pxa_defconfig b/arch/arm/configs/pxa_defconfig
index 2aac99f..1318f61 100644
--- a/arch/arm/configs/pxa_defconfig
+++ b/arch/arm/configs/pxa_defconfig
@@ -635,8 +635,7 @@
CONFIG_LEDS_TRIGGER_DEFAULT_ON=m
CONFIG_LEDS_TRIGGER_TRANSIENT=m
CONFIG_LEDS_TRIGGER_CAMERA=m
-CONFIG_EDAC=y
-CONFIG_EDAC_MM_EDAC=m
+CONFIG_EDAC=m
CONFIG_RTC_CLASS=y
CONFIG_RTC_DEBUG=y
CONFIG_RTC_DRV_DS1307=m
diff --git a/arch/arm/configs/stm32_defconfig b/arch/arm/configs/stm32_defconfig
index a9d8e3c..03437f8 100644
--- a/arch/arm/configs/stm32_defconfig
+++ b/arch/arm/configs/stm32_defconfig
@@ -75,5 +75,7 @@
# CONFIG_SCHED_DEBUG is not set
# CONFIG_DEBUG_BUGVERBOSE is not set
# CONFIG_FTRACE is not set
+CONFIG_CRYPTO=y
+CONFIG_CRYPTO_DEV_STM32=y
CONFIG_CRC_ITU_T=y
CONFIG_CRC7=y
diff --git a/arch/arm/crypto/Kconfig b/arch/arm/crypto/Kconfig
index a8fce93..b9adedc 100644
--- a/arch/arm/crypto/Kconfig
+++ b/arch/arm/crypto/Kconfig
@@ -73,7 +73,7 @@
depends on KERNEL_MODE_NEON
select CRYPTO_BLKCIPHER
select CRYPTO_SIMD
- select CRYPTO_AES_ARM
+ select CRYPTO_AES
help
Use a faster and more secure NEON based implementation of AES in CBC,
CTR and XTS modes
diff --git a/arch/arm/crypto/aes-neonbs-glue.c b/arch/arm/crypto/aes-neonbs-glue.c
index 2920b96..c763779 100644
--- a/arch/arm/crypto/aes-neonbs-glue.c
+++ b/arch/arm/crypto/aes-neonbs-glue.c
@@ -42,9 +42,6 @@
asmlinkage void aesbs_xts_decrypt(u8 out[], u8 const in[], u8 const rk[],
int rounds, int blocks, u8 iv[]);
-asmlinkage void __aes_arm_encrypt(const u32 rk[], int rounds, const u8 in[],
- u8 out[]);
-
struct aesbs_ctx {
int rounds;
u8 rk[13 * (8 * AES_BLOCK_SIZE) + 32] __aligned(AES_BLOCK_SIZE);
@@ -52,12 +49,12 @@
struct aesbs_cbc_ctx {
struct aesbs_ctx key;
- u32 enc[AES_MAX_KEYLENGTH_U32];
+ struct crypto_cipher *enc_tfm;
};
struct aesbs_xts_ctx {
struct aesbs_ctx key;
- u32 twkey[AES_MAX_KEYLENGTH_U32];
+ struct crypto_cipher *tweak_tfm;
};
static int aesbs_setkey(struct crypto_skcipher *tfm, const u8 *in_key,
@@ -132,20 +129,18 @@
ctx->key.rounds = 6 + key_len / 4;
- memcpy(ctx->enc, rk.key_enc, sizeof(ctx->enc));
-
kernel_neon_begin();
aesbs_convert_key(ctx->key.rk, rk.key_enc, ctx->key.rounds);
kernel_neon_end();
- return 0;
+ return crypto_cipher_setkey(ctx->enc_tfm, in_key, key_len);
}
static void cbc_encrypt_one(struct crypto_skcipher *tfm, const u8 *src, u8 *dst)
{
struct aesbs_cbc_ctx *ctx = crypto_skcipher_ctx(tfm);
- __aes_arm_encrypt(ctx->enc, ctx->key.rounds, src, dst);
+ crypto_cipher_encrypt_one(ctx->enc_tfm, dst, src);
}
static int cbc_encrypt(struct skcipher_request *req)
@@ -181,6 +176,23 @@
return err;
}
+static int cbc_init(struct crypto_tfm *tfm)
+{
+ struct aesbs_cbc_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ ctx->enc_tfm = crypto_alloc_cipher("aes", 0, 0);
+ if (IS_ERR(ctx->enc_tfm))
+ return PTR_ERR(ctx->enc_tfm);
+ return 0;
+}
+
+static void cbc_exit(struct crypto_tfm *tfm)
+{
+ struct aesbs_cbc_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ crypto_free_cipher(ctx->enc_tfm);
+}
+
static int ctr_encrypt(struct skcipher_request *req)
{
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
@@ -228,7 +240,6 @@
unsigned int key_len)
{
struct aesbs_xts_ctx *ctx = crypto_skcipher_ctx(tfm);
- struct crypto_aes_ctx rk;
int err;
err = xts_verify_key(tfm, in_key, key_len);
@@ -236,15 +247,30 @@
return err;
key_len /= 2;
- err = crypto_aes_expand_key(&rk, in_key + key_len, key_len);
+ err = crypto_cipher_setkey(ctx->tweak_tfm, in_key + key_len, key_len);
if (err)
return err;
- memcpy(ctx->twkey, rk.key_enc, sizeof(ctx->twkey));
-
return aesbs_setkey(tfm, in_key, key_len);
}
+static int xts_init(struct crypto_tfm *tfm)
+{
+ struct aesbs_xts_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ ctx->tweak_tfm = crypto_alloc_cipher("aes", 0, 0);
+ if (IS_ERR(ctx->tweak_tfm))
+ return PTR_ERR(ctx->tweak_tfm);
+ return 0;
+}
+
+static void xts_exit(struct crypto_tfm *tfm)
+{
+ struct aesbs_xts_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ crypto_free_cipher(ctx->tweak_tfm);
+}
+
static int __xts_crypt(struct skcipher_request *req,
void (*fn)(u8 out[], u8 const in[], u8 const rk[],
int rounds, int blocks, u8 iv[]))
@@ -256,7 +282,7 @@
err = skcipher_walk_virt(&walk, req, true);
- __aes_arm_encrypt(ctx->twkey, ctx->key.rounds, walk.iv, walk.iv);
+ crypto_cipher_encrypt_one(ctx->tweak_tfm, walk.iv, walk.iv);
kernel_neon_begin();
while (walk.nbytes >= AES_BLOCK_SIZE) {
@@ -309,6 +335,8 @@
.base.cra_ctxsize = sizeof(struct aesbs_cbc_ctx),
.base.cra_module = THIS_MODULE,
.base.cra_flags = CRYPTO_ALG_INTERNAL,
+ .base.cra_init = cbc_init,
+ .base.cra_exit = cbc_exit,
.min_keysize = AES_MIN_KEY_SIZE,
.max_keysize = AES_MAX_KEY_SIZE,
@@ -342,6 +370,8 @@
.base.cra_ctxsize = sizeof(struct aesbs_xts_ctx),
.base.cra_module = THIS_MODULE,
.base.cra_flags = CRYPTO_ALG_INTERNAL,
+ .base.cra_init = xts_init,
+ .base.cra_exit = xts_exit,
.min_keysize = 2 * AES_MIN_KEY_SIZE,
.max_keysize = 2 * AES_MAX_KEY_SIZE,
@@ -402,5 +432,5 @@
return err;
}
-module_init(aes_init);
+late_initcall(aes_init);
module_exit(aes_exit);
diff --git a/arch/arm/include/asm/Kbuild b/arch/arm/include/asm/Kbuild
index b14e8c7..3a36d99 100644
--- a/arch/arm/include/asm/Kbuild
+++ b/arch/arm/include/asm/Kbuild
@@ -7,6 +7,7 @@
generic-y += emergency-restart.h
generic-y += errno.h
generic-y += exec.h
+generic-y += extable.h
generic-y += ioctl.h
generic-y += ipcbuf.h
generic-y += irq_regs.h
diff --git a/arch/arm/include/asm/efi.h b/arch/arm/include/asm/efi.h
index e4e6a9d6..17f1f1a 100644
--- a/arch/arm/include/asm/efi.h
+++ b/arch/arm/include/asm/efi.h
@@ -85,6 +85,18 @@
*/
#define ZIMAGE_OFFSET_LIMIT SZ_128M
#define MIN_ZIMAGE_OFFSET MAX_UNCOMP_KERNEL_SIZE
-#define MAX_FDT_OFFSET ZIMAGE_OFFSET_LIMIT
+
+/* on ARM, the FDT should be located in the first 128 MB of RAM */
+static inline unsigned long efi_get_max_fdt_addr(unsigned long dram_base)
+{
+ return dram_base + ZIMAGE_OFFSET_LIMIT;
+}
+
+/* on ARM, the initrd should be loaded in a lowmem region */
+static inline unsigned long efi_get_max_initrd_addr(unsigned long dram_base,
+ unsigned long image_addr)
+{
+ return dram_base + SZ_512M;
+}
#endif /* _ASM_ARM_EFI_H */
diff --git a/arch/arm/include/asm/uaccess.h b/arch/arm/include/asm/uaccess.h
index b7e0125..2577405 100644
--- a/arch/arm/include/asm/uaccess.h
+++ b/arch/arm/include/asm/uaccess.h
@@ -12,8 +12,6 @@
* User space memory access functions
*/
#include <linux/string.h>
-#include <linux/thread_info.h>
-#include <asm/errno.h>
#include <asm/memory.h>
#include <asm/domain.h>
#include <asm/unified.h>
@@ -26,28 +24,7 @@
#define __put_user_unaligned __put_user
#endif
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
-/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-
-struct exception_table_entry
-{
- unsigned long insn, fixup;
-};
-
-extern int fixup_exception(struct pt_regs *regs);
+#include <asm/extable.h>
/*
* These two functions allow hooking accesses to userspace to increase
@@ -271,7 +248,7 @@
#define access_ok(type, addr, size) (__range_ok(addr, size) == 0)
#define user_addr_max() \
- (segment_eq(get_fs(), KERNEL_DS) ? ~0UL : get_fs())
+ (uaccess_kernel() ? ~0UL : get_fs())
/*
* The "__xxx" versions of the user access functions do not verify the
@@ -478,7 +455,7 @@
arm_copy_from_user(void *to, const void __user *from, unsigned long n);
static inline unsigned long __must_check
-__arch_copy_from_user(void *to, const void __user *from, unsigned long n)
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
unsigned int __ua_flags;
@@ -494,7 +471,7 @@
__copy_to_user_std(void __user *to, const void *from, unsigned long n);
static inline unsigned long __must_check
-__arch_copy_to_user(void __user *to, const void *from, unsigned long n)
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
{
#ifndef CONFIG_UACCESS_WITH_MEMCPY
unsigned int __ua_flags;
@@ -522,54 +499,22 @@
}
#else
-#define __arch_copy_from_user(to, from, n) \
- (memcpy(to, (void __force *)from, n), 0)
-#define __arch_copy_to_user(to, from, n) \
- (memcpy((void __force *)to, from, n), 0)
+static inline unsigned long
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
+{
+ memcpy(to, (const void __force *)from, n);
+ return 0;
+}
+static inline unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
+{
+ memcpy((void __force *)to, from, n);
+ return 0;
+}
#define __clear_user(addr, n) (memset((void __force *)addr, 0, n), 0)
#endif
-
-static inline unsigned long __must_check
-__copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- check_object_size(to, n, false);
- return __arch_copy_from_user(to, from, n);
-}
-
-static inline unsigned long __must_check
-copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- unsigned long res = n;
-
- check_object_size(to, n, false);
-
- if (likely(access_ok(VERIFY_READ, from, n)))
- res = __arch_copy_from_user(to, from, n);
- if (unlikely(res))
- memset(to + (n - res), 0, res);
- return res;
-}
-
-static inline unsigned long __must_check
-__copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- check_object_size(from, n, true);
-
- return __arch_copy_to_user(to, from, n);
-}
-
-static inline unsigned long __must_check
-copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- check_object_size(from, n, true);
-
- if (access_ok(VERIFY_WRITE, to, n))
- n = __arch_copy_to_user(to, from, n);
- return n;
-}
-
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
+#define INLINE_COPY_TO_USER
+#define INLINE_COPY_FROM_USER
static inline unsigned long __must_check clear_user(void __user *to, unsigned long n)
{
diff --git a/arch/arm/kernel/vmlinux-xip.lds.S b/arch/arm/kernel/vmlinux-xip.lds.S
index 37b2a11..8265b11 100644
--- a/arch/arm/kernel/vmlinux-xip.lds.S
+++ b/arch/arm/kernel/vmlinux-xip.lds.S
@@ -242,6 +242,8 @@
}
_edata_loc = __data_loc + SIZEOF(.data);
+ BUG_TABLE
+
#ifdef CONFIG_HAVE_TCM
/*
* We align everything to a page boundary so we can
diff --git a/arch/arm/kernel/vmlinux.lds.S b/arch/arm/kernel/vmlinux.lds.S
index ce18007..c83a7ba 100644
--- a/arch/arm/kernel/vmlinux.lds.S
+++ b/arch/arm/kernel/vmlinux.lds.S
@@ -262,6 +262,8 @@
}
_edata_loc = __data_loc + SIZEOF(.data);
+ BUG_TABLE
+
#ifdef CONFIG_HAVE_TCM
/*
* We align everything to a page boundary so we can
diff --git a/arch/arm/lib/uaccess_with_memcpy.c b/arch/arm/lib/uaccess_with_memcpy.c
index 6bd1089..9b4ed17 100644
--- a/arch/arm/lib/uaccess_with_memcpy.c
+++ b/arch/arm/lib/uaccess_with_memcpy.c
@@ -90,7 +90,7 @@
unsigned long ua_flags;
int atomic;
- if (unlikely(segment_eq(get_fs(), KERNEL_DS))) {
+ if (uaccess_kernel()) {
memcpy((void *)to, from, n);
return 0;
}
@@ -162,7 +162,7 @@
{
unsigned long ua_flags;
- if (unlikely(segment_eq(get_fs(), KERNEL_DS))) {
+ if (uaccess_kernel()) {
memset((void *)addr, 0, n);
return 0;
}
diff --git a/arch/arm/mach-moxart/Kconfig b/arch/arm/mach-moxart/Kconfig
index f69e28b..70db2ab 100644
--- a/arch/arm/mach-moxart/Kconfig
+++ b/arch/arm/mach-moxart/Kconfig
@@ -3,8 +3,8 @@
depends on ARCH_MULTI_V4
select CPU_FA526
select ARM_DMA_MEM_BUFFERABLE
+ select FARADAY_FTINTC010
select MOXART_TIMER
- select GENERIC_IRQ_CHIP
select GPIOLIB
select PHYLIB if NETDEVICES
help
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 3741859..67695fad 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2,6 +2,7 @@
def_bool y
select ACPI_CCA_REQUIRED if ACPI
select ACPI_GENERIC_GSI if ACPI
+ select ACPI_GTDT if ACPI
select ACPI_REDUCED_HARDWARE_ONLY if ACPI
select ACPI_MCFG if ACPI
select ACPI_SPCR_TABLE if ACPI
@@ -60,7 +61,6 @@
select HAVE_ALIGNED_STRUCT_PAGE if SLUB
select HAVE_ARCH_AUDITSYSCALL
select HAVE_ARCH_BITREVERSE
- select HAVE_ARCH_HARDENED_USERCOPY
select HAVE_ARCH_HUGE_VMAP
select HAVE_ARCH_JUMP_LABEL
select HAVE_ARCH_KASAN if SPARSEMEM_VMEMMAP && !(ARM64_16K_PAGES && ARM64_VA_BITS_48)
diff --git a/arch/arm64/boot/dts/amlogic/meson-gx.dtsi b/arch/arm64/boot/dts/amlogic/meson-gx.dtsi
index 5d995f7..620495a 100644
--- a/arch/arm64/boot/dts/amlogic/meson-gx.dtsi
+++ b/arch/arm64/boot/dts/amlogic/meson-gx.dtsi
@@ -380,7 +380,7 @@
#size-cells = <2>;
ranges = <0x0 0x0 0x0 0xc8834000 0x0 0x2000>;
- rng {
+ hwrng: rng {
compatible = "amlogic,meson-rng";
reg = <0x0 0x0 0x0 0x4>;
};
diff --git a/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi b/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi
index 04b3324..a375cb2 100644
--- a/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi
+++ b/arch/arm64/boot/dts/amlogic/meson-gxbb.dtsi
@@ -524,3 +524,8 @@
&vpu {
compatible = "amlogic,meson-gxbb-vpu", "amlogic,meson-gx-vpu";
};
+
+&hwrng {
+ clocks = <&clkc CLKID_RNG0>;
+ clock-names = "core";
+};
diff --git a/arch/arm64/boot/dts/hisilicon/hi6220-hikey.dts b/arch/arm64/boot/dts/hisilicon/hi6220-hikey.dts
index dba3c13..9b4ba71 100644
--- a/arch/arm64/boot/dts/hisilicon/hi6220-hikey.dts
+++ b/arch/arm64/boot/dts/hisilicon/hi6220-hikey.dts
@@ -98,6 +98,11 @@
assigned-clocks = <&sys_ctrl HI6220_UART1_SRC>;
assigned-clock-rates = <150000000>;
status = "ok";
+
+ bluetooth {
+ compatible = "ti,wl1835-st";
+ enable-gpios = <&gpio1 7 GPIO_ACTIVE_HIGH>;
+ };
};
uart2: uart@f7112000 {
diff --git a/arch/arm64/boot/dts/marvell/armada-ap806.dtsi b/arch/arm64/boot/dts/marvell/armada-ap806.dtsi
index a749ba2..5019c8f4 100644
--- a/arch/arm64/boot/dts/marvell/armada-ap806.dtsi
+++ b/arch/arm64/boot/dts/marvell/armada-ap806.dtsi
@@ -235,7 +235,8 @@
#clock-cells = <1>;
clock-output-names = "ap-cpu-cluster-0",
"ap-cpu-cluster-1",
- "ap-fixed", "ap-mss";
+ "ap-fixed", "ap-mss",
+ "ap-emmc";
reg = <0x6f4000 0x1000>;
};
};
diff --git a/arch/arm64/include/asm/arch_timer.h b/arch/arm64/include/asm/arch_timer.h
index b4b3400..74d08e4 100644
--- a/arch/arm64/include/asm/arch_timer.h
+++ b/arch/arm64/include/asm/arch_timer.h
@@ -25,6 +25,7 @@
#include <linux/bug.h>
#include <linux/init.h>
#include <linux/jump_label.h>
+#include <linux/smp.h>
#include <linux/types.h>
#include <clocksource/arm_arch_timer.h>
@@ -37,24 +38,44 @@
#define needs_unstable_timer_counter_workaround() false
#endif
+enum arch_timer_erratum_match_type {
+ ate_match_dt,
+ ate_match_local_cap_id,
+ ate_match_acpi_oem_info,
+};
+
+struct clock_event_device;
struct arch_timer_erratum_workaround {
- const char *id; /* Indicate the Erratum ID */
+ enum arch_timer_erratum_match_type match_type;
+ const void *id;
+ const char *desc;
u32 (*read_cntp_tval_el0)(void);
u32 (*read_cntv_tval_el0)(void);
u64 (*read_cntvct_el0)(void);
+ int (*set_next_event_phys)(unsigned long, struct clock_event_device *);
+ int (*set_next_event_virt)(unsigned long, struct clock_event_device *);
};
-extern const struct arch_timer_erratum_workaround *timer_unstable_counter_workaround;
+DECLARE_PER_CPU(const struct arch_timer_erratum_workaround *,
+ timer_unstable_counter_workaround);
-#define arch_timer_reg_read_stable(reg) \
-({ \
- u64 _val; \
- if (needs_unstable_timer_counter_workaround()) \
- _val = timer_unstable_counter_workaround->read_##reg();\
- else \
- _val = read_sysreg(reg); \
- _val; \
+#define arch_timer_reg_read_stable(reg) \
+({ \
+ u64 _val; \
+ if (needs_unstable_timer_counter_workaround()) { \
+ const struct arch_timer_erratum_workaround *wa; \
+ preempt_disable(); \
+ wa = __this_cpu_read(timer_unstable_counter_workaround); \
+ if (wa && wa->read_##reg) \
+ _val = wa->read_##reg(); \
+ else \
+ _val = read_sysreg(reg); \
+ preempt_enable(); \
+ } else { \
+ _val = read_sysreg(reg); \
+ } \
+ _val; \
})
/*
diff --git a/arch/arm64/include/asm/bug.h b/arch/arm64/include/asm/bug.h
index 561190d..a9be107 100644
--- a/arch/arm64/include/asm/bug.h
+++ b/arch/arm64/include/asm/bug.h
@@ -55,7 +55,7 @@
unreachable(); \
} while (0)
-#define __WARN_TAINT(taint) _BUG_FLAGS(BUGFLAG_TAINT(taint))
+#define __WARN_FLAGS(flags) _BUG_FLAGS(BUGFLAG_WARNING|(flags))
#endif /* ! CONFIG_GENERIC_BUG */
diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index fb78a5d..b3aab8a 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -37,7 +37,8 @@
#define ARM64_HAS_NO_FPSIMD 16
#define ARM64_WORKAROUND_REPEAT_TLBI 17
#define ARM64_WORKAROUND_QCOM_FALKOR_E1003 18
+#define ARM64_WORKAROUND_858921 19
-#define ARM64_NCAPS 19
+#define ARM64_NCAPS 20
#endif /* __ASM_CPUCAPS_H */
diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h
index fc50271..0984d1b 100644
--- a/arch/arm64/include/asm/cputype.h
+++ b/arch/arm64/include/asm/cputype.h
@@ -80,6 +80,7 @@
#define ARM_CPU_PART_FOUNDATION 0xD00
#define ARM_CPU_PART_CORTEX_A57 0xD07
#define ARM_CPU_PART_CORTEX_A53 0xD03
+#define ARM_CPU_PART_CORTEX_A73 0xD09
#define APM_CPU_PART_POTENZA 0x000
@@ -92,6 +93,7 @@
#define MIDR_CORTEX_A53 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A53)
#define MIDR_CORTEX_A57 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A57)
+#define MIDR_CORTEX_A73 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A73)
#define MIDR_THUNDERX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX)
#define MIDR_THUNDERX_81XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_81XX)
#define MIDR_QCOM_FALKOR_V1 MIDR_CPU_MODEL(ARM_CPU_IMP_QCOM, QCOM_CPU_PART_FALKOR_V1)
diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h
index e744528..8f3043a 100644
--- a/arch/arm64/include/asm/efi.h
+++ b/arch/arm64/include/asm/efi.h
@@ -1,6 +1,7 @@
#ifndef _ASM_EFI_H
#define _ASM_EFI_H
+#include <asm/boot.h>
#include <asm/cpufeature.h>
#include <asm/io.h>
#include <asm/mmu_context.h>
@@ -46,7 +47,28 @@
* 2MiB so we know it won't cross a 2MiB boundary.
*/
#define EFI_FDT_ALIGN SZ_2M /* used by allocate_new_fdt_and_exit_boot() */
-#define MAX_FDT_OFFSET SZ_512M
+
+/* on arm64, the FDT may be located anywhere in system RAM */
+static inline unsigned long efi_get_max_fdt_addr(unsigned long dram_base)
+{
+ return ULONG_MAX;
+}
+
+/*
+ * On arm64, we have to ensure that the initrd ends up in the linear region,
+ * which is a 1 GB aligned region of size '1UL << (VA_BITS - 1)' that is
+ * guaranteed to cover the kernel Image.
+ *
+ * Since the EFI stub is part of the kernel Image, we can relax the
+ * usual requirements in Documentation/arm64/booting.txt, which still
+ * apply to other bootloaders, and are required for some kernel
+ * configurations.
+ */
+static inline unsigned long efi_get_max_initrd_addr(unsigned long dram_base,
+ unsigned long image_addr)
+{
+ return (image_addr & ~(SZ_1G - 1UL)) + (1UL << (VA_BITS - 1));
+}
#define efi_call_early(f, ...) sys_table_arg->boottime->f(__VA_ARGS__)
#define __efi_call_early(f, ...) f(__VA_ARGS__)
diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h
index d14c478..ad42e79 100644
--- a/arch/arm64/include/asm/esr.h
+++ b/arch/arm64/include/asm/esr.h
@@ -175,6 +175,8 @@
#define ESR_ELx_SYS64_ISS_SYS_CTR_READ (ESR_ELx_SYS64_ISS_SYS_CTR | \
ESR_ELx_SYS64_ISS_DIR_READ)
+#define ESR_ELx_SYS64_ISS_SYS_CNTVCT (ESR_ELx_SYS64_ISS_SYS_VAL(3, 3, 2, 14, 0) | \
+ ESR_ELx_SYS64_ISS_DIR_READ)
#ifndef __ASSEMBLY__
#include <asm/types.h>
diff --git a/arch/arm64/include/asm/extable.h b/arch/arm64/include/asm/extable.h
new file mode 100644
index 0000000..42f50f1
--- /dev/null
+++ b/arch/arm64/include/asm/extable.h
@@ -0,0 +1,25 @@
+#ifndef __ASM_EXTABLE_H
+#define __ASM_EXTABLE_H
+
+/*
+ * The exception table consists of pairs of relative offsets: the first
+ * is the relative offset to an instruction that is allowed to fault,
+ * and the second is the relative offset at which the program should
+ * continue. No registers are modified, so it is entirely up to the
+ * continuation code to figure out what to do.
+ *
+ * All the routines below use bits of fixup code that are out of line
+ * with the main instruction path. This means when everything is well,
+ * we don't even have to jump over them. Further, they do not intrude
+ * on our cache or tlb entries.
+ */
+
+struct exception_table_entry
+{
+ int insn, fixup;
+};
+
+#define ARCH_HAS_RELATIVE_EXTABLE
+
+extern int fixup_exception(struct pt_regs *regs);
+#endif
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index aecc07e..29cb2ca 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -80,6 +80,7 @@
AARCH64_INSN_REGTYPE_RM,
AARCH64_INSN_REGTYPE_RD,
AARCH64_INSN_REGTYPE_RA,
+ AARCH64_INSN_REGTYPE_RS,
};
enum aarch64_insn_register {
@@ -188,6 +189,8 @@
AARCH64_INSN_LDST_STORE_PAIR_PRE_INDEX,
AARCH64_INSN_LDST_LOAD_PAIR_POST_INDEX,
AARCH64_INSN_LDST_STORE_PAIR_POST_INDEX,
+ AARCH64_INSN_LDST_LOAD_EX,
+ AARCH64_INSN_LDST_STORE_EX,
};
enum aarch64_insn_adsb_type {
@@ -240,6 +243,23 @@
AARCH64_INSN_LOGIC_BIC_SETFLAGS
};
+enum aarch64_insn_prfm_type {
+ AARCH64_INSN_PRFM_TYPE_PLD,
+ AARCH64_INSN_PRFM_TYPE_PLI,
+ AARCH64_INSN_PRFM_TYPE_PST,
+};
+
+enum aarch64_insn_prfm_target {
+ AARCH64_INSN_PRFM_TARGET_L1,
+ AARCH64_INSN_PRFM_TARGET_L2,
+ AARCH64_INSN_PRFM_TARGET_L3,
+};
+
+enum aarch64_insn_prfm_policy {
+ AARCH64_INSN_PRFM_POLICY_KEEP,
+ AARCH64_INSN_PRFM_POLICY_STRM,
+};
+
#define __AARCH64_INSN_FUNCS(abbr, mask, val) \
static __always_inline bool aarch64_insn_is_##abbr(u32 code) \
{ return (code & (mask)) == (val); } \
@@ -248,6 +268,7 @@
__AARCH64_INSN_FUNCS(adr, 0x9F000000, 0x10000000)
__AARCH64_INSN_FUNCS(adrp, 0x9F000000, 0x90000000)
+__AARCH64_INSN_FUNCS(prfm, 0x3FC00000, 0x39800000)
__AARCH64_INSN_FUNCS(prfm_lit, 0xFF000000, 0xD8000000)
__AARCH64_INSN_FUNCS(str_reg, 0x3FE0EC00, 0x38206800)
__AARCH64_INSN_FUNCS(ldr_reg, 0x3FE0EC00, 0x38606800)
@@ -357,6 +378,11 @@
int offset,
enum aarch64_insn_variant variant,
enum aarch64_insn_ldst_type type);
+u32 aarch64_insn_gen_load_store_ex(enum aarch64_insn_register reg,
+ enum aarch64_insn_register base,
+ enum aarch64_insn_register state,
+ enum aarch64_insn_size_type size,
+ enum aarch64_insn_ldst_type type);
u32 aarch64_insn_gen_add_sub_imm(enum aarch64_insn_register dst,
enum aarch64_insn_register src,
int imm, enum aarch64_insn_variant variant,
@@ -397,6 +423,10 @@
int shift,
enum aarch64_insn_variant variant,
enum aarch64_insn_logic_type type);
+u32 aarch64_insn_gen_prefetch(enum aarch64_insn_register base,
+ enum aarch64_insn_prfm_type type,
+ enum aarch64_insn_prfm_target target,
+ enum aarch64_insn_prfm_policy policy);
s32 aarch64_get_branch_offset(u32 insn);
u32 aarch64_set_branch_offset(u32 insn, s32 offset);
diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h
index 5308d69..ba49717 100644
--- a/arch/arm64/include/asm/uaccess.h
+++ b/arch/arm64/include/asm/uaccess.h
@@ -28,38 +28,12 @@
#include <linux/bitops.h>
#include <linux/kasan-checks.h>
#include <linux/string.h>
-#include <linux/thread_info.h>
#include <asm/cpufeature.h>
#include <asm/ptrace.h>
-#include <asm/errno.h>
#include <asm/memory.h>
#include <asm/compiler.h>
-
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
-/*
- * The exception table consists of pairs of relative offsets: the first
- * is the relative offset to an instruction that is allowed to fault,
- * and the second is the relative offset at which the program should
- * continue. No registers are modified, so it is entirely up to the
- * continuation code to figure out what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-
-struct exception_table_entry
-{
- int insn, fixup;
-};
-
-#define ARCH_HAS_RELATIVE_EXTABLE
-
-extern int fixup_exception(struct pt_regs *regs);
+#include <asm/extable.h>
#define KERNEL_DS (-1UL)
#define get_ds() (KERNEL_DS)
@@ -357,58 +331,13 @@
})
extern unsigned long __must_check __arch_copy_from_user(void *to, const void __user *from, unsigned long n);
+#define raw_copy_from_user __arch_copy_from_user
extern unsigned long __must_check __arch_copy_to_user(void __user *to, const void *from, unsigned long n);
-extern unsigned long __must_check __copy_in_user(void __user *to, const void __user *from, unsigned long n);
+#define raw_copy_to_user __arch_copy_to_user
+extern unsigned long __must_check raw_copy_in_user(void __user *to, const void __user *from, unsigned long n);
extern unsigned long __must_check __clear_user(void __user *addr, unsigned long n);
-
-static inline unsigned long __must_check __copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- kasan_check_write(to, n);
- check_object_size(to, n, false);
- return __arch_copy_from_user(to, from, n);
-}
-
-static inline unsigned long __must_check __copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- kasan_check_read(from, n);
- check_object_size(from, n, true);
- return __arch_copy_to_user(to, from, n);
-}
-
-static inline unsigned long __must_check copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- unsigned long res = n;
- kasan_check_write(to, n);
- check_object_size(to, n, false);
-
- if (access_ok(VERIFY_READ, from, n)) {
- res = __arch_copy_from_user(to, from, n);
- }
- if (unlikely(res))
- memset(to + (n - res), 0, res);
- return res;
-}
-
-static inline unsigned long __must_check copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- kasan_check_read(from, n);
- check_object_size(from, n, true);
-
- if (access_ok(VERIFY_WRITE, to, n)) {
- n = __arch_copy_to_user(to, from, n);
- }
- return n;
-}
-
-static inline unsigned long __must_check copy_in_user(void __user *to, const void __user *from, unsigned long n)
-{
- if (access_ok(VERIFY_READ, from, n) && access_ok(VERIFY_WRITE, to, n))
- n = __copy_in_user(to, from, n);
- return n;
-}
-
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
+#define INLINE_COPY_TO_USER
+#define INLINE_COPY_FROM_USER
static inline unsigned long __must_check clear_user(void __user *to, unsigned long n)
{
diff --git a/arch/arm64/include/asm/unistd.h b/arch/arm64/include/asm/unistd.h
index bdbeb06..a0baa9a 100644
--- a/arch/arm64/include/asm/unistd.h
+++ b/arch/arm64/include/asm/unistd.h
@@ -14,7 +14,6 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifdef CONFIG_COMPAT
-#define __ARCH_WANT_COMPAT_SYS_GETDENTS64
#define __ARCH_WANT_COMPAT_STAT64
#define __ARCH_WANT_SYS_GETHOSTNAME
#define __ARCH_WANT_SYS_PAUSE
diff --git a/arch/arm64/include/asm/unistd32.h b/arch/arm64/include/asm/unistd32.h
index c66b51a..ef29216 100644
--- a/arch/arm64/include/asm/unistd32.h
+++ b/arch/arm64/include/asm/unistd32.h
@@ -456,7 +456,7 @@
#define __NR_setfsgid32 216
__SYSCALL(__NR_setfsgid32, sys_setfsgid)
#define __NR_getdents64 217
-__SYSCALL(__NR_getdents64, compat_sys_getdents64)
+__SYSCALL(__NR_getdents64, sys_getdents64)
#define __NR_pivot_root 218
__SYSCALL(__NR_pivot_root, sys_pivot_root)
#define __NR_mincore 219
diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
index 64d9cbd6..e25c11e 100644
--- a/arch/arm64/kernel/acpi.c
+++ b/arch/arm64/kernel/acpi.c
@@ -18,6 +18,7 @@
#include <linux/acpi.h>
#include <linux/bootmem.h>
#include <linux/cpumask.h>
+#include <linux/efi-bgrt.h>
#include <linux/init.h>
#include <linux/irq.h>
#include <linux/irqdomain.h>
@@ -233,6 +234,8 @@
early_init_dt_scan_chosen_stdout();
} else {
parse_spcr(earlycon_init_is_deferred);
+ if (IS_ENABLED(CONFIG_ACPI_BGRT))
+ acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt);
}
}
diff --git a/arch/arm64/kernel/arm64ksyms.c b/arch/arm64/kernel/arm64ksyms.c
index e9c4dc9..67368c7 100644
--- a/arch/arm64/kernel/arm64ksyms.c
+++ b/arch/arm64/kernel/arm64ksyms.c
@@ -38,7 +38,7 @@
EXPORT_SYMBOL(__arch_copy_from_user);
EXPORT_SYMBOL(__arch_copy_to_user);
EXPORT_SYMBOL(__clear_user);
-EXPORT_SYMBOL(__copy_in_user);
+EXPORT_SYMBOL(raw_copy_in_user);
/* physical memory */
EXPORT_SYMBOL(memstart_addr);
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index f6cc67e..2ed2a76 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -53,6 +53,13 @@
.midr_range_min = min, \
.midr_range_max = max
+#define MIDR_ALL_VERSIONS(model) \
+ .def_scope = SCOPE_LOCAL_CPU, \
+ .matches = is_affected_midr_range, \
+ .midr_model = model, \
+ .midr_range_min = 0, \
+ .midr_range_max = (MIDR_VARIANT_MASK | MIDR_REVISION_MASK)
+
const struct arm64_cpu_capabilities arm64_errata[] = {
#if defined(CONFIG_ARM64_ERRATUM_826319) || \
defined(CONFIG_ARM64_ERRATUM_827319) || \
@@ -151,6 +158,14 @@
MIDR_CPU_VAR_REV(0, 0)),
},
#endif
+#ifdef CONFIG_ARM64_ERRATUM_858921
+ {
+ /* Cortex-A73 all versions */
+ .desc = "ARM erratum 858921",
+ .capability = ARM64_WORKAROUND_858921,
+ MIDR_ALL_VERSIONS(MIDR_CORTEX_A73),
+ },
+#endif
{
}
};
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index abda8e8..6eb77ae 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -1090,20 +1090,29 @@
* Check if the current CPU has a given feature capability.
* Should be called from non-preemptible context.
*/
-bool this_cpu_has_cap(unsigned int cap)
+static bool __this_cpu_has_cap(const struct arm64_cpu_capabilities *cap_array,
+ unsigned int cap)
{
const struct arm64_cpu_capabilities *caps;
if (WARN_ON(preemptible()))
return false;
- for (caps = arm64_features; caps->desc; caps++)
+ for (caps = cap_array; caps->desc; caps++)
if (caps->capability == cap && caps->matches)
return caps->matches(caps, SCOPE_LOCAL_CPU);
return false;
}
+extern const struct arm64_cpu_capabilities arm64_errata[];
+
+bool this_cpu_has_cap(unsigned int cap)
+{
+ return (__this_cpu_has_cap(arm64_features, cap) ||
+ __this_cpu_has_cap(arm64_errata, cap));
+}
+
void __init setup_cpu_features(void)
{
u32 cwg;
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 3a63954..b884a92 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -474,6 +474,7 @@
shift = 10;
break;
case AARCH64_INSN_REGTYPE_RM:
+ case AARCH64_INSN_REGTYPE_RS:
shift = 16;
break;
default:
@@ -757,6 +758,111 @@
offset >> shift);
}
+u32 aarch64_insn_gen_load_store_ex(enum aarch64_insn_register reg,
+ enum aarch64_insn_register base,
+ enum aarch64_insn_register state,
+ enum aarch64_insn_size_type size,
+ enum aarch64_insn_ldst_type type)
+{
+ u32 insn;
+
+ switch (type) {
+ case AARCH64_INSN_LDST_LOAD_EX:
+ insn = aarch64_insn_get_load_ex_value();
+ break;
+ case AARCH64_INSN_LDST_STORE_EX:
+ insn = aarch64_insn_get_store_ex_value();
+ break;
+ default:
+ pr_err("%s: unknown load/store exclusive encoding %d\n", __func__, type);
+ return AARCH64_BREAK_FAULT;
+ }
+
+ insn = aarch64_insn_encode_ldst_size(size, insn);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RT, insn,
+ reg);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn,
+ base);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RT2, insn,
+ AARCH64_INSN_REG_ZR);
+
+ return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RS, insn,
+ state);
+}
+
+static u32 aarch64_insn_encode_prfm_imm(enum aarch64_insn_prfm_type type,
+ enum aarch64_insn_prfm_target target,
+ enum aarch64_insn_prfm_policy policy,
+ u32 insn)
+{
+ u32 imm_type = 0, imm_target = 0, imm_policy = 0;
+
+ switch (type) {
+ case AARCH64_INSN_PRFM_TYPE_PLD:
+ break;
+ case AARCH64_INSN_PRFM_TYPE_PLI:
+ imm_type = BIT(0);
+ break;
+ case AARCH64_INSN_PRFM_TYPE_PST:
+ imm_type = BIT(1);
+ break;
+ default:
+ pr_err("%s: unknown prfm type encoding %d\n", __func__, type);
+ return AARCH64_BREAK_FAULT;
+ }
+
+ switch (target) {
+ case AARCH64_INSN_PRFM_TARGET_L1:
+ break;
+ case AARCH64_INSN_PRFM_TARGET_L2:
+ imm_target = BIT(0);
+ break;
+ case AARCH64_INSN_PRFM_TARGET_L3:
+ imm_target = BIT(1);
+ break;
+ default:
+ pr_err("%s: unknown prfm target encoding %d\n", __func__, target);
+ return AARCH64_BREAK_FAULT;
+ }
+
+ switch (policy) {
+ case AARCH64_INSN_PRFM_POLICY_KEEP:
+ break;
+ case AARCH64_INSN_PRFM_POLICY_STRM:
+ imm_policy = BIT(0);
+ break;
+ default:
+ pr_err("%s: unknown prfm policy encoding %d\n", __func__, policy);
+ return AARCH64_BREAK_FAULT;
+ }
+
+ /* In this case, imm5 is encoded into Rt field. */
+ insn &= ~GENMASK(4, 0);
+ insn |= imm_policy | (imm_target << 1) | (imm_type << 3);
+
+ return insn;
+}
+
+u32 aarch64_insn_gen_prefetch(enum aarch64_insn_register base,
+ enum aarch64_insn_prfm_type type,
+ enum aarch64_insn_prfm_target target,
+ enum aarch64_insn_prfm_policy policy)
+{
+ u32 insn = aarch64_insn_get_prfm_value();
+
+ insn = aarch64_insn_encode_ldst_size(AARCH64_INSN_SIZE_64, insn);
+
+ insn = aarch64_insn_encode_prfm_imm(type, target, policy, insn);
+
+ insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn,
+ base);
+
+ return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_12, insn, 0);
+}
+
u32 aarch64_insn_gen_add_sub_imm(enum aarch64_insn_register dst,
enum aarch64_insn_register src,
int imm, enum aarch64_insn_variant variant,
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index e52be6a..1de444e 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -505,6 +505,14 @@
regs->pc += 4;
}
+static void cntvct_read_handler(unsigned int esr, struct pt_regs *regs)
+{
+ int rt = (esr & ESR_ELx_SYS64_ISS_RT_MASK) >> ESR_ELx_SYS64_ISS_RT_SHIFT;
+
+ pt_regs_write_reg(regs, rt, arch_counter_get_cntvct());
+ regs->pc += 4;
+}
+
struct sys64_hook {
unsigned int esr_mask;
unsigned int esr_val;
@@ -523,6 +531,12 @@
.esr_val = ESR_ELx_SYS64_ISS_SYS_CTR_READ,
.handler = ctr_read_handler,
},
+ {
+ /* Trap read access to CNTVCT_EL0 */
+ .esr_mask = ESR_ELx_SYS64_ISS_SYS_OP_MASK,
+ .esr_val = ESR_ELx_SYS64_ISS_SYS_CNTVCT,
+ .handler = cntvct_read_handler,
+ },
{},
};
diff --git a/arch/arm64/lib/copy_in_user.S b/arch/arm64/lib/copy_in_user.S
index 47184c3..b24a830 100644
--- a/arch/arm64/lib/copy_in_user.S
+++ b/arch/arm64/lib/copy_in_user.S
@@ -64,14 +64,14 @@
.endm
end .req x5
-ENTRY(__copy_in_user)
+ENTRY(raw_copy_in_user)
uaccess_enable_not_uao x3, x4
add end, x0, x2
#include "copy_template.S"
uaccess_disable_not_uao x3
mov x0, #0
ret
-ENDPROC(__copy_in_user)
+ENDPROC(raw_copy_in_user)
.section .fixup,"ax"
.align 2
diff --git a/arch/arm64/net/bpf_jit.h b/arch/arm64/net/bpf_jit.h
index 7c16e54..b02a926 100644
--- a/arch/arm64/net/bpf_jit.h
+++ b/arch/arm64/net/bpf_jit.h
@@ -83,6 +83,25 @@
/* Rt = Rn[0]; Rt2 = Rn[8]; Rn += 16; */
#define A64_POP(Rt, Rt2, Rn) A64_LS_PAIR(Rt, Rt2, Rn, 16, LOAD, POST_INDEX)
+/* Load/store exclusive */
+#define A64_SIZE(sf) \
+ ((sf) ? AARCH64_INSN_SIZE_64 : AARCH64_INSN_SIZE_32)
+#define A64_LSX(sf, Rt, Rn, Rs, type) \
+ aarch64_insn_gen_load_store_ex(Rt, Rn, Rs, A64_SIZE(sf), \
+ AARCH64_INSN_LDST_##type)
+/* Rt = [Rn]; (atomic) */
+#define A64_LDXR(sf, Rt, Rn) \
+ A64_LSX(sf, Rt, Rn, A64_ZR, LOAD_EX)
+/* [Rn] = Rt; (atomic) Rs = [state] */
+#define A64_STXR(sf, Rt, Rn, Rs) \
+ A64_LSX(sf, Rt, Rn, Rs, STORE_EX)
+
+/* Prefetch */
+#define A64_PRFM(Rn, type, target, policy) \
+ aarch64_insn_gen_prefetch(Rn, AARCH64_INSN_PRFM_TYPE_##type, \
+ AARCH64_INSN_PRFM_TARGET_##target, \
+ AARCH64_INSN_PRFM_POLICY_##policy)
+
/* Add/subtract (immediate) */
#define A64_ADDSUB_IMM(sf, Rd, Rn, imm12, type) \
aarch64_insn_gen_add_sub_imm(Rd, Rn, imm12, \
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index a785554..d68abde 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -321,6 +321,7 @@
const s32 imm = insn->imm;
const int i = insn - ctx->prog->insnsi;
const bool is64 = BPF_CLASS(code) == BPF_ALU64;
+ const bool isdw = BPF_SIZE(code) == BPF_DW;
u8 jmp_cond;
s32 jmp_offset;
@@ -604,15 +605,6 @@
const struct bpf_insn insn1 = insn[1];
u64 imm64;
- if (insn1.code != 0 || insn1.src_reg != 0 ||
- insn1.dst_reg != 0 || insn1.off != 0) {
- /* Note: verifier in BPF core must catch invalid
- * instructions.
- */
- pr_err_once("Invalid BPF_LD_IMM64 instruction\n");
- return -EINVAL;
- }
-
imm64 = (u64)insn1.imm << 32 | (u32)imm;
emit_a64_mov_i64(dst, imm64, ctx);
@@ -690,7 +682,16 @@
case BPF_STX | BPF_XADD | BPF_W:
/* STX XADD: lock *(u64 *)(dst + off) += src */
case BPF_STX | BPF_XADD | BPF_DW:
- goto notyet;
+ emit_a64_mov_i(1, tmp, off, ctx);
+ emit(A64_ADD(1, tmp, tmp, dst), ctx);
+ emit(A64_PRFM(tmp, PST, L1, STRM), ctx);
+ emit(A64_LDXR(isdw, tmp2, tmp), ctx);
+ emit(A64_ADD(isdw, tmp2, tmp2, src), ctx);
+ emit(A64_STXR(isdw, tmp2, tmp, tmp2), ctx);
+ jmp_offset = -3;
+ check_imm19(jmp_offset);
+ emit(A64_CBNZ(0, tmp2, jmp_offset), ctx);
+ break;
/* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + imm)) */
case BPF_LD | BPF_ABS | BPF_W:
@@ -757,10 +758,6 @@
}
break;
}
-notyet:
- pr_info_once("*** NOT YET: opcode %02x ***\n", code);
- return -EFAULT;
-
default:
pr_err_once("unknown opcode %02x\n", code);
return -EINVAL;
@@ -779,14 +776,14 @@
int ret;
ret = build_insn(insn, ctx);
-
- if (ctx->image == NULL)
- ctx->offset[i] = ctx->idx;
-
if (ret > 0) {
i++;
+ if (ctx->image == NULL)
+ ctx->offset[i] = ctx->idx;
continue;
}
+ if (ctx->image == NULL)
+ ctx->offset[i] = ctx->idx;
if (ret)
return ret;
}
diff --git a/arch/avr32/Kconfig b/arch/avr32/Kconfig
deleted file mode 100644
index 7e75d45..0000000
--- a/arch/avr32/Kconfig
+++ /dev/null
@@ -1,288 +0,0 @@
-config AVR32
- def_bool y
- # With EXPERT=n, we get lots of stuff automatically selected
- # that we usually don't need on AVR32.
- select EXPERT
- select HAVE_CLK
- select HAVE_EXIT_THREAD
- select HAVE_OPROFILE
- select HAVE_KPROBES
- select VIRT_TO_BUS
- select GENERIC_IRQ_PROBE
- select GENERIC_ATOMIC64
- select HARDIRQS_SW_RESEND
- select GENERIC_IRQ_SHOW
- select ARCH_HAVE_CUSTOM_GPIO_H
- select ARCH_WANT_IPC_PARSE_VERSION
- select ARCH_HAVE_NMI_SAFE_CMPXCHG
- select GENERIC_CLOCKEVENTS
- select HAVE_MOD_ARCH_SPECIFIC
- select MODULES_USE_ELF_RELA
- select HAVE_NMI
- help
- AVR32 is a high-performance 32-bit RISC microprocessor core,
- designed for cost-sensitive embedded applications, with particular
- emphasis on low power consumption and high code density.
-
- There is an AVR32 Linux project with a web page at
- http://avr32linux.org/.
-
-config STACKTRACE_SUPPORT
- def_bool y
-
-config LOCKDEP_SUPPORT
- def_bool y
-
-config TRACE_IRQFLAGS_SUPPORT
- def_bool y
-
-config RWSEM_GENERIC_SPINLOCK
- def_bool y
-
-config RWSEM_XCHGADD_ALGORITHM
- def_bool n
-
-config ARCH_HAS_ILOG2_U32
- def_bool n
-
-config ARCH_HAS_ILOG2_U64
- def_bool n
-
-config GENERIC_HWEIGHT
- def_bool y
-
-config GENERIC_CALIBRATE_DELAY
- def_bool y
-
-config GENERIC_BUG
- def_bool y
- depends on BUG
-
-source "init/Kconfig"
-
-source "kernel/Kconfig.freezer"
-
-menu "System Type and features"
-
-config SUBARCH_AVR32B
- bool
-config MMU
- bool
-config PERFORMANCE_COUNTERS
- bool
-
-config PLATFORM_AT32AP
- bool
- select SUBARCH_AVR32B
- select MMU
- select PERFORMANCE_COUNTERS
- select GPIOLIB
- select GENERIC_ALLOCATOR
- select HAVE_FB_ATMEL
-
-#
-# CPU types
-#
-
-# AP7000 derivatives
-config CPU_AT32AP700X
- bool
- select PLATFORM_AT32AP
-config CPU_AT32AP7000
- bool
- select CPU_AT32AP700X
-config CPU_AT32AP7001
- bool
- select CPU_AT32AP700X
-config CPU_AT32AP7002
- bool
- select CPU_AT32AP700X
-
-# AP700X boards
-config BOARD_ATNGW100_COMMON
- bool
- select CPU_AT32AP7000
-
-choice
- prompt "AVR32 board type"
- default BOARD_ATSTK1000
-
-config BOARD_ATSTK1000
- bool "ATSTK1000 evaluation board"
-
-config BOARD_ATNGW100_MKI
- bool "ATNGW100 Network Gateway"
- select BOARD_ATNGW100_COMMON
-
-config BOARD_ATNGW100_MKII
- bool "ATNGW100 mkII Network Gateway"
- select BOARD_ATNGW100_COMMON
-
-config BOARD_HAMMERHEAD
- bool "Hammerhead board"
- select CPU_AT32AP7000
- select USB_ARCH_HAS_HCD
- help
- The Hammerhead platform is built around an AVR32 32-bit microcontroller from Atmel.
- It offers versatile peripherals, such as ethernet, usb device, usb host etc.
-
- The board also incorporates a power supply and is a Power over Ethernet (PoE) Powered
- Device (PD).
-
- Additionally, a Cyclone III FPGA from Altera is integrated on the board. The FPGA is
- mapped into the 32-bit AVR memory bus. The FPGA offers two DDR2 SDRAM interfaces, which
- will cover even the most exceptional need of memory bandwidth. Together with the onboard
- video decoder the board is ready for video processing.
-
- For more information see: http://www.miromico.ch/index.php/hammerhead.html
-
-config BOARD_FAVR_32
- bool "Favr-32 LCD-board"
- select CPU_AT32AP7000
-
-config BOARD_MERISC
- bool "Merisc board"
- select CPU_AT32AP7000
- help
- Merisc is the family name for a range of AVR32-based boards.
-
- The boards are designed to be used in a man-machine
- interfacing environment, utilizing a touch-based graphical
- user interface. They host a vast range of I/O peripherals as
- well as a large SDRAM & Flash memory bank.
-
- For more information see: http://www.martinsson.se/merisc
-
-config BOARD_MIMC200
- bool "MIMC200 CPU board"
- select CPU_AT32AP7000
-endchoice
-
-source "arch/avr32/boards/atstk1000/Kconfig"
-source "arch/avr32/boards/atngw100/Kconfig"
-source "arch/avr32/boards/hammerhead/Kconfig"
-source "arch/avr32/boards/favr-32/Kconfig"
-source "arch/avr32/boards/merisc/Kconfig"
-
-choice
- prompt "Boot loader type"
- default LOADER_U_BOOT
-
-config LOADER_U_BOOT
- bool "U-Boot (or similar) bootloader"
-endchoice
-
-source "arch/avr32/mach-at32ap/Kconfig"
-
-config LOAD_ADDRESS
- hex
- default 0x10000000 if LOADER_U_BOOT=y && CPU_AT32AP700X=y
-
-config ENTRY_ADDRESS
- hex
- default 0x90000000 if LOADER_U_BOOT=y && CPU_AT32AP700X=y
-
-config PHYS_OFFSET
- hex
- default 0x10000000 if CPU_AT32AP700X=y
-
-source "kernel/Kconfig.preempt"
-
-config QUICKLIST
- def_bool y
-
-config ARCH_HAVE_MEMORY_PRESENT
- def_bool n
-
-config NEED_NODE_MEMMAP_SIZE
- def_bool n
-
-config ARCH_FLATMEM_ENABLE
- def_bool y
-
-config ARCH_DISCONTIGMEM_ENABLE
- def_bool n
-
-config ARCH_SPARSEMEM_ENABLE
- def_bool n
-
-config NODES_SHIFT
- int
- default "2"
- depends on NEED_MULTIPLE_NODES
-
-source "mm/Kconfig"
-
-config OWNERSHIP_TRACE
- bool "Ownership trace support"
- default y
- help
- Say Y to generate an Ownership Trace message on every context switch,
- enabling Nexus-compliant debuggers to keep track of the PID of the
- currently executing task.
-
-config NMI_DEBUGGING
- bool "NMI Debugging"
- default n
- help
- Say Y here and pass the nmi_debug command-line parameter to
- the kernel to turn on NMI debugging. Depending on the value
- of the nmi_debug option, various pieces of information will
- be dumped to the console when a Non-Maskable Interrupt
- happens.
-
-# FPU emulation goes here
-
-source "kernel/Kconfig.hz"
-
-config CMDLINE
- string "Default kernel command line"
- default ""
- help
- If you don't have a boot loader capable of passing a command line string
- to the kernel, you may specify one here. As a minimum, you should specify
- the memory size and the root device (e.g., mem=8M, root=/dev/nfs).
-
-endmenu
-
-menu "Power management options"
-
-source "kernel/power/Kconfig"
-
-config ARCH_SUSPEND_POSSIBLE
- def_bool y
-
-menu "CPU Frequency scaling"
-source "drivers/cpufreq/Kconfig"
-endmenu
-
-endmenu
-
-menu "Bus options"
-
-config PCI
- bool
-
-source "drivers/pci/Kconfig"
-
-source "drivers/pcmcia/Kconfig"
-
-endmenu
-
-menu "Executable file formats"
-source "fs/Kconfig.binfmt"
-endmenu
-
-source "net/Kconfig"
-
-source "drivers/Kconfig"
-
-source "fs/Kconfig"
-
-source "arch/avr32/Kconfig.debug"
-
-source "security/Kconfig"
-
-source "crypto/Kconfig"
-
-source "lib/Kconfig"
diff --git a/arch/avr32/Kconfig.debug b/arch/avr32/Kconfig.debug
deleted file mode 100644
index 2283933..0000000
--- a/arch/avr32/Kconfig.debug
+++ /dev/null
@@ -1,9 +0,0 @@
-menu "Kernel hacking"
-
-config TRACE_IRQFLAGS_SUPPORT
- bool
- default y
-
-source "lib/Kconfig.debug"
-
-endmenu
diff --git a/arch/avr32/Makefile b/arch/avr32/Makefile
deleted file mode 100644
index dba48a5d..0000000
--- a/arch/avr32/Makefile
+++ /dev/null
@@ -1,84 +0,0 @@
-#
-# This file is subject to the terms and conditions of the GNU General Public
-# License. See the file "COPYING" in the main directory of this archive
-# for more details.
-#
-# Copyright (C) 2004-2006 Atmel Corporation.
-
-# Default target when executing plain make
-.PHONY: all
-all: uImage vmlinux.elf
-
-KBUILD_DEFCONFIG := atstk1002_defconfig
-
-KBUILD_CFLAGS += -pipe -fno-builtin -mno-pic -D__linux__
-KBUILD_AFLAGS += -mrelax -mno-pic
-KBUILD_CFLAGS_MODULE += -mno-relax
-LDFLAGS_vmlinux += --relax
-
-cpuflags-$(CONFIG_PLATFORM_AT32AP) += -march=ap
-
-KBUILD_CFLAGS += $(cpuflags-y)
-KBUILD_AFLAGS += $(cpuflags-y)
-
-CHECKFLAGS += -D__avr32__ -D__BIG_ENDIAN
-
-machine-$(CONFIG_PLATFORM_AT32AP) := at32ap
-machdirs := $(patsubst %,arch/avr32/mach-%/, $(machine-y))
-
-KBUILD_CPPFLAGS += $(patsubst %,-I$(srctree)/%include,$(machdirs))
-
-head-$(CONFIG_LOADER_U_BOOT) += arch/avr32/boot/u-boot/head.o
-head-y += arch/avr32/kernel/head.o
-core-y += $(machdirs)
-core-$(CONFIG_BOARD_ATSTK1000) += arch/avr32/boards/atstk1000/
-core-$(CONFIG_BOARD_ATNGW100_COMMON) += arch/avr32/boards/atngw100/
-core-$(CONFIG_BOARD_HAMMERHEAD) += arch/avr32/boards/hammerhead/
-core-$(CONFIG_BOARD_FAVR_32) += arch/avr32/boards/favr-32/
-core-$(CONFIG_BOARD_MERISC) += arch/avr32/boards/merisc/
-core-$(CONFIG_BOARD_MIMC200) += arch/avr32/boards/mimc200/
-core-$(CONFIG_LOADER_U_BOOT) += arch/avr32/boot/u-boot/
-core-y += arch/avr32/kernel/
-core-y += arch/avr32/mm/
-drivers-$(CONFIG_OPROFILE) += arch/avr32/oprofile/
-libs-y += arch/avr32/lib/
-
-BOOT_TARGETS := vmlinux.elf vmlinux.bin uImage uImage.srec
-
-.PHONY: $(BOOT_TARGETS) install
-
-boot := arch/$(ARCH)/boot/images
-
- KBUILD_IMAGE := $(boot)/uImage
-vmlinux.elf: KBUILD_IMAGE := $(boot)/vmlinux.elf
-vmlinux.cso: KBUILD_IMAGE := $(boot)/vmlinux.cso
-uImage.srec: KBUILD_IMAGE := $(boot)/uImage.srec
-uImage: KBUILD_IMAGE := $(boot)/uImage
-
-quiet_cmd_listing = LST $@
- cmd_listing = avr32-linux-objdump $(OBJDUMPFLAGS) -lS $< > $@
-quiet_cmd_disasm = DIS $@
- cmd_disasm = avr32-linux-objdump $(OBJDUMPFLAGS) -d $< > $@
-
-vmlinux.elf vmlinux.bin uImage.srec uImage vmlinux.cso: vmlinux
- $(Q)$(MAKE) $(build)=$(boot) $(boot)/$@
-
-install: vmlinux
- $(Q)$(MAKE) $(build)=$(boot) BOOTIMAGE=$(KBUILD_IMAGE) $@
-
-vmlinux.s: vmlinux
- $(call if_changed,disasm)
-
-vmlinux.lst: vmlinux
- $(call if_changed,listing)
-
-CLEAN_FILES += vmlinux.s vmlinux.lst
-
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
-
-define archhelp
- @echo '* vmlinux.elf - ELF image with load address 0'
- @echo ' vmlinux.cso - PathFinder CSO image'
- @echo '* uImage - Create a bootable image for U-Boot'
-endef
diff --git a/arch/avr32/boards/atngw100/Kconfig b/arch/avr32/boards/atngw100/Kconfig
deleted file mode 100644
index 4e55617..0000000
--- a/arch/avr32/boards/atngw100/Kconfig
+++ /dev/null
@@ -1,65 +0,0 @@
-# NGW100 customization
-
-if BOARD_ATNGW100_COMMON
-
-config BOARD_ATNGW100_MKII_LCD
- bool "Enable ATNGW100 mkII LCD interface"
- depends on BOARD_ATNGW100_MKII
- help
- This enables the LCD controller (LCDC) in the AT32AP7000. Since the
- LCDC is multiplexed with MACB1 (LAN) Ethernet port, only one can be
- enabled at a time.
-
- This choice enables the LCDC and disables the MACB1 interface marked
- LAN on the PCB.
-
-choice
- prompt "Select an NGW100 add-on board to support"
- default BOARD_ATNGW100_ADDON_NONE
-
-config BOARD_ATNGW100_ADDON_NONE
- bool "None"
-
-config BOARD_ATNGW100_EVKLCD10X
- bool "EVKLCD10X addon board"
- depends on BOARD_ATNGW100_MKI || BOARD_ATNGW100_MKII_LCD
- help
- This enables support for the EVKLCD100 (QVGA) or EVKLCD101 (VGA)
- addon board for the NGW100 and NGW100 mkII. By enabling this the LCD
- controller and AC97 controller is added as platform devices.
-
-config BOARD_ATNGW100_MRMT
- bool "Mediama RMT1/2 add-on board"
- help
- This enables support for the Mediama RMT1 or RMT2 board.
- RMT provides LCD support, AC97 codec and other
- optional peripherals to the Atmel NGW100.
-
- This choice disables the detect pin and the write-protect pin for the
- MCI platform device, since it conflicts with the LCD platform device.
- The MCI pins can be reenabled by editing the "add device function" but
- this may break the setup for other displays that use these pins.
-
-endchoice
-
-choice
- prompt "LCD panel resolution on EVKLCD10X"
- depends on BOARD_ATNGW100_EVKLCD10X
- default BOARD_ATNGW100_EVKLCD10X_VGA
-
-config BOARD_ATNGW100_EVKLCD10X_QVGA
- bool "QVGA (320x240)"
-
-config BOARD_ATNGW100_EVKLCD10X_VGA
- bool "VGA (640x480)"
-
-config BOARD_ATNGW100_EVKLCD10X_POW_QVGA
- bool "Powertip QVGA (320x240)"
-
-endchoice
-
-if BOARD_ATNGW100_MRMT
-source "arch/avr32/boards/atngw100/Kconfig_mrmt"
-endif
-
-endif # BOARD_ATNGW100_COMMON
diff --git a/arch/avr32/boards/atngw100/Kconfig_mrmt b/arch/avr32/boards/atngw100/Kconfig_mrmt
deleted file mode 100644
index 9a199a2..0000000
--- a/arch/avr32/boards/atngw100/Kconfig_mrmt
+++ /dev/null
@@ -1,80 +0,0 @@
-# RMT for NGW100 customization
-
-choice
- prompt "RMT Version"
- help
- Select the RMTx board version.
-
-config BOARD_MRMT_REV1
- bool "RMT1"
-config BOARD_MRMT_REV2
- bool "RMT2"
-
-endchoice
-
-config BOARD_MRMT_AC97
- bool "Enable AC97 CODEC"
- help
- Enable the UCB1400 AC97 CODEC driver.
-
-choice
- prompt "Touchscreen Driver"
- default BOARD_MRMT_ADS7846_TS
-
-config BOARD_MRMT_UCB1400_TS
- bool "Use UCB1400 Touchscreen"
-
-config BOARD_MRMT_ADS7846_TS
- bool "Use ADS7846 Touchscreen"
-
-endchoice
-
-choice
- prompt "RMTx LCD Selection"
- default BOARD_MRMT_LCD_DISABLE
-
-config BOARD_MRMT_LCD_DISABLE
- bool "LCD Disabled"
-
-config BOARD_MRMT_LCD_LQ043T3DX0X
- bool "Sharp LQ043T3DX0x or compatible"
- help
- If using RMT2, be sure to load the resistor pack selectors accordingly
-
-if BOARD_MRMT_REV2
-config BOARD_MRMT_LCD_KWH043GM08
- bool "Formike KWH043GM08 or compatible"
- help
- Be sure to load the RMT2 resistor pack selectors accordingly
-endif
-
-endchoice
-
-if !BOARD_MRMT_LCD_DISABLE
-config BOARD_MRMT_BL_PWM
- bool "Use PWM control for LCD Backlight"
- help
- Use PWM driver for controlling LCD Backlight.
- Otherwise, LCD Backlight is always on.
-endif
-
-config BOARD_MRMT_RTC_I2C
- bool "Use External RTC on I2C Bus"
- help
- RMT1 has an optional RTC device on the I2C bus.
- It is a SII S35390A. Be sure to select the
- matching RTC driver.
-
-choice
- prompt "Wireless Module on ttyS2"
- default BOARD_MRMT_WIRELESS_ZB
-
-config BOARD_MRMT_WIRELESS_ZB
- bool "Use ZigBee/802.15.4 Module"
-
-config BOARD_MRMT_WIRELESS_BT
- bool "Use Bluetooth (HCI) Module"
-
-config BOARD_MRMT_WIRELESS_NONE
- bool "Not Installed"
-endchoice
diff --git a/arch/avr32/boards/atngw100/Makefile b/arch/avr32/boards/atngw100/Makefile
deleted file mode 100644
index f4ebe42..0000000
--- a/arch/avr32/boards/atngw100/Makefile
+++ /dev/null
@@ -1,3 +0,0 @@
-obj-y += setup.o flash.o
-obj-$(CONFIG_BOARD_ATNGW100_EVKLCD10X) += evklcd10x.o
-obj-$(CONFIG_BOARD_ATNGW100_MRMT) += mrmt.o
diff --git a/arch/avr32/boards/atngw100/evklcd10x.c b/arch/avr32/boards/atngw100/evklcd10x.c
deleted file mode 100644
index 64919b0..0000000
--- a/arch/avr32/boards/atngw100/evklcd10x.c
+++ /dev/null
@@ -1,178 +0,0 @@
-/*
- * Board-specific setup code for the ATEVKLCD10X addon board to the ATNGW100
- * Network Gateway
- *
- * Copyright (C) 2008 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 as published by
- * the Free Software Foundation.
- */
-
-#include <linux/init.h>
-#include <linux/linkage.h>
-#include <linux/gpio.h>
-#include <linux/fb.h>
-#include <linux/platform_device.h>
-
-#include <video/atmel_lcdc.h>
-
-#include <asm/setup.h>
-
-#include <mach/at32ap700x.h>
-#include <mach/portmux.h>
-#include <mach/board.h>
-
-#include <sound/atmel-ac97c.h>
-
-static struct ac97c_platform_data __initdata ac97c0_data = {
- .reset_pin = GPIO_PIN_PB(19),
-};
-
-#ifdef CONFIG_BOARD_ATNGW100_EVKLCD10X_VGA
-static struct fb_videomode __initdata tcg057vglad_modes[] = {
- {
- .name = "640x480 @ 50",
- .refresh = 50,
- .xres = 640, .yres = 480,
- .pixclock = KHZ2PICOS(25180),
-
- .left_margin = 64, .right_margin = 96,
- .upper_margin = 34, .lower_margin = 11,
- .hsync_len = 64, .vsync_len = 15,
-
- .sync = 0,
- .vmode = FB_VMODE_NONINTERLACED,
- },
-};
-
-static struct fb_monspecs __initdata atevklcd10x_default_monspecs = {
- .manufacturer = "KYO",
- .monitor = "TCG057VGLAD",
- .modedb = tcg057vglad_modes,
- .modedb_len = ARRAY_SIZE(tcg057vglad_modes),
- .hfmin = 19948,
- .hfmax = 31478,
- .vfmin = 50,
- .vfmax = 67,
- .dclkmax = 28330000,
-};
-
-static struct atmel_lcdfb_pdata __initdata atevklcd10x_lcdc_data = {
- .default_bpp = 16,
- .default_dmacon = ATMEL_LCDC_DMAEN | ATMEL_LCDC_DMA2DEN,
- .default_lcdcon2 = (ATMEL_LCDC_DISTYPE_TFT
- | ATMEL_LCDC_CLKMOD_ALWAYSACTIVE
- | ATMEL_LCDC_MEMOR_BIG),
- .default_monspecs = &atevklcd10x_default_monspecs,
- .guard_time = 2,
-};
-#elif CONFIG_BOARD_ATNGW100_EVKLCD10X_QVGA
-static struct fb_videomode __initdata tcg057qvlad_modes[] = {
- {
- .name = "320x240 @ 50",
- .refresh = 50,
- .xres = 320, .yres = 240,
- .pixclock = KHZ2PICOS(6300),
-
- .left_margin = 34, .right_margin = 46,
- .upper_margin = 7, .lower_margin = 15,
- .hsync_len = 64, .vsync_len = 12,
-
- .sync = 0,
- .vmode = FB_VMODE_NONINTERLACED,
- },
-};
-
-static struct fb_monspecs __initdata atevklcd10x_default_monspecs = {
- .manufacturer = "KYO",
- .monitor = "TCG057QVLAD",
- .modedb = tcg057qvlad_modes,
- .modedb_len = ARRAY_SIZE(tcg057qvlad_modes),
- .hfmin = 19948,
- .hfmax = 31478,
- .vfmin = 50,
- .vfmax = 67,
- .dclkmax = 7000000,
-};
-
-static struct atmel_lcdfb_pdata __initdata atevklcd10x_lcdc_data = {
- .default_bpp = 16,
- .default_dmacon = ATMEL_LCDC_DMAEN | ATMEL_LCDC_DMA2DEN,
- .default_lcdcon2 = (ATMEL_LCDC_DISTYPE_TFT
- | ATMEL_LCDC_CLKMOD_ALWAYSACTIVE
- | ATMEL_LCDC_MEMOR_BIG),
- .default_monspecs = &atevklcd10x_default_monspecs,
- .guard_time = 2,
-};
-#elif CONFIG_BOARD_ATNGW100_EVKLCD10X_POW_QVGA
-static struct fb_videomode __initdata ph320240t_modes[] = {
- {
- .name = "320x240 @ 60",
- .refresh = 60,
- .xres = 320, .yres = 240,
- .pixclock = KHZ2PICOS(6300),
-
- .left_margin = 38, .right_margin = 20,
- .upper_margin = 15, .lower_margin = 5,
- .hsync_len = 30, .vsync_len = 3,
-
- .sync = 0,
- .vmode = FB_VMODE_NONINTERLACED,
- },
-};
-
-static struct fb_monspecs __initdata atevklcd10x_default_monspecs = {
- .manufacturer = "POW",
- .monitor = "PH320240T",
- .modedb = ph320240t_modes,
- .modedb_len = ARRAY_SIZE(ph320240t_modes),
- .hfmin = 14400,
- .hfmax = 21600,
- .vfmin = 50,
- .vfmax = 90,
- .dclkmax = 6400000,
-};
-
-static struct atmel_lcdfb_pdata __initdata atevklcd10x_lcdc_data = {
- .default_bpp = 16,
- .default_dmacon = ATMEL_LCDC_DMAEN | ATMEL_LCDC_DMA2DEN,
- .default_lcdcon2 = (ATMEL_LCDC_DISTYPE_TFT
- | ATMEL_LCDC_CLKMOD_ALWAYSACTIVE
- | ATMEL_LCDC_MEMOR_BIG),
- .default_monspecs = &atevklcd10x_default_monspecs,
- .guard_time = 2,
-};
-#endif
-
-static void atevklcd10x_lcdc_power_control(struct atmel_lcdfb_pdata *pdata, int on)
-{
- gpio_set_value(GPIO_PIN_PB(15), on);
-}
-
-static int __init atevklcd10x_init(void)
-{
- /* PB15 is connected to the enable line on the boost regulator
- * controlling the backlight for the LCD panel.
- */
- at32_select_gpio(GPIO_PIN_PB(15), AT32_GPIOF_OUTPUT);
- gpio_request(GPIO_PIN_PB(15), "backlight");
- gpio_direction_output(GPIO_PIN_PB(15), 0);
-
- atevklcd10x_lcdc_data.atmel_lcdfb_power_control =
- atevklcd10x_lcdc_power_control;
-
- at32_add_device_lcdc(0, &atevklcd10x_lcdc_data,
- fbmem_start, fbmem_size,
-#ifdef CONFIG_BOARD_ATNGW100_MKII
- ATMEL_LCDC_PRI_18BIT | ATMEL_LCDC_PC_DVAL
-#else
- ATMEL_LCDC_ALT_18BIT | ATMEL_LCDC_PE_DVAL
-#endif
- );
-
- at32_add_device_ac97c(0, &ac97c0_data, AC97C_BOTH);
-
- return 0;
-}
-postcore_initcall(atevklcd10x_init);
diff --git a/arch/avr32/boards/atngw100/flash.c b/arch/avr32/boards/atngw100/flash.c
deleted file mode 100644
index 55ccc9c..0000000
--- a/arch/avr32/boards/atngw100/flash.c
+++ /dev/null
@@ -1,98 +0,0 @@
-/*
- * ATNGW100 board-specific flash initialization
- *
- * Copyright (C) 2005-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/init.h>
-#include <linux/platform_device.h>
-#include <linux/mtd/mtd.h>
-#include <linux/mtd/partitions.h>
-#include <linux/mtd/physmap.h>
-
-#include <mach/smc.h>
-
-static struct smc_timing flash_timing __initdata = {
- .ncs_read_setup = 0,
- .nrd_setup = 40,
- .ncs_write_setup = 0,
- .nwe_setup = 10,
-
- .ncs_read_pulse = 80,
- .nrd_pulse = 40,
- .ncs_write_pulse = 65,
- .nwe_pulse = 55,
-
- .read_cycle = 120,
- .write_cycle = 120,
-};
-
-static struct smc_config flash_config __initdata = {
- .bus_width = 2,
- .nrd_controlled = 1,
- .nwe_controlled = 1,
- .byte_write = 1,
-};
-
-static struct mtd_partition flash_parts[] = {
- {
- .name = "u-boot",
- .offset = 0x00000000,
- .size = 0x00020000, /* 128 KiB */
- .mask_flags = MTD_WRITEABLE,
- },
- {
- .name = "root",
- .offset = 0x00020000,
- .size = 0x007d0000,
- },
- {
- .name = "env",
- .offset = 0x007f0000,
- .size = 0x00010000,
- .mask_flags = MTD_WRITEABLE,
- },
-};
-
-static struct physmap_flash_data flash_data = {
- .width = 2,
- .nr_parts = ARRAY_SIZE(flash_parts),
- .parts = flash_parts,
-};
-
-static struct resource flash_resource = {
- .start = 0x00000000,
- .end = 0x007fffff,
- .flags = IORESOURCE_MEM,
-};
-
-static struct platform_device flash_device = {
- .name = "physmap-flash",
- .id = 0,
- .resource = &flash_resource,
- .num_resources = 1,
- .dev = {
- .platform_data = &flash_data,
- },
-};
-
-/* This needs to be called after the SMC has been initialized */
-static int __init atngw100_flash_init(void)
-{
- int ret;
-
- smc_set_timing(&flash_config, &flash_timing);
- ret = smc_set_configuration(0, &flash_config);
- if (ret < 0) {
- printk(KERN_ERR "atngw100: failed to set NOR flash timing\n");
- return ret;
- }
-
- platform_device_register(&flash_device);
-
- return 0;
-}
-device_initcall(atngw100_flash_init);
diff --git a/arch/avr32/boards/atngw100/mrmt.c b/arch/avr32/boards/atngw100/mrmt.c
deleted file mode 100644
index 99b0a79..0000000
--- a/arch/avr32/boards/atngw100/mrmt.c
+++ /dev/null
@@ -1,382 +0,0 @@
-/*
- * Board-specific setup code for Remote Media Terminal 1 (RMT1)
- * add-on board for the ATNGW100 Network Gateway
- *
- * Copyright (C) 2008 Mediama Technologies
- * Based on ATNGW100 Network Gateway (Copyright (C) Atmel)
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/gpio.h>
-#include <linux/init.h>
-#include <linux/irq.h>
-#include <linux/linkage.h>
-#include <linux/platform_device.h>
-#include <linux/types.h>
-#include <linux/fb.h>
-#include <linux/leds.h>
-#include <linux/pwm.h>
-#include <linux/leds_pwm.h>
-#include <linux/input.h>
-#include <linux/gpio_keys.h>
-#include <linux/spi/spi.h>
-#include <linux/spi/ads7846.h>
-
-#include <video/atmel_lcdc.h>
-#include <sound/atmel-ac97c.h>
-
-#include <asm/delay.h>
-#include <asm/io.h>
-#include <asm/setup.h>
-
-#include <mach/at32ap700x.h>
-#include <mach/board.h>
-#include <mach/init.h>
-#include <mach/portmux.h>
-
-/* Define board-specifoic GPIO assignments */
-#define PIN_LCD_BL GPIO_PIN_PA(28)
-#define PWM_CH_BL 0 /* Must match with GPIO pin definition */
-#define PIN_LCD_DISP GPIO_PIN_PA(31)
-#define PIN_AC97_RST_N GPIO_PIN_PA(30)
-#define PB_EXTINT_BASE 25
-#define TS_IRQ 0
-#define PIN_TS_EXTINT GPIO_PIN_PB(PB_EXTINT_BASE+TS_IRQ)
-#define PIN_PB_LEFT GPIO_PIN_PB(11)
-#define PIN_PB_RIGHT GPIO_PIN_PB(12)
-#define PIN_PWR_SW_N GPIO_PIN_PB(14)
-#define PIN_PWR_ON GPIO_PIN_PB(13)
-#define PIN_ZB_RST_N GPIO_PIN_PA(21)
-#define PIN_BT_RST GPIO_PIN_PA(22)
-#define PIN_LED_SYS GPIO_PIN_PA(16)
-#define PIN_LED_A GPIO_PIN_PA(19)
-#define PIN_LED_B GPIO_PIN_PE(19)
-
-#ifdef CONFIG_BOARD_MRMT_LCD_LQ043T3DX0X
-/* Sharp LQ043T3DX0x (or compatible) panel */
-static struct fb_videomode __initdata lcd_fb_modes[] = {
- {
- .name = "480x272 @ 59.94Hz",
- .refresh = 59.94,
- .xres = 480, .yres = 272,
- .pixclock = KHZ2PICOS(9000),
-
- .left_margin = 2, .right_margin = 2,
- .upper_margin = 3, .lower_margin = 9,
- .hsync_len = 41, .vsync_len = 1,
-
- .sync = 0,
- .vmode = FB_VMODE_NONINTERLACED,
- },
-};
-
-static struct fb_monspecs __initdata lcd_fb_default_monspecs = {
- .manufacturer = "SHA",
- .monitor = "LQ043T3DX02",
- .modedb = lcd_fb_modes,
- .modedb_len = ARRAY_SIZE(lcd_fb_modes),
- .hfmin = 14915,
- .hfmax = 17638,
- .vfmin = 53,
- .vfmax = 61,
- .dclkmax = 9260000,
-};
-
-static struct atmel_lcdfb_pdata __initdata rmt_lcdc_data = {
- .default_bpp = 24,
- .default_dmacon = ATMEL_LCDC_DMAEN | ATMEL_LCDC_DMA2DEN,
- .default_lcdcon2 = (ATMEL_LCDC_DISTYPE_TFT
- | ATMEL_LCDC_CLKMOD_ALWAYSACTIVE
- | ATMEL_LCDC_INVCLK_NORMAL
- | ATMEL_LCDC_MEMOR_BIG),
- .lcd_wiring_mode = ATMEL_LCDC_WIRING_RGB,
- .default_monspecs = &lcd_fb_default_monspecs,
- .guard_time = 2,
-};
-#endif
-
-#ifdef CONFIG_BOARD_MRMT_LCD_KWH043GM08
-/* Sharp KWH043GM08-Fxx (or compatible) panel */
-static struct fb_videomode __initdata lcd_fb_modes[] = {
- {
- .name = "480x272 @ 59.94Hz",
- .refresh = 59.94,
- .xres = 480, .yres = 272,
- .pixclock = KHZ2PICOS(9000),
-
- .left_margin = 2, .right_margin = 2,
- .upper_margin = 3, .lower_margin = 9,
- .hsync_len = 41, .vsync_len = 1,
-
- .sync = 0,
- .vmode = FB_VMODE_NONINTERLACED,
- },
-};
-
-static struct fb_monspecs __initdata lcd_fb_default_monspecs = {
- .manufacturer = "FOR",
- .monitor = "KWH043GM08",
- .modedb = lcd_fb_modes,
- .modedb_len = ARRAY_SIZE(lcd_fb_modes),
- .hfmin = 14915,
- .hfmax = 17638,
- .vfmin = 53,
- .vfmax = 61,
- .dclkmax = 9260000,
-};
-
-static struct atmel_lcdfb_pdata __initdata rmt_lcdc_data = {
- .default_bpp = 24,
- .default_dmacon = ATMEL_LCDC_DMAEN | ATMEL_LCDC_DMA2DEN,
- .default_lcdcon2 = (ATMEL_LCDC_DISTYPE_TFT
- | ATMEL_LCDC_CLKMOD_ALWAYSACTIVE
- | ATMEL_LCDC_INVCLK_INVERTED
- | ATMEL_LCDC_MEMOR_BIG),
- .lcd_wiring_mode = ATMEL_LCDC_WIRING_RGB,
- .default_monspecs = &lcd_fb_default_monspecs,
- .guard_time = 2,
-};
-#endif
-
-#ifdef CONFIG_BOARD_MRMT_AC97
-static struct ac97c_platform_data __initdata ac97c0_data = {
- .reset_pin = PIN_AC97_RST_N,
-};
-#endif
-
-#ifdef CONFIG_BOARD_MRMT_UCB1400_TS
-/* NOTE: IRQ assignment relies on kernel module parameter */
-static struct platform_device rmt_ts_device = {
- .name = "ucb1400_ts",
- .id = -1,
-};
-#endif
-
-#ifdef CONFIG_BOARD_MRMT_BL_PWM
-/* PWM LEDs: LCD Backlight, etc */
-static struct pwm_lookup pwm_lookup[] = {
- PWM_LOOKUP("at91sam9rl-pwm", PWM_CH_BL, "leds_pwm", "ds1",
- 5000, PWM_POLARITY_INVERSED),
-};
-
-static struct led_pwm pwm_leds[] = {
- {
- .name = "backlight",
- .max_brightness = 255,
- },
-};
-
-static struct led_pwm_platform_data pwm_data = {
- .num_leds = ARRAY_SIZE(pwm_leds),
- .leds = pwm_leds,
-};
-
-static struct platform_device leds_pwm = {
- .name = "leds_pwm",
- .id = -1,
- .dev = {
- .platform_data = &pwm_data,
- },
-};
-#endif
-
-#ifdef CONFIG_BOARD_MRMT_ADS7846_TS
-static int ads7846_pendown_state(void)
-{
- return !gpio_get_value( PIN_TS_EXTINT ); /* PENIRQ.*/
-}
-
-static struct ads7846_platform_data ads_info = {
- .model = 7846,
- .keep_vref_on = 0, /* Use external VREF pin */
- .vref_delay_usecs = 0,
- .vref_mv = 3300, /* VREF = 3.3V */
- .settle_delay_usecs = 800,
- .penirq_recheck_delay_usecs = 800,
- .x_plate_ohms = 750,
- .y_plate_ohms = 300,
- .pressure_max = 4096,
- .debounce_max = 1,
- .debounce_rep = 0,
- .debounce_tol = (~0),
- .get_pendown_state = ads7846_pendown_state,
- .filter = NULL,
- .filter_init = NULL,
-};
-
-static struct spi_board_info spi01_board_info[] __initdata = {
- {
- .modalias = "ads7846",
- .max_speed_hz = 31250*26,
- .bus_num = 0,
- .chip_select = 1,
- .platform_data = &ads_info,
- .irq = AT32_EXTINT(TS_IRQ),
- },
-};
-#endif
-
-/* GPIO Keys: left, right, power, etc */
-static const struct gpio_keys_button rmt_gpio_keys_buttons[] = {
- [0] = {
- .type = EV_KEY,
- .code = KEY_POWER,
- .gpio = PIN_PWR_SW_N,
- .active_low = 1,
- .desc = "power button",
- },
- [1] = {
- .type = EV_KEY,
- .code = KEY_LEFT,
- .gpio = PIN_PB_LEFT,
- .active_low = 1,
- .desc = "left button",
- },
- [2] = {
- .type = EV_KEY,
- .code = KEY_RIGHT,
- .gpio = PIN_PB_RIGHT,
- .active_low = 1,
- .desc = "right button",
- },
-};
-
-static const struct gpio_keys_platform_data rmt_gpio_keys_data = {
- .nbuttons = ARRAY_SIZE(rmt_gpio_keys_buttons),
- .buttons = (void *) rmt_gpio_keys_buttons,
-};
-
-static struct platform_device rmt_gpio_keys = {
- .name = "gpio-keys",
- .id = -1,
- .dev = {
- .platform_data = (void *) &rmt_gpio_keys_data,
- }
-};
-
-#ifdef CONFIG_BOARD_MRMT_RTC_I2C
-static struct i2c_board_info __initdata mrmt1_i2c_rtc = {
- I2C_BOARD_INFO("s35390a", 0x30),
- .irq = 0,
-};
-#endif
-
-static void mrmt_power_off(void)
-{
- /* PWR_ON=0 will force power off */
- gpio_set_value( PIN_PWR_ON, 0 );
-}
-
-static int __init mrmt1_init(void)
-{
- gpio_set_value( PIN_PWR_ON, 1 ); /* Ensure PWR_ON is enabled */
-
- pm_power_off = mrmt_power_off;
-
- /* Setup USARTS (other than console) */
- at32_map_usart(2, 1, 0); /* USART 2: /dev/ttyS1, RMT1:DB9M */
- at32_map_usart(3, 2, ATMEL_USART_RTS | ATMEL_USART_CTS);
- /* USART 3: /dev/ttyS2, RMT1:Wireless, w/ RTS/CTS */
- at32_add_device_usart(1);
- at32_add_device_usart(2);
-
- /* Select GPIO Key pins */
- at32_select_gpio( PIN_PWR_SW_N, AT32_GPIOF_DEGLITCH);
- at32_select_gpio( PIN_PB_LEFT, AT32_GPIOF_DEGLITCH);
- at32_select_gpio( PIN_PB_RIGHT, AT32_GPIOF_DEGLITCH);
- platform_device_register(&rmt_gpio_keys);
-
-#ifdef CONFIG_BOARD_MRMT_RTC_I2C
- i2c_register_board_info(0, &mrmt1_i2c_rtc, 1);
-#endif
-
-#ifndef CONFIG_BOARD_MRMT_LCD_DISABLE
- /* User "alternate" LCDC inferface on Port E & D */
- /* NB: exclude LCDC_CC pin, as NGW100 reserves it for other use */
- at32_add_device_lcdc(0, &rmt_lcdc_data,
- fbmem_start, fbmem_size,
- (ATMEL_LCDC_ALT_24BIT | ATMEL_LCDC_PE_DVAL ) );
-#endif
-
-#ifdef CONFIG_BOARD_MRMT_AC97
- at32_add_device_ac97c(0, &ac97c0_data, AC97C_BOTH);
-#endif
-
-#ifdef CONFIG_BOARD_MRMT_ADS7846_TS
- /* Select the Touchscreen interrupt pin mode */
- at32_select_periph( GPIO_PIOB_BASE, 1 << (PB_EXTINT_BASE+TS_IRQ),
- GPIO_PERIPH_A, AT32_GPIOF_DEGLITCH);
- irq_set_irq_type(AT32_EXTINT(TS_IRQ), IRQ_TYPE_EDGE_FALLING);
- at32_spi_setup_slaves(0,spi01_board_info,ARRAY_SIZE(spi01_board_info));
- spi_register_board_info(spi01_board_info,ARRAY_SIZE(spi01_board_info));
-#endif
-
-#ifdef CONFIG_BOARD_MRMT_UCB1400_TS
- /* Select the Touchscreen interrupt pin mode */
- at32_select_periph( GPIO_PIOB_BASE, 1 << (PB_EXTINT_BASE+TS_IRQ),
- GPIO_PERIPH_A, AT32_GPIOF_DEGLITCH);
- platform_device_register(&rmt_ts_device);
-#endif
-
- at32_select_gpio( PIN_LCD_DISP, AT32_GPIOF_OUTPUT );
- gpio_request( PIN_LCD_DISP, "LCD_DISP" );
- gpio_direction_output( PIN_LCD_DISP, 0 ); /* LCD DISP */
-#ifdef CONFIG_BOARD_MRMT_LCD_DISABLE
- /* Keep Backlight and DISP off */
- at32_select_gpio( PIN_LCD_BL, AT32_GPIOF_OUTPUT );
- gpio_request( PIN_LCD_BL, "LCD_BL" );
- gpio_direction_output( PIN_LCD_BL, 0 ); /* Backlight */
-#else
- gpio_set_value( PIN_LCD_DISP, 1 ); /* DISP asserted first */
-#ifdef CONFIG_BOARD_MRMT_BL_PWM
- /* Use PWM for Backlight controls */
- at32_add_device_pwm(1 << PWM_CH_BL);
- pwm_add_table(pwm_lookup, ARRAY_SIZE(pwm_lookup));
- platform_device_register(&leds_pwm);
-#else
- /* Backlight always on */
- udelay( 1 );
- at32_select_gpio( PIN_LCD_BL, AT32_GPIOF_OUTPUT );
- gpio_request( PIN_LCD_BL, "LCD_BL" );
- gpio_direction_output( PIN_LCD_BL, 1 );
-#endif
-#endif
-
- /* Make sure BT and Zigbee modules in reset */
- at32_select_gpio( PIN_BT_RST, AT32_GPIOF_OUTPUT );
- gpio_request( PIN_BT_RST, "BT_RST" );
- gpio_direction_output( PIN_BT_RST, 1 );
- /* BT Module in Reset */
-
- at32_select_gpio( PIN_ZB_RST_N, AT32_GPIOF_OUTPUT );
- gpio_request( PIN_ZB_RST_N, "ZB_RST_N" );
- gpio_direction_output( PIN_ZB_RST_N, 0 );
- /* XBee Module in Reset */
-
-#ifdef CONFIG_BOARD_MRMT_WIRELESS_ZB
- udelay( 1000 );
- /* Unreset the XBee Module */
- gpio_set_value( PIN_ZB_RST_N, 1 );
-#endif
-#ifdef CONFIG_BOARD_MRMT_WIRELESS_BT
- udelay( 1000 );
- /* Unreset the BT Module */
- gpio_set_value( PIN_BT_RST, 0 );
-#endif
-
- return 0;
-}
-arch_initcall(mrmt1_init);
-
-static int __init mrmt1_early_init(void)
-{
- /* To maintain power-on signal in case boot loader did not already */
- at32_select_gpio( PIN_PWR_ON, AT32_GPIOF_OUTPUT );
- gpio_request( PIN_PWR_ON, "PIN_PWR_ON" );
- gpio_direction_output( PIN_PWR_ON, 1 );
-
- return 0;
-}
-core_initcall(mrmt1_early_init);
diff --git a/arch/avr32/boards/atngw100/setup.c b/arch/avr32/boards/atngw100/setup.c
deleted file mode 100644
index afeae89..0000000
--- a/arch/avr32/boards/atngw100/setup.c
+++ /dev/null
@@ -1,324 +0,0 @@
-/*
- * Board-specific setup code for the ATNGW100 Network Gateway
- *
- * Copyright (C) 2005-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/clk.h>
-#include <linux/etherdevice.h>
-#include <linux/gpio.h>
-#include <linux/irq.h>
-#include <linux/i2c.h>
-#include <linux/i2c-gpio.h>
-#include <linux/init.h>
-#include <linux/linkage.h>
-#include <linux/platform_device.h>
-#include <linux/types.h>
-#include <linux/leds.h>
-#include <linux/spi/spi.h>
-#include <linux/atmel-mci.h>
-#include <linux/usb/atmel_usba_udc.h>
-
-#include <asm/io.h>
-#include <asm/setup.h>
-
-#include <mach/at32ap700x.h>
-#include <mach/board.h>
-#include <mach/init.h>
-#include <mach/portmux.h>
-
-/* Oscillator frequencies. These are board-specific */
-unsigned long at32_board_osc_rates[3] = {
- [0] = 32768, /* 32.768 kHz on RTC osc */
- [1] = 20000000, /* 20 MHz on osc0 */
- [2] = 12000000, /* 12 MHz on osc1 */
-};
-
-/*
- * The ATNGW100 mkII is very similar to the ATNGW100. Both have the AT32AP7000
- * chip on board; the difference is that the ATNGW100 mkII has 128 MB 32-bit
- * SDRAM (the ATNGW100 has 32 MB 16-bit SDRAM) and 256 MB 16-bit NAND flash
- * (the ATNGW100 has none.)
- *
- * The RAM difference is handled by the boot loader, so the only difference we
- * end up handling here is the NAND flash, EBI pin reservation and if LCDC or
- * MACB1 should be enabled.
- */
-#ifdef CONFIG_BOARD_ATNGW100_MKII
-#include <linux/mtd/partitions.h>
-#include <mach/smc.h>
-
-static struct smc_timing nand_timing __initdata = {
- .ncs_read_setup = 0,
- .nrd_setup = 10,
- .ncs_write_setup = 0,
- .nwe_setup = 10,
-
- .ncs_read_pulse = 30,
- .nrd_pulse = 15,
- .ncs_write_pulse = 30,
- .nwe_pulse = 15,
-
- .read_cycle = 30,
- .write_cycle = 30,
-
- .ncs_read_recover = 0,
- .nrd_recover = 15,
- .ncs_write_recover = 0,
- /* WE# high -> RE# low min 60 ns */
- .nwe_recover = 50,
-};
-
-static struct smc_config nand_config __initdata = {
- .bus_width = 2,
- .nrd_controlled = 1,
- .nwe_controlled = 1,
- .nwait_mode = 0,
- .byte_write = 0,
- .tdf_cycles = 2,
- .tdf_mode = 0,
-};
-
-static struct mtd_partition nand_partitions[] = {
- {
- .name = "main",
- .offset = 0x00000000,
- .size = MTDPART_SIZ_FULL,
- },
-};
-
-
-static struct atmel_nand_data atngw100mkii_nand_data __initdata = {
- .cle = 21,
- .ale = 22,
- .rdy_pin = GPIO_PIN_PB(28),
- .enable_pin = GPIO_PIN_PE(23),
- .bus_width_16 = true,
- .ecc_mode = NAND_ECC_SOFT,
- .parts = nand_partitions,
- .num_parts = ARRAY_SIZE(nand_partitions),
-};
-#endif
-
-/* Initialized by bootloader-specific startup code. */
-struct tag *bootloader_tags __initdata;
-
-struct eth_addr {
- u8 addr[6];
-};
-static struct eth_addr __initdata hw_addr[2];
-static struct macb_platform_data __initdata eth_data[2];
-
-static struct spi_board_info spi0_board_info[] __initdata = {
- {
- .modalias = "mtd_dataflash",
- .max_speed_hz = 8000000,
- .chip_select = 0,
- },
-};
-
-static struct mci_platform_data __initdata mci0_data = {
- .slot[0] = {
- .bus_width = 4,
-#if defined(CONFIG_BOARD_ATNGW100_MKII)
- .detect_pin = GPIO_PIN_PC(25),
- .wp_pin = GPIO_PIN_PE(22),
-#else
- .detect_pin = GPIO_PIN_PC(25),
- .wp_pin = GPIO_PIN_PE(0),
-#endif
- },
-};
-
-static struct usba_platform_data atngw100_usba_data __initdata = {
-#if defined(CONFIG_BOARD_ATNGW100_MKII)
- .vbus_pin = GPIO_PIN_PE(26),
-#else
- .vbus_pin = -ENODEV,
-#endif
-};
-
-/*
- * The next two functions should go away as the boot loader is
- * supposed to initialize the macb address registers with a valid
- * ethernet address. But we need to keep it around for a while until
- * we can be reasonably sure the boot loader does this.
- *
- * The phy_id is ignored as the driver will probe for it.
- */
-static int __init parse_tag_ethernet(struct tag *tag)
-{
- int i;
-
- i = tag->u.ethernet.mac_index;
- if (i < ARRAY_SIZE(hw_addr))
- memcpy(hw_addr[i].addr, tag->u.ethernet.hw_address,
- sizeof(hw_addr[i].addr));
-
- return 0;
-}
-__tagtable(ATAG_ETHERNET, parse_tag_ethernet);
-
-static void __init set_hw_addr(struct platform_device *pdev)
-{
- struct resource *res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- const u8 *addr;
- void __iomem *regs;
- struct clk *pclk;
-
- if (!res)
- return;
- if (pdev->id >= ARRAY_SIZE(hw_addr))
- return;
-
- addr = hw_addr[pdev->id].addr;
- if (!is_valid_ether_addr(addr))
- return;
-
- /*
- * Since this is board-specific code, we'll cheat and use the
- * physical address directly as we happen to know that it's
- * the same as the virtual address.
- */
- regs = (void __iomem __force *)res->start;
- pclk = clk_get(&pdev->dev, "pclk");
- if (IS_ERR(pclk))
- return;
-
- clk_enable(pclk);
- __raw_writel((addr[3] << 24) | (addr[2] << 16)
- | (addr[1] << 8) | addr[0], regs + 0x98);
- __raw_writel((addr[5] << 8) | addr[4], regs + 0x9c);
- clk_disable(pclk);
- clk_put(pclk);
-}
-
-void __init setup_board(void)
-{
- at32_map_usart(1, 0, 0); /* USART 1: /dev/ttyS0, DB9 */
- at32_setup_serial_console(0);
-}
-
-static const struct gpio_led ngw_leds[] = {
- { .name = "sys", .gpio = GPIO_PIN_PA(16), .active_low = 1,
- .default_trigger = "heartbeat",
- },
- { .name = "a", .gpio = GPIO_PIN_PA(19), .active_low = 1, },
- { .name = "b", .gpio = GPIO_PIN_PE(19), .active_low = 1, },
-};
-
-static const struct gpio_led_platform_data ngw_led_data = {
- .num_leds = ARRAY_SIZE(ngw_leds),
- .leds = (void *) ngw_leds,
-};
-
-static struct platform_device ngw_gpio_leds = {
- .name = "leds-gpio",
- .id = -1,
- .dev = {
- .platform_data = (void *) &ngw_led_data,
- }
-};
-
-static struct i2c_gpio_platform_data i2c_gpio_data = {
- .sda_pin = GPIO_PIN_PA(6),
- .scl_pin = GPIO_PIN_PA(7),
- .sda_is_open_drain = 1,
- .scl_is_open_drain = 1,
- .udelay = 2, /* close to 100 kHz */
-};
-
-static struct platform_device i2c_gpio_device = {
- .name = "i2c-gpio",
- .id = 0,
- .dev = {
- .platform_data = &i2c_gpio_data,
- },
-};
-
-static struct i2c_board_info __initdata i2c_info[] = {
- /* NOTE: original ATtiny24 firmware is at address 0x0b */
-};
-
-static int __init atngw100_init(void)
-{
- unsigned i;
-
- /*
- * ATNGW100 mkII uses 32-bit SDRAM interface. Reserve the
- * SDRAM-specific pins so that nobody messes with them.
- */
-#ifdef CONFIG_BOARD_ATNGW100_MKII
- at32_reserve_pin(GPIO_PIOE_BASE, ATMEL_EBI_PE_DATA_ALL);
-
- smc_set_timing(&nand_config, &nand_timing);
- smc_set_configuration(3, &nand_config);
- at32_add_device_nand(0, &atngw100mkii_nand_data);
-#endif
-
- at32_add_device_usart(0);
-
- set_hw_addr(at32_add_device_eth(0, ð_data[0]));
-#ifndef CONFIG_BOARD_ATNGW100_MKII_LCD
- set_hw_addr(at32_add_device_eth(1, ð_data[1]));
-#endif
-
- at32_add_device_spi(0, spi0_board_info, ARRAY_SIZE(spi0_board_info));
- at32_add_device_mci(0, &mci0_data);
- at32_add_device_usba(0, &atngw100_usba_data);
-
- for (i = 0; i < ARRAY_SIZE(ngw_leds); i++) {
- at32_select_gpio(ngw_leds[i].gpio,
- AT32_GPIOF_OUTPUT | AT32_GPIOF_HIGH);
- }
- platform_device_register(&ngw_gpio_leds);
-
- /* all these i2c/smbus pins should have external pullups for
- * open-drain sharing among all I2C devices. SDA and SCL do;
- * PB28/EXTINT3 (ATNGW100) and PE21 (ATNGW100 mkII) doesn't; it should
- * be SMBALERT# (for PMBus), but it's not available off-board.
- */
-#ifdef CONFIG_BOARD_ATNGW100_MKII
- at32_select_periph(GPIO_PIOE_BASE, 1 << 21, 0, AT32_GPIOF_PULLUP);
-#else
- at32_select_periph(GPIO_PIOB_BASE, 1 << 28, 0, AT32_GPIOF_PULLUP);
-#endif
- at32_select_gpio(i2c_gpio_data.sda_pin,
- AT32_GPIOF_MULTIDRV | AT32_GPIOF_OUTPUT | AT32_GPIOF_HIGH);
- at32_select_gpio(i2c_gpio_data.scl_pin,
- AT32_GPIOF_MULTIDRV | AT32_GPIOF_OUTPUT | AT32_GPIOF_HIGH);
- platform_device_register(&i2c_gpio_device);
- i2c_register_board_info(0, i2c_info, ARRAY_SIZE(i2c_info));
-
- return 0;
-}
-postcore_initcall(atngw100_init);
-
-static int __init atngw100_arch_init(void)
-{
- /* PB30 (ATNGW100) and PE30 (ATNGW100 mkII) is the otherwise unused
- * jumper on the mainboard, with an external pullup; the jumper grounds
- * it. Use it however you like, including letting U-Boot or Linux tweak
- * boot sequences.
- */
-#ifdef CONFIG_BOARD_ATNGW100_MKII
- at32_select_gpio(GPIO_PIN_PE(30), 0);
- gpio_request(GPIO_PIN_PE(30), "j15");
- gpio_direction_input(GPIO_PIN_PE(30));
- gpio_export(GPIO_PIN_PE(30), false);
-#else
- at32_select_gpio(GPIO_PIN_PB(30), 0);
- gpio_request(GPIO_PIN_PB(30), "j15");
- gpio_direction_input(GPIO_PIN_PB(30));
- gpio_export(GPIO_PIN_PB(30), false);
-#endif
-
- /* set_irq_type() after the arch_initcall for EIC has run, and
- * before the I2C subsystem could try using this IRQ.
- */
- return irq_set_irq_type(AT32_EXTINT(3), IRQ_TYPE_EDGE_FALLING);
-}
-arch_initcall(atngw100_arch_init);
diff --git a/arch/avr32/boards/atstk1000/Kconfig b/arch/avr32/boards/atstk1000/Kconfig
deleted file mode 100644
index 8dc4821..0000000
--- a/arch/avr32/boards/atstk1000/Kconfig
+++ /dev/null
@@ -1,109 +0,0 @@
-# STK1000 customization
-
-if BOARD_ATSTK1000
-
-choice
- prompt "ATSTK1000 CPU daughterboard type"
- default BOARD_ATSTK1002
-
-config BOARD_ATSTK1002
- bool "ATSTK1002"
- select CPU_AT32AP7000
-
-config BOARD_ATSTK1003
- bool "ATSTK1003"
- select CPU_AT32AP7001
-
-config BOARD_ATSTK1004
- bool "ATSTK1004"
- select CPU_AT32AP7002
-
-config BOARD_ATSTK1006
- bool "ATSTK1006"
- select CPU_AT32AP7000
-
-endchoice
-
-
-config BOARD_ATSTK100X_CUSTOM
- bool "Non-default STK1002/STK1003/STK1004 jumper settings"
- help
- You will normally leave the jumpers on the CPU card at their
- default settings. If you need to use certain peripherals,
- you will need to change some of those jumpers.
-
-if BOARD_ATSTK100X_CUSTOM
-
-config BOARD_ATSTK100X_SW1_CUSTOM
- bool "SW1: use SSC1 (not SPI0)"
- help
- This also prevents using the external DAC as an audio interface,
- and means you can't initialize the on-board QVGA display.
-
-config BOARD_ATSTK100X_SW2_CUSTOM
- bool "SW2: use IRDA or TIMER0 (not UART-A, MMC/SD, and PS2-A)"
- help
- If you change this you'll want an updated boot loader putting
- the console on UART-C not UART-A.
-
-config BOARD_ATSTK100X_SW3_CUSTOM
- bool "SW3: use TIMER1 (not SSC0 and GCLK)"
- help
- This also prevents using the external DAC as an audio interface.
-
-config BOARD_ATSTK100X_SW4_CUSTOM
- bool "SW4: use ISI/Camera (not GPIOs, SPI1, and PS2-B)"
- help
- To use the camera interface you'll need a custom card (on the
- PCI-format connector) connect a video sensor.
-
-config BOARD_ATSTK1002_SW5_CUSTOM
- bool "SW5: use MACB1 (not LCDC)"
- depends on BOARD_ATSTK1002
-
-config BOARD_ATSTK1002_SW6_CUSTOM
- bool "SW6: more GPIOs (not MACB0)"
- depends on BOARD_ATSTK1002
-
-endif # custom
-
-config BOARD_ATSTK100X_SPI1
- bool "Configure SPI1 controller"
- depends on !BOARD_ATSTK100X_SW4_CUSTOM
- help
- All the signals for the second SPI controller are available on
- GPIO lines and accessed through the J1 jumper block. Say "y"
- here to configure that SPI controller.
-
-config BOARD_ATSTK1000_J2_LED
- bool
- default BOARD_ATSTK1000_J2_LED8 || BOARD_ATSTK1000_J2_RGB
-
-choice
- prompt "LEDs connected to J2:"
- depends on LEDS_GPIO && !BOARD_ATSTK100X_SW4_CUSTOM
- optional
- help
- Select this if you have jumpered the J2 jumper block to the
- LED0..LED7 amber leds, or to the RGB leds, using a ten-pin
- IDC cable. A default "heartbeat" trigger is provided, but
- you can of course override this.
-
-config BOARD_ATSTK1000_J2_LED8
- bool "LED0..LED7"
- help
- Select this if J2 is jumpered to LED0..LED7 amber leds.
-
-config BOARD_ATSTK1000_J2_RGB
- bool "RGB leds"
- help
- Select this if J2 is jumpered to the RGB leds.
-
-endchoice
-
-config BOARD_ATSTK1000_EXTDAC
- bool
- depends on !BOARD_ATSTK100X_SW1_CUSTOM && !BOARD_ATSTK100X_SW3_CUSTOM
- default y
-
-endif # stk 1000
diff --git a/arch/avr32/boards/atstk1000/Makefile b/arch/avr32/boards/atstk1000/Makefile
deleted file mode 100644
index edecee0..0000000
--- a/arch/avr32/boards/atstk1000/Makefile
+++ /dev/null
@@ -1,5 +0,0 @@
-obj-y += setup.o flash.o
-obj-$(CONFIG_BOARD_ATSTK1002) += atstk1002.o
-obj-$(CONFIG_BOARD_ATSTK1003) += atstk1003.o
-obj-$(CONFIG_BOARD_ATSTK1004) += atstk1004.o
-obj-$(CONFIG_BOARD_ATSTK1006) += atstk1002.o
diff --git a/arch/avr32/boards/atstk1000/atstk1000.h b/arch/avr32/boards/atstk1000/atstk1000.h
deleted file mode 100644
index 653cc09..0000000
--- a/arch/avr32/boards/atstk1000/atstk1000.h
+++ /dev/null
@@ -1,17 +0,0 @@
-/*
- * ATSTK1000 setup code: Daughterboard interface
- *
- * Copyright (C) 2007 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ARCH_AVR32_BOARDS_ATSTK1000_ATSTK1000_H
-#define __ARCH_AVR32_BOARDS_ATSTK1000_ATSTK1000_H
-
-extern struct atmel_lcdfb_pdata atstk1000_lcdc_data;
-
-void atstk1000_setup_j2_leds(void);
-
-#endif /* __ARCH_AVR32_BOARDS_ATSTK1000_ATSTK1000_H */
diff --git a/arch/avr32/boards/atstk1000/atstk1002.c b/arch/avr32/boards/atstk1000/atstk1002.c
deleted file mode 100644
index 6c80aba..0000000
--- a/arch/avr32/boards/atstk1000/atstk1002.c
+++ /dev/null
@@ -1,330 +0,0 @@
-/*
- * ATSTK1002/ATSTK1006 daughterboard-specific init code
- *
- * Copyright (C) 2005-2007 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/clk.h>
-#include <linux/etherdevice.h>
-#include <linux/init.h>
-#include <linux/kernel.h>
-#include <linux/platform_device.h>
-#include <linux/string.h>
-#include <linux/types.h>
-#include <linux/spi/spi.h>
-#include <linux/spi/at73c213.h>
-#include <linux/atmel-mci.h>
-
-#include <video/atmel_lcdc.h>
-
-#include <asm/io.h>
-#include <asm/setup.h>
-
-#include <mach/at32ap700x.h>
-#include <mach/board.h>
-#include <mach/init.h>
-#include <mach/portmux.h>
-
-#include "atstk1000.h"
-
-/* Oscillator frequencies. These are board specific */
-unsigned long at32_board_osc_rates[3] = {
- [0] = 32768, /* 32.768 kHz on RTC osc */
- [1] = 20000000, /* 20 MHz on osc0 */
- [2] = 12000000, /* 12 MHz on osc1 */
-};
-
-/*
- * The ATSTK1006 daughterboard is very similar to the ATSTK1002. Both
- * have the AT32AP7000 chip on board; the difference is that the
- * STK1006 has 128 MB SDRAM (the STK1002 uses the 8 MB SDRAM chip on
- * the STK1000 motherboard) and 256 MB NAND flash (the STK1002 has
- * none.)
- *
- * The RAM difference is handled by the boot loader, so the only
- * difference we end up handling here is the NAND flash.
- */
-#ifdef CONFIG_BOARD_ATSTK1006
-#include <linux/mtd/partitions.h>
-#include <mach/smc.h>
-
-static struct smc_timing nand_timing __initdata = {
- .ncs_read_setup = 0,
- .nrd_setup = 10,
- .ncs_write_setup = 0,
- .nwe_setup = 10,
-
- .ncs_read_pulse = 30,
- .nrd_pulse = 15,
- .ncs_write_pulse = 30,
- .nwe_pulse = 15,
-
- .read_cycle = 30,
- .write_cycle = 30,
-
- .ncs_read_recover = 0,
- .nrd_recover = 15,
- .ncs_write_recover = 0,
- /* WE# high -> RE# low min 60 ns */
- .nwe_recover = 50,
-};
-
-static struct smc_config nand_config __initdata = {
- .bus_width = 1,
- .nrd_controlled = 1,
- .nwe_controlled = 1,
- .nwait_mode = 0,
- .byte_write = 0,
- .tdf_cycles = 2,
- .tdf_mode = 0,
-};
-
-static struct mtd_partition nand_partitions[] = {
- {
- .name = "main",
- .offset = 0x00000000,
- .size = MTDPART_SIZ_FULL,
- },
-};
-
-static struct atmel_nand_data atstk1006_nand_data __initdata = {
- .cle = 21,
- .ale = 22,
- .rdy_pin = GPIO_PIN_PB(30),
- .enable_pin = GPIO_PIN_PB(29),
- .ecc_mode = NAND_ECC_SOFT,
- .parts = nand_partitions,
- .num_parts = ARRAY_SIZE(nand_partitions),
-};
-#endif
-
-struct eth_addr {
- u8 addr[6];
-};
-
-static struct eth_addr __initdata hw_addr[2];
-static struct macb_platform_data __initdata eth_data[2] = {
- {
- /*
- * The MDIO pullups on STK1000 are a bit too weak for
- * the autodetection to work properly, so we have to
- * mask out everything but the correct address.
- */
- .phy_mask = ~(1U << 16),
- },
- {
- .phy_mask = ~(1U << 17),
- },
-};
-
-#ifdef CONFIG_BOARD_ATSTK1000_EXTDAC
-static struct at73c213_board_info at73c213_data = {
- .ssc_id = 0,
- .shortname = "AVR32 STK1000 external DAC",
-};
-#endif
-
-#ifndef CONFIG_BOARD_ATSTK100X_SW1_CUSTOM
-static struct spi_board_info spi0_board_info[] __initdata = {
-#ifdef CONFIG_BOARD_ATSTK1000_EXTDAC
- {
- /* AT73C213 */
- .modalias = "at73c213",
- .max_speed_hz = 200000,
- .chip_select = 0,
- .mode = SPI_MODE_1,
- .platform_data = &at73c213_data,
- },
-#endif
- {
- /* QVGA display */
- .modalias = "ltv350qv",
- .max_speed_hz = 16000000,
- .chip_select = 1,
- .mode = SPI_MODE_3,
- },
-};
-#endif
-
-#ifdef CONFIG_BOARD_ATSTK100X_SPI1
-static struct spi_board_info spi1_board_info[] __initdata = { {
- /* patch in custom entries here */
-} };
-#endif
-
-/*
- * The next two functions should go away as the boot loader is
- * supposed to initialize the macb address registers with a valid
- * ethernet address. But we need to keep it around for a while until
- * we can be reasonably sure the boot loader does this.
- *
- * The phy_id is ignored as the driver will probe for it.
- */
-static int __init parse_tag_ethernet(struct tag *tag)
-{
- int i;
-
- i = tag->u.ethernet.mac_index;
- if (i < ARRAY_SIZE(hw_addr))
- memcpy(hw_addr[i].addr, tag->u.ethernet.hw_address,
- sizeof(hw_addr[i].addr));
-
- return 0;
-}
-__tagtable(ATAG_ETHERNET, parse_tag_ethernet);
-
-static void __init set_hw_addr(struct platform_device *pdev)
-{
- struct resource *res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- const u8 *addr;
- void __iomem *regs;
- struct clk *pclk;
-
- if (!res)
- return;
- if (pdev->id >= ARRAY_SIZE(hw_addr))
- return;
-
- addr = hw_addr[pdev->id].addr;
- if (!is_valid_ether_addr(addr))
- return;
-
- /*
- * Since this is board-specific code, we'll cheat and use the
- * physical address directly as we happen to know that it's
- * the same as the virtual address.
- */
- regs = (void __iomem __force *)res->start;
- pclk = clk_get(&pdev->dev, "pclk");
- if (IS_ERR(pclk))
- return;
-
- clk_enable(pclk);
- __raw_writel((addr[3] << 24) | (addr[2] << 16)
- | (addr[1] << 8) | addr[0], regs + 0x98);
- __raw_writel((addr[5] << 8) | addr[4], regs + 0x9c);
- clk_disable(pclk);
- clk_put(pclk);
-}
-
-#ifdef CONFIG_BOARD_ATSTK1000_EXTDAC
-static void __init atstk1002_setup_extdac(void)
-{
- struct clk *gclk;
- struct clk *pll;
-
- gclk = clk_get(NULL, "gclk0");
- if (IS_ERR(gclk))
- goto err_gclk;
- pll = clk_get(NULL, "pll0");
- if (IS_ERR(pll))
- goto err_pll;
-
- if (clk_set_parent(gclk, pll)) {
- pr_debug("STK1000: failed to set pll0 as parent for DAC clock\n");
- goto err_set_clk;
- }
-
- at32_select_periph(GPIO_PIOA_BASE, (1 << 30), GPIO_PERIPH_A, 0);
- at73c213_data.dac_clk = gclk;
-
-err_set_clk:
- clk_put(pll);
-err_pll:
- clk_put(gclk);
-err_gclk:
- return;
-}
-#else
-static void __init atstk1002_setup_extdac(void)
-{
-
-}
-#endif /* CONFIG_BOARD_ATSTK1000_EXTDAC */
-
-void __init setup_board(void)
-{
-#ifdef CONFIG_BOARD_ATSTK100X_SW2_CUSTOM
- at32_map_usart(0, 1, 0); /* USART 0/B: /dev/ttyS1, IRDA */
-#else
- at32_map_usart(1, 0, 0); /* USART 1/A: /dev/ttyS0, DB9 */
-#endif
- /* USART 2/unused: expansion connector */
- at32_map_usart(3, 2, 0); /* USART 3/C: /dev/ttyS2, DB9 */
-
- at32_setup_serial_console(0);
-}
-
-#ifndef CONFIG_BOARD_ATSTK100X_SW2_CUSTOM
-
-static struct mci_platform_data __initdata mci0_data = {
- .slot[0] = {
- .bus_width = 4,
-
-/* MMC card detect requires MACB0 *NOT* be used */
-#ifdef CONFIG_BOARD_ATSTK1002_SW6_CUSTOM
- .detect_pin = GPIO_PIN_PC(14), /* gpio30/sdcd */
- .wp_pin = GPIO_PIN_PC(15), /* gpio31/sdwp */
-#else
- .detect_pin = -ENODEV,
- .wp_pin = -ENODEV,
-#endif /* SW6 for sd{cd,wp} routing */
- },
-};
-
-#endif /* SW2 for MMC signal routing */
-
-static int __init atstk1002_init(void)
-{
- /*
- * ATSTK1000 uses 32-bit SDRAM interface. Reserve the
- * SDRAM-specific pins so that nobody messes with them.
- */
- at32_reserve_pin(GPIO_PIOE_BASE, ATMEL_EBI_PE_DATA_ALL);
-
-#ifdef CONFIG_BOARD_ATSTK1006
- smc_set_timing(&nand_config, &nand_timing);
- smc_set_configuration(3, &nand_config);
- at32_add_device_nand(0, &atstk1006_nand_data);
-#endif
-
-#ifdef CONFIG_BOARD_ATSTK100X_SW2_CUSTOM
- at32_add_device_usart(1);
-#else
- at32_add_device_usart(0);
-#endif
- at32_add_device_usart(2);
-
-#ifndef CONFIG_BOARD_ATSTK1002_SW6_CUSTOM
- set_hw_addr(at32_add_device_eth(0, ð_data[0]));
-#endif
-#ifndef CONFIG_BOARD_ATSTK100X_SW1_CUSTOM
- at32_add_device_spi(0, spi0_board_info, ARRAY_SIZE(spi0_board_info));
-#endif
-#ifdef CONFIG_BOARD_ATSTK100X_SPI1
- at32_add_device_spi(1, spi1_board_info, ARRAY_SIZE(spi1_board_info));
-#endif
-#ifndef CONFIG_BOARD_ATSTK100X_SW2_CUSTOM
- at32_add_device_mci(0, &mci0_data);
-#endif
-#ifdef CONFIG_BOARD_ATSTK1002_SW5_CUSTOM
- set_hw_addr(at32_add_device_eth(1, ð_data[1]));
-#else
- at32_add_device_lcdc(0, &atstk1000_lcdc_data,
- fbmem_start, fbmem_size,
- ATMEL_LCDC_PRI_24BIT | ATMEL_LCDC_PRI_CONTROL);
-#endif
- at32_add_device_usba(0, NULL);
-#ifndef CONFIG_BOARD_ATSTK100X_SW3_CUSTOM
- at32_add_device_ssc(0, ATMEL_SSC_TX);
-#endif
-
- atstk1000_setup_j2_leds();
- atstk1002_setup_extdac();
-
- return 0;
-}
-postcore_initcall(atstk1002_init);
diff --git a/arch/avr32/boards/atstk1000/atstk1003.c b/arch/avr32/boards/atstk1000/atstk1003.c
deleted file mode 100644
index ff7e232..0000000
--- a/arch/avr32/boards/atstk1000/atstk1003.c
+++ /dev/null
@@ -1,162 +0,0 @@
-/*
- * ATSTK1003 daughterboard-specific init code
- *
- * Copyright (C) 2007 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/clk.h>
-#include <linux/err.h>
-#include <linux/init.h>
-#include <linux/kernel.h>
-#include <linux/platform_device.h>
-#include <linux/string.h>
-#include <linux/types.h>
-
-#include <linux/spi/at73c213.h>
-#include <linux/spi/spi.h>
-#include <linux/atmel-mci.h>
-
-#include <asm/setup.h>
-
-#include <mach/at32ap700x.h>
-#include <mach/board.h>
-#include <mach/init.h>
-#include <mach/portmux.h>
-
-#include "atstk1000.h"
-
-/* Oscillator frequencies. These are board specific */
-unsigned long at32_board_osc_rates[3] = {
- [0] = 32768, /* 32.768 kHz on RTC osc */
- [1] = 20000000, /* 20 MHz on osc0 */
- [2] = 12000000, /* 12 MHz on osc1 */
-};
-
-#ifdef CONFIG_BOARD_ATSTK1000_EXTDAC
-static struct at73c213_board_info at73c213_data = {
- .ssc_id = 0,
- .shortname = "AVR32 STK1000 external DAC",
-};
-#endif
-
-#ifndef CONFIG_BOARD_ATSTK100X_SW1_CUSTOM
-static struct spi_board_info spi0_board_info[] __initdata = {
-#ifdef CONFIG_BOARD_ATSTK1000_EXTDAC
- {
- /* AT73C213 */
- .modalias = "at73c213",
- .max_speed_hz = 200000,
- .chip_select = 0,
- .mode = SPI_MODE_1,
- .platform_data = &at73c213_data,
- },
-#endif
- /*
- * We can control the LTV350QV LCD panel, but it isn't much
- * point since we don't have an LCD controller...
- */
-};
-#endif
-
-#ifdef CONFIG_BOARD_ATSTK100X_SPI1
-static struct spi_board_info spi1_board_info[] __initdata = { {
- /* patch in custom entries here */
-} };
-#endif
-
-#ifndef CONFIG_BOARD_ATSTK100X_SW2_CUSTOM
-static struct mci_platform_data __initdata mci0_data = {
- .slot[0] = {
- .bus_width = 4,
- .detect_pin = -ENODEV,
- .wp_pin = -ENODEV,
- },
-};
-#endif
-
-#ifdef CONFIG_BOARD_ATSTK1000_EXTDAC
-static void __init atstk1003_setup_extdac(void)
-{
- struct clk *gclk;
- struct clk *pll;
-
- gclk = clk_get(NULL, "gclk0");
- if (IS_ERR(gclk))
- goto err_gclk;
- pll = clk_get(NULL, "pll0");
- if (IS_ERR(pll))
- goto err_pll;
-
- if (clk_set_parent(gclk, pll)) {
- pr_debug("STK1000: failed to set pll0 as parent for DAC clock\n");
- goto err_set_clk;
- }
-
- at32_select_periph(GPIO_PIOA_BASE, (1 << 30), GPIO_PERIPH_A, 0);
- at73c213_data.dac_clk = gclk;
-
-err_set_clk:
- clk_put(pll);
-err_pll:
- clk_put(gclk);
-err_gclk:
- return;
-}
-#else
-static void __init atstk1003_setup_extdac(void)
-{
-
-}
-#endif /* CONFIG_BOARD_ATSTK1000_EXTDAC */
-
-void __init setup_board(void)
-{
-#ifdef CONFIG_BOARD_ATSTK100X_SW2_CUSTOM
- at32_map_usart(0, 1, 0); /* USART 0/B: /dev/ttyS1, IRDA */
-#else
- at32_map_usart(1, 0, 0); /* USART 1/A: /dev/ttyS0, DB9 */
-#endif
- /* USART 2/unused: expansion connector */
- at32_map_usart(3, 2, 0); /* USART 3/C: /dev/ttyS2, DB9 */
-
- at32_setup_serial_console(0);
-}
-
-static int __init atstk1003_init(void)
-{
- /*
- * ATSTK1000 uses 32-bit SDRAM interface. Reserve the
- * SDRAM-specific pins so that nobody messes with them.
- */
- at32_reserve_pin(GPIO_PIOE_BASE, ATMEL_EBI_PE_DATA_ALL);
-
-#ifdef CONFIG_BOARD_ATSTK100X_SW2_CUSTOM
- at32_add_device_usart(1);
-#else
- at32_add_device_usart(0);
-#endif
- at32_add_device_usart(2);
-
-#ifndef CONFIG_BOARD_ATSTK100X_SW1_CUSTOM
- at32_add_device_spi(0, spi0_board_info, ARRAY_SIZE(spi0_board_info));
-#endif
-#ifdef CONFIG_BOARD_ATSTK100X_SPI1
- at32_add_device_spi(1, spi1_board_info, ARRAY_SIZE(spi1_board_info));
-#endif
-#ifndef CONFIG_BOARD_ATSTK100X_SW2_CUSTOM
- at32_add_device_mci(0, &mci0_data);
-#endif
- at32_add_device_usba(0, NULL);
-#ifndef CONFIG_BOARD_ATSTK100X_SW3_CUSTOM
- at32_add_device_ssc(0, ATMEL_SSC_TX);
-#endif
-
- atstk1000_setup_j2_leds();
- atstk1003_setup_extdac();
-
- return 0;
-}
-postcore_initcall(atstk1003_init);
diff --git a/arch/avr32/boards/atstk1000/atstk1004.c b/arch/avr32/boards/atstk1000/atstk1004.c
deleted file mode 100644
index 69a9f0f..0000000
--- a/arch/avr32/boards/atstk1000/atstk1004.c
+++ /dev/null
@@ -1,164 +0,0 @@
-/*
- * ATSTK1003 daughterboard-specific init code
- *
- * Copyright (C) 2007 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/clk.h>
-#include <linux/err.h>
-#include <linux/init.h>
-#include <linux/kernel.h>
-#include <linux/platform_device.h>
-#include <linux/string.h>
-#include <linux/types.h>
-
-#include <linux/spi/at73c213.h>
-#include <linux/spi/spi.h>
-#include <linux/atmel-mci.h>
-
-#include <video/atmel_lcdc.h>
-
-#include <asm/setup.h>
-
-#include <mach/at32ap700x.h>
-#include <mach/board.h>
-#include <mach/init.h>
-#include <mach/portmux.h>
-
-#include "atstk1000.h"
-
-/* Oscillator frequencies. These are board specific */
-unsigned long at32_board_osc_rates[3] = {
- [0] = 32768, /* 32.768 kHz on RTC osc */
- [1] = 20000000, /* 20 MHz on osc0 */
- [2] = 12000000, /* 12 MHz on osc1 */
-};
-
-#ifdef CONFIG_BOARD_ATSTK1000_EXTDAC
-static struct at73c213_board_info at73c213_data = {
- .ssc_id = 0,
- .shortname = "AVR32 STK1000 external DAC",
-};
-#endif
-
-#ifndef CONFIG_BOARD_ATSTK100X_SW1_CUSTOM
-static struct spi_board_info spi0_board_info[] __initdata = {
-#ifdef CONFIG_BOARD_ATSTK1000_EXTDAC
- {
- /* AT73C213 */
- .modalias = "at73c213",
- .max_speed_hz = 200000,
- .chip_select = 0,
- .mode = SPI_MODE_1,
- .platform_data = &at73c213_data,
- },
-#endif
- {
- /* QVGA display */
- .modalias = "ltv350qv",
- .max_speed_hz = 16000000,
- .chip_select = 1,
- .mode = SPI_MODE_3,
- },
-};
-#endif
-
-#ifdef CONFIG_BOARD_ATSTK100X_SPI1
-static struct spi_board_info spi1_board_info[] __initdata = { {
- /* patch in custom entries here */
-} };
-#endif
-
-#ifndef CONFIG_BOARD_ATSTK100X_SW2_CUSTOM
-static struct mci_platform_data __initdata mci0_data = {
- .slot[0] = {
- .bus_width = 4,
- .detect_pin = -ENODEV,
- .wp_pin = -ENODEV,
- },
-};
-#endif
-
-#ifdef CONFIG_BOARD_ATSTK1000_EXTDAC
-static void __init atstk1004_setup_extdac(void)
-{
- struct clk *gclk;
- struct clk *pll;
-
- gclk = clk_get(NULL, "gclk0");
- if (IS_ERR(gclk))
- goto err_gclk;
- pll = clk_get(NULL, "pll0");
- if (IS_ERR(pll))
- goto err_pll;
-
- if (clk_set_parent(gclk, pll)) {
- pr_debug("STK1000: failed to set pll0 as parent for DAC clock\n");
- goto err_set_clk;
- }
-
- at32_select_periph(GPIO_PIOA_BASE, (1 << 30), GPIO_PERIPH_A, 0);
- at73c213_data.dac_clk = gclk;
-
-err_set_clk:
- clk_put(pll);
-err_pll:
- clk_put(gclk);
-err_gclk:
- return;
-}
-#else
-static void __init atstk1004_setup_extdac(void)
-{
-
-}
-#endif /* CONFIG_BOARD_ATSTK1000_EXTDAC */
-
-void __init setup_board(void)
-{
-#ifdef CONFIG_BOARD_ATSTK100X_SW2_CUSTOM
- at32_map_usart(0, 1, 0); /* USART 0/B: /dev/ttyS1, IRDA */
-#else
- at32_map_usart(1, 0, 0); /* USART 1/A: /dev/ttyS0, DB9 */
-#endif
- /* USART 2/unused: expansion connector */
- at32_map_usart(3, 2, 0); /* USART 3/C: /dev/ttyS2, DB9 */
-
- at32_setup_serial_console(0);
-}
-
-static int __init atstk1004_init(void)
-{
-#ifdef CONFIG_BOARD_ATSTK100X_SW2_CUSTOM
- at32_add_device_usart(1);
-#else
- at32_add_device_usart(0);
-#endif
- at32_add_device_usart(2);
-
-#ifndef CONFIG_BOARD_ATSTK100X_SW1_CUSTOM
- at32_add_device_spi(0, spi0_board_info, ARRAY_SIZE(spi0_board_info));
-#endif
-#ifdef CONFIG_BOARD_ATSTK100X_SPI1
- at32_add_device_spi(1, spi1_board_info, ARRAY_SIZE(spi1_board_info));
-#endif
-#ifndef CONFIG_BOARD_ATSTK100X_SW2_CUSTOM
- at32_add_device_mci(0, &mci0_data);
-#endif
- at32_add_device_lcdc(0, &atstk1000_lcdc_data,
- fbmem_start, fbmem_size,
- ATMEL_LCDC_PRI_24BIT | ATMEL_LCDC_PRI_CONTROL);
- at32_add_device_usba(0, NULL);
-#ifndef CONFIG_BOARD_ATSTK100X_SW3_CUSTOM
- at32_add_device_ssc(0, ATMEL_SSC_TX);
-#endif
-
- atstk1000_setup_j2_leds();
- atstk1004_setup_extdac();
-
- return 0;
-}
-postcore_initcall(atstk1004_init);
diff --git a/arch/avr32/boards/atstk1000/flash.c b/arch/avr32/boards/atstk1000/flash.c
deleted file mode 100644
index 6e4d561..0000000
--- a/arch/avr32/boards/atstk1000/flash.c
+++ /dev/null
@@ -1,98 +0,0 @@
-/*
- * ATSTK1000 board-specific flash initialization
- *
- * Copyright (C) 2005-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/init.h>
-#include <linux/platform_device.h>
-#include <linux/mtd/mtd.h>
-#include <linux/mtd/partitions.h>
-#include <linux/mtd/physmap.h>
-
-#include <mach/smc.h>
-
-static struct smc_timing flash_timing __initdata = {
- .ncs_read_setup = 0,
- .nrd_setup = 40,
- .ncs_write_setup = 0,
- .nwe_setup = 10,
-
- .ncs_read_pulse = 80,
- .nrd_pulse = 40,
- .ncs_write_pulse = 65,
- .nwe_pulse = 55,
-
- .read_cycle = 120,
- .write_cycle = 120,
-};
-
-static struct smc_config flash_config __initdata = {
- .bus_width = 2,
- .nrd_controlled = 1,
- .nwe_controlled = 1,
- .byte_write = 1,
-};
-
-static struct mtd_partition flash_parts[] = {
- {
- .name = "u-boot",
- .offset = 0x00000000,
- .size = 0x00020000, /* 128 KiB */
- .mask_flags = MTD_WRITEABLE,
- },
- {
- .name = "root",
- .offset = 0x00020000,
- .size = 0x007d0000,
- },
- {
- .name = "env",
- .offset = 0x007f0000,
- .size = 0x00010000,
- .mask_flags = MTD_WRITEABLE,
- },
-};
-
-static struct physmap_flash_data flash_data = {
- .width = 2,
- .nr_parts = ARRAY_SIZE(flash_parts),
- .parts = flash_parts,
-};
-
-static struct resource flash_resource = {
- .start = 0x00000000,
- .end = 0x007fffff,
- .flags = IORESOURCE_MEM,
-};
-
-static struct platform_device flash_device = {
- .name = "physmap-flash",
- .id = 0,
- .resource = &flash_resource,
- .num_resources = 1,
- .dev = {
- .platform_data = &flash_data,
- },
-};
-
-/* This needs to be called after the SMC has been initialized */
-static int __init atstk1000_flash_init(void)
-{
- int ret;
-
- smc_set_timing(&flash_config, &flash_timing);
- ret = smc_set_configuration(0, &flash_config);
- if (ret < 0) {
- printk(KERN_ERR "atstk1000: failed to set NOR flash timing\n");
- return ret;
- }
-
- platform_device_register(&flash_device);
-
- return 0;
-}
-device_initcall(atstk1000_flash_init);
diff --git a/arch/avr32/boards/atstk1000/setup.c b/arch/avr32/boards/atstk1000/setup.c
deleted file mode 100644
index b6b88f5..0000000
--- a/arch/avr32/boards/atstk1000/setup.c
+++ /dev/null
@@ -1,127 +0,0 @@
-/*
- * ATSTK1000 board-specific setup code.
- *
- * Copyright (C) 2005-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/bootmem.h>
-#include <linux/fb.h>
-#include <linux/init.h>
-#include <linux/platform_device.h>
-#include <linux/types.h>
-#include <linux/linkage.h>
-
-#include <video/atmel_lcdc.h>
-
-#include <asm/setup.h>
-
-#include <mach/at32ap700x.h>
-#include <mach/board.h>
-#include <mach/portmux.h>
-
-#include "atstk1000.h"
-
-/* Initialized by bootloader-specific startup code. */
-struct tag *bootloader_tags __initdata;
-
-static struct fb_videomode __initdata ltv350qv_modes[] = {
- {
- .name = "320x240 @ 75",
- .refresh = 75,
- .xres = 320, .yres = 240,
- .pixclock = KHZ2PICOS(6891),
-
- .left_margin = 17, .right_margin = 33,
- .upper_margin = 10, .lower_margin = 10,
- .hsync_len = 16, .vsync_len = 1,
-
- .sync = 0,
- .vmode = FB_VMODE_NONINTERLACED,
- },
-};
-
-static struct fb_monspecs __initdata atstk1000_default_monspecs = {
- .manufacturer = "SNG",
- .monitor = "LTV350QV",
- .modedb = ltv350qv_modes,
- .modedb_len = ARRAY_SIZE(ltv350qv_modes),
- .hfmin = 14820,
- .hfmax = 22230,
- .vfmin = 60,
- .vfmax = 90,
- .dclkmax = 30000000,
-};
-
-struct atmel_lcdfb_pdata __initdata atstk1000_lcdc_data = {
- .default_bpp = 24,
- .default_dmacon = ATMEL_LCDC_DMAEN | ATMEL_LCDC_DMA2DEN,
- .default_lcdcon2 = (ATMEL_LCDC_DISTYPE_TFT
- | ATMEL_LCDC_INVCLK
- | ATMEL_LCDC_CLKMOD_ALWAYSACTIVE
- | ATMEL_LCDC_MEMOR_BIG),
- .default_monspecs = &atstk1000_default_monspecs,
- .guard_time = 2,
-};
-
-#ifdef CONFIG_BOARD_ATSTK1000_J2_LED
-#include <linux/leds.h>
-
-static struct gpio_led stk1000_j2_led[] = {
-#ifdef CONFIG_BOARD_ATSTK1000_J2_LED8
-#define LEDSTRING "J2 jumpered to LED8"
- { .name = "led0:amber", .gpio = GPIO_PIN_PB( 8), },
- { .name = "led1:amber", .gpio = GPIO_PIN_PB( 9), },
- { .name = "led2:amber", .gpio = GPIO_PIN_PB(10), },
- { .name = "led3:amber", .gpio = GPIO_PIN_PB(13), },
- { .name = "led4:amber", .gpio = GPIO_PIN_PB(14), },
- { .name = "led5:amber", .gpio = GPIO_PIN_PB(15), },
- { .name = "led6:amber", .gpio = GPIO_PIN_PB(16), },
- { .name = "led7:amber", .gpio = GPIO_PIN_PB(30),
- .default_trigger = "heartbeat", },
-#else /* RGB */
-#define LEDSTRING "J2 jumpered to RGB LEDs"
- { .name = "r1:red", .gpio = GPIO_PIN_PB( 8), },
- { .name = "g1:green", .gpio = GPIO_PIN_PB(10), },
- { .name = "b1:blue", .gpio = GPIO_PIN_PB(14), },
-
- { .name = "r2:red", .gpio = GPIO_PIN_PB( 9),
- .default_trigger = "heartbeat", },
- { .name = "g2:green", .gpio = GPIO_PIN_PB(13), },
- { .name = "b2:blue", .gpio = GPIO_PIN_PB(15),
- .default_trigger = "heartbeat", },
- /* PB16, PB30 unused */
-#endif
-};
-
-static struct gpio_led_platform_data stk1000_j2_led_data = {
- .num_leds = ARRAY_SIZE(stk1000_j2_led),
- .leds = stk1000_j2_led,
-};
-
-static struct platform_device stk1000_j2_led_dev = {
- .name = "leds-gpio",
- .id = 2, /* gpio block J2 */
- .dev = {
- .platform_data = &stk1000_j2_led_data,
- },
-};
-
-void __init atstk1000_setup_j2_leds(void)
-{
- unsigned i;
-
- for (i = 0; i < ARRAY_SIZE(stk1000_j2_led); i++)
- at32_select_gpio(stk1000_j2_led[i].gpio, AT32_GPIOF_OUTPUT);
-
- printk("STK1000: " LEDSTRING "\n");
- platform_device_register(&stk1000_j2_led_dev);
-}
-#else /* CONFIG_BOARD_ATSTK1000_J2_LED */
-void __init atstk1000_setup_j2_leds(void)
-{
-
-}
-#endif /* CONFIG_BOARD_ATSTK1000_J2_LED */
diff --git a/arch/avr32/boards/favr-32/Kconfig b/arch/avr32/boards/favr-32/Kconfig
deleted file mode 100644
index 2c83d1d..0000000
--- a/arch/avr32/boards/favr-32/Kconfig
+++ /dev/null
@@ -1,22 +0,0 @@
-# Favr-32 customization
-
-if BOARD_FAVR_32
-
-config BOARD_FAVR32_ABDAC_RATE
- int "DAC target rate"
- default 44100
- range 32000 50000
- help
- Specify the target rate the internal DAC should try to match. This
- will use PLL1 to generate a frequency as close as possible to this
- rate.
-
- Must be within the range 32000 to 50000, which should be suitable to
- generate most other frequencies in power of 2 steps.
-
- Ex:
- 48000 will also suit 24000 and 12000
- 44100 will also suit 22050 and 11025
- 32000 will also suit 16000 and 8000
-
-endif # BOARD_FAVR_32
diff --git a/arch/avr32/boards/favr-32/Makefile b/arch/avr32/boards/favr-32/Makefile
deleted file mode 100644
index 234f215..0000000
--- a/arch/avr32/boards/favr-32/Makefile
+++ /dev/null
@@ -1 +0,0 @@
-obj-y += setup.o flash.o
diff --git a/arch/avr32/boards/favr-32/flash.c b/arch/avr32/boards/favr-32/flash.c
deleted file mode 100644
index 604bbd5..0000000
--- a/arch/avr32/boards/favr-32/flash.c
+++ /dev/null
@@ -1,98 +0,0 @@
-/*
- * Favr-32 board-specific flash initialization
- *
- * Copyright (C) 2008 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/init.h>
-#include <linux/platform_device.h>
-#include <linux/mtd/mtd.h>
-#include <linux/mtd/partitions.h>
-#include <linux/mtd/physmap.h>
-
-#include <mach/smc.h>
-
-static struct smc_timing flash_timing __initdata = {
- .ncs_read_setup = 0,
- .nrd_setup = 40,
- .ncs_write_setup = 0,
- .nwe_setup = 10,
-
- .ncs_read_pulse = 80,
- .nrd_pulse = 40,
- .ncs_write_pulse = 65,
- .nwe_pulse = 55,
-
- .read_cycle = 120,
- .write_cycle = 120,
-};
-
-static struct smc_config flash_config __initdata = {
- .bus_width = 2,
- .nrd_controlled = 1,
- .nwe_controlled = 1,
- .byte_write = 1,
-};
-
-static struct mtd_partition flash_parts[] = {
- {
- .name = "u-boot",
- .offset = 0x00000000,
- .size = 0x00020000, /* 128 KiB */
- .mask_flags = MTD_WRITEABLE,
- },
- {
- .name = "root",
- .offset = 0x00020000,
- .size = 0x007d0000,
- },
- {
- .name = "env",
- .offset = 0x007f0000,
- .size = 0x00010000,
- .mask_flags = MTD_WRITEABLE,
- },
-};
-
-static struct physmap_flash_data flash_data = {
- .width = 2,
- .nr_parts = ARRAY_SIZE(flash_parts),
- .parts = flash_parts,
-};
-
-static struct resource flash_resource = {
- .start = 0x00000000,
- .end = 0x007fffff,
- .flags = IORESOURCE_MEM,
-};
-
-static struct platform_device flash_device = {
- .name = "physmap-flash",
- .id = 0,
- .resource = &flash_resource,
- .num_resources = 1,
- .dev = {
- .platform_data = &flash_data,
- },
-};
-
-/* This needs to be called after the SMC has been initialized */
-static int __init favr32_flash_init(void)
-{
- int ret;
-
- smc_set_timing(&flash_config, &flash_timing);
- ret = smc_set_configuration(0, &flash_config);
- if (ret < 0) {
- printk(KERN_ERR "Favr-32: failed to set NOR flash timing\n");
- return ret;
- }
-
- platform_device_register(&flash_device);
-
- return 0;
-}
-device_initcall(favr32_flash_init);
diff --git a/arch/avr32/boards/favr-32/setup.c b/arch/avr32/boards/favr-32/setup.c
deleted file mode 100644
index 234cb07..0000000
--- a/arch/avr32/boards/favr-32/setup.c
+++ /dev/null
@@ -1,366 +0,0 @@
-/*
- * Favr-32 board-specific setup code.
- *
- * Copyright (C) 2008 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/clk.h>
-#include <linux/etherdevice.h>
-#include <linux/bootmem.h>
-#include <linux/fb.h>
-#include <linux/init.h>
-#include <linux/platform_device.h>
-#include <linux/types.h>
-#include <linux/linkage.h>
-#include <linux/gpio.h>
-#include <linux/leds.h>
-#include <linux/atmel-mci.h>
-#include <linux/pwm.h>
-#include <linux/pwm_backlight.h>
-#include <linux/regulator/fixed.h>
-#include <linux/regulator/machine.h>
-#include <linux/spi/spi.h>
-#include <linux/spi/ads7846.h>
-
-#include <sound/atmel-abdac.h>
-
-#include <video/atmel_lcdc.h>
-
-#include <asm/setup.h>
-
-#include <mach/at32ap700x.h>
-#include <mach/init.h>
-#include <mach/board.h>
-#include <mach/portmux.h>
-
-#define PWM_BL_CH 2
-
-/* Oscillator frequencies. These are board-specific */
-unsigned long at32_board_osc_rates[3] = {
- [0] = 32768, /* 32.768 kHz on RTC osc */
- [1] = 20000000, /* 20 MHz on osc0 */
- [2] = 12000000, /* 12 MHz on osc1 */
-};
-
-/* Initialized by bootloader-specific startup code. */
-struct tag *bootloader_tags __initdata;
-
-static struct atmel_abdac_pdata __initdata abdac0_data = {
-};
-
-struct eth_addr {
- u8 addr[6];
-};
-static struct eth_addr __initdata hw_addr[1];
-static struct macb_platform_data __initdata eth_data[1] = {
- {
- .phy_mask = ~(1U << 1),
- },
-};
-
-static int ads7843_get_pendown_state(void)
-{
- return !gpio_get_value(GPIO_PIN_PB(3));
-}
-
-static struct ads7846_platform_data ads7843_data = {
- .model = 7843,
- .get_pendown_state = ads7843_get_pendown_state,
- .pressure_max = 255,
- /*
- * Values below are for debounce filtering, these can be experimented
- * with further.
- */
- .debounce_max = 20,
- .debounce_rep = 4,
- .debounce_tol = 5,
-
- .keep_vref_on = true,
- .settle_delay_usecs = 500,
- .penirq_recheck_delay_usecs = 100,
-};
-
-static struct spi_board_info __initdata spi1_board_info[] = {
- {
- /* ADS7843 touch controller */
- .modalias = "ads7846",
- .max_speed_hz = 2000000,
- .chip_select = 0,
- .bus_num = 1,
- .platform_data = &ads7843_data,
- },
-};
-
-static struct mci_platform_data __initdata mci0_data = {
- .slot[0] = {
- .bus_width = 4,
- .detect_pin = -ENODEV,
- .wp_pin = -ENODEV,
- },
-};
-
-static struct fb_videomode __initdata lb104v03_modes[] = {
- {
- .name = "640x480 @ 50",
- .refresh = 50,
- .xres = 640, .yres = 480,
- .pixclock = KHZ2PICOS(25100),
-
- .left_margin = 90, .right_margin = 70,
- .upper_margin = 30, .lower_margin = 15,
- .hsync_len = 12, .vsync_len = 2,
-
- .sync = 0,
- .vmode = FB_VMODE_NONINTERLACED,
- },
-};
-
-static struct fb_monspecs __initdata favr32_default_monspecs = {
- .manufacturer = "LG",
- .monitor = "LB104V03",
- .modedb = lb104v03_modes,
- .modedb_len = ARRAY_SIZE(lb104v03_modes),
- .hfmin = 27273,
- .hfmax = 31111,
- .vfmin = 45,
- .vfmax = 60,
- .dclkmax = 28000000,
-};
-
-struct atmel_lcdfb_pdata __initdata favr32_lcdc_data = {
- .default_bpp = 16,
- .default_dmacon = ATMEL_LCDC_DMAEN | ATMEL_LCDC_DMA2DEN,
- .default_lcdcon2 = (ATMEL_LCDC_DISTYPE_TFT
- | ATMEL_LCDC_CLKMOD_ALWAYSACTIVE
- | ATMEL_LCDC_MEMOR_BIG),
- .default_monspecs = &favr32_default_monspecs,
- .guard_time = 2,
-};
-
-static struct gpio_led favr32_leds[] = {
- {
- .name = "green",
- .gpio = GPIO_PIN_PE(19),
- .default_trigger = "heartbeat",
- .active_low = 1,
- },
- {
- .name = "red",
- .gpio = GPIO_PIN_PE(20),
- .active_low = 1,
- },
-};
-
-static struct gpio_led_platform_data favr32_led_data = {
- .num_leds = ARRAY_SIZE(favr32_leds),
- .leds = favr32_leds,
-};
-
-static struct platform_device favr32_led_dev = {
- .name = "leds-gpio",
- .id = 0,
- .dev = {
- .platform_data = &favr32_led_data,
- },
-};
-
-/*
- * The next two functions should go away as the boot loader is
- * supposed to initialize the macb address registers with a valid
- * ethernet address. But we need to keep it around for a while until
- * we can be reasonably sure the boot loader does this.
- *
- * The phy_id is ignored as the driver will probe for it.
- */
-static int __init parse_tag_ethernet(struct tag *tag)
-{
- int i;
-
- i = tag->u.ethernet.mac_index;
- if (i < ARRAY_SIZE(hw_addr))
- memcpy(hw_addr[i].addr, tag->u.ethernet.hw_address,
- sizeof(hw_addr[i].addr));
-
- return 0;
-}
-__tagtable(ATAG_ETHERNET, parse_tag_ethernet);
-
-static void __init set_hw_addr(struct platform_device *pdev)
-{
- struct resource *res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- const u8 *addr;
- void __iomem *regs;
- struct clk *pclk;
-
- if (!res)
- return;
- if (pdev->id >= ARRAY_SIZE(hw_addr))
- return;
-
- addr = hw_addr[pdev->id].addr;
- if (!is_valid_ether_addr(addr))
- return;
-
- /*
- * Since this is board-specific code, we'll cheat and use the
- * physical address directly as we happen to know that it's
- * the same as the virtual address.
- */
- regs = (void __iomem __force *)res->start;
- pclk = clk_get(&pdev->dev, "pclk");
- if (IS_ERR(pclk))
- return;
-
- clk_enable(pclk);
- __raw_writel((addr[3] << 24) | (addr[2] << 16)
- | (addr[1] << 8) | addr[0], regs + 0x98);
- __raw_writel((addr[5] << 8) | addr[4], regs + 0x9c);
- clk_disable(pclk);
- clk_put(pclk);
-}
-
-void __init favr32_setup_leds(void)
-{
- unsigned i;
-
- for (i = 0; i < ARRAY_SIZE(favr32_leds); i++)
- at32_select_gpio(favr32_leds[i].gpio, AT32_GPIOF_OUTPUT);
-
- platform_device_register(&favr32_led_dev);
-}
-
-static struct pwm_lookup pwm_lookup[] = {
- PWM_LOOKUP("at91sam9rl-pwm", PWM_BL_CH, "pwm-backlight.0", NULL,
- 5000, PWM_POLARITY_INVERSED),
-};
-
-static struct regulator_consumer_supply fixed_power_consumers[] = {
- REGULATOR_SUPPLY("power", "pwm-backlight.0"),
-};
-
-static struct platform_pwm_backlight_data pwm_bl_data = {
- .enable_gpio = GPIO_PIN_PA(28),
- .max_brightness = 255,
- .dft_brightness = 255,
- .lth_brightness = 50,
-};
-
-static struct platform_device pwm_bl_device = {
- .name = "pwm-backlight",
- .dev = {
- .platform_data = &pwm_bl_data,
- },
-};
-
-static void __init favr32_setup_atmel_pwm_bl(void)
-{
- pwm_add_table(pwm_lookup, ARRAY_SIZE(pwm_lookup));
- regulator_register_always_on(0, "fixed", fixed_power_consumers,
- ARRAY_SIZE(fixed_power_consumers), 3300000);
- platform_device_register(&pwm_bl_device);
- at32_select_gpio(pwm_bl_data.enable_gpio, 0);
-}
-
-void __init setup_board(void)
-{
- at32_map_usart(3, 0, 0); /* USART 3 => /dev/ttyS0 */
- at32_setup_serial_console(0);
-}
-
-static int __init set_abdac_rate(struct platform_device *pdev)
-{
- int retval;
- struct clk *osc1;
- struct clk *pll1;
- struct clk *abdac;
-
- if (pdev == NULL)
- return -ENXIO;
-
- osc1 = clk_get(NULL, "osc1");
- if (IS_ERR(osc1)) {
- retval = PTR_ERR(osc1);
- goto out;
- }
-
- pll1 = clk_get(NULL, "pll1");
- if (IS_ERR(pll1)) {
- retval = PTR_ERR(pll1);
- goto out_osc1;
- }
-
- abdac = clk_get(&pdev->dev, "sample_clk");
- if (IS_ERR(abdac)) {
- retval = PTR_ERR(abdac);
- goto out_pll1;
- }
-
- retval = clk_set_parent(pll1, osc1);
- if (retval != 0)
- goto out_abdac;
-
- /*
- * Rate is 32000 to 50000 and ABDAC oversamples 256x. Multiply, in
- * power of 2, to a value above 80 MHz. Power of 2 so it is possible
- * for the generic clock to divide it down again and 80 MHz is the
- * lowest frequency for the PLL.
- */
- retval = clk_round_rate(pll1,
- CONFIG_BOARD_FAVR32_ABDAC_RATE * 256 * 16);
- if (retval <= 0) {
- retval = -EINVAL;
- goto out_abdac;
- }
-
- retval = clk_set_rate(pll1, retval);
- if (retval != 0)
- goto out_abdac;
-
- retval = clk_set_parent(abdac, pll1);
- if (retval != 0)
- goto out_abdac;
-
-out_abdac:
- clk_put(abdac);
-out_pll1:
- clk_put(pll1);
-out_osc1:
- clk_put(osc1);
-out:
- return retval;
-}
-
-static int __init favr32_init(void)
-{
- /*
- * Favr-32 uses 32-bit SDRAM interface. Reserve the SDRAM-specific
- * pins so that nobody messes with them.
- */
- at32_reserve_pin(GPIO_PIOE_BASE, ATMEL_EBI_PE_DATA_ALL);
-
- at32_select_gpio(GPIO_PIN_PB(3), 0); /* IRQ from ADS7843 */
-
- at32_add_device_usart(0);
-
- set_hw_addr(at32_add_device_eth(0, ð_data[0]));
-
- spi1_board_info[0].irq = gpio_to_irq(GPIO_PIN_PB(3));
-
- set_abdac_rate(at32_add_device_abdac(0, &abdac0_data));
-
- at32_add_device_pwm(1 << PWM_BL_CH);
- at32_add_device_spi(1, spi1_board_info, ARRAY_SIZE(spi1_board_info));
- at32_add_device_mci(0, &mci0_data);
- at32_add_device_usba(0, NULL);
- at32_add_device_lcdc(0, &favr32_lcdc_data, fbmem_start, fbmem_size, 0);
-
- favr32_setup_leds();
-
- favr32_setup_atmel_pwm_bl();
-
- return 0;
-}
-postcore_initcall(favr32_init);
diff --git a/arch/avr32/boards/hammerhead/Kconfig b/arch/avr32/boards/hammerhead/Kconfig
deleted file mode 100644
index 5c13d78..0000000
--- a/arch/avr32/boards/hammerhead/Kconfig
+++ /dev/null
@@ -1,43 +0,0 @@
-# Hammerhead customization
-
-if BOARD_HAMMERHEAD
-
-config BOARD_HAMMERHEAD_USB
- bool "Philips ISP116x-hcd USB support"
- help
- This enables USB support for Hammerheads internal ISP116x
- controller from Philips.
-
- Choose 'Y' here if you want to have your board USB driven.
-
-config BOARD_HAMMERHEAD_LCD
- bool "Atmel AT91/AT32 LCD support"
- help
- This enables LCD support for the Hammerhead board. You may
- also add support for framebuffer devices (AT91/AT32 LCD Controller)
- and framebuffer console support to get the most out of your LCD.
-
- Choose 'Y' here if you have ordered a Corona daugther board and
- want to have support for your Hantronix HDA-351T-LV LCD.
-
-config BOARD_HAMMERHEAD_SND
- bool "Atmel AC97 Sound support"
- help
- This enables Sound support for the Hammerhead board. You may
- also go through the ALSA settings to get it working.
-
- Choose 'Y' here if you have ordered a Corona daugther board and
- want to make your board funky.
-
-config BOARD_HAMMERHEAD_FPGA
- bool "Hammerhead FPGA Support"
- default y
- help
- This adds support for the Cyclone III FPGA from Altera
- found on Miromico's Hammerhead board.
-
- Choose 'Y' here if you want to have FPGA support enabled.
- You will have to choose the "Hammerhead FPGA Device Support" in
- Device Drivers->Misc to be able to use FPGA functionality.
-
-endif # BOARD_ATNGW100
diff --git a/arch/avr32/boards/hammerhead/Makefile b/arch/avr32/boards/hammerhead/Makefile
deleted file mode 100644
index c740aa1..0000000
--- a/arch/avr32/boards/hammerhead/Makefile
+++ /dev/null
@@ -1 +0,0 @@
-obj-y += setup.o flash.o
diff --git a/arch/avr32/boards/hammerhead/flash.c b/arch/avr32/boards/hammerhead/flash.c
deleted file mode 100644
index e86280c..0000000
--- a/arch/avr32/boards/hammerhead/flash.c
+++ /dev/null
@@ -1,381 +0,0 @@
-/*
- * Hammerhead board-specific flash initialization
- *
- * Copyright (C) 2008 Miromico AG
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/init.h>
-#include <linux/platform_device.h>
-#include <linux/mtd/mtd.h>
-#include <linux/mtd/partitions.h>
-#include <linux/mtd/physmap.h>
-#include <linux/usb/isp116x.h>
-#include <linux/dma-mapping.h>
-#include <linux/delay.h>
-
-#include <mach/portmux.h>
-#include <mach/at32ap700x.h>
-#include <mach/smc.h>
-
-#include "../../mach-at32ap/clock.h"
-#include "flash.h"
-
-
-#define HAMMERHEAD_USB_PERIPH_GCLK0 0x40000000
-#define HAMMERHEAD_USB_PERIPH_CS2 0x02000000
-#define HAMMERHEAD_USB_PERIPH_EXTINT0 0x02000000
-
-#define HAMMERHEAD_FPGA_PERIPH_MOSI 0x00000002
-#define HAMMERHEAD_FPGA_PERIPH_SCK 0x00000020
-#define HAMMERHEAD_FPGA_PERIPH_EXTINT3 0x10000000
-
-static struct smc_timing flash_timing __initdata = {
- .ncs_read_setup = 0,
- .nrd_setup = 40,
- .ncs_write_setup = 0,
- .nwe_setup = 10,
-
- .ncs_read_pulse = 80,
- .nrd_pulse = 40,
- .ncs_write_pulse = 65,
- .nwe_pulse = 55,
-
- .read_cycle = 120,
- .write_cycle = 120,
-};
-
-static struct smc_config flash_config __initdata = {
- .bus_width = 2,
- .nrd_controlled = 1,
- .nwe_controlled = 1,
- .byte_write = 1,
-};
-
-static struct mtd_partition flash_parts[] = {
- {
- .name = "u-boot",
- .offset = 0x00000000,
- .size = 0x00020000, /* 128 KiB */
- .mask_flags = MTD_WRITEABLE,
- },
- {
- .name = "root",
- .offset = 0x00020000,
- .size = 0x007d0000,
- },
- {
- .name = "env",
- .offset = 0x007f0000,
- .size = 0x00010000,
- .mask_flags = MTD_WRITEABLE,
- },
-};
-
-static struct physmap_flash_data flash_data = {
- .width = 2,
- .nr_parts = ARRAY_SIZE(flash_parts),
- .parts = flash_parts,
-};
-
-static struct resource flash_resource = {
- .start = 0x00000000,
- .end = 0x007fffff,
- .flags = IORESOURCE_MEM,
-};
-
-static struct platform_device flash_device = {
- .name = "physmap-flash",
- .id = 0,
- .resource = &flash_resource,
- .num_resources = 1,
- .dev = { .platform_data = &flash_data, },
-};
-
-#ifdef CONFIG_BOARD_HAMMERHEAD_USB
-
-static struct smc_timing isp1160_timing __initdata = {
- .ncs_read_setup = 75,
- .nrd_setup = 75,
- .ncs_write_setup = 75,
- .nwe_setup = 75,
-
-
- /* We use conservative timing settings, as the minimal settings aren't
- stable. There may be room for tweaking. */
- .ncs_read_pulse = 75, /* min. 33ns */
- .nrd_pulse = 75, /* min. 33ns */
- .ncs_write_pulse = 75, /* min. 26ns */
- .nwe_pulse = 75, /* min. 26ns */
-
- .read_cycle = 225, /* min. 143ns */
- .write_cycle = 225, /* min. 136ns */
-};
-
-static struct smc_config isp1160_config __initdata = {
- .bus_width = 2,
- .nrd_controlled = 1,
- .nwe_controlled = 1,
- .byte_write = 0,
-};
-
-/*
- * The platform delay function is only used to enforce the strange
- * read to write delay. This can not be configured in the SMC. All other
- * timings are controlled by the SMC (see timings obove)
- * So in isp116x-hcd.c we should comment out USE_PLATFORM_DELAY
- */
-void isp116x_delay(struct device *dev, int delay)
-{
- if (delay > 150)
- ndelay(delay - 150);
-}
-
-static struct isp116x_platform_data isp1160_data = {
- .sel15Kres = 1, /* use internal downstream resistors */
- .oc_enable = 0, /* external overcurrent detection */
- .int_edge_triggered = 0, /* interrupt is level triggered */
- .int_act_high = 0, /* interrupt is active low */
- .delay = isp116x_delay, /* platform delay function */
-};
-
-static struct resource isp1160_resource[] = {
- {
- .start = 0x08000000,
- .end = 0x08000001,
- .flags = IORESOURCE_MEM,
- },
- {
- .start = 0x08000002,
- .end = 0x08000003,
- .flags = IORESOURCE_MEM,
- },
- {
- .start = 64,
- .flags = IORESOURCE_IRQ,
- },
-};
-
-static struct platform_device isp1160_device = {
- .name = "isp116x-hcd",
- .id = 0,
- .resource = isp1160_resource,
- .num_resources = 3,
- .dev = {
- .platform_data = &isp1160_data,
- },
-};
-#endif
-
-#ifdef CONFIG_BOARD_HAMMERHEAD_USB
-static int __init hammerhead_usbh_init(void)
-{
- struct clk *gclk;
- struct clk *osc;
-
- int ret;
-
- /* setup smc for usbh */
- smc_set_timing(&isp1160_config, &isp1160_timing);
- ret = smc_set_configuration(2, &isp1160_config);
-
- if (ret < 0) {
- printk(KERN_ERR
- "hammerhead: failed to set ISP1160 USBH timing\n");
- return ret;
- }
-
- /* setup gclk0 to run from osc1 */
- gclk = clk_get(NULL, "gclk0");
- if (IS_ERR(gclk)) {
- ret = PTR_ERR(gclk);
- goto err_gclk;
- }
-
- osc = clk_get(NULL, "osc1");
- if (IS_ERR(osc)) {
- ret = PTR_ERR(osc);
- goto err_osc;
- }
-
- ret = clk_set_parent(gclk, osc);
- if (ret < 0) {
- pr_debug("hammerhead: failed to set osc1 for USBH clock\n");
- goto err_set_clk;
- }
-
- /* set clock to 6MHz */
- clk_set_rate(gclk, 6000000);
-
- /* and enable */
- clk_enable(gclk);
-
- /* select GCLK0 peripheral function */
- at32_select_periph(GPIO_PIOA_BASE, HAMMERHEAD_USB_PERIPH_GCLK0,
- GPIO_PERIPH_A, 0);
-
- /* enable CS2 peripheral function */
- at32_select_periph(GPIO_PIOE_BASE, HAMMERHEAD_USB_PERIPH_CS2,
- GPIO_PERIPH_A, 0);
-
- /* H_WAKEUP must be driven low */
- at32_select_gpio(GPIO_PIN_PA(8), AT32_GPIOF_OUTPUT);
-
- /* Select EXTINT0 for PB25 */
- at32_select_periph(GPIO_PIOB_BASE, HAMMERHEAD_USB_PERIPH_EXTINT0,
- GPIO_PERIPH_A, 0);
-
- /* register usbh device driver */
- platform_device_register(&isp1160_device);
-
- err_set_clk:
- clk_put(osc);
- err_osc:
- clk_put(gclk);
- err_gclk:
- return ret;
-}
-#endif
-
-#ifdef CONFIG_BOARD_HAMMERHEAD_FPGA
-static struct smc_timing fpga_timing __initdata = {
- .ncs_read_setup = 16,
- .nrd_setup = 32,
- .ncs_read_pulse = 48,
- .nrd_pulse = 32,
- .read_cycle = 64,
-
- .ncs_write_setup = 16,
- .nwe_setup = 16,
- .ncs_write_pulse = 32,
- .nwe_pulse = 32,
- .write_cycle = 64,
-};
-
-static struct smc_config fpga_config __initdata = {
- .bus_width = 4,
- .nrd_controlled = 1,
- .nwe_controlled = 1,
- .byte_write = 0,
-};
-
-static struct resource hh_fpga0_resource[] = {
- {
- .start = 0xffe00400,
- .end = 0xffe00400 + 0x3ff,
- .flags = IORESOURCE_MEM,
- },
- {
- .start = 4,
- .end = 4,
- .flags = IORESOURCE_IRQ,
- },
- {
- .start = 0x0c000000,
- .end = 0x0c000100,
- .flags = IORESOURCE_MEM,
- },
- {
- .start = 67,
- .end = 67,
- .flags = IORESOURCE_IRQ,
- },
-};
-
-static u64 hh_fpga0_dma_mask = DMA_BIT_MASK(32);
-static struct platform_device hh_fpga0_device = {
- .name = "hh_fpga",
- .id = 0,
- .dev = {
- .dma_mask = &hh_fpga0_dma_mask,
- .coherent_dma_mask = DMA_BIT_MASK(32),
- },
- .resource = hh_fpga0_resource,
- .num_resources = ARRAY_SIZE(hh_fpga0_resource),
-};
-
-static struct clk hh_fpga0_spi_clk = {
- .name = "spi_clk",
- .dev = &hh_fpga0_device.dev,
- .mode = pba_clk_mode,
- .get_rate = pba_clk_get_rate,
- .index = 1,
-};
-
-struct platform_device *__init at32_add_device_hh_fpga(void)
-{
- /* Select peripheral functionallity for SPI SCK and MOSI */
- at32_select_periph(GPIO_PIOB_BASE, HAMMERHEAD_FPGA_PERIPH_SCK,
- GPIO_PERIPH_B, 0);
- at32_select_periph(GPIO_PIOB_BASE, HAMMERHEAD_FPGA_PERIPH_MOSI,
- GPIO_PERIPH_B, 0);
-
- /* reserve all other needed gpio
- * We have on board pull ups, so there is no need
- * to enable gpio pull ups */
- /* INIT_DONE (input) */
- at32_select_gpio(GPIO_PIN_PB(0), 0);
-
- /* nSTATUS (input) */
- at32_select_gpio(GPIO_PIN_PB(2), 0);
-
- /* nCONFIG (output, low) */
- at32_select_gpio(GPIO_PIN_PB(3), AT32_GPIOF_OUTPUT);
-
- /* CONF_DONE (input) */
- at32_select_gpio(GPIO_PIN_PB(4), 0);
-
- /* Select EXTINT3 for PB28 (Interrupt from FPGA) */
- at32_select_periph(GPIO_PIOB_BASE, HAMMERHEAD_FPGA_PERIPH_EXTINT3,
- GPIO_PERIPH_A, 0);
-
- /* Get our parent clock */
- hh_fpga0_spi_clk.parent = clk_get(NULL, "pba");
- clk_put(hh_fpga0_spi_clk.parent);
-
- /* Register clock in at32 clock tree */
- at32_clk_register(&hh_fpga0_spi_clk);
-
- platform_device_register(&hh_fpga0_device);
- return &hh_fpga0_device;
-}
-#endif
-
-/* This needs to be called after the SMC has been initialized */
-static int __init hammerhead_flash_init(void)
-{
- int ret;
-
- smc_set_timing(&flash_config, &flash_timing);
- ret = smc_set_configuration(0, &flash_config);
-
- if (ret < 0) {
- printk(KERN_ERR "hammerhead: failed to set NOR flash timing\n");
- return ret;
- }
-
- platform_device_register(&flash_device);
-
-#ifdef CONFIG_BOARD_HAMMERHEAD_USB
- hammerhead_usbh_init();
-#endif
-
-#ifdef CONFIG_BOARD_HAMMERHEAD_FPGA
- /* Setup SMC for FPGA interface */
- smc_set_timing(&fpga_config, &fpga_timing);
- ret = smc_set_configuration(3, &fpga_config);
-#endif
-
-
- if (ret < 0) {
- printk(KERN_ERR "hammerhead: failed to set FPGA timing\n");
- return ret;
- }
-
- return 0;
-}
-
-device_initcall(hammerhead_flash_init);
diff --git a/arch/avr32/boards/hammerhead/flash.h b/arch/avr32/boards/hammerhead/flash.h
deleted file mode 100644
index ea70c62..0000000
--- a/arch/avr32/boards/hammerhead/flash.h
+++ /dev/null
@@ -1,6 +0,0 @@
-#ifndef __BOARDS_HAMMERHEAD_FLASH_H
-#define __BOARDS_HAMMERHEAD_FLASH_H
-
-struct platform_device *at32_add_device_hh_fpga(void);
-
-#endif /* __BOARDS_HAMMERHEAD_FLASH_H */
diff --git a/arch/avr32/boards/hammerhead/setup.c b/arch/avr32/boards/hammerhead/setup.c
deleted file mode 100644
index dc0e317..0000000
--- a/arch/avr32/boards/hammerhead/setup.c
+++ /dev/null
@@ -1,247 +0,0 @@
-/*
- * Board-specific setup code for the Miromico Hammerhead board
- *
- * Copyright (C) 2008 Miromico AG
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/atmel-mci.h>
-#include <linux/clk.h>
-#include <linux/fb.h>
-#include <linux/etherdevice.h>
-#include <linux/i2c.h>
-#include <linux/i2c-gpio.h>
-#include <linux/init.h>
-#include <linux/linkage.h>
-#include <linux/platform_device.h>
-#include <linux/types.h>
-#include <linux/spi/spi.h>
-
-#include <video/atmel_lcdc.h>
-
-#include <linux/io.h>
-#include <asm/setup.h>
-
-#include <mach/at32ap700x.h>
-#include <mach/board.h>
-#include <mach/init.h>
-#include <mach/portmux.h>
-
-#include <sound/atmel-ac97c.h>
-
-#include "../../mach-at32ap/clock.h"
-#include "flash.h"
-
-/* Oscillator frequencies. These are board-specific */
-unsigned long at32_board_osc_rates[3] = {
- [0] = 32768, /* 32.768 kHz on RTC osc */
- [1] = 25000000, /* 25MHz on osc0 */
- [2] = 12000000, /* 12 MHz on osc1 */
-};
-
-/* Initialized by bootloader-specific startup code. */
-struct tag *bootloader_tags __initdata;
-
-#ifdef CONFIG_BOARD_HAMMERHEAD_LCD
-static struct fb_videomode __initdata hda350tlv_modes[] = {
- {
- .name = "320x240 @ 75",
- .refresh = 75,
- .xres = 320,
- .yres = 240,
- .pixclock = KHZ2PICOS(6891),
-
- .left_margin = 48,
- .right_margin = 18,
- .upper_margin = 18,
- .lower_margin = 4,
- .hsync_len = 20,
- .vsync_len = 2,
-
- .sync = 0,
- .vmode = FB_VMODE_NONINTERLACED,
- },
-};
-
-static struct fb_monspecs __initdata hammerhead_hda350t_monspecs = {
- .manufacturer = "HAN",
- .monitor = "HDA350T-LV",
- .modedb = hda350tlv_modes,
- .modedb_len = ARRAY_SIZE(hda350tlv_modes),
- .hfmin = 14900,
- .hfmax = 22350,
- .vfmin = 60,
- .vfmax = 90,
- .dclkmax = 10000000,
-};
-
-struct atmel_lcdfb_pdata __initdata hammerhead_lcdc_data = {
- .default_bpp = 24,
- .default_dmacon = ATMEL_LCDC_DMAEN | ATMEL_LCDC_DMA2DEN,
- .default_lcdcon2 = (ATMEL_LCDC_DISTYPE_TFT
- | ATMEL_LCDC_INVCLK
- | ATMEL_LCDC_CLKMOD_ALWAYSACTIVE
- | ATMEL_LCDC_MEMOR_BIG),
- .default_monspecs = &hammerhead_hda350t_monspecs,
- .guard_time = 2,
-};
-#endif
-
-static struct mci_platform_data __initdata mci0_data = {
- .slot[0] = {
- .bus_width = 4,
- .detect_pin = -ENODEV,
- .wp_pin = -ENODEV,
- },
-};
-
-struct eth_addr {
- u8 addr[6];
-};
-
-static struct eth_addr __initdata hw_addr[1];
-static struct macb_platform_data __initdata eth_data[1];
-
-/*
- * The next two functions should go away as the boot loader is
- * supposed to initialize the macb address registers with a valid
- * ethernet address. But we need to keep it around for a while until
- * we can be reasonably sure the boot loader does this.
- *
- * The phy_id is ignored as the driver will probe for it.
- */
-static int __init parse_tag_ethernet(struct tag *tag)
-{
- int i = tag->u.ethernet.mac_index;
-
- if (i < ARRAY_SIZE(hw_addr))
- memcpy(hw_addr[i].addr, tag->u.ethernet.hw_address,
- sizeof(hw_addr[i].addr));
-
- return 0;
-}
-__tagtable(ATAG_ETHERNET, parse_tag_ethernet);
-
-static void __init set_hw_addr(struct platform_device *pdev)
-{
- struct resource *res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- const u8 *addr;
- void __iomem *regs;
- struct clk *pclk;
-
- if (!res)
- return;
-
- if (pdev->id >= ARRAY_SIZE(hw_addr))
- return;
-
- addr = hw_addr[pdev->id].addr;
-
- if (!is_valid_ether_addr(addr))
- return;
-
- /*
- * Since this is board-specific code, we'll cheat and use the
- * physical address directly as we happen to know that it's
- * the same as the virtual address.
- */
- regs = (void __iomem __force *)res->start;
- pclk = clk_get(&pdev->dev, "pclk");
-
- if (IS_ERR(pclk))
- return;
-
- clk_enable(pclk);
-
- __raw_writel((addr[3] << 24) | (addr[2] << 16) | (addr[1] << 8) |
- addr[0], regs + 0x98);
- __raw_writel((addr[5] << 8) | addr[4], regs + 0x9c);
-
- clk_disable(pclk);
- clk_put(pclk);
-}
-
-void __init setup_board(void)
-{
- at32_map_usart(1, 0, 0); /* USART 1: /dev/ttyS0, DB9 */
- at32_setup_serial_console(0);
-}
-
-static struct i2c_gpio_platform_data i2c_gpio_data = {
- .sda_pin = GPIO_PIN_PA(6),
- .scl_pin = GPIO_PIN_PA(7),
- .sda_is_open_drain = 1,
- .scl_is_open_drain = 1,
- .udelay = 2, /* close to 100 kHz */
-};
-
-static struct platform_device i2c_gpio_device = {
- .name = "i2c-gpio",
- .id = 0,
- .dev = { .platform_data = &i2c_gpio_data, },
-};
-
-static struct i2c_board_info __initdata i2c_info[] = {};
-
-#ifdef CONFIG_BOARD_HAMMERHEAD_SND
-static struct ac97c_platform_data ac97c_data = {
- .reset_pin = GPIO_PIN_PA(16),
-};
-#endif
-
-static int __init hammerhead_init(void)
-{
- /*
- * Hammerhead uses 32-bit SDRAM interface. Reserve the
- * SDRAM-specific pins so that nobody messes with them.
- */
- at32_reserve_pin(GPIO_PIOE_BASE, ATMEL_EBI_PE_DATA_ALL);
-
- at32_add_device_usart(0);
-
- /* Reserve PB29 (GCLK3). This pin is used as clock source
- * for ETH PHY (25MHz). GCLK3 setup is done by U-Boot.
- */
- at32_reserve_pin(GPIO_PIOB_BASE, (1<<29));
-
- /*
- * Hammerhead uses only one ethernet port, so we don't set
- * address of second port
- */
- set_hw_addr(at32_add_device_eth(0, ð_data[0]));
-
-#ifdef CONFIG_BOARD_HAMMERHEAD_FPGA
- at32_add_device_hh_fpga();
-#endif
- at32_add_device_mci(0, &mci0_data);
-
-#ifdef CONFIG_BOARD_HAMMERHEAD_USB
- at32_add_device_usba(0, NULL);
-#endif
-#ifdef CONFIG_BOARD_HAMMERHEAD_LCD
- at32_add_device_lcdc(0, &hammerhead_lcdc_data, fbmem_start,
- fbmem_size, ATMEL_LCDC_PRI_24BIT);
-#endif
-
- at32_select_gpio(i2c_gpio_data.sda_pin,
- AT32_GPIOF_MULTIDRV | AT32_GPIOF_OUTPUT |
- AT32_GPIOF_HIGH);
- at32_select_gpio(i2c_gpio_data.scl_pin,
- AT32_GPIOF_MULTIDRV | AT32_GPIOF_OUTPUT |
- AT32_GPIOF_HIGH);
- platform_device_register(&i2c_gpio_device);
- i2c_register_board_info(0, i2c_info, ARRAY_SIZE(i2c_info));
-
-#ifdef CONFIG_BOARD_HAMMERHEAD_SND
- at32_add_device_ac97c(0, &ac97c_data, AC97C_BOTH);
-#endif
-
- /* Select the Touchscreen interrupt pin mode */
- at32_select_periph(GPIO_PIOB_BASE, 0x08000000, GPIO_PERIPH_A, 0);
-
- return 0;
-}
-
-postcore_initcall(hammerhead_init);
diff --git a/arch/avr32/boards/merisc/Kconfig b/arch/avr32/boards/merisc/Kconfig
deleted file mode 100644
index 7e04327..0000000
--- a/arch/avr32/boards/merisc/Kconfig
+++ /dev/null
@@ -1,5 +0,0 @@
-# Merisc customization
-
-if BOARD_MERISC
-
-endif # BOARD_MERISC
diff --git a/arch/avr32/boards/merisc/Makefile b/arch/avr32/boards/merisc/Makefile
deleted file mode 100644
index d24c787..0000000
--- a/arch/avr32/boards/merisc/Makefile
+++ /dev/null
@@ -1 +0,0 @@
-obj-y += setup.o flash.o display.o merisc_sysfs.o
diff --git a/arch/avr32/boards/merisc/display.c b/arch/avr32/boards/merisc/display.c
deleted file mode 100644
index e7683ee..0000000
--- a/arch/avr32/boards/merisc/display.c
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Display setup code for the Merisc board
- *
- * Copyright (C) 2008 Martinsson Elektronik AB
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/init.h>
-#include <linux/platform_device.h>
-#include <linux/fb.h>
-#include <video/atmel_lcdc.h>
-#include <asm/setup.h>
-#include <mach/board.h>
-#include "merisc.h"
-
-static struct fb_videomode merisc_fb_videomode[] = {
- {
- .refresh = 44,
- .xres = 640,
- .yres = 480,
- .left_margin = 96,
- .right_margin = 96,
- .upper_margin = 34,
- .lower_margin = 8,
- .hsync_len = 64,
- .vsync_len = 64,
- .name = "640x480 @ 44",
- .pixclock = KHZ2PICOS(25180),
- .sync = 0,
- .vmode = FB_VMODE_NONINTERLACED,
- },
-};
-
-static struct fb_monspecs merisc_fb_monspecs = {
- .manufacturer = "Kyo",
- .monitor = "TCG075VG2AD",
- .modedb = merisc_fb_videomode,
- .modedb_len = ARRAY_SIZE(merisc_fb_videomode),
- .hfmin = 30000,
- .hfmax = 33333,
- .vfmin = 60,
- .vfmax = 90,
- .dclkmax = 30000000,
-};
-
-struct atmel_lcdfb_pdata merisc_lcdc_data = {
- .default_bpp = 24,
- .default_dmacon = ATMEL_LCDC_DMAEN | ATMEL_LCDC_DMA2DEN,
- .default_lcdcon2 = (ATMEL_LCDC_DISTYPE_TFT
- | ATMEL_LCDC_CLKMOD_ALWAYSACTIVE
- | ATMEL_LCDC_MEMOR_BIG),
- .default_monspecs = &merisc_fb_monspecs,
- .guard_time = 2,
-};
-
-static int __init merisc_display_init(void)
-{
- at32_add_device_lcdc(0, &merisc_lcdc_data, fbmem_start,
- fbmem_size, 0);
-
- return 0;
-}
-device_initcall(merisc_display_init);
diff --git a/arch/avr32/boards/merisc/flash.c b/arch/avr32/boards/merisc/flash.c
deleted file mode 100644
index 8e856fd..0000000
--- a/arch/avr32/boards/merisc/flash.c
+++ /dev/null
@@ -1,139 +0,0 @@
-/*
- * Merisc board-specific flash initialization
- *
- * Copyright (C) 2008 Martinsson Elektronik AB
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/init.h>
-#include <linux/platform_device.h>
-#include <linux/mtd/mtd.h>
-#include <linux/mtd/partitions.h>
-#include <linux/mtd/physmap.h>
-#include <mach/smc.h>
-
-/* Will be translated to units of 14.3 ns, rounded up */
-static struct smc_timing flash_timing __initdata = {
- .ncs_read_setup = 1 * 14,
- .nrd_setup = 5 * 14,
- .ncs_write_setup = 1 * 14,
- .nwe_setup = 2 * 14,
-
- .ncs_read_pulse = 12 * 14,
- .nrd_pulse = 7 * 14,
- .ncs_write_pulse = 8 * 14,
- .nwe_pulse = 4 * 14,
-
- .read_cycle = 14 * 14,
- .write_cycle = 10 * 14,
-};
-
-static struct smc_config flash_config __initdata = {
- .bus_width = 2,
- .nrd_controlled = 1,
- .nwe_controlled = 1,
- .byte_write = 1,
- .tdf_cycles = 3,
-};
-
-static struct mtd_partition flash_0_parts[] = {
- {
- .name = "boot",
- .offset = 0x00000000,
- .size = 0x00060000,
- .mask_flags = 0,
- },
- {
- .name = "kernel",
- .offset = 0x00060000,
- .size = 0x00200000,
- .mask_flags = 0,
- },
- {
- .name = "root",
- .offset = 0x00260000,
- .size = MTDPART_SIZ_FULL,
- .mask_flags = 0,
- },
-};
-
-static struct mtd_partition flash_1_parts[] = {
- {
- .name = "2ndflash",
- .offset = 0x00000000,
- .size = MTDPART_SIZ_FULL,
- .mask_flags = 0,
- },
-};
-
-static struct physmap_flash_data flash_data[] = {
- {
- .width = 2,
- .nr_parts = ARRAY_SIZE(flash_0_parts),
- .parts = flash_0_parts,
- },
- {
- .width = 2,
- .nr_parts = ARRAY_SIZE(flash_1_parts),
- .parts = flash_1_parts,
- }
-};
-
-static struct resource flash_resource[] = {
- {
- .start = 0x00000000,
- .end = 0x03ffffff,
- .flags = IORESOURCE_MEM,
- },
- {
- .start = 0x04000000,
- .end = 0x07ffffff,
- .flags = IORESOURCE_MEM,
- },
-};
-
-static struct platform_device flash_device[] = {
- {
- .name = "physmap-flash",
- .id = 0,
- .resource = &flash_resource[0],
- .num_resources = 1,
- .dev = {
- .platform_data = &flash_data[0],
- },
- },
- {
- .name = "physmap-flash",
- .id = 1,
- .resource = &flash_resource[1],
- .num_resources = 1,
- .dev = {
- .platform_data = &flash_data[1],
- },
- },
-};
-
-static int __init merisc_flash_init(void)
-{
- int ret;
- smc_set_timing(&flash_config, &flash_timing);
-
- ret = smc_set_configuration(0, &flash_config);
- if (ret < 0) {
- printk(KERN_ERR "Merisc: failed to set NOR flash timing #0\n");
- return ret;
- }
-
- ret = smc_set_configuration(4, &flash_config);
- if (ret < 0) {
- printk(KERN_ERR "Merisc: failed to set NOR flash timing #1\n");
- return ret;
- }
-
- platform_device_register(&flash_device[0]);
- platform_device_register(&flash_device[1]);
- return 0;
-}
-device_initcall(merisc_flash_init);
diff --git a/arch/avr32/boards/merisc/merisc.h b/arch/avr32/boards/merisc/merisc.h
deleted file mode 100644
index 50ffb2f..0000000
--- a/arch/avr32/boards/merisc/merisc.h
+++ /dev/null
@@ -1,18 +0,0 @@
-/*
- * Merisc exports
- *
- * Copyright (C) 2008 Martinsson Elektronik AB
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ARCH_AVR32_BOARDS_MERISC_MERISC_H
-#define __ARCH_AVR32_BOARDS_MERISC_MERISC_H
-
-const char *merisc_revision(void);
-const char *merisc_model(void);
-
-extern struct class merisc_class;
-
-#endif /* __ARCH_AVR32_BOARDS_MERISC_MERISC_H */
diff --git a/arch/avr32/boards/merisc/merisc_sysfs.c b/arch/avr32/boards/merisc/merisc_sysfs.c
deleted file mode 100644
index 5a25231..0000000
--- a/arch/avr32/boards/merisc/merisc_sysfs.c
+++ /dev/null
@@ -1,64 +0,0 @@
-/*
- * Merisc sysfs exports
- *
- * Copyright (C) 2008 Martinsson Elektronik AB
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/module.h>
-#include <linux/kernel.h>
-#include <linux/init.h>
-#include <linux/list.h>
-#include <linux/spinlock.h>
-#include <linux/device.h>
-#include <linux/timer.h>
-#include <linux/err.h>
-#include <linux/ctype.h>
-#include "merisc.h"
-
-static ssize_t merisc_model_show(struct class *class, char *buf)
-{
- ssize_t ret = 0;
-
- sprintf(buf, "%s\n", merisc_model());
- ret = strlen(buf) + 1;
-
- return ret;
-}
-
-static ssize_t merisc_revision_show(struct class *class, char *buf)
-{
- ssize_t ret = 0;
-
- sprintf(buf, "%s\n", merisc_revision());
- ret = strlen(buf) + 1;
-
- return ret;
-}
-
-static struct class_attribute merisc_class_attrs[] = {
- __ATTR(model, S_IRUGO, merisc_model_show, NULL),
- __ATTR(revision, S_IRUGO, merisc_revision_show, NULL),
- __ATTR_NULL,
-};
-
-struct class merisc_class = {
- .name = "merisc",
- .owner = THIS_MODULE,
- .class_attrs = merisc_class_attrs,
-};
-
-static int __init merisc_sysfs_init(void)
-{
- int status;
-
- status = class_register(&merisc_class);
- if (status < 0)
- return status;
-
- return 0;
-}
-
-postcore_initcall(merisc_sysfs_init);
diff --git a/arch/avr32/boards/merisc/setup.c b/arch/avr32/boards/merisc/setup.c
deleted file mode 100644
index 718a6d7..0000000
--- a/arch/avr32/boards/merisc/setup.c
+++ /dev/null
@@ -1,305 +0,0 @@
-/*
- * Board-specific setup code for the Merisc
- *
- * Copyright (C) 2008 Martinsson Elektronik AB
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/clk.h>
-#include <linux/etherdevice.h>
-#include <linux/i2c.h>
-#include <linux/i2c-gpio.h>
-#include <linux/gpio.h>
-#include <linux/init.h>
-#include <linux/linkage.h>
-#include <linux/platform_device.h>
-#include <linux/types.h>
-#include <linux/leds.h>
-#include <linux/spi/spi.h>
-#include <linux/spi/ads7846.h>
-#include <linux/irq.h>
-#include <linux/fb.h>
-#include <linux/atmel-mci.h>
-#include <linux/pwm.h>
-#include <linux/leds_pwm.h>
-
-#include <asm/io.h>
-#include <asm/setup.h>
-
-#include <mach/at32ap700x.h>
-#include <mach/board.h>
-#include <mach/init.h>
-#include <mach/portmux.h>
-
-#include "merisc.h"
-
-/* Holds the autodetected board model and revision */
-static int merisc_board_id;
-
-/* Initialized by bootloader-specific startup code. */
-struct tag *bootloader_tags __initdata;
-
-/* Oscillator frequencies. These are board specific */
-unsigned long at32_board_osc_rates[3] = {
- [0] = 32768, /* 32.768 kHz on RTC osc */
- [1] = 20000000, /* 20 MHz on osc0 */
- [2] = 12000000, /* 12 MHz on osc1 */
-};
-
-struct eth_addr {
- u8 addr[6];
-};
-
-static struct eth_addr __initdata hw_addr[2];
-static struct macb_platform_data __initdata eth_data[2];
-
-static int ads7846_get_pendown_state_PB26(void)
-{
- return !gpio_get_value(GPIO_PIN_PB(26));
-}
-
-static int ads7846_get_pendown_state_PB28(void)
-{
- return !gpio_get_value(GPIO_PIN_PB(28));
-}
-
-static struct ads7846_platform_data __initdata ads7846_data = {
- .model = 7846,
- .vref_delay_usecs = 100,
- .vref_mv = 0,
- .keep_vref_on = 0,
- .settle_delay_usecs = 150,
- .penirq_recheck_delay_usecs = 1,
- .x_plate_ohms = 800,
- .debounce_rep = 4,
- .debounce_max = 10,
- .debounce_tol = 50,
- .get_pendown_state = ads7846_get_pendown_state_PB26,
- .filter_init = NULL,
- .filter = NULL,
- .filter_cleanup = NULL,
-};
-
-static struct spi_board_info __initdata spi0_board_info[] = {
- {
- .modalias = "ads7846",
- .max_speed_hz = 3250000,
- .chip_select = 0,
- .bus_num = 0,
- .platform_data = &ads7846_data,
- .mode = SPI_MODE_0,
- },
-};
-
-static struct mci_platform_data __initdata mci0_data = {
- .slot[0] = {
- .bus_width = 4,
- .detect_pin = GPIO_PIN_PE(19),
- .wp_pin = GPIO_PIN_PE(20),
- .detect_is_active_high = true,
- },
-};
-
-static int __init parse_tag_ethernet(struct tag *tag)
-{
- int i;
-
- i = tag->u.ethernet.mac_index;
- if (i < ARRAY_SIZE(hw_addr)) {
- memcpy(hw_addr[i].addr, tag->u.ethernet.hw_address,
- sizeof(hw_addr[i].addr));
- }
-
- return 0;
-}
-__tagtable(ATAG_ETHERNET, parse_tag_ethernet);
-
-static void __init set_hw_addr(struct platform_device *pdev)
-{
- struct resource *res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- const u8 *addr;
- void __iomem *regs;
- struct clk *pclk;
-
- if (!res)
- return;
-
- if (pdev->id >= ARRAY_SIZE(hw_addr))
- return;
-
- addr = hw_addr[pdev->id].addr;
- if (!is_valid_ether_addr(addr))
- return;
-
- regs = (void __iomem __force *)res->start;
- pclk = clk_get(&pdev->dev, "pclk");
- if (IS_ERR(pclk))
- return;
-
- clk_enable(pclk);
- __raw_writel((addr[3] << 24) | (addr[2] << 16)
- | (addr[1] << 8) | addr[0], regs + 0x98);
- __raw_writel((addr[5] << 8) | addr[4], regs + 0x9c);
- clk_disable(pclk);
- clk_put(pclk);
-}
-
-static struct i2c_gpio_platform_data i2c_gpio_data = {
- .sda_pin = GPIO_PIN_PA(6),
- .scl_pin = GPIO_PIN_PA(7),
- .sda_is_open_drain = 1,
- .scl_is_open_drain = 1,
- .udelay = 2,
-};
-
-static struct platform_device i2c_gpio_device = {
- .name = "i2c-gpio",
- .id = 0,
- .dev = {
- .platform_data = &i2c_gpio_data,
- },
-};
-
-static struct i2c_board_info __initdata i2c_info[] = {
- {
- I2C_BOARD_INFO("pcf8563", 0x51)
- },
-};
-
-#if IS_ENABLED(CONFIG_LEDS_PWM)
-static struct pwm_lookup pwm_lookup[] = {
- PWM_LOOKUP("at91sam9rl-pwm", 0, "leds_pwm", "backlight",
- 5000, PWM_POLARITY_NORMAL),
-};
-
-static struct led_pwm pwm_leds[] = {
- {
- .name = "backlight",
- .max_brightness = 255,
- },
-};
-
-static struct led_pwm_platform_data pwm_data = {
- .num_leds = ARRAY_SIZE(pwm_leds),
- .leds = pwm_leds,
-};
-
-static struct platform_device leds_pwm = {
- .name = "leds_pwm",
- .id = -1,
- .dev = {
- .platform_data = &pwm_data,
- },
-};
-#endif
-
-const char *merisc_model(void)
-{
- switch (merisc_board_id) {
- case 0:
- case 1:
- return "500-01";
- case 2:
- return "BT";
- default:
- return "Unknown";
- }
-}
-
-const char *merisc_revision(void)
-{
- switch (merisc_board_id) {
- case 0:
- return "B";
- case 1:
- return "D";
- case 2:
- return "A";
- default:
- return "Unknown";
- }
-}
-
-static void detect_merisc_board_id(void)
-{
- /* Board ID pins MUST be set as input or the board may be damaged */
- at32_select_gpio(GPIO_PIN_PA(24), AT32_GPIOF_PULLUP);
- at32_select_gpio(GPIO_PIN_PA(25), AT32_GPIOF_PULLUP);
- at32_select_gpio(GPIO_PIN_PA(26), AT32_GPIOF_PULLUP);
- at32_select_gpio(GPIO_PIN_PA(27), AT32_GPIOF_PULLUP);
-
- merisc_board_id = !gpio_get_value(GPIO_PIN_PA(24)) +
- !gpio_get_value(GPIO_PIN_PA(25)) * 2 +
- !gpio_get_value(GPIO_PIN_PA(26)) * 4 +
- !gpio_get_value(GPIO_PIN_PA(27)) * 8;
-}
-
-void __init setup_board(void)
-{
- at32_map_usart(0, 0, 0);
- at32_map_usart(1, 1, 0);
- at32_map_usart(3, 3, 0);
- at32_setup_serial_console(1);
-}
-
-static int __init merisc_init(void)
-{
- detect_merisc_board_id();
-
- printk(KERN_NOTICE "BOARD: Merisc %s revision %s\n", merisc_model(),
- merisc_revision());
-
- /* Reserve pins for SDRAM */
- at32_reserve_pin(GPIO_PIOE_BASE, ATMEL_EBI_PE_DATA_ALL | (1 << 26));
-
- if (merisc_board_id >= 1)
- at32_map_usart(2, 2, 0);
-
- at32_add_device_usart(0);
- at32_add_device_usart(1);
- if (merisc_board_id >= 1)
- at32_add_device_usart(2);
- at32_add_device_usart(3);
- set_hw_addr(at32_add_device_eth(0, ð_data[0]));
-
- /* ADS7846 PENIRQ */
- if (merisc_board_id == 0) {
- ads7846_data.get_pendown_state = ads7846_get_pendown_state_PB26;
- at32_select_periph(GPIO_PIOB_BASE, 1 << 26,
- GPIO_PERIPH_A, AT32_GPIOF_PULLUP);
- spi0_board_info[0].irq = AT32_EXTINT(1);
- } else {
- ads7846_data.get_pendown_state = ads7846_get_pendown_state_PB28;
- at32_select_periph(GPIO_PIOB_BASE, 1 << 28, GPIO_PERIPH_A,
- AT32_GPIOF_PULLUP);
- spi0_board_info[0].irq = AT32_EXTINT(3);
- }
-
- /* ADS7846 busy pin */
- at32_select_gpio(GPIO_PIN_PA(4), AT32_GPIOF_PULLUP);
-
- at32_add_device_spi(0, spi0_board_info, ARRAY_SIZE(spi0_board_info));
-
- at32_add_device_mci(0, &mci0_data);
-
-#if IS_ENABLED(CONFIG_LEDS_PWM)
- pwm_add_table(pwm_lookup, ARRAY_SIZE(pwm_lookup));
- at32_add_device_pwm((1 << 0) | (1 << 2));
- platform_device_register(&leds_pwm);
-#else
- at32_add_device_pwm((1 << 2));
-#endif
-
- at32_select_gpio(i2c_gpio_data.sda_pin,
- AT32_GPIOF_MULTIDRV | AT32_GPIOF_OUTPUT | AT32_GPIOF_HIGH);
- at32_select_gpio(i2c_gpio_data.scl_pin,
- AT32_GPIOF_MULTIDRV | AT32_GPIOF_OUTPUT | AT32_GPIOF_HIGH);
- platform_device_register(&i2c_gpio_device);
-
- i2c_register_board_info(0, i2c_info, ARRAY_SIZE(i2c_info));
-
- return 0;
-}
-postcore_initcall(merisc_init);
diff --git a/arch/avr32/boards/mimc200/Makefile b/arch/avr32/boards/mimc200/Makefile
deleted file mode 100644
index c740aa1..0000000
--- a/arch/avr32/boards/mimc200/Makefile
+++ /dev/null
@@ -1 +0,0 @@
-obj-y += setup.o flash.o
diff --git a/arch/avr32/boards/mimc200/flash.c b/arch/avr32/boards/mimc200/flash.c
deleted file mode 100644
index d83d650..0000000
--- a/arch/avr32/boards/mimc200/flash.c
+++ /dev/null
@@ -1,143 +0,0 @@
-/*
- * MIMC200 board-specific flash initialization
- *
- * Copyright (C) 2008 Mercury IMC Ltd
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/init.h>
-#include <linux/platform_device.h>
-#include <linux/mtd/mtd.h>
-#include <linux/mtd/partitions.h>
-#include <linux/mtd/physmap.h>
-
-#include <mach/smc.h>
-
-static struct smc_timing flash_timing __initdata = {
- .ncs_read_setup = 0,
- .nrd_setup = 15,
- .ncs_write_setup = 0,
- .nwe_setup = 0,
-
- .ncs_read_pulse = 115,
- .nrd_pulse = 110,
- .ncs_write_pulse = 60,
- .nwe_pulse = 60,
-
- .read_cycle = 115,
- .write_cycle = 100,
-};
-
-static struct smc_config flash_config __initdata = {
- .bus_width = 2,
- .nrd_controlled = 1,
- .nwe_controlled = 1,
- .byte_write = 1,
-};
-
-/* system flash definition */
-
-static struct mtd_partition flash_parts_system[] = {
- {
- .name = "u-boot",
- .offset = 0x00000000,
- .size = 0x00020000, /* 128 KiB */
- .mask_flags = MTD_WRITEABLE,
- },
- {
- .name = "root",
- .offset = 0x00020000,
- .size = 0x007c0000,
- },
- {
- .name = "splash",
- .offset = 0x007e0000,
- .size = 0x00010000, /* 64KiB */
- },
- {
- .name = "env",
- .offset = 0x007f0000,
- .size = 0x00010000,
- .mask_flags = MTD_WRITEABLE,
- },
-};
-
-static struct physmap_flash_data flash_system = {
- .width = 2,
- .nr_parts = ARRAY_SIZE(flash_parts_system),
- .parts = flash_parts_system,
-};
-
-static struct resource flash_resource_system = {
- .start = 0x00000000,
- .end = 0x007fffff,
- .flags = IORESOURCE_MEM,
-};
-
-static struct platform_device flash_device_system = {
- .name = "physmap-flash",
- .id = 0,
- .resource = &flash_resource_system,
- .num_resources = 1,
- .dev = {
- .platform_data = &flash_system,
- },
-};
-
-/* data flash definition */
-
-static struct mtd_partition flash_parts_data[] = {
- {
- .name = "data",
- .offset = 0x00000000,
- .size = 0x00800000,
- },
-};
-
-static struct physmap_flash_data flash_data = {
- .width = 2,
- .nr_parts = ARRAY_SIZE(flash_parts_data),
- .parts = flash_parts_data,
-};
-
-static struct resource flash_resource_data = {
- .start = 0x08000000,
- .end = 0x087fffff,
- .flags = IORESOURCE_MEM,
-};
-
-static struct platform_device flash_device_data = {
- .name = "physmap-flash",
- .id = 1,
- .resource = &flash_resource_data,
- .num_resources = 1,
- .dev = {
- .platform_data = &flash_data,
- },
-};
-
-/* This needs to be called after the SMC has been initialized */
-static int __init mimc200_flash_init(void)
-{
- int ret;
-
- smc_set_timing(&flash_config, &flash_timing);
- ret = smc_set_configuration(0, &flash_config);
- if (ret < 0) {
- printk(KERN_ERR "mimc200: failed to set 'System' NOR flash timing\n");
- return ret;
- }
- ret = smc_set_configuration(1, &flash_config);
- if (ret < 0) {
- printk(KERN_ERR "mimc200: failed to set 'Data' NOR flash timing\n");
- return ret;
- }
-
- platform_device_register(&flash_device_system);
- platform_device_register(&flash_device_data);
-
- return 0;
-}
-device_initcall(mimc200_flash_init);
diff --git a/arch/avr32/boards/mimc200/setup.c b/arch/avr32/boards/mimc200/setup.c
deleted file mode 100644
index 1cb8e9c..0000000
--- a/arch/avr32/boards/mimc200/setup.c
+++ /dev/null
@@ -1,236 +0,0 @@
-/*
- * Board-specific setup code for the MIMC200
- *
- * Copyright (C) 2008 Mercury IMC Ltd
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-extern struct atmel_lcdfb_pdata mimc200_lcdc_data;
-
-#include <linux/clk.h>
-#include <linux/etherdevice.h>
-#include <linux/i2c-gpio.h>
-#include <linux/init.h>
-#include <linux/linkage.h>
-#include <linux/platform_device.h>
-#include <linux/types.h>
-#include <linux/leds.h>
-#include <linux/spi/spi.h>
-#include <linux/spi/eeprom.h>
-
-#include <video/atmel_lcdc.h>
-#include <linux/fb.h>
-
-#include <linux/atmel-mci.h>
-#include <linux/io.h>
-#include <asm/setup.h>
-
-#include <mach/at32ap700x.h>
-#include <mach/board.h>
-#include <mach/init.h>
-#include <mach/portmux.h>
-
-/* Oscillator frequencies. These are board-specific */
-unsigned long at32_board_osc_rates[3] = {
- [0] = 32768, /* 32.768 kHz on RTC osc */
- [1] = 10000000, /* 10 MHz on osc0 */
- [2] = 12000000, /* 12 MHz on osc1 */
-};
-
-/* Initialized by bootloader-specific startup code. */
-struct tag *bootloader_tags __initdata;
-
-static struct fb_videomode __initdata pt0434827_modes[] = {
- {
- .name = "480x272 @ 72",
- .refresh = 72,
- .xres = 480, .yres = 272,
- .pixclock = KHZ2PICOS(10000),
-
- .left_margin = 1, .right_margin = 1,
- .upper_margin = 12, .lower_margin = 1,
- .hsync_len = 42, .vsync_len = 1,
-
- .sync = 0,
- .vmode = FB_VMODE_NONINTERLACED,
- },
-};
-
-static struct fb_monspecs __initdata mimc200_default_monspecs = {
- .manufacturer = "PT",
- .monitor = "PT0434827-A401",
- .modedb = pt0434827_modes,
- .modedb_len = ARRAY_SIZE(pt0434827_modes),
- .hfmin = 14820,
- .hfmax = 22230,
- .vfmin = 60,
- .vfmax = 85,
- .dclkmax = 25200000,
-};
-
-struct atmel_lcdfb_pdata __initdata mimc200_lcdc_data = {
- .default_bpp = 16,
- .default_dmacon = ATMEL_LCDC_DMAEN | ATMEL_LCDC_DMA2DEN,
- .default_lcdcon2 = (ATMEL_LCDC_DISTYPE_TFT
- | ATMEL_LCDC_INVCLK
- | ATMEL_LCDC_CLKMOD_ALWAYSACTIVE
- | ATMEL_LCDC_MEMOR_BIG),
- .default_monspecs = &mimc200_default_monspecs,
- .guard_time = 2,
-};
-
-struct eth_addr {
- u8 addr[6];
-};
-static struct eth_addr __initdata hw_addr[2];
-static struct macb_platform_data __initdata eth_data[2];
-
-static struct spi_eeprom eeprom_25lc010 = {
- .name = "25lc010",
- .byte_len = 128,
- .page_size = 16,
- .flags = EE_ADDR1,
-};
-
-static struct spi_board_info spi0_board_info[] __initdata = {
- {
- .modalias = "rtc-ds1390",
- .max_speed_hz = 4000000,
- .chip_select = 2,
- },
- {
- .modalias = "at25",
- .max_speed_hz = 1000000,
- .chip_select = 1,
- .mode = SPI_MODE_3,
- .platform_data = &eeprom_25lc010,
- },
-};
-
-static struct mci_platform_data __initdata mci0_data = {
- .slot[0] = {
- .bus_width = 4,
- .detect_pin = GPIO_PIN_PA(26),
- .wp_pin = GPIO_PIN_PA(27),
- },
-};
-
-/*
- * The next two functions should go away as the boot loader is
- * supposed to initialize the macb address registers with a valid
- * ethernet address. But we need to keep it around for a while until
- * we can be reasonably sure the boot loader does this.
- *
- * The phy_id is ignored as the driver will probe for it.
- */
-static int __init parse_tag_ethernet(struct tag *tag)
-{
- int i;
-
- i = tag->u.ethernet.mac_index;
- if (i < ARRAY_SIZE(hw_addr))
- memcpy(hw_addr[i].addr, tag->u.ethernet.hw_address,
- sizeof(hw_addr[i].addr));
-
- return 0;
-}
-__tagtable(ATAG_ETHERNET, parse_tag_ethernet);
-
-static void __init set_hw_addr(struct platform_device *pdev)
-{
- struct resource *res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- const u8 *addr;
- void __iomem *regs;
- struct clk *pclk;
-
- if (!res)
- return;
- if (pdev->id >= ARRAY_SIZE(hw_addr))
- return;
-
- addr = hw_addr[pdev->id].addr;
- if (!is_valid_ether_addr(addr))
- return;
-
- /*
- * Since this is board-specific code, we'll cheat and use the
- * physical address directly as we happen to know that it's
- * the same as the virtual address.
- */
- regs = (void __iomem __force *)res->start;
- pclk = clk_get(&pdev->dev, "pclk");
- if (IS_ERR(pclk))
- return;
-
- clk_enable(pclk);
- __raw_writel((addr[3] << 24) | (addr[2] << 16)
- | (addr[1] << 8) | addr[0], regs + 0x98);
- __raw_writel((addr[5] << 8) | addr[4], regs + 0x9c);
- clk_disable(pclk);
- clk_put(pclk);
-}
-
-void __init setup_board(void)
-{
- at32_map_usart(0, 0, 0); /* USART 0: /dev/ttyS0 (TTL --> Altera) */
- at32_map_usart(1, 1, 0); /* USART 1: /dev/ttyS1 (RS232) */
- at32_map_usart(2, 2, 0); /* USART 2: /dev/ttyS2 (RS485) */
- at32_map_usart(3, 3, 0); /* USART 3: /dev/ttyS3 (RS422 Multidrop) */
-}
-
-static struct i2c_gpio_platform_data i2c_gpio_data = {
- .sda_pin = GPIO_PIN_PA(6),
- .scl_pin = GPIO_PIN_PA(7),
- .sda_is_open_drain = 1,
- .scl_is_open_drain = 1,
- .udelay = 2, /* close to 100 kHz */
-};
-
-static struct platform_device i2c_gpio_device = {
- .name = "i2c-gpio",
- .id = 0,
- .dev = {
- .platform_data = &i2c_gpio_data,
- },
-};
-
-static struct i2c_board_info __initdata i2c_info[] = {
-};
-
-static int __init mimc200_init(void)
-{
- /*
- * MIMC200 uses 16-bit SDRAM interface, so we don't need to
- * reserve any pins for it.
- */
-
- at32_add_device_usart(0);
- at32_add_device_usart(1);
- at32_add_device_usart(2);
- at32_add_device_usart(3);
-
- set_hw_addr(at32_add_device_eth(0, ð_data[0]));
- set_hw_addr(at32_add_device_eth(1, ð_data[1]));
-
- at32_add_device_spi(0, spi0_board_info, ARRAY_SIZE(spi0_board_info));
- at32_add_device_mci(0, &mci0_data);
- at32_add_device_usba(0, NULL);
-
- at32_select_periph(GPIO_PIOB_BASE, 1 << 28, 0, AT32_GPIOF_PULLUP);
- at32_select_gpio(i2c_gpio_data.sda_pin,
- AT32_GPIOF_MULTIDRV | AT32_GPIOF_OUTPUT | AT32_GPIOF_HIGH);
- at32_select_gpio(i2c_gpio_data.scl_pin,
- AT32_GPIOF_MULTIDRV | AT32_GPIOF_OUTPUT | AT32_GPIOF_HIGH);
- platform_device_register(&i2c_gpio_device);
- i2c_register_board_info(0, i2c_info, ARRAY_SIZE(i2c_info));
-
- at32_add_device_lcdc(0, &mimc200_lcdc_data,
- fbmem_start, fbmem_size,
- ATMEL_LCDC_CONTROL | ATMEL_LCDC_ALT_CONTROL | ATMEL_LCDC_ALT_24B_DATA);
-
- return 0;
-}
-postcore_initcall(mimc200_init);
diff --git a/arch/avr32/boot/images/.gitignore b/arch/avr32/boot/images/.gitignore
deleted file mode 100644
index 64ea9d0..0000000
--- a/arch/avr32/boot/images/.gitignore
+++ /dev/null
@@ -1,4 +0,0 @@
-uImage
-uImage.srec
-vmlinux.cso
-sfdwarf.log
diff --git a/arch/avr32/boot/images/Makefile b/arch/avr32/boot/images/Makefile
deleted file mode 100644
index 2a3b539..0000000
--- a/arch/avr32/boot/images/Makefile
+++ /dev/null
@@ -1,57 +0,0 @@
-#
-# Copyright (C) 2004-2006 Atmel Corporation
-#
-# This file is subject to the terms and conditions of the GNU General Public
-# License. See the file "COPYING" in the main directory of this archive
-# for more details.
-#
-
-extra-y := vmlinux.bin vmlinux.gz
-
-OBJCOPYFLAGS_vmlinux.bin := -O binary -R .note.gnu.build-id
-$(obj)/vmlinux.bin: vmlinux FORCE
- $(call if_changed,objcopy)
-
-$(obj)/vmlinux.gz: $(obj)/vmlinux.bin FORCE
- $(call if_changed,gzip)
-
-UIMAGE_LOADADDR = $(CONFIG_LOAD_ADDRESS)
-UIMAGE_ENTRYADDR = $(CONFIG_ENTRY_ADDRESS)
-UIMAGE_COMPRESSION = gzip
-
-targets += uImage uImage.srec
-$(obj)/uImage: $(obj)/vmlinux.gz
- $(call if_changed,uimage)
- @echo ' Image $@ is ready'
-
-OBJCOPYFLAGS_uImage.srec := -I binary -O srec
-$(obj)/uImage.srec: $(obj)/uImage
- $(call if_changed,objcopy)
-
-OBJCOPYFLAGS_vmlinux.elf := --change-section-lma .text-0x80000000 \
- --change-section-lma __ex_table-0x80000000 \
- --change-section-lma .rodata-0x80000000 \
- --change-section-lma .data-0x80000000 \
- --change-section-lma .init-0x80000000 \
- --change-section-lma .bss-0x80000000 \
- --change-section-lma __param-0x80000000 \
- --change-section-lma __ksymtab-0x80000000 \
- --change-section-lma __ksymtab_gpl-0x80000000 \
- --change-section-lma __kcrctab-0x80000000 \
- --change-section-lma __kcrctab_gpl-0x80000000 \
- --change-section-lma __ksymtab_strings-0x80000000 \
- --set-start 0xa0000000
-$(obj)/vmlinux.elf: vmlinux FORCE
- $(call if_changed,objcopy)
-
-quiet_cmd_sfdwarf = SFDWARF $@
- cmd_sfdwarf = sfdwarf $< TO $@ GNUAVR IW $(SFDWARF_FLAGS) > $(obj)/sfdwarf.log
-
-$(obj)/vmlinux.cso: $(obj)/vmlinux.elf FORCE
- $(call if_changed,sfdwarf)
-
-install: $(BOOTIMAGE)
- sh $(srctree)/install-kernel.sh $<
-
-# Generated files to be removed upon make clean
-clean-files := vmlinux.elf vmlinux.bin vmlinux.gz uImage uImage.srec
diff --git a/arch/avr32/boot/u-boot/Makefile b/arch/avr32/boot/u-boot/Makefile
deleted file mode 100644
index 125ddc9..0000000
--- a/arch/avr32/boot/u-boot/Makefile
+++ /dev/null
@@ -1,3 +0,0 @@
-extra-y := head.o
-
-obj-y := empty.o
diff --git a/arch/avr32/boot/u-boot/empty.S b/arch/avr32/boot/u-boot/empty.S
deleted file mode 100644
index 8ac91a5..0000000
--- a/arch/avr32/boot/u-boot/empty.S
+++ /dev/null
@@ -1 +0,0 @@
-/* Empty file */
diff --git a/arch/avr32/boot/u-boot/head.S b/arch/avr32/boot/u-boot/head.S
deleted file mode 100644
index 2ffc298..0000000
--- a/arch/avr32/boot/u-boot/head.S
+++ /dev/null
@@ -1,83 +0,0 @@
-/*
- * Startup code for use with the u-boot bootloader.
- *
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <asm/setup.h>
-#include <asm/thread_info.h>
-#include <asm/sysreg.h>
-
- /*
- * The kernel is loaded where we want it to be and all caches
- * have just been flushed. We get two parameters from u-boot:
- *
- * r12 contains a magic number (ATAG_MAGIC)
- * r11 points to a tag table providing information about
- * the system.
- */
- .section .init.text,"ax"
- .global _start
-_start:
- /* Initialize .bss */
- lddpc r2, bss_start_addr
- lddpc r3, end_addr
- mov r0, 0
- mov r1, 0
-1: st.d r2++, r0
- cp r2, r3
- brlo 1b
-
- /* Initialize status register */
- lddpc r0, init_sr
- mtsr SYSREG_SR, r0
-
- /* Set initial stack pointer */
- lddpc sp, stack_addr
- sub sp, -THREAD_SIZE
-
-#ifdef CONFIG_FRAME_POINTER
- /* Mark last stack frame */
- mov lr, 0
- mov r7, 0
-#endif
-
- /* Check if the boot loader actually provided a tag table */
- lddpc r0, magic_number
- cp.w r12, r0
- brne no_tag_table
-
- /*
- * Save the tag table address for later use. This must be done
- * _after_ .bss has been initialized...
- */
- lddpc r0, tag_table_addr
- st.w r0[0], r11
-
- /* Jump to loader-independent setup code */
- rjmp kernel_entry
-
- .align 2
-magic_number:
- .long ATAG_MAGIC
-tag_table_addr:
- .long bootloader_tags
-bss_start_addr:
- .long __bss_start
-end_addr:
- .long _end
-init_sr:
- .long 0x007f0000 /* Supervisor mode, everything masked */
-stack_addr:
- .long init_thread_union
-panic_addr:
- .long panic
-
-no_tag_table:
- sub r12, pc, (. - 2f)
- /* branch to panic() which can be far away with that construct */
- lddpc pc, panic_addr
-2: .asciz "Boot loader didn't provide correct magic number\n"
diff --git a/arch/avr32/configs/atngw100_defconfig b/arch/avr32/configs/atngw100_defconfig
deleted file mode 100644
index ce00300..0000000
--- a/arch/avr32/configs/atngw100_defconfig
+++ /dev/null
@@ -1,142 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_RELAY=y
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_COMPAT_BRK is not set
-CONFIG_PROFILING=y
-CONFIG_OPROFILE=m
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-CONFIG_BOARD_ATNGW100_MKI=y
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_NMI_DEBUGGING=y
-CONFIG_CPU_FREQ=y
-# CONFIG_CPU_FREQ_STAT is not set
-CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_XFRM_USER=y
-CONFIG_NET_KEY=y
-CONFIG_INET=y
-CONFIG_IP_MULTICAST=y
-CONFIG_IP_ADVANCED_ROUTER=y
-CONFIG_IP_PNP=y
-CONFIG_IP_PNP_DHCP=y
-CONFIG_IP_MROUTE=y
-CONFIG_IP_PIMSM_V1=y
-CONFIG_SYN_COOKIES=y
-CONFIG_INET_AH=y
-CONFIG_INET_ESP=y
-CONFIG_INET_IPCOMP=y
-# CONFIG_INET_LRO is not set
-CONFIG_IPV6=y
-CONFIG_INET6_AH=y
-CONFIG_INET6_ESP=y
-CONFIG_INET6_IPCOMP=y
-CONFIG_NETFILTER=y
-# CONFIG_NETFILTER_ADVANCED is not set
-CONFIG_NETFILTER_XTABLES=y
-CONFIG_BRIDGE=m
-CONFIG_VLAN_8021Q=m
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_AMDSTD=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_DATAFLASH=y
-CONFIG_MTD_UBI=y
-CONFIG_BLK_DEV_LOOP=m
-CONFIG_BLK_DEV_NBD=m
-CONFIG_BLK_DEV_RAM=m
-CONFIG_ATMEL_TCLIB=y
-CONFIG_NETDEVICES=y
-CONFIG_TUN=m
-CONFIG_MACB=y
-CONFIG_PPP=m
-CONFIG_PPP_BSDCOMP=m
-CONFIG_PPP_DEFLATE=m
-CONFIG_PPP_FILTER=y
-CONFIG_PPP_MPPE=m
-CONFIG_PPPOE=m
-CONFIG_PPP_ASYNC=m
-# CONFIG_INPUT is not set
-# CONFIG_SERIO is not set
-# CONFIG_VT is not set
-# CONFIG_LEGACY_PTYS is not set
-# CONFIG_DEVKMEM is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=m
-CONFIG_I2C_CHARDEV=m
-CONFIG_I2C_GPIO=m
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=m
-CONFIG_GPIO_SYSFS=y
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_USB_GADGET=y
-CONFIG_USB_GADGET_VBUS_DRAW=350
-CONFIG_USB_ZERO=m
-CONFIG_USB_ETH=m
-CONFIG_USB_GADGETFS=m
-CONFIG_USB_MASS_STORAGE=m
-CONFIG_USB_G_SERIAL=m
-CONFIG_USB_CDC_COMPOSITE=m
-CONFIG_MMC=y
-CONFIG_MMC_TEST=m
-CONFIG_MMC_ATMELMCI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_GPIO=y
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_TIMER=y
-CONFIG_LEDS_TRIGGER_HEARTBEAT=y
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_AT32AP700X=y
-CONFIG_DMADEVICES=y
-CONFIG_EXT2_FS=y
-CONFIG_EXT3_FS=y
-# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
-# CONFIG_EXT3_FS_XATTR is not set
-CONFIG_EXT4_FS=y
-# CONFIG_DNOTIFY is not set
-CONFIG_FUSE_FS=m
-CONFIG_MSDOS_FS=m
-CONFIG_VFAT_FS=m
-CONFIG_FAT_DEFAULT_CODEPAGE=850
-CONFIG_PROC_KCORE=y
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_UBIFS_FS=y
-CONFIG_NFS_FS=y
-CONFIG_ROOT_NFS=y
-CONFIG_NFSD=m
-CONFIG_NFSD_V3=y
-CONFIG_CIFS=m
-CONFIG_NLS_CODEPAGE_437=m
-CONFIG_NLS_CODEPAGE_850=m
-CONFIG_NLS_ISO8859_1=m
-CONFIG_NLS_UTF8=m
-CONFIG_DEBUG_FS=y
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-CONFIG_DETECT_HUNG_TASK=y
diff --git a/arch/avr32/configs/atngw100_evklcd100_defconfig b/arch/avr32/configs/atngw100_evklcd100_defconfig
deleted file mode 100644
index 01ff632..0000000
--- a/arch/avr32/configs/atngw100_evklcd100_defconfig
+++ /dev/null
@@ -1,158 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_RELAY=y
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_COMPAT_BRK is not set
-CONFIG_PROFILING=y
-CONFIG_OPROFILE=m
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-CONFIG_BOARD_ATNGW100_MKI=y
-CONFIG_BOARD_ATNGW100_EVKLCD10X=y
-CONFIG_BOARD_ATNGW100_EVKLCD10X_QVGA=y
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_NMI_DEBUGGING=y
-CONFIG_CPU_FREQ=y
-# CONFIG_CPU_FREQ_STAT is not set
-CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_XFRM_USER=y
-CONFIG_NET_KEY=y
-CONFIG_INET=y
-CONFIG_IP_MULTICAST=y
-CONFIG_IP_ADVANCED_ROUTER=y
-CONFIG_IP_PNP=y
-CONFIG_IP_PNP_DHCP=y
-CONFIG_IP_MROUTE=y
-CONFIG_IP_PIMSM_V1=y
-CONFIG_SYN_COOKIES=y
-CONFIG_INET_AH=y
-CONFIG_INET_ESP=y
-CONFIG_INET_IPCOMP=y
-# CONFIG_INET_LRO is not set
-CONFIG_IPV6=y
-CONFIG_INET6_AH=y
-CONFIG_INET6_ESP=y
-CONFIG_INET6_IPCOMP=y
-CONFIG_NETFILTER=y
-# CONFIG_NETFILTER_ADVANCED is not set
-CONFIG_NETFILTER_XTABLES=y
-CONFIG_BRIDGE=m
-CONFIG_VLAN_8021Q=m
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_AMDSTD=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_DATAFLASH=y
-CONFIG_MTD_UBI=y
-CONFIG_BLK_DEV_LOOP=m
-CONFIG_BLK_DEV_NBD=m
-CONFIG_BLK_DEV_RAM=m
-CONFIG_ATMEL_TCLIB=y
-CONFIG_NETDEVICES=y
-CONFIG_TUN=m
-CONFIG_MACB=y
-CONFIG_PPP=m
-CONFIG_PPP_BSDCOMP=m
-CONFIG_PPP_DEFLATE=m
-CONFIG_PPP_FILTER=y
-CONFIG_PPP_MPPE=m
-CONFIG_PPPOE=m
-CONFIG_PPP_ASYNC=m
-# CONFIG_INPUT_MOUSEDEV is not set
-CONFIG_INPUT_EVDEV=m
-# CONFIG_INPUT_KEYBOARD is not set
-# CONFIG_INPUT_MOUSE is not set
-CONFIG_INPUT_TOUCHSCREEN=y
-CONFIG_TOUCHSCREEN_WM97XX=m
-# CONFIG_SERIO is not set
-# CONFIG_LEGACY_PTYS is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=m
-CONFIG_I2C_CHARDEV=m
-CONFIG_I2C_GPIO=m
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=m
-CONFIG_GPIO_SYSFS=y
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_FB=y
-CONFIG_FB_ATMEL=y
-CONFIG_SOUND=y
-CONFIG_SND=y
-CONFIG_SND_MIXER_OSS=m
-CONFIG_SND_PCM_OSS=m
-CONFIG_SND_HRTIMER=y
-# CONFIG_SND_SUPPORT_OLD_API is not set
-# CONFIG_SND_DRIVERS is not set
-CONFIG_SND_ATMEL_AC97C=m
-# CONFIG_SND_SPI is not set
-CONFIG_USB_GADGET=y
-CONFIG_USB_GADGET_VBUS_DRAW=350
-CONFIG_USB_ZERO=m
-CONFIG_USB_ETH=m
-CONFIG_USB_GADGETFS=m
-CONFIG_USB_MASS_STORAGE=m
-CONFIG_USB_G_SERIAL=m
-CONFIG_USB_CDC_COMPOSITE=m
-CONFIG_MMC=y
-CONFIG_MMC_TEST=m
-CONFIG_MMC_ATMELMCI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_GPIO=y
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_TIMER=y
-CONFIG_LEDS_TRIGGER_HEARTBEAT=y
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_AT32AP700X=y
-CONFIG_DMADEVICES=y
-CONFIG_EXT2_FS=y
-CONFIG_EXT3_FS=y
-# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
-# CONFIG_EXT3_FS_XATTR is not set
-CONFIG_EXT4_FS=y
-# CONFIG_DNOTIFY is not set
-CONFIG_FUSE_FS=m
-CONFIG_MSDOS_FS=m
-CONFIG_VFAT_FS=m
-CONFIG_FAT_DEFAULT_CODEPAGE=850
-CONFIG_PROC_KCORE=y
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_UBIFS_FS=y
-CONFIG_NFS_FS=y
-CONFIG_ROOT_NFS=y
-CONFIG_NFSD=m
-CONFIG_NFSD_V3=y
-CONFIG_CIFS=m
-CONFIG_NLS_CODEPAGE_437=m
-CONFIG_NLS_CODEPAGE_850=m
-CONFIG_NLS_ISO8859_1=m
-CONFIG_NLS_UTF8=m
-CONFIG_DEBUG_FS=y
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-CONFIG_DETECT_HUNG_TASK=y
diff --git a/arch/avr32/configs/atngw100_evklcd101_defconfig b/arch/avr32/configs/atngw100_evklcd101_defconfig
deleted file mode 100644
index c4021df..0000000
--- a/arch/avr32/configs/atngw100_evklcd101_defconfig
+++ /dev/null
@@ -1,157 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_RELAY=y
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_COMPAT_BRK is not set
-CONFIG_PROFILING=y
-CONFIG_OPROFILE=m
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-CONFIG_BOARD_ATNGW100_MKI=y
-CONFIG_BOARD_ATNGW100_EVKLCD10X=y
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_NMI_DEBUGGING=y
-CONFIG_CPU_FREQ=y
-# CONFIG_CPU_FREQ_STAT is not set
-CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_XFRM_USER=y
-CONFIG_NET_KEY=y
-CONFIG_INET=y
-CONFIG_IP_MULTICAST=y
-CONFIG_IP_ADVANCED_ROUTER=y
-CONFIG_IP_PNP=y
-CONFIG_IP_PNP_DHCP=y
-CONFIG_IP_MROUTE=y
-CONFIG_IP_PIMSM_V1=y
-CONFIG_SYN_COOKIES=y
-CONFIG_INET_AH=y
-CONFIG_INET_ESP=y
-CONFIG_INET_IPCOMP=y
-# CONFIG_INET_LRO is not set
-CONFIG_IPV6=y
-CONFIG_INET6_AH=y
-CONFIG_INET6_ESP=y
-CONFIG_INET6_IPCOMP=y
-CONFIG_NETFILTER=y
-# CONFIG_NETFILTER_ADVANCED is not set
-CONFIG_NETFILTER_XTABLES=y
-CONFIG_BRIDGE=m
-CONFIG_VLAN_8021Q=m
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_AMDSTD=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_DATAFLASH=y
-CONFIG_MTD_UBI=y
-CONFIG_BLK_DEV_LOOP=m
-CONFIG_BLK_DEV_NBD=m
-CONFIG_BLK_DEV_RAM=m
-CONFIG_ATMEL_TCLIB=y
-CONFIG_NETDEVICES=y
-CONFIG_TUN=m
-CONFIG_MACB=y
-CONFIG_PPP=m
-CONFIG_PPP_BSDCOMP=m
-CONFIG_PPP_DEFLATE=m
-CONFIG_PPP_FILTER=y
-CONFIG_PPP_MPPE=m
-CONFIG_PPPOE=m
-CONFIG_PPP_ASYNC=m
-# CONFIG_INPUT_MOUSEDEV is not set
-CONFIG_INPUT_EVDEV=m
-# CONFIG_INPUT_KEYBOARD is not set
-# CONFIG_INPUT_MOUSE is not set
-CONFIG_INPUT_TOUCHSCREEN=y
-CONFIG_TOUCHSCREEN_WM97XX=m
-# CONFIG_SERIO is not set
-# CONFIG_LEGACY_PTYS is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=m
-CONFIG_I2C_CHARDEV=m
-CONFIG_I2C_GPIO=m
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=m
-CONFIG_GPIO_SYSFS=y
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_FB=y
-CONFIG_FB_ATMEL=y
-CONFIG_SOUND=y
-CONFIG_SND=y
-CONFIG_SND_MIXER_OSS=m
-CONFIG_SND_PCM_OSS=m
-CONFIG_SND_HRTIMER=y
-# CONFIG_SND_SUPPORT_OLD_API is not set
-# CONFIG_SND_DRIVERS is not set
-CONFIG_SND_ATMEL_AC97C=m
-# CONFIG_SND_SPI is not set
-CONFIG_USB_GADGET=y
-CONFIG_USB_GADGET_VBUS_DRAW=350
-CONFIG_USB_ZERO=m
-CONFIG_USB_ETH=m
-CONFIG_USB_GADGETFS=m
-CONFIG_USB_MASS_STORAGE=m
-CONFIG_USB_G_SERIAL=m
-CONFIG_USB_CDC_COMPOSITE=m
-CONFIG_MMC=y
-CONFIG_MMC_TEST=m
-CONFIG_MMC_ATMELMCI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_GPIO=y
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_TIMER=y
-CONFIG_LEDS_TRIGGER_HEARTBEAT=y
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_AT32AP700X=y
-CONFIG_DMADEVICES=y
-CONFIG_EXT2_FS=y
-CONFIG_EXT3_FS=y
-# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
-# CONFIG_EXT3_FS_XATTR is not set
-CONFIG_EXT4_FS=y
-# CONFIG_DNOTIFY is not set
-CONFIG_FUSE_FS=m
-CONFIG_MSDOS_FS=m
-CONFIG_VFAT_FS=m
-CONFIG_FAT_DEFAULT_CODEPAGE=850
-CONFIG_PROC_KCORE=y
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_UBIFS_FS=y
-CONFIG_NFS_FS=y
-CONFIG_ROOT_NFS=y
-CONFIG_NFSD=m
-CONFIG_NFSD_V3=y
-CONFIG_CIFS=m
-CONFIG_NLS_CODEPAGE_437=m
-CONFIG_NLS_CODEPAGE_850=m
-CONFIG_NLS_ISO8859_1=m
-CONFIG_NLS_UTF8=m
-CONFIG_DEBUG_FS=y
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-CONFIG_DETECT_HUNG_TASK=y
diff --git a/arch/avr32/configs/atngw100_mrmt_defconfig b/arch/avr32/configs/atngw100_mrmt_defconfig
deleted file mode 100644
index ffcc28d..0000000
--- a/arch/avr32/configs/atngw100_mrmt_defconfig
+++ /dev/null
@@ -1,136 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_BSD_PROCESS_ACCT=y
-CONFIG_BSD_PROCESS_ACCT_V3=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_SLUB_DEBUG is not set
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-CONFIG_MODULE_FORCE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-# CONFIG_OWNERSHIP_TRACE is not set
-# CONFIG_SUSPEND is not set
-CONFIG_PM=y
-CONFIG_CPU_FREQ=y
-CONFIG_CPU_FREQ_GOV_POWERSAVE=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_CPU_FREQ_GOV_ONDEMAND=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_INET=y
-CONFIG_IP_PNP=y
-CONFIG_IP_PNP_DHCP=y
-CONFIG_SYN_COOKIES=y
-# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
-# CONFIG_INET_XFRM_MODE_TUNNEL is not set
-# CONFIG_INET_XFRM_MODE_BEET is not set
-# CONFIG_INET_LRO is not set
-# CONFIG_IPV6 is not set
-CONFIG_BT=m
-CONFIG_BT_RFCOMM=m
-CONFIG_BT_RFCOMM_TTY=y
-CONFIG_BT_HIDP=m
-CONFIG_BT_HCIUART=m
-CONFIG_BT_HCIUART_H4=y
-CONFIG_BT_HCIUART_BCSP=y
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_AMDSTD=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_DATAFLASH=y
-CONFIG_BLK_DEV_LOOP=y
-CONFIG_NETDEVICES=y
-CONFIG_MACB=y
-# CONFIG_INPUT_MOUSEDEV is not set
-CONFIG_INPUT_EVDEV=y
-# CONFIG_KEYBOARD_ATKBD is not set
-CONFIG_KEYBOARD_GPIO=y
-# CONFIG_INPUT_MOUSE is not set
-CONFIG_INPUT_TOUCHSCREEN=y
-CONFIG_TOUCHSCREEN_ADS7846=m
-# CONFIG_SERIO is not set
-CONFIG_VT_HW_CONSOLE_BINDING=y
-# CONFIG_LEGACY_PTYS is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=y
-CONFIG_I2C_CHARDEV=y
-CONFIG_I2C_GPIO=y
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=y
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_FB=y
-CONFIG_FB_ATMEL=y
-CONFIG_LCD_CLASS_DEVICE=y
-CONFIG_SOUND=m
-CONFIG_SND=m
-CONFIG_SND_MIXER_OSS=m
-CONFIG_SND_PCM_OSS=m
-# CONFIG_SND_SUPPORT_OLD_API is not set
-# CONFIG_SND_VERBOSE_PROCFS is not set
-CONFIG_SND_ATMEL_AC97C=m
-# CONFIG_SND_SPI is not set
-CONFIG_USB_GADGET=m
-CONFIG_USB_GADGET_DEBUG_FILES=y
-CONFIG_USB_MASS_STORAGE=m
-CONFIG_USB_G_SERIAL=m
-CONFIG_MMC=y
-CONFIG_MMC_ATMELMCI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_GPIO=y
-CONFIG_LEDS_PWM=y
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_TIMER=y
-CONFIG_LEDS_TRIGGER_HEARTBEAT=y
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_S35390A=m
-CONFIG_RTC_DRV_AT32AP700X=m
-CONFIG_DMADEVICES=y
-CONFIG_UIO=y
-CONFIG_PWM=y
-CONFIG_PWM_ATMEL=y
-CONFIG_EXT2_FS=y
-CONFIG_EXT2_FS_XATTR=y
-CONFIG_EXT3_FS=y
-# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
-# CONFIG_DNOTIFY is not set
-CONFIG_MSDOS_FS=y
-CONFIG_VFAT_FS=y
-CONFIG_FAT_DEFAULT_CODEPAGE=850
-CONFIG_NTFS_FS=m
-CONFIG_NTFS_RW=y
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_NFS_FS=y
-CONFIG_ROOT_NFS=y
-CONFIG_CIFS=m
-CONFIG_CIFS_STATS=y
-CONFIG_CIFS_WEAK_PW_HASH=y
-CONFIG_CIFS_XATTR=y
-CONFIG_CIFS_POSIX=y
-CONFIG_NLS_CODEPAGE_437=y
-CONFIG_NLS_CODEPAGE_850=y
-CONFIG_NLS_ISO8859_1=y
-CONFIG_NLS_UTF8=y
-CONFIG_DEBUG_FS=y
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-CONFIG_DETECT_HUNG_TASK=y
-CONFIG_CRC_CCITT=y
diff --git a/arch/avr32/configs/atngw100mkii_defconfig b/arch/avr32/configs/atngw100mkii_defconfig
deleted file mode 100644
index 0496264..0000000
--- a/arch/avr32/configs/atngw100mkii_defconfig
+++ /dev/null
@@ -1,144 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_RELAY=y
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_COMPAT_BRK is not set
-CONFIG_PROFILING=y
-CONFIG_OPROFILE=m
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-CONFIG_BOARD_ATNGW100_MKII=y
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_NMI_DEBUGGING=y
-CONFIG_CPU_FREQ=y
-# CONFIG_CPU_FREQ_STAT is not set
-CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_XFRM_USER=y
-CONFIG_NET_KEY=y
-CONFIG_INET=y
-CONFIG_IP_MULTICAST=y
-CONFIG_IP_ADVANCED_ROUTER=y
-CONFIG_IP_PNP=y
-CONFIG_IP_PNP_DHCP=y
-CONFIG_IP_MROUTE=y
-CONFIG_IP_PIMSM_V1=y
-CONFIG_SYN_COOKIES=y
-CONFIG_INET_AH=y
-CONFIG_INET_ESP=y
-CONFIG_INET_IPCOMP=y
-# CONFIG_INET_LRO is not set
-CONFIG_IPV6=y
-CONFIG_INET6_AH=y
-CONFIG_INET6_ESP=y
-CONFIG_INET6_IPCOMP=y
-CONFIG_NETFILTER=y
-# CONFIG_NETFILTER_ADVANCED is not set
-CONFIG_NETFILTER_XTABLES=y
-CONFIG_BRIDGE=m
-CONFIG_VLAN_8021Q=m
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_INTELEXT=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_DATAFLASH=y
-CONFIG_MTD_NAND=y
-CONFIG_MTD_NAND_ATMEL=y
-CONFIG_MTD_UBI=y
-CONFIG_BLK_DEV_LOOP=m
-CONFIG_BLK_DEV_NBD=m
-CONFIG_BLK_DEV_RAM=m
-CONFIG_ATMEL_TCLIB=y
-CONFIG_NETDEVICES=y
-CONFIG_TUN=m
-CONFIG_MACB=y
-CONFIG_PPP=m
-CONFIG_PPP_BSDCOMP=m
-CONFIG_PPP_DEFLATE=m
-CONFIG_PPP_FILTER=y
-CONFIG_PPP_MPPE=m
-CONFIG_PPPOE=m
-CONFIG_PPP_ASYNC=m
-# CONFIG_INPUT is not set
-# CONFIG_SERIO is not set
-# CONFIG_VT is not set
-# CONFIG_LEGACY_PTYS is not set
-# CONFIG_DEVKMEM is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=m
-CONFIG_I2C_CHARDEV=m
-CONFIG_I2C_GPIO=m
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=m
-CONFIG_GPIO_SYSFS=y
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_USB_GADGET=y
-CONFIG_USB_GADGET_VBUS_DRAW=350
-CONFIG_USB_ZERO=m
-CONFIG_USB_ETH=m
-CONFIG_USB_GADGETFS=m
-CONFIG_USB_MASS_STORAGE=m
-CONFIG_USB_G_SERIAL=m
-CONFIG_USB_CDC_COMPOSITE=m
-CONFIG_MMC=y
-CONFIG_MMC_TEST=m
-CONFIG_MMC_ATMELMCI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_GPIO=y
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_TIMER=y
-CONFIG_LEDS_TRIGGER_HEARTBEAT=y
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_AT32AP700X=y
-CONFIG_DMADEVICES=y
-CONFIG_EXT2_FS=y
-CONFIG_EXT3_FS=y
-# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
-# CONFIG_EXT3_FS_XATTR is not set
-CONFIG_EXT4_FS=y
-# CONFIG_DNOTIFY is not set
-CONFIG_FUSE_FS=m
-CONFIG_MSDOS_FS=m
-CONFIG_VFAT_FS=m
-CONFIG_FAT_DEFAULT_CODEPAGE=850
-CONFIG_PROC_KCORE=y
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_UBIFS_FS=y
-CONFIG_NFS_FS=y
-CONFIG_ROOT_NFS=y
-CONFIG_NFSD=m
-CONFIG_NFSD_V3=y
-CONFIG_CIFS=m
-CONFIG_NLS_CODEPAGE_437=m
-CONFIG_NLS_CODEPAGE_850=m
-CONFIG_NLS_ISO8859_1=m
-CONFIG_NLS_UTF8=m
-CONFIG_DEBUG_FS=y
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-CONFIG_DETECT_HUNG_TASK=y
diff --git a/arch/avr32/configs/atngw100mkii_evklcd100_defconfig b/arch/avr32/configs/atngw100mkii_evklcd100_defconfig
deleted file mode 100644
index 89c2cda..0000000
--- a/arch/avr32/configs/atngw100mkii_evklcd100_defconfig
+++ /dev/null
@@ -1,161 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_RELAY=y
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_COMPAT_BRK is not set
-CONFIG_PROFILING=y
-CONFIG_OPROFILE=m
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-CONFIG_BOARD_ATNGW100_MKII=y
-CONFIG_BOARD_ATNGW100_MKII_LCD=y
-CONFIG_BOARD_ATNGW100_EVKLCD10X=y
-CONFIG_BOARD_ATNGW100_EVKLCD10X_QVGA=y
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_NMI_DEBUGGING=y
-CONFIG_CPU_FREQ=y
-# CONFIG_CPU_FREQ_STAT is not set
-CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_XFRM_USER=y
-CONFIG_NET_KEY=y
-CONFIG_INET=y
-CONFIG_IP_MULTICAST=y
-CONFIG_IP_ADVANCED_ROUTER=y
-CONFIG_IP_PNP=y
-CONFIG_IP_PNP_DHCP=y
-CONFIG_IP_MROUTE=y
-CONFIG_IP_PIMSM_V1=y
-CONFIG_SYN_COOKIES=y
-CONFIG_INET_AH=y
-CONFIG_INET_ESP=y
-CONFIG_INET_IPCOMP=y
-# CONFIG_INET_LRO is not set
-CONFIG_IPV6=y
-CONFIG_INET6_AH=y
-CONFIG_INET6_ESP=y
-CONFIG_INET6_IPCOMP=y
-CONFIG_NETFILTER=y
-# CONFIG_NETFILTER_ADVANCED is not set
-CONFIG_NETFILTER_XTABLES=y
-CONFIG_BRIDGE=m
-CONFIG_VLAN_8021Q=m
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_INTELEXT=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_DATAFLASH=y
-CONFIG_MTD_NAND=y
-CONFIG_MTD_NAND_ATMEL=y
-CONFIG_MTD_UBI=y
-CONFIG_BLK_DEV_LOOP=m
-CONFIG_BLK_DEV_NBD=m
-CONFIG_BLK_DEV_RAM=m
-CONFIG_ATMEL_TCLIB=y
-CONFIG_NETDEVICES=y
-CONFIG_TUN=m
-CONFIG_MACB=y
-CONFIG_PPP=m
-CONFIG_PPP_BSDCOMP=m
-CONFIG_PPP_DEFLATE=m
-CONFIG_PPP_FILTER=y
-CONFIG_PPP_MPPE=m
-CONFIG_PPPOE=m
-CONFIG_PPP_ASYNC=m
-# CONFIG_INPUT_MOUSEDEV is not set
-CONFIG_INPUT_EVDEV=m
-# CONFIG_INPUT_KEYBOARD is not set
-# CONFIG_INPUT_MOUSE is not set
-CONFIG_INPUT_TOUCHSCREEN=y
-CONFIG_TOUCHSCREEN_WM97XX=m
-# CONFIG_SERIO is not set
-# CONFIG_LEGACY_PTYS is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=m
-CONFIG_I2C_CHARDEV=m
-CONFIG_I2C_GPIO=m
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=m
-CONFIG_GPIO_SYSFS=y
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_FB=y
-CONFIG_FB_ATMEL=y
-CONFIG_SOUND=y
-CONFIG_SND=y
-CONFIG_SND_MIXER_OSS=m
-CONFIG_SND_PCM_OSS=m
-CONFIG_SND_HRTIMER=y
-# CONFIG_SND_SUPPORT_OLD_API is not set
-# CONFIG_SND_DRIVERS is not set
-CONFIG_SND_ATMEL_AC97C=m
-# CONFIG_SND_SPI is not set
-CONFIG_USB_GADGET=y
-CONFIG_USB_GADGET_VBUS_DRAW=350
-CONFIG_USB_ZERO=m
-CONFIG_USB_ETH=m
-CONFIG_USB_GADGETFS=m
-CONFIG_USB_MASS_STORAGE=m
-CONFIG_USB_G_SERIAL=m
-CONFIG_USB_CDC_COMPOSITE=m
-CONFIG_MMC=y
-CONFIG_MMC_TEST=m
-CONFIG_MMC_ATMELMCI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_GPIO=y
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_TIMER=y
-CONFIG_LEDS_TRIGGER_HEARTBEAT=y
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_AT32AP700X=y
-CONFIG_DMADEVICES=y
-CONFIG_EXT2_FS=y
-CONFIG_EXT3_FS=y
-# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
-# CONFIG_EXT3_FS_XATTR is not set
-CONFIG_EXT4_FS=y
-# CONFIG_DNOTIFY is not set
-CONFIG_FUSE_FS=m
-CONFIG_MSDOS_FS=m
-CONFIG_VFAT_FS=m
-CONFIG_FAT_DEFAULT_CODEPAGE=850
-CONFIG_PROC_KCORE=y
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_UBIFS_FS=y
-CONFIG_NFS_FS=y
-CONFIG_ROOT_NFS=y
-CONFIG_NFSD=m
-CONFIG_NFSD_V3=y
-CONFIG_CIFS=m
-CONFIG_NLS_CODEPAGE_437=m
-CONFIG_NLS_CODEPAGE_850=m
-CONFIG_NLS_ISO8859_1=m
-CONFIG_NLS_UTF8=m
-CONFIG_DEBUG_FS=y
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-CONFIG_DETECT_HUNG_TASK=y
diff --git a/arch/avr32/configs/atngw100mkii_evklcd101_defconfig b/arch/avr32/configs/atngw100mkii_evklcd101_defconfig
deleted file mode 100644
index 1b4d4a8..0000000
--- a/arch/avr32/configs/atngw100mkii_evklcd101_defconfig
+++ /dev/null
@@ -1,160 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_RELAY=y
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_COMPAT_BRK is not set
-CONFIG_PROFILING=y
-CONFIG_OPROFILE=m
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-CONFIG_BOARD_ATNGW100_MKII=y
-CONFIG_BOARD_ATNGW100_MKII_LCD=y
-CONFIG_BOARD_ATNGW100_EVKLCD10X=y
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_NMI_DEBUGGING=y
-CONFIG_CPU_FREQ=y
-# CONFIG_CPU_FREQ_STAT is not set
-CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_XFRM_USER=y
-CONFIG_NET_KEY=y
-CONFIG_INET=y
-CONFIG_IP_MULTICAST=y
-CONFIG_IP_ADVANCED_ROUTER=y
-CONFIG_IP_PNP=y
-CONFIG_IP_PNP_DHCP=y
-CONFIG_IP_MROUTE=y
-CONFIG_IP_PIMSM_V1=y
-CONFIG_SYN_COOKIES=y
-CONFIG_INET_AH=y
-CONFIG_INET_ESP=y
-CONFIG_INET_IPCOMP=y
-# CONFIG_INET_LRO is not set
-CONFIG_IPV6=y
-CONFIG_INET6_AH=y
-CONFIG_INET6_ESP=y
-CONFIG_INET6_IPCOMP=y
-CONFIG_NETFILTER=y
-# CONFIG_NETFILTER_ADVANCED is not set
-CONFIG_NETFILTER_XTABLES=y
-CONFIG_BRIDGE=m
-CONFIG_VLAN_8021Q=m
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_INTELEXT=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_DATAFLASH=y
-CONFIG_MTD_NAND=y
-CONFIG_MTD_NAND_ATMEL=y
-CONFIG_MTD_UBI=y
-CONFIG_BLK_DEV_LOOP=m
-CONFIG_BLK_DEV_NBD=m
-CONFIG_BLK_DEV_RAM=m
-CONFIG_ATMEL_TCLIB=y
-CONFIG_NETDEVICES=y
-CONFIG_TUN=m
-CONFIG_MACB=y
-CONFIG_PPP=m
-CONFIG_PPP_BSDCOMP=m
-CONFIG_PPP_DEFLATE=m
-CONFIG_PPP_FILTER=y
-CONFIG_PPP_MPPE=m
-CONFIG_PPPOE=m
-CONFIG_PPP_ASYNC=m
-# CONFIG_INPUT_MOUSEDEV is not set
-CONFIG_INPUT_EVDEV=m
-# CONFIG_INPUT_KEYBOARD is not set
-# CONFIG_INPUT_MOUSE is not set
-CONFIG_INPUT_TOUCHSCREEN=y
-CONFIG_TOUCHSCREEN_WM97XX=m
-# CONFIG_SERIO is not set
-# CONFIG_LEGACY_PTYS is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=m
-CONFIG_I2C_CHARDEV=m
-CONFIG_I2C_GPIO=m
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=m
-CONFIG_GPIO_SYSFS=y
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_FB=y
-CONFIG_FB_ATMEL=y
-CONFIG_SOUND=y
-CONFIG_SND=y
-CONFIG_SND_MIXER_OSS=m
-CONFIG_SND_PCM_OSS=m
-CONFIG_SND_HRTIMER=y
-# CONFIG_SND_SUPPORT_OLD_API is not set
-# CONFIG_SND_DRIVERS is not set
-CONFIG_SND_ATMEL_AC97C=m
-# CONFIG_SND_SPI is not set
-CONFIG_USB_GADGET=y
-CONFIG_USB_GADGET_VBUS_DRAW=350
-CONFIG_USB_ZERO=m
-CONFIG_USB_ETH=m
-CONFIG_USB_GADGETFS=m
-CONFIG_USB_MASS_STORAGE=m
-CONFIG_USB_G_SERIAL=m
-CONFIG_USB_CDC_COMPOSITE=m
-CONFIG_MMC=y
-CONFIG_MMC_TEST=m
-CONFIG_MMC_ATMELMCI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_GPIO=y
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_TIMER=y
-CONFIG_LEDS_TRIGGER_HEARTBEAT=y
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_AT32AP700X=y
-CONFIG_DMADEVICES=y
-CONFIG_EXT2_FS=y
-CONFIG_EXT3_FS=y
-# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
-# CONFIG_EXT3_FS_XATTR is not set
-CONFIG_EXT4_FS=y
-# CONFIG_DNOTIFY is not set
-CONFIG_FUSE_FS=m
-CONFIG_MSDOS_FS=m
-CONFIG_VFAT_FS=m
-CONFIG_FAT_DEFAULT_CODEPAGE=850
-CONFIG_PROC_KCORE=y
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_UBIFS_FS=y
-CONFIG_NFS_FS=y
-CONFIG_ROOT_NFS=y
-CONFIG_NFSD=m
-CONFIG_NFSD_V3=y
-CONFIG_CIFS=m
-CONFIG_NLS_CODEPAGE_437=m
-CONFIG_NLS_CODEPAGE_850=m
-CONFIG_NLS_ISO8859_1=m
-CONFIG_NLS_UTF8=m
-CONFIG_DEBUG_FS=y
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-CONFIG_DETECT_HUNG_TASK=y
diff --git a/arch/avr32/configs/atstk1002_defconfig b/arch/avr32/configs/atstk1002_defconfig
deleted file mode 100644
index 9b8b52e..0000000
--- a/arch/avr32/configs/atstk1002_defconfig
+++ /dev/null
@@ -1,157 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_RELAY=y
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_COMPAT_BRK is not set
-CONFIG_PROFILING=y
-CONFIG_OPROFILE=m
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_NMI_DEBUGGING=y
-CONFIG_CPU_FREQ=y
-# CONFIG_CPU_FREQ_STAT is not set
-CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_XFRM_USER=m
-CONFIG_NET_KEY=m
-CONFIG_INET=y
-CONFIG_IP_PNP=y
-CONFIG_IP_PNP_DHCP=y
-CONFIG_NET_IPIP=m
-CONFIG_NET_IPGRE_DEMUX=m
-CONFIG_NET_IPGRE=m
-CONFIG_INET_AH=m
-CONFIG_INET_ESP=m
-CONFIG_INET_XFRM_MODE_TRANSPORT=m
-CONFIG_INET_XFRM_MODE_TUNNEL=m
-CONFIG_INET_XFRM_MODE_BEET=m
-# CONFIG_INET_LRO is not set
-CONFIG_INET6_AH=m
-CONFIG_INET6_ESP=m
-CONFIG_INET6_IPCOMP=m
-CONFIG_IPV6_TUNNEL=m
-CONFIG_BRIDGE=m
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_AMDSTD=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_UBI=y
-CONFIG_BLK_DEV_LOOP=m
-CONFIG_BLK_DEV_NBD=m
-CONFIG_BLK_DEV_RAM=m
-CONFIG_ATMEL_TCLIB=y
-CONFIG_ATMEL_SSC=m
-# CONFIG_SCSI_PROC_FS is not set
-CONFIG_BLK_DEV_SD=m
-CONFIG_BLK_DEV_SR=m
-# CONFIG_SCSI_LOWLEVEL is not set
-CONFIG_ATA=m
-# CONFIG_SATA_PMP is not set
-CONFIG_PATA_AT32=m
-CONFIG_NETDEVICES=y
-CONFIG_TUN=m
-CONFIG_MACB=y
-CONFIG_PPP=m
-CONFIG_PPP_BSDCOMP=m
-CONFIG_PPP_DEFLATE=m
-CONFIG_PPP_ASYNC=m
-CONFIG_INPUT=m
-CONFIG_INPUT_EVDEV=m
-# CONFIG_KEYBOARD_ATKBD is not set
-CONFIG_KEYBOARD_GPIO=m
-# CONFIG_MOUSE_PS2 is not set
-CONFIG_MOUSE_GPIO=m
-# CONFIG_SERIO is not set
-# CONFIG_VT is not set
-# CONFIG_LEGACY_PTYS is not set
-# CONFIG_DEVKMEM is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=m
-CONFIG_I2C_CHARDEV=m
-CONFIG_I2C_GPIO=m
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=m
-CONFIG_GPIO_SYSFS=y
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_FB=y
-CONFIG_FB_ATMEL=y
-CONFIG_LCD_CLASS_DEVICE=y
-CONFIG_LCD_LTV350QV=y
-CONFIG_SOUND=m
-CONFIG_SND=m
-CONFIG_SND_MIXER_OSS=m
-CONFIG_SND_PCM_OSS=m
-# CONFIG_SND_SUPPORT_OLD_API is not set
-# CONFIG_SND_VERBOSE_PROCFS is not set
-CONFIG_SND_AT73C213=m
-CONFIG_USB_GADGET=y
-CONFIG_USB_ZERO=m
-CONFIG_USB_ETH=m
-CONFIG_USB_GADGETFS=m
-CONFIG_USB_MASS_STORAGE=m
-CONFIG_USB_G_SERIAL=m
-CONFIG_USB_CDC_COMPOSITE=m
-CONFIG_MMC=y
-CONFIG_MMC_TEST=m
-CONFIG_MMC_ATMELMCI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_GPIO=m
-CONFIG_LEDS_PWM=m
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_TIMER=m
-CONFIG_LEDS_TRIGGER_HEARTBEAT=m
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_AT32AP700X=y
-CONFIG_DMADEVICES=y
-CONFIG_PWM=y
-CONFIG_PWM_ATMEL=m
-CONFIG_EXT2_FS=y
-CONFIG_EXT3_FS=y
-# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
-# CONFIG_EXT3_FS_XATTR is not set
-CONFIG_EXT4_FS=y
-# CONFIG_DNOTIFY is not set
-CONFIG_FUSE_FS=m
-CONFIG_MSDOS_FS=m
-CONFIG_VFAT_FS=m
-CONFIG_FAT_DEFAULT_CODEPAGE=850
-CONFIG_PROC_KCORE=y
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_UBIFS_FS=y
-CONFIG_NFS_FS=y
-CONFIG_ROOT_NFS=y
-CONFIG_CIFS=m
-CONFIG_NLS_CODEPAGE_437=m
-CONFIG_NLS_CODEPAGE_850=m
-CONFIG_NLS_ISO8859_1=m
-CONFIG_NLS_UTF8=m
-CONFIG_DEBUG_FS=y
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-CONFIG_DETECT_HUNG_TASK=y
diff --git a/arch/avr32/configs/atstk1003_defconfig b/arch/avr32/configs/atstk1003_defconfig
deleted file mode 100644
index ccce1a0..0000000
--- a/arch/avr32/configs/atstk1003_defconfig
+++ /dev/null
@@ -1,137 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_RELAY=y
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_COMPAT_BRK is not set
-CONFIG_PROFILING=y
-CONFIG_OPROFILE=m
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-CONFIG_BOARD_ATSTK1003=y
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_NMI_DEBUGGING=y
-CONFIG_CPU_FREQ=y
-# CONFIG_CPU_FREQ_STAT is not set
-CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_INET=y
-# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
-# CONFIG_INET_XFRM_MODE_TUNNEL is not set
-# CONFIG_INET_XFRM_MODE_BEET is not set
-# CONFIG_INET_LRO is not set
-# CONFIG_INET_DIAG is not set
-# CONFIG_IPV6 is not set
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_AMDSTD=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_UBI=y
-CONFIG_BLK_DEV_LOOP=m
-CONFIG_BLK_DEV_NBD=m
-CONFIG_BLK_DEV_RAM=m
-CONFIG_ATMEL_TCLIB=y
-CONFIG_ATMEL_SSC=m
-# CONFIG_SCSI_PROC_FS is not set
-CONFIG_BLK_DEV_SD=m
-CONFIG_BLK_DEV_SR=m
-# CONFIG_SCSI_LOWLEVEL is not set
-CONFIG_ATA=m
-# CONFIG_SATA_PMP is not set
-CONFIG_PATA_AT32=m
-CONFIG_NETDEVICES=y
-CONFIG_PPP=m
-CONFIG_PPP_BSDCOMP=m
-CONFIG_PPP_DEFLATE=m
-CONFIG_PPP_ASYNC=m
-CONFIG_INPUT=m
-CONFIG_INPUT_EVDEV=m
-# CONFIG_KEYBOARD_ATKBD is not set
-CONFIG_KEYBOARD_GPIO=m
-# CONFIG_MOUSE_PS2 is not set
-CONFIG_MOUSE_GPIO=m
-# CONFIG_SERIO is not set
-# CONFIG_VT is not set
-# CONFIG_LEGACY_PTYS is not set
-# CONFIG_DEVKMEM is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=m
-CONFIG_I2C_CHARDEV=m
-CONFIG_I2C_GPIO=m
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=m
-CONFIG_GPIO_SYSFS=y
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_SOUND=m
-CONFIG_SND=m
-CONFIG_SND_MIXER_OSS=m
-CONFIG_SND_PCM_OSS=m
-# CONFIG_SND_DRIVERS is not set
-CONFIG_SND_AT73C213=m
-CONFIG_USB_GADGET=y
-CONFIG_USB_ZERO=m
-CONFIG_USB_ETH=m
-CONFIG_USB_GADGETFS=m
-CONFIG_USB_MASS_STORAGE=m
-CONFIG_USB_G_SERIAL=m
-CONFIG_USB_CDC_COMPOSITE=m
-CONFIG_MMC=y
-CONFIG_MMC_TEST=m
-CONFIG_MMC_ATMELMCI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_GPIO=m
-CONFIG_LEDS_PWM=m
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_TIMER=m
-CONFIG_LEDS_TRIGGER_HEARTBEAT=m
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_AT32AP700X=y
-CONFIG_DMADEVICES=y
-CONFIG_PWM=y
-CONFIG_PWM_ATMEL=m
-CONFIG_EXT2_FS=y
-CONFIG_EXT3_FS=y
-# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
-# CONFIG_EXT3_FS_XATTR is not set
-CONFIG_EXT4_FS=y
-# CONFIG_DNOTIFY is not set
-CONFIG_FUSE_FS=m
-CONFIG_MSDOS_FS=m
-CONFIG_VFAT_FS=m
-CONFIG_FAT_DEFAULT_CODEPAGE=850
-CONFIG_PROC_KCORE=y
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_UBIFS_FS=y
-# CONFIG_NETWORK_FILESYSTEMS is not set
-CONFIG_NLS_CODEPAGE_437=m
-CONFIG_NLS_CODEPAGE_850=m
-CONFIG_NLS_ISO8859_1=m
-CONFIG_NLS_UTF8=m
-CONFIG_DEBUG_FS=y
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-CONFIG_DETECT_HUNG_TASK=y
diff --git a/arch/avr32/configs/atstk1004_defconfig b/arch/avr32/configs/atstk1004_defconfig
deleted file mode 100644
index e64288f..0000000
--- a/arch/avr32/configs/atstk1004_defconfig
+++ /dev/null
@@ -1,135 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_RELAY=y
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_COMPAT_BRK is not set
-CONFIG_PROFILING=y
-CONFIG_OPROFILE=m
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-CONFIG_BOARD_ATSTK1004=y
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_NMI_DEBUGGING=y
-CONFIG_CPU_FREQ=y
-# CONFIG_CPU_FREQ_STAT is not set
-CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_INET=y
-# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
-# CONFIG_INET_XFRM_MODE_TUNNEL is not set
-# CONFIG_INET_XFRM_MODE_BEET is not set
-# CONFIG_INET_LRO is not set
-# CONFIG_INET_DIAG is not set
-# CONFIG_IPV6 is not set
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_AMDSTD=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_UBI=y
-CONFIG_BLK_DEV_LOOP=m
-CONFIG_BLK_DEV_NBD=m
-CONFIG_BLK_DEV_RAM=m
-CONFIG_ATMEL_TCLIB=y
-CONFIG_ATMEL_SSC=m
-# CONFIG_SCSI_PROC_FS is not set
-CONFIG_BLK_DEV_SD=m
-CONFIG_BLK_DEV_SR=m
-# CONFIG_SCSI_LOWLEVEL is not set
-CONFIG_ATA=m
-# CONFIG_SATA_PMP is not set
-CONFIG_PATA_AT32=m
-CONFIG_NETDEVICES=y
-CONFIG_PPP=m
-CONFIG_PPP_BSDCOMP=m
-CONFIG_PPP_DEFLATE=m
-CONFIG_PPP_ASYNC=m
-CONFIG_INPUT=m
-CONFIG_INPUT_EVDEV=m
-# CONFIG_KEYBOARD_ATKBD is not set
-CONFIG_KEYBOARD_GPIO=m
-# CONFIG_MOUSE_PS2 is not set
-CONFIG_MOUSE_GPIO=m
-# CONFIG_SERIO is not set
-# CONFIG_VT is not set
-# CONFIG_LEGACY_PTYS is not set
-# CONFIG_DEVKMEM is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=m
-CONFIG_I2C_CHARDEV=m
-CONFIG_I2C_GPIO=m
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=m
-CONFIG_GPIO_SYSFS=y
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_FB=y
-CONFIG_FB_ATMEL=y
-CONFIG_LCD_CLASS_DEVICE=y
-CONFIG_LCD_LTV350QV=y
-CONFIG_USB_GADGET=y
-CONFIG_USB_ZERO=m
-CONFIG_USB_ETH=m
-CONFIG_USB_GADGETFS=m
-CONFIG_USB_MASS_STORAGE=m
-CONFIG_USB_G_SERIAL=m
-CONFIG_USB_CDC_COMPOSITE=m
-CONFIG_MMC=y
-CONFIG_MMC_TEST=m
-CONFIG_MMC_ATMELMCI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_GPIO=m
-CONFIG_LEDS_PWM=m
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_TIMER=m
-CONFIG_LEDS_TRIGGER_HEARTBEAT=m
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_AT32AP700X=y
-CONFIG_DMADEVICES=y
-CONFIG_PWM=y
-CONFIG_PWM_ATMEL=m
-CONFIG_EXT2_FS=y
-CONFIG_EXT3_FS=y
-# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
-# CONFIG_EXT3_FS_XATTR is not set
-CONFIG_EXT4_FS=y
-# CONFIG_DNOTIFY is not set
-CONFIG_FUSE_FS=m
-CONFIG_MSDOS_FS=m
-CONFIG_VFAT_FS=m
-CONFIG_FAT_DEFAULT_CODEPAGE=850
-CONFIG_PROC_KCORE=y
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_UBIFS_FS=y
-# CONFIG_NETWORK_FILESYSTEMS is not set
-CONFIG_NLS_CODEPAGE_437=m
-CONFIG_NLS_CODEPAGE_850=m
-CONFIG_NLS_ISO8859_1=m
-CONFIG_NLS_UTF8=m
-CONFIG_DEBUG_FS=y
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-CONFIG_DETECT_HUNG_TASK=y
diff --git a/arch/avr32/configs/atstk1006_defconfig b/arch/avr32/configs/atstk1006_defconfig
deleted file mode 100644
index 7d669f7..0000000
--- a/arch/avr32/configs/atstk1006_defconfig
+++ /dev/null
@@ -1,160 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_RELAY=y
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_COMPAT_BRK is not set
-CONFIG_PROFILING=y
-CONFIG_OPROFILE=m
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-CONFIG_BOARD_ATSTK1006=y
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_NMI_DEBUGGING=y
-CONFIG_CPU_FREQ=y
-# CONFIG_CPU_FREQ_STAT is not set
-CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_XFRM_USER=m
-CONFIG_NET_KEY=m
-CONFIG_INET=y
-CONFIG_IP_PNP=y
-CONFIG_IP_PNP_DHCP=y
-CONFIG_NET_IPIP=m
-CONFIG_NET_IPGRE_DEMUX=m
-CONFIG_NET_IPGRE=m
-CONFIG_INET_AH=m
-CONFIG_INET_ESP=m
-CONFIG_INET_XFRM_MODE_TRANSPORT=m
-CONFIG_INET_XFRM_MODE_TUNNEL=m
-CONFIG_INET_XFRM_MODE_BEET=m
-# CONFIG_INET_LRO is not set
-CONFIG_INET6_AH=m
-CONFIG_INET6_ESP=m
-CONFIG_INET6_IPCOMP=m
-CONFIG_IPV6_TUNNEL=m
-CONFIG_BRIDGE=m
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_AMDSTD=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_NAND=y
-CONFIG_MTD_NAND_ATMEL=y
-CONFIG_MTD_UBI=y
-CONFIG_BLK_DEV_LOOP=m
-CONFIG_BLK_DEV_NBD=m
-CONFIG_BLK_DEV_RAM=m
-CONFIG_ATMEL_TCLIB=y
-CONFIG_ATMEL_SSC=m
-# CONFIG_SCSI_PROC_FS is not set
-CONFIG_BLK_DEV_SD=m
-CONFIG_BLK_DEV_SR=m
-# CONFIG_SCSI_LOWLEVEL is not set
-CONFIG_ATA=m
-# CONFIG_SATA_PMP is not set
-CONFIG_PATA_AT32=m
-CONFIG_NETDEVICES=y
-CONFIG_TUN=m
-CONFIG_MACB=y
-CONFIG_PPP=m
-CONFIG_PPP_BSDCOMP=m
-CONFIG_PPP_DEFLATE=m
-CONFIG_PPP_ASYNC=m
-CONFIG_INPUT=m
-CONFIG_INPUT_EVDEV=m
-# CONFIG_KEYBOARD_ATKBD is not set
-CONFIG_KEYBOARD_GPIO=m
-# CONFIG_MOUSE_PS2 is not set
-CONFIG_MOUSE_GPIO=m
-# CONFIG_SERIO is not set
-# CONFIG_VT is not set
-# CONFIG_LEGACY_PTYS is not set
-# CONFIG_DEVKMEM is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=m
-CONFIG_I2C_CHARDEV=m
-CONFIG_I2C_GPIO=m
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=m
-CONFIG_GPIO_SYSFS=y
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_FB=y
-CONFIG_FB_ATMEL=y
-CONFIG_LCD_CLASS_DEVICE=y
-CONFIG_LCD_LTV350QV=y
-CONFIG_SOUND=m
-CONFIG_SND=m
-CONFIG_SND_MIXER_OSS=m
-CONFIG_SND_PCM_OSS=m
-# CONFIG_SND_SUPPORT_OLD_API is not set
-# CONFIG_SND_VERBOSE_PROCFS is not set
-CONFIG_SND_AT73C213=m
-CONFIG_USB_GADGET=y
-CONFIG_USB_ZERO=m
-CONFIG_USB_ETH=m
-CONFIG_USB_GADGETFS=m
-CONFIG_USB_MASS_STORAGE=m
-CONFIG_USB_G_SERIAL=m
-CONFIG_USB_CDC_COMPOSITE=m
-CONFIG_MMC=y
-CONFIG_MMC_TEST=m
-CONFIG_MMC_ATMELMCI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_GPIO=m
-CONFIG_LEDS_PWM=m
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_TIMER=m
-CONFIG_LEDS_TRIGGER_HEARTBEAT=m
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_AT32AP700X=y
-CONFIG_DMADEVICES=y
-CONFIG_PWM=y
-CONFIG_PWM_ATMEL=m
-CONFIG_EXT2_FS=y
-CONFIG_EXT3_FS=y
-# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
-# CONFIG_EXT3_FS_XATTR is not set
-CONFIG_EXT4_FS=y
-# CONFIG_DNOTIFY is not set
-CONFIG_FUSE_FS=m
-CONFIG_MSDOS_FS=m
-CONFIG_VFAT_FS=m
-CONFIG_FAT_DEFAULT_CODEPAGE=850
-CONFIG_PROC_KCORE=y
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_UBIFS_FS=y
-CONFIG_NFS_FS=y
-CONFIG_ROOT_NFS=y
-CONFIG_CIFS=m
-CONFIG_NLS_CODEPAGE_437=m
-CONFIG_NLS_CODEPAGE_850=m
-CONFIG_NLS_ISO8859_1=m
-CONFIG_NLS_UTF8=m
-CONFIG_DEBUG_FS=y
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-CONFIG_DETECT_HUNG_TASK=y
diff --git a/arch/avr32/configs/favr-32_defconfig b/arch/avr32/configs/favr-32_defconfig
deleted file mode 100644
index 560c52f..0000000
--- a/arch/avr32/configs/favr-32_defconfig
+++ /dev/null
@@ -1,143 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_RELAY=y
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_COMPAT_BRK is not set
-CONFIG_PROFILING=y
-CONFIG_OPROFILE=m
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-CONFIG_BOARD_FAVR_32=y
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_NMI_DEBUGGING=y
-CONFIG_CPU_FREQ=y
-# CONFIG_CPU_FREQ_STAT is not set
-CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_XFRM_USER=m
-CONFIG_NET_KEY=m
-CONFIG_INET=y
-CONFIG_IP_PNP=y
-CONFIG_IP_PNP_DHCP=y
-CONFIG_NET_IPIP=m
-CONFIG_INET_AH=m
-CONFIG_INET_ESP=m
-CONFIG_INET_XFRM_MODE_TRANSPORT=m
-CONFIG_INET_XFRM_MODE_TUNNEL=m
-CONFIG_INET_XFRM_MODE_BEET=m
-# CONFIG_INET_LRO is not set
-CONFIG_IPV6=y
-CONFIG_INET6_AH=m
-CONFIG_INET6_ESP=m
-CONFIG_INET6_IPCOMP=m
-CONFIG_INET6_XFRM_MODE_TRANSPORT=m
-CONFIG_INET6_XFRM_MODE_TUNNEL=m
-CONFIG_INET6_XFRM_MODE_BEET=m
-CONFIG_IPV6_SIT=m
-CONFIG_IPV6_TUNNEL=m
-CONFIG_BRIDGE=m
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_AMDSTD=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_BLK_DEV_LOOP=m
-CONFIG_BLK_DEV_NBD=m
-CONFIG_BLK_DEV_RAM=m
-CONFIG_ATMEL_TCLIB=y
-CONFIG_ATMEL_SSC=m
-CONFIG_NETDEVICES=y
-CONFIG_MACB=y
-CONFIG_PPP=m
-CONFIG_PPP_BSDCOMP=m
-CONFIG_PPP_DEFLATE=m
-CONFIG_PPP_ASYNC=m
-CONFIG_INPUT_MOUSEDEV=m
-CONFIG_INPUT_EVDEV=m
-# CONFIG_KEYBOARD_ATKBD is not set
-CONFIG_KEYBOARD_GPIO=m
-# CONFIG_MOUSE_PS2 is not set
-CONFIG_MOUSE_GPIO=m
-CONFIG_INPUT_TOUCHSCREEN=y
-CONFIG_TOUCHSCREEN_ADS7846=m
-# CONFIG_SERIO is not set
-# CONFIG_CONSOLE_TRANSLATIONS is not set
-# CONFIG_LEGACY_PTYS is not set
-# CONFIG_DEVKMEM is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=m
-CONFIG_I2C_CHARDEV=m
-CONFIG_I2C_GPIO=m
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=m
-CONFIG_GPIO_SYSFS=y
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_FB=y
-CONFIG_FB_ATMEL=y
-# CONFIG_LCD_CLASS_DEVICE is not set
-CONFIG_BACKLIGHT_PWM=m
-CONFIG_SOUND=m
-CONFIG_SOUND_PRIME=m
-CONFIG_USB_GADGET=y
-CONFIG_USB_ZERO=m
-CONFIG_USB_ETH=m
-CONFIG_USB_GADGETFS=m
-CONFIG_USB_MASS_STORAGE=m
-CONFIG_USB_G_SERIAL=m
-CONFIG_USB_CDC_COMPOSITE=m
-CONFIG_MMC=y
-CONFIG_MMC_ATMELMCI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_GPIO=y
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_TIMER=y
-CONFIG_LEDS_TRIGGER_HEARTBEAT=y
-CONFIG_LEDS_TRIGGER_DEFAULT_ON=y
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_AT32AP700X=y
-CONFIG_DMADEVICES=y
-CONFIG_PWM=y
-CONFIG_PWM_ATMEL=y
-CONFIG_EXT2_FS=y
-CONFIG_EXT3_FS=y
-# CONFIG_EXT3_FS_XATTR is not set
-# CONFIG_DNOTIFY is not set
-CONFIG_FUSE_FS=m
-CONFIG_MSDOS_FS=m
-CONFIG_VFAT_FS=m
-CONFIG_PROC_KCORE=y
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-# CONFIG_JFFS2_FS_WRITEBUFFER is not set
-CONFIG_NFS_FS=y
-CONFIG_ROOT_NFS=y
-CONFIG_NLS_CODEPAGE_437=m
-CONFIG_NLS_ISO8859_1=m
-CONFIG_NLS_UTF8=m
-CONFIG_DEBUG_FS=y
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-# CONFIG_CRYPTO_HW is not set
diff --git a/arch/avr32/configs/hammerhead_defconfig b/arch/avr32/configs/hammerhead_defconfig
deleted file mode 100644
index d57fadb..0000000
--- a/arch/avr32/configs/hammerhead_defconfig
+++ /dev/null
@@ -1,145 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_BSD_PROCESS_ACCT=y
-CONFIG_BSD_PROCESS_ACCT_V3=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_COMPAT_BRK is not set
-CONFIG_PROFILING=y
-CONFIG_OPROFILE=m
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-CONFIG_MODULE_FORCE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-CONFIG_BOARD_HAMMERHEAD=y
-CONFIG_BOARD_HAMMERHEAD_USB=y
-CONFIG_BOARD_HAMMERHEAD_LCD=y
-CONFIG_BOARD_HAMMERHEAD_SND=y
-# CONFIG_BOARD_HAMMERHEAD_FPGA is not set
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_CPU_FREQ=y
-# CONFIG_CPU_FREQ_STAT is not set
-CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_XFRM_USER=y
-CONFIG_NET_KEY=y
-CONFIG_INET=y
-CONFIG_IP_MULTICAST=y
-CONFIG_IP_ADVANCED_ROUTER=y
-CONFIG_IP_PNP=y
-CONFIG_IP_PNP_DHCP=y
-CONFIG_IP_MROUTE=y
-CONFIG_IP_PIMSM_V1=y
-CONFIG_SYN_COOKIES=y
-CONFIG_INET_AH=y
-CONFIG_INET_ESP=y
-CONFIG_INET_IPCOMP=y
-# CONFIG_INET_LRO is not set
-# CONFIG_IPV6 is not set
-CONFIG_NETFILTER=y
-# CONFIG_NETFILTER_ADVANCED is not set
-CONFIG_NETFILTER_XTABLES=y
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_AMDSTD=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_DATAFLASH=y
-CONFIG_BLK_DEV_RAM=m
-CONFIG_ATMEL_TCLIB=y
-CONFIG_SCSI=m
-CONFIG_BLK_DEV_SD=m
-CONFIG_NETDEVICES=y
-CONFIG_MACB=y
-CONFIG_INPUT_FF_MEMLESS=m
-CONFIG_INPUT_EVDEV=m
-CONFIG_INPUT_TOUCHSCREEN=y
-# CONFIG_LEGACY_PTYS is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=m
-CONFIG_I2C_CHARDEV=m
-CONFIG_I2C_GPIO=m
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=m
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_FB=y
-CONFIG_FB_ATMEL=y
-CONFIG_FRAMEBUFFER_CONSOLE=y
-CONFIG_SOUND=m
-CONFIG_SND=m
-CONFIG_SND_SEQUENCER=m
-CONFIG_SND_MIXER_OSS=m
-CONFIG_SND_PCM_OSS=m
-CONFIG_SND_SEQUENCER_OSS=y
-# CONFIG_SND_SUPPORT_OLD_API is not set
-CONFIG_HID_A4TECH=m
-CONFIG_HID_APPLE=m
-CONFIG_HID_BELKIN=m
-CONFIG_HID_CHERRY=m
-CONFIG_HID_CHICONY=m
-CONFIG_HID_CYPRESS=m
-CONFIG_HID_EZKEY=m
-CONFIG_HID_GYRATION=m
-CONFIG_HID_LOGITECH=m
-CONFIG_HID_MICROSOFT=m
-CONFIG_HID_MONTEREY=m
-CONFIG_HID_PANTHERLORD=m
-CONFIG_HID_PETALYNX=m
-CONFIG_HID_SAMSUNG=m
-CONFIG_HID_SUNPLUS=m
-CONFIG_USB=m
-CONFIG_USB_MON=m
-CONFIG_USB_ISP116X_HCD=m
-CONFIG_USB_STORAGE=m
-CONFIG_USB_GADGET=y
-CONFIG_USB_ZERO=m
-CONFIG_USB_ETH=m
-CONFIG_USB_GADGETFS=m
-CONFIG_USB_MASS_STORAGE=m
-CONFIG_USB_G_SERIAL=m
-CONFIG_MMC=m
-CONFIG_MMC_ATMELMCI=m
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_AT32AP700X=y
-CONFIG_EXT2_FS=m
-# CONFIG_DNOTIFY is not set
-CONFIG_MSDOS_FS=y
-CONFIG_VFAT_FS=m
-CONFIG_FAT_DEFAULT_CODEPAGE=850
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_NFS_FS=y
-CONFIG_ROOT_NFS=y
-CONFIG_NLS_CODEPAGE_437=m
-CONFIG_NLS_CODEPAGE_850=m
-CONFIG_NLS_ISO8859_1=m
-CONFIG_NLS_UTF8=m
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-CONFIG_CRYPTO_ECB=m
-CONFIG_CRYPTO_PCBC=m
-CONFIG_CRYPTO_ARC4=m
-# CONFIG_CRYPTO_ANSI_CPRNG is not set
-CONFIG_CRC_CCITT=m
-CONFIG_CRC_ITU_T=m
-CONFIG_CRC7=m
diff --git a/arch/avr32/configs/merisc_defconfig b/arch/avr32/configs/merisc_defconfig
deleted file mode 100644
index e6a9cb7..0000000
--- a/arch/avr32/configs/merisc_defconfig
+++ /dev/null
@@ -1,115 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_BSD_PROCESS_ACCT=y
-CONFIG_BSD_PROCESS_ACCT_V3=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-CONFIG_MODULES=y
-CONFIG_MODULE_UNLOAD=y
-CONFIG_MODULE_FORCE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-CONFIG_BOARD_MERISC=y
-CONFIG_AP700X_32_BIT_SMC=y
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_XFRM_USER=y
-CONFIG_NET_KEY=y
-CONFIG_INET=y
-CONFIG_IP_MULTICAST=y
-CONFIG_IP_ADVANCED_ROUTER=y
-CONFIG_IP_PNP=y
-CONFIG_IP_PNP_DHCP=y
-CONFIG_IP_MROUTE=y
-CONFIG_IP_PIMSM_V1=y
-CONFIG_SYN_COOKIES=y
-CONFIG_INET_AH=y
-CONFIG_INET_ESP=y
-CONFIG_INET_IPCOMP=y
-# CONFIG_INET_LRO is not set
-# CONFIG_IPV6 is not set
-CONFIG_CAN=y
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_JEDECPROBE=y
-CONFIG_MTD_CFI_AMDSTD=y
-CONFIG_MTD_ABSENT=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_BLOCK2MTD=y
-CONFIG_BLK_DEV_LOOP=y
-CONFIG_ATMEL_SSC=y
-CONFIG_SCSI=y
-CONFIG_BLK_DEV_SD=y
-# CONFIG_SCSI_LOWLEVEL is not set
-CONFIG_NETDEVICES=y
-CONFIG_MACB=y
-# CONFIG_INPUT_MOUSEDEV is not set
-CONFIG_INPUT_EVDEV=y
-# CONFIG_KEYBOARD_ATKBD is not set
-# CONFIG_INPUT_MOUSE is not set
-CONFIG_INPUT_TOUCHSCREEN=y
-CONFIG_TOUCHSCREEN_ADS7846=y
-CONFIG_INPUT_MISC=y
-CONFIG_INPUT_UINPUT=y
-# CONFIG_SERIO is not set
-# CONFIG_CONSOLE_TRANSLATIONS is not set
-# CONFIG_LEGACY_PTYS is not set
-# CONFIG_DEVKMEM is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=y
-CONFIG_I2C_CHARDEV=y
-CONFIG_I2C_GPIO=y
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_SPI_SPIDEV=y
-CONFIG_GPIO_SYSFS=y
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_FB=y
-CONFIG_FB_ATMEL=y
-# CONFIG_LCD_CLASS_DEVICE is not set
-CONFIG_FRAMEBUFFER_CONSOLE=y
-CONFIG_LOGO=y
-CONFIG_MMC=y
-CONFIG_MMC_ATMELMCI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_PWM=y
-CONFIG_RTC_CLASS=y
-# CONFIG_RTC_HCTOSYS is not set
-CONFIG_RTC_DRV_PCF8563=y
-CONFIG_DMADEVICES=y
-CONFIG_UIO=y
-CONFIG_PWM=y
-CONFIG_PWM_ATMEL=m
-CONFIG_EXT2_FS=y
-# CONFIG_DNOTIFY is not set
-CONFIG_FUSE_FS=y
-CONFIG_MSDOS_FS=y
-CONFIG_VFAT_FS=y
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_JFFS2_FS_WBUF_VERIFY=y
-CONFIG_CRAMFS=y
-CONFIG_NFS_FS=y
-CONFIG_ROOT_NFS=y
-CONFIG_NLS_CODEPAGE_437=y
-CONFIG_NLS_CODEPAGE_850=y
-CONFIG_NLS_ISO8859_1=y
-CONFIG_NLS_UTF8=y
-# CONFIG_CRYPTO_ANSI_CPRNG is not set
-# CONFIG_CRYPTO_HW is not set
diff --git a/arch/avr32/configs/mimc200_defconfig b/arch/avr32/configs/mimc200_defconfig
deleted file mode 100644
index 49c7e89..0000000
--- a/arch/avr32/configs/mimc200_defconfig
+++ /dev/null
@@ -1,114 +0,0 @@
-# CONFIG_LOCALVERSION_AUTO is not set
-CONFIG_SYSVIPC=y
-CONFIG_POSIX_MQUEUE=y
-CONFIG_NO_HZ=y
-CONFIG_HIGH_RES_TIMERS=y
-CONFIG_BSD_PROCESS_ACCT=y
-CONFIG_BSD_PROCESS_ACCT_V3=y
-CONFIG_LOG_BUF_SHIFT=14
-CONFIG_BLK_DEV_INITRD=y
-CONFIG_CC_OPTIMIZE_FOR_SIZE=y
-# CONFIG_BASE_FULL is not set
-# CONFIG_COMPAT_BRK is not set
-CONFIG_PROFILING=y
-# CONFIG_BLK_DEV_BSG is not set
-# CONFIG_IOSCHED_DEADLINE is not set
-CONFIG_BOARD_MIMC200=y
-# CONFIG_OWNERSHIP_TRACE is not set
-CONFIG_NMI_DEBUGGING=y
-CONFIG_CPU_FREQ=y
-# CONFIG_CPU_FREQ_STAT is not set
-CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
-CONFIG_CPU_FREQ_GOV_USERSPACE=y
-CONFIG_AVR32_AT32AP_CPUFREQ=y
-CONFIG_NET=y
-CONFIG_PACKET=y
-CONFIG_UNIX=y
-CONFIG_XFRM_USER=y
-CONFIG_NET_KEY=y
-CONFIG_INET=y
-CONFIG_IP_MULTICAST=y
-CONFIG_IP_ADVANCED_ROUTER=y
-CONFIG_IP_PNP=y
-CONFIG_IP_PNP_DHCP=y
-CONFIG_IP_MROUTE=y
-CONFIG_IP_PIMSM_V1=y
-CONFIG_SYN_COOKIES=y
-CONFIG_INET_AH=y
-CONFIG_INET_ESP=y
-CONFIG_INET_IPCOMP=y
-# CONFIG_INET_LRO is not set
-CONFIG_INET6_AH=y
-CONFIG_INET6_ESP=y
-CONFIG_INET6_IPCOMP=y
-CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
-# CONFIG_PREVENT_FIRMWARE_BUILD is not set
-# CONFIG_FW_LOADER is not set
-CONFIG_MTD=y
-CONFIG_MTD_CMDLINE_PARTS=y
-CONFIG_MTD_BLOCK=y
-CONFIG_MTD_CFI=y
-CONFIG_MTD_CFI_AMDSTD=y
-CONFIG_MTD_PHYSMAP=y
-CONFIG_MTD_DATAFLASH=y
-CONFIG_ATMEL_TCLIB=y
-CONFIG_EEPROM_AT24=y
-CONFIG_EEPROM_AT25=y
-CONFIG_NETDEVICES=y
-CONFIG_MACB=y
-# CONFIG_INPUT is not set
-# CONFIG_SERIO is not set
-# CONFIG_VT is not set
-# CONFIG_LEGACY_PTYS is not set
-# CONFIG_DEVKMEM is not set
-CONFIG_SERIAL_ATMEL=y
-CONFIG_SERIAL_ATMEL_CONSOLE=y
-# CONFIG_HW_RANDOM is not set
-CONFIG_I2C=y
-CONFIG_I2C_CHARDEV=y
-CONFIG_I2C_GPIO=y
-CONFIG_SPI=y
-CONFIG_SPI_ATMEL=y
-CONFIG_GPIO_SYSFS=y
-# CONFIG_HWMON is not set
-CONFIG_WATCHDOG=y
-CONFIG_AT32AP700X_WDT=y
-CONFIG_FB=y
-CONFIG_FB_ATMEL=y
-# CONFIG_USB_SUPPORT is not set
-CONFIG_MMC=y
-CONFIG_MMC_TEST=y
-CONFIG_MMC_ATMELMCI=y
-CONFIG_MMC_SPI=y
-CONFIG_NEW_LEDS=y
-CONFIG_LEDS_CLASS=y
-CONFIG_LEDS_GPIO=y
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_TIMER=y
-CONFIG_LEDS_TRIGGER_HEARTBEAT=y
-CONFIG_LEDS_TRIGGER_DEFAULT_ON=y
-CONFIG_RTC_CLASS=y
-CONFIG_RTC_DRV_DS1390=y
-CONFIG_DMADEVICES=y
-CONFIG_EXT2_FS=y
-CONFIG_EXT3_FS=y
-# CONFIG_EXT3_FS_XATTR is not set
-# CONFIG_DNOTIFY is not set
-CONFIG_MSDOS_FS=y
-CONFIG_VFAT_FS=y
-CONFIG_FAT_DEFAULT_CODEPAGE=850
-CONFIG_TMPFS=y
-CONFIG_CONFIGFS_FS=y
-CONFIG_JFFS2_FS=y
-CONFIG_NFS_FS=y
-CONFIG_ROOT_NFS=y
-CONFIG_NLS_CODEPAGE_437=y
-CONFIG_NLS_CODEPAGE_850=y
-CONFIG_NLS_ISO8859_1=y
-CONFIG_NLS_UTF8=y
-CONFIG_FRAME_POINTER=y
-CONFIG_MAGIC_SYSRQ=y
-CONFIG_CRYPTO_ECB=y
-CONFIG_CRYPTO_PCBC=y
-CONFIG_CRYPTO_ARC4=y
-CONFIG_CRC_CCITT=y
diff --git a/arch/avr32/include/asm/Kbuild b/arch/avr32/include/asm/Kbuild
deleted file mode 100644
index 3d7ef2c..0000000
--- a/arch/avr32/include/asm/Kbuild
+++ /dev/null
@@ -1,23 +0,0 @@
-
-generic-y += clkdev.h
-generic-y += delay.h
-generic-y += device.h
-generic-y += div64.h
-generic-y += emergency-restart.h
-generic-y += exec.h
-generic-y += futex.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += local.h
-generic-y += local64.h
-generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += param.h
-generic-y += percpu.h
-generic-y += preempt.h
-generic-y += sections.h
-generic-y += topology.h
-generic-y += trace_clock.h
-generic-y += vga.h
-generic-y += word-at-a-time.h
-generic-y += xor.h
diff --git a/arch/avr32/include/asm/addrspace.h b/arch/avr32/include/asm/addrspace.h
deleted file mode 100644
index 5a47a79..0000000
--- a/arch/avr32/include/asm/addrspace.h
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Definitions for the address spaces of the AVR32 CPUs. Heavily based on
- * include/asm-sh/addrspace.h
- *
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_ADDRSPACE_H
-#define __ASM_AVR32_ADDRSPACE_H
-
-#ifdef CONFIG_MMU
-
-/* Memory segments when segmentation is enabled */
-#define P0SEG 0x00000000
-#define P1SEG 0x80000000
-#define P2SEG 0xa0000000
-#define P3SEG 0xc0000000
-#define P4SEG 0xe0000000
-
-/* Returns the privileged segment base of a given address */
-#define PXSEG(a) (((unsigned long)(a)) & 0xe0000000)
-
-/* Returns the physical address of a PnSEG (n=1,2) address */
-#define PHYSADDR(a) (((unsigned long)(a)) & 0x1fffffff)
-
-/*
- * Map an address to a certain privileged segment
- */
-#define P1SEGADDR(a) ((__typeof__(a))(((unsigned long)(a) & 0x1fffffff) \
- | P1SEG))
-#define P2SEGADDR(a) ((__typeof__(a))(((unsigned long)(a) & 0x1fffffff) \
- | P2SEG))
-#define P3SEGADDR(a) ((__typeof__(a))(((unsigned long)(a) & 0x1fffffff) \
- | P3SEG))
-#define P4SEGADDR(a) ((__typeof__(a))(((unsigned long)(a) & 0x1fffffff) \
- | P4SEG))
-
-#endif /* CONFIG_MMU */
-
-#endif /* __ASM_AVR32_ADDRSPACE_H */
diff --git a/arch/avr32/include/asm/asm-offsets.h b/arch/avr32/include/asm/asm-offsets.h
deleted file mode 100644
index d370ee3..0000000
--- a/arch/avr32/include/asm/asm-offsets.h
+++ /dev/null
@@ -1 +0,0 @@
-#include <generated/asm-offsets.h>
diff --git a/arch/avr32/include/asm/asm.h b/arch/avr32/include/asm/asm.h
deleted file mode 100644
index a2c64f4..0000000
--- a/arch/avr32/include/asm/asm.h
+++ /dev/null
@@ -1,102 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_ASM_H__
-#define __ASM_AVR32_ASM_H__
-
-#include <asm/sysreg.h>
-#include <asm/asm-offsets.h>
-#include <asm/thread_info.h>
-
-#define mask_interrupts ssrf SYSREG_GM_OFFSET
-#define mask_exceptions ssrf SYSREG_EM_OFFSET
-#define unmask_interrupts csrf SYSREG_GM_OFFSET
-#define unmask_exceptions csrf SYSREG_EM_OFFSET
-
-#ifdef CONFIG_FRAME_POINTER
- .macro save_fp
- st.w --sp, r7
- .endm
- .macro restore_fp
- ld.w r7, sp++
- .endm
- .macro zero_fp
- mov r7, 0
- .endm
-#else
- .macro save_fp
- .endm
- .macro restore_fp
- .endm
- .macro zero_fp
- .endm
-#endif
- .macro get_thread_info reg
- mov \reg, sp
- andl \reg, ~(THREAD_SIZE - 1) & 0xffff
- .endm
-
- /* Save and restore registers */
- .macro save_min sr, tmp=lr
- pushm lr
- mfsr \tmp, \sr
- zero_fp
- st.w --sp, \tmp
- .endm
-
- .macro restore_min sr, tmp=lr
- ld.w \tmp, sp++
- mtsr \sr, \tmp
- popm lr
- .endm
-
- .macro save_half sr, tmp=lr
- save_fp
- pushm r8-r9,r10,r11,r12,lr
- zero_fp
- mfsr \tmp, \sr
- st.w --sp, \tmp
- .endm
-
- .macro restore_half sr, tmp=lr
- ld.w \tmp, sp++
- mtsr \sr, \tmp
- popm r8-r9,r10,r11,r12,lr
- restore_fp
- .endm
-
- .macro save_full_user sr, tmp=lr
- stmts --sp, r0,r1,r2,r3,r4,r5,r6,r7,r8,r9,r10,r11,r12,sp,lr
- st.w --sp, lr
- zero_fp
- mfsr \tmp, \sr
- st.w --sp, \tmp
- .endm
-
- .macro restore_full_user sr, tmp=lr
- ld.w \tmp, sp++
- mtsr \sr, \tmp
- ld.w lr, sp++
- ldmts sp++, r0,r1,r2,r3,r4,r5,r6,r7,r8,r9,r10,r11,r12,sp,lr
- .endm
-
- /* uaccess macros */
- .macro branch_if_kernel scratch, label
- get_thread_info \scratch
- ld.w \scratch, \scratch[TI_flags]
- bld \scratch, TIF_USERSPACE
- brcc \label
- .endm
-
- .macro ret_if_privileged scratch, addr, size, ret
- sub \scratch, \size, 1
- add \scratch, \addr
- retcs \ret
- retmi \ret
- .endm
-
-#endif /* __ASM_AVR32_ASM_H__ */
diff --git a/arch/avr32/include/asm/atomic.h b/arch/avr32/include/asm/atomic.h
deleted file mode 100644
index 3d5ce38..0000000
--- a/arch/avr32/include/asm/atomic.h
+++ /dev/null
@@ -1,243 +0,0 @@
-/*
- * Atomic operations that C can't guarantee us. Useful for
- * resource counting etc.
- *
- * But use these as seldom as possible since they are slower than
- * regular operations.
- *
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_ATOMIC_H
-#define __ASM_AVR32_ATOMIC_H
-
-#include <linux/types.h>
-#include <asm/cmpxchg.h>
-
-#define ATOMIC_INIT(i) { (i) }
-
-#define atomic_read(v) READ_ONCE((v)->counter)
-#define atomic_set(v, i) WRITE_ONCE(((v)->counter), (i))
-
-#define ATOMIC_OP_RETURN(op, asm_op, asm_con) \
-static inline int __atomic_##op##_return(int i, atomic_t *v) \
-{ \
- int result; \
- \
- asm volatile( \
- "/* atomic_" #op "_return */\n" \
- "1: ssrf 5\n" \
- " ld.w %0, %2\n" \
- " " #asm_op " %0, %3\n" \
- " stcond %1, %0\n" \
- " brne 1b" \
- : "=&r" (result), "=o" (v->counter) \
- : "m" (v->counter), #asm_con (i) \
- : "cc"); \
- \
- return result; \
-}
-
-#define ATOMIC_FETCH_OP(op, asm_op, asm_con) \
-static inline int __atomic_fetch_##op(int i, atomic_t *v) \
-{ \
- int result, val; \
- \
- asm volatile( \
- "/* atomic_fetch_" #op " */\n" \
- "1: ssrf 5\n" \
- " ld.w %0, %3\n" \
- " mov %1, %0\n" \
- " " #asm_op " %1, %4\n" \
- " stcond %2, %1\n" \
- " brne 1b" \
- : "=&r" (result), "=&r" (val), "=o" (v->counter) \
- : "m" (v->counter), #asm_con (i) \
- : "cc"); \
- \
- return result; \
-}
-
-ATOMIC_OP_RETURN(sub, sub, rKs21)
-ATOMIC_OP_RETURN(add, add, r)
-ATOMIC_FETCH_OP (sub, sub, rKs21)
-ATOMIC_FETCH_OP (add, add, r)
-
-#define ATOMIC_OPS(op, asm_op) \
-ATOMIC_OP_RETURN(op, asm_op, r) \
-static inline void atomic_##op(int i, atomic_t *v) \
-{ \
- (void)__atomic_##op##_return(i, v); \
-} \
-ATOMIC_FETCH_OP(op, asm_op, r) \
-static inline int atomic_fetch_##op(int i, atomic_t *v) \
-{ \
- return __atomic_fetch_##op(i, v); \
-}
-
-ATOMIC_OPS(and, and)
-ATOMIC_OPS(or, or)
-ATOMIC_OPS(xor, eor)
-
-#undef ATOMIC_OPS
-#undef ATOMIC_FETCH_OP
-#undef ATOMIC_OP_RETURN
-
-/*
- * Probably found the reason why we want to use sub with the signed 21-bit
- * limit, it uses one less register than the add instruction that can add up to
- * 32-bit values.
- *
- * Both instructions are 32-bit, to use a 16-bit instruction the immediate is
- * very small; 4 bit.
- *
- * sub 32-bit, type IV, takes a register and subtracts a 21-bit immediate.
- * add 32-bit, type II, adds two register values together.
- */
-#define IS_21BIT_CONST(i) \
- (__builtin_constant_p(i) && ((i) >= -1048575) && ((i) <= 1048576))
-
-/*
- * atomic_add_return - add integer to atomic variable
- * @i: integer value to add
- * @v: pointer of type atomic_t
- *
- * Atomically adds @i to @v. Returns the resulting value.
- */
-static inline int atomic_add_return(int i, atomic_t *v)
-{
- if (IS_21BIT_CONST(i))
- return __atomic_sub_return(-i, v);
-
- return __atomic_add_return(i, v);
-}
-
-static inline int atomic_fetch_add(int i, atomic_t *v)
-{
- if (IS_21BIT_CONST(i))
- return __atomic_fetch_sub(-i, v);
-
- return __atomic_fetch_add(i, v);
-}
-
-/*
- * atomic_sub_return - subtract the atomic variable
- * @i: integer value to subtract
- * @v: pointer of type atomic_t
- *
- * Atomically subtracts @i from @v. Returns the resulting value.
- */
-static inline int atomic_sub_return(int i, atomic_t *v)
-{
- if (IS_21BIT_CONST(i))
- return __atomic_sub_return(i, v);
-
- return __atomic_add_return(-i, v);
-}
-
-static inline int atomic_fetch_sub(int i, atomic_t *v)
-{
- if (IS_21BIT_CONST(i))
- return __atomic_fetch_sub(i, v);
-
- return __atomic_fetch_add(-i, v);
-}
-
-/*
- * __atomic_add_unless - add unless the number is a given value
- * @v: pointer of type atomic_t
- * @a: the amount to add to v...
- * @u: ...unless v is equal to u.
- *
- * Atomically adds @a to @v, so long as it was not @u.
- * Returns the old value of @v.
-*/
-static inline int __atomic_add_unless(atomic_t *v, int a, int u)
-{
- int tmp, old = atomic_read(v);
-
- if (IS_21BIT_CONST(a)) {
- asm volatile(
- "/* __atomic_sub_unless */\n"
- "1: ssrf 5\n"
- " ld.w %0, %2\n"
- " cp.w %0, %4\n"
- " breq 1f\n"
- " sub %0, %3\n"
- " stcond %1, %0\n"
- " brne 1b\n"
- "1:"
- : "=&r"(tmp), "=o"(v->counter)
- : "m"(v->counter), "rKs21"(-a), "rKs21"(u)
- : "cc", "memory");
- } else {
- asm volatile(
- "/* __atomic_add_unless */\n"
- "1: ssrf 5\n"
- " ld.w %0, %2\n"
- " cp.w %0, %4\n"
- " breq 1f\n"
- " add %0, %3\n"
- " stcond %1, %0\n"
- " brne 1b\n"
- "1:"
- : "=&r"(tmp), "=o"(v->counter)
- : "m"(v->counter), "r"(a), "ir"(u)
- : "cc", "memory");
- }
-
- return old;
-}
-
-#undef IS_21BIT_CONST
-
-/*
- * atomic_sub_if_positive - conditionally subtract integer from atomic variable
- * @i: integer value to subtract
- * @v: pointer of type atomic_t
- *
- * Atomically test @v and subtract @i if @v is greater or equal than @i.
- * The function returns the old value of @v minus @i.
- */
-static inline int atomic_sub_if_positive(int i, atomic_t *v)
-{
- int result;
-
- asm volatile(
- "/* atomic_sub_if_positive */\n"
- "1: ssrf 5\n"
- " ld.w %0, %2\n"
- " sub %0, %3\n"
- " brlt 1f\n"
- " stcond %1, %0\n"
- " brne 1b\n"
- "1:"
- : "=&r"(result), "=o"(v->counter)
- : "m"(v->counter), "ir"(i)
- : "cc", "memory");
-
- return result;
-}
-
-#define atomic_xchg(v, new) (xchg(&((v)->counter), new))
-#define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n)))
-
-#define atomic_sub(i, v) (void)atomic_sub_return(i, v)
-#define atomic_add(i, v) (void)atomic_add_return(i, v)
-#define atomic_dec(v) atomic_sub(1, (v))
-#define atomic_inc(v) atomic_add(1, (v))
-
-#define atomic_dec_return(v) atomic_sub_return(1, v)
-#define atomic_inc_return(v) atomic_add_return(1, v)
-
-#define atomic_sub_and_test(i, v) (atomic_sub_return(i, v) == 0)
-#define atomic_inc_and_test(v) (atomic_add_return(1, v) == 0)
-#define atomic_dec_and_test(v) (atomic_sub_return(1, v) == 0)
-#define atomic_add_negative(i, v) (atomic_add_return(i, v) < 0)
-
-#define atomic_dec_if_positive(v) atomic_sub_if_positive(1, v)
-
-#endif /* __ASM_AVR32_ATOMIC_H */
diff --git a/arch/avr32/include/asm/barrier.h b/arch/avr32/include/asm/barrier.h
deleted file mode 100644
index 7151007..0000000
--- a/arch/avr32/include/asm/barrier.h
+++ /dev/null
@@ -1,22 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_BARRIER_H
-#define __ASM_AVR32_BARRIER_H
-
-/*
- * Weirdest thing ever.. no full barrier, but it has a write barrier!
- */
-#define wmb() asm volatile("sync 0" : : : "memory")
-
-#ifdef CONFIG_SMP
-# error "The AVR32 port does not support SMP"
-#endif
-
-#include <asm-generic/barrier.h>
-
-#endif /* __ASM_AVR32_BARRIER_H */
diff --git a/arch/avr32/include/asm/bitops.h b/arch/avr32/include/asm/bitops.h
deleted file mode 100644
index 910d537..0000000
--- a/arch/avr32/include/asm/bitops.h
+++ /dev/null
@@ -1,314 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_BITOPS_H
-#define __ASM_AVR32_BITOPS_H
-
-#ifndef _LINUX_BITOPS_H
-#error only <linux/bitops.h> can be included directly
-#endif
-
-#include <asm/byteorder.h>
-#include <asm/barrier.h>
-
-/*
- * set_bit - Atomically set a bit in memory
- * @nr: the bit to set
- * @addr: the address to start counting from
- *
- * This function is atomic and may not be reordered. See __set_bit()
- * if you do not require the atomic guarantees.
- *
- * Note that @nr may be almost arbitrarily large; this function is not
- * restricted to acting on a single-word quantity.
- */
-static inline void set_bit(int nr, volatile void * addr)
-{
- unsigned long *p = ((unsigned long *)addr) + nr / BITS_PER_LONG;
- unsigned long tmp;
-
- if (__builtin_constant_p(nr)) {
- asm volatile(
- "1: ssrf 5\n"
- " ld.w %0, %2\n"
- " sbr %0, %3\n"
- " stcond %1, %0\n"
- " brne 1b"
- : "=&r"(tmp), "=o"(*p)
- : "m"(*p), "i"(nr)
- : "cc");
- } else {
- unsigned long mask = 1UL << (nr % BITS_PER_LONG);
- asm volatile(
- "1: ssrf 5\n"
- " ld.w %0, %2\n"
- " or %0, %3\n"
- " stcond %1, %0\n"
- " brne 1b"
- : "=&r"(tmp), "=o"(*p)
- : "m"(*p), "r"(mask)
- : "cc");
- }
-}
-
-/*
- * clear_bit - Clears a bit in memory
- * @nr: Bit to clear
- * @addr: Address to start counting from
- *
- * clear_bit() is atomic and may not be reordered. However, it does
- * not contain a memory barrier, so if it is used for locking purposes,
- * you should call smp_mb__before_atomic() and/or smp_mb__after_atomic()
- * in order to ensure changes are visible on other processors.
- */
-static inline void clear_bit(int nr, volatile void * addr)
-{
- unsigned long *p = ((unsigned long *)addr) + nr / BITS_PER_LONG;
- unsigned long tmp;
-
- if (__builtin_constant_p(nr)) {
- asm volatile(
- "1: ssrf 5\n"
- " ld.w %0, %2\n"
- " cbr %0, %3\n"
- " stcond %1, %0\n"
- " brne 1b"
- : "=&r"(tmp), "=o"(*p)
- : "m"(*p), "i"(nr)
- : "cc");
- } else {
- unsigned long mask = 1UL << (nr % BITS_PER_LONG);
- asm volatile(
- "1: ssrf 5\n"
- " ld.w %0, %2\n"
- " andn %0, %3\n"
- " stcond %1, %0\n"
- " brne 1b"
- : "=&r"(tmp), "=o"(*p)
- : "m"(*p), "r"(mask)
- : "cc");
- }
-}
-
-/*
- * change_bit - Toggle a bit in memory
- * @nr: Bit to change
- * @addr: Address to start counting from
- *
- * change_bit() is atomic and may not be reordered.
- * Note that @nr may be almost arbitrarily large; this function is not
- * restricted to acting on a single-word quantity.
- */
-static inline void change_bit(int nr, volatile void * addr)
-{
- unsigned long *p = ((unsigned long *)addr) + nr / BITS_PER_LONG;
- unsigned long mask = 1UL << (nr % BITS_PER_LONG);
- unsigned long tmp;
-
- asm volatile(
- "1: ssrf 5\n"
- " ld.w %0, %2\n"
- " eor %0, %3\n"
- " stcond %1, %0\n"
- " brne 1b"
- : "=&r"(tmp), "=o"(*p)
- : "m"(*p), "r"(mask)
- : "cc");
-}
-
-/*
- * test_and_set_bit - Set a bit and return its old value
- * @nr: Bit to set
- * @addr: Address to count from
- *
- * This operation is atomic and cannot be reordered.
- * It also implies a memory barrier.
- */
-static inline int test_and_set_bit(int nr, volatile void * addr)
-{
- unsigned long *p = ((unsigned long *)addr) + nr / BITS_PER_LONG;
- unsigned long mask = 1UL << (nr % BITS_PER_LONG);
- unsigned long tmp, old;
-
- if (__builtin_constant_p(nr)) {
- asm volatile(
- "1: ssrf 5\n"
- " ld.w %0, %3\n"
- " mov %2, %0\n"
- " sbr %0, %4\n"
- " stcond %1, %0\n"
- " brne 1b"
- : "=&r"(tmp), "=o"(*p), "=&r"(old)
- : "m"(*p), "i"(nr)
- : "memory", "cc");
- } else {
- asm volatile(
- "1: ssrf 5\n"
- " ld.w %2, %3\n"
- " or %0, %2, %4\n"
- " stcond %1, %0\n"
- " brne 1b"
- : "=&r"(tmp), "=o"(*p), "=&r"(old)
- : "m"(*p), "r"(mask)
- : "memory", "cc");
- }
-
- return (old & mask) != 0;
-}
-
-/*
- * test_and_clear_bit - Clear a bit and return its old value
- * @nr: Bit to clear
- * @addr: Address to count from
- *
- * This operation is atomic and cannot be reordered.
- * It also implies a memory barrier.
- */
-static inline int test_and_clear_bit(int nr, volatile void * addr)
-{
- unsigned long *p = ((unsigned long *)addr) + nr / BITS_PER_LONG;
- unsigned long mask = 1UL << (nr % BITS_PER_LONG);
- unsigned long tmp, old;
-
- if (__builtin_constant_p(nr)) {
- asm volatile(
- "1: ssrf 5\n"
- " ld.w %0, %3\n"
- " mov %2, %0\n"
- " cbr %0, %4\n"
- " stcond %1, %0\n"
- " brne 1b"
- : "=&r"(tmp), "=o"(*p), "=&r"(old)
- : "m"(*p), "i"(nr)
- : "memory", "cc");
- } else {
- asm volatile(
- "1: ssrf 5\n"
- " ld.w %0, %3\n"
- " mov %2, %0\n"
- " andn %0, %4\n"
- " stcond %1, %0\n"
- " brne 1b"
- : "=&r"(tmp), "=o"(*p), "=&r"(old)
- : "m"(*p), "r"(mask)
- : "memory", "cc");
- }
-
- return (old & mask) != 0;
-}
-
-/*
- * test_and_change_bit - Change a bit and return its old value
- * @nr: Bit to change
- * @addr: Address to count from
- *
- * This operation is atomic and cannot be reordered.
- * It also implies a memory barrier.
- */
-static inline int test_and_change_bit(int nr, volatile void * addr)
-{
- unsigned long *p = ((unsigned long *)addr) + nr / BITS_PER_LONG;
- unsigned long mask = 1UL << (nr % BITS_PER_LONG);
- unsigned long tmp, old;
-
- asm volatile(
- "1: ssrf 5\n"
- " ld.w %2, %3\n"
- " eor %0, %2, %4\n"
- " stcond %1, %0\n"
- " brne 1b"
- : "=&r"(tmp), "=o"(*p), "=&r"(old)
- : "m"(*p), "r"(mask)
- : "memory", "cc");
-
- return (old & mask) != 0;
-}
-
-#include <asm-generic/bitops/non-atomic.h>
-
-/* Find First bit Set */
-static inline unsigned long __ffs(unsigned long word)
-{
- unsigned long result;
-
- asm("brev %1\n\t"
- "clz %0,%1"
- : "=r"(result), "=&r"(word)
- : "1"(word));
- return result;
-}
-
-/* Find First Zero */
-static inline unsigned long ffz(unsigned long word)
-{
- return __ffs(~word);
-}
-
-/* Find Last bit Set */
-static inline int fls(unsigned long word)
-{
- unsigned long result;
-
- asm("clz %0,%1" : "=r"(result) : "r"(word));
- return 32 - result;
-}
-
-static inline int __fls(unsigned long word)
-{
- return fls(word) - 1;
-}
-
-unsigned long find_first_zero_bit(const unsigned long *addr,
- unsigned long size);
-#define find_first_zero_bit find_first_zero_bit
-
-unsigned long find_next_zero_bit(const unsigned long *addr,
- unsigned long size,
- unsigned long offset);
-#define find_next_zero_bit find_next_zero_bit
-
-unsigned long find_first_bit(const unsigned long *addr,
- unsigned long size);
-#define find_first_bit find_first_bit
-
-unsigned long find_next_bit(const unsigned long *addr,
- unsigned long size,
- unsigned long offset);
-#define find_next_bit find_next_bit
-
-/*
- * ffs: find first bit set. This is defined the same way as
- * the libc and compiler builtin ffs routines, therefore
- * differs in spirit from the above ffz (man ffs).
- *
- * The difference is that bit numbering starts at 1, and if no bit is set,
- * the function returns 0.
- */
-static inline int ffs(unsigned long word)
-{
- if(word == 0)
- return 0;
- return __ffs(word) + 1;
-}
-
-#include <asm-generic/bitops/fls64.h>
-#include <asm-generic/bitops/sched.h>
-#include <asm-generic/bitops/hweight.h>
-#include <asm-generic/bitops/lock.h>
-
-extern unsigned long find_next_zero_bit_le(const void *addr,
- unsigned long size, unsigned long offset);
-#define find_next_zero_bit_le find_next_zero_bit_le
-
-extern unsigned long find_next_bit_le(const void *addr,
- unsigned long size, unsigned long offset);
-#define find_next_bit_le find_next_bit_le
-
-#include <asm-generic/bitops/le.h>
-#include <asm-generic/bitops/ext2-atomic.h>
-
-#endif /* __ASM_AVR32_BITOPS_H */
diff --git a/arch/avr32/include/asm/bug.h b/arch/avr32/include/asm/bug.h
deleted file mode 100644
index 85a92d0..0000000
--- a/arch/avr32/include/asm/bug.h
+++ /dev/null
@@ -1,78 +0,0 @@
-/*
- * Copyright (C) 2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_BUG_H
-#define __ASM_AVR32_BUG_H
-
-#ifdef CONFIG_BUG
-
-/*
- * According to our Chief Architect, this compact opcode is very
- * unlikely to ever be implemented.
- */
-#define AVR32_BUG_OPCODE 0x5df0
-
-#ifdef CONFIG_DEBUG_BUGVERBOSE
-
-#define _BUG_OR_WARN(flags) \
- asm volatile( \
- "1: .hword %0\n" \
- " .section __bug_table,\"a\",@progbits\n" \
- "2: .long 1b\n" \
- " .long %1\n" \
- " .short %2\n" \
- " .short %3\n" \
- " .org 2b + %4\n" \
- " .previous" \
- : \
- : "i"(AVR32_BUG_OPCODE), "i"(__FILE__), \
- "i"(__LINE__), "i"(flags), \
- "i"(sizeof(struct bug_entry)))
-
-#else
-
-#define _BUG_OR_WARN(flags) \
- asm volatile( \
- "1: .hword %0\n" \
- " .section __bug_table,\"a\",@progbits\n" \
- "2: .long 1b\n" \
- " .short %1\n" \
- " .org 2b + %2\n" \
- " .previous" \
- : \
- : "i"(AVR32_BUG_OPCODE), "i"(flags), \
- "i"(sizeof(struct bug_entry)))
-
-#endif /* CONFIG_DEBUG_BUGVERBOSE */
-
-#define BUG() \
- do { \
- _BUG_OR_WARN(0); \
- unreachable(); \
- } while (0)
-
-#define WARN_ON(condition) \
- ({ \
- int __ret_warn_on = !!(condition); \
- if (unlikely(__ret_warn_on)) \
- _BUG_OR_WARN(BUGFLAG_WARNING); \
- unlikely(__ret_warn_on); \
- })
-
-#define HAVE_ARCH_BUG
-#define HAVE_ARCH_WARN_ON
-
-#endif /* CONFIG_BUG */
-
-#include <asm-generic/bug.h>
-
-struct pt_regs;
-void die(const char *str, struct pt_regs *regs, long err);
-void _exception(long signr, struct pt_regs *regs, int code,
- unsigned long addr);
-
-#endif /* __ASM_AVR32_BUG_H */
diff --git a/arch/avr32/include/asm/bugs.h b/arch/avr32/include/asm/bugs.h
deleted file mode 100644
index 278661b..0000000
--- a/arch/avr32/include/asm/bugs.h
+++ /dev/null
@@ -1,15 +0,0 @@
-/*
- * This is included by init/main.c to check for architecture-dependent bugs.
- *
- * Needs:
- * void check_bugs(void);
- */
-#ifndef __ASM_AVR32_BUGS_H
-#define __ASM_AVR32_BUGS_H
-
-static void __init check_bugs(void)
-{
- boot_cpu_data.loops_per_jiffy = loops_per_jiffy;
-}
-
-#endif /* __ASM_AVR32_BUGS_H */
diff --git a/arch/avr32/include/asm/cache.h b/arch/avr32/include/asm/cache.h
deleted file mode 100644
index c3a58a1..0000000
--- a/arch/avr32/include/asm/cache.h
+++ /dev/null
@@ -1,38 +0,0 @@
-#ifndef __ASM_AVR32_CACHE_H
-#define __ASM_AVR32_CACHE_H
-
-#define L1_CACHE_SHIFT 5
-#define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT)
-
-/*
- * Memory returned by kmalloc() may be used for DMA, so we must make
- * sure that all such allocations are cache aligned. Otherwise,
- * unrelated code may cause parts of the buffer to be read into the
- * cache before the transfer is done, causing old data to be seen by
- * the CPU.
- */
-#define ARCH_DMA_MINALIGN L1_CACHE_BYTES
-
-#ifndef __ASSEMBLER__
-struct cache_info {
- unsigned int ways;
- unsigned int sets;
- unsigned int linesz;
-};
-#endif /* __ASSEMBLER */
-
-/* Cache operation constants */
-#define ICACHE_FLUSH 0x00
-#define ICACHE_INVALIDATE 0x01
-#define ICACHE_LOCK 0x02
-#define ICACHE_UNLOCK 0x03
-#define ICACHE_PREFETCH 0x04
-
-#define DCACHE_FLUSH 0x08
-#define DCACHE_LOCK 0x09
-#define DCACHE_UNLOCK 0x0a
-#define DCACHE_INVALIDATE 0x0b
-#define DCACHE_CLEAN 0x0c
-#define DCACHE_CLEAN_INVAL 0x0d
-
-#endif /* __ASM_AVR32_CACHE_H */
diff --git a/arch/avr32/include/asm/cacheflush.h b/arch/avr32/include/asm/cacheflush.h
deleted file mode 100644
index 96e5382..0000000
--- a/arch/avr32/include/asm/cacheflush.h
+++ /dev/null
@@ -1,132 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_CACHEFLUSH_H
-#define __ASM_AVR32_CACHEFLUSH_H
-
-/* Keep includes the same across arches. */
-#include <linux/mm.h>
-
-#define CACHE_OP_ICACHE_INVALIDATE 0x01
-#define CACHE_OP_DCACHE_INVALIDATE 0x0b
-#define CACHE_OP_DCACHE_CLEAN 0x0c
-#define CACHE_OP_DCACHE_CLEAN_INVAL 0x0d
-
-/*
- * Invalidate any cacheline containing virtual address vaddr without
- * writing anything back to memory.
- *
- * Note that this function may corrupt unrelated data structures when
- * applied on buffers that are not cacheline aligned in both ends.
- */
-static inline void invalidate_dcache_line(void *vaddr)
-{
- asm volatile("cache %0[0], %1"
- :
- : "r"(vaddr), "n"(CACHE_OP_DCACHE_INVALIDATE)
- : "memory");
-}
-
-/*
- * Make sure any cacheline containing virtual address vaddr is written
- * to memory.
- */
-static inline void clean_dcache_line(void *vaddr)
-{
- asm volatile("cache %0[0], %1"
- :
- : "r"(vaddr), "n"(CACHE_OP_DCACHE_CLEAN)
- : "memory");
-}
-
-/*
- * Make sure any cacheline containing virtual address vaddr is written
- * to memory and then invalidate it.
- */
-static inline void flush_dcache_line(void *vaddr)
-{
- asm volatile("cache %0[0], %1"
- :
- : "r"(vaddr), "n"(CACHE_OP_DCACHE_CLEAN_INVAL)
- : "memory");
-}
-
-/*
- * Invalidate any instruction cacheline containing virtual address
- * vaddr.
- */
-static inline void invalidate_icache_line(void *vaddr)
-{
- asm volatile("cache %0[0], %1"
- :
- : "r"(vaddr), "n"(CACHE_OP_ICACHE_INVALIDATE)
- : "memory");
-}
-
-/*
- * Applies the above functions on all lines that are touched by the
- * specified virtual address range.
- */
-void invalidate_dcache_region(void *start, size_t len);
-void clean_dcache_region(void *start, size_t len);
-void flush_dcache_region(void *start, size_t len);
-void invalidate_icache_region(void *start, size_t len);
-
-/*
- * Make sure any pending writes are completed before continuing.
- */
-#define flush_write_buffer() asm volatile("sync 0" : : : "memory")
-
-/*
- * The following functions are called when a virtual mapping changes.
- * We do not need to flush anything in this case.
- */
-#define flush_cache_all() do { } while (0)
-#define flush_cache_mm(mm) do { } while (0)
-#define flush_cache_dup_mm(mm) do { } while (0)
-#define flush_cache_range(vma, start, end) do { } while (0)
-#define flush_cache_page(vma, vmaddr, pfn) do { } while (0)
-#define flush_cache_vmap(start, end) do { } while (0)
-#define flush_cache_vunmap(start, end) do { } while (0)
-
-/*
- * I think we need to implement this one to be able to reliably
- * execute pages from RAMDISK. However, if we implement the
- * flush_dcache_*() functions, it might not be needed anymore.
- *
- * #define flush_icache_page(vma, page) do { } while (0)
- */
-extern void flush_icache_page(struct vm_area_struct *vma, struct page *page);
-
-/*
- * These are (I think) related to D-cache aliasing. We might need to
- * do something here, but only for certain configurations. No such
- * configurations exist at this time.
- */
-#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 0
-#define flush_dcache_page(page) do { } while (0)
-#define flush_dcache_mmap_lock(page) do { } while (0)
-#define flush_dcache_mmap_unlock(page) do { } while (0)
-
-/*
- * These are for I/D cache coherency. In this case, we do need to
- * flush with all configurations.
- */
-extern void flush_icache_range(unsigned long start, unsigned long end);
-
-extern void copy_to_user_page(struct vm_area_struct *vma, struct page *page,
- unsigned long vaddr, void *dst, const void *src,
- unsigned long len);
-
-static inline void copy_from_user_page(struct vm_area_struct *vma,
- struct page *page, unsigned long vaddr, void *dst,
- const void *src, unsigned long len)
-{
- memcpy(dst, src, len);
-}
-
-#endif /* __ASM_AVR32_CACHEFLUSH_H */
diff --git a/arch/avr32/include/asm/checksum.h b/arch/avr32/include/asm/checksum.h
deleted file mode 100644
index 4ab7d5b..0000000
--- a/arch/avr32/include/asm/checksum.h
+++ /dev/null
@@ -1,150 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_CHECKSUM_H
-#define __ASM_AVR32_CHECKSUM_H
-
-/*
- * computes the checksum of a memory block at buff, length len,
- * and adds in "sum" (32-bit)
- *
- * returns a 32-bit number suitable for feeding into itself
- * or csum_tcpudp_magic
- *
- * this function must be called with even lengths, except
- * for the last fragment, which may be odd
- *
- * it's best to have buff aligned on a 32-bit boundary
- */
-__wsum csum_partial(const void *buff, int len, __wsum sum);
-
-/*
- * the same as csum_partial, but copies from src while it
- * checksums, and handles user-space pointer exceptions correctly, when needed.
- *
- * here even more important to align src and dst on a 32-bit (or even
- * better 64-bit) boundary
- */
-__wsum csum_partial_copy_generic(const void *src, void *dst, int len,
- __wsum sum, int *src_err_ptr,
- int *dst_err_ptr);
-
-/*
- * Note: when you get a NULL pointer exception here this means someone
- * passed in an incorrect kernel address to one of these functions.
- *
- * If you use these functions directly please don't forget the
- * access_ok().
- */
-static inline
-__wsum csum_partial_copy_nocheck(const void *src, void *dst,
- int len, __wsum sum)
-{
- return csum_partial_copy_generic(src, dst, len, sum, NULL, NULL);
-}
-
-static inline
-__wsum csum_partial_copy_from_user(const void __user *src, void *dst,
- int len, __wsum sum, int *err_ptr)
-{
- return csum_partial_copy_generic((const void __force *)src, dst, len,
- sum, err_ptr, NULL);
-}
-
-/*
- * This is a version of ip_compute_csum() optimized for IP headers,
- * which always checksum on 4 octet boundaries.
- */
-static inline __sum16 ip_fast_csum(const void *iph, unsigned int ihl)
-{
- unsigned int sum, tmp;
-
- __asm__ __volatile__(
- " ld.w %0, %1++\n"
- " ld.w %3, %1++\n"
- " sub %2, 4\n"
- " add %0, %3\n"
- " ld.w %3, %1++\n"
- " adc %0, %0, %3\n"
- " ld.w %3, %1++\n"
- " adc %0, %0, %3\n"
- " acr %0\n"
- "1: ld.w %3, %1++\n"
- " add %0, %3\n"
- " acr %0\n"
- " sub %2, 1\n"
- " brne 1b\n"
- " lsl %3, %0, 16\n"
- " andl %0, 0\n"
- " mov %2, 0xffff\n"
- " add %0, %3\n"
- " adc %0, %0, %2\n"
- " com %0\n"
- " lsr %0, 16\n"
- : "=r"(sum), "=r"(iph), "=r"(ihl), "=r"(tmp)
- : "1"(iph), "2"(ihl)
- : "memory", "cc");
- return (__force __sum16)sum;
-}
-
-/*
- * Fold a partial checksum
- */
-
-static inline __sum16 csum_fold(__wsum sum)
-{
- unsigned int tmp;
-
- asm(" bfextu %1, %0, 0, 16\n"
- " lsr %0, 16\n"
- " add %0, %1\n"
- " bfextu %1, %0, 16, 16\n"
- " add %0, %1"
- : "=&r"(sum), "=&r"(tmp)
- : "0"(sum));
-
- return (__force __sum16)~sum;
-}
-
-static inline __wsum csum_tcpudp_nofold(__be32 saddr, __be32 daddr,
- __u32 len, __u8 proto,
- __wsum sum)
-{
- asm(" add %0, %1\n"
- " adc %0, %0, %2\n"
- " adc %0, %0, %3\n"
- " acr %0"
- : "=r"(sum)
- : "r"(daddr), "r"(saddr), "r"(len + proto),
- "0"(sum)
- : "cc");
-
- return sum;
-}
-
-/*
- * computes the checksum of the TCP/UDP pseudo-header
- * returns a 16-bit checksum, already complemented
- */
-static inline __sum16 csum_tcpudp_magic(__be32 saddr, __be32 daddr,
- __u32 len, __u8 proto,
- __wsum sum)
-{
- return csum_fold(csum_tcpudp_nofold(saddr,daddr,len,proto,sum));
-}
-
-/*
- * this routine is used for miscellaneous IP-like checksums, mainly
- * in icmp.c
- */
-
-static inline __sum16 ip_compute_csum(const void *buff, int len)
-{
- return csum_fold(csum_partial(buff, len, 0));
-}
-
-#endif /* __ASM_AVR32_CHECKSUM_H */
diff --git a/arch/avr32/include/asm/cmpxchg.h b/arch/avr32/include/asm/cmpxchg.h
deleted file mode 100644
index 572739b..0000000
--- a/arch/avr32/include/asm/cmpxchg.h
+++ /dev/null
@@ -1,115 +0,0 @@
-/*
- * Atomic operations that C can't guarantee us. Useful for
- * resource counting etc.
- *
- * But use these as seldom as possible since they are slower than
- * regular operations.
- *
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_CMPXCHG_H
-#define __ASM_AVR32_CMPXCHG_H
-
-#define xchg(ptr,x) \
- ((__typeof__(*(ptr)))__xchg((unsigned long)(x),(ptr),sizeof(*(ptr))))
-
-extern void __xchg_called_with_bad_pointer(void);
-
-static inline unsigned long xchg_u32(u32 val, volatile u32 *m)
-{
- u32 ret;
-
- asm volatile("xchg %[ret], %[m], %[val]"
- : [ret] "=&r"(ret), "=m"(*m)
- : "m"(*m), [m] "r"(m), [val] "r"(val)
- : "memory");
- return ret;
-}
-
-static inline unsigned long __xchg(unsigned long x,
- volatile void *ptr,
- int size)
-{
- switch(size) {
- case 4:
- return xchg_u32(x, ptr);
- default:
- __xchg_called_with_bad_pointer();
- return x;
- }
-}
-
-static inline unsigned long __cmpxchg_u32(volatile int *m, unsigned long old,
- unsigned long new)
-{
- __u32 ret;
-
- asm volatile(
- "1: ssrf 5\n"
- " ld.w %[ret], %[m]\n"
- " cp.w %[ret], %[old]\n"
- " brne 2f\n"
- " stcond %[m], %[new]\n"
- " brne 1b\n"
- "2:\n"
- : [ret] "=&r"(ret), [m] "=m"(*m)
- : "m"(m), [old] "Ks21r"(old), [new] "r"(new)
- : "memory", "cc");
- return ret;
-}
-
-extern unsigned long __cmpxchg_u64_unsupported_on_32bit_kernels(
- volatile int * m, unsigned long old, unsigned long new);
-#define __cmpxchg_u64 __cmpxchg_u64_unsupported_on_32bit_kernels
-
-/* This function doesn't exist, so you'll get a linker error
- if something tries to do an invalid cmpxchg(). */
-extern void __cmpxchg_called_with_bad_pointer(void);
-
-static inline unsigned long __cmpxchg(volatile void *ptr, unsigned long old,
- unsigned long new, int size)
-{
- switch (size) {
- case 4:
- return __cmpxchg_u32(ptr, old, new);
- case 8:
- return __cmpxchg_u64(ptr, old, new);
- }
-
- __cmpxchg_called_with_bad_pointer();
- return old;
-}
-
-#define cmpxchg(ptr, old, new) \
- ((typeof(*(ptr)))__cmpxchg((ptr), (unsigned long)(old), \
- (unsigned long)(new), \
- sizeof(*(ptr))))
-
-#include <asm-generic/cmpxchg-local.h>
-
-static inline unsigned long __cmpxchg_local(volatile void *ptr,
- unsigned long old,
- unsigned long new, int size)
-{
- switch (size) {
- case 4:
- return __cmpxchg_u32(ptr, old, new);
- default:
- return __cmpxchg_local_generic(ptr, old, new, size);
- }
-
- return old;
-}
-
-#define cmpxchg_local(ptr, old, new) \
- ((typeof(*(ptr)))__cmpxchg_local((ptr), (unsigned long)(old), \
- (unsigned long)(new), \
- sizeof(*(ptr))))
-
-#define cmpxchg64_local(ptr, o, n) __cmpxchg64_local_generic((ptr), (o), (n))
-
-#endif /* __ASM_AVR32_CMPXCHG_H */
diff --git a/arch/avr32/include/asm/current.h b/arch/avr32/include/asm/current.h
deleted file mode 100644
index c7b0549..0000000
--- a/arch/avr32/include/asm/current.h
+++ /dev/null
@@ -1,15 +0,0 @@
-#ifndef __ASM_AVR32_CURRENT_H
-#define __ASM_AVR32_CURRENT_H
-
-#include <linux/thread_info.h>
-
-struct task_struct;
-
-inline static struct task_struct * get_current(void)
-{
- return current_thread_info()->task;
-}
-
-#define current get_current()
-
-#endif /* __ASM_AVR32_CURRENT_H */
diff --git a/arch/avr32/include/asm/dma-mapping.h b/arch/avr32/include/asm/dma-mapping.h
deleted file mode 100644
index 7388451..0000000
--- a/arch/avr32/include/asm/dma-mapping.h
+++ /dev/null
@@ -1,14 +0,0 @@
-#ifndef __ASM_AVR32_DMA_MAPPING_H
-#define __ASM_AVR32_DMA_MAPPING_H
-
-extern void dma_cache_sync(struct device *dev, void *vaddr, size_t size,
- int direction);
-
-extern const struct dma_map_ops avr32_dma_ops;
-
-static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus)
-{
- return &avr32_dma_ops;
-}
-
-#endif /* __ASM_AVR32_DMA_MAPPING_H */
diff --git a/arch/avr32/include/asm/dma.h b/arch/avr32/include/asm/dma.h
deleted file mode 100644
index 9e91205..0000000
--- a/arch/avr32/include/asm/dma.h
+++ /dev/null
@@ -1,8 +0,0 @@
-#ifndef __ASM_AVR32_DMA_H
-#define __ASM_AVR32_DMA_H
-
-/* The maximum address that we can perform a DMA transfer to on this platform.
- * Not really applicable to AVR32, but some functions need it. */
-#define MAX_DMA_ADDRESS 0xffffffff
-
-#endif /* __ASM_AVR32_DMA_H */
diff --git a/arch/avr32/include/asm/elf.h b/arch/avr32/include/asm/elf.h
deleted file mode 100644
index 0388ece..0000000
--- a/arch/avr32/include/asm/elf.h
+++ /dev/null
@@ -1,105 +0,0 @@
-#ifndef __ASM_AVR32_ELF_H
-#define __ASM_AVR32_ELF_H
-
-/* AVR32 relocation numbers */
-#define R_AVR32_NONE 0
-#define R_AVR32_32 1
-#define R_AVR32_16 2
-#define R_AVR32_8 3
-#define R_AVR32_32_PCREL 4
-#define R_AVR32_16_PCREL 5
-#define R_AVR32_8_PCREL 6
-#define R_AVR32_DIFF32 7
-#define R_AVR32_DIFF16 8
-#define R_AVR32_DIFF8 9
-#define R_AVR32_GOT32 10
-#define R_AVR32_GOT16 11
-#define R_AVR32_GOT8 12
-#define R_AVR32_21S 13
-#define R_AVR32_16U 14
-#define R_AVR32_16S 15
-#define R_AVR32_8S 16
-#define R_AVR32_8S_EXT 17
-#define R_AVR32_22H_PCREL 18
-#define R_AVR32_18W_PCREL 19
-#define R_AVR32_16B_PCREL 20
-#define R_AVR32_16N_PCREL 21
-#define R_AVR32_14UW_PCREL 22
-#define R_AVR32_11H_PCREL 23
-#define R_AVR32_10UW_PCREL 24
-#define R_AVR32_9H_PCREL 25
-#define R_AVR32_9UW_PCREL 26
-#define R_AVR32_HI16 27
-#define R_AVR32_LO16 28
-#define R_AVR32_GOTPC 29
-#define R_AVR32_GOTCALL 30
-#define R_AVR32_LDA_GOT 31
-#define R_AVR32_GOT21S 32
-#define R_AVR32_GOT18SW 33
-#define R_AVR32_GOT16S 34
-#define R_AVR32_GOT7UW 35
-#define R_AVR32_32_CPENT 36
-#define R_AVR32_CPCALL 37
-#define R_AVR32_16_CP 38
-#define R_AVR32_9W_CP 39
-#define R_AVR32_RELATIVE 40
-#define R_AVR32_GLOB_DAT 41
-#define R_AVR32_JMP_SLOT 42
-#define R_AVR32_ALIGN 43
-
-/*
- * ELF register definitions..
- */
-
-#include <asm/ptrace.h>
-#include <asm/user.h>
-
-typedef unsigned long elf_greg_t;
-
-#define ELF_NGREG (sizeof (struct pt_regs) / sizeof (elf_greg_t))
-typedef elf_greg_t elf_gregset_t[ELF_NGREG];
-
-typedef struct user_fpu_struct elf_fpregset_t;
-
-/*
- * This is used to ensure we don't load something for the wrong architecture.
- */
-#define elf_check_arch(x) ( (x)->e_machine == EM_AVR32 )
-
-/*
- * These are used to set parameters in the core dumps.
- */
-#define ELF_CLASS ELFCLASS32
-#ifdef __LITTLE_ENDIAN__
-#define ELF_DATA ELFDATA2LSB
-#else
-#define ELF_DATA ELFDATA2MSB
-#endif
-#define ELF_ARCH EM_AVR32
-
-#define ELF_EXEC_PAGESIZE 4096
-
-/* This is the location that an ET_DYN program is loaded if exec'ed. Typical
- use of this is to invoke "./ld.so someprog" to test out a new version of
- the loader. We need to make sure that it is out of the way of the program
- that it will "exec", and that there is sufficient room for the brk. */
-
-#define ELF_ET_DYN_BASE (TASK_SIZE / 3 * 2)
-
-
-/* This yields a mask that user programs can use to figure out what
- instruction set this CPU supports. This could be done in user space,
- but it's not easy, and we've already done it here. */
-
-#define ELF_HWCAP (0)
-
-/* This yields a string that ld.so will use to load implementation
- specific libraries for optimization. This is more specific in
- intent than poking at uname or /proc/cpuinfo.
-
- For the moment, we have only optimizations for the Intel generations,
- but that could change... */
-
-#define ELF_PLATFORM (NULL)
-
-#endif /* __ASM_AVR32_ELF_H */
diff --git a/arch/avr32/include/asm/fb.h b/arch/avr32/include/asm/fb.h
deleted file mode 100644
index 41baf84..0000000
--- a/arch/avr32/include/asm/fb.h
+++ /dev/null
@@ -1,21 +0,0 @@
-#ifndef _ASM_FB_H_
-#define _ASM_FB_H_
-
-#include <linux/fb.h>
-#include <linux/fs.h>
-#include <asm/page.h>
-
-static inline void fb_pgprotect(struct file *file, struct vm_area_struct *vma,
- unsigned long off)
-{
- vma->vm_page_prot = __pgprot((pgprot_val(vma->vm_page_prot)
- & ~_PAGE_CACHABLE)
- | (_PAGE_BUFFER | _PAGE_DIRTY));
-}
-
-static inline int fb_is_primary_device(struct fb_info *info)
-{
- return 0;
-}
-
-#endif /* _ASM_FB_H_ */
diff --git a/arch/avr32/include/asm/ftrace.h b/arch/avr32/include/asm/ftrace.h
deleted file mode 100644
index 40a8c17..0000000
--- a/arch/avr32/include/asm/ftrace.h
+++ /dev/null
@@ -1 +0,0 @@
-/* empty */
diff --git a/arch/avr32/include/asm/gpio.h b/arch/avr32/include/asm/gpio.h
deleted file mode 100644
index b771f71..0000000
--- a/arch/avr32/include/asm/gpio.h
+++ /dev/null
@@ -1,6 +0,0 @@
-#ifndef __ASM_AVR32_GPIO_H
-#define __ASM_AVR32_GPIO_H
-
-#include <mach/gpio.h>
-
-#endif /* __ASM_AVR32_GPIO_H */
diff --git a/arch/avr32/include/asm/hardirq.h b/arch/avr32/include/asm/hardirq.h
deleted file mode 100644
index 9e36e3f..0000000
--- a/arch/avr32/include/asm/hardirq.h
+++ /dev/null
@@ -1,6 +0,0 @@
-#ifndef __ASM_AVR32_HARDIRQ_H
-#define __ASM_AVR32_HARDIRQ_H
-#ifndef __ASSEMBLY__
-#include <asm-generic/hardirq.h>
-#endif /* __ASSEMBLY__ */
-#endif /* __ASM_AVR32_HARDIRQ_H */
diff --git a/arch/avr32/include/asm/hw_irq.h b/arch/avr32/include/asm/hw_irq.h
deleted file mode 100644
index a36f9fc..0000000
--- a/arch/avr32/include/asm/hw_irq.h
+++ /dev/null
@@ -1,9 +0,0 @@
-#ifndef __ASM_AVR32_HW_IRQ_H
-#define __ASM_AVR32_HW_IRQ_H
-
-static inline void hw_resend_irq(struct irq_chip *h, unsigned int i)
-{
- /* Nothing to do */
-}
-
-#endif /* __ASM_AVR32_HW_IRQ_H */
diff --git a/arch/avr32/include/asm/io.h b/arch/avr32/include/asm/io.h
deleted file mode 100644
index f855646..0000000
--- a/arch/avr32/include/asm/io.h
+++ /dev/null
@@ -1,329 +0,0 @@
-#ifndef __ASM_AVR32_IO_H
-#define __ASM_AVR32_IO_H
-
-#include <linux/bug.h>
-#include <linux/kernel.h>
-#include <linux/string.h>
-#include <linux/types.h>
-
-#include <asm/addrspace.h>
-#include <asm/byteorder.h>
-
-#include <mach/io.h>
-
-/* virt_to_phys will only work when address is in P1 or P2 */
-static __inline__ unsigned long virt_to_phys(volatile void *address)
-{
- return PHYSADDR(address);
-}
-
-static __inline__ void * phys_to_virt(unsigned long address)
-{
- return (void *)P1SEGADDR(address);
-}
-
-#define cached_to_phys(addr) ((unsigned long)PHYSADDR(addr))
-#define uncached_to_phys(addr) ((unsigned long)PHYSADDR(addr))
-#define phys_to_cached(addr) ((void *)P1SEGADDR(addr))
-#define phys_to_uncached(addr) ((void *)P2SEGADDR(addr))
-
-/*
- * Generic IO read/write. These perform native-endian accesses. Note
- * that some architectures will want to re-define __raw_{read,write}w.
- */
-extern void __raw_writesb(void __iomem *addr, const void *data, int bytelen);
-extern void __raw_writesw(void __iomem *addr, const void *data, int wordlen);
-extern void __raw_writesl(void __iomem *addr, const void *data, int longlen);
-
-extern void __raw_readsb(const void __iomem *addr, void *data, int bytelen);
-extern void __raw_readsw(const void __iomem *addr, void *data, int wordlen);
-extern void __raw_readsl(const void __iomem *addr, void *data, int longlen);
-
-static inline void __raw_writeb(u8 v, volatile void __iomem *addr)
-{
- *(volatile u8 __force *)addr = v;
-}
-static inline void __raw_writew(u16 v, volatile void __iomem *addr)
-{
- *(volatile u16 __force *)addr = v;
-}
-static inline void __raw_writel(u32 v, volatile void __iomem *addr)
-{
- *(volatile u32 __force *)addr = v;
-}
-
-static inline u8 __raw_readb(const volatile void __iomem *addr)
-{
- return *(const volatile u8 __force *)addr;
-}
-static inline u16 __raw_readw(const volatile void __iomem *addr)
-{
- return *(const volatile u16 __force *)addr;
-}
-static inline u32 __raw_readl(const volatile void __iomem *addr)
-{
- return *(const volatile u32 __force *)addr;
-}
-
-/* Convert I/O port address to virtual address */
-#ifndef __io
-# define __io(p) ((void *)phys_to_uncached(p))
-#endif
-
-/*
- * Not really sure about the best way to slow down I/O on
- * AVR32. Defining it as a no-op until we have an actual test case.
- */
-#define SLOW_DOWN_IO do { } while (0)
-
-#define __BUILD_MEMORY_SINGLE(pfx, bwl, type) \
-static inline void \
-pfx##write##bwl(type val, volatile void __iomem *addr) \
-{ \
- volatile type *__addr; \
- type __val; \
- \
- __addr = (void *)__swizzle_addr_##bwl((unsigned long)(addr)); \
- __val = pfx##ioswab##bwl(__addr, val); \
- \
- BUILD_BUG_ON(sizeof(type) > sizeof(unsigned long)); \
- \
- *__addr = __val; \
-} \
- \
-static inline type pfx##read##bwl(const volatile void __iomem *addr) \
-{ \
- volatile type *__addr; \
- type __val; \
- \
- __addr = (void *)__swizzle_addr_##bwl((unsigned long)(addr)); \
- \
- BUILD_BUG_ON(sizeof(type) > sizeof(unsigned long)); \
- \
- __val = *__addr; \
- return pfx##ioswab##bwl(__addr, __val); \
-}
-
-#define __BUILD_IOPORT_SINGLE(pfx, bwl, type, p, slow) \
-static inline void pfx##out##bwl##p(type val, unsigned long port) \
-{ \
- volatile type *__addr; \
- type __val; \
- \
- __addr = __io(__swizzle_addr_##bwl(port)); \
- __val = pfx##ioswab##bwl(__addr, val); \
- \
- BUILD_BUG_ON(sizeof(type) > sizeof(unsigned long)); \
- \
- *__addr = __val; \
- slow; \
-} \
- \
-static inline type pfx##in##bwl##p(unsigned long port) \
-{ \
- volatile type *__addr; \
- type __val; \
- \
- __addr = __io(__swizzle_addr_##bwl(port)); \
- \
- BUILD_BUG_ON(sizeof(type) > sizeof(unsigned long)); \
- \
- __val = *__addr; \
- slow; \
- \
- return pfx##ioswab##bwl(__addr, __val); \
-}
-
-#define __BUILD_MEMORY_PFX(bus, bwl, type) \
- __BUILD_MEMORY_SINGLE(bus, bwl, type)
-
-#define BUILDIO_MEM(bwl, type) \
- __BUILD_MEMORY_PFX(, bwl, type) \
- __BUILD_MEMORY_PFX(__mem_, bwl, type)
-
-#define __BUILD_IOPORT_PFX(bus, bwl, type) \
- __BUILD_IOPORT_SINGLE(bus, bwl, type, ,) \
- __BUILD_IOPORT_SINGLE(bus, bwl, type, _p, SLOW_DOWN_IO)
-
-#define BUILDIO_IOPORT(bwl, type) \
- __BUILD_IOPORT_PFX(, bwl, type) \
- __BUILD_IOPORT_PFX(__mem_, bwl, type)
-
-BUILDIO_MEM(b, u8)
-BUILDIO_MEM(w, u16)
-BUILDIO_MEM(l, u32)
-
-BUILDIO_IOPORT(b, u8)
-BUILDIO_IOPORT(w, u16)
-BUILDIO_IOPORT(l, u32)
-
-#define readb_relaxed readb
-#define readw_relaxed readw
-#define readl_relaxed readl
-
-#define readb_be __raw_readb
-#define readw_be __raw_readw
-#define readl_be __raw_readl
-
-#define writeb_relaxed writeb
-#define writew_relaxed writew
-#define writel_relaxed writel
-
-#define writeb_be __raw_writeb
-#define writew_be __raw_writew
-#define writel_be __raw_writel
-
-#define __BUILD_MEMORY_STRING(bwl, type) \
-static inline void writes##bwl(volatile void __iomem *addr, \
- const void *data, unsigned int count) \
-{ \
- const type *__data = data; \
- \
- while (count--) \
- __mem_write##bwl(*__data++, addr); \
-} \
- \
-static inline void reads##bwl(const volatile void __iomem *addr, \
- void *data, unsigned int count) \
-{ \
- type *__data = data; \
- \
- while (count--) \
- *__data++ = __mem_read##bwl(addr); \
-}
-
-#define __BUILD_IOPORT_STRING(bwl, type) \
-static inline void outs##bwl(unsigned long port, const void *data, \
- unsigned int count) \
-{ \
- const type *__data = data; \
- \
- while (count--) \
- __mem_out##bwl(*__data++, port); \
-} \
- \
-static inline void ins##bwl(unsigned long port, void *data, \
- unsigned int count) \
-{ \
- type *__data = data; \
- \
- while (count--) \
- *__data++ = __mem_in##bwl(port); \
-}
-
-#define BUILDSTRING(bwl, type) \
- __BUILD_MEMORY_STRING(bwl, type) \
- __BUILD_IOPORT_STRING(bwl, type)
-
-BUILDSTRING(b, u8)
-BUILDSTRING(w, u16)
-BUILDSTRING(l, u32)
-
-/*
- * io{read,write}{8,16,32} macros in both le (for PCI style consumers) and native be
- */
-#ifndef ioread8
-
-#define ioread8(p) ((unsigned int)readb(p))
-
-#define ioread16(p) ((unsigned int)readw(p))
-#define ioread16be(p) ((unsigned int)__raw_readw(p))
-
-#define ioread32(p) ((unsigned int)readl(p))
-#define ioread32be(p) ((unsigned int)__raw_readl(p))
-
-#define iowrite8(v,p) writeb(v, p)
-
-#define iowrite16(v,p) writew(v, p)
-#define iowrite16be(v,p) __raw_writew(v, p)
-
-#define iowrite32(v,p) writel(v, p)
-#define iowrite32be(v,p) __raw_writel(v, p)
-
-#define ioread8_rep(p,d,c) readsb(p,d,c)
-#define ioread16_rep(p,d,c) readsw(p,d,c)
-#define ioread32_rep(p,d,c) readsl(p,d,c)
-
-#define iowrite8_rep(p,s,c) writesb(p,s,c)
-#define iowrite16_rep(p,s,c) writesw(p,s,c)
-#define iowrite32_rep(p,s,c) writesl(p,s,c)
-
-#endif
-
-static inline void memcpy_fromio(void * to, const volatile void __iomem *from,
- unsigned long count)
-{
- memcpy(to, (const void __force *)from, count);
-}
-
-static inline void memcpy_toio(volatile void __iomem *to, const void * from,
- unsigned long count)
-{
- memcpy((void __force *)to, from, count);
-}
-
-static inline void memset_io(volatile void __iomem *addr, unsigned char val,
- unsigned long count)
-{
- memset((void __force *)addr, val, count);
-}
-
-#define mmiowb()
-
-#define IO_SPACE_LIMIT 0xffffffff
-
-extern void __iomem *__ioremap(unsigned long offset, size_t size,
- unsigned long flags);
-extern void __iounmap(void __iomem *addr);
-
-/*
- * ioremap - map bus memory into CPU space
- * @offset bus address of the memory
- * @size size of the resource to map
- *
- * ioremap performs a platform specific sequence of operations to make
- * bus memory CPU accessible via the readb/.../writel functions and
- * the other mmio helpers. The returned address is not guaranteed to
- * be usable directly as a virtual address.
- */
-#define ioremap(offset, size) \
- __ioremap((offset), (size), 0)
-
-#define ioremap_nocache(offset, size) \
- __ioremap((offset), (size), 0)
-
-#define iounmap(addr) \
- __iounmap(addr)
-
-#define ioremap_wc ioremap_nocache
-#define ioremap_wt ioremap_nocache
-#define ioremap_uc ioremap_nocache
-
-#define cached(addr) P1SEGADDR(addr)
-#define uncached(addr) P2SEGADDR(addr)
-
-#define virt_to_bus virt_to_phys
-#define bus_to_virt phys_to_virt
-#define page_to_bus page_to_phys
-#define bus_to_page phys_to_page
-
-/*
- * Create a virtual mapping cookie for an IO port range. There exists
- * no such thing as port-based I/O on AVR32, so a regular ioremap()
- * should do what we need.
- */
-#define ioport_map(port, nr) ioremap(port, nr)
-#define ioport_unmap(port) iounmap(port)
-
-/*
- * Convert a physical pointer to a virtual kernel pointer for /dev/mem
- * access
- */
-#define xlate_dev_mem_ptr(p) __va(p)
-
-/*
- * Convert a virtual cached pointer to an uncached pointer
- */
-#define xlate_dev_kmem_ptr(p) p
-
-#endif /* __ASM_AVR32_IO_H */
diff --git a/arch/avr32/include/asm/irq.h b/arch/avr32/include/asm/irq.h
deleted file mode 100644
index 6fa8913f..0000000
--- a/arch/avr32/include/asm/irq.h
+++ /dev/null
@@ -1,24 +0,0 @@
-#ifndef __ASM_AVR32_IRQ_H
-#define __ASM_AVR32_IRQ_H
-
-#define NR_INTERNAL_IRQS 64
-
-#include <mach/irq.h>
-
-#ifndef NR_IRQS
-#define NR_IRQS (NR_INTERNAL_IRQS)
-#endif
-
-#define irq_canonicalize(i) (i)
-
-#ifndef __ASSEMBLER__
-int nmi_enable(void);
-void nmi_disable(void);
-
-/*
- * Returns a bitmask of pending interrupts in a group.
- */
-extern unsigned long intc_get_pending(unsigned int group);
-#endif
-
-#endif /* __ASM_AVR32_IOCTLS_H */
diff --git a/arch/avr32/include/asm/irqflags.h b/arch/avr32/include/asm/irqflags.h
deleted file mode 100644
index 006e948..0000000
--- a/arch/avr32/include/asm/irqflags.h
+++ /dev/null
@@ -1,61 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_IRQFLAGS_H
-#define __ASM_AVR32_IRQFLAGS_H
-
-#include <linux/types.h>
-#include <asm/sysreg.h>
-
-static inline unsigned long arch_local_save_flags(void)
-{
- return sysreg_read(SR);
-}
-
-/*
- * This will restore ALL status register flags, not only the interrupt
- * mask flag.
- *
- * The empty asm statement informs the compiler of this fact while
- * also serving as a barrier.
- */
-static inline void arch_local_irq_restore(unsigned long flags)
-{
- sysreg_write(SR, flags);
- asm volatile("" : : : "memory", "cc");
-}
-
-static inline void arch_local_irq_disable(void)
-{
- asm volatile("ssrf %0" : : "n"(SYSREG_GM_OFFSET) : "memory");
-}
-
-static inline void arch_local_irq_enable(void)
-{
- asm volatile("csrf %0" : : "n"(SYSREG_GM_OFFSET) : "memory");
-}
-
-static inline bool arch_irqs_disabled_flags(unsigned long flags)
-{
- return (flags & SYSREG_BIT(GM)) != 0;
-}
-
-static inline bool arch_irqs_disabled(void)
-{
- return arch_irqs_disabled_flags(arch_local_save_flags());
-}
-
-static inline unsigned long arch_local_irq_save(void)
-{
- unsigned long flags = arch_local_save_flags();
-
- arch_local_irq_disable();
-
- return flags;
-}
-
-#endif /* __ASM_AVR32_IRQFLAGS_H */
diff --git a/arch/avr32/include/asm/kdebug.h b/arch/avr32/include/asm/kdebug.h
deleted file mode 100644
index f930ce2..0000000
--- a/arch/avr32/include/asm/kdebug.h
+++ /dev/null
@@ -1,12 +0,0 @@
-#ifndef __ASM_AVR32_KDEBUG_H
-#define __ASM_AVR32_KDEBUG_H
-
-/* Grossly misnamed. */
-enum die_val {
- DIE_BREAKPOINT,
- DIE_SSTEP,
- DIE_NMI,
- DIE_OOPS,
-};
-
-#endif /* __ASM_AVR32_KDEBUG_H */
diff --git a/arch/avr32/include/asm/kmap_types.h b/arch/avr32/include/asm/kmap_types.h
deleted file mode 100644
index 479330b..0000000
--- a/arch/avr32/include/asm/kmap_types.h
+++ /dev/null
@@ -1,10 +0,0 @@
-#ifndef __ASM_AVR32_KMAP_TYPES_H
-#define __ASM_AVR32_KMAP_TYPES_H
-
-#ifdef CONFIG_DEBUG_HIGHMEM
-# define KM_TYPE_NR 29
-#else
-# define KM_TYPE_NR 14
-#endif
-
-#endif /* __ASM_AVR32_KMAP_TYPES_H */
diff --git a/arch/avr32/include/asm/kprobes.h b/arch/avr32/include/asm/kprobes.h
deleted file mode 100644
index 28dfc61a..0000000
--- a/arch/avr32/include/asm/kprobes.h
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Kernel Probes (KProbes)
- *
- * Copyright (C) 2005-2006 Atmel Corporation
- * Copyright (C) IBM Corporation, 2002, 2004
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_KPROBES_H
-#define __ASM_AVR32_KPROBES_H
-
-#include <asm-generic/kprobes.h>
-
-#define BREAKPOINT_INSTRUCTION 0xd673 /* breakpoint */
-
-#ifdef CONFIG_KPROBES
-#include <linux/types.h>
-
-typedef u16 kprobe_opcode_t;
-#define MAX_INSN_SIZE 2
-#define MAX_STACK_SIZE 64 /* 32 would probably be OK */
-
-#define kretprobe_blacklist_size 0
-
-#define arch_remove_kprobe(p) do { } while (0)
-
-/* Architecture specific copy of original instruction */
-struct arch_specific_insn {
- kprobe_opcode_t insn[MAX_INSN_SIZE];
-};
-
-struct prev_kprobe {
- struct kprobe *kp;
- unsigned int status;
-};
-
-/* per-cpu kprobe control block */
-struct kprobe_ctlblk {
- unsigned int kprobe_status;
- struct prev_kprobe prev_kprobe;
- struct pt_regs jprobe_saved_regs;
- char jprobes_stack[MAX_STACK_SIZE];
-};
-
-extern int kprobe_fault_handler(struct pt_regs *regs, int trapnr);
-extern int kprobe_exceptions_notify(struct notifier_block *self,
- unsigned long val, void *data);
-
-#define flush_insn_slot(p) do { } while (0)
-
-#endif /* CONFIG_KPROBES */
-#endif /* __ASM_AVR32_KPROBES_H */
diff --git a/arch/avr32/include/asm/linkage.h b/arch/avr32/include/asm/linkage.h
deleted file mode 100644
index f7b285e..0000000
--- a/arch/avr32/include/asm/linkage.h
+++ /dev/null
@@ -1,7 +0,0 @@
-#ifndef __ASM_LINKAGE_H
-#define __ASM_LINKAGE_H
-
-#define __ALIGN .balign 2
-#define __ALIGN_STR ".balign 2"
-
-#endif /* __ASM_LINKAGE_H */
diff --git a/arch/avr32/include/asm/mmu.h b/arch/avr32/include/asm/mmu.h
deleted file mode 100644
index 60c2d26..0000000
--- a/arch/avr32/include/asm/mmu.h
+++ /dev/null
@@ -1,10 +0,0 @@
-#ifndef __ASM_AVR32_MMU_H
-#define __ASM_AVR32_MMU_H
-
-/* Default "unsigned long" context */
-typedef unsigned long mm_context_t;
-
-#define MMU_ITLB_ENTRIES 64
-#define MMU_DTLB_ENTRIES 64
-
-#endif /* __ASM_AVR32_MMU_H */
diff --git a/arch/avr32/include/asm/mmu_context.h b/arch/avr32/include/asm/mmu_context.h
deleted file mode 100644
index cd87abb..0000000
--- a/arch/avr32/include/asm/mmu_context.h
+++ /dev/null
@@ -1,150 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * ASID handling taken from SH implementation.
- * Copyright (C) 1999 Niibe Yutaka
- * Copyright (C) 2003 Paul Mundt
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_MMU_CONTEXT_H
-#define __ASM_AVR32_MMU_CONTEXT_H
-
-#include <linux/mm_types.h>
-
-#include <asm/tlbflush.h>
-#include <asm/sysreg.h>
-#include <asm-generic/mm_hooks.h>
-
-/*
- * The MMU "context" consists of two things:
- * (a) TLB cache version
- * (b) ASID (Address Space IDentifier)
- */
-#define MMU_CONTEXT_ASID_MASK 0x000000ff
-#define MMU_CONTEXT_VERSION_MASK 0xffffff00
-#define MMU_CONTEXT_FIRST_VERSION 0x00000100
-#define NO_CONTEXT 0
-
-#define MMU_NO_ASID 0x100
-
-/* Virtual Page Number mask */
-#define MMU_VPN_MASK 0xfffff000
-
-/* Cache of MMU context last used */
-extern unsigned long mmu_context_cache;
-
-/*
- * Get MMU context if needed
- */
-static inline void
-get_mmu_context(struct mm_struct *mm)
-{
- unsigned long mc = mmu_context_cache;
-
- if (((mm->context ^ mc) & MMU_CONTEXT_VERSION_MASK) == 0)
- /* It's up to date, do nothing */
- return;
-
- /* It's old, we need to get new context with new version */
- mc = ++mmu_context_cache;
- if (!(mc & MMU_CONTEXT_ASID_MASK)) {
- /*
- * We have exhausted all ASIDs of this version.
- * Flush the TLB and start new cycle.
- */
- flush_tlb_all();
- /*
- * Fix version. Note that we avoid version #0
- * to distinguish NO_CONTEXT.
- */
- if (!mc)
- mmu_context_cache = mc = MMU_CONTEXT_FIRST_VERSION;
- }
- mm->context = mc;
-}
-
-/*
- * Initialize the context related info for a new mm_struct
- * instance.
- */
-static inline int init_new_context(struct task_struct *tsk,
- struct mm_struct *mm)
-{
- mm->context = NO_CONTEXT;
- return 0;
-}
-
-/*
- * Destroy context related info for an mm_struct that is about
- * to be put to rest.
- */
-static inline void destroy_context(struct mm_struct *mm)
-{
- /* Do nothing */
-}
-
-static inline void set_asid(unsigned long asid)
-{
- /* XXX: We're destroying TLBEHI[8:31] */
- sysreg_write(TLBEHI, asid & MMU_CONTEXT_ASID_MASK);
- cpu_sync_pipeline();
-}
-
-static inline unsigned long get_asid(void)
-{
- unsigned long asid;
-
- asid = sysreg_read(TLBEHI);
- return asid & MMU_CONTEXT_ASID_MASK;
-}
-
-static inline void activate_context(struct mm_struct *mm)
-{
- get_mmu_context(mm);
- set_asid(mm->context & MMU_CONTEXT_ASID_MASK);
-}
-
-static inline void switch_mm(struct mm_struct *prev,
- struct mm_struct *next,
- struct task_struct *tsk)
-{
- if (likely(prev != next)) {
- unsigned long __pgdir = (unsigned long)next->pgd;
-
- sysreg_write(PTBR, __pgdir);
- activate_context(next);
- }
-}
-
-#define deactivate_mm(tsk,mm) do { } while(0)
-
-#define activate_mm(prev, next) switch_mm((prev), (next), NULL)
-
-static inline void
-enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)
-{
-}
-
-
-static inline void enable_mmu(void)
-{
- sysreg_write(MMUCR, (SYSREG_BIT(MMUCR_S)
- | SYSREG_BIT(E)
- | SYSREG_BIT(MMUCR_I)));
- nop(); nop(); nop(); nop(); nop(); nop(); nop(); nop();
-
- if (mmu_context_cache == NO_CONTEXT)
- mmu_context_cache = MMU_CONTEXT_FIRST_VERSION;
-
- set_asid(mmu_context_cache & MMU_CONTEXT_ASID_MASK);
-}
-
-static inline void disable_mmu(void)
-{
- sysreg_write(MMUCR, SYSREG_BIT(MMUCR_S));
-}
-
-#endif /* __ASM_AVR32_MMU_CONTEXT_H */
diff --git a/arch/avr32/include/asm/module.h b/arch/avr32/include/asm/module.h
deleted file mode 100644
index 3f083d3..0000000
--- a/arch/avr32/include/asm/module.h
+++ /dev/null
@@ -1,26 +0,0 @@
-#ifndef __ASM_AVR32_MODULE_H
-#define __ASM_AVR32_MODULE_H
-
-#include <asm-generic/module.h>
-
-struct mod_arch_syminfo {
- unsigned long got_offset;
- int got_initialized;
-};
-
-struct mod_arch_specific {
- /* Starting offset of got in the module core memory. */
- unsigned long got_offset;
- /* Size of the got. */
- unsigned long got_size;
- /* Number of symbols in syminfo. */
- int nsyms;
- /* Additional symbol information (got offsets). */
- struct mod_arch_syminfo *syminfo;
-};
-
-#define MODULE_PROC_FAMILY "AVR32v1"
-
-#define MODULE_ARCH_VERMAGIC MODULE_PROC_FAMILY
-
-#endif /* __ASM_AVR32_MODULE_H */
diff --git a/arch/avr32/include/asm/ocd.h b/arch/avr32/include/asm/ocd.h
deleted file mode 100644
index 6bef094..0000000
--- a/arch/avr32/include/asm/ocd.h
+++ /dev/null
@@ -1,543 +0,0 @@
-/*
- * AVR32 OCD Interface and register definitions
- *
- * Copyright (C) 2004-2007 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_OCD_H
-#define __ASM_AVR32_OCD_H
-
-/* OCD Register offsets. Abbreviations used below:
- *
- * BP Breakpoint
- * Comm Communication
- * DT Data Trace
- * PC Program Counter
- * PID Process ID
- * R/W Read/Write
- * WP Watchpoint
- */
-#define OCD_DID 0x0000 /* Device ID */
-#define OCD_DC 0x0008 /* Development Control */
-#define OCD_DS 0x0010 /* Development Status */
-#define OCD_RWCS 0x001c /* R/W Access Control */
-#define OCD_RWA 0x0024 /* R/W Access Address */
-#define OCD_RWD 0x0028 /* R/W Access Data */
-#define OCD_WT 0x002c /* Watchpoint Trigger */
-#define OCD_DTC 0x0034 /* Data Trace Control */
-#define OCD_DTSA0 0x0038 /* DT Start Addr Channel 0 */
-#define OCD_DTSA1 0x003c /* DT Start Addr Channel 1 */
-#define OCD_DTEA0 0x0048 /* DT End Addr Channel 0 */
-#define OCD_DTEA1 0x004c /* DT End Addr Channel 1 */
-#define OCD_BWC0A 0x0058 /* PC BP/WP Control 0A */
-#define OCD_BWC0B 0x005c /* PC BP/WP Control 0B */
-#define OCD_BWC1A 0x0060 /* PC BP/WP Control 1A */
-#define OCD_BWC1B 0x0064 /* PC BP/WP Control 1B */
-#define OCD_BWC2A 0x0068 /* PC BP/WP Control 2A */
-#define OCD_BWC2B 0x006c /* PC BP/WP Control 2B */
-#define OCD_BWC3A 0x0070 /* Data BP/WP Control 3A */
-#define OCD_BWC3B 0x0074 /* Data BP/WP Control 3B */
-#define OCD_BWA0A 0x0078 /* PC BP/WP Address 0A */
-#define OCD_BWA0B 0x007c /* PC BP/WP Address 0B */
-#define OCD_BWA1A 0x0080 /* PC BP/WP Address 1A */
-#define OCD_BWA1B 0x0084 /* PC BP/WP Address 1B */
-#define OCD_BWA2A 0x0088 /* PC BP/WP Address 2A */
-#define OCD_BWA2B 0x008c /* PC BP/WP Address 2B */
-#define OCD_BWA3A 0x0090 /* Data BP/WP Address 3A */
-#define OCD_BWA3B 0x0094 /* Data BP/WP Address 3B */
-#define OCD_NXCFG 0x0100 /* Nexus Configuration */
-#define OCD_DINST 0x0104 /* Debug Instruction */
-#define OCD_DPC 0x0108 /* Debug Program Counter */
-#define OCD_CPUCM 0x010c /* CPU Control Mask */
-#define OCD_DCCPU 0x0110 /* Debug Comm CPU */
-#define OCD_DCEMU 0x0114 /* Debug Comm Emulator */
-#define OCD_DCSR 0x0118 /* Debug Comm Status */
-#define OCD_PID 0x011c /* Ownership Trace PID */
-#define OCD_EPC0 0x0120 /* Event Pair Control 0 */
-#define OCD_EPC1 0x0124 /* Event Pair Control 1 */
-#define OCD_EPC2 0x0128 /* Event Pair Control 2 */
-#define OCD_EPC3 0x012c /* Event Pair Control 3 */
-#define OCD_AXC 0x0130 /* AUX port Control */
-
-/* Bits in DID */
-#define OCD_DID_MID_START 1
-#define OCD_DID_MID_SIZE 11
-#define OCD_DID_PN_START 12
-#define OCD_DID_PN_SIZE 16
-#define OCD_DID_RN_START 28
-#define OCD_DID_RN_SIZE 4
-
-/* Bits in DC */
-#define OCD_DC_TM_START 0
-#define OCD_DC_TM_SIZE 2
-#define OCD_DC_EIC_START 3
-#define OCD_DC_EIC_SIZE 2
-#define OCD_DC_OVC_START 5
-#define OCD_DC_OVC_SIZE 3
-#define OCD_DC_SS_BIT 8
-#define OCD_DC_DBR_BIT 12
-#define OCD_DC_DBE_BIT 13
-#define OCD_DC_EOS_START 20
-#define OCD_DC_EOS_SIZE 2
-#define OCD_DC_SQA_BIT 22
-#define OCD_DC_IRP_BIT 23
-#define OCD_DC_IFM_BIT 24
-#define OCD_DC_TOZ_BIT 25
-#define OCD_DC_TSR_BIT 26
-#define OCD_DC_RID_BIT 27
-#define OCD_DC_ORP_BIT 28
-#define OCD_DC_MM_BIT 29
-#define OCD_DC_RES_BIT 30
-#define OCD_DC_ABORT_BIT 31
-
-/* Bits in DS */
-#define OCD_DS_SSS_BIT 0
-#define OCD_DS_SWB_BIT 1
-#define OCD_DS_HWB_BIT 2
-#define OCD_DS_HWE_BIT 3
-#define OCD_DS_STP_BIT 4
-#define OCD_DS_DBS_BIT 5
-#define OCD_DS_BP_START 8
-#define OCD_DS_BP_SIZE 8
-#define OCD_DS_INC_BIT 24
-#define OCD_DS_BOZ_BIT 25
-#define OCD_DS_DBA_BIT 26
-#define OCD_DS_EXB_BIT 27
-#define OCD_DS_NTBF_BIT 28
-
-/* Bits in RWCS */
-#define OCD_RWCS_DV_BIT 0
-#define OCD_RWCS_ERR_BIT 1
-#define OCD_RWCS_CNT_START 2
-#define OCD_RWCS_CNT_SIZE 14
-#define OCD_RWCS_CRC_BIT 19
-#define OCD_RWCS_NTBC_START 20
-#define OCD_RWCS_NTBC_SIZE 2
-#define OCD_RWCS_NTE_BIT 22
-#define OCD_RWCS_NTAP_BIT 23
-#define OCD_RWCS_WRAPPED_BIT 24
-#define OCD_RWCS_CCTRL_START 25
-#define OCD_RWCS_CCTRL_SIZE 2
-#define OCD_RWCS_SZ_START 27
-#define OCD_RWCS_SZ_SIZE 3
-#define OCD_RWCS_RW_BIT 30
-#define OCD_RWCS_AC_BIT 31
-
-/* Bits in RWA */
-#define OCD_RWA_RWA_START 0
-#define OCD_RWA_RWA_SIZE 32
-
-/* Bits in RWD */
-#define OCD_RWD_RWD_START 0
-#define OCD_RWD_RWD_SIZE 32
-
-/* Bits in WT */
-#define OCD_WT_DTE_START 20
-#define OCD_WT_DTE_SIZE 3
-#define OCD_WT_DTS_START 23
-#define OCD_WT_DTS_SIZE 3
-#define OCD_WT_PTE_START 26
-#define OCD_WT_PTE_SIZE 3
-#define OCD_WT_PTS_START 29
-#define OCD_WT_PTS_SIZE 3
-
-/* Bits in DTC */
-#define OCD_DTC_T0WP_BIT 0
-#define OCD_DTC_T1WP_BIT 1
-#define OCD_DTC_ASID0EN_BIT 2
-#define OCD_DTC_ASID0_START 3
-#define OCD_DTC_ASID0_SIZE 8
-#define OCD_DTC_ASID1EN_BIT 11
-#define OCD_DTC_ASID1_START 12
-#define OCD_DTC_ASID1_SIZE 8
-#define OCD_DTC_RWT1_START 28
-#define OCD_DTC_RWT1_SIZE 2
-#define OCD_DTC_RWT0_START 30
-#define OCD_DTC_RWT0_SIZE 2
-
-/* Bits in DTSA0 */
-#define OCD_DTSA0_DTSA_START 0
-#define OCD_DTSA0_DTSA_SIZE 32
-
-/* Bits in DTSA1 */
-#define OCD_DTSA1_DTSA_START 0
-#define OCD_DTSA1_DTSA_SIZE 32
-
-/* Bits in DTEA0 */
-#define OCD_DTEA0_DTEA_START 0
-#define OCD_DTEA0_DTEA_SIZE 32
-
-/* Bits in DTEA1 */
-#define OCD_DTEA1_DTEA_START 0
-#define OCD_DTEA1_DTEA_SIZE 32
-
-/* Bits in BWC0A */
-#define OCD_BWC0A_ASIDEN_BIT 0
-#define OCD_BWC0A_ASID_START 1
-#define OCD_BWC0A_ASID_SIZE 8
-#define OCD_BWC0A_EOC_BIT 14
-#define OCD_BWC0A_AME_BIT 25
-#define OCD_BWC0A_BWE_START 30
-#define OCD_BWC0A_BWE_SIZE 2
-
-/* Bits in BWC0B */
-#define OCD_BWC0B_ASIDEN_BIT 0
-#define OCD_BWC0B_ASID_START 1
-#define OCD_BWC0B_ASID_SIZE 8
-#define OCD_BWC0B_EOC_BIT 14
-#define OCD_BWC0B_AME_BIT 25
-#define OCD_BWC0B_BWE_START 30
-#define OCD_BWC0B_BWE_SIZE 2
-
-/* Bits in BWC1A */
-#define OCD_BWC1A_ASIDEN_BIT 0
-#define OCD_BWC1A_ASID_START 1
-#define OCD_BWC1A_ASID_SIZE 8
-#define OCD_BWC1A_EOC_BIT 14
-#define OCD_BWC1A_AME_BIT 25
-#define OCD_BWC1A_BWE_START 30
-#define OCD_BWC1A_BWE_SIZE 2
-
-/* Bits in BWC1B */
-#define OCD_BWC1B_ASIDEN_BIT 0
-#define OCD_BWC1B_ASID_START 1
-#define OCD_BWC1B_ASID_SIZE 8
-#define OCD_BWC1B_EOC_BIT 14
-#define OCD_BWC1B_AME_BIT 25
-#define OCD_BWC1B_BWE_START 30
-#define OCD_BWC1B_BWE_SIZE 2
-
-/* Bits in BWC2A */
-#define OCD_BWC2A_ASIDEN_BIT 0
-#define OCD_BWC2A_ASID_START 1
-#define OCD_BWC2A_ASID_SIZE 8
-#define OCD_BWC2A_EOC_BIT 14
-#define OCD_BWC2A_AMB_START 20
-#define OCD_BWC2A_AMB_SIZE 5
-#define OCD_BWC2A_AME_BIT 25
-#define OCD_BWC2A_BWE_START 30
-#define OCD_BWC2A_BWE_SIZE 2
-
-/* Bits in BWC2B */
-#define OCD_BWC2B_ASIDEN_BIT 0
-#define OCD_BWC2B_ASID_START 1
-#define OCD_BWC2B_ASID_SIZE 8
-#define OCD_BWC2B_EOC_BIT 14
-#define OCD_BWC2B_AME_BIT 25
-#define OCD_BWC2B_BWE_START 30
-#define OCD_BWC2B_BWE_SIZE 2
-
-/* Bits in BWC3A */
-#define OCD_BWC3A_ASIDEN_BIT 0
-#define OCD_BWC3A_ASID_START 1
-#define OCD_BWC3A_ASID_SIZE 8
-#define OCD_BWC3A_SIZE_START 9
-#define OCD_BWC3A_SIZE_SIZE 3
-#define OCD_BWC3A_EOC_BIT 14
-#define OCD_BWC3A_BWO_START 16
-#define OCD_BWC3A_BWO_SIZE 2
-#define OCD_BWC3A_BME_START 20
-#define OCD_BWC3A_BME_SIZE 4
-#define OCD_BWC3A_BRW_START 28
-#define OCD_BWC3A_BRW_SIZE 2
-#define OCD_BWC3A_BWE_START 30
-#define OCD_BWC3A_BWE_SIZE 2
-
-/* Bits in BWC3B */
-#define OCD_BWC3B_ASIDEN_BIT 0
-#define OCD_BWC3B_ASID_START 1
-#define OCD_BWC3B_ASID_SIZE 8
-#define OCD_BWC3B_SIZE_START 9
-#define OCD_BWC3B_SIZE_SIZE 3
-#define OCD_BWC3B_EOC_BIT 14
-#define OCD_BWC3B_BWO_START 16
-#define OCD_BWC3B_BWO_SIZE 2
-#define OCD_BWC3B_BME_START 20
-#define OCD_BWC3B_BME_SIZE 4
-#define OCD_BWC3B_BRW_START 28
-#define OCD_BWC3B_BRW_SIZE 2
-#define OCD_BWC3B_BWE_START 30
-#define OCD_BWC3B_BWE_SIZE 2
-
-/* Bits in BWA0A */
-#define OCD_BWA0A_BWA_START 0
-#define OCD_BWA0A_BWA_SIZE 32
-
-/* Bits in BWA0B */
-#define OCD_BWA0B_BWA_START 0
-#define OCD_BWA0B_BWA_SIZE 32
-
-/* Bits in BWA1A */
-#define OCD_BWA1A_BWA_START 0
-#define OCD_BWA1A_BWA_SIZE 32
-
-/* Bits in BWA1B */
-#define OCD_BWA1B_BWA_START 0
-#define OCD_BWA1B_BWA_SIZE 32
-
-/* Bits in BWA2A */
-#define OCD_BWA2A_BWA_START 0
-#define OCD_BWA2A_BWA_SIZE 32
-
-/* Bits in BWA2B */
-#define OCD_BWA2B_BWA_START 0
-#define OCD_BWA2B_BWA_SIZE 32
-
-/* Bits in BWA3A */
-#define OCD_BWA3A_BWA_START 0
-#define OCD_BWA3A_BWA_SIZE 32
-
-/* Bits in BWA3B */
-#define OCD_BWA3B_BWA_START 0
-#define OCD_BWA3B_BWA_SIZE 32
-
-/* Bits in NXCFG */
-#define OCD_NXCFG_NXARCH_START 0
-#define OCD_NXCFG_NXARCH_SIZE 4
-#define OCD_NXCFG_NXOCD_START 4
-#define OCD_NXCFG_NXOCD_SIZE 4
-#define OCD_NXCFG_NXPCB_START 8
-#define OCD_NXCFG_NXPCB_SIZE 4
-#define OCD_NXCFG_NXDB_START 12
-#define OCD_NXCFG_NXDB_SIZE 4
-#define OCD_NXCFG_MXMSEO_BIT 16
-#define OCD_NXCFG_NXMDO_START 17
-#define OCD_NXCFG_NXMDO_SIZE 4
-#define OCD_NXCFG_NXPT_BIT 21
-#define OCD_NXCFG_NXOT_BIT 22
-#define OCD_NXCFG_NXDWT_BIT 23
-#define OCD_NXCFG_NXDRT_BIT 24
-#define OCD_NXCFG_NXDTC_START 25
-#define OCD_NXCFG_NXDTC_SIZE 3
-#define OCD_NXCFG_NXDMA_BIT 28
-
-/* Bits in DINST */
-#define OCD_DINST_DINST_START 0
-#define OCD_DINST_DINST_SIZE 32
-
-/* Bits in CPUCM */
-#define OCD_CPUCM_BEM_BIT 1
-#define OCD_CPUCM_FEM_BIT 2
-#define OCD_CPUCM_REM_BIT 3
-#define OCD_CPUCM_IBEM_BIT 4
-#define OCD_CPUCM_IEEM_BIT 5
-
-/* Bits in DCCPU */
-#define OCD_DCCPU_DATA_START 0
-#define OCD_DCCPU_DATA_SIZE 32
-
-/* Bits in DCEMU */
-#define OCD_DCEMU_DATA_START 0
-#define OCD_DCEMU_DATA_SIZE 32
-
-/* Bits in DCSR */
-#define OCD_DCSR_CPUD_BIT 0
-#define OCD_DCSR_EMUD_BIT 1
-
-/* Bits in PID */
-#define OCD_PID_PROCESS_START 0
-#define OCD_PID_PROCESS_SIZE 32
-
-/* Bits in EPC0 */
-#define OCD_EPC0_RNG_START 0
-#define OCD_EPC0_RNG_SIZE 2
-#define OCD_EPC0_CE_BIT 4
-#define OCD_EPC0_ECNT_START 16
-#define OCD_EPC0_ECNT_SIZE 16
-
-/* Bits in EPC1 */
-#define OCD_EPC1_RNG_START 0
-#define OCD_EPC1_RNG_SIZE 2
-#define OCD_EPC1_ATB_BIT 5
-#define OCD_EPC1_AM_BIT 6
-
-/* Bits in EPC2 */
-#define OCD_EPC2_RNG_START 0
-#define OCD_EPC2_RNG_SIZE 2
-#define OCD_EPC2_DB_START 2
-#define OCD_EPC2_DB_SIZE 2
-
-/* Bits in EPC3 */
-#define OCD_EPC3_RNG_START 0
-#define OCD_EPC3_RNG_SIZE 2
-#define OCD_EPC3_DWE_BIT 2
-
-/* Bits in AXC */
-#define OCD_AXC_DIV_START 0
-#define OCD_AXC_DIV_SIZE 4
-#define OCD_AXC_AXE_BIT 8
-#define OCD_AXC_AXS_BIT 9
-#define OCD_AXC_DDR_BIT 10
-#define OCD_AXC_LS_BIT 11
-#define OCD_AXC_REX_BIT 12
-#define OCD_AXC_REXTEN_BIT 13
-
-/* Constants for DC:EIC */
-#define OCD_EIC_PROGRAM_AND_DATA_TRACE 0
-#define OCD_EIC_BREAKPOINT 1
-#define OCD_EIC_NOP 2
-
-/* Constants for DC:OVC */
-#define OCD_OVC_OVERRUN 0
-#define OCD_OVC_DELAY_CPU_BTM 1
-#define OCD_OVC_DELAY_CPU_DTM 2
-#define OCD_OVC_DELAY_CPU_BTM_DTM 3
-
-/* Constants for DC:EOS */
-#define OCD_EOS_NOP 0
-#define OCD_EOS_DEBUG_MODE 1
-#define OCD_EOS_BREAKPOINT_WATCHPOINT 2
-#define OCD_EOS_THQ 3
-
-/* Constants for RWCS:NTBC */
-#define OCD_NTBC_OVERWRITE 0
-#define OCD_NTBC_DISABLE 1
-#define OCD_NTBC_BREAKPOINT 2
-
-/* Constants for RWCS:CCTRL */
-#define OCD_CCTRL_AUTO 0
-#define OCD_CCTRL_CACHED 1
-#define OCD_CCTRL_UNCACHED 2
-
-/* Constants for RWCS:SZ */
-#define OCD_SZ_BYTE 0
-#define OCD_SZ_HALFWORD 1
-#define OCD_SZ_WORD 2
-
-/* Constants for WT:PTS */
-#define OCD_PTS_DISABLED 0
-#define OCD_PTS_PROGRAM_0B 1
-#define OCD_PTS_PROGRAM_1A 2
-#define OCD_PTS_PROGRAM_1B 3
-#define OCD_PTS_PROGRAM_2A 4
-#define OCD_PTS_PROGRAM_2B 5
-#define OCD_PTS_DATA_3A 6
-#define OCD_PTS_DATA_3B 7
-
-/* Constants for DTC:RWT1 */
-#define OCD_RWT1_NO_TRACE 0
-#define OCD_RWT1_DATA_READ 1
-#define OCD_RWT1_DATA_WRITE 2
-#define OCD_RWT1_DATA_READ_WRITE 3
-
-/* Constants for DTC:RWT0 */
-#define OCD_RWT0_NO_TRACE 0
-#define OCD_RWT0_DATA_READ 1
-#define OCD_RWT0_DATA_WRITE 2
-#define OCD_RWT0_DATA_READ_WRITE 3
-
-/* Constants for BWC0A:BWE */
-#define OCD_BWE_DISABLED 0
-#define OCD_BWE_BREAKPOINT_ENABLED 1
-#define OCD_BWE_WATCHPOINT_ENABLED 3
-
-/* Constants for BWC0B:BWE */
-#define OCD_BWE_DISABLED 0
-#define OCD_BWE_BREAKPOINT_ENABLED 1
-#define OCD_BWE_WATCHPOINT_ENABLED 3
-
-/* Constants for BWC1A:BWE */
-#define OCD_BWE_DISABLED 0
-#define OCD_BWE_BREAKPOINT_ENABLED 1
-#define OCD_BWE_WATCHPOINT_ENABLED 3
-
-/* Constants for BWC1B:BWE */
-#define OCD_BWE_DISABLED 0
-#define OCD_BWE_BREAKPOINT_ENABLED 1
-#define OCD_BWE_WATCHPOINT_ENABLED 3
-
-/* Constants for BWC2A:BWE */
-#define OCD_BWE_DISABLED 0
-#define OCD_BWE_BREAKPOINT_ENABLED 1
-#define OCD_BWE_WATCHPOINT_ENABLED 3
-
-/* Constants for BWC2B:BWE */
-#define OCD_BWE_DISABLED 0
-#define OCD_BWE_BREAKPOINT_ENABLED 1
-#define OCD_BWE_WATCHPOINT_ENABLED 3
-
-/* Constants for BWC3A:SIZE */
-#define OCD_SIZE_BYTE_ACCESS 4
-#define OCD_SIZE_HALFWORD_ACCESS 5
-#define OCD_SIZE_WORD_ACCESS 6
-#define OCD_SIZE_DOUBLE_WORD_ACCESS 7
-
-/* Constants for BWC3A:BRW */
-#define OCD_BRW_READ_BREAK 0
-#define OCD_BRW_WRITE_BREAK 1
-#define OCD_BRW_ANY_ACCES_BREAK 2
-
-/* Constants for BWC3A:BWE */
-#define OCD_BWE_DISABLED 0
-#define OCD_BWE_BREAKPOINT_ENABLED 1
-#define OCD_BWE_WATCHPOINT_ENABLED 3
-
-/* Constants for BWC3B:SIZE */
-#define OCD_SIZE_BYTE_ACCESS 4
-#define OCD_SIZE_HALFWORD_ACCESS 5
-#define OCD_SIZE_WORD_ACCESS 6
-#define OCD_SIZE_DOUBLE_WORD_ACCESS 7
-
-/* Constants for BWC3B:BRW */
-#define OCD_BRW_READ_BREAK 0
-#define OCD_BRW_WRITE_BREAK 1
-#define OCD_BRW_ANY_ACCES_BREAK 2
-
-/* Constants for BWC3B:BWE */
-#define OCD_BWE_DISABLED 0
-#define OCD_BWE_BREAKPOINT_ENABLED 1
-#define OCD_BWE_WATCHPOINT_ENABLED 3
-
-/* Constants for EPC0:RNG */
-#define OCD_RNG_DISABLED 0
-#define OCD_RNG_EXCLUSIVE 1
-#define OCD_RNG_INCLUSIVE 2
-
-/* Constants for EPC1:RNG */
-#define OCD_RNG_DISABLED 0
-#define OCD_RNG_EXCLUSIVE 1
-#define OCD_RNG_INCLUSIVE 2
-
-/* Constants for EPC2:RNG */
-#define OCD_RNG_DISABLED 0
-#define OCD_RNG_EXCLUSIVE 1
-#define OCD_RNG_INCLUSIVE 2
-
-/* Constants for EPC2:DB */
-#define OCD_DB_DISABLED 0
-#define OCD_DB_CHAINED_B 1
-#define OCD_DB_CHAINED_A 2
-#define OCD_DB_AHAINED_A_AND_B 3
-
-/* Constants for EPC3:RNG */
-#define OCD_RNG_DISABLED 0
-#define OCD_RNG_EXCLUSIVE 1
-#define OCD_RNG_INCLUSIVE 2
-
-#ifndef __ASSEMBLER__
-
-/* Register access macros */
-static inline unsigned long __ocd_read(unsigned int reg)
-{
- return __builtin_mfdr(reg);
-}
-
-static inline void __ocd_write(unsigned int reg, unsigned long value)
-{
- __builtin_mtdr(reg, value);
-}
-
-#define ocd_read(reg) __ocd_read(OCD_##reg)
-#define ocd_write(reg, value) __ocd_write(OCD_##reg, value)
-
-struct task_struct;
-
-void ocd_enable(struct task_struct *child);
-void ocd_disable(struct task_struct *child);
-
-#endif /* !__ASSEMBLER__ */
-
-#endif /* __ASM_AVR32_OCD_H */
diff --git a/arch/avr32/include/asm/page.h b/arch/avr32/include/asm/page.h
deleted file mode 100644
index c5d2a3e..0000000
--- a/arch/avr32/include/asm/page.h
+++ /dev/null
@@ -1,104 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_PAGE_H
-#define __ASM_AVR32_PAGE_H
-
-#include <linux/const.h>
-
-/* PAGE_SHIFT determines the page size */
-#define PAGE_SHIFT 12
-#define PAGE_SIZE (_AC(1, UL) << PAGE_SHIFT)
-#define PAGE_MASK (~(PAGE_SIZE-1))
-#define PTE_MASK PAGE_MASK
-
-#ifndef __ASSEMBLY__
-
-#include <asm/addrspace.h>
-
-extern void clear_page(void *to);
-extern void copy_page(void *to, void *from);
-
-#define clear_user_page(page, vaddr, pg) clear_page(page)
-#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
-
-/*
- * These are used to make use of C type-checking..
- */
-typedef struct { unsigned long pte; } pte_t;
-typedef struct { unsigned long pgd; } pgd_t;
-typedef struct { unsigned long pgprot; } pgprot_t;
-typedef struct page *pgtable_t;
-
-#define pte_val(x) ((x).pte)
-#define pgd_val(x) ((x).pgd)
-#define pgprot_val(x) ((x).pgprot)
-
-#define __pte(x) ((pte_t) { (x) })
-#define __pgd(x) ((pgd_t) { (x) })
-#define __pgprot(x) ((pgprot_t) { (x) })
-
-/* FIXME: These should be removed soon */
-extern unsigned long memory_start, memory_end;
-
-/* Pure 2^n version of get_order */
-static inline int get_order(unsigned long size)
-{
- unsigned lz;
-
- size = (size - 1) >> PAGE_SHIFT;
- asm("clz %0, %1" : "=r"(lz) : "r"(size));
- return 32 - lz;
-}
-
-#endif /* !__ASSEMBLY__ */
-
-/*
- * The hardware maps the virtual addresses 0x80000000 -> 0x9fffffff
- * permanently to the physical addresses 0x00000000 -> 0x1fffffff when
- * segmentation is enabled. We want to make use of this in order to
- * minimize TLB pressure.
- */
-#define PAGE_OFFSET (0x80000000UL)
-
-/*
- * ALSA uses virt_to_page() on DMA pages, which I'm not entirely sure
- * is a good idea. Anyway, we can't simply subtract PAGE_OFFSET here
- * in that case, so we'll have to mask out the three most significant
- * bits of the address instead...
- *
- * What's the difference between __pa() and virt_to_phys() anyway?
- */
-#define __pa(x) PHYSADDR(x)
-#define __va(x) ((void *)(P1SEGADDR(x)))
-
-#define MAP_NR(addr) (((unsigned long)(addr) - PAGE_OFFSET) >> PAGE_SHIFT)
-
-#define phys_to_page(phys) (pfn_to_page(phys >> PAGE_SHIFT))
-#define page_to_phys(page) (page_to_pfn(page) << PAGE_SHIFT)
-
-#ifndef CONFIG_NEED_MULTIPLE_NODES
-
-#define ARCH_PFN_OFFSET (CONFIG_PHYS_OFFSET >> PAGE_SHIFT)
-
-#define pfn_valid(pfn) ((pfn) >= ARCH_PFN_OFFSET && (pfn) < (ARCH_PFN_OFFSET + max_mapnr))
-#endif /* CONFIG_NEED_MULTIPLE_NODES */
-
-#define virt_to_page(kaddr) pfn_to_page(__pa(kaddr) >> PAGE_SHIFT)
-#define virt_addr_valid(kaddr) pfn_valid(__pa(kaddr) >> PAGE_SHIFT)
-
-#define VM_DATA_DEFAULT_FLAGS (VM_READ | VM_WRITE | \
- VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC)
-
-/*
- * Memory above this physical address will be considered highmem.
- */
-#define HIGHMEM_START 0x20000000UL
-
-#include <asm-generic/memory_model.h>
-
-#endif /* __ASM_AVR32_PAGE_H */
diff --git a/arch/avr32/include/asm/pci.h b/arch/avr32/include/asm/pci.h
deleted file mode 100644
index 0f5f134..0000000
--- a/arch/avr32/include/asm/pci.h
+++ /dev/null
@@ -1,8 +0,0 @@
-#ifndef __ASM_AVR32_PCI_H__
-#define __ASM_AVR32_PCI_H__
-
-/* We don't support PCI yet, but some drivers require this file anyway */
-
-#define PCI_DMA_BUS_IS_PHYS (1)
-
-#endif /* __ASM_AVR32_PCI_H__ */
diff --git a/arch/avr32/include/asm/pgalloc.h b/arch/avr32/include/asm/pgalloc.h
deleted file mode 100644
index db039cb..0000000
--- a/arch/avr32/include/asm/pgalloc.h
+++ /dev/null
@@ -1,102 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_PGALLOC_H
-#define __ASM_AVR32_PGALLOC_H
-
-#include <linux/mm.h>
-#include <linux/quicklist.h>
-#include <asm/page.h>
-#include <asm/pgtable.h>
-
-#define QUICK_PGD 0 /* Preserve kernel mappings over free */
-#define QUICK_PT 1 /* Zero on free */
-
-static inline void pmd_populate_kernel(struct mm_struct *mm,
- pmd_t *pmd, pte_t *pte)
-{
- set_pmd(pmd, __pmd((unsigned long)pte));
-}
-
-static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,
- pgtable_t pte)
-{
- set_pmd(pmd, __pmd((unsigned long)page_address(pte)));
-}
-#define pmd_pgtable(pmd) pmd_page(pmd)
-
-static inline void pgd_ctor(void *x)
-{
- pgd_t *pgd = x;
-
- memcpy(pgd + USER_PTRS_PER_PGD,
- swapper_pg_dir + USER_PTRS_PER_PGD,
- (PTRS_PER_PGD - USER_PTRS_PER_PGD) * sizeof(pgd_t));
-}
-
-/*
- * Allocate and free page tables
- */
-static inline pgd_t *pgd_alloc(struct mm_struct *mm)
-{
- return quicklist_alloc(QUICK_PGD, GFP_KERNEL, pgd_ctor);
-}
-
-static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
-{
- quicklist_free(QUICK_PGD, NULL, pgd);
-}
-
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
-{
- return quicklist_alloc(QUICK_PT, GFP_KERNEL, NULL);
-}
-
-static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
-{
- struct page *page;
- void *pg;
-
- pg = quicklist_alloc(QUICK_PT, GFP_KERNEL, NULL);
- if (!pg)
- return NULL;
-
- page = virt_to_page(pg);
- if (!pgtable_page_ctor(page)) {
- quicklist_free(QUICK_PT, NULL, pg);
- return NULL;
- }
-
- return page;
-}
-
-static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
-{
- quicklist_free(QUICK_PT, NULL, pte);
-}
-
-static inline void pte_free(struct mm_struct *mm, pgtable_t pte)
-{
- pgtable_page_dtor(pte);
- quicklist_free_page(QUICK_PT, NULL, pte);
-}
-
-#define __pte_free_tlb(tlb,pte,addr) \
-do { \
- pgtable_page_dtor(pte); \
- tlb_remove_page((tlb), pte); \
-} while (0)
-
-static inline void check_pgt_cache(void)
-{
- quicklist_trim(QUICK_PGD, NULL, 25, 16);
- quicklist_trim(QUICK_PT, NULL, 25, 16);
-}
-
-#endif /* __ASM_AVR32_PGALLOC_H */
diff --git a/arch/avr32/include/asm/pgtable-2level.h b/arch/avr32/include/asm/pgtable-2level.h
deleted file mode 100644
index d5b1c63..0000000
--- a/arch/avr32/include/asm/pgtable-2level.h
+++ /dev/null
@@ -1,48 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_PGTABLE_2LEVEL_H
-#define __ASM_AVR32_PGTABLE_2LEVEL_H
-
-#define __ARCH_USE_5LEVEL_HACK
-#include <asm-generic/pgtable-nopmd.h>
-
-/*
- * Traditional 2-level paging structure
- */
-#define PGDIR_SHIFT 22
-#define PTRS_PER_PGD 1024
-
-#define PTRS_PER_PTE 1024
-
-#ifndef __ASSEMBLY__
-#define pte_ERROR(e) \
- printk("%s:%d: bad pte %08lx.\n", __FILE__, __LINE__, pte_val(e))
-#define pgd_ERROR(e) \
- printk("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
-
-/*
- * Certain architectures need to do special things when PTEs
- * within a page table are directly modified. Thus, the following
- * hook is made available.
- */
-#define set_pte(pteptr, pteval) (*(pteptr) = pteval)
-#define set_pte_at(mm,addr,ptep,pteval) set_pte(ptep, pteval)
-
-/*
- * (pmds are folded into pgds so this doesn't get actually called,
- * but the define is needed for a generic inline function.)
- */
-#define set_pmd(pmdptr, pmdval) (*(pmdptr) = pmdval)
-
-#define pte_pfn(x) ((unsigned long)(((x).pte >> PAGE_SHIFT)))
-#define pfn_pte(pfn, prot) __pte(((pfn) << PAGE_SHIFT) | pgprot_val(prot))
-#define pfn_pmd(pfn, prot) __pmd(((pfn) << PAGE_SHIFT) | pgprot_val(prot))
-
-#endif /* !__ASSEMBLY__ */
-
-#endif /* __ASM_AVR32_PGTABLE_2LEVEL_H */
diff --git a/arch/avr32/include/asm/pgtable.h b/arch/avr32/include/asm/pgtable.h
deleted file mode 100644
index 3580066..0000000
--- a/arch/avr32/include/asm/pgtable.h
+++ /dev/null
@@ -1,347 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_PGTABLE_H
-#define __ASM_AVR32_PGTABLE_H
-
-#include <asm/addrspace.h>
-
-#ifndef __ASSEMBLY__
-#include <linux/sched.h>
-
-#endif /* !__ASSEMBLY__ */
-
-/*
- * Use two-level page tables just as the i386 (without PAE)
- */
-#include <asm/pgtable-2level.h>
-
-/*
- * The following code might need some cleanup when the values are
- * final...
- */
-#define PMD_SIZE (1UL << PMD_SHIFT)
-#define PMD_MASK (~(PMD_SIZE-1))
-#define PGDIR_SIZE (1UL << PGDIR_SHIFT)
-#define PGDIR_MASK (~(PGDIR_SIZE-1))
-
-#define USER_PTRS_PER_PGD (TASK_SIZE / PGDIR_SIZE)
-#define FIRST_USER_ADDRESS 0UL
-
-#ifndef __ASSEMBLY__
-extern pgd_t swapper_pg_dir[PTRS_PER_PGD];
-extern void paging_init(void);
-
-/*
- * ZERO_PAGE is a global shared page that is always zero: used for
- * zero-mapped memory areas etc.
- */
-extern struct page *empty_zero_page;
-#define ZERO_PAGE(vaddr) (empty_zero_page)
-
-/*
- * Just any arbitrary offset to the start of the vmalloc VM area: the
- * current 8 MiB value just means that there will be a 8 MiB "hole"
- * after the uncached physical memory (P2 segment) until the vmalloc
- * area starts. That means that any out-of-bounds memory accesses will
- * hopefully be caught; we don't know if the end of the P1/P2 segments
- * are actually used for anything, but it is anyway safer to let the
- * MMU catch these kinds of errors than to rely on the memory bus.
- *
- * A "hole" of the same size is added to the end of the P3 segment as
- * well. It might seem wasteful to use 16 MiB of virtual address space
- * on this, but we do have 512 MiB of it...
- *
- * The vmalloc() routines leave a hole of 4 KiB between each vmalloced
- * area for the same reason.
- */
-#define VMALLOC_OFFSET (8 * 1024 * 1024)
-#define VMALLOC_START (P3SEG + VMALLOC_OFFSET)
-#define VMALLOC_END (P4SEG - VMALLOC_OFFSET)
-#endif /* !__ASSEMBLY__ */
-
-/*
- * Page flags. Some of these flags are not directly supported by
- * hardware, so we have to emulate them.
- */
-#define _TLBEHI_BIT_VALID 9
-#define _TLBEHI_VALID (1 << _TLBEHI_BIT_VALID)
-
-#define _PAGE_BIT_WT 0 /* W-bit : write-through */
-#define _PAGE_BIT_DIRTY 1 /* D-bit : page changed */
-#define _PAGE_BIT_SZ0 2 /* SZ0-bit : Size of page */
-#define _PAGE_BIT_SZ1 3 /* SZ1-bit : Size of page */
-#define _PAGE_BIT_EXECUTE 4 /* X-bit : execute access allowed */
-#define _PAGE_BIT_RW 5 /* AP0-bit : write access allowed */
-#define _PAGE_BIT_USER 6 /* AP1-bit : user space access allowed */
-#define _PAGE_BIT_BUFFER 7 /* B-bit : bufferable */
-#define _PAGE_BIT_GLOBAL 8 /* G-bit : global (ignore ASID) */
-#define _PAGE_BIT_CACHABLE 9 /* C-bit : cachable */
-
-/* If we drop support for 1K pages, we get two extra bits */
-#define _PAGE_BIT_PRESENT 10
-#define _PAGE_BIT_ACCESSED 11 /* software: page was accessed */
-
-#define _PAGE_WT (1 << _PAGE_BIT_WT)
-#define _PAGE_DIRTY (1 << _PAGE_BIT_DIRTY)
-#define _PAGE_EXECUTE (1 << _PAGE_BIT_EXECUTE)
-#define _PAGE_RW (1 << _PAGE_BIT_RW)
-#define _PAGE_USER (1 << _PAGE_BIT_USER)
-#define _PAGE_BUFFER (1 << _PAGE_BIT_BUFFER)
-#define _PAGE_GLOBAL (1 << _PAGE_BIT_GLOBAL)
-#define _PAGE_CACHABLE (1 << _PAGE_BIT_CACHABLE)
-
-/* Software flags */
-#define _PAGE_ACCESSED (1 << _PAGE_BIT_ACCESSED)
-#define _PAGE_PRESENT (1 << _PAGE_BIT_PRESENT)
-
-/*
- * Page types, i.e. sizes. _PAGE_TYPE_NONE corresponds to what is
- * usually called _PAGE_PROTNONE on other architectures.
- *
- * XXX: Find out if _PAGE_PROTNONE is equivalent with !_PAGE_USER. If
- * so, we can encode all possible page sizes (although we can't really
- * support 1K pages anyway due to the _PAGE_PRESENT and _PAGE_ACCESSED
- * bits)
- *
- */
-#define _PAGE_TYPE_MASK ((1 << _PAGE_BIT_SZ0) | (1 << _PAGE_BIT_SZ1))
-#define _PAGE_TYPE_NONE (0 << _PAGE_BIT_SZ0)
-#define _PAGE_TYPE_SMALL (1 << _PAGE_BIT_SZ0)
-#define _PAGE_TYPE_MEDIUM (2 << _PAGE_BIT_SZ0)
-#define _PAGE_TYPE_LARGE (3 << _PAGE_BIT_SZ0)
-
-/*
- * Mask which drop software flags. We currently can't handle more than
- * 512 MiB of physical memory, so we can use bits 29-31 for other
- * stuff. With a fixed 4K page size, we can use bits 10-11 as well as
- * bits 2-3 (SZ)
- */
-#define _PAGE_FLAGS_HARDWARE_MASK 0xfffff3ff
-
-#define _PAGE_FLAGS_CACHE_MASK (_PAGE_CACHABLE | _PAGE_BUFFER | _PAGE_WT)
-
-/* Flags that may be modified by software */
-#define _PAGE_CHG_MASK (PTE_MASK | _PAGE_ACCESSED | _PAGE_DIRTY \
- | _PAGE_FLAGS_CACHE_MASK)
-
-#define _PAGE_FLAGS_READ (_PAGE_CACHABLE | _PAGE_BUFFER)
-#define _PAGE_FLAGS_WRITE (_PAGE_FLAGS_READ | _PAGE_RW | _PAGE_DIRTY)
-
-#define _PAGE_NORMAL(x) __pgprot((x) | _PAGE_PRESENT | _PAGE_TYPE_SMALL \
- | _PAGE_ACCESSED)
-
-#define PAGE_NONE (_PAGE_ACCESSED | _PAGE_TYPE_NONE)
-#define PAGE_READ (_PAGE_FLAGS_READ | _PAGE_USER)
-#define PAGE_EXEC (_PAGE_FLAGS_READ | _PAGE_EXECUTE | _PAGE_USER)
-#define PAGE_WRITE (_PAGE_FLAGS_WRITE | _PAGE_USER)
-#define PAGE_KERNEL _PAGE_NORMAL(_PAGE_FLAGS_WRITE | _PAGE_EXECUTE | _PAGE_GLOBAL)
-#define PAGE_KERNEL_RO _PAGE_NORMAL(_PAGE_FLAGS_READ | _PAGE_EXECUTE | _PAGE_GLOBAL)
-
-#define _PAGE_P(x) _PAGE_NORMAL((x) & ~(_PAGE_RW | _PAGE_DIRTY))
-#define _PAGE_S(x) _PAGE_NORMAL(x)
-
-#define PAGE_COPY _PAGE_P(PAGE_WRITE | PAGE_READ)
-#define PAGE_SHARED _PAGE_S(PAGE_WRITE | PAGE_READ)
-
-#ifndef __ASSEMBLY__
-/*
- * The hardware supports flags for write- and execute access. Read is
- * always allowed if the page is loaded into the TLB, so the "-w-",
- * "--x" and "-wx" mappings are implemented as "rw-", "r-x" and "rwx",
- * respectively.
- *
- * The "---" case is handled by software; the page will simply not be
- * loaded into the TLB if the page type is _PAGE_TYPE_NONE.
- */
-
-#define __P000 __pgprot(PAGE_NONE)
-#define __P001 _PAGE_P(PAGE_READ)
-#define __P010 _PAGE_P(PAGE_WRITE)
-#define __P011 _PAGE_P(PAGE_WRITE | PAGE_READ)
-#define __P100 _PAGE_P(PAGE_EXEC)
-#define __P101 _PAGE_P(PAGE_EXEC | PAGE_READ)
-#define __P110 _PAGE_P(PAGE_EXEC | PAGE_WRITE)
-#define __P111 _PAGE_P(PAGE_EXEC | PAGE_WRITE | PAGE_READ)
-
-#define __S000 __pgprot(PAGE_NONE)
-#define __S001 _PAGE_S(PAGE_READ)
-#define __S010 _PAGE_S(PAGE_WRITE)
-#define __S011 _PAGE_S(PAGE_WRITE | PAGE_READ)
-#define __S100 _PAGE_S(PAGE_EXEC)
-#define __S101 _PAGE_S(PAGE_EXEC | PAGE_READ)
-#define __S110 _PAGE_S(PAGE_EXEC | PAGE_WRITE)
-#define __S111 _PAGE_S(PAGE_EXEC | PAGE_WRITE | PAGE_READ)
-
-#define pte_none(x) (!pte_val(x))
-#define pte_present(x) (pte_val(x) & _PAGE_PRESENT)
-
-#define pte_clear(mm,addr,xp) \
- do { \
- set_pte_at(mm, addr, xp, __pte(0)); \
- } while (0)
-
-/*
- * The following only work if pte_present() is true.
- * Undefined behaviour if not..
- */
-static inline int pte_write(pte_t pte)
-{
- return pte_val(pte) & _PAGE_RW;
-}
-static inline int pte_dirty(pte_t pte)
-{
- return pte_val(pte) & _PAGE_DIRTY;
-}
-static inline int pte_young(pte_t pte)
-{
- return pte_val(pte) & _PAGE_ACCESSED;
-}
-static inline int pte_special(pte_t pte)
-{
- return 0;
-}
-
-/* Mutator functions for PTE bits */
-static inline pte_t pte_wrprotect(pte_t pte)
-{
- set_pte(&pte, __pte(pte_val(pte) & ~_PAGE_RW));
- return pte;
-}
-static inline pte_t pte_mkclean(pte_t pte)
-{
- set_pte(&pte, __pte(pte_val(pte) & ~_PAGE_DIRTY));
- return pte;
-}
-static inline pte_t pte_mkold(pte_t pte)
-{
- set_pte(&pte, __pte(pte_val(pte) & ~_PAGE_ACCESSED));
- return pte;
-}
-static inline pte_t pte_mkwrite(pte_t pte)
-{
- set_pte(&pte, __pte(pte_val(pte) | _PAGE_RW));
- return pte;
-}
-static inline pte_t pte_mkdirty(pte_t pte)
-{
- set_pte(&pte, __pte(pte_val(pte) | _PAGE_DIRTY));
- return pte;
-}
-static inline pte_t pte_mkyoung(pte_t pte)
-{
- set_pte(&pte, __pte(pte_val(pte) | _PAGE_ACCESSED));
- return pte;
-}
-static inline pte_t pte_mkspecial(pte_t pte)
-{
- return pte;
-}
-
-#define pmd_none(x) (!pmd_val(x))
-#define pmd_present(x) (pmd_val(x))
-
-static inline void pmd_clear(pmd_t *pmdp)
-{
- set_pmd(pmdp, __pmd(0));
-}
-
-#define pmd_bad(x) (pmd_val(x) & ~PAGE_MASK)
-
-/*
- * Permanent address of a page. We don't support highmem, so this is
- * trivial.
- */
-#define pages_to_mb(x) ((x) >> (20-PAGE_SHIFT))
-#define pte_page(x) (pfn_to_page(pte_pfn(x)))
-
-/*
- * Mark the prot value as uncacheable and unbufferable
- */
-#define pgprot_noncached(prot) \
- __pgprot(pgprot_val(prot) & ~(_PAGE_BUFFER | _PAGE_CACHABLE))
-
-/*
- * Mark the prot value as uncacheable but bufferable
- */
-#define pgprot_writecombine(prot) \
- __pgprot((pgprot_val(prot) & ~_PAGE_CACHABLE) | _PAGE_BUFFER)
-
-/*
- * Conversion functions: convert a page and protection to a page entry,
- * and a page entry and page directory to the page they refer to.
- *
- * extern pte_t mk_pte(struct page *page, pgprot_t pgprot)
- */
-#define mk_pte(page, pgprot) pfn_pte(page_to_pfn(page), (pgprot))
-
-static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
-{
- set_pte(&pte, __pte((pte_val(pte) & _PAGE_CHG_MASK)
- | pgprot_val(newprot)));
- return pte;
-}
-
-#define page_pte(page) page_pte_prot(page, __pgprot(0))
-
-#define pmd_page_vaddr(pmd) pmd_val(pmd)
-#define pmd_page(pmd) (virt_to_page(pmd_val(pmd)))
-
-/* to find an entry in a page-table-directory. */
-#define pgd_index(address) (((address) >> PGDIR_SHIFT) \
- & (PTRS_PER_PGD - 1))
-#define pgd_offset(mm, address) ((mm)->pgd + pgd_index(address))
-
-/* to find an entry in a kernel page-table-directory */
-#define pgd_offset_k(address) pgd_offset(&init_mm, address)
-
-/* Find an entry in the third-level page table.. */
-#define pte_index(address) \
- ((address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1))
-#define pte_offset(dir, address) \
- ((pte_t *) pmd_page_vaddr(*(dir)) + pte_index(address))
-#define pte_offset_kernel(dir, address) \
- ((pte_t *) pmd_page_vaddr(*(dir)) + pte_index(address))
-#define pte_offset_map(dir, address) pte_offset_kernel(dir, address)
-#define pte_unmap(pte) do { } while (0)
-
-struct vm_area_struct;
-extern void update_mmu_cache(struct vm_area_struct * vma,
- unsigned long address, pte_t *ptep);
-
-/*
- * Encode and decode a swap entry
- *
- * Constraints:
- * _PAGE_TYPE_* at bits 2-3 (for emulating _PAGE_PROTNONE)
- * _PAGE_PRESENT at bit 10
- *
- * We encode the type into bits 4-9 and offset into bits 11-31. This
- * gives us a 21 bits offset, or 2**21 * 4K = 8G usable swap space per
- * device, and 64 possible types.
- *
- * NOTE: We should set ZEROs at the position of _PAGE_PRESENT
- * and _PAGE_PROTNONE bits
- */
-#define __swp_type(x) (((x).val >> 4) & 0x3f)
-#define __swp_offset(x) ((x).val >> 11)
-#define __swp_entry(type, offset) ((swp_entry_t) { ((type) << 4) | ((offset) << 11) })
-#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
-#define __swp_entry_to_pte(x) ((pte_t) { (x).val })
-
-typedef pte_t *pte_addr_t;
-
-#define kern_addr_valid(addr) (1)
-
-/* No page table caches to initialize (?) */
-#define pgtable_cache_init() do { } while(0)
-
-#include <asm-generic/pgtable.h>
-
-#endif /* !__ASSEMBLY__ */
-
-#endif /* __ASM_AVR32_PGTABLE_H */
diff --git a/arch/avr32/include/asm/processor.h b/arch/avr32/include/asm/processor.h
deleted file mode 100644
index 972adcc..0000000
--- a/arch/avr32/include/asm/processor.h
+++ /dev/null
@@ -1,166 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_PROCESSOR_H
-#define __ASM_AVR32_PROCESSOR_H
-
-#include <asm/page.h>
-#include <asm/cache.h>
-
-#define TASK_SIZE 0x80000000
-
-#ifdef __KERNEL__
-#define STACK_TOP TASK_SIZE
-#define STACK_TOP_MAX STACK_TOP
-#endif
-
-#ifndef __ASSEMBLY__
-
-static inline void *current_text_addr(void)
-{
- register void *pc asm("pc");
- return pc;
-}
-
-enum arch_type {
- ARCH_AVR32A,
- ARCH_AVR32B,
- ARCH_MAX
-};
-
-enum cpu_type {
- CPU_MORGAN,
- CPU_AT32AP,
- CPU_MAX
-};
-
-enum tlb_config {
- TLB_NONE,
- TLB_SPLIT,
- TLB_UNIFIED,
- TLB_INVALID
-};
-
-#define AVR32_FEATURE_RMW (1 << 0)
-#define AVR32_FEATURE_DSP (1 << 1)
-#define AVR32_FEATURE_SIMD (1 << 2)
-#define AVR32_FEATURE_OCD (1 << 3)
-#define AVR32_FEATURE_PCTR (1 << 4)
-#define AVR32_FEATURE_JAVA (1 << 5)
-#define AVR32_FEATURE_FPU (1 << 6)
-
-struct avr32_cpuinfo {
- struct clk *clk;
- unsigned long loops_per_jiffy;
- enum arch_type arch_type;
- enum cpu_type cpu_type;
- unsigned short arch_revision;
- unsigned short cpu_revision;
- enum tlb_config tlb_config;
- unsigned long features;
- u32 device_id;
-
- struct cache_info icache;
- struct cache_info dcache;
-};
-
-static inline unsigned int avr32_get_manufacturer_id(struct avr32_cpuinfo *cpu)
-{
- return (cpu->device_id >> 1) & 0x7f;
-}
-static inline unsigned int avr32_get_product_number(struct avr32_cpuinfo *cpu)
-{
- return (cpu->device_id >> 12) & 0xffff;
-}
-static inline unsigned int avr32_get_chip_revision(struct avr32_cpuinfo *cpu)
-{
- return (cpu->device_id >> 28) & 0x0f;
-}
-
-extern struct avr32_cpuinfo boot_cpu_data;
-
-/* No SMP support so far */
-#define current_cpu_data boot_cpu_data
-
-/* This decides where the kernel will search for a free chunk of vm
- * space during mmap's
- */
-#define TASK_UNMAPPED_BASE (PAGE_ALIGN(TASK_SIZE / 3))
-
-#define cpu_relax() barrier()
-#define cpu_sync_pipeline() asm volatile("sub pc, -2" : : : "memory")
-
-struct cpu_context {
- unsigned long sr;
- unsigned long pc;
- unsigned long ksp; /* Kernel stack pointer */
- unsigned long r7;
- unsigned long r6;
- unsigned long r5;
- unsigned long r4;
- unsigned long r3;
- unsigned long r2;
- unsigned long r1;
- unsigned long r0;
-};
-
-/* This struct contains the CPU context as stored by switch_to() */
-struct thread_struct {
- struct cpu_context cpu_context;
- unsigned long single_step_addr;
- u16 single_step_insn;
-};
-
-#define INIT_THREAD { \
- .cpu_context = { \
- .ksp = sizeof(init_stack) + (long)&init_stack, \
- }, \
-}
-
-/*
- * Do necessary setup to start up a newly executed thread.
- */
-#define start_thread(regs, new_pc, new_sp) \
- do { \
- memset(regs, 0, sizeof(*regs)); \
- regs->sr = MODE_USER; \
- regs->pc = new_pc & ~1; \
- regs->sp = new_sp; \
- } while(0)
-
-struct task_struct;
-
-/* Free all resources held by a thread */
-extern void release_thread(struct task_struct *);
-
-/* Return saved PC of a blocked thread */
-#define thread_saved_pc(tsk) ((tsk)->thread.cpu_context.pc)
-
-struct pt_regs;
-extern unsigned long get_wchan(struct task_struct *p);
-extern void show_regs_log_lvl(struct pt_regs *regs, const char *log_lvl);
-extern void show_stack_log_lvl(struct task_struct *tsk, unsigned long sp,
- struct pt_regs *regs, const char *log_lvl);
-
-#define task_pt_regs(p) \
- ((struct pt_regs *)(THREAD_SIZE + task_stack_page(p)) - 1)
-
-#define KSTK_EIP(tsk) ((tsk)->thread.cpu_context.pc)
-#define KSTK_ESP(tsk) ((tsk)->thread.cpu_context.ksp)
-
-#define ARCH_HAS_PREFETCH
-
-static inline void prefetch(const void *x)
-{
- const char *c = x;
- asm volatile("pref %0" : : "r"(c));
-}
-#define PREFETCH_STRIDE L1_CACHE_BYTES
-
-#endif /* __ASSEMBLY__ */
-
-#endif /* __ASM_AVR32_PROCESSOR_H */
diff --git a/arch/avr32/include/asm/ptrace.h b/arch/avr32/include/asm/ptrace.h
deleted file mode 100644
index 630e4f9..0000000
--- a/arch/avr32/include/asm/ptrace.h
+++ /dev/null
@@ -1,45 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_PTRACE_H
-#define __ASM_AVR32_PTRACE_H
-
-#include <uapi/asm/ptrace.h>
-
-#ifndef __ASSEMBLY__
-
-#include <asm/ocd.h>
-
-#define arch_has_single_step() (1)
-
-#define arch_ptrace_attach(child) ocd_enable(child)
-
-#define user_mode(regs) (((regs)->sr & MODE_MASK) == MODE_USER)
-#define instruction_pointer(regs) ((regs)->pc)
-#define profile_pc(regs) instruction_pointer(regs)
-#define user_stack_pointer(regs) ((regs)->sp)
-
-static __inline__ int valid_user_regs(struct pt_regs *regs)
-{
- /*
- * Some of the Java bits might be acceptable if/when we
- * implement some support for that stuff...
- */
- if ((regs->sr & 0xffff0000) == 0)
- return 1;
-
- /*
- * Force status register flags to be sane and report this
- * illegal behaviour...
- */
- regs->sr &= 0x0000ffff;
- return 0;
-}
-
-
-#endif /* ! __ASSEMBLY__ */
-#endif /* __ASM_AVR32_PTRACE_H */
diff --git a/arch/avr32/include/asm/serial.h b/arch/avr32/include/asm/serial.h
deleted file mode 100644
index 5ecaebc..0000000
--- a/arch/avr32/include/asm/serial.h
+++ /dev/null
@@ -1,13 +0,0 @@
-#ifndef _ASM_SERIAL_H
-#define _ASM_SERIAL_H
-
-/*
- * This assumes you have a 1.8432 MHz clock for your UART.
- *
- * It'd be nice if someone built a serial card with a 24.576 MHz
- * clock, since the 16550A is capable of handling a top speed of 1.5
- * megabits/second; but this requires the faster clock.
- */
-#define BASE_BAUD (1843200 / 16)
-
-#endif /* _ASM_SERIAL_H */
diff --git a/arch/avr32/include/asm/setup.h b/arch/avr32/include/asm/setup.h
deleted file mode 100644
index 73490ae..0000000
--- a/arch/avr32/include/asm/setup.h
+++ /dev/null
@@ -1,144 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * Based on linux/include/asm-arm/setup.h
- * Copyright (C) 1997-1999 Russell King
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_SETUP_H__
-#define __ASM_AVR32_SETUP_H__
-
-#include <uapi/asm/setup.h>
-
-
-/* Magic number indicating that a tag table is present */
-#define ATAG_MAGIC 0xa2a25441
-
-#ifndef __ASSEMBLY__
-
-/*
- * Generic memory range, used by several tags.
- *
- * addr is always physical.
- * size is measured in bytes.
- * next is for use by the OS, e.g. for grouping regions into
- * linked lists.
- */
-struct tag_mem_range {
- u32 addr;
- u32 size;
- struct tag_mem_range * next;
-};
-
-/* The list ends with an ATAG_NONE node. */
-#define ATAG_NONE 0x00000000
-
-struct tag_header {
- u32 size;
- u32 tag;
-};
-
-/* The list must start with an ATAG_CORE node */
-#define ATAG_CORE 0x54410001
-
-struct tag_core {
- u32 flags;
- u32 pagesize;
- u32 rootdev;
-};
-
-/* it is allowed to have multiple ATAG_MEM nodes */
-#define ATAG_MEM 0x54410002
-/* ATAG_MEM uses tag_mem_range */
-
-/* command line: \0 terminated string */
-#define ATAG_CMDLINE 0x54410003
-
-struct tag_cmdline {
- char cmdline[1]; /* this is the minimum size */
-};
-
-/* Ramdisk image (may be compressed) */
-#define ATAG_RDIMG 0x54410004
-/* ATAG_RDIMG uses tag_mem_range */
-
-/* Information about various clocks present in the system */
-#define ATAG_CLOCK 0x54410005
-
-struct tag_clock {
- u32 clock_id; /* Which clock are we talking about? */
- u32 clock_flags; /* Special features */
- u64 clock_hz; /* Clock speed in Hz */
-};
-
-/* The clock types we know about */
-#define CLOCK_BOOTCPU 0
-
-/* Memory reserved for the system (e.g. the bootloader) */
-#define ATAG_RSVD_MEM 0x54410006
-/* ATAG_RSVD_MEM uses tag_mem_range */
-
-/* Ethernet information */
-
-#define ATAG_ETHERNET 0x54410007
-
-struct tag_ethernet {
- u8 mac_index;
- u8 mii_phy_addr;
- u8 hw_address[6];
-};
-
-#define ETH_INVALID_PHY 0xff
-
-/* board information */
-#define ATAG_BOARDINFO 0x54410008
-
-struct tag_boardinfo {
- u32 board_number;
-};
-
-struct tag {
- struct tag_header hdr;
- union {
- struct tag_core core;
- struct tag_mem_range mem_range;
- struct tag_cmdline cmdline;
- struct tag_clock clock;
- struct tag_ethernet ethernet;
- struct tag_boardinfo boardinfo;
- } u;
-};
-
-struct tagtable {
- u32 tag;
- int (*parse)(struct tag *);
-};
-
-#define __tag __used __attribute__((__section__(".taglist.init")))
-#define __tagtable(tag, fn) \
- static struct tagtable __tagtable_##fn __tag = { tag, fn }
-
-#define tag_member_present(tag,member) \
- ((unsigned long)(&((struct tag *)0L)->member + 1) \
- <= (tag)->hdr.size * 4)
-
-#define tag_next(t) ((struct tag *)((u32 *)(t) + (t)->hdr.size))
-#define tag_size(type) ((sizeof(struct tag_header) + sizeof(struct type)) >> 2)
-
-#define for_each_tag(t,base) \
- for (t = base; t->hdr.size; t = tag_next(t))
-
-extern struct tag *bootloader_tags;
-
-extern resource_size_t fbmem_start;
-extern resource_size_t fbmem_size;
-extern u32 board_number;
-
-void setup_processor(void);
-
-#endif /* !__ASSEMBLY__ */
-
-#endif /* __ASM_AVR32_SETUP_H__ */
diff --git a/arch/avr32/include/asm/shmparam.h b/arch/avr32/include/asm/shmparam.h
deleted file mode 100644
index 3681266..0000000
--- a/arch/avr32/include/asm/shmparam.h
+++ /dev/null
@@ -1,6 +0,0 @@
-#ifndef __ASM_AVR32_SHMPARAM_H
-#define __ASM_AVR32_SHMPARAM_H
-
-#define SHMLBA PAGE_SIZE /* attach addr a multiple of this */
-
-#endif /* __ASM_AVR32_SHMPARAM_H */
diff --git a/arch/avr32/include/asm/signal.h b/arch/avr32/include/asm/signal.h
deleted file mode 100644
index d875eb6..0000000
--- a/arch/avr32/include/asm/signal.h
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_SIGNAL_H
-#define __ASM_AVR32_SIGNAL_H
-
-#include <uapi/asm/signal.h>
-
-/* Most things should be clean enough to redefine this at will, if care
- is taken to make libc match. */
-
-#define _NSIG 64
-#define _NSIG_BPW 32
-#define _NSIG_WORDS (_NSIG / _NSIG_BPW)
-
-typedef unsigned long old_sigset_t; /* at least 32 bits */
-
-typedef struct {
- unsigned long sig[_NSIG_WORDS];
-} sigset_t;
-
-#define __ARCH_HAS_SA_RESTORER
-
-#include <asm/sigcontext.h>
-#undef __HAVE_ARCH_SIG_BITOPS
-
-#endif
diff --git a/arch/avr32/include/asm/string.h b/arch/avr32/include/asm/string.h
deleted file mode 100644
index c91a623..0000000
--- a/arch/avr32/include/asm/string.h
+++ /dev/null
@@ -1,17 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_STRING_H
-#define __ASM_AVR32_STRING_H
-
-#define __HAVE_ARCH_MEMSET
-extern void *memset(void *b, int c, size_t len);
-
-#define __HAVE_ARCH_MEMCPY
-extern void *memcpy(void *to, const void *from, size_t len);
-
-#endif /* __ASM_AVR32_STRING_H */
diff --git a/arch/avr32/include/asm/switch_to.h b/arch/avr32/include/asm/switch_to.h
deleted file mode 100644
index 6f00581..0000000
--- a/arch/avr32/include/asm/switch_to.h
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_SWITCH_TO_H
-#define __ASM_AVR32_SWITCH_TO_H
-
-/*
- * Help PathFinder and other Nexus-compliant debuggers keep track of
- * the current PID by emitting an Ownership Trace Message each time we
- * switch task.
- */
-#ifdef CONFIG_OWNERSHIP_TRACE
-#include <asm/ocd.h>
-#define ocd_switch(prev, next) \
- do { \
- ocd_write(PID, prev->pid); \
- ocd_write(PID, next->pid); \
- } while(0)
-#else
-#define ocd_switch(prev, next)
-#endif
-
-/*
- * switch_to(prev, next, last) should switch from task `prev' to task
- * `next'. `prev' will never be the same as `next'.
- *
- * We just delegate everything to the __switch_to assembly function,
- * which is implemented in arch/avr32/kernel/switch_to.S
- *
- * mb() tells GCC not to cache `current' across this call.
- */
-struct cpu_context;
-struct task_struct;
-extern struct task_struct *__switch_to(struct task_struct *,
- struct cpu_context *,
- struct cpu_context *);
-#define switch_to(prev, next, last) \
- do { \
- ocd_switch(prev, next); \
- last = __switch_to(prev, &prev->thread.cpu_context + 1, \
- &next->thread.cpu_context); \
- } while (0)
-
-
-#endif /* __ASM_AVR32_SWITCH_TO_H */
diff --git a/arch/avr32/include/asm/syscalls.h b/arch/avr32/include/asm/syscalls.h
deleted file mode 100644
index 244f2ac..0000000
--- a/arch/avr32/include/asm/syscalls.h
+++ /dev/null
@@ -1,21 +0,0 @@
-/*
- * syscalls.h - Linux syscall interfaces (arch-specific)
- *
- * Copyright (c) 2008 Jaswinder Singh
- *
- * This file is released under the GPLv2.
- * See the file COPYING for more details.
- */
-
-#ifndef _ASM_AVR32_SYSCALLS_H
-#define _ASM_AVR32_SYSCALLS_H
-
-#include <linux/compiler.h>
-#include <linux/linkage.h>
-#include <linux/types.h>
-#include <linux/signal.h>
-
-/* mm/cache.c */
-asmlinkage int sys_cacheflush(int, void __user *, size_t);
-
-#endif /* _ASM_AVR32_SYSCALLS_H */
diff --git a/arch/avr32/include/asm/sysreg.h b/arch/avr32/include/asm/sysreg.h
deleted file mode 100644
index d4e0950..0000000
--- a/arch/avr32/include/asm/sysreg.h
+++ /dev/null
@@ -1,291 +0,0 @@
-/*
- * AVR32 System Registers
- *
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_SYSREG_H
-#define __ASM_AVR32_SYSREG_H
-
-/* sysreg register offsets */
-#define SYSREG_SR 0x0000
-#define SYSREG_EVBA 0x0004
-#define SYSREG_ACBA 0x0008
-#define SYSREG_CPUCR 0x000c
-#define SYSREG_ECR 0x0010
-#define SYSREG_RSR_SUP 0x0014
-#define SYSREG_RSR_INT0 0x0018
-#define SYSREG_RSR_INT1 0x001c
-#define SYSREG_RSR_INT2 0x0020
-#define SYSREG_RSR_INT3 0x0024
-#define SYSREG_RSR_EX 0x0028
-#define SYSREG_RSR_NMI 0x002c
-#define SYSREG_RSR_DBG 0x0030
-#define SYSREG_RAR_SUP 0x0034
-#define SYSREG_RAR_INT0 0x0038
-#define SYSREG_RAR_INT1 0x003c
-#define SYSREG_RAR_INT2 0x0040
-#define SYSREG_RAR_INT3 0x0044
-#define SYSREG_RAR_EX 0x0048
-#define SYSREG_RAR_NMI 0x004c
-#define SYSREG_RAR_DBG 0x0050
-#define SYSREG_JECR 0x0054
-#define SYSREG_JOSP 0x0058
-#define SYSREG_JAVA_LV0 0x005c
-#define SYSREG_JAVA_LV1 0x0060
-#define SYSREG_JAVA_LV2 0x0064
-#define SYSREG_JAVA_LV3 0x0068
-#define SYSREG_JAVA_LV4 0x006c
-#define SYSREG_JAVA_LV5 0x0070
-#define SYSREG_JAVA_LV6 0x0074
-#define SYSREG_JAVA_LV7 0x0078
-#define SYSREG_JTBA 0x007c
-#define SYSREG_JBCR 0x0080
-#define SYSREG_CONFIG0 0x0100
-#define SYSREG_CONFIG1 0x0104
-#define SYSREG_COUNT 0x0108
-#define SYSREG_COMPARE 0x010c
-#define SYSREG_TLBEHI 0x0110
-#define SYSREG_TLBELO 0x0114
-#define SYSREG_PTBR 0x0118
-#define SYSREG_TLBEAR 0x011c
-#define SYSREG_MMUCR 0x0120
-#define SYSREG_TLBARLO 0x0124
-#define SYSREG_TLBARHI 0x0128
-#define SYSREG_PCCNT 0x012c
-#define SYSREG_PCNT0 0x0130
-#define SYSREG_PCNT1 0x0134
-#define SYSREG_PCCR 0x0138
-#define SYSREG_BEAR 0x013c
-#define SYSREG_SABAL 0x0300
-#define SYSREG_SABAH 0x0304
-#define SYSREG_SABD 0x0308
-
-/* Bitfields in SR */
-#define SYSREG_SR_C_OFFSET 0
-#define SYSREG_SR_C_SIZE 1
-#define SYSREG_Z_OFFSET 1
-#define SYSREG_Z_SIZE 1
-#define SYSREG_SR_N_OFFSET 2
-#define SYSREG_SR_N_SIZE 1
-#define SYSREG_SR_V_OFFSET 3
-#define SYSREG_SR_V_SIZE 1
-#define SYSREG_Q_OFFSET 4
-#define SYSREG_Q_SIZE 1
-#define SYSREG_L_OFFSET 5
-#define SYSREG_L_SIZE 1
-#define SYSREG_T_OFFSET 14
-#define SYSREG_T_SIZE 1
-#define SYSREG_SR_R_OFFSET 15
-#define SYSREG_SR_R_SIZE 1
-#define SYSREG_GM_OFFSET 16
-#define SYSREG_GM_SIZE 1
-#define SYSREG_I0M_OFFSET 17
-#define SYSREG_I0M_SIZE 1
-#define SYSREG_I1M_OFFSET 18
-#define SYSREG_I1M_SIZE 1
-#define SYSREG_I2M_OFFSET 19
-#define SYSREG_I2M_SIZE 1
-#define SYSREG_I3M_OFFSET 20
-#define SYSREG_I3M_SIZE 1
-#define SYSREG_EM_OFFSET 21
-#define SYSREG_EM_SIZE 1
-#define SYSREG_MODE_OFFSET 22
-#define SYSREG_MODE_SIZE 3
-#define SYSREG_M0_OFFSET 22
-#define SYSREG_M0_SIZE 1
-#define SYSREG_M1_OFFSET 23
-#define SYSREG_M1_SIZE 1
-#define SYSREG_M2_OFFSET 24
-#define SYSREG_M2_SIZE 1
-#define SYSREG_SR_D_OFFSET 26
-#define SYSREG_SR_D_SIZE 1
-#define SYSREG_DM_OFFSET 27
-#define SYSREG_DM_SIZE 1
-#define SYSREG_SR_J_OFFSET 28
-#define SYSREG_SR_J_SIZE 1
-#define SYSREG_H_OFFSET 29
-#define SYSREG_H_SIZE 1
-
-/* Bitfields in CPUCR */
-#define SYSREG_BI_OFFSET 0
-#define SYSREG_BI_SIZE 1
-#define SYSREG_BE_OFFSET 1
-#define SYSREG_BE_SIZE 1
-#define SYSREG_FE_OFFSET 2
-#define SYSREG_FE_SIZE 1
-#define SYSREG_RE_OFFSET 3
-#define SYSREG_RE_SIZE 1
-#define SYSREG_IBE_OFFSET 4
-#define SYSREG_IBE_SIZE 1
-#define SYSREG_IEE_OFFSET 5
-#define SYSREG_IEE_SIZE 1
-
-/* Bitfields in CONFIG0 */
-#define SYSREG_CONFIG0_R_OFFSET 0
-#define SYSREG_CONFIG0_R_SIZE 1
-#define SYSREG_CONFIG0_D_OFFSET 1
-#define SYSREG_CONFIG0_D_SIZE 1
-#define SYSREG_CONFIG0_S_OFFSET 2
-#define SYSREG_CONFIG0_S_SIZE 1
-#define SYSREG_CONFIG0_O_OFFSET 3
-#define SYSREG_CONFIG0_O_SIZE 1
-#define SYSREG_CONFIG0_P_OFFSET 4
-#define SYSREG_CONFIG0_P_SIZE 1
-#define SYSREG_CONFIG0_J_OFFSET 5
-#define SYSREG_CONFIG0_J_SIZE 1
-#define SYSREG_CONFIG0_F_OFFSET 6
-#define SYSREG_CONFIG0_F_SIZE 1
-#define SYSREG_MMUT_OFFSET 7
-#define SYSREG_MMUT_SIZE 3
-#define SYSREG_AR_OFFSET 10
-#define SYSREG_AR_SIZE 3
-#define SYSREG_AT_OFFSET 13
-#define SYSREG_AT_SIZE 3
-#define SYSREG_PROCESSORREVISION_OFFSET 16
-#define SYSREG_PROCESSORREVISION_SIZE 8
-#define SYSREG_PROCESSORID_OFFSET 24
-#define SYSREG_PROCESSORID_SIZE 8
-
-/* Bitfields in CONFIG1 */
-#define SYSREG_DASS_OFFSET 0
-#define SYSREG_DASS_SIZE 3
-#define SYSREG_DLSZ_OFFSET 3
-#define SYSREG_DLSZ_SIZE 3
-#define SYSREG_DSET_OFFSET 6
-#define SYSREG_DSET_SIZE 4
-#define SYSREG_IASS_OFFSET 10
-#define SYSREG_IASS_SIZE 3
-#define SYSREG_ILSZ_OFFSET 13
-#define SYSREG_ILSZ_SIZE 3
-#define SYSREG_ISET_OFFSET 16
-#define SYSREG_ISET_SIZE 4
-#define SYSREG_DMMUSZ_OFFSET 20
-#define SYSREG_DMMUSZ_SIZE 6
-#define SYSREG_IMMUSZ_OFFSET 26
-#define SYSREG_IMMUSZ_SIZE 6
-
-/* Bitfields in TLBEHI */
-#define SYSREG_ASID_OFFSET 0
-#define SYSREG_ASID_SIZE 8
-#define SYSREG_TLBEHI_I_OFFSET 8
-#define SYSREG_TLBEHI_I_SIZE 1
-#define SYSREG_TLBEHI_V_OFFSET 9
-#define SYSREG_TLBEHI_V_SIZE 1
-#define SYSREG_VPN_OFFSET 10
-#define SYSREG_VPN_SIZE 22
-
-/* Bitfields in TLBELO */
-#define SYSREG_W_OFFSET 0
-#define SYSREG_W_SIZE 1
-#define SYSREG_TLBELO_D_OFFSET 1
-#define SYSREG_TLBELO_D_SIZE 1
-#define SYSREG_SZ_OFFSET 2
-#define SYSREG_SZ_SIZE 2
-#define SYSREG_AP_OFFSET 4
-#define SYSREG_AP_SIZE 3
-#define SYSREG_B_OFFSET 7
-#define SYSREG_B_SIZE 1
-#define SYSREG_G_OFFSET 8
-#define SYSREG_G_SIZE 1
-#define SYSREG_TLBELO_C_OFFSET 9
-#define SYSREG_TLBELO_C_SIZE 1
-#define SYSREG_PFN_OFFSET 10
-#define SYSREG_PFN_SIZE 22
-
-/* Bitfields in MMUCR */
-#define SYSREG_E_OFFSET 0
-#define SYSREG_E_SIZE 1
-#define SYSREG_M_OFFSET 1
-#define SYSREG_M_SIZE 1
-#define SYSREG_MMUCR_I_OFFSET 2
-#define SYSREG_MMUCR_I_SIZE 1
-#define SYSREG_MMUCR_N_OFFSET 3
-#define SYSREG_MMUCR_N_SIZE 1
-#define SYSREG_MMUCR_S_OFFSET 4
-#define SYSREG_MMUCR_S_SIZE 1
-#define SYSREG_DLA_OFFSET 8
-#define SYSREG_DLA_SIZE 6
-#define SYSREG_DRP_OFFSET 14
-#define SYSREG_DRP_SIZE 6
-#define SYSREG_ILA_OFFSET 20
-#define SYSREG_ILA_SIZE 6
-#define SYSREG_IRP_OFFSET 26
-#define SYSREG_IRP_SIZE 6
-
-/* Bitfields in PCCR */
-#define SYSREG_PCCR_E_OFFSET 0
-#define SYSREG_PCCR_E_SIZE 1
-#define SYSREG_PCCR_R_OFFSET 1
-#define SYSREG_PCCR_R_SIZE 1
-#define SYSREG_PCCR_C_OFFSET 2
-#define SYSREG_PCCR_C_SIZE 1
-#define SYSREG_PCCR_S_OFFSET 3
-#define SYSREG_PCCR_S_SIZE 1
-#define SYSREG_IEC_OFFSET 4
-#define SYSREG_IEC_SIZE 1
-#define SYSREG_IE0_OFFSET 5
-#define SYSREG_IE0_SIZE 1
-#define SYSREG_IE1_OFFSET 6
-#define SYSREG_IE1_SIZE 1
-#define SYSREG_FC_OFFSET 8
-#define SYSREG_FC_SIZE 1
-#define SYSREG_F0_OFFSET 9
-#define SYSREG_F0_SIZE 1
-#define SYSREG_F1_OFFSET 10
-#define SYSREG_F1_SIZE 1
-#define SYSREG_CONF0_OFFSET 12
-#define SYSREG_CONF0_SIZE 6
-#define SYSREG_CONF1_OFFSET 18
-#define SYSREG_CONF1_SIZE 6
-
-/* Constants for ECR */
-#define ECR_UNRECOVERABLE 0
-#define ECR_TLB_MULTIPLE 1
-#define ECR_BUS_ERROR_WRITE 2
-#define ECR_BUS_ERROR_READ 3
-#define ECR_NMI 4
-#define ECR_ADDR_ALIGN_X 5
-#define ECR_PROTECTION_X 6
-#define ECR_DEBUG 7
-#define ECR_ILLEGAL_OPCODE 8
-#define ECR_UNIMPL_INSTRUCTION 9
-#define ECR_PRIVILEGE_VIOLATION 10
-#define ECR_FPE 11
-#define ECR_COPROC_ABSENT 12
-#define ECR_ADDR_ALIGN_R 13
-#define ECR_ADDR_ALIGN_W 14
-#define ECR_PROTECTION_R 15
-#define ECR_PROTECTION_W 16
-#define ECR_DTLB_MODIFIED 17
-#define ECR_TLB_MISS_X 20
-#define ECR_TLB_MISS_R 24
-#define ECR_TLB_MISS_W 28
-
-/* Bit manipulation macros */
-#define SYSREG_BIT(name) \
- (1 << SYSREG_##name##_OFFSET)
-#define SYSREG_BF(name,value) \
- (((value) & ((1 << SYSREG_##name##_SIZE) - 1)) \
- << SYSREG_##name##_OFFSET)
-#define SYSREG_BFEXT(name,value)\
- (((value) >> SYSREG_##name##_OFFSET) \
- & ((1 << SYSREG_##name##_SIZE) - 1))
-#define SYSREG_BFINS(name,value,old) \
- (((old) & ~(((1 << SYSREG_##name##_SIZE) - 1) \
- << SYSREG_##name##_OFFSET)) \
- | SYSREG_BF(name,value))
-
-/* Register access macros */
-#ifdef __CHECKER__
-extern unsigned long __builtin_mfsr(unsigned long reg);
-extern void __builtin_mtsr(unsigned long reg, unsigned long value);
-#endif
-
-#define sysreg_read(reg) __builtin_mfsr(SYSREG_##reg)
-#define sysreg_write(reg, value) __builtin_mtsr(SYSREG_##reg, value)
-
-#endif /* __ASM_AVR32_SYSREG_H */
diff --git a/arch/avr32/include/asm/termios.h b/arch/avr32/include/asm/termios.h
deleted file mode 100644
index 9d59437..0000000
--- a/arch/avr32/include/asm/termios.h
+++ /dev/null
@@ -1,23 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_TERMIOS_H
-#define __ASM_AVR32_TERMIOS_H
-
-#include <uapi/asm/termios.h>
-
-/* intr=^C quit=^\ erase=del kill=^U
- eof=^D vtime=\0 vmin=\1 sxtc=\0
- start=^Q stop=^S susp=^Z eol=\0
- reprint=^R discard=^U werase=^W lnext=^V
- eol2=\0
-*/
-#define INIT_C_CC "\003\034\177\025\004\0\1\0\021\023\032\0\022\017\027\026\0"
-
-#include <asm-generic/termios-base.h>
-
-#endif /* __ASM_AVR32_TERMIOS_H */
diff --git a/arch/avr32/include/asm/thread_info.h b/arch/avr32/include/asm/thread_info.h
deleted file mode 100644
index d4d3079..0000000
--- a/arch/avr32/include/asm/thread_info.h
+++ /dev/null
@@ -1,103 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_THREAD_INFO_H
-#define __ASM_AVR32_THREAD_INFO_H
-
-#include <asm/page.h>
-
-#define THREAD_SIZE_ORDER 1
-#define THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER)
-
-#ifndef __ASSEMBLY__
-#include <asm/types.h>
-
-struct task_struct;
-
-struct thread_info {
- struct task_struct *task; /* main task structure */
- unsigned long flags; /* low level flags */
- __u32 cpu;
- __s32 preempt_count; /* 0 => preemptable, <0 => BUG */
- __u32 rar_saved; /* return address... */
- __u32 rsr_saved; /* ...and status register
- saved by debug handler
- when setting up
- trampoline */
- __u8 supervisor_stack[0];
-};
-
-#define INIT_THREAD_INFO(tsk) \
-{ \
- .task = &tsk, \
- .flags = 0, \
- .cpu = 0, \
- .preempt_count = INIT_PREEMPT_COUNT, \
-}
-
-#define init_thread_info (init_thread_union.thread_info)
-#define init_stack (init_thread_union.stack)
-
-/*
- * Get the thread information struct from C.
- * We do the usual trick and use the lower end of the stack for this
- */
-static inline struct thread_info *current_thread_info(void)
-{
- unsigned long addr = ~(THREAD_SIZE - 1);
-
- asm("and %0, sp" : "=r"(addr) : "0"(addr));
- return (struct thread_info *)addr;
-}
-
-#define get_thread_info(ti) get_task_struct((ti)->task)
-#define put_thread_info(ti) put_task_struct((ti)->task)
-
-#endif /* !__ASSEMBLY__ */
-
-/*
- * Thread information flags
- * - these are process state flags that various assembly files may need to access
- * - pending work-to-be-done flags are in LSW
- * - other flags in MSW
- */
-#define TIF_SYSCALL_TRACE 0 /* syscall trace active */
-#define TIF_SIGPENDING 1 /* signal pending */
-#define TIF_NEED_RESCHED 2 /* rescheduling necessary */
-#define TIF_BREAKPOINT 4 /* enter monitor mode on return */
-#define TIF_SINGLE_STEP 5 /* single step in progress */
-#define TIF_MEMDIE 6 /* is terminating due to OOM killer */
-#define TIF_RESTORE_SIGMASK 7 /* restore signal mask in do_signal */
-#define TIF_CPU_GOING_TO_SLEEP 8 /* CPU is entering sleep 0 mode */
-#define TIF_NOTIFY_RESUME 9 /* callback before returning to user */
-#define TIF_DEBUG 30 /* debugging enabled */
-#define TIF_USERSPACE 31 /* true if FS sets userspace */
-
-#define _TIF_SYSCALL_TRACE (1 << TIF_SYSCALL_TRACE)
-#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)
-#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)
-#define _TIF_BREAKPOINT (1 << TIF_BREAKPOINT)
-#define _TIF_SINGLE_STEP (1 << TIF_SINGLE_STEP)
-#define _TIF_MEMDIE (1 << TIF_MEMDIE)
-#define _TIF_CPU_GOING_TO_SLEEP (1 << TIF_CPU_GOING_TO_SLEEP)
-#define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)
-
-/* Note: The masks below must never span more than 16 bits! */
-
-/* work to do on interrupt/exception return */
-#define _TIF_WORK_MASK \
- (_TIF_SIGPENDING \
- | _TIF_NOTIFY_RESUME \
- | _TIF_NEED_RESCHED \
- | _TIF_BREAKPOINT)
-
-/* work to do on any return to userspace */
-#define _TIF_ALLWORK_MASK (_TIF_WORK_MASK | _TIF_SYSCALL_TRACE)
-/* work to do on return from debug mode */
-#define _TIF_DBGWORK_MASK (_TIF_WORK_MASK & ~_TIF_BREAKPOINT)
-
-#endif /* __ASM_AVR32_THREAD_INFO_H */
diff --git a/arch/avr32/include/asm/timex.h b/arch/avr32/include/asm/timex.h
deleted file mode 100644
index 187dcf3..0000000
--- a/arch/avr32/include/asm/timex.h
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_TIMEX_H
-#define __ASM_AVR32_TIMEX_H
-
-/*
- * This is the frequency of the timer used for Linux's timer interrupt.
- * The value should be defined as accurate as possible or under certain
- * circumstances Linux timekeeping might become inaccurate or fail.
- *
- * For many system the exact clockrate of the timer isn't known but due to
- * the way this value is used we can get away with a wrong value as long
- * as this value is:
- *
- * - a multiple of HZ
- * - a divisor of the actual rate
- *
- * 500000 is a good such cheat value.
- *
- * The obscure number 1193182 is the same as used by the original i8254
- * time in legacy PC hardware; the chip is never found in AVR32 systems.
- */
-#define CLOCK_TICK_RATE 500000 /* Underlying HZ */
-
-typedef unsigned long cycles_t;
-
-static inline cycles_t get_cycles (void)
-{
- return 0;
-}
-
-#define ARCH_HAS_READ_CURRENT_TIMER
-
-#endif /* __ASM_AVR32_TIMEX_H */
diff --git a/arch/avr32/include/asm/tlb.h b/arch/avr32/include/asm/tlb.h
deleted file mode 100644
index 5c55f9c..0000000
--- a/arch/avr32/include/asm/tlb.h
+++ /dev/null
@@ -1,32 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_TLB_H
-#define __ASM_AVR32_TLB_H
-
-#define tlb_start_vma(tlb, vma) \
- flush_cache_range(vma, vma->vm_start, vma->vm_end)
-
-#define tlb_end_vma(tlb, vma) \
- flush_tlb_range(vma, vma->vm_start, vma->vm_end)
-
-#define __tlb_remove_tlb_entry(tlb, pte, address) do { } while(0)
-
-/*
- * Flush whole TLB for MM
- */
-#define tlb_flush(tlb) flush_tlb_mm((tlb)->mm)
-
-#include <asm-generic/tlb.h>
-
-/*
- * For debugging purposes
- */
-extern void show_dtlb_entry(unsigned int index);
-extern void dump_dtlb(void);
-
-#endif /* __ASM_AVR32_TLB_H */
diff --git a/arch/avr32/include/asm/tlbflush.h b/arch/avr32/include/asm/tlbflush.h
deleted file mode 100644
index bf90a78..0000000
--- a/arch/avr32/include/asm/tlbflush.h
+++ /dev/null
@@ -1,32 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_TLBFLUSH_H
-#define __ASM_AVR32_TLBFLUSH_H
-
-#include <asm/mmu.h>
-
-/*
- * TLB flushing:
- *
- * - flush_tlb() flushes the current mm struct TLBs
- * - flush_tlb_all() flushes all processes' TLB entries
- * - flush_tlb_mm(mm) flushes the specified mm context TLBs
- * - flush_tlb_page(vma, vmaddr) flushes one page
- * - flush_tlb_range(vma, start, end) flushes a range of pages
- * - flush_tlb_kernel_range(start, end) flushes a range of kernel pages
- */
-extern void flush_tlb(void);
-extern void flush_tlb_all(void);
-extern void flush_tlb_mm(struct mm_struct *mm);
-extern void flush_tlb_range(struct vm_area_struct *vma, unsigned long start,
- unsigned long end);
-extern void flush_tlb_page(struct vm_area_struct *vma, unsigned long page);
-
-extern void flush_tlb_kernel_range(unsigned long start, unsigned long end);
-
-#endif /* __ASM_AVR32_TLBFLUSH_H */
diff --git a/arch/avr32/include/asm/traps.h b/arch/avr32/include/asm/traps.h
deleted file mode 100644
index 6a8fb94..0000000
--- a/arch/avr32/include/asm/traps.h
+++ /dev/null
@@ -1,23 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_TRAPS_H
-#define __ASM_AVR32_TRAPS_H
-
-#include <linux/list.h>
-
-struct undef_hook {
- struct list_head node;
- u32 insn_mask;
- u32 insn_val;
- int (*fn)(struct pt_regs *regs, u32 insn);
-};
-
-void register_undef_hook(struct undef_hook *hook);
-void unregister_undef_hook(struct undef_hook *hook);
-
-#endif /* __ASM_AVR32_TRAPS_H */
diff --git a/arch/avr32/include/asm/types.h b/arch/avr32/include/asm/types.h
deleted file mode 100644
index 5932405..0000000
--- a/arch/avr32/include/asm/types.h
+++ /dev/null
@@ -1,19 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_TYPES_H
-#define __ASM_AVR32_TYPES_H
-
-#include <uapi/asm/types.h>
-
-/*
- * These aren't exported outside the kernel to avoid name space clashes
- */
-
-#define BITS_PER_LONG 32
-
-#endif /* __ASM_AVR32_TYPES_H */
diff --git a/arch/avr32/include/asm/uaccess.h b/arch/avr32/include/asm/uaccess.h
deleted file mode 100644
index b1ec1fa..0000000
--- a/arch/avr32/include/asm/uaccess.h
+++ /dev/null
@@ -1,337 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_UACCESS_H
-#define __ASM_AVR32_UACCESS_H
-
-#include <linux/errno.h>
-#include <linux/sched.h>
-
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
-typedef struct {
- unsigned int is_user_space;
-} mm_segment_t;
-
-/*
- * The fs value determines whether argument validity checking should be
- * performed or not. If get_fs() == USER_DS, checking is performed, with
- * get_fs() == KERNEL_DS, checking is bypassed.
- *
- * For historical reasons (Data Segment Register?), these macros are misnamed.
- */
-#define MAKE_MM_SEG(s) ((mm_segment_t) { (s) })
-#define segment_eq(a, b) ((a).is_user_space == (b).is_user_space)
-
-#define USER_ADDR_LIMIT 0x80000000
-
-#define KERNEL_DS MAKE_MM_SEG(0)
-#define USER_DS MAKE_MM_SEG(1)
-
-#define get_ds() (KERNEL_DS)
-
-static inline mm_segment_t get_fs(void)
-{
- return MAKE_MM_SEG(test_thread_flag(TIF_USERSPACE));
-}
-
-static inline void set_fs(mm_segment_t s)
-{
- if (s.is_user_space)
- set_thread_flag(TIF_USERSPACE);
- else
- clear_thread_flag(TIF_USERSPACE);
-}
-
-/*
- * Test whether a block of memory is a valid user space address.
- * Returns 0 if the range is valid, nonzero otherwise.
- *
- * We do the following checks:
- * 1. Is the access from kernel space?
- * 2. Does (addr + size) set the carry bit?
- * 3. Is (addr + size) a negative number (i.e. >= 0x80000000)?
- *
- * If yes on the first check, access is granted.
- * If no on any of the others, access is denied.
- */
-#define __range_ok(addr, size) \
- (test_thread_flag(TIF_USERSPACE) \
- && (((unsigned long)(addr) >= 0x80000000) \
- || ((unsigned long)(size) > 0x80000000) \
- || (((unsigned long)(addr) + (unsigned long)(size)) > 0x80000000)))
-
-#define access_ok(type, addr, size) (likely(__range_ok(addr, size) == 0))
-
-/* Generic arbitrary sized copy. Return the number of bytes NOT copied */
-extern __kernel_size_t __copy_user(void *to, const void *from,
- __kernel_size_t n);
-
-extern __kernel_size_t copy_to_user(void __user *to, const void *from,
- __kernel_size_t n);
-extern __kernel_size_t ___copy_from_user(void *to, const void __user *from,
- __kernel_size_t n);
-
-static inline __kernel_size_t __copy_to_user(void __user *to, const void *from,
- __kernel_size_t n)
-{
- return __copy_user((void __force *)to, from, n);
-}
-static inline __kernel_size_t __copy_from_user(void *to,
- const void __user *from,
- __kernel_size_t n)
-{
- return __copy_user(to, (const void __force *)from, n);
-}
-static inline __kernel_size_t copy_from_user(void *to,
- const void __user *from,
- __kernel_size_t n)
-{
- size_t res = ___copy_from_user(to, from, n);
- if (unlikely(res))
- memset(to + (n - res), 0, res);
- return res;
-}
-
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
-
-/*
- * put_user: - Write a simple value into user space.
- * @x: Value to copy to user space.
- * @ptr: Destination address, in user space.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * This macro copies a single simple value from kernel space to user
- * space. It supports simple types like char and int, but not larger
- * data types like structures or arrays.
- *
- * @ptr must have pointer-to-simple-variable type, and @x must be assignable
- * to the result of dereferencing @ptr.
- *
- * Returns zero on success, or -EFAULT on error.
- */
-#define put_user(x, ptr) \
- __put_user_check((x), (ptr), sizeof(*(ptr)))
-
-/*
- * get_user: - Get a simple variable from user space.
- * @x: Variable to store result.
- * @ptr: Source address, in user space.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * This macro copies a single simple variable from user space to kernel
- * space. It supports simple types like char and int, but not larger
- * data types like structures or arrays.
- *
- * @ptr must have pointer-to-simple-variable type, and the result of
- * dereferencing @ptr must be assignable to @x without a cast.
- *
- * Returns zero on success, or -EFAULT on error.
- * On error, the variable @x is set to zero.
- */
-#define get_user(x, ptr) \
- __get_user_check((x), (ptr), sizeof(*(ptr)))
-
-/*
- * __put_user: - Write a simple value into user space, with less checking.
- * @x: Value to copy to user space.
- * @ptr: Destination address, in user space.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * This macro copies a single simple value from kernel space to user
- * space. It supports simple types like char and int, but not larger
- * data types like structures or arrays.
- *
- * @ptr must have pointer-to-simple-variable type, and @x must be assignable
- * to the result of dereferencing @ptr.
- *
- * Caller must check the pointer with access_ok() before calling this
- * function.
- *
- * Returns zero on success, or -EFAULT on error.
- */
-#define __put_user(x, ptr) \
- __put_user_nocheck((x), (ptr), sizeof(*(ptr)))
-
-/*
- * __get_user: - Get a simple variable from user space, with less checking.
- * @x: Variable to store result.
- * @ptr: Source address, in user space.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * This macro copies a single simple variable from user space to kernel
- * space. It supports simple types like char and int, but not larger
- * data types like structures or arrays.
- *
- * @ptr must have pointer-to-simple-variable type, and the result of
- * dereferencing @ptr must be assignable to @x without a cast.
- *
- * Caller must check the pointer with access_ok() before calling this
- * function.
- *
- * Returns zero on success, or -EFAULT on error.
- * On error, the variable @x is set to zero.
- */
-#define __get_user(x, ptr) \
- __get_user_nocheck((x), (ptr), sizeof(*(ptr)))
-
-extern int __get_user_bad(void);
-extern int __put_user_bad(void);
-
-#define __get_user_nocheck(x, ptr, size) \
-({ \
- unsigned long __gu_val = 0; \
- int __gu_err = 0; \
- \
- switch (size) { \
- case 1: __get_user_asm("ub", __gu_val, ptr, __gu_err); break; \
- case 2: __get_user_asm("uh", __gu_val, ptr, __gu_err); break; \
- case 4: __get_user_asm("w", __gu_val, ptr, __gu_err); break; \
- default: __gu_err = __get_user_bad(); break; \
- } \
- \
- x = (__force typeof(*(ptr)))__gu_val; \
- __gu_err; \
-})
-
-#define __get_user_check(x, ptr, size) \
-({ \
- unsigned long __gu_val = 0; \
- const typeof(*(ptr)) __user * __gu_addr = (ptr); \
- int __gu_err = 0; \
- \
- if (access_ok(VERIFY_READ, __gu_addr, size)) { \
- switch (size) { \
- case 1: \
- __get_user_asm("ub", __gu_val, __gu_addr, \
- __gu_err); \
- break; \
- case 2: \
- __get_user_asm("uh", __gu_val, __gu_addr, \
- __gu_err); \
- break; \
- case 4: \
- __get_user_asm("w", __gu_val, __gu_addr, \
- __gu_err); \
- break; \
- default: \
- __gu_err = __get_user_bad(); \
- break; \
- } \
- } else { \
- __gu_err = -EFAULT; \
- } \
- x = (__force typeof(*(ptr)))__gu_val; \
- __gu_err; \
-})
-
-#define __get_user_asm(suffix, __gu_val, ptr, __gu_err) \
- asm volatile( \
- "1: ld." suffix " %1, %3 \n" \
- "2: \n" \
- " .subsection 1 \n" \
- "3: mov %0, %4 \n" \
- " rjmp 2b \n" \
- " .subsection 0 \n" \
- " .section __ex_table, \"a\" \n" \
- " .long 1b, 3b \n" \
- " .previous \n" \
- : "=r"(__gu_err), "=r"(__gu_val) \
- : "0"(__gu_err), "m"(*(ptr)), "i"(-EFAULT))
-
-#define __put_user_nocheck(x, ptr, size) \
-({ \
- typeof(*(ptr)) __pu_val; \
- int __pu_err = 0; \
- \
- __pu_val = (x); \
- switch (size) { \
- case 1: __put_user_asm("b", ptr, __pu_val, __pu_err); break; \
- case 2: __put_user_asm("h", ptr, __pu_val, __pu_err); break; \
- case 4: __put_user_asm("w", ptr, __pu_val, __pu_err); break; \
- case 8: __put_user_asm("d", ptr, __pu_val, __pu_err); break; \
- default: __pu_err = __put_user_bad(); break; \
- } \
- __pu_err; \
-})
-
-#define __put_user_check(x, ptr, size) \
-({ \
- typeof(*(ptr)) __pu_val; \
- typeof(*(ptr)) __user *__pu_addr = (ptr); \
- int __pu_err = 0; \
- \
- __pu_val = (x); \
- if (access_ok(VERIFY_WRITE, __pu_addr, size)) { \
- switch (size) { \
- case 1: \
- __put_user_asm("b", __pu_addr, __pu_val, \
- __pu_err); \
- break; \
- case 2: \
- __put_user_asm("h", __pu_addr, __pu_val, \
- __pu_err); \
- break; \
- case 4: \
- __put_user_asm("w", __pu_addr, __pu_val, \
- __pu_err); \
- break; \
- case 8: \
- __put_user_asm("d", __pu_addr, __pu_val, \
- __pu_err); \
- break; \
- default: \
- __pu_err = __put_user_bad(); \
- break; \
- } \
- } else { \
- __pu_err = -EFAULT; \
- } \
- __pu_err; \
-})
-
-#define __put_user_asm(suffix, ptr, __pu_val, __gu_err) \
- asm volatile( \
- "1: st." suffix " %1, %3 \n" \
- "2: \n" \
- " .subsection 1 \n" \
- "3: mov %0, %4 \n" \
- " rjmp 2b \n" \
- " .subsection 0 \n" \
- " .section __ex_table, \"a\" \n" \
- " .long 1b, 3b \n" \
- " .previous \n" \
- : "=r"(__gu_err), "=m"(*(ptr)) \
- : "0"(__gu_err), "r"(__pu_val), "i"(-EFAULT))
-
-extern __kernel_size_t clear_user(void __user *addr, __kernel_size_t size);
-extern __kernel_size_t __clear_user(void __user *addr, __kernel_size_t size);
-
-extern long strncpy_from_user(char *dst, const char __user *src, long count);
-extern long __strncpy_from_user(char *dst, const char __user *src, long count);
-
-extern long strnlen_user(const char __user *__s, long __n);
-extern long __strnlen_user(const char __user *__s, long __n);
-
-#define strlen_user(s) strnlen_user(s, ~0UL >> 1)
-
-struct exception_table_entry
-{
- unsigned long insn, fixup;
-};
-
-#endif /* __ASM_AVR32_UACCESS_H */
diff --git a/arch/avr32/include/asm/ucontext.h b/arch/avr32/include/asm/ucontext.h
deleted file mode 100644
index ac7259c..0000000
--- a/arch/avr32/include/asm/ucontext.h
+++ /dev/null
@@ -1,12 +0,0 @@
-#ifndef __ASM_AVR32_UCONTEXT_H
-#define __ASM_AVR32_UCONTEXT_H
-
-struct ucontext {
- unsigned long uc_flags;
- struct ucontext * uc_link;
- stack_t uc_stack;
- struct sigcontext uc_mcontext;
- sigset_t uc_sigmask;
-};
-
-#endif /* __ASM_AVR32_UCONTEXT_H */
diff --git a/arch/avr32/include/asm/unaligned.h b/arch/avr32/include/asm/unaligned.h
deleted file mode 100644
index 0418772..0000000
--- a/arch/avr32/include/asm/unaligned.h
+++ /dev/null
@@ -1,21 +0,0 @@
-#ifndef _ASM_AVR32_UNALIGNED_H
-#define _ASM_AVR32_UNALIGNED_H
-
-/*
- * AVR32 can handle some unaligned accesses, depending on the
- * implementation. The AVR32 AP implementation can handle unaligned
- * words, but halfwords must be halfword-aligned, and doublewords must
- * be word-aligned.
- *
- * However, swapped word loads must be word-aligned so we can't
- * optimize word loads in general.
- */
-
-#include <linux/unaligned/be_struct.h>
-#include <linux/unaligned/le_byteshift.h>
-#include <linux/unaligned/generic.h>
-
-#define get_unaligned __get_unaligned_be
-#define put_unaligned __put_unaligned_be
-
-#endif /* _ASM_AVR32_UNALIGNED_H */
diff --git a/arch/avr32/include/asm/unistd.h b/arch/avr32/include/asm/unistd.h
deleted file mode 100644
index 2011bee..0000000
--- a/arch/avr32/include/asm/unistd.h
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_UNISTD_H
-#define __ASM_AVR32_UNISTD_H
-
-#include <uapi/asm/unistd.h>
-
-#define NR_syscalls 321
-
-/* Old stuff */
-#define __IGNORE_uselib
-#define __IGNORE_mmap
-
-/* NUMA stuff */
-#define __IGNORE_mbind
-#define __IGNORE_get_mempolicy
-#define __IGNORE_set_mempolicy
-#define __IGNORE_migrate_pages
-#define __IGNORE_move_pages
-
-/* SMP stuff */
-#define __IGNORE_getcpu
-
-#define __ARCH_WANT_STAT64
-#define __ARCH_WANT_SYS_ALARM
-#define __ARCH_WANT_SYS_GETHOSTNAME
-#define __ARCH_WANT_SYS_PAUSE
-#define __ARCH_WANT_SYS_TIME
-#define __ARCH_WANT_SYS_UTIME
-#define __ARCH_WANT_SYS_WAITPID
-#define __ARCH_WANT_SYS_FADVISE64
-#define __ARCH_WANT_SYS_GETPGRP
-#define __ARCH_WANT_SYS_LLSEEK
-#define __ARCH_WANT_SYS_GETPGRP
-#define __ARCH_WANT_SYS_FORK
-#define __ARCH_WANT_SYS_VFORK
-#define __ARCH_WANT_SYS_CLONE
-
-#endif /* __ASM_AVR32_UNISTD_H */
diff --git a/arch/avr32/include/asm/user.h b/arch/avr32/include/asm/user.h
deleted file mode 100644
index 7e9152f..0000000
--- a/arch/avr32/include/asm/user.h
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * Note: We may not need these definitions for AVR32, as we don't
- * support a.out.
- */
-#ifndef __ASM_AVR32_USER_H
-#define __ASM_AVR32_USER_H
-
-#include <linux/types.h>
-#include <asm/ptrace.h>
-#include <asm/page.h>
-
-/*
- * Core file format: The core file is written in such a way that gdb
- * can understand it and provide useful information to the user (under
- * linux we use the `trad-core' bfd). The file contents are as follows:
- *
- * upage: 1 page consisting of a user struct that tells gdb
- * what is present in the file. Directly after this is a
- * copy of the task_struct, which is currently not used by gdb,
- * but it may come in handy at some point. All of the registers
- * are stored as part of the upage. The upage should always be
- * only one page long.
- * data: The data segment follows next. We use current->end_text to
- * current->brk to pick up all of the user variables, plus any memory
- * that may have been sbrk'ed. No attempt is made to determine if a
- * page is demand-zero or if a page is totally unused, we just cover
- * the entire range. All of the addresses are rounded in such a way
- * that an integral number of pages is written.
- * stack: We need the stack information in order to get a meaningful
- * backtrace. We need to write the data from usp to
- * current->start_stack, so we round each of these in order to be able
- * to write an integer number of pages.
- */
-
-struct user_fpu_struct {
- /* We have no FPU (yet) */
-};
-
-struct user {
- struct pt_regs regs; /* entire machine state */
- size_t u_tsize; /* text size (pages) */
- size_t u_dsize; /* data size (pages) */
- size_t u_ssize; /* stack size (pages) */
- unsigned long start_code; /* text starting address */
- unsigned long start_data; /* data starting address */
- unsigned long start_stack; /* stack starting address */
- long int signal; /* signal causing core dump */
- unsigned long u_ar0; /* help gdb find registers */
- unsigned long magic; /* identifies a core file */
- char u_comm[32]; /* user command name */
-};
-
-#define NBPG PAGE_SIZE
-#define UPAGES 1
-#define HOST_TEXT_START_ADDR (u.start_code)
-#define HOST_DATA_START_ADDR (u.start_data)
-#define HOST_STACK_END_ADDR (u.start_stack + u.u_ssize * NBPG)
-
-#endif /* __ASM_AVR32_USER_H */
diff --git a/arch/avr32/include/uapi/asm/Kbuild b/arch/avr32/include/uapi/asm/Kbuild
deleted file mode 100644
index 08d8a3d..0000000
--- a/arch/avr32/include/uapi/asm/Kbuild
+++ /dev/null
@@ -1,36 +0,0 @@
-# UAPI Header export list
-include include/uapi/asm-generic/Kbuild.asm
-
-header-y += auxvec.h
-header-y += byteorder.h
-header-y += cachectl.h
-header-y += msgbuf.h
-header-y += param.h
-header-y += posix_types.h
-header-y += ptrace.h
-header-y += sembuf.h
-header-y += setup.h
-header-y += shmbuf.h
-header-y += sigcontext.h
-header-y += signal.h
-header-y += socket.h
-header-y += sockios.h
-header-y += stat.h
-header-y += swab.h
-header-y += termbits.h
-header-y += termios.h
-header-y += types.h
-header-y += unistd.h
-generic-y += bitsperlong.h
-generic-y += errno.h
-generic-y += fcntl.h
-generic-y += ioctl.h
-generic-y += ioctls.h
-generic-y += ipcbuf.h
-generic-y += kvm_para.h
-generic-y += mman.h
-generic-y += param.h
-generic-y += poll.h
-generic-y += resource.h
-generic-y += siginfo.h
-generic-y += statfs.h
diff --git a/arch/avr32/include/uapi/asm/auxvec.h b/arch/avr32/include/uapi/asm/auxvec.h
deleted file mode 100644
index 4f02da3..0000000
--- a/arch/avr32/include/uapi/asm/auxvec.h
+++ /dev/null
@@ -1,4 +0,0 @@
-#ifndef _UAPI__ASM_AVR32_AUXVEC_H
-#define _UAPI__ASM_AVR32_AUXVEC_H
-
-#endif /* _UAPI__ASM_AVR32_AUXVEC_H */
diff --git a/arch/avr32/include/uapi/asm/byteorder.h b/arch/avr32/include/uapi/asm/byteorder.h
deleted file mode 100644
index 71242f0..0000000
--- a/arch/avr32/include/uapi/asm/byteorder.h
+++ /dev/null
@@ -1,9 +0,0 @@
-/*
- * AVR32 endian-conversion functions.
- */
-#ifndef _UAPI__ASM_AVR32_BYTEORDER_H
-#define _UAPI__ASM_AVR32_BYTEORDER_H
-
-#include <linux/byteorder/big_endian.h>
-
-#endif /* _UAPI__ASM_AVR32_BYTEORDER_H */
diff --git a/arch/avr32/include/uapi/asm/cachectl.h b/arch/avr32/include/uapi/asm/cachectl.h
deleted file mode 100644
index 573a958..0000000
--- a/arch/avr32/include/uapi/asm/cachectl.h
+++ /dev/null
@@ -1,11 +0,0 @@
-#ifndef _UAPI__ASM_AVR32_CACHECTL_H
-#define _UAPI__ASM_AVR32_CACHECTL_H
-
-/*
- * Operations that can be performed through the cacheflush system call
- */
-
-/* Clean the data cache, then invalidate the icache */
-#define CACHE_IFLUSH 0
-
-#endif /* _UAPI__ASM_AVR32_CACHECTL_H */
diff --git a/arch/avr32/include/uapi/asm/msgbuf.h b/arch/avr32/include/uapi/asm/msgbuf.h
deleted file mode 100644
index 9eae6ef..0000000
--- a/arch/avr32/include/uapi/asm/msgbuf.h
+++ /dev/null
@@ -1,31 +0,0 @@
-#ifndef _UAPI__ASM_AVR32_MSGBUF_H
-#define _UAPI__ASM_AVR32_MSGBUF_H
-
-/*
- * The msqid64_ds structure for i386 architecture.
- * Note extra padding because this structure is passed back and forth
- * between kernel and user space.
- *
- * Pad space is left for:
- * - 64-bit time_t to solve y2038 problem
- * - 2 miscellaneous 32-bit values
- */
-
-struct msqid64_ds {
- struct ipc64_perm msg_perm;
- __kernel_time_t msg_stime; /* last msgsnd time */
- unsigned long __unused1;
- __kernel_time_t msg_rtime; /* last msgrcv time */
- unsigned long __unused2;
- __kernel_time_t msg_ctime; /* last change time */
- unsigned long __unused3;
- unsigned long msg_cbytes; /* current number of bytes on queue */
- unsigned long msg_qnum; /* number of messages in queue */
- unsigned long msg_qbytes; /* max number of bytes on queue */
- __kernel_pid_t msg_lspid; /* pid of last msgsnd */
- __kernel_pid_t msg_lrpid; /* last receive pid */
- unsigned long __unused4;
- unsigned long __unused5;
-};
-
-#endif /* _UAPI__ASM_AVR32_MSGBUF_H */
diff --git a/arch/avr32/include/uapi/asm/posix_types.h b/arch/avr32/include/uapi/asm/posix_types.h
deleted file mode 100644
index 5b813a8..0000000
--- a/arch/avr32/include/uapi/asm/posix_types.h
+++ /dev/null
@@ -1,37 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef _UAPI__ASM_AVR32_POSIX_TYPES_H
-#define _UAPI__ASM_AVR32_POSIX_TYPES_H
-
-/*
- * This file is generally used by user-level software, so you need to
- * be a little careful about namespace pollution etc. Also, we cannot
- * assume GCC is being used.
- */
-
-typedef unsigned short __kernel_mode_t;
-#define __kernel_mode_t __kernel_mode_t
-
-typedef unsigned short __kernel_ipc_pid_t;
-#define __kernel_ipc_pid_t __kernel_ipc_pid_t
-
-typedef unsigned long __kernel_size_t;
-typedef long __kernel_ssize_t;
-typedef int __kernel_ptrdiff_t;
-#define __kernel_size_t __kernel_size_t
-
-typedef unsigned short __kernel_old_uid_t;
-typedef unsigned short __kernel_old_gid_t;
-#define __kernel_old_uid_t __kernel_old_uid_t
-
-typedef unsigned short __kernel_old_dev_t;
-#define __kernel_old_dev_t __kernel_old_dev_t
-
-#include <asm-generic/posix_types.h>
-
-#endif /* _UAPI__ASM_AVR32_POSIX_TYPES_H */
diff --git a/arch/avr32/include/uapi/asm/ptrace.h b/arch/avr32/include/uapi/asm/ptrace.h
deleted file mode 100644
index fe8c162..0000000
--- a/arch/avr32/include/uapi/asm/ptrace.h
+++ /dev/null
@@ -1,126 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef _UAPI__ASM_AVR32_PTRACE_H
-#define _UAPI__ASM_AVR32_PTRACE_H
-
-#define PTRACE_GETREGS 12
-#define PTRACE_SETREGS 13
-
-/*
- * Status Register bits
- */
-#define SR_H 0x20000000
-#define SR_J 0x10000000
-#define SR_DM 0x08000000
-#define SR_D 0x04000000
-#define MODE_NMI 0x01c00000
-#define MODE_EXCEPTION 0x01800000
-#define MODE_INT3 0x01400000
-#define MODE_INT2 0x01000000
-#define MODE_INT1 0x00c00000
-#define MODE_INT0 0x00800000
-#define MODE_SUPERVISOR 0x00400000
-#define MODE_USER 0x00000000
-#define MODE_MASK 0x01c00000
-#define SR_EM 0x00200000
-#define SR_I3M 0x00100000
-#define SR_I2M 0x00080000
-#define SR_I1M 0x00040000
-#define SR_I0M 0x00020000
-#define SR_GM 0x00010000
-
-#define SR_H_BIT 29
-#define SR_J_BIT 28
-#define SR_DM_BIT 27
-#define SR_D_BIT 26
-#define MODE_SHIFT 22
-#define SR_EM_BIT 21
-#define SR_I3M_BIT 20
-#define SR_I2M_BIT 19
-#define SR_I1M_BIT 18
-#define SR_I0M_BIT 17
-#define SR_GM_BIT 16
-
-/* The user-visible part */
-#define SR_L 0x00000020
-#define SR_Q 0x00000010
-#define SR_V 0x00000008
-#define SR_N 0x00000004
-#define SR_Z 0x00000002
-#define SR_C 0x00000001
-
-#define SR_L_BIT 5
-#define SR_Q_BIT 4
-#define SR_V_BIT 3
-#define SR_N_BIT 2
-#define SR_Z_BIT 1
-#define SR_C_BIT 0
-
-/*
- * The order is defined by the stmts instruction. r0 is stored first,
- * so it gets the highest address.
- *
- * Registers 0-12 are general-purpose registers (r12 is normally used for
- * the function return value).
- * Register 13 is the stack pointer
- * Register 14 is the link register
- * Register 15 is the program counter (retrieved from the RAR sysreg)
- */
-#define FRAME_SIZE_FULL 72
-#define REG_R12_ORIG 68
-#define REG_R0 64
-#define REG_R1 60
-#define REG_R2 56
-#define REG_R3 52
-#define REG_R4 48
-#define REG_R5 44
-#define REG_R6 40
-#define REG_R7 36
-#define REG_R8 32
-#define REG_R9 28
-#define REG_R10 24
-#define REG_R11 20
-#define REG_R12 16
-#define REG_SP 12
-#define REG_LR 8
-
-#define FRAME_SIZE_MIN 8
-#define REG_PC 4
-#define REG_SR 0
-
-#ifndef __ASSEMBLY__
-struct pt_regs {
- /* These are always saved */
- unsigned long sr;
- unsigned long pc;
-
- /* These are sometimes saved */
- unsigned long lr;
- unsigned long sp;
- unsigned long r12;
- unsigned long r11;
- unsigned long r10;
- unsigned long r9;
- unsigned long r8;
- unsigned long r7;
- unsigned long r6;
- unsigned long r5;
- unsigned long r4;
- unsigned long r3;
- unsigned long r2;
- unsigned long r1;
- unsigned long r0;
-
- /* Only saved on system call */
- unsigned long r12_orig;
-};
-
-
-#endif /* ! __ASSEMBLY__ */
-
-#endif /* _UAPI__ASM_AVR32_PTRACE_H */
diff --git a/arch/avr32/include/uapi/asm/sembuf.h b/arch/avr32/include/uapi/asm/sembuf.h
deleted file mode 100644
index 6c6f7cf..0000000
--- a/arch/avr32/include/uapi/asm/sembuf.h
+++ /dev/null
@@ -1,25 +0,0 @@
-#ifndef _UAPI__ASM_AVR32_SEMBUF_H
-#define _UAPI__ASM_AVR32_SEMBUF_H
-
-/*
-* The semid64_ds structure for AVR32 architecture.
- * Note extra padding because this structure is passed back and forth
- * between kernel and user space.
- *
- * Pad space is left for:
- * - 64-bit time_t to solve y2038 problem
- * - 2 miscellaneous 32-bit values
- */
-
-struct semid64_ds {
- struct ipc64_perm sem_perm; /* permissions .. see ipc.h */
- __kernel_time_t sem_otime; /* last semop time */
- unsigned long __unused1;
- __kernel_time_t sem_ctime; /* last change time */
- unsigned long __unused2;
- unsigned long sem_nsems; /* no. of semaphores in array */
- unsigned long __unused3;
- unsigned long __unused4;
-};
-
-#endif /* _UAPI__ASM_AVR32_SEMBUF_H */
diff --git a/arch/avr32/include/uapi/asm/setup.h b/arch/avr32/include/uapi/asm/setup.h
deleted file mode 100644
index a654df7..0000000
--- a/arch/avr32/include/uapi/asm/setup.h
+++ /dev/null
@@ -1,16 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * Based on linux/include/asm-arm/setup.h
- * Copyright (C) 1997-1999 Russell King
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef _UAPI__ASM_AVR32_SETUP_H__
-#define _UAPI__ASM_AVR32_SETUP_H__
-
-#define COMMAND_LINE_SIZE 256
-
-#endif /* _UAPI__ASM_AVR32_SETUP_H__ */
diff --git a/arch/avr32/include/uapi/asm/shmbuf.h b/arch/avr32/include/uapi/asm/shmbuf.h
deleted file mode 100644
index b94cf8b..0000000
--- a/arch/avr32/include/uapi/asm/shmbuf.h
+++ /dev/null
@@ -1,42 +0,0 @@
-#ifndef _UAPI__ASM_AVR32_SHMBUF_H
-#define _UAPI__ASM_AVR32_SHMBUF_H
-
-/*
- * The shmid64_ds structure for i386 architecture.
- * Note extra padding because this structure is passed back and forth
- * between kernel and user space.
- *
- * Pad space is left for:
- * - 64-bit time_t to solve y2038 problem
- * - 2 miscellaneous 32-bit values
- */
-
-struct shmid64_ds {
- struct ipc64_perm shm_perm; /* operation perms */
- size_t shm_segsz; /* size of segment (bytes) */
- __kernel_time_t shm_atime; /* last attach time */
- unsigned long __unused1;
- __kernel_time_t shm_dtime; /* last detach time */
- unsigned long __unused2;
- __kernel_time_t shm_ctime; /* last change time */
- unsigned long __unused3;
- __kernel_pid_t shm_cpid; /* pid of creator */
- __kernel_pid_t shm_lpid; /* pid of last operator */
- unsigned long shm_nattch; /* no. of current attaches */
- unsigned long __unused4;
- unsigned long __unused5;
-};
-
-struct shminfo64 {
- unsigned long shmmax;
- unsigned long shmmin;
- unsigned long shmmni;
- unsigned long shmseg;
- unsigned long shmall;
- unsigned long __unused1;
- unsigned long __unused2;
- unsigned long __unused3;
- unsigned long __unused4;
-};
-
-#endif /* _UAPI__ASM_AVR32_SHMBUF_H */
diff --git a/arch/avr32/include/uapi/asm/sigcontext.h b/arch/avr32/include/uapi/asm/sigcontext.h
deleted file mode 100644
index 27e56bf..0000000
--- a/arch/avr32/include/uapi/asm/sigcontext.h
+++ /dev/null
@@ -1,34 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef _UAPI__ASM_AVR32_SIGCONTEXT_H
-#define _UAPI__ASM_AVR32_SIGCONTEXT_H
-
-struct sigcontext {
- unsigned long oldmask;
-
- /* CPU registers */
- unsigned long sr;
- unsigned long pc;
- unsigned long lr;
- unsigned long sp;
- unsigned long r12;
- unsigned long r11;
- unsigned long r10;
- unsigned long r9;
- unsigned long r8;
- unsigned long r7;
- unsigned long r6;
- unsigned long r5;
- unsigned long r4;
- unsigned long r3;
- unsigned long r2;
- unsigned long r1;
- unsigned long r0;
-};
-
-#endif /* _UAPI__ASM_AVR32_SIGCONTEXT_H */
diff --git a/arch/avr32/include/uapi/asm/signal.h b/arch/avr32/include/uapi/asm/signal.h
deleted file mode 100644
index ffe8c77..0000000
--- a/arch/avr32/include/uapi/asm/signal.h
+++ /dev/null
@@ -1,121 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef _UAPI__ASM_AVR32_SIGNAL_H
-#define _UAPI__ASM_AVR32_SIGNAL_H
-
-#include <linux/types.h>
-
-/* Avoid too many header ordering problems. */
-struct siginfo;
-
-#ifndef __KERNEL__
-/* Here we must cater to libcs that poke about in kernel headers. */
-
-#define NSIG 32
-typedef unsigned long sigset_t;
-
-#endif /* __KERNEL__ */
-
-#define SIGHUP 1
-#define SIGINT 2
-#define SIGQUIT 3
-#define SIGILL 4
-#define SIGTRAP 5
-#define SIGABRT 6
-#define SIGIOT 6
-#define SIGBUS 7
-#define SIGFPE 8
-#define SIGKILL 9
-#define SIGUSR1 10
-#define SIGSEGV 11
-#define SIGUSR2 12
-#define SIGPIPE 13
-#define SIGALRM 14
-#define SIGTERM 15
-#define SIGSTKFLT 16
-#define SIGCHLD 17
-#define SIGCONT 18
-#define SIGSTOP 19
-#define SIGTSTP 20
-#define SIGTTIN 21
-#define SIGTTOU 22
-#define SIGURG 23
-#define SIGXCPU 24
-#define SIGXFSZ 25
-#define SIGVTALRM 26
-#define SIGPROF 27
-#define SIGWINCH 28
-#define SIGIO 29
-#define SIGPOLL SIGIO
-/*
-#define SIGLOST 29
-*/
-#define SIGPWR 30
-#define SIGSYS 31
-#define SIGUNUSED 31
-
-/* These should not be considered constants from userland. */
-#define SIGRTMIN 32
-#define SIGRTMAX (_NSIG-1)
-
-/*
- * SA_FLAGS values:
- *
- * SA_NOCLDSTOP flag to turn off SIGCHLD when children stop.
- * SA_NOCLDWAIT flag on SIGCHLD to inhibit zombies.
- * SA_SIGINFO deliver the signal with SIGINFO structs
- * SA_ONSTACK indicates that a registered stack_t will be used.
- * SA_RESTART flag to get restarting signals (which were the default long ago)
- * SA_NODEFER prevents the current signal from being masked in the handler.
- * SA_RESETHAND clears the handler when the signal is delivered.
- *
- * SA_ONESHOT and SA_NOMASK are the historical Linux names for the Single
- * Unix names RESETHAND and NODEFER respectively.
- */
-#define SA_NOCLDSTOP 0x00000001
-#define SA_NOCLDWAIT 0x00000002
-#define SA_SIGINFO 0x00000004
-#define SA_RESTORER 0x04000000
-#define SA_ONSTACK 0x08000000
-#define SA_RESTART 0x10000000
-#define SA_NODEFER 0x40000000
-#define SA_RESETHAND 0x80000000
-
-#define SA_NOMASK SA_NODEFER
-#define SA_ONESHOT SA_RESETHAND
-
-#define MINSIGSTKSZ 2048
-#define SIGSTKSZ 8192
-
-#include <asm-generic/signal-defs.h>
-
-#ifndef __KERNEL__
-/* Here we must cater to libcs that poke about in kernel headers. */
-
-struct sigaction {
- union {
- __sighandler_t _sa_handler;
- void (*_sa_sigaction)(int, struct siginfo *, void *);
- } _u;
- sigset_t sa_mask;
- unsigned long sa_flags;
- void (*sa_restorer)(void);
-};
-
-#define sa_handler _u._sa_handler
-#define sa_sigaction _u._sa_sigaction
-
-#endif /* __KERNEL__ */
-
-typedef struct sigaltstack {
- void __user *ss_sp;
- int ss_flags;
- size_t ss_size;
-} stack_t;
-
-#endif /* _UAPI__ASM_AVR32_SIGNAL_H */
diff --git a/arch/avr32/include/uapi/asm/socket.h b/arch/avr32/include/uapi/asm/socket.h
deleted file mode 100644
index 5a65042..0000000
--- a/arch/avr32/include/uapi/asm/socket.h
+++ /dev/null
@@ -1,95 +0,0 @@
-#ifndef _UAPI__ASM_AVR32_SOCKET_H
-#define _UAPI__ASM_AVR32_SOCKET_H
-
-#include <asm/sockios.h>
-
-/* For setsockopt(2) */
-#define SOL_SOCKET 1
-
-#define SO_DEBUG 1
-#define SO_REUSEADDR 2
-#define SO_TYPE 3
-#define SO_ERROR 4
-#define SO_DONTROUTE 5
-#define SO_BROADCAST 6
-#define SO_SNDBUF 7
-#define SO_RCVBUF 8
-#define SO_SNDBUFFORCE 32
-#define SO_RCVBUFFORCE 33
-#define SO_KEEPALIVE 9
-#define SO_OOBINLINE 10
-#define SO_NO_CHECK 11
-#define SO_PRIORITY 12
-#define SO_LINGER 13
-#define SO_BSDCOMPAT 14
-#define SO_REUSEPORT 15
-#define SO_PASSCRED 16
-#define SO_PEERCRED 17
-#define SO_RCVLOWAT 18
-#define SO_SNDLOWAT 19
-#define SO_RCVTIMEO 20
-#define SO_SNDTIMEO 21
-
-/* Security levels - as per NRL IPv6 - don't actually do anything */
-#define SO_SECURITY_AUTHENTICATION 22
-#define SO_SECURITY_ENCRYPTION_TRANSPORT 23
-#define SO_SECURITY_ENCRYPTION_NETWORK 24
-
-#define SO_BINDTODEVICE 25
-
-/* Socket filtering */
-#define SO_ATTACH_FILTER 26
-#define SO_DETACH_FILTER 27
-#define SO_GET_FILTER SO_ATTACH_FILTER
-
-#define SO_PEERNAME 28
-#define SO_TIMESTAMP 29
-#define SCM_TIMESTAMP SO_TIMESTAMP
-
-#define SO_ACCEPTCONN 30
-
-#define SO_PEERSEC 31
-#define SO_PASSSEC 34
-#define SO_TIMESTAMPNS 35
-#define SCM_TIMESTAMPNS SO_TIMESTAMPNS
-
-#define SO_MARK 36
-
-#define SO_TIMESTAMPING 37
-#define SCM_TIMESTAMPING SO_TIMESTAMPING
-
-#define SO_PROTOCOL 38
-#define SO_DOMAIN 39
-
-#define SO_RXQ_OVFL 40
-
-#define SO_WIFI_STATUS 41
-#define SCM_WIFI_STATUS SO_WIFI_STATUS
-#define SO_PEEK_OFF 42
-
-/* Instruct lower device to use last 4-bytes of skb data as FCS */
-#define SO_NOFCS 43
-
-#define SO_LOCK_FILTER 44
-
-#define SO_SELECT_ERR_QUEUE 45
-
-#define SO_BUSY_POLL 46
-
-#define SO_MAX_PACING_RATE 47
-
-#define SO_BPF_EXTENSIONS 48
-
-#define SO_INCOMING_CPU 49
-
-#define SO_ATTACH_BPF 50
-#define SO_DETACH_BPF SO_DETACH_FILTER
-
-#define SO_ATTACH_REUSEPORT_CBPF 51
-#define SO_ATTACH_REUSEPORT_EBPF 52
-
-#define SO_CNX_ADVICE 53
-
-#define SCM_TIMESTAMPING_OPT_STATS 54
-
-#endif /* _UAPI__ASM_AVR32_SOCKET_H */
diff --git a/arch/avr32/include/uapi/asm/sockios.h b/arch/avr32/include/uapi/asm/sockios.h
deleted file mode 100644
index d04785453..0000000
--- a/arch/avr32/include/uapi/asm/sockios.h
+++ /dev/null
@@ -1,13 +0,0 @@
-#ifndef _UAPI__ASM_AVR32_SOCKIOS_H
-#define _UAPI__ASM_AVR32_SOCKIOS_H
-
-/* Socket-level I/O control calls. */
-#define FIOSETOWN 0x8901
-#define SIOCSPGRP 0x8902
-#define FIOGETOWN 0x8903
-#define SIOCGPGRP 0x8904
-#define SIOCATMARK 0x8905
-#define SIOCGSTAMP 0x8906 /* Get stamp (timeval) */
-#define SIOCGSTAMPNS 0x8907 /* Get stamp (timespec) */
-
-#endif /* _UAPI__ASM_AVR32_SOCKIOS_H */
diff --git a/arch/avr32/include/uapi/asm/stat.h b/arch/avr32/include/uapi/asm/stat.h
deleted file mode 100644
index c06acef..0000000
--- a/arch/avr32/include/uapi/asm/stat.h
+++ /dev/null
@@ -1,79 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef _UAPI__ASM_AVR32_STAT_H
-#define _UAPI__ASM_AVR32_STAT_H
-
-struct __old_kernel_stat {
- unsigned short st_dev;
- unsigned short st_ino;
- unsigned short st_mode;
- unsigned short st_nlink;
- unsigned short st_uid;
- unsigned short st_gid;
- unsigned short st_rdev;
- unsigned long st_size;
- unsigned long st_atime;
- unsigned long st_mtime;
- unsigned long st_ctime;
-};
-
-struct stat {
- unsigned long st_dev;
- unsigned long st_ino;
- unsigned short st_mode;
- unsigned short st_nlink;
- unsigned short st_uid;
- unsigned short st_gid;
- unsigned long st_rdev;
- unsigned long st_size;
- unsigned long st_blksize;
- unsigned long st_blocks;
- unsigned long st_atime;
- unsigned long st_atime_nsec;
- unsigned long st_mtime;
- unsigned long st_mtime_nsec;
- unsigned long st_ctime;
- unsigned long st_ctime_nsec;
- unsigned long __unused4;
- unsigned long __unused5;
-};
-
-#define STAT_HAVE_NSEC 1
-
-struct stat64 {
- unsigned long long st_dev;
-
- unsigned long long st_ino;
- unsigned int st_mode;
- unsigned int st_nlink;
-
- unsigned long st_uid;
- unsigned long st_gid;
-
- unsigned long long st_rdev;
-
- long long st_size;
- unsigned long __pad1; /* align 64-bit st_blocks */
- unsigned long st_blksize;
-
- unsigned long long st_blocks; /* Number 512-byte blocks allocated. */
-
- unsigned long st_atime;
- unsigned long st_atime_nsec;
-
- unsigned long st_mtime;
- unsigned long st_mtime_nsec;
-
- unsigned long st_ctime;
- unsigned long st_ctime_nsec;
-
- unsigned long __unused1;
- unsigned long __unused2;
-};
-
-#endif /* _UAPI__ASM_AVR32_STAT_H */
diff --git a/arch/avr32/include/uapi/asm/swab.h b/arch/avr32/include/uapi/asm/swab.h
deleted file mode 100644
index 1a03549..0000000
--- a/arch/avr32/include/uapi/asm/swab.h
+++ /dev/null
@@ -1,35 +0,0 @@
-/*
- * AVR32 byteswapping functions.
- */
-#ifndef _UAPI__ASM_AVR32_SWAB_H
-#define _UAPI__ASM_AVR32_SWAB_H
-
-#include <linux/types.h>
-#include <linux/compiler.h>
-
-#define __SWAB_64_THRU_32__
-
-#ifdef __CHECKER__
-extern unsigned long __builtin_bswap_32(unsigned long x);
-extern unsigned short __builtin_bswap_16(unsigned short x);
-#endif
-
-/*
- * avr32-linux-gcc versions earlier than 4.2 improperly sign-extends
- * the result.
- */
-#if !(__GNUC__ == 4 && __GNUC_MINOR__ < 2)
-static inline __attribute_const__ __u16 __arch_swab16(__u16 val)
-{
- return __builtin_bswap_16(val);
-}
-#define __arch_swab16 __arch_swab16
-
-static inline __attribute_const__ __u32 __arch_swab32(__u32 val)
-{
- return __builtin_bswap_32(val);
-}
-#define __arch_swab32 __arch_swab32
-#endif
-
-#endif /* _UAPI__ASM_AVR32_SWAB_H */
diff --git a/arch/avr32/include/uapi/asm/termbits.h b/arch/avr32/include/uapi/asm/termbits.h
deleted file mode 100644
index 32789cc..0000000
--- a/arch/avr32/include/uapi/asm/termbits.h
+++ /dev/null
@@ -1,196 +0,0 @@
-#ifndef _UAPI__ASM_AVR32_TERMBITS_H
-#define _UAPI__ASM_AVR32_TERMBITS_H
-
-#include <linux/posix_types.h>
-
-typedef unsigned char cc_t;
-typedef unsigned int speed_t;
-typedef unsigned int tcflag_t;
-
-#define NCCS 19
-struct termios {
- tcflag_t c_iflag; /* input mode flags */
- tcflag_t c_oflag; /* output mode flags */
- tcflag_t c_cflag; /* control mode flags */
- tcflag_t c_lflag; /* local mode flags */
- cc_t c_line; /* line discipline */
- cc_t c_cc[NCCS]; /* control characters */
-};
-
-struct termios2 {
- tcflag_t c_iflag; /* input mode flags */
- tcflag_t c_oflag; /* output mode flags */
- tcflag_t c_cflag; /* control mode flags */
- tcflag_t c_lflag; /* local mode flags */
- cc_t c_line; /* line discipline */
- cc_t c_cc[NCCS]; /* control characters */
- speed_t c_ispeed; /* input speed */
- speed_t c_ospeed; /* output speed */
-};
-
-struct ktermios {
- tcflag_t c_iflag; /* input mode flags */
- tcflag_t c_oflag; /* output mode flags */
- tcflag_t c_cflag; /* control mode flags */
- tcflag_t c_lflag; /* local mode flags */
- cc_t c_line; /* line discipline */
- cc_t c_cc[NCCS]; /* control characters */
- speed_t c_ispeed; /* input speed */
- speed_t c_ospeed; /* output speed */
-};
-
-/* c_cc characters */
-#define VINTR 0
-#define VQUIT 1
-#define VERASE 2
-#define VKILL 3
-#define VEOF 4
-#define VTIME 5
-#define VMIN 6
-#define VSWTC 7
-#define VSTART 8
-#define VSTOP 9
-#define VSUSP 10
-#define VEOL 11
-#define VREPRINT 12
-#define VDISCARD 13
-#define VWERASE 14
-#define VLNEXT 15
-#define VEOL2 16
-
-/* c_iflag bits */
-#define IGNBRK 0000001
-#define BRKINT 0000002
-#define IGNPAR 0000004
-#define PARMRK 0000010
-#define INPCK 0000020
-#define ISTRIP 0000040
-#define INLCR 0000100
-#define IGNCR 0000200
-#define ICRNL 0000400
-#define IUCLC 0001000
-#define IXON 0002000
-#define IXANY 0004000
-#define IXOFF 0010000
-#define IMAXBEL 0020000
-#define IUTF8 0040000
-
-/* c_oflag bits */
-#define OPOST 0000001
-#define OLCUC 0000002
-#define ONLCR 0000004
-#define OCRNL 0000010
-#define ONOCR 0000020
-#define ONLRET 0000040
-#define OFILL 0000100
-#define OFDEL 0000200
-#define NLDLY 0000400
-#define NL0 0000000
-#define NL1 0000400
-#define CRDLY 0003000
-#define CR0 0000000
-#define CR1 0001000
-#define CR2 0002000
-#define CR3 0003000
-#define TABDLY 0014000
-#define TAB0 0000000
-#define TAB1 0004000
-#define TAB2 0010000
-#define TAB3 0014000
-#define XTABS 0014000
-#define BSDLY 0020000
-#define BS0 0000000
-#define BS1 0020000
-#define VTDLY 0040000
-#define VT0 0000000
-#define VT1 0040000
-#define FFDLY 0100000
-#define FF0 0000000
-#define FF1 0100000
-
-/* c_cflag bit meaning */
-#define CBAUD 0010017
-#define B0 0000000 /* hang up */
-#define B50 0000001
-#define B75 0000002
-#define B110 0000003
-#define B134 0000004
-#define B150 0000005
-#define B200 0000006
-#define B300 0000007
-#define B600 0000010
-#define B1200 0000011
-#define B1800 0000012
-#define B2400 0000013
-#define B4800 0000014
-#define B9600 0000015
-#define B19200 0000016
-#define B38400 0000017
-#define EXTA B19200
-#define EXTB B38400
-#define CSIZE 0000060
-#define CS5 0000000
-#define CS6 0000020
-#define CS7 0000040
-#define CS8 0000060
-#define CSTOPB 0000100
-#define CREAD 0000200
-#define PARENB 0000400
-#define PARODD 0001000
-#define HUPCL 0002000
-#define CLOCAL 0004000
-#define CBAUDEX 0010000
-#define B57600 0010001
-#define B115200 0010002
-#define B230400 0010003
-#define B460800 0010004
-#define B500000 0010005
-#define B576000 0010006
-#define B921600 0010007
-#define B1000000 0010010
-#define B1152000 0010011
-#define B1500000 0010012
-#define B2000000 0010013
-#define B2500000 0010014
-#define B3000000 0010015
-#define B3500000 0010016
-#define B4000000 0010017
-#define CIBAUD 002003600000 /* input baud rate (not used) */
-#define CMSPAR 010000000000 /* mark or space (stick) parity */
-#define CRTSCTS 020000000000 /* flow control */
-
-/* c_lflag bits */
-#define ISIG 0000001
-#define ICANON 0000002
-#define XCASE 0000004
-#define ECHO 0000010
-#define ECHOE 0000020
-#define ECHOK 0000040
-#define ECHONL 0000100
-#define NOFLSH 0000200
-#define TOSTOP 0000400
-#define ECHOCTL 0001000
-#define ECHOPRT 0002000
-#define ECHOKE 0004000
-#define FLUSHO 0010000
-#define PENDIN 0040000
-#define IEXTEN 0100000
-#define EXTPROC 0200000
-
-/* tcflow() and TCXONC use these */
-#define TCOOFF 0
-#define TCOON 1
-#define TCIOFF 2
-#define TCION 3
-
-/* tcflush() and TCFLSH use these */
-#define TCIFLUSH 0
-#define TCOFLUSH 1
-#define TCIOFLUSH 2
-
-/* tcsetattr uses these */
-#define TCSANOW 0
-#define TCSADRAIN 1
-#define TCSAFLUSH 2
-
-#endif /* _UAPI__ASM_AVR32_TERMBITS_H */
diff --git a/arch/avr32/include/uapi/asm/termios.h b/arch/avr32/include/uapi/asm/termios.h
deleted file mode 100644
index c8a0081..0000000
--- a/arch/avr32/include/uapi/asm/termios.h
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef _UAPI__ASM_AVR32_TERMIOS_H
-#define _UAPI__ASM_AVR32_TERMIOS_H
-
-#include <asm/termbits.h>
-#include <asm/ioctls.h>
-
-struct winsize {
- unsigned short ws_row;
- unsigned short ws_col;
- unsigned short ws_xpixel;
- unsigned short ws_ypixel;
-};
-
-#define NCC 8
-struct termio {
- unsigned short c_iflag; /* input mode flags */
- unsigned short c_oflag; /* output mode flags */
- unsigned short c_cflag; /* control mode flags */
- unsigned short c_lflag; /* local mode flags */
- unsigned char c_line; /* line discipline */
- unsigned char c_cc[NCC]; /* control characters */
-};
-
-/* modem lines */
-#define TIOCM_LE 0x001
-#define TIOCM_DTR 0x002
-#define TIOCM_RTS 0x004
-#define TIOCM_ST 0x008
-#define TIOCM_SR 0x010
-#define TIOCM_CTS 0x020
-#define TIOCM_CAR 0x040
-#define TIOCM_RNG 0x080
-#define TIOCM_DSR 0x100
-#define TIOCM_CD TIOCM_CAR
-#define TIOCM_RI TIOCM_RNG
-#define TIOCM_OUT1 0x2000
-#define TIOCM_OUT2 0x4000
-#define TIOCM_LOOP 0x8000
-
-/* ioctl (fd, TIOCSERGETLSR, &result) where result may be as below */
-
-#endif /* _UAPI__ASM_AVR32_TERMIOS_H */
diff --git a/arch/avr32/include/uapi/asm/types.h b/arch/avr32/include/uapi/asm/types.h
deleted file mode 100644
index 7c986c4..0000000
--- a/arch/avr32/include/uapi/asm/types.h
+++ /dev/null
@@ -1,13 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef _UAPI__ASM_AVR32_TYPES_H
-#define _UAPI__ASM_AVR32_TYPES_H
-
-#include <asm-generic/int-ll64.h>
-
-#endif /* _UAPI__ASM_AVR32_TYPES_H */
diff --git a/arch/avr32/include/uapi/asm/unistd.h b/arch/avr32/include/uapi/asm/unistd.h
deleted file mode 100644
index 236505d..0000000
--- a/arch/avr32/include/uapi/asm/unistd.h
+++ /dev/null
@@ -1,347 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef _UAPI__ASM_AVR32_UNISTD_H
-#define _UAPI__ASM_AVR32_UNISTD_H
-
-/*
- * This file contains the system call numbers.
- */
-
-#define __NR_restart_syscall 0
-#define __NR_exit 1
-#define __NR_fork 2
-#define __NR_read 3
-#define __NR_write 4
-#define __NR_open 5
-#define __NR_close 6
-#define __NR_umask 7
-#define __NR_creat 8
-#define __NR_link 9
-#define __NR_unlink 10
-#define __NR_execve 11
-#define __NR_chdir 12
-#define __NR_time 13
-#define __NR_mknod 14
-#define __NR_chmod 15
-#define __NR_chown 16
-#define __NR_lchown 17
-#define __NR_lseek 18
-#define __NR__llseek 19
-#define __NR_getpid 20
-#define __NR_mount 21
-#define __NR_umount2 22
-#define __NR_setuid 23
-#define __NR_getuid 24
-#define __NR_stime 25
-#define __NR_ptrace 26
-#define __NR_alarm 27
-#define __NR_pause 28
-#define __NR_utime 29
-#define __NR_stat 30
-#define __NR_fstat 31
-#define __NR_lstat 32
-#define __NR_access 33
-#define __NR_chroot 34
-#define __NR_sync 35
-#define __NR_fsync 36
-#define __NR_kill 37
-#define __NR_rename 38
-#define __NR_mkdir 39
-#define __NR_rmdir 40
-#define __NR_dup 41
-#define __NR_pipe 42
-#define __NR_times 43
-#define __NR_clone 44
-#define __NR_brk 45
-#define __NR_setgid 46
-#define __NR_getgid 47
-#define __NR_getcwd 48
-#define __NR_geteuid 49
-#define __NR_getegid 50
-#define __NR_acct 51
-#define __NR_setfsuid 52
-#define __NR_setfsgid 53
-#define __NR_ioctl 54
-#define __NR_fcntl 55
-#define __NR_setpgid 56
-#define __NR_mremap 57
-#define __NR_setresuid 58
-#define __NR_getresuid 59
-#define __NR_setreuid 60
-#define __NR_setregid 61
-#define __NR_ustat 62
-#define __NR_dup2 63
-#define __NR_getppid 64
-#define __NR_getpgrp 65
-#define __NR_setsid 66
-#define __NR_rt_sigaction 67
-#define __NR_rt_sigreturn 68
-#define __NR_rt_sigprocmask 69
-#define __NR_rt_sigpending 70
-#define __NR_rt_sigtimedwait 71
-#define __NR_rt_sigqueueinfo 72
-#define __NR_rt_sigsuspend 73
-#define __NR_sethostname 74
-#define __NR_setrlimit 75
-#define __NR_getrlimit 76 /* SuS compliant getrlimit */
-#define __NR_getrusage 77
-#define __NR_gettimeofday 78
-#define __NR_settimeofday 79
-#define __NR_getgroups 80
-#define __NR_setgroups 81
-#define __NR_select 82
-#define __NR_symlink 83
-#define __NR_fchdir 84
-#define __NR_readlink 85
-#define __NR_pread 86
-#define __NR_pwrite 87
-#define __NR_swapon 88
-#define __NR_reboot 89
-#define __NR_mmap2 90
-#define __NR_munmap 91
-#define __NR_truncate 92
-#define __NR_ftruncate 93
-#define __NR_fchmod 94
-#define __NR_fchown 95
-#define __NR_getpriority 96
-#define __NR_setpriority 97
-#define __NR_wait4 98
-#define __NR_statfs 99
-#define __NR_fstatfs 100
-#define __NR_vhangup 101
-#define __NR_sigaltstack 102
-#define __NR_syslog 103
-#define __NR_setitimer 104
-#define __NR_getitimer 105
-#define __NR_swapoff 106
-#define __NR_sysinfo 107
-/* 108 was __NR_ipc for a little while */
-#define __NR_sendfile 109
-#define __NR_setdomainname 110
-#define __NR_uname 111
-#define __NR_adjtimex 112
-#define __NR_mprotect 113
-#define __NR_vfork 114
-#define __NR_init_module 115
-#define __NR_delete_module 116
-#define __NR_quotactl 117
-#define __NR_getpgid 118
-#define __NR_bdflush 119
-#define __NR_sysfs 120
-#define __NR_personality 121
-#define __NR_afs_syscall 122 /* Syscall for Andrew File System */
-#define __NR_getdents 123
-#define __NR_flock 124
-#define __NR_msync 125
-#define __NR_readv 126
-#define __NR_writev 127
-#define __NR_getsid 128
-#define __NR_fdatasync 129
-#define __NR__sysctl 130
-#define __NR_mlock 131
-#define __NR_munlock 132
-#define __NR_mlockall 133
-#define __NR_munlockall 134
-#define __NR_sched_setparam 135
-#define __NR_sched_getparam 136
-#define __NR_sched_setscheduler 137
-#define __NR_sched_getscheduler 138
-#define __NR_sched_yield 139
-#define __NR_sched_get_priority_max 140
-#define __NR_sched_get_priority_min 141
-#define __NR_sched_rr_get_interval 142
-#define __NR_nanosleep 143
-#define __NR_poll 144
-#define __NR_nfsservctl 145
-#define __NR_setresgid 146
-#define __NR_getresgid 147
-#define __NR_prctl 148
-#define __NR_socket 149
-#define __NR_bind 150
-#define __NR_connect 151
-#define __NR_listen 152
-#define __NR_accept 153
-#define __NR_getsockname 154
-#define __NR_getpeername 155
-#define __NR_socketpair 156
-#define __NR_send 157
-#define __NR_recv 158
-#define __NR_sendto 159
-#define __NR_recvfrom 160
-#define __NR_shutdown 161
-#define __NR_setsockopt 162
-#define __NR_getsockopt 163
-#define __NR_sendmsg 164
-#define __NR_recvmsg 165
-#define __NR_truncate64 166
-#define __NR_ftruncate64 167
-#define __NR_stat64 168
-#define __NR_lstat64 169
-#define __NR_fstat64 170
-#define __NR_pivot_root 171
-#define __NR_mincore 172
-#define __NR_madvise 173
-#define __NR_getdents64 174
-#define __NR_fcntl64 175
-#define __NR_gettid 176
-#define __NR_readahead 177
-#define __NR_setxattr 178
-#define __NR_lsetxattr 179
-#define __NR_fsetxattr 180
-#define __NR_getxattr 181
-#define __NR_lgetxattr 182
-#define __NR_fgetxattr 183
-#define __NR_listxattr 184
-#define __NR_llistxattr 185
-#define __NR_flistxattr 186
-#define __NR_removexattr 187
-#define __NR_lremovexattr 188
-#define __NR_fremovexattr 189
-#define __NR_tkill 190
-#define __NR_sendfile64 191
-#define __NR_futex 192
-#define __NR_sched_setaffinity 193
-#define __NR_sched_getaffinity 194
-#define __NR_capget 195
-#define __NR_capset 196
-#define __NR_io_setup 197
-#define __NR_io_destroy 198
-#define __NR_io_getevents 199
-#define __NR_io_submit 200
-#define __NR_io_cancel 201
-#define __NR_fadvise64 202
-#define __NR_exit_group 203
-#define __NR_lookup_dcookie 204
-#define __NR_epoll_create 205
-#define __NR_epoll_ctl 206
-#define __NR_epoll_wait 207
-#define __NR_remap_file_pages 208
-#define __NR_set_tid_address 209
-#define __NR_timer_create 210
-#define __NR_timer_settime 211
-#define __NR_timer_gettime 212
-#define __NR_timer_getoverrun 213
-#define __NR_timer_delete 214
-#define __NR_clock_settime 215
-#define __NR_clock_gettime 216
-#define __NR_clock_getres 217
-#define __NR_clock_nanosleep 218
-#define __NR_statfs64 219
-#define __NR_fstatfs64 220
-#define __NR_tgkill 221
-/* 222 reserved for tux */
-#define __NR_utimes 223
-#define __NR_fadvise64_64 224
-#define __NR_cacheflush 225
-#define __NR_vserver 226
-#define __NR_mq_open 227
-#define __NR_mq_unlink 228
-#define __NR_mq_timedsend 229
-#define __NR_mq_timedreceive 230
-#define __NR_mq_notify 231
-#define __NR_mq_getsetattr 232
-#define __NR_kexec_load 233
-#define __NR_waitid 234
-#define __NR_add_key 235
-#define __NR_request_key 236
-#define __NR_keyctl 237
-#define __NR_ioprio_set 238
-#define __NR_ioprio_get 239
-#define __NR_inotify_init 240
-#define __NR_inotify_add_watch 241
-#define __NR_inotify_rm_watch 242
-#define __NR_openat 243
-#define __NR_mkdirat 244
-#define __NR_mknodat 245
-#define __NR_fchownat 246
-#define __NR_futimesat 247
-#define __NR_fstatat64 248
-#define __NR_unlinkat 249
-#define __NR_renameat 250
-#define __NR_linkat 251
-#define __NR_symlinkat 252
-#define __NR_readlinkat 253
-#define __NR_fchmodat 254
-#define __NR_faccessat 255
-#define __NR_pselect6 256
-#define __NR_ppoll 257
-#define __NR_unshare 258
-#define __NR_set_robust_list 259
-#define __NR_get_robust_list 260
-#define __NR_splice 261
-#define __NR_sync_file_range 262
-#define __NR_tee 263
-#define __NR_vmsplice 264
-#define __NR_epoll_pwait 265
-#define __NR_msgget 266
-#define __NR_msgsnd 267
-#define __NR_msgrcv 268
-#define __NR_msgctl 269
-#define __NR_semget 270
-#define __NR_semop 271
-#define __NR_semctl 272
-#define __NR_semtimedop 273
-#define __NR_shmat 274
-#define __NR_shmget 275
-#define __NR_shmdt 276
-#define __NR_shmctl 277
-#define __NR_utimensat 278
-#define __NR_signalfd 279
-/* 280 was __NR_timerfd */
-#define __NR_eventfd 281
-/* 282 was half-implemented __NR_recvmmsg */
-#define __NR_setns 283
-#define __NR_pread64 284
-#define __NR_pwrite64 285
-#define __NR_timerfd_create 286
-#define __NR_fallocate 287
-#define __NR_timerfd_settime 288
-#define __NR_timerfd_gettime 289
-#define __NR_signalfd4 290
-#define __NR_eventfd2 291
-#define __NR_epoll_create1 292
-#define __NR_dup3 293
-#define __NR_pipe2 294
-#define __NR_inotify_init1 295
-#define __NR_preadv 296
-#define __NR_pwritev 297
-#define __NR_rt_tgsigqueueinfo 298
-#define __NR_perf_event_open 299
-#define __NR_recvmmsg 300
-#define __NR_fanotify_init 301
-#define __NR_fanotify_mark 302
-#define __NR_prlimit64 303
-#define __NR_name_to_handle_at 304
-#define __NR_open_by_handle_at 305
-#define __NR_clock_adjtime 306
-#define __NR_syncfs 307
-#define __NR_sendmmsg 308
-#define __NR_process_vm_readv 309
-#define __NR_process_vm_writev 310
-#define __NR_kcmp 311
-#define __NR_finit_module 312
-#define __NR_sched_setattr 313
-#define __NR_sched_getattr 314
-#define __NR_renameat2 315
-#define __NR_seccomp 316
-#define __NR_getrandom 317
-#define __NR_memfd_create 318
-#define __NR_bpf 319
-#define __NR_execveat 320
-#define __NR_accept4 321
-#define __NR_userfaultfd 322
-#define __NR_membarrier 323
-#define __NR_mlock2 324
-#define __NR_copy_file_range 325
-#define __NR_preadv2 326
-#define __NR_pwritev2 327
-#define __NR_pkey_mprotect 328
-#define __NR_pkey_alloc 329
-#define __NR_pkey_free 330
-
-#endif /* _UAPI__ASM_AVR32_UNISTD_H */
diff --git a/arch/avr32/kernel/.gitignore b/arch/avr32/kernel/.gitignore
deleted file mode 100644
index c5f676c..0000000
--- a/arch/avr32/kernel/.gitignore
+++ /dev/null
@@ -1 +0,0 @@
-vmlinux.lds
diff --git a/arch/avr32/kernel/Makefile b/arch/avr32/kernel/Makefile
deleted file mode 100644
index 119a2e4..0000000
--- a/arch/avr32/kernel/Makefile
+++ /dev/null
@@ -1,15 +0,0 @@
-#
-# Makefile for the Linux/AVR32 kernel.
-#
-
-extra-y := head.o vmlinux.lds
-
-obj-$(CONFIG_SUBARCH_AVR32B) += entry-avr32b.o
-obj-y += syscall_table.o syscall-stubs.o irq.o
-obj-y += setup.o traps.o ocd.o ptrace.o
-obj-y += signal.o process.o time.o
-obj-y += switch_to.o cpu.o
-obj-$(CONFIG_MODULES) += module.o avr32_ksyms.o
-obj-$(CONFIG_KPROBES) += kprobes.o
-obj-$(CONFIG_STACKTRACE) += stacktrace.o
-obj-$(CONFIG_NMI_DEBUGGING) += nmi_debug.o
diff --git a/arch/avr32/kernel/asm-offsets.c b/arch/avr32/kernel/asm-offsets.c
deleted file mode 100644
index 2c9764f..0000000
--- a/arch/avr32/kernel/asm-offsets.c
+++ /dev/null
@@ -1,24 +0,0 @@
-/*
- * Generate definitions needed by assembly language modules.
- * This code generates raw asm output which is post-processed
- * to extract and format the required data.
- */
-
-#include <linux/mm.h>
-#include <linux/sched.h>
-#include <linux/thread_info.h>
-#include <linux/kbuild.h>
-
-void foo(void)
-{
- OFFSET(TI_task, thread_info, task);
- OFFSET(TI_flags, thread_info, flags);
- OFFSET(TI_cpu, thread_info, cpu);
- OFFSET(TI_preempt_count, thread_info, preempt_count);
- OFFSET(TI_rar_saved, thread_info, rar_saved);
- OFFSET(TI_rsr_saved, thread_info, rsr_saved);
- BLANK();
- OFFSET(TSK_active_mm, task_struct, active_mm);
- BLANK();
- OFFSET(MM_pgd, mm_struct, pgd);
-}
diff --git a/arch/avr32/kernel/avr32_ksyms.c b/arch/avr32/kernel/avr32_ksyms.c
deleted file mode 100644
index 0d05fd0..0000000
--- a/arch/avr32/kernel/avr32_ksyms.c
+++ /dev/null
@@ -1,70 +0,0 @@
-/*
- * Export AVR32-specific functions for loadable modules.
- *
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/delay.h>
-#include <linux/io.h>
-#include <linux/module.h>
-
-#include <asm/checksum.h>
-#include <linux/uaccess.h>
-
-/*
- * GCC functions
- */
-extern unsigned long long __avr32_lsl64(unsigned long long u, unsigned long b);
-extern unsigned long long __avr32_lsr64(unsigned long long u, unsigned long b);
-extern unsigned long long __avr32_asr64(unsigned long long u, unsigned long b);
-EXPORT_SYMBOL(__avr32_lsl64);
-EXPORT_SYMBOL(__avr32_lsr64);
-EXPORT_SYMBOL(__avr32_asr64);
-
-/*
- * String functions
- */
-EXPORT_SYMBOL(memset);
-EXPORT_SYMBOL(memcpy);
-
-EXPORT_SYMBOL(clear_page);
-EXPORT_SYMBOL(copy_page);
-
-/*
- * Userspace access stuff.
- */
-EXPORT_SYMBOL(___copy_from_user);
-EXPORT_SYMBOL(copy_to_user);
-EXPORT_SYMBOL(__copy_user);
-EXPORT_SYMBOL(strncpy_from_user);
-EXPORT_SYMBOL(__strncpy_from_user);
-EXPORT_SYMBOL(clear_user);
-EXPORT_SYMBOL(__clear_user);
-EXPORT_SYMBOL(strnlen_user);
-
-EXPORT_SYMBOL(csum_partial);
-EXPORT_SYMBOL(csum_partial_copy_generic);
-
-/* Delay loops (lib/delay.S) */
-EXPORT_SYMBOL(__ndelay);
-EXPORT_SYMBOL(__udelay);
-EXPORT_SYMBOL(__const_udelay);
-
-/* Bit operations (lib/findbit.S) */
-EXPORT_SYMBOL(find_first_zero_bit);
-EXPORT_SYMBOL(find_next_zero_bit);
-EXPORT_SYMBOL(find_first_bit);
-EXPORT_SYMBOL(find_next_bit);
-EXPORT_SYMBOL(find_next_bit_le);
-EXPORT_SYMBOL(find_next_zero_bit_le);
-
-/* I/O primitives (lib/io-*.S) */
-EXPORT_SYMBOL(__raw_readsb);
-EXPORT_SYMBOL(__raw_readsw);
-EXPORT_SYMBOL(__raw_readsl);
-EXPORT_SYMBOL(__raw_writesb);
-EXPORT_SYMBOL(__raw_writesw);
-EXPORT_SYMBOL(__raw_writesl);
diff --git a/arch/avr32/kernel/cpu.c b/arch/avr32/kernel/cpu.c
deleted file mode 100644
index 0341ae2..0000000
--- a/arch/avr32/kernel/cpu.c
+++ /dev/null
@@ -1,410 +0,0 @@
-/*
- * Copyright (C) 2005-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/init.h>
-#include <linux/device.h>
-#include <linux/seq_file.h>
-#include <linux/cpu.h>
-#include <linux/module.h>
-#include <linux/percpu.h>
-#include <linux/param.h>
-#include <linux/errno.h>
-#include <linux/clk.h>
-
-#include <asm/setup.h>
-#include <asm/sysreg.h>
-
-static DEFINE_PER_CPU(struct cpu, cpu_devices);
-
-#ifdef CONFIG_PERFORMANCE_COUNTERS
-
-/*
- * XXX: If/when a SMP-capable implementation of AVR32 will ever be
- * made, we must make sure that the code executes on the correct CPU.
- */
-static ssize_t show_pc0event(struct device *dev,
- struct device_attribute *attr, char *buf)
-{
- unsigned long pccr;
-
- pccr = sysreg_read(PCCR);
- return sprintf(buf, "0x%lx\n", (pccr >> 12) & 0x3f);
-}
-static ssize_t store_pc0event(struct device *dev,
- struct device_attribute *attr, const char *buf,
- size_t count)
-{
- unsigned long val;
- int ret;
-
- ret = kstrtoul(buf, 0, &val);
- if (ret)
- return ret;
- if (val > 0x3f)
- return -EINVAL;
- val = (val << 12) | (sysreg_read(PCCR) & 0xfffc0fff);
- sysreg_write(PCCR, val);
- return count;
-}
-static ssize_t show_pc0count(struct device *dev,
- struct device_attribute *attr, char *buf)
-{
- unsigned long pcnt0;
-
- pcnt0 = sysreg_read(PCNT0);
- return sprintf(buf, "%lu\n", pcnt0);
-}
-static ssize_t store_pc0count(struct device *dev,
- struct device_attribute *attr,
- const char *buf, size_t count)
-{
- unsigned long val;
- int ret;
-
- ret = kstrtoul(buf, 0, &val);
- if (ret)
- return ret;
- sysreg_write(PCNT0, val);
-
- return count;
-}
-
-static ssize_t show_pc1event(struct device *dev,
- struct device_attribute *attr, char *buf)
-{
- unsigned long pccr;
-
- pccr = sysreg_read(PCCR);
- return sprintf(buf, "0x%lx\n", (pccr >> 18) & 0x3f);
-}
-static ssize_t store_pc1event(struct device *dev,
- struct device_attribute *attr, const char *buf,
- size_t count)
-{
- unsigned long val;
- int ret;
-
- ret = kstrtoul(buf, 0, &val);
- if (ret)
- return ret;
- if (val > 0x3f)
- return -EINVAL;
- val = (val << 18) | (sysreg_read(PCCR) & 0xff03ffff);
- sysreg_write(PCCR, val);
- return count;
-}
-static ssize_t show_pc1count(struct device *dev,
- struct device_attribute *attr, char *buf)
-{
- unsigned long pcnt1;
-
- pcnt1 = sysreg_read(PCNT1);
- return sprintf(buf, "%lu\n", pcnt1);
-}
-static ssize_t store_pc1count(struct device *dev,
- struct device_attribute *attr, const char *buf,
- size_t count)
-{
- unsigned long val;
- int ret;
-
- ret = kstrtoul(buf, 0, &val);
- if (ret)
- return ret;
- sysreg_write(PCNT1, val);
-
- return count;
-}
-
-static ssize_t show_pccycles(struct device *dev,
- struct device_attribute *attr, char *buf)
-{
- unsigned long pccnt;
-
- pccnt = sysreg_read(PCCNT);
- return sprintf(buf, "%lu\n", pccnt);
-}
-static ssize_t store_pccycles(struct device *dev,
- struct device_attribute *attr, const char *buf,
- size_t count)
-{
- unsigned long val;
- int ret;
-
- ret = kstrtoul(buf, 0, &val);
- if (ret)
- return ret;
- sysreg_write(PCCNT, val);
-
- return count;
-}
-
-static ssize_t show_pcenable(struct device *dev,
- struct device_attribute *attr, char *buf)
-{
- unsigned long pccr;
-
- pccr = sysreg_read(PCCR);
- return sprintf(buf, "%c\n", (pccr & 1)?'1':'0');
-}
-static ssize_t store_pcenable(struct device *dev,
- struct device_attribute *attr, const char *buf,
- size_t count)
-{
- unsigned long pccr, val;
- int ret;
-
- ret = kstrtoul(buf, 0, &val);
- if (ret)
- return ret;
- if (val)
- val = 1;
-
- pccr = sysreg_read(PCCR);
- pccr = (pccr & ~1UL) | val;
- sysreg_write(PCCR, pccr);
-
- return count;
-}
-
-static DEVICE_ATTR(pc0event, 0600, show_pc0event, store_pc0event);
-static DEVICE_ATTR(pc0count, 0600, show_pc0count, store_pc0count);
-static DEVICE_ATTR(pc1event, 0600, show_pc1event, store_pc1event);
-static DEVICE_ATTR(pc1count, 0600, show_pc1count, store_pc1count);
-static DEVICE_ATTR(pccycles, 0600, show_pccycles, store_pccycles);
-static DEVICE_ATTR(pcenable, 0600, show_pcenable, store_pcenable);
-
-#endif /* CONFIG_PERFORMANCE_COUNTERS */
-
-static int __init topology_init(void)
-{
- int cpu;
-
- for_each_possible_cpu(cpu) {
- struct cpu *c = &per_cpu(cpu_devices, cpu);
-
- register_cpu(c, cpu);
-
-#ifdef CONFIG_PERFORMANCE_COUNTERS
- device_create_file(&c->dev, &dev_attr_pc0event);
- device_create_file(&c->dev, &dev_attr_pc0count);
- device_create_file(&c->dev, &dev_attr_pc1event);
- device_create_file(&c->dev, &dev_attr_pc1count);
- device_create_file(&c->dev, &dev_attr_pccycles);
- device_create_file(&c->dev, &dev_attr_pcenable);
-#endif
- }
-
- return 0;
-}
-
-subsys_initcall(topology_init);
-
-struct chip_id_map {
- u16 mid;
- u16 pn;
- const char *name;
-};
-
-static const struct chip_id_map chip_names[] = {
- { .mid = 0x1f, .pn = 0x1e82, .name = "AT32AP700x" },
-};
-#define NR_CHIP_NAMES ARRAY_SIZE(chip_names)
-
-static const char *cpu_names[] = {
- "Morgan",
- "AP7",
-};
-#define NR_CPU_NAMES ARRAY_SIZE(cpu_names)
-
-static const char *arch_names[] = {
- "AVR32A",
- "AVR32B",
-};
-#define NR_ARCH_NAMES ARRAY_SIZE(arch_names)
-
-static const char *mmu_types[] = {
- "No MMU",
- "ITLB and DTLB",
- "Shared TLB",
- "MPU"
-};
-
-static const char *cpu_feature_flags[] = {
- "rmw", "dsp", "simd", "ocd", "perfctr", "java", "fpu",
-};
-
-static const char *get_chip_name(struct avr32_cpuinfo *cpu)
-{
- unsigned int i;
- unsigned int mid = avr32_get_manufacturer_id(cpu);
- unsigned int pn = avr32_get_product_number(cpu);
-
- for (i = 0; i < NR_CHIP_NAMES; i++) {
- if (chip_names[i].mid == mid && chip_names[i].pn == pn)
- return chip_names[i].name;
- }
-
- return "(unknown)";
-}
-
-void __init setup_processor(void)
-{
- unsigned long config0, config1;
- unsigned long features;
- unsigned cpu_id, cpu_rev, arch_id, arch_rev, mmu_type;
- unsigned device_id;
- unsigned tmp;
- unsigned i;
-
- config0 = sysreg_read(CONFIG0);
- config1 = sysreg_read(CONFIG1);
- cpu_id = SYSREG_BFEXT(PROCESSORID, config0);
- cpu_rev = SYSREG_BFEXT(PROCESSORREVISION, config0);
- arch_id = SYSREG_BFEXT(AT, config0);
- arch_rev = SYSREG_BFEXT(AR, config0);
- mmu_type = SYSREG_BFEXT(MMUT, config0);
-
- device_id = ocd_read(DID);
-
- boot_cpu_data.arch_type = arch_id;
- boot_cpu_data.cpu_type = cpu_id;
- boot_cpu_data.arch_revision = arch_rev;
- boot_cpu_data.cpu_revision = cpu_rev;
- boot_cpu_data.tlb_config = mmu_type;
- boot_cpu_data.device_id = device_id;
-
- tmp = SYSREG_BFEXT(ILSZ, config1);
- if (tmp) {
- boot_cpu_data.icache.ways = 1 << SYSREG_BFEXT(IASS, config1);
- boot_cpu_data.icache.sets = 1 << SYSREG_BFEXT(ISET, config1);
- boot_cpu_data.icache.linesz = 1 << (tmp + 1);
- }
- tmp = SYSREG_BFEXT(DLSZ, config1);
- if (tmp) {
- boot_cpu_data.dcache.ways = 1 << SYSREG_BFEXT(DASS, config1);
- boot_cpu_data.dcache.sets = 1 << SYSREG_BFEXT(DSET, config1);
- boot_cpu_data.dcache.linesz = 1 << (tmp + 1);
- }
-
- if ((cpu_id >= NR_CPU_NAMES) || (arch_id >= NR_ARCH_NAMES)) {
- printk ("Unknown CPU configuration (ID %02x, arch %02x), "
- "continuing anyway...\n",
- cpu_id, arch_id);
- return;
- }
-
- printk ("CPU: %s chip revision %c\n", get_chip_name(&boot_cpu_data),
- avr32_get_chip_revision(&boot_cpu_data) + 'A');
- printk ("CPU: %s [%02x] core revision %d (%s arch revision %d)\n",
- cpu_names[cpu_id], cpu_id, cpu_rev,
- arch_names[arch_id], arch_rev);
- printk ("CPU: MMU configuration: %s\n", mmu_types[mmu_type]);
-
- printk ("CPU: features:");
- features = 0;
- if (config0 & SYSREG_BIT(CONFIG0_R))
- features |= AVR32_FEATURE_RMW;
- if (config0 & SYSREG_BIT(CONFIG0_D))
- features |= AVR32_FEATURE_DSP;
- if (config0 & SYSREG_BIT(CONFIG0_S))
- features |= AVR32_FEATURE_SIMD;
- if (config0 & SYSREG_BIT(CONFIG0_O))
- features |= AVR32_FEATURE_OCD;
- if (config0 & SYSREG_BIT(CONFIG0_P))
- features |= AVR32_FEATURE_PCTR;
- if (config0 & SYSREG_BIT(CONFIG0_J))
- features |= AVR32_FEATURE_JAVA;
- if (config0 & SYSREG_BIT(CONFIG0_F))
- features |= AVR32_FEATURE_FPU;
-
- for (i = 0; i < ARRAY_SIZE(cpu_feature_flags); i++)
- if (features & (1 << i))
- printk(" %s", cpu_feature_flags[i]);
-
- printk("\n");
- boot_cpu_data.features = features;
-}
-
-#ifdef CONFIG_PROC_FS
-static int c_show(struct seq_file *m, void *v)
-{
- unsigned int icache_size, dcache_size;
- unsigned int cpu = smp_processor_id();
- unsigned int freq;
- unsigned int i;
-
- icache_size = boot_cpu_data.icache.ways *
- boot_cpu_data.icache.sets *
- boot_cpu_data.icache.linesz;
- dcache_size = boot_cpu_data.dcache.ways *
- boot_cpu_data.dcache.sets *
- boot_cpu_data.dcache.linesz;
-
- seq_printf(m, "processor\t: %d\n", cpu);
-
- seq_printf(m, "chip type\t: %s revision %c\n",
- get_chip_name(&boot_cpu_data),
- avr32_get_chip_revision(&boot_cpu_data) + 'A');
- if (boot_cpu_data.arch_type < NR_ARCH_NAMES)
- seq_printf(m, "cpu arch\t: %s revision %d\n",
- arch_names[boot_cpu_data.arch_type],
- boot_cpu_data.arch_revision);
- if (boot_cpu_data.cpu_type < NR_CPU_NAMES)
- seq_printf(m, "cpu core\t: %s revision %d\n",
- cpu_names[boot_cpu_data.cpu_type],
- boot_cpu_data.cpu_revision);
-
- freq = (clk_get_rate(boot_cpu_data.clk) + 500) / 1000;
- seq_printf(m, "cpu MHz\t\t: %u.%03u\n", freq / 1000, freq % 1000);
-
- seq_printf(m, "i-cache\t\t: %dK (%u ways x %u sets x %u)\n",
- icache_size >> 10,
- boot_cpu_data.icache.ways,
- boot_cpu_data.icache.sets,
- boot_cpu_data.icache.linesz);
- seq_printf(m, "d-cache\t\t: %dK (%u ways x %u sets x %u)\n",
- dcache_size >> 10,
- boot_cpu_data.dcache.ways,
- boot_cpu_data.dcache.sets,
- boot_cpu_data.dcache.linesz);
-
- seq_printf(m, "features\t:");
- for (i = 0; i < ARRAY_SIZE(cpu_feature_flags); i++)
- if (boot_cpu_data.features & (1 << i))
- seq_printf(m, " %s", cpu_feature_flags[i]);
-
- seq_printf(m, "\nbogomips\t: %lu.%02lu\n",
- boot_cpu_data.loops_per_jiffy / (500000/HZ),
- (boot_cpu_data.loops_per_jiffy / (5000/HZ)) % 100);
-
- return 0;
-}
-
-static void *c_start(struct seq_file *m, loff_t *pos)
-{
- return *pos < 1 ? (void *)1 : NULL;
-}
-
-static void *c_next(struct seq_file *m, void *v, loff_t *pos)
-{
- ++*pos;
- return NULL;
-}
-
-static void c_stop(struct seq_file *m, void *v)
-{
-
-}
-
-const struct seq_operations cpuinfo_op = {
- .start = c_start,
- .next = c_next,
- .stop = c_stop,
- .show = c_show
-};
-#endif /* CONFIG_PROC_FS */
diff --git a/arch/avr32/kernel/entry-avr32b.S b/arch/avr32/kernel/entry-avr32b.S
deleted file mode 100644
index 7301f48..0000000
--- a/arch/avr32/kernel/entry-avr32b.S
+++ /dev/null
@@ -1,877 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-/*
- * This file contains the low-level entry-points into the kernel, that is,
- * exception handlers, debug trap handlers, interrupt handlers and the
- * system call handler.
- */
-#include <linux/errno.h>
-
-#include <asm/asm.h>
-#include <asm/hardirq.h>
-#include <asm/irq.h>
-#include <asm/ocd.h>
-#include <asm/page.h>
-#include <asm/pgtable.h>
-#include <asm/ptrace.h>
-#include <asm/sysreg.h>
-#include <asm/thread_info.h>
-#include <asm/unistd.h>
-
-#ifdef CONFIG_PREEMPT
-# define preempt_stop mask_interrupts
-#else
-# define preempt_stop
-# define fault_resume_kernel fault_restore_all
-#endif
-
-#define __MASK(x) ((1 << (x)) - 1)
-#define IRQ_MASK ((__MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT) | \
- (__MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT))
-
- .section .ex.text,"ax",@progbits
- .align 2
-exception_vectors:
- bral handle_critical
- .align 2
- bral handle_critical
- .align 2
- bral do_bus_error_write
- .align 2
- bral do_bus_error_read
- .align 2
- bral do_nmi_ll
- .align 2
- bral handle_address_fault
- .align 2
- bral handle_protection_fault
- .align 2
- bral handle_debug
- .align 2
- bral do_illegal_opcode_ll
- .align 2
- bral do_illegal_opcode_ll
- .align 2
- bral do_illegal_opcode_ll
- .align 2
- bral do_fpe_ll
- .align 2
- bral do_illegal_opcode_ll
- .align 2
- bral handle_address_fault
- .align 2
- bral handle_address_fault
- .align 2
- bral handle_protection_fault
- .align 2
- bral handle_protection_fault
- .align 2
- bral do_dtlb_modified
-
-#define tlbmiss_save pushm r0-r3
-#define tlbmiss_restore popm r0-r3
-
- .org 0x50
- .global itlb_miss
-itlb_miss:
- tlbmiss_save
- rjmp tlb_miss_common
-
- .org 0x60
-dtlb_miss_read:
- tlbmiss_save
- rjmp tlb_miss_common
-
- .org 0x70
-dtlb_miss_write:
- tlbmiss_save
-
- .global tlb_miss_common
- .align 2
-tlb_miss_common:
- mfsr r0, SYSREG_TLBEAR
- mfsr r1, SYSREG_PTBR
-
- /*
- * First level lookup: The PGD contains virtual pointers to
- * the second-level page tables, but they may be NULL if not
- * present.
- */
-pgtbl_lookup:
- lsr r2, r0, PGDIR_SHIFT
- ld.w r3, r1[r2 << 2]
- bfextu r1, r0, PAGE_SHIFT, PGDIR_SHIFT - PAGE_SHIFT
- cp.w r3, 0
- breq page_table_not_present
-
- /* Second level lookup */
- ld.w r2, r3[r1 << 2]
- mfsr r0, SYSREG_TLBARLO
- bld r2, _PAGE_BIT_PRESENT
- brcc page_not_present
-
- /* Mark the page as accessed */
- sbr r2, _PAGE_BIT_ACCESSED
- st.w r3[r1 << 2], r2
-
- /* Drop software flags */
- andl r2, _PAGE_FLAGS_HARDWARE_MASK & 0xffff
- mtsr SYSREG_TLBELO, r2
-
- /* Figure out which entry we want to replace */
- mfsr r1, SYSREG_MMUCR
- clz r2, r0
- brcc 1f
- mov r3, -1 /* All entries have been accessed, */
- mov r2, 0 /* so start at 0 */
- mtsr SYSREG_TLBARLO, r3 /* and reset TLBAR */
-
-1: bfins r1, r2, SYSREG_DRP_OFFSET, SYSREG_DRP_SIZE
- mtsr SYSREG_MMUCR, r1
- tlbw
-
- tlbmiss_restore
- rete
-
- /* The slow path of the TLB miss handler */
- .align 2
-page_table_not_present:
- /* Do we need to synchronize with swapper_pg_dir? */
- bld r0, 31
- brcs sync_with_swapper_pg_dir
-
-page_not_present:
- tlbmiss_restore
- sub sp, 4
- stmts --sp, r0-lr
- call save_full_context_ex
- mfsr r12, SYSREG_ECR
- mov r11, sp
- call do_page_fault
- rjmp ret_from_exception
-
- .align 2
-sync_with_swapper_pg_dir:
- /*
- * If swapper_pg_dir contains a non-NULL second-level page
- * table pointer, copy it into the current PGD. If not, we
- * must handle it as a full-blown page fault.
- *
- * Jumping back to pgtbl_lookup causes an unnecessary lookup,
- * but it is guaranteed to be a cache hit, it won't happen
- * very often, and we absolutely do not want to sacrifice any
- * performance in the fast path in order to improve this.
- */
- mov r1, lo(swapper_pg_dir)
- orh r1, hi(swapper_pg_dir)
- ld.w r3, r1[r2 << 2]
- cp.w r3, 0
- breq page_not_present
- mfsr r1, SYSREG_PTBR
- st.w r1[r2 << 2], r3
- rjmp pgtbl_lookup
-
- /*
- * We currently have two bytes left at this point until we
- * crash into the system call handler...
- *
- * Don't worry, the assembler will let us know.
- */
-
-
- /* --- System Call --- */
-
- .org 0x100
-system_call:
-#ifdef CONFIG_PREEMPT
- mask_interrupts
-#endif
- pushm r12 /* r12_orig */
- stmts --sp, r0-lr
-
- mfsr r0, SYSREG_RAR_SUP
- mfsr r1, SYSREG_RSR_SUP
-#ifdef CONFIG_PREEMPT
- unmask_interrupts
-#endif
- zero_fp
- stm --sp, r0-r1
-
- /* check for syscall tracing */
- get_thread_info r0
- ld.w r1, r0[TI_flags]
- bld r1, TIF_SYSCALL_TRACE
- brcs syscall_trace_enter
-
-syscall_trace_cont:
- cp.w r8, NR_syscalls
- brhs syscall_badsys
-
- lddpc lr, syscall_table_addr
- ld.w lr, lr[r8 << 2]
- mov r8, r5 /* 5th argument (6th is pushed by stub) */
- icall lr
-
- .global syscall_return
-syscall_return:
- get_thread_info r0
- mask_interrupts /* make sure we don't miss an interrupt
- setting need_resched or sigpending
- between sampling and the rets */
-
- /* Store the return value so that the correct value is loaded below */
- stdsp sp[REG_R12], r12
-
- ld.w r1, r0[TI_flags]
- andl r1, _TIF_ALLWORK_MASK, COH
- brne syscall_exit_work
-
-syscall_exit_cont:
- popm r8-r9
- mtsr SYSREG_RAR_SUP, r8
- mtsr SYSREG_RSR_SUP, r9
- ldmts sp++, r0-lr
- sub sp, -4 /* r12_orig */
- rets
-
- .align 2
-syscall_table_addr:
- .long sys_call_table
-
-syscall_badsys:
- mov r12, -ENOSYS
- rjmp syscall_return
-
- .global ret_from_fork
-ret_from_fork:
- call schedule_tail
- mov r12, 0
- rjmp syscall_return
-
- .global ret_from_kernel_thread
-ret_from_kernel_thread:
- call schedule_tail
- mov r12, r0
- mov lr, r2 /* syscall_return */
- mov pc, r1
-
-syscall_trace_enter:
- pushm r8-r12
- call syscall_trace
- popm r8-r12
- rjmp syscall_trace_cont
-
-syscall_exit_work:
- bld r1, TIF_SYSCALL_TRACE
- brcc 1f
- unmask_interrupts
- call syscall_trace
- mask_interrupts
- ld.w r1, r0[TI_flags]
-
-1: bld r1, TIF_NEED_RESCHED
- brcc 2f
- unmask_interrupts
- call schedule
- mask_interrupts
- ld.w r1, r0[TI_flags]
- rjmp 1b
-
-2: mov r2, _TIF_SIGPENDING | _TIF_NOTIFY_RESUME
- tst r1, r2
- breq 3f
- unmask_interrupts
- mov r12, sp
- mov r11, r0
- call do_notify_resume
- mask_interrupts
- ld.w r1, r0[TI_flags]
- rjmp 1b
-
-3: bld r1, TIF_BREAKPOINT
- brcc syscall_exit_cont
- rjmp enter_monitor_mode
-
- /* This function expects to find offending PC in SYSREG_RAR_EX */
- .type save_full_context_ex, @function
- .align 2
-save_full_context_ex:
- mfsr r11, SYSREG_RAR_EX
- sub r9, pc, . - debug_trampoline
- mfsr r8, SYSREG_RSR_EX
- cp.w r9, r11
- breq 3f
- mov r12, r8
- andh r8, (MODE_MASK >> 16), COH
- brne 2f
-
-1: pushm r11, r12 /* PC and SR */
- unmask_exceptions
- ret r12
-
-2: sub r10, sp, -(FRAME_SIZE_FULL - REG_LR)
- stdsp sp[4], r10 /* replace saved SP */
- rjmp 1b
-
- /*
- * The debug handler set up a trampoline to make us
- * automatically enter monitor mode upon return, but since
- * we're saving the full context, we must assume that the
- * exception handler might want to alter the return address
- * and/or status register. So we need to restore the original
- * context and enter monitor mode manually after the exception
- * has been handled.
- */
-3: get_thread_info r8
- ld.w r11, r8[TI_rar_saved]
- ld.w r12, r8[TI_rsr_saved]
- rjmp 1b
- .size save_full_context_ex, . - save_full_context_ex
-
- /* Low-level exception handlers */
-handle_critical:
- /*
- * AT32AP700x errata:
- *
- * After a Java stack overflow or underflow trap, any CPU
- * memory access may cause erratic behavior. This will happen
- * when the four least significant bits of the JOSP system
- * register contains any value between 9 and 15 (inclusive).
- *
- * Possible workarounds:
- * - Don't use the Java Extension Module
- * - Ensure that the stack overflow and underflow trap
- * handlers do not do any memory access or trigger any
- * exceptions before the overflow/underflow condition is
- * cleared (by incrementing or decrementing the JOSP)
- * - Make sure that JOSP does not contain any problematic
- * value before doing any exception or interrupt
- * processing.
- * - Set up a critical exception handler which writes a
- * known-to-be-safe value, e.g. 4, to JOSP before doing
- * any further processing.
- *
- * We'll use the last workaround for now since we cannot
- * guarantee that user space processes don't use Java mode.
- * Non-well-behaving userland will be terminated with extreme
- * prejudice.
- */
-#ifdef CONFIG_CPU_AT32AP700X
- /*
- * There's a chance we can't touch memory, so temporarily
- * borrow PTBR to save the stack pointer while we fix things
- * up...
- */
- mtsr SYSREG_PTBR, sp
- mov sp, 4
- mtsr SYSREG_JOSP, sp
- mfsr sp, SYSREG_PTBR
- sub pc, -2
-
- /* Push most of pt_regs on stack. We'll do the rest later */
- sub sp, 4
- pushm r0-r12
-
- /* PTBR mirrors current_thread_info()->task->active_mm->pgd */
- get_thread_info r0
- ld.w r1, r0[TI_task]
- ld.w r2, r1[TSK_active_mm]
- ld.w r3, r2[MM_pgd]
- mtsr SYSREG_PTBR, r3
-#else
- sub sp, 4
- pushm r0-r12
-#endif
- sub r0, sp, -(14 * 4)
- mov r1, lr
- mfsr r2, SYSREG_RAR_EX
- mfsr r3, SYSREG_RSR_EX
- pushm r0-r3
-
- mfsr r12, SYSREG_ECR
- mov r11, sp
- call do_critical_exception
-
- /* We should never get here... */
-bad_return:
- sub r12, pc, (. - 1f)
- lddpc pc, 2f
- .align 2
-1: .asciz "Return from critical exception!"
-2: .long panic
-
- .align 1
-do_bus_error_write:
- sub sp, 4
- stmts --sp, r0-lr
- call save_full_context_ex
- mov r11, 1
- rjmp 1f
-
-do_bus_error_read:
- sub sp, 4
- stmts --sp, r0-lr
- call save_full_context_ex
- mov r11, 0
-1: mfsr r12, SYSREG_BEAR
- mov r10, sp
- call do_bus_error
- rjmp ret_from_exception
-
- .align 1
-do_nmi_ll:
- sub sp, 4
- stmts --sp, r0-lr
- mfsr r9, SYSREG_RSR_NMI
- mfsr r8, SYSREG_RAR_NMI
- bfextu r0, r9, MODE_SHIFT, 3
- brne 2f
-
-1: pushm r8, r9 /* PC and SR */
- mfsr r12, SYSREG_ECR
- mov r11, sp
- call do_nmi
- popm r8-r9
- mtsr SYSREG_RAR_NMI, r8
- tst r0, r0
- mtsr SYSREG_RSR_NMI, r9
- brne 3f
-
- ldmts sp++, r0-lr
- sub sp, -4 /* skip r12_orig */
- rete
-
-2: sub r10, sp, -(FRAME_SIZE_FULL - REG_LR)
- stdsp sp[4], r10 /* replace saved SP */
- rjmp 1b
-
-3: popm lr
- sub sp, -4 /* skip sp */
- popm r0-r12
- sub sp, -4 /* skip r12_orig */
- rete
-
-handle_address_fault:
- sub sp, 4
- stmts --sp, r0-lr
- call save_full_context_ex
- mfsr r12, SYSREG_ECR
- mov r11, sp
- call do_address_exception
- rjmp ret_from_exception
-
-handle_protection_fault:
- sub sp, 4
- stmts --sp, r0-lr
- call save_full_context_ex
- mfsr r12, SYSREG_ECR
- mov r11, sp
- call do_page_fault
- rjmp ret_from_exception
-
- .align 1
-do_illegal_opcode_ll:
- sub sp, 4
- stmts --sp, r0-lr
- call save_full_context_ex
- mfsr r12, SYSREG_ECR
- mov r11, sp
- call do_illegal_opcode
- rjmp ret_from_exception
-
-do_dtlb_modified:
- pushm r0-r3
- mfsr r1, SYSREG_TLBEAR
- mfsr r0, SYSREG_PTBR
- lsr r2, r1, PGDIR_SHIFT
- ld.w r0, r0[r2 << 2]
- lsl r1, (32 - PGDIR_SHIFT)
- lsr r1, (32 - PGDIR_SHIFT) + PAGE_SHIFT
-
- /* Translate to virtual address in P1 */
- andl r0, 0xf000
- sbr r0, 31
- add r2, r0, r1 << 2
- ld.w r3, r2[0]
- sbr r3, _PAGE_BIT_DIRTY
- mov r0, r3
- st.w r2[0], r3
-
- /* The page table is up-to-date. Update the TLB entry as well */
- andl r0, lo(_PAGE_FLAGS_HARDWARE_MASK)
- mtsr SYSREG_TLBELO, r0
-
- /* MMUCR[DRP] is updated automatically, so let's go... */
- tlbw
-
- popm r0-r3
- rete
-
-do_fpe_ll:
- sub sp, 4
- stmts --sp, r0-lr
- call save_full_context_ex
- unmask_interrupts
- mov r12, 26
- mov r11, sp
- call do_fpe
- rjmp ret_from_exception
-
-ret_from_exception:
- mask_interrupts
- lddsp r4, sp[REG_SR]
-
- andh r4, (MODE_MASK >> 16), COH
- brne fault_resume_kernel
-
- get_thread_info r0
- ld.w r1, r0[TI_flags]
- andl r1, _TIF_WORK_MASK, COH
- brne fault_exit_work
-
-fault_resume_user:
- popm r8-r9
- mask_exceptions
- mtsr SYSREG_RAR_EX, r8
- mtsr SYSREG_RSR_EX, r9
- ldmts sp++, r0-lr
- sub sp, -4
- rete
-
-fault_resume_kernel:
-#ifdef CONFIG_PREEMPT
- get_thread_info r0
- ld.w r2, r0[TI_preempt_count]
- cp.w r2, 0
- brne 1f
- ld.w r1, r0[TI_flags]
- bld r1, TIF_NEED_RESCHED
- brcc 1f
- lddsp r4, sp[REG_SR]
- bld r4, SYSREG_GM_OFFSET
- brcs 1f
- call preempt_schedule_irq
-1:
-#endif
-
- popm r8-r9
- mask_exceptions
- mfsr r1, SYSREG_SR
- mtsr SYSREG_RAR_EX, r8
- mtsr SYSREG_RSR_EX, r9
- popm lr
- sub sp, -4 /* ignore SP */
- popm r0-r12
- sub sp, -4 /* ignore r12_orig */
- rete
-
-irq_exit_work:
- /* Switch to exception mode so that we can share the same code. */
- mfsr r8, SYSREG_SR
- cbr r8, SYSREG_M0_OFFSET
- orh r8, hi(SYSREG_BIT(M1) | SYSREG_BIT(M2))
- mtsr SYSREG_SR, r8
- sub pc, -2
- get_thread_info r0
- ld.w r1, r0[TI_flags]
-
-fault_exit_work:
- bld r1, TIF_NEED_RESCHED
- brcc 1f
- unmask_interrupts
- call schedule
- mask_interrupts
- ld.w r1, r0[TI_flags]
- rjmp fault_exit_work
-
-1: mov r2, _TIF_SIGPENDING | _TIF_NOTIFY_RESUME
- tst r1, r2
- breq 2f
- unmask_interrupts
- mov r12, sp
- mov r11, r0
- call do_notify_resume
- mask_interrupts
- ld.w r1, r0[TI_flags]
- rjmp fault_exit_work
-
-2: bld r1, TIF_BREAKPOINT
- brcc fault_resume_user
- rjmp enter_monitor_mode
-
- .section .kprobes.text, "ax", @progbits
- .type handle_debug, @function
-handle_debug:
- sub sp, 4 /* r12_orig */
- stmts --sp, r0-lr
- mfsr r8, SYSREG_RAR_DBG
- mfsr r9, SYSREG_RSR_DBG
- unmask_exceptions
- pushm r8-r9
- bfextu r9, r9, SYSREG_MODE_OFFSET, SYSREG_MODE_SIZE
- brne debug_fixup_regs
-
-.Ldebug_fixup_cont:
-#ifdef CONFIG_TRACE_IRQFLAGS
- call trace_hardirqs_off
-#endif
- mov r12, sp
- call do_debug
- mov sp, r12
-
- lddsp r2, sp[REG_SR]
- bfextu r3, r2, SYSREG_MODE_OFFSET, SYSREG_MODE_SIZE
- brne debug_resume_kernel
-
- get_thread_info r0
- ld.w r1, r0[TI_flags]
- mov r2, _TIF_DBGWORK_MASK
- tst r1, r2
- brne debug_exit_work
-
- bld r1, TIF_SINGLE_STEP
- brcc 1f
- mfdr r4, OCD_DC
- sbr r4, OCD_DC_SS_BIT
- mtdr OCD_DC, r4
-
-1: popm r10,r11
- mask_exceptions
- mtsr SYSREG_RSR_DBG, r11
- mtsr SYSREG_RAR_DBG, r10
-#ifdef CONFIG_TRACE_IRQFLAGS
- call trace_hardirqs_on
-1:
-#endif
- ldmts sp++, r0-lr
- sub sp, -4
- retd
- .size handle_debug, . - handle_debug
-
- /* Mode of the trapped context is in r9 */
- .type debug_fixup_regs, @function
-debug_fixup_regs:
- mfsr r8, SYSREG_SR
- mov r10, r8
- bfins r8, r9, SYSREG_MODE_OFFSET, SYSREG_MODE_SIZE
- mtsr SYSREG_SR, r8
- sub pc, -2
- stdsp sp[REG_LR], lr
- mtsr SYSREG_SR, r10
- sub pc, -2
- sub r8, sp, -FRAME_SIZE_FULL
- stdsp sp[REG_SP], r8
- rjmp .Ldebug_fixup_cont
- .size debug_fixup_regs, . - debug_fixup_regs
-
- .type debug_resume_kernel, @function
-debug_resume_kernel:
- mask_exceptions
- popm r10, r11
- mtsr SYSREG_RAR_DBG, r10
- mtsr SYSREG_RSR_DBG, r11
-#ifdef CONFIG_TRACE_IRQFLAGS
- bld r11, SYSREG_GM_OFFSET
- brcc 1f
- call trace_hardirqs_on
-1:
-#endif
- mfsr r2, SYSREG_SR
- mov r1, r2
- bfins r2, r3, SYSREG_MODE_OFFSET, SYSREG_MODE_SIZE
- mtsr SYSREG_SR, r2
- sub pc, -2
- popm lr
- mtsr SYSREG_SR, r1
- sub pc, -2
- sub sp, -4 /* skip SP */
- popm r0-r12
- sub sp, -4
- retd
- .size debug_resume_kernel, . - debug_resume_kernel
-
- .type debug_exit_work, @function
-debug_exit_work:
- /*
- * We must return from Monitor Mode using a retd, and we must
- * not schedule since that involves the D bit in SR getting
- * cleared by something other than the debug hardware. This
- * may cause undefined behaviour according to the Architecture
- * manual.
- *
- * So we fix up the return address and status and return to a
- * stub below in Exception mode. From there, we can follow the
- * normal exception return path.
- *
- * The real return address and status registers are stored on
- * the stack in the way the exception return path understands,
- * so no need to fix anything up there.
- */
- sub r8, pc, . - fault_exit_work
- mtsr SYSREG_RAR_DBG, r8
- mov r9, 0
- orh r9, hi(SR_EM | SR_GM | MODE_EXCEPTION)
- mtsr SYSREG_RSR_DBG, r9
- sub pc, -2
- retd
- .size debug_exit_work, . - debug_exit_work
-
- .set rsr_int0, SYSREG_RSR_INT0
- .set rsr_int1, SYSREG_RSR_INT1
- .set rsr_int2, SYSREG_RSR_INT2
- .set rsr_int3, SYSREG_RSR_INT3
- .set rar_int0, SYSREG_RAR_INT0
- .set rar_int1, SYSREG_RAR_INT1
- .set rar_int2, SYSREG_RAR_INT2
- .set rar_int3, SYSREG_RAR_INT3
-
- .macro IRQ_LEVEL level
- .type irq_level\level, @function
-irq_level\level:
- sub sp, 4 /* r12_orig */
- stmts --sp,r0-lr
- mfsr r8, rar_int\level
- mfsr r9, rsr_int\level
-
-#ifdef CONFIG_PREEMPT
- sub r11, pc, (. - system_call)
- cp.w r11, r8
- breq 4f
-#endif
-
- pushm r8-r9
-
- mov r11, sp
- mov r12, \level
-
- call do_IRQ
-
- lddsp r4, sp[REG_SR]
- bfextu r4, r4, SYSREG_M0_OFFSET, 3
- cp.w r4, MODE_SUPERVISOR >> SYSREG_M0_OFFSET
- breq 2f
- cp.w r4, MODE_USER >> SYSREG_M0_OFFSET
-#ifdef CONFIG_PREEMPT
- brne 3f
-#else
- brne 1f
-#endif
-
- get_thread_info r0
- ld.w r1, r0[TI_flags]
- andl r1, _TIF_WORK_MASK, COH
- brne irq_exit_work
-
-1:
-#ifdef CONFIG_TRACE_IRQFLAGS
- call trace_hardirqs_on
-#endif
- popm r8-r9
- mtsr rar_int\level, r8
- mtsr rsr_int\level, r9
- ldmts sp++,r0-lr
- sub sp, -4 /* ignore r12_orig */
- rete
-
-#ifdef CONFIG_PREEMPT
-4: mask_interrupts
- mfsr r8, rsr_int\level
- sbr r8, 16
- mtsr rsr_int\level, r8
- ldmts sp++, r0-lr
- sub sp, -4 /* ignore r12_orig */
- rete
-#endif
-
-2: get_thread_info r0
- ld.w r1, r0[TI_flags]
- bld r1, TIF_CPU_GOING_TO_SLEEP
-#ifdef CONFIG_PREEMPT
- brcc 3f
-#else
- brcc 1b
-#endif
- sub r1, pc, . - cpu_idle_skip_sleep
- stdsp sp[REG_PC], r1
-#ifdef CONFIG_PREEMPT
-3: get_thread_info r0
- ld.w r2, r0[TI_preempt_count]
- cp.w r2, 0
- brne 1b
- ld.w r1, r0[TI_flags]
- bld r1, TIF_NEED_RESCHED
- brcc 1b
- lddsp r4, sp[REG_SR]
- bld r4, SYSREG_GM_OFFSET
- brcs 1b
- call preempt_schedule_irq
-#endif
- rjmp 1b
- .endm
-
- .section .irq.text,"ax",@progbits
-
- .global irq_level0
- .global irq_level1
- .global irq_level2
- .global irq_level3
- IRQ_LEVEL 0
- IRQ_LEVEL 1
- IRQ_LEVEL 2
- IRQ_LEVEL 3
-
- .section .kprobes.text, "ax", @progbits
- .type enter_monitor_mode, @function
-enter_monitor_mode:
- /*
- * We need to enter monitor mode to do a single step. The
- * monitor code will alter the return address so that we
- * return directly to the user instead of returning here.
- */
- breakpoint
- rjmp breakpoint_failed
-
- .size enter_monitor_mode, . - enter_monitor_mode
-
- .type debug_trampoline, @function
- .global debug_trampoline
-debug_trampoline:
- /*
- * Save the registers on the stack so that the monitor code
- * can find them easily.
- */
- sub sp, 4 /* r12_orig */
- stmts --sp, r0-lr
- get_thread_info r0
- ld.w r8, r0[TI_rar_saved]
- ld.w r9, r0[TI_rsr_saved]
- pushm r8-r9
-
- /*
- * The monitor code will alter the return address so we don't
- * return here.
- */
- breakpoint
- rjmp breakpoint_failed
- .size debug_trampoline, . - debug_trampoline
-
- .type breakpoint_failed, @function
-breakpoint_failed:
- /*
- * Something went wrong. Perhaps the debug hardware isn't
- * enabled?
- */
- lda.w r12, msg_breakpoint_failed
- mov r11, sp
- mov r10, 9 /* SIGKILL */
- call die
-1: rjmp 1b
-
-msg_breakpoint_failed:
- .asciz "Failed to enter Debug Mode"
diff --git a/arch/avr32/kernel/head.S b/arch/avr32/kernel/head.S
deleted file mode 100644
index 59eae6d..0000000
--- a/arch/avr32/kernel/head.S
+++ /dev/null
@@ -1,22 +0,0 @@
-/*
- * Non-board-specific low-level startup code
- *
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/linkage.h>
-
-#include <asm/page.h>
-
- .section .init.text,"ax"
- .global kernel_entry
-kernel_entry:
- /* Start the show */
- lddpc pc, kernel_start_addr
-
- .align 2
-kernel_start_addr:
- .long start_kernel
diff --git a/arch/avr32/kernel/irq.c b/arch/avr32/kernel/irq.c
deleted file mode 100644
index 900e49b..0000000
--- a/arch/avr32/kernel/irq.c
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * Based on arch/i386/kernel/irq.c
- * Copyright (C) 1992, 1998 Linus Torvalds, Ingo Molnar
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/interrupt.h>
-#include <linux/irq.h>
-#include <linux/kernel_stat.h>
-#include <linux/proc_fs.h>
-#include <linux/seq_file.h>
-#include <linux/device.h>
-
-/* May be overridden by platform code */
-int __weak nmi_enable(void)
-{
- return -ENOSYS;
-}
-
-void __weak nmi_disable(void)
-{
-
-}
diff --git a/arch/avr32/kernel/kprobes.c b/arch/avr32/kernel/kprobes.c
deleted file mode 100644
index a94ece4..0000000
--- a/arch/avr32/kernel/kprobes.c
+++ /dev/null
@@ -1,267 +0,0 @@
-/*
- * Kernel Probes (KProbes)
- *
- * Copyright (C) 2005-2006 Atmel Corporation
- *
- * Based on arch/ppc64/kernel/kprobes.c
- * Copyright (C) IBM Corporation, 2002, 2004
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/kprobes.h>
-#include <linux/ptrace.h>
-
-#include <asm/cacheflush.h>
-#include <linux/kdebug.h>
-#include <asm/ocd.h>
-
-DEFINE_PER_CPU(struct kprobe *, current_kprobe);
-static unsigned long kprobe_status;
-static struct pt_regs jprobe_saved_regs;
-
-struct kretprobe_blackpoint kretprobe_blacklist[] = {{NULL, NULL}};
-
-int __kprobes arch_prepare_kprobe(struct kprobe *p)
-{
- int ret = 0;
-
- if ((unsigned long)p->addr & 0x01) {
- printk("Attempt to register kprobe at an unaligned address\n");
- ret = -EINVAL;
- }
-
- /* XXX: Might be a good idea to check if p->addr is a valid
- * kernel address as well... */
-
- if (!ret) {
- pr_debug("copy kprobe at %p\n", p->addr);
- memcpy(p->ainsn.insn, p->addr, MAX_INSN_SIZE * sizeof(kprobe_opcode_t));
- p->opcode = *p->addr;
- }
-
- return ret;
-}
-
-void __kprobes arch_arm_kprobe(struct kprobe *p)
-{
- pr_debug("arming kprobe at %p\n", p->addr);
- ocd_enable(NULL);
- *p->addr = BREAKPOINT_INSTRUCTION;
- flush_icache_range((unsigned long)p->addr,
- (unsigned long)p->addr + sizeof(kprobe_opcode_t));
-}
-
-void __kprobes arch_disarm_kprobe(struct kprobe *p)
-{
- pr_debug("disarming kprobe at %p\n", p->addr);
- ocd_disable(NULL);
- *p->addr = p->opcode;
- flush_icache_range((unsigned long)p->addr,
- (unsigned long)p->addr + sizeof(kprobe_opcode_t));
-}
-
-static void __kprobes prepare_singlestep(struct kprobe *p, struct pt_regs *regs)
-{
- unsigned long dc;
-
- pr_debug("preparing to singlestep over %p (PC=%08lx)\n",
- p->addr, regs->pc);
-
- BUG_ON(!(sysreg_read(SR) & SYSREG_BIT(SR_D)));
-
- dc = ocd_read(DC);
- dc |= 1 << OCD_DC_SS_BIT;
- ocd_write(DC, dc);
-
- /*
- * We must run the instruction from its original location
- * since it may actually reference PC.
- *
- * TODO: Do the instruction replacement directly in icache.
- */
- *p->addr = p->opcode;
- flush_icache_range((unsigned long)p->addr,
- (unsigned long)p->addr + sizeof(kprobe_opcode_t));
-}
-
-static void __kprobes resume_execution(struct kprobe *p, struct pt_regs *regs)
-{
- unsigned long dc;
-
- pr_debug("resuming execution at PC=%08lx\n", regs->pc);
-
- dc = ocd_read(DC);
- dc &= ~(1 << OCD_DC_SS_BIT);
- ocd_write(DC, dc);
-
- *p->addr = BREAKPOINT_INSTRUCTION;
- flush_icache_range((unsigned long)p->addr,
- (unsigned long)p->addr + sizeof(kprobe_opcode_t));
-}
-
-static void __kprobes set_current_kprobe(struct kprobe *p)
-{
- __this_cpu_write(current_kprobe, p);
-}
-
-static int __kprobes kprobe_handler(struct pt_regs *regs)
-{
- struct kprobe *p;
- void *addr = (void *)regs->pc;
- int ret = 0;
-
- pr_debug("kprobe_handler: kprobe_running=%p\n",
- kprobe_running());
-
- /*
- * We don't want to be preempted for the entire
- * duration of kprobe processing
- */
- preempt_disable();
-
- /* Check that we're not recursing */
- if (kprobe_running()) {
- p = get_kprobe(addr);
- if (p) {
- if (kprobe_status == KPROBE_HIT_SS) {
- printk("FIXME: kprobe hit while single-stepping!\n");
- goto no_kprobe;
- }
-
- printk("FIXME: kprobe hit while handling another kprobe\n");
- goto no_kprobe;
- } else {
- p = kprobe_running();
- if (p->break_handler && p->break_handler(p, regs))
- goto ss_probe;
- }
- /* If it's not ours, can't be delete race, (we hold lock). */
- goto no_kprobe;
- }
-
- p = get_kprobe(addr);
- if (!p)
- goto no_kprobe;
-
- kprobe_status = KPROBE_HIT_ACTIVE;
- set_current_kprobe(p);
- if (p->pre_handler && p->pre_handler(p, regs))
- /* handler has already set things up, so skip ss setup */
- return 1;
-
-ss_probe:
- prepare_singlestep(p, regs);
- kprobe_status = KPROBE_HIT_SS;
- return 1;
-
-no_kprobe:
- preempt_enable_no_resched();
- return ret;
-}
-
-static int __kprobes post_kprobe_handler(struct pt_regs *regs)
-{
- struct kprobe *cur = kprobe_running();
-
- pr_debug("post_kprobe_handler, cur=%p\n", cur);
-
- if (!cur)
- return 0;
-
- if (cur->post_handler) {
- kprobe_status = KPROBE_HIT_SSDONE;
- cur->post_handler(cur, regs, 0);
- }
-
- resume_execution(cur, regs);
- reset_current_kprobe();
- preempt_enable_no_resched();
-
- return 1;
-}
-
-int __kprobes kprobe_fault_handler(struct pt_regs *regs, int trapnr)
-{
- struct kprobe *cur = kprobe_running();
-
- pr_debug("kprobe_fault_handler: trapnr=%d\n", trapnr);
-
- if (cur->fault_handler && cur->fault_handler(cur, regs, trapnr))
- return 1;
-
- if (kprobe_status & KPROBE_HIT_SS) {
- resume_execution(cur, regs);
- preempt_enable_no_resched();
- }
- return 0;
-}
-
-/*
- * Wrapper routine to for handling exceptions.
- */
-int __kprobes kprobe_exceptions_notify(struct notifier_block *self,
- unsigned long val, void *data)
-{
- struct die_args *args = (struct die_args *)data;
- int ret = NOTIFY_DONE;
-
- pr_debug("kprobe_exceptions_notify: val=%lu, data=%p\n",
- val, data);
-
- switch (val) {
- case DIE_BREAKPOINT:
- if (kprobe_handler(args->regs))
- ret = NOTIFY_STOP;
- break;
- case DIE_SSTEP:
- if (post_kprobe_handler(args->regs))
- ret = NOTIFY_STOP;
- break;
- default:
- break;
- }
-
- return ret;
-}
-
-int __kprobes setjmp_pre_handler(struct kprobe *p, struct pt_regs *regs)
-{
- struct jprobe *jp = container_of(p, struct jprobe, kp);
-
- memcpy(&jprobe_saved_regs, regs, sizeof(struct pt_regs));
-
- /*
- * TODO: We should probably save some of the stack here as
- * well, since gcc may pass arguments on the stack for certain
- * functions (lots of arguments, large aggregates, varargs)
- */
-
- /* setup return addr to the jprobe handler routine */
- regs->pc = (unsigned long)jp->entry;
- return 1;
-}
-
-void __kprobes jprobe_return(void)
-{
- asm volatile("breakpoint" ::: "memory");
-}
-
-int __kprobes longjmp_break_handler(struct kprobe *p, struct pt_regs *regs)
-{
- /*
- * FIXME - we should ideally be validating that we got here 'cos
- * of the "trap" in jprobe_return() above, before restoring the
- * saved regs...
- */
- memcpy(regs, &jprobe_saved_regs, sizeof(struct pt_regs));
- return 1;
-}
-
-int __init arch_init_kprobes(void)
-{
- /* TODO: Register kretprobe trampoline */
- return 0;
-}
diff --git a/arch/avr32/kernel/module.c b/arch/avr32/kernel/module.c
deleted file mode 100644
index 2b4c54c..0000000
--- a/arch/avr32/kernel/module.c
+++ /dev/null
@@ -1,291 +0,0 @@
-/*
- * AVR32-specific kernel module loader
- *
- * Copyright (C) 2005-2006 Atmel Corporation
- *
- * GOT initialization parts are based on the s390 version
- * Copyright (C) 2002, 2003 IBM Deutschland Entwicklung GmbH,
- * IBM Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/bug.h>
-#include <linux/elf.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/moduleloader.h>
-#include <linux/vmalloc.h>
-
-void module_arch_freeing_init(struct module *mod)
-{
- vfree(mod->arch.syminfo);
- mod->arch.syminfo = NULL;
-}
-
-static inline int check_rela(Elf32_Rela *rela, struct module *module,
- char *strings, Elf32_Sym *symbols)
-{
- struct mod_arch_syminfo *info;
-
- info = module->arch.syminfo + ELF32_R_SYM(rela->r_info);
- switch (ELF32_R_TYPE(rela->r_info)) {
- case R_AVR32_GOT32:
- case R_AVR32_GOT16:
- case R_AVR32_GOT8:
- case R_AVR32_GOT21S:
- case R_AVR32_GOT18SW: /* mcall */
- case R_AVR32_GOT16S: /* ld.w */
- if (rela->r_addend != 0) {
- printk(KERN_ERR
- "GOT relocation against %s at offset %u with addend\n",
- strings + symbols[ELF32_R_SYM(rela->r_info)].st_name,
- rela->r_offset);
- return -ENOEXEC;
- }
- if (info->got_offset == -1UL) {
- info->got_offset = module->arch.got_size;
- module->arch.got_size += sizeof(void *);
- }
- pr_debug("GOT[%3lu] %s\n", info->got_offset,
- strings + symbols[ELF32_R_SYM(rela->r_info)].st_name);
- break;
- }
-
- return 0;
-}
-
-int module_frob_arch_sections(Elf_Ehdr *hdr, Elf_Shdr *sechdrs,
- char *secstrings, struct module *module)
-{
- Elf32_Shdr *symtab;
- Elf32_Sym *symbols;
- Elf32_Rela *rela;
- char *strings;
- int nrela, i, j;
- int ret;
-
- /* Find the symbol table */
- symtab = NULL;
- for (i = 0; i < hdr->e_shnum; i++)
- switch (sechdrs[i].sh_type) {
- case SHT_SYMTAB:
- symtab = &sechdrs[i];
- break;
- }
- if (!symtab) {
- printk(KERN_ERR "module %s: no symbol table\n", module->name);
- return -ENOEXEC;
- }
-
- /* Allocate room for one syminfo structure per symbol. */
- module->arch.nsyms = symtab->sh_size / sizeof(Elf_Sym);
- module->arch.syminfo = vmalloc(module->arch.nsyms
- * sizeof(struct mod_arch_syminfo));
- if (!module->arch.syminfo)
- return -ENOMEM;
-
- symbols = (void *)hdr + symtab->sh_offset;
- strings = (void *)hdr + sechdrs[symtab->sh_link].sh_offset;
- for (i = 0; i < module->arch.nsyms; i++) {
- if (symbols[i].st_shndx == SHN_UNDEF &&
- strcmp(strings + symbols[i].st_name,
- "_GLOBAL_OFFSET_TABLE_") == 0)
- /* "Define" it as absolute. */
- symbols[i].st_shndx = SHN_ABS;
- module->arch.syminfo[i].got_offset = -1UL;
- module->arch.syminfo[i].got_initialized = 0;
- }
-
- /* Allocate GOT entries for symbols that need it. */
- module->arch.got_size = 0;
- for (i = 0; i < hdr->e_shnum; i++) {
- if (sechdrs[i].sh_type != SHT_RELA)
- continue;
- nrela = sechdrs[i].sh_size / sizeof(Elf32_Rela);
- rela = (void *)hdr + sechdrs[i].sh_offset;
- for (j = 0; j < nrela; j++) {
- ret = check_rela(rela + j, module,
- strings, symbols);
- if (ret)
- goto out_free_syminfo;
- }
- }
-
- /*
- * Increase core size to make room for GOT and set start
- * offset for GOT.
- */
- module->core_layout.size = ALIGN(module->core_layout.size, 4);
- module->arch.got_offset = module->core_layout.size;
- module->core_layout.size += module->arch.got_size;
-
- return 0;
-
-out_free_syminfo:
- vfree(module->arch.syminfo);
- module->arch.syminfo = NULL;
-
- return ret;
-}
-
-static inline int reloc_overflow(struct module *module, const char *reloc_name,
- Elf32_Addr relocation)
-{
- printk(KERN_ERR "module %s: Value %lx does not fit relocation %s\n",
- module->name, (unsigned long)relocation, reloc_name);
- return -ENOEXEC;
-}
-
-#define get_u16(loc) (*((uint16_t *)loc))
-#define put_u16(loc, val) (*((uint16_t *)loc) = (val))
-
-int apply_relocate_add(Elf32_Shdr *sechdrs, const char *strtab,
- unsigned int symindex, unsigned int relindex,
- struct module *module)
-{
- Elf32_Shdr *symsec = sechdrs + symindex;
- Elf32_Shdr *relsec = sechdrs + relindex;
- Elf32_Shdr *dstsec = sechdrs + relsec->sh_info;
- Elf32_Rela *rel = (void *)relsec->sh_addr;
- unsigned int i;
- int ret = 0;
-
- for (i = 0; i < relsec->sh_size / sizeof(Elf32_Rela); i++, rel++) {
- struct mod_arch_syminfo *info;
- Elf32_Sym *sym;
- Elf32_Addr relocation;
- uint32_t *location;
- uint32_t value;
-
- location = (void *)dstsec->sh_addr + rel->r_offset;
- sym = (Elf32_Sym *)symsec->sh_addr + ELF32_R_SYM(rel->r_info);
- relocation = sym->st_value + rel->r_addend;
-
- info = module->arch.syminfo + ELF32_R_SYM(rel->r_info);
-
- /* Initialize GOT entry if necessary */
- switch (ELF32_R_TYPE(rel->r_info)) {
- case R_AVR32_GOT32:
- case R_AVR32_GOT16:
- case R_AVR32_GOT8:
- case R_AVR32_GOT21S:
- case R_AVR32_GOT18SW:
- case R_AVR32_GOT16S:
- if (!info->got_initialized) {
- Elf32_Addr *gotent;
-
- gotent = (module->core_layout.base
- + module->arch.got_offset
- + info->got_offset);
- *gotent = relocation;
- info->got_initialized = 1;
- }
-
- relocation = info->got_offset;
- break;
- }
-
- switch (ELF32_R_TYPE(rel->r_info)) {
- case R_AVR32_32:
- case R_AVR32_32_CPENT:
- *location = relocation;
- break;
- case R_AVR32_22H_PCREL:
- relocation -= (Elf32_Addr)location;
- if ((relocation & 0xffe00001) != 0
- && (relocation & 0xffc00001) != 0xffc00000)
- return reloc_overflow(module,
- "R_AVR32_22H_PCREL",
- relocation);
- relocation >>= 1;
-
- value = *location;
- value = ((value & 0xe1ef0000)
- | (relocation & 0xffff)
- | ((relocation & 0x10000) << 4)
- | ((relocation & 0x1e0000) << 8));
- *location = value;
- break;
- case R_AVR32_11H_PCREL:
- relocation -= (Elf32_Addr)location;
- if ((relocation & 0xfffffc01) != 0
- && (relocation & 0xfffff801) != 0xfffff800)
- return reloc_overflow(module,
- "R_AVR32_11H_PCREL",
- relocation);
- value = get_u16(location);
- value = ((value & 0xf00c)
- | ((relocation & 0x1fe) << 3)
- | ((relocation & 0x600) >> 9));
- put_u16(location, value);
- break;
- case R_AVR32_9H_PCREL:
- relocation -= (Elf32_Addr)location;
- if ((relocation & 0xffffff01) != 0
- && (relocation & 0xfffffe01) != 0xfffffe00)
- return reloc_overflow(module,
- "R_AVR32_9H_PCREL",
- relocation);
- value = get_u16(location);
- value = ((value & 0xf00f)
- | ((relocation & 0x1fe) << 3));
- put_u16(location, value);
- break;
- case R_AVR32_9UW_PCREL:
- relocation -= ((Elf32_Addr)location) & 0xfffffffc;
- if ((relocation & 0xfffffc03) != 0)
- return reloc_overflow(module,
- "R_AVR32_9UW_PCREL",
- relocation);
- value = get_u16(location);
- value = ((value & 0xf80f)
- | ((relocation & 0x1fc) << 2));
- put_u16(location, value);
- break;
- case R_AVR32_GOTPC:
- /*
- * R6 = PC - (PC - GOT)
- *
- * At this point, relocation contains the
- * value of PC. Just subtract the value of
- * GOT, and we're done.
- */
- pr_debug("GOTPC: PC=0x%x, got_offset=0x%lx, core=0x%p\n",
- relocation, module->arch.got_offset,
- module->core_layout.base);
- relocation -= ((unsigned long)module->core_layout.base
- + module->arch.got_offset);
- *location = relocation;
- break;
- case R_AVR32_GOT18SW:
- if ((relocation & 0xfffe0003) != 0
- && (relocation & 0xfffc0000) != 0xfffc0000)
- return reloc_overflow(module, "R_AVR32_GOT18SW",
- relocation);
- relocation >>= 2;
- /* fall through */
- case R_AVR32_GOT16S:
- if ((relocation & 0xffff8000) != 0
- && (relocation & 0xffff0000) != 0xffff0000)
- return reloc_overflow(module, "R_AVR32_GOT16S",
- relocation);
- pr_debug("GOT reloc @ 0x%x -> %u\n",
- rel->r_offset, relocation);
- value = *location;
- value = ((value & 0xffff0000)
- | (relocation & 0xffff));
- *location = value;
- break;
-
- default:
- printk(KERN_ERR "module %s: Unknown relocation: %u\n",
- module->name, ELF32_R_TYPE(rel->r_info));
- return -ENOEXEC;
- }
- }
-
- return ret;
-}
diff --git a/arch/avr32/kernel/nmi_debug.c b/arch/avr32/kernel/nmi_debug.c
deleted file mode 100644
index 2582304..0000000
--- a/arch/avr32/kernel/nmi_debug.c
+++ /dev/null
@@ -1,83 +0,0 @@
-/*
- * Copyright (C) 2007 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/delay.h>
-#include <linux/kdebug.h>
-#include <linux/notifier.h>
-#include <linux/sched.h>
-#include <linux/sched/debug.h>
-
-#include <asm/irq.h>
-
-enum nmi_action {
- NMI_SHOW_STATE = 1 << 0,
- NMI_SHOW_REGS = 1 << 1,
- NMI_DIE = 1 << 2,
- NMI_DEBOUNCE = 1 << 3,
-};
-
-static unsigned long nmi_actions;
-
-static int nmi_debug_notify(struct notifier_block *self,
- unsigned long val, void *data)
-{
- struct die_args *args = data;
-
- if (likely(val != DIE_NMI))
- return NOTIFY_DONE;
-
- if (nmi_actions & NMI_SHOW_STATE)
- show_state();
- if (nmi_actions & NMI_SHOW_REGS)
- show_regs(args->regs);
- if (nmi_actions & NMI_DEBOUNCE)
- mdelay(10);
- if (nmi_actions & NMI_DIE)
- return NOTIFY_BAD;
-
- return NOTIFY_OK;
-}
-
-static struct notifier_block nmi_debug_nb = {
- .notifier_call = nmi_debug_notify,
-};
-
-static int __init nmi_debug_setup(char *str)
-{
- char *p, *sep;
-
- register_die_notifier(&nmi_debug_nb);
- if (nmi_enable()) {
- printk(KERN_WARNING "Unable to enable NMI.\n");
- return 0;
- }
-
- if (*str != '=')
- return 0;
-
- for (p = str + 1; *p; p = sep + 1) {
- sep = strchr(p, ',');
- if (sep)
- *sep = 0;
- if (strcmp(p, "state") == 0)
- nmi_actions |= NMI_SHOW_STATE;
- else if (strcmp(p, "regs") == 0)
- nmi_actions |= NMI_SHOW_REGS;
- else if (strcmp(p, "debounce") == 0)
- nmi_actions |= NMI_DEBOUNCE;
- else if (strcmp(p, "die") == 0)
- nmi_actions |= NMI_DIE;
- else
- printk(KERN_WARNING "NMI: Unrecognized action `%s'\n",
- p);
- if (!sep)
- break;
- }
-
- return 0;
-}
-__setup("nmi_debug", nmi_debug_setup);
diff --git a/arch/avr32/kernel/ocd.c b/arch/avr32/kernel/ocd.c
deleted file mode 100644
index 1b0245d..0000000
--- a/arch/avr32/kernel/ocd.c
+++ /dev/null
@@ -1,167 +0,0 @@
-/*
- * Copyright (C) 2007 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/init.h>
-#include <linux/sched.h>
-#include <linux/spinlock.h>
-
-#include <asm/ocd.h>
-
-static long ocd_count;
-static spinlock_t ocd_lock;
-
-/**
- * ocd_enable - enable on-chip debugging
- * @child: task to be debugged
- *
- * If @child is non-NULL, ocd_enable() first checks if debugging has
- * already been enabled for @child, and if it has, does nothing.
- *
- * If @child is NULL (e.g. when debugging the kernel), or debugging
- * has not already been enabled for it, ocd_enable() increments the
- * reference count and enables the debugging hardware.
- */
-void ocd_enable(struct task_struct *child)
-{
- u32 dc;
-
- if (child)
- pr_debug("ocd_enable: child=%s [%u]\n",
- child->comm, child->pid);
- else
- pr_debug("ocd_enable (no child)\n");
-
- if (!child || !test_and_set_tsk_thread_flag(child, TIF_DEBUG)) {
- spin_lock(&ocd_lock);
- ocd_count++;
- dc = ocd_read(DC);
- dc |= (1 << OCD_DC_MM_BIT) | (1 << OCD_DC_DBE_BIT);
- ocd_write(DC, dc);
- spin_unlock(&ocd_lock);
- }
-}
-
-/**
- * ocd_disable - disable on-chip debugging
- * @child: task that was being debugged, but isn't anymore
- *
- * If @child is non-NULL, ocd_disable() checks if debugging is enabled
- * for @child, and if it isn't, does nothing.
- *
- * If @child is NULL (e.g. when debugging the kernel), or debugging is
- * enabled, ocd_disable() decrements the reference count, and if it
- * reaches zero, disables the debugging hardware.
- */
-void ocd_disable(struct task_struct *child)
-{
- u32 dc;
-
- if (!child)
- pr_debug("ocd_disable (no child)\n");
- else if (test_tsk_thread_flag(child, TIF_DEBUG))
- pr_debug("ocd_disable: child=%s [%u]\n",
- child->comm, child->pid);
-
- if (!child || test_and_clear_tsk_thread_flag(child, TIF_DEBUG)) {
- spin_lock(&ocd_lock);
- ocd_count--;
-
- WARN_ON(ocd_count < 0);
-
- if (ocd_count <= 0) {
- dc = ocd_read(DC);
- dc &= ~((1 << OCD_DC_MM_BIT) | (1 << OCD_DC_DBE_BIT));
- ocd_write(DC, dc);
- }
- spin_unlock(&ocd_lock);
- }
-}
-
-#ifdef CONFIG_DEBUG_FS
-#include <linux/debugfs.h>
-#include <linux/module.h>
-
-static struct dentry *ocd_debugfs_root;
-static struct dentry *ocd_debugfs_DC;
-static struct dentry *ocd_debugfs_DS;
-static struct dentry *ocd_debugfs_count;
-
-static int ocd_DC_get(void *data, u64 *val)
-{
- *val = ocd_read(DC);
- return 0;
-}
-static int ocd_DC_set(void *data, u64 val)
-{
- ocd_write(DC, val);
- return 0;
-}
-DEFINE_SIMPLE_ATTRIBUTE(fops_DC, ocd_DC_get, ocd_DC_set, "0x%08llx\n");
-
-static int ocd_DS_get(void *data, u64 *val)
-{
- *val = ocd_read(DS);
- return 0;
-}
-DEFINE_SIMPLE_ATTRIBUTE(fops_DS, ocd_DS_get, NULL, "0x%08llx\n");
-
-static int ocd_count_get(void *data, u64 *val)
-{
- *val = ocd_count;
- return 0;
-}
-DEFINE_SIMPLE_ATTRIBUTE(fops_count, ocd_count_get, NULL, "%lld\n");
-
-static void ocd_debugfs_init(void)
-{
- struct dentry *root;
-
- root = debugfs_create_dir("ocd", NULL);
- if (IS_ERR(root) || !root)
- goto err_root;
- ocd_debugfs_root = root;
-
- ocd_debugfs_DC = debugfs_create_file("DC", S_IRUSR | S_IWUSR,
- root, NULL, &fops_DC);
- if (!ocd_debugfs_DC)
- goto err_DC;
-
- ocd_debugfs_DS = debugfs_create_file("DS", S_IRUSR, root,
- NULL, &fops_DS);
- if (!ocd_debugfs_DS)
- goto err_DS;
-
- ocd_debugfs_count = debugfs_create_file("count", S_IRUSR, root,
- NULL, &fops_count);
- if (!ocd_debugfs_count)
- goto err_count;
-
- return;
-
-err_count:
- debugfs_remove(ocd_debugfs_DS);
-err_DS:
- debugfs_remove(ocd_debugfs_DC);
-err_DC:
- debugfs_remove(ocd_debugfs_root);
-err_root:
- printk(KERN_WARNING "OCD: Failed to create debugfs entries\n");
-}
-#else
-static inline void ocd_debugfs_init(void)
-{
-
-}
-#endif
-
-static int __init ocd_init(void)
-{
- spin_lock_init(&ocd_lock);
- ocd_debugfs_init();
- return 0;
-}
-arch_initcall(ocd_init);
diff --git a/arch/avr32/kernel/process.c b/arch/avr32/kernel/process.c
deleted file mode 100644
index ad0dfcc..0000000
--- a/arch/avr32/kernel/process.c
+++ /dev/null
@@ -1,358 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/sched.h>
-#include <linux/sched/debug.h>
-#include <linux/sched/task.h>
-#include <linux/sched/task_stack.h>
-#include <linux/module.h>
-#include <linux/kallsyms.h>
-#include <linux/fs.h>
-#include <linux/pm.h>
-#include <linux/ptrace.h>
-#include <linux/slab.h>
-#include <linux/reboot.h>
-#include <linux/tick.h>
-#include <linux/uaccess.h>
-#include <linux/unistd.h>
-
-#include <asm/sysreg.h>
-#include <asm/ocd.h>
-#include <asm/syscalls.h>
-
-#include <mach/pm.h>
-
-void (*pm_power_off)(void);
-EXPORT_SYMBOL(pm_power_off);
-
-/*
- * This file handles the architecture-dependent parts of process handling..
- */
-
-void arch_cpu_idle(void)
-{
- cpu_enter_idle();
-}
-
-void machine_halt(void)
-{
- /*
- * Enter Stop mode. The 32 kHz oscillator will keep running so
- * the RTC will keep the time properly and the system will
- * boot quickly.
- */
- asm volatile("sleep 3\n\t"
- "sub pc, -2");
-}
-
-void machine_power_off(void)
-{
- if (pm_power_off)
- pm_power_off();
-}
-
-void machine_restart(char *cmd)
-{
- ocd_write(DC, (1 << OCD_DC_DBE_BIT));
- ocd_write(DC, (1 << OCD_DC_RES_BIT));
- while (1) ;
-}
-
-/*
- * Free current thread data structures etc
- */
-void exit_thread(struct task_struct *tsk)
-{
- ocd_disable(tsk);
-}
-
-void flush_thread(void)
-{
- /* nothing to do */
-}
-
-void release_thread(struct task_struct *dead_task)
-{
- /* do nothing */
-}
-
-static void dump_mem(const char *str, const char *log_lvl,
- unsigned long bottom, unsigned long top)
-{
- unsigned long p;
- int i;
-
- printk("%s%s(0x%08lx to 0x%08lx)\n", log_lvl, str, bottom, top);
-
- for (p = bottom & ~31; p < top; ) {
- printk("%s%04lx: ", log_lvl, p & 0xffff);
-
- for (i = 0; i < 8; i++, p += 4) {
- unsigned int val;
-
- if (p < bottom || p >= top)
- printk(" ");
- else {
- if (__get_user(val, (unsigned int __user *)p)) {
- printk("\n");
- goto out;
- }
- printk("%08x ", val);
- }
- }
- printk("\n");
- }
-
-out:
- return;
-}
-
-static inline int valid_stack_ptr(struct thread_info *tinfo, unsigned long p)
-{
- return (p > (unsigned long)tinfo)
- && (p < (unsigned long)tinfo + THREAD_SIZE - 3);
-}
-
-#ifdef CONFIG_FRAME_POINTER
-static void show_trace_log_lvl(struct task_struct *tsk, unsigned long *sp,
- struct pt_regs *regs, const char *log_lvl)
-{
- unsigned long lr, fp;
- struct thread_info *tinfo;
-
- if (regs)
- fp = regs->r7;
- else if (tsk == current)
- asm("mov %0, r7" : "=r"(fp));
- else
- fp = tsk->thread.cpu_context.r7;
-
- /*
- * Walk the stack as long as the frame pointer (a) is within
- * the kernel stack of the task, and (b) it doesn't move
- * downwards.
- */
- tinfo = task_thread_info(tsk);
- printk("%sCall trace:\n", log_lvl);
- while (valid_stack_ptr(tinfo, fp)) {
- unsigned long new_fp;
-
- lr = *(unsigned long *)fp;
-#ifdef CONFIG_KALLSYMS
- printk("%s [<%08lx>] ", log_lvl, lr);
-#else
- printk(" [<%08lx>] ", lr);
-#endif
- print_symbol("%s\n", lr);
-
- new_fp = *(unsigned long *)(fp + 4);
- if (new_fp <= fp)
- break;
- fp = new_fp;
- }
- printk("\n");
-}
-#else
-static void show_trace_log_lvl(struct task_struct *tsk, unsigned long *sp,
- struct pt_regs *regs, const char *log_lvl)
-{
- unsigned long addr;
-
- printk("%sCall trace:\n", log_lvl);
-
- while (!kstack_end(sp)) {
- addr = *sp++;
- if (kernel_text_address(addr)) {
-#ifdef CONFIG_KALLSYMS
- printk("%s [<%08lx>] ", log_lvl, addr);
-#else
- printk(" [<%08lx>] ", addr);
-#endif
- print_symbol("%s\n", addr);
- }
- }
- printk("\n");
-}
-#endif
-
-void show_stack_log_lvl(struct task_struct *tsk, unsigned long sp,
- struct pt_regs *regs, const char *log_lvl)
-{
- struct thread_info *tinfo;
-
- if (sp == 0) {
- if (tsk)
- sp = tsk->thread.cpu_context.ksp;
- else
- sp = (unsigned long)&tinfo;
- }
- if (!tsk)
- tsk = current;
-
- tinfo = task_thread_info(tsk);
-
- if (valid_stack_ptr(tinfo, sp)) {
- dump_mem("Stack: ", log_lvl, sp,
- THREAD_SIZE + (unsigned long)tinfo);
- show_trace_log_lvl(tsk, (unsigned long *)sp, regs, log_lvl);
- }
-}
-
-void show_stack(struct task_struct *tsk, unsigned long *stack)
-{
- show_stack_log_lvl(tsk, (unsigned long)stack, NULL, "");
-}
-
-static const char *cpu_modes[] = {
- "Application", "Supervisor", "Interrupt level 0", "Interrupt level 1",
- "Interrupt level 2", "Interrupt level 3", "Exception", "NMI"
-};
-
-void show_regs_log_lvl(struct pt_regs *regs, const char *log_lvl)
-{
- unsigned long sp = regs->sp;
- unsigned long lr = regs->lr;
- unsigned long mode = (regs->sr & MODE_MASK) >> MODE_SHIFT;
-
- show_regs_print_info(log_lvl);
-
- if (!user_mode(regs)) {
- sp = (unsigned long)regs + FRAME_SIZE_FULL;
-
- printk("%s", log_lvl);
- print_symbol("PC is at %s\n", instruction_pointer(regs));
- printk("%s", log_lvl);
- print_symbol("LR is at %s\n", lr);
- }
-
- printk("%spc : [<%08lx>] lr : [<%08lx>] %s\n"
- "%ssp : %08lx r12: %08lx r11: %08lx\n",
- log_lvl, instruction_pointer(regs), lr, print_tainted(),
- log_lvl, sp, regs->r12, regs->r11);
- printk("%sr10: %08lx r9 : %08lx r8 : %08lx\n",
- log_lvl, regs->r10, regs->r9, regs->r8);
- printk("%sr7 : %08lx r6 : %08lx r5 : %08lx r4 : %08lx\n",
- log_lvl, regs->r7, regs->r6, regs->r5, regs->r4);
- printk("%sr3 : %08lx r2 : %08lx r1 : %08lx r0 : %08lx\n",
- log_lvl, regs->r3, regs->r2, regs->r1, regs->r0);
- printk("%sFlags: %c%c%c%c%c\n", log_lvl,
- regs->sr & SR_Q ? 'Q' : 'q',
- regs->sr & SR_V ? 'V' : 'v',
- regs->sr & SR_N ? 'N' : 'n',
- regs->sr & SR_Z ? 'Z' : 'z',
- regs->sr & SR_C ? 'C' : 'c');
- printk("%sMode bits: %c%c%c%c%c%c%c%c%c%c\n", log_lvl,
- regs->sr & SR_H ? 'H' : 'h',
- regs->sr & SR_J ? 'J' : 'j',
- regs->sr & SR_DM ? 'M' : 'm',
- regs->sr & SR_D ? 'D' : 'd',
- regs->sr & SR_EM ? 'E' : 'e',
- regs->sr & SR_I3M ? '3' : '.',
- regs->sr & SR_I2M ? '2' : '.',
- regs->sr & SR_I1M ? '1' : '.',
- regs->sr & SR_I0M ? '0' : '.',
- regs->sr & SR_GM ? 'G' : 'g');
- printk("%sCPU Mode: %s\n", log_lvl, cpu_modes[mode]);
-}
-
-void show_regs(struct pt_regs *regs)
-{
- unsigned long sp = regs->sp;
-
- if (!user_mode(regs))
- sp = (unsigned long)regs + FRAME_SIZE_FULL;
-
- show_regs_log_lvl(regs, "");
- show_trace_log_lvl(current, (unsigned long *)sp, regs, "");
-}
-EXPORT_SYMBOL(show_regs);
-
-/* Fill in the fpu structure for a core dump. This is easy -- we don't have any */
-int dump_fpu(struct pt_regs *regs, elf_fpregset_t *fpu)
-{
- /* Not valid */
- return 0;
-}
-
-asmlinkage void ret_from_fork(void);
-asmlinkage void ret_from_kernel_thread(void);
-asmlinkage void syscall_return(void);
-
-int copy_thread(unsigned long clone_flags, unsigned long usp,
- unsigned long arg,
- struct task_struct *p)
-{
- struct pt_regs *childregs = task_pt_regs(p);
-
- if (unlikely(p->flags & PF_KTHREAD)) {
- memset(childregs, 0, sizeof(struct pt_regs));
- p->thread.cpu_context.r0 = arg;
- p->thread.cpu_context.r1 = usp; /* fn */
- p->thread.cpu_context.r2 = (unsigned long)syscall_return;
- p->thread.cpu_context.pc = (unsigned long)ret_from_kernel_thread;
- childregs->sr = MODE_SUPERVISOR;
- } else {
- *childregs = *current_pt_regs();
- if (usp)
- childregs->sp = usp;
- childregs->r12 = 0; /* Set return value for child */
- p->thread.cpu_context.pc = (unsigned long)ret_from_fork;
- }
-
- p->thread.cpu_context.sr = MODE_SUPERVISOR | SR_GM;
- p->thread.cpu_context.ksp = (unsigned long)childregs;
-
- clear_tsk_thread_flag(p, TIF_DEBUG);
- if ((clone_flags & CLONE_PTRACE) && test_thread_flag(TIF_DEBUG))
- ocd_enable(p);
-
- return 0;
-}
-
-/*
- * This function is supposed to answer the question "who called
- * schedule()?"
- */
-unsigned long get_wchan(struct task_struct *p)
-{
- unsigned long pc;
- unsigned long stack_page;
-
- if (!p || p == current || p->state == TASK_RUNNING)
- return 0;
-
- stack_page = (unsigned long)task_stack_page(p);
- BUG_ON(!stack_page);
-
- /*
- * The stored value of PC is either the address right after
- * the call to __switch_to() or ret_from_fork.
- */
- pc = thread_saved_pc(p);
- if (in_sched_functions(pc)) {
-#ifdef CONFIG_FRAME_POINTER
- unsigned long fp = p->thread.cpu_context.r7;
- BUG_ON(fp < stack_page || fp > (THREAD_SIZE + stack_page));
- pc = *(unsigned long *)fp;
-#else
- /*
- * We depend on the frame size of schedule here, which
- * is actually quite ugly. It might be possible to
- * determine the frame size automatically at build
- * time by doing this:
- * - compile sched/core.c
- * - disassemble the resulting sched.o
- * - look for 'sub sp,??' shortly after '<schedule>:'
- */
- unsigned long sp = p->thread.cpu_context.ksp + 16;
- BUG_ON(sp < stack_page || sp > (THREAD_SIZE + stack_page));
- pc = *(unsigned long *)sp;
-#endif
- }
-
- return pc;
-}
diff --git a/arch/avr32/kernel/ptrace.c b/arch/avr32/kernel/ptrace.c
deleted file mode 100644
index 41a14e9..0000000
--- a/arch/avr32/kernel/ptrace.c
+++ /dev/null
@@ -1,357 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#undef DEBUG
-#include <linux/kernel.h>
-#include <linux/sched.h>
-#include <linux/sched/task_stack.h>
-#include <linux/mm.h>
-#include <linux/ptrace.h>
-#include <linux/errno.h>
-#include <linux/user.h>
-#include <linux/security.h>
-#include <linux/unistd.h>
-#include <linux/notifier.h>
-
-#include <asm/traps.h>
-#include <linux/uaccess.h>
-#include <asm/ocd.h>
-#include <asm/mmu_context.h>
-#include <linux/kdebug.h>
-
-static struct pt_regs *get_user_regs(struct task_struct *tsk)
-{
- return (struct pt_regs *)((unsigned long)task_stack_page(tsk) +
- THREAD_SIZE - sizeof(struct pt_regs));
-}
-
-void user_enable_single_step(struct task_struct *tsk)
-{
- pr_debug("user_enable_single_step: pid=%u, PC=0x%08lx, SR=0x%08lx\n",
- tsk->pid, task_pt_regs(tsk)->pc, task_pt_regs(tsk)->sr);
-
- /*
- * We can't schedule in Debug mode, so when TIF_BREAKPOINT is
- * set, the system call or exception handler will do a
- * breakpoint to enter monitor mode before returning to
- * userspace.
- *
- * The monitor code will then notice that TIF_SINGLE_STEP is
- * set and return to userspace with single stepping enabled.
- * The CPU will then enter monitor mode again after exactly
- * one instruction has been executed, and the monitor code
- * will then send a SIGTRAP to the process.
- */
- set_tsk_thread_flag(tsk, TIF_BREAKPOINT);
- set_tsk_thread_flag(tsk, TIF_SINGLE_STEP);
-}
-
-void user_disable_single_step(struct task_struct *child)
-{
- /* XXX(hch): a no-op here seems wrong.. */
-}
-
-/*
- * Called by kernel/ptrace.c when detaching
- *
- * Make sure any single step bits, etc. are not set
- */
-void ptrace_disable(struct task_struct *child)
-{
- clear_tsk_thread_flag(child, TIF_SINGLE_STEP);
- clear_tsk_thread_flag(child, TIF_BREAKPOINT);
- ocd_disable(child);
-}
-
-/*
- * Read the word at offset "offset" into the task's "struct user". We
- * actually access the pt_regs struct stored on the kernel stack.
- */
-static int ptrace_read_user(struct task_struct *tsk, unsigned long offset,
- unsigned long __user *data)
-{
- unsigned long *regs;
- unsigned long value;
-
- if (offset & 3 || offset >= sizeof(struct user)) {
- printk("ptrace_read_user: invalid offset 0x%08lx\n", offset);
- return -EIO;
- }
-
- regs = (unsigned long *)get_user_regs(tsk);
-
- value = 0;
- if (offset < sizeof(struct pt_regs))
- value = regs[offset / sizeof(regs[0])];
-
- pr_debug("ptrace_read_user(%s[%u], %#lx, %p) -> %#lx\n",
- tsk->comm, tsk->pid, offset, data, value);
-
- return put_user(value, data);
-}
-
-/*
- * Write the word "value" to offset "offset" into the task's "struct
- * user". We actually access the pt_regs struct stored on the kernel
- * stack.
- */
-static int ptrace_write_user(struct task_struct *tsk, unsigned long offset,
- unsigned long value)
-{
- unsigned long *regs;
-
- pr_debug("ptrace_write_user(%s[%u], %#lx, %#lx)\n",
- tsk->comm, tsk->pid, offset, value);
-
- if (offset & 3 || offset >= sizeof(struct user)) {
- pr_debug(" invalid offset 0x%08lx\n", offset);
- return -EIO;
- }
-
- if (offset >= sizeof(struct pt_regs))
- return 0;
-
- regs = (unsigned long *)get_user_regs(tsk);
- regs[offset / sizeof(regs[0])] = value;
-
- return 0;
-}
-
-static int ptrace_getregs(struct task_struct *tsk, void __user *uregs)
-{
- struct pt_regs *regs = get_user_regs(tsk);
-
- return copy_to_user(uregs, regs, sizeof(*regs)) ? -EFAULT : 0;
-}
-
-static int ptrace_setregs(struct task_struct *tsk, const void __user *uregs)
-{
- struct pt_regs newregs;
- int ret;
-
- ret = -EFAULT;
- if (copy_from_user(&newregs, uregs, sizeof(newregs)) == 0) {
- struct pt_regs *regs = get_user_regs(tsk);
-
- ret = -EINVAL;
- if (valid_user_regs(&newregs)) {
- *regs = newregs;
- ret = 0;
- }
- }
-
- return ret;
-}
-
-long arch_ptrace(struct task_struct *child, long request,
- unsigned long addr, unsigned long data)
-{
- int ret;
- void __user *datap = (void __user *) data;
-
- switch (request) {
- /* Read the word at location addr in the child process */
- case PTRACE_PEEKTEXT:
- case PTRACE_PEEKDATA:
- ret = generic_ptrace_peekdata(child, addr, data);
- break;
-
- case PTRACE_PEEKUSR:
- ret = ptrace_read_user(child, addr, datap);
- break;
-
- /* Write the word in data at location addr */
- case PTRACE_POKETEXT:
- case PTRACE_POKEDATA:
- ret = generic_ptrace_pokedata(child, addr, data);
- break;
-
- case PTRACE_POKEUSR:
- ret = ptrace_write_user(child, addr, data);
- break;
-
- case PTRACE_GETREGS:
- ret = ptrace_getregs(child, datap);
- break;
-
- case PTRACE_SETREGS:
- ret = ptrace_setregs(child, datap);
- break;
-
- default:
- ret = ptrace_request(child, request, addr, data);
- break;
- }
-
- return ret;
-}
-
-asmlinkage void syscall_trace(void)
-{
- if (!test_thread_flag(TIF_SYSCALL_TRACE))
- return;
- if (!(current->ptrace & PT_PTRACED))
- return;
-
- /* The 0x80 provides a way for the tracing parent to
- * distinguish between a syscall stop and SIGTRAP delivery */
- ptrace_notify(SIGTRAP | ((current->ptrace & PT_TRACESYSGOOD)
- ? 0x80 : 0));
-
- /*
- * this isn't the same as continuing with a signal, but it
- * will do for normal use. strace only continues with a
- * signal if the stopping signal is not SIGTRAP. -brl
- */
- if (current->exit_code) {
- pr_debug("syscall_trace: sending signal %d to PID %u\n",
- current->exit_code, current->pid);
- send_sig(current->exit_code, current, 1);
- current->exit_code = 0;
- }
-}
-
-/*
- * debug_trampoline() is an assembly stub which will store all user
- * registers on the stack and execute a breakpoint instruction.
- *
- * If we single-step into an exception handler which runs with
- * interrupts disabled the whole time so it doesn't have to check for
- * pending work, its return address will be modified so that it ends
- * up returning to debug_trampoline.
- *
- * If the exception handler decides to store the user context and
- * enable interrupts after all, it will restore the original return
- * address and status register value. Before it returns, it will
- * notice that TIF_BREAKPOINT is set and execute a breakpoint
- * instruction.
- */
-extern void debug_trampoline(void);
-
-asmlinkage struct pt_regs *do_debug(struct pt_regs *regs)
-{
- struct thread_info *ti;
- unsigned long trampoline_addr;
- u32 status;
- u32 ctrl;
- int code;
-
- status = ocd_read(DS);
- ti = current_thread_info();
- code = TRAP_BRKPT;
-
- pr_debug("do_debug: status=0x%08x PC=0x%08lx SR=0x%08lx tif=0x%08lx\n",
- status, regs->pc, regs->sr, ti->flags);
-
- if (!user_mode(regs)) {
- unsigned long die_val = DIE_BREAKPOINT;
-
- if (status & (1 << OCD_DS_SSS_BIT))
- die_val = DIE_SSTEP;
-
- if (notify_die(die_val, "ptrace", regs, 0, 0, SIGTRAP)
- == NOTIFY_STOP)
- return regs;
-
- if ((status & (1 << OCD_DS_SWB_BIT))
- && test_and_clear_ti_thread_flag(
- ti, TIF_BREAKPOINT)) {
- /*
- * Explicit breakpoint from trampoline or
- * exception/syscall/interrupt handler.
- *
- * The real saved regs are on the stack right
- * after the ones we saved on entry.
- */
- regs++;
- pr_debug(" -> TIF_BREAKPOINT done, adjusted regs:"
- "PC=0x%08lx SR=0x%08lx\n",
- regs->pc, regs->sr);
- BUG_ON(!user_mode(regs));
-
- if (test_thread_flag(TIF_SINGLE_STEP)) {
- pr_debug("Going to do single step...\n");
- return regs;
- }
-
- /*
- * No TIF_SINGLE_STEP means we're done
- * stepping over a syscall. Do the trap now.
- */
- code = TRAP_TRACE;
- } else if ((status & (1 << OCD_DS_SSS_BIT))
- && test_ti_thread_flag(ti, TIF_SINGLE_STEP)) {
-
- pr_debug("Stepped into something, "
- "setting TIF_BREAKPOINT...\n");
- set_ti_thread_flag(ti, TIF_BREAKPOINT);
-
- /*
- * We stepped into an exception, interrupt or
- * syscall handler. Some exception handlers
- * don't check for pending work, so we need to
- * set up a trampoline just in case.
- *
- * The exception entry code will undo the
- * trampoline stuff if it does a full context
- * save (which also means that it'll check for
- * pending work later.)
- */
- if ((regs->sr & MODE_MASK) == MODE_EXCEPTION) {
- trampoline_addr
- = (unsigned long)&debug_trampoline;
-
- pr_debug("Setting up trampoline...\n");
- ti->rar_saved = sysreg_read(RAR_EX);
- ti->rsr_saved = sysreg_read(RSR_EX);
- sysreg_write(RAR_EX, trampoline_addr);
- sysreg_write(RSR_EX, (MODE_EXCEPTION
- | SR_EM | SR_GM));
- BUG_ON(ti->rsr_saved & MODE_MASK);
- }
-
- /*
- * If we stepped into a system call, we
- * shouldn't do a single step after we return
- * since the return address is right after the
- * "scall" instruction we were told to step
- * over.
- */
- if ((regs->sr & MODE_MASK) == MODE_SUPERVISOR) {
- pr_debug("Supervisor; no single step\n");
- clear_ti_thread_flag(ti, TIF_SINGLE_STEP);
- }
-
- ctrl = ocd_read(DC);
- ctrl &= ~(1 << OCD_DC_SS_BIT);
- ocd_write(DC, ctrl);
-
- return regs;
- } else {
- printk(KERN_ERR "Unexpected OCD_DS value: 0x%08x\n",
- status);
- printk(KERN_ERR "Thread flags: 0x%08lx\n", ti->flags);
- die("Unhandled debug trap in kernel mode",
- regs, SIGTRAP);
- }
- } else if (status & (1 << OCD_DS_SSS_BIT)) {
- /* Single step in user mode */
- code = TRAP_TRACE;
-
- ctrl = ocd_read(DC);
- ctrl &= ~(1 << OCD_DC_SS_BIT);
- ocd_write(DC, ctrl);
- }
-
- pr_debug("Sending SIGTRAP: code=%d PC=0x%08lx SR=0x%08lx\n",
- code, regs->pc, regs->sr);
-
- clear_thread_flag(TIF_SINGLE_STEP);
- _exception(SIGTRAP, regs, code, instruction_pointer(regs));
-
- return regs;
-}
diff --git a/arch/avr32/kernel/setup.c b/arch/avr32/kernel/setup.c
deleted file mode 100644
index e692889..0000000
--- a/arch/avr32/kernel/setup.c
+++ /dev/null
@@ -1,609 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/clk.h>
-#include <linux/init.h>
-#include <linux/initrd.h>
-#include <linux/sched.h>
-#include <linux/console.h>
-#include <linux/ioport.h>
-#include <linux/bootmem.h>
-#include <linux/fs.h>
-#include <linux/module.h>
-#include <linux/pfn.h>
-#include <linux/root_dev.h>
-#include <linux/cpu.h>
-#include <linux/kernel.h>
-
-#include <asm/sections.h>
-#include <asm/processor.h>
-#include <asm/pgtable.h>
-#include <asm/setup.h>
-#include <asm/sysreg.h>
-
-#include <mach/board.h>
-#include <mach/init.h>
-
-extern int root_mountflags;
-
-/*
- * Initialize loops_per_jiffy as 5000000 (500MIPS).
- * Better make it too large than too small...
- */
-struct avr32_cpuinfo boot_cpu_data = {
- .loops_per_jiffy = 5000000
-};
-EXPORT_SYMBOL(boot_cpu_data);
-
-static char __initdata command_line[COMMAND_LINE_SIZE];
-
-/*
- * Standard memory resources
- */
-static struct resource __initdata kernel_data = {
- .name = "Kernel data",
- .start = 0,
- .end = 0,
- .flags = IORESOURCE_SYSTEM_RAM,
-};
-static struct resource __initdata kernel_code = {
- .name = "Kernel code",
- .start = 0,
- .end = 0,
- .flags = IORESOURCE_SYSTEM_RAM,
- .sibling = &kernel_data,
-};
-
-/*
- * Available system RAM and reserved regions as singly linked
- * lists. These lists are traversed using the sibling pointer in
- * struct resource and are kept sorted at all times.
- */
-static struct resource *__initdata system_ram;
-static struct resource *__initdata reserved = &kernel_code;
-
-/*
- * We need to allocate these before the bootmem allocator is up and
- * running, so we need this "cache". 32 entries are probably enough
- * for all but the most insanely complex systems.
- */
-static struct resource __initdata res_cache[32];
-static unsigned int __initdata res_cache_next_free;
-
-static void __init resource_init(void)
-{
- struct resource *mem, *res;
- struct resource *new;
-
- kernel_code.start = __pa(init_mm.start_code);
-
- for (mem = system_ram; mem; mem = mem->sibling) {
- new = alloc_bootmem_low(sizeof(struct resource));
- memcpy(new, mem, sizeof(struct resource));
-
- new->sibling = NULL;
- if (request_resource(&iomem_resource, new))
- printk(KERN_WARNING "Bad RAM resource %08x-%08x\n",
- mem->start, mem->end);
- }
-
- for (res = reserved; res; res = res->sibling) {
- new = alloc_bootmem_low(sizeof(struct resource));
- memcpy(new, res, sizeof(struct resource));
-
- new->sibling = NULL;
- if (insert_resource(&iomem_resource, new))
- printk(KERN_WARNING
- "Bad reserved resource %s (%08x-%08x)\n",
- res->name, res->start, res->end);
- }
-}
-
-static void __init
-add_physical_memory(resource_size_t start, resource_size_t end)
-{
- struct resource *new, *next, **pprev;
-
- for (pprev = &system_ram, next = system_ram; next;
- pprev = &next->sibling, next = next->sibling) {
- if (end < next->start)
- break;
- if (start <= next->end) {
- printk(KERN_WARNING
- "Warning: Physical memory map is broken\n");
- printk(KERN_WARNING
- "Warning: %08x-%08x overlaps %08x-%08x\n",
- start, end, next->start, next->end);
- return;
- }
- }
-
- if (res_cache_next_free >= ARRAY_SIZE(res_cache)) {
- printk(KERN_WARNING
- "Warning: Failed to add physical memory %08x-%08x\n",
- start, end);
- return;
- }
-
- new = &res_cache[res_cache_next_free++];
- new->start = start;
- new->end = end;
- new->name = "System RAM";
- new->flags = IORESOURCE_SYSTEM_RAM;
-
- *pprev = new;
-}
-
-static int __init
-add_reserved_region(resource_size_t start, resource_size_t end,
- const char *name)
-{
- struct resource *new, *next, **pprev;
-
- if (end < start)
- return -EINVAL;
-
- if (res_cache_next_free >= ARRAY_SIZE(res_cache))
- return -ENOMEM;
-
- for (pprev = &reserved, next = reserved; next;
- pprev = &next->sibling, next = next->sibling) {
- if (end < next->start)
- break;
- if (start <= next->end)
- return -EBUSY;
- }
-
- new = &res_cache[res_cache_next_free++];
- new->start = start;
- new->end = end;
- new->name = name;
- new->sibling = next;
- new->flags = IORESOURCE_MEM;
-
- *pprev = new;
-
- return 0;
-}
-
-static unsigned long __init
-find_free_region(const struct resource *mem, resource_size_t size,
- resource_size_t align)
-{
- struct resource *res;
- unsigned long target;
-
- target = ALIGN(mem->start, align);
- for (res = reserved; res; res = res->sibling) {
- if ((target + size) <= res->start)
- break;
- if (target <= res->end)
- target = ALIGN(res->end + 1, align);
- }
-
- if ((target + size) > (mem->end + 1))
- return mem->end + 1;
-
- return target;
-}
-
-static int __init
-alloc_reserved_region(resource_size_t *start, resource_size_t size,
- resource_size_t align, const char *name)
-{
- struct resource *mem;
- resource_size_t target;
- int ret;
-
- for (mem = system_ram; mem; mem = mem->sibling) {
- target = find_free_region(mem, size, align);
- if (target <= mem->end) {
- ret = add_reserved_region(target, target + size - 1,
- name);
- if (!ret)
- *start = target;
- return ret;
- }
- }
-
- return -ENOMEM;
-}
-
-/*
- * Early framebuffer allocation. Works as follows:
- * - If fbmem_size is zero, nothing will be allocated or reserved.
- * - If fbmem_start is zero when setup_bootmem() is called,
- * a block of fbmem_size bytes will be reserved before bootmem
- * initialization. It will be aligned to the largest page size
- * that fbmem_size is a multiple of.
- * - If fbmem_start is nonzero, an area of size fbmem_size will be
- * reserved at the physical address fbmem_start if possible. If
- * it collides with other reserved memory, a different block of
- * same size will be allocated, just as if fbmem_start was zero.
- *
- * Board-specific code may use these variables to set up platform data
- * for the framebuffer driver if fbmem_size is nonzero.
- */
-resource_size_t __initdata fbmem_start;
-resource_size_t __initdata fbmem_size;
-
-/*
- * "fbmem=xxx[kKmM]" allocates the specified amount of boot memory for
- * use as framebuffer.
- *
- * "fbmem=xxx[kKmM]@yyy[kKmM]" defines a memory region of size xxx and
- * starting at yyy to be reserved for use as framebuffer.
- *
- * The kernel won't verify that the memory region starting at yyy
- * actually contains usable RAM.
- */
-static int __init early_parse_fbmem(char *p)
-{
- int ret;
- unsigned long align;
-
- fbmem_size = memparse(p, &p);
- if (*p == '@') {
- fbmem_start = memparse(p + 1, &p);
- ret = add_reserved_region(fbmem_start,
- fbmem_start + fbmem_size - 1,
- "Framebuffer");
- if (ret) {
- printk(KERN_WARNING
- "Failed to reserve framebuffer memory\n");
- fbmem_start = 0;
- }
- }
-
- if (!fbmem_start) {
- if ((fbmem_size & 0x000fffffUL) == 0)
- align = 0x100000; /* 1 MiB */
- else if ((fbmem_size & 0x0000ffffUL) == 0)
- align = 0x10000; /* 64 KiB */
- else
- align = 0x1000; /* 4 KiB */
-
- ret = alloc_reserved_region(&fbmem_start, fbmem_size,
- align, "Framebuffer");
- if (ret) {
- printk(KERN_WARNING
- "Failed to allocate framebuffer memory\n");
- fbmem_size = 0;
- } else {
- memset(__va(fbmem_start), 0, fbmem_size);
- }
- }
-
- return 0;
-}
-early_param("fbmem", early_parse_fbmem);
-
-/*
- * Pick out the memory size. We look for mem=size@start,
- * where start and size are "size[KkMmGg]"
- */
-static int __init early_mem(char *p)
-{
- resource_size_t size, start;
-
- start = system_ram->start;
- size = memparse(p, &p);
- if (*p == '@')
- start = memparse(p + 1, &p);
-
- system_ram->start = start;
- system_ram->end = system_ram->start + size - 1;
- return 0;
-}
-early_param("mem", early_mem);
-
-static int __init parse_tag_core(struct tag *tag)
-{
- if (tag->hdr.size > 2) {
- if ((tag->u.core.flags & 1) == 0)
- root_mountflags &= ~MS_RDONLY;
- ROOT_DEV = new_decode_dev(tag->u.core.rootdev);
- }
- return 0;
-}
-__tagtable(ATAG_CORE, parse_tag_core);
-
-static int __init parse_tag_mem(struct tag *tag)
-{
- unsigned long start, end;
-
- /*
- * Ignore zero-sized entries. If we're running standalone, the
- * SDRAM code may emit such entries if something goes
- * wrong...
- */
- if (tag->u.mem_range.size == 0)
- return 0;
-
- start = tag->u.mem_range.addr;
- end = tag->u.mem_range.addr + tag->u.mem_range.size - 1;
-
- add_physical_memory(start, end);
- return 0;
-}
-__tagtable(ATAG_MEM, parse_tag_mem);
-
-static int __init parse_tag_rdimg(struct tag *tag)
-{
-#ifdef CONFIG_BLK_DEV_INITRD
- struct tag_mem_range *mem = &tag->u.mem_range;
- int ret;
-
- if (initrd_start) {
- printk(KERN_WARNING
- "Warning: Only the first initrd image will be used\n");
- return 0;
- }
-
- ret = add_reserved_region(mem->addr, mem->addr + mem->size - 1,
- "initrd");
- if (ret) {
- printk(KERN_WARNING
- "Warning: Failed to reserve initrd memory\n");
- return ret;
- }
-
- initrd_start = (unsigned long)__va(mem->addr);
- initrd_end = initrd_start + mem->size;
-#else
- printk(KERN_WARNING "RAM disk image present, but "
- "no initrd support in kernel, ignoring\n");
-#endif
-
- return 0;
-}
-__tagtable(ATAG_RDIMG, parse_tag_rdimg);
-
-static int __init parse_tag_rsvd_mem(struct tag *tag)
-{
- struct tag_mem_range *mem = &tag->u.mem_range;
-
- return add_reserved_region(mem->addr, mem->addr + mem->size - 1,
- "Reserved");
-}
-__tagtable(ATAG_RSVD_MEM, parse_tag_rsvd_mem);
-
-static int __init parse_tag_cmdline(struct tag *tag)
-{
- strlcpy(boot_command_line, tag->u.cmdline.cmdline, COMMAND_LINE_SIZE);
- return 0;
-}
-__tagtable(ATAG_CMDLINE, parse_tag_cmdline);
-
-static int __init parse_tag_clock(struct tag *tag)
-{
- /*
- * We'll figure out the clocks by peeking at the system
- * manager regs directly.
- */
- return 0;
-}
-__tagtable(ATAG_CLOCK, parse_tag_clock);
-
-/*
- * The board_number correspond to the bd->bi_board_number in U-Boot. This
- * parameter is only available during initialisation and can be used in some
- * kind of board identification.
- */
-u32 __initdata board_number;
-
-static int __init parse_tag_boardinfo(struct tag *tag)
-{
- board_number = tag->u.boardinfo.board_number;
-
- return 0;
-}
-__tagtable(ATAG_BOARDINFO, parse_tag_boardinfo);
-
-/*
- * Scan the tag table for this tag, and call its parse function. The
- * tag table is built by the linker from all the __tagtable
- * declarations.
- */
-static int __init parse_tag(struct tag *tag)
-{
- extern struct tagtable __tagtable_begin, __tagtable_end;
- struct tagtable *t;
-
- for (t = &__tagtable_begin; t < &__tagtable_end; t++)
- if (tag->hdr.tag == t->tag) {
- t->parse(tag);
- break;
- }
-
- return t < &__tagtable_end;
-}
-
-/*
- * Parse all tags in the list we got from the boot loader
- */
-static void __init parse_tags(struct tag *t)
-{
- for (; t->hdr.tag != ATAG_NONE; t = tag_next(t))
- if (!parse_tag(t))
- printk(KERN_WARNING
- "Ignoring unrecognised tag 0x%08x\n",
- t->hdr.tag);
-}
-
-/*
- * Find a free memory region large enough for storing the
- * bootmem bitmap.
- */
-static unsigned long __init
-find_bootmap_pfn(const struct resource *mem)
-{
- unsigned long bootmap_pages, bootmap_len;
- unsigned long node_pages = PFN_UP(resource_size(mem));
- unsigned long bootmap_start;
-
- bootmap_pages = bootmem_bootmap_pages(node_pages);
- bootmap_len = bootmap_pages << PAGE_SHIFT;
-
- /*
- * Find a large enough region without reserved pages for
- * storing the bootmem bitmap. We can take advantage of the
- * fact that all lists have been sorted.
- *
- * We have to check that we don't collide with any reserved
- * regions, which includes the kernel image and any RAMDISK
- * images.
- */
- bootmap_start = find_free_region(mem, bootmap_len, PAGE_SIZE);
-
- return bootmap_start >> PAGE_SHIFT;
-}
-
-#define MAX_LOWMEM HIGHMEM_START
-#define MAX_LOWMEM_PFN PFN_DOWN(MAX_LOWMEM)
-
-static void __init setup_bootmem(void)
-{
- unsigned bootmap_size;
- unsigned long first_pfn, bootmap_pfn, pages;
- unsigned long max_pfn, max_low_pfn;
- unsigned node = 0;
- struct resource *res;
-
- printk(KERN_INFO "Physical memory:\n");
- for (res = system_ram; res; res = res->sibling)
- printk(" %08x-%08x\n", res->start, res->end);
- printk(KERN_INFO "Reserved memory:\n");
- for (res = reserved; res; res = res->sibling)
- printk(" %08x-%08x: %s\n",
- res->start, res->end, res->name);
-
- nodes_clear(node_online_map);
-
- if (system_ram->sibling)
- printk(KERN_WARNING "Only using first memory bank\n");
-
- for (res = system_ram; res; res = NULL) {
- first_pfn = PFN_UP(res->start);
- max_low_pfn = max_pfn = PFN_DOWN(res->end + 1);
- bootmap_pfn = find_bootmap_pfn(res);
- if (bootmap_pfn > max_pfn)
- panic("No space for bootmem bitmap!\n");
-
- if (max_low_pfn > MAX_LOWMEM_PFN) {
- max_low_pfn = MAX_LOWMEM_PFN;
-#ifndef CONFIG_HIGHMEM
- /*
- * Lowmem is memory that can be addressed
- * directly through P1/P2
- */
- printk(KERN_WARNING
- "Node %u: Only %ld MiB of memory will be used.\n",
- node, MAX_LOWMEM >> 20);
- printk(KERN_WARNING "Use a HIGHMEM enabled kernel.\n");
-#else
-#error HIGHMEM is not supported by AVR32 yet
-#endif
- }
-
- /* Initialize the boot-time allocator with low memory only. */
- bootmap_size = init_bootmem_node(NODE_DATA(node), bootmap_pfn,
- first_pfn, max_low_pfn);
-
- /*
- * Register fully available RAM pages with the bootmem
- * allocator.
- */
- pages = max_low_pfn - first_pfn;
- free_bootmem_node (NODE_DATA(node), PFN_PHYS(first_pfn),
- PFN_PHYS(pages));
-
- /* Reserve space for the bootmem bitmap... */
- reserve_bootmem_node(NODE_DATA(node),
- PFN_PHYS(bootmap_pfn),
- bootmap_size,
- BOOTMEM_DEFAULT);
-
- /* ...and any other reserved regions. */
- for (res = reserved; res; res = res->sibling) {
- if (res->start > PFN_PHYS(max_pfn))
- break;
-
- /*
- * resource_init will complain about partial
- * overlaps, so we'll just ignore such
- * resources for now.
- */
- if (res->start >= PFN_PHYS(first_pfn)
- && res->end < PFN_PHYS(max_pfn))
- reserve_bootmem_node(NODE_DATA(node),
- res->start,
- resource_size(res),
- BOOTMEM_DEFAULT);
- }
-
- node_set_online(node);
- }
-}
-
-void __init setup_arch (char **cmdline_p)
-{
- struct clk *cpu_clk;
-
- init_mm.start_code = (unsigned long)_stext;
- init_mm.end_code = (unsigned long)_etext;
- init_mm.end_data = (unsigned long)_edata;
- init_mm.brk = (unsigned long)_end;
-
- /*
- * Include .init section to make allocations easier. It will
- * be removed before the resource is actually requested.
- */
- kernel_code.start = __pa(__init_begin);
- kernel_code.end = __pa(init_mm.end_code - 1);
- kernel_data.start = __pa(init_mm.end_code);
- kernel_data.end = __pa(init_mm.brk - 1);
-
- parse_tags(bootloader_tags);
-
- setup_processor();
- setup_platform();
- setup_board();
-
- cpu_clk = clk_get(NULL, "cpu");
- if (IS_ERR(cpu_clk)) {
- printk(KERN_WARNING "Warning: Unable to get CPU clock\n");
- } else {
- unsigned long cpu_hz = clk_get_rate(cpu_clk);
-
- /*
- * Well, duh, but it's probably a good idea to
- * increment the use count.
- */
- clk_enable(cpu_clk);
-
- boot_cpu_data.clk = cpu_clk;
- boot_cpu_data.loops_per_jiffy = cpu_hz * 4;
- printk("CPU: Running at %lu.%03lu MHz\n",
- ((cpu_hz + 500) / 1000) / 1000,
- ((cpu_hz + 500) / 1000) % 1000);
- }
-
- strlcpy(command_line, boot_command_line, COMMAND_LINE_SIZE);
- *cmdline_p = command_line;
- parse_early_param();
-
- setup_bootmem();
-
-#ifdef CONFIG_VT
- conswitchp = &dummy_con;
-#endif
-
- paging_init();
- resource_init();
-}
diff --git a/arch/avr32/kernel/signal.c b/arch/avr32/kernel/signal.c
deleted file mode 100644
index b5fcc49..0000000
--- a/arch/avr32/kernel/signal.c
+++ /dev/null
@@ -1,288 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * Based on linux/arch/sh/kernel/signal.c
- * Copyright (C) 1999, 2000 Niibe Yutaka & Kaz Kojima
- * Copyright (C) 1991, 1992 Linus Torvalds
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/sched.h>
-#include <linux/mm.h>
-#include <linux/errno.h>
-#include <linux/ptrace.h>
-#include <linux/unistd.h>
-#include <linux/tracehook.h>
-
-#include <linux/uaccess.h>
-#include <asm/ucontext.h>
-#include <asm/syscalls.h>
-
-struct rt_sigframe
-{
- struct siginfo info;
- struct ucontext uc;
- unsigned long retcode;
-};
-
-static int
-restore_sigcontext(struct pt_regs *regs, struct sigcontext __user *sc)
-{
- int err = 0;
-
-#define COPY(x) err |= __get_user(regs->x, &sc->x)
- COPY(sr);
- COPY(pc);
- COPY(lr);
- COPY(sp);
- COPY(r12);
- COPY(r11);
- COPY(r10);
- COPY(r9);
- COPY(r8);
- COPY(r7);
- COPY(r6);
- COPY(r5);
- COPY(r4);
- COPY(r3);
- COPY(r2);
- COPY(r1);
- COPY(r0);
-#undef COPY
-
- /*
- * Don't allow anyone to pretend they're running in supervisor
- * mode or something...
- */
- err |= !valid_user_regs(regs);
-
- return err;
-}
-
-
-asmlinkage int sys_rt_sigreturn(struct pt_regs *regs)
-{
- struct rt_sigframe __user *frame;
- sigset_t set;
-
- /* Always make any pending restarted system calls return -EINTR */
- current->restart_block.fn = do_no_restart_syscall;
-
- frame = (struct rt_sigframe __user *)regs->sp;
- pr_debug("SIG return: frame = %p\n", frame);
-
- if (!access_ok(VERIFY_READ, frame, sizeof(*frame)))
- goto badframe;
-
- if (__copy_from_user(&set, &frame->uc.uc_sigmask, sizeof(set)))
- goto badframe;
-
- set_current_blocked(&set);
-
- if (restore_sigcontext(regs, &frame->uc.uc_mcontext))
- goto badframe;
-
- if (restore_altstack(&frame->uc.uc_stack))
- goto badframe;
-
- pr_debug("Context restored: pc = %08lx, lr = %08lx, sp = %08lx\n",
- regs->pc, regs->lr, regs->sp);
-
- return regs->r12;
-
-badframe:
- force_sig(SIGSEGV, current);
- return 0;
-}
-
-static int
-setup_sigcontext(struct sigcontext __user *sc, struct pt_regs *regs)
-{
- int err = 0;
-
-#define COPY(x) err |= __put_user(regs->x, &sc->x)
- COPY(sr);
- COPY(pc);
- COPY(lr);
- COPY(sp);
- COPY(r12);
- COPY(r11);
- COPY(r10);
- COPY(r9);
- COPY(r8);
- COPY(r7);
- COPY(r6);
- COPY(r5);
- COPY(r4);
- COPY(r3);
- COPY(r2);
- COPY(r1);
- COPY(r0);
-#undef COPY
-
- return err;
-}
-
-static inline void __user *
-get_sigframe(struct ksignal *ksig, struct pt_regs *regs, int framesize)
-{
- unsigned long sp = sigsp(regs->sp, ksig);
-
- return (void __user *)((sp - framesize) & ~3);
-}
-
-static int
-setup_rt_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs)
-{
- struct rt_sigframe __user *frame;
- int err = 0;
-
- frame = get_sigframe(ksig, regs, sizeof(*frame));
- err = -EFAULT;
- if (!access_ok(VERIFY_WRITE, frame, sizeof (*frame)))
- goto out;
-
- /*
- * Set up the return code:
- *
- * mov r8, __NR_rt_sigreturn
- * scall
- *
- * Note: This will blow up since we're using a non-executable
- * stack. Better use SA_RESTORER.
- */
-#if __NR_rt_sigreturn > 127
-# error __NR_rt_sigreturn must be < 127 to fit in a short mov
-#endif
- err = __put_user(0x3008d733 | (__NR_rt_sigreturn << 20),
- &frame->retcode);
-
- err |= copy_siginfo_to_user(&frame->info, &ksig->info);
-
- /* Set up the ucontext */
- err |= __put_user(0, &frame->uc.uc_flags);
- err |= __put_user(NULL, &frame->uc.uc_link);
- err |= __save_altstack(&frame->uc.uc_stack, regs->sp);
- err |= setup_sigcontext(&frame->uc.uc_mcontext, regs);
- err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set));
-
- if (err)
- goto out;
-
- regs->r12 = ksig->sig;
- regs->r11 = (unsigned long) &frame->info;
- regs->r10 = (unsigned long) &frame->uc;
- regs->sp = (unsigned long) frame;
- if (ksig->ka.sa.sa_flags & SA_RESTORER)
- regs->lr = (unsigned long)ksig->ka.sa.sa_restorer;
- else {
- printk(KERN_NOTICE "[%s:%d] did not set SA_RESTORER\n",
- current->comm, current->pid);
- regs->lr = (unsigned long) &frame->retcode;
- }
-
- pr_debug("SIG deliver [%s:%d]: sig=%d sp=0x%lx pc=0x%lx->0x%p lr=0x%lx\n",
- current->comm, current->pid, ksig->sig, regs->sp,
- regs->pc, ksig->ka.sa.sa_handler, regs->lr);
-
- regs->pc = (unsigned long)ksig->ka.sa.sa_handler;
-
-out:
- return err;
-}
-
-static inline void setup_syscall_restart(struct pt_regs *regs)
-{
- if (regs->r12 == -ERESTART_RESTARTBLOCK)
- regs->r8 = __NR_restart_syscall;
- else
- regs->r12 = regs->r12_orig;
- regs->pc -= 2;
-}
-
-static inline void
-handle_signal(struct ksignal *ksig, struct pt_regs *regs, int syscall)
-{
- int ret;
-
- /*
- * Set up the stack frame
- */
- ret = setup_rt_frame(ksig, sigmask_to_save(), regs);
-
- /*
- * Check that the resulting registers are sane
- */
- ret |= !valid_user_regs(regs);
-
- /*
- * Block the signal if we were successful.
- */
- signal_setup_done(ret, ksig, 0);
-}
-
-/*
- * Note that 'init' is a special process: it doesn't get signals it
- * doesn't want to handle. Thus you cannot kill init even with a
- * SIGKILL even by mistake.
- */
-static void do_signal(struct pt_regs *regs, int syscall)
-{
- struct ksignal ksig;
-
- /*
- * We want the common case to go fast, which is why we may in
- * certain cases get here from kernel mode. Just return
- * without doing anything if so.
- */
- if (!user_mode(regs))
- return;
-
- get_signal(&ksig);
- if (syscall) {
- switch (regs->r12) {
- case -ERESTART_RESTARTBLOCK:
- case -ERESTARTNOHAND:
- if (ksig.sig > 0) {
- regs->r12 = -EINTR;
- break;
- }
- /* fall through */
- case -ERESTARTSYS:
- if (ksig.sig > 0 && !(ksig.ka.sa.sa_flags & SA_RESTART)) {
- regs->r12 = -EINTR;
- break;
- }
- /* fall through */
- case -ERESTARTNOINTR:
- setup_syscall_restart(regs);
- }
- }
-
- if (!ksig.sig) {
- /* No signal to deliver -- put the saved sigmask back */
- restore_saved_sigmask();
- return;
- }
-
- handle_signal(&ksig, regs, syscall);
-}
-
-asmlinkage void do_notify_resume(struct pt_regs *regs, struct thread_info *ti)
-{
- int syscall = 0;
-
- if ((sysreg_read(SR) & MODE_MASK) == MODE_SUPERVISOR)
- syscall = 1;
-
- if (ti->flags & _TIF_SIGPENDING)
- do_signal(regs, syscall);
-
- if (ti->flags & _TIF_NOTIFY_RESUME) {
- clear_thread_flag(TIF_NOTIFY_RESUME);
- tracehook_notify_resume(regs);
- }
-}
diff --git a/arch/avr32/kernel/stacktrace.c b/arch/avr32/kernel/stacktrace.c
deleted file mode 100644
index f8cc995..0000000
--- a/arch/avr32/kernel/stacktrace.c
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Stack trace management functions
- *
- * Copyright (C) 2007 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/sched.h>
-#include <linux/sched/task_stack.h>
-#include <linux/stacktrace.h>
-#include <linux/thread_info.h>
-#include <linux/module.h>
-
-register unsigned long current_frame_pointer asm("r7");
-
-struct stackframe {
- unsigned long lr;
- unsigned long fp;
-};
-
-/*
- * Save stack-backtrace addresses into a stack_trace buffer.
- */
-void save_stack_trace(struct stack_trace *trace)
-{
- unsigned long low, high;
- unsigned long fp;
- struct stackframe *frame;
- int skip = trace->skip;
-
- low = (unsigned long)task_stack_page(current);
- high = low + THREAD_SIZE;
- fp = current_frame_pointer;
-
- while (fp >= low && fp <= (high - 8)) {
- frame = (struct stackframe *)fp;
-
- if (skip) {
- skip--;
- } else {
- trace->entries[trace->nr_entries++] = frame->lr;
- if (trace->nr_entries >= trace->max_entries)
- break;
- }
-
- /*
- * The next frame must be at a higher address than the
- * current frame.
- */
- low = fp + 8;
- fp = frame->fp;
- }
-}
-EXPORT_SYMBOL_GPL(save_stack_trace);
diff --git a/arch/avr32/kernel/switch_to.S b/arch/avr32/kernel/switch_to.S
deleted file mode 100644
index a48d046..0000000
--- a/arch/avr32/kernel/switch_to.S
+++ /dev/null
@@ -1,35 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <asm/sysreg.h>
-
- .text
- .global __switch_to
- .type __switch_to, @function
-
- /* Switch thread context from "prev" to "next", returning "last"
- * r12 : prev
- * r11 : &prev->thread + 1
- * r10 : &next->thread
- */
-__switch_to:
- stm --r11, r0,r1,r2,r3,r4,r5,r6,r7,sp,lr
- mfsr r9, SYSREG_SR
- st.w --r11, r9
- ld.w r8, r10++
- /*
- * schedule() may have been called from a mode with a different
- * set of registers. Make sure we don't lose anything here.
- */
- pushm r10,r12
- mtsr SYSREG_SR, r8
- frs /* flush the return stack */
- sub pc, -2 /* flush the pipeline */
- popm r10,r12
- ldm r10++, r0,r1,r2,r3,r4,r5,r6,r7,sp,pc
- .size __switch_to, . - __switch_to
diff --git a/arch/avr32/kernel/syscall-stubs.S b/arch/avr32/kernel/syscall-stubs.S
deleted file mode 100644
index cb25653..0000000
--- a/arch/avr32/kernel/syscall-stubs.S
+++ /dev/null
@@ -1,153 +0,0 @@
-/*
- * Copyright (C) 2005-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-/*
- * Stubs for syscalls that require access to pt_regs or that take more
- * than five parameters.
- */
-
-#define ARG6 r3
-
- .text
- .global __sys_rt_sigsuspend
- .type __sys_rt_sigsuspend,@function
-__sys_rt_sigsuspend:
- mov r10, sp
- rjmp sys_rt_sigsuspend
-
- .global __sys_rt_sigreturn
- .type __sys_rt_sigreturn,@function
-__sys_rt_sigreturn:
- mov r12, sp
- rjmp sys_rt_sigreturn
-
- .global __sys_mmap2
- .type __sys_mmap2,@function
-__sys_mmap2:
- pushm lr
- st.w --sp, ARG6
- call sys_mmap_pgoff
- sub sp, -4
- popm pc
-
- .global __sys_sendto
- .type __sys_sendto,@function
-__sys_sendto:
- pushm lr
- st.w --sp, ARG6
- call sys_sendto
- sub sp, -4
- popm pc
-
- .global __sys_recvfrom
- .type __sys_recvfrom,@function
-__sys_recvfrom:
- pushm lr
- st.w --sp, ARG6
- call sys_recvfrom
- sub sp, -4
- popm pc
-
- .global __sys_pselect6
- .type __sys_pselect6,@function
-__sys_pselect6:
- pushm lr
- st.w --sp, ARG6
- call sys_pselect6
- sub sp, -4
- popm pc
-
- .global __sys_splice
- .type __sys_splice,@function
-__sys_splice:
- pushm lr
- st.w --sp, ARG6
- call sys_splice
- sub sp, -4
- popm pc
-
- .global __sys_epoll_pwait
- .type __sys_epoll_pwait,@function
-__sys_epoll_pwait:
- pushm lr
- st.w --sp, ARG6
- call sys_epoll_pwait
- sub sp, -4
- popm pc
-
- .global __sys_sync_file_range
- .type __sys_sync_file_range,@function
-__sys_sync_file_range:
- pushm lr
- st.w --sp, ARG6
- call sys_sync_file_range
- sub sp, -4
- popm pc
-
- .global __sys_fallocate
- .type __sys_fallocate,@function
-__sys_fallocate:
- pushm lr
- st.w --sp, ARG6
- call sys_fallocate
- sub sp, -4
- popm pc
-
- .global __sys_fanotify_mark
- .type __sys_fanotify_mark,@function
-__sys_fanotify_mark:
- pushm lr
- st.w --sp, ARG6
- call sys_fanotify_mark
- sub sp, -4
- popm pc
-
- .global __sys_process_vm_readv
- .type __sys_process_vm_readv,@function
-__sys_process_vm_readv:
- pushm lr
- st.w --sp, ARG6
- call sys_process_vm_readv
- sub sp, -4
- popm pc
-
- .global __sys_process_vm_writev
- .type __sys_process_vm_writev,@function
-__sys_process_vm_writev:
- pushm lr
- st.w --sp, ARG6
- call sys_process_vm_writev
- sub sp, -4
- popm pc
-
- .global __sys_copy_file_range
- .type __sys_copy_file_range,@function
-__sys_copy_file_range:
- pushm lr
- st.w --sp, ARG6
- call sys_copy_file_range
- sub sp, -4
- popm pc
-
- .global __sys_preadv2
- .type __sys_preadv2,@function
-__sys_preadv2:
- pushm lr
- st.w --sp, ARG6
- call sys_preadv2
- sub sp, -4
- popm pc
-
- .global __sys_pwritev2
- .type __sys_pwritev2,@function
-__sys_pwritev2:
- pushm lr
- st.w --sp, ARG6
- call sys_pwritev2
- sub sp, -4
- popm pc
diff --git a/arch/avr32/kernel/syscall_table.S b/arch/avr32/kernel/syscall_table.S
deleted file mode 100644
index 774ce57..0000000
--- a/arch/avr32/kernel/syscall_table.S
+++ /dev/null
@@ -1,347 +0,0 @@
-/*
- * AVR32 system call table
- *
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
- .section .rodata,"a",@progbits
- .type sys_call_table,@object
- .global sys_call_table
- .align 2
-sys_call_table:
- .long sys_restart_syscall
- .long sys_exit
- .long sys_fork
- .long sys_read
- .long sys_write
- .long sys_open
- .long sys_close
- .long sys_umask
- .long sys_creat
- .long sys_link
- .long sys_unlink /* 10 */
- .long sys_execve
- .long sys_chdir
- .long sys_time
- .long sys_mknod
- .long sys_chmod
- .long sys_chown
- .long sys_lchown
- .long sys_lseek
- .long sys_llseek
- .long sys_getpid /* 20 */
- .long sys_mount
- .long sys_umount
- .long sys_setuid
- .long sys_getuid
- .long sys_stime
- .long sys_ptrace
- .long sys_alarm
- .long sys_pause
- .long sys_utime
- .long sys_newstat /* 30 */
- .long sys_newfstat
- .long sys_newlstat
- .long sys_access
- .long sys_chroot
- .long sys_sync
- .long sys_fsync
- .long sys_kill
- .long sys_rename
- .long sys_mkdir
- .long sys_rmdir /* 40 */
- .long sys_dup
- .long sys_pipe
- .long sys_times
- .long sys_clone
- .long sys_brk
- .long sys_setgid
- .long sys_getgid
- .long sys_getcwd
- .long sys_geteuid
- .long sys_getegid /* 50 */
- .long sys_acct
- .long sys_setfsuid
- .long sys_setfsgid
- .long sys_ioctl
- .long sys_fcntl
- .long sys_setpgid
- .long sys_mremap
- .long sys_setresuid
- .long sys_getresuid
- .long sys_setreuid /* 60 */
- .long sys_setregid
- .long sys_ustat
- .long sys_dup2
- .long sys_getppid
- .long sys_getpgrp
- .long sys_setsid
- .long sys_rt_sigaction
- .long __sys_rt_sigreturn
- .long sys_rt_sigprocmask
- .long sys_rt_sigpending /* 70 */
- .long sys_rt_sigtimedwait
- .long sys_rt_sigqueueinfo
- .long __sys_rt_sigsuspend
- .long sys_sethostname
- .long sys_setrlimit
- .long sys_getrlimit
- .long sys_getrusage
- .long sys_gettimeofday
- .long sys_settimeofday
- .long sys_getgroups /* 80 */
- .long sys_setgroups
- .long sys_select
- .long sys_symlink
- .long sys_fchdir
- .long sys_readlink
- .long sys_pread64
- .long sys_pwrite64
- .long sys_swapon
- .long sys_reboot
- .long __sys_mmap2 /* 90 */
- .long sys_munmap
- .long sys_truncate
- .long sys_ftruncate
- .long sys_fchmod
- .long sys_fchown
- .long sys_getpriority
- .long sys_setpriority
- .long sys_wait4
- .long sys_statfs
- .long sys_fstatfs /* 100 */
- .long sys_vhangup
- .long sys_sigaltstack
- .long sys_syslog
- .long sys_setitimer
- .long sys_getitimer
- .long sys_swapoff
- .long sys_sysinfo
- .long sys_ni_syscall /* was sys_ipc briefly */
- .long sys_sendfile
- .long sys_setdomainname /* 110 */
- .long sys_newuname
- .long sys_adjtimex
- .long sys_mprotect
- .long sys_vfork
- .long sys_init_module
- .long sys_delete_module
- .long sys_quotactl
- .long sys_getpgid
- .long sys_bdflush
- .long sys_sysfs /* 120 */
- .long sys_personality
- .long sys_ni_syscall /* reserved for afs_syscall */
- .long sys_getdents
- .long sys_flock
- .long sys_msync
- .long sys_readv
- .long sys_writev
- .long sys_getsid
- .long sys_fdatasync
- .long sys_sysctl /* 130 */
- .long sys_mlock
- .long sys_munlock
- .long sys_mlockall
- .long sys_munlockall
- .long sys_sched_setparam
- .long sys_sched_getparam
- .long sys_sched_setscheduler
- .long sys_sched_getscheduler
- .long sys_sched_yield
- .long sys_sched_get_priority_max /* 140 */
- .long sys_sched_get_priority_min
- .long sys_sched_rr_get_interval
- .long sys_nanosleep
- .long sys_poll
- .long sys_ni_syscall /* 145 was nfsservctl */
- .long sys_setresgid
- .long sys_getresgid
- .long sys_prctl
- .long sys_socket
- .long sys_bind /* 150 */
- .long sys_connect
- .long sys_listen
- .long sys_accept
- .long sys_getsockname
- .long sys_getpeername
- .long sys_socketpair
- .long sys_send
- .long sys_recv
- .long __sys_sendto
- .long __sys_recvfrom /* 160 */
- .long sys_shutdown
- .long sys_setsockopt
- .long sys_getsockopt
- .long sys_sendmsg
- .long sys_recvmsg
- .long sys_truncate64
- .long sys_ftruncate64
- .long sys_stat64
- .long sys_lstat64
- .long sys_fstat64 /* 170 */
- .long sys_pivot_root
- .long sys_mincore
- .long sys_madvise
- .long sys_getdents64
- .long sys_fcntl64
- .long sys_gettid
- .long sys_readahead
- .long sys_setxattr
- .long sys_lsetxattr
- .long sys_fsetxattr /* 180 */
- .long sys_getxattr
- .long sys_lgetxattr
- .long sys_fgetxattr
- .long sys_listxattr
- .long sys_llistxattr
- .long sys_flistxattr
- .long sys_removexattr
- .long sys_lremovexattr
- .long sys_fremovexattr
- .long sys_tkill /* 190 */
- .long sys_sendfile64
- .long sys_futex
- .long sys_sched_setaffinity
- .long sys_sched_getaffinity
- .long sys_capget
- .long sys_capset
- .long sys_io_setup
- .long sys_io_destroy
- .long sys_io_getevents
- .long sys_io_submit /* 200 */
- .long sys_io_cancel
- .long sys_fadvise64
- .long sys_exit_group
- .long sys_lookup_dcookie
- .long sys_epoll_create
- .long sys_epoll_ctl
- .long sys_epoll_wait
- .long sys_remap_file_pages
- .long sys_set_tid_address
- .long sys_timer_create /* 210 */
- .long sys_timer_settime
- .long sys_timer_gettime
- .long sys_timer_getoverrun
- .long sys_timer_delete
- .long sys_clock_settime
- .long sys_clock_gettime
- .long sys_clock_getres
- .long sys_clock_nanosleep
- .long sys_statfs64
- .long sys_fstatfs64 /* 220 */
- .long sys_tgkill
- .long sys_ni_syscall /* reserved for TUX */
- .long sys_utimes
- .long sys_fadvise64_64
- .long sys_cacheflush
- .long sys_ni_syscall /* sys_vserver */
- .long sys_mq_open
- .long sys_mq_unlink
- .long sys_mq_timedsend
- .long sys_mq_timedreceive /* 230 */
- .long sys_mq_notify
- .long sys_mq_getsetattr
- .long sys_kexec_load
- .long sys_waitid
- .long sys_add_key
- .long sys_request_key
- .long sys_keyctl
- .long sys_ioprio_set
- .long sys_ioprio_get
- .long sys_inotify_init /* 240 */
- .long sys_inotify_add_watch
- .long sys_inotify_rm_watch
- .long sys_openat
- .long sys_mkdirat
- .long sys_mknodat
- .long sys_fchownat
- .long sys_futimesat
- .long sys_fstatat64
- .long sys_unlinkat
- .long sys_renameat /* 250 */
- .long sys_linkat
- .long sys_symlinkat
- .long sys_readlinkat
- .long sys_fchmodat
- .long sys_faccessat
- .long __sys_pselect6
- .long sys_ppoll
- .long sys_unshare
- .long sys_set_robust_list
- .long sys_get_robust_list /* 260 */
- .long __sys_splice
- .long __sys_sync_file_range
- .long sys_tee
- .long sys_vmsplice
- .long __sys_epoll_pwait
- .long sys_msgget
- .long sys_msgsnd
- .long sys_msgrcv
- .long sys_msgctl
- .long sys_semget /* 270 */
- .long sys_semop
- .long sys_semctl
- .long sys_semtimedop
- .long sys_shmat
- .long sys_shmget
- .long sys_shmdt
- .long sys_shmctl
- .long sys_utimensat
- .long sys_signalfd
- .long sys_ni_syscall /* 280, was sys_timerfd */
- .long sys_eventfd
- .long sys_ni_syscall /* 282, was half-implemented recvmmsg */
- .long sys_setns
- .long sys_pread64
- .long sys_pwrite64
- .long sys_timerfd_create
- .long __sys_fallocate
- .long sys_timerfd_settime
- .long sys_timerfd_gettime
- .long sys_signalfd4 /* 290 */
- .long sys_eventfd2
- .long sys_epoll_create1
- .long sys_dup3
- .long sys_pipe2
- .long sys_inotify_init1
- .long sys_preadv
- .long sys_pwritev
- .long sys_rt_tgsigqueueinfo
- .long sys_perf_event_open
- .long sys_recvmmsg /* 300 */
- .long sys_fanotify_init
- .long __sys_fanotify_mark
- .long sys_prlimit64
- .long sys_name_to_handle_at
- .long sys_open_by_handle_at
- .long sys_clock_adjtime
- .long sys_syncfs
- .long sys_sendmmsg
- .long __sys_process_vm_readv
- .long __sys_process_vm_writev /* 310 */
- .long sys_kcmp
- .long sys_finit_module
- .long sys_sched_setattr
- .long sys_sched_getattr
- .long sys_renameat2
- .long sys_seccomp
- .long sys_getrandom
- .long sys_memfd_create
- .long sys_bpf
- .long sys_execveat /* 320 */
- .long sys_accept4
- .long sys_userfaultfd
- .long sys_membarrier
- .long sys_mlock2
- .long __sys_copy_file_range
- .long __sys_preadv2
- .long __sys_pwritev2
- .long sys_pkey_mprotect
- .long sys_pkey_alloc
- .long sys_pkey_free /* 330 */
- .long sys_ni_syscall /* r8 is saturated at nr_syscalls */
diff --git a/arch/avr32/kernel/time.c b/arch/avr32/kernel/time.c
deleted file mode 100644
index 4d9b696..0000000
--- a/arch/avr32/kernel/time.c
+++ /dev/null
@@ -1,161 +0,0 @@
-/*
- * Copyright (C) 2004-2007 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/clk.h>
-#include <linux/clockchips.h>
-#include <linux/init.h>
-#include <linux/interrupt.h>
-#include <linux/irq.h>
-#include <linux/kernel.h>
-#include <linux/time.h>
-#include <linux/cpu.h>
-
-#include <asm/sysreg.h>
-
-#include <mach/pm.h>
-
-static bool disable_cpu_idle_poll;
-
-static u64 read_cycle_count(struct clocksource *cs)
-{
- return (u64)sysreg_read(COUNT);
-}
-
-/*
- * The architectural cycle count registers are a fine clocksource unless
- * the system idle loop use sleep states like "idle": the CPU cycles
- * measured by COUNT (and COMPARE) don't happen during sleep states.
- * Their duration also changes if cpufreq changes the CPU clock rate.
- * So we rate the clocksource using COUNT as very low quality.
- */
-static struct clocksource counter = {
- .name = "avr32_counter",
- .rating = 50,
- .read = read_cycle_count,
- .mask = CLOCKSOURCE_MASK(32),
- .flags = CLOCK_SOURCE_IS_CONTINUOUS,
-};
-
-static irqreturn_t timer_interrupt(int irq, void *dev_id)
-{
- struct clock_event_device *evdev = dev_id;
-
- if (unlikely(!(intc_get_pending(0) & 1)))
- return IRQ_NONE;
-
- /*
- * Disable the interrupt until the clockevent subsystem
- * reprograms it.
- */
- sysreg_write(COMPARE, 0);
-
- evdev->event_handler(evdev);
- return IRQ_HANDLED;
-}
-
-static struct irqaction timer_irqaction = {
- .handler = timer_interrupt,
- /* Oprofile uses the same irq as the timer, so allow it to be shared */
- .flags = IRQF_TIMER | IRQF_SHARED,
- .name = "avr32_comparator",
-};
-
-static int comparator_next_event(unsigned long delta,
- struct clock_event_device *evdev)
-{
- unsigned long flags;
-
- raw_local_irq_save(flags);
-
- /* The time to read COUNT then update COMPARE must be less
- * than the min_delta_ns value for this clockevent source.
- */
- sysreg_write(COMPARE, (sysreg_read(COUNT) + delta) ? : 1);
-
- raw_local_irq_restore(flags);
-
- return 0;
-}
-
-static int comparator_shutdown(struct clock_event_device *evdev)
-{
- pr_debug("%s: %s\n", __func__, evdev->name);
- sysreg_write(COMPARE, 0);
-
- if (disable_cpu_idle_poll) {
- disable_cpu_idle_poll = false;
- /*
- * Only disable idle poll if we have forced that
- * in a previous call.
- */
- cpu_idle_poll_ctrl(false);
- }
- return 0;
-}
-
-static int comparator_set_oneshot(struct clock_event_device *evdev)
-{
- pr_debug("%s: %s\n", __func__, evdev->name);
-
- disable_cpu_idle_poll = true;
- /*
- * If we're using the COUNT and COMPARE registers we
- * need to force idle poll.
- */
- cpu_idle_poll_ctrl(true);
-
- return 0;
-}
-
-static struct clock_event_device comparator = {
- .name = "avr32_comparator",
- .features = CLOCK_EVT_FEAT_ONESHOT,
- .shift = 16,
- .rating = 50,
- .set_next_event = comparator_next_event,
- .set_state_shutdown = comparator_shutdown,
- .set_state_oneshot = comparator_set_oneshot,
- .tick_resume = comparator_set_oneshot,
-};
-
-void read_persistent_clock(struct timespec *ts)
-{
- ts->tv_sec = mktime(2007, 1, 1, 0, 0, 0);
- ts->tv_nsec = 0;
-}
-
-void __init time_init(void)
-{
- unsigned long counter_hz;
- int ret;
-
- /* figure rate for counter */
- counter_hz = clk_get_rate(boot_cpu_data.clk);
- ret = clocksource_register_hz(&counter, counter_hz);
- if (ret)
- pr_debug("timer: could not register clocksource: %d\n", ret);
-
- /* setup COMPARE clockevent */
- comparator.mult = div_sc(counter_hz, NSEC_PER_SEC, comparator.shift);
- comparator.max_delta_ns = clockevent_delta2ns((u32)~0, &comparator);
- comparator.min_delta_ns = clockevent_delta2ns(50, &comparator) + 1;
- comparator.cpumask = cpumask_of(0);
-
- sysreg_write(COMPARE, 0);
- timer_irqaction.dev_id = &comparator;
-
- ret = setup_irq(0, &timer_irqaction);
- if (ret)
- pr_debug("timer: could not request IRQ 0: %d\n", ret);
- else {
- clockevents_register_device(&comparator);
-
- pr_info("%s: irq 0, %lu.%03lu MHz\n", comparator.name,
- ((counter_hz + 500) / 1000) / 1000,
- ((counter_hz + 500) / 1000) % 1000);
- }
-}
diff --git a/arch/avr32/kernel/traps.c b/arch/avr32/kernel/traps.c
deleted file mode 100644
index 50b5413..0000000
--- a/arch/avr32/kernel/traps.c
+++ /dev/null
@@ -1,262 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/bug.h>
-#include <linux/hardirq.h>
-#include <linux/init.h>
-#include <linux/kallsyms.h>
-#include <linux/kdebug.h>
-#include <linux/extable.h>
-#include <linux/module.h> /* print_modules */
-#include <linux/notifier.h>
-#include <linux/sched/signal.h>
-#include <linux/uaccess.h>
-
-#include <asm/addrspace.h>
-#include <asm/mmu_context.h>
-#include <asm/ocd.h>
-#include <asm/sysreg.h>
-#include <asm/traps.h>
-
-static DEFINE_SPINLOCK(die_lock);
-
-void die(const char *str, struct pt_regs *regs, long err)
-{
- static int die_counter;
-
- console_verbose();
- spin_lock_irq(&die_lock);
- bust_spinlocks(1);
-
- printk(KERN_ALERT "Oops: %s, sig: %ld [#%d]\n",
- str, err, ++die_counter);
-
- printk(KERN_EMERG);
-
-#ifdef CONFIG_PREEMPT
- printk(KERN_CONT "PREEMPT ");
-#endif
-#ifdef CONFIG_FRAME_POINTER
- printk(KERN_CONT "FRAME_POINTER ");
-#endif
- if (current_cpu_data.features & AVR32_FEATURE_OCD) {
- unsigned long did = ocd_read(DID);
- printk(KERN_CONT "chip: 0x%03lx:0x%04lx rev %lu\n",
- (did >> 1) & 0x7ff,
- (did >> 12) & 0x7fff,
- (did >> 28) & 0xf);
- } else {
- printk(KERN_CONT "cpu: arch %u r%u / core %u r%u\n",
- current_cpu_data.arch_type,
- current_cpu_data.arch_revision,
- current_cpu_data.cpu_type,
- current_cpu_data.cpu_revision);
- }
-
- print_modules();
- show_regs_log_lvl(regs, KERN_EMERG);
- show_stack_log_lvl(current, regs->sp, regs, KERN_EMERG);
- bust_spinlocks(0);
- add_taint(TAINT_DIE, LOCKDEP_NOW_UNRELIABLE);
- spin_unlock_irq(&die_lock);
-
- if (in_interrupt())
- panic("Fatal exception in interrupt");
-
- if (panic_on_oops)
- panic("Fatal exception");
-
- do_exit(err);
-}
-
-void _exception(long signr, struct pt_regs *regs, int code,
- unsigned long addr)
-{
- siginfo_t info;
-
- if (!user_mode(regs)) {
- const struct exception_table_entry *fixup;
-
- /* Are we prepared to handle this kernel fault? */
- fixup = search_exception_tables(regs->pc);
- if (fixup) {
- regs->pc = fixup->fixup;
- return;
- }
- die("Unhandled exception in kernel mode", regs, signr);
- }
-
- memset(&info, 0, sizeof(info));
- info.si_signo = signr;
- info.si_code = code;
- info.si_addr = (void __user *)addr;
- force_sig_info(signr, &info, current);
-}
-
-asmlinkage void do_nmi(unsigned long ecr, struct pt_regs *regs)
-{
- int ret;
-
- nmi_enter();
-
- ret = notify_die(DIE_NMI, "NMI", regs, 0, ecr, SIGINT);
- switch (ret) {
- case NOTIFY_OK:
- case NOTIFY_STOP:
- break;
- case NOTIFY_BAD:
- die("Fatal Non-Maskable Interrupt", regs, SIGINT);
- default:
- printk(KERN_ALERT "Got NMI, but nobody cared. Disabling...\n");
- nmi_disable();
- break;
- }
- nmi_exit();
-}
-
-asmlinkage void do_critical_exception(unsigned long ecr, struct pt_regs *regs)
-{
- die("Critical exception", regs, SIGKILL);
-}
-
-asmlinkage void do_address_exception(unsigned long ecr, struct pt_regs *regs)
-{
- _exception(SIGBUS, regs, BUS_ADRALN, regs->pc);
-}
-
-/* This way of handling undefined instructions is stolen from ARM */
-static LIST_HEAD(undef_hook);
-static DEFINE_SPINLOCK(undef_lock);
-
-void register_undef_hook(struct undef_hook *hook)
-{
- spin_lock_irq(&undef_lock);
- list_add(&hook->node, &undef_hook);
- spin_unlock_irq(&undef_lock);
-}
-
-void unregister_undef_hook(struct undef_hook *hook)
-{
- spin_lock_irq(&undef_lock);
- list_del(&hook->node);
- spin_unlock_irq(&undef_lock);
-}
-
-static int do_cop_absent(u32 insn)
-{
- int cop_nr;
- u32 cpucr;
-
- if ((insn & 0xfdf00000) == 0xf1900000)
- /* LDC0 */
- cop_nr = 0;
- else
- cop_nr = (insn >> 13) & 0x7;
-
- /* Try enabling the coprocessor */
- cpucr = sysreg_read(CPUCR);
- cpucr |= (1 << (24 + cop_nr));
- sysreg_write(CPUCR, cpucr);
-
- cpucr = sysreg_read(CPUCR);
- if (!(cpucr & (1 << (24 + cop_nr))))
- return -ENODEV;
-
- return 0;
-}
-
-#ifdef CONFIG_BUG
-int is_valid_bugaddr(unsigned long pc)
-{
- unsigned short opcode;
-
- if (pc < PAGE_OFFSET)
- return 0;
- if (probe_kernel_address((u16 *)pc, opcode))
- return 0;
-
- return opcode == AVR32_BUG_OPCODE;
-}
-#endif
-
-asmlinkage void do_illegal_opcode(unsigned long ecr, struct pt_regs *regs)
-{
- u32 insn;
- struct undef_hook *hook;
- void __user *pc;
- long code;
-
-#ifdef CONFIG_BUG
- if (!user_mode(regs) && (ecr == ECR_ILLEGAL_OPCODE)) {
- enum bug_trap_type type;
-
- type = report_bug(regs->pc, regs);
- switch (type) {
- case BUG_TRAP_TYPE_NONE:
- break;
- case BUG_TRAP_TYPE_WARN:
- regs->pc += 2;
- return;
- case BUG_TRAP_TYPE_BUG:
- die("Kernel BUG", regs, SIGKILL);
- }
- }
-#endif
-
- local_irq_enable();
-
- if (user_mode(regs)) {
- pc = (void __user *)instruction_pointer(regs);
- if (get_user(insn, (u32 __user *)pc))
- goto invalid_area;
-
- if (ecr == ECR_COPROC_ABSENT && !do_cop_absent(insn))
- return;
-
- spin_lock_irq(&undef_lock);
- list_for_each_entry(hook, &undef_hook, node) {
- if ((insn & hook->insn_mask) == hook->insn_val) {
- if (hook->fn(regs, insn) == 0) {
- spin_unlock_irq(&undef_lock);
- return;
- }
- }
- }
- spin_unlock_irq(&undef_lock);
- }
-
- switch (ecr) {
- case ECR_PRIVILEGE_VIOLATION:
- code = ILL_PRVOPC;
- break;
- case ECR_COPROC_ABSENT:
- code = ILL_COPROC;
- break;
- default:
- code = ILL_ILLOPC;
- break;
- }
-
- _exception(SIGILL, regs, code, regs->pc);
- return;
-
-invalid_area:
- _exception(SIGSEGV, regs, SEGV_MAPERR, regs->pc);
-}
-
-asmlinkage void do_fpe(unsigned long ecr, struct pt_regs *regs)
-{
- /* We have no FPU yet */
- _exception(SIGILL, regs, ILL_COPROC, regs->pc);
-}
-
-
-void __init trap_init(void)
-{
-
-}
diff --git a/arch/avr32/kernel/vmlinux.lds.S b/arch/avr32/kernel/vmlinux.lds.S
deleted file mode 100644
index 17f2730..0000000
--- a/arch/avr32/kernel/vmlinux.lds.S
+++ /dev/null
@@ -1,89 +0,0 @@
-/*
- * AVR32 linker script for the Linux kernel
- *
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#define LOAD_OFFSET 0x00000000
-#include <asm-generic/vmlinux.lds.h>
-#include <asm/cache.h>
-#include <asm/thread_info.h>
-
-OUTPUT_FORMAT("elf32-avr32", "elf32-avr32", "elf32-avr32")
-OUTPUT_ARCH(avr32)
-ENTRY(_start)
-
-/* Big endian */
-jiffies = jiffies_64 + 4;
-
-SECTIONS
-{
- . = CONFIG_ENTRY_ADDRESS;
- .init : AT(ADDR(.init) - LOAD_OFFSET) {
- _text = .;
- __init_begin = .;
- _sinittext = .;
- *(.text.reset)
- INIT_TEXT
- /*
- * .exit.text is discarded at runtime, not
- * link time, to deal with references from
- * __bug_table
- */
- EXIT_TEXT
- _einittext = .;
- . = ALIGN(4);
- __tagtable_begin = .;
- *(.taglist.init)
- __tagtable_end = .;
- }
- INIT_DATA_SECTION(16)
- . = ALIGN(PAGE_SIZE);
- __init_end = .;
-
- .text : AT(ADDR(.text) - LOAD_OFFSET) {
- _evba = .;
- _stext = .;
- *(.ex.text)
- *(.irq.text)
- KPROBES_TEXT
- TEXT_TEXT
- SCHED_TEXT
- CPUIDLE_TEXT
- LOCK_TEXT
- *(.fixup)
- *(.gnu.warning)
- _etext = .;
- } = 0xd703d703
-
- EXCEPTION_TABLE(4)
- RODATA
-
- .data : AT(ADDR(.data) - LOAD_OFFSET) {
- _data = .;
- _sdata = .;
-
- INIT_TASK_DATA(THREAD_SIZE)
- PAGE_ALIGNED_DATA(PAGE_SIZE);
- CACHELINE_ALIGNED_DATA(L1_CACHE_BYTES)
- *(.data.rel*)
- DATA_DATA
- CONSTRUCTORS
-
- _edata = .;
- }
-
- BSS_SECTION(0, 8, 8)
- _end = .;
-
- DWARF_DEBUG
-
- /* When something in the kernel is NOT compiled as a module, the module
- * cleanup code and data are put into these segments. Both can then be
- * thrown away, as cleanup code is never called unless it's a module.
- */
- DISCARDS
-}
diff --git a/arch/avr32/lib/Makefile b/arch/avr32/lib/Makefile
deleted file mode 100644
index 084d95b..0000000
--- a/arch/avr32/lib/Makefile
+++ /dev/null
@@ -1,11 +0,0 @@
-#
-# Makefile for AVR32-specific library files
-#
-
-lib-y := copy_user.o clear_user.o
-lib-y += strncpy_from_user.o strnlen_user.o
-lib-y += delay.o memset.o memcpy.o findbit.o
-lib-y += csum_partial.o csum_partial_copy_generic.o
-lib-y += io-readsw.o io-readsl.o io-writesw.o io-writesl.o
-lib-y += io-readsb.o io-writesb.o
-lib-y += __avr32_lsl64.o __avr32_lsr64.o __avr32_asr64.o
diff --git a/arch/avr32/lib/__avr32_asr64.S b/arch/avr32/lib/__avr32_asr64.S
deleted file mode 100644
index 368b6bc..0000000
--- a/arch/avr32/lib/__avr32_asr64.S
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Copyright (C) 2005-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
- /*
- * DWtype __avr32_asr64(DWtype u, word_type b)
- */
- .text
- .global __avr32_asr64
- .type __avr32_asr64,@function
-__avr32_asr64:
- cp.w r12, 0
- reteq r12
-
- rsub r9, r12, 32
- brle 1f
-
- lsl r8, r11, r9
- lsr r10, r10, r12
- asr r11, r11, r12
- or r10, r8
- retal r12
-
-1: neg r9
- asr r10, r11, r9
- asr r11, 31
- retal r12
diff --git a/arch/avr32/lib/__avr32_lsl64.S b/arch/avr32/lib/__avr32_lsl64.S
deleted file mode 100644
index f1dbc2b..0000000
--- a/arch/avr32/lib/__avr32_lsl64.S
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Copyright (C) 2005-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
- /*
- * DWtype __avr32_lsl64(DWtype u, word_type b)
- */
- .text
- .global __avr32_lsl64
- .type __avr32_lsl64,@function
-__avr32_lsl64:
- cp.w r12, 0
- reteq r12
-
- rsub r9, r12, 32
- brle 1f
-
- lsr r8, r10, r9
- lsl r10, r10, r12
- lsl r11, r11, r12
- or r11, r8
- retal r12
-
-1: neg r9
- lsl r11, r10, r9
- mov r10, 0
- retal r12
diff --git a/arch/avr32/lib/__avr32_lsr64.S b/arch/avr32/lib/__avr32_lsr64.S
deleted file mode 100644
index e65bb7f..0000000
--- a/arch/avr32/lib/__avr32_lsr64.S
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Copyright (C) 2005-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
- /*
- * DWtype __avr32_lsr64(DWtype u, word_type b)
- */
- .text
- .global __avr32_lsr64
- .type __avr32_lsr64,@function
-__avr32_lsr64:
- cp.w r12, 0
- reteq r12
-
- rsub r9, r12, 32
- brle 1f
-
- lsl r8, r11, r9
- lsr r11, r11, r12
- lsr r10, r10, r12
- or r10, r8
- retal r12
-
-1: neg r9
- lsr r10, r11, r9
- mov r11, 0
- retal r12
diff --git a/arch/avr32/lib/clear_user.S b/arch/avr32/lib/clear_user.S
deleted file mode 100644
index d8991b6..0000000
--- a/arch/avr32/lib/clear_user.S
+++ /dev/null
@@ -1,76 +0,0 @@
-/*
- * Copyright 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <asm/page.h>
-#include <asm/thread_info.h>
-#include <asm/asm.h>
-
- .text
- .align 1
- .global clear_user
- .type clear_user, "function"
-clear_user:
- branch_if_kernel r8, __clear_user
- ret_if_privileged r8, r12, r11, r11
-
- .global __clear_user
- .type __clear_user, "function"
-__clear_user:
- mov r9, r12
- mov r8, 0
- andl r9, 3, COH
- brne 5f
-
-1: sub r11, 4
- brlt 2f
-
-10: st.w r12++, r8
- sub r11, 4
- brge 10b
-
-2: sub r11, -4
- reteq 0
-
- /* Unaligned count or address */
- bld r11, 1
- brcc 12f
-11: st.h r12++, r8
- sub r11, 2
- reteq 0
-12: st.b r12++, r8
- retal 0
-
- /* Unaligned address */
-5: cp.w r11, 4
- brlt 2b
-
- lsl r9, 2
- add pc, pc, r9
-13: st.b r12++, r8
- sub r11, 1
-14: st.b r12++, r8
- sub r11, 1
-15: st.b r12++, r8
- sub r11, 1
- rjmp 1b
-
- .size clear_user, . - clear_user
- .size __clear_user, . - __clear_user
-
- .section .fixup, "ax"
- .align 1
-18: sub r11, -4
-19: retal r11
-
- .section __ex_table, "a"
- .align 2
- .long 10b, 18b
- .long 11b, 19b
- .long 12b, 19b
- .long 13b, 19b
- .long 14b, 19b
- .long 15b, 19b
diff --git a/arch/avr32/lib/copy_user.S b/arch/avr32/lib/copy_user.S
deleted file mode 100644
index 0753734..0000000
--- a/arch/avr32/lib/copy_user.S
+++ /dev/null
@@ -1,119 +0,0 @@
-/*
- * Copy to/from userspace with optional address space checking.
- *
- * Copyright 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <asm/page.h>
-#include <asm/thread_info.h>
-#include <asm/asm.h>
-
- /*
- * __kernel_size_t
- * __copy_user(void *to, const void *from, __kernel_size_t n)
- *
- * Returns the number of bytes not copied. Might be off by
- * max 3 bytes if we get a fault in the main loop.
- *
- * The address-space checking functions simply fall through to
- * the non-checking version.
- */
- .text
- .align 1
- .global ___copy_from_user
- .type ___copy_from_user, @function
-___copy_from_user:
- branch_if_kernel r8, __copy_user
- ret_if_privileged r8, r11, r10, r10
- rjmp __copy_user
- .size ___copy_from_user, . - ___copy_from_user
-
- .global copy_to_user
- .type copy_to_user, @function
-copy_to_user:
- branch_if_kernel r8, __copy_user
- ret_if_privileged r8, r12, r10, r10
- .size copy_to_user, . - copy_to_user
-
- .global __copy_user
- .type __copy_user, @function
-__copy_user:
- mov r9, r11
- andl r9, 3, COH
- brne 6f
-
- /* At this point, from is word-aligned */
-1: sub r10, 4
- brlt 3f
-
-2:
-10: ld.w r8, r11++
-11: st.w r12++, r8
- sub r10, 4
- brge 2b
-
-3: sub r10, -4
- reteq 0
-
- /*
- * Handle unaligned count. Need to be careful with r10 here so
- * that we return the correct value even if we get a fault
- */
-4:
-20: ld.ub r8, r11++
-21: st.b r12++, r8
- sub r10, 1
- reteq 0
-22: ld.ub r8, r11++
-23: st.b r12++, r8
- sub r10, 1
- reteq 0
-24: ld.ub r8, r11++
-25: st.b r12++, r8
- retal 0
-
- /* Handle unaligned from-pointer */
-6: cp.w r10, 4
- brlt 4b
- rsub r9, r9, 4
-
-30: ld.ub r8, r11++
-31: st.b r12++, r8
- sub r10, 1
- sub r9, 1
- breq 1b
-32: ld.ub r8, r11++
-33: st.b r12++, r8
- sub r10, 1
- sub r9, 1
- breq 1b
-34: ld.ub r8, r11++
-35: st.b r12++, r8
- sub r10, 1
- rjmp 1b
- .size __copy_user, . - __copy_user
-
- .section .fixup,"ax"
- .align 1
-19: sub r10, -4
-29: retal r10
-
- .section __ex_table,"a"
- .align 2
- .long 10b, 19b
- .long 11b, 19b
- .long 20b, 29b
- .long 21b, 29b
- .long 22b, 29b
- .long 23b, 29b
- .long 24b, 29b
- .long 25b, 29b
- .long 30b, 29b
- .long 31b, 29b
- .long 32b, 29b
- .long 33b, 29b
- .long 34b, 29b
- .long 35b, 29b
diff --git a/arch/avr32/lib/csum_partial.S b/arch/avr32/lib/csum_partial.S
deleted file mode 100644
index 6a262b5..0000000
--- a/arch/avr32/lib/csum_partial.S
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
- /*
- * unsigned int csum_partial(const unsigned char *buff,
- * int len, unsigned int sum)
- */
- .text
- .global csum_partial
- .type csum_partial,"function"
- .align 1
-csum_partial:
- /* checksum complete words, aligned or not */
-3: sub r11, 4
- brlt 5f
-4: ld.w r9, r12++
- add r10, r9
- acr r10
- sub r11, 4
- brge 4b
-
- /* return if we had a whole number of words */
-5: sub r11, -4
- reteq r10
-
- /* checksum any remaining bytes at the end */
- mov r9, 0
- mov r8, 0
- cp r11, 2
- brlt 6f
- ld.uh r9, r12++
- sub r11, 2
- breq 7f
- lsl r9, 16
-6: ld.ub r8, r12++
- lsl r8, 8
-7: or r9, r8
- add r10, r9
- acr r10
-
- retal r10
- .size csum_partial, . - csum_partial
diff --git a/arch/avr32/lib/csum_partial_copy_generic.S b/arch/avr32/lib/csum_partial_copy_generic.S
deleted file mode 100644
index a3a0f9b..0000000
--- a/arch/avr32/lib/csum_partial_copy_generic.S
+++ /dev/null
@@ -1,99 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <asm/errno.h>
-#include <asm/asm.h>
-
- /*
- * unsigned int csum_partial_copy_generic(const char *src, char *dst, int len
- * int sum, int *src_err_ptr,
- * int *dst_err_ptr)
- *
- * Copy src to dst while checksumming, otherwise like csum_partial.
- */
-
- .macro ld_src size, reg, ptr
-9999: ld.\size \reg, \ptr
- .section __ex_table, "a"
- .long 9999b, fixup_ld_src
- .previous
- .endm
-
- .macro st_dst size, ptr, reg
-9999: st.\size \ptr, \reg
- .section __ex_table, "a"
- .long 9999b, fixup_st_dst
- .previous
- .endm
-
- .text
- .global csum_partial_copy_generic
- .type csum_partial_copy_generic,"function"
- .align 1
-csum_partial_copy_generic:
- pushm r4-r7,lr
-
- /* The inner loop */
-1: sub r10, 4
- brlt 5f
-2: ld_src w, r5, r12++
- st_dst w, r11++, r5
- add r9, r5
- acr r9
- sub r10, 4
- brge 2b
-
- /* return if we had a whole number of words */
-5: sub r10, -4
- brne 7f
-
-6: mov r12, r9
- popm r4-r7,pc
-
- /* handle additional bytes at the tail */
-7: mov r5, 0
- mov r4, 32
-8: ld_src ub, r6, r12++
- st_dst b, r11++, r6
- lsl r5, 8
- sub r4, 8
- bfins r5, r6, 0, 8
- sub r10, 1
- brne 8b
-
- lsl r5, r5, r4
- add r9, r5
- acr r9
- rjmp 6b
-
- /* Exception handler */
- .section .fixup,"ax"
- .align 1
-fixup_ld_src:
- mov r9, -EFAULT
- cp.w r8, 0
- breq 1f
- st.w r8[0], r9
-
-1: /*
- * TODO: zero the complete destination - computing the rest
- * is too much work
- */
-
- mov r9, 0
- rjmp 6b
-
-fixup_st_dst:
- mov r9, -EFAULT
- lddsp r8, sp[20]
- cp.w r8, 0
- breq 1f
- st.w r8[0], r9
-1: mov r9, 0
- rjmp 6b
-
- .previous
diff --git a/arch/avr32/lib/delay.c b/arch/avr32/lib/delay.c
deleted file mode 100644
index c2f4a07..0000000
--- a/arch/avr32/lib/delay.c
+++ /dev/null
@@ -1,57 +0,0 @@
-/*
- * Precise Delay Loops for avr32
- *
- * Copyright (C) 1993 Linus Torvalds
- * Copyright (C) 1997 Martin Mares <mj@atrey.karlin.mff.cuni.cz>
- * Copyright (C) 2005-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/delay.h>
-#include <linux/module.h>
-#include <linux/timex.h>
-#include <linux/param.h>
-#include <linux/types.h>
-#include <linux/init.h>
-
-#include <asm/processor.h>
-#include <asm/sysreg.h>
-
-int read_current_timer(unsigned long *timer_value)
-{
- *timer_value = sysreg_read(COUNT);
- return 0;
-}
-
-void __delay(unsigned long loops)
-{
- unsigned bclock, now;
-
- bclock = sysreg_read(COUNT);
- do {
- now = sysreg_read(COUNT);
- } while ((now - bclock) < loops);
-}
-
-inline void __const_udelay(unsigned long xloops)
-{
- unsigned long long loops;
-
- asm("mulu.d %0, %1, %2"
- : "=r"(loops)
- : "r"(current_cpu_data.loops_per_jiffy * HZ), "r"(xloops));
- __delay(loops >> 32);
-}
-
-void __udelay(unsigned long usecs)
-{
- __const_udelay(usecs * 0x000010c7); /* 2**32 / 1000000 (rounded up) */
-}
-
-void __ndelay(unsigned long nsecs)
-{
- __const_udelay(nsecs * 0x00005); /* 2**32 / 1000000000 (rounded up) */
-}
diff --git a/arch/avr32/lib/findbit.S b/arch/avr32/lib/findbit.S
deleted file mode 100644
index b935864..0000000
--- a/arch/avr32/lib/findbit.S
+++ /dev/null
@@ -1,185 +0,0 @@
-/*
- * Copyright (C) 2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/linkage.h>
-
- .text
- /*
- * unsigned long find_first_zero_bit(const unsigned long *addr,
- * unsigned long size)
- */
-ENTRY(find_first_zero_bit)
- cp.w r11, 0
- reteq r11
- mov r9, r11
-1: ld.w r8, r12[0]
- com r8
- brne .L_found
- sub r12, -4
- sub r9, 32
- brgt 1b
- retal r11
-
- /*
- * unsigned long find_next_zero_bit(const unsigned long *addr,
- * unsigned long size,
- * unsigned long offset)
- */
-ENTRY(find_next_zero_bit)
- lsr r8, r10, 5
- sub r9, r11, r10
- retle r11
-
- lsl r8, 2
- add r12, r8
- andl r10, 31, COH
- breq 1f
-
- /* offset is not word-aligned. Handle the first (32 - r10) bits */
- ld.w r8, r12[0]
- com r8
- sub r12, -4
- lsr r8, r8, r10
- brne .L_found
-
- /* r9 = r9 - (32 - r10) = r9 + r10 - 32 */
- add r9, r10
- sub r9, 32
- retle r11
-
- /* Main loop. offset must be word-aligned */
-1: ld.w r8, r12[0]
- com r8
- brne .L_found
- sub r12, -4
- sub r9, 32
- brgt 1b
- retal r11
-
- /* Common return path for when a bit is actually found. */
-.L_found:
- brev r8
- clz r10, r8
- rsub r9, r11
- add r10, r9
-
- /* XXX: If we don't have to return exactly "size" when the bit
- is not found, we may drop this "min" thing */
- min r12, r11, r10
- retal r12
-
- /*
- * unsigned long find_first_bit(const unsigned long *addr,
- * unsigned long size)
- */
-ENTRY(find_first_bit)
- cp.w r11, 0
- reteq r11
- mov r9, r11
-1: ld.w r8, r12[0]
- cp.w r8, 0
- brne .L_found
- sub r12, -4
- sub r9, 32
- brgt 1b
- retal r11
-
- /*
- * unsigned long find_next_bit(const unsigned long *addr,
- * unsigned long size,
- * unsigned long offset)
- */
-ENTRY(find_next_bit)
- lsr r8, r10, 5
- sub r9, r11, r10
- retle r11
-
- lsl r8, 2
- add r12, r8
- andl r10, 31, COH
- breq 1f
-
- /* offset is not word-aligned. Handle the first (32 - r10) bits */
- ld.w r8, r12[0]
- sub r12, -4
- lsr r8, r8, r10
- brne .L_found
-
- /* r9 = r9 - (32 - r10) = r9 + r10 - 32 */
- add r9, r10
- sub r9, 32
- retle r11
-
- /* Main loop. offset must be word-aligned */
-1: ld.w r8, r12[0]
- cp.w r8, 0
- brne .L_found
- sub r12, -4
- sub r9, 32
- brgt 1b
- retal r11
-
-ENTRY(find_next_bit_le)
- lsr r8, r10, 5
- sub r9, r11, r10
- retle r11
-
- lsl r8, 2
- add r12, r8
- andl r10, 31, COH
- breq 1f
-
- /* offset is not word-aligned. Handle the first (32 - r10) bits */
- ldswp.w r8, r12[0]
- sub r12, -4
- lsr r8, r8, r10
- brne .L_found
-
- /* r9 = r9 - (32 - r10) = r9 + r10 - 32 */
- add r9, r10
- sub r9, 32
- retle r11
-
- /* Main loop. offset must be word-aligned */
-1: ldswp.w r8, r12[0]
- cp.w r8, 0
- brne .L_found
- sub r12, -4
- sub r9, 32
- brgt 1b
- retal r11
-
-ENTRY(find_next_zero_bit_le)
- lsr r8, r10, 5
- sub r9, r11, r10
- retle r11
-
- lsl r8, 2
- add r12, r8
- andl r10, 31, COH
- breq 1f
-
- /* offset is not word-aligned. Handle the first (32 - r10) bits */
- ldswp.w r8, r12[0]
- sub r12, -4
- com r8
- lsr r8, r8, r10
- brne .L_found
-
- /* r9 = r9 - (32 - r10) = r9 + r10 - 32 */
- add r9, r10
- sub r9, 32
- retle r11
-
- /* Main loop. offset must be word-aligned */
-1: ldswp.w r8, r12[0]
- com r8
- brne .L_found
- sub r12, -4
- sub r9, 32
- brgt 1b
- retal r11
diff --git a/arch/avr32/lib/io-readsb.S b/arch/avr32/lib/io-readsb.S
deleted file mode 100644
index cb2d869..0000000
--- a/arch/avr32/lib/io-readsb.S
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
- .text
-.Lnot_word_aligned:
-1: ld.ub r8, r12[0]
- sub r10, 1
- st.b r11++, r8
- reteq r12
- tst r11, r9
- brne 1b
-
- /* fall through */
-
- .global __raw_readsb
- .type __raw_readsb,@function
-__raw_readsb:
- cp.w r10, 0
- mov r9, 3
- reteq r12
-
- tst r11, r9
- brne .Lnot_word_aligned
-
- sub r10, 4
- brlt 2f
-
-1: ldins.b r8:t, r12[0]
- ldins.b r8:u, r12[0]
- ldins.b r8:l, r12[0]
- ldins.b r8:b, r12[0]
- st.w r11++, r8
- sub r10, 4
- brge 1b
-
-2: sub r10, -4
- reteq r12
-
-3: ld.ub r8, r12[0]
- sub r10, 1
- st.b r11++, r8
- brne 3b
-
- retal r12
diff --git a/arch/avr32/lib/io-readsl.S b/arch/avr32/lib/io-readsl.S
deleted file mode 100644
index b103511..0000000
--- a/arch/avr32/lib/io-readsl.S
+++ /dev/null
@@ -1,24 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
- .global __raw_readsl
- .type __raw_readsl,@function
-__raw_readsl:
- cp.w r10, 0
- reteq r12
-
- /*
- * If r11 isn't properly aligned, we might get an exception on
- * some implementations. But there's not much we can do about it.
- */
-1: ld.w r8, r12[0]
- sub r10, 1
- st.w r11++, r8
- brne 1b
-
- retal r12
diff --git a/arch/avr32/lib/io-readsw.S b/arch/avr32/lib/io-readsw.S
deleted file mode 100644
index 456be99..0000000
--- a/arch/avr32/lib/io-readsw.S
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-.Lnot_word_aligned:
- /*
- * Bad alignment will cause a hardware exception, which is as
- * good as anything. No need for us to check for proper alignment.
- */
- ld.uh r8, r12[0]
- sub r10, 1
- st.h r11++, r8
-
- /* fall through */
-
- .global __raw_readsw
- .type __raw_readsw,@function
-__raw_readsw:
- cp.w r10, 0
- reteq r12
- mov r9, 3
- tst r11, r9
- brne .Lnot_word_aligned
-
- sub r10, 2
- brlt 2f
-
-1: ldins.h r8:t, r12[0]
- ldins.h r8:b, r12[0]
- st.w r11++, r8
- sub r10, 2
- brge 1b
-
-2: sub r10, -2
- reteq r12
-
- ld.uh r8, r12[0]
- st.h r11++, r8
- retal r12
diff --git a/arch/avr32/lib/io-writesb.S b/arch/avr32/lib/io-writesb.S
deleted file mode 100644
index b4ebaac..0000000
--- a/arch/avr32/lib/io-writesb.S
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
- .text
-.Lnot_word_aligned:
-1: ld.ub r8, r11++
- sub r10, 1
- st.b r12[0], r8
- reteq r12
- tst r11, r9
- brne 1b
-
- /* fall through */
-
- .global __raw_writesb
- .type __raw_writesb,@function
-__raw_writesb:
- cp.w r10, 0
- mov r9, 3
- reteq r12
-
- tst r11, r9
- brne .Lnot_word_aligned
-
- sub r10, 4
- brlt 2f
-
-1: ld.w r8, r11++
- bfextu r9, r8, 24, 8
- st.b r12[0], r9
- bfextu r9, r8, 16, 8
- st.b r12[0], r9
- bfextu r9, r8, 8, 8
- st.b r12[0], r9
- st.b r12[0], r8
- sub r10, 4
- brge 1b
-
-2: sub r10, -4
- reteq r12
-
-3: ld.ub r8, r11++
- sub r10, 1
- st.b r12[0], r8
- brne 3b
-
- retal r12
diff --git a/arch/avr32/lib/io-writesl.S b/arch/avr32/lib/io-writesl.S
deleted file mode 100644
index 22138b3..0000000
--- a/arch/avr32/lib/io-writesl.S
+++ /dev/null
@@ -1,20 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
- .global __raw_writesl
- .type __raw_writesl,@function
-__raw_writesl:
- cp.w r10, 0
- reteq r12
-
-1: ld.w r8, r11++
- sub r10, 1
- st.w r12[0], r8
- brne 1b
-
- retal r12
diff --git a/arch/avr32/lib/io-writesw.S b/arch/avr32/lib/io-writesw.S
deleted file mode 100644
index 8c4a53f..0000000
--- a/arch/avr32/lib/io-writesw.S
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-.Lnot_word_aligned:
- ld.uh r8, r11++
- sub r10, 1
- st.h r12[0], r8
-
- .global __raw_writesw
- .type __raw_writesw,@function
-__raw_writesw:
- cp.w r10, 0
- mov r9, 3
- reteq r12
- tst r11, r9
- brne .Lnot_word_aligned
-
- sub r10, 2
- brlt 2f
-
-1: ld.w r8, r11++
- bfextu r9, r8, 16, 16
- st.h r12[0], r9
- st.h r12[0], r8
- sub r10, 2
- brge 1b
-
-2: sub r10, -2
- reteq r12
-
- ld.uh r8, r11++
- st.h r12[0], r8
- retal r12
diff --git a/arch/avr32/lib/memcpy.S b/arch/avr32/lib/memcpy.S
deleted file mode 100644
index c2ca49d..0000000
--- a/arch/avr32/lib/memcpy.S
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
- /*
- * void *memcpy(void *to, const void *from, unsigned long n)
- *
- * This implementation does word-aligned loads in the main loop,
- * possibly sacrificing alignment of stores.
- *
- * Hopefully, in most cases, both "to" and "from" will be
- * word-aligned to begin with.
- */
- .text
- .global memcpy
- .type memcpy, @function
-memcpy:
- mov r9, r11
- andl r9, 3, COH
- brne 1f
-
- /* At this point, "from" is word-aligned */
-2: mov r9, r12
-5: sub r10, 4
- brlt 4f
-
-3: ld.w r8, r11++
- sub r10, 4
- st.w r12++, r8
- brge 3b
-
-4: neg r10
- reteq r9
-
- /* Handle unaligned count */
- lsl r10, 2
- add pc, pc, r10
- ld.ub r8, r11++
- st.b r12++, r8
- ld.ub r8, r11++
- st.b r12++, r8
- ld.ub r8, r11++
- st.b r12++, r8
- retal r9
-
- /* Handle unaligned "from" pointer */
-1: sub r10, 4
- movlt r9, r12
- brlt 4b
- add r10, r9
- lsl r9, 2
- add pc, pc, r9
- ld.ub r8, r11++
- st.b r12++, r8
- ld.ub r8, r11++
- st.b r12++, r8
- ld.ub r8, r11++
- st.b r12++, r8
- mov r8, r12
- add pc, pc, r9
- sub r8, 1
- nop
- sub r8, 1
- nop
- sub r8, 1
- nop
- mov r9, r8
- rjmp 5b
diff --git a/arch/avr32/lib/memset.S b/arch/avr32/lib/memset.S
deleted file mode 100644
index 40da32c..0000000
--- a/arch/avr32/lib/memset.S
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * Based on linux/arch/arm/lib/memset.S
- * Copyright (C) 1995-2000 Russell King
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * ASM optimised string functions
- */
-#include <asm/asm.h>
-
- /*
- * r12: void *b
- * r11: int c
- * r10: size_t len
- *
- * Returns b in r12
- */
- .text
- .global memset
- .type memset, @function
- .align 5
-memset:
- mov r9, r12
- mov r8, r12
- or r11, r11, r11 << 8
- andl r9, 3, COH
- brne 1f
-
-2: or r11, r11, r11 << 16
- sub r10, 4
- brlt 5f
-
- /* Let's do some real work */
-4: st.w r8++, r11
- sub r10, 4
- brge 4b
-
- /*
- * When we get here, we've got less than 4 bytes to set. r10
- * might be negative.
- */
-5: sub r10, -4
- reteq r12
-
- /* Fastpath ends here, exactly 32 bytes from memset */
-
- /* Handle unaligned count or pointer */
- bld r10, 1
- brcc 6f
- st.b r8++, r11
- st.b r8++, r11
- bld r10, 0
- retcc r12
-6: st.b r8++, r11
- retal r12
-
- /* Handle unaligned pointer */
-1: sub r10, 4
- brlt 5b
- add r10, r9
- lsl r9, 1
- add pc, r9
- st.b r8++, r11
- st.b r8++, r11
- st.b r8++, r11
- rjmp 2b
-
- .size memset, . - memset
diff --git a/arch/avr32/lib/strncpy_from_user.S b/arch/avr32/lib/strncpy_from_user.S
deleted file mode 100644
index 72bd505..0000000
--- a/arch/avr32/lib/strncpy_from_user.S
+++ /dev/null
@@ -1,60 +0,0 @@
-/*
- * Copy to/from userspace with optional address space checking.
- *
- * Copyright 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/errno.h>
-
-#include <asm/page.h>
-#include <asm/thread_info.h>
-#include <asm/asm.h>
-
- /*
- * long strncpy_from_user(char *dst, const char *src, long count)
- *
- * On success, returns the length of the string, not including
- * the terminating NUL.
- *
- * If the string is longer than count, returns count
- *
- * If userspace access fails, returns -EFAULT
- */
- .text
- .align 1
- .global strncpy_from_user
- .type strncpy_from_user, "function"
-strncpy_from_user:
- mov r9, -EFAULT
- branch_if_kernel r8, __strncpy_from_user
- ret_if_privileged r8, r11, r10, r9
-
- .global __strncpy_from_user
- .type __strncpy_from_user, "function"
-__strncpy_from_user:
- cp.w r10, 0
- reteq 0
-
- mov r9, r10
-
-1: ld.ub r8, r11++
- st.b r12++, r8
- cp.w r8, 0
- breq 2f
- sub r9, 1
- brne 1b
-
-2: sub r10, r9
- retal r10
-
- .section .fixup, "ax"
- .align 1
-3: mov r12, -EFAULT
- retal r12
-
- .section __ex_table, "a"
- .align 2
- .long 1b, 3b
diff --git a/arch/avr32/lib/strnlen_user.S b/arch/avr32/lib/strnlen_user.S
deleted file mode 100644
index e46f4724..0000000
--- a/arch/avr32/lib/strnlen_user.S
+++ /dev/null
@@ -1,67 +0,0 @@
-/*
- * Copy to/from userspace with optional address space checking.
- *
- * Copyright 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <asm/page.h>
-#include <asm/thread_info.h>
-#include <asm/processor.h>
-#include <asm/asm.h>
-
- .text
- .align 1
- .global strnlen_user
- .type strnlen_user, "function"
-strnlen_user:
- branch_if_kernel r8, __strnlen_user
- sub r8, r11, 1
- add r8, r12
- retcs 0
- brmi adjust_length /* do a closer inspection */
-
- .global __strnlen_user
- .type __strnlen_user, "function"
-__strnlen_user:
- mov r10, r12
-
-10: ld.ub r8, r12++
- cp.w r8, 0
- breq 2f
- sub r11, 1
- brne 10b
-
- sub r12, -1
-2: sub r12, r10
- retal r12
-
-
- .type adjust_length, "function"
-adjust_length:
- cp.w r12, 0 /* addr must always be < TASK_SIZE */
- retmi 0
-
- pushm lr
- lddpc lr, _task_size
- sub r11, lr, r12
- mov r9, r11
- call __strnlen_user
- cp.w r12, r9
- brgt 1f
- popm pc
-1: popm pc, r12=0
-
- .align 2
-_task_size:
- .long TASK_SIZE
-
- .section .fixup, "ax"
- .align 1
-19: retal 0
-
- .section __ex_table, "a"
- .align 2
- .long 10b, 19b
diff --git a/arch/avr32/mach-at32ap/Kconfig b/arch/avr32/mach-at32ap/Kconfig
deleted file mode 100644
index a7bbcc8..0000000
--- a/arch/avr32/mach-at32ap/Kconfig
+++ /dev/null
@@ -1,31 +0,0 @@
-if PLATFORM_AT32AP
-
-menu "Atmel AVR32 AP options"
-
-choice
- prompt "AT32AP700x static memory bus width"
- depends on CPU_AT32AP700X
- default AP700X_16_BIT_SMC
- help
- Define the width of the AP7000 external static memory interface.
- This is used to determine how to mangle the address and/or data
- when doing little-endian port access.
-
- The current code can only support a single external memory bus
- width for all chip selects, excluding the flash (which is using
- raw access and is thus not affected by any of this.)
-
-config AP700X_32_BIT_SMC
- bool "32 bit"
-
-config AP700X_16_BIT_SMC
- bool "16 bit"
-
-config AP700X_8_BIT_SMC
- bool "8 bit"
-
-endchoice
-
-endmenu
-
-endif # PLATFORM_AT32AP
diff --git a/arch/avr32/mach-at32ap/Makefile b/arch/avr32/mach-at32ap/Makefile
deleted file mode 100644
index fc09ec4..0000000
--- a/arch/avr32/mach-at32ap/Makefile
+++ /dev/null
@@ -1,8 +0,0 @@
-obj-y += pdc.o clock.o intc.o extint.o pio.o hsmc.o
-obj-y += hmatrix.o
-obj-$(CONFIG_CPU_AT32AP700X) += at32ap700x.o pm-at32ap700x.o
-obj-$(CONFIG_PM) += pm.o
-
-ifeq ($(CONFIG_PM_DEBUG),y)
-CFLAGS_pm.o += -DDEBUG
-endif
diff --git a/arch/avr32/mach-at32ap/at32ap700x.c b/arch/avr32/mach-at32ap/at32ap700x.c
deleted file mode 100644
index 00d6dcc..0000000
--- a/arch/avr32/mach-at32ap/at32ap700x.c
+++ /dev/null
@@ -1,2368 +0,0 @@
-/*
- * Copyright (C) 2005-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/clk.h>
-#include <linux/delay.h>
-#include <linux/platform_data/dma-dw.h>
-#include <linux/fb.h>
-#include <linux/init.h>
-#include <linux/platform_device.h>
-#include <linux/dma-mapping.h>
-#include <linux/slab.h>
-#include <linux/gpio.h>
-#include <linux/spi/spi.h>
-#include <linux/usb/atmel_usba_udc.h>
-
-#include <linux/atmel-mci.h>
-
-#include <asm/io.h>
-#include <asm/irq.h>
-
-#include <mach/at32ap700x.h>
-#include <mach/board.h>
-#include <mach/hmatrix.h>
-#include <mach/portmux.h>
-#include <mach/sram.h>
-
-#include <sound/atmel-abdac.h>
-#include <sound/atmel-ac97c.h>
-
-#include <video/atmel_lcdc.h>
-
-#include "clock.h"
-#include "pio.h"
-#include "pm.h"
-
-
-#define PBMEM(base) \
- { \
- .start = base, \
- .end = base + 0x3ff, \
- .flags = IORESOURCE_MEM, \
- }
-#define IRQ(num) \
- { \
- .start = num, \
- .end = num, \
- .flags = IORESOURCE_IRQ, \
- }
-#define NAMED_IRQ(num, _name) \
- { \
- .start = num, \
- .end = num, \
- .name = _name, \
- .flags = IORESOURCE_IRQ, \
- }
-
-/* REVISIT these assume *every* device supports DMA, but several
- * don't ... tc, smc, pio, rtc, watchdog, pwm, ps2, and more.
- */
-#define DEFINE_DEV(_name, _id) \
-static u64 _name##_id##_dma_mask = DMA_BIT_MASK(32); \
-static struct platform_device _name##_id##_device = { \
- .name = #_name, \
- .id = _id, \
- .dev = { \
- .dma_mask = &_name##_id##_dma_mask, \
- .coherent_dma_mask = DMA_BIT_MASK(32), \
- }, \
- .resource = _name##_id##_resource, \
- .num_resources = ARRAY_SIZE(_name##_id##_resource), \
-}
-#define DEFINE_DEV_DATA(_name, _id) \
-static u64 _name##_id##_dma_mask = DMA_BIT_MASK(32); \
-static struct platform_device _name##_id##_device = { \
- .name = #_name, \
- .id = _id, \
- .dev = { \
- .dma_mask = &_name##_id##_dma_mask, \
- .platform_data = &_name##_id##_data, \
- .coherent_dma_mask = DMA_BIT_MASK(32), \
- }, \
- .resource = _name##_id##_resource, \
- .num_resources = ARRAY_SIZE(_name##_id##_resource), \
-}
-
-#define select_peripheral(port, pin_mask, periph, flags) \
- at32_select_periph(GPIO_##port##_BASE, pin_mask, \
- GPIO_##periph, flags)
-
-#define DEV_CLK(_name, devname, bus, _index) \
-static struct clk devname##_##_name = { \
- .name = #_name, \
- .dev = &devname##_device.dev, \
- .parent = &bus##_clk, \
- .mode = bus##_clk_mode, \
- .get_rate = bus##_clk_get_rate, \
- .index = _index, \
-}
-
-static DEFINE_SPINLOCK(pm_lock);
-
-static struct clk osc0;
-static struct clk osc1;
-
-static unsigned long osc_get_rate(struct clk *clk)
-{
- return at32_board_osc_rates[clk->index];
-}
-
-static unsigned long pll_get_rate(struct clk *clk, unsigned long control)
-{
- unsigned long div, mul, rate;
-
- div = PM_BFEXT(PLLDIV, control) + 1;
- mul = PM_BFEXT(PLLMUL, control) + 1;
-
- rate = clk->parent->get_rate(clk->parent);
- rate = (rate + div / 2) / div;
- rate *= mul;
-
- return rate;
-}
-
-static long pll_set_rate(struct clk *clk, unsigned long rate,
- u32 *pll_ctrl)
-{
- unsigned long mul;
- unsigned long mul_best_fit = 0;
- unsigned long div;
- unsigned long div_min;
- unsigned long div_max;
- unsigned long div_best_fit = 0;
- unsigned long base;
- unsigned long pll_in;
- unsigned long actual = 0;
- unsigned long rate_error;
- unsigned long rate_error_prev = ~0UL;
- u32 ctrl;
-
- /* Rate must be between 80 MHz and 200 Mhz. */
- if (rate < 80000000UL || rate > 200000000UL)
- return -EINVAL;
-
- ctrl = PM_BF(PLLOPT, 4);
- base = clk->parent->get_rate(clk->parent);
-
- /* PLL input frequency must be between 6 MHz and 32 MHz. */
- div_min = DIV_ROUND_UP(base, 32000000UL);
- div_max = base / 6000000UL;
-
- if (div_max < div_min)
- return -EINVAL;
-
- for (div = div_min; div <= div_max; div++) {
- pll_in = (base + div / 2) / div;
- mul = (rate + pll_in / 2) / pll_in;
-
- if (mul == 0)
- continue;
-
- actual = pll_in * mul;
- rate_error = abs(actual - rate);
-
- if (rate_error < rate_error_prev) {
- mul_best_fit = mul;
- div_best_fit = div;
- rate_error_prev = rate_error;
- }
-
- if (rate_error == 0)
- break;
- }
-
- if (div_best_fit == 0)
- return -EINVAL;
-
- ctrl |= PM_BF(PLLMUL, mul_best_fit - 1);
- ctrl |= PM_BF(PLLDIV, div_best_fit - 1);
- ctrl |= PM_BF(PLLCOUNT, 16);
-
- if (clk->parent == &osc1)
- ctrl |= PM_BIT(PLLOSC);
-
- *pll_ctrl = ctrl;
-
- return actual;
-}
-
-static unsigned long pll0_get_rate(struct clk *clk)
-{
- u32 control;
-
- control = pm_readl(PLL0);
-
- return pll_get_rate(clk, control);
-}
-
-static void pll1_mode(struct clk *clk, int enabled)
-{
- unsigned long timeout;
- u32 status;
- u32 ctrl;
-
- ctrl = pm_readl(PLL1);
-
- if (enabled) {
- if (!PM_BFEXT(PLLMUL, ctrl) && !PM_BFEXT(PLLDIV, ctrl)) {
- pr_debug("clk %s: failed to enable, rate not set\n",
- clk->name);
- return;
- }
-
- ctrl |= PM_BIT(PLLEN);
- pm_writel(PLL1, ctrl);
-
- /* Wait for PLL lock. */
- for (timeout = 10000; timeout; timeout--) {
- status = pm_readl(ISR);
- if (status & PM_BIT(LOCK1))
- break;
- udelay(10);
- }
-
- if (!(status & PM_BIT(LOCK1)))
- printk(KERN_ERR "clk %s: timeout waiting for lock\n",
- clk->name);
- } else {
- ctrl &= ~PM_BIT(PLLEN);
- pm_writel(PLL1, ctrl);
- }
-}
-
-static unsigned long pll1_get_rate(struct clk *clk)
-{
- u32 control;
-
- control = pm_readl(PLL1);
-
- return pll_get_rate(clk, control);
-}
-
-static long pll1_set_rate(struct clk *clk, unsigned long rate, int apply)
-{
- u32 ctrl = 0;
- unsigned long actual_rate;
-
- actual_rate = pll_set_rate(clk, rate, &ctrl);
-
- if (apply) {
- if (actual_rate != rate)
- return -EINVAL;
- if (clk->users > 0)
- return -EBUSY;
- pr_debug(KERN_INFO "clk %s: new rate %lu (actual rate %lu)\n",
- clk->name, rate, actual_rate);
- pm_writel(PLL1, ctrl);
- }
-
- return actual_rate;
-}
-
-static int pll1_set_parent(struct clk *clk, struct clk *parent)
-{
- u32 ctrl;
-
- if (clk->users > 0)
- return -EBUSY;
-
- ctrl = pm_readl(PLL1);
- WARN_ON(ctrl & PM_BIT(PLLEN));
-
- if (parent == &osc0)
- ctrl &= ~PM_BIT(PLLOSC);
- else if (parent == &osc1)
- ctrl |= PM_BIT(PLLOSC);
- else
- return -EINVAL;
-
- pm_writel(PLL1, ctrl);
- clk->parent = parent;
-
- return 0;
-}
-
-/*
- * The AT32AP7000 has five primary clock sources: One 32kHz
- * oscillator, two crystal oscillators and two PLLs.
- */
-static struct clk osc32k = {
- .name = "osc32k",
- .get_rate = osc_get_rate,
- .users = 1,
- .index = 0,
-};
-static struct clk osc0 = {
- .name = "osc0",
- .get_rate = osc_get_rate,
- .users = 1,
- .index = 1,
-};
-static struct clk osc1 = {
- .name = "osc1",
- .get_rate = osc_get_rate,
- .index = 2,
-};
-static struct clk pll0 = {
- .name = "pll0",
- .get_rate = pll0_get_rate,
- .parent = &osc0,
-};
-static struct clk pll1 = {
- .name = "pll1",
- .mode = pll1_mode,
- .get_rate = pll1_get_rate,
- .set_rate = pll1_set_rate,
- .set_parent = pll1_set_parent,
- .parent = &osc0,
-};
-
-/*
- * The main clock can be either osc0 or pll0. The boot loader may
- * have chosen one for us, so we don't really know which one until we
- * have a look at the SM.
- */
-static struct clk *main_clock;
-
-/*
- * Synchronous clocks are generated from the main clock. The clocks
- * must satisfy the constraint
- * fCPU >= fHSB >= fPB
- * i.e. each clock must not be faster than its parent.
- */
-static unsigned long bus_clk_get_rate(struct clk *clk, unsigned int shift)
-{
- return main_clock->get_rate(main_clock) >> shift;
-};
-
-static void cpu_clk_mode(struct clk *clk, int enabled)
-{
- unsigned long flags;
- u32 mask;
-
- spin_lock_irqsave(&pm_lock, flags);
- mask = pm_readl(CPU_MASK);
- if (enabled)
- mask |= 1 << clk->index;
- else
- mask &= ~(1 << clk->index);
- pm_writel(CPU_MASK, mask);
- spin_unlock_irqrestore(&pm_lock, flags);
-}
-
-static unsigned long cpu_clk_get_rate(struct clk *clk)
-{
- unsigned long cksel, shift = 0;
-
- cksel = pm_readl(CKSEL);
- if (cksel & PM_BIT(CPUDIV))
- shift = PM_BFEXT(CPUSEL, cksel) + 1;
-
- return bus_clk_get_rate(clk, shift);
-}
-
-static long cpu_clk_set_rate(struct clk *clk, unsigned long rate, int apply)
-{
- u32 control;
- unsigned long parent_rate, child_div, actual_rate, div;
-
- parent_rate = clk->parent->get_rate(clk->parent);
- control = pm_readl(CKSEL);
-
- if (control & PM_BIT(HSBDIV))
- child_div = 1 << (PM_BFEXT(HSBSEL, control) + 1);
- else
- child_div = 1;
-
- if (rate > 3 * (parent_rate / 4) || child_div == 1) {
- actual_rate = parent_rate;
- control &= ~PM_BIT(CPUDIV);
- } else {
- unsigned int cpusel;
- div = (parent_rate + rate / 2) / rate;
- if (div > child_div)
- div = child_div;
- cpusel = (div > 1) ? (fls(div) - 2) : 0;
- control = PM_BIT(CPUDIV) | PM_BFINS(CPUSEL, cpusel, control);
- actual_rate = parent_rate / (1 << (cpusel + 1));
- }
-
- pr_debug("clk %s: new rate %lu (actual rate %lu)\n",
- clk->name, rate, actual_rate);
-
- if (apply)
- pm_writel(CKSEL, control);
-
- return actual_rate;
-}
-
-static void hsb_clk_mode(struct clk *clk, int enabled)
-{
- unsigned long flags;
- u32 mask;
-
- spin_lock_irqsave(&pm_lock, flags);
- mask = pm_readl(HSB_MASK);
- if (enabled)
- mask |= 1 << clk->index;
- else
- mask &= ~(1 << clk->index);
- pm_writel(HSB_MASK, mask);
- spin_unlock_irqrestore(&pm_lock, flags);
-}
-
-static unsigned long hsb_clk_get_rate(struct clk *clk)
-{
- unsigned long cksel, shift = 0;
-
- cksel = pm_readl(CKSEL);
- if (cksel & PM_BIT(HSBDIV))
- shift = PM_BFEXT(HSBSEL, cksel) + 1;
-
- return bus_clk_get_rate(clk, shift);
-}
-
-void pba_clk_mode(struct clk *clk, int enabled)
-{
- unsigned long flags;
- u32 mask;
-
- spin_lock_irqsave(&pm_lock, flags);
- mask = pm_readl(PBA_MASK);
- if (enabled)
- mask |= 1 << clk->index;
- else
- mask &= ~(1 << clk->index);
- pm_writel(PBA_MASK, mask);
- spin_unlock_irqrestore(&pm_lock, flags);
-}
-
-unsigned long pba_clk_get_rate(struct clk *clk)
-{
- unsigned long cksel, shift = 0;
-
- cksel = pm_readl(CKSEL);
- if (cksel & PM_BIT(PBADIV))
- shift = PM_BFEXT(PBASEL, cksel) + 1;
-
- return bus_clk_get_rate(clk, shift);
-}
-
-static void pbb_clk_mode(struct clk *clk, int enabled)
-{
- unsigned long flags;
- u32 mask;
-
- spin_lock_irqsave(&pm_lock, flags);
- mask = pm_readl(PBB_MASK);
- if (enabled)
- mask |= 1 << clk->index;
- else
- mask &= ~(1 << clk->index);
- pm_writel(PBB_MASK, mask);
- spin_unlock_irqrestore(&pm_lock, flags);
-}
-
-static unsigned long pbb_clk_get_rate(struct clk *clk)
-{
- unsigned long cksel, shift = 0;
-
- cksel = pm_readl(CKSEL);
- if (cksel & PM_BIT(PBBDIV))
- shift = PM_BFEXT(PBBSEL, cksel) + 1;
-
- return bus_clk_get_rate(clk, shift);
-}
-
-static struct clk cpu_clk = {
- .name = "cpu",
- .get_rate = cpu_clk_get_rate,
- .set_rate = cpu_clk_set_rate,
- .users = 1,
-};
-static struct clk hsb_clk = {
- .name = "hsb",
- .parent = &cpu_clk,
- .get_rate = hsb_clk_get_rate,
-};
-static struct clk pba_clk = {
- .name = "pba",
- .parent = &hsb_clk,
- .mode = hsb_clk_mode,
- .get_rate = pba_clk_get_rate,
- .index = 1,
-};
-static struct clk pbb_clk = {
- .name = "pbb",
- .parent = &hsb_clk,
- .mode = hsb_clk_mode,
- .get_rate = pbb_clk_get_rate,
- .users = 1,
- .index = 2,
-};
-
-/* --------------------------------------------------------------------
- * Generic Clock operations
- * -------------------------------------------------------------------- */
-
-static void genclk_mode(struct clk *clk, int enabled)
-{
- u32 control;
-
- control = pm_readl(GCCTRL(clk->index));
- if (enabled)
- control |= PM_BIT(CEN);
- else
- control &= ~PM_BIT(CEN);
- pm_writel(GCCTRL(clk->index), control);
-}
-
-static unsigned long genclk_get_rate(struct clk *clk)
-{
- u32 control;
- unsigned long div = 1;
-
- control = pm_readl(GCCTRL(clk->index));
- if (control & PM_BIT(DIVEN))
- div = 2 * (PM_BFEXT(DIV, control) + 1);
-
- return clk->parent->get_rate(clk->parent) / div;
-}
-
-static long genclk_set_rate(struct clk *clk, unsigned long rate, int apply)
-{
- u32 control;
- unsigned long parent_rate, actual_rate, div;
-
- parent_rate = clk->parent->get_rate(clk->parent);
- control = pm_readl(GCCTRL(clk->index));
-
- if (rate > 3 * parent_rate / 4) {
- actual_rate = parent_rate;
- control &= ~PM_BIT(DIVEN);
- } else {
- div = (parent_rate + rate) / (2 * rate) - 1;
- control = PM_BFINS(DIV, div, control) | PM_BIT(DIVEN);
- actual_rate = parent_rate / (2 * (div + 1));
- }
-
- dev_dbg(clk->dev, "clk %s: new rate %lu (actual rate %lu)\n",
- clk->name, rate, actual_rate);
-
- if (apply)
- pm_writel(GCCTRL(clk->index), control);
-
- return actual_rate;
-}
-
-int genclk_set_parent(struct clk *clk, struct clk *parent)
-{
- u32 control;
-
- dev_dbg(clk->dev, "clk %s: new parent %s (was %s)\n",
- clk->name, parent->name, clk->parent->name);
-
- control = pm_readl(GCCTRL(clk->index));
-
- if (parent == &osc1 || parent == &pll1)
- control |= PM_BIT(OSCSEL);
- else if (parent == &osc0 || parent == &pll0)
- control &= ~PM_BIT(OSCSEL);
- else
- return -EINVAL;
-
- if (parent == &pll0 || parent == &pll1)
- control |= PM_BIT(PLLSEL);
- else
- control &= ~PM_BIT(PLLSEL);
-
- pm_writel(GCCTRL(clk->index), control);
- clk->parent = parent;
-
- return 0;
-}
-
-static void __init genclk_init_parent(struct clk *clk)
-{
- u32 control;
- struct clk *parent;
-
- BUG_ON(clk->index > 7);
-
- control = pm_readl(GCCTRL(clk->index));
- if (control & PM_BIT(OSCSEL))
- parent = (control & PM_BIT(PLLSEL)) ? &pll1 : &osc1;
- else
- parent = (control & PM_BIT(PLLSEL)) ? &pll0 : &osc0;
-
- clk->parent = parent;
-}
-
-static struct resource dw_dmac0_resource[] = {
- PBMEM(0xff200000),
- IRQ(2),
-};
-DEFINE_DEV(dw_dmac, 0);
-DEV_CLK(hclk, dw_dmac0, hsb, 10);
-
-/* --------------------------------------------------------------------
- * System peripherals
- * -------------------------------------------------------------------- */
-static struct resource at32_pm0_resource[] = {
- {
- .start = 0xfff00000,
- .end = 0xfff0007f,
- .flags = IORESOURCE_MEM,
- },
- IRQ(20),
-};
-
-static struct resource at32ap700x_rtc0_resource[] = {
- {
- .start = 0xfff00080,
- .end = 0xfff000af,
- .flags = IORESOURCE_MEM,
- },
- IRQ(21),
-};
-
-static struct resource at32_wdt0_resource[] = {
- {
- .start = 0xfff000b0,
- .end = 0xfff000cf,
- .flags = IORESOURCE_MEM,
- },
-};
-
-static struct resource at32_eic0_resource[] = {
- {
- .start = 0xfff00100,
- .end = 0xfff0013f,
- .flags = IORESOURCE_MEM,
- },
- IRQ(19),
-};
-
-DEFINE_DEV(at32_pm, 0);
-DEFINE_DEV(at32ap700x_rtc, 0);
-DEFINE_DEV(at32_wdt, 0);
-DEFINE_DEV(at32_eic, 0);
-
-/*
- * Peripheral clock for PM, RTC, WDT and EIC. PM will ensure that this
- * is always running.
- */
-static struct clk at32_pm_pclk = {
- .name = "pclk",
- .dev = &at32_pm0_device.dev,
- .parent = &pbb_clk,
- .mode = pbb_clk_mode,
- .get_rate = pbb_clk_get_rate,
- .users = 1,
- .index = 0,
-};
-
-static struct resource intc0_resource[] = {
- PBMEM(0xfff00400),
-};
-struct platform_device at32_intc0_device = {
- .name = "intc",
- .id = 0,
- .resource = intc0_resource,
- .num_resources = ARRAY_SIZE(intc0_resource),
-};
-DEV_CLK(pclk, at32_intc0, pbb, 1);
-
-static struct clk ebi_clk = {
- .name = "ebi",
- .parent = &hsb_clk,
- .mode = hsb_clk_mode,
- .get_rate = hsb_clk_get_rate,
- .users = 1,
-};
-static struct clk hramc_clk = {
- .name = "hramc",
- .parent = &hsb_clk,
- .mode = hsb_clk_mode,
- .get_rate = hsb_clk_get_rate,
- .users = 1,
- .index = 3,
-};
-static struct clk sdramc_clk = {
- .name = "sdramc_clk",
- .parent = &pbb_clk,
- .mode = pbb_clk_mode,
- .get_rate = pbb_clk_get_rate,
- .users = 1,
- .index = 14,
-};
-
-static struct resource smc0_resource[] = {
- PBMEM(0xfff03400),
-};
-DEFINE_DEV(smc, 0);
-DEV_CLK(pclk, smc0, pbb, 13);
-DEV_CLK(mck, smc0, hsb, 0);
-
-static struct platform_device pdc_device = {
- .name = "pdc",
- .id = 0,
-};
-DEV_CLK(hclk, pdc, hsb, 4);
-DEV_CLK(pclk, pdc, pba, 16);
-
-static struct clk pico_clk = {
- .name = "pico",
- .parent = &cpu_clk,
- .mode = cpu_clk_mode,
- .get_rate = cpu_clk_get_rate,
- .users = 1,
-};
-
-/* --------------------------------------------------------------------
- * HMATRIX
- * -------------------------------------------------------------------- */
-
-struct clk at32_hmatrix_clk = {
- .name = "hmatrix_clk",
- .parent = &pbb_clk,
- .mode = pbb_clk_mode,
- .get_rate = pbb_clk_get_rate,
- .index = 2,
- .users = 1,
-};
-
-/*
- * Set bits in the HMATRIX Special Function Register (SFR) used by the
- * External Bus Interface (EBI). This can be used to enable special
- * features like CompactFlash support, NAND Flash support, etc. on
- * certain chipselects.
- */
-static inline void set_ebi_sfr_bits(u32 mask)
-{
- hmatrix_sfr_set_bits(HMATRIX_SLAVE_EBI, mask);
-}
-
-/* --------------------------------------------------------------------
- * Timer/Counter (TC)
- * -------------------------------------------------------------------- */
-
-static struct resource at32_tcb0_resource[] = {
- PBMEM(0xfff00c00),
- IRQ(22),
-};
-static struct platform_device at32_tcb0_device = {
- .name = "atmel_tcb",
- .id = 0,
- .resource = at32_tcb0_resource,
- .num_resources = ARRAY_SIZE(at32_tcb0_resource),
-};
-DEV_CLK(t0_clk, at32_tcb0, pbb, 3);
-
-static struct resource at32_tcb1_resource[] = {
- PBMEM(0xfff01000),
- IRQ(23),
-};
-static struct platform_device at32_tcb1_device = {
- .name = "atmel_tcb",
- .id = 1,
- .resource = at32_tcb1_resource,
- .num_resources = ARRAY_SIZE(at32_tcb1_resource),
-};
-DEV_CLK(t0_clk, at32_tcb1, pbb, 4);
-
-/* --------------------------------------------------------------------
- * PIO
- * -------------------------------------------------------------------- */
-
-static struct resource pio0_resource[] = {
- PBMEM(0xffe02800),
- IRQ(13),
-};
-DEFINE_DEV(pio, 0);
-DEV_CLK(mck, pio0, pba, 10);
-
-static struct resource pio1_resource[] = {
- PBMEM(0xffe02c00),
- IRQ(14),
-};
-DEFINE_DEV(pio, 1);
-DEV_CLK(mck, pio1, pba, 11);
-
-static struct resource pio2_resource[] = {
- PBMEM(0xffe03000),
- IRQ(15),
-};
-DEFINE_DEV(pio, 2);
-DEV_CLK(mck, pio2, pba, 12);
-
-static struct resource pio3_resource[] = {
- PBMEM(0xffe03400),
- IRQ(16),
-};
-DEFINE_DEV(pio, 3);
-DEV_CLK(mck, pio3, pba, 13);
-
-static struct resource pio4_resource[] = {
- PBMEM(0xffe03800),
- IRQ(17),
-};
-DEFINE_DEV(pio, 4);
-DEV_CLK(mck, pio4, pba, 14);
-
-static int __init system_device_init(void)
-{
- platform_device_register(&at32_pm0_device);
- platform_device_register(&at32_intc0_device);
- platform_device_register(&at32ap700x_rtc0_device);
- platform_device_register(&at32_wdt0_device);
- platform_device_register(&at32_eic0_device);
- platform_device_register(&smc0_device);
- platform_device_register(&pdc_device);
- platform_device_register(&dw_dmac0_device);
-
- platform_device_register(&at32_tcb0_device);
- platform_device_register(&at32_tcb1_device);
-
- platform_device_register(&pio0_device);
- platform_device_register(&pio1_device);
- platform_device_register(&pio2_device);
- platform_device_register(&pio3_device);
- platform_device_register(&pio4_device);
-
- return 0;
-}
-core_initcall(system_device_init);
-
-/* --------------------------------------------------------------------
- * PSIF
- * -------------------------------------------------------------------- */
-static struct resource atmel_psif0_resource[] __initdata = {
- {
- .start = 0xffe03c00,
- .end = 0xffe03cff,
- .flags = IORESOURCE_MEM,
- },
- IRQ(18),
-};
-static struct clk atmel_psif0_pclk = {
- .name = "pclk",
- .parent = &pba_clk,
- .mode = pba_clk_mode,
- .get_rate = pba_clk_get_rate,
- .index = 15,
-};
-
-static struct resource atmel_psif1_resource[] __initdata = {
- {
- .start = 0xffe03d00,
- .end = 0xffe03dff,
- .flags = IORESOURCE_MEM,
- },
- IRQ(18),
-};
-static struct clk atmel_psif1_pclk = {
- .name = "pclk",
- .parent = &pba_clk,
- .mode = pba_clk_mode,
- .get_rate = pba_clk_get_rate,
- .index = 15,
-};
-
-struct platform_device *__init at32_add_device_psif(unsigned int id)
-{
- struct platform_device *pdev;
- u32 pin_mask;
-
- if (!(id == 0 || id == 1))
- return NULL;
-
- pdev = platform_device_alloc("atmel_psif", id);
- if (!pdev)
- return NULL;
-
- switch (id) {
- case 0:
- pin_mask = (1 << 8) | (1 << 9); /* CLOCK & DATA */
-
- if (platform_device_add_resources(pdev, atmel_psif0_resource,
- ARRAY_SIZE(atmel_psif0_resource)))
- goto err_add_resources;
- atmel_psif0_pclk.dev = &pdev->dev;
- select_peripheral(PIOA, pin_mask, PERIPH_A, 0);
- break;
- case 1:
- pin_mask = (1 << 11) | (1 << 12); /* CLOCK & DATA */
-
- if (platform_device_add_resources(pdev, atmel_psif1_resource,
- ARRAY_SIZE(atmel_psif1_resource)))
- goto err_add_resources;
- atmel_psif1_pclk.dev = &pdev->dev;
- select_peripheral(PIOB, pin_mask, PERIPH_A, 0);
- break;
- default:
- return NULL;
- }
-
- platform_device_add(pdev);
- return pdev;
-
-err_add_resources:
- platform_device_put(pdev);
- return NULL;
-}
-
-/* --------------------------------------------------------------------
- * USART
- * -------------------------------------------------------------------- */
-
-static struct atmel_uart_data atmel_usart0_data = {
- .use_dma_tx = 1,
- .use_dma_rx = 1,
-};
-static struct resource atmel_usart0_resource[] = {
- PBMEM(0xffe00c00),
- IRQ(6),
-};
-DEFINE_DEV_DATA(atmel_usart, 0);
-DEV_CLK(usart, atmel_usart0, pba, 3);
-
-static struct atmel_uart_data atmel_usart1_data = {
- .use_dma_tx = 1,
- .use_dma_rx = 1,
-};
-static struct resource atmel_usart1_resource[] = {
- PBMEM(0xffe01000),
- IRQ(7),
-};
-DEFINE_DEV_DATA(atmel_usart, 1);
-DEV_CLK(usart, atmel_usart1, pba, 4);
-
-static struct atmel_uart_data atmel_usart2_data = {
- .use_dma_tx = 1,
- .use_dma_rx = 1,
-};
-static struct resource atmel_usart2_resource[] = {
- PBMEM(0xffe01400),
- IRQ(8),
-};
-DEFINE_DEV_DATA(atmel_usart, 2);
-DEV_CLK(usart, atmel_usart2, pba, 5);
-
-static struct atmel_uart_data atmel_usart3_data = {
- .use_dma_tx = 1,
- .use_dma_rx = 1,
-};
-static struct resource atmel_usart3_resource[] = {
- PBMEM(0xffe01800),
- IRQ(9),
-};
-DEFINE_DEV_DATA(atmel_usart, 3);
-DEV_CLK(usart, atmel_usart3, pba, 6);
-
-static inline void configure_usart0_pins(int flags)
-{
- u32 pin_mask = (1 << 8) | (1 << 9); /* RXD & TXD */
- if (flags & ATMEL_USART_RTS) pin_mask |= (1 << 6);
- if (flags & ATMEL_USART_CTS) pin_mask |= (1 << 7);
- if (flags & ATMEL_USART_CLK) pin_mask |= (1 << 10);
-
- select_peripheral(PIOA, pin_mask, PERIPH_B, AT32_GPIOF_PULLUP);
-}
-
-static inline void configure_usart1_pins(int flags)
-{
- u32 pin_mask = (1 << 17) | (1 << 18); /* RXD & TXD */
- if (flags & ATMEL_USART_RTS) pin_mask |= (1 << 19);
- if (flags & ATMEL_USART_CTS) pin_mask |= (1 << 20);
- if (flags & ATMEL_USART_CLK) pin_mask |= (1 << 16);
-
- select_peripheral(PIOA, pin_mask, PERIPH_A, AT32_GPIOF_PULLUP);
-}
-
-static inline void configure_usart2_pins(int flags)
-{
- u32 pin_mask = (1 << 26) | (1 << 27); /* RXD & TXD */
- if (flags & ATMEL_USART_RTS) pin_mask |= (1 << 30);
- if (flags & ATMEL_USART_CTS) pin_mask |= (1 << 29);
- if (flags & ATMEL_USART_CLK) pin_mask |= (1 << 28);
-
- select_peripheral(PIOB, pin_mask, PERIPH_B, AT32_GPIOF_PULLUP);
-}
-
-static inline void configure_usart3_pins(int flags)
-{
- u32 pin_mask = (1 << 18) | (1 << 17); /* RXD & TXD */
- if (flags & ATMEL_USART_RTS) pin_mask |= (1 << 16);
- if (flags & ATMEL_USART_CTS) pin_mask |= (1 << 15);
- if (flags & ATMEL_USART_CLK) pin_mask |= (1 << 19);
-
- select_peripheral(PIOB, pin_mask, PERIPH_B, AT32_GPIOF_PULLUP);
-}
-
-static struct platform_device *__initdata at32_usarts[4];
-
-void __init at32_map_usart(unsigned int hw_id, unsigned int line, int flags)
-{
- struct platform_device *pdev;
- struct atmel_uart_data *pdata;
-
- switch (hw_id) {
- case 0:
- pdev = &atmel_usart0_device;
- configure_usart0_pins(flags);
- break;
- case 1:
- pdev = &atmel_usart1_device;
- configure_usart1_pins(flags);
- break;
- case 2:
- pdev = &atmel_usart2_device;
- configure_usart2_pins(flags);
- break;
- case 3:
- pdev = &atmel_usart3_device;
- configure_usart3_pins(flags);
- break;
- default:
- return;
- }
-
- if (PXSEG(pdev->resource[0].start) == P4SEG) {
- /* Addresses in the P4 segment are permanently mapped 1:1 */
- struct atmel_uart_data *data = pdev->dev.platform_data;
- data->regs = (void __iomem *)pdev->resource[0].start;
- }
-
- pdev->id = line;
- pdata = pdev->dev.platform_data;
- pdata->num = line;
- at32_usarts[line] = pdev;
-}
-
-struct platform_device *__init at32_add_device_usart(unsigned int id)
-{
- platform_device_register(at32_usarts[id]);
- return at32_usarts[id];
-}
-
-void __init at32_setup_serial_console(unsigned int usart_id)
-{
-#ifdef CONFIG_SERIAL_ATMEL
- atmel_default_console_device = at32_usarts[usart_id];
-#endif
-}
-
-/* --------------------------------------------------------------------
- * Ethernet
- * -------------------------------------------------------------------- */
-
-#ifdef CONFIG_CPU_AT32AP7000
-static struct macb_platform_data macb0_data;
-static struct resource macb0_resource[] = {
- PBMEM(0xfff01800),
- IRQ(25),
-};
-DEFINE_DEV_DATA(macb, 0);
-DEV_CLK(hclk, macb0, hsb, 8);
-DEV_CLK(pclk, macb0, pbb, 6);
-
-static struct macb_platform_data macb1_data;
-static struct resource macb1_resource[] = {
- PBMEM(0xfff01c00),
- IRQ(26),
-};
-DEFINE_DEV_DATA(macb, 1);
-DEV_CLK(hclk, macb1, hsb, 9);
-DEV_CLK(pclk, macb1, pbb, 7);
-
-struct platform_device *__init
-at32_add_device_eth(unsigned int id, struct macb_platform_data *data)
-{
- struct platform_device *pdev;
- u32 pin_mask;
-
- switch (id) {
- case 0:
- pdev = &macb0_device;
-
- pin_mask = (1 << 3); /* TXD0 */
- pin_mask |= (1 << 4); /* TXD1 */
- pin_mask |= (1 << 7); /* TXEN */
- pin_mask |= (1 << 8); /* TXCK */
- pin_mask |= (1 << 9); /* RXD0 */
- pin_mask |= (1 << 10); /* RXD1 */
- pin_mask |= (1 << 13); /* RXER */
- pin_mask |= (1 << 15); /* RXDV */
- pin_mask |= (1 << 16); /* MDC */
- pin_mask |= (1 << 17); /* MDIO */
-
- if (!data->is_rmii) {
- pin_mask |= (1 << 0); /* COL */
- pin_mask |= (1 << 1); /* CRS */
- pin_mask |= (1 << 2); /* TXER */
- pin_mask |= (1 << 5); /* TXD2 */
- pin_mask |= (1 << 6); /* TXD3 */
- pin_mask |= (1 << 11); /* RXD2 */
- pin_mask |= (1 << 12); /* RXD3 */
- pin_mask |= (1 << 14); /* RXCK */
-#ifndef CONFIG_BOARD_MIMC200
- pin_mask |= (1 << 18); /* SPD */
-#endif
- }
-
- select_peripheral(PIOC, pin_mask, PERIPH_A, 0);
-
- break;
-
- case 1:
- pdev = &macb1_device;
-
- pin_mask = (1 << 13); /* TXD0 */
- pin_mask |= (1 << 14); /* TXD1 */
- pin_mask |= (1 << 11); /* TXEN */
- pin_mask |= (1 << 12); /* TXCK */
- pin_mask |= (1 << 10); /* RXD0 */
- pin_mask |= (1 << 6); /* RXD1 */
- pin_mask |= (1 << 5); /* RXER */
- pin_mask |= (1 << 4); /* RXDV */
- pin_mask |= (1 << 3); /* MDC */
- pin_mask |= (1 << 2); /* MDIO */
-
-#ifndef CONFIG_BOARD_MIMC200
- if (!data->is_rmii)
- pin_mask |= (1 << 15); /* SPD */
-#endif
-
- select_peripheral(PIOD, pin_mask, PERIPH_B, 0);
-
- if (!data->is_rmii) {
- pin_mask = (1 << 19); /* COL */
- pin_mask |= (1 << 23); /* CRS */
- pin_mask |= (1 << 26); /* TXER */
- pin_mask |= (1 << 27); /* TXD2 */
- pin_mask |= (1 << 28); /* TXD3 */
- pin_mask |= (1 << 29); /* RXD2 */
- pin_mask |= (1 << 30); /* RXD3 */
- pin_mask |= (1 << 24); /* RXCK */
-
- select_peripheral(PIOC, pin_mask, PERIPH_B, 0);
- }
- break;
-
- default:
- return NULL;
- }
-
- memcpy(pdev->dev.platform_data, data, sizeof(struct macb_platform_data));
- platform_device_register(pdev);
-
- return pdev;
-}
-#endif
-
-/* --------------------------------------------------------------------
- * SPI
- * -------------------------------------------------------------------- */
-static struct resource atmel_spi0_resource[] = {
- PBMEM(0xffe00000),
- IRQ(3),
-};
-DEFINE_DEV(atmel_spi, 0);
-DEV_CLK(spi_clk, atmel_spi0, pba, 0);
-
-static struct resource atmel_spi1_resource[] = {
- PBMEM(0xffe00400),
- IRQ(4),
-};
-DEFINE_DEV(atmel_spi, 1);
-DEV_CLK(spi_clk, atmel_spi1, pba, 1);
-
-void __init
-at32_spi_setup_slaves(unsigned int bus_num, struct spi_board_info *b, unsigned int n)
-{
- /*
- * Manage the chipselects as GPIOs, normally using the same pins
- * the SPI controller expects; but boards can use other pins.
- */
- static u8 __initdata spi_pins[][4] = {
- { GPIO_PIN_PA(3), GPIO_PIN_PA(4),
- GPIO_PIN_PA(5), GPIO_PIN_PA(20) },
- { GPIO_PIN_PB(2), GPIO_PIN_PB(3),
- GPIO_PIN_PB(4), GPIO_PIN_PA(27) },
- };
- unsigned int pin, mode;
-
- /* There are only 2 SPI controllers */
- if (bus_num > 1)
- return;
-
- for (; n; n--, b++) {
- b->bus_num = bus_num;
- if (b->chip_select >= 4)
- continue;
- pin = (unsigned)b->controller_data;
- if (!pin) {
- pin = spi_pins[bus_num][b->chip_select];
- b->controller_data = (void *)pin;
- }
- mode = AT32_GPIOF_OUTPUT;
- if (!(b->mode & SPI_CS_HIGH))
- mode |= AT32_GPIOF_HIGH;
- at32_select_gpio(pin, mode);
- }
-}
-
-struct platform_device *__init
-at32_add_device_spi(unsigned int id, struct spi_board_info *b, unsigned int n)
-{
- struct platform_device *pdev;
- u32 pin_mask;
-
- switch (id) {
- case 0:
- pdev = &atmel_spi0_device;
- pin_mask = (1 << 1) | (1 << 2); /* MOSI & SCK */
-
- /* pullup MISO so a level is always defined */
- select_peripheral(PIOA, (1 << 0), PERIPH_A, AT32_GPIOF_PULLUP);
- select_peripheral(PIOA, pin_mask, PERIPH_A, 0);
-
- at32_spi_setup_slaves(0, b, n);
- break;
-
- case 1:
- pdev = &atmel_spi1_device;
- pin_mask = (1 << 1) | (1 << 5); /* MOSI */
-
- /* pullup MISO so a level is always defined */
- select_peripheral(PIOB, (1 << 0), PERIPH_B, AT32_GPIOF_PULLUP);
- select_peripheral(PIOB, pin_mask, PERIPH_B, 0);
-
- at32_spi_setup_slaves(1, b, n);
- break;
-
- default:
- return NULL;
- }
-
- spi_register_board_info(b, n);
- platform_device_register(pdev);
- return pdev;
-}
-
-/* --------------------------------------------------------------------
- * TWI
- * -------------------------------------------------------------------- */
-static struct resource atmel_twi0_resource[] __initdata = {
- PBMEM(0xffe00800),
- IRQ(5),
-};
-static struct clk atmel_twi0_pclk = {
- .name = "twi_pclk",
- .parent = &pba_clk,
- .mode = pba_clk_mode,
- .get_rate = pba_clk_get_rate,
- .index = 2,
-};
-
-struct platform_device *__init at32_add_device_twi(unsigned int id,
- struct i2c_board_info *b,
- unsigned int n)
-{
- struct platform_device *pdev;
- u32 pin_mask;
-
- if (id != 0)
- return NULL;
-
- pdev = platform_device_alloc("atmel_twi", id);
- if (!pdev)
- return NULL;
-
- if (platform_device_add_resources(pdev, atmel_twi0_resource,
- ARRAY_SIZE(atmel_twi0_resource)))
- goto err_add_resources;
-
- pin_mask = (1 << 6) | (1 << 7); /* SDA & SDL */
-
- select_peripheral(PIOA, pin_mask, PERIPH_A, 0);
-
- atmel_twi0_pclk.dev = &pdev->dev;
-
- if (b)
- i2c_register_board_info(id, b, n);
-
- platform_device_add(pdev);
- return pdev;
-
-err_add_resources:
- platform_device_put(pdev);
- return NULL;
-}
-
-/* --------------------------------------------------------------------
- * MMC
- * -------------------------------------------------------------------- */
-static struct resource atmel_mci0_resource[] __initdata = {
- PBMEM(0xfff02400),
- IRQ(28),
-};
-static struct clk atmel_mci0_pclk = {
- .name = "mci_clk",
- .parent = &pbb_clk,
- .mode = pbb_clk_mode,
- .get_rate = pbb_clk_get_rate,
- .index = 9,
-};
-
-static bool at32_mci_dma_filter(struct dma_chan *chan, void *pdata)
-{
- struct dw_dma_slave *sl = pdata;
-
- if (!sl)
- return false;
-
- if (sl->dma_dev == chan->device->dev) {
- chan->private = sl;
- return true;
- }
-
- return false;
-}
-
-struct platform_device *__init
-at32_add_device_mci(unsigned int id, struct mci_platform_data *data)
-{
- struct platform_device *pdev;
- struct dw_dma_slave *slave;
- u32 pioa_mask;
- u32 piob_mask;
-
- if (id != 0 || !data)
- return NULL;
-
- /* Must have at least one usable slot */
- if (!data->slot[0].bus_width && !data->slot[1].bus_width)
- return NULL;
-
- pdev = platform_device_alloc("atmel_mci", id);
- if (!pdev)
- goto fail;
-
- if (platform_device_add_resources(pdev, atmel_mci0_resource,
- ARRAY_SIZE(atmel_mci0_resource)))
- goto fail;
-
- slave = kzalloc(sizeof(*slave), GFP_KERNEL);
- if (!slave)
- goto fail;
-
- slave->dma_dev = &dw_dmac0_device.dev;
- slave->src_id = 0;
- slave->dst_id = 1;
- slave->m_master = 1;
- slave->p_master = 0;
-
- data->dma_slave = slave;
- data->dma_filter = at32_mci_dma_filter;
-
- if (platform_device_add_data(pdev, data,
- sizeof(struct mci_platform_data)))
- goto fail_free;
-
- /* CLK line is common to both slots */
- pioa_mask = 1 << 10;
-
- switch (data->slot[0].bus_width) {
- case 4:
- pioa_mask |= 1 << 13; /* DATA1 */
- pioa_mask |= 1 << 14; /* DATA2 */
- pioa_mask |= 1 << 15; /* DATA3 */
- /* fall through */
- case 1:
- pioa_mask |= 1 << 11; /* CMD */
- pioa_mask |= 1 << 12; /* DATA0 */
-
- if (gpio_is_valid(data->slot[0].detect_pin))
- at32_select_gpio(data->slot[0].detect_pin, 0);
- if (gpio_is_valid(data->slot[0].wp_pin))
- at32_select_gpio(data->slot[0].wp_pin, 0);
- break;
- case 0:
- /* Slot is unused */
- break;
- default:
- goto fail_free;
- }
-
- select_peripheral(PIOA, pioa_mask, PERIPH_A, 0);
- piob_mask = 0;
-
- switch (data->slot[1].bus_width) {
- case 4:
- piob_mask |= 1 << 8; /* DATA1 */
- piob_mask |= 1 << 9; /* DATA2 */
- piob_mask |= 1 << 10; /* DATA3 */
- /* fall through */
- case 1:
- piob_mask |= 1 << 6; /* CMD */
- piob_mask |= 1 << 7; /* DATA0 */
- select_peripheral(PIOB, piob_mask, PERIPH_B, 0);
-
- if (gpio_is_valid(data->slot[1].detect_pin))
- at32_select_gpio(data->slot[1].detect_pin, 0);
- if (gpio_is_valid(data->slot[1].wp_pin))
- at32_select_gpio(data->slot[1].wp_pin, 0);
- break;
- case 0:
- /* Slot is unused */
- break;
- default:
- if (!data->slot[0].bus_width)
- goto fail_free;
-
- data->slot[1].bus_width = 0;
- break;
- }
-
- atmel_mci0_pclk.dev = &pdev->dev;
-
- platform_device_add(pdev);
- return pdev;
-
-fail_free:
- kfree(slave);
-fail:
- data->dma_slave = NULL;
- platform_device_put(pdev);
- return NULL;
-}
-
-/* --------------------------------------------------------------------
- * LCDC
- * -------------------------------------------------------------------- */
-#if defined(CONFIG_CPU_AT32AP7000) || defined(CONFIG_CPU_AT32AP7002)
-static struct atmel_lcdfb_pdata atmel_lcdfb0_data;
-static struct resource atmel_lcdfb0_resource[] = {
- {
- .start = 0xff000000,
- .end = 0xff000fff,
- .flags = IORESOURCE_MEM,
- },
- IRQ(1),
- {
- /* Placeholder for pre-allocated fb memory */
- .start = 0x00000000,
- .end = 0x00000000,
- .flags = 0,
- },
-};
-DEFINE_DEV_DATA(atmel_lcdfb, 0);
-DEV_CLK(hclk, atmel_lcdfb0, hsb, 7);
-static struct clk atmel_lcdfb0_pixclk = {
- .name = "lcdc_clk",
- .dev = &atmel_lcdfb0_device.dev,
- .mode = genclk_mode,
- .get_rate = genclk_get_rate,
- .set_rate = genclk_set_rate,
- .set_parent = genclk_set_parent,
- .index = 7,
-};
-
-struct platform_device *__init
-at32_add_device_lcdc(unsigned int id, struct atmel_lcdfb_pdata *data,
- unsigned long fbmem_start, unsigned long fbmem_len,
- u64 pin_mask)
-{
- struct platform_device *pdev;
- struct atmel_lcdfb_pdata *info;
- struct fb_monspecs *monspecs;
- struct fb_videomode *modedb;
- unsigned int modedb_size;
- u32 portc_mask, portd_mask, porte_mask;
-
- /*
- * Do a deep copy of the fb data, monspecs and modedb. Make
- * sure all allocations are done before setting up the
- * portmux.
- */
- monspecs = kmemdup(data->default_monspecs,
- sizeof(struct fb_monspecs), GFP_KERNEL);
- if (!monspecs)
- return NULL;
-
- modedb_size = sizeof(struct fb_videomode) * monspecs->modedb_len;
- modedb = kmemdup(monspecs->modedb, modedb_size, GFP_KERNEL);
- if (!modedb)
- goto err_dup_modedb;
- monspecs->modedb = modedb;
-
- switch (id) {
- case 0:
- pdev = &atmel_lcdfb0_device;
-
- if (pin_mask == 0ULL)
- /* Default to "full" lcdc control signals and 24bit */
- pin_mask = ATMEL_LCDC_PRI_24BIT | ATMEL_LCDC_PRI_CONTROL;
-
- /* LCDC on port C */
- portc_mask = pin_mask & 0xfff80000;
- select_peripheral(PIOC, portc_mask, PERIPH_A, 0);
-
- /* LCDC on port D */
- portd_mask = pin_mask & 0x0003ffff;
- select_peripheral(PIOD, portd_mask, PERIPH_A, 0);
-
- /* LCDC on port E */
- porte_mask = (pin_mask >> 32) & 0x0007ffff;
- select_peripheral(PIOE, porte_mask, PERIPH_B, 0);
-
- clk_set_parent(&atmel_lcdfb0_pixclk, &pll0);
- clk_set_rate(&atmel_lcdfb0_pixclk, clk_get_rate(&pll0));
- break;
-
- default:
- goto err_invalid_id;
- }
-
- if (fbmem_len) {
- pdev->resource[2].start = fbmem_start;
- pdev->resource[2].end = fbmem_start + fbmem_len - 1;
- pdev->resource[2].flags = IORESOURCE_MEM;
- }
-
- info = pdev->dev.platform_data;
- memcpy(info, data, sizeof(struct atmel_lcdfb_pdata));
- info->default_monspecs = monspecs;
-
- pdev->name = "at32ap-lcdfb";
-
- platform_device_register(pdev);
- return pdev;
-
-err_invalid_id:
- kfree(modedb);
-err_dup_modedb:
- kfree(monspecs);
- return NULL;
-}
-#endif
-
-/* --------------------------------------------------------------------
- * PWM
- * -------------------------------------------------------------------- */
-static struct resource atmel_pwm0_resource[] __initdata = {
- PBMEM(0xfff01400),
- IRQ(24),
-};
-static struct clk atmel_pwm0_mck = {
- .name = "at91sam9rl-pwm",
- .parent = &pbb_clk,
- .mode = pbb_clk_mode,
- .get_rate = pbb_clk_get_rate,
- .index = 5,
-};
-
-struct platform_device *__init at32_add_device_pwm(u32 mask)
-{
- struct platform_device *pdev;
- u32 pin_mask;
-
- if (!mask)
- return NULL;
-
- pdev = platform_device_alloc("at91sam9rl-pwm", 0);
- if (!pdev)
- return NULL;
-
- if (platform_device_add_resources(pdev, atmel_pwm0_resource,
- ARRAY_SIZE(atmel_pwm0_resource)))
- goto out_free_pdev;
-
- pin_mask = 0;
- if (mask & (1 << 0))
- pin_mask |= (1 << 28);
- if (mask & (1 << 1))
- pin_mask |= (1 << 29);
- if (pin_mask > 0)
- select_peripheral(PIOA, pin_mask, PERIPH_A, 0);
-
- pin_mask = 0;
- if (mask & (1 << 2))
- pin_mask |= (1 << 21);
- if (mask & (1 << 3))
- pin_mask |= (1 << 22);
- if (pin_mask > 0)
- select_peripheral(PIOA, pin_mask, PERIPH_B, 0);
-
- atmel_pwm0_mck.dev = &pdev->dev;
-
- platform_device_add(pdev);
-
- return pdev;
-
-out_free_pdev:
- platform_device_put(pdev);
- return NULL;
-}
-
-/* --------------------------------------------------------------------
- * SSC
- * -------------------------------------------------------------------- */
-static struct resource ssc0_resource[] = {
- PBMEM(0xffe01c00),
- IRQ(10),
-};
-DEFINE_DEV(ssc, 0);
-DEV_CLK(pclk, ssc0, pba, 7);
-
-static struct resource ssc1_resource[] = {
- PBMEM(0xffe02000),
- IRQ(11),
-};
-DEFINE_DEV(ssc, 1);
-DEV_CLK(pclk, ssc1, pba, 8);
-
-static struct resource ssc2_resource[] = {
- PBMEM(0xffe02400),
- IRQ(12),
-};
-DEFINE_DEV(ssc, 2);
-DEV_CLK(pclk, ssc2, pba, 9);
-
-struct platform_device *__init
-at32_add_device_ssc(unsigned int id, unsigned int flags)
-{
- struct platform_device *pdev;
- u32 pin_mask = 0;
-
- switch (id) {
- case 0:
- pdev = &ssc0_device;
- if (flags & ATMEL_SSC_RF)
- pin_mask |= (1 << 21); /* RF */
- if (flags & ATMEL_SSC_RK)
- pin_mask |= (1 << 22); /* RK */
- if (flags & ATMEL_SSC_TK)
- pin_mask |= (1 << 23); /* TK */
- if (flags & ATMEL_SSC_TF)
- pin_mask |= (1 << 24); /* TF */
- if (flags & ATMEL_SSC_TD)
- pin_mask |= (1 << 25); /* TD */
- if (flags & ATMEL_SSC_RD)
- pin_mask |= (1 << 26); /* RD */
-
- if (pin_mask > 0)
- select_peripheral(PIOA, pin_mask, PERIPH_A, 0);
-
- break;
- case 1:
- pdev = &ssc1_device;
- if (flags & ATMEL_SSC_RF)
- pin_mask |= (1 << 0); /* RF */
- if (flags & ATMEL_SSC_RK)
- pin_mask |= (1 << 1); /* RK */
- if (flags & ATMEL_SSC_TK)
- pin_mask |= (1 << 2); /* TK */
- if (flags & ATMEL_SSC_TF)
- pin_mask |= (1 << 3); /* TF */
- if (flags & ATMEL_SSC_TD)
- pin_mask |= (1 << 4); /* TD */
- if (flags & ATMEL_SSC_RD)
- pin_mask |= (1 << 5); /* RD */
-
- if (pin_mask > 0)
- select_peripheral(PIOA, pin_mask, PERIPH_B, 0);
-
- break;
- case 2:
- pdev = &ssc2_device;
- if (flags & ATMEL_SSC_TD)
- pin_mask |= (1 << 13); /* TD */
- if (flags & ATMEL_SSC_RD)
- pin_mask |= (1 << 14); /* RD */
- if (flags & ATMEL_SSC_TK)
- pin_mask |= (1 << 15); /* TK */
- if (flags & ATMEL_SSC_TF)
- pin_mask |= (1 << 16); /* TF */
- if (flags & ATMEL_SSC_RF)
- pin_mask |= (1 << 17); /* RF */
- if (flags & ATMEL_SSC_RK)
- pin_mask |= (1 << 18); /* RK */
-
- if (pin_mask > 0)
- select_peripheral(PIOB, pin_mask, PERIPH_A, 0);
-
- break;
- default:
- return NULL;
- }
-
- platform_device_register(pdev);
- return pdev;
-}
-
-/* --------------------------------------------------------------------
- * USB Device Controller
- * -------------------------------------------------------------------- */
-static struct resource usba0_resource[] __initdata = {
- {
- .start = 0xff300000,
- .end = 0xff3fffff,
- .flags = IORESOURCE_MEM,
- }, {
- .start = 0xfff03000,
- .end = 0xfff033ff,
- .flags = IORESOURCE_MEM,
- },
- IRQ(31),
-};
-static struct clk usba0_pclk = {
- .name = "pclk",
- .parent = &pbb_clk,
- .mode = pbb_clk_mode,
- .get_rate = pbb_clk_get_rate,
- .index = 12,
-};
-static struct clk usba0_hclk = {
- .name = "hclk",
- .parent = &hsb_clk,
- .mode = hsb_clk_mode,
- .get_rate = hsb_clk_get_rate,
- .index = 6,
-};
-
-#define EP(nam, idx, maxpkt, maxbk, dma, isoc) \
- [idx] = { \
- .name = nam, \
- .index = idx, \
- .fifo_size = maxpkt, \
- .nr_banks = maxbk, \
- .can_dma = dma, \
- .can_isoc = isoc, \
- }
-
-static struct usba_ep_data at32_usba_ep[] __initdata = {
- EP("ep0", 0, 64, 1, 0, 0),
- EP("ep1", 1, 512, 2, 1, 1),
- EP("ep2", 2, 512, 2, 1, 1),
- EP("ep3-int", 3, 64, 3, 1, 0),
- EP("ep4-int", 4, 64, 3, 1, 0),
- EP("ep5", 5, 1024, 3, 1, 1),
- EP("ep6", 6, 1024, 3, 1, 1),
-};
-
-#undef EP
-
-struct platform_device *__init
-at32_add_device_usba(unsigned int id, struct usba_platform_data *data)
-{
- /*
- * pdata doesn't have room for any endpoints, so we need to
- * append room for the ones we need right after it.
- */
- struct {
- struct usba_platform_data pdata;
- struct usba_ep_data ep[7];
- } usba_data;
- struct platform_device *pdev;
-
- if (id != 0)
- return NULL;
-
- pdev = platform_device_alloc("atmel_usba_udc", 0);
- if (!pdev)
- return NULL;
-
- if (platform_device_add_resources(pdev, usba0_resource,
- ARRAY_SIZE(usba0_resource)))
- goto out_free_pdev;
-
- if (data) {
- usba_data.pdata.vbus_pin = data->vbus_pin;
- usba_data.pdata.vbus_pin_inverted = data->vbus_pin_inverted;
- } else {
- usba_data.pdata.vbus_pin = -EINVAL;
- usba_data.pdata.vbus_pin_inverted = -EINVAL;
- }
-
- data = &usba_data.pdata;
- data->num_ep = ARRAY_SIZE(at32_usba_ep);
- memcpy(data->ep, at32_usba_ep, sizeof(at32_usba_ep));
-
- if (platform_device_add_data(pdev, data, sizeof(usba_data)))
- goto out_free_pdev;
-
- if (gpio_is_valid(data->vbus_pin))
- at32_select_gpio(data->vbus_pin, 0);
-
- usba0_pclk.dev = &pdev->dev;
- usba0_hclk.dev = &pdev->dev;
-
- platform_device_add(pdev);
-
- return pdev;
-
-out_free_pdev:
- platform_device_put(pdev);
- return NULL;
-}
-
-/* --------------------------------------------------------------------
- * IDE / CompactFlash
- * -------------------------------------------------------------------- */
-#if defined(CONFIG_CPU_AT32AP7000) || defined(CONFIG_CPU_AT32AP7001)
-static struct resource at32_smc_cs4_resource[] __initdata = {
- {
- .start = 0x04000000,
- .end = 0x07ffffff,
- .flags = IORESOURCE_MEM,
- },
- IRQ(~0UL), /* Magic IRQ will be overridden */
-};
-static struct resource at32_smc_cs5_resource[] __initdata = {
- {
- .start = 0x20000000,
- .end = 0x23ffffff,
- .flags = IORESOURCE_MEM,
- },
- IRQ(~0UL), /* Magic IRQ will be overridden */
-};
-
-static int __init at32_init_ide_or_cf(struct platform_device *pdev,
- unsigned int cs, unsigned int extint)
-{
- static unsigned int extint_pin_map[4] __initdata = {
- (1 << 25),
- (1 << 26),
- (1 << 27),
- (1 << 28),
- };
- static bool common_pins_initialized __initdata = false;
- unsigned int extint_pin;
- int ret;
- u32 pin_mask;
-
- if (extint >= ARRAY_SIZE(extint_pin_map))
- return -EINVAL;
- extint_pin = extint_pin_map[extint];
-
- switch (cs) {
- case 4:
- ret = platform_device_add_resources(pdev,
- at32_smc_cs4_resource,
- ARRAY_SIZE(at32_smc_cs4_resource));
- if (ret)
- return ret;
-
- /* NCS4 -> OE_N */
- select_peripheral(PIOE, (1 << 21), PERIPH_A, 0);
- hmatrix_sfr_set_bits(HMATRIX_SLAVE_EBI, HMATRIX_EBI_CF0_ENABLE);
- break;
- case 5:
- ret = platform_device_add_resources(pdev,
- at32_smc_cs5_resource,
- ARRAY_SIZE(at32_smc_cs5_resource));
- if (ret)
- return ret;
-
- /* NCS5 -> OE_N */
- select_peripheral(PIOE, (1 << 22), PERIPH_A, 0);
- hmatrix_sfr_set_bits(HMATRIX_SLAVE_EBI, HMATRIX_EBI_CF1_ENABLE);
- break;
- default:
- return -EINVAL;
- }
-
- if (!common_pins_initialized) {
- pin_mask = (1 << 19); /* CFCE1 -> CS0_N */
- pin_mask |= (1 << 20); /* CFCE2 -> CS1_N */
- pin_mask |= (1 << 23); /* CFRNW -> DIR */
- pin_mask |= (1 << 24); /* NWAIT <- IORDY */
-
- select_peripheral(PIOE, pin_mask, PERIPH_A, 0);
-
- common_pins_initialized = true;
- }
-
- select_peripheral(PIOB, extint_pin, PERIPH_A, AT32_GPIOF_DEGLITCH);
-
- pdev->resource[1].start = EIM_IRQ_BASE + extint;
- pdev->resource[1].end = pdev->resource[1].start;
-
- return 0;
-}
-
-struct platform_device *__init
-at32_add_device_ide(unsigned int id, unsigned int extint,
- struct ide_platform_data *data)
-{
- struct platform_device *pdev;
-
- pdev = platform_device_alloc("at32_ide", id);
- if (!pdev)
- goto fail;
-
- if (platform_device_add_data(pdev, data,
- sizeof(struct ide_platform_data)))
- goto fail;
-
- if (at32_init_ide_or_cf(pdev, data->cs, extint))
- goto fail;
-
- platform_device_add(pdev);
- return pdev;
-
-fail:
- platform_device_put(pdev);
- return NULL;
-}
-
-struct platform_device *__init
-at32_add_device_cf(unsigned int id, unsigned int extint,
- struct cf_platform_data *data)
-{
- struct platform_device *pdev;
-
- pdev = platform_device_alloc("at32_cf", id);
- if (!pdev)
- goto fail;
-
- if (platform_device_add_data(pdev, data,
- sizeof(struct cf_platform_data)))
- goto fail;
-
- if (at32_init_ide_or_cf(pdev, data->cs, extint))
- goto fail;
-
- if (gpio_is_valid(data->detect_pin))
- at32_select_gpio(data->detect_pin, AT32_GPIOF_DEGLITCH);
- if (gpio_is_valid(data->reset_pin))
- at32_select_gpio(data->reset_pin, 0);
- if (gpio_is_valid(data->vcc_pin))
- at32_select_gpio(data->vcc_pin, 0);
- /* READY is used as extint, so we can't select it as gpio */
-
- platform_device_add(pdev);
- return pdev;
-
-fail:
- platform_device_put(pdev);
- return NULL;
-}
-#endif
-
-/* --------------------------------------------------------------------
- * NAND Flash / SmartMedia
- * -------------------------------------------------------------------- */
-static struct resource smc_cs3_resource[] __initdata = {
- {
- .start = 0x0c000000,
- .end = 0x0fffffff,
- .flags = IORESOURCE_MEM,
- }, {
- .start = 0xfff03c00,
- .end = 0xfff03fff,
- .flags = IORESOURCE_MEM,
- },
-};
-
-struct platform_device *__init
-at32_add_device_nand(unsigned int id, struct atmel_nand_data *data)
-{
- struct platform_device *pdev;
-
- if (id != 0 || !data)
- return NULL;
-
- pdev = platform_device_alloc("atmel_nand", id);
- if (!pdev)
- goto fail;
-
- if (platform_device_add_resources(pdev, smc_cs3_resource,
- ARRAY_SIZE(smc_cs3_resource)))
- goto fail;
-
- /* For at32ap7000, we use the reset workaround for nand driver */
- data->need_reset_workaround = true;
-
- if (platform_device_add_data(pdev, data,
- sizeof(struct atmel_nand_data)))
- goto fail;
-
- hmatrix_sfr_set_bits(HMATRIX_SLAVE_EBI, HMATRIX_EBI_NAND_ENABLE);
- if (data->enable_pin)
- at32_select_gpio(data->enable_pin,
- AT32_GPIOF_OUTPUT | AT32_GPIOF_HIGH);
- if (data->rdy_pin)
- at32_select_gpio(data->rdy_pin, 0);
- if (data->det_pin)
- at32_select_gpio(data->det_pin, 0);
-
- platform_device_add(pdev);
- return pdev;
-
-fail:
- platform_device_put(pdev);
- return NULL;
-}
-
-/* --------------------------------------------------------------------
- * AC97C
- * -------------------------------------------------------------------- */
-static struct resource atmel_ac97c0_resource[] __initdata = {
- PBMEM(0xfff02800),
- IRQ(29),
-};
-static struct clk atmel_ac97c0_pclk = {
- .name = "pclk",
- .parent = &pbb_clk,
- .mode = pbb_clk_mode,
- .get_rate = pbb_clk_get_rate,
- .index = 10,
-};
-
-struct platform_device *__init
-at32_add_device_ac97c(unsigned int id, struct ac97c_platform_data *data,
- unsigned int flags)
-{
- struct platform_device *pdev;
- struct dw_dma_slave *rx_dws;
- struct dw_dma_slave *tx_dws;
- struct ac97c_platform_data _data;
- u32 pin_mask;
-
- if (id != 0)
- return NULL;
-
- pdev = platform_device_alloc("atmel_ac97c", id);
- if (!pdev)
- return NULL;
-
- if (platform_device_add_resources(pdev, atmel_ac97c0_resource,
- ARRAY_SIZE(atmel_ac97c0_resource)))
- goto out_free_resources;
-
- if (!data) {
- data = &_data;
- memset(data, 0, sizeof(struct ac97c_platform_data));
- data->reset_pin = -ENODEV;
- }
-
- rx_dws = &data->rx_dws;
- tx_dws = &data->tx_dws;
-
- /* Check if DMA slave interface for capture should be configured. */
- if (flags & AC97C_CAPTURE) {
- rx_dws->dma_dev = &dw_dmac0_device.dev;
- rx_dws->src_id = 3;
- rx_dws->m_master = 0;
- rx_dws->p_master = 1;
- }
-
- /* Check if DMA slave interface for playback should be configured. */
- if (flags & AC97C_PLAYBACK) {
- tx_dws->dma_dev = &dw_dmac0_device.dev;
- tx_dws->dst_id = 4;
- tx_dws->m_master = 0;
- tx_dws->p_master = 1;
- }
-
- if (platform_device_add_data(pdev, data,
- sizeof(struct ac97c_platform_data)))
- goto out_free_resources;
-
- /* SDO | SYNC | SCLK | SDI */
- pin_mask = (1 << 20) | (1 << 21) | (1 << 22) | (1 << 23);
-
- select_peripheral(PIOB, pin_mask, PERIPH_B, 0);
-
- if (gpio_is_valid(data->reset_pin))
- at32_select_gpio(data->reset_pin, AT32_GPIOF_OUTPUT
- | AT32_GPIOF_HIGH);
-
- atmel_ac97c0_pclk.dev = &pdev->dev;
-
- platform_device_add(pdev);
- return pdev;
-
-out_free_resources:
- platform_device_put(pdev);
- return NULL;
-}
-
-/* --------------------------------------------------------------------
- * ABDAC
- * -------------------------------------------------------------------- */
-static struct resource abdac0_resource[] __initdata = {
- PBMEM(0xfff02000),
- IRQ(27),
-};
-static struct clk abdac0_pclk = {
- .name = "pclk",
- .parent = &pbb_clk,
- .mode = pbb_clk_mode,
- .get_rate = pbb_clk_get_rate,
- .index = 8,
-};
-static struct clk abdac0_sample_clk = {
- .name = "sample_clk",
- .mode = genclk_mode,
- .get_rate = genclk_get_rate,
- .set_rate = genclk_set_rate,
- .set_parent = genclk_set_parent,
- .index = 6,
-};
-
-struct platform_device *__init
-at32_add_device_abdac(unsigned int id, struct atmel_abdac_pdata *data)
-{
- struct platform_device *pdev;
- struct dw_dma_slave *dws;
- u32 pin_mask;
-
- if (id != 0 || !data)
- return NULL;
-
- pdev = platform_device_alloc("atmel_abdac", id);
- if (!pdev)
- return NULL;
-
- if (platform_device_add_resources(pdev, abdac0_resource,
- ARRAY_SIZE(abdac0_resource)))
- goto out_free_resources;
-
- dws = &data->dws;
-
- dws->dma_dev = &dw_dmac0_device.dev;
- dws->dst_id = 2;
- dws->m_master = 0;
- dws->p_master = 1;
-
- if (platform_device_add_data(pdev, data,
- sizeof(struct atmel_abdac_pdata)))
- goto out_free_resources;
-
- pin_mask = (1 << 20) | (1 << 22); /* DATA1 & DATAN1 */
- pin_mask |= (1 << 21) | (1 << 23); /* DATA0 & DATAN0 */
-
- select_peripheral(PIOB, pin_mask, PERIPH_A, 0);
-
- abdac0_pclk.dev = &pdev->dev;
- abdac0_sample_clk.dev = &pdev->dev;
-
- platform_device_add(pdev);
- return pdev;
-
-out_free_resources:
- platform_device_put(pdev);
- return NULL;
-}
-
-/* --------------------------------------------------------------------
- * GCLK
- * -------------------------------------------------------------------- */
-static struct clk gclk0 = {
- .name = "gclk0",
- .mode = genclk_mode,
- .get_rate = genclk_get_rate,
- .set_rate = genclk_set_rate,
- .set_parent = genclk_set_parent,
- .index = 0,
-};
-static struct clk gclk1 = {
- .name = "gclk1",
- .mode = genclk_mode,
- .get_rate = genclk_get_rate,
- .set_rate = genclk_set_rate,
- .set_parent = genclk_set_parent,
- .index = 1,
-};
-static struct clk gclk2 = {
- .name = "gclk2",
- .mode = genclk_mode,
- .get_rate = genclk_get_rate,
- .set_rate = genclk_set_rate,
- .set_parent = genclk_set_parent,
- .index = 2,
-};
-static struct clk gclk3 = {
- .name = "gclk3",
- .mode = genclk_mode,
- .get_rate = genclk_get_rate,
- .set_rate = genclk_set_rate,
- .set_parent = genclk_set_parent,
- .index = 3,
-};
-static struct clk gclk4 = {
- .name = "gclk4",
- .mode = genclk_mode,
- .get_rate = genclk_get_rate,
- .set_rate = genclk_set_rate,
- .set_parent = genclk_set_parent,
- .index = 4,
-};
-
-static __initdata struct clk *init_clocks[] = {
- &osc32k,
- &osc0,
- &osc1,
- &pll0,
- &pll1,
- &cpu_clk,
- &hsb_clk,
- &pba_clk,
- &pbb_clk,
- &at32_pm_pclk,
- &at32_intc0_pclk,
- &at32_hmatrix_clk,
- &ebi_clk,
- &hramc_clk,
- &sdramc_clk,
- &smc0_pclk,
- &smc0_mck,
- &pdc_hclk,
- &pdc_pclk,
- &dw_dmac0_hclk,
- &pico_clk,
- &pio0_mck,
- &pio1_mck,
- &pio2_mck,
- &pio3_mck,
- &pio4_mck,
- &at32_tcb0_t0_clk,
- &at32_tcb1_t0_clk,
- &atmel_psif0_pclk,
- &atmel_psif1_pclk,
- &atmel_usart0_usart,
- &atmel_usart1_usart,
- &atmel_usart2_usart,
- &atmel_usart3_usart,
- &atmel_pwm0_mck,
-#if defined(CONFIG_CPU_AT32AP7000)
- &macb0_hclk,
- &macb0_pclk,
- &macb1_hclk,
- &macb1_pclk,
-#endif
- &atmel_spi0_spi_clk,
- &atmel_spi1_spi_clk,
- &atmel_twi0_pclk,
- &atmel_mci0_pclk,
-#if defined(CONFIG_CPU_AT32AP7000) || defined(CONFIG_CPU_AT32AP7002)
- &atmel_lcdfb0_hclk,
- &atmel_lcdfb0_pixclk,
-#endif
- &ssc0_pclk,
- &ssc1_pclk,
- &ssc2_pclk,
- &usba0_hclk,
- &usba0_pclk,
- &atmel_ac97c0_pclk,
- &abdac0_pclk,
- &abdac0_sample_clk,
- &gclk0,
- &gclk1,
- &gclk2,
- &gclk3,
- &gclk4,
-};
-
-void __init setup_platform(void)
-{
- u32 cpu_mask = 0, hsb_mask = 0, pba_mask = 0, pbb_mask = 0;
- int i;
-
- if (pm_readl(MCCTRL) & PM_BIT(PLLSEL)) {
- main_clock = &pll0;
- cpu_clk.parent = &pll0;
- } else {
- main_clock = &osc0;
- cpu_clk.parent = &osc0;
- }
-
- if (pm_readl(PLL0) & PM_BIT(PLLOSC))
- pll0.parent = &osc1;
- if (pm_readl(PLL1) & PM_BIT(PLLOSC))
- pll1.parent = &osc1;
-
- genclk_init_parent(&gclk0);
- genclk_init_parent(&gclk1);
- genclk_init_parent(&gclk2);
- genclk_init_parent(&gclk3);
- genclk_init_parent(&gclk4);
-#if defined(CONFIG_CPU_AT32AP7000) || defined(CONFIG_CPU_AT32AP7002)
- genclk_init_parent(&atmel_lcdfb0_pixclk);
-#endif
- genclk_init_parent(&abdac0_sample_clk);
-
- /*
- * Build initial dynamic clock list by registering all clocks
- * from the array.
- * At the same time, turn on all clocks that have at least one
- * user already, and turn off everything else. We only do this
- * for module clocks, and even though it isn't particularly
- * pretty to check the address of the mode function, it should
- * do the trick...
- */
- for (i = 0; i < ARRAY_SIZE(init_clocks); i++) {
- struct clk *clk = init_clocks[i];
-
- /* first, register clock */
- at32_clk_register(clk);
-
- if (clk->users == 0)
- continue;
-
- if (clk->mode == &cpu_clk_mode)
- cpu_mask |= 1 << clk->index;
- else if (clk->mode == &hsb_clk_mode)
- hsb_mask |= 1 << clk->index;
- else if (clk->mode == &pba_clk_mode)
- pba_mask |= 1 << clk->index;
- else if (clk->mode == &pbb_clk_mode)
- pbb_mask |= 1 << clk->index;
- }
-
- pm_writel(CPU_MASK, cpu_mask);
- pm_writel(HSB_MASK, hsb_mask);
- pm_writel(PBA_MASK, pba_mask);
- pm_writel(PBB_MASK, pbb_mask);
-
- /* Initialize the port muxes */
- at32_init_pio(&pio0_device);
- at32_init_pio(&pio1_device);
- at32_init_pio(&pio2_device);
- at32_init_pio(&pio3_device);
- at32_init_pio(&pio4_device);
-}
-
-struct gen_pool *sram_pool;
-
-static int __init sram_init(void)
-{
- struct gen_pool *pool;
-
- /* 1KiB granularity */
- pool = gen_pool_create(10, -1);
- if (!pool)
- goto fail;
-
- if (gen_pool_add(pool, 0x24000000, 0x8000, -1))
- goto err_pool_add;
-
- sram_pool = pool;
- return 0;
-
-err_pool_add:
- gen_pool_destroy(pool);
-fail:
- pr_err("Failed to create SRAM pool\n");
- return -ENOMEM;
-}
-core_initcall(sram_init);
diff --git a/arch/avr32/mach-at32ap/clock.c b/arch/avr32/mach-at32ap/clock.c
deleted file mode 100644
index fdf1cae..0000000
--- a/arch/avr32/mach-at32ap/clock.c
+++ /dev/null
@@ -1,334 +0,0 @@
-/*
- * Clock management for AT32AP CPUs
- *
- * Copyright (C) 2006 Atmel Corporation
- *
- * Based on arch/arm/mach-at91/clock.c
- * Copyright (C) 2005 David Brownell
- * Copyright (C) 2005 Ivan Kokshaysky
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/clk.h>
-#include <linux/err.h>
-#include <linux/export.h>
-#include <linux/device.h>
-#include <linux/string.h>
-#include <linux/list.h>
-
-#include <mach/chip.h>
-
-#include "clock.h"
-
-/* at32 clock list */
-static LIST_HEAD(at32_clock_list);
-
-static DEFINE_SPINLOCK(clk_lock);
-static DEFINE_SPINLOCK(clk_list_lock);
-
-void at32_clk_register(struct clk *clk)
-{
- spin_lock(&clk_list_lock);
- /* add the new item to the end of the list */
- list_add_tail(&clk->list, &at32_clock_list);
- spin_unlock(&clk_list_lock);
-}
-
-static struct clk *__clk_get(struct device *dev, const char *id)
-{
- struct clk *clk;
-
- list_for_each_entry(clk, &at32_clock_list, list) {
- if (clk->dev == dev && strcmp(id, clk->name) == 0) {
- return clk;
- }
- }
-
- return ERR_PTR(-ENOENT);
-}
-
-struct clk *clk_get(struct device *dev, const char *id)
-{
- struct clk *clk;
-
- spin_lock(&clk_list_lock);
- clk = __clk_get(dev, id);
- spin_unlock(&clk_list_lock);
-
- return clk;
-}
-
-EXPORT_SYMBOL(clk_get);
-
-void clk_put(struct clk *clk)
-{
- /* clocks are static for now, we can't free them */
-}
-EXPORT_SYMBOL(clk_put);
-
-static void __clk_enable(struct clk *clk)
-{
- if (clk->parent)
- __clk_enable(clk->parent);
- if (clk->users++ == 0 && clk->mode)
- clk->mode(clk, 1);
-}
-
-int clk_enable(struct clk *clk)
-{
- unsigned long flags;
-
- if (!clk)
- return 0;
-
- spin_lock_irqsave(&clk_lock, flags);
- __clk_enable(clk);
- spin_unlock_irqrestore(&clk_lock, flags);
-
- return 0;
-}
-EXPORT_SYMBOL(clk_enable);
-
-static void __clk_disable(struct clk *clk)
-{
- if (clk->users == 0) {
- printk(KERN_ERR "%s: mismatched disable\n", clk->name);
- WARN_ON(1);
- return;
- }
-
- if (--clk->users == 0 && clk->mode)
- clk->mode(clk, 0);
- if (clk->parent)
- __clk_disable(clk->parent);
-}
-
-void clk_disable(struct clk *clk)
-{
- unsigned long flags;
-
- if (IS_ERR_OR_NULL(clk))
- return;
-
- spin_lock_irqsave(&clk_lock, flags);
- __clk_disable(clk);
- spin_unlock_irqrestore(&clk_lock, flags);
-}
-EXPORT_SYMBOL(clk_disable);
-
-unsigned long clk_get_rate(struct clk *clk)
-{
- unsigned long flags;
- unsigned long rate;
-
- if (!clk)
- return 0;
-
- spin_lock_irqsave(&clk_lock, flags);
- rate = clk->get_rate(clk);
- spin_unlock_irqrestore(&clk_lock, flags);
-
- return rate;
-}
-EXPORT_SYMBOL(clk_get_rate);
-
-long clk_round_rate(struct clk *clk, unsigned long rate)
-{
- unsigned long flags, actual_rate;
-
- if (!clk)
- return 0;
-
- if (!clk->set_rate)
- return -ENOSYS;
-
- spin_lock_irqsave(&clk_lock, flags);
- actual_rate = clk->set_rate(clk, rate, 0);
- spin_unlock_irqrestore(&clk_lock, flags);
-
- return actual_rate;
-}
-EXPORT_SYMBOL(clk_round_rate);
-
-int clk_set_rate(struct clk *clk, unsigned long rate)
-{
- unsigned long flags;
- long ret;
-
- if (!clk)
- return 0;
-
- if (!clk->set_rate)
- return -ENOSYS;
-
- spin_lock_irqsave(&clk_lock, flags);
- ret = clk->set_rate(clk, rate, 1);
- spin_unlock_irqrestore(&clk_lock, flags);
-
- return (ret < 0) ? ret : 0;
-}
-EXPORT_SYMBOL(clk_set_rate);
-
-int clk_set_parent(struct clk *clk, struct clk *parent)
-{
- unsigned long flags;
- int ret;
-
- if (!clk)
- return 0;
-
- if (!clk->set_parent)
- return -ENOSYS;
-
- spin_lock_irqsave(&clk_lock, flags);
- ret = clk->set_parent(clk, parent);
- spin_unlock_irqrestore(&clk_lock, flags);
-
- return ret;
-}
-EXPORT_SYMBOL(clk_set_parent);
-
-struct clk *clk_get_parent(struct clk *clk)
-{
- return !clk ? NULL : clk->parent;
-}
-EXPORT_SYMBOL(clk_get_parent);
-
-
-
-#ifdef CONFIG_DEBUG_FS
-
-/* /sys/kernel/debug/at32ap_clk */
-
-#include <linux/io.h>
-#include <linux/debugfs.h>
-#include <linux/seq_file.h>
-#include "pm.h"
-
-
-#define NEST_DELTA 2
-#define NEST_MAX 6
-
-struct clkinf {
- struct seq_file *s;
- unsigned nest;
-};
-
-static void
-dump_clock(struct clk *parent, struct clkinf *r)
-{
- unsigned nest = r->nest;
- char buf[16 + NEST_MAX];
- struct clk *clk;
- unsigned i;
-
- /* skip clocks coupled to devices that aren't registered */
- if (parent->dev && !dev_name(parent->dev) && !parent->users)
- return;
-
- /* <nest spaces> name <pad to end> */
- memset(buf, ' ', sizeof(buf) - 1);
- buf[sizeof(buf) - 1] = 0;
- i = strlen(parent->name);
- memcpy(buf + nest, parent->name,
- min(i, (unsigned)(sizeof(buf) - 1 - nest)));
-
- seq_printf(r->s, "%s%c users=%2d %-3s %9ld Hz",
- buf, parent->set_parent ? '*' : ' ',
- parent->users,
- parent->users ? "on" : "off", /* NOTE: not-paranoid!! */
- clk_get_rate(parent));
- if (parent->dev)
- seq_printf(r->s, ", for %s", dev_name(parent->dev));
- seq_putc(r->s, '\n');
-
- /* cost of this scan is small, but not linear... */
- r->nest = nest + NEST_DELTA;
-
- list_for_each_entry(clk, &at32_clock_list, list) {
- if (clk->parent == parent)
- dump_clock(clk, r);
- }
- r->nest = nest;
-}
-
-static int clk_show(struct seq_file *s, void *unused)
-{
- struct clkinf r;
- int i;
- struct clk *clk;
-
- /* show all the power manager registers */
- seq_printf(s,
- "MCCTRL = %8x\n"
- "CKSEL = %8x\n"
- "CPUMASK = %8x\n"
- "HSBMASK = %8x\n"
- "PBAMASK = %8x\n"
- "PBBMASK = %8x\n"
- "PLL0 = %8x\n"
- "PLL1 = %8x\n"
- "IMR = %8x\n",
- pm_readl(MCCTRL),
- pm_readl(CKSEL),
- pm_readl(CPU_MASK),
- pm_readl(HSB_MASK),
- pm_readl(PBA_MASK),
- pm_readl(PBB_MASK),
- pm_readl(PLL0),
- pm_readl(PLL1),
- pm_readl(IMR));
- for (i = 0; i < 8; i++) {
- if (i == 5)
- continue;
- seq_printf(s, "GCCTRL%d = %8x\n", i, pm_readl(GCCTRL(i)));
- }
-
- seq_putc(s, '\n');
- r.s = s;
- r.nest = 0;
- /* protected from changes on the list while dumping */
- spin_lock(&clk_list_lock);
-
- /* show clock tree as derived from the three oscillators */
- clk = __clk_get(NULL, "osc32k");
- dump_clock(clk, &r);
- clk_put(clk);
-
- clk = __clk_get(NULL, "osc0");
- dump_clock(clk, &r);
- clk_put(clk);
-
- clk = __clk_get(NULL, "osc1");
- dump_clock(clk, &r);
- clk_put(clk);
-
- spin_unlock(&clk_list_lock);
-
- return 0;
-}
-
-static int clk_open(struct inode *inode, struct file *file)
-{
- return single_open(file, clk_show, NULL);
-}
-
-static const struct file_operations clk_operations = {
- .open = clk_open,
- .read = seq_read,
- .llseek = seq_lseek,
- .release = single_release,
-};
-
-static int __init clk_debugfs_init(void)
-{
- (void) debugfs_create_file("at32ap_clk", S_IFREG | S_IRUGO,
- NULL, NULL, &clk_operations);
-
- return 0;
-}
-postcore_initcall(clk_debugfs_init);
-
-#endif
diff --git a/arch/avr32/mach-at32ap/clock.h b/arch/avr32/mach-at32ap/clock.h
deleted file mode 100644
index 4c7ebbd..0000000
--- a/arch/avr32/mach-at32ap/clock.h
+++ /dev/null
@@ -1,35 +0,0 @@
-/*
- * Clock management for AT32AP CPUs
- *
- * Copyright (C) 2006 Atmel Corporation
- *
- * Based on arch/arm/mach-at91/clock.c
- * Copyright (C) 2005 David Brownell
- * Copyright (C) 2005 Ivan Kokshaysky
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/clk.h>
-#include <linux/list.h>
-
-
-void at32_clk_register(struct clk *clk);
-
-struct clk {
- struct list_head list; /* linking element */
- const char *name; /* Clock name/function */
- struct device *dev; /* Device the clock is used by */
- struct clk *parent; /* Parent clock, if any */
- void (*mode)(struct clk *clk, int enabled);
- unsigned long (*get_rate)(struct clk *clk);
- long (*set_rate)(struct clk *clk, unsigned long rate,
- int apply);
- int (*set_parent)(struct clk *clk, struct clk *parent);
- u16 users; /* Enabled if non-zero */
- u16 index; /* Sibling index */
-};
-
-unsigned long pba_clk_get_rate(struct clk *clk);
-void pba_clk_mode(struct clk *clk, int enabled);
diff --git a/arch/avr32/mach-at32ap/extint.c b/arch/avr32/mach-at32ap/extint.c
deleted file mode 100644
index 96cabad..0000000
--- a/arch/avr32/mach-at32ap/extint.c
+++ /dev/null
@@ -1,271 +0,0 @@
-/*
- * External interrupt handling for AT32AP CPUs
- *
- * Copyright (C) 2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/errno.h>
-#include <linux/init.h>
-#include <linux/interrupt.h>
-#include <linux/irq.h>
-#include <linux/platform_device.h>
-#include <linux/random.h>
-#include <linux/slab.h>
-
-#include <asm/io.h>
-
-/* EIC register offsets */
-#define EIC_IER 0x0000
-#define EIC_IDR 0x0004
-#define EIC_IMR 0x0008
-#define EIC_ISR 0x000c
-#define EIC_ICR 0x0010
-#define EIC_MODE 0x0014
-#define EIC_EDGE 0x0018
-#define EIC_LEVEL 0x001c
-#define EIC_NMIC 0x0024
-
-/* Bitfields in NMIC */
-#define EIC_NMIC_ENABLE (1 << 0)
-
-/* Bit manipulation macros */
-#define EIC_BIT(name) \
- (1 << EIC_##name##_OFFSET)
-#define EIC_BF(name,value) \
- (((value) & ((1 << EIC_##name##_SIZE) - 1)) \
- << EIC_##name##_OFFSET)
-#define EIC_BFEXT(name,value) \
- (((value) >> EIC_##name##_OFFSET) \
- & ((1 << EIC_##name##_SIZE) - 1))
-#define EIC_BFINS(name,value,old) \
- (((old) & ~(((1 << EIC_##name##_SIZE) - 1) \
- << EIC_##name##_OFFSET)) \
- | EIC_BF(name,value))
-
-/* Register access macros */
-#define eic_readl(port,reg) \
- __raw_readl((port)->regs + EIC_##reg)
-#define eic_writel(port,reg,value) \
- __raw_writel((value), (port)->regs + EIC_##reg)
-
-struct eic {
- void __iomem *regs;
- struct irq_chip *chip;
- unsigned int first_irq;
-};
-
-static struct eic *nmi_eic;
-static bool nmi_enabled;
-
-static void eic_ack_irq(struct irq_data *d)
-{
- struct eic *eic = irq_data_get_irq_chip_data(d);
- eic_writel(eic, ICR, 1 << (d->irq - eic->first_irq));
-}
-
-static void eic_mask_irq(struct irq_data *d)
-{
- struct eic *eic = irq_data_get_irq_chip_data(d);
- eic_writel(eic, IDR, 1 << (d->irq - eic->first_irq));
-}
-
-static void eic_mask_ack_irq(struct irq_data *d)
-{
- struct eic *eic = irq_data_get_irq_chip_data(d);
- eic_writel(eic, ICR, 1 << (d->irq - eic->first_irq));
- eic_writel(eic, IDR, 1 << (d->irq - eic->first_irq));
-}
-
-static void eic_unmask_irq(struct irq_data *d)
-{
- struct eic *eic = irq_data_get_irq_chip_data(d);
- eic_writel(eic, IER, 1 << (d->irq - eic->first_irq));
-}
-
-static int eic_set_irq_type(struct irq_data *d, unsigned int flow_type)
-{
- struct eic *eic = irq_data_get_irq_chip_data(d);
- unsigned int irq = d->irq;
- unsigned int i = irq - eic->first_irq;
- u32 mode, edge, level;
-
- flow_type &= IRQ_TYPE_SENSE_MASK;
- if (flow_type == IRQ_TYPE_NONE)
- flow_type = IRQ_TYPE_LEVEL_LOW;
-
- mode = eic_readl(eic, MODE);
- edge = eic_readl(eic, EDGE);
- level = eic_readl(eic, LEVEL);
-
- switch (flow_type) {
- case IRQ_TYPE_LEVEL_LOW:
- mode |= 1 << i;
- level &= ~(1 << i);
- break;
- case IRQ_TYPE_LEVEL_HIGH:
- mode |= 1 << i;
- level |= 1 << i;
- break;
- case IRQ_TYPE_EDGE_RISING:
- mode &= ~(1 << i);
- edge |= 1 << i;
- break;
- case IRQ_TYPE_EDGE_FALLING:
- mode &= ~(1 << i);
- edge &= ~(1 << i);
- break;
- default:
- return -EINVAL;
- }
-
- eic_writel(eic, MODE, mode);
- eic_writel(eic, EDGE, edge);
- eic_writel(eic, LEVEL, level);
-
- irqd_set_trigger_type(d, flow_type);
- if (flow_type & (IRQ_TYPE_LEVEL_LOW | IRQ_TYPE_LEVEL_HIGH))
- irq_set_handler_locked(d, handle_level_irq);
- else
- irq_set_handler_locked(d, handle_edge_irq);
-
- return IRQ_SET_MASK_OK_NOCOPY;
-}
-
-static struct irq_chip eic_chip = {
- .name = "eic",
- .irq_ack = eic_ack_irq,
- .irq_mask = eic_mask_irq,
- .irq_mask_ack = eic_mask_ack_irq,
- .irq_unmask = eic_unmask_irq,
- .irq_set_type = eic_set_irq_type,
-};
-
-static void demux_eic_irq(struct irq_desc *desc)
-{
- struct eic *eic = irq_desc_get_handler_data(desc);
- unsigned long status, pending;
- unsigned int i;
-
- status = eic_readl(eic, ISR);
- pending = status & eic_readl(eic, IMR);
-
- while (pending) {
- i = fls(pending) - 1;
- pending &= ~(1 << i);
-
- generic_handle_irq(i + eic->first_irq);
- }
-}
-
-int nmi_enable(void)
-{
- nmi_enabled = true;
-
- if (nmi_eic)
- eic_writel(nmi_eic, NMIC, EIC_NMIC_ENABLE);
-
- return 0;
-}
-
-void nmi_disable(void)
-{
- if (nmi_eic)
- eic_writel(nmi_eic, NMIC, 0);
-
- nmi_enabled = false;
-}
-
-static int __init eic_probe(struct platform_device *pdev)
-{
- struct eic *eic;
- struct resource *regs;
- unsigned int i;
- unsigned int nr_of_irqs;
- unsigned int int_irq;
- int ret;
- u32 pattern;
-
- regs = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- int_irq = platform_get_irq(pdev, 0);
- if (!regs || (int)int_irq <= 0) {
- dev_dbg(&pdev->dev, "missing regs and/or irq resource\n");
- return -ENXIO;
- }
-
- ret = -ENOMEM;
- eic = kzalloc(sizeof(struct eic), GFP_KERNEL);
- if (!eic) {
- dev_dbg(&pdev->dev, "no memory for eic structure\n");
- goto err_kzalloc;
- }
-
- eic->first_irq = EIM_IRQ_BASE + 32 * pdev->id;
- eic->regs = ioremap(regs->start, resource_size(regs));
- if (!eic->regs) {
- dev_dbg(&pdev->dev, "failed to map regs\n");
- goto err_ioremap;
- }
-
- /*
- * Find out how many interrupt lines that are actually
- * implemented in hardware.
- */
- eic_writel(eic, IDR, ~0UL);
- eic_writel(eic, MODE, ~0UL);
- pattern = eic_readl(eic, MODE);
- nr_of_irqs = fls(pattern);
-
- /* Trigger on low level unless overridden by driver */
- eic_writel(eic, EDGE, 0UL);
- eic_writel(eic, LEVEL, 0UL);
-
- eic->chip = &eic_chip;
-
- for (i = 0; i < nr_of_irqs; i++) {
- irq_set_chip_and_handler(eic->first_irq + i, &eic_chip,
- handle_level_irq);
- irq_set_chip_data(eic->first_irq + i, eic);
- }
-
- irq_set_chained_handler_and_data(int_irq, demux_eic_irq, eic);
-
- if (pdev->id == 0) {
- nmi_eic = eic;
- if (nmi_enabled)
- /*
- * Someone tried to enable NMI before we were
- * ready. Do it now.
- */
- nmi_enable();
- }
-
- dev_info(&pdev->dev,
- "External Interrupt Controller at 0x%p, IRQ %u\n",
- eic->regs, int_irq);
- dev_info(&pdev->dev,
- "Handling %u external IRQs, starting with IRQ %u\n",
- nr_of_irqs, eic->first_irq);
-
- return 0;
-
-err_ioremap:
- kfree(eic);
-err_kzalloc:
- return ret;
-}
-
-static struct platform_driver eic_driver = {
- .driver = {
- .name = "at32_eic",
- },
-};
-
-static int __init eic_init(void)
-{
- return platform_driver_probe(&eic_driver, eic_probe);
-}
-arch_initcall(eic_init);
diff --git a/arch/avr32/mach-at32ap/hmatrix.c b/arch/avr32/mach-at32ap/hmatrix.c
deleted file mode 100644
index 48f5ede..0000000
--- a/arch/avr32/mach-at32ap/hmatrix.c
+++ /dev/null
@@ -1,88 +0,0 @@
-/*
- * High-Speed Bus Matrix helper functions
- *
- * Copyright (C) 2008 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/clk.h>
-#include <linux/io.h>
-
-#include <mach/chip.h>
-#include <mach/hmatrix.h>
-
-static inline void __hmatrix_write_reg(unsigned long offset, u32 value)
-{
- __raw_writel(value, (void __iomem __force *)(HMATRIX_BASE + offset));
-}
-
-static inline u32 __hmatrix_read_reg(unsigned long offset)
-{
- return __raw_readl((void __iomem __force *)(HMATRIX_BASE + offset));
-}
-
-/**
- * hmatrix_write_reg - write HMATRIX configuration register
- * @offset: register offset
- * @value: value to be written to the register at @offset
- */
-void hmatrix_write_reg(unsigned long offset, u32 value)
-{
- clk_enable(&at32_hmatrix_clk);
- __hmatrix_write_reg(offset, value);
- __hmatrix_read_reg(offset);
- clk_disable(&at32_hmatrix_clk);
-}
-
-/**
- * hmatrix_read_reg - read HMATRIX configuration register
- * @offset: register offset
- *
- * Returns the value of the register at @offset.
- */
-u32 hmatrix_read_reg(unsigned long offset)
-{
- u32 value;
-
- clk_enable(&at32_hmatrix_clk);
- value = __hmatrix_read_reg(offset);
- clk_disable(&at32_hmatrix_clk);
-
- return value;
-}
-
-/**
- * hmatrix_sfr_set_bits - set bits in a slave's Special Function Register
- * @slave_id: operate on the SFR belonging to this slave
- * @mask: mask of bits to be set in the SFR
- */
-void hmatrix_sfr_set_bits(unsigned int slave_id, u32 mask)
-{
- u32 value;
-
- clk_enable(&at32_hmatrix_clk);
- value = __hmatrix_read_reg(HMATRIX_SFR(slave_id));
- value |= mask;
- __hmatrix_write_reg(HMATRIX_SFR(slave_id), value);
- __hmatrix_read_reg(HMATRIX_SFR(slave_id));
- clk_disable(&at32_hmatrix_clk);
-}
-
-/**
- * hmatrix_sfr_set_bits - clear bits in a slave's Special Function Register
- * @slave_id: operate on the SFR belonging to this slave
- * @mask: mask of bits to be cleared in the SFR
- */
-void hmatrix_sfr_clear_bits(unsigned int slave_id, u32 mask)
-{
- u32 value;
-
- clk_enable(&at32_hmatrix_clk);
- value = __hmatrix_read_reg(HMATRIX_SFR(slave_id));
- value &= ~mask;
- __hmatrix_write_reg(HMATRIX_SFR(slave_id), value);
- __hmatrix_read_reg(HMATRIX_SFR(slave_id));
- clk_disable(&at32_hmatrix_clk);
-}
diff --git a/arch/avr32/mach-at32ap/hsmc.c b/arch/avr32/mach-at32ap/hsmc.c
deleted file mode 100644
index f66245e..0000000
--- a/arch/avr32/mach-at32ap/hsmc.c
+++ /dev/null
@@ -1,282 +0,0 @@
-/*
- * Static Memory Controller for AT32 chips
- *
- * Copyright (C) 2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/clk.h>
-#include <linux/err.h>
-#include <linux/init.h>
-#include <linux/module.h>
-#include <linux/platform_device.h>
-#include <linux/slab.h>
-
-#include <asm/io.h>
-#include <mach/smc.h>
-
-#include "hsmc.h"
-
-#define NR_CHIP_SELECTS 6
-
-struct hsmc {
- void __iomem *regs;
- struct clk *pclk;
- struct clk *mck;
-};
-
-static struct hsmc *hsmc;
-
-void smc_set_timing(struct smc_config *config,
- const struct smc_timing *timing)
-{
- int recover;
- int cycle;
-
- unsigned long mul;
-
- /* Reset all SMC timings */
- config->ncs_read_setup = 0;
- config->nrd_setup = 0;
- config->ncs_write_setup = 0;
- config->nwe_setup = 0;
- config->ncs_read_pulse = 0;
- config->nrd_pulse = 0;
- config->ncs_write_pulse = 0;
- config->nwe_pulse = 0;
- config->read_cycle = 0;
- config->write_cycle = 0;
-
- /*
- * cycles = x / T = x * f
- * = ((x * 1000000000) * ((f * 65536) / 1000000000)) / 65536
- * = ((x * 1000000000) * (((f / 10000) * 65536) / 100000)) / 65536
- */
- mul = (clk_get_rate(hsmc->mck) / 10000) << 16;
- mul /= 100000;
-
-#define ns2cyc(x) ((((x) * mul) + 65535) >> 16)
-
- if (timing->ncs_read_setup > 0)
- config->ncs_read_setup = ns2cyc(timing->ncs_read_setup);
-
- if (timing->nrd_setup > 0)
- config->nrd_setup = ns2cyc(timing->nrd_setup);
-
- if (timing->ncs_write_setup > 0)
- config->ncs_write_setup = ns2cyc(timing->ncs_write_setup);
-
- if (timing->nwe_setup > 0)
- config->nwe_setup = ns2cyc(timing->nwe_setup);
-
- if (timing->ncs_read_pulse > 0)
- config->ncs_read_pulse = ns2cyc(timing->ncs_read_pulse);
-
- if (timing->nrd_pulse > 0)
- config->nrd_pulse = ns2cyc(timing->nrd_pulse);
-
- if (timing->ncs_write_pulse > 0)
- config->ncs_write_pulse = ns2cyc(timing->ncs_write_pulse);
-
- if (timing->nwe_pulse > 0)
- config->nwe_pulse = ns2cyc(timing->nwe_pulse);
-
- if (timing->read_cycle > 0)
- config->read_cycle = ns2cyc(timing->read_cycle);
-
- if (timing->write_cycle > 0)
- config->write_cycle = ns2cyc(timing->write_cycle);
-
- /* Extend read cycle in needed */
- if (timing->ncs_read_recover > 0)
- recover = ns2cyc(timing->ncs_read_recover);
- else
- recover = 1;
-
- cycle = config->ncs_read_setup + config->ncs_read_pulse + recover;
-
- if (config->read_cycle < cycle)
- config->read_cycle = cycle;
-
- /* Extend read cycle in needed */
- if (timing->nrd_recover > 0)
- recover = ns2cyc(timing->nrd_recover);
- else
- recover = 1;
-
- cycle = config->nrd_setup + config->nrd_pulse + recover;
-
- if (config->read_cycle < cycle)
- config->read_cycle = cycle;
-
- /* Extend write cycle in needed */
- if (timing->ncs_write_recover > 0)
- recover = ns2cyc(timing->ncs_write_recover);
- else
- recover = 1;
-
- cycle = config->ncs_write_setup + config->ncs_write_pulse + recover;
-
- if (config->write_cycle < cycle)
- config->write_cycle = cycle;
-
- /* Extend write cycle in needed */
- if (timing->nwe_recover > 0)
- recover = ns2cyc(timing->nwe_recover);
- else
- recover = 1;
-
- cycle = config->nwe_setup + config->nwe_pulse + recover;
-
- if (config->write_cycle < cycle)
- config->write_cycle = cycle;
-}
-EXPORT_SYMBOL(smc_set_timing);
-
-int smc_set_configuration(int cs, const struct smc_config *config)
-{
- unsigned long offset;
- u32 setup, pulse, cycle, mode;
-
- if (!hsmc)
- return -ENODEV;
- if (cs >= NR_CHIP_SELECTS)
- return -EINVAL;
-
- setup = (HSMC_BF(NWE_SETUP, config->nwe_setup)
- | HSMC_BF(NCS_WR_SETUP, config->ncs_write_setup)
- | HSMC_BF(NRD_SETUP, config->nrd_setup)
- | HSMC_BF(NCS_RD_SETUP, config->ncs_read_setup));
- pulse = (HSMC_BF(NWE_PULSE, config->nwe_pulse)
- | HSMC_BF(NCS_WR_PULSE, config->ncs_write_pulse)
- | HSMC_BF(NRD_PULSE, config->nrd_pulse)
- | HSMC_BF(NCS_RD_PULSE, config->ncs_read_pulse));
- cycle = (HSMC_BF(NWE_CYCLE, config->write_cycle)
- | HSMC_BF(NRD_CYCLE, config->read_cycle));
-
- switch (config->bus_width) {
- case 1:
- mode = HSMC_BF(DBW, HSMC_DBW_8_BITS);
- break;
- case 2:
- mode = HSMC_BF(DBW, HSMC_DBW_16_BITS);
- break;
- case 4:
- mode = HSMC_BF(DBW, HSMC_DBW_32_BITS);
- break;
- default:
- return -EINVAL;
- }
-
- switch (config->nwait_mode) {
- case 0:
- mode |= HSMC_BF(EXNW_MODE, HSMC_EXNW_MODE_DISABLED);
- break;
- case 1:
- mode |= HSMC_BF(EXNW_MODE, HSMC_EXNW_MODE_RESERVED);
- break;
- case 2:
- mode |= HSMC_BF(EXNW_MODE, HSMC_EXNW_MODE_FROZEN);
- break;
- case 3:
- mode |= HSMC_BF(EXNW_MODE, HSMC_EXNW_MODE_READY);
- break;
- default:
- return -EINVAL;
- }
-
- if (config->tdf_cycles) {
- mode |= HSMC_BF(TDF_CYCLES, config->tdf_cycles);
- }
-
- if (config->nrd_controlled)
- mode |= HSMC_BIT(READ_MODE);
- if (config->nwe_controlled)
- mode |= HSMC_BIT(WRITE_MODE);
- if (config->byte_write)
- mode |= HSMC_BIT(BAT);
- if (config->tdf_mode)
- mode |= HSMC_BIT(TDF_MODE);
-
- pr_debug("smc cs%d: setup/%08x pulse/%08x cycle/%08x mode/%08x\n",
- cs, setup, pulse, cycle, mode);
-
- offset = cs * 0x10;
- hsmc_writel(hsmc, SETUP0 + offset, setup);
- hsmc_writel(hsmc, PULSE0 + offset, pulse);
- hsmc_writel(hsmc, CYCLE0 + offset, cycle);
- hsmc_writel(hsmc, MODE0 + offset, mode);
- hsmc_readl(hsmc, MODE0); /* I/O barrier */
-
- return 0;
-}
-EXPORT_SYMBOL(smc_set_configuration);
-
-static int hsmc_probe(struct platform_device *pdev)
-{
- struct resource *regs;
- struct clk *pclk, *mck;
- int ret;
-
- if (hsmc)
- return -EBUSY;
-
- regs = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!regs)
- return -ENXIO;
- pclk = clk_get(&pdev->dev, "pclk");
- if (IS_ERR(pclk))
- return PTR_ERR(pclk);
- mck = clk_get(&pdev->dev, "mck");
- if (IS_ERR(mck)) {
- ret = PTR_ERR(mck);
- goto out_put_pclk;
- }
-
- ret = -ENOMEM;
- hsmc = kzalloc(sizeof(struct hsmc), GFP_KERNEL);
- if (!hsmc)
- goto out_put_clocks;
-
- clk_enable(pclk);
- clk_enable(mck);
-
- hsmc->pclk = pclk;
- hsmc->mck = mck;
- hsmc->regs = ioremap(regs->start, resource_size(regs));
- if (!hsmc->regs)
- goto out_disable_clocks;
-
- dev_info(&pdev->dev, "Atmel Static Memory Controller at 0x%08lx\n",
- (unsigned long)regs->start);
-
- platform_set_drvdata(pdev, hsmc);
-
- return 0;
-
-out_disable_clocks:
- clk_disable(mck);
- clk_disable(pclk);
- kfree(hsmc);
-out_put_clocks:
- clk_put(mck);
-out_put_pclk:
- clk_put(pclk);
- hsmc = NULL;
- return ret;
-}
-
-static struct platform_driver hsmc_driver = {
- .probe = hsmc_probe,
- .driver = {
- .name = "smc",
- },
-};
-
-static int __init hsmc_init(void)
-{
- return platform_driver_register(&hsmc_driver);
-}
-core_initcall(hsmc_init);
diff --git a/arch/avr32/mach-at32ap/hsmc.h b/arch/avr32/mach-at32ap/hsmc.h
deleted file mode 100644
index d1d48e2..0000000
--- a/arch/avr32/mach-at32ap/hsmc.h
+++ /dev/null
@@ -1,127 +0,0 @@
-/*
- * Register definitions for Atmel Static Memory Controller (SMC)
- *
- * Copyright (C) 2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_HSMC_H__
-#define __ASM_AVR32_HSMC_H__
-
-/* HSMC register offsets */
-#define HSMC_SETUP0 0x0000
-#define HSMC_PULSE0 0x0004
-#define HSMC_CYCLE0 0x0008
-#define HSMC_MODE0 0x000c
-#define HSMC_SETUP1 0x0010
-#define HSMC_PULSE1 0x0014
-#define HSMC_CYCLE1 0x0018
-#define HSMC_MODE1 0x001c
-#define HSMC_SETUP2 0x0020
-#define HSMC_PULSE2 0x0024
-#define HSMC_CYCLE2 0x0028
-#define HSMC_MODE2 0x002c
-#define HSMC_SETUP3 0x0030
-#define HSMC_PULSE3 0x0034
-#define HSMC_CYCLE3 0x0038
-#define HSMC_MODE3 0x003c
-#define HSMC_SETUP4 0x0040
-#define HSMC_PULSE4 0x0044
-#define HSMC_CYCLE4 0x0048
-#define HSMC_MODE4 0x004c
-#define HSMC_SETUP5 0x0050
-#define HSMC_PULSE5 0x0054
-#define HSMC_CYCLE5 0x0058
-#define HSMC_MODE5 0x005c
-
-/* Bitfields in SETUP0 */
-#define HSMC_NWE_SETUP_OFFSET 0
-#define HSMC_NWE_SETUP_SIZE 6
-#define HSMC_NCS_WR_SETUP_OFFSET 8
-#define HSMC_NCS_WR_SETUP_SIZE 6
-#define HSMC_NRD_SETUP_OFFSET 16
-#define HSMC_NRD_SETUP_SIZE 6
-#define HSMC_NCS_RD_SETUP_OFFSET 24
-#define HSMC_NCS_RD_SETUP_SIZE 6
-
-/* Bitfields in PULSE0 */
-#define HSMC_NWE_PULSE_OFFSET 0
-#define HSMC_NWE_PULSE_SIZE 7
-#define HSMC_NCS_WR_PULSE_OFFSET 8
-#define HSMC_NCS_WR_PULSE_SIZE 7
-#define HSMC_NRD_PULSE_OFFSET 16
-#define HSMC_NRD_PULSE_SIZE 7
-#define HSMC_NCS_RD_PULSE_OFFSET 24
-#define HSMC_NCS_RD_PULSE_SIZE 7
-
-/* Bitfields in CYCLE0 */
-#define HSMC_NWE_CYCLE_OFFSET 0
-#define HSMC_NWE_CYCLE_SIZE 9
-#define HSMC_NRD_CYCLE_OFFSET 16
-#define HSMC_NRD_CYCLE_SIZE 9
-
-/* Bitfields in MODE0 */
-#define HSMC_READ_MODE_OFFSET 0
-#define HSMC_READ_MODE_SIZE 1
-#define HSMC_WRITE_MODE_OFFSET 1
-#define HSMC_WRITE_MODE_SIZE 1
-#define HSMC_EXNW_MODE_OFFSET 4
-#define HSMC_EXNW_MODE_SIZE 2
-#define HSMC_BAT_OFFSET 8
-#define HSMC_BAT_SIZE 1
-#define HSMC_DBW_OFFSET 12
-#define HSMC_DBW_SIZE 2
-#define HSMC_TDF_CYCLES_OFFSET 16
-#define HSMC_TDF_CYCLES_SIZE 4
-#define HSMC_TDF_MODE_OFFSET 20
-#define HSMC_TDF_MODE_SIZE 1
-#define HSMC_PMEN_OFFSET 24
-#define HSMC_PMEN_SIZE 1
-#define HSMC_PS_OFFSET 28
-#define HSMC_PS_SIZE 2
-
-/* Constants for READ_MODE */
-#define HSMC_READ_MODE_NCS_CONTROLLED 0
-#define HSMC_READ_MODE_NRD_CONTROLLED 1
-
-/* Constants for WRITE_MODE */
-#define HSMC_WRITE_MODE_NCS_CONTROLLED 0
-#define HSMC_WRITE_MODE_NWE_CONTROLLED 1
-
-/* Constants for EXNW_MODE */
-#define HSMC_EXNW_MODE_DISABLED 0
-#define HSMC_EXNW_MODE_RESERVED 1
-#define HSMC_EXNW_MODE_FROZEN 2
-#define HSMC_EXNW_MODE_READY 3
-
-/* Constants for BAT */
-#define HSMC_BAT_BYTE_SELECT 0
-#define HSMC_BAT_BYTE_WRITE 1
-
-/* Constants for DBW */
-#define HSMC_DBW_8_BITS 0
-#define HSMC_DBW_16_BITS 1
-#define HSMC_DBW_32_BITS 2
-
-/* Bit manipulation macros */
-#define HSMC_BIT(name) \
- (1 << HSMC_##name##_OFFSET)
-#define HSMC_BF(name,value) \
- (((value) & ((1 << HSMC_##name##_SIZE) - 1)) \
- << HSMC_##name##_OFFSET)
-#define HSMC_BFEXT(name,value) \
- (((value) >> HSMC_##name##_OFFSET) \
- & ((1 << HSMC_##name##_SIZE) - 1))
-#define HSMC_BFINS(name,value,old) \
- (((old) & ~(((1 << HSMC_##name##_SIZE) - 1) \
- << HSMC_##name##_OFFSET)) | HSMC_BF(name,value))
-
-/* Register access macros */
-#define hsmc_readl(port,reg) \
- __raw_readl((port)->regs + HSMC_##reg)
-#define hsmc_writel(port,reg,value) \
- __raw_writel((value), (port)->regs + HSMC_##reg)
-
-#endif /* __ASM_AVR32_HSMC_H__ */
diff --git a/arch/avr32/mach-at32ap/include/mach/at32ap700x.h b/arch/avr32/mach-at32ap/include/mach/at32ap700x.h
deleted file mode 100644
index b9222bf..0000000
--- a/arch/avr32/mach-at32ap/include/mach/at32ap700x.h
+++ /dev/null
@@ -1,245 +0,0 @@
-/*
- * Pin definitions for AT32AP7000.
- *
- * Copyright (C) 2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_ARCH_AT32AP700X_H__
-#define __ASM_ARCH_AT32AP700X_H__
-
-#define GPIO_PERIPH_A 0
-#define GPIO_PERIPH_B 1
-
-/*
- * Pin numbers identifying specific GPIO pins on the chip. They can
- * also be converted to IRQ numbers by passing them through
- * gpio_to_irq().
- */
-#define GPIO_PIOA_BASE (0)
-#define GPIO_PIOB_BASE (GPIO_PIOA_BASE + 32)
-#define GPIO_PIOC_BASE (GPIO_PIOB_BASE + 32)
-#define GPIO_PIOD_BASE (GPIO_PIOC_BASE + 32)
-#define GPIO_PIOE_BASE (GPIO_PIOD_BASE + 32)
-
-#define GPIO_PIN_PA(N) (GPIO_PIOA_BASE + (N))
-#define GPIO_PIN_PB(N) (GPIO_PIOB_BASE + (N))
-#define GPIO_PIN_PC(N) (GPIO_PIOC_BASE + (N))
-#define GPIO_PIN_PD(N) (GPIO_PIOD_BASE + (N))
-#define GPIO_PIN_PE(N) (GPIO_PIOE_BASE + (N))
-
-
-/*
- * DMAC peripheral hardware handshaking interfaces, used with dw_dmac
- */
-#define DMAC_MCI_RX 0
-#define DMAC_MCI_TX 1
-#define DMAC_DAC_TX 2
-#define DMAC_AC97_A_RX 3
-#define DMAC_AC97_A_TX 4
-#define DMAC_AC97_B_RX 5
-#define DMAC_AC97_B_TX 6
-#define DMAC_DMAREQ_0 7
-#define DMAC_DMAREQ_1 8
-#define DMAC_DMAREQ_2 9
-#define DMAC_DMAREQ_3 10
-
-/* HSB master IDs */
-#define HMATRIX_MASTER_CPU_DCACHE 0
-#define HMATRIX_MASTER_CPU_ICACHE 1
-#define HMATRIX_MASTER_PDC 2
-#define HMATRIX_MASTER_ISI 3
-#define HMATRIX_MASTER_USBA 4
-#define HMATRIX_MASTER_LCDC 5
-#define HMATRIX_MASTER_MACB0 6
-#define HMATRIX_MASTER_MACB1 7
-#define HMATRIX_MASTER_DMACA_M0 8
-#define HMATRIX_MASTER_DMACA_M1 9
-
-/* HSB slave IDs */
-#define HMATRIX_SLAVE_SRAM0 0
-#define HMATRIX_SLAVE_SRAM1 1
-#define HMATRIX_SLAVE_PBA 2
-#define HMATRIX_SLAVE_PBB 3
-#define HMATRIX_SLAVE_EBI 4
-#define HMATRIX_SLAVE_USBA 5
-#define HMATRIX_SLAVE_LCDC 6
-#define HMATRIX_SLAVE_DMACA 7
-
-/* Bits in HMATRIX SFR4 (EBI) */
-#define HMATRIX_EBI_SDRAM_ENABLE (1 << 1)
-#define HMATRIX_EBI_NAND_ENABLE (1 << 3)
-#define HMATRIX_EBI_CF0_ENABLE (1 << 4)
-#define HMATRIX_EBI_CF1_ENABLE (1 << 5)
-#define HMATRIX_EBI_PULLUP_DISABLE (1 << 8)
-
-/*
- * Base addresses of controllers that may be accessed early by
- * platform code.
- */
-#define PM_BASE 0xfff00000
-#define HMATRIX_BASE 0xfff00800
-#define SDRAMC_BASE 0xfff03800
-
-/* LCDC on port C */
-#define ATMEL_LCDC_PC_CC (1ULL << 19)
-#define ATMEL_LCDC_PC_HSYNC (1ULL << 20)
-#define ATMEL_LCDC_PC_PCLK (1ULL << 21)
-#define ATMEL_LCDC_PC_VSYNC (1ULL << 22)
-#define ATMEL_LCDC_PC_DVAL (1ULL << 23)
-#define ATMEL_LCDC_PC_MODE (1ULL << 24)
-#define ATMEL_LCDC_PC_PWR (1ULL << 25)
-#define ATMEL_LCDC_PC_DATA0 (1ULL << 26)
-#define ATMEL_LCDC_PC_DATA1 (1ULL << 27)
-#define ATMEL_LCDC_PC_DATA2 (1ULL << 28)
-#define ATMEL_LCDC_PC_DATA3 (1ULL << 29)
-#define ATMEL_LCDC_PC_DATA4 (1ULL << 30)
-#define ATMEL_LCDC_PC_DATA5 (1ULL << 31)
-
-/* LCDC on port D */
-#define ATMEL_LCDC_PD_DATA6 (1ULL << 0)
-#define ATMEL_LCDC_PD_DATA7 (1ULL << 1)
-#define ATMEL_LCDC_PD_DATA8 (1ULL << 2)
-#define ATMEL_LCDC_PD_DATA9 (1ULL << 3)
-#define ATMEL_LCDC_PD_DATA10 (1ULL << 4)
-#define ATMEL_LCDC_PD_DATA11 (1ULL << 5)
-#define ATMEL_LCDC_PD_DATA12 (1ULL << 6)
-#define ATMEL_LCDC_PD_DATA13 (1ULL << 7)
-#define ATMEL_LCDC_PD_DATA14 (1ULL << 8)
-#define ATMEL_LCDC_PD_DATA15 (1ULL << 9)
-#define ATMEL_LCDC_PD_DATA16 (1ULL << 10)
-#define ATMEL_LCDC_PD_DATA17 (1ULL << 11)
-#define ATMEL_LCDC_PD_DATA18 (1ULL << 12)
-#define ATMEL_LCDC_PD_DATA19 (1ULL << 13)
-#define ATMEL_LCDC_PD_DATA20 (1ULL << 14)
-#define ATMEL_LCDC_PD_DATA21 (1ULL << 15)
-#define ATMEL_LCDC_PD_DATA22 (1ULL << 16)
-#define ATMEL_LCDC_PD_DATA23 (1ULL << 17)
-
-/* LCDC on port E */
-#define ATMEL_LCDC_PE_CC (1ULL << (32 + 0))
-#define ATMEL_LCDC_PE_DVAL (1ULL << (32 + 1))
-#define ATMEL_LCDC_PE_MODE (1ULL << (32 + 2))
-#define ATMEL_LCDC_PE_DATA0 (1ULL << (32 + 3))
-#define ATMEL_LCDC_PE_DATA1 (1ULL << (32 + 4))
-#define ATMEL_LCDC_PE_DATA2 (1ULL << (32 + 5))
-#define ATMEL_LCDC_PE_DATA3 (1ULL << (32 + 6))
-#define ATMEL_LCDC_PE_DATA4 (1ULL << (32 + 7))
-#define ATMEL_LCDC_PE_DATA8 (1ULL << (32 + 8))
-#define ATMEL_LCDC_PE_DATA9 (1ULL << (32 + 9))
-#define ATMEL_LCDC_PE_DATA10 (1ULL << (32 + 10))
-#define ATMEL_LCDC_PE_DATA11 (1ULL << (32 + 11))
-#define ATMEL_LCDC_PE_DATA12 (1ULL << (32 + 12))
-#define ATMEL_LCDC_PE_DATA16 (1ULL << (32 + 13))
-#define ATMEL_LCDC_PE_DATA17 (1ULL << (32 + 14))
-#define ATMEL_LCDC_PE_DATA18 (1ULL << (32 + 15))
-#define ATMEL_LCDC_PE_DATA19 (1ULL << (32 + 16))
-#define ATMEL_LCDC_PE_DATA20 (1ULL << (32 + 17))
-#define ATMEL_LCDC_PE_DATA21 (1ULL << (32 + 18))
-
-
-#define ATMEL_LCDC(PORT, PIN) (ATMEL_LCDC_##PORT##_##PIN)
-
-
-#define ATMEL_LCDC_PRI_24B_DATA ( \
- ATMEL_LCDC(PC, DATA0) | ATMEL_LCDC(PC, DATA1) | \
- ATMEL_LCDC(PC, DATA2) | ATMEL_LCDC(PC, DATA3) | \
- ATMEL_LCDC(PC, DATA4) | ATMEL_LCDC(PC, DATA5) | \
- ATMEL_LCDC(PD, DATA6) | ATMEL_LCDC(PD, DATA7) | \
- ATMEL_LCDC(PD, DATA8) | ATMEL_LCDC(PD, DATA9) | \
- ATMEL_LCDC(PD, DATA10) | ATMEL_LCDC(PD, DATA11) | \
- ATMEL_LCDC(PD, DATA12) | ATMEL_LCDC(PD, DATA13) | \
- ATMEL_LCDC(PD, DATA14) | ATMEL_LCDC(PD, DATA15) | \
- ATMEL_LCDC(PD, DATA16) | ATMEL_LCDC(PD, DATA17) | \
- ATMEL_LCDC(PD, DATA18) | ATMEL_LCDC(PD, DATA19) | \
- ATMEL_LCDC(PD, DATA20) | ATMEL_LCDC(PD, DATA21) | \
- ATMEL_LCDC(PD, DATA22) | ATMEL_LCDC(PD, DATA23))
-
-#define ATMEL_LCDC_ALT_24B_DATA ( \
- ATMEL_LCDC(PE, DATA0) | ATMEL_LCDC(PE, DATA1) | \
- ATMEL_LCDC(PE, DATA2) | ATMEL_LCDC(PE, DATA3) | \
- ATMEL_LCDC(PE, DATA4) | ATMEL_LCDC(PC, DATA5) | \
- ATMEL_LCDC(PD, DATA6) | ATMEL_LCDC(PD, DATA7) | \
- ATMEL_LCDC(PE, DATA8) | ATMEL_LCDC(PE, DATA9) | \
- ATMEL_LCDC(PE, DATA10) | ATMEL_LCDC(PE, DATA11) | \
- ATMEL_LCDC(PE, DATA12) | ATMEL_LCDC(PD, DATA13) | \
- ATMEL_LCDC(PD, DATA14) | ATMEL_LCDC(PD, DATA15) | \
- ATMEL_LCDC(PE, DATA16) | ATMEL_LCDC(PE, DATA17) | \
- ATMEL_LCDC(PE, DATA18) | ATMEL_LCDC(PE, DATA19) | \
- ATMEL_LCDC(PE, DATA20) | ATMEL_LCDC(PE, DATA21) | \
- ATMEL_LCDC(PD, DATA22) | ATMEL_LCDC(PD, DATA23))
-
-#define ATMEL_LCDC_PRI_18B_DATA ( \
- ATMEL_LCDC(PC, DATA2) | ATMEL_LCDC(PC, DATA3) | \
- ATMEL_LCDC(PC, DATA4) | ATMEL_LCDC(PC, DATA5) | \
- ATMEL_LCDC(PD, DATA6) | ATMEL_LCDC(PD, DATA7) | \
- ATMEL_LCDC(PD, DATA10) | ATMEL_LCDC(PD, DATA11) | \
- ATMEL_LCDC(PD, DATA12) | ATMEL_LCDC(PD, DATA13) | \
- ATMEL_LCDC(PD, DATA14) | ATMEL_LCDC(PD, DATA15) | \
- ATMEL_LCDC(PD, DATA18) | ATMEL_LCDC(PD, DATA19) | \
- ATMEL_LCDC(PD, DATA20) | ATMEL_LCDC(PD, DATA21) | \
- ATMEL_LCDC(PD, DATA22) | ATMEL_LCDC(PD, DATA23))
-
-#define ATMEL_LCDC_ALT_18B_DATA ( \
- ATMEL_LCDC(PE, DATA2) | ATMEL_LCDC(PE, DATA3) | \
- ATMEL_LCDC(PE, DATA4) | ATMEL_LCDC(PC, DATA5) | \
- ATMEL_LCDC(PD, DATA6) | ATMEL_LCDC(PD, DATA7) | \
- ATMEL_LCDC(PE, DATA10) | ATMEL_LCDC(PE, DATA11) | \
- ATMEL_LCDC(PE, DATA12) | ATMEL_LCDC(PD, DATA13) | \
- ATMEL_LCDC(PD, DATA14) | ATMEL_LCDC(PD, DATA15) | \
- ATMEL_LCDC(PE, DATA18) | ATMEL_LCDC(PE, DATA19) | \
- ATMEL_LCDC(PE, DATA20) | ATMEL_LCDC(PE, DATA21) | \
- ATMEL_LCDC(PD, DATA22) | ATMEL_LCDC(PD, DATA23))
-
-#define ATMEL_LCDC_PRI_15B_DATA ( \
- ATMEL_LCDC(PC, DATA3) | ATMEL_LCDC(PC, DATA4) | \
- ATMEL_LCDC(PC, DATA5) | ATMEL_LCDC(PD, DATA6) | \
- ATMEL_LCDC(PD, DATA7) | \
- ATMEL_LCDC(PD, DATA11) | ATMEL_LCDC(PD, DATA12) | \
- ATMEL_LCDC(PD, DATA13) | ATMEL_LCDC(PD, DATA14) | \
- ATMEL_LCDC(PD, DATA15) | \
- ATMEL_LCDC(PD, DATA19) | ATMEL_LCDC(PD, DATA20) | \
- ATMEL_LCDC(PD, DATA21) | ATMEL_LCDC(PD, DATA22) | \
- ATMEL_LCDC(PD, DATA23))
-
-#define ATMEL_LCDC_ALT_15B_DATA ( \
- ATMEL_LCDC(PE, DATA3) | ATMEL_LCDC(PE, DATA4) | \
- ATMEL_LCDC(PC, DATA5) | ATMEL_LCDC(PD, DATA6) | \
- ATMEL_LCDC(PD, DATA7) | \
- ATMEL_LCDC(PE, DATA11) | ATMEL_LCDC(PE, DATA12) | \
- ATMEL_LCDC(PD, DATA13) | ATMEL_LCDC(PD, DATA14) | \
- ATMEL_LCDC(PD, DATA15) | \
- ATMEL_LCDC(PE, DATA19) | ATMEL_LCDC(PE, DATA20) | \
- ATMEL_LCDC(PE, DATA21) | ATMEL_LCDC(PD, DATA22) | \
- ATMEL_LCDC(PD, DATA23))
-
-#define ATMEL_LCDC_PRI_CONTROL ( \
- ATMEL_LCDC(PC, CC) | ATMEL_LCDC(PC, DVAL) | \
- ATMEL_LCDC(PC, MODE) | ATMEL_LCDC(PC, PWR))
-
-#define ATMEL_LCDC_ALT_CONTROL ( \
- ATMEL_LCDC(PE, CC) | ATMEL_LCDC(PE, DVAL) | \
- ATMEL_LCDC(PE, MODE) | ATMEL_LCDC(PC, PWR))
-
-#define ATMEL_LCDC_CONTROL ( \
- ATMEL_LCDC(PC, HSYNC) | ATMEL_LCDC(PC, VSYNC) | \
- ATMEL_LCDC(PC, PCLK))
-
-#define ATMEL_LCDC_PRI_24BIT (ATMEL_LCDC_CONTROL | ATMEL_LCDC_PRI_24B_DATA)
-
-#define ATMEL_LCDC_ALT_24BIT (ATMEL_LCDC_CONTROL | ATMEL_LCDC_ALT_24B_DATA)
-
-#define ATMEL_LCDC_PRI_18BIT (ATMEL_LCDC_CONTROL | ATMEL_LCDC_PRI_18B_DATA)
-
-#define ATMEL_LCDC_ALT_18BIT (ATMEL_LCDC_CONTROL | ATMEL_LCDC_ALT_18B_DATA)
-
-#define ATMEL_LCDC_PRI_15BIT (ATMEL_LCDC_CONTROL | ATMEL_LCDC_PRI_15B_DATA)
-
-#define ATMEL_LCDC_ALT_15BIT (ATMEL_LCDC_CONTROL | ATMEL_LCDC_ALT_15B_DATA)
-
-/* Bitmask for all EBI data (D16..D31) pins on port E */
-#define ATMEL_EBI_PE_DATA_ALL (0x0000FFFF)
-
-#endif /* __ASM_ARCH_AT32AP700X_H__ */
diff --git a/arch/avr32/mach-at32ap/include/mach/board.h b/arch/avr32/mach-at32ap/include/mach/board.h
deleted file mode 100644
index f1a316d..0000000
--- a/arch/avr32/mach-at32ap/include/mach/board.h
+++ /dev/null
@@ -1,115 +0,0 @@
-/*
- * Platform data definitions.
- */
-#ifndef __ASM_ARCH_BOARD_H
-#define __ASM_ARCH_BOARD_H
-
-#include <linux/types.h>
-#include <linux/serial.h>
-#include <linux/platform_data/macb.h>
-#include <linux/platform_data/atmel.h>
-
-#define GPIO_PIN_NONE (-1)
-
-/*
- * Clock rates for various on-board oscillators. The number of entries
- * in this array is chip-dependent.
- */
-extern unsigned long at32_board_osc_rates[];
-
-/*
- * This used to add essential system devices, but this is now done
- * automatically. Please don't use it in new board code.
- */
-static inline void __deprecated at32_add_system_devices(void)
-{
-
-}
-
-extern struct platform_device *atmel_default_console_device;
-
-/* Flags for selecting USART extra pins */
-#define ATMEL_USART_RTS 0x01
-#define ATMEL_USART_CTS 0x02
-#define ATMEL_USART_CLK 0x04
-
-void at32_map_usart(unsigned int hw_id, unsigned int line, int flags);
-struct platform_device *at32_add_device_usart(unsigned int id);
-
-struct platform_device *
-at32_add_device_eth(unsigned int id, struct macb_platform_data *data);
-
-struct spi_board_info;
-struct platform_device *
-at32_add_device_spi(unsigned int id, struct spi_board_info *b, unsigned int n);
-void at32_spi_setup_slaves(unsigned int bus_num, struct spi_board_info *b, unsigned int n);
-
-struct atmel_lcdfb_pdata;
-struct platform_device *
-at32_add_device_lcdc(unsigned int id, struct atmel_lcdfb_pdata *data,
- unsigned long fbmem_start, unsigned long fbmem_len,
- u64 pin_mask);
-
-struct usba_platform_data;
-struct platform_device *
-at32_add_device_usba(unsigned int id, struct usba_platform_data *data);
-
-struct ide_platform_data {
- u8 cs;
-};
-struct platform_device *
-at32_add_device_ide(unsigned int id, unsigned int extint,
- struct ide_platform_data *data);
-
-/* mask says which PWM channels to mux */
-struct platform_device *at32_add_device_pwm(u32 mask);
-
-/* depending on what's hooked up, not all SSC pins will be used */
-#define ATMEL_SSC_TK 0x01
-#define ATMEL_SSC_TF 0x02
-#define ATMEL_SSC_TD 0x04
-#define ATMEL_SSC_TX (ATMEL_SSC_TK | ATMEL_SSC_TF | ATMEL_SSC_TD)
-
-#define ATMEL_SSC_RK 0x10
-#define ATMEL_SSC_RF 0x20
-#define ATMEL_SSC_RD 0x40
-#define ATMEL_SSC_RX (ATMEL_SSC_RK | ATMEL_SSC_RF | ATMEL_SSC_RD)
-
-struct platform_device *
-at32_add_device_ssc(unsigned int id, unsigned int flags);
-
-struct i2c_board_info;
-struct platform_device *at32_add_device_twi(unsigned int id,
- struct i2c_board_info *b,
- unsigned int n);
-
-struct mci_platform_data;
-struct platform_device *
-at32_add_device_mci(unsigned int id, struct mci_platform_data *data);
-
-struct ac97c_platform_data;
-struct platform_device *
-at32_add_device_ac97c(unsigned int id, struct ac97c_platform_data *data,
- unsigned int flags);
-
-struct atmel_abdac_pdata;
-struct platform_device *
-at32_add_device_abdac(unsigned int id, struct atmel_abdac_pdata *data);
-
-struct platform_device *at32_add_device_psif(unsigned int id);
-
-struct cf_platform_data {
- int detect_pin;
- int reset_pin;
- int vcc_pin;
- int ready_pin;
- u8 cs;
-};
-struct platform_device *
-at32_add_device_cf(unsigned int id, unsigned int extint,
- struct cf_platform_data *data);
-
-struct platform_device *
-at32_add_device_nand(unsigned int id, struct atmel_nand_data *data);
-
-#endif /* __ASM_ARCH_BOARD_H */
diff --git a/arch/avr32/mach-at32ap/include/mach/chip.h b/arch/avr32/mach-at32ap/include/mach/chip.h
deleted file mode 100644
index 5efca6d..0000000
--- a/arch/avr32/mach-at32ap/include/mach/chip.h
+++ /dev/null
@@ -1,19 +0,0 @@
-/*
- * AVR32 chip-specific definitions
- *
- * Copyright (C) 2008 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_ARCH_CHIP_H__
-#define __ASM_AVR32_ARCH_CHIP_H__
-
-#if defined(CONFIG_CPU_AT32AP700X)
-# include <mach/at32ap700x.h>
-#else
-# error Unknown chip type selected
-#endif
-
-#endif /* __ASM_AVR32_ARCH_CHIP_H__ */
diff --git a/arch/avr32/mach-at32ap/include/mach/cpu.h b/arch/avr32/mach-at32ap/include/mach/cpu.h
deleted file mode 100644
index 4181086..0000000
--- a/arch/avr32/mach-at32ap/include/mach/cpu.h
+++ /dev/null
@@ -1,23 +0,0 @@
-/*
- * AVR32 CPU identification
- *
- * Copyright (C) 2007 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_ARCH_CPU_H
-#define __ASM_ARCH_CPU_H
-
-/*
- * Only AT32AP7000 is defined for now. We can identify the specific
- * chip at runtime, but I'm not sure if it's really worth it.
- */
-#ifdef CONFIG_CPU_AT32AP700X
-# define cpu_is_at32ap7000() (1)
-#else
-# define cpu_is_at32ap7000() (0)
-#endif
-
-#endif /* __ASM_ARCH_CPU_H */
diff --git a/arch/avr32/mach-at32ap/include/mach/gpio.h b/arch/avr32/mach-at32ap/include/mach/gpio.h
deleted file mode 100644
index 0180f58..0000000
--- a/arch/avr32/mach-at32ap/include/mach/gpio.h
+++ /dev/null
@@ -1,45 +0,0 @@
-#ifndef __ASM_AVR32_ARCH_GPIO_H
-#define __ASM_AVR32_ARCH_GPIO_H
-
-#include <linux/compiler.h>
-#include <asm/irq.h>
-
-
-/* Some GPIO chips can manage IRQs; some can't. The exact numbers can
- * be changed if needed, but for the moment they're not configurable.
- */
-#define ARCH_NR_GPIOS (NR_GPIO_IRQS + 2 * 32)
-
-
-/* Arch-neutral GPIO API, supporting both "native" and external GPIOs. */
-#include <asm-generic/gpio.h>
-
-static inline int gpio_get_value(unsigned int gpio)
-{
- return __gpio_get_value(gpio);
-}
-
-static inline void gpio_set_value(unsigned int gpio, int value)
-{
- __gpio_set_value(gpio, value);
-}
-
-static inline int gpio_cansleep(unsigned int gpio)
-{
- return __gpio_cansleep(gpio);
-}
-
-
-static inline int gpio_to_irq(unsigned int gpio)
-{
- if (gpio < NR_GPIO_IRQS)
- return gpio + GPIO_IRQ_BASE;
- return -EINVAL;
-}
-
-static inline int irq_to_gpio(unsigned int irq)
-{
- return irq - GPIO_IRQ_BASE;
-}
-
-#endif /* __ASM_AVR32_ARCH_GPIO_H */
diff --git a/arch/avr32/mach-at32ap/include/mach/hmatrix.h b/arch/avr32/mach-at32ap/include/mach/hmatrix.h
deleted file mode 100644
index 7a368f2..0000000
--- a/arch/avr32/mach-at32ap/include/mach/hmatrix.h
+++ /dev/null
@@ -1,55 +0,0 @@
-/*
- * High-Speed Bus Matrix configuration registers
- *
- * Copyright (C) 2008 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __HMATRIX_H
-#define __HMATRIX_H
-
-extern struct clk at32_hmatrix_clk;
-
-void hmatrix_write_reg(unsigned long offset, u32 value);
-u32 hmatrix_read_reg(unsigned long offset);
-
-void hmatrix_sfr_set_bits(unsigned int slave_id, u32 mask);
-void hmatrix_sfr_clear_bits(unsigned int slave_id, u32 mask);
-
-/* Master Configuration register */
-#define HMATRIX_MCFG(m) (0x0000 + 4 * (m))
-/* Undefined length burst limit */
-# define HMATRIX_MCFG_ULBT_INFINITE 0 /* Infinite length */
-# define HMATRIX_MCFG_ULBT_SINGLE 1 /* Single Access */
-# define HMATRIX_MCFG_ULBT_FOUR_BEAT 2 /* Four beat */
-# define HMATRIX_MCFG_ULBT_EIGHT_BEAT 3 /* Eight beat */
-# define HMATRIX_MCFG_ULBT_SIXTEEN_BEAT 4 /* Sixteen beat */
-
-/* Slave Configuration register */
-#define HMATRIX_SCFG(s) (0x0040 + 4 * (s))
-# define HMATRIX_SCFG_SLOT_CYCLE(x) ((x) << 0) /* Max burst cycles */
-# define HMATRIX_SCFG_DEFMSTR_NONE ( 0 << 16) /* No default master */
-# define HMATRIX_SCFG_DEFMSTR_LAST ( 1 << 16) /* Last def master */
-# define HMATRIX_SCFG_DEFMSTR_FIXED ( 2 << 16) /* Fixed def master */
-# define HMATRIX_SCFG_FIXED_DEFMSTR(m) ((m) << 18) /* Fixed master ID */
-# define HMATRIX_SCFG_ARBT_ROUND_ROBIN ( 0 << 24) /* RR arbitration */
-# define HMATRIX_SCFG_ARBT_FIXED_PRIO ( 1 << 24) /* Fixed priority */
-
-/* Slave Priority register A (master 0..7) */
-#define HMATRIX_PRAS(s) (0x0080 + 8 * (s))
-# define HMATRIX_PRAS_PRIO(m, p) ((p) << ((m) * 4))
-
-/* Slave Priority register A (master 8..15) */
-#define HMATRIX_PRBS(s) (0x0084 + 8 * (s))
-# define HMATRIX_PRBS_PRIO(m, p) ((p) << (((m) - 8) * 4))
-
-/* Master Remap Control Register */
-#define HMATRIX_MRCR 0x0100
-# define HMATRIX_MRCR_REMAP(m) ( 1 << (m)) /* Remap master m */
-
-/* Special Function Register. Bit definitions are chip-specific */
-#define HMATRIX_SFR(s) (0x0110 + 4 * (s))
-
-#endif /* __HMATRIX_H */
diff --git a/arch/avr32/mach-at32ap/include/mach/init.h b/arch/avr32/mach-at32ap/include/mach/init.h
deleted file mode 100644
index bc40e3d..0000000
--- a/arch/avr32/mach-at32ap/include/mach/init.h
+++ /dev/null
@@ -1,18 +0,0 @@
-/*
- * AT32AP platform initialization calls.
- *
- * Copyright (C) 2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_AT32AP_INIT_H__
-#define __ASM_AVR32_AT32AP_INIT_H__
-
-void setup_platform(void);
-void setup_board(void);
-
-void at32_setup_serial_console(unsigned int usart_id);
-
-#endif /* __ASM_AVR32_AT32AP_INIT_H__ */
diff --git a/arch/avr32/mach-at32ap/include/mach/io.h b/arch/avr32/mach-at32ap/include/mach/io.h
deleted file mode 100644
index 22ea79b..0000000
--- a/arch/avr32/mach-at32ap/include/mach/io.h
+++ /dev/null
@@ -1,38 +0,0 @@
-#ifndef __ASM_AVR32_ARCH_AT32AP_IO_H
-#define __ASM_AVR32_ARCH_AT32AP_IO_H
-
-#include <linux/swab.h>
-
-#if defined(CONFIG_AP700X_32_BIT_SMC)
-# define __swizzle_addr_b(addr) (addr ^ 3UL)
-# define __swizzle_addr_w(addr) (addr ^ 2UL)
-# define __swizzle_addr_l(addr) (addr)
-# define ioswabb(a, x) (x)
-# define ioswabw(a, x) (x)
-# define ioswabl(a, x) (x)
-# define __mem_ioswabb(a, x) (x)
-# define __mem_ioswabw(a, x) swab16(x)
-# define __mem_ioswabl(a, x) swab32(x)
-#elif defined(CONFIG_AP700X_16_BIT_SMC)
-# define __swizzle_addr_b(addr) (addr ^ 1UL)
-# define __swizzle_addr_w(addr) (addr)
-# define __swizzle_addr_l(addr) (addr)
-# define ioswabb(a, x) (x)
-# define ioswabw(a, x) (x)
-# define ioswabl(a, x) swahw32(x)
-# define __mem_ioswabb(a, x) (x)
-# define __mem_ioswabw(a, x) swab16(x)
-# define __mem_ioswabl(a, x) swahb32(x)
-#else
-# define __swizzle_addr_b(addr) (addr)
-# define __swizzle_addr_w(addr) (addr)
-# define __swizzle_addr_l(addr) (addr)
-# define ioswabb(a, x) (x)
-# define ioswabw(a, x) swab16(x)
-# define ioswabl(a, x) swab32(x)
-# define __mem_ioswabb(a, x) (x)
-# define __mem_ioswabw(a, x) (x)
-# define __mem_ioswabl(a, x) (x)
-#endif
-
-#endif /* __ASM_AVR32_ARCH_AT32AP_IO_H */
diff --git a/arch/avr32/mach-at32ap/include/mach/irq.h b/arch/avr32/mach-at32ap/include/mach/irq.h
deleted file mode 100644
index 608e350..0000000
--- a/arch/avr32/mach-at32ap/include/mach/irq.h
+++ /dev/null
@@ -1,14 +0,0 @@
-#ifndef __ASM_AVR32_ARCH_IRQ_H
-#define __ASM_AVR32_ARCH_IRQ_H
-
-#define EIM_IRQ_BASE NR_INTERNAL_IRQS
-#define NR_EIM_IRQS 32
-#define AT32_EXTINT(n) (EIM_IRQ_BASE + (n))
-
-#define GPIO_IRQ_BASE (EIM_IRQ_BASE + NR_EIM_IRQS)
-#define NR_GPIO_CTLR (5 /*internal*/ + 1 /*external*/)
-#define NR_GPIO_IRQS (NR_GPIO_CTLR * 32)
-
-#define NR_IRQS (GPIO_IRQ_BASE + NR_GPIO_IRQS)
-
-#endif /* __ASM_AVR32_ARCH_IRQ_H */
diff --git a/arch/avr32/mach-at32ap/include/mach/pm.h b/arch/avr32/mach-at32ap/include/mach/pm.h
deleted file mode 100644
index f29ff2c..0000000
--- a/arch/avr32/mach-at32ap/include/mach/pm.h
+++ /dev/null
@@ -1,27 +0,0 @@
-/*
- * AVR32 AP Power Management.
- *
- * Copyright (C) 2008 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_ARCH_PM_H
-#define __ASM_AVR32_ARCH_PM_H
-
-/* Possible arguments to the "sleep" instruction */
-#define CPU_SLEEP_IDLE 0
-#define CPU_SLEEP_FROZEN 1
-#define CPU_SLEEP_STANDBY 2
-#define CPU_SLEEP_STOP 3
-#define CPU_SLEEP_STATIC 5
-
-#ifndef __ASSEMBLY__
-extern void cpu_enter_idle(void);
-extern void cpu_enter_standby(unsigned long sdramc_base);
-
-void intc_set_suspend_handler(unsigned long offset);
-#endif
-
-#endif /* __ASM_AVR32_ARCH_PM_H */
diff --git a/arch/avr32/mach-at32ap/include/mach/portmux.h b/arch/avr32/mach-at32ap/include/mach/portmux.h
deleted file mode 100644
index 4873024..0000000
--- a/arch/avr32/mach-at32ap/include/mach/portmux.h
+++ /dev/null
@@ -1,30 +0,0 @@
-/*
- * AT32 portmux interface.
- *
- * Copyright (C) 2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_ARCH_PORTMUX_H__
-#define __ASM_ARCH_PORTMUX_H__
-
-/*
- * Set up pin multiplexing, called from board init only.
- *
- * The following flags determine the initial state of the pin.
- */
-#define AT32_GPIOF_PULLUP 0x00000001 /* (not-OUT) Enable pull-up */
-#define AT32_GPIOF_OUTPUT 0x00000002 /* (OUT) Enable output driver */
-#define AT32_GPIOF_HIGH 0x00000004 /* (OUT) Set output high */
-#define AT32_GPIOF_DEGLITCH 0x00000008 /* (IN) Filter glitches */
-#define AT32_GPIOF_MULTIDRV 0x00000010 /* Enable multidriver option */
-
-void at32_select_periph(unsigned int port, unsigned int pin,
- unsigned int periph, unsigned long flags);
-void at32_select_gpio(unsigned int pin, unsigned long flags);
-void at32_deselect_pin(unsigned int pin);
-void at32_reserve_pin(unsigned int port, u32 pin_mask);
-
-#endif /* __ASM_ARCH_PORTMUX_H__ */
diff --git a/arch/avr32/mach-at32ap/include/mach/smc.h b/arch/avr32/mach-at32ap/include/mach/smc.h
deleted file mode 100644
index c98eea4..0000000
--- a/arch/avr32/mach-at32ap/include/mach/smc.h
+++ /dev/null
@@ -1,113 +0,0 @@
-/*
- * Static Memory Controller for AT32 chips
- *
- * Copyright (C) 2006 Atmel Corporation
- *
- * Inspired by the OMAP2 General-Purpose Memory Controller interface
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ARCH_AT32AP_SMC_H
-#define __ARCH_AT32AP_SMC_H
-
-/*
- * All timing parameters are in nanoseconds.
- */
-struct smc_timing {
- /* Delay from address valid to assertion of given strobe */
- int ncs_read_setup;
- int nrd_setup;
- int ncs_write_setup;
- int nwe_setup;
-
- /* Pulse length of given strobe */
- int ncs_read_pulse;
- int nrd_pulse;
- int ncs_write_pulse;
- int nwe_pulse;
-
- /* Total cycle length of given operation */
- int read_cycle;
- int write_cycle;
-
- /* Minimal recovery times, will extend cycle if needed */
- int ncs_read_recover;
- int nrd_recover;
- int ncs_write_recover;
- int nwe_recover;
-};
-
-/*
- * All timing parameters are in clock cycles.
- */
-struct smc_config {
-
- /* Delay from address valid to assertion of given strobe */
- u8 ncs_read_setup;
- u8 nrd_setup;
- u8 ncs_write_setup;
- u8 nwe_setup;
-
- /* Pulse length of given strobe */
- u8 ncs_read_pulse;
- u8 nrd_pulse;
- u8 ncs_write_pulse;
- u8 nwe_pulse;
-
- /* Total cycle length of given operation */
- u8 read_cycle;
- u8 write_cycle;
-
- /* Bus width in bytes */
- u8 bus_width;
-
- /*
- * 0: Data is sampled on rising edge of NCS
- * 1: Data is sampled on rising edge of NRD
- */
- unsigned int nrd_controlled:1;
-
- /*
- * 0: Data is driven on falling edge of NCS
- * 1: Data is driven on falling edge of NWR
- */
- unsigned int nwe_controlled:1;
-
- /*
- * 0: NWAIT is disabled
- * 1: Reserved
- * 2: NWAIT is frozen mode
- * 3: NWAIT in ready mode
- */
- unsigned int nwait_mode:2;
-
- /*
- * 0: Byte select access type
- * 1: Byte write access type
- */
- unsigned int byte_write:1;
-
- /*
- * Number of clock cycles before data is released after
- * the rising edge of the read controlling signal
- *
- * Total cycles from SMC is tdf_cycles + 1
- */
- unsigned int tdf_cycles:4;
-
- /*
- * 0: TDF optimization disabled
- * 1: TDF optimization enabled
- */
- unsigned int tdf_mode:1;
-};
-
-extern void smc_set_timing(struct smc_config *config,
- const struct smc_timing *timing);
-
-extern int smc_set_configuration(int cs, const struct smc_config *config);
-extern struct smc_config *smc_get_configuration(int cs);
-
-#endif /* __ARCH_AT32AP_SMC_H */
diff --git a/arch/avr32/mach-at32ap/include/mach/sram.h b/arch/avr32/mach-at32ap/include/mach/sram.h
deleted file mode 100644
index 4838dae..0000000
--- a/arch/avr32/mach-at32ap/include/mach/sram.h
+++ /dev/null
@@ -1,30 +0,0 @@
-/*
- * Simple SRAM allocator
- *
- * Copyright (C) 2008 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ASM_AVR32_ARCH_SRAM_H
-#define __ASM_AVR32_ARCH_SRAM_H
-
-#include <linux/genalloc.h>
-
-extern struct gen_pool *sram_pool;
-
-static inline unsigned long sram_alloc(size_t len)
-{
- if (!sram_pool)
- return 0UL;
-
- return gen_pool_alloc(sram_pool, len);
-}
-
-static inline void sram_free(unsigned long addr, size_t len)
-{
- return gen_pool_free(sram_pool, addr, len);
-}
-
-#endif /* __ASM_AVR32_ARCH_SRAM_H */
diff --git a/arch/avr32/mach-at32ap/intc.c b/arch/avr32/mach-at32ap/intc.c
deleted file mode 100644
index aaff83c..0000000
--- a/arch/avr32/mach-at32ap/intc.c
+++ /dev/null
@@ -1,200 +0,0 @@
-/*
- * Copyright (C) 2006, 2008 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/clk.h>
-#include <linux/err.h>
-#include <linux/init.h>
-#include <linux/interrupt.h>
-#include <linux/irq.h>
-#include <linux/platform_device.h>
-#include <linux/syscore_ops.h>
-#include <linux/export.h>
-
-#include <asm/io.h>
-
-#include "intc.h"
-
-struct intc {
- void __iomem *regs;
- struct irq_chip chip;
-#ifdef CONFIG_PM
- unsigned long suspend_ipr;
- unsigned long saved_ipr[64];
-#endif
-};
-
-extern struct platform_device at32_intc0_device;
-
-/*
- * TODO: We may be able to implement mask/unmask by setting IxM flags
- * in the status register.
- */
-static void intc_mask_irq(struct irq_data *d)
-{
-
-}
-
-static void intc_unmask_irq(struct irq_data *d)
-{
-
-}
-
-static struct intc intc0 = {
- .chip = {
- .name = "intc",
- .irq_mask = intc_mask_irq,
- .irq_unmask = intc_unmask_irq,
- },
-};
-
-/*
- * All interrupts go via intc at some point.
- */
-asmlinkage void do_IRQ(int level, struct pt_regs *regs)
-{
- struct pt_regs *old_regs;
- unsigned int irq;
- unsigned long status_reg;
-
- local_irq_disable();
-
- old_regs = set_irq_regs(regs);
-
- irq_enter();
-
- irq = intc_readl(&intc0, INTCAUSE0 - 4 * level);
- generic_handle_irq(irq);
-
- /*
- * Clear all interrupt level masks so that we may handle
- * interrupts during softirq processing. If this is a nested
- * interrupt, interrupts must stay globally disabled until we
- * return.
- */
- status_reg = sysreg_read(SR);
- status_reg &= ~(SYSREG_BIT(I0M) | SYSREG_BIT(I1M)
- | SYSREG_BIT(I2M) | SYSREG_BIT(I3M));
- sysreg_write(SR, status_reg);
-
- irq_exit();
-
- set_irq_regs(old_regs);
-}
-
-void __init init_IRQ(void)
-{
- extern void _evba(void);
- extern void irq_level0(void);
- struct resource *regs;
- struct clk *pclk;
- unsigned int i;
- u32 offset, readback;
-
- regs = platform_get_resource(&at32_intc0_device, IORESOURCE_MEM, 0);
- if (!regs) {
- printk(KERN_EMERG "intc: no mmio resource defined\n");
- goto fail;
- }
- pclk = clk_get(&at32_intc0_device.dev, "pclk");
- if (IS_ERR(pclk)) {
- printk(KERN_EMERG "intc: no clock defined\n");
- goto fail;
- }
-
- clk_enable(pclk);
-
- intc0.regs = ioremap(regs->start, resource_size(regs));
- if (!intc0.regs) {
- printk(KERN_EMERG "intc: failed to map registers (0x%08lx)\n",
- (unsigned long)regs->start);
- goto fail;
- }
-
- /*
- * Initialize all interrupts to level 0 (lowest priority). The
- * priority level may be changed by calling
- * irq_set_priority().
- *
- */
- offset = (unsigned long)&irq_level0 - (unsigned long)&_evba;
- for (i = 0; i < NR_INTERNAL_IRQS; i++) {
- intc_writel(&intc0, INTPR0 + 4 * i, offset);
- readback = intc_readl(&intc0, INTPR0 + 4 * i);
- if (readback == offset)
- irq_set_chip_and_handler(i, &intc0.chip,
- handle_simple_irq);
- }
-
- /* Unmask all interrupt levels */
- sysreg_write(SR, (sysreg_read(SR)
- & ~(SR_I3M | SR_I2M | SR_I1M | SR_I0M)));
-
- return;
-
-fail:
- panic("Interrupt controller initialization failed!\n");
-}
-
-#ifdef CONFIG_PM
-void intc_set_suspend_handler(unsigned long offset)
-{
- intc0.suspend_ipr = offset;
-}
-
-static int intc_suspend(void)
-{
- int i;
-
- if (unlikely(!irqs_disabled())) {
- pr_err("intc_suspend: called with interrupts enabled\n");
- return -EINVAL;
- }
-
- if (unlikely(!intc0.suspend_ipr)) {
- pr_err("intc_suspend: suspend_ipr not initialized\n");
- return -EINVAL;
- }
-
- for (i = 0; i < 64; i++) {
- intc0.saved_ipr[i] = intc_readl(&intc0, INTPR0 + 4 * i);
- intc_writel(&intc0, INTPR0 + 4 * i, intc0.suspend_ipr);
- }
-
- return 0;
-}
-
-static void intc_resume(void)
-{
- int i;
-
- for (i = 0; i < 64; i++)
- intc_writel(&intc0, INTPR0 + 4 * i, intc0.saved_ipr[i]);
-}
-#else
-#define intc_suspend NULL
-#define intc_resume NULL
-#endif
-
-static struct syscore_ops intc_syscore_ops = {
- .suspend = intc_suspend,
- .resume = intc_resume,
-};
-
-static int __init intc_init_syscore(void)
-{
- register_syscore_ops(&intc_syscore_ops);
-
- return 0;
-}
-device_initcall(intc_init_syscore);
-
-unsigned long intc_get_pending(unsigned int group)
-{
- return intc_readl(&intc0, INTREQ0 + 4 * group);
-}
-EXPORT_SYMBOL_GPL(intc_get_pending);
diff --git a/arch/avr32/mach-at32ap/intc.h b/arch/avr32/mach-at32ap/intc.h
deleted file mode 100644
index 4d3664e..0000000
--- a/arch/avr32/mach-at32ap/intc.h
+++ /dev/null
@@ -1,329 +0,0 @@
-/*
- * Automatically generated by gen-header.xsl
- */
-#ifndef __ASM_AVR32_PERIHP_INTC_H__
-#define __ASM_AVR32_PERIHP_INTC_H__
-
-#define INTC_NUM_INT_GRPS 33
-
-#define INTC_INTPR0 0x0
-# define INTC_INTPR0_INTLEV_OFFSET 30
-# define INTC_INTPR0_INTLEV_SIZE 2
-# define INTC_INTPR0_OFFSET_OFFSET 0
-# define INTC_INTPR0_OFFSET_SIZE 24
-#define INTC_INTREQ0 0x100
-# define INTC_INTREQ0_IREQUEST0_OFFSET 0
-# define INTC_INTREQ0_IREQUEST0_SIZE 1
-# define INTC_INTREQ0_IREQUEST1_OFFSET 1
-# define INTC_INTREQ0_IREQUEST1_SIZE 1
-#define INTC_INTPR1 0x4
-# define INTC_INTPR1_INTLEV_OFFSET 30
-# define INTC_INTPR1_INTLEV_SIZE 2
-# define INTC_INTPR1_OFFSET_OFFSET 0
-# define INTC_INTPR1_OFFSET_SIZE 24
-#define INTC_INTREQ1 0x104
-# define INTC_INTREQ1_IREQUEST32_OFFSET 0
-# define INTC_INTREQ1_IREQUEST32_SIZE 1
-# define INTC_INTREQ1_IREQUEST33_OFFSET 1
-# define INTC_INTREQ1_IREQUEST33_SIZE 1
-# define INTC_INTREQ1_IREQUEST34_OFFSET 2
-# define INTC_INTREQ1_IREQUEST34_SIZE 1
-# define INTC_INTREQ1_IREQUEST35_OFFSET 3
-# define INTC_INTREQ1_IREQUEST35_SIZE 1
-# define INTC_INTREQ1_IREQUEST36_OFFSET 4
-# define INTC_INTREQ1_IREQUEST36_SIZE 1
-# define INTC_INTREQ1_IREQUEST37_OFFSET 5
-# define INTC_INTREQ1_IREQUEST37_SIZE 1
-#define INTC_INTPR2 0x8
-# define INTC_INTPR2_INTLEV_OFFSET 30
-# define INTC_INTPR2_INTLEV_SIZE 2
-# define INTC_INTPR2_OFFSET_OFFSET 0
-# define INTC_INTPR2_OFFSET_SIZE 24
-#define INTC_INTREQ2 0x108
-# define INTC_INTREQ2_IREQUEST64_OFFSET 0
-# define INTC_INTREQ2_IREQUEST64_SIZE 1
-# define INTC_INTREQ2_IREQUEST65_OFFSET 1
-# define INTC_INTREQ2_IREQUEST65_SIZE 1
-# define INTC_INTREQ2_IREQUEST66_OFFSET 2
-# define INTC_INTREQ2_IREQUEST66_SIZE 1
-# define INTC_INTREQ2_IREQUEST67_OFFSET 3
-# define INTC_INTREQ2_IREQUEST67_SIZE 1
-# define INTC_INTREQ2_IREQUEST68_OFFSET 4
-# define INTC_INTREQ2_IREQUEST68_SIZE 1
-#define INTC_INTPR3 0xc
-# define INTC_INTPR3_INTLEV_OFFSET 30
-# define INTC_INTPR3_INTLEV_SIZE 2
-# define INTC_INTPR3_OFFSET_OFFSET 0
-# define INTC_INTPR3_OFFSET_SIZE 24
-#define INTC_INTREQ3 0x10c
-# define INTC_INTREQ3_IREQUEST96_OFFSET 0
-# define INTC_INTREQ3_IREQUEST96_SIZE 1
-#define INTC_INTPR4 0x10
-# define INTC_INTPR4_INTLEV_OFFSET 30
-# define INTC_INTPR4_INTLEV_SIZE 2
-# define INTC_INTPR4_OFFSET_OFFSET 0
-# define INTC_INTPR4_OFFSET_SIZE 24
-#define INTC_INTREQ4 0x110
-# define INTC_INTREQ4_IREQUEST128_OFFSET 0
-# define INTC_INTREQ4_IREQUEST128_SIZE 1
-#define INTC_INTPR5 0x14
-# define INTC_INTPR5_INTLEV_OFFSET 30
-# define INTC_INTPR5_INTLEV_SIZE 2
-# define INTC_INTPR5_OFFSET_OFFSET 0
-# define INTC_INTPR5_OFFSET_SIZE 24
-#define INTC_INTREQ5 0x114
-# define INTC_INTREQ5_IREQUEST160_OFFSET 0
-# define INTC_INTREQ5_IREQUEST160_SIZE 1
-#define INTC_INTPR6 0x18
-# define INTC_INTPR6_INTLEV_OFFSET 30
-# define INTC_INTPR6_INTLEV_SIZE 2
-# define INTC_INTPR6_OFFSET_OFFSET 0
-# define INTC_INTPR6_OFFSET_SIZE 24
-#define INTC_INTREQ6 0x118
-# define INTC_INTREQ6_IREQUEST192_OFFSET 0
-# define INTC_INTREQ6_IREQUEST192_SIZE 1
-#define INTC_INTPR7 0x1c
-# define INTC_INTPR7_INTLEV_OFFSET 30
-# define INTC_INTPR7_INTLEV_SIZE 2
-# define INTC_INTPR7_OFFSET_OFFSET 0
-# define INTC_INTPR7_OFFSET_SIZE 24
-#define INTC_INTREQ7 0x11c
-# define INTC_INTREQ7_IREQUEST224_OFFSET 0
-# define INTC_INTREQ7_IREQUEST224_SIZE 1
-#define INTC_INTPR8 0x20
-# define INTC_INTPR8_INTLEV_OFFSET 30
-# define INTC_INTPR8_INTLEV_SIZE 2
-# define INTC_INTPR8_OFFSET_OFFSET 0
-# define INTC_INTPR8_OFFSET_SIZE 24
-#define INTC_INTREQ8 0x120
-# define INTC_INTREQ8_IREQUEST256_OFFSET 0
-# define INTC_INTREQ8_IREQUEST256_SIZE 1
-#define INTC_INTPR9 0x24
-# define INTC_INTPR9_INTLEV_OFFSET 30
-# define INTC_INTPR9_INTLEV_SIZE 2
-# define INTC_INTPR9_OFFSET_OFFSET 0
-# define INTC_INTPR9_OFFSET_SIZE 24
-#define INTC_INTREQ9 0x124
-# define INTC_INTREQ9_IREQUEST288_OFFSET 0
-# define INTC_INTREQ9_IREQUEST288_SIZE 1
-#define INTC_INTPR10 0x28
-# define INTC_INTPR10_INTLEV_OFFSET 30
-# define INTC_INTPR10_INTLEV_SIZE 2
-# define INTC_INTPR10_OFFSET_OFFSET 0
-# define INTC_INTPR10_OFFSET_SIZE 24
-#define INTC_INTREQ10 0x128
-# define INTC_INTREQ10_IREQUEST320_OFFSET 0
-# define INTC_INTREQ10_IREQUEST320_SIZE 1
-#define INTC_INTPR11 0x2c
-# define INTC_INTPR11_INTLEV_OFFSET 30
-# define INTC_INTPR11_INTLEV_SIZE 2
-# define INTC_INTPR11_OFFSET_OFFSET 0
-# define INTC_INTPR11_OFFSET_SIZE 24
-#define INTC_INTREQ11 0x12c
-# define INTC_INTREQ11_IREQUEST352_OFFSET 0
-# define INTC_INTREQ11_IREQUEST352_SIZE 1
-#define INTC_INTPR12 0x30
-# define INTC_INTPR12_INTLEV_OFFSET 30
-# define INTC_INTPR12_INTLEV_SIZE 2
-# define INTC_INTPR12_OFFSET_OFFSET 0
-# define INTC_INTPR12_OFFSET_SIZE 24
-#define INTC_INTREQ12 0x130
-# define INTC_INTREQ12_IREQUEST384_OFFSET 0
-# define INTC_INTREQ12_IREQUEST384_SIZE 1
-#define INTC_INTPR13 0x34
-# define INTC_INTPR13_INTLEV_OFFSET 30
-# define INTC_INTPR13_INTLEV_SIZE 2
-# define INTC_INTPR13_OFFSET_OFFSET 0
-# define INTC_INTPR13_OFFSET_SIZE 24
-#define INTC_INTREQ13 0x134
-# define INTC_INTREQ13_IREQUEST416_OFFSET 0
-# define INTC_INTREQ13_IREQUEST416_SIZE 1
-#define INTC_INTPR14 0x38
-# define INTC_INTPR14_INTLEV_OFFSET 30
-# define INTC_INTPR14_INTLEV_SIZE 2
-# define INTC_INTPR14_OFFSET_OFFSET 0
-# define INTC_INTPR14_OFFSET_SIZE 24
-#define INTC_INTREQ14 0x138
-# define INTC_INTREQ14_IREQUEST448_OFFSET 0
-# define INTC_INTREQ14_IREQUEST448_SIZE 1
-#define INTC_INTPR15 0x3c
-# define INTC_INTPR15_INTLEV_OFFSET 30
-# define INTC_INTPR15_INTLEV_SIZE 2
-# define INTC_INTPR15_OFFSET_OFFSET 0
-# define INTC_INTPR15_OFFSET_SIZE 24
-#define INTC_INTREQ15 0x13c
-# define INTC_INTREQ15_IREQUEST480_OFFSET 0
-# define INTC_INTREQ15_IREQUEST480_SIZE 1
-#define INTC_INTPR16 0x40
-# define INTC_INTPR16_INTLEV_OFFSET 30
-# define INTC_INTPR16_INTLEV_SIZE 2
-# define INTC_INTPR16_OFFSET_OFFSET 0
-# define INTC_INTPR16_OFFSET_SIZE 24
-#define INTC_INTREQ16 0x140
-# define INTC_INTREQ16_IREQUEST512_OFFSET 0
-# define INTC_INTREQ16_IREQUEST512_SIZE 1
-#define INTC_INTPR17 0x44
-# define INTC_INTPR17_INTLEV_OFFSET 30
-# define INTC_INTPR17_INTLEV_SIZE 2
-# define INTC_INTPR17_OFFSET_OFFSET 0
-# define INTC_INTPR17_OFFSET_SIZE 24
-#define INTC_INTREQ17 0x144
-# define INTC_INTREQ17_IREQUEST544_OFFSET 0
-# define INTC_INTREQ17_IREQUEST544_SIZE 1
-#define INTC_INTPR18 0x48
-# define INTC_INTPR18_INTLEV_OFFSET 30
-# define INTC_INTPR18_INTLEV_SIZE 2
-# define INTC_INTPR18_OFFSET_OFFSET 0
-# define INTC_INTPR18_OFFSET_SIZE 24
-#define INTC_INTREQ18 0x148
-# define INTC_INTREQ18_IREQUEST576_OFFSET 0
-# define INTC_INTREQ18_IREQUEST576_SIZE 1
-#define INTC_INTPR19 0x4c
-# define INTC_INTPR19_INTLEV_OFFSET 30
-# define INTC_INTPR19_INTLEV_SIZE 2
-# define INTC_INTPR19_OFFSET_OFFSET 0
-# define INTC_INTPR19_OFFSET_SIZE 24
-#define INTC_INTREQ19 0x14c
-# define INTC_INTREQ19_IREQUEST608_OFFSET 0
-# define INTC_INTREQ19_IREQUEST608_SIZE 1
-# define INTC_INTREQ19_IREQUEST609_OFFSET 1
-# define INTC_INTREQ19_IREQUEST609_SIZE 1
-# define INTC_INTREQ19_IREQUEST610_OFFSET 2
-# define INTC_INTREQ19_IREQUEST610_SIZE 1
-# define INTC_INTREQ19_IREQUEST611_OFFSET 3
-# define INTC_INTREQ19_IREQUEST611_SIZE 1
-#define INTC_INTPR20 0x50
-# define INTC_INTPR20_INTLEV_OFFSET 30
-# define INTC_INTPR20_INTLEV_SIZE 2
-# define INTC_INTPR20_OFFSET_OFFSET 0
-# define INTC_INTPR20_OFFSET_SIZE 24
-#define INTC_INTREQ20 0x150
-# define INTC_INTREQ20_IREQUEST640_OFFSET 0
-# define INTC_INTREQ20_IREQUEST640_SIZE 1
-#define INTC_INTPR21 0x54
-# define INTC_INTPR21_INTLEV_OFFSET 30
-# define INTC_INTPR21_INTLEV_SIZE 2
-# define INTC_INTPR21_OFFSET_OFFSET 0
-# define INTC_INTPR21_OFFSET_SIZE 24
-#define INTC_INTREQ21 0x154
-# define INTC_INTREQ21_IREQUEST672_OFFSET 0
-# define INTC_INTREQ21_IREQUEST672_SIZE 1
-#define INTC_INTPR22 0x58
-# define INTC_INTPR22_INTLEV_OFFSET 30
-# define INTC_INTPR22_INTLEV_SIZE 2
-# define INTC_INTPR22_OFFSET_OFFSET 0
-# define INTC_INTPR22_OFFSET_SIZE 24
-#define INTC_INTREQ22 0x158
-# define INTC_INTREQ22_IREQUEST704_OFFSET 0
-# define INTC_INTREQ22_IREQUEST704_SIZE 1
-# define INTC_INTREQ22_IREQUEST705_OFFSET 1
-# define INTC_INTREQ22_IREQUEST705_SIZE 1
-# define INTC_INTREQ22_IREQUEST706_OFFSET 2
-# define INTC_INTREQ22_IREQUEST706_SIZE 1
-#define INTC_INTPR23 0x5c
-# define INTC_INTPR23_INTLEV_OFFSET 30
-# define INTC_INTPR23_INTLEV_SIZE 2
-# define INTC_INTPR23_OFFSET_OFFSET 0
-# define INTC_INTPR23_OFFSET_SIZE 24
-#define INTC_INTREQ23 0x15c
-# define INTC_INTREQ23_IREQUEST736_OFFSET 0
-# define INTC_INTREQ23_IREQUEST736_SIZE 1
-# define INTC_INTREQ23_IREQUEST737_OFFSET 1
-# define INTC_INTREQ23_IREQUEST737_SIZE 1
-# define INTC_INTREQ23_IREQUEST738_OFFSET 2
-# define INTC_INTREQ23_IREQUEST738_SIZE 1
-#define INTC_INTPR24 0x60
-# define INTC_INTPR24_INTLEV_OFFSET 30
-# define INTC_INTPR24_INTLEV_SIZE 2
-# define INTC_INTPR24_OFFSET_OFFSET 0
-# define INTC_INTPR24_OFFSET_SIZE 24
-#define INTC_INTREQ24 0x160
-# define INTC_INTREQ24_IREQUEST768_OFFSET 0
-# define INTC_INTREQ24_IREQUEST768_SIZE 1
-#define INTC_INTPR25 0x64
-# define INTC_INTPR25_INTLEV_OFFSET 30
-# define INTC_INTPR25_INTLEV_SIZE 2
-# define INTC_INTPR25_OFFSET_OFFSET 0
-# define INTC_INTPR25_OFFSET_SIZE 24
-#define INTC_INTREQ25 0x164
-# define INTC_INTREQ25_IREQUEST800_OFFSET 0
-# define INTC_INTREQ25_IREQUEST800_SIZE 1
-#define INTC_INTPR26 0x68
-# define INTC_INTPR26_INTLEV_OFFSET 30
-# define INTC_INTPR26_INTLEV_SIZE 2
-# define INTC_INTPR26_OFFSET_OFFSET 0
-# define INTC_INTPR26_OFFSET_SIZE 24
-#define INTC_INTREQ26 0x168
-# define INTC_INTREQ26_IREQUEST832_OFFSET 0
-# define INTC_INTREQ26_IREQUEST832_SIZE 1
-#define INTC_INTPR27 0x6c
-# define INTC_INTPR27_INTLEV_OFFSET 30
-# define INTC_INTPR27_INTLEV_SIZE 2
-# define INTC_INTPR27_OFFSET_OFFSET 0
-# define INTC_INTPR27_OFFSET_SIZE 24
-#define INTC_INTREQ27 0x16c
-# define INTC_INTREQ27_IREQUEST864_OFFSET 0
-# define INTC_INTREQ27_IREQUEST864_SIZE 1
-#define INTC_INTPR28 0x70
-# define INTC_INTPR28_INTLEV_OFFSET 30
-# define INTC_INTPR28_INTLEV_SIZE 2
-# define INTC_INTPR28_OFFSET_OFFSET 0
-# define INTC_INTPR28_OFFSET_SIZE 24
-#define INTC_INTREQ28 0x170
-# define INTC_INTREQ28_IREQUEST896_OFFSET 0
-# define INTC_INTREQ28_IREQUEST896_SIZE 1
-#define INTC_INTPR29 0x74
-# define INTC_INTPR29_INTLEV_OFFSET 30
-# define INTC_INTPR29_INTLEV_SIZE 2
-# define INTC_INTPR29_OFFSET_OFFSET 0
-# define INTC_INTPR29_OFFSET_SIZE 24
-#define INTC_INTREQ29 0x174
-# define INTC_INTREQ29_IREQUEST928_OFFSET 0
-# define INTC_INTREQ29_IREQUEST928_SIZE 1
-#define INTC_INTPR30 0x78
-# define INTC_INTPR30_INTLEV_OFFSET 30
-# define INTC_INTPR30_INTLEV_SIZE 2
-# define INTC_INTPR30_OFFSET_OFFSET 0
-# define INTC_INTPR30_OFFSET_SIZE 24
-#define INTC_INTREQ30 0x178
-# define INTC_INTREQ30_IREQUEST960_OFFSET 0
-# define INTC_INTREQ30_IREQUEST960_SIZE 1
-#define INTC_INTPR31 0x7c
-# define INTC_INTPR31_INTLEV_OFFSET 30
-# define INTC_INTPR31_INTLEV_SIZE 2
-# define INTC_INTPR31_OFFSET_OFFSET 0
-# define INTC_INTPR31_OFFSET_SIZE 24
-#define INTC_INTREQ31 0x17c
-# define INTC_INTREQ31_IREQUEST992_OFFSET 0
-# define INTC_INTREQ31_IREQUEST992_SIZE 1
-#define INTC_INTPR32 0x80
-# define INTC_INTPR32_INTLEV_OFFSET 30
-# define INTC_INTPR32_INTLEV_SIZE 2
-# define INTC_INTPR32_OFFSET_OFFSET 0
-# define INTC_INTPR32_OFFSET_SIZE 24
-#define INTC_INTREQ32 0x180
-# define INTC_INTREQ32_IREQUEST1024_OFFSET 0
-# define INTC_INTREQ32_IREQUEST1024_SIZE 1
-#define INTC_INTCAUSE0 0x20c
-# define INTC_INTCAUSE0_CAUSEGRP_OFFSET 0
-# define INTC_INTCAUSE0_CAUSEGRP_SIZE 6
-#define INTC_INTCAUSE1 0x208
-# define INTC_INTCAUSE1_CAUSEGRP_OFFSET 0
-# define INTC_INTCAUSE1_CAUSEGRP_SIZE 6
-#define INTC_INTCAUSE2 0x204
-# define INTC_INTCAUSE2_CAUSEGRP_OFFSET 0
-# define INTC_INTCAUSE2_CAUSEGRP_SIZE 6
-#define INTC_INTCAUSE3 0x200
-# define INTC_INTCAUSE3_CAUSEGRP_OFFSET 0
-# define INTC_INTCAUSE3_CAUSEGRP_SIZE 6
-
-#define INTC_BIT(name) (1 << INTC_##name##_OFFSET)
-#define INTC_MKBF(name, value) (((value) & ((1 << INTC_##name##_SIZE) - 1)) << INTC_##name##_OFFSET)
-#define INTC_GETBF(name, value) (((value) >> INTC_##name##_OFFSET) & ((1 << INTC_##name##_SIZE) - 1))
-
-#define intc_readl(port,reg) \
- __raw_readl((port)->regs + INTC_##reg)
-#define intc_writel(port,reg,value) \
- __raw_writel((value), (port)->regs + INTC_##reg)
-
-#endif /* __ASM_AVR32_PERIHP_INTC_H__ */
diff --git a/arch/avr32/mach-at32ap/pdc.c b/arch/avr32/mach-at32ap/pdc.c
deleted file mode 100644
index 61ab15a..0000000
--- a/arch/avr32/mach-at32ap/pdc.c
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
- * Copyright (C) 2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/clk.h>
-#include <linux/err.h>
-#include <linux/init.h>
-#include <linux/platform_device.h>
-
-static int __init pdc_probe(struct platform_device *pdev)
-{
- struct clk *pclk, *hclk;
-
- pclk = clk_get(&pdev->dev, "pclk");
- if (IS_ERR(pclk)) {
- dev_err(&pdev->dev, "no pclk defined\n");
- return PTR_ERR(pclk);
- }
- hclk = clk_get(&pdev->dev, "hclk");
- if (IS_ERR(hclk)) {
- dev_err(&pdev->dev, "no hclk defined\n");
- clk_put(pclk);
- return PTR_ERR(hclk);
- }
-
- clk_enable(pclk);
- clk_enable(hclk);
-
- dev_info(&pdev->dev, "Atmel Peripheral DMA Controller enabled\n");
- return 0;
-}
-
-static struct platform_driver pdc_driver = {
- .driver = {
- .name = "pdc",
- },
-};
-
-static int __init pdc_init(void)
-{
- return platform_driver_probe(&pdc_driver, pdc_probe);
-}
-arch_initcall(pdc_init);
diff --git a/arch/avr32/mach-at32ap/pio.c b/arch/avr32/mach-at32ap/pio.c
deleted file mode 100644
index 7fae6ec..0000000
--- a/arch/avr32/mach-at32ap/pio.c
+++ /dev/null
@@ -1,470 +0,0 @@
-/*
- * Atmel PIO2 Port Multiplexer support
- *
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/clk.h>
-#include <linux/debugfs.h>
-#include <linux/export.h>
-#include <linux/fs.h>
-#include <linux/platform_device.h>
-#include <linux/irq.h>
-#include <linux/gpio.h>
-
-#include <asm/io.h>
-
-#include <mach/portmux.h>
-
-#include "pio.h"
-
-#define MAX_NR_PIO_DEVICES 8
-
-struct pio_device {
- struct gpio_chip chip;
- void __iomem *regs;
- const struct platform_device *pdev;
- struct clk *clk;
- u32 pinmux_mask;
- char name[8];
-};
-
-static struct pio_device pio_dev[MAX_NR_PIO_DEVICES];
-
-static struct pio_device *gpio_to_pio(unsigned int gpio)
-{
- struct pio_device *pio;
- unsigned int index;
-
- index = gpio >> 5;
- if (index >= MAX_NR_PIO_DEVICES)
- return NULL;
- pio = &pio_dev[index];
- if (!pio->regs)
- return NULL;
-
- return pio;
-}
-
-/* Pin multiplexing API */
-static DEFINE_SPINLOCK(pio_lock);
-
-void __init at32_select_periph(unsigned int port, u32 pin_mask,
- unsigned int periph, unsigned long flags)
-{
- struct pio_device *pio;
-
- /* assign and verify pio */
- pio = gpio_to_pio(port);
- if (unlikely(!pio)) {
- printk(KERN_WARNING "pio: invalid port %u\n", port);
- goto fail;
- }
-
- /* Test if any of the requested pins is already muxed */
- spin_lock(&pio_lock);
- if (unlikely(pio->pinmux_mask & pin_mask)) {
- printk(KERN_WARNING "%s: pin(s) busy (requested 0x%x, busy 0x%x)\n",
- pio->name, pin_mask, pio->pinmux_mask & pin_mask);
- spin_unlock(&pio_lock);
- goto fail;
- }
-
- pio->pinmux_mask |= pin_mask;
-
- /* enable pull ups */
- pio_writel(pio, PUER, pin_mask);
-
- /* select either peripheral A or B */
- if (periph)
- pio_writel(pio, BSR, pin_mask);
- else
- pio_writel(pio, ASR, pin_mask);
-
- /* enable peripheral control */
- pio_writel(pio, PDR, pin_mask);
-
- /* Disable pull ups if not requested. */
- if (!(flags & AT32_GPIOF_PULLUP))
- pio_writel(pio, PUDR, pin_mask);
-
- spin_unlock(&pio_lock);
-
- return;
-
-fail:
- dump_stack();
-}
-
-void __init at32_select_gpio(unsigned int pin, unsigned long flags)
-{
- struct pio_device *pio;
- unsigned int pin_index = pin & 0x1f;
- u32 mask = 1 << pin_index;
-
- pio = gpio_to_pio(pin);
- if (unlikely(!pio)) {
- printk("pio: invalid pin %u\n", pin);
- goto fail;
- }
-
- if (unlikely(test_and_set_bit(pin_index, &pio->pinmux_mask))) {
- printk("%s: pin %u is busy\n", pio->name, pin_index);
- goto fail;
- }
-
- if (flags & AT32_GPIOF_OUTPUT) {
- if (flags & AT32_GPIOF_HIGH)
- pio_writel(pio, SODR, mask);
- else
- pio_writel(pio, CODR, mask);
- if (flags & AT32_GPIOF_MULTIDRV)
- pio_writel(pio, MDER, mask);
- else
- pio_writel(pio, MDDR, mask);
- pio_writel(pio, PUDR, mask);
- pio_writel(pio, OER, mask);
- } else {
- if (flags & AT32_GPIOF_PULLUP)
- pio_writel(pio, PUER, mask);
- else
- pio_writel(pio, PUDR, mask);
- if (flags & AT32_GPIOF_DEGLITCH)
- pio_writel(pio, IFER, mask);
- else
- pio_writel(pio, IFDR, mask);
- pio_writel(pio, ODR, mask);
- }
-
- pio_writel(pio, PER, mask);
-
- return;
-
-fail:
- dump_stack();
-}
-
-/*
- * Undo a previous pin reservation. Will not affect the hardware
- * configuration.
- */
-void at32_deselect_pin(unsigned int pin)
-{
- struct pio_device *pio;
- unsigned int pin_index = pin & 0x1f;
-
- pio = gpio_to_pio(pin);
- if (unlikely(!pio)) {
- printk("pio: invalid pin %u\n", pin);
- dump_stack();
- return;
- }
-
- clear_bit(pin_index, &pio->pinmux_mask);
-}
-
-/* Reserve a pin, preventing anyone else from changing its configuration. */
-void __init at32_reserve_pin(unsigned int port, u32 pin_mask)
-{
- struct pio_device *pio;
-
- /* assign and verify pio */
- pio = gpio_to_pio(port);
- if (unlikely(!pio)) {
- printk(KERN_WARNING "pio: invalid port %u\n", port);
- goto fail;
- }
-
- /* Test if any of the requested pins is already muxed */
- spin_lock(&pio_lock);
- if (unlikely(pio->pinmux_mask & pin_mask)) {
- printk(KERN_WARNING "%s: pin(s) busy (req. 0x%x, busy 0x%x)\n",
- pio->name, pin_mask, pio->pinmux_mask & pin_mask);
- spin_unlock(&pio_lock);
- goto fail;
- }
-
- /* Reserve pins */
- pio->pinmux_mask |= pin_mask;
- spin_unlock(&pio_lock);
- return;
-
-fail:
- dump_stack();
-}
-
-/*--------------------------------------------------------------------------*/
-
-/* GPIO API */
-
-static int direction_input(struct gpio_chip *chip, unsigned offset)
-{
- struct pio_device *pio = gpiochip_get_data(chip);
- u32 mask = 1 << offset;
-
- if (!(pio_readl(pio, PSR) & mask))
- return -EINVAL;
-
- pio_writel(pio, ODR, mask);
- return 0;
-}
-
-static int gpio_get(struct gpio_chip *chip, unsigned offset)
-{
- struct pio_device *pio = gpiochip_get_data(chip);
-
- return (pio_readl(pio, PDSR) >> offset) & 1;
-}
-
-static void gpio_set(struct gpio_chip *chip, unsigned offset, int value);
-
-static int direction_output(struct gpio_chip *chip, unsigned offset, int value)
-{
- struct pio_device *pio = gpiochip_get_data(chip);
- u32 mask = 1 << offset;
-
- if (!(pio_readl(pio, PSR) & mask))
- return -EINVAL;
-
- gpio_set(chip, offset, value);
- pio_writel(pio, OER, mask);
- return 0;
-}
-
-static void gpio_set(struct gpio_chip *chip, unsigned offset, int value)
-{
- struct pio_device *pio = gpiochip_get_data(chip);
- u32 mask = 1 << offset;
-
- if (value)
- pio_writel(pio, SODR, mask);
- else
- pio_writel(pio, CODR, mask);
-}
-
-/*--------------------------------------------------------------------------*/
-
-/* GPIO IRQ support */
-
-static void gpio_irq_mask(struct irq_data *d)
-{
- unsigned gpio = irq_to_gpio(d->irq);
- struct pio_device *pio = &pio_dev[gpio >> 5];
-
- pio_writel(pio, IDR, 1 << (gpio & 0x1f));
-}
-
-static void gpio_irq_unmask(struct irq_data *d)
-{
- unsigned gpio = irq_to_gpio(d->irq);
- struct pio_device *pio = &pio_dev[gpio >> 5];
-
- pio_writel(pio, IER, 1 << (gpio & 0x1f));
-}
-
-static int gpio_irq_type(struct irq_data *d, unsigned type)
-{
- if (type != IRQ_TYPE_EDGE_BOTH && type != IRQ_TYPE_NONE)
- return -EINVAL;
-
- return 0;
-}
-
-static struct irq_chip gpio_irqchip = {
- .name = "gpio",
- .irq_mask = gpio_irq_mask,
- .irq_unmask = gpio_irq_unmask,
- .irq_set_type = gpio_irq_type,
-};
-
-static void gpio_irq_handler(struct irq_desc *desc)
-{
- struct pio_device *pio = irq_desc_get_chip_data(desc);
- unsigned gpio_irq;
-
- gpio_irq = (unsigned) irq_desc_get_handler_data(desc);
- for (;;) {
- u32 isr;
-
- /* ack pending GPIO interrupts */
- isr = pio_readl(pio, ISR) & pio_readl(pio, IMR);
- if (!isr)
- break;
- do {
- int i;
-
- i = ffs(isr) - 1;
- isr &= ~(1 << i);
-
- i += gpio_irq;
- generic_handle_irq(i);
- } while (isr);
- }
-}
-
-static void __init
-gpio_irq_setup(struct pio_device *pio, int irq, int gpio_irq)
-{
- unsigned i;
-
- irq_set_chip_data(irq, pio);
-
- for (i = 0; i < 32; i++, gpio_irq++) {
- irq_set_chip_data(gpio_irq, pio);
- irq_set_chip_and_handler(gpio_irq, &gpio_irqchip,
- handle_simple_irq);
- }
-
- irq_set_chained_handler_and_data(irq, gpio_irq_handler,
- (void *)gpio_irq);
-}
-
-/*--------------------------------------------------------------------------*/
-
-#ifdef CONFIG_DEBUG_FS
-
-#include <linux/seq_file.h>
-
-/*
- * This shows more info than the generic gpio dump code:
- * pullups, deglitching, open drain drive.
- */
-static void pio_bank_show(struct seq_file *s, struct gpio_chip *chip)
-{
- struct pio_device *pio = gpiochip_get_data(chip);
- u32 psr, osr, imr, pdsr, pusr, ifsr, mdsr;
- unsigned i;
- u32 mask;
- char bank;
-
- psr = pio_readl(pio, PSR);
- osr = pio_readl(pio, OSR);
- imr = pio_readl(pio, IMR);
- pdsr = pio_readl(pio, PDSR);
- pusr = pio_readl(pio, PUSR);
- ifsr = pio_readl(pio, IFSR);
- mdsr = pio_readl(pio, MDSR);
-
- bank = 'A' + pio->pdev->id;
-
- for (i = 0, mask = 1; i < 32; i++, mask <<= 1) {
- const char *label;
-
- label = gpiochip_is_requested(chip, i);
- if (!label && (imr & mask))
- label = "[irq]";
- if (!label)
- continue;
-
- seq_printf(s, " gpio-%-3d P%c%-2d (%-12s) %s %s %s",
- chip->base + i, bank, i,
- label,
- (osr & mask) ? "out" : "in ",
- (mask & pdsr) ? "hi" : "lo",
- (mask & pusr) ? " " : "up");
- if (ifsr & mask)
- seq_puts(s, " deglitch");
- if ((osr & mdsr) & mask)
- seq_puts(s, " open-drain");
- if (imr & mask)
- seq_printf(s, " irq-%d edge-both",
- gpio_to_irq(chip->base + i));
- seq_putc(s, '\n');
- }
-}
-
-#else
-#define pio_bank_show NULL
-#endif
-
-
-/*--------------------------------------------------------------------------*/
-
-static int __init pio_probe(struct platform_device *pdev)
-{
- struct pio_device *pio = NULL;
- int irq = platform_get_irq(pdev, 0);
- int gpio_irq_base = GPIO_IRQ_BASE + pdev->id * 32;
-
- BUG_ON(pdev->id >= MAX_NR_PIO_DEVICES);
- pio = &pio_dev[pdev->id];
- BUG_ON(!pio->regs);
-
- pio->chip.label = pio->name;
- pio->chip.base = pdev->id * 32;
- pio->chip.ngpio = 32;
- pio->chip.parent = &pdev->dev;
- pio->chip.owner = THIS_MODULE;
-
- pio->chip.direction_input = direction_input;
- pio->chip.get = gpio_get;
- pio->chip.direction_output = direction_output;
- pio->chip.set = gpio_set;
- pio->chip.dbg_show = pio_bank_show;
-
- gpiochip_add_data(&pio->chip, pio);
-
- gpio_irq_setup(pio, irq, gpio_irq_base);
-
- platform_set_drvdata(pdev, pio);
-
- printk(KERN_DEBUG "%s: base 0x%p, irq %d chains %d..%d\n",
- pio->name, pio->regs, irq, gpio_irq_base, gpio_irq_base + 31);
-
- return 0;
-}
-
-static struct platform_driver pio_driver = {
- .driver = {
- .name = "pio",
- },
-};
-
-static int __init pio_init(void)
-{
- return platform_driver_probe(&pio_driver, pio_probe);
-}
-postcore_initcall(pio_init);
-
-void __init at32_init_pio(struct platform_device *pdev)
-{
- struct resource *regs;
- struct pio_device *pio;
-
- if (pdev->id >= MAX_NR_PIO_DEVICES) {
- dev_err(&pdev->dev, "only %d PIO devices supported\n",
- MAX_NR_PIO_DEVICES);
- return;
- }
-
- pio = &pio_dev[pdev->id];
- snprintf(pio->name, sizeof(pio->name), "pio%d", pdev->id);
-
- regs = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!regs) {
- dev_err(&pdev->dev, "no mmio resource defined\n");
- return;
- }
-
- pio->clk = clk_get(&pdev->dev, "mck");
- if (IS_ERR(pio->clk))
- /*
- * This is a fatal error, but if we continue we might
- * be so lucky that we manage to initialize the
- * console and display this message...
- */
- dev_err(&pdev->dev, "no mck clock defined\n");
- else
- clk_enable(pio->clk);
-
- pio->pdev = pdev;
- pio->regs = ioremap(regs->start, resource_size(regs));
-
- /* start with irqs disabled and acked */
- pio_writel(pio, IDR, ~0UL);
- (void) pio_readl(pio, ISR);
-}
diff --git a/arch/avr32/mach-at32ap/pio.h b/arch/avr32/mach-at32ap/pio.h
deleted file mode 100644
index 9484dfc..0000000
--- a/arch/avr32/mach-at32ap/pio.h
+++ /dev/null
@@ -1,180 +0,0 @@
-/*
- * Atmel PIO2 Port Multiplexer support
- *
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#ifndef __ARCH_AVR32_AT32AP_PIO_H__
-#define __ARCH_AVR32_AT32AP_PIO_H__
-
-/* PIO register offsets */
-#define PIO_PER 0x0000
-#define PIO_PDR 0x0004
-#define PIO_PSR 0x0008
-#define PIO_OER 0x0010
-#define PIO_ODR 0x0014
-#define PIO_OSR 0x0018
-#define PIO_IFER 0x0020
-#define PIO_IFDR 0x0024
-#define PIO_IFSR 0x0028
-#define PIO_SODR 0x0030
-#define PIO_CODR 0x0034
-#define PIO_ODSR 0x0038
-#define PIO_PDSR 0x003c
-#define PIO_IER 0x0040
-#define PIO_IDR 0x0044
-#define PIO_IMR 0x0048
-#define PIO_ISR 0x004c
-#define PIO_MDER 0x0050
-#define PIO_MDDR 0x0054
-#define PIO_MDSR 0x0058
-#define PIO_PUDR 0x0060
-#define PIO_PUER 0x0064
-#define PIO_PUSR 0x0068
-#define PIO_ASR 0x0070
-#define PIO_BSR 0x0074
-#define PIO_ABSR 0x0078
-#define PIO_OWER 0x00a0
-#define PIO_OWDR 0x00a4
-#define PIO_OWSR 0x00a8
-
-/* Bitfields in PER */
-
-/* Bitfields in PDR */
-
-/* Bitfields in PSR */
-
-/* Bitfields in OER */
-
-/* Bitfields in ODR */
-
-/* Bitfields in OSR */
-
-/* Bitfields in IFER */
-
-/* Bitfields in IFDR */
-
-/* Bitfields in IFSR */
-
-/* Bitfields in SODR */
-
-/* Bitfields in CODR */
-
-/* Bitfields in ODSR */
-
-/* Bitfields in PDSR */
-
-/* Bitfields in IER */
-
-/* Bitfields in IDR */
-
-/* Bitfields in IMR */
-
-/* Bitfields in ISR */
-
-/* Bitfields in MDER */
-
-/* Bitfields in MDDR */
-
-/* Bitfields in MDSR */
-
-/* Bitfields in PUDR */
-
-/* Bitfields in PUER */
-
-/* Bitfields in PUSR */
-
-/* Bitfields in ASR */
-
-/* Bitfields in BSR */
-
-/* Bitfields in ABSR */
-#define PIO_P0_OFFSET 0
-#define PIO_P0_SIZE 1
-#define PIO_P1_OFFSET 1
-#define PIO_P1_SIZE 1
-#define PIO_P2_OFFSET 2
-#define PIO_P2_SIZE 1
-#define PIO_P3_OFFSET 3
-#define PIO_P3_SIZE 1
-#define PIO_P4_OFFSET 4
-#define PIO_P4_SIZE 1
-#define PIO_P5_OFFSET 5
-#define PIO_P5_SIZE 1
-#define PIO_P6_OFFSET 6
-#define PIO_P6_SIZE 1
-#define PIO_P7_OFFSET 7
-#define PIO_P7_SIZE 1
-#define PIO_P8_OFFSET 8
-#define PIO_P8_SIZE 1
-#define PIO_P9_OFFSET 9
-#define PIO_P9_SIZE 1
-#define PIO_P10_OFFSET 10
-#define PIO_P10_SIZE 1
-#define PIO_P11_OFFSET 11
-#define PIO_P11_SIZE 1
-#define PIO_P12_OFFSET 12
-#define PIO_P12_SIZE 1
-#define PIO_P13_OFFSET 13
-#define PIO_P13_SIZE 1
-#define PIO_P14_OFFSET 14
-#define PIO_P14_SIZE 1
-#define PIO_P15_OFFSET 15
-#define PIO_P15_SIZE 1
-#define PIO_P16_OFFSET 16
-#define PIO_P16_SIZE 1
-#define PIO_P17_OFFSET 17
-#define PIO_P17_SIZE 1
-#define PIO_P18_OFFSET 18
-#define PIO_P18_SIZE 1
-#define PIO_P19_OFFSET 19
-#define PIO_P19_SIZE 1
-#define PIO_P20_OFFSET 20
-#define PIO_P20_SIZE 1
-#define PIO_P21_OFFSET 21
-#define PIO_P21_SIZE 1
-#define PIO_P22_OFFSET 22
-#define PIO_P22_SIZE 1
-#define PIO_P23_OFFSET 23
-#define PIO_P23_SIZE 1
-#define PIO_P24_OFFSET 24
-#define PIO_P24_SIZE 1
-#define PIO_P25_OFFSET 25
-#define PIO_P25_SIZE 1
-#define PIO_P26_OFFSET 26
-#define PIO_P26_SIZE 1
-#define PIO_P27_OFFSET 27
-#define PIO_P27_SIZE 1
-#define PIO_P28_OFFSET 28
-#define PIO_P28_SIZE 1
-#define PIO_P29_OFFSET 29
-#define PIO_P29_SIZE 1
-#define PIO_P30_OFFSET 30
-#define PIO_P30_SIZE 1
-#define PIO_P31_OFFSET 31
-#define PIO_P31_SIZE 1
-
-/* Bitfields in OWER */
-
-/* Bitfields in OWDR */
-
-/* Bitfields in OWSR */
-
-/* Bit manipulation macros */
-#define PIO_BIT(name) (1 << PIO_##name##_OFFSET)
-#define PIO_BF(name,value) (((value) & ((1 << PIO_##name##_SIZE) - 1)) << PIO_##name##_OFFSET)
-#define PIO_BFEXT(name,value) (((value) >> PIO_##name##_OFFSET) & ((1 << PIO_##name##_SIZE) - 1))
-#define PIO_BFINS(name,value,old) (((old) & ~(((1 << PIO_##name##_SIZE) - 1) << PIO_##name##_OFFSET)) | PIO_BF(name,value))
-
-/* Register access macros */
-#define pio_readl(port,reg) \
- __raw_readl((port)->regs + PIO_##reg)
-#define pio_writel(port,reg,value) \
- __raw_writel((value), (port)->regs + PIO_##reg)
-
-void at32_init_pio(struct platform_device *pdev);
-
-#endif /* __ARCH_AVR32_AT32AP_PIO_H__ */
diff --git a/arch/avr32/mach-at32ap/pm-at32ap700x.S b/arch/avr32/mach-at32ap/pm-at32ap700x.S
deleted file mode 100644
index 1c8e4e6..0000000
--- a/arch/avr32/mach-at32ap/pm-at32ap700x.S
+++ /dev/null
@@ -1,167 +0,0 @@
-/*
- * Low-level Power Management code.
- *
- * Copyright (C) 2008 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <asm/asm.h>
-#include <asm/asm-offsets.h>
-#include <asm/thread_info.h>
-#include <mach/pm.h>
-
-#include "pm.h"
-#include "sdramc.h"
-
-/* Same as 0xfff00000 but fits in a 21 bit signed immediate */
-#define PM_BASE -0x100000
-
- /* Keep this close to the irq handlers */
- .section .irq.text, "ax", @progbits
-
- /*
- * void cpu_enter_idle(void)
- *
- * Put the CPU into "idle" mode, in which it will consume
- * significantly less power.
- *
- * If an interrupt comes along in the window between
- * unmask_interrupts and the sleep instruction below, the
- * interrupt code will adjust the return address so that we
- * never execute the sleep instruction. This is required
- * because the AP7000 doesn't unmask interrupts when entering
- * sleep modes; later CPUs may not need this workaround.
- */
- .global cpu_enter_idle
- .type cpu_enter_idle, @function
-cpu_enter_idle:
- mask_interrupts
- get_thread_info r8
- ld.w r9, r8[TI_flags]
- bld r9, TIF_NEED_RESCHED
- brcs .Lret_from_sleep
- sbr r9, TIF_CPU_GOING_TO_SLEEP
- st.w r8[TI_flags], r9
- unmask_interrupts
- sleep CPU_SLEEP_IDLE
- .size cpu_enter_idle, . - cpu_enter_idle
-
- /*
- * Common return path for PM functions that don't run from
- * SRAM.
- */
- .global cpu_idle_skip_sleep
- .type cpu_idle_skip_sleep, @function
-cpu_idle_skip_sleep:
- mask_interrupts
- ld.w r9, r8[TI_flags]
- cbr r9, TIF_CPU_GOING_TO_SLEEP
- st.w r8[TI_flags], r9
-.Lret_from_sleep:
- unmask_interrupts
- retal r12
- .size cpu_idle_skip_sleep, . - cpu_idle_skip_sleep
-
-#ifdef CONFIG_PM
- .section .init.text, "ax", @progbits
-
- .global pm_exception
- .type pm_exception, @function
-pm_exception:
- /*
- * Exceptions are masked when we switch to this handler, so
- * we'll only get "unrecoverable" exceptions (offset 0.)
- */
- sub r12, pc, . - .Lpanic_msg
- lddpc pc, .Lpanic_addr
-
- .align 2
-.Lpanic_addr:
- .long panic
-.Lpanic_msg:
- .asciz "Unrecoverable exception during suspend\n"
- .size pm_exception, . - pm_exception
-
- .global pm_irq0
- .type pm_irq0, @function
-pm_irq0:
- /* Disable interrupts and return after the sleep instruction */
- mfsr r9, SYSREG_RSR_INT0
- mtsr SYSREG_RAR_INT0, r8
- sbr r9, SYSREG_GM_OFFSET
- mtsr SYSREG_RSR_INT0, r9
- rete
-
- /*
- * void cpu_enter_standby(unsigned long sdramc_base)
- *
- * Enter PM_SUSPEND_STANDBY mode. At this point, all drivers
- * are suspended and interrupts are disabled. Interrupts
- * marked as 'wakeup' event sources may still come along and
- * get us out of here.
- *
- * The SDRAM will be put into self-refresh mode (which does
- * not require a clock from the CPU), and the CPU will be put
- * into "frozen" mode (HSB bus stopped). The SDRAM controller
- * will automatically bring the SDRAM into normal mode on the
- * first access, and the power manager will automatically
- * start the HSB and CPU clocks upon a wakeup event.
- *
- * This code uses the same "skip sleep" technique as above.
- * It is very important that we jump directly to
- * cpu_after_sleep after the sleep instruction since that's
- * where we'll end up if the interrupt handler decides that we
- * need to skip the sleep instruction.
- */
- .global pm_standby
- .type pm_standby, @function
-pm_standby:
- /*
- * interrupts are already masked at this point, and EVBA
- * points to pm_exception above.
- */
- ld.w r10, r12[SDRAMC_LPR]
- sub r8, pc, . - 1f /* return address for irq handler */
- mov r11, SDRAMC_LPR_LPCB_SELF_RFR
- bfins r10, r11, 0, 2 /* LPCB <- self Refresh */
- sync 0 /* flush write buffer */
- st.w r12[SDRAMC_LPR], r10 /* put SDRAM in self-refresh mode */
- ld.w r11, r12[SDRAMC_LPR]
- unmask_interrupts
- sleep CPU_SLEEP_FROZEN
-1: mask_interrupts
- retal r12
- .size pm_standby, . - pm_standby
-
- .global pm_suspend_to_ram
- .type pm_suspend_to_ram, @function
-pm_suspend_to_ram:
- /*
- * interrupts are already masked at this point, and EVBA
- * points to pm_exception above.
- */
- mov r11, 0
- cache r11[2], 8 /* clean all dcache lines */
- sync 0 /* flush write buffer */
- ld.w r10, r12[SDRAMC_LPR]
- sub r8, pc, . - 1f /* return address for irq handler */
- mov r11, SDRAMC_LPR_LPCB_SELF_RFR
- bfins r10, r11, 0, 2 /* LPCB <- self refresh */
- st.w r12[SDRAMC_LPR], r10 /* put SDRAM in self-refresh mode */
- ld.w r11, r12[SDRAMC_LPR]
-
- unmask_interrupts
- sleep CPU_SLEEP_STOP
-1: mask_interrupts
-
- retal r12
- .size pm_suspend_to_ram, . - pm_suspend_to_ram
-
- .global pm_sram_end
- .type pm_sram_end, @function
-pm_sram_end:
- .size pm_sram_end, 0
-
-#endif /* CONFIG_PM */
diff --git a/arch/avr32/mach-at32ap/pm.c b/arch/avr32/mach-at32ap/pm.c
deleted file mode 100644
index db19084..0000000
--- a/arch/avr32/mach-at32ap/pm.c
+++ /dev/null
@@ -1,243 +0,0 @@
-/*
- * AVR32 AP Power Management
- *
- * Copyright (C) 2008 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * version 2 as published by the Free Software Foundation.
- */
-#include <linux/io.h>
-#include <linux/suspend.h>
-#include <linux/vmalloc.h>
-
-#include <asm/cacheflush.h>
-#include <asm/sysreg.h>
-
-#include <mach/chip.h>
-#include <mach/pm.h>
-#include <mach/sram.h>
-
-#include "sdramc.h"
-
-#define SRAM_PAGE_FLAGS (SYSREG_BIT(TLBELO_D) | SYSREG_BF(SZ, 1) \
- | SYSREG_BF(AP, 3) | SYSREG_BIT(G))
-
-
-static unsigned long pm_sram_start;
-static size_t pm_sram_size;
-static struct vm_struct *pm_sram_area;
-
-static void (*avr32_pm_enter_standby)(unsigned long sdramc_base);
-static void (*avr32_pm_enter_str)(unsigned long sdramc_base);
-
-/*
- * Must be called with interrupts disabled. Exceptions will be masked
- * on return (i.e. all exceptions will be "unrecoverable".)
- */
-static void *avr32_pm_map_sram(void)
-{
- unsigned long vaddr;
- unsigned long page_addr;
- u32 tlbehi;
- u32 mmucr;
-
- vaddr = (unsigned long)pm_sram_area->addr;
- page_addr = pm_sram_start & PAGE_MASK;
-
- /*
- * Mask exceptions and grab the first TLB entry. We won't be
- * needing it while sleeping.
- */
- asm volatile("ssrf %0" : : "i"(SYSREG_EM_OFFSET) : "memory");
-
- mmucr = sysreg_read(MMUCR);
- tlbehi = sysreg_read(TLBEHI);
- sysreg_write(MMUCR, SYSREG_BFINS(DRP, 0, mmucr));
-
- tlbehi = SYSREG_BF(ASID, SYSREG_BFEXT(ASID, tlbehi));
- tlbehi |= vaddr & PAGE_MASK;
- tlbehi |= SYSREG_BIT(TLBEHI_V);
-
- sysreg_write(TLBELO, page_addr | SRAM_PAGE_FLAGS);
- sysreg_write(TLBEHI, tlbehi);
- __builtin_tlbw();
-
- return (void *)(vaddr + pm_sram_start - page_addr);
-}
-
-/*
- * Must be called with interrupts disabled. Exceptions will be
- * unmasked on return.
- */
-static void avr32_pm_unmap_sram(void)
-{
- u32 mmucr;
- u32 tlbehi;
- u32 tlbarlo;
-
- /* Going to update TLB entry at index 0 */
- mmucr = sysreg_read(MMUCR);
- tlbehi = sysreg_read(TLBEHI);
- sysreg_write(MMUCR, SYSREG_BFINS(DRP, 0, mmucr));
-
- /* Clear the "valid" bit */
- tlbehi = SYSREG_BF(ASID, SYSREG_BFEXT(ASID, tlbehi));
- sysreg_write(TLBEHI, tlbehi);
-
- /* Mark it as "not accessed" */
- tlbarlo = sysreg_read(TLBARLO);
- sysreg_write(TLBARLO, tlbarlo | 0x80000000U);
-
- /* Update the TLB */
- __builtin_tlbw();
-
- /* Unmask exceptions */
- asm volatile("csrf %0" : : "i"(SYSREG_EM_OFFSET) : "memory");
-}
-
-static int avr32_pm_valid_state(suspend_state_t state)
-{
- switch (state) {
- case PM_SUSPEND_ON:
- case PM_SUSPEND_STANDBY:
- case PM_SUSPEND_MEM:
- return 1;
-
- default:
- return 0;
- }
-}
-
-static int avr32_pm_enter(suspend_state_t state)
-{
- u32 lpr_saved;
- u32 evba_saved;
- void *sram;
-
- switch (state) {
- case PM_SUSPEND_STANDBY:
- sram = avr32_pm_map_sram();
-
- /* Switch to in-sram exception handlers */
- evba_saved = sysreg_read(EVBA);
- sysreg_write(EVBA, (unsigned long)sram);
-
- /*
- * Save the LPR register so that we can re-enable
- * SDRAM Low Power mode on resume.
- */
- lpr_saved = sdramc_readl(LPR);
- pr_debug("%s: Entering standby...\n", __func__);
- avr32_pm_enter_standby(SDRAMC_BASE);
- sdramc_writel(LPR, lpr_saved);
-
- /* Switch back to regular exception handlers */
- sysreg_write(EVBA, evba_saved);
-
- avr32_pm_unmap_sram();
- break;
-
- case PM_SUSPEND_MEM:
- sram = avr32_pm_map_sram();
-
- /* Switch to in-sram exception handlers */
- evba_saved = sysreg_read(EVBA);
- sysreg_write(EVBA, (unsigned long)sram);
-
- /*
- * Save the LPR register so that we can re-enable
- * SDRAM Low Power mode on resume.
- */
- lpr_saved = sdramc_readl(LPR);
- pr_debug("%s: Entering suspend-to-ram...\n", __func__);
- avr32_pm_enter_str(SDRAMC_BASE);
- sdramc_writel(LPR, lpr_saved);
-
- /* Switch back to regular exception handlers */
- sysreg_write(EVBA, evba_saved);
-
- avr32_pm_unmap_sram();
- break;
-
- case PM_SUSPEND_ON:
- pr_debug("%s: Entering idle...\n", __func__);
- cpu_enter_idle();
- break;
-
- default:
- pr_debug("%s: Invalid suspend state %d\n", __func__, state);
- goto out;
- }
-
- pr_debug("%s: wakeup\n", __func__);
-
-out:
- return 0;
-}
-
-static const struct platform_suspend_ops avr32_pm_ops = {
- .valid = avr32_pm_valid_state,
- .enter = avr32_pm_enter,
-};
-
-static unsigned long __init avr32_pm_offset(void *symbol)
-{
- extern u8 pm_exception[];
-
- return (unsigned long)symbol - (unsigned long)pm_exception;
-}
-
-static int __init avr32_pm_init(void)
-{
- extern u8 pm_exception[];
- extern u8 pm_irq0[];
- extern u8 pm_standby[];
- extern u8 pm_suspend_to_ram[];
- extern u8 pm_sram_end[];
- void *dst;
-
- /*
- * To keep things simple, we depend on not needing more than a
- * single page.
- */
- pm_sram_size = avr32_pm_offset(pm_sram_end);
- if (pm_sram_size > PAGE_SIZE)
- goto err;
-
- pm_sram_start = sram_alloc(pm_sram_size);
- if (!pm_sram_start)
- goto err_alloc_sram;
-
- /* Grab a virtual area we can use later on. */
- pm_sram_area = get_vm_area(pm_sram_size, VM_IOREMAP);
- if (!pm_sram_area)
- goto err_vm_area;
- pm_sram_area->phys_addr = pm_sram_start;
-
- local_irq_disable();
- dst = avr32_pm_map_sram();
- memcpy(dst, pm_exception, pm_sram_size);
- flush_dcache_region(dst, pm_sram_size);
- invalidate_icache_region(dst, pm_sram_size);
- avr32_pm_unmap_sram();
- local_irq_enable();
-
- avr32_pm_enter_standby = dst + avr32_pm_offset(pm_standby);
- avr32_pm_enter_str = dst + avr32_pm_offset(pm_suspend_to_ram);
- intc_set_suspend_handler(avr32_pm_offset(pm_irq0));
-
- suspend_set_ops(&avr32_pm_ops);
-
- printk("AVR32 AP Power Management enabled\n");
-
- return 0;
-
-err_vm_area:
- sram_free(pm_sram_start, pm_sram_size);
-err_alloc_sram:
-err:
- pr_err("AVR32 Power Management initialization failed\n");
- return -ENOMEM;
-}
-arch_initcall(avr32_pm_init);
diff --git a/arch/avr32/mach-at32ap/pm.h b/arch/avr32/mach-at32ap/pm.h
deleted file mode 100644
index 532a373..0000000
--- a/arch/avr32/mach-at32ap/pm.h
+++ /dev/null
@@ -1,112 +0,0 @@
-/*
- * Register definitions for the Power Manager (PM)
- */
-#ifndef __ARCH_AVR32_MACH_AT32AP_PM_H__
-#define __ARCH_AVR32_MACH_AT32AP_PM_H__
-
-/* PM register offsets */
-#define PM_MCCTRL 0x0000
-#define PM_CKSEL 0x0004
-#define PM_CPU_MASK 0x0008
-#define PM_HSB_MASK 0x000c
-#define PM_PBA_MASK 0x0010
-#define PM_PBB_MASK 0x0014
-#define PM_PLL0 0x0020
-#define PM_PLL1 0x0024
-#define PM_IER 0x0040
-#define PM_IDR 0x0044
-#define PM_IMR 0x0048
-#define PM_ISR 0x004c
-#define PM_ICR 0x0050
-#define PM_GCCTRL(x) (0x0060 + 4 * (x))
-#define PM_RCAUSE 0x00c0
-
-/* Bitfields in CKSEL */
-#define PM_CPUSEL_OFFSET 0
-#define PM_CPUSEL_SIZE 3
-#define PM_CPUDIV_OFFSET 7
-#define PM_CPUDIV_SIZE 1
-#define PM_HSBSEL_OFFSET 8
-#define PM_HSBSEL_SIZE 3
-#define PM_HSBDIV_OFFSET 15
-#define PM_HSBDIV_SIZE 1
-#define PM_PBASEL_OFFSET 16
-#define PM_PBASEL_SIZE 3
-#define PM_PBADIV_OFFSET 23
-#define PM_PBADIV_SIZE 1
-#define PM_PBBSEL_OFFSET 24
-#define PM_PBBSEL_SIZE 3
-#define PM_PBBDIV_OFFSET 31
-#define PM_PBBDIV_SIZE 1
-
-/* Bitfields in PLL0 */
-#define PM_PLLEN_OFFSET 0
-#define PM_PLLEN_SIZE 1
-#define PM_PLLOSC_OFFSET 1
-#define PM_PLLOSC_SIZE 1
-#define PM_PLLOPT_OFFSET 2
-#define PM_PLLOPT_SIZE 3
-#define PM_PLLDIV_OFFSET 8
-#define PM_PLLDIV_SIZE 8
-#define PM_PLLMUL_OFFSET 16
-#define PM_PLLMUL_SIZE 8
-#define PM_PLLCOUNT_OFFSET 24
-#define PM_PLLCOUNT_SIZE 6
-#define PM_PLLTEST_OFFSET 31
-#define PM_PLLTEST_SIZE 1
-
-/* Bitfields in ICR */
-#define PM_LOCK0_OFFSET 0
-#define PM_LOCK0_SIZE 1
-#define PM_LOCK1_OFFSET 1
-#define PM_LOCK1_SIZE 1
-#define PM_WAKE_OFFSET 2
-#define PM_WAKE_SIZE 1
-#define PM_CKRDY_OFFSET 5
-#define PM_CKRDY_SIZE 1
-#define PM_MSKRDY_OFFSET 6
-#define PM_MSKRDY_SIZE 1
-
-/* Bitfields in GCCTRL0 */
-#define PM_OSCSEL_OFFSET 0
-#define PM_OSCSEL_SIZE 1
-#define PM_PLLSEL_OFFSET 1
-#define PM_PLLSEL_SIZE 1
-#define PM_CEN_OFFSET 2
-#define PM_CEN_SIZE 1
-#define PM_DIVEN_OFFSET 4
-#define PM_DIVEN_SIZE 1
-#define PM_DIV_OFFSET 8
-#define PM_DIV_SIZE 8
-
-/* Bitfields in RCAUSE */
-#define PM_POR_OFFSET 0
-#define PM_POR_SIZE 1
-#define PM_EXT_OFFSET 2
-#define PM_EXT_SIZE 1
-#define PM_WDT_OFFSET 3
-#define PM_WDT_SIZE 1
-#define PM_NTAE_OFFSET 4
-#define PM_NTAE_SIZE 1
-
-/* Bit manipulation macros */
-#define PM_BIT(name) \
- (1 << PM_##name##_OFFSET)
-#define PM_BF(name,value) \
- (((value) & ((1 << PM_##name##_SIZE) - 1)) \
- << PM_##name##_OFFSET)
-#define PM_BFEXT(name,value) \
- (((value) >> PM_##name##_OFFSET) \
- & ((1 << PM_##name##_SIZE) - 1))
-#define PM_BFINS(name,value,old)\
- (((old) & ~(((1 << PM_##name##_SIZE) - 1) \
- << PM_##name##_OFFSET)) \
- | PM_BF(name,value))
-
-/* Register access macros */
-#define pm_readl(reg) \
- __raw_readl((void __iomem __force *)PM_BASE + PM_##reg)
-#define pm_writel(reg,value) \
- __raw_writel((value), (void __iomem __force *)PM_BASE + PM_##reg)
-
-#endif /* __ARCH_AVR32_MACH_AT32AP_PM_H__ */
diff --git a/arch/avr32/mach-at32ap/sdramc.h b/arch/avr32/mach-at32ap/sdramc.h
deleted file mode 100644
index 66eeaed..0000000
--- a/arch/avr32/mach-at32ap/sdramc.h
+++ /dev/null
@@ -1,76 +0,0 @@
-/*
- * Register definitions for the AT32AP SDRAM Controller
- *
- * Copyright (C) 2008 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * version 2 as published by the Free Software Foundation.
- */
-
-/* Register offsets */
-#define SDRAMC_MR 0x0000
-#define SDRAMC_TR 0x0004
-#define SDRAMC_CR 0x0008
-#define SDRAMC_HSR 0x000c
-#define SDRAMC_LPR 0x0010
-#define SDRAMC_IER 0x0014
-#define SDRAMC_IDR 0x0018
-#define SDRAMC_IMR 0x001c
-#define SDRAMC_ISR 0x0020
-#define SDRAMC_MDR 0x0024
-
-/* MR - Mode Register */
-#define SDRAMC_MR_MODE_NORMAL ( 0 << 0)
-#define SDRAMC_MR_MODE_NOP ( 1 << 0)
-#define SDRAMC_MR_MODE_BANKS_PRECHARGE ( 2 << 0)
-#define SDRAMC_MR_MODE_LOAD_MODE ( 3 << 0)
-#define SDRAMC_MR_MODE_AUTO_REFRESH ( 4 << 0)
-#define SDRAMC_MR_MODE_EXT_LOAD_MODE ( 5 << 0)
-#define SDRAMC_MR_MODE_POWER_DOWN ( 6 << 0)
-
-/* CR - Configuration Register */
-#define SDRAMC_CR_NC_8_BITS ( 0 << 0)
-#define SDRAMC_CR_NC_9_BITS ( 1 << 0)
-#define SDRAMC_CR_NC_10_BITS ( 2 << 0)
-#define SDRAMC_CR_NC_11_BITS ( 3 << 0)
-#define SDRAMC_CR_NR_11_BITS ( 0 << 2)
-#define SDRAMC_CR_NR_12_BITS ( 1 << 2)
-#define SDRAMC_CR_NR_13_BITS ( 2 << 2)
-#define SDRAMC_CR_NB_2_BANKS ( 0 << 4)
-#define SDRAMC_CR_NB_4_BANKS ( 1 << 4)
-#define SDRAMC_CR_CAS(x) ((x) << 5)
-#define SDRAMC_CR_DBW_32_BITS ( 0 << 7)
-#define SDRAMC_CR_DBW_16_BITS ( 1 << 7)
-#define SDRAMC_CR_TWR(x) ((x) << 8)
-#define SDRAMC_CR_TRC(x) ((x) << 12)
-#define SDRAMC_CR_TRP(x) ((x) << 16)
-#define SDRAMC_CR_TRCD(x) ((x) << 20)
-#define SDRAMC_CR_TRAS(x) ((x) << 24)
-#define SDRAMC_CR_TXSR(x) ((x) << 28)
-
-/* HSR - High Speed Register */
-#define SDRAMC_HSR_DA ( 1 << 0)
-
-/* LPR - Low Power Register */
-#define SDRAMC_LPR_LPCB_INHIBIT ( 0 << 0)
-#define SDRAMC_LPR_LPCB_SELF_RFR ( 1 << 0)
-#define SDRAMC_LPR_LPCB_PDOWN ( 2 << 0)
-#define SDRAMC_LPR_LPCB_DEEP_PDOWN ( 3 << 0)
-#define SDRAMC_LPR_PASR(x) ((x) << 4)
-#define SDRAMC_LPR_TCSR(x) ((x) << 8)
-#define SDRAMC_LPR_DS(x) ((x) << 10)
-#define SDRAMC_LPR_TIMEOUT(x) ((x) << 12)
-
-/* IER/IDR/IMR/ISR - Interrupt Enable/Disable/Mask/Status Register */
-#define SDRAMC_ISR_RES ( 1 << 0)
-
-/* MDR - Memory Device Register */
-#define SDRAMC_MDR_MD_SDRAM ( 0 << 0)
-#define SDRAMC_MDR_MD_LOW_PWR_SDRAM ( 1 << 0)
-
-/* Register access macros */
-#define sdramc_readl(reg) \
- __raw_readl((void __iomem __force *)SDRAMC_BASE + SDRAMC_##reg)
-#define sdramc_writel(reg, value) \
- __raw_writel(value, (void __iomem __force *)SDRAMC_BASE + SDRAMC_##reg)
diff --git a/arch/avr32/mm/Makefile b/arch/avr32/mm/Makefile
deleted file mode 100644
index 0066491..0000000
--- a/arch/avr32/mm/Makefile
+++ /dev/null
@@ -1,6 +0,0 @@
-#
-# Makefile for the Linux/AVR32 kernel.
-#
-
-obj-y += init.o clear_page.o copy_page.o dma-coherent.o
-obj-y += ioremap.o cache.o fault.o tlb.o
diff --git a/arch/avr32/mm/cache.c b/arch/avr32/mm/cache.c
deleted file mode 100644
index d947682..0000000
--- a/arch/avr32/mm/cache.c
+++ /dev/null
@@ -1,163 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/highmem.h>
-#include <linux/unistd.h>
-
-#include <asm/cacheflush.h>
-#include <asm/cachectl.h>
-#include <asm/processor.h>
-#include <linux/uaccess.h>
-#include <asm/syscalls.h>
-
-/*
- * If you attempt to flush anything more than this, you need superuser
- * privileges. The value is completely arbitrary.
- */
-#define CACHEFLUSH_MAX_LEN 1024
-
-void invalidate_dcache_region(void *start, size_t size)
-{
- unsigned long v, begin, end, linesz, mask;
-
- linesz = boot_cpu_data.dcache.linesz;
- mask = linesz - 1;
-
- /* when first and/or last cachelines are shared, flush them
- * instead of invalidating ... never discard valid data!
- */
- begin = (unsigned long)start;
- end = begin + size;
-
- if (begin & mask) {
- flush_dcache_line(start);
- begin += linesz;
- }
- if (end & mask) {
- flush_dcache_line((void *)end);
- end &= ~mask;
- }
-
- /* remaining cachelines only need invalidation */
- for (v = begin; v < end; v += linesz)
- invalidate_dcache_line((void *)v);
- flush_write_buffer();
-}
-
-void clean_dcache_region(void *start, size_t size)
-{
- unsigned long v, begin, end, linesz;
-
- linesz = boot_cpu_data.dcache.linesz;
- begin = (unsigned long)start & ~(linesz - 1);
- end = ((unsigned long)start + size + linesz - 1) & ~(linesz - 1);
-
- for (v = begin; v < end; v += linesz)
- clean_dcache_line((void *)v);
- flush_write_buffer();
-}
-
-void flush_dcache_region(void *start, size_t size)
-{
- unsigned long v, begin, end, linesz;
-
- linesz = boot_cpu_data.dcache.linesz;
- begin = (unsigned long)start & ~(linesz - 1);
- end = ((unsigned long)start + size + linesz - 1) & ~(linesz - 1);
-
- for (v = begin; v < end; v += linesz)
- flush_dcache_line((void *)v);
- flush_write_buffer();
-}
-
-void invalidate_icache_region(void *start, size_t size)
-{
- unsigned long v, begin, end, linesz;
-
- linesz = boot_cpu_data.icache.linesz;
- begin = (unsigned long)start & ~(linesz - 1);
- end = ((unsigned long)start + size + linesz - 1) & ~(linesz - 1);
-
- for (v = begin; v < end; v += linesz)
- invalidate_icache_line((void *)v);
-}
-
-static inline void __flush_icache_range(unsigned long start, unsigned long end)
-{
- unsigned long v, linesz;
-
- linesz = boot_cpu_data.dcache.linesz;
- for (v = start; v < end; v += linesz) {
- clean_dcache_line((void *)v);
- invalidate_icache_line((void *)v);
- }
-
- flush_write_buffer();
-}
-
-/*
- * This one is called after a module has been loaded.
- */
-void flush_icache_range(unsigned long start, unsigned long end)
-{
- unsigned long linesz;
-
- linesz = boot_cpu_data.dcache.linesz;
- __flush_icache_range(start & ~(linesz - 1),
- (end + linesz - 1) & ~(linesz - 1));
-}
-EXPORT_SYMBOL(flush_icache_range);
-
-/*
- * This one is called from __do_fault() and do_swap_page().
- */
-void flush_icache_page(struct vm_area_struct *vma, struct page *page)
-{
- if (vma->vm_flags & VM_EXEC) {
- void *v = page_address(page);
- __flush_icache_range((unsigned long)v, (unsigned long)v + PAGE_SIZE);
- }
-}
-
-asmlinkage int sys_cacheflush(int operation, void __user *addr, size_t len)
-{
- int ret;
-
- if (len > CACHEFLUSH_MAX_LEN) {
- ret = -EPERM;
- if (!capable(CAP_SYS_ADMIN))
- goto out;
- }
-
- ret = -EFAULT;
- if (!access_ok(VERIFY_WRITE, addr, len))
- goto out;
-
- switch (operation) {
- case CACHE_IFLUSH:
- flush_icache_range((unsigned long)addr,
- (unsigned long)addr + len);
- ret = 0;
- break;
- default:
- ret = -EINVAL;
- }
-
-out:
- return ret;
-}
-
-void copy_to_user_page(struct vm_area_struct *vma, struct page *page,
- unsigned long vaddr, void *dst, const void *src,
- unsigned long len)
-{
- memcpy(dst, src, len);
- if (vma->vm_flags & VM_EXEC)
- flush_icache_range((unsigned long)dst,
- (unsigned long)dst + len);
-}
diff --git a/arch/avr32/mm/clear_page.S b/arch/avr32/mm/clear_page.S
deleted file mode 100644
index 5d70dca..0000000
--- a/arch/avr32/mm/clear_page.S
+++ /dev/null
@@ -1,25 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/linkage.h>
-#include <asm/page.h>
-
-/*
- * clear_page
- * r12: P1 address (to)
- */
- .text
- .global clear_page
-clear_page:
- sub r9, r12, -PAGE_SIZE
- mov r10, 0
- mov r11, 0
-0: st.d r12++, r10
- cp r12, r9
- brne 0b
- mov pc, lr
diff --git a/arch/avr32/mm/copy_page.S b/arch/avr32/mm/copy_page.S
deleted file mode 100644
index c2b3752..0000000
--- a/arch/avr32/mm/copy_page.S
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/linkage.h>
-#include <asm/page.h>
-
-/*
- * copy_page
- *
- * r12 to (P1 address)
- * r11 from (P1 address)
- * r8-r10 scratch
- */
- .text
- .global copy_page
-copy_page:
- sub r10, r11, -(1 << PAGE_SHIFT)
- /* pref r11[0] */
-1: /* pref r11[8] */
- ld.d r8, r11++
- st.d r12++, r8
- cp r11, r10
- brlo 1b
- mov pc, lr
diff --git a/arch/avr32/mm/dma-coherent.c b/arch/avr32/mm/dma-coherent.c
deleted file mode 100644
index 555222d..0000000
--- a/arch/avr32/mm/dma-coherent.c
+++ /dev/null
@@ -1,202 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/dma-mapping.h>
-#include <linux/gfp.h>
-#include <linux/export.h>
-#include <linux/mm.h>
-#include <linux/device.h>
-#include <linux/scatterlist.h>
-
-#include <asm/processor.h>
-#include <asm/cacheflush.h>
-#include <asm/io.h>
-#include <asm/addrspace.h>
-
-void dma_cache_sync(struct device *dev, void *vaddr, size_t size, int direction)
-{
- /*
- * No need to sync an uncached area
- */
- if (PXSEG(vaddr) == P2SEG)
- return;
-
- switch (direction) {
- case DMA_FROM_DEVICE: /* invalidate only */
- invalidate_dcache_region(vaddr, size);
- break;
- case DMA_TO_DEVICE: /* writeback only */
- clean_dcache_region(vaddr, size);
- break;
- case DMA_BIDIRECTIONAL: /* writeback and invalidate */
- flush_dcache_region(vaddr, size);
- break;
- default:
- BUG();
- }
-}
-EXPORT_SYMBOL(dma_cache_sync);
-
-static struct page *__dma_alloc(struct device *dev, size_t size,
- dma_addr_t *handle, gfp_t gfp)
-{
- struct page *page, *free, *end;
- int order;
-
- /* Following is a work-around (a.k.a. hack) to prevent pages
- * with __GFP_COMP being passed to split_page() which cannot
- * handle them. The real problem is that this flag probably
- * should be 0 on AVR32 as it is not supported on this
- * platform--see CONFIG_HUGETLB_PAGE. */
- gfp &= ~(__GFP_COMP);
-
- size = PAGE_ALIGN(size);
- order = get_order(size);
-
- page = alloc_pages(gfp, order);
- if (!page)
- return NULL;
- split_page(page, order);
-
- /*
- * When accessing physical memory with valid cache data, we
- * get a cache hit even if the virtual memory region is marked
- * as uncached.
- *
- * Since the memory is newly allocated, there is no point in
- * doing a writeback. If the previous owner cares, he should
- * have flushed the cache before releasing the memory.
- */
- invalidate_dcache_region(phys_to_virt(page_to_phys(page)), size);
-
- *handle = page_to_bus(page);
- free = page + (size >> PAGE_SHIFT);
- end = page + (1 << order);
-
- /*
- * Free any unused pages
- */
- while (free < end) {
- __free_page(free);
- free++;
- }
-
- return page;
-}
-
-static void __dma_free(struct device *dev, size_t size,
- struct page *page, dma_addr_t handle)
-{
- struct page *end = page + (PAGE_ALIGN(size) >> PAGE_SHIFT);
-
- while (page < end)
- __free_page(page++);
-}
-
-static void *avr32_dma_alloc(struct device *dev, size_t size,
- dma_addr_t *handle, gfp_t gfp, unsigned long attrs)
-{
- struct page *page;
- dma_addr_t phys;
-
- page = __dma_alloc(dev, size, handle, gfp);
- if (!page)
- return NULL;
- phys = page_to_phys(page);
-
- if (attrs & DMA_ATTR_WRITE_COMBINE) {
- /* Now, map the page into P3 with write-combining turned on */
- *handle = phys;
- return __ioremap(phys, size, _PAGE_BUFFER);
- } else {
- return phys_to_uncached(phys);
- }
-}
-
-static void avr32_dma_free(struct device *dev, size_t size,
- void *cpu_addr, dma_addr_t handle, unsigned long attrs)
-{
- struct page *page;
-
- if (attrs & DMA_ATTR_WRITE_COMBINE) {
- iounmap(cpu_addr);
-
- page = phys_to_page(handle);
- } else {
- void *addr = phys_to_cached(uncached_to_phys(cpu_addr));
-
- pr_debug("avr32_dma_free addr %p (phys %08lx) size %u\n",
- cpu_addr, (unsigned long)handle, (unsigned)size);
-
- BUG_ON(!virt_addr_valid(addr));
- page = virt_to_page(addr);
- }
-
- __dma_free(dev, size, page, handle);
-}
-
-static dma_addr_t avr32_dma_map_page(struct device *dev, struct page *page,
- unsigned long offset, size_t size,
- enum dma_data_direction direction, unsigned long attrs)
-{
- void *cpu_addr = page_address(page) + offset;
-
- if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC))
- dma_cache_sync(dev, cpu_addr, size, direction);
- return virt_to_bus(cpu_addr);
-}
-
-static int avr32_dma_map_sg(struct device *dev, struct scatterlist *sglist,
- int nents, enum dma_data_direction direction,
- unsigned long attrs)
-{
- int i;
- struct scatterlist *sg;
-
- for_each_sg(sglist, sg, nents, i) {
- char *virt;
-
- sg->dma_address = page_to_bus(sg_page(sg)) + sg->offset;
- virt = sg_virt(sg);
-
- if (attrs & DMA_ATTR_SKIP_CPU_SYNC)
- continue;
-
- dma_cache_sync(dev, virt, sg->length, direction);
- }
-
- return nents;
-}
-
-static void avr32_dma_sync_single_for_device(struct device *dev,
- dma_addr_t dma_handle, size_t size,
- enum dma_data_direction direction)
-{
- dma_cache_sync(dev, bus_to_virt(dma_handle), size, direction);
-}
-
-static void avr32_dma_sync_sg_for_device(struct device *dev,
- struct scatterlist *sglist, int nents,
- enum dma_data_direction direction)
-{
- int i;
- struct scatterlist *sg;
-
- for_each_sg(sglist, sg, nents, i)
- dma_cache_sync(dev, sg_virt(sg), sg->length, direction);
-}
-
-const struct dma_map_ops avr32_dma_ops = {
- .alloc = avr32_dma_alloc,
- .free = avr32_dma_free,
- .map_page = avr32_dma_map_page,
- .map_sg = avr32_dma_map_sg,
- .sync_single_for_device = avr32_dma_sync_single_for_device,
- .sync_sg_for_device = avr32_dma_sync_sg_for_device,
-};
-EXPORT_SYMBOL(avr32_dma_ops);
diff --git a/arch/avr32/mm/fault.c b/arch/avr32/mm/fault.c
deleted file mode 100644
index b3977e9..0000000
--- a/arch/avr32/mm/fault.c
+++ /dev/null
@@ -1,268 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * Based on linux/arch/sh/mm/fault.c:
- * Copyright (C) 1999 Niibe Yutaka
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/mm.h>
-#include <linux/extable.h>
-#include <linux/pagemap.h>
-#include <linux/kdebug.h>
-#include <linux/kprobes.h>
-#include <linux/uaccess.h>
-
-#include <asm/mmu_context.h>
-#include <asm/sysreg.h>
-#include <asm/tlb.h>
-
-#ifdef CONFIG_KPROBES
-static inline int notify_page_fault(struct pt_regs *regs, int trap)
-{
- int ret = 0;
-
- if (!user_mode(regs)) {
- if (kprobe_running() && kprobe_fault_handler(regs, trap))
- ret = 1;
- }
-
- return ret;
-}
-#else
-static inline int notify_page_fault(struct pt_regs *regs, int trap)
-{
- return 0;
-}
-#endif
-
-int exception_trace = 1;
-
-/*
- * This routine handles page faults. It determines the address and the
- * problem, and then passes it off to one of the appropriate routines.
- *
- * ecr is the Exception Cause Register. Possible values are:
- * 6: Protection fault (instruction access)
- * 15: Protection fault (read access)
- * 16: Protection fault (write access)
- * 20: Page not found (instruction access)
- * 24: Page not found (read access)
- * 28: Page not found (write access)
- */
-asmlinkage void do_page_fault(unsigned long ecr, struct pt_regs *regs)
-{
- struct task_struct *tsk;
- struct mm_struct *mm;
- struct vm_area_struct *vma;
- const struct exception_table_entry *fixup;
- unsigned long address;
- unsigned long page;
- long signr;
- int code;
- int fault;
- unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
-
- if (notify_page_fault(regs, ecr))
- return;
-
- address = sysreg_read(TLBEAR);
-
- tsk = current;
- mm = tsk->mm;
-
- signr = SIGSEGV;
- code = SEGV_MAPERR;
-
- /*
- * If we're in an interrupt or have no user context, we must
- * not take the fault...
- */
- if (faulthandler_disabled() || !mm || regs->sr & SYSREG_BIT(GM))
- goto no_context;
-
- local_irq_enable();
-
- if (user_mode(regs))
- flags |= FAULT_FLAG_USER;
-retry:
- down_read(&mm->mmap_sem);
-
- vma = find_vma(mm, address);
- if (!vma)
- goto bad_area;
- if (vma->vm_start <= address)
- goto good_area;
- if (!(vma->vm_flags & VM_GROWSDOWN))
- goto bad_area;
- if (expand_stack(vma, address))
- goto bad_area;
-
- /*
- * Ok, we have a good vm_area for this memory access, so we
- * can handle it...
- */
-good_area:
- code = SEGV_ACCERR;
-
- switch (ecr) {
- case ECR_PROTECTION_X:
- case ECR_TLB_MISS_X:
- if (!(vma->vm_flags & VM_EXEC))
- goto bad_area;
- break;
- case ECR_PROTECTION_R:
- case ECR_TLB_MISS_R:
- if (!(vma->vm_flags & (VM_READ | VM_WRITE | VM_EXEC)))
- goto bad_area;
- break;
- case ECR_PROTECTION_W:
- case ECR_TLB_MISS_W:
- if (!(vma->vm_flags & VM_WRITE))
- goto bad_area;
- flags |= FAULT_FLAG_WRITE;
- break;
- default:
- panic("Unhandled case %lu in do_page_fault!", ecr);
- }
-
- /*
- * If for any reason at all we couldn't handle the fault, make
- * sure we exit gracefully rather than endlessly redo the
- * fault.
- */
- fault = handle_mm_fault(vma, address, flags);
-
- if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
- return;
-
- if (unlikely(fault & VM_FAULT_ERROR)) {
- if (fault & VM_FAULT_OOM)
- goto out_of_memory;
- else if (fault & VM_FAULT_SIGSEGV)
- goto bad_area;
- else if (fault & VM_FAULT_SIGBUS)
- goto do_sigbus;
- BUG();
- }
-
- if (flags & FAULT_FLAG_ALLOW_RETRY) {
- if (fault & VM_FAULT_MAJOR)
- tsk->maj_flt++;
- else
- tsk->min_flt++;
- if (fault & VM_FAULT_RETRY) {
- flags &= ~FAULT_FLAG_ALLOW_RETRY;
- flags |= FAULT_FLAG_TRIED;
-
- /*
- * No need to up_read(&mm->mmap_sem) as we would have
- * already released it in __lock_page_or_retry() in
- * mm/filemap.c.
- */
- goto retry;
- }
- }
-
- up_read(&mm->mmap_sem);
- return;
-
- /*
- * Something tried to access memory that isn't in our memory
- * map. Fix it, but check if it's kernel or user first...
- */
-bad_area:
- up_read(&mm->mmap_sem);
-
- if (user_mode(regs)) {
- if (exception_trace && printk_ratelimit())
- printk("%s%s[%d]: segfault at %08lx pc %08lx "
- "sp %08lx ecr %lu\n",
- is_global_init(tsk) ? KERN_EMERG : KERN_INFO,
- tsk->comm, tsk->pid, address, regs->pc,
- regs->sp, ecr);
- _exception(SIGSEGV, regs, code, address);
- return;
- }
-
-no_context:
- /* Are we prepared to handle this kernel fault? */
- fixup = search_exception_tables(regs->pc);
- if (fixup) {
- regs->pc = fixup->fixup;
- return;
- }
-
- /*
- * Oops. The kernel tried to access some bad page. We'll have
- * to terminate things with extreme prejudice.
- */
- if (address < PAGE_SIZE)
- printk(KERN_ALERT
- "Unable to handle kernel NULL pointer dereference");
- else
- printk(KERN_ALERT
- "Unable to handle kernel paging request");
- printk(" at virtual address %08lx\n", address);
-
- page = sysreg_read(PTBR);
- printk(KERN_ALERT "ptbr = %08lx", page);
- if (address >= TASK_SIZE)
- page = (unsigned long)swapper_pg_dir;
- if (page) {
- page = ((unsigned long *)page)[address >> 22];
- printk(" pgd = %08lx", page);
- if (page & _PAGE_PRESENT) {
- page &= PAGE_MASK;
- address &= 0x003ff000;
- page = ((unsigned long *)__va(page))[address >> PAGE_SHIFT];
- printk(" pte = %08lx", page);
- }
- }
- printk("\n");
- die("Kernel access of bad area", regs, signr);
- return;
-
- /*
- * We ran out of memory, or some other thing happened to us
- * that made us unable to handle the page fault gracefully.
- */
-out_of_memory:
- up_read(&mm->mmap_sem);
- if (!user_mode(regs))
- goto no_context;
- pagefault_out_of_memory();
- return;
-
-do_sigbus:
- up_read(&mm->mmap_sem);
-
- /* Kernel mode? Handle exceptions or die */
- signr = SIGBUS;
- code = BUS_ADRERR;
- if (!user_mode(regs))
- goto no_context;
-
- if (exception_trace)
- printk("%s%s[%d]: bus error at %08lx pc %08lx "
- "sp %08lx ecr %lu\n",
- is_global_init(tsk) ? KERN_EMERG : KERN_INFO,
- tsk->comm, tsk->pid, address, regs->pc,
- regs->sp, ecr);
-
- _exception(SIGBUS, regs, BUS_ADRERR, address);
-}
-
-asmlinkage void do_bus_error(unsigned long addr, int write_access,
- struct pt_regs *regs)
-{
- printk(KERN_ALERT
- "Bus error at physical address 0x%08lx (%s access)\n",
- addr, write_access ? "write" : "read");
- printk(KERN_INFO "DTLB dump:\n");
- dump_dtlb();
- die("Bus Error", regs, SIGKILL);
-}
diff --git a/arch/avr32/mm/init.c b/arch/avr32/mm/init.c
deleted file mode 100644
index def5391..0000000
--- a/arch/avr32/mm/init.c
+++ /dev/null
@@ -1,125 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/kernel.h>
-#include <linux/gfp.h>
-#include <linux/mm.h>
-#include <linux/swap.h>
-#include <linux/init.h>
-#include <linux/mmzone.h>
-#include <linux/module.h>
-#include <linux/bootmem.h>
-#include <linux/pagemap.h>
-#include <linux/nodemask.h>
-
-#include <asm/page.h>
-#include <asm/mmu_context.h>
-#include <asm/tlb.h>
-#include <asm/io.h>
-#include <asm/dma.h>
-#include <asm/setup.h>
-#include <asm/sections.h>
-
-pgd_t swapper_pg_dir[PTRS_PER_PGD] __page_aligned_data;
-
-struct page *empty_zero_page;
-EXPORT_SYMBOL(empty_zero_page);
-
-/*
- * Cache of MMU context last used.
- */
-unsigned long mmu_context_cache = NO_CONTEXT;
-
-/*
- * paging_init() sets up the page tables
- *
- * This routine also unmaps the page at virtual kernel address 0, so
- * that we can trap those pesky NULL-reference errors in the kernel.
- */
-void __init paging_init(void)
-{
- extern unsigned long _evba;
- void *zero_page;
- int nid;
-
- /*
- * Make sure we can handle exceptions before enabling
- * paging. Not that we should ever _get_ any exceptions this
- * early, but you never know...
- */
- printk("Exception vectors start at %p\n", &_evba);
- sysreg_write(EVBA, (unsigned long)&_evba);
-
- /*
- * Since we are ready to handle exceptions now, we should let
- * the CPU generate them...
- */
- __asm__ __volatile__ ("csrf %0" : : "i"(SR_EM_BIT));
-
- /*
- * Allocate the zero page. The allocator will panic if it
- * can't satisfy the request, so no need to check.
- */
- zero_page = alloc_bootmem_low_pages_node(NODE_DATA(0),
- PAGE_SIZE);
-
- sysreg_write(PTBR, (unsigned long)swapper_pg_dir);
- enable_mmu();
- printk ("CPU: Paging enabled\n");
-
- for_each_online_node(nid) {
- pg_data_t *pgdat = NODE_DATA(nid);
- unsigned long zones_size[MAX_NR_ZONES];
- unsigned long low, start_pfn;
-
- start_pfn = pgdat->bdata->node_min_pfn;
- low = pgdat->bdata->node_low_pfn;
-
- memset(zones_size, 0, sizeof(zones_size));
- zones_size[ZONE_NORMAL] = low - start_pfn;
-
- printk("Node %u: start_pfn = 0x%lx, low = 0x%lx\n",
- nid, start_pfn, low);
-
- free_area_init_node(nid, zones_size, start_pfn, NULL);
-
- printk("Node %u: mem_map starts at %p\n",
- pgdat->node_id, pgdat->node_mem_map);
- }
-
- mem_map = NODE_DATA(0)->node_mem_map;
-
- empty_zero_page = virt_to_page(zero_page);
- flush_dcache_page(empty_zero_page);
-}
-
-void __init mem_init(void)
-{
- pg_data_t *pgdat;
-
- high_memory = NULL;
- for_each_online_pgdat(pgdat)
- high_memory = max_t(void *, high_memory,
- __va(pgdat_end_pfn(pgdat) << PAGE_SHIFT));
-
- set_max_mapnr(MAP_NR(high_memory));
- free_all_bootmem();
- mem_init_print_info(NULL);
-}
-
-void free_initmem(void)
-{
- free_initmem_default(-1);
-}
-
-#ifdef CONFIG_BLK_DEV_INITRD
-void free_initrd_mem(unsigned long start, unsigned long end)
-{
- free_reserved_area((void *)start, (void *)end, -1, "initrd");
-}
-#endif
diff --git a/arch/avr32/mm/ioremap.c b/arch/avr32/mm/ioremap.c
deleted file mode 100644
index 7def0d8..0000000
--- a/arch/avr32/mm/ioremap.c
+++ /dev/null
@@ -1,93 +0,0 @@
-/*
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/vmalloc.h>
-#include <linux/mm.h>
-#include <linux/module.h>
-#include <linux/io.h>
-#include <linux/slab.h>
-
-#include <asm/pgtable.h>
-#include <asm/addrspace.h>
-
-/*
- * Re-map an arbitrary physical address space into the kernel virtual
- * address space. Needed when the kernel wants to access physical
- * memory directly.
- */
-void __iomem *__ioremap(unsigned long phys_addr, size_t size,
- unsigned long flags)
-{
- unsigned long addr;
- struct vm_struct *area;
- unsigned long offset, last_addr;
- pgprot_t prot;
-
- /*
- * Check if we can simply use the P4 segment. This area is
- * uncacheable, so if caching/buffering is requested, we can't
- * use it.
- */
- if ((phys_addr >= P4SEG) && (flags == 0))
- return (void __iomem *)phys_addr;
-
- /* Don't allow wraparound or zero size */
- last_addr = phys_addr + size - 1;
- if (!size || last_addr < phys_addr)
- return NULL;
-
- /*
- * XXX: When mapping regular RAM, we'd better make damn sure
- * it's never used for anything else. But this is really the
- * caller's responsibility...
- */
- if (PHYSADDR(P2SEGADDR(phys_addr)) == phys_addr)
- return (void __iomem *)P2SEGADDR(phys_addr);
-
- /* Mappings have to be page-aligned */
- offset = phys_addr & ~PAGE_MASK;
- phys_addr &= PAGE_MASK;
- size = PAGE_ALIGN(last_addr + 1) - phys_addr;
-
- prot = __pgprot(_PAGE_PRESENT | _PAGE_GLOBAL | _PAGE_RW | _PAGE_DIRTY
- | _PAGE_ACCESSED | _PAGE_TYPE_SMALL | flags);
-
- /*
- * Ok, go for it..
- */
- area = get_vm_area(size, VM_IOREMAP);
- if (!area)
- return NULL;
- area->phys_addr = phys_addr;
- addr = (unsigned long )area->addr;
- if (ioremap_page_range(addr, addr + size, phys_addr, prot)) {
- vunmap((void *)addr);
- return NULL;
- }
-
- return (void __iomem *)(offset + (char *)addr);
-}
-EXPORT_SYMBOL(__ioremap);
-
-void __iounmap(void __iomem *addr)
-{
- struct vm_struct *p;
-
- if ((unsigned long)addr >= P4SEG)
- return;
- if (PXSEG(addr) == P2SEG)
- return;
-
- p = remove_vm_area((void *)(PAGE_MASK & (unsigned long __force)addr));
- if (unlikely(!p)) {
- printk (KERN_ERR "iounmap: bad address %p\n", addr);
- return;
- }
-
- kfree (p);
-}
-EXPORT_SYMBOL(__iounmap);
diff --git a/arch/avr32/mm/tlb.c b/arch/avr32/mm/tlb.c
deleted file mode 100644
index 0da2310..0000000
--- a/arch/avr32/mm/tlb.c
+++ /dev/null
@@ -1,375 +0,0 @@
-/*
- * AVR32 TLB operations
- *
- * Copyright (C) 2004-2006 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-#include <linux/mm.h>
-
-#include <asm/mmu_context.h>
-
-/* TODO: Get the correct number from the CONFIG1 system register */
-#define NR_TLB_ENTRIES 32
-
-static void show_dtlb_entry(unsigned int index)
-{
- u32 tlbehi, tlbehi_save, tlbelo, mmucr, mmucr_save;
- unsigned long flags;
-
- local_irq_save(flags);
- mmucr_save = sysreg_read(MMUCR);
- tlbehi_save = sysreg_read(TLBEHI);
- mmucr = SYSREG_BFINS(DRP, index, mmucr_save);
- sysreg_write(MMUCR, mmucr);
-
- __builtin_tlbr();
- cpu_sync_pipeline();
-
- tlbehi = sysreg_read(TLBEHI);
- tlbelo = sysreg_read(TLBELO);
-
- printk("%2u: %c %c %02x %05x %05x %o %o %c %c %c %c\n",
- index,
- SYSREG_BFEXT(TLBEHI_V, tlbehi) ? '1' : '0',
- SYSREG_BFEXT(G, tlbelo) ? '1' : '0',
- SYSREG_BFEXT(ASID, tlbehi),
- SYSREG_BFEXT(VPN, tlbehi) >> 2,
- SYSREG_BFEXT(PFN, tlbelo) >> 2,
- SYSREG_BFEXT(AP, tlbelo),
- SYSREG_BFEXT(SZ, tlbelo),
- SYSREG_BFEXT(TLBELO_C, tlbelo) ? 'C' : ' ',
- SYSREG_BFEXT(B, tlbelo) ? 'B' : ' ',
- SYSREG_BFEXT(W, tlbelo) ? 'W' : ' ',
- SYSREG_BFEXT(TLBELO_D, tlbelo) ? 'D' : ' ');
-
- sysreg_write(MMUCR, mmucr_save);
- sysreg_write(TLBEHI, tlbehi_save);
- cpu_sync_pipeline();
- local_irq_restore(flags);
-}
-
-void dump_dtlb(void)
-{
- unsigned int i;
-
- printk("ID V G ASID VPN PFN AP SZ C B W D\n");
- for (i = 0; i < NR_TLB_ENTRIES; i++)
- show_dtlb_entry(i);
-}
-
-static void update_dtlb(unsigned long address, pte_t pte)
-{
- u32 tlbehi;
- u32 mmucr;
-
- /*
- * We're not changing the ASID here, so no need to flush the
- * pipeline.
- */
- tlbehi = sysreg_read(TLBEHI);
- tlbehi = SYSREG_BF(ASID, SYSREG_BFEXT(ASID, tlbehi));
- tlbehi |= address & MMU_VPN_MASK;
- tlbehi |= SYSREG_BIT(TLBEHI_V);
- sysreg_write(TLBEHI, tlbehi);
-
- /* Does this mapping already exist? */
- __builtin_tlbs();
- mmucr = sysreg_read(MMUCR);
-
- if (mmucr & SYSREG_BIT(MMUCR_N)) {
- /* Not found -- pick a not-recently-accessed entry */
- unsigned int rp;
- u32 tlbar = sysreg_read(TLBARLO);
-
- rp = 32 - fls(tlbar);
- if (rp == 32) {
- rp = 0;
- sysreg_write(TLBARLO, -1L);
- }
-
- mmucr = SYSREG_BFINS(DRP, rp, mmucr);
- sysreg_write(MMUCR, mmucr);
- }
-
- sysreg_write(TLBELO, pte_val(pte) & _PAGE_FLAGS_HARDWARE_MASK);
-
- /* Let's go */
- __builtin_tlbw();
-}
-
-void update_mmu_cache(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep)
-{
- unsigned long flags;
-
- /* ptrace may call this routine */
- if (vma && current->active_mm != vma->vm_mm)
- return;
-
- local_irq_save(flags);
- update_dtlb(address, *ptep);
- local_irq_restore(flags);
-}
-
-static void __flush_tlb_page(unsigned long asid, unsigned long page)
-{
- u32 mmucr, tlbehi;
-
- /*
- * Caller is responsible for masking out non-PFN bits in page
- * and changing the current ASID if necessary. This means that
- * we don't need to flush the pipeline after writing TLBEHI.
- */
- tlbehi = page | asid;
- sysreg_write(TLBEHI, tlbehi);
-
- __builtin_tlbs();
- mmucr = sysreg_read(MMUCR);
-
- if (!(mmucr & SYSREG_BIT(MMUCR_N))) {
- unsigned int entry;
- u32 tlbarlo;
-
- /* Clear the "valid" bit */
- sysreg_write(TLBEHI, tlbehi);
-
- /* mark the entry as "not accessed" */
- entry = SYSREG_BFEXT(DRP, mmucr);
- tlbarlo = sysreg_read(TLBARLO);
- tlbarlo |= (0x80000000UL >> entry);
- sysreg_write(TLBARLO, tlbarlo);
-
- /* update the entry with valid bit clear */
- __builtin_tlbw();
- }
-}
-
-void flush_tlb_page(struct vm_area_struct *vma, unsigned long page)
-{
- if (vma->vm_mm && vma->vm_mm->context != NO_CONTEXT) {
- unsigned long flags, asid;
- unsigned long saved_asid = MMU_NO_ASID;
-
- asid = vma->vm_mm->context & MMU_CONTEXT_ASID_MASK;
- page &= PAGE_MASK;
-
- local_irq_save(flags);
- if (vma->vm_mm != current->mm) {
- saved_asid = get_asid();
- set_asid(asid);
- }
-
- __flush_tlb_page(asid, page);
-
- if (saved_asid != MMU_NO_ASID)
- set_asid(saved_asid);
- local_irq_restore(flags);
- }
-}
-
-void flush_tlb_range(struct vm_area_struct *vma, unsigned long start,
- unsigned long end)
-{
- struct mm_struct *mm = vma->vm_mm;
-
- if (mm->context != NO_CONTEXT) {
- unsigned long flags;
- int size;
-
- local_irq_save(flags);
- size = (end - start + (PAGE_SIZE - 1)) >> PAGE_SHIFT;
-
- if (size > (MMU_DTLB_ENTRIES / 4)) { /* Too many entries to flush */
- mm->context = NO_CONTEXT;
- if (mm == current->mm)
- activate_context(mm);
- } else {
- unsigned long asid;
- unsigned long saved_asid;
-
- asid = mm->context & MMU_CONTEXT_ASID_MASK;
- saved_asid = MMU_NO_ASID;
-
- start &= PAGE_MASK;
- end += (PAGE_SIZE - 1);
- end &= PAGE_MASK;
-
- if (mm != current->mm) {
- saved_asid = get_asid();
- set_asid(asid);
- }
-
- while (start < end) {
- __flush_tlb_page(asid, start);
- start += PAGE_SIZE;
- }
- if (saved_asid != MMU_NO_ASID)
- set_asid(saved_asid);
- }
- local_irq_restore(flags);
- }
-}
-
-/*
- * This function depends on the pages to be flushed having the G
- * (global) bit set in their pte. This is true for all
- * PAGE_KERNEL(_RO) pages.
- */
-void flush_tlb_kernel_range(unsigned long start, unsigned long end)
-{
- unsigned long flags;
- int size;
-
- size = (end - start + (PAGE_SIZE - 1)) >> PAGE_SHIFT;
- if (size > (MMU_DTLB_ENTRIES / 4)) { /* Too many entries to flush */
- flush_tlb_all();
- } else {
- unsigned long asid;
-
- local_irq_save(flags);
- asid = get_asid();
-
- start &= PAGE_MASK;
- end += (PAGE_SIZE - 1);
- end &= PAGE_MASK;
-
- while (start < end) {
- __flush_tlb_page(asid, start);
- start += PAGE_SIZE;
- }
- local_irq_restore(flags);
- }
-}
-
-void flush_tlb_mm(struct mm_struct *mm)
-{
- /* Invalidate all TLB entries of this process by getting a new ASID */
- if (mm->context != NO_CONTEXT) {
- unsigned long flags;
-
- local_irq_save(flags);
- mm->context = NO_CONTEXT;
- if (mm == current->mm)
- activate_context(mm);
- local_irq_restore(flags);
- }
-}
-
-void flush_tlb_all(void)
-{
- unsigned long flags;
-
- local_irq_save(flags);
- sysreg_write(MMUCR, sysreg_read(MMUCR) | SYSREG_BIT(MMUCR_I));
- local_irq_restore(flags);
-}
-
-#ifdef CONFIG_PROC_FS
-
-#include <linux/seq_file.h>
-#include <linux/proc_fs.h>
-#include <linux/init.h>
-
-static void *tlb_start(struct seq_file *tlb, loff_t *pos)
-{
- static unsigned long tlb_index;
-
- if (*pos >= NR_TLB_ENTRIES)
- return NULL;
-
- tlb_index = 0;
- return &tlb_index;
-}
-
-static void *tlb_next(struct seq_file *tlb, void *v, loff_t *pos)
-{
- unsigned long *index = v;
-
- if (*index >= NR_TLB_ENTRIES - 1)
- return NULL;
-
- ++*pos;
- ++*index;
- return index;
-}
-
-static void tlb_stop(struct seq_file *tlb, void *v)
-{
-
-}
-
-static int tlb_show(struct seq_file *tlb, void *v)
-{
- unsigned int tlbehi, tlbehi_save, tlbelo, mmucr, mmucr_save;
- unsigned long flags;
- unsigned long *index = v;
-
- if (*index == 0)
- seq_puts(tlb, "ID V G ASID VPN PFN AP SZ C B W D\n");
-
- BUG_ON(*index >= NR_TLB_ENTRIES);
-
- local_irq_save(flags);
- mmucr_save = sysreg_read(MMUCR);
- tlbehi_save = sysreg_read(TLBEHI);
- mmucr = SYSREG_BFINS(DRP, *index, mmucr_save);
- sysreg_write(MMUCR, mmucr);
-
- /* TLBR might change the ASID */
- __builtin_tlbr();
- cpu_sync_pipeline();
-
- tlbehi = sysreg_read(TLBEHI);
- tlbelo = sysreg_read(TLBELO);
-
- sysreg_write(MMUCR, mmucr_save);
- sysreg_write(TLBEHI, tlbehi_save);
- cpu_sync_pipeline();
- local_irq_restore(flags);
-
- seq_printf(tlb, "%2lu: %c %c %02x %05x %05x %o %o %c %c %c %c\n",
- *index,
- SYSREG_BFEXT(TLBEHI_V, tlbehi) ? '1' : '0',
- SYSREG_BFEXT(G, tlbelo) ? '1' : '0',
- SYSREG_BFEXT(ASID, tlbehi),
- SYSREG_BFEXT(VPN, tlbehi) >> 2,
- SYSREG_BFEXT(PFN, tlbelo) >> 2,
- SYSREG_BFEXT(AP, tlbelo),
- SYSREG_BFEXT(SZ, tlbelo),
- SYSREG_BFEXT(TLBELO_C, tlbelo) ? '1' : '0',
- SYSREG_BFEXT(B, tlbelo) ? '1' : '0',
- SYSREG_BFEXT(W, tlbelo) ? '1' : '0',
- SYSREG_BFEXT(TLBELO_D, tlbelo) ? '1' : '0');
-
- return 0;
-}
-
-static const struct seq_operations tlb_ops = {
- .start = tlb_start,
- .next = tlb_next,
- .stop = tlb_stop,
- .show = tlb_show,
-};
-
-static int tlb_open(struct inode *inode, struct file *file)
-{
- return seq_open(file, &tlb_ops);
-}
-
-static const struct file_operations proc_tlb_operations = {
- .open = tlb_open,
- .read = seq_read,
- .llseek = seq_lseek,
- .release = seq_release,
-};
-
-static int __init proctlb_init(void)
-{
- proc_create("tlb", 0, NULL, &proc_tlb_operations);
- return 0;
-}
-late_initcall(proctlb_init);
-#endif /* CONFIG_PROC_FS */
diff --git a/arch/avr32/oprofile/Makefile b/arch/avr32/oprofile/Makefile
deleted file mode 100644
index e0eb520..0000000
--- a/arch/avr32/oprofile/Makefile
+++ /dev/null
@@ -1,8 +0,0 @@
-obj-$(CONFIG_OPROFILE) += oprofile.o
-
-oprofile-y := $(addprefix ../../../drivers/oprofile/, \
- oprof.o cpu_buffer.o buffer_sync.o \
- event_buffer.o oprofile_files.o \
- oprofilefs.o oprofile_stats.o \
- timer_int.o)
-oprofile-y += op_model_avr32.o backtrace.o
diff --git a/arch/avr32/oprofile/backtrace.c b/arch/avr32/oprofile/backtrace.c
deleted file mode 100644
index 29cf2f1..0000000
--- a/arch/avr32/oprofile/backtrace.c
+++ /dev/null
@@ -1,81 +0,0 @@
-/*
- * AVR32 specific backtracing code for oprofile
- *
- * Copyright 2008 Weinmann GmbH
- *
- * Author: Nikolaus Voss <n.voss@weinmann.de>
- *
- * Based on i386 oprofile backtrace code by John Levon and David Smith
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- */
-
-#include <linux/oprofile.h>
-#include <linux/ptrace.h>
-#include <linux/uaccess.h>
-
-/* The first two words of each frame on the stack look like this if we have
- * frame pointers */
-struct frame_head {
- unsigned long lr;
- struct frame_head *fp;
-};
-
-/* copied from arch/avr32/kernel/process.c */
-static inline int valid_stack_ptr(struct thread_info *tinfo, unsigned long p)
-{
- return (p > (unsigned long)tinfo)
- && (p < (unsigned long)tinfo + THREAD_SIZE - 3);
-}
-
-/* copied from arch/x86/oprofile/backtrace.c */
-static struct frame_head *dump_user_backtrace(struct frame_head *head)
-{
- struct frame_head bufhead[2];
-
- /* Also check accessibility of one struct frame_head beyond */
- if (!access_ok(VERIFY_READ, head, sizeof(bufhead)))
- return NULL;
- if (__copy_from_user_inatomic(bufhead, head, sizeof(bufhead)))
- return NULL;
-
- oprofile_add_trace(bufhead[0].lr);
-
- /* frame pointers should strictly progress back up the stack
- * (towards higher addresses) */
- if (bufhead[0].fp <= head)
- return NULL;
-
- return bufhead[0].fp;
-}
-
-void avr32_backtrace(struct pt_regs * const regs, unsigned int depth)
-{
- /* Get first frame pointer */
- struct frame_head *head = (struct frame_head *)(regs->r7);
-
- if (!user_mode(regs)) {
-#ifdef CONFIG_FRAME_POINTER
- /*
- * Traverse the kernel stack from frame to frame up to
- * "depth" steps.
- */
- while (depth-- && valid_stack_ptr(task_thread_info(current),
- (unsigned long)head)) {
- oprofile_add_trace(head->lr);
- if (head->fp <= head)
- break;
- head = head->fp;
- }
-#endif
- } else {
- /* Assume we have frame pointers in user mode process */
- while (depth-- && head)
- head = dump_user_backtrace(head);
- }
-}
-
-
diff --git a/arch/avr32/oprofile/op_model_avr32.c b/arch/avr32/oprofile/op_model_avr32.c
deleted file mode 100644
index 08308be..0000000
--- a/arch/avr32/oprofile/op_model_avr32.c
+++ /dev/null
@@ -1,236 +0,0 @@
-/*
- * AVR32 Performance Counter Driver
- *
- * Copyright (C) 2005-2007 Atmel Corporation
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * Author: Ronny Pedersen
- */
-#include <linux/errno.h>
-#include <linux/interrupt.h>
-#include <linux/irq.h>
-#include <linux/oprofile.h>
-#include <linux/sched.h>
-#include <linux/types.h>
-
-#include <asm/sysreg.h>
-
-#define AVR32_PERFCTR_IRQ_GROUP 0
-#define AVR32_PERFCTR_IRQ_LINE 1
-
-void avr32_backtrace(struct pt_regs * const regs, unsigned int depth);
-
-enum { PCCNT, PCNT0, PCNT1, NR_counter };
-
-struct avr32_perf_counter {
- unsigned long enabled;
- unsigned long event;
- unsigned long count;
- unsigned long unit_mask;
- unsigned long kernel;
- unsigned long user;
-
- u32 ie_mask;
- u32 flag_mask;
-};
-
-static struct avr32_perf_counter counter[NR_counter] = {
- {
- .ie_mask = SYSREG_BIT(IEC),
- .flag_mask = SYSREG_BIT(FC),
- }, {
- .ie_mask = SYSREG_BIT(IE0),
- .flag_mask = SYSREG_BIT(F0),
- }, {
- .ie_mask = SYSREG_BIT(IE1),
- .flag_mask = SYSREG_BIT(F1),
- },
-};
-
-static void avr32_perf_counter_reset(void)
-{
- /* Reset all counter and disable/clear all interrupts */
- sysreg_write(PCCR, (SYSREG_BIT(PCCR_R)
- | SYSREG_BIT(PCCR_C)
- | SYSREG_BIT(FC)
- | SYSREG_BIT(F0)
- | SYSREG_BIT(F1)));
-}
-
-static irqreturn_t avr32_perf_counter_interrupt(int irq, void *dev_id)
-{
- struct avr32_perf_counter *ctr = dev_id;
- struct pt_regs *regs;
- u32 pccr;
-
- if (likely(!(intc_get_pending(AVR32_PERFCTR_IRQ_GROUP)
- & (1 << AVR32_PERFCTR_IRQ_LINE))))
- return IRQ_NONE;
-
- regs = get_irq_regs();
- pccr = sysreg_read(PCCR);
-
- /* Clear the interrupt flags we're about to handle */
- sysreg_write(PCCR, pccr);
-
- /* PCCNT */
- if (ctr->enabled && (pccr & ctr->flag_mask)) {
- sysreg_write(PCCNT, -ctr->count);
- oprofile_add_sample(regs, PCCNT);
- }
- ctr++;
- /* PCNT0 */
- if (ctr->enabled && (pccr & ctr->flag_mask)) {
- sysreg_write(PCNT0, -ctr->count);
- oprofile_add_sample(regs, PCNT0);
- }
- ctr++;
- /* PCNT1 */
- if (ctr->enabled && (pccr & ctr->flag_mask)) {
- sysreg_write(PCNT1, -ctr->count);
- oprofile_add_sample(regs, PCNT1);
- }
-
- return IRQ_HANDLED;
-}
-
-static int avr32_perf_counter_create_files(struct dentry *root)
-{
- struct dentry *dir;
- unsigned int i;
- char filename[4];
-
- for (i = 0; i < NR_counter; i++) {
- snprintf(filename, sizeof(filename), "%u", i);
- dir = oprofilefs_mkdir(root, filename);
-
- oprofilefs_create_ulong(dir, "enabled",
- &counter[i].enabled);
- oprofilefs_create_ulong(dir, "event",
- &counter[i].event);
- oprofilefs_create_ulong(dir, "count",
- &counter[i].count);
-
- /* Dummy entries */
- oprofilefs_create_ulong(dir, "kernel",
- &counter[i].kernel);
- oprofilefs_create_ulong(dir, "user",
- &counter[i].user);
- oprofilefs_create_ulong(dir, "unit_mask",
- &counter[i].unit_mask);
- }
-
- return 0;
-}
-
-static int avr32_perf_counter_setup(void)
-{
- struct avr32_perf_counter *ctr;
- u32 pccr;
- int ret;
- int i;
-
- pr_debug("avr32_perf_counter_setup\n");
-
- if (sysreg_read(PCCR) & SYSREG_BIT(PCCR_E)) {
- printk(KERN_ERR
- "oprofile: setup: perf counter already enabled\n");
- return -EBUSY;
- }
-
- ret = request_irq(AVR32_PERFCTR_IRQ_GROUP,
- avr32_perf_counter_interrupt, IRQF_SHARED,
- "oprofile", counter);
- if (ret)
- return ret;
-
- avr32_perf_counter_reset();
-
- pccr = 0;
- for (i = PCCNT; i < NR_counter; i++) {
- ctr = &counter[i];
- if (!ctr->enabled)
- continue;
-
- pr_debug("enabling counter %d...\n", i);
-
- pccr |= ctr->ie_mask;
-
- switch (i) {
- case PCCNT:
- /* PCCNT always counts cycles, so no events */
- sysreg_write(PCCNT, -ctr->count);
- break;
- case PCNT0:
- pccr |= SYSREG_BF(CONF0, ctr->event);
- sysreg_write(PCNT0, -ctr->count);
- break;
- case PCNT1:
- pccr |= SYSREG_BF(CONF1, ctr->event);
- sysreg_write(PCNT1, -ctr->count);
- break;
- }
- }
-
- pr_debug("oprofile: writing 0x%x to PCCR...\n", pccr);
-
- sysreg_write(PCCR, pccr);
-
- return 0;
-}
-
-static void avr32_perf_counter_shutdown(void)
-{
- pr_debug("avr32_perf_counter_shutdown\n");
-
- avr32_perf_counter_reset();
- free_irq(AVR32_PERFCTR_IRQ_GROUP, counter);
-}
-
-static int avr32_perf_counter_start(void)
-{
- pr_debug("avr32_perf_counter_start\n");
-
- sysreg_write(PCCR, sysreg_read(PCCR) | SYSREG_BIT(PCCR_E));
-
- return 0;
-}
-
-static void avr32_perf_counter_stop(void)
-{
- pr_debug("avr32_perf_counter_stop\n");
-
- sysreg_write(PCCR, sysreg_read(PCCR) & ~SYSREG_BIT(PCCR_E));
-}
-
-static struct oprofile_operations avr32_perf_counter_ops __initdata = {
- .create_files = avr32_perf_counter_create_files,
- .setup = avr32_perf_counter_setup,
- .shutdown = avr32_perf_counter_shutdown,
- .start = avr32_perf_counter_start,
- .stop = avr32_perf_counter_stop,
- .cpu_type = "avr32",
-};
-
-int __init oprofile_arch_init(struct oprofile_operations *ops)
-{
- if (!(current_cpu_data.features & AVR32_FEATURE_PCTR))
- return -ENODEV;
-
- memcpy(ops, &avr32_perf_counter_ops,
- sizeof(struct oprofile_operations));
-
- ops->backtrace = avr32_backtrace;
-
- printk(KERN_INFO "oprofile: using AVR32 performance monitoring.\n");
-
- return 0;
-}
-
-void oprofile_arch_exit(void)
-{
-
-}
diff --git a/arch/blackfin/include/asm/Kbuild b/arch/blackfin/include/asm/Kbuild
index 625db8a..dc4ef9a 100644
--- a/arch/blackfin/include/asm/Kbuild
+++ b/arch/blackfin/include/asm/Kbuild
@@ -7,6 +7,7 @@
generic-y += div64.h
generic-y += emergency-restart.h
generic-y += errno.h
+generic-y += extable.h
generic-y += fb.h
generic-y += futex.h
generic-y += hw_irq.h
diff --git a/arch/blackfin/include/asm/uaccess.h b/arch/blackfin/include/asm/uaccess.h
index 0eff88a..f54a34f 100644
--- a/arch/blackfin/include/asm/uaccess.h
+++ b/arch/blackfin/include/asm/uaccess.h
@@ -12,7 +12,6 @@
/*
* User space memory access functions
*/
-#include <linux/sched.h>
#include <linux/mm.h>
#include <linux/string.h>
@@ -29,9 +28,6 @@
#define segment_eq(a, b) ((a) == (b))
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
#define access_ok(type, addr, size) _access_ok((unsigned long)(addr), (size))
/*
@@ -46,22 +42,7 @@
extern int _access_ok(unsigned long addr, unsigned long size);
#endif
-/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-
-struct exception_table_entry {
- unsigned long insn, fixup;
-};
+#include <asm/extable.h>
/*
* These are the main single-value transfer routines. They automatically
@@ -163,41 +144,23 @@
: "a" (__ptr(ptr))); \
})
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
-
static inline unsigned long __must_check
-__copy_from_user(void *to, const void __user *from, unsigned long n)
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
memcpy(to, (const void __force *)from, n);
return 0;
}
static inline unsigned long __must_check
-__copy_to_user(void __user *to, const void *from, unsigned long n)
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
{
memcpy((void __force *)to, from, n);
SSYNC();
return 0;
}
-static inline unsigned long __must_check
-copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- if (likely(access_ok(VERIFY_READ, from, n)))
- return __copy_from_user(to, from, n);
- memset(to, 0, n);
- return n;
-}
-
-static inline unsigned long __must_check
-copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- if (likely(access_ok(VERIFY_WRITE, to, n)))
- return __copy_to_user(to, from, n);
- return n;
-}
-
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
/*
* Copy a null terminated string from userspace.
*/
diff --git a/arch/blackfin/kernel/process.c b/arch/blackfin/kernel/process.c
index 89d5162..8981485 100644
--- a/arch/blackfin/kernel/process.c
+++ b/arch/blackfin/kernel/process.c
@@ -370,7 +370,7 @@
/* Check that things do not wrap around */
if (addr > ULONG_MAX - size)
return 0;
- if (segment_eq(get_fs(), KERNEL_DS))
+ if (uaccess_kernel())
return 1;
#ifdef CONFIG_MTD_UCLINUX
if (1)
diff --git a/arch/blackfin/kernel/time-ts.c b/arch/blackfin/kernel/time-ts.c
index 0e9fcf8..0135055 100644
--- a/arch/blackfin/kernel/time-ts.c
+++ b/arch/blackfin/kernel/time-ts.c
@@ -230,7 +230,9 @@
clock_tick = get_sclk();
evt->mult = div_sc(clock_tick, NSEC_PER_SEC, evt->shift);
evt->max_delta_ns = clockevent_delta2ns(-1, evt);
+ evt->max_delta_ticks = (unsigned long)-1;
evt->min_delta_ns = clockevent_delta2ns(100, evt);
+ evt->min_delta_ticks = 100;
evt->cpumask = cpumask_of(0);
@@ -344,7 +346,9 @@
clock_tick = get_cclk() / TIME_SCALE;
evt->mult = div_sc(clock_tick, NSEC_PER_SEC, evt->shift);
evt->max_delta_ns = clockevent_delta2ns(-1, evt);
+ evt->max_delta_ticks = (unsigned long)-1;
evt->min_delta_ns = clockevent_delta2ns(100, evt);
+ evt->min_delta_ticks = 100;
evt->cpumask = cpumask_of(cpu);
diff --git a/arch/blackfin/kernel/vmlinux.lds.S b/arch/blackfin/kernel/vmlinux.lds.S
index 68069a1..334ef81 100644
--- a/arch/blackfin/kernel/vmlinux.lds.S
+++ b/arch/blackfin/kernel/vmlinux.lds.S
@@ -115,6 +115,8 @@
__data_lma = LOADADDR(.data);
__data_len = SIZEOF(.data);
+ BUG_TABLE
+
/* The init section should be last, so when we free it, it goes into
* the general memory pool, and (hopefully) will decrease fragmentation
* a tiny bit. The init section has a _requirement_ that it be
diff --git a/arch/c6x/include/asm/Kbuild b/arch/c6x/include/asm/Kbuild
index 82619c3..f0eaf047 100644
--- a/arch/c6x/include/asm/Kbuild
+++ b/arch/c6x/include/asm/Kbuild
@@ -12,6 +12,7 @@
generic-y += emergency-restart.h
generic-y += errno.h
generic-y += exec.h
+generic-y += extable.h
generic-y += fb.h
generic-y += fcntl.h
generic-y += futex.h
diff --git a/arch/c6x/include/asm/uaccess.h b/arch/c6x/include/asm/uaccess.h
index 453dd26..ba67568 100644
--- a/arch/c6x/include/asm/uaccess.h
+++ b/arch/c6x/include/asm/uaccess.h
@@ -13,17 +13,11 @@
#include <linux/compiler.h>
#include <linux/string.h>
-#ifdef CONFIG_ACCESS_CHECK
-#define __access_ok _access_ok
-#endif
-
/*
- * __copy_from_user/copy_to_user are based on ones in asm-generic/uaccess.h
- *
* C6X supports unaligned 32 and 64 bit loads and stores.
*/
-static inline __must_check long __copy_from_user(void *to,
- const void __user *from, unsigned long n)
+static inline __must_check unsigned long
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
u32 tmp32;
u64 tmp64;
@@ -58,8 +52,8 @@
return 0;
}
-static inline __must_check long __copy_to_user(void __user *to,
- const void *from, unsigned long n)
+static inline __must_check unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
{
u32 tmp32;
u64 tmp64;
@@ -93,9 +87,8 @@
memcpy((void __force *)to, from, n);
return 0;
}
-
-#define __copy_to_user __copy_to_user
-#define __copy_from_user __copy_from_user
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
extern int _access_ok(unsigned long addr, unsigned long size);
#ifdef CONFIG_ACCESS_CHECK
diff --git a/arch/c6x/kernel/sys_c6x.c b/arch/c6x/kernel/sys_c6x.c
index 3e9bdfb..a742ae25 100644
--- a/arch/c6x/kernel/sys_c6x.c
+++ b/arch/c6x/kernel/sys_c6x.c
@@ -23,7 +23,7 @@
if (!addr || addr > (0xffffffffUL - (size - 1)))
goto _bad_access;
- if (segment_eq(get_fs(), KERNEL_DS))
+ if (uaccess_kernel())
return 1;
if (memory_start <= addr && (addr + size - 1) < memory_end)
diff --git a/arch/c6x/kernel/vmlinux.lds.S b/arch/c6x/kernel/vmlinux.lds.S
index a1a5c16..29ebea4 100644
--- a/arch/c6x/kernel/vmlinux.lds.S
+++ b/arch/c6x/kernel/vmlinux.lds.S
@@ -128,6 +128,8 @@
. = ALIGN(8);
}
+ BUG_TABLE
+
_edata = .;
__bss_start = .;
diff --git a/arch/c6x/platforms/timer64.c b/arch/c6x/platforms/timer64.c
index c19901e..0bd0452 100644
--- a/arch/c6x/platforms/timer64.c
+++ b/arch/c6x/platforms/timer64.c
@@ -234,7 +234,9 @@
clockevents_calc_mult_shift(cd, c6x_core_freq / TIMER_DIVISOR, 5);
cd->max_delta_ns = clockevent_delta2ns(0x7fffffff, cd);
+ cd->max_delta_ticks = 0x7fffffff;
cd->min_delta_ns = clockevent_delta2ns(250, cd);
+ cd->min_delta_ticks = 250;
cd->cpumask = cpumask_of(smp_processor_id());
diff --git a/arch/cris/arch-v10/lib/usercopy.c b/arch/cris/arch-v10/lib/usercopy.c
index 1ba7cc0..48fa37f 100644
--- a/arch/cris/arch-v10/lib/usercopy.c
+++ b/arch/cris/arch-v10/lib/usercopy.c
@@ -188,11 +188,10 @@
}
EXPORT_SYMBOL(__copy_user);
-/* Copy from user to kernel, zeroing the bytes that were inaccessible in
- userland. The return-value is the number of bytes that were
+/* Copy from user to kernel. The return-value is the number of bytes that were
inaccessible. */
-unsigned long __copy_user_zeroing(void *pdst, const void __user *psrc,
+unsigned long __copy_user_in(void *pdst, const void __user *psrc,
unsigned long pn)
{
/* We want the parameters put in special registers.
@@ -217,19 +216,17 @@
{
__asm_copy_from_user_1 (dst, src, retn);
n--;
+ if (retn)
+ goto exception;
}
if (((unsigned long) src & 2) && n >= 2)
{
__asm_copy_from_user_2 (dst, src, retn);
n -= 2;
+ if (retn)
+ goto exception;
}
-
- /* We only need one check after the unalignment-adjustments, because
- if both adjustments were done, either both or neither reference
- had an exception. */
- if (retn != 0)
- goto copy_exception_bytes;
}
/* Decide which copying method to use. */
@@ -328,7 +325,7 @@
n -= 4;
if (retn)
- goto copy_exception_bytes;
+ goto exception;
}
/* If we get here, there were no memory read faults. */
@@ -356,20 +353,10 @@
bytes. */
return retn;
-copy_exception_bytes:
- /* We already have "retn" bytes cleared, and need to clear the
- remaining "n" bytes. A non-optimized simple byte-for-byte in-line
- memset is preferred here, since this isn't speed-critical code and
- we'd rather have this a leaf-function than calling memset. */
- {
- char *endp;
- for (endp = dst + n; dst < endp; dst++)
- *dst = 0;
- }
-
+exception:
return retn + n;
}
-EXPORT_SYMBOL(__copy_user_zeroing);
+EXPORT_SYMBOL(__copy_user_in);
/* Zero userspace. */
unsigned long __do_clear_user(void __user *pto, unsigned long pn)
diff --git a/arch/cris/arch-v32/lib/usercopy.c b/arch/cris/arch-v32/lib/usercopy.c
index 05e58da..20b6080 100644
--- a/arch/cris/arch-v32/lib/usercopy.c
+++ b/arch/cris/arch-v32/lib/usercopy.c
@@ -156,10 +156,9 @@
}
EXPORT_SYMBOL(__copy_user);
-/* Copy from user to kernel, zeroing the bytes that were inaccessible in
- userland. The return-value is the number of bytes that were
+/* Copy from user to kernel. The return-value is the number of bytes that were
inaccessible. */
-unsigned long __copy_user_zeroing(void *pdst, const void __user *psrc,
+unsigned long __copy_user_in(void *pdst, const void __user *psrc,
unsigned long pn)
{
/* We want the parameters put in special registers.
@@ -184,19 +183,18 @@
{
__asm_copy_from_user_1 (dst, src, retn);
n--;
+ if (retn != 0)
+ goto exception;
}
if (((unsigned long) src & 2) && n >= 2)
{
__asm_copy_from_user_2 (dst, src, retn);
n -= 2;
+ if (retn != 0)
+ goto exception;
}
- /* We only need one check after the unalignment-adjustments, because
- if both adjustments were done, either both or neither reference
- had an exception. */
- if (retn != 0)
- goto copy_exception_bytes;
}
/* Movem is dirt cheap. The overheap is low enough to always use the
@@ -279,7 +277,7 @@
n -= 4;
if (retn)
- goto copy_exception_bytes;
+ goto exception;
}
/* If we get here, there were no memory read faults. */
@@ -307,20 +305,10 @@
bytes. */
return retn;
-copy_exception_bytes:
- /* We already have "retn" bytes cleared, and need to clear the
- remaining "n" bytes. A non-optimized simple byte-for-byte in-line
- memset is preferred here, since this isn't speed-critical code and
- we'd rather have this a leaf-function than calling memset. */
- {
- char *endp;
- for (endp = dst + n; dst < endp; dst++)
- *dst = 0;
- }
-
+exception:
return retn + n;
}
-EXPORT_SYMBOL(__copy_user_zeroing);
+EXPORT_SYMBOL(__copy_user_in);
/* Zero userspace. */
unsigned long __do_clear_user(void __user *pto, unsigned long pn)
diff --git a/arch/cris/include/arch-v10/arch/uaccess.h b/arch/cris/include/arch-v10/arch/uaccess.h
index 65b02d9..5477c98 100644
--- a/arch/cris/include/arch-v10/arch/uaccess.h
+++ b/arch/cris/include/arch-v10/arch/uaccess.h
@@ -172,16 +172,14 @@
__asm_copy_user_cont(to, from, ret, \
" move.b [%1+],$r9\n" \
"2: move.b $r9,[%0+]\n", \
- "3: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "3: addq 1,%2\n", \
" .dword 2b,3b\n")
#define __asm_copy_from_user_2x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
__asm_copy_user_cont(to, from, ret, \
" move.w [%1+],$r9\n" \
"2: move.w $r9,[%0+]\n" COPY, \
- "3: addq 2,%2\n" \
- " clear.w [%0+]\n" FIXUP, \
+ "3: addq 2,%2\n" FIXUP, \
" .dword 2b,3b\n" TENTRY)
#define __asm_copy_from_user_2(to, from, ret) \
@@ -191,16 +189,14 @@
__asm_copy_from_user_2x_cont(to, from, ret, \
" move.b [%1+],$r9\n" \
"4: move.b $r9,[%0+]\n", \
- "5: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "5: addq 1,%2\n", \
" .dword 4b,5b\n")
#define __asm_copy_from_user_4x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
__asm_copy_user_cont(to, from, ret, \
" move.d [%1+],$r9\n" \
"2: move.d $r9,[%0+]\n" COPY, \
- "3: addq 4,%2\n" \
- " clear.d [%0+]\n" FIXUP, \
+ "3: addq 4,%2\n" FIXUP, \
" .dword 2b,3b\n" TENTRY)
#define __asm_copy_from_user_4(to, from, ret) \
@@ -210,8 +206,7 @@
__asm_copy_from_user_4x_cont(to, from, ret, \
" move.b [%1+],$r9\n" \
"4: move.b $r9,[%0+]\n", \
- "5: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "5: addq 1,%2\n", \
" .dword 4b,5b\n")
#define __asm_copy_from_user_6x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -219,7 +214,7 @@
" move.w [%1+],$r9\n" \
"4: move.w $r9,[%0+]\n" COPY, \
"5: addq 2,%2\n" \
- " clear.w [%0+]\n" FIXUP, \
+ FIXUP, \
" .dword 4b,5b\n" TENTRY)
#define __asm_copy_from_user_6(to, from, ret) \
@@ -229,8 +224,7 @@
__asm_copy_from_user_6x_cont(to, from, ret, \
" move.b [%1+],$r9\n" \
"6: move.b $r9,[%0+]\n", \
- "7: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "7: addq 1,%2\n", \
" .dword 6b,7b\n")
#define __asm_copy_from_user_8x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -238,7 +232,7 @@
" move.d [%1+],$r9\n" \
"4: move.d $r9,[%0+]\n" COPY, \
"5: addq 4,%2\n" \
- " clear.d [%0+]\n" FIXUP, \
+ FIXUP, \
" .dword 4b,5b\n" TENTRY)
#define __asm_copy_from_user_8(to, from, ret) \
@@ -248,8 +242,7 @@
__asm_copy_from_user_8x_cont(to, from, ret, \
" move.b [%1+],$r9\n" \
"6: move.b $r9,[%0+]\n", \
- "7: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "7: addq 1,%2\n", \
" .dword 6b,7b\n")
#define __asm_copy_from_user_10x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -257,7 +250,7 @@
" move.w [%1+],$r9\n" \
"6: move.w $r9,[%0+]\n" COPY, \
"7: addq 2,%2\n" \
- " clear.w [%0+]\n" FIXUP, \
+ FIXUP, \
" .dword 6b,7b\n" TENTRY)
#define __asm_copy_from_user_10(to, from, ret) \
@@ -267,8 +260,7 @@
__asm_copy_from_user_10x_cont(to, from, ret, \
" move.b [%1+],$r9\n" \
"8: move.b $r9,[%0+]\n", \
- "9: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "9: addq 1,%2\n", \
" .dword 8b,9b\n")
#define __asm_copy_from_user_12x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -276,7 +268,7 @@
" move.d [%1+],$r9\n" \
"6: move.d $r9,[%0+]\n" COPY, \
"7: addq 4,%2\n" \
- " clear.d [%0+]\n" FIXUP, \
+ FIXUP, \
" .dword 6b,7b\n" TENTRY)
#define __asm_copy_from_user_12(to, from, ret) \
@@ -286,8 +278,7 @@
__asm_copy_from_user_12x_cont(to, from, ret, \
" move.b [%1+],$r9\n" \
"8: move.b $r9,[%0+]\n", \
- "9: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "9: addq 1,%2\n", \
" .dword 8b,9b\n")
#define __asm_copy_from_user_14x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -295,7 +286,7 @@
" move.w [%1+],$r9\n" \
"8: move.w $r9,[%0+]\n" COPY, \
"9: addq 2,%2\n" \
- " clear.w [%0+]\n" FIXUP, \
+ FIXUP, \
" .dword 8b,9b\n" TENTRY)
#define __asm_copy_from_user_14(to, from, ret) \
@@ -305,8 +296,7 @@
__asm_copy_from_user_14x_cont(to, from, ret, \
" move.b [%1+],$r9\n" \
"10: move.b $r9,[%0+]\n", \
- "11: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "11: addq 1,%2\n", \
" .dword 10b,11b\n")
#define __asm_copy_from_user_16x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -314,7 +304,7 @@
" move.d [%1+],$r9\n" \
"8: move.d $r9,[%0+]\n" COPY, \
"9: addq 4,%2\n" \
- " clear.d [%0+]\n" FIXUP, \
+ FIXUP, \
" .dword 8b,9b\n" TENTRY)
#define __asm_copy_from_user_16(to, from, ret) \
@@ -325,7 +315,7 @@
" move.d [%1+],$r9\n" \
"10: move.d $r9,[%0+]\n" COPY, \
"11: addq 4,%2\n" \
- " clear.d [%0+]\n" FIXUP, \
+ FIXUP, \
" .dword 10b,11b\n" TENTRY)
#define __asm_copy_from_user_20(to, from, ret) \
@@ -336,7 +326,7 @@
" move.d [%1+],$r9\n" \
"12: move.d $r9,[%0+]\n" COPY, \
"13: addq 4,%2\n" \
- " clear.d [%0+]\n" FIXUP, \
+ FIXUP, \
" .dword 12b,13b\n" TENTRY)
#define __asm_copy_from_user_24(to, from, ret) \
diff --git a/arch/cris/include/arch-v32/arch/uaccess.h b/arch/cris/include/arch-v32/arch/uaccess.h
index 3196019..dc2ce09 100644
--- a/arch/cris/include/arch-v32/arch/uaccess.h
+++ b/arch/cris/include/arch-v32/arch/uaccess.h
@@ -178,8 +178,7 @@
"2: move.b [%1+],$acr\n" \
" move.b $acr,[%0+]\n", \
"3: addq 1,%2\n" \
- " jump 1b\n" \
- " clear.b [%0+]\n", \
+ " jump 1b\n", \
" .dword 2b,3b\n")
#define __asm_copy_from_user_2x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -189,8 +188,7 @@
" move.w $acr,[%0+]\n", \
FIXUP \
"3: addq 2,%2\n" \
- " jump 1b\n" \
- " clear.w [%0+]\n", \
+ " jump 1b\n", \
TENTRY \
" .dword 2b,3b\n")
@@ -201,8 +199,7 @@
__asm_copy_from_user_2x_cont(to, from, ret, \
"4: move.b [%1+],$acr\n" \
" move.b $acr,[%0+]\n", \
- "5: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "5: addq 1,%2\n", \
" .dword 4b,5b\n")
#define __asm_copy_from_user_4x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -212,8 +209,7 @@
" move.d $acr,[%0+]\n", \
FIXUP \
"3: addq 4,%2\n" \
- " jump 1b\n" \
- " clear.d [%0+]\n", \
+ " jump 1b\n", \
TENTRY \
" .dword 2b,3b\n")
@@ -224,8 +220,7 @@
__asm_copy_from_user_4x_cont(to, from, ret, \
"4: move.b [%1+],$acr\n" \
" move.b $acr,[%0+]\n", \
- "5: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "5: addq 1,%2\n", \
" .dword 4b,5b\n")
#define __asm_copy_from_user_6x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -234,8 +229,7 @@
"4: move.w [%1+],$acr\n" \
" move.w $acr,[%0+]\n", \
FIXUP \
- "5: addq 2,%2\n" \
- " clear.w [%0+]\n", \
+ "5: addq 2,%2\n", \
TENTRY \
" .dword 4b,5b\n")
@@ -246,8 +240,7 @@
__asm_copy_from_user_6x_cont(to, from, ret, \
"6: move.b [%1+],$acr\n" \
" move.b $acr,[%0+]\n", \
- "7: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "7: addq 1,%2\n", \
" .dword 6b,7b\n")
#define __asm_copy_from_user_8x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -256,8 +249,7 @@
"4: move.d [%1+],$acr\n" \
" move.d $acr,[%0+]\n", \
FIXUP \
- "5: addq 4,%2\n" \
- " clear.d [%0+]\n", \
+ "5: addq 4,%2\n", \
TENTRY \
" .dword 4b,5b\n")
@@ -268,8 +260,7 @@
__asm_copy_from_user_8x_cont(to, from, ret, \
"6: move.b [%1+],$acr\n" \
" move.b $acr,[%0+]\n", \
- "7: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "7: addq 1,%2\n", \
" .dword 6b,7b\n")
#define __asm_copy_from_user_10x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -278,8 +269,7 @@
"6: move.w [%1+],$acr\n" \
" move.w $acr,[%0+]\n", \
FIXUP \
- "7: addq 2,%2\n" \
- " clear.w [%0+]\n", \
+ "7: addq 2,%2\n", \
TENTRY \
" .dword 6b,7b\n")
@@ -290,8 +280,7 @@
__asm_copy_from_user_10x_cont(to, from, ret, \
"8: move.b [%1+],$acr\n" \
" move.b $acr,[%0+]\n", \
- "9: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "9: addq 1,%2\n", \
" .dword 8b,9b\n")
#define __asm_copy_from_user_12x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -300,8 +289,7 @@
"6: move.d [%1+],$acr\n" \
" move.d $acr,[%0+]\n", \
FIXUP \
- "7: addq 4,%2\n" \
- " clear.d [%0+]\n", \
+ "7: addq 4,%2\n", \
TENTRY \
" .dword 6b,7b\n")
@@ -312,8 +300,7 @@
__asm_copy_from_user_12x_cont(to, from, ret, \
"8: move.b [%1+],$acr\n" \
" move.b $acr,[%0+]\n", \
- "9: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "9: addq 1,%2\n", \
" .dword 8b,9b\n")
#define __asm_copy_from_user_14x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -322,8 +309,7 @@
"8: move.w [%1+],$acr\n" \
" move.w $acr,[%0+]\n", \
FIXUP \
- "9: addq 2,%2\n" \
- " clear.w [%0+]\n", \
+ "9: addq 2,%2\n", \
TENTRY \
" .dword 8b,9b\n")
@@ -334,8 +320,7 @@
__asm_copy_from_user_14x_cont(to, from, ret, \
"10: move.b [%1+],$acr\n" \
" move.b $acr,[%0+]\n", \
- "11: addq 1,%2\n" \
- " clear.b [%0+]\n", \
+ "11: addq 1,%2\n", \
" .dword 10b,11b\n")
#define __asm_copy_from_user_16x_cont(to, from, ret, COPY, FIXUP, TENTRY) \
@@ -344,8 +329,7 @@
"8: move.d [%1+],$acr\n" \
" move.d $acr,[%0+]\n", \
FIXUP \
- "9: addq 4,%2\n" \
- " clear.d [%0+]\n", \
+ "9: addq 4,%2\n", \
TENTRY \
" .dword 8b,9b\n")
@@ -358,8 +342,7 @@
"10: move.d [%1+],$acr\n" \
" move.d $acr,[%0+]\n", \
FIXUP \
- "11: addq 4,%2\n" \
- " clear.d [%0+]\n", \
+ "11: addq 4,%2\n", \
TENTRY \
" .dword 10b,11b\n")
@@ -372,8 +355,7 @@
"12: move.d [%1+],$acr\n" \
" move.d $acr,[%0+]\n", \
FIXUP \
- "13: addq 4,%2\n" \
- " clear.d [%0+]\n", \
+ "13: addq 4,%2\n", \
TENTRY \
" .dword 12b,13b\n")
diff --git a/arch/cris/include/asm/Kbuild b/arch/cris/include/asm/Kbuild
index 0f5132b..2890099 100644
--- a/arch/cris/include/asm/Kbuild
+++ b/arch/cris/include/asm/Kbuild
@@ -9,6 +9,7 @@
generic-y += div64.h
generic-y += errno.h
generic-y += exec.h
+generic-y += extable.h
generic-y += emergency-restart.h
generic-y += fcntl.h
generic-y += futex.h
diff --git a/arch/cris/include/asm/uaccess.h b/arch/cris/include/asm/uaccess.h
index 56c7d57..0d473ae 100644
--- a/arch/cris/include/asm/uaccess.h
+++ b/arch/cris/include/asm/uaccess.h
@@ -15,15 +15,9 @@
#ifndef _CRIS_UACCESS_H
#define _CRIS_UACCESS_H
-#ifndef __ASSEMBLY__
-#include <linux/sched.h>
-#include <linux/errno.h>
#include <asm/processor.h>
#include <asm/page.h>
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
/*
* The fs value determines whether argument validity checking should be
* performed or not. If get_fs() == USER_DS, checking is performed, with
@@ -49,30 +43,14 @@
#define segment_eq(a, b) ((a).seg == (b).seg)
-#define __kernel_ok (segment_eq(get_fs(), KERNEL_DS))
+#define __kernel_ok (uaccess_kernel())
#define __user_ok(addr, size) \
(((size) <= TASK_SIZE) && ((addr) <= TASK_SIZE-(size)))
#define __access_ok(addr, size) (__kernel_ok || __user_ok((addr), (size)))
#define access_ok(type, addr, size) __access_ok((unsigned long)(addr), (size))
#include <arch/uaccess.h>
-
-/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-
-struct exception_table_entry {
- unsigned long insn, fixup;
-};
+#include <asm/extable.h>
/*
* These are the main single-value transfer routines. They automatically
@@ -191,7 +169,7 @@
live in lib/usercopy.c */
extern unsigned long __copy_user(void __user *to, const void *from, unsigned long n);
-extern unsigned long __copy_user_zeroing(void *to, const void __user *from, unsigned long n);
+extern unsigned long __copy_user_in(void *to, const void __user *from, unsigned long n);
extern unsigned long __do_clear_user(void __user *to, unsigned long n);
static inline long
@@ -258,7 +236,7 @@
else if (n == 24)
__asm_copy_from_user_24(to, from, ret);
else
- ret = __copy_user_zeroing(to, from, n);
+ ret = __copy_user_in(to, from, n);
return ret;
}
@@ -358,64 +336,33 @@
return __do_clear_user(to, n);
}
-static inline size_t copy_from_user(void *to, const void __user *from, size_t n)
+static inline unsigned long
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
- if (unlikely(!access_ok(VERIFY_READ, from, n))) {
- memset(to, 0, n);
- return n;
- }
if (__builtin_constant_p(n))
return __constant_copy_from_user(to, from, n);
else
- return __copy_user_zeroing(to, from, n);
+ return __copy_user_in(to, from, n);
}
-static inline size_t copy_to_user(void __user *to, const void *from, size_t n)
+static inline unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
{
- if (unlikely(!access_ok(VERIFY_WRITE, to, n)))
- return n;
if (__builtin_constant_p(n))
return __constant_copy_to_user(to, from, n);
else
return __copy_user(to, from, n);
}
-/* We let the __ versions of copy_from/to_user inline, because they're often
- * used in fast paths and have only a small space overhead.
- */
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
static inline unsigned long
-__generic_copy_from_user_nocheck(void *to, const void __user *from,
- unsigned long n)
-{
- return __copy_user_zeroing(to, from, n);
-}
-
-static inline unsigned long
-__generic_copy_to_user_nocheck(void __user *to, const void *from,
- unsigned long n)
-{
- return __copy_user(to, from, n);
-}
-
-static inline unsigned long
-__generic_clear_user_nocheck(void __user *to, unsigned long n)
+__clear_user(void __user *to, unsigned long n)
{
return __do_clear_user(to, n);
}
-/* without checking */
-
-#define __copy_to_user(to, from, n) \
- __generic_copy_to_user_nocheck((to), (from), (n))
-#define __copy_from_user(to, from, n) \
- __generic_copy_from_user_nocheck((to), (from), (n))
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
-#define __clear_user(to, n) __generic_clear_user_nocheck((to), (n))
-
#define strlen_user(str) strnlen_user((str), 0x7ffffffe)
-#endif /* __ASSEMBLY__ */
-
#endif /* _CRIS_UACCESS_H */
diff --git a/arch/cris/kernel/vmlinux.lds.S b/arch/cris/kernel/vmlinux.lds.S
index 9795862..867f237 100644
--- a/arch/cris/kernel/vmlinux.lds.S
+++ b/arch/cris/kernel/vmlinux.lds.S
@@ -68,6 +68,8 @@
__edata = . ; /* End of data section. */
_edata = . ;
+ BUG_TABLE
+
INIT_TASK_DATA_SECTION(PAGE_SIZE)
. = ALIGN(PAGE_SIZE); /* Init code and data. */
diff --git a/arch/frv/include/asm/Kbuild b/arch/frv/include/asm/Kbuild
index c33b467..cce3bc3 100644
--- a/arch/frv/include/asm/Kbuild
+++ b/arch/frv/include/asm/Kbuild
@@ -1,6 +1,7 @@
generic-y += clkdev.h
generic-y += exec.h
+generic-y += extable.h
generic-y += irq_work.h
generic-y += mcs_spinlock.h
generic-y += mm-arch-hooks.h
diff --git a/arch/frv/include/asm/uaccess.h b/arch/frv/include/asm/uaccess.h
index c0f4057e..e4e33b4 100644
--- a/arch/frv/include/asm/uaccess.h
+++ b/arch/frv/include/asm/uaccess.h
@@ -15,16 +15,13 @@
/*
* User space memory access functions
*/
-#include <linux/sched.h>
#include <linux/mm.h>
#include <asm/segment.h>
#include <asm/sections.h>
+#include <asm/extable.h>
#define __ptr(x) ((unsigned long __force *)(x))
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
/*
* check that a range of addresses falls within the current address limit
*/
@@ -63,26 +60,6 @@
#define access_ok(type,addr,size) (__range_ok((void __user *)(addr), (size)) == 0)
#define __access_ok(addr,size) (__range_ok((addr), (size)) == 0)
-/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-struct exception_table_entry
-{
- unsigned long insn, fixup;
-};
-
-/* Returns 0 if exception not found and fixup otherwise. */
-extern unsigned long search_exception_table(unsigned long);
-
/*
* These are the main single-value transfer routines. They automatically
@@ -256,23 +233,44 @@
/*
*
*/
+
#define ____force(x) (__force void *)(void __user *)(x)
#ifdef CONFIG_MMU
extern long __memset_user(void *dst, unsigned long count);
extern long __memcpy_user(void *dst, const void *src, unsigned long count);
#define __clear_user(dst,count) __memset_user(____force(dst), (count))
-#define __copy_from_user_inatomic(to, from, n) __memcpy_user((to), ____force(from), (n))
-#define __copy_to_user_inatomic(to, from, n) __memcpy_user(____force(to), (from), (n))
#else
#define __clear_user(dst,count) (memset(____force(dst), 0, (count)), 0)
-#define __copy_from_user_inatomic(to, from, n) (memcpy((to), ____force(from), (n)), 0)
-#define __copy_to_user_inatomic(to, from, n) (memcpy(____force(to), (from), (n)), 0)
#endif
+static inline unsigned long
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
+{
+#ifdef CONFIG_MMU
+ return __memcpy_user(to, (__force const void *)from, n);
+#else
+ memcpy(to, (__force const void *)from, n);
+ return 0;
+#endif
+}
+
+static inline unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
+{
+#ifdef CONFIG_MMU
+ return __memcpy_user((__force void *)to, from, n);
+#else
+ memcpy((__force void *)to, from, n);
+ return 0;
+#endif
+}
+#define INLINE_COPY_TO_USER
+#define INLINE_COPY_FROM_USER
+
static inline unsigned long __must_check
clear_user(void __user *to, unsigned long n)
{
@@ -281,43 +279,9 @@
return n;
}
-static inline unsigned long __must_check
-__copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- might_fault();
- return __copy_to_user_inatomic(to, from, n);
-}
-
-static inline unsigned long
-__copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- might_fault();
- return __copy_from_user_inatomic(to, from, n);
-}
-
-static inline long copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- unsigned long ret = n;
-
- if (likely(__access_ok(from, n)))
- ret = __copy_from_user(to, from, n);
-
- if (unlikely(ret != 0))
- memset(to + (n - ret), 0, ret);
-
- return ret;
-}
-
-static inline long copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- return likely(__access_ok(to, n)) ? __copy_to_user(to, from, n) : n;
-}
-
extern long strncpy_from_user(char *dst, const char __user *src, long count);
extern long strnlen_user(const char __user *src, long count);
#define strlen_user(str) strnlen_user(str, 32767)
-extern unsigned long search_exception_table(unsigned long addr);
-
#endif /* _ASM_UACCESS_H */
diff --git a/arch/frv/include/uapi/asm/socket.h b/arch/frv/include/uapi/asm/socket.h
index 81e0353..1ccf456 100644
--- a/arch/frv/include/uapi/asm/socket.h
+++ b/arch/frv/include/uapi/asm/socket.h
@@ -92,5 +92,11 @@
#define SCM_TIMESTAMPING_OPT_STATS 54
+#define SO_MEMINFO 55
+
+#define SO_INCOMING_NAPI_ID 56
+
+#define SO_COOKIE 57
+
#endif /* _ASM_SOCKET_H */
diff --git a/arch/frv/kernel/traps.c b/arch/frv/kernel/traps.c
index ce29991..fb08ebe 100644
--- a/arch/frv/kernel/traps.c
+++ b/arch/frv/kernel/traps.c
@@ -360,13 +360,8 @@
siginfo_t info;
#ifdef CONFIG_MMU
- unsigned long fixup;
-
- fixup = search_exception_table(__frame->pc);
- if (fixup) {
- __frame->pc = fixup;
+ if (fixup_exception(__frame))
return;
- }
#endif
die_if_kernel("-- Memory Access Exception --\n"
diff --git a/arch/frv/kernel/vmlinux.lds.S b/arch/frv/kernel/vmlinux.lds.S
index aa6e573..3f44dcb 100644
--- a/arch/frv/kernel/vmlinux.lds.S
+++ b/arch/frv/kernel/vmlinux.lds.S
@@ -102,6 +102,8 @@
_edata = .; /* End of data section */
+ BUG_TABLE
+
/* GP section */
. = ALIGN(L1_CACHE_BYTES);
_gp = . + 2048;
diff --git a/arch/frv/mm/extable.c b/arch/frv/mm/extable.c
index a0e8b3e..9198ddd 100644
--- a/arch/frv/mm/extable.c
+++ b/arch/frv/mm/extable.c
@@ -10,40 +10,39 @@
extern const void __memcpy_end, __memcpy_user_error_lr, __memcpy_user_error_handler;
extern spinlock_t modlist_lock;
-
-/*****************************************************************************/
-/*
- * see if there's a fixup handler available to deal with a kernel fault
- */
-unsigned long search_exception_table(unsigned long pc)
+int fixup_exception(struct pt_regs *regs)
{
const struct exception_table_entry *extab;
+ unsigned long pc = regs->pc;
/* determine if the fault lay during a memcpy_user or a memset_user */
- if (__frame->lr == (unsigned long) &__memset_user_error_lr &&
+ if (regs->lr == (unsigned long) &__memset_user_error_lr &&
(unsigned long) &memset <= pc && pc < (unsigned long) &__memset_end
) {
/* the fault occurred in a protected memset
* - we search for the return address (in LR) instead of the program counter
* - it was probably during a clear_user()
*/
- return (unsigned long) &__memset_user_error_handler;
+ regs->pc = (unsigned long) &__memset_user_error_handler;
+ return 1;
}
- if (__frame->lr == (unsigned long) &__memcpy_user_error_lr &&
+ if (regs->lr == (unsigned long) &__memcpy_user_error_lr &&
(unsigned long) &memcpy <= pc && pc < (unsigned long) &__memcpy_end
) {
/* the fault occurred in a protected memset
* - we search for the return address (in LR) instead of the program counter
* - it was probably during a copy_to/from_user()
*/
- return (unsigned long) &__memcpy_user_error_handler;
+ regs->pc = (unsigned long) &__memcpy_user_error_handler;
+ return 1;
}
extab = search_exception_tables(pc);
- if (extab)
- return extab->fixup;
+ if (extab) {
+ regs->pc = extab->fixup;
+ return 1;
+ }
return 0;
-
-} /* end search_exception_table() */
+}
diff --git a/arch/frv/mm/fault.c b/arch/frv/mm/fault.c
index 614a46c..179e79e 100644
--- a/arch/frv/mm/fault.c
+++ b/arch/frv/mm/fault.c
@@ -33,7 +33,7 @@
{
struct vm_area_struct *vma;
struct mm_struct *mm;
- unsigned long _pme, lrai, lrad, fixup;
+ unsigned long _pme, lrai, lrad;
unsigned long flags = 0;
siginfo_t info;
pgd_t *pge;
@@ -201,10 +201,8 @@
no_context:
/* are we prepared to handle this kernel fault? */
- if ((fixup = search_exception_table(__frame->pc)) != 0) {
- __frame->pc = fixup;
+ if (fixup_exception(__frame))
return;
- }
/*
* Oops. The kernel tried to access some bad page. We'll have to
diff --git a/arch/h8300/include/asm/Kbuild b/arch/h8300/include/asm/Kbuild
index 341740c..757cdeb 100644
--- a/arch/h8300/include/asm/Kbuild
+++ b/arch/h8300/include/asm/Kbuild
@@ -13,6 +13,7 @@
generic-y += emergency-restart.h
generic-y += errno.h
generic-y += exec.h
+generic-y += extable.h
generic-y += fb.h
generic-y += fcntl.h
generic-y += ftrace.h
@@ -68,7 +69,6 @@
generic-y += trace_clock.h
generic-y += topology.h
generic-y += types.h
-generic-y += uaccess.h
generic-y += ucontext.h
generic-y += unaligned.h
generic-y += vga.h
diff --git a/arch/h8300/include/asm/uaccess.h b/arch/h8300/include/asm/uaccess.h
new file mode 100644
index 0000000..6f6144a
--- /dev/null
+++ b/arch/h8300/include/asm/uaccess.h
@@ -0,0 +1,54 @@
+#ifndef _ASM_UACCESS_H
+#define _ASM_UACCESS_H
+
+#include <linux/string.h>
+
+static inline __must_check unsigned long
+raw_copy_from_user(void *to, const void __user * from, unsigned long n)
+{
+ if (__builtin_constant_p(n)) {
+ switch(n) {
+ case 1:
+ *(u8 *)to = *(u8 __force *)from;
+ return 0;
+ case 2:
+ *(u16 *)to = *(u16 __force *)from;
+ return 0;
+ case 4:
+ *(u32 *)to = *(u32 __force *)from;
+ return 0;
+ }
+ }
+
+ memcpy(to, (const void __force *)from, n);
+ return 0;
+}
+
+static inline __must_check unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
+{
+ if (__builtin_constant_p(n)) {
+ switch(n) {
+ case 1:
+ *(u8 __force *)to = *(u8 *)from;
+ return 0;
+ case 2:
+ *(u16 __force *)to = *(u16 *)from;
+ return 0;
+ case 4:
+ *(u32 __force *)to = *(u32 *)from;
+ return 0;
+ default:
+ break;
+ }
+ }
+
+ memcpy((void __force *)to, from, n);
+ return 0;
+}
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
+
+#include <asm-generic/uaccess.h>
+
+#endif
diff --git a/arch/hexagon/include/asm/Kbuild b/arch/hexagon/include/asm/Kbuild
index 797b64a..a2036bf 100644
--- a/arch/hexagon/include/asm/Kbuild
+++ b/arch/hexagon/include/asm/Kbuild
@@ -11,6 +11,7 @@
generic-y += div64.h
generic-y += emergency-restart.h
generic-y += errno.h
+generic-y += extable.h
generic-y += fb.h
generic-y += fcntl.h
generic-y += ftrace.h
diff --git a/arch/hexagon/include/asm/uaccess.h b/arch/hexagon/include/asm/uaccess.h
index f61cfb2..458b698 100644
--- a/arch/hexagon/include/asm/uaccess.h
+++ b/arch/hexagon/include/asm/uaccess.h
@@ -23,7 +23,6 @@
/*
* User space memory access functions
*/
-#include <linux/sched.h>
#include <linux/mm.h>
#include <asm/segment.h>
#include <asm/sections.h>
@@ -50,8 +49,6 @@
* reasonably simple and not *too* slow. After all, we've got the
* MMU for backup.
*/
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
#define __access_ok(addr, size) \
((get_fs().seg == KERNEL_DS.seg) || \
@@ -68,19 +65,12 @@
*/
/* Assembly somewhat optimized copy routines */
-unsigned long __copy_from_user_hexagon(void *to, const void __user *from,
+unsigned long raw_copy_from_user(void *to, const void __user *from,
unsigned long n);
-unsigned long __copy_to_user_hexagon(void __user *to, const void *from,
+unsigned long raw_copy_to_user(void __user *to, const void *from,
unsigned long n);
-
-#define __copy_from_user(to, from, n) __copy_from_user_hexagon(to, from, n)
-#define __copy_to_user(to, from, n) __copy_to_user_hexagon(to, from, n)
-
-/*
- * XXX todo: some additonal performance gain is possible by
- * implementing __copy_to/from_user_inatomic, which is much
- * like __copy_to/from_user, but performs slightly less checking.
- */
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
__kernel_size_t __clear_user_hexagon(void __user *dest, unsigned long count);
#define __clear_user(a, s) __clear_user_hexagon((a), (s))
@@ -107,10 +97,14 @@
return -EFAULT;
if (res > n) {
- copy_from_user(dst, src, n);
+ long left = raw_copy_from_user(dst, src, n);
+ if (unlikely(left))
+ memset(dst + (n - left), 0, left);
return n;
} else {
- copy_from_user(dst, src, res);
+ long left = raw_copy_from_user(dst, src, res);
+ if (unlikely(left))
+ memset(dst + (res - left), 0, left);
return res-1;
}
}
diff --git a/arch/hexagon/kernel/hexagon_ksyms.c b/arch/hexagon/kernel/hexagon_ksyms.c
index af9dec4..00bcad9 100644
--- a/arch/hexagon/kernel/hexagon_ksyms.c
+++ b/arch/hexagon/kernel/hexagon_ksyms.c
@@ -25,8 +25,8 @@
/* Additional functions */
EXPORT_SYMBOL(__clear_user_hexagon);
-EXPORT_SYMBOL(__copy_from_user_hexagon);
-EXPORT_SYMBOL(__copy_to_user_hexagon);
+EXPORT_SYMBOL(raw_copy_from_user);
+EXPORT_SYMBOL(raw_copy_to_user);
EXPORT_SYMBOL(__iounmap);
EXPORT_SYMBOL(__strnlen_user);
EXPORT_SYMBOL(__vmgetie);
diff --git a/arch/hexagon/kernel/time.c b/arch/hexagon/kernel/time.c
index ff4e9bf..29b1f57 100644
--- a/arch/hexagon/kernel/time.c
+++ b/arch/hexagon/kernel/time.c
@@ -199,7 +199,9 @@
clockevents_calc_mult_shift(ce_dev, sleep_clk_freq, 4);
ce_dev->max_delta_ns = clockevent_delta2ns(0x7fffffff, ce_dev);
+ ce_dev->max_delta_ticks = 0x7fffffff;
ce_dev->min_delta_ns = clockevent_delta2ns(0xf, ce_dev);
+ ce_dev->min_delta_ticks = 0xf;
#ifdef CONFIG_SMP
setup_percpu_clockdev();
diff --git a/arch/hexagon/mm/copy_from_user.S b/arch/hexagon/mm/copy_from_user.S
index 7fc94f3..7da066f 100644
--- a/arch/hexagon/mm/copy_from_user.S
+++ b/arch/hexagon/mm/copy_from_user.S
@@ -44,7 +44,7 @@
#define bytes r2
#define loopcount r5
-#define FUNCNAME __copy_from_user_hexagon
+#define FUNCNAME raw_copy_from_user
#include "copy_user_template.S"
/* LOAD FAULTS from COPY_FROM_USER */
diff --git a/arch/hexagon/mm/copy_to_user.S b/arch/hexagon/mm/copy_to_user.S
index 0cfbcc0..a7b7f8d 100644
--- a/arch/hexagon/mm/copy_to_user.S
+++ b/arch/hexagon/mm/copy_to_user.S
@@ -43,7 +43,7 @@
#define bytes r2
#define loopcount r5
-#define FUNCNAME __copy_to_user_hexagon
+#define FUNCNAME raw_copy_to_user
#include "copy_user_template.S"
/* STORE FAULTS from COPY_TO_USER */
diff --git a/arch/ia64/Kconfig b/arch/ia64/Kconfig
index 18ca6a9..6a15083 100644
--- a/arch/ia64/Kconfig
+++ b/arch/ia64/Kconfig
@@ -52,7 +52,6 @@
select MODULES_USE_ELF_RELA
select ARCH_USE_CMPXCHG_LOCKREF
select HAVE_ARCH_AUDITSYSCALL
- select HAVE_ARCH_HARDENED_USERCOPY
default y
help
The Itanium Processor Family is Intel's 64-bit successor to
diff --git a/arch/ia64/include/asm/extable.h b/arch/ia64/include/asm/extable.h
new file mode 100644
index 0000000..20376e7
--- /dev/null
+++ b/arch/ia64/include/asm/extable.h
@@ -0,0 +1,11 @@
+#ifndef _ASM_IA64_EXTABLE_H
+#define _ASM_IA64_EXTABLE_H
+
+#define ARCH_HAS_RELATIVE_EXTABLE
+
+struct exception_table_entry {
+ int insn; /* location-relative address of insn this fixup is for */
+ int fixup; /* location-relative continuation addr.; if bit 2 is set, r9 is set to 0 */
+};
+
+#endif
diff --git a/arch/ia64/include/asm/uaccess.h b/arch/ia64/include/asm/uaccess.h
index 471044b..82a7646 100644
--- a/arch/ia64/include/asm/uaccess.h
+++ b/arch/ia64/include/asm/uaccess.h
@@ -33,14 +33,13 @@
*/
#include <linux/compiler.h>
-#include <linux/errno.h>
-#include <linux/sched.h>
#include <linux/page-flags.h>
#include <linux/mm.h>
#include <asm/intrinsics.h>
#include <asm/pgtable.h>
#include <asm/io.h>
+#include <asm/extable.h>
/*
* For historical reasons, the following macros are grossly misnamed:
@@ -48,9 +47,6 @@
#define KERNEL_DS ((mm_segment_t) { ~0UL }) /* cf. access_ok() */
#define USER_DS ((mm_segment_t) { TASK_SIZE-1 }) /* cf. access_ok() */
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
#define get_ds() (KERNEL_DS)
#define get_fs() (current_thread_info()->addr_limit)
#define set_fs(x) (current_thread_info()->addr_limit = (x))
@@ -63,14 +59,14 @@
* address TASK_SIZE is never valid. We also need to make sure that the address doesn't
* point inside the virtually mapped linear page table.
*/
-#define __access_ok(addr, size, segment) \
-({ \
- __chk_user_ptr(addr); \
- (likely((unsigned long) (addr) <= (segment).seg) \
- && ((segment).seg == KERNEL_DS.seg \
- || likely(REGION_OFFSET((unsigned long) (addr)) < RGN_MAP_LIMIT))); \
-})
-#define access_ok(type, addr, size) __access_ok((addr), (size), get_fs())
+static inline int __access_ok(const void __user *p, unsigned long size)
+{
+ unsigned long addr = (unsigned long)p;
+ unsigned long seg = get_fs().seg;
+ return likely(addr <= seg) &&
+ (seg == KERNEL_DS.seg || likely(REGION_OFFSET(addr) < RGN_MAP_LIMIT));
+}
+#define access_ok(type, addr, size) __access_ok((addr), (size))
/*
* These are the main single-value transfer routines. They automatically
@@ -80,8 +76,8 @@
* (a) re-use the arguments for side effects (sizeof/typeof is ok)
* (b) require any knowledge of processes at this stage
*/
-#define put_user(x, ptr) __put_user_check((__typeof__(*(ptr))) (x), (ptr), sizeof(*(ptr)), get_fs())
-#define get_user(x, ptr) __get_user_check((x), (ptr), sizeof(*(ptr)), get_fs())
+#define put_user(x, ptr) __put_user_check((__typeof__(*(ptr))) (x), (ptr), sizeof(*(ptr)))
+#define get_user(x, ptr) __get_user_check((x), (ptr), sizeof(*(ptr)))
/*
* The "__xxx" versions do not do address space checking, useful when
@@ -184,13 +180,13 @@
* could clobber r8 and r9 (among others). Thus, be careful not to evaluate it while
* using r8/r9.
*/
-#define __do_get_user(check, x, ptr, size, segment) \
+#define __do_get_user(check, x, ptr, size) \
({ \
const __typeof__(*(ptr)) __user *__gu_ptr = (ptr); \
__typeof__ (size) __gu_size = (size); \
long __gu_err = -EFAULT; \
unsigned long __gu_val = 0; \
- if (!check || __access_ok(__gu_ptr, size, segment)) \
+ if (!check || __access_ok(__gu_ptr, size)) \
switch (__gu_size) { \
case 1: __get_user_size(__gu_val, __gu_ptr, 1, __gu_err); break; \
case 2: __get_user_size(__gu_val, __gu_ptr, 2, __gu_err); break; \
@@ -202,8 +198,8 @@
__gu_err; \
})
-#define __get_user_nocheck(x, ptr, size) __do_get_user(0, x, ptr, size, KERNEL_DS)
-#define __get_user_check(x, ptr, size, segment) __do_get_user(1, x, ptr, size, segment)
+#define __get_user_nocheck(x, ptr, size) __do_get_user(0, x, ptr, size)
+#define __get_user_check(x, ptr, size) __do_get_user(1, x, ptr, size)
extern void __put_user_unknown (void);
@@ -211,14 +207,14 @@
* Evaluating arguments X, PTR, SIZE, and SEGMENT may involve subroutine-calls, which
* could clobber r8 (among others). Thus, be careful not to evaluate them while using r8.
*/
-#define __do_put_user(check, x, ptr, size, segment) \
+#define __do_put_user(check, x, ptr, size) \
({ \
__typeof__ (x) __pu_x = (x); \
__typeof__ (*(ptr)) __user *__pu_ptr = (ptr); \
__typeof__ (size) __pu_size = (size); \
long __pu_err = -EFAULT; \
\
- if (!check || __access_ok(__pu_ptr, __pu_size, segment)) \
+ if (!check || __access_ok(__pu_ptr, __pu_size)) \
switch (__pu_size) { \
case 1: __put_user_size(__pu_x, __pu_ptr, 1, __pu_err); break; \
case 2: __put_user_size(__pu_x, __pu_ptr, 2, __pu_err); break; \
@@ -229,8 +225,8 @@
__pu_err; \
})
-#define __put_user_nocheck(x, ptr, size) __do_put_user(0, x, ptr, size, KERNEL_DS)
-#define __put_user_check(x, ptr, size, segment) __do_put_user(1, x, ptr, size, segment)
+#define __put_user_nocheck(x, ptr, size) __do_put_user(0, x, ptr, size)
+#define __put_user_check(x, ptr, size) __do_put_user(1, x, ptr, size)
/*
* Complex access routines
@@ -239,56 +235,19 @@
unsigned long count);
static inline unsigned long
-__copy_to_user (void __user *to, const void *from, unsigned long count)
+raw_copy_to_user(void __user *to, const void *from, unsigned long count)
{
- check_object_size(from, count, true);
-
return __copy_user(to, (__force void __user *) from, count);
}
static inline unsigned long
-__copy_from_user (void *to, const void __user *from, unsigned long count)
+raw_copy_from_user(void *to, const void __user *from, unsigned long count)
{
- check_object_size(to, count, false);
-
return __copy_user((__force void __user *) to, from, count);
}
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
-#define copy_to_user(to, from, n) \
-({ \
- void __user *__cu_to = (to); \
- const void *__cu_from = (from); \
- long __cu_len = (n); \
- \
- if (__access_ok(__cu_to, __cu_len, get_fs())) { \
- check_object_size(__cu_from, __cu_len, true); \
- __cu_len = __copy_user(__cu_to, (__force void __user *) __cu_from, __cu_len); \
- } \
- __cu_len; \
-})
-
-static inline unsigned long
-copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- check_object_size(to, n, false);
- if (likely(__access_ok(from, n, get_fs())))
- n = __copy_user((__force void __user *) to, from, n);
- else
- memset(to, 0, n);
- return n;
-}
-
-#define __copy_in_user(to, from, size) __copy_user((to), (from), (size))
-
-static inline unsigned long
-copy_in_user (void __user *to, const void __user *from, unsigned long n)
-{
- if (likely(access_ok(VERIFY_READ, from, n) && access_ok(VERIFY_WRITE, to, n)))
- n = __copy_user(to, from, n);
- return n;
-}
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
extern unsigned long __do_clear_user (void __user *, unsigned long);
@@ -297,7 +256,7 @@
#define clear_user(to, n) \
({ \
unsigned long __cu_len = (n); \
- if (__access_ok(to, __cu_len, get_fs())) \
+ if (__access_ok(to, __cu_len)) \
__cu_len = __do_clear_user(to, __cu_len); \
__cu_len; \
})
@@ -313,7 +272,7 @@
({ \
const char __user * __sfu_from = (from); \
long __sfu_ret = -EFAULT; \
- if (__access_ok(__sfu_from, 0, get_fs())) \
+ if (__access_ok(__sfu_from, 0)) \
__sfu_ret = __strncpy_from_user((to), __sfu_from, (n)); \
__sfu_ret; \
})
@@ -325,7 +284,7 @@
({ \
const char __user *__su_str = (str); \
unsigned long __su_ret = 0; \
- if (__access_ok(__su_str, 0, get_fs())) \
+ if (__access_ok(__su_str, 0)) \
__su_ret = __strlen_user(__su_str); \
__su_ret; \
})
@@ -341,18 +300,11 @@
({ \
const char __user *__su_str = (str); \
unsigned long __su_ret = 0; \
- if (__access_ok(__su_str, 0, get_fs())) \
+ if (__access_ok(__su_str, 0)) \
__su_ret = __strnlen_user(__su_str, len); \
__su_ret; \
})
-#define ARCH_HAS_RELATIVE_EXTABLE
-
-struct exception_table_entry {
- int insn; /* location-relative address of insn this fixup is for */
- int fixup; /* location-relative continuation addr.; if bit 2 is set, r9 is set to 0 */
-};
-
#define ARCH_HAS_TRANSLATE_MEM_PTR 1
static __inline__ void *
xlate_dev_mem_ptr(phys_addr_t p)
diff --git a/arch/ia64/include/uapi/asm/socket.h b/arch/ia64/include/uapi/asm/socket.h
index 57feb0c..2c3f4b4 100644
--- a/arch/ia64/include/uapi/asm/socket.h
+++ b/arch/ia64/include/uapi/asm/socket.h
@@ -101,4 +101,10 @@
#define SCM_TIMESTAMPING_OPT_STATS 54
+#define SO_MEMINFO 55
+
+#define SO_INCOMING_NAPI_ID 56
+
+#define SO_COOKIE 57
+
#endif /* _ASM_IA64_SOCKET_H */
diff --git a/arch/ia64/kernel/module.c b/arch/ia64/kernel/module.c
index 6ab0ae7..d1d945c 100644
--- a/arch/ia64/kernel/module.c
+++ b/arch/ia64/kernel/module.c
@@ -153,7 +153,7 @@
static int
apply_imm64 (struct module *mod, struct insn *insn, uint64_t val)
{
- if (slot(insn) != 2) {
+ if (slot(insn) != 1 && slot(insn) != 2) {
printk(KERN_ERR "%s: invalid slot number %d for IMM64\n",
mod->name, slot(insn));
return 0;
@@ -165,7 +165,7 @@
static int
apply_imm60 (struct module *mod, struct insn *insn, uint64_t val)
{
- if (slot(insn) != 2) {
+ if (slot(insn) != 1 && slot(insn) != 2) {
printk(KERN_ERR "%s: invalid slot number %d for IMM60\n",
mod->name, slot(insn));
return 0;
diff --git a/arch/ia64/kernel/salinfo.c b/arch/ia64/kernel/salinfo.c
index d194d5c..63dc9cd 100644
--- a/arch/ia64/kernel/salinfo.c
+++ b/arch/ia64/kernel/salinfo.c
@@ -179,14 +179,14 @@
const u8 *efi_guid;
u8 **oemdata;
u64 *oemdata_size;
- int ret;
};
-static void
+static long
salinfo_platform_oemdata_cpu(void *context)
{
struct salinfo_platform_oemdata_parms *parms = context;
- parms->ret = salinfo_platform_oemdata(parms->efi_guid, parms->oemdata, parms->oemdata_size);
+
+ return salinfo_platform_oemdata(parms->efi_guid, parms->oemdata, parms->oemdata_size);
}
static void
@@ -380,16 +380,7 @@
return 0;
}
-static void
-call_on_cpu(int cpu, void (*fn)(void *), void *arg)
-{
- cpumask_t save_cpus_allowed = current->cpus_allowed;
- set_cpus_allowed_ptr(current, cpumask_of(cpu));
- (*fn)(arg);
- set_cpus_allowed_ptr(current, &save_cpus_allowed);
-}
-
-static void
+static long
salinfo_log_read_cpu(void *context)
{
struct salinfo_data *data = context;
@@ -399,6 +390,7 @@
/* Clear corrected errors as they are read from SAL */
if (rh->severity == sal_log_severity_corrected)
ia64_sal_clear_state_info(data->type);
+ return 0;
}
static void
@@ -430,7 +422,7 @@
spin_unlock_irqrestore(&data_saved_lock, flags);
if (!data->saved_num)
- call_on_cpu(cpu, salinfo_log_read_cpu, data);
+ work_on_cpu_safe(cpu, salinfo_log_read_cpu, data);
if (!data->log_size) {
data->state = STATE_NO_DATA;
cpumask_clear_cpu(cpu, &data->cpu_event);
@@ -459,11 +451,13 @@
return simple_read_from_buffer(buffer, count, ppos, buf, bufsize);
}
-static void
+static long
salinfo_log_clear_cpu(void *context)
{
struct salinfo_data *data = context;
+
ia64_sal_clear_state_info(data->type);
+ return 0;
}
static int
@@ -486,7 +480,7 @@
rh = (sal_log_record_header_t *)(data->log_buffer);
/* Corrected errors have already been cleared from SAL */
if (rh->severity != sal_log_severity_corrected)
- call_on_cpu(cpu, salinfo_log_clear_cpu, data);
+ work_on_cpu_safe(cpu, salinfo_log_clear_cpu, data);
/* clearing a record may make a new record visible */
salinfo_log_new_read(cpu, data);
if (data->state == STATE_LOG_RECORD) {
@@ -531,9 +525,8 @@
.oemdata = &data->oemdata,
.oemdata_size = &data->oemdata_size
};
- call_on_cpu(cpu, salinfo_platform_oemdata_cpu, &parms);
- if (parms.ret)
- count = parms.ret;
+ count = work_on_cpu_safe(cpu, salinfo_platform_oemdata_cpu,
+ &parms);
} else
data->oemdata_size = 0;
} else
diff --git a/arch/ia64/kernel/topology.c b/arch/ia64/kernel/topology.c
index 1a68f01..d76529c 100644
--- a/arch/ia64/kernel/topology.c
+++ b/arch/ia64/kernel/topology.c
@@ -355,18 +355,12 @@
unsigned long i, j;
struct cache_info *this_object;
int retval = 0;
- cpumask_t oldmask;
if (all_cpu_cache_info[cpu].kobj.parent)
return 0;
- oldmask = current->cpus_allowed;
- retval = set_cpus_allowed_ptr(current, cpumask_of(cpu));
- if (unlikely(retval))
- return retval;
retval = cpu_cache_sysfs_init(cpu);
- set_cpus_allowed_ptr(current, &oldmask);
if (unlikely(retval < 0))
return retval;
diff --git a/arch/ia64/kernel/vmlinux.lds.S b/arch/ia64/kernel/vmlinux.lds.S
index f89d20c..798026d 100644
--- a/arch/ia64/kernel/vmlinux.lds.S
+++ b/arch/ia64/kernel/vmlinux.lds.S
@@ -192,6 +192,8 @@
CONSTRUCTORS
}
+ BUG_TABLE
+
. = ALIGN(16); /* gp must be 16-byte aligned for exc. table */
.got : AT(ADDR(.got) - LOAD_OFFSET) {
*(.got.plt)
diff --git a/arch/ia64/lib/memcpy_mck.S b/arch/ia64/lib/memcpy_mck.S
index b264b6a..bbbadc4 100644
--- a/arch/ia64/lib/memcpy_mck.S
+++ b/arch/ia64/lib/memcpy_mck.S
@@ -556,9 +556,6 @@
#define D r22
#define F r28
-#define memset_arg0 r32
-#define memset_arg2 r33
-
#define saved_retval loc0
#define saved_rtlink loc1
#define saved_pfs_stack loc2
@@ -622,7 +619,7 @@
* (faulting_addr - orig_dst) -> len to faulting st address
* B = (cur_dst - orig_dst) -> len copied so far
* C = A - B -> len need to be copied
- * D = orig_len - A -> len need to be zeroed
+ * D = orig_len - A -> len need to be left along
*/
(p6) sub A = F, saved_in0
(p7) sub A = F, saved_in1
@@ -638,9 +635,6 @@
sub D = saved_in2, A
;;
cmp.gt p8,p0=C,r0 // more than 1 byte?
- add memset_arg0=saved_in0, A
-(p6) mov memset_arg2=0 // copy_to_user should not call memset
-(p7) mov memset_arg2=D // copy_from_user need to have kbuf zeroed
mov r8=0
mov saved_retval = D
mov saved_rtlink = b0
@@ -652,11 +646,6 @@
;;
add saved_retval=saved_retval,r8 // above might return non-zero value
- cmp.gt p8,p0=memset_arg2,r0 // more than 1 byte?
- mov out0=memset_arg0 // *s
- mov out1=r0 // c
- mov out2=memset_arg2 // n
-(p8) br.call.sptk.few b0=memset
;;
mov retval=saved_retval
diff --git a/arch/ia64/mm/extable.c b/arch/ia64/mm/extable.c
index 4edb816..10dd4a6 100644
--- a/arch/ia64/mm/extable.c
+++ b/arch/ia64/mm/extable.c
@@ -5,7 +5,10 @@
* David Mosberger-Tang <davidm@hpl.hp.com>
*/
-#include <linux/uaccess.h>
+#include <asm/ptrace.h>
+#include <asm/extable.h>
+#include <asm/errno.h>
+#include <asm/processor.h>
void
ia64_handle_exception (struct pt_regs *regs, const struct exception_table_entry *e)
diff --git a/arch/ia64/sn/kernel/sn2/sn_hwperf.c b/arch/ia64/sn/kernel/sn2/sn_hwperf.c
index 52704f1..55febd6 100644
--- a/arch/ia64/sn/kernel/sn2/sn_hwperf.c
+++ b/arch/ia64/sn/kernel/sn2/sn_hwperf.c
@@ -598,12 +598,17 @@
op_info->ret = r;
}
+static long sn_hwperf_call_sal_work(void *info)
+{
+ sn_hwperf_call_sal(info);
+ return 0;
+}
+
static int sn_hwperf_op_cpu(struct sn_hwperf_op_info *op_info)
{
u32 cpu;
u32 use_ipi;
int r = 0;
- cpumask_t save_allowed;
cpu = (op_info->a->arg & SN_HWPERF_ARG_CPU_MASK) >> 32;
use_ipi = op_info->a->arg & SN_HWPERF_ARG_USE_IPI_MASK;
@@ -629,13 +634,9 @@
/* use an interprocessor interrupt to call SAL */
smp_call_function_single(cpu, sn_hwperf_call_sal,
op_info, 1);
- }
- else {
- /* migrate the task before calling SAL */
- save_allowed = current->cpus_allowed;
- set_cpus_allowed_ptr(current, cpumask_of(cpu));
- sn_hwperf_call_sal(op_info);
- set_cpus_allowed_ptr(current, &save_allowed);
+ } else {
+ /* Call on the target CPU */
+ work_on_cpu_safe(cpu, sn_hwperf_call_sal_work, op_info);
}
}
r = op_info->ret;
diff --git a/arch/m32r/include/asm/Kbuild b/arch/m32r/include/asm/Kbuild
index deb2987..c000ffa 100644
--- a/arch/m32r/include/asm/Kbuild
+++ b/arch/m32r/include/asm/Kbuild
@@ -2,6 +2,7 @@
generic-y += clkdev.h
generic-y += current.h
generic-y += exec.h
+generic-y += extable.h
generic-y += irq_work.h
generic-y += kvm_para.h
generic-y += mcs_spinlock.h
diff --git a/arch/m32r/include/asm/uaccess.h b/arch/m32r/include/asm/uaccess.h
index 6f89821..07be349 100644
--- a/arch/m32r/include/asm/uaccess.h
+++ b/arch/m32r/include/asm/uaccess.h
@@ -11,13 +11,9 @@
/*
* User space memory access functions
*/
-#include <linux/errno.h>
-#include <linux/thread_info.h>
#include <asm/page.h>
#include <asm/setup.h>
-
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
+#include <linux/prefetch.h>
/*
* The fs value determines whether argument validity checking should be
@@ -114,25 +110,7 @@
}
#endif /* CONFIG_MMU */
-/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-
-struct exception_table_entry
-{
- unsigned long insn, fixup;
-};
-
-extern int fixup_exception(struct pt_regs *regs);
+#include <asm/extable.h>
/*
* These are the main single-value transfer routines. They automatically
@@ -483,173 +461,24 @@
: "r14", "memory"); \
} while (0)
-#define __copy_user_zeroing(to, from, size) \
-do { \
- unsigned long __dst, __src, __c; \
- __asm__ __volatile__ ( \
- " mv r14, %0\n" \
- " or r14, %1\n" \
- " beq %0, %1, 9f\n" \
- " beqz %2, 9f\n" \
- " and3 r14, r14, #3\n" \
- " bnez r14, 2f\n" \
- " and3 %2, %2, #3\n" \
- " beqz %3, 2f\n" \
- " addi %0, #-4 ; word_copy \n" \
- " .fillinsn\n" \
- "0: ld r14, @%1+\n" \
- " addi %3, #-1\n" \
- " .fillinsn\n" \
- "1: st r14, @+%0\n" \
- " bnez %3, 0b\n" \
- " beqz %2, 9f\n" \
- " addi %0, #4\n" \
- " .fillinsn\n" \
- "2: ldb r14, @%1 ; byte_copy \n" \
- " .fillinsn\n" \
- "3: stb r14, @%0\n" \
- " addi %1, #1\n" \
- " addi %2, #-1\n" \
- " addi %0, #1\n" \
- " bnez %2, 2b\n" \
- " .fillinsn\n" \
- "9:\n" \
- ".section .fixup,\"ax\"\n" \
- " .balign 4\n" \
- "5: addi %3, #1\n" \
- " addi %1, #-4\n" \
- " .fillinsn\n" \
- "6: slli %3, #2\n" \
- " add %2, %3\n" \
- " addi %0, #4\n" \
- " .fillinsn\n" \
- "7: ldi r14, #0 ; store zero \n" \
- " .fillinsn\n" \
- "8: addi %2, #-1\n" \
- " stb r14, @%0 ; ACE? \n" \
- " addi %0, #1\n" \
- " bnez %2, 8b\n" \
- " seth r14, #high(9b)\n" \
- " or3 r14, r14, #low(9b)\n" \
- " jmp r14\n" \
- ".previous\n" \
- ".section __ex_table,\"a\"\n" \
- " .balign 4\n" \
- " .long 0b,6b\n" \
- " .long 1b,5b\n" \
- " .long 2b,7b\n" \
- " .long 3b,7b\n" \
- ".previous\n" \
- : "=&r" (__dst), "=&r" (__src), "=&r" (size), \
- "=&r" (__c) \
- : "0" (to), "1" (from), "2" (size), "3" (size / 4) \
- : "r14", "memory"); \
-} while (0)
-
-
/* We let the __ versions of copy_from/to_user inline, because they're often
* used in fast paths and have only a small space overhead.
*/
-static inline unsigned long __generic_copy_from_user_nocheck(void *to,
- const void __user *from, unsigned long n)
+static inline unsigned long
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
- __copy_user_zeroing(to, from, n);
- return n;
-}
-
-static inline unsigned long __generic_copy_to_user_nocheck(void __user *to,
- const void *from, unsigned long n)
-{
+ prefetchw(to);
__copy_user(to, from, n);
return n;
}
-unsigned long __generic_copy_to_user(void __user *, const void *, unsigned long);
-unsigned long __generic_copy_from_user(void *, const void __user *, unsigned long);
-
-/**
- * __copy_to_user: - Copy a block of data into user space, with less checking.
- * @to: Destination address, in user space.
- * @from: Source address, in kernel space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from kernel space to user space. Caller must check
- * the specified block with access_ok() before calling this function.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- */
-#define __copy_to_user(to, from, n) \
- __generic_copy_to_user_nocheck((to), (from), (n))
-
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
-
-/**
- * copy_to_user: - Copy a block of data into user space.
- * @to: Destination address, in user space.
- * @from: Source address, in kernel space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from kernel space to user space.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- */
-#define copy_to_user(to, from, n) \
-({ \
- might_fault(); \
- __generic_copy_to_user((to), (from), (n)); \
-})
-
-/**
- * __copy_from_user: - Copy a block of data from user space, with less checking. * @to: Destination address, in kernel space.
- * @from: Source address, in user space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from user space to kernel space. Caller must check
- * the specified block with access_ok() before calling this function.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- *
- * If some data could not be copied, this function will pad the copied
- * data to the requested size using zero bytes.
- */
-#define __copy_from_user(to, from, n) \
- __generic_copy_from_user_nocheck((to), (from), (n))
-
-/**
- * copy_from_user: - Copy a block of data from user space.
- * @to: Destination address, in kernel space.
- * @from: Source address, in user space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from user space to kernel space.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- *
- * If some data could not be copied, this function will pad the copied
- * data to the requested size using zero bytes.
- */
-#define copy_from_user(to, from, n) \
-({ \
- might_fault(); \
- __generic_copy_from_user((to), (from), (n)); \
-})
+static inline unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
+{
+ prefetch(from);
+ __copy_user(to, from, n);
+ return n;
+}
long __must_check strncpy_from_user(char *dst, const char __user *src,
long count);
diff --git a/arch/m32r/include/uapi/asm/socket.h b/arch/m32r/include/uapi/asm/socket.h
index 5853f8e9..ae6548d 100644
--- a/arch/m32r/include/uapi/asm/socket.h
+++ b/arch/m32r/include/uapi/asm/socket.h
@@ -92,4 +92,10 @@
#define SCM_TIMESTAMPING_OPT_STATS 54
+#define SO_MEMINFO 55
+
+#define SO_INCOMING_NAPI_ID 56
+
+#define SO_COOKIE 57
+
#endif /* _ASM_M32R_SOCKET_H */
diff --git a/arch/m32r/kernel/m32r_ksyms.c b/arch/m32r/kernel/m32r_ksyms.c
index d763f0b..a4d43b5 100644
--- a/arch/m32r/kernel/m32r_ksyms.c
+++ b/arch/m32r/kernel/m32r_ksyms.c
@@ -26,8 +26,6 @@
EXPORT_SYMBOL(__strncpy_from_user);
EXPORT_SYMBOL(clear_user);
EXPORT_SYMBOL(__clear_user);
-EXPORT_SYMBOL(__generic_copy_from_user);
-EXPORT_SYMBOL(__generic_copy_to_user);
EXPORT_SYMBOL(strnlen_user);
#ifdef CONFIG_SMP
diff --git a/arch/m32r/lib/usercopy.c b/arch/m32r/lib/usercopy.c
index fd03f27..b3ef2c8 100644
--- a/arch/m32r/lib/usercopy.c
+++ b/arch/m32r/lib/usercopy.c
@@ -11,27 +11,6 @@
#include <linux/thread_info.h>
#include <linux/uaccess.h>
-unsigned long
-__generic_copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- prefetch(from);
- if (access_ok(VERIFY_WRITE, to, n))
- __copy_user(to,from,n);
- return n;
-}
-
-unsigned long
-__generic_copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- prefetchw(to);
- if (access_ok(VERIFY_READ, from, n))
- __copy_user_zeroing(to,from,n);
- else
- memset(to, 0, n);
- return n;
-}
-
-
/*
* Copy a null terminated string from userspace.
*/
diff --git a/arch/m68k/coldfire/pit.c b/arch/m68k/coldfire/pit.c
index 175553d..6c08780 100644
--- a/arch/m68k/coldfire/pit.c
+++ b/arch/m68k/coldfire/pit.c
@@ -149,8 +149,10 @@
cf_pit_clockevent.mult = div_sc(FREQ, NSEC_PER_SEC, 32);
cf_pit_clockevent.max_delta_ns =
clockevent_delta2ns(0xFFFF, &cf_pit_clockevent);
+ cf_pit_clockevent.max_delta_ticks = 0xFFFF;
cf_pit_clockevent.min_delta_ns =
clockevent_delta2ns(0x3f, &cf_pit_clockevent);
+ cf_pit_clockevent.min_delta_ticks = 0x3f;
clockevents_register_device(&cf_pit_clockevent);
setup_irq(MCF_IRQ_PIT1, &pit_irq);
diff --git a/arch/m68k/include/asm/Kbuild b/arch/m68k/include/asm/Kbuild
index d4f9ccb..82005d2 100644
--- a/arch/m68k/include/asm/Kbuild
+++ b/arch/m68k/include/asm/Kbuild
@@ -5,6 +5,7 @@
generic-y += emergency-restart.h
generic-y += errno.h
generic-y += exec.h
+generic-y += extable.h
generic-y += futex.h
generic-y += hw_irq.h
generic-y += ioctl.h
diff --git a/arch/m68k/include/asm/processor.h b/arch/m68k/include/asm/processor.h
index f5f790c..77239e8 100644
--- a/arch/m68k/include/asm/processor.h
+++ b/arch/m68k/include/asm/processor.h
@@ -122,16 +122,6 @@
wrusp(usp);
}
-#ifdef CONFIG_MMU
-extern int handle_kernel_fault(struct pt_regs *regs);
-#else
-static inline int handle_kernel_fault(struct pt_regs *regs)
-{
- /* Any fault in kernel is fatal on non-mmu */
- return 0;
-}
-#endif
-
/* Forward declaration, a strange C thing */
struct task_struct;
diff --git a/arch/m68k/include/asm/uaccess.h b/arch/m68k/include/asm/uaccess.h
index 3fadc4a..67b3481 100644
--- a/arch/m68k/include/asm/uaccess.h
+++ b/arch/m68k/include/asm/uaccess.h
@@ -4,6 +4,7 @@
#include <asm/uaccess_mm.h>
#endif
+#include <asm/extable.h>
#ifdef CONFIG_CPU_HAS_NO_UNALIGNED
#include <asm-generic/uaccess-unaligned.h>
#else
diff --git a/arch/m68k/include/asm/uaccess_mm.h b/arch/m68k/include/asm/uaccess_mm.h
index d228601..ef856ff 100644
--- a/arch/m68k/include/asm/uaccess_mm.h
+++ b/arch/m68k/include/asm/uaccess_mm.h
@@ -5,14 +5,9 @@
* User space memory access functions
*/
#include <linux/compiler.h>
-#include <linux/errno.h>
#include <linux/types.h>
-#include <linux/sched.h>
#include <asm/segment.h>
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
/* We let the MMU do all checking */
static inline int access_ok(int type, const void __user *addr,
unsigned long size)
@@ -36,24 +31,6 @@
#define MOVES "move"
#endif
-/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-
-struct exception_table_entry
-{
- unsigned long insn, fixup;
-};
-
extern int __put_user_bad(void);
extern int __get_user_bad(void);
@@ -202,39 +179,55 @@
unsigned long __generic_copy_from_user(void *to, const void __user *from, unsigned long n);
unsigned long __generic_copy_to_user(void __user *to, const void *from, unsigned long n);
-#define __constant_copy_from_user_asm(res, to, from, tmp, n, s1, s2, s3)\
+#define __suffix0
+#define __suffix1 b
+#define __suffix2 w
+#define __suffix4 l
+
+#define ____constant_copy_from_user_asm(res, to, from, tmp, n1, n2, n3, s1, s2, s3)\
asm volatile ("\n" \
"1: "MOVES"."#s1" (%2)+,%3\n" \
" move."#s1" %3,(%1)+\n" \
+ " .ifnc \""#s2"\",\"\"\n" \
"2: "MOVES"."#s2" (%2)+,%3\n" \
" move."#s2" %3,(%1)+\n" \
" .ifnc \""#s3"\",\"\"\n" \
"3: "MOVES"."#s3" (%2)+,%3\n" \
" move."#s3" %3,(%1)+\n" \
" .endif\n" \
+ " .endif\n" \
"4:\n" \
" .section __ex_table,\"a\"\n" \
" .align 4\n" \
" .long 1b,10f\n" \
+ " .ifnc \""#s2"\",\"\"\n" \
" .long 2b,20f\n" \
" .ifnc \""#s3"\",\"\"\n" \
" .long 3b,30f\n" \
" .endif\n" \
+ " .endif\n" \
" .previous\n" \
"\n" \
" .section .fixup,\"ax\"\n" \
" .even\n" \
- "10: clr."#s1" (%1)+\n" \
- "20: clr."#s2" (%1)+\n" \
+ "10: addq.l #"#n1",%0\n" \
+ " .ifnc \""#s2"\",\"\"\n" \
+ "20: addq.l #"#n2",%0\n" \
" .ifnc \""#s3"\",\"\"\n" \
- "30: clr."#s3" (%1)+\n" \
+ "30: addq.l #"#n3",%0\n" \
" .endif\n" \
- " moveq.l #"#n",%0\n" \
+ " .endif\n" \
" jra 4b\n" \
" .previous\n" \
: "+d" (res), "+&a" (to), "+a" (from), "=&d" (tmp) \
: : "memory")
+#define ___constant_copy_from_user_asm(res, to, from, tmp, n1, n2, n3, s1, s2, s3)\
+ ____constant_copy_from_user_asm(res, to, from, tmp, n1, n2, n3, s1, s2, s3)
+#define __constant_copy_from_user_asm(res, to, from, tmp, n1, n2, n3) \
+ ___constant_copy_from_user_asm(res, to, from, tmp, n1, n2, n3, \
+ __suffix##n1, __suffix##n2, __suffix##n3)
+
static __always_inline unsigned long
__constant_copy_from_user(void *to, const void __user *from, unsigned long n)
{
@@ -242,37 +235,37 @@
switch (n) {
case 1:
- __get_user_asm(res, *(u8 *)to, (u8 __user *)from, u8, b, d, 1);
+ __constant_copy_from_user_asm(res, to, from, tmp, 1, 0, 0);
break;
case 2:
- __get_user_asm(res, *(u16 *)to, (u16 __user *)from, u16, w, r, 2);
+ __constant_copy_from_user_asm(res, to, from, tmp, 2, 0, 0);
break;
case 3:
- __constant_copy_from_user_asm(res, to, from, tmp, 3, w, b,);
+ __constant_copy_from_user_asm(res, to, from, tmp, 2, 1, 0);
break;
case 4:
- __get_user_asm(res, *(u32 *)to, (u32 __user *)from, u32, l, r, 4);
+ __constant_copy_from_user_asm(res, to, from, tmp, 4, 0, 0);
break;
case 5:
- __constant_copy_from_user_asm(res, to, from, tmp, 5, l, b,);
+ __constant_copy_from_user_asm(res, to, from, tmp, 4, 1, 0);
break;
case 6:
- __constant_copy_from_user_asm(res, to, from, tmp, 6, l, w,);
+ __constant_copy_from_user_asm(res, to, from, tmp, 4, 2, 0);
break;
case 7:
- __constant_copy_from_user_asm(res, to, from, tmp, 7, l, w, b);
+ __constant_copy_from_user_asm(res, to, from, tmp, 4, 2, 1);
break;
case 8:
- __constant_copy_from_user_asm(res, to, from, tmp, 8, l, l,);
+ __constant_copy_from_user_asm(res, to, from, tmp, 4, 4, 0);
break;
case 9:
- __constant_copy_from_user_asm(res, to, from, tmp, 9, l, l, b);
+ __constant_copy_from_user_asm(res, to, from, tmp, 4, 4, 1);
break;
case 10:
- __constant_copy_from_user_asm(res, to, from, tmp, 10, l, l, w);
+ __constant_copy_from_user_asm(res, to, from, tmp, 4, 4, 2);
break;
case 12:
- __constant_copy_from_user_asm(res, to, from, tmp, 12, l, l, l);
+ __constant_copy_from_user_asm(res, to, from, tmp, 4, 4, 4);
break;
default:
/* we limit the inlined version to 3 moves */
@@ -363,24 +356,26 @@
return res;
}
-#define __copy_from_user(to, from, n) \
-(__builtin_constant_p(n) ? \
- __constant_copy_from_user(to, from, n) : \
- __generic_copy_from_user(to, from, n))
+static inline unsigned long
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
+{
+ if (__builtin_constant_p(n))
+ return __constant_copy_from_user(to, from, n);
+ return __generic_copy_from_user(to, from, n);
+}
-#define __copy_to_user(to, from, n) \
-(__builtin_constant_p(n) ? \
- __constant_copy_to_user(to, from, n) : \
- __generic_copy_to_user(to, from, n))
-
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
-
-#define copy_from_user(to, from, n) __copy_from_user(to, from, n)
-#define copy_to_user(to, from, n) __copy_to_user(to, from, n)
+static inline unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
+{
+ if (__builtin_constant_p(n))
+ return __constant_copy_to_user(to, from, n);
+ return __generic_copy_to_user(to, from, n);
+}
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
#define user_addr_max() \
- (segment_eq(get_fs(), USER_DS) ? TASK_SIZE : ~0UL)
+ (uaccess_kernel() ? ~0UL : TASK_SIZE)
extern long strncpy_from_user(char *dst, const char __user *src, long count);
extern __must_check long strlen_user(const char __user *str);
diff --git a/arch/m68k/include/asm/uaccess_no.h b/arch/m68k/include/asm/uaccess_no.h
index 36deeb3..e482c38 100644
--- a/arch/m68k/include/asm/uaccess_no.h
+++ b/arch/m68k/include/asm/uaccess_no.h
@@ -4,15 +4,11 @@
/*
* User space memory access functions
*/
-#include <linux/sched.h>
#include <linux/mm.h>
#include <linux/string.h>
#include <asm/segment.h>
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
#define access_ok(type,addr,size) _access_ok((unsigned long)(addr),(size))
/*
@@ -27,25 +23,6 @@
}
/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-
-struct exception_table_entry
-{
- unsigned long insn, fixup;
-};
-
-
-/*
* These are the main single-value transfer routines. They automatically
* use the right size if we just have the right pointer type.
*/
@@ -124,13 +101,21 @@
: "=d" (x) \
: "m" (*__ptr(ptr)))
-#define copy_from_user(to, from, n) (memcpy(to, from, n), 0)
-#define copy_to_user(to, from, n) (memcpy(to, from, n), 0)
+static inline unsigned long
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
+{
+ memcpy(to, (__force const void *)from, n);
+ return 0;
+}
-#define __copy_from_user(to, from, n) copy_from_user(to, from, n)
-#define __copy_to_user(to, from, n) copy_to_user(to, from, n)
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
+static inline unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
+{
+ memcpy((__force void *)to, from, n);
+ return 0;
+}
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
/*
* Copy a null terminated string from userspace.
diff --git a/arch/m68k/kernel/signal.c b/arch/m68k/kernel/signal.c
index 093b7c4..6f945bb 100644
--- a/arch/m68k/kernel/signal.c
+++ b/arch/m68k/kernel/signal.c
@@ -88,7 +88,7 @@
return frame_size_change[f];
}
-int handle_kernel_fault(struct pt_regs *regs)
+int fixup_exception(struct pt_regs *regs)
{
const struct exception_table_entry *fixup;
struct pt_regs *tregs;
diff --git a/arch/m68k/kernel/traps.c b/arch/m68k/kernel/traps.c
index a926d2c..c1cc4e9 100644
--- a/arch/m68k/kernel/traps.c
+++ b/arch/m68k/kernel/traps.c
@@ -1016,8 +1016,13 @@
/* traced a trapping instruction on a 68020/30,
* real exception will be executed afterwards.
*/
- } else if (!handle_kernel_fault(&fp->ptregs))
- bad_super_trap(fp);
+ return;
+ }
+#ifdef CONFIG_MMU
+ if (fixup_exception(&fp->ptregs))
+ return;
+#endif
+ bad_super_trap(fp);
return;
}
diff --git a/arch/m68k/lib/uaccess.c b/arch/m68k/lib/uaccess.c
index a76b73a..7646e46 100644
--- a/arch/m68k/lib/uaccess.c
+++ b/arch/m68k/lib/uaccess.c
@@ -30,19 +30,13 @@
"6:\n"
" .section .fixup,\"ax\"\n"
" .even\n"
- "10: move.l %0,%3\n"
- "7: clr.l (%2)+\n"
- " subq.l #1,%3\n"
- " jne 7b\n"
- " lsl.l #2,%0\n"
+ "10: lsl.l #2,%0\n"
" btst #1,%5\n"
" jeq 8f\n"
- "30: clr.w (%2)+\n"
- " addq.l #2,%0\n"
+ "30: addq.l #2,%0\n"
"8: btst #0,%5\n"
" jeq 6b\n"
- "50: clr.b (%2)+\n"
- " addq.l #1,%0\n"
+ "50: addq.l #1,%0\n"
" jra 6b\n"
" .previous\n"
"\n"
diff --git a/arch/m68k/mac/config.c b/arch/m68k/mac/config.c
index 9dc65a4..22123f7 100644
--- a/arch/m68k/mac/config.c
+++ b/arch/m68k/mac/config.c
@@ -150,7 +150,7 @@
void __init config_mac(void)
{
if (!MACH_IS_MAC)
- printk(KERN_ERR "ERROR: no Mac, but config_mac() called!!\n");
+ pr_err("ERROR: no Mac, but config_mac() called!!\n");
mach_sched_init = mac_sched_init;
mach_init_IRQ = mac_init_IRQ;
@@ -837,8 +837,7 @@
/* no bootinfo model id -> NetBSD booter was used! */
/* XXX FIXME: breaks for model > 31 */
model = (mac_bi_data.cpuid >> 2) & 63;
- printk(KERN_WARNING "No bootinfo model ID, using cpuid instead "
- "(obsolete bootloader?)\n");
+ pr_warn("No bootinfo model ID, using cpuid instead (obsolete bootloader?)\n");
}
macintosh_config = mac_data_table;
@@ -880,14 +879,13 @@
*/
iop_preinit();
- printk(KERN_INFO "Detected Macintosh model: %d\n", model);
+ pr_info("Detected Macintosh model: %d\n", model);
/*
* Report booter data:
*/
printk(KERN_DEBUG " Penguin bootinfo data:\n");
- printk(KERN_DEBUG " Video: addr 0x%lx "
- "row 0x%lx depth %lx dimensions %ld x %ld\n",
+ printk(KERN_DEBUG " Video: addr 0x%lx row 0x%lx depth %lx dimensions %ld x %ld\n",
mac_bi_data.videoaddr, mac_bi_data.videorow,
mac_bi_data.videodepth, mac_bi_data.dimensions & 0xFFFF,
mac_bi_data.dimensions >> 16);
@@ -912,7 +910,7 @@
static void __init mac_report_hardware(void)
{
- printk(KERN_INFO "Apple Macintosh %s\n", macintosh_config->name);
+ pr_info("Apple Macintosh %s\n", macintosh_config->name);
}
static void mac_get_model(char *str)
@@ -921,15 +919,6 @@
strcat(str, macintosh_config->name);
}
-static struct resource swim_rsrc = { .flags = IORESOURCE_MEM };
-
-static struct platform_device swim_pdev = {
- .name = "swim",
- .id = -1,
- .num_resources = 1,
- .resource = &swim_rsrc,
-};
-
static const struct resource mac_scsi_iifx_rsrc[] __initconst = {
{
.flags = IORESOURCE_IRQ,
@@ -994,26 +983,6 @@
},
};
-static struct platform_device esp_0_pdev = {
- .name = "mac_esp",
- .id = 0,
-};
-
-static struct platform_device esp_1_pdev = {
- .name = "mac_esp",
- .id = 1,
-};
-
-static struct platform_device sonic_pdev = {
- .name = "macsonic",
- .id = -1,
-};
-
-static struct platform_device mace_pdev = {
- .name = "macmace",
- .id = -1,
-};
-
int __init mac_platform_init(void)
{
u8 *swim_base;
@@ -1045,9 +1014,13 @@
}
if (swim_base) {
- swim_rsrc.start = (resource_size_t) swim_base,
- swim_rsrc.end = (resource_size_t) swim_base + 0x2000,
- platform_device_register(&swim_pdev);
+ struct resource swim_rsrc = {
+ .flags = IORESOURCE_MEM,
+ .start = (resource_size_t)swim_base,
+ .end = (resource_size_t)swim_base + 0x2000,
+ };
+
+ platform_device_register_simple("swim", -1, &swim_rsrc, 1);
}
/*
@@ -1057,13 +1030,13 @@
switch (macintosh_config->scsi_type) {
case MAC_SCSI_QUADRA:
case MAC_SCSI_QUADRA3:
- platform_device_register(&esp_0_pdev);
+ platform_device_register_simple("mac_esp", 0, NULL, 0);
break;
case MAC_SCSI_QUADRA2:
- platform_device_register(&esp_0_pdev);
+ platform_device_register_simple("mac_esp", 0, NULL, 0);
if ((macintosh_config->ident == MAC_MODEL_Q900) ||
(macintosh_config->ident == MAC_MODEL_Q950))
- platform_device_register(&esp_1_pdev);
+ platform_device_register_simple("mac_esp", 1, NULL, 0);
break;
case MAC_SCSI_IIFX:
/* Addresses from The Guide to Mac Family Hardware.
@@ -1129,10 +1102,10 @@
switch (macintosh_config->ether_type) {
case MAC_ETHER_SONIC:
- platform_device_register(&sonic_pdev);
+ platform_device_register_simple("macsonic", -1, NULL, 0);
break;
case MAC_ETHER_MACE:
- platform_device_register(&mace_pdev);
+ platform_device_register_simple("macmace", -1, NULL, 0);
break;
}
diff --git a/arch/m68k/mac/iop.c b/arch/m68k/mac/iop.c
index 7990b6f..4c1e606 100644
--- a/arch/m68k/mac/iop.c
+++ b/arch/m68k/mac/iop.c
@@ -115,7 +115,17 @@
#include <asm/macints.h>
#include <asm/mac_iop.h>
-/*#define DEBUG_IOP*/
+#ifdef DEBUG
+#define iop_pr_debug(fmt, ...) \
+ printk(KERN_DEBUG "%s: " fmt, __func__, ##__VA_ARGS__)
+#define iop_pr_cont(fmt, ...) \
+ printk(KERN_CONT fmt, ##__VA_ARGS__)
+#else
+#define iop_pr_debug(fmt, ...) \
+ no_printk(KERN_DEBUG "%s: " fmt, __func__, ##__VA_ARGS__)
+#define iop_pr_cont(fmt, ...) \
+ no_printk(KERN_CONT fmt, ##__VA_ARGS__)
+#endif
/* Non-zero if the IOPs are present */
@@ -200,7 +210,7 @@
return retval;
}
-static struct iop_msg *iop_alloc_msg(void)
+static struct iop_msg *iop_get_unused_msg(void)
{
int i;
unsigned long flags;
@@ -219,11 +229,6 @@
return NULL;
}
-static void iop_free_msg(struct iop_msg *msg)
-{
- msg->status = IOP_MSGSTATUS_UNUSED;
-}
-
/*
* This is called by the startup code before anything else. Its purpose
* is to find and initialize the IOPs early in the boot sequence, so that
@@ -268,10 +273,10 @@
int i;
if (iop_scc_present) {
- printk("IOP: detected SCC IOP at %p\n", iop_base[IOP_NUM_SCC]);
+ pr_info("IOP: detected SCC IOP at %p\n", iop_base[IOP_NUM_SCC]);
}
if (iop_ism_present) {
- printk("IOP: detected ISM IOP at %p\n", iop_base[IOP_NUM_ISM]);
+ pr_info("IOP: detected ISM IOP at %p\n", iop_base[IOP_NUM_ISM]);
iop_start(iop_base[IOP_NUM_ISM]);
iop_alive(iop_base[IOP_NUM_ISM]); /* clears the alive flag */
}
@@ -310,9 +315,9 @@
pr_err("Couldn't register ISM IOP interrupt\n");
}
if (!iop_alive(iop_base[IOP_NUM_ISM])) {
- printk("IOP: oh my god, they killed the ISM IOP!\n");
+ pr_warn("IOP: oh my god, they killed the ISM IOP!\n");
} else {
- printk("IOP: the ISM IOP seems to be alive.\n");
+ pr_warn("IOP: the ISM IOP seems to be alive.\n");
}
}
}
@@ -349,9 +354,8 @@
int chan = msg->channel;
int i,offset;
-#ifdef DEBUG_IOP
- printk("iop_complete(%p): iop %d chan %d\n", msg, msg->iop_num, msg->channel);
-#endif
+ iop_pr_debug("msg %p iop_num %d channel %d\n", msg, msg->iop_num,
+ msg->channel);
offset = IOP_ADDR_RECV_MSG + (msg->channel * IOP_MSG_LEN);
@@ -363,7 +367,7 @@
IOP_ADDR_RECV_STATE + chan, IOP_MSG_COMPLETE);
iop_interrupt(iop_base[msg->iop_num]);
- iop_free_msg(msg);
+ msg->status = IOP_MSGSTATUS_UNUSED;
}
/*
@@ -394,12 +398,10 @@
static void iop_handle_send(uint iop_num, uint chan)
{
volatile struct mac_iop *iop = iop_base[iop_num];
- struct iop_msg *msg,*msg2;
+ struct iop_msg *msg;
int i,offset;
-#ifdef DEBUG_IOP
- printk("iop_handle_send: iop %d channel %d\n", iop_num, chan);
-#endif
+ iop_pr_debug("iop_num %d chan %d\n", iop_num, chan);
iop_writeb(iop, IOP_ADDR_SEND_STATE + chan, IOP_MSG_IDLE);
@@ -411,10 +413,8 @@
msg->reply[i] = iop_readb(iop, offset);
}
if (msg->handler) (*msg->handler)(msg);
- msg2 = msg;
+ msg->status = IOP_MSGSTATUS_UNUSED;
msg = msg->next;
- iop_free_msg(msg2);
-
iop_send_queue[iop_num][chan] = msg;
if (msg) iop_do_send(msg);
}
@@ -430,11 +430,9 @@
int i,offset;
struct iop_msg *msg;
-#ifdef DEBUG_IOP
- printk("iop_handle_recv: iop %d channel %d\n", iop_num, chan);
-#endif
+ iop_pr_debug("iop_num %d chan %d\n", iop_num, chan);
- msg = iop_alloc_msg();
+ msg = iop_get_unused_msg();
msg->iop_num = iop_num;
msg->channel = chan;
msg->status = IOP_MSGSTATUS_UNSOL;
@@ -454,14 +452,9 @@
if (msg->handler) {
(*msg->handler)(msg);
} else {
-#ifdef DEBUG_IOP
- printk("iop_handle_recv: unclaimed message on iop %d channel %d\n", iop_num, chan);
- printk("iop_handle_recv:");
- for (i = 0 ; i < IOP_MSG_LEN ; i++) {
- printk(" %02X", (uint) msg->message[i]);
- }
- printk("\n");
-#endif
+ iop_pr_debug("unclaimed message on iop_num %d chan %d\n",
+ iop_num, chan);
+ iop_pr_debug("%*ph\n", IOP_MSG_LEN, msg->message);
iop_complete_message(msg);
}
}
@@ -484,7 +477,7 @@
if (chan >= NUM_IOP_CHAN) return -EINVAL;
if (msg_len > IOP_MSG_LEN) return -EINVAL;
- msg = iop_alloc_msg();
+ msg = iop_get_unused_msg();
if (!msg) return -ENOMEM;
msg->next = NULL;
@@ -574,50 +567,34 @@
volatile struct mac_iop *iop = iop_base[iop_num];
int i,state;
-#ifdef DEBUG_IOP
- printk("iop_ism_irq: status = %02X\n", (uint) iop->status_ctrl);
-#endif
+ iop_pr_debug("status %02X\n", iop->status_ctrl);
/* INT0 indicates a state change on an outgoing message channel */
if (iop->status_ctrl & IOP_INT0) {
iop->status_ctrl = IOP_INT0 | IOP_RUN | IOP_AUTOINC;
-#ifdef DEBUG_IOP
- printk("iop_ism_irq: new status = %02X, send states",
- (uint) iop->status_ctrl);
-#endif
+ iop_pr_debug("new status %02X, send states", iop->status_ctrl);
for (i = 0 ; i < NUM_IOP_CHAN ; i++) {
state = iop_readb(iop, IOP_ADDR_SEND_STATE + i);
-#ifdef DEBUG_IOP
- printk(" %02X", state);
-#endif
+ iop_pr_cont(" %02X", state);
if (state == IOP_MSG_COMPLETE) {
iop_handle_send(iop_num, i);
}
}
-#ifdef DEBUG_IOP
- printk("\n");
-#endif
+ iop_pr_cont("\n");
}
if (iop->status_ctrl & IOP_INT1) { /* INT1 for incoming msgs */
iop->status_ctrl = IOP_INT1 | IOP_RUN | IOP_AUTOINC;
-#ifdef DEBUG_IOP
- printk("iop_ism_irq: new status = %02X, recv states",
- (uint) iop->status_ctrl);
-#endif
+ iop_pr_debug("new status %02X, recv states", iop->status_ctrl);
for (i = 0 ; i < NUM_IOP_CHAN ; i++) {
state = iop_readb(iop, IOP_ADDR_RECV_STATE + i);
-#ifdef DEBUG_IOP
- printk(" %02X", state);
-#endif
+ iop_pr_cont(" %02X", state);
if (state == IOP_MSG_NEW) {
iop_handle_recv(iop_num, i);
}
}
-#ifdef DEBUG_IOP
- printk("\n");
-#endif
+ iop_pr_cont("\n");
}
return IRQ_HANDLED;
}
diff --git a/arch/m68k/mac/misc.c b/arch/m68k/mac/misc.c
index 5b01704..8aa8792 100644
--- a/arch/m68k/mac/misc.c
+++ b/arch/m68k/mac/misc.c
@@ -281,8 +281,7 @@
last_result.idata = result.idata;
}
- pr_err("via_read_time: failed to read a stable value; "
- "got 0x%08lx then 0x%08lx\n",
+ pr_err("via_read_time: failed to read a stable value; got 0x%08lx then 0x%08lx\n",
last_result.idata, result.idata);
return 0;
@@ -465,7 +464,7 @@
#endif
}
local_irq_enable();
- printk("It is now safe to turn off your Macintosh.\n");
+ pr_crit("It is now safe to turn off your Macintosh.\n");
while(1);
}
@@ -556,7 +555,7 @@
/* should never get here */
local_irq_enable();
- printk ("Restart failed. Please restart manually.\n");
+ pr_crit("Restart failed. Please restart manually.\n");
while(1);
}
@@ -661,17 +660,13 @@
unmktime(now, 0,
&t->tm_year, &t->tm_mon, &t->tm_mday,
&t->tm_hour, &t->tm_min, &t->tm_sec);
-#if 0
- printk("mac_hwclk: read %04d-%02d-%-2d %02d:%02d:%02d\n",
- t->tm_year + 1900, t->tm_mon + 1, t->tm_mday,
- t->tm_hour, t->tm_min, t->tm_sec);
-#endif
+ pr_debug("%s: read %04d-%02d-%-2d %02d:%02d:%02d\n",
+ __func__, t->tm_year + 1900, t->tm_mon + 1, t->tm_mday,
+ t->tm_hour, t->tm_min, t->tm_sec);
} else { /* write */
-#if 0
- printk("mac_hwclk: tried to write %04d-%02d-%-2d %02d:%02d:%02d\n",
- t->tm_year + 1900, t->tm_mon + 1, t->tm_mday,
- t->tm_hour, t->tm_min, t->tm_sec);
-#endif
+ pr_debug("%s: tried to write %04d-%02d-%-2d %02d:%02d:%02d\n",
+ __func__, t->tm_year + 1900, t->tm_mon + 1, t->tm_mday,
+ t->tm_hour, t->tm_min, t->tm_sec);
now = mktime(t->tm_year + 1900, t->tm_mon + 1, t->tm_mday,
t->tm_hour, t->tm_min, t->tm_sec);
diff --git a/arch/m68k/mm/fault.c b/arch/m68k/mm/fault.c
index bd66a0b..2795e4c 100644
--- a/arch/m68k/mm/fault.c
+++ b/arch/m68k/mm/fault.c
@@ -32,7 +32,7 @@
force_sig_info(siginfo.si_signo,
&siginfo, current);
} else {
- if (handle_kernel_fault(regs))
+ if (fixup_exception(regs))
return -1;
//if (siginfo.si_signo == SIGBUS)
diff --git a/arch/metag/include/asm/Kbuild b/arch/metag/include/asm/Kbuild
index f9b9df5..8f94055 100644
--- a/arch/metag/include/asm/Kbuild
+++ b/arch/metag/include/asm/Kbuild
@@ -8,6 +8,7 @@
generic-y += emergency-restart.h
generic-y += errno.h
generic-y += exec.h
+generic-y += extable.h
generic-y += fb.h
generic-y += fcntl.h
generic-y += futex.h
diff --git a/arch/metag/include/asm/uaccess.h b/arch/metag/include/asm/uaccess.h
index 07238b3..5ebc285 100644
--- a/arch/metag/include/asm/uaccess.h
+++ b/arch/metag/include/asm/uaccess.h
@@ -4,10 +4,6 @@
/*
* User space memory access functions
*/
-#include <linux/sched.h>
-
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
/*
* The fs value determines whether argument validity checking should be
@@ -28,7 +24,7 @@
#define segment_eq(a, b) ((a).seg == (b).seg)
-#define __kernel_ok (segment_eq(get_fs(), KERNEL_DS))
+#define __kernel_ok (uaccess_kernel())
/*
* Explicitly allow NULL pointers here. Parts of the kernel such
* as readv/writev use access_ok to validate pointers, but want
@@ -51,28 +47,7 @@
#define access_ok(type, addr, size) __access_ok((unsigned long)(addr), \
(unsigned long)(size))
-static inline int verify_area(int type, const void *addr, unsigned long size)
-{
- return access_ok(type, addr, size) ? 0 : -EFAULT;
-}
-
-/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-struct exception_table_entry {
- unsigned long insn, fixup;
-};
-
-extern int fixup_exception(struct pt_regs *regs);
+#include <asm/extable.h>
/*
* These are the main single-value transfer routines. They automatically
@@ -199,35 +174,8 @@
extern unsigned long raw_copy_from_user(void *to, const void __user *from,
unsigned long n);
-
-static inline unsigned long
-copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- unsigned long res = n;
- if (likely(access_ok(VERIFY_READ, from, n)))
- res = raw_copy_from_user(to, from, n);
- if (unlikely(res))
- memset(to + (n - res), 0, res);
- return res;
-}
-
-#define __copy_from_user(to, from, n) raw_copy_from_user(to, from, n)
-#define __copy_from_user_inatomic __copy_from_user
-
-extern unsigned long __must_check __copy_user(void __user *to,
- const void *from,
- unsigned long n);
-
-static inline unsigned long copy_to_user(void __user *to, const void *from,
- unsigned long n)
-{
- if (access_ok(VERIFY_WRITE, to, n))
- return __copy_user(to, from, n);
- return n;
-}
-
-#define __copy_to_user(to, from, n) __copy_user(to, from, n)
-#define __copy_to_user_inatomic __copy_to_user
+extern unsigned long raw_copy_to_user(void __user *to, const void *from,
+ unsigned long n);
/*
* Zero Userspace
diff --git a/arch/metag/kernel/stacktrace.c b/arch/metag/kernel/stacktrace.c
index 91ffc4b..09d67b7 100644
--- a/arch/metag/kernel/stacktrace.c
+++ b/arch/metag/kernel/stacktrace.c
@@ -31,8 +31,6 @@
}
#endif
-#define ALIGN_DOWN(addr, size) ((addr)&(~((size)-1)))
-
/*
* Unwind the current stack frame and store the new register values in the
* structure passed as argument. Unwinding is equivalent to a function return,
diff --git a/arch/metag/lib/usercopy.c b/arch/metag/lib/usercopy.c
index 2792fc6..e8a4ea8 100644
--- a/arch/metag/lib/usercopy.c
+++ b/arch/metag/lib/usercopy.c
@@ -548,8 +548,8 @@
"SUB %1, %1, D0Ar2\n" \
"SUB %3, %3, D1Ar1\n")
-unsigned long __copy_user(void __user *pdst, const void *psrc,
- unsigned long n)
+unsigned long raw_copy_to_user(void __user *pdst, const void *psrc,
+ unsigned long n)
{
register char __user *dst asm ("A0.2") = pdst;
register const char *src asm ("A1.2") = psrc;
@@ -654,7 +654,7 @@
*/
return retn;
}
-EXPORT_SYMBOL(__copy_user);
+EXPORT_SYMBOL(raw_copy_to_user);
#define __asm_copy_from_user_1(to, from, ret) \
__asm_copy_user_cont(to, from, ret, \
diff --git a/arch/microblaze/include/asm/Kbuild b/arch/microblaze/include/asm/Kbuild
index 1732ec1..56830ff 100644
--- a/arch/microblaze/include/asm/Kbuild
+++ b/arch/microblaze/include/asm/Kbuild
@@ -3,6 +3,7 @@
generic-y += clkdev.h
generic-y += device.h
generic-y += exec.h
+generic-y += extable.h
generic-y += irq_work.h
generic-y += mcs_spinlock.h
generic-y += mm-arch-hooks.h
diff --git a/arch/microblaze/include/asm/uaccess.h b/arch/microblaze/include/asm/uaccess.h
index 253a67e..38f2c9c 100644
--- a/arch/microblaze/include/asm/uaccess.h
+++ b/arch/microblaze/include/asm/uaccess.h
@@ -11,22 +11,15 @@
#ifndef _ASM_MICROBLAZE_UACCESS_H
#define _ASM_MICROBLAZE_UACCESS_H
-#ifdef __KERNEL__
-#ifndef __ASSEMBLY__
-
#include <linux/kernel.h>
-#include <linux/errno.h>
-#include <linux/sched.h> /* RLIMIT_FSIZE */
#include <linux/mm.h>
#include <asm/mmu.h>
#include <asm/page.h>
#include <asm/pgtable.h>
+#include <asm/extable.h>
#include <linux/string.h>
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
/*
* On Microblaze the fs value is actually the top of the corresponding
* address space.
@@ -55,22 +48,6 @@
# define segment_eq(a, b) ((a).seg == (b).seg)
-/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-struct exception_table_entry {
- unsigned long insn, fixup;
-};
-
#ifndef CONFIG_MMU
/* Check against bounds of physical memory */
@@ -359,39 +336,19 @@
__gu_err; \
})
-
-/* copy_to_from_user */
-#define __copy_from_user(to, from, n) \
- __copy_tofrom_user((__force void __user *)(to), \
- (void __user *)(from), (n))
-#define __copy_from_user_inatomic(to, from, n) \
- __copy_from_user((to), (from), (n))
-
-static inline long copy_from_user(void *to,
- const void __user *from, unsigned long n)
+static inline unsigned long
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
- unsigned long res = n;
- might_fault();
- if (likely(access_ok(VERIFY_READ, from, n)))
- res = __copy_from_user(to, from, n);
- if (unlikely(res))
- memset(to + (n - res), 0, res);
- return res;
+ return __copy_tofrom_user((__force void __user *)to, from, n);
}
-#define __copy_to_user(to, from, n) \
- __copy_tofrom_user((void __user *)(to), \
- (__force const void __user *)(from), (n))
-#define __copy_to_user_inatomic(to, from, n) __copy_to_user((to), (from), (n))
-
-static inline long copy_to_user(void __user *to,
- const void *from, unsigned long n)
+static inline unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
{
- might_fault();
- if (access_ok(VERIFY_WRITE, to, n))
- return __copy_to_user(to, from, n);
- return n;
+ return __copy_tofrom_user(to, (__force const void __user *)from, n);
}
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
/*
* Copy a null terminated string from userspace.
@@ -422,7 +379,4 @@
return __strnlen_user(src, n);
}
-#endif /* __ASSEMBLY__ */
-#endif /* __KERNEL__ */
-
#endif /* _ASM_MICROBLAZE_UACCESS_H */
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index e0bb576..d6d545a 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -68,7 +68,6 @@
select HANDLE_DOMAIN_IRQ
select HAVE_EXIT_THREAD
select HAVE_REGS_AND_STACK_ACCESS_API
- select HAVE_ARCH_HARDENED_USERCOPY
menu "Machine selection"
diff --git a/arch/mips/Makefile b/arch/mips/Makefile
index 8ef9c02..02a1787 100644
--- a/arch/mips/Makefile
+++ b/arch/mips/Makefile
@@ -489,7 +489,7 @@
$(Q)$(CONFIG_SHELL) $(srctree)/scripts/kconfig/merge_config.sh \
-m -O $(objtree) $(srctree)/arch/$(ARCH)/configs/generic_defconfig $^ \
$(foreach board,$(BOARDS),$(generic_config_dir)/board-$(board).config)
- $(Q)$(MAKE) olddefconfig
+ $(Q)$(MAKE) -f $(srctree)/Makefile olddefconfig
#
# Prevent generic merge_config rules attempting to merge single fragments
@@ -503,8 +503,8 @@
#
.PHONY: sead3_defconfig
sead3_defconfig:
- $(Q)$(MAKE) 32r2el_defconfig BOARDS=sead-3
+ $(Q)$(MAKE) -f $(srctree)/Makefile 32r2el_defconfig BOARDS=sead-3
.PHONY: sead3micro_defconfig
sead3micro_defconfig:
- $(Q)$(MAKE) micro32r2el_defconfig BOARDS=sead-3
+ $(Q)$(MAKE) -f $(srctree)/Makefile micro32r2el_defconfig BOARDS=sead-3
diff --git a/arch/mips/alchemy/common/time.c b/arch/mips/alchemy/common/time.c
index e1bec5a..32d1333 100644
--- a/arch/mips/alchemy/common/time.c
+++ b/arch/mips/alchemy/common/time.c
@@ -138,7 +138,9 @@
cd->shift = 32;
cd->mult = div_sc(32768, NSEC_PER_SEC, cd->shift);
cd->max_delta_ns = clockevent_delta2ns(0xffffffff, cd);
- cd->min_delta_ns = clockevent_delta2ns(9, cd); /* ~0.28ms */
+ cd->max_delta_ticks = 0xffffffff;
+ cd->min_delta_ns = clockevent_delta2ns(9, cd);
+ cd->min_delta_ticks = 9; /* ~0.28ms */
clockevents_register_device(cd);
setup_irq(m2int, &au1x_rtcmatch2_irqaction);
diff --git a/arch/mips/cavium-octeon/octeon-memcpy.S b/arch/mips/cavium-octeon/octeon-memcpy.S
index cfd97f6..0a7c983 100644
--- a/arch/mips/cavium-octeon/octeon-memcpy.S
+++ b/arch/mips/cavium-octeon/octeon-memcpy.S
@@ -140,15 +140,6 @@
.set noat
/*
- * t7 is used as a flag to note inatomic mode.
- */
-LEAF(__copy_user_inatomic)
-EXPORT_SYMBOL(__copy_user_inatomic)
- b __copy_user_common
- li t7, 1
- END(__copy_user_inatomic)
-
-/*
* A combined memcpy/__copy_user
* __copy_user sets len to 0 for success; else to an upper bound of
* the number of uncopied bytes.
@@ -161,8 +152,6 @@
__memcpy:
FEXPORT(__copy_user)
EXPORT_SYMBOL(__copy_user)
- li t7, 0 /* not inatomic */
-__copy_user_common:
/*
* Note: dst & src may be unaligned, len may be 0
* Temps
@@ -414,25 +403,7 @@
LOAD t0, TI_TASK($28)
LOAD t0, THREAD_BUADDR(t0) # t0 is just past last good address
SUB len, AT, t0 # len number of uncopied bytes
- bnez t7, 2f /* Skip the zeroing out part if inatomic */
- /*
- * Here's where we rely on src and dst being incremented in tandem,
- * See (3) above.
- * dst += (fault addr - src) to put dst at first byte to clear
- */
- ADD dst, t0 # compute start address in a1
- SUB dst, src
- /*
- * Clear len bytes starting at dst. Can't call __bzero because it
- * might modify len. An inefficient loop for these rare times...
- */
- beqz len, done
- SUB src, len, 1
-1: sb zero, 0(dst)
- ADD dst, dst, 1
- bnez src, 1b
- SUB src, src, 1
-2: jr ra
+ jr ra
nop
diff --git a/arch/mips/configs/cavium_octeon_defconfig b/arch/mips/configs/cavium_octeon_defconfig
index 31e3c4d..d4fda41 100644
--- a/arch/mips/configs/cavium_octeon_defconfig
+++ b/arch/mips/configs/cavium_octeon_defconfig
@@ -127,6 +127,11 @@
CONFIG_USB_EHCI_HCD_PLATFORM=m
CONFIG_USB_OHCI_HCD=m
CONFIG_USB_OHCI_HCD_PLATFORM=m
+CONFIG_MMC=y
+# CONFIG_PWRSEQ_EMMC is not set
+# CONFIG_PWRSEQ_SIMPLE is not set
+# CONFIG_MMC_BLOCK_BOUNCE is not set
+CONFIG_MMC_CAVIUM_OCTEON=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_DS1307=y
CONFIG_STAGING=y
diff --git a/arch/mips/include/asm/asm-prototypes.h b/arch/mips/include/asm/asm-prototypes.h
index a160cf6..6e28971 100644
--- a/arch/mips/include/asm/asm-prototypes.h
+++ b/arch/mips/include/asm/asm-prototypes.h
@@ -3,3 +3,4 @@
#include <asm/fpu.h>
#include <asm-generic/asm-prototypes.h>
#include <asm/uaccess.h>
+#include <asm/ftrace.h>
diff --git a/arch/mips/include/asm/checksum.h b/arch/mips/include/asm/checksum.h
index c8b574f..77cad23 100644
--- a/arch/mips/include/asm/checksum.h
+++ b/arch/mips/include/asm/checksum.h
@@ -50,7 +50,7 @@
__wsum sum, int *err_ptr)
{
might_fault();
- if (segment_eq(get_fs(), get_ds()))
+ if (uaccess_kernel())
return __csum_partial_copy_kernel((__force void *)src, dst,
len, sum, err_ptr);
else
@@ -82,7 +82,7 @@
{
might_fault();
if (access_ok(VERIFY_WRITE, dst, len)) {
- if (segment_eq(get_fs(), get_ds()))
+ if (uaccess_kernel())
return __csum_partial_copy_kernel(src,
(__force void *)dst,
len, sum, err_ptr);
diff --git a/arch/mips/include/asm/r4kcache.h b/arch/mips/include/asm/r4kcache.h
index 55fd94e..7f12d7e 100644
--- a/arch/mips/include/asm/r4kcache.h
+++ b/arch/mips/include/asm/r4kcache.h
@@ -20,7 +20,7 @@
#include <asm/cpu-features.h>
#include <asm/cpu-type.h>
#include <asm/mipsmtregs.h>
-#include <linux/uaccess.h> /* for segment_eq() */
+#include <linux/uaccess.h> /* for uaccess_kernel() */
extern void (*r4k_blast_dcache)(void);
extern void (*r4k_blast_icache)(void);
@@ -714,7 +714,7 @@
\
__##pfx##flush_prologue \
\
- if (segment_eq(get_fs(), USER_DS)) { \
+ if (!uaccess_kernel()) { \
while (1) { \
protected_cachee_op(hitop, addr); \
if (addr == aend) \
diff --git a/arch/mips/include/asm/uaccess.h b/arch/mips/include/asm/uaccess.h
index 5347cfe..99e629a 100644
--- a/arch/mips/include/asm/uaccess.h
+++ b/arch/mips/include/asm/uaccess.h
@@ -12,8 +12,6 @@
#define _ASM_UACCESS_H
#include <linux/kernel.h>
-#include <linux/errno.h>
-#include <linux/thread_info.h>
#include <linux/string.h>
#include <asm/asm-eva.h>
#include <asm/extable.h>
@@ -71,9 +69,6 @@
#define USER_DS ((mm_segment_t) { __UA_LIMIT })
#endif
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
#define get_ds() (KERNEL_DS)
#define get_fs() (current_thread_info()->addr_limit)
#define set_fs(x) (current_thread_info()->addr_limit = (x))
@@ -93,7 +88,7 @@
if (!IS_ENABLED(CONFIG_EVA))
return false;
- return segment_eq(get_fs(), get_ds());
+ return uaccess_kernel();
}
/*
@@ -133,23 +128,14 @@
* this function, memory access functions may still return -EFAULT.
*/
-#define __access_mask get_fs().seg
-
-#define __access_ok(addr, size, mask) \
-({ \
- unsigned long __addr = (unsigned long) (addr); \
- unsigned long __size = size; \
- unsigned long __mask = mask; \
- unsigned long __ok; \
- \
- __chk_user_ptr(addr); \
- __ok = (signed long)(__mask & (__addr | (__addr + __size) | \
- __ua_size(__size))); \
- __ok == 0; \
-})
+static inline int __access_ok(const void __user *p, unsigned long size)
+{
+ unsigned long addr = (unsigned long)p;
+ return (get_fs().seg & (addr | (addr + size) | __ua_size(size))) == 0;
+}
#define access_ok(type, addr, size) \
- likely(__access_ok((addr), (size), __access_mask))
+ likely(__access_ok((addr), (size)))
/*
* put_user: - Write a simple value into user space.
@@ -811,8 +797,30 @@
extern size_t __copy_user(void *__to, const void *__from, size_t __n);
-#ifndef CONFIG_EVA
-#define __invoke_copy_to_user(to, from, n) \
+#define __invoke_copy_from(func, to, from, n) \
+({ \
+ register void *__cu_to_r __asm__("$4"); \
+ register const void __user *__cu_from_r __asm__("$5"); \
+ register long __cu_len_r __asm__("$6"); \
+ \
+ __cu_to_r = (to); \
+ __cu_from_r = (from); \
+ __cu_len_r = (n); \
+ __asm__ __volatile__( \
+ ".set\tnoreorder\n\t" \
+ __MODULE_JAL(func) \
+ ".set\tnoat\n\t" \
+ __UA_ADDU "\t$1, %1, %2\n\t" \
+ ".set\tat\n\t" \
+ ".set\treorder" \
+ : "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r) \
+ : \
+ : "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31", \
+ DADDI_SCRATCH, "memory"); \
+ __cu_len_r; \
+})
+
+#define __invoke_copy_to(func, to, from, n) \
({ \
register void __user *__cu_to_r __asm__("$4"); \
register const void *__cu_from_r __asm__("$5"); \
@@ -822,161 +830,7 @@
__cu_from_r = (from); \
__cu_len_r = (n); \
__asm__ __volatile__( \
- __MODULE_JAL(__copy_user) \
- : "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r) \
- : \
- : "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31", \
- DADDI_SCRATCH, "memory"); \
- __cu_len_r; \
-})
-
-#define __invoke_copy_to_kernel(to, from, n) \
- __invoke_copy_to_user(to, from, n)
-
-#endif
-
-/*
- * __copy_to_user: - Copy a block of data into user space, with less checking.
- * @to: Destination address, in user space.
- * @from: Source address, in kernel space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from kernel space to user space. Caller must check
- * the specified block with access_ok() before calling this function.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- */
-#define __copy_to_user(to, from, n) \
-({ \
- void __user *__cu_to; \
- const void *__cu_from; \
- long __cu_len; \
- \
- __cu_to = (to); \
- __cu_from = (from); \
- __cu_len = (n); \
- \
- check_object_size(__cu_from, __cu_len, true); \
- might_fault(); \
- \
- if (eva_kernel_access()) \
- __cu_len = __invoke_copy_to_kernel(__cu_to, __cu_from, \
- __cu_len); \
- else \
- __cu_len = __invoke_copy_to_user(__cu_to, __cu_from, \
- __cu_len); \
- __cu_len; \
-})
-
-extern size_t __copy_user_inatomic(void *__to, const void *__from, size_t __n);
-
-#define __copy_to_user_inatomic(to, from, n) \
-({ \
- void __user *__cu_to; \
- const void *__cu_from; \
- long __cu_len; \
- \
- __cu_to = (to); \
- __cu_from = (from); \
- __cu_len = (n); \
- \
- check_object_size(__cu_from, __cu_len, true); \
- \
- if (eva_kernel_access()) \
- __cu_len = __invoke_copy_to_kernel(__cu_to, __cu_from, \
- __cu_len); \
- else \
- __cu_len = __invoke_copy_to_user(__cu_to, __cu_from, \
- __cu_len); \
- __cu_len; \
-})
-
-#define __copy_from_user_inatomic(to, from, n) \
-({ \
- void *__cu_to; \
- const void __user *__cu_from; \
- long __cu_len; \
- \
- __cu_to = (to); \
- __cu_from = (from); \
- __cu_len = (n); \
- \
- check_object_size(__cu_to, __cu_len, false); \
- \
- if (eva_kernel_access()) \
- __cu_len = __invoke_copy_from_kernel_inatomic(__cu_to, \
- __cu_from,\
- __cu_len);\
- else \
- __cu_len = __invoke_copy_from_user_inatomic(__cu_to, \
- __cu_from, \
- __cu_len); \
- __cu_len; \
-})
-
-/*
- * copy_to_user: - Copy a block of data into user space.
- * @to: Destination address, in user space.
- * @from: Source address, in kernel space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from kernel space to user space.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- */
-#define copy_to_user(to, from, n) \
-({ \
- void __user *__cu_to; \
- const void *__cu_from; \
- long __cu_len; \
- \
- __cu_to = (to); \
- __cu_from = (from); \
- __cu_len = (n); \
- \
- check_object_size(__cu_from, __cu_len, true); \
- \
- if (eva_kernel_access()) { \
- __cu_len = __invoke_copy_to_kernel(__cu_to, \
- __cu_from, \
- __cu_len); \
- } else { \
- if (access_ok(VERIFY_WRITE, __cu_to, __cu_len)) { \
- might_fault(); \
- __cu_len = __invoke_copy_to_user(__cu_to, \
- __cu_from, \
- __cu_len); \
- } \
- } \
- __cu_len; \
-})
-
-#ifndef CONFIG_EVA
-
-#define __invoke_copy_from_user(to, from, n) \
-({ \
- register void *__cu_to_r __asm__("$4"); \
- register const void __user *__cu_from_r __asm__("$5"); \
- register long __cu_len_r __asm__("$6"); \
- \
- __cu_to_r = (to); \
- __cu_from_r = (from); \
- __cu_len_r = (n); \
- __asm__ __volatile__( \
- ".set\tnoreorder\n\t" \
- __MODULE_JAL(__copy_user) \
- ".set\tnoat\n\t" \
- __UA_ADDU "\t$1, %1, %2\n\t" \
- ".set\tat\n\t" \
- ".set\treorder" \
+ __MODULE_JAL(func) \
: "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r) \
: \
: "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31", \
@@ -985,261 +839,78 @@
})
#define __invoke_copy_from_kernel(to, from, n) \
- __invoke_copy_from_user(to, from, n)
+ __invoke_copy_from(__copy_user, to, from, n)
-/* For userland <-> userland operations */
-#define ___invoke_copy_in_user(to, from, n) \
- __invoke_copy_from_user(to, from, n)
+#define __invoke_copy_to_kernel(to, from, n) \
+ __invoke_copy_to(__copy_user, to, from, n)
-/* For kernel <-> kernel operations */
#define ___invoke_copy_in_kernel(to, from, n) \
- __invoke_copy_from_user(to, from, n)
+ __invoke_copy_from(__copy_user, to, from, n)
-#define __invoke_copy_from_user_inatomic(to, from, n) \
-({ \
- register void *__cu_to_r __asm__("$4"); \
- register const void __user *__cu_from_r __asm__("$5"); \
- register long __cu_len_r __asm__("$6"); \
- \
- __cu_to_r = (to); \
- __cu_from_r = (from); \
- __cu_len_r = (n); \
- __asm__ __volatile__( \
- ".set\tnoreorder\n\t" \
- __MODULE_JAL(__copy_user_inatomic) \
- ".set\tnoat\n\t" \
- __UA_ADDU "\t$1, %1, %2\n\t" \
- ".set\tat\n\t" \
- ".set\treorder" \
- : "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r) \
- : \
- : "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31", \
- DADDI_SCRATCH, "memory"); \
- __cu_len_r; \
-})
+#ifndef CONFIG_EVA
+#define __invoke_copy_from_user(to, from, n) \
+ __invoke_copy_from(__copy_user, to, from, n)
-#define __invoke_copy_from_kernel_inatomic(to, from, n) \
- __invoke_copy_from_user_inatomic(to, from, n) \
+#define __invoke_copy_to_user(to, from, n) \
+ __invoke_copy_to(__copy_user, to, from, n)
+
+#define ___invoke_copy_in_user(to, from, n) \
+ __invoke_copy_from(__copy_user, to, from, n)
#else
/* EVA specific functions */
-extern size_t __copy_user_inatomic_eva(void *__to, const void *__from,
- size_t __n);
extern size_t __copy_from_user_eva(void *__to, const void *__from,
size_t __n);
extern size_t __copy_to_user_eva(void *__to, const void *__from,
size_t __n);
extern size_t __copy_in_user_eva(void *__to, const void *__from, size_t __n);
-#define __invoke_copy_from_user_eva_generic(to, from, n, func_ptr) \
-({ \
- register void *__cu_to_r __asm__("$4"); \
- register const void __user *__cu_from_r __asm__("$5"); \
- register long __cu_len_r __asm__("$6"); \
- \
- __cu_to_r = (to); \
- __cu_from_r = (from); \
- __cu_len_r = (n); \
- __asm__ __volatile__( \
- ".set\tnoreorder\n\t" \
- __MODULE_JAL(func_ptr) \
- ".set\tnoat\n\t" \
- __UA_ADDU "\t$1, %1, %2\n\t" \
- ".set\tat\n\t" \
- ".set\treorder" \
- : "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r) \
- : \
- : "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31", \
- DADDI_SCRATCH, "memory"); \
- __cu_len_r; \
-})
-
-#define __invoke_copy_to_user_eva_generic(to, from, n, func_ptr) \
-({ \
- register void *__cu_to_r __asm__("$4"); \
- register const void __user *__cu_from_r __asm__("$5"); \
- register long __cu_len_r __asm__("$6"); \
- \
- __cu_to_r = (to); \
- __cu_from_r = (from); \
- __cu_len_r = (n); \
- __asm__ __volatile__( \
- __MODULE_JAL(func_ptr) \
- : "+r" (__cu_to_r), "+r" (__cu_from_r), "+r" (__cu_len_r) \
- : \
- : "$8", "$9", "$10", "$11", "$12", "$14", "$15", "$24", "$31", \
- DADDI_SCRATCH, "memory"); \
- __cu_len_r; \
-})
-
/*
* Source or destination address is in userland. We need to go through
* the TLB
*/
#define __invoke_copy_from_user(to, from, n) \
- __invoke_copy_from_user_eva_generic(to, from, n, __copy_from_user_eva)
-
-#define __invoke_copy_from_user_inatomic(to, from, n) \
- __invoke_copy_from_user_eva_generic(to, from, n, \
- __copy_user_inatomic_eva)
+ __invoke_copy_from(__copy_from_user_eva, to, from, n)
#define __invoke_copy_to_user(to, from, n) \
- __invoke_copy_to_user_eva_generic(to, from, n, __copy_to_user_eva)
+ __invoke_copy_to(__copy_to_user_eva, to, from, n)
#define ___invoke_copy_in_user(to, from, n) \
- __invoke_copy_from_user_eva_generic(to, from, n, __copy_in_user_eva)
-
-/*
- * Source or destination address in the kernel. We are not going through
- * the TLB
- */
-#define __invoke_copy_from_kernel(to, from, n) \
- __invoke_copy_from_user_eva_generic(to, from, n, __copy_user)
-
-#define __invoke_copy_from_kernel_inatomic(to, from, n) \
- __invoke_copy_from_user_eva_generic(to, from, n, __copy_user_inatomic)
-
-#define __invoke_copy_to_kernel(to, from, n) \
- __invoke_copy_to_user_eva_generic(to, from, n, __copy_user)
-
-#define ___invoke_copy_in_kernel(to, from, n) \
- __invoke_copy_from_user_eva_generic(to, from, n, __copy_user)
+ __invoke_copy_from(__copy_in_user_eva, to, from, n)
#endif /* CONFIG_EVA */
-/*
- * __copy_from_user: - Copy a block of data from user space, with less checking.
- * @to: Destination address, in kernel space.
- * @from: Source address, in user space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from user space to kernel space. Caller must check
- * the specified block with access_ok() before calling this function.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- *
- * If some data could not be copied, this function will pad the copied
- * data to the requested size using zero bytes.
- */
-#define __copy_from_user(to, from, n) \
-({ \
- void *__cu_to; \
- const void __user *__cu_from; \
- long __cu_len; \
- \
- __cu_to = (to); \
- __cu_from = (from); \
- __cu_len = (n); \
- \
- check_object_size(__cu_to, __cu_len, false); \
- \
- if (eva_kernel_access()) { \
- __cu_len = __invoke_copy_from_kernel(__cu_to, \
- __cu_from, \
- __cu_len); \
- } else { \
- might_fault(); \
- __cu_len = __invoke_copy_from_user(__cu_to, __cu_from, \
- __cu_len); \
- } \
- __cu_len; \
-})
+static inline unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
+{
+ if (eva_kernel_access())
+ return __invoke_copy_to_kernel(to, from, n);
+ else
+ return __invoke_copy_to_user(to, from, n);
+}
-/*
- * copy_from_user: - Copy a block of data from user space.
- * @to: Destination address, in kernel space.
- * @from: Source address, in user space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from user space to kernel space.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- *
- * If some data could not be copied, this function will pad the copied
- * data to the requested size using zero bytes.
- */
-#define copy_from_user(to, from, n) \
-({ \
- void *__cu_to; \
- const void __user *__cu_from; \
- long __cu_len; \
- \
- __cu_to = (to); \
- __cu_from = (from); \
- __cu_len = (n); \
- \
- check_object_size(__cu_to, __cu_len, false); \
- \
- if (eva_kernel_access()) { \
- __cu_len = __invoke_copy_from_kernel(__cu_to, \
- __cu_from, \
- __cu_len); \
- } else { \
- if (access_ok(VERIFY_READ, __cu_from, __cu_len)) { \
- might_fault(); \
- __cu_len = __invoke_copy_from_user(__cu_to, \
- __cu_from, \
- __cu_len); \
- } else { \
- memset(__cu_to, 0, __cu_len); \
- } \
- } \
- __cu_len; \
-})
+static inline unsigned long
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
+{
+ if (eva_kernel_access())
+ return __invoke_copy_from_kernel(to, from, n);
+ else
+ return __invoke_copy_from_user(to, from, n);
+}
-#define __copy_in_user(to, from, n) \
-({ \
- void __user *__cu_to; \
- const void __user *__cu_from; \
- long __cu_len; \
- \
- __cu_to = (to); \
- __cu_from = (from); \
- __cu_len = (n); \
- if (eva_kernel_access()) { \
- __cu_len = ___invoke_copy_in_kernel(__cu_to, __cu_from, \
- __cu_len); \
- } else { \
- might_fault(); \
- __cu_len = ___invoke_copy_in_user(__cu_to, __cu_from, \
- __cu_len); \
- } \
- __cu_len; \
-})
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
-#define copy_in_user(to, from, n) \
-({ \
- void __user *__cu_to; \
- const void __user *__cu_from; \
- long __cu_len; \
- \
- __cu_to = (to); \
- __cu_from = (from); \
- __cu_len = (n); \
- if (eva_kernel_access()) { \
- __cu_len = ___invoke_copy_in_kernel(__cu_to,__cu_from, \
- __cu_len); \
- } else { \
- if (likely(access_ok(VERIFY_READ, __cu_from, __cu_len) &&\
- access_ok(VERIFY_WRITE, __cu_to, __cu_len))) {\
- might_fault(); \
- __cu_len = ___invoke_copy_in_user(__cu_to, \
- __cu_from, \
- __cu_len); \
- } \
- } \
- __cu_len; \
-})
+static inline unsigned long
+raw_copy_in_user(void __user*to, const void __user *from, unsigned long n)
+{
+ if (eva_kernel_access())
+ return ___invoke_copy_in_kernel(to, from, n);
+ else
+ return ___invoke_copy_in_user(to, from, n);
+}
extern __kernel_size_t __bzero_kernel(void __user *addr, __kernel_size_t size);
extern __kernel_size_t __bzero(void __user *addr, __kernel_size_t size);
diff --git a/arch/mips/include/uapi/asm/socket.h b/arch/mips/include/uapi/asm/socket.h
index 566ecdc..3418ec9 100644
--- a/arch/mips/include/uapi/asm/socket.h
+++ b/arch/mips/include/uapi/asm/socket.h
@@ -110,4 +110,10 @@
#define SCM_TIMESTAMPING_OPT_STATS 54
+#define SO_MEMINFO 55
+
+#define SO_INCOMING_NAPI_ID 56
+
+#define SO_COOKIE 57
+
#endif /* _UAPI_ASM_SOCKET_H */
diff --git a/arch/mips/jz4740/time.c b/arch/mips/jz4740/time.c
index bcf8f8c..bb1ad51 100644
--- a/arch/mips/jz4740/time.c
+++ b/arch/mips/jz4740/time.c
@@ -145,7 +145,9 @@
clockevent_set_clock(&jz4740_clockevent, clk_rate);
jz4740_clockevent.min_delta_ns = clockevent_delta2ns(100, &jz4740_clockevent);
+ jz4740_clockevent.min_delta_ticks = 100;
jz4740_clockevent.max_delta_ns = clockevent_delta2ns(0xffff, &jz4740_clockevent);
+ jz4740_clockevent.max_delta_ticks = 0xffff;
jz4740_clockevent.cpumask = cpumask_of(0);
clockevents_register_device(&jz4740_clockevent);
diff --git a/arch/mips/kernel/cevt-bcm1480.c b/arch/mips/kernel/cevt-bcm1480.c
index 940ac00..8f9f2da 100644
--- a/arch/mips/kernel/cevt-bcm1480.c
+++ b/arch/mips/kernel/cevt-bcm1480.c
@@ -123,7 +123,9 @@
CLOCK_EVT_FEAT_ONESHOT;
clockevent_set_clock(cd, V_SCD_TIMER_FREQ);
cd->max_delta_ns = clockevent_delta2ns(0x7fffff, cd);
+ cd->max_delta_ticks = 0x7fffff;
cd->min_delta_ns = clockevent_delta2ns(2, cd);
+ cd->min_delta_ticks = 2;
cd->rating = 200;
cd->irq = irq;
cd->cpumask = cpumask_of(cpu);
diff --git a/arch/mips/kernel/cevt-ds1287.c b/arch/mips/kernel/cevt-ds1287.c
index 77a5ddf..61ad907 100644
--- a/arch/mips/kernel/cevt-ds1287.c
+++ b/arch/mips/kernel/cevt-ds1287.c
@@ -128,7 +128,9 @@
cd->irq = irq;
clockevent_set_clock(cd, 32768);
cd->max_delta_ns = clockevent_delta2ns(0x7fffffff, cd);
+ cd->max_delta_ticks = 0x7fffffff;
cd->min_delta_ns = clockevent_delta2ns(0x300, cd);
+ cd->min_delta_ticks = 0x300;
cd->cpumask = cpumask_of(0);
clockevents_register_device(&ds1287_clockevent);
diff --git a/arch/mips/kernel/cevt-gt641xx.c b/arch/mips/kernel/cevt-gt641xx.c
index 6604005..fd90c82 100644
--- a/arch/mips/kernel/cevt-gt641xx.c
+++ b/arch/mips/kernel/cevt-gt641xx.c
@@ -152,7 +152,9 @@
cd->rating = 200 + gt641xx_base_clock / 10000000;
clockevent_set_clock(cd, gt641xx_base_clock);
cd->max_delta_ns = clockevent_delta2ns(0x7fffffff, cd);
+ cd->max_delta_ticks = 0x7fffffff;
cd->min_delta_ns = clockevent_delta2ns(0x300, cd);
+ cd->min_delta_ticks = 0x300;
cd->cpumask = cpumask_of(0);
clockevents_register_device(>641xx_timer0_clockevent);
diff --git a/arch/mips/kernel/cevt-r4k.c b/arch/mips/kernel/cevt-r4k.c
index 804d2a2..dd6a18b 100644
--- a/arch/mips/kernel/cevt-r4k.c
+++ b/arch/mips/kernel/cevt-r4k.c
@@ -80,7 +80,7 @@
}
/* Sorted insert of 75th percentile into buf2 */
- for (k = 0; k < i; ++k) {
+ for (k = 0; k < i && k < ARRAY_SIZE(buf2); ++k) {
if (buf1[ARRAY_SIZE(buf1) - 1] < buf2[k]) {
l = min_t(unsigned int,
i, ARRAY_SIZE(buf2) - 1);
diff --git a/arch/mips/kernel/cevt-sb1250.c b/arch/mips/kernel/cevt-sb1250.c
index 3d860ef..9d1edb5 100644
--- a/arch/mips/kernel/cevt-sb1250.c
+++ b/arch/mips/kernel/cevt-sb1250.c
@@ -123,7 +123,9 @@
CLOCK_EVT_FEAT_ONESHOT;
clockevent_set_clock(cd, V_SCD_TIMER_FREQ);
cd->max_delta_ns = clockevent_delta2ns(0x7fffff, cd);
+ cd->max_delta_ticks = 0x7fffff;
cd->min_delta_ns = clockevent_delta2ns(2, cd);
+ cd->min_delta_ticks = 2;
cd->rating = 200;
cd->irq = irq;
cd->cpumask = cpumask_of(cpu);
diff --git a/arch/mips/kernel/cevt-txx9.c b/arch/mips/kernel/cevt-txx9.c
index aaca60d..7b17c8f 100644
--- a/arch/mips/kernel/cevt-txx9.c
+++ b/arch/mips/kernel/cevt-txx9.c
@@ -196,7 +196,9 @@
clockevent_set_clock(cd, TIMER_CLK(imbusclk));
cd->max_delta_ns =
clockevent_delta2ns(0xffffffff >> (32 - TXX9_TIMER_BITS), cd);
+ cd->max_delta_ticks = 0xffffffff >> (32 - TXX9_TIMER_BITS);
cd->min_delta_ns = clockevent_delta2ns(0xf, cd);
+ cd->min_delta_ticks = 0xf;
cd->irq = irq;
cd->cpumask = cpumask_of(0),
clockevents_register_device(cd);
diff --git a/arch/mips/kernel/elf.c b/arch/mips/kernel/elf.c
index 6430bff..5c429d7 100644
--- a/arch/mips/kernel/elf.c
+++ b/arch/mips/kernel/elf.c
@@ -257,7 +257,7 @@
else if ((prog_req.fr1 && prog_req.frdefault) ||
(prog_req.single && !prog_req.frdefault))
/* Make sure 64-bit MIPS III/IV/64R1 will not pick FR1 */
- state->overall_fp_mode = ((current_cpu_data.fpu_id & MIPS_FPIR_F64) &&
+ state->overall_fp_mode = ((raw_current_cpu_data.fpu_id & MIPS_FPIR_F64) &&
cpu_has_mips_r2_r6) ?
FP_FR1 : FP_FR0;
else if (prog_req.fr1)
diff --git a/arch/mips/kernel/kgdb.c b/arch/mips/kernel/kgdb.c
index 1f4bd22..eb6c0d5 100644
--- a/arch/mips/kernel/kgdb.c
+++ b/arch/mips/kernel/kgdb.c
@@ -244,9 +244,6 @@
void sleeping_thread_to_gdb_regs(unsigned long *gdb_regs, struct task_struct *p)
{
int reg;
- struct thread_info *ti = task_thread_info(p);
- unsigned long ksp = (unsigned long)ti + THREAD_SIZE - 32;
- struct pt_regs *regs = (struct pt_regs *)ksp - 1;
#if (KGDB_GDB_REG_SIZE == 32)
u32 *ptr = (u32 *)gdb_regs;
#else
@@ -254,25 +251,46 @@
#endif
for (reg = 0; reg < 16; reg++)
- *(ptr++) = regs->regs[reg];
+ *(ptr++) = 0;
/* S0 - S7 */
- for (reg = 16; reg < 24; reg++)
- *(ptr++) = regs->regs[reg];
+ *(ptr++) = p->thread.reg16;
+ *(ptr++) = p->thread.reg17;
+ *(ptr++) = p->thread.reg18;
+ *(ptr++) = p->thread.reg19;
+ *(ptr++) = p->thread.reg20;
+ *(ptr++) = p->thread.reg21;
+ *(ptr++) = p->thread.reg22;
+ *(ptr++) = p->thread.reg23;
for (reg = 24; reg < 28; reg++)
*(ptr++) = 0;
/* GP, SP, FP, RA */
- for (reg = 28; reg < 32; reg++)
- *(ptr++) = regs->regs[reg];
+ *(ptr++) = (long)p;
+ *(ptr++) = p->thread.reg29;
+ *(ptr++) = p->thread.reg30;
+ *(ptr++) = p->thread.reg31;
- *(ptr++) = regs->cp0_status;
- *(ptr++) = regs->lo;
- *(ptr++) = regs->hi;
- *(ptr++) = regs->cp0_badvaddr;
- *(ptr++) = regs->cp0_cause;
- *(ptr++) = regs->cp0_epc;
+ *(ptr++) = p->thread.cp0_status;
+
+ /* lo, hi */
+ *(ptr++) = 0;
+ *(ptr++) = 0;
+
+ /*
+ * BadVAddr, Cause
+ * Ideally these would come from the last exception frame up the stack
+ * but that requires unwinding, otherwise we can't know much for sure.
+ */
+ *(ptr++) = 0;
+ *(ptr++) = 0;
+
+ /*
+ * PC
+ * use return address (RA), i.e. the moment after return from resume()
+ */
+ *(ptr++) = p->thread.reg31;
}
void kgdb_arch_set_pc(struct pt_regs *regs, unsigned long pc)
diff --git a/arch/mips/kernel/mips-r2-to-r6-emul.c b/arch/mips/kernel/mips-r2-to-r6-emul.c
index d8f1cf1..550e7d0 100644
--- a/arch/mips/kernel/mips-r2-to-r6-emul.c
+++ b/arch/mips/kernel/mips-r2-to-r6-emul.c
@@ -1200,7 +1200,7 @@
case lwl_op:
rt = regs->regs[MIPSInst_RT(inst)];
vaddr = regs->regs[MIPSInst_RS(inst)] + MIPSInst_SIMM(inst);
- if (!access_ok(VERIFY_READ, vaddr, 4)) {
+ if (!access_ok(VERIFY_READ, (void __user *)vaddr, 4)) {
current->thread.cp0_baduaddr = vaddr;
err = SIGSEGV;
break;
@@ -1273,7 +1273,7 @@
case lwr_op:
rt = regs->regs[MIPSInst_RT(inst)];
vaddr = regs->regs[MIPSInst_RS(inst)] + MIPSInst_SIMM(inst);
- if (!access_ok(VERIFY_READ, vaddr, 4)) {
+ if (!access_ok(VERIFY_READ, (void __user *)vaddr, 4)) {
current->thread.cp0_baduaddr = vaddr;
err = SIGSEGV;
break;
@@ -1347,7 +1347,7 @@
case swl_op:
rt = regs->regs[MIPSInst_RT(inst)];
vaddr = regs->regs[MIPSInst_RS(inst)] + MIPSInst_SIMM(inst);
- if (!access_ok(VERIFY_WRITE, vaddr, 4)) {
+ if (!access_ok(VERIFY_WRITE, (void __user *)vaddr, 4)) {
current->thread.cp0_baduaddr = vaddr;
err = SIGSEGV;
break;
@@ -1417,7 +1417,7 @@
case swr_op:
rt = regs->regs[MIPSInst_RT(inst)];
vaddr = regs->regs[MIPSInst_RS(inst)] + MIPSInst_SIMM(inst);
- if (!access_ok(VERIFY_WRITE, vaddr, 4)) {
+ if (!access_ok(VERIFY_WRITE, (void __user *)vaddr, 4)) {
current->thread.cp0_baduaddr = vaddr;
err = SIGSEGV;
break;
@@ -1492,7 +1492,7 @@
rt = regs->regs[MIPSInst_RT(inst)];
vaddr = regs->regs[MIPSInst_RS(inst)] + MIPSInst_SIMM(inst);
- if (!access_ok(VERIFY_READ, vaddr, 8)) {
+ if (!access_ok(VERIFY_READ, (void __user *)vaddr, 8)) {
current->thread.cp0_baduaddr = vaddr;
err = SIGSEGV;
break;
@@ -1611,7 +1611,7 @@
rt = regs->regs[MIPSInst_RT(inst)];
vaddr = regs->regs[MIPSInst_RS(inst)] + MIPSInst_SIMM(inst);
- if (!access_ok(VERIFY_READ, vaddr, 8)) {
+ if (!access_ok(VERIFY_READ, (void __user *)vaddr, 8)) {
current->thread.cp0_baduaddr = vaddr;
err = SIGSEGV;
break;
@@ -1730,7 +1730,7 @@
rt = regs->regs[MIPSInst_RT(inst)];
vaddr = regs->regs[MIPSInst_RS(inst)] + MIPSInst_SIMM(inst);
- if (!access_ok(VERIFY_WRITE, vaddr, 8)) {
+ if (!access_ok(VERIFY_WRITE, (void __user *)vaddr, 8)) {
current->thread.cp0_baduaddr = vaddr;
err = SIGSEGV;
break;
@@ -1848,7 +1848,7 @@
rt = regs->regs[MIPSInst_RT(inst)];
vaddr = regs->regs[MIPSInst_RS(inst)] + MIPSInst_SIMM(inst);
- if (!access_ok(VERIFY_WRITE, vaddr, 8)) {
+ if (!access_ok(VERIFY_WRITE, (void __user *)vaddr, 8)) {
current->thread.cp0_baduaddr = vaddr;
err = SIGSEGV;
break;
@@ -1965,7 +1965,7 @@
err = SIGBUS;
break;
}
- if (!access_ok(VERIFY_READ, vaddr, 4)) {
+ if (!access_ok(VERIFY_READ, (void __user *)vaddr, 4)) {
current->thread.cp0_baduaddr = vaddr;
err = SIGBUS;
break;
@@ -2021,7 +2021,7 @@
err = SIGBUS;
break;
}
- if (!access_ok(VERIFY_WRITE, vaddr, 4)) {
+ if (!access_ok(VERIFY_WRITE, (void __user *)vaddr, 4)) {
current->thread.cp0_baduaddr = vaddr;
err = SIGBUS;
break;
@@ -2084,7 +2084,7 @@
err = SIGBUS;
break;
}
- if (!access_ok(VERIFY_READ, vaddr, 8)) {
+ if (!access_ok(VERIFY_READ, (void __user *)vaddr, 8)) {
current->thread.cp0_baduaddr = vaddr;
err = SIGBUS;
break;
@@ -2145,7 +2145,7 @@
err = SIGBUS;
break;
}
- if (!access_ok(VERIFY_WRITE, vaddr, 8)) {
+ if (!access_ok(VERIFY_WRITE, (void __user *)vaddr, 8)) {
current->thread.cp0_baduaddr = vaddr;
err = SIGBUS;
break;
diff --git a/arch/mips/kernel/perf_event_mipsxx.c b/arch/mips/kernel/perf_event_mipsxx.c
index 8c35b31..9452b02 100644
--- a/arch/mips/kernel/perf_event_mipsxx.c
+++ b/arch/mips/kernel/perf_event_mipsxx.c
@@ -1446,6 +1446,11 @@
HANDLE_COUNTER(0)
}
+#ifdef CONFIG_MIPS_PERF_SHARED_TC_COUNTERS
+ read_unlock(&pmuint_rwlock);
+#endif
+ resume_local_counters();
+
/*
* Do all the work for the pending perf events. We can do this
* in here because the performance counter interrupt is a regular
@@ -1454,10 +1459,6 @@
if (handled == IRQ_HANDLED)
irq_work_run();
-#ifdef CONFIG_MIPS_PERF_SHARED_TC_COUNTERS
- read_unlock(&pmuint_rwlock);
-#endif
- resume_local_counters();
return handled;
}
diff --git a/arch/mips/kernel/relocate.c b/arch/mips/kernel/relocate.c
index 9103beb..2d1a0c4 100644
--- a/arch/mips/kernel/relocate.c
+++ b/arch/mips/kernel/relocate.c
@@ -18,7 +18,7 @@
#include <linux/kernel.h>
#include <linux/libfdt.h>
#include <linux/of_fdt.h>
-#include <linux/sched.h>
+#include <linux/sched/task.h>
#include <linux/start_kernel.h>
#include <linux/string.h>
#include <linux/printk.h>
diff --git a/arch/mips/kernel/smp-cps.c b/arch/mips/kernel/smp-cps.c
index 6d45f05..795b4aa 100644
--- a/arch/mips/kernel/smp-cps.c
+++ b/arch/mips/kernel/smp-cps.c
@@ -422,13 +422,12 @@
local_irq_disable();
idle_task_exit();
cpu = smp_processor_id();
+ core = cpu_data[cpu].core;
cpu_death = CPU_DEATH_POWER;
pr_debug("CPU%d going offline\n", cpu);
if (cpu_has_mipsmt || cpu_has_vp) {
- core = cpu_data[cpu].core;
-
/* Look for another online VPE within the core */
for_each_online_cpu(cpu_death_sibling) {
if (cpu_data[cpu_death_sibling].core != core)
diff --git a/arch/mips/kernel/syscall.c b/arch/mips/kernel/syscall.c
index f1d17ec..1dfa7f5 100644
--- a/arch/mips/kernel/syscall.c
+++ b/arch/mips/kernel/syscall.c
@@ -98,7 +98,7 @@
if (unlikely(addr & 3))
return -EINVAL;
- if (unlikely(!access_ok(VERIFY_WRITE, addr, 4)))
+ if (unlikely(!access_ok(VERIFY_WRITE, (const void __user *)addr, 4)))
return -EINVAL;
if (cpu_has_llsc && R10000_LLSC_WAR) {
diff --git a/arch/mips/kernel/unaligned.c b/arch/mips/kernel/unaligned.c
index 7ed9835..f806ee5 100644
--- a/arch/mips/kernel/unaligned.c
+++ b/arch/mips/kernel/unaligned.c
@@ -1026,7 +1026,7 @@
goto sigbus;
if (IS_ENABLED(CONFIG_EVA)) {
- if (segment_eq(get_fs(), get_ds()))
+ if (uaccess_kernel())
LoadHW(addr, value, res);
else
LoadHWE(addr, value, res);
@@ -1045,7 +1045,7 @@
goto sigbus;
if (IS_ENABLED(CONFIG_EVA)) {
- if (segment_eq(get_fs(), get_ds()))
+ if (uaccess_kernel())
LoadW(addr, value, res);
else
LoadWE(addr, value, res);
@@ -1064,7 +1064,7 @@
goto sigbus;
if (IS_ENABLED(CONFIG_EVA)) {
- if (segment_eq(get_fs(), get_ds()))
+ if (uaccess_kernel())
LoadHWU(addr, value, res);
else
LoadHWUE(addr, value, res);
@@ -1132,7 +1132,7 @@
value = regs->regs[insn.i_format.rt];
if (IS_ENABLED(CONFIG_EVA)) {
- if (segment_eq(get_fs(), get_ds()))
+ if (uaccess_kernel())
StoreHW(addr, value, res);
else
StoreHWE(addr, value, res);
@@ -1152,7 +1152,7 @@
value = regs->regs[insn.i_format.rt];
if (IS_ENABLED(CONFIG_EVA)) {
- if (segment_eq(get_fs(), get_ds()))
+ if (uaccess_kernel())
StoreW(addr, value, res);
else
StoreWE(addr, value, res);
diff --git a/arch/mips/kernel/vmlinux.lds.S b/arch/mips/kernel/vmlinux.lds.S
index f0a0e6d..8ca2371 100644
--- a/arch/mips/kernel/vmlinux.lds.S
+++ b/arch/mips/kernel/vmlinux.lds.S
@@ -97,6 +97,7 @@
DATA_DATA
CONSTRUCTORS
}
+ BUG_TABLE
_gp = . + 0x8000;
.lit8 : {
*(.lit8)
diff --git a/arch/mips/lib/memcpy.S b/arch/mips/lib/memcpy.S
index c3031f1..3114a2e 100644
--- a/arch/mips/lib/memcpy.S
+++ b/arch/mips/lib/memcpy.S
@@ -562,39 +562,9 @@
LOADK t0, THREAD_BUADDR(t0) # t0 is just past last good address
nop
SUB len, AT, t0 # len number of uncopied bytes
- bnez t6, .Ldone\@ /* Skip the zeroing part if inatomic */
- /*
- * Here's where we rely on src and dst being incremented in tandem,
- * See (3) above.
- * dst += (fault addr - src) to put dst at first byte to clear
- */
- ADD dst, t0 # compute start address in a1
- SUB dst, src
- /*
- * Clear len bytes starting at dst. Can't call __bzero because it
- * might modify len. An inefficient loop for these rare times...
- */
- .set reorder /* DADDI_WAR */
- SUB src, len, 1
- beqz len, .Ldone\@
- .set noreorder
-1: sb zero, 0(dst)
- ADD dst, dst, 1
-#ifndef CONFIG_CPU_DADDI_WORKAROUNDS
- bnez src, 1b
- SUB src, src, 1
-#else
- .set push
- .set noat
- li v1, 1
- bnez src, 1b
- SUB src, src, v1
- .set pop
-#endif
jr ra
nop
-
#define SEXC(n) \
.set reorder; /* DADDI_WAR */ \
.Ls_exc_p ## n ## u\@: \
@@ -673,15 +643,6 @@
END(__rmemcpy)
/*
- * t6 is used as a flag to note inatomic mode.
- */
-LEAF(__copy_user_inatomic)
-EXPORT_SYMBOL(__copy_user_inatomic)
- b __copy_user_common
- li t6, 1
- END(__copy_user_inatomic)
-
-/*
* A combined memcpy/__copy_user
* __copy_user sets len to 0 for success; else to an upper bound of
* the number of uncopied bytes.
@@ -694,8 +655,6 @@
.L__memcpy:
FEXPORT(__copy_user)
EXPORT_SYMBOL(__copy_user)
- li t6, 0 /* not inatomic */
-__copy_user_common:
/* Legacy Mode, user <-> user */
__BUILD_COPY_USER LEGACY_MODE USEROP USEROP
@@ -708,20 +667,12 @@
* space
*/
-LEAF(__copy_user_inatomic_eva)
-EXPORT_SYMBOL(__copy_user_inatomic_eva)
- b __copy_from_user_common
- li t6, 1
- END(__copy_user_inatomic_eva)
-
/*
* __copy_from_user (EVA)
*/
LEAF(__copy_from_user_eva)
EXPORT_SYMBOL(__copy_from_user_eva)
- li t6, 0 /* not inatomic */
-__copy_from_user_common:
__BUILD_COPY_USER EVA_MODE USEROP KERNELOP
END(__copy_from_user_eva)
diff --git a/arch/mips/loongson32/common/time.c b/arch/mips/loongson32/common/time.c
index e6f972d..1c4332a 100644
--- a/arch/mips/loongson32/common/time.c
+++ b/arch/mips/loongson32/common/time.c
@@ -199,7 +199,9 @@
clockevent_set_clock(cd, mips_hpt_frequency);
cd->max_delta_ns = clockevent_delta2ns(0xffffff, cd);
+ cd->max_delta_ticks = 0xffffff;
cd->min_delta_ns = clockevent_delta2ns(0x000300, cd);
+ cd->min_delta_ticks = 0x000300;
cd->cpumask = cpumask_of(smp_processor_id());
clockevents_register_device(cd);
diff --git a/arch/mips/loongson64/common/cs5536/cs5536_mfgpt.c b/arch/mips/loongson64/common/cs5536/cs5536_mfgpt.c
index b817d6d..a6adcc4 100644
--- a/arch/mips/loongson64/common/cs5536/cs5536_mfgpt.c
+++ b/arch/mips/loongson64/common/cs5536/cs5536_mfgpt.c
@@ -123,7 +123,9 @@
cd->cpumask = cpumask_of(cpu);
clockevent_set_clock(cd, MFGPT_TICK_RATE);
cd->max_delta_ns = clockevent_delta2ns(0xffff, cd);
+ cd->max_delta_ticks = 0xffff;
cd->min_delta_ns = clockevent_delta2ns(0xf, cd);
+ cd->min_delta_ticks = 0xf;
/* Enable MFGPT0 Comparator 2 Output to the Interrupt Mapper */
_wrmsr(DIVIL_MSR_REG(MFGPT_IRQ), 0, 0x100);
diff --git a/arch/mips/loongson64/loongson-3/hpet.c b/arch/mips/loongson64/loongson-3/hpet.c
index 24afe36..4df9d4b 100644
--- a/arch/mips/loongson64/loongson-3/hpet.c
+++ b/arch/mips/loongson64/loongson-3/hpet.c
@@ -241,7 +241,9 @@
cd->cpumask = cpumask_of(cpu);
clockevent_set_clock(cd, HPET_FREQ);
cd->max_delta_ns = clockevent_delta2ns(0x7fffffff, cd);
+ cd->max_delta_ticks = 0x7fffffff;
cd->min_delta_ns = clockevent_delta2ns(HPET_MIN_PROG_DELTA, cd);
+ cd->min_delta_ticks = HPET_MIN_PROG_DELTA;
clockevents_register_device(cd);
setup_irq(HPET_T0_IRQ, &hpet_irq);
diff --git a/arch/mips/mti-malta/malta-int.c b/arch/mips/mti-malta/malta-int.c
index cb675ec..54f56d5 100644
--- a/arch/mips/mti-malta/malta-int.c
+++ b/arch/mips/mti-malta/malta-int.c
@@ -232,6 +232,17 @@
{
int corehi_irq;
+ /*
+ * Preallocate the i8259's expected virq's here. Since irqchip_init()
+ * will probe the irqchips in hierarchial order, i8259 is probed last.
+ * If anything allocates a virq before the i8259 is probed, it will
+ * be given one of the i8259's expected range and consequently setup
+ * of the i8259 will fail.
+ */
+ WARN(irq_alloc_descs(I8259A_IRQ_BASE, I8259A_IRQ_BASE,
+ 16, numa_node_id()) < 0,
+ "Cannot reserve i8259 virqs at IRQ%d\n", I8259A_IRQ_BASE);
+
i8259_set_poll(mips_pcibios_iack);
irqchip_init();
diff --git a/arch/mips/mti-malta/malta-time.c b/arch/mips/mti-malta/malta-time.c
index 1829a90..289edcf 100644
--- a/arch/mips/mti-malta/malta-time.c
+++ b/arch/mips/mti-malta/malta-time.c
@@ -21,6 +21,7 @@
#include <linux/i8253.h>
#include <linux/init.h>
#include <linux/kernel_stat.h>
+#include <linux/libfdt.h>
#include <linux/math64.h>
#include <linux/sched.h>
#include <linux/spinlock.h>
@@ -207,6 +208,33 @@
CMOS_WRITE(ctrl & ~RTC_SET, RTC_CONTROL);
}
+#ifdef CONFIG_CLKSRC_MIPS_GIC
+static u32 gic_frequency_dt;
+
+static struct property gic_frequency_prop = {
+ .name = "clock-frequency",
+ .length = sizeof(u32),
+ .value = &gic_frequency_dt,
+};
+
+static void update_gic_frequency_dt(void)
+{
+ struct device_node *node;
+
+ gic_frequency_dt = cpu_to_be32(gic_frequency);
+
+ node = of_find_compatible_node(NULL, NULL, "mti,gic-timer");
+ if (!node) {
+ pr_err("mti,gic-timer device node not found\n");
+ return;
+ }
+
+ if (of_update_property(node, &gic_frequency_prop) < 0)
+ pr_err("error updating gic frequency property\n");
+}
+
+#endif
+
void __init plat_time_init(void)
{
unsigned int prid = read_c0_prid() & (PRID_COMP_MASK | PRID_IMP_MASK);
@@ -236,7 +264,8 @@
printk("GIC frequency %d.%02d MHz\n", freq/1000000,
(freq%1000000)*100/1000000);
#ifdef CONFIG_CLKSRC_MIPS_GIC
- gic_clocksource_init(gic_frequency);
+ update_gic_frequency_dt();
+ clocksource_probe();
#endif
}
#endif
diff --git a/arch/mips/oprofile/backtrace.c b/arch/mips/oprofile/backtrace.c
index 5e645c9..16ace55 100644
--- a/arch/mips/oprofile/backtrace.c
+++ b/arch/mips/oprofile/backtrace.c
@@ -18,7 +18,7 @@
static inline int get_mem(unsigned long addr, unsigned long *result)
{
unsigned long *address = (unsigned long *) addr;
- if (!access_ok(VERIFY_READ, addr, sizeof(unsigned long)))
+ if (!access_ok(VERIFY_READ, address, sizeof(unsigned long)))
return -1;
if (__copy_from_user_inatomic(result, address, sizeof(unsigned long)))
return -3;
diff --git a/arch/mips/pci/pci-legacy.c b/arch/mips/pci/pci-legacy.c
index 014649b..3a84f6c 100644
--- a/arch/mips/pci/pci-legacy.c
+++ b/arch/mips/pci/pci-legacy.c
@@ -190,7 +190,7 @@
}
INIT_LIST_HEAD(&hose->list);
- list_add(&hose->list, &controllers);
+ list_add_tail(&hose->list, &controllers);
/*
* Do not panic here but later - this might happen before console init.
diff --git a/arch/mips/ralink/cevt-rt3352.c b/arch/mips/ralink/cevt-rt3352.c
index f24eee0..b8a1376 100644
--- a/arch/mips/ralink/cevt-rt3352.c
+++ b/arch/mips/ralink/cevt-rt3352.c
@@ -129,7 +129,9 @@
systick.dev.name = np->name;
clockevents_calc_mult_shift(&systick.dev, SYSTICK_FREQ, 60);
systick.dev.max_delta_ns = clockevent_delta2ns(0x7fff, &systick.dev);
+ systick.dev.max_delta_ticks = 0x7fff;
systick.dev.min_delta_ns = clockevent_delta2ns(0x3, &systick.dev);
+ systick.dev.min_delta_ticks = 0x3;
systick.dev.irq = irq_of_parse_and_map(np, 0);
if (!systick.dev.irq) {
pr_err("%s: request_irq failed", np->name);
diff --git a/arch/mips/sgi-ip27/ip27-timer.c b/arch/mips/sgi-ip27/ip27-timer.c
index 695c51b..a53f0c8 100644
--- a/arch/mips/sgi-ip27/ip27-timer.c
+++ b/arch/mips/sgi-ip27/ip27-timer.c
@@ -113,7 +113,9 @@
cd->features = CLOCK_EVT_FEAT_ONESHOT;
clockevent_set_clock(cd, CYCLES_PER_SEC);
cd->max_delta_ns = clockevent_delta2ns(0xfffffffffffff, cd);
+ cd->max_delta_ticks = 0xfffffffffffff;
cd->min_delta_ns = clockevent_delta2ns(0x300, cd);
+ cd->min_delta_ticks = 0x300;
cd->rating = 200;
cd->irq = irq;
cd->cpumask = cpumask_of(cpu);
diff --git a/arch/mn10300/include/asm/Kbuild b/arch/mn10300/include/asm/Kbuild
index 97f64c7..ed810e7 100644
--- a/arch/mn10300/include/asm/Kbuild
+++ b/arch/mn10300/include/asm/Kbuild
@@ -2,6 +2,7 @@
generic-y += barrier.h
generic-y += clkdev.h
generic-y += exec.h
+generic-y += extable.h
generic-y += irq_work.h
generic-y += mcs_spinlock.h
generic-y += mm-arch-hooks.h
diff --git a/arch/mn10300/include/asm/uaccess.h b/arch/mn10300/include/asm/uaccess.h
index 2eedf6f..c696647 100644
--- a/arch/mn10300/include/asm/uaccess.h
+++ b/arch/mn10300/include/asm/uaccess.h
@@ -14,13 +14,8 @@
/*
* User space memory access functions
*/
-#include <linux/thread_info.h>
#include <linux/kernel.h>
#include <asm/page.h>
-#include <asm/errno.h>
-
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
/*
* The fs value determines whether argument validity checking should be
@@ -71,26 +66,7 @@
#define access_ok(type, addr, size) (__range_ok((addr), (size)) == 0)
#define __access_ok(addr, size) (__range_ok((addr), (size)) == 0)
-/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-
-struct exception_table_entry
-{
- unsigned long insn, fixup;
-};
-
-/* Returns 0 if exception not found and fixup otherwise. */
-extern int fixup_exception(struct pt_regs *regs);
+#include <asm/extable.h>
#define put_user(x, ptr) __put_user_check((x), (ptr), sizeof(*(ptr)))
#define get_user(x, ptr) __get_user_check((x), (ptr), sizeof(*(ptr)))
@@ -299,171 +275,20 @@
} \
} while (0)
-#define __copy_user_zeroing(to, from, size) \
-do { \
- if (size) { \
- void *__to = to; \
- const void *__from = from; \
- int w; \
- asm volatile( \
- "0: movbu (%0),%3;\n" \
- "1: movbu %3,(%1);\n" \
- " inc %0;\n" \
- " inc %1;\n" \
- " add -1,%2;\n" \
- " bne 0b;\n" \
- "2:\n" \
- " .section .fixup,\"ax\"\n" \
- "3:\n" \
- " mov %2,%0\n" \
- " clr %3\n" \
- "4: movbu %3,(%1);\n" \
- " inc %1;\n" \
- " add -1,%2;\n" \
- " bne 4b;\n" \
- " mov %0,%2\n" \
- " jmp 2b\n" \
- " .previous\n" \
- " .section __ex_table,\"a\"\n" \
- " .balign 4\n" \
- " .long 0b,3b\n" \
- " .long 1b,3b\n" \
- " .previous\n" \
- : "=a"(__from), "=a"(__to), "=r"(size), "=&r"(w)\
- : "0"(__from), "1"(__to), "2"(size) \
- : "cc", "memory"); \
- } \
-} while (0)
-
-/* We let the __ versions of copy_from/to_user inline, because they're often
- * used in fast paths and have only a small space overhead.
- */
-static inline
-unsigned long __generic_copy_from_user_nocheck(void *to, const void *from,
- unsigned long n)
-{
- __copy_user_zeroing(to, from, n);
- return n;
-}
-
-static inline
-unsigned long __generic_copy_to_user_nocheck(void *to, const void *from,
- unsigned long n)
+static inline unsigned long
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
__copy_user(to, from, n);
return n;
}
-
-#if 0
-#error "don't use - these macros don't increment to & from pointers"
-/* Optimize just a little bit when we know the size of the move. */
-#define __constant_copy_user(to, from, size) \
-do { \
- asm volatile( \
- " mov %0,a0;\n" \
- "0: movbu (%1),d3;\n" \
- "1: movbu d3,(%2);\n" \
- " add -1,a0;\n" \
- " bne 0b;\n" \
- "2:;" \
- ".section .fixup,\"ax\"\n" \
- "3: jmp 2b\n" \
- ".previous\n" \
- ".section __ex_table,\"a\"\n" \
- " .balign 4\n" \
- " .long 0b,3b\n" \
- " .long 1b,3b\n" \
- ".previous" \
- : \
- : "d"(size), "d"(to), "d"(from) \
- : "d3", "a0"); \
-} while (0)
-
-/* Optimize just a little bit when we know the size of the move. */
-#define __constant_copy_user_zeroing(to, from, size) \
-do { \
- asm volatile( \
- " mov %0,a0;\n" \
- "0: movbu (%1),d3;\n" \
- "1: movbu d3,(%2);\n" \
- " add -1,a0;\n" \
- " bne 0b;\n" \
- "2:;" \
- ".section .fixup,\"ax\"\n" \
- "3: jmp 2b\n" \
- ".previous\n" \
- ".section __ex_table,\"a\"\n" \
- " .balign 4\n" \
- " .long 0b,3b\n" \
- " .long 1b,3b\n" \
- ".previous" \
- : \
- : "d"(size), "d"(to), "d"(from) \
- : "d3", "a0"); \
-} while (0)
-
-static inline
-unsigned long __constant_copy_to_user(void *to, const void *from,
- unsigned long n)
+static inline unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
{
- if (access_ok(VERIFY_WRITE, to, n))
- __constant_copy_user(to, from, n);
+ __copy_user(to, from, n);
return n;
}
-static inline
-unsigned long __constant_copy_from_user(void *to, const void *from,
- unsigned long n)
-{
- if (access_ok(VERIFY_READ, from, n))
- __constant_copy_user_zeroing(to, from, n);
- return n;
-}
-
-static inline
-unsigned long __constant_copy_to_user_nocheck(void *to, const void *from,
- unsigned long n)
-{
- __constant_copy_user(to, from, n);
- return n;
-}
-
-static inline
-unsigned long __constant_copy_from_user_nocheck(void *to, const void *from,
- unsigned long n)
-{
- __constant_copy_user_zeroing(to, from, n);
- return n;
-}
-#endif
-
-extern unsigned long __generic_copy_to_user(void __user *, const void *,
- unsigned long);
-extern unsigned long __generic_copy_from_user(void *, const void __user *,
- unsigned long);
-
-#define __copy_to_user_inatomic(to, from, n) \
- __generic_copy_to_user_nocheck((to), (from), (n))
-#define __copy_from_user_inatomic(to, from, n) \
- __generic_copy_from_user_nocheck((to), (from), (n))
-
-#define __copy_to_user(to, from, n) \
-({ \
- might_fault(); \
- __copy_to_user_inatomic((to), (from), (n)); \
-})
-
-#define __copy_from_user(to, from, n) \
-({ \
- might_fault(); \
- __copy_from_user_inatomic((to), (from), (n)); \
-})
-
-
-#define copy_to_user(to, from, n) __generic_copy_to_user((to), (from), (n))
-#define copy_from_user(to, from, n) __generic_copy_from_user((to), (from), (n))
-
extern long strncpy_from_user(char *dst, const char __user *src, long count);
extern long __strncpy_from_user(char *dst, const char __user *src, long count);
extern long strnlen_user(const char __user *str, long n);
diff --git a/arch/mn10300/include/uapi/asm/socket.h b/arch/mn10300/include/uapi/asm/socket.h
index 0e12527..4526e92 100644
--- a/arch/mn10300/include/uapi/asm/socket.h
+++ b/arch/mn10300/include/uapi/asm/socket.h
@@ -92,4 +92,10 @@
#define SCM_TIMESTAMPING_OPT_STATS 54
+#define SO_MEMINFO 55
+
+#define SO_INCOMING_NAPI_ID 56
+
+#define SO_COOKIE 57
+
#endif /* _ASM_SOCKET_H */
diff --git a/arch/mn10300/kernel/cevt-mn10300.c b/arch/mn10300/kernel/cevt-mn10300.c
index d9b34dd..2b21bbc9 100644
--- a/arch/mn10300/kernel/cevt-mn10300.c
+++ b/arch/mn10300/kernel/cevt-mn10300.c
@@ -98,7 +98,9 @@
/* Calculate the min / max delta */
cd->max_delta_ns = clockevent_delta2ns(TMJCBR_MAX, cd);
+ cd->max_delta_ticks = TMJCBR_MAX;
cd->min_delta_ns = clockevent_delta2ns(100, cd);
+ cd->min_delta_ticks = 100;
cd->rating = 200;
cd->cpumask = cpumask_of(smp_processor_id());
diff --git a/arch/mn10300/kernel/mn10300_ksyms.c b/arch/mn10300/kernel/mn10300_ksyms.c
index ec6c4f8..5e9f919 100644
--- a/arch/mn10300/kernel/mn10300_ksyms.c
+++ b/arch/mn10300/kernel/mn10300_ksyms.c
@@ -26,8 +26,6 @@
EXPORT_SYMBOL(__strncpy_from_user);
EXPORT_SYMBOL(clear_user);
EXPORT_SYMBOL(__clear_user);
-EXPORT_SYMBOL(__generic_copy_from_user);
-EXPORT_SYMBOL(__generic_copy_to_user);
EXPORT_SYMBOL(strnlen_user);
extern u64 __ashrdi3(u64, unsigned);
diff --git a/arch/mn10300/lib/usercopy.c b/arch/mn10300/lib/usercopy.c
index ce8899e..cece179 100644
--- a/arch/mn10300/lib/usercopy.c
+++ b/arch/mn10300/lib/usercopy.c
@@ -11,24 +11,6 @@
*/
#include <linux/uaccess.h>
-unsigned long
-__generic_copy_to_user(void *to, const void *from, unsigned long n)
-{
- if (access_ok(VERIFY_WRITE, to, n))
- __copy_user(to, from, n);
- return n;
-}
-
-unsigned long
-__generic_copy_from_user(void *to, const void *from, unsigned long n)
-{
- if (access_ok(VERIFY_READ, from, n))
- __copy_user_zeroing(to, from, n);
- else
- memset(to, 0, n);
- return n;
-}
-
/*
* Copy a null terminated string from userspace.
*/
diff --git a/arch/nios2/include/asm/Kbuild b/arch/nios2/include/asm/Kbuild
index aaa3c21..87e70f2 100644
--- a/arch/nios2/include/asm/Kbuild
+++ b/arch/nios2/include/asm/Kbuild
@@ -13,6 +13,7 @@
generic-y += emergency-restart.h
generic-y += errno.h
generic-y += exec.h
+generic-y += extable.h
generic-y += fb.h
generic-y += fcntl.h
generic-y += ftrace.h
diff --git a/arch/nios2/include/asm/uaccess.h b/arch/nios2/include/asm/uaccess.h
index 0ab8232..727bd95 100644
--- a/arch/nios2/include/asm/uaccess.h
+++ b/arch/nios2/include/asm/uaccess.h
@@ -13,33 +13,11 @@
#ifndef _ASM_NIOS2_UACCESS_H
#define _ASM_NIOS2_UACCESS_H
-#include <linux/errno.h>
-#include <linux/thread_info.h>
#include <linux/string.h>
#include <asm/page.h>
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
-/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-struct exception_table_entry {
- unsigned long insn;
- unsigned long fixup;
-};
-
-extern int fixup_exception(struct pt_regs *regs);
+#include <asm/extable.h>
/*
* Segment stuff
@@ -95,36 +73,17 @@
return __clear_user(to, n);
}
-extern long __copy_from_user(void *to, const void __user *from,
- unsigned long n);
-extern long __copy_to_user(void __user *to, const void *from, unsigned long n);
-
-static inline long copy_from_user(void *to, const void __user *from,
- unsigned long n)
-{
- unsigned long res = n;
- if (access_ok(VERIFY_READ, from, n))
- res = __copy_from_user(to, from, n);
- if (unlikely(res))
- memset(to + (n - res), 0, res);
- return res;
-}
-
-static inline long copy_to_user(void __user *to, const void *from,
- unsigned long n)
-{
- if (!access_ok(VERIFY_WRITE, to, n))
- return n;
- return __copy_to_user(to, from, n);
-}
+extern unsigned long
+raw_copy_from_user(void *to, const void __user *from, unsigned long n);
+extern unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n);
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
extern long strncpy_from_user(char *__to, const char __user *__from,
long __len);
extern long strnlen_user(const char __user *s, long n);
-#define __copy_from_user_inatomic __copy_from_user
-#define __copy_to_user_inatomic __copy_to_user
-
/* Optimized macros */
#define __get_user_asm(val, insn, addr, err) \
{ \
diff --git a/arch/nios2/mm/uaccess.c b/arch/nios2/mm/uaccess.c
index 7663e15..8049833 100644
--- a/arch/nios2/mm/uaccess.c
+++ b/arch/nios2/mm/uaccess.c
@@ -10,9 +10,9 @@
#include <linux/export.h>
#include <linux/uaccess.h>
-asm(".global __copy_from_user\n"
- " .type __copy_from_user, @function\n"
- "__copy_from_user:\n"
+asm(".global raw_copy_from_user\n"
+ " .type raw_copy_from_user, @function\n"
+ "raw_copy_from_user:\n"
" movi r2,7\n"
" mov r3,r4\n"
" bge r2,r6,1f\n"
@@ -65,12 +65,12 @@
".word 7b,13b\n"
".previous\n"
);
-EXPORT_SYMBOL(__copy_from_user);
+EXPORT_SYMBOL(raw_copy_from_user);
asm(
- " .global __copy_to_user\n"
- " .type __copy_to_user, @function\n"
- "__copy_to_user:\n"
+ " .global raw_copy_to_user\n"
+ " .type raw_copy_to_user, @function\n"
+ "raw_copy_to_user:\n"
" movi r2,7\n"
" mov r3,r4\n"
" bge r2,r6,1f\n"
@@ -127,7 +127,7 @@
".word 11b,13b\n"
".word 12b,13b\n"
".previous\n");
-EXPORT_SYMBOL(__copy_to_user);
+EXPORT_SYMBOL(raw_copy_to_user);
long strncpy_from_user(char *__to, const char __user *__from, long __len)
{
diff --git a/arch/openrisc/include/asm/Kbuild b/arch/openrisc/include/asm/Kbuild
index fb01873..df8e2f7 100644
--- a/arch/openrisc/include/asm/Kbuild
+++ b/arch/openrisc/include/asm/Kbuild
@@ -16,6 +16,7 @@
generic-y += emergency-restart.h
generic-y += errno.h
generic-y += exec.h
+generic-y += extable.h
generic-y += fb.h
generic-y += fcntl.h
generic-y += ftrace.h
diff --git a/arch/openrisc/include/asm/uaccess.h b/arch/openrisc/include/asm/uaccess.h
index 1311e6b..a557a7c 100644
--- a/arch/openrisc/include/asm/uaccess.h
+++ b/arch/openrisc/include/asm/uaccess.h
@@ -22,14 +22,10 @@
/*
* User space memory access functions
*/
-#include <linux/errno.h>
-#include <linux/thread_info.h>
#include <linux/prefetch.h>
#include <linux/string.h>
#include <asm/page.h>
-
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
+#include <asm/extable.h>
/*
* The fs value determines whether argument validity checking should be
@@ -66,23 +62,6 @@
__range_ok((unsigned long)addr, (unsigned long)size)
/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-
-struct exception_table_entry {
- unsigned long insn, fixup;
-};
-
-/*
* These are the main single-value transfer routines. They automatically
* use the right size if we just have the right pointer type.
*
@@ -257,34 +236,18 @@
extern unsigned long __must_check
__copy_tofrom_user(void *to, const void *from, unsigned long size);
-
-#define __copy_from_user(to, from, size) \
- __copy_tofrom_user(to, from, size)
-#define __copy_to_user(to, from, size) \
- __copy_tofrom_user(to, from, size)
-
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
-
static inline unsigned long
-copy_from_user(void *to, const void *from, unsigned long n)
+raw_copy_from_user(void *to, const void __user *from, unsigned long size)
{
- unsigned long res = n;
-
- if (likely(access_ok(VERIFY_READ, from, n)))
- res = __copy_tofrom_user(to, from, n);
- if (unlikely(res))
- memset(to + (n - res), 0, res);
- return res;
+ return __copy_tofrom_user(to, (__force const void *)from, size);
}
-
static inline unsigned long
-copy_to_user(void *to, const void *from, unsigned long n)
+raw_copy_to_user(void *to, const void __user *from, unsigned long size)
{
- if (likely(access_ok(VERIFY_WRITE, to, n)))
- n = __copy_tofrom_user(to, from, n);
- return n;
+ return __copy_tofrom_user((__force void *)to, from, size);
}
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
extern unsigned long __clear_user(void *addr, unsigned long size);
@@ -297,7 +260,7 @@
}
#define user_addr_max() \
- (segment_eq(get_fs(), USER_DS) ? TASK_SIZE : ~0UL)
+ (uaccess_kernel() ? ~0UL : TASK_SIZE)
extern long strncpy_from_user(char *dest, const char __user *src, long count);
diff --git a/arch/parisc/Kconfig b/arch/parisc/Kconfig
index ad294b3..531da9e 100644
--- a/arch/parisc/Kconfig
+++ b/arch/parisc/Kconfig
@@ -26,7 +26,6 @@
select SYSCTL_ARCH_UNALIGN_ALLOW
select SYSCTL_EXCEPTION_TRACE
select HAVE_MOD_ARCH_SPECIFIC
- select HAVE_ARCH_HARDENED_USERCOPY
select VIRT_TO_BUS
select MODULES_USE_ELF_RELA
select CLONE_BACKWARDS
diff --git a/arch/parisc/include/asm/bug.h b/arch/parisc/include/asm/bug.h
index 62a3333..d274227 100644
--- a/arch/parisc/include/asm/bug.h
+++ b/arch/parisc/include/asm/bug.h
@@ -46,7 +46,7 @@
#endif
#ifdef CONFIG_DEBUG_BUGVERBOSE
-#define __WARN_TAINT(taint) \
+#define __WARN_FLAGS(flags) \
do { \
asm volatile("\n" \
"1:\t" PARISC_BUG_BREAK_ASM "\n" \
@@ -56,11 +56,11 @@
"\t.org 2b+%c3\n" \
"\t.popsection" \
: : "i" (__FILE__), "i" (__LINE__), \
- "i" (BUGFLAG_TAINT(taint)), \
+ "i" (BUGFLAG_WARNING|(flags)), \
"i" (sizeof(struct bug_entry)) ); \
} while(0)
#else
-#define __WARN_TAINT(taint) \
+#define __WARN_FLAGS(flags) \
do { \
asm volatile("\n" \
"1:\t" PARISC_BUG_BREAK_ASM "\n" \
@@ -69,7 +69,7 @@
"\t.short %c0\n" \
"\t.org 2b+%c1\n" \
"\t.popsection" \
- : : "i" (BUGFLAG_TAINT(taint)), \
+ : : "i" (BUGFLAG_WARNING|(flags)), \
"i" (sizeof(struct bug_entry)) ); \
} while(0)
#endif
diff --git a/arch/parisc/include/asm/futex.h b/arch/parisc/include/asm/futex.h
index ac8bd58..0ba1430 100644
--- a/arch/parisc/include/asm/futex.h
+++ b/arch/parisc/include/asm/futex.h
@@ -109,7 +109,7 @@
/* futex.c wants to do a cmpxchg_inatomic on kernel NULL, which is
* our gateway page, and causes no end of trouble...
*/
- if (segment_eq(KERNEL_DS, get_fs()) && !uaddr)
+ if (uaccess_kernel() && !uaddr)
return -EFAULT;
if (!access_ok(VERIFY_WRITE, uaddr, sizeof(u32)))
diff --git a/arch/parisc/include/asm/uaccess.h b/arch/parisc/include/asm/uaccess.h
index cbd4f4a..6b113f3 100644
--- a/arch/parisc/include/asm/uaccess.h
+++ b/arch/parisc/include/asm/uaccess.h
@@ -6,15 +6,10 @@
*/
#include <asm/page.h>
#include <asm/cache.h>
-#include <asm/errno.h>
#include <asm-generic/uaccess-unaligned.h>
#include <linux/bug.h>
#include <linux/string.h>
-#include <linux/thread_info.h>
-
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
#define KERNEL_DS ((mm_segment_t){0})
#define USER_DS ((mm_segment_t){1})
@@ -216,9 +211,6 @@
* Complex access routines -- external declarations
*/
-extern unsigned long lcopy_to_user(void __user *, const void *, unsigned long);
-extern unsigned long lcopy_from_user(void *, const void __user *, unsigned long);
-extern unsigned long lcopy_in_user(void __user *, const void __user *, unsigned long);
extern long strncpy_from_user(char *, const char __user *, long);
extern unsigned lclear_user(void __user *, unsigned long);
extern long lstrnlen_user(const char __user *, long);
@@ -232,59 +224,14 @@
#define clear_user lclear_user
#define __clear_user lclear_user
-unsigned long __must_check __copy_to_user(void __user *dst, const void *src,
- unsigned long len);
-unsigned long __must_check __copy_from_user(void *dst, const void __user *src,
- unsigned long len);
-unsigned long copy_in_user(void __user *dst, const void __user *src,
- unsigned long len);
-#define __copy_in_user copy_in_user
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
-
-extern void __compiletime_error("usercopy buffer size is too small")
-__bad_copy_user(void);
-
-static inline void copy_user_overflow(int size, unsigned long count)
-{
- WARN(1, "Buffer overflow detected (%d < %lu)!\n", size, count);
-}
-
-static __always_inline unsigned long __must_check
-copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- int sz = __compiletime_object_size(to);
- unsigned long ret = n;
-
- if (likely(sz < 0 || sz >= n)) {
- check_object_size(to, n, false);
- ret = __copy_from_user(to, from, n);
- } else if (!__builtin_constant_p(n))
- copy_user_overflow(sz, n);
- else
- __bad_copy_user();
-
- if (unlikely(ret))
- memset(to + (n - ret), 0, ret);
-
- return ret;
-}
-
-static __always_inline unsigned long __must_check
-copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- int sz = __compiletime_object_size(from);
-
- if (likely(sz < 0 || sz >= n)) {
- check_object_size(from, n, true);
- n = __copy_to_user(to, from, n);
- } else if (!__builtin_constant_p(n))
- copy_user_overflow(sz, n);
- else
- __bad_copy_user();
-
- return n;
-}
+unsigned long __must_check raw_copy_to_user(void __user *dst, const void *src,
+ unsigned long len);
+unsigned long __must_check raw_copy_from_user(void *dst, const void __user *src,
+ unsigned long len);
+unsigned long __must_check raw_copy_in_user(void __user *dst, const void __user *src,
+ unsigned long len);
+#define INLINE_COPY_TO_USER
+#define INLINE_COPY_FROM_USER
struct pt_regs;
int fixup_exception(struct pt_regs *regs);
diff --git a/arch/parisc/include/uapi/asm/socket.h b/arch/parisc/include/uapi/asm/socket.h
index 7a109b7..5147018 100644
--- a/arch/parisc/include/uapi/asm/socket.h
+++ b/arch/parisc/include/uapi/asm/socket.h
@@ -91,4 +91,10 @@
#define SCM_TIMESTAMPING_OPT_STATS 0x402F
+#define SO_MEMINFO 0x4030
+
+#define SO_INCOMING_NAPI_ID 0x4031
+
+#define SO_COOKIE 0x4032
+
#endif /* _UAPI_ASM_SOCKET_H */
diff --git a/arch/parisc/lib/memcpy.c b/arch/parisc/lib/memcpy.c
index b3d47ec..99115cd 100644
--- a/arch/parisc/lib/memcpy.c
+++ b/arch/parisc/lib/memcpy.c
@@ -29,32 +29,32 @@
DECLARE_PER_CPU(struct exception_data, exception_data);
-#define get_user_space() (segment_eq(get_fs(), KERNEL_DS) ? 0 : mfsp(3))
+#define get_user_space() (uaccess_kernel() ? 0 : mfsp(3))
#define get_kernel_space() (0)
/* Returns 0 for success, otherwise, returns number of bytes not transferred. */
extern unsigned long pa_memcpy(void *dst, const void *src,
unsigned long len);
-unsigned long __copy_to_user(void __user *dst, const void *src,
- unsigned long len)
+unsigned long raw_copy_to_user(void __user *dst, const void *src,
+ unsigned long len)
{
mtsp(get_kernel_space(), 1);
mtsp(get_user_space(), 2);
return pa_memcpy((void __force *)dst, src, len);
}
-EXPORT_SYMBOL(__copy_to_user);
+EXPORT_SYMBOL(raw_copy_to_user);
-unsigned long __copy_from_user(void *dst, const void __user *src,
+unsigned long raw_copy_from_user(void *dst, const void __user *src,
unsigned long len)
{
mtsp(get_user_space(), 1);
mtsp(get_kernel_space(), 2);
return pa_memcpy(dst, (void __force *)src, len);
}
-EXPORT_SYMBOL(__copy_from_user);
+EXPORT_SYMBOL(raw_copy_from_user);
-unsigned long copy_in_user(void __user *dst, const void __user *src, unsigned long len)
+unsigned long raw_copy_in_user(void __user *dst, const void __user *src, unsigned long len)
{
mtsp(get_user_space(), 1);
mtsp(get_user_space(), 2);
@@ -70,7 +70,7 @@
return dst;
}
-EXPORT_SYMBOL(copy_in_user);
+EXPORT_SYMBOL(raw_copy_in_user);
EXPORT_SYMBOL(memcpy);
long probe_kernel_read(void *dst, const void *src, size_t size)
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 97a8bc8..0533826 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -117,7 +117,6 @@
select GENERIC_STRNLEN_USER
select GENERIC_TIME_VSYSCALL_OLD
select HAVE_ARCH_AUDITSYSCALL
- select HAVE_ARCH_HARDENED_USERCOPY
select HAVE_ARCH_JUMP_LABEL
select HAVE_ARCH_KGDB
select HAVE_ARCH_SECCOMP_FILTER
diff --git a/arch/powerpc/configs/85xx-hw.config b/arch/powerpc/configs/85xx-hw.config
index 528ff0e..c03d0fb 100644
--- a/arch/powerpc/configs/85xx-hw.config
+++ b/arch/powerpc/configs/85xx-hw.config
@@ -16,9 +16,8 @@
CONFIG_DMADEVICES=y
CONFIG_E1000E=y
CONFIG_E1000=y
-CONFIG_EDAC_MM_EDAC=y
-CONFIG_EDAC_MPC85XX=y
CONFIG_EDAC=y
+CONFIG_EDAC_MPC85XX=y
CONFIG_EEPROM_AT24=y
CONFIG_EEPROM_LEGACY=y
CONFIG_FB_FSL_DIU=y
diff --git a/arch/powerpc/configs/85xx/ge_imp3a_defconfig b/arch/powerpc/configs/85xx/ge_imp3a_defconfig
index c79283b..a917f7afb 100644
--- a/arch/powerpc/configs/85xx/ge_imp3a_defconfig
+++ b/arch/powerpc/configs/85xx/ge_imp3a_defconfig
@@ -155,7 +155,6 @@
CONFIG_USB_OHCI_HCD_PPC_OF_LE=y
CONFIG_USB_STORAGE=y
CONFIG_EDAC=y
-CONFIG_EDAC_MM_EDAC=y
CONFIG_EDAC_MPC85XX=y
CONFIG_RTC_CLASS=y
# CONFIG_RTC_INTF_PROC is not set
diff --git a/arch/powerpc/configs/85xx/xes_mpc85xx_defconfig b/arch/powerpc/configs/85xx/xes_mpc85xx_defconfig
index dbd961d..72900b8 100644
--- a/arch/powerpc/configs/85xx/xes_mpc85xx_defconfig
+++ b/arch/powerpc/configs/85xx/xes_mpc85xx_defconfig
@@ -116,7 +116,6 @@
CONFIG_LEDS_TRIGGER_TIMER=y
CONFIG_LEDS_TRIGGER_HEARTBEAT=y
CONFIG_EDAC=y
-CONFIG_EDAC_MM_EDAC=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_DS1307=y
CONFIG_RTC_DRV_CMOS=y
diff --git a/arch/powerpc/configs/cell_defconfig b/arch/powerpc/configs/cell_defconfig
index 2d7fcbe..aa56459 100644
--- a/arch/powerpc/configs/cell_defconfig
+++ b/arch/powerpc/configs/cell_defconfig
@@ -179,7 +179,6 @@
CONFIG_INFINIBAND_IPOIB=m
CONFIG_INFINIBAND_IPOIB_DEBUG_DATA=y
CONFIG_EDAC=y
-CONFIG_EDAC_MM_EDAC=y
CONFIG_EDAC_CELL=y
CONFIG_UIO=m
CONFIG_EXT2_FS=y
diff --git a/arch/powerpc/configs/pasemi_defconfig b/arch/powerpc/configs/pasemi_defconfig
index 5553c5c..fe43ff4 100644
--- a/arch/powerpc/configs/pasemi_defconfig
+++ b/arch/powerpc/configs/pasemi_defconfig
@@ -142,7 +142,6 @@
CONFIG_USB_SL811_HCD=y
CONFIG_USB_STORAGE=y
CONFIG_EDAC=y
-CONFIG_EDAC_MM_EDAC=y
CONFIG_EDAC_PASEMI=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_DS1307=y
diff --git a/arch/powerpc/configs/ppc64_defconfig b/arch/powerpc/configs/ppc64_defconfig
index 4f1288b..f2e03f0 100644
--- a/arch/powerpc/configs/ppc64_defconfig
+++ b/arch/powerpc/configs/ppc64_defconfig
@@ -262,7 +262,6 @@
CONFIG_INFINIBAND_SRP=m
CONFIG_INFINIBAND_ISER=m
CONFIG_EDAC=y
-CONFIG_EDAC_MM_EDAC=y
CONFIG_EDAC_PASEMI=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_DS1307=y
diff --git a/arch/powerpc/configs/ppc64e_defconfig b/arch/powerpc/configs/ppc64e_defconfig
index 11a3473..6340e6c 100644
--- a/arch/powerpc/configs/ppc64e_defconfig
+++ b/arch/powerpc/configs/ppc64e_defconfig
@@ -173,7 +173,6 @@
CONFIG_INFINIBAND_IPOIB=m
CONFIG_INFINIBAND_ISER=m
CONFIG_EDAC=y
-CONFIG_EDAC_MM_EDAC=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_DS1307=y
CONFIG_FS_DAX=y
diff --git a/arch/powerpc/configs/ppc6xx_defconfig b/arch/powerpc/configs/ppc6xx_defconfig
index 1d2d69d..18d0d60 100644
--- a/arch/powerpc/configs/ppc6xx_defconfig
+++ b/arch/powerpc/configs/ppc6xx_defconfig
@@ -988,8 +988,7 @@
CONFIG_LEDS_TRIGGER_DEFAULT_ON=m
CONFIG_ACCESSIBILITY=y
CONFIG_A11Y_BRAILLE_CONSOLE=y
-CONFIG_EDAC=y
-CONFIG_EDAC_MM_EDAC=m
+CONFIG_EDAC=m
CONFIG_RTC_CLASS=y
# CONFIG_RTC_HCTOSYS is not set
CONFIG_RTC_DRV_DS1307=m
diff --git a/arch/powerpc/crypto/Makefile b/arch/powerpc/crypto/Makefile
index 87f4045..67eca3a 100644
--- a/arch/powerpc/crypto/Makefile
+++ b/arch/powerpc/crypto/Makefile
@@ -10,6 +10,8 @@
obj-$(CONFIG_CRYPTO_SHA1_PPC_SPE) += sha1-ppc-spe.o
obj-$(CONFIG_CRYPTO_SHA256_PPC_SPE) += sha256-ppc-spe.o
obj-$(CONFIG_CRYPTO_CRC32C_VPMSUM) += crc32c-vpmsum.o
+obj-$(CONFIG_CRYPTO_CRCT10DIF_VPMSUM) += crct10dif-vpmsum.o
+obj-$(CONFIG_CRYPTO_VPMSUM_TESTER) += crc-vpmsum_test.o
aes-ppc-spe-y := aes-spe-core.o aes-spe-keys.o aes-tab-4k.o aes-spe-modes.o aes-spe-glue.o
md5-ppc-y := md5-asm.o md5-glue.o
@@ -17,3 +19,4 @@
sha1-ppc-spe-y := sha1-spe-asm.o sha1-spe-glue.o
sha256-ppc-spe-y := sha256-spe-asm.o sha256-spe-glue.o
crc32c-vpmsum-y := crc32c-vpmsum_asm.o crc32c-vpmsum_glue.o
+crct10dif-vpmsum-y := crct10dif-vpmsum_asm.o crct10dif-vpmsum_glue.o
diff --git a/arch/powerpc/crypto/crc-vpmsum_test.c b/arch/powerpc/crypto/crc-vpmsum_test.c
new file mode 100644
index 0000000..0153a9c
--- /dev/null
+++ b/arch/powerpc/crypto/crc-vpmsum_test.c
@@ -0,0 +1,137 @@
+/*
+ * CRC vpmsum tester
+ * Copyright 2017 Daniel Axtens, IBM Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/crc-t10dif.h>
+#include <linux/crc32.h>
+#include <crypto/internal/hash.h>
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/kernel.h>
+#include <linux/cpufeature.h>
+#include <asm/switch_to.h>
+
+static unsigned long iterations = 10000;
+
+#define MAX_CRC_LENGTH 65535
+
+
+static int __init crc_test_init(void)
+{
+ u16 crc16 = 0, verify16 = 0;
+ u32 crc32 = 0, verify32 = 0;
+ __le32 verify32le = 0;
+ unsigned char *data;
+ unsigned long i;
+ int ret;
+
+ struct crypto_shash *crct10dif_tfm;
+ struct crypto_shash *crc32c_tfm;
+
+ if (!cpu_has_feature(CPU_FTR_ARCH_207S))
+ return -ENODEV;
+
+ data = kmalloc(MAX_CRC_LENGTH, GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+
+ crct10dif_tfm = crypto_alloc_shash("crct10dif", 0, 0);
+
+ if (IS_ERR(crct10dif_tfm)) {
+ pr_err("Error allocating crc-t10dif\n");
+ goto free_buf;
+ }
+
+ crc32c_tfm = crypto_alloc_shash("crc32c", 0, 0);
+
+ if (IS_ERR(crc32c_tfm)) {
+ pr_err("Error allocating crc32c\n");
+ goto free_16;
+ }
+
+ do {
+ SHASH_DESC_ON_STACK(crct10dif_shash, crct10dif_tfm);
+ SHASH_DESC_ON_STACK(crc32c_shash, crc32c_tfm);
+
+ crct10dif_shash->tfm = crct10dif_tfm;
+ ret = crypto_shash_init(crct10dif_shash);
+
+ if (ret) {
+ pr_err("Error initing crc-t10dif\n");
+ goto free_32;
+ }
+
+
+ crc32c_shash->tfm = crc32c_tfm;
+ ret = crypto_shash_init(crc32c_shash);
+
+ if (ret) {
+ pr_err("Error initing crc32c\n");
+ goto free_32;
+ }
+
+ pr_info("crc-vpmsum_test begins, %lu iterations\n", iterations);
+ for (i=0; i<iterations; i++) {
+ size_t len, offset;
+
+ get_random_bytes(data, MAX_CRC_LENGTH);
+ get_random_bytes(&len, sizeof(len));
+ get_random_bytes(&offset, sizeof(offset));
+
+ len %= MAX_CRC_LENGTH;
+ offset &= 15;
+ if (len <= offset)
+ continue;
+ len -= offset;
+
+ crypto_shash_update(crct10dif_shash, data+offset, len);
+ crypto_shash_final(crct10dif_shash, (u8 *)(&crc16));
+ verify16 = crc_t10dif_generic(verify16, data+offset, len);
+
+
+ if (crc16 != verify16) {
+ pr_err("FAILURE in CRC16: got 0x%04x expected 0x%04x (len %lu)\n",
+ crc16, verify16, len);
+ break;
+ }
+
+ crypto_shash_update(crc32c_shash, data+offset, len);
+ crypto_shash_final(crc32c_shash, (u8 *)(&crc32));
+ verify32 = le32_to_cpu(verify32le);
+ verify32le = ~cpu_to_le32(__crc32c_le(~verify32, data+offset, len));
+ if (crc32 != (u32)verify32le) {
+ pr_err("FAILURE in CRC32: got 0x%08x expected 0x%08x (len %lu)\n",
+ crc32, verify32, len);
+ break;
+ }
+ }
+ pr_info("crc-vpmsum_test done, completed %lu iterations\n", i);
+ } while (0);
+
+free_32:
+ crypto_free_shash(crc32c_tfm);
+
+free_16:
+ crypto_free_shash(crct10dif_tfm);
+
+free_buf:
+ kfree(data);
+
+ return 0;
+}
+
+static void __exit crc_test_exit(void) {}
+
+module_init(crc_test_init);
+module_exit(crc_test_exit);
+module_param(iterations, long, 0400);
+
+MODULE_AUTHOR("Daniel Axtens <dja@axtens.net>");
+MODULE_DESCRIPTION("Vector polynomial multiply-sum CRC tester");
+MODULE_LICENSE("GPL");
diff --git a/arch/powerpc/crypto/crc32-vpmsum_core.S b/arch/powerpc/crypto/crc32-vpmsum_core.S
new file mode 100644
index 0000000..aadb59c
--- /dev/null
+++ b/arch/powerpc/crypto/crc32-vpmsum_core.S
@@ -0,0 +1,755 @@
+/*
+ * Core of the accelerated CRC algorithm.
+ * In your file, define the constants and CRC_FUNCTION_NAME
+ * Then include this file.
+ *
+ * Calculate the checksum of data that is 16 byte aligned and a multiple of
+ * 16 bytes.
+ *
+ * The first step is to reduce it to 1024 bits. We do this in 8 parallel
+ * chunks in order to mask the latency of the vpmsum instructions. If we
+ * have more than 32 kB of data to checksum we repeat this step multiple
+ * times, passing in the previous 1024 bits.
+ *
+ * The next step is to reduce the 1024 bits to 64 bits. This step adds
+ * 32 bits of 0s to the end - this matches what a CRC does. We just
+ * calculate constants that land the data in this 32 bits.
+ *
+ * We then use fixed point Barrett reduction to compute a mod n over GF(2)
+ * for n = CRC using POWER8 instructions. We use x = 32.
+ *
+ * http://en.wikipedia.org/wiki/Barrett_reduction
+ *
+ * Copyright (C) 2015 Anton Blanchard <anton@au.ibm.com>, IBM
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+*/
+
+#include <asm/ppc_asm.h>
+#include <asm/ppc-opcode.h>
+
+#define MAX_SIZE 32768
+
+ .text
+
+#if defined(__BIG_ENDIAN__) && defined(REFLECT)
+#define BYTESWAP_DATA
+#elif defined(__LITTLE_ENDIAN__) && !defined(REFLECT)
+#define BYTESWAP_DATA
+#else
+#undef BYTESWAP_DATA
+#endif
+
+#define off16 r25
+#define off32 r26
+#define off48 r27
+#define off64 r28
+#define off80 r29
+#define off96 r30
+#define off112 r31
+
+#define const1 v24
+#define const2 v25
+
+#define byteswap v26
+#define mask_32bit v27
+#define mask_64bit v28
+#define zeroes v29
+
+#ifdef BYTESWAP_DATA
+#define VPERM(A, B, C, D) vperm A, B, C, D
+#else
+#define VPERM(A, B, C, D)
+#endif
+
+/* unsigned int CRC_FUNCTION_NAME(unsigned int crc, void *p, unsigned long len) */
+FUNC_START(CRC_FUNCTION_NAME)
+ std r31,-8(r1)
+ std r30,-16(r1)
+ std r29,-24(r1)
+ std r28,-32(r1)
+ std r27,-40(r1)
+ std r26,-48(r1)
+ std r25,-56(r1)
+
+ li off16,16
+ li off32,32
+ li off48,48
+ li off64,64
+ li off80,80
+ li off96,96
+ li off112,112
+ li r0,0
+
+ /* Enough room for saving 10 non volatile VMX registers */
+ subi r6,r1,56+10*16
+ subi r7,r1,56+2*16
+
+ stvx v20,0,r6
+ stvx v21,off16,r6
+ stvx v22,off32,r6
+ stvx v23,off48,r6
+ stvx v24,off64,r6
+ stvx v25,off80,r6
+ stvx v26,off96,r6
+ stvx v27,off112,r6
+ stvx v28,0,r7
+ stvx v29,off16,r7
+
+ mr r10,r3
+
+ vxor zeroes,zeroes,zeroes
+ vspltisw v0,-1
+
+ vsldoi mask_32bit,zeroes,v0,4
+ vsldoi mask_64bit,zeroes,v0,8
+
+ /* Get the initial value into v8 */
+ vxor v8,v8,v8
+ MTVRD(v8, R3)
+#ifdef REFLECT
+ vsldoi v8,zeroes,v8,8 /* shift into bottom 32 bits */
+#else
+ vsldoi v8,v8,zeroes,4 /* shift into top 32 bits */
+#endif
+
+#ifdef BYTESWAP_DATA
+ addis r3,r2,.byteswap_constant@toc@ha
+ addi r3,r3,.byteswap_constant@toc@l
+
+ lvx byteswap,0,r3
+ addi r3,r3,16
+#endif
+
+ cmpdi r5,256
+ blt .Lshort
+
+ rldicr r6,r5,0,56
+
+ /* Checksum in blocks of MAX_SIZE */
+1: lis r7,MAX_SIZE@h
+ ori r7,r7,MAX_SIZE@l
+ mr r9,r7
+ cmpd r6,r7
+ bgt 2f
+ mr r7,r6
+2: subf r6,r7,r6
+
+ /* our main loop does 128 bytes at a time */
+ srdi r7,r7,7
+
+ /*
+ * Work out the offset into the constants table to start at. Each
+ * constant is 16 bytes, and it is used against 128 bytes of input
+ * data - 128 / 16 = 8
+ */
+ sldi r8,r7,4
+ srdi r9,r9,3
+ subf r8,r8,r9
+
+ /* We reduce our final 128 bytes in a separate step */
+ addi r7,r7,-1
+ mtctr r7
+
+ addis r3,r2,.constants@toc@ha
+ addi r3,r3,.constants@toc@l
+
+ /* Find the start of our constants */
+ add r3,r3,r8
+
+ /* zero v0-v7 which will contain our checksums */
+ vxor v0,v0,v0
+ vxor v1,v1,v1
+ vxor v2,v2,v2
+ vxor v3,v3,v3
+ vxor v4,v4,v4
+ vxor v5,v5,v5
+ vxor v6,v6,v6
+ vxor v7,v7,v7
+
+ lvx const1,0,r3
+
+ /*
+ * If we are looping back to consume more data we use the values
+ * already in v16-v23.
+ */
+ cmpdi r0,1
+ beq 2f
+
+ /* First warm up pass */
+ lvx v16,0,r4
+ lvx v17,off16,r4
+ VPERM(v16,v16,v16,byteswap)
+ VPERM(v17,v17,v17,byteswap)
+ lvx v18,off32,r4
+ lvx v19,off48,r4
+ VPERM(v18,v18,v18,byteswap)
+ VPERM(v19,v19,v19,byteswap)
+ lvx v20,off64,r4
+ lvx v21,off80,r4
+ VPERM(v20,v20,v20,byteswap)
+ VPERM(v21,v21,v21,byteswap)
+ lvx v22,off96,r4
+ lvx v23,off112,r4
+ VPERM(v22,v22,v22,byteswap)
+ VPERM(v23,v23,v23,byteswap)
+ addi r4,r4,8*16
+
+ /* xor in initial value */
+ vxor v16,v16,v8
+
+2: bdz .Lfirst_warm_up_done
+
+ addi r3,r3,16
+ lvx const2,0,r3
+
+ /* Second warm up pass */
+ VPMSUMD(v8,v16,const1)
+ lvx v16,0,r4
+ VPERM(v16,v16,v16,byteswap)
+ ori r2,r2,0
+
+ VPMSUMD(v9,v17,const1)
+ lvx v17,off16,r4
+ VPERM(v17,v17,v17,byteswap)
+ ori r2,r2,0
+
+ VPMSUMD(v10,v18,const1)
+ lvx v18,off32,r4
+ VPERM(v18,v18,v18,byteswap)
+ ori r2,r2,0
+
+ VPMSUMD(v11,v19,const1)
+ lvx v19,off48,r4
+ VPERM(v19,v19,v19,byteswap)
+ ori r2,r2,0
+
+ VPMSUMD(v12,v20,const1)
+ lvx v20,off64,r4
+ VPERM(v20,v20,v20,byteswap)
+ ori r2,r2,0
+
+ VPMSUMD(v13,v21,const1)
+ lvx v21,off80,r4
+ VPERM(v21,v21,v21,byteswap)
+ ori r2,r2,0
+
+ VPMSUMD(v14,v22,const1)
+ lvx v22,off96,r4
+ VPERM(v22,v22,v22,byteswap)
+ ori r2,r2,0
+
+ VPMSUMD(v15,v23,const1)
+ lvx v23,off112,r4
+ VPERM(v23,v23,v23,byteswap)
+
+ addi r4,r4,8*16
+
+ bdz .Lfirst_cool_down
+
+ /*
+ * main loop. We modulo schedule it such that it takes three iterations
+ * to complete - first iteration load, second iteration vpmsum, third
+ * iteration xor.
+ */
+ .balign 16
+4: lvx const1,0,r3
+ addi r3,r3,16
+ ori r2,r2,0
+
+ vxor v0,v0,v8
+ VPMSUMD(v8,v16,const2)
+ lvx v16,0,r4
+ VPERM(v16,v16,v16,byteswap)
+ ori r2,r2,0
+
+ vxor v1,v1,v9
+ VPMSUMD(v9,v17,const2)
+ lvx v17,off16,r4
+ VPERM(v17,v17,v17,byteswap)
+ ori r2,r2,0
+
+ vxor v2,v2,v10
+ VPMSUMD(v10,v18,const2)
+ lvx v18,off32,r4
+ VPERM(v18,v18,v18,byteswap)
+ ori r2,r2,0
+
+ vxor v3,v3,v11
+ VPMSUMD(v11,v19,const2)
+ lvx v19,off48,r4
+ VPERM(v19,v19,v19,byteswap)
+ lvx const2,0,r3
+ ori r2,r2,0
+
+ vxor v4,v4,v12
+ VPMSUMD(v12,v20,const1)
+ lvx v20,off64,r4
+ VPERM(v20,v20,v20,byteswap)
+ ori r2,r2,0
+
+ vxor v5,v5,v13
+ VPMSUMD(v13,v21,const1)
+ lvx v21,off80,r4
+ VPERM(v21,v21,v21,byteswap)
+ ori r2,r2,0
+
+ vxor v6,v6,v14
+ VPMSUMD(v14,v22,const1)
+ lvx v22,off96,r4
+ VPERM(v22,v22,v22,byteswap)
+ ori r2,r2,0
+
+ vxor v7,v7,v15
+ VPMSUMD(v15,v23,const1)
+ lvx v23,off112,r4
+ VPERM(v23,v23,v23,byteswap)
+
+ addi r4,r4,8*16
+
+ bdnz 4b
+
+.Lfirst_cool_down:
+ /* First cool down pass */
+ lvx const1,0,r3
+ addi r3,r3,16
+
+ vxor v0,v0,v8
+ VPMSUMD(v8,v16,const1)
+ ori r2,r2,0
+
+ vxor v1,v1,v9
+ VPMSUMD(v9,v17,const1)
+ ori r2,r2,0
+
+ vxor v2,v2,v10
+ VPMSUMD(v10,v18,const1)
+ ori r2,r2,0
+
+ vxor v3,v3,v11
+ VPMSUMD(v11,v19,const1)
+ ori r2,r2,0
+
+ vxor v4,v4,v12
+ VPMSUMD(v12,v20,const1)
+ ori r2,r2,0
+
+ vxor v5,v5,v13
+ VPMSUMD(v13,v21,const1)
+ ori r2,r2,0
+
+ vxor v6,v6,v14
+ VPMSUMD(v14,v22,const1)
+ ori r2,r2,0
+
+ vxor v7,v7,v15
+ VPMSUMD(v15,v23,const1)
+ ori r2,r2,0
+
+.Lsecond_cool_down:
+ /* Second cool down pass */
+ vxor v0,v0,v8
+ vxor v1,v1,v9
+ vxor v2,v2,v10
+ vxor v3,v3,v11
+ vxor v4,v4,v12
+ vxor v5,v5,v13
+ vxor v6,v6,v14
+ vxor v7,v7,v15
+
+#ifdef REFLECT
+ /*
+ * vpmsumd produces a 96 bit result in the least significant bits
+ * of the register. Since we are bit reflected we have to shift it
+ * left 32 bits so it occupies the least significant bits in the
+ * bit reflected domain.
+ */
+ vsldoi v0,v0,zeroes,4
+ vsldoi v1,v1,zeroes,4
+ vsldoi v2,v2,zeroes,4
+ vsldoi v3,v3,zeroes,4
+ vsldoi v4,v4,zeroes,4
+ vsldoi v5,v5,zeroes,4
+ vsldoi v6,v6,zeroes,4
+ vsldoi v7,v7,zeroes,4
+#endif
+
+ /* xor with last 1024 bits */
+ lvx v8,0,r4
+ lvx v9,off16,r4
+ VPERM(v8,v8,v8,byteswap)
+ VPERM(v9,v9,v9,byteswap)
+ lvx v10,off32,r4
+ lvx v11,off48,r4
+ VPERM(v10,v10,v10,byteswap)
+ VPERM(v11,v11,v11,byteswap)
+ lvx v12,off64,r4
+ lvx v13,off80,r4
+ VPERM(v12,v12,v12,byteswap)
+ VPERM(v13,v13,v13,byteswap)
+ lvx v14,off96,r4
+ lvx v15,off112,r4
+ VPERM(v14,v14,v14,byteswap)
+ VPERM(v15,v15,v15,byteswap)
+
+ addi r4,r4,8*16
+
+ vxor v16,v0,v8
+ vxor v17,v1,v9
+ vxor v18,v2,v10
+ vxor v19,v3,v11
+ vxor v20,v4,v12
+ vxor v21,v5,v13
+ vxor v22,v6,v14
+ vxor v23,v7,v15
+
+ li r0,1
+ cmpdi r6,0
+ addi r6,r6,128
+ bne 1b
+
+ /* Work out how many bytes we have left */
+ andi. r5,r5,127
+
+ /* Calculate where in the constant table we need to start */
+ subfic r6,r5,128
+ add r3,r3,r6
+
+ /* How many 16 byte chunks are in the tail */
+ srdi r7,r5,4
+ mtctr r7
+
+ /*
+ * Reduce the previously calculated 1024 bits to 64 bits, shifting
+ * 32 bits to include the trailing 32 bits of zeros
+ */
+ lvx v0,0,r3
+ lvx v1,off16,r3
+ lvx v2,off32,r3
+ lvx v3,off48,r3
+ lvx v4,off64,r3
+ lvx v5,off80,r3
+ lvx v6,off96,r3
+ lvx v7,off112,r3
+ addi r3,r3,8*16
+
+ VPMSUMW(v0,v16,v0)
+ VPMSUMW(v1,v17,v1)
+ VPMSUMW(v2,v18,v2)
+ VPMSUMW(v3,v19,v3)
+ VPMSUMW(v4,v20,v4)
+ VPMSUMW(v5,v21,v5)
+ VPMSUMW(v6,v22,v6)
+ VPMSUMW(v7,v23,v7)
+
+ /* Now reduce the tail (0 - 112 bytes) */
+ cmpdi r7,0
+ beq 1f
+
+ lvx v16,0,r4
+ lvx v17,0,r3
+ VPERM(v16,v16,v16,byteswap)
+ VPMSUMW(v16,v16,v17)
+ vxor v0,v0,v16
+ bdz 1f
+
+ lvx v16,off16,r4
+ lvx v17,off16,r3
+ VPERM(v16,v16,v16,byteswap)
+ VPMSUMW(v16,v16,v17)
+ vxor v0,v0,v16
+ bdz 1f
+
+ lvx v16,off32,r4
+ lvx v17,off32,r3
+ VPERM(v16,v16,v16,byteswap)
+ VPMSUMW(v16,v16,v17)
+ vxor v0,v0,v16
+ bdz 1f
+
+ lvx v16,off48,r4
+ lvx v17,off48,r3
+ VPERM(v16,v16,v16,byteswap)
+ VPMSUMW(v16,v16,v17)
+ vxor v0,v0,v16
+ bdz 1f
+
+ lvx v16,off64,r4
+ lvx v17,off64,r3
+ VPERM(v16,v16,v16,byteswap)
+ VPMSUMW(v16,v16,v17)
+ vxor v0,v0,v16
+ bdz 1f
+
+ lvx v16,off80,r4
+ lvx v17,off80,r3
+ VPERM(v16,v16,v16,byteswap)
+ VPMSUMW(v16,v16,v17)
+ vxor v0,v0,v16
+ bdz 1f
+
+ lvx v16,off96,r4
+ lvx v17,off96,r3
+ VPERM(v16,v16,v16,byteswap)
+ VPMSUMW(v16,v16,v17)
+ vxor v0,v0,v16
+
+ /* Now xor all the parallel chunks together */
+1: vxor v0,v0,v1
+ vxor v2,v2,v3
+ vxor v4,v4,v5
+ vxor v6,v6,v7
+
+ vxor v0,v0,v2
+ vxor v4,v4,v6
+
+ vxor v0,v0,v4
+
+.Lbarrett_reduction:
+ /* Barrett constants */
+ addis r3,r2,.barrett_constants@toc@ha
+ addi r3,r3,.barrett_constants@toc@l
+
+ lvx const1,0,r3
+ lvx const2,off16,r3
+
+ vsldoi v1,v0,v0,8
+ vxor v0,v0,v1 /* xor two 64 bit results together */
+
+#ifdef REFLECT
+ /* shift left one bit */
+ vspltisb v1,1
+ vsl v0,v0,v1
+#endif
+
+ vand v0,v0,mask_64bit
+#ifndef REFLECT
+ /*
+ * Now for the Barrett reduction algorithm. The idea is to calculate q,
+ * the multiple of our polynomial that we need to subtract. By
+ * doing the computation 2x bits higher (ie 64 bits) and shifting the
+ * result back down 2x bits, we round down to the nearest multiple.
+ */
+ VPMSUMD(v1,v0,const1) /* ma */
+ vsldoi v1,zeroes,v1,8 /* q = floor(ma/(2^64)) */
+ VPMSUMD(v1,v1,const2) /* qn */
+ vxor v0,v0,v1 /* a - qn, subtraction is xor in GF(2) */
+
+ /*
+ * Get the result into r3. We need to shift it left 8 bytes:
+ * V0 [ 0 1 2 X ]
+ * V0 [ 0 X 2 3 ]
+ */
+ vsldoi v0,v0,zeroes,8 /* shift result into top 64 bits */
+#else
+ /*
+ * The reflected version of Barrett reduction. Instead of bit
+ * reflecting our data (which is expensive to do), we bit reflect our
+ * constants and our algorithm, which means the intermediate data in
+ * our vector registers goes from 0-63 instead of 63-0. We can reflect
+ * the algorithm because we don't carry in mod 2 arithmetic.
+ */
+ vand v1,v0,mask_32bit /* bottom 32 bits of a */
+ VPMSUMD(v1,v1,const1) /* ma */
+ vand v1,v1,mask_32bit /* bottom 32bits of ma */
+ VPMSUMD(v1,v1,const2) /* qn */
+ vxor v0,v0,v1 /* a - qn, subtraction is xor in GF(2) */
+
+ /*
+ * Since we are bit reflected, the result (ie the low 32 bits) is in
+ * the high 32 bits. We just need to shift it left 4 bytes
+ * V0 [ 0 1 X 3 ]
+ * V0 [ 0 X 2 3 ]
+ */
+ vsldoi v0,v0,zeroes,4 /* shift result into top 64 bits of */
+#endif
+
+ /* Get it into r3 */
+ MFVRD(R3, v0)
+
+.Lout:
+ subi r6,r1,56+10*16
+ subi r7,r1,56+2*16
+
+ lvx v20,0,r6
+ lvx v21,off16,r6
+ lvx v22,off32,r6
+ lvx v23,off48,r6
+ lvx v24,off64,r6
+ lvx v25,off80,r6
+ lvx v26,off96,r6
+ lvx v27,off112,r6
+ lvx v28,0,r7
+ lvx v29,off16,r7
+
+ ld r31,-8(r1)
+ ld r30,-16(r1)
+ ld r29,-24(r1)
+ ld r28,-32(r1)
+ ld r27,-40(r1)
+ ld r26,-48(r1)
+ ld r25,-56(r1)
+
+ blr
+
+.Lfirst_warm_up_done:
+ lvx const1,0,r3
+ addi r3,r3,16
+
+ VPMSUMD(v8,v16,const1)
+ VPMSUMD(v9,v17,const1)
+ VPMSUMD(v10,v18,const1)
+ VPMSUMD(v11,v19,const1)
+ VPMSUMD(v12,v20,const1)
+ VPMSUMD(v13,v21,const1)
+ VPMSUMD(v14,v22,const1)
+ VPMSUMD(v15,v23,const1)
+
+ b .Lsecond_cool_down
+
+.Lshort:
+ cmpdi r5,0
+ beq .Lzero
+
+ addis r3,r2,.short_constants@toc@ha
+ addi r3,r3,.short_constants@toc@l
+
+ /* Calculate where in the constant table we need to start */
+ subfic r6,r5,256
+ add r3,r3,r6
+
+ /* How many 16 byte chunks? */
+ srdi r7,r5,4
+ mtctr r7
+
+ vxor v19,v19,v19
+ vxor v20,v20,v20
+
+ lvx v0,0,r4
+ lvx v16,0,r3
+ VPERM(v0,v0,v16,byteswap)
+ vxor v0,v0,v8 /* xor in initial value */
+ VPMSUMW(v0,v0,v16)
+ bdz .Lv0
+
+ lvx v1,off16,r4
+ lvx v17,off16,r3
+ VPERM(v1,v1,v17,byteswap)
+ VPMSUMW(v1,v1,v17)
+ bdz .Lv1
+
+ lvx v2,off32,r4
+ lvx v16,off32,r3
+ VPERM(v2,v2,v16,byteswap)
+ VPMSUMW(v2,v2,v16)
+ bdz .Lv2
+
+ lvx v3,off48,r4
+ lvx v17,off48,r3
+ VPERM(v3,v3,v17,byteswap)
+ VPMSUMW(v3,v3,v17)
+ bdz .Lv3
+
+ lvx v4,off64,r4
+ lvx v16,off64,r3
+ VPERM(v4,v4,v16,byteswap)
+ VPMSUMW(v4,v4,v16)
+ bdz .Lv4
+
+ lvx v5,off80,r4
+ lvx v17,off80,r3
+ VPERM(v5,v5,v17,byteswap)
+ VPMSUMW(v5,v5,v17)
+ bdz .Lv5
+
+ lvx v6,off96,r4
+ lvx v16,off96,r3
+ VPERM(v6,v6,v16,byteswap)
+ VPMSUMW(v6,v6,v16)
+ bdz .Lv6
+
+ lvx v7,off112,r4
+ lvx v17,off112,r3
+ VPERM(v7,v7,v17,byteswap)
+ VPMSUMW(v7,v7,v17)
+ bdz .Lv7
+
+ addi r3,r3,128
+ addi r4,r4,128
+
+ lvx v8,0,r4
+ lvx v16,0,r3
+ VPERM(v8,v8,v16,byteswap)
+ VPMSUMW(v8,v8,v16)
+ bdz .Lv8
+
+ lvx v9,off16,r4
+ lvx v17,off16,r3
+ VPERM(v9,v9,v17,byteswap)
+ VPMSUMW(v9,v9,v17)
+ bdz .Lv9
+
+ lvx v10,off32,r4
+ lvx v16,off32,r3
+ VPERM(v10,v10,v16,byteswap)
+ VPMSUMW(v10,v10,v16)
+ bdz .Lv10
+
+ lvx v11,off48,r4
+ lvx v17,off48,r3
+ VPERM(v11,v11,v17,byteswap)
+ VPMSUMW(v11,v11,v17)
+ bdz .Lv11
+
+ lvx v12,off64,r4
+ lvx v16,off64,r3
+ VPERM(v12,v12,v16,byteswap)
+ VPMSUMW(v12,v12,v16)
+ bdz .Lv12
+
+ lvx v13,off80,r4
+ lvx v17,off80,r3
+ VPERM(v13,v13,v17,byteswap)
+ VPMSUMW(v13,v13,v17)
+ bdz .Lv13
+
+ lvx v14,off96,r4
+ lvx v16,off96,r3
+ VPERM(v14,v14,v16,byteswap)
+ VPMSUMW(v14,v14,v16)
+ bdz .Lv14
+
+ lvx v15,off112,r4
+ lvx v17,off112,r3
+ VPERM(v15,v15,v17,byteswap)
+ VPMSUMW(v15,v15,v17)
+
+.Lv15: vxor v19,v19,v15
+.Lv14: vxor v20,v20,v14
+.Lv13: vxor v19,v19,v13
+.Lv12: vxor v20,v20,v12
+.Lv11: vxor v19,v19,v11
+.Lv10: vxor v20,v20,v10
+.Lv9: vxor v19,v19,v9
+.Lv8: vxor v20,v20,v8
+.Lv7: vxor v19,v19,v7
+.Lv6: vxor v20,v20,v6
+.Lv5: vxor v19,v19,v5
+.Lv4: vxor v20,v20,v4
+.Lv3: vxor v19,v19,v3
+.Lv2: vxor v20,v20,v2
+.Lv1: vxor v19,v19,v1
+.Lv0: vxor v20,v20,v0
+
+ vxor v0,v19,v20
+
+ b .Lbarrett_reduction
+
+.Lzero:
+ mr r3,r10
+ b .Lout
+
+FUNC_END(CRC_FUNCTION_NAME)
diff --git a/arch/powerpc/crypto/crc32c-vpmsum_asm.S b/arch/powerpc/crypto/crc32c-vpmsum_asm.S
index dc640b2..d2bea48 100644
--- a/arch/powerpc/crypto/crc32c-vpmsum_asm.S
+++ b/arch/powerpc/crypto/crc32c-vpmsum_asm.S
@@ -1,20 +1,5 @@
/*
- * Calculate the checksum of data that is 16 byte aligned and a multiple of
- * 16 bytes.
- *
- * The first step is to reduce it to 1024 bits. We do this in 8 parallel
- * chunks in order to mask the latency of the vpmsum instructions. If we
- * have more than 32 kB of data to checksum we repeat this step multiple
- * times, passing in the previous 1024 bits.
- *
- * The next step is to reduce the 1024 bits to 64 bits. This step adds
- * 32 bits of 0s to the end - this matches what a CRC does. We just
- * calculate constants that land the data in this 32 bits.
- *
- * We then use fixed point Barrett reduction to compute a mod n over GF(2)
- * for n = CRC using POWER8 instructions. We use x = 32.
- *
- * http://en.wikipedia.org/wiki/Barrett_reduction
+ * Calculate a crc32c with vpmsum acceleration
*
* Copyright (C) 2015 Anton Blanchard <anton@au.ibm.com>, IBM
*
@@ -23,9 +8,6 @@
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*/
-#include <asm/ppc_asm.h>
-#include <asm/ppc-opcode.h>
-
.section .rodata
.balign 16
@@ -33,7 +15,6 @@
/* byte reverse permute constant */
.octa 0x0F0E0D0C0B0A09080706050403020100
-#define MAX_SIZE 32768
.constants:
/* Reduce 262144 kbits to 1024 bits */
@@ -860,694 +841,6 @@
/* 33 bit reflected Barrett constant n */
.octa 0x00000000000000000000000105ec76f1
- .text
-
-#if defined(__BIG_ENDIAN__)
-#define BYTESWAP_DATA
-#else
-#undef BYTESWAP_DATA
-#endif
-
-#define off16 r25
-#define off32 r26
-#define off48 r27
-#define off64 r28
-#define off80 r29
-#define off96 r30
-#define off112 r31
-
-#define const1 v24
-#define const2 v25
-
-#define byteswap v26
-#define mask_32bit v27
-#define mask_64bit v28
-#define zeroes v29
-
-#ifdef BYTESWAP_DATA
-#define VPERM(A, B, C, D) vperm A, B, C, D
-#else
-#define VPERM(A, B, C, D)
-#endif
-
-/* unsigned int __crc32c_vpmsum(unsigned int crc, void *p, unsigned long len) */
-FUNC_START(__crc32c_vpmsum)
- std r31,-8(r1)
- std r30,-16(r1)
- std r29,-24(r1)
- std r28,-32(r1)
- std r27,-40(r1)
- std r26,-48(r1)
- std r25,-56(r1)
-
- li off16,16
- li off32,32
- li off48,48
- li off64,64
- li off80,80
- li off96,96
- li off112,112
- li r0,0
-
- /* Enough room for saving 10 non volatile VMX registers */
- subi r6,r1,56+10*16
- subi r7,r1,56+2*16
-
- stvx v20,0,r6
- stvx v21,off16,r6
- stvx v22,off32,r6
- stvx v23,off48,r6
- stvx v24,off64,r6
- stvx v25,off80,r6
- stvx v26,off96,r6
- stvx v27,off112,r6
- stvx v28,0,r7
- stvx v29,off16,r7
-
- mr r10,r3
-
- vxor zeroes,zeroes,zeroes
- vspltisw v0,-1
-
- vsldoi mask_32bit,zeroes,v0,4
- vsldoi mask_64bit,zeroes,v0,8
-
- /* Get the initial value into v8 */
- vxor v8,v8,v8
- MTVRD(v8, R3)
- vsldoi v8,zeroes,v8,8 /* shift into bottom 32 bits */
-
-#ifdef BYTESWAP_DATA
- addis r3,r2,.byteswap_constant@toc@ha
- addi r3,r3,.byteswap_constant@toc@l
-
- lvx byteswap,0,r3
- addi r3,r3,16
-#endif
-
- cmpdi r5,256
- blt .Lshort
-
- rldicr r6,r5,0,56
-
- /* Checksum in blocks of MAX_SIZE */
-1: lis r7,MAX_SIZE@h
- ori r7,r7,MAX_SIZE@l
- mr r9,r7
- cmpd r6,r7
- bgt 2f
- mr r7,r6
-2: subf r6,r7,r6
-
- /* our main loop does 128 bytes at a time */
- srdi r7,r7,7
-
- /*
- * Work out the offset into the constants table to start at. Each
- * constant is 16 bytes, and it is used against 128 bytes of input
- * data - 128 / 16 = 8
- */
- sldi r8,r7,4
- srdi r9,r9,3
- subf r8,r8,r9
-
- /* We reduce our final 128 bytes in a separate step */
- addi r7,r7,-1
- mtctr r7
-
- addis r3,r2,.constants@toc@ha
- addi r3,r3,.constants@toc@l
-
- /* Find the start of our constants */
- add r3,r3,r8
-
- /* zero v0-v7 which will contain our checksums */
- vxor v0,v0,v0
- vxor v1,v1,v1
- vxor v2,v2,v2
- vxor v3,v3,v3
- vxor v4,v4,v4
- vxor v5,v5,v5
- vxor v6,v6,v6
- vxor v7,v7,v7
-
- lvx const1,0,r3
-
- /*
- * If we are looping back to consume more data we use the values
- * already in v16-v23.
- */
- cmpdi r0,1
- beq 2f
-
- /* First warm up pass */
- lvx v16,0,r4
- lvx v17,off16,r4
- VPERM(v16,v16,v16,byteswap)
- VPERM(v17,v17,v17,byteswap)
- lvx v18,off32,r4
- lvx v19,off48,r4
- VPERM(v18,v18,v18,byteswap)
- VPERM(v19,v19,v19,byteswap)
- lvx v20,off64,r4
- lvx v21,off80,r4
- VPERM(v20,v20,v20,byteswap)
- VPERM(v21,v21,v21,byteswap)
- lvx v22,off96,r4
- lvx v23,off112,r4
- VPERM(v22,v22,v22,byteswap)
- VPERM(v23,v23,v23,byteswap)
- addi r4,r4,8*16
-
- /* xor in initial value */
- vxor v16,v16,v8
-
-2: bdz .Lfirst_warm_up_done
-
- addi r3,r3,16
- lvx const2,0,r3
-
- /* Second warm up pass */
- VPMSUMD(v8,v16,const1)
- lvx v16,0,r4
- VPERM(v16,v16,v16,byteswap)
- ori r2,r2,0
-
- VPMSUMD(v9,v17,const1)
- lvx v17,off16,r4
- VPERM(v17,v17,v17,byteswap)
- ori r2,r2,0
-
- VPMSUMD(v10,v18,const1)
- lvx v18,off32,r4
- VPERM(v18,v18,v18,byteswap)
- ori r2,r2,0
-
- VPMSUMD(v11,v19,const1)
- lvx v19,off48,r4
- VPERM(v19,v19,v19,byteswap)
- ori r2,r2,0
-
- VPMSUMD(v12,v20,const1)
- lvx v20,off64,r4
- VPERM(v20,v20,v20,byteswap)
- ori r2,r2,0
-
- VPMSUMD(v13,v21,const1)
- lvx v21,off80,r4
- VPERM(v21,v21,v21,byteswap)
- ori r2,r2,0
-
- VPMSUMD(v14,v22,const1)
- lvx v22,off96,r4
- VPERM(v22,v22,v22,byteswap)
- ori r2,r2,0
-
- VPMSUMD(v15,v23,const1)
- lvx v23,off112,r4
- VPERM(v23,v23,v23,byteswap)
-
- addi r4,r4,8*16
-
- bdz .Lfirst_cool_down
-
- /*
- * main loop. We modulo schedule it such that it takes three iterations
- * to complete - first iteration load, second iteration vpmsum, third
- * iteration xor.
- */
- .balign 16
-4: lvx const1,0,r3
- addi r3,r3,16
- ori r2,r2,0
-
- vxor v0,v0,v8
- VPMSUMD(v8,v16,const2)
- lvx v16,0,r4
- VPERM(v16,v16,v16,byteswap)
- ori r2,r2,0
-
- vxor v1,v1,v9
- VPMSUMD(v9,v17,const2)
- lvx v17,off16,r4
- VPERM(v17,v17,v17,byteswap)
- ori r2,r2,0
-
- vxor v2,v2,v10
- VPMSUMD(v10,v18,const2)
- lvx v18,off32,r4
- VPERM(v18,v18,v18,byteswap)
- ori r2,r2,0
-
- vxor v3,v3,v11
- VPMSUMD(v11,v19,const2)
- lvx v19,off48,r4
- VPERM(v19,v19,v19,byteswap)
- lvx const2,0,r3
- ori r2,r2,0
-
- vxor v4,v4,v12
- VPMSUMD(v12,v20,const1)
- lvx v20,off64,r4
- VPERM(v20,v20,v20,byteswap)
- ori r2,r2,0
-
- vxor v5,v5,v13
- VPMSUMD(v13,v21,const1)
- lvx v21,off80,r4
- VPERM(v21,v21,v21,byteswap)
- ori r2,r2,0
-
- vxor v6,v6,v14
- VPMSUMD(v14,v22,const1)
- lvx v22,off96,r4
- VPERM(v22,v22,v22,byteswap)
- ori r2,r2,0
-
- vxor v7,v7,v15
- VPMSUMD(v15,v23,const1)
- lvx v23,off112,r4
- VPERM(v23,v23,v23,byteswap)
-
- addi r4,r4,8*16
-
- bdnz 4b
-
-.Lfirst_cool_down:
- /* First cool down pass */
- lvx const1,0,r3
- addi r3,r3,16
-
- vxor v0,v0,v8
- VPMSUMD(v8,v16,const1)
- ori r2,r2,0
-
- vxor v1,v1,v9
- VPMSUMD(v9,v17,const1)
- ori r2,r2,0
-
- vxor v2,v2,v10
- VPMSUMD(v10,v18,const1)
- ori r2,r2,0
-
- vxor v3,v3,v11
- VPMSUMD(v11,v19,const1)
- ori r2,r2,0
-
- vxor v4,v4,v12
- VPMSUMD(v12,v20,const1)
- ori r2,r2,0
-
- vxor v5,v5,v13
- VPMSUMD(v13,v21,const1)
- ori r2,r2,0
-
- vxor v6,v6,v14
- VPMSUMD(v14,v22,const1)
- ori r2,r2,0
-
- vxor v7,v7,v15
- VPMSUMD(v15,v23,const1)
- ori r2,r2,0
-
-.Lsecond_cool_down:
- /* Second cool down pass */
- vxor v0,v0,v8
- vxor v1,v1,v9
- vxor v2,v2,v10
- vxor v3,v3,v11
- vxor v4,v4,v12
- vxor v5,v5,v13
- vxor v6,v6,v14
- vxor v7,v7,v15
-
- /*
- * vpmsumd produces a 96 bit result in the least significant bits
- * of the register. Since we are bit reflected we have to shift it
- * left 32 bits so it occupies the least significant bits in the
- * bit reflected domain.
- */
- vsldoi v0,v0,zeroes,4
- vsldoi v1,v1,zeroes,4
- vsldoi v2,v2,zeroes,4
- vsldoi v3,v3,zeroes,4
- vsldoi v4,v4,zeroes,4
- vsldoi v5,v5,zeroes,4
- vsldoi v6,v6,zeroes,4
- vsldoi v7,v7,zeroes,4
-
- /* xor with last 1024 bits */
- lvx v8,0,r4
- lvx v9,off16,r4
- VPERM(v8,v8,v8,byteswap)
- VPERM(v9,v9,v9,byteswap)
- lvx v10,off32,r4
- lvx v11,off48,r4
- VPERM(v10,v10,v10,byteswap)
- VPERM(v11,v11,v11,byteswap)
- lvx v12,off64,r4
- lvx v13,off80,r4
- VPERM(v12,v12,v12,byteswap)
- VPERM(v13,v13,v13,byteswap)
- lvx v14,off96,r4
- lvx v15,off112,r4
- VPERM(v14,v14,v14,byteswap)
- VPERM(v15,v15,v15,byteswap)
-
- addi r4,r4,8*16
-
- vxor v16,v0,v8
- vxor v17,v1,v9
- vxor v18,v2,v10
- vxor v19,v3,v11
- vxor v20,v4,v12
- vxor v21,v5,v13
- vxor v22,v6,v14
- vxor v23,v7,v15
-
- li r0,1
- cmpdi r6,0
- addi r6,r6,128
- bne 1b
-
- /* Work out how many bytes we have left */
- andi. r5,r5,127
-
- /* Calculate where in the constant table we need to start */
- subfic r6,r5,128
- add r3,r3,r6
-
- /* How many 16 byte chunks are in the tail */
- srdi r7,r5,4
- mtctr r7
-
- /*
- * Reduce the previously calculated 1024 bits to 64 bits, shifting
- * 32 bits to include the trailing 32 bits of zeros
- */
- lvx v0,0,r3
- lvx v1,off16,r3
- lvx v2,off32,r3
- lvx v3,off48,r3
- lvx v4,off64,r3
- lvx v5,off80,r3
- lvx v6,off96,r3
- lvx v7,off112,r3
- addi r3,r3,8*16
-
- VPMSUMW(v0,v16,v0)
- VPMSUMW(v1,v17,v1)
- VPMSUMW(v2,v18,v2)
- VPMSUMW(v3,v19,v3)
- VPMSUMW(v4,v20,v4)
- VPMSUMW(v5,v21,v5)
- VPMSUMW(v6,v22,v6)
- VPMSUMW(v7,v23,v7)
-
- /* Now reduce the tail (0 - 112 bytes) */
- cmpdi r7,0
- beq 1f
-
- lvx v16,0,r4
- lvx v17,0,r3
- VPERM(v16,v16,v16,byteswap)
- VPMSUMW(v16,v16,v17)
- vxor v0,v0,v16
- bdz 1f
-
- lvx v16,off16,r4
- lvx v17,off16,r3
- VPERM(v16,v16,v16,byteswap)
- VPMSUMW(v16,v16,v17)
- vxor v0,v0,v16
- bdz 1f
-
- lvx v16,off32,r4
- lvx v17,off32,r3
- VPERM(v16,v16,v16,byteswap)
- VPMSUMW(v16,v16,v17)
- vxor v0,v0,v16
- bdz 1f
-
- lvx v16,off48,r4
- lvx v17,off48,r3
- VPERM(v16,v16,v16,byteswap)
- VPMSUMW(v16,v16,v17)
- vxor v0,v0,v16
- bdz 1f
-
- lvx v16,off64,r4
- lvx v17,off64,r3
- VPERM(v16,v16,v16,byteswap)
- VPMSUMW(v16,v16,v17)
- vxor v0,v0,v16
- bdz 1f
-
- lvx v16,off80,r4
- lvx v17,off80,r3
- VPERM(v16,v16,v16,byteswap)
- VPMSUMW(v16,v16,v17)
- vxor v0,v0,v16
- bdz 1f
-
- lvx v16,off96,r4
- lvx v17,off96,r3
- VPERM(v16,v16,v16,byteswap)
- VPMSUMW(v16,v16,v17)
- vxor v0,v0,v16
-
- /* Now xor all the parallel chunks together */
-1: vxor v0,v0,v1
- vxor v2,v2,v3
- vxor v4,v4,v5
- vxor v6,v6,v7
-
- vxor v0,v0,v2
- vxor v4,v4,v6
-
- vxor v0,v0,v4
-
-.Lbarrett_reduction:
- /* Barrett constants */
- addis r3,r2,.barrett_constants@toc@ha
- addi r3,r3,.barrett_constants@toc@l
-
- lvx const1,0,r3
- lvx const2,off16,r3
-
- vsldoi v1,v0,v0,8
- vxor v0,v0,v1 /* xor two 64 bit results together */
-
- /* shift left one bit */
- vspltisb v1,1
- vsl v0,v0,v1
-
- vand v0,v0,mask_64bit
-
- /*
- * The reflected version of Barrett reduction. Instead of bit
- * reflecting our data (which is expensive to do), we bit reflect our
- * constants and our algorithm, which means the intermediate data in
- * our vector registers goes from 0-63 instead of 63-0. We can reflect
- * the algorithm because we don't carry in mod 2 arithmetic.
- */
- vand v1,v0,mask_32bit /* bottom 32 bits of a */
- VPMSUMD(v1,v1,const1) /* ma */
- vand v1,v1,mask_32bit /* bottom 32bits of ma */
- VPMSUMD(v1,v1,const2) /* qn */
- vxor v0,v0,v1 /* a - qn, subtraction is xor in GF(2) */
-
- /*
- * Since we are bit reflected, the result (ie the low 32 bits) is in
- * the high 32 bits. We just need to shift it left 4 bytes
- * V0 [ 0 1 X 3 ]
- * V0 [ 0 X 2 3 ]
- */
- vsldoi v0,v0,zeroes,4 /* shift result into top 64 bits of */
-
- /* Get it into r3 */
- MFVRD(R3, v0)
-
-.Lout:
- subi r6,r1,56+10*16
- subi r7,r1,56+2*16
-
- lvx v20,0,r6
- lvx v21,off16,r6
- lvx v22,off32,r6
- lvx v23,off48,r6
- lvx v24,off64,r6
- lvx v25,off80,r6
- lvx v26,off96,r6
- lvx v27,off112,r6
- lvx v28,0,r7
- lvx v29,off16,r7
-
- ld r31,-8(r1)
- ld r30,-16(r1)
- ld r29,-24(r1)
- ld r28,-32(r1)
- ld r27,-40(r1)
- ld r26,-48(r1)
- ld r25,-56(r1)
-
- blr
-
-.Lfirst_warm_up_done:
- lvx const1,0,r3
- addi r3,r3,16
-
- VPMSUMD(v8,v16,const1)
- VPMSUMD(v9,v17,const1)
- VPMSUMD(v10,v18,const1)
- VPMSUMD(v11,v19,const1)
- VPMSUMD(v12,v20,const1)
- VPMSUMD(v13,v21,const1)
- VPMSUMD(v14,v22,const1)
- VPMSUMD(v15,v23,const1)
-
- b .Lsecond_cool_down
-
-.Lshort:
- cmpdi r5,0
- beq .Lzero
-
- addis r3,r2,.short_constants@toc@ha
- addi r3,r3,.short_constants@toc@l
-
- /* Calculate where in the constant table we need to start */
- subfic r6,r5,256
- add r3,r3,r6
-
- /* How many 16 byte chunks? */
- srdi r7,r5,4
- mtctr r7
-
- vxor v19,v19,v19
- vxor v20,v20,v20
-
- lvx v0,0,r4
- lvx v16,0,r3
- VPERM(v0,v0,v16,byteswap)
- vxor v0,v0,v8 /* xor in initial value */
- VPMSUMW(v0,v0,v16)
- bdz .Lv0
-
- lvx v1,off16,r4
- lvx v17,off16,r3
- VPERM(v1,v1,v17,byteswap)
- VPMSUMW(v1,v1,v17)
- bdz .Lv1
-
- lvx v2,off32,r4
- lvx v16,off32,r3
- VPERM(v2,v2,v16,byteswap)
- VPMSUMW(v2,v2,v16)
- bdz .Lv2
-
- lvx v3,off48,r4
- lvx v17,off48,r3
- VPERM(v3,v3,v17,byteswap)
- VPMSUMW(v3,v3,v17)
- bdz .Lv3
-
- lvx v4,off64,r4
- lvx v16,off64,r3
- VPERM(v4,v4,v16,byteswap)
- VPMSUMW(v4,v4,v16)
- bdz .Lv4
-
- lvx v5,off80,r4
- lvx v17,off80,r3
- VPERM(v5,v5,v17,byteswap)
- VPMSUMW(v5,v5,v17)
- bdz .Lv5
-
- lvx v6,off96,r4
- lvx v16,off96,r3
- VPERM(v6,v6,v16,byteswap)
- VPMSUMW(v6,v6,v16)
- bdz .Lv6
-
- lvx v7,off112,r4
- lvx v17,off112,r3
- VPERM(v7,v7,v17,byteswap)
- VPMSUMW(v7,v7,v17)
- bdz .Lv7
-
- addi r3,r3,128
- addi r4,r4,128
-
- lvx v8,0,r4
- lvx v16,0,r3
- VPERM(v8,v8,v16,byteswap)
- VPMSUMW(v8,v8,v16)
- bdz .Lv8
-
- lvx v9,off16,r4
- lvx v17,off16,r3
- VPERM(v9,v9,v17,byteswap)
- VPMSUMW(v9,v9,v17)
- bdz .Lv9
-
- lvx v10,off32,r4
- lvx v16,off32,r3
- VPERM(v10,v10,v16,byteswap)
- VPMSUMW(v10,v10,v16)
- bdz .Lv10
-
- lvx v11,off48,r4
- lvx v17,off48,r3
- VPERM(v11,v11,v17,byteswap)
- VPMSUMW(v11,v11,v17)
- bdz .Lv11
-
- lvx v12,off64,r4
- lvx v16,off64,r3
- VPERM(v12,v12,v16,byteswap)
- VPMSUMW(v12,v12,v16)
- bdz .Lv12
-
- lvx v13,off80,r4
- lvx v17,off80,r3
- VPERM(v13,v13,v17,byteswap)
- VPMSUMW(v13,v13,v17)
- bdz .Lv13
-
- lvx v14,off96,r4
- lvx v16,off96,r3
- VPERM(v14,v14,v16,byteswap)
- VPMSUMW(v14,v14,v16)
- bdz .Lv14
-
- lvx v15,off112,r4
- lvx v17,off112,r3
- VPERM(v15,v15,v17,byteswap)
- VPMSUMW(v15,v15,v17)
-
-.Lv15: vxor v19,v19,v15
-.Lv14: vxor v20,v20,v14
-.Lv13: vxor v19,v19,v13
-.Lv12: vxor v20,v20,v12
-.Lv11: vxor v19,v19,v11
-.Lv10: vxor v20,v20,v10
-.Lv9: vxor v19,v19,v9
-.Lv8: vxor v20,v20,v8
-.Lv7: vxor v19,v19,v7
-.Lv6: vxor v20,v20,v6
-.Lv5: vxor v19,v19,v5
-.Lv4: vxor v20,v20,v4
-.Lv3: vxor v19,v19,v3
-.Lv2: vxor v20,v20,v2
-.Lv1: vxor v19,v19,v1
-.Lv0: vxor v20,v20,v0
-
- vxor v0,v19,v20
-
- b .Lbarrett_reduction
-
-.Lzero:
- mr r3,r10
- b .Lout
-
-FUNC_END(__crc32_vpmsum)
+#define CRC_FUNCTION_NAME __crc32c_vpmsum
+#define REFLECT
+#include "crc32-vpmsum_core.S"
diff --git a/arch/powerpc/crypto/crct10dif-vpmsum_asm.S b/arch/powerpc/crypto/crct10dif-vpmsum_asm.S
new file mode 100644
index 0000000..5e3d81a
--- /dev/null
+++ b/arch/powerpc/crypto/crct10dif-vpmsum_asm.S
@@ -0,0 +1,850 @@
+/*
+ * Calculate a CRC T10DIF with vpmsum acceleration
+ *
+ * Constants generated by crc32-vpmsum, available at
+ * https://github.com/antonblanchard/crc32-vpmsum
+ *
+ * crc32-vpmsum is
+ * Copyright (C) 2015 Anton Blanchard <anton@au.ibm.com>, IBM
+ * and is available under the GPL v2 or later.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+ .section .rodata
+.balign 16
+
+.byteswap_constant:
+ /* byte reverse permute constant */
+ .octa 0x0F0E0D0C0B0A09080706050403020100
+
+.constants:
+
+ /* Reduce 262144 kbits to 1024 bits */
+ /* x^261184 mod p(x), x^261120 mod p(x) */
+ .octa 0x0000000056d300000000000052550000
+
+ /* x^260160 mod p(x), x^260096 mod p(x) */
+ .octa 0x00000000ee67000000000000a1e40000
+
+ /* x^259136 mod p(x), x^259072 mod p(x) */
+ .octa 0x0000000060830000000000004ad10000
+
+ /* x^258112 mod p(x), x^258048 mod p(x) */
+ .octa 0x000000008cfe0000000000009ab40000
+
+ /* x^257088 mod p(x), x^257024 mod p(x) */
+ .octa 0x000000003e93000000000000fdb50000
+
+ /* x^256064 mod p(x), x^256000 mod p(x) */
+ .octa 0x000000003c2000000000000045480000
+
+ /* x^255040 mod p(x), x^254976 mod p(x) */
+ .octa 0x00000000b1fc0000000000008d690000
+
+ /* x^254016 mod p(x), x^253952 mod p(x) */
+ .octa 0x00000000f82b00000000000024ad0000
+
+ /* x^252992 mod p(x), x^252928 mod p(x) */
+ .octa 0x0000000044420000000000009f1a0000
+
+ /* x^251968 mod p(x), x^251904 mod p(x) */
+ .octa 0x00000000e88c00000000000066ec0000
+
+ /* x^250944 mod p(x), x^250880 mod p(x) */
+ .octa 0x00000000385c000000000000c87d0000
+
+ /* x^249920 mod p(x), x^249856 mod p(x) */
+ .octa 0x000000003227000000000000c8ff0000
+
+ /* x^248896 mod p(x), x^248832 mod p(x) */
+ .octa 0x00000000a9a900000000000033440000
+
+ /* x^247872 mod p(x), x^247808 mod p(x) */
+ .octa 0x00000000abaa00000000000066eb0000
+
+ /* x^246848 mod p(x), x^246784 mod p(x) */
+ .octa 0x000000001ac3000000000000c4ef0000
+
+ /* x^245824 mod p(x), x^245760 mod p(x) */
+ .octa 0x0000000063f000000000000056f30000
+
+ /* x^244800 mod p(x), x^244736 mod p(x) */
+ .octa 0x0000000032cc00000000000002050000
+
+ /* x^243776 mod p(x), x^243712 mod p(x) */
+ .octa 0x00000000f8b5000000000000568e0000
+
+ /* x^242752 mod p(x), x^242688 mod p(x) */
+ .octa 0x000000008db100000000000064290000
+
+ /* x^241728 mod p(x), x^241664 mod p(x) */
+ .octa 0x0000000059ca0000000000006b660000
+
+ /* x^240704 mod p(x), x^240640 mod p(x) */
+ .octa 0x000000005f5c00000000000018f80000
+
+ /* x^239680 mod p(x), x^239616 mod p(x) */
+ .octa 0x0000000061af000000000000b6090000
+
+ /* x^238656 mod p(x), x^238592 mod p(x) */
+ .octa 0x00000000e29e000000000000099a0000
+
+ /* x^237632 mod p(x), x^237568 mod p(x) */
+ .octa 0x000000000975000000000000a8360000
+
+ /* x^236608 mod p(x), x^236544 mod p(x) */
+ .octa 0x0000000043900000000000004f570000
+
+ /* x^235584 mod p(x), x^235520 mod p(x) */
+ .octa 0x00000000f9cd000000000000134c0000
+
+ /* x^234560 mod p(x), x^234496 mod p(x) */
+ .octa 0x000000007c29000000000000ec380000
+
+ /* x^233536 mod p(x), x^233472 mod p(x) */
+ .octa 0x000000004c6a000000000000b0d10000
+
+ /* x^232512 mod p(x), x^232448 mod p(x) */
+ .octa 0x00000000e7290000000000007d3e0000
+
+ /* x^231488 mod p(x), x^231424 mod p(x) */
+ .octa 0x00000000f1ab000000000000f0b20000
+
+ /* x^230464 mod p(x), x^230400 mod p(x) */
+ .octa 0x0000000039db0000000000009c270000
+
+ /* x^229440 mod p(x), x^229376 mod p(x) */
+ .octa 0x000000005e2800000000000092890000
+
+ /* x^228416 mod p(x), x^228352 mod p(x) */
+ .octa 0x00000000d44e000000000000d5ee0000
+
+ /* x^227392 mod p(x), x^227328 mod p(x) */
+ .octa 0x00000000cd0a00000000000041f50000
+
+ /* x^226368 mod p(x), x^226304 mod p(x) */
+ .octa 0x00000000c5b400000000000010520000
+
+ /* x^225344 mod p(x), x^225280 mod p(x) */
+ .octa 0x00000000fd2100000000000042170000
+
+ /* x^224320 mod p(x), x^224256 mod p(x) */
+ .octa 0x000000002f2500000000000095c20000
+
+ /* x^223296 mod p(x), x^223232 mod p(x) */
+ .octa 0x000000001b0100000000000001ce0000
+
+ /* x^222272 mod p(x), x^222208 mod p(x) */
+ .octa 0x000000000d430000000000002aca0000
+
+ /* x^221248 mod p(x), x^221184 mod p(x) */
+ .octa 0x0000000030a6000000000000385e0000
+
+ /* x^220224 mod p(x), x^220160 mod p(x) */
+ .octa 0x00000000e37b0000000000006f7a0000
+
+ /* x^219200 mod p(x), x^219136 mod p(x) */
+ .octa 0x00000000873600000000000024320000
+
+ /* x^218176 mod p(x), x^218112 mod p(x) */
+ .octa 0x00000000e9fb000000000000bd9c0000
+
+ /* x^217152 mod p(x), x^217088 mod p(x) */
+ .octa 0x000000003b9500000000000054bc0000
+
+ /* x^216128 mod p(x), x^216064 mod p(x) */
+ .octa 0x00000000133e000000000000a4660000
+
+ /* x^215104 mod p(x), x^215040 mod p(x) */
+ .octa 0x00000000784500000000000079930000
+
+ /* x^214080 mod p(x), x^214016 mod p(x) */
+ .octa 0x00000000b9800000000000001bb80000
+
+ /* x^213056 mod p(x), x^212992 mod p(x) */
+ .octa 0x00000000687600000000000024400000
+
+ /* x^212032 mod p(x), x^211968 mod p(x) */
+ .octa 0x00000000aff300000000000029e10000
+
+ /* x^211008 mod p(x), x^210944 mod p(x) */
+ .octa 0x0000000024b50000000000005ded0000
+
+ /* x^209984 mod p(x), x^209920 mod p(x) */
+ .octa 0x0000000017e8000000000000b12e0000
+
+ /* x^208960 mod p(x), x^208896 mod p(x) */
+ .octa 0x00000000128400000000000026d20000
+
+ /* x^207936 mod p(x), x^207872 mod p(x) */
+ .octa 0x000000002115000000000000a32a0000
+
+ /* x^206912 mod p(x), x^206848 mod p(x) */
+ .octa 0x000000009595000000000000a1210000
+
+ /* x^205888 mod p(x), x^205824 mod p(x) */
+ .octa 0x00000000281e000000000000ee8b0000
+
+ /* x^204864 mod p(x), x^204800 mod p(x) */
+ .octa 0x0000000006010000000000003d0d0000
+
+ /* x^203840 mod p(x), x^203776 mod p(x) */
+ .octa 0x00000000e2b600000000000034e90000
+
+ /* x^202816 mod p(x), x^202752 mod p(x) */
+ .octa 0x000000001bd40000000000004cdb0000
+
+ /* x^201792 mod p(x), x^201728 mod p(x) */
+ .octa 0x00000000df2800000000000030e90000
+
+ /* x^200768 mod p(x), x^200704 mod p(x) */
+ .octa 0x0000000049c200000000000042590000
+
+ /* x^199744 mod p(x), x^199680 mod p(x) */
+ .octa 0x000000009b97000000000000df950000
+
+ /* x^198720 mod p(x), x^198656 mod p(x) */
+ .octa 0x000000006184000000000000da7b0000
+
+ /* x^197696 mod p(x), x^197632 mod p(x) */
+ .octa 0x00000000461700000000000012510000
+
+ /* x^196672 mod p(x), x^196608 mod p(x) */
+ .octa 0x000000009b40000000000000f37e0000
+
+ /* x^195648 mod p(x), x^195584 mod p(x) */
+ .octa 0x00000000eeb2000000000000ecf10000
+
+ /* x^194624 mod p(x), x^194560 mod p(x) */
+ .octa 0x00000000b2e800000000000050f20000
+
+ /* x^193600 mod p(x), x^193536 mod p(x) */
+ .octa 0x00000000f59a000000000000e0b30000
+
+ /* x^192576 mod p(x), x^192512 mod p(x) */
+ .octa 0x00000000467f0000000000004d5a0000
+
+ /* x^191552 mod p(x), x^191488 mod p(x) */
+ .octa 0x00000000da92000000000000bb010000
+
+ /* x^190528 mod p(x), x^190464 mod p(x) */
+ .octa 0x000000001e1000000000000022a40000
+
+ /* x^189504 mod p(x), x^189440 mod p(x) */
+ .octa 0x0000000058fe000000000000836f0000
+
+ /* x^188480 mod p(x), x^188416 mod p(x) */
+ .octa 0x00000000b9ce000000000000d78d0000
+
+ /* x^187456 mod p(x), x^187392 mod p(x) */
+ .octa 0x0000000022210000000000004f8d0000
+
+ /* x^186432 mod p(x), x^186368 mod p(x) */
+ .octa 0x00000000744600000000000033760000
+
+ /* x^185408 mod p(x), x^185344 mod p(x) */
+ .octa 0x000000001c2e000000000000a1e50000
+
+ /* x^184384 mod p(x), x^184320 mod p(x) */
+ .octa 0x00000000dcc8000000000000a1a40000
+
+ /* x^183360 mod p(x), x^183296 mod p(x) */
+ .octa 0x00000000910f00000000000019a20000
+
+ /* x^182336 mod p(x), x^182272 mod p(x) */
+ .octa 0x0000000055d5000000000000f6ae0000
+
+ /* x^181312 mod p(x), x^181248 mod p(x) */
+ .octa 0x00000000c8ba000000000000a7ac0000
+
+ /* x^180288 mod p(x), x^180224 mod p(x) */
+ .octa 0x0000000031f8000000000000eea20000
+
+ /* x^179264 mod p(x), x^179200 mod p(x) */
+ .octa 0x000000001966000000000000c4d90000
+
+ /* x^178240 mod p(x), x^178176 mod p(x) */
+ .octa 0x00000000b9810000000000002b470000
+
+ /* x^177216 mod p(x), x^177152 mod p(x) */
+ .octa 0x000000008303000000000000f7cf0000
+
+ /* x^176192 mod p(x), x^176128 mod p(x) */
+ .octa 0x000000002ce500000000000035b30000
+
+ /* x^175168 mod p(x), x^175104 mod p(x) */
+ .octa 0x000000002fae0000000000000c7c0000
+
+ /* x^174144 mod p(x), x^174080 mod p(x) */
+ .octa 0x00000000f50c0000000000009edf0000
+
+ /* x^173120 mod p(x), x^173056 mod p(x) */
+ .octa 0x00000000714f00000000000004cd0000
+
+ /* x^172096 mod p(x), x^172032 mod p(x) */
+ .octa 0x00000000c161000000000000541b0000
+
+ /* x^171072 mod p(x), x^171008 mod p(x) */
+ .octa 0x0000000021c8000000000000e2700000
+
+ /* x^170048 mod p(x), x^169984 mod p(x) */
+ .octa 0x00000000b93d00000000000009a60000
+
+ /* x^169024 mod p(x), x^168960 mod p(x) */
+ .octa 0x00000000fbcf000000000000761c0000
+
+ /* x^168000 mod p(x), x^167936 mod p(x) */
+ .octa 0x0000000026350000000000009db30000
+
+ /* x^166976 mod p(x), x^166912 mod p(x) */
+ .octa 0x00000000b64f0000000000003e9f0000
+
+ /* x^165952 mod p(x), x^165888 mod p(x) */
+ .octa 0x00000000bd0e00000000000078590000
+
+ /* x^164928 mod p(x), x^164864 mod p(x) */
+ .octa 0x00000000d9360000000000008bc80000
+
+ /* x^163904 mod p(x), x^163840 mod p(x) */
+ .octa 0x000000002f140000000000008c9f0000
+
+ /* x^162880 mod p(x), x^162816 mod p(x) */
+ .octa 0x000000006a270000000000006af70000
+
+ /* x^161856 mod p(x), x^161792 mod p(x) */
+ .octa 0x000000006685000000000000e5210000
+
+ /* x^160832 mod p(x), x^160768 mod p(x) */
+ .octa 0x0000000062da00000000000008290000
+
+ /* x^159808 mod p(x), x^159744 mod p(x) */
+ .octa 0x00000000bb4b000000000000e4d00000
+
+ /* x^158784 mod p(x), x^158720 mod p(x) */
+ .octa 0x00000000d2490000000000004ae10000
+
+ /* x^157760 mod p(x), x^157696 mod p(x) */
+ .octa 0x00000000c85b00000000000000e70000
+
+ /* x^156736 mod p(x), x^156672 mod p(x) */
+ .octa 0x00000000c37a00000000000015650000
+
+ /* x^155712 mod p(x), x^155648 mod p(x) */
+ .octa 0x0000000018530000000000001c2f0000
+
+ /* x^154688 mod p(x), x^154624 mod p(x) */
+ .octa 0x00000000b46600000000000037bd0000
+
+ /* x^153664 mod p(x), x^153600 mod p(x) */
+ .octa 0x00000000439b00000000000012190000
+
+ /* x^152640 mod p(x), x^152576 mod p(x) */
+ .octa 0x00000000b1260000000000005ece0000
+
+ /* x^151616 mod p(x), x^151552 mod p(x) */
+ .octa 0x00000000d8110000000000002a5e0000
+
+ /* x^150592 mod p(x), x^150528 mod p(x) */
+ .octa 0x00000000099f00000000000052330000
+
+ /* x^149568 mod p(x), x^149504 mod p(x) */
+ .octa 0x00000000f9f9000000000000f9120000
+
+ /* x^148544 mod p(x), x^148480 mod p(x) */
+ .octa 0x000000005cc00000000000000ddc0000
+
+ /* x^147520 mod p(x), x^147456 mod p(x) */
+ .octa 0x00000000343b00000000000012200000
+
+ /* x^146496 mod p(x), x^146432 mod p(x) */
+ .octa 0x000000009222000000000000d12b0000
+
+ /* x^145472 mod p(x), x^145408 mod p(x) */
+ .octa 0x00000000d781000000000000eb2d0000
+
+ /* x^144448 mod p(x), x^144384 mod p(x) */
+ .octa 0x000000000bf400000000000058970000
+
+ /* x^143424 mod p(x), x^143360 mod p(x) */
+ .octa 0x00000000094200000000000013690000
+
+ /* x^142400 mod p(x), x^142336 mod p(x) */
+ .octa 0x00000000d55100000000000051950000
+
+ /* x^141376 mod p(x), x^141312 mod p(x) */
+ .octa 0x000000008f11000000000000954b0000
+
+ /* x^140352 mod p(x), x^140288 mod p(x) */
+ .octa 0x00000000140f000000000000b29e0000
+
+ /* x^139328 mod p(x), x^139264 mod p(x) */
+ .octa 0x00000000c6db000000000000db5d0000
+
+ /* x^138304 mod p(x), x^138240 mod p(x) */
+ .octa 0x00000000715b000000000000dfaf0000
+
+ /* x^137280 mod p(x), x^137216 mod p(x) */
+ .octa 0x000000000dea000000000000e3b60000
+
+ /* x^136256 mod p(x), x^136192 mod p(x) */
+ .octa 0x000000006f94000000000000ddaf0000
+
+ /* x^135232 mod p(x), x^135168 mod p(x) */
+ .octa 0x0000000024e1000000000000e4f70000
+
+ /* x^134208 mod p(x), x^134144 mod p(x) */
+ .octa 0x000000008810000000000000aa110000
+
+ /* x^133184 mod p(x), x^133120 mod p(x) */
+ .octa 0x0000000030c2000000000000a8e60000
+
+ /* x^132160 mod p(x), x^132096 mod p(x) */
+ .octa 0x00000000e6d0000000000000ccf30000
+
+ /* x^131136 mod p(x), x^131072 mod p(x) */
+ .octa 0x000000004da000000000000079bf0000
+
+ /* x^130112 mod p(x), x^130048 mod p(x) */
+ .octa 0x000000007759000000000000b3a30000
+
+ /* x^129088 mod p(x), x^129024 mod p(x) */
+ .octa 0x00000000597400000000000028790000
+
+ /* x^128064 mod p(x), x^128000 mod p(x) */
+ .octa 0x000000007acd000000000000b5820000
+
+ /* x^127040 mod p(x), x^126976 mod p(x) */
+ .octa 0x00000000e6e400000000000026ad0000
+
+ /* x^126016 mod p(x), x^125952 mod p(x) */
+ .octa 0x000000006d49000000000000985b0000
+
+ /* x^124992 mod p(x), x^124928 mod p(x) */
+ .octa 0x000000000f0800000000000011520000
+
+ /* x^123968 mod p(x), x^123904 mod p(x) */
+ .octa 0x000000002c7f000000000000846c0000
+
+ /* x^122944 mod p(x), x^122880 mod p(x) */
+ .octa 0x000000005ce7000000000000ae1d0000
+
+ /* x^121920 mod p(x), x^121856 mod p(x) */
+ .octa 0x00000000d4cb000000000000e21d0000
+
+ /* x^120896 mod p(x), x^120832 mod p(x) */
+ .octa 0x000000003a2300000000000019bb0000
+
+ /* x^119872 mod p(x), x^119808 mod p(x) */
+ .octa 0x000000000e1700000000000095290000
+
+ /* x^118848 mod p(x), x^118784 mod p(x) */
+ .octa 0x000000006e6400000000000050d20000
+
+ /* x^117824 mod p(x), x^117760 mod p(x) */
+ .octa 0x000000008d5c0000000000000cd10000
+
+ /* x^116800 mod p(x), x^116736 mod p(x) */
+ .octa 0x00000000ef310000000000007b570000
+
+ /* x^115776 mod p(x), x^115712 mod p(x) */
+ .octa 0x00000000645d00000000000053d60000
+
+ /* x^114752 mod p(x), x^114688 mod p(x) */
+ .octa 0x0000000018fc00000000000077510000
+
+ /* x^113728 mod p(x), x^113664 mod p(x) */
+ .octa 0x000000000cb3000000000000a7b70000
+
+ /* x^112704 mod p(x), x^112640 mod p(x) */
+ .octa 0x00000000991b000000000000d0780000
+
+ /* x^111680 mod p(x), x^111616 mod p(x) */
+ .octa 0x00000000845a000000000000be3c0000
+
+ /* x^110656 mod p(x), x^110592 mod p(x) */
+ .octa 0x00000000d3a9000000000000df020000
+
+ /* x^109632 mod p(x), x^109568 mod p(x) */
+ .octa 0x0000000017d7000000000000063e0000
+
+ /* x^108608 mod p(x), x^108544 mod p(x) */
+ .octa 0x000000007a860000000000008ab40000
+
+ /* x^107584 mod p(x), x^107520 mod p(x) */
+ .octa 0x00000000fd7c000000000000c7bd0000
+
+ /* x^106560 mod p(x), x^106496 mod p(x) */
+ .octa 0x00000000a56b000000000000efd60000
+
+ /* x^105536 mod p(x), x^105472 mod p(x) */
+ .octa 0x0000000010e400000000000071380000
+
+ /* x^104512 mod p(x), x^104448 mod p(x) */
+ .octa 0x00000000994500000000000004d30000
+
+ /* x^103488 mod p(x), x^103424 mod p(x) */
+ .octa 0x00000000b83c0000000000003b0e0000
+
+ /* x^102464 mod p(x), x^102400 mod p(x) */
+ .octa 0x00000000d6c10000000000008b020000
+
+ /* x^101440 mod p(x), x^101376 mod p(x) */
+ .octa 0x000000009efc000000000000da940000
+
+ /* x^100416 mod p(x), x^100352 mod p(x) */
+ .octa 0x000000005e87000000000000f9f70000
+
+ /* x^99392 mod p(x), x^99328 mod p(x) */
+ .octa 0x000000006c9b00000000000045e40000
+
+ /* x^98368 mod p(x), x^98304 mod p(x) */
+ .octa 0x00000000178a00000000000083940000
+
+ /* x^97344 mod p(x), x^97280 mod p(x) */
+ .octa 0x00000000f0c8000000000000f0a00000
+
+ /* x^96320 mod p(x), x^96256 mod p(x) */
+ .octa 0x00000000f699000000000000b74b0000
+
+ /* x^95296 mod p(x), x^95232 mod p(x) */
+ .octa 0x00000000316d000000000000c1cf0000
+
+ /* x^94272 mod p(x), x^94208 mod p(x) */
+ .octa 0x00000000987e00000000000072680000
+
+ /* x^93248 mod p(x), x^93184 mod p(x) */
+ .octa 0x00000000acff000000000000e0ab0000
+
+ /* x^92224 mod p(x), x^92160 mod p(x) */
+ .octa 0x00000000a1f6000000000000c5a80000
+
+ /* x^91200 mod p(x), x^91136 mod p(x) */
+ .octa 0x0000000061bd000000000000cf690000
+
+ /* x^90176 mod p(x), x^90112 mod p(x) */
+ .octa 0x00000000c9f2000000000000cbcc0000
+
+ /* x^89152 mod p(x), x^89088 mod p(x) */
+ .octa 0x000000005a33000000000000de050000
+
+ /* x^88128 mod p(x), x^88064 mod p(x) */
+ .octa 0x00000000e416000000000000ccd70000
+
+ /* x^87104 mod p(x), x^87040 mod p(x) */
+ .octa 0x0000000058930000000000002f670000
+
+ /* x^86080 mod p(x), x^86016 mod p(x) */
+ .octa 0x00000000a9d3000000000000152f0000
+
+ /* x^85056 mod p(x), x^84992 mod p(x) */
+ .octa 0x00000000c114000000000000ecc20000
+
+ /* x^84032 mod p(x), x^83968 mod p(x) */
+ .octa 0x00000000b9270000000000007c890000
+
+ /* x^83008 mod p(x), x^82944 mod p(x) */
+ .octa 0x000000002e6000000000000006ee0000
+
+ /* x^81984 mod p(x), x^81920 mod p(x) */
+ .octa 0x00000000dfc600000000000009100000
+
+ /* x^80960 mod p(x), x^80896 mod p(x) */
+ .octa 0x000000004911000000000000ad4e0000
+
+ /* x^79936 mod p(x), x^79872 mod p(x) */
+ .octa 0x00000000ae1b000000000000b04d0000
+
+ /* x^78912 mod p(x), x^78848 mod p(x) */
+ .octa 0x0000000005fa000000000000e9900000
+
+ /* x^77888 mod p(x), x^77824 mod p(x) */
+ .octa 0x0000000004a1000000000000cc6f0000
+
+ /* x^76864 mod p(x), x^76800 mod p(x) */
+ .octa 0x00000000af73000000000000ed110000
+
+ /* x^75840 mod p(x), x^75776 mod p(x) */
+ .octa 0x0000000082530000000000008f7e0000
+
+ /* x^74816 mod p(x), x^74752 mod p(x) */
+ .octa 0x00000000cfdc000000000000594f0000
+
+ /* x^73792 mod p(x), x^73728 mod p(x) */
+ .octa 0x00000000a6b6000000000000a8750000
+
+ /* x^72768 mod p(x), x^72704 mod p(x) */
+ .octa 0x00000000fd76000000000000aa0c0000
+
+ /* x^71744 mod p(x), x^71680 mod p(x) */
+ .octa 0x0000000006f500000000000071db0000
+
+ /* x^70720 mod p(x), x^70656 mod p(x) */
+ .octa 0x0000000037ca000000000000ab0c0000
+
+ /* x^69696 mod p(x), x^69632 mod p(x) */
+ .octa 0x00000000d7ab000000000000b7a00000
+
+ /* x^68672 mod p(x), x^68608 mod p(x) */
+ .octa 0x00000000440800000000000090d30000
+
+ /* x^67648 mod p(x), x^67584 mod p(x) */
+ .octa 0x00000000186100000000000054730000
+
+ /* x^66624 mod p(x), x^66560 mod p(x) */
+ .octa 0x000000007368000000000000a3a20000
+
+ /* x^65600 mod p(x), x^65536 mod p(x) */
+ .octa 0x0000000026d0000000000000f9040000
+
+ /* x^64576 mod p(x), x^64512 mod p(x) */
+ .octa 0x00000000fe770000000000009c0a0000
+
+ /* x^63552 mod p(x), x^63488 mod p(x) */
+ .octa 0x000000002cba000000000000d1e70000
+
+ /* x^62528 mod p(x), x^62464 mod p(x) */
+ .octa 0x00000000f8bd0000000000005ac10000
+
+ /* x^61504 mod p(x), x^61440 mod p(x) */
+ .octa 0x000000007372000000000000d68d0000
+
+ /* x^60480 mod p(x), x^60416 mod p(x) */
+ .octa 0x00000000f37f00000000000089f60000
+
+ /* x^59456 mod p(x), x^59392 mod p(x) */
+ .octa 0x00000000078400000000000008a90000
+
+ /* x^58432 mod p(x), x^58368 mod p(x) */
+ .octa 0x00000000d3e400000000000042360000
+
+ /* x^57408 mod p(x), x^57344 mod p(x) */
+ .octa 0x00000000eba800000000000092d50000
+
+ /* x^56384 mod p(x), x^56320 mod p(x) */
+ .octa 0x00000000afbe000000000000b4d50000
+
+ /* x^55360 mod p(x), x^55296 mod p(x) */
+ .octa 0x00000000d8ca000000000000c9060000
+
+ /* x^54336 mod p(x), x^54272 mod p(x) */
+ .octa 0x00000000c2d00000000000008f4f0000
+
+ /* x^53312 mod p(x), x^53248 mod p(x) */
+ .octa 0x00000000373200000000000028690000
+
+ /* x^52288 mod p(x), x^52224 mod p(x) */
+ .octa 0x0000000046ae000000000000c3b30000
+
+ /* x^51264 mod p(x), x^51200 mod p(x) */
+ .octa 0x00000000b243000000000000f8700000
+
+ /* x^50240 mod p(x), x^50176 mod p(x) */
+ .octa 0x00000000f7f500000000000029eb0000
+
+ /* x^49216 mod p(x), x^49152 mod p(x) */
+ .octa 0x000000000c7e000000000000fe730000
+
+ /* x^48192 mod p(x), x^48128 mod p(x) */
+ .octa 0x00000000c38200000000000096000000
+
+ /* x^47168 mod p(x), x^47104 mod p(x) */
+ .octa 0x000000008956000000000000683c0000
+
+ /* x^46144 mod p(x), x^46080 mod p(x) */
+ .octa 0x00000000422d0000000000005f1e0000
+
+ /* x^45120 mod p(x), x^45056 mod p(x) */
+ .octa 0x00000000ac0f0000000000006f810000
+
+ /* x^44096 mod p(x), x^44032 mod p(x) */
+ .octa 0x00000000ce30000000000000031f0000
+
+ /* x^43072 mod p(x), x^43008 mod p(x) */
+ .octa 0x000000003d43000000000000455a0000
+
+ /* x^42048 mod p(x), x^41984 mod p(x) */
+ .octa 0x000000007ebe000000000000a6050000
+
+ /* x^41024 mod p(x), x^40960 mod p(x) */
+ .octa 0x00000000976e00000000000077eb0000
+
+ /* x^40000 mod p(x), x^39936 mod p(x) */
+ .octa 0x000000000872000000000000389c0000
+
+ /* x^38976 mod p(x), x^38912 mod p(x) */
+ .octa 0x000000008979000000000000c7b20000
+
+ /* x^37952 mod p(x), x^37888 mod p(x) */
+ .octa 0x000000005c1e0000000000001d870000
+
+ /* x^36928 mod p(x), x^36864 mod p(x) */
+ .octa 0x00000000aebb00000000000045810000
+
+ /* x^35904 mod p(x), x^35840 mod p(x) */
+ .octa 0x000000004f7e0000000000006d4a0000
+
+ /* x^34880 mod p(x), x^34816 mod p(x) */
+ .octa 0x00000000ea98000000000000b9200000
+
+ /* x^33856 mod p(x), x^33792 mod p(x) */
+ .octa 0x00000000f39600000000000022f20000
+
+ /* x^32832 mod p(x), x^32768 mod p(x) */
+ .octa 0x000000000bc500000000000041ca0000
+
+ /* x^31808 mod p(x), x^31744 mod p(x) */
+ .octa 0x00000000786400000000000078500000
+
+ /* x^30784 mod p(x), x^30720 mod p(x) */
+ .octa 0x00000000be970000000000009e7e0000
+
+ /* x^29760 mod p(x), x^29696 mod p(x) */
+ .octa 0x00000000dd6d000000000000a53c0000
+
+ /* x^28736 mod p(x), x^28672 mod p(x) */
+ .octa 0x000000004c3f00000000000039340000
+
+ /* x^27712 mod p(x), x^27648 mod p(x) */
+ .octa 0x0000000093a4000000000000b58e0000
+
+ /* x^26688 mod p(x), x^26624 mod p(x) */
+ .octa 0x0000000050fb00000000000062d40000
+
+ /* x^25664 mod p(x), x^25600 mod p(x) */
+ .octa 0x00000000f505000000000000a26f0000
+
+ /* x^24640 mod p(x), x^24576 mod p(x) */
+ .octa 0x0000000064f900000000000065e60000
+
+ /* x^23616 mod p(x), x^23552 mod p(x) */
+ .octa 0x00000000e8c2000000000000aad90000
+
+ /* x^22592 mod p(x), x^22528 mod p(x) */
+ .octa 0x00000000720b000000000000a3b00000
+
+ /* x^21568 mod p(x), x^21504 mod p(x) */
+ .octa 0x00000000e992000000000000d2680000
+
+ /* x^20544 mod p(x), x^20480 mod p(x) */
+ .octa 0x000000009132000000000000cf4c0000
+
+ /* x^19520 mod p(x), x^19456 mod p(x) */
+ .octa 0x00000000608a00000000000076610000
+
+ /* x^18496 mod p(x), x^18432 mod p(x) */
+ .octa 0x000000009948000000000000fb9f0000
+
+ /* x^17472 mod p(x), x^17408 mod p(x) */
+ .octa 0x00000000173000000000000003770000
+
+ /* x^16448 mod p(x), x^16384 mod p(x) */
+ .octa 0x000000006fe300000000000004880000
+
+ /* x^15424 mod p(x), x^15360 mod p(x) */
+ .octa 0x00000000e15300000000000056a70000
+
+ /* x^14400 mod p(x), x^14336 mod p(x) */
+ .octa 0x0000000092d60000000000009dfd0000
+
+ /* x^13376 mod p(x), x^13312 mod p(x) */
+ .octa 0x0000000002fd00000000000074c80000
+
+ /* x^12352 mod p(x), x^12288 mod p(x) */
+ .octa 0x00000000c78b000000000000a3ec0000
+
+ /* x^11328 mod p(x), x^11264 mod p(x) */
+ .octa 0x000000009262000000000000b3530000
+
+ /* x^10304 mod p(x), x^10240 mod p(x) */
+ .octa 0x0000000084f200000000000047bf0000
+
+ /* x^9280 mod p(x), x^9216 mod p(x) */
+ .octa 0x0000000067ee000000000000e97c0000
+
+ /* x^8256 mod p(x), x^8192 mod p(x) */
+ .octa 0x00000000535b00000000000091e10000
+
+ /* x^7232 mod p(x), x^7168 mod p(x) */
+ .octa 0x000000007ebb00000000000055060000
+
+ /* x^6208 mod p(x), x^6144 mod p(x) */
+ .octa 0x00000000c6a1000000000000fd360000
+
+ /* x^5184 mod p(x), x^5120 mod p(x) */
+ .octa 0x000000001be500000000000055860000
+
+ /* x^4160 mod p(x), x^4096 mod p(x) */
+ .octa 0x00000000ae0e0000000000005bd00000
+
+ /* x^3136 mod p(x), x^3072 mod p(x) */
+ .octa 0x0000000022040000000000008db20000
+
+ /* x^2112 mod p(x), x^2048 mod p(x) */
+ .octa 0x00000000c9eb000000000000efe20000
+
+ /* x^1088 mod p(x), x^1024 mod p(x) */
+ .octa 0x0000000039b400000000000051d10000
+
+.short_constants:
+
+ /* Reduce final 1024-2048 bits to 64 bits, shifting 32 bits to include the trailing 32 bits of zeros */
+ /* x^2048 mod p(x), x^2016 mod p(x), x^1984 mod p(x), x^1952 mod p(x) */
+ .octa 0xefe20000dccf00009440000033590000
+
+ /* x^1920 mod p(x), x^1888 mod p(x), x^1856 mod p(x), x^1824 mod p(x) */
+ .octa 0xee6300002f3f000062180000e0ed0000
+
+ /* x^1792 mod p(x), x^1760 mod p(x), x^1728 mod p(x), x^1696 mod p(x) */
+ .octa 0xcf5f000017ef0000ccbe000023d30000
+
+ /* x^1664 mod p(x), x^1632 mod p(x), x^1600 mod p(x), x^1568 mod p(x) */
+ .octa 0x6d0c0000a30e00000920000042630000
+
+ /* x^1536 mod p(x), x^1504 mod p(x), x^1472 mod p(x), x^1440 mod p(x) */
+ .octa 0x21d30000932b0000a7a00000efcc0000
+
+ /* x^1408 mod p(x), x^1376 mod p(x), x^1344 mod p(x), x^1312 mod p(x) */
+ .octa 0x10be00000b310000666f00000d1c0000
+
+ /* x^1280 mod p(x), x^1248 mod p(x), x^1216 mod p(x), x^1184 mod p(x) */
+ .octa 0x1f240000ce9e0000caad0000589e0000
+
+ /* x^1152 mod p(x), x^1120 mod p(x), x^1088 mod p(x), x^1056 mod p(x) */
+ .octa 0x29610000d02b000039b400007cf50000
+
+ /* x^1024 mod p(x), x^992 mod p(x), x^960 mod p(x), x^928 mod p(x) */
+ .octa 0x51d100009d9d00003c0e0000bfd60000
+
+ /* x^896 mod p(x), x^864 mod p(x), x^832 mod p(x), x^800 mod p(x) */
+ .octa 0xda390000ceae000013830000713c0000
+
+ /* x^768 mod p(x), x^736 mod p(x), x^704 mod p(x), x^672 mod p(x) */
+ .octa 0xb67800001e16000085c0000080a60000
+
+ /* x^640 mod p(x), x^608 mod p(x), x^576 mod p(x), x^544 mod p(x) */
+ .octa 0x0db40000f7f90000371d0000e6580000
+
+ /* x^512 mod p(x), x^480 mod p(x), x^448 mod p(x), x^416 mod p(x) */
+ .octa 0x87e70000044c0000aadb0000a4970000
+
+ /* x^384 mod p(x), x^352 mod p(x), x^320 mod p(x), x^288 mod p(x) */
+ .octa 0x1f990000ad180000d8b30000e7b50000
+
+ /* x^256 mod p(x), x^224 mod p(x), x^192 mod p(x), x^160 mod p(x) */
+ .octa 0xbe6c00006ee300004c1a000006df0000
+
+ /* x^128 mod p(x), x^96 mod p(x), x^64 mod p(x), x^32 mod p(x) */
+ .octa 0xfb0b00002d560000136800008bb70000
+
+
+.barrett_constants:
+ /* Barrett constant m - (4^32)/n */
+ .octa 0x000000000000000000000001f65a57f8 /* x^64 div p(x) */
+ /* Barrett constant n */
+ .octa 0x0000000000000000000000018bb70000
+
+#define CRC_FUNCTION_NAME __crct10dif_vpmsum
+#include "crc32-vpmsum_core.S"
diff --git a/arch/powerpc/crypto/crct10dif-vpmsum_glue.c b/arch/powerpc/crypto/crct10dif-vpmsum_glue.c
new file mode 100644
index 0000000..02ea277
--- /dev/null
+++ b/arch/powerpc/crypto/crct10dif-vpmsum_glue.c
@@ -0,0 +1,128 @@
+/*
+ * Calculate a CRC T10-DIF with vpmsum acceleration
+ *
+ * Copyright 2017, Daniel Axtens, IBM Corporation.
+ * [based on crc32c-vpmsum_glue.c]
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ */
+
+#include <linux/crc-t10dif.h>
+#include <crypto/internal/hash.h>
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/kernel.h>
+#include <linux/cpufeature.h>
+#include <asm/switch_to.h>
+
+#define VMX_ALIGN 16
+#define VMX_ALIGN_MASK (VMX_ALIGN-1)
+
+#define VECTOR_BREAKPOINT 64
+
+u32 __crct10dif_vpmsum(u32 crc, unsigned char const *p, size_t len);
+
+static u16 crct10dif_vpmsum(u16 crci, unsigned char const *p, size_t len)
+{
+ unsigned int prealign;
+ unsigned int tail;
+ u32 crc = crci;
+
+ if (len < (VECTOR_BREAKPOINT + VMX_ALIGN) || in_interrupt())
+ return crc_t10dif_generic(crc, p, len);
+
+ if ((unsigned long)p & VMX_ALIGN_MASK) {
+ prealign = VMX_ALIGN - ((unsigned long)p & VMX_ALIGN_MASK);
+ crc = crc_t10dif_generic(crc, p, prealign);
+ len -= prealign;
+ p += prealign;
+ }
+
+ if (len & ~VMX_ALIGN_MASK) {
+ crc <<= 16;
+ preempt_disable();
+ pagefault_disable();
+ enable_kernel_altivec();
+ crc = __crct10dif_vpmsum(crc, p, len & ~VMX_ALIGN_MASK);
+ disable_kernel_altivec();
+ pagefault_enable();
+ preempt_enable();
+ crc >>= 16;
+ }
+
+ tail = len & VMX_ALIGN_MASK;
+ if (tail) {
+ p += len & ~VMX_ALIGN_MASK;
+ crc = crc_t10dif_generic(crc, p, tail);
+ }
+
+ return crc & 0xffff;
+}
+
+static int crct10dif_vpmsum_init(struct shash_desc *desc)
+{
+ u16 *crc = shash_desc_ctx(desc);
+
+ *crc = 0;
+ return 0;
+}
+
+static int crct10dif_vpmsum_update(struct shash_desc *desc, const u8 *data,
+ unsigned int length)
+{
+ u16 *crc = shash_desc_ctx(desc);
+
+ *crc = crct10dif_vpmsum(*crc, data, length);
+
+ return 0;
+}
+
+
+static int crct10dif_vpmsum_final(struct shash_desc *desc, u8 *out)
+{
+ u16 *crcp = shash_desc_ctx(desc);
+
+ *(u16 *)out = *crcp;
+ return 0;
+}
+
+static struct shash_alg alg = {
+ .init = crct10dif_vpmsum_init,
+ .update = crct10dif_vpmsum_update,
+ .final = crct10dif_vpmsum_final,
+ .descsize = CRC_T10DIF_DIGEST_SIZE,
+ .digestsize = CRC_T10DIF_DIGEST_SIZE,
+ .base = {
+ .cra_name = "crct10dif",
+ .cra_driver_name = "crct10dif-vpmsum",
+ .cra_priority = 200,
+ .cra_blocksize = CRC_T10DIF_BLOCK_SIZE,
+ .cra_module = THIS_MODULE,
+ }
+};
+
+static int __init crct10dif_vpmsum_mod_init(void)
+{
+ if (!cpu_has_feature(CPU_FTR_ARCH_207S))
+ return -ENODEV;
+
+ return crypto_register_shash(&alg);
+}
+
+static void __exit crct10dif_vpmsum_mod_fini(void)
+{
+ crypto_unregister_shash(&alg);
+}
+
+module_cpu_feature_match(PPC_MODULE_FEATURE_VEC_CRYPTO, crct10dif_vpmsum_mod_init);
+module_exit(crct10dif_vpmsum_mod_fini);
+
+MODULE_AUTHOR("Daniel Axtens <dja@axtens.net>");
+MODULE_DESCRIPTION("CRCT10DIF using vector polynomial multiply-sum instructions");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_CRYPTO("crct10dif");
+MODULE_ALIAS_CRYPTO("crct10dif-vpmsum");
diff --git a/arch/powerpc/include/asm/bug.h b/arch/powerpc/include/asm/bug.h
index 3a39283..f2c562a 100644
--- a/arch/powerpc/include/asm/bug.h
+++ b/arch/powerpc/include/asm/bug.h
@@ -85,12 +85,12 @@
} \
} while (0)
-#define __WARN_TAINT(taint) do { \
+#define __WARN_FLAGS(flags) do { \
__asm__ __volatile__( \
"1: twi 31,0,0\n" \
_EMIT_BUG_ENTRY \
: : "i" (__FILE__), "i" (__LINE__), \
- "i" (BUGFLAG_TAINT(taint)), \
+ "i" (BUGFLAG_WARNING|(flags)), \
"i" (sizeof(struct bug_entry))); \
} while (0)
diff --git a/arch/powerpc/include/asm/extable.h b/arch/powerpc/include/asm/extable.h
new file mode 100644
index 0000000..07cc45c
--- /dev/null
+++ b/arch/powerpc/include/asm/extable.h
@@ -0,0 +1,29 @@
+#ifndef _ARCH_POWERPC_EXTABLE_H
+#define _ARCH_POWERPC_EXTABLE_H
+
+/*
+ * The exception table consists of pairs of relative addresses: the first is
+ * the address of an instruction that is allowed to fault, and the second is
+ * the address at which the program should continue. No registers are
+ * modified, so it is entirely up to the continuation code to figure out what
+ * to do.
+ *
+ * All the routines below use bits of fixup code that are out of line with the
+ * main instruction path. This means when everything is well, we don't even
+ * have to jump over them. Further, they do not intrude on our cache or tlb
+ * entries.
+ */
+
+#define ARCH_HAS_RELATIVE_EXTABLE
+
+struct exception_table_entry {
+ int insn;
+ int fixup;
+};
+
+static inline unsigned long extable_fixup(const struct exception_table_entry *x)
+{
+ return (unsigned long)&x->fixup + x->fixup;
+}
+
+#endif
diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h
index b9e3f0a..ecf9885 100644
--- a/arch/powerpc/include/asm/mmu_context.h
+++ b/arch/powerpc/include/asm/mmu_context.h
@@ -163,11 +163,5 @@
/* by default, allow everything */
return true;
}
-
-static inline bool arch_pte_access_permitted(pte_t pte, bool write)
-{
- /* by default, allow everything */
- return true;
-}
#endif /* __KERNEL__ */
#endif /* __ASM_POWERPC_MMU_CONTEXT_H */
diff --git a/arch/powerpc/include/asm/thread_info.h b/arch/powerpc/include/asm/thread_info.h
index 87e4b2d..6fc6464 100644
--- a/arch/powerpc/include/asm/thread_info.h
+++ b/arch/powerpc/include/asm/thread_info.h
@@ -92,6 +92,7 @@
TIF_NEED_RESCHED */
#define TIF_32BIT 4 /* 32 bit binary */
#define TIF_RESTORE_TM 5 /* need to restore TM FP/VEC/VSX */
+#define TIF_PATCH_PENDING 6 /* pending live patching update */
#define TIF_SYSCALL_AUDIT 7 /* syscall auditing active */
#define TIF_SINGLESTEP 8 /* singlestepping active */
#define TIF_NOHZ 9 /* in adaptive nohz mode */
@@ -115,6 +116,7 @@
#define _TIF_POLLING_NRFLAG (1<<TIF_POLLING_NRFLAG)
#define _TIF_32BIT (1<<TIF_32BIT)
#define _TIF_RESTORE_TM (1<<TIF_RESTORE_TM)
+#define _TIF_PATCH_PENDING (1<<TIF_PATCH_PENDING)
#define _TIF_SYSCALL_AUDIT (1<<TIF_SYSCALL_AUDIT)
#define _TIF_SINGLESTEP (1<<TIF_SINGLESTEP)
#define _TIF_SECCOMP (1<<TIF_SECCOMP)
@@ -131,7 +133,7 @@
#define _TIF_USER_WORK_MASK (_TIF_SIGPENDING | _TIF_NEED_RESCHED | \
_TIF_NOTIFY_RESUME | _TIF_UPROBE | \
- _TIF_RESTORE_TM)
+ _TIF_RESTORE_TM | _TIF_PATCH_PENDING)
#define _TIF_PERSYSCALL_MASK (_TIF_RESTOREALL|_TIF_NOERROR)
/* Bits in local_flags */
diff --git a/arch/powerpc/include/asm/uaccess.h b/arch/powerpc/include/asm/uaccess.h
index 0e6add3..5c0d8a8 100644
--- a/arch/powerpc/include/asm/uaccess.h
+++ b/arch/powerpc/include/asm/uaccess.h
@@ -1,18 +1,11 @@
#ifndef _ARCH_POWERPC_UACCESS_H
#define _ARCH_POWERPC_UACCESS_H
-#ifdef __KERNEL__
-#ifndef __ASSEMBLY__
-
-#include <linux/sched.h>
-#include <linux/errno.h>
#include <asm/asm-compat.h>
#include <asm/ppc_asm.h>
#include <asm/processor.h>
#include <asm/page.h>
-
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
+#include <asm/extable.h>
/*
* The fs value determines whether argument validity checking should be
@@ -64,31 +57,6 @@
__access_ok((__force unsigned long)(addr), (size), get_fs()))
/*
- * The exception table consists of pairs of relative addresses: the first is
- * the address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out what
- * to do.
- *
- * All the routines below use bits of fixup code that are out of line with the
- * main instruction path. This means when everything is well, we don't even
- * have to jump over them. Further, they do not intrude on our cache or tlb
- * entries.
- */
-
-#define ARCH_HAS_RELATIVE_EXTABLE
-
-struct exception_table_entry {
- int insn;
- int fixup;
-};
-
-static inline unsigned long extable_fixup(const struct exception_table_entry *x)
-{
- return (unsigned long)&x->fixup + x->fixup;
-}
-
-/*
* These are the main single-value transfer routines. They automatically
* use the right size if we just have the right pointer type.
*
@@ -301,42 +269,19 @@
#ifndef __powerpc64__
-static inline unsigned long copy_from_user(void *to,
- const void __user *from, unsigned long n)
-{
- if (likely(access_ok(VERIFY_READ, from, n))) {
- check_object_size(to, n, false);
- return __copy_tofrom_user((__force void __user *)to, from, n);
- }
- memset(to, 0, n);
- return n;
-}
-
-static inline unsigned long copy_to_user(void __user *to,
- const void *from, unsigned long n)
-{
- if (access_ok(VERIFY_WRITE, to, n)) {
- check_object_size(from, n, true);
- return __copy_tofrom_user(to, (__force void __user *)from, n);
- }
- return n;
-}
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
#else /* __powerpc64__ */
-#define __copy_in_user(to, from, size) \
- __copy_tofrom_user((to), (from), (size))
-
-extern unsigned long copy_from_user(void *to, const void __user *from,
- unsigned long n);
-extern unsigned long copy_to_user(void __user *to, const void *from,
- unsigned long n);
-extern unsigned long copy_in_user(void __user *to, const void __user *from,
- unsigned long n);
-
+static inline unsigned long
+raw_copy_in_user(void __user *to, const void __user *from, unsigned long n)
+{
+ return __copy_tofrom_user(to, from, n);
+}
#endif /* __powerpc64__ */
-static inline unsigned long __copy_from_user_inatomic(void *to,
+static inline unsigned long raw_copy_from_user(void *to,
const void __user *from, unsigned long n)
{
if (__builtin_constant_p(n) && (n <= 8)) {
@@ -360,12 +305,10 @@
return 0;
}
- check_object_size(to, n, false);
-
return __copy_tofrom_user((__force void __user *)to, from, n);
}
-static inline unsigned long __copy_to_user_inatomic(void __user *to,
+static inline unsigned long raw_copy_to_user(void __user *to,
const void *from, unsigned long n)
{
if (__builtin_constant_p(n) && (n <= 8)) {
@@ -389,25 +332,9 @@
return 0;
}
- check_object_size(from, n, true);
-
return __copy_tofrom_user(to, (__force const void __user *)from, n);
}
-static inline unsigned long __copy_from_user(void *to,
- const void __user *from, unsigned long size)
-{
- might_fault();
- return __copy_from_user_inatomic(to, from, size);
-}
-
-static inline unsigned long __copy_to_user(void __user *to,
- const void *from, unsigned long size)
-{
- might_fault();
- return __copy_to_user_inatomic(to, from, size);
-}
-
extern unsigned long __clear_user(void __user *addr, unsigned long size);
static inline unsigned long clear_user(void __user *addr, unsigned long size)
@@ -422,7 +349,4 @@
extern __must_check long strlen_user(const char __user *str);
extern __must_check long strnlen_user(const char __user *str, long n);
-#endif /* __ASSEMBLY__ */
-#endif /* __KERNEL__ */
-
#endif /* _ARCH_POWERPC_UACCESS_H */
diff --git a/arch/powerpc/include/uapi/asm/socket.h b/arch/powerpc/include/uapi/asm/socket.h
index 44583a5..58e2ec0 100644
--- a/arch/powerpc/include/uapi/asm/socket.h
+++ b/arch/powerpc/include/uapi/asm/socket.h
@@ -99,4 +99,10 @@
#define SCM_TIMESTAMPING_OPT_STATS 54
+#define SO_MEMINFO 55
+
+#define SO_INCOMING_NAPI_ID 56
+
+#define SO_COOKIE 57
+
#endif /* _ASM_POWERPC_SOCKET_H */
diff --git a/arch/powerpc/kernel/nvram_64.c b/arch/powerpc/kernel/nvram_64.c
index d5e2b83..eae61b0 100644
--- a/arch/powerpc/kernel/nvram_64.c
+++ b/arch/powerpc/kernel/nvram_64.c
@@ -389,51 +389,40 @@
/**
* nvram_pstore_write - pstore write callback for nvram
- * @type: Type of message logged
- * @reason: reason behind dump (oops/panic)
- * @id: identifier to indicate the write performed
- * @part: pstore writes data to registered buffer in parts,
- * part number will indicate the same.
- * @count: Indicates oops count
- * @compressed: Flag to indicate the log is compressed
- * @size: number of bytes written to the registered buffer
- * @psi: registered pstore_info structure
+ * @record: pstore record to write, with @id to be set
*
* Called by pstore_dump() when an oops or panic report is logged in the
* printk buffer.
* Returns 0 on successful write.
*/
-static int nvram_pstore_write(enum pstore_type_id type,
- enum kmsg_dump_reason reason,
- u64 *id, unsigned int part, int count,
- bool compressed, size_t size,
- struct pstore_info *psi)
+static int nvram_pstore_write(struct pstore_record *record)
{
int rc;
unsigned int err_type = ERR_TYPE_KERNEL_PANIC;
struct oops_log_info *oops_hdr = (struct oops_log_info *) oops_buf;
/* part 1 has the recent messages from printk buffer */
- if (part > 1 || (type != PSTORE_TYPE_DMESG))
+ if (record->part > 1 || (record->type != PSTORE_TYPE_DMESG))
return -1;
if (clobbering_unread_rtas_event())
return -1;
oops_hdr->version = cpu_to_be16(OOPS_HDR_VERSION);
- oops_hdr->report_length = cpu_to_be16(size);
+ oops_hdr->report_length = cpu_to_be16(record->size);
oops_hdr->timestamp = cpu_to_be64(ktime_get_real_seconds());
- if (compressed)
+ if (record->compressed)
err_type = ERR_TYPE_KERNEL_PANIC_GZ;
rc = nvram_write_os_partition(&oops_log_partition, oops_buf,
- (int) (sizeof(*oops_hdr) + size), err_type, count);
+ (int) (sizeof(*oops_hdr) + record->size), err_type,
+ record->count);
if (rc != 0)
return rc;
- *id = part;
+ record->id = record->part;
return 0;
}
@@ -442,10 +431,7 @@
* Returns the length of the data we read from each partition.
* Returns 0 if we've been called before.
*/
-static ssize_t nvram_pstore_read(u64 *id, enum pstore_type_id *type,
- int *count, struct timespec *time, char **buf,
- bool *compressed, ssize_t *ecc_notice_size,
- struct pstore_info *psi)
+static ssize_t nvram_pstore_read(struct pstore_record *record)
{
struct oops_log_info *oops_hdr;
unsigned int err_type, id_no, size = 0;
@@ -459,40 +445,40 @@
switch (nvram_type_ids[read_type]) {
case PSTORE_TYPE_DMESG:
part = &oops_log_partition;
- *type = PSTORE_TYPE_DMESG;
+ record->type = PSTORE_TYPE_DMESG;
break;
case PSTORE_TYPE_PPC_COMMON:
sig = NVRAM_SIG_SYS;
part = &common_partition;
- *type = PSTORE_TYPE_PPC_COMMON;
- *id = PSTORE_TYPE_PPC_COMMON;
- time->tv_sec = 0;
- time->tv_nsec = 0;
+ record->type = PSTORE_TYPE_PPC_COMMON;
+ record->id = PSTORE_TYPE_PPC_COMMON;
+ record->time.tv_sec = 0;
+ record->time.tv_nsec = 0;
break;
#ifdef CONFIG_PPC_PSERIES
case PSTORE_TYPE_PPC_RTAS:
part = &rtas_log_partition;
- *type = PSTORE_TYPE_PPC_RTAS;
- time->tv_sec = last_rtas_event;
- time->tv_nsec = 0;
+ record->type = PSTORE_TYPE_PPC_RTAS;
+ record->time.tv_sec = last_rtas_event;
+ record->time.tv_nsec = 0;
break;
case PSTORE_TYPE_PPC_OF:
sig = NVRAM_SIG_OF;
part = &of_config_partition;
- *type = PSTORE_TYPE_PPC_OF;
- *id = PSTORE_TYPE_PPC_OF;
- time->tv_sec = 0;
- time->tv_nsec = 0;
+ record->type = PSTORE_TYPE_PPC_OF;
+ record->id = PSTORE_TYPE_PPC_OF;
+ record->time.tv_sec = 0;
+ record->time.tv_nsec = 0;
break;
#endif
#ifdef CONFIG_PPC_POWERNV
case PSTORE_TYPE_PPC_OPAL:
sig = NVRAM_SIG_FW;
part = &skiboot_partition;
- *type = PSTORE_TYPE_PPC_OPAL;
- *id = PSTORE_TYPE_PPC_OPAL;
- time->tv_sec = 0;
- time->tv_nsec = 0;
+ record->type = PSTORE_TYPE_PPC_OPAL;
+ record->id = PSTORE_TYPE_PPC_OPAL;
+ record->time.tv_sec = 0;
+ record->time.tv_nsec = 0;
break;
#endif
default:
@@ -520,10 +506,10 @@
return 0;
}
- *count = 0;
+ record->count = 0;
if (part->os_partition)
- *id = id_no;
+ record->id = id_no;
if (nvram_type_ids[read_type] == PSTORE_TYPE_DMESG) {
size_t length, hdr_size;
@@ -533,34 +519,35 @@
/* Old format oops header had 2-byte record size */
hdr_size = sizeof(u16);
length = be16_to_cpu(oops_hdr->version);
- time->tv_sec = 0;
- time->tv_nsec = 0;
+ record->time.tv_sec = 0;
+ record->time.tv_nsec = 0;
} else {
hdr_size = sizeof(*oops_hdr);
length = be16_to_cpu(oops_hdr->report_length);
- time->tv_sec = be64_to_cpu(oops_hdr->timestamp);
- time->tv_nsec = 0;
+ record->time.tv_sec = be64_to_cpu(oops_hdr->timestamp);
+ record->time.tv_nsec = 0;
}
- *buf = kmemdup(buff + hdr_size, length, GFP_KERNEL);
+ record->buf = kmemdup(buff + hdr_size, length, GFP_KERNEL);
kfree(buff);
- if (*buf == NULL)
+ if (record->buf == NULL)
return -ENOMEM;
- *ecc_notice_size = 0;
+ record->ecc_notice_size = 0;
if (err_type == ERR_TYPE_KERNEL_PANIC_GZ)
- *compressed = true;
+ record->compressed = true;
else
- *compressed = false;
+ record->compressed = false;
return length;
}
- *buf = buff;
+ record->buf = buff;
return part->size;
}
static struct pstore_info nvram_pstore_info = {
.owner = THIS_MODULE,
.name = "nvram",
+ .flags = PSTORE_FLAGS_DMESG,
.open = nvram_pstore_open,
.read = nvram_pstore_read,
.write = nvram_pstore_write,
diff --git a/arch/powerpc/kernel/signal.c b/arch/powerpc/kernel/signal.c
index 3a36711..e9436c5 100644
--- a/arch/powerpc/kernel/signal.c
+++ b/arch/powerpc/kernel/signal.c
@@ -14,6 +14,7 @@
#include <linux/uprobes.h>
#include <linux/key.h>
#include <linux/context_tracking.h>
+#include <linux/livepatch.h>
#include <asm/hw_breakpoint.h>
#include <linux/uaccess.h>
#include <asm/unistd.h>
@@ -162,6 +163,9 @@
tracehook_notify_resume(regs);
}
+ if (thread_info_flags & _TIF_PATCH_PENDING)
+ klp_update_patch_state(current);
+
user_enter();
}
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index 46f89e6..d68ed1f 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -787,24 +787,21 @@
{ NULL, },
};
+static __init long smp_setup_cpu_workfn(void *data __always_unused)
+{
+ smp_ops->setup_cpu(boot_cpuid);
+ return 0;
+}
+
void __init smp_cpus_done(unsigned int max_cpus)
{
- cpumask_var_t old_mask;
-
- /* We want the setup_cpu() here to be called from CPU 0, but our
- * init thread may have been "borrowed" by another CPU in the meantime
- * se we pin us down to CPU 0 for a short while
+ /*
+ * We want the setup_cpu() here to be called on the boot CPU, but
+ * init might run on any CPU, so make sure it's invoked on the boot
+ * CPU.
*/
- alloc_cpumask_var(&old_mask, GFP_NOWAIT);
- cpumask_copy(old_mask, ¤t->cpus_allowed);
- set_cpus_allowed_ptr(current, cpumask_of(boot_cpuid));
-
if (smp_ops && smp_ops->setup_cpu)
- smp_ops->setup_cpu(boot_cpuid);
-
- set_cpus_allowed_ptr(current, old_mask);
-
- free_cpumask_var(old_mask);
+ work_on_cpu_safe(boot_cpuid, smp_setup_cpu_workfn, NULL);
if (smp_ops && smp_ops->bringup_done)
smp_ops->bringup_done();
@@ -812,7 +809,6 @@
dump_numa_cpu_topology();
set_sched_topology(powerpc_topology);
-
}
#ifdef CONFIG_HOTPLUG_CPU
diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c
index 07b9072..2b33cfa 100644
--- a/arch/powerpc/kernel/time.c
+++ b/arch/powerpc/kernel/time.c
@@ -995,8 +995,10 @@
decrementer_clockevent.max_delta_ns =
clockevent_delta2ns(decrementer_max, &decrementer_clockevent);
+ decrementer_clockevent.max_delta_ticks = decrementer_max;
decrementer_clockevent.min_delta_ns =
clockevent_delta2ns(2, &decrementer_clockevent);
+ decrementer_clockevent.min_delta_ticks = 2;
register_decrementer_clockevent(cpu);
}
diff --git a/arch/powerpc/kernel/vmlinux.lds.S b/arch/powerpc/kernel/vmlinux.lds.S
index 7394b77..1c24c89 100644
--- a/arch/powerpc/kernel/vmlinux.lds.S
+++ b/arch/powerpc/kernel/vmlinux.lds.S
@@ -312,6 +312,8 @@
NOSAVE_DATA
}
+ BUG_TABLE
+
. = ALIGN(PAGE_SIZE);
_edata = .;
PROVIDE32 (edata = .);
diff --git a/arch/powerpc/lib/Makefile b/arch/powerpc/lib/Makefile
index 2b5e090..ed7dfce 100644
--- a/arch/powerpc/lib/Makefile
+++ b/arch/powerpc/lib/Makefile
@@ -14,7 +14,7 @@
obj-$(CONFIG_PPC32) += div64.o copy_32.o
-obj64-y += copypage_64.o copyuser_64.o usercopy_64.o mem_64.o hweight_64.o \
+obj64-y += copypage_64.o copyuser_64.o mem_64.o hweight_64.o \
copyuser_power7.o string_64.o copypage_power7.o memcpy_power7.o \
memcpy_64.o memcmp_64.o
diff --git a/arch/powerpc/lib/copy_32.S b/arch/powerpc/lib/copy_32.S
index ff0d894..8aedbb5 100644
--- a/arch/powerpc/lib/copy_32.S
+++ b/arch/powerpc/lib/copy_32.S
@@ -477,18 +477,6 @@
bdnz 130b
/* then clear out the destination: r3 bytes starting at 4(r6) */
132: mfctr r3
- srwi. r0,r3,2
- li r9,0
- mtctr r0
- beq 113f
-112: stwu r9,4(r6)
- bdnz 112b
-113: andi. r0,r3,3
- mtctr r0
- beq 120f
-114: stb r9,4(r6)
- addi r6,r6,1
- bdnz 114b
120: blr
EX_TABLE(30b,108b)
@@ -497,7 +485,5 @@
EX_TABLE(41b,111b)
EX_TABLE(130b,132b)
EX_TABLE(131b,120b)
- EX_TABLE(112b,120b)
- EX_TABLE(114b,120b)
EXPORT_SYMBOL(__copy_tofrom_user)
diff --git a/arch/powerpc/lib/copyuser_64.S b/arch/powerpc/lib/copyuser_64.S
index aee6e24..08da06e 100644
--- a/arch/powerpc/lib/copyuser_64.S
+++ b/arch/powerpc/lib/copyuser_64.S
@@ -319,32 +319,9 @@
blr
/*
- * here we have trapped again, need to clear ctr bytes starting at r3
+ * here we have trapped again, amount remaining is in ctr.
*/
-143: mfctr r5
- li r0,0
- mr r4,r3
- mr r3,r5 /* return the number of bytes not copied */
-1: andi. r9,r4,7
- beq 3f
-90: stb r0,0(r4)
- addic. r5,r5,-1
- addi r4,r4,1
- bne 1b
- blr
-3: cmpldi cr1,r5,8
- srdi r9,r5,3
- andi. r5,r5,7
- blt cr1,93f
- mtctr r9
-91: std r0,0(r4)
- addi r4,r4,8
- bdnz 91b
-93: beqlr
- mtctr r5
-92: stb r0,0(r4)
- addi r4,r4,1
- bdnz 92b
+143: mfctr r3
blr
/*
@@ -389,10 +366,7 @@
ld r5,-8(r1)
add r6,r6,r5
subf r3,r3,r6 /* #bytes not copied */
-190:
-191:
-192:
- blr /* #bytes not copied in r3 */
+ blr
EX_TABLE(20b,120b)
EX_TABLE(220b,320b)
@@ -451,9 +425,6 @@
EX_TABLE(88b,188b)
EX_TABLE(43b,143b)
EX_TABLE(89b,189b)
- EX_TABLE(90b,190b)
- EX_TABLE(91b,191b)
- EX_TABLE(92b,192b)
/*
* Routine to copy a whole page of data, optimized for POWER4.
diff --git a/arch/powerpc/lib/usercopy_64.c b/arch/powerpc/lib/usercopy_64.c
deleted file mode 100644
index 9bd3a3d..0000000
--- a/arch/powerpc/lib/usercopy_64.c
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Functions which are too large to be inlined.
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * as published by the Free Software Foundation; either version
- * 2 of the License, or (at your option) any later version.
- */
-#include <linux/module.h>
-#include <linux/uaccess.h>
-
-unsigned long copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- if (likely(access_ok(VERIFY_READ, from, n)))
- n = __copy_from_user(to, from, n);
- else
- memset(to, 0, n);
- return n;
-}
-
-unsigned long copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- if (likely(access_ok(VERIFY_WRITE, to, n)))
- n = __copy_to_user(to, from, n);
- return n;
-}
-
-unsigned long copy_in_user(void __user *to, const void __user *from,
- unsigned long n)
-{
- might_sleep();
- if (likely(access_ok(VERIFY_READ, from, n) &&
- access_ok(VERIFY_WRITE, to, n)))
- n =__copy_tofrom_user(to, from, n);
- return n;
-}
-
-EXPORT_SYMBOL(copy_from_user);
-EXPORT_SYMBOL(copy_to_user);
-EXPORT_SYMBOL(copy_in_user);
-
diff --git a/arch/s390/Kbuild b/arch/s390/Kbuild
index e256592..eae2c64 100644
--- a/arch/s390/Kbuild
+++ b/arch/s390/Kbuild
@@ -1,7 +1,7 @@
obj-y += kernel/
obj-y += mm/
obj-$(CONFIG_KVM) += kvm/
-obj-$(CONFIG_CRYPTO_HW) += crypto/
+obj-y += crypto/
obj-$(CONFIG_S390_HYPFS_FS) += hypfs/
obj-$(CONFIG_APPLDATA_BASE) += appldata/
obj-y += net/
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index a2dcef0..e161faf 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -105,6 +105,7 @@
select ARCH_INLINE_WRITE_UNLOCK_IRQRESTORE
select ARCH_SAVE_PAGE_KEYS if HIBERNATION
select ARCH_SUPPORTS_ATOMIC_RMW
+ select ARCH_SUPPORTS_DEFERRED_STRUCT_PAGE_INIT
select ARCH_SUPPORTS_NUMA_BALANCING
select ARCH_USE_BUILTIN_BSWAP
select ARCH_USE_CMPXCHG_LOCKREF
@@ -123,8 +124,6 @@
select GENERIC_TIME_VSYSCALL
select HAVE_ALIGNED_STRUCT_PAGE if SLUB
select HAVE_ARCH_AUDITSYSCALL
- select HAVE_ARCH_EARLY_PFN_TO_NID
- select HAVE_ARCH_HARDENED_USERCOPY
select HAVE_ARCH_JUMP_LABEL
select CPU_NO_EFFICIENT_FFS if !HAVE_MARCH_Z9_109_FEATURES
select HAVE_ARCH_SECCOMP_FILTER
@@ -507,6 +506,21 @@
source kernel/Kconfig.hz
+config ARCH_RANDOM
+ def_bool y
+ prompt "s390 architectural random number generation API"
+ help
+ Enable the s390 architectural random number generation API
+ to provide random data for all consumers within the Linux
+ kernel.
+
+ When enabled the arch_random_* functions declared in linux/random.h
+ are implemented. The implementation is based on the s390 CPACF
+ instruction subfunction TRNG which provides a real true random
+ number generator.
+
+ If unsure, say Y.
+
endmenu
menu "Memory setup"
@@ -537,6 +551,16 @@
source "mm/Kconfig"
+config MAX_PHYSMEM_BITS
+ int "Maximum size of supported physical memory in bits (42-53)"
+ range 42 53
+ default "46"
+ help
+ This option specifies the maximum supported size of physical memory
+ in bits. Supported is any size between 2^42 (4TB) and 2^53 (8PB).
+ Increasing the number of bits also increases the kernel image size.
+ By default 46 bits (64TB) are supported.
+
config PACK_STACK
def_bool y
prompt "Pack kernel stack"
@@ -614,7 +638,7 @@
config PCI_NR_FUNCTIONS
int "Maximum number of PCI functions (1-4096)"
range 1 4096
- default "64"
+ default "128"
help
This allows you to specify the maximum number of PCI functions which
this kernel will support.
@@ -672,6 +696,16 @@
To compile this driver as a module, choose M here: the
module will be called eadm_sch.
+config VFIO_CCW
+ def_tristate n
+ prompt "Support for VFIO-CCW subchannels"
+ depends on S390_CCW_IOMMU && VFIO_MDEV
+ help
+ This driver allows usage of I/O subchannels via VFIO-CCW.
+
+ To compile this driver as a module, choose M here: the
+ module will be called vfio_ccw.
+
endmenu
menu "Dump support"
diff --git a/arch/s390/configs/default_defconfig b/arch/s390/configs/default_defconfig
index 4b176fe..a5039fa 100644
--- a/arch/s390/configs/default_defconfig
+++ b/arch/s390/configs/default_defconfig
@@ -73,6 +73,7 @@
CONFIG_ZBUD=m
CONFIG_ZSMALLOC=m
CONFIG_ZSMALLOC_STAT=y
+CONFIG_DEFERRED_STRUCT_PAGE_INIT=y
CONFIG_IDLE_PAGE_TRACKING=y
CONFIG_PCI=y
CONFIG_PCI_DEBUG=y
diff --git a/arch/s390/configs/gcov_defconfig b/arch/s390/configs/gcov_defconfig
index 0de46cc..83970b5 100644
--- a/arch/s390/configs/gcov_defconfig
+++ b/arch/s390/configs/gcov_defconfig
@@ -72,6 +72,7 @@
CONFIG_ZBUD=m
CONFIG_ZSMALLOC=m
CONFIG_ZSMALLOC_STAT=y
+CONFIG_DEFERRED_STRUCT_PAGE_INIT=y
CONFIG_IDLE_PAGE_TRACKING=y
CONFIG_PCI=y
CONFIG_HOTPLUG_PCI=y
diff --git a/arch/s390/configs/performance_defconfig b/arch/s390/configs/performance_defconfig
index e167557..fbc6542 100644
--- a/arch/s390/configs/performance_defconfig
+++ b/arch/s390/configs/performance_defconfig
@@ -70,6 +70,7 @@
CONFIG_ZBUD=m
CONFIG_ZSMALLOC=m
CONFIG_ZSMALLOC_STAT=y
+CONFIG_DEFERRED_STRUCT_PAGE_INIT=y
CONFIG_IDLE_PAGE_TRACKING=y
CONFIG_PCI=y
CONFIG_HOTPLUG_PCI=y
diff --git a/arch/s390/configs/zfcpdump_defconfig b/arch/s390/configs/zfcpdump_defconfig
index 4366a3e..e23d97c1 100644
--- a/arch/s390/configs/zfcpdump_defconfig
+++ b/arch/s390/configs/zfcpdump_defconfig
@@ -35,7 +35,6 @@
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_FC_ATTRS=y
-CONFIG_SCSI_SRP_ATTRS=y
CONFIG_ZFCP=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
# CONFIG_INPUT_KEYBOARD is not set
diff --git a/arch/s390/crypto/Makefile b/arch/s390/crypto/Makefile
index 402c530..678d986 100644
--- a/arch/s390/crypto/Makefile
+++ b/arch/s390/crypto/Makefile
@@ -10,5 +10,6 @@
obj-$(CONFIG_S390_PRNG) += prng.o
obj-$(CONFIG_CRYPTO_GHASH_S390) += ghash_s390.o
obj-$(CONFIG_CRYPTO_CRC32_S390) += crc32-vx_s390.o
+obj-$(CONFIG_ARCH_RANDOM) += arch_random.o
crc32-vx_s390-y := crc32-vx.o crc32le-vx.o crc32be-vx.o
diff --git a/arch/s390/crypto/arch_random.c b/arch/s390/crypto/arch_random.c
new file mode 100644
index 0000000..9317b3e
--- /dev/null
+++ b/arch/s390/crypto/arch_random.c
@@ -0,0 +1,31 @@
+/*
+ * s390 arch random implementation.
+ *
+ * Copyright IBM Corp. 2017
+ * Author(s): Harald Freudenberger <freude@de.ibm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License (version 2 only)
+ * as published by the Free Software Foundation.
+ *
+ */
+
+#include <linux/kernel.h>
+#include <linux/atomic.h>
+#include <linux/static_key.h>
+#include <asm/cpacf.h>
+
+DEFINE_STATIC_KEY_FALSE(s390_arch_random_available);
+
+atomic64_t s390_arch_random_counter = ATOMIC64_INIT(0);
+EXPORT_SYMBOL(s390_arch_random_counter);
+
+static int __init s390_arch_random_init(void)
+{
+ /* check if subfunction CPACF_PRNO_TRNG is available */
+ if (cpacf_query_func(CPACF_PRNO, CPACF_PRNO_TRNG))
+ static_branch_enable(&s390_arch_random_available);
+
+ return 0;
+}
+arch_initcall(s390_arch_random_init);
diff --git a/arch/s390/crypto/paes_s390.c b/arch/s390/crypto/paes_s390.c
index 716b172..a4e903e 100644
--- a/arch/s390/crypto/paes_s390.c
+++ b/arch/s390/crypto/paes_s390.c
@@ -616,7 +616,7 @@
module_init(paes_s390_init);
module_exit(paes_s390_fini);
-MODULE_ALIAS_CRYPTO("aes-all");
+MODULE_ALIAS_CRYPTO("paes");
MODULE_DESCRIPTION("Rijndael (AES) Cipher Algorithm with protected keys");
MODULE_LICENSE("GPL");
diff --git a/arch/s390/crypto/prng.c b/arch/s390/crypto/prng.c
index 5a3ec04..3e47c4a 100644
--- a/arch/s390/crypto/prng.c
+++ b/arch/s390/crypto/prng.c
@@ -81,7 +81,7 @@
u64 byte_counter;
};
-struct ppno_ws_s {
+struct prno_ws_s {
u32 res;
u32 reseed_counter;
u64 stream_bytes;
@@ -93,7 +93,7 @@
struct mutex mutex;
union {
struct prng_ws_s prngws;
- struct ppno_ws_s ppnows;
+ struct prno_ws_s prnows;
};
u8 *buf;
u32 rest;
@@ -306,12 +306,12 @@
0x36, 0x8c, 0x5a, 0x9f, 0x7a, 0x4b, 0x3e, 0xe2 };
u8 buf[sizeof(random)];
- struct ppno_ws_s ws;
+ struct prno_ws_s ws;
memset(&ws, 0, sizeof(ws));
/* initial seed */
- cpacf_ppno(CPACF_PPNO_SHA512_DRNG_SEED,
+ cpacf_prno(CPACF_PRNO_SHA512_DRNG_SEED,
&ws, NULL, 0, seed, sizeof(seed));
/* check working states V and C */
@@ -324,9 +324,9 @@
}
/* generate random bytes */
- cpacf_ppno(CPACF_PPNO_SHA512_DRNG_GEN,
+ cpacf_prno(CPACF_PRNO_SHA512_DRNG_GEN,
&ws, buf, sizeof(buf), NULL, 0);
- cpacf_ppno(CPACF_PPNO_SHA512_DRNG_GEN,
+ cpacf_prno(CPACF_PRNO_SHA512_DRNG_GEN,
&ws, buf, sizeof(buf), NULL, 0);
/* check against expected data */
@@ -374,16 +374,16 @@
/* followed by 16 bytes of unique nonce */
get_tod_clock_ext(seed + 64 + 32);
- /* initial seed of the ppno drng */
- cpacf_ppno(CPACF_PPNO_SHA512_DRNG_SEED,
- &prng_data->ppnows, NULL, 0, seed, sizeof(seed));
+ /* initial seed of the prno drng */
+ cpacf_prno(CPACF_PRNO_SHA512_DRNG_SEED,
+ &prng_data->prnows, NULL, 0, seed, sizeof(seed));
/* if fips mode is enabled, generate a first block of random
bytes for the FIPS 140-2 Conditional Self Test */
if (fips_enabled) {
prng_data->prev = prng_data->buf + prng_chunk_size;
- cpacf_ppno(CPACF_PPNO_SHA512_DRNG_GEN,
- &prng_data->ppnows,
+ cpacf_prno(CPACF_PRNO_SHA512_DRNG_GEN,
+ &prng_data->prnows,
prng_data->prev, prng_chunk_size, NULL, 0);
}
@@ -412,9 +412,9 @@
if (ret != sizeof(seed))
return ret;
- /* do a reseed of the ppno drng with this bytestring */
- cpacf_ppno(CPACF_PPNO_SHA512_DRNG_SEED,
- &prng_data->ppnows, NULL, 0, seed, sizeof(seed));
+ /* do a reseed of the prno drng with this bytestring */
+ cpacf_prno(CPACF_PRNO_SHA512_DRNG_SEED,
+ &prng_data->prnows, NULL, 0, seed, sizeof(seed));
return 0;
}
@@ -425,15 +425,15 @@
int ret;
/* reseed needed ? */
- if (prng_data->ppnows.reseed_counter > prng_reseed_limit) {
+ if (prng_data->prnows.reseed_counter > prng_reseed_limit) {
ret = prng_sha512_reseed();
if (ret)
return ret;
}
- /* PPNO generate */
- cpacf_ppno(CPACF_PPNO_SHA512_DRNG_GEN,
- &prng_data->ppnows, buf, nbytes, NULL, 0);
+ /* PRNO generate */
+ cpacf_prno(CPACF_PRNO_SHA512_DRNG_GEN,
+ &prng_data->prnows, buf, nbytes, NULL, 0);
/* FIPS 140-2 Conditional Self Test */
if (fips_enabled) {
@@ -653,7 +653,7 @@
if (mutex_lock_interruptible(&prng_data->mutex))
return -ERESTARTSYS;
if (prng_mode == PRNG_MODE_SHA512)
- counter = prng_data->ppnows.stream_bytes;
+ counter = prng_data->prnows.stream_bytes;
else
counter = prng_data->prngws.byte_counter;
mutex_unlock(&prng_data->mutex);
@@ -774,8 +774,8 @@
/* choose prng mode */
if (prng_mode != PRNG_MODE_TDES) {
- /* check for MSA5 support for PPNO operations */
- if (!cpacf_query_func(CPACF_PPNO, CPACF_PPNO_SHA512_DRNG_GEN)) {
+ /* check for MSA5 support for PRNO operations */
+ if (!cpacf_query_func(CPACF_PRNO, CPACF_PRNO_SHA512_DRNG_GEN)) {
if (prng_mode == PRNG_MODE_SHA512) {
pr_err("The prng module cannot "
"start in SHA-512 mode\n");
diff --git a/arch/s390/include/asm/Kbuild b/arch/s390/include/asm/Kbuild
index 8aea32f..7e3481e 100644
--- a/arch/s390/include/asm/Kbuild
+++ b/arch/s390/include/asm/Kbuild
@@ -1,8 +1,14 @@
generic-y += asm-offsets.h
generic-y += clkdev.h
generic-y += dma-contiguous.h
+generic-y += div64.h
+generic-y += emergency-restart.h
generic-y += export.h
+generic-y += irq_regs.h
generic-y += irq_work.h
+generic-y += kmap_types.h
+generic-y += local.h
+generic-y += local64.h
generic-y += mcs_spinlock.h
generic-y += mm-arch-hooks.h
generic-y += preempt.h
diff --git a/arch/s390/include/asm/archrandom.h b/arch/s390/include/asm/archrandom.h
new file mode 100644
index 0000000..6033901
--- /dev/null
+++ b/arch/s390/include/asm/archrandom.h
@@ -0,0 +1,69 @@
+/*
+ * Kernel interface for the s390 arch_random_* functions
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author: Harald Freudenberger <freude@de.ibm.com>
+ *
+ */
+
+#ifndef _ASM_S390_ARCHRANDOM_H
+#define _ASM_S390_ARCHRANDOM_H
+
+#ifdef CONFIG_ARCH_RANDOM
+
+#include <linux/static_key.h>
+#include <linux/atomic.h>
+#include <asm/cpacf.h>
+
+DECLARE_STATIC_KEY_FALSE(s390_arch_random_available);
+extern atomic64_t s390_arch_random_counter;
+
+static void s390_arch_random_generate(u8 *buf, unsigned int nbytes)
+{
+ cpacf_trng(NULL, 0, buf, nbytes);
+ atomic64_add(nbytes, &s390_arch_random_counter);
+}
+
+static inline bool arch_has_random(void)
+{
+ if (static_branch_likely(&s390_arch_random_available))
+ return true;
+ return false;
+}
+
+static inline bool arch_has_random_seed(void)
+{
+ return arch_has_random();
+}
+
+static inline bool arch_get_random_long(unsigned long *v)
+{
+ if (static_branch_likely(&s390_arch_random_available)) {
+ s390_arch_random_generate((u8 *)v, sizeof(*v));
+ return true;
+ }
+ return false;
+}
+
+static inline bool arch_get_random_int(unsigned int *v)
+{
+ if (static_branch_likely(&s390_arch_random_available)) {
+ s390_arch_random_generate((u8 *)v, sizeof(*v));
+ return true;
+ }
+ return false;
+}
+
+static inline bool arch_get_random_seed_long(unsigned long *v)
+{
+ return arch_get_random_long(v);
+}
+
+static inline bool arch_get_random_seed_int(unsigned int *v)
+{
+ return arch_get_random_int(v);
+}
+
+#endif /* CONFIG_ARCH_RANDOM */
+#endif /* _ASM_S390_ARCHRANDOM_H */
diff --git a/arch/s390/include/asm/atomic_ops.h b/arch/s390/include/asm/atomic_ops.h
index ac9e2b9..ba6d294 100644
--- a/arch/s390/include/asm/atomic_ops.h
+++ b/arch/s390/include/asm/atomic_ops.h
@@ -111,20 +111,22 @@
static inline int __atomic_cmpxchg(int *ptr, int old, int new)
{
- asm volatile(
- " cs %[old],%[new],%[ptr]"
- : [old] "+d" (old), [ptr] "+Q" (*ptr)
- : [new] "d" (new) : "cc", "memory");
- return old;
+ return __sync_val_compare_and_swap(ptr, old, new);
+}
+
+static inline int __atomic_cmpxchg_bool(int *ptr, int old, int new)
+{
+ return __sync_bool_compare_and_swap(ptr, old, new);
}
static inline long __atomic64_cmpxchg(long *ptr, long old, long new)
{
- asm volatile(
- " csg %[old],%[new],%[ptr]"
- : [old] "+d" (old), [ptr] "+Q" (*ptr)
- : [new] "d" (new) : "cc", "memory");
- return old;
+ return __sync_val_compare_and_swap(ptr, old, new);
+}
+
+static inline long __atomic64_cmpxchg_bool(long *ptr, long old, long new)
+{
+ return __sync_bool_compare_and_swap(ptr, old, new);
}
#endif /* __ARCH_S390_ATOMIC_OPS__ */
diff --git a/arch/s390/include/asm/bitops.h b/arch/s390/include/asm/bitops.h
index d92047d..99902b7 100644
--- a/arch/s390/include/asm/bitops.h
+++ b/arch/s390/include/asm/bitops.h
@@ -15,14 +15,6 @@
* end up numbered:
* |63..............0|127............64|191...........128|255...........192|
*
- * There are a few little-endian macros used mostly for filesystem
- * bitmaps, these work on similar bit array layouts, but byte-oriented:
- * |7...0|15...8|23...16|31...24|39...32|47...40|55...48|63...56|
- *
- * The main difference is that bit 3-5 in the bit number field needs to be
- * reversed compared to the big-endian bit fields. This can be achieved by
- * XOR with 0x38.
- *
* We also have special functions which work with an MSB0 encoding.
* The bits are numbered:
* |0..............63|64............127|128...........191|192...........255|
@@ -253,6 +245,11 @@
unsigned long find_next_bit_inv(const unsigned long *addr, unsigned long size,
unsigned long offset);
+#define for_each_set_bit_inv(bit, addr, size) \
+ for ((bit) = find_first_bit_inv((addr), (size)); \
+ (bit) < (size); \
+ (bit) = find_next_bit_inv((addr), (size), (bit) + 1))
+
static inline void set_bit_inv(unsigned long nr, volatile unsigned long *ptr)
{
return set_bit(nr ^ (BITS_PER_LONG - 1), ptr);
diff --git a/arch/s390/include/asm/bug.h b/arch/s390/include/asm/bug.h
index bf90d1f..1bbd9db 100644
--- a/arch/s390/include/asm/bug.h
+++ b/arch/s390/include/asm/bug.h
@@ -46,8 +46,8 @@
unreachable(); \
} while (0)
-#define __WARN_TAINT(taint) do { \
- __EMIT_BUG(BUGFLAG_TAINT(taint)); \
+#define __WARN_FLAGS(flags) do { \
+ __EMIT_BUG(BUGFLAG_WARNING|(flags)); \
} while (0)
#define WARN_ON(x) ({ \
diff --git a/arch/s390/include/asm/cio.h b/arch/s390/include/asm/cio.h
index f7ed88c..7a38ca8 100644
--- a/arch/s390/include/asm/cio.h
+++ b/arch/s390/include/asm/cio.h
@@ -33,6 +33,24 @@
__u32 cda;
} __attribute__ ((packed,aligned(8)));
+/**
+ * struct ccw0 - channel command word
+ * @cmd_code: command code
+ * @cda: data address
+ * @flags: flags, like IDA addressing, etc.
+ * @reserved: will be ignored
+ * @count: byte count
+ *
+ * The format-0 ccw structure.
+ */
+struct ccw0 {
+ __u8 cmd_code;
+ __u32 cda : 24;
+ __u8 flags;
+ __u8 reserved;
+ __u16 count;
+} __packed __aligned(8);
+
#define CCW_FLAG_DC 0x80
#define CCW_FLAG_CC 0x40
#define CCW_FLAG_SLI 0x20
diff --git a/arch/s390/include/asm/cpacf.h b/arch/s390/include/asm/cpacf.h
index e2dfbf2..e06f255 100644
--- a/arch/s390/include/asm/cpacf.h
+++ b/arch/s390/include/asm/cpacf.h
@@ -25,7 +25,8 @@
#define CPACF_KMO 0xb92b /* MSA4 */
#define CPACF_PCC 0xb92c /* MSA4 */
#define CPACF_KMCTR 0xb92d /* MSA4 */
-#define CPACF_PPNO 0xb93c /* MSA5 */
+#define CPACF_PRNO 0xb93c /* MSA5 */
+#define CPACF_KMA 0xb929 /* MSA8 */
/*
* En/decryption modifier bits
@@ -123,12 +124,14 @@
#define CPACF_PCKMO_ENC_AES_256_KEY 0x14
/*
- * Function codes for the PPNO (PERFORM PSEUDORANDOM NUMBER OPERATION)
+ * Function codes for the PRNO (PERFORM RANDOM NUMBER OPERATION)
* instruction
*/
-#define CPACF_PPNO_QUERY 0x00
-#define CPACF_PPNO_SHA512_DRNG_GEN 0x03
-#define CPACF_PPNO_SHA512_DRNG_SEED 0x83
+#define CPACF_PRNO_QUERY 0x00
+#define CPACF_PRNO_SHA512_DRNG_GEN 0x03
+#define CPACF_PRNO_SHA512_DRNG_SEED 0x83
+#define CPACF_PRNO_TRNG_Q_R2C_RATIO 0x70
+#define CPACF_PRNO_TRNG 0x72
typedef struct { unsigned char bytes[16]; } cpacf_mask_t;
@@ -149,8 +152,8 @@
asm volatile(
" spm 0\n" /* pckmo doesn't change the cc */
- /* Parameter registers are ignored, but may not be 0 */
- "0: .insn rrf,%[opc] << 16,2,2,2,0\n"
+ /* Parameter regs are ignored, but must be nonzero and unique */
+ "0: .insn rrf,%[opc] << 16,2,4,6,0\n"
" brc 1,0b\n" /* handle partial completion */
: "=m" (*mask)
: [fc] "d" (r0), [pba] "a" (r1), [opc] "i" (opcode)
@@ -173,7 +176,7 @@
case CPACF_PCC:
case CPACF_KMCTR:
return test_facility(77); /* check for MSA4 */
- case CPACF_PPNO:
+ case CPACF_PRNO:
return test_facility(57); /* check for MSA5 */
default:
BUG();
@@ -373,18 +376,18 @@
}
/**
- * cpacf_ppno() - executes the PPNO (PERFORM PSEUDORANDOM NUMBER OPERATION)
+ * cpacf_prno() - executes the PRNO (PERFORM RANDOM NUMBER OPERATION)
* instruction
- * @func: the function code passed to PPNO; see CPACF_PPNO_xxx defines
+ * @func: the function code passed to PRNO; see CPACF_PRNO_xxx defines
* @param: address of parameter block; see POP for details on each func
* @dest: address of destination memory area
* @dest_len: size of destination memory area in bytes
* @seed: address of seed data
* @seed_len: size of seed data in bytes
*/
-static inline void cpacf_ppno(unsigned long func, void *param,
- u8 *dest, long dest_len,
- const u8 *seed, long seed_len)
+static inline void cpacf_prno(unsigned long func, void *param,
+ u8 *dest, unsigned long dest_len,
+ const u8 *seed, unsigned long seed_len)
{
register unsigned long r0 asm("0") = (unsigned long) func;
register unsigned long r1 asm("1") = (unsigned long) param;
@@ -398,7 +401,32 @@
" brc 1,0b\n" /* handle partial completion */
: [dst] "+a" (r2), [dlen] "+d" (r3)
: [fc] "d" (r0), [pba] "a" (r1),
- [seed] "a" (r4), [slen] "d" (r5), [opc] "i" (CPACF_PPNO)
+ [seed] "a" (r4), [slen] "d" (r5), [opc] "i" (CPACF_PRNO)
+ : "cc", "memory");
+}
+
+/**
+ * cpacf_trng() - executes the TRNG subfunction of the PRNO instruction
+ * @ucbuf: buffer for unconditioned data
+ * @ucbuf_len: amount of unconditioned data to fetch in bytes
+ * @cbuf: buffer for conditioned data
+ * @cbuf_len: amount of conditioned data to fetch in bytes
+ */
+static inline void cpacf_trng(u8 *ucbuf, unsigned long ucbuf_len,
+ u8 *cbuf, unsigned long cbuf_len)
+{
+ register unsigned long r0 asm("0") = (unsigned long) CPACF_PRNO_TRNG;
+ register unsigned long r2 asm("2") = (unsigned long) ucbuf;
+ register unsigned long r3 asm("3") = (unsigned long) ucbuf_len;
+ register unsigned long r4 asm("4") = (unsigned long) cbuf;
+ register unsigned long r5 asm("5") = (unsigned long) cbuf_len;
+
+ asm volatile (
+ "0: .insn rre,%[opc] << 16,%[ucbuf],%[cbuf]\n"
+ " brc 1,0b\n" /* handle partial completion */
+ : [ucbuf] "+a" (r2), [ucbuflen] "+d" (r3),
+ [cbuf] "+a" (r4), [cbuflen] "+d" (r5)
+ : [fc] "d" (r0), [opc] "i" (CPACF_PRNO)
: "cc", "memory");
}
diff --git a/arch/s390/include/asm/cpu_mf.h b/arch/s390/include/asm/cpu_mf.h
index d1e0707..05480e4 100644
--- a/arch/s390/include/asm/cpu_mf.h
+++ b/arch/s390/include/asm/cpu_mf.h
@@ -20,9 +20,11 @@
#define CPU_MF_INT_SF_PRA (1 << 29) /* program request alert */
#define CPU_MF_INT_SF_SACA (1 << 23) /* sampler auth. change alert */
#define CPU_MF_INT_SF_LSDA (1 << 22) /* loss of sample data alert */
+#define CPU_MF_INT_CF_MTDA (1 << 15) /* loss of MT ctr. data alert */
#define CPU_MF_INT_CF_CACA (1 << 7) /* counter auth. change alert */
#define CPU_MF_INT_CF_LCDA (1 << 6) /* loss of counter data alert */
-#define CPU_MF_INT_CF_MASK (CPU_MF_INT_CF_CACA|CPU_MF_INT_CF_LCDA)
+#define CPU_MF_INT_CF_MASK (CPU_MF_INT_CF_MTDA|CPU_MF_INT_CF_CACA| \
+ CPU_MF_INT_CF_LCDA)
#define CPU_MF_INT_SF_MASK (CPU_MF_INT_SF_IAE|CPU_MF_INT_SF_ISE| \
CPU_MF_INT_SF_PRA|CPU_MF_INT_SF_SACA| \
CPU_MF_INT_SF_LSDA)
@@ -172,7 +174,7 @@
/* Extract CPU counter */
static inline int __ecctr(u64 ctr, u64 *content)
{
- register u64 _content asm("4") = 0;
+ u64 _content;
int cc;
asm volatile (
diff --git a/arch/s390/include/asm/div64.h b/arch/s390/include/asm/div64.h
deleted file mode 100644
index 6cd978c..0000000
--- a/arch/s390/include/asm/div64.h
+++ /dev/null
@@ -1 +0,0 @@
-#include <asm-generic/div64.h>
diff --git a/arch/s390/include/asm/elf.h b/arch/s390/include/asm/elf.h
index 1d48880..e8f6230 100644
--- a/arch/s390/include/asm/elf.h
+++ b/arch/s390/include/asm/elf.h
@@ -105,6 +105,7 @@
#define HWCAP_S390_VXRS 2048
#define HWCAP_S390_VXRS_BCD 4096
#define HWCAP_S390_VXRS_EXT 8192
+#define HWCAP_S390_GS 16384
/* Internal bits, not exposed via elf */
#define HWCAP_INT_SIE 1UL
diff --git a/arch/s390/include/asm/emergency-restart.h b/arch/s390/include/asm/emergency-restart.h
deleted file mode 100644
index 108d8c4..0000000
--- a/arch/s390/include/asm/emergency-restart.h
+++ /dev/null
@@ -1,6 +0,0 @@
-#ifndef _ASM_EMERGENCY_RESTART_H
-#define _ASM_EMERGENCY_RESTART_H
-
-#include <asm-generic/emergency-restart.h>
-
-#endif /* _ASM_EMERGENCY_RESTART_H */
diff --git a/arch/s390/include/asm/extable.h b/arch/s390/include/asm/extable.h
new file mode 100644
index 0000000..16cfe2d
--- /dev/null
+++ b/arch/s390/include/asm/extable.h
@@ -0,0 +1,28 @@
+#ifndef __S390_EXTABLE_H
+#define __S390_EXTABLE_H
+/*
+ * The exception table consists of pairs of addresses: the first is the
+ * address of an instruction that is allowed to fault, and the second is
+ * the address at which the program should continue. No registers are
+ * modified, so it is entirely up to the continuation code to figure out
+ * what to do.
+ *
+ * All the routines below use bits of fixup code that are out of line
+ * with the main instruction path. This means when everything is well,
+ * we don't even have to jump over them. Further, they do not intrude
+ * on our cache or tlb entries.
+ */
+
+struct exception_table_entry
+{
+ int insn, fixup;
+};
+
+static inline unsigned long extable_fixup(const struct exception_table_entry *x)
+{
+ return (unsigned long)&x->fixup + x->fixup;
+}
+
+#define ARCH_HAS_RELATIVE_EXTABLE
+
+#endif
diff --git a/arch/s390/include/asm/facility.h b/arch/s390/include/asm/facility.h
index 09b406d..cb60d5c 100644
--- a/arch/s390/include/asm/facility.h
+++ b/arch/s390/include/asm/facility.h
@@ -8,14 +8,11 @@
#define __ASM_FACILITY_H
#include <generated/facilities.h>
-
-#ifndef __ASSEMBLY__
-
#include <linux/string.h>
#include <linux/preempt.h>
#include <asm/lowcore.h>
-#define MAX_FACILITY_BIT (256*8) /* stfle_fac_list has 256 bytes */
+#define MAX_FACILITY_BIT (sizeof(((struct lowcore *)0)->stfle_fac_list) * 8)
static inline int __test_facility(unsigned long nr, void *facilities)
{
@@ -72,5 +69,4 @@
preempt_enable();
}
-#endif /* __ASSEMBLY__ */
#endif /* __ASM_FACILITY_H */
diff --git a/arch/s390/include/asm/irq_regs.h b/arch/s390/include/asm/irq_regs.h
deleted file mode 100644
index 3dd9c0b..0000000
--- a/arch/s390/include/asm/irq_regs.h
+++ /dev/null
@@ -1 +0,0 @@
-#include <asm-generic/irq_regs.h>
diff --git a/arch/s390/include/asm/isc.h b/arch/s390/include/asm/isc.h
index 68d7d68..8a0b721 100644
--- a/arch/s390/include/asm/isc.h
+++ b/arch/s390/include/asm/isc.h
@@ -16,6 +16,7 @@
#define CONSOLE_ISC 1 /* console I/O subchannel */
#define EADM_SCH_ISC 4 /* EADM subchannels */
#define CHSC_SCH_ISC 7 /* CHSC subchannels */
+#define VFIO_CCW_ISC IO_SCH_ISC /* VFIO-CCW I/O subchannels */
/* Adapter interrupts. */
#define QDIO_AIRQ_ISC IO_SCH_ISC /* I/O subchannel in qdio mode */
#define PCI_ISC 2 /* PCI I/O subchannels */
diff --git a/arch/s390/include/asm/kmap_types.h b/arch/s390/include/asm/kmap_types.h
deleted file mode 100644
index 0a88622..0000000
--- a/arch/s390/include/asm/kmap_types.h
+++ /dev/null
@@ -1,6 +0,0 @@
-#ifndef _ASM_KMAP_TYPES_H
-#define _ASM_KMAP_TYPES_H
-
-#include <asm-generic/kmap_types.h>
-
-#endif
diff --git a/arch/s390/include/asm/local.h b/arch/s390/include/asm/local.h
deleted file mode 100644
index c11c530..0000000
--- a/arch/s390/include/asm/local.h
+++ /dev/null
@@ -1 +0,0 @@
-#include <asm-generic/local.h>
diff --git a/arch/s390/include/asm/local64.h b/arch/s390/include/asm/local64.h
deleted file mode 100644
index 36c93b5..0000000
--- a/arch/s390/include/asm/local64.h
+++ /dev/null
@@ -1 +0,0 @@
-#include <asm-generic/local64.h>
diff --git a/arch/s390/include/asm/lowcore.h b/arch/s390/include/asm/lowcore.h
index 61261e0..8a5b082 100644
--- a/arch/s390/include/asm/lowcore.h
+++ b/arch/s390/include/asm/lowcore.h
@@ -157,8 +157,8 @@
__u64 stfle_fac_list[32]; /* 0x0f00 */
__u8 pad_0x1000[0x11b0-0x1000]; /* 0x1000 */
- /* Pointer to vector register save area */
- __u64 vector_save_area_addr; /* 0x11b0 */
+ /* Pointer to the machine check extended save area */
+ __u64 mcesad; /* 0x11b0 */
/* 64 bit extparam used for pfault/diag 250: defined by architecture */
__u64 ext_params2; /* 0x11B8 */
@@ -182,10 +182,7 @@
/* Transaction abort diagnostic block */
__u8 pgm_tdb[256]; /* 0x1800 */
- __u8 pad_0x1900[0x1c00-0x1900]; /* 0x1900 */
-
- /* Software defined save area for vector registers */
- __u8 vector_save_area[1024]; /* 0x1c00 */
+ __u8 pad_0x1900[0x2000-0x1900]; /* 0x1900 */
} __packed;
#define S390_lowcore (*((struct lowcore *) 0))
diff --git a/arch/s390/include/asm/mman.h b/arch/s390/include/asm/mman.h
index b55a59e..b79813d 100644
--- a/arch/s390/include/asm/mman.h
+++ b/arch/s390/include/asm/mman.h
@@ -8,8 +8,4 @@
#include <uapi/asm/mman.h>
-#ifndef __ASSEMBLY__
-int s390_mmap_check(unsigned long addr, unsigned long len, unsigned long flags);
-#define arch_mmap_check(addr, len, flags) s390_mmap_check(addr, len, flags)
-#endif
#endif /* __S390_MMAN_H__ */
diff --git a/arch/s390/include/asm/mmu.h b/arch/s390/include/asm/mmu.h
index bea785d..bd6f303 100644
--- a/arch/s390/include/asm/mmu.h
+++ b/arch/s390/include/asm/mmu.h
@@ -22,6 +22,8 @@
unsigned int has_pgste:1;
/* The mmu context uses storage keys. */
unsigned int use_skey:1;
+ /* The mmu context uses CMMA. */
+ unsigned int use_cmma:1;
} mm_context_t;
#define INIT_MM_CONTEXT(name) \
diff --git a/arch/s390/include/asm/mmu_context.h b/arch/s390/include/asm/mmu_context.h
index 6e31d87..8712e11 100644
--- a/arch/s390/include/asm/mmu_context.h
+++ b/arch/s390/include/asm/mmu_context.h
@@ -28,6 +28,7 @@
mm->context.alloc_pgste = page_table_allocate_pgste;
mm->context.has_pgste = 0;
mm->context.use_skey = 0;
+ mm->context.use_cmma = 0;
#endif
switch (mm->context.asce_limit) {
case 1UL << 42:
@@ -156,10 +157,4 @@
/* by default, allow everything */
return true;
}
-
-static inline bool arch_pte_access_permitted(pte_t pte, bool write)
-{
- /* by default, allow everything */
- return true;
-}
#endif /* __S390_MMU_CONTEXT_H */
diff --git a/arch/s390/include/asm/nmi.h b/arch/s390/include/asm/nmi.h
index b75fd91..e3e8895 100644
--- a/arch/s390/include/asm/nmi.h
+++ b/arch/s390/include/asm/nmi.h
@@ -58,7 +58,9 @@
u64 ie : 1; /* 32 indirect storage error */
u64 ar : 1; /* 33 access register validity */
u64 da : 1; /* 34 delayed access exception */
- u64 : 7; /* 35-41 */
+ u64 : 1; /* 35 */
+ u64 gs : 1; /* 36 guarded storage registers */
+ u64 : 5; /* 37-41 */
u64 pr : 1; /* 42 tod programmable register validity */
u64 fc : 1; /* 43 fp control register validity */
u64 ap : 1; /* 44 ancillary report */
@@ -69,6 +71,14 @@
};
};
+#define MCESA_ORIGIN_MASK (~0x3ffUL)
+#define MCESA_LC_MASK (0xfUL)
+
+struct mcesa {
+ u8 vector_save_area[1024];
+ u8 guarded_storage_save_area[32];
+};
+
struct pt_regs;
extern void s390_handle_mcck(void);
diff --git a/arch/s390/include/asm/page-states.h b/arch/s390/include/asm/page-states.h
new file mode 100644
index 0000000..42267a2
--- /dev/null
+++ b/arch/s390/include/asm/page-states.h
@@ -0,0 +1,19 @@
+/*
+ * Copyright IBM Corp. 2017
+ * Author(s): Claudio Imbrenda <imbrenda@linux.vnet.ibm.com>
+ */
+
+#ifndef PAGE_STATES_H
+#define PAGE_STATES_H
+
+#define ESSA_GET_STATE 0
+#define ESSA_SET_STABLE 1
+#define ESSA_SET_UNUSED 2
+#define ESSA_SET_VOLATILE 3
+#define ESSA_SET_POT_VOLATILE 4
+#define ESSA_SET_STABLE_RESIDENT 5
+#define ESSA_SET_STABLE_IF_RESIDENT 6
+
+#define ESSA_MAX ESSA_SET_STABLE_IF_RESIDENT
+
+#endif
diff --git a/arch/s390/include/asm/perf_event.h b/arch/s390/include/asm/perf_event.h
index c64c0be..dd32beb 100644
--- a/arch/s390/include/asm/perf_event.h
+++ b/arch/s390/include/asm/perf_event.h
@@ -1,7 +1,7 @@
/*
* Performance event support - s390 specific definitions.
*
- * Copyright IBM Corp. 2009, 2013
+ * Copyright IBM Corp. 2009, 2017
* Author(s): Martin Schwidefsky <schwidefsky@de.ibm.com>
* Hendrik Brueckner <brueckner@linux.vnet.ibm.com>
*/
@@ -47,7 +47,7 @@
};
/* Perf PMU definitions for the counter facility */
-#define PERF_CPUM_CF_MAX_CTR 256
+#define PERF_CPUM_CF_MAX_CTR 0xffffUL /* Max ctr for ECCTR */
/* Perf PMU definitions for the sampling facility */
#define PERF_CPUM_SF_MAX_CTR 2
diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index ecec682..e6e3b88 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -372,10 +372,12 @@
#define PGSTE_VSIE_BIT 0x0000200000000000UL /* ref'd in a shadow table */
/* Guest Page State used for virtualization */
-#define _PGSTE_GPS_ZERO 0x0000000080000000UL
-#define _PGSTE_GPS_USAGE_MASK 0x0000000003000000UL
-#define _PGSTE_GPS_USAGE_STABLE 0x0000000000000000UL
-#define _PGSTE_GPS_USAGE_UNUSED 0x0000000001000000UL
+#define _PGSTE_GPS_ZERO 0x0000000080000000UL
+#define _PGSTE_GPS_USAGE_MASK 0x0000000003000000UL
+#define _PGSTE_GPS_USAGE_STABLE 0x0000000000000000UL
+#define _PGSTE_GPS_USAGE_UNUSED 0x0000000001000000UL
+#define _PGSTE_GPS_USAGE_POT_VOLATILE 0x0000000002000000UL
+#define _PGSTE_GPS_USAGE_VOLATILE _PGSTE_GPS_USAGE_MASK
/*
* A user page table pointer has the space-switch-event bit, the
@@ -1041,6 +1043,12 @@
int get_guest_storage_key(struct mm_struct *mm, unsigned long addr,
unsigned char *key);
+int set_pgste_bits(struct mm_struct *mm, unsigned long addr,
+ unsigned long bits, unsigned long value);
+int get_pgste(struct mm_struct *mm, unsigned long hva, unsigned long *pgstep);
+int pgste_perform_essa(struct mm_struct *mm, unsigned long hva, int orc,
+ unsigned long *oldpte, unsigned long *oldpgste);
+
/*
* Certain architectures need to do special things when PTEs
* within a page table are directly modified. Thus, the following
diff --git a/arch/s390/include/asm/pkey.h b/arch/s390/include/asm/pkey.h
index b48aef4..4c48459 100644
--- a/arch/s390/include/asm/pkey.h
+++ b/arch/s390/include/asm/pkey.h
@@ -87,4 +87,25 @@
int pkey_skey2pkey(const struct pkey_seckey *seckey,
struct pkey_protkey *protkey);
+/*
+ * Verify the given secure key for being able to be useable with
+ * the pkey module. Check for correct key type and check for having at
+ * least one crypto card being able to handle this key (master key
+ * or old master key verification pattern matches).
+ * Return some info about the key: keysize in bits, keytype (currently
+ * only AES), flag if key is wrapped with an old MKVP.
+ * @param seckey pointer to buffer with the input secure key
+ * @param pcardnr pointer to cardnr, receives the card number on success
+ * @param pdomain pointer to domain, receives the domain number on success
+ * @param pkeysize pointer to keysize, receives the bitsize of the key
+ * @param pattributes pointer to attributes, receives additional info
+ * PKEY_VERIFY_ATTR_AES if the key is an AES key
+ * PKEY_VERIFY_ATTR_OLD_MKVP if key has old mkvp stored in
+ * @return 0 on success, negative errno value on failure. If no card could
+ * be found which is able to handle this key, -ENODEV is returned.
+ */
+int pkey_verifykey(const struct pkey_seckey *seckey,
+ u16 *pcardnr, u16 *pdomain,
+ u16 *pkeysize, u32 *pattributes);
+
#endif /* _KAPI_PKEY_H */
diff --git a/arch/s390/include/asm/processor.h b/arch/s390/include/asm/processor.h
index e498871..60d395f 100644
--- a/arch/s390/include/asm/processor.h
+++ b/arch/s390/include/asm/processor.h
@@ -91,14 +91,15 @@
* User space process size: 2GB for 31 bit, 4TB or 8PT for 64 bit.
*/
-#define TASK_SIZE_OF(tsk) ((tsk)->mm ? \
- (tsk)->mm->context.asce_limit : TASK_MAX_SIZE)
+#define TASK_SIZE_OF(tsk) (test_tsk_thread_flag(tsk, TIF_31BIT) ? \
+ (1UL << 31) : (1UL << 53))
#define TASK_UNMAPPED_BASE (test_thread_flag(TIF_31BIT) ? \
(1UL << 30) : (1UL << 41))
#define TASK_SIZE TASK_SIZE_OF(current)
-#define TASK_MAX_SIZE (1UL << 53)
+#define TASK_SIZE_MAX (1UL << 53)
-#define STACK_TOP (1UL << (test_thread_flag(TIF_31BIT) ? 31:42))
+#define STACK_TOP (test_thread_flag(TIF_31BIT) ? \
+ (1UL << 31) : (1UL << 42))
#define STACK_TOP_MAX (1UL << 42)
#define HAVE_ARCH_PICK_MMAP_LAYOUT
@@ -135,6 +136,8 @@
struct list_head list;
/* cpu runtime instrumentation */
struct runtime_instr_cb *ri_cb;
+ struct gs_cb *gs_cb; /* Current guarded storage cb */
+ struct gs_cb *gs_bc_cb; /* Broadcast guarded storage cb */
unsigned char trap_tdb[256]; /* Transaction abort diagnose block */
/*
* Warning: 'fpu' is dynamically-sized. It *MUST* be at
@@ -215,6 +218,9 @@
/* Free all resources held by a thread. */
extern void release_thread(struct task_struct *);
+/* Free guarded storage control block for current */
+void exit_thread_gs(void);
+
/*
* Return saved PC of a blocked thread.
*/
diff --git a/arch/s390/include/asm/setup.h b/arch/s390/include/asm/setup.h
index 30bdb5a..cd78155 100644
--- a/arch/s390/include/asm/setup.h
+++ b/arch/s390/include/asm/setup.h
@@ -29,8 +29,8 @@
#define MACHINE_FLAG_TE _BITUL(11)
#define MACHINE_FLAG_TLB_LC _BITUL(12)
#define MACHINE_FLAG_VX _BITUL(13)
-#define MACHINE_FLAG_CAD _BITUL(14)
-#define MACHINE_FLAG_NX _BITUL(15)
+#define MACHINE_FLAG_NX _BITUL(14)
+#define MACHINE_FLAG_GS _BITUL(15)
#define LPP_MAGIC _BITUL(31)
#define LPP_PFAULT_PID_MASK _AC(0xffffffff, UL)
@@ -68,8 +68,8 @@
#define MACHINE_HAS_TE (S390_lowcore.machine_flags & MACHINE_FLAG_TE)
#define MACHINE_HAS_TLB_LC (S390_lowcore.machine_flags & MACHINE_FLAG_TLB_LC)
#define MACHINE_HAS_VX (S390_lowcore.machine_flags & MACHINE_FLAG_VX)
-#define MACHINE_HAS_CAD (S390_lowcore.machine_flags & MACHINE_FLAG_CAD)
#define MACHINE_HAS_NX (S390_lowcore.machine_flags & MACHINE_FLAG_NX)
+#define MACHINE_HAS_GS (S390_lowcore.machine_flags & MACHINE_FLAG_GS)
/*
* Console mode. Override with conmode=
diff --git a/arch/s390/include/asm/sparsemem.h b/arch/s390/include/asm/sparsemem.h
index 487428b..334e279 100644
--- a/arch/s390/include/asm/sparsemem.h
+++ b/arch/s390/include/asm/sparsemem.h
@@ -2,6 +2,6 @@
#define _ASM_S390_SPARSEMEM_H
#define SECTION_SIZE_BITS 28
-#define MAX_PHYSMEM_BITS 46
+#define MAX_PHYSMEM_BITS CONFIG_MAX_PHYSMEM_BITS
#endif /* _ASM_S390_SPARSEMEM_H */
diff --git a/arch/s390/include/asm/spinlock.h b/arch/s390/include/asm/spinlock.h
index ffc4504..f7838ec 100644
--- a/arch/s390/include/asm/spinlock.h
+++ b/arch/s390/include/asm/spinlock.h
@@ -10,6 +10,7 @@
#define __ASM_SPINLOCK_H
#include <linux/smp.h>
+#include <asm/atomic_ops.h>
#include <asm/barrier.h>
#include <asm/processor.h>
@@ -17,12 +18,6 @@
extern int spin_retry;
-static inline int
-_raw_compare_and_swap(unsigned int *lock, unsigned int old, unsigned int new)
-{
- return __sync_bool_compare_and_swap(lock, old, new);
-}
-
#ifndef CONFIG_SMP
static inline bool arch_vcpu_is_preempted(int cpu) { return false; }
#else
@@ -40,7 +35,7 @@
* (the type definitions are in asm/spinlock_types.h)
*/
-void arch_lock_relax(unsigned int cpu);
+void arch_lock_relax(int cpu);
void arch_spin_lock_wait(arch_spinlock_t *);
int arch_spin_trylock_retry(arch_spinlock_t *);
@@ -70,7 +65,7 @@
{
barrier();
return likely(arch_spin_value_unlocked(*lp) &&
- _raw_compare_and_swap(&lp->lock, 0, SPINLOCK_LOCKVAL));
+ __atomic_cmpxchg_bool(&lp->lock, 0, SPINLOCK_LOCKVAL));
}
static inline void arch_spin_lock(arch_spinlock_t *lp)
@@ -95,7 +90,7 @@
static inline void arch_spin_unlock(arch_spinlock_t *lp)
{
- typecheck(unsigned int, lp->lock);
+ typecheck(int, lp->lock);
asm volatile(
"st %1,%0\n"
: "+Q" (lp->lock)
@@ -141,16 +136,16 @@
static inline int arch_read_trylock_once(arch_rwlock_t *rw)
{
- unsigned int old = ACCESS_ONCE(rw->lock);
- return likely((int) old >= 0 &&
- _raw_compare_and_swap(&rw->lock, old, old + 1));
+ int old = ACCESS_ONCE(rw->lock);
+ return likely(old >= 0 &&
+ __atomic_cmpxchg_bool(&rw->lock, old, old + 1));
}
static inline int arch_write_trylock_once(arch_rwlock_t *rw)
{
- unsigned int old = ACCESS_ONCE(rw->lock);
+ int old = ACCESS_ONCE(rw->lock);
return likely(old == 0 &&
- _raw_compare_and_swap(&rw->lock, 0, 0x80000000));
+ __atomic_cmpxchg_bool(&rw->lock, 0, 0x80000000));
}
#ifdef CONFIG_HAVE_MARCH_Z196_FEATURES
@@ -161,9 +156,9 @@
#define __RAW_LOCK(ptr, op_val, op_string) \
({ \
- unsigned int old_val; \
+ int old_val; \
\
- typecheck(unsigned int *, ptr); \
+ typecheck(int *, ptr); \
asm volatile( \
op_string " %0,%2,%1\n" \
"bcr 14,0\n" \
@@ -175,9 +170,9 @@
#define __RAW_UNLOCK(ptr, op_val, op_string) \
({ \
- unsigned int old_val; \
+ int old_val; \
\
- typecheck(unsigned int *, ptr); \
+ typecheck(int *, ptr); \
asm volatile( \
op_string " %0,%2,%1\n" \
: "=d" (old_val), "+Q" (*ptr) \
@@ -187,14 +182,14 @@
})
extern void _raw_read_lock_wait(arch_rwlock_t *lp);
-extern void _raw_write_lock_wait(arch_rwlock_t *lp, unsigned int prev);
+extern void _raw_write_lock_wait(arch_rwlock_t *lp, int prev);
static inline void arch_read_lock(arch_rwlock_t *rw)
{
- unsigned int old;
+ int old;
old = __RAW_LOCK(&rw->lock, 1, __RAW_OP_ADD);
- if ((int) old < 0)
+ if (old < 0)
_raw_read_lock_wait(rw);
}
@@ -205,7 +200,7 @@
static inline void arch_write_lock(arch_rwlock_t *rw)
{
- unsigned int old;
+ int old;
old = __RAW_LOCK(&rw->lock, 0x80000000, __RAW_OP_OR);
if (old != 0)
@@ -232,11 +227,11 @@
static inline void arch_read_unlock(arch_rwlock_t *rw)
{
- unsigned int old;
+ int old;
do {
old = ACCESS_ONCE(rw->lock);
- } while (!_raw_compare_and_swap(&rw->lock, old, old - 1));
+ } while (!__atomic_cmpxchg_bool(&rw->lock, old, old - 1));
}
static inline void arch_write_lock(arch_rwlock_t *rw)
@@ -248,7 +243,7 @@
static inline void arch_write_unlock(arch_rwlock_t *rw)
{
- typecheck(unsigned int, rw->lock);
+ typecheck(int, rw->lock);
rw->owner = 0;
asm volatile(
diff --git a/arch/s390/include/asm/spinlock_types.h b/arch/s390/include/asm/spinlock_types.h
index d84b693..fe755ee 100644
--- a/arch/s390/include/asm/spinlock_types.h
+++ b/arch/s390/include/asm/spinlock_types.h
@@ -6,14 +6,14 @@
#endif
typedef struct {
- unsigned int lock;
+ int lock;
} __attribute__ ((aligned (4))) arch_spinlock_t;
#define __ARCH_SPIN_LOCK_UNLOCKED { .lock = 0, }
typedef struct {
- unsigned int lock;
- unsigned int owner;
+ int lock;
+ int owner;
} arch_rwlock_t;
#define __ARCH_RW_LOCK_UNLOCKED { 0 }
diff --git a/arch/s390/include/asm/switch_to.h b/arch/s390/include/asm/switch_to.h
index 12d45f0..f6c2b58 100644
--- a/arch/s390/include/asm/switch_to.h
+++ b/arch/s390/include/asm/switch_to.h
@@ -10,6 +10,7 @@
#include <linux/thread_info.h>
#include <asm/fpu/api.h>
#include <asm/ptrace.h>
+#include <asm/guarded_storage.h>
extern struct task_struct *__switch_to(void *, void *);
extern void update_cr_regs(struct task_struct *task);
@@ -33,12 +34,14 @@
save_fpu_regs(); \
save_access_regs(&prev->thread.acrs[0]); \
save_ri_cb(prev->thread.ri_cb); \
+ save_gs_cb(prev->thread.gs_cb); \
} \
if (next->mm) { \
update_cr_regs(next); \
set_cpu_flag(CIF_FPU); \
restore_access_regs(&next->thread.acrs[0]); \
restore_ri_cb(next->thread.ri_cb, prev->thread.ri_cb); \
+ restore_gs_cb(next->thread.gs_cb); \
} \
prev = __switch_to(prev,next); \
} while (0)
diff --git a/arch/s390/include/asm/sysinfo.h b/arch/s390/include/asm/sysinfo.h
index 229326c..73bff45 100644
--- a/arch/s390/include/asm/sysinfo.h
+++ b/arch/s390/include/asm/sysinfo.h
@@ -142,7 +142,15 @@
extern int topology_max_mnest;
-#define TOPOLOGY_CORE_BITS 64
+/*
+ * Returns the maximum nesting level supported by the cpu topology code.
+ * The current maximum level is 4 which is the drawer level.
+ */
+static inline int topology_mnest_limit(void)
+{
+ return min(topology_max_mnest, 4);
+}
+
#define TOPOLOGY_NR_MAG 6
struct topology_core {
@@ -152,7 +160,7 @@
unsigned char pp:2;
unsigned char reserved1;
unsigned short origin;
- unsigned long mask[TOPOLOGY_CORE_BITS / BITS_PER_LONG];
+ unsigned long mask;
};
struct topology_container {
diff --git a/arch/s390/include/asm/thread_info.h b/arch/s390/include/asm/thread_info.h
index a5b54a4..0b3ee08 100644
--- a/arch/s390/include/asm/thread_info.h
+++ b/arch/s390/include/asm/thread_info.h
@@ -51,14 +51,14 @@
/*
* thread information flags bit numbers
*/
+/* _TIF_WORK bits */
#define TIF_NOTIFY_RESUME 0 /* callback before returning to user */
#define TIF_SIGPENDING 1 /* signal pending */
#define TIF_NEED_RESCHED 2 /* rescheduling necessary */
-#define TIF_SYSCALL_TRACE 3 /* syscall trace active */
-#define TIF_SYSCALL_AUDIT 4 /* syscall auditing active */
-#define TIF_SECCOMP 5 /* secure computing */
-#define TIF_SYSCALL_TRACEPOINT 6 /* syscall tracepoint instrumentation */
-#define TIF_UPROBE 7 /* breakpointed or single-stepping */
+#define TIF_UPROBE 3 /* breakpointed or single-stepping */
+#define TIF_GUARDED_STORAGE 4 /* load guarded storage control block */
+#define TIF_PATCH_PENDING 5 /* pending live patching update */
+
#define TIF_31BIT 16 /* 32bit process */
#define TIF_MEMDIE 17 /* is terminating due to OOM killer */
#define TIF_RESTORE_SIGMASK 18 /* restore signal mask in do_signal() */
@@ -66,15 +66,25 @@
#define TIF_BLOCK_STEP 20 /* This task is block stepped */
#define TIF_UPROBE_SINGLESTEP 21 /* This task is uprobe single stepped */
+/* _TIF_TRACE bits */
+#define TIF_SYSCALL_TRACE 24 /* syscall trace active */
+#define TIF_SYSCALL_AUDIT 25 /* syscall auditing active */
+#define TIF_SECCOMP 26 /* secure computing */
+#define TIF_SYSCALL_TRACEPOINT 27 /* syscall tracepoint instrumentation */
+
#define _TIF_NOTIFY_RESUME _BITUL(TIF_NOTIFY_RESUME)
#define _TIF_SIGPENDING _BITUL(TIF_SIGPENDING)
#define _TIF_NEED_RESCHED _BITUL(TIF_NEED_RESCHED)
+#define _TIF_UPROBE _BITUL(TIF_UPROBE)
+#define _TIF_GUARDED_STORAGE _BITUL(TIF_GUARDED_STORAGE)
+#define _TIF_PATCH_PENDING _BITUL(TIF_PATCH_PENDING)
+
+#define _TIF_31BIT _BITUL(TIF_31BIT)
+#define _TIF_SINGLE_STEP _BITUL(TIF_SINGLE_STEP)
+
#define _TIF_SYSCALL_TRACE _BITUL(TIF_SYSCALL_TRACE)
#define _TIF_SYSCALL_AUDIT _BITUL(TIF_SYSCALL_AUDIT)
#define _TIF_SECCOMP _BITUL(TIF_SECCOMP)
#define _TIF_SYSCALL_TRACEPOINT _BITUL(TIF_SYSCALL_TRACEPOINT)
-#define _TIF_UPROBE _BITUL(TIF_UPROBE)
-#define _TIF_31BIT _BITUL(TIF_31BIT)
-#define _TIF_SINGLE_STEP _BITUL(TIF_SINGLE_STEP)
#endif /* _ASM_THREAD_INFO_H */
diff --git a/arch/s390/include/asm/uaccess.h b/arch/s390/include/asm/uaccess.h
index 3ea1554..78f3f09 100644
--- a/arch/s390/include/asm/uaccess.h
+++ b/arch/s390/include/asm/uaccess.h
@@ -12,13 +12,9 @@
/*
* User space memory access functions
*/
-#include <linux/sched.h>
-#include <linux/errno.h>
#include <asm/processor.h>
#include <asm/ctl_reg.h>
-
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
+#include <asm/extable.h>
/*
@@ -42,7 +38,7 @@
static inline void set_fs(mm_segment_t fs)
{
current->thread.mm_segment = fs;
- if (segment_eq(fs, KERNEL_DS)) {
+ if (uaccess_kernel()) {
set_cpu_flag(CIF_ASCE_SECONDARY);
__ctl_load(S390_lowcore.kernel_asce, 7, 7);
} else {
@@ -64,72 +60,14 @@
#define access_ok(type, addr, size) __access_ok(addr, size)
-/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
+unsigned long __must_check
+raw_copy_from_user(void *to, const void __user *from, unsigned long n);
-struct exception_table_entry
-{
- int insn, fixup;
-};
+unsigned long __must_check
+raw_copy_to_user(void __user *to, const void *from, unsigned long n);
-static inline unsigned long extable_fixup(const struct exception_table_entry *x)
-{
- return (unsigned long)&x->fixup + x->fixup;
-}
-
-#define ARCH_HAS_RELATIVE_EXTABLE
-
-/**
- * __copy_from_user: - Copy a block of data from user space, with less checking.
- * @to: Destination address, in kernel space.
- * @from: Source address, in user space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from user space to kernel space. Caller must check
- * the specified block with access_ok() before calling this function.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- *
- * If some data could not be copied, this function will pad the copied
- * data to the requested size using zero bytes.
- */
-unsigned long __must_check __copy_from_user(void *to, const void __user *from,
- unsigned long n);
-
-/**
- * __copy_to_user: - Copy a block of data into user space, with less checking.
- * @to: Destination address, in user space.
- * @from: Source address, in kernel space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from kernel space to user space. Caller must check
- * the specified block with access_ok() before calling this function.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- */
-unsigned long __must_check __copy_to_user(void __user *to, const void *from,
- unsigned long n);
-
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
#ifdef CONFIG_HAVE_MARCH_Z10_FEATURES
@@ -218,13 +156,13 @@
static inline int __put_user_fn(void *x, void __user *ptr, unsigned long size)
{
- size = __copy_to_user(ptr, x, size);
+ size = raw_copy_to_user(ptr, x, size);
return size ? -EFAULT : 0;
}
static inline int __get_user_fn(void *x, const void __user *ptr, unsigned long size)
{
- size = __copy_from_user(x, ptr, size);
+ size = raw_copy_from_user(x, ptr, size);
return size ? -EFAULT : 0;
}
@@ -314,77 +252,8 @@
#define __put_user_unaligned __put_user
#define __get_user_unaligned __get_user
-extern void __compiletime_error("usercopy buffer size is too small")
-__bad_copy_user(void);
-
-static inline void copy_user_overflow(int size, unsigned long count)
-{
- WARN(1, "Buffer overflow detected (%d < %lu)!\n", size, count);
-}
-
-/**
- * copy_to_user: - Copy a block of data into user space.
- * @to: Destination address, in user space.
- * @from: Source address, in kernel space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from kernel space to user space.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- */
-static inline unsigned long __must_check
-copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- might_fault();
- return __copy_to_user(to, from, n);
-}
-
-/**
- * copy_from_user: - Copy a block of data from user space.
- * @to: Destination address, in kernel space.
- * @from: Source address, in user space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from user space to kernel space.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- *
- * If some data could not be copied, this function will pad the copied
- * data to the requested size using zero bytes.
- */
-static inline unsigned long __must_check
-copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- unsigned int sz = __compiletime_object_size(to);
-
- might_fault();
- if (unlikely(sz != -1 && sz < n)) {
- if (!__builtin_constant_p(n))
- copy_user_overflow(sz, n);
- else
- __bad_copy_user();
- return n;
- }
- return __copy_from_user(to, from, n);
-}
-
unsigned long __must_check
-__copy_in_user(void __user *to, const void __user *from, unsigned long n);
-
-static inline unsigned long __must_check
-copy_in_user(void __user *to, const void __user *from, unsigned long n)
-{
- might_fault();
- return __copy_in_user(to, from, n);
-}
+raw_copy_in_user(void __user *to, const void __user *from, unsigned long n);
/*
* Copy a null terminated string from userspace.
diff --git a/arch/s390/include/uapi/asm/Kbuild b/arch/s390/include/uapi/asm/Kbuild
index 6848ba5..addb09c 100644
--- a/arch/s390/include/uapi/asm/Kbuild
+++ b/arch/s390/include/uapi/asm/Kbuild
@@ -1,6 +1,16 @@
# UAPI Header export list
include include/uapi/asm-generic/Kbuild.asm
+generic-y += errno.h
+generic-y += fcntl.h
+generic-y += ioctl.h
+generic-y += mman.h
+generic-y += param.h
+generic-y += poll.h
+generic-y += resource.h
+generic-y += sockios.h
+generic-y += termbits.h
+
header-y += auxvec.h
header-y += bitsperlong.h
header-y += byteorder.h
@@ -11,25 +21,20 @@
header-y += dasd.h
header-y += debug.h
header-y += errno.h
-header-y += fcntl.h
+header-y += guarded_storage.h
header-y += hypfs.h
-header-y += ioctl.h
header-y += ioctls.h
header-y += ipcbuf.h
header-y += kvm.h
header-y += kvm_para.h
header-y += kvm_perf.h
header-y += kvm_virtio.h
-header-y += mman.h
header-y += monwriter.h
header-y += msgbuf.h
-header-y += param.h
header-y += pkey.h
-header-y += poll.h
header-y += posix_types.h
header-y += ptrace.h
header-y += qeth.h
-header-y += resource.h
header-y += schid.h
header-y += sclp_ctl.h
header-y += sembuf.h
@@ -40,12 +45,10 @@
header-y += siginfo.h
header-y += signal.h
header-y += socket.h
-header-y += sockios.h
header-y += stat.h
header-y += statfs.h
header-y += swab.h
header-y += tape390.h
-header-y += termbits.h
header-y += termios.h
header-y += types.h
header-y += ucontext.h
diff --git a/arch/s390/include/uapi/asm/errno.h b/arch/s390/include/uapi/asm/errno.h
deleted file mode 100644
index 395e97d..0000000
--- a/arch/s390/include/uapi/asm/errno.h
+++ /dev/null
@@ -1,11 +0,0 @@
-/*
- * S390 version
- *
- */
-
-#ifndef _S390_ERRNO_H
-#define _S390_ERRNO_H
-
-#include <asm-generic/errno.h>
-
-#endif
diff --git a/arch/s390/include/uapi/asm/fcntl.h b/arch/s390/include/uapi/asm/fcntl.h
deleted file mode 100644
index 46ab12d..0000000
--- a/arch/s390/include/uapi/asm/fcntl.h
+++ /dev/null
@@ -1 +0,0 @@
-#include <asm-generic/fcntl.h>
diff --git a/arch/s390/include/uapi/asm/guarded_storage.h b/arch/s390/include/uapi/asm/guarded_storage.h
new file mode 100644
index 0000000..852850e
--- /dev/null
+++ b/arch/s390/include/uapi/asm/guarded_storage.h
@@ -0,0 +1,77 @@
+#ifndef _GUARDED_STORAGE_H
+#define _GUARDED_STORAGE_H
+
+#include <linux/types.h>
+
+struct gs_cb {
+ __u64 reserved;
+ __u64 gsd;
+ __u64 gssm;
+ __u64 gs_epl_a;
+};
+
+struct gs_epl {
+ __u8 pad1;
+ union {
+ __u8 gs_eam;
+ struct {
+ __u8 : 6;
+ __u8 e : 1;
+ __u8 b : 1;
+ };
+ };
+ union {
+ __u8 gs_eci;
+ struct {
+ __u8 tx : 1;
+ __u8 cx : 1;
+ __u8 : 5;
+ __u8 in : 1;
+ };
+ };
+ union {
+ __u8 gs_eai;
+ struct {
+ __u8 : 1;
+ __u8 t : 1;
+ __u8 as : 2;
+ __u8 ar : 4;
+ };
+ };
+ __u32 pad2;
+ __u64 gs_eha;
+ __u64 gs_eia;
+ __u64 gs_eoa;
+ __u64 gs_eir;
+ __u64 gs_era;
+};
+
+#define GS_ENABLE 0
+#define GS_DISABLE 1
+#define GS_SET_BC_CB 2
+#define GS_CLEAR_BC_CB 3
+#define GS_BROADCAST 4
+
+static inline void load_gs_cb(struct gs_cb *gs_cb)
+{
+ asm volatile(".insn rxy,0xe3000000004d,0,%0" : : "Q" (*gs_cb));
+}
+
+static inline void store_gs_cb(struct gs_cb *gs_cb)
+{
+ asm volatile(".insn rxy,0xe30000000049,0,%0" : : "Q" (*gs_cb));
+}
+
+static inline void save_gs_cb(struct gs_cb *gs_cb)
+{
+ if (gs_cb)
+ store_gs_cb(gs_cb);
+}
+
+static inline void restore_gs_cb(struct gs_cb *gs_cb)
+{
+ if (gs_cb)
+ load_gs_cb(gs_cb);
+}
+
+#endif /* _GUARDED_STORAGE_H */
diff --git a/arch/s390/include/uapi/asm/ioctl.h b/arch/s390/include/uapi/asm/ioctl.h
deleted file mode 100644
index b279fe0..0000000
--- a/arch/s390/include/uapi/asm/ioctl.h
+++ /dev/null
@@ -1 +0,0 @@
-#include <asm-generic/ioctl.h>
diff --git a/arch/s390/include/uapi/asm/mman.h b/arch/s390/include/uapi/asm/mman.h
deleted file mode 100644
index de23da1..0000000
--- a/arch/s390/include/uapi/asm/mman.h
+++ /dev/null
@@ -1,6 +0,0 @@
-/*
- * S390 version
- *
- * Derived from "include/asm-i386/mman.h"
- */
-#include <asm-generic/mman.h>
diff --git a/arch/s390/include/uapi/asm/param.h b/arch/s390/include/uapi/asm/param.h
deleted file mode 100644
index c616821..0000000
--- a/arch/s390/include/uapi/asm/param.h
+++ /dev/null
@@ -1,6 +0,0 @@
-#ifndef _ASMS390_PARAM_H
-#define _ASMS390_PARAM_H
-
-#include <asm-generic/param.h>
-
-#endif /* _ASMS390_PARAM_H */
diff --git a/arch/s390/include/uapi/asm/pkey.h b/arch/s390/include/uapi/asm/pkey.h
index ed7f19c..e6c04fa 100644
--- a/arch/s390/include/uapi/asm/pkey.h
+++ b/arch/s390/include/uapi/asm/pkey.h
@@ -109,4 +109,23 @@
};
#define PKEY_SKEY2PKEY _IOWR(PKEY_IOCTL_MAGIC, 0x06, struct pkey_skey2pkey)
+/*
+ * Verify the given secure key for being able to be useable with
+ * the pkey module. Check for correct key type and check for having at
+ * least one crypto card being able to handle this key (master key
+ * or old master key verification pattern matches).
+ * Return some info about the key: keysize in bits, keytype (currently
+ * only AES), flag if key is wrapped with an old MKVP.
+ */
+struct pkey_verifykey {
+ struct pkey_seckey seckey; /* in: the secure key blob */
+ __u16 cardnr; /* out: card number */
+ __u16 domain; /* out: domain number */
+ __u16 keysize; /* out: key size in bits */
+ __u32 attributes; /* out: attribute bits */
+};
+#define PKEY_VERIFYKEY _IOWR(PKEY_IOCTL_MAGIC, 0x07, struct pkey_verifykey)
+#define PKEY_VERIFY_ATTR_AES 0x00000001 /* key is an AES key */
+#define PKEY_VERIFY_ATTR_OLD_MKVP 0x00000100 /* key has old MKVP value */
+
#endif /* _UAPI_PKEY_H */
diff --git a/arch/s390/include/uapi/asm/poll.h b/arch/s390/include/uapi/asm/poll.h
deleted file mode 100644
index c98509d..0000000
--- a/arch/s390/include/uapi/asm/poll.h
+++ /dev/null
@@ -1 +0,0 @@
-#include <asm-generic/poll.h>
diff --git a/arch/s390/include/uapi/asm/resource.h b/arch/s390/include/uapi/asm/resource.h
deleted file mode 100644
index ec23d1c..0000000
--- a/arch/s390/include/uapi/asm/resource.h
+++ /dev/null
@@ -1,13 +0,0 @@
-/*
- * S390 version
- *
- * Derived from "include/asm-i386/resources.h"
- */
-
-#ifndef _S390_RESOURCE_H
-#define _S390_RESOURCE_H
-
-#include <asm-generic/resource.h>
-
-#endif
-
diff --git a/arch/s390/include/uapi/asm/socket.h b/arch/s390/include/uapi/asm/socket.h
index b24a64c..e8e5ecf 100644
--- a/arch/s390/include/uapi/asm/socket.h
+++ b/arch/s390/include/uapi/asm/socket.h
@@ -98,4 +98,10 @@
#define SCM_TIMESTAMPING_OPT_STATS 54
+#define SO_MEMINFO 55
+
+#define SO_INCOMING_NAPI_ID 56
+
+#define SO_COOKIE 57
+
#endif /* _ASM_SOCKET_H */
diff --git a/arch/s390/include/uapi/asm/sockios.h b/arch/s390/include/uapi/asm/sockios.h
deleted file mode 100644
index 6f60eee..0000000
--- a/arch/s390/include/uapi/asm/sockios.h
+++ /dev/null
@@ -1,6 +0,0 @@
-#ifndef _ASM_S390_SOCKIOS_H
-#define _ASM_S390_SOCKIOS_H
-
-#include <asm-generic/sockios.h>
-
-#endif
diff --git a/arch/s390/include/uapi/asm/termbits.h b/arch/s390/include/uapi/asm/termbits.h
deleted file mode 100644
index 71bf6ac..0000000
--- a/arch/s390/include/uapi/asm/termbits.h
+++ /dev/null
@@ -1,6 +0,0 @@
-#ifndef _ASM_S390_TERMBITS_H
-#define _ASM_S390_TERMBITS_H
-
-#include <asm-generic/termbits.h>
-
-#endif
diff --git a/arch/s390/include/uapi/asm/unistd.h b/arch/s390/include/uapi/asm/unistd.h
index 152de9b..ea42290 100644
--- a/arch/s390/include/uapi/asm/unistd.h
+++ b/arch/s390/include/uapi/asm/unistd.h
@@ -313,7 +313,7 @@
#define __NR_copy_file_range 375
#define __NR_preadv2 376
#define __NR_pwritev2 377
-/* Number 378 is reserved for guarded storage */
+#define __NR_s390_guarded_storage 378
#define __NR_statx 379
#define NR_syscalls 380
diff --git a/arch/s390/kernel/Makefile b/arch/s390/kernel/Makefile
index 060ce54..adb3fe2 100644
--- a/arch/s390/kernel/Makefile
+++ b/arch/s390/kernel/Makefile
@@ -51,14 +51,12 @@
#
CFLAGS_ptrace.o += -DUTS_MACHINE='"$(UTS_MACHINE)"'
-CFLAGS_sysinfo.o += -w
-
obj-y := traps.o time.o process.o base.o early.o setup.o idle.o vtime.o
obj-y += processor.o sys_s390.o ptrace.o signal.o cpcmd.o ebcdic.o nmi.o
obj-y += debug.o irq.o ipl.o dis.o diag.o vdso.o als.o
obj-y += sysinfo.o jump_label.o lgr.o os_info.o machine_kexec.o pgm_check.o
-obj-y += runtime_instr.o cache.o fpu.o dumpstack.o
-obj-y += entry.o reipl.o relocate_kernel.o
+obj-y += runtime_instr.o cache.o fpu.o dumpstack.o guarded_storage.o
+obj-y += entry.o reipl.o relocate_kernel.o kdebugfs.o
extra-y += head.o head64.o vmlinux.lds
diff --git a/arch/s390/kernel/asm-offsets.c b/arch/s390/kernel/asm-offsets.c
index c4b3570..6bb2963 100644
--- a/arch/s390/kernel/asm-offsets.c
+++ b/arch/s390/kernel/asm-offsets.c
@@ -175,7 +175,7 @@
/* software defined ABI-relevant lowcore locations 0xe00 - 0xe20 */
OFFSET(__LC_DUMP_REIPL, lowcore, ipib);
/* hardware defined lowcore locations 0x1000 - 0x18ff */
- OFFSET(__LC_VX_SAVE_AREA_ADDR, lowcore, vector_save_area_addr);
+ OFFSET(__LC_MCESAD, lowcore, mcesad);
OFFSET(__LC_EXT_PARAMS2, lowcore, ext_params2);
OFFSET(__LC_FPREGS_SAVE_AREA, lowcore, floating_pt_save_area);
OFFSET(__LC_GPREGS_SAVE_AREA, lowcore, gpregs_save_area);
diff --git a/arch/s390/kernel/compat_wrapper.c b/arch/s390/kernel/compat_wrapper.c
index e89cc2e..986642a 100644
--- a/arch/s390/kernel/compat_wrapper.c
+++ b/arch/s390/kernel/compat_wrapper.c
@@ -178,4 +178,5 @@
COMPAT_SYSCALL_WRAP6(sendto, int, fd, void __user *, buff, size_t, len, unsigned int, flags, struct sockaddr __user *, addr, int, addr_len);
COMPAT_SYSCALL_WRAP3(mlock2, unsigned long, start, size_t, len, int, flags);
COMPAT_SYSCALL_WRAP6(copy_file_range, int, fd_in, loff_t __user *, off_in, int, fd_out, loff_t __user *, off_out, size_t, len, unsigned int, flags);
+COMPAT_SYSCALL_WRAP2(s390_guarded_storage, int, command, struct gs_cb *, gs_cb);
COMPAT_SYSCALL_WRAP5(statx, int, dfd, const char __user *, path, unsigned, flags, unsigned, mask, struct statx __user *, buffer);
diff --git a/arch/s390/kernel/crash_dump.c b/arch/s390/kernel/crash_dump.c
index dd1d5c6..d628afc 100644
--- a/arch/s390/kernel/crash_dump.c
+++ b/arch/s390/kernel/crash_dump.c
@@ -429,6 +429,20 @@
}
/*
+ * Initialize final note (needed for /proc/vmcore code)
+ */
+static void *nt_final(void *ptr)
+{
+ Elf64_Nhdr *note;
+
+ note = (Elf64_Nhdr *) ptr;
+ note->n_namesz = 0;
+ note->n_descsz = 0;
+ note->n_type = 0;
+ return PTR_ADD(ptr, sizeof(Elf64_Nhdr));
+}
+
+/*
* Initialize ELF header (new kernel)
*/
static void *ehdr_init(Elf64_Ehdr *ehdr, int mem_chunk_cnt)
@@ -515,6 +529,7 @@
if (sa->prefix != 0)
ptr = fill_cpu_elf_notes(ptr, cpu++, sa);
ptr = nt_vmcoreinfo(ptr);
+ ptr = nt_final(ptr);
memset(phdr, 0, sizeof(*phdr));
phdr->p_type = PT_NOTE;
phdr->p_offset = notes_offset;
diff --git a/arch/s390/kernel/early.c b/arch/s390/kernel/early.c
index 4e65c79..5d20182 100644
--- a/arch/s390/kernel/early.c
+++ b/arch/s390/kernel/early.c
@@ -231,9 +231,29 @@
S390_lowcore.machine_flags |= MACHINE_FLAG_VM;
}
+/* Remove leading, trailing and double whitespace. */
+static inline void strim_all(char *str)
+{
+ char *s;
+
+ s = strim(str);
+ if (s != str)
+ memmove(str, s, strlen(s));
+ while (*str) {
+ if (!isspace(*str++))
+ continue;
+ if (isspace(*str)) {
+ s = skip_spaces(str);
+ memmove(str, s, strlen(s) + 1);
+ }
+ }
+}
+
static noinline __init void setup_arch_string(void)
{
struct sysinfo_1_1_1 *mach = (struct sysinfo_1_1_1 *)&sysinfo_page;
+ struct sysinfo_3_2_2 *vm = (struct sysinfo_3_2_2 *)&sysinfo_page;
+ char mstr[80], hvstr[17];
if (stsi(mach, 1, 1, 1))
return;
@@ -241,14 +261,21 @@
EBCASC(mach->type, sizeof(mach->type));
EBCASC(mach->model, sizeof(mach->model));
EBCASC(mach->model_capacity, sizeof(mach->model_capacity));
- dump_stack_set_arch_desc("%-16.16s %-4.4s %-16.16s %-16.16s (%s)",
- mach->manufacturer,
- mach->type,
- mach->model,
- mach->model_capacity,
- MACHINE_IS_LPAR ? "LPAR" :
- MACHINE_IS_VM ? "z/VM" :
- MACHINE_IS_KVM ? "KVM" : "unknown");
+ sprintf(mstr, "%-16.16s %-4.4s %-16.16s %-16.16s",
+ mach->manufacturer, mach->type,
+ mach->model, mach->model_capacity);
+ strim_all(mstr);
+ if (stsi(vm, 3, 2, 2) == 0 && vm->count) {
+ EBCASC(vm->vm[0].cpi, sizeof(vm->vm[0].cpi));
+ sprintf(hvstr, "%-16.16s", vm->vm[0].cpi);
+ strim_all(hvstr);
+ } else {
+ sprintf(hvstr, "%s",
+ MACHINE_IS_LPAR ? "LPAR" :
+ MACHINE_IS_VM ? "z/VM" :
+ MACHINE_IS_KVM ? "KVM" : "unknown");
+ }
+ dump_stack_set_arch_desc("%s (%s)", mstr, hvstr);
}
static __init void setup_topology(void)
@@ -358,6 +385,8 @@
S390_lowcore.machine_flags |= MACHINE_FLAG_NX;
__ctl_set_bit(0, 20);
}
+ if (test_facility(133))
+ S390_lowcore.machine_flags |= MACHINE_FLAG_GS;
}
static inline void save_vector_registers(void)
@@ -375,7 +404,7 @@
rc = kstrtobool(str, &enabled);
if (!rc && !enabled)
- S390_lowcore.machine_flags &= ~MACHINE_HAS_TOPOLOGY;
+ S390_lowcore.machine_flags &= ~MACHINE_FLAG_TOPOLOGY;
return rc;
}
early_param("topology", topology_setup);
@@ -405,23 +434,16 @@
static int __init cad_setup(char *str)
{
- int val;
+ bool enabled;
+ int rc;
- get_option(&str, &val);
- if (val && test_facility(128))
- S390_lowcore.machine_flags |= MACHINE_FLAG_CAD;
- return 0;
-}
-early_param("cad", cad_setup);
-
-static int __init cad_init(void)
-{
- if (MACHINE_HAS_CAD)
+ rc = kstrtobool(str, &enabled);
+ if (!rc && enabled && test_facility(128))
/* Enable problem state CAD. */
__ctl_set_bit(2, 3);
- return 0;
+ return rc;
}
-early_initcall(cad_init);
+early_param("cad", cad_setup);
static __init void memmove_early(void *dst, const void *src, size_t n)
{
diff --git a/arch/s390/kernel/entry.S b/arch/s390/kernel/entry.S
index 6a7d737..a5f5d3b 100644
--- a/arch/s390/kernel/entry.S
+++ b/arch/s390/kernel/entry.S
@@ -47,7 +47,7 @@
STACK_INIT = STACK_SIZE - STACK_FRAME_OVERHEAD - __PT_SIZE
_TIF_WORK = (_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_NEED_RESCHED | \
- _TIF_UPROBE)
+ _TIF_UPROBE | _TIF_GUARDED_STORAGE | _TIF_PATCH_PENDING)
_TIF_TRACE = (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | _TIF_SECCOMP | \
_TIF_SYSCALL_TRACEPOINT)
_CIF_WORK = (_CIF_MCCK_PENDING | _CIF_ASCE_PRIMARY | \
@@ -189,8 +189,6 @@
stg %r3,__LC_CURRENT # store task struct of next
stg %r15,__LC_KERNEL_STACK # store end of kernel stack
lg %r15,__THREAD_ksp(%r1) # load kernel stack of next
- /* c4 is used in guest detection: arch/s390/kernel/perf_cpum_sf.c */
- lctl %c4,%c4,__TASK_pid(%r3) # load pid to control reg. 4
mvc __LC_CURRENT_PID(4,%r0),__TASK_pid(%r3) # store pid of next
lmg %r6,%r15,__SF_GPRS(%r15) # load gprs of next task
TSTMSK __LC_MACHINE_FLAGS,MACHINE_FLAG_LPP
@@ -332,8 +330,15 @@
TSTMSK __TI_flags(%r12),_TIF_UPROBE
jo .Lsysc_uprobe_notify
#endif
+ TSTMSK __TI_flags(%r12),_TIF_GUARDED_STORAGE
+ jo .Lsysc_guarded_storage
TSTMSK __PT_FLAGS(%r11),_PIF_PER_TRAP
jo .Lsysc_singlestep
+#ifdef CONFIG_LIVEPATCH
+ TSTMSK __TI_flags(%r12),_TIF_PATCH_PENDING
+ jo .Lsysc_patch_pending # handle live patching just before
+ # signals and possible syscall restart
+#endif
TSTMSK __TI_flags(%r12),_TIF_SIGPENDING
jo .Lsysc_sigpending
TSTMSK __TI_flags(%r12),_TIF_NOTIFY_RESUME
@@ -409,6 +414,23 @@
#endif
#
+# _TIF_GUARDED_STORAGE is set, call guarded_storage_load
+#
+.Lsysc_guarded_storage:
+ lgr %r2,%r11 # pass pointer to pt_regs
+ larl %r14,.Lsysc_return
+ jg gs_load_bc_cb
+#
+# _TIF_PATCH_PENDING is set, call klp_update_patch_state
+#
+#ifdef CONFIG_LIVEPATCH
+.Lsysc_patch_pending:
+ lg %r2,__LC_CURRENT # pass pointer to task struct
+ larl %r14,.Lsysc_return
+ jg klp_update_patch_state
+#endif
+
+#
# _PIF_PER_TRAP is set, call do_per_trap
#
.Lsysc_singlestep:
@@ -659,10 +681,16 @@
jo .Lio_mcck_pending
TSTMSK __TI_flags(%r12),_TIF_NEED_RESCHED
jo .Lio_reschedule
+#ifdef CONFIG_LIVEPATCH
+ TSTMSK __TI_flags(%r12),_TIF_PATCH_PENDING
+ jo .Lio_patch_pending
+#endif
TSTMSK __TI_flags(%r12),_TIF_SIGPENDING
jo .Lio_sigpending
TSTMSK __TI_flags(%r12),_TIF_NOTIFY_RESUME
jo .Lio_notify_resume
+ TSTMSK __TI_flags(%r12),_TIF_GUARDED_STORAGE
+ jo .Lio_guarded_storage
TSTMSK __LC_CPU_FLAGS,_CIF_FPU
jo .Lio_vxrs
TSTMSK __LC_CPU_FLAGS,(_CIF_ASCE_PRIMARY|_CIF_ASCE_SECONDARY)
@@ -697,6 +725,18 @@
jg load_fpu_regs
#
+# _TIF_GUARDED_STORAGE is set, call guarded_storage_load
+#
+.Lio_guarded_storage:
+ # TRACE_IRQS_ON already done at .Lio_return
+ ssm __LC_SVC_NEW_PSW # reenable interrupts
+ lgr %r2,%r11 # pass pointer to pt_regs
+ brasl %r14,gs_load_bc_cb
+ ssm __LC_PGM_NEW_PSW # disable I/O and ext. interrupts
+ TRACE_IRQS_OFF
+ j .Lio_return
+
+#
# _TIF_NEED_RESCHED is set, call schedule
#
.Lio_reschedule:
@@ -708,6 +748,16 @@
j .Lio_return
#
+# _TIF_PATCH_PENDING is set, call klp_update_patch_state
+#
+#ifdef CONFIG_LIVEPATCH
+.Lio_patch_pending:
+ lg %r2,__LC_CURRENT # pass pointer to task struct
+ larl %r14,.Lio_return
+ jg klp_update_patch_state
+#endif
+
+#
# _TIF_SIGPENDING or is set, call do_signal
#
.Lio_sigpending:
diff --git a/arch/s390/kernel/entry.h b/arch/s390/kernel/entry.h
index 33f9018..dbf5f7e 100644
--- a/arch/s390/kernel/entry.h
+++ b/arch/s390/kernel/entry.h
@@ -74,12 +74,14 @@
long sys_s390_personality(unsigned int personality);
long sys_s390_runtime_instr(int command, int signum);
+long sys_s390_guarded_storage(int command, struct gs_cb __user *);
long sys_s390_pci_mmio_write(unsigned long, const void __user *, size_t);
long sys_s390_pci_mmio_read(unsigned long, void __user *, size_t);
DECLARE_PER_CPU(u64, mt_cycles[8]);
void verify_facilities(void);
+void gs_load_bc_cb(struct pt_regs *regs);
void set_fs_fixup(void);
#endif /* _ENTRY_H */
diff --git a/arch/s390/kernel/guarded_storage.c b/arch/s390/kernel/guarded_storage.c
new file mode 100644
index 0000000..6f06474
--- /dev/null
+++ b/arch/s390/kernel/guarded_storage.c
@@ -0,0 +1,128 @@
+/*
+ * Copyright IBM Corp. 2016
+ * Author(s): Martin Schwidefsky <schwidefsky@de.ibm.com>
+ */
+
+#include <linux/kernel.h>
+#include <linux/syscalls.h>
+#include <linux/signal.h>
+#include <linux/mm.h>
+#include <linux/slab.h>
+#include <asm/guarded_storage.h>
+#include "entry.h"
+
+void exit_thread_gs(void)
+{
+ kfree(current->thread.gs_cb);
+ kfree(current->thread.gs_bc_cb);
+ current->thread.gs_cb = current->thread.gs_bc_cb = NULL;
+}
+
+static int gs_enable(void)
+{
+ struct gs_cb *gs_cb;
+
+ if (!current->thread.gs_cb) {
+ gs_cb = kzalloc(sizeof(*gs_cb), GFP_KERNEL);
+ if (!gs_cb)
+ return -ENOMEM;
+ gs_cb->gsd = 25;
+ preempt_disable();
+ __ctl_set_bit(2, 4);
+ load_gs_cb(gs_cb);
+ current->thread.gs_cb = gs_cb;
+ preempt_enable();
+ }
+ return 0;
+}
+
+static int gs_disable(void)
+{
+ if (current->thread.gs_cb) {
+ preempt_disable();
+ kfree(current->thread.gs_cb);
+ current->thread.gs_cb = NULL;
+ __ctl_clear_bit(2, 4);
+ preempt_enable();
+ }
+ return 0;
+}
+
+static int gs_set_bc_cb(struct gs_cb __user *u_gs_cb)
+{
+ struct gs_cb *gs_cb;
+
+ gs_cb = current->thread.gs_bc_cb;
+ if (!gs_cb) {
+ gs_cb = kzalloc(sizeof(*gs_cb), GFP_KERNEL);
+ if (!gs_cb)
+ return -ENOMEM;
+ current->thread.gs_bc_cb = gs_cb;
+ }
+ if (copy_from_user(gs_cb, u_gs_cb, sizeof(*gs_cb)))
+ return -EFAULT;
+ return 0;
+}
+
+static int gs_clear_bc_cb(void)
+{
+ struct gs_cb *gs_cb;
+
+ gs_cb = current->thread.gs_bc_cb;
+ current->thread.gs_bc_cb = NULL;
+ kfree(gs_cb);
+ return 0;
+}
+
+void gs_load_bc_cb(struct pt_regs *regs)
+{
+ struct gs_cb *gs_cb;
+
+ preempt_disable();
+ clear_thread_flag(TIF_GUARDED_STORAGE);
+ gs_cb = current->thread.gs_bc_cb;
+ if (gs_cb) {
+ kfree(current->thread.gs_cb);
+ current->thread.gs_bc_cb = NULL;
+ __ctl_set_bit(2, 4);
+ load_gs_cb(gs_cb);
+ current->thread.gs_cb = gs_cb;
+ }
+ preempt_enable();
+}
+
+static int gs_broadcast(void)
+{
+ struct task_struct *sibling;
+
+ read_lock(&tasklist_lock);
+ for_each_thread(current, sibling) {
+ if (!sibling->thread.gs_bc_cb)
+ continue;
+ if (test_and_set_tsk_thread_flag(sibling, TIF_GUARDED_STORAGE))
+ kick_process(sibling);
+ }
+ read_unlock(&tasklist_lock);
+ return 0;
+}
+
+SYSCALL_DEFINE2(s390_guarded_storage, int, command,
+ struct gs_cb __user *, gs_cb)
+{
+ if (!MACHINE_HAS_GS)
+ return -EOPNOTSUPP;
+ switch (command) {
+ case GS_ENABLE:
+ return gs_enable();
+ case GS_DISABLE:
+ return gs_disable();
+ case GS_SET_BC_CB:
+ return gs_set_bc_cb(gs_cb);
+ case GS_CLEAR_BC_CB:
+ return gs_clear_bc_cb();
+ case GS_BROADCAST:
+ return gs_broadcast();
+ default:
+ return -EINVAL;
+ }
+}
diff --git a/arch/s390/kernel/head.S b/arch/s390/kernel/head.S
index 0b5ebf8..eff5b31 100644
--- a/arch/s390/kernel/head.S
+++ b/arch/s390/kernel/head.S
@@ -25,7 +25,6 @@
#include <linux/linkage.h>
#include <asm/asm-offsets.h>
#include <asm/thread_info.h>
-#include <asm/facility.h>
#include <asm/page.h>
#include <asm/ptrace.h>
diff --git a/arch/s390/kernel/head64.S b/arch/s390/kernel/head64.S
index 482d352..31c91f2 100644
--- a/arch/s390/kernel/head64.S
+++ b/arch/s390/kernel/head64.S
@@ -52,7 +52,7 @@
.quad 0 # cr1: primary space segment table
.quad .Lduct # cr2: dispatchable unit control table
.quad 0 # cr3: instruction authorization
- .quad 0 # cr4: instruction authorization
+ .quad 0xffff # cr4: instruction authorization
.quad .Lduct # cr5: primary-aste origin
.quad 0 # cr6: I/O interrupts
.quad 0 # cr7: secondary space segment table
diff --git a/arch/s390/kernel/kdebugfs.c b/arch/s390/kernel/kdebugfs.c
new file mode 100644
index 0000000..ee85e17
--- /dev/null
+++ b/arch/s390/kernel/kdebugfs.c
@@ -0,0 +1,15 @@
+#include <linux/debugfs.h>
+#include <linux/export.h>
+#include <linux/init.h>
+
+struct dentry *arch_debugfs_dir;
+EXPORT_SYMBOL(arch_debugfs_dir);
+
+static int __init arch_kdebugfs_init(void)
+{
+ arch_debugfs_dir = debugfs_create_dir("s390", NULL);
+ if (IS_ERR(arch_debugfs_dir))
+ arch_debugfs_dir = NULL;
+ return 0;
+}
+postcore_initcall(arch_kdebugfs_init);
diff --git a/arch/s390/kernel/machine_kexec.c b/arch/s390/kernel/machine_kexec.c
index 3074c1d..db5658d 100644
--- a/arch/s390/kernel/machine_kexec.c
+++ b/arch/s390/kernel/machine_kexec.c
@@ -27,6 +27,7 @@
#include <asm/cacheflush.h>
#include <asm/os_info.h>
#include <asm/switch_to.h>
+#include <asm/nmi.h>
typedef void (*relocate_kernel_t)(kimage_entry_t *, unsigned long);
@@ -102,6 +103,8 @@
*/
static noinline void __machine_kdump(void *image)
{
+ struct mcesa *mcesa;
+ unsigned long cr2_old, cr2_new;
int this_cpu, cpu;
lgr_info_log();
@@ -114,8 +117,16 @@
continue;
}
/* Store status of the boot CPU */
+ mcesa = (struct mcesa *)(S390_lowcore.mcesad & MCESA_ORIGIN_MASK);
if (MACHINE_HAS_VX)
- save_vx_regs((void *) &S390_lowcore.vector_save_area);
+ save_vx_regs((__vector128 *) mcesa->vector_save_area);
+ if (MACHINE_HAS_GS) {
+ __ctl_store(cr2_old, 2, 2);
+ cr2_new = cr2_old | (1UL << 4);
+ __ctl_load(cr2_new, 2, 2);
+ save_gs_cb((struct gs_cb *) mcesa->guarded_storage_save_area);
+ __ctl_load(cr2_old, 2, 2);
+ }
/*
* To create a good backchain for this CPU in the dump store_status
* is passed the address of a function. The address is saved into
diff --git a/arch/s390/kernel/nmi.c b/arch/s390/kernel/nmi.c
index 9bf8327..9855895 100644
--- a/arch/s390/kernel/nmi.c
+++ b/arch/s390/kernel/nmi.c
@@ -106,6 +106,7 @@
int kill_task;
u64 zero;
void *fpt_save_area;
+ struct mcesa *mcesa;
kill_task = 0;
zero = 0;
@@ -165,6 +166,7 @@
: : "Q" (S390_lowcore.fpt_creg_save_area));
}
+ mcesa = (struct mcesa *)(S390_lowcore.mcesad & MCESA_ORIGIN_MASK);
if (!MACHINE_HAS_VX) {
/* Validate floating point registers */
asm volatile(
@@ -209,8 +211,8 @@
" la 1,%0\n"
" .word 0xe70f,0x1000,0x0036\n" /* vlm 0,15,0(1) */
" .word 0xe70f,0x1100,0x0c36\n" /* vlm 16,31,256(1) */
- : : "Q" (*(struct vx_array *)
- &S390_lowcore.vector_save_area) : "1");
+ : : "Q" (*(struct vx_array *) mcesa->vector_save_area)
+ : "1");
__ctl_load(S390_lowcore.cregs_save_area[0], 0, 0);
}
/* Validate access registers */
@@ -224,6 +226,19 @@
*/
kill_task = 1;
}
+ /* Validate guarded storage registers */
+ if (MACHINE_HAS_GS && (S390_lowcore.cregs_save_area[2] & (1UL << 4))) {
+ if (!mci.gs)
+ /*
+ * Guarded storage register can't be restored and
+ * the current processes uses guarded storage.
+ * It has to be terminated.
+ */
+ kill_task = 1;
+ else
+ load_gs_cb((struct gs_cb *)
+ mcesa->guarded_storage_save_area);
+ }
/*
* We don't even try to validate the TOD register, since we simply
* can't write something sensible into that register.
diff --git a/arch/s390/kernel/perf_cpum_cf.c b/arch/s390/kernel/perf_cpum_cf.c
index 1aba10e..746d034 100644
--- a/arch/s390/kernel/perf_cpum_cf.c
+++ b/arch/s390/kernel/perf_cpum_cf.c
@@ -1,7 +1,7 @@
/*
* Performance event support for s390x - CPU-measurement Counter Facility
*
- * Copyright IBM Corp. 2012
+ * Copyright IBM Corp. 2012, 2017
* Author(s): Hendrik Brueckner <brueckner@linux.vnet.ibm.com>
*
* This program is free software; you can redistribute it and/or modify
@@ -22,19 +22,12 @@
#include <asm/irq.h>
#include <asm/cpu_mf.h>
-/* CPU-measurement counter facility supports these CPU counter sets:
- * For CPU counter sets:
- * Basic counter set: 0-31
- * Problem-state counter set: 32-63
- * Crypto-activity counter set: 64-127
- * Extented counter set: 128-159
- */
enum cpumf_ctr_set {
- /* CPU counter sets */
- CPUMF_CTR_SET_BASIC = 0,
- CPUMF_CTR_SET_USER = 1,
- CPUMF_CTR_SET_CRYPTO = 2,
- CPUMF_CTR_SET_EXT = 3,
+ CPUMF_CTR_SET_BASIC = 0, /* Basic Counter Set */
+ CPUMF_CTR_SET_USER = 1, /* Problem-State Counter Set */
+ CPUMF_CTR_SET_CRYPTO = 2, /* Crypto-Activity Counter Set */
+ CPUMF_CTR_SET_EXT = 3, /* Extended Counter Set */
+ CPUMF_CTR_SET_MT_DIAG = 4, /* MT-diagnostic Counter Set */
/* Maximum number of counter sets */
CPUMF_CTR_SET_MAX,
@@ -47,6 +40,7 @@
[CPUMF_CTR_SET_USER] = 0x04,
[CPUMF_CTR_SET_CRYPTO] = 0x08,
[CPUMF_CTR_SET_EXT] = 0x01,
+ [CPUMF_CTR_SET_MT_DIAG] = 0x20,
};
static void ctr_set_enable(u64 *state, int ctr_set)
@@ -76,19 +70,20 @@
};
static DEFINE_PER_CPU(struct cpu_hw_events, cpu_hw_events) = {
.ctr_set = {
- [CPUMF_CTR_SET_BASIC] = ATOMIC_INIT(0),
- [CPUMF_CTR_SET_USER] = ATOMIC_INIT(0),
- [CPUMF_CTR_SET_CRYPTO] = ATOMIC_INIT(0),
- [CPUMF_CTR_SET_EXT] = ATOMIC_INIT(0),
+ [CPUMF_CTR_SET_BASIC] = ATOMIC_INIT(0),
+ [CPUMF_CTR_SET_USER] = ATOMIC_INIT(0),
+ [CPUMF_CTR_SET_CRYPTO] = ATOMIC_INIT(0),
+ [CPUMF_CTR_SET_EXT] = ATOMIC_INIT(0),
+ [CPUMF_CTR_SET_MT_DIAG] = ATOMIC_INIT(0),
},
.state = 0,
.flags = 0,
.txn_flags = 0,
};
-static int get_counter_set(u64 event)
+static enum cpumf_ctr_set get_counter_set(u64 event)
{
- int set = -1;
+ int set = CPUMF_CTR_SET_MAX;
if (event < 32)
set = CPUMF_CTR_SET_BASIC;
@@ -98,34 +93,17 @@
set = CPUMF_CTR_SET_CRYPTO;
else if (event < 256)
set = CPUMF_CTR_SET_EXT;
+ else if (event >= 448 && event < 496)
+ set = CPUMF_CTR_SET_MT_DIAG;
return set;
}
-static int validate_event(const struct hw_perf_event *hwc)
-{
- switch (hwc->config_base) {
- case CPUMF_CTR_SET_BASIC:
- case CPUMF_CTR_SET_USER:
- case CPUMF_CTR_SET_CRYPTO:
- case CPUMF_CTR_SET_EXT:
- /* check for reserved counters */
- if ((hwc->config >= 6 && hwc->config <= 31) ||
- (hwc->config >= 38 && hwc->config <= 63) ||
- (hwc->config >= 80 && hwc->config <= 127))
- return -EOPNOTSUPP;
- break;
- default:
- return -EINVAL;
- }
-
- return 0;
-}
-
static int validate_ctr_version(const struct hw_perf_event *hwc)
{
struct cpu_hw_events *cpuhw;
int err = 0;
+ u16 mtdiag_ctl;
cpuhw = &get_cpu_var(cpu_hw_events);
@@ -145,6 +123,27 @@
(cpuhw->info.csvn > 2 && hwc->config > 255))
err = -EOPNOTSUPP;
break;
+ case CPUMF_CTR_SET_MT_DIAG:
+ if (cpuhw->info.csvn <= 3)
+ err = -EOPNOTSUPP;
+ /*
+ * MT-diagnostic counters are read-only. The counter set
+ * is automatically enabled and activated on all CPUs with
+ * multithreading (SMT). Deactivation of multithreading
+ * also disables the counter set. State changes are ignored
+ * by lcctl(). Because Linux controls SMT enablement through
+ * a kernel parameter only, the counter set is either disabled
+ * or enabled and active.
+ *
+ * Thus, the counters can only be used if SMT is on and the
+ * counter set is enabled and active.
+ */
+ mtdiag_ctl = cpumf_state_ctl[CPUMF_CTR_SET_MT_DIAG];
+ if (!((cpuhw->info.auth_ctl & mtdiag_ctl) &&
+ (cpuhw->info.enable_ctl & mtdiag_ctl) &&
+ (cpuhw->info.act_ctl & mtdiag_ctl)))
+ err = -EOPNOTSUPP;
+ break;
}
put_cpu_var(cpu_hw_events);
@@ -250,6 +249,11 @@
/* loss of counter data alert */
if (alert & CPU_MF_INT_CF_LCDA)
pr_err("CPU[%i] Counter data was lost\n", smp_processor_id());
+
+ /* loss of MT counter data alert */
+ if (alert & CPU_MF_INT_CF_MTDA)
+ pr_warn("CPU[%i] MT counter data was lost\n",
+ smp_processor_id());
}
#define PMC_INIT 0
@@ -330,6 +334,7 @@
{
struct perf_event_attr *attr = &event->attr;
struct hw_perf_event *hwc = &event->hw;
+ enum cpumf_ctr_set set;
int err;
u64 ev;
@@ -370,25 +375,30 @@
if (ev == -1)
return -ENOENT;
- if (ev >= PERF_CPUM_CF_MAX_CTR)
+ if (ev > PERF_CPUM_CF_MAX_CTR)
return -EINVAL;
- /* Use the hardware perf event structure to store the counter number
- * in 'config' member and the counter set to which the counter belongs
- * in the 'config_base'. The counter set (config_base) is then used
- * to enable/disable the counters.
- */
- hwc->config = ev;
- hwc->config_base = get_counter_set(ev);
-
- /* Validate the counter that is assigned to this event.
- * Because the counter facility can use numerous counters at the
- * same time without constraints, it is not necessary to explicitly
- * validate event groups (event->group_leader != event).
- */
- err = validate_event(hwc);
- if (err)
- return err;
+ /* Obtain the counter set to which the specified counter belongs */
+ set = get_counter_set(ev);
+ switch (set) {
+ case CPUMF_CTR_SET_BASIC:
+ case CPUMF_CTR_SET_USER:
+ case CPUMF_CTR_SET_CRYPTO:
+ case CPUMF_CTR_SET_EXT:
+ case CPUMF_CTR_SET_MT_DIAG:
+ /*
+ * Use the hardware perf event structure to store the
+ * counter number in the 'config' member and the counter
+ * set number in the 'config_base'. The counter set number
+ * is then later used to enable/disable the counter(s).
+ */
+ hwc->config = ev;
+ hwc->config_base = set;
+ break;
+ case CPUMF_CTR_SET_MAX:
+ /* The counter could not be associated to a counter set */
+ return -EINVAL;
+ };
/* Initialize for using the CPU-measurement counter facility */
if (!atomic_inc_not_zero(&num_events)) {
@@ -452,7 +462,7 @@
return err;
}
-static int hw_perf_event_update(struct perf_event *event)
+static void hw_perf_event_update(struct perf_event *event)
{
u64 prev, new, delta;
int err;
@@ -461,14 +471,12 @@
prev = local64_read(&event->hw.prev_count);
err = ecctr(event->hw.config, &new);
if (err)
- goto out;
+ return;
} while (local64_cmpxchg(&event->hw.prev_count, prev, new) != prev);
delta = (prev <= new) ? new - prev
: (-1ULL - prev) + new + 1; /* overflow */
local64_add(delta, &event->count);
-out:
- return err;
}
static void cpumf_pmu_read(struct perf_event *event)
diff --git a/arch/s390/kernel/perf_cpum_cf_events.c b/arch/s390/kernel/perf_cpum_cf_events.c
index c343ac2..d313328 100644
--- a/arch/s390/kernel/perf_cpum_cf_events.c
+++ b/arch/s390/kernel/perf_cpum_cf_events.c
@@ -114,8 +114,64 @@
CPUMF_EVENT_ATTR(cf_zec12, TX_NC_TABORT, 0x00b1);
CPUMF_EVENT_ATTR(cf_zec12, TX_C_TABORT_NO_SPECIAL, 0x00b2);
CPUMF_EVENT_ATTR(cf_zec12, TX_C_TABORT_SPECIAL, 0x00b3);
+CPUMF_EVENT_ATTR(cf_z13, L1D_WRITES_RO_EXCL, 0x0080);
+CPUMF_EVENT_ATTR(cf_z13, DTLB1_WRITES, 0x0081);
+CPUMF_EVENT_ATTR(cf_z13, DTLB1_MISSES, 0x0082);
+CPUMF_EVENT_ATTR(cf_z13, DTLB1_HPAGE_WRITES, 0x0083);
+CPUMF_EVENT_ATTR(cf_z13, DTLB1_GPAGE_WRITES, 0x0084);
+CPUMF_EVENT_ATTR(cf_z13, L1D_L2D_SOURCED_WRITES, 0x0085);
+CPUMF_EVENT_ATTR(cf_z13, ITLB1_WRITES, 0x0086);
+CPUMF_EVENT_ATTR(cf_z13, ITLB1_MISSES, 0x0087);
+CPUMF_EVENT_ATTR(cf_z13, L1I_L2I_SOURCED_WRITES, 0x0088);
+CPUMF_EVENT_ATTR(cf_z13, TLB2_PTE_WRITES, 0x0089);
+CPUMF_EVENT_ATTR(cf_z13, TLB2_CRSTE_HPAGE_WRITES, 0x008a);
+CPUMF_EVENT_ATTR(cf_z13, TLB2_CRSTE_WRITES, 0x008b);
+CPUMF_EVENT_ATTR(cf_z13, TX_C_TEND, 0x008c);
+CPUMF_EVENT_ATTR(cf_z13, TX_NC_TEND, 0x008d);
+CPUMF_EVENT_ATTR(cf_z13, L1C_TLB1_MISSES, 0x008f);
+CPUMF_EVENT_ATTR(cf_z13, L1D_ONCHIP_L3_SOURCED_WRITES, 0x0090);
+CPUMF_EVENT_ATTR(cf_z13, L1D_ONCHIP_L3_SOURCED_WRITES_IV, 0x0091);
+CPUMF_EVENT_ATTR(cf_z13, L1D_ONNODE_L4_SOURCED_WRITES, 0x0092);
+CPUMF_EVENT_ATTR(cf_z13, L1D_ONNODE_L3_SOURCED_WRITES_IV, 0x0093);
+CPUMF_EVENT_ATTR(cf_z13, L1D_ONNODE_L3_SOURCED_WRITES, 0x0094);
+CPUMF_EVENT_ATTR(cf_z13, L1D_ONDRAWER_L4_SOURCED_WRITES, 0x0095);
+CPUMF_EVENT_ATTR(cf_z13, L1D_ONDRAWER_L3_SOURCED_WRITES_IV, 0x0096);
+CPUMF_EVENT_ATTR(cf_z13, L1D_ONDRAWER_L3_SOURCED_WRITES, 0x0097);
+CPUMF_EVENT_ATTR(cf_z13, L1D_OFFDRAWER_SCOL_L4_SOURCED_WRITES, 0x0098);
+CPUMF_EVENT_ATTR(cf_z13, L1D_OFFDRAWER_SCOL_L3_SOURCED_WRITES_IV, 0x0099);
+CPUMF_EVENT_ATTR(cf_z13, L1D_OFFDRAWER_SCOL_L3_SOURCED_WRITES, 0x009a);
+CPUMF_EVENT_ATTR(cf_z13, L1D_OFFDRAWER_FCOL_L4_SOURCED_WRITES, 0x009b);
+CPUMF_EVENT_ATTR(cf_z13, L1D_OFFDRAWER_FCOL_L3_SOURCED_WRITES_IV, 0x009c);
+CPUMF_EVENT_ATTR(cf_z13, L1D_OFFDRAWER_FCOL_L3_SOURCED_WRITES, 0x009d);
+CPUMF_EVENT_ATTR(cf_z13, L1D_ONNODE_MEM_SOURCED_WRITES, 0x009e);
+CPUMF_EVENT_ATTR(cf_z13, L1D_ONDRAWER_MEM_SOURCED_WRITES, 0x009f);
+CPUMF_EVENT_ATTR(cf_z13, L1D_OFFDRAWER_MEM_SOURCED_WRITES, 0x00a0);
+CPUMF_EVENT_ATTR(cf_z13, L1D_ONCHIP_MEM_SOURCED_WRITES, 0x00a1);
+CPUMF_EVENT_ATTR(cf_z13, L1I_ONCHIP_L3_SOURCED_WRITES, 0x00a2);
+CPUMF_EVENT_ATTR(cf_z13, L1I_ONCHIP_L3_SOURCED_WRITES_IV, 0x00a3);
+CPUMF_EVENT_ATTR(cf_z13, L1I_ONNODE_L4_SOURCED_WRITES, 0x00a4);
+CPUMF_EVENT_ATTR(cf_z13, L1I_ONNODE_L3_SOURCED_WRITES_IV, 0x00a5);
+CPUMF_EVENT_ATTR(cf_z13, L1I_ONNODE_L3_SOURCED_WRITES, 0x00a6);
+CPUMF_EVENT_ATTR(cf_z13, L1I_ONDRAWER_L4_SOURCED_WRITES, 0x00a7);
+CPUMF_EVENT_ATTR(cf_z13, L1I_ONDRAWER_L3_SOURCED_WRITES_IV, 0x00a8);
+CPUMF_EVENT_ATTR(cf_z13, L1I_ONDRAWER_L3_SOURCED_WRITES, 0x00a9);
+CPUMF_EVENT_ATTR(cf_z13, L1I_OFFDRAWER_SCOL_L4_SOURCED_WRITES, 0x00aa);
+CPUMF_EVENT_ATTR(cf_z13, L1I_OFFDRAWER_SCOL_L3_SOURCED_WRITES_IV, 0x00ab);
+CPUMF_EVENT_ATTR(cf_z13, L1I_OFFDRAWER_SCOL_L3_SOURCED_WRITES, 0x00ac);
+CPUMF_EVENT_ATTR(cf_z13, L1I_OFFDRAWER_FCOL_L4_SOURCED_WRITES, 0x00ad);
+CPUMF_EVENT_ATTR(cf_z13, L1I_OFFDRAWER_FCOL_L3_SOURCED_WRITES_IV, 0x00ae);
+CPUMF_EVENT_ATTR(cf_z13, L1I_OFFDRAWER_FCOL_L3_SOURCED_WRITES, 0x00af);
+CPUMF_EVENT_ATTR(cf_z13, L1I_ONNODE_MEM_SOURCED_WRITES, 0x00b0);
+CPUMF_EVENT_ATTR(cf_z13, L1I_ONDRAWER_MEM_SOURCED_WRITES, 0x00b1);
+CPUMF_EVENT_ATTR(cf_z13, L1I_OFFDRAWER_MEM_SOURCED_WRITES, 0x00b2);
+CPUMF_EVENT_ATTR(cf_z13, L1I_ONCHIP_MEM_SOURCED_WRITES, 0x00b3);
+CPUMF_EVENT_ATTR(cf_z13, TX_NC_TABORT, 0x00da);
+CPUMF_EVENT_ATTR(cf_z13, TX_C_TABORT_NO_SPECIAL, 0x00db);
+CPUMF_EVENT_ATTR(cf_z13, TX_C_TABORT_SPECIAL, 0x00dc);
+CPUMF_EVENT_ATTR(cf_z13, MT_DIAG_CYCLES_ONE_THR_ACTIVE, 0x01c0);
+CPUMF_EVENT_ATTR(cf_z13, MT_DIAG_CYCLES_TWO_THR_ACTIVE, 0x01c1);
-static struct attribute *cpumcf_pmu_event_attr[] = {
+static struct attribute *cpumcf_pmu_event_attr[] __initdata = {
CPUMF_EVENT_PTR(cf, CPU_CYCLES),
CPUMF_EVENT_PTR(cf, INSTRUCTIONS),
CPUMF_EVENT_PTR(cf, L1I_DIR_WRITES),
@@ -236,28 +292,87 @@
NULL,
};
+static struct attribute *cpumcf_z13_pmu_event_attr[] __initdata = {
+ CPUMF_EVENT_PTR(cf_z13, L1D_WRITES_RO_EXCL),
+ CPUMF_EVENT_PTR(cf_z13, DTLB1_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, DTLB1_MISSES),
+ CPUMF_EVENT_PTR(cf_z13, DTLB1_HPAGE_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, DTLB1_GPAGE_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_L2D_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, ITLB1_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, ITLB1_MISSES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_L2I_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, TLB2_PTE_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, TLB2_CRSTE_HPAGE_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, TLB2_CRSTE_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, TX_C_TEND),
+ CPUMF_EVENT_PTR(cf_z13, TX_NC_TEND),
+ CPUMF_EVENT_PTR(cf_z13, L1C_TLB1_MISSES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_ONCHIP_L3_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_ONCHIP_L3_SOURCED_WRITES_IV),
+ CPUMF_EVENT_PTR(cf_z13, L1D_ONNODE_L4_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_ONNODE_L3_SOURCED_WRITES_IV),
+ CPUMF_EVENT_PTR(cf_z13, L1D_ONNODE_L3_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_ONDRAWER_L4_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_ONDRAWER_L3_SOURCED_WRITES_IV),
+ CPUMF_EVENT_PTR(cf_z13, L1D_ONDRAWER_L3_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_OFFDRAWER_SCOL_L4_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_OFFDRAWER_SCOL_L3_SOURCED_WRITES_IV),
+ CPUMF_EVENT_PTR(cf_z13, L1D_OFFDRAWER_SCOL_L3_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_OFFDRAWER_FCOL_L4_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_OFFDRAWER_FCOL_L3_SOURCED_WRITES_IV),
+ CPUMF_EVENT_PTR(cf_z13, L1D_OFFDRAWER_FCOL_L3_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_ONNODE_MEM_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_ONDRAWER_MEM_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_OFFDRAWER_MEM_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1D_ONCHIP_MEM_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_ONCHIP_L3_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_ONCHIP_L3_SOURCED_WRITES_IV),
+ CPUMF_EVENT_PTR(cf_z13, L1I_ONNODE_L4_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_ONNODE_L3_SOURCED_WRITES_IV),
+ CPUMF_EVENT_PTR(cf_z13, L1I_ONNODE_L3_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_ONDRAWER_L4_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_ONDRAWER_L3_SOURCED_WRITES_IV),
+ CPUMF_EVENT_PTR(cf_z13, L1I_ONDRAWER_L3_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_OFFDRAWER_SCOL_L4_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_OFFDRAWER_SCOL_L3_SOURCED_WRITES_IV),
+ CPUMF_EVENT_PTR(cf_z13, L1I_OFFDRAWER_SCOL_L3_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_OFFDRAWER_FCOL_L4_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_OFFDRAWER_FCOL_L3_SOURCED_WRITES_IV),
+ CPUMF_EVENT_PTR(cf_z13, L1I_OFFDRAWER_FCOL_L3_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_ONNODE_MEM_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_ONDRAWER_MEM_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_OFFDRAWER_MEM_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, L1I_ONCHIP_MEM_SOURCED_WRITES),
+ CPUMF_EVENT_PTR(cf_z13, TX_NC_TABORT),
+ CPUMF_EVENT_PTR(cf_z13, TX_C_TABORT_NO_SPECIAL),
+ CPUMF_EVENT_PTR(cf_z13, TX_C_TABORT_SPECIAL),
+ CPUMF_EVENT_PTR(cf_z13, MT_DIAG_CYCLES_ONE_THR_ACTIVE),
+ CPUMF_EVENT_PTR(cf_z13, MT_DIAG_CYCLES_TWO_THR_ACTIVE),
+ NULL,
+};
+
/* END: CPUM_CF COUNTER DEFINITIONS ===================================== */
-static struct attribute_group cpumsf_pmu_events_group = {
+static struct attribute_group cpumcf_pmu_events_group = {
.name = "events",
- .attrs = cpumcf_pmu_event_attr,
};
PMU_FORMAT_ATTR(event, "config:0-63");
-static struct attribute *cpumsf_pmu_format_attr[] = {
+static struct attribute *cpumcf_pmu_format_attr[] = {
&format_attr_event.attr,
NULL,
};
-static struct attribute_group cpumsf_pmu_format_group = {
+static struct attribute_group cpumcf_pmu_format_group = {
.name = "format",
- .attrs = cpumsf_pmu_format_attr,
+ .attrs = cpumcf_pmu_format_attr,
};
-static const struct attribute_group *cpumsf_pmu_attr_groups[] = {
- &cpumsf_pmu_events_group,
- &cpumsf_pmu_format_group,
+static const struct attribute_group *cpumcf_pmu_attr_groups[] = {
+ &cpumcf_pmu_events_group,
+ &cpumcf_pmu_format_group,
NULL,
};
@@ -290,6 +405,7 @@
__init const struct attribute_group **cpumf_cf_event_group(void)
{
struct attribute **combined, **model;
+ struct attribute *none[] = { NULL };
struct cpuid cpu_id;
get_cpu_id(&cpu_id);
@@ -306,17 +422,17 @@
case 0x2828:
model = cpumcf_zec12_pmu_event_attr;
break;
+ case 0x2964:
+ case 0x2965:
+ model = cpumcf_z13_pmu_event_attr;
+ break;
default:
- model = NULL;
+ model = none;
break;
}
- if (!model)
- goto out;
-
combined = merge_attr(cpumcf_pmu_event_attr, model);
if (combined)
- cpumsf_pmu_events_group.attrs = combined;
-out:
- return cpumsf_pmu_attr_groups;
+ cpumcf_pmu_events_group.attrs = combined;
+ return cpumcf_pmu_attr_groups;
}
diff --git a/arch/s390/kernel/perf_cpum_sf.c b/arch/s390/kernel/perf_cpum_sf.c
index 1c0b585..9a4f279 100644
--- a/arch/s390/kernel/perf_cpum_sf.c
+++ b/arch/s390/kernel/perf_cpum_sf.c
@@ -1009,8 +1009,8 @@
* sample. Some early samples or samples from guests without
* lpp usage would be misaccounted to the host. We use the asn
* value as an addon heuristic to detect most of these guest samples.
- * If the value differs from the host hpp value, we assume to be a
- * KVM guest.
+ * If the value differs from 0xffff (the host value), we assume to
+ * be a KVM guest.
*/
switch (sfr->basic.CL) {
case 1: /* logical partition */
@@ -1020,8 +1020,7 @@
sde_regs->in_guest = 1;
break;
default: /* old machine, use heuristics */
- if (sfr->basic.gpp ||
- sfr->basic.prim_asn != (u16)sfr->basic.hpp)
+ if (sfr->basic.gpp || sfr->basic.prim_asn != 0xffff)
sde_regs->in_guest = 1;
break;
}
diff --git a/arch/s390/kernel/process.c b/arch/s390/kernel/process.c
index f29e41c..999d715 100644
--- a/arch/s390/kernel/process.c
+++ b/arch/s390/kernel/process.c
@@ -73,8 +73,10 @@
*/
void exit_thread(struct task_struct *tsk)
{
- if (tsk == current)
+ if (tsk == current) {
exit_thread_runtime_instr();
+ exit_thread_gs();
+ }
}
void flush_thread(void)
@@ -159,6 +161,9 @@
/* Don't copy runtime instrumentation info */
p->thread.ri_cb = NULL;
frame->childregs.psw.mask &= ~PSW_MASK_RI;
+ /* Don't copy guarded storage control block */
+ p->thread.gs_cb = NULL;
+ p->thread.gs_bc_cb = NULL;
/* Set a new TLS ? */
if (clone_flags & CLONE_SETTLS) {
diff --git a/arch/s390/kernel/processor.c b/arch/s390/kernel/processor.c
index 928b929..778cd65 100644
--- a/arch/s390/kernel/processor.c
+++ b/arch/s390/kernel/processor.c
@@ -7,6 +7,7 @@
#define pr_fmt(fmt) KMSG_COMPONENT ": " fmt
#include <linux/cpufeature.h>
+#include <linux/bitops.h>
#include <linux/kernel.h>
#include <linux/sched/mm.h>
#include <linux/init.h>
@@ -91,11 +92,23 @@
}
EXPORT_SYMBOL(cpu_have_feature);
+static void show_facilities(struct seq_file *m)
+{
+ unsigned int bit;
+ long *facilities;
+
+ facilities = (long *)&S390_lowcore.stfle_fac_list;
+ seq_puts(m, "facilities :");
+ for_each_set_bit_inv(bit, facilities, MAX_FACILITY_BIT)
+ seq_printf(m, " %d", bit);
+ seq_putc(m, '\n');
+}
+
static void show_cpu_summary(struct seq_file *m, void *v)
{
static const char *hwcap_str[] = {
"esan3", "zarch", "stfle", "msa", "ldisp", "eimm", "dfp",
- "edat", "etf3eh", "highgprs", "te", "vx", "vxd", "vxe"
+ "edat", "etf3eh", "highgprs", "te", "vx", "vxd", "vxe", "gs"
};
static const char * const int_hwcap_str[] = {
"sie"
@@ -116,6 +129,7 @@
if (int_hwcap_str[i] && (int_hwcap & (1UL << i)))
seq_printf(m, "%s ", int_hwcap_str[i]);
seq_puts(m, "\n");
+ show_facilities(m);
show_cacheinfo(m);
for_each_online_cpu(cpu) {
struct cpuid *id = &per_cpu(cpu_info.cpu_id, cpu);
diff --git a/arch/s390/kernel/ptrace.c b/arch/s390/kernel/ptrace.c
index c14df0a..488c5bb 100644
--- a/arch/s390/kernel/ptrace.c
+++ b/arch/s390/kernel/ptrace.c
@@ -44,30 +44,42 @@
struct pt_regs *regs = task_pt_regs(task);
struct thread_struct *thread = &task->thread;
struct per_regs old, new;
+ unsigned long cr0_old, cr0_new;
+ unsigned long cr2_old, cr2_new;
+ int cr0_changed, cr2_changed;
+ __ctl_store(cr0_old, 0, 0);
+ __ctl_store(cr2_old, 2, 2);
+ cr0_new = cr0_old;
+ cr2_new = cr2_old;
/* Take care of the enable/disable of transactional execution. */
if (MACHINE_HAS_TE) {
- unsigned long cr, cr_new;
-
- __ctl_store(cr, 0, 0);
/* Set or clear transaction execution TXC bit 8. */
- cr_new = cr | (1UL << 55);
+ cr0_new |= (1UL << 55);
if (task->thread.per_flags & PER_FLAG_NO_TE)
- cr_new &= ~(1UL << 55);
- if (cr_new != cr)
- __ctl_load(cr_new, 0, 0);
+ cr0_new &= ~(1UL << 55);
/* Set or clear transaction execution TDC bits 62 and 63. */
- __ctl_store(cr, 2, 2);
- cr_new = cr & ~3UL;
+ cr2_new &= ~3UL;
if (task->thread.per_flags & PER_FLAG_TE_ABORT_RAND) {
if (task->thread.per_flags & PER_FLAG_TE_ABORT_RAND_TEND)
- cr_new |= 1UL;
+ cr2_new |= 1UL;
else
- cr_new |= 2UL;
+ cr2_new |= 2UL;
}
- if (cr_new != cr)
- __ctl_load(cr_new, 2, 2);
}
+ /* Take care of enable/disable of guarded storage. */
+ if (MACHINE_HAS_GS) {
+ cr2_new &= ~(1UL << 4);
+ if (task->thread.gs_cb)
+ cr2_new |= (1UL << 4);
+ }
+ /* Load control register 0/2 iff changed */
+ cr0_changed = cr0_new != cr0_old;
+ cr2_changed = cr2_new != cr2_old;
+ if (cr0_changed)
+ __ctl_load(cr0_new, 0, 0);
+ if (cr2_changed)
+ __ctl_load(cr2_new, 2, 2);
/* Copy user specified PER registers */
new.control = thread->per_user.control;
new.start = thread->per_user.start;
@@ -1137,6 +1149,74 @@
data, 0, sizeof(unsigned int));
}
+static int s390_gs_cb_get(struct task_struct *target,
+ const struct user_regset *regset,
+ unsigned int pos, unsigned int count,
+ void *kbuf, void __user *ubuf)
+{
+ struct gs_cb *data = target->thread.gs_cb;
+
+ if (!MACHINE_HAS_GS)
+ return -ENODEV;
+ if (!data)
+ return -ENODATA;
+ return user_regset_copyout(&pos, &count, &kbuf, &ubuf,
+ data, 0, sizeof(struct gs_cb));
+}
+
+static int s390_gs_cb_set(struct task_struct *target,
+ const struct user_regset *regset,
+ unsigned int pos, unsigned int count,
+ const void *kbuf, const void __user *ubuf)
+{
+ struct gs_cb *data = target->thread.gs_cb;
+
+ if (!MACHINE_HAS_GS)
+ return -ENODEV;
+ if (!data) {
+ data = kzalloc(sizeof(*data), GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+ target->thread.gs_cb = data;
+ }
+ return user_regset_copyin(&pos, &count, &kbuf, &ubuf,
+ data, 0, sizeof(struct gs_cb));
+}
+
+static int s390_gs_bc_get(struct task_struct *target,
+ const struct user_regset *regset,
+ unsigned int pos, unsigned int count,
+ void *kbuf, void __user *ubuf)
+{
+ struct gs_cb *data = target->thread.gs_bc_cb;
+
+ if (!MACHINE_HAS_GS)
+ return -ENODEV;
+ if (!data)
+ return -ENODATA;
+ return user_regset_copyout(&pos, &count, &kbuf, &ubuf,
+ data, 0, sizeof(struct gs_cb));
+}
+
+static int s390_gs_bc_set(struct task_struct *target,
+ const struct user_regset *regset,
+ unsigned int pos, unsigned int count,
+ const void *kbuf, const void __user *ubuf)
+{
+ struct gs_cb *data = target->thread.gs_bc_cb;
+
+ if (!MACHINE_HAS_GS)
+ return -ENODEV;
+ if (!data) {
+ data = kzalloc(sizeof(*data), GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+ target->thread.gs_bc_cb = data;
+ }
+ return user_regset_copyin(&pos, &count, &kbuf, &ubuf,
+ data, 0, sizeof(struct gs_cb));
+}
+
static const struct user_regset s390_regsets[] = {
{
.core_note_type = NT_PRSTATUS,
@@ -1194,6 +1274,22 @@
.get = s390_vxrs_high_get,
.set = s390_vxrs_high_set,
},
+ {
+ .core_note_type = NT_S390_GS_CB,
+ .n = sizeof(struct gs_cb) / sizeof(__u64),
+ .size = sizeof(__u64),
+ .align = sizeof(__u64),
+ .get = s390_gs_cb_get,
+ .set = s390_gs_cb_set,
+ },
+ {
+ .core_note_type = NT_S390_GS_BC,
+ .n = sizeof(struct gs_cb) / sizeof(__u64),
+ .size = sizeof(__u64),
+ .align = sizeof(__u64),
+ .get = s390_gs_bc_get,
+ .set = s390_gs_bc_set,
+ },
};
static const struct user_regset_view user_s390_view = {
@@ -1422,6 +1518,14 @@
.get = s390_compat_regs_high_get,
.set = s390_compat_regs_high_set,
},
+ {
+ .core_note_type = NT_S390_GS_CB,
+ .n = sizeof(struct gs_cb) / sizeof(__u64),
+ .size = sizeof(__u64),
+ .align = sizeof(__u64),
+ .get = s390_gs_cb_get,
+ .set = s390_gs_cb_set,
+ },
};
static const struct user_regset_view user_s390_compat_view = {
diff --git a/arch/s390/kernel/setup.c b/arch/s390/kernel/setup.c
index 911dc0b..3ae756c 100644
--- a/arch/s390/kernel/setup.c
+++ b/arch/s390/kernel/setup.c
@@ -339,9 +339,15 @@
lc->stfl_fac_list = S390_lowcore.stfl_fac_list;
memcpy(lc->stfle_fac_list, S390_lowcore.stfle_fac_list,
MAX_FACILITY_BIT/8);
- if (MACHINE_HAS_VX)
- lc->vector_save_area_addr =
- (unsigned long) &lc->vector_save_area;
+ if (MACHINE_HAS_VX || MACHINE_HAS_GS) {
+ unsigned long bits, size;
+
+ bits = MACHINE_HAS_GS ? 11 : 10;
+ size = 1UL << bits;
+ lc->mcesad = (__u64) memblock_virt_alloc(size, size);
+ if (MACHINE_HAS_GS)
+ lc->mcesad |= bits;
+ }
lc->vdso_per_cpu_data = (unsigned long) &lc->paste[0];
lc->sync_enter_timer = S390_lowcore.sync_enter_timer;
lc->async_enter_timer = S390_lowcore.async_enter_timer;
@@ -779,6 +785,12 @@
elf_hwcap |= HWCAP_S390_VXRS_BCD;
}
+ /*
+ * Guarded storage support HWCAP_S390_GS is bit 12.
+ */
+ if (MACHINE_HAS_GS)
+ elf_hwcap |= HWCAP_S390_GS;
+
get_cpu_id(&cpu_id);
add_device_randomness(&cpu_id, sizeof(cpu_id));
switch (cpu_id.machine) {
diff --git a/arch/s390/kernel/smp.c b/arch/s390/kernel/smp.c
index 5dab859..363000a 100644
--- a/arch/s390/kernel/smp.c
+++ b/arch/s390/kernel/smp.c
@@ -51,6 +51,7 @@
#include <asm/os_info.h>
#include <asm/sigp.h>
#include <asm/idle.h>
+#include <asm/nmi.h>
#include "entry.h"
enum {
@@ -78,6 +79,8 @@
static u8 boot_core_type;
static struct pcpu pcpu_devices[NR_CPUS];
+static struct kmem_cache *pcpu_mcesa_cache;
+
unsigned int smp_cpu_mt_shift;
EXPORT_SYMBOL(smp_cpu_mt_shift);
@@ -188,8 +191,10 @@
static int pcpu_alloc_lowcore(struct pcpu *pcpu, int cpu)
{
unsigned long async_stack, panic_stack;
+ unsigned long mcesa_origin, mcesa_bits;
struct lowcore *lc;
+ mcesa_origin = mcesa_bits = 0;
if (pcpu != &pcpu_devices[0]) {
pcpu->lowcore = (struct lowcore *)
__get_free_pages(GFP_KERNEL | GFP_DMA, LC_ORDER);
@@ -197,20 +202,27 @@
panic_stack = __get_free_page(GFP_KERNEL);
if (!pcpu->lowcore || !panic_stack || !async_stack)
goto out;
+ if (MACHINE_HAS_VX || MACHINE_HAS_GS) {
+ mcesa_origin = (unsigned long)
+ kmem_cache_alloc(pcpu_mcesa_cache, GFP_KERNEL);
+ if (!mcesa_origin)
+ goto out;
+ mcesa_bits = MACHINE_HAS_GS ? 11 : 0;
+ }
} else {
async_stack = pcpu->lowcore->async_stack - ASYNC_FRAME_OFFSET;
panic_stack = pcpu->lowcore->panic_stack - PANIC_FRAME_OFFSET;
+ mcesa_origin = pcpu->lowcore->mcesad & MCESA_ORIGIN_MASK;
+ mcesa_bits = pcpu->lowcore->mcesad & MCESA_LC_MASK;
}
lc = pcpu->lowcore;
memcpy(lc, &S390_lowcore, 512);
memset((char *) lc + 512, 0, sizeof(*lc) - 512);
lc->async_stack = async_stack + ASYNC_FRAME_OFFSET;
lc->panic_stack = panic_stack + PANIC_FRAME_OFFSET;
+ lc->mcesad = mcesa_origin | mcesa_bits;
lc->cpu_nr = cpu;
lc->spinlock_lockval = arch_spin_lockval(cpu);
- if (MACHINE_HAS_VX)
- lc->vector_save_area_addr =
- (unsigned long) &lc->vector_save_area;
if (vdso_alloc_per_cpu(lc))
goto out;
lowcore_ptr[cpu] = lc;
@@ -218,6 +230,9 @@
return 0;
out:
if (pcpu != &pcpu_devices[0]) {
+ if (mcesa_origin)
+ kmem_cache_free(pcpu_mcesa_cache,
+ (void *) mcesa_origin);
free_page(panic_stack);
free_pages(async_stack, ASYNC_ORDER);
free_pages((unsigned long) pcpu->lowcore, LC_ORDER);
@@ -229,11 +244,17 @@
static void pcpu_free_lowcore(struct pcpu *pcpu)
{
+ unsigned long mcesa_origin;
+
pcpu_sigp_retry(pcpu, SIGP_SET_PREFIX, 0);
lowcore_ptr[pcpu - pcpu_devices] = NULL;
vdso_free_per_cpu(pcpu->lowcore);
if (pcpu == &pcpu_devices[0])
return;
+ if (MACHINE_HAS_VX || MACHINE_HAS_GS) {
+ mcesa_origin = pcpu->lowcore->mcesad & MCESA_ORIGIN_MASK;
+ kmem_cache_free(pcpu_mcesa_cache, (void *) mcesa_origin);
+ }
free_page(pcpu->lowcore->panic_stack-PANIC_FRAME_OFFSET);
free_pages(pcpu->lowcore->async_stack-ASYNC_FRAME_OFFSET, ASYNC_ORDER);
free_pages((unsigned long) pcpu->lowcore, LC_ORDER);
@@ -550,9 +571,11 @@
if (__pcpu_sigp_relax(pcpu->address, SIGP_STORE_STATUS_AT_ADDRESS,
pa) != SIGP_CC_ORDER_CODE_ACCEPTED)
return -EIO;
- if (!MACHINE_HAS_VX)
+ if (!MACHINE_HAS_VX && !MACHINE_HAS_GS)
return 0;
- pa = __pa(pcpu->lowcore->vector_save_area_addr);
+ pa = __pa(pcpu->lowcore->mcesad & MCESA_ORIGIN_MASK);
+ if (MACHINE_HAS_GS)
+ pa |= pcpu->lowcore->mcesad & MCESA_LC_MASK;
if (__pcpu_sigp_relax(pcpu->address, SIGP_STORE_ADDITIONAL_STATUS,
pa) != SIGP_CC_ORDER_CODE_ACCEPTED)
return -EIO;
@@ -897,12 +920,22 @@
void __init smp_prepare_cpus(unsigned int max_cpus)
{
+ unsigned long size;
+
/* request the 0x1201 emergency signal external interrupt */
if (register_external_irq(EXT_IRQ_EMERGENCY_SIG, do_ext_call_interrupt))
panic("Couldn't request external interrupt 0x1201");
/* request the 0x1202 external call external interrupt */
if (register_external_irq(EXT_IRQ_EXTERNAL_CALL, do_ext_call_interrupt))
panic("Couldn't request external interrupt 0x1202");
+ /* create slab cache for the machine-check-extended-save-areas */
+ if (MACHINE_HAS_VX || MACHINE_HAS_GS) {
+ size = 1UL << (MACHINE_HAS_GS ? 11 : 10);
+ pcpu_mcesa_cache = kmem_cache_create("nmi_save_areas",
+ size, size, 0, NULL);
+ if (!pcpu_mcesa_cache)
+ panic("Couldn't create nmi save area cache");
+ }
}
void __init smp_prepare_boot_cpu(void)
diff --git a/arch/s390/kernel/syscalls.S b/arch/s390/kernel/syscalls.S
index 2659b5c..54fce7b 100644
--- a/arch/s390/kernel/syscalls.S
+++ b/arch/s390/kernel/syscalls.S
@@ -386,5 +386,5 @@
SYSCALL(sys_copy_file_range,compat_sys_copy_file_range) /* 375 */
SYSCALL(sys_preadv2,compat_sys_preadv2)
SYSCALL(sys_pwritev2,compat_sys_pwritev2)
-NI_SYSCALL
+SYSCALL(sys_s390_guarded_storage,compat_sys_s390_guarded_storage) /* 378 */
SYSCALL(sys_statx,compat_sys_statx)
diff --git a/arch/s390/kernel/sysinfo.c b/arch/s390/kernel/sysinfo.c
index 12b6b13..eefcb54 100644
--- a/arch/s390/kernel/sysinfo.c
+++ b/arch/s390/kernel/sysinfo.c
@@ -4,6 +4,7 @@
* Martin Schwidefsky <schwidefsky@de.ibm.com>,
*/
+#include <linux/debugfs.h>
#include <linux/kernel.h>
#include <linux/mm.h>
#include <linux/proc_fs.h>
@@ -13,6 +14,7 @@
#include <linux/export.h>
#include <linux/slab.h>
#include <asm/ebcdic.h>
+#include <asm/debug.h>
#include <asm/sysinfo.h>
#include <asm/cpcmd.h>
#include <asm/topology.h>
@@ -485,3 +487,99 @@
"%lu.%02lu BogoMIPS preset\n", loops_per_jiffy/(500000/HZ),
(loops_per_jiffy/(5000/HZ)) % 100);
}
+
+#ifdef CONFIG_DEBUG_FS
+
+#define STSI_FILE(fc, s1, s2) \
+static int stsi_open_##fc##_##s1##_##s2(struct inode *inode, struct file *file)\
+{ \
+ file->private_data = (void *) get_zeroed_page(GFP_KERNEL); \
+ if (!file->private_data) \
+ return -ENOMEM; \
+ if (stsi(file->private_data, fc, s1, s2)) { \
+ free_page((unsigned long)file->private_data); \
+ file->private_data = NULL; \
+ return -EACCES; \
+ } \
+ return nonseekable_open(inode, file); \
+} \
+ \
+static const struct file_operations stsi_##fc##_##s1##_##s2##_fs_ops = { \
+ .open = stsi_open_##fc##_##s1##_##s2, \
+ .release = stsi_release, \
+ .read = stsi_read, \
+ .llseek = no_llseek, \
+};
+
+static int stsi_release(struct inode *inode, struct file *file)
+{
+ free_page((unsigned long)file->private_data);
+ return 0;
+}
+
+static ssize_t stsi_read(struct file *file, char __user *buf, size_t size, loff_t *ppos)
+{
+ return simple_read_from_buffer(buf, size, ppos, file->private_data, PAGE_SIZE);
+}
+
+STSI_FILE( 1, 1, 1);
+STSI_FILE( 1, 2, 1);
+STSI_FILE( 1, 2, 2);
+STSI_FILE( 2, 2, 1);
+STSI_FILE( 2, 2, 2);
+STSI_FILE( 3, 2, 2);
+STSI_FILE(15, 1, 2);
+STSI_FILE(15, 1, 3);
+STSI_FILE(15, 1, 4);
+STSI_FILE(15, 1, 5);
+STSI_FILE(15, 1, 6);
+
+struct stsi_file {
+ const struct file_operations *fops;
+ char *name;
+};
+
+static struct stsi_file stsi_file[] __initdata = {
+ {.fops = &stsi_1_1_1_fs_ops, .name = "1_1_1"},
+ {.fops = &stsi_1_2_1_fs_ops, .name = "1_2_1"},
+ {.fops = &stsi_1_2_2_fs_ops, .name = "1_2_2"},
+ {.fops = &stsi_2_2_1_fs_ops, .name = "2_2_1"},
+ {.fops = &stsi_2_2_2_fs_ops, .name = "2_2_2"},
+ {.fops = &stsi_3_2_2_fs_ops, .name = "3_2_2"},
+ {.fops = &stsi_15_1_2_fs_ops, .name = "15_1_2"},
+ {.fops = &stsi_15_1_3_fs_ops, .name = "15_1_3"},
+ {.fops = &stsi_15_1_4_fs_ops, .name = "15_1_4"},
+ {.fops = &stsi_15_1_5_fs_ops, .name = "15_1_5"},
+ {.fops = &stsi_15_1_6_fs_ops, .name = "15_1_6"},
+};
+
+static u8 stsi_0_0_0;
+
+static __init int stsi_init_debugfs(void)
+{
+ struct dentry *stsi_root;
+ struct stsi_file *sf;
+ int lvl, i;
+
+ stsi_root = debugfs_create_dir("stsi", arch_debugfs_dir);
+ if (IS_ERR_OR_NULL(stsi_root))
+ return 0;
+ lvl = stsi(NULL, 0, 0, 0);
+ if (lvl > 0)
+ stsi_0_0_0 = lvl;
+ debugfs_create_u8("0_0_0", 0400, stsi_root, &stsi_0_0_0);
+ for (i = 0; i < ARRAY_SIZE(stsi_file); i++) {
+ sf = &stsi_file[i];
+ debugfs_create_file(sf->name, 0400, stsi_root, NULL, sf->fops);
+ }
+ if (IS_ENABLED(CONFIG_SCHED_TOPOLOGY) && MACHINE_HAS_TOPOLOGY) {
+ char link_to[10];
+
+ sprintf(link_to, "15_1_%d", topology_mnest_limit());
+ debugfs_create_symlink("topology", stsi_root, link_to);
+ }
+ return 0;
+}
+device_initcall(stsi_init_debugfs);
+
+#endif /* CONFIG_DEBUG_FS */
diff --git a/arch/s390/kernel/time.c b/arch/s390/kernel/time.c
index c31da46..c3a52f9 100644
--- a/arch/s390/kernel/time.c
+++ b/arch/s390/kernel/time.c
@@ -158,7 +158,9 @@
cd->mult = 16777;
cd->shift = 12;
cd->min_delta_ns = 1;
+ cd->min_delta_ticks = 1;
cd->max_delta_ns = LONG_MAX;
+ cd->max_delta_ticks = ULONG_MAX;
cd->rating = 400;
cd->cpumask = cpumask_of(cpu);
cd->set_next_event = s390_next_event;
diff --git a/arch/s390/kernel/topology.c b/arch/s390/kernel/topology.c
index 17660e8..bb47c92 100644
--- a/arch/s390/kernel/topology.c
+++ b/arch/s390/kernel/topology.c
@@ -83,6 +83,8 @@
return mask;
}
+#define TOPOLOGY_CORE_BITS 64
+
static void add_cpus_to_mask(struct topology_core *tl_core,
struct mask_info *drawer,
struct mask_info *book,
@@ -91,7 +93,7 @@
struct cpu_topology_s390 *topo;
unsigned int core;
- for_each_set_bit(core, &tl_core->mask[0], TOPOLOGY_CORE_BITS) {
+ for_each_set_bit(core, &tl_core->mask, TOPOLOGY_CORE_BITS) {
unsigned int rcore;
int lcpu, i;
@@ -244,7 +246,7 @@
void store_topology(struct sysinfo_15_1_x *info)
{
- stsi(info, 15, 1, min(topology_max_mnest, 4));
+ stsi(info, 15, 1, topology_mnest_limit());
}
static int __arch_update_cpu_topology(void)
diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
index 0f8f141..169558d 100644
--- a/arch/s390/kvm/interrupt.c
+++ b/arch/s390/kvm/interrupt.c
@@ -420,8 +420,8 @@
save_access_regs(vcpu->run->s.regs.acrs);
/* Extended save area */
- rc = read_guest_lc(vcpu, __LC_VX_SAVE_AREA_ADDR, &ext_sa_addr,
- sizeof(unsigned long));
+ rc = read_guest_lc(vcpu, __LC_MCESAD, &ext_sa_addr,
+ sizeof(unsigned long));
/* Only bits 0-53 are used for address formation */
ext_sa_addr &= ~0x3ffUL;
if (!rc && mci.vr && ext_sa_addr && test_kvm_facility(vcpu->kvm, 129)) {
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index fd6cd05b..d5c5c91 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -273,7 +273,7 @@
kvm_s390_available_subfunc.pcc);
}
if (test_facility(57)) /* MSA5 */
- __cpacf_query(CPACF_PPNO, (cpacf_mask_t *)
+ __cpacf_query(CPACF_PRNO, (cpacf_mask_t *)
kvm_s390_available_subfunc.ppno);
if (MACHINE_HAS_ESOP)
@@ -1512,9 +1512,9 @@
kvm->arch.mem_limit = KVM_S390_NO_MEM_LIMIT;
} else {
if (sclp.hamax == U64_MAX)
- kvm->arch.mem_limit = TASK_MAX_SIZE;
+ kvm->arch.mem_limit = TASK_SIZE_MAX;
else
- kvm->arch.mem_limit = min_t(unsigned long, TASK_MAX_SIZE,
+ kvm->arch.mem_limit = min_t(unsigned long, TASK_SIZE_MAX,
sclp.hamax + 1);
kvm->arch.gmap = gmap_create(current->mm, kvm->arch.mem_limit - 1);
if (!kvm->arch.gmap)
diff --git a/arch/s390/lib/spinlock.c b/arch/s390/lib/spinlock.c
index ba427eb..ffb15bd 100644
--- a/arch/s390/lib/spinlock.c
+++ b/arch/s390/lib/spinlock.c
@@ -17,7 +17,7 @@
static int __init spin_retry_init(void)
{
if (spin_retry < 0)
- spin_retry = MACHINE_HAS_CAD ? 10 : 1000;
+ spin_retry = 1000;
return 0;
}
early_initcall(spin_retry_init);
@@ -32,23 +32,17 @@
}
__setup("spin_retry=", spin_retry_setup);
-static inline void _raw_compare_and_delay(unsigned int *lock, unsigned int old)
-{
- asm(".insn rsy,0xeb0000000022,%0,0,%1" : : "d" (old), "Q" (*lock));
-}
-
void arch_spin_lock_wait(arch_spinlock_t *lp)
{
- unsigned int cpu = SPINLOCK_LOCKVAL;
- unsigned int owner;
- int count, first_diag;
+ int cpu = SPINLOCK_LOCKVAL;
+ int owner, count, first_diag;
first_diag = 1;
while (1) {
owner = ACCESS_ONCE(lp->lock);
/* Try to get the lock if it is free. */
if (!owner) {
- if (_raw_compare_and_swap(&lp->lock, 0, cpu))
+ if (__atomic_cmpxchg_bool(&lp->lock, 0, cpu))
return;
continue;
}
@@ -61,8 +55,6 @@
/* Loop for a while on the lock value. */
count = spin_retry;
do {
- if (MACHINE_HAS_CAD)
- _raw_compare_and_delay(&lp->lock, owner);
owner = ACCESS_ONCE(lp->lock);
} while (owner && count-- > 0);
if (!owner)
@@ -82,9 +74,8 @@
void arch_spin_lock_wait_flags(arch_spinlock_t *lp, unsigned long flags)
{
- unsigned int cpu = SPINLOCK_LOCKVAL;
- unsigned int owner;
- int count, first_diag;
+ int cpu = SPINLOCK_LOCKVAL;
+ int owner, count, first_diag;
local_irq_restore(flags);
first_diag = 1;
@@ -93,7 +84,7 @@
/* Try to get the lock if it is free. */
if (!owner) {
local_irq_disable();
- if (_raw_compare_and_swap(&lp->lock, 0, cpu))
+ if (__atomic_cmpxchg_bool(&lp->lock, 0, cpu))
return;
local_irq_restore(flags);
continue;
@@ -107,8 +98,6 @@
/* Loop for a while on the lock value. */
count = spin_retry;
do {
- if (MACHINE_HAS_CAD)
- _raw_compare_and_delay(&lp->lock, owner);
owner = ACCESS_ONCE(lp->lock);
} while (owner && count-- > 0);
if (!owner)
@@ -128,18 +117,16 @@
int arch_spin_trylock_retry(arch_spinlock_t *lp)
{
- unsigned int cpu = SPINLOCK_LOCKVAL;
- unsigned int owner;
- int count;
+ int cpu = SPINLOCK_LOCKVAL;
+ int owner, count;
for (count = spin_retry; count > 0; count--) {
owner = READ_ONCE(lp->lock);
/* Try to get the lock if it is free. */
if (!owner) {
- if (_raw_compare_and_swap(&lp->lock, 0, cpu))
+ if (__atomic_cmpxchg_bool(&lp->lock, 0, cpu))
return 1;
- } else if (MACHINE_HAS_CAD)
- _raw_compare_and_delay(&lp->lock, owner);
+ }
}
return 0;
}
@@ -147,8 +134,8 @@
void _raw_read_lock_wait(arch_rwlock_t *rw)
{
- unsigned int owner, old;
int count = spin_retry;
+ int owner, old;
#ifdef CONFIG_HAVE_MARCH_Z196_FEATURES
__RAW_LOCK(&rw->lock, -1, __RAW_OP_ADD);
@@ -162,12 +149,9 @@
}
old = ACCESS_ONCE(rw->lock);
owner = ACCESS_ONCE(rw->owner);
- if ((int) old < 0) {
- if (MACHINE_HAS_CAD)
- _raw_compare_and_delay(&rw->lock, old);
+ if (old < 0)
continue;
- }
- if (_raw_compare_and_swap(&rw->lock, old, old + 1))
+ if (__atomic_cmpxchg_bool(&rw->lock, old, old + 1))
return;
}
}
@@ -175,17 +159,14 @@
int _raw_read_trylock_retry(arch_rwlock_t *rw)
{
- unsigned int old;
int count = spin_retry;
+ int old;
while (count-- > 0) {
old = ACCESS_ONCE(rw->lock);
- if ((int) old < 0) {
- if (MACHINE_HAS_CAD)
- _raw_compare_and_delay(&rw->lock, old);
+ if (old < 0)
continue;
- }
- if (_raw_compare_and_swap(&rw->lock, old, old + 1))
+ if (__atomic_cmpxchg_bool(&rw->lock, old, old + 1))
return 1;
}
return 0;
@@ -194,10 +175,10 @@
#ifdef CONFIG_HAVE_MARCH_Z196_FEATURES
-void _raw_write_lock_wait(arch_rwlock_t *rw, unsigned int prev)
+void _raw_write_lock_wait(arch_rwlock_t *rw, int prev)
{
- unsigned int owner, old;
int count = spin_retry;
+ int owner, old;
owner = 0;
while (1) {
@@ -209,14 +190,12 @@
old = ACCESS_ONCE(rw->lock);
owner = ACCESS_ONCE(rw->owner);
smp_mb();
- if ((int) old >= 0) {
+ if (old >= 0) {
prev = __RAW_LOCK(&rw->lock, 0x80000000, __RAW_OP_OR);
old = prev;
}
- if ((old & 0x7fffffff) == 0 && (int) prev >= 0)
+ if ((old & 0x7fffffff) == 0 && prev >= 0)
break;
- if (MACHINE_HAS_CAD)
- _raw_compare_and_delay(&rw->lock, old);
}
}
EXPORT_SYMBOL(_raw_write_lock_wait);
@@ -225,8 +204,8 @@
void _raw_write_lock_wait(arch_rwlock_t *rw)
{
- unsigned int owner, old, prev;
int count = spin_retry;
+ int owner, old, prev;
prev = 0x80000000;
owner = 0;
@@ -238,15 +217,13 @@
}
old = ACCESS_ONCE(rw->lock);
owner = ACCESS_ONCE(rw->owner);
- if ((int) old >= 0 &&
- _raw_compare_and_swap(&rw->lock, old, old | 0x80000000))
+ if (old >= 0 &&
+ __atomic_cmpxchg_bool(&rw->lock, old, old | 0x80000000))
prev = old;
else
smp_mb();
- if ((old & 0x7fffffff) == 0 && (int) prev >= 0)
+ if ((old & 0x7fffffff) == 0 && prev >= 0)
break;
- if (MACHINE_HAS_CAD)
- _raw_compare_and_delay(&rw->lock, old);
}
}
EXPORT_SYMBOL(_raw_write_lock_wait);
@@ -255,24 +232,21 @@
int _raw_write_trylock_retry(arch_rwlock_t *rw)
{
- unsigned int old;
int count = spin_retry;
+ int old;
while (count-- > 0) {
old = ACCESS_ONCE(rw->lock);
- if (old) {
- if (MACHINE_HAS_CAD)
- _raw_compare_and_delay(&rw->lock, old);
+ if (old)
continue;
- }
- if (_raw_compare_and_swap(&rw->lock, 0, 0x80000000))
+ if (__atomic_cmpxchg_bool(&rw->lock, 0, 0x80000000))
return 1;
}
return 0;
}
EXPORT_SYMBOL(_raw_write_trylock_retry);
-void arch_lock_relax(unsigned int cpu)
+void arch_lock_relax(int cpu)
{
if (!cpu)
return;
diff --git a/arch/s390/lib/uaccess.c b/arch/s390/lib/uaccess.c
index f481fcd..1e5bb2b 100644
--- a/arch/s390/lib/uaccess.c
+++ b/arch/s390/lib/uaccess.c
@@ -26,7 +26,7 @@
tmp1 = -4096UL;
asm volatile(
"0: .insn ss,0xc80000000000,0(%0,%2),0(%1),0\n"
- "9: jz 7f\n"
+ "6: jz 4f\n"
"1: algr %0,%3\n"
" slgr %1,%3\n"
" slgr %2,%3\n"
@@ -35,23 +35,13 @@
" nr %4,%3\n" /* %4 = (ptr + 4095) & -4096 */
" slgr %4,%1\n"
" clgr %0,%4\n" /* copy crosses next page boundary? */
- " jnh 4f\n"
+ " jnh 5f\n"
"3: .insn ss,0xc80000000000,0(%4,%2),0(%1),0\n"
- "10:slgr %0,%4\n"
- " algr %2,%4\n"
- "4: lghi %4,-1\n"
- " algr %4,%0\n" /* copy remaining size, subtract 1 */
- " bras %3,6f\n" /* memset loop */
- " xc 0(1,%2),0(%2)\n"
- "5: xc 0(256,%2),0(%2)\n"
- " la %2,256(%2)\n"
- "6: aghi %4,-256\n"
- " jnm 5b\n"
- " ex %4,0(%3)\n"
- " j 8f\n"
- "7: slgr %0,%0\n"
- "8:\n"
- EX_TABLE(0b,2b) EX_TABLE(3b,4b) EX_TABLE(9b,2b) EX_TABLE(10b,4b)
+ "7: slgr %0,%4\n"
+ " j 5f\n"
+ "4: slgr %0,%0\n"
+ "5:\n"
+ EX_TABLE(0b,2b) EX_TABLE(3b,5b) EX_TABLE(6b,2b) EX_TABLE(7b,5b)
: "+a" (size), "+a" (ptr), "+a" (x), "+a" (tmp1), "=a" (tmp2)
: "d" (reg0) : "cc", "memory");
return size;
@@ -67,49 +57,38 @@
asm volatile(
" sacf 0\n"
"0: mvcp 0(%0,%2),0(%1),%3\n"
- "10:jz 8f\n"
+ "7: jz 5f\n"
"1: algr %0,%3\n"
" la %1,256(%1)\n"
" la %2,256(%2)\n"
"2: mvcp 0(%0,%2),0(%1),%3\n"
- "11:jnz 1b\n"
- " j 8f\n"
+ "8: jnz 1b\n"
+ " j 5f\n"
"3: la %4,255(%1)\n" /* %4 = ptr + 255 */
" lghi %3,-4096\n"
" nr %4,%3\n" /* %4 = (ptr + 255) & -4096 */
" slgr %4,%1\n"
" clgr %0,%4\n" /* copy crosses next page boundary? */
- " jnh 5f\n"
+ " jnh 6f\n"
"4: mvcp 0(%4,%2),0(%1),%3\n"
- "12:slgr %0,%4\n"
- " algr %2,%4\n"
- "5: lghi %4,-1\n"
- " algr %4,%0\n" /* copy remaining size, subtract 1 */
- " bras %3,7f\n" /* memset loop */
- " xc 0(1,%2),0(%2)\n"
- "6: xc 0(256,%2),0(%2)\n"
- " la %2,256(%2)\n"
- "7: aghi %4,-256\n"
- " jnm 6b\n"
- " ex %4,0(%3)\n"
- " j 9f\n"
- "8: slgr %0,%0\n"
- "9: sacf 768\n"
- EX_TABLE(0b,3b) EX_TABLE(2b,3b) EX_TABLE(4b,5b)
- EX_TABLE(10b,3b) EX_TABLE(11b,3b) EX_TABLE(12b,5b)
+ "9: slgr %0,%4\n"
+ " j 6f\n"
+ "5: slgr %0,%0\n"
+ "6: sacf 768\n"
+ EX_TABLE(0b,3b) EX_TABLE(2b,3b) EX_TABLE(4b,6b)
+ EX_TABLE(7b,3b) EX_TABLE(8b,3b) EX_TABLE(9b,6b)
: "+a" (size), "+a" (ptr), "+a" (x), "+a" (tmp1), "=a" (tmp2)
: : "cc", "memory");
return size;
}
-unsigned long __copy_from_user(void *to, const void __user *from, unsigned long n)
+unsigned long raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
- check_object_size(to, n, false);
if (static_branch_likely(&have_mvcos))
return copy_from_user_mvcos(to, from, n);
return copy_from_user_mvcp(to, from, n);
}
-EXPORT_SYMBOL(__copy_from_user);
+EXPORT_SYMBOL(raw_copy_from_user);
static inline unsigned long copy_to_user_mvcos(void __user *ptr, const void *x,
unsigned long size)
@@ -176,14 +155,13 @@
return size;
}
-unsigned long __copy_to_user(void __user *to, const void *from, unsigned long n)
+unsigned long raw_copy_to_user(void __user *to, const void *from, unsigned long n)
{
- check_object_size(from, n, true);
if (static_branch_likely(&have_mvcos))
return copy_to_user_mvcos(to, from, n);
return copy_to_user_mvcs(to, from, n);
}
-EXPORT_SYMBOL(__copy_to_user);
+EXPORT_SYMBOL(raw_copy_to_user);
static inline unsigned long copy_in_user_mvcos(void __user *to, const void __user *from,
unsigned long size)
@@ -240,13 +218,13 @@
return size;
}
-unsigned long __copy_in_user(void __user *to, const void __user *from, unsigned long n)
+unsigned long raw_copy_in_user(void __user *to, const void __user *from, unsigned long n)
{
if (static_branch_likely(&have_mvcos))
return copy_in_user_mvcos(to, from, n);
return copy_in_user_mvc(to, from, n);
}
-EXPORT_SYMBOL(__copy_in_user);
+EXPORT_SYMBOL(raw_copy_in_user);
static inline unsigned long clear_user_mvcos(void __user *to, unsigned long size)
{
diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c
index a07b1ec..7f6db1e 100644
--- a/arch/s390/mm/gmap.c
+++ b/arch/s390/mm/gmap.c
@@ -431,7 +431,7 @@
if ((from | to | len) & (PMD_SIZE - 1))
return -EINVAL;
if (len == 0 || from + len < from || to + len < to ||
- from + len - 1 > TASK_MAX_SIZE || to + len - 1 > gmap->asce_end)
+ from + len - 1 > TASK_SIZE_MAX || to + len - 1 > gmap->asce_end)
return -EINVAL;
flush = 0;
@@ -2004,20 +2004,12 @@
* Called with sg->parent->shadow_lock.
*/
static void gmap_shadow_notify(struct gmap *sg, unsigned long vmaddr,
- unsigned long offset, pte_t *pte)
+ unsigned long gaddr, pte_t *pte)
{
struct gmap_rmap *rmap, *rnext, *head;
- unsigned long gaddr, start, end, bits, raddr;
- unsigned long *table;
+ unsigned long start, end, bits, raddr;
BUG_ON(!gmap_is_shadow(sg));
- spin_lock(&sg->parent->guest_table_lock);
- table = radix_tree_lookup(&sg->parent->host_to_guest,
- vmaddr >> PMD_SHIFT);
- gaddr = table ? __gmap_segment_gaddr(table) + offset : 0;
- spin_unlock(&sg->parent->guest_table_lock);
- if (!table)
- return;
spin_lock(&sg->guest_table_lock);
if (sg->removed) {
@@ -2076,7 +2068,7 @@
void ptep_notify(struct mm_struct *mm, unsigned long vmaddr,
pte_t *pte, unsigned long bits)
{
- unsigned long offset, gaddr;
+ unsigned long offset, gaddr = 0;
unsigned long *table;
struct gmap *gmap, *sg, *next;
@@ -2084,22 +2076,23 @@
offset = offset * (4096 / sizeof(pte_t));
rcu_read_lock();
list_for_each_entry_rcu(gmap, &mm->context.gmap_list, list) {
- if (!list_empty(&gmap->children) && (bits & PGSTE_VSIE_BIT)) {
- spin_lock(&gmap->shadow_lock);
- list_for_each_entry_safe(sg, next,
- &gmap->children, list)
- gmap_shadow_notify(sg, vmaddr, offset, pte);
- spin_unlock(&gmap->shadow_lock);
- }
- if (!(bits & PGSTE_IN_BIT))
- continue;
spin_lock(&gmap->guest_table_lock);
table = radix_tree_lookup(&gmap->host_to_guest,
vmaddr >> PMD_SHIFT);
if (table)
gaddr = __gmap_segment_gaddr(table) + offset;
spin_unlock(&gmap->guest_table_lock);
- if (table)
+ if (!table)
+ continue;
+
+ if (!list_empty(&gmap->children) && (bits & PGSTE_VSIE_BIT)) {
+ spin_lock(&gmap->shadow_lock);
+ list_for_each_entry_safe(sg, next,
+ &gmap->children, list)
+ gmap_shadow_notify(sg, vmaddr, gaddr, pte);
+ spin_unlock(&gmap->shadow_lock);
+ }
+ if (bits & PGSTE_IN_BIT)
gmap_call_notifier(gmap, gaddr, gaddr + PAGE_SIZE - 1);
}
rcu_read_unlock();
diff --git a/arch/s390/mm/gup.c b/arch/s390/mm/gup.c
index 18d4107e..b7b779c 100644
--- a/arch/s390/mm/gup.c
+++ b/arch/s390/mm/gup.c
@@ -211,7 +211,7 @@
addr = start;
len = (unsigned long) nr_pages << PAGE_SHIFT;
end = start + len;
- if ((end <= start) || (end > TASK_SIZE))
+ if ((end <= start) || (end > mm->context.asce_limit))
return 0;
/*
* local_irq_save() doesn't prevent pagetable teardown, but does
diff --git a/arch/s390/mm/mmap.c b/arch/s390/mm/mmap.c
index 5061861..b017dae 100644
--- a/arch/s390/mm/mmap.c
+++ b/arch/s390/mm/mmap.c
@@ -89,19 +89,20 @@
struct mm_struct *mm = current->mm;
struct vm_area_struct *vma;
struct vm_unmapped_area_info info;
+ int rc;
if (len > TASK_SIZE - mmap_min_addr)
return -ENOMEM;
if (flags & MAP_FIXED)
- return addr;
+ goto check_asce_limit;
if (addr) {
addr = PAGE_ALIGN(addr);
vma = find_vma(mm, addr);
if (TASK_SIZE - len >= addr && addr >= mmap_min_addr &&
(!vma || addr + len <= vma->vm_start))
- return addr;
+ goto check_asce_limit;
}
info.flags = 0;
@@ -113,7 +114,18 @@
else
info.align_mask = 0;
info.align_offset = pgoff << PAGE_SHIFT;
- return vm_unmapped_area(&info);
+ addr = vm_unmapped_area(&info);
+ if (addr & ~PAGE_MASK)
+ return addr;
+
+check_asce_limit:
+ if (addr + len > current->mm->context.asce_limit) {
+ rc = crst_table_upgrade(mm);
+ if (rc)
+ return (unsigned long) rc;
+ }
+
+ return addr;
}
unsigned long
@@ -125,13 +137,14 @@
struct mm_struct *mm = current->mm;
unsigned long addr = addr0;
struct vm_unmapped_area_info info;
+ int rc;
/* requested length too big for entire address space */
if (len > TASK_SIZE - mmap_min_addr)
return -ENOMEM;
if (flags & MAP_FIXED)
- return addr;
+ goto check_asce_limit;
/* requesting a specific address */
if (addr) {
@@ -139,7 +152,7 @@
vma = find_vma(mm, addr);
if (TASK_SIZE - len >= addr && addr >= mmap_min_addr &&
(!vma || addr + len <= vma->vm_start))
- return addr;
+ goto check_asce_limit;
}
info.flags = VM_UNMAPPED_AREA_TOPDOWN;
@@ -165,65 +178,20 @@
info.low_limit = TASK_UNMAPPED_BASE;
info.high_limit = TASK_SIZE;
addr = vm_unmapped_area(&info);
+ if (addr & ~PAGE_MASK)
+ return addr;
+ }
+
+check_asce_limit:
+ if (addr + len > current->mm->context.asce_limit) {
+ rc = crst_table_upgrade(mm);
+ if (rc)
+ return (unsigned long) rc;
}
return addr;
}
-int s390_mmap_check(unsigned long addr, unsigned long len, unsigned long flags)
-{
- if (is_compat_task() || TASK_SIZE >= TASK_MAX_SIZE)
- return 0;
- if (!(flags & MAP_FIXED))
- addr = 0;
- if ((addr + len) >= TASK_SIZE)
- return crst_table_upgrade(current->mm);
- return 0;
-}
-
-static unsigned long
-s390_get_unmapped_area(struct file *filp, unsigned long addr,
- unsigned long len, unsigned long pgoff, unsigned long flags)
-{
- struct mm_struct *mm = current->mm;
- unsigned long area;
- int rc;
-
- area = arch_get_unmapped_area(filp, addr, len, pgoff, flags);
- if (!(area & ~PAGE_MASK))
- return area;
- if (area == -ENOMEM && !is_compat_task() && TASK_SIZE < TASK_MAX_SIZE) {
- /* Upgrade the page table to 4 levels and retry. */
- rc = crst_table_upgrade(mm);
- if (rc)
- return (unsigned long) rc;
- area = arch_get_unmapped_area(filp, addr, len, pgoff, flags);
- }
- return area;
-}
-
-static unsigned long
-s390_get_unmapped_area_topdown(struct file *filp, const unsigned long addr,
- const unsigned long len, const unsigned long pgoff,
- const unsigned long flags)
-{
- struct mm_struct *mm = current->mm;
- unsigned long area;
- int rc;
-
- area = arch_get_unmapped_area_topdown(filp, addr, len, pgoff, flags);
- if (!(area & ~PAGE_MASK))
- return area;
- if (area == -ENOMEM && !is_compat_task() && TASK_SIZE < TASK_MAX_SIZE) {
- /* Upgrade the page table to 4 levels and retry. */
- rc = crst_table_upgrade(mm);
- if (rc)
- return (unsigned long) rc;
- area = arch_get_unmapped_area_topdown(filp, addr, len,
- pgoff, flags);
- }
- return area;
-}
/*
* This function, called very early during the creation of a new
* process VM image, sets up which VM layout function to use:
@@ -241,9 +209,9 @@
*/
if (mmap_is_legacy()) {
mm->mmap_base = mmap_base_legacy(random_factor);
- mm->get_unmapped_area = s390_get_unmapped_area;
+ mm->get_unmapped_area = arch_get_unmapped_area;
} else {
mm->mmap_base = mmap_base(random_factor);
- mm->get_unmapped_area = s390_get_unmapped_area_topdown;
+ mm->get_unmapped_area = arch_get_unmapped_area_topdown;
}
}
diff --git a/arch/s390/mm/page-states.c b/arch/s390/mm/page-states.c
index 3330ea1..69a7b01 100644
--- a/arch/s390/mm/page-states.c
+++ b/arch/s390/mm/page-states.c
@@ -13,8 +13,7 @@
#include <linux/gfp.h>
#include <linux/init.h>
-#define ESSA_SET_STABLE 1
-#define ESSA_SET_UNUSED 2
+#include <asm/page-states.h>
static int cmma_flag = 1;
diff --git a/arch/s390/mm/pageattr.c b/arch/s390/mm/pageattr.c
index fc5dc33..fc321c5 100644
--- a/arch/s390/mm/pageattr.c
+++ b/arch/s390/mm/pageattr.c
@@ -94,7 +94,7 @@
new = pte_wrprotect(new);
else if (flags & SET_MEMORY_RW)
new = pte_mkwrite(pte_mkdirty(new));
- if ((flags & SET_MEMORY_NX) && MACHINE_HAS_NX)
+ if (flags & SET_MEMORY_NX)
pte_val(new) |= _PAGE_NOEXEC;
else if (flags & SET_MEMORY_X)
pte_val(new) &= ~_PAGE_NOEXEC;
@@ -144,7 +144,7 @@
new = pmd_wrprotect(new);
else if (flags & SET_MEMORY_RW)
new = pmd_mkwrite(pmd_mkdirty(new));
- if ((flags & SET_MEMORY_NX) && MACHINE_HAS_NX)
+ if (flags & SET_MEMORY_NX)
pmd_val(new) |= _SEGMENT_ENTRY_NOEXEC;
else if (flags & SET_MEMORY_X)
pmd_val(new) &= ~_SEGMENT_ENTRY_NOEXEC;
@@ -221,7 +221,7 @@
new = pud_wrprotect(new);
else if (flags & SET_MEMORY_RW)
new = pud_mkwrite(pud_mkdirty(new));
- if ((flags & SET_MEMORY_NX) && MACHINE_HAS_NX)
+ if (flags & SET_MEMORY_NX)
pud_val(new) |= _REGION_ENTRY_NOEXEC;
else if (flags & SET_MEMORY_X)
pud_val(new) &= ~_REGION_ENTRY_NOEXEC;
@@ -288,6 +288,10 @@
int __set_memory(unsigned long addr, int numpages, unsigned long flags)
{
+ if (!MACHINE_HAS_NX)
+ flags &= ~(SET_MEMORY_NX | SET_MEMORY_X);
+ if (!flags)
+ return 0;
addr &= PAGE_MASK;
return change_page_attr(addr, addr + numpages * PAGE_SIZE, flags);
}
diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c
index 995f785..f502cbe 100644
--- a/arch/s390/mm/pgalloc.c
+++ b/arch/s390/mm/pgalloc.c
@@ -95,7 +95,6 @@
mm->context.asce_limit = 1UL << 53;
mm->context.asce = __pa(mm->pgd) | _ASCE_TABLE_LENGTH |
_ASCE_USER_BITS | _ASCE_TYPE_REGION2;
- mm->task_size = mm->context.asce_limit;
spin_unlock_bh(&mm->page_table_lock);
on_each_cpu(__crst_table_upgrade, mm, 0);
@@ -119,7 +118,6 @@
mm->context.asce_limit = 1UL << 31;
mm->context.asce = __pa(mm->pgd) | _ASCE_TABLE_LENGTH |
_ASCE_USER_BITS | _ASCE_TYPE_SEGMENT;
- mm->task_size = mm->context.asce_limit;
crst_table_free(mm, (unsigned long *) pgd);
if (current->active_mm == mm)
@@ -144,7 +142,7 @@
struct page *page;
unsigned long *table;
- page = alloc_page(GFP_KERNEL|__GFP_REPEAT);
+ page = alloc_page(GFP_KERNEL);
if (page) {
table = (unsigned long *) page_to_phys(page);
clear_table(table, _PAGE_INVALID, PAGE_SIZE/2);
diff --git a/arch/s390/mm/pgtable.c b/arch/s390/mm/pgtable.c
index 463e5ef..947b66a 100644
--- a/arch/s390/mm/pgtable.c
+++ b/arch/s390/mm/pgtable.c
@@ -23,6 +23,7 @@
#include <asm/tlb.h>
#include <asm/tlbflush.h>
#include <asm/mmu_context.h>
+#include <asm/page-states.h>
static inline pte_t ptep_flush_direct(struct mm_struct *mm,
unsigned long addr, pte_t *ptep)
@@ -787,4 +788,156 @@
return 0;
}
EXPORT_SYMBOL(get_guest_storage_key);
+
+/**
+ * pgste_perform_essa - perform ESSA actions on the PGSTE.
+ * @mm: the memory context. It must have PGSTEs, no check is performed here!
+ * @hva: the host virtual address of the page whose PGSTE is to be processed
+ * @orc: the specific action to perform, see the ESSA_SET_* macros.
+ * @oldpte: the PTE will be saved there if the pointer is not NULL.
+ * @oldpgste: the old PGSTE will be saved there if the pointer is not NULL.
+ *
+ * Return: 1 if the page is to be added to the CBRL, otherwise 0,
+ * or < 0 in case of error. -EINVAL is returned for invalid values
+ * of orc, -EFAULT for invalid addresses.
+ */
+int pgste_perform_essa(struct mm_struct *mm, unsigned long hva, int orc,
+ unsigned long *oldpte, unsigned long *oldpgste)
+{
+ unsigned long pgstev;
+ spinlock_t *ptl;
+ pgste_t pgste;
+ pte_t *ptep;
+ int res = 0;
+
+ WARN_ON_ONCE(orc > ESSA_MAX);
+ if (unlikely(orc > ESSA_MAX))
+ return -EINVAL;
+ ptep = get_locked_pte(mm, hva, &ptl);
+ if (unlikely(!ptep))
+ return -EFAULT;
+ pgste = pgste_get_lock(ptep);
+ pgstev = pgste_val(pgste);
+ if (oldpte)
+ *oldpte = pte_val(*ptep);
+ if (oldpgste)
+ *oldpgste = pgstev;
+
+ switch (orc) {
+ case ESSA_GET_STATE:
+ break;
+ case ESSA_SET_STABLE:
+ pgstev &= ~_PGSTE_GPS_USAGE_MASK;
+ pgstev |= _PGSTE_GPS_USAGE_STABLE;
+ break;
+ case ESSA_SET_UNUSED:
+ pgstev &= ~_PGSTE_GPS_USAGE_MASK;
+ pgstev |= _PGSTE_GPS_USAGE_UNUSED;
+ if (pte_val(*ptep) & _PAGE_INVALID)
+ res = 1;
+ break;
+ case ESSA_SET_VOLATILE:
+ pgstev &= ~_PGSTE_GPS_USAGE_MASK;
+ pgstev |= _PGSTE_GPS_USAGE_VOLATILE;
+ if (pte_val(*ptep) & _PAGE_INVALID)
+ res = 1;
+ break;
+ case ESSA_SET_POT_VOLATILE:
+ pgstev &= ~_PGSTE_GPS_USAGE_MASK;
+ if (!(pte_val(*ptep) & _PAGE_INVALID)) {
+ pgstev |= _PGSTE_GPS_USAGE_POT_VOLATILE;
+ break;
+ }
+ if (pgstev & _PGSTE_GPS_ZERO) {
+ pgstev |= _PGSTE_GPS_USAGE_VOLATILE;
+ break;
+ }
+ if (!(pgstev & PGSTE_GC_BIT)) {
+ pgstev |= _PGSTE_GPS_USAGE_VOLATILE;
+ res = 1;
+ break;
+ }
+ break;
+ case ESSA_SET_STABLE_RESIDENT:
+ pgstev &= ~_PGSTE_GPS_USAGE_MASK;
+ pgstev |= _PGSTE_GPS_USAGE_STABLE;
+ /*
+ * Since the resident state can go away any time after this
+ * call, we will not make this page resident. We can revisit
+ * this decision if a guest will ever start using this.
+ */
+ break;
+ case ESSA_SET_STABLE_IF_RESIDENT:
+ if (!(pte_val(*ptep) & _PAGE_INVALID)) {
+ pgstev &= ~_PGSTE_GPS_USAGE_MASK;
+ pgstev |= _PGSTE_GPS_USAGE_STABLE;
+ }
+ break;
+ default:
+ /* we should never get here! */
+ break;
+ }
+ /* If we are discarding a page, set it to logical zero */
+ if (res)
+ pgstev |= _PGSTE_GPS_ZERO;
+
+ pgste_val(pgste) = pgstev;
+ pgste_set_unlock(ptep, pgste);
+ pte_unmap_unlock(ptep, ptl);
+ return res;
+}
+EXPORT_SYMBOL(pgste_perform_essa);
+
+/**
+ * set_pgste_bits - set specific PGSTE bits.
+ * @mm: the memory context. It must have PGSTEs, no check is performed here!
+ * @hva: the host virtual address of the page whose PGSTE is to be processed
+ * @bits: a bitmask representing the bits that will be touched
+ * @value: the values of the bits to be written. Only the bits in the mask
+ * will be written.
+ *
+ * Return: 0 on success, < 0 in case of error.
+ */
+int set_pgste_bits(struct mm_struct *mm, unsigned long hva,
+ unsigned long bits, unsigned long value)
+{
+ spinlock_t *ptl;
+ pgste_t new;
+ pte_t *ptep;
+
+ ptep = get_locked_pte(mm, hva, &ptl);
+ if (unlikely(!ptep))
+ return -EFAULT;
+ new = pgste_get_lock(ptep);
+
+ pgste_val(new) &= ~bits;
+ pgste_val(new) |= value & bits;
+
+ pgste_set_unlock(ptep, new);
+ pte_unmap_unlock(ptep, ptl);
+ return 0;
+}
+EXPORT_SYMBOL(set_pgste_bits);
+
+/**
+ * get_pgste - get the current PGSTE for the given address.
+ * @mm: the memory context. It must have PGSTEs, no check is performed here!
+ * @hva: the host virtual address of the page whose PGSTE is to be processed
+ * @pgstep: will be written with the current PGSTE for the given address.
+ *
+ * Return: 0 on success, < 0 in case of error.
+ */
+int get_pgste(struct mm_struct *mm, unsigned long hva, unsigned long *pgstep)
+{
+ spinlock_t *ptl;
+ pte_t *ptep;
+
+ ptep = get_locked_pte(mm, hva, &ptl);
+ if (unlikely(!ptep))
+ return -EFAULT;
+ *pgstep = pgste_val(pgste_get(ptep));
+ pte_unmap_unlock(ptep, ptl);
+ return 0;
+}
+EXPORT_SYMBOL(get_pgste);
#endif
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index 364b9d8..8051df1 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -60,16 +60,8 @@
static struct airq_iv *zpci_aisb_iv;
static struct airq_iv *zpci_aibv[ZPCI_NR_DEVICES];
-/* Adapter interrupt definitions */
-static void zpci_irq_handler(struct airq_struct *airq);
-
-static struct airq_struct zpci_airq = {
- .handler = zpci_irq_handler,
- .isc = PCI_ISC,
-};
-
#define ZPCI_IOMAP_ENTRIES \
- min(((unsigned long) CONFIG_PCI_NR_FUNCTIONS * PCI_BAR_COUNT), \
+ min(((unsigned long) ZPCI_NR_DEVICES * PCI_BAR_COUNT / 2), \
ZPCI_IOMAP_MAX_ENTRIES)
static DEFINE_SPINLOCK(zpci_iomap_lock);
@@ -214,8 +206,6 @@
return rc;
}
-#define ZPCI_PCIAS_CFGSPC 15
-
static int zpci_cfg_load(struct zpci_dev *zdev, int offset, u32 *val, u8 len)
{
u64 req = ZPCI_CREATE_REQ(zdev->fh, ZPCI_PCIAS_CFGSPC, len);
@@ -507,6 +497,11 @@
}
}
+static struct airq_struct zpci_airq = {
+ .handler = zpci_irq_handler,
+ .isc = PCI_ISC,
+};
+
static int __init zpci_irq_init(void)
{
int rc;
@@ -871,11 +866,6 @@
}
EXPORT_SYMBOL(zpci_report_error);
-static inline int barsize(u8 size)
-{
- return (size) ? (1 << size) >> 10 : 0;
-}
-
static int zpci_mem_init(void)
{
BUILD_BUG_ON(!is_power_of_2(__alignof__(struct zpci_fmb)) ||
diff --git a/arch/score/include/asm/Kbuild b/arch/score/include/asm/Kbuild
index 926943a..e3a8d0f 100644
--- a/arch/score/include/asm/Kbuild
+++ b/arch/score/include/asm/Kbuild
@@ -4,6 +4,7 @@
generic-y += barrier.h
generic-y += clkdev.h
generic-y += current.h
+generic-y += extable.h
generic-y += irq_work.h
generic-y += mcs_spinlock.h
generic-y += mm-arch-hooks.h
diff --git a/arch/score/include/asm/extable.h b/arch/score/include/asm/extable.h
deleted file mode 100644
index c4423ccf..0000000
--- a/arch/score/include/asm/extable.h
+++ /dev/null
@@ -1,11 +0,0 @@
-#ifndef _ASM_SCORE_EXTABLE_H
-#define _ASM_SCORE_EXTABLE_H
-
-struct exception_table_entry {
- unsigned long insn;
- unsigned long fixup;
-};
-
-struct pt_regs;
-extern int fixup_exception(struct pt_regs *regs);
-#endif
diff --git a/arch/score/include/asm/uaccess.h b/arch/score/include/asm/uaccess.h
index db58ab9..916e5db 100644
--- a/arch/score/include/asm/uaccess.h
+++ b/arch/score/include/asm/uaccess.h
@@ -2,13 +2,8 @@
#define __SCORE_UACCESS_H
#include <linux/kernel.h>
-#include <linux/errno.h>
-#include <linux/thread_info.h>
#include <asm/extable.h>
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
#define get_ds() (KERNEL_DS)
#define get_fs() (current_thread_info()->addr_limit)
#define segment_eq(a, b) ((a).seg == (b).seg)
@@ -300,61 +295,19 @@
extern int __copy_tofrom_user(void *to, const void *from, unsigned long len);
static inline unsigned long
-copy_from_user(void *to, const void *from, unsigned long len)
+raw_copy_from_user(void *to, const void __user *from, unsigned long len)
{
- unsigned long res = len;
-
- if (likely(access_ok(VERIFY_READ, from, len)))
- res = __copy_tofrom_user(to, from, len);
-
- if (unlikely(res))
- memset(to + (len - res), 0, res);
-
- return res;
+ return __copy_tofrom_user(to, (__force const void *)from, len);
}
static inline unsigned long
-copy_to_user(void *to, const void *from, unsigned long len)
+raw_copy_to_user(void __user *to, const void *from, unsigned long len)
{
- if (likely(access_ok(VERIFY_WRITE, to, len)))
- len = __copy_tofrom_user(to, from, len);
-
- return len;
+ return __copy_tofrom_user((__force void *)to, from, len);
}
-static inline unsigned long
-__copy_from_user(void *to, const void *from, unsigned long len)
-{
- unsigned long left = __copy_tofrom_user(to, from, len);
- if (unlikely(left))
- memset(to + (len - left), 0, left);
- return left;
-}
-
-#define __copy_to_user(to, from, len) \
- __copy_tofrom_user((to), (from), (len))
-
-static inline unsigned long
-__copy_to_user_inatomic(void *to, const void *from, unsigned long len)
-{
- return __copy_to_user(to, from, len);
-}
-
-static inline unsigned long
-__copy_from_user_inatomic(void *to, const void *from, unsigned long len)
-{
- return __copy_tofrom_user(to, from, len);
-}
-
-#define __copy_in_user(to, from, len) __copy_tofrom_user(to, from, len)
-
-static inline unsigned long
-copy_in_user(void *to, const void *from, unsigned long len)
-{
- if (access_ok(VERIFY_READ, from, len) &&
- access_ok(VERFITY_WRITE, to, len))
- return __copy_tofrom_user(to, from, len);
-}
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
/*
* __clear_user: - Zero a block of memory in user space, with less checking.
diff --git a/arch/score/kernel/time.c b/arch/score/kernel/time.c
index 679b8d7..29aafc7 100644
--- a/arch/score/kernel/time.c
+++ b/arch/score/kernel/time.c
@@ -81,8 +81,10 @@
score_clockevent.shift);
score_clockevent.max_delta_ns = clockevent_delta2ns((u32)~0,
&score_clockevent);
+ score_clockevent.max_delta_ticks = (u32)~0;
score_clockevent.min_delta_ns = clockevent_delta2ns(50,
&score_clockevent) + 1;
+ score_clockevent.min_delta_ticks = 50;
score_clockevent.cpumask = cpumask_of(0);
clockevents_register_device(&score_clockevent);
}
diff --git a/arch/sh/include/asm/bug.h b/arch/sh/include/asm/bug.h
index dcf2780..1b77f06 100644
--- a/arch/sh/include/asm/bug.h
+++ b/arch/sh/include/asm/bug.h
@@ -50,7 +50,7 @@
"i" (sizeof(struct bug_entry))); \
} while (0)
-#define __WARN_TAINT(taint) \
+#define __WARN_FLAGS(flags) \
do { \
__asm__ __volatile__ ( \
"1:\t.short %O0\n" \
@@ -59,7 +59,7 @@
: "n" (TRAPA_BUG_OPCODE), \
"i" (__FILE__), \
"i" (__LINE__), \
- "i" (BUGFLAG_TAINT(taint)), \
+ "i" (BUGFLAG_WARNING|(flags)), \
"i" (sizeof(struct bug_entry))); \
} while (0)
diff --git a/arch/sh/include/asm/extable.h b/arch/sh/include/asm/extable.h
new file mode 100644
index 0000000..df2ee2f
--- /dev/null
+++ b/arch/sh/include/asm/extable.h
@@ -0,0 +1,10 @@
+#ifndef __ASM_SH_EXTABLE_H
+#define __ASM_SH_EXTABLE_H
+
+#include <asm-generic/extable.h>
+
+#if defined(CONFIG_SUPERH64) && defined(CONFIG_MMU)
+#define ARCH_HAS_SEARCH_EXTABLE
+#endif
+
+#endif
diff --git a/arch/sh/include/asm/uaccess.h b/arch/sh/include/asm/uaccess.h
index c4f0fee..2722b61 100644
--- a/arch/sh/include/asm/uaccess.h
+++ b/arch/sh/include/asm/uaccess.h
@@ -1,12 +1,8 @@
#ifndef __ASM_SH_UACCESS_H
#define __ASM_SH_UACCESS_H
-#include <linux/errno.h>
-#include <linux/sched.h>
#include <asm/segment.h>
-
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
+#include <asm/extable.h>
#define __addr_ok(addr) \
((unsigned long __force)(addr) < current_thread_info()->addr_limit.seg)
@@ -112,19 +108,18 @@
__kernel_size_t __copy_user(void *to, const void *from, __kernel_size_t n);
static __always_inline unsigned long
-__copy_from_user(void *to, const void __user *from, unsigned long n)
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
return __copy_user(to, (__force void *)from, n);
}
static __always_inline unsigned long __must_check
-__copy_to_user(void __user *to, const void *from, unsigned long n)
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
{
return __copy_user((__force void *)to, from, n);
}
-
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
/*
* Clear the area and return remaining number of bytes
@@ -144,55 +139,6 @@
__cl_size; \
})
-static inline unsigned long
-copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- unsigned long __copy_from = (unsigned long) from;
- __kernel_size_t __copy_size = (__kernel_size_t) n;
-
- if (__copy_size && __access_ok(__copy_from, __copy_size))
- __copy_size = __copy_user(to, from, __copy_size);
-
- if (unlikely(__copy_size))
- memset(to + (n - __copy_size), 0, __copy_size);
-
- return __copy_size;
-}
-
-static inline unsigned long
-copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- unsigned long __copy_to = (unsigned long) to;
- __kernel_size_t __copy_size = (__kernel_size_t) n;
-
- if (__copy_size && __access_ok(__copy_to, __copy_size))
- return __copy_user(to, from, __copy_size);
-
- return __copy_size;
-}
-
-/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-struct exception_table_entry {
- unsigned long insn, fixup;
-};
-
-#if defined(CONFIG_SUPERH64) && defined(CONFIG_MMU)
-#define ARCH_HAS_SEARCH_EXTABLE
-#endif
-
-int fixup_exception(struct pt_regs *regs);
-
extern void *set_exception_table_vec(unsigned int vec, void *handler);
static inline void *set_exception_table_evt(unsigned int evt, void *handler)
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index a59deae..58243b0 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -31,7 +31,8 @@
select ARCH_WANT_IPC_PARSE_VERSION
select GENERIC_PCI_IOMAP
select HAVE_NMI_WATCHDOG if SPARC64
- select HAVE_CBPF_JIT
+ select HAVE_CBPF_JIT if SPARC32
+ select HAVE_EBPF_JIT if SPARC64
select HAVE_DEBUG_BUGVERBOSE
select GENERIC_SMP_IDLE_THREAD
select GENERIC_CLOCKEVENTS
@@ -42,7 +43,6 @@
select OLD_SIGSUSPEND
select ARCH_HAS_SG_CHAIN
select CPU_NO_EFFICIENT_FFS
- select HAVE_ARCH_HARDENED_USERCOPY
select LOCKDEP_SMALL if LOCKDEP
select ARCH_WANT_RELAX_ORDER
@@ -82,6 +82,7 @@
select HAVE_ARCH_AUDITSYSCALL
select ARCH_SUPPORTS_ATOMIC_RMW
select HAVE_NMI
+ select HAVE_REGS_AND_STACK_ACCESS_API
config ARCH_DEFCONFIG
string
diff --git a/arch/sparc/include/asm/ptrace.h b/arch/sparc/include/asm/ptrace.h
index ca57f08..d73428e 100644
--- a/arch/sparc/include/asm/ptrace.h
+++ b/arch/sparc/include/asm/ptrace.h
@@ -83,7 +83,8 @@
#define MAX_REG_OFFSET (offsetof(struct pt_regs, magic))
-extern int regs_query_register_offset(const char *name);
+int regs_query_register_offset(const char *name);
+unsigned long regs_get_kernel_stack_nth(struct pt_regs *regs, unsigned int n);
/**
* regs_get_register() - get register value from its offset
diff --git a/arch/sparc/include/asm/uaccess.h b/arch/sparc/include/asm/uaccess.h
index bd56c28..9e068bf 100644
--- a/arch/sparc/include/asm/uaccess.h
+++ b/arch/sparc/include/asm/uaccess.h
@@ -7,7 +7,7 @@
#endif
#define user_addr_max() \
- (segment_eq(get_fs(), USER_DS) ? TASK_SIZE : ~0UL)
+ (uaccess_kernel() ? ~0UL : TASK_SIZE)
long strncpy_from_user(char *dest, const char __user *src, long count);
diff --git a/arch/sparc/include/asm/uaccess_32.h b/arch/sparc/include/asm/uaccess_32.h
index ea55f86..12ebee2 100644
--- a/arch/sparc/include/asm/uaccess_32.h
+++ b/arch/sparc/include/asm/uaccess_32.h
@@ -7,14 +7,8 @@
#ifndef _ASM_UACCESS_H
#define _ASM_UACCESS_H
-#ifdef __KERNEL__
#include <linux/compiler.h>
-#include <linux/sched.h>
#include <linux/string.h>
-#include <linux/errno.h>
-#endif
-
-#ifndef __ASSEMBLY__
#include <asm/processor.h>
@@ -30,9 +24,6 @@
#define KERNEL_DS ((mm_segment_t) { 0 })
#define USER_DS ((mm_segment_t) { -1 })
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
#define get_ds() (KERNEL_DS)
#define get_fs() (current->thread.current_ds)
#define set_fs(val) ((current->thread.current_ds) = (val))
@@ -45,7 +36,7 @@
* large size and address near to PAGE_OFFSET - a fault will break his intentions.
*/
#define __user_ok(addr, size) ({ (void)(size); (addr) < STACK_TOP; })
-#define __kernel_ok (segment_eq(get_fs(), KERNEL_DS))
+#define __kernel_ok (uaccess_kernel())
#define __access_ok(addr, size) (__user_ok((addr) & get_fs().seg, (size)))
#define access_ok(type, addr, size) \
({ (void)(type); __access_ok((unsigned long)(addr), size); })
@@ -80,8 +71,6 @@
/* Returns 0 if exception not found and fixup otherwise. */
unsigned long search_extables_range(unsigned long addr, unsigned long *g2);
-void __ret_efault(void);
-
/* Uh, these should become the main single-value transfer routines..
* They automatically use the right size if we just have the right
* pointer type..
@@ -246,39 +235,18 @@
unsigned long __copy_user(void __user *to, const void __user *from, unsigned long size);
-static inline unsigned long copy_to_user(void __user *to, const void *from, unsigned long n)
+static inline unsigned long raw_copy_to_user(void __user *to, const void *from, unsigned long n)
{
- if (n && __access_ok((unsigned long) to, n)) {
- check_object_size(from, n, true);
- return __copy_user(to, (__force void __user *) from, n);
- } else
- return n;
-}
-
-static inline unsigned long __copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- check_object_size(from, n, true);
return __copy_user(to, (__force void __user *) from, n);
}
-static inline unsigned long copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- if (n && __access_ok((unsigned long) from, n)) {
- check_object_size(to, n, false);
- return __copy_user((__force void __user *) to, from, n);
- } else {
- memset(to, 0, n);
- return n;
- }
-}
-
-static inline unsigned long __copy_from_user(void *to, const void __user *from, unsigned long n)
+static inline unsigned long raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
return __copy_user((__force void __user *) to, from, n);
}
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
static inline unsigned long __clear_user(void __user *addr, unsigned long size)
{
@@ -312,6 +280,4 @@
__must_check long strlen_user(const char __user *str);
__must_check long strnlen_user(const char __user *str, long n);
-#endif /* __ASSEMBLY__ */
-
#endif /* _ASM_UACCESS_H */
diff --git a/arch/sparc/include/asm/uaccess_64.h b/arch/sparc/include/asm/uaccess_64.h
index 5373136..6096d67 100644
--- a/arch/sparc/include/asm/uaccess_64.h
+++ b/arch/sparc/include/asm/uaccess_64.h
@@ -5,18 +5,12 @@
* User space memory access functions
*/
-#ifdef __KERNEL__
-#include <linux/errno.h>
#include <linux/compiler.h>
#include <linux/string.h>
-#include <linux/thread_info.h>
#include <asm/asi.h>
#include <asm/spitfire.h>
#include <asm-generic/uaccess-unaligned.h>
#include <asm/extable_64.h>
-#endif
-
-#ifndef __ASSEMBLY__
#include <asm/processor.h>
@@ -36,9 +30,6 @@
#define KERNEL_DS ((mm_segment_t) { ASI_P })
#define USER_DS ((mm_segment_t) { ASI_AIUS }) /* har har har */
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
#define get_fs() ((mm_segment_t){(current_thread_info()->current_ds)})
#define get_ds() (KERNEL_DS)
@@ -185,39 +176,19 @@
int __get_user_bad(void);
-unsigned long __must_check ___copy_from_user(void *to,
+unsigned long __must_check raw_copy_from_user(void *to,
const void __user *from,
unsigned long size);
-static inline unsigned long __must_check
-copy_from_user(void *to, const void __user *from, unsigned long size)
-{
- check_object_size(to, size, false);
- return ___copy_from_user(to, from, size);
-}
-#define __copy_from_user copy_from_user
-
-unsigned long __must_check ___copy_to_user(void __user *to,
+unsigned long __must_check raw_copy_to_user(void __user *to,
const void *from,
unsigned long size);
-static inline unsigned long __must_check
-copy_to_user(void __user *to, const void *from, unsigned long size)
-{
- check_object_size(from, size, true);
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
- return ___copy_to_user(to, from, size);
-}
-#define __copy_to_user copy_to_user
-
-unsigned long __must_check ___copy_in_user(void __user *to,
+unsigned long __must_check raw_copy_in_user(void __user *to,
const void __user *from,
unsigned long size);
-static inline unsigned long __must_check
-copy_in_user(void __user *to, void __user *from, unsigned long size)
-{
- return ___copy_in_user(to, from, size);
-}
-#define __copy_in_user copy_in_user
unsigned long __must_check __clear_user(void __user *, unsigned long);
@@ -226,14 +197,9 @@
__must_check long strlen_user(const char __user *str);
__must_check long strnlen_user(const char __user *str, long n);
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
-
struct pt_regs;
unsigned long compute_effective_address(struct pt_regs *,
unsigned int insn,
unsigned int rd);
-#endif /* __ASSEMBLY__ */
-
#endif /* _ASM_UACCESS_H */
diff --git a/arch/sparc/include/uapi/asm/socket.h b/arch/sparc/include/uapi/asm/socket.h
index a25dc32..3f4ad19 100644
--- a/arch/sparc/include/uapi/asm/socket.h
+++ b/arch/sparc/include/uapi/asm/socket.h
@@ -88,6 +88,12 @@
#define SCM_TIMESTAMPING_OPT_STATS 0x0038
+#define SO_MEMINFO 0x0039
+
+#define SO_INCOMING_NAPI_ID 0x003a
+
+#define SO_COOKIE 0x003b
+
/* Security levels - as per NRL IPv6 - don't actually do anything */
#define SO_SECURITY_AUTHENTICATION 0x5001
#define SO_SECURITY_ENCRYPTION_TRANSPORT 0x5002
diff --git a/arch/sparc/include/uapi/asm/unistd.h b/arch/sparc/include/uapi/asm/unistd.h
index 36eee81..ae77df7 100644
--- a/arch/sparc/include/uapi/asm/unistd.h
+++ b/arch/sparc/include/uapi/asm/unistd.h
@@ -425,8 +425,9 @@
#define __NR_copy_file_range 357
#define __NR_preadv2 358
#define __NR_pwritev2 359
+#define __NR_statx 360
-#define NR_syscalls 360
+#define NR_syscalls 361
/* Bitmask values returned from kern_features system call. */
#define KERN_FEATURE_MIXED_MODE_STACK 0x00000001
@@ -442,4 +443,9 @@
#define __IGNORE_getresgid
#endif
+/* Sparc doesn't have protection keys. */
+#define __IGNORE_pkey_mprotect
+#define __IGNORE_pkey_alloc
+#define __IGNORE_pkey_free
+
#endif /* _UAPI_SPARC_UNISTD_H */
diff --git a/arch/sparc/kernel/head_32.S b/arch/sparc/kernel/head_32.S
index 7bb317b..7274e43 100644
--- a/arch/sparc/kernel/head_32.S
+++ b/arch/sparc/kernel/head_32.S
@@ -809,10 +809,3 @@
.word 0
.word 0
.word t_irq14
-
- .section ".fixup",#alloc,#execinstr
- .globl __ret_efault
-__ret_efault:
- ret
- restore %g0, -EFAULT, %o0
-EXPORT_SYMBOL(__ret_efault)
diff --git a/arch/sparc/kernel/ptrace_64.c b/arch/sparc/kernel/ptrace_64.c
index fc5124c..e1d965e 100644
--- a/arch/sparc/kernel/ptrace_64.c
+++ b/arch/sparc/kernel/ptrace_64.c
@@ -1162,3 +1162,39 @@
return roff->offset;
return -EINVAL;
}
+
+/**
+ * regs_within_kernel_stack() - check the address in the stack
+ * @regs: pt_regs which contains kernel stack pointer.
+ * @addr: address which is checked.
+ *
+ * regs_within_kernel_stack() checks @addr is within the kernel stack page(s).
+ * If @addr is within the kernel stack, it returns true. If not, returns false.
+ */
+static inline int regs_within_kernel_stack(struct pt_regs *regs,
+ unsigned long addr)
+{
+ unsigned long ksp = kernel_stack_pointer(regs) + STACK_BIAS;
+ return ((addr & ~(THREAD_SIZE - 1)) ==
+ (ksp & ~(THREAD_SIZE - 1)));
+}
+
+/**
+ * regs_get_kernel_stack_nth() - get Nth entry of the stack
+ * @regs: pt_regs which contains kernel stack pointer.
+ * @n: stack entry number.
+ *
+ * regs_get_kernel_stack_nth() returns @n th entry of the kernel stack which
+ * is specified by @regs. If the @n th entry is NOT in the kernel stack,
+ * this returns 0.
+ */
+unsigned long regs_get_kernel_stack_nth(struct pt_regs *regs, unsigned int n)
+{
+ unsigned long ksp = kernel_stack_pointer(regs) + STACK_BIAS;
+ unsigned long *addr = (unsigned long *)ksp;
+ addr += n;
+ if (regs_within_kernel_stack(regs, (unsigned long)addr))
+ return *addr;
+ else
+ return 0;
+}
diff --git a/arch/sparc/kernel/sysfs.c b/arch/sparc/kernel/sysfs.c
index d63fc61..5fd352b 100644
--- a/arch/sparc/kernel/sysfs.c
+++ b/arch/sparc/kernel/sysfs.c
@@ -98,27 +98,7 @@
.name = "mmu_stats",
};
-/* XXX convert to rusty's on_one_cpu */
-static unsigned long run_on_cpu(unsigned long cpu,
- unsigned long (*func)(unsigned long),
- unsigned long arg)
-{
- cpumask_t old_affinity;
- unsigned long ret;
-
- cpumask_copy(&old_affinity, ¤t->cpus_allowed);
- /* should return -EINVAL to userspace */
- if (set_cpus_allowed_ptr(current, cpumask_of(cpu)))
- return 0;
-
- ret = func(arg);
-
- set_cpus_allowed_ptr(current, &old_affinity);
-
- return ret;
-}
-
-static unsigned long read_mmustat_enable(unsigned long junk)
+static long read_mmustat_enable(void *data __maybe_unused)
{
unsigned long ra = 0;
@@ -127,11 +107,11 @@
return ra != 0;
}
-static unsigned long write_mmustat_enable(unsigned long val)
+static long write_mmustat_enable(void *data)
{
- unsigned long ra, orig_ra;
+ unsigned long ra, orig_ra, *val = data;
- if (val)
+ if (*val)
ra = __pa(&per_cpu(mmu_stats, smp_processor_id()));
else
ra = 0UL;
@@ -142,7 +122,8 @@
static ssize_t show_mmustat_enable(struct device *s,
struct device_attribute *attr, char *buf)
{
- unsigned long val = run_on_cpu(s->id, read_mmustat_enable, 0);
+ long val = work_on_cpu(s->id, read_mmustat_enable, NULL);
+
return sprintf(buf, "%lx\n", val);
}
@@ -150,13 +131,15 @@
struct device_attribute *attr, const char *buf,
size_t count)
{
- unsigned long val, err;
- int ret = sscanf(buf, "%lu", &val);
+ unsigned long val;
+ long err;
+ int ret;
+ ret = sscanf(buf, "%lu", &val);
if (ret != 1)
return -EINVAL;
- err = run_on_cpu(s->id, write_mmustat_enable, val);
+ err = work_on_cpu(s->id, write_mmustat_enable, &val);
if (err)
return -EIO;
diff --git a/arch/sparc/kernel/systbls_32.S b/arch/sparc/kernel/systbls_32.S
index eac7f0d..5253e89 100644
--- a/arch/sparc/kernel/systbls_32.S
+++ b/arch/sparc/kernel/systbls_32.S
@@ -89,3 +89,4 @@
/*345*/ .long sys_renameat2, sys_seccomp, sys_getrandom, sys_memfd_create, sys_bpf
/*350*/ .long sys_execveat, sys_membarrier, sys_userfaultfd, sys_bind, sys_listen
/*355*/ .long sys_setsockopt, sys_mlock2, sys_copy_file_range, sys_preadv2, sys_pwritev2
+/*360*/ .long sys_statx
diff --git a/arch/sparc/kernel/systbls_64.S b/arch/sparc/kernel/systbls_64.S
index b0f17ff..82339f6 100644
--- a/arch/sparc/kernel/systbls_64.S
+++ b/arch/sparc/kernel/systbls_64.S
@@ -90,6 +90,7 @@
.word sys32_renameat2, sys_seccomp, sys_getrandom, sys_memfd_create, sys_bpf
/*350*/ .word sys32_execveat, sys_membarrier, sys_userfaultfd, sys_bind, sys_listen
.word compat_sys_setsockopt, sys_mlock2, sys_copy_file_range, compat_sys_preadv2, compat_sys_pwritev2
+/*360*/ .word sys_statx
#endif /* CONFIG_COMPAT */
@@ -171,3 +172,4 @@
.word sys_renameat2, sys_seccomp, sys_getrandom, sys_memfd_create, sys_bpf
/*350*/ .word sys64_execveat, sys_membarrier, sys_userfaultfd, sys_bind, sys_listen
.word sys_setsockopt, sys_mlock2, sys_copy_file_range, sys_preadv2, sys_pwritev2
+/*360*/ .word sys_statx
diff --git a/arch/sparc/kernel/time_32.c b/arch/sparc/kernel/time_32.c
index 244062b..9f575df 100644
--- a/arch/sparc/kernel/time_32.c
+++ b/arch/sparc/kernel/time_32.c
@@ -228,7 +228,9 @@
ce->mult = div_sc(sparc_config.clock_rate, NSEC_PER_SEC,
ce->shift);
ce->max_delta_ns = clockevent_delta2ns(sparc_config.clock_rate, ce);
+ ce->max_delta_ticks = (unsigned long)sparc_config.clock_rate;
ce->min_delta_ns = clockevent_delta2ns(100, ce);
+ ce->min_delta_ticks = 100;
clockevents_register_device(ce);
}
diff --git a/arch/sparc/kernel/time_64.c b/arch/sparc/kernel/time_64.c
index 12a6d35..98d05de 100644
--- a/arch/sparc/kernel/time_64.c
+++ b/arch/sparc/kernel/time_64.c
@@ -796,8 +796,10 @@
sparc64_clockevent.max_delta_ns =
clockevent_delta2ns(0x7fffffffffffffffUL, &sparc64_clockevent);
+ sparc64_clockevent.max_delta_ticks = 0x7fffffffffffffffUL;
sparc64_clockevent.min_delta_ns =
clockevent_delta2ns(0xF, &sparc64_clockevent);
+ sparc64_clockevent.min_delta_ticks = 0xF;
printk("clockevent: mult[%x] shift[%d]\n",
sparc64_clockevent.mult, sparc64_clockevent.shift);
diff --git a/arch/sparc/lib/GENcopy_from_user.S b/arch/sparc/lib/GENcopy_from_user.S
index 69a439f..8aa16ef 100644
--- a/arch/sparc/lib/GENcopy_from_user.S
+++ b/arch/sparc/lib/GENcopy_from_user.S
@@ -23,7 +23,7 @@
#define PREAMBLE \
rd %asi, %g1; \
cmp %g1, ASI_AIUS; \
- bne,pn %icc, ___copy_in_user; \
+ bne,pn %icc, raw_copy_in_user; \
nop
#endif
diff --git a/arch/sparc/lib/GENcopy_to_user.S b/arch/sparc/lib/GENcopy_to_user.S
index 9947427..311c8fa 100644
--- a/arch/sparc/lib/GENcopy_to_user.S
+++ b/arch/sparc/lib/GENcopy_to_user.S
@@ -27,7 +27,7 @@
#define PREAMBLE \
rd %asi, %g1; \
cmp %g1, ASI_AIUS; \
- bne,pn %icc, ___copy_in_user; \
+ bne,pn %icc, raw_copy_in_user; \
nop
#endif
diff --git a/arch/sparc/lib/GENpatch.S b/arch/sparc/lib/GENpatch.S
index fab9e89..95e2f1f 100644
--- a/arch/sparc/lib/GENpatch.S
+++ b/arch/sparc/lib/GENpatch.S
@@ -26,8 +26,8 @@
.type generic_patch_copyops,#function
generic_patch_copyops:
GEN_DO_PATCH(memcpy, GENmemcpy)
- GEN_DO_PATCH(___copy_from_user, GENcopy_from_user)
- GEN_DO_PATCH(___copy_to_user, GENcopy_to_user)
+ GEN_DO_PATCH(raw_copy_from_user, GENcopy_from_user)
+ GEN_DO_PATCH(raw_copy_to_user, GENcopy_to_user)
retl
nop
.size generic_patch_copyops,.-generic_patch_copyops
diff --git a/arch/sparc/lib/NG2copy_from_user.S b/arch/sparc/lib/NG2copy_from_user.S
index b79a699..0d8a018 100644
--- a/arch/sparc/lib/NG2copy_from_user.S
+++ b/arch/sparc/lib/NG2copy_from_user.S
@@ -36,7 +36,7 @@
#define PREAMBLE \
rd %asi, %g1; \
cmp %g1, ASI_AIUS; \
- bne,pn %icc, ___copy_in_user; \
+ bne,pn %icc, raw_copy_in_user; \
nop
#endif
diff --git a/arch/sparc/lib/NG2copy_to_user.S b/arch/sparc/lib/NG2copy_to_user.S
index dcec55f..a7a0ea0 100644
--- a/arch/sparc/lib/NG2copy_to_user.S
+++ b/arch/sparc/lib/NG2copy_to_user.S
@@ -45,7 +45,7 @@
#define PREAMBLE \
rd %asi, %g1; \
cmp %g1, ASI_AIUS; \
- bne,pn %icc, ___copy_in_user; \
+ bne,pn %icc, raw_copy_in_user; \
nop
#endif
diff --git a/arch/sparc/lib/NG2patch.S b/arch/sparc/lib/NG2patch.S
index 28c36f0..56ccc19 100644
--- a/arch/sparc/lib/NG2patch.S
+++ b/arch/sparc/lib/NG2patch.S
@@ -26,8 +26,8 @@
.type niagara2_patch_copyops,#function
niagara2_patch_copyops:
NG_DO_PATCH(memcpy, NG2memcpy)
- NG_DO_PATCH(___copy_from_user, NG2copy_from_user)
- NG_DO_PATCH(___copy_to_user, NG2copy_to_user)
+ NG_DO_PATCH(raw_copy_from_user, NG2copy_from_user)
+ NG_DO_PATCH(raw_copy_to_user, NG2copy_to_user)
retl
nop
.size niagara2_patch_copyops,.-niagara2_patch_copyops
diff --git a/arch/sparc/lib/NG4copy_from_user.S b/arch/sparc/lib/NG4copy_from_user.S
index 16a286c..5bb506b 100644
--- a/arch/sparc/lib/NG4copy_from_user.S
+++ b/arch/sparc/lib/NG4copy_from_user.S
@@ -31,7 +31,7 @@
#define PREAMBLE \
rd %asi, %g1; \
cmp %g1, ASI_AIUS; \
- bne,pn %icc, ___copy_in_user; \
+ bne,pn %icc, raw_copy_in_user; \
nop
#endif
diff --git a/arch/sparc/lib/NG4copy_to_user.S b/arch/sparc/lib/NG4copy_to_user.S
index 6b0276f..a82d4d4 100644
--- a/arch/sparc/lib/NG4copy_to_user.S
+++ b/arch/sparc/lib/NG4copy_to_user.S
@@ -40,7 +40,7 @@
#define PREAMBLE \
rd %asi, %g1; \
cmp %g1, ASI_AIUS; \
- bne,pn %icc, ___copy_in_user; \
+ bne,pn %icc, raw_copy_in_user; \
nop
#endif
diff --git a/arch/sparc/lib/NG4patch.S b/arch/sparc/lib/NG4patch.S
index a114cbc..3cc0f8c 100644
--- a/arch/sparc/lib/NG4patch.S
+++ b/arch/sparc/lib/NG4patch.S
@@ -26,8 +26,8 @@
.type niagara4_patch_copyops,#function
niagara4_patch_copyops:
NG_DO_PATCH(memcpy, NG4memcpy)
- NG_DO_PATCH(___copy_from_user, NG4copy_from_user)
- NG_DO_PATCH(___copy_to_user, NG4copy_to_user)
+ NG_DO_PATCH(raw_copy_from_user, NG4copy_from_user)
+ NG_DO_PATCH(raw_copy_to_user, NG4copy_to_user)
retl
nop
.size niagara4_patch_copyops,.-niagara4_patch_copyops
diff --git a/arch/sparc/lib/NGcopy_from_user.S b/arch/sparc/lib/NGcopy_from_user.S
index 9cd42fc..2333b6f 100644
--- a/arch/sparc/lib/NGcopy_from_user.S
+++ b/arch/sparc/lib/NGcopy_from_user.S
@@ -25,7 +25,7 @@
#define PREAMBLE \
rd %asi, %g1; \
cmp %g1, ASI_AIUS; \
- bne,pn %icc, ___copy_in_user; \
+ bne,pn %icc, raw_copy_in_user; \
nop
#endif
diff --git a/arch/sparc/lib/NGcopy_to_user.S b/arch/sparc/lib/NGcopy_to_user.S
index 5c358af..07ba20b 100644
--- a/arch/sparc/lib/NGcopy_to_user.S
+++ b/arch/sparc/lib/NGcopy_to_user.S
@@ -28,7 +28,7 @@
#define PREAMBLE \
rd %asi, %g1; \
cmp %g1, ASI_AIUS; \
- bne,pn %icc, ___copy_in_user; \
+ bne,pn %icc, raw_copy_in_user; \
nop
#endif
diff --git a/arch/sparc/lib/NGpatch.S b/arch/sparc/lib/NGpatch.S
index 3b0674f..62ccda7 100644
--- a/arch/sparc/lib/NGpatch.S
+++ b/arch/sparc/lib/NGpatch.S
@@ -26,8 +26,8 @@
.type niagara_patch_copyops,#function
niagara_patch_copyops:
NG_DO_PATCH(memcpy, NGmemcpy)
- NG_DO_PATCH(___copy_from_user, NGcopy_from_user)
- NG_DO_PATCH(___copy_to_user, NGcopy_to_user)
+ NG_DO_PATCH(raw_copy_from_user, NGcopy_from_user)
+ NG_DO_PATCH(raw_copy_to_user, NGcopy_to_user)
retl
nop
.size niagara_patch_copyops,.-niagara_patch_copyops
diff --git a/arch/sparc/lib/U1copy_from_user.S b/arch/sparc/lib/U1copy_from_user.S
index bb6ff73..9a6e68a 100644
--- a/arch/sparc/lib/U1copy_from_user.S
+++ b/arch/sparc/lib/U1copy_from_user.S
@@ -19,7 +19,7 @@
.text; \
.align 4;
-#define FUNC_NAME ___copy_from_user
+#define FUNC_NAME raw_copy_from_user
#define LOAD(type,addr,dest) type##a [addr] %asi, dest
#define LOAD_BLK(addr,dest) ldda [addr] ASI_BLK_AIUS, dest
#define EX_RETVAL(x) 0
@@ -31,7 +31,7 @@
#define PREAMBLE \
rd %asi, %g1; \
cmp %g1, ASI_AIUS; \
- bne,pn %icc, ___copy_in_user; \
+ bne,pn %icc, raw_copy_in_user; \
nop; \
#include "U1memcpy.S"
diff --git a/arch/sparc/lib/U1copy_to_user.S b/arch/sparc/lib/U1copy_to_user.S
index ed92ce73..d7b2849 100644
--- a/arch/sparc/lib/U1copy_to_user.S
+++ b/arch/sparc/lib/U1copy_to_user.S
@@ -19,7 +19,7 @@
.text; \
.align 4;
-#define FUNC_NAME ___copy_to_user
+#define FUNC_NAME raw_copy_to_user
#define STORE(type,src,addr) type##a src, [addr] ASI_AIUS
#define STORE_BLK(src,addr) stda src, [addr] ASI_BLK_AIUS
#define EX_RETVAL(x) 0
@@ -31,7 +31,7 @@
#define PREAMBLE \
rd %asi, %g1; \
cmp %g1, ASI_AIUS; \
- bne,pn %icc, ___copy_in_user; \
+ bne,pn %icc, raw_copy_in_user; \
nop; \
#include "U1memcpy.S"
diff --git a/arch/sparc/lib/U3copy_to_user.S b/arch/sparc/lib/U3copy_to_user.S
index c4ee858..f48fb87 100644
--- a/arch/sparc/lib/U3copy_to_user.S
+++ b/arch/sparc/lib/U3copy_to_user.S
@@ -31,7 +31,7 @@
#define PREAMBLE \
rd %asi, %g1; \
cmp %g1, ASI_AIUS; \
- bne,pn %icc, ___copy_in_user; \
+ bne,pn %icc, raw_copy_in_user; \
nop; \
#include "U3memcpy.S"
diff --git a/arch/sparc/lib/U3patch.S b/arch/sparc/lib/U3patch.S
index ecc3026..91cd653 100644
--- a/arch/sparc/lib/U3patch.S
+++ b/arch/sparc/lib/U3patch.S
@@ -26,8 +26,8 @@
.type cheetah_patch_copyops,#function
cheetah_patch_copyops:
ULTRA3_DO_PATCH(memcpy, U3memcpy)
- ULTRA3_DO_PATCH(___copy_from_user, U3copy_from_user)
- ULTRA3_DO_PATCH(___copy_to_user, U3copy_to_user)
+ ULTRA3_DO_PATCH(raw_copy_from_user, U3copy_from_user)
+ ULTRA3_DO_PATCH(raw_copy_to_user, U3copy_to_user)
retl
nop
.size cheetah_patch_copyops,.-cheetah_patch_copyops
diff --git a/arch/sparc/lib/copy_in_user.S b/arch/sparc/lib/copy_in_user.S
index 0252b21..1b73bb8 100644
--- a/arch/sparc/lib/copy_in_user.S
+++ b/arch/sparc/lib/copy_in_user.S
@@ -44,7 +44,7 @@
* to copy register windows around during thread cloning.
*/
-ENTRY(___copy_in_user) /* %o0=dst, %o1=src, %o2=len */
+ENTRY(raw_copy_in_user) /* %o0=dst, %o1=src, %o2=len */
cmp %o2, 0
be,pn %XCC, 85f
or %o0, %o1, %o3
@@ -105,5 +105,5 @@
add %o0, 1, %o0
retl
clr %o0
-ENDPROC(___copy_in_user)
-EXPORT_SYMBOL(___copy_in_user)
+ENDPROC(raw_copy_in_user)
+EXPORT_SYMBOL(raw_copy_in_user)
diff --git a/arch/sparc/lib/copy_user.S b/arch/sparc/lib/copy_user.S
index cea644d..bc243ee 100644
--- a/arch/sparc/lib/copy_user.S
+++ b/arch/sparc/lib/copy_user.S
@@ -364,21 +364,7 @@
97:
mov %o2, %g3
fixupretl:
- sethi %hi(PAGE_OFFSET), %g1
- cmp %o0, %g1
- blu 1f
- cmp %o1, %g1
- bgeu 1f
- ld [%g6 + TI_PREEMPT], %g1
- cmp %g1, 0
- bne 1f
- nop
- save %sp, -64, %sp
- mov %i0, %o0
- call __bzero
- mov %g3, %o1
- restore
-1: retl
+ retl
mov %g3, %o0
/* exception routine sets %g2 to (broken_insn - first_insn)>>2 */
diff --git a/arch/sparc/net/Makefile b/arch/sparc/net/Makefile
index 1306a58..76fa8e9 100644
--- a/arch/sparc/net/Makefile
+++ b/arch/sparc/net/Makefile
@@ -1,4 +1,4 @@
#
# Arch-specific network modules
#
-obj-$(CONFIG_BPF_JIT) += bpf_jit_asm.o bpf_jit_comp.o
+obj-$(CONFIG_BPF_JIT) += bpf_jit_asm_$(BITS).o bpf_jit_comp_$(BITS).o
diff --git a/arch/sparc/net/bpf_jit.h b/arch/sparc/net/bpf_jit_32.h
similarity index 96%
rename from arch/sparc/net/bpf_jit.h
rename to arch/sparc/net/bpf_jit_32.h
index 33d6b37..d5c069b 100644
--- a/arch/sparc/net/bpf_jit.h
+++ b/arch/sparc/net/bpf_jit_32.h
@@ -39,7 +39,7 @@
#define r_TMP2 G2
#define r_OFF G3
-/* assembly code in arch/sparc/net/bpf_jit_asm.S */
+/* assembly code in arch/sparc/net/bpf_jit_asm_32.S */
extern u32 bpf_jit_load_word[];
extern u32 bpf_jit_load_half[];
extern u32 bpf_jit_load_byte[];
diff --git a/arch/sparc/net/bpf_jit_64.h b/arch/sparc/net/bpf_jit_64.h
new file mode 100644
index 0000000..74abd45
--- /dev/null
+++ b/arch/sparc/net/bpf_jit_64.h
@@ -0,0 +1,66 @@
+#ifndef _BPF_JIT_H
+#define _BPF_JIT_H
+
+#ifndef __ASSEMBLER__
+#define G0 0x00
+#define G1 0x01
+#define G2 0x02
+#define G3 0x03
+#define G6 0x06
+#define G7 0x07
+#define O0 0x08
+#define O1 0x09
+#define O2 0x0a
+#define O3 0x0b
+#define O4 0x0c
+#define O5 0x0d
+#define SP 0x0e
+#define O7 0x0f
+#define L0 0x10
+#define L1 0x11
+#define L2 0x12
+#define L3 0x13
+#define L4 0x14
+#define L5 0x15
+#define L6 0x16
+#define L7 0x17
+#define I0 0x18
+#define I1 0x19
+#define I2 0x1a
+#define I3 0x1b
+#define I4 0x1c
+#define I5 0x1d
+#define FP 0x1e
+#define I7 0x1f
+
+#define r_SKB L0
+#define r_HEADLEN L4
+#define r_SKB_DATA L5
+#define r_TMP G1
+#define r_TMP2 G3
+
+/* assembly code in arch/sparc/net/bpf_jit_asm_64.S */
+extern u32 bpf_jit_load_word[];
+extern u32 bpf_jit_load_half[];
+extern u32 bpf_jit_load_byte[];
+extern u32 bpf_jit_load_byte_msh[];
+extern u32 bpf_jit_load_word_positive_offset[];
+extern u32 bpf_jit_load_half_positive_offset[];
+extern u32 bpf_jit_load_byte_positive_offset[];
+extern u32 bpf_jit_load_byte_msh_positive_offset[];
+extern u32 bpf_jit_load_word_negative_offset[];
+extern u32 bpf_jit_load_half_negative_offset[];
+extern u32 bpf_jit_load_byte_negative_offset[];
+extern u32 bpf_jit_load_byte_msh_negative_offset[];
+
+#else
+#define r_RESULT %o0
+#define r_SKB %o0
+#define r_OFF %o1
+#define r_HEADLEN %l4
+#define r_SKB_DATA %l5
+#define r_TMP %g1
+#define r_TMP2 %g3
+#endif
+
+#endif /* _BPF_JIT_H */
diff --git a/arch/sparc/net/bpf_jit_asm.S b/arch/sparc/net/bpf_jit_asm_32.S
similarity index 95%
rename from arch/sparc/net/bpf_jit_asm.S
rename to arch/sparc/net/bpf_jit_asm_32.S
index 8c83f4b..dcc402f 100644
--- a/arch/sparc/net/bpf_jit_asm.S
+++ b/arch/sparc/net/bpf_jit_asm_32.S
@@ -1,18 +1,11 @@
#include <asm/ptrace.h>
-#include "bpf_jit.h"
+#include "bpf_jit_32.h"
-#ifdef CONFIG_SPARC64
-#define SAVE_SZ 176
-#define SCRATCH_OFF STACK_BIAS + 128
-#define BE_PTR(label) be,pn %xcc, label
-#define SIGN_EXTEND(reg) sra reg, 0, reg
-#else
#define SAVE_SZ 96
#define SCRATCH_OFF 72
#define BE_PTR(label) be label
#define SIGN_EXTEND(reg)
-#endif
#define SKF_MAX_NEG_OFF (-0x200000) /* SKF_LL_OFF from filter.h */
diff --git a/arch/sparc/net/bpf_jit_asm.S b/arch/sparc/net/bpf_jit_asm_64.S
similarity index 64%
copy from arch/sparc/net/bpf_jit_asm.S
copy to arch/sparc/net/bpf_jit_asm_64.S
index 8c83f4b..3b3f146 100644
--- a/arch/sparc/net/bpf_jit_asm.S
+++ b/arch/sparc/net/bpf_jit_asm_64.S
@@ -1,18 +1,11 @@
#include <asm/ptrace.h>
-#include "bpf_jit.h"
+#include "bpf_jit_64.h"
-#ifdef CONFIG_SPARC64
#define SAVE_SZ 176
#define SCRATCH_OFF STACK_BIAS + 128
#define BE_PTR(label) be,pn %xcc, label
#define SIGN_EXTEND(reg) sra reg, 0, reg
-#else
-#define SAVE_SZ 96
-#define SCRATCH_OFF 72
-#define BE_PTR(label) be label
-#define SIGN_EXTEND(reg)
-#endif
#define SKF_MAX_NEG_OFF (-0x200000) /* SKF_LL_OFF from filter.h */
@@ -32,7 +25,7 @@
bne load_word_unaligned
nop
retl
- ld [r_TMP], r_A
+ ld [r_TMP], r_RESULT
load_word_unaligned:
ldub [r_TMP + 0x0], r_OFF
ldub [r_TMP + 0x1], r_TMP2
@@ -44,7 +37,7 @@
ldub [r_TMP + 0x3], r_TMP2
sll r_OFF, 8, r_OFF
retl
- or r_OFF, r_TMP2, r_A
+ or r_OFF, r_TMP2, r_RESULT
.globl bpf_jit_load_half
bpf_jit_load_half:
@@ -61,13 +54,13 @@
bne load_half_unaligned
nop
retl
- lduh [r_TMP], r_A
+ lduh [r_TMP], r_RESULT
load_half_unaligned:
ldub [r_TMP + 0x0], r_OFF
ldub [r_TMP + 0x1], r_TMP2
sll r_OFF, 8, r_OFF
retl
- or r_OFF, r_TMP2, r_A
+ or r_OFF, r_TMP2, r_RESULT
.globl bpf_jit_load_byte
bpf_jit_load_byte:
@@ -80,27 +73,12 @@
bge bpf_slow_path_byte
nop
retl
- ldub [r_SKB_DATA + r_OFF], r_A
-
- .globl bpf_jit_load_byte_msh
-bpf_jit_load_byte_msh:
- cmp r_OFF, 0
- bl bpf_slow_path_byte_msh_neg
- nop
- .globl bpf_jit_load_byte_msh_positive_offset
-bpf_jit_load_byte_msh_positive_offset:
- cmp r_OFF, r_HEADLEN
- bge bpf_slow_path_byte_msh
- nop
- ldub [r_SKB_DATA + r_OFF], r_OFF
- and r_OFF, 0xf, r_OFF
- retl
- sll r_OFF, 2, r_X
+ ldub [r_SKB_DATA + r_OFF], r_RESULT
#define bpf_slow_path_common(LEN) \
save %sp, -SAVE_SZ, %sp; \
mov %i0, %o0; \
- mov r_OFF, %o1; \
+ mov %i1, %o1; \
add %fp, SCRATCH_OFF, %o2; \
call skb_copy_bits; \
mov (LEN), %o3; \
@@ -110,33 +88,26 @@
bpf_slow_path_word:
bpf_slow_path_common(4)
bl bpf_error
- ld [%sp + SCRATCH_OFF], r_A
+ ld [%sp + SCRATCH_OFF], r_RESULT
retl
nop
bpf_slow_path_half:
bpf_slow_path_common(2)
bl bpf_error
- lduh [%sp + SCRATCH_OFF], r_A
+ lduh [%sp + SCRATCH_OFF], r_RESULT
retl
nop
bpf_slow_path_byte:
bpf_slow_path_common(1)
bl bpf_error
- ldub [%sp + SCRATCH_OFF], r_A
+ ldub [%sp + SCRATCH_OFF], r_RESULT
retl
nop
-bpf_slow_path_byte_msh:
- bpf_slow_path_common(1)
- bl bpf_error
- ldub [%sp + SCRATCH_OFF], r_A
- and r_OFF, 0xf, r_OFF
- retl
- sll r_OFF, 2, r_X
#define bpf_negative_common(LEN) \
save %sp, -SAVE_SZ, %sp; \
mov %i0, %o0; \
- mov r_OFF, %o1; \
+ mov %i1, %o1; \
SIGN_EXTEND(%o1); \
call bpf_internal_load_pointer_neg_helper; \
mov (LEN), %o2; \
@@ -157,7 +128,7 @@
bne load_word_unaligned
nop
retl
- ld [r_TMP], r_A
+ ld [r_TMP], r_RESULT
bpf_slow_path_half_neg:
sethi %hi(SKF_MAX_NEG_OFF), r_TMP
@@ -171,7 +142,7 @@
bne load_half_unaligned
nop
retl
- lduh [r_TMP], r_A
+ lduh [r_TMP], r_RESULT
bpf_slow_path_byte_neg:
sethi %hi(SKF_MAX_NEG_OFF), r_TMP
@@ -182,27 +153,9 @@
bpf_jit_load_byte_negative_offset:
bpf_negative_common(1)
retl
- ldub [r_TMP], r_A
-
-bpf_slow_path_byte_msh_neg:
- sethi %hi(SKF_MAX_NEG_OFF), r_TMP
- cmp r_OFF, r_TMP
- bl bpf_error
- nop
- .globl bpf_jit_load_byte_msh_negative_offset
-bpf_jit_load_byte_msh_negative_offset:
- bpf_negative_common(1)
- ldub [r_TMP], r_OFF
- and r_OFF, 0xf, r_OFF
- retl
- sll r_OFF, 2, r_X
+ ldub [r_TMP], r_RESULT
bpf_error:
- /* Make the JIT program return zero. The JIT epilogue
- * stores away the original %o7 into r_saved_O7. The
- * normal leaf function return is to use "retl" which
- * would evalute to "jmpl %o7 + 8, %g0" but we want to
- * use the saved value thus the sequence you see here.
- */
- jmpl r_saved_O7 + 8, %g0
- clr %o0
+ /* Make the JIT program itself return zero. */
+ ret
+ restore %g0, %g0, %o0
diff --git a/arch/sparc/net/bpf_jit_comp.c b/arch/sparc/net/bpf_jit_comp_32.c
similarity index 94%
rename from arch/sparc/net/bpf_jit_comp.c
rename to arch/sparc/net/bpf_jit_comp_32.c
index a6d9204..d193748 100644
--- a/arch/sparc/net/bpf_jit_comp.c
+++ b/arch/sparc/net/bpf_jit_comp_32.c
@@ -8,7 +8,7 @@
#include <asm/cacheflush.h>
#include <asm/ptrace.h>
-#include "bpf_jit.h"
+#include "bpf_jit_32.h"
int bpf_jit_enable __read_mostly;
@@ -17,24 +17,6 @@
return value + 0x1000 < 0x2000;
}
-static void bpf_flush_icache(void *start_, void *end_)
-{
-#ifdef CONFIG_SPARC64
- /* Cheetah's I-cache is fully coherent. */
- if (tlb_type == spitfire) {
- unsigned long start = (unsigned long) start_;
- unsigned long end = (unsigned long) end_;
-
- start &= ~7UL;
- end = (end + 7UL) & ~7UL;
- while (start < end) {
- flushi(start);
- start += 32;
- }
- }
-#endif
-}
-
#define SEEN_DATAREF 1 /* might call external helpers */
#define SEEN_XREG 2 /* ebx is used */
#define SEEN_MEM 4 /* use mem[] for temporary storage */
@@ -82,11 +64,7 @@
#define BE (F2(0, 2) | CONDE)
#define BNE (F2(0, 2) | CONDNE)
-#ifdef CONFIG_SPARC64
-#define BE_PTR (F2(0, 1) | CONDE | (2 << 20))
-#else
#define BE_PTR BE
-#endif
#define SETHI(K, REG) \
(F2(0, 0x4) | RD(REG) | (((K) >> 10) & 0x3fffff))
@@ -116,13 +94,8 @@
#define LD64 F3(3, 0x0b)
#define ST32 F3(3, 0x04)
-#ifdef CONFIG_SPARC64
-#define LDPTR LD64
-#define BASE_STACKFRAME 176
-#else
#define LDPTR LD32
#define BASE_STACKFRAME 96
-#endif
#define LD32I (LD32 | IMMED)
#define LD8I (LD8 | IMMED)
@@ -234,11 +207,7 @@
__emit_load8(BASE, STRUCT, FIELD, DEST); \
} while (0)
-#ifdef CONFIG_SPARC64
-#define BIAS (STACK_BIAS - 4)
-#else
#define BIAS (-4)
-#endif
#define emit_ldmem(OFF, DEST) \
do { *prog++ = LD32I | RS1(SP) | S13(BIAS - (OFF)) | RD(DEST); \
@@ -249,13 +218,8 @@
} while (0)
#ifdef CONFIG_SMP
-#ifdef CONFIG_SPARC64
-#define emit_load_cpu(REG) \
- emit_load16(G6, struct thread_info, cpu, REG)
-#else
#define emit_load_cpu(REG) \
emit_load32(G6, struct thread_info, cpu, REG)
-#endif
#else
#define emit_load_cpu(REG) emit_clear(REG)
#endif
@@ -486,7 +450,6 @@
if (K == 1)
break;
emit_write_y(G0);
-#ifdef CONFIG_SPARC32
/* The Sparc v8 architecture requires
* three instructions between a %y
* register write and the first use.
@@ -494,31 +457,21 @@
emit_nop();
emit_nop();
emit_nop();
-#endif
emit_alu_K(DIV, K);
break;
case BPF_ALU | BPF_DIV | BPF_X: /* A /= X; */
emit_cmpi(r_X, 0);
if (pc_ret0 > 0) {
t_offset = addrs[pc_ret0 - 1];
-#ifdef CONFIG_SPARC32
emit_branch(BE, t_offset + 20);
-#else
- emit_branch(BE, t_offset + 8);
-#endif
emit_nop(); /* delay slot */
} else {
emit_branch_off(BNE, 16);
emit_nop();
-#ifdef CONFIG_SPARC32
emit_jump(cleanup_addr + 20);
-#else
- emit_jump(cleanup_addr + 8);
-#endif
emit_clear(r_A);
}
emit_write_y(G0);
-#ifdef CONFIG_SPARC32
/* The Sparc v8 architecture requires
* three instructions between a %y
* register write and the first use.
@@ -526,7 +479,6 @@
emit_nop();
emit_nop();
emit_nop();
-#endif
emit_alu_X(DIV);
break;
case BPF_ALU | BPF_NEG:
@@ -797,7 +749,6 @@
bpf_jit_dump(flen, proglen, pass + 1, image);
if (image) {
- bpf_flush_icache(image, image + proglen);
fp->bpf_func = (void *)image;
fp->jited = 1;
}
diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c
new file mode 100644
index 0000000..21de774
--- /dev/null
+++ b/arch/sparc/net/bpf_jit_comp_64.c
@@ -0,0 +1,1566 @@
+#include <linux/moduleloader.h>
+#include <linux/workqueue.h>
+#include <linux/netdevice.h>
+#include <linux/filter.h>
+#include <linux/bpf.h>
+#include <linux/cache.h>
+#include <linux/if_vlan.h>
+
+#include <asm/cacheflush.h>
+#include <asm/ptrace.h>
+
+#include "bpf_jit_64.h"
+
+int bpf_jit_enable __read_mostly;
+
+static inline bool is_simm13(unsigned int value)
+{
+ return value + 0x1000 < 0x2000;
+}
+
+static inline bool is_simm10(unsigned int value)
+{
+ return value + 0x200 < 0x400;
+}
+
+static inline bool is_simm5(unsigned int value)
+{
+ return value + 0x10 < 0x20;
+}
+
+static inline bool is_sethi(unsigned int value)
+{
+ return (value & ~0x3fffff) == 0;
+}
+
+static void bpf_flush_icache(void *start_, void *end_)
+{
+ /* Cheetah's I-cache is fully coherent. */
+ if (tlb_type == spitfire) {
+ unsigned long start = (unsigned long) start_;
+ unsigned long end = (unsigned long) end_;
+
+ start &= ~7UL;
+ end = (end + 7UL) & ~7UL;
+ while (start < end) {
+ flushi(start);
+ start += 32;
+ }
+ }
+}
+
+#define SEEN_DATAREF 1 /* might call external helpers */
+#define SEEN_XREG 2 /* ebx is used */
+#define SEEN_MEM 4 /* use mem[] for temporary storage */
+
+#define S13(X) ((X) & 0x1fff)
+#define S5(X) ((X) & 0x1f)
+#define IMMED 0x00002000
+#define RD(X) ((X) << 25)
+#define RS1(X) ((X) << 14)
+#define RS2(X) ((X))
+#define OP(X) ((X) << 30)
+#define OP2(X) ((X) << 22)
+#define OP3(X) ((X) << 19)
+#define COND(X) (((X) & 0xf) << 25)
+#define CBCOND(X) (((X) & 0x1f) << 25)
+#define F1(X) OP(X)
+#define F2(X, Y) (OP(X) | OP2(Y))
+#define F3(X, Y) (OP(X) | OP3(Y))
+#define ASI(X) (((X) & 0xff) << 5)
+
+#define CONDN COND(0x0)
+#define CONDE COND(0x1)
+#define CONDLE COND(0x2)
+#define CONDL COND(0x3)
+#define CONDLEU COND(0x4)
+#define CONDCS COND(0x5)
+#define CONDNEG COND(0x6)
+#define CONDVC COND(0x7)
+#define CONDA COND(0x8)
+#define CONDNE COND(0x9)
+#define CONDG COND(0xa)
+#define CONDGE COND(0xb)
+#define CONDGU COND(0xc)
+#define CONDCC COND(0xd)
+#define CONDPOS COND(0xe)
+#define CONDVS COND(0xf)
+
+#define CONDGEU CONDCC
+#define CONDLU CONDCS
+
+#define WDISP22(X) (((X) >> 2) & 0x3fffff)
+#define WDISP19(X) (((X) >> 2) & 0x7ffff)
+
+/* The 10-bit branch displacement for CBCOND is split into two fields */
+static u32 WDISP10(u32 off)
+{
+ u32 ret = ((off >> 2) & 0xff) << 5;
+
+ ret |= ((off >> (2 + 8)) & 0x03) << 19;
+
+ return ret;
+}
+
+#define CBCONDE CBCOND(0x09)
+#define CBCONDLE CBCOND(0x0a)
+#define CBCONDL CBCOND(0x0b)
+#define CBCONDLEU CBCOND(0x0c)
+#define CBCONDCS CBCOND(0x0d)
+#define CBCONDN CBCOND(0x0e)
+#define CBCONDVS CBCOND(0x0f)
+#define CBCONDNE CBCOND(0x19)
+#define CBCONDG CBCOND(0x1a)
+#define CBCONDGE CBCOND(0x1b)
+#define CBCONDGU CBCOND(0x1c)
+#define CBCONDCC CBCOND(0x1d)
+#define CBCONDPOS CBCOND(0x1e)
+#define CBCONDVC CBCOND(0x1f)
+
+#define CBCONDGEU CBCONDCC
+#define CBCONDLU CBCONDCS
+
+#define ANNUL (1 << 29)
+#define XCC (1 << 21)
+
+#define BRANCH (F2(0, 1) | XCC)
+#define CBCOND_OP (F2(0, 3) | XCC)
+
+#define BA (BRANCH | CONDA)
+#define BG (BRANCH | CONDG)
+#define BGU (BRANCH | CONDGU)
+#define BLEU (BRANCH | CONDLEU)
+#define BGE (BRANCH | CONDGE)
+#define BGEU (BRANCH | CONDGEU)
+#define BLU (BRANCH | CONDLU)
+#define BE (BRANCH | CONDE)
+#define BNE (BRANCH | CONDNE)
+
+#define SETHI(K, REG) \
+ (F2(0, 0x4) | RD(REG) | (((K) >> 10) & 0x3fffff))
+#define OR_LO(K, REG) \
+ (F3(2, 0x02) | IMMED | RS1(REG) | ((K) & 0x3ff) | RD(REG))
+
+#define ADD F3(2, 0x00)
+#define AND F3(2, 0x01)
+#define ANDCC F3(2, 0x11)
+#define OR F3(2, 0x02)
+#define XOR F3(2, 0x03)
+#define SUB F3(2, 0x04)
+#define SUBCC F3(2, 0x14)
+#define MUL F3(2, 0x0a)
+#define MULX F3(2, 0x09)
+#define UDIVX F3(2, 0x0d)
+#define DIV F3(2, 0x0e)
+#define SLL F3(2, 0x25)
+#define SLLX (F3(2, 0x25)|(1<<12))
+#define SRA F3(2, 0x27)
+#define SRAX (F3(2, 0x27)|(1<<12))
+#define SRL F3(2, 0x26)
+#define SRLX (F3(2, 0x26)|(1<<12))
+#define JMPL F3(2, 0x38)
+#define SAVE F3(2, 0x3c)
+#define RESTORE F3(2, 0x3d)
+#define CALL F1(1)
+#define BR F2(0, 0x01)
+#define RD_Y F3(2, 0x28)
+#define WR_Y F3(2, 0x30)
+
+#define LD32 F3(3, 0x00)
+#define LD8 F3(3, 0x01)
+#define LD16 F3(3, 0x02)
+#define LD64 F3(3, 0x0b)
+#define LD64A F3(3, 0x1b)
+#define ST8 F3(3, 0x05)
+#define ST16 F3(3, 0x06)
+#define ST32 F3(3, 0x04)
+#define ST64 F3(3, 0x0e)
+
+#define CAS F3(3, 0x3c)
+#define CASX F3(3, 0x3e)
+
+#define LDPTR LD64
+#define BASE_STACKFRAME 176
+
+#define LD32I (LD32 | IMMED)
+#define LD8I (LD8 | IMMED)
+#define LD16I (LD16 | IMMED)
+#define LD64I (LD64 | IMMED)
+#define LDPTRI (LDPTR | IMMED)
+#define ST32I (ST32 | IMMED)
+
+struct jit_ctx {
+ struct bpf_prog *prog;
+ unsigned int *offset;
+ int idx;
+ int epilogue_offset;
+ bool tmp_1_used;
+ bool tmp_2_used;
+ bool tmp_3_used;
+ bool saw_ld_abs_ind;
+ bool saw_frame_pointer;
+ bool saw_call;
+ bool saw_tail_call;
+ u32 *image;
+};
+
+#define TMP_REG_1 (MAX_BPF_JIT_REG + 0)
+#define TMP_REG_2 (MAX_BPF_JIT_REG + 1)
+#define SKB_HLEN_REG (MAX_BPF_JIT_REG + 2)
+#define SKB_DATA_REG (MAX_BPF_JIT_REG + 3)
+#define TMP_REG_3 (MAX_BPF_JIT_REG + 4)
+
+/* Map BPF registers to SPARC registers */
+static const int bpf2sparc[] = {
+ /* return value from in-kernel function, and exit value from eBPF */
+ [BPF_REG_0] = O5,
+
+ /* arguments from eBPF program to in-kernel function */
+ [BPF_REG_1] = O0,
+ [BPF_REG_2] = O1,
+ [BPF_REG_3] = O2,
+ [BPF_REG_4] = O3,
+ [BPF_REG_5] = O4,
+
+ /* callee saved registers that in-kernel function will preserve */
+ [BPF_REG_6] = L0,
+ [BPF_REG_7] = L1,
+ [BPF_REG_8] = L2,
+ [BPF_REG_9] = L3,
+
+ /* read-only frame pointer to access stack */
+ [BPF_REG_FP] = L6,
+
+ [BPF_REG_AX] = G7,
+
+ /* temporary register for internal BPF JIT */
+ [TMP_REG_1] = G1,
+ [TMP_REG_2] = G2,
+ [TMP_REG_3] = G3,
+
+ [SKB_HLEN_REG] = L4,
+ [SKB_DATA_REG] = L5,
+};
+
+static void emit(const u32 insn, struct jit_ctx *ctx)
+{
+ if (ctx->image != NULL)
+ ctx->image[ctx->idx] = insn;
+
+ ctx->idx++;
+}
+
+static void emit_call(u32 *func, struct jit_ctx *ctx)
+{
+ if (ctx->image != NULL) {
+ void *here = &ctx->image[ctx->idx];
+ unsigned int off;
+
+ off = (void *)func - here;
+ ctx->image[ctx->idx] = CALL | ((off >> 2) & 0x3fffffff);
+ }
+ ctx->idx++;
+}
+
+static void emit_nop(struct jit_ctx *ctx)
+{
+ emit(SETHI(0, G0), ctx);
+}
+
+static void emit_reg_move(u32 from, u32 to, struct jit_ctx *ctx)
+{
+ emit(OR | RS1(G0) | RS2(from) | RD(to), ctx);
+}
+
+/* Emit 32-bit constant, zero extended. */
+static void emit_set_const(s32 K, u32 reg, struct jit_ctx *ctx)
+{
+ emit(SETHI(K, reg), ctx);
+ emit(OR_LO(K, reg), ctx);
+}
+
+/* Emit 32-bit constant, sign extended. */
+static void emit_set_const_sext(s32 K, u32 reg, struct jit_ctx *ctx)
+{
+ if (K >= 0) {
+ emit(SETHI(K, reg), ctx);
+ emit(OR_LO(K, reg), ctx);
+ } else {
+ u32 hbits = ~(u32) K;
+ u32 lbits = -0x400 | (u32) K;
+
+ emit(SETHI(hbits, reg), ctx);
+ emit(XOR | IMMED | RS1(reg) | S13(lbits) | RD(reg), ctx);
+ }
+}
+
+static void emit_alu(u32 opcode, u32 src, u32 dst, struct jit_ctx *ctx)
+{
+ emit(opcode | RS1(dst) | RS2(src) | RD(dst), ctx);
+}
+
+static void emit_alu3(u32 opcode, u32 a, u32 b, u32 c, struct jit_ctx *ctx)
+{
+ emit(opcode | RS1(a) | RS2(b) | RD(c), ctx);
+}
+
+static void emit_alu_K(unsigned int opcode, unsigned int dst, unsigned int imm,
+ struct jit_ctx *ctx)
+{
+ bool small_immed = is_simm13(imm);
+ unsigned int insn = opcode;
+
+ insn |= RS1(dst) | RD(dst);
+ if (small_immed) {
+ emit(insn | IMMED | S13(imm), ctx);
+ } else {
+ unsigned int tmp = bpf2sparc[TMP_REG_1];
+
+ ctx->tmp_1_used = true;
+
+ emit_set_const_sext(imm, tmp, ctx);
+ emit(insn | RS2(tmp), ctx);
+ }
+}
+
+static void emit_alu3_K(unsigned int opcode, unsigned int src, unsigned int imm,
+ unsigned int dst, struct jit_ctx *ctx)
+{
+ bool small_immed = is_simm13(imm);
+ unsigned int insn = opcode;
+
+ insn |= RS1(src) | RD(dst);
+ if (small_immed) {
+ emit(insn | IMMED | S13(imm), ctx);
+ } else {
+ unsigned int tmp = bpf2sparc[TMP_REG_1];
+
+ ctx->tmp_1_used = true;
+
+ emit_set_const_sext(imm, tmp, ctx);
+ emit(insn | RS2(tmp), ctx);
+ }
+}
+
+static void emit_loadimm32(s32 K, unsigned int dest, struct jit_ctx *ctx)
+{
+ if (K >= 0 && is_simm13(K)) {
+ /* or %g0, K, DEST */
+ emit(OR | IMMED | RS1(G0) | S13(K) | RD(dest), ctx);
+ } else {
+ emit_set_const(K, dest, ctx);
+ }
+}
+
+static void emit_loadimm(s32 K, unsigned int dest, struct jit_ctx *ctx)
+{
+ if (is_simm13(K)) {
+ /* or %g0, K, DEST */
+ emit(OR | IMMED | RS1(G0) | S13(K) | RD(dest), ctx);
+ } else {
+ emit_set_const(K, dest, ctx);
+ }
+}
+
+static void emit_loadimm_sext(s32 K, unsigned int dest, struct jit_ctx *ctx)
+{
+ if (is_simm13(K)) {
+ /* or %g0, K, DEST */
+ emit(OR | IMMED | RS1(G0) | S13(K) | RD(dest), ctx);
+ } else {
+ emit_set_const_sext(K, dest, ctx);
+ }
+}
+
+static void analyze_64bit_constant(u32 high_bits, u32 low_bits,
+ int *hbsp, int *lbsp, int *abbasp)
+{
+ int lowest_bit_set, highest_bit_set, all_bits_between_are_set;
+ int i;
+
+ lowest_bit_set = highest_bit_set = -1;
+ i = 0;
+ do {
+ if ((lowest_bit_set == -1) && ((low_bits >> i) & 1))
+ lowest_bit_set = i;
+ if ((highest_bit_set == -1) && ((high_bits >> (32 - i - 1)) & 1))
+ highest_bit_set = (64 - i - 1);
+ } while (++i < 32 && (highest_bit_set == -1 ||
+ lowest_bit_set == -1));
+ if (i == 32) {
+ i = 0;
+ do {
+ if (lowest_bit_set == -1 && ((high_bits >> i) & 1))
+ lowest_bit_set = i + 32;
+ if (highest_bit_set == -1 &&
+ ((low_bits >> (32 - i - 1)) & 1))
+ highest_bit_set = 32 - i - 1;
+ } while (++i < 32 && (highest_bit_set == -1 ||
+ lowest_bit_set == -1));
+ }
+
+ all_bits_between_are_set = 1;
+ for (i = lowest_bit_set; i <= highest_bit_set; i++) {
+ if (i < 32) {
+ if ((low_bits & (1 << i)) != 0)
+ continue;
+ } else {
+ if ((high_bits & (1 << (i - 32))) != 0)
+ continue;
+ }
+ all_bits_between_are_set = 0;
+ break;
+ }
+ *hbsp = highest_bit_set;
+ *lbsp = lowest_bit_set;
+ *abbasp = all_bits_between_are_set;
+}
+
+static unsigned long create_simple_focus_bits(unsigned long high_bits,
+ unsigned long low_bits,
+ int lowest_bit_set, int shift)
+{
+ long hi, lo;
+
+ if (lowest_bit_set < 32) {
+ lo = (low_bits >> lowest_bit_set) << shift;
+ hi = ((high_bits << (32 - lowest_bit_set)) << shift);
+ } else {
+ lo = 0;
+ hi = ((high_bits >> (lowest_bit_set - 32)) << shift);
+ }
+ return hi | lo;
+}
+
+static bool const64_is_2insns(unsigned long high_bits,
+ unsigned long low_bits)
+{
+ int highest_bit_set, lowest_bit_set, all_bits_between_are_set;
+
+ if (high_bits == 0 || high_bits == 0xffffffff)
+ return true;
+
+ analyze_64bit_constant(high_bits, low_bits,
+ &highest_bit_set, &lowest_bit_set,
+ &all_bits_between_are_set);
+
+ if ((highest_bit_set == 63 || lowest_bit_set == 0) &&
+ all_bits_between_are_set != 0)
+ return true;
+
+ if (highest_bit_set - lowest_bit_set < 21)
+ return true;
+
+ return false;
+}
+
+static void sparc_emit_set_const64_quick2(unsigned long high_bits,
+ unsigned long low_imm,
+ unsigned int dest,
+ int shift_count, struct jit_ctx *ctx)
+{
+ emit_loadimm32(high_bits, dest, ctx);
+
+ /* Now shift it up into place. */
+ emit_alu_K(SLLX, dest, shift_count, ctx);
+
+ /* If there is a low immediate part piece, finish up by
+ * putting that in as well.
+ */
+ if (low_imm != 0)
+ emit(OR | IMMED | RS1(dest) | S13(low_imm) | RD(dest), ctx);
+}
+
+static void emit_loadimm64(u64 K, unsigned int dest, struct jit_ctx *ctx)
+{
+ int all_bits_between_are_set, lowest_bit_set, highest_bit_set;
+ unsigned int tmp = bpf2sparc[TMP_REG_1];
+ u32 low_bits = (K & 0xffffffff);
+ u32 high_bits = (K >> 32);
+
+ /* These two tests also take care of all of the one
+ * instruction cases.
+ */
+ if (high_bits == 0xffffffff && (low_bits & 0x80000000))
+ return emit_loadimm_sext(K, dest, ctx);
+ if (high_bits == 0x00000000)
+ return emit_loadimm32(K, dest, ctx);
+
+ analyze_64bit_constant(high_bits, low_bits, &highest_bit_set,
+ &lowest_bit_set, &all_bits_between_are_set);
+
+ /* 1) mov -1, %reg
+ * sllx %reg, shift, %reg
+ * 2) mov -1, %reg
+ * srlx %reg, shift, %reg
+ * 3) mov some_small_const, %reg
+ * sllx %reg, shift, %reg
+ */
+ if (((highest_bit_set == 63 || lowest_bit_set == 0) &&
+ all_bits_between_are_set != 0) ||
+ ((highest_bit_set - lowest_bit_set) < 12)) {
+ int shift = lowest_bit_set;
+ long the_const = -1;
+
+ if ((highest_bit_set != 63 && lowest_bit_set != 0) ||
+ all_bits_between_are_set == 0) {
+ the_const =
+ create_simple_focus_bits(high_bits, low_bits,
+ lowest_bit_set, 0);
+ } else if (lowest_bit_set == 0)
+ shift = -(63 - highest_bit_set);
+
+ emit(OR | IMMED | RS1(G0) | S13(the_const) | RD(dest), ctx);
+ if (shift > 0)
+ emit_alu_K(SLLX, dest, shift, ctx);
+ else if (shift < 0)
+ emit_alu_K(SRLX, dest, -shift, ctx);
+
+ return;
+ }
+
+ /* Now a range of 22 or less bits set somewhere.
+ * 1) sethi %hi(focus_bits), %reg
+ * sllx %reg, shift, %reg
+ * 2) sethi %hi(focus_bits), %reg
+ * srlx %reg, shift, %reg
+ */
+ if ((highest_bit_set - lowest_bit_set) < 21) {
+ unsigned long focus_bits =
+ create_simple_focus_bits(high_bits, low_bits,
+ lowest_bit_set, 10);
+
+ emit(SETHI(focus_bits, dest), ctx);
+
+ /* If lowest_bit_set == 10 then a sethi alone could
+ * have done it.
+ */
+ if (lowest_bit_set < 10)
+ emit_alu_K(SRLX, dest, 10 - lowest_bit_set, ctx);
+ else if (lowest_bit_set > 10)
+ emit_alu_K(SLLX, dest, lowest_bit_set - 10, ctx);
+ return;
+ }
+
+ /* Ok, now 3 instruction sequences. */
+ if (low_bits == 0) {
+ emit_loadimm32(high_bits, dest, ctx);
+ emit_alu_K(SLLX, dest, 32, ctx);
+ return;
+ }
+
+ /* We may be able to do something quick
+ * when the constant is negated, so try that.
+ */
+ if (const64_is_2insns((~high_bits) & 0xffffffff,
+ (~low_bits) & 0xfffffc00)) {
+ /* NOTE: The trailing bits get XOR'd so we need the
+ * non-negated bits, not the negated ones.
+ */
+ unsigned long trailing_bits = low_bits & 0x3ff;
+
+ if ((((~high_bits) & 0xffffffff) == 0 &&
+ ((~low_bits) & 0x80000000) == 0) ||
+ (((~high_bits) & 0xffffffff) == 0xffffffff &&
+ ((~low_bits) & 0x80000000) != 0)) {
+ unsigned long fast_int = (~low_bits & 0xffffffff);
+
+ if ((is_sethi(fast_int) &&
+ (~high_bits & 0xffffffff) == 0)) {
+ emit(SETHI(fast_int, dest), ctx);
+ } else if (is_simm13(fast_int)) {
+ emit(OR | IMMED | RS1(G0) | S13(fast_int) | RD(dest), ctx);
+ } else {
+ emit_loadimm64(fast_int, dest, ctx);
+ }
+ } else {
+ u64 n = ((~low_bits) & 0xfffffc00) |
+ (((unsigned long)((~high_bits) & 0xffffffff))<<32);
+ emit_loadimm64(n, dest, ctx);
+ }
+
+ low_bits = -0x400 | trailing_bits;
+
+ emit(XOR | IMMED | RS1(dest) | S13(low_bits) | RD(dest), ctx);
+ return;
+ }
+
+ /* 1) sethi %hi(xxx), %reg
+ * or %reg, %lo(xxx), %reg
+ * sllx %reg, yyy, %reg
+ */
+ if ((highest_bit_set - lowest_bit_set) < 32) {
+ unsigned long focus_bits =
+ create_simple_focus_bits(high_bits, low_bits,
+ lowest_bit_set, 0);
+
+ /* So what we know is that the set bits straddle the
+ * middle of the 64-bit word.
+ */
+ sparc_emit_set_const64_quick2(focus_bits, 0, dest,
+ lowest_bit_set, ctx);
+ return;
+ }
+
+ /* 1) sethi %hi(high_bits), %reg
+ * or %reg, %lo(high_bits), %reg
+ * sllx %reg, 32, %reg
+ * or %reg, low_bits, %reg
+ */
+ if (is_simm13(low_bits) && ((int)low_bits > 0)) {
+ sparc_emit_set_const64_quick2(high_bits, low_bits,
+ dest, 32, ctx);
+ return;
+ }
+
+ /* Oh well, we tried... Do a full 64-bit decomposition. */
+ ctx->tmp_1_used = true;
+
+ emit_loadimm32(high_bits, tmp, ctx);
+ emit_loadimm32(low_bits, dest, ctx);
+ emit_alu_K(SLLX, tmp, 32, ctx);
+ emit(OR | RS1(dest) | RS2(tmp) | RD(dest), ctx);
+}
+
+static void emit_branch(unsigned int br_opc, unsigned int from_idx, unsigned int to_idx,
+ struct jit_ctx *ctx)
+{
+ unsigned int off = to_idx - from_idx;
+
+ if (br_opc & XCC)
+ emit(br_opc | WDISP19(off << 2), ctx);
+ else
+ emit(br_opc | WDISP22(off << 2), ctx);
+}
+
+static void emit_cbcond(unsigned int cb_opc, unsigned int from_idx, unsigned int to_idx,
+ const u8 dst, const u8 src, struct jit_ctx *ctx)
+{
+ unsigned int off = to_idx - from_idx;
+
+ emit(cb_opc | WDISP10(off << 2) | RS1(dst) | RS2(src), ctx);
+}
+
+static void emit_cbcondi(unsigned int cb_opc, unsigned int from_idx, unsigned int to_idx,
+ const u8 dst, s32 imm, struct jit_ctx *ctx)
+{
+ unsigned int off = to_idx - from_idx;
+
+ emit(cb_opc | IMMED | WDISP10(off << 2) | RS1(dst) | S5(imm), ctx);
+}
+
+#define emit_read_y(REG, CTX) emit(RD_Y | RD(REG), CTX)
+#define emit_write_y(REG, CTX) emit(WR_Y | IMMED | RS1(REG) | S13(0), CTX)
+
+#define emit_cmp(R1, R2, CTX) \
+ emit(SUBCC | RS1(R1) | RS2(R2) | RD(G0), CTX)
+
+#define emit_cmpi(R1, IMM, CTX) \
+ emit(SUBCC | IMMED | RS1(R1) | S13(IMM) | RD(G0), CTX)
+
+#define emit_btst(R1, R2, CTX) \
+ emit(ANDCC | RS1(R1) | RS2(R2) | RD(G0), CTX)
+
+#define emit_btsti(R1, IMM, CTX) \
+ emit(ANDCC | IMMED | RS1(R1) | S13(IMM) | RD(G0), CTX)
+
+static int emit_compare_and_branch(const u8 code, const u8 dst, u8 src,
+ const s32 imm, bool is_imm, int branch_dst,
+ struct jit_ctx *ctx)
+{
+ bool use_cbcond = (sparc64_elf_hwcap & AV_SPARC_CBCOND) != 0;
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+
+ branch_dst = ctx->offset[branch_dst];
+
+ if (!is_simm10(branch_dst - ctx->idx) ||
+ BPF_OP(code) == BPF_JSET)
+ use_cbcond = false;
+
+ if (is_imm) {
+ bool fits = true;
+
+ if (use_cbcond) {
+ if (!is_simm5(imm))
+ fits = false;
+ } else if (!is_simm13(imm)) {
+ fits = false;
+ }
+ if (!fits) {
+ ctx->tmp_1_used = true;
+ emit_loadimm_sext(imm, tmp, ctx);
+ src = tmp;
+ is_imm = false;
+ }
+ }
+
+ if (!use_cbcond) {
+ u32 br_opcode;
+
+ if (BPF_OP(code) == BPF_JSET) {
+ if (is_imm)
+ emit_btsti(dst, imm, ctx);
+ else
+ emit_btst(dst, src, ctx);
+ } else {
+ if (is_imm)
+ emit_cmpi(dst, imm, ctx);
+ else
+ emit_cmp(dst, src, ctx);
+ }
+ switch (BPF_OP(code)) {
+ case BPF_JEQ:
+ br_opcode = BE;
+ break;
+ case BPF_JGT:
+ br_opcode = BGU;
+ break;
+ case BPF_JGE:
+ br_opcode = BGEU;
+ break;
+ case BPF_JSET:
+ case BPF_JNE:
+ br_opcode = BNE;
+ break;
+ case BPF_JSGT:
+ br_opcode = BG;
+ break;
+ case BPF_JSGE:
+ br_opcode = BGE;
+ break;
+ default:
+ /* Make sure we dont leak kernel information to the
+ * user.
+ */
+ return -EFAULT;
+ }
+ emit_branch(br_opcode, ctx->idx, branch_dst, ctx);
+ emit_nop(ctx);
+ } else {
+ u32 cbcond_opcode;
+
+ switch (BPF_OP(code)) {
+ case BPF_JEQ:
+ cbcond_opcode = CBCONDE;
+ break;
+ case BPF_JGT:
+ cbcond_opcode = CBCONDGU;
+ break;
+ case BPF_JGE:
+ cbcond_opcode = CBCONDGEU;
+ break;
+ case BPF_JNE:
+ cbcond_opcode = CBCONDNE;
+ break;
+ case BPF_JSGT:
+ cbcond_opcode = CBCONDG;
+ break;
+ case BPF_JSGE:
+ cbcond_opcode = CBCONDGE;
+ break;
+ default:
+ /* Make sure we dont leak kernel information to the
+ * user.
+ */
+ return -EFAULT;
+ }
+ cbcond_opcode |= CBCOND_OP;
+ if (is_imm)
+ emit_cbcondi(cbcond_opcode, ctx->idx, branch_dst,
+ dst, imm, ctx);
+ else
+ emit_cbcond(cbcond_opcode, ctx->idx, branch_dst,
+ dst, src, ctx);
+ }
+ return 0;
+}
+
+static void load_skb_regs(struct jit_ctx *ctx, u8 r_skb)
+{
+ const u8 r_headlen = bpf2sparc[SKB_HLEN_REG];
+ const u8 r_data = bpf2sparc[SKB_DATA_REG];
+ const u8 r_tmp = bpf2sparc[TMP_REG_1];
+ unsigned int off;
+
+ off = offsetof(struct sk_buff, len);
+ emit(LD32I | RS1(r_skb) | S13(off) | RD(r_headlen), ctx);
+
+ off = offsetof(struct sk_buff, data_len);
+ emit(LD32I | RS1(r_skb) | S13(off) | RD(r_tmp), ctx);
+
+ emit(SUB | RS1(r_headlen) | RS2(r_tmp) | RD(r_headlen), ctx);
+
+ off = offsetof(struct sk_buff, data);
+ emit(LDPTRI | RS1(r_skb) | S13(off) | RD(r_data), ctx);
+}
+
+/* Just skip the save instruction and the ctx register move. */
+#define BPF_TAILCALL_PROLOGUE_SKIP 16
+#define BPF_TAILCALL_CNT_SP_OFF (STACK_BIAS + 128)
+
+static void build_prologue(struct jit_ctx *ctx)
+{
+ s32 stack_needed = BASE_STACKFRAME;
+
+ if (ctx->saw_frame_pointer || ctx->saw_tail_call)
+ stack_needed += MAX_BPF_STACK;
+
+ if (ctx->saw_tail_call)
+ stack_needed += 8;
+
+ /* save %sp, -176, %sp */
+ emit(SAVE | IMMED | RS1(SP) | S13(-stack_needed) | RD(SP), ctx);
+
+ /* tail_call_cnt = 0 */
+ if (ctx->saw_tail_call) {
+ u32 off = BPF_TAILCALL_CNT_SP_OFF;
+
+ emit(ST32 | IMMED | RS1(SP) | S13(off) | RD(G0), ctx);
+ } else {
+ emit_nop(ctx);
+ }
+ if (ctx->saw_frame_pointer) {
+ const u8 vfp = bpf2sparc[BPF_REG_FP];
+
+ emit(ADD | IMMED | RS1(FP) | S13(STACK_BIAS) | RD(vfp), ctx);
+ }
+
+ emit_reg_move(I0, O0, ctx);
+ /* If you add anything here, adjust BPF_TAILCALL_PROLOGUE_SKIP above. */
+
+ if (ctx->saw_ld_abs_ind)
+ load_skb_regs(ctx, bpf2sparc[BPF_REG_1]);
+}
+
+static void build_epilogue(struct jit_ctx *ctx)
+{
+ ctx->epilogue_offset = ctx->idx;
+
+ /* ret (jmpl %i7 + 8, %g0) */
+ emit(JMPL | IMMED | RS1(I7) | S13(8) | RD(G0), ctx);
+
+ /* restore %i5, %g0, %o0 */
+ emit(RESTORE | RS1(bpf2sparc[BPF_REG_0]) | RS2(G0) | RD(O0), ctx);
+}
+
+static void emit_tail_call(struct jit_ctx *ctx)
+{
+ const u8 bpf_array = bpf2sparc[BPF_REG_2];
+ const u8 bpf_index = bpf2sparc[BPF_REG_3];
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+ u32 off;
+
+ ctx->saw_tail_call = true;
+
+ off = offsetof(struct bpf_array, map.max_entries);
+ emit(LD32 | IMMED | RS1(bpf_array) | S13(off) | RD(tmp), ctx);
+ emit_cmp(bpf_index, tmp, ctx);
+#define OFFSET1 17
+ emit_branch(BGEU, ctx->idx, ctx->idx + OFFSET1, ctx);
+ emit_nop(ctx);
+
+ off = BPF_TAILCALL_CNT_SP_OFF;
+ emit(LD32 | IMMED | RS1(SP) | S13(off) | RD(tmp), ctx);
+ emit_cmpi(tmp, MAX_TAIL_CALL_CNT, ctx);
+#define OFFSET2 13
+ emit_branch(BGU, ctx->idx, ctx->idx + OFFSET2, ctx);
+ emit_nop(ctx);
+
+ emit_alu_K(ADD, tmp, 1, ctx);
+ off = BPF_TAILCALL_CNT_SP_OFF;
+ emit(ST32 | IMMED | RS1(SP) | S13(off) | RD(tmp), ctx);
+
+ emit_alu3_K(SLL, bpf_index, 3, tmp, ctx);
+ emit_alu(ADD, bpf_array, tmp, ctx);
+ off = offsetof(struct bpf_array, ptrs);
+ emit(LD64 | IMMED | RS1(tmp) | S13(off) | RD(tmp), ctx);
+
+ emit_cmpi(tmp, 0, ctx);
+#define OFFSET3 5
+ emit_branch(BE, ctx->idx, ctx->idx + OFFSET3, ctx);
+ emit_nop(ctx);
+
+ off = offsetof(struct bpf_prog, bpf_func);
+ emit(LD64 | IMMED | RS1(tmp) | S13(off) | RD(tmp), ctx);
+
+ off = BPF_TAILCALL_PROLOGUE_SKIP;
+ emit(JMPL | IMMED | RS1(tmp) | S13(off) | RD(G0), ctx);
+ emit_nop(ctx);
+}
+
+static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx)
+{
+ const u8 code = insn->code;
+ const u8 dst = bpf2sparc[insn->dst_reg];
+ const u8 src = bpf2sparc[insn->src_reg];
+ const int i = insn - ctx->prog->insnsi;
+ const s16 off = insn->off;
+ const s32 imm = insn->imm;
+ u32 *func;
+
+ if (insn->src_reg == BPF_REG_FP)
+ ctx->saw_frame_pointer = true;
+
+ switch (code) {
+ /* dst = src */
+ case BPF_ALU | BPF_MOV | BPF_X:
+ emit_alu3_K(SRL, src, 0, dst, ctx);
+ break;
+ case BPF_ALU64 | BPF_MOV | BPF_X:
+ emit_reg_move(src, dst, ctx);
+ break;
+ /* dst = dst OP src */
+ case BPF_ALU | BPF_ADD | BPF_X:
+ case BPF_ALU64 | BPF_ADD | BPF_X:
+ emit_alu(ADD, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_SUB | BPF_X:
+ case BPF_ALU64 | BPF_SUB | BPF_X:
+ emit_alu(SUB, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_AND | BPF_X:
+ case BPF_ALU64 | BPF_AND | BPF_X:
+ emit_alu(AND, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_OR | BPF_X:
+ case BPF_ALU64 | BPF_OR | BPF_X:
+ emit_alu(OR, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_XOR | BPF_X:
+ case BPF_ALU64 | BPF_XOR | BPF_X:
+ emit_alu(XOR, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_MUL | BPF_X:
+ emit_alu(MUL, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_MUL | BPF_X:
+ emit_alu(MULX, src, dst, ctx);
+ break;
+ case BPF_ALU | BPF_DIV | BPF_X:
+ emit_cmp(src, G0, ctx);
+ emit_branch(BE|ANNUL, ctx->idx, ctx->epilogue_offset, ctx);
+ emit_loadimm(0, bpf2sparc[BPF_REG_0], ctx);
+
+ emit_write_y(G0, ctx);
+ emit_alu(DIV, src, dst, ctx);
+ break;
+
+ case BPF_ALU64 | BPF_DIV | BPF_X:
+ emit_cmp(src, G0, ctx);
+ emit_branch(BE|ANNUL, ctx->idx, ctx->epilogue_offset, ctx);
+ emit_loadimm(0, bpf2sparc[BPF_REG_0], ctx);
+
+ emit_alu(UDIVX, src, dst, ctx);
+ break;
+
+ case BPF_ALU | BPF_MOD | BPF_X: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+
+ ctx->tmp_1_used = true;
+
+ emit_cmp(src, G0, ctx);
+ emit_branch(BE|ANNUL, ctx->idx, ctx->epilogue_offset, ctx);
+ emit_loadimm(0, bpf2sparc[BPF_REG_0], ctx);
+
+ emit_write_y(G0, ctx);
+ emit_alu3(DIV, dst, src, tmp, ctx);
+ emit_alu3(MULX, tmp, src, tmp, ctx);
+ emit_alu3(SUB, dst, tmp, dst, ctx);
+ goto do_alu32_trunc;
+ }
+ case BPF_ALU64 | BPF_MOD | BPF_X: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+
+ ctx->tmp_1_used = true;
+
+ emit_cmp(src, G0, ctx);
+ emit_branch(BE|ANNUL, ctx->idx, ctx->epilogue_offset, ctx);
+ emit_loadimm(0, bpf2sparc[BPF_REG_0], ctx);
+
+ emit_alu3(UDIVX, dst, src, tmp, ctx);
+ emit_alu3(MULX, tmp, src, tmp, ctx);
+ emit_alu3(SUB, dst, tmp, dst, ctx);
+ break;
+ }
+ case BPF_ALU | BPF_LSH | BPF_X:
+ emit_alu(SLL, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_LSH | BPF_X:
+ emit_alu(SLLX, src, dst, ctx);
+ break;
+ case BPF_ALU | BPF_RSH | BPF_X:
+ emit_alu(SRL, src, dst, ctx);
+ break;
+ case BPF_ALU64 | BPF_RSH | BPF_X:
+ emit_alu(SRLX, src, dst, ctx);
+ break;
+ case BPF_ALU | BPF_ARSH | BPF_X:
+ emit_alu(SRA, src, dst, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_ARSH | BPF_X:
+ emit_alu(SRAX, src, dst, ctx);
+ break;
+
+ /* dst = -dst */
+ case BPF_ALU | BPF_NEG:
+ case BPF_ALU64 | BPF_NEG:
+ emit(SUB | RS1(0) | RS2(dst) | RD(dst), ctx);
+ goto do_alu32_trunc;
+
+ case BPF_ALU | BPF_END | BPF_FROM_BE:
+ switch (imm) {
+ case 16:
+ emit_alu_K(SLL, dst, 16, ctx);
+ emit_alu_K(SRL, dst, 16, ctx);
+ break;
+ case 32:
+ emit_alu_K(SRL, dst, 0, ctx);
+ break;
+ case 64:
+ /* nop */
+ break;
+
+ }
+ break;
+
+ /* dst = BSWAP##imm(dst) */
+ case BPF_ALU | BPF_END | BPF_FROM_LE: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+ const u8 tmp2 = bpf2sparc[TMP_REG_2];
+
+ ctx->tmp_1_used = true;
+ switch (imm) {
+ case 16:
+ emit_alu3_K(AND, dst, 0xff, tmp, ctx);
+ emit_alu3_K(SRL, dst, 8, dst, ctx);
+ emit_alu3_K(AND, dst, 0xff, dst, ctx);
+ emit_alu3_K(SLL, tmp, 8, tmp, ctx);
+ emit_alu(OR, tmp, dst, ctx);
+ break;
+
+ case 32:
+ ctx->tmp_2_used = true;
+ emit_alu3_K(SRL, dst, 24, tmp, ctx); /* tmp = dst >> 24 */
+ emit_alu3_K(SRL, dst, 16, tmp2, ctx); /* tmp2 = dst >> 16 */
+ emit_alu3_K(AND, tmp2, 0xff, tmp2, ctx);/* tmp2 = tmp2 & 0xff */
+ emit_alu3_K(SLL, tmp2, 8, tmp2, ctx); /* tmp2 = tmp2 << 8 */
+ emit_alu(OR, tmp2, tmp, ctx); /* tmp = tmp | tmp2 */
+ emit_alu3_K(SRL, dst, 8, tmp2, ctx); /* tmp2 = dst >> 8 */
+ emit_alu3_K(AND, tmp2, 0xff, tmp2, ctx);/* tmp2 = tmp2 & 0xff */
+ emit_alu3_K(SLL, tmp2, 16, tmp2, ctx); /* tmp2 = tmp2 << 16 */
+ emit_alu(OR, tmp2, tmp, ctx); /* tmp = tmp | tmp2 */
+ emit_alu3_K(AND, dst, 0xff, dst, ctx); /* dst = dst & 0xff */
+ emit_alu3_K(SLL, dst, 24, dst, ctx); /* dst = dst << 24 */
+ emit_alu(OR, tmp, dst, ctx); /* dst = dst | tmp */
+ break;
+
+ case 64:
+ emit_alu3_K(ADD, SP, STACK_BIAS + 128, tmp, ctx);
+ emit(ST64 | RS1(tmp) | RS2(G0) | RD(dst), ctx);
+ emit(LD64A | ASI(ASI_PL) | RS1(tmp) | RS2(G0) | RD(dst), ctx);
+ break;
+ }
+ break;
+ }
+ /* dst = imm */
+ case BPF_ALU | BPF_MOV | BPF_K:
+ emit_loadimm32(imm, dst, ctx);
+ break;
+ case BPF_ALU64 | BPF_MOV | BPF_K:
+ emit_loadimm_sext(imm, dst, ctx);
+ break;
+ /* dst = dst OP imm */
+ case BPF_ALU | BPF_ADD | BPF_K:
+ case BPF_ALU64 | BPF_ADD | BPF_K:
+ emit_alu_K(ADD, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_SUB | BPF_K:
+ case BPF_ALU64 | BPF_SUB | BPF_K:
+ emit_alu_K(SUB, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_AND | BPF_K:
+ case BPF_ALU64 | BPF_AND | BPF_K:
+ emit_alu_K(AND, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_OR | BPF_K:
+ case BPF_ALU64 | BPF_OR | BPF_K:
+ emit_alu_K(OR, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_XOR | BPF_K:
+ case BPF_ALU64 | BPF_XOR | BPF_K:
+ emit_alu_K(XOR, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU | BPF_MUL | BPF_K:
+ emit_alu_K(MUL, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_MUL | BPF_K:
+ emit_alu_K(MULX, dst, imm, ctx);
+ break;
+ case BPF_ALU | BPF_DIV | BPF_K:
+ if (imm == 0)
+ return -EINVAL;
+
+ emit_write_y(G0, ctx);
+ emit_alu_K(DIV, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_DIV | BPF_K:
+ if (imm == 0)
+ return -EINVAL;
+
+ emit_alu_K(UDIVX, dst, imm, ctx);
+ break;
+ case BPF_ALU64 | BPF_MOD | BPF_K:
+ case BPF_ALU | BPF_MOD | BPF_K: {
+ const u8 tmp = bpf2sparc[TMP_REG_2];
+ unsigned int div;
+
+ if (imm == 0)
+ return -EINVAL;
+
+ div = (BPF_CLASS(code) == BPF_ALU64) ? UDIVX : DIV;
+
+ ctx->tmp_2_used = true;
+
+ if (BPF_CLASS(code) != BPF_ALU64)
+ emit_write_y(G0, ctx);
+ if (is_simm13(imm)) {
+ emit(div | IMMED | RS1(dst) | S13(imm) | RD(tmp), ctx);
+ emit(MULX | IMMED | RS1(tmp) | S13(imm) | RD(tmp), ctx);
+ emit(SUB | RS1(dst) | RS2(tmp) | RD(dst), ctx);
+ } else {
+ const u8 tmp1 = bpf2sparc[TMP_REG_1];
+
+ ctx->tmp_1_used = true;
+
+ emit_set_const_sext(imm, tmp1, ctx);
+ emit(div | RS1(dst) | RS2(tmp1) | RD(tmp), ctx);
+ emit(MULX | RS1(tmp) | RS2(tmp1) | RD(tmp), ctx);
+ emit(SUB | RS1(dst) | RS2(tmp) | RD(dst), ctx);
+ }
+ goto do_alu32_trunc;
+ }
+ case BPF_ALU | BPF_LSH | BPF_K:
+ emit_alu_K(SLL, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_LSH | BPF_K:
+ emit_alu_K(SLLX, dst, imm, ctx);
+ break;
+ case BPF_ALU | BPF_RSH | BPF_K:
+ emit_alu_K(SRL, dst, imm, ctx);
+ break;
+ case BPF_ALU64 | BPF_RSH | BPF_K:
+ emit_alu_K(SRLX, dst, imm, ctx);
+ break;
+ case BPF_ALU | BPF_ARSH | BPF_K:
+ emit_alu_K(SRA, dst, imm, ctx);
+ goto do_alu32_trunc;
+ case BPF_ALU64 | BPF_ARSH | BPF_K:
+ emit_alu_K(SRAX, dst, imm, ctx);
+ break;
+
+ do_alu32_trunc:
+ if (BPF_CLASS(code) == BPF_ALU)
+ emit_alu_K(SRL, dst, 0, ctx);
+ break;
+
+ /* JUMP off */
+ case BPF_JMP | BPF_JA:
+ emit_branch(BA, ctx->idx, ctx->offset[i + off], ctx);
+ emit_nop(ctx);
+ break;
+ /* IF (dst COND src) JUMP off */
+ case BPF_JMP | BPF_JEQ | BPF_X:
+ case BPF_JMP | BPF_JGT | BPF_X:
+ case BPF_JMP | BPF_JGE | BPF_X:
+ case BPF_JMP | BPF_JNE | BPF_X:
+ case BPF_JMP | BPF_JSGT | BPF_X:
+ case BPF_JMP | BPF_JSGE | BPF_X:
+ case BPF_JMP | BPF_JSET | BPF_X: {
+ int err;
+
+ err = emit_compare_and_branch(code, dst, src, 0, false, i + off, ctx);
+ if (err)
+ return err;
+ break;
+ }
+ /* IF (dst COND imm) JUMP off */
+ case BPF_JMP | BPF_JEQ | BPF_K:
+ case BPF_JMP | BPF_JGT | BPF_K:
+ case BPF_JMP | BPF_JGE | BPF_K:
+ case BPF_JMP | BPF_JNE | BPF_K:
+ case BPF_JMP | BPF_JSGT | BPF_K:
+ case BPF_JMP | BPF_JSGE | BPF_K:
+ case BPF_JMP | BPF_JSET | BPF_K: {
+ int err;
+
+ err = emit_compare_and_branch(code, dst, 0, imm, true, i + off, ctx);
+ if (err)
+ return err;
+ break;
+ }
+
+ /* function call */
+ case BPF_JMP | BPF_CALL:
+ {
+ u8 *func = ((u8 *)__bpf_call_base) + imm;
+
+ ctx->saw_call = true;
+
+ emit_call((u32 *)func, ctx);
+ emit_nop(ctx);
+
+ emit_reg_move(O0, bpf2sparc[BPF_REG_0], ctx);
+
+ if (bpf_helper_changes_pkt_data(func) && ctx->saw_ld_abs_ind)
+ load_skb_regs(ctx, bpf2sparc[BPF_REG_6]);
+ break;
+ }
+
+ /* tail call */
+ case BPF_JMP | BPF_CALL |BPF_X:
+ emit_tail_call(ctx);
+ break;
+
+ /* function return */
+ case BPF_JMP | BPF_EXIT:
+ /* Optimization: when last instruction is EXIT,
+ simply fallthrough to epilogue. */
+ if (i == ctx->prog->len - 1)
+ break;
+ emit_branch(BA, ctx->idx, ctx->epilogue_offset, ctx);
+ emit_nop(ctx);
+ break;
+
+ /* dst = imm64 */
+ case BPF_LD | BPF_IMM | BPF_DW:
+ {
+ const struct bpf_insn insn1 = insn[1];
+ u64 imm64;
+
+ imm64 = (u64)insn1.imm << 32 | (u32)imm;
+ emit_loadimm64(imm64, dst, ctx);
+
+ return 1;
+ }
+
+ /* LDX: dst = *(size *)(src + off) */
+ case BPF_LDX | BPF_MEM | BPF_W:
+ case BPF_LDX | BPF_MEM | BPF_H:
+ case BPF_LDX | BPF_MEM | BPF_B:
+ case BPF_LDX | BPF_MEM | BPF_DW: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+ u32 opcode = 0, rs2;
+
+ ctx->tmp_1_used = true;
+ switch (BPF_SIZE(code)) {
+ case BPF_W:
+ opcode = LD32;
+ break;
+ case BPF_H:
+ opcode = LD16;
+ break;
+ case BPF_B:
+ opcode = LD8;
+ break;
+ case BPF_DW:
+ opcode = LD64;
+ break;
+ }
+
+ if (is_simm13(off)) {
+ opcode |= IMMED;
+ rs2 = S13(off);
+ } else {
+ emit_loadimm(off, tmp, ctx);
+ rs2 = RS2(tmp);
+ }
+ emit(opcode | RS1(src) | rs2 | RD(dst), ctx);
+ break;
+ }
+ /* ST: *(size *)(dst + off) = imm */
+ case BPF_ST | BPF_MEM | BPF_W:
+ case BPF_ST | BPF_MEM | BPF_H:
+ case BPF_ST | BPF_MEM | BPF_B:
+ case BPF_ST | BPF_MEM | BPF_DW: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+ const u8 tmp2 = bpf2sparc[TMP_REG_2];
+ u32 opcode = 0, rs2;
+
+ ctx->tmp_2_used = true;
+ emit_loadimm(imm, tmp2, ctx);
+
+ switch (BPF_SIZE(code)) {
+ case BPF_W:
+ opcode = ST32;
+ break;
+ case BPF_H:
+ opcode = ST16;
+ break;
+ case BPF_B:
+ opcode = ST8;
+ break;
+ case BPF_DW:
+ opcode = ST64;
+ break;
+ }
+
+ if (is_simm13(off)) {
+ opcode |= IMMED;
+ rs2 = S13(off);
+ } else {
+ ctx->tmp_1_used = true;
+ emit_loadimm(off, tmp, ctx);
+ rs2 = RS2(tmp);
+ }
+ emit(opcode | RS1(dst) | rs2 | RD(tmp2), ctx);
+ break;
+ }
+
+ /* STX: *(size *)(dst + off) = src */
+ case BPF_STX | BPF_MEM | BPF_W:
+ case BPF_STX | BPF_MEM | BPF_H:
+ case BPF_STX | BPF_MEM | BPF_B:
+ case BPF_STX | BPF_MEM | BPF_DW: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+ u32 opcode = 0, rs2;
+
+ switch (BPF_SIZE(code)) {
+ case BPF_W:
+ opcode = ST32;
+ break;
+ case BPF_H:
+ opcode = ST16;
+ break;
+ case BPF_B:
+ opcode = ST8;
+ break;
+ case BPF_DW:
+ opcode = ST64;
+ break;
+ }
+ if (is_simm13(off)) {
+ opcode |= IMMED;
+ rs2 = S13(off);
+ } else {
+ ctx->tmp_1_used = true;
+ emit_loadimm(off, tmp, ctx);
+ rs2 = RS2(tmp);
+ }
+ emit(opcode | RS1(dst) | rs2 | RD(src), ctx);
+ break;
+ }
+
+ /* STX XADD: lock *(u32 *)(dst + off) += src */
+ case BPF_STX | BPF_XADD | BPF_W: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+ const u8 tmp2 = bpf2sparc[TMP_REG_2];
+ const u8 tmp3 = bpf2sparc[TMP_REG_3];
+
+ ctx->tmp_1_used = true;
+ ctx->tmp_2_used = true;
+ ctx->tmp_3_used = true;
+ emit_loadimm(off, tmp, ctx);
+ emit_alu3(ADD, dst, tmp, tmp, ctx);
+
+ emit(LD32 | RS1(tmp) | RS2(G0) | RD(tmp2), ctx);
+ emit_alu3(ADD, tmp2, src, tmp3, ctx);
+ emit(CAS | ASI(ASI_P) | RS1(tmp) | RS2(tmp2) | RD(tmp3), ctx);
+ emit_cmp(tmp2, tmp3, ctx);
+ emit_branch(BNE, 4, 0, ctx);
+ emit_nop(ctx);
+ break;
+ }
+ /* STX XADD: lock *(u64 *)(dst + off) += src */
+ case BPF_STX | BPF_XADD | BPF_DW: {
+ const u8 tmp = bpf2sparc[TMP_REG_1];
+ const u8 tmp2 = bpf2sparc[TMP_REG_2];
+ const u8 tmp3 = bpf2sparc[TMP_REG_3];
+
+ ctx->tmp_1_used = true;
+ ctx->tmp_2_used = true;
+ ctx->tmp_3_used = true;
+ emit_loadimm(off, tmp, ctx);
+ emit_alu3(ADD, dst, tmp, tmp, ctx);
+
+ emit(LD64 | RS1(tmp) | RS2(G0) | RD(tmp2), ctx);
+ emit_alu3(ADD, tmp2, src, tmp3, ctx);
+ emit(CASX | ASI(ASI_P) | RS1(tmp) | RS2(tmp2) | RD(tmp3), ctx);
+ emit_cmp(tmp2, tmp3, ctx);
+ emit_branch(BNE, 4, 0, ctx);
+ emit_nop(ctx);
+ break;
+ }
+#define CHOOSE_LOAD_FUNC(K, func) \
+ ((int)K < 0 ? ((int)K >= SKF_LL_OFF ? func##_negative_offset : func) : func##_positive_offset)
+
+ /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + imm)) */
+ case BPF_LD | BPF_ABS | BPF_W:
+ func = CHOOSE_LOAD_FUNC(imm, bpf_jit_load_word);
+ goto common_load;
+ case BPF_LD | BPF_ABS | BPF_H:
+ func = CHOOSE_LOAD_FUNC(imm, bpf_jit_load_half);
+ goto common_load;
+ case BPF_LD | BPF_ABS | BPF_B:
+ func = CHOOSE_LOAD_FUNC(imm, bpf_jit_load_byte);
+ goto common_load;
+ /* R0 = ntohx(*(size *)(((struct sk_buff *)R6)->data + src + imm)) */
+ case BPF_LD | BPF_IND | BPF_W:
+ func = bpf_jit_load_word;
+ goto common_load;
+ case BPF_LD | BPF_IND | BPF_H:
+ func = bpf_jit_load_half;
+ goto common_load;
+
+ case BPF_LD | BPF_IND | BPF_B:
+ func = bpf_jit_load_byte;
+ common_load:
+ ctx->saw_ld_abs_ind = true;
+
+ emit_reg_move(bpf2sparc[BPF_REG_6], O0, ctx);
+ emit_loadimm(imm, O1, ctx);
+
+ if (BPF_MODE(code) == BPF_IND)
+ emit_alu(ADD, src, O1, ctx);
+
+ emit_call(func, ctx);
+ emit_alu_K(SRA, O1, 0, ctx);
+
+ emit_reg_move(O0, bpf2sparc[BPF_REG_0], ctx);
+ break;
+
+ default:
+ pr_err_once("unknown opcode %02x\n", code);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int build_body(struct jit_ctx *ctx)
+{
+ const struct bpf_prog *prog = ctx->prog;
+ int i;
+
+ for (i = 0; i < prog->len; i++) {
+ const struct bpf_insn *insn = &prog->insnsi[i];
+ int ret;
+
+ ret = build_insn(insn, ctx);
+
+ if (ret > 0) {
+ i++;
+ ctx->offset[i] = ctx->idx;
+ continue;
+ }
+ ctx->offset[i] = ctx->idx;
+ if (ret)
+ return ret;
+ }
+ return 0;
+}
+
+static void jit_fill_hole(void *area, unsigned int size)
+{
+ u32 *ptr;
+ /* We are guaranteed to have aligned memory. */
+ for (ptr = area; size >= sizeof(u32); size -= sizeof(u32))
+ *ptr++ = 0x91d02005; /* ta 5 */
+}
+
+struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
+{
+ struct bpf_prog *tmp, *orig_prog = prog;
+ struct bpf_binary_header *header;
+ bool tmp_blinded = false;
+ struct jit_ctx ctx;
+ u32 image_size;
+ u8 *image_ptr;
+ int pass;
+
+ if (!bpf_jit_enable)
+ return orig_prog;
+
+ tmp = bpf_jit_blind_constants(prog);
+ /* If blinding was requested and we failed during blinding,
+ * we must fall back to the interpreter.
+ */
+ if (IS_ERR(tmp))
+ return orig_prog;
+ if (tmp != prog) {
+ tmp_blinded = true;
+ prog = tmp;
+ }
+
+ memset(&ctx, 0, sizeof(ctx));
+ ctx.prog = prog;
+
+ ctx.offset = kcalloc(prog->len, sizeof(unsigned int), GFP_KERNEL);
+ if (ctx.offset == NULL) {
+ prog = orig_prog;
+ goto out;
+ }
+
+ /* Fake pass to detect features used, and get an accurate assessment
+ * of what the final image size will be.
+ */
+ if (build_body(&ctx)) {
+ prog = orig_prog;
+ goto out_off;
+ }
+ build_prologue(&ctx);
+ build_epilogue(&ctx);
+
+ /* Now we know the actual image size. */
+ image_size = sizeof(u32) * ctx.idx;
+ header = bpf_jit_binary_alloc(image_size, &image_ptr,
+ sizeof(u32), jit_fill_hole);
+ if (header == NULL) {
+ prog = orig_prog;
+ goto out_off;
+ }
+
+ ctx.image = (u32 *)image_ptr;
+
+ for (pass = 1; pass < 3; pass++) {
+ ctx.idx = 0;
+
+ build_prologue(&ctx);
+
+ if (build_body(&ctx)) {
+ bpf_jit_binary_free(header);
+ prog = orig_prog;
+ goto out_off;
+ }
+
+ build_epilogue(&ctx);
+
+ if (bpf_jit_enable > 1)
+ pr_info("Pass %d: shrink = %d, seen = [%c%c%c%c%c%c%c]\n", pass,
+ image_size - (ctx.idx * 4),
+ ctx.tmp_1_used ? '1' : ' ',
+ ctx.tmp_2_used ? '2' : ' ',
+ ctx.tmp_3_used ? '3' : ' ',
+ ctx.saw_ld_abs_ind ? 'L' : ' ',
+ ctx.saw_frame_pointer ? 'F' : ' ',
+ ctx.saw_call ? 'C' : ' ',
+ ctx.saw_tail_call ? 'T' : ' ');
+ }
+
+ if (bpf_jit_enable > 1)
+ bpf_jit_dump(prog->len, image_size, pass, ctx.image);
+
+ bpf_flush_icache(header, (u8 *)header + (header->pages * PAGE_SIZE));
+
+ bpf_jit_binary_lock_ro(header);
+
+ prog->bpf_func = (void *)ctx.image;
+ prog->jited = 1;
+
+out_off:
+ kfree(ctx.offset);
+out:
+ if (tmp_blinded)
+ bpf_jit_prog_release_other(prog, prog == orig_prog ?
+ tmp : orig_prog);
+ return prog;
+}
diff --git a/arch/tile/configs/tilegx_defconfig b/arch/tile/configs/tilegx_defconfig
index fd122ef..0d925fa 100644
--- a/arch/tile/configs/tilegx_defconfig
+++ b/arch/tile/configs/tilegx_defconfig
@@ -249,7 +249,6 @@
CONFIG_USB_OHCI_HCD=y
CONFIG_USB_STORAGE=y
CONFIG_EDAC=y
-CONFIG_EDAC_MM_EDAC=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_TILE=y
CONFIG_EXT2_FS=y
diff --git a/arch/tile/configs/tilepro_defconfig b/arch/tile/configs/tilepro_defconfig
index eb6a559..149d8e8 100644
--- a/arch/tile/configs/tilepro_defconfig
+++ b/arch/tile/configs/tilepro_defconfig
@@ -358,7 +358,6 @@
# CONFIG_VGA_ARB is not set
# CONFIG_USB_SUPPORT is not set
CONFIG_EDAC=y
-CONFIG_EDAC_MM_EDAC=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_TILE=y
CONFIG_EXT2_FS=y
diff --git a/arch/tile/include/asm/Kbuild b/arch/tile/include/asm/Kbuild
index aa48b6e..24c44e9 100644
--- a/arch/tile/include/asm/Kbuild
+++ b/arch/tile/include/asm/Kbuild
@@ -7,6 +7,7 @@
generic-y += emergency-restart.h
generic-y += errno.h
generic-y += exec.h
+generic-y += extable.h
generic-y += fb.h
generic-y += fcntl.h
generic-y += hw_irq.h
diff --git a/arch/tile/include/asm/uaccess.h b/arch/tile/include/asm/uaccess.h
index a77369e..a803f6b 100644
--- a/arch/tile/include/asm/uaccess.h
+++ b/arch/tile/include/asm/uaccess.h
@@ -18,15 +18,11 @@
/*
* User space memory access functions
*/
-#include <linux/sched.h>
#include <linux/mm.h>
#include <asm-generic/uaccess-unaligned.h>
#include <asm/processor.h>
#include <asm/page.h>
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
/*
* The fs value determines whether argument validity checking should be
* performed or not. If get_fs() == USER_DS, checking is performed, with
@@ -102,24 +98,7 @@
likely(__range_ok((unsigned long)(addr), (size)) == 0); \
})
-/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-
-struct exception_table_entry {
- unsigned long insn, fixup;
-};
-
-extern int fixup_exception(struct pt_regs *regs);
+#include <asm/extable.h>
/*
* This is a type: either unsigned long, if the argument fits into
@@ -334,145 +313,16 @@
((x) = 0, -EFAULT); \
})
-/**
- * __copy_to_user() - copy data into user space, with less checking.
- * @to: Destination address, in user space.
- * @from: Source address, in kernel space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from kernel space to user space. Caller must check
- * the specified block with access_ok() before calling this function.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- *
- * An alternate version - __copy_to_user_inatomic() - is designed
- * to be called from atomic context, typically bracketed by calls
- * to pagefault_disable() and pagefault_enable().
- */
-extern unsigned long __must_check __copy_to_user_inatomic(
- void __user *to, const void *from, unsigned long n);
-
-static inline unsigned long __must_check
-__copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- might_fault();
- return __copy_to_user_inatomic(to, from, n);
-}
-
-static inline unsigned long __must_check
-copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- if (access_ok(VERIFY_WRITE, to, n))
- n = __copy_to_user(to, from, n);
- return n;
-}
-
-/**
- * __copy_from_user() - copy data from user space, with less checking.
- * @to: Destination address, in kernel space.
- * @from: Source address, in user space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from user space to kernel space. Caller must check
- * the specified block with access_ok() before calling this function.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- *
- * If some data could not be copied, this function will pad the copied
- * data to the requested size using zero bytes.
- *
- * An alternate version - __copy_from_user_inatomic() - is designed
- * to be called from atomic context, typically bracketed by calls
- * to pagefault_disable() and pagefault_enable(). This version
- * does *NOT* pad with zeros.
- */
-extern unsigned long __must_check __copy_from_user_inatomic(
- void *to, const void __user *from, unsigned long n);
-extern unsigned long __must_check __copy_from_user_zeroing(
- void *to, const void __user *from, unsigned long n);
-
-static inline unsigned long __must_check
-__copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- might_fault();
- return __copy_from_user_zeroing(to, from, n);
-}
-
-static inline unsigned long __must_check
-_copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- if (access_ok(VERIFY_READ, from, n))
- n = __copy_from_user(to, from, n);
- else
- memset(to, 0, n);
- return n;
-}
-
-extern void __compiletime_error("usercopy buffer size is too small")
-__bad_copy_user(void);
-
-static inline void copy_user_overflow(int size, unsigned long count)
-{
- WARN(1, "Buffer overflow detected (%d < %lu)!\n", size, count);
-}
-
-static inline unsigned long __must_check copy_from_user(void *to,
- const void __user *from,
- unsigned long n)
-{
- int sz = __compiletime_object_size(to);
-
- if (likely(sz == -1 || sz >= n))
- n = _copy_from_user(to, from, n);
- else if (!__builtin_constant_p(n))
- copy_user_overflow(sz, n);
- else
- __bad_copy_user();
-
- return n;
-}
+extern unsigned long __must_check
+raw_copy_to_user(void __user *to, const void *from, unsigned long n);
+extern unsigned long __must_check
+raw_copy_from_user(void *to, const void __user *from, unsigned long n);
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
#ifdef __tilegx__
-/**
- * __copy_in_user() - copy data within user space, with less checking.
- * @to: Destination address, in user space.
- * @from: Source address, in user space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from user space to user space. Caller must check
- * the specified blocks with access_ok() before calling this function.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- */
-extern unsigned long __copy_in_user_inatomic(
+extern unsigned long raw_copy_in_user(
void __user *to, const void __user *from, unsigned long n);
-
-static inline unsigned long __must_check
-__copy_in_user(void __user *to, const void __user *from, unsigned long n)
-{
- might_fault();
- return __copy_in_user_inatomic(to, from, n);
-}
-
-static inline unsigned long __must_check
-copy_in_user(void __user *to, const void __user *from, unsigned long n)
-{
- if (access_ok(VERIFY_WRITE, to, n) && access_ok(VERIFY_READ, from, n))
- n = __copy_in_user(to, from, n);
- return n;
-}
#endif
diff --git a/arch/tile/kernel/time.c b/arch/tile/kernel/time.c
index 5bd4e88..6643ffb 100644
--- a/arch/tile/kernel/time.c
+++ b/arch/tile/kernel/time.c
@@ -155,6 +155,8 @@
.name = "tile timer",
.features = CLOCK_EVT_FEAT_ONESHOT,
.min_delta_ns = 1000,
+ .min_delta_ticks = 1,
+ .max_delta_ticks = MAX_TICK,
.rating = 100,
.irq = -1,
.set_next_event = tile_timer_set_next_event,
diff --git a/arch/tile/lib/exports.c b/arch/tile/lib/exports.c
index c5369fe..ecce8e1 100644
--- a/arch/tile/lib/exports.c
+++ b/arch/tile/lib/exports.c
@@ -38,11 +38,10 @@
/* arch/tile/lib/, various memcpy files */
EXPORT_SYMBOL(memcpy);
-EXPORT_SYMBOL(__copy_to_user_inatomic);
-EXPORT_SYMBOL(__copy_from_user_inatomic);
-EXPORT_SYMBOL(__copy_from_user_zeroing);
+EXPORT_SYMBOL(raw_copy_to_user);
+EXPORT_SYMBOL(raw_copy_from_user);
#ifdef __tilegx__
-EXPORT_SYMBOL(__copy_in_user_inatomic);
+EXPORT_SYMBOL(raw_copy_in_user);
#endif
/* hypervisor glue */
diff --git a/arch/tile/lib/memcpy_32.S b/arch/tile/lib/memcpy_32.S
index a2771ae..270f126 100644
--- a/arch/tile/lib/memcpy_32.S
+++ b/arch/tile/lib/memcpy_32.S
@@ -24,7 +24,6 @@
#define IS_MEMCPY 0
#define IS_COPY_FROM_USER 1
-#define IS_COPY_FROM_USER_ZEROING 2
#define IS_COPY_TO_USER -1
.section .text.memcpy_common, "ax"
@@ -42,40 +41,31 @@
9
-/* __copy_from_user_inatomic takes the kernel target address in r0,
+/* raw_copy_from_user takes the kernel target address in r0,
* the user source in r1, and the bytes to copy in r2.
* It returns the number of uncopiable bytes (hopefully zero) in r0.
*/
-ENTRY(__copy_from_user_inatomic)
-.type __copy_from_user_inatomic, @function
- FEEDBACK_ENTER_EXPLICIT(__copy_from_user_inatomic, \
+ENTRY(raw_copy_from_user)
+.type raw_copy_from_user, @function
+ FEEDBACK_ENTER_EXPLICIT(raw_copy_from_user, \
.text.memcpy_common, \
- .Lend_memcpy_common - __copy_from_user_inatomic)
+ .Lend_memcpy_common - raw_copy_from_user)
{ movei r29, IS_COPY_FROM_USER; j memcpy_common }
- .size __copy_from_user_inatomic, . - __copy_from_user_inatomic
+ .size raw_copy_from_user, . - raw_copy_from_user
-/* __copy_from_user_zeroing is like __copy_from_user_inatomic, but
- * any uncopiable bytes are zeroed in the target.
- */
-ENTRY(__copy_from_user_zeroing)
-.type __copy_from_user_zeroing, @function
- FEEDBACK_REENTER(__copy_from_user_inatomic)
- { movei r29, IS_COPY_FROM_USER_ZEROING; j memcpy_common }
- .size __copy_from_user_zeroing, . - __copy_from_user_zeroing
-
-/* __copy_to_user_inatomic takes the user target address in r0,
+/* raw_copy_to_user takes the user target address in r0,
* the kernel source in r1, and the bytes to copy in r2.
* It returns the number of uncopiable bytes (hopefully zero) in r0.
*/
-ENTRY(__copy_to_user_inatomic)
-.type __copy_to_user_inatomic, @function
- FEEDBACK_REENTER(__copy_from_user_inatomic)
+ENTRY(raw_copy_to_user)
+.type raw_copy_to_user, @function
+ FEEDBACK_REENTER(raw_copy_from_user)
{ movei r29, IS_COPY_TO_USER; j memcpy_common }
- .size __copy_to_user_inatomic, . - __copy_to_user_inatomic
+ .size raw_copy_to_user, . - raw_copy_to_user
ENTRY(memcpy)
.type memcpy, @function
- FEEDBACK_REENTER(__copy_from_user_inatomic)
+ FEEDBACK_REENTER(raw_copy_from_user)
{ movei r29, IS_MEMCPY }
.size memcpy, . - memcpy
/* Fall through */
@@ -520,12 +510,7 @@
{ bnzt r2, copy_from_user_fixup_loop }
.Lcopy_from_user_fixup_zero_remainder:
- { bbs r29, 2f } /* low bit set means IS_COPY_FROM_USER */
- /* byte-at-a-time loop faulted, so zero the rest. */
- { move r3, r2; bz r2, 2f /* should be impossible, but handle it. */ }
-1: { sb r0, zero; addi r0, r0, 1; addi r3, r3, -1 }
- { bnzt r3, 1b }
-2: move lr, r27
+ move lr, r27
{ move r0, r2; jrp lr }
copy_to_user_fixup_loop:
diff --git a/arch/tile/lib/memcpy_user_64.c b/arch/tile/lib/memcpy_user_64.c
index 97bbb60..a3fea9f 100644
--- a/arch/tile/lib/memcpy_user_64.c
+++ b/arch/tile/lib/memcpy_user_64.c
@@ -51,7 +51,7 @@
__v; \
})
-#define USERCOPY_FUNC __copy_to_user_inatomic
+#define USERCOPY_FUNC raw_copy_to_user
#define ST1(p, v) _ST((p), st1, (v))
#define ST2(p, v) _ST((p), st2, (v))
#define ST4(p, v) _ST((p), st4, (v))
@@ -62,7 +62,7 @@
#define LD8 LD
#include "memcpy_64.c"
-#define USERCOPY_FUNC __copy_from_user_inatomic
+#define USERCOPY_FUNC raw_copy_from_user
#define ST1 ST
#define ST2 ST
#define ST4 ST
@@ -73,7 +73,7 @@
#define LD8(p) _LD((p), ld)
#include "memcpy_64.c"
-#define USERCOPY_FUNC __copy_in_user_inatomic
+#define USERCOPY_FUNC raw_copy_in_user
#define ST1(p, v) _ST((p), st1, (v))
#define ST2(p, v) _ST((p), st2, (v))
#define ST4(p, v) _ST((p), st4, (v))
@@ -83,12 +83,3 @@
#define LD4(p) _LD((p), ld4u)
#define LD8(p) _LD((p), ld)
#include "memcpy_64.c"
-
-unsigned long __copy_from_user_zeroing(void *to, const void __user *from,
- unsigned long n)
-{
- unsigned long rc = __copy_from_user_inatomic(to, from, n);
- if (unlikely(rc))
- memset(to + n - rc, 0, rc);
- return rc;
-}
diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common
index fd44385..ed9c5b5 100644
--- a/arch/um/Kconfig.common
+++ b/arch/um/Kconfig.common
@@ -50,11 +50,6 @@
bool
default y
-config GENERIC_BUG
- bool
- default y
- depends on BUG
-
config HZ
int
default 100
diff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild
index e9d42aa..50a32c3 100644
--- a/arch/um/include/asm/Kbuild
+++ b/arch/um/include/asm/Kbuild
@@ -6,6 +6,7 @@
generic-y += device.h
generic-y += emergency-restart.h
generic-y += exec.h
+generic-y += extable.h
generic-y += ftrace.h
generic-y += futex.h
generic-y += hardirq.h
diff --git a/arch/um/include/asm/mmu_context.h b/arch/um/include/asm/mmu_context.h
index 94ac273..b668e35 100644
--- a/arch/um/include/asm/mmu_context.h
+++ b/arch/um/include/asm/mmu_context.h
@@ -37,12 +37,6 @@
return true;
}
-static inline bool arch_pte_access_permitted(pte_t pte, bool write)
-{
- /* by default, allow everything */
- return true;
-}
-
/*
* end asm-generic/mm_hooks.h functions
*/
diff --git a/arch/um/include/asm/uaccess.h b/arch/um/include/asm/uaccess.h
index 3705620..cc00fc5 100644
--- a/arch/um/include/asm/uaccess.h
+++ b/arch/um/include/asm/uaccess.h
@@ -7,7 +7,6 @@
#ifndef __UM_UACCESS_H
#define __UM_UACCESS_H
-#include <asm/thread_info.h>
#include <asm/elf.h>
#define __under_task_size(addr, size) \
@@ -22,8 +21,8 @@
#define __addr_range_nowrap(addr, size) \
((unsigned long) (addr) <= ((unsigned long) (addr) + (size)))
-extern long __copy_from_user(void *to, const void __user *from, unsigned long n);
-extern long __copy_to_user(void __user *to, const void *from, unsigned long n);
+extern unsigned long raw_copy_from_user(void *to, const void __user *from, unsigned long n);
+extern unsigned long raw_copy_to_user(void __user *to, const void *from, unsigned long n);
extern long __strncpy_from_user(char *dst, const char __user *src, long count);
extern long __strnlen_user(const void __user *str, long len);
extern unsigned long __clear_user(void __user *mem, unsigned long len);
@@ -32,12 +31,10 @@
/* Teach asm-generic/uaccess.h that we have C functions for these. */
#define __access_ok __access_ok
#define __clear_user __clear_user
-#define __copy_to_user __copy_to_user
-#define __copy_from_user __copy_from_user
#define __strnlen_user __strnlen_user
#define __strncpy_from_user __strncpy_from_user
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
#include <asm-generic/uaccess.h>
@@ -46,7 +43,7 @@
return __addr_range_nowrap(addr, size) &&
(__under_task_size(addr, size) ||
__access_ok_vsyscall(addr, size) ||
- segment_eq(get_fs(), KERNEL_DS));
+ uaccess_kernel());
}
#endif
diff --git a/arch/um/include/shared/os.h b/arch/um/include/shared/os.h
index de5d572..cd1fa97 100644
--- a/arch/um/include/shared/os.h
+++ b/arch/um/include/shared/os.h
@@ -302,8 +302,8 @@
extern void maybe_sigio_broken(int fd, int read);
extern void sigio_broken(int fd, int read);
-/* sys-x86_64/prctl.c */
-extern int os_arch_prctl(int pid, int code, unsigned long *addr);
+/* prctl.c */
+extern int os_arch_prctl(int pid, int option, unsigned long *arg2);
/* tty.c */
extern int get_pty(void);
diff --git a/arch/um/kernel/skas/uaccess.c b/arch/um/kernel/skas/uaccess.c
index 85ac8ad..d450797 100644
--- a/arch/um/kernel/skas/uaccess.c
+++ b/arch/um/kernel/skas/uaccess.c
@@ -139,16 +139,16 @@
return 0;
}
-long __copy_from_user(void *to, const void __user *from, unsigned long n)
+unsigned long raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
- if (segment_eq(get_fs(), KERNEL_DS)) {
+ if (uaccess_kernel()) {
memcpy(to, (__force void*)from, n);
return 0;
}
return buffer_op((unsigned long) from, n, 0, copy_chunk_from_user, &to);
}
-EXPORT_SYMBOL(__copy_from_user);
+EXPORT_SYMBOL(raw_copy_from_user);
static int copy_chunk_to_user(unsigned long to, int len, void *arg)
{
@@ -159,16 +159,16 @@
return 0;
}
-long __copy_to_user(void __user *to, const void *from, unsigned long n)
+unsigned long raw_copy_to_user(void __user *to, const void *from, unsigned long n)
{
- if (segment_eq(get_fs(), KERNEL_DS)) {
+ if (uaccess_kernel()) {
memcpy((__force void *) to, from, n);
return 0;
}
return buffer_op((unsigned long) to, n, 1, copy_chunk_to_user, &from);
}
-EXPORT_SYMBOL(__copy_to_user);
+EXPORT_SYMBOL(raw_copy_to_user);
static int strncpy_chunk_from_user(unsigned long from, int len, void *arg)
{
@@ -189,7 +189,7 @@
long n;
char *ptr = dst;
- if (segment_eq(get_fs(), KERNEL_DS)) {
+ if (uaccess_kernel()) {
strncpy(dst, (__force void *) src, count);
return strnlen(dst, count);
}
@@ -210,7 +210,7 @@
unsigned long __clear_user(void __user *mem, unsigned long len)
{
- if (segment_eq(get_fs(), KERNEL_DS)) {
+ if (uaccess_kernel()) {
memset((__force void*)mem, 0, len);
return 0;
}
@@ -235,7 +235,7 @@
{
int count = 0, n;
- if (segment_eq(get_fs(), KERNEL_DS))
+ if (uaccess_kernel())
return strnlen((__force char*)str, len) + 1;
n = buffer_op((unsigned long) str, len, 0, strnlen_chunk, &count);
diff --git a/arch/um/kernel/time.c b/arch/um/kernel/time.c
index ba87a27..0b034eb 100644
--- a/arch/um/kernel/time.c
+++ b/arch/um/kernel/time.c
@@ -65,7 +65,9 @@
.set_next_event = itimer_next_event,
.shift = 0,
.max_delta_ns = 0xffffffff,
- .min_delta_ns = TIMER_MIN_DELTA, //microsecond resolution should be enough for anyone, same as 640K RAM
+ .max_delta_ticks = 0xffffffff,
+ .min_delta_ns = TIMER_MIN_DELTA,
+ .min_delta_ticks = TIMER_MIN_DELTA, // microsecond resolution should be enough for anyone, same as 640K RAM
.irq = 0,
.mult = 1,
};
diff --git a/arch/unicore32/include/asm/Kbuild b/arch/unicore32/include/asm/Kbuild
index 84205fe..e9ad511 100644
--- a/arch/unicore32/include/asm/Kbuild
+++ b/arch/unicore32/include/asm/Kbuild
@@ -10,6 +10,7 @@
generic-y += emergency-restart.h
generic-y += errno.h
generic-y += exec.h
+generic-y += extable.h
generic-y += fb.h
generic-y += fcntl.h
generic-y += ftrace.h
diff --git a/arch/unicore32/include/asm/mmu_context.h b/arch/unicore32/include/asm/mmu_context.h
index 62dfc64..59b06b4 100644
--- a/arch/unicore32/include/asm/mmu_context.h
+++ b/arch/unicore32/include/asm/mmu_context.h
@@ -103,10 +103,4 @@
/* by default, allow everything */
return true;
}
-
-static inline bool arch_pte_access_permitted(pte_t pte, bool write)
-{
- /* by default, allow everything */
- return true;
-}
#endif
diff --git a/arch/unicore32/include/asm/uaccess.h b/arch/unicore32/include/asm/uaccess.h
index 897e11ad..1d55f2f 100644
--- a/arch/unicore32/include/asm/uaccess.h
+++ b/arch/unicore32/include/asm/uaccess.h
@@ -12,35 +12,30 @@
#ifndef __UNICORE_UACCESS_H__
#define __UNICORE_UACCESS_H__
-#include <linux/thread_info.h>
-#include <linux/errno.h>
-
#include <asm/memory.h>
-#define __copy_from_user __copy_from_user
-#define __copy_to_user __copy_to_user
#define __strncpy_from_user __strncpy_from_user
#define __strnlen_user __strnlen_user
#define __clear_user __clear_user
-#define __kernel_ok (segment_eq(get_fs(), KERNEL_DS))
+#define __kernel_ok (uaccess_kernel())
#define __user_ok(addr, size) (((size) <= TASK_SIZE) \
&& ((addr) <= TASK_SIZE - (size)))
#define __access_ok(addr, size) (__kernel_ok || __user_ok((addr), (size)))
extern unsigned long __must_check
-__copy_from_user(void *to, const void __user *from, unsigned long n);
+raw_copy_from_user(void *to, const void __user *from, unsigned long n);
extern unsigned long __must_check
-__copy_to_user(void __user *to, const void *from, unsigned long n);
+raw_copy_to_user(void __user *to, const void *from, unsigned long n);
extern unsigned long __must_check
__clear_user(void __user *addr, unsigned long n);
extern unsigned long __must_check
__strncpy_from_user(char *to, const char __user *from, unsigned long count);
extern unsigned long
__strnlen_user(const char __user *s, long n);
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
#include <asm-generic/uaccess.h>
-extern int fixup_exception(struct pt_regs *regs);
-
#endif /* __UNICORE_UACCESS_H__ */
diff --git a/arch/unicore32/kernel/ksyms.c b/arch/unicore32/kernel/ksyms.c
index 0323528..dcc72ee 100644
--- a/arch/unicore32/kernel/ksyms.c
+++ b/arch/unicore32/kernel/ksyms.c
@@ -46,8 +46,8 @@
EXPORT_SYMBOL(copy_page);
-EXPORT_SYMBOL(__copy_from_user);
-EXPORT_SYMBOL(__copy_to_user);
+EXPORT_SYMBOL(raw_copy_from_user);
+EXPORT_SYMBOL(raw_copy_to_user);
EXPORT_SYMBOL(__clear_user);
EXPORT_SYMBOL(__ashldi3);
diff --git a/arch/unicore32/kernel/process.c b/arch/unicore32/kernel/process.c
index d22c1dc..ddaf78a 100644
--- a/arch/unicore32/kernel/process.c
+++ b/arch/unicore32/kernel/process.c
@@ -178,7 +178,7 @@
buf, interrupts_enabled(regs) ? "n" : "ff",
fast_interrupts_enabled(regs) ? "n" : "ff",
processor_modes[processor_mode(regs)],
- segment_eq(get_fs(), get_ds()) ? "kernel" : "user");
+ uaccess_kernel() ? "kernel" : "user");
{
unsigned int ctrl;
diff --git a/arch/unicore32/kernel/time.c b/arch/unicore32/kernel/time.c
index fceaa67..c6b3fa3 100644
--- a/arch/unicore32/kernel/time.c
+++ b/arch/unicore32/kernel/time.c
@@ -91,8 +91,10 @@
ckevt_puv3_osmr0.max_delta_ns =
clockevent_delta2ns(0x7fffffff, &ckevt_puv3_osmr0);
+ ckevt_puv3_osmr0.max_delta_ticks = 0x7fffffff;
ckevt_puv3_osmr0.min_delta_ns =
clockevent_delta2ns(MIN_OSCR_DELTA * 2, &ckevt_puv3_osmr0) + 1;
+ ckevt_puv3_osmr0.min_delta_ticks = MIN_OSCR_DELTA * 2;
ckevt_puv3_osmr0.cpumask = cpumask_of(0);
setup_irq(IRQ_TIMER0, &puv3_timer_irq);
diff --git a/arch/unicore32/lib/copy_from_user.S b/arch/unicore32/lib/copy_from_user.S
index ab0767ea..5f80fcb 100644
--- a/arch/unicore32/lib/copy_from_user.S
+++ b/arch/unicore32/lib/copy_from_user.S
@@ -16,7 +16,7 @@
/*
* Prototype:
*
- * size_t __copy_from_user(void *to, const void *from, size_t n)
+ * size_t raw_copy_from_user(void *to, const void *from, size_t n)
*
* Purpose:
*
@@ -87,22 +87,18 @@
.text
-ENTRY(__copy_from_user)
+ENTRY(raw_copy_from_user)
#include "copy_template.S"
-ENDPROC(__copy_from_user)
+ENDPROC(raw_copy_from_user)
.pushsection .fixup,"ax"
.align 0
copy_abort_preamble
- ldm.w (r1, r2), [sp]+
- sub r3, r0, r1
- rsub r2, r3, r2
- stw r2, [sp]
- mov r1, #0
- b.l memset
- ldw.w r0, [sp]+, #4
+ ldm.w (r1, r2, r3), [sp]+
+ sub r0, r0, r1
+ rsub r0, r0, r2
copy_abort_end
.popsection
diff --git a/arch/unicore32/lib/copy_to_user.S b/arch/unicore32/lib/copy_to_user.S
index 6e22151..857c681 100644
--- a/arch/unicore32/lib/copy_to_user.S
+++ b/arch/unicore32/lib/copy_to_user.S
@@ -16,7 +16,7 @@
/*
* Prototype:
*
- * size_t __copy_to_user(void *to, const void *from, size_t n)
+ * size_t raw_copy_to_user(void *to, const void *from, size_t n)
*
* Purpose:
*
@@ -79,11 +79,11 @@
.text
-WEAK(__copy_to_user)
+WEAK(raw_copy_to_user)
#include "copy_template.S"
-ENDPROC(__copy_to_user)
+ENDPROC(raw_copy_to_user)
.pushsection .fixup,"ax"
.align 0
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index cc98d5a..cd18994 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -98,7 +98,6 @@
select HAVE_ACPI_APEI_NMI if ACPI
select HAVE_ALIGNED_STRUCT_PAGE if SLUB
select HAVE_ARCH_AUDITSYSCALL
- select HAVE_ARCH_HARDENED_USERCOPY
select HAVE_ARCH_HUGE_VMAP if X86_64 || X86_PAE
select HAVE_ARCH_JUMP_LABEL
select HAVE_ARCH_KASAN if X86_64 && SPARSEMEM_VMEMMAP
@@ -106,6 +105,7 @@
select HAVE_ARCH_KMEMCHECK
select HAVE_ARCH_MMAP_RND_BITS if MMU
select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT
+ select HAVE_ARCH_COMPAT_MMAP_BASES if MMU && COMPAT
select HAVE_ARCH_SECCOMP_FILTER
select HAVE_ARCH_TRACEHOOK
select HAVE_ARCH_TRANSPARENT_HUGEPAGE
@@ -127,7 +127,7 @@
select HAVE_EBPF_JIT if X86_64
select HAVE_EFFICIENT_UNALIGNED_ACCESS
select HAVE_EXIT_THREAD
- select HAVE_FENTRY if X86_64
+ select HAVE_FENTRY if X86_64 || DYNAMIC_FTRACE
select HAVE_FTRACE_MCOUNT_RECORD
select HAVE_FUNCTION_GRAPH_TRACER
select HAVE_FUNCTION_TRACER
@@ -160,6 +160,7 @@
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
select HAVE_REGS_AND_STACK_ACCESS_API
+ select HAVE_RELIABLE_STACKTRACE if X86_64 && FRAME_POINTER && STACK_VALIDATION
select HAVE_STACK_VALIDATION if X86_64
select HAVE_SYSCALL_TRACEPOINTS
select HAVE_UNSTABLE_SCHED_CLOCK
@@ -290,6 +291,7 @@
config KASAN_SHADOW_OFFSET
hex
depends on KASAN
+ default 0xdff8000000000000 if X86_5LEVEL
default 0xdffffc0000000000
config HAVE_INTEL_TXT
@@ -1043,6 +1045,14 @@
The action the kernel takes depends on the severity of the problem,
ranging from warning messages to halting the machine.
+config X86_MCELOG_LEGACY
+ bool "Support for deprecated /dev/mcelog character device"
+ depends on X86_MCE
+ ---help---
+ Enable support for /dev/mcelog which is needed by the old mcelog
+ userspace logging daemon. Consider switching to the new generation
+ rasdaemon solution.
+
config X86_MCE_INTEL
def_bool y
prompt "Intel MCE features"
@@ -1072,7 +1082,7 @@
def_bool y
config X86_MCE_INJECT
- depends on X86_MCE && X86_LOCAL_APIC
+ depends on X86_MCE && X86_LOCAL_APIC && X86_MCELOG_LEGACY
tristate "Machine check injector support"
---help---
Provide support for injecting machine checks for testing purposes.
@@ -1966,7 +1976,7 @@
config RANDOMIZE_BASE
bool "Randomize the address of the kernel image (KASLR)"
depends on RELOCATABLE
- default n
+ default y
---help---
In support of Kernel Address Space Layout Randomization (KASLR),
this randomizes the physical address at which the kernel image
@@ -1996,7 +2006,7 @@
theoretically possible, but the implementations are further
limited due to memory layouts.
- If unsure, say N.
+ If unsure, say Y.
# Relocation on x86 needs some additional build support
config X86_NEED_RELOCS
@@ -2045,7 +2055,7 @@
configuration have in average 30,000 different possible virtual
addresses for each memory section.
- If unsure, say N.
+ If unsure, say Y.
config RANDOMIZE_MEMORY_PHYSICAL_PADDING
hex "Physical memory mapping padding" if EXPERT
diff --git a/arch/x86/Kconfig.debug b/arch/x86/Kconfig.debug
index 63c1d13..fcb7604 100644
--- a/arch/x86/Kconfig.debug
+++ b/arch/x86/Kconfig.debug
@@ -5,6 +5,9 @@
source "lib/Kconfig.debug"
+config EARLY_PRINTK_USB
+ bool
+
config X86_VERBOSE_BOOTUP
bool "Enable verbose x86 bootup info messages"
default y
@@ -23,19 +26,20 @@
This is useful for kernel debugging when your machine crashes very
early before the console code is initialized. For normal operation
it is not recommended because it looks ugly and doesn't cooperate
- with klogd/syslogd or the X server. You should normally N here,
+ with klogd/syslogd or the X server. You should normally say N here,
unless you want to debug such a crash.
config EARLY_PRINTK_DBGP
bool "Early printk via EHCI debug port"
depends on EARLY_PRINTK && PCI
+ select EARLY_PRINTK_USB
---help---
Write kernel log output directly into the EHCI debug port.
This is useful for kernel debugging when your machine crashes very
early before the console code is initialized. For normal operation
it is not recommended because it looks ugly and doesn't cooperate
- with klogd/syslogd or the X server. You should normally N here,
+ with klogd/syslogd or the X server. You should normally say N here,
unless you want to debug such a crash. You need usb debug device.
config EARLY_PRINTK_EFI
@@ -48,6 +52,25 @@
This is useful for kernel debugging when your machine crashes very
early before the console code is initialized.
+config EARLY_PRINTK_USB_XDBC
+ bool "Early printk via the xHCI debug port"
+ depends on EARLY_PRINTK && PCI
+ select EARLY_PRINTK_USB
+ ---help---
+ Write kernel log output directly into the xHCI debug port.
+
+ One use for this feature is kernel debugging, for example when your
+ machine crashes very early before the regular console code is
+ initialized. Other uses include simpler, lockless logging instead of
+ a full-blown printk console driver + klogd.
+
+ For normal production environments this is normally not recommended,
+ because it doesn't feed events into klogd/syslogd and doesn't try to
+ print anything on the screen.
+
+ You should normally say N here, unless you want to debug early
+ crashes or need a very simple printk logging facility.
+
config X86_PTDUMP_CORE
def_bool n
diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index a94a4d1..4430dd4 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -88,10 +88,10 @@
KBUILD_CFLAGS += -m64
# Align jump targets to 1 byte, not the default 16 bytes:
- KBUILD_CFLAGS += -falign-jumps=1
+ KBUILD_CFLAGS += $(call cc-option,-falign-jumps=1)
# Pack loops tightly as well:
- KBUILD_CFLAGS += -falign-loops=1
+ KBUILD_CFLAGS += $(call cc-option,-falign-loops=1)
# Don't autogenerate traditional x87 instructions
KBUILD_CFLAGS += $(call cc-option,-mno-80387)
@@ -154,6 +154,14 @@
else
ifeq ($(call cc-option-yn, -mfentry), n)
ACCUMULATE_OUTGOING_ARGS := 1
+
+ # GCC ignores '-maccumulate-outgoing-args' when used with '-Os'.
+ # If '-Os' is enabled, disable it and print a warning.
+ ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE
+ undefine CONFIG_CC_OPTIMIZE_FOR_SIZE
+ $(warning Disabling CONFIG_CC_OPTIMIZE_FOR_SIZE. Your compiler does not have -mfentry so you cannot optimize for size with CONFIG_FUNCTION_GRAPH_TRACER.)
+ endif
+
endif
endif
endif
diff --git a/arch/x86/boot/boot.h b/arch/x86/boot/boot.h
index 9b42b6d..ef5a9cc 100644
--- a/arch/x86/boot/boot.h
+++ b/arch/x86/boot/boot.h
@@ -16,7 +16,7 @@
#ifndef BOOT_BOOT_H
#define BOOT_BOOT_H
-#define STACK_SIZE 512 /* Minimum number of bytes for stack */
+#define STACK_SIZE 1024 /* Minimum number of bytes for stack */
#ifndef __ASSEMBLY__
diff --git a/arch/x86/boot/compressed/eboot.c b/arch/x86/boot/compressed/eboot.c
index 801c7a1..cbf4b87 100644
--- a/arch/x86/boot/compressed/eboot.c
+++ b/arch/x86/boot/compressed/eboot.c
@@ -9,7 +9,9 @@
#include <linux/efi.h>
#include <linux/pci.h>
+
#include <asm/efi.h>
+#include <asm/e820/types.h>
#include <asm/setup.h>
#include <asm/desc.h>
@@ -729,7 +731,7 @@
unsigned long size;
e820ext->type = SETUP_E820_EXT;
- e820ext->len = nr_entries * sizeof(struct e820entry);
+ e820ext->len = nr_entries * sizeof(struct boot_e820_entry);
e820ext->next = 0;
data = (struct setup_data *)(unsigned long)params->hdr.setup_data;
@@ -746,9 +748,9 @@
static efi_status_t setup_e820(struct boot_params *params,
struct setup_data *e820ext, u32 e820ext_size)
{
- struct e820entry *e820_map = ¶ms->e820_map[0];
+ struct boot_e820_entry *entry = params->e820_table;
struct efi_info *efi = ¶ms->efi_info;
- struct e820entry *prev = NULL;
+ struct boot_e820_entry *prev = NULL;
u32 nr_entries;
u32 nr_desc;
int i;
@@ -773,15 +775,15 @@
case EFI_MEMORY_MAPPED_IO:
case EFI_MEMORY_MAPPED_IO_PORT_SPACE:
case EFI_PAL_CODE:
- e820_type = E820_RESERVED;
+ e820_type = E820_TYPE_RESERVED;
break;
case EFI_UNUSABLE_MEMORY:
- e820_type = E820_UNUSABLE;
+ e820_type = E820_TYPE_UNUSABLE;
break;
case EFI_ACPI_RECLAIM_MEMORY:
- e820_type = E820_ACPI;
+ e820_type = E820_TYPE_ACPI;
break;
case EFI_LOADER_CODE:
@@ -789,15 +791,15 @@
case EFI_BOOT_SERVICES_CODE:
case EFI_BOOT_SERVICES_DATA:
case EFI_CONVENTIONAL_MEMORY:
- e820_type = E820_RAM;
+ e820_type = E820_TYPE_RAM;
break;
case EFI_ACPI_MEMORY_NVS:
- e820_type = E820_NVS;
+ e820_type = E820_TYPE_NVS;
break;
case EFI_PERSISTENT_MEMORY:
- e820_type = E820_PMEM;
+ e820_type = E820_TYPE_PMEM;
break;
default:
@@ -811,26 +813,26 @@
continue;
}
- if (nr_entries == ARRAY_SIZE(params->e820_map)) {
- u32 need = (nr_desc - i) * sizeof(struct e820entry) +
+ if (nr_entries == ARRAY_SIZE(params->e820_table)) {
+ u32 need = (nr_desc - i) * sizeof(struct e820_entry) +
sizeof(struct setup_data);
if (!e820ext || e820ext_size < need)
return EFI_BUFFER_TOO_SMALL;
/* boot_params map full, switch to e820 extended */
- e820_map = (struct e820entry *)e820ext->data;
+ entry = (struct boot_e820_entry *)e820ext->data;
}
- e820_map->addr = d->phys_addr;
- e820_map->size = d->num_pages << PAGE_SHIFT;
- e820_map->type = e820_type;
- prev = e820_map++;
+ entry->addr = d->phys_addr;
+ entry->size = d->num_pages << PAGE_SHIFT;
+ entry->type = e820_type;
+ prev = entry++;
nr_entries++;
}
- if (nr_entries > ARRAY_SIZE(params->e820_map)) {
- u32 nr_e820ext = nr_entries - ARRAY_SIZE(params->e820_map);
+ if (nr_entries > ARRAY_SIZE(params->e820_table)) {
+ u32 nr_e820ext = nr_entries - ARRAY_SIZE(params->e820_table);
add_e820ext(params, e820ext, nr_e820ext);
nr_entries -= nr_e820ext;
@@ -848,7 +850,7 @@
unsigned long size;
size = sizeof(struct setup_data) +
- sizeof(struct e820entry) * nr_desc;
+ sizeof(struct e820_entry) * nr_desc;
if (*e820ext) {
efi_call_early(free_pool, *e820ext);
@@ -884,9 +886,9 @@
if (first) {
nr_desc = *map->buff_size / *map->desc_size;
- if (nr_desc > ARRAY_SIZE(p->boot_params->e820_map)) {
+ if (nr_desc > ARRAY_SIZE(p->boot_params->e820_table)) {
u32 nr_e820ext = nr_desc -
- ARRAY_SIZE(p->boot_params->e820_map);
+ ARRAY_SIZE(p->boot_params->e820_table);
status = alloc_e820ext(nr_e820ext, &p->e820ext,
&p->e820ext_size);
diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c
index 8b7c9e7..54c24f0 100644
--- a/arch/x86/boot/compressed/kaslr.c
+++ b/arch/x86/boot/compressed/kaslr.c
@@ -426,7 +426,7 @@
return 0;
}
-static void process_e820_entry(struct e820entry *entry,
+static void process_e820_entry(struct boot_e820_entry *entry,
unsigned long minimum,
unsigned long image_size)
{
@@ -435,7 +435,7 @@
unsigned long start_orig;
/* Skip non-RAM entries. */
- if (entry->type != E820_RAM)
+ if (entry->type != E820_TYPE_RAM)
return;
/* On 32-bit, ignore entries entirely above our maximum. */
@@ -518,7 +518,7 @@
/* Verify potential e820 positions, appending to slots list. */
for (i = 0; i < boot_params->e820_entries; i++) {
- process_e820_entry(&boot_params->e820_map[i], minimum,
+ process_e820_entry(&boot_params->e820_table[i], minimum,
image_size);
if (slot_area_index == MAX_SLOT_AREA) {
debug_putstr("Aborted e820 scan (slot_areas full)!\n");
@@ -597,10 +597,17 @@
add_identity_map(random_addr, output_size);
*output = random_addr;
}
+
+ /*
+ * This loads the identity mapping page table.
+ * This should only be done if a new physical address
+ * is found for the kernel, otherwise we should keep
+ * the old page table to make it be like the "nokaslr"
+ * case.
+ */
+ finalize_identity_maps();
}
- /* This actually loads the identity pagetable on x86_64. */
- finalize_identity_maps();
/* Pick random virtual address starting from LOAD_PHYSICAL_ADDR. */
if (IS_ENABLED(CONFIG_X86_64))
diff --git a/arch/x86/boot/cpucheck.c b/arch/x86/boot/cpucheck.c
index 4ad7d70..8f0c4c9 100644
--- a/arch/x86/boot/cpucheck.c
+++ b/arch/x86/boot/cpucheck.c
@@ -44,6 +44,15 @@
0, /* REQUIRED_MASK5 not implemented in this file */
REQUIRED_MASK6,
0, /* REQUIRED_MASK7 not implemented in this file */
+ 0, /* REQUIRED_MASK8 not implemented in this file */
+ 0, /* REQUIRED_MASK9 not implemented in this file */
+ 0, /* REQUIRED_MASK10 not implemented in this file */
+ 0, /* REQUIRED_MASK11 not implemented in this file */
+ 0, /* REQUIRED_MASK12 not implemented in this file */
+ 0, /* REQUIRED_MASK13 not implemented in this file */
+ 0, /* REQUIRED_MASK14 not implemented in this file */
+ 0, /* REQUIRED_MASK15 not implemented in this file */
+ REQUIRED_MASK16,
};
#define A32(a, b, c, d) (((d) << 24)+((c) << 16)+((b) << 8)+(a))
diff --git a/arch/x86/boot/cpuflags.c b/arch/x86/boot/cpuflags.c
index 6687ab9..9e77c23 100644
--- a/arch/x86/boot/cpuflags.c
+++ b/arch/x86/boot/cpuflags.c
@@ -70,16 +70,19 @@
# define EBX_REG "=b"
#endif
-static inline void cpuid(u32 id, u32 *a, u32 *b, u32 *c, u32 *d)
+static inline void cpuid_count(u32 id, u32 count,
+ u32 *a, u32 *b, u32 *c, u32 *d)
{
asm volatile(".ifnc %%ebx,%3 ; movl %%ebx,%3 ; .endif \n\t"
"cpuid \n\t"
".ifnc %%ebx,%3 ; xchgl %%ebx,%3 ; .endif \n\t"
: "=a" (*a), "=c" (*c), "=d" (*d), EBX_REG (*b)
- : "a" (id)
+ : "a" (id), "c" (count)
);
}
+#define cpuid(id, a, b, c, d) cpuid_count(id, 0, a, b, c, d)
+
void get_cpuflags(void)
{
u32 max_intel_level, max_amd_level;
@@ -108,6 +111,11 @@
cpu.model += ((tfms >> 16) & 0xf) << 4;
}
+ if (max_intel_level >= 0x00000007) {
+ cpuid_count(0x00000007, 0, &ignored, &ignored,
+ &cpu.flags[16], &ignored);
+ }
+
cpuid(0x80000000, &max_amd_level, &ignored, &ignored,
&ignored);
diff --git a/arch/x86/boot/header.S b/arch/x86/boot/header.S
index 3dd5be3..2ed8f0c 100644
--- a/arch/x86/boot/header.S
+++ b/arch/x86/boot/header.S
@@ -18,7 +18,6 @@
#include <asm/segment.h>
#include <generated/utsrelease.h>
#include <asm/boot.h>
-#include <asm/e820.h>
#include <asm/page_types.h>
#include <asm/setup.h>
#include <asm/bootparam.h>
diff --git a/arch/x86/boot/memory.c b/arch/x86/boot/memory.c
index db75d07..d9c28c8 100644
--- a/arch/x86/boot/memory.c
+++ b/arch/x86/boot/memory.c
@@ -21,8 +21,8 @@
{
int count = 0;
struct biosregs ireg, oreg;
- struct e820entry *desc = boot_params.e820_map;
- static struct e820entry buf; /* static so it is zeroed */
+ struct boot_e820_entry *desc = boot_params.e820_table;
+ static struct boot_e820_entry buf; /* static so it is zeroed */
initregs(&ireg);
ireg.ax = 0xe820;
@@ -66,7 +66,7 @@
*desc++ = buf;
count++;
- } while (ireg.ebx && count < ARRAY_SIZE(boot_params.e820_map));
+ } while (ireg.ebx && count < ARRAY_SIZE(boot_params.e820_table));
return boot_params.e820_entries = count;
}
diff --git a/arch/x86/configs/i386_defconfig b/arch/x86/configs/i386_defconfig
index 5fa6ee2..6cf79e1 100644
--- a/arch/x86/configs/i386_defconfig
+++ b/arch/x86/configs/i386_defconfig
@@ -57,6 +57,8 @@
CONFIG_HZ_1000=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
+CONFIG_RANDOMIZE_BASE=y
+CONFIG_RANDOMIZE_MEMORY=y
# CONFIG_COMPAT_VDSO is not set
CONFIG_HIBERNATION=y
CONFIG_PM_DEBUG=y
diff --git a/arch/x86/configs/x86_64_defconfig b/arch/x86/configs/x86_64_defconfig
index 6205d3b..de45f57 100644
--- a/arch/x86/configs/x86_64_defconfig
+++ b/arch/x86/configs/x86_64_defconfig
@@ -55,6 +55,8 @@
CONFIG_HZ_1000=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
+CONFIG_RANDOMIZE_BASE=y
+CONFIG_RANDOMIZE_MEMORY=y
# CONFIG_COMPAT_VDSO is not set
CONFIG_HIBERNATION=y
CONFIG_PM_DEBUG=y
diff --git a/arch/x86/crypto/aes_ctrby8_avx-x86_64.S b/arch/x86/crypto/aes_ctrby8_avx-x86_64.S
index a916c4a..5f6a5af 100644
--- a/arch/x86/crypto/aes_ctrby8_avx-x86_64.S
+++ b/arch/x86/crypto/aes_ctrby8_avx-x86_64.S
@@ -65,7 +65,6 @@
#include <linux/linkage.h>
#include <asm/inst.h>
-#define CONCAT(a,b) a##b
#define VMOVDQ vmovdqu
#define xdata0 %xmm0
@@ -92,8 +91,6 @@
#define num_bytes %r8
#define tmp %r10
-#define DDQ(i) CONCAT(ddq_add_,i)
-#define XMM(i) CONCAT(%xmm, i)
#define DDQ_DATA 0
#define XDATA 1
#define KEY_128 1
@@ -131,12 +128,12 @@
/* generate a unique variable for ddq_add_x */
.macro setddq n
- var_ddq_add = DDQ(\n)
+ var_ddq_add = ddq_add_\n
.endm
/* generate a unique variable for xmm register */
.macro setxdata n
- var_xdata = XMM(\n)
+ var_xdata = %xmm\n
.endm
/* club the numeric 'id' to the symbol 'name' */
diff --git a/arch/x86/crypto/camellia_glue.c b/arch/x86/crypto/camellia_glue.c
index aa76cad..af4840a 100644
--- a/arch/x86/crypto/camellia_glue.c
+++ b/arch/x86/crypto/camellia_glue.c
@@ -1522,7 +1522,7 @@
struct scatterlist *src, unsigned int nbytes)
{
struct camellia_xts_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
- be128 buf[2 * 4];
+ le128 buf[2 * 4];
struct xts_crypt_req req = {
.tbuf = buf,
.tbuflen = sizeof(buf),
@@ -1540,7 +1540,7 @@
struct scatterlist *src, unsigned int nbytes)
{
struct camellia_xts_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
- be128 buf[2 * 4];
+ le128 buf[2 * 4];
struct xts_crypt_req req = {
.tbuf = buf,
.tbuflen = sizeof(buf),
diff --git a/arch/x86/crypto/glue_helper.c b/arch/x86/crypto/glue_helper.c
index 260a060..24ac9fa 100644
--- a/arch/x86/crypto/glue_helper.c
+++ b/arch/x86/crypto/glue_helper.c
@@ -27,6 +27,7 @@
#include <linux/module.h>
#include <crypto/b128ops.h>
+#include <crypto/gf128mul.h>
#include <crypto/internal/skcipher.h>
#include <crypto/lrw.h>
#include <crypto/xts.h>
@@ -457,7 +458,7 @@
le128 ivblk = *iv;
/* generate next IV */
- le128_gf128mul_x_ble(iv, &ivblk);
+ gf128mul_x_ble(iv, &ivblk);
/* CC <- T xor C */
u128_xor(dst, src, (u128 *)&ivblk);
diff --git a/arch/x86/crypto/serpent_sse2_glue.c b/arch/x86/crypto/serpent_sse2_glue.c
index 644f97a..ac0e831 100644
--- a/arch/x86/crypto/serpent_sse2_glue.c
+++ b/arch/x86/crypto/serpent_sse2_glue.c
@@ -328,7 +328,7 @@
struct scatterlist *src, unsigned int nbytes)
{
struct serpent_xts_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
- be128 buf[SERPENT_PARALLEL_BLOCKS];
+ le128 buf[SERPENT_PARALLEL_BLOCKS];
struct crypt_priv crypt_ctx = {
.ctx = &ctx->crypt_ctx,
.fpu_enabled = false,
@@ -355,7 +355,7 @@
struct scatterlist *src, unsigned int nbytes)
{
struct serpent_xts_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
- be128 buf[SERPENT_PARALLEL_BLOCKS];
+ le128 buf[SERPENT_PARALLEL_BLOCKS];
struct crypt_priv crypt_ctx = {
.ctx = &ctx->crypt_ctx,
.fpu_enabled = false,
diff --git a/arch/x86/crypto/twofish_glue_3way.c b/arch/x86/crypto/twofish_glue_3way.c
index 2ebb5e9..243e90a 100644
--- a/arch/x86/crypto/twofish_glue_3way.c
+++ b/arch/x86/crypto/twofish_glue_3way.c
@@ -296,7 +296,7 @@
struct scatterlist *src, unsigned int nbytes)
{
struct twofish_xts_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
- be128 buf[3];
+ le128 buf[3];
struct xts_crypt_req req = {
.tbuf = buf,
.tbuflen = sizeof(buf),
@@ -314,7 +314,7 @@
struct scatterlist *src, unsigned int nbytes)
{
struct twofish_xts_ctx *ctx = crypto_blkcipher_ctx(desc->tfm);
- be128 buf[3];
+ le128 buf[3];
struct xts_crypt_req req = {
.tbuf = buf,
.tbuflen = sizeof(buf),
diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c
index 370c42c..cdefcfd 100644
--- a/arch/x86/entry/common.c
+++ b/arch/x86/entry/common.c
@@ -22,6 +22,7 @@
#include <linux/context_tracking.h>
#include <linux/user-return-notifier.h>
#include <linux/uprobes.h>
+#include <linux/livepatch.h>
#include <asm/desc.h>
#include <asm/traps.h>
@@ -130,14 +131,13 @@
#define EXIT_TO_USERMODE_LOOP_FLAGS \
(_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_UPROBE | \
- _TIF_NEED_RESCHED | _TIF_USER_RETURN_NOTIFY)
+ _TIF_NEED_RESCHED | _TIF_USER_RETURN_NOTIFY | _TIF_PATCH_PENDING)
static void exit_to_usermode_loop(struct pt_regs *regs, u32 cached_flags)
{
/*
* In order to return to user mode, we need to have IRQs off with
- * none of _TIF_SIGPENDING, _TIF_NOTIFY_RESUME, _TIF_USER_RETURN_NOTIFY,
- * _TIF_UPROBE, or _TIF_NEED_RESCHED set. Several of these flags
+ * none of EXIT_TO_USERMODE_LOOP_FLAGS set. Several of these flags
* can be set at any time on preemptable kernels if we have IRQs on,
* so we need to loop. Disabling preemption wouldn't help: doing the
* work to clear some of the flags can sleep.
@@ -164,6 +164,9 @@
if (cached_flags & _TIF_USER_RETURN_NOTIFY)
fire_user_return_notifiers();
+ if (cached_flags & _TIF_PATCH_PENDING)
+ klp_update_patch_state(current);
+
/* Disable IRQs and retry */
local_irq_disable();
diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S
index 57f7ec3..50bc269 100644
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -35,16 +35,13 @@
#include <asm/errno.h>
#include <asm/segment.h>
#include <asm/smp.h>
-#include <asm/page_types.h>
#include <asm/percpu.h>
#include <asm/processor-flags.h>
-#include <asm/ftrace.h>
#include <asm/irq_vectors.h>
#include <asm/cpufeatures.h>
#include <asm/alternative-asm.h>
#include <asm/asm.h>
#include <asm/smap.h>
-#include <asm/export.h>
#include <asm/frame.h>
.section .entry.text, "ax"
@@ -585,7 +582,7 @@
* will soon execute iret and the tracer was already set to
* the irqstate after the IRET:
*/
- DISABLE_INTERRUPTS(CLBR_EAX)
+ DISABLE_INTERRUPTS(CLBR_ANY)
lss (%esp), %esp /* switch to espfix segment */
jmp .Lrestore_nocheck
#endif
@@ -886,172 +883,6 @@
#endif /* CONFIG_HYPERV */
-#ifdef CONFIG_FUNCTION_TRACER
-#ifdef CONFIG_DYNAMIC_FTRACE
-
-ENTRY(mcount)
- ret
-END(mcount)
-
-ENTRY(ftrace_caller)
- pushl %eax
- pushl %ecx
- pushl %edx
- pushl $0 /* Pass NULL as regs pointer */
- movl 4*4(%esp), %eax
- movl 0x4(%ebp), %edx
- movl function_trace_op, %ecx
- subl $MCOUNT_INSN_SIZE, %eax
-
-.globl ftrace_call
-ftrace_call:
- call ftrace_stub
-
- addl $4, %esp /* skip NULL pointer */
- popl %edx
- popl %ecx
- popl %eax
-.Lftrace_ret:
-#ifdef CONFIG_FUNCTION_GRAPH_TRACER
-.globl ftrace_graph_call
-ftrace_graph_call:
- jmp ftrace_stub
-#endif
-
-/* This is weak to keep gas from relaxing the jumps */
-WEAK(ftrace_stub)
- ret
-END(ftrace_caller)
-
-ENTRY(ftrace_regs_caller)
- pushf /* push flags before compare (in cs location) */
-
- /*
- * i386 does not save SS and ESP when coming from kernel.
- * Instead, to get sp, ®s->sp is used (see ptrace.h).
- * Unfortunately, that means eflags must be at the same location
- * as the current return ip is. We move the return ip into the
- * ip location, and move flags into the return ip location.
- */
- pushl 4(%esp) /* save return ip into ip slot */
-
- pushl $0 /* Load 0 into orig_ax */
- pushl %gs
- pushl %fs
- pushl %es
- pushl %ds
- pushl %eax
- pushl %ebp
- pushl %edi
- pushl %esi
- pushl %edx
- pushl %ecx
- pushl %ebx
-
- movl 13*4(%esp), %eax /* Get the saved flags */
- movl %eax, 14*4(%esp) /* Move saved flags into regs->flags location */
- /* clobbering return ip */
- movl $__KERNEL_CS, 13*4(%esp)
-
- movl 12*4(%esp), %eax /* Load ip (1st parameter) */
- subl $MCOUNT_INSN_SIZE, %eax /* Adjust ip */
- movl 0x4(%ebp), %edx /* Load parent ip (2nd parameter) */
- movl function_trace_op, %ecx /* Save ftrace_pos in 3rd parameter */
- pushl %esp /* Save pt_regs as 4th parameter */
-
-GLOBAL(ftrace_regs_call)
- call ftrace_stub
-
- addl $4, %esp /* Skip pt_regs */
- movl 14*4(%esp), %eax /* Move flags back into cs */
- movl %eax, 13*4(%esp) /* Needed to keep addl from modifying flags */
- movl 12*4(%esp), %eax /* Get return ip from regs->ip */
- movl %eax, 14*4(%esp) /* Put return ip back for ret */
-
- popl %ebx
- popl %ecx
- popl %edx
- popl %esi
- popl %edi
- popl %ebp
- popl %eax
- popl %ds
- popl %es
- popl %fs
- popl %gs
- addl $8, %esp /* Skip orig_ax and ip */
- popf /* Pop flags at end (no addl to corrupt flags) */
- jmp .Lftrace_ret
-
- popf
- jmp ftrace_stub
-#else /* ! CONFIG_DYNAMIC_FTRACE */
-
-ENTRY(mcount)
- cmpl $__PAGE_OFFSET, %esp
- jb ftrace_stub /* Paging not enabled yet? */
-
- cmpl $ftrace_stub, ftrace_trace_function
- jnz .Ltrace
-#ifdef CONFIG_FUNCTION_GRAPH_TRACER
- cmpl $ftrace_stub, ftrace_graph_return
- jnz ftrace_graph_caller
-
- cmpl $ftrace_graph_entry_stub, ftrace_graph_entry
- jnz ftrace_graph_caller
-#endif
-.globl ftrace_stub
-ftrace_stub:
- ret
-
- /* taken from glibc */
-.Ltrace:
- pushl %eax
- pushl %ecx
- pushl %edx
- movl 0xc(%esp), %eax
- movl 0x4(%ebp), %edx
- subl $MCOUNT_INSN_SIZE, %eax
-
- call *ftrace_trace_function
-
- popl %edx
- popl %ecx
- popl %eax
- jmp ftrace_stub
-END(mcount)
-#endif /* CONFIG_DYNAMIC_FTRACE */
-EXPORT_SYMBOL(mcount)
-#endif /* CONFIG_FUNCTION_TRACER */
-
-#ifdef CONFIG_FUNCTION_GRAPH_TRACER
-ENTRY(ftrace_graph_caller)
- pushl %eax
- pushl %ecx
- pushl %edx
- movl 0xc(%esp), %eax
- lea 0x4(%ebp), %edx
- movl (%ebp), %ecx
- subl $MCOUNT_INSN_SIZE, %eax
- call prepare_ftrace_return
- popl %edx
- popl %ecx
- popl %eax
- ret
-END(ftrace_graph_caller)
-
-.globl return_to_handler
-return_to_handler:
- pushl %eax
- pushl %edx
- movl %ebp, %eax
- call ftrace_return_to_handler
- movl %eax, %ecx
- popl %edx
- popl %eax
- jmp *%ecx
-#endif
-
#ifdef CONFIG_TRACING
ENTRY(trace_page_fault)
ASM_CLAC
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 044d18e..607d72c 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -212,7 +212,7 @@
* If we see that no exit work is required (which we are required
* to check with IRQs off), then we can go straight to SYSRET64.
*/
- DISABLE_INTERRUPTS(CLBR_NONE)
+ DISABLE_INTERRUPTS(CLBR_ANY)
TRACE_IRQS_OFF
movq PER_CPU_VAR(current_task), %r11
testl $_TIF_ALLWORK_MASK, TASK_TI_flags(%r11)
@@ -233,7 +233,7 @@
* raise(3) will trigger this, for example. IRQs are off.
*/
TRACE_IRQS_ON
- ENABLE_INTERRUPTS(CLBR_NONE)
+ ENABLE_INTERRUPTS(CLBR_ANY)
SAVE_EXTRA_REGS
movq %rsp, %rdi
call syscall_return_slowpath /* returns with IRQs disabled */
@@ -265,12 +265,9 @@
*
* If width of "canonical tail" ever becomes variable, this will need
* to be updated to remain correct on both old and new CPUs.
+ *
+ * Change top 16 bits to be the sign-extension of 47th bit
*/
- .ifne __VIRTUAL_MASK_SHIFT - 47
- .error "virtual address width changed -- SYSRET checks need update"
- .endif
-
- /* Change top 16 bits to be the sign-extension of 47th bit */
shl $(64 - (__VIRTUAL_MASK_SHIFT+1)), %rcx
sar $(64 - (__VIRTUAL_MASK_SHIFT+1)), %rcx
@@ -343,7 +340,7 @@
* Called from fast path -- disable IRQs again, pop return address
* and jump to slow path
*/
- DISABLE_INTERRUPTS(CLBR_NONE)
+ DISABLE_INTERRUPTS(CLBR_ANY)
TRACE_IRQS_OFF
popq %rax
jmp entry_SYSCALL64_slow_path
@@ -518,7 +515,7 @@
interrupt do_IRQ
/* 0(%rsp): old RSP */
ret_from_intr:
- DISABLE_INTERRUPTS(CLBR_NONE)
+ DISABLE_INTERRUPTS(CLBR_ANY)
TRACE_IRQS_OFF
decl PER_CPU_VAR(irq_count)
@@ -1051,7 +1048,7 @@
* On entry, ebx is "no swapgs" flag (1: don't need swapgs, 0: need it)
*/
ENTRY(paranoid_exit)
- DISABLE_INTERRUPTS(CLBR_NONE)
+ DISABLE_INTERRUPTS(CLBR_ANY)
TRACE_IRQS_OFF_DEBUG
testl %ebx, %ebx /* swapgs needed? */
jnz paranoid_exit_no_swapgs
@@ -1156,10 +1153,9 @@
* 0: user gsbase is loaded, we need SWAPGS and standard preparation for return to usermode
*/
ENTRY(error_exit)
- movl %ebx, %eax
- DISABLE_INTERRUPTS(CLBR_NONE)
+ DISABLE_INTERRUPTS(CLBR_ANY)
TRACE_IRQS_OFF
- testl %eax, %eax
+ testl %ebx, %ebx
jnz retint_kernel
jmp retint_user
END(error_exit)
diff --git a/arch/x86/entry/syscalls/syscall_32.tbl b/arch/x86/entry/syscalls/syscall_32.tbl
index 9ba050f..448ac21 100644
--- a/arch/x86/entry/syscalls/syscall_32.tbl
+++ b/arch/x86/entry/syscalls/syscall_32.tbl
@@ -226,7 +226,7 @@
217 i386 pivot_root sys_pivot_root
218 i386 mincore sys_mincore
219 i386 madvise sys_madvise
-220 i386 getdents64 sys_getdents64 compat_sys_getdents64
+220 i386 getdents64 sys_getdents64
221 i386 fcntl64 sys_fcntl64 compat_sys_fcntl64
# 222 is unused
# 223 is unused
@@ -390,3 +390,4 @@
381 i386 pkey_alloc sys_pkey_alloc
382 i386 pkey_free sys_pkey_free
383 i386 statx sys_statx
+384 i386 arch_prctl sys_arch_prctl compat_sys_arch_prctl
diff --git a/arch/x86/entry/vdso/vclock_gettime.c b/arch/x86/entry/vdso/vclock_gettime.c
index 9d4d6e1..fa8dbfc 100644
--- a/arch/x86/entry/vdso/vclock_gettime.c
+++ b/arch/x86/entry/vdso/vclock_gettime.c
@@ -17,6 +17,7 @@
#include <asm/unistd.h>
#include <asm/msr.h>
#include <asm/pvclock.h>
+#include <asm/mshyperv.h>
#include <linux/math64.h>
#include <linux/time.h>
#include <linux/kernel.h>
@@ -32,6 +33,11 @@
__attribute__((visibility("hidden")));
#endif
+#ifdef CONFIG_HYPERV_TSCPAGE
+extern u8 hvclock_page
+ __attribute__((visibility("hidden")));
+#endif
+
#ifndef BUILD_VDSO32
notrace static long vdso_fallback_gettime(long clock, struct timespec *ts)
@@ -141,6 +147,20 @@
return last;
}
#endif
+#ifdef CONFIG_HYPERV_TSCPAGE
+static notrace u64 vread_hvclock(int *mode)
+{
+ const struct ms_hyperv_tsc_page *tsc_pg =
+ (const struct ms_hyperv_tsc_page *)&hvclock_page;
+ u64 current_tick = hv_read_tsc_page(tsc_pg);
+
+ if (current_tick != U64_MAX)
+ return current_tick;
+
+ *mode = VCLOCK_NONE;
+ return 0;
+}
+#endif
notrace static u64 vread_tsc(void)
{
@@ -173,6 +193,10 @@
else if (gtod->vclock_mode == VCLOCK_PVCLOCK)
cycles = vread_pvclock(mode);
#endif
+#ifdef CONFIG_HYPERV_TSCPAGE
+ else if (gtod->vclock_mode == VCLOCK_HVCLOCK)
+ cycles = vread_hvclock(mode);
+#endif
else
return 0;
v = (cycles - gtod->cycle_last) & gtod->mask;
diff --git a/arch/x86/entry/vdso/vdso-layout.lds.S b/arch/x86/entry/vdso/vdso-layout.lds.S
index a708aa9..8ebb4b6 100644
--- a/arch/x86/entry/vdso/vdso-layout.lds.S
+++ b/arch/x86/entry/vdso/vdso-layout.lds.S
@@ -25,7 +25,7 @@
* segment.
*/
- vvar_start = . - 2 * PAGE_SIZE;
+ vvar_start = . - 3 * PAGE_SIZE;
vvar_page = vvar_start;
/* Place all vvars at the offsets in asm/vvar.h. */
@@ -36,6 +36,7 @@
#undef EMIT_VVAR
pvclock_page = vvar_start + PAGE_SIZE;
+ hvclock_page = vvar_start + 2 * PAGE_SIZE;
. = SIZEOF_HEADERS;
diff --git a/arch/x86/entry/vdso/vdso2c.c b/arch/x86/entry/vdso/vdso2c.c
index 491020b..0780a44 100644
--- a/arch/x86/entry/vdso/vdso2c.c
+++ b/arch/x86/entry/vdso/vdso2c.c
@@ -74,6 +74,7 @@
sym_vvar_page,
sym_hpet_page,
sym_pvclock_page,
+ sym_hvclock_page,
sym_VDSO_FAKE_SECTION_TABLE_START,
sym_VDSO_FAKE_SECTION_TABLE_END,
};
@@ -82,6 +83,7 @@
sym_vvar_page,
sym_hpet_page,
sym_pvclock_page,
+ sym_hvclock_page,
};
struct vdso_sym {
@@ -94,6 +96,7 @@
[sym_vvar_page] = {"vvar_page", true},
[sym_hpet_page] = {"hpet_page", true},
[sym_pvclock_page] = {"pvclock_page", true},
+ [sym_hvclock_page] = {"hvclock_page", true},
[sym_VDSO_FAKE_SECTION_TABLE_START] = {
"VDSO_FAKE_SECTION_TABLE_START", false
},
diff --git a/arch/x86/entry/vdso/vma.c b/arch/x86/entry/vdso/vma.c
index 226ca70..139ad77 100644
--- a/arch/x86/entry/vdso/vma.c
+++ b/arch/x86/entry/vdso/vma.c
@@ -22,6 +22,7 @@
#include <asm/page.h>
#include <asm/desc.h>
#include <asm/cpufeature.h>
+#include <asm/mshyperv.h>
#if defined(CONFIG_X86_64)
unsigned int __read_mostly vdso64_enabled = 1;
@@ -121,6 +122,12 @@
vmf->address,
__pa(pvti) >> PAGE_SHIFT);
}
+ } else if (sym_offset == image->sym_hvclock_page) {
+ struct ms_hyperv_tsc_page *tsc_pg = hv_get_tsc_page();
+
+ if (tsc_pg && vclock_was_used(VCLOCK_HVCLOCK))
+ ret = vm_insert_pfn(vma, vmf->address,
+ vmalloc_to_pfn(tsc_pg));
}
if (ret == 0 || ret == -EBUSY)
@@ -354,7 +361,7 @@
d.p = 1; /* Present */
d.d = 1; /* 32-bit */
- write_gdt_entry(get_cpu_gdt_table(cpu), GDT_ENTRY_PER_CPU, &d, DESCTYPE_S);
+ write_gdt_entry(get_cpu_gdt_rw(cpu), GDT_ENTRY_PER_CPU, &d, DESCTYPE_S);
}
static int vgetcpu_online(unsigned int cpu)
diff --git a/arch/x86/events/amd/iommu.c b/arch/x86/events/amd/iommu.c
index b28200d..3641e24 100644
--- a/arch/x86/events/amd/iommu.c
+++ b/arch/x86/events/amd/iommu.c
@@ -11,6 +11,8 @@
* published by the Free Software Foundation.
*/
+#define pr_fmt(fmt) "perf/amd_iommu: " fmt
+
#include <linux/perf_event.h>
#include <linux/init.h>
#include <linux/cpumask.h>
@@ -21,44 +23,42 @@
#define COUNTER_SHIFT 16
-#define _GET_BANK(ev) ((u8)(ev->hw.extra_reg.reg >> 8))
-#define _GET_CNTR(ev) ((u8)(ev->hw.extra_reg.reg))
+/* iommu pmu conf masks */
+#define GET_CSOURCE(x) ((x)->conf & 0xFFULL)
+#define GET_DEVID(x) (((x)->conf >> 8) & 0xFFFFULL)
+#define GET_DOMID(x) (((x)->conf >> 24) & 0xFFFFULL)
+#define GET_PASID(x) (((x)->conf >> 40) & 0xFFFFFULL)
-/* iommu pmu config masks */
-#define _GET_CSOURCE(ev) ((ev->hw.config & 0xFFULL))
-#define _GET_DEVID(ev) ((ev->hw.config >> 8) & 0xFFFFULL)
-#define _GET_PASID(ev) ((ev->hw.config >> 24) & 0xFFFFULL)
-#define _GET_DOMID(ev) ((ev->hw.config >> 40) & 0xFFFFULL)
-#define _GET_DEVID_MASK(ev) ((ev->hw.extra_reg.config) & 0xFFFFULL)
-#define _GET_PASID_MASK(ev) ((ev->hw.extra_reg.config >> 16) & 0xFFFFULL)
-#define _GET_DOMID_MASK(ev) ((ev->hw.extra_reg.config >> 32) & 0xFFFFULL)
+/* iommu pmu conf1 masks */
+#define GET_DEVID_MASK(x) ((x)->conf1 & 0xFFFFULL)
+#define GET_DOMID_MASK(x) (((x)->conf1 >> 16) & 0xFFFFULL)
+#define GET_PASID_MASK(x) (((x)->conf1 >> 32) & 0xFFFFFULL)
-static struct perf_amd_iommu __perf_iommu;
+#define IOMMU_NAME_SIZE 16
struct perf_amd_iommu {
+ struct list_head list;
struct pmu pmu;
+ struct amd_iommu *iommu;
+ char name[IOMMU_NAME_SIZE];
u8 max_banks;
u8 max_counters;
u64 cntr_assign_mask;
raw_spinlock_t lock;
- const struct attribute_group *attr_groups[4];
};
-#define format_group attr_groups[0]
-#define cpumask_group attr_groups[1]
-#define events_group attr_groups[2]
-#define null_group attr_groups[3]
+static LIST_HEAD(perf_amd_iommu_list);
/*---------------------------------------------
* sysfs format attributes
*---------------------------------------------*/
PMU_FORMAT_ATTR(csource, "config:0-7");
PMU_FORMAT_ATTR(devid, "config:8-23");
-PMU_FORMAT_ATTR(pasid, "config:24-39");
-PMU_FORMAT_ATTR(domid, "config:40-55");
+PMU_FORMAT_ATTR(domid, "config:24-39");
+PMU_FORMAT_ATTR(pasid, "config:40-59");
PMU_FORMAT_ATTR(devid_mask, "config1:0-15");
-PMU_FORMAT_ATTR(pasid_mask, "config1:16-31");
-PMU_FORMAT_ATTR(domid_mask, "config1:32-47");
+PMU_FORMAT_ATTR(domid_mask, "config1:16-31");
+PMU_FORMAT_ATTR(pasid_mask, "config1:32-51");
static struct attribute *iommu_format_attrs[] = {
&format_attr_csource.attr,
@@ -79,6 +79,10 @@
/*---------------------------------------------
* sysfs events attributes
*---------------------------------------------*/
+static struct attribute_group amd_iommu_events_group = {
+ .name = "events",
+};
+
struct amd_iommu_event_desc {
struct kobj_attribute attr;
const char *event;
@@ -150,30 +154,34 @@
/*---------------------------------------------*/
-static int get_next_avail_iommu_bnk_cntr(struct perf_amd_iommu *perf_iommu)
+static int get_next_avail_iommu_bnk_cntr(struct perf_event *event)
{
+ struct perf_amd_iommu *piommu = container_of(event->pmu, struct perf_amd_iommu, pmu);
+ int max_cntrs = piommu->max_counters;
+ int max_banks = piommu->max_banks;
+ u32 shift, bank, cntr;
unsigned long flags;
- int shift, bank, cntr, retval;
- int max_banks = perf_iommu->max_banks;
- int max_cntrs = perf_iommu->max_counters;
+ int retval;
- raw_spin_lock_irqsave(&perf_iommu->lock, flags);
+ raw_spin_lock_irqsave(&piommu->lock, flags);
for (bank = 0, shift = 0; bank < max_banks; bank++) {
for (cntr = 0; cntr < max_cntrs; cntr++) {
shift = bank + (bank*3) + cntr;
- if (perf_iommu->cntr_assign_mask & (1ULL<<shift)) {
+ if (piommu->cntr_assign_mask & BIT_ULL(shift)) {
continue;
} else {
- perf_iommu->cntr_assign_mask |= (1ULL<<shift);
- retval = ((u16)((u16)bank<<8) | (u8)(cntr));
+ piommu->cntr_assign_mask |= BIT_ULL(shift);
+ event->hw.iommu_bank = bank;
+ event->hw.iommu_cntr = cntr;
+ retval = 0;
goto out;
}
}
}
retval = -ENOSPC;
out:
- raw_spin_unlock_irqrestore(&perf_iommu->lock, flags);
+ raw_spin_unlock_irqrestore(&piommu->lock, flags);
return retval;
}
@@ -202,8 +210,6 @@
static int perf_iommu_event_init(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
- struct perf_amd_iommu *perf_iommu;
- u64 config, config1;
/* test the event attr type check for PMU enumeration */
if (event->attr.type != event->pmu->type)
@@ -225,80 +231,62 @@
if (event->cpu < 0)
return -EINVAL;
- perf_iommu = &__perf_iommu;
-
- if (event->pmu != &perf_iommu->pmu)
- return -ENOENT;
-
- if (perf_iommu) {
- config = event->attr.config;
- config1 = event->attr.config1;
- } else {
- return -EINVAL;
- }
-
- /* integrate with iommu base devid (0000), assume one iommu */
- perf_iommu->max_banks =
- amd_iommu_pc_get_max_banks(IOMMU_BASE_DEVID);
- perf_iommu->max_counters =
- amd_iommu_pc_get_max_counters(IOMMU_BASE_DEVID);
- if ((perf_iommu->max_banks == 0) || (perf_iommu->max_counters == 0))
- return -EINVAL;
-
/* update the hw_perf_event struct with the iommu config data */
- hwc->config = config;
- hwc->extra_reg.config = config1;
+ hwc->conf = event->attr.config;
+ hwc->conf1 = event->attr.config1;
return 0;
}
+static inline struct amd_iommu *perf_event_2_iommu(struct perf_event *ev)
+{
+ return (container_of(ev->pmu, struct perf_amd_iommu, pmu))->iommu;
+}
+
static void perf_iommu_enable_event(struct perf_event *ev)
{
- u8 csource = _GET_CSOURCE(ev);
- u16 devid = _GET_DEVID(ev);
+ struct amd_iommu *iommu = perf_event_2_iommu(ev);
+ struct hw_perf_event *hwc = &ev->hw;
+ u8 bank = hwc->iommu_bank;
+ u8 cntr = hwc->iommu_cntr;
u64 reg = 0ULL;
- reg = csource;
- amd_iommu_pc_get_set_reg_val(devid,
- _GET_BANK(ev), _GET_CNTR(ev) ,
- IOMMU_PC_COUNTER_SRC_REG, ®, true);
+ reg = GET_CSOURCE(hwc);
+ amd_iommu_pc_set_reg(iommu, bank, cntr, IOMMU_PC_COUNTER_SRC_REG, ®);
- reg = 0ULL | devid | (_GET_DEVID_MASK(ev) << 32);
+ reg = GET_DEVID_MASK(hwc);
+ reg = GET_DEVID(hwc) | (reg << 32);
if (reg)
- reg |= (1UL << 31);
- amd_iommu_pc_get_set_reg_val(devid,
- _GET_BANK(ev), _GET_CNTR(ev) ,
- IOMMU_PC_DEVID_MATCH_REG, ®, true);
+ reg |= BIT(31);
+ amd_iommu_pc_set_reg(iommu, bank, cntr, IOMMU_PC_DEVID_MATCH_REG, ®);
- reg = 0ULL | _GET_PASID(ev) | (_GET_PASID_MASK(ev) << 32);
+ reg = GET_PASID_MASK(hwc);
+ reg = GET_PASID(hwc) | (reg << 32);
if (reg)
- reg |= (1UL << 31);
- amd_iommu_pc_get_set_reg_val(devid,
- _GET_BANK(ev), _GET_CNTR(ev) ,
- IOMMU_PC_PASID_MATCH_REG, ®, true);
+ reg |= BIT(31);
+ amd_iommu_pc_set_reg(iommu, bank, cntr, IOMMU_PC_PASID_MATCH_REG, ®);
- reg = 0ULL | _GET_DOMID(ev) | (_GET_DOMID_MASK(ev) << 32);
+ reg = GET_DOMID_MASK(hwc);
+ reg = GET_DOMID(hwc) | (reg << 32);
if (reg)
- reg |= (1UL << 31);
- amd_iommu_pc_get_set_reg_val(devid,
- _GET_BANK(ev), _GET_CNTR(ev) ,
- IOMMU_PC_DOMID_MATCH_REG, ®, true);
+ reg |= BIT(31);
+ amd_iommu_pc_set_reg(iommu, bank, cntr, IOMMU_PC_DOMID_MATCH_REG, ®);
}
static void perf_iommu_disable_event(struct perf_event *event)
{
+ struct amd_iommu *iommu = perf_event_2_iommu(event);
+ struct hw_perf_event *hwc = &event->hw;
u64 reg = 0ULL;
- amd_iommu_pc_get_set_reg_val(_GET_DEVID(event),
- _GET_BANK(event), _GET_CNTR(event),
- IOMMU_PC_COUNTER_SRC_REG, ®, true);
+ amd_iommu_pc_set_reg(iommu, hwc->iommu_bank, hwc->iommu_cntr,
+ IOMMU_PC_COUNTER_SRC_REG, ®);
}
static void perf_iommu_start(struct perf_event *event, int flags)
{
struct hw_perf_event *hwc = &event->hw;
- pr_debug("perf: amd_iommu:perf_iommu_start\n");
if (WARN_ON_ONCE(!(hwc->state & PERF_HES_STOPPED)))
return;
@@ -306,10 +294,11 @@
hwc->state = 0;
if (flags & PERF_EF_RELOAD) {
- u64 prev_raw_count = local64_read(&hwc->prev_count);
- amd_iommu_pc_get_set_reg_val(_GET_DEVID(event),
- _GET_BANK(event), _GET_CNTR(event),
- IOMMU_PC_COUNTER_REG, &prev_raw_count, true);
+ u64 prev_raw_count = local64_read(&hwc->prev_count);
+ struct amd_iommu *iommu = perf_event_2_iommu(event);
+
+ amd_iommu_pc_set_reg(iommu, hwc->iommu_bank, hwc->iommu_cntr,
+ IOMMU_PC_COUNTER_REG, &prev_raw_count);
}
perf_iommu_enable_event(event);
@@ -319,37 +308,30 @@
static void perf_iommu_read(struct perf_event *event)
{
- u64 count = 0ULL;
- u64 prev_raw_count = 0ULL;
- u64 delta = 0ULL;
+ u64 count, prev, delta;
struct hw_perf_event *hwc = &event->hw;
- pr_debug("perf: amd_iommu:perf_iommu_read\n");
+ struct amd_iommu *iommu = perf_event_2_iommu(event);
- amd_iommu_pc_get_set_reg_val(_GET_DEVID(event),
- _GET_BANK(event), _GET_CNTR(event),
- IOMMU_PC_COUNTER_REG, &count, false);
-
- /* IOMMU pc counter register is only 48 bits */
- count &= 0xFFFFFFFFFFFFULL;
-
- prev_raw_count = local64_read(&hwc->prev_count);
- if (local64_cmpxchg(&hwc->prev_count, prev_raw_count,
- count) != prev_raw_count)
+ if (amd_iommu_pc_get_reg(iommu, hwc->iommu_bank, hwc->iommu_cntr,
+ IOMMU_PC_COUNTER_REG, &count))
return;
- /* Handling 48-bit counter overflowing */
- delta = (count << COUNTER_SHIFT) - (prev_raw_count << COUNTER_SHIFT);
+ /* IOMMU pc counter register is only 48 bits */
+ count &= GENMASK_ULL(47, 0);
+
+ prev = local64_read(&hwc->prev_count);
+ if (local64_cmpxchg(&hwc->prev_count, prev, count) != prev)
+ return;
+
+ /* Handle 48-bit counter overflow */
+ delta = (count << COUNTER_SHIFT) - (prev << COUNTER_SHIFT);
delta >>= COUNTER_SHIFT;
local64_add(delta, &event->count);
-
}
static void perf_iommu_stop(struct perf_event *event, int flags)
{
struct hw_perf_event *hwc = &event->hw;
- u64 config;
-
- pr_debug("perf: amd_iommu:perf_iommu_stop\n");
if (hwc->state & PERF_HES_UPTODATE)
return;
@@ -361,7 +343,6 @@
if (hwc->state & PERF_HES_UPTODATE)
return;
- config = hwc->config;
perf_iommu_read(event);
hwc->state |= PERF_HES_UPTODATE;
}
@@ -369,17 +350,12 @@
static int perf_iommu_add(struct perf_event *event, int flags)
{
int retval;
- struct perf_amd_iommu *perf_iommu =
- container_of(event->pmu, struct perf_amd_iommu, pmu);
- pr_debug("perf: amd_iommu:perf_iommu_add\n");
event->hw.state = PERF_HES_UPTODATE | PERF_HES_STOPPED;
/* request an iommu bank/counter */
- retval = get_next_avail_iommu_bnk_cntr(perf_iommu);
- if (retval != -ENOSPC)
- event->hw.extra_reg.reg = (u16)retval;
- else
+ retval = get_next_avail_iommu_bnk_cntr(event);
+ if (retval)
return retval;
if (flags & PERF_EF_START)
@@ -390,115 +366,124 @@
static void perf_iommu_del(struct perf_event *event, int flags)
{
+ struct hw_perf_event *hwc = &event->hw;
struct perf_amd_iommu *perf_iommu =
container_of(event->pmu, struct perf_amd_iommu, pmu);
- pr_debug("perf: amd_iommu:perf_iommu_del\n");
perf_iommu_stop(event, PERF_EF_UPDATE);
/* clear the assigned iommu bank/counter */
clear_avail_iommu_bnk_cntr(perf_iommu,
- _GET_BANK(event),
- _GET_CNTR(event));
+ hwc->iommu_bank, hwc->iommu_cntr);
perf_event_update_userpage(event);
}
-static __init int _init_events_attrs(struct perf_amd_iommu *perf_iommu)
+static __init int _init_events_attrs(void)
{
- struct attribute **attrs;
- struct attribute_group *attr_group;
int i = 0, j;
+ struct attribute **attrs;
while (amd_iommu_v2_event_descs[i].attr.attr.name)
i++;
- attr_group = kzalloc(sizeof(struct attribute *)
- * (i + 1) + sizeof(*attr_group), GFP_KERNEL);
- if (!attr_group)
+ attrs = kzalloc(sizeof(struct attribute **) * (i + 1), GFP_KERNEL);
+ if (!attrs)
return -ENOMEM;
- attrs = (struct attribute **)(attr_group + 1);
for (j = 0; j < i; j++)
attrs[j] = &amd_iommu_v2_event_descs[j].attr.attr;
- attr_group->name = "events";
- attr_group->attrs = attrs;
- perf_iommu->events_group = attr_group;
-
+ amd_iommu_events_group.attrs = attrs;
return 0;
}
-static __init void amd_iommu_pc_exit(void)
-{
- if (__perf_iommu.events_group != NULL) {
- kfree(__perf_iommu.events_group);
- __perf_iommu.events_group = NULL;
- }
-}
+const struct attribute_group *amd_iommu_attr_groups[] = {
+ &amd_iommu_format_group,
+ &amd_iommu_cpumask_group,
+ &amd_iommu_events_group,
+ NULL,
+};
-static __init int _init_perf_amd_iommu(
- struct perf_amd_iommu *perf_iommu, char *name)
+static struct pmu iommu_pmu = {
+ .event_init = perf_iommu_event_init,
+ .add = perf_iommu_add,
+ .del = perf_iommu_del,
+ .start = perf_iommu_start,
+ .stop = perf_iommu_stop,
+ .read = perf_iommu_read,
+ .task_ctx_nr = perf_invalid_context,
+ .attr_groups = amd_iommu_attr_groups,
+};
+
+static __init int init_one_iommu(unsigned int idx)
{
+ struct perf_amd_iommu *perf_iommu;
int ret;
+ perf_iommu = kzalloc(sizeof(struct perf_amd_iommu), GFP_KERNEL);
+ if (!perf_iommu)
+ return -ENOMEM;
+
raw_spin_lock_init(&perf_iommu->lock);
- /* Init format attributes */
- perf_iommu->format_group = &amd_iommu_format_group;
+ perf_iommu->pmu = iommu_pmu;
+ perf_iommu->iommu = get_amd_iommu(idx);
+ perf_iommu->max_banks = amd_iommu_pc_get_max_banks(idx);
+ perf_iommu->max_counters = amd_iommu_pc_get_max_counters(idx);
- /* Init cpumask attributes to only core 0 */
- cpumask_set_cpu(0, &iommu_cpumask);
- perf_iommu->cpumask_group = &amd_iommu_cpumask_group;
-
- /* Init events attributes */
- if (_init_events_attrs(perf_iommu) != 0)
- pr_err("perf: amd_iommu: Only support raw events.\n");
-
- /* Init null attributes */
- perf_iommu->null_group = NULL;
- perf_iommu->pmu.attr_groups = perf_iommu->attr_groups;
-
- ret = perf_pmu_register(&perf_iommu->pmu, name, -1);
- if (ret) {
- pr_err("perf: amd_iommu: Failed to initialized.\n");
- amd_iommu_pc_exit();
- } else {
- pr_info("perf: amd_iommu: Detected. (%d banks, %d counters/bank)\n",
- amd_iommu_pc_get_max_banks(IOMMU_BASE_DEVID),
- amd_iommu_pc_get_max_counters(IOMMU_BASE_DEVID));
+ if (!perf_iommu->iommu ||
+ !perf_iommu->max_banks ||
+ !perf_iommu->max_counters) {
+ kfree(perf_iommu);
+ return -EINVAL;
}
+ snprintf(perf_iommu->name, IOMMU_NAME_SIZE, "amd_iommu_%u", idx);
+
+ ret = perf_pmu_register(&perf_iommu->pmu, perf_iommu->name, -1);
+ if (!ret) {
+ pr_info("Detected AMD IOMMU #%d (%d banks, %d counters/bank).\n",
+ idx, perf_iommu->max_banks, perf_iommu->max_counters);
+ list_add_tail(&perf_iommu->list, &perf_amd_iommu_list);
+ } else {
+ pr_warn("Error initializing IOMMU %d.\n", idx);
+ kfree(perf_iommu);
+ }
return ret;
}
-static struct perf_amd_iommu __perf_iommu = {
- .pmu = {
- .task_ctx_nr = perf_invalid_context,
- .event_init = perf_iommu_event_init,
- .add = perf_iommu_add,
- .del = perf_iommu_del,
- .start = perf_iommu_start,
- .stop = perf_iommu_stop,
- .read = perf_iommu_read,
- },
- .max_banks = 0x00,
- .max_counters = 0x00,
- .cntr_assign_mask = 0ULL,
- .format_group = NULL,
- .cpumask_group = NULL,
- .events_group = NULL,
- .null_group = NULL,
-};
-
static __init int amd_iommu_pc_init(void)
{
+ unsigned int i, cnt = 0;
+ int ret;
+
/* Make sure the IOMMU PC resource is available */
if (!amd_iommu_pc_supported())
return -ENODEV;
- _init_perf_amd_iommu(&__perf_iommu, "amd_iommu");
+ ret = _init_events_attrs();
+ if (ret)
+ return ret;
+ /*
+ * An IOMMU PMU is specific to an IOMMU, and can function independently.
+ * So we go through all IOMMUs and ignore the one that fails init
+ * unless all IOMMU are failing.
+ */
+ for (i = 0; i < amd_iommu_get_num_iommus(); i++) {
+ ret = init_one_iommu(i);
+ if (!ret)
+ cnt++;
+ }
+
+ if (!cnt) {
+ kfree(amd_iommu_events_group.attrs);
+ return -ENODEV;
+ }
+
+ /* Init cpumask attributes to only core 0 */
+ cpumask_set_cpu(0, &iommu_cpumask);
return 0;
}
diff --git a/arch/x86/events/amd/iommu.h b/arch/x86/events/amd/iommu.h
index 845d173..62e0702 100644
--- a/arch/x86/events/amd/iommu.h
+++ b/arch/x86/events/amd/iommu.h
@@ -24,17 +24,23 @@
#define PC_MAX_SPEC_BNKS 64
#define PC_MAX_SPEC_CNTRS 16
-/* iommu pc reg masks*/
-#define IOMMU_BASE_DEVID 0x0000
+struct amd_iommu;
/* amd_iommu_init.c external support functions */
+extern int amd_iommu_get_num_iommus(void);
+
extern bool amd_iommu_pc_supported(void);
-extern u8 amd_iommu_pc_get_max_banks(u16 devid);
+extern u8 amd_iommu_pc_get_max_banks(unsigned int idx);
-extern u8 amd_iommu_pc_get_max_counters(u16 devid);
+extern u8 amd_iommu_pc_get_max_counters(unsigned int idx);
-extern int amd_iommu_pc_get_set_reg_val(u16 devid, u8 bank, u8 cntr,
- u8 fxn, u64 *value, bool is_write);
+extern int amd_iommu_pc_set_reg(struct amd_iommu *iommu, u8 bank, u8 cntr,
+ u8 fxn, u64 *value);
+
+extern int amd_iommu_pc_get_reg(struct amd_iommu *iommu, u8 bank, u8 cntr,
+ u8 fxn, u64 *value);
+
+extern struct amd_iommu *get_amd_iommu(int idx);
#endif /*_PERF_EVENT_AMD_IOMMU_H_*/
diff --git a/arch/x86/events/amd/uncore.c b/arch/x86/events/amd/uncore.c
index 4d1f7f2d..ad44af0 100644
--- a/arch/x86/events/amd/uncore.c
+++ b/arch/x86/events/amd/uncore.c
@@ -30,6 +30,9 @@
#define COUNTER_SHIFT 16
+#undef pr_fmt
+#define pr_fmt(fmt) "amd_uncore: " fmt
+
static int num_counters_llc;
static int num_counters_nb;
@@ -509,51 +512,34 @@
int ret = -ENODEV;
if (boot_cpu_data.x86_vendor != X86_VENDOR_AMD)
- goto fail_nodev;
-
- switch(boot_cpu_data.x86) {
- case 23:
- /* Family 17h: */
- num_counters_nb = NUM_COUNTERS_NB;
- num_counters_llc = NUM_COUNTERS_L3;
- /*
- * For Family17h, the NorthBridge counters are
- * re-purposed as Data Fabric counters. Also, support is
- * added for L3 counters. The pmus are exported based on
- * family as either L2 or L3 and NB or DF.
- */
- amd_nb_pmu.name = "amd_df";
- amd_llc_pmu.name = "amd_l3";
- format_attr_event_df.show = &event_show_df;
- format_attr_event_l3.show = &event_show_l3;
- break;
- case 22:
- /* Family 16h - may change: */
- num_counters_nb = NUM_COUNTERS_NB;
- num_counters_llc = NUM_COUNTERS_L2;
- amd_nb_pmu.name = "amd_nb";
- amd_llc_pmu.name = "amd_l2";
- format_attr_event_df = format_attr_event;
- format_attr_event_l3 = format_attr_event;
- break;
- default:
- /*
- * All prior families have the same number of
- * NorthBridge and Last Level Cache counters
- */
- num_counters_nb = NUM_COUNTERS_NB;
- num_counters_llc = NUM_COUNTERS_L2;
- amd_nb_pmu.name = "amd_nb";
- amd_llc_pmu.name = "amd_l2";
- format_attr_event_df = format_attr_event;
- format_attr_event_l3 = format_attr_event;
- break;
- }
- amd_nb_pmu.attr_groups = amd_uncore_attr_groups_df;
- amd_llc_pmu.attr_groups = amd_uncore_attr_groups_l3;
+ return -ENODEV;
if (!boot_cpu_has(X86_FEATURE_TOPOEXT))
- goto fail_nodev;
+ return -ENODEV;
+
+ if (boot_cpu_data.x86 == 0x17) {
+ /*
+ * For F17h, the Northbridge counters are repurposed as Data
+ * Fabric counters. Also, L3 counters are supported too. The PMUs
+ * are exported based on family as either L2 or L3 and NB or DF.
+ */
+ num_counters_nb = NUM_COUNTERS_NB;
+ num_counters_llc = NUM_COUNTERS_L3;
+ amd_nb_pmu.name = "amd_df";
+ amd_llc_pmu.name = "amd_l3";
+ format_attr_event_df.show = &event_show_df;
+ format_attr_event_l3.show = &event_show_l3;
+ } else {
+ num_counters_nb = NUM_COUNTERS_NB;
+ num_counters_llc = NUM_COUNTERS_L2;
+ amd_nb_pmu.name = "amd_nb";
+ amd_llc_pmu.name = "amd_l2";
+ format_attr_event_df = format_attr_event;
+ format_attr_event_l3 = format_attr_event;
+ }
+
+ amd_nb_pmu.attr_groups = amd_uncore_attr_groups_df;
+ amd_llc_pmu.attr_groups = amd_uncore_attr_groups_l3;
if (boot_cpu_has(X86_FEATURE_PERFCTR_NB)) {
amd_uncore_nb = alloc_percpu(struct amd_uncore *);
@@ -565,7 +551,7 @@
if (ret)
goto fail_nb;
- pr_info("perf: AMD NB counters detected\n");
+ pr_info("AMD NB counters detected\n");
ret = 0;
}
@@ -579,7 +565,7 @@
if (ret)
goto fail_llc;
- pr_info("perf: AMD LLC counters detected\n");
+ pr_info("AMD LLC counters detected\n");
ret = 0;
}
@@ -615,7 +601,6 @@
if (amd_uncore_nb)
free_percpu(amd_uncore_nb);
-fail_nodev:
return ret;
}
device_initcall(amd_uncore_init);
diff --git a/arch/x86/events/intel/bts.c b/arch/x86/events/intel/bts.c
index 982c9e3..8ae8c5c 100644
--- a/arch/x86/events/intel/bts.c
+++ b/arch/x86/events/intel/bts.c
@@ -63,7 +63,6 @@
unsigned int cur_buf;
bool snapshot;
local_t data_size;
- local_t lost;
local_t head;
unsigned long end;
void **data_pages;
@@ -199,7 +198,8 @@
return;
if (ds->bts_index >= ds->bts_absolute_maximum)
- local_inc(&buf->lost);
+ perf_aux_output_flag(&bts->handle,
+ PERF_AUX_FLAG_TRUNCATED);
/*
* old and head are always in the same physical buffer, so we
@@ -276,7 +276,7 @@
return;
fail_end_stop:
- perf_aux_output_end(&bts->handle, 0, false);
+ perf_aux_output_end(&bts->handle, 0);
fail_stop:
event->hw.state = PERF_HES_STOPPED;
@@ -319,9 +319,8 @@
bts->handle.head =
local_xchg(&buf->data_size,
buf->nr_pages << PAGE_SHIFT);
-
- perf_aux_output_end(&bts->handle, local_xchg(&buf->data_size, 0),
- !!local_xchg(&buf->lost, 0));
+ perf_aux_output_end(&bts->handle,
+ local_xchg(&buf->data_size, 0));
}
cpuc->ds->bts_index = bts->ds_back.bts_buffer_base;
@@ -484,8 +483,7 @@
if (old_head == local_read(&buf->head))
return handled;
- perf_aux_output_end(&bts->handle, local_xchg(&buf->data_size, 0),
- !!local_xchg(&buf->lost, 0));
+ perf_aux_output_end(&bts->handle, local_xchg(&buf->data_size, 0));
buf = perf_aux_output_begin(&bts->handle, event);
if (buf)
@@ -500,7 +498,7 @@
* cleared handle::event
*/
barrier();
- perf_aux_output_end(&bts->handle, 0, false);
+ perf_aux_output_end(&bts->handle, 0);
}
}
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index eb1484c..a6d91d4 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -1553,6 +1553,27 @@
},
};
+EVENT_ATTR_STR(topdown-total-slots, td_total_slots_glm, "event=0x3c");
+EVENT_ATTR_STR(topdown-total-slots.scale, td_total_slots_scale_glm, "3");
+/* UOPS_NOT_DELIVERED.ANY */
+EVENT_ATTR_STR(topdown-fetch-bubbles, td_fetch_bubbles_glm, "event=0x9c");
+/* ISSUE_SLOTS_NOT_CONSUMED.RECOVERY */
+EVENT_ATTR_STR(topdown-recovery-bubbles, td_recovery_bubbles_glm, "event=0xca,umask=0x02");
+/* UOPS_RETIRED.ANY */
+EVENT_ATTR_STR(topdown-slots-retired, td_slots_retired_glm, "event=0xc2");
+/* UOPS_ISSUED.ANY */
+EVENT_ATTR_STR(topdown-slots-issued, td_slots_issued_glm, "event=0x0e");
+
+static struct attribute *glm_events_attrs[] = {
+ EVENT_PTR(td_total_slots_glm),
+ EVENT_PTR(td_total_slots_scale_glm),
+ EVENT_PTR(td_fetch_bubbles_glm),
+ EVENT_PTR(td_recovery_bubbles_glm),
+ EVENT_PTR(td_slots_issued_glm),
+ EVENT_PTR(td_slots_retired_glm),
+ NULL
+};
+
static struct extra_reg intel_glm_extra_regs[] __read_mostly = {
/* must define OFFCORE_RSP_X first, see intel_fixup_er() */
INTEL_UEVENT_EXTRA_REG(0x01b7, MSR_OFFCORE_RSP_0, 0x760005ffbfull, RSP_0),
@@ -2130,7 +2151,7 @@
* counters from the GLOBAL_STATUS mask and we always process PEBS
* events via drain_pebs().
*/
- status &= ~cpuc->pebs_enabled;
+ status &= ~(cpuc->pebs_enabled & PEBS_COUNTER_MASK);
/*
* PEBS overflow sets bit 62 in the global status register
@@ -3750,6 +3771,7 @@
x86_pmu.pebs_prec_dist = true;
x86_pmu.lbr_pt_coexist = true;
x86_pmu.flags |= PMU_FL_HAS_RSP_1;
+ x86_pmu.cpu_events = glm_events_attrs;
pr_cont("Goldmont events, ");
break;
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 9dfeeec..c6d23ff 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -1222,7 +1222,7 @@
/* clear non-PEBS bit and re-check */
pebs_status = p->status & cpuc->pebs_enabled;
- pebs_status &= (1ULL << MAX_PEBS_EVENTS) - 1;
+ pebs_status &= PEBS_COUNTER_MASK;
if (pebs_status == (1 << bit))
return at;
}
diff --git a/arch/x86/events/intel/pt.c b/arch/x86/events/intel/pt.c
index 5900471..ae8324d 100644
--- a/arch/x86/events/intel/pt.c
+++ b/arch/x86/events/intel/pt.c
@@ -28,6 +28,7 @@
#include <asm/insn.h>
#include <asm/io.h>
#include <asm/intel_pt.h>
+#include <asm/intel-family.h>
#include "../perf_event.h"
#include "pt.h"
@@ -98,6 +99,7 @@
.name = "caps",
};
+PMU_FORMAT_ATTR(pt, "config:0" );
PMU_FORMAT_ATTR(cyc, "config:1" );
PMU_FORMAT_ATTR(pwr_evt, "config:4" );
PMU_FORMAT_ATTR(fup_on_ptw, "config:5" );
@@ -105,11 +107,13 @@
PMU_FORMAT_ATTR(tsc, "config:10" );
PMU_FORMAT_ATTR(noretcomp, "config:11" );
PMU_FORMAT_ATTR(ptw, "config:12" );
+PMU_FORMAT_ATTR(branch, "config:13" );
PMU_FORMAT_ATTR(mtc_period, "config:14-17" );
PMU_FORMAT_ATTR(cyc_thresh, "config:19-22" );
PMU_FORMAT_ATTR(psb_period, "config:24-27" );
static struct attribute *pt_formats_attr[] = {
+ &format_attr_pt.attr,
&format_attr_cyc.attr,
&format_attr_pwr_evt.attr,
&format_attr_fup_on_ptw.attr,
@@ -117,6 +121,7 @@
&format_attr_tsc.attr,
&format_attr_noretcomp.attr,
&format_attr_ptw.attr,
+ &format_attr_branch.attr,
&format_attr_mtc_period.attr,
&format_attr_cyc_thresh.attr,
&format_attr_psb_period.attr,
@@ -197,6 +202,19 @@
pt_pmu.tsc_art_den = eax;
}
+ /* model-specific quirks */
+ switch (boot_cpu_data.x86_model) {
+ case INTEL_FAM6_BROADWELL_CORE:
+ case INTEL_FAM6_BROADWELL_XEON_D:
+ case INTEL_FAM6_BROADWELL_GT3E:
+ case INTEL_FAM6_BROADWELL_X:
+ /* not setting BRANCH_EN will #GP, erratum BDM106 */
+ pt_pmu.branch_en_always_on = true;
+ break;
+ default:
+ break;
+ }
+
if (boot_cpu_has(X86_FEATURE_VMX)) {
/*
* Intel SDM, 36.5 "Tracing post-VMXON" says that
@@ -263,8 +281,20 @@
#define RTIT_CTL_PTW (RTIT_CTL_PTW_EN | \
RTIT_CTL_FUP_ON_PTW)
-#define PT_CONFIG_MASK (RTIT_CTL_TSC_EN | \
+/*
+ * Bit 0 (TraceEn) in the attr.config is meaningless as the
+ * corresponding bit in the RTIT_CTL can only be controlled
+ * by the driver; therefore, repurpose it to mean: pass
+ * through the bit that was previously assumed to be always
+ * on for PT, thereby allowing the user to *not* set it if
+ * they so wish. See also pt_event_valid() and pt_config().
+ */
+#define RTIT_CTL_PASSTHROUGH RTIT_CTL_TRACEEN
+
+#define PT_CONFIG_MASK (RTIT_CTL_TRACEEN | \
+ RTIT_CTL_TSC_EN | \
RTIT_CTL_DISRETC | \
+ RTIT_CTL_BRANCH_EN | \
RTIT_CTL_CYC_PSB | \
RTIT_CTL_MTC | \
RTIT_CTL_PWR_EVT_EN | \
@@ -332,6 +362,33 @@
return false;
}
+ /*
+ * Setting bit 0 (TraceEn in RTIT_CTL MSR) in the attr.config
+ * clears the assomption that BranchEn must always be enabled,
+ * as was the case with the first implementation of PT.
+ * If this bit is not set, the legacy behavior is preserved
+ * for compatibility with the older userspace.
+ *
+ * Re-using bit 0 for this purpose is fine because it is never
+ * directly set by the user; previous attempts at setting it in
+ * the attr.config resulted in -EINVAL.
+ */
+ if (config & RTIT_CTL_PASSTHROUGH) {
+ /*
+ * Disallow not setting BRANCH_EN where BRANCH_EN is
+ * always required.
+ */
+ if (pt_pmu.branch_en_always_on &&
+ !(config & RTIT_CTL_BRANCH_EN))
+ return false;
+ } else {
+ /*
+ * Disallow BRANCH_EN without the PASSTHROUGH.
+ */
+ if (config & RTIT_CTL_BRANCH_EN)
+ return false;
+ }
+
return true;
}
@@ -411,6 +468,7 @@
static void pt_config(struct perf_event *event)
{
+ struct pt *pt = this_cpu_ptr(&pt_ctx);
u64 reg;
if (!event->hw.itrace_started) {
@@ -419,7 +477,20 @@
}
reg = pt_config_filters(event);
- reg |= RTIT_CTL_TOPA | RTIT_CTL_BRANCH_EN | RTIT_CTL_TRACEEN;
+ reg |= RTIT_CTL_TOPA | RTIT_CTL_TRACEEN;
+
+ /*
+ * Previously, we had BRANCH_EN on by default, but now that PT has
+ * grown features outside of branch tracing, it is useful to allow
+ * the user to disable it. Setting bit 0 in the event's attr.config
+ * allows BRANCH_EN to pass through instead of being always on. See
+ * also the comment in pt_event_valid().
+ */
+ if (event->attr.config & BIT(0)) {
+ reg |= event->attr.config & RTIT_CTL_BRANCH_EN;
+ } else {
+ reg |= RTIT_CTL_BRANCH_EN;
+ }
if (!event->attr.exclude_kernel)
reg |= RTIT_CTL_OS;
@@ -429,11 +500,15 @@
reg |= (event->attr.config & PT_CONFIG_MASK);
event->hw.config = reg;
- wrmsrl(MSR_IA32_RTIT_CTL, reg);
+ if (READ_ONCE(pt->vmx_on))
+ perf_aux_output_flag(&pt->handle, PERF_AUX_FLAG_PARTIAL);
+ else
+ wrmsrl(MSR_IA32_RTIT_CTL, reg);
}
static void pt_config_stop(struct perf_event *event)
{
+ struct pt *pt = this_cpu_ptr(&pt_ctx);
u64 ctl = READ_ONCE(event->hw.config);
/* may be already stopped by a PMI */
@@ -441,7 +516,8 @@
return;
ctl &= ~RTIT_CTL_TRACEEN;
- wrmsrl(MSR_IA32_RTIT_CTL, ctl);
+ if (!READ_ONCE(pt->vmx_on))
+ wrmsrl(MSR_IA32_RTIT_CTL, ctl);
WRITE_ONCE(event->hw.config, ctl);
@@ -753,7 +829,8 @@
*/
if (!pt_cap_get(PT_CAP_topa_multiple_entries) ||
buf->output_off == sizes(TOPA_ENTRY(buf->cur, buf->cur_idx)->size)) {
- local_inc(&buf->lost);
+ perf_aux_output_flag(&pt->handle,
+ PERF_AUX_FLAG_TRUNCATED);
advance++;
}
}
@@ -846,8 +923,10 @@
/* can't stop in the middle of an output region */
if (buf->output_off + handle->size + 1 <
- sizes(TOPA_ENTRY(buf->cur, buf->cur_idx)->size))
+ sizes(TOPA_ENTRY(buf->cur, buf->cur_idx)->size)) {
+ perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED);
return -EINVAL;
+ }
/* single entry ToPA is handled by marking all regions STOP=1 INT=1 */
@@ -1171,12 +1250,6 @@
if (!READ_ONCE(pt->handle_nmi))
return;
- /*
- * If VMX is on and PT does not support it, don't touch anything.
- */
- if (READ_ONCE(pt->vmx_on))
- return;
-
if (!event)
return;
@@ -1192,8 +1265,7 @@
pt_update_head(pt);
- perf_aux_output_end(&pt->handle, local_xchg(&buf->data_size, 0),
- local_xchg(&buf->lost, 0));
+ perf_aux_output_end(&pt->handle, local_xchg(&buf->data_size, 0));
if (!event->hw.state) {
int ret;
@@ -1208,7 +1280,7 @@
/* snapshot counters don't use PMI, so it's safe */
ret = pt_buffer_reset_markers(buf, &pt->handle);
if (ret) {
- perf_aux_output_end(&pt->handle, 0, true);
+ perf_aux_output_end(&pt->handle, 0);
return;
}
@@ -1237,12 +1309,19 @@
local_irq_save(flags);
WRITE_ONCE(pt->vmx_on, on);
- if (on) {
- /* prevent pt_config_stop() from writing RTIT_CTL */
- event = pt->handle.event;
- if (event)
- event->hw.config = 0;
- }
+ /*
+ * If an AUX transaction is in progress, it will contain
+ * gap(s), so flag it PARTIAL to inform the user.
+ */
+ event = pt->handle.event;
+ if (event)
+ perf_aux_output_flag(&pt->handle,
+ PERF_AUX_FLAG_PARTIAL);
+
+ /* Turn PTs back on */
+ if (!on && event)
+ wrmsrl(MSR_IA32_RTIT_CTL, event->hw.config);
+
local_irq_restore(flags);
}
EXPORT_SYMBOL_GPL(intel_pt_handle_vmx);
@@ -1257,9 +1336,6 @@
struct pt *pt = this_cpu_ptr(&pt_ctx);
struct pt_buffer *buf;
- if (READ_ONCE(pt->vmx_on))
- return;
-
buf = perf_aux_output_begin(&pt->handle, event);
if (!buf)
goto fail_stop;
@@ -1280,7 +1356,7 @@
return;
fail_end_stop:
- perf_aux_output_end(&pt->handle, 0, true);
+ perf_aux_output_end(&pt->handle, 0);
fail_stop:
hwc->state = PERF_HES_STOPPED;
}
@@ -1321,8 +1397,7 @@
pt->handle.head =
local_xchg(&buf->data_size,
buf->nr_pages << PAGE_SHIFT);
- perf_aux_output_end(&pt->handle, local_xchg(&buf->data_size, 0),
- local_xchg(&buf->lost, 0));
+ perf_aux_output_end(&pt->handle, local_xchg(&buf->data_size, 0));
}
}
diff --git a/arch/x86/events/intel/pt.h b/arch/x86/events/intel/pt.h
index 53473c2..0eb41d0 100644
--- a/arch/x86/events/intel/pt.h
+++ b/arch/x86/events/intel/pt.h
@@ -110,6 +110,7 @@
struct pmu pmu;
u32 caps[PT_CPUID_REGS_NUM * PT_CPUID_LEAVES];
bool vmx;
+ bool branch_en_always_on;
unsigned long max_nonturbo_ratio;
unsigned int tsc_art_num;
unsigned int tsc_art_den;
@@ -143,7 +144,6 @@
size_t output_off;
unsigned long nr_pages;
local_t data_size;
- local_t lost;
local64_t head;
bool snapshot;
unsigned long stop_pos, intr_pos;
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index bcbb1d2..be3d362 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -79,6 +79,7 @@
/* The maximal number of PEBS events: */
#define MAX_PEBS_EVENTS 8
+#define PEBS_COUNTER_MASK ((1ULL << MAX_PEBS_EVENTS) - 1)
/*
* Flags PEBS can handle without an PMI.
diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
index 8bef70e..2b01421 100644
--- a/arch/x86/hyperv/hv_init.c
+++ b/arch/x86/hyperv/hv_init.c
@@ -27,45 +27,22 @@
#include <linux/clockchips.h>
-#ifdef CONFIG_X86_64
+#ifdef CONFIG_HYPERV_TSCPAGE
static struct ms_hyperv_tsc_page *tsc_pg;
+struct ms_hyperv_tsc_page *hv_get_tsc_page(void)
+{
+ return tsc_pg;
+}
+
static u64 read_hv_clock_tsc(struct clocksource *arg)
{
- u64 current_tick;
+ u64 current_tick = hv_read_tsc_page(tsc_pg);
- if (tsc_pg->tsc_sequence != 0) {
- /*
- * Use the tsc page to compute the value.
- */
+ if (current_tick == U64_MAX)
+ rdmsrl(HV_X64_MSR_TIME_REF_COUNT, current_tick);
- while (1) {
- u64 tmp;
- u32 sequence = tsc_pg->tsc_sequence;
- u64 cur_tsc;
- u64 scale = tsc_pg->tsc_scale;
- s64 offset = tsc_pg->tsc_offset;
-
- rdtscll(cur_tsc);
- /* current_tick = ((cur_tsc *scale) >> 64) + offset */
- asm("mulq %3"
- : "=d" (current_tick), "=a" (tmp)
- : "a" (cur_tsc), "r" (scale));
-
- current_tick += offset;
- if (tsc_pg->tsc_sequence == sequence)
- return current_tick;
-
- if (tsc_pg->tsc_sequence != 0)
- continue;
- /*
- * Fallback using MSR method.
- */
- break;
- }
- }
- rdmsrl(HV_X64_MSR_TIME_REF_COUNT, current_tick);
return current_tick;
}
@@ -139,7 +116,7 @@
/*
* Register Hyper-V specific clocksource.
*/
-#ifdef CONFIG_X86_64
+#ifdef CONFIG_HYPERV_TSCPAGE
if (ms_hyperv.features & HV_X64_MSR_REFERENCE_TSC_AVAILABLE) {
union hv_x64_msr_hypercall_contents tsc_msr;
@@ -155,6 +132,9 @@
tsc_msr.guest_physical_address = vmalloc_to_pfn(tsc_pg);
wrmsrl(HV_X64_MSR_REFERENCE_TSC, tsc_msr.as_uint64);
+
+ hyperv_cs_tsc.archdata.vclock_mode = VCLOCK_HVCLOCK;
+
clocksource_register_hz(&hyperv_cs_tsc, NSEC_PER_SEC/100);
return;
}
diff --git a/arch/x86/include/asm/acpi.h b/arch/x86/include/asm/acpi.h
index 395b695..2efc768 100644
--- a/arch/x86/include/asm/acpi.h
+++ b/arch/x86/include/asm/acpi.h
@@ -52,6 +52,8 @@
extern int acpi_sci_override_gsi;
void acpi_pic_sci_set_trigger(unsigned int, u16);
+struct device;
+
extern int (*__acpi_register_gsi)(struct device *dev, u32 gsi,
int trigger, int polarity);
extern void (*__acpi_unregister_gsi)(u32 gsi);
diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h
index 730ef65..bdffcd9 100644
--- a/arch/x86/include/asm/apic.h
+++ b/arch/x86/include/asm/apic.h
@@ -252,12 +252,6 @@
#define x2apic_supported() (0)
#endif /* !CONFIG_X86_X2APIC */
-#ifdef CONFIG_X86_64
-#define SET_APIC_ID(x) (apic->set_apic_id(x))
-#else
-
-#endif
-
/*
* Copyright 2004 James Cleverdon, IBM.
* Subject to the GNU Public License, v.2
@@ -299,6 +293,7 @@
int (*phys_pkg_id)(int cpuid_apic, int index_msb);
unsigned int (*get_apic_id)(unsigned long x);
+ /* Can't be NULL on 64-bit */
unsigned long (*set_apic_id)(unsigned int id);
int (*cpu_mask_to_apicid_and)(const struct cpumask *cpumask,
diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h
index 14635c5..caa5798 100644
--- a/arch/x86/include/asm/atomic.h
+++ b/arch/x86/include/asm/atomic.h
@@ -186,6 +186,12 @@
return cmpxchg(&v->counter, old, new);
}
+#define atomic_try_cmpxchg atomic_try_cmpxchg
+static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
+{
+ return try_cmpxchg(&v->counter, old, new);
+}
+
static inline int atomic_xchg(atomic_t *v, int new)
{
return xchg(&v->counter, new);
@@ -201,16 +207,12 @@
}
#define ATOMIC_FETCH_OP(op, c_op) \
-static inline int atomic_fetch_##op(int i, atomic_t *v) \
+static inline int atomic_fetch_##op(int i, atomic_t *v) \
{ \
- int old, val = atomic_read(v); \
- for (;;) { \
- old = atomic_cmpxchg(v, val, val c_op i); \
- if (old == val) \
- break; \
- val = old; \
- } \
- return old; \
+ int val = atomic_read(v); \
+ do { \
+ } while (!atomic_try_cmpxchg(v, &val, val c_op i)); \
+ return val; \
}
#define ATOMIC_OPS(op, c_op) \
@@ -236,16 +238,11 @@
*/
static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
{
- int c, old;
- c = atomic_read(v);
- for (;;) {
- if (unlikely(c == (u)))
+ int c = atomic_read(v);
+ do {
+ if (unlikely(c == u))
break;
- old = atomic_cmpxchg((v), c, c + (a));
- if (likely(old == c))
- break;
- c = old;
- }
+ } while (!atomic_try_cmpxchg(v, &c, c + a));
return c;
}
diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h
index 89ed2f6..6189a43 100644
--- a/arch/x86/include/asm/atomic64_64.h
+++ b/arch/x86/include/asm/atomic64_64.h
@@ -176,6 +176,12 @@
return cmpxchg(&v->counter, old, new);
}
+#define atomic64_try_cmpxchg atomic64_try_cmpxchg
+static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, long *old, long new)
+{
+ return try_cmpxchg(&v->counter, old, new);
+}
+
static inline long atomic64_xchg(atomic64_t *v, long new)
{
return xchg(&v->counter, new);
@@ -192,17 +198,12 @@
*/
static inline bool atomic64_add_unless(atomic64_t *v, long a, long u)
{
- long c, old;
- c = atomic64_read(v);
- for (;;) {
- if (unlikely(c == (u)))
- break;
- old = atomic64_cmpxchg((v), c, c + (a));
- if (likely(old == c))
- break;
- c = old;
- }
- return c != (u);
+ long c = atomic64_read(v);
+ do {
+ if (unlikely(c == u))
+ return false;
+ } while (!atomic64_try_cmpxchg(v, &c, c + a));
+ return true;
}
#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
@@ -216,17 +217,12 @@
*/
static inline long atomic64_dec_if_positive(atomic64_t *v)
{
- long c, old, dec;
- c = atomic64_read(v);
- for (;;) {
+ long dec, c = atomic64_read(v);
+ do {
dec = c - 1;
if (unlikely(dec < 0))
break;
- old = atomic64_cmpxchg((v), c, dec);
- if (likely(old == c))
- break;
- c = old;
- }
+ } while (!atomic64_try_cmpxchg(v, &c, dec));
return dec;
}
@@ -242,14 +238,10 @@
#define ATOMIC64_FETCH_OP(op, c_op) \
static inline long atomic64_fetch_##op(long i, atomic64_t *v) \
{ \
- long old, val = atomic64_read(v); \
- for (;;) { \
- old = atomic64_cmpxchg(v, val, val c_op i); \
- if (old == val) \
- break; \
- val = old; \
- } \
- return old; \
+ long val = atomic64_read(v); \
+ do { \
+ } while (!atomic64_try_cmpxchg(v, &val, val c_op i)); \
+ return val; \
}
#define ATOMIC64_OPS(op, c_op) \
diff --git a/arch/x86/include/asm/bug.h b/arch/x86/include/asm/bug.h
index ba38ebb..39e702d 100644
--- a/arch/x86/include/asm/bug.h
+++ b/arch/x86/include/asm/bug.h
@@ -1,36 +1,82 @@
#ifndef _ASM_X86_BUG_H
#define _ASM_X86_BUG_H
-#define HAVE_ARCH_BUG
+#include <linux/stringify.h>
+
+/*
+ * Since some emulators terminate on UD2, we cannot use it for WARN.
+ * Since various instruction decoders disagree on the length of UD1,
+ * we cannot use it either. So use UD0 for WARN.
+ *
+ * (binutils knows about "ud1" but {en,de}codes it as 2 bytes, whereas
+ * our kernel decoder thinks it takes a ModRM byte, which seems consistent
+ * with various things like the Intel SDM instruction encoding rules)
+ */
+
+#define ASM_UD0 ".byte 0x0f, 0xff"
+#define ASM_UD1 ".byte 0x0f, 0xb9" /* + ModRM */
+#define ASM_UD2 ".byte 0x0f, 0x0b"
+
+#define INSN_UD0 0xff0f
+#define INSN_UD2 0x0b0f
+
+#define LEN_UD0 2
+
+#ifdef CONFIG_GENERIC_BUG
+
+#ifdef CONFIG_X86_32
+# define __BUG_REL(val) ".long " __stringify(val)
+#else
+# define __BUG_REL(val) ".long " __stringify(val) " - 2b"
+#endif
#ifdef CONFIG_DEBUG_BUGVERBOSE
-#ifdef CONFIG_X86_32
-# define __BUG_C0 "2:\t.long 1b, %c0\n"
-#else
-# define __BUG_C0 "2:\t.long 1b - 2b, %c0 - 2b\n"
-#endif
+#define _BUG_FLAGS(ins, flags) \
+do { \
+ asm volatile("1:\t" ins "\n" \
+ ".pushsection __bug_table,\"a\"\n" \
+ "2:\t" __BUG_REL(1b) "\t# bug_entry::bug_addr\n" \
+ "\t" __BUG_REL(%c0) "\t# bug_entry::file\n" \
+ "\t.word %c1" "\t# bug_entry::line\n" \
+ "\t.word %c2" "\t# bug_entry::flags\n" \
+ "\t.org 2b+%c3\n" \
+ ".popsection" \
+ : : "i" (__FILE__), "i" (__LINE__), \
+ "i" (flags), \
+ "i" (sizeof(struct bug_entry))); \
+} while (0)
+#else /* !CONFIG_DEBUG_BUGVERBOSE */
+
+#define _BUG_FLAGS(ins, flags) \
+do { \
+ asm volatile("1:\t" ins "\n" \
+ ".pushsection __bug_table,\"a\"\n" \
+ "2:\t" __BUG_REL(1b) "\t# bug_entry::bug_addr\n" \
+ "\t.word %c0" "\t# bug_entry::flags\n" \
+ "\t.org 2b+%c1\n" \
+ ".popsection" \
+ : : "i" (flags), \
+ "i" (sizeof(struct bug_entry))); \
+} while (0)
+
+#endif /* CONFIG_DEBUG_BUGVERBOSE */
+
+#else
+
+#define _BUG_FLAGS(ins, flags) asm volatile(ins)
+
+#endif /* CONFIG_GENERIC_BUG */
+
+#define HAVE_ARCH_BUG
#define BUG() \
do { \
- asm volatile("1:\tud2\n" \
- ".pushsection __bug_table,\"a\"\n" \
- __BUG_C0 \
- "\t.word %c1, 0\n" \
- "\t.org 2b+%c2\n" \
- ".popsection" \
- : : "i" (__FILE__), "i" (__LINE__), \
- "i" (sizeof(struct bug_entry))); \
+ _BUG_FLAGS(ASM_UD2, 0); \
unreachable(); \
} while (0)
-#else
-#define BUG() \
-do { \
- asm volatile("ud2"); \
- unreachable(); \
-} while (0)
-#endif
+#define __WARN_FLAGS(flags) _BUG_FLAGS(ASM_UD0, BUGFLAG_WARNING|(flags))
#include <asm-generic/bug.h>
diff --git a/arch/x86/include/asm/clocksource.h b/arch/x86/include/asm/clocksource.h
index eae33c7..47bea8c 100644
--- a/arch/x86/include/asm/clocksource.h
+++ b/arch/x86/include/asm/clocksource.h
@@ -6,7 +6,8 @@
#define VCLOCK_NONE 0 /* No vDSO clock available. */
#define VCLOCK_TSC 1 /* vDSO should use vread_tsc. */
#define VCLOCK_PVCLOCK 2 /* vDSO should use vread_pvclock. */
-#define VCLOCK_MAX 2
+#define VCLOCK_HVCLOCK 3 /* vDSO should use vread_hvclock. */
+#define VCLOCK_MAX 3
struct arch_clocksource_data {
int vclock_mode;
diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h
index 97848cd..d90296d 100644
--- a/arch/x86/include/asm/cmpxchg.h
+++ b/arch/x86/include/asm/cmpxchg.h
@@ -153,6 +153,76 @@
#define cmpxchg_local(ptr, old, new) \
__cmpxchg_local(ptr, old, new, sizeof(*(ptr)))
+
+#define __raw_try_cmpxchg(_ptr, _pold, _new, size, lock) \
+({ \
+ bool success; \
+ __typeof__(_ptr) _old = (_pold); \
+ __typeof__(*(_ptr)) __old = *_old; \
+ __typeof__(*(_ptr)) __new = (_new); \
+ switch (size) { \
+ case __X86_CASE_B: \
+ { \
+ volatile u8 *__ptr = (volatile u8 *)(_ptr); \
+ asm volatile(lock "cmpxchgb %[new], %[ptr]" \
+ CC_SET(z) \
+ : CC_OUT(z) (success), \
+ [ptr] "+m" (*__ptr), \
+ [old] "+a" (__old) \
+ : [new] "q" (__new) \
+ : "memory"); \
+ break; \
+ } \
+ case __X86_CASE_W: \
+ { \
+ volatile u16 *__ptr = (volatile u16 *)(_ptr); \
+ asm volatile(lock "cmpxchgw %[new], %[ptr]" \
+ CC_SET(z) \
+ : CC_OUT(z) (success), \
+ [ptr] "+m" (*__ptr), \
+ [old] "+a" (__old) \
+ : [new] "r" (__new) \
+ : "memory"); \
+ break; \
+ } \
+ case __X86_CASE_L: \
+ { \
+ volatile u32 *__ptr = (volatile u32 *)(_ptr); \
+ asm volatile(lock "cmpxchgl %[new], %[ptr]" \
+ CC_SET(z) \
+ : CC_OUT(z) (success), \
+ [ptr] "+m" (*__ptr), \
+ [old] "+a" (__old) \
+ : [new] "r" (__new) \
+ : "memory"); \
+ break; \
+ } \
+ case __X86_CASE_Q: \
+ { \
+ volatile u64 *__ptr = (volatile u64 *)(_ptr); \
+ asm volatile(lock "cmpxchgq %[new], %[ptr]" \
+ CC_SET(z) \
+ : CC_OUT(z) (success), \
+ [ptr] "+m" (*__ptr), \
+ [old] "+a" (__old) \
+ : [new] "r" (__new) \
+ : "memory"); \
+ break; \
+ } \
+ default: \
+ __cmpxchg_wrong_size(); \
+ } \
+ if (unlikely(!success)) \
+ *_old = __old; \
+ likely(success); \
+})
+
+#define __try_cmpxchg(ptr, pold, new, size) \
+ __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
+
+#define try_cmpxchg(ptr, pold, new) \
+ __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
+
/*
* xadd() adds "inc" to "*ptr" and atomically returns the previous
* value of "*ptr".
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index b04bb6d..2701e5f 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -187,6 +187,7 @@
* Reuse free bits when adding new feature flags!
*/
#define X86_FEATURE_RING3MWAIT ( 7*32+ 0) /* Ring 3 MONITOR/MWAIT */
+#define X86_FEATURE_CPUID_FAULT ( 7*32+ 1) /* Intel CPUID faulting */
#define X86_FEATURE_CPB ( 7*32+ 2) /* AMD Core Performance Boost */
#define X86_FEATURE_EPB ( 7*32+ 3) /* IA32_ENERGY_PERF_BIAS support */
#define X86_FEATURE_CAT_L3 ( 7*32+ 4) /* Cache Allocation Technology L3 */
@@ -201,6 +202,8 @@
#define X86_FEATURE_AVX512_4VNNIW (7*32+16) /* AVX-512 Neural Network Instructions */
#define X86_FEATURE_AVX512_4FMAPS (7*32+17) /* AVX-512 Multiply Accumulation Single precision */
+#define X86_FEATURE_MBA ( 7*32+18) /* Memory Bandwidth Allocation */
+
/* Virtualization flags: Linux defined, word 8 */
#define X86_FEATURE_TPR_SHADOW ( 8*32+ 0) /* Intel TPR Shadow */
#define X86_FEATURE_VNMI ( 8*32+ 1) /* Intel Virtual NMI */
diff --git a/arch/x86/include/asm/crypto/glue_helper.h b/arch/x86/include/asm/crypto/glue_helper.h
index 29e53ea..ed8b66d 100644
--- a/arch/x86/include/asm/crypto/glue_helper.h
+++ b/arch/x86/include/asm/crypto/glue_helper.h
@@ -125,16 +125,6 @@
i->b = cpu_to_le64(b);
}
-static inline void le128_gf128mul_x_ble(le128 *dst, const le128 *src)
-{
- u64 a = le64_to_cpu(src->a);
- u64 b = le64_to_cpu(src->b);
- u64 _tt = ((s64)a >> 63) & 0x87;
-
- dst->a = cpu_to_le64((a << 1) ^ (b >> 63));
- dst->b = cpu_to_le64((b << 1) ^ _tt);
-}
-
extern int glue_ecb_crypt_128bit(const struct common_glue_ctx *gctx,
struct blkcipher_desc *desc,
struct scatterlist *dst,
diff --git a/arch/x86/include/asm/desc.h b/arch/x86/include/asm/desc.h
index 1548ca9..d0a21b1 100644
--- a/arch/x86/include/asm/desc.h
+++ b/arch/x86/include/asm/desc.h
@@ -4,6 +4,7 @@
#include <asm/desc_defs.h>
#include <asm/ldt.h>
#include <asm/mmu.h>
+#include <asm/fixmap.h>
#include <linux/smp.h>
#include <linux/percpu.h>
@@ -45,11 +46,43 @@
DECLARE_PER_CPU_PAGE_ALIGNED(struct gdt_page, gdt_page);
-static inline struct desc_struct *get_cpu_gdt_table(unsigned int cpu)
+/* Provide the original GDT */
+static inline struct desc_struct *get_cpu_gdt_rw(unsigned int cpu)
{
return per_cpu(gdt_page, cpu).gdt;
}
+/* Provide the current original GDT */
+static inline struct desc_struct *get_current_gdt_rw(void)
+{
+ return this_cpu_ptr(&gdt_page)->gdt;
+}
+
+/* Get the fixmap index for a specific processor */
+static inline unsigned int get_cpu_gdt_ro_index(int cpu)
+{
+ return FIX_GDT_REMAP_BEGIN + cpu;
+}
+
+/* Provide the fixmap address of the remapped GDT */
+static inline struct desc_struct *get_cpu_gdt_ro(int cpu)
+{
+ unsigned int idx = get_cpu_gdt_ro_index(cpu);
+ return (struct desc_struct *)__fix_to_virt(idx);
+}
+
+/* Provide the current read-only GDT */
+static inline struct desc_struct *get_current_gdt_ro(void)
+{
+ return get_cpu_gdt_ro(smp_processor_id());
+}
+
+/* Provide the physical address of the GDT page. */
+static inline phys_addr_t get_cpu_gdt_paddr(unsigned int cpu)
+{
+ return per_cpu_ptr_to_phys(get_cpu_gdt_rw(cpu));
+}
+
#ifdef CONFIG_X86_64
static inline void pack_gate(gate_desc *gate, unsigned type, unsigned long func,
@@ -174,7 +207,7 @@
static inline void __set_tss_desc(unsigned cpu, unsigned int entry, void *addr)
{
- struct desc_struct *d = get_cpu_gdt_table(cpu);
+ struct desc_struct *d = get_cpu_gdt_rw(cpu);
tss_desc tss;
set_tssldt_descriptor(&tss, (unsigned long)addr, DESC_TSS,
@@ -194,22 +227,90 @@
set_tssldt_descriptor(&ldt, (unsigned long)addr, DESC_LDT,
entries * LDT_ENTRY_SIZE - 1);
- write_gdt_entry(get_cpu_gdt_table(cpu), GDT_ENTRY_LDT,
+ write_gdt_entry(get_cpu_gdt_rw(cpu), GDT_ENTRY_LDT,
&ldt, DESC_LDT);
asm volatile("lldt %w0"::"q" (GDT_ENTRY_LDT*8));
}
}
+static inline void native_load_gdt(const struct desc_ptr *dtr)
+{
+ asm volatile("lgdt %0"::"m" (*dtr));
+}
+
+static inline void native_load_idt(const struct desc_ptr *dtr)
+{
+ asm volatile("lidt %0"::"m" (*dtr));
+}
+
+static inline void native_store_gdt(struct desc_ptr *dtr)
+{
+ asm volatile("sgdt %0":"=m" (*dtr));
+}
+
+static inline void native_store_idt(struct desc_ptr *dtr)
+{
+ asm volatile("sidt %0":"=m" (*dtr));
+}
+
+/*
+ * The LTR instruction marks the TSS GDT entry as busy. On 64-bit, the GDT is
+ * a read-only remapping. To prevent a page fault, the GDT is switched to the
+ * original writeable version when needed.
+ */
+#ifdef CONFIG_X86_64
+static inline void native_load_tr_desc(void)
+{
+ struct desc_ptr gdt;
+ int cpu = raw_smp_processor_id();
+ bool restore = 0;
+ struct desc_struct *fixmap_gdt;
+
+ native_store_gdt(&gdt);
+ fixmap_gdt = get_cpu_gdt_ro(cpu);
+
+ /*
+ * If the current GDT is the read-only fixmap, swap to the original
+ * writeable version. Swap back at the end.
+ */
+ if (gdt.address == (unsigned long)fixmap_gdt) {
+ load_direct_gdt(cpu);
+ restore = 1;
+ }
+ asm volatile("ltr %w0"::"q" (GDT_ENTRY_TSS*8));
+ if (restore)
+ load_fixmap_gdt(cpu);
+}
+#else
static inline void native_load_tr_desc(void)
{
asm volatile("ltr %w0"::"q" (GDT_ENTRY_TSS*8));
}
+#endif
+
+static inline unsigned long native_store_tr(void)
+{
+ unsigned long tr;
+
+ asm volatile("str %0":"=r" (tr));
+
+ return tr;
+}
+
+static inline void native_load_tls(struct thread_struct *t, unsigned int cpu)
+{
+ struct desc_struct *gdt = get_cpu_gdt_rw(cpu);
+ unsigned int i;
+
+ for (i = 0; i < GDT_ENTRY_TLS_ENTRIES; i++)
+ gdt[GDT_ENTRY_TLS_MIN + i] = t->tls_array[i];
+}
DECLARE_PER_CPU(bool, __tss_limit_invalid);
static inline void force_reload_TR(void)
{
- struct desc_struct *d = get_cpu_gdt_table(smp_processor_id());
+ struct desc_struct *d = get_current_gdt_rw();
tss_desc tss;
memcpy(&tss, &d[GDT_ENTRY_TSS], sizeof(tss_desc));
@@ -257,44 +358,6 @@
this_cpu_write(__tss_limit_invalid, true);
}
-static inline void native_load_gdt(const struct desc_ptr *dtr)
-{
- asm volatile("lgdt %0"::"m" (*dtr));
-}
-
-static inline void native_load_idt(const struct desc_ptr *dtr)
-{
- asm volatile("lidt %0"::"m" (*dtr));
-}
-
-static inline void native_store_gdt(struct desc_ptr *dtr)
-{
- asm volatile("sgdt %0":"=m" (*dtr));
-}
-
-static inline void native_store_idt(struct desc_ptr *dtr)
-{
- asm volatile("sidt %0":"=m" (*dtr));
-}
-
-static inline unsigned long native_store_tr(void)
-{
- unsigned long tr;
-
- asm volatile("str %0":"=r" (tr));
-
- return tr;
-}
-
-static inline void native_load_tls(struct thread_struct *t, unsigned int cpu)
-{
- struct desc_struct *gdt = get_cpu_gdt_table(cpu);
- unsigned int i;
-
- for (i = 0; i < GDT_ENTRY_TLS_ENTRIES; i++)
- gdt[GDT_ENTRY_TLS_MIN + i] = t->tls_array[i];
-}
-
/* This intentionally ignores lm, since 32-bit apps don't have that field. */
#define LDT_empty(info) \
((info)->base_addr == 0 && \
diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h
index 85599ad..5dff775 100644
--- a/arch/x86/include/asm/disabled-features.h
+++ b/arch/x86/include/asm/disabled-features.h
@@ -36,6 +36,12 @@
# define DISABLE_OSPKE (1<<(X86_FEATURE_OSPKE & 31))
#endif /* CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS */
+#ifdef CONFIG_X86_5LEVEL
+# define DISABLE_LA57 0
+#else
+# define DISABLE_LA57 (1<<(X86_FEATURE_LA57 & 31))
+#endif
+
/*
* Make sure to add features to the correct mask
*/
@@ -55,7 +61,7 @@
#define DISABLED_MASK13 0
#define DISABLED_MASK14 0
#define DISABLED_MASK15 0
-#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE)
+#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57)
#define DISABLED_MASK17 0
#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 18)
diff --git a/arch/x86/include/asm/e820.h b/arch/x86/include/asm/e820.h
deleted file mode 100644
index 67313f3..0000000
--- a/arch/x86/include/asm/e820.h
+++ /dev/null
@@ -1,73 +0,0 @@
-#ifndef _ASM_X86_E820_H
-#define _ASM_X86_E820_H
-
-/*
- * E820_X_MAX is the maximum size of the extended E820 table. The extended
- * table may contain up to 3 extra E820 entries per possible NUMA node, so we
- * make room for 3 * MAX_NUMNODES possible entries, beyond the standard 128.
- * Also note that E820_X_MAX *must* be defined before we include uapi/asm/e820.h.
- */
-#include <linux/numa.h>
-#define E820_X_MAX (E820MAX + 3 * MAX_NUMNODES)
-
-#include <uapi/asm/e820.h>
-
-#ifndef __ASSEMBLY__
-/* see comment in arch/x86/kernel/e820.c */
-extern struct e820map *e820;
-extern struct e820map *e820_saved;
-
-extern unsigned long pci_mem_start;
-extern int e820_any_mapped(u64 start, u64 end, unsigned type);
-extern int e820_all_mapped(u64 start, u64 end, unsigned type);
-extern void e820_add_region(u64 start, u64 size, int type);
-extern void e820_print_map(char *who);
-extern int
-sanitize_e820_map(struct e820entry *biosmap, int max_nr_map, u32 *pnr_map);
-extern u64 e820_update_range(u64 start, u64 size, unsigned old_type,
- unsigned new_type);
-extern u64 e820_remove_range(u64 start, u64 size, unsigned old_type,
- int checktype);
-extern void update_e820(void);
-extern void e820_setup_gap(void);
-struct setup_data;
-extern void parse_e820_ext(u64 phys_addr, u32 data_len);
-
-#if defined(CONFIG_X86_64) || \
- (defined(CONFIG_X86_32) && defined(CONFIG_HIBERNATION))
-extern void e820_mark_nosave_regions(unsigned long limit_pfn);
-#else
-static inline void e820_mark_nosave_regions(unsigned long limit_pfn)
-{
-}
-#endif
-
-extern unsigned long e820_end_of_ram_pfn(void);
-extern unsigned long e820_end_of_low_ram_pfn(void);
-extern u64 early_reserve_e820(u64 sizet, u64 align);
-
-void memblock_x86_fill(void);
-void memblock_find_dma_reserve(void);
-
-extern void finish_e820_parsing(void);
-extern void e820_reserve_resources(void);
-extern void e820_reserve_resources_late(void);
-extern void setup_memory_map(void);
-extern char *default_machine_specific_memory_setup(void);
-
-extern void e820_reallocate_tables(void);
-
-/*
- * Returns true iff the specified range [s,e) is completely contained inside
- * the ISA region.
- */
-static inline bool is_ISA_range(u64 s, u64 e)
-{
- return s >= ISA_START_ADDRESS && e <= ISA_END_ADDRESS;
-}
-
-#endif /* __ASSEMBLY__ */
-#include <linux/ioport.h>
-
-#define HIGH_MEMORY (1024*1024)
-#endif /* _ASM_X86_E820_H */
diff --git a/arch/x86/include/asm/e820/api.h b/arch/x86/include/asm/e820/api.h
new file mode 100644
index 0000000..8e0f8b8
--- /dev/null
+++ b/arch/x86/include/asm/e820/api.h
@@ -0,0 +1,50 @@
+#ifndef _ASM_E820_API_H
+#define _ASM_E820_API_H
+
+#include <asm/e820/types.h>
+
+extern struct e820_table *e820_table;
+extern struct e820_table *e820_table_firmware;
+
+extern unsigned long pci_mem_start;
+
+extern bool e820__mapped_any(u64 start, u64 end, enum e820_type type);
+extern bool e820__mapped_all(u64 start, u64 end, enum e820_type type);
+
+extern void e820__range_add (u64 start, u64 size, enum e820_type type);
+extern u64 e820__range_update(u64 start, u64 size, enum e820_type old_type, enum e820_type new_type);
+extern u64 e820__range_remove(u64 start, u64 size, enum e820_type old_type, bool check_type);
+
+extern void e820__print_table(char *who);
+extern int e820__update_table(struct e820_table *table);
+extern void e820__update_table_print(void);
+
+extern unsigned long e820__end_of_ram_pfn(void);
+extern unsigned long e820__end_of_low_ram_pfn(void);
+
+extern u64 e820__memblock_alloc_reserved(u64 size, u64 align);
+extern void e820__memblock_setup(void);
+
+extern void e820__reserve_setup_data(void);
+extern void e820__finish_early_params(void);
+extern void e820__reserve_resources(void);
+extern void e820__reserve_resources_late(void);
+
+extern void e820__memory_setup(void);
+extern void e820__memory_setup_extended(u64 phys_addr, u32 data_len);
+extern char *e820__memory_setup_default(void);
+extern void e820__setup_pci_gap(void);
+
+extern void e820__reallocate_tables(void);
+extern void e820__register_nosave_regions(unsigned long limit_pfn);
+
+/*
+ * Returns true iff the specified range [start,end) is completely contained inside
+ * the ISA region.
+ */
+static inline bool is_ISA_range(u64 start, u64 end)
+{
+ return start >= ISA_START_ADDRESS && end <= ISA_END_ADDRESS;
+}
+
+#endif /* _ASM_E820_API_H */
diff --git a/arch/x86/include/asm/e820/types.h b/arch/x86/include/asm/e820/types.h
new file mode 100644
index 0000000..4adeed0
--- /dev/null
+++ b/arch/x86/include/asm/e820/types.h
@@ -0,0 +1,104 @@
+#ifndef _ASM_E820_TYPES_H
+#define _ASM_E820_TYPES_H
+
+#include <uapi/asm/bootparam.h>
+
+/*
+ * These are the E820 types known to the kernel:
+ */
+enum e820_type {
+ E820_TYPE_RAM = 1,
+ E820_TYPE_RESERVED = 2,
+ E820_TYPE_ACPI = 3,
+ E820_TYPE_NVS = 4,
+ E820_TYPE_UNUSABLE = 5,
+ E820_TYPE_PMEM = 7,
+
+ /*
+ * This is a non-standardized way to represent ADR or
+ * NVDIMM regions that persist over a reboot.
+ *
+ * The kernel will ignore their special capabilities
+ * unless the CONFIG_X86_PMEM_LEGACY=y option is set.
+ *
+ * ( Note that older platforms also used 6 for the same
+ * type of memory, but newer versions switched to 12 as
+ * 6 was assigned differently. Some time they will learn... )
+ */
+ E820_TYPE_PRAM = 12,
+
+ /*
+ * Reserved RAM used by the kernel itself if
+ * CONFIG_INTEL_TXT=y is enabled, memory of this type
+ * will be included in the S3 integrity calculation
+ * and so should not include any memory that the BIOS
+ * might alter over the S3 transition:
+ */
+ E820_TYPE_RESERVED_KERN = 128,
+};
+
+/*
+ * A single E820 map entry, describing a memory range of [addr...addr+size-1],
+ * of 'type' memory type:
+ *
+ * (We pack it because there can be thousands of them on large systems.)
+ */
+struct e820_entry {
+ u64 addr;
+ u64 size;
+ enum e820_type type;
+} __attribute__((packed));
+
+/*
+ * The legacy E820 BIOS limits us to 128 (E820_MAX_ENTRIES_ZEROPAGE) nodes
+ * due to the constrained space in the zeropage.
+ *
+ * On large systems we can easily have thousands of nodes with RAM,
+ * which cannot be fit into so few entries - so we have a mechanism
+ * to extend the e820 table size at build-time, via the E820_MAX_ENTRIES
+ * define below.
+ *
+ * ( Those extra entries are enumerated via the EFI memory map, not
+ * via the legacy zeropage mechanism. )
+ *
+ * Size our internal memory map tables to have room for these additional
+ * entries, based on a heuristic calculation: up to three entries per
+ * NUMA node, plus E820_MAX_ENTRIES_ZEROPAGE for some extra space.
+ *
+ * This allows for bootstrap/firmware quirks such as possible duplicate
+ * E820 entries that might need room in the same arrays, prior to the
+ * call to e820__update_table() to remove duplicates. The allowance
+ * of three memory map entries per node is "enough" entries for
+ * the initial hardware platform motivating this mechanism to make
+ * use of additional EFI map entries. Future platforms may want
+ * to allow more than three entries per node or otherwise refine
+ * this size.
+ */
+
+#include <linux/numa.h>
+
+#define E820_MAX_ENTRIES (E820_MAX_ENTRIES_ZEROPAGE + 3*MAX_NUMNODES)
+
+/*
+ * The whole array of E820 entries:
+ */
+struct e820_table {
+ __u32 nr_entries;
+ struct e820_entry entries[E820_MAX_ENTRIES];
+};
+
+/*
+ * Various well-known legacy memory ranges in physical memory:
+ */
+#define ISA_START_ADDRESS 0x000a0000
+#define ISA_END_ADDRESS 0x00100000
+
+#define BIOS_BEGIN 0x000a0000
+#define BIOS_END 0x00100000
+
+#define HIGH_MEMORY 0x00100000
+
+#define BIOS_ROM_BASE 0xffe00000
+#define BIOS_ROM_END 0xffffffff
+
+#endif /* _ASM_E820_TYPES_H */
diff --git a/arch/x86/include/asm/elf.h b/arch/x86/include/asm/elf.h
index 3762536..e8ab9a4 100644
--- a/arch/x86/include/asm/elf.h
+++ b/arch/x86/include/asm/elf.h
@@ -293,8 +293,23 @@
} \
} while (0)
+/*
+ * True on X86_32 or when emulating IA32 on X86_64
+ */
+static inline int mmap_is_ia32(void)
+{
+ return IS_ENABLED(CONFIG_X86_32) ||
+ (IS_ENABLED(CONFIG_COMPAT) &&
+ test_thread_flag(TIF_ADDR32));
+}
+
+extern unsigned long tasksize_32bit(void);
+extern unsigned long tasksize_64bit(void);
+extern unsigned long get_mmap_base(int is_legacy);
+
#ifdef CONFIG_X86_32
+#define __STACK_RND_MASK(is32bit) (0x7ff)
#define STACK_RND_MASK (0x7ff)
#define ARCH_DLINFO ARCH_DLINFO_IA32
@@ -304,7 +319,8 @@
#else /* CONFIG_X86_32 */
/* 1GB for 64bit, 8MB for 32bit */
-#define STACK_RND_MASK (test_thread_flag(TIF_ADDR32) ? 0x7ff : 0x3fffff)
+#define __STACK_RND_MASK(is32bit) ((is32bit) ? 0x7ff : 0x3fffff)
+#define STACK_RND_MASK __STACK_RND_MASK(mmap_is_ia32())
#define ARCH_DLINFO \
do { \
@@ -348,16 +364,6 @@
int uses_interp);
#define compat_arch_setup_additional_pages compat_arch_setup_additional_pages
-/*
- * True on X86_32 or when emulating IA32 on X86_64
- */
-static inline int mmap_is_ia32(void)
-{
- return IS_ENABLED(CONFIG_X86_32) ||
- (IS_ENABLED(CONFIG_COMPAT) &&
- test_thread_flag(TIF_ADDR32));
-}
-
/* Do not change the values. See get_align_mask() */
enum align_flags {
ALIGN_VA_32 = BIT(0),
diff --git a/arch/x86/include/asm/fixmap.h b/arch/x86/include/asm/fixmap.h
index 8554f96..b65155c 100644
--- a/arch/x86/include/asm/fixmap.h
+++ b/arch/x86/include/asm/fixmap.h
@@ -100,6 +100,10 @@
#ifdef CONFIG_X86_INTEL_MID
FIX_LNW_VRTC,
#endif
+ /* Fixmap entries to remap the GDTs, one per processor. */
+ FIX_GDT_REMAP_BEGIN,
+ FIX_GDT_REMAP_END = FIX_GDT_REMAP_BEGIN + NR_CPUS - 1,
+
__end_of_permanent_fixed_addresses,
/*
diff --git a/arch/x86/include/asm/gart.h b/arch/x86/include/asm/gart.h
index 156cd5d..1d26809 100644
--- a/arch/x86/include/asm/gart.h
+++ b/arch/x86/include/asm/gart.h
@@ -1,7 +1,7 @@
#ifndef _ASM_X86_GART_H
#define _ASM_X86_GART_H
-#include <asm/e820.h>
+#include <asm/e820/api.h>
extern void set_up_gart_resume(u32, u32);
@@ -97,7 +97,7 @@
printk(KERN_INFO "Aperture beyond 4GB. Ignoring.\n");
return 0;
}
- if (e820_any_mapped(aper_base, aper_base + aper_size, E820_RAM)) {
+ if (e820__mapped_any(aper_base, aper_base + aper_size, E820_TYPE_RAM)) {
printk(KERN_INFO "Aperture pointing to e820 RAM. Ignoring.\n");
return 0;
}
diff --git a/arch/x86/include/asm/intel-family.h b/arch/x86/include/asm/intel-family.h
index 9814db4..75b748a 100644
--- a/arch/x86/include/asm/intel-family.h
+++ b/arch/x86/include/asm/intel-family.h
@@ -12,6 +12,7 @@
*/
#define INTEL_FAM6_CORE_YONAH 0x0E
+
#define INTEL_FAM6_CORE2_MEROM 0x0F
#define INTEL_FAM6_CORE2_MEROM_L 0x16
#define INTEL_FAM6_CORE2_PENRYN 0x17
@@ -21,6 +22,7 @@
#define INTEL_FAM6_NEHALEM_G 0x1F /* Auburndale / Havendale */
#define INTEL_FAM6_NEHALEM_EP 0x1A
#define INTEL_FAM6_NEHALEM_EX 0x2E
+
#define INTEL_FAM6_WESTMERE 0x25
#define INTEL_FAM6_WESTMERE_EP 0x2C
#define INTEL_FAM6_WESTMERE_EX 0x2F
@@ -36,9 +38,9 @@
#define INTEL_FAM6_HASWELL_GT3E 0x46
#define INTEL_FAM6_BROADWELL_CORE 0x3D
-#define INTEL_FAM6_BROADWELL_XEON_D 0x56
#define INTEL_FAM6_BROADWELL_GT3E 0x47
#define INTEL_FAM6_BROADWELL_X 0x4F
+#define INTEL_FAM6_BROADWELL_XEON_D 0x56
#define INTEL_FAM6_SKYLAKE_MOBILE 0x4E
#define INTEL_FAM6_SKYLAKE_DESKTOP 0x5E
@@ -59,8 +61,8 @@
#define INTEL_FAM6_ATOM_MERRIFIELD 0x4A /* Tangier */
#define INTEL_FAM6_ATOM_MOOREFIELD 0x5A /* Anniedale */
#define INTEL_FAM6_ATOM_GOLDMONT 0x5C
-#define INTEL_FAM6_ATOM_GEMINI_LAKE 0x7A
#define INTEL_FAM6_ATOM_DENVERTON 0x5F /* Goldmont Microserver */
+#define INTEL_FAM6_ATOM_GEMINI_LAKE 0x7A
/* Xeon Phi */
diff --git a/arch/x86/include/asm/intel_rdt.h b/arch/x86/include/asm/intel_rdt.h
index 0d64397..597dc49 100644
--- a/arch/x86/include/asm/intel_rdt.h
+++ b/arch/x86/include/asm/intel_rdt.h
@@ -12,6 +12,7 @@
#define IA32_L3_QOS_CFG 0xc81
#define IA32_L3_CBM_BASE 0xc90
#define IA32_L2_CBM_BASE 0xd10
+#define IA32_MBA_THRTL_BASE 0xd50
#define L3_QOS_CDP_ENABLE 0x01ULL
@@ -37,23 +38,30 @@
/* rdtgroup.flags */
#define RDT_DELETED 1
+/* rftype.flags */
+#define RFTYPE_FLAGS_CPUS_LIST 1
+
/* List of all resource groups */
extern struct list_head rdt_all_groups;
+extern int max_name_width, max_data_width;
+
int __init rdtgroup_init(void);
/**
* struct rftype - describe each file in the resctrl file system
- * @name: file name
- * @mode: access mode
- * @kf_ops: operations
- * @seq_show: show content of the file
- * @write: write to the file
+ * @name: File name
+ * @mode: Access mode
+ * @kf_ops: File operations
+ * @flags: File specific RFTYPE_FLAGS_* flags
+ * @seq_show: Show content of the file
+ * @write: Write to the file
*/
struct rftype {
char *name;
umode_t mode;
struct kernfs_ops *kf_ops;
+ unsigned long flags;
int (*seq_show)(struct kernfs_open_file *of,
struct seq_file *sf, void *v);
@@ -67,54 +75,21 @@
};
/**
- * struct rdt_resource - attributes of an RDT resource
- * @enabled: Is this feature enabled on this machine
- * @capable: Is this feature available on this machine
- * @name: Name to use in "schemata" file
- * @num_closid: Number of CLOSIDs available
- * @max_cbm: Largest Cache Bit Mask allowed
- * @min_cbm_bits: Minimum number of consecutive bits to be set
- * in a cache bit mask
- * @domains: All domains for this resource
- * @num_domains: Number of domains active
- * @msr_base: Base MSR address for CBMs
- * @tmp_cbms: Scratch space when updating schemata
- * @num_tmp_cbms: Number of CBMs in tmp_cbms
- * @cache_level: Which cache level defines scope of this domain
- * @cbm_idx_multi: Multiplier of CBM index
- * @cbm_idx_offset: Offset of CBM index. CBM index is computed by:
- * closid * cbm_idx_multi + cbm_idx_offset
- */
-struct rdt_resource {
- bool enabled;
- bool capable;
- char *name;
- int num_closid;
- int cbm_len;
- int min_cbm_bits;
- u32 max_cbm;
- struct list_head domains;
- int num_domains;
- int msr_base;
- u32 *tmp_cbms;
- int num_tmp_cbms;
- int cache_level;
- int cbm_idx_multi;
- int cbm_idx_offset;
-};
-
-/**
* struct rdt_domain - group of cpus sharing an RDT resource
* @list: all instances of this resource
* @id: unique id for this instance
* @cpu_mask: which cpus share this resource
- * @cbm: array of cache bit masks (indexed by CLOSID)
+ * @ctrl_val: array of cache or mem ctrl values (indexed by CLOSID)
+ * @new_ctrl: new ctrl value to be loaded
+ * @have_new_ctrl: did user provide new_ctrl for this domain
*/
struct rdt_domain {
struct list_head list;
int id;
struct cpumask cpu_mask;
- u32 *cbm;
+ u32 *ctrl_val;
+ u32 new_ctrl;
+ bool have_new_ctrl;
};
/**
@@ -129,6 +104,83 @@
int high;
};
+/**
+ * struct rdt_cache - Cache allocation related data
+ * @cbm_len: Length of the cache bit mask
+ * @min_cbm_bits: Minimum number of consecutive bits to be set
+ * @cbm_idx_mult: Multiplier of CBM index
+ * @cbm_idx_offset: Offset of CBM index. CBM index is computed by:
+ * closid * cbm_idx_multi + cbm_idx_offset
+ * in a cache bit mask
+ */
+struct rdt_cache {
+ unsigned int cbm_len;
+ unsigned int min_cbm_bits;
+ unsigned int cbm_idx_mult;
+ unsigned int cbm_idx_offset;
+};
+
+/**
+ * struct rdt_membw - Memory bandwidth allocation related data
+ * @max_delay: Max throttle delay. Delay is the hardware
+ * representation for memory bandwidth.
+ * @min_bw: Minimum memory bandwidth percentage user can request
+ * @bw_gran: Granularity at which the memory bandwidth is allocated
+ * @delay_linear: True if memory B/W delay is in linear scale
+ * @mb_map: Mapping of memory B/W percentage to memory B/W delay
+ */
+struct rdt_membw {
+ u32 max_delay;
+ u32 min_bw;
+ u32 bw_gran;
+ u32 delay_linear;
+ u32 *mb_map;
+};
+
+/**
+ * struct rdt_resource - attributes of an RDT resource
+ * @enabled: Is this feature enabled on this machine
+ * @capable: Is this feature available on this machine
+ * @name: Name to use in "schemata" file
+ * @num_closid: Number of CLOSIDs available
+ * @cache_level: Which cache level defines scope of this resource
+ * @default_ctrl: Specifies default cache cbm or memory B/W percent.
+ * @msr_base: Base MSR address for CBMs
+ * @msr_update: Function pointer to update QOS MSRs
+ * @data_width: Character width of data when displaying
+ * @domains: All domains for this resource
+ * @cache: Cache allocation related data
+ * @info_files: resctrl info files for the resource
+ * @nr_info_files: Number of info files
+ * @format_str: Per resource format string to show domain value
+ * @parse_ctrlval: Per resource function pointer to parse control values
+ */
+struct rdt_resource {
+ bool enabled;
+ bool capable;
+ char *name;
+ int num_closid;
+ int cache_level;
+ u32 default_ctrl;
+ unsigned int msr_base;
+ void (*msr_update) (struct rdt_domain *d, struct msr_param *m,
+ struct rdt_resource *r);
+ int data_width;
+ struct list_head domains;
+ struct rdt_cache cache;
+ struct rdt_membw membw;
+ struct rftype *info_files;
+ int nr_info_files;
+ const char *format_str;
+ int (*parse_ctrlval) (char *buf, struct rdt_resource *r,
+ struct rdt_domain *d);
+};
+
+void rdt_get_cache_infofile(struct rdt_resource *r);
+void rdt_get_mba_infofile(struct rdt_resource *r);
+int parse_cbm(char *buf, struct rdt_resource *r, struct rdt_domain *d);
+int parse_bw(char *buf, struct rdt_resource *r, struct rdt_domain *d);
+
extern struct mutex rdtgroup_mutex;
extern struct rdt_resource rdt_resources_all[];
@@ -142,6 +194,7 @@
RDT_RESOURCE_L3DATA,
RDT_RESOURCE_L3CODE,
RDT_RESOURCE_L2,
+ RDT_RESOURCE_MBA,
/* Must be the last */
RDT_NUM_RESOURCES,
@@ -149,7 +202,7 @@
#define for_each_capable_rdt_resource(r) \
for (r = rdt_resources_all; r < rdt_resources_all + RDT_NUM_RESOURCES;\
- r++) \
+ r++) \
if (r->capable)
#define for_each_enabled_rdt_resource(r) \
@@ -165,8 +218,16 @@
unsigned int full;
};
-/* CPUID.(EAX=10H, ECX=ResID=1).EDX */
-union cpuid_0x10_1_edx {
+/* CPUID.(EAX=10H, ECX=ResID=3).EAX */
+union cpuid_0x10_3_eax {
+ struct {
+ unsigned int max_delay:12;
+ } split;
+ unsigned int full;
+};
+
+/* CPUID.(EAX=10H, ECX=ResID).EDX */
+union cpuid_0x10_x_edx {
struct {
unsigned int cos_max:16;
} split;
@@ -175,7 +236,7 @@
DECLARE_PER_CPU_READ_MOSTLY(int, cpu_closid);
-void rdt_cbm_update(void *arg);
+void rdt_ctrl_update(void *arg);
struct rdtgroup *rdtgroup_kn_lock_live(struct kernfs_node *kn);
void rdtgroup_kn_unlock(struct kernfs_node *kn);
ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
diff --git a/arch/x86/include/asm/kasan.h b/arch/x86/include/asm/kasan.h
index 1410b56..f527b02 100644
--- a/arch/x86/include/asm/kasan.h
+++ b/arch/x86/include/asm/kasan.h
@@ -11,9 +11,12 @@
* 'kernel address space start' >> KASAN_SHADOW_SCALE_SHIFT
*/
#define KASAN_SHADOW_START (KASAN_SHADOW_OFFSET + \
- (0xffff800000000000ULL >> 3))
-/* 47 bits for kernel address -> (47 - 3) bits for shadow */
-#define KASAN_SHADOW_END (KASAN_SHADOW_START + (1ULL << (47 - 3)))
+ ((-1UL << __VIRTUAL_MASK_SHIFT) >> 3))
+/*
+ * 47 bits for kernel address -> (47 - 3) bits for shadow
+ * 56 bits for kernel address -> (56 - 3) bits for shadow
+ */
+#define KASAN_SHADOW_END (KASAN_SHADOW_START + (1ULL << (__VIRTUAL_MASK_SHIFT - 3)))
#ifndef __ASSEMBLY__
diff --git a/arch/x86/include/asm/kexec.h b/arch/x86/include/asm/kexec.h
index 282630e..70ef205 100644
--- a/arch/x86/include/asm/kexec.h
+++ b/arch/x86/include/asm/kexec.h
@@ -164,6 +164,7 @@
};
#else
struct kimage_arch {
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
diff --git a/arch/x86/include/asm/kprobes.h b/arch/x86/include/asm/kprobes.h
index 2005816..34b984c 100644
--- a/arch/x86/include/asm/kprobes.h
+++ b/arch/x86/include/asm/kprobes.h
@@ -72,14 +72,13 @@
/* copy of the original instruction */
kprobe_opcode_t *insn;
/*
- * boostable = -1: This instruction type is not boostable.
- * boostable = 0: This instruction type is boostable.
- * boostable = 1: This instruction has been boosted: we have
+ * boostable = false: This instruction type is not boostable.
+ * boostable = true: This instruction has been boosted: we have
* added a relative jump after the instruction copy in insn,
* so no single-step and fixup are needed (unless there's
* a post_handler or break_handler).
*/
- int boostable;
+ bool boostable;
bool if_modifier;
};
diff --git a/arch/x86/include/asm/mce.h b/arch/x86/include/asm/mce.h
index e638736..4fd5195 100644
--- a/arch/x86/include/asm/mce.h
+++ b/arch/x86/include/asm/mce.h
@@ -128,7 +128,7 @@
* debugging tools. Each entry is only valid when its finished flag
* is set.
*/
-struct mce_log {
+struct mce_log_buffer {
char signature[12]; /* "MACHINECHECK" */
unsigned len; /* = MCE_LOG_LEN */
unsigned next;
@@ -191,10 +191,12 @@
extern struct mca_msr_regs msr_ops;
enum mce_notifier_prios {
- MCE_PRIO_SRAO = INT_MAX,
- MCE_PRIO_EXTLOG = INT_MAX - 1,
- MCE_PRIO_NFIT = INT_MAX - 2,
- MCE_PRIO_EDAC = INT_MAX - 3,
+ MCE_PRIO_FIRST = INT_MAX,
+ MCE_PRIO_SRAO = INT_MAX - 1,
+ MCE_PRIO_EXTLOG = INT_MAX - 2,
+ MCE_PRIO_NFIT = INT_MAX - 3,
+ MCE_PRIO_EDAC = INT_MAX - 4,
+ MCE_PRIO_MCELOG = 1,
MCE_PRIO_LOWEST = 0,
};
diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h
index 306c7e12..68b329d 100644
--- a/arch/x86/include/asm/mmu_context.h
+++ b/arch/x86/include/asm/mmu_context.h
@@ -268,8 +268,4 @@
return __pkru_allows_pkey(vma_pkey(vma), write);
}
-static inline bool arch_pte_access_permitted(pte_t pte, bool write)
-{
- return __pkru_allows_pkey(pte_flags_pkey(pte_flags(pte)), write);
-}
#endif /* _ASM_X86_MMU_CONTEXT_H */
diff --git a/arch/x86/include/asm/mpspec.h b/arch/x86/include/asm/mpspec.h
index 3200704..831eb78 100644
--- a/arch/x86/include/asm/mpspec.h
+++ b/arch/x86/include/asm/mpspec.h
@@ -64,7 +64,7 @@
}
#ifdef CONFIG_X86_MPPARSE
-extern void early_reserve_e820_mpc_new(void);
+extern void e820__memblock_alloc_reserved_mpc_new(void);
extern int enable_update_mptable;
extern int default_mpc_apic_id(struct mpc_cpu *m);
extern void default_smp_read_mpc_oem(struct mpc_table *mpc);
@@ -76,7 +76,7 @@
extern void default_find_smp_config(void);
extern void default_get_smp_config(unsigned int early);
#else
-static inline void early_reserve_e820_mpc_new(void) { }
+static inline void e820__memblock_alloc_reserved_mpc_new(void) { }
#define enable_update_mptable 0
#define default_mpc_apic_id NULL
#define default_smp_read_mpc_oem NULL
diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
index 7c9c895..fba1007 100644
--- a/arch/x86/include/asm/mshyperv.h
+++ b/arch/x86/include/asm/mshyperv.h
@@ -176,4 +176,58 @@
bool hv_is_hypercall_page_setup(void);
void hyperv_cleanup(void);
#endif
+#ifdef CONFIG_HYPERV_TSCPAGE
+struct ms_hyperv_tsc_page *hv_get_tsc_page(void);
+static inline u64 hv_read_tsc_page(const struct ms_hyperv_tsc_page *tsc_pg)
+{
+ u64 scale, offset, cur_tsc;
+ u32 sequence;
+
+ /*
+ * The protocol for reading Hyper-V TSC page is specified in Hypervisor
+ * Top-Level Functional Specification ver. 3.0 and above. To get the
+ * reference time we must do the following:
+ * - READ ReferenceTscSequence
+ * A special '0' value indicates the time source is unreliable and we
+ * need to use something else. The currently published specification
+ * versions (up to 4.0b) contain a mistake and wrongly claim '-1'
+ * instead of '0' as the special value, see commit c35b82ef0294.
+ * - ReferenceTime =
+ * ((RDTSC() * ReferenceTscScale) >> 64) + ReferenceTscOffset
+ * - READ ReferenceTscSequence again. In case its value has changed
+ * since our first reading we need to discard ReferenceTime and repeat
+ * the whole sequence as the hypervisor was updating the page in
+ * between.
+ */
+ do {
+ sequence = READ_ONCE(tsc_pg->tsc_sequence);
+ if (!sequence)
+ return U64_MAX;
+ /*
+ * Make sure we read sequence before we read other values from
+ * TSC page.
+ */
+ smp_rmb();
+
+ scale = READ_ONCE(tsc_pg->tsc_scale);
+ offset = READ_ONCE(tsc_pg->tsc_offset);
+ cur_tsc = rdtsc_ordered();
+
+ /*
+ * Make sure we read sequence after we read all other values
+ * from TSC page.
+ */
+ smp_rmb();
+
+ } while (READ_ONCE(tsc_pg->tsc_sequence) != sequence);
+
+ return mul_u64_u64_shr(cur_tsc, scale, 64) + offset;
+}
+
+#else
+static inline struct ms_hyperv_tsc_page *hv_get_tsc_page(void)
+{
+ return NULL;
+}
+#endif
#endif
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index d8b5f8a..673f9ac 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -45,6 +45,8 @@
#define MSR_IA32_PERFCTR1 0x000000c2
#define MSR_FSB_FREQ 0x000000cd
#define MSR_PLATFORM_INFO 0x000000ce
+#define MSR_PLATFORM_INFO_CPUID_FAULT_BIT 31
+#define MSR_PLATFORM_INFO_CPUID_FAULT BIT_ULL(MSR_PLATFORM_INFO_CPUID_FAULT_BIT)
#define MSR_PKG_CST_CONFIG_CONTROL 0x000000e2
#define NHM_C3_AUTO_DEMOTE (1UL << 25)
@@ -127,6 +129,7 @@
/* DEBUGCTLMSR bits (others vary by model): */
#define DEBUGCTLMSR_LBR (1UL << 0) /* last branch recording */
+#define DEBUGCTLMSR_BTF_SHIFT 1
#define DEBUGCTLMSR_BTF (1UL << 1) /* single-step on branches */
#define DEBUGCTLMSR_TR (1UL << 6)
#define DEBUGCTLMSR_BTS (1UL << 7)
@@ -552,10 +555,12 @@
#define MSR_IA32_MISC_ENABLE_IP_PREF_DISABLE_BIT 39
#define MSR_IA32_MISC_ENABLE_IP_PREF_DISABLE (1ULL << MSR_IA32_MISC_ENABLE_IP_PREF_DISABLE_BIT)
-/* MISC_FEATURE_ENABLES non-architectural features */
-#define MSR_MISC_FEATURE_ENABLES 0x00000140
+/* MISC_FEATURES_ENABLES non-architectural features */
+#define MSR_MISC_FEATURES_ENABLES 0x00000140
-#define MSR_MISC_FEATURE_ENABLES_RING3MWAIT_BIT 1
+#define MSR_MISC_FEATURES_ENABLES_CPUID_FAULT_BIT 0
+#define MSR_MISC_FEATURES_ENABLES_CPUID_FAULT BIT_ULL(MSR_MISC_FEATURES_ENABLES_CPUID_FAULT_BIT)
+#define MSR_MISC_FEATURES_ENABLES_RING3MWAIT_BIT 1
#define MSR_IA32_TSC_DEADLINE 0x000006E0
diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h
index b3bebf9..b4a0d43 100644
--- a/arch/x86/include/asm/page_64.h
+++ b/arch/x86/include/asm/page_64.h
@@ -4,6 +4,7 @@
#include <asm/page_64_types.h>
#ifndef __ASSEMBLY__
+#include <asm/alternative.h>
/* duplicated to the one in bootmem.h */
extern unsigned long max_pfn;
@@ -34,7 +35,20 @@
#define pfn_valid(pfn) ((pfn) < max_pfn)
#endif
-void clear_page(void *page);
+void clear_page_orig(void *page);
+void clear_page_rep(void *page);
+void clear_page_erms(void *page);
+
+static inline void clear_page(void *page)
+{
+ alternative_call_2(clear_page_orig,
+ clear_page_rep, X86_FEATURE_REP_GOOD,
+ clear_page_erms, X86_FEATURE_ERMS,
+ "=D" (page),
+ "0" (page)
+ : "memory", "rax", "rcx");
+}
+
void copy_page(void *to, void *from);
#endif /* !__ASSEMBLY__ */
diff --git a/arch/x86/include/asm/page_64_types.h b/arch/x86/include/asm/page_64_types.h
index 9215e05..3f5f08b 100644
--- a/arch/x86/include/asm/page_64_types.h
+++ b/arch/x86/include/asm/page_64_types.h
@@ -36,7 +36,12 @@
* hypervisor to fit. Choosing 16 slots here is arbitrary, but it's
* what Xen requires.
*/
+#ifdef CONFIG_X86_5LEVEL
+#define __PAGE_OFFSET_BASE _AC(0xff10000000000000, UL)
+#else
#define __PAGE_OFFSET_BASE _AC(0xffff880000000000, UL)
+#endif
+
#ifdef CONFIG_RANDOMIZE_MEMORY
#define __PAGE_OFFSET page_offset_base
#else
@@ -46,8 +51,13 @@
#define __START_KERNEL_map _AC(0xffffffff80000000, UL)
/* See Documentation/x86/x86_64/mm.txt for a description of the memory map. */
+#ifdef CONFIG_X86_5LEVEL
+#define __PHYSICAL_MASK_SHIFT 52
+#define __VIRTUAL_MASK_SHIFT 56
+#else
#define __PHYSICAL_MASK_SHIFT 46
#define __VIRTUAL_MASK_SHIFT 47
+#endif
/*
* Kernel image size is limited to 1GiB due to the fixmap living in the
diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index 0489884..55fa56f 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -357,6 +357,16 @@
PVOP_VCALL1(pv_mmu_ops.release_pud, pfn);
}
+static inline void paravirt_alloc_p4d(struct mm_struct *mm, unsigned long pfn)
+{
+ PVOP_VCALL2(pv_mmu_ops.alloc_p4d, mm, pfn);
+}
+
+static inline void paravirt_release_p4d(unsigned long pfn)
+{
+ PVOP_VCALL1(pv_mmu_ops.release_p4d, pfn);
+}
+
static inline void pte_update(struct mm_struct *mm, unsigned long addr,
pte_t *ptep)
{
@@ -536,7 +546,7 @@
PVOP_VCALL2(pv_mmu_ops.set_pud, pudp,
val);
}
-#if CONFIG_PGTABLE_LEVELS == 4
+#if CONFIG_PGTABLE_LEVELS >= 4
static inline pud_t __pud(pudval_t val)
{
pudval_t ret;
@@ -565,16 +575,42 @@
return ret;
}
+static inline void pud_clear(pud_t *pudp)
+{
+ set_pud(pudp, __pud(0));
+}
+
+static inline void set_p4d(p4d_t *p4dp, p4d_t p4d)
+{
+ p4dval_t val = native_p4d_val(p4d);
+
+ if (sizeof(p4dval_t) > sizeof(long))
+ PVOP_VCALL3(pv_mmu_ops.set_p4d, p4dp,
+ val, (u64)val >> 32);
+ else
+ PVOP_VCALL2(pv_mmu_ops.set_p4d, p4dp,
+ val);
+}
+
+#if CONFIG_PGTABLE_LEVELS >= 5
+
+static inline p4d_t __p4d(p4dval_t val)
+{
+ p4dval_t ret = PVOP_CALLEE1(p4dval_t, pv_mmu_ops.make_p4d, val);
+
+ return (p4d_t) { ret };
+}
+
+static inline p4dval_t p4d_val(p4d_t p4d)
+{
+ return PVOP_CALLEE1(p4dval_t, pv_mmu_ops.p4d_val, p4d.p4d);
+}
+
static inline void set_pgd(pgd_t *pgdp, pgd_t pgd)
{
pgdval_t val = native_pgd_val(pgd);
- if (sizeof(pgdval_t) > sizeof(long))
- PVOP_VCALL3(pv_mmu_ops.set_pgd, pgdp,
- val, (u64)val >> 32);
- else
- PVOP_VCALL2(pv_mmu_ops.set_pgd, pgdp,
- val);
+ PVOP_VCALL2(pv_mmu_ops.set_pgd, pgdp, val);
}
static inline void pgd_clear(pgd_t *pgdp)
@@ -582,9 +618,11 @@
set_pgd(pgdp, __pgd(0));
}
-static inline void pud_clear(pud_t *pudp)
+#endif /* CONFIG_PGTABLE_LEVELS == 5 */
+
+static inline void p4d_clear(p4d_t *p4dp)
{
- set_pud(pudp, __pud(0));
+ set_p4d(p4dp, __p4d(0));
}
#endif /* CONFIG_PGTABLE_LEVELS == 4 */
diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h
index b060f96..7465d6f 100644
--- a/arch/x86/include/asm/paravirt_types.h
+++ b/arch/x86/include/asm/paravirt_types.h
@@ -238,9 +238,11 @@
void (*alloc_pte)(struct mm_struct *mm, unsigned long pfn);
void (*alloc_pmd)(struct mm_struct *mm, unsigned long pfn);
void (*alloc_pud)(struct mm_struct *mm, unsigned long pfn);
+ void (*alloc_p4d)(struct mm_struct *mm, unsigned long pfn);
void (*release_pte)(unsigned long pfn);
void (*release_pmd)(unsigned long pfn);
void (*release_pud)(unsigned long pfn);
+ void (*release_p4d)(unsigned long pfn);
/* Pagetable manipulation functions */
void (*set_pte)(pte_t *ptep, pte_t pteval);
@@ -279,12 +281,21 @@
struct paravirt_callee_save pmd_val;
struct paravirt_callee_save make_pmd;
-#if CONFIG_PGTABLE_LEVELS == 4
+#if CONFIG_PGTABLE_LEVELS >= 4
struct paravirt_callee_save pud_val;
struct paravirt_callee_save make_pud;
- void (*set_pgd)(pgd_t *pudp, pgd_t pgdval);
-#endif /* CONFIG_PGTABLE_LEVELS == 4 */
+ void (*set_p4d)(p4d_t *p4dp, p4d_t p4dval);
+
+#if CONFIG_PGTABLE_LEVELS >= 5
+ struct paravirt_callee_save p4d_val;
+ struct paravirt_callee_save make_p4d;
+
+ void (*set_pgd)(pgd_t *pgdp, pgd_t pgdval);
+#endif /* CONFIG_PGTABLE_LEVELS >= 5 */
+
+#endif /* CONFIG_PGTABLE_LEVELS >= 4 */
+
#endif /* CONFIG_PGTABLE_LEVELS >= 3 */
struct pv_lazy_ops lazy_mode;
diff --git a/arch/x86/include/asm/pci_x86.h b/arch/x86/include/asm/pci_x86.h
index d08eacd2..9f1b21f 100644
--- a/arch/x86/include/asm/pci_x86.h
+++ b/arch/x86/include/asm/pci_x86.h
@@ -4,6 +4,8 @@
* (c) 1999 Martin Mares <mj@ucw.cz>
*/
+#include <linux/ioport.h>
+
#undef DEBUG
#ifdef DEBUG
diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h
index b6d4259..b2d0cd8 100644
--- a/arch/x86/include/asm/pgalloc.h
+++ b/arch/x86/include/asm/pgalloc.h
@@ -17,9 +17,11 @@
static inline void paravirt_alloc_pmd_clone(unsigned long pfn, unsigned long clonepfn,
unsigned long start, unsigned long count) {}
static inline void paravirt_alloc_pud(struct mm_struct *mm, unsigned long pfn) {}
+static inline void paravirt_alloc_p4d(struct mm_struct *mm, unsigned long pfn) {}
static inline void paravirt_release_pte(unsigned long pfn) {}
static inline void paravirt_release_pmd(unsigned long pfn) {}
static inline void paravirt_release_pud(unsigned long pfn) {}
+static inline void paravirt_release_p4d(unsigned long pfn) {}
#endif
/*
@@ -121,10 +123,10 @@
#endif /* CONFIG_X86_PAE */
#if CONFIG_PGTABLE_LEVELS > 3
-static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud)
+static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4d, pud_t *pud)
{
paravirt_alloc_pud(mm, __pa(pud) >> PAGE_SHIFT);
- set_pgd(pgd, __pgd(_PAGE_TABLE | __pa(pud)));
+ set_p4d(p4d, __p4d(_PAGE_TABLE | __pa(pud)));
}
static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr)
@@ -150,6 +152,37 @@
___pud_free_tlb(tlb, pud);
}
+#if CONFIG_PGTABLE_LEVELS > 4
+static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4d)
+{
+ paravirt_alloc_p4d(mm, __pa(p4d) >> PAGE_SHIFT);
+ set_pgd(pgd, __pgd(_PAGE_TABLE | __pa(p4d)));
+}
+
+static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr)
+{
+ gfp_t gfp = GFP_KERNEL_ACCOUNT;
+
+ if (mm == &init_mm)
+ gfp &= ~__GFP_ACCOUNT;
+ return (p4d_t *)get_zeroed_page(gfp);
+}
+
+static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d)
+{
+ BUG_ON((unsigned long)p4d & (PAGE_SIZE-1));
+ free_page((unsigned long)p4d);
+}
+
+extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d);
+
+static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d,
+ unsigned long address)
+{
+ ___p4d_free_tlb(tlb, p4d);
+}
+
+#endif /* CONFIG_PGTABLE_LEVELS > 4 */
#endif /* CONFIG_PGTABLE_LEVELS > 3 */
#endif /* CONFIG_PGTABLE_LEVELS > 2 */
diff --git a/arch/x86/include/asm/pgtable-2level_types.h b/arch/x86/include/asm/pgtable-2level_types.h
index 3925764..373ab1d 100644
--- a/arch/x86/include/asm/pgtable-2level_types.h
+++ b/arch/x86/include/asm/pgtable-2level_types.h
@@ -7,6 +7,7 @@
typedef unsigned long pteval_t;
typedef unsigned long pmdval_t;
typedef unsigned long pudval_t;
+typedef unsigned long p4dval_t;
typedef unsigned long pgdval_t;
typedef unsigned long pgprotval_t;
diff --git a/arch/x86/include/asm/pgtable-3level_types.h b/arch/x86/include/asm/pgtable-3level_types.h
index bcc8962..b8a4341 100644
--- a/arch/x86/include/asm/pgtable-3level_types.h
+++ b/arch/x86/include/asm/pgtable-3level_types.h
@@ -7,6 +7,7 @@
typedef u64 pteval_t;
typedef u64 pmdval_t;
typedef u64 pudval_t;
+typedef u64 p4dval_t;
typedef u64 pgdval_t;
typedef u64 pgprotval_t;
diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index 585ee0d..f5af95a 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -2,8 +2,6 @@
#define _ASM_X86_PGTABLE_H
#include <asm/page.h>
-#include <asm/e820.h>
-
#include <asm/pgtable_types.h>
/*
@@ -53,11 +51,19 @@
#define set_pmd(pmdp, pmd) native_set_pmd(pmdp, pmd)
-#ifndef __PAGETABLE_PUD_FOLDED
+#ifndef __PAGETABLE_P4D_FOLDED
#define set_pgd(pgdp, pgd) native_set_pgd(pgdp, pgd)
#define pgd_clear(pgd) native_pgd_clear(pgd)
#endif
+#ifndef set_p4d
+# define set_p4d(p4dp, p4d) native_set_p4d(p4dp, p4d)
+#endif
+
+#ifndef __PAGETABLE_PUD_FOLDED
+#define p4d_clear(p4d) native_p4d_clear(p4d)
+#endif
+
#ifndef set_pud
# define set_pud(pudp, pud) native_set_pud(pudp, pud)
#endif
@@ -74,6 +80,11 @@
#define pgd_val(x) native_pgd_val(x)
#define __pgd(x) native_make_pgd(x)
+#ifndef __PAGETABLE_P4D_FOLDED
+#define p4d_val(x) native_p4d_val(x)
+#define __p4d(x) native_make_p4d(x)
+#endif
+
#ifndef __PAGETABLE_PUD_FOLDED
#define pud_val(x) native_pud_val(x)
#define __pud(x) native_make_pud(x)
@@ -179,6 +190,17 @@
return (pud_val(pud) & pud_pfn_mask(pud)) >> PAGE_SHIFT;
}
+static inline unsigned long p4d_pfn(p4d_t p4d)
+{
+ return (p4d_val(p4d) & p4d_pfn_mask(p4d)) >> PAGE_SHIFT;
+}
+
+static inline int p4d_large(p4d_t p4d)
+{
+ /* No 512 GiB pages yet */
+ return 0;
+}
+
#define pte_page(pte) pfn_to_page(pte_pfn(pte))
static inline int pmd_large(pmd_t pte)
@@ -538,6 +560,7 @@
#define pte_pgprot(x) __pgprot(pte_flags(x))
#define pmd_pgprot(x) __pgprot(pmd_flags(x))
#define pud_pgprot(x) __pgprot(pud_flags(x))
+#define p4d_pgprot(x) __pgprot(p4d_flags(x))
#define canon_pgprot(p) __pgprot(massage_pgprot(p))
@@ -587,6 +610,7 @@
#include <linux/mm_types.h>
#include <linux/mmdebug.h>
#include <linux/log2.h>
+#include <asm/fixmap.h>
static inline int pte_none(pte_t pte)
{
@@ -770,7 +794,52 @@
}
#endif /* CONFIG_PGTABLE_LEVELS > 2 */
+static inline unsigned long pud_index(unsigned long address)
+{
+ return (address >> PUD_SHIFT) & (PTRS_PER_PUD - 1);
+}
+
#if CONFIG_PGTABLE_LEVELS > 3
+static inline int p4d_none(p4d_t p4d)
+{
+ return (native_p4d_val(p4d) & ~(_PAGE_KNL_ERRATUM_MASK)) == 0;
+}
+
+static inline int p4d_present(p4d_t p4d)
+{
+ return p4d_flags(p4d) & _PAGE_PRESENT;
+}
+
+static inline unsigned long p4d_page_vaddr(p4d_t p4d)
+{
+ return (unsigned long)__va(p4d_val(p4d) & p4d_pfn_mask(p4d));
+}
+
+/*
+ * Currently stuck as a macro due to indirect forward reference to
+ * linux/mmzone.h's __section_mem_map_addr() definition:
+ */
+#define p4d_page(p4d) \
+ pfn_to_page((p4d_val(p4d) & p4d_pfn_mask(p4d)) >> PAGE_SHIFT)
+
+/* Find an entry in the third-level page table.. */
+static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address)
+{
+ return (pud_t *)p4d_page_vaddr(*p4d) + pud_index(address);
+}
+
+static inline int p4d_bad(p4d_t p4d)
+{
+ return (p4d_flags(p4d) & ~(_KERNPG_TABLE | _PAGE_USER)) != 0;
+}
+#endif /* CONFIG_PGTABLE_LEVELS > 3 */
+
+static inline unsigned long p4d_index(unsigned long address)
+{
+ return (address >> P4D_SHIFT) & (PTRS_PER_P4D - 1);
+}
+
+#if CONFIG_PGTABLE_LEVELS > 4
static inline int pgd_present(pgd_t pgd)
{
return pgd_flags(pgd) & _PAGE_PRESENT;
@@ -788,14 +857,9 @@
#define pgd_page(pgd) pfn_to_page(pgd_val(pgd) >> PAGE_SHIFT)
/* to find an entry in a page-table-directory. */
-static inline unsigned long pud_index(unsigned long address)
+static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address)
{
- return (address >> PUD_SHIFT) & (PTRS_PER_PUD - 1);
-}
-
-static inline pud_t *pud_offset(pgd_t *pgd, unsigned long address)
-{
- return (pud_t *)pgd_page_vaddr(*pgd) + pud_index(address);
+ return (p4d_t *)pgd_page_vaddr(*pgd) + p4d_index(address);
}
static inline int pgd_bad(pgd_t pgd)
@@ -813,7 +877,7 @@
*/
return !native_pgd_val(pgd);
}
-#endif /* CONFIG_PGTABLE_LEVELS > 3 */
+#endif /* CONFIG_PGTABLE_LEVELS > 4 */
#endif /* __ASSEMBLY__ */
@@ -845,6 +909,7 @@
extern int direct_gbpages;
void init_mem_mapping(void);
void early_alloc_pgt_buf(void);
+extern void memblock_find_dma_reserve(void);
#ifdef CONFIG_X86_64
/* Realmode trampoline initialization. */
diff --git a/arch/x86/include/asm/pgtable_32.h b/arch/x86/include/asm/pgtable_32.h
index fbc7336..bfab5567 100644
--- a/arch/x86/include/asm/pgtable_32.h
+++ b/arch/x86/include/asm/pgtable_32.h
@@ -14,7 +14,6 @@
*/
#ifndef __ASSEMBLY__
#include <asm/processor.h>
-#include <asm/fixmap.h>
#include <linux/threads.h>
#include <asm/paravirt.h>
diff --git a/arch/x86/include/asm/pgtable_64.h b/arch/x86/include/asm/pgtable_64.h
index 73c7ccc..9991224 100644
--- a/arch/x86/include/asm/pgtable_64.h
+++ b/arch/x86/include/asm/pgtable_64.h
@@ -35,15 +35,22 @@
#define pud_ERROR(e) \
pr_err("%s:%d: bad pud %p(%016lx)\n", \
__FILE__, __LINE__, &(e), pud_val(e))
+
+#if CONFIG_PGTABLE_LEVELS >= 5
+#define p4d_ERROR(e) \
+ pr_err("%s:%d: bad p4d %p(%016lx)\n", \
+ __FILE__, __LINE__, &(e), p4d_val(e))
+#endif
+
#define pgd_ERROR(e) \
pr_err("%s:%d: bad pgd %p(%016lx)\n", \
__FILE__, __LINE__, &(e), pgd_val(e))
struct mm_struct;
+void set_pte_vaddr_p4d(p4d_t *p4d_page, unsigned long vaddr, pte_t new_pte);
void set_pte_vaddr_pud(pud_t *pud_page, unsigned long vaddr, pte_t new_pte);
-
static inline void native_pte_clear(struct mm_struct *mm, unsigned long addr,
pte_t *ptep)
{
@@ -121,6 +128,20 @@
#endif
}
+static inline void native_set_p4d(p4d_t *p4dp, p4d_t p4d)
+{
+ *p4dp = p4d;
+}
+
+static inline void native_p4d_clear(p4d_t *p4d)
+{
+#ifdef CONFIG_X86_5LEVEL
+ native_set_p4d(p4d, native_make_p4d(0));
+#else
+ native_set_p4d(p4d, (p4d_t) { .pgd = native_make_pgd(0)});
+#endif
+}
+
static inline void native_set_pgd(pgd_t *pgdp, pgd_t pgd)
{
*pgdp = pgd;
diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h
index 3a26420..06470da 100644
--- a/arch/x86/include/asm/pgtable_64_types.h
+++ b/arch/x86/include/asm/pgtable_64_types.h
@@ -13,6 +13,7 @@
typedef unsigned long pteval_t;
typedef unsigned long pmdval_t;
typedef unsigned long pudval_t;
+typedef unsigned long p4dval_t;
typedef unsigned long pgdval_t;
typedef unsigned long pgprotval_t;
@@ -22,12 +23,32 @@
#define SHARED_KERNEL_PMD 0
+#ifdef CONFIG_X86_5LEVEL
+
+/*
+ * PGDIR_SHIFT determines what a top-level page table entry can map
+ */
+#define PGDIR_SHIFT 48
+#define PTRS_PER_PGD 512
+
+/*
+ * 4th level page in 5-level paging case
+ */
+#define P4D_SHIFT 39
+#define PTRS_PER_P4D 512
+#define P4D_SIZE (_AC(1, UL) << P4D_SHIFT)
+#define P4D_MASK (~(P4D_SIZE - 1))
+
+#else /* CONFIG_X86_5LEVEL */
+
/*
* PGDIR_SHIFT determines what a top-level page table entry can map
*/
#define PGDIR_SHIFT 39
#define PTRS_PER_PGD 512
+#endif /* CONFIG_X86_5LEVEL */
+
/*
* 3rd level page
*/
@@ -55,9 +76,15 @@
/* See Documentation/x86/x86_64/mm.txt for a description of the memory map. */
#define MAXMEM _AC(__AC(1, UL) << MAX_PHYSMEM_BITS, UL)
+#ifdef CONFIG_X86_5LEVEL
+#define VMALLOC_SIZE_TB _AC(16384, UL)
+#define __VMALLOC_BASE _AC(0xff92000000000000, UL)
+#define __VMEMMAP_BASE _AC(0xffd4000000000000, UL)
+#else
#define VMALLOC_SIZE_TB _AC(32, UL)
#define __VMALLOC_BASE _AC(0xffffc90000000000, UL)
#define __VMEMMAP_BASE _AC(0xffffea0000000000, UL)
+#endif
#ifdef CONFIG_RANDOMIZE_MEMORY
#define VMALLOC_START vmalloc_base
#define VMEMMAP_START vmemmap_base
@@ -67,10 +94,11 @@
#endif /* CONFIG_RANDOMIZE_MEMORY */
#define VMALLOC_END (VMALLOC_START + _AC((VMALLOC_SIZE_TB << 40) - 1, UL))
#define MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE)
-#define MODULES_END _AC(0xffffffffff000000, UL)
+/* The module sections ends with the start of the fixmap */
+#define MODULES_END __fix_to_virt(__end_of_fixed_addresses + 1)
#define MODULES_LEN (MODULES_END - MODULES_VADDR)
#define ESPFIX_PGD_ENTRY _AC(-2, UL)
-#define ESPFIX_BASE_ADDR (ESPFIX_PGD_ENTRY << PGDIR_SHIFT)
+#define ESPFIX_BASE_ADDR (ESPFIX_PGD_ENTRY << P4D_SHIFT)
#define EFI_VA_START ( -4 * (_AC(1, UL) << 30))
#define EFI_VA_END (-68 * (_AC(1, UL) << 30))
diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h
index 6248433..bf9638e 100644
--- a/arch/x86/include/asm/pgtable_types.h
+++ b/arch/x86/include/asm/pgtable_types.h
@@ -272,9 +272,28 @@
return native_pgd_val(pgd) & PTE_FLAGS_MASK;
}
-#if CONFIG_PGTABLE_LEVELS > 3
-#include <asm-generic/5level-fixup.h>
+#if CONFIG_PGTABLE_LEVELS > 4
+typedef struct { p4dval_t p4d; } p4d_t;
+static inline p4d_t native_make_p4d(pudval_t val)
+{
+ return (p4d_t) { val };
+}
+
+static inline p4dval_t native_p4d_val(p4d_t p4d)
+{
+ return p4d.p4d;
+}
+#else
+#include <asm-generic/pgtable-nop4d.h>
+
+static inline p4dval_t native_p4d_val(p4d_t p4d)
+{
+ return native_pgd_val(p4d.pgd);
+}
+#endif
+
+#if CONFIG_PGTABLE_LEVELS > 3
typedef struct { pudval_t pud; } pud_t;
static inline pud_t native_make_pud(pmdval_t val)
@@ -287,12 +306,11 @@
return pud.pud;
}
#else
-#define __ARCH_USE_5LEVEL_HACK
#include <asm-generic/pgtable-nopud.h>
static inline pudval_t native_pud_val(pud_t pud)
{
- return native_pgd_val(pud.pgd);
+ return native_pgd_val(pud.p4d.pgd);
}
#endif
@@ -309,15 +327,30 @@
return pmd.pmd;
}
#else
-#define __ARCH_USE_5LEVEL_HACK
#include <asm-generic/pgtable-nopmd.h>
static inline pmdval_t native_pmd_val(pmd_t pmd)
{
- return native_pgd_val(pmd.pud.pgd);
+ return native_pgd_val(pmd.pud.p4d.pgd);
}
#endif
+static inline p4dval_t p4d_pfn_mask(p4d_t p4d)
+{
+ /* No 512 GiB huge pages yet */
+ return PTE_PFN_MASK;
+}
+
+static inline p4dval_t p4d_flags_mask(p4d_t p4d)
+{
+ return ~p4d_pfn_mask(p4d);
+}
+
+static inline p4dval_t p4d_flags(p4d_t p4d)
+{
+ return native_p4d_val(p4d) & p4d_flags_mask(p4d);
+}
+
static inline pudval_t pud_pfn_mask(pud_t pud)
{
if (native_pud_val(pud) & _PAGE_PSE)
@@ -461,6 +494,7 @@
PG_LEVEL_4K,
PG_LEVEL_2M,
PG_LEVEL_1G,
+ PG_LEVEL_512G,
PG_LEVEL_NUM
};
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index f385eca..3cada99 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -80,7 +80,7 @@
/*
* CPU type and hardware bug flags. Kept separately for each CPU.
- * Members of this structure are referenced in head.S, so think twice
+ * Members of this structure are referenced in head_32.S, so think twice
* before touching them. [mj]
*/
@@ -89,14 +89,7 @@
__u8 x86_vendor; /* CPU vendor */
__u8 x86_model;
__u8 x86_mask;
-#ifdef CONFIG_X86_32
- char wp_works_ok; /* It doesn't on 386's */
-
- /* Problems on some 486Dx4's and old 386's: */
- char rfu;
- char pad0;
- char pad1;
-#else
+#ifdef CONFIG_X86_64
/* Number of 4K pages in DTLB/ITLB combined(in pages): */
int x86_tlbsize;
#endif
@@ -716,6 +709,8 @@
extern void cpu_set_gdt(int);
extern void switch_to_new_gdt(int);
+extern void load_direct_gdt(int);
+extern void load_fixmap_gdt(int);
extern void load_percpu_segment(int);
extern void cpu_init(void);
@@ -797,6 +792,7 @@
/*
* User space process size: 3GB (default).
*/
+#define IA32_PAGE_OFFSET PAGE_OFFSET
#define TASK_SIZE PAGE_OFFSET
#define TASK_SIZE_MAX TASK_SIZE
#define STACK_TOP TASK_SIZE
@@ -873,7 +869,8 @@
* This decides where the kernel will search for a free chunk of vm
* space during mmap's.
*/
-#define TASK_UNMAPPED_BASE (PAGE_ALIGN(TASK_SIZE / 3))
+#define __TASK_UNMAPPED_BASE(task_size) (PAGE_ALIGN(task_size / 3))
+#define TASK_UNMAPPED_BASE __TASK_UNMAPPED_BASE(TASK_SIZE)
#define KSTK_EIP(task) (task_pt_regs(task)->ip)
@@ -884,6 +881,8 @@
extern int get_tsc_mode(unsigned long adr);
extern int set_tsc_mode(unsigned int val);
+DECLARE_PER_CPU(u64, msr_misc_features_shadow);
+
/* Register/unregister a process' MPX related resource */
#define MPX_ENABLE_MANAGEMENT() mpx_enable_management()
#define MPX_DISABLE_MANAGEMENT() mpx_disable_management()
diff --git a/arch/x86/include/asm/proto.h b/arch/x86/include/asm/proto.h
index 9b9b30b1..8d3964f 100644
--- a/arch/x86/include/asm/proto.h
+++ b/arch/x86/include/asm/proto.h
@@ -9,6 +9,7 @@
#ifdef CONFIG_X86_64
void entry_SYSCALL_64(void);
+long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2);
#endif
#ifdef CONFIG_X86_32
@@ -30,6 +31,7 @@
extern int reboot_force;
-long do_arch_prctl(struct task_struct *task, int code, unsigned long addr);
+long do_arch_prctl_common(struct task_struct *task, int option,
+ unsigned long cpuid_enabled);
#endif /* _ASM_X86_PROTO_H */
diff --git a/arch/x86/include/asm/reboot.h b/arch/x86/include/asm/reboot.h
index 2cb1cc2..fc62ba8 100644
--- a/arch/x86/include/asm/reboot.h
+++ b/arch/x86/include/asm/reboot.h
@@ -15,6 +15,7 @@
};
extern struct machine_ops machine_ops;
+extern int crashing_cpu;
void native_machine_crash_shutdown(struct pt_regs *regs);
void native_machine_shutdown(void);
diff --git a/arch/x86/include/asm/required-features.h b/arch/x86/include/asm/required-features.h
index fac9a5c..d91ba04 100644
--- a/arch/x86/include/asm/required-features.h
+++ b/arch/x86/include/asm/required-features.h
@@ -53,6 +53,12 @@
# define NEED_MOVBE 0
#endif
+#ifdef CONFIG_X86_5LEVEL
+# define NEED_LA57 (1<<(X86_FEATURE_LA57 & 31))
+#else
+# define NEED_LA57 0
+#endif
+
#ifdef CONFIG_X86_64
#ifdef CONFIG_PARAVIRT
/* Paravirtualized systems may not have PSE or PGE available */
@@ -98,7 +104,7 @@
#define REQUIRED_MASK13 0
#define REQUIRED_MASK14 0
#define REQUIRED_MASK15 0
-#define REQUIRED_MASK16 0
+#define REQUIRED_MASK16 (NEED_LA57)
#define REQUIRED_MASK17 0
#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 18)
diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h
index 026ea82..47103ec 100644
--- a/arch/x86/include/asm/smp.h
+++ b/arch/x86/include/asm/smp.h
@@ -149,6 +149,19 @@
#define cpu_physical_id(cpu) per_cpu(x86_cpu_to_apicid, cpu)
#define cpu_acpi_id(cpu) per_cpu(x86_cpu_to_acpiid, cpu)
+/*
+ * This function is needed by all SMP systems. It must _always_ be valid
+ * from the initial startup. We map APIC_BASE very early in page_setup(),
+ * so this is correct in the x86 case.
+ */
+#define raw_smp_processor_id() (this_cpu_read(cpu_number))
+
+#ifdef CONFIG_X86_32
+extern int safe_smp_processor_id(void);
+#else
+# define safe_smp_processor_id() smp_processor_id()
+#endif
+
#else /* !CONFIG_SMP */
#define wbinvd_on_cpu(cpu) wbinvd()
static inline int wbinvd_on_all_cpus(void)
@@ -161,22 +174,6 @@
extern unsigned disabled_cpus;
-#ifdef CONFIG_X86_32_SMP
-/*
- * This function is needed by all SMP systems. It must _always_ be valid
- * from the initial startup. We map APIC_BASE very early in page_setup(),
- * so this is correct in the x86 case.
- */
-#define raw_smp_processor_id() (this_cpu_read(cpu_number))
-extern int safe_smp_processor_id(void);
-
-#elif defined(CONFIG_X86_64_SMP)
-#define raw_smp_processor_id() (this_cpu_read(cpu_number))
-
-#define safe_smp_processor_id() smp_processor_id()
-
-#endif
-
#ifdef CONFIG_X86_LOCAL_APIC
#ifndef CONFIG_X86_64
@@ -191,11 +188,7 @@
extern int hard_smp_processor_id(void);
#else /* CONFIG_X86_LOCAL_APIC */
-
-# ifndef CONFIG_SMP
-# define hard_smp_processor_id() 0
-# endif
-
+#define hard_smp_processor_id() 0
#endif /* CONFIG_X86_LOCAL_APIC */
#ifdef CONFIG_DEBUG_NMI_SELFTEST
diff --git a/arch/x86/include/asm/sparsemem.h b/arch/x86/include/asm/sparsemem.h
index 4517d6b..1f5bee2c 100644
--- a/arch/x86/include/asm/sparsemem.h
+++ b/arch/x86/include/asm/sparsemem.h
@@ -26,8 +26,13 @@
# endif
#else /* CONFIG_X86_32 */
# define SECTION_SIZE_BITS 27 /* matt - 128 is convenient right now */
-# define MAX_PHYSADDR_BITS 44
-# define MAX_PHYSMEM_BITS 46
+# ifdef CONFIG_X86_5LEVEL
+# define MAX_PHYSADDR_BITS 52
+# define MAX_PHYSMEM_BITS 52
+# else
+# define MAX_PHYSADDR_BITS 44
+# define MAX_PHYSMEM_BITS 46
+# endif
#endif
#endif /* CONFIG_SPARSEMEM */
diff --git a/arch/x86/include/asm/stackprotector.h b/arch/x86/include/asm/stackprotector.h
index 58505f0..dcbd9bc 100644
--- a/arch/x86/include/asm/stackprotector.h
+++ b/arch/x86/include/asm/stackprotector.h
@@ -87,7 +87,7 @@
{
#ifdef CONFIG_X86_32
unsigned long canary = (unsigned long)&per_cpu(stack_canary, cpu);
- struct desc_struct *gdt_table = get_cpu_gdt_table(cpu);
+ struct desc_struct *gdt_table = get_cpu_gdt_rw(cpu);
struct desc_struct desc;
desc = gdt_table[GDT_ENTRY_STACK_CANARY];
diff --git a/arch/x86/include/asm/thread_info.h b/arch/x86/include/asm/thread_info.h
index ad6f5eb0..e00e1bd 100644
--- a/arch/x86/include/asm/thread_info.h
+++ b/arch/x86/include/asm/thread_info.h
@@ -73,9 +73,6 @@
* thread information flags
* - these are process state flags that various assembly files
* may need to access
- * - pending work-to-be-done flags are in LSW
- * - other flags in MSW
- * Warning: layout of LSW is hardcoded in entry.S
*/
#define TIF_SYSCALL_TRACE 0 /* syscall trace active */
#define TIF_NOTIFY_RESUME 1 /* callback before returning to user */
@@ -87,6 +84,8 @@
#define TIF_SECCOMP 8 /* secure computing */
#define TIF_USER_RETURN_NOTIFY 11 /* notify kernel of userspace return */
#define TIF_UPROBE 12 /* breakpointed or singlestepping */
+#define TIF_PATCH_PENDING 13 /* pending live patching update */
+#define TIF_NOCPUID 15 /* CPUID is not accessible in userland */
#define TIF_NOTSC 16 /* TSC is not accessible in userland */
#define TIF_IA32 17 /* IA32 compatibility process */
#define TIF_NOHZ 19 /* in adaptive nohz mode */
@@ -103,13 +102,15 @@
#define _TIF_SYSCALL_TRACE (1 << TIF_SYSCALL_TRACE)
#define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)
#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)
-#define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP)
#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)
+#define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP)
#define _TIF_SYSCALL_EMU (1 << TIF_SYSCALL_EMU)
#define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT)
#define _TIF_SECCOMP (1 << TIF_SECCOMP)
#define _TIF_USER_RETURN_NOTIFY (1 << TIF_USER_RETURN_NOTIFY)
#define _TIF_UPROBE (1 << TIF_UPROBE)
+#define _TIF_PATCH_PENDING (1 << TIF_PATCH_PENDING)
+#define _TIF_NOCPUID (1 << TIF_NOCPUID)
#define _TIF_NOTSC (1 << TIF_NOTSC)
#define _TIF_IA32 (1 << TIF_IA32)
#define _TIF_NOHZ (1 << TIF_NOHZ)
@@ -133,12 +134,14 @@
/* work to do on any return to user space */
#define _TIF_ALLWORK_MASK \
- ((0x0000FFFF & ~_TIF_SECCOMP) | _TIF_SYSCALL_TRACEPOINT | \
- _TIF_NOHZ)
+ (_TIF_SYSCALL_TRACE | _TIF_NOTIFY_RESUME | _TIF_SIGPENDING | \
+ _TIF_NEED_RESCHED | _TIF_SINGLESTEP | _TIF_SYSCALL_EMU | \
+ _TIF_SYSCALL_AUDIT | _TIF_USER_RETURN_NOTIFY | _TIF_UPROBE | \
+ _TIF_PATCH_PENDING | _TIF_NOHZ | _TIF_SYSCALL_TRACEPOINT)
/* flags to check in __switch_to() */
#define _TIF_WORK_CTXSW \
- (_TIF_IO_BITMAP|_TIF_NOTSC|_TIF_BLOCKSTEP)
+ (_TIF_IO_BITMAP|_TIF_NOCPUID|_TIF_NOTSC|_TIF_BLOCKSTEP)
#define _TIF_WORK_CTXSW_PREV (_TIF_WORK_CTXSW|_TIF_USER_RETURN_NOTIFY)
#define _TIF_WORK_CTXSW_NEXT (_TIF_WORK_CTXSW)
@@ -168,9 +171,9 @@
* entirely contained by a single stack frame.
*
* Returns:
- * 1 if within a frame
- * -1 if placed across a frame boundary (or outside stack)
- * 0 unable to determine (no frame pointers, etc)
+ * GOOD_FRAME if within a frame
+ * BAD_STACK if placed across a frame boundary (or outside stack)
+ * NOT_STACK unable to determine (no frame pointers, etc)
*/
static inline int arch_within_stack_frames(const void * const stack,
const void * const stackend,
@@ -197,13 +200,14 @@
* the copy as invalid.
*/
if (obj + len <= frame)
- return obj >= oldframe + 2 * sizeof(void *) ? 1 : -1;
+ return obj >= oldframe + 2 * sizeof(void *) ?
+ GOOD_FRAME : BAD_STACK;
oldframe = frame;
frame = *(const void * const *)frame;
}
- return -1;
+ return BAD_STACK;
#else
- return 0;
+ return NOT_STACK;
#endif
}
@@ -239,6 +243,8 @@
extern void arch_task_cache_init(void);
extern int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
extern void arch_release_task_struct(struct task_struct *tsk);
+extern void arch_setup_new_exec(void);
+#define arch_setup_new_exec arch_setup_new_exec
#endif /* !__ASSEMBLY__ */
#endif /* _ASM_X86_THREAD_INFO_H */
diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h
index fc5abff..6ed9ea4 100644
--- a/arch/x86/include/asm/tlbflush.h
+++ b/arch/x86/include/asm/tlbflush.h
@@ -110,6 +110,16 @@
}
}
+static inline void cr4_toggle_bits(unsigned long mask)
+{
+ unsigned long cr4;
+
+ cr4 = this_cpu_read(cpu_tlbstate.cr4);
+ cr4 ^= mask;
+ this_cpu_write(cpu_tlbstate.cr4, cr4);
+ __write_cr4(cr4);
+}
+
/* Read the CR4 shadow. */
static inline unsigned long cr4_read_shadow(void)
{
@@ -205,7 +215,6 @@
/*
* TLB flushing:
*
- * - flush_tlb() flushes the current mm struct TLBs
* - flush_tlb_all() flushes all processes TLBs
* - flush_tlb_mm(mm) flushes the specified mm context TLB's
* - flush_tlb_page(vma, vmaddr) flushes one page
@@ -237,11 +246,6 @@
__flush_tlb_all();
}
-static inline void flush_tlb(void)
-{
- __flush_tlb_up();
-}
-
static inline void local_flush_tlb(void)
{
__flush_tlb_up();
@@ -303,14 +307,11 @@
flush_tlb_mm_range(vma->vm_mm, start, end, vma->vm_flags)
extern void flush_tlb_all(void);
-extern void flush_tlb_current_task(void);
extern void flush_tlb_page(struct vm_area_struct *, unsigned long);
extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
unsigned long end, unsigned long vmflag);
extern void flush_tlb_kernel_range(unsigned long start, unsigned long end);
-#define flush_tlb() flush_tlb_current_task()
-
void native_flush_tlb_others(const struct cpumask *cpumask,
struct mm_struct *mm,
unsigned long start, unsigned long end);
diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h
index ea14831..68766b2 100644
--- a/arch/x86/include/asm/uaccess.h
+++ b/arch/x86/include/asm/uaccess.h
@@ -3,19 +3,14 @@
/*
* User space memory access functions
*/
-#include <linux/errno.h>
#include <linux/compiler.h>
#include <linux/kasan-checks.h>
-#include <linux/thread_info.h>
#include <linux/string.h>
#include <asm/asm.h>
#include <asm/page.h>
#include <asm/smap.h>
#include <asm/extable.h>
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
/*
* The fs value determines whether argument validity checking should be
* performed or not. If get_fs() == USER_DS, checking is performed, with
@@ -384,6 +379,18 @@
: "=r" (err), ltype(x) \
: "m" (__m(addr)), "i" (errret), "0" (err))
+#define __get_user_asm_nozero(x, addr, err, itype, rtype, ltype, errret) \
+ asm volatile("\n" \
+ "1: mov"itype" %2,%"rtype"1\n" \
+ "2:\n" \
+ ".section .fixup,\"ax\"\n" \
+ "3: mov %3,%0\n" \
+ " jmp 2b\n" \
+ ".previous\n" \
+ _ASM_EXTABLE(1b, 3b) \
+ : "=r" (err), ltype(x) \
+ : "m" (__m(addr)), "i" (errret), "0" (err))
+
/*
* This doesn't do __uaccess_begin/end - the exception handling
* around it must do that.
@@ -675,59 +682,6 @@
# include <asm/uaccess_64.h>
#endif
-unsigned long __must_check _copy_from_user(void *to, const void __user *from,
- unsigned n);
-unsigned long __must_check _copy_to_user(void __user *to, const void *from,
- unsigned n);
-
-extern void __compiletime_error("usercopy buffer size is too small")
-__bad_copy_user(void);
-
-static inline void copy_user_overflow(int size, unsigned long count)
-{
- WARN(1, "Buffer overflow detected (%d < %lu)!\n", size, count);
-}
-
-static __always_inline unsigned long __must_check
-copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- int sz = __compiletime_object_size(to);
-
- might_fault();
-
- kasan_check_write(to, n);
-
- if (likely(sz < 0 || sz >= n)) {
- check_object_size(to, n, false);
- n = _copy_from_user(to, from, n);
- } else if (!__builtin_constant_p(n))
- copy_user_overflow(sz, n);
- else
- __bad_copy_user();
-
- return n;
-}
-
-static __always_inline unsigned long __must_check
-copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- int sz = __compiletime_object_size(from);
-
- kasan_check_read(from, n);
-
- might_fault();
-
- if (likely(sz < 0 || sz >= n)) {
- check_object_size(from, n, true);
- n = _copy_to_user(to, from, n);
- } else if (!__builtin_constant_p(n))
- copy_user_overflow(sz, n);
- else
- __bad_copy_user();
-
- return n;
-}
-
/*
* We rely on the nested NMI work to allow atomic faults from the NMI path; the
* nested NMI paths are careful to preserve CR2.
diff --git a/arch/x86/include/asm/uaccess_32.h b/arch/x86/include/asm/uaccess_32.h
index 7d3bdd1..aeda9bb 100644
--- a/arch/x86/include/asm/uaccess_32.h
+++ b/arch/x86/include/asm/uaccess_32.h
@@ -4,149 +4,52 @@
/*
* User space memory access functions
*/
-#include <linux/errno.h>
-#include <linux/thread_info.h>
#include <linux/string.h>
#include <asm/asm.h>
#include <asm/page.h>
-unsigned long __must_check __copy_to_user_ll
- (void __user *to, const void *from, unsigned long n);
-unsigned long __must_check __copy_from_user_ll
- (void *to, const void __user *from, unsigned long n);
-unsigned long __must_check __copy_from_user_ll_nozero
- (void *to, const void __user *from, unsigned long n);
-unsigned long __must_check __copy_from_user_ll_nocache
- (void *to, const void __user *from, unsigned long n);
+unsigned long __must_check __copy_user_ll
+ (void *to, const void *from, unsigned long n);
unsigned long __must_check __copy_from_user_ll_nocache_nozero
(void *to, const void __user *from, unsigned long n);
-/**
- * __copy_to_user_inatomic: - Copy a block of data into user space, with less checking.
- * @to: Destination address, in user space.
- * @from: Source address, in kernel space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only.
- *
- * Copy data from kernel space to user space. Caller must check
- * the specified block with access_ok() before calling this function.
- * The caller should also make sure he pins the user space address
- * so that we don't result in page fault and sleep.
- */
static __always_inline unsigned long __must_check
-__copy_to_user_inatomic(void __user *to, const void *from, unsigned long n)
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
{
- check_object_size(from, n, true);
- return __copy_to_user_ll(to, from, n);
-}
-
-/**
- * __copy_to_user: - Copy a block of data into user space, with less checking.
- * @to: Destination address, in user space.
- * @from: Source address, in kernel space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from kernel space to user space. Caller must check
- * the specified block with access_ok() before calling this function.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- */
-static __always_inline unsigned long __must_check
-__copy_to_user(void __user *to, const void *from, unsigned long n)
-{
- might_fault();
- return __copy_to_user_inatomic(to, from, n);
+ return __copy_user_ll((__force void *)to, from, n);
}
static __always_inline unsigned long
-__copy_from_user_inatomic(void *to, const void __user *from, unsigned long n)
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
- return __copy_from_user_ll_nozero(to, from, n);
-}
-
-/**
- * __copy_from_user: - Copy a block of data from user space, with less checking.
- * @to: Destination address, in kernel space.
- * @from: Source address, in user space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from user space to kernel space. Caller must check
- * the specified block with access_ok() before calling this function.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- *
- * If some data could not be copied, this function will pad the copied
- * data to the requested size using zero bytes.
- *
- * An alternate version - __copy_from_user_inatomic() - may be called from
- * atomic context and will fail rather than sleep. In this case the
- * uncopied bytes will *NOT* be padded with zeros. See fs/filemap.h
- * for explanation of why this is needed.
- */
-static __always_inline unsigned long
-__copy_from_user(void *to, const void __user *from, unsigned long n)
-{
- might_fault();
- check_object_size(to, n, false);
if (__builtin_constant_p(n)) {
unsigned long ret;
switch (n) {
case 1:
+ ret = 0;
__uaccess_begin();
- __get_user_size(*(u8 *)to, from, 1, ret, 1);
+ __get_user_asm_nozero(*(u8 *)to, from, ret,
+ "b", "b", "=q", 1);
__uaccess_end();
return ret;
case 2:
+ ret = 0;
__uaccess_begin();
- __get_user_size(*(u16 *)to, from, 2, ret, 2);
+ __get_user_asm_nozero(*(u16 *)to, from, ret,
+ "w", "w", "=r", 2);
__uaccess_end();
return ret;
case 4:
+ ret = 0;
__uaccess_begin();
- __get_user_size(*(u32 *)to, from, 4, ret, 4);
+ __get_user_asm_nozero(*(u32 *)to, from, ret,
+ "l", "k", "=r", 4);
__uaccess_end();
return ret;
}
}
- return __copy_from_user_ll(to, from, n);
-}
-
-static __always_inline unsigned long __copy_from_user_nocache(void *to,
- const void __user *from, unsigned long n)
-{
- might_fault();
- if (__builtin_constant_p(n)) {
- unsigned long ret;
-
- switch (n) {
- case 1:
- __uaccess_begin();
- __get_user_size(*(u8 *)to, from, 1, ret, 1);
- __uaccess_end();
- return ret;
- case 2:
- __uaccess_begin();
- __get_user_size(*(u16 *)to, from, 2, ret, 2);
- __uaccess_end();
- return ret;
- case 4:
- __uaccess_begin();
- __get_user_size(*(u32 *)to, from, 4, ret, 4);
- __uaccess_end();
- return ret;
- }
- }
- return __copy_from_user_ll_nocache(to, from, n);
+ return __copy_user_ll(to, (__force const void *)from, n);
}
static __always_inline unsigned long
diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h
index 673059a..c5504b9 100644
--- a/arch/x86/include/asm/uaccess_64.h
+++ b/arch/x86/include/asm/uaccess_64.h
@@ -5,7 +5,6 @@
* User space memory access functions
*/
#include <linux/compiler.h>
-#include <linux/errno.h>
#include <linux/lockdep.h>
#include <linux/kasan-checks.h>
#include <asm/alternative.h>
@@ -46,58 +45,54 @@
return ret;
}
-__must_check unsigned long
-copy_in_user(void __user *to, const void __user *from, unsigned len);
-
-static __always_inline __must_check
-int __copy_from_user_nocheck(void *dst, const void __user *src, unsigned size)
+static __always_inline __must_check unsigned long
+raw_copy_from_user(void *dst, const void __user *src, unsigned long size)
{
int ret = 0;
- check_object_size(dst, size, false);
if (!__builtin_constant_p(size))
return copy_user_generic(dst, (__force void *)src, size);
switch (size) {
case 1:
__uaccess_begin();
- __get_user_asm(*(u8 *)dst, (u8 __user *)src,
+ __get_user_asm_nozero(*(u8 *)dst, (u8 __user *)src,
ret, "b", "b", "=q", 1);
__uaccess_end();
return ret;
case 2:
__uaccess_begin();
- __get_user_asm(*(u16 *)dst, (u16 __user *)src,
+ __get_user_asm_nozero(*(u16 *)dst, (u16 __user *)src,
ret, "w", "w", "=r", 2);
__uaccess_end();
return ret;
case 4:
__uaccess_begin();
- __get_user_asm(*(u32 *)dst, (u32 __user *)src,
+ __get_user_asm_nozero(*(u32 *)dst, (u32 __user *)src,
ret, "l", "k", "=r", 4);
__uaccess_end();
return ret;
case 8:
__uaccess_begin();
- __get_user_asm(*(u64 *)dst, (u64 __user *)src,
+ __get_user_asm_nozero(*(u64 *)dst, (u64 __user *)src,
ret, "q", "", "=r", 8);
__uaccess_end();
return ret;
case 10:
__uaccess_begin();
- __get_user_asm(*(u64 *)dst, (u64 __user *)src,
+ __get_user_asm_nozero(*(u64 *)dst, (u64 __user *)src,
ret, "q", "", "=r", 10);
if (likely(!ret))
- __get_user_asm(*(u16 *)(8 + (char *)dst),
+ __get_user_asm_nozero(*(u16 *)(8 + (char *)dst),
(u16 __user *)(8 + (char __user *)src),
ret, "w", "w", "=r", 2);
__uaccess_end();
return ret;
case 16:
__uaccess_begin();
- __get_user_asm(*(u64 *)dst, (u64 __user *)src,
+ __get_user_asm_nozero(*(u64 *)dst, (u64 __user *)src,
ret, "q", "", "=r", 16);
if (likely(!ret))
- __get_user_asm(*(u64 *)(8 + (char *)dst),
+ __get_user_asm_nozero(*(u64 *)(8 + (char *)dst),
(u64 __user *)(8 + (char __user *)src),
ret, "q", "", "=r", 8);
__uaccess_end();
@@ -107,20 +102,11 @@
}
}
-static __always_inline __must_check
-int __copy_from_user(void *dst, const void __user *src, unsigned size)
-{
- might_fault();
- kasan_check_write(dst, size);
- return __copy_from_user_nocheck(dst, src, size);
-}
-
-static __always_inline __must_check
-int __copy_to_user_nocheck(void __user *dst, const void *src, unsigned size)
+static __always_inline __must_check unsigned long
+raw_copy_to_user(void __user *dst, const void *src, unsigned long size)
{
int ret = 0;
- check_object_size(src, size, true);
if (!__builtin_constant_p(size))
return copy_user_generic((__force void *)dst, src, size);
switch (size) {
@@ -176,100 +162,16 @@
}
static __always_inline __must_check
-int __copy_to_user(void __user *dst, const void *src, unsigned size)
+unsigned long raw_copy_in_user(void __user *dst, const void __user *src, unsigned long size)
{
- might_fault();
- kasan_check_read(src, size);
- return __copy_to_user_nocheck(dst, src, size);
-}
-
-static __always_inline __must_check
-int __copy_in_user(void __user *dst, const void __user *src, unsigned size)
-{
- int ret = 0;
-
- might_fault();
- if (!__builtin_constant_p(size))
- return copy_user_generic((__force void *)dst,
- (__force void *)src, size);
- switch (size) {
- case 1: {
- u8 tmp;
- __uaccess_begin();
- __get_user_asm(tmp, (u8 __user *)src,
- ret, "b", "b", "=q", 1);
- if (likely(!ret))
- __put_user_asm(tmp, (u8 __user *)dst,
- ret, "b", "b", "iq", 1);
- __uaccess_end();
- return ret;
- }
- case 2: {
- u16 tmp;
- __uaccess_begin();
- __get_user_asm(tmp, (u16 __user *)src,
- ret, "w", "w", "=r", 2);
- if (likely(!ret))
- __put_user_asm(tmp, (u16 __user *)dst,
- ret, "w", "w", "ir", 2);
- __uaccess_end();
- return ret;
- }
-
- case 4: {
- u32 tmp;
- __uaccess_begin();
- __get_user_asm(tmp, (u32 __user *)src,
- ret, "l", "k", "=r", 4);
- if (likely(!ret))
- __put_user_asm(tmp, (u32 __user *)dst,
- ret, "l", "k", "ir", 4);
- __uaccess_end();
- return ret;
- }
- case 8: {
- u64 tmp;
- __uaccess_begin();
- __get_user_asm(tmp, (u64 __user *)src,
- ret, "q", "", "=r", 8);
- if (likely(!ret))
- __put_user_asm(tmp, (u64 __user *)dst,
- ret, "q", "", "er", 8);
- __uaccess_end();
- return ret;
- }
- default:
- return copy_user_generic((__force void *)dst,
- (__force void *)src, size);
- }
-}
-
-static __must_check __always_inline int
-__copy_from_user_inatomic(void *dst, const void __user *src, unsigned size)
-{
- kasan_check_write(dst, size);
- return __copy_from_user_nocheck(dst, src, size);
-}
-
-static __must_check __always_inline int
-__copy_to_user_inatomic(void __user *dst, const void *src, unsigned size)
-{
- kasan_check_read(src, size);
- return __copy_to_user_nocheck(dst, src, size);
+ return copy_user_generic((__force void *)dst,
+ (__force void *)src, size);
}
extern long __copy_user_nocache(void *dst, const void __user *src,
unsigned size, int zerorest);
static inline int
-__copy_from_user_nocache(void *dst, const void __user *src, unsigned size)
-{
- might_fault();
- kasan_check_write(dst, size);
- return __copy_user_nocache(dst, src, size, 1);
-}
-
-static inline int
__copy_from_user_inatomic_nocache(void *dst, const void __user *src,
unsigned size)
{
diff --git a/arch/x86/include/asm/unistd.h b/arch/x86/include/asm/unistd.h
index 32712a9..1ba1536 100644
--- a/arch/x86/include/asm/unistd.h
+++ b/arch/x86/include/asm/unistd.h
@@ -23,7 +23,6 @@
# include <asm/unistd_64.h>
# include <asm/unistd_64_x32.h>
# define __ARCH_WANT_COMPAT_SYS_TIME
-# define __ARCH_WANT_COMPAT_SYS_GETDENTS64
# define __ARCH_WANT_COMPAT_SYS_PREADV64
# define __ARCH_WANT_COMPAT_SYS_PWRITEV64
# define __ARCH_WANT_COMPAT_SYS_PREADV64V2
diff --git a/arch/x86/include/asm/unwind.h b/arch/x86/include/asm/unwind.h
index 6fa75b1..e667649 100644
--- a/arch/x86/include/asm/unwind.h
+++ b/arch/x86/include/asm/unwind.h
@@ -11,9 +11,12 @@
unsigned long stack_mask;
struct task_struct *task;
int graph_idx;
+ bool error;
#ifdef CONFIG_FRAME_POINTER
+ bool got_irq;
unsigned long *bp, *orig_sp;
struct pt_regs *regs;
+ unsigned long ip;
#else
unsigned long *sp;
#endif
@@ -40,6 +43,11 @@
__unwind_start(state, task, regs, first_frame);
}
+static inline bool unwind_error(struct unwind_state *state)
+{
+ return state->error;
+}
+
#ifdef CONFIG_FRAME_POINTER
static inline
diff --git a/arch/x86/include/asm/uv/uv_bau.h b/arch/x86/include/asm/uv/uv_bau.h
index 57ab86d..7cac798 100644
--- a/arch/x86/include/asm/uv/uv_bau.h
+++ b/arch/x86/include/asm/uv/uv_bau.h
@@ -185,6 +185,15 @@
#define MSG_REGULAR 1
#define MSG_RETRY 2
+#define BAU_DESC_QUALIFIER 0x534749
+
+enum uv_bau_version {
+ UV_BAU_V1 = 1,
+ UV_BAU_V2,
+ UV_BAU_V3,
+ UV_BAU_V4,
+};
+
/*
* Distribution: 32 bytes (256 bits) (bytes 0-0x1f of descriptor)
* If the 'multilevel' flag in the header portion of the descriptor
@@ -222,20 +231,32 @@
* the s/w ack bit vector ]
*/
-/*
- * The payload is software-defined for INTD transactions
+/**
+ * struct uv1_2_3_bau_msg_payload - defines payload for INTD transactions
+ * @address: Signifies a page or all TLB's of the cpu
+ * @sending_cpu: CPU from which the message originates
+ * @acknowledge_count: CPUs on the destination Hub that received the interrupt
*/
-struct bau_msg_payload {
- unsigned long address; /* signifies a page or all
- TLB's of the cpu */
- /* 64 bits */
- unsigned short sending_cpu; /* filled in by sender */
- /* 16 bits */
- unsigned short acknowledge_count; /* filled in by destination */
- /* 16 bits */
- unsigned int reserved1:32; /* not usable */
+struct uv1_2_3_bau_msg_payload {
+ u64 address;
+ u16 sending_cpu;
+ u16 acknowledge_count;
};
+/**
+ * struct uv4_bau_msg_payload - defines payload for INTD transactions
+ * @address: Signifies a page or all TLB's of the cpu
+ * @sending_cpu: CPU from which the message originates
+ * @acknowledge_count: CPUs on the destination Hub that received the interrupt
+ * @qualifier: Set by source to verify origin of INTD broadcast
+ */
+struct uv4_bau_msg_payload {
+ u64 address;
+ u16 sending_cpu;
+ u16 acknowledge_count;
+ u32 reserved:8;
+ u32 qualifier:24;
+};
/*
* UV1 Message header: 16 bytes (128 bits) (bytes 0x30-0x3f of descriptor)
@@ -385,17 +406,6 @@
/* bits 127:120 */
};
-/* Abstracted BAU functions */
-struct bau_operations {
- unsigned long (*read_l_sw_ack)(void);
- unsigned long (*read_g_sw_ack)(int pnode);
- unsigned long (*bau_gpa_to_offset)(unsigned long vaddr);
- void (*write_l_sw_ack)(unsigned long mmr);
- void (*write_g_sw_ack)(int pnode, unsigned long mmr);
- void (*write_payload_first)(int pnode, unsigned long mmr);
- void (*write_payload_last)(int pnode, unsigned long mmr);
-};
-
/*
* The activation descriptor:
* The format of the message to send, plus all accompanying control
@@ -411,7 +421,10 @@
struct uv2_3_bau_msg_header uv2_3_hdr;
} header;
- struct bau_msg_payload payload;
+ union bau_payload_header {
+ struct uv1_2_3_bau_msg_payload uv1_2_3;
+ struct uv4_bau_msg_payload uv4;
+ } payload;
};
/* UV1:
* -payload-- ---------header------
@@ -588,8 +601,12 @@
struct socket_desc socket[2];
};
-/*
- * one per-cpu; to locate the software tables
+/**
+ * struct bau_control
+ * @status_mmr: location of status mmr, determined by uvhub_cpu
+ * @status_index: index of ERR|BUSY bits in status mmr, determined by uvhub_cpu
+ *
+ * Per-cpu control struct containing CPU topology information and BAU tuneables.
*/
struct bau_control {
struct bau_desc *descriptor_base;
@@ -607,6 +624,8 @@
int timeout_tries;
int ipi_attempts;
int conseccompletes;
+ u64 status_mmr;
+ int status_index;
bool nobau;
short baudisabled;
short cpu;
@@ -644,6 +663,19 @@
struct hub_and_pnode *thp;
};
+/* Abstracted BAU functions */
+struct bau_operations {
+ unsigned long (*read_l_sw_ack)(void);
+ unsigned long (*read_g_sw_ack)(int pnode);
+ unsigned long (*bau_gpa_to_offset)(unsigned long vaddr);
+ void (*write_l_sw_ack)(unsigned long mmr);
+ void (*write_g_sw_ack)(int pnode, unsigned long mmr);
+ void (*write_payload_first)(int pnode, unsigned long mmr);
+ void (*write_payload_last)(int pnode, unsigned long mmr);
+ int (*wait_completion)(struct bau_desc*,
+ struct bau_control*, long try);
+};
+
static inline void write_mmr_data_broadcast(int pnode, unsigned long mmr_image)
{
write_gmmr(pnode, UVH_BAU_DATA_BROADCAST, mmr_image);
diff --git a/arch/x86/include/asm/vdso.h b/arch/x86/include/asm/vdso.h
index 2444189..bccdf49 100644
--- a/arch/x86/include/asm/vdso.h
+++ b/arch/x86/include/asm/vdso.h
@@ -20,6 +20,7 @@
long sym_vvar_page;
long sym_hpet_page;
long sym_pvclock_page;
+ long sym_hvclock_page;
long sym_VDSO32_NOTE_MASK;
long sym___kernel_sigreturn;
long sym___kernel_rt_sigreturn;
diff --git a/arch/x86/include/asm/xen/page.h b/arch/x86/include/asm/xen/page.h
index 33cbd3d..8a5a02b 100644
--- a/arch/x86/include/asm/xen/page.h
+++ b/arch/x86/include/asm/xen/page.h
@@ -6,6 +6,7 @@
#include <linux/spinlock.h>
#include <linux/pfn.h>
#include <linux/mm.h>
+#include <linux/device.h>
#include <linux/uaccess.h>
#include <asm/page.h>
@@ -279,13 +280,17 @@
#define pmd_val_ma(v) ((v).pmd)
#ifdef __PAGETABLE_PUD_FOLDED
-#define pud_val_ma(v) ((v).pgd.pgd)
+#define pud_val_ma(v) ((v).p4d.pgd.pgd)
#else
#define pud_val_ma(v) ((v).pud)
#endif
#define __pmd_ma(x) ((pmd_t) { (x) } )
-#define pgd_val_ma(x) ((x).pgd)
+#ifdef __PAGETABLE_P4D_FOLDED
+#define p4d_val_ma(x) ((x).pgd.pgd)
+#else
+#define p4d_val_ma(x) ((x).p4d)
+#endif
void xen_set_domain_pte(pte_t *ptep, pte_t pteval, unsigned domid);
diff --git a/arch/x86/include/uapi/asm/bootparam.h b/arch/x86/include/uapi/asm/bootparam.h
index 07244ea..ddef37b 100644
--- a/arch/x86/include/uapi/asm/bootparam.h
+++ b/arch/x86/include/uapi/asm/bootparam.h
@@ -34,7 +34,6 @@
#include <linux/screen_info.h>
#include <linux/apm_bios.h>
#include <linux/edd.h>
-#include <asm/e820.h>
#include <asm/ist.h>
#include <video/edid.h>
@@ -111,6 +110,21 @@
__u32 efi_memmap_hi;
};
+/*
+ * This is the maximum number of entries in struct boot_params::e820_table
+ * (the zeropage), which is part of the x86 boot protocol ABI:
+ */
+#define E820_MAX_ENTRIES_ZEROPAGE 128
+
+/*
+ * The E820 memory region entry of the boot protocol ABI:
+ */
+struct boot_e820_entry {
+ __u64 addr;
+ __u64 size;
+ __u32 type;
+} __attribute__((packed));
+
/* The so-called "zeropage" */
struct boot_params {
struct screen_info screen_info; /* 0x000 */
@@ -153,7 +167,7 @@
struct setup_header hdr; /* setup header */ /* 0x1f1 */
__u8 _pad7[0x290-0x1f1-sizeof(struct setup_header)];
__u32 edd_mbr_sig_buffer[EDD_MBR_SIG_MAX]; /* 0x290 */
- struct e820entry e820_map[E820MAX]; /* 0x2d0 */
+ struct boot_e820_entry e820_table[E820_MAX_ENTRIES_ZEROPAGE]; /* 0x2d0 */
__u8 _pad8[48]; /* 0xcd0 */
struct edd_info eddbuf[EDDMAXNR]; /* 0xd00 */
__u8 _pad9[276]; /* 0xeec */
diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h
index 835aa51..c457655 100644
--- a/arch/x86/include/uapi/asm/prctl.h
+++ b/arch/x86/include/uapi/asm/prctl.h
@@ -1,10 +1,13 @@
#ifndef _ASM_X86_PRCTL_H
#define _ASM_X86_PRCTL_H
-#define ARCH_SET_GS 0x1001
-#define ARCH_SET_FS 0x1002
-#define ARCH_GET_FS 0x1003
-#define ARCH_GET_GS 0x1004
+#define ARCH_SET_GS 0x1001
+#define ARCH_SET_FS 0x1002
+#define ARCH_GET_FS 0x1003
+#define ARCH_GET_GS 0x1004
+
+#define ARCH_GET_CPUID 0x1011
+#define ARCH_SET_CPUID 0x1012
#define ARCH_MAP_VDSO_X32 0x2001
#define ARCH_MAP_VDSO_32 0x2002
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 84c0059..4b99423 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -27,7 +27,7 @@
OBJECT_FILES_NON_STANDARD_head_$(BITS).o := y
OBJECT_FILES_NON_STANDARD_relocate_kernel_$(BITS).o := y
-OBJECT_FILES_NON_STANDARD_mcount_$(BITS).o := y
+OBJECT_FILES_NON_STANDARD_ftrace_$(BITS).o := y
OBJECT_FILES_NON_STANDARD_test_nx.o := y
# If instrumentation of this dir is enabled, boot hangs during first second.
@@ -46,7 +46,7 @@
obj-y += setup.o x86_init.o i8259.o irqinit.o jump_label.o
obj-$(CONFIG_IRQ_WORK) += irq_work.o
obj-y += probe_roms.o
-obj-$(CONFIG_X86_64) += sys_x86_64.o mcount_64.o
+obj-$(CONFIG_X86_64) += sys_x86_64.o
obj-$(CONFIG_X86_ESPFIX64) += espfix_64.o
obj-$(CONFIG_SYSFS) += ksysfs.o
obj-y += bootflag.o e820.o
@@ -82,6 +82,7 @@
obj-$(CONFIG_X86_REBOOTFIXUPS) += reboot_fixups_32.o
obj-$(CONFIG_DYNAMIC_FTRACE) += ftrace.o
obj-$(CONFIG_LIVEPATCH) += livepatch.o
+obj-$(CONFIG_FUNCTION_TRACER) += ftrace_$(BITS).o
obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += ftrace.o
obj-$(CONFIG_FTRACE_SYSCALLS) += ftrace.o
obj-$(CONFIG_X86_TSC) += trace_clock.o
diff --git a/arch/x86/kernel/acpi/boot.c b/arch/x86/kernel/acpi/boot.c
index b2879cc23..6bb6806 100644
--- a/arch/x86/kernel/acpi/boot.c
+++ b/arch/x86/kernel/acpi/boot.c
@@ -37,6 +37,7 @@
#include <linux/pci.h>
#include <linux/efi-bgrt.h>
+#include <asm/e820/api.h>
#include <asm/irqdomain.h>
#include <asm/pci_x86.h>
#include <asm/pgtable.h>
@@ -1564,12 +1565,6 @@
return 0;
}
-static int __init acpi_parse_bgrt(struct acpi_table_header *table)
-{
- efi_bgrt_init(table);
- return 0;
-}
-
int __init acpi_boot_init(void)
{
/* those are executed after early-quirks are executed */
@@ -1729,6 +1724,6 @@
void __init arch_reserve_mem_area(acpi_physical_address addr, size_t size)
{
- e820_add_region(addr, size, E820_ACPI);
- update_e820();
+ e820__range_add(addr, size, E820_TYPE_ACPI);
+ e820__update_table_print();
}
diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c
index 4858733..ed01481 100644
--- a/arch/x86/kernel/acpi/sleep.c
+++ b/arch/x86/kernel/acpi/sleep.c
@@ -101,7 +101,7 @@
#ifdef CONFIG_SMP
initial_stack = (unsigned long)temp_stack + sizeof(temp_stack);
early_gdt_descr.address =
- (unsigned long)get_cpu_gdt_table(smp_processor_id());
+ (unsigned long)get_cpu_gdt_rw(smp_processor_id());
initial_gs = per_cpu_offset(smp_processor_id());
#endif
initial_code = (unsigned long)wakeup_long64;
diff --git a/arch/x86/kernel/aperture_64.c b/arch/x86/kernel/aperture_64.c
index 0a2bb1f..ef2859f 100644
--- a/arch/x86/kernel/aperture_64.c
+++ b/arch/x86/kernel/aperture_64.c
@@ -21,7 +21,7 @@
#include <linux/pci.h>
#include <linux/bitops.h>
#include <linux/suspend.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/io.h>
#include <asm/iommu.h>
#include <asm/gart.h>
@@ -306,13 +306,13 @@
fix = 1;
if (gart_fix_e820 && !fix && aper_enabled) {
- if (e820_any_mapped(aper_base, aper_base + aper_size,
- E820_RAM)) {
+ if (e820__mapped_any(aper_base, aper_base + aper_size,
+ E820_TYPE_RAM)) {
/* reserve it, so we can reuse it in second kernel */
pr_info("e820: reserve [mem %#010Lx-%#010Lx] for GART\n",
aper_base, aper_base + aper_size - 1);
- e820_add_region(aper_base, aper_size, E820_RESERVED);
- update_e820();
+ e820__range_add(aper_base, aper_size, E820_TYPE_RESERVED);
+ e820__update_table_print();
}
}
diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 8ccb7ef..2d75faf 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -731,8 +731,10 @@
TICK_NSEC, lapic_clockevent.shift);
lapic_clockevent.max_delta_ns =
clockevent_delta2ns(0x7FFFFF, &lapic_clockevent);
+ lapic_clockevent.max_delta_ticks = 0x7FFFFF;
lapic_clockevent.min_delta_ns =
clockevent_delta2ns(0xF, &lapic_clockevent);
+ lapic_clockevent.min_delta_ticks = 0xF;
lapic_clockevent.features &= ~CLOCK_EVT_FEAT_DUMMY;
return 0;
}
@@ -778,8 +780,10 @@
lapic_clockevent.shift);
lapic_clockevent.max_delta_ns =
clockevent_delta2ns(0x7FFFFFFF, &lapic_clockevent);
+ lapic_clockevent.max_delta_ticks = 0x7FFFFFFF;
lapic_clockevent.min_delta_ns =
clockevent_delta2ns(0xF, &lapic_clockevent);
+ lapic_clockevent.min_delta_ticks = 0xF;
lapic_timer_frequency = (delta * APIC_DIVISOR) / LAPIC_CAL_LOOPS;
@@ -1789,8 +1793,8 @@
apic_phys = mp_lapic_addr;
/*
- * acpi lapic path already maps that address in
- * acpi_register_lapic_address()
+ * If the system has ACPI MADT tables or MP info, the LAPIC
+ * address is already registered.
*/
if (!acpi_lapic && !smp_found_config)
register_lapic_address(apic_phys);
@@ -2237,7 +2241,7 @@
static void __init apic_bsp_up_setup(void)
{
#ifdef CONFIG_X86_64
- apic_write(APIC_ID, SET_APIC_ID(boot_cpu_physical_apicid));
+ apic_write(APIC_ID, apic->set_apic_id(boot_cpu_physical_apicid));
#else
/*
* Hack: In case of kdump, after a crash, kernel might be booting
@@ -2627,7 +2631,7 @@
}
/*
- * need call insert after e820_reserve_resources()
+ * need call insert after e820__reserve_resources()
* that is using request_resource
*/
late_initcall(lapic_insert_resource);
diff --git a/arch/x86/kernel/apic/apic_noop.c b/arch/x86/kernel/apic/apic_noop.c
index b109e43..2262eb6 100644
--- a/arch/x86/kernel/apic/apic_noop.c
+++ b/arch/x86/kernel/apic/apic_noop.c
@@ -26,7 +26,7 @@
#include <linux/interrupt.h>
#include <asm/acpi.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
static void noop_init_apic_ldr(void) { }
static void noop_send_IPI(int cpu, int vector) { }
diff --git a/arch/x86/kernel/apic/probe_32.c b/arch/x86/kernel/apic/probe_32.c
index c48264e..2e8f7f0 100644
--- a/arch/x86/kernel/apic/probe_32.c
+++ b/arch/x86/kernel/apic/probe_32.c
@@ -25,7 +25,7 @@
#include <linux/interrupt.h>
#include <asm/acpi.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#ifdef CONFIG_HOTPLUG_CPU
#define DEFAULT_SEND_IPI (1)
diff --git a/arch/x86/kernel/apic/x2apic_uv_x.c b/arch/x86/kernel/apic/x2apic_uv_x.c
index 86f20cc..b487b3a 100644
--- a/arch/x86/kernel/apic/x2apic_uv_x.c
+++ b/arch/x86/kernel/apic/x2apic_uv_x.c
@@ -34,6 +34,7 @@
#include <asm/uv/bios.h>
#include <asm/uv/uv.h>
#include <asm/apic.h>
+#include <asm/e820/api.h>
#include <asm/ipi.h>
#include <asm/smp.h>
#include <asm/x86_init.h>
diff --git a/arch/x86/kernel/apm_32.c b/arch/x86/kernel/apm_32.c
index 5a41454..446b0d3 100644
--- a/arch/x86/kernel/apm_32.c
+++ b/arch/x86/kernel/apm_32.c
@@ -609,7 +609,7 @@
cpu = get_cpu();
BUG_ON(cpu != 0);
- gdt = get_cpu_gdt_table(cpu);
+ gdt = get_cpu_gdt_rw(cpu);
save_desc_40 = gdt[0x40 / 8];
gdt[0x40 / 8] = bad_bios_desc;
@@ -685,7 +685,7 @@
cpu = get_cpu();
BUG_ON(cpu != 0);
- gdt = get_cpu_gdt_table(cpu);
+ gdt = get_cpu_gdt_rw(cpu);
save_desc_40 = gdt[0x40 / 8];
gdt[0x40 / 8] = bad_bios_desc;
@@ -2352,7 +2352,7 @@
* Note we only set APM segments on CPU zero, since we pin the APM
* code to that CPU.
*/
- gdt = get_cpu_gdt_table(0);
+ gdt = get_cpu_gdt_rw(0);
set_desc_base(&gdt[APM_CS >> 3],
(unsigned long)__va((unsigned long)apm_info.bios.cseg << 4));
set_desc_base(&gdt[APM_CS_16 >> 3],
diff --git a/arch/x86/kernel/cpu/centaur.c b/arch/x86/kernel/cpu/centaur.c
index 43955ee..44207b7 100644
--- a/arch/x86/kernel/cpu/centaur.c
+++ b/arch/x86/kernel/cpu/centaur.c
@@ -3,7 +3,7 @@
#include <linux/sched/clock.h>
#include <asm/cpufeature.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/mtrr.h>
#include <asm/msr.h>
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 58094a1..8ee3211 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -448,19 +448,60 @@
load_stack_canary_segment();
}
+/* Setup the fixmap mapping only once per-processor */
+static inline void setup_fixmap_gdt(int cpu)
+{
+#ifdef CONFIG_X86_64
+ /* On 64-bit systems, we use a read-only fixmap GDT. */
+ pgprot_t prot = PAGE_KERNEL_RO;
+#else
+ /*
+ * On native 32-bit systems, the GDT cannot be read-only because
+ * our double fault handler uses a task gate, and entering through
+ * a task gate needs to change an available TSS to busy. If the GDT
+ * is read-only, that will triple fault.
+ *
+ * On Xen PV, the GDT must be read-only because the hypervisor requires
+ * it.
+ */
+ pgprot_t prot = boot_cpu_has(X86_FEATURE_XENPV) ?
+ PAGE_KERNEL_RO : PAGE_KERNEL;
+#endif
+
+ __set_fixmap(get_cpu_gdt_ro_index(cpu), get_cpu_gdt_paddr(cpu), prot);
+}
+
+/* Load the original GDT from the per-cpu structure */
+void load_direct_gdt(int cpu)
+{
+ struct desc_ptr gdt_descr;
+
+ gdt_descr.address = (long)get_cpu_gdt_rw(cpu);
+ gdt_descr.size = GDT_SIZE - 1;
+ load_gdt(&gdt_descr);
+}
+EXPORT_SYMBOL_GPL(load_direct_gdt);
+
+/* Load a fixmap remapping of the per-cpu GDT */
+void load_fixmap_gdt(int cpu)
+{
+ struct desc_ptr gdt_descr;
+
+ gdt_descr.address = (long)get_cpu_gdt_ro(cpu);
+ gdt_descr.size = GDT_SIZE - 1;
+ load_gdt(&gdt_descr);
+}
+EXPORT_SYMBOL_GPL(load_fixmap_gdt);
+
/*
* Current gdt points %fs at the "master" per-cpu area: after this,
* it's on the real one.
*/
void switch_to_new_gdt(int cpu)
{
- struct desc_ptr gdt_descr;
-
- gdt_descr.address = (long)get_cpu_gdt_table(cpu);
- gdt_descr.size = GDT_SIZE - 1;
- load_gdt(&gdt_descr);
+ /* Load the original GDT */
+ load_direct_gdt(cpu);
/* Reload the per-cpu base */
-
load_percpu_segment(cpu);
}
@@ -1526,6 +1567,9 @@
if (is_uv_system())
uv_cpu_init();
+
+ setup_fixmap_gdt(cpu);
+ load_fixmap_gdt(cpu);
}
#else
@@ -1581,6 +1625,9 @@
dbg_restore_debug_regs();
fpu__init_cpu();
+
+ setup_fixmap_gdt(cpu);
+ load_fixmap_gdt(cpu);
}
#endif
diff --git a/arch/x86/kernel/cpu/intel.c b/arch/x86/kernel/cpu/intel.c
index 0631977..dfa90a3 100644
--- a/arch/x86/kernel/cpu/intel.c
+++ b/arch/x86/kernel/cpu/intel.c
@@ -90,16 +90,12 @@
return;
}
- if (ring3mwait_disabled) {
- msr_clear_bit(MSR_MISC_FEATURE_ENABLES,
- MSR_MISC_FEATURE_ENABLES_RING3MWAIT_BIT);
+ if (ring3mwait_disabled)
return;
- }
-
- msr_set_bit(MSR_MISC_FEATURE_ENABLES,
- MSR_MISC_FEATURE_ENABLES_RING3MWAIT_BIT);
set_cpu_cap(c, X86_FEATURE_RING3MWAIT);
+ this_cpu_or(msr_misc_features_shadow,
+ 1UL << MSR_MISC_FEATURES_ENABLES_RING3MWAIT_BIT);
if (c == &boot_cpu_data)
ELF_HWCAP2 |= HWCAP2_RING3MWAIT;
@@ -488,6 +484,34 @@
init_intel_energy_perf(c);
}
+static void init_cpuid_fault(struct cpuinfo_x86 *c)
+{
+ u64 msr;
+
+ if (!rdmsrl_safe(MSR_PLATFORM_INFO, &msr)) {
+ if (msr & MSR_PLATFORM_INFO_CPUID_FAULT)
+ set_cpu_cap(c, X86_FEATURE_CPUID_FAULT);
+ }
+}
+
+static void init_intel_misc_features(struct cpuinfo_x86 *c)
+{
+ u64 msr;
+
+ if (rdmsrl_safe(MSR_MISC_FEATURES_ENABLES, &msr))
+ return;
+
+ /* Clear all MISC features */
+ this_cpu_write(msr_misc_features_shadow, 0);
+
+ /* Check features and update capabilities and shadow control bits */
+ init_cpuid_fault(c);
+ probe_xeon_phi_r3mwait(c);
+
+ msr = this_cpu_read(msr_misc_features_shadow);
+ wrmsrl(MSR_MISC_FEATURES_ENABLES, msr);
+}
+
static void init_intel(struct cpuinfo_x86 *c)
{
unsigned int l2 = 0;
@@ -602,7 +626,7 @@
init_intel_energy_perf(c);
- probe_xeon_phi_r3mwait(c);
+ init_intel_misc_features(c);
}
#ifdef CONFIG_X86_32
diff --git a/arch/x86/kernel/cpu/intel_rdt.c b/arch/x86/kernel/cpu/intel_rdt.c
index 5a533fe..5b36646 100644
--- a/arch/x86/kernel/cpu/intel_rdt.c
+++ b/arch/x86/kernel/cpu/intel_rdt.c
@@ -32,55 +32,98 @@
#include <asm/intel-family.h>
#include <asm/intel_rdt.h>
+#define MAX_MBA_BW 100u
+#define MBA_IS_LINEAR 0x4
+
/* Mutex to protect rdtgroup access. */
DEFINE_MUTEX(rdtgroup_mutex);
DEFINE_PER_CPU_READ_MOSTLY(int, cpu_closid);
+/*
+ * Used to store the max resource name width and max resource data width
+ * to display the schemata in a tabular format
+ */
+int max_name_width, max_data_width;
+
+static void
+mba_wrmsr(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r);
+static void
+cat_wrmsr(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r);
+
#define domain_init(id) LIST_HEAD_INIT(rdt_resources_all[id].domains)
struct rdt_resource rdt_resources_all[] = {
{
- .name = "L3",
- .domains = domain_init(RDT_RESOURCE_L3),
- .msr_base = IA32_L3_CBM_BASE,
- .min_cbm_bits = 1,
- .cache_level = 3,
- .cbm_idx_multi = 1,
- .cbm_idx_offset = 0
+ .name = "L3",
+ .domains = domain_init(RDT_RESOURCE_L3),
+ .msr_base = IA32_L3_CBM_BASE,
+ .msr_update = cat_wrmsr,
+ .cache_level = 3,
+ .cache = {
+ .min_cbm_bits = 1,
+ .cbm_idx_mult = 1,
+ .cbm_idx_offset = 0,
+ },
+ .parse_ctrlval = parse_cbm,
+ .format_str = "%d=%0*x",
},
{
- .name = "L3DATA",
- .domains = domain_init(RDT_RESOURCE_L3DATA),
- .msr_base = IA32_L3_CBM_BASE,
- .min_cbm_bits = 1,
- .cache_level = 3,
- .cbm_idx_multi = 2,
- .cbm_idx_offset = 0
+ .name = "L3DATA",
+ .domains = domain_init(RDT_RESOURCE_L3DATA),
+ .msr_base = IA32_L3_CBM_BASE,
+ .msr_update = cat_wrmsr,
+ .cache_level = 3,
+ .cache = {
+ .min_cbm_bits = 1,
+ .cbm_idx_mult = 2,
+ .cbm_idx_offset = 0,
+ },
+ .parse_ctrlval = parse_cbm,
+ .format_str = "%d=%0*x",
},
{
- .name = "L3CODE",
- .domains = domain_init(RDT_RESOURCE_L3CODE),
- .msr_base = IA32_L3_CBM_BASE,
- .min_cbm_bits = 1,
- .cache_level = 3,
- .cbm_idx_multi = 2,
- .cbm_idx_offset = 1
+ .name = "L3CODE",
+ .domains = domain_init(RDT_RESOURCE_L3CODE),
+ .msr_base = IA32_L3_CBM_BASE,
+ .msr_update = cat_wrmsr,
+ .cache_level = 3,
+ .cache = {
+ .min_cbm_bits = 1,
+ .cbm_idx_mult = 2,
+ .cbm_idx_offset = 1,
+ },
+ .parse_ctrlval = parse_cbm,
+ .format_str = "%d=%0*x",
},
{
- .name = "L2",
- .domains = domain_init(RDT_RESOURCE_L2),
- .msr_base = IA32_L2_CBM_BASE,
- .min_cbm_bits = 1,
- .cache_level = 2,
- .cbm_idx_multi = 1,
- .cbm_idx_offset = 0
+ .name = "L2",
+ .domains = domain_init(RDT_RESOURCE_L2),
+ .msr_base = IA32_L2_CBM_BASE,
+ .msr_update = cat_wrmsr,
+ .cache_level = 2,
+ .cache = {
+ .min_cbm_bits = 1,
+ .cbm_idx_mult = 1,
+ .cbm_idx_offset = 0,
+ },
+ .parse_ctrlval = parse_cbm,
+ .format_str = "%d=%0*x",
+ },
+ {
+ .name = "MB",
+ .domains = domain_init(RDT_RESOURCE_MBA),
+ .msr_base = IA32_MBA_THRTL_BASE,
+ .msr_update = mba_wrmsr,
+ .cache_level = 3,
+ .parse_ctrlval = parse_bw,
+ .format_str = "%d=%*d",
},
};
-static int cbm_idx(struct rdt_resource *r, int closid)
+static unsigned int cbm_idx(struct rdt_resource *r, unsigned int closid)
{
- return closid * r->cbm_idx_multi + r->cbm_idx_offset;
+ return closid * r->cache.cbm_idx_mult + r->cache.cbm_idx_offset;
}
/*
@@ -118,9 +161,9 @@
return false;
r->num_closid = 4;
- r->cbm_len = 20;
- r->max_cbm = max_cbm;
- r->min_cbm_bits = 2;
+ r->default_ctrl = max_cbm;
+ r->cache.cbm_len = 20;
+ r->cache.min_cbm_bits = 2;
r->capable = true;
r->enabled = true;
@@ -130,16 +173,66 @@
return false;
}
-static void rdt_get_config(int idx, struct rdt_resource *r)
+/*
+ * rdt_get_mb_table() - get a mapping of bandwidth(b/w) percentage values
+ * exposed to user interface and the h/w understandable delay values.
+ *
+ * The non-linear delay values have the granularity of power of two
+ * and also the h/w does not guarantee a curve for configured delay
+ * values vs. actual b/w enforced.
+ * Hence we need a mapping that is pre calibrated so the user can
+ * express the memory b/w as a percentage value.
+ */
+static inline bool rdt_get_mb_table(struct rdt_resource *r)
+{
+ /*
+ * There are no Intel SKUs as of now to support non-linear delay.
+ */
+ pr_info("MBA b/w map not implemented for cpu:%d, model:%d",
+ boot_cpu_data.x86, boot_cpu_data.x86_model);
+
+ return false;
+}
+
+static bool rdt_get_mem_config(struct rdt_resource *r)
+{
+ union cpuid_0x10_3_eax eax;
+ union cpuid_0x10_x_edx edx;
+ u32 ebx, ecx;
+
+ cpuid_count(0x00000010, 3, &eax.full, &ebx, &ecx, &edx.full);
+ r->num_closid = edx.split.cos_max + 1;
+ r->membw.max_delay = eax.split.max_delay + 1;
+ r->default_ctrl = MAX_MBA_BW;
+ if (ecx & MBA_IS_LINEAR) {
+ r->membw.delay_linear = true;
+ r->membw.min_bw = MAX_MBA_BW - r->membw.max_delay;
+ r->membw.bw_gran = MAX_MBA_BW - r->membw.max_delay;
+ } else {
+ if (!rdt_get_mb_table(r))
+ return false;
+ }
+ r->data_width = 3;
+ rdt_get_mba_infofile(r);
+
+ r->capable = true;
+ r->enabled = true;
+
+ return true;
+}
+
+static void rdt_get_cache_config(int idx, struct rdt_resource *r)
{
union cpuid_0x10_1_eax eax;
- union cpuid_0x10_1_edx edx;
+ union cpuid_0x10_x_edx edx;
u32 ebx, ecx;
cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx, &edx.full);
r->num_closid = edx.split.cos_max + 1;
- r->cbm_len = eax.split.cbm_len + 1;
- r->max_cbm = BIT_MASK(eax.split.cbm_len + 1) - 1;
+ r->cache.cbm_len = eax.split.cbm_len + 1;
+ r->default_ctrl = BIT_MASK(eax.split.cbm_len + 1) - 1;
+ r->data_width = (r->cache.cbm_len + 3) / 4;
+ rdt_get_cache_infofile(r);
r->capable = true;
r->enabled = true;
}
@@ -150,8 +243,9 @@
struct rdt_resource *r = &rdt_resources_all[type];
r->num_closid = r_l3->num_closid / 2;
- r->cbm_len = r_l3->cbm_len;
- r->max_cbm = r_l3->max_cbm;
+ r->cache.cbm_len = r_l3->cache.cbm_len;
+ r->default_ctrl = r_l3->default_ctrl;
+ r->data_width = (r->cache.cbm_len + 3) / 4;
r->capable = true;
/*
* By default, CDP is disabled. CDP can be enabled by mount parameter
@@ -160,33 +254,6 @@
r->enabled = false;
}
-static inline bool get_rdt_resources(void)
-{
- bool ret = false;
-
- if (cache_alloc_hsw_probe())
- return true;
-
- if (!boot_cpu_has(X86_FEATURE_RDT_A))
- return false;
-
- if (boot_cpu_has(X86_FEATURE_CAT_L3)) {
- rdt_get_config(1, &rdt_resources_all[RDT_RESOURCE_L3]);
- if (boot_cpu_has(X86_FEATURE_CDP_L3)) {
- rdt_get_cdp_l3_config(RDT_RESOURCE_L3DATA);
- rdt_get_cdp_l3_config(RDT_RESOURCE_L3CODE);
- }
- ret = true;
- }
- if (boot_cpu_has(X86_FEATURE_CAT_L2)) {
- /* CPUID 0x10.2 fields are same format at 0x10.1 */
- rdt_get_config(2, &rdt_resources_all[RDT_RESOURCE_L2]);
- ret = true;
- }
-
- return ret;
-}
-
static int get_cache_id(int cpu, int level)
{
struct cpu_cacheinfo *ci = get_cpu_cacheinfo(cpu);
@@ -200,29 +267,55 @@
return -1;
}
-void rdt_cbm_update(void *arg)
+/*
+ * Map the memory b/w percentage value to delay values
+ * that can be written to QOS_MSRs.
+ * There are currently no SKUs which support non linear delay values.
+ */
+static u32 delay_bw_map(unsigned long bw, struct rdt_resource *r)
{
- struct msr_param *m = (struct msr_param *)arg;
+ if (r->membw.delay_linear)
+ return MAX_MBA_BW - bw;
+
+ pr_warn_once("Non Linear delay-bw map not supported but queried\n");
+ return r->default_ctrl;
+}
+
+static void
+mba_wrmsr(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r)
+{
+ unsigned int i;
+
+ /* Write the delay values for mba. */
+ for (i = m->low; i < m->high; i++)
+ wrmsrl(r->msr_base + i, delay_bw_map(d->ctrl_val[i], r));
+}
+
+static void
+cat_wrmsr(struct rdt_domain *d, struct msr_param *m, struct rdt_resource *r)
+{
+ unsigned int i;
+
+ for (i = m->low; i < m->high; i++)
+ wrmsrl(r->msr_base + cbm_idx(r, i), d->ctrl_val[i]);
+}
+
+void rdt_ctrl_update(void *arg)
+{
+ struct msr_param *m = arg;
struct rdt_resource *r = m->res;
- int i, cpu = smp_processor_id();
+ int cpu = smp_processor_id();
struct rdt_domain *d;
list_for_each_entry(d, &r->domains, list) {
/* Find the domain that contains this CPU */
- if (cpumask_test_cpu(cpu, &d->cpu_mask))
- goto found;
+ if (cpumask_test_cpu(cpu, &d->cpu_mask)) {
+ r->msr_update(d, m, r);
+ return;
+ }
}
- pr_info_once("cpu %d not found in any domain for resource %s\n",
+ pr_warn_once("cpu %d not found in any domain for resource %s\n",
cpu, r->name);
-
- return;
-
-found:
- for (i = m->low; i < m->high; i++) {
- int idx = cbm_idx(r, i);
-
- wrmsrl(r->msr_base + idx, d->cbm[i]);
- }
}
/*
@@ -258,6 +351,32 @@
return NULL;
}
+static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_domain *d)
+{
+ struct msr_param m;
+ u32 *dc;
+ int i;
+
+ dc = kmalloc_array(r->num_closid, sizeof(*d->ctrl_val), GFP_KERNEL);
+ if (!dc)
+ return -ENOMEM;
+
+ d->ctrl_val = dc;
+
+ /*
+ * Initialize the Control MSRs to having no control.
+ * For Cache Allocation: Set all bits in cbm
+ * For Memory Allocation: Set b/w requested to 100
+ */
+ for (i = 0; i < r->num_closid; i++, dc++)
+ *dc = r->default_ctrl;
+
+ m.low = 0;
+ m.high = r->num_closid;
+ r->msr_update(d, &m, r);
+ return 0;
+}
+
/*
* domain_add_cpu - Add a cpu to a resource's domain list.
*
@@ -273,7 +392,7 @@
*/
static void domain_add_cpu(int cpu, struct rdt_resource *r)
{
- int i, id = get_cache_id(cpu, r->cache_level);
+ int id = get_cache_id(cpu, r->cache_level);
struct list_head *add_pos = NULL;
struct rdt_domain *d;
@@ -294,22 +413,13 @@
d->id = id;
- d->cbm = kmalloc_array(r->num_closid, sizeof(*d->cbm), GFP_KERNEL);
- if (!d->cbm) {
+ if (domain_setup_ctrlval(r, d)) {
kfree(d);
return;
}
- for (i = 0; i < r->num_closid; i++) {
- int idx = cbm_idx(r, i);
-
- d->cbm[i] = r->max_cbm;
- wrmsrl(r->msr_base + idx, d->cbm[i]);
- }
-
cpumask_set_cpu(cpu, &d->cpu_mask);
list_add_tail(&d->list, add_pos);
- r->num_domains++;
}
static void domain_remove_cpu(int cpu, struct rdt_resource *r)
@@ -325,8 +435,7 @@
cpumask_clear_cpu(cpu, &d->cpu_mask);
if (cpumask_empty(&d->cpu_mask)) {
- r->num_domains--;
- kfree(d->cbm);
+ kfree(d->ctrl_val);
list_del(&d->list);
kfree(d);
}
@@ -374,6 +483,57 @@
return 0;
}
+/*
+ * Choose a width for the resource name and resource data based on the
+ * resource that has widest name and cbm.
+ */
+static __init void rdt_init_padding(void)
+{
+ struct rdt_resource *r;
+ int cl;
+
+ for_each_capable_rdt_resource(r) {
+ cl = strlen(r->name);
+ if (cl > max_name_width)
+ max_name_width = cl;
+
+ if (r->data_width > max_data_width)
+ max_data_width = r->data_width;
+ }
+}
+
+static __init bool get_rdt_resources(void)
+{
+ bool ret = false;
+
+ if (cache_alloc_hsw_probe())
+ return true;
+
+ if (!boot_cpu_has(X86_FEATURE_RDT_A))
+ return false;
+
+ if (boot_cpu_has(X86_FEATURE_CAT_L3)) {
+ rdt_get_cache_config(1, &rdt_resources_all[RDT_RESOURCE_L3]);
+ if (boot_cpu_has(X86_FEATURE_CDP_L3)) {
+ rdt_get_cdp_l3_config(RDT_RESOURCE_L3DATA);
+ rdt_get_cdp_l3_config(RDT_RESOURCE_L3CODE);
+ }
+ ret = true;
+ }
+ if (boot_cpu_has(X86_FEATURE_CAT_L2)) {
+ /* CPUID 0x10.2 fields are same format at 0x10.1 */
+ rdt_get_cache_config(2, &rdt_resources_all[RDT_RESOURCE_L2]);
+ ret = true;
+ }
+
+ if (boot_cpu_has(X86_FEATURE_MBA)) {
+ if (rdt_get_mem_config(&rdt_resources_all[RDT_RESOURCE_MBA]))
+ ret = true;
+ }
+
+ return ret;
+}
+
static int __init intel_rdt_late_init(void)
{
struct rdt_resource *r;
@@ -382,6 +542,8 @@
if (!get_rdt_resources())
return -ENODEV;
+ rdt_init_padding();
+
state = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN,
"x86/rdt/cat:online:",
intel_rdt_online_cpu, intel_rdt_offline_cpu);
diff --git a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
index 9ac2a5c..f5af0cc 100644
--- a/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
+++ b/arch/x86/kernel/cpu/intel_rdt_rdtgroup.c
@@ -174,6 +174,13 @@
.seq_show = rdtgroup_seqfile_show,
};
+static bool is_cpu_list(struct kernfs_open_file *of)
+{
+ struct rftype *rft = of->kn->priv;
+
+ return rft->flags & RFTYPE_FLAGS_CPUS_LIST;
+}
+
static int rdtgroup_cpus_show(struct kernfs_open_file *of,
struct seq_file *s, void *v)
{
@@ -182,10 +189,12 @@
rdtgrp = rdtgroup_kn_lock_live(of->kn);
- if (rdtgrp)
- seq_printf(s, "%*pb\n", cpumask_pr_args(&rdtgrp->cpu_mask));
- else
+ if (rdtgrp) {
+ seq_printf(s, is_cpu_list(of) ? "%*pbl\n" : "%*pb\n",
+ cpumask_pr_args(&rdtgrp->cpu_mask));
+ } else {
ret = -ENOENT;
+ }
rdtgroup_kn_unlock(of->kn);
return ret;
@@ -252,7 +261,11 @@
goto unlock;
}
- ret = cpumask_parse(buf, newmask);
+ if (is_cpu_list(of))
+ ret = cpulist_parse(buf, newmask);
+ else
+ ret = cpumask_parse(buf, newmask);
+
if (ret)
goto unlock;
@@ -473,6 +486,14 @@
.seq_show = rdtgroup_cpus_show,
},
{
+ .name = "cpus_list",
+ .mode = 0644,
+ .kf_ops = &rdtgroup_kf_single_ops,
+ .write = rdtgroup_cpus_write,
+ .seq_show = rdtgroup_cpus_show,
+ .flags = RFTYPE_FLAGS_CPUS_LIST,
+ },
+ {
.name = "tasks",
.mode = 0644,
.kf_ops = &rdtgroup_kf_single_ops,
@@ -494,17 +515,15 @@
struct rdt_resource *r = of->kn->parent->priv;
seq_printf(seq, "%d\n", r->num_closid);
-
return 0;
}
-static int rdt_cbm_mask_show(struct kernfs_open_file *of,
+static int rdt_default_ctrl_show(struct kernfs_open_file *of,
struct seq_file *seq, void *v)
{
struct rdt_resource *r = of->kn->parent->priv;
- seq_printf(seq, "%x\n", r->max_cbm);
-
+ seq_printf(seq, "%x\n", r->default_ctrl);
return 0;
}
@@ -513,13 +532,39 @@
{
struct rdt_resource *r = of->kn->parent->priv;
- seq_printf(seq, "%d\n", r->min_cbm_bits);
+ seq_printf(seq, "%u\n", r->cache.min_cbm_bits);
+ return 0;
+}
+static int rdt_min_bw_show(struct kernfs_open_file *of,
+ struct seq_file *seq, void *v)
+{
+ struct rdt_resource *r = of->kn->parent->priv;
+
+ seq_printf(seq, "%u\n", r->membw.min_bw);
+ return 0;
+}
+
+static int rdt_bw_gran_show(struct kernfs_open_file *of,
+ struct seq_file *seq, void *v)
+{
+ struct rdt_resource *r = of->kn->parent->priv;
+
+ seq_printf(seq, "%u\n", r->membw.bw_gran);
+ return 0;
+}
+
+static int rdt_delay_linear_show(struct kernfs_open_file *of,
+ struct seq_file *seq, void *v)
+{
+ struct rdt_resource *r = of->kn->parent->priv;
+
+ seq_printf(seq, "%u\n", r->membw.delay_linear);
return 0;
}
/* rdtgroup information files for one cache resource. */
-static struct rftype res_info_files[] = {
+static struct rftype res_cache_info_files[] = {
{
.name = "num_closids",
.mode = 0444,
@@ -530,7 +575,7 @@
.name = "cbm_mask",
.mode = 0444,
.kf_ops = &rdtgroup_kf_single_ops,
- .seq_show = rdt_cbm_mask_show,
+ .seq_show = rdt_default_ctrl_show,
},
{
.name = "min_cbm_bits",
@@ -540,11 +585,52 @@
},
};
+/* rdtgroup information files for memory bandwidth. */
+static struct rftype res_mba_info_files[] = {
+ {
+ .name = "num_closids",
+ .mode = 0444,
+ .kf_ops = &rdtgroup_kf_single_ops,
+ .seq_show = rdt_num_closids_show,
+ },
+ {
+ .name = "min_bandwidth",
+ .mode = 0444,
+ .kf_ops = &rdtgroup_kf_single_ops,
+ .seq_show = rdt_min_bw_show,
+ },
+ {
+ .name = "bandwidth_gran",
+ .mode = 0444,
+ .kf_ops = &rdtgroup_kf_single_ops,
+ .seq_show = rdt_bw_gran_show,
+ },
+ {
+ .name = "delay_linear",
+ .mode = 0444,
+ .kf_ops = &rdtgroup_kf_single_ops,
+ .seq_show = rdt_delay_linear_show,
+ },
+};
+
+void rdt_get_mba_infofile(struct rdt_resource *r)
+{
+ r->info_files = res_mba_info_files;
+ r->nr_info_files = ARRAY_SIZE(res_mba_info_files);
+}
+
+void rdt_get_cache_infofile(struct rdt_resource *r)
+{
+ r->info_files = res_cache_info_files;
+ r->nr_info_files = ARRAY_SIZE(res_cache_info_files);
+}
+
static int rdtgroup_create_info_dir(struct kernfs_node *parent_kn)
{
struct kernfs_node *kn_subdir;
+ struct rftype *res_info_files;
struct rdt_resource *r;
- int ret;
+ int ret, len;
/* create the directory */
kn_info = kernfs_create_dir(parent_kn, "info", parent_kn->mode, NULL);
@@ -563,8 +649,11 @@
ret = rdtgroup_kn_set_ugid(kn_subdir);
if (ret)
goto out_destroy;
- ret = rdtgroup_add_files(kn_subdir, res_info_files,
- ARRAY_SIZE(res_info_files));
+
+ res_info_files = r->info_files;
+ len = r->nr_info_files;
+
+ ret = rdtgroup_add_files(kn_subdir, res_info_files, len);
if (ret)
goto out_destroy;
kernfs_activate(kn_subdir);
@@ -780,7 +869,7 @@
return dentry;
}
-static int reset_all_cbms(struct rdt_resource *r)
+static int reset_all_ctrls(struct rdt_resource *r)
{
struct msr_param msr_param;
cpumask_var_t cpu_mask;
@@ -803,14 +892,14 @@
cpumask_set_cpu(cpumask_any(&d->cpu_mask), cpu_mask);
for (i = 0; i < r->num_closid; i++)
- d->cbm[i] = r->max_cbm;
+ d->ctrl_val[i] = r->default_ctrl;
}
cpu = get_cpu();
/* Update CBM on this cpu if it's in cpu_mask. */
if (cpumask_test_cpu(cpu, cpu_mask))
- rdt_cbm_update(&msr_param);
+ rdt_ctrl_update(&msr_param);
/* Update CBM on all other cpus in cpu_mask. */
- smp_call_function_many(cpu_mask, rdt_cbm_update, &msr_param, 1);
+ smp_call_function_many(cpu_mask, rdt_ctrl_update, &msr_param, 1);
put_cpu();
free_cpumask_var(cpu_mask);
@@ -896,7 +985,7 @@
/*Put everything back to default values. */
for_each_enabled_rdt_resource(r)
- reset_all_cbms(r);
+ reset_all_ctrls(r);
cdp_disable();
rmdir_all_sub();
static_branch_disable(&rdt_enable_key);
diff --git a/arch/x86/kernel/cpu/intel_rdt_schemata.c b/arch/x86/kernel/cpu/intel_rdt_schemata.c
index badd2b3..406d7a6 100644
--- a/arch/x86/kernel/cpu/intel_rdt_schemata.c
+++ b/arch/x86/kernel/cpu/intel_rdt_schemata.c
@@ -29,26 +29,77 @@
#include <asm/intel_rdt.h>
/*
+ * Check whether MBA bandwidth percentage value is correct. The value is
+ * checked against the minimum and max bandwidth values specified by the
+ * hardware. The allocated bandwidth percentage is rounded to the next
+ * control step available on the hardware.
+ */
+static bool bw_validate(char *buf, unsigned long *data, struct rdt_resource *r)
+{
+ unsigned long bw;
+ int ret;
+
+ /*
+ * Only linear delay values is supported for current Intel SKUs.
+ */
+ if (!r->membw.delay_linear)
+ return false;
+
+ ret = kstrtoul(buf, 10, &bw);
+ if (ret)
+ return false;
+
+ if (bw < r->membw.min_bw || bw > r->default_ctrl)
+ return false;
+
+ *data = roundup(bw, (unsigned long)r->membw.bw_gran);
+ return true;
+}
+
+int parse_bw(char *buf, struct rdt_resource *r, struct rdt_domain *d)
+{
+ unsigned long data;
+
+ if (d->have_new_ctrl)
+ return -EINVAL;
+
+ if (!bw_validate(buf, &data, r))
+ return -EINVAL;
+ d->new_ctrl = data;
+ d->have_new_ctrl = true;
+
+ return 0;
+}
+
+/*
* Check whether a cache bit mask is valid. The SDM says:
* Please note that all (and only) contiguous '1' combinations
* are allowed (e.g. FFFFH, 0FF0H, 003CH, etc.).
* Additionally Haswell requires at least two bits set.
*/
-static bool cbm_validate(unsigned long var, struct rdt_resource *r)
+static bool cbm_validate(char *buf, unsigned long *data, struct rdt_resource *r)
{
- unsigned long first_bit, zero_bit;
+ unsigned long first_bit, zero_bit, val;
+ unsigned int cbm_len = r->cache.cbm_len;
+ int ret;
- if (var == 0 || var > r->max_cbm)
+ ret = kstrtoul(buf, 16, &val);
+ if (ret)
return false;
- first_bit = find_first_bit(&var, r->cbm_len);
- zero_bit = find_next_zero_bit(&var, r->cbm_len, first_bit);
-
- if (find_next_bit(&var, r->cbm_len, zero_bit) < r->cbm_len)
+ if (val == 0 || val > r->default_ctrl)
return false;
- if ((zero_bit - first_bit) < r->min_cbm_bits)
+ first_bit = find_first_bit(&val, cbm_len);
+ zero_bit = find_next_zero_bit(&val, cbm_len, first_bit);
+
+ if (find_next_bit(&val, cbm_len, zero_bit) < cbm_len)
return false;
+
+ if ((zero_bit - first_bit) < r->cache.min_cbm_bits)
+ return false;
+
+ *data = val;
return true;
}
@@ -56,17 +107,17 @@
* Read one cache bit mask (hex). Check that it is valid for the current
* resource type.
*/
-static int parse_cbm(char *buf, struct rdt_resource *r)
+int parse_cbm(char *buf, struct rdt_resource *r, struct rdt_domain *d)
{
unsigned long data;
- int ret;
- ret = kstrtoul(buf, 16, &data);
- if (ret)
- return ret;
- if (!cbm_validate(data, r))
+ if (d->have_new_ctrl)
return -EINVAL;
- r->tmp_cbms[r->num_tmp_cbms++] = data;
+
+ if(!cbm_validate(buf, &data, r))
+ return -EINVAL;
+ d->new_ctrl = data;
+ d->have_new_ctrl = true;
return 0;
}
@@ -74,8 +125,8 @@
/*
* For each domain in this resource we expect to find a series of:
* id=mask
- * separated by ";". The "id" is in decimal, and must appear in the
- * right order.
+ * separated by ";". The "id" is in decimal, and must match one of
+ * the "id"s for this resource.
*/
static int parse_line(char *line, struct rdt_resource *r)
{
@@ -83,21 +134,22 @@
struct rdt_domain *d;
unsigned long dom_id;
- list_for_each_entry(d, &r->domains, list) {
- dom = strsep(&line, ";");
- if (!dom)
- return -EINVAL;
- id = strsep(&dom, "=");
- if (kstrtoul(id, 10, &dom_id) || dom_id != d->id)
- return -EINVAL;
- if (parse_cbm(dom, r))
- return -EINVAL;
- }
-
- /* Any garbage at the end of the line? */
- if (line && line[0])
+next:
+ if (!line || line[0] == '\0')
+ return 0;
+ dom = strsep(&line, ";");
+ id = strsep(&dom, "=");
+ if (!dom || kstrtoul(id, 10, &dom_id))
return -EINVAL;
- return 0;
+ dom = strim(dom);
+ list_for_each_entry(d, &r->domains, list) {
+ if (d->id == dom_id) {
+ if (r->parse_ctrlval(dom, r, d))
+ return -EINVAL;
+ goto next;
+ }
+ }
+ return -EINVAL;
}
static int update_domains(struct rdt_resource *r, int closid)
@@ -105,7 +157,7 @@
struct msr_param msr_param;
cpumask_var_t cpu_mask;
struct rdt_domain *d;
- int cpu, idx = 0;
+ int cpu;
if (!zalloc_cpumask_var(&cpu_mask, GFP_KERNEL))
return -ENOMEM;
@@ -115,30 +167,46 @@
msr_param.res = r;
list_for_each_entry(d, &r->domains, list) {
- cpumask_set_cpu(cpumask_any(&d->cpu_mask), cpu_mask);
- d->cbm[msr_param.low] = r->tmp_cbms[idx++];
+ if (d->have_new_ctrl && d->new_ctrl != d->ctrl_val[closid]) {
+ cpumask_set_cpu(cpumask_any(&d->cpu_mask), cpu_mask);
+ d->ctrl_val[closid] = d->new_ctrl;
+ }
}
+ if (cpumask_empty(cpu_mask))
+ goto done;
cpu = get_cpu();
/* Update CBM on this cpu if it's in cpu_mask. */
if (cpumask_test_cpu(cpu, cpu_mask))
- rdt_cbm_update(&msr_param);
+ rdt_ctrl_update(&msr_param);
/* Update CBM on other cpus. */
- smp_call_function_many(cpu_mask, rdt_cbm_update, &msr_param, 1);
+ smp_call_function_many(cpu_mask, rdt_ctrl_update, &msr_param, 1);
put_cpu();
+done:
free_cpumask_var(cpu_mask);
return 0;
}
+static int rdtgroup_parse_resource(char *resname, char *tok, int closid)
+{
+ struct rdt_resource *r;
+
+ for_each_enabled_rdt_resource(r) {
+ if (!strcmp(resname, r->name) && closid < r->num_closid)
+ return parse_line(tok, r);
+ }
+ return -EINVAL;
+}
+
ssize_t rdtgroup_schemata_write(struct kernfs_open_file *of,
char *buf, size_t nbytes, loff_t off)
{
struct rdtgroup *rdtgrp;
+ struct rdt_domain *dom;
struct rdt_resource *r;
char *tok, *resname;
int closid, ret = 0;
- u32 *l3_cbms = NULL;
/* Valid input requires a trailing newline */
if (nbytes == 0 || buf[nbytes - 1] != '\n')
@@ -153,44 +221,20 @@
closid = rdtgrp->closid;
- /* get scratch space to save all the masks while we validate input */
for_each_enabled_rdt_resource(r) {
- r->tmp_cbms = kcalloc(r->num_domains, sizeof(*l3_cbms),
- GFP_KERNEL);
- if (!r->tmp_cbms) {
- ret = -ENOMEM;
- goto out;
- }
- r->num_tmp_cbms = 0;
+ list_for_each_entry(dom, &r->domains, list)
+ dom->have_new_ctrl = false;
}
while ((tok = strsep(&buf, "\n")) != NULL) {
- resname = strsep(&tok, ":");
+ resname = strim(strsep(&tok, ":"));
if (!tok) {
ret = -EINVAL;
goto out;
}
- for_each_enabled_rdt_resource(r) {
- if (!strcmp(resname, r->name) &&
- closid < r->num_closid) {
- ret = parse_line(tok, r);
- if (ret)
- goto out;
- break;
- }
- }
- if (!r->name) {
- ret = -EINVAL;
+ ret = rdtgroup_parse_resource(resname, tok, closid);
+ if (ret)
goto out;
- }
- }
-
- /* Did the parser find all the masks we need? */
- for_each_enabled_rdt_resource(r) {
- if (r->num_tmp_cbms != r->num_domains) {
- ret = -EINVAL;
- goto out;
- }
}
for_each_enabled_rdt_resource(r) {
@@ -200,10 +244,6 @@
}
out:
- for_each_enabled_rdt_resource(r) {
- kfree(r->tmp_cbms);
- r->tmp_cbms = NULL;
- }
rdtgroup_kn_unlock(of->kn);
return ret ?: nbytes;
}
@@ -213,11 +253,12 @@
struct rdt_domain *dom;
bool sep = false;
- seq_printf(s, "%s:", r->name);
+ seq_printf(s, "%*s:", max_name_width, r->name);
list_for_each_entry(dom, &r->domains, list) {
if (sep)
seq_puts(s, ";");
- seq_printf(s, "%d=%x", dom->id, dom->cbm[closid]);
+ seq_printf(s, r->format_str, dom->id, max_data_width,
+ dom->ctrl_val[closid]);
sep = true;
}
seq_puts(s, "\n");
diff --git a/arch/x86/kernel/cpu/mcheck/Makefile b/arch/x86/kernel/cpu/mcheck/Makefile
index a3311c8..43051f0 100644
--- a/arch/x86/kernel/cpu/mcheck/Makefile
+++ b/arch/x86/kernel/cpu/mcheck/Makefile
@@ -9,3 +9,5 @@
obj-$(CONFIG_X86_THERMAL_VECTOR) += therm_throt.o
obj-$(CONFIG_ACPI_APEI) += mce-apei.o
+
+obj-$(CONFIG_X86_MCELOG_LEGACY) += dev-mcelog.o
diff --git a/arch/x86/kernel/cpu/mcheck/dev-mcelog.c b/arch/x86/kernel/cpu/mcheck/dev-mcelog.c
new file mode 100644
index 0000000..9c632cb8
--- /dev/null
+++ b/arch/x86/kernel/cpu/mcheck/dev-mcelog.c
@@ -0,0 +1,397 @@
+/*
+ * /dev/mcelog driver
+ *
+ * K8 parts Copyright 2002,2003 Andi Kleen, SuSE Labs.
+ * Rest from unknown author(s).
+ * 2004 Andi Kleen. Rewrote most of it.
+ * Copyright 2008 Intel Corporation
+ * Author: Andi Kleen
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/miscdevice.h>
+#include <linux/slab.h>
+#include <linux/kmod.h>
+#include <linux/poll.h>
+
+#include "mce-internal.h"
+
+static DEFINE_MUTEX(mce_chrdev_read_mutex);
+
+static char mce_helper[128];
+static char *mce_helper_argv[2] = { mce_helper, NULL };
+
+#define mce_log_get_idx_check(p) \
+({ \
+ RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held() && \
+ !lockdep_is_held(&mce_chrdev_read_mutex), \
+ "suspicious mce_log_get_idx_check() usage"); \
+ smp_load_acquire(&(p)); \
+})
+
+/*
+ * Lockless MCE logging infrastructure.
+ * This avoids deadlocks on printk locks without having to break locks. Also
+ * separate MCEs from kernel messages to avoid bogus bug reports.
+ */
+
+static struct mce_log_buffer mcelog = {
+ .signature = MCE_LOG_SIGNATURE,
+ .len = MCE_LOG_LEN,
+ .recordlen = sizeof(struct mce),
+};
+
+static DECLARE_WAIT_QUEUE_HEAD(mce_chrdev_wait);
+
+/* User mode helper program triggered by machine check event */
+extern char mce_helper[128];
+
+static int dev_mce_log(struct notifier_block *nb, unsigned long val,
+ void *data)
+{
+ struct mce *mce = (struct mce *)data;
+ unsigned int next, entry;
+
+ wmb();
+ for (;;) {
+ entry = mce_log_get_idx_check(mcelog.next);
+ for (;;) {
+
+ /*
+ * When the buffer fills up discard new entries.
+ * Assume that the earlier errors are the more
+ * interesting ones:
+ */
+ if (entry >= MCE_LOG_LEN) {
+ set_bit(MCE_OVERFLOW,
+ (unsigned long *)&mcelog.flags);
+ return NOTIFY_OK;
+ }
+ /* Old left over entry. Skip: */
+ if (mcelog.entry[entry].finished) {
+ entry++;
+ continue;
+ }
+ break;
+ }
+ smp_rmb();
+ next = entry + 1;
+ if (cmpxchg(&mcelog.next, entry, next) == entry)
+ break;
+ }
+ memcpy(mcelog.entry + entry, mce, sizeof(struct mce));
+ wmb();
+ mcelog.entry[entry].finished = 1;
+ wmb();
+
+ /* wake processes polling /dev/mcelog */
+ wake_up_interruptible(&mce_chrdev_wait);
+
+ return NOTIFY_OK;
+}
+
+static struct notifier_block dev_mcelog_nb = {
+ .notifier_call = dev_mce_log,
+ .priority = MCE_PRIO_MCELOG,
+};
+
+static void mce_do_trigger(struct work_struct *work)
+{
+ call_usermodehelper(mce_helper, mce_helper_argv, NULL, UMH_NO_WAIT);
+}
+
+static DECLARE_WORK(mce_trigger_work, mce_do_trigger);
+
+
+void mce_work_trigger(void)
+{
+ if (mce_helper[0])
+ schedule_work(&mce_trigger_work);
+}
+
+static ssize_t
+show_trigger(struct device *s, struct device_attribute *attr, char *buf)
+{
+ strcpy(buf, mce_helper);
+ strcat(buf, "\n");
+ return strlen(mce_helper) + 1;
+}
+
+static ssize_t set_trigger(struct device *s, struct device_attribute *attr,
+ const char *buf, size_t siz)
+{
+ char *p;
+
+ strncpy(mce_helper, buf, sizeof(mce_helper));
+ mce_helper[sizeof(mce_helper)-1] = 0;
+ p = strchr(mce_helper, '\n');
+
+ if (p)
+ *p = 0;
+
+ return strlen(mce_helper) + !!p;
+}
+
+DEVICE_ATTR(trigger, 0644, show_trigger, set_trigger);
+
+/*
+ * mce_chrdev: Character device /dev/mcelog to read and clear the MCE log.
+ */
+
+static DEFINE_SPINLOCK(mce_chrdev_state_lock);
+static int mce_chrdev_open_count; /* #times opened */
+static int mce_chrdev_open_exclu; /* already open exclusive? */
+
+static int mce_chrdev_open(struct inode *inode, struct file *file)
+{
+ spin_lock(&mce_chrdev_state_lock);
+
+ if (mce_chrdev_open_exclu ||
+ (mce_chrdev_open_count && (file->f_flags & O_EXCL))) {
+ spin_unlock(&mce_chrdev_state_lock);
+
+ return -EBUSY;
+ }
+
+ if (file->f_flags & O_EXCL)
+ mce_chrdev_open_exclu = 1;
+ mce_chrdev_open_count++;
+
+ spin_unlock(&mce_chrdev_state_lock);
+
+ return nonseekable_open(inode, file);
+}
+
+static int mce_chrdev_release(struct inode *inode, struct file *file)
+{
+ spin_lock(&mce_chrdev_state_lock);
+
+ mce_chrdev_open_count--;
+ mce_chrdev_open_exclu = 0;
+
+ spin_unlock(&mce_chrdev_state_lock);
+
+ return 0;
+}
+
+static void collect_tscs(void *data)
+{
+ unsigned long *cpu_tsc = (unsigned long *)data;
+
+ cpu_tsc[smp_processor_id()] = rdtsc();
+}
+
+static int mce_apei_read_done;
+
+/* Collect MCE record of previous boot in persistent storage via APEI ERST. */
+static int __mce_read_apei(char __user **ubuf, size_t usize)
+{
+ int rc;
+ u64 record_id;
+ struct mce m;
+
+ if (usize < sizeof(struct mce))
+ return -EINVAL;
+
+ rc = apei_read_mce(&m, &record_id);
+ /* Error or no more MCE record */
+ if (rc <= 0) {
+ mce_apei_read_done = 1;
+ /*
+ * When ERST is disabled, mce_chrdev_read() should return
+ * "no record" instead of "no device."
+ */
+ if (rc == -ENODEV)
+ return 0;
+ return rc;
+ }
+ rc = -EFAULT;
+ if (copy_to_user(*ubuf, &m, sizeof(struct mce)))
+ return rc;
+ /*
+ * In fact, we should have cleared the record after that has
+ * been flushed to the disk or sent to network in
+ * /sbin/mcelog, but we have no interface to support that now,
+ * so just clear it to avoid duplication.
+ */
+ rc = apei_clear_mce(record_id);
+ if (rc) {
+ mce_apei_read_done = 1;
+ return rc;
+ }
+ *ubuf += sizeof(struct mce);
+
+ return 0;
+}
+
+static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf,
+ size_t usize, loff_t *off)
+{
+ char __user *buf = ubuf;
+ unsigned long *cpu_tsc;
+ unsigned prev, next;
+ int i, err;
+
+ cpu_tsc = kmalloc(nr_cpu_ids * sizeof(long), GFP_KERNEL);
+ if (!cpu_tsc)
+ return -ENOMEM;
+
+ mutex_lock(&mce_chrdev_read_mutex);
+
+ if (!mce_apei_read_done) {
+ err = __mce_read_apei(&buf, usize);
+ if (err || buf != ubuf)
+ goto out;
+ }
+
+ next = mce_log_get_idx_check(mcelog.next);
+
+ /* Only supports full reads right now */
+ err = -EINVAL;
+ if (*off != 0 || usize < MCE_LOG_LEN*sizeof(struct mce))
+ goto out;
+
+ err = 0;
+ prev = 0;
+ do {
+ for (i = prev; i < next; i++) {
+ unsigned long start = jiffies;
+ struct mce *m = &mcelog.entry[i];
+
+ while (!m->finished) {
+ if (time_after_eq(jiffies, start + 2)) {
+ memset(m, 0, sizeof(*m));
+ goto timeout;
+ }
+ cpu_relax();
+ }
+ smp_rmb();
+ err |= copy_to_user(buf, m, sizeof(*m));
+ buf += sizeof(*m);
+timeout:
+ ;
+ }
+
+ memset(mcelog.entry + prev, 0,
+ (next - prev) * sizeof(struct mce));
+ prev = next;
+ next = cmpxchg(&mcelog.next, prev, 0);
+ } while (next != prev);
+
+ synchronize_sched();
+
+ /*
+ * Collect entries that were still getting written before the
+ * synchronize.
+ */
+ on_each_cpu(collect_tscs, cpu_tsc, 1);
+
+ for (i = next; i < MCE_LOG_LEN; i++) {
+ struct mce *m = &mcelog.entry[i];
+
+ if (m->finished && m->tsc < cpu_tsc[m->cpu]) {
+ err |= copy_to_user(buf, m, sizeof(*m));
+ smp_rmb();
+ buf += sizeof(*m);
+ memset(m, 0, sizeof(*m));
+ }
+ }
+
+ if (err)
+ err = -EFAULT;
+
+out:
+ mutex_unlock(&mce_chrdev_read_mutex);
+ kfree(cpu_tsc);
+
+ return err ? err : buf - ubuf;
+}
+
+static unsigned int mce_chrdev_poll(struct file *file, poll_table *wait)
+{
+ poll_wait(file, &mce_chrdev_wait, wait);
+ if (READ_ONCE(mcelog.next))
+ return POLLIN | POLLRDNORM;
+ if (!mce_apei_read_done && apei_check_mce())
+ return POLLIN | POLLRDNORM;
+ return 0;
+}
+
+static long mce_chrdev_ioctl(struct file *f, unsigned int cmd,
+ unsigned long arg)
+{
+ int __user *p = (int __user *)arg;
+
+ if (!capable(CAP_SYS_ADMIN))
+ return -EPERM;
+
+ switch (cmd) {
+ case MCE_GET_RECORD_LEN:
+ return put_user(sizeof(struct mce), p);
+ case MCE_GET_LOG_LEN:
+ return put_user(MCE_LOG_LEN, p);
+ case MCE_GETCLEAR_FLAGS: {
+ unsigned flags;
+
+ do {
+ flags = mcelog.flags;
+ } while (cmpxchg(&mcelog.flags, flags, 0) != flags);
+
+ return put_user(flags, p);
+ }
+ default:
+ return -ENOTTY;
+ }
+}
+
+static ssize_t (*mce_write)(struct file *filp, const char __user *ubuf,
+ size_t usize, loff_t *off);
+
+void register_mce_write_callback(ssize_t (*fn)(struct file *filp,
+ const char __user *ubuf,
+ size_t usize, loff_t *off))
+{
+ mce_write = fn;
+}
+EXPORT_SYMBOL_GPL(register_mce_write_callback);
+
+static ssize_t mce_chrdev_write(struct file *filp, const char __user *ubuf,
+ size_t usize, loff_t *off)
+{
+ if (mce_write)
+ return mce_write(filp, ubuf, usize, off);
+ else
+ return -EINVAL;
+}
+
+static const struct file_operations mce_chrdev_ops = {
+ .open = mce_chrdev_open,
+ .release = mce_chrdev_release,
+ .read = mce_chrdev_read,
+ .write = mce_chrdev_write,
+ .poll = mce_chrdev_poll,
+ .unlocked_ioctl = mce_chrdev_ioctl,
+ .llseek = no_llseek,
+};
+
+static struct miscdevice mce_chrdev_device = {
+ MISC_MCELOG_MINOR,
+ "mcelog",
+ &mce_chrdev_ops,
+};
+
+static __init int dev_mcelog_init_device(void)
+{
+ int err;
+
+ /* register character device /dev/mcelog */
+ err = misc_register(&mce_chrdev_device);
+ if (err) {
+ pr_err("Unable to init device /dev/mcelog (rc: %d)\n", err);
+ return err;
+ }
+ mce_register_decode_chain(&dev_mcelog_nb);
+ return 0;
+}
+device_initcall_sync(dev_mcelog_init_device);
diff --git a/arch/x86/kernel/cpu/mcheck/mce-genpool.c b/arch/x86/kernel/cpu/mcheck/mce-genpool.c
index 1e5a50c..217cd44 100644
--- a/arch/x86/kernel/cpu/mcheck/mce-genpool.c
+++ b/arch/x86/kernel/cpu/mcheck/mce-genpool.c
@@ -85,7 +85,7 @@
head = llist_reverse_order(head);
llist_for_each_entry_safe(node, tmp, head, llnode) {
mce = &node->mce;
- atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, mce);
+ blocking_notifier_call_chain(&x86_mce_decoder_chain, 0, mce);
gen_pool_free(mce_evt_pool, (unsigned long)node, sizeof(*node));
}
}
diff --git a/arch/x86/kernel/cpu/mcheck/mce-internal.h b/arch/x86/kernel/cpu/mcheck/mce-internal.h
index 903043e..654ad06 100644
--- a/arch/x86/kernel/cpu/mcheck/mce-internal.h
+++ b/arch/x86/kernel/cpu/mcheck/mce-internal.h
@@ -13,7 +13,7 @@
MCE_PANIC_SEVERITY,
};
-extern struct atomic_notifier_head x86_mce_decoder_chain;
+extern struct blocking_notifier_head x86_mce_decoder_chain;
#define ATTR_LEN 16
#define INITIAL_CHECK_INTERVAL 5 * 60 /* 5 minutes */
@@ -96,3 +96,11 @@
m1->addr != m2->addr ||
m1->misc != m2->misc;
}
+
+extern struct device_attribute dev_attr_trigger;
+
+#ifdef CONFIG_X86_MCELOG_LEGACY
+extern void mce_work_trigger(void);
+#else
+static inline void mce_work_trigger(void) { }
+#endif
diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c
index 5accfbd..5abd4bf 100644
--- a/arch/x86/kernel/cpu/mcheck/mce.c
+++ b/arch/x86/kernel/cpu/mcheck/mce.c
@@ -35,6 +35,7 @@
#include <linux/poll.h>
#include <linux/nmi.h>
#include <linux/cpu.h>
+#include <linux/ras.h>
#include <linux/smp.h>
#include <linux/fs.h>
#include <linux/mm.h>
@@ -49,20 +50,11 @@
#include <asm/tlbflush.h>
#include <asm/mce.h>
#include <asm/msr.h>
+#include <asm/reboot.h>
#include "mce-internal.h"
-static DEFINE_MUTEX(mce_chrdev_read_mutex);
-
-static int mce_chrdev_open_count; /* #times opened */
-
-#define mce_log_get_idx_check(p) \
-({ \
- RCU_LOCKDEP_WARN(!rcu_read_lock_sched_held() && \
- !lockdep_is_held(&mce_chrdev_read_mutex), \
- "suspicious mce_log_get_idx_check() usage"); \
- smp_load_acquire(&(p)); \
-})
+static DEFINE_MUTEX(mce_log_mutex);
#define CREATE_TRACE_POINTS
#include <trace/events/mce.h>
@@ -87,15 +79,9 @@
.monarch_timeout = -1
};
-/* User mode helper program triggered by machine check event */
-static unsigned long mce_need_notify;
-static char mce_helper[128];
-static char *mce_helper_argv[2] = { mce_helper, NULL };
-
-static DECLARE_WAIT_QUEUE_HEAD(mce_chrdev_wait);
-
static DEFINE_PER_CPU(struct mce, mces_seen);
-static int cpu_missing;
+static unsigned long mce_need_notify;
+static int cpu_missing;
/*
* MCA banks polled by the period polling timer for corrected events.
@@ -123,7 +109,7 @@
* CPU/chipset specific EDAC code can register a notifier call here to print
* MCE errors in a human-readable form.
*/
-ATOMIC_NOTIFIER_HEAD(x86_mce_decoder_chain);
+BLOCKING_NOTIFIER_HEAD(x86_mce_decoder_chain);
/* Do initial initialization of a struct mce */
void mce_setup(struct mce *m)
@@ -145,82 +131,38 @@
DEFINE_PER_CPU(struct mce, injectm);
EXPORT_PER_CPU_SYMBOL_GPL(injectm);
-/*
- * Lockless MCE logging infrastructure.
- * This avoids deadlocks on printk locks without having to break locks. Also
- * separate MCEs from kernel messages to avoid bogus bug reports.
- */
-
-static struct mce_log mcelog = {
- .signature = MCE_LOG_SIGNATURE,
- .len = MCE_LOG_LEN,
- .recordlen = sizeof(struct mce),
-};
-
-void mce_log(struct mce *mce)
+void mce_log(struct mce *m)
{
- unsigned next, entry;
-
- /* Emit the trace record: */
- trace_mce_record(mce);
-
- if (!mce_gen_pool_add(mce))
+ if (!mce_gen_pool_add(m))
irq_work_queue(&mce_irq_work);
-
- wmb();
- for (;;) {
- entry = mce_log_get_idx_check(mcelog.next);
- for (;;) {
-
- /*
- * When the buffer fills up discard new entries.
- * Assume that the earlier errors are the more
- * interesting ones:
- */
- if (entry >= MCE_LOG_LEN) {
- set_bit(MCE_OVERFLOW,
- (unsigned long *)&mcelog.flags);
- return;
- }
- /* Old left over entry. Skip: */
- if (mcelog.entry[entry].finished) {
- entry++;
- continue;
- }
- break;
- }
- smp_rmb();
- next = entry + 1;
- if (cmpxchg(&mcelog.next, entry, next) == entry)
- break;
- }
- memcpy(mcelog.entry + entry, mce, sizeof(struct mce));
- wmb();
- mcelog.entry[entry].finished = 1;
- wmb();
-
- set_bit(0, &mce_need_notify);
}
void mce_inject_log(struct mce *m)
{
- mutex_lock(&mce_chrdev_read_mutex);
+ mutex_lock(&mce_log_mutex);
mce_log(m);
- mutex_unlock(&mce_chrdev_read_mutex);
+ mutex_unlock(&mce_log_mutex);
}
EXPORT_SYMBOL_GPL(mce_inject_log);
static struct notifier_block mce_srao_nb;
+/*
+ * We run the default notifier if we have only the SRAO, the first and the
+ * default notifier registered. I.e., the mandatory NUM_DEFAULT_NOTIFIERS
+ * notifiers registered on the chain.
+ */
+#define NUM_DEFAULT_NOTIFIERS 3
static atomic_t num_notifiers;
void mce_register_decode_chain(struct notifier_block *nb)
{
+ if (WARN_ON(nb->priority > MCE_PRIO_MCELOG && nb->priority < MCE_PRIO_EDAC))
+ return;
+
atomic_inc(&num_notifiers);
- WARN_ON(nb->priority > MCE_PRIO_LOWEST && nb->priority < MCE_PRIO_EDAC);
-
- atomic_notifier_chain_register(&x86_mce_decoder_chain, nb);
+ blocking_notifier_chain_register(&x86_mce_decoder_chain, nb);
}
EXPORT_SYMBOL_GPL(mce_register_decode_chain);
@@ -228,7 +170,7 @@
{
atomic_dec(&num_notifiers);
- atomic_notifier_chain_unregister(&x86_mce_decoder_chain, nb);
+ blocking_notifier_chain_unregister(&x86_mce_decoder_chain, nb);
}
EXPORT_SYMBOL_GPL(mce_unregister_decode_chain);
@@ -321,18 +263,7 @@
static void print_mce(struct mce *m)
{
- int ret = 0;
-
__print_mce(m);
-
- /*
- * Print out human-readable details about the MCE error,
- * (if the CPU has an implementation for that)
- */
- ret = atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, m);
- if (ret == NOTIFY_STOP)
- return;
-
pr_emerg_ratelimited(HW_ERR "Run the above through 'mcelog --ascii'\n");
}
@@ -521,7 +452,6 @@
static void mce_irq_work_cb(struct irq_work *entry)
{
- mce_notify_irq();
mce_schedule_work();
}
@@ -550,20 +480,97 @@
*/
static int mce_usable_address(struct mce *m)
{
- if (!(m->status & MCI_STATUS_MISCV) || !(m->status & MCI_STATUS_ADDRV))
+ if (!(m->status & MCI_STATUS_ADDRV))
return 0;
/* Checks after this one are Intel-specific: */
if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
return 1;
+ if (!(m->status & MCI_STATUS_MISCV))
+ return 0;
+
if (MCI_MISC_ADDR_LSB(m->misc) > PAGE_SHIFT)
return 0;
+
if (MCI_MISC_ADDR_MODE(m->misc) != MCI_MISC_ADDR_PHYS)
return 0;
+
return 1;
}
+static bool memory_error(struct mce *m)
+{
+ struct cpuinfo_x86 *c = &boot_cpu_data;
+
+ if (c->x86_vendor == X86_VENDOR_AMD) {
+ /* ErrCodeExt[20:16] */
+ u8 xec = (m->status >> 16) & 0x1f;
+
+ return (xec == 0x0 || xec == 0x8);
+ } else if (c->x86_vendor == X86_VENDOR_INTEL) {
+ /*
+ * Intel SDM Volume 3B - 15.9.2 Compound Error Codes
+ *
+ * Bit 7 of the MCACOD field of IA32_MCi_STATUS is used for
+ * indicating a memory error. Bit 8 is used for indicating a
+ * cache hierarchy error. The combination of bit 2 and bit 3
+ * is used for indicating a `generic' cache hierarchy error
+ * But we can't just blindly check the above bits, because if
+ * bit 11 is set, then it is a bus/interconnect error - and
+ * either way the above bits just gives more detail on what
+ * bus/interconnect error happened. Note that bit 12 can be
+ * ignored, as it's the "filter" bit.
+ */
+ return (m->status & 0xef80) == BIT(7) ||
+ (m->status & 0xef00) == BIT(8) ||
+ (m->status & 0xeffc) == 0xc;
+ }
+
+ return false;
+}
+
+static bool cec_add_mce(struct mce *m)
+{
+ if (!m)
+ return false;
+
+ /* We eat only correctable DRAM errors with usable addresses. */
+ if (memory_error(m) &&
+ !(m->status & MCI_STATUS_UC) &&
+ mce_usable_address(m))
+ if (!cec_add_elem(m->addr >> PAGE_SHIFT))
+ return true;
+
+ return false;
+}
+
+static int mce_first_notifier(struct notifier_block *nb, unsigned long val,
+ void *data)
+{
+ struct mce *m = (struct mce *)data;
+
+ if (!m)
+ return NOTIFY_DONE;
+
+ if (cec_add_mce(m))
+ return NOTIFY_STOP;
+
+ /* Emit the trace record: */
+ trace_mce_record(m);
+
+ set_bit(0, &mce_need_notify);
+
+ mce_notify_irq();
+
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block first_nb = {
+ .notifier_call = mce_first_notifier,
+ .priority = MCE_PRIO_FIRST,
+};
+
static int srao_decode_notifier(struct notifier_block *nb, unsigned long val,
void *data)
{
@@ -593,15 +600,7 @@
if (!m)
return NOTIFY_DONE;
- /*
- * Run the default notifier if we have only the SRAO
- * notifier and us registered.
- */
- if (atomic_read(&num_notifiers) > 2)
- return NOTIFY_DONE;
-
- /* Don't print when mcelog is running */
- if (mce_chrdev_open_count > 0)
+ if (atomic_read(&num_notifiers) > NUM_DEFAULT_NOTIFIERS)
return NOTIFY_DONE;
__print_mce(m);
@@ -654,37 +653,6 @@
}
}
-static bool memory_error(struct mce *m)
-{
- struct cpuinfo_x86 *c = &boot_cpu_data;
-
- if (c->x86_vendor == X86_VENDOR_AMD) {
- /* ErrCodeExt[20:16] */
- u8 xec = (m->status >> 16) & 0x1f;
-
- return (xec == 0x0 || xec == 0x8);
- } else if (c->x86_vendor == X86_VENDOR_INTEL) {
- /*
- * Intel SDM Volume 3B - 15.9.2 Compound Error Codes
- *
- * Bit 7 of the MCACOD field of IA32_MCi_STATUS is used for
- * indicating a memory error. Bit 8 is used for indicating a
- * cache hierarchy error. The combination of bit 2 and bit 3
- * is used for indicating a `generic' cache hierarchy error
- * But we can't just blindly check the above bits, because if
- * bit 11 is set, then it is a bus/interconnect error - and
- * either way the above bits just gives more detail on what
- * bus/interconnect error happened. Note that bit 12 can be
- * ignored, as it's the "filter" bit.
- */
- return (m->status & 0xef80) == BIT(7) ||
- (m->status & 0xef00) == BIT(8) ||
- (m->status & 0xeffc) == 0xc;
- }
-
- return false;
-}
-
DEFINE_PER_CPU(unsigned, mce_poll_count);
/*
@@ -1133,9 +1101,22 @@
* on Intel.
*/
int lmce = 1;
+ int cpu = smp_processor_id();
- /* If this CPU is offline, just bail out. */
- if (cpu_is_offline(smp_processor_id())) {
+ /*
+ * Cases where we avoid rendezvous handler timeout:
+ * 1) If this CPU is offline.
+ *
+ * 2) If crashing_cpu was set, e.g. we're entering kdump and we need to
+ * skip those CPUs which remain looping in the 1st kernel - see
+ * crash_nmi_callback().
+ *
+ * Note: there still is a small window between kexec-ing and the new,
+ * kdump kernel establishing a new #MC handler where a broadcasted MCE
+ * might not get handled properly.
+ */
+ if (cpu_is_offline(cpu) ||
+ (crashing_cpu != -1 && crashing_cpu != cpu)) {
u64 mcgstatus;
mcgstatus = mce_rdmsrl(MSR_IA32_MCG_STATUS);
@@ -1405,13 +1386,6 @@
del_timer_sync(&per_cpu(mce_timer, cpu));
}
-static void mce_do_trigger(struct work_struct *work)
-{
- call_usermodehelper(mce_helper, mce_helper_argv, NULL, UMH_NO_WAIT);
-}
-
-static DECLARE_WORK(mce_trigger_work, mce_do_trigger);
-
/*
* Notify the user(s) about new machine check events.
* Can be called from interrupt context, but not from machine check/NMI
@@ -1423,11 +1397,7 @@
static DEFINE_RATELIMIT_STATE(ratelimit, 60*HZ, 2);
if (test_and_clear_bit(0, &mce_need_notify)) {
- /* wake processes polling /dev/mcelog */
- wake_up_interruptible(&mce_chrdev_wait);
-
- if (mce_helper[0])
- schedule_work(&mce_trigger_work);
+ mce_work_trigger();
if (__ratelimit(&ratelimit))
pr_info(HW_ERR "Machine check events logged\n");
@@ -1694,6 +1664,25 @@
return 0;
}
+/*
+ * Init basic CPU features needed for early decoding of MCEs.
+ */
+static void __mcheck_cpu_init_early(struct cpuinfo_x86 *c)
+{
+ if (c->x86_vendor == X86_VENDOR_AMD) {
+ mce_flags.overflow_recov = !!cpu_has(c, X86_FEATURE_OVERFLOW_RECOV);
+ mce_flags.succor = !!cpu_has(c, X86_FEATURE_SUCCOR);
+ mce_flags.smca = !!cpu_has(c, X86_FEATURE_SMCA);
+
+ if (mce_flags.smca) {
+ msr_ops.ctl = smca_ctl_reg;
+ msr_ops.status = smca_status_reg;
+ msr_ops.addr = smca_addr_reg;
+ msr_ops.misc = smca_misc_reg;
+ }
+ }
+}
+
static void __mcheck_cpu_init_vendor(struct cpuinfo_x86 *c)
{
switch (c->x86_vendor) {
@@ -1703,21 +1692,7 @@
break;
case X86_VENDOR_AMD: {
- mce_flags.overflow_recov = !!cpu_has(c, X86_FEATURE_OVERFLOW_RECOV);
- mce_flags.succor = !!cpu_has(c, X86_FEATURE_SUCCOR);
- mce_flags.smca = !!cpu_has(c, X86_FEATURE_SMCA);
-
- /*
- * Install proper ops for Scalable MCA enabled processors
- */
- if (mce_flags.smca) {
- msr_ops.ctl = smca_ctl_reg;
- msr_ops.status = smca_status_reg;
- msr_ops.addr = smca_addr_reg;
- msr_ops.misc = smca_misc_reg;
- }
mce_amd_feature_init(c);
-
break;
}
@@ -1804,6 +1779,7 @@
machine_check_vector = do_machine_check;
+ __mcheck_cpu_init_early(c);
__mcheck_cpu_init_generic();
__mcheck_cpu_init_vendor(c);
__mcheck_cpu_init_clear_banks();
@@ -1829,251 +1805,6 @@
}
-/*
- * mce_chrdev: Character device /dev/mcelog to read and clear the MCE log.
- */
-
-static DEFINE_SPINLOCK(mce_chrdev_state_lock);
-static int mce_chrdev_open_exclu; /* already open exclusive? */
-
-static int mce_chrdev_open(struct inode *inode, struct file *file)
-{
- spin_lock(&mce_chrdev_state_lock);
-
- if (mce_chrdev_open_exclu ||
- (mce_chrdev_open_count && (file->f_flags & O_EXCL))) {
- spin_unlock(&mce_chrdev_state_lock);
-
- return -EBUSY;
- }
-
- if (file->f_flags & O_EXCL)
- mce_chrdev_open_exclu = 1;
- mce_chrdev_open_count++;
-
- spin_unlock(&mce_chrdev_state_lock);
-
- return nonseekable_open(inode, file);
-}
-
-static int mce_chrdev_release(struct inode *inode, struct file *file)
-{
- spin_lock(&mce_chrdev_state_lock);
-
- mce_chrdev_open_count--;
- mce_chrdev_open_exclu = 0;
-
- spin_unlock(&mce_chrdev_state_lock);
-
- return 0;
-}
-
-static void collect_tscs(void *data)
-{
- unsigned long *cpu_tsc = (unsigned long *)data;
-
- cpu_tsc[smp_processor_id()] = rdtsc();
-}
-
-static int mce_apei_read_done;
-
-/* Collect MCE record of previous boot in persistent storage via APEI ERST. */
-static int __mce_read_apei(char __user **ubuf, size_t usize)
-{
- int rc;
- u64 record_id;
- struct mce m;
-
- if (usize < sizeof(struct mce))
- return -EINVAL;
-
- rc = apei_read_mce(&m, &record_id);
- /* Error or no more MCE record */
- if (rc <= 0) {
- mce_apei_read_done = 1;
- /*
- * When ERST is disabled, mce_chrdev_read() should return
- * "no record" instead of "no device."
- */
- if (rc == -ENODEV)
- return 0;
- return rc;
- }
- rc = -EFAULT;
- if (copy_to_user(*ubuf, &m, sizeof(struct mce)))
- return rc;
- /*
- * In fact, we should have cleared the record after that has
- * been flushed to the disk or sent to network in
- * /sbin/mcelog, but we have no interface to support that now,
- * so just clear it to avoid duplication.
- */
- rc = apei_clear_mce(record_id);
- if (rc) {
- mce_apei_read_done = 1;
- return rc;
- }
- *ubuf += sizeof(struct mce);
-
- return 0;
-}
-
-static ssize_t mce_chrdev_read(struct file *filp, char __user *ubuf,
- size_t usize, loff_t *off)
-{
- char __user *buf = ubuf;
- unsigned long *cpu_tsc;
- unsigned prev, next;
- int i, err;
-
- cpu_tsc = kmalloc(nr_cpu_ids * sizeof(long), GFP_KERNEL);
- if (!cpu_tsc)
- return -ENOMEM;
-
- mutex_lock(&mce_chrdev_read_mutex);
-
- if (!mce_apei_read_done) {
- err = __mce_read_apei(&buf, usize);
- if (err || buf != ubuf)
- goto out;
- }
-
- next = mce_log_get_idx_check(mcelog.next);
-
- /* Only supports full reads right now */
- err = -EINVAL;
- if (*off != 0 || usize < MCE_LOG_LEN*sizeof(struct mce))
- goto out;
-
- err = 0;
- prev = 0;
- do {
- for (i = prev; i < next; i++) {
- unsigned long start = jiffies;
- struct mce *m = &mcelog.entry[i];
-
- while (!m->finished) {
- if (time_after_eq(jiffies, start + 2)) {
- memset(m, 0, sizeof(*m));
- goto timeout;
- }
- cpu_relax();
- }
- smp_rmb();
- err |= copy_to_user(buf, m, sizeof(*m));
- buf += sizeof(*m);
-timeout:
- ;
- }
-
- memset(mcelog.entry + prev, 0,
- (next - prev) * sizeof(struct mce));
- prev = next;
- next = cmpxchg(&mcelog.next, prev, 0);
- } while (next != prev);
-
- synchronize_sched();
-
- /*
- * Collect entries that were still getting written before the
- * synchronize.
- */
- on_each_cpu(collect_tscs, cpu_tsc, 1);
-
- for (i = next; i < MCE_LOG_LEN; i++) {
- struct mce *m = &mcelog.entry[i];
-
- if (m->finished && m->tsc < cpu_tsc[m->cpu]) {
- err |= copy_to_user(buf, m, sizeof(*m));
- smp_rmb();
- buf += sizeof(*m);
- memset(m, 0, sizeof(*m));
- }
- }
-
- if (err)
- err = -EFAULT;
-
-out:
- mutex_unlock(&mce_chrdev_read_mutex);
- kfree(cpu_tsc);
-
- return err ? err : buf - ubuf;
-}
-
-static unsigned int mce_chrdev_poll(struct file *file, poll_table *wait)
-{
- poll_wait(file, &mce_chrdev_wait, wait);
- if (READ_ONCE(mcelog.next))
- return POLLIN | POLLRDNORM;
- if (!mce_apei_read_done && apei_check_mce())
- return POLLIN | POLLRDNORM;
- return 0;
-}
-
-static long mce_chrdev_ioctl(struct file *f, unsigned int cmd,
- unsigned long arg)
-{
- int __user *p = (int __user *)arg;
-
- if (!capable(CAP_SYS_ADMIN))
- return -EPERM;
-
- switch (cmd) {
- case MCE_GET_RECORD_LEN:
- return put_user(sizeof(struct mce), p);
- case MCE_GET_LOG_LEN:
- return put_user(MCE_LOG_LEN, p);
- case MCE_GETCLEAR_FLAGS: {
- unsigned flags;
-
- do {
- flags = mcelog.flags;
- } while (cmpxchg(&mcelog.flags, flags, 0) != flags);
-
- return put_user(flags, p);
- }
- default:
- return -ENOTTY;
- }
-}
-
-static ssize_t (*mce_write)(struct file *filp, const char __user *ubuf,
- size_t usize, loff_t *off);
-
-void register_mce_write_callback(ssize_t (*fn)(struct file *filp,
- const char __user *ubuf,
- size_t usize, loff_t *off))
-{
- mce_write = fn;
-}
-EXPORT_SYMBOL_GPL(register_mce_write_callback);
-
-static ssize_t mce_chrdev_write(struct file *filp, const char __user *ubuf,
- size_t usize, loff_t *off)
-{
- if (mce_write)
- return mce_write(filp, ubuf, usize, off);
- else
- return -EINVAL;
-}
-
-static const struct file_operations mce_chrdev_ops = {
- .open = mce_chrdev_open,
- .release = mce_chrdev_release,
- .read = mce_chrdev_read,
- .write = mce_chrdev_write,
- .poll = mce_chrdev_poll,
- .unlocked_ioctl = mce_chrdev_ioctl,
- .llseek = no_llseek,
-};
-
-static struct miscdevice mce_chrdev_device = {
- MISC_MCELOG_MINOR,
- "mcelog",
- &mce_chrdev_ops,
-};
-
static void __mce_disable_bank(void *arg)
{
int bank = *((int *)arg);
@@ -2147,6 +1878,7 @@
int __init mcheck_init(void)
{
mcheck_intel_therm_init();
+ mce_register_decode_chain(&first_nb);
mce_register_decode_chain(&mce_srao_nb);
mce_register_decode_chain(&mce_default_nb);
mcheck_vendor_init_severity();
@@ -2291,29 +2023,6 @@
return size;
}
-static ssize_t
-show_trigger(struct device *s, struct device_attribute *attr, char *buf)
-{
- strcpy(buf, mce_helper);
- strcat(buf, "\n");
- return strlen(mce_helper) + 1;
-}
-
-static ssize_t set_trigger(struct device *s, struct device_attribute *attr,
- const char *buf, size_t siz)
-{
- char *p;
-
- strncpy(mce_helper, buf, sizeof(mce_helper));
- mce_helper[sizeof(mce_helper)-1] = 0;
- p = strchr(mce_helper, '\n');
-
- if (p)
- *p = 0;
-
- return strlen(mce_helper) + !!p;
-}
-
static ssize_t set_ignore_ce(struct device *s,
struct device_attribute *attr,
const char *buf, size_t size)
@@ -2370,7 +2079,6 @@
return ret;
}
-static DEVICE_ATTR(trigger, 0644, show_trigger, set_trigger);
static DEVICE_INT_ATTR(tolerant, 0644, mca_cfg.tolerant);
static DEVICE_INT_ATTR(monarch_timeout, 0644, mca_cfg.monarch_timeout);
static DEVICE_BOOL_ATTR(dont_log_ce, 0644, mca_cfg.dont_log_ce);
@@ -2393,7 +2101,9 @@
static struct device_attribute *mce_device_attrs[] = {
&dev_attr_tolerant.attr,
&dev_attr_check_interval.attr,
+#ifdef CONFIG_X86_MCELOG_LEGACY
&dev_attr_trigger,
+#endif
&dev_attr_monarch_timeout.attr,
&dev_attr_dont_log_ce.attr,
&dev_attr_ignore_ce.attr,
@@ -2567,7 +2277,6 @@
static __init int mcheck_init_device(void)
{
- enum cpuhp_state hp_online;
int err;
if (!mce_available(&boot_cpu_data)) {
@@ -2595,21 +2304,11 @@
mce_cpu_online, mce_cpu_pre_down);
if (err < 0)
goto err_out_online;
- hp_online = err;
register_syscore_ops(&mce_syscore_ops);
- /* register character device /dev/mcelog */
- err = misc_register(&mce_chrdev_device);
- if (err)
- goto err_register;
-
return 0;
-err_register:
- unregister_syscore_ops(&mce_syscore_ops);
- cpuhp_remove_state(hp_online);
-
err_out_online:
cpuhp_remove_state(CPUHP_X86_MCE_DEAD);
@@ -2617,7 +2316,7 @@
free_cpumask_var(mce_device_initialized);
err_out:
- pr_err("Unable to init device /dev/mcelog (rc: %d)\n", err);
+ pr_err("Unable to init MCE device (rc: %d)\n", err);
return err;
}
@@ -2696,6 +2395,7 @@
static_branch_inc(&mcsafe_key);
mcheck_debugfs_init();
+ cec_init();
/*
* Flush out everything that has been logged during early boot, now that
diff --git a/arch/x86/kernel/cpu/mcheck/mce_intel.c b/arch/x86/kernel/cpu/mcheck/mce_intel.c
index 190b3e6..e84db79 100644
--- a/arch/x86/kernel/cpu/mcheck/mce_intel.c
+++ b/arch/x86/kernel/cpu/mcheck/mce_intel.c
@@ -481,6 +481,9 @@
case INTEL_FAM6_BROADWELL_XEON_D:
case INTEL_FAM6_BROADWELL_X:
case INTEL_FAM6_SKYLAKE_X:
+ case INTEL_FAM6_XEON_PHI_KNL:
+ case INTEL_FAM6_XEON_PHI_KNM:
+
if (rdmsrl_safe(MSR_PPIN_CTL, &val))
return;
diff --git a/arch/x86/kernel/cpu/mtrr/cleanup.c b/arch/x86/kernel/cpu/mtrr/cleanup.c
index 3b442b6..765afd5 100644
--- a/arch/x86/kernel/cpu/mtrr/cleanup.c
+++ b/arch/x86/kernel/cpu/mtrr/cleanup.c
@@ -27,7 +27,7 @@
#include <linux/range.h>
#include <asm/processor.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/mtrr.h>
#include <asm/msr.h>
@@ -860,7 +860,7 @@
trim_size <<= PAGE_SHIFT;
trim_size -= trim_start;
- return e820_update_range(trim_start, trim_size, E820_RAM, E820_RESERVED);
+ return e820__range_update(trim_start, trim_size, E820_TYPE_RAM, E820_TYPE_RESERVED);
}
/**
@@ -978,7 +978,7 @@
WARN_ON(1);
pr_info("update e820 for mtrr\n");
- update_e820();
+ e820__update_table_print();
return 1;
}
diff --git a/arch/x86/kernel/cpu/mtrr/main.c b/arch/x86/kernel/cpu/mtrr/main.c
index 24e87e7..2bce84d 100644
--- a/arch/x86/kernel/cpu/mtrr/main.c
+++ b/arch/x86/kernel/cpu/mtrr/main.c
@@ -48,7 +48,7 @@
#include <linux/syscore_ops.h>
#include <asm/cpufeature.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/mtrr.h>
#include <asm/msr.h>
#include <asm/pat.h>
diff --git a/arch/x86/kernel/cpu/proc.c b/arch/x86/kernel/cpu/proc.c
index 18ca99f..6df621a 100644
--- a/arch/x86/kernel/cpu/proc.c
+++ b/arch/x86/kernel/cpu/proc.c
@@ -31,14 +31,13 @@
"fpu\t\t: %s\n"
"fpu_exception\t: %s\n"
"cpuid level\t: %d\n"
- "wp\t\t: %s\n",
+ "wp\t\t: yes\n",
static_cpu_has_bug(X86_BUG_FDIV) ? "yes" : "no",
static_cpu_has_bug(X86_BUG_F00F) ? "yes" : "no",
static_cpu_has_bug(X86_BUG_COMA) ? "yes" : "no",
static_cpu_has(X86_FEATURE_FPU) ? "yes" : "no",
static_cpu_has(X86_FEATURE_FPU) ? "yes" : "no",
- c->cpuid_level,
- c->wp_works_ok ? "yes" : "no");
+ c->cpuid_level);
}
#else
static void show_cpuinfo_misc(struct seq_file *m, struct cpuinfo_x86 *c)
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index d979406..23c2350 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -27,6 +27,7 @@
{ X86_FEATURE_CAT_L3, CPUID_EBX, 1, 0x00000010, 0 },
{ X86_FEATURE_CAT_L2, CPUID_EBX, 2, 0x00000010, 0 },
{ X86_FEATURE_CDP_L3, CPUID_ECX, 2, 0x00000010, 1 },
+ { X86_FEATURE_MBA, CPUID_EBX, 3, 0x00000010, 0 },
{ X86_FEATURE_HW_PSTATE, CPUID_EDX, 7, 0x80000007, 0 },
{ X86_FEATURE_CPB, CPUID_EDX, 9, 0x80000007, 0 },
{ X86_FEATURE_PROC_FEEDBACK, CPUID_EDX, 11, 0x80000007, 0 },
diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 3741461..22217ec 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -29,6 +29,7 @@
#include <asm/nmi.h>
#include <asm/hw_irq.h>
#include <asm/apic.h>
+#include <asm/e820/types.h>
#include <asm/io_apic.h>
#include <asm/hpet.h>
#include <linux/kdebug.h>
@@ -503,16 +504,16 @@
return ret;
}
-static int add_e820_entry(struct boot_params *params, struct e820entry *entry)
+static int add_e820_entry(struct boot_params *params, struct e820_entry *entry)
{
unsigned int nr_e820_entries;
nr_e820_entries = params->e820_entries;
- if (nr_e820_entries >= E820MAX)
+ if (nr_e820_entries >= E820_MAX_ENTRIES_ZEROPAGE)
return 1;
- memcpy(¶ms->e820_map[nr_e820_entries], entry,
- sizeof(struct e820entry));
+ memcpy(¶ms->e820_table[nr_e820_entries], entry,
+ sizeof(struct e820_entry));
params->e820_entries++;
return 0;
}
@@ -521,7 +522,7 @@
{
struct crash_memmap_data *cmd = arg;
struct boot_params *params = cmd->params;
- struct e820entry ei;
+ struct e820_entry ei;
ei.addr = start;
ei.size = end - start + 1;
@@ -560,7 +561,7 @@
{
int i, ret = 0;
unsigned long flags;
- struct e820entry ei;
+ struct e820_entry ei;
struct crash_memmap_data cmd;
struct crash_mem *cmem;
@@ -574,17 +575,17 @@
/* Add first 640K segment */
ei.addr = image->arch.backup_src_start;
ei.size = image->arch.backup_src_sz;
- ei.type = E820_RAM;
+ ei.type = E820_TYPE_RAM;
add_e820_entry(params, &ei);
/* Add ACPI tables */
- cmd.type = E820_ACPI;
+ cmd.type = E820_TYPE_ACPI;
flags = IORESOURCE_MEM | IORESOURCE_BUSY;
walk_iomem_res_desc(IORES_DESC_ACPI_TABLES, flags, 0, -1, &cmd,
memmap_entry_callback);
/* Add ACPI Non-volatile Storage */
- cmd.type = E820_NVS;
+ cmd.type = E820_TYPE_NVS;
walk_iomem_res_desc(IORES_DESC_ACPI_NV_STORAGE, flags, 0, -1, &cmd,
memmap_entry_callback);
@@ -592,7 +593,7 @@
if (crashk_low_res.end) {
ei.addr = crashk_low_res.start;
ei.size = crashk_low_res.end - crashk_low_res.start + 1;
- ei.type = E820_RAM;
+ ei.type = E820_TYPE_RAM;
add_e820_entry(params, &ei);
}
@@ -609,7 +610,7 @@
if (ei.size < PAGE_SIZE)
continue;
ei.addr = cmem->ranges[i].start;
- ei.type = E820_RAM;
+ ei.type = E820_TYPE_RAM;
add_e820_entry(params, &ei);
}
diff --git a/arch/x86/kernel/dumpstack.c b/arch/x86/kernel/dumpstack.c
index 09d4ac0..dbce3cc 100644
--- a/arch/x86/kernel/dumpstack.c
+++ b/arch/x86/kernel/dumpstack.c
@@ -77,7 +77,7 @@
* - softirq stack
* - hardirq stack
*/
- for (regs = NULL; stack; stack = stack_info.next_sp) {
+ for (regs = NULL; stack; stack = PTR_ALIGN(stack_info.next_sp, sizeof(long))) {
const char *stack_name;
/*
@@ -289,9 +289,6 @@
unsigned long flags = oops_begin();
int sig = SIGSEGV;
- if (!user_mode(regs))
- report_bug(regs->ip, regs);
-
if (__die(str, regs, err))
sig = 0;
oops_end(flags, regs, sig);
diff --git a/arch/x86/kernel/dumpstack_32.c b/arch/x86/kernel/dumpstack_32.c
index b0b3a3d..e5f0b40 100644
--- a/arch/x86/kernel/dumpstack_32.c
+++ b/arch/x86/kernel/dumpstack_32.c
@@ -162,15 +162,3 @@
}
pr_cont("\n");
}
-
-int is_valid_bugaddr(unsigned long ip)
-{
- unsigned short ud2;
-
- if (ip < PAGE_OFFSET)
- return 0;
- if (probe_kernel_address((unsigned short *)ip, ud2))
- return 0;
-
- return ud2 == 0x0b0f;
-}
diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c
index a8b117e..3e1471d 100644
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -178,13 +178,3 @@
}
pr_cont("\n");
}
-
-int is_valid_bugaddr(unsigned long ip)
-{
- unsigned short ud2;
-
- if (__copy_from_user(&ud2, (const void __user *) ip, sizeof(ud2)))
- return 0;
-
- return ud2 == 0x0b0f;
-}
diff --git a/arch/x86/kernel/e820.c b/arch/x86/kernel/e820.c
index b2bbad6..d78a586 100644
--- a/arch/x86/kernel/e820.c
+++ b/arch/x86/kernel/e820.c
@@ -1,49 +1,55 @@
/*
- * Handle the memory map.
- * The functions here do the job until bootmem takes over.
+ * Low level x86 E820 memory map handling functions.
*
- * Getting sanitize_e820_map() in sync with i386 version by applying change:
- * - Provisions for empty E820 memory regions (reported by certain BIOSes).
- * Alex Achenbach <xela@slit.de>, December 2002.
- * Venkatesh Pallipadi <venkatesh.pallipadi@intel.com>
+ * The firmware and bootloader passes us the "E820 table", which is the primary
+ * physical memory layout description available about x86 systems.
*
+ * The kernel takes the E820 memory layout and optionally modifies it with
+ * quirks and other tweaks, and feeds that into the generic Linux memory
+ * allocation code routines via a platform independent interface (memblock, etc.).
*/
-#include <linux/kernel.h>
-#include <linux/types.h>
-#include <linux/init.h>
#include <linux/crash_dump.h>
-#include <linux/export.h>
#include <linux/bootmem.h>
-#include <linux/pfn.h>
#include <linux/suspend.h>
#include <linux/acpi.h>
#include <linux/firmware-map.h>
#include <linux/memblock.h>
#include <linux/sort.h>
-#include <asm/e820.h>
-#include <asm/proto.h>
+#include <asm/e820/api.h>
#include <asm/setup.h>
-#include <asm/cpufeature.h>
/*
- * The e820 map is the map that gets modified e.g. with command line parameters
- * and that is also registered with modifications in the kernel resource tree
- * with the iomem_resource as parent.
+ * We organize the E820 table into two main data structures:
*
- * The e820_saved is directly saved after the BIOS-provided memory map is
- * copied. It doesn't get modified afterwards. It's registered for the
- * /sys/firmware/memmap interface.
+ * - 'e820_table_firmware': the original firmware version passed to us by the
+ * bootloader - not modified by the kernel. We use this to:
*
- * That memory map is not modified and is used as base for kexec. The kexec'd
- * kernel should get the same memory map as the firmware provides. Then the
- * user can e.g. boot the original kernel with mem=1G while still booting the
- * next kernel with full memory.
+ * - inform the user about the firmware's notion of memory layout
+ * via /sys/firmware/memmap
+ *
+ * - the hibernation code uses it to generate a kernel-independent MD5
+ * fingerprint of the physical memory layout of a system.
+ *
+ * - kexec, which is a bootloader in disguise, uses the original E820
+ * layout to pass to the kexec-ed kernel. This way the original kernel
+ * can have a restricted E820 map while the kexec()-ed kexec-kernel
+ * can have access to full memory - etc.
+ *
+ * - 'e820_table': this is the main E820 table that is massaged by the
+ * low level x86 platform code, or modified by boot parameters, before
+ * passed on to higher level MM layers.
+ *
+ * Once the E820 map has been converted to the standard Linux memory layout
+ * information its role stops - modifying it has no effect and does not get
+ * re-propagated. So itsmain role is a temporary bootstrap storage of firmware
+ * specific memory layout data during early bootup.
*/
-static struct e820map initial_e820 __initdata;
-static struct e820map initial_e820_saved __initdata;
-struct e820map *e820 __refdata = &initial_e820;
-struct e820map *e820_saved __refdata = &initial_e820_saved;
+static struct e820_table e820_table_init __initdata;
+static struct e820_table e820_table_firmware_init __initdata;
+
+struct e820_table *e820_table __refdata = &e820_table_init;
+struct e820_table *e820_table_firmware __refdata = &e820_table_firmware_init;
/* For PCI or other memory-mapped resources */
unsigned long pci_mem_start = 0xaeedbabe;
@@ -55,51 +61,53 @@
* This function checks if any part of the range <start,end> is mapped
* with type.
*/
-int
-e820_any_mapped(u64 start, u64 end, unsigned type)
+bool e820__mapped_any(u64 start, u64 end, enum e820_type type)
{
int i;
- for (i = 0; i < e820->nr_map; i++) {
- struct e820entry *ei = &e820->map[i];
+ for (i = 0; i < e820_table->nr_entries; i++) {
+ struct e820_entry *entry = &e820_table->entries[i];
- if (type && ei->type != type)
+ if (type && entry->type != type)
continue;
- if (ei->addr >= end || ei->addr + ei->size <= start)
+ if (entry->addr >= end || entry->addr + entry->size <= start)
continue;
return 1;
}
return 0;
}
-EXPORT_SYMBOL_GPL(e820_any_mapped);
+EXPORT_SYMBOL_GPL(e820__mapped_any);
/*
- * This function checks if the entire range <start,end> is mapped with type.
+ * This function checks if the entire <start,end> range is mapped with 'type'.
*
- * Note: this function only works correct if the e820 table is sorted and
- * not-overlapping, which is the case
+ * Note: this function only works correctly once the E820 table is sorted and
+ * not-overlapping (at least for the range specified), which is the case normally.
*/
-int __init e820_all_mapped(u64 start, u64 end, unsigned type)
+bool __init e820__mapped_all(u64 start, u64 end, enum e820_type type)
{
int i;
- for (i = 0; i < e820->nr_map; i++) {
- struct e820entry *ei = &e820->map[i];
+ for (i = 0; i < e820_table->nr_entries; i++) {
+ struct e820_entry *entry = &e820_table->entries[i];
- if (type && ei->type != type)
- continue;
- /* is the region (part) in overlap with the current region ?*/
- if (ei->addr >= end || ei->addr + ei->size <= start)
+ if (type && entry->type != type)
continue;
- /* if the region is at the beginning of <start,end> we move
- * start to the end of the region since it's ok until there
- */
- if (ei->addr <= start)
- start = ei->addr + ei->size;
+ /* Is the region (part) in overlap with the current region? */
+ if (entry->addr >= end || entry->addr + entry->size <= start)
+ continue;
+
/*
- * if start is now at or beyond end, we're done, full
- * coverage
+ * If the region is at the beginning of <start,end> we move
+ * 'start' to the end of the region since it's ok until there
+ */
+ if (entry->addr <= start)
+ start = entry->addr + entry->size;
+
+ /*
+ * If 'start' is now at or beyond 'end', we're done, full
+ * coverage of the desired range exists:
*/
if (start >= end)
return 1;
@@ -108,94 +116,77 @@
}
/*
- * Add a memory region to the kernel e820 map.
+ * Add a memory region to the kernel E820 map.
*/
-static void __init __e820_add_region(struct e820map *e820x, u64 start, u64 size,
- int type)
+static void __init __e820__range_add(struct e820_table *table, u64 start, u64 size, enum e820_type type)
{
- int x = e820x->nr_map;
+ int x = table->nr_entries;
- if (x >= ARRAY_SIZE(e820x->map)) {
- printk(KERN_ERR "e820: too many entries; ignoring [mem %#010llx-%#010llx]\n",
- (unsigned long long) start,
- (unsigned long long) (start + size - 1));
+ if (x >= ARRAY_SIZE(table->entries)) {
+ pr_err("e820: too many entries; ignoring [mem %#010llx-%#010llx]\n", start, start + size - 1);
return;
}
- e820x->map[x].addr = start;
- e820x->map[x].size = size;
- e820x->map[x].type = type;
- e820x->nr_map++;
+ table->entries[x].addr = start;
+ table->entries[x].size = size;
+ table->entries[x].type = type;
+ table->nr_entries++;
}
-void __init e820_add_region(u64 start, u64 size, int type)
+void __init e820__range_add(u64 start, u64 size, enum e820_type type)
{
- __e820_add_region(e820, start, size, type);
+ __e820__range_add(e820_table, start, size, type);
}
-static void __init e820_print_type(u32 type)
+static void __init e820_print_type(enum e820_type type)
{
switch (type) {
- case E820_RAM:
- case E820_RESERVED_KERN:
- printk(KERN_CONT "usable");
- break;
- case E820_RESERVED:
- printk(KERN_CONT "reserved");
- break;
- case E820_ACPI:
- printk(KERN_CONT "ACPI data");
- break;
- case E820_NVS:
- printk(KERN_CONT "ACPI NVS");
- break;
- case E820_UNUSABLE:
- printk(KERN_CONT "unusable");
- break;
- case E820_PMEM:
- case E820_PRAM:
- printk(KERN_CONT "persistent (type %u)", type);
- break;
- default:
- printk(KERN_CONT "type %u", type);
- break;
+ case E820_TYPE_RAM: /* Fall through: */
+ case E820_TYPE_RESERVED_KERN: pr_cont("usable"); break;
+ case E820_TYPE_RESERVED: pr_cont("reserved"); break;
+ case E820_TYPE_ACPI: pr_cont("ACPI data"); break;
+ case E820_TYPE_NVS: pr_cont("ACPI NVS"); break;
+ case E820_TYPE_UNUSABLE: pr_cont("unusable"); break;
+ case E820_TYPE_PMEM: /* Fall through: */
+ case E820_TYPE_PRAM: pr_cont("persistent (type %u)", type); break;
+ default: pr_cont("type %u", type); break;
}
}
-void __init e820_print_map(char *who)
+void __init e820__print_table(char *who)
{
int i;
- for (i = 0; i < e820->nr_map; i++) {
- printk(KERN_INFO "%s: [mem %#018Lx-%#018Lx] ", who,
- (unsigned long long) e820->map[i].addr,
- (unsigned long long)
- (e820->map[i].addr + e820->map[i].size - 1));
- e820_print_type(e820->map[i].type);
- printk(KERN_CONT "\n");
+ for (i = 0; i < e820_table->nr_entries; i++) {
+ pr_info("%s: [mem %#018Lx-%#018Lx] ", who,
+ e820_table->entries[i].addr,
+ e820_table->entries[i].addr + e820_table->entries[i].size - 1);
+
+ e820_print_type(e820_table->entries[i].type);
+ pr_cont("\n");
}
}
/*
- * Sanitize the BIOS e820 map.
+ * Sanitize an E820 map.
*
- * Some e820 responses include overlapping entries. The following
- * replaces the original e820 map with a new one, removing overlaps,
+ * Some E820 layouts include overlapping entries. The following
+ * replaces the original E820 map with a new one, removing overlaps,
* and resolving conflicting memory types in favor of highest
* numbered type.
*
- * The input parameter biosmap points to an array of 'struct
- * e820entry' which on entry has elements in the range [0, *pnr_map)
- * valid, and which has space for up to max_nr_map entries.
- * On return, the resulting sanitized e820 map entries will be in
- * overwritten in the same location, starting at biosmap.
+ * The input parameter 'entries' points to an array of 'struct
+ * e820_entry' which on entry has elements in the range [0, *nr_entries)
+ * valid, and which has space for up to max_nr_entries entries.
+ * On return, the resulting sanitized E820 map entries will be in
+ * overwritten in the same location, starting at 'entries'.
*
- * The integer pointed to by pnr_map must be valid on entry (the
- * current number of valid entries located at biosmap). If the
- * sanitizing succeeds the *pnr_map will be updated with the new
- * number of valid entries (something no more than max_nr_map).
+ * The integer pointed to by nr_entries must be valid on entry (the
+ * current number of valid entries located at 'entries'). If the
+ * sanitizing succeeds the *nr_entries will be updated with the new
+ * number of valid entries (something no more than max_nr_entries).
*
- * The return value from sanitize_e820_map() is zero if it
+ * The return value from e820__update_table() is zero if it
* successfully 'sanitized' the map entries passed in, and is -1
* if it did nothing, which can happen if either of (1) it was
* only passed one map entry, or (2) any of the input map entries
@@ -238,10 +229,17 @@
* ______________________4_
*/
struct change_member {
- struct e820entry *pbios; /* pointer to original bios entry */
- unsigned long long addr; /* address for this change point */
+ /* Pointer to the original entry: */
+ struct e820_entry *entry;
+ /* Address for this change point: */
+ unsigned long long addr;
};
+static struct change_member change_point_list[2*E820_MAX_ENTRIES] __initdata;
+static struct change_member *change_point[2*E820_MAX_ENTRIES] __initdata;
+static struct e820_entry *overlap_list[E820_MAX_ENTRIES] __initdata;
+static struct e820_entry new_entries[E820_MAX_ENTRIES] __initdata;
+
static int __init cpcompare(const void *a, const void *b)
{
struct change_member * const *app = a, * const *bpp = b;
@@ -249,164 +247,140 @@
/*
* Inputs are pointers to two elements of change_point[]. If their
- * addresses are unequal, their difference dominates. If the addresses
+ * addresses are not equal, their difference dominates. If the addresses
* are equal, then consider one that represents the end of its region
* to be greater than one that does not.
*/
if (ap->addr != bp->addr)
return ap->addr > bp->addr ? 1 : -1;
- return (ap->addr != ap->pbios->addr) - (bp->addr != bp->pbios->addr);
+ return (ap->addr != ap->entry->addr) - (bp->addr != bp->entry->addr);
}
-int __init sanitize_e820_map(struct e820entry *biosmap, int max_nr_map,
- u32 *pnr_map)
+int __init e820__update_table(struct e820_table *table)
{
- static struct change_member change_point_list[2*E820_X_MAX] __initdata;
- static struct change_member *change_point[2*E820_X_MAX] __initdata;
- static struct e820entry *overlap_list[E820_X_MAX] __initdata;
- static struct e820entry new_bios[E820_X_MAX] __initdata;
- unsigned long current_type, last_type;
+ struct e820_entry *entries = table->entries;
+ u32 max_nr_entries = ARRAY_SIZE(table->entries);
+ enum e820_type current_type, last_type;
unsigned long long last_addr;
- int chgidx;
- int overlap_entries;
- int new_bios_entry;
- int old_nr, new_nr, chg_nr;
- int i;
+ u32 new_nr_entries, overlap_entries;
+ u32 i, chg_idx, chg_nr;
- /* if there's only one memory region, don't bother */
- if (*pnr_map < 2)
+ /* If there's only one memory region, don't bother: */
+ if (table->nr_entries < 2)
return -1;
- old_nr = *pnr_map;
- BUG_ON(old_nr > max_nr_map);
+ BUG_ON(table->nr_entries > max_nr_entries);
- /* bail out if we find any unreasonable addresses in bios map */
- for (i = 0; i < old_nr; i++)
- if (biosmap[i].addr + biosmap[i].size < biosmap[i].addr)
+ /* Bail out if we find any unreasonable addresses in the map: */
+ for (i = 0; i < table->nr_entries; i++) {
+ if (entries[i].addr + entries[i].size < entries[i].addr)
return -1;
+ }
- /* create pointers for initial change-point information (for sorting) */
- for (i = 0; i < 2 * old_nr; i++)
+ /* Create pointers for initial change-point information (for sorting): */
+ for (i = 0; i < 2 * table->nr_entries; i++)
change_point[i] = &change_point_list[i];
- /* record all known change-points (starting and ending addresses),
- omitting those that are for empty memory regions */
- chgidx = 0;
- for (i = 0; i < old_nr; i++) {
- if (biosmap[i].size != 0) {
- change_point[chgidx]->addr = biosmap[i].addr;
- change_point[chgidx++]->pbios = &biosmap[i];
- change_point[chgidx]->addr = biosmap[i].addr +
- biosmap[i].size;
- change_point[chgidx++]->pbios = &biosmap[i];
+ /*
+ * Record all known change-points (starting and ending addresses),
+ * omitting empty memory regions:
+ */
+ chg_idx = 0;
+ for (i = 0; i < table->nr_entries; i++) {
+ if (entries[i].size != 0) {
+ change_point[chg_idx]->addr = entries[i].addr;
+ change_point[chg_idx++]->entry = &entries[i];
+ change_point[chg_idx]->addr = entries[i].addr + entries[i].size;
+ change_point[chg_idx++]->entry = &entries[i];
}
}
- chg_nr = chgidx;
+ chg_nr = chg_idx;
- /* sort change-point list by memory addresses (low -> high) */
- sort(change_point, chg_nr, sizeof *change_point, cpcompare, NULL);
+ /* Sort change-point list by memory addresses (low -> high): */
+ sort(change_point, chg_nr, sizeof(*change_point), cpcompare, NULL);
- /* create a new bios memory map, removing overlaps */
- overlap_entries = 0; /* number of entries in the overlap table */
- new_bios_entry = 0; /* index for creating new bios map entries */
- last_type = 0; /* start with undefined memory type */
- last_addr = 0; /* start with 0 as last starting address */
+ /* Create a new memory map, removing overlaps: */
+ overlap_entries = 0; /* Number of entries in the overlap table */
+ new_nr_entries = 0; /* Index for creating new map entries */
+ last_type = 0; /* Start with undefined memory type */
+ last_addr = 0; /* Start with 0 as last starting address */
- /* loop through change-points, determining affect on the new bios map */
- for (chgidx = 0; chgidx < chg_nr; chgidx++) {
- /* keep track of all overlapping bios entries */
- if (change_point[chgidx]->addr ==
- change_point[chgidx]->pbios->addr) {
- /*
- * add map entry to overlap list (> 1 entry
- * implies an overlap)
- */
- overlap_list[overlap_entries++] =
- change_point[chgidx]->pbios;
+ /* Loop through change-points, determining effect on the new map: */
+ for (chg_idx = 0; chg_idx < chg_nr; chg_idx++) {
+ /* Keep track of all overlapping entries */
+ if (change_point[chg_idx]->addr == change_point[chg_idx]->entry->addr) {
+ /* Add map entry to overlap list (> 1 entry implies an overlap) */
+ overlap_list[overlap_entries++] = change_point[chg_idx]->entry;
} else {
- /*
- * remove entry from list (order independent,
- * so swap with last)
- */
+ /* Remove entry from list (order independent, so swap with last): */
for (i = 0; i < overlap_entries; i++) {
- if (overlap_list[i] ==
- change_point[chgidx]->pbios)
- overlap_list[i] =
- overlap_list[overlap_entries-1];
+ if (overlap_list[i] == change_point[chg_idx]->entry)
+ overlap_list[i] = overlap_list[overlap_entries-1];
}
overlap_entries--;
}
/*
- * if there are overlapping entries, decide which
+ * If there are overlapping entries, decide which
* "type" to use (larger value takes precedence --
* 1=usable, 2,3,4,4+=unusable)
*/
current_type = 0;
- for (i = 0; i < overlap_entries; i++)
+ for (i = 0; i < overlap_entries; i++) {
if (overlap_list[i]->type > current_type)
current_type = overlap_list[i]->type;
- /*
- * continue building up new bios map based on this
- * information
- */
- if (current_type != last_type || current_type == E820_PRAM) {
+ }
+
+ /* Continue building up new map based on this information: */
+ if (current_type != last_type || current_type == E820_TYPE_PRAM) {
if (last_type != 0) {
- new_bios[new_bios_entry].size =
- change_point[chgidx]->addr - last_addr;
- /*
- * move forward only if the new size
- * was non-zero
- */
- if (new_bios[new_bios_entry].size != 0)
- /*
- * no more space left for new
- * bios entries ?
- */
- if (++new_bios_entry >= max_nr_map)
+ new_entries[new_nr_entries].size = change_point[chg_idx]->addr - last_addr;
+ /* Move forward only if the new size was non-zero: */
+ if (new_entries[new_nr_entries].size != 0)
+ /* No more space left for new entries? */
+ if (++new_nr_entries >= max_nr_entries)
break;
}
if (current_type != 0) {
- new_bios[new_bios_entry].addr =
- change_point[chgidx]->addr;
- new_bios[new_bios_entry].type = current_type;
- last_addr = change_point[chgidx]->addr;
+ new_entries[new_nr_entries].addr = change_point[chg_idx]->addr;
+ new_entries[new_nr_entries].type = current_type;
+ last_addr = change_point[chg_idx]->addr;
}
last_type = current_type;
}
}
- /* retain count for new bios entries */
- new_nr = new_bios_entry;
- /* copy new bios mapping into original location */
- memcpy(biosmap, new_bios, new_nr * sizeof(struct e820entry));
- *pnr_map = new_nr;
+ /* Copy the new entries into the original location: */
+ memcpy(entries, new_entries, new_nr_entries*sizeof(*entries));
+ table->nr_entries = new_nr_entries;
return 0;
}
-static int __init __append_e820_map(struct e820entry *biosmap, int nr_map)
+static int __init __append_e820_table(struct boot_e820_entry *entries, u32 nr_entries)
{
- while (nr_map) {
- u64 start = biosmap->addr;
- u64 size = biosmap->size;
- u64 end = start + size - 1;
- u32 type = biosmap->type;
+ struct boot_e820_entry *entry = entries;
- /* Overflow in 64 bits? Ignore the memory map. */
+ while (nr_entries) {
+ u64 start = entry->addr;
+ u64 size = entry->size;
+ u64 end = start + size - 1;
+ u32 type = entry->type;
+
+ /* Ignore the entry on 64-bit overflow: */
if (start > end && likely(size))
return -1;
- e820_add_region(start, size, type);
+ e820__range_add(start, size, type);
- biosmap++;
- nr_map--;
+ entry++;
+ nr_entries--;
}
return 0;
}
/*
- * Copy the BIOS e820 map into a safe place.
+ * Copy the BIOS E820 map into a safe place.
*
* Sanity-check it while we're at it..
*
@@ -414,18 +388,17 @@
* will have given us a memory map that we can use to properly
* set up memory. If we aren't, we'll fake a memory map.
*/
-static int __init append_e820_map(struct e820entry *biosmap, int nr_map)
+static int __init append_e820_table(struct boot_e820_entry *entries, u32 nr_entries)
{
/* Only one memory region (or negative)? Ignore it */
- if (nr_map < 2)
+ if (nr_entries < 2)
return -1;
- return __append_e820_map(biosmap, nr_map);
+ return __append_e820_table(entries, nr_entries);
}
-static u64 __init __e820_update_range(struct e820map *e820x, u64 start,
- u64 size, unsigned old_type,
- unsigned new_type)
+static u64 __init
+__e820__range_update(struct e820_table *table, u64 start, u64 size, enum e820_type old_type, enum e820_type new_type)
{
u64 end;
unsigned int i;
@@ -437,77 +410,73 @@
size = ULLONG_MAX - start;
end = start + size;
- printk(KERN_DEBUG "e820: update [mem %#010Lx-%#010Lx] ",
- (unsigned long long) start, (unsigned long long) (end - 1));
+ printk(KERN_DEBUG "e820: update [mem %#010Lx-%#010Lx] ", start, end - 1);
e820_print_type(old_type);
- printk(KERN_CONT " ==> ");
+ pr_cont(" ==> ");
e820_print_type(new_type);
- printk(KERN_CONT "\n");
+ pr_cont("\n");
- for (i = 0; i < e820x->nr_map; i++) {
- struct e820entry *ei = &e820x->map[i];
+ for (i = 0; i < table->nr_entries; i++) {
+ struct e820_entry *entry = &table->entries[i];
u64 final_start, final_end;
- u64 ei_end;
+ u64 entry_end;
- if (ei->type != old_type)
+ if (entry->type != old_type)
continue;
- ei_end = ei->addr + ei->size;
- /* totally covered by new range? */
- if (ei->addr >= start && ei_end <= end) {
- ei->type = new_type;
- real_updated_size += ei->size;
+ entry_end = entry->addr + entry->size;
+
+ /* Completely covered by new range? */
+ if (entry->addr >= start && entry_end <= end) {
+ entry->type = new_type;
+ real_updated_size += entry->size;
continue;
}
- /* new range is totally covered? */
- if (ei->addr < start && ei_end > end) {
- __e820_add_region(e820x, start, size, new_type);
- __e820_add_region(e820x, end, ei_end - end, ei->type);
- ei->size = start - ei->addr;
+ /* New range is completely covered? */
+ if (entry->addr < start && entry_end > end) {
+ __e820__range_add(table, start, size, new_type);
+ __e820__range_add(table, end, entry_end - end, entry->type);
+ entry->size = start - entry->addr;
real_updated_size += size;
continue;
}
- /* partially covered */
- final_start = max(start, ei->addr);
- final_end = min(end, ei_end);
+ /* Partially covered: */
+ final_start = max(start, entry->addr);
+ final_end = min(end, entry_end);
if (final_start >= final_end)
continue;
- __e820_add_region(e820x, final_start, final_end - final_start,
- new_type);
+ __e820__range_add(table, final_start, final_end - final_start, new_type);
real_updated_size += final_end - final_start;
/*
- * left range could be head or tail, so need to update
- * size at first.
+ * Left range could be head or tail, so need to update
+ * its size first:
*/
- ei->size -= final_end - final_start;
- if (ei->addr < final_start)
+ entry->size -= final_end - final_start;
+ if (entry->addr < final_start)
continue;
- ei->addr = final_end;
+
+ entry->addr = final_end;
}
return real_updated_size;
}
-u64 __init e820_update_range(u64 start, u64 size, unsigned old_type,
- unsigned new_type)
+u64 __init e820__range_update(u64 start, u64 size, enum e820_type old_type, enum e820_type new_type)
{
- return __e820_update_range(e820, start, size, old_type, new_type);
+ return __e820__range_update(e820_table, start, size, old_type, new_type);
}
-static u64 __init e820_update_range_saved(u64 start, u64 size,
- unsigned old_type, unsigned new_type)
+static u64 __init e820__range_update_firmware(u64 start, u64 size, enum e820_type old_type, enum e820_type new_type)
{
- return __e820_update_range(e820_saved, start, size, old_type,
- new_type);
+ return __e820__range_update(e820_table_firmware, start, size, old_type, new_type);
}
-/* make e820 not cover the range */
-u64 __init e820_remove_range(u64 start, u64 size, unsigned old_type,
- int checktype)
+/* Remove a range of memory from the E820 table: */
+u64 __init e820__range_remove(u64 start, u64 size, enum e820_type old_type, bool check_type)
{
int i;
u64 end;
@@ -517,85 +486,89 @@
size = ULLONG_MAX - start;
end = start + size;
- printk(KERN_DEBUG "e820: remove [mem %#010Lx-%#010Lx] ",
- (unsigned long long) start, (unsigned long long) (end - 1));
- if (checktype)
+ printk(KERN_DEBUG "e820: remove [mem %#010Lx-%#010Lx] ", start, end - 1);
+ if (check_type)
e820_print_type(old_type);
- printk(KERN_CONT "\n");
+ pr_cont("\n");
- for (i = 0; i < e820->nr_map; i++) {
- struct e820entry *ei = &e820->map[i];
+ for (i = 0; i < e820_table->nr_entries; i++) {
+ struct e820_entry *entry = &e820_table->entries[i];
u64 final_start, final_end;
- u64 ei_end;
+ u64 entry_end;
- if (checktype && ei->type != old_type)
+ if (check_type && entry->type != old_type)
continue;
- ei_end = ei->addr + ei->size;
- /* totally covered? */
- if (ei->addr >= start && ei_end <= end) {
- real_removed_size += ei->size;
- memset(ei, 0, sizeof(struct e820entry));
+ entry_end = entry->addr + entry->size;
+
+ /* Completely covered? */
+ if (entry->addr >= start && entry_end <= end) {
+ real_removed_size += entry->size;
+ memset(entry, 0, sizeof(*entry));
continue;
}
- /* new range is totally covered? */
- if (ei->addr < start && ei_end > end) {
- e820_add_region(end, ei_end - end, ei->type);
- ei->size = start - ei->addr;
+ /* Is the new range completely covered? */
+ if (entry->addr < start && entry_end > end) {
+ e820__range_add(end, entry_end - end, entry->type);
+ entry->size = start - entry->addr;
real_removed_size += size;
continue;
}
- /* partially covered */
- final_start = max(start, ei->addr);
- final_end = min(end, ei_end);
+ /* Partially covered: */
+ final_start = max(start, entry->addr);
+ final_end = min(end, entry_end);
if (final_start >= final_end)
continue;
+
real_removed_size += final_end - final_start;
/*
- * left range could be head or tail, so need to update
- * size at first.
+ * Left range could be head or tail, so need to update
+ * the size first:
*/
- ei->size -= final_end - final_start;
- if (ei->addr < final_start)
+ entry->size -= final_end - final_start;
+ if (entry->addr < final_start)
continue;
- ei->addr = final_end;
+
+ entry->addr = final_end;
}
return real_removed_size;
}
-void __init update_e820(void)
+void __init e820__update_table_print(void)
{
- if (sanitize_e820_map(e820->map, ARRAY_SIZE(e820->map), &e820->nr_map))
+ if (e820__update_table(e820_table))
return;
- printk(KERN_INFO "e820: modified physical RAM map:\n");
- e820_print_map("modified");
+
+ pr_info("e820: modified physical RAM map:\n");
+ e820__print_table("modified");
}
-static void __init update_e820_saved(void)
+
+static void __init e820__update_table_firmware(void)
{
- sanitize_e820_map(e820_saved->map, ARRAY_SIZE(e820_saved->map),
- &e820_saved->nr_map);
+ e820__update_table(e820_table_firmware);
}
+
#define MAX_GAP_END 0x100000000ull
+
/*
- * Search for a gap in the e820 memory space from 0 to MAX_GAP_END.
+ * Search for a gap in the E820 memory space from 0 to MAX_GAP_END (4GB).
*/
-static int __init e820_search_gap(unsigned long *gapstart,
- unsigned long *gapsize)
+static int __init e820_search_gap(unsigned long *gapstart, unsigned long *gapsize)
{
unsigned long long last = MAX_GAP_END;
- int i = e820->nr_map;
+ int i = e820_table->nr_entries;
int found = 0;
while (--i >= 0) {
- unsigned long long start = e820->map[i].addr;
- unsigned long long end = start + e820->map[i].size;
+ unsigned long long start = e820_table->entries[i].addr;
+ unsigned long long end = start + e820_table->entries[i].size;
/*
* Since "last" is at most 4GB, we know we'll
- * fit in 32 bits if this condition is true
+ * fit in 32 bits if this condition is true:
*/
if (last > end) {
unsigned long gap = last - end;
@@ -613,12 +586,14 @@
}
/*
- * Search for the biggest gap in the low 32 bits of the e820
- * memory space. We pass this space to PCI to assign MMIO resources
- * for hotplug or unconfigured devices in.
+ * Search for the biggest gap in the low 32 bits of the E820
+ * memory space. We pass this space to the PCI subsystem, so
+ * that it can assign MMIO resources for hotplug or
+ * unconfigured devices in.
+ *
* Hopefully the BIOS let enough space left.
*/
-__init void e820_setup_gap(void)
+__init void e820__setup_pci_gap(void)
{
unsigned long gapstart, gapsize;
int found;
@@ -629,138 +604,143 @@
if (!found) {
#ifdef CONFIG_X86_64
gapstart = (max_pfn << PAGE_SHIFT) + 1024*1024;
- printk(KERN_ERR
- "e820: cannot find a gap in the 32bit address range\n"
- "e820: PCI devices with unassigned 32bit BARs may break!\n");
+ pr_err(
+ "e820: Cannot find an available gap in the 32-bit address range\n"
+ "e820: PCI devices with unassigned 32-bit BARs may not work!\n");
#else
gapstart = 0x10000000;
#endif
}
/*
- * e820_reserve_resources_late protect stolen RAM already
+ * e820__reserve_resources_late() protects stolen RAM already:
*/
pci_mem_start = gapstart;
- printk(KERN_INFO
- "e820: [mem %#010lx-%#010lx] available for PCI devices\n",
- gapstart, gapstart + gapsize - 1);
+ pr_info("e820: [mem %#010lx-%#010lx] available for PCI devices\n", gapstart, gapstart + gapsize - 1);
}
/*
* Called late during init, in free_initmem().
*
- * Initial e820 and e820_saved are largish __initdata arrays.
- * Copy them to (usually much smaller) dynamically allocated area.
- * This is done after all tweaks we ever do to them:
- * all functions which modify them are __init functions,
- * they won't exist after this point.
+ * Initial e820_table and e820_table_firmware are largish __initdata arrays.
+ *
+ * Copy them to a (usually much smaller) dynamically allocated area that is
+ * sized precisely after the number of e820 entries.
+ *
+ * This is done after we've performed all the fixes and tweaks to the tables.
+ * All functions which modify them are __init functions, which won't exist
+ * after free_initmem().
*/
-__init void e820_reallocate_tables(void)
+__init void e820__reallocate_tables(void)
{
- struct e820map *n;
+ struct e820_table *n;
int size;
- size = offsetof(struct e820map, map) + sizeof(struct e820entry) * e820->nr_map;
+ size = offsetof(struct e820_table, entries) + sizeof(struct e820_entry)*e820_table->nr_entries;
n = kmalloc(size, GFP_KERNEL);
BUG_ON(!n);
- memcpy(n, e820, size);
- e820 = n;
+ memcpy(n, e820_table, size);
+ e820_table = n;
- size = offsetof(struct e820map, map) + sizeof(struct e820entry) * e820_saved->nr_map;
+ size = offsetof(struct e820_table, entries) + sizeof(struct e820_entry)*e820_table_firmware->nr_entries;
n = kmalloc(size, GFP_KERNEL);
BUG_ON(!n);
- memcpy(n, e820_saved, size);
- e820_saved = n;
+ memcpy(n, e820_table_firmware, size);
+ e820_table_firmware = n;
}
-/**
- * Because of the size limitation of struct boot_params, only first
- * 128 E820 memory entries are passed to kernel via
- * boot_params.e820_map, others are passed via SETUP_E820_EXT node of
- * linked list of struct setup_data, which is parsed here.
+/*
+ * Because of the small fixed size of struct boot_params, only the first
+ * 128 E820 memory entries are passed to the kernel via boot_params.e820_table,
+ * the remaining (if any) entries are passed via the SETUP_E820_EXT node of
+ * struct setup_data, which is parsed here.
*/
-void __init parse_e820_ext(u64 phys_addr, u32 data_len)
+void __init e820__memory_setup_extended(u64 phys_addr, u32 data_len)
{
int entries;
- struct e820entry *extmap;
+ struct boot_e820_entry *extmap;
struct setup_data *sdata;
sdata = early_memremap(phys_addr, data_len);
- entries = sdata->len / sizeof(struct e820entry);
- extmap = (struct e820entry *)(sdata->data);
- __append_e820_map(extmap, entries);
- sanitize_e820_map(e820->map, ARRAY_SIZE(e820->map), &e820->nr_map);
+ entries = sdata->len / sizeof(*extmap);
+ extmap = (struct boot_e820_entry *)(sdata->data);
+
+ __append_e820_table(extmap, entries);
+ e820__update_table(e820_table);
+
early_memunmap(sdata, data_len);
- printk(KERN_INFO "e820: extended physical RAM map:\n");
- e820_print_map("extended");
+ pr_info("e820: extended physical RAM map:\n");
+ e820__print_table("extended");
}
-#if defined(CONFIG_X86_64) || \
- (defined(CONFIG_X86_32) && defined(CONFIG_HIBERNATION))
-/**
+/*
* Find the ranges of physical addresses that do not correspond to
- * e820 RAM areas and mark the corresponding pages as nosave for
- * hibernation (32 bit) or software suspend and suspend to RAM (64 bit).
+ * E820 RAM areas and register the corresponding pages as 'nosave' for
+ * hibernation (32-bit) or software suspend and suspend to RAM (64-bit).
*
- * This function requires the e820 map to be sorted and without any
+ * This function requires the E820 map to be sorted and without any
* overlapping entries.
*/
-void __init e820_mark_nosave_regions(unsigned long limit_pfn)
+void __init e820__register_nosave_regions(unsigned long limit_pfn)
{
int i;
unsigned long pfn = 0;
- for (i = 0; i < e820->nr_map; i++) {
- struct e820entry *ei = &e820->map[i];
+ for (i = 0; i < e820_table->nr_entries; i++) {
+ struct e820_entry *entry = &e820_table->entries[i];
- if (pfn < PFN_UP(ei->addr))
- register_nosave_region(pfn, PFN_UP(ei->addr));
+ if (pfn < PFN_UP(entry->addr))
+ register_nosave_region(pfn, PFN_UP(entry->addr));
- pfn = PFN_DOWN(ei->addr + ei->size);
+ pfn = PFN_DOWN(entry->addr + entry->size);
- if (ei->type != E820_RAM && ei->type != E820_RESERVED_KERN)
- register_nosave_region(PFN_UP(ei->addr), pfn);
+ if (entry->type != E820_TYPE_RAM && entry->type != E820_TYPE_RESERVED_KERN)
+ register_nosave_region(PFN_UP(entry->addr), pfn);
if (pfn >= limit_pfn)
break;
}
}
-#endif
#ifdef CONFIG_ACPI
-/**
- * Mark ACPI NVS memory region, so that we can save/restore it during
- * hibernation and the subsequent resume.
+/*
+ * Register ACPI NVS memory regions, so that we can save/restore them during
+ * hibernation and the subsequent resume:
*/
-static int __init e820_mark_nvs_memory(void)
+static int __init e820__register_nvs_regions(void)
{
int i;
- for (i = 0; i < e820->nr_map; i++) {
- struct e820entry *ei = &e820->map[i];
+ for (i = 0; i < e820_table->nr_entries; i++) {
+ struct e820_entry *entry = &e820_table->entries[i];
- if (ei->type == E820_NVS)
- acpi_nvs_register(ei->addr, ei->size);
+ if (entry->type == E820_TYPE_NVS)
+ acpi_nvs_register(entry->addr, entry->size);
}
return 0;
}
-core_initcall(e820_mark_nvs_memory);
+core_initcall(e820__register_nvs_regions);
#endif
/*
- * pre allocated 4k and reserved it in memblock and e820_saved
+ * Allocate the requested number of bytes with the requsted alignment
+ * and return (the physical address) to the caller. Also register this
+ * range in the 'firmware' E820 table as a reserved range.
+ *
+ * This allows kexec to fake a new mptable, as if it came from the real
+ * system.
*/
-u64 __init early_reserve_e820(u64 size, u64 align)
+u64 __init e820__memblock_alloc_reserved(u64 size, u64 align)
{
u64 addr;
addr = __memblock_alloc_base(size, align, MEMBLOCK_ALLOC_ACCESSIBLE);
if (addr) {
- e820_update_range_saved(addr, size, E820_RAM, E820_RESERVED);
- printk(KERN_INFO "e820: update e820_saved for early_reserve_e820\n");
- update_e820_saved();
+ e820__range_update_firmware(addr, size, E820_TYPE_RAM, E820_TYPE_RESERVED);
+ pr_info("e820: update e820_table_firmware for e820__memblock_alloc_reserved()\n");
+ e820__update_table_firmware();
}
return addr;
@@ -779,22 +759,22 @@
/*
* Find the highest page frame number we have available
*/
-static unsigned long __init e820_end_pfn(unsigned long limit_pfn, unsigned type)
+static unsigned long __init e820_end_pfn(unsigned long limit_pfn, enum e820_type type)
{
int i;
unsigned long last_pfn = 0;
unsigned long max_arch_pfn = MAX_ARCH_PFN;
- for (i = 0; i < e820->nr_map; i++) {
- struct e820entry *ei = &e820->map[i];
+ for (i = 0; i < e820_table->nr_entries; i++) {
+ struct e820_entry *entry = &e820_table->entries[i];
unsigned long start_pfn;
unsigned long end_pfn;
- if (ei->type != type)
+ if (entry->type != type)
continue;
- start_pfn = ei->addr >> PAGE_SHIFT;
- end_pfn = (ei->addr + ei->size) >> PAGE_SHIFT;
+ start_pfn = entry->addr >> PAGE_SHIFT;
+ end_pfn = (entry->addr + entry->size) >> PAGE_SHIFT;
if (start_pfn >= limit_pfn)
continue;
@@ -809,18 +789,19 @@
if (last_pfn > max_arch_pfn)
last_pfn = max_arch_pfn;
- printk(KERN_INFO "e820: last_pfn = %#lx max_arch_pfn = %#lx\n",
+ pr_info("e820: last_pfn = %#lx max_arch_pfn = %#lx\n",
last_pfn, max_arch_pfn);
return last_pfn;
}
-unsigned long __init e820_end_of_ram_pfn(void)
+
+unsigned long __init e820__end_of_ram_pfn(void)
{
- return e820_end_pfn(MAX_ARCH_PFN, E820_RAM);
+ return e820_end_pfn(MAX_ARCH_PFN, E820_TYPE_RAM);
}
-unsigned long __init e820_end_of_low_ram_pfn(void)
+unsigned long __init e820__end_of_low_ram_pfn(void)
{
- return e820_end_pfn(1UL << (32 - PAGE_SHIFT), E820_RAM);
+ return e820_end_pfn(1UL << (32 - PAGE_SHIFT), E820_TYPE_RAM);
}
static void __init early_panic(char *msg)
@@ -831,7 +812,7 @@
static int userdef __initdata;
-/* "mem=nopentium" disables the 4MB page tables. */
+/* The "mem=nopentium" boot option disables 4MB page tables on 32-bit kernels: */
static int __init parse_memopt(char *p)
{
u64 mem_size;
@@ -844,17 +825,19 @@
setup_clear_cpu_cap(X86_FEATURE_PSE);
return 0;
#else
- printk(KERN_WARNING "mem=nopentium ignored! (only supported on x86_32)\n");
+ pr_warn("mem=nopentium ignored! (only supported on x86_32)\n");
return -EINVAL;
#endif
}
userdef = 1;
mem_size = memparse(p, &p);
- /* don't remove all of memory when handling "mem={invalid}" param */
+
+ /* Don't remove all memory when getting "mem={invalid}" parameter: */
if (mem_size == 0)
return -EINVAL;
- e820_remove_range(mem_size, ULLONG_MAX - mem_size, E820_RAM, 1);
+
+ e820__range_remove(mem_size, ULLONG_MAX - mem_size, E820_TYPE_RAM, 1);
return 0;
}
@@ -872,12 +855,12 @@
#ifdef CONFIG_CRASH_DUMP
/*
* If we are doing a crash dump, we still need to know
- * the real mem size before original memory map is
+ * the real memory size before the original memory map is
* reset.
*/
- saved_max_pfn = e820_end_of_ram_pfn();
+ saved_max_pfn = e820__end_of_ram_pfn();
#endif
- e820->nr_map = 0;
+ e820_table->nr_entries = 0;
userdef = 1;
return 0;
}
@@ -890,21 +873,23 @@
userdef = 1;
if (*p == '@') {
start_at = memparse(p+1, &p);
- e820_add_region(start_at, mem_size, E820_RAM);
+ e820__range_add(start_at, mem_size, E820_TYPE_RAM);
} else if (*p == '#') {
start_at = memparse(p+1, &p);
- e820_add_region(start_at, mem_size, E820_ACPI);
+ e820__range_add(start_at, mem_size, E820_TYPE_ACPI);
} else if (*p == '$') {
start_at = memparse(p+1, &p);
- e820_add_region(start_at, mem_size, E820_RESERVED);
+ e820__range_add(start_at, mem_size, E820_TYPE_RESERVED);
} else if (*p == '!') {
start_at = memparse(p+1, &p);
- e820_add_region(start_at, mem_size, E820_PRAM);
- } else
- e820_remove_range(mem_size, ULLONG_MAX - mem_size, E820_RAM, 1);
+ e820__range_add(start_at, mem_size, E820_TYPE_PRAM);
+ } else {
+ e820__range_remove(mem_size, ULLONG_MAX - mem_size, E820_TYPE_RAM, 1);
+ }
return *p == '\0' ? 0 : -EINVAL;
}
+
static int __init parse_memmap_opt(char *str)
{
while (str) {
@@ -921,68 +906,97 @@
}
early_param("memmap", parse_memmap_opt);
-void __init finish_e820_parsing(void)
+/*
+ * Reserve all entries from the bootloader's extensible data nodes list,
+ * because if present we are going to use it later on to fetch e820
+ * entries from it:
+ */
+void __init e820__reserve_setup_data(void)
+{
+ struct setup_data *data;
+ u64 pa_data;
+
+ pa_data = boot_params.hdr.setup_data;
+ if (!pa_data)
+ return;
+
+ while (pa_data) {
+ data = early_memremap(pa_data, sizeof(*data));
+ e820__range_update(pa_data, sizeof(*data)+data->len, E820_TYPE_RAM, E820_TYPE_RESERVED_KERN);
+ pa_data = data->next;
+ early_memunmap(data, sizeof(*data));
+ }
+
+ e820__update_table(e820_table);
+
+ memcpy(e820_table_firmware, e820_table, sizeof(*e820_table_firmware));
+
+ pr_info("extended physical RAM map:\n");
+ e820__print_table("reserve setup_data");
+}
+
+/*
+ * Called after parse_early_param(), after early parameters (such as mem=)
+ * have been processed, in which case we already have an E820 table filled in
+ * via the parameter callback function(s), but it's not sorted and printed yet:
+ */
+void __init e820__finish_early_params(void)
{
if (userdef) {
- if (sanitize_e820_map(e820->map, ARRAY_SIZE(e820->map),
- &e820->nr_map) < 0)
+ if (e820__update_table(e820_table) < 0)
early_panic("Invalid user supplied memory map");
- printk(KERN_INFO "e820: user-defined physical RAM map:\n");
- e820_print_map("user");
+ pr_info("e820: user-defined physical RAM map:\n");
+ e820__print_table("user");
}
}
-static const char *__init e820_type_to_string(int e820_type)
+static const char *__init e820_type_to_string(struct e820_entry *entry)
{
- switch (e820_type) {
- case E820_RESERVED_KERN:
- case E820_RAM: return "System RAM";
- case E820_ACPI: return "ACPI Tables";
- case E820_NVS: return "ACPI Non-volatile Storage";
- case E820_UNUSABLE: return "Unusable memory";
- case E820_PRAM: return "Persistent Memory (legacy)";
- case E820_PMEM: return "Persistent Memory";
- default: return "reserved";
+ switch (entry->type) {
+ case E820_TYPE_RESERVED_KERN: /* Fall-through: */
+ case E820_TYPE_RAM: return "System RAM";
+ case E820_TYPE_ACPI: return "ACPI Tables";
+ case E820_TYPE_NVS: return "ACPI Non-volatile Storage";
+ case E820_TYPE_UNUSABLE: return "Unusable memory";
+ case E820_TYPE_PRAM: return "Persistent Memory (legacy)";
+ case E820_TYPE_PMEM: return "Persistent Memory";
+ case E820_TYPE_RESERVED: return "Reserved";
+ default: return "Unknown E820 type";
}
}
-static unsigned long __init e820_type_to_iomem_type(int e820_type)
+static unsigned long __init e820_type_to_iomem_type(struct e820_entry *entry)
{
- switch (e820_type) {
- case E820_RESERVED_KERN:
- case E820_RAM:
- return IORESOURCE_SYSTEM_RAM;
- case E820_ACPI:
- case E820_NVS:
- case E820_UNUSABLE:
- case E820_PRAM:
- case E820_PMEM:
- default:
- return IORESOURCE_MEM;
+ switch (entry->type) {
+ case E820_TYPE_RESERVED_KERN: /* Fall-through: */
+ case E820_TYPE_RAM: return IORESOURCE_SYSTEM_RAM;
+ case E820_TYPE_ACPI: /* Fall-through: */
+ case E820_TYPE_NVS: /* Fall-through: */
+ case E820_TYPE_UNUSABLE: /* Fall-through: */
+ case E820_TYPE_PRAM: /* Fall-through: */
+ case E820_TYPE_PMEM: /* Fall-through: */
+ case E820_TYPE_RESERVED: /* Fall-through: */
+ default: return IORESOURCE_MEM;
}
}
-static unsigned long __init e820_type_to_iores_desc(int e820_type)
+static unsigned long __init e820_type_to_iores_desc(struct e820_entry *entry)
{
- switch (e820_type) {
- case E820_ACPI:
- return IORES_DESC_ACPI_TABLES;
- case E820_NVS:
- return IORES_DESC_ACPI_NV_STORAGE;
- case E820_PMEM:
- return IORES_DESC_PERSISTENT_MEMORY;
- case E820_PRAM:
- return IORES_DESC_PERSISTENT_MEMORY_LEGACY;
- case E820_RESERVED_KERN:
- case E820_RAM:
- case E820_UNUSABLE:
- default:
- return IORES_DESC_NONE;
+ switch (entry->type) {
+ case E820_TYPE_ACPI: return IORES_DESC_ACPI_TABLES;
+ case E820_TYPE_NVS: return IORES_DESC_ACPI_NV_STORAGE;
+ case E820_TYPE_PMEM: return IORES_DESC_PERSISTENT_MEMORY;
+ case E820_TYPE_PRAM: return IORES_DESC_PERSISTENT_MEMORY_LEGACY;
+ case E820_TYPE_RESERVED_KERN: /* Fall-through: */
+ case E820_TYPE_RAM: /* Fall-through: */
+ case E820_TYPE_UNUSABLE: /* Fall-through: */
+ case E820_TYPE_RESERVED: /* Fall-through: */
+ default: return IORES_DESC_NONE;
}
}
-static bool __init do_mark_busy(u32 type, struct resource *res)
+static bool __init do_mark_busy(enum e820_type type, struct resource *res)
{
/* this is the legacy bios/dos rom-shadow + mmio region */
if (res->start < (1ULL<<20))
@@ -993,61 +1007,71 @@
* for exclusive use of a driver
*/
switch (type) {
- case E820_RESERVED:
- case E820_PRAM:
- case E820_PMEM:
+ case E820_TYPE_RESERVED:
+ case E820_TYPE_PRAM:
+ case E820_TYPE_PMEM:
return false;
+ case E820_TYPE_RESERVED_KERN:
+ case E820_TYPE_RAM:
+ case E820_TYPE_ACPI:
+ case E820_TYPE_NVS:
+ case E820_TYPE_UNUSABLE:
default:
return true;
}
}
/*
- * Mark e820 reserved areas as busy for the resource manager.
+ * Mark E820 reserved areas as busy for the resource manager:
*/
+
static struct resource __initdata *e820_res;
-void __init e820_reserve_resources(void)
+
+void __init e820__reserve_resources(void)
{
int i;
struct resource *res;
u64 end;
- res = alloc_bootmem(sizeof(struct resource) * e820->nr_map);
+ res = alloc_bootmem(sizeof(*res) * e820_table->nr_entries);
e820_res = res;
- for (i = 0; i < e820->nr_map; i++) {
- end = e820->map[i].addr + e820->map[i].size - 1;
+
+ for (i = 0; i < e820_table->nr_entries; i++) {
+ struct e820_entry *entry = e820_table->entries + i;
+
+ end = entry->addr + entry->size - 1;
if (end != (resource_size_t)end) {
res++;
continue;
}
- res->name = e820_type_to_string(e820->map[i].type);
- res->start = e820->map[i].addr;
- res->end = end;
-
- res->flags = e820_type_to_iomem_type(e820->map[i].type);
- res->desc = e820_type_to_iores_desc(e820->map[i].type);
+ res->start = entry->addr;
+ res->end = end;
+ res->name = e820_type_to_string(entry);
+ res->flags = e820_type_to_iomem_type(entry);
+ res->desc = e820_type_to_iores_desc(entry);
/*
- * don't register the region that could be conflicted with
- * pci device BAR resource and insert them later in
- * pcibios_resource_survey()
+ * Don't register the region that could be conflicted with
+ * PCI device BAR resources and insert them later in
+ * pcibios_resource_survey():
*/
- if (do_mark_busy(e820->map[i].type, res)) {
+ if (do_mark_busy(entry->type, res)) {
res->flags |= IORESOURCE_BUSY;
insert_resource(&iomem_resource, res);
}
res++;
}
- for (i = 0; i < e820_saved->nr_map; i++) {
- struct e820entry *entry = &e820_saved->map[i];
- firmware_map_add_early(entry->addr,
- entry->addr + entry->size,
- e820_type_to_string(entry->type));
+ for (i = 0; i < e820_table_firmware->nr_entries; i++) {
+ struct e820_entry *entry = e820_table_firmware->entries + i;
+
+ firmware_map_add_early(entry->addr, entry->addr + entry->size, e820_type_to_string(entry));
}
}
-/* How much should we pad RAM ending depending on where it is? */
+/*
+ * How much should we pad the end of RAM, depending on where it is?
+ */
static unsigned long __init ram_alignment(resource_size_t pos)
{
unsigned long mb = pos >> 20;
@@ -1066,64 +1090,59 @@
#define MAX_RESOURCE_SIZE ((resource_size_t)-1)
-void __init e820_reserve_resources_late(void)
+void __init e820__reserve_resources_late(void)
{
int i;
struct resource *res;
res = e820_res;
- for (i = 0; i < e820->nr_map; i++) {
+ for (i = 0; i < e820_table->nr_entries; i++) {
if (!res->parent && res->end)
insert_resource_expand_to_fit(&iomem_resource, res);
res++;
}
/*
- * Try to bump up RAM regions to reasonable boundaries to
+ * Try to bump up RAM regions to reasonable boundaries, to
* avoid stolen RAM:
*/
- for (i = 0; i < e820->nr_map; i++) {
- struct e820entry *entry = &e820->map[i];
+ for (i = 0; i < e820_table->nr_entries; i++) {
+ struct e820_entry *entry = &e820_table->entries[i];
u64 start, end;
- if (entry->type != E820_RAM)
+ if (entry->type != E820_TYPE_RAM)
continue;
+
start = entry->addr + entry->size;
end = round_up(start, ram_alignment(start)) - 1;
if (end > MAX_RESOURCE_SIZE)
end = MAX_RESOURCE_SIZE;
if (start >= end)
continue;
- printk(KERN_DEBUG
- "e820: reserve RAM buffer [mem %#010llx-%#010llx]\n",
- start, end);
- reserve_region_with_split(&iomem_resource, start, end,
- "RAM buffer");
+
+ printk(KERN_DEBUG "e820: reserve RAM buffer [mem %#010llx-%#010llx]\n", start, end);
+ reserve_region_with_split(&iomem_resource, start, end, "RAM buffer");
}
}
-char *__init default_machine_specific_memory_setup(void)
+/*
+ * Pass the firmware (bootloader) E820 map to the kernel and process it:
+ */
+char *__init e820__memory_setup_default(void)
{
char *who = "BIOS-e820";
- u32 new_nr;
+
/*
* Try to copy the BIOS-supplied E820-map.
*
* Otherwise fake a memory map; one section from 0k->640k,
* the next section from 1mb->appropriate_mem_k
*/
- new_nr = boot_params.e820_entries;
- sanitize_e820_map(boot_params.e820_map,
- ARRAY_SIZE(boot_params.e820_map),
- &new_nr);
- boot_params.e820_entries = new_nr;
- if (append_e820_map(boot_params.e820_map, boot_params.e820_entries)
- < 0) {
+ if (append_e820_table(boot_params.e820_table, boot_params.e820_entries) < 0) {
u64 mem_size;
- /* compare results from other methods and take the greater */
- if (boot_params.alt_mem_k
- < boot_params.screen_info.ext_mem_k) {
+ /* Compare results from other methods and take the one that gives more RAM: */
+ if (boot_params.alt_mem_k < boot_params.screen_info.ext_mem_k) {
mem_size = boot_params.screen_info.ext_mem_k;
who = "BIOS-88";
} else {
@@ -1131,84 +1150,68 @@
who = "BIOS-e801";
}
- e820->nr_map = 0;
- e820_add_region(0, LOWMEMSIZE(), E820_RAM);
- e820_add_region(HIGH_MEMORY, mem_size << 10, E820_RAM);
+ e820_table->nr_entries = 0;
+ e820__range_add(0, LOWMEMSIZE(), E820_TYPE_RAM);
+ e820__range_add(HIGH_MEMORY, mem_size << 10, E820_TYPE_RAM);
}
- /* In case someone cares... */
+ /* We just appended a lot of ranges, sanitize the table: */
+ e820__update_table(e820_table);
+
return who;
}
-void __init setup_memory_map(void)
+/*
+ * Calls e820__memory_setup_default() in essence to pick up the firmware/bootloader
+ * E820 map - with an optional platform quirk available for virtual platforms
+ * to override this method of boot environment processing:
+ */
+void __init e820__memory_setup(void)
{
char *who;
+ /* This is a firmware interface ABI - make sure we don't break it: */
+ BUILD_BUG_ON(sizeof(struct boot_e820_entry) != 20);
+
who = x86_init.resources.memory_setup();
- memcpy(e820_saved, e820, sizeof(struct e820map));
- printk(KERN_INFO "e820: BIOS-provided physical RAM map:\n");
- e820_print_map(who);
+
+ memcpy(e820_table_firmware, e820_table, sizeof(*e820_table_firmware));
+
+ pr_info("e820: BIOS-provided physical RAM map:\n");
+ e820__print_table(who);
}
-void __init memblock_x86_fill(void)
+void __init e820__memblock_setup(void)
{
int i;
u64 end;
/*
- * EFI may have more than 128 entries
- * We are safe to enable resizing, beause memblock_x86_fill()
- * is rather later for x86
+ * The bootstrap memblock region count maximum is 128 entries
+ * (INIT_MEMBLOCK_REGIONS), but EFI might pass us more E820 entries
+ * than that - so allow memblock resizing.
+ *
+ * This is safe, because this call happens pretty late during x86 setup,
+ * so we know about reserved memory regions already. (This is important
+ * so that memblock resizing does no stomp over reserved areas.)
*/
memblock_allow_resize();
- for (i = 0; i < e820->nr_map; i++) {
- struct e820entry *ei = &e820->map[i];
+ for (i = 0; i < e820_table->nr_entries; i++) {
+ struct e820_entry *entry = &e820_table->entries[i];
- end = ei->addr + ei->size;
+ end = entry->addr + entry->size;
if (end != (resource_size_t)end)
continue;
- if (ei->type != E820_RAM && ei->type != E820_RESERVED_KERN)
+ if (entry->type != E820_TYPE_RAM && entry->type != E820_TYPE_RESERVED_KERN)
continue;
- memblock_add(ei->addr, ei->size);
+ memblock_add(entry->addr, entry->size);
}
- /* throw away partial pages */
+ /* Throw away partial pages: */
memblock_trim_memory(PAGE_SIZE);
memblock_dump_all();
}
-
-void __init memblock_find_dma_reserve(void)
-{
-#ifdef CONFIG_X86_64
- u64 nr_pages = 0, nr_free_pages = 0;
- unsigned long start_pfn, end_pfn;
- phys_addr_t start, end;
- int i;
- u64 u;
-
- /*
- * need to find out used area below MAX_DMA_PFN
- * need to use memblock to get free size in [0, MAX_DMA_PFN]
- * at first, and assume boot_mem will not take below MAX_DMA_PFN
- */
- for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, NULL) {
- start_pfn = min(start_pfn, MAX_DMA_PFN);
- end_pfn = min(end_pfn, MAX_DMA_PFN);
- nr_pages += end_pfn - start_pfn;
- }
-
- for_each_free_mem_range(u, NUMA_NO_NODE, MEMBLOCK_NONE, &start, &end,
- NULL) {
- start_pfn = min_t(unsigned long, PFN_UP(start), MAX_DMA_PFN);
- end_pfn = min_t(unsigned long, PFN_DOWN(end), MAX_DMA_PFN);
- if (start_pfn < end_pfn)
- nr_free_pages += end_pfn - start_pfn;
- }
-
- set_dma_reserve(nr_pages - nr_free_pages);
-#endif
-}
diff --git a/arch/x86/kernel/early-quirks.c b/arch/x86/kernel/early-quirks.c
index 6a08e25..ff7e4b3 100644
--- a/arch/x86/kernel/early-quirks.c
+++ b/arch/x86/kernel/early-quirks.c
@@ -546,8 +546,8 @@
&base, &end);
/* Mark this space as reserved */
- e820_add_region(base, size, E820_RESERVED);
- sanitize_e820_map(e820->map, ARRAY_SIZE(e820->map), &e820->nr_map);
+ e820__range_add(base, size, E820_TYPE_RESERVED);
+ e820__update_table(e820_table);
}
static void __init intel_graphics_quirks(int num, int slot, int func)
diff --git a/arch/x86/kernel/early_printk.c b/arch/x86/kernel/early_printk.c
index 8a12199..0f08403 100644
--- a/arch/x86/kernel/early_printk.c
+++ b/arch/x86/kernel/early_printk.c
@@ -17,6 +17,7 @@
#include <asm/intel-mid.h>
#include <asm/pgtable.h>
#include <linux/usb/ehci_def.h>
+#include <linux/usb/xhci-dbgp.h>
#include <linux/efi.h>
#include <asm/efi.h>
#include <asm/pci_x86.h>
@@ -381,6 +382,10 @@
if (!strncmp(buf, "efi", 3))
early_console_register(&early_efi_console, keep);
#endif
+#ifdef CONFIG_EARLY_PRINTK_USB_XDBC
+ if (!strncmp(buf, "xdbc", 4))
+ early_xdbc_parse_parameter(buf + 4);
+#endif
buf++;
}
diff --git a/arch/x86/kernel/espfix_64.c b/arch/x86/kernel/espfix_64.c
index 04f89ca..8e598a1 100644
--- a/arch/x86/kernel/espfix_64.c
+++ b/arch/x86/kernel/espfix_64.c
@@ -50,11 +50,11 @@
#define ESPFIX_STACKS_PER_PAGE (PAGE_SIZE/ESPFIX_STACK_SIZE)
/* There is address space for how many espfix pages? */
-#define ESPFIX_PAGE_SPACE (1UL << (PGDIR_SHIFT-PAGE_SHIFT-16))
+#define ESPFIX_PAGE_SPACE (1UL << (P4D_SHIFT-PAGE_SHIFT-16))
#define ESPFIX_MAX_CPUS (ESPFIX_STACKS_PER_PAGE * ESPFIX_PAGE_SPACE)
#if CONFIG_NR_CPUS > ESPFIX_MAX_CPUS
-# error "Need more than one PGD for the ESPFIX hack"
+# error "Need more virtual address space for the ESPFIX hack"
#endif
#define PGALLOC_GFP (GFP_KERNEL | __GFP_NOTRACK | __GFP_ZERO)
@@ -121,11 +121,13 @@
void __init init_espfix_bsp(void)
{
- pgd_t *pgd_p;
+ pgd_t *pgd;
+ p4d_t *p4d;
/* Install the espfix pud into the kernel page directory */
- pgd_p = &init_level4_pgt[pgd_index(ESPFIX_BASE_ADDR)];
- pgd_populate(&init_mm, pgd_p, (pud_t *)espfix_pud_page);
+ pgd = &init_level4_pgt[pgd_index(ESPFIX_BASE_ADDR)];
+ p4d = p4d_alloc(&init_mm, pgd, ESPFIX_BASE_ADDR);
+ p4d_populate(&init_mm, p4d, espfix_pud_page);
/* Randomize the locations */
init_espfix_random();
diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index cbd73eb..5b71535 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -29,12 +29,6 @@
#include <asm/ftrace.h>
#include <asm/nops.h>
-#if defined(CONFIG_FUNCTION_GRAPH_TRACER) && \
- !defined(CC_USING_FENTRY) && \
- !defined(CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE)
-# error The following combination is not supported: ((compiler missing -mfentry) || (CONFIG_X86_32 and !CONFIG_DYNAMIC_FTRACE)) && CONFIG_FUNCTION_GRAPH_TRACER && CONFIG_CC_OPTIMIZE_FOR_SIZE
-#endif
-
#ifdef CONFIG_DYNAMIC_FTRACE
int ftrace_arch_code_modify_prepare(void)
@@ -989,6 +983,18 @@
unsigned long return_hooker = (unsigned long)
&return_to_handler;
+ /*
+ * When resuming from suspend-to-ram, this function can be indirectly
+ * called from early CPU startup code while the CPU is in real mode,
+ * which would fail miserably. Make sure the stack pointer is a
+ * virtual address.
+ *
+ * This check isn't as accurate as virt_addr_valid(), but it should be
+ * good enough for this purpose, and it's fast.
+ */
+ if (unlikely((long)__builtin_frame_address(0) >= 0))
+ return;
+
if (unlikely(ftrace_graph_is_dead()))
return;
diff --git a/arch/x86/kernel/ftrace_32.S b/arch/x86/kernel/ftrace_32.S
new file mode 100644
index 0000000..722a145
--- /dev/null
+++ b/arch/x86/kernel/ftrace_32.S
@@ -0,0 +1,244 @@
+/*
+ * Copyright (C) 2017 Steven Rostedt, VMware Inc.
+ */
+
+#include <linux/linkage.h>
+#include <asm/page_types.h>
+#include <asm/segment.h>
+#include <asm/export.h>
+#include <asm/ftrace.h>
+
+#ifdef CC_USING_FENTRY
+# define function_hook __fentry__
+EXPORT_SYMBOL(__fentry__)
+#else
+# define function_hook mcount
+EXPORT_SYMBOL(mcount)
+#endif
+
+#ifdef CONFIG_DYNAMIC_FTRACE
+
+/* mcount uses a frame pointer even if CONFIG_FRAME_POINTER is not set */
+#if !defined(CC_USING_FENTRY) || defined(CONFIG_FRAME_POINTER)
+# define USING_FRAME_POINTER
+#endif
+
+#ifdef USING_FRAME_POINTER
+# define MCOUNT_FRAME 1 /* using frame = true */
+#else
+# define MCOUNT_FRAME 0 /* using frame = false */
+#endif
+
+ENTRY(function_hook)
+ ret
+END(function_hook)
+
+ENTRY(ftrace_caller)
+
+#ifdef USING_FRAME_POINTER
+# ifdef CC_USING_FENTRY
+ /*
+ * Frame pointers are of ip followed by bp.
+ * Since fentry is an immediate jump, we are left with
+ * parent-ip, function-ip. We need to add a frame with
+ * parent-ip followed by ebp.
+ */
+ pushl 4(%esp) /* parent ip */
+ pushl %ebp
+ movl %esp, %ebp
+ pushl 2*4(%esp) /* function ip */
+# endif
+ /* For mcount, the function ip is directly above */
+ pushl %ebp
+ movl %esp, %ebp
+#endif
+ pushl %eax
+ pushl %ecx
+ pushl %edx
+ pushl $0 /* Pass NULL as regs pointer */
+
+#ifdef USING_FRAME_POINTER
+ /* Load parent ebp into edx */
+ movl 4*4(%esp), %edx
+#else
+ /* There's no frame pointer, load the appropriate stack addr instead */
+ lea 4*4(%esp), %edx
+#endif
+
+ movl (MCOUNT_FRAME+4)*4(%esp), %eax /* load the rip */
+ /* Get the parent ip */
+ movl 4(%edx), %edx /* edx has ebp */
+
+ movl function_trace_op, %ecx
+ subl $MCOUNT_INSN_SIZE, %eax
+
+.globl ftrace_call
+ftrace_call:
+ call ftrace_stub
+
+ addl $4, %esp /* skip NULL pointer */
+ popl %edx
+ popl %ecx
+ popl %eax
+#ifdef USING_FRAME_POINTER
+ popl %ebp
+# ifdef CC_USING_FENTRY
+ addl $4,%esp /* skip function ip */
+ popl %ebp /* this is the orig bp */
+ addl $4, %esp /* skip parent ip */
+# endif
+#endif
+.Lftrace_ret:
+#ifdef CONFIG_FUNCTION_GRAPH_TRACER
+.globl ftrace_graph_call
+ftrace_graph_call:
+ jmp ftrace_stub
+#endif
+
+/* This is weak to keep gas from relaxing the jumps */
+WEAK(ftrace_stub)
+ ret
+END(ftrace_caller)
+
+ENTRY(ftrace_regs_caller)
+ /*
+ * i386 does not save SS and ESP when coming from kernel.
+ * Instead, to get sp, ®s->sp is used (see ptrace.h).
+ * Unfortunately, that means eflags must be at the same location
+ * as the current return ip is. We move the return ip into the
+ * regs->ip location, and move flags into the return ip location.
+ */
+ pushl $__KERNEL_CS
+ pushl 4(%esp) /* Save the return ip */
+ pushl $0 /* Load 0 into orig_ax */
+ pushl %gs
+ pushl %fs
+ pushl %es
+ pushl %ds
+ pushl %eax
+
+ /* Get flags and place them into the return ip slot */
+ pushf
+ popl %eax
+ movl %eax, 8*4(%esp)
+
+ pushl %ebp
+ pushl %edi
+ pushl %esi
+ pushl %edx
+ pushl %ecx
+ pushl %ebx
+
+ movl 12*4(%esp), %eax /* Load ip (1st parameter) */
+ subl $MCOUNT_INSN_SIZE, %eax /* Adjust ip */
+#ifdef CC_USING_FENTRY
+ movl 15*4(%esp), %edx /* Load parent ip (2nd parameter) */
+#else
+ movl 0x4(%ebp), %edx /* Load parent ip (2nd parameter) */
+#endif
+ movl function_trace_op, %ecx /* Save ftrace_pos in 3rd parameter */
+ pushl %esp /* Save pt_regs as 4th parameter */
+
+GLOBAL(ftrace_regs_call)
+ call ftrace_stub
+
+ addl $4, %esp /* Skip pt_regs */
+
+ /* restore flags */
+ push 14*4(%esp)
+ popf
+
+ /* Move return ip back to its original location */
+ movl 12*4(%esp), %eax
+ movl %eax, 14*4(%esp)
+
+ popl %ebx
+ popl %ecx
+ popl %edx
+ popl %esi
+ popl %edi
+ popl %ebp
+ popl %eax
+ popl %ds
+ popl %es
+ popl %fs
+ popl %gs
+
+ /* use lea to not affect flags */
+ lea 3*4(%esp), %esp /* Skip orig_ax, ip and cs */
+
+ jmp .Lftrace_ret
+#else /* ! CONFIG_DYNAMIC_FTRACE */
+
+ENTRY(function_hook)
+ cmpl $__PAGE_OFFSET, %esp
+ jb ftrace_stub /* Paging not enabled yet? */
+
+ cmpl $ftrace_stub, ftrace_trace_function
+ jnz .Ltrace
+#ifdef CONFIG_FUNCTION_GRAPH_TRACER
+ cmpl $ftrace_stub, ftrace_graph_return
+ jnz ftrace_graph_caller
+
+ cmpl $ftrace_graph_entry_stub, ftrace_graph_entry
+ jnz ftrace_graph_caller
+#endif
+.globl ftrace_stub
+ftrace_stub:
+ ret
+
+ /* taken from glibc */
+.Ltrace:
+ pushl %eax
+ pushl %ecx
+ pushl %edx
+ movl 0xc(%esp), %eax
+ movl 0x4(%ebp), %edx
+ subl $MCOUNT_INSN_SIZE, %eax
+
+ call *ftrace_trace_function
+
+ popl %edx
+ popl %ecx
+ popl %eax
+ jmp ftrace_stub
+END(function_hook)
+#endif /* CONFIG_DYNAMIC_FTRACE */
+
+#ifdef CONFIG_FUNCTION_GRAPH_TRACER
+ENTRY(ftrace_graph_caller)
+ pushl %eax
+ pushl %ecx
+ pushl %edx
+ movl 3*4(%esp), %eax
+ /* Even with frame pointers, fentry doesn't have one here */
+#ifdef CC_USING_FENTRY
+ lea 4*4(%esp), %edx
+ movl $0, %ecx
+#else
+ lea 0x4(%ebp), %edx
+ movl (%ebp), %ecx
+#endif
+ subl $MCOUNT_INSN_SIZE, %eax
+ call prepare_ftrace_return
+ popl %edx
+ popl %ecx
+ popl %eax
+ ret
+END(ftrace_graph_caller)
+
+.globl return_to_handler
+return_to_handler:
+ pushl %eax
+ pushl %edx
+#ifdef CC_USING_FENTRY
+ movl $0, %eax
+#else
+ movl %ebp, %eax
+#endif
+ call ftrace_return_to_handler
+ movl %eax, %ecx
+ popl %edx
+ popl %eax
+ jmp *%ecx
+#endif
diff --git a/arch/x86/kernel/mcount_64.S b/arch/x86/kernel/ftrace_64.S
similarity index 98%
rename from arch/x86/kernel/mcount_64.S
rename to arch/x86/kernel/ftrace_64.S
index 7b0d3da..1dfac63 100644
--- a/arch/x86/kernel/mcount_64.S
+++ b/arch/x86/kernel/ftrace_64.S
@@ -1,6 +1,4 @@
/*
- * linux/arch/x86_64/mcount_64.S
- *
* Copyright (C) 2014 Steven Rostedt, Red Hat Inc
*/
@@ -13,9 +11,6 @@
.code64
.section .entry.text, "ax"
-
-#ifdef CONFIG_FUNCTION_TRACER
-
#ifdef CC_USING_FENTRY
# define function_hook __fentry__
EXPORT_SYMBOL(__fentry__)
@@ -297,7 +292,6 @@
jmp fgraph_trace
END(function_hook)
#endif /* CONFIG_DYNAMIC_FTRACE */
-#endif /* CONFIG_FUNCTION_TRACER */
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
ENTRY(ftrace_graph_caller)
diff --git a/arch/x86/kernel/head32.c b/arch/x86/kernel/head32.c
index e5fb436..538ec01 100644
--- a/arch/x86/kernel/head32.c
+++ b/arch/x86/kernel/head32.c
@@ -12,7 +12,7 @@
#include <asm/setup.h>
#include <asm/sections.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/page.h>
#include <asm/apic.h>
#include <asm/io_apic.h>
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index b5785c1..43b7002 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -24,7 +24,7 @@
#include <asm/tlbflush.h>
#include <asm/sections.h>
#include <asm/kdebug.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/bios_ebda.h>
#include <asm/bootparam_utils.h>
#include <asm/microcode.h>
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index b467b14..ac9d327 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -269,10 +269,8 @@
/* rsi is pointer to real mode structure with interesting info.
pass it to C */
movq %rsi, %rdi
- jmp start_cpu
-ENDPROC(secondary_startup_64)
-ENTRY(start_cpu)
+.Ljump_to_C_code:
/*
* Jump to run C code and to be on a real kernel address.
* Since we are running on identity-mapped space we have to jump
@@ -305,7 +303,7 @@
pushq %rax # target address in negative space
lretq
.Lafter_lret:
-ENDPROC(start_cpu)
+ENDPROC(secondary_startup_64)
#include "verify_cpu.S"
@@ -313,11 +311,11 @@
/*
* Boot CPU0 entry point. It's called from play_dead(). Everything has been set
* up already except stack. We just set up stack here. Then call
- * start_secondary() via start_cpu().
+ * start_secondary() via .Ljump_to_C_code.
*/
ENTRY(start_cpu0)
movq initial_stack(%rip), %rsp
- jmp start_cpu
+ jmp .Ljump_to_C_code
ENDPROC(start_cpu0)
#endif
diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c
index 4d8183b..f34fe74 100644
--- a/arch/x86/kernel/irq.c
+++ b/arch/x86/kernel/irq.c
@@ -394,6 +394,9 @@
!cpumask_subset(&affinity_new, &online_new))
this_count++;
}
+ /* No need to check any further. */
+ if (!this_count)
+ return 0;
count = 0;
for_each_online_cpu(cpu) {
@@ -411,8 +414,10 @@
for (vector = FIRST_EXTERNAL_VECTOR;
vector < first_system_vector; vector++) {
if (!test_bit(vector, used_vectors) &&
- IS_ERR_OR_NULL(per_cpu(vector_irq, cpu)[vector]))
- count++;
+ IS_ERR_OR_NULL(per_cpu(vector_irq, cpu)[vector])) {
+ if (++count == this_count)
+ return 0;
+ }
}
}
diff --git a/arch/x86/kernel/irqinit.c b/arch/x86/kernel/irqinit.c
index 1423ab1..7468c69 100644
--- a/arch/x86/kernel/irqinit.c
+++ b/arch/x86/kernel/irqinit.c
@@ -195,7 +195,5 @@
if (!acpi_ioapic && !of_ioapic && nr_legacy_irqs())
setup_irq(2, &irq2);
-#ifdef CONFIG_X86_32
irq_ctx_init(smp_processor_id());
-#endif
}
diff --git a/arch/x86/kernel/kexec-bzimage64.c b/arch/x86/kernel/kexec-bzimage64.c
index d0a814a..9d7fd5e 100644
--- a/arch/x86/kernel/kexec-bzimage64.c
+++ b/arch/x86/kernel/kexec-bzimage64.c
@@ -25,6 +25,7 @@
#include <asm/setup.h>
#include <asm/crash.h>
#include <asm/efi.h>
+#include <asm/e820/api.h>
#include <asm/kexec-bzimage64.h>
#define MAX_ELFCOREHDR_STR_LEN 30 /* elfcorehdr=0x<64bit-value> */
@@ -99,15 +100,14 @@
{
unsigned int nr_e820_entries;
- nr_e820_entries = e820_saved->nr_map;
+ nr_e820_entries = e820_table_firmware->nr_entries;
- /* TODO: Pass entries more than E820MAX in bootparams setup data */
- if (nr_e820_entries > E820MAX)
- nr_e820_entries = E820MAX;
+ /* TODO: Pass entries more than E820_MAX_ENTRIES_ZEROPAGE in bootparams setup data */
+ if (nr_e820_entries > E820_MAX_ENTRIES_ZEROPAGE)
+ nr_e820_entries = E820_MAX_ENTRIES_ZEROPAGE;
params->e820_entries = nr_e820_entries;
- memcpy(¶ms->e820_map, &e820_saved->map,
- nr_e820_entries * sizeof(struct e820entry));
+ memcpy(¶ms->e820_table, &e820_table_firmware->entries, nr_e820_entries*sizeof(struct e820_entry));
return 0;
}
@@ -232,10 +232,10 @@
nr_e820_entries = params->e820_entries;
for (i = 0; i < nr_e820_entries; i++) {
- if (params->e820_map[i].type != E820_RAM)
+ if (params->e820_table[i].type != E820_TYPE_RAM)
continue;
- start = params->e820_map[i].addr;
- end = params->e820_map[i].addr + params->e820_map[i].size - 1;
+ start = params->e820_table[i].addr;
+ end = params->e820_table[i].addr + params->e820_table[i].size - 1;
if ((start <= 0x100000) && end > 0x100000) {
mem_k = (end >> 10) - (0x100000 >> 10);
diff --git a/arch/x86/kernel/kprobes/common.h b/arch/x86/kernel/kprobes/common.h
index d688826..db2182d 100644
--- a/arch/x86/kernel/kprobes/common.h
+++ b/arch/x86/kernel/kprobes/common.h
@@ -67,7 +67,7 @@
#endif
/* Ensure if the instruction can be boostable */
-extern int can_boost(kprobe_opcode_t *instruction, void *addr);
+extern int can_boost(struct insn *insn, void *orig_addr);
/* Recover instruction if given address is probed */
extern unsigned long recover_probed_instruction(kprobe_opcode_t *buf,
unsigned long addr);
@@ -75,7 +75,7 @@
* Copy an instruction and adjust the displacement if the instruction
* uses the %rip-relative addressing mode.
*/
-extern int __copy_instruction(u8 *dest, u8 *src);
+extern int __copy_instruction(u8 *dest, u8 *src, struct insn *insn);
/* Generate a relative-jump/call instruction */
extern void synthesize_reljump(void *from, void *to);
diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 993fa4f..19e1f2a 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -164,42 +164,38 @@
NOKPROBE_SYMBOL(skip_prefixes);
/*
- * Returns non-zero if opcode is boostable.
+ * Returns non-zero if INSN is boostable.
* RIP relative instructions are adjusted at copying time in 64 bits mode
*/
-int can_boost(kprobe_opcode_t *opcodes, void *addr)
+int can_boost(struct insn *insn, void *addr)
{
kprobe_opcode_t opcode;
- kprobe_opcode_t *orig_opcodes = opcodes;
if (search_exception_tables((unsigned long)addr))
return 0; /* Page fault may occur on this address. */
-retry:
- if (opcodes - orig_opcodes > MAX_INSN_SIZE - 1)
- return 0;
- opcode = *(opcodes++);
-
/* 2nd-byte opcode */
- if (opcode == 0x0f) {
- if (opcodes - orig_opcodes > MAX_INSN_SIZE - 1)
- return 0;
- return test_bit(*opcodes,
+ if (insn->opcode.nbytes == 2)
+ return test_bit(insn->opcode.bytes[1],
(unsigned long *)twobyte_is_boostable);
- }
+
+ if (insn->opcode.nbytes != 1)
+ return 0;
+
+ /* Can't boost Address-size override prefix */
+ if (unlikely(inat_is_address_size_prefix(insn->attr)))
+ return 0;
+
+ opcode = insn->opcode.bytes[0];
switch (opcode & 0xf0) {
-#ifdef CONFIG_X86_64
- case 0x40:
- goto retry; /* REX prefix is boostable */
-#endif
case 0x60:
- if (0x63 < opcode && opcode < 0x67)
- goto retry; /* prefixes */
- /* can't boost Address-size override and bound */
- return (opcode != 0x62 && opcode != 0x67);
+ /* can't boost "bound" */
+ return (opcode != 0x62);
case 0x70:
return 0; /* can't boost conditional jump */
+ case 0x90:
+ return opcode != 0x9a; /* can't boost call far */
case 0xc0:
/* can't boost software-interruptions */
return (0xc1 < opcode && opcode < 0xcc) || opcode == 0xcf;
@@ -210,14 +206,9 @@
/* can boost in/out and absolute jmps */
return ((opcode & 0x04) || opcode == 0xea);
case 0xf0:
- if ((opcode & 0x0c) == 0 && opcode != 0xf1)
- goto retry; /* lock/rep(ne) prefix */
/* clear and set flags are boostable */
return (opcode == 0xf5 || (0xf7 < opcode && opcode < 0xfe));
default:
- /* segment override prefixes are boostable */
- if (opcode == 0x26 || opcode == 0x36 || opcode == 0x3e)
- goto retry; /* prefixes */
/* CS override prefix and call are not boostable */
return (opcode != 0x2e && opcode != 0x9a);
}
@@ -264,7 +255,10 @@
* Fortunately, we know that the original code is the ideal 5-byte
* long NOP.
*/
- memcpy(buf, (void *)addr, MAX_INSN_SIZE * sizeof(kprobe_opcode_t));
+ if (probe_kernel_read(buf, (void *)addr,
+ MAX_INSN_SIZE * sizeof(kprobe_opcode_t)))
+ return 0UL;
+
if (faddr)
memcpy(buf, ideal_nops[NOP_ATOMIC5], 5);
else
@@ -276,7 +270,7 @@
* Recover the probed instruction at addr for further analysis.
* Caller must lock kprobes by kprobe_mutex, or disable preemption
* for preventing to release referencing kprobes.
- * Returns zero if the instruction can not get recovered.
+ * Returns zero if the instruction can not get recovered (or access failed).
*/
unsigned long recover_probed_instruction(kprobe_opcode_t *buf, unsigned long addr)
{
@@ -348,37 +342,36 @@
}
/*
- * Copy an instruction and adjust the displacement if the instruction
- * uses the %rip-relative addressing mode.
- * If it does, Return the address of the 32-bit displacement word.
- * If not, return null.
- * Only applicable to 64-bit x86.
+ * Copy an instruction with recovering modified instruction by kprobes
+ * and adjust the displacement if the instruction uses the %rip-relative
+ * addressing mode.
+ * This returns the length of copied instruction, or 0 if it has an error.
*/
-int __copy_instruction(u8 *dest, u8 *src)
+int __copy_instruction(u8 *dest, u8 *src, struct insn *insn)
{
- struct insn insn;
kprobe_opcode_t buf[MAX_INSN_SIZE];
- int length;
unsigned long recovered_insn =
recover_probed_instruction(buf, (unsigned long)src);
- if (!recovered_insn)
+ if (!recovered_insn || !insn)
return 0;
- kernel_insn_init(&insn, (void *)recovered_insn, MAX_INSN_SIZE);
- insn_get_length(&insn);
- length = insn.length;
+
+ /* This can access kernel text if given address is not recovered */
+ if (probe_kernel_read(dest, (void *)recovered_insn, MAX_INSN_SIZE))
+ return 0;
+
+ kernel_insn_init(insn, dest, MAX_INSN_SIZE);
+ insn_get_length(insn);
/* Another subsystem puts a breakpoint, failed to recover */
- if (insn.opcode.bytes[0] == BREAKPOINT_INSTRUCTION)
+ if (insn->opcode.bytes[0] == BREAKPOINT_INSTRUCTION)
return 0;
- memcpy(dest, insn.kaddr, length);
#ifdef CONFIG_X86_64
- if (insn_rip_relative(&insn)) {
+ /* Only x86_64 has RIP relative instructions */
+ if (insn_rip_relative(insn)) {
s64 newdisp;
u8 *disp;
- kernel_insn_init(&insn, dest, length);
- insn_get_displacement(&insn);
/*
* The copied instruction uses the %rip-relative addressing
* mode. Adjust the displacement for the difference between
@@ -391,36 +384,57 @@
* extension of the original signed 32-bit displacement would
* have given.
*/
- newdisp = (u8 *) src + (s64) insn.displacement.value - (u8 *) dest;
+ newdisp = (u8 *) src + (s64) insn->displacement.value
+ - (u8 *) dest;
if ((s64) (s32) newdisp != newdisp) {
pr_err("Kprobes error: new displacement does not fit into s32 (%llx)\n", newdisp);
- pr_err("\tSrc: %p, Dest: %p, old disp: %x\n", src, dest, insn.displacement.value);
+ pr_err("\tSrc: %p, Dest: %p, old disp: %x\n",
+ src, dest, insn->displacement.value);
return 0;
}
- disp = (u8 *) dest + insn_offset_displacement(&insn);
+ disp = (u8 *) dest + insn_offset_displacement(insn);
*(s32 *) disp = (s32) newdisp;
}
#endif
- return length;
+ return insn->length;
+}
+
+/* Prepare reljump right after instruction to boost */
+static void prepare_boost(struct kprobe *p, struct insn *insn)
+{
+ if (can_boost(insn, p->addr) &&
+ MAX_INSN_SIZE - insn->length >= RELATIVEJUMP_SIZE) {
+ /*
+ * These instructions can be executed directly if it
+ * jumps back to correct address.
+ */
+ synthesize_reljump(p->ainsn.insn + insn->length,
+ p->addr + insn->length);
+ p->ainsn.boostable = true;
+ } else {
+ p->ainsn.boostable = false;
+ }
}
static int arch_copy_kprobe(struct kprobe *p)
{
- int ret;
+ struct insn insn;
+ int len;
+
+ set_memory_rw((unsigned long)p->ainsn.insn & PAGE_MASK, 1);
/* Copy an instruction with recovering if other optprobe modifies it.*/
- ret = __copy_instruction(p->ainsn.insn, p->addr);
- if (!ret)
+ len = __copy_instruction(p->ainsn.insn, p->addr, &insn);
+ if (!len)
return -EINVAL;
/*
* __copy_instruction can modify the displacement of the instruction,
* but it doesn't affect boostable check.
*/
- if (can_boost(p->ainsn.insn, p->addr))
- p->ainsn.boostable = 0;
- else
- p->ainsn.boostable = -1;
+ prepare_boost(p, &insn);
+
+ set_memory_ro((unsigned long)p->ainsn.insn & PAGE_MASK, 1);
/* Check whether the instruction modifies Interrupt Flag or not */
p->ainsn.if_modifier = is_IF_modifier(p->ainsn.insn);
@@ -459,7 +473,7 @@
void arch_remove_kprobe(struct kprobe *p)
{
if (p->ainsn.insn) {
- free_insn_slot(p->ainsn.insn, (p->ainsn.boostable == 1));
+ free_insn_slot(p->ainsn.insn, p->ainsn.boostable);
p->ainsn.insn = NULL;
}
}
@@ -531,7 +545,7 @@
return;
#if !defined(CONFIG_PREEMPT)
- if (p->ainsn.boostable == 1 && !p->post_handler) {
+ if (p->ainsn.boostable && !p->post_handler) {
/* Boost up -- we can execute copied instructions directly */
if (!reenter)
reset_current_kprobe();
@@ -851,7 +865,7 @@
case 0xcf:
case 0xea: /* jmp absolute -- ip is correct */
/* ip is already adjusted, no more changes required */
- p->ainsn.boostable = 1;
+ p->ainsn.boostable = true;
goto no_change;
case 0xe8: /* call relative - Fix return addr */
*tos = orig_ip + (*tos - copy_ip);
@@ -876,28 +890,13 @@
* jmp near and far, absolute indirect
* ip is correct. And this is boostable
*/
- p->ainsn.boostable = 1;
+ p->ainsn.boostable = true;
goto no_change;
}
default:
break;
}
- if (p->ainsn.boostable == 0) {
- if ((regs->ip > copy_ip) &&
- (regs->ip - copy_ip) + 5 < MAX_INSN_SIZE) {
- /*
- * These instructions can be executed directly if it
- * jumps back to correct address.
- */
- synthesize_reljump((void *)regs->ip,
- (void *)orig_ip + (regs->ip - copy_ip));
- p->ainsn.boostable = 1;
- } else {
- p->ainsn.boostable = -1;
- }
- }
-
regs->ip += orig_ip - copy_ip;
no_change:
diff --git a/arch/x86/kernel/kprobes/ftrace.c b/arch/x86/kernel/kprobes/ftrace.c
index 5f8f0b3..041f7b6 100644
--- a/arch/x86/kernel/kprobes/ftrace.c
+++ b/arch/x86/kernel/kprobes/ftrace.c
@@ -94,6 +94,6 @@
int arch_prepare_kprobe_ftrace(struct kprobe *p)
{
p->ainsn.insn = NULL;
- p->ainsn.boostable = -1;
+ p->ainsn.boostable = false;
return 0;
}
diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c
index 3e7c6e5..9aadff3 100644
--- a/arch/x86/kernel/kprobes/opt.c
+++ b/arch/x86/kernel/kprobes/opt.c
@@ -65,7 +65,10 @@
* overwritten by jump destination address. In this case, original
* bytes must be recovered from op->optinsn.copied_insn buffer.
*/
- memcpy(buf, (void *)addr, MAX_INSN_SIZE * sizeof(kprobe_opcode_t));
+ if (probe_kernel_read(buf, (void *)addr,
+ MAX_INSN_SIZE * sizeof(kprobe_opcode_t)))
+ return 0UL;
+
if (addr == (unsigned long)kp->addr) {
buf[0] = kp->opcode;
memcpy(buf + 1, op->optinsn.copied_insn, RELATIVE_ADDR_SIZE);
@@ -174,11 +177,12 @@
static int copy_optimized_instructions(u8 *dest, u8 *src)
{
+ struct insn insn;
int len = 0, ret;
while (len < RELATIVEJUMP_SIZE) {
- ret = __copy_instruction(dest + len, src + len);
- if (!ret || !can_boost(dest + len, src + len))
+ ret = __copy_instruction(dest + len, src + len, &insn);
+ if (!ret || !can_boost(&insn, src + len))
return -EINVAL;
len += ret;
}
@@ -350,6 +354,7 @@
}
buf = (u8 *)op->optinsn.insn;
+ set_memory_rw((unsigned long)buf & PAGE_MASK, 1);
/* Copy instructions into the out-of-line buffer */
ret = copy_optimized_instructions(buf + TMPL_END_IDX, op->kp.addr);
@@ -372,6 +377,8 @@
synthesize_reljump(buf + TMPL_END_IDX + op->optinsn.size,
(u8 *)op->kp.addr + op->optinsn.size);
+ set_memory_ro((unsigned long)buf & PAGE_MASK, 1);
+
flush_icache_range((unsigned long) buf,
(unsigned long) buf + TMPL_END_IDX +
op->optinsn.size + RELATIVEJUMP_SIZE);
diff --git a/arch/x86/kernel/machine_kexec_32.c b/arch/x86/kernel/machine_kexec_32.c
index 469b23d..5f43cec 100644
--- a/arch/x86/kernel/machine_kexec_32.c
+++ b/arch/x86/kernel/machine_kexec_32.c
@@ -103,6 +103,7 @@
pgd_t *pgd, pmd_t *pmd, pte_t *pte,
unsigned long vaddr, unsigned long paddr)
{
+ p4d_t *p4d;
pud_t *pud;
pgd += pgd_index(vaddr);
@@ -110,7 +111,8 @@
if (!(pgd_val(*pgd) & _PAGE_PRESENT))
set_pgd(pgd, __pgd(__pa(pmd) | _PAGE_PRESENT));
#endif
- pud = pud_offset(pgd, vaddr);
+ p4d = p4d_offset(pgd, vaddr);
+ pud = pud_offset(p4d, vaddr);
pmd = pmd_offset(pud, vaddr);
if (!(pmd_val(*pmd) & _PAGE_PRESENT))
set_pmd(pmd, __pmd(__pa(pte) | _PAGE_TABLE));
diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c
index 857cdbd..085c3b3 100644
--- a/arch/x86/kernel/machine_kexec_64.c
+++ b/arch/x86/kernel/machine_kexec_64.c
@@ -36,6 +36,7 @@
static void free_transition_pgtable(struct kimage *image)
{
+ free_page((unsigned long)image->arch.p4d);
free_page((unsigned long)image->arch.pud);
free_page((unsigned long)image->arch.pmd);
free_page((unsigned long)image->arch.pte);
@@ -43,6 +44,7 @@
static int init_transition_pgtable(struct kimage *image, pgd_t *pgd)
{
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
@@ -53,13 +55,21 @@
paddr = __pa(page_address(image->control_code_page)+PAGE_SIZE);
pgd += pgd_index(vaddr);
if (!pgd_present(*pgd)) {
+ p4d = (p4d_t *)get_zeroed_page(GFP_KERNEL);
+ if (!p4d)
+ goto err;
+ image->arch.p4d = p4d;
+ set_pgd(pgd, __pgd(__pa(p4d) | _KERNPG_TABLE));
+ }
+ p4d = p4d_offset(pgd, vaddr);
+ if (!p4d_present(*p4d)) {
pud = (pud_t *)get_zeroed_page(GFP_KERNEL);
if (!pud)
goto err;
image->arch.pud = pud;
- set_pgd(pgd, __pgd(__pa(pud) | _KERNPG_TABLE));
+ set_p4d(p4d, __p4d(__pa(pud) | _KERNPG_TABLE));
}
- pud = pud_offset(pgd, vaddr);
+ pud = pud_offset(p4d, vaddr);
if (!pud_present(*pud)) {
pmd = (pmd_t *)get_zeroed_page(GFP_KERNEL);
if (!pmd)
diff --git a/arch/x86/kernel/mpparse.c b/arch/x86/kernel/mpparse.c
index 0f8d204..0d904d7 100644
--- a/arch/x86/kernel/mpparse.c
+++ b/arch/x86/kernel/mpparse.c
@@ -26,7 +26,7 @@
#include <asm/io_apic.h>
#include <asm/proto.h>
#include <asm/bios_ebda.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/setup.h>
#include <asm/smp.h>
@@ -826,10 +826,10 @@
}
early_param("alloc_mptable", parse_alloc_mptable_opt);
-void __init early_reserve_e820_mpc_new(void)
+void __init e820__memblock_alloc_reserved_mpc_new(void)
{
if (enable_update_mptable && alloc_mptable)
- mpc_new_phys = early_reserve_e820(mpc_new_length, 4);
+ mpc_new_phys = e820__memblock_alloc_reserved(mpc_new_length, 4);
}
static int __init update_mp_table(void)
diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c
index a723ae94..446c8aa 100644
--- a/arch/x86/kernel/nmi.c
+++ b/arch/x86/kernel/nmi.c
@@ -222,17 +222,6 @@
pr_emerg("NMI: PCI system error (SERR) for reason %02x on CPU %d.\n",
reason, smp_processor_id());
- /*
- * On some machines, PCI SERR line is used to report memory
- * errors. EDAC makes use of it.
- */
-#if defined(CONFIG_EDAC)
- if (edac_handler_set()) {
- edac_atomic_assert_error();
- return;
- }
-#endif
-
if (panic_on_unrecovered_nmi)
nmi_panic(regs, "NMI: Not continuing");
diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
index 4797e87..3586996 100644
--- a/arch/x86/kernel/paravirt.c
+++ b/arch/x86/kernel/paravirt.c
@@ -405,9 +405,11 @@
.alloc_pte = paravirt_nop,
.alloc_pmd = paravirt_nop,
.alloc_pud = paravirt_nop,
+ .alloc_p4d = paravirt_nop,
.release_pte = paravirt_nop,
.release_pmd = paravirt_nop,
.release_pud = paravirt_nop,
+ .release_p4d = paravirt_nop,
.set_pte = native_set_pte,
.set_pte_at = native_set_pte_at,
@@ -430,12 +432,19 @@
.pmd_val = PTE_IDENT,
.make_pmd = PTE_IDENT,
-#if CONFIG_PGTABLE_LEVELS == 4
+#if CONFIG_PGTABLE_LEVELS >= 4
.pud_val = PTE_IDENT,
.make_pud = PTE_IDENT,
+ .set_p4d = native_set_p4d,
+
+#if CONFIG_PGTABLE_LEVELS >= 5
+ .p4d_val = PTE_IDENT,
+ .make_p4d = PTE_IDENT,
+
.set_pgd = native_set_pgd,
-#endif
+#endif /* CONFIG_PGTABLE_LEVELS >= 5 */
+#endif /* CONFIG_PGTABLE_LEVELS >= 4 */
#endif /* CONFIG_PGTABLE_LEVELS >= 3 */
.pte_val = PTE_IDENT,
diff --git a/arch/x86/kernel/pci-calgary_64.c b/arch/x86/kernel/pci-calgary_64.c
index 0c150c0..fda7867 100644
--- a/arch/x86/kernel/pci-calgary_64.c
+++ b/arch/x86/kernel/pci-calgary_64.c
@@ -1007,9 +1007,8 @@
writel(cpu_to_be32(val32), target);
readl(target); /* flush */
- init_timer(&tbl->watchdog_timer);
- tbl->watchdog_timer.function = &calgary_watchdog;
- tbl->watchdog_timer.data = (unsigned long)dev;
+ setup_timer(&tbl->watchdog_timer, &calgary_watchdog,
+ (unsigned long)dev);
mod_timer(&tbl->watchdog_timer, jiffies);
}
diff --git a/arch/x86/kernel/probe_roms.c b/arch/x86/kernel/probe_roms.c
index d5f15c3..963e3fb 100644
--- a/arch/x86/kernel/probe_roms.c
+++ b/arch/x86/kernel/probe_roms.c
@@ -14,7 +14,7 @@
#include <asm/probe_roms.h>
#include <asm/pci-direct.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/mmzone.h>
#include <asm/setup.h>
#include <asm/sections.h>
diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
index f675915..0bb8842 100644
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -37,6 +37,7 @@
#include <asm/vm86.h>
#include <asm/switch_to.h>
#include <asm/desc.h>
+#include <asm/prctl.h>
/*
* per-CPU TSS segments. Threads are completely 'soft' on Linux,
@@ -124,11 +125,6 @@
fpu__clear(&tsk->thread.fpu);
}
-static void hard_disable_TSC(void)
-{
- cr4_set_bits(X86_CR4_TSD);
-}
-
void disable_TSC(void)
{
preempt_disable();
@@ -137,15 +133,10 @@
* Must flip the CPU state synchronously with
* TIF_NOTSC in the current running context.
*/
- hard_disable_TSC();
+ cr4_set_bits(X86_CR4_TSD);
preempt_enable();
}
-static void hard_enable_TSC(void)
-{
- cr4_clear_bits(X86_CR4_TSD);
-}
-
static void enable_TSC(void)
{
preempt_disable();
@@ -154,7 +145,7 @@
* Must flip the CPU state synchronously with
* TIF_NOTSC in the current running context.
*/
- hard_enable_TSC();
+ cr4_clear_bits(X86_CR4_TSD);
preempt_enable();
}
@@ -182,54 +173,129 @@
return 0;
}
-void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p,
- struct tss_struct *tss)
+DEFINE_PER_CPU(u64, msr_misc_features_shadow);
+
+static void set_cpuid_faulting(bool on)
{
- struct thread_struct *prev, *next;
+ u64 msrval;
- prev = &prev_p->thread;
- next = &next_p->thread;
+ msrval = this_cpu_read(msr_misc_features_shadow);
+ msrval &= ~MSR_MISC_FEATURES_ENABLES_CPUID_FAULT;
+ msrval |= (on << MSR_MISC_FEATURES_ENABLES_CPUID_FAULT_BIT);
+ this_cpu_write(msr_misc_features_shadow, msrval);
+ wrmsrl(MSR_MISC_FEATURES_ENABLES, msrval);
+}
- if (test_tsk_thread_flag(prev_p, TIF_BLOCKSTEP) ^
- test_tsk_thread_flag(next_p, TIF_BLOCKSTEP)) {
- unsigned long debugctl = get_debugctlmsr();
-
- debugctl &= ~DEBUGCTLMSR_BTF;
- if (test_tsk_thread_flag(next_p, TIF_BLOCKSTEP))
- debugctl |= DEBUGCTLMSR_BTF;
-
- update_debugctlmsr(debugctl);
+static void disable_cpuid(void)
+{
+ preempt_disable();
+ if (!test_and_set_thread_flag(TIF_NOCPUID)) {
+ /*
+ * Must flip the CPU state synchronously with
+ * TIF_NOCPUID in the current running context.
+ */
+ set_cpuid_faulting(true);
}
+ preempt_enable();
+}
- if (test_tsk_thread_flag(prev_p, TIF_NOTSC) ^
- test_tsk_thread_flag(next_p, TIF_NOTSC)) {
- /* prev and next are different */
- if (test_tsk_thread_flag(next_p, TIF_NOTSC))
- hard_disable_TSC();
- else
- hard_enable_TSC();
+static void enable_cpuid(void)
+{
+ preempt_disable();
+ if (test_and_clear_thread_flag(TIF_NOCPUID)) {
+ /*
+ * Must flip the CPU state synchronously with
+ * TIF_NOCPUID in the current running context.
+ */
+ set_cpuid_faulting(false);
}
+ preempt_enable();
+}
- if (test_tsk_thread_flag(next_p, TIF_IO_BITMAP)) {
+static int get_cpuid_mode(void)
+{
+ return !test_thread_flag(TIF_NOCPUID);
+}
+
+static int set_cpuid_mode(struct task_struct *task, unsigned long cpuid_enabled)
+{
+ if (!static_cpu_has(X86_FEATURE_CPUID_FAULT))
+ return -ENODEV;
+
+ if (cpuid_enabled)
+ enable_cpuid();
+ else
+ disable_cpuid();
+
+ return 0;
+}
+
+/*
+ * Called immediately after a successful exec.
+ */
+void arch_setup_new_exec(void)
+{
+ /* If cpuid was previously disabled for this task, re-enable it. */
+ if (test_thread_flag(TIF_NOCPUID))
+ enable_cpuid();
+}
+
+static inline void switch_to_bitmap(struct tss_struct *tss,
+ struct thread_struct *prev,
+ struct thread_struct *next,
+ unsigned long tifp, unsigned long tifn)
+{
+ if (tifn & _TIF_IO_BITMAP) {
/*
* Copy the relevant range of the IO bitmap.
* Normally this is 128 bytes or less:
*/
memcpy(tss->io_bitmap, next->io_bitmap_ptr,
max(prev->io_bitmap_max, next->io_bitmap_max));
-
/*
* Make sure that the TSS limit is correct for the CPU
* to notice the IO bitmap.
*/
refresh_tss_limit();
- } else if (test_tsk_thread_flag(prev_p, TIF_IO_BITMAP)) {
+ } else if (tifp & _TIF_IO_BITMAP) {
/*
* Clear any possible leftover bits:
*/
memset(tss->io_bitmap, 0xff, prev->io_bitmap_max);
}
+}
+
+void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p,
+ struct tss_struct *tss)
+{
+ struct thread_struct *prev, *next;
+ unsigned long tifp, tifn;
+
+ prev = &prev_p->thread;
+ next = &next_p->thread;
+
+ tifn = READ_ONCE(task_thread_info(next_p)->flags);
+ tifp = READ_ONCE(task_thread_info(prev_p)->flags);
+ switch_to_bitmap(tss, prev, next, tifp, tifn);
+
propagate_user_return_notify(prev_p, next_p);
+
+ if ((tifp & _TIF_BLOCKSTEP || tifn & _TIF_BLOCKSTEP) &&
+ arch_has_block_step()) {
+ unsigned long debugctl, msk;
+
+ rdmsrl(MSR_IA32_DEBUGCTLMSR, debugctl);
+ debugctl &= ~DEBUGCTLMSR_BTF;
+ msk = tifn & _TIF_BLOCKSTEP;
+ debugctl |= (msk >> TIF_BLOCKSTEP) << DEBUGCTLMSR_BTF_SHIFT;
+ wrmsrl(MSR_IA32_DEBUGCTLMSR, debugctl);
+ }
+
+ if ((tifp ^ tifn) & _TIF_NOTSC)
+ cr4_toggle_bits(X86_CR4_TSD);
+
+ if ((tifp ^ tifn) & _TIF_NOCPUID)
+ set_cpuid_faulting(!!(tifn & _TIF_NOCPUID));
}
/*
@@ -550,3 +616,16 @@
put_task_stack(p);
return ret;
}
+
+long do_arch_prctl_common(struct task_struct *task, int option,
+ unsigned long cpuid_enabled)
+{
+ switch (option) {
+ case ARCH_GET_CPUID:
+ return get_cpuid_mode();
+ case ARCH_SET_CPUID:
+ return set_cpuid_mode(task, cpuid_enabled);
+ }
+
+ return -EINVAL;
+}
diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c
index 4c818f8..ff40e74 100644
--- a/arch/x86/kernel/process_32.c
+++ b/arch/x86/kernel/process_32.c
@@ -37,6 +37,7 @@
#include <linux/uaccess.h>
#include <linux/io.h>
#include <linux/kdebug.h>
+#include <linux/syscalls.h>
#include <asm/pgtable.h>
#include <asm/ldt.h>
@@ -56,6 +57,7 @@
#include <asm/switch_to.h>
#include <asm/vm86.h>
#include <asm/intel_rdt.h>
+#include <asm/proto.h>
void __show_regs(struct pt_regs *regs, int all)
{
@@ -304,3 +306,8 @@
return prev_p;
}
+
+SYSCALL_DEFINE2(arch_prctl, int, option, unsigned long, arg2)
+{
+ return do_arch_prctl_common(current, option, arg2);
+}
diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c
index d6b784a..825a1e4 100644
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -37,6 +37,7 @@
#include <linux/uaccess.h>
#include <linux/io.h>
#include <linux/ftrace.h>
+#include <linux/syscalls.h>
#include <asm/pgtable.h>
#include <asm/processor.h>
@@ -52,6 +53,11 @@
#include <asm/xen/hypervisor.h>
#include <asm/vdso.h>
#include <asm/intel_rdt.h>
+#include <asm/unistd.h>
+#ifdef CONFIG_IA32_EMULATION
+/* Not included via unistd.h */
+#include <asm/unistd_32_ia32.h>
+#endif
__visible DEFINE_PER_CPU(unsigned long, rsp_scratch);
@@ -204,7 +210,7 @@
(struct user_desc __user *)tls, 0);
else
#endif
- err = do_arch_prctl(p, ARCH_SET_FS, tls);
+ err = do_arch_prctl_64(p, ARCH_SET_FS, tls);
if (err)
goto out;
}
@@ -493,6 +499,8 @@
clear_thread_flag(TIF_IA32);
clear_thread_flag(TIF_ADDR32);
clear_thread_flag(TIF_X32);
+ /* Pretend that this comes from a 64bit execve */
+ task_pt_regs(current)->orig_ax = __NR_execve;
/* Ensure the corresponding mm is not marked. */
if (current->mm)
@@ -505,32 +513,50 @@
current->personality &= ~READ_IMPLIES_EXEC;
}
+static void __set_personality_x32(void)
+{
+#ifdef CONFIG_X86_X32
+ clear_thread_flag(TIF_IA32);
+ set_thread_flag(TIF_X32);
+ if (current->mm)
+ current->mm->context.ia32_compat = TIF_X32;
+ current->personality &= ~READ_IMPLIES_EXEC;
+ /*
+ * in_compat_syscall() uses the presence of the x32 syscall bit
+ * flag to determine compat status. The x86 mmap() code relies on
+ * the syscall bitness so set x32 syscall bit right here to make
+ * in_compat_syscall() work during exec().
+ *
+ * Pretend to come from a x32 execve.
+ */
+ task_pt_regs(current)->orig_ax = __NR_x32_execve | __X32_SYSCALL_BIT;
+ current->thread.status &= ~TS_COMPAT;
+#endif
+}
+
+static void __set_personality_ia32(void)
+{
+#ifdef CONFIG_IA32_EMULATION
+ set_thread_flag(TIF_IA32);
+ clear_thread_flag(TIF_X32);
+ if (current->mm)
+ current->mm->context.ia32_compat = TIF_IA32;
+ current->personality |= force_personality32;
+ /* Prepare the first "return" to user space */
+ task_pt_regs(current)->orig_ax = __NR_ia32_execve;
+ current->thread.status |= TS_COMPAT;
+#endif
+}
+
void set_personality_ia32(bool x32)
{
- /* inherit personality from parent */
-
/* Make sure to be in 32bit mode */
set_thread_flag(TIF_ADDR32);
- /* Mark the associated mm as containing 32-bit tasks. */
- if (x32) {
- clear_thread_flag(TIF_IA32);
- set_thread_flag(TIF_X32);
- if (current->mm)
- current->mm->context.ia32_compat = TIF_X32;
- current->personality &= ~READ_IMPLIES_EXEC;
- /* in_compat_syscall() uses the presence of the x32
- syscall bit flag to determine compat status */
- current->thread.status &= ~TS_COMPAT;
- } else {
- set_thread_flag(TIF_IA32);
- clear_thread_flag(TIF_X32);
- if (current->mm)
- current->mm->context.ia32_compat = TIF_IA32;
- current->personality |= force_personality32;
- /* Prepare the first "return" to user space */
- current->thread.status |= TS_COMPAT;
- }
+ if (x32)
+ __set_personality_x32();
+ else
+ __set_personality_ia32();
}
EXPORT_SYMBOL_GPL(set_personality_ia32);
@@ -547,70 +573,72 @@
}
#endif
-long do_arch_prctl(struct task_struct *task, int code, unsigned long addr)
+long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)
{
int ret = 0;
int doit = task == current;
int cpu;
- switch (code) {
+ switch (option) {
case ARCH_SET_GS:
- if (addr >= TASK_SIZE_MAX)
+ if (arg2 >= TASK_SIZE_MAX)
return -EPERM;
cpu = get_cpu();
task->thread.gsindex = 0;
- task->thread.gsbase = addr;
+ task->thread.gsbase = arg2;
if (doit) {
load_gs_index(0);
- ret = wrmsrl_safe(MSR_KERNEL_GS_BASE, addr);
+ ret = wrmsrl_safe(MSR_KERNEL_GS_BASE, arg2);
}
put_cpu();
break;
case ARCH_SET_FS:
/* Not strictly needed for fs, but do it for symmetry
with gs */
- if (addr >= TASK_SIZE_MAX)
+ if (arg2 >= TASK_SIZE_MAX)
return -EPERM;
cpu = get_cpu();
task->thread.fsindex = 0;
- task->thread.fsbase = addr;
+ task->thread.fsbase = arg2;
if (doit) {
/* set the selector to 0 to not confuse __switch_to */
loadsegment(fs, 0);
- ret = wrmsrl_safe(MSR_FS_BASE, addr);
+ ret = wrmsrl_safe(MSR_FS_BASE, arg2);
}
put_cpu();
break;
case ARCH_GET_FS: {
unsigned long base;
+
if (doit)
rdmsrl(MSR_FS_BASE, base);
else
base = task->thread.fsbase;
- ret = put_user(base, (unsigned long __user *)addr);
+ ret = put_user(base, (unsigned long __user *)arg2);
break;
}
case ARCH_GET_GS: {
unsigned long base;
+
if (doit)
rdmsrl(MSR_KERNEL_GS_BASE, base);
else
base = task->thread.gsbase;
- ret = put_user(base, (unsigned long __user *)addr);
+ ret = put_user(base, (unsigned long __user *)arg2);
break;
}
#ifdef CONFIG_CHECKPOINT_RESTORE
# ifdef CONFIG_X86_X32_ABI
case ARCH_MAP_VDSO_X32:
- return prctl_map_vdso(&vdso_image_x32, addr);
+ return prctl_map_vdso(&vdso_image_x32, arg2);
# endif
# if defined CONFIG_X86_32 || defined CONFIG_IA32_EMULATION
case ARCH_MAP_VDSO_32:
- return prctl_map_vdso(&vdso_image_32, addr);
+ return prctl_map_vdso(&vdso_image_32, arg2);
# endif
case ARCH_MAP_VDSO_64:
- return prctl_map_vdso(&vdso_image_64, addr);
+ return prctl_map_vdso(&vdso_image_64, arg2);
#endif
default:
@@ -621,11 +649,24 @@
return ret;
}
-long sys_arch_prctl(int code, unsigned long addr)
+SYSCALL_DEFINE2(arch_prctl, int, option, unsigned long, arg2)
{
- return do_arch_prctl(current, code, addr);
+ long ret;
+
+ ret = do_arch_prctl_64(current, option, arg2);
+ if (ret == -EINVAL)
+ ret = do_arch_prctl_common(current, option, arg2);
+
+ return ret;
}
+#ifdef CONFIG_IA32_EMULATION
+COMPAT_SYSCALL_DEFINE2(arch_prctl, int, option, unsigned long, arg2)
+{
+ return do_arch_prctl_common(current, option, arg2);
+}
+#endif
+
unsigned long KSTK_ESP(struct task_struct *task)
{
return task_pt_regs(task)->sp;
diff --git a/arch/x86/kernel/ptrace.c b/arch/x86/kernel/ptrace.c
index 2364b23..f37d181 100644
--- a/arch/x86/kernel/ptrace.c
+++ b/arch/x86/kernel/ptrace.c
@@ -396,12 +396,12 @@
if (value >= TASK_SIZE_MAX)
return -EIO;
/*
- * When changing the segment base, use do_arch_prctl
+ * When changing the segment base, use do_arch_prctl_64
* to set either thread.fs or thread.fsindex and the
* corresponding GDT slot.
*/
if (child->thread.fsbase != value)
- return do_arch_prctl(child, ARCH_SET_FS, value);
+ return do_arch_prctl_64(child, ARCH_SET_FS, value);
return 0;
case offsetof(struct user_regs_struct,gs_base):
/*
@@ -410,7 +410,7 @@
if (value >= TASK_SIZE_MAX)
return -EIO;
if (child->thread.gsbase != value)
- return do_arch_prctl(child, ARCH_SET_GS, value);
+ return do_arch_prctl_64(child, ARCH_SET_GS, value);
return 0;
#endif
}
@@ -869,7 +869,7 @@
Works just like arch_prctl, except that the arguments
are reversed. */
case PTRACE_ARCH_PRCTL:
- ret = do_arch_prctl(child, data, addr);
+ ret = do_arch_prctl_64(child, data, addr);
break;
#endif
diff --git a/arch/x86/kernel/reboot.c b/arch/x86/kernel/reboot.c
index 067f981..2544700 100644
--- a/arch/x86/kernel/reboot.c
+++ b/arch/x86/kernel/reboot.c
@@ -765,10 +765,11 @@
#endif
+/* This is the CPU performing the emergency shutdown work. */
+int crashing_cpu = -1;
+
#if defined(CONFIG_SMP)
-/* This keeps a track of which one is crashing cpu. */
-static int crashing_cpu;
static nmi_shootdown_cb shootdown_callback;
static atomic_t waiting_for_crash_ipi;
diff --git a/arch/x86/kernel/resource.c b/arch/x86/kernel/resource.c
index 2408c16..5ab3895 100644
--- a/arch/x86/kernel/resource.c
+++ b/arch/x86/kernel/resource.c
@@ -1,5 +1,5 @@
#include <linux/ioport.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
static void resource_clip(struct resource *res, resource_size_t start,
resource_size_t end)
@@ -25,10 +25,10 @@
static void remove_e820_regions(struct resource *avail)
{
int i;
- struct e820entry *entry;
+ struct e820_entry *entry;
- for (i = 0; i < e820->nr_map; i++) {
- entry = &e820->map[i];
+ for (i = 0; i < e820_table->nr_entries; i++) {
+ entry = &e820_table->entries[i];
resource_clip(avail, entry->addr,
entry->addr + entry->size - 1);
diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index 4bf0c89..603a166 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -70,12 +70,13 @@
#include <linux/tboot.h>
#include <linux/jiffies.h>
+#include <linux/usb/xhci-dbgp.h>
#include <video/edid.h>
#include <asm/mtrr.h>
#include <asm/apic.h>
#include <asm/realmode.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/mpspec.h>
#include <asm/setup.h>
#include <asm/efi.h>
@@ -119,7 +120,7 @@
* max_low_pfn_mapped: highest direct mapped pfn under 4GB
* max_pfn_mapped: highest direct mapped pfn over 4GB
*
- * The direct mapping only covers E820_RAM regions, so the ranges and gaps are
+ * The direct mapping only covers E820_TYPE_RAM regions, so the ranges and gaps are
* represented by pfn_mapped
*/
unsigned long max_low_pfn_mapped;
@@ -173,14 +174,11 @@
#ifdef CONFIG_X86_32
-/* cpu data as detected by the assembly code in head.S */
-struct cpuinfo_x86 new_cpu_data = {
- .wp_works_ok = -1,
-};
+/* cpu data as detected by the assembly code in head_32.S */
+struct cpuinfo_x86 new_cpu_data;
+
/* common cpu data for all cpus */
-struct cpuinfo_x86 boot_cpu_data __read_mostly = {
- .wp_works_ok = -1,
-};
+struct cpuinfo_x86 boot_cpu_data __read_mostly;
EXPORT_SYMBOL(boot_cpu_data);
unsigned int def_to_bigsmp;
@@ -426,7 +424,7 @@
switch (data_type) {
case SETUP_E820_EXT:
- parse_e820_ext(pa_data, data_len);
+ e820__memory_setup_extended(pa_data, data_len);
break;
case SETUP_DTB:
add_dtb(pa_data);
@@ -441,29 +439,6 @@
}
}
-static void __init e820_reserve_setup_data(void)
-{
- struct setup_data *data;
- u64 pa_data;
-
- pa_data = boot_params.hdr.setup_data;
- if (!pa_data)
- return;
-
- while (pa_data) {
- data = early_memremap(pa_data, sizeof(*data));
- e820_update_range(pa_data, sizeof(*data)+data->len,
- E820_RAM, E820_RESERVED_KERN);
- pa_data = data->next;
- early_memunmap(data, sizeof(*data));
- }
-
- sanitize_e820_map(e820->map, ARRAY_SIZE(e820->map), &e820->nr_map);
- memcpy(e820_saved, e820, sizeof(struct e820map));
- printk(KERN_INFO "extended physical RAM map:\n");
- e820_print_map("reserve setup_data");
-}
-
static void __init memblock_x86_reserve_range_setup_data(void)
{
struct setup_data *data;
@@ -756,16 +731,16 @@
* since some BIOSes are known to corrupt low memory. See the
* Kconfig help text for X86_RESERVE_LOW.
*/
- e820_update_range(0, PAGE_SIZE, E820_RAM, E820_RESERVED);
+ e820__range_update(0, PAGE_SIZE, E820_TYPE_RAM, E820_TYPE_RESERVED);
/*
* special case: Some BIOSen report the PC BIOS
* area (640->1Mb) as ram even though it is not.
* take them out.
*/
- e820_remove_range(BIOS_BEGIN, BIOS_END - BIOS_BEGIN, E820_RAM, 1);
+ e820__range_remove(BIOS_BEGIN, BIOS_END - BIOS_BEGIN, E820_TYPE_RAM, 1);
- sanitize_e820_map(e820->map, ARRAY_SIZE(e820->map), &e820->nr_map);
+ e820__update_table(e820_table);
}
/* called before trim_bios_range() to spare extra sanitize */
@@ -775,18 +750,18 @@
u64 size = __pa_symbol(_end) - start;
/*
- * Complain if .text .data and .bss are not marked as E820_RAM and
+ * Complain if .text .data and .bss are not marked as E820_TYPE_RAM and
* attempt to fix it by adding the range. We may have a confused BIOS,
* or the user may have used memmap=exactmap or memmap=xxM$yyM to
* exclude kernel range. If we really are running on top non-RAM,
* we will crash later anyways.
*/
- if (e820_all_mapped(start, start + size, E820_RAM))
+ if (e820__mapped_all(start, start + size, E820_TYPE_RAM))
return;
- pr_warn(".text .data .bss are not marked as E820_RAM!\n");
- e820_remove_range(start, size, E820_RAM, 0);
- e820_add_region(start, size, E820_RAM);
+ pr_warn(".text .data .bss are not marked as E820_TYPE_RAM!\n");
+ e820__range_remove(start, size, E820_TYPE_RAM, 0);
+ e820__range_add(start, size, E820_TYPE_RAM);
}
static unsigned reserve_low = CONFIG_X86_RESERVE_LOW << 10;
@@ -837,6 +812,26 @@
return 0;
}
+static void __init simple_udelay_calibration(void)
+{
+ unsigned int tsc_khz, cpu_khz;
+ unsigned long lpj;
+
+ if (!boot_cpu_has(X86_FEATURE_TSC))
+ return;
+
+ cpu_khz = x86_platform.calibrate_cpu();
+ tsc_khz = x86_platform.calibrate_tsc();
+
+ tsc_khz = tsc_khz ? : cpu_khz;
+ if (!tsc_khz)
+ return;
+
+ lpj = tsc_khz * 1000;
+ do_div(lpj, HZ);
+ loops_per_jiffy = lpj;
+}
+
/*
* Determine if we were loaded by an EFI loader. If so, then we have also been
* passed the efi memmap, systab, etc., so we should use these data structures
@@ -939,7 +934,7 @@
x86_init.oem.arch_setup();
iomem_resource.end = (1ULL << boot_cpu_data.x86_phys_bits) - 1;
- setup_memory_map();
+ e820__memory_setup();
parse_setup_data();
copy_edd();
@@ -985,6 +980,8 @@
*/
x86_configure_nx();
+ simple_udelay_calibration();
+
parse_early_param();
#ifdef CONFIG_MEMORY_HOTPLUG
@@ -1028,9 +1025,8 @@
early_dump_pci_devices();
#endif
- /* update the e820_saved too */
- e820_reserve_setup_data();
- finish_e820_parsing();
+ e820__reserve_setup_data();
+ e820__finish_early_params();
if (efi_enabled(EFI_BOOT))
efi_init();
@@ -1056,11 +1052,11 @@
trim_bios_range();
#ifdef CONFIG_X86_32
if (ppro_with_ram_bug()) {
- e820_update_range(0x70000000ULL, 0x40000ULL, E820_RAM,
- E820_RESERVED);
- sanitize_e820_map(e820->map, ARRAY_SIZE(e820->map), &e820->nr_map);
+ e820__range_update(0x70000000ULL, 0x40000ULL, E820_TYPE_RAM,
+ E820_TYPE_RESERVED);
+ e820__update_table(e820_table);
printk(KERN_INFO "fixed physical RAM map:\n");
- e820_print_map("bad_ppro");
+ e820__print_table("bad_ppro");
}
#else
early_gart_iommu_check();
@@ -1070,12 +1066,12 @@
* partially used pages are not usable - thus
* we are rounding upwards:
*/
- max_pfn = e820_end_of_ram_pfn();
+ max_pfn = e820__end_of_ram_pfn();
/* update e820 for memory not covered by WB MTRRs */
mtrr_bp_init();
if (mtrr_trim_uncached_memory(max_pfn))
- max_pfn = e820_end_of_ram_pfn();
+ max_pfn = e820__end_of_ram_pfn();
max_possible_pfn = max_pfn;
@@ -1094,7 +1090,7 @@
/* How many end-of-memory variables you have, grandma! */
/* need this before calling reserve_initrd */
if (max_pfn > (1UL<<(32 - PAGE_SHIFT)))
- max_low_pfn = e820_end_of_low_ram_pfn();
+ max_low_pfn = e820__end_of_low_ram_pfn();
else
max_low_pfn = max_pfn;
@@ -1111,7 +1107,7 @@
early_alloc_pgt_buf();
/*
- * Need to conclude brk, before memblock_x86_fill()
+ * Need to conclude brk, before e820__memblock_setup()
* it could use memblock_find_in_range, could overlap with
* brk area.
*/
@@ -1120,7 +1116,10 @@
cleanup_highmap();
memblock_set_current_limit(ISA_END_ADDRESS);
- memblock_x86_fill();
+ e820__memblock_setup();
+
+ if (!early_xdbc_setup_hardware())
+ early_xdbc_register_console();
reserve_bios_regions();
@@ -1137,7 +1136,7 @@
}
/* preallocate 4k for mptable mpc */
- early_reserve_e820_mpc_new();
+ e820__memblock_alloc_reserved_mpc_new();
#ifdef CONFIG_X86_CHECK_BIOS_CORRUPTION
setup_bios_corruption_check();
@@ -1226,21 +1225,6 @@
kasan_init();
-#ifdef CONFIG_X86_32
- /* sync back kernel address range */
- clone_pgd_range(initial_page_table + KERNEL_PGD_BOUNDARY,
- swapper_pg_dir + KERNEL_PGD_BOUNDARY,
- KERNEL_PGD_PTRS);
-
- /*
- * sync back low identity map too. It is used for example
- * in the 32-bit EFI stub.
- */
- clone_pgd_range(initial_page_table,
- swapper_pg_dir + KERNEL_PGD_BOUNDARY,
- min(KERNEL_PGD_PTRS, KERNEL_PGD_BOUNDARY));
-#endif
-
tboot_probe();
map_vsyscall();
@@ -1275,12 +1259,12 @@
kvm_guest_init();
- e820_reserve_resources();
- e820_mark_nosave_regions(max_low_pfn);
+ e820__reserve_resources();
+ e820__register_nosave_regions(max_low_pfn);
x86_init.resources.reserve_resources();
- e820_setup_gap();
+ e820__setup_pci_gap();
#ifdef CONFIG_VT
#if defined(CONFIG_VGA_CONSOLE)
diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c
index 9820d6d..bb1e8cc 100644
--- a/arch/x86/kernel/setup_percpu.c
+++ b/arch/x86/kernel/setup_percpu.c
@@ -160,7 +160,7 @@
pack_descriptor(&gdt, per_cpu_offset(cpu), 0xFFFFF,
0x2 | DESCTYPE_S, 0x8);
gdt.s = 1;
- write_gdt_entry(get_cpu_gdt_table(cpu),
+ write_gdt_entry(get_cpu_gdt_rw(cpu),
GDT_ENTRY_PERCPU, &gdt, DESCTYPE_S);
#endif
}
@@ -288,4 +288,25 @@
/* Setup cpu initialized, callin, callout masks */
setup_cpu_local_masks();
+
+#ifdef CONFIG_X86_32
+ /*
+ * Sync back kernel address range. We want to make sure that
+ * all kernel mappings, including percpu mappings, are available
+ * in the smpboot asm. We can't reliably pick up percpu
+ * mappings using vmalloc_fault(), because exception dispatch
+ * needs percpu data.
+ */
+ clone_pgd_range(initial_page_table + KERNEL_PGD_BOUNDARY,
+ swapper_pg_dir + KERNEL_PGD_BOUNDARY,
+ KERNEL_PGD_PTRS);
+
+ /*
+ * sync back low identity map too. It is used for example
+ * in the 32-bit EFI stub.
+ */
+ clone_pgd_range(initial_page_table,
+ swapper_pg_dir + KERNEL_PGD_BOUNDARY,
+ min(KERNEL_PGD_PTRS, KERNEL_PGD_BOUNDARY));
+#endif
}
diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c
index d3c66a1..d798c0d 100644
--- a/arch/x86/kernel/smp.c
+++ b/arch/x86/kernel/smp.c
@@ -33,6 +33,7 @@
#include <asm/mce.h>
#include <asm/trace/irq_vectors.h>
#include <asm/kexec.h>
+#include <asm/virtext.h>
/*
* Some notes on x86 processor bugs affecting SMP operation:
@@ -124,7 +125,7 @@
static void native_smp_send_reschedule(int cpu)
{
if (unlikely(cpu_is_offline(cpu))) {
- WARN_ON(1);
+ WARN(1, "sched: Unexpected reschedule of offline CPU#%d!\n", cpu);
return;
}
apic->send_IPI(cpu, RESCHEDULE_VECTOR);
@@ -162,6 +163,7 @@
if (raw_smp_processor_id() == atomic_read(&stopping_cpu))
return NMI_HANDLED;
+ cpu_emergency_vmxoff();
stop_this_cpu(NULL);
return NMI_HANDLED;
@@ -174,6 +176,7 @@
asmlinkage __visible void smp_reboot_interrupt(void)
{
ipi_entering_ack_irq();
+ cpu_emergency_vmxoff();
stop_this_cpu(NULL);
irq_exit();
}
diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index bd1f1ad..f04479a 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -983,7 +983,7 @@
unsigned long timeout;
idle->thread.sp = (unsigned long)task_pt_regs(idle);
- early_gdt_descr.address = (unsigned long)get_cpu_gdt_table(cpu);
+ early_gdt_descr.address = (unsigned long)get_cpu_gdt_rw(cpu);
initial_code = (unsigned long)start_secondary;
initial_stack = idle->thread.sp;
diff --git a/arch/x86/kernel/stacktrace.c b/arch/x86/kernel/stacktrace.c
index 8e2b79b..8dabd7b 100644
--- a/arch/x86/kernel/stacktrace.c
+++ b/arch/x86/kernel/stacktrace.c
@@ -76,6 +76,101 @@
}
EXPORT_SYMBOL_GPL(save_stack_trace_tsk);
+#ifdef CONFIG_HAVE_RELIABLE_STACKTRACE
+
+#define STACKTRACE_DUMP_ONCE(task) ({ \
+ static bool __section(.data.unlikely) __dumped; \
+ \
+ if (!__dumped) { \
+ __dumped = true; \
+ WARN_ON(1); \
+ show_stack(task, NULL); \
+ } \
+})
+
+static int __save_stack_trace_reliable(struct stack_trace *trace,
+ struct task_struct *task)
+{
+ struct unwind_state state;
+ struct pt_regs *regs;
+ unsigned long addr;
+
+ for (unwind_start(&state, task, NULL, NULL); !unwind_done(&state);
+ unwind_next_frame(&state)) {
+
+ regs = unwind_get_entry_regs(&state);
+ if (regs) {
+ /*
+ * Kernel mode registers on the stack indicate an
+ * in-kernel interrupt or exception (e.g., preemption
+ * or a page fault), which can make frame pointers
+ * unreliable.
+ */
+ if (!user_mode(regs))
+ return -EINVAL;
+
+ /*
+ * The last frame contains the user mode syscall
+ * pt_regs. Skip it and finish the unwind.
+ */
+ unwind_next_frame(&state);
+ if (!unwind_done(&state)) {
+ STACKTRACE_DUMP_ONCE(task);
+ return -EINVAL;
+ }
+ break;
+ }
+
+ addr = unwind_get_return_address(&state);
+
+ /*
+ * A NULL or invalid return address probably means there's some
+ * generated code which __kernel_text_address() doesn't know
+ * about.
+ */
+ if (!addr) {
+ STACKTRACE_DUMP_ONCE(task);
+ return -EINVAL;
+ }
+
+ if (save_stack_address(trace, addr, false))
+ return -EINVAL;
+ }
+
+ /* Check for stack corruption */
+ if (unwind_error(&state)) {
+ STACKTRACE_DUMP_ONCE(task);
+ return -EINVAL;
+ }
+
+ if (trace->nr_entries < trace->max_entries)
+ trace->entries[trace->nr_entries++] = ULONG_MAX;
+
+ return 0;
+}
+
+/*
+ * This function returns an error if it detects any unreliable features of the
+ * stack. Otherwise it guarantees that the stack trace is reliable.
+ *
+ * If the task is not 'current', the caller *must* ensure the task is inactive.
+ */
+int save_stack_trace_tsk_reliable(struct task_struct *tsk,
+ struct stack_trace *trace)
+{
+ int ret;
+
+ if (!try_get_task_stack(tsk))
+ return -EINVAL;
+
+ ret = __save_stack_trace_reliable(trace, tsk);
+
+ put_task_stack(tsk);
+
+ return ret;
+}
+#endif /* CONFIG_HAVE_RELIABLE_STACKTRACE */
+
/* Userspace stacktrace - based on kernel/trace/trace_sysprof.c */
struct stack_frame_user {
@@ -138,4 +233,3 @@
if (trace->nr_entries < trace->max_entries)
trace->entries[trace->nr_entries++] = ULONG_MAX;
}
-
diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c
index 50215a4..207b8f2 100644
--- a/arch/x86/kernel/sys_x86_64.c
+++ b/arch/x86/kernel/sys_x86_64.c
@@ -17,6 +17,8 @@
#include <linux/uaccess.h>
#include <linux/elf.h>
+#include <asm/elf.h>
+#include <asm/compat.h>
#include <asm/ia32.h>
#include <asm/syscalls.h>
@@ -101,7 +103,7 @@
static void find_start_end(unsigned long flags, unsigned long *begin,
unsigned long *end)
{
- if (!test_thread_flag(TIF_ADDR32) && (flags & MAP_32BIT)) {
+ if (!in_compat_syscall() && (flags & MAP_32BIT)) {
/* This is usually used needed to map code in small
model, so it needs to be in the first 31bit. Limit
it to that. This means we need to move the
@@ -114,10 +116,11 @@
if (current->flags & PF_RANDOMIZE) {
*begin = randomize_page(*begin, 0x02000000);
}
- } else {
- *begin = current->mm->mmap_legacy_base;
- *end = TASK_SIZE;
+ return;
}
+
+ *begin = get_mmap_base(1);
+ *end = in_compat_syscall() ? tasksize_32bit() : tasksize_64bit();
}
unsigned long
@@ -176,7 +179,7 @@
return addr;
/* for MAP_32BIT mappings we force the legacy mmap base */
- if (!test_thread_flag(TIF_ADDR32) && (flags & MAP_32BIT))
+ if (!in_compat_syscall() && (flags & MAP_32BIT))
goto bottomup;
/* requesting a specific address */
@@ -191,7 +194,7 @@
info.flags = VM_UNMAPPED_AREA_TOPDOWN;
info.length = len;
info.low_limit = PAGE_SIZE;
- info.high_limit = mm->mmap_base;
+ info.high_limit = get_mmap_base(0);
info.align_mask = 0;
info.align_offset = pgoff << PAGE_SHIFT;
if (filp) {
diff --git a/arch/x86/kernel/tboot.c b/arch/x86/kernel/tboot.c
index b868fa1..d4c8011 100644
--- a/arch/x86/kernel/tboot.c
+++ b/arch/x86/kernel/tboot.c
@@ -42,7 +42,7 @@
#include <asm/fixmap.h>
#include <asm/proto.h>
#include <asm/setup.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/io.h>
#include "../realmode/rm/wakeup.h"
@@ -68,9 +68,9 @@
* also verify that it is mapped as we expect it before calling
* set_fixmap(), to reduce chance of garbage value causing crash
*/
- if (!e820_any_mapped(boot_params.tboot_addr,
- boot_params.tboot_addr, E820_RESERVED)) {
- pr_warning("non-0 tboot_addr but it is not of type E820_RESERVED\n");
+ if (!e820__mapped_any(boot_params.tboot_addr,
+ boot_params.tboot_addr, E820_TYPE_RESERVED)) {
+ pr_warning("non-0 tboot_addr but it is not of type E820_TYPE_RESERVED\n");
return;
}
@@ -118,12 +118,16 @@
pgprot_t prot)
{
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
pgd = pgd_offset(&tboot_mm, vaddr);
- pud = pud_alloc(&tboot_mm, pgd, vaddr);
+ p4d = p4d_alloc(&tboot_mm, pgd, vaddr);
+ if (!p4d)
+ return -1;
+ pud = pud_alloc(&tboot_mm, p4d, vaddr);
if (!pud)
return -1;
pmd = pmd_alloc(&tboot_mm, pud, vaddr);
@@ -188,12 +192,12 @@
tboot->num_mac_regions = 0;
- for (i = 0; i < e820->nr_map; i++) {
- if ((e820->map[i].type != E820_RAM)
- && (e820->map[i].type != E820_RESERVED_KERN))
+ for (i = 0; i < e820_table->nr_entries; i++) {
+ if ((e820_table->entries[i].type != E820_TYPE_RAM)
+ && (e820_table->entries[i].type != E820_TYPE_RESERVED_KERN))
continue;
- add_mac_region(e820->map[i].addr, e820->map[i].size);
+ add_mac_region(e820_table->entries[i].addr, e820_table->entries[i].size);
}
tboot->acpi_sinfo.kernel_s3_resume_vector =
diff --git a/arch/x86/kernel/tls.c b/arch/x86/kernel/tls.c
index 6c89344..dcd699b 100644
--- a/arch/x86/kernel/tls.c
+++ b/arch/x86/kernel/tls.c
@@ -92,10 +92,17 @@
cpu = get_cpu();
while (n-- > 0) {
- if (LDT_empty(info) || LDT_zero(info))
+ if (LDT_empty(info) || LDT_zero(info)) {
desc->a = desc->b = 0;
- else
+ } else {
fill_ldt(desc, info);
+
+ /*
+ * Always set the accessed bit so that the CPU
+ * doesn't try to write to the (read-only) GDT.
+ */
+ desc->type |= 1;
+ }
++info;
++desc;
}
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 4e49637..3995d3a 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -169,6 +169,37 @@
preempt_disable();
}
+int is_valid_bugaddr(unsigned long addr)
+{
+ unsigned short ud;
+
+ if (addr < TASK_SIZE_MAX)
+ return 0;
+
+ if (probe_kernel_address((unsigned short *)addr, ud))
+ return 0;
+
+ return ud == INSN_UD0 || ud == INSN_UD2;
+}
+
+static int fixup_bug(struct pt_regs *regs, int trapnr)
+{
+ if (trapnr != X86_TRAP_UD)
+ return 0;
+
+ switch (report_bug(regs->ip, regs)) {
+ case BUG_TRAP_TYPE_NONE:
+ case BUG_TRAP_TYPE_BUG:
+ break;
+
+ case BUG_TRAP_TYPE_WARN:
+ regs->ip += LEN_UD0;
+ return 1;
+ }
+
+ return 0;
+}
+
static nokprobe_inline int
do_trap_no_signal(struct task_struct *tsk, int trapnr, char *str,
struct pt_regs *regs, long error_code)
@@ -187,12 +218,15 @@
}
if (!user_mode(regs)) {
- if (!fixup_exception(regs, trapnr)) {
- tsk->thread.error_code = error_code;
- tsk->thread.trap_nr = trapnr;
- die(str, regs, error_code);
- }
- return 0;
+ if (fixup_exception(regs, trapnr))
+ return 0;
+
+ if (fixup_bug(regs, trapnr))
+ return 0;
+
+ tsk->thread.error_code = error_code;
+ tsk->thread.trap_nr = trapnr;
+ die(str, regs, error_code);
}
return -1;
diff --git a/arch/x86/kernel/unwind_frame.c b/arch/x86/kernel/unwind_frame.c
index 0833926..82c6d7f 100644
--- a/arch/x86/kernel/unwind_frame.c
+++ b/arch/x86/kernel/unwind_frame.c
@@ -1,6 +1,8 @@
#include <linux/sched.h>
#include <linux/sched/task.h>
#include <linux/sched/task_stack.h>
+#include <linux/interrupt.h>
+#include <asm/sections.h>
#include <asm/ptrace.h>
#include <asm/bitops.h>
#include <asm/stacktrace.h>
@@ -23,53 +25,53 @@
val; \
})
-static void unwind_dump(struct unwind_state *state, unsigned long *sp)
+static void unwind_dump(struct unwind_state *state)
{
static bool dumped_before = false;
bool prev_zero, zero = false;
- unsigned long word;
+ unsigned long word, *sp;
+ struct stack_info stack_info = {0};
+ unsigned long visit_mask = 0;
if (dumped_before)
return;
dumped_before = true;
- printk_deferred("unwind stack type:%d next_sp:%p mask:%lx graph_idx:%d\n",
+ printk_deferred("unwind stack type:%d next_sp:%p mask:0x%lx graph_idx:%d\n",
state->stack_info.type, state->stack_info.next_sp,
state->stack_mask, state->graph_idx);
- for (sp = state->orig_sp; sp < state->stack_info.end; sp++) {
- word = READ_ONCE_NOCHECK(*sp);
+ for (sp = state->orig_sp; sp; sp = PTR_ALIGN(stack_info.next_sp, sizeof(long))) {
+ if (get_stack_info(sp, state->task, &stack_info, &visit_mask))
+ break;
- prev_zero = zero;
- zero = word == 0;
+ for (; sp < stack_info.end; sp++) {
- if (zero) {
- if (!prev_zero)
- printk_deferred("%p: %016x ...\n", sp, 0);
- continue;
+ word = READ_ONCE_NOCHECK(*sp);
+
+ prev_zero = zero;
+ zero = word == 0;
+
+ if (zero) {
+ if (!prev_zero)
+ printk_deferred("%p: %0*x ...\n",
+ sp, BITS_PER_LONG/4, 0);
+ continue;
+ }
+
+ printk_deferred("%p: %0*lx (%pB)\n",
+ sp, BITS_PER_LONG/4, word, (void *)word);
}
-
- printk_deferred("%p: %016lx (%pB)\n", sp, word, (void *)word);
}
}
unsigned long unwind_get_return_address(struct unwind_state *state)
{
- unsigned long addr;
- unsigned long *addr_p = unwind_get_return_address_ptr(state);
-
if (unwind_done(state))
return 0;
- if (state->regs && user_mode(state->regs))
- return 0;
-
- addr = READ_ONCE_TASK_STACK(state->task, *addr_p);
- addr = ftrace_graph_ret_addr(state->task, &state->graph_idx, addr,
- addr_p);
-
- return __kernel_text_address(addr) ? addr : 0;
+ return __kernel_text_address(state->ip) ? state->ip : 0;
}
EXPORT_SYMBOL_GPL(unwind_get_return_address);
@@ -82,16 +84,41 @@
return sizeof(*regs);
}
+static bool in_entry_code(unsigned long ip)
+{
+ char *addr = (char *)ip;
+
+ if (addr >= __entry_text_start && addr < __entry_text_end)
+ return true;
+
+#if defined(CONFIG_FUNCTION_GRAPH_TRACER) || defined(CONFIG_KASAN)
+ if (addr >= __irqentry_text_start && addr < __irqentry_text_end)
+ return true;
+#endif
+
+ return false;
+}
+
+static inline unsigned long *last_frame(struct unwind_state *state)
+{
+ return (unsigned long *)task_pt_regs(state->task) - 2;
+}
+
#ifdef CONFIG_X86_32
#define GCC_REALIGN_WORDS 3
#else
#define GCC_REALIGN_WORDS 1
#endif
+static inline unsigned long *last_aligned_frame(struct unwind_state *state)
+{
+ return last_frame(state) - GCC_REALIGN_WORDS;
+}
+
static bool is_last_task_frame(struct unwind_state *state)
{
- unsigned long *last_bp = (unsigned long *)task_pt_regs(state->task) - 2;
- unsigned long *aligned_bp = last_bp - GCC_REALIGN_WORDS;
+ unsigned long *last_bp = last_frame(state);
+ unsigned long *aligned_bp = last_aligned_frame(state);
/*
* We have to check for the last task frame at two different locations
@@ -135,26 +162,70 @@
return (struct pt_regs *)(regs & ~0x1);
}
-static bool update_stack_state(struct unwind_state *state, void *addr,
- size_t len)
+static bool update_stack_state(struct unwind_state *state,
+ unsigned long *next_bp)
{
struct stack_info *info = &state->stack_info;
- enum stack_type orig_type = info->type;
+ enum stack_type prev_type = info->type;
+ struct pt_regs *regs;
+ unsigned long *frame, *prev_frame_end, *addr_p, addr;
+ size_t len;
+
+ if (state->regs)
+ prev_frame_end = (void *)state->regs + regs_size(state->regs);
+ else
+ prev_frame_end = (void *)state->bp + FRAME_HEADER_SIZE;
+
+ /* Is the next frame pointer an encoded pointer to pt_regs? */
+ regs = decode_frame_pointer(next_bp);
+ if (regs) {
+ frame = (unsigned long *)regs;
+ len = regs_size(regs);
+ state->got_irq = true;
+ } else {
+ frame = next_bp;
+ len = FRAME_HEADER_SIZE;
+ }
/*
- * If addr isn't on the current stack, switch to the next one.
+ * If the next bp isn't on the current stack, switch to the next one.
*
* We may have to traverse multiple stacks to deal with the possibility
- * that 'info->next_sp' could point to an empty stack and 'addr' could
- * be on a subsequent stack.
+ * that info->next_sp could point to an empty stack and the next bp
+ * could be on a subsequent stack.
*/
- while (!on_stack(info, addr, len))
+ while (!on_stack(info, frame, len))
if (get_stack_info(info->next_sp, state->task, info,
&state->stack_mask))
return false;
- if (!state->orig_sp || info->type != orig_type)
- state->orig_sp = addr;
+ /* Make sure it only unwinds up and doesn't overlap the prev frame: */
+ if (state->orig_sp && state->stack_info.type == prev_type &&
+ frame < prev_frame_end)
+ return false;
+
+ /* Move state to the next frame: */
+ if (regs) {
+ state->regs = regs;
+ state->bp = NULL;
+ } else {
+ state->bp = next_bp;
+ state->regs = NULL;
+ }
+
+ /* Save the return address: */
+ if (state->regs && user_mode(state->regs))
+ state->ip = 0;
+ else {
+ addr_p = unwind_get_return_address_ptr(state);
+ addr = READ_ONCE_TASK_STACK(state->task, *addr_p);
+ state->ip = ftrace_graph_ret_addr(state->task, &state->graph_idx,
+ addr, addr_p);
+ }
+
+ /* Save the original stack pointer for unwind_dump(): */
+ if (!state->orig_sp)
+ state->orig_sp = frame;
return true;
}
@@ -162,14 +233,12 @@
bool unwind_next_frame(struct unwind_state *state)
{
struct pt_regs *regs;
- unsigned long *next_bp, *next_frame;
- size_t next_len;
- enum stack_type prev_type = state->stack_info.type;
+ unsigned long *next_bp;
if (unwind_done(state))
return false;
- /* have we reached the end? */
+ /* Have we reached the end? */
if (state->regs && user_mode(state->regs))
goto the_end;
@@ -197,58 +266,25 @@
*/
state->regs = regs;
state->bp = NULL;
+ state->ip = 0;
return true;
}
- /* get the next frame pointer */
+ /* Get the next frame pointer: */
if (state->regs)
next_bp = (unsigned long *)state->regs->bp;
else
- next_bp = (unsigned long *)READ_ONCE_TASK_STACK(state->task,*state->bp);
+ next_bp = (unsigned long *)READ_ONCE_TASK_STACK(state->task, *state->bp);
- /* is the next frame pointer an encoded pointer to pt_regs? */
- regs = decode_frame_pointer(next_bp);
- if (regs) {
- next_frame = (unsigned long *)regs;
- next_len = sizeof(*regs);
- } else {
- next_frame = next_bp;
- next_len = FRAME_HEADER_SIZE;
- }
-
- /* make sure the next frame's data is accessible */
- if (!update_stack_state(state, next_frame, next_len)) {
- /*
- * Don't warn on bad regs->bp. An interrupt in entry code
- * might cause a false positive warning.
- */
- if (state->regs)
- goto the_end;
-
+ /* Move to the next frame if it's safe: */
+ if (!update_stack_state(state, next_bp))
goto bad_address;
- }
-
- /* Make sure it only unwinds up and doesn't overlap the last frame: */
- if (state->stack_info.type == prev_type) {
- if (state->regs && (void *)next_frame < (void *)state->regs + regs_size(state->regs))
- goto bad_address;
-
- if (state->bp && (void *)next_frame < (void *)state->bp + FRAME_HEADER_SIZE)
- goto bad_address;
- }
-
- /* move to the next frame */
- if (regs) {
- state->regs = regs;
- state->bp = NULL;
- } else {
- state->bp = next_bp;
- state->regs = NULL;
- }
return true;
bad_address:
+ state->error = true;
+
/*
* When unwinding a non-current task, the task might actually be
* running on another CPU, in which case it could be modifying its
@@ -259,18 +295,29 @@
if (state->task != current)
goto the_end;
+ /*
+ * Don't warn if the unwinder got lost due to an interrupt in entry
+ * code or in the C handler before the first frame pointer got set up:
+ */
+ if (state->got_irq && in_entry_code(state->ip))
+ goto the_end;
+ if (state->regs &&
+ state->regs->sp >= (unsigned long)last_aligned_frame(state) &&
+ state->regs->sp < (unsigned long)task_pt_regs(state->task))
+ goto the_end;
+
if (state->regs) {
printk_deferred_once(KERN_WARNING
"WARNING: kernel stack regs at %p in %s:%d has bad 'bp' value %p\n",
state->regs, state->task->comm,
- state->task->pid, next_frame);
- unwind_dump(state, (unsigned long *)state->regs);
+ state->task->pid, next_bp);
+ unwind_dump(state);
} else {
printk_deferred_once(KERN_WARNING
"WARNING: kernel stack frame pointer at %p in %s:%d has bad value %p\n",
state->bp, state->task->comm,
- state->task->pid, next_frame);
- unwind_dump(state, state->bp);
+ state->task->pid, next_bp);
+ unwind_dump(state);
}
the_end:
state->stack_info.type = STACK_TYPE_UNKNOWN;
@@ -281,35 +328,24 @@
void __unwind_start(struct unwind_state *state, struct task_struct *task,
struct pt_regs *regs, unsigned long *first_frame)
{
- unsigned long *bp, *frame;
- size_t len;
+ unsigned long *bp;
memset(state, 0, sizeof(*state));
state->task = task;
+ state->got_irq = (regs);
- /* don't even attempt to start from user mode regs */
+ /* Don't even attempt to start from user mode regs: */
if (regs && user_mode(regs)) {
state->stack_info.type = STACK_TYPE_UNKNOWN;
return;
}
- /* set up the starting stack frame */
bp = get_frame_pointer(task, regs);
- regs = decode_frame_pointer(bp);
- if (regs) {
- state->regs = regs;
- frame = (unsigned long *)regs;
- len = sizeof(*regs);
- } else {
- state->bp = bp;
- frame = bp;
- len = FRAME_HEADER_SIZE;
- }
- /* initialize stack info and make sure the frame data is accessible */
- get_stack_info(frame, state->task, &state->stack_info,
+ /* Initialize stack info and make sure the frame data is accessible: */
+ get_stack_info(bp, state->task, &state->stack_info,
&state->stack_mask);
- update_stack_state(state, frame, len);
+ update_stack_state(state, bp);
/*
* The caller can provide the address of the first frame directly
diff --git a/arch/x86/kernel/unwind_guess.c b/arch/x86/kernel/unwind_guess.c
index 22881dd..039f367 100644
--- a/arch/x86/kernel/unwind_guess.c
+++ b/arch/x86/kernel/unwind_guess.c
@@ -34,7 +34,7 @@
return true;
}
- state->sp = info->next_sp;
+ state->sp = PTR_ALIGN(info->next_sp, sizeof(long));
} while (!get_stack_info(state->sp, state->task, info,
&state->stack_mask));
@@ -49,7 +49,7 @@
memset(state, 0, sizeof(*state));
state->task = task;
- state->sp = first_frame;
+ state->sp = PTR_ALIGN(first_frame, sizeof(long));
get_stack_info(first_frame, state->task, &state->stack_info,
&state->stack_mask);
diff --git a/arch/x86/kernel/vm86_32.c b/arch/x86/kernel/vm86_32.c
index 23ee89c..7924a53 100644
--- a/arch/x86/kernel/vm86_32.c
+++ b/arch/x86/kernel/vm86_32.c
@@ -164,6 +164,7 @@
struct vm_area_struct *vma;
spinlock_t *ptl;
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
@@ -173,7 +174,10 @@
pgd = pgd_offset(mm, 0xA0000);
if (pgd_none_or_clear_bad(pgd))
goto out;
- pud = pud_offset(pgd, 0xA0000);
+ p4d = p4d_offset(pgd, 0xA0000);
+ if (p4d_none_or_clear_bad(p4d))
+ goto out;
+ pud = pud_offset(p4d, 0xA0000);
if (pud_none_or_clear_bad(pud))
goto out;
pmd = pmd_offset(pud, 0xA0000);
@@ -193,7 +197,7 @@
pte_unmap_unlock(pte, ptl);
out:
up_write(&mm->mmap_sem);
- flush_tlb();
+ flush_tlb_mm_range(mm, 0xA0000, 0xA0000 + 32*PAGE_SIZE, 0UL);
}
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index c74ae9c..c8a3b61 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -146,6 +146,7 @@
_edata = .;
} :data
+ BUG_TABLE
. = ALIGN(PAGE_SIZE);
__vvar_page = .;
diff --git a/arch/x86/kernel/x86_init.c b/arch/x86/kernel/x86_init.c
index 11a93f0..a088b2c 100644
--- a/arch/x86/kernel/x86_init.c
+++ b/arch/x86/kernel/x86_init.c
@@ -14,7 +14,7 @@
#include <asm/mpspec.h>
#include <asm/setup.h>
#include <asm/apic.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/time.h>
#include <asm/irq.h>
#include <asm/io_apic.h>
@@ -38,7 +38,7 @@
.resources = {
.probe_roms = probe_roms,
.reserve_resources = reserve_standard_io_resources,
- .memory_setup = default_machine_specific_memory_setup,
+ .memory_setup = e820__memory_setup_default,
},
.mpparse = {
diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c
index 5fba706..5f48f62 100644
--- a/arch/x86/kvm/svm.c
+++ b/arch/x86/kvm/svm.c
@@ -741,7 +741,6 @@
struct svm_cpu_data *sd;
uint64_t efer;
- struct desc_ptr gdt_descr;
struct desc_struct *gdt;
int me = raw_smp_processor_id();
@@ -763,8 +762,7 @@
sd->max_asid = cpuid_ebx(SVM_CPUID_FUNC) - 1;
sd->next_asid = sd->max_asid + 1;
- native_store_gdt(&gdt_descr);
- gdt = (struct desc_struct *)gdt_descr.address;
+ gdt = get_current_gdt_rw();
sd->tss_desc = (struct kvm_ldttss_desc *)(gdt + GDT_ENTRY_TSS);
wrmsrl(MSR_EFER, efer | EFER_SVME);
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index 259e9b2..1a471e5 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -935,7 +935,6 @@
* when a CPU is brought down, and we need to VMCLEAR all VMCSs loaded on it.
*/
static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu);
-static DEFINE_PER_CPU(struct desc_ptr, host_gdt);
/*
* We maintian a per-CPU linked-list of vCPU, so in wakeup_handler() we
@@ -2057,14 +2056,13 @@
*/
static unsigned long segment_base(u16 selector)
{
- struct desc_ptr *gdt = this_cpu_ptr(&host_gdt);
struct desc_struct *table;
unsigned long v;
if (!(selector & ~SEGMENT_RPL_MASK))
return 0;
- table = (struct desc_struct *)gdt->address;
+ table = get_current_gdt_ro();
if ((selector & SEGMENT_TI_MASK) == SEGMENT_LDT) {
u16 ldt_selector = kvm_read_ldt();
@@ -2169,7 +2167,7 @@
#endif
if (vmx->host_state.msr_host_bndcfgs)
wrmsrl(MSR_IA32_BNDCFGS, vmx->host_state.msr_host_bndcfgs);
- load_gdt(this_cpu_ptr(&host_gdt));
+ load_fixmap_gdt(raw_smp_processor_id());
}
static void vmx_load_host_state(struct vcpu_vmx *vmx)
@@ -2271,7 +2269,7 @@
}
if (!already_loaded) {
- struct desc_ptr *gdt = this_cpu_ptr(&host_gdt);
+ void *gdt = get_current_gdt_ro();
unsigned long sysenter_esp;
kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu);
@@ -2282,7 +2280,7 @@
*/
vmcs_writel(HOST_TR_BASE,
(unsigned long)this_cpu_ptr(&cpu_tss));
- vmcs_writel(HOST_GDTR_BASE, gdt->address);
+ vmcs_writel(HOST_GDTR_BASE, (unsigned long)gdt); /* 22.2.4 */
/*
* VM exits change the host TR limit to 0x67 after a VM
@@ -3471,8 +3469,6 @@
ept_sync_global();
}
- native_store_gdt(this_cpu_ptr(&host_gdt));
-
return 0;
}
diff --git a/arch/x86/lguest/boot.c b/arch/x86/lguest/boot.c
index d3289d7..9947269 100644
--- a/arch/x86/lguest/boot.c
+++ b/arch/x86/lguest/boot.c
@@ -67,7 +67,7 @@
#include <asm/pgtable.h>
#include <asm/desc.h>
#include <asm/setup.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/mce.h>
#include <asm/io.h>
#include <asm/fpu/api.h>
@@ -994,7 +994,9 @@
.mult = 1,
.shift = 0,
.min_delta_ns = LG_CLOCK_MIN_DELTA,
+ .min_delta_ticks = LG_CLOCK_MIN_DELTA,
.max_delta_ns = LG_CLOCK_MAX_DELTA,
+ .max_delta_ticks = LG_CLOCK_MAX_DELTA,
};
/*
@@ -1178,9 +1180,9 @@
* The Linux bootloader header contains an "e820" memory map: the
* Launcher populated the first entry with our memory limit.
*/
- e820_add_region(boot_params.e820_map[0].addr,
- boot_params.e820_map[0].size,
- boot_params.e820_map[0].type);
+ e820__range_add(boot_params.e820_table[0].addr,
+ boot_params.e820_table[0].size,
+ boot_params.e820_table[0].type);
/* This string is for the boot messages. */
return "LGUEST";
diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S
index 5e2af3a..81b1635 100644
--- a/arch/x86/lib/clear_page_64.S
+++ b/arch/x86/lib/clear_page_64.S
@@ -14,20 +14,15 @@
* Zero a page.
* %rdi - page
*/
-ENTRY(clear_page)
-
- ALTERNATIVE_2 "jmp clear_page_orig", "", X86_FEATURE_REP_GOOD, \
- "jmp clear_page_c_e", X86_FEATURE_ERMS
-
+ENTRY(clear_page_rep)
movl $4096/8,%ecx
xorl %eax,%eax
rep stosq
ret
-ENDPROC(clear_page)
-EXPORT_SYMBOL(clear_page)
+ENDPROC(clear_page_rep)
+EXPORT_SYMBOL_GPL(clear_page_rep)
ENTRY(clear_page_orig)
-
xorl %eax,%eax
movl $4096/64,%ecx
.p2align 4
@@ -47,10 +42,12 @@
nop
ret
ENDPROC(clear_page_orig)
+EXPORT_SYMBOL_GPL(clear_page_orig)
-ENTRY(clear_page_c_e)
+ENTRY(clear_page_erms)
movl $4096,%ecx
xorl %eax,%eax
rep stosb
ret
-ENDPROC(clear_page_c_e)
+ENDPROC(clear_page_erms)
+EXPORT_SYMBOL_GPL(clear_page_erms)
diff --git a/arch/x86/lib/delay.c b/arch/x86/lib/delay.c
index a8e91ae..29df077 100644
--- a/arch/x86/lib/delay.c
+++ b/arch/x86/lib/delay.c
@@ -93,6 +93,13 @@
{
u64 start, end, delay, loops = __loops;
+ /*
+ * Timer value of 0 causes MWAITX to wait indefinitely, unless there
+ * is a store on the memory monitored by MONITORX.
+ */
+ if (loops == 0)
+ return;
+
start = rdtsc_ordered();
for (;;) {
diff --git a/arch/x86/lib/kaslr.c b/arch/x86/lib/kaslr.c
index 121f59c..5761a4f 100644
--- a/arch/x86/lib/kaslr.c
+++ b/arch/x86/lib/kaslr.c
@@ -8,7 +8,7 @@
#include <asm/kaslr.h>
#include <asm/msr.h>
#include <asm/archrandom.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/io.h>
/*
diff --git a/arch/x86/lib/usercopy.c b/arch/x86/lib/usercopy.c
index c074799..c8c6ad0 100644
--- a/arch/x86/lib/usercopy.c
+++ b/arch/x86/lib/usercopy.c
@@ -4,12 +4,9 @@
* For licencing details see kernel-base/COPYING
*/
-#include <linux/highmem.h>
+#include <linux/uaccess.h>
#include <linux/export.h>
-#include <asm/word-at-a-time.h>
-#include <linux/sched.h>
-
/*
* We rely on the nested NMI work to allow atomic faults from the NMI path; the
* nested NMI paths are careful to preserve CR2.
@@ -34,52 +31,3 @@
return ret;
}
EXPORT_SYMBOL_GPL(copy_from_user_nmi);
-
-/**
- * copy_to_user: - Copy a block of data into user space.
- * @to: Destination address, in user space.
- * @from: Source address, in kernel space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from kernel space to user space.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- */
-unsigned long _copy_to_user(void __user *to, const void *from, unsigned n)
-{
- if (access_ok(VERIFY_WRITE, to, n))
- n = __copy_to_user(to, from, n);
- return n;
-}
-EXPORT_SYMBOL(_copy_to_user);
-
-/**
- * copy_from_user: - Copy a block of data from user space.
- * @to: Destination address, in kernel space.
- * @from: Source address, in user space.
- * @n: Number of bytes to copy.
- *
- * Context: User context only. This function may sleep if pagefaults are
- * enabled.
- *
- * Copy data from user space to kernel space.
- *
- * Returns number of bytes that could not be copied.
- * On success, this will be zero.
- *
- * If some data could not be copied, this function will pad the copied
- * data to the requested size using zero bytes.
- */
-unsigned long _copy_from_user(void *to, const void __user *from, unsigned n)
-{
- if (access_ok(VERIFY_READ, from, n))
- n = __copy_from_user(to, from, n);
- else
- memset(to, 0, n);
- return n;
-}
-EXPORT_SYMBOL(_copy_from_user);
diff --git a/arch/x86/lib/usercopy_32.c b/arch/x86/lib/usercopy_32.c
index 1f65ff65..bd057a4 100644
--- a/arch/x86/lib/usercopy_32.c
+++ b/arch/x86/lib/usercopy_32.c
@@ -5,12 +5,7 @@
* Copyright 1997 Andi Kleen <ak@muc.de>
* Copyright 1997 Linus Torvalds
*/
-#include <linux/mm.h>
-#include <linux/highmem.h>
-#include <linux/blkdev.h>
#include <linux/export.h>
-#include <linux/backing-dev.h>
-#include <linux/interrupt.h>
#include <linux/uaccess.h>
#include <asm/mmx.h>
#include <asm/asm.h>
@@ -201,197 +196,6 @@
return size;
}
-static unsigned long
-__copy_user_zeroing_intel(void *to, const void __user *from, unsigned long size)
-{
- int d0, d1;
- __asm__ __volatile__(
- " .align 2,0x90\n"
- "0: movl 32(%4), %%eax\n"
- " cmpl $67, %0\n"
- " jbe 2f\n"
- "1: movl 64(%4), %%eax\n"
- " .align 2,0x90\n"
- "2: movl 0(%4), %%eax\n"
- "21: movl 4(%4), %%edx\n"
- " movl %%eax, 0(%3)\n"
- " movl %%edx, 4(%3)\n"
- "3: movl 8(%4), %%eax\n"
- "31: movl 12(%4),%%edx\n"
- " movl %%eax, 8(%3)\n"
- " movl %%edx, 12(%3)\n"
- "4: movl 16(%4), %%eax\n"
- "41: movl 20(%4), %%edx\n"
- " movl %%eax, 16(%3)\n"
- " movl %%edx, 20(%3)\n"
- "10: movl 24(%4), %%eax\n"
- "51: movl 28(%4), %%edx\n"
- " movl %%eax, 24(%3)\n"
- " movl %%edx, 28(%3)\n"
- "11: movl 32(%4), %%eax\n"
- "61: movl 36(%4), %%edx\n"
- " movl %%eax, 32(%3)\n"
- " movl %%edx, 36(%3)\n"
- "12: movl 40(%4), %%eax\n"
- "71: movl 44(%4), %%edx\n"
- " movl %%eax, 40(%3)\n"
- " movl %%edx, 44(%3)\n"
- "13: movl 48(%4), %%eax\n"
- "81: movl 52(%4), %%edx\n"
- " movl %%eax, 48(%3)\n"
- " movl %%edx, 52(%3)\n"
- "14: movl 56(%4), %%eax\n"
- "91: movl 60(%4), %%edx\n"
- " movl %%eax, 56(%3)\n"
- " movl %%edx, 60(%3)\n"
- " addl $-64, %0\n"
- " addl $64, %4\n"
- " addl $64, %3\n"
- " cmpl $63, %0\n"
- " ja 0b\n"
- "5: movl %0, %%eax\n"
- " shrl $2, %0\n"
- " andl $3, %%eax\n"
- " cld\n"
- "6: rep; movsl\n"
- " movl %%eax,%0\n"
- "7: rep; movsb\n"
- "8:\n"
- ".section .fixup,\"ax\"\n"
- "9: lea 0(%%eax,%0,4),%0\n"
- "16: pushl %0\n"
- " pushl %%eax\n"
- " xorl %%eax,%%eax\n"
- " rep; stosb\n"
- " popl %%eax\n"
- " popl %0\n"
- " jmp 8b\n"
- ".previous\n"
- _ASM_EXTABLE(0b,16b)
- _ASM_EXTABLE(1b,16b)
- _ASM_EXTABLE(2b,16b)
- _ASM_EXTABLE(21b,16b)
- _ASM_EXTABLE(3b,16b)
- _ASM_EXTABLE(31b,16b)
- _ASM_EXTABLE(4b,16b)
- _ASM_EXTABLE(41b,16b)
- _ASM_EXTABLE(10b,16b)
- _ASM_EXTABLE(51b,16b)
- _ASM_EXTABLE(11b,16b)
- _ASM_EXTABLE(61b,16b)
- _ASM_EXTABLE(12b,16b)
- _ASM_EXTABLE(71b,16b)
- _ASM_EXTABLE(13b,16b)
- _ASM_EXTABLE(81b,16b)
- _ASM_EXTABLE(14b,16b)
- _ASM_EXTABLE(91b,16b)
- _ASM_EXTABLE(6b,9b)
- _ASM_EXTABLE(7b,16b)
- : "=&c"(size), "=&D" (d0), "=&S" (d1)
- : "1"(to), "2"(from), "0"(size)
- : "eax", "edx", "memory");
- return size;
-}
-
-/*
- * Non Temporal Hint version of __copy_user_zeroing_intel. It is cache aware.
- * hyoshiok@miraclelinux.com
- */
-
-static unsigned long __copy_user_zeroing_intel_nocache(void *to,
- const void __user *from, unsigned long size)
-{
- int d0, d1;
-
- __asm__ __volatile__(
- " .align 2,0x90\n"
- "0: movl 32(%4), %%eax\n"
- " cmpl $67, %0\n"
- " jbe 2f\n"
- "1: movl 64(%4), %%eax\n"
- " .align 2,0x90\n"
- "2: movl 0(%4), %%eax\n"
- "21: movl 4(%4), %%edx\n"
- " movnti %%eax, 0(%3)\n"
- " movnti %%edx, 4(%3)\n"
- "3: movl 8(%4), %%eax\n"
- "31: movl 12(%4),%%edx\n"
- " movnti %%eax, 8(%3)\n"
- " movnti %%edx, 12(%3)\n"
- "4: movl 16(%4), %%eax\n"
- "41: movl 20(%4), %%edx\n"
- " movnti %%eax, 16(%3)\n"
- " movnti %%edx, 20(%3)\n"
- "10: movl 24(%4), %%eax\n"
- "51: movl 28(%4), %%edx\n"
- " movnti %%eax, 24(%3)\n"
- " movnti %%edx, 28(%3)\n"
- "11: movl 32(%4), %%eax\n"
- "61: movl 36(%4), %%edx\n"
- " movnti %%eax, 32(%3)\n"
- " movnti %%edx, 36(%3)\n"
- "12: movl 40(%4), %%eax\n"
- "71: movl 44(%4), %%edx\n"
- " movnti %%eax, 40(%3)\n"
- " movnti %%edx, 44(%3)\n"
- "13: movl 48(%4), %%eax\n"
- "81: movl 52(%4), %%edx\n"
- " movnti %%eax, 48(%3)\n"
- " movnti %%edx, 52(%3)\n"
- "14: movl 56(%4), %%eax\n"
- "91: movl 60(%4), %%edx\n"
- " movnti %%eax, 56(%3)\n"
- " movnti %%edx, 60(%3)\n"
- " addl $-64, %0\n"
- " addl $64, %4\n"
- " addl $64, %3\n"
- " cmpl $63, %0\n"
- " ja 0b\n"
- " sfence \n"
- "5: movl %0, %%eax\n"
- " shrl $2, %0\n"
- " andl $3, %%eax\n"
- " cld\n"
- "6: rep; movsl\n"
- " movl %%eax,%0\n"
- "7: rep; movsb\n"
- "8:\n"
- ".section .fixup,\"ax\"\n"
- "9: lea 0(%%eax,%0,4),%0\n"
- "16: pushl %0\n"
- " pushl %%eax\n"
- " xorl %%eax,%%eax\n"
- " rep; stosb\n"
- " popl %%eax\n"
- " popl %0\n"
- " jmp 8b\n"
- ".previous\n"
- _ASM_EXTABLE(0b,16b)
- _ASM_EXTABLE(1b,16b)
- _ASM_EXTABLE(2b,16b)
- _ASM_EXTABLE(21b,16b)
- _ASM_EXTABLE(3b,16b)
- _ASM_EXTABLE(31b,16b)
- _ASM_EXTABLE(4b,16b)
- _ASM_EXTABLE(41b,16b)
- _ASM_EXTABLE(10b,16b)
- _ASM_EXTABLE(51b,16b)
- _ASM_EXTABLE(11b,16b)
- _ASM_EXTABLE(61b,16b)
- _ASM_EXTABLE(12b,16b)
- _ASM_EXTABLE(71b,16b)
- _ASM_EXTABLE(13b,16b)
- _ASM_EXTABLE(81b,16b)
- _ASM_EXTABLE(14b,16b)
- _ASM_EXTABLE(91b,16b)
- _ASM_EXTABLE(6b,9b)
- _ASM_EXTABLE(7b,16b)
- : "=&c"(size), "=&D" (d0), "=&S" (d1)
- : "1"(to), "2"(from), "0"(size)
- : "eax", "edx", "memory");
- return size;
-}
-
static unsigned long __copy_user_intel_nocache(void *to,
const void __user *from, unsigned long size)
{
@@ -486,12 +290,8 @@
* Leave these declared but undefined. They should not be any references to
* them
*/
-unsigned long __copy_user_zeroing_intel(void *to, const void __user *from,
- unsigned long size);
unsigned long __copy_user_intel(void __user *to, const void *from,
unsigned long size);
-unsigned long __copy_user_zeroing_intel_nocache(void *to,
- const void __user *from, unsigned long size);
#endif /* CONFIG_X86_INTEL_USERCOPY */
/* Generic arbitrary sized copy. */
@@ -528,47 +328,7 @@
: "memory"); \
} while (0)
-#define __copy_user_zeroing(to, from, size) \
-do { \
- int __d0, __d1, __d2; \
- __asm__ __volatile__( \
- " cmp $7,%0\n" \
- " jbe 1f\n" \
- " movl %1,%0\n" \
- " negl %0\n" \
- " andl $7,%0\n" \
- " subl %0,%3\n" \
- "4: rep; movsb\n" \
- " movl %3,%0\n" \
- " shrl $2,%0\n" \
- " andl $3,%3\n" \
- " .align 2,0x90\n" \
- "0: rep; movsl\n" \
- " movl %3,%0\n" \
- "1: rep; movsb\n" \
- "2:\n" \
- ".section .fixup,\"ax\"\n" \
- "5: addl %3,%0\n" \
- " jmp 6f\n" \
- "3: lea 0(%3,%0,4),%0\n" \
- "6: pushl %0\n" \
- " pushl %%eax\n" \
- " xorl %%eax,%%eax\n" \
- " rep; stosb\n" \
- " popl %%eax\n" \
- " popl %0\n" \
- " jmp 2b\n" \
- ".previous\n" \
- _ASM_EXTABLE(4b,5b) \
- _ASM_EXTABLE(0b,3b) \
- _ASM_EXTABLE(1b,6b) \
- : "=&c"(size), "=&D" (__d0), "=&S" (__d1), "=r"(__d2) \
- : "3"(size), "0"(size), "1"(to), "2"(from) \
- : "memory"); \
-} while (0)
-
-unsigned long __copy_to_user_ll(void __user *to, const void *from,
- unsigned long n)
+unsigned long __copy_user_ll(void *to, const void *from, unsigned long n)
{
stac();
if (movsl_is_ok(to, from, n))
@@ -578,51 +338,7 @@
clac();
return n;
}
-EXPORT_SYMBOL(__copy_to_user_ll);
-
-unsigned long __copy_from_user_ll(void *to, const void __user *from,
- unsigned long n)
-{
- stac();
- if (movsl_is_ok(to, from, n))
- __copy_user_zeroing(to, from, n);
- else
- n = __copy_user_zeroing_intel(to, from, n);
- clac();
- return n;
-}
-EXPORT_SYMBOL(__copy_from_user_ll);
-
-unsigned long __copy_from_user_ll_nozero(void *to, const void __user *from,
- unsigned long n)
-{
- stac();
- if (movsl_is_ok(to, from, n))
- __copy_user(to, from, n);
- else
- n = __copy_user_intel((void __user *)to,
- (const void *)from, n);
- clac();
- return n;
-}
-EXPORT_SYMBOL(__copy_from_user_ll_nozero);
-
-unsigned long __copy_from_user_ll_nocache(void *to, const void __user *from,
- unsigned long n)
-{
- stac();
-#ifdef CONFIG_X86_INTEL_USERCOPY
- if (n > 64 && static_cpu_has(X86_FEATURE_XMM2))
- n = __copy_user_zeroing_intel_nocache(to, from, n);
- else
- __copy_user_zeroing(to, from, n);
-#else
- __copy_user_zeroing(to, from, n);
-#endif
- clac();
- return n;
-}
-EXPORT_SYMBOL(__copy_from_user_ll_nocache);
+EXPORT_SYMBOL(__copy_user_ll);
unsigned long __copy_from_user_ll_nocache_nozero(void *to, const void __user *from,
unsigned long n)
diff --git a/arch/x86/lib/usercopy_64.c b/arch/x86/lib/usercopy_64.c
index 6987358..3b7c40a 100644
--- a/arch/x86/lib/usercopy_64.c
+++ b/arch/x86/lib/usercopy_64.c
@@ -54,15 +54,6 @@
}
EXPORT_SYMBOL(clear_user);
-unsigned long copy_in_user(void __user *to, const void __user *from, unsigned len)
-{
- if (access_ok(VERIFY_WRITE, to, len) && access_ok(VERIFY_READ, from, len)) {
- return copy_user_generic((__force void *)to, (__force void *)from, len);
- }
- return len;
-}
-EXPORT_SYMBOL(copy_in_user);
-
/*
* Try to copy last bytes and clear the rest if needed.
* Since protection fault in copy_from/to_user is not a normal situation,
@@ -80,9 +71,5 @@
break;
}
clac();
-
- /* If the destination is a kernel buffer, we always clear the end */
- if (!__addr_ok(to))
- memset(to, 0, len);
return len;
}
diff --git a/arch/x86/mm/amdtopology.c b/arch/x86/mm/amdtopology.c
index d1c7de0..91f501b 100644
--- a/arch/x86/mm/amdtopology.c
+++ b/arch/x86/mm/amdtopology.c
@@ -19,7 +19,7 @@
#include <asm/types.h>
#include <asm/mmzone.h>
#include <asm/proto.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/pci-direct.h>
#include <asm/numa.h>
#include <asm/mpspec.h>
diff --git a/arch/x86/mm/dump_pagetables.c b/arch/x86/mm/dump_pagetables.c
index 58b5bee..bce6990 100644
--- a/arch/x86/mm/dump_pagetables.c
+++ b/arch/x86/mm/dump_pagetables.c
@@ -110,7 +110,8 @@
#define PTE_LEVEL_MULT (PAGE_SIZE)
#define PMD_LEVEL_MULT (PTRS_PER_PTE * PTE_LEVEL_MULT)
#define PUD_LEVEL_MULT (PTRS_PER_PMD * PMD_LEVEL_MULT)
-#define PGD_LEVEL_MULT (PTRS_PER_PUD * PUD_LEVEL_MULT)
+#define P4D_LEVEL_MULT (PTRS_PER_PUD * PUD_LEVEL_MULT)
+#define PGD_LEVEL_MULT (PTRS_PER_P4D * P4D_LEVEL_MULT)
#define pt_dump_seq_printf(m, to_dmesg, fmt, args...) \
({ \
@@ -286,14 +287,13 @@
}
}
-static void walk_pte_level(struct seq_file *m, struct pg_state *st, pmd_t addr,
- unsigned long P)
+static void walk_pte_level(struct seq_file *m, struct pg_state *st, pmd_t addr, unsigned long P)
{
int i;
pte_t *start;
pgprotval_t prot;
- start = (pte_t *) pmd_page_vaddr(addr);
+ start = (pte_t *)pmd_page_vaddr(addr);
for (i = 0; i < PTRS_PER_PTE; i++) {
prot = pte_flags(*start);
st->current_address = normalize_addr(P + i * PTE_LEVEL_MULT);
@@ -304,14 +304,13 @@
#if PTRS_PER_PMD > 1
-static void walk_pmd_level(struct seq_file *m, struct pg_state *st, pud_t addr,
- unsigned long P)
+static void walk_pmd_level(struct seq_file *m, struct pg_state *st, pud_t addr, unsigned long P)
{
int i;
pmd_t *start;
pgprotval_t prot;
- start = (pmd_t *) pud_page_vaddr(addr);
+ start = (pmd_t *)pud_page_vaddr(addr);
for (i = 0; i < PTRS_PER_PMD; i++) {
st->current_address = normalize_addr(P + i * PMD_LEVEL_MULT);
if (!pmd_none(*start)) {
@@ -347,15 +346,14 @@
return checkwx && prev_pud && (pud_val(*prev_pud) == pud_val(*pud));
}
-static void walk_pud_level(struct seq_file *m, struct pg_state *st, pgd_t addr,
- unsigned long P)
+static void walk_pud_level(struct seq_file *m, struct pg_state *st, p4d_t addr, unsigned long P)
{
int i;
pud_t *start;
pgprotval_t prot;
pud_t *prev_pud = NULL;
- start = (pud_t *) pgd_page_vaddr(addr);
+ start = (pud_t *)p4d_page_vaddr(addr);
for (i = 0; i < PTRS_PER_PUD; i++) {
st->current_address = normalize_addr(P + i * PUD_LEVEL_MULT);
@@ -377,9 +375,42 @@
}
#else
-#define walk_pud_level(m,s,a,p) walk_pmd_level(m,s,__pud(pgd_val(a)),p)
-#define pgd_large(a) pud_large(__pud(pgd_val(a)))
-#define pgd_none(a) pud_none(__pud(pgd_val(a)))
+#define walk_pud_level(m,s,a,p) walk_pmd_level(m,s,__pud(p4d_val(a)),p)
+#define p4d_large(a) pud_large(__pud(p4d_val(a)))
+#define p4d_none(a) pud_none(__pud(p4d_val(a)))
+#endif
+
+#if PTRS_PER_P4D > 1
+
+static void walk_p4d_level(struct seq_file *m, struct pg_state *st, pgd_t addr, unsigned long P)
+{
+ int i;
+ p4d_t *start;
+ pgprotval_t prot;
+
+ start = (p4d_t *)pgd_page_vaddr(addr);
+
+ for (i = 0; i < PTRS_PER_P4D; i++) {
+ st->current_address = normalize_addr(P + i * P4D_LEVEL_MULT);
+ if (!p4d_none(*start)) {
+ if (p4d_large(*start) || !p4d_present(*start)) {
+ prot = p4d_flags(*start);
+ note_page(m, st, __pgprot(prot), 2);
+ } else {
+ walk_pud_level(m, st, *start,
+ P + i * P4D_LEVEL_MULT);
+ }
+ } else
+ note_page(m, st, __pgprot(0), 2);
+
+ start++;
+ }
+}
+
+#else
+#define walk_p4d_level(m,s,a,p) walk_pud_level(m,s,__p4d(pgd_val(a)),p)
+#define pgd_large(a) p4d_large(__p4d(pgd_val(a)))
+#define pgd_none(a) p4d_none(__p4d(pgd_val(a)))
#endif
static inline bool is_hypervisor_range(int idx)
@@ -424,7 +455,7 @@
prot = pgd_flags(*start);
note_page(m, &st, __pgprot(prot), 1);
} else {
- walk_pud_level(m, &st, *start,
+ walk_p4d_level(m, &st, *start,
i * PGD_LEVEL_MULT);
}
} else
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index 428e3176..8ad91a0 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -253,6 +253,7 @@
{
unsigned index = pgd_index(address);
pgd_t *pgd_k;
+ p4d_t *p4d, *p4d_k;
pud_t *pud, *pud_k;
pmd_t *pmd, *pmd_k;
@@ -265,10 +266,15 @@
/*
* set_pgd(pgd, *pgd_k); here would be useless on PAE
* and redundant with the set_pmd() on non-PAE. As would
- * set_pud.
+ * set_p4d/set_pud.
*/
- pud = pud_offset(pgd, address);
- pud_k = pud_offset(pgd_k, address);
+ p4d = p4d_offset(pgd, address);
+ p4d_k = p4d_offset(pgd_k, address);
+ if (!p4d_present(*p4d_k))
+ return NULL;
+
+ pud = pud_offset(p4d, address);
+ pud_k = pud_offset(p4d_k, address);
if (!pud_present(*pud_k))
return NULL;
@@ -384,6 +390,8 @@
{
pgd_t *base = __va(read_cr3());
pgd_t *pgd = &base[pgd_index(address)];
+ p4d_t *p4d;
+ pud_t *pud;
pmd_t *pmd;
pte_t *pte;
@@ -392,7 +400,9 @@
if (!low_pfn(pgd_val(*pgd) >> PAGE_SHIFT) || !pgd_present(*pgd))
goto out;
#endif
- pmd = pmd_offset(pud_offset(pgd, address), address);
+ p4d = p4d_offset(pgd, address);
+ pud = pud_offset(p4d, address);
+ pmd = pmd_offset(pud, address);
printk(KERN_CONT "*pde = %0*Lx ", sizeof(*pmd) * 2, (u64)pmd_val(*pmd));
/*
@@ -425,6 +435,7 @@
static noinline int vmalloc_fault(unsigned long address)
{
pgd_t *pgd, *pgd_ref;
+ p4d_t *p4d, *p4d_ref;
pud_t *pud, *pud_ref;
pmd_t *pmd, *pmd_ref;
pte_t *pte, *pte_ref;
@@ -448,17 +459,37 @@
if (pgd_none(*pgd)) {
set_pgd(pgd, *pgd_ref);
arch_flush_lazy_mmu_mode();
- } else {
+ } else if (CONFIG_PGTABLE_LEVELS > 4) {
+ /*
+ * With folded p4d, pgd_none() is always false, so the pgd may
+ * point to an empty page table entry and pgd_page_vaddr()
+ * will return garbage.
+ *
+ * We will do the correct sanity check on the p4d level.
+ */
BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref));
}
+ /* With 4-level paging, copying happens on the p4d level. */
+ p4d = p4d_offset(pgd, address);
+ p4d_ref = p4d_offset(pgd_ref, address);
+ if (p4d_none(*p4d_ref))
+ return -1;
+
+ if (p4d_none(*p4d)) {
+ set_p4d(p4d, *p4d_ref);
+ arch_flush_lazy_mmu_mode();
+ } else {
+ BUG_ON(p4d_pfn(*p4d) != p4d_pfn(*p4d_ref));
+ }
+
/*
* Below here mismatches are bugs because these lower tables
* are shared:
*/
- pud = pud_offset(pgd, address);
- pud_ref = pud_offset(pgd_ref, address);
+ pud = pud_offset(p4d, address);
+ pud_ref = pud_offset(p4d_ref, address);
if (pud_none(*pud_ref))
return -1;
@@ -526,6 +557,7 @@
{
pgd_t *base = __va(read_cr3() & PHYSICAL_PAGE_MASK);
pgd_t *pgd = base + pgd_index(address);
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
@@ -538,7 +570,15 @@
if (!pgd_present(*pgd))
goto out;
- pud = pud_offset(pgd, address);
+ p4d = p4d_offset(pgd, address);
+ if (bad_address(p4d))
+ goto bad;
+
+ printk("P4D %lx ", p4d_val(*p4d));
+ if (!p4d_present(*p4d) || p4d_large(*p4d))
+ goto out;
+
+ pud = pud_offset(p4d, address);
if (bad_address(pud))
goto bad;
@@ -1082,6 +1122,7 @@
spurious_fault(unsigned long error_code, unsigned long address)
{
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
@@ -1104,7 +1145,14 @@
if (!pgd_present(*pgd))
return 0;
- pud = pud_offset(pgd, address);
+ p4d = p4d_offset(pgd, address);
+ if (!p4d_present(*p4d))
+ return 0;
+
+ if (p4d_large(*p4d))
+ return spurious_fault_check(error_code, (pte_t *) p4d);
+
+ pud = pud_offset(p4d, address);
if (!pud_present(*pud))
return 0;
diff --git a/arch/x86/mm/gup.c b/arch/x86/mm/gup.c
index 1f3b6ef..456dfdf 100644
--- a/arch/x86/mm/gup.c
+++ b/arch/x86/mm/gup.c
@@ -76,9 +76,9 @@
}
/*
- * 'pteval' can come from a pte, pmd or pud. We only check
+ * 'pteval' can come from a pte, pmd, pud or p4d. We only check
* _PAGE_PRESENT, _PAGE_USER, and _PAGE_RW in here which are the
- * same value on all 3 types.
+ * same value on all 4 types.
*/
static inline int pte_allows_gup(unsigned long pteval, int write)
{
@@ -295,13 +295,13 @@
return 1;
}
-static int gup_pud_range(pgd_t pgd, unsigned long addr, unsigned long end,
+static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end,
int write, struct page **pages, int *nr)
{
unsigned long next;
pud_t *pudp;
- pudp = pud_offset(&pgd, addr);
+ pudp = pud_offset(&p4d, addr);
do {
pud_t pud = *pudp;
@@ -320,6 +320,27 @@
return 1;
}
+static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end,
+ int write, struct page **pages, int *nr)
+{
+ unsigned long next;
+ p4d_t *p4dp;
+
+ p4dp = p4d_offset(&pgd, addr);
+ do {
+ p4d_t p4d = *p4dp;
+
+ next = p4d_addr_end(addr, end);
+ if (p4d_none(p4d))
+ return 0;
+ BUILD_BUG_ON(p4d_large(p4d));
+ if (!gup_pud_range(p4d, addr, next, write, pages, nr))
+ return 0;
+ } while (p4dp++, addr = next, addr != end);
+
+ return 1;
+}
+
/*
* Like get_user_pages_fast() except its IRQ-safe in that it won't fall
* back to the regular GUP.
@@ -368,7 +389,7 @@
next = pgd_addr_end(addr, end);
if (pgd_none(pgd))
break;
- if (!gup_pud_range(pgd, addr, next, write, pages, &nr))
+ if (!gup_p4d_range(pgd, addr, next, write, pages, &nr))
break;
} while (pgdp++, addr = next, addr != end);
local_irq_restore(flags);
@@ -440,7 +461,7 @@
next = pgd_addr_end(addr, end);
if (pgd_none(pgd))
goto slow;
- if (!gup_pud_range(pgd, addr, next, write, pages, &nr))
+ if (!gup_p4d_range(pgd, addr, next, write, pages, &nr))
goto slow;
} while (pgdp++, addr = next, addr != end);
local_irq_enable();
diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c
index c5066a2..302f43f 100644
--- a/arch/x86/mm/hugetlbpage.c
+++ b/arch/x86/mm/hugetlbpage.c
@@ -12,10 +12,12 @@
#include <linux/pagemap.h>
#include <linux/err.h>
#include <linux/sysctl.h>
+#include <linux/compat.h>
#include <asm/mman.h>
#include <asm/tlb.h>
#include <asm/tlbflush.h>
#include <asm/pgalloc.h>
+#include <asm/elf.h>
#if 0 /* This is just for testing */
struct page *
@@ -82,8 +84,9 @@
info.flags = 0;
info.length = len;
- info.low_limit = current->mm->mmap_legacy_base;
- info.high_limit = TASK_SIZE;
+ info.low_limit = get_mmap_base(1);
+ info.high_limit = in_compat_syscall() ?
+ tasksize_32bit() : tasksize_64bit();
info.align_mask = PAGE_MASK & ~huge_page_mask(h);
info.align_offset = 0;
return vm_unmapped_area(&info);
@@ -100,7 +103,7 @@
info.flags = VM_UNMAPPED_AREA_TOPDOWN;
info.length = len;
info.low_limit = PAGE_SIZE;
- info.high_limit = current->mm->mmap_base;
+ info.high_limit = get_mmap_base(0);
info.align_mask = PAGE_MASK & ~huge_page_mask(h);
info.align_offset = 0;
addr = vm_unmapped_area(&info);
diff --git a/arch/x86/mm/ident_map.c b/arch/x86/mm/ident_map.c
index 4473cb4..04210a2 100644
--- a/arch/x86/mm/ident_map.c
+++ b/arch/x86/mm/ident_map.c
@@ -45,6 +45,34 @@
return 0;
}
+static int ident_p4d_init(struct x86_mapping_info *info, p4d_t *p4d_page,
+ unsigned long addr, unsigned long end)
+{
+ unsigned long next;
+
+ for (; addr < end; addr = next) {
+ p4d_t *p4d = p4d_page + p4d_index(addr);
+ pud_t *pud;
+
+ next = (addr & P4D_MASK) + P4D_SIZE;
+ if (next > end)
+ next = end;
+
+ if (p4d_present(*p4d)) {
+ pud = pud_offset(p4d, 0);
+ ident_pud_init(info, pud, addr, next);
+ continue;
+ }
+ pud = (pud_t *)info->alloc_pgt_page(info->context);
+ if (!pud)
+ return -ENOMEM;
+ ident_pud_init(info, pud, addr, next);
+ set_p4d(p4d, __p4d(__pa(pud) | _KERNPG_TABLE));
+ }
+
+ return 0;
+}
+
int kernel_ident_mapping_init(struct x86_mapping_info *info, pgd_t *pgd_page,
unsigned long pstart, unsigned long pend)
{
@@ -55,27 +83,36 @@
for (; addr < end; addr = next) {
pgd_t *pgd = pgd_page + pgd_index(addr);
- pud_t *pud;
+ p4d_t *p4d;
next = (addr & PGDIR_MASK) + PGDIR_SIZE;
if (next > end)
next = end;
if (pgd_present(*pgd)) {
- pud = pud_offset(pgd, 0);
- result = ident_pud_init(info, pud, addr, next);
+ p4d = p4d_offset(pgd, 0);
+ result = ident_p4d_init(info, p4d, addr, next);
if (result)
return result;
continue;
}
- pud = (pud_t *)info->alloc_pgt_page(info->context);
- if (!pud)
+ p4d = (p4d_t *)info->alloc_pgt_page(info->context);
+ if (!p4d)
return -ENOMEM;
- result = ident_pud_init(info, pud, addr, next);
+ result = ident_p4d_init(info, p4d, addr, next);
if (result)
return result;
- set_pgd(pgd, __pgd(__pa(pud) | _KERNPG_TABLE));
+ if (IS_ENABLED(CONFIG_X86_5LEVEL)) {
+ set_pgd(pgd, __pgd(__pa(p4d) | _KERNPG_TABLE));
+ } else {
+ /*
+ * With p4d folded, pgd is equal to p4d.
+ * The pgd entry has to point to the pud page table in this case.
+ */
+ pud_t *pud = pud_offset(p4d, 0);
+ set_pgd(pgd, __pgd(__pa(pud) | _KERNPG_TABLE));
+ }
}
return 0;
diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c
index 889e761..138bad2 100644
--- a/arch/x86/mm/init.c
+++ b/arch/x86/mm/init.c
@@ -6,7 +6,7 @@
#include <linux/bootmem.h> /* for max_low_pfn */
#include <asm/cacheflush.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/init.h>
#include <asm/page.h>
#include <asm/page_types.h>
@@ -373,14 +373,14 @@
return nr_range;
}
-struct range pfn_mapped[E820_X_MAX];
+struct range pfn_mapped[E820_MAX_ENTRIES];
int nr_pfn_mapped;
static void add_pfn_range_mapped(unsigned long start_pfn, unsigned long end_pfn)
{
- nr_pfn_mapped = add_range_with_merge(pfn_mapped, E820_X_MAX,
+ nr_pfn_mapped = add_range_with_merge(pfn_mapped, E820_MAX_ENTRIES,
nr_pfn_mapped, start_pfn, end_pfn);
- nr_pfn_mapped = clean_sort_range(pfn_mapped, E820_X_MAX);
+ nr_pfn_mapped = clean_sort_range(pfn_mapped, E820_MAX_ENTRIES);
max_pfn_mapped = max(max_pfn_mapped, end_pfn);
@@ -430,7 +430,7 @@
/*
* We need to iterate through the E820 memory map and create direct mappings
- * for only E820_RAM and E820_KERN_RESERVED regions. We cannot simply
+ * for only E820_TYPE_RAM and E820_KERN_RESERVED regions. We cannot simply
* create direct mappings for all pfns from [0 to max_low_pfn) and
* [4GB to max_pfn) because of possible memory holes in high addresses
* that cannot be marked as UC by fixed/variable range MTRRs.
@@ -720,7 +720,7 @@
void __ref free_initmem(void)
{
- e820_reallocate_tables();
+ e820__reallocate_tables();
free_init_pages("unused kernel",
(unsigned long)(&__init_begin),
@@ -743,6 +743,53 @@
}
#endif
+/*
+ * Calculate the precise size of the DMA zone (first 16 MB of RAM),
+ * and pass it to the MM layer - to help it set zone watermarks more
+ * accurately.
+ *
+ * Done on 64-bit systems only for the time being, although 32-bit systems
+ * might benefit from this as well.
+ */
+void __init memblock_find_dma_reserve(void)
+{
+#ifdef CONFIG_X86_64
+ u64 nr_pages = 0, nr_free_pages = 0;
+ unsigned long start_pfn, end_pfn;
+ phys_addr_t start_addr, end_addr;
+ int i;
+ u64 u;
+
+ /*
+ * Iterate over all memory ranges (free and reserved ones alike),
+ * to calculate the total number of pages in the first 16 MB of RAM:
+ */
+ nr_pages = 0;
+ for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, NULL) {
+ start_pfn = min(start_pfn, MAX_DMA_PFN);
+ end_pfn = min(end_pfn, MAX_DMA_PFN);
+
+ nr_pages += end_pfn - start_pfn;
+ }
+
+ /*
+ * Iterate over free memory ranges to calculate the number of free
+ * pages in the DMA zone, while not counting potential partial
+ * pages at the beginning or the end of the range:
+ */
+ nr_free_pages = 0;
+ for_each_free_mem_range(u, NUMA_NO_NODE, MEMBLOCK_NONE, &start_addr, &end_addr, NULL) {
+ start_pfn = min_t(unsigned long, PFN_UP(start_addr), MAX_DMA_PFN);
+ end_pfn = min_t(unsigned long, PFN_DOWN(end_addr), MAX_DMA_PFN);
+
+ if (start_pfn < end_pfn)
+ nr_free_pages += end_pfn - start_pfn;
+ }
+
+ set_dma_reserve(nr_pages - nr_free_pages);
+#endif
+}
+
void __init zone_sizes_init(void)
{
unsigned long max_zone_pfns[MAX_NR_ZONES];
diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c
index 2b4b53e..f34d275 100644
--- a/arch/x86/mm/init_32.c
+++ b/arch/x86/mm/init_32.c
@@ -38,7 +38,7 @@
#include <asm/pgtable.h>
#include <asm/dma.h>
#include <asm/fixmap.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/apic.h>
#include <asm/bugs.h>
#include <asm/tlb.h>
@@ -56,8 +56,6 @@
unsigned long highstart_pfn, highend_pfn;
-static noinline int do_test_wp_bit(void);
-
bool __read_mostly __vmalloc_start_set = false;
/*
@@ -67,6 +65,7 @@
*/
static pmd_t * __init one_md_table_init(pgd_t *pgd)
{
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd_table;
@@ -75,13 +74,15 @@
pmd_table = (pmd_t *)alloc_low_page();
paravirt_alloc_pmd(&init_mm, __pa(pmd_table) >> PAGE_SHIFT);
set_pgd(pgd, __pgd(__pa(pmd_table) | _PAGE_PRESENT));
- pud = pud_offset(pgd, 0);
+ p4d = p4d_offset(pgd, 0);
+ pud = pud_offset(p4d, 0);
BUG_ON(pmd_table != pmd_offset(pud, 0));
return pmd_table;
}
#endif
- pud = pud_offset(pgd, 0);
+ p4d = p4d_offset(pgd, 0);
+ pud = pud_offset(p4d, 0);
pmd_table = pmd_offset(pud, 0);
return pmd_table;
@@ -390,8 +391,11 @@
static inline pte_t *kmap_get_fixmap_pte(unsigned long vaddr)
{
- return pte_offset_kernel(pmd_offset(pud_offset(pgd_offset_k(vaddr),
- vaddr), vaddr), vaddr);
+ pgd_t *pgd = pgd_offset_k(vaddr);
+ p4d_t *p4d = p4d_offset(pgd, vaddr);
+ pud_t *pud = pud_offset(p4d, vaddr);
+ pmd_t *pmd = pmd_offset(pud, vaddr);
+ return pte_offset_kernel(pmd, vaddr);
}
static void __init kmap_init(void)
@@ -410,6 +414,7 @@
{
unsigned long vaddr;
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
@@ -418,7 +423,8 @@
page_table_range_init(vaddr, vaddr + PAGE_SIZE*LAST_PKMAP, pgd_base);
pgd = swapper_pg_dir + pgd_index(vaddr);
- pud = pud_offset(pgd, vaddr);
+ p4d = p4d_offset(pgd, vaddr);
+ pud = pud_offset(p4d, vaddr);
pmd = pmd_offset(pud, vaddr);
pte = pte_offset_kernel(pmd, vaddr);
pkmap_page_table = pte;
@@ -450,6 +456,7 @@
{
unsigned long pfn, va;
pgd_t *pgd, *base = swapper_pg_dir;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
@@ -469,7 +476,8 @@
if (!pgd_present(*pgd))
break;
- pud = pud_offset(pgd, va);
+ p4d = p4d_offset(pgd, va);
+ pud = pud_offset(p4d, va);
pmd = pmd_offset(pud, va);
if (!pmd_present(*pmd))
break;
@@ -716,20 +724,20 @@
*/
static void __init test_wp_bit(void)
{
- printk(KERN_INFO
- "Checking if this processor honours the WP bit even in supervisor mode...");
+ char z = 0;
- /* Any page-aligned address will do, the test is non-destructive */
- __set_fixmap(FIX_WP_TEST, __pa(&swapper_pg_dir), PAGE_KERNEL_RO);
- boot_cpu_data.wp_works_ok = do_test_wp_bit();
- clear_fixmap(FIX_WP_TEST);
+ printk(KERN_INFO "Checking if this processor honours the WP bit even in supervisor mode...");
- if (!boot_cpu_data.wp_works_ok) {
- printk(KERN_CONT "No.\n");
- panic("Linux doesn't support CPUs with broken WP.");
- } else {
+ __set_fixmap(FIX_WP_TEST, __pa_symbol(empty_zero_page), PAGE_KERNEL_RO);
+
+ if (probe_kernel_write((char *)fix_to_virt(FIX_WP_TEST), &z, 1)) {
+ clear_fixmap(FIX_WP_TEST);
printk(KERN_CONT "Ok.\n");
+ return;
}
+
+ printk(KERN_CONT "No.\n");
+ panic("Linux doesn't support CPUs with broken WP.");
}
void __init mem_init(void)
@@ -811,8 +819,7 @@
BUG_ON(VMALLOC_START >= VMALLOC_END);
BUG_ON((unsigned long)high_memory > VMALLOC_START);
- if (boot_cpu_data.wp_works_ok < 0)
- test_wp_bit();
+ test_wp_bit();
}
#ifdef CONFIG_MEMORY_HOTPLUG
@@ -840,30 +847,6 @@
#endif
#endif
-/*
- * This function cannot be __init, since exceptions don't work in that
- * section. Put this after the callers, so that it cannot be inlined.
- */
-static noinline int do_test_wp_bit(void)
-{
- char tmp_reg;
- int flag;
-
- __asm__ __volatile__(
- " movb %0, %1 \n"
- "1: movb %1, %0 \n"
- " xorl %2, %2 \n"
- "2: \n"
- _ASM_EXTABLE(1b,2b)
- :"=m" (*(char *)fix_to_virt(FIX_WP_TEST)),
- "=q" (tmp_reg),
- "=r" (flag)
- :"2" (1)
- :"memory");
-
- return flag;
-}
-
int kernel_set_to_readonly __read_mostly;
void set_kernel_text_rw(void)
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 15173d3..745e5e1 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -41,7 +41,7 @@
#include <asm/pgalloc.h>
#include <asm/dma.h>
#include <asm/fixmap.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/apic.h>
#include <asm/tlb.h>
#include <asm/mmu_context.h>
@@ -97,28 +97,38 @@
unsigned long address;
for (address = start; address <= end; address += PGDIR_SIZE) {
- const pgd_t *pgd_ref = pgd_offset_k(address);
+ pgd_t *pgd_ref = pgd_offset_k(address);
+ const p4d_t *p4d_ref;
struct page *page;
- if (pgd_none(*pgd_ref))
+ /*
+ * With folded p4d, pgd_none() is always false, we need to
+ * handle synchonization on p4d level.
+ */
+ BUILD_BUG_ON(pgd_none(*pgd_ref));
+ p4d_ref = p4d_offset(pgd_ref, address);
+
+ if (p4d_none(*p4d_ref))
continue;
spin_lock(&pgd_lock);
list_for_each_entry(page, &pgd_list, lru) {
pgd_t *pgd;
+ p4d_t *p4d;
spinlock_t *pgt_lock;
pgd = (pgd_t *)page_address(page) + pgd_index(address);
+ p4d = p4d_offset(pgd, address);
/* the pgt_lock only for Xen */
pgt_lock = &pgd_page_get_mm(page)->page_table_lock;
spin_lock(pgt_lock);
- if (!pgd_none(*pgd_ref) && !pgd_none(*pgd))
- BUG_ON(pgd_page_vaddr(*pgd)
- != pgd_page_vaddr(*pgd_ref));
+ if (!p4d_none(*p4d_ref) && !p4d_none(*p4d))
+ BUG_ON(p4d_page_vaddr(*p4d)
+ != p4d_page_vaddr(*p4d_ref));
- if (pgd_none(*pgd))
- set_pgd(pgd, *pgd_ref);
+ if (p4d_none(*p4d))
+ set_p4d(p4d, *p4d_ref);
spin_unlock(pgt_lock);
}
@@ -149,16 +159,28 @@
return ptr;
}
-static pud_t *fill_pud(pgd_t *pgd, unsigned long vaddr)
+static p4d_t *fill_p4d(pgd_t *pgd, unsigned long vaddr)
{
if (pgd_none(*pgd)) {
- pud_t *pud = (pud_t *)spp_getpage();
- pgd_populate(&init_mm, pgd, pud);
- if (pud != pud_offset(pgd, 0))
+ p4d_t *p4d = (p4d_t *)spp_getpage();
+ pgd_populate(&init_mm, pgd, p4d);
+ if (p4d != p4d_offset(pgd, 0))
printk(KERN_ERR "PAGETABLE BUG #00! %p <-> %p\n",
- pud, pud_offset(pgd, 0));
+ p4d, p4d_offset(pgd, 0));
}
- return pud_offset(pgd, vaddr);
+ return p4d_offset(pgd, vaddr);
+}
+
+static pud_t *fill_pud(p4d_t *p4d, unsigned long vaddr)
+{
+ if (p4d_none(*p4d)) {
+ pud_t *pud = (pud_t *)spp_getpage();
+ p4d_populate(&init_mm, p4d, pud);
+ if (pud != pud_offset(p4d, 0))
+ printk(KERN_ERR "PAGETABLE BUG #01! %p <-> %p\n",
+ pud, pud_offset(p4d, 0));
+ }
+ return pud_offset(p4d, vaddr);
}
static pmd_t *fill_pmd(pud_t *pud, unsigned long vaddr)
@@ -167,7 +189,7 @@
pmd_t *pmd = (pmd_t *) spp_getpage();
pud_populate(&init_mm, pud, pmd);
if (pmd != pmd_offset(pud, 0))
- printk(KERN_ERR "PAGETABLE BUG #01! %p <-> %p\n",
+ printk(KERN_ERR "PAGETABLE BUG #02! %p <-> %p\n",
pmd, pmd_offset(pud, 0));
}
return pmd_offset(pud, vaddr);
@@ -179,20 +201,15 @@
pte_t *pte = (pte_t *) spp_getpage();
pmd_populate_kernel(&init_mm, pmd, pte);
if (pte != pte_offset_kernel(pmd, 0))
- printk(KERN_ERR "PAGETABLE BUG #02!\n");
+ printk(KERN_ERR "PAGETABLE BUG #03!\n");
}
return pte_offset_kernel(pmd, vaddr);
}
-void set_pte_vaddr_pud(pud_t *pud_page, unsigned long vaddr, pte_t new_pte)
+static void __set_pte_vaddr(pud_t *pud, unsigned long vaddr, pte_t new_pte)
{
- pud_t *pud;
- pmd_t *pmd;
- pte_t *pte;
-
- pud = pud_page + pud_index(vaddr);
- pmd = fill_pmd(pud, vaddr);
- pte = fill_pte(pmd, vaddr);
+ pmd_t *pmd = fill_pmd(pud, vaddr);
+ pte_t *pte = fill_pte(pmd, vaddr);
set_pte(pte, new_pte);
@@ -203,10 +220,25 @@
__flush_tlb_one(vaddr);
}
+void set_pte_vaddr_p4d(p4d_t *p4d_page, unsigned long vaddr, pte_t new_pte)
+{
+ p4d_t *p4d = p4d_page + p4d_index(vaddr);
+ pud_t *pud = fill_pud(p4d, vaddr);
+
+ __set_pte_vaddr(pud, vaddr, new_pte);
+}
+
+void set_pte_vaddr_pud(pud_t *pud_page, unsigned long vaddr, pte_t new_pte)
+{
+ pud_t *pud = pud_page + pud_index(vaddr);
+
+ __set_pte_vaddr(pud, vaddr, new_pte);
+}
+
void set_pte_vaddr(unsigned long vaddr, pte_t pteval)
{
pgd_t *pgd;
- pud_t *pud_page;
+ p4d_t *p4d_page;
pr_debug("set_pte_vaddr %lx to %lx\n", vaddr, native_pte_val(pteval));
@@ -216,17 +248,20 @@
"PGD FIXMAP MISSING, it should be setup in head.S!\n");
return;
}
- pud_page = (pud_t*)pgd_page_vaddr(*pgd);
- set_pte_vaddr_pud(pud_page, vaddr, pteval);
+
+ p4d_page = p4d_offset(pgd, 0);
+ set_pte_vaddr_p4d(p4d_page, vaddr, pteval);
}
pmd_t * __init populate_extra_pmd(unsigned long vaddr)
{
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pgd = pgd_offset_k(vaddr);
- pud = fill_pud(pgd, vaddr);
+ p4d = fill_p4d(pgd, vaddr);
+ pud = fill_pud(p4d, vaddr);
return fill_pmd(pud, vaddr);
}
@@ -245,6 +280,7 @@
enum page_cache_mode cache)
{
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pgprot_t prot;
@@ -255,11 +291,17 @@
for (; size; phys += PMD_SIZE, size -= PMD_SIZE) {
pgd = pgd_offset_k((unsigned long)__va(phys));
if (pgd_none(*pgd)) {
- pud = (pud_t *) spp_getpage();
- set_pgd(pgd, __pgd(__pa(pud) | _KERNPG_TABLE |
+ p4d = (p4d_t *) spp_getpage();
+ set_pgd(pgd, __pgd(__pa(p4d) | _KERNPG_TABLE |
_PAGE_USER));
}
- pud = pud_offset(pgd, (unsigned long)__va(phys));
+ p4d = p4d_offset(pgd, (unsigned long)__va(phys));
+ if (p4d_none(*p4d)) {
+ pud = (pud_t *) spp_getpage();
+ set_p4d(p4d, __p4d(__pa(pud) | _KERNPG_TABLE |
+ _PAGE_USER));
+ }
+ pud = pud_offset(p4d, (unsigned long)__va(phys));
if (pud_none(*pud)) {
pmd = (pmd_t *) spp_getpage();
set_pud(pud, __pud(__pa(pmd) | _KERNPG_TABLE |
@@ -337,10 +379,10 @@
paddr_next = (paddr & PAGE_MASK) + PAGE_SIZE;
if (paddr >= paddr_end) {
if (!after_bootmem &&
- !e820_any_mapped(paddr & PAGE_MASK, paddr_next,
- E820_RAM) &&
- !e820_any_mapped(paddr & PAGE_MASK, paddr_next,
- E820_RESERVED_KERN))
+ !e820__mapped_any(paddr & PAGE_MASK, paddr_next,
+ E820_TYPE_RAM) &&
+ !e820__mapped_any(paddr & PAGE_MASK, paddr_next,
+ E820_TYPE_RESERVED_KERN))
set_pte(pte, __pte(0));
continue;
}
@@ -392,10 +434,10 @@
paddr_next = (paddr & PMD_MASK) + PMD_SIZE;
if (paddr >= paddr_end) {
if (!after_bootmem &&
- !e820_any_mapped(paddr & PMD_MASK, paddr_next,
- E820_RAM) &&
- !e820_any_mapped(paddr & PMD_MASK, paddr_next,
- E820_RESERVED_KERN))
+ !e820__mapped_any(paddr & PMD_MASK, paddr_next,
+ E820_TYPE_RAM) &&
+ !e820__mapped_any(paddr & PMD_MASK, paddr_next,
+ E820_TYPE_RESERVED_KERN))
set_pmd(pmd, __pmd(0));
continue;
}
@@ -478,10 +520,10 @@
if (paddr >= paddr_end) {
if (!after_bootmem &&
- !e820_any_mapped(paddr & PUD_MASK, paddr_next,
- E820_RAM) &&
- !e820_any_mapped(paddr & PUD_MASK, paddr_next,
- E820_RESERVED_KERN))
+ !e820__mapped_any(paddr & PUD_MASK, paddr_next,
+ E820_TYPE_RAM) &&
+ !e820__mapped_any(paddr & PUD_MASK, paddr_next,
+ E820_TYPE_RESERVED_KERN))
set_pud(pud, __pud(0));
continue;
}
@@ -563,12 +605,15 @@
for (; vaddr < vaddr_end; vaddr = vaddr_next) {
pgd_t *pgd = pgd_offset_k(vaddr);
+ p4d_t *p4d;
pud_t *pud;
vaddr_next = (vaddr & PGDIR_MASK) + PGDIR_SIZE;
- if (pgd_val(*pgd)) {
- pud = (pud_t *)pgd_page_vaddr(*pgd);
+ BUILD_BUG_ON(pgd_none(*pgd));
+ p4d = p4d_offset(pgd, vaddr);
+ if (p4d_val(*p4d)) {
+ pud = (pud_t *)p4d_page_vaddr(*p4d);
paddr_last = phys_pud_init(pud, __pa(vaddr),
__pa(vaddr_end),
page_size_mask);
@@ -580,7 +625,7 @@
page_size_mask);
spin_lock(&init_mm.page_table_lock);
- pgd_populate(&init_mm, pgd, pud);
+ p4d_populate(&init_mm, p4d, pud);
spin_unlock(&init_mm.page_table_lock);
pgd_changed = true;
}
@@ -726,6 +771,24 @@
spin_unlock(&init_mm.page_table_lock);
}
+static void __meminit free_pud_table(pud_t *pud_start, p4d_t *p4d)
+{
+ pud_t *pud;
+ int i;
+
+ for (i = 0; i < PTRS_PER_PUD; i++) {
+ pud = pud_start + i;
+ if (!pud_none(*pud))
+ return;
+ }
+
+ /* free a pud talbe */
+ free_pagetable(p4d_page(*p4d), 0);
+ spin_lock(&init_mm.page_table_lock);
+ p4d_clear(p4d);
+ spin_unlock(&init_mm.page_table_lock);
+}
+
static void __meminit
remove_pte_table(pte_t *pte_start, unsigned long addr, unsigned long end,
bool direct)
@@ -899,7 +962,7 @@
continue;
}
- pmd_base = (pmd_t *)pud_page_vaddr(*pud);
+ pmd_base = pmd_offset(pud, 0);
remove_pmd_table(pmd_base, addr, next, direct);
free_pmd_table(pmd_base, pud);
}
@@ -908,6 +971,32 @@
update_page_count(PG_LEVEL_1G, -pages);
}
+static void __meminit
+remove_p4d_table(p4d_t *p4d_start, unsigned long addr, unsigned long end,
+ bool direct)
+{
+ unsigned long next, pages = 0;
+ pud_t *pud_base;
+ p4d_t *p4d;
+
+ p4d = p4d_start + p4d_index(addr);
+ for (; addr < end; addr = next, p4d++) {
+ next = p4d_addr_end(addr, end);
+
+ if (!p4d_present(*p4d))
+ continue;
+
+ BUILD_BUG_ON(p4d_large(*p4d));
+
+ pud_base = pud_offset(p4d, 0);
+ remove_pud_table(pud_base, addr, next, direct);
+ free_pud_table(pud_base, p4d);
+ }
+
+ if (direct)
+ update_page_count(PG_LEVEL_512G, -pages);
+}
+
/* start and end are both virtual address. */
static void __meminit
remove_pagetable(unsigned long start, unsigned long end, bool direct)
@@ -915,7 +1004,7 @@
unsigned long next;
unsigned long addr;
pgd_t *pgd;
- pud_t *pud;
+ p4d_t *p4d;
for (addr = start; addr < end; addr = next) {
next = pgd_addr_end(addr, end);
@@ -924,8 +1013,8 @@
if (!pgd_present(*pgd))
continue;
- pud = (pud_t *)pgd_page_vaddr(*pgd);
- remove_pud_table(pud, addr, next, direct);
+ p4d = p4d_offset(pgd, 0);
+ remove_p4d_table(p4d, addr, next, direct);
}
flush_tlb_all();
@@ -1090,6 +1179,7 @@
{
unsigned long above = ((long)addr) >> __VIRTUAL_MASK_SHIFT;
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
@@ -1101,7 +1191,11 @@
if (pgd_none(*pgd))
return 0;
- pud = pud_offset(pgd, addr);
+ p4d = p4d_offset(pgd, addr);
+ if (p4d_none(*p4d))
+ return 0;
+
+ pud = pud_offset(p4d, addr);
if (pud_none(*pud))
return 0;
@@ -1158,6 +1252,7 @@
unsigned long addr;
unsigned long next;
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
@@ -1168,7 +1263,11 @@
if (!pgd)
return -ENOMEM;
- pud = vmemmap_pud_populate(pgd, addr, node);
+ p4d = vmemmap_p4d_populate(pgd, addr, node);
+ if (!p4d)
+ return -ENOMEM;
+
+ pud = vmemmap_pud_populate(p4d, addr, node);
if (!pud)
return -ENOMEM;
@@ -1236,6 +1335,7 @@
unsigned long end = (unsigned long)(start_page + size);
unsigned long next;
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
unsigned int nr_pages;
@@ -1251,7 +1351,14 @@
}
get_page_bootmem(section_nr, pgd_page(*pgd), MIX_SECTION_INFO);
- pud = pud_offset(pgd, addr);
+ p4d = p4d_offset(pgd, addr);
+ if (p4d_none(*p4d)) {
+ next = (addr + PAGE_SIZE) & PAGE_MASK;
+ continue;
+ }
+ get_page_bootmem(section_nr, p4d_page(*p4d), MIX_SECTION_INFO);
+
+ pud = pud_offset(p4d, addr);
if (pud_none(*pud)) {
next = (addr + PAGE_SIZE) & PAGE_MASK;
continue;
diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c
index 7aaa2635..e4f7b25 100644
--- a/arch/x86/mm/ioremap.c
+++ b/arch/x86/mm/ioremap.c
@@ -9,12 +9,13 @@
#include <linux/bootmem.h>
#include <linux/init.h>
#include <linux/io.h>
+#include <linux/ioport.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
#include <linux/mmiotrace.h>
#include <asm/cacheflush.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/fixmap.h>
#include <asm/pgtable.h>
#include <asm/tlbflush.h>
@@ -425,7 +426,8 @@
/* Don't assume we're using swapper_pg_dir at this point */
pgd_t *base = __va(read_cr3());
pgd_t *pgd = &base[pgd_index(addr)];
- pud_t *pud = pud_offset(pgd, addr);
+ p4d_t *p4d = p4d_offset(pgd, addr);
+ pud_t *pud = pud_offset(p4d, addr);
pmd_t *pmd = pmd_offset(pud, addr);
return pmd;
diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index 4c90cfd..0c7d812 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -8,11 +8,12 @@
#include <linux/sched/task.h>
#include <linux/vmalloc.h>
+#include <asm/e820/types.h>
#include <asm/tlbflush.h>
#include <asm/sections.h>
extern pgd_t early_level4_pgt[PTRS_PER_PGD];
-extern struct range pfn_mapped[E820_X_MAX];
+extern struct range pfn_mapped[E820_MAX_ENTRIES];
static int __init map_range(struct range *range)
{
@@ -33,8 +34,19 @@
static void __init clear_pgds(unsigned long start,
unsigned long end)
{
- for (; start < end; start += PGDIR_SIZE)
- pgd_clear(pgd_offset_k(start));
+ pgd_t *pgd;
+
+ for (; start < end; start += PGDIR_SIZE) {
+ pgd = pgd_offset_k(start);
+ /*
+ * With folded p4d, pgd_clear() is nop, use p4d_clear()
+ * instead.
+ */
+ if (CONFIG_PGTABLE_LEVELS < 5)
+ p4d_clear(p4d_offset(pgd, start));
+ else
+ pgd_clear(pgd);
+ }
}
static void __init kasan_map_early_shadow(pgd_t *pgd)
@@ -44,8 +56,18 @@
unsigned long end = KASAN_SHADOW_END;
for (i = pgd_index(start); start < end; i++) {
- pgd[i] = __pgd(__pa_nodebug(kasan_zero_pud)
- | _KERNPG_TABLE);
+ switch (CONFIG_PGTABLE_LEVELS) {
+ case 4:
+ pgd[i] = __pgd(__pa_nodebug(kasan_zero_pud) |
+ _KERNPG_TABLE);
+ break;
+ case 5:
+ pgd[i] = __pgd(__pa_nodebug(kasan_zero_p4d) |
+ _KERNPG_TABLE);
+ break;
+ default:
+ BUILD_BUG();
+ }
start += PGDIR_SIZE;
}
}
@@ -73,6 +95,7 @@
pteval_t pte_val = __pa_nodebug(kasan_zero_page) | __PAGE_KERNEL;
pmdval_t pmd_val = __pa_nodebug(kasan_zero_pte) | _KERNPG_TABLE;
pudval_t pud_val = __pa_nodebug(kasan_zero_pmd) | _KERNPG_TABLE;
+ p4dval_t p4d_val = __pa_nodebug(kasan_zero_pud) | _KERNPG_TABLE;
for (i = 0; i < PTRS_PER_PTE; i++)
kasan_zero_pte[i] = __pte(pte_val);
@@ -83,6 +106,9 @@
for (i = 0; i < PTRS_PER_PUD; i++)
kasan_zero_pud[i] = __pud(pud_val);
+ for (i = 0; CONFIG_PGTABLE_LEVELS >= 5 && i < PTRS_PER_P4D; i++)
+ kasan_zero_p4d[i] = __p4d(p4d_val);
+
kasan_map_early_shadow(early_level4_pgt);
kasan_map_early_shadow(init_level4_pgt);
}
@@ -104,7 +130,7 @@
kasan_populate_zero_shadow((void *)KASAN_SHADOW_START,
kasan_mem_to_shadow((void *)PAGE_OFFSET));
- for (i = 0; i < E820_X_MAX; i++) {
+ for (i = 0; i < E820_MAX_ENTRIES; i++) {
if (pfn_mapped[i].end == 0)
break;
diff --git a/arch/x86/mm/mmap.c b/arch/x86/mm/mmap.c
index 7940166..19ad095 100644
--- a/arch/x86/mm/mmap.c
+++ b/arch/x86/mm/mmap.c
@@ -30,30 +30,44 @@
#include <linux/limits.h>
#include <linux/sched/signal.h>
#include <linux/sched/mm.h>
+#include <linux/compat.h>
#include <asm/elf.h>
struct va_alignment __read_mostly va_align = {
.flags = -1,
};
-static unsigned long stack_maxrandom_size(void)
+unsigned long tasksize_32bit(void)
+{
+ return IA32_PAGE_OFFSET;
+}
+
+unsigned long tasksize_64bit(void)
+{
+ return TASK_SIZE_MAX;
+}
+
+static unsigned long stack_maxrandom_size(unsigned long task_size)
{
unsigned long max = 0;
if ((current->flags & PF_RANDOMIZE) &&
!(current->personality & ADDR_NO_RANDOMIZE)) {
- max = ((-1UL) & STACK_RND_MASK) << PAGE_SHIFT;
+ max = (-1UL) & __STACK_RND_MASK(task_size == tasksize_32bit());
+ max <<= PAGE_SHIFT;
}
return max;
}
-/*
- * Top of mmap area (just below the process stack).
- *
- * Leave an at least ~128 MB hole with possible stack randomization.
- */
-#define MIN_GAP (128*1024*1024UL + stack_maxrandom_size())
-#define MAX_GAP (TASK_SIZE/6*5)
+#ifdef CONFIG_COMPAT
+# define mmap32_rnd_bits mmap_rnd_compat_bits
+# define mmap64_rnd_bits mmap_rnd_bits
+#else
+# define mmap32_rnd_bits mmap_rnd_bits
+# define mmap64_rnd_bits mmap_rnd_bits
+#endif
+
+#define SIZE_128M (128 * 1024 * 1024UL)
static int mmap_is_legacy(void)
{
@@ -66,54 +80,91 @@
return sysctl_legacy_va_layout;
}
-unsigned long arch_mmap_rnd(void)
+static unsigned long arch_rnd(unsigned int rndbits)
{
- unsigned long rnd;
-
- if (mmap_is_ia32())
-#ifdef CONFIG_COMPAT
- rnd = get_random_long() & ((1UL << mmap_rnd_compat_bits) - 1);
-#else
- rnd = get_random_long() & ((1UL << mmap_rnd_bits) - 1);
-#endif
- else
- rnd = get_random_long() & ((1UL << mmap_rnd_bits) - 1);
-
- return rnd << PAGE_SHIFT;
+ return (get_random_long() & ((1UL << rndbits) - 1)) << PAGE_SHIFT;
}
-static unsigned long mmap_base(unsigned long rnd)
+unsigned long arch_mmap_rnd(void)
+{
+ if (!(current->flags & PF_RANDOMIZE))
+ return 0;
+ return arch_rnd(mmap_is_ia32() ? mmap32_rnd_bits : mmap64_rnd_bits);
+}
+
+static unsigned long mmap_base(unsigned long rnd, unsigned long task_size)
{
unsigned long gap = rlimit(RLIMIT_STACK);
+ unsigned long gap_min, gap_max;
- if (gap < MIN_GAP)
- gap = MIN_GAP;
- else if (gap > MAX_GAP)
- gap = MAX_GAP;
+ /*
+ * Top of mmap area (just below the process stack).
+ * Leave an at least ~128 MB hole with possible stack randomization.
+ */
+ gap_min = SIZE_128M + stack_maxrandom_size(task_size);
+ gap_max = (task_size / 6) * 5;
- return PAGE_ALIGN(TASK_SIZE - gap - rnd);
+ if (gap < gap_min)
+ gap = gap_min;
+ else if (gap > gap_max)
+ gap = gap_max;
+
+ return PAGE_ALIGN(task_size - gap - rnd);
+}
+
+static unsigned long mmap_legacy_base(unsigned long rnd,
+ unsigned long task_size)
+{
+ return __TASK_UNMAPPED_BASE(task_size) + rnd;
}
/*
* This function, called very early during the creation of a new
* process VM image, sets up which VM layout function to use:
*/
+static void arch_pick_mmap_base(unsigned long *base, unsigned long *legacy_base,
+ unsigned long random_factor, unsigned long task_size)
+{
+ *legacy_base = mmap_legacy_base(random_factor, task_size);
+ if (mmap_is_legacy())
+ *base = *legacy_base;
+ else
+ *base = mmap_base(random_factor, task_size);
+}
+
void arch_pick_mmap_layout(struct mm_struct *mm)
{
- unsigned long random_factor = 0UL;
-
- if (current->flags & PF_RANDOMIZE)
- random_factor = arch_mmap_rnd();
-
- mm->mmap_legacy_base = TASK_UNMAPPED_BASE + random_factor;
-
- if (mmap_is_legacy()) {
- mm->mmap_base = mm->mmap_legacy_base;
+ if (mmap_is_legacy())
mm->get_unmapped_area = arch_get_unmapped_area;
- } else {
- mm->mmap_base = mmap_base(random_factor);
+ else
mm->get_unmapped_area = arch_get_unmapped_area_topdown;
+
+ arch_pick_mmap_base(&mm->mmap_base, &mm->mmap_legacy_base,
+ arch_rnd(mmap64_rnd_bits), tasksize_64bit());
+
+#ifdef CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES
+ /*
+ * The mmap syscall mapping base decision depends solely on the
+ * syscall type (64-bit or compat). This applies for 64bit
+ * applications and 32bit applications. The 64bit syscall uses
+ * mmap_base, the compat syscall uses mmap_compat_base.
+ */
+ arch_pick_mmap_base(&mm->mmap_compat_base, &mm->mmap_compat_legacy_base,
+ arch_rnd(mmap32_rnd_bits), tasksize_32bit());
+#endif
+}
+
+unsigned long get_mmap_base(int is_legacy)
+{
+ struct mm_struct *mm = current->mm;
+
+#ifdef CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES
+ if (in_compat_syscall()) {
+ return is_legacy ? mm->mmap_compat_legacy_base
+ : mm->mmap_compat_base;
}
+#endif
+ return is_legacy ? mm->mmap_legacy_base : mm->mmap_base;
}
const char *arch_vma_name(struct vm_area_struct *vma)
diff --git a/arch/x86/mm/mmio-mod.c b/arch/x86/mm/mmio-mod.c
index bef3662..4d434dd 100644
--- a/arch/x86/mm/mmio-mod.c
+++ b/arch/x86/mm/mmio-mod.c
@@ -32,7 +32,7 @@
#include <linux/kallsyms.h>
#include <asm/pgtable.h>
#include <linux/mmiotrace.h>
-#include <asm/e820.h> /* for ISA_START_ADDRESS */
+#include <asm/e820/api.h> /* for ISA_START_ADDRESS */
#include <linux/atomic.h>
#include <linux/percpu.h>
#include <linux/cpu.h>
diff --git a/arch/x86/mm/mpx.c b/arch/x86/mm/mpx.c
index cd44ae7..1c34b76 100644
--- a/arch/x86/mm/mpx.c
+++ b/arch/x86/mm/mpx.c
@@ -526,15 +526,7 @@
if (!kernel_managing_mpx_tables(current->mm))
return -EINVAL;
- if (do_mpx_bt_fault()) {
- force_sig(SIGSEGV, current);
- /*
- * The force_sig() is essentially "handling" this
- * exception, so we do not pass up the error
- * from do_mpx_bt_fault().
- */
- }
- return 0;
+ return do_mpx_bt_fault();
}
/*
diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c
index 12dcad7..25504d5 100644
--- a/arch/x86/mm/numa.c
+++ b/arch/x86/mm/numa.c
@@ -12,7 +12,7 @@
#include <linux/sched.h>
#include <linux/topology.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/proto.h>
#include <asm/dma.h>
#include <asm/amd_nb.h>
@@ -201,7 +201,7 @@
nd_pa = __memblock_alloc_base(nd_size, SMP_CACHE_BYTES,
MEMBLOCK_ALLOC_ACCESSIBLE);
if (!nd_pa) {
- pr_err("Cannot find %zu bytes in node %d\n",
+ pr_err("Cannot find %zu bytes in any node (initial node: %d)\n",
nd_size, nid);
return;
}
@@ -225,7 +225,7 @@
* numa_cleanup_meminfo - Cleanup a numa_meminfo
* @mi: numa_meminfo to clean up
*
- * Sanitize @mi by merging and removing unncessary memblks. Also check for
+ * Sanitize @mi by merging and removing unnecessary memblks. Also check for
* conflicts and clear unused memblks.
*
* RETURNS:
diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c
index 28d4213..56b22fa 100644
--- a/arch/x86/mm/pageattr.c
+++ b/arch/x86/mm/pageattr.c
@@ -15,7 +15,7 @@
#include <linux/pci.h>
#include <linux/vmalloc.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/processor.h>
#include <asm/tlbflush.h>
#include <asm/sections.h>
@@ -346,6 +346,7 @@
pte_t *lookup_address_in_pgd(pgd_t *pgd, unsigned long address,
unsigned int *level)
{
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
@@ -354,7 +355,15 @@
if (pgd_none(*pgd))
return NULL;
- pud = pud_offset(pgd, address);
+ p4d = p4d_offset(pgd, address);
+ if (p4d_none(*p4d))
+ return NULL;
+
+ *level = PG_LEVEL_512G;
+ if (p4d_large(*p4d) || !p4d_present(*p4d))
+ return (pte_t *)p4d;
+
+ pud = pud_offset(p4d, address);
if (pud_none(*pud))
return NULL;
@@ -406,13 +415,18 @@
pmd_t *lookup_pmd_address(unsigned long address)
{
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pgd = pgd_offset_k(address);
if (pgd_none(*pgd))
return NULL;
- pud = pud_offset(pgd, address);
+ p4d = p4d_offset(pgd, address);
+ if (p4d_none(*p4d) || p4d_large(*p4d) || !p4d_present(*p4d))
+ return NULL;
+
+ pud = pud_offset(p4d, address);
if (pud_none(*pud) || pud_large(*pud) || !pud_present(*pud))
return NULL;
@@ -477,11 +491,13 @@
list_for_each_entry(page, &pgd_list, lru) {
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pgd = (pgd_t *)page_address(page) + pgd_index(address);
- pud = pud_offset(pgd, address);
+ p4d = p4d_offset(pgd, address);
+ pud = pud_offset(p4d, address);
pmd = pmd_offset(pud, address);
set_pte_atomic((pte_t *)pmd, pte);
}
@@ -836,9 +852,9 @@
pud_clear(pud);
}
-static void unmap_pud_range(pgd_t *pgd, unsigned long start, unsigned long end)
+static void unmap_pud_range(p4d_t *p4d, unsigned long start, unsigned long end)
{
- pud_t *pud = pud_offset(pgd, start);
+ pud_t *pud = pud_offset(p4d, start);
/*
* Not on a GB page boundary?
@@ -1004,8 +1020,8 @@
return num_pages;
}
-static long populate_pud(struct cpa_data *cpa, unsigned long start, pgd_t *pgd,
- pgprot_t pgprot)
+static int populate_pud(struct cpa_data *cpa, unsigned long start, p4d_t *p4d,
+ pgprot_t pgprot)
{
pud_t *pud;
unsigned long end;
@@ -1026,7 +1042,7 @@
cur_pages = (pre_end - start) >> PAGE_SHIFT;
cur_pages = min_t(int, (int)cpa->numpages, cur_pages);
- pud = pud_offset(pgd, start);
+ pud = pud_offset(p4d, start);
/*
* Need a PMD page?
@@ -1047,7 +1063,7 @@
if (cpa->numpages == cur_pages)
return cur_pages;
- pud = pud_offset(pgd, start);
+ pud = pud_offset(p4d, start);
pud_pgprot = pgprot_4k_2_large(pgprot);
/*
@@ -1067,7 +1083,7 @@
if (start < end) {
long tmp;
- pud = pud_offset(pgd, start);
+ pud = pud_offset(p4d, start);
if (pud_none(*pud))
if (alloc_pmd_page(pud))
return -1;
@@ -1090,33 +1106,43 @@
{
pgprot_t pgprot = __pgprot(_KERNPG_TABLE);
pud_t *pud = NULL; /* shut up gcc */
+ p4d_t *p4d;
pgd_t *pgd_entry;
long ret;
pgd_entry = cpa->pgd + pgd_index(addr);
+ if (pgd_none(*pgd_entry)) {
+ p4d = (p4d_t *)get_zeroed_page(GFP_KERNEL | __GFP_NOTRACK);
+ if (!p4d)
+ return -1;
+
+ set_pgd(pgd_entry, __pgd(__pa(p4d) | _KERNPG_TABLE));
+ }
+
/*
* Allocate a PUD page and hand it down for mapping.
*/
- if (pgd_none(*pgd_entry)) {
+ p4d = p4d_offset(pgd_entry, addr);
+ if (p4d_none(*p4d)) {
pud = (pud_t *)get_zeroed_page(GFP_KERNEL | __GFP_NOTRACK);
if (!pud)
return -1;
- set_pgd(pgd_entry, __pgd(__pa(pud) | _KERNPG_TABLE));
+ set_p4d(p4d, __p4d(__pa(pud) | _KERNPG_TABLE));
}
pgprot_val(pgprot) &= ~pgprot_val(cpa->mask_clr);
pgprot_val(pgprot) |= pgprot_val(cpa->mask_set);
- ret = populate_pud(cpa, addr, pgd_entry, pgprot);
+ ret = populate_pud(cpa, addr, p4d, pgprot);
if (ret < 0) {
/*
* Leave the PUD page in place in case some other CPU or thread
* already found it, but remove any useless entries we just
* added to it.
*/
- unmap_pud_range(pgd_entry, addr,
+ unmap_pud_range(p4d, addr,
addr + (cpa->numpages << PAGE_SHIFT));
return ret;
}
diff --git a/arch/x86/mm/pat.c b/arch/x86/mm/pat.c
index efc32bc..9b78685 100644
--- a/arch/x86/mm/pat.c
+++ b/arch/x86/mm/pat.c
@@ -10,6 +10,7 @@
#include <linux/seq_file.h>
#include <linux/bootmem.h>
#include <linux/debugfs.h>
+#include <linux/ioport.h>
#include <linux/kernel.h>
#include <linux/pfn_t.h>
#include <linux/slab.h>
@@ -23,7 +24,7 @@
#include <asm/x86_init.h>
#include <asm/pgtable.h>
#include <asm/fcntl.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/mtrr.h>
#include <asm/page.h>
#include <asm/msr.h>
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index 6cbdff2..508a708 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -81,6 +81,14 @@
paravirt_release_pud(__pa(pud) >> PAGE_SHIFT);
tlb_remove_page(tlb, virt_to_page(pud));
}
+
+#if CONFIG_PGTABLE_LEVELS > 4
+void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d)
+{
+ paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT);
+ tlb_remove_page(tlb, virt_to_page(p4d));
+}
+#endif /* CONFIG_PGTABLE_LEVELS > 4 */
#endif /* CONFIG_PGTABLE_LEVELS > 3 */
#endif /* CONFIG_PGTABLE_LEVELS > 2 */
@@ -120,7 +128,7 @@
references from swapper_pg_dir. */
if (CONFIG_PGTABLE_LEVELS == 2 ||
(CONFIG_PGTABLE_LEVELS == 3 && SHARED_KERNEL_PMD) ||
- CONFIG_PGTABLE_LEVELS == 4) {
+ CONFIG_PGTABLE_LEVELS >= 4) {
clone_pgd_range(pgd + KERNEL_PGD_BOUNDARY,
swapper_pg_dir + KERNEL_PGD_BOUNDARY,
KERNEL_PGD_PTRS);
@@ -261,13 +269,15 @@
static void pgd_prepopulate_pmd(struct mm_struct *mm, pgd_t *pgd, pmd_t *pmds[])
{
+ p4d_t *p4d;
pud_t *pud;
int i;
if (PREALLOCATED_PMDS == 0) /* Work around gcc-3.4.x bug */
return;
- pud = pud_offset(pgd, 0);
+ p4d = p4d_offset(pgd, 0);
+ pud = pud_offset(p4d, 0);
for (i = 0; i < PREALLOCATED_PMDS; i++, pud++) {
pmd_t *pmd = pmds[i];
@@ -580,6 +590,28 @@
}
#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP
+#ifdef CONFIG_X86_5LEVEL
+/**
+ * p4d_set_huge - setup kernel P4D mapping
+ *
+ * No 512GB pages yet -- always return 0
+ */
+int p4d_set_huge(p4d_t *p4d, phys_addr_t addr, pgprot_t prot)
+{
+ return 0;
+}
+
+/**
+ * p4d_clear_huge - clear kernel P4D mapping when it is set
+ *
+ * No 512GB pages yet -- always return 0
+ */
+int p4d_clear_huge(p4d_t *p4d)
+{
+ return 0;
+}
+#endif
+
/**
* pud_set_huge - setup kernel PUD mapping
*
diff --git a/arch/x86/mm/pgtable_32.c b/arch/x86/mm/pgtable_32.c
index 9adce77..b9bd5b8 100644
--- a/arch/x86/mm/pgtable_32.c
+++ b/arch/x86/mm/pgtable_32.c
@@ -12,7 +12,7 @@
#include <asm/pgtable.h>
#include <asm/pgalloc.h>
#include <asm/fixmap.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/tlb.h>
#include <asm/tlbflush.h>
#include <asm/io.h>
@@ -26,6 +26,7 @@
void set_pte_vaddr(unsigned long vaddr, pte_t pteval)
{
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
@@ -35,7 +36,12 @@
BUG();
return;
}
- pud = pud_offset(pgd, vaddr);
+ p4d = p4d_offset(pgd, vaddr);
+ if (p4d_none(*p4d)) {
+ BUG();
+ return;
+ }
+ pud = pud_offset(p4d, vaddr);
if (pud_none(*pud)) {
BUG();
return;
diff --git a/arch/x86/mm/srat.c b/arch/x86/mm/srat.c
index 35fe695..3ea20d6 100644
--- a/arch/x86/mm/srat.c
+++ b/arch/x86/mm/srat.c
@@ -18,7 +18,7 @@
#include <linux/mm.h>
#include <asm/proto.h>
#include <asm/numa.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/apic.h>
#include <asm/uv/uv.h>
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index a7655f6..6e7bedf 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -263,8 +263,6 @@
{
struct flush_tlb_info info;
- if (end == 0)
- end = start + PAGE_SIZE;
info.flush_mm = mm;
info.flush_start = start;
info.flush_end = end;
@@ -289,23 +287,6 @@
smp_call_function_many(cpumask, flush_tlb_func, &info, 1);
}
-void flush_tlb_current_task(void)
-{
- struct mm_struct *mm = current->mm;
-
- preempt_disable();
-
- count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL);
-
- /* This is an implicit full barrier that synchronizes with switch_mm. */
- local_flush_tlb();
-
- trace_tlb_flush(TLB_LOCAL_SHOOTDOWN, TLB_FLUSH_ALL);
- if (cpumask_any_but(mm_cpumask(mm), smp_processor_id()) < nr_cpu_ids)
- flush_tlb_others(mm_cpumask(mm), mm, 0UL, TLB_FLUSH_ALL);
- preempt_enable();
-}
-
/*
* See Documentation/x86/tlb.txt for details. We choose 33
* because it is large enough to cover the vast majority (at
@@ -326,6 +307,12 @@
unsigned long base_pages_to_flush = TLB_FLUSH_ALL;
preempt_disable();
+
+ if ((end != TLB_FLUSH_ALL) && !(vmflag & VM_HUGETLB))
+ base_pages_to_flush = (end - start) >> PAGE_SHIFT;
+ if (base_pages_to_flush > tlb_single_page_flush_ceiling)
+ base_pages_to_flush = TLB_FLUSH_ALL;
+
if (current->active_mm != mm) {
/* Synchronize with switch_mm. */
smp_mb();
@@ -342,15 +329,11 @@
goto out;
}
- if ((end != TLB_FLUSH_ALL) && !(vmflag & VM_HUGETLB))
- base_pages_to_flush = (end - start) >> PAGE_SHIFT;
-
/*
* Both branches below are implicit full barriers (MOV to CR or
* INVLPG) that synchronize with switch_mm.
*/
- if (base_pages_to_flush > tlb_single_page_flush_ceiling) {
- base_pages_to_flush = TLB_FLUSH_ALL;
+ if (base_pages_to_flush == TLB_FLUSH_ALL) {
count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL);
local_flush_tlb();
} else {
@@ -393,7 +376,7 @@
}
if (cpumask_any_but(mm_cpumask(mm), smp_processor_id()) < nr_cpu_ids)
- flush_tlb_others(mm_cpumask(mm), mm, start, 0UL);
+ flush_tlb_others(mm_cpumask(mm), mm, start, start + PAGE_SIZE);
preempt_enable();
}
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 32322ce..14f840d 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -490,13 +490,6 @@
break;
case BPF_LD | BPF_IMM | BPF_DW:
- if (insn[1].code != 0 || insn[1].src_reg != 0 ||
- insn[1].dst_reg != 0 || insn[1].off != 0) {
- /* verifier must catch invalid insns */
- pr_err("invalid BPF_LD_IMM64 insn\n");
- return -EINVAL;
- }
-
/* optimization: if imm64 is zero, use 'xor <dst>,<dst>'
* to save 7 bytes.
*/
diff --git a/arch/x86/pci/i386.c b/arch/x86/pci/i386.c
index 0a9f2ca..6fa84d5 100644
--- a/arch/x86/pci/i386.c
+++ b/arch/x86/pci/i386.c
@@ -34,7 +34,7 @@
#include <linux/bootmem.h>
#include <asm/pat.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/pci_x86.h>
#include <asm/io_apic.h>
@@ -398,7 +398,7 @@
list_for_each_entry(bus, &pci_root_buses, node)
pcibios_allocate_resources(bus, 1);
- e820_reserve_resources_late();
+ e820__reserve_resources_late();
/*
* Insert the IO APIC resources after PCI initialization has
* occurred to handle IO APICS that are mapped in on a BAR in
diff --git a/arch/x86/pci/mmconfig-shared.c b/arch/x86/pci/mmconfig-shared.c
index dd30b7e..d1b47d5 100644
--- a/arch/x86/pci/mmconfig-shared.c
+++ b/arch/x86/pci/mmconfig-shared.c
@@ -18,7 +18,7 @@
#include <linux/slab.h>
#include <linux/mutex.h>
#include <linux/rculist.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/pci_x86.h>
#include <asm/acpi.h>
@@ -423,7 +423,7 @@
return AE_OK;
}
-static int is_acpi_reserved(u64 start, u64 end, unsigned not_used)
+static bool is_acpi_reserved(u64 start, u64 end, unsigned not_used)
{
struct resource mcfg_res;
@@ -440,11 +440,11 @@
return mcfg_res.flags;
}
-typedef int (*check_reserved_t)(u64 start, u64 end, unsigned type);
+typedef bool (*check_reserved_t)(u64 start, u64 end, unsigned type);
-static int __ref is_mmconf_reserved(check_reserved_t is_reserved,
- struct pci_mmcfg_region *cfg,
- struct device *dev, int with_e820)
+static bool __ref is_mmconf_reserved(check_reserved_t is_reserved,
+ struct pci_mmcfg_region *cfg,
+ struct device *dev, int with_e820)
{
u64 addr = cfg->res.start;
u64 size = resource_size(&cfg->res);
@@ -452,7 +452,7 @@
int num_buses;
char *method = with_e820 ? "E820" : "ACPI motherboard resources";
- while (!is_reserved(addr, addr + size, E820_RESERVED)) {
+ while (!is_reserved(addr, addr + size, E820_TYPE_RESERVED)) {
size >>= 1;
if (size < (16UL<<20))
break;
@@ -494,8 +494,8 @@
return 1;
}
-static int __ref pci_mmcfg_check_reserved(struct device *dev,
- struct pci_mmcfg_region *cfg, int early)
+static bool __ref
+pci_mmcfg_check_reserved(struct device *dev, struct pci_mmcfg_region *cfg, int early)
{
if (!early && !acpi_disabled) {
if (is_mmconf_reserved(is_acpi_reserved, cfg, dev, 0))
@@ -514,7 +514,7 @@
}
/*
- * e820_all_mapped() is marked as __init.
+ * e820__mapped_all() is marked as __init.
* All entries from ACPI MCFG table have been checked at boot time.
* For MCFG information constructed from hotpluggable host bridge's
* _CBA method, just assume it's reserved.
@@ -525,7 +525,7 @@
/* Don't try to do this check unless configuration
type 1 is available. how about type 2 ?*/
if (raw_pci_ops)
- return is_mmconf_reserved(e820_all_mapped, cfg, dev, 1);
+ return is_mmconf_reserved(e820__mapped_all, cfg, dev, 1);
return 0;
}
diff --git a/arch/x86/pci/mmconfig_32.c b/arch/x86/pci/mmconfig_32.c
index 43984bc..3e9e166 100644
--- a/arch/x86/pci/mmconfig_32.c
+++ b/arch/x86/pci/mmconfig_32.c
@@ -12,7 +12,7 @@
#include <linux/pci.h>
#include <linux/init.h>
#include <linux/rcupdate.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/pci_x86.h>
/* Assume systems with more busses have correct MCFG */
diff --git a/arch/x86/pci/mmconfig_64.c b/arch/x86/pci/mmconfig_64.c
index bea5249..f1c1aa0 100644
--- a/arch/x86/pci/mmconfig_64.c
+++ b/arch/x86/pci/mmconfig_64.c
@@ -10,7 +10,7 @@
#include <linux/acpi.h>
#include <linux/bitmap.h>
#include <linux/rcupdate.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/pci_x86.h>
#define PREFIX "PCI: "
diff --git a/arch/x86/pci/pcbios.c b/arch/x86/pci/pcbios.c
index 1d97cea..29e9ba6 100644
--- a/arch/x86/pci/pcbios.c
+++ b/arch/x86/pci/pcbios.c
@@ -7,7 +7,9 @@
#include <linux/slab.h>
#include <linux/module.h>
#include <linux/uaccess.h>
+
#include <asm/pci_x86.h>
+#include <asm/e820/types.h>
#include <asm/pci-functions.h>
#include <asm/cacheflush.h>
diff --git a/arch/x86/platform/efi/Makefile b/arch/x86/platform/efi/Makefile
index 066619b..f1d83b3 100644
--- a/arch/x86/platform/efi/Makefile
+++ b/arch/x86/platform/efi/Makefile
@@ -1,6 +1,5 @@
OBJECT_FILES_NON_STANDARD_efi_thunk_$(BITS).o := y
obj-$(CONFIG_EFI) += quirks.o efi.o efi_$(BITS).o efi_stub_$(BITS).o
-obj-$(CONFIG_ACPI_BGRT) += efi-bgrt.o
obj-$(CONFIG_EARLY_PRINTK_EFI) += early_printk.o
obj-$(CONFIG_EFI_MIXED) += efi_thunk_$(BITS).o
diff --git a/arch/x86/platform/efi/efi.c b/arch/x86/platform/efi/efi.c
index 565dff3..a15cf81 100644
--- a/arch/x86/platform/efi/efi.c
+++ b/arch/x86/platform/efi/efi.c
@@ -47,6 +47,7 @@
#include <asm/setup.h>
#include <asm/efi.h>
+#include <asm/e820/api.h>
#include <asm/time.h>
#include <asm/cacheflush.h>
#include <asm/tlbflush.h>
@@ -139,21 +140,21 @@
case EFI_BOOT_SERVICES_DATA:
case EFI_CONVENTIONAL_MEMORY:
if (md->attribute & EFI_MEMORY_WB)
- e820_type = E820_RAM;
+ e820_type = E820_TYPE_RAM;
else
- e820_type = E820_RESERVED;
+ e820_type = E820_TYPE_RESERVED;
break;
case EFI_ACPI_RECLAIM_MEMORY:
- e820_type = E820_ACPI;
+ e820_type = E820_TYPE_ACPI;
break;
case EFI_ACPI_MEMORY_NVS:
- e820_type = E820_NVS;
+ e820_type = E820_TYPE_NVS;
break;
case EFI_UNUSABLE_MEMORY:
- e820_type = E820_UNUSABLE;
+ e820_type = E820_TYPE_UNUSABLE;
break;
case EFI_PERSISTENT_MEMORY:
- e820_type = E820_PMEM;
+ e820_type = E820_TYPE_PMEM;
break;
default:
/*
@@ -161,12 +162,12 @@
* EFI_RUNTIME_SERVICES_DATA EFI_MEMORY_MAPPED_IO
* EFI_MEMORY_MAPPED_IO_PORT_SPACE EFI_PAL_CODE
*/
- e820_type = E820_RESERVED;
+ e820_type = E820_TYPE_RESERVED;
break;
}
- e820_add_region(start, size, e820_type);
+ e820__range_add(start, size, e820_type);
}
- sanitize_e820_map(e820->map, ARRAY_SIZE(e820->map), &e820->nr_map);
+ e820__update_table(e820_table);
}
int __init efi_memblock_x86_reserve_range(void)
diff --git a/arch/x86/platform/efi/efi_32.c b/arch/x86/platform/efi/efi_32.c
index cef39b0..3481268 100644
--- a/arch/x86/platform/efi/efi_32.c
+++ b/arch/x86/platform/efi/efi_32.c
@@ -68,7 +68,7 @@
load_cr3(initial_page_table);
__flush_tlb_all();
- gdt_descr.address = __pa(get_cpu_gdt_table(0));
+ gdt_descr.address = get_cpu_gdt_paddr(0);
gdt_descr.size = GDT_SIZE - 1;
load_gdt(&gdt_descr);
@@ -79,7 +79,7 @@
{
struct desc_ptr gdt_descr;
- gdt_descr.address = (unsigned long)get_cpu_gdt_table(0);
+ gdt_descr.address = (unsigned long)get_cpu_gdt_rw(0);
gdt_descr.size = GDT_SIZE - 1;
load_gdt(&gdt_descr);
diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
index a4695da..c488625 100644
--- a/arch/x86/platform/efi/efi_64.c
+++ b/arch/x86/platform/efi/efi_64.c
@@ -35,7 +35,7 @@
#include <asm/setup.h>
#include <asm/page.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/pgtable.h>
#include <asm/tlbflush.h>
#include <asm/proto.h>
@@ -47,7 +47,7 @@
#include <asm/pgalloc.h>
/*
- * We allocate runtime services regions bottom-up, starting from -4G, i.e.
+ * We allocate runtime services regions top-down, starting from -4G, i.e.
* 0xffff_ffff_0000_0000 and limit EFI VA mapping space to 64G.
*/
static u64 efi_va = EFI_VA_START;
@@ -135,6 +135,7 @@
int __init efi_alloc_page_tables(void)
{
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
gfp_t gfp_mask;
@@ -147,14 +148,19 @@
return -ENOMEM;
pgd = efi_pgd + pgd_index(EFI_VA_END);
-
- pud = pud_alloc_one(NULL, 0);
- if (!pud) {
+ p4d = p4d_alloc(&init_mm, pgd, EFI_VA_END);
+ if (!p4d) {
free_page((unsigned long)efi_pgd);
return -ENOMEM;
}
- pgd_populate(NULL, pgd, pud);
+ pud = pud_alloc(&init_mm, p4d, EFI_VA_END);
+ if (!pud) {
+ if (CONFIG_PGTABLE_LEVELS > 4)
+ free_page((unsigned long) pgd_page_vaddr(*pgd));
+ free_page((unsigned long)efi_pgd);
+ return -ENOMEM;
+ }
return 0;
}
@@ -166,6 +172,7 @@
{
unsigned num_entries;
pgd_t *pgd_k, *pgd_efi;
+ p4d_t *p4d_k, *p4d_efi;
pud_t *pud_k, *pud_efi;
if (efi_enabled(EFI_OLD_MEMMAP))
@@ -190,23 +197,37 @@
memcpy(pgd_efi, pgd_k, sizeof(pgd_t) * num_entries);
/*
+ * As with PGDs, we share all P4D entries apart from the one entry
+ * that covers the EFI runtime mapping space.
+ */
+ BUILD_BUG_ON(p4d_index(EFI_VA_END) != p4d_index(MODULES_END));
+ BUILD_BUG_ON((EFI_VA_START & P4D_MASK) != (EFI_VA_END & P4D_MASK));
+
+ pgd_efi = efi_pgd + pgd_index(EFI_VA_END);
+ pgd_k = pgd_offset_k(EFI_VA_END);
+ p4d_efi = p4d_offset(pgd_efi, 0);
+ p4d_k = p4d_offset(pgd_k, 0);
+
+ num_entries = p4d_index(EFI_VA_END);
+ memcpy(p4d_efi, p4d_k, sizeof(p4d_t) * num_entries);
+
+ /*
* We share all the PUD entries apart from those that map the
* EFI regions. Copy around them.
*/
BUILD_BUG_ON((EFI_VA_START & ~PUD_MASK) != 0);
BUILD_BUG_ON((EFI_VA_END & ~PUD_MASK) != 0);
- pgd_efi = efi_pgd + pgd_index(EFI_VA_END);
- pud_efi = pud_offset(pgd_efi, 0);
-
- pgd_k = pgd_offset_k(EFI_VA_END);
- pud_k = pud_offset(pgd_k, 0);
+ p4d_efi = p4d_offset(pgd_efi, EFI_VA_END);
+ p4d_k = p4d_offset(pgd_k, EFI_VA_END);
+ pud_efi = pud_offset(p4d_efi, 0);
+ pud_k = pud_offset(p4d_k, 0);
num_entries = pud_index(EFI_VA_END);
memcpy(pud_efi, pud_k, sizeof(pud_t) * num_entries);
- pud_efi = pud_offset(pgd_efi, EFI_VA_START);
- pud_k = pud_offset(pgd_k, EFI_VA_START);
+ pud_efi = pud_offset(p4d_efi, EFI_VA_START);
+ pud_k = pud_offset(p4d_k, EFI_VA_START);
num_entries = PTRS_PER_PUD - pud_index(EFI_VA_START);
memcpy(pud_efi, pud_k, sizeof(pud_t) * num_entries);
diff --git a/arch/x86/platform/efi/quirks.c b/arch/x86/platform/efi/quirks.c
index cdfe8c6..2661599 100644
--- a/arch/x86/platform/efi/quirks.c
+++ b/arch/x86/platform/efi/quirks.c
@@ -11,6 +11,8 @@
#include <linux/bootmem.h>
#include <linux/acpi.h>
#include <linux/dmi.h>
+
+#include <asm/e820/api.h>
#include <asm/efi.h>
#include <asm/uv/uv.h>
@@ -244,14 +246,14 @@
* else. We must only reserve (and then free) regions:
*
* - Not within any part of the kernel
- * - Not the BIOS reserved area (E820_RESERVED, E820_NVS, etc)
+ * - Not the BIOS reserved area (E820_TYPE_RESERVED, E820_TYPE_NVS, etc)
*/
static bool can_free_region(u64 start, u64 size)
{
if (start + size > __pa_symbol(_text) && start <= __pa_symbol(_end))
return false;
- if (!e820_all_mapped(start, start+size, E820_RAM))
+ if (!e820__mapped_all(start, start+size, E820_TYPE_RAM))
return false;
return true;
@@ -284,7 +286,7 @@
* A good example of a critical region that must not be
* freed is page zero (first 4Kb of memory), which may
* contain boot services code/data but is marked
- * E820_RESERVED by trim_bios_range().
+ * E820_TYPE_RESERVED by trim_bios_range().
*/
if (!already_reserved) {
memblock_reserve(start, size);
diff --git a/arch/x86/platform/intel-mid/device_libs/Makefile b/arch/x86/platform/intel-mid/device_libs/Makefile
index 3dbde04..53e0235 100644
--- a/arch/x86/platform/intel-mid/device_libs/Makefile
+++ b/arch/x86/platform/intel-mid/device_libs/Makefile
@@ -2,8 +2,9 @@
obj-$(subst m,y,$(CONFIG_PINCTRL_MERRIFIELD)) += platform_mrfld_pinctrl.o
# SDHCI Devices
obj-$(subst m,y,$(CONFIG_MMC_SDHCI_PCI)) += platform_mrfld_sd.o
-# WiFi
+# WiFi + BT
obj-$(subst m,y,$(CONFIG_BRCMFMAC_SDIO)) += platform_bcm43xx.o
+obj-$(subst m,y,$(CONFIG_BT_HCIUART_BCM)) += platform_bt.o
# IPC Devices
obj-$(subst m,y,$(CONFIG_MFD_INTEL_MSIC)) += platform_msic.o
obj-$(subst m,y,$(CONFIG_SND_MFLD_MACHINE)) += platform_msic_audio.o
diff --git a/arch/x86/platform/intel-mid/device_libs/platform_bt.c b/arch/x86/platform/intel-mid/device_libs/platform_bt.c
new file mode 100644
index 0000000..5a0483e
--- /dev/null
+++ b/arch/x86/platform/intel-mid/device_libs/platform_bt.c
@@ -0,0 +1,108 @@
+/*
+ * Bluetooth platform data initialization file
+ *
+ * (C) Copyright 2017 Intel Corporation
+ * Author: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; version 2
+ * of the License.
+ */
+
+#include <linux/gpio/machine.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+
+#include <asm/cpu_device_id.h>
+#include <asm/intel-family.h>
+#include <asm/intel-mid.h>
+
+struct bt_sfi_data {
+ struct device *dev;
+ const char *name;
+ int (*setup)(struct bt_sfi_data *ddata);
+};
+
+static struct gpiod_lookup_table tng_bt_sfi_gpio_table = {
+ .dev_id = "hci_bcm",
+ .table = {
+ GPIO_LOOKUP("0000:00:0c.0", -1, "device-wakeup", GPIO_ACTIVE_HIGH),
+ GPIO_LOOKUP("0000:00:0c.0", -1, "shutdown", GPIO_ACTIVE_HIGH),
+ GPIO_LOOKUP("0000:00:0c.0", -1, "host-wakeup", GPIO_ACTIVE_HIGH),
+ { },
+ },
+};
+
+#define TNG_BT_SFI_GPIO_DEVICE_WAKEUP "bt_wakeup"
+#define TNG_BT_SFI_GPIO_SHUTDOWN "BT-reset"
+#define TNG_BT_SFI_GPIO_HOST_WAKEUP "bt_uart_enable"
+
+static int __init tng_bt_sfi_setup(struct bt_sfi_data *ddata)
+{
+ struct gpiod_lookup_table *table = &tng_bt_sfi_gpio_table;
+ struct gpiod_lookup *lookup = table->table;
+ struct pci_dev *pdev;
+
+ /* Connected to /dev/ttyS0 */
+ pdev = pci_get_domain_bus_and_slot(0, 0, PCI_DEVFN(4, 1));
+ if (!pdev)
+ return -ENODEV;
+
+ ddata->dev = &pdev->dev;
+ ddata->name = table->dev_id;
+
+ lookup[0].chip_hwnum = get_gpio_by_name(TNG_BT_SFI_GPIO_DEVICE_WAKEUP);
+ lookup[1].chip_hwnum = get_gpio_by_name(TNG_BT_SFI_GPIO_SHUTDOWN);
+ lookup[2].chip_hwnum = get_gpio_by_name(TNG_BT_SFI_GPIO_HOST_WAKEUP);
+
+ gpiod_add_lookup_table(table);
+ return 0;
+}
+
+static struct bt_sfi_data tng_bt_sfi_data __initdata = {
+ .setup = tng_bt_sfi_setup,
+};
+
+#define ICPU(model, ddata) \
+ { X86_VENDOR_INTEL, 6, model, X86_FEATURE_ANY, (kernel_ulong_t)&ddata }
+
+static const struct x86_cpu_id bt_sfi_cpu_ids[] = {
+ ICPU(INTEL_FAM6_ATOM_MERRIFIELD, tng_bt_sfi_data),
+ {}
+};
+
+static int __init bt_sfi_init(void)
+{
+ struct platform_device_info info;
+ struct platform_device *pdev;
+ const struct x86_cpu_id *id;
+ struct bt_sfi_data *ddata;
+ int ret;
+
+ id = x86_match_cpu(bt_sfi_cpu_ids);
+ if (!id)
+ return -ENODEV;
+
+ ddata = (struct bt_sfi_data *)id->driver_data;
+ if (!ddata)
+ return -ENODEV;
+
+ ret = ddata->setup(ddata);
+ if (ret)
+ return ret;
+
+ memset(&info, 0, sizeof(info));
+ info.fwnode = ddata->dev->fwnode;
+ info.parent = ddata->dev;
+ info.name = ddata->name,
+ info.id = PLATFORM_DEVID_NONE,
+
+ pdev = platform_device_register_full(&info);
+ if (IS_ERR(pdev))
+ return PTR_ERR(pdev);
+
+ dev_info(ddata->dev, "Registered Bluetooth device: %s\n", ddata->name);
+ return 0;
+}
+device_initcall(bt_sfi_init);
diff --git a/arch/x86/platform/uv/tlb_uv.c b/arch/x86/platform/uv/tlb_uv.c
index f25982c..42e65fe 100644
--- a/arch/x86/platform/uv/tlb_uv.c
+++ b/arch/x86/platform/uv/tlb_uv.c
@@ -23,28 +23,7 @@
#include <asm/irq_vectors.h>
#include <asm/timer.h>
-static struct bau_operations ops;
-
-static struct bau_operations uv123_bau_ops = {
- .bau_gpa_to_offset = uv_gpa_to_offset,
- .read_l_sw_ack = read_mmr_sw_ack,
- .read_g_sw_ack = read_gmmr_sw_ack,
- .write_l_sw_ack = write_mmr_sw_ack,
- .write_g_sw_ack = write_gmmr_sw_ack,
- .write_payload_first = write_mmr_payload_first,
- .write_payload_last = write_mmr_payload_last,
-};
-
-static struct bau_operations uv4_bau_ops = {
- .bau_gpa_to_offset = uv_gpa_to_soc_phys_ram,
- .read_l_sw_ack = read_mmr_proc_sw_ack,
- .read_g_sw_ack = read_gmmr_proc_sw_ack,
- .write_l_sw_ack = write_mmr_proc_sw_ack,
- .write_g_sw_ack = write_gmmr_proc_sw_ack,
- .write_payload_first = write_mmr_proc_payload_first,
- .write_payload_last = write_mmr_proc_payload_last,
-};
-
+static struct bau_operations ops __ro_after_init;
/* timeouts in nanoseconds (indexed by UVH_AGING_PRESCALE_SEL urgency7 30:28) */
static int timeout_base_ns[] = {
@@ -548,11 +527,12 @@
* return COMPLETE, RETRY(PLUGGED or TIMEOUT) or GIVEUP
*/
static int uv1_wait_completion(struct bau_desc *bau_desc,
- unsigned long mmr_offset, int right_shift,
struct bau_control *bcp, long try)
{
unsigned long descriptor_status;
cycles_t ttm;
+ u64 mmr_offset = bcp->status_mmr;
+ int right_shift = bcp->status_index;
struct ptc_stats *stat = bcp->statp;
descriptor_status = uv1_read_status(mmr_offset, right_shift);
@@ -640,11 +620,12 @@
}
static int uv2_3_wait_completion(struct bau_desc *bau_desc,
- unsigned long mmr_offset, int right_shift,
struct bau_control *bcp, long try)
{
unsigned long descriptor_stat;
cycles_t ttm;
+ u64 mmr_offset = bcp->status_mmr;
+ int right_shift = bcp->status_index;
int desc = bcp->uvhub_cpu;
long busy_reps = 0;
struct ptc_stats *stat = bcp->statp;
@@ -706,28 +687,59 @@
}
/*
- * There are 2 status registers; each and array[32] of 2 bits. Set up for
- * which register to read and position in that register based on cpu in
- * current hub.
+ * Returns the status of current BAU message for cpu desc as a bit field
+ * [Error][Busy][Aux]
*/
-static int wait_completion(struct bau_desc *bau_desc, struct bau_control *bcp, long try)
+static u64 read_status(u64 status_mmr, int index, int desc)
{
- int right_shift;
- unsigned long mmr_offset;
+ u64 stat;
+
+ stat = ((read_lmmr(status_mmr) >> index) & UV_ACT_STATUS_MASK) << 1;
+ stat |= (read_lmmr(UVH_LB_BAU_SB_ACTIVATION_STATUS_2) >> desc) & 0x1;
+
+ return stat;
+}
+
+static int uv4_wait_completion(struct bau_desc *bau_desc,
+ struct bau_control *bcp, long try)
+{
+ struct ptc_stats *stat = bcp->statp;
+ u64 descriptor_stat;
+ u64 mmr = bcp->status_mmr;
+ int index = bcp->status_index;
int desc = bcp->uvhub_cpu;
- if (desc < UV_CPUS_PER_AS) {
- mmr_offset = UVH_LB_BAU_SB_ACTIVATION_STATUS_0;
- right_shift = desc * UV_ACT_STATUS_SIZE;
- } else {
- mmr_offset = UVH_LB_BAU_SB_ACTIVATION_STATUS_1;
- right_shift = ((desc - UV_CPUS_PER_AS) * UV_ACT_STATUS_SIZE);
- }
+ descriptor_stat = read_status(mmr, index, desc);
- if (bcp->uvhub_version == 1)
- return uv1_wait_completion(bau_desc, mmr_offset, right_shift, bcp, try);
- else
- return uv2_3_wait_completion(bau_desc, mmr_offset, right_shift, bcp, try);
+ /* spin on the status MMR, waiting for it to go idle */
+ while (descriptor_stat != UV2H_DESC_IDLE) {
+ switch (descriptor_stat) {
+ case UV2H_DESC_SOURCE_TIMEOUT:
+ stat->s_stimeout++;
+ return FLUSH_GIVEUP;
+
+ case UV2H_DESC_DEST_TIMEOUT:
+ stat->s_dtimeout++;
+ bcp->conseccompletes = 0;
+ return FLUSH_RETRY_TIMEOUT;
+
+ case UV2H_DESC_DEST_STRONG_NACK:
+ stat->s_plugged++;
+ bcp->conseccompletes = 0;
+ return FLUSH_RETRY_PLUGGED;
+
+ case UV2H_DESC_DEST_PUT_ERR:
+ bcp->conseccompletes = 0;
+ return FLUSH_GIVEUP;
+
+ default:
+ /* descriptor_stat is still BUSY */
+ cpu_relax();
+ }
+ descriptor_stat = read_status(mmr, index, desc);
+ }
+ bcp->conseccompletes++;
+ return FLUSH_COMPLETE;
}
/*
@@ -918,7 +930,7 @@
struct uv1_bau_msg_header *uv1_hdr = NULL;
struct uv2_3_bau_msg_header *uv2_3_hdr = NULL;
- if (bcp->uvhub_version == 1) {
+ if (bcp->uvhub_version == UV_BAU_V1) {
uv1 = 1;
uv1_throttle(hmaster, stat);
}
@@ -958,7 +970,7 @@
write_mmr_activation(index);
try++;
- completion_stat = wait_completion(bau_desc, bcp, try);
+ completion_stat = ops.wait_completion(bau_desc, bcp, try);
handle_cmplt(completion_stat, bau_desc, bcp, hmaster, stat);
@@ -1114,15 +1126,12 @@
unsigned long end,
unsigned int cpu)
{
- int locals = 0;
- int remotes = 0;
- int hubs = 0;
+ int locals = 0, remotes = 0, hubs = 0;
struct bau_desc *bau_desc;
struct cpumask *flush_mask;
struct ptc_stats *stat;
struct bau_control *bcp;
- unsigned long descriptor_status;
- unsigned long status;
+ unsigned long descriptor_status, status, address;
bcp = &per_cpu(bau_control, cpu);
@@ -1171,10 +1180,24 @@
record_send_statistics(stat, locals, hubs, remotes, bau_desc);
if (!end || (end - start) <= PAGE_SIZE)
- bau_desc->payload.address = start;
+ address = start;
else
- bau_desc->payload.address = TLB_FLUSH_ALL;
- bau_desc->payload.sending_cpu = cpu;
+ address = TLB_FLUSH_ALL;
+
+ switch (bcp->uvhub_version) {
+ case UV_BAU_V1:
+ case UV_BAU_V2:
+ case UV_BAU_V3:
+ bau_desc->payload.uv1_2_3.address = address;
+ bau_desc->payload.uv1_2_3.sending_cpu = cpu;
+ break;
+ case UV_BAU_V4:
+ bau_desc->payload.uv4.address = address;
+ bau_desc->payload.uv4.sending_cpu = cpu;
+ bau_desc->payload.uv4.qualifier = BAU_DESC_QUALIFIER;
+ break;
+ }
+
/*
* uv_flush_send_and_wait returns 0 if all cpu's were messaged,
* or 1 if it gave up and the original cpumask should be returned.
@@ -1296,7 +1319,7 @@
msgdesc.msg_slot = msg - msgdesc.queue_first;
msgdesc.msg = msg;
- if (bcp->uvhub_version == 2)
+ if (bcp->uvhub_version == UV_BAU_V2)
process_uv2_message(&msgdesc, bcp);
else
/* no error workaround for uv1 or uv3 */
@@ -1838,7 +1861,7 @@
* and the payload queue tail must be maintained by the kernel.
*/
bcp = &per_cpu(bau_control, smp_processor_id());
- if (bcp->uvhub_version <= 3) {
+ if (bcp->uvhub_version <= UV_BAU_V3) {
tail = first;
gnode = uv_gpa_to_gnode(uv_gpa(pqp));
first = (gnode << UV_PAYLOADQ_GNODE_SHIFT) | tail;
@@ -2034,8 +2057,7 @@
struct bau_control **smasterp,
struct bau_control **hmasterp)
{
- int i;
- int cpu;
+ int i, cpu, uvhub_cpu;
struct bau_control *bcp;
for (i = 0; i < sdp->num_cpus; i++) {
@@ -2052,19 +2074,33 @@
bcp->socket_master = *smasterp;
bcp->uvhub = bdp->uvhub;
if (is_uv1_hub())
- bcp->uvhub_version = 1;
+ bcp->uvhub_version = UV_BAU_V1;
else if (is_uv2_hub())
- bcp->uvhub_version = 2;
+ bcp->uvhub_version = UV_BAU_V2;
else if (is_uv3_hub())
- bcp->uvhub_version = 3;
+ bcp->uvhub_version = UV_BAU_V3;
else if (is_uv4_hub())
- bcp->uvhub_version = 4;
+ bcp->uvhub_version = UV_BAU_V4;
else {
pr_emerg("uvhub version not 1, 2, 3, or 4\n");
return 1;
}
bcp->uvhub_master = *hmasterp;
- bcp->uvhub_cpu = uv_cpu_blade_processor_id(cpu);
+ uvhub_cpu = uv_cpu_blade_processor_id(cpu);
+ bcp->uvhub_cpu = uvhub_cpu;
+
+ /*
+ * The ERROR and BUSY status registers are located pairwise over
+ * the STATUS_0 and STATUS_1 mmrs; each an array[32] of 2 bits.
+ */
+ if (uvhub_cpu < UV_CPUS_PER_AS) {
+ bcp->status_mmr = UVH_LB_BAU_SB_ACTIVATION_STATUS_0;
+ bcp->status_index = uvhub_cpu * UV_ACT_STATUS_SIZE;
+ } else {
+ bcp->status_mmr = UVH_LB_BAU_SB_ACTIVATION_STATUS_1;
+ bcp->status_index = (uvhub_cpu - UV_CPUS_PER_AS)
+ * UV_ACT_STATUS_SIZE;
+ }
if (bcp->uvhub_cpu >= MAX_CPUS_PER_UVHUB) {
pr_emerg("%d cpus per uvhub invalid\n",
@@ -2147,6 +2183,39 @@
return 1;
}
+static const struct bau_operations uv1_bau_ops __initconst = {
+ .bau_gpa_to_offset = uv_gpa_to_offset,
+ .read_l_sw_ack = read_mmr_sw_ack,
+ .read_g_sw_ack = read_gmmr_sw_ack,
+ .write_l_sw_ack = write_mmr_sw_ack,
+ .write_g_sw_ack = write_gmmr_sw_ack,
+ .write_payload_first = write_mmr_payload_first,
+ .write_payload_last = write_mmr_payload_last,
+ .wait_completion = uv1_wait_completion,
+};
+
+static const struct bau_operations uv2_3_bau_ops __initconst = {
+ .bau_gpa_to_offset = uv_gpa_to_offset,
+ .read_l_sw_ack = read_mmr_sw_ack,
+ .read_g_sw_ack = read_gmmr_sw_ack,
+ .write_l_sw_ack = write_mmr_sw_ack,
+ .write_g_sw_ack = write_gmmr_sw_ack,
+ .write_payload_first = write_mmr_payload_first,
+ .write_payload_last = write_mmr_payload_last,
+ .wait_completion = uv2_3_wait_completion,
+};
+
+static const struct bau_operations uv4_bau_ops __initconst = {
+ .bau_gpa_to_offset = uv_gpa_to_soc_phys_ram,
+ .read_l_sw_ack = read_mmr_proc_sw_ack,
+ .read_g_sw_ack = read_gmmr_proc_sw_ack,
+ .write_l_sw_ack = write_mmr_proc_sw_ack,
+ .write_g_sw_ack = write_gmmr_proc_sw_ack,
+ .write_payload_first = write_mmr_proc_payload_first,
+ .write_payload_last = write_mmr_proc_payload_last,
+ .wait_completion = uv4_wait_completion,
+};
+
/*
* Initialization of BAU-related structures
*/
@@ -2166,11 +2235,11 @@
if (is_uv4_hub())
ops = uv4_bau_ops;
else if (is_uv3_hub())
- ops = uv123_bau_ops;
+ ops = uv2_3_bau_ops;
else if (is_uv2_hub())
- ops = uv123_bau_ops;
+ ops = uv2_3_bau_ops;
else if (is_uv1_hub())
- ops = uv123_bau_ops;
+ ops = uv1_bau_ops;
for_each_possible_cpu(cur_cpu) {
mask = &per_cpu(uv_flush_tlb_mask, cur_cpu);
diff --git a/arch/x86/platform/uv/uv_time.c b/arch/x86/platform/uv/uv_time.c
index 2ee7632..b082d71 100644
--- a/arch/x86/platform/uv/uv_time.c
+++ b/arch/x86/platform/uv/uv_time.c
@@ -390,9 +390,11 @@
clock_event_device_uv.min_delta_ns = NSEC_PER_SEC /
sn_rtc_cycles_per_second;
+ clock_event_device_uv.min_delta_ticks = 1;
clock_event_device_uv.max_delta_ns = clocksource_uv.mask *
(NSEC_PER_SEC / sn_rtc_cycles_per_second);
+ clock_event_device_uv.max_delta_ticks = clocksource_uv.mask;
rc = schedule_on_each_cpu(uv_rtc_register_clockevents);
if (rc) {
diff --git a/arch/x86/power/cpu.c b/arch/x86/power/cpu.c
index 66ade16..6b05a92 100644
--- a/arch/x86/power/cpu.c
+++ b/arch/x86/power/cpu.c
@@ -95,7 +95,7 @@
* 'pmode_gdt' in wakeup_start.
*/
ctxt->gdt_desc.size = GDT_SIZE - 1;
- ctxt->gdt_desc.address = (unsigned long)get_cpu_gdt_table(smp_processor_id());
+ ctxt->gdt_desc.address = (unsigned long)get_cpu_gdt_rw(smp_processor_id());
store_tr(ctxt->tr);
@@ -162,7 +162,7 @@
int cpu = smp_processor_id();
struct tss_struct *t = &per_cpu(cpu_tss, cpu);
#ifdef CONFIG_X86_64
- struct desc_struct *desc = get_cpu_gdt_table(cpu);
+ struct desc_struct *desc = get_cpu_gdt_rw(cpu);
tss_desc tss;
#endif
set_tss_desc(cpu, t); /*
@@ -183,6 +183,9 @@
load_mm_ldt(current->active_mm); /* This does lldt */
fpu__resume_cpu();
+
+ /* The processor is back on the direct GDT, load back the fixmap */
+ load_fixmap_gdt(cpu);
}
/**
diff --git a/arch/x86/power/hibernate_32.c b/arch/x86/power/hibernate_32.c
index 9f14bd3..c35fdb5 100644
--- a/arch/x86/power/hibernate_32.c
+++ b/arch/x86/power/hibernate_32.c
@@ -32,6 +32,7 @@
*/
static pmd_t *resume_one_md_table_init(pgd_t *pgd)
{
+ p4d_t *p4d;
pud_t *pud;
pmd_t *pmd_table;
@@ -41,11 +42,13 @@
return NULL;
set_pgd(pgd, __pgd(__pa(pmd_table) | _PAGE_PRESENT));
- pud = pud_offset(pgd, 0);
+ p4d = p4d_offset(pgd, 0);
+ pud = pud_offset(p4d, 0);
BUG_ON(pmd_table != pmd_offset(pud, 0));
#else
- pud = pud_offset(pgd, 0);
+ p4d = p4d_offset(pgd, 0);
+ pud = pud_offset(p4d, 0);
pmd_table = pmd_offset(pud, 0);
#endif
diff --git a/arch/x86/power/hibernate_64.c b/arch/x86/power/hibernate_64.c
index ded2e82..6a61194 100644
--- a/arch/x86/power/hibernate_64.c
+++ b/arch/x86/power/hibernate_64.c
@@ -16,6 +16,7 @@
#include <crypto/hash.h>
+#include <asm/e820/api.h>
#include <asm/init.h>
#include <asm/proto.h>
#include <asm/page.h>
@@ -49,6 +50,7 @@
{
pmd_t *pmd;
pud_t *pud;
+ p4d_t *p4d;
/*
* The new mapping only has to cover the page containing the image
@@ -63,6 +65,13 @@
* the virtual address space after switching over to the original page
* tables used by the image kernel.
*/
+
+ if (IS_ENABLED(CONFIG_X86_5LEVEL)) {
+ p4d = (p4d_t *)get_safe_page(GFP_ATOMIC);
+ if (!p4d)
+ return -ENOMEM;
+ }
+
pud = (pud_t *)get_safe_page(GFP_ATOMIC);
if (!pud)
return -ENOMEM;
@@ -75,8 +84,13 @@
__pmd((jump_address_phys & PMD_MASK) | __PAGE_KERNEL_LARGE_EXEC));
set_pud(pud + pud_index(restore_jump_address),
__pud(__pa(pmd) | _KERNPG_TABLE));
- set_pgd(pgd + pgd_index(restore_jump_address),
- __pgd(__pa(pud) | _KERNPG_TABLE));
+ if (IS_ENABLED(CONFIG_X86_5LEVEL)) {
+ set_p4d(p4d + p4d_index(restore_jump_address), __p4d(__pa(pud) | _KERNPG_TABLE));
+ set_pgd(pgd + pgd_index(restore_jump_address), __pgd(__pa(p4d) | _KERNPG_TABLE));
+ } else {
+ /* No p4d for 4-level paging: point the pgd to the pud page table */
+ set_pgd(pgd + pgd_index(restore_jump_address), __pgd(__pa(pud) | _KERNPG_TABLE));
+ }
return 0;
}
@@ -124,7 +138,10 @@
static int relocate_restore_code(void)
{
pgd_t *pgd;
+ p4d_t *p4d;
pud_t *pud;
+ pmd_t *pmd;
+ pte_t *pte;
relocated_restore_code = get_safe_page(GFP_ATOMIC);
if (!relocated_restore_code)
@@ -134,22 +151,25 @@
/* Make the page containing the relocated code executable */
pgd = (pgd_t *)__va(read_cr3()) + pgd_index(relocated_restore_code);
- pud = pud_offset(pgd, relocated_restore_code);
+ p4d = p4d_offset(pgd, relocated_restore_code);
+ if (p4d_large(*p4d)) {
+ set_p4d(p4d, __p4d(p4d_val(*p4d) & ~_PAGE_NX));
+ goto out;
+ }
+ pud = pud_offset(p4d, relocated_restore_code);
if (pud_large(*pud)) {
set_pud(pud, __pud(pud_val(*pud) & ~_PAGE_NX));
- } else {
- pmd_t *pmd = pmd_offset(pud, relocated_restore_code);
-
- if (pmd_large(*pmd)) {
- set_pmd(pmd, __pmd(pmd_val(*pmd) & ~_PAGE_NX));
- } else {
- pte_t *pte = pte_offset_kernel(pmd, relocated_restore_code);
-
- set_pte(pte, __pte(pte_val(*pte) & ~_PAGE_NX));
- }
+ goto out;
}
+ pmd = pmd_offset(pud, relocated_restore_code);
+ if (pmd_large(*pmd)) {
+ set_pmd(pmd, __pmd(pmd_val(*pmd) & ~_PAGE_NX));
+ goto out;
+ }
+ pte = pte_offset_kernel(pmd, relocated_restore_code);
+ set_pte(pte, __pte(pte_val(*pte) & ~_PAGE_NX));
+out:
__flush_tlb_all();
-
return 0;
}
@@ -195,12 +215,12 @@
#if IS_BUILTIN(CONFIG_CRYPTO_MD5)
/**
- * get_e820_md5 - calculate md5 according to given e820 map
+ * get_e820_md5 - calculate md5 according to given e820 table
*
- * @map: the e820 map to be calculated
+ * @table: the e820 table to be calculated
* @buf: the md5 result to be stored to
*/
-static int get_e820_md5(struct e820map *map, void *buf)
+static int get_e820_md5(struct e820_table *table, void *buf)
{
struct scatterlist sg;
struct crypto_ahash *tfm;
@@ -213,10 +233,9 @@
{
AHASH_REQUEST_ON_STACK(req, tfm);
- size = offsetof(struct e820map, map)
- + sizeof(struct e820entry) * map->nr_map;
+ size = offsetof(struct e820_table, entries) + sizeof(struct e820_entry) * table->nr_entries;
ahash_request_set_tfm(req, tfm);
- sg_init_one(&sg, (u8 *)map, size);
+ sg_init_one(&sg, (u8 *)table, size);
ahash_request_set_callback(req, 0, NULL, NULL);
ahash_request_set_crypt(req, &sg, buf, size);
@@ -231,7 +250,7 @@
static void hibernation_e820_save(void *buf)
{
- get_e820_md5(e820_saved, buf);
+ get_e820_md5(e820_table_firmware, buf);
}
static bool hibernation_e820_mismatch(void *buf)
@@ -244,7 +263,7 @@
if (!memcmp(result, buf, MD5_DIGEST_SIZE))
return false;
- ret = get_e820_md5(e820_saved, result);
+ ret = get_e820_md5(e820_table_firmware, result);
if (ret)
return true;
diff --git a/arch/x86/ras/Kconfig b/arch/x86/ras/Kconfig
index 0bc60a3..2a2d89d 100644
--- a/arch/x86/ras/Kconfig
+++ b/arch/x86/ras/Kconfig
@@ -7,3 +7,17 @@
aspects of the MCE handling code.
WARNING: Do not even assume this interface is staying stable!
+
+config RAS_CEC
+ bool "Correctable Errors Collector"
+ depends on X86_MCE && MEMORY_FAILURE && DEBUG_FS
+ ---help---
+ This is a small cache which collects correctable memory errors per 4K
+ page PFN and counts their repeated occurrence. Once the counter for a
+ PFN overflows, we try to soft-offline that page as we take it to mean
+ that it has reached a relatively high error count and would probably
+ be best if we don't use it anymore.
+
+ Bear in mind that this is absolutely useless if your platform doesn't
+ have ECC DIMMs and doesn't have DRAM ECC checking enabled in the BIOS.
+
diff --git a/arch/x86/um/Makefile b/arch/x86/um/Makefile
index e7e7055..46cbbfe 100644
--- a/arch/x86/um/Makefile
+++ b/arch/x86/um/Makefile
@@ -8,7 +8,7 @@
BITS := 64
endif
-obj-y = bug.o bugs_$(BITS).o delay.o fault.o ldt.o \
+obj-y = bugs_$(BITS).o delay.o fault.o ldt.o \
ptrace_$(BITS).o ptrace_user.o setjmp_$(BITS).o signal.o \
stub_$(BITS).o stub_segv.o \
sys_call_table_$(BITS).o sysrq_$(BITS).o tls_$(BITS).o \
@@ -16,7 +16,7 @@
ifeq ($(CONFIG_X86_32),y)
-obj-y += checksum_32.o
+obj-y += checksum_32.o syscalls_32.o
obj-$(CONFIG_ELF_CORE) += elfcore.o
subarch-y = ../lib/string_32.o ../lib/atomic64_32.o ../lib/atomic64_cx8_32.o
diff --git a/arch/x86/um/asm/ptrace.h b/arch/x86/um/asm/ptrace.h
index e59eef2..b291ca5 100644
--- a/arch/x86/um/asm/ptrace.h
+++ b/arch/x86/um/asm/ptrace.h
@@ -78,7 +78,7 @@
return -ENOSYS;
}
-extern long arch_prctl(struct task_struct *task, int code,
+extern long arch_prctl(struct task_struct *task, int option,
unsigned long __user *addr);
#endif
diff --git a/arch/x86/um/bug.c b/arch/x86/um/bug.c
deleted file mode 100644
index e8034e3..0000000
--- a/arch/x86/um/bug.c
+++ /dev/null
@@ -1,21 +0,0 @@
-/*
- * Copyright (C) 2006 Jeff Dike (jdike@addtoit.com)
- * Licensed under the GPL V2
- */
-
-#include <linux/uaccess.h>
-
-/*
- * Mostly copied from i386/x86_86 - eliminated the eip < PAGE_OFFSET because
- * that's not relevant in skas mode.
- */
-
-int is_valid_bugaddr(unsigned long eip)
-{
- unsigned short ud2;
-
- if (probe_kernel_address((unsigned short __user *)eip, ud2))
- return 0;
-
- return ud2 == 0x0b0f;
-}
diff --git a/arch/x86/um/os-Linux/prctl.c b/arch/x86/um/os-Linux/prctl.c
index 96eb2bd..8431e87 100644
--- a/arch/x86/um/os-Linux/prctl.c
+++ b/arch/x86/um/os-Linux/prctl.c
@@ -6,7 +6,7 @@
#include <sys/ptrace.h>
#include <asm/ptrace.h>
-int os_arch_prctl(int pid, int code, unsigned long *addr)
+int os_arch_prctl(int pid, int option, unsigned long *arg2)
{
- return ptrace(PTRACE_ARCH_PRCTL, pid, (unsigned long) addr, code);
+ return ptrace(PTRACE_ARCH_PRCTL, pid, (unsigned long) arg2, option);
}
diff --git a/arch/x86/um/syscalls_32.c b/arch/x86/um/syscalls_32.c
new file mode 100644
index 0000000..627d688
--- /dev/null
+++ b/arch/x86/um/syscalls_32.c
@@ -0,0 +1,7 @@
+#include <linux/syscalls.h>
+#include <os.h>
+
+SYSCALL_DEFINE2(arch_prctl, int, option, unsigned long, arg2)
+{
+ return -EINVAL;
+}
diff --git a/arch/x86/um/syscalls_64.c b/arch/x86/um/syscalls_64.c
index 10d9070..58f5166 100644
--- a/arch/x86/um/syscalls_64.c
+++ b/arch/x86/um/syscalls_64.c
@@ -7,13 +7,15 @@
#include <linux/sched.h>
#include <linux/sched/mm.h>
+#include <linux/syscalls.h>
#include <linux/uaccess.h>
#include <asm/prctl.h> /* XXX This should get the constants from libc */
#include <os.h>
-long arch_prctl(struct task_struct *task, int code, unsigned long __user *addr)
+long arch_prctl(struct task_struct *task, int option,
+ unsigned long __user *arg2)
{
- unsigned long *ptr = addr, tmp;
+ unsigned long *ptr = arg2, tmp;
long ret;
int pid = task->mm->context.id.u.pid;
@@ -30,7 +32,7 @@
* arch_prctl is run on the host, then the registers are read
* back.
*/
- switch (code) {
+ switch (option) {
case ARCH_SET_FS:
case ARCH_SET_GS:
ret = restore_registers(pid, ¤t->thread.regs.regs);
@@ -50,11 +52,11 @@
ptr = &tmp;
}
- ret = os_arch_prctl(pid, code, ptr);
+ ret = os_arch_prctl(pid, option, ptr);
if (ret)
return ret;
- switch (code) {
+ switch (option) {
case ARCH_SET_FS:
current->thread.arch.fs = (unsigned long) ptr;
ret = save_registers(pid, ¤t->thread.regs.regs);
@@ -63,19 +65,19 @@
ret = save_registers(pid, ¤t->thread.regs.regs);
break;
case ARCH_GET_FS:
- ret = put_user(tmp, addr);
+ ret = put_user(tmp, arg2);
break;
case ARCH_GET_GS:
- ret = put_user(tmp, addr);
+ ret = put_user(tmp, arg2);
break;
}
return ret;
}
-long sys_arch_prctl(int code, unsigned long addr)
+SYSCALL_DEFINE2(arch_prctl, int, option, unsigned long, arg2)
{
- return arch_prctl(current, code, (unsigned long __user *) addr);
+ return arch_prctl(current, option, (unsigned long __user *) arg2);
}
void arch_switch_to(struct task_struct *to)
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index ec1d5c4..30822e8 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -76,6 +76,7 @@
#include <asm/mwait.h>
#include <asm/pci_x86.h>
#include <asm/cpu.h>
+#include <asm/e820/api.h>
#ifdef CONFIG_ACPI
#include <linux/acpi.h>
@@ -710,7 +711,7 @@
*shadow = t->tls_array[i];
- gdt = get_cpu_gdt_table(cpu);
+ gdt = get_cpu_gdt_rw(cpu);
maddr = arbitrary_virt_to_machine(&gdt[GDT_ENTRY_TLS_MIN+i]);
mc = __xen_mc_entry(0);
@@ -1595,7 +1596,6 @@
/* set up basic CPUID stuff */
cpu_detect(&new_cpu_data);
set_cpu_cap(&new_cpu_data, X86_FEATURE_FPU);
- new_cpu_data.wp_works_ok = 1;
new_cpu_data.x86_capability[CPUID_1_EDX] = cpuid_edx(1);
#endif
@@ -1690,34 +1690,32 @@
memset(&pvh_bootparams, 0, sizeof(pvh_bootparams));
- memmap.nr_entries = ARRAY_SIZE(pvh_bootparams.e820_map);
- set_xen_guest_handle(memmap.buffer, pvh_bootparams.e820_map);
+ memmap.nr_entries = ARRAY_SIZE(pvh_bootparams.e820_table);
+ set_xen_guest_handle(memmap.buffer, pvh_bootparams.e820_table);
rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap);
if (rc) {
xen_raw_printk("XENMEM_memory_map failed (%d)\n", rc);
BUG();
}
- if (memmap.nr_entries < E820MAX - 1) {
- pvh_bootparams.e820_map[memmap.nr_entries].addr =
+ if (memmap.nr_entries < E820_MAX_ENTRIES_ZEROPAGE - 1) {
+ pvh_bootparams.e820_table[memmap.nr_entries].addr =
ISA_START_ADDRESS;
- pvh_bootparams.e820_map[memmap.nr_entries].size =
+ pvh_bootparams.e820_table[memmap.nr_entries].size =
ISA_END_ADDRESS - ISA_START_ADDRESS;
- pvh_bootparams.e820_map[memmap.nr_entries].type =
- E820_RESERVED;
+ pvh_bootparams.e820_table[memmap.nr_entries].type =
+ E820_TYPE_RESERVED;
memmap.nr_entries++;
} else
xen_raw_printk("Warning: Can fit ISA range into e820\n");
- sanitize_e820_map(pvh_bootparams.e820_map,
- ARRAY_SIZE(pvh_bootparams.e820_map),
- &memmap.nr_entries);
-
pvh_bootparams.e820_entries = memmap.nr_entries;
for (i = 0; i < pvh_bootparams.e820_entries; i++)
- e820_add_region(pvh_bootparams.e820_map[i].addr,
- pvh_bootparams.e820_map[i].size,
- pvh_bootparams.e820_map[i].type);
+ e820__range_add(pvh_bootparams.e820_table[i].addr,
+ pvh_bootparams.e820_table[i].size,
+ pvh_bootparams.e820_table[i].type);
+
+ e820__update_table(e820_table);
pvh_bootparams.hdr.cmd_line_ptr =
pvh_start_info.cmdline_paddr;
diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index 37cb5aa..f226038 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -58,7 +58,7 @@
#include <asm/mmu_context.h>
#include <asm/setup.h>
#include <asm/paravirt.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/linkage.h>
#include <asm/page.h>
#include <asm/init.h>
@@ -535,40 +535,41 @@
return user_ptr;
}
-static void __xen_set_pgd_hyper(pgd_t *ptr, pgd_t val)
+static void __xen_set_p4d_hyper(p4d_t *ptr, p4d_t val)
{
struct mmu_update u;
u.ptr = virt_to_machine(ptr).maddr;
- u.val = pgd_val_ma(val);
+ u.val = p4d_val_ma(val);
xen_extend_mmu_update(&u);
}
/*
- * Raw hypercall-based set_pgd, intended for in early boot before
+ * Raw hypercall-based set_p4d, intended for in early boot before
* there's a page structure. This implies:
* 1. The only existing pagetable is the kernel's
* 2. It is always pinned
* 3. It has no user pagetable attached to it
*/
-static void __init xen_set_pgd_hyper(pgd_t *ptr, pgd_t val)
+static void __init xen_set_p4d_hyper(p4d_t *ptr, p4d_t val)
{
preempt_disable();
xen_mc_batch();
- __xen_set_pgd_hyper(ptr, val);
+ __xen_set_p4d_hyper(ptr, val);
xen_mc_issue(PARAVIRT_LAZY_MMU);
preempt_enable();
}
-static void xen_set_pgd(pgd_t *ptr, pgd_t val)
+static void xen_set_p4d(p4d_t *ptr, p4d_t val)
{
- pgd_t *user_ptr = xen_get_user_pgd(ptr);
+ pgd_t *user_ptr = xen_get_user_pgd((pgd_t *)ptr);
+ pgd_t pgd_val;
- trace_xen_mmu_set_pgd(ptr, user_ptr, val);
+ trace_xen_mmu_set_p4d(ptr, (p4d_t *)user_ptr, val);
/* If page is not pinned, we can just update the entry
directly */
@@ -576,7 +577,8 @@
*ptr = val;
if (user_ptr) {
WARN_ON(xen_page_pinned(user_ptr));
- *user_ptr = val;
+ pgd_val.pgd = p4d_val_ma(val);
+ *user_ptr = pgd_val;
}
return;
}
@@ -585,14 +587,72 @@
user updates together. */
xen_mc_batch();
- __xen_set_pgd_hyper(ptr, val);
+ __xen_set_p4d_hyper(ptr, val);
if (user_ptr)
- __xen_set_pgd_hyper(user_ptr, val);
+ __xen_set_p4d_hyper((p4d_t *)user_ptr, val);
xen_mc_issue(PARAVIRT_LAZY_MMU);
}
#endif /* CONFIG_PGTABLE_LEVELS == 4 */
+static int xen_pmd_walk(struct mm_struct *mm, pmd_t *pmd,
+ int (*func)(struct mm_struct *mm, struct page *, enum pt_level),
+ bool last, unsigned long limit)
+{
+ int i, nr, flush = 0;
+
+ nr = last ? pmd_index(limit) + 1 : PTRS_PER_PMD;
+ for (i = 0; i < nr; i++) {
+ if (!pmd_none(pmd[i]))
+ flush |= (*func)(mm, pmd_page(pmd[i]), PT_PTE);
+ }
+ return flush;
+}
+
+static int xen_pud_walk(struct mm_struct *mm, pud_t *pud,
+ int (*func)(struct mm_struct *mm, struct page *, enum pt_level),
+ bool last, unsigned long limit)
+{
+ int i, nr, flush = 0;
+
+ nr = last ? pud_index(limit) + 1 : PTRS_PER_PUD;
+ for (i = 0; i < nr; i++) {
+ pmd_t *pmd;
+
+ if (pud_none(pud[i]))
+ continue;
+
+ pmd = pmd_offset(&pud[i], 0);
+ if (PTRS_PER_PMD > 1)
+ flush |= (*func)(mm, virt_to_page(pmd), PT_PMD);
+ flush |= xen_pmd_walk(mm, pmd, func,
+ last && i == nr - 1, limit);
+ }
+ return flush;
+}
+
+static int xen_p4d_walk(struct mm_struct *mm, p4d_t *p4d,
+ int (*func)(struct mm_struct *mm, struct page *, enum pt_level),
+ bool last, unsigned long limit)
+{
+ int i, nr, flush = 0;
+
+ nr = last ? p4d_index(limit) + 1 : PTRS_PER_P4D;
+ for (i = 0; i < nr; i++) {
+ pud_t *pud;
+
+ if (p4d_none(p4d[i]))
+ continue;
+
+ pud = pud_offset(&p4d[i], 0);
+ if (PTRS_PER_PUD > 1)
+ flush |= (*func)(mm, virt_to_page(pud), PT_PUD);
+ flush |= xen_pud_walk(mm, pud, func,
+ last && i == nr - 1, limit);
+ }
+ return flush;
+}
+
/*
* (Yet another) pagetable walker. This one is intended for pinning a
* pagetable. This means that it walks a pagetable and calls the
@@ -613,10 +673,8 @@
enum pt_level),
unsigned long limit)
{
- int flush = 0;
+ int i, nr, flush = 0;
unsigned hole_low, hole_high;
- unsigned pgdidx_limit, pudidx_limit, pmdidx_limit;
- unsigned pgdidx, pudidx, pmdidx;
/* The limit is the last byte to be touched */
limit--;
@@ -633,65 +691,22 @@
hole_low = pgd_index(USER_LIMIT);
hole_high = pgd_index(PAGE_OFFSET);
- pgdidx_limit = pgd_index(limit);
-#if PTRS_PER_PUD > 1
- pudidx_limit = pud_index(limit);
-#else
- pudidx_limit = 0;
-#endif
-#if PTRS_PER_PMD > 1
- pmdidx_limit = pmd_index(limit);
-#else
- pmdidx_limit = 0;
-#endif
+ nr = pgd_index(limit) + 1;
+ for (i = 0; i < nr; i++) {
+ p4d_t *p4d;
- for (pgdidx = 0; pgdidx <= pgdidx_limit; pgdidx++) {
- pud_t *pud;
-
- if (pgdidx >= hole_low && pgdidx < hole_high)
+ if (i >= hole_low && i < hole_high)
continue;
- if (!pgd_val(pgd[pgdidx]))
+ if (pgd_none(pgd[i]))
continue;
- pud = pud_offset(&pgd[pgdidx], 0);
-
- if (PTRS_PER_PUD > 1) /* not folded */
- flush |= (*func)(mm, virt_to_page(pud), PT_PUD);
-
- for (pudidx = 0; pudidx < PTRS_PER_PUD; pudidx++) {
- pmd_t *pmd;
-
- if (pgdidx == pgdidx_limit &&
- pudidx > pudidx_limit)
- goto out;
-
- if (pud_none(pud[pudidx]))
- continue;
-
- pmd = pmd_offset(&pud[pudidx], 0);
-
- if (PTRS_PER_PMD > 1) /* not folded */
- flush |= (*func)(mm, virt_to_page(pmd), PT_PMD);
-
- for (pmdidx = 0; pmdidx < PTRS_PER_PMD; pmdidx++) {
- struct page *pte;
-
- if (pgdidx == pgdidx_limit &&
- pudidx == pudidx_limit &&
- pmdidx > pmdidx_limit)
- goto out;
-
- if (pmd_none(pmd[pmdidx]))
- continue;
-
- pte = pmd_page(pmd[pmdidx]);
- flush |= (*func)(mm, pte, PT_PTE);
- }
- }
+ p4d = p4d_offset(&pgd[i], 0);
+ if (PTRS_PER_P4D > 1)
+ flush |= (*func)(mm, virt_to_page(p4d), PT_P4D);
+ flush |= xen_p4d_walk(mm, p4d, func, i == nr - 1, limit);
}
-out:
/* Do the top level last, so that the callbacks can use it as
a cue to do final things like tlb flushes. */
flush |= (*func)(mm, virt_to_page(pgd), PT_PGD);
@@ -1150,57 +1165,97 @@
xen_free_ro_pages(pa, PAGE_SIZE);
}
+static void __init xen_cleanmfnmap_pmd(pmd_t *pmd, bool unpin)
+{
+ unsigned long pa;
+ pte_t *pte_tbl;
+ int i;
+
+ if (pmd_large(*pmd)) {
+ pa = pmd_val(*pmd) & PHYSICAL_PAGE_MASK;
+ xen_free_ro_pages(pa, PMD_SIZE);
+ return;
+ }
+
+ pte_tbl = pte_offset_kernel(pmd, 0);
+ for (i = 0; i < PTRS_PER_PTE; i++) {
+ if (pte_none(pte_tbl[i]))
+ continue;
+ pa = pte_pfn(pte_tbl[i]) << PAGE_SHIFT;
+ xen_free_ro_pages(pa, PAGE_SIZE);
+ }
+ set_pmd(pmd, __pmd(0));
+ xen_cleanmfnmap_free_pgtbl(pte_tbl, unpin);
+}
+
+static void __init xen_cleanmfnmap_pud(pud_t *pud, bool unpin)
+{
+ unsigned long pa;
+ pmd_t *pmd_tbl;
+ int i;
+
+ if (pud_large(*pud)) {
+ pa = pud_val(*pud) & PHYSICAL_PAGE_MASK;
+ xen_free_ro_pages(pa, PUD_SIZE);
+ return;
+ }
+
+ pmd_tbl = pmd_offset(pud, 0);
+ for (i = 0; i < PTRS_PER_PMD; i++) {
+ if (pmd_none(pmd_tbl[i]))
+ continue;
+ xen_cleanmfnmap_pmd(pmd_tbl + i, unpin);
+ }
+ set_pud(pud, __pud(0));
+ xen_cleanmfnmap_free_pgtbl(pmd_tbl, unpin);
+}
+
+static void __init xen_cleanmfnmap_p4d(p4d_t *p4d, bool unpin)
+{
+ unsigned long pa;
+ pud_t *pud_tbl;
+ int i;
+
+ if (p4d_large(*p4d)) {
+ pa = p4d_val(*p4d) & PHYSICAL_PAGE_MASK;
+ xen_free_ro_pages(pa, P4D_SIZE);
+ return;
+ }
+
+ pud_tbl = pud_offset(p4d, 0);
+ for (i = 0; i < PTRS_PER_PUD; i++) {
+ if (pud_none(pud_tbl[i]))
+ continue;
+ xen_cleanmfnmap_pud(pud_tbl + i, unpin);
+ }
+ set_p4d(p4d, __p4d(0));
+ xen_cleanmfnmap_free_pgtbl(pud_tbl, unpin);
+}
+
/*
* Since it is well isolated we can (and since it is perhaps large we should)
* also free the page tables mapping the initial P->M table.
*/
static void __init xen_cleanmfnmap(unsigned long vaddr)
{
- unsigned long va = vaddr & PMD_MASK;
- unsigned long pa;
- pgd_t *pgd = pgd_offset_k(va);
- pud_t *pud_page = pud_offset(pgd, 0);
- pud_t *pud;
- pmd_t *pmd;
- pte_t *pte;
+ pgd_t *pgd;
+ p4d_t *p4d;
unsigned int i;
bool unpin;
unpin = (vaddr == 2 * PGDIR_SIZE);
- set_pgd(pgd, __pgd(0));
- do {
- pud = pud_page + pud_index(va);
- if (pud_none(*pud)) {
- va += PUD_SIZE;
- } else if (pud_large(*pud)) {
- pa = pud_val(*pud) & PHYSICAL_PAGE_MASK;
- xen_free_ro_pages(pa, PUD_SIZE);
- va += PUD_SIZE;
- } else {
- pmd = pmd_offset(pud, va);
- if (pmd_large(*pmd)) {
- pa = pmd_val(*pmd) & PHYSICAL_PAGE_MASK;
- xen_free_ro_pages(pa, PMD_SIZE);
- } else if (!pmd_none(*pmd)) {
- pte = pte_offset_kernel(pmd, va);
- set_pmd(pmd, __pmd(0));
- for (i = 0; i < PTRS_PER_PTE; ++i) {
- if (pte_none(pte[i]))
- break;
- pa = pte_pfn(pte[i]) << PAGE_SHIFT;
- xen_free_ro_pages(pa, PAGE_SIZE);
- }
- xen_cleanmfnmap_free_pgtbl(pte, unpin);
- }
- va += PMD_SIZE;
- if (pmd_index(va))
- continue;
- set_pud(pud, __pud(0));
- xen_cleanmfnmap_free_pgtbl(pmd, unpin);
- }
-
- } while (pud_index(va) || pmd_index(va));
- xen_cleanmfnmap_free_pgtbl(pud_page, unpin);
+ vaddr &= PMD_MASK;
+ pgd = pgd_offset_k(vaddr);
+ p4d = p4d_offset(pgd, 0);
+ for (i = 0; i < PTRS_PER_P4D; i++) {
+ if (p4d_none(p4d[i]))
+ continue;
+ xen_cleanmfnmap_p4d(p4d + i, unpin);
+ }
+ if (IS_ENABLED(CONFIG_X86_5LEVEL)) {
+ set_pgd(pgd, __pgd(0));
+ xen_cleanmfnmap_free_pgtbl(p4d, unpin);
+ }
}
static void __init xen_pagetable_p2m_free(void)
@@ -1538,7 +1593,6 @@
BUG_ON(PagePinned(virt_to_page(xen_get_user_pgd(pgd))));
}
#endif
-
return ret;
}
@@ -1730,7 +1784,7 @@
xen_release_ptpage(pfn, PT_PMD);
}
-#if CONFIG_PGTABLE_LEVELS == 4
+#if CONFIG_PGTABLE_LEVELS >= 4
static void xen_alloc_pud(struct mm_struct *mm, unsigned long pfn)
{
xen_alloc_ptpage(mm, pfn, PT_PUD);
@@ -2071,21 +2125,27 @@
*/
void __init xen_relocate_p2m(void)
{
- phys_addr_t size, new_area, pt_phys, pmd_phys, pud_phys;
+ phys_addr_t size, new_area, pt_phys, pmd_phys, pud_phys, p4d_phys;
unsigned long p2m_pfn, p2m_pfn_end, n_frames, pfn, pfn_end;
- int n_pte, n_pt, n_pmd, n_pud, idx_pte, idx_pt, idx_pmd, idx_pud;
+ int n_pte, n_pt, n_pmd, n_pud, n_p4d, idx_pte, idx_pt, idx_pmd, idx_pud, idx_p4d;
pte_t *pt;
pmd_t *pmd;
pud_t *pud;
+ p4d_t *p4d = NULL;
pgd_t *pgd;
unsigned long *new_p2m;
+ int save_pud;
size = PAGE_ALIGN(xen_start_info->nr_pages * sizeof(unsigned long));
n_pte = roundup(size, PAGE_SIZE) >> PAGE_SHIFT;
n_pt = roundup(size, PMD_SIZE) >> PMD_SHIFT;
n_pmd = roundup(size, PUD_SIZE) >> PUD_SHIFT;
- n_pud = roundup(size, PGDIR_SIZE) >> PGDIR_SHIFT;
- n_frames = n_pte + n_pt + n_pmd + n_pud;
+ n_pud = roundup(size, P4D_SIZE) >> P4D_SHIFT;
+ if (PTRS_PER_P4D > 1)
+ n_p4d = roundup(size, PGDIR_SIZE) >> PGDIR_SHIFT;
+ else
+ n_p4d = 0;
+ n_frames = n_pte + n_pt + n_pmd + n_pud + n_p4d;
new_area = xen_find_free_area(PFN_PHYS(n_frames));
if (!new_area) {
@@ -2101,55 +2161,76 @@
* To avoid any possible virtual address collision, just use
* 2 * PUD_SIZE for the new area.
*/
- pud_phys = new_area;
+ p4d_phys = new_area;
+ pud_phys = p4d_phys + PFN_PHYS(n_p4d);
pmd_phys = pud_phys + PFN_PHYS(n_pud);
pt_phys = pmd_phys + PFN_PHYS(n_pmd);
p2m_pfn = PFN_DOWN(pt_phys) + n_pt;
pgd = __va(read_cr3());
new_p2m = (unsigned long *)(2 * PGDIR_SIZE);
- for (idx_pud = 0; idx_pud < n_pud; idx_pud++) {
- pud = early_memremap(pud_phys, PAGE_SIZE);
- clear_page(pud);
- for (idx_pmd = 0; idx_pmd < min(n_pmd, PTRS_PER_PUD);
- idx_pmd++) {
- pmd = early_memremap(pmd_phys, PAGE_SIZE);
- clear_page(pmd);
- for (idx_pt = 0; idx_pt < min(n_pt, PTRS_PER_PMD);
- idx_pt++) {
- pt = early_memremap(pt_phys, PAGE_SIZE);
- clear_page(pt);
- for (idx_pte = 0;
- idx_pte < min(n_pte, PTRS_PER_PTE);
- idx_pte++) {
- set_pte(pt + idx_pte,
- pfn_pte(p2m_pfn, PAGE_KERNEL));
- p2m_pfn++;
- }
- n_pte -= PTRS_PER_PTE;
- early_memunmap(pt, PAGE_SIZE);
- make_lowmem_page_readonly(__va(pt_phys));
- pin_pagetable_pfn(MMUEXT_PIN_L1_TABLE,
- PFN_DOWN(pt_phys));
- set_pmd(pmd + idx_pt,
- __pmd(_PAGE_TABLE | pt_phys));
- pt_phys += PAGE_SIZE;
- }
- n_pt -= PTRS_PER_PMD;
- early_memunmap(pmd, PAGE_SIZE);
- make_lowmem_page_readonly(__va(pmd_phys));
- pin_pagetable_pfn(MMUEXT_PIN_L2_TABLE,
- PFN_DOWN(pmd_phys));
- set_pud(pud + idx_pmd, __pud(_PAGE_TABLE | pmd_phys));
- pmd_phys += PAGE_SIZE;
+ idx_p4d = 0;
+ save_pud = n_pud;
+ do {
+ if (n_p4d > 0) {
+ p4d = early_memremap(p4d_phys, PAGE_SIZE);
+ clear_page(p4d);
+ n_pud = min(save_pud, PTRS_PER_P4D);
}
- n_pmd -= PTRS_PER_PUD;
- early_memunmap(pud, PAGE_SIZE);
- make_lowmem_page_readonly(__va(pud_phys));
- pin_pagetable_pfn(MMUEXT_PIN_L3_TABLE, PFN_DOWN(pud_phys));
- set_pgd(pgd + 2 + idx_pud, __pgd(_PAGE_TABLE | pud_phys));
- pud_phys += PAGE_SIZE;
- }
+ for (idx_pud = 0; idx_pud < n_pud; idx_pud++) {
+ pud = early_memremap(pud_phys, PAGE_SIZE);
+ clear_page(pud);
+ for (idx_pmd = 0; idx_pmd < min(n_pmd, PTRS_PER_PUD);
+ idx_pmd++) {
+ pmd = early_memremap(pmd_phys, PAGE_SIZE);
+ clear_page(pmd);
+ for (idx_pt = 0; idx_pt < min(n_pt, PTRS_PER_PMD);
+ idx_pt++) {
+ pt = early_memremap(pt_phys, PAGE_SIZE);
+ clear_page(pt);
+ for (idx_pte = 0;
+ idx_pte < min(n_pte, PTRS_PER_PTE);
+ idx_pte++) {
+ set_pte(pt + idx_pte,
+ pfn_pte(p2m_pfn, PAGE_KERNEL));
+ p2m_pfn++;
+ }
+ n_pte -= PTRS_PER_PTE;
+ early_memunmap(pt, PAGE_SIZE);
+ make_lowmem_page_readonly(__va(pt_phys));
+ pin_pagetable_pfn(MMUEXT_PIN_L1_TABLE,
+ PFN_DOWN(pt_phys));
+ set_pmd(pmd + idx_pt,
+ __pmd(_PAGE_TABLE | pt_phys));
+ pt_phys += PAGE_SIZE;
+ }
+ n_pt -= PTRS_PER_PMD;
+ early_memunmap(pmd, PAGE_SIZE);
+ make_lowmem_page_readonly(__va(pmd_phys));
+ pin_pagetable_pfn(MMUEXT_PIN_L2_TABLE,
+ PFN_DOWN(pmd_phys));
+ set_pud(pud + idx_pmd, __pud(_PAGE_TABLE | pmd_phys));
+ pmd_phys += PAGE_SIZE;
+ }
+ n_pmd -= PTRS_PER_PUD;
+ early_memunmap(pud, PAGE_SIZE);
+ make_lowmem_page_readonly(__va(pud_phys));
+ pin_pagetable_pfn(MMUEXT_PIN_L3_TABLE, PFN_DOWN(pud_phys));
+ if (n_p4d > 0)
+ set_p4d(p4d + idx_pud, __p4d(_PAGE_TABLE | pud_phys));
+ else
+ set_pgd(pgd + 2 + idx_pud, __pgd(_PAGE_TABLE | pud_phys));
+ pud_phys += PAGE_SIZE;
+ }
+ if (n_p4d > 0) {
+ save_pud -= PTRS_PER_P4D;
+ early_memunmap(p4d, PAGE_SIZE);
+ make_lowmem_page_readonly(__va(p4d_phys));
+ pin_pagetable_pfn(MMUEXT_PIN_L4_TABLE, PFN_DOWN(p4d_phys));
+ set_pgd(pgd + 2 + idx_p4d, __pgd(_PAGE_TABLE | p4d_phys));
+ p4d_phys += PAGE_SIZE;
+ }
+ } while (++idx_p4d < n_p4d);
/* Now copy the old p2m info to the new area. */
memcpy(new_p2m, xen_p2m_addr, size);
@@ -2326,6 +2407,7 @@
#endif
case FIX_TEXT_POKE0:
case FIX_TEXT_POKE1:
+ case FIX_GDT_REMAP_BEGIN ... FIX_GDT_REMAP_END:
/* All local page mappings */
pte = pfn_pte(phys, prot);
break;
@@ -2378,8 +2460,8 @@
pv_mmu_ops.set_pte = xen_set_pte;
pv_mmu_ops.set_pmd = xen_set_pmd;
pv_mmu_ops.set_pud = xen_set_pud;
-#if CONFIG_PGTABLE_LEVELS == 4
- pv_mmu_ops.set_pgd = xen_set_pgd;
+#if CONFIG_PGTABLE_LEVELS >= 4
+ pv_mmu_ops.set_p4d = xen_set_p4d;
#endif
/* This will work as long as patching hasn't happened yet
@@ -2388,7 +2470,7 @@
pv_mmu_ops.alloc_pmd = xen_alloc_pmd;
pv_mmu_ops.release_pte = xen_release_pte;
pv_mmu_ops.release_pmd = xen_release_pmd;
-#if CONFIG_PGTABLE_LEVELS == 4
+#if CONFIG_PGTABLE_LEVELS >= 4
pv_mmu_ops.alloc_pud = xen_alloc_pud;
pv_mmu_ops.release_pud = xen_release_pud;
#endif
@@ -2454,10 +2536,10 @@
.make_pmd = PV_CALLEE_SAVE(xen_make_pmd),
.pmd_val = PV_CALLEE_SAVE(xen_pmd_val),
-#if CONFIG_PGTABLE_LEVELS == 4
+#if CONFIG_PGTABLE_LEVELS >= 4
.pud_val = PV_CALLEE_SAVE(xen_pud_val),
.make_pud = PV_CALLEE_SAVE(xen_make_pud),
- .set_pgd = xen_set_pgd_hyper,
+ .set_p4d = xen_set_p4d_hyper,
.alloc_pud = xen_alloc_pmd_init,
.release_pud = xen_release_pmd_init,
diff --git a/arch/x86/xen/mmu.h b/arch/x86/xen/mmu.h
index 73809bb..3fe2b32 100644
--- a/arch/x86/xen/mmu.h
+++ b/arch/x86/xen/mmu.h
@@ -5,6 +5,7 @@
enum pt_level {
PT_PGD,
+ PT_P4D,
PT_PUD,
PT_PMD,
PT_PTE
diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index a8c306c..a5bf7c4 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -14,7 +14,7 @@
#include <asm/elf.h>
#include <asm/vdso.h>
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/setup.h>
#include <asm/acpi.h>
#include <asm/numa.h>
@@ -41,8 +41,7 @@
unsigned long xen_released_pages;
/* E820 map used during setting up memory. */
-static struct e820entry xen_e820_map[E820_X_MAX] __initdata;
-static u32 xen_e820_map_entries __initdata;
+static struct e820_table xen_e820_table __initdata;
/*
* Buffer used to remap identity mapped pages. We only need the virtual space.
@@ -198,15 +197,15 @@
*/
static unsigned long __init xen_find_pfn_range(unsigned long *min_pfn)
{
- const struct e820entry *entry = xen_e820_map;
+ const struct e820_entry *entry = xen_e820_table.entries;
unsigned int i;
unsigned long done = 0;
- for (i = 0; i < xen_e820_map_entries; i++, entry++) {
+ for (i = 0; i < xen_e820_table.nr_entries; i++, entry++) {
unsigned long s_pfn;
unsigned long e_pfn;
- if (entry->type != E820_RAM)
+ if (entry->type != E820_TYPE_RAM)
continue;
e_pfn = PFN_DOWN(entry->addr + entry->size);
@@ -457,7 +456,7 @@
{
phys_addr_t start = 0;
unsigned long ret_val = 0;
- const struct e820entry *entry = xen_e820_map;
+ const struct e820_entry *entry = xen_e820_table.entries;
int i;
/*
@@ -471,13 +470,13 @@
* example) the DMI tables in a reserved region that begins on
* a non-page boundary.
*/
- for (i = 0; i < xen_e820_map_entries; i++, entry++) {
+ for (i = 0; i < xen_e820_table.nr_entries; i++, entry++) {
phys_addr_t end = entry->addr + entry->size;
- if (entry->type == E820_RAM || i == xen_e820_map_entries - 1) {
+ if (entry->type == E820_TYPE_RAM || i == xen_e820_table.nr_entries - 1) {
unsigned long start_pfn = PFN_DOWN(start);
unsigned long end_pfn = PFN_UP(end);
- if (entry->type == E820_RAM)
+ if (entry->type == E820_TYPE_RAM)
end_pfn = PFN_UP(entry->addr);
if (start_pfn < end_pfn)
@@ -591,28 +590,28 @@
phys_addr_t end = start + size;
/* Align RAM regions to page boundaries. */
- if (type == E820_RAM) {
+ if (type == E820_TYPE_RAM) {
start = PAGE_ALIGN(start);
end &= ~((phys_addr_t)PAGE_SIZE - 1);
}
- e820_add_region(start, end - start, type);
+ e820__range_add(start, end - start, type);
}
static void __init xen_ignore_unusable(void)
{
- struct e820entry *entry = xen_e820_map;
+ struct e820_entry *entry = xen_e820_table.entries;
unsigned int i;
- for (i = 0; i < xen_e820_map_entries; i++, entry++) {
- if (entry->type == E820_UNUSABLE)
- entry->type = E820_RAM;
+ for (i = 0; i < xen_e820_table.nr_entries; i++, entry++) {
+ if (entry->type == E820_TYPE_UNUSABLE)
+ entry->type = E820_TYPE_RAM;
}
}
bool __init xen_is_e820_reserved(phys_addr_t start, phys_addr_t size)
{
- struct e820entry *entry;
+ struct e820_entry *entry;
unsigned mapcnt;
phys_addr_t end;
@@ -620,10 +619,10 @@
return false;
end = start + size;
- entry = xen_e820_map;
+ entry = xen_e820_table.entries;
- for (mapcnt = 0; mapcnt < xen_e820_map_entries; mapcnt++) {
- if (entry->type == E820_RAM && entry->addr <= start &&
+ for (mapcnt = 0; mapcnt < xen_e820_table.nr_entries; mapcnt++) {
+ if (entry->type == E820_TYPE_RAM && entry->addr <= start &&
(entry->addr + entry->size) >= end)
return false;
@@ -645,10 +644,10 @@
{
unsigned mapcnt;
phys_addr_t addr, start;
- struct e820entry *entry = xen_e820_map;
+ struct e820_entry *entry = xen_e820_table.entries;
- for (mapcnt = 0; mapcnt < xen_e820_map_entries; mapcnt++, entry++) {
- if (entry->type != E820_RAM || entry->size < size)
+ for (mapcnt = 0; mapcnt < xen_e820_table.nr_entries; mapcnt++, entry++) {
+ if (entry->type != E820_TYPE_RAM || entry->size < size)
continue;
start = entry->addr;
for (addr = start; addr < start + size; addr += PAGE_SIZE) {
@@ -750,8 +749,8 @@
max_pfn = min(max_pfn, xen_start_info->nr_pages);
mem_end = PFN_PHYS(max_pfn);
- memmap.nr_entries = ARRAY_SIZE(xen_e820_map);
- set_xen_guest_handle(memmap.buffer, xen_e820_map);
+ memmap.nr_entries = ARRAY_SIZE(xen_e820_table.entries);
+ set_xen_guest_handle(memmap.buffer, xen_e820_table.entries);
op = xen_initial_domain() ?
XENMEM_machine_memory_map :
@@ -760,16 +759,16 @@
if (rc == -ENOSYS) {
BUG_ON(xen_initial_domain());
memmap.nr_entries = 1;
- xen_e820_map[0].addr = 0ULL;
- xen_e820_map[0].size = mem_end;
+ xen_e820_table.entries[0].addr = 0ULL;
+ xen_e820_table.entries[0].size = mem_end;
/* 8MB slack (to balance backend allocations). */
- xen_e820_map[0].size += 8ULL << 20;
- xen_e820_map[0].type = E820_RAM;
+ xen_e820_table.entries[0].size += 8ULL << 20;
+ xen_e820_table.entries[0].type = E820_TYPE_RAM;
rc = 0;
}
BUG_ON(rc);
BUG_ON(memmap.nr_entries == 0);
- xen_e820_map_entries = memmap.nr_entries;
+ xen_e820_table.nr_entries = memmap.nr_entries;
/*
* Xen won't allow a 1:1 mapping to be created to UNUSABLE
@@ -783,8 +782,7 @@
xen_ignore_unusable();
/* Make sure the Xen-supplied memory map is well-ordered. */
- sanitize_e820_map(xen_e820_map, ARRAY_SIZE(xen_e820_map),
- &xen_e820_map_entries);
+ e820__update_table(&xen_e820_table);
max_pages = xen_get_max_pages();
@@ -811,15 +809,15 @@
extra_pages = min3(EXTRA_MEM_RATIO * min(max_pfn, PFN_DOWN(MAXMEM)),
extra_pages, max_pages - max_pfn);
i = 0;
- addr = xen_e820_map[0].addr;
- size = xen_e820_map[0].size;
- while (i < xen_e820_map_entries) {
+ addr = xen_e820_table.entries[0].addr;
+ size = xen_e820_table.entries[0].size;
+ while (i < xen_e820_table.nr_entries) {
bool discard = false;
chunk_size = size;
- type = xen_e820_map[i].type;
+ type = xen_e820_table.entries[i].type;
- if (type == E820_RAM) {
+ if (type == E820_TYPE_RAM) {
if (addr < mem_end) {
chunk_size = min(size, mem_end - addr);
} else if (extra_pages) {
@@ -840,9 +838,9 @@
size -= chunk_size;
if (size == 0) {
i++;
- if (i < xen_e820_map_entries) {
- addr = xen_e820_map[i].addr;
- size = xen_e820_map[i].size;
+ if (i < xen_e820_table.nr_entries) {
+ addr = xen_e820_table.entries[i].addr;
+ size = xen_e820_table.entries[i].size;
}
}
}
@@ -858,10 +856,9 @@
* reserve ISA memory anyway because too many things poke
* about in there.
*/
- e820_add_region(ISA_START_ADDRESS, ISA_END_ADDRESS - ISA_START_ADDRESS,
- E820_RESERVED);
+ e820__range_add(ISA_START_ADDRESS, ISA_END_ADDRESS - ISA_START_ADDRESS, E820_TYPE_RESERVED);
- sanitize_e820_map(e820->map, ARRAY_SIZE(e820->map), &e820->nr_map);
+ e820__update_table(e820_table);
/*
* Check whether the kernel itself conflicts with the target E820 map.
@@ -915,6 +912,37 @@
}
/*
+ * Machine specific memory setup for auto-translated guests.
+ */
+char * __init xen_auto_xlated_memory_setup(void)
+{
+ struct xen_memory_map memmap;
+ int i;
+ int rc;
+
+ memmap.nr_entries = ARRAY_SIZE(xen_e820_table.entries);
+ set_xen_guest_handle(memmap.buffer, xen_e820_table.entries);
+
+ rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap);
+ if (rc < 0)
+ panic("No memory map (%d)\n", rc);
+
+ xen_e820_table.nr_entries = memmap.nr_entries;
+
+ e820__update_table(&xen_e820_table);
+
+ for (i = 0; i < xen_e820_table.nr_entries; i++)
+ e820__range_add(xen_e820_table.entries[i].addr, xen_e820_table.entries[i].size, xen_e820_table.entries[i].type);
+
+ /* Remove p2m info, it is not needed. */
+ xen_start_info->mfn_list = 0;
+ xen_start_info->first_p2m_pfn = 0;
+ xen_start_info->nr_p2m_frames = 0;
+
+ return "Xen";
+}
+
+/*
* Set the bit indicating "nosegneg" library variants should be used.
* We only need to bother in pure 32-bit mode; compat 32-bit processes
* can have un-truncated segments, so wrapping around is allowed.
@@ -999,8 +1027,8 @@
void __init xen_arch_setup(void)
{
xen_panic_handler_init();
-
- xen_pvmmu_arch_setup();
+ if (!xen_feature(XENFEAT_auto_translated_physmap))
+ xen_pvmmu_arch_setup();
#ifdef CONFIG_ACPI
if (!(xen_start_info->flags & SIF_INITDOMAIN)) {
diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c
index 7ff2f1b..eaa3616 100644
--- a/arch/x86/xen/smp.c
+++ b/arch/x86/xen/smp.c
@@ -392,7 +392,7 @@
if (ctxt == NULL)
return -ENOMEM;
- gdt = get_cpu_gdt_table(cpu);
+ gdt = get_cpu_gdt_rw(cpu);
#ifdef CONFIG_X86_32
ctxt->user_regs.fs = __KERNEL_PERCPU;
diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
index 1e69956..7a30892 100644
--- a/arch/x86/xen/time.c
+++ b/arch/x86/xen/time.c
@@ -209,7 +209,9 @@
.features = CLOCK_EVT_FEAT_ONESHOT,
.max_delta_ns = 0xffffffff,
+ .max_delta_ticks = 0xffffffff,
.min_delta_ns = TIMER_SLOP,
+ .min_delta_ticks = TIMER_SLOP,
.mult = 1,
.shift = 0,
@@ -268,7 +270,9 @@
.features = CLOCK_EVT_FEAT_ONESHOT,
.max_delta_ns = 0xffffffff,
+ .max_delta_ticks = 0xffffffff,
.min_delta_ns = TIMER_SLOP,
+ .min_delta_ticks = TIMER_SLOP,
.mult = 1,
.shift = 0,
diff --git a/arch/xtensa/include/asm/Kbuild b/arch/xtensa/include/asm/Kbuild
index f41408c..cc23e9e 100644
--- a/arch/xtensa/include/asm/Kbuild
+++ b/arch/xtensa/include/asm/Kbuild
@@ -6,6 +6,7 @@
generic-y += emergency-restart.h
generic-y += errno.h
generic-y += exec.h
+generic-y += extable.h
generic-y += fcntl.h
generic-y += hardirq.h
generic-y += ioctl.h
diff --git a/arch/xtensa/include/asm/asm-uaccess.h b/arch/xtensa/include/asm/asm-uaccess.h
index a7a1100..dfdf9fa 100644
--- a/arch/xtensa/include/asm/asm-uaccess.h
+++ b/arch/xtensa/include/asm/asm-uaccess.h
@@ -19,9 +19,6 @@
#include <linux/errno.h>
#include <asm/types.h>
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
#include <asm/current.h>
#include <asm/asm-offsets.h>
#include <asm/processor.h>
diff --git a/arch/xtensa/include/asm/uaccess.h b/arch/xtensa/include/asm/uaccess.h
index 848a3d7..2e7bac0 100644
--- a/arch/xtensa/include/asm/uaccess.h
+++ b/arch/xtensa/include/asm/uaccess.h
@@ -16,14 +16,9 @@
#ifndef _XTENSA_UACCESS_H
#define _XTENSA_UACCESS_H
-#include <linux/errno.h>
#include <linux/prefetch.h>
#include <asm/types.h>
-
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
-#include <linux/sched.h>
+#include <asm/extable.h>
/*
* The fs value determines whether argument validity checking should
@@ -43,7 +38,7 @@
#define segment_eq(a, b) ((a).seg == (b).seg)
-#define __kernel_ok (segment_eq(get_fs(), KERNEL_DS))
+#define __kernel_ok (uaccess_kernel())
#define __user_ok(addr, size) \
(((size) <= TASK_SIZE)&&((addr) <= TASK_SIZE-(size)))
#define __access_ok(addr, size) (__kernel_ok || __user_ok((addr), (size)))
@@ -239,60 +234,22 @@
* Copy to/from user space
*/
-/*
- * We use a generic, arbitrary-sized copy subroutine. The Xtensa
- * architecture would cause heavy code bloat if we tried to inline
- * these functions and provide __constant_copy_* equivalents like the
- * i386 versions. __xtensa_copy_user is quite efficient. See the
- * .fixup section of __xtensa_copy_user for a discussion on the
- * X_zeroing equivalents for Xtensa.
- */
-
extern unsigned __xtensa_copy_user(void *to, const void *from, unsigned n);
-#define __copy_user(to, from, size) __xtensa_copy_user(to, from, size)
-
static inline unsigned long
-__generic_copy_from_user_nocheck(void *to, const void *from, unsigned long n)
-{
- return __copy_user(to, from, n);
-}
-
-static inline unsigned long
-__generic_copy_to_user_nocheck(void *to, const void *from, unsigned long n)
-{
- return __copy_user(to, from, n);
-}
-
-static inline unsigned long
-__generic_copy_to_user(void *to, const void *from, unsigned long n)
-{
- prefetch(from);
- if (access_ok(VERIFY_WRITE, to, n))
- return __copy_user(to, from, n);
- return n;
-}
-
-static inline unsigned long
-__generic_copy_from_user(void *to, const void *from, unsigned long n)
+raw_copy_from_user(void *to, const void __user *from, unsigned long n)
{
prefetchw(to);
- if (access_ok(VERIFY_READ, from, n))
- return __copy_user(to, from, n);
- else
- memset(to, 0, n);
- return n;
+ return __xtensa_copy_user(to, (__force const void *)from, n);
}
-
-#define copy_to_user(to, from, n) __generic_copy_to_user((to), (from), (n))
-#define copy_from_user(to, from, n) __generic_copy_from_user((to), (from), (n))
-#define __copy_to_user(to, from, n) \
- __generic_copy_to_user_nocheck((to), (from), (n))
-#define __copy_from_user(to, from, n) \
- __generic_copy_from_user_nocheck((to), (from), (n))
-#define __copy_to_user_inatomic __copy_to_user
-#define __copy_from_user_inatomic __copy_from_user
-
+static inline unsigned long
+raw_copy_to_user(void __user *to, const void *from, unsigned long n)
+{
+ prefetch(from);
+ return __xtensa_copy_user((__force void *)to, from, n);
+}
+#define INLINE_COPY_FROM_USER
+#define INLINE_COPY_TO_USER
/*
* We need to return the number of bytes not cleared. Our memset()
@@ -348,10 +305,4 @@
return __strnlen_user(str, len);
}
-
-struct exception_table_entry
-{
- unsigned long insn, fixup;
-};
-
#endif /* _XTENSA_UACCESS_H */
diff --git a/arch/xtensa/include/uapi/asm/socket.h b/arch/xtensa/include/uapi/asm/socket.h
index 9fdbe1f..1eb6d2f 100644
--- a/arch/xtensa/include/uapi/asm/socket.h
+++ b/arch/xtensa/include/uapi/asm/socket.h
@@ -103,4 +103,10 @@
#define SCM_TIMESTAMPING_OPT_STATS 54
+#define SO_MEMINFO 55
+
+#define SO_INCOMING_NAPI_ID 56
+
+#define SO_COOKIE 57
+
#endif /* _XTENSA_SOCKET_H */
diff --git a/arch/xtensa/lib/usercopy.S b/arch/xtensa/lib/usercopy.S
index 7ea4dd6..d9cd766 100644
--- a/arch/xtensa/lib/usercopy.S
+++ b/arch/xtensa/lib/usercopy.S
@@ -102,9 +102,9 @@
bltui a4, 7, .Lbytecopy # do short copies byte by byte
# copy 1 byte
- EX(l8ui, a6, a3, 0, l_fixup)
+ EX(l8ui, a6, a3, 0, fixup)
addi a3, a3, 1
- EX(s8i, a6, a5, 0, s_fixup)
+ EX(s8i, a6, a5, 0, fixup)
addi a5, a5, 1
addi a4, a4, -1
bbci.l a5, 1, .Ldstaligned # if dst is now aligned, then
@@ -112,11 +112,11 @@
.Ldst2mod4: # dst 16-bit aligned
# copy 2 bytes
bltui a4, 6, .Lbytecopy # do short copies byte by byte
- EX(l8ui, a6, a3, 0, l_fixup)
- EX(l8ui, a7, a3, 1, l_fixup)
+ EX(l8ui, a6, a3, 0, fixup)
+ EX(l8ui, a7, a3, 1, fixup)
addi a3, a3, 2
- EX(s8i, a6, a5, 0, s_fixup)
- EX(s8i, a7, a5, 1, s_fixup)
+ EX(s8i, a6, a5, 0, fixup)
+ EX(s8i, a7, a5, 1, fixup)
addi a5, a5, 2
addi a4, a4, -2
j .Ldstaligned # dst is now aligned, return to main algorithm
@@ -135,9 +135,9 @@
add a7, a3, a4 # a7 = end address for source
#endif /* !XCHAL_HAVE_LOOPS */
.Lnextbyte:
- EX(l8ui, a6, a3, 0, l_fixup)
+ EX(l8ui, a6, a3, 0, fixup)
addi a3, a3, 1
- EX(s8i, a6, a5, 0, s_fixup)
+ EX(s8i, a6, a5, 0, fixup)
addi a5, a5, 1
#if !XCHAL_HAVE_LOOPS
blt a3, a7, .Lnextbyte
@@ -161,15 +161,15 @@
add a8, a8, a3 # a8 = end of last 16B source chunk
#endif /* !XCHAL_HAVE_LOOPS */
.Loop1:
- EX(l32i, a6, a3, 0, l_fixup)
- EX(l32i, a7, a3, 4, l_fixup)
- EX(s32i, a6, a5, 0, s_fixup)
- EX(l32i, a6, a3, 8, l_fixup)
- EX(s32i, a7, a5, 4, s_fixup)
- EX(l32i, a7, a3, 12, l_fixup)
- EX(s32i, a6, a5, 8, s_fixup)
+ EX(l32i, a6, a3, 0, fixup)
+ EX(l32i, a7, a3, 4, fixup)
+ EX(s32i, a6, a5, 0, fixup)
+ EX(l32i, a6, a3, 8, fixup)
+ EX(s32i, a7, a5, 4, fixup)
+ EX(l32i, a7, a3, 12, fixup)
+ EX(s32i, a6, a5, 8, fixup)
addi a3, a3, 16
- EX(s32i, a7, a5, 12, s_fixup)
+ EX(s32i, a7, a5, 12, fixup)
addi a5, a5, 16
#if !XCHAL_HAVE_LOOPS
blt a3, a8, .Loop1
@@ -177,31 +177,31 @@
.Loop1done:
bbci.l a4, 3, .L2
# copy 8 bytes
- EX(l32i, a6, a3, 0, l_fixup)
- EX(l32i, a7, a3, 4, l_fixup)
+ EX(l32i, a6, a3, 0, fixup)
+ EX(l32i, a7, a3, 4, fixup)
addi a3, a3, 8
- EX(s32i, a6, a5, 0, s_fixup)
- EX(s32i, a7, a5, 4, s_fixup)
+ EX(s32i, a6, a5, 0, fixup)
+ EX(s32i, a7, a5, 4, fixup)
addi a5, a5, 8
.L2:
bbci.l a4, 2, .L3
# copy 4 bytes
- EX(l32i, a6, a3, 0, l_fixup)
+ EX(l32i, a6, a3, 0, fixup)
addi a3, a3, 4
- EX(s32i, a6, a5, 0, s_fixup)
+ EX(s32i, a6, a5, 0, fixup)
addi a5, a5, 4
.L3:
bbci.l a4, 1, .L4
# copy 2 bytes
- EX(l16ui, a6, a3, 0, l_fixup)
+ EX(l16ui, a6, a3, 0, fixup)
addi a3, a3, 2
- EX(s16i, a6, a5, 0, s_fixup)
+ EX(s16i, a6, a5, 0, fixup)
addi a5, a5, 2
.L4:
bbci.l a4, 0, .L5
# copy 1 byte
- EX(l8ui, a6, a3, 0, l_fixup)
- EX(s8i, a6, a5, 0, s_fixup)
+ EX(l8ui, a6, a3, 0, fixup)
+ EX(s8i, a6, a5, 0, fixup)
.L5:
movi a2, 0 # return success for len bytes copied
retw
@@ -217,7 +217,7 @@
# copy 16 bytes per iteration for word-aligned dst and unaligned src
and a10, a3, a8 # save unalignment offset for below
sub a3, a3, a10 # align a3 (to avoid sim warnings only; not needed for hardware)
- EX(l32i, a6, a3, 0, l_fixup) # load first word
+ EX(l32i, a6, a3, 0, fixup) # load first word
#if XCHAL_HAVE_LOOPS
loopnez a7, .Loop2done
#else /* !XCHAL_HAVE_LOOPS */
@@ -226,19 +226,19 @@
add a12, a12, a3 # a12 = end of last 16B source chunk
#endif /* !XCHAL_HAVE_LOOPS */
.Loop2:
- EX(l32i, a7, a3, 4, l_fixup)
- EX(l32i, a8, a3, 8, l_fixup)
+ EX(l32i, a7, a3, 4, fixup)
+ EX(l32i, a8, a3, 8, fixup)
ALIGN( a6, a6, a7)
- EX(s32i, a6, a5, 0, s_fixup)
- EX(l32i, a9, a3, 12, l_fixup)
+ EX(s32i, a6, a5, 0, fixup)
+ EX(l32i, a9, a3, 12, fixup)
ALIGN( a7, a7, a8)
- EX(s32i, a7, a5, 4, s_fixup)
- EX(l32i, a6, a3, 16, l_fixup)
+ EX(s32i, a7, a5, 4, fixup)
+ EX(l32i, a6, a3, 16, fixup)
ALIGN( a8, a8, a9)
- EX(s32i, a8, a5, 8, s_fixup)
+ EX(s32i, a8, a5, 8, fixup)
addi a3, a3, 16
ALIGN( a9, a9, a6)
- EX(s32i, a9, a5, 12, s_fixup)
+ EX(s32i, a9, a5, 12, fixup)
addi a5, a5, 16
#if !XCHAL_HAVE_LOOPS
blt a3, a12, .Loop2
@@ -246,39 +246,39 @@
.Loop2done:
bbci.l a4, 3, .L12
# copy 8 bytes
- EX(l32i, a7, a3, 4, l_fixup)
- EX(l32i, a8, a3, 8, l_fixup)
+ EX(l32i, a7, a3, 4, fixup)
+ EX(l32i, a8, a3, 8, fixup)
ALIGN( a6, a6, a7)
- EX(s32i, a6, a5, 0, s_fixup)
+ EX(s32i, a6, a5, 0, fixup)
addi a3, a3, 8
ALIGN( a7, a7, a8)
- EX(s32i, a7, a5, 4, s_fixup)
+ EX(s32i, a7, a5, 4, fixup)
addi a5, a5, 8
mov a6, a8
.L12:
bbci.l a4, 2, .L13
# copy 4 bytes
- EX(l32i, a7, a3, 4, l_fixup)
+ EX(l32i, a7, a3, 4, fixup)
addi a3, a3, 4
ALIGN( a6, a6, a7)
- EX(s32i, a6, a5, 0, s_fixup)
+ EX(s32i, a6, a5, 0, fixup)
addi a5, a5, 4
mov a6, a7
.L13:
add a3, a3, a10 # readjust a3 with correct misalignment
bbci.l a4, 1, .L14
# copy 2 bytes
- EX(l8ui, a6, a3, 0, l_fixup)
- EX(l8ui, a7, a3, 1, l_fixup)
+ EX(l8ui, a6, a3, 0, fixup)
+ EX(l8ui, a7, a3, 1, fixup)
addi a3, a3, 2
- EX(s8i, a6, a5, 0, s_fixup)
- EX(s8i, a7, a5, 1, s_fixup)
+ EX(s8i, a6, a5, 0, fixup)
+ EX(s8i, a7, a5, 1, fixup)
addi a5, a5, 2
.L14:
bbci.l a4, 0, .L15
# copy 1 byte
- EX(l8ui, a6, a3, 0, l_fixup)
- EX(s8i, a6, a5, 0, s_fixup)
+ EX(l8ui, a6, a3, 0, fixup)
+ EX(s8i, a6, a5, 0, fixup)
.L15:
movi a2, 0 # return success for len bytes copied
retw
@@ -291,30 +291,10 @@
* bytes_copied = a5 - a2
* retval = bytes_not_copied = original len - bytes_copied
* retval = a11 - (a5 - a2)
- *
- * Clearing the remaining pieces of kernel memory plugs security
- * holes. This functionality is the equivalent of the *_zeroing
- * functions that some architectures provide.
*/
-.Lmemset:
- .word memset
-s_fixup:
+fixup:
sub a2, a5, a2 /* a2 <-- bytes copied */
sub a2, a11, a2 /* a2 <-- bytes not copied */
retw
-
-l_fixup:
- sub a2, a5, a2 /* a2 <-- bytes copied */
- sub a2, a11, a2 /* a2 <-- bytes not copied == return value */
-
- /* void *memset(void *s, int c, size_t n); */
- mov a6, a5 /* s */
- movi a7, 0 /* c */
- mov a8, a2 /* n */
- l32r a4, .Lmemset
- callx4 a4
- /* Ignore memset return value in a6. */
- /* a2 still contains bytes not copied. */
- retw
diff --git a/block/Kconfig b/block/Kconfig
index e9f780f..89cd28f 100644
--- a/block/Kconfig
+++ b/block/Kconfig
@@ -115,6 +115,18 @@
See Documentation/cgroups/blkio-controller.txt for more information.
+config BLK_DEV_THROTTLING_LOW
+ bool "Block throttling .low limit interface support (EXPERIMENTAL)"
+ depends on BLK_DEV_THROTTLING
+ default n
+ ---help---
+ Add .low limit interface for block throttling. The low limit is a best
+ effort limit to prioritize cgroups. Depending on the setting, the limit
+ can be used to protect cgroups in terms of bandwidth/iops and better
+ utilize disk resource.
+
+ Note, this is an experimental interface and could be changed someday.
+
config BLK_CMDLINE_PARSER
bool "Block device command line partition parser"
default n
diff --git a/block/Kconfig.iosched b/block/Kconfig.iosched
index 58fc868..fd2cefa 100644
--- a/block/Kconfig.iosched
+++ b/block/Kconfig.iosched
@@ -40,6 +40,7 @@
Enable group IO scheduling in CFQ.
choice
+
prompt "Default I/O scheduler"
default DEFAULT_CFQ
help
@@ -69,6 +70,35 @@
---help---
MQ version of the deadline IO scheduler.
+config MQ_IOSCHED_KYBER
+ tristate "Kyber I/O scheduler"
+ default y
+ ---help---
+ The Kyber I/O scheduler is a low-overhead scheduler suitable for
+ multiqueue and other fast devices. Given target latencies for reads and
+ synchronous writes, it will self-tune queue depths to achieve that
+ goal.
+
+config IOSCHED_BFQ
+ tristate "BFQ I/O scheduler"
+ default n
+ ---help---
+ BFQ I/O scheduler for BLK-MQ. BFQ distributes the bandwidth of
+ of the device among all processes according to their weights,
+ regardless of the device parameters and with any workload. It
+ also guarantees a low latency to interactive and soft
+ real-time applications. Details in
+ Documentation/block/bfq-iosched.txt
+
+config BFQ_GROUP_IOSCHED
+ bool "BFQ hierarchical scheduling support"
+ depends on IOSCHED_BFQ && BLK_CGROUP
+ default n
+ ---help---
+
+ Enable hierarchical scheduling in BFQ, using the blkio
+ (cgroups-v1) or io (cgroups-v2) controller.
+
endmenu
endif
diff --git a/block/Makefile b/block/Makefile
index 081bb68..2b281cf2 100644
--- a/block/Makefile
+++ b/block/Makefile
@@ -20,6 +20,9 @@
obj-$(CONFIG_IOSCHED_DEADLINE) += deadline-iosched.o
obj-$(CONFIG_IOSCHED_CFQ) += cfq-iosched.o
obj-$(CONFIG_MQ_IOSCHED_DEADLINE) += mq-deadline.o
+obj-$(CONFIG_MQ_IOSCHED_KYBER) += kyber-iosched.o
+bfq-y := bfq-iosched.o bfq-wf2q.o bfq-cgroup.o
+obj-$(CONFIG_IOSCHED_BFQ) += bfq.o
obj-$(CONFIG_BLOCK_COMPAT) += compat_ioctl.o
obj-$(CONFIG_BLK_CMDLINE_PARSER) += cmdline-parser.o
diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c
new file mode 100644
index 0000000..c8a32fb
--- /dev/null
+++ b/block/bfq-cgroup.c
@@ -0,0 +1,1139 @@
+/*
+ * cgroups support for the BFQ I/O scheduler.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/blkdev.h>
+#include <linux/cgroup.h>
+#include <linux/elevator.h>
+#include <linux/ktime.h>
+#include <linux/rbtree.h>
+#include <linux/ioprio.h>
+#include <linux/sbitmap.h>
+#include <linux/delay.h>
+
+#include "bfq-iosched.h"
+
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+
+/* bfqg stats flags */
+enum bfqg_stats_flags {
+ BFQG_stats_waiting = 0,
+ BFQG_stats_idling,
+ BFQG_stats_empty,
+};
+
+#define BFQG_FLAG_FNS(name) \
+static void bfqg_stats_mark_##name(struct bfqg_stats *stats) \
+{ \
+ stats->flags |= (1 << BFQG_stats_##name); \
+} \
+static void bfqg_stats_clear_##name(struct bfqg_stats *stats) \
+{ \
+ stats->flags &= ~(1 << BFQG_stats_##name); \
+} \
+static int bfqg_stats_##name(struct bfqg_stats *stats) \
+{ \
+ return (stats->flags & (1 << BFQG_stats_##name)) != 0; \
+} \
+
+BFQG_FLAG_FNS(waiting)
+BFQG_FLAG_FNS(idling)
+BFQG_FLAG_FNS(empty)
+#undef BFQG_FLAG_FNS
+
+/* This should be called with the queue_lock held. */
+static void bfqg_stats_update_group_wait_time(struct bfqg_stats *stats)
+{
+ unsigned long long now;
+
+ if (!bfqg_stats_waiting(stats))
+ return;
+
+ now = sched_clock();
+ if (time_after64(now, stats->start_group_wait_time))
+ blkg_stat_add(&stats->group_wait_time,
+ now - stats->start_group_wait_time);
+ bfqg_stats_clear_waiting(stats);
+}
+
+/* This should be called with the queue_lock held. */
+static void bfqg_stats_set_start_group_wait_time(struct bfq_group *bfqg,
+ struct bfq_group *curr_bfqg)
+{
+ struct bfqg_stats *stats = &bfqg->stats;
+
+ if (bfqg_stats_waiting(stats))
+ return;
+ if (bfqg == curr_bfqg)
+ return;
+ stats->start_group_wait_time = sched_clock();
+ bfqg_stats_mark_waiting(stats);
+}
+
+/* This should be called with the queue_lock held. */
+static void bfqg_stats_end_empty_time(struct bfqg_stats *stats)
+{
+ unsigned long long now;
+
+ if (!bfqg_stats_empty(stats))
+ return;
+
+ now = sched_clock();
+ if (time_after64(now, stats->start_empty_time))
+ blkg_stat_add(&stats->empty_time,
+ now - stats->start_empty_time);
+ bfqg_stats_clear_empty(stats);
+}
+
+void bfqg_stats_update_dequeue(struct bfq_group *bfqg)
+{
+ blkg_stat_add(&bfqg->stats.dequeue, 1);
+}
+
+void bfqg_stats_set_start_empty_time(struct bfq_group *bfqg)
+{
+ struct bfqg_stats *stats = &bfqg->stats;
+
+ if (blkg_rwstat_total(&stats->queued))
+ return;
+
+ /*
+ * group is already marked empty. This can happen if bfqq got new
+ * request in parent group and moved to this group while being added
+ * to service tree. Just ignore the event and move on.
+ */
+ if (bfqg_stats_empty(stats))
+ return;
+
+ stats->start_empty_time = sched_clock();
+ bfqg_stats_mark_empty(stats);
+}
+
+void bfqg_stats_update_idle_time(struct bfq_group *bfqg)
+{
+ struct bfqg_stats *stats = &bfqg->stats;
+
+ if (bfqg_stats_idling(stats)) {
+ unsigned long long now = sched_clock();
+
+ if (time_after64(now, stats->start_idle_time))
+ blkg_stat_add(&stats->idle_time,
+ now - stats->start_idle_time);
+ bfqg_stats_clear_idling(stats);
+ }
+}
+
+void bfqg_stats_set_start_idle_time(struct bfq_group *bfqg)
+{
+ struct bfqg_stats *stats = &bfqg->stats;
+
+ stats->start_idle_time = sched_clock();
+ bfqg_stats_mark_idling(stats);
+}
+
+void bfqg_stats_update_avg_queue_size(struct bfq_group *bfqg)
+{
+ struct bfqg_stats *stats = &bfqg->stats;
+
+ blkg_stat_add(&stats->avg_queue_size_sum,
+ blkg_rwstat_total(&stats->queued));
+ blkg_stat_add(&stats->avg_queue_size_samples, 1);
+ bfqg_stats_update_group_wait_time(stats);
+}
+
+/*
+ * blk-cgroup policy-related handlers
+ * The following functions help in converting between blk-cgroup
+ * internal structures and BFQ-specific structures.
+ */
+
+static struct bfq_group *pd_to_bfqg(struct blkg_policy_data *pd)
+{
+ return pd ? container_of(pd, struct bfq_group, pd) : NULL;
+}
+
+struct blkcg_gq *bfqg_to_blkg(struct bfq_group *bfqg)
+{
+ return pd_to_blkg(&bfqg->pd);
+}
+
+static struct bfq_group *blkg_to_bfqg(struct blkcg_gq *blkg)
+{
+ return pd_to_bfqg(blkg_to_pd(blkg, &blkcg_policy_bfq));
+}
+
+/*
+ * bfq_group handlers
+ * The following functions help in navigating the bfq_group hierarchy
+ * by allowing to find the parent of a bfq_group or the bfq_group
+ * associated to a bfq_queue.
+ */
+
+static struct bfq_group *bfqg_parent(struct bfq_group *bfqg)
+{
+ struct blkcg_gq *pblkg = bfqg_to_blkg(bfqg)->parent;
+
+ return pblkg ? blkg_to_bfqg(pblkg) : NULL;
+}
+
+struct bfq_group *bfqq_group(struct bfq_queue *bfqq)
+{
+ struct bfq_entity *group_entity = bfqq->entity.parent;
+
+ return group_entity ? container_of(group_entity, struct bfq_group,
+ entity) :
+ bfqq->bfqd->root_group;
+}
+
+/*
+ * The following two functions handle get and put of a bfq_group by
+ * wrapping the related blk-cgroup hooks.
+ */
+
+static void bfqg_get(struct bfq_group *bfqg)
+{
+ return blkg_get(bfqg_to_blkg(bfqg));
+}
+
+void bfqg_put(struct bfq_group *bfqg)
+{
+ return blkg_put(bfqg_to_blkg(bfqg));
+}
+
+void bfqg_stats_update_io_add(struct bfq_group *bfqg, struct bfq_queue *bfqq,
+ unsigned int op)
+{
+ blkg_rwstat_add(&bfqg->stats.queued, op, 1);
+ bfqg_stats_end_empty_time(&bfqg->stats);
+ if (!(bfqq == ((struct bfq_data *)bfqg->bfqd)->in_service_queue))
+ bfqg_stats_set_start_group_wait_time(bfqg, bfqq_group(bfqq));
+}
+
+void bfqg_stats_update_io_remove(struct bfq_group *bfqg, unsigned int op)
+{
+ blkg_rwstat_add(&bfqg->stats.queued, op, -1);
+}
+
+void bfqg_stats_update_io_merged(struct bfq_group *bfqg, unsigned int op)
+{
+ blkg_rwstat_add(&bfqg->stats.merged, op, 1);
+}
+
+void bfqg_stats_update_completion(struct bfq_group *bfqg, uint64_t start_time,
+ uint64_t io_start_time, unsigned int op)
+{
+ struct bfqg_stats *stats = &bfqg->stats;
+ unsigned long long now = sched_clock();
+
+ if (time_after64(now, io_start_time))
+ blkg_rwstat_add(&stats->service_time, op,
+ now - io_start_time);
+ if (time_after64(io_start_time, start_time))
+ blkg_rwstat_add(&stats->wait_time, op,
+ io_start_time - start_time);
+}
+
+/* @stats = 0 */
+static void bfqg_stats_reset(struct bfqg_stats *stats)
+{
+ /* queued stats shouldn't be cleared */
+ blkg_rwstat_reset(&stats->merged);
+ blkg_rwstat_reset(&stats->service_time);
+ blkg_rwstat_reset(&stats->wait_time);
+ blkg_stat_reset(&stats->time);
+ blkg_stat_reset(&stats->avg_queue_size_sum);
+ blkg_stat_reset(&stats->avg_queue_size_samples);
+ blkg_stat_reset(&stats->dequeue);
+ blkg_stat_reset(&stats->group_wait_time);
+ blkg_stat_reset(&stats->idle_time);
+ blkg_stat_reset(&stats->empty_time);
+}
+
+/* @to += @from */
+static void bfqg_stats_add_aux(struct bfqg_stats *to, struct bfqg_stats *from)
+{
+ if (!to || !from)
+ return;
+
+ /* queued stats shouldn't be cleared */
+ blkg_rwstat_add_aux(&to->merged, &from->merged);
+ blkg_rwstat_add_aux(&to->service_time, &from->service_time);
+ blkg_rwstat_add_aux(&to->wait_time, &from->wait_time);
+ blkg_stat_add_aux(&from->time, &from->time);
+ blkg_stat_add_aux(&to->avg_queue_size_sum, &from->avg_queue_size_sum);
+ blkg_stat_add_aux(&to->avg_queue_size_samples,
+ &from->avg_queue_size_samples);
+ blkg_stat_add_aux(&to->dequeue, &from->dequeue);
+ blkg_stat_add_aux(&to->group_wait_time, &from->group_wait_time);
+ blkg_stat_add_aux(&to->idle_time, &from->idle_time);
+ blkg_stat_add_aux(&to->empty_time, &from->empty_time);
+}
+
+/*
+ * Transfer @bfqg's stats to its parent's aux counts so that the ancestors'
+ * recursive stats can still account for the amount used by this bfqg after
+ * it's gone.
+ */
+static void bfqg_stats_xfer_dead(struct bfq_group *bfqg)
+{
+ struct bfq_group *parent;
+
+ if (!bfqg) /* root_group */
+ return;
+
+ parent = bfqg_parent(bfqg);
+
+ lockdep_assert_held(bfqg_to_blkg(bfqg)->q->queue_lock);
+
+ if (unlikely(!parent))
+ return;
+
+ bfqg_stats_add_aux(&parent->stats, &bfqg->stats);
+ bfqg_stats_reset(&bfqg->stats);
+}
+
+void bfq_init_entity(struct bfq_entity *entity, struct bfq_group *bfqg)
+{
+ struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity);
+
+ entity->weight = entity->new_weight;
+ entity->orig_weight = entity->new_weight;
+ if (bfqq) {
+ bfqq->ioprio = bfqq->new_ioprio;
+ bfqq->ioprio_class = bfqq->new_ioprio_class;
+ bfqg_get(bfqg);
+ }
+ entity->parent = bfqg->my_entity; /* NULL for root group */
+ entity->sched_data = &bfqg->sched_data;
+}
+
+static void bfqg_stats_exit(struct bfqg_stats *stats)
+{
+ blkg_rwstat_exit(&stats->merged);
+ blkg_rwstat_exit(&stats->service_time);
+ blkg_rwstat_exit(&stats->wait_time);
+ blkg_rwstat_exit(&stats->queued);
+ blkg_stat_exit(&stats->time);
+ blkg_stat_exit(&stats->avg_queue_size_sum);
+ blkg_stat_exit(&stats->avg_queue_size_samples);
+ blkg_stat_exit(&stats->dequeue);
+ blkg_stat_exit(&stats->group_wait_time);
+ blkg_stat_exit(&stats->idle_time);
+ blkg_stat_exit(&stats->empty_time);
+}
+
+static int bfqg_stats_init(struct bfqg_stats *stats, gfp_t gfp)
+{
+ if (blkg_rwstat_init(&stats->merged, gfp) ||
+ blkg_rwstat_init(&stats->service_time, gfp) ||
+ blkg_rwstat_init(&stats->wait_time, gfp) ||
+ blkg_rwstat_init(&stats->queued, gfp) ||
+ blkg_stat_init(&stats->time, gfp) ||
+ blkg_stat_init(&stats->avg_queue_size_sum, gfp) ||
+ blkg_stat_init(&stats->avg_queue_size_samples, gfp) ||
+ blkg_stat_init(&stats->dequeue, gfp) ||
+ blkg_stat_init(&stats->group_wait_time, gfp) ||
+ blkg_stat_init(&stats->idle_time, gfp) ||
+ blkg_stat_init(&stats->empty_time, gfp)) {
+ bfqg_stats_exit(stats);
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+static struct bfq_group_data *cpd_to_bfqgd(struct blkcg_policy_data *cpd)
+{
+ return cpd ? container_of(cpd, struct bfq_group_data, pd) : NULL;
+}
+
+static struct bfq_group_data *blkcg_to_bfqgd(struct blkcg *blkcg)
+{
+ return cpd_to_bfqgd(blkcg_to_cpd(blkcg, &blkcg_policy_bfq));
+}
+
+struct blkcg_policy_data *bfq_cpd_alloc(gfp_t gfp)
+{
+ struct bfq_group_data *bgd;
+
+ bgd = kzalloc(sizeof(*bgd), gfp);
+ if (!bgd)
+ return NULL;
+ return &bgd->pd;
+}
+
+void bfq_cpd_init(struct blkcg_policy_data *cpd)
+{
+ struct bfq_group_data *d = cpd_to_bfqgd(cpd);
+
+ d->weight = cgroup_subsys_on_dfl(io_cgrp_subsys) ?
+ CGROUP_WEIGHT_DFL : BFQ_WEIGHT_LEGACY_DFL;
+}
+
+void bfq_cpd_free(struct blkcg_policy_data *cpd)
+{
+ kfree(cpd_to_bfqgd(cpd));
+}
+
+struct blkg_policy_data *bfq_pd_alloc(gfp_t gfp, int node)
+{
+ struct bfq_group *bfqg;
+
+ bfqg = kzalloc_node(sizeof(*bfqg), gfp, node);
+ if (!bfqg)
+ return NULL;
+
+ if (bfqg_stats_init(&bfqg->stats, gfp)) {
+ kfree(bfqg);
+ return NULL;
+ }
+
+ return &bfqg->pd;
+}
+
+void bfq_pd_init(struct blkg_policy_data *pd)
+{
+ struct blkcg_gq *blkg = pd_to_blkg(pd);
+ struct bfq_group *bfqg = blkg_to_bfqg(blkg);
+ struct bfq_data *bfqd = blkg->q->elevator->elevator_data;
+ struct bfq_entity *entity = &bfqg->entity;
+ struct bfq_group_data *d = blkcg_to_bfqgd(blkg->blkcg);
+
+ entity->orig_weight = entity->weight = entity->new_weight = d->weight;
+ entity->my_sched_data = &bfqg->sched_data;
+ bfqg->my_entity = entity; /*
+ * the root_group's will be set to NULL
+ * in bfq_init_queue()
+ */
+ bfqg->bfqd = bfqd;
+ bfqg->active_entities = 0;
+ bfqg->rq_pos_tree = RB_ROOT;
+}
+
+void bfq_pd_free(struct blkg_policy_data *pd)
+{
+ struct bfq_group *bfqg = pd_to_bfqg(pd);
+
+ bfqg_stats_exit(&bfqg->stats);
+ return kfree(bfqg);
+}
+
+void bfq_pd_reset_stats(struct blkg_policy_data *pd)
+{
+ struct bfq_group *bfqg = pd_to_bfqg(pd);
+
+ bfqg_stats_reset(&bfqg->stats);
+}
+
+static void bfq_group_set_parent(struct bfq_group *bfqg,
+ struct bfq_group *parent)
+{
+ struct bfq_entity *entity;
+
+ entity = &bfqg->entity;
+ entity->parent = parent->my_entity;
+ entity->sched_data = &parent->sched_data;
+}
+
+static struct bfq_group *bfq_lookup_bfqg(struct bfq_data *bfqd,
+ struct blkcg *blkcg)
+{
+ struct blkcg_gq *blkg;
+
+ blkg = blkg_lookup(blkcg, bfqd->queue);
+ if (likely(blkg))
+ return blkg_to_bfqg(blkg);
+ return NULL;
+}
+
+struct bfq_group *bfq_find_set_group(struct bfq_data *bfqd,
+ struct blkcg *blkcg)
+{
+ struct bfq_group *bfqg, *parent;
+ struct bfq_entity *entity;
+
+ bfqg = bfq_lookup_bfqg(bfqd, blkcg);
+
+ if (unlikely(!bfqg))
+ return NULL;
+
+ /*
+ * Update chain of bfq_groups as we might be handling a leaf group
+ * which, along with some of its relatives, has not been hooked yet
+ * to the private hierarchy of BFQ.
+ */
+ entity = &bfqg->entity;
+ for_each_entity(entity) {
+ bfqg = container_of(entity, struct bfq_group, entity);
+ if (bfqg != bfqd->root_group) {
+ parent = bfqg_parent(bfqg);
+ if (!parent)
+ parent = bfqd->root_group;
+ bfq_group_set_parent(bfqg, parent);
+ }
+ }
+
+ return bfqg;
+}
+
+/**
+ * bfq_bfqq_move - migrate @bfqq to @bfqg.
+ * @bfqd: queue descriptor.
+ * @bfqq: the queue to move.
+ * @bfqg: the group to move to.
+ *
+ * Move @bfqq to @bfqg, deactivating it from its old group and reactivating
+ * it on the new one. Avoid putting the entity on the old group idle tree.
+ *
+ * Must be called under the queue lock; the cgroup owning @bfqg must
+ * not disappear (by now this just means that we are called under
+ * rcu_read_lock()).
+ */
+void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ struct bfq_group *bfqg)
+{
+ struct bfq_entity *entity = &bfqq->entity;
+
+ /* If bfqq is empty, then bfq_bfqq_expire also invokes
+ * bfq_del_bfqq_busy, thereby removing bfqq and its entity
+ * from data structures related to current group. Otherwise we
+ * need to remove bfqq explicitly with bfq_deactivate_bfqq, as
+ * we do below.
+ */
+ if (bfqq == bfqd->in_service_queue)
+ bfq_bfqq_expire(bfqd, bfqd->in_service_queue,
+ false, BFQQE_PREEMPTED);
+
+ if (bfq_bfqq_busy(bfqq))
+ bfq_deactivate_bfqq(bfqd, bfqq, false, false);
+ else if (entity->on_st)
+ bfq_put_idle_entity(bfq_entity_service_tree(entity), entity);
+ bfqg_put(bfqq_group(bfqq));
+
+ /*
+ * Here we use a reference to bfqg. We don't need a refcounter
+ * as the cgroup reference will not be dropped, so that its
+ * destroy() callback will not be invoked.
+ */
+ entity->parent = bfqg->my_entity;
+ entity->sched_data = &bfqg->sched_data;
+ bfqg_get(bfqg);
+
+ if (bfq_bfqq_busy(bfqq)) {
+ bfq_pos_tree_add_move(bfqd, bfqq);
+ bfq_activate_bfqq(bfqd, bfqq);
+ }
+
+ if (!bfqd->in_service_queue && !bfqd->rq_in_driver)
+ bfq_schedule_dispatch(bfqd);
+}
+
+/**
+ * __bfq_bic_change_cgroup - move @bic to @cgroup.
+ * @bfqd: the queue descriptor.
+ * @bic: the bic to move.
+ * @blkcg: the blk-cgroup to move to.
+ *
+ * Move bic to blkcg, assuming that bfqd->queue is locked; the caller
+ * has to make sure that the reference to cgroup is valid across the call.
+ *
+ * NOTE: an alternative approach might have been to store the current
+ * cgroup in bfqq and getting a reference to it, reducing the lookup
+ * time here, at the price of slightly more complex code.
+ */
+static struct bfq_group *__bfq_bic_change_cgroup(struct bfq_data *bfqd,
+ struct bfq_io_cq *bic,
+ struct blkcg *blkcg)
+{
+ struct bfq_queue *async_bfqq = bic_to_bfqq(bic, 0);
+ struct bfq_queue *sync_bfqq = bic_to_bfqq(bic, 1);
+ struct bfq_group *bfqg;
+ struct bfq_entity *entity;
+
+ bfqg = bfq_find_set_group(bfqd, blkcg);
+
+ if (unlikely(!bfqg))
+ bfqg = bfqd->root_group;
+
+ if (async_bfqq) {
+ entity = &async_bfqq->entity;
+
+ if (entity->sched_data != &bfqg->sched_data) {
+ bic_set_bfqq(bic, NULL, 0);
+ bfq_log_bfqq(bfqd, async_bfqq,
+ "bic_change_group: %p %d",
+ async_bfqq, async_bfqq->ref);
+ bfq_put_queue(async_bfqq);
+ }
+ }
+
+ if (sync_bfqq) {
+ entity = &sync_bfqq->entity;
+ if (entity->sched_data != &bfqg->sched_data)
+ bfq_bfqq_move(bfqd, sync_bfqq, bfqg);
+ }
+
+ return bfqg;
+}
+
+void bfq_bic_update_cgroup(struct bfq_io_cq *bic, struct bio *bio)
+{
+ struct bfq_data *bfqd = bic_to_bfqd(bic);
+ struct bfq_group *bfqg = NULL;
+ uint64_t serial_nr;
+
+ rcu_read_lock();
+ serial_nr = bio_blkcg(bio)->css.serial_nr;
+
+ /*
+ * Check whether blkcg has changed. The condition may trigger
+ * spuriously on a newly created cic but there's no harm.
+ */
+ if (unlikely(!bfqd) || likely(bic->blkcg_serial_nr == serial_nr))
+ goto out;
+
+ bfqg = __bfq_bic_change_cgroup(bfqd, bic, bio_blkcg(bio));
+ bic->blkcg_serial_nr = serial_nr;
+out:
+ rcu_read_unlock();
+}
+
+/**
+ * bfq_flush_idle_tree - deactivate any entity on the idle tree of @st.
+ * @st: the service tree being flushed.
+ */
+static void bfq_flush_idle_tree(struct bfq_service_tree *st)
+{
+ struct bfq_entity *entity = st->first_idle;
+
+ for (; entity ; entity = st->first_idle)
+ __bfq_deactivate_entity(entity, false);
+}
+
+/**
+ * bfq_reparent_leaf_entity - move leaf entity to the root_group.
+ * @bfqd: the device data structure with the root group.
+ * @entity: the entity to move.
+ */
+static void bfq_reparent_leaf_entity(struct bfq_data *bfqd,
+ struct bfq_entity *entity)
+{
+ struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity);
+
+ bfq_bfqq_move(bfqd, bfqq, bfqd->root_group);
+}
+
+/**
+ * bfq_reparent_active_entities - move to the root group all active
+ * entities.
+ * @bfqd: the device data structure with the root group.
+ * @bfqg: the group to move from.
+ * @st: the service tree with the entities.
+ *
+ * Needs queue_lock to be taken and reference to be valid over the call.
+ */
+static void bfq_reparent_active_entities(struct bfq_data *bfqd,
+ struct bfq_group *bfqg,
+ struct bfq_service_tree *st)
+{
+ struct rb_root *active = &st->active;
+ struct bfq_entity *entity = NULL;
+
+ if (!RB_EMPTY_ROOT(&st->active))
+ entity = bfq_entity_of(rb_first(active));
+
+ for (; entity ; entity = bfq_entity_of(rb_first(active)))
+ bfq_reparent_leaf_entity(bfqd, entity);
+
+ if (bfqg->sched_data.in_service_entity)
+ bfq_reparent_leaf_entity(bfqd,
+ bfqg->sched_data.in_service_entity);
+}
+
+/**
+ * bfq_pd_offline - deactivate the entity associated with @pd,
+ * and reparent its children entities.
+ * @pd: descriptor of the policy going offline.
+ *
+ * blkio already grabs the queue_lock for us, so no need to use
+ * RCU-based magic
+ */
+void bfq_pd_offline(struct blkg_policy_data *pd)
+{
+ struct bfq_service_tree *st;
+ struct bfq_group *bfqg = pd_to_bfqg(pd);
+ struct bfq_data *bfqd = bfqg->bfqd;
+ struct bfq_entity *entity = bfqg->my_entity;
+ unsigned long flags;
+ int i;
+
+ if (!entity) /* root group */
+ return;
+
+ spin_lock_irqsave(&bfqd->lock, flags);
+ /*
+ * Empty all service_trees belonging to this group before
+ * deactivating the group itself.
+ */
+ for (i = 0; i < BFQ_IOPRIO_CLASSES; i++) {
+ st = bfqg->sched_data.service_tree + i;
+
+ /*
+ * The idle tree may still contain bfq_queues belonging
+ * to exited task because they never migrated to a different
+ * cgroup from the one being destroyed now. No one else
+ * can access them so it's safe to act without any lock.
+ */
+ bfq_flush_idle_tree(st);
+
+ /*
+ * It may happen that some queues are still active
+ * (busy) upon group destruction (if the corresponding
+ * processes have been forced to terminate). We move
+ * all the leaf entities corresponding to these queues
+ * to the root_group.
+ * Also, it may happen that the group has an entity
+ * in service, which is disconnected from the active
+ * tree: it must be moved, too.
+ * There is no need to put the sync queues, as the
+ * scheduler has taken no reference.
+ */
+ bfq_reparent_active_entities(bfqd, bfqg, st);
+ }
+
+ __bfq_deactivate_entity(entity, false);
+ bfq_put_async_queues(bfqd, bfqg);
+
+ spin_unlock_irqrestore(&bfqd->lock, flags);
+ /*
+ * @blkg is going offline and will be ignored by
+ * blkg_[rw]stat_recursive_sum(). Transfer stats to the parent so
+ * that they don't get lost. If IOs complete after this point, the
+ * stats for them will be lost. Oh well...
+ */
+ bfqg_stats_xfer_dead(bfqg);
+}
+
+void bfq_end_wr_async(struct bfq_data *bfqd)
+{
+ struct blkcg_gq *blkg;
+
+ list_for_each_entry(blkg, &bfqd->queue->blkg_list, q_node) {
+ struct bfq_group *bfqg = blkg_to_bfqg(blkg);
+
+ bfq_end_wr_async_queues(bfqd, bfqg);
+ }
+ bfq_end_wr_async_queues(bfqd, bfqd->root_group);
+}
+
+static int bfq_io_show_weight(struct seq_file *sf, void *v)
+{
+ struct blkcg *blkcg = css_to_blkcg(seq_css(sf));
+ struct bfq_group_data *bfqgd = blkcg_to_bfqgd(blkcg);
+ unsigned int val = 0;
+
+ if (bfqgd)
+ val = bfqgd->weight;
+
+ seq_printf(sf, "%u\n", val);
+
+ return 0;
+}
+
+static int bfq_io_set_weight_legacy(struct cgroup_subsys_state *css,
+ struct cftype *cftype,
+ u64 val)
+{
+ struct blkcg *blkcg = css_to_blkcg(css);
+ struct bfq_group_data *bfqgd = blkcg_to_bfqgd(blkcg);
+ struct blkcg_gq *blkg;
+ int ret = -ERANGE;
+
+ if (val < BFQ_MIN_WEIGHT || val > BFQ_MAX_WEIGHT)
+ return ret;
+
+ ret = 0;
+ spin_lock_irq(&blkcg->lock);
+ bfqgd->weight = (unsigned short)val;
+ hlist_for_each_entry(blkg, &blkcg->blkg_list, blkcg_node) {
+ struct bfq_group *bfqg = blkg_to_bfqg(blkg);
+
+ if (!bfqg)
+ continue;
+ /*
+ * Setting the prio_changed flag of the entity
+ * to 1 with new_weight == weight would re-set
+ * the value of the weight to its ioprio mapping.
+ * Set the flag only if necessary.
+ */
+ if ((unsigned short)val != bfqg->entity.new_weight) {
+ bfqg->entity.new_weight = (unsigned short)val;
+ /*
+ * Make sure that the above new value has been
+ * stored in bfqg->entity.new_weight before
+ * setting the prio_changed flag. In fact,
+ * this flag may be read asynchronously (in
+ * critical sections protected by a different
+ * lock than that held here), and finding this
+ * flag set may cause the execution of the code
+ * for updating parameters whose value may
+ * depend also on bfqg->entity.new_weight (in
+ * __bfq_entity_update_weight_prio).
+ * This barrier makes sure that the new value
+ * of bfqg->entity.new_weight is correctly
+ * seen in that code.
+ */
+ smp_wmb();
+ bfqg->entity.prio_changed = 1;
+ }
+ }
+ spin_unlock_irq(&blkcg->lock);
+
+ return ret;
+}
+
+static ssize_t bfq_io_set_weight(struct kernfs_open_file *of,
+ char *buf, size_t nbytes,
+ loff_t off)
+{
+ u64 weight;
+ /* First unsigned long found in the file is used */
+ int ret = kstrtoull(strim(buf), 0, &weight);
+
+ if (ret)
+ return ret;
+
+ return bfq_io_set_weight_legacy(of_css(of), NULL, weight);
+}
+
+static int bfqg_print_stat(struct seq_file *sf, void *v)
+{
+ blkcg_print_blkgs(sf, css_to_blkcg(seq_css(sf)), blkg_prfill_stat,
+ &blkcg_policy_bfq, seq_cft(sf)->private, false);
+ return 0;
+}
+
+static int bfqg_print_rwstat(struct seq_file *sf, void *v)
+{
+ blkcg_print_blkgs(sf, css_to_blkcg(seq_css(sf)), blkg_prfill_rwstat,
+ &blkcg_policy_bfq, seq_cft(sf)->private, true);
+ return 0;
+}
+
+static u64 bfqg_prfill_stat_recursive(struct seq_file *sf,
+ struct blkg_policy_data *pd, int off)
+{
+ u64 sum = blkg_stat_recursive_sum(pd_to_blkg(pd),
+ &blkcg_policy_bfq, off);
+ return __blkg_prfill_u64(sf, pd, sum);
+}
+
+static u64 bfqg_prfill_rwstat_recursive(struct seq_file *sf,
+ struct blkg_policy_data *pd, int off)
+{
+ struct blkg_rwstat sum = blkg_rwstat_recursive_sum(pd_to_blkg(pd),
+ &blkcg_policy_bfq,
+ off);
+ return __blkg_prfill_rwstat(sf, pd, &sum);
+}
+
+static int bfqg_print_stat_recursive(struct seq_file *sf, void *v)
+{
+ blkcg_print_blkgs(sf, css_to_blkcg(seq_css(sf)),
+ bfqg_prfill_stat_recursive, &blkcg_policy_bfq,
+ seq_cft(sf)->private, false);
+ return 0;
+}
+
+static int bfqg_print_rwstat_recursive(struct seq_file *sf, void *v)
+{
+ blkcg_print_blkgs(sf, css_to_blkcg(seq_css(sf)),
+ bfqg_prfill_rwstat_recursive, &blkcg_policy_bfq,
+ seq_cft(sf)->private, true);
+ return 0;
+}
+
+static u64 bfqg_prfill_sectors(struct seq_file *sf, struct blkg_policy_data *pd,
+ int off)
+{
+ u64 sum = blkg_rwstat_total(&pd->blkg->stat_bytes);
+
+ return __blkg_prfill_u64(sf, pd, sum >> 9);
+}
+
+static int bfqg_print_stat_sectors(struct seq_file *sf, void *v)
+{
+ blkcg_print_blkgs(sf, css_to_blkcg(seq_css(sf)),
+ bfqg_prfill_sectors, &blkcg_policy_bfq, 0, false);
+ return 0;
+}
+
+static u64 bfqg_prfill_sectors_recursive(struct seq_file *sf,
+ struct blkg_policy_data *pd, int off)
+{
+ struct blkg_rwstat tmp = blkg_rwstat_recursive_sum(pd->blkg, NULL,
+ offsetof(struct blkcg_gq, stat_bytes));
+ u64 sum = atomic64_read(&tmp.aux_cnt[BLKG_RWSTAT_READ]) +
+ atomic64_read(&tmp.aux_cnt[BLKG_RWSTAT_WRITE]);
+
+ return __blkg_prfill_u64(sf, pd, sum >> 9);
+}
+
+static int bfqg_print_stat_sectors_recursive(struct seq_file *sf, void *v)
+{
+ blkcg_print_blkgs(sf, css_to_blkcg(seq_css(sf)),
+ bfqg_prfill_sectors_recursive, &blkcg_policy_bfq, 0,
+ false);
+ return 0;
+}
+
+static u64 bfqg_prfill_avg_queue_size(struct seq_file *sf,
+ struct blkg_policy_data *pd, int off)
+{
+ struct bfq_group *bfqg = pd_to_bfqg(pd);
+ u64 samples = blkg_stat_read(&bfqg->stats.avg_queue_size_samples);
+ u64 v = 0;
+
+ if (samples) {
+ v = blkg_stat_read(&bfqg->stats.avg_queue_size_sum);
+ v = div64_u64(v, samples);
+ }
+ __blkg_prfill_u64(sf, pd, v);
+ return 0;
+}
+
+/* print avg_queue_size */
+static int bfqg_print_avg_queue_size(struct seq_file *sf, void *v)
+{
+ blkcg_print_blkgs(sf, css_to_blkcg(seq_css(sf)),
+ bfqg_prfill_avg_queue_size, &blkcg_policy_bfq,
+ 0, false);
+ return 0;
+}
+
+struct bfq_group *bfq_create_group_hierarchy(struct bfq_data *bfqd, int node)
+{
+ int ret;
+
+ ret = blkcg_activate_policy(bfqd->queue, &blkcg_policy_bfq);
+ if (ret)
+ return NULL;
+
+ return blkg_to_bfqg(bfqd->queue->root_blkg);
+}
+
+struct blkcg_policy blkcg_policy_bfq = {
+ .dfl_cftypes = bfq_blkg_files,
+ .legacy_cftypes = bfq_blkcg_legacy_files,
+
+ .cpd_alloc_fn = bfq_cpd_alloc,
+ .cpd_init_fn = bfq_cpd_init,
+ .cpd_bind_fn = bfq_cpd_init,
+ .cpd_free_fn = bfq_cpd_free,
+
+ .pd_alloc_fn = bfq_pd_alloc,
+ .pd_init_fn = bfq_pd_init,
+ .pd_offline_fn = bfq_pd_offline,
+ .pd_free_fn = bfq_pd_free,
+ .pd_reset_stats_fn = bfq_pd_reset_stats,
+};
+
+struct cftype bfq_blkcg_legacy_files[] = {
+ {
+ .name = "bfq.weight",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .seq_show = bfq_io_show_weight,
+ .write_u64 = bfq_io_set_weight_legacy,
+ },
+
+ /* statistics, covers only the tasks in the bfqg */
+ {
+ .name = "bfq.time",
+ .private = offsetof(struct bfq_group, stats.time),
+ .seq_show = bfqg_print_stat,
+ },
+ {
+ .name = "bfq.sectors",
+ .seq_show = bfqg_print_stat_sectors,
+ },
+ {
+ .name = "bfq.io_service_bytes",
+ .private = (unsigned long)&blkcg_policy_bfq,
+ .seq_show = blkg_print_stat_bytes,
+ },
+ {
+ .name = "bfq.io_serviced",
+ .private = (unsigned long)&blkcg_policy_bfq,
+ .seq_show = blkg_print_stat_ios,
+ },
+ {
+ .name = "bfq.io_service_time",
+ .private = offsetof(struct bfq_group, stats.service_time),
+ .seq_show = bfqg_print_rwstat,
+ },
+ {
+ .name = "bfq.io_wait_time",
+ .private = offsetof(struct bfq_group, stats.wait_time),
+ .seq_show = bfqg_print_rwstat,
+ },
+ {
+ .name = "bfq.io_merged",
+ .private = offsetof(struct bfq_group, stats.merged),
+ .seq_show = bfqg_print_rwstat,
+ },
+ {
+ .name = "bfq.io_queued",
+ .private = offsetof(struct bfq_group, stats.queued),
+ .seq_show = bfqg_print_rwstat,
+ },
+
+ /* the same statictics which cover the bfqg and its descendants */
+ {
+ .name = "bfq.time_recursive",
+ .private = offsetof(struct bfq_group, stats.time),
+ .seq_show = bfqg_print_stat_recursive,
+ },
+ {
+ .name = "bfq.sectors_recursive",
+ .seq_show = bfqg_print_stat_sectors_recursive,
+ },
+ {
+ .name = "bfq.io_service_bytes_recursive",
+ .private = (unsigned long)&blkcg_policy_bfq,
+ .seq_show = blkg_print_stat_bytes_recursive,
+ },
+ {
+ .name = "bfq.io_serviced_recursive",
+ .private = (unsigned long)&blkcg_policy_bfq,
+ .seq_show = blkg_print_stat_ios_recursive,
+ },
+ {
+ .name = "bfq.io_service_time_recursive",
+ .private = offsetof(struct bfq_group, stats.service_time),
+ .seq_show = bfqg_print_rwstat_recursive,
+ },
+ {
+ .name = "bfq.io_wait_time_recursive",
+ .private = offsetof(struct bfq_group, stats.wait_time),
+ .seq_show = bfqg_print_rwstat_recursive,
+ },
+ {
+ .name = "bfq.io_merged_recursive",
+ .private = offsetof(struct bfq_group, stats.merged),
+ .seq_show = bfqg_print_rwstat_recursive,
+ },
+ {
+ .name = "bfq.io_queued_recursive",
+ .private = offsetof(struct bfq_group, stats.queued),
+ .seq_show = bfqg_print_rwstat_recursive,
+ },
+ {
+ .name = "bfq.avg_queue_size",
+ .seq_show = bfqg_print_avg_queue_size,
+ },
+ {
+ .name = "bfq.group_wait_time",
+ .private = offsetof(struct bfq_group, stats.group_wait_time),
+ .seq_show = bfqg_print_stat,
+ },
+ {
+ .name = "bfq.idle_time",
+ .private = offsetof(struct bfq_group, stats.idle_time),
+ .seq_show = bfqg_print_stat,
+ },
+ {
+ .name = "bfq.empty_time",
+ .private = offsetof(struct bfq_group, stats.empty_time),
+ .seq_show = bfqg_print_stat,
+ },
+ {
+ .name = "bfq.dequeue",
+ .private = offsetof(struct bfq_group, stats.dequeue),
+ .seq_show = bfqg_print_stat,
+ },
+ { } /* terminate */
+};
+
+struct cftype bfq_blkg_files[] = {
+ {
+ .name = "bfq.weight",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .seq_show = bfq_io_show_weight,
+ .write = bfq_io_set_weight,
+ },
+ {} /* terminate */
+};
+
+#else /* CONFIG_BFQ_GROUP_IOSCHED */
+
+void bfqg_stats_update_io_add(struct bfq_group *bfqg, struct bfq_queue *bfqq,
+ unsigned int op) { }
+void bfqg_stats_update_io_remove(struct bfq_group *bfqg, unsigned int op) { }
+void bfqg_stats_update_io_merged(struct bfq_group *bfqg, unsigned int op) { }
+void bfqg_stats_update_completion(struct bfq_group *bfqg, uint64_t start_time,
+ uint64_t io_start_time, unsigned int op) { }
+void bfqg_stats_update_dequeue(struct bfq_group *bfqg) { }
+void bfqg_stats_set_start_empty_time(struct bfq_group *bfqg) { }
+void bfqg_stats_update_idle_time(struct bfq_group *bfqg) { }
+void bfqg_stats_set_start_idle_time(struct bfq_group *bfqg) { }
+void bfqg_stats_update_avg_queue_size(struct bfq_group *bfqg) { }
+
+void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ struct bfq_group *bfqg) {}
+
+void bfq_init_entity(struct bfq_entity *entity, struct bfq_group *bfqg)
+{
+ struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity);
+
+ entity->weight = entity->new_weight;
+ entity->orig_weight = entity->new_weight;
+ if (bfqq) {
+ bfqq->ioprio = bfqq->new_ioprio;
+ bfqq->ioprio_class = bfqq->new_ioprio_class;
+ }
+ entity->sched_data = &bfqg->sched_data;
+}
+
+void bfq_bic_update_cgroup(struct bfq_io_cq *bic, struct bio *bio) {}
+
+void bfq_end_wr_async(struct bfq_data *bfqd)
+{
+ bfq_end_wr_async_queues(bfqd, bfqd->root_group);
+}
+
+struct bfq_group *bfq_find_set_group(struct bfq_data *bfqd, struct blkcg *blkcg)
+{
+ return bfqd->root_group;
+}
+
+struct bfq_group *bfqq_group(struct bfq_queue *bfqq)
+{
+ return bfqq->bfqd->root_group;
+}
+
+struct bfq_group *bfq_create_group_hierarchy(struct bfq_data *bfqd, int node)
+{
+ struct bfq_group *bfqg;
+ int i;
+
+ bfqg = kmalloc_node(sizeof(*bfqg), GFP_KERNEL | __GFP_ZERO, node);
+ if (!bfqg)
+ return NULL;
+
+ for (i = 0; i < BFQ_IOPRIO_CLASSES; i++)
+ bfqg->sched_data.service_tree[i] = BFQ_SERVICE_TREE_INIT;
+
+ return bfqg;
+}
+#endif /* CONFIG_BFQ_GROUP_IOSCHED */
diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
new file mode 100644
index 0000000..bd8499e
--- /dev/null
+++ b/block/bfq-iosched.c
@@ -0,0 +1,5047 @@
+/*
+ * Budget Fair Queueing (BFQ) I/O scheduler.
+ *
+ * Based on ideas and code from CFQ:
+ * Copyright (C) 2003 Jens Axboe <axboe@kernel.dk>
+ *
+ * Copyright (C) 2008 Fabio Checconi <fabio@gandalf.sssup.it>
+ * Paolo Valente <paolo.valente@unimore.it>
+ *
+ * Copyright (C) 2010 Paolo Valente <paolo.valente@unimore.it>
+ * Arianna Avanzini <avanzini@google.com>
+ *
+ * Copyright (C) 2017 Paolo Valente <paolo.valente@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * BFQ is a proportional-share I/O scheduler, with some extra
+ * low-latency capabilities. BFQ also supports full hierarchical
+ * scheduling through cgroups. Next paragraphs provide an introduction
+ * on BFQ inner workings. Details on BFQ benefits, usage and
+ * limitations can be found in Documentation/block/bfq-iosched.txt.
+ *
+ * BFQ is a proportional-share storage-I/O scheduling algorithm based
+ * on the slice-by-slice service scheme of CFQ. But BFQ assigns
+ * budgets, measured in number of sectors, to processes instead of
+ * time slices. The device is not granted to the in-service process
+ * for a given time slice, but until it has exhausted its assigned
+ * budget. This change from the time to the service domain enables BFQ
+ * to distribute the device throughput among processes as desired,
+ * without any distortion due to throughput fluctuations, or to device
+ * internal queueing. BFQ uses an ad hoc internal scheduler, called
+ * B-WF2Q+, to schedule processes according to their budgets. More
+ * precisely, BFQ schedules queues associated with processes. Each
+ * process/queue is assigned a user-configurable weight, and B-WF2Q+
+ * guarantees that each queue receives a fraction of the throughput
+ * proportional to its weight. Thanks to the accurate policy of
+ * B-WF2Q+, BFQ can afford to assign high budgets to I/O-bound
+ * processes issuing sequential requests (to boost the throughput),
+ * and yet guarantee a low latency to interactive and soft real-time
+ * applications.
+ *
+ * In particular, to provide these low-latency guarantees, BFQ
+ * explicitly privileges the I/O of two classes of time-sensitive
+ * applications: interactive and soft real-time. This feature enables
+ * BFQ to provide applications in these classes with a very low
+ * latency. Finally, BFQ also features additional heuristics for
+ * preserving both a low latency and a high throughput on NCQ-capable,
+ * rotational or flash-based devices, and to get the job done quickly
+ * for applications consisting in many I/O-bound processes.
+ *
+ * BFQ is described in [1], where also a reference to the initial, more
+ * theoretical paper on BFQ can be found. The interested reader can find
+ * in the latter paper full details on the main algorithm, as well as
+ * formulas of the guarantees and formal proofs of all the properties.
+ * With respect to the version of BFQ presented in these papers, this
+ * implementation adds a few more heuristics, such as the one that
+ * guarantees a low latency to soft real-time applications, and a
+ * hierarchical extension based on H-WF2Q+.
+ *
+ * B-WF2Q+ is based on WF2Q+, which is described in [2], together with
+ * H-WF2Q+, while the augmented tree used here to implement B-WF2Q+
+ * with O(log N) complexity derives from the one introduced with EEVDF
+ * in [3].
+ *
+ * [1] P. Valente, A. Avanzini, "Evolution of the BFQ Storage I/O
+ * Scheduler", Proceedings of the First Workshop on Mobile System
+ * Technologies (MST-2015), May 2015.
+ * http://algogroup.unimore.it/people/paolo/disk_sched/mst-2015.pdf
+ *
+ * [2] Jon C.R. Bennett and H. Zhang, "Hierarchical Packet Fair Queueing
+ * Algorithms", IEEE/ACM Transactions on Networking, 5(5):675-689,
+ * Oct 1997.
+ *
+ * http://www.cs.cmu.edu/~hzhang/papers/TON-97-Oct.ps.gz
+ *
+ * [3] I. Stoica and H. Abdel-Wahab, "Earliest Eligible Virtual Deadline
+ * First: A Flexible and Accurate Mechanism for Proportional Share
+ * Resource Allocation", technical report.
+ *
+ * http://www.cs.berkeley.edu/~istoica/papers/eevdf-tr-95.pdf
+ */
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/blkdev.h>
+#include <linux/cgroup.h>
+#include <linux/elevator.h>
+#include <linux/ktime.h>
+#include <linux/rbtree.h>
+#include <linux/ioprio.h>
+#include <linux/sbitmap.h>
+#include <linux/delay.h>
+
+#include "blk.h"
+#include "blk-mq.h"
+#include "blk-mq-tag.h"
+#include "blk-mq-sched.h"
+#include "bfq-iosched.h"
+
+#define BFQ_BFQQ_FNS(name) \
+void bfq_mark_bfqq_##name(struct bfq_queue *bfqq) \
+{ \
+ __set_bit(BFQQF_##name, &(bfqq)->flags); \
+} \
+void bfq_clear_bfqq_##name(struct bfq_queue *bfqq) \
+{ \
+ __clear_bit(BFQQF_##name, &(bfqq)->flags); \
+} \
+int bfq_bfqq_##name(const struct bfq_queue *bfqq) \
+{ \
+ return test_bit(BFQQF_##name, &(bfqq)->flags); \
+}
+
+BFQ_BFQQ_FNS(just_created);
+BFQ_BFQQ_FNS(busy);
+BFQ_BFQQ_FNS(wait_request);
+BFQ_BFQQ_FNS(non_blocking_wait_rq);
+BFQ_BFQQ_FNS(fifo_expire);
+BFQ_BFQQ_FNS(idle_window);
+BFQ_BFQQ_FNS(sync);
+BFQ_BFQQ_FNS(IO_bound);
+BFQ_BFQQ_FNS(in_large_burst);
+BFQ_BFQQ_FNS(coop);
+BFQ_BFQQ_FNS(split_coop);
+BFQ_BFQQ_FNS(softrt_update);
+#undef BFQ_BFQQ_FNS \
+
+/* Expiration time of sync (0) and async (1) requests, in ns. */
+static const u64 bfq_fifo_expire[2] = { NSEC_PER_SEC / 4, NSEC_PER_SEC / 8 };
+
+/* Maximum backwards seek (magic number lifted from CFQ), in KiB. */
+static const int bfq_back_max = 16 * 1024;
+
+/* Penalty of a backwards seek, in number of sectors. */
+static const int bfq_back_penalty = 2;
+
+/* Idling period duration, in ns. */
+static u64 bfq_slice_idle = NSEC_PER_SEC / 125;
+
+/* Minimum number of assigned budgets for which stats are safe to compute. */
+static const int bfq_stats_min_budgets = 194;
+
+/* Default maximum budget values, in sectors and number of requests. */
+static const int bfq_default_max_budget = 16 * 1024;
+
+/*
+ * Async to sync throughput distribution is controlled as follows:
+ * when an async request is served, the entity is charged the number
+ * of sectors of the request, multiplied by the factor below
+ */
+static const int bfq_async_charge_factor = 10;
+
+/* Default timeout values, in jiffies, approximating CFQ defaults. */
+const int bfq_timeout = HZ / 8;
+
+static struct kmem_cache *bfq_pool;
+
+/* Below this threshold (in ns), we consider thinktime immediate. */
+#define BFQ_MIN_TT (2 * NSEC_PER_MSEC)
+
+/* hw_tag detection: parallel requests threshold and min samples needed. */
+#define BFQ_HW_QUEUE_THRESHOLD 4
+#define BFQ_HW_QUEUE_SAMPLES 32
+
+#define BFQQ_SEEK_THR (sector_t)(8 * 100)
+#define BFQQ_SECT_THR_NONROT (sector_t)(2 * 32)
+#define BFQQ_CLOSE_THR (sector_t)(8 * 1024)
+#define BFQQ_SEEKY(bfqq) (hweight32(bfqq->seek_history) > 32/8)
+
+/* Min number of samples required to perform peak-rate update */
+#define BFQ_RATE_MIN_SAMPLES 32
+/* Min observation time interval required to perform a peak-rate update (ns) */
+#define BFQ_RATE_MIN_INTERVAL (300*NSEC_PER_MSEC)
+/* Target observation time interval for a peak-rate update (ns) */
+#define BFQ_RATE_REF_INTERVAL NSEC_PER_SEC
+
+/* Shift used for peak rate fixed precision calculations. */
+#define BFQ_RATE_SHIFT 16
+
+/*
+ * By default, BFQ computes the duration of the weight raising for
+ * interactive applications automatically, using the following formula:
+ * duration = (R / r) * T, where r is the peak rate of the device, and
+ * R and T are two reference parameters.
+ * In particular, R is the peak rate of the reference device (see below),
+ * and T is a reference time: given the systems that are likely to be
+ * installed on the reference device according to its speed class, T is
+ * about the maximum time needed, under BFQ and while reading two files in
+ * parallel, to load typical large applications on these systems.
+ * In practice, the slower/faster the device at hand is, the more/less it
+ * takes to load applications with respect to the reference device.
+ * Accordingly, the longer/shorter BFQ grants weight raising to interactive
+ * applications.
+ *
+ * BFQ uses four different reference pairs (R, T), depending on:
+ * . whether the device is rotational or non-rotational;
+ * . whether the device is slow, such as old or portable HDDs, as well as
+ * SD cards, or fast, such as newer HDDs and SSDs.
+ *
+ * The device's speed class is dynamically (re)detected in
+ * bfq_update_peak_rate() every time the estimated peak rate is updated.
+ *
+ * In the following definitions, R_slow[0]/R_fast[0] and
+ * T_slow[0]/T_fast[0] are the reference values for a slow/fast
+ * rotational device, whereas R_slow[1]/R_fast[1] and
+ * T_slow[1]/T_fast[1] are the reference values for a slow/fast
+ * non-rotational device. Finally, device_speed_thresh are the
+ * thresholds used to switch between speed classes. The reference
+ * rates are not the actual peak rates of the devices used as a
+ * reference, but slightly lower values. The reason for using these
+ * slightly lower values is that the peak-rate estimator tends to
+ * yield slightly lower values than the actual peak rate (it can yield
+ * the actual peak rate only if there is only one process doing I/O,
+ * and the process does sequential I/O).
+ *
+ * Both the reference peak rates and the thresholds are measured in
+ * sectors/usec, left-shifted by BFQ_RATE_SHIFT.
+ */
+static int R_slow[2] = {1000, 10700};
+static int R_fast[2] = {14000, 33000};
+/*
+ * To improve readability, a conversion function is used to initialize the
+ * following arrays, which entails that they can be initialized only in a
+ * function.
+ */
+static int T_slow[2];
+static int T_fast[2];
+static int device_speed_thresh[2];
+
+#define RQ_BIC(rq) ((struct bfq_io_cq *) (rq)->elv.priv[0])
+#define RQ_BFQQ(rq) ((rq)->elv.priv[1])
+
+struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync)
+{
+ return bic->bfqq[is_sync];
+}
+
+void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync)
+{
+ bic->bfqq[is_sync] = bfqq;
+}
+
+struct bfq_data *bic_to_bfqd(struct bfq_io_cq *bic)
+{
+ return bic->icq.q->elevator->elevator_data;
+}
+
+/**
+ * icq_to_bic - convert iocontext queue structure to bfq_io_cq.
+ * @icq: the iocontext queue.
+ */
+static struct bfq_io_cq *icq_to_bic(struct io_cq *icq)
+{
+ /* bic->icq is the first member, %NULL will convert to %NULL */
+ return container_of(icq, struct bfq_io_cq, icq);
+}
+
+/**
+ * bfq_bic_lookup - search into @ioc a bic associated to @bfqd.
+ * @bfqd: the lookup key.
+ * @ioc: the io_context of the process doing I/O.
+ * @q: the request queue.
+ */
+static struct bfq_io_cq *bfq_bic_lookup(struct bfq_data *bfqd,
+ struct io_context *ioc,
+ struct request_queue *q)
+{
+ if (ioc) {
+ unsigned long flags;
+ struct bfq_io_cq *icq;
+
+ spin_lock_irqsave(q->queue_lock, flags);
+ icq = icq_to_bic(ioc_lookup_icq(ioc, q));
+ spin_unlock_irqrestore(q->queue_lock, flags);
+
+ return icq;
+ }
+
+ return NULL;
+}
+
+/*
+ * Scheduler run of queue, if there are requests pending and no one in the
+ * driver that will restart queueing.
+ */
+void bfq_schedule_dispatch(struct bfq_data *bfqd)
+{
+ if (bfqd->queued != 0) {
+ bfq_log(bfqd, "schedule dispatch");
+ blk_mq_run_hw_queues(bfqd->queue, true);
+ }
+}
+
+#define bfq_class_idle(bfqq) ((bfqq)->ioprio_class == IOPRIO_CLASS_IDLE)
+#define bfq_class_rt(bfqq) ((bfqq)->ioprio_class == IOPRIO_CLASS_RT)
+
+#define bfq_sample_valid(samples) ((samples) > 80)
+
+/*
+ * Lifted from AS - choose which of rq1 and rq2 that is best served now.
+ * We choose the request that is closesr to the head right now. Distance
+ * behind the head is penalized and only allowed to a certain extent.
+ */
+static struct request *bfq_choose_req(struct bfq_data *bfqd,
+ struct request *rq1,
+ struct request *rq2,
+ sector_t last)
+{
+ sector_t s1, s2, d1 = 0, d2 = 0;
+ unsigned long back_max;
+#define BFQ_RQ1_WRAP 0x01 /* request 1 wraps */
+#define BFQ_RQ2_WRAP 0x02 /* request 2 wraps */
+ unsigned int wrap = 0; /* bit mask: requests behind the disk head? */
+
+ if (!rq1 || rq1 == rq2)
+ return rq2;
+ if (!rq2)
+ return rq1;
+
+ if (rq_is_sync(rq1) && !rq_is_sync(rq2))
+ return rq1;
+ else if (rq_is_sync(rq2) && !rq_is_sync(rq1))
+ return rq2;
+ if ((rq1->cmd_flags & REQ_META) && !(rq2->cmd_flags & REQ_META))
+ return rq1;
+ else if ((rq2->cmd_flags & REQ_META) && !(rq1->cmd_flags & REQ_META))
+ return rq2;
+
+ s1 = blk_rq_pos(rq1);
+ s2 = blk_rq_pos(rq2);
+
+ /*
+ * By definition, 1KiB is 2 sectors.
+ */
+ back_max = bfqd->bfq_back_max * 2;
+
+ /*
+ * Strict one way elevator _except_ in the case where we allow
+ * short backward seeks which are biased as twice the cost of a
+ * similar forward seek.
+ */
+ if (s1 >= last)
+ d1 = s1 - last;
+ else if (s1 + back_max >= last)
+ d1 = (last - s1) * bfqd->bfq_back_penalty;
+ else
+ wrap |= BFQ_RQ1_WRAP;
+
+ if (s2 >= last)
+ d2 = s2 - last;
+ else if (s2 + back_max >= last)
+ d2 = (last - s2) * bfqd->bfq_back_penalty;
+ else
+ wrap |= BFQ_RQ2_WRAP;
+
+ /* Found required data */
+
+ /*
+ * By doing switch() on the bit mask "wrap" we avoid having to
+ * check two variables for all permutations: --> faster!
+ */
+ switch (wrap) {
+ case 0: /* common case for CFQ: rq1 and rq2 not wrapped */
+ if (d1 < d2)
+ return rq1;
+ else if (d2 < d1)
+ return rq2;
+
+ if (s1 >= s2)
+ return rq1;
+ else
+ return rq2;
+
+ case BFQ_RQ2_WRAP:
+ return rq1;
+ case BFQ_RQ1_WRAP:
+ return rq2;
+ case BFQ_RQ1_WRAP|BFQ_RQ2_WRAP: /* both rqs wrapped */
+ default:
+ /*
+ * Since both rqs are wrapped,
+ * start with the one that's further behind head
+ * (--> only *one* back seek required),
+ * since back seek takes more time than forward.
+ */
+ if (s1 <= s2)
+ return rq1;
+ else
+ return rq2;
+ }
+}
+
+static struct bfq_queue *
+bfq_rq_pos_tree_lookup(struct bfq_data *bfqd, struct rb_root *root,
+ sector_t sector, struct rb_node **ret_parent,
+ struct rb_node ***rb_link)
+{
+ struct rb_node **p, *parent;
+ struct bfq_queue *bfqq = NULL;
+
+ parent = NULL;
+ p = &root->rb_node;
+ while (*p) {
+ struct rb_node **n;
+
+ parent = *p;
+ bfqq = rb_entry(parent, struct bfq_queue, pos_node);
+
+ /*
+ * Sort strictly based on sector. Smallest to the left,
+ * largest to the right.
+ */
+ if (sector > blk_rq_pos(bfqq->next_rq))
+ n = &(*p)->rb_right;
+ else if (sector < blk_rq_pos(bfqq->next_rq))
+ n = &(*p)->rb_left;
+ else
+ break;
+ p = n;
+ bfqq = NULL;
+ }
+
+ *ret_parent = parent;
+ if (rb_link)
+ *rb_link = p;
+
+ bfq_log(bfqd, "rq_pos_tree_lookup %llu: returning %d",
+ (unsigned long long)sector,
+ bfqq ? bfqq->pid : 0);
+
+ return bfqq;
+}
+
+void bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq)
+{
+ struct rb_node **p, *parent;
+ struct bfq_queue *__bfqq;
+
+ if (bfqq->pos_root) {
+ rb_erase(&bfqq->pos_node, bfqq->pos_root);
+ bfqq->pos_root = NULL;
+ }
+
+ if (bfq_class_idle(bfqq))
+ return;
+ if (!bfqq->next_rq)
+ return;
+
+ bfqq->pos_root = &bfq_bfqq_to_bfqg(bfqq)->rq_pos_tree;
+ __bfqq = bfq_rq_pos_tree_lookup(bfqd, bfqq->pos_root,
+ blk_rq_pos(bfqq->next_rq), &parent, &p);
+ if (!__bfqq) {
+ rb_link_node(&bfqq->pos_node, parent, p);
+ rb_insert_color(&bfqq->pos_node, bfqq->pos_root);
+ } else
+ bfqq->pos_root = NULL;
+}
+
+/*
+ * Tell whether there are active queues or groups with differentiated weights.
+ */
+static bool bfq_differentiated_weights(struct bfq_data *bfqd)
+{
+ /*
+ * For weights to differ, at least one of the trees must contain
+ * at least two nodes.
+ */
+ return (!RB_EMPTY_ROOT(&bfqd->queue_weights_tree) &&
+ (bfqd->queue_weights_tree.rb_node->rb_left ||
+ bfqd->queue_weights_tree.rb_node->rb_right)
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ ) ||
+ (!RB_EMPTY_ROOT(&bfqd->group_weights_tree) &&
+ (bfqd->group_weights_tree.rb_node->rb_left ||
+ bfqd->group_weights_tree.rb_node->rb_right)
+#endif
+ );
+}
+
+/*
+ * The following function returns true if every queue must receive the
+ * same share of the throughput (this condition is used when deciding
+ * whether idling may be disabled, see the comments in the function
+ * bfq_bfqq_may_idle()).
+ *
+ * Such a scenario occurs when:
+ * 1) all active queues have the same weight,
+ * 2) all active groups at the same level in the groups tree have the same
+ * weight,
+ * 3) all active groups at the same level in the groups tree have the same
+ * number of children.
+ *
+ * Unfortunately, keeping the necessary state for evaluating exactly the
+ * above symmetry conditions would be quite complex and time-consuming.
+ * Therefore this function evaluates, instead, the following stronger
+ * sub-conditions, for which it is much easier to maintain the needed
+ * state:
+ * 1) all active queues have the same weight,
+ * 2) all active groups have the same weight,
+ * 3) all active groups have at most one active child each.
+ * In particular, the last two conditions are always true if hierarchical
+ * support and the cgroups interface are not enabled, thus no state needs
+ * to be maintained in this case.
+ */
+static bool bfq_symmetric_scenario(struct bfq_data *bfqd)
+{
+ return !bfq_differentiated_weights(bfqd);
+}
+
+/*
+ * If the weight-counter tree passed as input contains no counter for
+ * the weight of the input entity, then add that counter; otherwise just
+ * increment the existing counter.
+ *
+ * Note that weight-counter trees contain few nodes in mostly symmetric
+ * scenarios. For example, if all queues have the same weight, then the
+ * weight-counter tree for the queues may contain at most one node.
+ * This holds even if low_latency is on, because weight-raised queues
+ * are not inserted in the tree.
+ * In most scenarios, the rate at which nodes are created/destroyed
+ * should be low too.
+ */
+void bfq_weights_tree_add(struct bfq_data *bfqd, struct bfq_entity *entity,
+ struct rb_root *root)
+{
+ struct rb_node **new = &(root->rb_node), *parent = NULL;
+
+ /*
+ * Do not insert if the entity is already associated with a
+ * counter, which happens if:
+ * 1) the entity is associated with a queue,
+ * 2) a request arrival has caused the queue to become both
+ * non-weight-raised, and hence change its weight, and
+ * backlogged; in this respect, each of the two events
+ * causes an invocation of this function,
+ * 3) this is the invocation of this function caused by the
+ * second event. This second invocation is actually useless,
+ * and we handle this fact by exiting immediately. More
+ * efficient or clearer solutions might possibly be adopted.
+ */
+ if (entity->weight_counter)
+ return;
+
+ while (*new) {
+ struct bfq_weight_counter *__counter = container_of(*new,
+ struct bfq_weight_counter,
+ weights_node);
+ parent = *new;
+
+ if (entity->weight == __counter->weight) {
+ entity->weight_counter = __counter;
+ goto inc_counter;
+ }
+ if (entity->weight < __counter->weight)
+ new = &((*new)->rb_left);
+ else
+ new = &((*new)->rb_right);
+ }
+
+ entity->weight_counter = kzalloc(sizeof(struct bfq_weight_counter),
+ GFP_ATOMIC);
+
+ /*
+ * In the unlucky event of an allocation failure, we just
+ * exit. This will cause the weight of entity to not be
+ * considered in bfq_differentiated_weights, which, in its
+ * turn, causes the scenario to be deemed wrongly symmetric in
+ * case entity's weight would have been the only weight making
+ * the scenario asymmetric. On the bright side, no unbalance
+ * will however occur when entity becomes inactive again (the
+ * invocation of this function is triggered by an activation
+ * of entity). In fact, bfq_weights_tree_remove does nothing
+ * if !entity->weight_counter.
+ */
+ if (unlikely(!entity->weight_counter))
+ return;
+
+ entity->weight_counter->weight = entity->weight;
+ rb_link_node(&entity->weight_counter->weights_node, parent, new);
+ rb_insert_color(&entity->weight_counter->weights_node, root);
+
+inc_counter:
+ entity->weight_counter->num_active++;
+}
+
+/*
+ * Decrement the weight counter associated with the entity, and, if the
+ * counter reaches 0, remove the counter from the tree.
+ * See the comments to the function bfq_weights_tree_add() for considerations
+ * about overhead.
+ */
+void bfq_weights_tree_remove(struct bfq_data *bfqd, struct bfq_entity *entity,
+ struct rb_root *root)
+{
+ if (!entity->weight_counter)
+ return;
+
+ entity->weight_counter->num_active--;
+ if (entity->weight_counter->num_active > 0)
+ goto reset_entity_pointer;
+
+ rb_erase(&entity->weight_counter->weights_node, root);
+ kfree(entity->weight_counter);
+
+reset_entity_pointer:
+ entity->weight_counter = NULL;
+}
+
+/*
+ * Return expired entry, or NULL to just start from scratch in rbtree.
+ */
+static struct request *bfq_check_fifo(struct bfq_queue *bfqq,
+ struct request *last)
+{
+ struct request *rq;
+
+ if (bfq_bfqq_fifo_expire(bfqq))
+ return NULL;
+
+ bfq_mark_bfqq_fifo_expire(bfqq);
+
+ rq = rq_entry_fifo(bfqq->fifo.next);
+
+ if (rq == last || ktime_get_ns() < rq->fifo_time)
+ return NULL;
+
+ bfq_log_bfqq(bfqq->bfqd, bfqq, "check_fifo: returned %p", rq);
+ return rq;
+}
+
+static struct request *bfq_find_next_rq(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq,
+ struct request *last)
+{
+ struct rb_node *rbnext = rb_next(&last->rb_node);
+ struct rb_node *rbprev = rb_prev(&last->rb_node);
+ struct request *next, *prev = NULL;
+
+ /* Follow expired path, else get first next available. */
+ next = bfq_check_fifo(bfqq, last);
+ if (next)
+ return next;
+
+ if (rbprev)
+ prev = rb_entry_rq(rbprev);
+
+ if (rbnext)
+ next = rb_entry_rq(rbnext);
+ else {
+ rbnext = rb_first(&bfqq->sort_list);
+ if (rbnext && rbnext != &last->rb_node)
+ next = rb_entry_rq(rbnext);
+ }
+
+ return bfq_choose_req(bfqd, next, prev, blk_rq_pos(last));
+}
+
+/* see the definition of bfq_async_charge_factor for details */
+static unsigned long bfq_serv_to_charge(struct request *rq,
+ struct bfq_queue *bfqq)
+{
+ if (bfq_bfqq_sync(bfqq) || bfqq->wr_coeff > 1)
+ return blk_rq_sectors(rq);
+
+ /*
+ * If there are no weight-raised queues, then amplify service
+ * by just the async charge factor; otherwise amplify service
+ * by twice the async charge factor, to further reduce latency
+ * for weight-raised queues.
+ */
+ if (bfqq->bfqd->wr_busy_queues == 0)
+ return blk_rq_sectors(rq) * bfq_async_charge_factor;
+
+ return blk_rq_sectors(rq) * 2 * bfq_async_charge_factor;
+}
+
+/**
+ * bfq_updated_next_req - update the queue after a new next_rq selection.
+ * @bfqd: the device data the queue belongs to.
+ * @bfqq: the queue to update.
+ *
+ * If the first request of a queue changes we make sure that the queue
+ * has enough budget to serve at least its first request (if the
+ * request has grown). We do this because if the queue has not enough
+ * budget for its first request, it has to go through two dispatch
+ * rounds to actually get it dispatched.
+ */
+static void bfq_updated_next_req(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq)
+{
+ struct bfq_entity *entity = &bfqq->entity;
+ struct request *next_rq = bfqq->next_rq;
+ unsigned long new_budget;
+
+ if (!next_rq)
+ return;
+
+ if (bfqq == bfqd->in_service_queue)
+ /*
+ * In order not to break guarantees, budgets cannot be
+ * changed after an entity has been selected.
+ */
+ return;
+
+ new_budget = max_t(unsigned long, bfqq->max_budget,
+ bfq_serv_to_charge(next_rq, bfqq));
+ if (entity->budget != new_budget) {
+ entity->budget = new_budget;
+ bfq_log_bfqq(bfqd, bfqq, "updated next rq: new budget %lu",
+ new_budget);
+ bfq_requeue_bfqq(bfqd, bfqq);
+ }
+}
+
+static void
+bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_io_cq *bic)
+{
+ if (bic->saved_idle_window)
+ bfq_mark_bfqq_idle_window(bfqq);
+ else
+ bfq_clear_bfqq_idle_window(bfqq);
+
+ if (bic->saved_IO_bound)
+ bfq_mark_bfqq_IO_bound(bfqq);
+ else
+ bfq_clear_bfqq_IO_bound(bfqq);
+
+ bfqq->ttime = bic->saved_ttime;
+ bfqq->wr_coeff = bic->saved_wr_coeff;
+ bfqq->wr_start_at_switch_to_srt = bic->saved_wr_start_at_switch_to_srt;
+ bfqq->last_wr_start_finish = bic->saved_last_wr_start_finish;
+ bfqq->wr_cur_max_time = bic->saved_wr_cur_max_time;
+
+ if (bfqq->wr_coeff > 1 && (bfq_bfqq_in_large_burst(bfqq) ||
+ time_is_before_jiffies(bfqq->last_wr_start_finish +
+ bfqq->wr_cur_max_time))) {
+ bfq_log_bfqq(bfqq->bfqd, bfqq,
+ "resume state: switching off wr");
+
+ bfqq->wr_coeff = 1;
+ }
+
+ /* make sure weight will be updated, however we got here */
+ bfqq->entity.prio_changed = 1;
+}
+
+static int bfqq_process_refs(struct bfq_queue *bfqq)
+{
+ return bfqq->ref - bfqq->allocated - bfqq->entity.on_st;
+}
+
+/* Empty burst list and add just bfqq (see comments on bfq_handle_burst) */
+static void bfq_reset_burst_list(struct bfq_data *bfqd, struct bfq_queue *bfqq)
+{
+ struct bfq_queue *item;
+ struct hlist_node *n;
+
+ hlist_for_each_entry_safe(item, n, &bfqd->burst_list, burst_list_node)
+ hlist_del_init(&item->burst_list_node);
+ hlist_add_head(&bfqq->burst_list_node, &bfqd->burst_list);
+ bfqd->burst_size = 1;
+ bfqd->burst_parent_entity = bfqq->entity.parent;
+}
+
+/* Add bfqq to the list of queues in current burst (see bfq_handle_burst) */
+static void bfq_add_to_burst(struct bfq_data *bfqd, struct bfq_queue *bfqq)
+{
+ /* Increment burst size to take into account also bfqq */
+ bfqd->burst_size++;
+
+ if (bfqd->burst_size == bfqd->bfq_large_burst_thresh) {
+ struct bfq_queue *pos, *bfqq_item;
+ struct hlist_node *n;
+
+ /*
+ * Enough queues have been activated shortly after each
+ * other to consider this burst as large.
+ */
+ bfqd->large_burst = true;
+
+ /*
+ * We can now mark all queues in the burst list as
+ * belonging to a large burst.
+ */
+ hlist_for_each_entry(bfqq_item, &bfqd->burst_list,
+ burst_list_node)
+ bfq_mark_bfqq_in_large_burst(bfqq_item);
+ bfq_mark_bfqq_in_large_burst(bfqq);
+
+ /*
+ * From now on, and until the current burst finishes, any
+ * new queue being activated shortly after the last queue
+ * was inserted in the burst can be immediately marked as
+ * belonging to a large burst. So the burst list is not
+ * needed any more. Remove it.
+ */
+ hlist_for_each_entry_safe(pos, n, &bfqd->burst_list,
+ burst_list_node)
+ hlist_del_init(&pos->burst_list_node);
+ } else /*
+ * Burst not yet large: add bfqq to the burst list. Do
+ * not increment the ref counter for bfqq, because bfqq
+ * is removed from the burst list before freeing bfqq
+ * in put_queue.
+ */
+ hlist_add_head(&bfqq->burst_list_node, &bfqd->burst_list);
+}
+
+/*
+ * If many queues belonging to the same group happen to be created
+ * shortly after each other, then the processes associated with these
+ * queues have typically a common goal. In particular, bursts of queue
+ * creations are usually caused by services or applications that spawn
+ * many parallel threads/processes. Examples are systemd during boot,
+ * or git grep. To help these processes get their job done as soon as
+ * possible, it is usually better to not grant either weight-raising
+ * or device idling to their queues.
+ *
+ * In this comment we describe, firstly, the reasons why this fact
+ * holds, and, secondly, the next function, which implements the main
+ * steps needed to properly mark these queues so that they can then be
+ * treated in a different way.
+ *
+ * The above services or applications benefit mostly from a high
+ * throughput: the quicker the requests of the activated queues are
+ * cumulatively served, the sooner the target job of these queues gets
+ * completed. As a consequence, weight-raising any of these queues,
+ * which also implies idling the device for it, is almost always
+ * counterproductive. In most cases it just lowers throughput.
+ *
+ * On the other hand, a burst of queue creations may be caused also by
+ * the start of an application that does not consist of a lot of
+ * parallel I/O-bound threads. In fact, with a complex application,
+ * several short processes may need to be executed to start-up the
+ * application. In this respect, to start an application as quickly as
+ * possible, the best thing to do is in any case to privilege the I/O
+ * related to the application with respect to all other
+ * I/O. Therefore, the best strategy to start as quickly as possible
+ * an application that causes a burst of queue creations is to
+ * weight-raise all the queues created during the burst. This is the
+ * exact opposite of the best strategy for the other type of bursts.
+ *
+ * In the end, to take the best action for each of the two cases, the
+ * two types of bursts need to be distinguished. Fortunately, this
+ * seems relatively easy, by looking at the sizes of the bursts. In
+ * particular, we found a threshold such that only bursts with a
+ * larger size than that threshold are apparently caused by
+ * services or commands such as systemd or git grep. For brevity,
+ * hereafter we call just 'large' these bursts. BFQ *does not*
+ * weight-raise queues whose creation occurs in a large burst. In
+ * addition, for each of these queues BFQ performs or does not perform
+ * idling depending on which choice boosts the throughput more. The
+ * exact choice depends on the device and request pattern at
+ * hand.
+ *
+ * Unfortunately, false positives may occur while an interactive task
+ * is starting (e.g., an application is being started). The
+ * consequence is that the queues associated with the task do not
+ * enjoy weight raising as expected. Fortunately these false positives
+ * are very rare. They typically occur if some service happens to
+ * start doing I/O exactly when the interactive task starts.
+ *
+ * Turning back to the next function, it implements all the steps
+ * needed to detect the occurrence of a large burst and to properly
+ * mark all the queues belonging to it (so that they can then be
+ * treated in a different way). This goal is achieved by maintaining a
+ * "burst list" that holds, temporarily, the queues that belong to the
+ * burst in progress. The list is then used to mark these queues as
+ * belonging to a large burst if the burst does become large. The main
+ * steps are the following.
+ *
+ * . when the very first queue is created, the queue is inserted into the
+ * list (as it could be the first queue in a possible burst)
+ *
+ * . if the current burst has not yet become large, and a queue Q that does
+ * not yet belong to the burst is activated shortly after the last time
+ * at which a new queue entered the burst list, then the function appends
+ * Q to the burst list
+ *
+ * . if, as a consequence of the previous step, the burst size reaches
+ * the large-burst threshold, then
+ *
+ * . all the queues in the burst list are marked as belonging to a
+ * large burst
+ *
+ * . the burst list is deleted; in fact, the burst list already served
+ * its purpose (keeping temporarily track of the queues in a burst,
+ * so as to be able to mark them as belonging to a large burst in the
+ * previous sub-step), and now is not needed any more
+ *
+ * . the device enters a large-burst mode
+ *
+ * . if a queue Q that does not belong to the burst is created while
+ * the device is in large-burst mode and shortly after the last time
+ * at which a queue either entered the burst list or was marked as
+ * belonging to the current large burst, then Q is immediately marked
+ * as belonging to a large burst.
+ *
+ * . if a queue Q that does not belong to the burst is created a while
+ * later, i.e., not shortly after, than the last time at which a queue
+ * either entered the burst list or was marked as belonging to the
+ * current large burst, then the current burst is deemed as finished and:
+ *
+ * . the large-burst mode is reset if set
+ *
+ * . the burst list is emptied
+ *
+ * . Q is inserted in the burst list, as Q may be the first queue
+ * in a possible new burst (then the burst list contains just Q
+ * after this step).
+ */
+static void bfq_handle_burst(struct bfq_data *bfqd, struct bfq_queue *bfqq)
+{
+ /*
+ * If bfqq is already in the burst list or is part of a large
+ * burst, or finally has just been split, then there is
+ * nothing else to do.
+ */
+ if (!hlist_unhashed(&bfqq->burst_list_node) ||
+ bfq_bfqq_in_large_burst(bfqq) ||
+ time_is_after_eq_jiffies(bfqq->split_time +
+ msecs_to_jiffies(10)))
+ return;
+
+ /*
+ * If bfqq's creation happens late enough, or bfqq belongs to
+ * a different group than the burst group, then the current
+ * burst is finished, and related data structures must be
+ * reset.
+ *
+ * In this respect, consider the special case where bfqq is
+ * the very first queue created after BFQ is selected for this
+ * device. In this case, last_ins_in_burst and
+ * burst_parent_entity are not yet significant when we get
+ * here. But it is easy to verify that, whether or not the
+ * following condition is true, bfqq will end up being
+ * inserted into the burst list. In particular the list will
+ * happen to contain only bfqq. And this is exactly what has
+ * to happen, as bfqq may be the first queue of the first
+ * burst.
+ */
+ if (time_is_before_jiffies(bfqd->last_ins_in_burst +
+ bfqd->bfq_burst_interval) ||
+ bfqq->entity.parent != bfqd->burst_parent_entity) {
+ bfqd->large_burst = false;
+ bfq_reset_burst_list(bfqd, bfqq);
+ goto end;
+ }
+
+ /*
+ * If we get here, then bfqq is being activated shortly after the
+ * last queue. So, if the current burst is also large, we can mark
+ * bfqq as belonging to this large burst immediately.
+ */
+ if (bfqd->large_burst) {
+ bfq_mark_bfqq_in_large_burst(bfqq);
+ goto end;
+ }
+
+ /*
+ * If we get here, then a large-burst state has not yet been
+ * reached, but bfqq is being activated shortly after the last
+ * queue. Then we add bfqq to the burst.
+ */
+ bfq_add_to_burst(bfqd, bfqq);
+end:
+ /*
+ * At this point, bfqq either has been added to the current
+ * burst or has caused the current burst to terminate and a
+ * possible new burst to start. In particular, in the second
+ * case, bfqq has become the first queue in the possible new
+ * burst. In both cases last_ins_in_burst needs to be moved
+ * forward.
+ */
+ bfqd->last_ins_in_burst = jiffies;
+}
+
+static int bfq_bfqq_budget_left(struct bfq_queue *bfqq)
+{
+ struct bfq_entity *entity = &bfqq->entity;
+
+ return entity->budget - entity->service;
+}
+
+/*
+ * If enough samples have been computed, return the current max budget
+ * stored in bfqd, which is dynamically updated according to the
+ * estimated disk peak rate; otherwise return the default max budget
+ */
+static int bfq_max_budget(struct bfq_data *bfqd)
+{
+ if (bfqd->budgets_assigned < bfq_stats_min_budgets)
+ return bfq_default_max_budget;
+ else
+ return bfqd->bfq_max_budget;
+}
+
+/*
+ * Return min budget, which is a fraction of the current or default
+ * max budget (trying with 1/32)
+ */
+static int bfq_min_budget(struct bfq_data *bfqd)
+{
+ if (bfqd->budgets_assigned < bfq_stats_min_budgets)
+ return bfq_default_max_budget / 32;
+ else
+ return bfqd->bfq_max_budget / 32;
+}
+
+/*
+ * The next function, invoked after the input queue bfqq switches from
+ * idle to busy, updates the budget of bfqq. The function also tells
+ * whether the in-service queue should be expired, by returning
+ * true. The purpose of expiring the in-service queue is to give bfqq
+ * the chance to possibly preempt the in-service queue, and the reason
+ * for preempting the in-service queue is to achieve one of the two
+ * goals below.
+ *
+ * 1. Guarantee to bfqq its reserved bandwidth even if bfqq has
+ * expired because it has remained idle. In particular, bfqq may have
+ * expired for one of the following two reasons:
+ *
+ * - BFQQE_NO_MORE_REQUESTS bfqq did not enjoy any device idling
+ * and did not make it to issue a new request before its last
+ * request was served;
+ *
+ * - BFQQE_TOO_IDLE bfqq did enjoy device idling, but did not issue
+ * a new request before the expiration of the idling-time.
+ *
+ * Even if bfqq has expired for one of the above reasons, the process
+ * associated with the queue may be however issuing requests greedily,
+ * and thus be sensitive to the bandwidth it receives (bfqq may have
+ * remained idle for other reasons: CPU high load, bfqq not enjoying
+ * idling, I/O throttling somewhere in the path from the process to
+ * the I/O scheduler, ...). But if, after every expiration for one of
+ * the above two reasons, bfqq has to wait for the service of at least
+ * one full budget of another queue before being served again, then
+ * bfqq is likely to get a much lower bandwidth or resource time than
+ * its reserved ones. To address this issue, two countermeasures need
+ * to be taken.
+ *
+ * First, the budget and the timestamps of bfqq need to be updated in
+ * a special way on bfqq reactivation: they need to be updated as if
+ * bfqq did not remain idle and did not expire. In fact, if they are
+ * computed as if bfqq expired and remained idle until reactivation,
+ * then the process associated with bfqq is treated as if, instead of
+ * being greedy, it stopped issuing requests when bfqq remained idle,
+ * and restarts issuing requests only on this reactivation. In other
+ * words, the scheduler does not help the process recover the "service
+ * hole" between bfqq expiration and reactivation. As a consequence,
+ * the process receives a lower bandwidth than its reserved one. In
+ * contrast, to recover this hole, the budget must be updated as if
+ * bfqq was not expired at all before this reactivation, i.e., it must
+ * be set to the value of the remaining budget when bfqq was
+ * expired. Along the same line, timestamps need to be assigned the
+ * value they had the last time bfqq was selected for service, i.e.,
+ * before last expiration. Thus timestamps need to be back-shifted
+ * with respect to their normal computation (see [1] for more details
+ * on this tricky aspect).
+ *
+ * Secondly, to allow the process to recover the hole, the in-service
+ * queue must be expired too, to give bfqq the chance to preempt it
+ * immediately. In fact, if bfqq has to wait for a full budget of the
+ * in-service queue to be completed, then it may become impossible to
+ * let the process recover the hole, even if the back-shifted
+ * timestamps of bfqq are lower than those of the in-service queue. If
+ * this happens for most or all of the holes, then the process may not
+ * receive its reserved bandwidth. In this respect, it is worth noting
+ * that, being the service of outstanding requests unpreemptible, a
+ * little fraction of the holes may however be unrecoverable, thereby
+ * causing a little loss of bandwidth.
+ *
+ * The last important point is detecting whether bfqq does need this
+ * bandwidth recovery. In this respect, the next function deems the
+ * process associated with bfqq greedy, and thus allows it to recover
+ * the hole, if: 1) the process is waiting for the arrival of a new
+ * request (which implies that bfqq expired for one of the above two
+ * reasons), and 2) such a request has arrived soon. The first
+ * condition is controlled through the flag non_blocking_wait_rq,
+ * while the second through the flag arrived_in_time. If both
+ * conditions hold, then the function computes the budget in the
+ * above-described special way, and signals that the in-service queue
+ * should be expired. Timestamp back-shifting is done later in
+ * __bfq_activate_entity.
+ *
+ * 2. Reduce latency. Even if timestamps are not backshifted to let
+ * the process associated with bfqq recover a service hole, bfqq may
+ * however happen to have, after being (re)activated, a lower finish
+ * timestamp than the in-service queue. That is, the next budget of
+ * bfqq may have to be completed before the one of the in-service
+ * queue. If this is the case, then preempting the in-service queue
+ * allows this goal to be achieved, apart from the unpreemptible,
+ * outstanding requests mentioned above.
+ *
+ * Unfortunately, regardless of which of the above two goals one wants
+ * to achieve, service trees need first to be updated to know whether
+ * the in-service queue must be preempted. To have service trees
+ * correctly updated, the in-service queue must be expired and
+ * rescheduled, and bfqq must be scheduled too. This is one of the
+ * most costly operations (in future versions, the scheduling
+ * mechanism may be re-designed in such a way to make it possible to
+ * know whether preemption is needed without needing to update service
+ * trees). In addition, queue preemptions almost always cause random
+ * I/O, and thus loss of throughput. Because of these facts, the next
+ * function adopts the following simple scheme to avoid both costly
+ * operations and too frequent preemptions: it requests the expiration
+ * of the in-service queue (unconditionally) only for queues that need
+ * to recover a hole, or that either are weight-raised or deserve to
+ * be weight-raised.
+ */
+static bool bfq_bfqq_update_budg_for_activation(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq,
+ bool arrived_in_time,
+ bool wr_or_deserves_wr)
+{
+ struct bfq_entity *entity = &bfqq->entity;
+
+ if (bfq_bfqq_non_blocking_wait_rq(bfqq) && arrived_in_time) {
+ /*
+ * We do not clear the flag non_blocking_wait_rq here, as
+ * the latter is used in bfq_activate_bfqq to signal
+ * that timestamps need to be back-shifted (and is
+ * cleared right after).
+ */
+
+ /*
+ * In next assignment we rely on that either
+ * entity->service or entity->budget are not updated
+ * on expiration if bfqq is empty (see
+ * __bfq_bfqq_recalc_budget). Thus both quantities
+ * remain unchanged after such an expiration, and the
+ * following statement therefore assigns to
+ * entity->budget the remaining budget on such an
+ * expiration. For clarity, entity->service is not
+ * updated on expiration in any case, and, in normal
+ * operation, is reset only when bfqq is selected for
+ * service (see bfq_get_next_queue).
+ */
+ entity->budget = min_t(unsigned long,
+ bfq_bfqq_budget_left(bfqq),
+ bfqq->max_budget);
+
+ return true;
+ }
+
+ entity->budget = max_t(unsigned long, bfqq->max_budget,
+ bfq_serv_to_charge(bfqq->next_rq, bfqq));
+ bfq_clear_bfqq_non_blocking_wait_rq(bfqq);
+ return wr_or_deserves_wr;
+}
+
+static unsigned int bfq_wr_duration(struct bfq_data *bfqd)
+{
+ u64 dur;
+
+ if (bfqd->bfq_wr_max_time > 0)
+ return bfqd->bfq_wr_max_time;
+
+ dur = bfqd->RT_prod;
+ do_div(dur, bfqd->peak_rate);
+
+ /*
+ * Limit duration between 3 and 13 seconds. Tests show that
+ * higher values than 13 seconds often yield the opposite of
+ * the desired result, i.e., worsen responsiveness by letting
+ * non-interactive and non-soft-real-time applications
+ * preserve weight raising for a too long time interval.
+ *
+ * On the other end, lower values than 3 seconds make it
+ * difficult for most interactive tasks to complete their jobs
+ * before weight-raising finishes.
+ */
+ if (dur > msecs_to_jiffies(13000))
+ dur = msecs_to_jiffies(13000);
+ else if (dur < msecs_to_jiffies(3000))
+ dur = msecs_to_jiffies(3000);
+
+ return dur;
+}
+
+static void bfq_update_bfqq_wr_on_rq_arrival(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq,
+ unsigned int old_wr_coeff,
+ bool wr_or_deserves_wr,
+ bool interactive,
+ bool in_burst,
+ bool soft_rt)
+{
+ if (old_wr_coeff == 1 && wr_or_deserves_wr) {
+ /* start a weight-raising period */
+ if (interactive) {
+ bfqq->wr_coeff = bfqd->bfq_wr_coeff;
+ bfqq->wr_cur_max_time = bfq_wr_duration(bfqd);
+ } else {
+ bfqq->wr_start_at_switch_to_srt = jiffies;
+ bfqq->wr_coeff = bfqd->bfq_wr_coeff *
+ BFQ_SOFTRT_WEIGHT_FACTOR;
+ bfqq->wr_cur_max_time =
+ bfqd->bfq_wr_rt_max_time;
+ }
+
+ /*
+ * If needed, further reduce budget to make sure it is
+ * close to bfqq's backlog, so as to reduce the
+ * scheduling-error component due to a too large
+ * budget. Do not care about throughput consequences,
+ * but only about latency. Finally, do not assign a
+ * too small budget either, to avoid increasing
+ * latency by causing too frequent expirations.
+ */
+ bfqq->entity.budget = min_t(unsigned long,
+ bfqq->entity.budget,
+ 2 * bfq_min_budget(bfqd));
+ } else if (old_wr_coeff > 1) {
+ if (interactive) { /* update wr coeff and duration */
+ bfqq->wr_coeff = bfqd->bfq_wr_coeff;
+ bfqq->wr_cur_max_time = bfq_wr_duration(bfqd);
+ } else if (in_burst)
+ bfqq->wr_coeff = 1;
+ else if (soft_rt) {
+ /*
+ * The application is now or still meeting the
+ * requirements for being deemed soft rt. We
+ * can then correctly and safely (re)charge
+ * the weight-raising duration for the
+ * application with the weight-raising
+ * duration for soft rt applications.
+ *
+ * In particular, doing this recharge now, i.e.,
+ * before the weight-raising period for the
+ * application finishes, reduces the probability
+ * of the following negative scenario:
+ * 1) the weight of a soft rt application is
+ * raised at startup (as for any newly
+ * created application),
+ * 2) since the application is not interactive,
+ * at a certain time weight-raising is
+ * stopped for the application,
+ * 3) at that time the application happens to
+ * still have pending requests, and hence
+ * is destined to not have a chance to be
+ * deemed soft rt before these requests are
+ * completed (see the comments to the
+ * function bfq_bfqq_softrt_next_start()
+ * for details on soft rt detection),
+ * 4) these pending requests experience a high
+ * latency because the application is not
+ * weight-raised while they are pending.
+ */
+ if (bfqq->wr_cur_max_time !=
+ bfqd->bfq_wr_rt_max_time) {
+ bfqq->wr_start_at_switch_to_srt =
+ bfqq->last_wr_start_finish;
+
+ bfqq->wr_cur_max_time =
+ bfqd->bfq_wr_rt_max_time;
+ bfqq->wr_coeff = bfqd->bfq_wr_coeff *
+ BFQ_SOFTRT_WEIGHT_FACTOR;
+ }
+ bfqq->last_wr_start_finish = jiffies;
+ }
+ }
+}
+
+static bool bfq_bfqq_idle_for_long_time(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq)
+{
+ return bfqq->dispatched == 0 &&
+ time_is_before_jiffies(
+ bfqq->budget_timeout +
+ bfqd->bfq_wr_min_idle_time);
+}
+
+static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq,
+ int old_wr_coeff,
+ struct request *rq,
+ bool *interactive)
+{
+ bool soft_rt, in_burst, wr_or_deserves_wr,
+ bfqq_wants_to_preempt,
+ idle_for_long_time = bfq_bfqq_idle_for_long_time(bfqd, bfqq),
+ /*
+ * See the comments on
+ * bfq_bfqq_update_budg_for_activation for
+ * details on the usage of the next variable.
+ */
+ arrived_in_time = ktime_get_ns() <=
+ bfqq->ttime.last_end_request +
+ bfqd->bfq_slice_idle * 3;
+
+ bfqg_stats_update_io_add(bfqq_group(RQ_BFQQ(rq)), bfqq, rq->cmd_flags);
+
+ /*
+ * bfqq deserves to be weight-raised if:
+ * - it is sync,
+ * - it does not belong to a large burst,
+ * - it has been idle for enough time or is soft real-time,
+ * - is linked to a bfq_io_cq (it is not shared in any sense).
+ */
+ in_burst = bfq_bfqq_in_large_burst(bfqq);
+ soft_rt = bfqd->bfq_wr_max_softrt_rate > 0 &&
+ !in_burst &&
+ time_is_before_jiffies(bfqq->soft_rt_next_start);
+ *interactive = !in_burst && idle_for_long_time;
+ wr_or_deserves_wr = bfqd->low_latency &&
+ (bfqq->wr_coeff > 1 ||
+ (bfq_bfqq_sync(bfqq) &&
+ bfqq->bic && (*interactive || soft_rt)));
+
+ /*
+ * Using the last flag, update budget and check whether bfqq
+ * may want to preempt the in-service queue.
+ */
+ bfqq_wants_to_preempt =
+ bfq_bfqq_update_budg_for_activation(bfqd, bfqq,
+ arrived_in_time,
+ wr_or_deserves_wr);
+
+ /*
+ * If bfqq happened to be activated in a burst, but has been
+ * idle for much more than an interactive queue, then we
+ * assume that, in the overall I/O initiated in the burst, the
+ * I/O associated with bfqq is finished. So bfqq does not need
+ * to be treated as a queue belonging to a burst
+ * anymore. Accordingly, we reset bfqq's in_large_burst flag
+ * if set, and remove bfqq from the burst list if it's
+ * there. We do not decrement burst_size, because the fact
+ * that bfqq does not need to belong to the burst list any
+ * more does not invalidate the fact that bfqq was created in
+ * a burst.
+ */
+ if (likely(!bfq_bfqq_just_created(bfqq)) &&
+ idle_for_long_time &&
+ time_is_before_jiffies(
+ bfqq->budget_timeout +
+ msecs_to_jiffies(10000))) {
+ hlist_del_init(&bfqq->burst_list_node);
+ bfq_clear_bfqq_in_large_burst(bfqq);
+ }
+
+ bfq_clear_bfqq_just_created(bfqq);
+
+
+ if (!bfq_bfqq_IO_bound(bfqq)) {
+ if (arrived_in_time) {
+ bfqq->requests_within_timer++;
+ if (bfqq->requests_within_timer >=
+ bfqd->bfq_requests_within_timer)
+ bfq_mark_bfqq_IO_bound(bfqq);
+ } else
+ bfqq->requests_within_timer = 0;
+ }
+
+ if (bfqd->low_latency) {
+ if (unlikely(time_is_after_jiffies(bfqq->split_time)))
+ /* wraparound */
+ bfqq->split_time =
+ jiffies - bfqd->bfq_wr_min_idle_time - 1;
+
+ if (time_is_before_jiffies(bfqq->split_time +
+ bfqd->bfq_wr_min_idle_time)) {
+ bfq_update_bfqq_wr_on_rq_arrival(bfqd, bfqq,
+ old_wr_coeff,
+ wr_or_deserves_wr,
+ *interactive,
+ in_burst,
+ soft_rt);
+
+ if (old_wr_coeff != bfqq->wr_coeff)
+ bfqq->entity.prio_changed = 1;
+ }
+ }
+
+ bfqq->last_idle_bklogged = jiffies;
+ bfqq->service_from_backlogged = 0;
+ bfq_clear_bfqq_softrt_update(bfqq);
+
+ bfq_add_bfqq_busy(bfqd, bfqq);
+
+ /*
+ * Expire in-service queue only if preemption may be needed
+ * for guarantees. In this respect, the function
+ * next_queue_may_preempt just checks a simple, necessary
+ * condition, and not a sufficient condition based on
+ * timestamps. In fact, for the latter condition to be
+ * evaluated, timestamps would need first to be updated, and
+ * this operation is quite costly (see the comments on the
+ * function bfq_bfqq_update_budg_for_activation).
+ */
+ if (bfqd->in_service_queue && bfqq_wants_to_preempt &&
+ bfqd->in_service_queue->wr_coeff < bfqq->wr_coeff &&
+ next_queue_may_preempt(bfqd))
+ bfq_bfqq_expire(bfqd, bfqd->in_service_queue,
+ false, BFQQE_PREEMPTED);
+}
+
+static void bfq_add_request(struct request *rq)
+{
+ struct bfq_queue *bfqq = RQ_BFQQ(rq);
+ struct bfq_data *bfqd = bfqq->bfqd;
+ struct request *next_rq, *prev;
+ unsigned int old_wr_coeff = bfqq->wr_coeff;
+ bool interactive = false;
+
+ bfq_log_bfqq(bfqd, bfqq, "add_request %d", rq_is_sync(rq));
+ bfqq->queued[rq_is_sync(rq)]++;
+ bfqd->queued++;
+
+ elv_rb_add(&bfqq->sort_list, rq);
+
+ /*
+ * Check if this request is a better next-serve candidate.
+ */
+ prev = bfqq->next_rq;
+ next_rq = bfq_choose_req(bfqd, bfqq->next_rq, rq, bfqd->last_position);
+ bfqq->next_rq = next_rq;
+
+ /*
+ * Adjust priority tree position, if next_rq changes.
+ */
+ if (prev != bfqq->next_rq)
+ bfq_pos_tree_add_move(bfqd, bfqq);
+
+ if (!bfq_bfqq_busy(bfqq)) /* switching to busy ... */
+ bfq_bfqq_handle_idle_busy_switch(bfqd, bfqq, old_wr_coeff,
+ rq, &interactive);
+ else {
+ if (bfqd->low_latency && old_wr_coeff == 1 && !rq_is_sync(rq) &&
+ time_is_before_jiffies(
+ bfqq->last_wr_start_finish +
+ bfqd->bfq_wr_min_inter_arr_async)) {
+ bfqq->wr_coeff = bfqd->bfq_wr_coeff;
+ bfqq->wr_cur_max_time = bfq_wr_duration(bfqd);
+
+ bfqd->wr_busy_queues++;
+ bfqq->entity.prio_changed = 1;
+ }
+ if (prev != bfqq->next_rq)
+ bfq_updated_next_req(bfqd, bfqq);
+ }
+
+ /*
+ * Assign jiffies to last_wr_start_finish in the following
+ * cases:
+ *
+ * . if bfqq is not going to be weight-raised, because, for
+ * non weight-raised queues, last_wr_start_finish stores the
+ * arrival time of the last request; as of now, this piece
+ * of information is used only for deciding whether to
+ * weight-raise async queues
+ *
+ * . if bfqq is not weight-raised, because, if bfqq is now
+ * switching to weight-raised, then last_wr_start_finish
+ * stores the time when weight-raising starts
+ *
+ * . if bfqq is interactive, because, regardless of whether
+ * bfqq is currently weight-raised, the weight-raising
+ * period must start or restart (this case is considered
+ * separately because it is not detected by the above
+ * conditions, if bfqq is already weight-raised)
+ *
+ * last_wr_start_finish has to be updated also if bfqq is soft
+ * real-time, because the weight-raising period is constantly
+ * restarted on idle-to-busy transitions for these queues, but
+ * this is already done in bfq_bfqq_handle_idle_busy_switch if
+ * needed.
+ */
+ if (bfqd->low_latency &&
+ (old_wr_coeff == 1 || bfqq->wr_coeff == 1 || interactive))
+ bfqq->last_wr_start_finish = jiffies;
+}
+
+static struct request *bfq_find_rq_fmerge(struct bfq_data *bfqd,
+ struct bio *bio,
+ struct request_queue *q)
+{
+ struct bfq_queue *bfqq = bfqd->bio_bfqq;
+
+
+ if (bfqq)
+ return elv_rb_find(&bfqq->sort_list, bio_end_sector(bio));
+
+ return NULL;
+}
+
+static sector_t get_sdist(sector_t last_pos, struct request *rq)
+{
+ if (last_pos)
+ return abs(blk_rq_pos(rq) - last_pos);
+
+ return 0;
+}
+
+#if 0 /* Still not clear if we can do without next two functions */
+static void bfq_activate_request(struct request_queue *q, struct request *rq)
+{
+ struct bfq_data *bfqd = q->elevator->elevator_data;
+
+ bfqd->rq_in_driver++;
+}
+
+static void bfq_deactivate_request(struct request_queue *q, struct request *rq)
+{
+ struct bfq_data *bfqd = q->elevator->elevator_data;
+
+ bfqd->rq_in_driver--;
+}
+#endif
+
+static void bfq_remove_request(struct request_queue *q,
+ struct request *rq)
+{
+ struct bfq_queue *bfqq = RQ_BFQQ(rq);
+ struct bfq_data *bfqd = bfqq->bfqd;
+ const int sync = rq_is_sync(rq);
+
+ if (bfqq->next_rq == rq) {
+ bfqq->next_rq = bfq_find_next_rq(bfqd, bfqq, rq);
+ bfq_updated_next_req(bfqd, bfqq);
+ }
+
+ if (rq->queuelist.prev != &rq->queuelist)
+ list_del_init(&rq->queuelist);
+ bfqq->queued[sync]--;
+ bfqd->queued--;
+ elv_rb_del(&bfqq->sort_list, rq);
+
+ elv_rqhash_del(q, rq);
+ if (q->last_merge == rq)
+ q->last_merge = NULL;
+
+ if (RB_EMPTY_ROOT(&bfqq->sort_list)) {
+ bfqq->next_rq = NULL;
+
+ if (bfq_bfqq_busy(bfqq) && bfqq != bfqd->in_service_queue) {
+ bfq_del_bfqq_busy(bfqd, bfqq, false);
+ /*
+ * bfqq emptied. In normal operation, when
+ * bfqq is empty, bfqq->entity.service and
+ * bfqq->entity.budget must contain,
+ * respectively, the service received and the
+ * budget used last time bfqq emptied. These
+ * facts do not hold in this case, as at least
+ * this last removal occurred while bfqq is
+ * not in service. To avoid inconsistencies,
+ * reset both bfqq->entity.service and
+ * bfqq->entity.budget, if bfqq has still a
+ * process that may issue I/O requests to it.
+ */
+ bfqq->entity.budget = bfqq->entity.service = 0;
+ }
+
+ /*
+ * Remove queue from request-position tree as it is empty.
+ */
+ if (bfqq->pos_root) {
+ rb_erase(&bfqq->pos_node, bfqq->pos_root);
+ bfqq->pos_root = NULL;
+ }
+ }
+
+ if (rq->cmd_flags & REQ_META)
+ bfqq->meta_pending--;
+
+ bfqg_stats_update_io_remove(bfqq_group(bfqq), rq->cmd_flags);
+}
+
+static bool bfq_bio_merge(struct blk_mq_hw_ctx *hctx, struct bio *bio)
+{
+ struct request_queue *q = hctx->queue;
+ struct bfq_data *bfqd = q->elevator->elevator_data;
+ struct request *free = NULL;
+ /*
+ * bfq_bic_lookup grabs the queue_lock: invoke it now and
+ * store its return value for later use, to avoid nesting
+ * queue_lock inside the bfqd->lock. We assume that the bic
+ * returned by bfq_bic_lookup does not go away before
+ * bfqd->lock is taken.
+ */
+ struct bfq_io_cq *bic = bfq_bic_lookup(bfqd, current->io_context, q);
+ bool ret;
+
+ spin_lock_irq(&bfqd->lock);
+
+ if (bic)
+ bfqd->bio_bfqq = bic_to_bfqq(bic, op_is_sync(bio->bi_opf));
+ else
+ bfqd->bio_bfqq = NULL;
+ bfqd->bio_bic = bic;
+
+ ret = blk_mq_sched_try_merge(q, bio, &free);
+
+ if (free)
+ blk_mq_free_request(free);
+ spin_unlock_irq(&bfqd->lock);
+
+ return ret;
+}
+
+static int bfq_request_merge(struct request_queue *q, struct request **req,
+ struct bio *bio)
+{
+ struct bfq_data *bfqd = q->elevator->elevator_data;
+ struct request *__rq;
+
+ __rq = bfq_find_rq_fmerge(bfqd, bio, q);
+ if (__rq && elv_bio_merge_ok(__rq, bio)) {
+ *req = __rq;
+ return ELEVATOR_FRONT_MERGE;
+ }
+
+ return ELEVATOR_NO_MERGE;
+}
+
+static void bfq_request_merged(struct request_queue *q, struct request *req,
+ enum elv_merge type)
+{
+ if (type == ELEVATOR_FRONT_MERGE &&
+ rb_prev(&req->rb_node) &&
+ blk_rq_pos(req) <
+ blk_rq_pos(container_of(rb_prev(&req->rb_node),
+ struct request, rb_node))) {
+ struct bfq_queue *bfqq = RQ_BFQQ(req);
+ struct bfq_data *bfqd = bfqq->bfqd;
+ struct request *prev, *next_rq;
+
+ /* Reposition request in its sort_list */
+ elv_rb_del(&bfqq->sort_list, req);
+ elv_rb_add(&bfqq->sort_list, req);
+
+ /* Choose next request to be served for bfqq */
+ prev = bfqq->next_rq;
+ next_rq = bfq_choose_req(bfqd, bfqq->next_rq, req,
+ bfqd->last_position);
+ bfqq->next_rq = next_rq;
+ /*
+ * If next_rq changes, update both the queue's budget to
+ * fit the new request and the queue's position in its
+ * rq_pos_tree.
+ */
+ if (prev != bfqq->next_rq) {
+ bfq_updated_next_req(bfqd, bfqq);
+ bfq_pos_tree_add_move(bfqd, bfqq);
+ }
+ }
+}
+
+static void bfq_requests_merged(struct request_queue *q, struct request *rq,
+ struct request *next)
+{
+ struct bfq_queue *bfqq = RQ_BFQQ(rq), *next_bfqq = RQ_BFQQ(next);
+
+ if (!RB_EMPTY_NODE(&rq->rb_node))
+ goto end;
+ spin_lock_irq(&bfqq->bfqd->lock);
+
+ /*
+ * If next and rq belong to the same bfq_queue and next is older
+ * than rq, then reposition rq in the fifo (by substituting next
+ * with rq). Otherwise, if next and rq belong to different
+ * bfq_queues, never reposition rq: in fact, we would have to
+ * reposition it with respect to next's position in its own fifo,
+ * which would most certainly be too expensive with respect to
+ * the benefits.
+ */
+ if (bfqq == next_bfqq &&
+ !list_empty(&rq->queuelist) && !list_empty(&next->queuelist) &&
+ next->fifo_time < rq->fifo_time) {
+ list_del_init(&rq->queuelist);
+ list_replace_init(&next->queuelist, &rq->queuelist);
+ rq->fifo_time = next->fifo_time;
+ }
+
+ if (bfqq->next_rq == next)
+ bfqq->next_rq = rq;
+
+ bfq_remove_request(q, next);
+
+ spin_unlock_irq(&bfqq->bfqd->lock);
+end:
+ bfqg_stats_update_io_merged(bfqq_group(bfqq), next->cmd_flags);
+}
+
+/* Must be called with bfqq != NULL */
+static void bfq_bfqq_end_wr(struct bfq_queue *bfqq)
+{
+ if (bfq_bfqq_busy(bfqq))
+ bfqq->bfqd->wr_busy_queues--;
+ bfqq->wr_coeff = 1;
+ bfqq->wr_cur_max_time = 0;
+ bfqq->last_wr_start_finish = jiffies;
+ /*
+ * Trigger a weight change on the next invocation of
+ * __bfq_entity_update_weight_prio.
+ */
+ bfqq->entity.prio_changed = 1;
+}
+
+void bfq_end_wr_async_queues(struct bfq_data *bfqd,
+ struct bfq_group *bfqg)
+{
+ int i, j;
+
+ for (i = 0; i < 2; i++)
+ for (j = 0; j < IOPRIO_BE_NR; j++)
+ if (bfqg->async_bfqq[i][j])
+ bfq_bfqq_end_wr(bfqg->async_bfqq[i][j]);
+ if (bfqg->async_idle_bfqq)
+ bfq_bfqq_end_wr(bfqg->async_idle_bfqq);
+}
+
+static void bfq_end_wr(struct bfq_data *bfqd)
+{
+ struct bfq_queue *bfqq;
+
+ spin_lock_irq(&bfqd->lock);
+
+ list_for_each_entry(bfqq, &bfqd->active_list, bfqq_list)
+ bfq_bfqq_end_wr(bfqq);
+ list_for_each_entry(bfqq, &bfqd->idle_list, bfqq_list)
+ bfq_bfqq_end_wr(bfqq);
+ bfq_end_wr_async(bfqd);
+
+ spin_unlock_irq(&bfqd->lock);
+}
+
+static sector_t bfq_io_struct_pos(void *io_struct, bool request)
+{
+ if (request)
+ return blk_rq_pos(io_struct);
+ else
+ return ((struct bio *)io_struct)->bi_iter.bi_sector;
+}
+
+static int bfq_rq_close_to_sector(void *io_struct, bool request,
+ sector_t sector)
+{
+ return abs(bfq_io_struct_pos(io_struct, request) - sector) <=
+ BFQQ_CLOSE_THR;
+}
+
+static struct bfq_queue *bfqq_find_close(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq,
+ sector_t sector)
+{
+ struct rb_root *root = &bfq_bfqq_to_bfqg(bfqq)->rq_pos_tree;
+ struct rb_node *parent, *node;
+ struct bfq_queue *__bfqq;
+
+ if (RB_EMPTY_ROOT(root))
+ return NULL;
+
+ /*
+ * First, if we find a request starting at the end of the last
+ * request, choose it.
+ */
+ __bfqq = bfq_rq_pos_tree_lookup(bfqd, root, sector, &parent, NULL);
+ if (__bfqq)
+ return __bfqq;
+
+ /*
+ * If the exact sector wasn't found, the parent of the NULL leaf
+ * will contain the closest sector (rq_pos_tree sorted by
+ * next_request position).
+ */
+ __bfqq = rb_entry(parent, struct bfq_queue, pos_node);
+ if (bfq_rq_close_to_sector(__bfqq->next_rq, true, sector))
+ return __bfqq;
+
+ if (blk_rq_pos(__bfqq->next_rq) < sector)
+ node = rb_next(&__bfqq->pos_node);
+ else
+ node = rb_prev(&__bfqq->pos_node);
+ if (!node)
+ return NULL;
+
+ __bfqq = rb_entry(node, struct bfq_queue, pos_node);
+ if (bfq_rq_close_to_sector(__bfqq->next_rq, true, sector))
+ return __bfqq;
+
+ return NULL;
+}
+
+static struct bfq_queue *bfq_find_close_cooperator(struct bfq_data *bfqd,
+ struct bfq_queue *cur_bfqq,
+ sector_t sector)
+{
+ struct bfq_queue *bfqq;
+
+ /*
+ * We shall notice if some of the queues are cooperating,
+ * e.g., working closely on the same area of the device. In
+ * that case, we can group them together and: 1) don't waste
+ * time idling, and 2) serve the union of their requests in
+ * the best possible order for throughput.
+ */
+ bfqq = bfqq_find_close(bfqd, cur_bfqq, sector);
+ if (!bfqq || bfqq == cur_bfqq)
+ return NULL;
+
+ return bfqq;
+}
+
+static struct bfq_queue *
+bfq_setup_merge(struct bfq_queue *bfqq, struct bfq_queue *new_bfqq)
+{
+ int process_refs, new_process_refs;
+ struct bfq_queue *__bfqq;
+
+ /*
+ * If there are no process references on the new_bfqq, then it is
+ * unsafe to follow the ->new_bfqq chain as other bfqq's in the chain
+ * may have dropped their last reference (not just their last process
+ * reference).
+ */
+ if (!bfqq_process_refs(new_bfqq))
+ return NULL;
+
+ /* Avoid a circular list and skip interim queue merges. */
+ while ((__bfqq = new_bfqq->new_bfqq)) {
+ if (__bfqq == bfqq)
+ return NULL;
+ new_bfqq = __bfqq;
+ }
+
+ process_refs = bfqq_process_refs(bfqq);
+ new_process_refs = bfqq_process_refs(new_bfqq);
+ /*
+ * If the process for the bfqq has gone away, there is no
+ * sense in merging the queues.
+ */
+ if (process_refs == 0 || new_process_refs == 0)
+ return NULL;
+
+ bfq_log_bfqq(bfqq->bfqd, bfqq, "scheduling merge with queue %d",
+ new_bfqq->pid);
+
+ /*
+ * Merging is just a redirection: the requests of the process
+ * owning one of the two queues are redirected to the other queue.
+ * The latter queue, in its turn, is set as shared if this is the
+ * first time that the requests of some process are redirected to
+ * it.
+ *
+ * We redirect bfqq to new_bfqq and not the opposite, because
+ * we are in the context of the process owning bfqq, thus we
+ * have the io_cq of this process. So we can immediately
+ * configure this io_cq to redirect the requests of the
+ * process to new_bfqq. In contrast, the io_cq of new_bfqq is
+ * not available any more (new_bfqq->bic == NULL).
+ *
+ * Anyway, even in case new_bfqq coincides with the in-service
+ * queue, redirecting requests the in-service queue is the
+ * best option, as we feed the in-service queue with new
+ * requests close to the last request served and, by doing so,
+ * are likely to increase the throughput.
+ */
+ bfqq->new_bfqq = new_bfqq;
+ new_bfqq->ref += process_refs;
+ return new_bfqq;
+}
+
+static bool bfq_may_be_close_cooperator(struct bfq_queue *bfqq,
+ struct bfq_queue *new_bfqq)
+{
+ if (bfq_class_idle(bfqq) || bfq_class_idle(new_bfqq) ||
+ (bfqq->ioprio_class != new_bfqq->ioprio_class))
+ return false;
+
+ /*
+ * If either of the queues has already been detected as seeky,
+ * then merging it with the other queue is unlikely to lead to
+ * sequential I/O.
+ */
+ if (BFQQ_SEEKY(bfqq) || BFQQ_SEEKY(new_bfqq))
+ return false;
+
+ /*
+ * Interleaved I/O is known to be done by (some) applications
+ * only for reads, so it does not make sense to merge async
+ * queues.
+ */
+ if (!bfq_bfqq_sync(bfqq) || !bfq_bfqq_sync(new_bfqq))
+ return false;
+
+ return true;
+}
+
+/*
+ * If this function returns true, then bfqq cannot be merged. The idea
+ * is that true cooperation happens very early after processes start
+ * to do I/O. Usually, late cooperations are just accidental false
+ * positives. In case bfqq is weight-raised, such false positives
+ * would evidently degrade latency guarantees for bfqq.
+ */
+static bool wr_from_too_long(struct bfq_queue *bfqq)
+{
+ return bfqq->wr_coeff > 1 &&
+ time_is_before_jiffies(bfqq->last_wr_start_finish +
+ msecs_to_jiffies(100));
+}
+
+/*
+ * Attempt to schedule a merge of bfqq with the currently in-service
+ * queue or with a close queue among the scheduled queues. Return
+ * NULL if no merge was scheduled, a pointer to the shared bfq_queue
+ * structure otherwise.
+ *
+ * The OOM queue is not allowed to participate to cooperation: in fact, since
+ * the requests temporarily redirected to the OOM queue could be redirected
+ * again to dedicated queues at any time, the state needed to correctly
+ * handle merging with the OOM queue would be quite complex and expensive
+ * to maintain. Besides, in such a critical condition as an out of memory,
+ * the benefits of queue merging may be little relevant, or even negligible.
+ *
+ * Weight-raised queues can be merged only if their weight-raising
+ * period has just started. In fact cooperating processes are usually
+ * started together. Thus, with this filter we avoid false positives
+ * that would jeopardize low-latency guarantees.
+ *
+ * WARNING: queue merging may impair fairness among non-weight raised
+ * queues, for at least two reasons: 1) the original weight of a
+ * merged queue may change during the merged state, 2) even being the
+ * weight the same, a merged queue may be bloated with many more
+ * requests than the ones produced by its originally-associated
+ * process.
+ */
+static struct bfq_queue *
+bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ void *io_struct, bool request)
+{
+ struct bfq_queue *in_service_bfqq, *new_bfqq;
+
+ if (bfqq->new_bfqq)
+ return bfqq->new_bfqq;
+
+ if (!io_struct ||
+ wr_from_too_long(bfqq) ||
+ unlikely(bfqq == &bfqd->oom_bfqq))
+ return NULL;
+
+ /* If there is only one backlogged queue, don't search. */
+ if (bfqd->busy_queues == 1)
+ return NULL;
+
+ in_service_bfqq = bfqd->in_service_queue;
+
+ if (!in_service_bfqq || in_service_bfqq == bfqq
+ || wr_from_too_long(in_service_bfqq) ||
+ unlikely(in_service_bfqq == &bfqd->oom_bfqq))
+ goto check_scheduled;
+
+ if (bfq_rq_close_to_sector(io_struct, request, bfqd->last_position) &&
+ bfqq->entity.parent == in_service_bfqq->entity.parent &&
+ bfq_may_be_close_cooperator(bfqq, in_service_bfqq)) {
+ new_bfqq = bfq_setup_merge(bfqq, in_service_bfqq);
+ if (new_bfqq)
+ return new_bfqq;
+ }
+ /*
+ * Check whether there is a cooperator among currently scheduled
+ * queues. The only thing we need is that the bio/request is not
+ * NULL, as we need it to establish whether a cooperator exists.
+ */
+check_scheduled:
+ new_bfqq = bfq_find_close_cooperator(bfqd, bfqq,
+ bfq_io_struct_pos(io_struct, request));
+
+ if (new_bfqq && !wr_from_too_long(new_bfqq) &&
+ likely(new_bfqq != &bfqd->oom_bfqq) &&
+ bfq_may_be_close_cooperator(bfqq, new_bfqq))
+ return bfq_setup_merge(bfqq, new_bfqq);
+
+ return NULL;
+}
+
+static void bfq_bfqq_save_state(struct bfq_queue *bfqq)
+{
+ struct bfq_io_cq *bic = bfqq->bic;
+
+ /*
+ * If !bfqq->bic, the queue is already shared or its requests
+ * have already been redirected to a shared queue; both idle window
+ * and weight raising state have already been saved. Do nothing.
+ */
+ if (!bic)
+ return;
+
+ bic->saved_ttime = bfqq->ttime;
+ bic->saved_idle_window = bfq_bfqq_idle_window(bfqq);
+ bic->saved_IO_bound = bfq_bfqq_IO_bound(bfqq);
+ bic->saved_in_large_burst = bfq_bfqq_in_large_burst(bfqq);
+ bic->was_in_burst_list = !hlist_unhashed(&bfqq->burst_list_node);
+ bic->saved_wr_coeff = bfqq->wr_coeff;
+ bic->saved_wr_start_at_switch_to_srt = bfqq->wr_start_at_switch_to_srt;
+ bic->saved_last_wr_start_finish = bfqq->last_wr_start_finish;
+ bic->saved_wr_cur_max_time = bfqq->wr_cur_max_time;
+}
+
+static void
+bfq_merge_bfqqs(struct bfq_data *bfqd, struct bfq_io_cq *bic,
+ struct bfq_queue *bfqq, struct bfq_queue *new_bfqq)
+{
+ bfq_log_bfqq(bfqd, bfqq, "merging with queue %lu",
+ (unsigned long)new_bfqq->pid);
+ /* Save weight raising and idle window of the merged queues */
+ bfq_bfqq_save_state(bfqq);
+ bfq_bfqq_save_state(new_bfqq);
+ if (bfq_bfqq_IO_bound(bfqq))
+ bfq_mark_bfqq_IO_bound(new_bfqq);
+ bfq_clear_bfqq_IO_bound(bfqq);
+
+ /*
+ * If bfqq is weight-raised, then let new_bfqq inherit
+ * weight-raising. To reduce false positives, neglect the case
+ * where bfqq has just been created, but has not yet made it
+ * to be weight-raised (which may happen because EQM may merge
+ * bfqq even before bfq_add_request is executed for the first
+ * time for bfqq). Handling this case would however be very
+ * easy, thanks to the flag just_created.
+ */
+ if (new_bfqq->wr_coeff == 1 && bfqq->wr_coeff > 1) {
+ new_bfqq->wr_coeff = bfqq->wr_coeff;
+ new_bfqq->wr_cur_max_time = bfqq->wr_cur_max_time;
+ new_bfqq->last_wr_start_finish = bfqq->last_wr_start_finish;
+ new_bfqq->wr_start_at_switch_to_srt =
+ bfqq->wr_start_at_switch_to_srt;
+ if (bfq_bfqq_busy(new_bfqq))
+ bfqd->wr_busy_queues++;
+ new_bfqq->entity.prio_changed = 1;
+ }
+
+ if (bfqq->wr_coeff > 1) { /* bfqq has given its wr to new_bfqq */
+ bfqq->wr_coeff = 1;
+ bfqq->entity.prio_changed = 1;
+ if (bfq_bfqq_busy(bfqq))
+ bfqd->wr_busy_queues--;
+ }
+
+ bfq_log_bfqq(bfqd, new_bfqq, "merge_bfqqs: wr_busy %d",
+ bfqd->wr_busy_queues);
+
+ /*
+ * Merge queues (that is, let bic redirect its requests to new_bfqq)
+ */
+ bic_set_bfqq(bic, new_bfqq, 1);
+ bfq_mark_bfqq_coop(new_bfqq);
+ /*
+ * new_bfqq now belongs to at least two bics (it is a shared queue):
+ * set new_bfqq->bic to NULL. bfqq either:
+ * - does not belong to any bic any more, and hence bfqq->bic must
+ * be set to NULL, or
+ * - is a queue whose owning bics have already been redirected to a
+ * different queue, hence the queue is destined to not belong to
+ * any bic soon and bfqq->bic is already NULL (therefore the next
+ * assignment causes no harm).
+ */
+ new_bfqq->bic = NULL;
+ bfqq->bic = NULL;
+ /* release process reference to bfqq */
+ bfq_put_queue(bfqq);
+}
+
+static bool bfq_allow_bio_merge(struct request_queue *q, struct request *rq,
+ struct bio *bio)
+{
+ struct bfq_data *bfqd = q->elevator->elevator_data;
+ bool is_sync = op_is_sync(bio->bi_opf);
+ struct bfq_queue *bfqq = bfqd->bio_bfqq, *new_bfqq;
+
+ /*
+ * Disallow merge of a sync bio into an async request.
+ */
+ if (is_sync && !rq_is_sync(rq))
+ return false;
+
+ /*
+ * Lookup the bfqq that this bio will be queued with. Allow
+ * merge only if rq is queued there.
+ */
+ if (!bfqq)
+ return false;
+
+ /*
+ * We take advantage of this function to perform an early merge
+ * of the queues of possible cooperating processes.
+ */
+ new_bfqq = bfq_setup_cooperator(bfqd, bfqq, bio, false);
+ if (new_bfqq) {
+ /*
+ * bic still points to bfqq, then it has not yet been
+ * redirected to some other bfq_queue, and a queue
+ * merge beween bfqq and new_bfqq can be safely
+ * fulfillled, i.e., bic can be redirected to new_bfqq
+ * and bfqq can be put.
+ */
+ bfq_merge_bfqqs(bfqd, bfqd->bio_bic, bfqq,
+ new_bfqq);
+ /*
+ * If we get here, bio will be queued into new_queue,
+ * so use new_bfqq to decide whether bio and rq can be
+ * merged.
+ */
+ bfqq = new_bfqq;
+
+ /*
+ * Change also bqfd->bio_bfqq, as
+ * bfqd->bio_bic now points to new_bfqq, and
+ * this function may be invoked again (and then may
+ * use again bqfd->bio_bfqq).
+ */
+ bfqd->bio_bfqq = bfqq;
+ }
+
+ return bfqq == RQ_BFQQ(rq);
+}
+
+/*
+ * Set the maximum time for the in-service queue to consume its
+ * budget. This prevents seeky processes from lowering the throughput.
+ * In practice, a time-slice service scheme is used with seeky
+ * processes.
+ */
+static void bfq_set_budget_timeout(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq)
+{
+ unsigned int timeout_coeff;
+
+ if (bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time)
+ timeout_coeff = 1;
+ else
+ timeout_coeff = bfqq->entity.weight / bfqq->entity.orig_weight;
+
+ bfqd->last_budget_start = ktime_get();
+
+ bfqq->budget_timeout = jiffies +
+ bfqd->bfq_timeout * timeout_coeff;
+}
+
+static void __bfq_set_in_service_queue(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq)
+{
+ if (bfqq) {
+ bfqg_stats_update_avg_queue_size(bfqq_group(bfqq));
+ bfq_clear_bfqq_fifo_expire(bfqq);
+
+ bfqd->budgets_assigned = (bfqd->budgets_assigned * 7 + 256) / 8;
+
+ if (time_is_before_jiffies(bfqq->last_wr_start_finish) &&
+ bfqq->wr_coeff > 1 &&
+ bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time &&
+ time_is_before_jiffies(bfqq->budget_timeout)) {
+ /*
+ * For soft real-time queues, move the start
+ * of the weight-raising period forward by the
+ * time the queue has not received any
+ * service. Otherwise, a relatively long
+ * service delay is likely to cause the
+ * weight-raising period of the queue to end,
+ * because of the short duration of the
+ * weight-raising period of a soft real-time
+ * queue. It is worth noting that this move
+ * is not so dangerous for the other queues,
+ * because soft real-time queues are not
+ * greedy.
+ *
+ * To not add a further variable, we use the
+ * overloaded field budget_timeout to
+ * determine for how long the queue has not
+ * received service, i.e., how much time has
+ * elapsed since the queue expired. However,
+ * this is a little imprecise, because
+ * budget_timeout is set to jiffies if bfqq
+ * not only expires, but also remains with no
+ * request.
+ */
+ if (time_after(bfqq->budget_timeout,
+ bfqq->last_wr_start_finish))
+ bfqq->last_wr_start_finish +=
+ jiffies - bfqq->budget_timeout;
+ else
+ bfqq->last_wr_start_finish = jiffies;
+ }
+
+ bfq_set_budget_timeout(bfqd, bfqq);
+ bfq_log_bfqq(bfqd, bfqq,
+ "set_in_service_queue, cur-budget = %d",
+ bfqq->entity.budget);
+ }
+
+ bfqd->in_service_queue = bfqq;
+}
+
+/*
+ * Get and set a new queue for service.
+ */
+static struct bfq_queue *bfq_set_in_service_queue(struct bfq_data *bfqd)
+{
+ struct bfq_queue *bfqq = bfq_get_next_queue(bfqd);
+
+ __bfq_set_in_service_queue(bfqd, bfqq);
+ return bfqq;
+}
+
+static void bfq_arm_slice_timer(struct bfq_data *bfqd)
+{
+ struct bfq_queue *bfqq = bfqd->in_service_queue;
+ u32 sl;
+
+ bfq_mark_bfqq_wait_request(bfqq);
+
+ /*
+ * We don't want to idle for seeks, but we do want to allow
+ * fair distribution of slice time for a process doing back-to-back
+ * seeks. So allow a little bit of time for him to submit a new rq.
+ */
+ sl = bfqd->bfq_slice_idle;
+ /*
+ * Unless the queue is being weight-raised or the scenario is
+ * asymmetric, grant only minimum idle time if the queue
+ * is seeky. A long idling is preserved for a weight-raised
+ * queue, or, more in general, in an asymmetric scenario,
+ * because a long idling is needed for guaranteeing to a queue
+ * its reserved share of the throughput (in particular, it is
+ * needed if the queue has a higher weight than some other
+ * queue).
+ */
+ if (BFQQ_SEEKY(bfqq) && bfqq->wr_coeff == 1 &&
+ bfq_symmetric_scenario(bfqd))
+ sl = min_t(u64, sl, BFQ_MIN_TT);
+
+ bfqd->last_idling_start = ktime_get();
+ hrtimer_start(&bfqd->idle_slice_timer, ns_to_ktime(sl),
+ HRTIMER_MODE_REL);
+ bfqg_stats_set_start_idle_time(bfqq_group(bfqq));
+}
+
+/*
+ * In autotuning mode, max_budget is dynamically recomputed as the
+ * amount of sectors transferred in timeout at the estimated peak
+ * rate. This enables BFQ to utilize a full timeslice with a full
+ * budget, even if the in-service queue is served at peak rate. And
+ * this maximises throughput with sequential workloads.
+ */
+static unsigned long bfq_calc_max_budget(struct bfq_data *bfqd)
+{
+ return (u64)bfqd->peak_rate * USEC_PER_MSEC *
+ jiffies_to_msecs(bfqd->bfq_timeout)>>BFQ_RATE_SHIFT;
+}
+
+/*
+ * Update parameters related to throughput and responsiveness, as a
+ * function of the estimated peak rate. See comments on
+ * bfq_calc_max_budget(), and on T_slow and T_fast arrays.
+ */
+static void update_thr_responsiveness_params(struct bfq_data *bfqd)
+{
+ int dev_type = blk_queue_nonrot(bfqd->queue);
+
+ if (bfqd->bfq_user_max_budget == 0)
+ bfqd->bfq_max_budget =
+ bfq_calc_max_budget(bfqd);
+
+ if (bfqd->device_speed == BFQ_BFQD_FAST &&
+ bfqd->peak_rate < device_speed_thresh[dev_type]) {
+ bfqd->device_speed = BFQ_BFQD_SLOW;
+ bfqd->RT_prod = R_slow[dev_type] *
+ T_slow[dev_type];
+ } else if (bfqd->device_speed == BFQ_BFQD_SLOW &&
+ bfqd->peak_rate > device_speed_thresh[dev_type]) {
+ bfqd->device_speed = BFQ_BFQD_FAST;
+ bfqd->RT_prod = R_fast[dev_type] *
+ T_fast[dev_type];
+ }
+
+ bfq_log(bfqd,
+"dev_type %s dev_speed_class = %s (%llu sects/sec), thresh %llu setcs/sec",
+ dev_type == 0 ? "ROT" : "NONROT",
+ bfqd->device_speed == BFQ_BFQD_FAST ? "FAST" : "SLOW",
+ bfqd->device_speed == BFQ_BFQD_FAST ?
+ (USEC_PER_SEC*(u64)R_fast[dev_type])>>BFQ_RATE_SHIFT :
+ (USEC_PER_SEC*(u64)R_slow[dev_type])>>BFQ_RATE_SHIFT,
+ (USEC_PER_SEC*(u64)device_speed_thresh[dev_type])>>
+ BFQ_RATE_SHIFT);
+}
+
+static void bfq_reset_rate_computation(struct bfq_data *bfqd,
+ struct request *rq)
+{
+ if (rq != NULL) { /* new rq dispatch now, reset accordingly */
+ bfqd->last_dispatch = bfqd->first_dispatch = ktime_get_ns();
+ bfqd->peak_rate_samples = 1;
+ bfqd->sequential_samples = 0;
+ bfqd->tot_sectors_dispatched = bfqd->last_rq_max_size =
+ blk_rq_sectors(rq);
+ } else /* no new rq dispatched, just reset the number of samples */
+ bfqd->peak_rate_samples = 0; /* full re-init on next disp. */
+
+ bfq_log(bfqd,
+ "reset_rate_computation at end, sample %u/%u tot_sects %llu",
+ bfqd->peak_rate_samples, bfqd->sequential_samples,
+ bfqd->tot_sectors_dispatched);
+}
+
+static void bfq_update_rate_reset(struct bfq_data *bfqd, struct request *rq)
+{
+ u32 rate, weight, divisor;
+
+ /*
+ * For the convergence property to hold (see comments on
+ * bfq_update_peak_rate()) and for the assessment to be
+ * reliable, a minimum number of samples must be present, and
+ * a minimum amount of time must have elapsed. If not so, do
+ * not compute new rate. Just reset parameters, to get ready
+ * for a new evaluation attempt.
+ */
+ if (bfqd->peak_rate_samples < BFQ_RATE_MIN_SAMPLES ||
+ bfqd->delta_from_first < BFQ_RATE_MIN_INTERVAL)
+ goto reset_computation;
+
+ /*
+ * If a new request completion has occurred after last
+ * dispatch, then, to approximate the rate at which requests
+ * have been served by the device, it is more precise to
+ * extend the observation interval to the last completion.
+ */
+ bfqd->delta_from_first =
+ max_t(u64, bfqd->delta_from_first,
+ bfqd->last_completion - bfqd->first_dispatch);
+
+ /*
+ * Rate computed in sects/usec, and not sects/nsec, for
+ * precision issues.
+ */
+ rate = div64_ul(bfqd->tot_sectors_dispatched<<BFQ_RATE_SHIFT,
+ div_u64(bfqd->delta_from_first, NSEC_PER_USEC));
+
+ /*
+ * Peak rate not updated if:
+ * - the percentage of sequential dispatches is below 3/4 of the
+ * total, and rate is below the current estimated peak rate
+ * - rate is unreasonably high (> 20M sectors/sec)
+ */
+ if ((bfqd->sequential_samples < (3 * bfqd->peak_rate_samples)>>2 &&
+ rate <= bfqd->peak_rate) ||
+ rate > 20<<BFQ_RATE_SHIFT)
+ goto reset_computation;
+
+ /*
+ * We have to update the peak rate, at last! To this purpose,
+ * we use a low-pass filter. We compute the smoothing constant
+ * of the filter as a function of the 'weight' of the new
+ * measured rate.
+ *
+ * As can be seen in next formulas, we define this weight as a
+ * quantity proportional to how sequential the workload is,
+ * and to how long the observation time interval is.
+ *
+ * The weight runs from 0 to 8. The maximum value of the
+ * weight, 8, yields the minimum value for the smoothing
+ * constant. At this minimum value for the smoothing constant,
+ * the measured rate contributes for half of the next value of
+ * the estimated peak rate.
+ *
+ * So, the first step is to compute the weight as a function
+ * of how sequential the workload is. Note that the weight
+ * cannot reach 9, because bfqd->sequential_samples cannot
+ * become equal to bfqd->peak_rate_samples, which, in its
+ * turn, holds true because bfqd->sequential_samples is not
+ * incremented for the first sample.
+ */
+ weight = (9 * bfqd->sequential_samples) / bfqd->peak_rate_samples;
+
+ /*
+ * Second step: further refine the weight as a function of the
+ * duration of the observation interval.
+ */
+ weight = min_t(u32, 8,
+ div_u64(weight * bfqd->delta_from_first,
+ BFQ_RATE_REF_INTERVAL));
+
+ /*
+ * Divisor ranging from 10, for minimum weight, to 2, for
+ * maximum weight.
+ */
+ divisor = 10 - weight;
+
+ /*
+ * Finally, update peak rate:
+ *
+ * peak_rate = peak_rate * (divisor-1) / divisor + rate / divisor
+ */
+ bfqd->peak_rate *= divisor-1;
+ bfqd->peak_rate /= divisor;
+ rate /= divisor; /* smoothing constant alpha = 1/divisor */
+
+ bfqd->peak_rate += rate;
+ update_thr_responsiveness_params(bfqd);
+
+reset_computation:
+ bfq_reset_rate_computation(bfqd, rq);
+}
+
+/*
+ * Update the read/write peak rate (the main quantity used for
+ * auto-tuning, see update_thr_responsiveness_params()).
+ *
+ * It is not trivial to estimate the peak rate (correctly): because of
+ * the presence of sw and hw queues between the scheduler and the
+ * device components that finally serve I/O requests, it is hard to
+ * say exactly when a given dispatched request is served inside the
+ * device, and for how long. As a consequence, it is hard to know
+ * precisely at what rate a given set of requests is actually served
+ * by the device.
+ *
+ * On the opposite end, the dispatch time of any request is trivially
+ * available, and, from this piece of information, the "dispatch rate"
+ * of requests can be immediately computed. So, the idea in the next
+ * function is to use what is known, namely request dispatch times
+ * (plus, when useful, request completion times), to estimate what is
+ * unknown, namely in-device request service rate.
+ *
+ * The main issue is that, because of the above facts, the rate at
+ * which a certain set of requests is dispatched over a certain time
+ * interval can vary greatly with respect to the rate at which the
+ * same requests are then served. But, since the size of any
+ * intermediate queue is limited, and the service scheme is lossless
+ * (no request is silently dropped), the following obvious convergence
+ * property holds: the number of requests dispatched MUST become
+ * closer and closer to the number of requests completed as the
+ * observation interval grows. This is the key property used in
+ * the next function to estimate the peak service rate as a function
+ * of the observed dispatch rate. The function assumes to be invoked
+ * on every request dispatch.
+ */
+static void bfq_update_peak_rate(struct bfq_data *bfqd, struct request *rq)
+{
+ u64 now_ns = ktime_get_ns();
+
+ if (bfqd->peak_rate_samples == 0) { /* first dispatch */
+ bfq_log(bfqd, "update_peak_rate: goto reset, samples %d",
+ bfqd->peak_rate_samples);
+ bfq_reset_rate_computation(bfqd, rq);
+ goto update_last_values; /* will add one sample */
+ }
+
+ /*
+ * Device idle for very long: the observation interval lasting
+ * up to this dispatch cannot be a valid observation interval
+ * for computing a new peak rate (similarly to the late-
+ * completion event in bfq_completed_request()). Go to
+ * update_rate_and_reset to have the following three steps
+ * taken:
+ * - close the observation interval at the last (previous)
+ * request dispatch or completion
+ * - compute rate, if possible, for that observation interval
+ * - start a new observation interval with this dispatch
+ */
+ if (now_ns - bfqd->last_dispatch > 100*NSEC_PER_MSEC &&
+ bfqd->rq_in_driver == 0)
+ goto update_rate_and_reset;
+
+ /* Update sampling information */
+ bfqd->peak_rate_samples++;
+
+ if ((bfqd->rq_in_driver > 0 ||
+ now_ns - bfqd->last_completion < BFQ_MIN_TT)
+ && get_sdist(bfqd->last_position, rq) < BFQQ_SEEK_THR)
+ bfqd->sequential_samples++;
+
+ bfqd->tot_sectors_dispatched += blk_rq_sectors(rq);
+
+ /* Reset max observed rq size every 32 dispatches */
+ if (likely(bfqd->peak_rate_samples % 32))
+ bfqd->last_rq_max_size =
+ max_t(u32, blk_rq_sectors(rq), bfqd->last_rq_max_size);
+ else
+ bfqd->last_rq_max_size = blk_rq_sectors(rq);
+
+ bfqd->delta_from_first = now_ns - bfqd->first_dispatch;
+
+ /* Target observation interval not yet reached, go on sampling */
+ if (bfqd->delta_from_first < BFQ_RATE_REF_INTERVAL)
+ goto update_last_values;
+
+update_rate_and_reset:
+ bfq_update_rate_reset(bfqd, rq);
+update_last_values:
+ bfqd->last_position = blk_rq_pos(rq) + blk_rq_sectors(rq);
+ bfqd->last_dispatch = now_ns;
+}
+
+/*
+ * Remove request from internal lists.
+ */
+static void bfq_dispatch_remove(struct request_queue *q, struct request *rq)
+{
+ struct bfq_queue *bfqq = RQ_BFQQ(rq);
+
+ /*
+ * For consistency, the next instruction should have been
+ * executed after removing the request from the queue and
+ * dispatching it. We execute instead this instruction before
+ * bfq_remove_request() (and hence introduce a temporary
+ * inconsistency), for efficiency. In fact, should this
+ * dispatch occur for a non in-service bfqq, this anticipated
+ * increment prevents two counters related to bfqq->dispatched
+ * from risking to be, first, uselessly decremented, and then
+ * incremented again when the (new) value of bfqq->dispatched
+ * happens to be taken into account.
+ */
+ bfqq->dispatched++;
+ bfq_update_peak_rate(q->elevator->elevator_data, rq);
+
+ bfq_remove_request(q, rq);
+}
+
+static void __bfq_bfqq_expire(struct bfq_data *bfqd, struct bfq_queue *bfqq)
+{
+ /*
+ * If this bfqq is shared between multiple processes, check
+ * to make sure that those processes are still issuing I/Os
+ * within the mean seek distance. If not, it may be time to
+ * break the queues apart again.
+ */
+ if (bfq_bfqq_coop(bfqq) && BFQQ_SEEKY(bfqq))
+ bfq_mark_bfqq_split_coop(bfqq);
+
+ if (RB_EMPTY_ROOT(&bfqq->sort_list)) {
+ if (bfqq->dispatched == 0)
+ /*
+ * Overloading budget_timeout field to store
+ * the time at which the queue remains with no
+ * backlog and no outstanding request; used by
+ * the weight-raising mechanism.
+ */
+ bfqq->budget_timeout = jiffies;
+
+ bfq_del_bfqq_busy(bfqd, bfqq, true);
+ } else {
+ bfq_requeue_bfqq(bfqd, bfqq);
+ /*
+ * Resort priority tree of potential close cooperators.
+ */
+ bfq_pos_tree_add_move(bfqd, bfqq);
+ }
+
+ /*
+ * All in-service entities must have been properly deactivated
+ * or requeued before executing the next function, which
+ * resets all in-service entites as no more in service.
+ */
+ __bfq_bfqd_reset_in_service(bfqd);
+}
+
+/**
+ * __bfq_bfqq_recalc_budget - try to adapt the budget to the @bfqq behavior.
+ * @bfqd: device data.
+ * @bfqq: queue to update.
+ * @reason: reason for expiration.
+ *
+ * Handle the feedback on @bfqq budget at queue expiration.
+ * See the body for detailed comments.
+ */
+static void __bfq_bfqq_recalc_budget(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq,
+ enum bfqq_expiration reason)
+{
+ struct request *next_rq;
+ int budget, min_budget;
+
+ min_budget = bfq_min_budget(bfqd);
+
+ if (bfqq->wr_coeff == 1)
+ budget = bfqq->max_budget;
+ else /*
+ * Use a constant, low budget for weight-raised queues,
+ * to help achieve a low latency. Keep it slightly higher
+ * than the minimum possible budget, to cause a little
+ * bit fewer expirations.
+ */
+ budget = 2 * min_budget;
+
+ bfq_log_bfqq(bfqd, bfqq, "recalc_budg: last budg %d, budg left %d",
+ bfqq->entity.budget, bfq_bfqq_budget_left(bfqq));
+ bfq_log_bfqq(bfqd, bfqq, "recalc_budg: last max_budg %d, min budg %d",
+ budget, bfq_min_budget(bfqd));
+ bfq_log_bfqq(bfqd, bfqq, "recalc_budg: sync %d, seeky %d",
+ bfq_bfqq_sync(bfqq), BFQQ_SEEKY(bfqd->in_service_queue));
+
+ if (bfq_bfqq_sync(bfqq) && bfqq->wr_coeff == 1) {
+ switch (reason) {
+ /*
+ * Caveat: in all the following cases we trade latency
+ * for throughput.
+ */
+ case BFQQE_TOO_IDLE:
+ /*
+ * This is the only case where we may reduce
+ * the budget: if there is no request of the
+ * process still waiting for completion, then
+ * we assume (tentatively) that the timer has
+ * expired because the batch of requests of
+ * the process could have been served with a
+ * smaller budget. Hence, betting that
+ * process will behave in the same way when it
+ * becomes backlogged again, we reduce its
+ * next budget. As long as we guess right,
+ * this budget cut reduces the latency
+ * experienced by the process.
+ *
+ * However, if there are still outstanding
+ * requests, then the process may have not yet
+ * issued its next request just because it is
+ * still waiting for the completion of some of
+ * the still outstanding ones. So in this
+ * subcase we do not reduce its budget, on the
+ * contrary we increase it to possibly boost
+ * the throughput, as discussed in the
+ * comments to the BUDGET_TIMEOUT case.
+ */
+ if (bfqq->dispatched > 0) /* still outstanding reqs */
+ budget = min(budget * 2, bfqd->bfq_max_budget);
+ else {
+ if (budget > 5 * min_budget)
+ budget -= 4 * min_budget;
+ else
+ budget = min_budget;
+ }
+ break;
+ case BFQQE_BUDGET_TIMEOUT:
+ /*
+ * We double the budget here because it gives
+ * the chance to boost the throughput if this
+ * is not a seeky process (and has bumped into
+ * this timeout because of, e.g., ZBR).
+ */
+ budget = min(budget * 2, bfqd->bfq_max_budget);
+ break;
+ case BFQQE_BUDGET_EXHAUSTED:
+ /*
+ * The process still has backlog, and did not
+ * let either the budget timeout or the disk
+ * idling timeout expire. Hence it is not
+ * seeky, has a short thinktime and may be
+ * happy with a higher budget too. So
+ * definitely increase the budget of this good
+ * candidate to boost the disk throughput.
+ */
+ budget = min(budget * 4, bfqd->bfq_max_budget);
+ break;
+ case BFQQE_NO_MORE_REQUESTS:
+ /*
+ * For queues that expire for this reason, it
+ * is particularly important to keep the
+ * budget close to the actual service they
+ * need. Doing so reduces the timestamp
+ * misalignment problem described in the
+ * comments in the body of
+ * __bfq_activate_entity. In fact, suppose
+ * that a queue systematically expires for
+ * BFQQE_NO_MORE_REQUESTS and presents a
+ * new request in time to enjoy timestamp
+ * back-shifting. The larger the budget of the
+ * queue is with respect to the service the
+ * queue actually requests in each service
+ * slot, the more times the queue can be
+ * reactivated with the same virtual finish
+ * time. It follows that, even if this finish
+ * time is pushed to the system virtual time
+ * to reduce the consequent timestamp
+ * misalignment, the queue unjustly enjoys for
+ * many re-activations a lower finish time
+ * than all newly activated queues.
+ *
+ * The service needed by bfqq is measured
+ * quite precisely by bfqq->entity.service.
+ * Since bfqq does not enjoy device idling,
+ * bfqq->entity.service is equal to the number
+ * of sectors that the process associated with
+ * bfqq requested to read/write before waiting
+ * for request completions, or blocking for
+ * other reasons.
+ */
+ budget = max_t(int, bfqq->entity.service, min_budget);
+ break;
+ default:
+ return;
+ }
+ } else if (!bfq_bfqq_sync(bfqq)) {
+ /*
+ * Async queues get always the maximum possible
+ * budget, as for them we do not care about latency
+ * (in addition, their ability to dispatch is limited
+ * by the charging factor).
+ */
+ budget = bfqd->bfq_max_budget;
+ }
+
+ bfqq->max_budget = budget;
+
+ if (bfqd->budgets_assigned >= bfq_stats_min_budgets &&
+ !bfqd->bfq_user_max_budget)
+ bfqq->max_budget = min(bfqq->max_budget, bfqd->bfq_max_budget);
+
+ /*
+ * If there is still backlog, then assign a new budget, making
+ * sure that it is large enough for the next request. Since
+ * the finish time of bfqq must be kept in sync with the
+ * budget, be sure to call __bfq_bfqq_expire() *after* this
+ * update.
+ *
+ * If there is no backlog, then no need to update the budget;
+ * it will be updated on the arrival of a new request.
+ */
+ next_rq = bfqq->next_rq;
+ if (next_rq)
+ bfqq->entity.budget = max_t(unsigned long, bfqq->max_budget,
+ bfq_serv_to_charge(next_rq, bfqq));
+
+ bfq_log_bfqq(bfqd, bfqq, "head sect: %u, new budget %d",
+ next_rq ? blk_rq_sectors(next_rq) : 0,
+ bfqq->entity.budget);
+}
+
+/*
+ * Return true if the process associated with bfqq is "slow". The slow
+ * flag is used, in addition to the budget timeout, to reduce the
+ * amount of service provided to seeky processes, and thus reduce
+ * their chances to lower the throughput. More details in the comments
+ * on the function bfq_bfqq_expire().
+ *
+ * An important observation is in order: as discussed in the comments
+ * on the function bfq_update_peak_rate(), with devices with internal
+ * queues, it is hard if ever possible to know when and for how long
+ * an I/O request is processed by the device (apart from the trivial
+ * I/O pattern where a new request is dispatched only after the
+ * previous one has been completed). This makes it hard to evaluate
+ * the real rate at which the I/O requests of each bfq_queue are
+ * served. In fact, for an I/O scheduler like BFQ, serving a
+ * bfq_queue means just dispatching its requests during its service
+ * slot (i.e., until the budget of the queue is exhausted, or the
+ * queue remains idle, or, finally, a timeout fires). But, during the
+ * service slot of a bfq_queue, around 100 ms at most, the device may
+ * be even still processing requests of bfq_queues served in previous
+ * service slots. On the opposite end, the requests of the in-service
+ * bfq_queue may be completed after the service slot of the queue
+ * finishes.
+ *
+ * Anyway, unless more sophisticated solutions are used
+ * (where possible), the sum of the sizes of the requests dispatched
+ * during the service slot of a bfq_queue is probably the only
+ * approximation available for the service received by the bfq_queue
+ * during its service slot. And this sum is the quantity used in this
+ * function to evaluate the I/O speed of a process.
+ */
+static bool bfq_bfqq_is_slow(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ bool compensate, enum bfqq_expiration reason,
+ unsigned long *delta_ms)
+{
+ ktime_t delta_ktime;
+ u32 delta_usecs;
+ bool slow = BFQQ_SEEKY(bfqq); /* if delta too short, use seekyness */
+
+ if (!bfq_bfqq_sync(bfqq))
+ return false;
+
+ if (compensate)
+ delta_ktime = bfqd->last_idling_start;
+ else
+ delta_ktime = ktime_get();
+ delta_ktime = ktime_sub(delta_ktime, bfqd->last_budget_start);
+ delta_usecs = ktime_to_us(delta_ktime);
+
+ /* don't use too short time intervals */
+ if (delta_usecs < 1000) {
+ if (blk_queue_nonrot(bfqd->queue))
+ /*
+ * give same worst-case guarantees as idling
+ * for seeky
+ */
+ *delta_ms = BFQ_MIN_TT / NSEC_PER_MSEC;
+ else /* charge at least one seek */
+ *delta_ms = bfq_slice_idle / NSEC_PER_MSEC;
+
+ return slow;
+ }
+
+ *delta_ms = delta_usecs / USEC_PER_MSEC;
+
+ /*
+ * Use only long (> 20ms) intervals to filter out excessive
+ * spikes in service rate estimation.
+ */
+ if (delta_usecs > 20000) {
+ /*
+ * Caveat for rotational devices: processes doing I/O
+ * in the slower disk zones tend to be slow(er) even
+ * if not seeky. In this respect, the estimated peak
+ * rate is likely to be an average over the disk
+ * surface. Accordingly, to not be too harsh with
+ * unlucky processes, a process is deemed slow only if
+ * its rate has been lower than half of the estimated
+ * peak rate.
+ */
+ slow = bfqq->entity.service < bfqd->bfq_max_budget / 2;
+ }
+
+ bfq_log_bfqq(bfqd, bfqq, "bfq_bfqq_is_slow: slow %d", slow);
+
+ return slow;
+}
+
+/*
+ * To be deemed as soft real-time, an application must meet two
+ * requirements. First, the application must not require an average
+ * bandwidth higher than the approximate bandwidth required to playback or
+ * record a compressed high-definition video.
+ * The next function is invoked on the completion of the last request of a
+ * batch, to compute the next-start time instant, soft_rt_next_start, such
+ * that, if the next request of the application does not arrive before
+ * soft_rt_next_start, then the above requirement on the bandwidth is met.
+ *
+ * The second requirement is that the request pattern of the application is
+ * isochronous, i.e., that, after issuing a request or a batch of requests,
+ * the application stops issuing new requests until all its pending requests
+ * have been completed. After that, the application may issue a new batch,
+ * and so on.
+ * For this reason the next function is invoked to compute
+ * soft_rt_next_start only for applications that meet this requirement,
+ * whereas soft_rt_next_start is set to infinity for applications that do
+ * not.
+ *
+ * Unfortunately, even a greedy application may happen to behave in an
+ * isochronous way if the CPU load is high. In fact, the application may
+ * stop issuing requests while the CPUs are busy serving other processes,
+ * then restart, then stop again for a while, and so on. In addition, if
+ * the disk achieves a low enough throughput with the request pattern
+ * issued by the application (e.g., because the request pattern is random
+ * and/or the device is slow), then the application may meet the above
+ * bandwidth requirement too. To prevent such a greedy application to be
+ * deemed as soft real-time, a further rule is used in the computation of
+ * soft_rt_next_start: soft_rt_next_start must be higher than the current
+ * time plus the maximum time for which the arrival of a request is waited
+ * for when a sync queue becomes idle, namely bfqd->bfq_slice_idle.
+ * This filters out greedy applications, as the latter issue instead their
+ * next request as soon as possible after the last one has been completed
+ * (in contrast, when a batch of requests is completed, a soft real-time
+ * application spends some time processing data).
+ *
+ * Unfortunately, the last filter may easily generate false positives if
+ * only bfqd->bfq_slice_idle is used as a reference time interval and one
+ * or both the following cases occur:
+ * 1) HZ is so low that the duration of a jiffy is comparable to or higher
+ * than bfqd->bfq_slice_idle. This happens, e.g., on slow devices with
+ * HZ=100.
+ * 2) jiffies, instead of increasing at a constant rate, may stop increasing
+ * for a while, then suddenly 'jump' by several units to recover the lost
+ * increments. This seems to happen, e.g., inside virtual machines.
+ * To address this issue, we do not use as a reference time interval just
+ * bfqd->bfq_slice_idle, but bfqd->bfq_slice_idle plus a few jiffies. In
+ * particular we add the minimum number of jiffies for which the filter
+ * seems to be quite precise also in embedded systems and KVM/QEMU virtual
+ * machines.
+ */
+static unsigned long bfq_bfqq_softrt_next_start(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq)
+{
+ return max(bfqq->last_idle_bklogged +
+ HZ * bfqq->service_from_backlogged /
+ bfqd->bfq_wr_max_softrt_rate,
+ jiffies + nsecs_to_jiffies(bfqq->bfqd->bfq_slice_idle) + 4);
+}
+
+/*
+ * Return the farthest future time instant according to jiffies
+ * macros.
+ */
+static unsigned long bfq_greatest_from_now(void)
+{
+ return jiffies + MAX_JIFFY_OFFSET;
+}
+
+/*
+ * Return the farthest past time instant according to jiffies
+ * macros.
+ */
+static unsigned long bfq_smallest_from_now(void)
+{
+ return jiffies - MAX_JIFFY_OFFSET;
+}
+
+/**
+ * bfq_bfqq_expire - expire a queue.
+ * @bfqd: device owning the queue.
+ * @bfqq: the queue to expire.
+ * @compensate: if true, compensate for the time spent idling.
+ * @reason: the reason causing the expiration.
+ *
+ * If the process associated with bfqq does slow I/O (e.g., because it
+ * issues random requests), we charge bfqq with the time it has been
+ * in service instead of the service it has received (see
+ * bfq_bfqq_charge_time for details on how this goal is achieved). As
+ * a consequence, bfqq will typically get higher timestamps upon
+ * reactivation, and hence it will be rescheduled as if it had
+ * received more service than what it has actually received. In the
+ * end, bfqq receives less service in proportion to how slowly its
+ * associated process consumes its budgets (and hence how seriously it
+ * tends to lower the throughput). In addition, this time-charging
+ * strategy guarantees time fairness among slow processes. In
+ * contrast, if the process associated with bfqq is not slow, we
+ * charge bfqq exactly with the service it has received.
+ *
+ * Charging time to the first type of queues and the exact service to
+ * the other has the effect of using the WF2Q+ policy to schedule the
+ * former on a timeslice basis, without violating service domain
+ * guarantees among the latter.
+ */
+void bfq_bfqq_expire(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq,
+ bool compensate,
+ enum bfqq_expiration reason)
+{
+ bool slow;
+ unsigned long delta = 0;
+ struct bfq_entity *entity = &bfqq->entity;
+ int ref;
+
+ /*
+ * Check whether the process is slow (see bfq_bfqq_is_slow).
+ */
+ slow = bfq_bfqq_is_slow(bfqd, bfqq, compensate, reason, &delta);
+
+ /*
+ * Increase service_from_backlogged before next statement,
+ * because the possible next invocation of
+ * bfq_bfqq_charge_time would likely inflate
+ * entity->service. In contrast, service_from_backlogged must
+ * contain real service, to enable the soft real-time
+ * heuristic to correctly compute the bandwidth consumed by
+ * bfqq.
+ */
+ bfqq->service_from_backlogged += entity->service;
+
+ /*
+ * As above explained, charge slow (typically seeky) and
+ * timed-out queues with the time and not the service
+ * received, to favor sequential workloads.
+ *
+ * Processes doing I/O in the slower disk zones will tend to
+ * be slow(er) even if not seeky. Therefore, since the
+ * estimated peak rate is actually an average over the disk
+ * surface, these processes may timeout just for bad luck. To
+ * avoid punishing them, do not charge time to processes that
+ * succeeded in consuming at least 2/3 of their budget. This
+ * allows BFQ to preserve enough elasticity to still perform
+ * bandwidth, and not time, distribution with little unlucky
+ * or quasi-sequential processes.
+ */
+ if (bfqq->wr_coeff == 1 &&
+ (slow ||
+ (reason == BFQQE_BUDGET_TIMEOUT &&
+ bfq_bfqq_budget_left(bfqq) >= entity->budget / 3)))
+ bfq_bfqq_charge_time(bfqd, bfqq, delta);
+
+ if (reason == BFQQE_TOO_IDLE &&
+ entity->service <= 2 * entity->budget / 10)
+ bfq_clear_bfqq_IO_bound(bfqq);
+
+ if (bfqd->low_latency && bfqq->wr_coeff == 1)
+ bfqq->last_wr_start_finish = jiffies;
+
+ if (bfqd->low_latency && bfqd->bfq_wr_max_softrt_rate > 0 &&
+ RB_EMPTY_ROOT(&bfqq->sort_list)) {
+ /*
+ * If we get here, and there are no outstanding
+ * requests, then the request pattern is isochronous
+ * (see the comments on the function
+ * bfq_bfqq_softrt_next_start()). Thus we can compute
+ * soft_rt_next_start. If, instead, the queue still
+ * has outstanding requests, then we have to wait for
+ * the completion of all the outstanding requests to
+ * discover whether the request pattern is actually
+ * isochronous.
+ */
+ if (bfqq->dispatched == 0)
+ bfqq->soft_rt_next_start =
+ bfq_bfqq_softrt_next_start(bfqd, bfqq);
+ else {
+ /*
+ * The application is still waiting for the
+ * completion of one or more requests:
+ * prevent it from possibly being incorrectly
+ * deemed as soft real-time by setting its
+ * soft_rt_next_start to infinity. In fact,
+ * without this assignment, the application
+ * would be incorrectly deemed as soft
+ * real-time if:
+ * 1) it issued a new request before the
+ * completion of all its in-flight
+ * requests, and
+ * 2) at that time, its soft_rt_next_start
+ * happened to be in the past.
+ */
+ bfqq->soft_rt_next_start =
+ bfq_greatest_from_now();
+ /*
+ * Schedule an update of soft_rt_next_start to when
+ * the task may be discovered to be isochronous.
+ */
+ bfq_mark_bfqq_softrt_update(bfqq);
+ }
+ }
+
+ bfq_log_bfqq(bfqd, bfqq,
+ "expire (%d, slow %d, num_disp %d, idle_win %d)", reason,
+ slow, bfqq->dispatched, bfq_bfqq_idle_window(bfqq));
+
+ /*
+ * Increase, decrease or leave budget unchanged according to
+ * reason.
+ */
+ __bfq_bfqq_recalc_budget(bfqd, bfqq, reason);
+ ref = bfqq->ref;
+ __bfq_bfqq_expire(bfqd, bfqq);
+
+ /* mark bfqq as waiting a request only if a bic still points to it */
+ if (ref > 1 && !bfq_bfqq_busy(bfqq) &&
+ reason != BFQQE_BUDGET_TIMEOUT &&
+ reason != BFQQE_BUDGET_EXHAUSTED)
+ bfq_mark_bfqq_non_blocking_wait_rq(bfqq);
+}
+
+/*
+ * Budget timeout is not implemented through a dedicated timer, but
+ * just checked on request arrivals and completions, as well as on
+ * idle timer expirations.
+ */
+static bool bfq_bfqq_budget_timeout(struct bfq_queue *bfqq)
+{
+ return time_is_before_eq_jiffies(bfqq->budget_timeout);
+}
+
+/*
+ * If we expire a queue that is actively waiting (i.e., with the
+ * device idled) for the arrival of a new request, then we may incur
+ * the timestamp misalignment problem described in the body of the
+ * function __bfq_activate_entity. Hence we return true only if this
+ * condition does not hold, or if the queue is slow enough to deserve
+ * only to be kicked off for preserving a high throughput.
+ */
+static bool bfq_may_expire_for_budg_timeout(struct bfq_queue *bfqq)
+{
+ bfq_log_bfqq(bfqq->bfqd, bfqq,
+ "may_budget_timeout: wait_request %d left %d timeout %d",
+ bfq_bfqq_wait_request(bfqq),
+ bfq_bfqq_budget_left(bfqq) >= bfqq->entity.budget / 3,
+ bfq_bfqq_budget_timeout(bfqq));
+
+ return (!bfq_bfqq_wait_request(bfqq) ||
+ bfq_bfqq_budget_left(bfqq) >= bfqq->entity.budget / 3)
+ &&
+ bfq_bfqq_budget_timeout(bfqq);
+}
+
+/*
+ * For a queue that becomes empty, device idling is allowed only if
+ * this function returns true for the queue. As a consequence, since
+ * device idling plays a critical role in both throughput boosting and
+ * service guarantees, the return value of this function plays a
+ * critical role in both these aspects as well.
+ *
+ * In a nutshell, this function returns true only if idling is
+ * beneficial for throughput or, even if detrimental for throughput,
+ * idling is however necessary to preserve service guarantees (low
+ * latency, desired throughput distribution, ...). In particular, on
+ * NCQ-capable devices, this function tries to return false, so as to
+ * help keep the drives' internal queues full, whenever this helps the
+ * device boost the throughput without causing any service-guarantee
+ * issue.
+ *
+ * In more detail, the return value of this function is obtained by,
+ * first, computing a number of boolean variables that take into
+ * account throughput and service-guarantee issues, and, then,
+ * combining these variables in a logical expression. Most of the
+ * issues taken into account are not trivial. We discuss these issues
+ * individually while introducing the variables.
+ */
+static bool bfq_bfqq_may_idle(struct bfq_queue *bfqq)
+{
+ struct bfq_data *bfqd = bfqq->bfqd;
+ bool idling_boosts_thr, idling_boosts_thr_without_issues,
+ idling_needed_for_service_guarantees,
+ asymmetric_scenario;
+
+ if (bfqd->strict_guarantees)
+ return true;
+
+ /*
+ * The next variable takes into account the cases where idling
+ * boosts the throughput.
+ *
+ * The value of the variable is computed considering, first, that
+ * idling is virtually always beneficial for the throughput if:
+ * (a) the device is not NCQ-capable, or
+ * (b) regardless of the presence of NCQ, the device is rotational
+ * and the request pattern for bfqq is I/O-bound and sequential.
+ *
+ * Secondly, and in contrast to the above item (b), idling an
+ * NCQ-capable flash-based device would not boost the
+ * throughput even with sequential I/O; rather it would lower
+ * the throughput in proportion to how fast the device
+ * is. Accordingly, the next variable is true if any of the
+ * above conditions (a) and (b) is true, and, in particular,
+ * happens to be false if bfqd is an NCQ-capable flash-based
+ * device.
+ */
+ idling_boosts_thr = !bfqd->hw_tag ||
+ (!blk_queue_nonrot(bfqd->queue) && bfq_bfqq_IO_bound(bfqq) &&
+ bfq_bfqq_idle_window(bfqq));
+
+ /*
+ * The value of the next variable,
+ * idling_boosts_thr_without_issues, is equal to that of
+ * idling_boosts_thr, unless a special case holds. In this
+ * special case, described below, idling may cause problems to
+ * weight-raised queues.
+ *
+ * When the request pool is saturated (e.g., in the presence
+ * of write hogs), if the processes associated with
+ * non-weight-raised queues ask for requests at a lower rate,
+ * then processes associated with weight-raised queues have a
+ * higher probability to get a request from the pool
+ * immediately (or at least soon) when they need one. Thus
+ * they have a higher probability to actually get a fraction
+ * of the device throughput proportional to their high
+ * weight. This is especially true with NCQ-capable drives,
+ * which enqueue several requests in advance, and further
+ * reorder internally-queued requests.
+ *
+ * For this reason, we force to false the value of
+ * idling_boosts_thr_without_issues if there are weight-raised
+ * busy queues. In this case, and if bfqq is not weight-raised,
+ * this guarantees that the device is not idled for bfqq (if,
+ * instead, bfqq is weight-raised, then idling will be
+ * guaranteed by another variable, see below). Combined with
+ * the timestamping rules of BFQ (see [1] for details), this
+ * behavior causes bfqq, and hence any sync non-weight-raised
+ * queue, to get a lower number of requests served, and thus
+ * to ask for a lower number of requests from the request
+ * pool, before the busy weight-raised queues get served
+ * again. This often mitigates starvation problems in the
+ * presence of heavy write workloads and NCQ, thereby
+ * guaranteeing a higher application and system responsiveness
+ * in these hostile scenarios.
+ */
+ idling_boosts_thr_without_issues = idling_boosts_thr &&
+ bfqd->wr_busy_queues == 0;
+
+ /*
+ * There is then a case where idling must be performed not
+ * for throughput concerns, but to preserve service
+ * guarantees.
+ *
+ * To introduce this case, we can note that allowing the drive
+ * to enqueue more than one request at a time, and hence
+ * delegating de facto final scheduling decisions to the
+ * drive's internal scheduler, entails loss of control on the
+ * actual request service order. In particular, the critical
+ * situation is when requests from different processes happen
+ * to be present, at the same time, in the internal queue(s)
+ * of the drive. In such a situation, the drive, by deciding
+ * the service order of the internally-queued requests, does
+ * determine also the actual throughput distribution among
+ * these processes. But the drive typically has no notion or
+ * concern about per-process throughput distribution, and
+ * makes its decisions only on a per-request basis. Therefore,
+ * the service distribution enforced by the drive's internal
+ * scheduler is likely to coincide with the desired
+ * device-throughput distribution only in a completely
+ * symmetric scenario where:
+ * (i) each of these processes must get the same throughput as
+ * the others;
+ * (ii) all these processes have the same I/O pattern
+ (either sequential or random).
+ * In fact, in such a scenario, the drive will tend to treat
+ * the requests of each of these processes in about the same
+ * way as the requests of the others, and thus to provide
+ * each of these processes with about the same throughput
+ * (which is exactly the desired throughput distribution). In
+ * contrast, in any asymmetric scenario, device idling is
+ * certainly needed to guarantee that bfqq receives its
+ * assigned fraction of the device throughput (see [1] for
+ * details).
+ *
+ * We address this issue by controlling, actually, only the
+ * symmetry sub-condition (i), i.e., provided that
+ * sub-condition (i) holds, idling is not performed,
+ * regardless of whether sub-condition (ii) holds. In other
+ * words, only if sub-condition (i) holds, then idling is
+ * allowed, and the device tends to be prevented from queueing
+ * many requests, possibly of several processes. The reason
+ * for not controlling also sub-condition (ii) is that we
+ * exploit preemption to preserve guarantees in case of
+ * symmetric scenarios, even if (ii) does not hold, as
+ * explained in the next two paragraphs.
+ *
+ * Even if a queue, say Q, is expired when it remains idle, Q
+ * can still preempt the new in-service queue if the next
+ * request of Q arrives soon (see the comments on
+ * bfq_bfqq_update_budg_for_activation). If all queues and
+ * groups have the same weight, this form of preemption,
+ * combined with the hole-recovery heuristic described in the
+ * comments on function bfq_bfqq_update_budg_for_activation,
+ * are enough to preserve a correct bandwidth distribution in
+ * the mid term, even without idling. In fact, even if not
+ * idling allows the internal queues of the device to contain
+ * many requests, and thus to reorder requests, we can rather
+ * safely assume that the internal scheduler still preserves a
+ * minimum of mid-term fairness. The motivation for using
+ * preemption instead of idling is that, by not idling,
+ * service guarantees are preserved without minimally
+ * sacrificing throughput. In other words, both a high
+ * throughput and its desired distribution are obtained.
+ *
+ * More precisely, this preemption-based, idleless approach
+ * provides fairness in terms of IOPS, and not sectors per
+ * second. This can be seen with a simple example. Suppose
+ * that there are two queues with the same weight, but that
+ * the first queue receives requests of 8 sectors, while the
+ * second queue receives requests of 1024 sectors. In
+ * addition, suppose that each of the two queues contains at
+ * most one request at a time, which implies that each queue
+ * always remains idle after it is served. Finally, after
+ * remaining idle, each queue receives very quickly a new
+ * request. It follows that the two queues are served
+ * alternatively, preempting each other if needed. This
+ * implies that, although both queues have the same weight,
+ * the queue with large requests receives a service that is
+ * 1024/8 times as high as the service received by the other
+ * queue.
+ *
+ * On the other hand, device idling is performed, and thus
+ * pure sector-domain guarantees are provided, for the
+ * following queues, which are likely to need stronger
+ * throughput guarantees: weight-raised queues, and queues
+ * with a higher weight than other queues. When such queues
+ * are active, sub-condition (i) is false, which triggers
+ * device idling.
+ *
+ * According to the above considerations, the next variable is
+ * true (only) if sub-condition (i) holds. To compute the
+ * value of this variable, we not only use the return value of
+ * the function bfq_symmetric_scenario(), but also check
+ * whether bfqq is being weight-raised, because
+ * bfq_symmetric_scenario() does not take into account also
+ * weight-raised queues (see comments on
+ * bfq_weights_tree_add()).
+ *
+ * As a side note, it is worth considering that the above
+ * device-idling countermeasures may however fail in the
+ * following unlucky scenario: if idling is (correctly)
+ * disabled in a time period during which all symmetry
+ * sub-conditions hold, and hence the device is allowed to
+ * enqueue many requests, but at some later point in time some
+ * sub-condition stops to hold, then it may become impossible
+ * to let requests be served in the desired order until all
+ * the requests already queued in the device have been served.
+ */
+ asymmetric_scenario = bfqq->wr_coeff > 1 ||
+ !bfq_symmetric_scenario(bfqd);
+
+ /*
+ * Finally, there is a case where maximizing throughput is the
+ * best choice even if it may cause unfairness toward
+ * bfqq. Such a case is when bfqq became active in a burst of
+ * queue activations. Queues that became active during a large
+ * burst benefit only from throughput, as discussed in the
+ * comments on bfq_handle_burst. Thus, if bfqq became active
+ * in a burst and not idling the device maximizes throughput,
+ * then the device must no be idled, because not idling the
+ * device provides bfqq and all other queues in the burst with
+ * maximum benefit. Combining this and the above case, we can
+ * now establish when idling is actually needed to preserve
+ * service guarantees.
+ */
+ idling_needed_for_service_guarantees =
+ asymmetric_scenario && !bfq_bfqq_in_large_burst(bfqq);
+
+ /*
+ * We have now all the components we need to compute the return
+ * value of the function, which is true only if both the following
+ * conditions hold:
+ * 1) bfqq is sync, because idling make sense only for sync queues;
+ * 2) idling either boosts the throughput (without issues), or
+ * is necessary to preserve service guarantees.
+ */
+ return bfq_bfqq_sync(bfqq) &&
+ (idling_boosts_thr_without_issues ||
+ idling_needed_for_service_guarantees);
+}
+
+/*
+ * If the in-service queue is empty but the function bfq_bfqq_may_idle
+ * returns true, then:
+ * 1) the queue must remain in service and cannot be expired, and
+ * 2) the device must be idled to wait for the possible arrival of a new
+ * request for the queue.
+ * See the comments on the function bfq_bfqq_may_idle for the reasons
+ * why performing device idling is the best choice to boost the throughput
+ * and preserve service guarantees when bfq_bfqq_may_idle itself
+ * returns true.
+ */
+static bool bfq_bfqq_must_idle(struct bfq_queue *bfqq)
+{
+ struct bfq_data *bfqd = bfqq->bfqd;
+
+ return RB_EMPTY_ROOT(&bfqq->sort_list) && bfqd->bfq_slice_idle != 0 &&
+ bfq_bfqq_may_idle(bfqq);
+}
+
+/*
+ * Select a queue for service. If we have a current queue in service,
+ * check whether to continue servicing it, or retrieve and set a new one.
+ */
+static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd)
+{
+ struct bfq_queue *bfqq;
+ struct request *next_rq;
+ enum bfqq_expiration reason = BFQQE_BUDGET_TIMEOUT;
+
+ bfqq = bfqd->in_service_queue;
+ if (!bfqq)
+ goto new_queue;
+
+ bfq_log_bfqq(bfqd, bfqq, "select_queue: already in-service queue");
+
+ if (bfq_may_expire_for_budg_timeout(bfqq) &&
+ !bfq_bfqq_wait_request(bfqq) &&
+ !bfq_bfqq_must_idle(bfqq))
+ goto expire;
+
+check_queue:
+ /*
+ * This loop is rarely executed more than once. Even when it
+ * happens, it is much more convenient to re-execute this loop
+ * than to return NULL and trigger a new dispatch to get a
+ * request served.
+ */
+ next_rq = bfqq->next_rq;
+ /*
+ * If bfqq has requests queued and it has enough budget left to
+ * serve them, keep the queue, otherwise expire it.
+ */
+ if (next_rq) {
+ if (bfq_serv_to_charge(next_rq, bfqq) >
+ bfq_bfqq_budget_left(bfqq)) {
+ /*
+ * Expire the queue for budget exhaustion,
+ * which makes sure that the next budget is
+ * enough to serve the next request, even if
+ * it comes from the fifo expired path.
+ */
+ reason = BFQQE_BUDGET_EXHAUSTED;
+ goto expire;
+ } else {
+ /*
+ * The idle timer may be pending because we may
+ * not disable disk idling even when a new request
+ * arrives.
+ */
+ if (bfq_bfqq_wait_request(bfqq)) {
+ /*
+ * If we get here: 1) at least a new request
+ * has arrived but we have not disabled the
+ * timer because the request was too small,
+ * 2) then the block layer has unplugged
+ * the device, causing the dispatch to be
+ * invoked.
+ *
+ * Since the device is unplugged, now the
+ * requests are probably large enough to
+ * provide a reasonable throughput.
+ * So we disable idling.
+ */
+ bfq_clear_bfqq_wait_request(bfqq);
+ hrtimer_try_to_cancel(&bfqd->idle_slice_timer);
+ bfqg_stats_update_idle_time(bfqq_group(bfqq));
+ }
+ goto keep_queue;
+ }
+ }
+
+ /*
+ * No requests pending. However, if the in-service queue is idling
+ * for a new request, or has requests waiting for a completion and
+ * may idle after their completion, then keep it anyway.
+ */
+ if (bfq_bfqq_wait_request(bfqq) ||
+ (bfqq->dispatched != 0 && bfq_bfqq_may_idle(bfqq))) {
+ bfqq = NULL;
+ goto keep_queue;
+ }
+
+ reason = BFQQE_NO_MORE_REQUESTS;
+expire:
+ bfq_bfqq_expire(bfqd, bfqq, false, reason);
+new_queue:
+ bfqq = bfq_set_in_service_queue(bfqd);
+ if (bfqq) {
+ bfq_log_bfqq(bfqd, bfqq, "select_queue: checking new queue");
+ goto check_queue;
+ }
+keep_queue:
+ if (bfqq)
+ bfq_log_bfqq(bfqd, bfqq, "select_queue: returned this queue");
+ else
+ bfq_log(bfqd, "select_queue: no queue returned");
+
+ return bfqq;
+}
+
+static void bfq_update_wr_data(struct bfq_data *bfqd, struct bfq_queue *bfqq)
+{
+ struct bfq_entity *entity = &bfqq->entity;
+
+ if (bfqq->wr_coeff > 1) { /* queue is being weight-raised */
+ bfq_log_bfqq(bfqd, bfqq,
+ "raising period dur %u/%u msec, old coeff %u, w %d(%d)",
+ jiffies_to_msecs(jiffies - bfqq->last_wr_start_finish),
+ jiffies_to_msecs(bfqq->wr_cur_max_time),
+ bfqq->wr_coeff,
+ bfqq->entity.weight, bfqq->entity.orig_weight);
+
+ if (entity->prio_changed)
+ bfq_log_bfqq(bfqd, bfqq, "WARN: pending prio change");
+
+ /*
+ * If the queue was activated in a burst, or too much
+ * time has elapsed from the beginning of this
+ * weight-raising period, then end weight raising.
+ */
+ if (bfq_bfqq_in_large_burst(bfqq))
+ bfq_bfqq_end_wr(bfqq);
+ else if (time_is_before_jiffies(bfqq->last_wr_start_finish +
+ bfqq->wr_cur_max_time)) {
+ if (bfqq->wr_cur_max_time != bfqd->bfq_wr_rt_max_time ||
+ time_is_before_jiffies(bfqq->wr_start_at_switch_to_srt +
+ bfq_wr_duration(bfqd)))
+ bfq_bfqq_end_wr(bfqq);
+ else {
+ /* switch back to interactive wr */
+ bfqq->wr_coeff = bfqd->bfq_wr_coeff;
+ bfqq->wr_cur_max_time = bfq_wr_duration(bfqd);
+ bfqq->last_wr_start_finish =
+ bfqq->wr_start_at_switch_to_srt;
+ bfqq->entity.prio_changed = 1;
+ }
+ }
+ }
+ /* Update weight both if it must be raised and if it must be lowered */
+ if ((entity->weight > entity->orig_weight) != (bfqq->wr_coeff > 1))
+ __bfq_entity_update_weight_prio(
+ bfq_entity_service_tree(entity),
+ entity);
+}
+
+/*
+ * Dispatch next request from bfqq.
+ */
+static struct request *bfq_dispatch_rq_from_bfqq(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq)
+{
+ struct request *rq = bfqq->next_rq;
+ unsigned long service_to_charge;
+
+ service_to_charge = bfq_serv_to_charge(rq, bfqq);
+
+ bfq_bfqq_served(bfqq, service_to_charge);
+
+ bfq_dispatch_remove(bfqd->queue, rq);
+
+ /*
+ * If weight raising has to terminate for bfqq, then next
+ * function causes an immediate update of bfqq's weight,
+ * without waiting for next activation. As a consequence, on
+ * expiration, bfqq will be timestamped as if has never been
+ * weight-raised during this service slot, even if it has
+ * received part or even most of the service as a
+ * weight-raised queue. This inflates bfqq's timestamps, which
+ * is beneficial, as bfqq is then more willing to leave the
+ * device immediately to possible other weight-raised queues.
+ */
+ bfq_update_wr_data(bfqd, bfqq);
+
+ /*
+ * Expire bfqq, pretending that its budget expired, if bfqq
+ * belongs to CLASS_IDLE and other queues are waiting for
+ * service.
+ */
+ if (bfqd->busy_queues > 1 && bfq_class_idle(bfqq))
+ goto expire;
+
+ return rq;
+
+expire:
+ bfq_bfqq_expire(bfqd, bfqq, false, BFQQE_BUDGET_EXHAUSTED);
+ return rq;
+}
+
+static bool bfq_has_work(struct blk_mq_hw_ctx *hctx)
+{
+ struct bfq_data *bfqd = hctx->queue->elevator->elevator_data;
+
+ /*
+ * Avoiding lock: a race on bfqd->busy_queues should cause at
+ * most a call to dispatch for nothing
+ */
+ return !list_empty_careful(&bfqd->dispatch) ||
+ bfqd->busy_queues > 0;
+}
+
+static struct request *__bfq_dispatch_request(struct blk_mq_hw_ctx *hctx)
+{
+ struct bfq_data *bfqd = hctx->queue->elevator->elevator_data;
+ struct request *rq = NULL;
+ struct bfq_queue *bfqq = NULL;
+
+ if (!list_empty(&bfqd->dispatch)) {
+ rq = list_first_entry(&bfqd->dispatch, struct request,
+ queuelist);
+ list_del_init(&rq->queuelist);
+
+ bfqq = RQ_BFQQ(rq);
+
+ if (bfqq) {
+ /*
+ * Increment counters here, because this
+ * dispatch does not follow the standard
+ * dispatch flow (where counters are
+ * incremented)
+ */
+ bfqq->dispatched++;
+
+ goto inc_in_driver_start_rq;
+ }
+
+ /*
+ * We exploit the put_rq_private hook to decrement
+ * rq_in_driver, but put_rq_private will not be
+ * invoked on this request. So, to avoid unbalance,
+ * just start this request, without incrementing
+ * rq_in_driver. As a negative consequence,
+ * rq_in_driver is deceptively lower than it should be
+ * while this request is in service. This may cause
+ * bfq_schedule_dispatch to be invoked uselessly.
+ *
+ * As for implementing an exact solution, the
+ * put_request hook, if defined, is probably invoked
+ * also on this request. So, by exploiting this hook,
+ * we could 1) increment rq_in_driver here, and 2)
+ * decrement it in put_request. Such a solution would
+ * let the value of the counter be always accurate,
+ * but it would entail using an extra interface
+ * function. This cost seems higher than the benefit,
+ * being the frequency of non-elevator-private
+ * requests very low.
+ */
+ goto start_rq;
+ }
+
+ bfq_log(bfqd, "dispatch requests: %d busy queues", bfqd->busy_queues);
+
+ if (bfqd->busy_queues == 0)
+ goto exit;
+
+ /*
+ * Force device to serve one request at a time if
+ * strict_guarantees is true. Forcing this service scheme is
+ * currently the ONLY way to guarantee that the request
+ * service order enforced by the scheduler is respected by a
+ * queueing device. Otherwise the device is free even to make
+ * some unlucky request wait for as long as the device
+ * wishes.
+ *
+ * Of course, serving one request at at time may cause loss of
+ * throughput.
+ */
+ if (bfqd->strict_guarantees && bfqd->rq_in_driver > 0)
+ goto exit;
+
+ bfqq = bfq_select_queue(bfqd);
+ if (!bfqq)
+ goto exit;
+
+ rq = bfq_dispatch_rq_from_bfqq(bfqd, bfqq);
+
+ if (rq) {
+inc_in_driver_start_rq:
+ bfqd->rq_in_driver++;
+start_rq:
+ rq->rq_flags |= RQF_STARTED;
+ }
+exit:
+ return rq;
+}
+
+static struct request *bfq_dispatch_request(struct blk_mq_hw_ctx *hctx)
+{
+ struct bfq_data *bfqd = hctx->queue->elevator->elevator_data;
+ struct request *rq;
+
+ spin_lock_irq(&bfqd->lock);
+
+ rq = __bfq_dispatch_request(hctx);
+ spin_unlock_irq(&bfqd->lock);
+
+ return rq;
+}
+
+/*
+ * Task holds one reference to the queue, dropped when task exits. Each rq
+ * in-flight on this queue also holds a reference, dropped when rq is freed.
+ *
+ * Scheduler lock must be held here. Recall not to use bfqq after calling
+ * this function on it.
+ */
+void bfq_put_queue(struct bfq_queue *bfqq)
+{
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ struct bfq_group *bfqg = bfqq_group(bfqq);
+#endif
+
+ if (bfqq->bfqd)
+ bfq_log_bfqq(bfqq->bfqd, bfqq, "put_queue: %p %d",
+ bfqq, bfqq->ref);
+
+ bfqq->ref--;
+ if (bfqq->ref)
+ return;
+
+ if (bfq_bfqq_sync(bfqq))
+ /*
+ * The fact that this queue is being destroyed does not
+ * invalidate the fact that this queue may have been
+ * activated during the current burst. As a consequence,
+ * although the queue does not exist anymore, and hence
+ * needs to be removed from the burst list if there,
+ * the burst size has not to be decremented.
+ */
+ hlist_del_init(&bfqq->burst_list_node);
+
+ kmem_cache_free(bfq_pool, bfqq);
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ bfqg_put(bfqg);
+#endif
+}
+
+static void bfq_put_cooperator(struct bfq_queue *bfqq)
+{
+ struct bfq_queue *__bfqq, *next;
+
+ /*
+ * If this queue was scheduled to merge with another queue, be
+ * sure to drop the reference taken on that queue (and others in
+ * the merge chain). See bfq_setup_merge and bfq_merge_bfqqs.
+ */
+ __bfqq = bfqq->new_bfqq;
+ while (__bfqq) {
+ if (__bfqq == bfqq)
+ break;
+ next = __bfqq->new_bfqq;
+ bfq_put_queue(__bfqq);
+ __bfqq = next;
+ }
+}
+
+static void bfq_exit_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq)
+{
+ if (bfqq == bfqd->in_service_queue) {
+ __bfq_bfqq_expire(bfqd, bfqq);
+ bfq_schedule_dispatch(bfqd);
+ }
+
+ bfq_log_bfqq(bfqd, bfqq, "exit_bfqq: %p, %d", bfqq, bfqq->ref);
+
+ bfq_put_cooperator(bfqq);
+
+ bfq_put_queue(bfqq); /* release process reference */
+}
+
+static void bfq_exit_icq_bfqq(struct bfq_io_cq *bic, bool is_sync)
+{
+ struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync);
+ struct bfq_data *bfqd;
+
+ if (bfqq)
+ bfqd = bfqq->bfqd; /* NULL if scheduler already exited */
+
+ if (bfqq && bfqd) {
+ unsigned long flags;
+
+ spin_lock_irqsave(&bfqd->lock, flags);
+ bfq_exit_bfqq(bfqd, bfqq);
+ bic_set_bfqq(bic, NULL, is_sync);
+ spin_unlock_irqrestore(&bfqd->lock, flags);
+ }
+}
+
+static void bfq_exit_icq(struct io_cq *icq)
+{
+ struct bfq_io_cq *bic = icq_to_bic(icq);
+
+ bfq_exit_icq_bfqq(bic, true);
+ bfq_exit_icq_bfqq(bic, false);
+}
+
+/*
+ * Update the entity prio values; note that the new values will not
+ * be used until the next (re)activation.
+ */
+static void
+bfq_set_next_ioprio_data(struct bfq_queue *bfqq, struct bfq_io_cq *bic)
+{
+ struct task_struct *tsk = current;
+ int ioprio_class;
+ struct bfq_data *bfqd = bfqq->bfqd;
+
+ if (!bfqd)
+ return;
+
+ ioprio_class = IOPRIO_PRIO_CLASS(bic->ioprio);
+ switch (ioprio_class) {
+ default:
+ dev_err(bfqq->bfqd->queue->backing_dev_info->dev,
+ "bfq: bad prio class %d\n", ioprio_class);
+ case IOPRIO_CLASS_NONE:
+ /*
+ * No prio set, inherit CPU scheduling settings.
+ */
+ bfqq->new_ioprio = task_nice_ioprio(tsk);
+ bfqq->new_ioprio_class = task_nice_ioclass(tsk);
+ break;
+ case IOPRIO_CLASS_RT:
+ bfqq->new_ioprio = IOPRIO_PRIO_DATA(bic->ioprio);
+ bfqq->new_ioprio_class = IOPRIO_CLASS_RT;
+ break;
+ case IOPRIO_CLASS_BE:
+ bfqq->new_ioprio = IOPRIO_PRIO_DATA(bic->ioprio);
+ bfqq->new_ioprio_class = IOPRIO_CLASS_BE;
+ break;
+ case IOPRIO_CLASS_IDLE:
+ bfqq->new_ioprio_class = IOPRIO_CLASS_IDLE;
+ bfqq->new_ioprio = 7;
+ bfq_clear_bfqq_idle_window(bfqq);
+ break;
+ }
+
+ if (bfqq->new_ioprio >= IOPRIO_BE_NR) {
+ pr_crit("bfq_set_next_ioprio_data: new_ioprio %d\n",
+ bfqq->new_ioprio);
+ bfqq->new_ioprio = IOPRIO_BE_NR;
+ }
+
+ bfqq->entity.new_weight = bfq_ioprio_to_weight(bfqq->new_ioprio);
+ bfqq->entity.prio_changed = 1;
+}
+
+static struct bfq_queue *bfq_get_queue(struct bfq_data *bfqd,
+ struct bio *bio, bool is_sync,
+ struct bfq_io_cq *bic);
+
+static void bfq_check_ioprio_change(struct bfq_io_cq *bic, struct bio *bio)
+{
+ struct bfq_data *bfqd = bic_to_bfqd(bic);
+ struct bfq_queue *bfqq;
+ int ioprio = bic->icq.ioc->ioprio;
+
+ /*
+ * This condition may trigger on a newly created bic, be sure to
+ * drop the lock before returning.
+ */
+ if (unlikely(!bfqd) || likely(bic->ioprio == ioprio))
+ return;
+
+ bic->ioprio = ioprio;
+
+ bfqq = bic_to_bfqq(bic, false);
+ if (bfqq) {
+ /* release process reference on this queue */
+ bfq_put_queue(bfqq);
+ bfqq = bfq_get_queue(bfqd, bio, BLK_RW_ASYNC, bic);
+ bic_set_bfqq(bic, bfqq, false);
+ }
+
+ bfqq = bic_to_bfqq(bic, true);
+ if (bfqq)
+ bfq_set_next_ioprio_data(bfqq, bic);
+}
+
+static void bfq_init_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ struct bfq_io_cq *bic, pid_t pid, int is_sync)
+{
+ RB_CLEAR_NODE(&bfqq->entity.rb_node);
+ INIT_LIST_HEAD(&bfqq->fifo);
+ INIT_HLIST_NODE(&bfqq->burst_list_node);
+
+ bfqq->ref = 0;
+ bfqq->bfqd = bfqd;
+
+ if (bic)
+ bfq_set_next_ioprio_data(bfqq, bic);
+
+ if (is_sync) {
+ if (!bfq_class_idle(bfqq))
+ bfq_mark_bfqq_idle_window(bfqq);
+ bfq_mark_bfqq_sync(bfqq);
+ bfq_mark_bfqq_just_created(bfqq);
+ } else
+ bfq_clear_bfqq_sync(bfqq);
+
+ /* set end request to minus infinity from now */
+ bfqq->ttime.last_end_request = ktime_get_ns() + 1;
+
+ bfq_mark_bfqq_IO_bound(bfqq);
+
+ bfqq->pid = pid;
+
+ /* Tentative initial value to trade off between thr and lat */
+ bfqq->max_budget = (2 * bfq_max_budget(bfqd)) / 3;
+ bfqq->budget_timeout = bfq_smallest_from_now();
+
+ bfqq->wr_coeff = 1;
+ bfqq->last_wr_start_finish = jiffies;
+ bfqq->wr_start_at_switch_to_srt = bfq_smallest_from_now();
+ bfqq->split_time = bfq_smallest_from_now();
+
+ /*
+ * Set to the value for which bfqq will not be deemed as
+ * soft rt when it becomes backlogged.
+ */
+ bfqq->soft_rt_next_start = bfq_greatest_from_now();
+
+ /* first request is almost certainly seeky */
+ bfqq->seek_history = 1;
+}
+
+static struct bfq_queue **bfq_async_queue_prio(struct bfq_data *bfqd,
+ struct bfq_group *bfqg,
+ int ioprio_class, int ioprio)
+{
+ switch (ioprio_class) {
+ case IOPRIO_CLASS_RT:
+ return &bfqg->async_bfqq[0][ioprio];
+ case IOPRIO_CLASS_NONE:
+ ioprio = IOPRIO_NORM;
+ /* fall through */
+ case IOPRIO_CLASS_BE:
+ return &bfqg->async_bfqq[1][ioprio];
+ case IOPRIO_CLASS_IDLE:
+ return &bfqg->async_idle_bfqq;
+ default:
+ return NULL;
+ }
+}
+
+static struct bfq_queue *bfq_get_queue(struct bfq_data *bfqd,
+ struct bio *bio, bool is_sync,
+ struct bfq_io_cq *bic)
+{
+ const int ioprio = IOPRIO_PRIO_DATA(bic->ioprio);
+ const int ioprio_class = IOPRIO_PRIO_CLASS(bic->ioprio);
+ struct bfq_queue **async_bfqq = NULL;
+ struct bfq_queue *bfqq;
+ struct bfq_group *bfqg;
+
+ rcu_read_lock();
+
+ bfqg = bfq_find_set_group(bfqd, bio_blkcg(bio));
+ if (!bfqg) {
+ bfqq = &bfqd->oom_bfqq;
+ goto out;
+ }
+
+ if (!is_sync) {
+ async_bfqq = bfq_async_queue_prio(bfqd, bfqg, ioprio_class,
+ ioprio);
+ bfqq = *async_bfqq;
+ if (bfqq)
+ goto out;
+ }
+
+ bfqq = kmem_cache_alloc_node(bfq_pool,
+ GFP_NOWAIT | __GFP_ZERO | __GFP_NOWARN,
+ bfqd->queue->node);
+
+ if (bfqq) {
+ bfq_init_bfqq(bfqd, bfqq, bic, current->pid,
+ is_sync);
+ bfq_init_entity(&bfqq->entity, bfqg);
+ bfq_log_bfqq(bfqd, bfqq, "allocated");
+ } else {
+ bfqq = &bfqd->oom_bfqq;
+ bfq_log_bfqq(bfqd, bfqq, "using oom bfqq");
+ goto out;
+ }
+
+ /*
+ * Pin the queue now that it's allocated, scheduler exit will
+ * prune it.
+ */
+ if (async_bfqq) {
+ bfqq->ref++; /*
+ * Extra group reference, w.r.t. sync
+ * queue. This extra reference is removed
+ * only if bfqq->bfqg disappears, to
+ * guarantee that this queue is not freed
+ * until its group goes away.
+ */
+ bfq_log_bfqq(bfqd, bfqq, "get_queue, bfqq not in async: %p, %d",
+ bfqq, bfqq->ref);
+ *async_bfqq = bfqq;
+ }
+
+out:
+ bfqq->ref++; /* get a process reference to this queue */
+ bfq_log_bfqq(bfqd, bfqq, "get_queue, at end: %p, %d", bfqq, bfqq->ref);
+ rcu_read_unlock();
+ return bfqq;
+}
+
+static void bfq_update_io_thinktime(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq)
+{
+ struct bfq_ttime *ttime = &bfqq->ttime;
+ u64 elapsed = ktime_get_ns() - bfqq->ttime.last_end_request;
+
+ elapsed = min_t(u64, elapsed, 2ULL * bfqd->bfq_slice_idle);
+
+ ttime->ttime_samples = (7*bfqq->ttime.ttime_samples + 256) / 8;
+ ttime->ttime_total = div_u64(7*ttime->ttime_total + 256*elapsed, 8);
+ ttime->ttime_mean = div64_ul(ttime->ttime_total + 128,
+ ttime->ttime_samples);
+}
+
+static void
+bfq_update_io_seektime(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ struct request *rq)
+{
+ bfqq->seek_history <<= 1;
+ bfqq->seek_history |=
+ get_sdist(bfqq->last_request_pos, rq) > BFQQ_SEEK_THR &&
+ (!blk_queue_nonrot(bfqd->queue) ||
+ blk_rq_sectors(rq) < BFQQ_SECT_THR_NONROT);
+}
+
+/*
+ * Disable idle window if the process thinks too long or seeks so much that
+ * it doesn't matter.
+ */
+static void bfq_update_idle_window(struct bfq_data *bfqd,
+ struct bfq_queue *bfqq,
+ struct bfq_io_cq *bic)
+{
+ int enable_idle;
+
+ /* Don't idle for async or idle io prio class. */
+ if (!bfq_bfqq_sync(bfqq) || bfq_class_idle(bfqq))
+ return;
+
+ /* Idle window just restored, statistics are meaningless. */
+ if (time_is_after_eq_jiffies(bfqq->split_time +
+ bfqd->bfq_wr_min_idle_time))
+ return;
+
+ enable_idle = bfq_bfqq_idle_window(bfqq);
+
+ if (atomic_read(&bic->icq.ioc->active_ref) == 0 ||
+ bfqd->bfq_slice_idle == 0 ||
+ (bfqd->hw_tag && BFQQ_SEEKY(bfqq) &&
+ bfqq->wr_coeff == 1))
+ enable_idle = 0;
+ else if (bfq_sample_valid(bfqq->ttime.ttime_samples)) {
+ if (bfqq->ttime.ttime_mean > bfqd->bfq_slice_idle &&
+ bfqq->wr_coeff == 1)
+ enable_idle = 0;
+ else
+ enable_idle = 1;
+ }
+ bfq_log_bfqq(bfqd, bfqq, "update_idle_window: enable_idle %d",
+ enable_idle);
+
+ if (enable_idle)
+ bfq_mark_bfqq_idle_window(bfqq);
+ else
+ bfq_clear_bfqq_idle_window(bfqq);
+}
+
+/*
+ * Called when a new fs request (rq) is added to bfqq. Check if there's
+ * something we should do about it.
+ */
+static void bfq_rq_enqueued(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ struct request *rq)
+{
+ struct bfq_io_cq *bic = RQ_BIC(rq);
+
+ if (rq->cmd_flags & REQ_META)
+ bfqq->meta_pending++;
+
+ bfq_update_io_thinktime(bfqd, bfqq);
+ bfq_update_io_seektime(bfqd, bfqq, rq);
+ if (bfqq->entity.service > bfq_max_budget(bfqd) / 8 ||
+ !BFQQ_SEEKY(bfqq))
+ bfq_update_idle_window(bfqd, bfqq, bic);
+
+ bfq_log_bfqq(bfqd, bfqq,
+ "rq_enqueued: idle_window=%d (seeky %d)",
+ bfq_bfqq_idle_window(bfqq), BFQQ_SEEKY(bfqq));
+
+ bfqq->last_request_pos = blk_rq_pos(rq) + blk_rq_sectors(rq);
+
+ if (bfqq == bfqd->in_service_queue && bfq_bfqq_wait_request(bfqq)) {
+ bool small_req = bfqq->queued[rq_is_sync(rq)] == 1 &&
+ blk_rq_sectors(rq) < 32;
+ bool budget_timeout = bfq_bfqq_budget_timeout(bfqq);
+
+ /*
+ * There is just this request queued: if the request
+ * is small and the queue is not to be expired, then
+ * just exit.
+ *
+ * In this way, if the device is being idled to wait
+ * for a new request from the in-service queue, we
+ * avoid unplugging the device and committing the
+ * device to serve just a small request. On the
+ * contrary, we wait for the block layer to decide
+ * when to unplug the device: hopefully, new requests
+ * will be merged to this one quickly, then the device
+ * will be unplugged and larger requests will be
+ * dispatched.
+ */
+ if (small_req && !budget_timeout)
+ return;
+
+ /*
+ * A large enough request arrived, or the queue is to
+ * be expired: in both cases disk idling is to be
+ * stopped, so clear wait_request flag and reset
+ * timer.
+ */
+ bfq_clear_bfqq_wait_request(bfqq);
+ hrtimer_try_to_cancel(&bfqd->idle_slice_timer);
+ bfqg_stats_update_idle_time(bfqq_group(bfqq));
+
+ /*
+ * The queue is not empty, because a new request just
+ * arrived. Hence we can safely expire the queue, in
+ * case of budget timeout, without risking that the
+ * timestamps of the queue are not updated correctly.
+ * See [1] for more details.
+ */
+ if (budget_timeout)
+ bfq_bfqq_expire(bfqd, bfqq, false,
+ BFQQE_BUDGET_TIMEOUT);
+ }
+}
+
+static void __bfq_insert_request(struct bfq_data *bfqd, struct request *rq)
+{
+ struct bfq_queue *bfqq = RQ_BFQQ(rq),
+ *new_bfqq = bfq_setup_cooperator(bfqd, bfqq, rq, true);
+
+ if (new_bfqq) {
+ if (bic_to_bfqq(RQ_BIC(rq), 1) != bfqq)
+ new_bfqq = bic_to_bfqq(RQ_BIC(rq), 1);
+ /*
+ * Release the request's reference to the old bfqq
+ * and make sure one is taken to the shared queue.
+ */
+ new_bfqq->allocated++;
+ bfqq->allocated--;
+ new_bfqq->ref++;
+ bfq_clear_bfqq_just_created(bfqq);
+ /*
+ * If the bic associated with the process
+ * issuing this request still points to bfqq
+ * (and thus has not been already redirected
+ * to new_bfqq or even some other bfq_queue),
+ * then complete the merge and redirect it to
+ * new_bfqq.
+ */
+ if (bic_to_bfqq(RQ_BIC(rq), 1) == bfqq)
+ bfq_merge_bfqqs(bfqd, RQ_BIC(rq),
+ bfqq, new_bfqq);
+ /*
+ * rq is about to be enqueued into new_bfqq,
+ * release rq reference on bfqq
+ */
+ bfq_put_queue(bfqq);
+ rq->elv.priv[1] = new_bfqq;
+ bfqq = new_bfqq;
+ }
+
+ bfq_add_request(rq);
+
+ rq->fifo_time = ktime_get_ns() + bfqd->bfq_fifo_expire[rq_is_sync(rq)];
+ list_add_tail(&rq->queuelist, &bfqq->fifo);
+
+ bfq_rq_enqueued(bfqd, bfqq, rq);
+}
+
+static void bfq_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,
+ bool at_head)
+{
+ struct request_queue *q = hctx->queue;
+ struct bfq_data *bfqd = q->elevator->elevator_data;
+
+ spin_lock_irq(&bfqd->lock);
+ if (blk_mq_sched_try_insert_merge(q, rq)) {
+ spin_unlock_irq(&bfqd->lock);
+ return;
+ }
+
+ spin_unlock_irq(&bfqd->lock);
+
+ blk_mq_sched_request_inserted(rq);
+
+ spin_lock_irq(&bfqd->lock);
+ if (at_head || blk_rq_is_passthrough(rq)) {
+ if (at_head)
+ list_add(&rq->queuelist, &bfqd->dispatch);
+ else
+ list_add_tail(&rq->queuelist, &bfqd->dispatch);
+ } else {
+ __bfq_insert_request(bfqd, rq);
+
+ if (rq_mergeable(rq)) {
+ elv_rqhash_add(q, rq);
+ if (!q->last_merge)
+ q->last_merge = rq;
+ }
+ }
+
+ spin_unlock_irq(&bfqd->lock);
+}
+
+static void bfq_insert_requests(struct blk_mq_hw_ctx *hctx,
+ struct list_head *list, bool at_head)
+{
+ while (!list_empty(list)) {
+ struct request *rq;
+
+ rq = list_first_entry(list, struct request, queuelist);
+ list_del_init(&rq->queuelist);
+ bfq_insert_request(hctx, rq, at_head);
+ }
+}
+
+static void bfq_update_hw_tag(struct bfq_data *bfqd)
+{
+ bfqd->max_rq_in_driver = max_t(int, bfqd->max_rq_in_driver,
+ bfqd->rq_in_driver);
+
+ if (bfqd->hw_tag == 1)
+ return;
+
+ /*
+ * This sample is valid if the number of outstanding requests
+ * is large enough to allow a queueing behavior. Note that the
+ * sum is not exact, as it's not taking into account deactivated
+ * requests.
+ */
+ if (bfqd->rq_in_driver + bfqd->queued < BFQ_HW_QUEUE_THRESHOLD)
+ return;
+
+ if (bfqd->hw_tag_samples++ < BFQ_HW_QUEUE_SAMPLES)
+ return;
+
+ bfqd->hw_tag = bfqd->max_rq_in_driver > BFQ_HW_QUEUE_THRESHOLD;
+ bfqd->max_rq_in_driver = 0;
+ bfqd->hw_tag_samples = 0;
+}
+
+static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
+{
+ u64 now_ns;
+ u32 delta_us;
+
+ bfq_update_hw_tag(bfqd);
+
+ bfqd->rq_in_driver--;
+ bfqq->dispatched--;
+
+ if (!bfqq->dispatched && !bfq_bfqq_busy(bfqq)) {
+ /*
+ * Set budget_timeout (which we overload to store the
+ * time at which the queue remains with no backlog and
+ * no outstanding request; used by the weight-raising
+ * mechanism).
+ */
+ bfqq->budget_timeout = jiffies;
+
+ bfq_weights_tree_remove(bfqd, &bfqq->entity,
+ &bfqd->queue_weights_tree);
+ }
+
+ now_ns = ktime_get_ns();
+
+ bfqq->ttime.last_end_request = now_ns;
+
+ /*
+ * Using us instead of ns, to get a reasonable precision in
+ * computing rate in next check.
+ */
+ delta_us = div_u64(now_ns - bfqd->last_completion, NSEC_PER_USEC);
+
+ /*
+ * If the request took rather long to complete, and, according
+ * to the maximum request size recorded, this completion latency
+ * implies that the request was certainly served at a very low
+ * rate (less than 1M sectors/sec), then the whole observation
+ * interval that lasts up to this time instant cannot be a
+ * valid time interval for computing a new peak rate. Invoke
+ * bfq_update_rate_reset to have the following three steps
+ * taken:
+ * - close the observation interval at the last (previous)
+ * request dispatch or completion
+ * - compute rate, if possible, for that observation interval
+ * - reset to zero samples, which will trigger a proper
+ * re-initialization of the observation interval on next
+ * dispatch
+ */
+ if (delta_us > BFQ_MIN_TT/NSEC_PER_USEC &&
+ (bfqd->last_rq_max_size<<BFQ_RATE_SHIFT)/delta_us <
+ 1UL<<(BFQ_RATE_SHIFT - 10))
+ bfq_update_rate_reset(bfqd, NULL);
+ bfqd->last_completion = now_ns;
+
+ /*
+ * If we are waiting to discover whether the request pattern
+ * of the task associated with the queue is actually
+ * isochronous, and both requisites for this condition to hold
+ * are now satisfied, then compute soft_rt_next_start (see the
+ * comments on the function bfq_bfqq_softrt_next_start()). We
+ * schedule this delayed check when bfqq expires, if it still
+ * has in-flight requests.
+ */
+ if (bfq_bfqq_softrt_update(bfqq) && bfqq->dispatched == 0 &&
+ RB_EMPTY_ROOT(&bfqq->sort_list))
+ bfqq->soft_rt_next_start =
+ bfq_bfqq_softrt_next_start(bfqd, bfqq);
+
+ /*
+ * If this is the in-service queue, check if it needs to be expired,
+ * or if we want to idle in case it has no pending requests.
+ */
+ if (bfqd->in_service_queue == bfqq) {
+ if (bfqq->dispatched == 0 && bfq_bfqq_must_idle(bfqq)) {
+ bfq_arm_slice_timer(bfqd);
+ return;
+ } else if (bfq_may_expire_for_budg_timeout(bfqq))
+ bfq_bfqq_expire(bfqd, bfqq, false,
+ BFQQE_BUDGET_TIMEOUT);
+ else if (RB_EMPTY_ROOT(&bfqq->sort_list) &&
+ (bfqq->dispatched == 0 ||
+ !bfq_bfqq_may_idle(bfqq)))
+ bfq_bfqq_expire(bfqd, bfqq, false,
+ BFQQE_NO_MORE_REQUESTS);
+ }
+}
+
+static void bfq_put_rq_priv_body(struct bfq_queue *bfqq)
+{
+ bfqq->allocated--;
+
+ bfq_put_queue(bfqq);
+}
+
+static void bfq_put_rq_private(struct request_queue *q, struct request *rq)
+{
+ struct bfq_queue *bfqq = RQ_BFQQ(rq);
+ struct bfq_data *bfqd = bfqq->bfqd;
+
+ if (rq->rq_flags & RQF_STARTED)
+ bfqg_stats_update_completion(bfqq_group(bfqq),
+ rq_start_time_ns(rq),
+ rq_io_start_time_ns(rq),
+ rq->cmd_flags);
+
+ if (likely(rq->rq_flags & RQF_STARTED)) {
+ unsigned long flags;
+
+ spin_lock_irqsave(&bfqd->lock, flags);
+
+ bfq_completed_request(bfqq, bfqd);
+ bfq_put_rq_priv_body(bfqq);
+
+ spin_unlock_irqrestore(&bfqd->lock, flags);
+ } else {
+ /*
+ * Request rq may be still/already in the scheduler,
+ * in which case we need to remove it. And we cannot
+ * defer such a check and removal, to avoid
+ * inconsistencies in the time interval from the end
+ * of this function to the start of the deferred work.
+ * This situation seems to occur only in process
+ * context, as a consequence of a merge. In the
+ * current version of the code, this implies that the
+ * lock is held.
+ */
+
+ if (!RB_EMPTY_NODE(&rq->rb_node))
+ bfq_remove_request(q, rq);
+ bfq_put_rq_priv_body(bfqq);
+ }
+
+ rq->elv.priv[0] = NULL;
+ rq->elv.priv[1] = NULL;
+}
+
+/*
+ * Returns NULL if a new bfqq should be allocated, or the old bfqq if this
+ * was the last process referring to that bfqq.
+ */
+static struct bfq_queue *
+bfq_split_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq)
+{
+ bfq_log_bfqq(bfqq->bfqd, bfqq, "splitting queue");
+
+ if (bfqq_process_refs(bfqq) == 1) {
+ bfqq->pid = current->pid;
+ bfq_clear_bfqq_coop(bfqq);
+ bfq_clear_bfqq_split_coop(bfqq);
+ return bfqq;
+ }
+
+ bic_set_bfqq(bic, NULL, 1);
+
+ bfq_put_cooperator(bfqq);
+
+ bfq_put_queue(bfqq);
+ return NULL;
+}
+
+static struct bfq_queue *bfq_get_bfqq_handle_split(struct bfq_data *bfqd,
+ struct bfq_io_cq *bic,
+ struct bio *bio,
+ bool split, bool is_sync,
+ bool *new_queue)
+{
+ struct bfq_queue *bfqq = bic_to_bfqq(bic, is_sync);
+
+ if (likely(bfqq && bfqq != &bfqd->oom_bfqq))
+ return bfqq;
+
+ if (new_queue)
+ *new_queue = true;
+
+ if (bfqq)
+ bfq_put_queue(bfqq);
+ bfqq = bfq_get_queue(bfqd, bio, is_sync, bic);
+
+ bic_set_bfqq(bic, bfqq, is_sync);
+ if (split && is_sync) {
+ if ((bic->was_in_burst_list && bfqd->large_burst) ||
+ bic->saved_in_large_burst)
+ bfq_mark_bfqq_in_large_burst(bfqq);
+ else {
+ bfq_clear_bfqq_in_large_burst(bfqq);
+ if (bic->was_in_burst_list)
+ hlist_add_head(&bfqq->burst_list_node,
+ &bfqd->burst_list);
+ }
+ bfqq->split_time = jiffies;
+ }
+
+ return bfqq;
+}
+
+/*
+ * Allocate bfq data structures associated with this request.
+ */
+static int bfq_get_rq_private(struct request_queue *q, struct request *rq,
+ struct bio *bio)
+{
+ struct bfq_data *bfqd = q->elevator->elevator_data;
+ struct bfq_io_cq *bic = icq_to_bic(rq->elv.icq);
+ const int is_sync = rq_is_sync(rq);
+ struct bfq_queue *bfqq;
+ bool new_queue = false;
+ bool split = false;
+
+ spin_lock_irq(&bfqd->lock);
+
+ if (!bic)
+ goto queue_fail;
+
+ bfq_check_ioprio_change(bic, bio);
+
+ bfq_bic_update_cgroup(bic, bio);
+
+ bfqq = bfq_get_bfqq_handle_split(bfqd, bic, bio, false, is_sync,
+ &new_queue);
+
+ if (likely(!new_queue)) {
+ /* If the queue was seeky for too long, break it apart. */
+ if (bfq_bfqq_coop(bfqq) && bfq_bfqq_split_coop(bfqq)) {
+ bfq_log_bfqq(bfqd, bfqq, "breaking apart bfqq");
+
+ /* Update bic before losing reference to bfqq */
+ if (bfq_bfqq_in_large_burst(bfqq))
+ bic->saved_in_large_burst = true;
+
+ bfqq = bfq_split_bfqq(bic, bfqq);
+ split = true;
+
+ if (!bfqq)
+ bfqq = bfq_get_bfqq_handle_split(bfqd, bic, bio,
+ true, is_sync,
+ NULL);
+ }
+ }
+
+ bfqq->allocated++;
+ bfqq->ref++;
+ bfq_log_bfqq(bfqd, bfqq, "get_request %p: bfqq %p, %d",
+ rq, bfqq, bfqq->ref);
+
+ rq->elv.priv[0] = bic;
+ rq->elv.priv[1] = bfqq;
+
+ /*
+ * If a bfq_queue has only one process reference, it is owned
+ * by only this bic: we can then set bfqq->bic = bic. in
+ * addition, if the queue has also just been split, we have to
+ * resume its state.
+ */
+ if (likely(bfqq != &bfqd->oom_bfqq) && bfqq_process_refs(bfqq) == 1) {
+ bfqq->bic = bic;
+ if (split) {
+ /*
+ * The queue has just been split from a shared
+ * queue: restore the idle window and the
+ * possible weight raising period.
+ */
+ bfq_bfqq_resume_state(bfqq, bic);
+ }
+ }
+
+ if (unlikely(bfq_bfqq_just_created(bfqq)))
+ bfq_handle_burst(bfqd, bfqq);
+
+ spin_unlock_irq(&bfqd->lock);
+
+ return 0;
+
+queue_fail:
+ spin_unlock_irq(&bfqd->lock);
+
+ return 1;
+}
+
+static void bfq_idle_slice_timer_body(struct bfq_queue *bfqq)
+{
+ struct bfq_data *bfqd = bfqq->bfqd;
+ enum bfqq_expiration reason;
+ unsigned long flags;
+
+ spin_lock_irqsave(&bfqd->lock, flags);
+ bfq_clear_bfqq_wait_request(bfqq);
+
+ if (bfqq != bfqd->in_service_queue) {
+ spin_unlock_irqrestore(&bfqd->lock, flags);
+ return;
+ }
+
+ if (bfq_bfqq_budget_timeout(bfqq))
+ /*
+ * Also here the queue can be safely expired
+ * for budget timeout without wasting
+ * guarantees
+ */
+ reason = BFQQE_BUDGET_TIMEOUT;
+ else if (bfqq->queued[0] == 0 && bfqq->queued[1] == 0)
+ /*
+ * The queue may not be empty upon timer expiration,
+ * because we may not disable the timer when the
+ * first request of the in-service queue arrives
+ * during disk idling.
+ */
+ reason = BFQQE_TOO_IDLE;
+ else
+ goto schedule_dispatch;
+
+ bfq_bfqq_expire(bfqd, bfqq, true, reason);
+
+schedule_dispatch:
+ spin_unlock_irqrestore(&bfqd->lock, flags);
+ bfq_schedule_dispatch(bfqd);
+}
+
+/*
+ * Handler of the expiration of the timer running if the in-service queue
+ * is idling inside its time slice.
+ */
+static enum hrtimer_restart bfq_idle_slice_timer(struct hrtimer *timer)
+{
+ struct bfq_data *bfqd = container_of(timer, struct bfq_data,
+ idle_slice_timer);
+ struct bfq_queue *bfqq = bfqd->in_service_queue;
+
+ /*
+ * Theoretical race here: the in-service queue can be NULL or
+ * different from the queue that was idling if a new request
+ * arrives for the current queue and there is a full dispatch
+ * cycle that changes the in-service queue. This can hardly
+ * happen, but in the worst case we just expire a queue too
+ * early.
+ */
+ if (bfqq)
+ bfq_idle_slice_timer_body(bfqq);
+
+ return HRTIMER_NORESTART;
+}
+
+static void __bfq_put_async_bfqq(struct bfq_data *bfqd,
+ struct bfq_queue **bfqq_ptr)
+{
+ struct bfq_queue *bfqq = *bfqq_ptr;
+
+ bfq_log(bfqd, "put_async_bfqq: %p", bfqq);
+ if (bfqq) {
+ bfq_bfqq_move(bfqd, bfqq, bfqd->root_group);
+
+ bfq_log_bfqq(bfqd, bfqq, "put_async_bfqq: putting %p, %d",
+ bfqq, bfqq->ref);
+ bfq_put_queue(bfqq);
+ *bfqq_ptr = NULL;
+ }
+}
+
+/*
+ * Release all the bfqg references to its async queues. If we are
+ * deallocating the group these queues may still contain requests, so
+ * we reparent them to the root cgroup (i.e., the only one that will
+ * exist for sure until all the requests on a device are gone).
+ */
+void bfq_put_async_queues(struct bfq_data *bfqd, struct bfq_group *bfqg)
+{
+ int i, j;
+
+ for (i = 0; i < 2; i++)
+ for (j = 0; j < IOPRIO_BE_NR; j++)
+ __bfq_put_async_bfqq(bfqd, &bfqg->async_bfqq[i][j]);
+
+ __bfq_put_async_bfqq(bfqd, &bfqg->async_idle_bfqq);
+}
+
+static void bfq_exit_queue(struct elevator_queue *e)
+{
+ struct bfq_data *bfqd = e->elevator_data;
+ struct bfq_queue *bfqq, *n;
+
+ hrtimer_cancel(&bfqd->idle_slice_timer);
+
+ spin_lock_irq(&bfqd->lock);
+ list_for_each_entry_safe(bfqq, n, &bfqd->idle_list, bfqq_list)
+ bfq_deactivate_bfqq(bfqd, bfqq, false, false);
+ spin_unlock_irq(&bfqd->lock);
+
+ hrtimer_cancel(&bfqd->idle_slice_timer);
+
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ blkcg_deactivate_policy(bfqd->queue, &blkcg_policy_bfq);
+#else
+ spin_lock_irq(&bfqd->lock);
+ bfq_put_async_queues(bfqd, bfqd->root_group);
+ kfree(bfqd->root_group);
+ spin_unlock_irq(&bfqd->lock);
+#endif
+
+ kfree(bfqd);
+}
+
+static void bfq_init_root_group(struct bfq_group *root_group,
+ struct bfq_data *bfqd)
+{
+ int i;
+
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ root_group->entity.parent = NULL;
+ root_group->my_entity = NULL;
+ root_group->bfqd = bfqd;
+#endif
+ root_group->rq_pos_tree = RB_ROOT;
+ for (i = 0; i < BFQ_IOPRIO_CLASSES; i++)
+ root_group->sched_data.service_tree[i] = BFQ_SERVICE_TREE_INIT;
+ root_group->sched_data.bfq_class_idle_last_service = jiffies;
+}
+
+static int bfq_init_queue(struct request_queue *q, struct elevator_type *e)
+{
+ struct bfq_data *bfqd;
+ struct elevator_queue *eq;
+
+ eq = elevator_alloc(q, e);
+ if (!eq)
+ return -ENOMEM;
+
+ bfqd = kzalloc_node(sizeof(*bfqd), GFP_KERNEL, q->node);
+ if (!bfqd) {
+ kobject_put(&eq->kobj);
+ return -ENOMEM;
+ }
+ eq->elevator_data = bfqd;
+
+ spin_lock_irq(q->queue_lock);
+ q->elevator = eq;
+ spin_unlock_irq(q->queue_lock);
+
+ /*
+ * Our fallback bfqq if bfq_find_alloc_queue() runs into OOM issues.
+ * Grab a permanent reference to it, so that the normal code flow
+ * will not attempt to free it.
+ */
+ bfq_init_bfqq(bfqd, &bfqd->oom_bfqq, NULL, 1, 0);
+ bfqd->oom_bfqq.ref++;
+ bfqd->oom_bfqq.new_ioprio = BFQ_DEFAULT_QUEUE_IOPRIO;
+ bfqd->oom_bfqq.new_ioprio_class = IOPRIO_CLASS_BE;
+ bfqd->oom_bfqq.entity.new_weight =
+ bfq_ioprio_to_weight(bfqd->oom_bfqq.new_ioprio);
+
+ /* oom_bfqq does not participate to bursts */
+ bfq_clear_bfqq_just_created(&bfqd->oom_bfqq);
+
+ /*
+ * Trigger weight initialization, according to ioprio, at the
+ * oom_bfqq's first activation. The oom_bfqq's ioprio and ioprio
+ * class won't be changed any more.
+ */
+ bfqd->oom_bfqq.entity.prio_changed = 1;
+
+ bfqd->queue = q;
+
+ INIT_LIST_HEAD(&bfqd->dispatch);
+
+ hrtimer_init(&bfqd->idle_slice_timer, CLOCK_MONOTONIC,
+ HRTIMER_MODE_REL);
+ bfqd->idle_slice_timer.function = bfq_idle_slice_timer;
+
+ bfqd->queue_weights_tree = RB_ROOT;
+ bfqd->group_weights_tree = RB_ROOT;
+
+ INIT_LIST_HEAD(&bfqd->active_list);
+ INIT_LIST_HEAD(&bfqd->idle_list);
+ INIT_HLIST_HEAD(&bfqd->burst_list);
+
+ bfqd->hw_tag = -1;
+
+ bfqd->bfq_max_budget = bfq_default_max_budget;
+
+ bfqd->bfq_fifo_expire[0] = bfq_fifo_expire[0];
+ bfqd->bfq_fifo_expire[1] = bfq_fifo_expire[1];
+ bfqd->bfq_back_max = bfq_back_max;
+ bfqd->bfq_back_penalty = bfq_back_penalty;
+ bfqd->bfq_slice_idle = bfq_slice_idle;
+ bfqd->bfq_timeout = bfq_timeout;
+
+ bfqd->bfq_requests_within_timer = 120;
+
+ bfqd->bfq_large_burst_thresh = 8;
+ bfqd->bfq_burst_interval = msecs_to_jiffies(180);
+
+ bfqd->low_latency = true;
+
+ /*
+ * Trade-off between responsiveness and fairness.
+ */
+ bfqd->bfq_wr_coeff = 30;
+ bfqd->bfq_wr_rt_max_time = msecs_to_jiffies(300);
+ bfqd->bfq_wr_max_time = 0;
+ bfqd->bfq_wr_min_idle_time = msecs_to_jiffies(2000);
+ bfqd->bfq_wr_min_inter_arr_async = msecs_to_jiffies(500);
+ bfqd->bfq_wr_max_softrt_rate = 7000; /*
+ * Approximate rate required
+ * to playback or record a
+ * high-definition compressed
+ * video.
+ */
+ bfqd->wr_busy_queues = 0;
+
+ /*
+ * Begin by assuming, optimistically, that the device is a
+ * high-speed one, and that its peak rate is equal to 2/3 of
+ * the highest reference rate.
+ */
+ bfqd->RT_prod = R_fast[blk_queue_nonrot(bfqd->queue)] *
+ T_fast[blk_queue_nonrot(bfqd->queue)];
+ bfqd->peak_rate = R_fast[blk_queue_nonrot(bfqd->queue)] * 2 / 3;
+ bfqd->device_speed = BFQ_BFQD_FAST;
+
+ spin_lock_init(&bfqd->lock);
+
+ /*
+ * The invocation of the next bfq_create_group_hierarchy
+ * function is the head of a chain of function calls
+ * (bfq_create_group_hierarchy->blkcg_activate_policy->
+ * blk_mq_freeze_queue) that may lead to the invocation of the
+ * has_work hook function. For this reason,
+ * bfq_create_group_hierarchy is invoked only after all
+ * scheduler data has been initialized, apart from the fields
+ * that can be initialized only after invoking
+ * bfq_create_group_hierarchy. This, in particular, enables
+ * has_work to correctly return false. Of course, to avoid
+ * other inconsistencies, the blk-mq stack must then refrain
+ * from invoking further scheduler hooks before this init
+ * function is finished.
+ */
+ bfqd->root_group = bfq_create_group_hierarchy(bfqd, q->node);
+ if (!bfqd->root_group)
+ goto out_free;
+ bfq_init_root_group(bfqd->root_group, bfqd);
+ bfq_init_entity(&bfqd->oom_bfqq.entity, bfqd->root_group);
+
+
+ return 0;
+
+out_free:
+ kfree(bfqd);
+ kobject_put(&eq->kobj);
+ return -ENOMEM;
+}
+
+static void bfq_slab_kill(void)
+{
+ kmem_cache_destroy(bfq_pool);
+}
+
+static int __init bfq_slab_setup(void)
+{
+ bfq_pool = KMEM_CACHE(bfq_queue, 0);
+ if (!bfq_pool)
+ return -ENOMEM;
+ return 0;
+}
+
+static ssize_t bfq_var_show(unsigned int var, char *page)
+{
+ return sprintf(page, "%u\n", var);
+}
+
+static ssize_t bfq_var_store(unsigned long *var, const char *page,
+ size_t count)
+{
+ unsigned long new_val;
+ int ret = kstrtoul(page, 10, &new_val);
+
+ if (ret == 0)
+ *var = new_val;
+
+ return count;
+}
+
+#define SHOW_FUNCTION(__FUNC, __VAR, __CONV) \
+static ssize_t __FUNC(struct elevator_queue *e, char *page) \
+{ \
+ struct bfq_data *bfqd = e->elevator_data; \
+ u64 __data = __VAR; \
+ if (__CONV == 1) \
+ __data = jiffies_to_msecs(__data); \
+ else if (__CONV == 2) \
+ __data = div_u64(__data, NSEC_PER_MSEC); \
+ return bfq_var_show(__data, (page)); \
+}
+SHOW_FUNCTION(bfq_fifo_expire_sync_show, bfqd->bfq_fifo_expire[1], 2);
+SHOW_FUNCTION(bfq_fifo_expire_async_show, bfqd->bfq_fifo_expire[0], 2);
+SHOW_FUNCTION(bfq_back_seek_max_show, bfqd->bfq_back_max, 0);
+SHOW_FUNCTION(bfq_back_seek_penalty_show, bfqd->bfq_back_penalty, 0);
+SHOW_FUNCTION(bfq_slice_idle_show, bfqd->bfq_slice_idle, 2);
+SHOW_FUNCTION(bfq_max_budget_show, bfqd->bfq_user_max_budget, 0);
+SHOW_FUNCTION(bfq_timeout_sync_show, bfqd->bfq_timeout, 1);
+SHOW_FUNCTION(bfq_strict_guarantees_show, bfqd->strict_guarantees, 0);
+SHOW_FUNCTION(bfq_low_latency_show, bfqd->low_latency, 0);
+#undef SHOW_FUNCTION
+
+#define USEC_SHOW_FUNCTION(__FUNC, __VAR) \
+static ssize_t __FUNC(struct elevator_queue *e, char *page) \
+{ \
+ struct bfq_data *bfqd = e->elevator_data; \
+ u64 __data = __VAR; \
+ __data = div_u64(__data, NSEC_PER_USEC); \
+ return bfq_var_show(__data, (page)); \
+}
+USEC_SHOW_FUNCTION(bfq_slice_idle_us_show, bfqd->bfq_slice_idle);
+#undef USEC_SHOW_FUNCTION
+
+#define STORE_FUNCTION(__FUNC, __PTR, MIN, MAX, __CONV) \
+static ssize_t \
+__FUNC(struct elevator_queue *e, const char *page, size_t count) \
+{ \
+ struct bfq_data *bfqd = e->elevator_data; \
+ unsigned long uninitialized_var(__data); \
+ int ret = bfq_var_store(&__data, (page), count); \
+ if (__data < (MIN)) \
+ __data = (MIN); \
+ else if (__data > (MAX)) \
+ __data = (MAX); \
+ if (__CONV == 1) \
+ *(__PTR) = msecs_to_jiffies(__data); \
+ else if (__CONV == 2) \
+ *(__PTR) = (u64)__data * NSEC_PER_MSEC; \
+ else \
+ *(__PTR) = __data; \
+ return ret; \
+}
+STORE_FUNCTION(bfq_fifo_expire_sync_store, &bfqd->bfq_fifo_expire[1], 1,
+ INT_MAX, 2);
+STORE_FUNCTION(bfq_fifo_expire_async_store, &bfqd->bfq_fifo_expire[0], 1,
+ INT_MAX, 2);
+STORE_FUNCTION(bfq_back_seek_max_store, &bfqd->bfq_back_max, 0, INT_MAX, 0);
+STORE_FUNCTION(bfq_back_seek_penalty_store, &bfqd->bfq_back_penalty, 1,
+ INT_MAX, 0);
+STORE_FUNCTION(bfq_slice_idle_store, &bfqd->bfq_slice_idle, 0, INT_MAX, 2);
+#undef STORE_FUNCTION
+
+#define USEC_STORE_FUNCTION(__FUNC, __PTR, MIN, MAX) \
+static ssize_t __FUNC(struct elevator_queue *e, const char *page, size_t count)\
+{ \
+ struct bfq_data *bfqd = e->elevator_data; \
+ unsigned long uninitialized_var(__data); \
+ int ret = bfq_var_store(&__data, (page), count); \
+ if (__data < (MIN)) \
+ __data = (MIN); \
+ else if (__data > (MAX)) \
+ __data = (MAX); \
+ *(__PTR) = (u64)__data * NSEC_PER_USEC; \
+ return ret; \
+}
+USEC_STORE_FUNCTION(bfq_slice_idle_us_store, &bfqd->bfq_slice_idle, 0,
+ UINT_MAX);
+#undef USEC_STORE_FUNCTION
+
+static ssize_t bfq_max_budget_store(struct elevator_queue *e,
+ const char *page, size_t count)
+{
+ struct bfq_data *bfqd = e->elevator_data;
+ unsigned long uninitialized_var(__data);
+ int ret = bfq_var_store(&__data, (page), count);
+
+ if (__data == 0)
+ bfqd->bfq_max_budget = bfq_calc_max_budget(bfqd);
+ else {
+ if (__data > INT_MAX)
+ __data = INT_MAX;
+ bfqd->bfq_max_budget = __data;
+ }
+
+ bfqd->bfq_user_max_budget = __data;
+
+ return ret;
+}
+
+/*
+ * Leaving this name to preserve name compatibility with cfq
+ * parameters, but this timeout is used for both sync and async.
+ */
+static ssize_t bfq_timeout_sync_store(struct elevator_queue *e,
+ const char *page, size_t count)
+{
+ struct bfq_data *bfqd = e->elevator_data;
+ unsigned long uninitialized_var(__data);
+ int ret = bfq_var_store(&__data, (page), count);
+
+ if (__data < 1)
+ __data = 1;
+ else if (__data > INT_MAX)
+ __data = INT_MAX;
+
+ bfqd->bfq_timeout = msecs_to_jiffies(__data);
+ if (bfqd->bfq_user_max_budget == 0)
+ bfqd->bfq_max_budget = bfq_calc_max_budget(bfqd);
+
+ return ret;
+}
+
+static ssize_t bfq_strict_guarantees_store(struct elevator_queue *e,
+ const char *page, size_t count)
+{
+ struct bfq_data *bfqd = e->elevator_data;
+ unsigned long uninitialized_var(__data);
+ int ret = bfq_var_store(&__data, (page), count);
+
+ if (__data > 1)
+ __data = 1;
+ if (!bfqd->strict_guarantees && __data == 1
+ && bfqd->bfq_slice_idle < 8 * NSEC_PER_MSEC)
+ bfqd->bfq_slice_idle = 8 * NSEC_PER_MSEC;
+
+ bfqd->strict_guarantees = __data;
+
+ return ret;
+}
+
+static ssize_t bfq_low_latency_store(struct elevator_queue *e,
+ const char *page, size_t count)
+{
+ struct bfq_data *bfqd = e->elevator_data;
+ unsigned long uninitialized_var(__data);
+ int ret = bfq_var_store(&__data, (page), count);
+
+ if (__data > 1)
+ __data = 1;
+ if (__data == 0 && bfqd->low_latency != 0)
+ bfq_end_wr(bfqd);
+ bfqd->low_latency = __data;
+
+ return ret;
+}
+
+#define BFQ_ATTR(name) \
+ __ATTR(name, 0644, bfq_##name##_show, bfq_##name##_store)
+
+static struct elv_fs_entry bfq_attrs[] = {
+ BFQ_ATTR(fifo_expire_sync),
+ BFQ_ATTR(fifo_expire_async),
+ BFQ_ATTR(back_seek_max),
+ BFQ_ATTR(back_seek_penalty),
+ BFQ_ATTR(slice_idle),
+ BFQ_ATTR(slice_idle_us),
+ BFQ_ATTR(max_budget),
+ BFQ_ATTR(timeout_sync),
+ BFQ_ATTR(strict_guarantees),
+ BFQ_ATTR(low_latency),
+ __ATTR_NULL
+};
+
+static struct elevator_type iosched_bfq_mq = {
+ .ops.mq = {
+ .get_rq_priv = bfq_get_rq_private,
+ .put_rq_priv = bfq_put_rq_private,
+ .exit_icq = bfq_exit_icq,
+ .insert_requests = bfq_insert_requests,
+ .dispatch_request = bfq_dispatch_request,
+ .next_request = elv_rb_latter_request,
+ .former_request = elv_rb_former_request,
+ .allow_merge = bfq_allow_bio_merge,
+ .bio_merge = bfq_bio_merge,
+ .request_merge = bfq_request_merge,
+ .requests_merged = bfq_requests_merged,
+ .request_merged = bfq_request_merged,
+ .has_work = bfq_has_work,
+ .init_sched = bfq_init_queue,
+ .exit_sched = bfq_exit_queue,
+ },
+
+ .uses_mq = true,
+ .icq_size = sizeof(struct bfq_io_cq),
+ .icq_align = __alignof__(struct bfq_io_cq),
+ .elevator_attrs = bfq_attrs,
+ .elevator_name = "bfq",
+ .elevator_owner = THIS_MODULE,
+};
+
+static int __init bfq_init(void)
+{
+ int ret;
+
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ ret = blkcg_policy_register(&blkcg_policy_bfq);
+ if (ret)
+ return ret;
+#endif
+
+ ret = -ENOMEM;
+ if (bfq_slab_setup())
+ goto err_pol_unreg;
+
+ /*
+ * Times to load large popular applications for the typical
+ * systems installed on the reference devices (see the
+ * comments before the definitions of the next two
+ * arrays). Actually, we use slightly slower values, as the
+ * estimated peak rate tends to be smaller than the actual
+ * peak rate. The reason for this last fact is that estimates
+ * are computed over much shorter time intervals than the long
+ * intervals typically used for benchmarking. Why? First, to
+ * adapt more quickly to variations. Second, because an I/O
+ * scheduler cannot rely on a peak-rate-evaluation workload to
+ * be run for a long time.
+ */
+ T_slow[0] = msecs_to_jiffies(3500); /* actually 4 sec */
+ T_slow[1] = msecs_to_jiffies(6000); /* actually 6.5 sec */
+ T_fast[0] = msecs_to_jiffies(7000); /* actually 8 sec */
+ T_fast[1] = msecs_to_jiffies(2500); /* actually 3 sec */
+
+ /*
+ * Thresholds that determine the switch between speed classes
+ * (see the comments before the definition of the array
+ * device_speed_thresh). These thresholds are biased towards
+ * transitions to the fast class. This is safer than the
+ * opposite bias. In fact, a wrong transition to the slow
+ * class results in short weight-raising periods, because the
+ * speed of the device then tends to be higher that the
+ * reference peak rate. On the opposite end, a wrong
+ * transition to the fast class tends to increase
+ * weight-raising periods, because of the opposite reason.
+ */
+ device_speed_thresh[0] = (4 * R_slow[0]) / 3;
+ device_speed_thresh[1] = (4 * R_slow[1]) / 3;
+
+ ret = elv_register(&iosched_bfq_mq);
+ if (ret)
+ goto err_pol_unreg;
+
+ return 0;
+
+err_pol_unreg:
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ blkcg_policy_unregister(&blkcg_policy_bfq);
+#endif
+ return ret;
+}
+
+static void __exit bfq_exit(void)
+{
+ elv_unregister(&iosched_bfq_mq);
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ blkcg_policy_unregister(&blkcg_policy_bfq);
+#endif
+ bfq_slab_kill();
+}
+
+module_init(bfq_init);
+module_exit(bfq_exit);
+
+MODULE_AUTHOR("Paolo Valente");
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("MQ Budget Fair Queueing I/O Scheduler");
diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
new file mode 100644
index 0000000..ae783c0
--- /dev/null
+++ b/block/bfq-iosched.h
@@ -0,0 +1,941 @@
+/*
+ * Header file for the BFQ I/O scheduler: data structures and
+ * prototypes of interface functions among BFQ components.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#ifndef _BFQ_H
+#define _BFQ_H
+
+#include <linux/blktrace_api.h>
+#include <linux/hrtimer.h>
+#include <linux/blk-cgroup.h>
+
+#define BFQ_IOPRIO_CLASSES 3
+#define BFQ_CL_IDLE_TIMEOUT (HZ/5)
+
+#define BFQ_MIN_WEIGHT 1
+#define BFQ_MAX_WEIGHT 1000
+#define BFQ_WEIGHT_CONVERSION_COEFF 10
+
+#define BFQ_DEFAULT_QUEUE_IOPRIO 4
+
+#define BFQ_WEIGHT_LEGACY_DFL 100
+#define BFQ_DEFAULT_GRP_IOPRIO 0
+#define BFQ_DEFAULT_GRP_CLASS IOPRIO_CLASS_BE
+
+/*
+ * Soft real-time applications are extremely more latency sensitive
+ * than interactive ones. Over-raise the weight of the former to
+ * privilege them against the latter.
+ */
+#define BFQ_SOFTRT_WEIGHT_FACTOR 100
+
+struct bfq_entity;
+
+/**
+ * struct bfq_service_tree - per ioprio_class service tree.
+ *
+ * Each service tree represents a B-WF2Q+ scheduler on its own. Each
+ * ioprio_class has its own independent scheduler, and so its own
+ * bfq_service_tree. All the fields are protected by the queue lock
+ * of the containing bfqd.
+ */
+struct bfq_service_tree {
+ /* tree for active entities (i.e., those backlogged) */
+ struct rb_root active;
+ /* tree for idle entities (i.e., not backlogged, with V <= F_i)*/
+ struct rb_root idle;
+
+ /* idle entity with minimum F_i */
+ struct bfq_entity *first_idle;
+ /* idle entity with maximum F_i */
+ struct bfq_entity *last_idle;
+
+ /* scheduler virtual time */
+ u64 vtime;
+ /* scheduler weight sum; active and idle entities contribute to it */
+ unsigned long wsum;
+};
+
+/**
+ * struct bfq_sched_data - multi-class scheduler.
+ *
+ * bfq_sched_data is the basic scheduler queue. It supports three
+ * ioprio_classes, and can be used either as a toplevel queue or as an
+ * intermediate queue on a hierarchical setup. @next_in_service
+ * points to the active entity of the sched_data service trees that
+ * will be scheduled next. It is used to reduce the number of steps
+ * needed for each hierarchical-schedule update.
+ *
+ * The supported ioprio_classes are the same as in CFQ, in descending
+ * priority order, IOPRIO_CLASS_RT, IOPRIO_CLASS_BE, IOPRIO_CLASS_IDLE.
+ * Requests from higher priority queues are served before all the
+ * requests from lower priority queues; among requests of the same
+ * queue requests are served according to B-WF2Q+.
+ * All the fields are protected by the queue lock of the containing bfqd.
+ */
+struct bfq_sched_data {
+ /* entity in service */
+ struct bfq_entity *in_service_entity;
+ /* head-of-line entity (see comments above) */
+ struct bfq_entity *next_in_service;
+ /* array of service trees, one per ioprio_class */
+ struct bfq_service_tree service_tree[BFQ_IOPRIO_CLASSES];
+ /* last time CLASS_IDLE was served */
+ unsigned long bfq_class_idle_last_service;
+
+};
+
+/**
+ * struct bfq_weight_counter - counter of the number of all active entities
+ * with a given weight.
+ */
+struct bfq_weight_counter {
+ unsigned int weight; /* weight of the entities this counter refers to */
+ unsigned int num_active; /* nr of active entities with this weight */
+ /*
+ * Weights tree member (see bfq_data's @queue_weights_tree and
+ * @group_weights_tree)
+ */
+ struct rb_node weights_node;
+};
+
+/**
+ * struct bfq_entity - schedulable entity.
+ *
+ * A bfq_entity is used to represent either a bfq_queue (leaf node in the
+ * cgroup hierarchy) or a bfq_group into the upper level scheduler. Each
+ * entity belongs to the sched_data of the parent group in the cgroup
+ * hierarchy. Non-leaf entities have also their own sched_data, stored
+ * in @my_sched_data.
+ *
+ * Each entity stores independently its priority values; this would
+ * allow different weights on different devices, but this
+ * functionality is not exported to userspace by now. Priorities and
+ * weights are updated lazily, first storing the new values into the
+ * new_* fields, then setting the @prio_changed flag. As soon as
+ * there is a transition in the entity state that allows the priority
+ * update to take place the effective and the requested priority
+ * values are synchronized.
+ *
+ * Unless cgroups are used, the weight value is calculated from the
+ * ioprio to export the same interface as CFQ. When dealing with
+ * ``well-behaved'' queues (i.e., queues that do not spend too much
+ * time to consume their budget and have true sequential behavior, and
+ * when there are no external factors breaking anticipation) the
+ * relative weights at each level of the cgroups hierarchy should be
+ * guaranteed. All the fields are protected by the queue lock of the
+ * containing bfqd.
+ */
+struct bfq_entity {
+ /* service_tree member */
+ struct rb_node rb_node;
+ /* pointer to the weight counter associated with this entity */
+ struct bfq_weight_counter *weight_counter;
+
+ /*
+ * Flag, true if the entity is on a tree (either the active or
+ * the idle one of its service_tree) or is in service.
+ */
+ bool on_st;
+
+ /* B-WF2Q+ start and finish timestamps [sectors/weight] */
+ u64 start, finish;
+
+ /* tree the entity is enqueued into; %NULL if not on a tree */
+ struct rb_root *tree;
+
+ /*
+ * minimum start time of the (active) subtree rooted at this
+ * entity; used for O(log N) lookups into active trees
+ */
+ u64 min_start;
+
+ /* amount of service received during the last service slot */
+ int service;
+
+ /* budget, used also to calculate F_i: F_i = S_i + @budget / @weight */
+ int budget;
+
+ /* weight of the queue */
+ int weight;
+ /* next weight if a change is in progress */
+ int new_weight;
+
+ /* original weight, used to implement weight boosting */
+ int orig_weight;
+
+ /* parent entity, for hierarchical scheduling */
+ struct bfq_entity *parent;
+
+ /*
+ * For non-leaf nodes in the hierarchy, the associated
+ * scheduler queue, %NULL on leaf nodes.
+ */
+ struct bfq_sched_data *my_sched_data;
+ /* the scheduler queue this entity belongs to */
+ struct bfq_sched_data *sched_data;
+
+ /* flag, set to request a weight, ioprio or ioprio_class change */
+ int prio_changed;
+};
+
+struct bfq_group;
+
+/**
+ * struct bfq_ttime - per process thinktime stats.
+ */
+struct bfq_ttime {
+ /* completion time of the last request */
+ u64 last_end_request;
+
+ /* total process thinktime */
+ u64 ttime_total;
+ /* number of thinktime samples */
+ unsigned long ttime_samples;
+ /* average process thinktime */
+ u64 ttime_mean;
+};
+
+/**
+ * struct bfq_queue - leaf schedulable entity.
+ *
+ * A bfq_queue is a leaf request queue; it can be associated with an
+ * io_context or more, if it is async or shared between cooperating
+ * processes. @cgroup holds a reference to the cgroup, to be sure that it
+ * does not disappear while a bfqq still references it (mostly to avoid
+ * races between request issuing and task migration followed by cgroup
+ * destruction).
+ * All the fields are protected by the queue lock of the containing bfqd.
+ */
+struct bfq_queue {
+ /* reference counter */
+ int ref;
+ /* parent bfq_data */
+ struct bfq_data *bfqd;
+
+ /* current ioprio and ioprio class */
+ unsigned short ioprio, ioprio_class;
+ /* next ioprio and ioprio class if a change is in progress */
+ unsigned short new_ioprio, new_ioprio_class;
+
+ /*
+ * Shared bfq_queue if queue is cooperating with one or more
+ * other queues.
+ */
+ struct bfq_queue *new_bfqq;
+ /* request-position tree member (see bfq_group's @rq_pos_tree) */
+ struct rb_node pos_node;
+ /* request-position tree root (see bfq_group's @rq_pos_tree) */
+ struct rb_root *pos_root;
+
+ /* sorted list of pending requests */
+ struct rb_root sort_list;
+ /* if fifo isn't expired, next request to serve */
+ struct request *next_rq;
+ /* number of sync and async requests queued */
+ int queued[2];
+ /* number of requests currently allocated */
+ int allocated;
+ /* number of pending metadata requests */
+ int meta_pending;
+ /* fifo list of requests in sort_list */
+ struct list_head fifo;
+
+ /* entity representing this queue in the scheduler */
+ struct bfq_entity entity;
+
+ /* maximum budget allowed from the feedback mechanism */
+ int max_budget;
+ /* budget expiration (in jiffies) */
+ unsigned long budget_timeout;
+
+ /* number of requests on the dispatch list or inside driver */
+ int dispatched;
+
+ /* status flags */
+ unsigned long flags;
+
+ /* node for active/idle bfqq list inside parent bfqd */
+ struct list_head bfqq_list;
+
+ /* associated @bfq_ttime struct */
+ struct bfq_ttime ttime;
+
+ /* bit vector: a 1 for each seeky requests in history */
+ u32 seek_history;
+
+ /* node for the device's burst list */
+ struct hlist_node burst_list_node;
+
+ /* position of the last request enqueued */
+ sector_t last_request_pos;
+
+ /* Number of consecutive pairs of request completion and
+ * arrival, such that the queue becomes idle after the
+ * completion, but the next request arrives within an idle
+ * time slice; used only if the queue's IO_bound flag has been
+ * cleared.
+ */
+ unsigned int requests_within_timer;
+
+ /* pid of the process owning the queue, used for logging purposes */
+ pid_t pid;
+
+ /*
+ * Pointer to the bfq_io_cq owning the bfq_queue, set to %NULL
+ * if the queue is shared.
+ */
+ struct bfq_io_cq *bic;
+
+ /* current maximum weight-raising time for this queue */
+ unsigned long wr_cur_max_time;
+ /*
+ * Minimum time instant such that, only if a new request is
+ * enqueued after this time instant in an idle @bfq_queue with
+ * no outstanding requests, then the task associated with the
+ * queue it is deemed as soft real-time (see the comments on
+ * the function bfq_bfqq_softrt_next_start())
+ */
+ unsigned long soft_rt_next_start;
+ /*
+ * Start time of the current weight-raising period if
+ * the @bfq-queue is being weight-raised, otherwise
+ * finish time of the last weight-raising period.
+ */
+ unsigned long last_wr_start_finish;
+ /* factor by which the weight of this queue is multiplied */
+ unsigned int wr_coeff;
+ /*
+ * Time of the last transition of the @bfq_queue from idle to
+ * backlogged.
+ */
+ unsigned long last_idle_bklogged;
+ /*
+ * Cumulative service received from the @bfq_queue since the
+ * last transition from idle to backlogged.
+ */
+ unsigned long service_from_backlogged;
+
+ /*
+ * Value of wr start time when switching to soft rt
+ */
+ unsigned long wr_start_at_switch_to_srt;
+
+ unsigned long split_time; /* time of last split */
+};
+
+/**
+ * struct bfq_io_cq - per (request_queue, io_context) structure.
+ */
+struct bfq_io_cq {
+ /* associated io_cq structure */
+ struct io_cq icq; /* must be the first member */
+ /* array of two process queues, the sync and the async */
+ struct bfq_queue *bfqq[2];
+ /* per (request_queue, blkcg) ioprio */
+ int ioprio;
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ uint64_t blkcg_serial_nr; /* the current blkcg serial */
+#endif
+ /*
+ * Snapshot of the idle window before merging; taken to
+ * remember this value while the queue is merged, so as to be
+ * able to restore it in case of split.
+ */
+ bool saved_idle_window;
+ /*
+ * Same purpose as the previous two fields for the I/O bound
+ * classification of a queue.
+ */
+ bool saved_IO_bound;
+
+ /*
+ * Same purpose as the previous fields for the value of the
+ * field keeping the queue's belonging to a large burst
+ */
+ bool saved_in_large_burst;
+ /*
+ * True if the queue belonged to a burst list before its merge
+ * with another cooperating queue.
+ */
+ bool was_in_burst_list;
+
+ /*
+ * Similar to previous fields: save wr information.
+ */
+ unsigned long saved_wr_coeff;
+ unsigned long saved_last_wr_start_finish;
+ unsigned long saved_wr_start_at_switch_to_srt;
+ unsigned int saved_wr_cur_max_time;
+ struct bfq_ttime saved_ttime;
+};
+
+enum bfq_device_speed {
+ BFQ_BFQD_FAST,
+ BFQ_BFQD_SLOW,
+};
+
+/**
+ * struct bfq_data - per-device data structure.
+ *
+ * All the fields are protected by @lock.
+ */
+struct bfq_data {
+ /* device request queue */
+ struct request_queue *queue;
+ /* dispatch queue */
+ struct list_head dispatch;
+
+ /* root bfq_group for the device */
+ struct bfq_group *root_group;
+
+ /*
+ * rbtree of weight counters of @bfq_queues, sorted by
+ * weight. Used to keep track of whether all @bfq_queues have
+ * the same weight. The tree contains one counter for each
+ * distinct weight associated to some active and not
+ * weight-raised @bfq_queue (see the comments to the functions
+ * bfq_weights_tree_[add|remove] for further details).
+ */
+ struct rb_root queue_weights_tree;
+ /*
+ * rbtree of non-queue @bfq_entity weight counters, sorted by
+ * weight. Used to keep track of whether all @bfq_groups have
+ * the same weight. The tree contains one counter for each
+ * distinct weight associated to some active @bfq_group (see
+ * the comments to the functions bfq_weights_tree_[add|remove]
+ * for further details).
+ */
+ struct rb_root group_weights_tree;
+
+ /*
+ * Number of bfq_queues containing requests (including the
+ * queue in service, even if it is idling).
+ */
+ int busy_queues;
+ /* number of weight-raised busy @bfq_queues */
+ int wr_busy_queues;
+ /* number of queued requests */
+ int queued;
+ /* number of requests dispatched and waiting for completion */
+ int rq_in_driver;
+
+ /*
+ * Maximum number of requests in driver in the last
+ * @hw_tag_samples completed requests.
+ */
+ int max_rq_in_driver;
+ /* number of samples used to calculate hw_tag */
+ int hw_tag_samples;
+ /* flag set to one if the driver is showing a queueing behavior */
+ int hw_tag;
+
+ /* number of budgets assigned */
+ int budgets_assigned;
+
+ /*
+ * Timer set when idling (waiting) for the next request from
+ * the queue in service.
+ */
+ struct hrtimer idle_slice_timer;
+
+ /* bfq_queue in service */
+ struct bfq_queue *in_service_queue;
+
+ /* on-disk position of the last served request */
+ sector_t last_position;
+
+ /* time of last request completion (ns) */
+ u64 last_completion;
+
+ /* time of first rq dispatch in current observation interval (ns) */
+ u64 first_dispatch;
+ /* time of last rq dispatch in current observation interval (ns) */
+ u64 last_dispatch;
+
+ /* beginning of the last budget */
+ ktime_t last_budget_start;
+ /* beginning of the last idle slice */
+ ktime_t last_idling_start;
+
+ /* number of samples in current observation interval */
+ int peak_rate_samples;
+ /* num of samples of seq dispatches in current observation interval */
+ u32 sequential_samples;
+ /* total num of sectors transferred in current observation interval */
+ u64 tot_sectors_dispatched;
+ /* max rq size seen during current observation interval (sectors) */
+ u32 last_rq_max_size;
+ /* time elapsed from first dispatch in current observ. interval (us) */
+ u64 delta_from_first;
+ /*
+ * Current estimate of the device peak rate, measured in
+ * [BFQ_RATE_SHIFT * sectors/usec]. The left-shift by
+ * BFQ_RATE_SHIFT is performed to increase precision in
+ * fixed-point calculations.
+ */
+ u32 peak_rate;
+
+ /* maximum budget allotted to a bfq_queue before rescheduling */
+ int bfq_max_budget;
+
+ /* list of all the bfq_queues active on the device */
+ struct list_head active_list;
+ /* list of all the bfq_queues idle on the device */
+ struct list_head idle_list;
+
+ /*
+ * Timeout for async/sync requests; when it fires, requests
+ * are served in fifo order.
+ */
+ u64 bfq_fifo_expire[2];
+ /* weight of backward seeks wrt forward ones */
+ unsigned int bfq_back_penalty;
+ /* maximum allowed backward seek */
+ unsigned int bfq_back_max;
+ /* maximum idling time */
+ u32 bfq_slice_idle;
+
+ /* user-configured max budget value (0 for auto-tuning) */
+ int bfq_user_max_budget;
+ /*
+ * Timeout for bfq_queues to consume their budget; used to
+ * prevent seeky queues from imposing long latencies to
+ * sequential or quasi-sequential ones (this also implies that
+ * seeky queues cannot receive guarantees in the service
+ * domain; after a timeout they are charged for the time they
+ * have been in service, to preserve fairness among them, but
+ * without service-domain guarantees).
+ */
+ unsigned int bfq_timeout;
+
+ /*
+ * Number of consecutive requests that must be issued within
+ * the idle time slice to set again idling to a queue which
+ * was marked as non-I/O-bound (see the definition of the
+ * IO_bound flag for further details).
+ */
+ unsigned int bfq_requests_within_timer;
+
+ /*
+ * Force device idling whenever needed to provide accurate
+ * service guarantees, without caring about throughput
+ * issues. CAVEAT: this may even increase latencies, in case
+ * of useless idling for processes that did stop doing I/O.
+ */
+ bool strict_guarantees;
+
+ /*
+ * Last time at which a queue entered the current burst of
+ * queues being activated shortly after each other; for more
+ * details about this and the following parameters related to
+ * a burst of activations, see the comments on the function
+ * bfq_handle_burst.
+ */
+ unsigned long last_ins_in_burst;
+ /*
+ * Reference time interval used to decide whether a queue has
+ * been activated shortly after @last_ins_in_burst.
+ */
+ unsigned long bfq_burst_interval;
+ /* number of queues in the current burst of queue activations */
+ int burst_size;
+
+ /* common parent entity for the queues in the burst */
+ struct bfq_entity *burst_parent_entity;
+ /* Maximum burst size above which the current queue-activation
+ * burst is deemed as 'large'.
+ */
+ unsigned long bfq_large_burst_thresh;
+ /* true if a large queue-activation burst is in progress */
+ bool large_burst;
+ /*
+ * Head of the burst list (as for the above fields, more
+ * details in the comments on the function bfq_handle_burst).
+ */
+ struct hlist_head burst_list;
+
+ /* if set to true, low-latency heuristics are enabled */
+ bool low_latency;
+ /*
+ * Maximum factor by which the weight of a weight-raised queue
+ * is multiplied.
+ */
+ unsigned int bfq_wr_coeff;
+ /* maximum duration of a weight-raising period (jiffies) */
+ unsigned int bfq_wr_max_time;
+
+ /* Maximum weight-raising duration for soft real-time processes */
+ unsigned int bfq_wr_rt_max_time;
+ /*
+ * Minimum idle period after which weight-raising may be
+ * reactivated for a queue (in jiffies).
+ */
+ unsigned int bfq_wr_min_idle_time;
+ /*
+ * Minimum period between request arrivals after which
+ * weight-raising may be reactivated for an already busy async
+ * queue (in jiffies).
+ */
+ unsigned long bfq_wr_min_inter_arr_async;
+
+ /* Max service-rate for a soft real-time queue, in sectors/sec */
+ unsigned int bfq_wr_max_softrt_rate;
+ /*
+ * Cached value of the product R*T, used for computing the
+ * maximum duration of weight raising automatically.
+ */
+ u64 RT_prod;
+ /* device-speed class for the low-latency heuristic */
+ enum bfq_device_speed device_speed;
+
+ /* fallback dummy bfqq for extreme OOM conditions */
+ struct bfq_queue oom_bfqq;
+
+ spinlock_t lock;
+
+ /*
+ * bic associated with the task issuing current bio for
+ * merging. This and the next field are used as a support to
+ * be able to perform the bic lookup, needed by bio-merge
+ * functions, before the scheduler lock is taken, and thus
+ * avoid taking the request-queue lock while the scheduler
+ * lock is being held.
+ */
+ struct bfq_io_cq *bio_bic;
+ /* bfqq associated with the task issuing current bio for merging */
+ struct bfq_queue *bio_bfqq;
+};
+
+enum bfqq_state_flags {
+ BFQQF_just_created = 0, /* queue just allocated */
+ BFQQF_busy, /* has requests or is in service */
+ BFQQF_wait_request, /* waiting for a request */
+ BFQQF_non_blocking_wait_rq, /*
+ * waiting for a request
+ * without idling the device
+ */
+ BFQQF_fifo_expire, /* FIFO checked in this slice */
+ BFQQF_idle_window, /* slice idling enabled */
+ BFQQF_sync, /* synchronous queue */
+ BFQQF_IO_bound, /*
+ * bfqq has timed-out at least once
+ * having consumed at most 2/10 of
+ * its budget
+ */
+ BFQQF_in_large_burst, /*
+ * bfqq activated in a large burst,
+ * see comments to bfq_handle_burst.
+ */
+ BFQQF_softrt_update, /*
+ * may need softrt-next-start
+ * update
+ */
+ BFQQF_coop, /* bfqq is shared */
+ BFQQF_split_coop /* shared bfqq will be split */
+};
+
+#define BFQ_BFQQ_FNS(name) \
+void bfq_mark_bfqq_##name(struct bfq_queue *bfqq); \
+void bfq_clear_bfqq_##name(struct bfq_queue *bfqq); \
+int bfq_bfqq_##name(const struct bfq_queue *bfqq);
+
+BFQ_BFQQ_FNS(just_created);
+BFQ_BFQQ_FNS(busy);
+BFQ_BFQQ_FNS(wait_request);
+BFQ_BFQQ_FNS(non_blocking_wait_rq);
+BFQ_BFQQ_FNS(fifo_expire);
+BFQ_BFQQ_FNS(idle_window);
+BFQ_BFQQ_FNS(sync);
+BFQ_BFQQ_FNS(IO_bound);
+BFQ_BFQQ_FNS(in_large_burst);
+BFQ_BFQQ_FNS(coop);
+BFQ_BFQQ_FNS(split_coop);
+BFQ_BFQQ_FNS(softrt_update);
+#undef BFQ_BFQQ_FNS
+
+/* Expiration reasons. */
+enum bfqq_expiration {
+ BFQQE_TOO_IDLE = 0, /*
+ * queue has been idling for
+ * too long
+ */
+ BFQQE_BUDGET_TIMEOUT, /* budget took too long to be used */
+ BFQQE_BUDGET_EXHAUSTED, /* budget consumed */
+ BFQQE_NO_MORE_REQUESTS, /* the queue has no more requests */
+ BFQQE_PREEMPTED /* preemption in progress */
+};
+
+struct bfqg_stats {
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ /* number of ios merged */
+ struct blkg_rwstat merged;
+ /* total time spent on device in ns, may not be accurate w/ queueing */
+ struct blkg_rwstat service_time;
+ /* total time spent waiting in scheduler queue in ns */
+ struct blkg_rwstat wait_time;
+ /* number of IOs queued up */
+ struct blkg_rwstat queued;
+ /* total disk time and nr sectors dispatched by this group */
+ struct blkg_stat time;
+ /* sum of number of ios queued across all samples */
+ struct blkg_stat avg_queue_size_sum;
+ /* count of samples taken for average */
+ struct blkg_stat avg_queue_size_samples;
+ /* how many times this group has been removed from service tree */
+ struct blkg_stat dequeue;
+ /* total time spent waiting for it to be assigned a timeslice. */
+ struct blkg_stat group_wait_time;
+ /* time spent idling for this blkcg_gq */
+ struct blkg_stat idle_time;
+ /* total time with empty current active q with other requests queued */
+ struct blkg_stat empty_time;
+ /* fields after this shouldn't be cleared on stat reset */
+ uint64_t start_group_wait_time;
+ uint64_t start_idle_time;
+ uint64_t start_empty_time;
+ uint16_t flags;
+#endif /* CONFIG_BFQ_GROUP_IOSCHED */
+};
+
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+
+/*
+ * struct bfq_group_data - per-blkcg storage for the blkio subsystem.
+ *
+ * @ps: @blkcg_policy_storage that this structure inherits
+ * @weight: weight of the bfq_group
+ */
+struct bfq_group_data {
+ /* must be the first member */
+ struct blkcg_policy_data pd;
+
+ unsigned int weight;
+};
+
+/**
+ * struct bfq_group - per (device, cgroup) data structure.
+ * @entity: schedulable entity to insert into the parent group sched_data.
+ * @sched_data: own sched_data, to contain child entities (they may be
+ * both bfq_queues and bfq_groups).
+ * @bfqd: the bfq_data for the device this group acts upon.
+ * @async_bfqq: array of async queues for all the tasks belonging to
+ * the group, one queue per ioprio value per ioprio_class,
+ * except for the idle class that has only one queue.
+ * @async_idle_bfqq: async queue for the idle class (ioprio is ignored).
+ * @my_entity: pointer to @entity, %NULL for the toplevel group; used
+ * to avoid too many special cases during group creation/
+ * migration.
+ * @stats: stats for this bfqg.
+ * @active_entities: number of active entities belonging to the group;
+ * unused for the root group. Used to know whether there
+ * are groups with more than one active @bfq_entity
+ * (see the comments to the function
+ * bfq_bfqq_may_idle()).
+ * @rq_pos_tree: rbtree sorted by next_request position, used when
+ * determining if two or more queues have interleaving
+ * requests (see bfq_find_close_cooperator()).
+ *
+ * Each (device, cgroup) pair has its own bfq_group, i.e., for each cgroup
+ * there is a set of bfq_groups, each one collecting the lower-level
+ * entities belonging to the group that are acting on the same device.
+ *
+ * Locking works as follows:
+ * o @bfqd is protected by the queue lock, RCU is used to access it
+ * from the readers.
+ * o All the other fields are protected by the @bfqd queue lock.
+ */
+struct bfq_group {
+ /* must be the first member */
+ struct blkg_policy_data pd;
+
+ struct bfq_entity entity;
+ struct bfq_sched_data sched_data;
+
+ void *bfqd;
+
+ struct bfq_queue *async_bfqq[2][IOPRIO_BE_NR];
+ struct bfq_queue *async_idle_bfqq;
+
+ struct bfq_entity *my_entity;
+
+ int active_entities;
+
+ struct rb_root rq_pos_tree;
+
+ struct bfqg_stats stats;
+};
+
+#else
+struct bfq_group {
+ struct bfq_sched_data sched_data;
+
+ struct bfq_queue *async_bfqq[2][IOPRIO_BE_NR];
+ struct bfq_queue *async_idle_bfqq;
+
+ struct rb_root rq_pos_tree;
+};
+#endif
+
+struct bfq_queue *bfq_entity_to_bfqq(struct bfq_entity *entity);
+
+/* --------------- main algorithm interface ----------------- */
+
+#define BFQ_SERVICE_TREE_INIT ((struct bfq_service_tree) \
+ { RB_ROOT, RB_ROOT, NULL, NULL, 0, 0 })
+
+extern const int bfq_timeout;
+
+struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync);
+void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync);
+struct bfq_data *bic_to_bfqd(struct bfq_io_cq *bic);
+void bfq_requeue_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq);
+void bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq);
+void bfq_weights_tree_add(struct bfq_data *bfqd, struct bfq_entity *entity,
+ struct rb_root *root);
+void bfq_weights_tree_remove(struct bfq_data *bfqd, struct bfq_entity *entity,
+ struct rb_root *root);
+void bfq_bfqq_expire(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ bool compensate, enum bfqq_expiration reason);
+void bfq_put_queue(struct bfq_queue *bfqq);
+void bfq_end_wr_async_queues(struct bfq_data *bfqd, struct bfq_group *bfqg);
+void bfq_schedule_dispatch(struct bfq_data *bfqd);
+void bfq_put_async_queues(struct bfq_data *bfqd, struct bfq_group *bfqg);
+
+/* ------------ end of main algorithm interface -------------- */
+
+/* ---------------- cgroups-support interface ---------------- */
+
+void bfqg_stats_update_io_add(struct bfq_group *bfqg, struct bfq_queue *bfqq,
+ unsigned int op);
+void bfqg_stats_update_io_remove(struct bfq_group *bfqg, unsigned int op);
+void bfqg_stats_update_io_merged(struct bfq_group *bfqg, unsigned int op);
+void bfqg_stats_update_completion(struct bfq_group *bfqg, uint64_t start_time,
+ uint64_t io_start_time, unsigned int op);
+void bfqg_stats_update_dequeue(struct bfq_group *bfqg);
+void bfqg_stats_set_start_empty_time(struct bfq_group *bfqg);
+void bfqg_stats_update_idle_time(struct bfq_group *bfqg);
+void bfqg_stats_set_start_idle_time(struct bfq_group *bfqg);
+void bfqg_stats_update_avg_queue_size(struct bfq_group *bfqg);
+void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ struct bfq_group *bfqg);
+
+void bfq_init_entity(struct bfq_entity *entity, struct bfq_group *bfqg);
+void bfq_bic_update_cgroup(struct bfq_io_cq *bic, struct bio *bio);
+void bfq_end_wr_async(struct bfq_data *bfqd);
+struct bfq_group *bfq_find_set_group(struct bfq_data *bfqd,
+ struct blkcg *blkcg);
+struct blkcg_gq *bfqg_to_blkg(struct bfq_group *bfqg);
+struct bfq_group *bfqq_group(struct bfq_queue *bfqq);
+struct bfq_group *bfq_create_group_hierarchy(struct bfq_data *bfqd, int node);
+void bfqg_put(struct bfq_group *bfqg);
+
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+extern struct cftype bfq_blkcg_legacy_files[];
+extern struct cftype bfq_blkg_files[];
+extern struct blkcg_policy blkcg_policy_bfq;
+#endif
+
+/* ------------- end of cgroups-support interface ------------- */
+
+/* - interface of the internal hierarchical B-WF2Q+ scheduler - */
+
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+/* both next loops stop at one of the child entities of the root group */
+#define for_each_entity(entity) \
+ for (; entity ; entity = entity->parent)
+
+/*
+ * For each iteration, compute parent in advance, so as to be safe if
+ * entity is deallocated during the iteration. Such a deallocation may
+ * happen as a consequence of a bfq_put_queue that frees the bfq_queue
+ * containing entity.
+ */
+#define for_each_entity_safe(entity, parent) \
+ for (; entity && ({ parent = entity->parent; 1; }); entity = parent)
+
+#else /* CONFIG_BFQ_GROUP_IOSCHED */
+/*
+ * Next two macros are fake loops when cgroups support is not
+ * enabled. I fact, in such a case, there is only one level to go up
+ * (to reach the root group).
+ */
+#define for_each_entity(entity) \
+ for (; entity ; entity = NULL)
+
+#define for_each_entity_safe(entity, parent) \
+ for (parent = NULL; entity ; entity = parent)
+#endif /* CONFIG_BFQ_GROUP_IOSCHED */
+
+struct bfq_group *bfq_bfqq_to_bfqg(struct bfq_queue *bfqq);
+struct bfq_queue *bfq_entity_to_bfqq(struct bfq_entity *entity);
+struct bfq_service_tree *bfq_entity_service_tree(struct bfq_entity *entity);
+struct bfq_entity *bfq_entity_of(struct rb_node *node);
+unsigned short bfq_ioprio_to_weight(int ioprio);
+void bfq_put_idle_entity(struct bfq_service_tree *st,
+ struct bfq_entity *entity);
+struct bfq_service_tree *
+__bfq_entity_update_weight_prio(struct bfq_service_tree *old_st,
+ struct bfq_entity *entity);
+void bfq_bfqq_served(struct bfq_queue *bfqq, int served);
+void bfq_bfqq_charge_time(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ unsigned long time_ms);
+bool __bfq_deactivate_entity(struct bfq_entity *entity,
+ bool ins_into_idle_tree);
+bool next_queue_may_preempt(struct bfq_data *bfqd);
+struct bfq_queue *bfq_get_next_queue(struct bfq_data *bfqd);
+void __bfq_bfqd_reset_in_service(struct bfq_data *bfqd);
+void bfq_deactivate_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ bool ins_into_idle_tree, bool expiration);
+void bfq_activate_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq);
+void bfq_requeue_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq);
+void bfq_del_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ bool expiration);
+void bfq_add_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq);
+
+/* --------------- end of interface of B-WF2Q+ ---------------- */
+
+/* Logging facilities. */
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+struct bfq_group *bfqq_group(struct bfq_queue *bfqq);
+
+#define bfq_log_bfqq(bfqd, bfqq, fmt, args...) do { \
+ char __pbuf[128]; \
+ \
+ blkg_path(bfqg_to_blkg(bfqq_group(bfqq)), __pbuf, sizeof(__pbuf)); \
+ blk_add_trace_msg((bfqd)->queue, "bfq%d%c %s " fmt, (bfqq)->pid, \
+ bfq_bfqq_sync((bfqq)) ? 'S' : 'A', \
+ __pbuf, ##args); \
+} while (0)
+
+#define bfq_log_bfqg(bfqd, bfqg, fmt, args...) do { \
+ char __pbuf[128]; \
+ \
+ blkg_path(bfqg_to_blkg(bfqg), __pbuf, sizeof(__pbuf)); \
+ blk_add_trace_msg((bfqd)->queue, "%s " fmt, __pbuf, ##args); \
+} while (0)
+
+#else /* CONFIG_BFQ_GROUP_IOSCHED */
+
+#define bfq_log_bfqq(bfqd, bfqq, fmt, args...) \
+ blk_add_trace_msg((bfqd)->queue, "bfq%d%c " fmt, (bfqq)->pid, \
+ bfq_bfqq_sync((bfqq)) ? 'S' : 'A', \
+ ##args)
+#define bfq_log_bfqg(bfqd, bfqg, fmt, args...) do {} while (0)
+
+#endif /* CONFIG_BFQ_GROUP_IOSCHED */
+
+#define bfq_log(bfqd, fmt, args...) \
+ blk_add_trace_msg((bfqd)->queue, "bfq " fmt, ##args)
+
+#endif /* _BFQ_H */
diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c
new file mode 100644
index 0000000..b4fc3e4
--- /dev/null
+++ b/block/bfq-wf2q.c
@@ -0,0 +1,1616 @@
+/*
+ * Hierarchical Budget Worst-case Fair Weighted Fair Queueing
+ * (B-WF2Q+): hierarchical scheduling algorithm by which the BFQ I/O
+ * scheduler schedules generic entities. The latter can represent
+ * either single bfq queues (associated with processes) or groups of
+ * bfq queues (associated with cgroups).
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#include "bfq-iosched.h"
+
+/**
+ * bfq_gt - compare two timestamps.
+ * @a: first ts.
+ * @b: second ts.
+ *
+ * Return @a > @b, dealing with wrapping correctly.
+ */
+static int bfq_gt(u64 a, u64 b)
+{
+ return (s64)(a - b) > 0;
+}
+
+static struct bfq_entity *bfq_root_active_entity(struct rb_root *tree)
+{
+ struct rb_node *node = tree->rb_node;
+
+ return rb_entry(node, struct bfq_entity, rb_node);
+}
+
+static unsigned int bfq_class_idx(struct bfq_entity *entity)
+{
+ struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity);
+
+ return bfqq ? bfqq->ioprio_class - 1 :
+ BFQ_DEFAULT_GRP_CLASS - 1;
+}
+
+static struct bfq_entity *bfq_lookup_next_entity(struct bfq_sched_data *sd);
+
+static bool bfq_update_parent_budget(struct bfq_entity *next_in_service);
+
+/**
+ * bfq_update_next_in_service - update sd->next_in_service
+ * @sd: sched_data for which to perform the update.
+ * @new_entity: if not NULL, pointer to the entity whose activation,
+ * requeueing or repositionig triggered the invocation of
+ * this function.
+ *
+ * This function is called to update sd->next_in_service, which, in
+ * its turn, may change as a consequence of the insertion or
+ * extraction of an entity into/from one of the active trees of
+ * sd. These insertions/extractions occur as a consequence of
+ * activations/deactivations of entities, with some activations being
+ * 'true' activations, and other activations being requeueings (i.e.,
+ * implementing the second, requeueing phase of the mechanism used to
+ * reposition an entity in its active tree; see comments on
+ * __bfq_activate_entity and __bfq_requeue_entity for details). In
+ * both the last two activation sub-cases, new_entity points to the
+ * just activated or requeued entity.
+ *
+ * Returns true if sd->next_in_service changes in such a way that
+ * entity->parent may become the next_in_service for its parent
+ * entity.
+ */
+static bool bfq_update_next_in_service(struct bfq_sched_data *sd,
+ struct bfq_entity *new_entity)
+{
+ struct bfq_entity *next_in_service = sd->next_in_service;
+ bool parent_sched_may_change = false;
+
+ /*
+ * If this update is triggered by the activation, requeueing
+ * or repositiong of an entity that does not coincide with
+ * sd->next_in_service, then a full lookup in the active tree
+ * can be avoided. In fact, it is enough to check whether the
+ * just-modified entity has a higher priority than
+ * sd->next_in_service, or, even if it has the same priority
+ * as sd->next_in_service, is eligible and has a lower virtual
+ * finish time than sd->next_in_service. If this compound
+ * condition holds, then the new entity becomes the new
+ * next_in_service. Otherwise no change is needed.
+ */
+ if (new_entity && new_entity != sd->next_in_service) {
+ /*
+ * Flag used to decide whether to replace
+ * sd->next_in_service with new_entity. Tentatively
+ * set to true, and left as true if
+ * sd->next_in_service is NULL.
+ */
+ bool replace_next = true;
+
+ /*
+ * If there is already a next_in_service candidate
+ * entity, then compare class priorities or timestamps
+ * to decide whether to replace sd->service_tree with
+ * new_entity.
+ */
+ if (next_in_service) {
+ unsigned int new_entity_class_idx =
+ bfq_class_idx(new_entity);
+ struct bfq_service_tree *st =
+ sd->service_tree + new_entity_class_idx;
+
+ /*
+ * For efficiency, evaluate the most likely
+ * sub-condition first.
+ */
+ replace_next =
+ (new_entity_class_idx ==
+ bfq_class_idx(next_in_service)
+ &&
+ !bfq_gt(new_entity->start, st->vtime)
+ &&
+ bfq_gt(next_in_service->finish,
+ new_entity->finish))
+ ||
+ new_entity_class_idx <
+ bfq_class_idx(next_in_service);
+ }
+
+ if (replace_next)
+ next_in_service = new_entity;
+ } else /* invoked because of a deactivation: lookup needed */
+ next_in_service = bfq_lookup_next_entity(sd);
+
+ if (next_in_service) {
+ parent_sched_may_change = !sd->next_in_service ||
+ bfq_update_parent_budget(next_in_service);
+ }
+
+ sd->next_in_service = next_in_service;
+
+ if (!next_in_service)
+ return parent_sched_may_change;
+
+ return parent_sched_may_change;
+}
+
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+
+struct bfq_group *bfq_bfqq_to_bfqg(struct bfq_queue *bfqq)
+{
+ struct bfq_entity *group_entity = bfqq->entity.parent;
+
+ if (!group_entity)
+ group_entity = &bfqq->bfqd->root_group->entity;
+
+ return container_of(group_entity, struct bfq_group, entity);
+}
+
+/*
+ * Returns true if this budget changes may let next_in_service->parent
+ * become the next_in_service entity for its parent entity.
+ */
+static bool bfq_update_parent_budget(struct bfq_entity *next_in_service)
+{
+ struct bfq_entity *bfqg_entity;
+ struct bfq_group *bfqg;
+ struct bfq_sched_data *group_sd;
+ bool ret = false;
+
+ group_sd = next_in_service->sched_data;
+
+ bfqg = container_of(group_sd, struct bfq_group, sched_data);
+ /*
+ * bfq_group's my_entity field is not NULL only if the group
+ * is not the root group. We must not touch the root entity
+ * as it must never become an in-service entity.
+ */
+ bfqg_entity = bfqg->my_entity;
+ if (bfqg_entity) {
+ if (bfqg_entity->budget > next_in_service->budget)
+ ret = true;
+ bfqg_entity->budget = next_in_service->budget;
+ }
+
+ return ret;
+}
+
+/*
+ * This function tells whether entity stops being a candidate for next
+ * service, according to the following logic.
+ *
+ * This function is invoked for an entity that is about to be set in
+ * service. If such an entity is a queue, then the entity is no longer
+ * a candidate for next service (i.e, a candidate entity to serve
+ * after the in-service entity is expired). The function then returns
+ * true.
+ *
+ * In contrast, the entity could stil be a candidate for next service
+ * if it is not a queue, and has more than one child. In fact, even if
+ * one of its children is about to be set in service, other children
+ * may still be the next to serve. As a consequence, a non-queue
+ * entity is not a candidate for next-service only if it has only one
+ * child. And only if this condition holds, then the function returns
+ * true for a non-queue entity.
+ */
+static bool bfq_no_longer_next_in_service(struct bfq_entity *entity)
+{
+ struct bfq_group *bfqg;
+
+ if (bfq_entity_to_bfqq(entity))
+ return true;
+
+ bfqg = container_of(entity, struct bfq_group, entity);
+
+ if (bfqg->active_entities == 1)
+ return true;
+
+ return false;
+}
+
+#else /* CONFIG_BFQ_GROUP_IOSCHED */
+
+struct bfq_group *bfq_bfqq_to_bfqg(struct bfq_queue *bfqq)
+{
+ return bfqq->bfqd->root_group;
+}
+
+static bool bfq_update_parent_budget(struct bfq_entity *next_in_service)
+{
+ return false;
+}
+
+static bool bfq_no_longer_next_in_service(struct bfq_entity *entity)
+{
+ return true;
+}
+
+#endif /* CONFIG_BFQ_GROUP_IOSCHED */
+
+/*
+ * Shift for timestamp calculations. This actually limits the maximum
+ * service allowed in one timestamp delta (small shift values increase it),
+ * the maximum total weight that can be used for the queues in the system
+ * (big shift values increase it), and the period of virtual time
+ * wraparounds.
+ */
+#define WFQ_SERVICE_SHIFT 22
+
+struct bfq_queue *bfq_entity_to_bfqq(struct bfq_entity *entity)
+{
+ struct bfq_queue *bfqq = NULL;
+
+ if (!entity->my_sched_data)
+ bfqq = container_of(entity, struct bfq_queue, entity);
+
+ return bfqq;
+}
+
+
+/**
+ * bfq_delta - map service into the virtual time domain.
+ * @service: amount of service.
+ * @weight: scale factor (weight of an entity or weight sum).
+ */
+static u64 bfq_delta(unsigned long service, unsigned long weight)
+{
+ u64 d = (u64)service << WFQ_SERVICE_SHIFT;
+
+ do_div(d, weight);
+ return d;
+}
+
+/**
+ * bfq_calc_finish - assign the finish time to an entity.
+ * @entity: the entity to act upon.
+ * @service: the service to be charged to the entity.
+ */
+static void bfq_calc_finish(struct bfq_entity *entity, unsigned long service)
+{
+ struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity);
+
+ entity->finish = entity->start +
+ bfq_delta(service, entity->weight);
+
+ if (bfqq) {
+ bfq_log_bfqq(bfqq->bfqd, bfqq,
+ "calc_finish: serv %lu, w %d",
+ service, entity->weight);
+ bfq_log_bfqq(bfqq->bfqd, bfqq,
+ "calc_finish: start %llu, finish %llu, delta %llu",
+ entity->start, entity->finish,
+ bfq_delta(service, entity->weight));
+ }
+}
+
+/**
+ * bfq_entity_of - get an entity from a node.
+ * @node: the node field of the entity.
+ *
+ * Convert a node pointer to the relative entity. This is used only
+ * to simplify the logic of some functions and not as the generic
+ * conversion mechanism because, e.g., in the tree walking functions,
+ * the check for a %NULL value would be redundant.
+ */
+struct bfq_entity *bfq_entity_of(struct rb_node *node)
+{
+ struct bfq_entity *entity = NULL;
+
+ if (node)
+ entity = rb_entry(node, struct bfq_entity, rb_node);
+
+ return entity;
+}
+
+/**
+ * bfq_extract - remove an entity from a tree.
+ * @root: the tree root.
+ * @entity: the entity to remove.
+ */
+static void bfq_extract(struct rb_root *root, struct bfq_entity *entity)
+{
+ entity->tree = NULL;
+ rb_erase(&entity->rb_node, root);
+}
+
+/**
+ * bfq_idle_extract - extract an entity from the idle tree.
+ * @st: the service tree of the owning @entity.
+ * @entity: the entity being removed.
+ */
+static void bfq_idle_extract(struct bfq_service_tree *st,
+ struct bfq_entity *entity)
+{
+ struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity);
+ struct rb_node *next;
+
+ if (entity == st->first_idle) {
+ next = rb_next(&entity->rb_node);
+ st->first_idle = bfq_entity_of(next);
+ }
+
+ if (entity == st->last_idle) {
+ next = rb_prev(&entity->rb_node);
+ st->last_idle = bfq_entity_of(next);
+ }
+
+ bfq_extract(&st->idle, entity);
+
+ if (bfqq)
+ list_del(&bfqq->bfqq_list);
+}
+
+/**
+ * bfq_insert - generic tree insertion.
+ * @root: tree root.
+ * @entity: entity to insert.
+ *
+ * This is used for the idle and the active tree, since they are both
+ * ordered by finish time.
+ */
+static void bfq_insert(struct rb_root *root, struct bfq_entity *entity)
+{
+ struct bfq_entity *entry;
+ struct rb_node **node = &root->rb_node;
+ struct rb_node *parent = NULL;
+
+ while (*node) {
+ parent = *node;
+ entry = rb_entry(parent, struct bfq_entity, rb_node);
+
+ if (bfq_gt(entry->finish, entity->finish))
+ node = &parent->rb_left;
+ else
+ node = &parent->rb_right;
+ }
+
+ rb_link_node(&entity->rb_node, parent, node);
+ rb_insert_color(&entity->rb_node, root);
+
+ entity->tree = root;
+}
+
+/**
+ * bfq_update_min - update the min_start field of a entity.
+ * @entity: the entity to update.
+ * @node: one of its children.
+ *
+ * This function is called when @entity may store an invalid value for
+ * min_start due to updates to the active tree. The function assumes
+ * that the subtree rooted at @node (which may be its left or its right
+ * child) has a valid min_start value.
+ */
+static void bfq_update_min(struct bfq_entity *entity, struct rb_node *node)
+{
+ struct bfq_entity *child;
+
+ if (node) {
+ child = rb_entry(node, struct bfq_entity, rb_node);
+ if (bfq_gt(entity->min_start, child->min_start))
+ entity->min_start = child->min_start;
+ }
+}
+
+/**
+ * bfq_update_active_node - recalculate min_start.
+ * @node: the node to update.
+ *
+ * @node may have changed position or one of its children may have moved,
+ * this function updates its min_start value. The left and right subtrees
+ * are assumed to hold a correct min_start value.
+ */
+static void bfq_update_active_node(struct rb_node *node)
+{
+ struct bfq_entity *entity = rb_entry(node, struct bfq_entity, rb_node);
+
+ entity->min_start = entity->start;
+ bfq_update_min(entity, node->rb_right);
+ bfq_update_min(entity, node->rb_left);
+}
+
+/**
+ * bfq_update_active_tree - update min_start for the whole active tree.
+ * @node: the starting node.
+ *
+ * @node must be the deepest modified node after an update. This function
+ * updates its min_start using the values held by its children, assuming
+ * that they did not change, and then updates all the nodes that may have
+ * changed in the path to the root. The only nodes that may have changed
+ * are the ones in the path or their siblings.
+ */
+static void bfq_update_active_tree(struct rb_node *node)
+{
+ struct rb_node *parent;
+
+up:
+ bfq_update_active_node(node);
+
+ parent = rb_parent(node);
+ if (!parent)
+ return;
+
+ if (node == parent->rb_left && parent->rb_right)
+ bfq_update_active_node(parent->rb_right);
+ else if (parent->rb_left)
+ bfq_update_active_node(parent->rb_left);
+
+ node = parent;
+ goto up;
+}
+
+/**
+ * bfq_active_insert - insert an entity in the active tree of its
+ * group/device.
+ * @st: the service tree of the entity.
+ * @entity: the entity being inserted.
+ *
+ * The active tree is ordered by finish time, but an extra key is kept
+ * per each node, containing the minimum value for the start times of
+ * its children (and the node itself), so it's possible to search for
+ * the eligible node with the lowest finish time in logarithmic time.
+ */
+static void bfq_active_insert(struct bfq_service_tree *st,
+ struct bfq_entity *entity)
+{
+ struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity);
+ struct rb_node *node = &entity->rb_node;
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ struct bfq_sched_data *sd = NULL;
+ struct bfq_group *bfqg = NULL;
+ struct bfq_data *bfqd = NULL;
+#endif
+
+ bfq_insert(&st->active, entity);
+
+ if (node->rb_left)
+ node = node->rb_left;
+ else if (node->rb_right)
+ node = node->rb_right;
+
+ bfq_update_active_tree(node);
+
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ sd = entity->sched_data;
+ bfqg = container_of(sd, struct bfq_group, sched_data);
+ bfqd = (struct bfq_data *)bfqg->bfqd;
+#endif
+ if (bfqq)
+ list_add(&bfqq->bfqq_list, &bfqq->bfqd->active_list);
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ else /* bfq_group */
+ bfq_weights_tree_add(bfqd, entity, &bfqd->group_weights_tree);
+
+ if (bfqg != bfqd->root_group)
+ bfqg->active_entities++;
+#endif
+}
+
+/**
+ * bfq_ioprio_to_weight - calc a weight from an ioprio.
+ * @ioprio: the ioprio value to convert.
+ */
+unsigned short bfq_ioprio_to_weight(int ioprio)
+{
+ return (IOPRIO_BE_NR - ioprio) * BFQ_WEIGHT_CONVERSION_COEFF;
+}
+
+/**
+ * bfq_weight_to_ioprio - calc an ioprio from a weight.
+ * @weight: the weight value to convert.
+ *
+ * To preserve as much as possible the old only-ioprio user interface,
+ * 0 is used as an escape ioprio value for weights (numerically) equal or
+ * larger than IOPRIO_BE_NR * BFQ_WEIGHT_CONVERSION_COEFF.
+ */
+static unsigned short bfq_weight_to_ioprio(int weight)
+{
+ return max_t(int, 0,
+ IOPRIO_BE_NR * BFQ_WEIGHT_CONVERSION_COEFF - weight);
+}
+
+static void bfq_get_entity(struct bfq_entity *entity)
+{
+ struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity);
+
+ if (bfqq) {
+ bfqq->ref++;
+ bfq_log_bfqq(bfqq->bfqd, bfqq, "get_entity: %p %d",
+ bfqq, bfqq->ref);
+ }
+}
+
+/**
+ * bfq_find_deepest - find the deepest node that an extraction can modify.
+ * @node: the node being removed.
+ *
+ * Do the first step of an extraction in an rb tree, looking for the
+ * node that will replace @node, and returning the deepest node that
+ * the following modifications to the tree can touch. If @node is the
+ * last node in the tree return %NULL.
+ */
+static struct rb_node *bfq_find_deepest(struct rb_node *node)
+{
+ struct rb_node *deepest;
+
+ if (!node->rb_right && !node->rb_left)
+ deepest = rb_parent(node);
+ else if (!node->rb_right)
+ deepest = node->rb_left;
+ else if (!node->rb_left)
+ deepest = node->rb_right;
+ else {
+ deepest = rb_next(node);
+ if (deepest->rb_right)
+ deepest = deepest->rb_right;
+ else if (rb_parent(deepest) != node)
+ deepest = rb_parent(deepest);
+ }
+
+ return deepest;
+}
+
+/**
+ * bfq_active_extract - remove an entity from the active tree.
+ * @st: the service_tree containing the tree.
+ * @entity: the entity being removed.
+ */
+static void bfq_active_extract(struct bfq_service_tree *st,
+ struct bfq_entity *entity)
+{
+ struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity);
+ struct rb_node *node;
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ struct bfq_sched_data *sd = NULL;
+ struct bfq_group *bfqg = NULL;
+ struct bfq_data *bfqd = NULL;
+#endif
+
+ node = bfq_find_deepest(&entity->rb_node);
+ bfq_extract(&st->active, entity);
+
+ if (node)
+ bfq_update_active_tree(node);
+
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ sd = entity->sched_data;
+ bfqg = container_of(sd, struct bfq_group, sched_data);
+ bfqd = (struct bfq_data *)bfqg->bfqd;
+#endif
+ if (bfqq)
+ list_del(&bfqq->bfqq_list);
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ else /* bfq_group */
+ bfq_weights_tree_remove(bfqd, entity,
+ &bfqd->group_weights_tree);
+
+ if (bfqg != bfqd->root_group)
+ bfqg->active_entities--;
+#endif
+}
+
+/**
+ * bfq_idle_insert - insert an entity into the idle tree.
+ * @st: the service tree containing the tree.
+ * @entity: the entity to insert.
+ */
+static void bfq_idle_insert(struct bfq_service_tree *st,
+ struct bfq_entity *entity)
+{
+ struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity);
+ struct bfq_entity *first_idle = st->first_idle;
+ struct bfq_entity *last_idle = st->last_idle;
+
+ if (!first_idle || bfq_gt(first_idle->finish, entity->finish))
+ st->first_idle = entity;
+ if (!last_idle || bfq_gt(entity->finish, last_idle->finish))
+ st->last_idle = entity;
+
+ bfq_insert(&st->idle, entity);
+
+ if (bfqq)
+ list_add(&bfqq->bfqq_list, &bfqq->bfqd->idle_list);
+}
+
+/**
+ * bfq_forget_entity - do not consider entity any longer for scheduling
+ * @st: the service tree.
+ * @entity: the entity being removed.
+ * @is_in_service: true if entity is currently the in-service entity.
+ *
+ * Forget everything about @entity. In addition, if entity represents
+ * a queue, and the latter is not in service, then release the service
+ * reference to the queue (the one taken through bfq_get_entity). In
+ * fact, in this case, there is really no more service reference to
+ * the queue, as the latter is also outside any service tree. If,
+ * instead, the queue is in service, then __bfq_bfqd_reset_in_service
+ * will take care of putting the reference when the queue finally
+ * stops being served.
+ */
+static void bfq_forget_entity(struct bfq_service_tree *st,
+ struct bfq_entity *entity,
+ bool is_in_service)
+{
+ struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity);
+
+ entity->on_st = false;
+ st->wsum -= entity->weight;
+ if (bfqq && !is_in_service)
+ bfq_put_queue(bfqq);
+}
+
+/**
+ * bfq_put_idle_entity - release the idle tree ref of an entity.
+ * @st: service tree for the entity.
+ * @entity: the entity being released.
+ */
+void bfq_put_idle_entity(struct bfq_service_tree *st, struct bfq_entity *entity)
+{
+ bfq_idle_extract(st, entity);
+ bfq_forget_entity(st, entity,
+ entity == entity->sched_data->in_service_entity);
+}
+
+/**
+ * bfq_forget_idle - update the idle tree if necessary.
+ * @st: the service tree to act upon.
+ *
+ * To preserve the global O(log N) complexity we only remove one entry here;
+ * as the idle tree will not grow indefinitely this can be done safely.
+ */
+static void bfq_forget_idle(struct bfq_service_tree *st)
+{
+ struct bfq_entity *first_idle = st->first_idle;
+ struct bfq_entity *last_idle = st->last_idle;
+
+ if (RB_EMPTY_ROOT(&st->active) && last_idle &&
+ !bfq_gt(last_idle->finish, st->vtime)) {
+ /*
+ * Forget the whole idle tree, increasing the vtime past
+ * the last finish time of idle entities.
+ */
+ st->vtime = last_idle->finish;
+ }
+
+ if (first_idle && !bfq_gt(first_idle->finish, st->vtime))
+ bfq_put_idle_entity(st, first_idle);
+}
+
+struct bfq_service_tree *bfq_entity_service_tree(struct bfq_entity *entity)
+{
+ struct bfq_sched_data *sched_data = entity->sched_data;
+ unsigned int idx = bfq_class_idx(entity);
+
+ return sched_data->service_tree + idx;
+}
+
+
+struct bfq_service_tree *
+__bfq_entity_update_weight_prio(struct bfq_service_tree *old_st,
+ struct bfq_entity *entity)
+{
+ struct bfq_service_tree *new_st = old_st;
+
+ if (entity->prio_changed) {
+ struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity);
+ unsigned int prev_weight, new_weight;
+ struct bfq_data *bfqd = NULL;
+ struct rb_root *root;
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ struct bfq_sched_data *sd;
+ struct bfq_group *bfqg;
+#endif
+
+ if (bfqq)
+ bfqd = bfqq->bfqd;
+#ifdef CONFIG_BFQ_GROUP_IOSCHED
+ else {
+ sd = entity->my_sched_data;
+ bfqg = container_of(sd, struct bfq_group, sched_data);
+ bfqd = (struct bfq_data *)bfqg->bfqd;
+ }
+#endif
+
+ old_st->wsum -= entity->weight;
+
+ if (entity->new_weight != entity->orig_weight) {
+ if (entity->new_weight < BFQ_MIN_WEIGHT ||
+ entity->new_weight > BFQ_MAX_WEIGHT) {
+ pr_crit("update_weight_prio: new_weight %d\n",
+ entity->new_weight);
+ if (entity->new_weight < BFQ_MIN_WEIGHT)
+ entity->new_weight = BFQ_MIN_WEIGHT;
+ else
+ entity->new_weight = BFQ_MAX_WEIGHT;
+ }
+ entity->orig_weight = entity->new_weight;
+ if (bfqq)
+ bfqq->ioprio =
+ bfq_weight_to_ioprio(entity->orig_weight);
+ }
+
+ if (bfqq)
+ bfqq->ioprio_class = bfqq->new_ioprio_class;
+ entity->prio_changed = 0;
+
+ /*
+ * NOTE: here we may be changing the weight too early,
+ * this will cause unfairness. The correct approach
+ * would have required additional complexity to defer
+ * weight changes to the proper time instants (i.e.,
+ * when entity->finish <= old_st->vtime).
+ */
+ new_st = bfq_entity_service_tree(entity);
+
+ prev_weight = entity->weight;
+ new_weight = entity->orig_weight *
+ (bfqq ? bfqq->wr_coeff : 1);
+ /*
+ * If the weight of the entity changes, remove the entity
+ * from its old weight counter (if there is a counter
+ * associated with the entity), and add it to the counter
+ * associated with its new weight.
+ */
+ if (prev_weight != new_weight) {
+ root = bfqq ? &bfqd->queue_weights_tree :
+ &bfqd->group_weights_tree;
+ bfq_weights_tree_remove(bfqd, entity, root);
+ }
+ entity->weight = new_weight;
+ /*
+ * Add the entity to its weights tree only if it is
+ * not associated with a weight-raised queue.
+ */
+ if (prev_weight != new_weight &&
+ (bfqq ? bfqq->wr_coeff == 1 : 1))
+ /* If we get here, root has been initialized. */
+ bfq_weights_tree_add(bfqd, entity, root);
+
+ new_st->wsum += entity->weight;
+
+ if (new_st != old_st)
+ entity->start = new_st->vtime;
+ }
+
+ return new_st;
+}
+
+/**
+ * bfq_bfqq_served - update the scheduler status after selection for
+ * service.
+ * @bfqq: the queue being served.
+ * @served: bytes to transfer.
+ *
+ * NOTE: this can be optimized, as the timestamps of upper level entities
+ * are synchronized every time a new bfqq is selected for service. By now,
+ * we keep it to better check consistency.
+ */
+void bfq_bfqq_served(struct bfq_queue *bfqq, int served)
+{
+ struct bfq_entity *entity = &bfqq->entity;
+ struct bfq_service_tree *st;
+
+ for_each_entity(entity) {
+ st = bfq_entity_service_tree(entity);
+
+ entity->service += served;
+
+ st->vtime += bfq_delta(served, st->wsum);
+ bfq_forget_idle(st);
+ }
+ bfqg_stats_set_start_empty_time(bfqq_group(bfqq));
+ bfq_log_bfqq(bfqq->bfqd, bfqq, "bfqq_served %d secs", served);
+}
+
+/**
+ * bfq_bfqq_charge_time - charge an amount of service equivalent to the length
+ * of the time interval during which bfqq has been in
+ * service.
+ * @bfqd: the device
+ * @bfqq: the queue that needs a service update.
+ * @time_ms: the amount of time during which the queue has received service
+ *
+ * If a queue does not consume its budget fast enough, then providing
+ * the queue with service fairness may impair throughput, more or less
+ * severely. For this reason, queues that consume their budget slowly
+ * are provided with time fairness instead of service fairness. This
+ * goal is achieved through the BFQ scheduling engine, even if such an
+ * engine works in the service, and not in the time domain. The trick
+ * is charging these queues with an inflated amount of service, equal
+ * to the amount of service that they would have received during their
+ * service slot if they had been fast, i.e., if their requests had
+ * been dispatched at a rate equal to the estimated peak rate.
+ *
+ * It is worth noting that time fairness can cause important
+ * distortions in terms of bandwidth distribution, on devices with
+ * internal queueing. The reason is that I/O requests dispatched
+ * during the service slot of a queue may be served after that service
+ * slot is finished, and may have a total processing time loosely
+ * correlated with the duration of the service slot. This is
+ * especially true for short service slots.
+ */
+void bfq_bfqq_charge_time(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ unsigned long time_ms)
+{
+ struct bfq_entity *entity = &bfqq->entity;
+ int tot_serv_to_charge = entity->service;
+ unsigned int timeout_ms = jiffies_to_msecs(bfq_timeout);
+
+ if (time_ms > 0 && time_ms < timeout_ms)
+ tot_serv_to_charge =
+ (bfqd->bfq_max_budget * time_ms) / timeout_ms;
+
+ if (tot_serv_to_charge < entity->service)
+ tot_serv_to_charge = entity->service;
+
+ /* Increase budget to avoid inconsistencies */
+ if (tot_serv_to_charge > entity->budget)
+ entity->budget = tot_serv_to_charge;
+
+ bfq_bfqq_served(bfqq,
+ max_t(int, 0, tot_serv_to_charge - entity->service));
+}
+
+static void bfq_update_fin_time_enqueue(struct bfq_entity *entity,
+ struct bfq_service_tree *st,
+ bool backshifted)
+{
+ struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity);
+
+ st = __bfq_entity_update_weight_prio(st, entity);
+ bfq_calc_finish(entity, entity->budget);
+
+ /*
+ * If some queues enjoy backshifting for a while, then their
+ * (virtual) finish timestamps may happen to become lower and
+ * lower than the system virtual time. In particular, if
+ * these queues often happen to be idle for short time
+ * periods, and during such time periods other queues with
+ * higher timestamps happen to be busy, then the backshifted
+ * timestamps of the former queues can become much lower than
+ * the system virtual time. In fact, to serve the queues with
+ * higher timestamps while the ones with lower timestamps are
+ * idle, the system virtual time may be pushed-up to much
+ * higher values than the finish timestamps of the idle
+ * queues. As a consequence, the finish timestamps of all new
+ * or newly activated queues may end up being much larger than
+ * those of lucky queues with backshifted timestamps. The
+ * latter queues may then monopolize the device for a lot of
+ * time. This would simply break service guarantees.
+ *
+ * To reduce this problem, push up a little bit the
+ * backshifted timestamps of the queue associated with this
+ * entity (only a queue can happen to have the backshifted
+ * flag set): just enough to let the finish timestamp of the
+ * queue be equal to the current value of the system virtual
+ * time. This may introduce a little unfairness among queues
+ * with backshifted timestamps, but it does not break
+ * worst-case fairness guarantees.
+ *
+ * As a special case, if bfqq is weight-raised, push up
+ * timestamps much less, to keep very low the probability that
+ * this push up causes the backshifted finish timestamps of
+ * weight-raised queues to become higher than the backshifted
+ * finish timestamps of non weight-raised queues.
+ */
+ if (backshifted && bfq_gt(st->vtime, entity->finish)) {
+ unsigned long delta = st->vtime - entity->finish;
+
+ if (bfqq)
+ delta /= bfqq->wr_coeff;
+
+ entity->start += delta;
+ entity->finish += delta;
+ }
+
+ bfq_active_insert(st, entity);
+}
+
+/**
+ * __bfq_activate_entity - handle activation of entity.
+ * @entity: the entity being activated.
+ * @non_blocking_wait_rq: true if entity was waiting for a request
+ *
+ * Called for a 'true' activation, i.e., if entity is not active and
+ * one of its children receives a new request.
+ *
+ * Basically, this function updates the timestamps of entity and
+ * inserts entity into its active tree, ater possible extracting it
+ * from its idle tree.
+ */
+static void __bfq_activate_entity(struct bfq_entity *entity,
+ bool non_blocking_wait_rq)
+{
+ struct bfq_service_tree *st = bfq_entity_service_tree(entity);
+ bool backshifted = false;
+ unsigned long long min_vstart;
+
+ /* See comments on bfq_fqq_update_budg_for_activation */
+ if (non_blocking_wait_rq && bfq_gt(st->vtime, entity->finish)) {
+ backshifted = true;
+ min_vstart = entity->finish;
+ } else
+ min_vstart = st->vtime;
+
+ if (entity->tree == &st->idle) {
+ /*
+ * Must be on the idle tree, bfq_idle_extract() will
+ * check for that.
+ */
+ bfq_idle_extract(st, entity);
+ entity->start = bfq_gt(min_vstart, entity->finish) ?
+ min_vstart : entity->finish;
+ } else {
+ /*
+ * The finish time of the entity may be invalid, and
+ * it is in the past for sure, otherwise the queue
+ * would have been on the idle tree.
+ */
+ entity->start = min_vstart;
+ st->wsum += entity->weight;
+ /*
+ * entity is about to be inserted into a service tree,
+ * and then set in service: get a reference to make
+ * sure entity does not disappear until it is no
+ * longer in service or scheduled for service.
+ */
+ bfq_get_entity(entity);
+
+ entity->on_st = true;
+ }
+
+ bfq_update_fin_time_enqueue(entity, st, backshifted);
+}
+
+/**
+ * __bfq_requeue_entity - handle requeueing or repositioning of an entity.
+ * @entity: the entity being requeued or repositioned.
+ *
+ * Requeueing is needed if this entity stops being served, which
+ * happens if a leaf descendant entity has expired. On the other hand,
+ * repositioning is needed if the next_inservice_entity for the child
+ * entity has changed. See the comments inside the function for
+ * details.
+ *
+ * Basically, this function: 1) removes entity from its active tree if
+ * present there, 2) updates the timestamps of entity and 3) inserts
+ * entity back into its active tree (in the new, right position for
+ * the new values of the timestamps).
+ */
+static void __bfq_requeue_entity(struct bfq_entity *entity)
+{
+ struct bfq_sched_data *sd = entity->sched_data;
+ struct bfq_service_tree *st = bfq_entity_service_tree(entity);
+
+ if (entity == sd->in_service_entity) {
+ /*
+ * We are requeueing the current in-service entity,
+ * which may have to be done for one of the following
+ * reasons:
+ * - entity represents the in-service queue, and the
+ * in-service queue is being requeued after an
+ * expiration;
+ * - entity represents a group, and its budget has
+ * changed because one of its child entities has
+ * just been either activated or requeued for some
+ * reason; the timestamps of the entity need then to
+ * be updated, and the entity needs to be enqueued
+ * or repositioned accordingly.
+ *
+ * In particular, before requeueing, the start time of
+ * the entity must be moved forward to account for the
+ * service that the entity has received while in
+ * service. This is done by the next instructions. The
+ * finish time will then be updated according to this
+ * new value of the start time, and to the budget of
+ * the entity.
+ */
+ bfq_calc_finish(entity, entity->service);
+ entity->start = entity->finish;
+ /*
+ * In addition, if the entity had more than one child
+ * when set in service, then was not extracted from
+ * the active tree. This implies that the position of
+ * the entity in the active tree may need to be
+ * changed now, because we have just updated the start
+ * time of the entity, and we will update its finish
+ * time in a moment (the requeueing is then, more
+ * precisely, a repositioning in this case). To
+ * implement this repositioning, we: 1) dequeue the
+ * entity here, 2) update the finish time and
+ * requeue the entity according to the new
+ * timestamps below.
+ */
+ if (entity->tree)
+ bfq_active_extract(st, entity);
+ } else { /* The entity is already active, and not in service */
+ /*
+ * In this case, this function gets called only if the
+ * next_in_service entity below this entity has
+ * changed, and this change has caused the budget of
+ * this entity to change, which, finally implies that
+ * the finish time of this entity must be
+ * updated. Such an update may cause the scheduling,
+ * i.e., the position in the active tree, of this
+ * entity to change. We handle this change by: 1)
+ * dequeueing the entity here, 2) updating the finish
+ * time and requeueing the entity according to the new
+ * timestamps below. This is the same approach as the
+ * non-extracted-entity sub-case above.
+ */
+ bfq_active_extract(st, entity);
+ }
+
+ bfq_update_fin_time_enqueue(entity, st, false);
+}
+
+static void __bfq_activate_requeue_entity(struct bfq_entity *entity,
+ struct bfq_sched_data *sd,
+ bool non_blocking_wait_rq)
+{
+ struct bfq_service_tree *st = bfq_entity_service_tree(entity);
+
+ if (sd->in_service_entity == entity || entity->tree == &st->active)
+ /*
+ * in service or already queued on the active tree,
+ * requeue or reposition
+ */
+ __bfq_requeue_entity(entity);
+ else
+ /*
+ * Not in service and not queued on its active tree:
+ * the activity is idle and this is a true activation.
+ */
+ __bfq_activate_entity(entity, non_blocking_wait_rq);
+}
+
+
+/**
+ * bfq_activate_entity - activate or requeue an entity representing a bfq_queue,
+ * and activate, requeue or reposition all ancestors
+ * for which such an update becomes necessary.
+ * @entity: the entity to activate.
+ * @non_blocking_wait_rq: true if this entity was waiting for a request
+ * @requeue: true if this is a requeue, which implies that bfqq is
+ * being expired; thus ALL its ancestors stop being served and must
+ * therefore be requeued
+ */
+static void bfq_activate_requeue_entity(struct bfq_entity *entity,
+ bool non_blocking_wait_rq,
+ bool requeue)
+{
+ struct bfq_sched_data *sd;
+
+ for_each_entity(entity) {
+ sd = entity->sched_data;
+ __bfq_activate_requeue_entity(entity, sd, non_blocking_wait_rq);
+
+ if (!bfq_update_next_in_service(sd, entity) && !requeue)
+ break;
+ }
+}
+
+/**
+ * __bfq_deactivate_entity - deactivate an entity from its service tree.
+ * @entity: the entity to deactivate.
+ * @ins_into_idle_tree: if false, the entity will not be put into the
+ * idle tree.
+ *
+ * Deactivates an entity, independently from its previous state. Must
+ * be invoked only if entity is on a service tree. Extracts the entity
+ * from that tree, and if necessary and allowed, puts it on the idle
+ * tree.
+ */
+bool __bfq_deactivate_entity(struct bfq_entity *entity, bool ins_into_idle_tree)
+{
+ struct bfq_sched_data *sd = entity->sched_data;
+ struct bfq_service_tree *st = bfq_entity_service_tree(entity);
+ int is_in_service = entity == sd->in_service_entity;
+
+ if (!entity->on_st) /* entity never activated, or already inactive */
+ return false;
+
+ if (is_in_service)
+ bfq_calc_finish(entity, entity->service);
+
+ if (entity->tree == &st->active)
+ bfq_active_extract(st, entity);
+ else if (!is_in_service && entity->tree == &st->idle)
+ bfq_idle_extract(st, entity);
+
+ if (!ins_into_idle_tree || !bfq_gt(entity->finish, st->vtime))
+ bfq_forget_entity(st, entity, is_in_service);
+ else
+ bfq_idle_insert(st, entity);
+
+ return true;
+}
+
+/**
+ * bfq_deactivate_entity - deactivate an entity representing a bfq_queue.
+ * @entity: the entity to deactivate.
+ * @ins_into_idle_tree: true if the entity can be put on the idle tree
+ */
+static void bfq_deactivate_entity(struct bfq_entity *entity,
+ bool ins_into_idle_tree,
+ bool expiration)
+{
+ struct bfq_sched_data *sd;
+ struct bfq_entity *parent = NULL;
+
+ for_each_entity_safe(entity, parent) {
+ sd = entity->sched_data;
+
+ if (!__bfq_deactivate_entity(entity, ins_into_idle_tree)) {
+ /*
+ * entity is not in any tree any more, so
+ * this deactivation is a no-op, and there is
+ * nothing to change for upper-level entities
+ * (in case of expiration, this can never
+ * happen).
+ */
+ return;
+ }
+
+ if (sd->next_in_service == entity)
+ /*
+ * entity was the next_in_service entity,
+ * then, since entity has just been
+ * deactivated, a new one must be found.
+ */
+ bfq_update_next_in_service(sd, NULL);
+
+ if (sd->next_in_service)
+ /*
+ * The parent entity is still backlogged,
+ * because next_in_service is not NULL. So, no
+ * further upwards deactivation must be
+ * performed. Yet, next_in_service has
+ * changed. Then the schedule does need to be
+ * updated upwards.
+ */
+ break;
+
+ /*
+ * If we get here, then the parent is no more
+ * backlogged and we need to propagate the
+ * deactivation upwards. Thus let the loop go on.
+ */
+
+ /*
+ * Also let parent be queued into the idle tree on
+ * deactivation, to preserve service guarantees, and
+ * assuming that who invoked this function does not
+ * need parent entities too to be removed completely.
+ */
+ ins_into_idle_tree = true;
+ }
+
+ /*
+ * If the deactivation loop is fully executed, then there are
+ * no more entities to touch and next loop is not executed at
+ * all. Otherwise, requeue remaining entities if they are
+ * about to stop receiving service, or reposition them if this
+ * is not the case.
+ */
+ entity = parent;
+ for_each_entity(entity) {
+ /*
+ * Invoke __bfq_requeue_entity on entity, even if
+ * already active, to requeue/reposition it in the
+ * active tree (because sd->next_in_service has
+ * changed)
+ */
+ __bfq_requeue_entity(entity);
+
+ sd = entity->sched_data;
+ if (!bfq_update_next_in_service(sd, entity) &&
+ !expiration)
+ /*
+ * next_in_service unchanged or not causing
+ * any change in entity->parent->sd, and no
+ * requeueing needed for expiration: stop
+ * here.
+ */
+ break;
+ }
+}
+
+/**
+ * bfq_calc_vtime_jump - compute the value to which the vtime should jump,
+ * if needed, to have at least one entity eligible.
+ * @st: the service tree to act upon.
+ *
+ * Assumes that st is not empty.
+ */
+static u64 bfq_calc_vtime_jump(struct bfq_service_tree *st)
+{
+ struct bfq_entity *root_entity = bfq_root_active_entity(&st->active);
+
+ if (bfq_gt(root_entity->min_start, st->vtime))
+ return root_entity->min_start;
+
+ return st->vtime;
+}
+
+static void bfq_update_vtime(struct bfq_service_tree *st, u64 new_value)
+{
+ if (new_value > st->vtime) {
+ st->vtime = new_value;
+ bfq_forget_idle(st);
+ }
+}
+
+/**
+ * bfq_first_active_entity - find the eligible entity with
+ * the smallest finish time
+ * @st: the service tree to select from.
+ * @vtime: the system virtual to use as a reference for eligibility
+ *
+ * This function searches the first schedulable entity, starting from the
+ * root of the tree and going on the left every time on this side there is
+ * a subtree with at least one eligible (start >= vtime) entity. The path on
+ * the right is followed only if a) the left subtree contains no eligible
+ * entities and b) no eligible entity has been found yet.
+ */
+static struct bfq_entity *bfq_first_active_entity(struct bfq_service_tree *st,
+ u64 vtime)
+{
+ struct bfq_entity *entry, *first = NULL;
+ struct rb_node *node = st->active.rb_node;
+
+ while (node) {
+ entry = rb_entry(node, struct bfq_entity, rb_node);
+left:
+ if (!bfq_gt(entry->start, vtime))
+ first = entry;
+
+ if (node->rb_left) {
+ entry = rb_entry(node->rb_left,
+ struct bfq_entity, rb_node);
+ if (!bfq_gt(entry->min_start, vtime)) {
+ node = node->rb_left;
+ goto left;
+ }
+ }
+ if (first)
+ break;
+ node = node->rb_right;
+ }
+
+ return first;
+}
+
+/**
+ * __bfq_lookup_next_entity - return the first eligible entity in @st.
+ * @st: the service tree.
+ *
+ * If there is no in-service entity for the sched_data st belongs to,
+ * then return the entity that will be set in service if:
+ * 1) the parent entity this st belongs to is set in service;
+ * 2) no entity belonging to such parent entity undergoes a state change
+ * that would influence the timestamps of the entity (e.g., becomes idle,
+ * becomes backlogged, changes its budget, ...).
+ *
+ * In this first case, update the virtual time in @st too (see the
+ * comments on this update inside the function).
+ *
+ * In constrast, if there is an in-service entity, then return the
+ * entity that would be set in service if not only the above
+ * conditions, but also the next one held true: the currently
+ * in-service entity, on expiration,
+ * 1) gets a finish time equal to the current one, or
+ * 2) is not eligible any more, or
+ * 3) is idle.
+ */
+static struct bfq_entity *
+__bfq_lookup_next_entity(struct bfq_service_tree *st, bool in_service)
+{
+ struct bfq_entity *entity;
+ u64 new_vtime;
+
+ if (RB_EMPTY_ROOT(&st->active))
+ return NULL;
+
+ /*
+ * Get the value of the system virtual time for which at
+ * least one entity is eligible.
+ */
+ new_vtime = bfq_calc_vtime_jump(st);
+
+ /*
+ * If there is no in-service entity for the sched_data this
+ * active tree belongs to, then push the system virtual time
+ * up to the value that guarantees that at least one entity is
+ * eligible. If, instead, there is an in-service entity, then
+ * do not make any such update, because there is already an
+ * eligible entity, namely the in-service one (even if the
+ * entity is not on st, because it was extracted when set in
+ * service).
+ */
+ if (!in_service)
+ bfq_update_vtime(st, new_vtime);
+
+ entity = bfq_first_active_entity(st, new_vtime);
+
+ return entity;
+}
+
+/**
+ * bfq_lookup_next_entity - return the first eligible entity in @sd.
+ * @sd: the sched_data.
+ *
+ * This function is invoked when there has been a change in the trees
+ * for sd, and we need know what is the new next entity after this
+ * change.
+ */
+static struct bfq_entity *bfq_lookup_next_entity(struct bfq_sched_data *sd)
+{
+ struct bfq_service_tree *st = sd->service_tree;
+ struct bfq_service_tree *idle_class_st = st + (BFQ_IOPRIO_CLASSES - 1);
+ struct bfq_entity *entity = NULL;
+ int class_idx = 0;
+
+ /*
+ * Choose from idle class, if needed to guarantee a minimum
+ * bandwidth to this class (and if there is some active entity
+ * in idle class). This should also mitigate
+ * priority-inversion problems in case a low priority task is
+ * holding file system resources.
+ */
+ if (time_is_before_jiffies(sd->bfq_class_idle_last_service +
+ BFQ_CL_IDLE_TIMEOUT)) {
+ if (!RB_EMPTY_ROOT(&idle_class_st->active))
+ class_idx = BFQ_IOPRIO_CLASSES - 1;
+ /* About to be served if backlogged, or not yet backlogged */
+ sd->bfq_class_idle_last_service = jiffies;
+ }
+
+ /*
+ * Find the next entity to serve for the highest-priority
+ * class, unless the idle class needs to be served.
+ */
+ for (; class_idx < BFQ_IOPRIO_CLASSES; class_idx++) {
+ entity = __bfq_lookup_next_entity(st + class_idx,
+ sd->in_service_entity);
+
+ if (entity)
+ break;
+ }
+
+ if (!entity)
+ return NULL;
+
+ return entity;
+}
+
+bool next_queue_may_preempt(struct bfq_data *bfqd)
+{
+ struct bfq_sched_data *sd = &bfqd->root_group->sched_data;
+
+ return sd->next_in_service != sd->in_service_entity;
+}
+
+/*
+ * Get next queue for service.
+ */
+struct bfq_queue *bfq_get_next_queue(struct bfq_data *bfqd)
+{
+ struct bfq_entity *entity = NULL;
+ struct bfq_sched_data *sd;
+ struct bfq_queue *bfqq;
+
+ if (bfqd->busy_queues == 0)
+ return NULL;
+
+ /*
+ * Traverse the path from the root to the leaf entity to
+ * serve. Set in service all the entities visited along the
+ * way.
+ */
+ sd = &bfqd->root_group->sched_data;
+ for (; sd ; sd = entity->my_sched_data) {
+ /*
+ * WARNING. We are about to set the in-service entity
+ * to sd->next_in_service, i.e., to the (cached) value
+ * returned by bfq_lookup_next_entity(sd) the last
+ * time it was invoked, i.e., the last time when the
+ * service order in sd changed as a consequence of the
+ * activation or deactivation of an entity. In this
+ * respect, if we execute bfq_lookup_next_entity(sd)
+ * in this very moment, it may, although with low
+ * probability, yield a different entity than that
+ * pointed to by sd->next_in_service. This rare event
+ * happens in case there was no CLASS_IDLE entity to
+ * serve for sd when bfq_lookup_next_entity(sd) was
+ * invoked for the last time, while there is now one
+ * such entity.
+ *
+ * If the above event happens, then the scheduling of
+ * such entity in CLASS_IDLE is postponed until the
+ * service of the sd->next_in_service entity
+ * finishes. In fact, when the latter is expired,
+ * bfq_lookup_next_entity(sd) gets called again,
+ * exactly to update sd->next_in_service.
+ */
+
+ /* Make next_in_service entity become in_service_entity */
+ entity = sd->next_in_service;
+ sd->in_service_entity = entity;
+
+ /*
+ * Reset the accumulator of the amount of service that
+ * the entity is about to receive.
+ */
+ entity->service = 0;
+
+ /*
+ * If entity is no longer a candidate for next
+ * service, then we extract it from its active tree,
+ * for the following reason. To further boost the
+ * throughput in some special case, BFQ needs to know
+ * which is the next candidate entity to serve, while
+ * there is already an entity in service. In this
+ * respect, to make it easy to compute/update the next
+ * candidate entity to serve after the current
+ * candidate has been set in service, there is a case
+ * where it is necessary to extract the current
+ * candidate from its service tree. Such a case is
+ * when the entity just set in service cannot be also
+ * a candidate for next service. Details about when
+ * this conditions holds are reported in the comments
+ * on the function bfq_no_longer_next_in_service()
+ * invoked below.
+ */
+ if (bfq_no_longer_next_in_service(entity))
+ bfq_active_extract(bfq_entity_service_tree(entity),
+ entity);
+
+ /*
+ * For the same reason why we may have just extracted
+ * entity from its active tree, we may need to update
+ * next_in_service for the sched_data of entity too,
+ * regardless of whether entity has been extracted.
+ * In fact, even if entity has not been extracted, a
+ * descendant entity may get extracted. Such an event
+ * would cause a change in next_in_service for the
+ * level of the descendant entity, and thus possibly
+ * back to upper levels.
+ *
+ * We cannot perform the resulting needed update
+ * before the end of this loop, because, to know which
+ * is the correct next-to-serve candidate entity for
+ * each level, we need first to find the leaf entity
+ * to set in service. In fact, only after we know
+ * which is the next-to-serve leaf entity, we can
+ * discover whether the parent entity of the leaf
+ * entity becomes the next-to-serve, and so on.
+ */
+
+ }
+
+ bfqq = bfq_entity_to_bfqq(entity);
+
+ /*
+ * We can finally update all next-to-serve entities along the
+ * path from the leaf entity just set in service to the root.
+ */
+ for_each_entity(entity) {
+ struct bfq_sched_data *sd = entity->sched_data;
+
+ if (!bfq_update_next_in_service(sd, NULL))
+ break;
+ }
+
+ return bfqq;
+}
+
+void __bfq_bfqd_reset_in_service(struct bfq_data *bfqd)
+{
+ struct bfq_queue *in_serv_bfqq = bfqd->in_service_queue;
+ struct bfq_entity *in_serv_entity = &in_serv_bfqq->entity;
+ struct bfq_entity *entity = in_serv_entity;
+
+ bfq_clear_bfqq_wait_request(in_serv_bfqq);
+ hrtimer_try_to_cancel(&bfqd->idle_slice_timer);
+ bfqd->in_service_queue = NULL;
+
+ /*
+ * When this function is called, all in-service entities have
+ * been properly deactivated or requeued, so we can safely
+ * execute the final step: reset in_service_entity along the
+ * path from entity to the root.
+ */
+ for_each_entity(entity)
+ entity->sched_data->in_service_entity = NULL;
+
+ /*
+ * in_serv_entity is no longer in service, so, if it is in no
+ * service tree either, then release the service reference to
+ * the queue it represents (taken with bfq_get_entity).
+ */
+ if (!in_serv_entity->on_st)
+ bfq_put_queue(in_serv_bfqq);
+}
+
+void bfq_deactivate_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ bool ins_into_idle_tree, bool expiration)
+{
+ struct bfq_entity *entity = &bfqq->entity;
+
+ bfq_deactivate_entity(entity, ins_into_idle_tree, expiration);
+}
+
+void bfq_activate_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq)
+{
+ struct bfq_entity *entity = &bfqq->entity;
+
+ bfq_activate_requeue_entity(entity, bfq_bfqq_non_blocking_wait_rq(bfqq),
+ false);
+ bfq_clear_bfqq_non_blocking_wait_rq(bfqq);
+}
+
+void bfq_requeue_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq)
+{
+ struct bfq_entity *entity = &bfqq->entity;
+
+ bfq_activate_requeue_entity(entity, false,
+ bfqq == bfqd->in_service_queue);
+}
+
+/*
+ * Called when the bfqq no longer has requests pending, remove it from
+ * the service tree. As a special case, it can be invoked during an
+ * expiration.
+ */
+void bfq_del_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq,
+ bool expiration)
+{
+ bfq_log_bfqq(bfqd, bfqq, "del from busy");
+
+ bfq_clear_bfqq_busy(bfqq);
+
+ bfqd->busy_queues--;
+
+ if (!bfqq->dispatched)
+ bfq_weights_tree_remove(bfqd, &bfqq->entity,
+ &bfqd->queue_weights_tree);
+
+ if (bfqq->wr_coeff > 1)
+ bfqd->wr_busy_queues--;
+
+ bfqg_stats_update_dequeue(bfqq_group(bfqq));
+
+ bfq_deactivate_bfqq(bfqd, bfqq, true, expiration);
+}
+
+/*
+ * Called when an inactive queue receives a new request.
+ */
+void bfq_add_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq)
+{
+ bfq_log_bfqq(bfqd, bfqq, "add to busy");
+
+ bfq_activate_bfqq(bfqd, bfqq);
+
+ bfq_mark_bfqq_busy(bfqq);
+ bfqd->busy_queues++;
+
+ if (!bfqq->dispatched)
+ if (bfqq->wr_coeff == 1)
+ bfq_weights_tree_add(bfqd, &bfqq->entity,
+ &bfqd->queue_weights_tree);
+
+ if (bfqq->wr_coeff > 1)
+ bfqd->wr_busy_queues++;
+}
diff --git a/block/bio.c b/block/bio.c
index e75878f..f4d20718 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -30,6 +30,7 @@
#include <linux/cgroup.h>
#include <trace/events/block.h>
+#include "blk.h"
/*
* Test patch to inline a certain number of bi_io_vec's inside the bio
@@ -427,7 +428,8 @@
* RETURNS:
* Pointer to new bio on success, NULL on failure.
*/
-struct bio *bio_alloc_bioset(gfp_t gfp_mask, int nr_iovecs, struct bio_set *bs)
+struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned int nr_iovecs,
+ struct bio_set *bs)
{
gfp_t saved_gfp = gfp_mask;
unsigned front_pad;
@@ -1824,6 +1826,11 @@
* bio_endio() will end I/O on the whole bio. bio_endio() is the preferred
* way to end I/O on a bio. No one should call bi_end_io() directly on a
* bio unless they own it and thus know that it has an end_io function.
+ *
+ * bio_endio() can be called several times on a bio that has been chained
+ * using bio_chain(). The ->bi_end_io() function will only be called the
+ * last time. At this point the BLK_TA_COMPLETE tracing event will be
+ * generated if BIO_TRACE_COMPLETION is set.
**/
void bio_endio(struct bio *bio)
{
@@ -1844,6 +1851,13 @@
goto again;
}
+ if (bio->bi_bdev && bio_flagged(bio, BIO_TRACE_COMPLETION)) {
+ trace_block_bio_complete(bdev_get_queue(bio->bi_bdev),
+ bio, bio->bi_error);
+ bio_clear_flag(bio, BIO_TRACE_COMPLETION);
+ }
+
+ blk_throtl_bio_endio(bio);
if (bio->bi_end_io)
bio->bi_end_io(bio);
}
@@ -1882,6 +1896,9 @@
bio_advance(bio, split->bi_iter.bi_size);
+ if (bio_flagged(bio, BIO_TRACE_COMPLETION))
+ bio_set_flag(bio, BIO_TRACE_COMPLETION);
+
return split;
}
EXPORT_SYMBOL(bio_split);
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index bbe7ee0..7c29471 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -772,6 +772,27 @@
}
EXPORT_SYMBOL_GPL(blkg_rwstat_recursive_sum);
+/* Performs queue bypass and policy enabled checks then looks up blkg. */
+static struct blkcg_gq *blkg_lookup_check(struct blkcg *blkcg,
+ const struct blkcg_policy *pol,
+ struct request_queue *q)
+{
+ WARN_ON_ONCE(!rcu_read_lock_held());
+ lockdep_assert_held(q->queue_lock);
+
+ if (!blkcg_policy_enabled(q, pol))
+ return ERR_PTR(-EOPNOTSUPP);
+
+ /*
+ * This could be the first entry point of blkcg implementation and
+ * we shouldn't allow anything to go through for a bypassing queue.
+ */
+ if (unlikely(blk_queue_bypass(q)))
+ return ERR_PTR(blk_queue_dying(q) ? -ENODEV : -EBUSY);
+
+ return __blkg_lookup(blkcg, q, true /* update_hint */);
+}
+
/**
* blkg_conf_prep - parse and prepare for per-blkg config update
* @blkcg: target block cgroup
@@ -789,6 +810,7 @@
__acquires(rcu) __acquires(disk->queue->queue_lock)
{
struct gendisk *disk;
+ struct request_queue *q;
struct blkcg_gq *blkg;
struct module *owner;
unsigned int major, minor;
@@ -807,44 +829,95 @@
if (!disk)
return -ENODEV;
if (part) {
- owner = disk->fops->owner;
- put_disk(disk);
- module_put(owner);
- return -ENODEV;
+ ret = -ENODEV;
+ goto fail;
}
+ q = disk->queue;
+
rcu_read_lock();
- spin_lock_irq(disk->queue->queue_lock);
+ spin_lock_irq(q->queue_lock);
- if (blkcg_policy_enabled(disk->queue, pol))
- blkg = blkg_lookup_create(blkcg, disk->queue);
- else
- blkg = ERR_PTR(-EOPNOTSUPP);
-
+ blkg = blkg_lookup_check(blkcg, pol, q);
if (IS_ERR(blkg)) {
ret = PTR_ERR(blkg);
- rcu_read_unlock();
- spin_unlock_irq(disk->queue->queue_lock);
- owner = disk->fops->owner;
- put_disk(disk);
- module_put(owner);
- /*
- * If queue was bypassing, we should retry. Do so after a
- * short msleep(). It isn't strictly necessary but queue
- * can be bypassing for some time and it's always nice to
- * avoid busy looping.
- */
- if (ret == -EBUSY) {
- msleep(10);
- ret = restart_syscall();
- }
- return ret;
+ goto fail_unlock;
}
+ if (blkg)
+ goto success;
+
+ /*
+ * Create blkgs walking down from blkcg_root to @blkcg, so that all
+ * non-root blkgs have access to their parents.
+ */
+ while (true) {
+ struct blkcg *pos = blkcg;
+ struct blkcg *parent;
+ struct blkcg_gq *new_blkg;
+
+ parent = blkcg_parent(blkcg);
+ while (parent && !__blkg_lookup(parent, q, false)) {
+ pos = parent;
+ parent = blkcg_parent(parent);
+ }
+
+ /* Drop locks to do new blkg allocation with GFP_KERNEL. */
+ spin_unlock_irq(q->queue_lock);
+ rcu_read_unlock();
+
+ new_blkg = blkg_alloc(pos, q, GFP_KERNEL);
+ if (unlikely(!new_blkg)) {
+ ret = -ENOMEM;
+ goto fail;
+ }
+
+ rcu_read_lock();
+ spin_lock_irq(q->queue_lock);
+
+ blkg = blkg_lookup_check(pos, pol, q);
+ if (IS_ERR(blkg)) {
+ ret = PTR_ERR(blkg);
+ goto fail_unlock;
+ }
+
+ if (blkg) {
+ blkg_free(new_blkg);
+ } else {
+ blkg = blkg_create(pos, q, new_blkg);
+ if (unlikely(IS_ERR(blkg))) {
+ ret = PTR_ERR(blkg);
+ goto fail_unlock;
+ }
+ }
+
+ if (pos == blkcg)
+ goto success;
+ }
+success:
ctx->disk = disk;
ctx->blkg = blkg;
ctx->body = body;
return 0;
+
+fail_unlock:
+ spin_unlock_irq(q->queue_lock);
+ rcu_read_unlock();
+fail:
+ owner = disk->fops->owner;
+ put_disk(disk);
+ module_put(owner);
+ /*
+ * If queue was bypassing, we should retry. Do so after a
+ * short msleep(). It isn't strictly necessary but queue
+ * can be bypassing for some time and it's always nice to
+ * avoid busy looping.
+ */
+ if (ret == -EBUSY) {
+ msleep(10);
+ ret = restart_syscall();
+ }
+ return ret;
}
EXPORT_SYMBOL_GPL(blkg_conf_prep);
diff --git a/block/blk-core.c b/block/blk-core.c
index d772c22..24886b6 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -268,10 +268,8 @@
struct blk_mq_hw_ctx *hctx;
int i;
- queue_for_each_hw_ctx(q, hctx, i) {
- cancel_work_sync(&hctx->run_work);
- cancel_delayed_work_sync(&hctx->delay_work);
- }
+ queue_for_each_hw_ctx(q, hctx, i)
+ cancel_delayed_work_sync(&hctx->run_work);
} else {
cancel_delayed_work_sync(&q->delay_work);
}
@@ -500,6 +498,13 @@
queue_flag_set(QUEUE_FLAG_DYING, q);
spin_unlock_irq(q->queue_lock);
+ /*
+ * When queue DYING flag is set, we need to block new req
+ * entering queue, so we call blk_freeze_queue_start() to
+ * prevent I/O from crossing blk_queue_enter().
+ */
+ blk_freeze_queue_start(q);
+
if (q->mq_ops)
blk_mq_wake_waiters(q);
else {
@@ -556,9 +561,13 @@
* prevent that q->request_fn() gets invoked after draining finished.
*/
blk_freeze_queue(q);
- spin_lock_irq(lock);
- if (!q->mq_ops)
+ if (!q->mq_ops) {
+ spin_lock_irq(lock);
__blk_drain_queue(q, true);
+ } else {
+ blk_mq_debugfs_unregister_mq(q);
+ spin_lock_irq(lock);
+ }
queue_flag_set(QUEUE_FLAG_DEAD, q);
spin_unlock_irq(lock);
@@ -669,6 +678,15 @@
if (nowait)
return -EBUSY;
+ /*
+ * read pair of barrier in blk_freeze_queue_start(),
+ * we need to order reading __PERCPU_REF_DEAD flag of
+ * .q_usage_counter and reading .mq_freeze_depth or
+ * queue dying flag, otherwise the following wait may
+ * never return if the two reads are reordered.
+ */
+ smp_rmb();
+
ret = wait_event_interruptible(q->mq_freeze_wq,
!atomic_read(&q->mq_freeze_depth) ||
blk_queue_dying(q));
@@ -720,6 +738,10 @@
if (!q->backing_dev_info)
goto fail_split;
+ q->stats = blk_alloc_queue_stats();
+ if (!q->stats)
+ goto fail_stats;
+
q->backing_dev_info->ra_pages =
(VM_MAX_READAHEAD * 1024) / PAGE_SIZE;
q->backing_dev_info->capabilities = BDI_CAP_CGROUP_WRITEBACK;
@@ -776,6 +798,8 @@
fail_ref:
percpu_ref_exit(&q->q_usage_counter);
fail_bdi:
+ blk_free_queue_stats(q->stats);
+fail_stats:
bdi_put(q->backing_dev_info);
fail_split:
bioset_free(q->bio_split);
@@ -889,7 +913,6 @@
q->exit_rq_fn(q, q->fq->flush_rq);
out_free_flush_queue:
blk_free_flush_queue(q->fq);
- wbt_exit(q);
return -ENOMEM;
}
EXPORT_SYMBOL(blk_init_allocated_queue);
@@ -1128,7 +1151,6 @@
blk_rq_init(q, rq);
blk_rq_set_rl(rq, rl);
- blk_rq_set_prio(rq, ioc);
rq->cmd_flags = op;
rq->rq_flags = rq_flags;
@@ -1608,17 +1630,23 @@
return ret;
}
-void init_request_from_bio(struct request *req, struct bio *bio)
+void blk_init_request_from_bio(struct request *req, struct bio *bio)
{
+ struct io_context *ioc = rq_ioc(bio);
+
if (bio->bi_opf & REQ_RAHEAD)
req->cmd_flags |= REQ_FAILFAST_MASK;
- req->errors = 0;
req->__sector = bio->bi_iter.bi_sector;
if (ioprio_valid(bio_prio(bio)))
req->ioprio = bio_prio(bio);
+ else if (ioc)
+ req->ioprio = ioc->ioprio;
+ else
+ req->ioprio = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_NONE, 0);
blk_rq_bio_prep(req->q, req, bio);
}
+EXPORT_SYMBOL_GPL(blk_init_request_from_bio);
static blk_qc_t blk_queue_bio(struct request_queue *q, struct bio *bio)
{
@@ -1709,7 +1737,7 @@
* We don't worry about that case for efficiency. It won't happen
* often, and the elevators are able to handle it.
*/
- init_request_from_bio(req, bio);
+ blk_init_request_from_bio(req, bio);
if (test_bit(QUEUE_FLAG_SAME_COMP, &q->queue_flags))
req->cpu = raw_smp_processor_id();
@@ -1936,7 +1964,13 @@
if (!blkcg_bio_issue_check(q, bio))
return false;
- trace_block_bio_queue(q, bio);
+ if (!bio_flagged(bio, BIO_TRACE_COMPLETION)) {
+ trace_block_bio_queue(q, bio);
+ /* Now that enqueuing has been traced, we need to trace
+ * completion as well.
+ */
+ bio_set_flag(bio, BIO_TRACE_COMPLETION);
+ }
return true;
not_supported:
@@ -2478,7 +2512,7 @@
blk_dequeue_request(req);
if (test_bit(QUEUE_FLAG_STATS, &req->q->queue_flags)) {
- blk_stat_set_issue_time(&req->issue_stat);
+ blk_stat_set_issue(&req->issue_stat, blk_rq_sectors(req));
req->rq_flags |= RQF_STATS;
wbt_issue(req->q->rq_wb, &req->issue_stat);
}
@@ -2540,22 +2574,11 @@
{
int total_bytes;
- trace_block_rq_complete(req->q, req, nr_bytes);
+ trace_block_rq_complete(req, error, nr_bytes);
if (!req->bio)
return false;
- /*
- * For fs requests, rq is just carrier of independent bio's
- * and each partial completion should be handled separately.
- * Reset per-request error on each partial completion.
- *
- * TODO: tj: This is too subtle. It would be better to let
- * low level drivers do what they see fit.
- */
- if (!blk_rq_is_passthrough(req))
- req->errors = 0;
-
if (error && !blk_rq_is_passthrough(req) &&
!(req->rq_flags & RQF_QUIET)) {
char *error_type;
@@ -2601,6 +2624,8 @@
if (bio_bytes == bio->bi_iter.bi_size)
req->bio = bio->bi_next;
+ /* Completion has already been traced */
+ bio_clear_flag(bio, BIO_TRACE_COMPLETION);
req_bio_endio(req, bio, bio_bytes, error);
total_bytes += bio_bytes;
@@ -2699,7 +2724,7 @@
struct request_queue *q = req->q;
if (req->rq_flags & RQF_STATS)
- blk_stat_add(&q->rq_stats[rq_data_dir(req)], req);
+ blk_stat_add(req);
if (req->rq_flags & RQF_QUEUED)
blk_queue_end_tag(q, req);
@@ -2776,7 +2801,7 @@
* %false - we are done with this request
* %true - still buffers pending for this request
**/
-bool __blk_end_bidi_request(struct request *rq, int error,
+static bool __blk_end_bidi_request(struct request *rq, int error,
unsigned int nr_bytes, unsigned int bidi_bytes)
{
if (blk_update_bidi_request(rq, error, nr_bytes, bidi_bytes))
@@ -2829,43 +2854,6 @@
EXPORT_SYMBOL(blk_end_request_all);
/**
- * blk_end_request_cur - Helper function to finish the current request chunk.
- * @rq: the request to finish the current chunk for
- * @error: %0 for success, < %0 for error
- *
- * Description:
- * Complete the current consecutively mapped chunk from @rq.
- *
- * Return:
- * %false - we are done with this request
- * %true - still buffers pending for this request
- */
-bool blk_end_request_cur(struct request *rq, int error)
-{
- return blk_end_request(rq, error, blk_rq_cur_bytes(rq));
-}
-EXPORT_SYMBOL(blk_end_request_cur);
-
-/**
- * blk_end_request_err - Finish a request till the next failure boundary.
- * @rq: the request to finish till the next failure boundary for
- * @error: must be negative errno
- *
- * Description:
- * Complete @rq till the next failure boundary.
- *
- * Return:
- * %false - we are done with this request
- * %true - still buffers pending for this request
- */
-bool blk_end_request_err(struct request *rq, int error)
-{
- WARN_ON(error >= 0);
- return blk_end_request(rq, error, blk_rq_err_bytes(rq));
-}
-EXPORT_SYMBOL_GPL(blk_end_request_err);
-
-/**
* __blk_end_request - Helper function for drivers to complete the request.
* @rq: the request being processed
* @error: %0 for success, < %0 for error
@@ -2924,26 +2912,6 @@
}
EXPORT_SYMBOL(__blk_end_request_cur);
-/**
- * __blk_end_request_err - Finish a request till the next failure boundary.
- * @rq: the request to finish till the next failure boundary for
- * @error: must be negative errno
- *
- * Description:
- * Complete @rq till the next failure boundary. Must be called
- * with queue lock held.
- *
- * Return:
- * %false - we are done with this request
- * %true - still buffers pending for this request
- */
-bool __blk_end_request_err(struct request *rq, int error)
-{
- WARN_ON(error >= 0);
- return __blk_end_request(rq, error, blk_rq_err_bytes(rq));
-}
-EXPORT_SYMBOL_GPL(__blk_end_request_err);
-
void blk_rq_bio_prep(struct request_queue *q, struct request *rq,
struct bio *bio)
{
@@ -3106,6 +3074,13 @@
}
EXPORT_SYMBOL(kblockd_schedule_work_on);
+int kblockd_mod_delayed_work_on(int cpu, struct delayed_work *dwork,
+ unsigned long delay)
+{
+ return mod_delayed_work_on(cpu, kblockd_workqueue, dwork, delay);
+}
+EXPORT_SYMBOL(kblockd_mod_delayed_work_on);
+
int kblockd_schedule_delayed_work(struct delayed_work *dwork,
unsigned long delay)
{
diff --git a/block/blk-exec.c b/block/blk-exec.c
index 8cd0e9b..a9451e3 100644
--- a/block/blk-exec.c
+++ b/block/blk-exec.c
@@ -69,8 +69,7 @@
if (unlikely(blk_queue_dying(q))) {
rq->rq_flags |= RQF_QUIET;
- rq->errors = -ENXIO;
- __blk_end_request_all(rq, rq->errors);
+ __blk_end_request_all(rq, -ENXIO);
spin_unlock_irq(q->queue_lock);
return;
}
@@ -92,11 +91,10 @@
* Insert a fully prepared request at the back of the I/O scheduler queue
* for execution and wait for completion.
*/
-int blk_execute_rq(struct request_queue *q, struct gendisk *bd_disk,
+void blk_execute_rq(struct request_queue *q, struct gendisk *bd_disk,
struct request *rq, int at_head)
{
DECLARE_COMPLETION_ONSTACK(wait);
- int err = 0;
unsigned long hang_check;
rq->end_io_data = &wait;
@@ -108,10 +106,5 @@
while (!wait_for_completion_io_timeout(&wait, hang_check * (HZ/2)));
else
wait_for_completion_io(&wait);
-
- if (rq->errors)
- err = -EIO;
-
- return err;
}
EXPORT_SYMBOL(blk_execute_rq);
diff --git a/block/blk-flush.c b/block/blk-flush.c
index 0d5a9c1..c4e0880 100644
--- a/block/blk-flush.c
+++ b/block/blk-flush.c
@@ -447,7 +447,7 @@
if (q->mq_ops)
blk_mq_end_request(rq, 0);
else
- __blk_end_bidi_request(rq, 0, 0, 0);
+ __blk_end_request(rq, 0, 0);
return;
}
@@ -497,8 +497,7 @@
* Description:
* Issue a flush for the block device in question. Caller can supply
* room for storing the error offset in case of a flush error, if they
- * wish to. If WAIT flag is not passed then caller may check only what
- * request was pushed in some internal queue for later handling.
+ * wish to.
*/
int blkdev_issue_flush(struct block_device *bdev, gfp_t gfp_mask,
sector_t *error_sector)
diff --git a/block/blk-integrity.c b/block/blk-integrity.c
index 9f0ff5b..0f891a9 100644
--- a/block/blk-integrity.c
+++ b/block/blk-integrity.c
@@ -389,7 +389,7 @@
return 0;
}
-static struct blk_integrity_profile nop_profile = {
+static const struct blk_integrity_profile nop_profile = {
.name = "nop",
.generate_fn = blk_integrity_nop_fn,
.verify_fn = blk_integrity_nop_fn,
@@ -412,12 +412,13 @@
bi->flags = BLK_INTEGRITY_VERIFY | BLK_INTEGRITY_GENERATE |
template->flags;
- bi->interval_exp = ilog2(queue_logical_block_size(disk->queue));
+ bi->interval_exp = template->interval_exp ? :
+ ilog2(queue_logical_block_size(disk->queue));
bi->profile = template->profile ? template->profile : &nop_profile;
bi->tuple_size = template->tuple_size;
bi->tag_size = template->tag_size;
- blk_integrity_revalidate(disk);
+ disk->queue->backing_dev_info->capabilities |= BDI_CAP_STABLE_WRITES;
}
EXPORT_SYMBOL(blk_integrity_register);
@@ -430,26 +431,11 @@
*/
void blk_integrity_unregister(struct gendisk *disk)
{
- blk_integrity_revalidate(disk);
+ disk->queue->backing_dev_info->capabilities &= ~BDI_CAP_STABLE_WRITES;
memset(&disk->queue->integrity, 0, sizeof(struct blk_integrity));
}
EXPORT_SYMBOL(blk_integrity_unregister);
-void blk_integrity_revalidate(struct gendisk *disk)
-{
- struct blk_integrity *bi = &disk->queue->integrity;
-
- if (!(disk->flags & GENHD_FL_UP))
- return;
-
- if (bi->profile)
- disk->queue->backing_dev_info->capabilities |=
- BDI_CAP_STABLE_WRITES;
- else
- disk->queue->backing_dev_info->capabilities &=
- ~BDI_CAP_STABLE_WRITES;
-}
-
void blk_integrity_add(struct gendisk *disk)
{
if (kobject_init_and_add(&disk->integrity_kobj, &integrity_ktype,
diff --git a/block/blk-lib.c b/block/blk-lib.c
index ed1e78e..e8caecd 100644
--- a/block/blk-lib.c
+++ b/block/blk-lib.c
@@ -37,17 +37,12 @@
return -ENXIO;
if (flags & BLKDEV_DISCARD_SECURE) {
- if (flags & BLKDEV_DISCARD_ZERO)
- return -EOPNOTSUPP;
if (!blk_queue_secure_erase(q))
return -EOPNOTSUPP;
op = REQ_OP_SECURE_ERASE;
} else {
if (!blk_queue_discard(q))
return -EOPNOTSUPP;
- if ((flags & BLKDEV_DISCARD_ZERO) &&
- !q->limits.discard_zeroes_data)
- return -EOPNOTSUPP;
op = REQ_OP_DISCARD;
}
@@ -109,7 +104,7 @@
* @sector: start sector
* @nr_sects: number of sectors to discard
* @gfp_mask: memory allocation flags (for bio_alloc)
- * @flags: BLKDEV_IFL_* flags to control behaviour
+ * @flags: BLKDEV_DISCARD_* flags to control behaviour
*
* Description:
* Issue a discard request for the sectors in question.
@@ -126,7 +121,7 @@
&bio);
if (!ret && bio) {
ret = submit_bio_wait(bio);
- if (ret == -EOPNOTSUPP && !(flags & BLKDEV_DISCARD_ZERO))
+ if (ret == -EOPNOTSUPP)
ret = 0;
bio_put(bio);
}
@@ -226,20 +221,9 @@
}
EXPORT_SYMBOL(blkdev_issue_write_same);
-/**
- * __blkdev_issue_write_zeroes - generate number of bios with WRITE ZEROES
- * @bdev: blockdev to issue
- * @sector: start sector
- * @nr_sects: number of sectors to write
- * @gfp_mask: memory allocation flags (for bio_alloc)
- * @biop: pointer to anchor bio
- *
- * Description:
- * Generate and issue number of bios(REQ_OP_WRITE_ZEROES) with zerofiled pages.
- */
static int __blkdev_issue_write_zeroes(struct block_device *bdev,
sector_t sector, sector_t nr_sects, gfp_t gfp_mask,
- struct bio **biop)
+ struct bio **biop, unsigned flags)
{
struct bio *bio = *biop;
unsigned int max_write_zeroes_sectors;
@@ -258,7 +242,9 @@
bio = next_bio(bio, 0, gfp_mask);
bio->bi_iter.bi_sector = sector;
bio->bi_bdev = bdev;
- bio_set_op_attrs(bio, REQ_OP_WRITE_ZEROES, 0);
+ bio->bi_opf = REQ_OP_WRITE_ZEROES;
+ if (flags & BLKDEV_ZERO_NOUNMAP)
+ bio->bi_opf |= REQ_NOUNMAP;
if (nr_sects > max_write_zeroes_sectors) {
bio->bi_iter.bi_size = max_write_zeroes_sectors << 9;
@@ -282,14 +268,27 @@
* @nr_sects: number of sectors to write
* @gfp_mask: memory allocation flags (for bio_alloc)
* @biop: pointer to anchor bio
- * @discard: discard flag
+ * @flags: controls detailed behavior
*
* Description:
- * Generate and issue number of bios with zerofiled pages.
+ * Zero-fill a block range, either using hardware offload or by explicitly
+ * writing zeroes to the device.
+ *
+ * Note that this function may fail with -EOPNOTSUPP if the driver signals
+ * zeroing offload support, but the device fails to process the command (for
+ * some devices there is no non-destructive way to verify whether this
+ * operation is actually supported). In this case the caller should call
+ * retry the call to blkdev_issue_zeroout() and the fallback path will be used.
+ *
+ * If a device is using logical block provisioning, the underlying space will
+ * not be released if %flags contains BLKDEV_ZERO_NOUNMAP.
+ *
+ * If %flags contains BLKDEV_ZERO_NOFALLBACK, the function will return
+ * -EOPNOTSUPP if no explicit hardware offload for zeroing is provided.
*/
int __blkdev_issue_zeroout(struct block_device *bdev, sector_t sector,
sector_t nr_sects, gfp_t gfp_mask, struct bio **biop,
- bool discard)
+ unsigned flags)
{
int ret;
int bi_size = 0;
@@ -302,8 +301,8 @@
return -EINVAL;
ret = __blkdev_issue_write_zeroes(bdev, sector, nr_sects, gfp_mask,
- biop);
- if (ret == 0 || (ret && ret != -EOPNOTSUPP))
+ biop, flags);
+ if (ret != -EOPNOTSUPP || (flags & BLKDEV_ZERO_NOFALLBACK))
goto out;
ret = 0;
@@ -337,40 +336,23 @@
* @sector: start sector
* @nr_sects: number of sectors to write
* @gfp_mask: memory allocation flags (for bio_alloc)
- * @discard: whether to discard the block range
+ * @flags: controls detailed behavior
*
* Description:
- * Zero-fill a block range. If the discard flag is set and the block
- * device guarantees that subsequent READ operations to the block range
- * in question will return zeroes, the blocks will be discarded. Should
- * the discard request fail, if the discard flag is not set, or if
- * discard_zeroes_data is not supported, this function will resort to
- * zeroing the blocks manually, thus provisioning (allocating,
- * anchoring) them. If the block device supports WRITE ZEROES or WRITE SAME
- * command(s), blkdev_issue_zeroout() will use it to optimize the process of
- * clearing the block range. Otherwise the zeroing will be performed
- * using regular WRITE calls.
+ * Zero-fill a block range, either using hardware offload or by explicitly
+ * writing zeroes to the device. See __blkdev_issue_zeroout() for the
+ * valid values for %flags.
*/
int blkdev_issue_zeroout(struct block_device *bdev, sector_t sector,
- sector_t nr_sects, gfp_t gfp_mask, bool discard)
+ sector_t nr_sects, gfp_t gfp_mask, unsigned flags)
{
int ret;
struct bio *bio = NULL;
struct blk_plug plug;
- if (discard) {
- if (!blkdev_issue_discard(bdev, sector, nr_sects, gfp_mask,
- BLKDEV_DISCARD_ZERO))
- return 0;
- }
-
- if (!blkdev_issue_write_same(bdev, sector, nr_sects, gfp_mask,
- ZERO_PAGE(0)))
- return 0;
-
blk_start_plug(&plug);
ret = __blkdev_issue_zeroout(bdev, sector, nr_sects, gfp_mask,
- &bio, discard);
+ &bio, flags);
if (ret == 0 && bio) {
ret = submit_bio_wait(bio);
bio_put(bio);
diff --git a/block/blk-merge.c b/block/blk-merge.c
index 2afa262..3990ae4 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -54,6 +54,20 @@
return bio_split(bio, split_sectors, GFP_NOIO, bs);
}
+static struct bio *blk_bio_write_zeroes_split(struct request_queue *q,
+ struct bio *bio, struct bio_set *bs, unsigned *nsegs)
+{
+ *nsegs = 1;
+
+ if (!q->limits.max_write_zeroes_sectors)
+ return NULL;
+
+ if (bio_sectors(bio) <= q->limits.max_write_zeroes_sectors)
+ return NULL;
+
+ return bio_split(bio, q->limits.max_write_zeroes_sectors, GFP_NOIO, bs);
+}
+
static struct bio *blk_bio_write_same_split(struct request_queue *q,
struct bio *bio,
struct bio_set *bs,
@@ -200,8 +214,7 @@
split = blk_bio_discard_split(q, *bio, bs, &nsegs);
break;
case REQ_OP_WRITE_ZEROES:
- split = NULL;
- nsegs = (*bio)->bi_phys_segments;
+ split = blk_bio_write_zeroes_split(q, *bio, bs, &nsegs);
break;
case REQ_OP_WRITE_SAME:
split = blk_bio_write_same_split(q, *bio, bs, &nsegs);
diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c
index f6d9179..bcd2a7d 100644
--- a/block/blk-mq-debugfs.c
+++ b/block/blk-mq-debugfs.c
@@ -43,11 +43,157 @@
return ret;
}
+static int blk_flags_show(struct seq_file *m, const unsigned long flags,
+ const char *const *flag_name, int flag_name_count)
+{
+ bool sep = false;
+ int i;
+
+ for (i = 0; i < sizeof(flags) * BITS_PER_BYTE; i++) {
+ if (!(flags & BIT(i)))
+ continue;
+ if (sep)
+ seq_puts(m, " ");
+ sep = true;
+ if (i < flag_name_count && flag_name[i])
+ seq_puts(m, flag_name[i]);
+ else
+ seq_printf(m, "%d", i);
+ }
+ return 0;
+}
+
+static const char *const blk_queue_flag_name[] = {
+ [QUEUE_FLAG_QUEUED] = "QUEUED",
+ [QUEUE_FLAG_STOPPED] = "STOPPED",
+ [QUEUE_FLAG_SYNCFULL] = "SYNCFULL",
+ [QUEUE_FLAG_ASYNCFULL] = "ASYNCFULL",
+ [QUEUE_FLAG_DYING] = "DYING",
+ [QUEUE_FLAG_BYPASS] = "BYPASS",
+ [QUEUE_FLAG_BIDI] = "BIDI",
+ [QUEUE_FLAG_NOMERGES] = "NOMERGES",
+ [QUEUE_FLAG_SAME_COMP] = "SAME_COMP",
+ [QUEUE_FLAG_FAIL_IO] = "FAIL_IO",
+ [QUEUE_FLAG_STACKABLE] = "STACKABLE",
+ [QUEUE_FLAG_NONROT] = "NONROT",
+ [QUEUE_FLAG_IO_STAT] = "IO_STAT",
+ [QUEUE_FLAG_DISCARD] = "DISCARD",
+ [QUEUE_FLAG_NOXMERGES] = "NOXMERGES",
+ [QUEUE_FLAG_ADD_RANDOM] = "ADD_RANDOM",
+ [QUEUE_FLAG_SECERASE] = "SECERASE",
+ [QUEUE_FLAG_SAME_FORCE] = "SAME_FORCE",
+ [QUEUE_FLAG_DEAD] = "DEAD",
+ [QUEUE_FLAG_INIT_DONE] = "INIT_DONE",
+ [QUEUE_FLAG_NO_SG_MERGE] = "NO_SG_MERGE",
+ [QUEUE_FLAG_POLL] = "POLL",
+ [QUEUE_FLAG_WC] = "WC",
+ [QUEUE_FLAG_FUA] = "FUA",
+ [QUEUE_FLAG_FLUSH_NQ] = "FLUSH_NQ",
+ [QUEUE_FLAG_DAX] = "DAX",
+ [QUEUE_FLAG_STATS] = "STATS",
+ [QUEUE_FLAG_POLL_STATS] = "POLL_STATS",
+ [QUEUE_FLAG_REGISTERED] = "REGISTERED",
+};
+
+static int blk_queue_flags_show(struct seq_file *m, void *v)
+{
+ struct request_queue *q = m->private;
+
+ blk_flags_show(m, q->queue_flags, blk_queue_flag_name,
+ ARRAY_SIZE(blk_queue_flag_name));
+ seq_puts(m, "\n");
+ return 0;
+}
+
+static ssize_t blk_queue_flags_store(struct file *file, const char __user *ubuf,
+ size_t len, loff_t *offp)
+{
+ struct request_queue *q = file_inode(file)->i_private;
+ char op[16] = { }, *s;
+
+ len = min(len, sizeof(op) - 1);
+ if (copy_from_user(op, ubuf, len))
+ return -EFAULT;
+ s = op;
+ strsep(&s, " \t\n"); /* strip trailing whitespace */
+ if (strcmp(op, "run") == 0) {
+ blk_mq_run_hw_queues(q, true);
+ } else if (strcmp(op, "start") == 0) {
+ blk_mq_start_stopped_hw_queues(q, true);
+ } else {
+ pr_err("%s: unsupported operation %s. Use either 'run' or 'start'\n",
+ __func__, op);
+ return -EINVAL;
+ }
+ return len;
+}
+
+static int blk_queue_flags_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, blk_queue_flags_show, inode->i_private);
+}
+
+static const struct file_operations blk_queue_flags_fops = {
+ .open = blk_queue_flags_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+ .write = blk_queue_flags_store,
+};
+
+static void print_stat(struct seq_file *m, struct blk_rq_stat *stat)
+{
+ if (stat->nr_samples) {
+ seq_printf(m, "samples=%d, mean=%lld, min=%llu, max=%llu",
+ stat->nr_samples, stat->mean, stat->min, stat->max);
+ } else {
+ seq_puts(m, "samples=0");
+ }
+}
+
+static int queue_poll_stat_show(struct seq_file *m, void *v)
+{
+ struct request_queue *q = m->private;
+ int bucket;
+
+ for (bucket = 0; bucket < BLK_MQ_POLL_STATS_BKTS/2; bucket++) {
+ seq_printf(m, "read (%d Bytes): ", 1 << (9+bucket));
+ print_stat(m, &q->poll_stat[2*bucket]);
+ seq_puts(m, "\n");
+
+ seq_printf(m, "write (%d Bytes): ", 1 << (9+bucket));
+ print_stat(m, &q->poll_stat[2*bucket+1]);
+ seq_puts(m, "\n");
+ }
+ return 0;
+}
+
+static int queue_poll_stat_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, queue_poll_stat_show, inode->i_private);
+}
+
+static const struct file_operations queue_poll_stat_fops = {
+ .open = queue_poll_stat_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static const char *const hctx_state_name[] = {
+ [BLK_MQ_S_STOPPED] = "STOPPED",
+ [BLK_MQ_S_TAG_ACTIVE] = "TAG_ACTIVE",
+ [BLK_MQ_S_SCHED_RESTART] = "SCHED_RESTART",
+ [BLK_MQ_S_TAG_WAITING] = "TAG_WAITING",
+
+};
static int hctx_state_show(struct seq_file *m, void *v)
{
struct blk_mq_hw_ctx *hctx = m->private;
- seq_printf(m, "0x%lx\n", hctx->state);
+ blk_flags_show(m, hctx->state, hctx_state_name,
+ ARRAY_SIZE(hctx_state_name));
+ seq_puts(m, "\n");
return 0;
}
@@ -63,11 +209,35 @@
.release = single_release,
};
+static const char *const alloc_policy_name[] = {
+ [BLK_TAG_ALLOC_FIFO] = "fifo",
+ [BLK_TAG_ALLOC_RR] = "rr",
+};
+
+static const char *const hctx_flag_name[] = {
+ [ilog2(BLK_MQ_F_SHOULD_MERGE)] = "SHOULD_MERGE",
+ [ilog2(BLK_MQ_F_TAG_SHARED)] = "TAG_SHARED",
+ [ilog2(BLK_MQ_F_SG_MERGE)] = "SG_MERGE",
+ [ilog2(BLK_MQ_F_BLOCKING)] = "BLOCKING",
+ [ilog2(BLK_MQ_F_NO_SCHED)] = "NO_SCHED",
+};
+
static int hctx_flags_show(struct seq_file *m, void *v)
{
struct blk_mq_hw_ctx *hctx = m->private;
+ const int alloc_policy = BLK_MQ_FLAG_TO_ALLOC_POLICY(hctx->flags);
- seq_printf(m, "0x%lx\n", hctx->flags);
+ seq_puts(m, "alloc_policy=");
+ if (alloc_policy < ARRAY_SIZE(alloc_policy_name) &&
+ alloc_policy_name[alloc_policy])
+ seq_puts(m, alloc_policy_name[alloc_policy]);
+ else
+ seq_printf(m, "%d", alloc_policy);
+ seq_puts(m, " ");
+ blk_flags_show(m,
+ hctx->flags ^ BLK_ALLOC_POLICY_TO_MQ_FLAG(alloc_policy),
+ hctx_flag_name, ARRAY_SIZE(hctx_flag_name));
+ seq_puts(m, "\n");
return 0;
}
@@ -83,13 +253,83 @@
.release = single_release,
};
+static const char *const op_name[] = {
+ [REQ_OP_READ] = "READ",
+ [REQ_OP_WRITE] = "WRITE",
+ [REQ_OP_FLUSH] = "FLUSH",
+ [REQ_OP_DISCARD] = "DISCARD",
+ [REQ_OP_ZONE_REPORT] = "ZONE_REPORT",
+ [REQ_OP_SECURE_ERASE] = "SECURE_ERASE",
+ [REQ_OP_ZONE_RESET] = "ZONE_RESET",
+ [REQ_OP_WRITE_SAME] = "WRITE_SAME",
+ [REQ_OP_WRITE_ZEROES] = "WRITE_ZEROES",
+ [REQ_OP_SCSI_IN] = "SCSI_IN",
+ [REQ_OP_SCSI_OUT] = "SCSI_OUT",
+ [REQ_OP_DRV_IN] = "DRV_IN",
+ [REQ_OP_DRV_OUT] = "DRV_OUT",
+};
+
+static const char *const cmd_flag_name[] = {
+ [__REQ_FAILFAST_DEV] = "FAILFAST_DEV",
+ [__REQ_FAILFAST_TRANSPORT] = "FAILFAST_TRANSPORT",
+ [__REQ_FAILFAST_DRIVER] = "FAILFAST_DRIVER",
+ [__REQ_SYNC] = "SYNC",
+ [__REQ_META] = "META",
+ [__REQ_PRIO] = "PRIO",
+ [__REQ_NOMERGE] = "NOMERGE",
+ [__REQ_IDLE] = "IDLE",
+ [__REQ_INTEGRITY] = "INTEGRITY",
+ [__REQ_FUA] = "FUA",
+ [__REQ_PREFLUSH] = "PREFLUSH",
+ [__REQ_RAHEAD] = "RAHEAD",
+ [__REQ_BACKGROUND] = "BACKGROUND",
+ [__REQ_NR_BITS] = "NR_BITS",
+};
+
+static const char *const rqf_name[] = {
+ [ilog2((__force u32)RQF_SORTED)] = "SORTED",
+ [ilog2((__force u32)RQF_STARTED)] = "STARTED",
+ [ilog2((__force u32)RQF_QUEUED)] = "QUEUED",
+ [ilog2((__force u32)RQF_SOFTBARRIER)] = "SOFTBARRIER",
+ [ilog2((__force u32)RQF_FLUSH_SEQ)] = "FLUSH_SEQ",
+ [ilog2((__force u32)RQF_MIXED_MERGE)] = "MIXED_MERGE",
+ [ilog2((__force u32)RQF_MQ_INFLIGHT)] = "MQ_INFLIGHT",
+ [ilog2((__force u32)RQF_DONTPREP)] = "DONTPREP",
+ [ilog2((__force u32)RQF_PREEMPT)] = "PREEMPT",
+ [ilog2((__force u32)RQF_COPY_USER)] = "COPY_USER",
+ [ilog2((__force u32)RQF_FAILED)] = "FAILED",
+ [ilog2((__force u32)RQF_QUIET)] = "QUIET",
+ [ilog2((__force u32)RQF_ELVPRIV)] = "ELVPRIV",
+ [ilog2((__force u32)RQF_IO_STAT)] = "IO_STAT",
+ [ilog2((__force u32)RQF_ALLOCED)] = "ALLOCED",
+ [ilog2((__force u32)RQF_PM)] = "PM",
+ [ilog2((__force u32)RQF_HASHED)] = "HASHED",
+ [ilog2((__force u32)RQF_STATS)] = "STATS",
+ [ilog2((__force u32)RQF_SPECIAL_PAYLOAD)] = "SPECIAL_PAYLOAD",
+};
+
static int blk_mq_debugfs_rq_show(struct seq_file *m, void *v)
{
struct request *rq = list_entry_rq(v);
+ const struct blk_mq_ops *const mq_ops = rq->q->mq_ops;
+ const unsigned int op = rq->cmd_flags & REQ_OP_MASK;
- seq_printf(m, "%p {.cmd_flags=0x%x, .rq_flags=0x%x, .tag=%d, .internal_tag=%d}\n",
- rq, rq->cmd_flags, (__force unsigned int)rq->rq_flags,
- rq->tag, rq->internal_tag);
+ seq_printf(m, "%p {.op=", rq);
+ if (op < ARRAY_SIZE(op_name) && op_name[op])
+ seq_printf(m, "%s", op_name[op]);
+ else
+ seq_printf(m, "%d", op);
+ seq_puts(m, ", .cmd_flags=");
+ blk_flags_show(m, rq->cmd_flags & ~REQ_OP_MASK, cmd_flag_name,
+ ARRAY_SIZE(cmd_flag_name));
+ seq_puts(m, ", .rq_flags=");
+ blk_flags_show(m, (__force unsigned int)rq->rq_flags, rqf_name,
+ ARRAY_SIZE(rqf_name));
+ seq_printf(m, ", .tag=%d, .internal_tag=%d", rq->tag,
+ rq->internal_tag);
+ if (mq_ops->show_rq)
+ mq_ops->show_rq(m, rq);
+ seq_puts(m, "}\n");
return 0;
}
@@ -322,60 +562,6 @@
.release = single_release,
};
-static void print_stat(struct seq_file *m, struct blk_rq_stat *stat)
-{
- seq_printf(m, "samples=%d, mean=%lld, min=%llu, max=%llu",
- stat->nr_samples, stat->mean, stat->min, stat->max);
-}
-
-static int hctx_stats_show(struct seq_file *m, void *v)
-{
- struct blk_mq_hw_ctx *hctx = m->private;
- struct blk_rq_stat stat[2];
-
- blk_stat_init(&stat[BLK_STAT_READ]);
- blk_stat_init(&stat[BLK_STAT_WRITE]);
-
- blk_hctx_stat_get(hctx, stat);
-
- seq_puts(m, "read: ");
- print_stat(m, &stat[BLK_STAT_READ]);
- seq_puts(m, "\n");
-
- seq_puts(m, "write: ");
- print_stat(m, &stat[BLK_STAT_WRITE]);
- seq_puts(m, "\n");
- return 0;
-}
-
-static int hctx_stats_open(struct inode *inode, struct file *file)
-{
- return single_open(file, hctx_stats_show, inode->i_private);
-}
-
-static ssize_t hctx_stats_write(struct file *file, const char __user *buf,
- size_t count, loff_t *ppos)
-{
- struct seq_file *m = file->private_data;
- struct blk_mq_hw_ctx *hctx = m->private;
- struct blk_mq_ctx *ctx;
- int i;
-
- hctx_for_each_ctx(hctx, ctx, i) {
- blk_stat_init(&ctx->stat[BLK_STAT_READ]);
- blk_stat_init(&ctx->stat[BLK_STAT_WRITE]);
- }
- return count;
-}
-
-static const struct file_operations hctx_stats_fops = {
- .open = hctx_stats_open,
- .read = seq_read,
- .write = hctx_stats_write,
- .llseek = seq_lseek,
- .release = single_release,
-};
-
static int hctx_dispatched_show(struct seq_file *m, void *v)
{
struct blk_mq_hw_ctx *hctx = m->private;
@@ -636,6 +822,12 @@
.release = single_release,
};
+static const struct blk_mq_debugfs_attr blk_mq_debugfs_queue_attrs[] = {
+ {"poll_stat", 0400, &queue_poll_stat_fops},
+ {"state", 0600, &blk_queue_flags_fops},
+ {},
+};
+
static const struct blk_mq_debugfs_attr blk_mq_debugfs_hctx_attrs[] = {
{"state", 0400, &hctx_state_fops},
{"flags", 0400, &hctx_flags_fops},
@@ -646,7 +838,6 @@
{"sched_tags", 0400, &hctx_sched_tags_fops},
{"sched_tags_bitmap", 0400, &hctx_sched_tags_bitmap_fops},
{"io_poll", 0600, &hctx_io_poll_fops},
- {"stats", 0600, &hctx_stats_fops},
{"dispatched", 0600, &hctx_dispatched_fops},
{"queued", 0600, &hctx_queued_fops},
{"run", 0600, &hctx_run_fops},
@@ -662,16 +853,17 @@
{},
};
-int blk_mq_debugfs_register(struct request_queue *q, const char *name)
+int blk_mq_debugfs_register(struct request_queue *q)
{
if (!blk_debugfs_root)
return -ENOENT;
- q->debugfs_dir = debugfs_create_dir(name, blk_debugfs_root);
+ q->debugfs_dir = debugfs_create_dir(kobject_name(q->kobj.parent),
+ blk_debugfs_root);
if (!q->debugfs_dir)
goto err;
- if (blk_mq_debugfs_register_hctxs(q))
+ if (blk_mq_debugfs_register_mq(q))
goto err;
return 0;
@@ -741,7 +933,7 @@
return 0;
}
-int blk_mq_debugfs_register_hctxs(struct request_queue *q)
+int blk_mq_debugfs_register_mq(struct request_queue *q)
{
struct blk_mq_hw_ctx *hctx;
int i;
@@ -753,6 +945,9 @@
if (!q->mq_debugfs_dir)
goto err;
+ if (!debugfs_create_files(q->mq_debugfs_dir, q, blk_mq_debugfs_queue_attrs))
+ goto err;
+
queue_for_each_hw_ctx(q, hctx, i) {
if (blk_mq_debugfs_register_hctx(q, hctx))
goto err;
@@ -761,11 +956,11 @@
return 0;
err:
- blk_mq_debugfs_unregister_hctxs(q);
+ blk_mq_debugfs_unregister_mq(q);
return -ENOMEM;
}
-void blk_mq_debugfs_unregister_hctxs(struct request_queue *q)
+void blk_mq_debugfs_unregister_mq(struct request_queue *q)
{
debugfs_remove_recursive(q->mq_debugfs_dir);
q->mq_debugfs_dir = NULL;
diff --git a/block/blk-mq-pci.c b/block/blk-mq-pci.c
index 966c216..0c3354c 100644
--- a/block/blk-mq-pci.c
+++ b/block/blk-mq-pci.c
@@ -23,7 +23,7 @@
* @pdev: PCI device associated with @set.
*
* This function assumes the PCI device @pdev has at least as many available
- * interrupt vetors as @set has queues. It will then queuery the vector
+ * interrupt vectors as @set has queues. It will then query the vector
* corresponding to each queue for it's affinity mask and built queue mapping
* that maps a queue to the CPUs that have irq affinity for the corresponding
* vector.
diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c
index c974a1b..8b361e1 100644
--- a/block/blk-mq-sched.c
+++ b/block/blk-mq-sched.c
@@ -30,43 +30,6 @@
}
EXPORT_SYMBOL_GPL(blk_mq_sched_free_hctx_data);
-int blk_mq_sched_init_hctx_data(struct request_queue *q, size_t size,
- int (*init)(struct blk_mq_hw_ctx *),
- void (*exit)(struct blk_mq_hw_ctx *))
-{
- struct blk_mq_hw_ctx *hctx;
- int ret;
- int i;
-
- queue_for_each_hw_ctx(q, hctx, i) {
- hctx->sched_data = kmalloc_node(size, GFP_KERNEL, hctx->numa_node);
- if (!hctx->sched_data) {
- ret = -ENOMEM;
- goto error;
- }
-
- if (init) {
- ret = init(hctx);
- if (ret) {
- /*
- * We don't want to give exit() a partially
- * initialized sched_data. init() must clean up
- * if it fails.
- */
- kfree(hctx->sched_data);
- hctx->sched_data = NULL;
- goto error;
- }
- }
- }
-
- return 0;
-error:
- blk_mq_sched_free_hctx_data(q, exit);
- return ret;
-}
-EXPORT_SYMBOL_GPL(blk_mq_sched_init_hctx_data);
-
static void __blk_mq_sched_assign_ioc(struct request_queue *q,
struct request *rq,
struct bio *bio,
@@ -119,7 +82,11 @@
if (likely(!data->hctx))
data->hctx = blk_mq_map_queue(q, data->ctx->cpu);
- if (e) {
+ /*
+ * For a reserved tag, allocate a normal request since we might
+ * have driver dependencies on the value of the internal tag.
+ */
+ if (e && !(data->flags & BLK_MQ_REQ_RESERVED)) {
data->flags |= BLK_MQ_REQ_INTERNAL;
/*
@@ -227,22 +194,6 @@
}
}
-void blk_mq_sched_move_to_dispatch(struct blk_mq_hw_ctx *hctx,
- struct list_head *rq_list,
- struct request *(*get_rq)(struct blk_mq_hw_ctx *))
-{
- do {
- struct request *rq;
-
- rq = get_rq(hctx);
- if (!rq)
- break;
-
- list_add_tail(&rq->queuelist, rq_list);
- } while (1);
-}
-EXPORT_SYMBOL_GPL(blk_mq_sched_move_to_dispatch);
-
bool blk_mq_sched_try_merge(struct request_queue *q, struct bio *bio,
struct request **merged_request)
{
@@ -508,11 +459,24 @@
unsigned int hctx_idx)
{
struct elevator_queue *e = q->elevator;
+ int ret;
if (!e)
return 0;
- return blk_mq_sched_alloc_tags(q, hctx, hctx_idx);
+ ret = blk_mq_sched_alloc_tags(q, hctx, hctx_idx);
+ if (ret)
+ return ret;
+
+ if (e->type->ops.mq.init_hctx) {
+ ret = e->type->ops.mq.init_hctx(hctx, hctx_idx);
+ if (ret) {
+ blk_mq_sched_free_tags(q->tag_set, hctx, hctx_idx);
+ return ret;
+ }
+ }
+
+ return 0;
}
void blk_mq_sched_exit_hctx(struct request_queue *q, struct blk_mq_hw_ctx *hctx,
@@ -523,12 +487,18 @@
if (!e)
return;
+ if (e->type->ops.mq.exit_hctx && hctx->sched_data) {
+ e->type->ops.mq.exit_hctx(hctx, hctx_idx);
+ hctx->sched_data = NULL;
+ }
+
blk_mq_sched_free_tags(q->tag_set, hctx, hctx_idx);
}
int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e)
{
struct blk_mq_hw_ctx *hctx;
+ struct elevator_queue *eq;
unsigned int i;
int ret;
@@ -553,6 +523,18 @@
if (ret)
goto err;
+ if (e->ops.mq.init_hctx) {
+ queue_for_each_hw_ctx(q, hctx, i) {
+ ret = e->ops.mq.init_hctx(hctx, i);
+ if (ret) {
+ eq = q->elevator;
+ blk_mq_exit_sched(q, eq);
+ kobject_put(&eq->kobj);
+ return ret;
+ }
+ }
+ }
+
return 0;
err:
@@ -563,6 +545,17 @@
void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e)
{
+ struct blk_mq_hw_ctx *hctx;
+ unsigned int i;
+
+ if (e->type->ops.mq.exit_hctx) {
+ queue_for_each_hw_ctx(q, hctx, i) {
+ if (hctx->sched_data) {
+ e->type->ops.mq.exit_hctx(hctx, i);
+ hctx->sched_data = NULL;
+ }
+ }
+ }
if (e->type->ops.mq.exit_sched)
e->type->ops.mq.exit_sched(e);
blk_mq_sched_tags_teardown(q);
diff --git a/block/blk-mq-sched.h b/block/blk-mq-sched.h
index 3a9e6e4..edafb53 100644
--- a/block/blk-mq-sched.h
+++ b/block/blk-mq-sched.h
@@ -4,10 +4,6 @@
#include "blk-mq.h"
#include "blk-mq-tag.h"
-int blk_mq_sched_init_hctx_data(struct request_queue *q, size_t size,
- int (*init)(struct blk_mq_hw_ctx *),
- void (*exit)(struct blk_mq_hw_ctx *));
-
void blk_mq_sched_free_hctx_data(struct request_queue *q,
void (*exit)(struct blk_mq_hw_ctx *));
@@ -28,9 +24,6 @@
struct list_head *list, bool run_queue_async);
void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx);
-void blk_mq_sched_move_to_dispatch(struct blk_mq_hw_ctx *hctx,
- struct list_head *rq_list,
- struct request *(*get_rq)(struct blk_mq_hw_ctx *));
int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e);
void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e);
@@ -86,17 +79,12 @@
return true;
}
-static inline void
-blk_mq_sched_completed_request(struct blk_mq_hw_ctx *hctx, struct request *rq)
+static inline void blk_mq_sched_completed_request(struct request *rq)
{
- struct elevator_queue *e = hctx->queue->elevator;
+ struct elevator_queue *e = rq->q->elevator;
if (e && e->type->ops.mq.completed_request)
- e->type->ops.mq.completed_request(hctx, rq);
-
- BUG_ON(rq->internal_tag == -1);
-
- blk_mq_put_tag(hctx, hctx->sched_tags, rq->mq_ctx, rq->internal_tag);
+ e->type->ops.mq.completed_request(rq);
}
static inline void blk_mq_sched_started_request(struct request *rq)
diff --git a/block/blk-mq-sysfs.c b/block/blk-mq-sysfs.c
index d745ab8..ec0afdf 100644
--- a/block/blk-mq-sysfs.c
+++ b/block/blk-mq-sysfs.c
@@ -253,10 +253,12 @@
struct blk_mq_hw_ctx *hctx;
int i;
+ lockdep_assert_held(&q->sysfs_lock);
+
queue_for_each_hw_ctx(q, hctx, i)
blk_mq_unregister_hctx(hctx);
- blk_mq_debugfs_unregister_hctxs(q);
+ blk_mq_debugfs_unregister_mq(q);
kobject_uevent(&q->mq_kobj, KOBJ_REMOVE);
kobject_del(&q->mq_kobj);
@@ -267,9 +269,9 @@
void blk_mq_unregister_dev(struct device *dev, struct request_queue *q)
{
- blk_mq_disable_hotplug();
+ mutex_lock(&q->sysfs_lock);
__blk_mq_unregister_dev(dev, q);
- blk_mq_enable_hotplug();
+ mutex_unlock(&q->sysfs_lock);
}
void blk_mq_hctx_kobj_init(struct blk_mq_hw_ctx *hctx)
@@ -302,12 +304,13 @@
}
}
-int blk_mq_register_dev(struct device *dev, struct request_queue *q)
+int __blk_mq_register_dev(struct device *dev, struct request_queue *q)
{
struct blk_mq_hw_ctx *hctx;
int ret, i;
- blk_mq_disable_hotplug();
+ WARN_ON_ONCE(!q->kobj.parent);
+ lockdep_assert_held(&q->sysfs_lock);
ret = kobject_add(&q->mq_kobj, kobject_get(&dev->kobj), "%s", "mq");
if (ret < 0)
@@ -315,20 +318,38 @@
kobject_uevent(&q->mq_kobj, KOBJ_ADD);
- blk_mq_debugfs_register(q, kobject_name(&dev->kobj));
+ blk_mq_debugfs_register(q);
queue_for_each_hw_ctx(q, hctx, i) {
ret = blk_mq_register_hctx(hctx);
if (ret)
- break;
+ goto unreg;
}
- if (ret)
- __blk_mq_unregister_dev(dev, q);
- else
- q->mq_sysfs_init_done = true;
+ q->mq_sysfs_init_done = true;
+
out:
- blk_mq_enable_hotplug();
+ return ret;
+
+unreg:
+ while (--i >= 0)
+ blk_mq_unregister_hctx(q->queue_hw_ctx[i]);
+
+ blk_mq_debugfs_unregister_mq(q);
+
+ kobject_uevent(&q->mq_kobj, KOBJ_REMOVE);
+ kobject_del(&q->mq_kobj);
+ kobject_put(&dev->kobj);
+ return ret;
+}
+
+int blk_mq_register_dev(struct device *dev, struct request_queue *q)
+{
+ int ret;
+
+ mutex_lock(&q->sysfs_lock);
+ ret = __blk_mq_register_dev(dev, q);
+ mutex_unlock(&q->sysfs_lock);
return ret;
}
@@ -339,13 +360,17 @@
struct blk_mq_hw_ctx *hctx;
int i;
+ mutex_lock(&q->sysfs_lock);
if (!q->mq_sysfs_init_done)
- return;
+ goto unlock;
- blk_mq_debugfs_unregister_hctxs(q);
+ blk_mq_debugfs_unregister_mq(q);
queue_for_each_hw_ctx(q, hctx, i)
blk_mq_unregister_hctx(hctx);
+
+unlock:
+ mutex_unlock(&q->sysfs_lock);
}
int blk_mq_sysfs_register(struct request_queue *q)
@@ -353,10 +378,11 @@
struct blk_mq_hw_ctx *hctx;
int i, ret = 0;
+ mutex_lock(&q->sysfs_lock);
if (!q->mq_sysfs_init_done)
- return ret;
+ goto unlock;
- blk_mq_debugfs_register_hctxs(q);
+ blk_mq_debugfs_register_mq(q);
queue_for_each_hw_ctx(q, hctx, i) {
ret = blk_mq_register_hctx(hctx);
@@ -364,5 +390,8 @@
break;
}
+unlock:
+ mutex_unlock(&q->sysfs_lock);
+
return ret;
}
diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
index 9d97bfc..d0be72c 100644
--- a/block/blk-mq-tag.c
+++ b/block/blk-mq-tag.c
@@ -96,7 +96,10 @@
if (!(data->flags & BLK_MQ_REQ_INTERNAL) &&
!hctx_may_queue(data->hctx, bt))
return -1;
- return __sbitmap_queue_get(bt);
+ if (data->shallow_depth)
+ return __sbitmap_queue_get_shallow(bt, data->shallow_depth);
+ else
+ return __sbitmap_queue_get(bt);
}
unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index c7836a1..bf90684 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -39,6 +39,26 @@
static DEFINE_MUTEX(all_q_mutex);
static LIST_HEAD(all_q_list);
+static void blk_mq_poll_stats_start(struct request_queue *q);
+static void blk_mq_poll_stats_fn(struct blk_stat_callback *cb);
+
+static int blk_mq_poll_stats_bkt(const struct request *rq)
+{
+ int ddir, bytes, bucket;
+
+ ddir = rq_data_dir(rq);
+ bytes = blk_rq_bytes(rq);
+
+ bucket = ddir + 2*(ilog2(bytes) - 9);
+
+ if (bucket < 0)
+ return -1;
+ else if (bucket >= BLK_MQ_POLL_STATS_BKTS)
+ return ddir + BLK_MQ_POLL_STATS_BKTS - 2;
+
+ return bucket;
+}
+
/*
* Check if any of the ctx's have pending work in this hardware queue
*/
@@ -65,7 +85,7 @@
sbitmap_clear_bit(&hctx->ctx_map, ctx->index_hw);
}
-void blk_mq_freeze_queue_start(struct request_queue *q)
+void blk_freeze_queue_start(struct request_queue *q)
{
int freeze_depth;
@@ -75,7 +95,7 @@
blk_mq_run_hw_queues(q, false);
}
}
-EXPORT_SYMBOL_GPL(blk_mq_freeze_queue_start);
+EXPORT_SYMBOL_GPL(blk_freeze_queue_start);
void blk_mq_freeze_queue_wait(struct request_queue *q)
{
@@ -105,7 +125,7 @@
* no blk_unfreeze_queue(), and blk_freeze_queue() is not
* exported to drivers as the only user for unfreeze is blk_mq.
*/
- blk_mq_freeze_queue_start(q);
+ blk_freeze_queue_start(q);
blk_mq_freeze_queue_wait(q);
}
@@ -210,7 +230,6 @@
#endif
rq->special = NULL;
/* tag was already set */
- rq->errors = 0;
rq->extra_len = 0;
INIT_LIST_HEAD(&rq->timeout_list);
@@ -347,7 +366,7 @@
if (rq->tag != -1)
blk_mq_put_tag(hctx, hctx->tags, ctx, rq->tag);
if (sched_tag != -1)
- blk_mq_sched_completed_request(hctx, rq);
+ blk_mq_put_tag(hctx, hctx->sched_tags, ctx, sched_tag);
blk_mq_sched_restart(hctx);
blk_queue_exit(q);
}
@@ -365,6 +384,7 @@
{
blk_mq_finish_hctx_request(blk_mq_map_queue(rq->q, rq->mq_ctx->cpu), rq);
}
+EXPORT_SYMBOL_GPL(blk_mq_finish_request);
void blk_mq_free_request(struct request *rq)
{
@@ -402,12 +422,19 @@
rq->q->softirq_done_fn(rq);
}
-static void blk_mq_ipi_complete_request(struct request *rq)
+static void __blk_mq_complete_request(struct request *rq)
{
struct blk_mq_ctx *ctx = rq->mq_ctx;
bool shared = false;
int cpu;
+ if (rq->internal_tag != -1)
+ blk_mq_sched_completed_request(rq);
+ if (rq->rq_flags & RQF_STATS) {
+ blk_mq_poll_stats_start(rq->q);
+ blk_stat_add(rq);
+ }
+
if (!test_bit(QUEUE_FLAG_SAME_COMP, &rq->q->queue_flags)) {
rq->q->softirq_done_fn(rq);
return;
@@ -428,33 +455,6 @@
put_cpu();
}
-static void blk_mq_stat_add(struct request *rq)
-{
- if (rq->rq_flags & RQF_STATS) {
- /*
- * We could rq->mq_ctx here, but there's less of a risk
- * of races if we have the completion event add the stats
- * to the local software queue.
- */
- struct blk_mq_ctx *ctx;
-
- ctx = __blk_mq_get_ctx(rq->q, raw_smp_processor_id());
- blk_stat_add(&ctx->stat[rq_data_dir(rq)], rq);
- }
-}
-
-static void __blk_mq_complete_request(struct request *rq)
-{
- struct request_queue *q = rq->q;
-
- blk_mq_stat_add(rq);
-
- if (!q->softirq_done_fn)
- blk_mq_end_request(rq, rq->errors);
- else
- blk_mq_ipi_complete_request(rq);
-}
-
/**
* blk_mq_complete_request - end I/O on a request
* @rq: the request being processed
@@ -463,16 +463,14 @@
* Ends all I/O on a request. It does not handle partial completions.
* The actual completion happens out-of-order, through a IPI handler.
**/
-void blk_mq_complete_request(struct request *rq, int error)
+void blk_mq_complete_request(struct request *rq)
{
struct request_queue *q = rq->q;
if (unlikely(blk_should_fake_timeout(q)))
return;
- if (!blk_mark_rq_complete(rq)) {
- rq->errors = error;
+ if (!blk_mark_rq_complete(rq))
__blk_mq_complete_request(rq);
- }
}
EXPORT_SYMBOL(blk_mq_complete_request);
@@ -491,7 +489,7 @@
trace_block_rq_issue(q, rq);
if (test_bit(QUEUE_FLAG_STATS, &q->queue_flags)) {
- blk_stat_set_issue_time(&rq->issue_stat);
+ blk_stat_set_issue(&rq->issue_stat, blk_rq_sectors(rq));
rq->rq_flags |= RQF_STATS;
wbt_issue(q->rq_wb, &rq->issue_stat);
}
@@ -526,6 +524,15 @@
}
EXPORT_SYMBOL(blk_mq_start_request);
+/*
+ * When we reach here because queue is busy, REQ_ATOM_COMPLETE
+ * flag isn't set yet, so there may be race with timeout handler,
+ * but given rq->deadline is just set in .queue_rq() under
+ * this situation, the race won't be possible in reality because
+ * rq->timeout should be set as big enough to cover the window
+ * between blk_mq_start_request() called from .queue_rq() and
+ * clearing REQ_ATOM_STARTED here.
+ */
static void __blk_mq_requeue_request(struct request *rq)
{
struct request_queue *q = rq->q;
@@ -633,8 +640,7 @@
rq = list_first_entry(&rq_list, struct request, queuelist);
list_del_init(&rq->queuelist);
- rq->errors = -EIO;
- blk_mq_end_request(rq, rq->errors);
+ blk_mq_end_request(rq, -EIO);
}
}
EXPORT_SYMBOL(blk_mq_abort_requeue_list);
@@ -666,7 +672,7 @@
* just be ignored. This can happen due to the bitflag ordering.
* Timeout first checks if STARTED is set, and if it is, assumes
* the request is active. But if we race with completion, then
- * we both flags will get cleared. So check here again, and ignore
+ * both flags will get cleared. So check here again, and ignore
* a timeout event with a request that isn't active.
*/
if (!test_bit(REQ_ATOM_STARTED, &req->atomic_flags))
@@ -699,6 +705,19 @@
if (!test_bit(REQ_ATOM_STARTED, &rq->atomic_flags))
return;
+ /*
+ * The rq being checked may have been freed and reallocated
+ * out already here, we avoid this race by checking rq->deadline
+ * and REQ_ATOM_COMPLETE flag together:
+ *
+ * - if rq->deadline is observed as new value because of
+ * reusing, the rq won't be timed out because of timing.
+ * - if rq->deadline is observed as previous value,
+ * REQ_ATOM_COMPLETE flag won't be cleared in reuse path
+ * because we put a barrier between setting rq->deadline
+ * and clearing the flag in blk_mq_start_request(), so
+ * this rq won't be timed out too.
+ */
if (time_after_eq(jiffies, rq->deadline)) {
if (!blk_mark_rq_complete(rq))
blk_mq_rq_timed_out(rq, reserved);
@@ -727,7 +746,7 @@
* percpu_ref_tryget directly, because we need to be able to
* obtain a reference even in the short window between the queue
* starting to freeze, by dropping the first reference in
- * blk_mq_freeze_queue_start, and the moment the last request is
+ * blk_freeze_queue_start, and the moment the last request is
* consumed, marked by the instant q_usage_counter reaches
* zero.
*/
@@ -845,6 +864,8 @@
.flags = wait ? 0 : BLK_MQ_REQ_NOWAIT,
};
+ might_sleep_if(wait);
+
if (rq->tag != -1)
goto done;
@@ -964,20 +985,12 @@
{
struct blk_mq_hw_ctx *hctx;
struct request *rq;
- LIST_HEAD(driver_list);
- struct list_head *dptr;
int errors, queued, ret = BLK_MQ_RQ_QUEUE_OK;
if (list_empty(list))
return false;
/*
- * Start off with dptr being NULL, so we start the first request
- * immediately, even if we have more pending.
- */
- dptr = NULL;
-
- /*
* Now process all the entries, sending them to the driver.
*/
errors = queued = 0;
@@ -993,23 +1006,21 @@
* The initial allocation attempt failed, so we need to
* rerun the hardware queue when a tag is freed.
*/
- if (blk_mq_dispatch_wait_add(hctx)) {
- /*
- * It's possible that a tag was freed in the
- * window between the allocation failure and
- * adding the hardware queue to the wait queue.
- */
- if (!blk_mq_get_driver_tag(rq, &hctx, false))
- break;
- } else {
+ if (!blk_mq_dispatch_wait_add(hctx))
break;
- }
+
+ /*
+ * It's possible that a tag was freed in the window
+ * between the allocation failure and adding the
+ * hardware queue to the wait queue.
+ */
+ if (!blk_mq_get_driver_tag(rq, &hctx, false))
+ break;
}
list_del_init(&rq->queuelist);
bd.rq = rq;
- bd.list = dptr;
/*
* Flag last if we have no more requests, or if we have more
@@ -1038,20 +1049,12 @@
pr_err("blk-mq: bad return on queue: %d\n", ret);
case BLK_MQ_RQ_QUEUE_ERROR:
errors++;
- rq->errors = -EIO;
- blk_mq_end_request(rq, rq->errors);
+ blk_mq_end_request(rq, -EIO);
break;
}
if (ret == BLK_MQ_RQ_QUEUE_BUSY)
break;
-
- /*
- * We've done the first request. If we have more than 1
- * left in the list, set dptr to defer issue.
- */
- if (!dptr && list->next != list->prev)
- dptr = &driver_list;
} while (!list_empty(list));
hctx->dispatched[queued_to_index(queued)]++;
@@ -1062,8 +1065,8 @@
*/
if (!list_empty(list)) {
/*
- * If we got a driver tag for the next request already,
- * free it again.
+ * If an I/O scheduler has been configured and we got a driver
+ * tag for the next request already, free it again.
*/
rq = list_first_entry(list, struct request, queuelist);
blk_mq_put_driver_tag(rq);
@@ -1073,16 +1076,24 @@
spin_unlock(&hctx->lock);
/*
- * the queue is expected stopped with BLK_MQ_RQ_QUEUE_BUSY, but
- * it's possible the queue is stopped and restarted again
- * before this. Queue restart will dispatch requests. And since
- * requests in rq_list aren't added into hctx->dispatch yet,
- * the requests in rq_list might get lost.
+ * If SCHED_RESTART was set by the caller of this function and
+ * it is no longer set that means that it was cleared by another
+ * thread and hence that a queue rerun is needed.
*
- * blk_mq_run_hw_queue() already checks the STOPPED bit
+ * If TAG_WAITING is set that means that an I/O scheduler has
+ * been configured and another thread is waiting for a driver
+ * tag. To guarantee fairness, do not rerun this hardware queue
+ * but let the other thread grab the driver tag.
*
- * If RESTART or TAG_WAITING is set, then let completion restart
- * the queue instead of potentially looping here.
+ * If no I/O scheduler has been configured it is possible that
+ * the hardware queue got stopped and restarted before requests
+ * were pushed back onto the dispatch list. Rerun the queue to
+ * avoid starvation. Notes:
+ * - blk_mq_run_hw_queue() checks whether or not a queue has
+ * been stopped before rerunning a queue.
+ * - Some but not all block drivers stop a queue before
+ * returning BLK_MQ_RQ_QUEUE_BUSY. Two exceptions are scsi-mq
+ * and dm-rq.
*/
if (!blk_mq_sched_needs_restart(hctx) &&
!test_bit(BLK_MQ_S_TAG_WAITING, &hctx->state))
@@ -1104,6 +1115,8 @@
blk_mq_sched_dispatch_requests(hctx);
rcu_read_unlock();
} else {
+ might_sleep();
+
srcu_idx = srcu_read_lock(&hctx->queue_rq_srcu);
blk_mq_sched_dispatch_requests(hctx);
srcu_read_unlock(&hctx->queue_rq_srcu, srcu_idx);
@@ -1153,13 +1166,9 @@
put_cpu();
}
- if (msecs == 0)
- kblockd_schedule_work_on(blk_mq_hctx_next_cpu(hctx),
- &hctx->run_work);
- else
- kblockd_schedule_delayed_work_on(blk_mq_hctx_next_cpu(hctx),
- &hctx->delayed_run_work,
- msecs_to_jiffies(msecs));
+ kblockd_schedule_delayed_work_on(blk_mq_hctx_next_cpu(hctx),
+ &hctx->run_work,
+ msecs_to_jiffies(msecs));
}
void blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs)
@@ -1172,6 +1181,7 @@
{
__blk_mq_delay_run_hw_queue(hctx, async, 0);
}
+EXPORT_SYMBOL(blk_mq_run_hw_queue);
void blk_mq_run_hw_queues(struct request_queue *q, bool async)
{
@@ -1210,8 +1220,7 @@
void blk_mq_stop_hw_queue(struct blk_mq_hw_ctx *hctx)
{
- cancel_work(&hctx->run_work);
- cancel_delayed_work(&hctx->delay_work);
+ cancel_delayed_work_sync(&hctx->run_work);
set_bit(BLK_MQ_S_STOPPED, &hctx->state);
}
EXPORT_SYMBOL(blk_mq_stop_hw_queue);
@@ -1268,38 +1277,40 @@
{
struct blk_mq_hw_ctx *hctx;
- hctx = container_of(work, struct blk_mq_hw_ctx, run_work);
+ hctx = container_of(work, struct blk_mq_hw_ctx, run_work.work);
+
+ /*
+ * If we are stopped, don't run the queue. The exception is if
+ * BLK_MQ_S_START_ON_RUN is set. For that case, we auto-clear
+ * the STOPPED bit and run it.
+ */
+ if (test_bit(BLK_MQ_S_STOPPED, &hctx->state)) {
+ if (!test_bit(BLK_MQ_S_START_ON_RUN, &hctx->state))
+ return;
+
+ clear_bit(BLK_MQ_S_START_ON_RUN, &hctx->state);
+ clear_bit(BLK_MQ_S_STOPPED, &hctx->state);
+ }
__blk_mq_run_hw_queue(hctx);
}
-static void blk_mq_delayed_run_work_fn(struct work_struct *work)
-{
- struct blk_mq_hw_ctx *hctx;
-
- hctx = container_of(work, struct blk_mq_hw_ctx, delayed_run_work.work);
-
- __blk_mq_run_hw_queue(hctx);
-}
-
-static void blk_mq_delay_work_fn(struct work_struct *work)
-{
- struct blk_mq_hw_ctx *hctx;
-
- hctx = container_of(work, struct blk_mq_hw_ctx, delay_work.work);
-
- if (test_and_clear_bit(BLK_MQ_S_STOPPED, &hctx->state))
- __blk_mq_run_hw_queue(hctx);
-}
void blk_mq_delay_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs)
{
if (unlikely(!blk_mq_hw_queue_mapped(hctx)))
return;
+ /*
+ * Stop the hw queue, then modify currently delayed work.
+ * This should prevent us from running the queue prematurely.
+ * Mark the queue as auto-clearing STOPPED when it runs.
+ */
blk_mq_stop_hw_queue(hctx);
- kblockd_schedule_delayed_work_on(blk_mq_hctx_next_cpu(hctx),
- &hctx->delay_work, msecs_to_jiffies(msecs));
+ set_bit(BLK_MQ_S_START_ON_RUN, &hctx->state);
+ kblockd_mod_delayed_work_on(blk_mq_hctx_next_cpu(hctx),
+ &hctx->run_work,
+ msecs_to_jiffies(msecs));
}
EXPORT_SYMBOL(blk_mq_delay_queue);
@@ -1408,7 +1419,7 @@
static void blk_mq_bio_to_request(struct request *rq, struct bio *bio)
{
- init_request_from_bio(rq, bio);
+ blk_init_request_from_bio(rq, bio);
blk_account_io_start(rq, true);
}
@@ -1453,14 +1464,13 @@
return blk_tag_to_qc_t(rq->internal_tag, hctx->queue_num, true);
}
-static void blk_mq_try_issue_directly(struct request *rq, blk_qc_t *cookie,
+static void __blk_mq_try_issue_directly(struct request *rq, blk_qc_t *cookie,
bool may_sleep)
{
struct request_queue *q = rq->q;
struct blk_mq_queue_data bd = {
.rq = rq,
- .list = NULL,
- .last = 1
+ .last = true,
};
struct blk_mq_hw_ctx *hctx;
blk_qc_t new_cookie;
@@ -1485,31 +1495,42 @@
return;
}
- __blk_mq_requeue_request(rq);
-
if (ret == BLK_MQ_RQ_QUEUE_ERROR) {
*cookie = BLK_QC_T_NONE;
- rq->errors = -EIO;
- blk_mq_end_request(rq, rq->errors);
+ blk_mq_end_request(rq, -EIO);
return;
}
+ __blk_mq_requeue_request(rq);
insert:
blk_mq_sched_insert_request(rq, false, true, false, may_sleep);
}
-/*
- * Multiple hardware queue variant. This will not use per-process plugs,
- * but will attempt to bypass the hctx queueing if we can go straight to
- * hardware for SYNC IO.
- */
+static void blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx,
+ struct request *rq, blk_qc_t *cookie)
+{
+ if (!(hctx->flags & BLK_MQ_F_BLOCKING)) {
+ rcu_read_lock();
+ __blk_mq_try_issue_directly(rq, cookie, false);
+ rcu_read_unlock();
+ } else {
+ unsigned int srcu_idx;
+
+ might_sleep();
+
+ srcu_idx = srcu_read_lock(&hctx->queue_rq_srcu);
+ __blk_mq_try_issue_directly(rq, cookie, true);
+ srcu_read_unlock(&hctx->queue_rq_srcu, srcu_idx);
+ }
+}
+
static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)
{
const int is_sync = op_is_sync(bio->bi_opf);
const int is_flush_fua = op_is_flush(bio->bi_opf);
struct blk_mq_alloc_data data = { .flags = 0 };
struct request *rq;
- unsigned int request_count = 0, srcu_idx;
+ unsigned int request_count = 0;
struct blk_plug *plug;
struct request *same_queue_rq = NULL;
blk_qc_t cookie;
@@ -1545,147 +1566,21 @@
cookie = request_to_qc_t(data.hctx, rq);
- if (unlikely(is_flush_fua)) {
- if (q->elevator)
- goto elv_insert;
- blk_mq_bio_to_request(rq, bio);
- blk_insert_flush(rq);
- goto run_queue;
- }
-
plug = current->plug;
- /*
- * If the driver supports defer issued based on 'last', then
- * queue it up like normal since we can potentially save some
- * CPU this way.
- */
- if (((plug && !blk_queue_nomerges(q)) || is_sync) &&
- !(data.hctx->flags & BLK_MQ_F_DEFER_ISSUE)) {
- struct request *old_rq = NULL;
-
- blk_mq_bio_to_request(rq, bio);
-
- /*
- * We do limited plugging. If the bio can be merged, do that.
- * Otherwise the existing request in the plug list will be
- * issued. So the plug list will have one request at most
- */
- if (plug) {
- /*
- * The plug list might get flushed before this. If that
- * happens, same_queue_rq is invalid and plug list is
- * empty
- */
- if (same_queue_rq && !list_empty(&plug->mq_list)) {
- old_rq = same_queue_rq;
- list_del_init(&old_rq->queuelist);
- }
- list_add_tail(&rq->queuelist, &plug->mq_list);
- } else /* is_sync */
- old_rq = rq;
+ if (unlikely(is_flush_fua)) {
blk_mq_put_ctx(data.ctx);
- if (!old_rq)
- goto done;
-
- if (!(data.hctx->flags & BLK_MQ_F_BLOCKING)) {
- rcu_read_lock();
- blk_mq_try_issue_directly(old_rq, &cookie, false);
- rcu_read_unlock();
+ blk_mq_bio_to_request(rq, bio);
+ if (q->elevator) {
+ blk_mq_sched_insert_request(rq, false, true, true,
+ true);
} else {
- srcu_idx = srcu_read_lock(&data.hctx->queue_rq_srcu);
- blk_mq_try_issue_directly(old_rq, &cookie, true);
- srcu_read_unlock(&data.hctx->queue_rq_srcu, srcu_idx);
+ blk_insert_flush(rq);
+ blk_mq_run_hw_queue(data.hctx, true);
}
- goto done;
- }
-
- if (q->elevator) {
-elv_insert:
- blk_mq_put_ctx(data.ctx);
- blk_mq_bio_to_request(rq, bio);
- blk_mq_sched_insert_request(rq, false, true,
- !is_sync || is_flush_fua, true);
- goto done;
- }
- if (!blk_mq_merge_queue_io(data.hctx, data.ctx, rq, bio)) {
- /*
- * For a SYNC request, send it to the hardware immediately. For
- * an ASYNC request, just ensure that we run it later on. The
- * latter allows for merging opportunities and more efficient
- * dispatching.
- */
-run_queue:
- blk_mq_run_hw_queue(data.hctx, !is_sync || is_flush_fua);
- }
- blk_mq_put_ctx(data.ctx);
-done:
- return cookie;
-}
-
-/*
- * Single hardware queue variant. This will attempt to use any per-process
- * plug for merging and IO deferral.
- */
-static blk_qc_t blk_sq_make_request(struct request_queue *q, struct bio *bio)
-{
- const int is_sync = op_is_sync(bio->bi_opf);
- const int is_flush_fua = op_is_flush(bio->bi_opf);
- struct blk_plug *plug;
- unsigned int request_count = 0;
- struct blk_mq_alloc_data data = { .flags = 0 };
- struct request *rq;
- blk_qc_t cookie;
- unsigned int wb_acct;
-
- blk_queue_bounce(q, &bio);
-
- if (bio_integrity_enabled(bio) && bio_integrity_prep(bio)) {
- bio_io_error(bio);
- return BLK_QC_T_NONE;
- }
-
- blk_queue_split(q, &bio, q->bio_split);
-
- if (!is_flush_fua && !blk_queue_nomerges(q)) {
- if (blk_attempt_plug_merge(q, bio, &request_count, NULL))
- return BLK_QC_T_NONE;
- } else
- request_count = blk_plug_queued_count(q);
-
- if (blk_mq_sched_bio_merge(q, bio))
- return BLK_QC_T_NONE;
-
- wb_acct = wbt_wait(q->rq_wb, bio, NULL);
-
- trace_block_getrq(q, bio, bio->bi_opf);
-
- rq = blk_mq_sched_get_request(q, bio, bio->bi_opf, &data);
- if (unlikely(!rq)) {
- __wbt_done(q->rq_wb, wb_acct);
- return BLK_QC_T_NONE;
- }
-
- wbt_track(&rq->issue_stat, wb_acct);
-
- cookie = request_to_qc_t(data.hctx, rq);
-
- if (unlikely(is_flush_fua)) {
- if (q->elevator)
- goto elv_insert;
- blk_mq_bio_to_request(rq, bio);
- blk_insert_flush(rq);
- goto run_queue;
- }
-
- /*
- * A task plug currently exists. Since this is completely lockless,
- * utilize that to temporarily store requests until the task is
- * either done or scheduled away.
- */
- plug = current->plug;
- if (plug) {
+ } else if (plug && q->nr_hw_queues == 1) {
struct request *last = NULL;
+ blk_mq_put_ctx(data.ctx);
blk_mq_bio_to_request(rq, bio);
/*
@@ -1694,13 +1589,14 @@
*/
if (list_empty(&plug->mq_list))
request_count = 0;
+ else if (blk_queue_nomerges(q))
+ request_count = blk_plug_queued_count(q);
+
if (!request_count)
trace_block_plug(q);
else
last = list_entry_rq(plug->mq_list.prev);
- blk_mq_put_ctx(data.ctx);
-
if (request_count >= BLK_MAX_REQUEST_COUNT || (last &&
blk_rq_bytes(last) >= BLK_PLUG_FLUSH_SIZE)) {
blk_flush_plug_list(plug, false);
@@ -1708,30 +1604,41 @@
}
list_add_tail(&rq->queuelist, &plug->mq_list);
- return cookie;
- }
+ } else if (plug && !blk_queue_nomerges(q)) {
+ blk_mq_bio_to_request(rq, bio);
- if (q->elevator) {
-elv_insert:
+ /*
+ * We do limited plugging. If the bio can be merged, do that.
+ * Otherwise the existing request in the plug list will be
+ * issued. So the plug list will have one request at most
+ * The plug list might get flushed before this. If that happens,
+ * the plug list is empty, and same_queue_rq is invalid.
+ */
+ if (list_empty(&plug->mq_list))
+ same_queue_rq = NULL;
+ if (same_queue_rq)
+ list_del_init(&same_queue_rq->queuelist);
+ list_add_tail(&rq->queuelist, &plug->mq_list);
+
+ blk_mq_put_ctx(data.ctx);
+
+ if (same_queue_rq)
+ blk_mq_try_issue_directly(data.hctx, same_queue_rq,
+ &cookie);
+ } else if (q->nr_hw_queues > 1 && is_sync) {
blk_mq_put_ctx(data.ctx);
blk_mq_bio_to_request(rq, bio);
- blk_mq_sched_insert_request(rq, false, true,
- !is_sync || is_flush_fua, true);
- goto done;
- }
- if (!blk_mq_merge_queue_io(data.hctx, data.ctx, rq, bio)) {
- /*
- * For a SYNC request, send it to the hardware immediately. For
- * an ASYNC request, just ensure that we run it later on. The
- * latter allows for merging opportunities and more efficient
- * dispatching.
- */
-run_queue:
- blk_mq_run_hw_queue(data.hctx, !is_sync || is_flush_fua);
- }
+ blk_mq_try_issue_directly(data.hctx, rq, &cookie);
+ } else if (q->elevator) {
+ blk_mq_put_ctx(data.ctx);
+ blk_mq_bio_to_request(rq, bio);
+ blk_mq_sched_insert_request(rq, false, true, true, true);
+ } else if (!blk_mq_merge_queue_io(data.hctx, data.ctx, rq, bio)) {
+ blk_mq_put_ctx(data.ctx);
+ blk_mq_run_hw_queue(data.hctx, true);
+ } else
+ blk_mq_put_ctx(data.ctx);
- blk_mq_put_ctx(data.ctx);
-done:
return cookie;
}
@@ -1988,9 +1895,7 @@
if (node == NUMA_NO_NODE)
node = hctx->numa_node = set->numa_node;
- INIT_WORK(&hctx->run_work, blk_mq_run_work_fn);
- INIT_DELAYED_WORK(&hctx->delayed_run_work, blk_mq_delayed_run_work_fn);
- INIT_DELAYED_WORK(&hctx->delay_work, blk_mq_delay_work_fn);
+ INIT_DELAYED_WORK(&hctx->run_work, blk_mq_run_work_fn);
spin_lock_init(&hctx->lock);
INIT_LIST_HEAD(&hctx->dispatch);
hctx->queue = q;
@@ -2067,8 +1972,6 @@
spin_lock_init(&__ctx->lock);
INIT_LIST_HEAD(&__ctx->rq_list);
__ctx->queue = q;
- blk_stat_init(&__ctx->stat[BLK_STAT_READ]);
- blk_stat_init(&__ctx->stat[BLK_STAT_WRITE]);
/* If the cpu isn't online, the cpu is mapped to first hctx */
if (!cpu_online(i))
@@ -2215,6 +2118,8 @@
{
struct request_queue *q;
+ lockdep_assert_held(&set->tag_list_lock);
+
list_for_each_entry(q, &set->tag_list, tag_set_list) {
blk_mq_freeze_queue(q);
queue_set_hctx_shared(q, shared);
@@ -2227,7 +2132,8 @@
struct blk_mq_tag_set *set = q->tag_set;
mutex_lock(&set->tag_list_lock);
- list_del_init(&q->tag_set_list);
+ list_del_rcu(&q->tag_set_list);
+ INIT_LIST_HEAD(&q->tag_set_list);
if (list_is_singular(&set->tag_list)) {
/* just transitioned to unshared */
set->flags &= ~BLK_MQ_F_TAG_SHARED;
@@ -2235,6 +2141,8 @@
blk_mq_update_tag_set_depth(set, false);
}
mutex_unlock(&set->tag_list_lock);
+
+ synchronize_rcu();
}
static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
@@ -2252,7 +2160,7 @@
}
if (set->flags & BLK_MQ_F_TAG_SHARED)
queue_set_hctx_shared(q, true);
- list_add_tail(&q->tag_set_list, &set->tag_list);
+ list_add_tail_rcu(&q->tag_set_list, &set->tag_list);
mutex_unlock(&set->tag_list_lock);
}
@@ -2364,6 +2272,12 @@
/* mark the queue as mq asap */
q->mq_ops = set->ops;
+ q->poll_cb = blk_stat_alloc_callback(blk_mq_poll_stats_fn,
+ blk_mq_poll_stats_bkt,
+ BLK_MQ_POLL_STATS_BKTS, q);
+ if (!q->poll_cb)
+ goto err_exit;
+
q->queue_ctx = alloc_percpu(struct blk_mq_ctx);
if (!q->queue_ctx)
goto err_exit;
@@ -2398,10 +2312,7 @@
INIT_LIST_HEAD(&q->requeue_list);
spin_lock_init(&q->requeue_lock);
- if (q->nr_hw_queues > 1)
- blk_queue_make_request(q, blk_mq_make_request);
- else
- blk_queue_make_request(q, blk_sq_make_request);
+ blk_queue_make_request(q, blk_mq_make_request);
/*
* Do this after blk_queue_make_request() overrides it...
@@ -2456,8 +2367,6 @@
list_del_init(&q->all_q_node);
mutex_unlock(&all_q_mutex);
- wbt_exit(q);
-
blk_mq_del_queue_tag_set(q);
blk_mq_exit_hw_queues(q, set, set->nr_hw_queues);
@@ -2502,7 +2411,7 @@
* take place in parallel.
*/
list_for_each_entry(q, &all_q_list, all_q_node)
- blk_mq_freeze_queue_start(q);
+ blk_freeze_queue_start(q);
list_for_each_entry(q, &all_q_list, all_q_node)
blk_mq_freeze_queue_wait(q);
@@ -2743,6 +2652,8 @@
{
struct request_queue *q;
+ lockdep_assert_held(&set->tag_list_lock);
+
if (nr_hw_queues > nr_cpu_ids)
nr_hw_queues = nr_cpu_ids;
if (nr_hw_queues < 1 || nr_hw_queues == set->nr_hw_queues)
@@ -2755,16 +2666,6 @@
blk_mq_update_queue_map(set);
list_for_each_entry(q, &set->tag_list, tag_set_list) {
blk_mq_realloc_hw_ctxs(set, q);
-
- /*
- * Manually set the make_request_fn as blk_queue_make_request
- * resets a lot of the queue settings.
- */
- if (q->nr_hw_queues > 1)
- q->make_request_fn = blk_mq_make_request;
- else
- q->make_request_fn = blk_sq_make_request;
-
blk_mq_queue_reinit(q, cpu_online_mask);
}
@@ -2773,39 +2674,69 @@
}
EXPORT_SYMBOL_GPL(blk_mq_update_nr_hw_queues);
+/* Enable polling stats and return whether they were already enabled. */
+static bool blk_poll_stats_enable(struct request_queue *q)
+{
+ if (test_bit(QUEUE_FLAG_POLL_STATS, &q->queue_flags) ||
+ test_and_set_bit(QUEUE_FLAG_POLL_STATS, &q->queue_flags))
+ return true;
+ blk_stat_add_callback(q, q->poll_cb);
+ return false;
+}
+
+static void blk_mq_poll_stats_start(struct request_queue *q)
+{
+ /*
+ * We don't arm the callback if polling stats are not enabled or the
+ * callback is already active.
+ */
+ if (!test_bit(QUEUE_FLAG_POLL_STATS, &q->queue_flags) ||
+ blk_stat_is_active(q->poll_cb))
+ return;
+
+ blk_stat_activate_msecs(q->poll_cb, 100);
+}
+
+static void blk_mq_poll_stats_fn(struct blk_stat_callback *cb)
+{
+ struct request_queue *q = cb->data;
+ int bucket;
+
+ for (bucket = 0; bucket < BLK_MQ_POLL_STATS_BKTS; bucket++) {
+ if (cb->stat[bucket].nr_samples)
+ q->poll_stat[bucket] = cb->stat[bucket];
+ }
+}
+
static unsigned long blk_mq_poll_nsecs(struct request_queue *q,
struct blk_mq_hw_ctx *hctx,
struct request *rq)
{
- struct blk_rq_stat stat[2];
unsigned long ret = 0;
+ int bucket;
/*
* If stats collection isn't on, don't sleep but turn it on for
* future users
*/
- if (!blk_stat_enable(q))
+ if (!blk_poll_stats_enable(q))
return 0;
/*
- * We don't have to do this once per IO, should optimize this
- * to just use the current window of stats until it changes
- */
- memset(&stat, 0, sizeof(stat));
- blk_hctx_stat_get(hctx, stat);
-
- /*
* As an optimistic guess, use half of the mean service time
* for this type of request. We can (and should) make this smarter.
* For instance, if the completion latencies are tight, we can
* get closer than just half the mean. This is especially
* important on devices where the completion latencies are longer
- * than ~10 usec.
+ * than ~10 usec. We do use the stats for the relevant IO size
+ * if available which does lead to better estimates.
*/
- if (req_op(rq) == REQ_OP_READ && stat[BLK_STAT_READ].nr_samples)
- ret = (stat[BLK_STAT_READ].mean + 1) / 2;
- else if (req_op(rq) == REQ_OP_WRITE && stat[BLK_STAT_WRITE].nr_samples)
- ret = (stat[BLK_STAT_WRITE].mean + 1) / 2;
+ bucket = blk_mq_poll_stats_bkt(rq);
+ if (bucket < 0)
+ return ret;
+
+ if (q->poll_stat[bucket].nr_samples)
+ ret = (q->poll_stat[bucket].mean + 1) / 2;
return ret;
}
diff --git a/block/blk-mq.h b/block/blk-mq.h
index 660a17e..2814a14 100644
--- a/block/blk-mq.h
+++ b/block/blk-mq.h
@@ -20,7 +20,6 @@
/* incremented at completion time */
unsigned long ____cacheline_aligned_in_smp rq_completed[2];
- struct blk_rq_stat stat[2];
struct request_queue *queue;
struct kobject kobj;
@@ -79,6 +78,7 @@
*/
extern void blk_mq_sysfs_init(struct request_queue *q);
extern void blk_mq_sysfs_deinit(struct request_queue *q);
+extern int __blk_mq_register_dev(struct device *dev, struct request_queue *q);
extern int blk_mq_sysfs_register(struct request_queue *q);
extern void blk_mq_sysfs_unregister(struct request_queue *q);
extern void blk_mq_hctx_kobj_init(struct blk_mq_hw_ctx *hctx);
@@ -87,13 +87,12 @@
* debugfs helpers
*/
#ifdef CONFIG_BLK_DEBUG_FS
-int blk_mq_debugfs_register(struct request_queue *q, const char *name);
+int blk_mq_debugfs_register(struct request_queue *q);
void blk_mq_debugfs_unregister(struct request_queue *q);
-int blk_mq_debugfs_register_hctxs(struct request_queue *q);
-void blk_mq_debugfs_unregister_hctxs(struct request_queue *q);
+int blk_mq_debugfs_register_mq(struct request_queue *q);
+void blk_mq_debugfs_unregister_mq(struct request_queue *q);
#else
-static inline int blk_mq_debugfs_register(struct request_queue *q,
- const char *name)
+static inline int blk_mq_debugfs_register(struct request_queue *q)
{
return 0;
}
@@ -102,12 +101,12 @@
{
}
-static inline int blk_mq_debugfs_register_hctxs(struct request_queue *q)
+static inline int blk_mq_debugfs_register_mq(struct request_queue *q)
{
return 0;
}
-static inline void blk_mq_debugfs_unregister_hctxs(struct request_queue *q)
+static inline void blk_mq_debugfs_unregister_mq(struct request_queue *q)
{
}
#endif
@@ -142,6 +141,7 @@
/* input parameter */
struct request_queue *q;
unsigned int flags;
+ unsigned int shallow_depth;
/* input & output parameter */
struct blk_mq_ctx *ctx;
diff --git a/block/blk-settings.c b/block/blk-settings.c
index 1e7174f..4fa81ed3 100644
--- a/block/blk-settings.c
+++ b/block/blk-settings.c
@@ -103,7 +103,6 @@
lim->discard_granularity = 0;
lim->discard_alignment = 0;
lim->discard_misaligned = 0;
- lim->discard_zeroes_data = 0;
lim->logical_block_size = lim->physical_block_size = lim->io_min = 512;
lim->bounce_pfn = (unsigned long)(BLK_BOUNCE_ANY >> PAGE_SHIFT);
lim->alignment_offset = 0;
@@ -127,7 +126,6 @@
blk_set_default_limits(lim);
/* Inherit limits from component devices */
- lim->discard_zeroes_data = 1;
lim->max_segments = USHRT_MAX;
lim->max_discard_segments = 1;
lim->max_hw_sectors = UINT_MAX;
@@ -609,7 +607,6 @@
t->io_opt = lcm_not_zero(t->io_opt, b->io_opt);
t->cluster &= b->cluster;
- t->discard_zeroes_data &= b->discard_zeroes_data;
/* Physical block size a multiple of the logical block size? */
if (t->physical_block_size & (t->logical_block_size - 1)) {
diff --git a/block/blk-stat.c b/block/blk-stat.c
index 186fcb9..6c2f409 100644
--- a/block/blk-stat.c
+++ b/block/blk-stat.c
@@ -4,10 +4,27 @@
* Copyright (C) 2016 Jens Axboe
*/
#include <linux/kernel.h>
+#include <linux/rculist.h>
#include <linux/blk-mq.h>
#include "blk-stat.h"
#include "blk-mq.h"
+#include "blk.h"
+
+#define BLK_RQ_STAT_BATCH 64
+
+struct blk_queue_stats {
+ struct list_head callbacks;
+ spinlock_t lock;
+ bool enable_accounting;
+};
+
+static void blk_stat_init(struct blk_rq_stat *stat)
+{
+ stat->min = -1ULL;
+ stat->max = stat->nr_samples = stat->mean = 0;
+ stat->batch = stat->nr_batch = 0;
+}
static void blk_stat_flush_batch(struct blk_rq_stat *stat)
{
@@ -48,166 +65,10 @@
dst->nr_samples += src->nr_samples;
}
-static void blk_mq_stat_get(struct request_queue *q, struct blk_rq_stat *dst)
+static void __blk_stat_add(struct blk_rq_stat *stat, u64 value)
{
- struct blk_mq_hw_ctx *hctx;
- struct blk_mq_ctx *ctx;
- uint64_t latest = 0;
- int i, j, nr;
-
- blk_stat_init(&dst[BLK_STAT_READ]);
- blk_stat_init(&dst[BLK_STAT_WRITE]);
-
- nr = 0;
- do {
- uint64_t newest = 0;
-
- queue_for_each_hw_ctx(q, hctx, i) {
- hctx_for_each_ctx(hctx, ctx, j) {
- blk_stat_flush_batch(&ctx->stat[BLK_STAT_READ]);
- blk_stat_flush_batch(&ctx->stat[BLK_STAT_WRITE]);
-
- if (!ctx->stat[BLK_STAT_READ].nr_samples &&
- !ctx->stat[BLK_STAT_WRITE].nr_samples)
- continue;
- if (ctx->stat[BLK_STAT_READ].time > newest)
- newest = ctx->stat[BLK_STAT_READ].time;
- if (ctx->stat[BLK_STAT_WRITE].time > newest)
- newest = ctx->stat[BLK_STAT_WRITE].time;
- }
- }
-
- /*
- * No samples
- */
- if (!newest)
- break;
-
- if (newest > latest)
- latest = newest;
-
- queue_for_each_hw_ctx(q, hctx, i) {
- hctx_for_each_ctx(hctx, ctx, j) {
- if (ctx->stat[BLK_STAT_READ].time == newest) {
- blk_stat_sum(&dst[BLK_STAT_READ],
- &ctx->stat[BLK_STAT_READ]);
- nr++;
- }
- if (ctx->stat[BLK_STAT_WRITE].time == newest) {
- blk_stat_sum(&dst[BLK_STAT_WRITE],
- &ctx->stat[BLK_STAT_WRITE]);
- nr++;
- }
- }
- }
- /*
- * If we race on finding an entry, just loop back again.
- * Should be very rare.
- */
- } while (!nr);
-
- dst[BLK_STAT_READ].time = dst[BLK_STAT_WRITE].time = latest;
-}
-
-void blk_queue_stat_get(struct request_queue *q, struct blk_rq_stat *dst)
-{
- if (q->mq_ops)
- blk_mq_stat_get(q, dst);
- else {
- blk_stat_flush_batch(&q->rq_stats[BLK_STAT_READ]);
- blk_stat_flush_batch(&q->rq_stats[BLK_STAT_WRITE]);
- memcpy(&dst[BLK_STAT_READ], &q->rq_stats[BLK_STAT_READ],
- sizeof(struct blk_rq_stat));
- memcpy(&dst[BLK_STAT_WRITE], &q->rq_stats[BLK_STAT_WRITE],
- sizeof(struct blk_rq_stat));
- }
-}
-
-void blk_hctx_stat_get(struct blk_mq_hw_ctx *hctx, struct blk_rq_stat *dst)
-{
- struct blk_mq_ctx *ctx;
- unsigned int i, nr;
-
- nr = 0;
- do {
- uint64_t newest = 0;
-
- hctx_for_each_ctx(hctx, ctx, i) {
- blk_stat_flush_batch(&ctx->stat[BLK_STAT_READ]);
- blk_stat_flush_batch(&ctx->stat[BLK_STAT_WRITE]);
-
- if (!ctx->stat[BLK_STAT_READ].nr_samples &&
- !ctx->stat[BLK_STAT_WRITE].nr_samples)
- continue;
-
- if (ctx->stat[BLK_STAT_READ].time > newest)
- newest = ctx->stat[BLK_STAT_READ].time;
- if (ctx->stat[BLK_STAT_WRITE].time > newest)
- newest = ctx->stat[BLK_STAT_WRITE].time;
- }
-
- if (!newest)
- break;
-
- hctx_for_each_ctx(hctx, ctx, i) {
- if (ctx->stat[BLK_STAT_READ].time == newest) {
- blk_stat_sum(&dst[BLK_STAT_READ],
- &ctx->stat[BLK_STAT_READ]);
- nr++;
- }
- if (ctx->stat[BLK_STAT_WRITE].time == newest) {
- blk_stat_sum(&dst[BLK_STAT_WRITE],
- &ctx->stat[BLK_STAT_WRITE]);
- nr++;
- }
- }
- /*
- * If we race on finding an entry, just loop back again.
- * Should be very rare, as the window is only updated
- * occasionally
- */
- } while (!nr);
-}
-
-static void __blk_stat_init(struct blk_rq_stat *stat, s64 time_now)
-{
- stat->min = -1ULL;
- stat->max = stat->nr_samples = stat->mean = 0;
- stat->batch = stat->nr_batch = 0;
- stat->time = time_now & BLK_STAT_NSEC_MASK;
-}
-
-void blk_stat_init(struct blk_rq_stat *stat)
-{
- __blk_stat_init(stat, ktime_to_ns(ktime_get()));
-}
-
-static bool __blk_stat_is_current(struct blk_rq_stat *stat, s64 now)
-{
- return (now & BLK_STAT_NSEC_MASK) == (stat->time & BLK_STAT_NSEC_MASK);
-}
-
-bool blk_stat_is_current(struct blk_rq_stat *stat)
-{
- return __blk_stat_is_current(stat, ktime_to_ns(ktime_get()));
-}
-
-void blk_stat_add(struct blk_rq_stat *stat, struct request *rq)
-{
- s64 now, value;
-
- now = __blk_stat_time(ktime_to_ns(ktime_get()));
- if (now < blk_stat_time(&rq->issue_stat))
- return;
-
- if (!__blk_stat_is_current(stat, now))
- __blk_stat_init(stat, now);
-
- value = now - blk_stat_time(&rq->issue_stat);
- if (value > stat->max)
- stat->max = value;
- if (value < stat->min)
- stat->min = value;
+ stat->min = min(stat->min, value);
+ stat->max = max(stat->max, value);
if (stat->batch + value < stat->batch ||
stat->nr_batch + 1 == BLK_RQ_STAT_BATCH)
@@ -217,40 +78,172 @@
stat->nr_batch++;
}
-void blk_stat_clear(struct request_queue *q)
+void blk_stat_add(struct request *rq)
{
- if (q->mq_ops) {
- struct blk_mq_hw_ctx *hctx;
- struct blk_mq_ctx *ctx;
- int i, j;
+ struct request_queue *q = rq->q;
+ struct blk_stat_callback *cb;
+ struct blk_rq_stat *stat;
+ int bucket;
+ s64 now, value;
- queue_for_each_hw_ctx(q, hctx, i) {
- hctx_for_each_ctx(hctx, ctx, j) {
- blk_stat_init(&ctx->stat[BLK_STAT_READ]);
- blk_stat_init(&ctx->stat[BLK_STAT_WRITE]);
- }
+ now = __blk_stat_time(ktime_to_ns(ktime_get()));
+ if (now < blk_stat_time(&rq->issue_stat))
+ return;
+
+ value = now - blk_stat_time(&rq->issue_stat);
+
+ blk_throtl_stat_add(rq, value);
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(cb, &q->stats->callbacks, list) {
+ if (blk_stat_is_active(cb)) {
+ bucket = cb->bucket_fn(rq);
+ if (bucket < 0)
+ continue;
+ stat = &this_cpu_ptr(cb->cpu_stat)[bucket];
+ __blk_stat_add(stat, value);
}
- } else {
- blk_stat_init(&q->rq_stats[BLK_STAT_READ]);
- blk_stat_init(&q->rq_stats[BLK_STAT_WRITE]);
}
+ rcu_read_unlock();
}
-void blk_stat_set_issue_time(struct blk_issue_stat *stat)
+static void blk_stat_timer_fn(unsigned long data)
{
- stat->time = (stat->time & BLK_STAT_MASK) |
- (ktime_to_ns(ktime_get()) & BLK_STAT_TIME_MASK);
-}
+ struct blk_stat_callback *cb = (void *)data;
+ unsigned int bucket;
+ int cpu;
-/*
- * Enable stat tracking, return whether it was enabled
- */
-bool blk_stat_enable(struct request_queue *q)
-{
- if (!test_bit(QUEUE_FLAG_STATS, &q->queue_flags)) {
- set_bit(QUEUE_FLAG_STATS, &q->queue_flags);
- return false;
+ for (bucket = 0; bucket < cb->buckets; bucket++)
+ blk_stat_init(&cb->stat[bucket]);
+
+ for_each_online_cpu(cpu) {
+ struct blk_rq_stat *cpu_stat;
+
+ cpu_stat = per_cpu_ptr(cb->cpu_stat, cpu);
+ for (bucket = 0; bucket < cb->buckets; bucket++) {
+ blk_stat_sum(&cb->stat[bucket], &cpu_stat[bucket]);
+ blk_stat_init(&cpu_stat[bucket]);
+ }
}
- return true;
+ cb->timer_fn(cb);
+}
+
+struct blk_stat_callback *
+blk_stat_alloc_callback(void (*timer_fn)(struct blk_stat_callback *),
+ int (*bucket_fn)(const struct request *),
+ unsigned int buckets, void *data)
+{
+ struct blk_stat_callback *cb;
+
+ cb = kmalloc(sizeof(*cb), GFP_KERNEL);
+ if (!cb)
+ return NULL;
+
+ cb->stat = kmalloc_array(buckets, sizeof(struct blk_rq_stat),
+ GFP_KERNEL);
+ if (!cb->stat) {
+ kfree(cb);
+ return NULL;
+ }
+ cb->cpu_stat = __alloc_percpu(buckets * sizeof(struct blk_rq_stat),
+ __alignof__(struct blk_rq_stat));
+ if (!cb->cpu_stat) {
+ kfree(cb->stat);
+ kfree(cb);
+ return NULL;
+ }
+
+ cb->timer_fn = timer_fn;
+ cb->bucket_fn = bucket_fn;
+ cb->data = data;
+ cb->buckets = buckets;
+ setup_timer(&cb->timer, blk_stat_timer_fn, (unsigned long)cb);
+
+ return cb;
+}
+EXPORT_SYMBOL_GPL(blk_stat_alloc_callback);
+
+void blk_stat_add_callback(struct request_queue *q,
+ struct blk_stat_callback *cb)
+{
+ unsigned int bucket;
+ int cpu;
+
+ for_each_possible_cpu(cpu) {
+ struct blk_rq_stat *cpu_stat;
+
+ cpu_stat = per_cpu_ptr(cb->cpu_stat, cpu);
+ for (bucket = 0; bucket < cb->buckets; bucket++)
+ blk_stat_init(&cpu_stat[bucket]);
+ }
+
+ spin_lock(&q->stats->lock);
+ list_add_tail_rcu(&cb->list, &q->stats->callbacks);
+ set_bit(QUEUE_FLAG_STATS, &q->queue_flags);
+ spin_unlock(&q->stats->lock);
+}
+EXPORT_SYMBOL_GPL(blk_stat_add_callback);
+
+void blk_stat_remove_callback(struct request_queue *q,
+ struct blk_stat_callback *cb)
+{
+ spin_lock(&q->stats->lock);
+ list_del_rcu(&cb->list);
+ if (list_empty(&q->stats->callbacks) && !q->stats->enable_accounting)
+ clear_bit(QUEUE_FLAG_STATS, &q->queue_flags);
+ spin_unlock(&q->stats->lock);
+
+ del_timer_sync(&cb->timer);
+}
+EXPORT_SYMBOL_GPL(blk_stat_remove_callback);
+
+static void blk_stat_free_callback_rcu(struct rcu_head *head)
+{
+ struct blk_stat_callback *cb;
+
+ cb = container_of(head, struct blk_stat_callback, rcu);
+ free_percpu(cb->cpu_stat);
+ kfree(cb->stat);
+ kfree(cb);
+}
+
+void blk_stat_free_callback(struct blk_stat_callback *cb)
+{
+ if (cb)
+ call_rcu(&cb->rcu, blk_stat_free_callback_rcu);
+}
+EXPORT_SYMBOL_GPL(blk_stat_free_callback);
+
+void blk_stat_enable_accounting(struct request_queue *q)
+{
+ spin_lock(&q->stats->lock);
+ q->stats->enable_accounting = true;
+ set_bit(QUEUE_FLAG_STATS, &q->queue_flags);
+ spin_unlock(&q->stats->lock);
+}
+
+struct blk_queue_stats *blk_alloc_queue_stats(void)
+{
+ struct blk_queue_stats *stats;
+
+ stats = kmalloc(sizeof(*stats), GFP_KERNEL);
+ if (!stats)
+ return NULL;
+
+ INIT_LIST_HEAD(&stats->callbacks);
+ spin_lock_init(&stats->lock);
+ stats->enable_accounting = false;
+
+ return stats;
+}
+
+void blk_free_queue_stats(struct blk_queue_stats *stats)
+{
+ if (!stats)
+ return;
+
+ WARN_ON(!list_empty(&stats->callbacks));
+
+ kfree(stats);
}
diff --git a/block/blk-stat.h b/block/blk-stat.h
index a2050a0..2fb20d1 100644
--- a/block/blk-stat.h
+++ b/block/blk-stat.h
@@ -1,33 +1,85 @@
#ifndef BLK_STAT_H
#define BLK_STAT_H
-/*
- * ~0.13s window as a power-of-2 (2^27 nsecs)
- */
-#define BLK_STAT_NSEC 134217728ULL
-#define BLK_STAT_NSEC_MASK ~(BLK_STAT_NSEC - 1)
+#include <linux/kernel.h>
+#include <linux/blkdev.h>
+#include <linux/ktime.h>
+#include <linux/rcupdate.h>
+#include <linux/timer.h>
/*
- * Upper 3 bits can be used elsewhere
+ * from upper:
+ * 3 bits: reserved for other usage
+ * 12 bits: size
+ * 49 bits: time
*/
#define BLK_STAT_RES_BITS 3
-#define BLK_STAT_SHIFT (64 - BLK_STAT_RES_BITS)
-#define BLK_STAT_TIME_MASK ((1ULL << BLK_STAT_SHIFT) - 1)
-#define BLK_STAT_MASK ~BLK_STAT_TIME_MASK
+#define BLK_STAT_SIZE_BITS 12
+#define BLK_STAT_RES_SHIFT (64 - BLK_STAT_RES_BITS)
+#define BLK_STAT_SIZE_SHIFT (BLK_STAT_RES_SHIFT - BLK_STAT_SIZE_BITS)
+#define BLK_STAT_TIME_MASK ((1ULL << BLK_STAT_SIZE_SHIFT) - 1)
+#define BLK_STAT_SIZE_MASK \
+ (((1ULL << BLK_STAT_SIZE_BITS) - 1) << BLK_STAT_SIZE_SHIFT)
+#define BLK_STAT_RES_MASK (~((1ULL << BLK_STAT_RES_SHIFT) - 1))
-enum {
- BLK_STAT_READ = 0,
- BLK_STAT_WRITE,
+/**
+ * struct blk_stat_callback - Block statistics callback.
+ *
+ * A &struct blk_stat_callback is associated with a &struct request_queue. While
+ * @timer is active, that queue's request completion latencies are sorted into
+ * buckets by @bucket_fn and added to a per-cpu buffer, @cpu_stat. When the
+ * timer fires, @cpu_stat is flushed to @stat and @timer_fn is invoked.
+ */
+struct blk_stat_callback {
+ /*
+ * @list: RCU list of callbacks for a &struct request_queue.
+ */
+ struct list_head list;
+
+ /**
+ * @timer: Timer for the next callback invocation.
+ */
+ struct timer_list timer;
+
+ /**
+ * @cpu_stat: Per-cpu statistics buckets.
+ */
+ struct blk_rq_stat __percpu *cpu_stat;
+
+ /**
+ * @bucket_fn: Given a request, returns which statistics bucket it
+ * should be accounted under. Return -1 for no bucket for this
+ * request.
+ */
+ int (*bucket_fn)(const struct request *);
+
+ /**
+ * @buckets: Number of statistics buckets.
+ */
+ unsigned int buckets;
+
+ /**
+ * @stat: Array of statistics buckets.
+ */
+ struct blk_rq_stat *stat;
+
+ /**
+ * @fn: Callback function.
+ */
+ void (*timer_fn)(struct blk_stat_callback *);
+
+ /**
+ * @data: Private pointer for the user.
+ */
+ void *data;
+
+ struct rcu_head rcu;
};
-void blk_stat_add(struct blk_rq_stat *, struct request *);
-void blk_hctx_stat_get(struct blk_mq_hw_ctx *, struct blk_rq_stat *);
-void blk_queue_stat_get(struct request_queue *, struct blk_rq_stat *);
-void blk_stat_clear(struct request_queue *);
-void blk_stat_init(struct blk_rq_stat *);
-bool blk_stat_is_current(struct blk_rq_stat *);
-void blk_stat_set_issue_time(struct blk_issue_stat *);
-bool blk_stat_enable(struct request_queue *);
+struct blk_queue_stats *blk_alloc_queue_stats(void);
+void blk_free_queue_stats(struct blk_queue_stats *);
+
+void blk_stat_add(struct request *);
static inline u64 __blk_stat_time(u64 time)
{
@@ -36,7 +88,117 @@
static inline u64 blk_stat_time(struct blk_issue_stat *stat)
{
- return __blk_stat_time(stat->time);
+ return __blk_stat_time(stat->stat);
+}
+
+static inline sector_t blk_capped_size(sector_t size)
+{
+ return size & ((1ULL << BLK_STAT_SIZE_BITS) - 1);
+}
+
+static inline sector_t blk_stat_size(struct blk_issue_stat *stat)
+{
+ return (stat->stat & BLK_STAT_SIZE_MASK) >> BLK_STAT_SIZE_SHIFT;
+}
+
+static inline void blk_stat_set_issue(struct blk_issue_stat *stat,
+ sector_t size)
+{
+ stat->stat = (stat->stat & BLK_STAT_RES_MASK) |
+ (ktime_to_ns(ktime_get()) & BLK_STAT_TIME_MASK) |
+ (((u64)blk_capped_size(size)) << BLK_STAT_SIZE_SHIFT);
+}
+
+/* record time/size info in request but not add a callback */
+void blk_stat_enable_accounting(struct request_queue *q);
+
+/**
+ * blk_stat_alloc_callback() - Allocate a block statistics callback.
+ * @timer_fn: Timer callback function.
+ * @bucket_fn: Bucket callback function.
+ * @buckets: Number of statistics buckets.
+ * @data: Value for the @data field of the &struct blk_stat_callback.
+ *
+ * See &struct blk_stat_callback for details on the callback functions.
+ *
+ * Return: &struct blk_stat_callback on success or NULL on ENOMEM.
+ */
+struct blk_stat_callback *
+blk_stat_alloc_callback(void (*timer_fn)(struct blk_stat_callback *),
+ int (*bucket_fn)(const struct request *),
+ unsigned int buckets, void *data);
+
+/**
+ * blk_stat_add_callback() - Add a block statistics callback to be run on a
+ * request queue.
+ * @q: The request queue.
+ * @cb: The callback.
+ *
+ * Note that a single &struct blk_stat_callback can only be added to a single
+ * &struct request_queue.
+ */
+void blk_stat_add_callback(struct request_queue *q,
+ struct blk_stat_callback *cb);
+
+/**
+ * blk_stat_remove_callback() - Remove a block statistics callback from a
+ * request queue.
+ * @q: The request queue.
+ * @cb: The callback.
+ *
+ * When this returns, the callback is not running on any CPUs and will not be
+ * called again unless readded.
+ */
+void blk_stat_remove_callback(struct request_queue *q,
+ struct blk_stat_callback *cb);
+
+/**
+ * blk_stat_free_callback() - Free a block statistics callback.
+ * @cb: The callback.
+ *
+ * @cb may be NULL, in which case this does nothing. If it is not NULL, @cb must
+ * not be associated with a request queue. I.e., if it was previously added with
+ * blk_stat_add_callback(), it must also have been removed since then with
+ * blk_stat_remove_callback().
+ */
+void blk_stat_free_callback(struct blk_stat_callback *cb);
+
+/**
+ * blk_stat_is_active() - Check if a block statistics callback is currently
+ * gathering statistics.
+ * @cb: The callback.
+ */
+static inline bool blk_stat_is_active(struct blk_stat_callback *cb)
+{
+ return timer_pending(&cb->timer);
+}
+
+/**
+ * blk_stat_activate_nsecs() - Gather block statistics during a time window in
+ * nanoseconds.
+ * @cb: The callback.
+ * @nsecs: Number of nanoseconds to gather statistics for.
+ *
+ * The timer callback will be called when the window expires.
+ */
+static inline void blk_stat_activate_nsecs(struct blk_stat_callback *cb,
+ u64 nsecs)
+{
+ mod_timer(&cb->timer, jiffies + nsecs_to_jiffies(nsecs));
+}
+
+/**
+ * blk_stat_activate_msecs() - Gather block statistics during a time window in
+ * milliseconds.
+ * @cb: The callback.
+ * @msecs: Number of milliseconds to gather statistics for.
+ *
+ * The timer callback will be called when the window expires.
+ */
+static inline void blk_stat_activate_msecs(struct blk_stat_callback *cb,
+ unsigned int msecs)
+{
+ mod_timer(&cb->timer, jiffies + msecs_to_jiffies(msecs));
}
#endif
diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index 37f0b3a..3f37813 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -208,7 +208,7 @@
static ssize_t queue_discard_zeroes_data_show(struct request_queue *q, char *page)
{
- return queue_var_show(queue_discard_zeroes_data(q), page);
+ return queue_var_show(0, page);
}
static ssize_t queue_write_same_max_show(struct request_queue *q, char *page)
@@ -503,26 +503,6 @@
return queue_var_show(blk_queue_dax(q), page);
}
-static ssize_t print_stat(char *page, struct blk_rq_stat *stat, const char *pre)
-{
- return sprintf(page, "%s samples=%llu, mean=%lld, min=%lld, max=%lld\n",
- pre, (long long) stat->nr_samples,
- (long long) stat->mean, (long long) stat->min,
- (long long) stat->max);
-}
-
-static ssize_t queue_stats_show(struct request_queue *q, char *page)
-{
- struct blk_rq_stat stat[2];
- ssize_t ret;
-
- blk_queue_stat_get(q, stat);
-
- ret = print_stat(page, &stat[BLK_STAT_READ], "read :");
- ret += print_stat(page + ret, &stat[BLK_STAT_WRITE], "write:");
- return ret;
-}
-
static struct queue_sysfs_entry queue_requests_entry = {
.attr = {.name = "nr_requests", .mode = S_IRUGO | S_IWUSR },
.show = queue_requests_show,
@@ -691,17 +671,20 @@
.show = queue_dax_show,
};
-static struct queue_sysfs_entry queue_stats_entry = {
- .attr = {.name = "stats", .mode = S_IRUGO },
- .show = queue_stats_show,
-};
-
static struct queue_sysfs_entry queue_wb_lat_entry = {
.attr = {.name = "wbt_lat_usec", .mode = S_IRUGO | S_IWUSR },
.show = queue_wb_lat_show,
.store = queue_wb_lat_store,
};
+#ifdef CONFIG_BLK_DEV_THROTTLING_LOW
+static struct queue_sysfs_entry throtl_sample_time_entry = {
+ .attr = {.name = "throttle_sample_time", .mode = S_IRUGO | S_IWUSR },
+ .show = blk_throtl_sample_time_show,
+ .store = blk_throtl_sample_time_store,
+};
+#endif
+
static struct attribute *default_attrs[] = {
&queue_requests_entry.attr,
&queue_ra_entry.attr,
@@ -733,9 +716,11 @@
&queue_poll_entry.attr,
&queue_wc_entry.attr,
&queue_dax_entry.attr,
- &queue_stats_entry.attr,
&queue_wb_lat_entry.attr,
&queue_poll_delay_entry.attr,
+#ifdef CONFIG_BLK_DEV_THROTTLING_LOW
+ &throtl_sample_time_entry.attr,
+#endif
NULL,
};
@@ -810,7 +795,9 @@
struct request_queue *q =
container_of(kobj, struct request_queue, kobj);
- wbt_exit(q);
+ if (test_bit(QUEUE_FLAG_POLL_STATS, &q->queue_flags))
+ blk_stat_remove_callback(q, q->poll_cb);
+ blk_stat_free_callback(q->poll_cb);
bdi_put(q->backing_dev_info);
blkcg_exit_queue(q);
@@ -819,6 +806,8 @@
elevator_exit(q, q->elevator);
}
+ blk_free_queue_stats(q->stats);
+
blk_exit_rl(&q->root_rl);
if (q->queue_tags)
@@ -855,23 +844,6 @@
.release = blk_release_queue,
};
-static void blk_wb_init(struct request_queue *q)
-{
-#ifndef CONFIG_BLK_WBT_MQ
- if (q->mq_ops)
- return;
-#endif
-#ifndef CONFIG_BLK_WBT_SQ
- if (q->request_fn)
- return;
-#endif
-
- /*
- * If this fails, we don't get throttling
- */
- wbt_init(q);
-}
-
int blk_register_queue(struct gendisk *disk)
{
int ret;
@@ -881,6 +853,11 @@
if (WARN_ON(!q))
return -ENXIO;
+ WARN_ONCE(test_bit(QUEUE_FLAG_REGISTERED, &q->queue_flags),
+ "%s is registering an already registered queue\n",
+ kobject_name(&dev->kobj));
+ queue_flag_set_unlocked(QUEUE_FLAG_REGISTERED, q);
+
/*
* SCSI probing may synchronously create and destroy a lot of
* request_queues for non-existent devices. Shutting down a fully
@@ -900,9 +877,6 @@
if (ret)
return ret;
- if (q->mq_ops)
- blk_mq_register_dev(dev, q);
-
/* Prevent changes through sysfs until registration is completed. */
mutex_lock(&q->sysfs_lock);
@@ -912,9 +886,14 @@
goto unlock;
}
+ if (q->mq_ops)
+ __blk_mq_register_dev(dev, q);
+
kobject_uevent(&q->kobj, KOBJ_ADD);
- blk_wb_init(q);
+ wbt_enable_default(q);
+
+ blk_throtl_register_queue(q);
if (q->request_fn || (q->mq_ops && q->elevator)) {
ret = elv_register_queue(q);
@@ -939,6 +918,11 @@
if (WARN_ON(!q))
return;
+ queue_flag_clear_unlocked(QUEUE_FLAG_REGISTERED, q);
+
+ wbt_exit(q);
+
+
if (q->mq_ops)
blk_mq_unregister_dev(disk_to_dev(disk), q);
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 8fab716..b78db2e 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -18,8 +18,17 @@
/* Total max dispatch from all groups in one round */
static int throtl_quantum = 32;
-/* Throttling is performed over 100ms slice and after that slice is renewed */
-static unsigned long throtl_slice = HZ/10; /* 100 ms */
+/* Throttling is performed over a slice and after that slice is renewed */
+#define DFL_THROTL_SLICE_HD (HZ / 10)
+#define DFL_THROTL_SLICE_SSD (HZ / 50)
+#define MAX_THROTL_SLICE (HZ)
+#define DFL_IDLE_THRESHOLD_SSD (1000L) /* 1 ms */
+#define DFL_IDLE_THRESHOLD_HD (100L * 1000) /* 100 ms */
+#define MAX_IDLE_TIME (5L * 1000 * 1000) /* 5 s */
+/* default latency target is 0, eg, guarantee IO latency by default */
+#define DFL_LATENCY_TARGET (0)
+
+#define SKIP_LATENCY (((u64)1) << BLK_STAT_RES_SHIFT)
static struct blkcg_policy blkcg_policy_throtl;
@@ -83,6 +92,12 @@
#define rb_entry_tg(node) rb_entry((node), struct throtl_grp, rb_node)
+enum {
+ LIMIT_LOW,
+ LIMIT_MAX,
+ LIMIT_CNT,
+};
+
struct throtl_grp {
/* must be the first member */
struct blkg_policy_data pd;
@@ -119,20 +134,54 @@
/* are there any throtl rules between this group and td? */
bool has_rules[2];
- /* bytes per second rate limits */
- uint64_t bps[2];
+ /* internally used bytes per second rate limits */
+ uint64_t bps[2][LIMIT_CNT];
+ /* user configured bps limits */
+ uint64_t bps_conf[2][LIMIT_CNT];
- /* IOPS limits */
- unsigned int iops[2];
+ /* internally used IOPS limits */
+ unsigned int iops[2][LIMIT_CNT];
+ /* user configured IOPS limits */
+ unsigned int iops_conf[2][LIMIT_CNT];
/* Number of bytes disptached in current slice */
uint64_t bytes_disp[2];
/* Number of bio's dispatched in current slice */
unsigned int io_disp[2];
+ unsigned long last_low_overflow_time[2];
+
+ uint64_t last_bytes_disp[2];
+ unsigned int last_io_disp[2];
+
+ unsigned long last_check_time;
+
+ unsigned long latency_target; /* us */
/* When did we start a new slice */
unsigned long slice_start[2];
unsigned long slice_end[2];
+
+ unsigned long last_finish_time; /* ns / 1024 */
+ unsigned long checked_last_finish_time; /* ns / 1024 */
+ unsigned long avg_idletime; /* ns / 1024 */
+ unsigned long idletime_threshold; /* us */
+
+ unsigned int bio_cnt; /* total bios */
+ unsigned int bad_bio_cnt; /* bios exceeding latency threshold */
+ unsigned long bio_cnt_reset_time;
+};
+
+/* We measure latency for request size from <= 4k to >= 1M */
+#define LATENCY_BUCKET_SIZE 9
+
+struct latency_bucket {
+ unsigned long total_latency; /* ns / 1024 */
+ int samples;
+};
+
+struct avg_latency_bucket {
+ unsigned long latency; /* ns / 1024 */
+ bool valid;
};
struct throtl_data
@@ -145,8 +194,26 @@
/* Total Number of queued bios on READ and WRITE lists */
unsigned int nr_queued[2];
+ unsigned int throtl_slice;
+
/* Work for dispatching throttled bios */
struct work_struct dispatch_work;
+ unsigned int limit_index;
+ bool limit_valid[LIMIT_CNT];
+
+ unsigned long dft_idletime_threshold; /* us */
+
+ unsigned long low_upgrade_time;
+ unsigned long low_downgrade_time;
+
+ unsigned int scale;
+
+ struct latency_bucket tmp_buckets[LATENCY_BUCKET_SIZE];
+ struct avg_latency_bucket avg_buckets[LATENCY_BUCKET_SIZE];
+ struct latency_bucket __percpu *latency_buckets;
+ unsigned long last_calculate_time;
+
+ bool track_bio_latency;
};
static void throtl_pending_timer_fn(unsigned long arg);
@@ -198,6 +265,76 @@
return container_of(sq, struct throtl_data, service_queue);
}
+/*
+ * cgroup's limit in LIMIT_MAX is scaled if low limit is set. This scale is to
+ * make the IO dispatch more smooth.
+ * Scale up: linearly scale up according to lapsed time since upgrade. For
+ * every throtl_slice, the limit scales up 1/2 .low limit till the
+ * limit hits .max limit
+ * Scale down: exponentially scale down if a cgroup doesn't hit its .low limit
+ */
+static uint64_t throtl_adjusted_limit(uint64_t low, struct throtl_data *td)
+{
+ /* arbitrary value to avoid too big scale */
+ if (td->scale < 4096 && time_after_eq(jiffies,
+ td->low_upgrade_time + td->scale * td->throtl_slice))
+ td->scale = (jiffies - td->low_upgrade_time) / td->throtl_slice;
+
+ return low + (low >> 1) * td->scale;
+}
+
+static uint64_t tg_bps_limit(struct throtl_grp *tg, int rw)
+{
+ struct blkcg_gq *blkg = tg_to_blkg(tg);
+ struct throtl_data *td;
+ uint64_t ret;
+
+ if (cgroup_subsys_on_dfl(io_cgrp_subsys) && !blkg->parent)
+ return U64_MAX;
+
+ td = tg->td;
+ ret = tg->bps[rw][td->limit_index];
+ if (ret == 0 && td->limit_index == LIMIT_LOW)
+ return tg->bps[rw][LIMIT_MAX];
+
+ if (td->limit_index == LIMIT_MAX && tg->bps[rw][LIMIT_LOW] &&
+ tg->bps[rw][LIMIT_LOW] != tg->bps[rw][LIMIT_MAX]) {
+ uint64_t adjusted;
+
+ adjusted = throtl_adjusted_limit(tg->bps[rw][LIMIT_LOW], td);
+ ret = min(tg->bps[rw][LIMIT_MAX], adjusted);
+ }
+ return ret;
+}
+
+static unsigned int tg_iops_limit(struct throtl_grp *tg, int rw)
+{
+ struct blkcg_gq *blkg = tg_to_blkg(tg);
+ struct throtl_data *td;
+ unsigned int ret;
+
+ if (cgroup_subsys_on_dfl(io_cgrp_subsys) && !blkg->parent)
+ return UINT_MAX;
+ td = tg->td;
+ ret = tg->iops[rw][td->limit_index];
+ if (ret == 0 && tg->td->limit_index == LIMIT_LOW)
+ return tg->iops[rw][LIMIT_MAX];
+
+ if (td->limit_index == LIMIT_MAX && tg->iops[rw][LIMIT_LOW] &&
+ tg->iops[rw][LIMIT_LOW] != tg->iops[rw][LIMIT_MAX]) {
+ uint64_t adjusted;
+
+ adjusted = throtl_adjusted_limit(tg->iops[rw][LIMIT_LOW], td);
+ if (adjusted > UINT_MAX)
+ adjusted = UINT_MAX;
+ ret = min_t(unsigned int, tg->iops[rw][LIMIT_MAX], adjusted);
+ }
+ return ret;
+}
+
+#define request_bucket_index(sectors) \
+ clamp_t(int, order_base_2(sectors) - 3, 0, LATENCY_BUCKET_SIZE - 1)
+
/**
* throtl_log - log debug message via blktrace
* @sq: the service_queue being reported
@@ -334,10 +471,17 @@
}
RB_CLEAR_NODE(&tg->rb_node);
- tg->bps[READ] = -1;
- tg->bps[WRITE] = -1;
- tg->iops[READ] = -1;
- tg->iops[WRITE] = -1;
+ tg->bps[READ][LIMIT_MAX] = U64_MAX;
+ tg->bps[WRITE][LIMIT_MAX] = U64_MAX;
+ tg->iops[READ][LIMIT_MAX] = UINT_MAX;
+ tg->iops[WRITE][LIMIT_MAX] = UINT_MAX;
+ tg->bps_conf[READ][LIMIT_MAX] = U64_MAX;
+ tg->bps_conf[WRITE][LIMIT_MAX] = U64_MAX;
+ tg->iops_conf[READ][LIMIT_MAX] = UINT_MAX;
+ tg->iops_conf[WRITE][LIMIT_MAX] = UINT_MAX;
+ /* LIMIT_LOW will have default value 0 */
+
+ tg->latency_target = DFL_LATENCY_TARGET;
return &tg->pd;
}
@@ -366,6 +510,8 @@
if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
sq->parent_sq = &blkg_to_tg(blkg->parent)->service_queue;
tg->td = td;
+
+ tg->idletime_threshold = td->dft_idletime_threshold;
}
/*
@@ -376,20 +522,59 @@
static void tg_update_has_rules(struct throtl_grp *tg)
{
struct throtl_grp *parent_tg = sq_to_tg(tg->service_queue.parent_sq);
+ struct throtl_data *td = tg->td;
int rw;
for (rw = READ; rw <= WRITE; rw++)
tg->has_rules[rw] = (parent_tg && parent_tg->has_rules[rw]) ||
- (tg->bps[rw] != -1 || tg->iops[rw] != -1);
+ (td->limit_valid[td->limit_index] &&
+ (tg_bps_limit(tg, rw) != U64_MAX ||
+ tg_iops_limit(tg, rw) != UINT_MAX));
}
static void throtl_pd_online(struct blkg_policy_data *pd)
{
+ struct throtl_grp *tg = pd_to_tg(pd);
/*
* We don't want new groups to escape the limits of its ancestors.
* Update has_rules[] after a new group is brought online.
*/
- tg_update_has_rules(pd_to_tg(pd));
+ tg_update_has_rules(tg);
+}
+
+static void blk_throtl_update_limit_valid(struct throtl_data *td)
+{
+ struct cgroup_subsys_state *pos_css;
+ struct blkcg_gq *blkg;
+ bool low_valid = false;
+
+ rcu_read_lock();
+ blkg_for_each_descendant_post(blkg, pos_css, td->queue->root_blkg) {
+ struct throtl_grp *tg = blkg_to_tg(blkg);
+
+ if (tg->bps[READ][LIMIT_LOW] || tg->bps[WRITE][LIMIT_LOW] ||
+ tg->iops[READ][LIMIT_LOW] || tg->iops[WRITE][LIMIT_LOW])
+ low_valid = true;
+ }
+ rcu_read_unlock();
+
+ td->limit_valid[LIMIT_LOW] = low_valid;
+}
+
+static void throtl_upgrade_state(struct throtl_data *td);
+static void throtl_pd_offline(struct blkg_policy_data *pd)
+{
+ struct throtl_grp *tg = pd_to_tg(pd);
+
+ tg->bps[READ][LIMIT_LOW] = 0;
+ tg->bps[WRITE][LIMIT_LOW] = 0;
+ tg->iops[READ][LIMIT_LOW] = 0;
+ tg->iops[WRITE][LIMIT_LOW] = 0;
+
+ blk_throtl_update_limit_valid(tg->td);
+
+ if (!tg->td->limit_valid[tg->td->limit_index])
+ throtl_upgrade_state(tg->td);
}
static void throtl_pd_free(struct blkg_policy_data *pd)
@@ -499,6 +684,17 @@
static void throtl_schedule_pending_timer(struct throtl_service_queue *sq,
unsigned long expires)
{
+ unsigned long max_expire = jiffies + 8 * sq_to_tg(sq)->td->throtl_slice;
+
+ /*
+ * Since we are adjusting the throttle limit dynamically, the sleep
+ * time calculated according to previous limit might be invalid. It's
+ * possible the cgroup sleep time is very long and no other cgroups
+ * have IO running so notify the limit changes. Make sure the cgroup
+ * doesn't sleep too long to avoid the missed notification.
+ */
+ if (time_after(expires, max_expire))
+ expires = max_expire;
mod_timer(&sq->pending_timer, expires);
throtl_log(sq, "schedule timer. delay=%lu jiffies=%lu",
expires - jiffies, jiffies);
@@ -556,7 +752,7 @@
if (time_after_eq(start, tg->slice_start[rw]))
tg->slice_start[rw] = start;
- tg->slice_end[rw] = jiffies + throtl_slice;
+ tg->slice_end[rw] = jiffies + tg->td->throtl_slice;
throtl_log(&tg->service_queue,
"[%c] new slice with credit start=%lu end=%lu jiffies=%lu",
rw == READ ? 'R' : 'W', tg->slice_start[rw],
@@ -568,7 +764,7 @@
tg->bytes_disp[rw] = 0;
tg->io_disp[rw] = 0;
tg->slice_start[rw] = jiffies;
- tg->slice_end[rw] = jiffies + throtl_slice;
+ tg->slice_end[rw] = jiffies + tg->td->throtl_slice;
throtl_log(&tg->service_queue,
"[%c] new slice start=%lu end=%lu jiffies=%lu",
rw == READ ? 'R' : 'W', tg->slice_start[rw],
@@ -578,13 +774,13 @@
static inline void throtl_set_slice_end(struct throtl_grp *tg, bool rw,
unsigned long jiffy_end)
{
- tg->slice_end[rw] = roundup(jiffy_end, throtl_slice);
+ tg->slice_end[rw] = roundup(jiffy_end, tg->td->throtl_slice);
}
static inline void throtl_extend_slice(struct throtl_grp *tg, bool rw,
unsigned long jiffy_end)
{
- tg->slice_end[rw] = roundup(jiffy_end, throtl_slice);
+ tg->slice_end[rw] = roundup(jiffy_end, tg->td->throtl_slice);
throtl_log(&tg->service_queue,
"[%c] extend slice start=%lu end=%lu jiffies=%lu",
rw == READ ? 'R' : 'W', tg->slice_start[rw],
@@ -624,19 +820,20 @@
* is bad because it does not allow new slice to start.
*/
- throtl_set_slice_end(tg, rw, jiffies + throtl_slice);
+ throtl_set_slice_end(tg, rw, jiffies + tg->td->throtl_slice);
time_elapsed = jiffies - tg->slice_start[rw];
- nr_slices = time_elapsed / throtl_slice;
+ nr_slices = time_elapsed / tg->td->throtl_slice;
if (!nr_slices)
return;
- tmp = tg->bps[rw] * throtl_slice * nr_slices;
+ tmp = tg_bps_limit(tg, rw) * tg->td->throtl_slice * nr_slices;
do_div(tmp, HZ);
bytes_trim = tmp;
- io_trim = (tg->iops[rw] * throtl_slice * nr_slices)/HZ;
+ io_trim = (tg_iops_limit(tg, rw) * tg->td->throtl_slice * nr_slices) /
+ HZ;
if (!bytes_trim && !io_trim)
return;
@@ -651,7 +848,7 @@
else
tg->io_disp[rw] = 0;
- tg->slice_start[rw] += nr_slices * throtl_slice;
+ tg->slice_start[rw] += nr_slices * tg->td->throtl_slice;
throtl_log(&tg->service_queue,
"[%c] trim slice nr=%lu bytes=%llu io=%lu start=%lu end=%lu jiffies=%lu",
@@ -671,9 +868,9 @@
/* Slice has just started. Consider one slice interval */
if (!jiffy_elapsed)
- jiffy_elapsed_rnd = throtl_slice;
+ jiffy_elapsed_rnd = tg->td->throtl_slice;
- jiffy_elapsed_rnd = roundup(jiffy_elapsed_rnd, throtl_slice);
+ jiffy_elapsed_rnd = roundup(jiffy_elapsed_rnd, tg->td->throtl_slice);
/*
* jiffy_elapsed_rnd should not be a big value as minimum iops can be
@@ -682,7 +879,7 @@
* have been trimmed.
*/
- tmp = (u64)tg->iops[rw] * jiffy_elapsed_rnd;
+ tmp = (u64)tg_iops_limit(tg, rw) * jiffy_elapsed_rnd;
do_div(tmp, HZ);
if (tmp > UINT_MAX)
@@ -697,7 +894,7 @@
}
/* Calc approx time to dispatch */
- jiffy_wait = ((tg->io_disp[rw] + 1) * HZ)/tg->iops[rw] + 1;
+ jiffy_wait = ((tg->io_disp[rw] + 1) * HZ) / tg_iops_limit(tg, rw) + 1;
if (jiffy_wait > jiffy_elapsed)
jiffy_wait = jiffy_wait - jiffy_elapsed;
@@ -720,11 +917,11 @@
/* Slice has just started. Consider one slice interval */
if (!jiffy_elapsed)
- jiffy_elapsed_rnd = throtl_slice;
+ jiffy_elapsed_rnd = tg->td->throtl_slice;
- jiffy_elapsed_rnd = roundup(jiffy_elapsed_rnd, throtl_slice);
+ jiffy_elapsed_rnd = roundup(jiffy_elapsed_rnd, tg->td->throtl_slice);
- tmp = tg->bps[rw] * jiffy_elapsed_rnd;
+ tmp = tg_bps_limit(tg, rw) * jiffy_elapsed_rnd;
do_div(tmp, HZ);
bytes_allowed = tmp;
@@ -736,7 +933,7 @@
/* Calc approx time to dispatch */
extra_bytes = tg->bytes_disp[rw] + bio->bi_iter.bi_size - bytes_allowed;
- jiffy_wait = div64_u64(extra_bytes * HZ, tg->bps[rw]);
+ jiffy_wait = div64_u64(extra_bytes * HZ, tg_bps_limit(tg, rw));
if (!jiffy_wait)
jiffy_wait = 1;
@@ -771,7 +968,8 @@
bio != throtl_peek_queued(&tg->service_queue.queued[rw]));
/* If tg->bps = -1, then BW is unlimited */
- if (tg->bps[rw] == -1 && tg->iops[rw] == -1) {
+ if (tg_bps_limit(tg, rw) == U64_MAX &&
+ tg_iops_limit(tg, rw) == UINT_MAX) {
if (wait)
*wait = 0;
return true;
@@ -787,8 +985,10 @@
if (throtl_slice_used(tg, rw) && !(tg->service_queue.nr_queued[rw]))
throtl_start_new_slice(tg, rw);
else {
- if (time_before(tg->slice_end[rw], jiffies + throtl_slice))
- throtl_extend_slice(tg, rw, jiffies + throtl_slice);
+ if (time_before(tg->slice_end[rw],
+ jiffies + tg->td->throtl_slice))
+ throtl_extend_slice(tg, rw,
+ jiffies + tg->td->throtl_slice);
}
if (tg_with_in_bps_limit(tg, bio, &bps_wait) &&
@@ -816,6 +1016,8 @@
/* Charge the bio to the group */
tg->bytes_disp[rw] += bio->bi_iter.bi_size;
tg->io_disp[rw]++;
+ tg->last_bytes_disp[rw] += bio->bi_iter.bi_size;
+ tg->last_io_disp[rw]++;
/*
* BIO_THROTTLED is used to prevent the same bio to be throttled
@@ -999,6 +1201,8 @@
return nr_disp;
}
+static bool throtl_can_upgrade(struct throtl_data *td,
+ struct throtl_grp *this_tg);
/**
* throtl_pending_timer_fn - timer function for service_queue->pending_timer
* @arg: the throtl_service_queue being serviced
@@ -1025,6 +1229,9 @@
int ret;
spin_lock_irq(q->queue_lock);
+ if (throtl_can_upgrade(td, NULL))
+ throtl_upgrade_state(td);
+
again:
parent_sq = sq->parent_sq;
dispatched = false;
@@ -1112,7 +1319,7 @@
struct throtl_grp *tg = pd_to_tg(pd);
u64 v = *(u64 *)((void *)tg + off);
- if (v == -1)
+ if (v == U64_MAX)
return 0;
return __blkg_prfill_u64(sf, pd, v);
}
@@ -1123,7 +1330,7 @@
struct throtl_grp *tg = pd_to_tg(pd);
unsigned int v = *(unsigned int *)((void *)tg + off);
- if (v == -1)
+ if (v == UINT_MAX)
return 0;
return __blkg_prfill_u64(sf, pd, v);
}
@@ -1150,8 +1357,8 @@
throtl_log(&tg->service_queue,
"limit change rbps=%llu wbps=%llu riops=%u wiops=%u",
- tg->bps[READ], tg->bps[WRITE],
- tg->iops[READ], tg->iops[WRITE]);
+ tg_bps_limit(tg, READ), tg_bps_limit(tg, WRITE),
+ tg_iops_limit(tg, READ), tg_iops_limit(tg, WRITE));
/*
* Update has_rules[] flags for the updated tg's subtree. A tg is
@@ -1197,7 +1404,7 @@
if (sscanf(ctx.body, "%llu", &v) != 1)
goto out_finish;
if (!v)
- v = -1;
+ v = U64_MAX;
tg = blkg_to_tg(ctx.blkg);
@@ -1228,25 +1435,25 @@
static struct cftype throtl_legacy_files[] = {
{
.name = "throttle.read_bps_device",
- .private = offsetof(struct throtl_grp, bps[READ]),
+ .private = offsetof(struct throtl_grp, bps[READ][LIMIT_MAX]),
.seq_show = tg_print_conf_u64,
.write = tg_set_conf_u64,
},
{
.name = "throttle.write_bps_device",
- .private = offsetof(struct throtl_grp, bps[WRITE]),
+ .private = offsetof(struct throtl_grp, bps[WRITE][LIMIT_MAX]),
.seq_show = tg_print_conf_u64,
.write = tg_set_conf_u64,
},
{
.name = "throttle.read_iops_device",
- .private = offsetof(struct throtl_grp, iops[READ]),
+ .private = offsetof(struct throtl_grp, iops[READ][LIMIT_MAX]),
.seq_show = tg_print_conf_uint,
.write = tg_set_conf_uint,
},
{
.name = "throttle.write_iops_device",
- .private = offsetof(struct throtl_grp, iops[WRITE]),
+ .private = offsetof(struct throtl_grp, iops[WRITE][LIMIT_MAX]),
.seq_show = tg_print_conf_uint,
.write = tg_set_conf_uint,
},
@@ -1263,48 +1470,87 @@
{ } /* terminate */
};
-static u64 tg_prfill_max(struct seq_file *sf, struct blkg_policy_data *pd,
+static u64 tg_prfill_limit(struct seq_file *sf, struct blkg_policy_data *pd,
int off)
{
struct throtl_grp *tg = pd_to_tg(pd);
const char *dname = blkg_dev_name(pd->blkg);
char bufs[4][21] = { "max", "max", "max", "max" };
+ u64 bps_dft;
+ unsigned int iops_dft;
+ char idle_time[26] = "";
+ char latency_time[26] = "";
if (!dname)
return 0;
- if (tg->bps[READ] == -1 && tg->bps[WRITE] == -1 &&
- tg->iops[READ] == -1 && tg->iops[WRITE] == -1)
+
+ if (off == LIMIT_LOW) {
+ bps_dft = 0;
+ iops_dft = 0;
+ } else {
+ bps_dft = U64_MAX;
+ iops_dft = UINT_MAX;
+ }
+
+ if (tg->bps_conf[READ][off] == bps_dft &&
+ tg->bps_conf[WRITE][off] == bps_dft &&
+ tg->iops_conf[READ][off] == iops_dft &&
+ tg->iops_conf[WRITE][off] == iops_dft &&
+ (off != LIMIT_LOW ||
+ (tg->idletime_threshold == tg->td->dft_idletime_threshold &&
+ tg->latency_target == DFL_LATENCY_TARGET)))
return 0;
- if (tg->bps[READ] != -1)
- snprintf(bufs[0], sizeof(bufs[0]), "%llu", tg->bps[READ]);
- if (tg->bps[WRITE] != -1)
- snprintf(bufs[1], sizeof(bufs[1]), "%llu", tg->bps[WRITE]);
- if (tg->iops[READ] != -1)
- snprintf(bufs[2], sizeof(bufs[2]), "%u", tg->iops[READ]);
- if (tg->iops[WRITE] != -1)
- snprintf(bufs[3], sizeof(bufs[3]), "%u", tg->iops[WRITE]);
+ if (tg->bps_conf[READ][off] != bps_dft)
+ snprintf(bufs[0], sizeof(bufs[0]), "%llu",
+ tg->bps_conf[READ][off]);
+ if (tg->bps_conf[WRITE][off] != bps_dft)
+ snprintf(bufs[1], sizeof(bufs[1]), "%llu",
+ tg->bps_conf[WRITE][off]);
+ if (tg->iops_conf[READ][off] != iops_dft)
+ snprintf(bufs[2], sizeof(bufs[2]), "%u",
+ tg->iops_conf[READ][off]);
+ if (tg->iops_conf[WRITE][off] != iops_dft)
+ snprintf(bufs[3], sizeof(bufs[3]), "%u",
+ tg->iops_conf[WRITE][off]);
+ if (off == LIMIT_LOW) {
+ if (tg->idletime_threshold == ULONG_MAX)
+ strcpy(idle_time, " idle=max");
+ else
+ snprintf(idle_time, sizeof(idle_time), " idle=%lu",
+ tg->idletime_threshold);
- seq_printf(sf, "%s rbps=%s wbps=%s riops=%s wiops=%s\n",
- dname, bufs[0], bufs[1], bufs[2], bufs[3]);
+ if (tg->latency_target == ULONG_MAX)
+ strcpy(latency_time, " latency=max");
+ else
+ snprintf(latency_time, sizeof(latency_time),
+ " latency=%lu", tg->latency_target);
+ }
+
+ seq_printf(sf, "%s rbps=%s wbps=%s riops=%s wiops=%s%s%s\n",
+ dname, bufs[0], bufs[1], bufs[2], bufs[3], idle_time,
+ latency_time);
return 0;
}
-static int tg_print_max(struct seq_file *sf, void *v)
+static int tg_print_limit(struct seq_file *sf, void *v)
{
- blkcg_print_blkgs(sf, css_to_blkcg(seq_css(sf)), tg_prfill_max,
+ blkcg_print_blkgs(sf, css_to_blkcg(seq_css(sf)), tg_prfill_limit,
&blkcg_policy_throtl, seq_cft(sf)->private, false);
return 0;
}
-static ssize_t tg_set_max(struct kernfs_open_file *of,
+static ssize_t tg_set_limit(struct kernfs_open_file *of,
char *buf, size_t nbytes, loff_t off)
{
struct blkcg *blkcg = css_to_blkcg(of_css(of));
struct blkg_conf_ctx ctx;
struct throtl_grp *tg;
u64 v[4];
+ unsigned long idle_time;
+ unsigned long latency_time;
int ret;
+ int index = of_cft(of)->private;
ret = blkg_conf_prep(blkcg, &blkcg_policy_throtl, buf, &ctx);
if (ret)
@@ -1312,15 +1558,17 @@
tg = blkg_to_tg(ctx.blkg);
- v[0] = tg->bps[READ];
- v[1] = tg->bps[WRITE];
- v[2] = tg->iops[READ];
- v[3] = tg->iops[WRITE];
+ v[0] = tg->bps_conf[READ][index];
+ v[1] = tg->bps_conf[WRITE][index];
+ v[2] = tg->iops_conf[READ][index];
+ v[3] = tg->iops_conf[WRITE][index];
+ idle_time = tg->idletime_threshold;
+ latency_time = tg->latency_target;
while (true) {
char tok[27]; /* wiops=18446744073709551616 */
char *p;
- u64 val = -1;
+ u64 val = U64_MAX;
int len;
if (sscanf(ctx.body, "%26s%n", tok, &len) != 1)
@@ -1348,15 +1596,43 @@
v[2] = min_t(u64, val, UINT_MAX);
else if (!strcmp(tok, "wiops"))
v[3] = min_t(u64, val, UINT_MAX);
+ else if (off == LIMIT_LOW && !strcmp(tok, "idle"))
+ idle_time = val;
+ else if (off == LIMIT_LOW && !strcmp(tok, "latency"))
+ latency_time = val;
else
goto out_finish;
}
- tg->bps[READ] = v[0];
- tg->bps[WRITE] = v[1];
- tg->iops[READ] = v[2];
- tg->iops[WRITE] = v[3];
+ tg->bps_conf[READ][index] = v[0];
+ tg->bps_conf[WRITE][index] = v[1];
+ tg->iops_conf[READ][index] = v[2];
+ tg->iops_conf[WRITE][index] = v[3];
+ if (index == LIMIT_MAX) {
+ tg->bps[READ][index] = v[0];
+ tg->bps[WRITE][index] = v[1];
+ tg->iops[READ][index] = v[2];
+ tg->iops[WRITE][index] = v[3];
+ }
+ tg->bps[READ][LIMIT_LOW] = min(tg->bps_conf[READ][LIMIT_LOW],
+ tg->bps_conf[READ][LIMIT_MAX]);
+ tg->bps[WRITE][LIMIT_LOW] = min(tg->bps_conf[WRITE][LIMIT_LOW],
+ tg->bps_conf[WRITE][LIMIT_MAX]);
+ tg->iops[READ][LIMIT_LOW] = min(tg->iops_conf[READ][LIMIT_LOW],
+ tg->iops_conf[READ][LIMIT_MAX]);
+ tg->iops[WRITE][LIMIT_LOW] = min(tg->iops_conf[WRITE][LIMIT_LOW],
+ tg->iops_conf[WRITE][LIMIT_MAX]);
+
+ if (index == LIMIT_LOW) {
+ blk_throtl_update_limit_valid(tg->td);
+ if (tg->td->limit_valid[LIMIT_LOW])
+ tg->td->limit_index = LIMIT_LOW;
+ tg->idletime_threshold = (idle_time == ULONG_MAX) ?
+ ULONG_MAX : idle_time;
+ tg->latency_target = (latency_time == ULONG_MAX) ?
+ ULONG_MAX : latency_time;
+ }
tg_conf_updated(tg);
ret = 0;
out_finish:
@@ -1365,11 +1641,21 @@
}
static struct cftype throtl_files[] = {
+#ifdef CONFIG_BLK_DEV_THROTTLING_LOW
+ {
+ .name = "low",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .seq_show = tg_print_limit,
+ .write = tg_set_limit,
+ .private = LIMIT_LOW,
+ },
+#endif
{
.name = "max",
.flags = CFTYPE_NOT_ON_ROOT,
- .seq_show = tg_print_max,
- .write = tg_set_max,
+ .seq_show = tg_print_limit,
+ .write = tg_set_limit,
+ .private = LIMIT_MAX,
},
{ } /* terminate */
};
@@ -1388,9 +1674,376 @@
.pd_alloc_fn = throtl_pd_alloc,
.pd_init_fn = throtl_pd_init,
.pd_online_fn = throtl_pd_online,
+ .pd_offline_fn = throtl_pd_offline,
.pd_free_fn = throtl_pd_free,
};
+static unsigned long __tg_last_low_overflow_time(struct throtl_grp *tg)
+{
+ unsigned long rtime = jiffies, wtime = jiffies;
+
+ if (tg->bps[READ][LIMIT_LOW] || tg->iops[READ][LIMIT_LOW])
+ rtime = tg->last_low_overflow_time[READ];
+ if (tg->bps[WRITE][LIMIT_LOW] || tg->iops[WRITE][LIMIT_LOW])
+ wtime = tg->last_low_overflow_time[WRITE];
+ return min(rtime, wtime);
+}
+
+/* tg should not be an intermediate node */
+static unsigned long tg_last_low_overflow_time(struct throtl_grp *tg)
+{
+ struct throtl_service_queue *parent_sq;
+ struct throtl_grp *parent = tg;
+ unsigned long ret = __tg_last_low_overflow_time(tg);
+
+ while (true) {
+ parent_sq = parent->service_queue.parent_sq;
+ parent = sq_to_tg(parent_sq);
+ if (!parent)
+ break;
+
+ /*
+ * The parent doesn't have low limit, it always reaches low
+ * limit. Its overflow time is useless for children
+ */
+ if (!parent->bps[READ][LIMIT_LOW] &&
+ !parent->iops[READ][LIMIT_LOW] &&
+ !parent->bps[WRITE][LIMIT_LOW] &&
+ !parent->iops[WRITE][LIMIT_LOW])
+ continue;
+ if (time_after(__tg_last_low_overflow_time(parent), ret))
+ ret = __tg_last_low_overflow_time(parent);
+ }
+ return ret;
+}
+
+static bool throtl_tg_is_idle(struct throtl_grp *tg)
+{
+ /*
+ * cgroup is idle if:
+ * - single idle is too long, longer than a fixed value (in case user
+ * configure a too big threshold) or 4 times of slice
+ * - average think time is more than threshold
+ * - IO latency is largely below threshold
+ */
+ unsigned long time = jiffies_to_usecs(4 * tg->td->throtl_slice);
+
+ time = min_t(unsigned long, MAX_IDLE_TIME, time);
+ return (ktime_get_ns() >> 10) - tg->last_finish_time > time ||
+ tg->avg_idletime > tg->idletime_threshold ||
+ (tg->latency_target && tg->bio_cnt &&
+ tg->bad_bio_cnt * 5 < tg->bio_cnt);
+}
+
+static bool throtl_tg_can_upgrade(struct throtl_grp *tg)
+{
+ struct throtl_service_queue *sq = &tg->service_queue;
+ bool read_limit, write_limit;
+
+ /*
+ * if cgroup reaches low limit (if low limit is 0, the cgroup always
+ * reaches), it's ok to upgrade to next limit
+ */
+ read_limit = tg->bps[READ][LIMIT_LOW] || tg->iops[READ][LIMIT_LOW];
+ write_limit = tg->bps[WRITE][LIMIT_LOW] || tg->iops[WRITE][LIMIT_LOW];
+ if (!read_limit && !write_limit)
+ return true;
+ if (read_limit && sq->nr_queued[READ] &&
+ (!write_limit || sq->nr_queued[WRITE]))
+ return true;
+ if (write_limit && sq->nr_queued[WRITE] &&
+ (!read_limit || sq->nr_queued[READ]))
+ return true;
+
+ if (time_after_eq(jiffies,
+ tg_last_low_overflow_time(tg) + tg->td->throtl_slice) &&
+ throtl_tg_is_idle(tg))
+ return true;
+ return false;
+}
+
+static bool throtl_hierarchy_can_upgrade(struct throtl_grp *tg)
+{
+ while (true) {
+ if (throtl_tg_can_upgrade(tg))
+ return true;
+ tg = sq_to_tg(tg->service_queue.parent_sq);
+ if (!tg || !tg_to_blkg(tg)->parent)
+ return false;
+ }
+ return false;
+}
+
+static bool throtl_can_upgrade(struct throtl_data *td,
+ struct throtl_grp *this_tg)
+{
+ struct cgroup_subsys_state *pos_css;
+ struct blkcg_gq *blkg;
+
+ if (td->limit_index != LIMIT_LOW)
+ return false;
+
+ if (time_before(jiffies, td->low_downgrade_time + td->throtl_slice))
+ return false;
+
+ rcu_read_lock();
+ blkg_for_each_descendant_post(blkg, pos_css, td->queue->root_blkg) {
+ struct throtl_grp *tg = blkg_to_tg(blkg);
+
+ if (tg == this_tg)
+ continue;
+ if (!list_empty(&tg_to_blkg(tg)->blkcg->css.children))
+ continue;
+ if (!throtl_hierarchy_can_upgrade(tg)) {
+ rcu_read_unlock();
+ return false;
+ }
+ }
+ rcu_read_unlock();
+ return true;
+}
+
+static void throtl_upgrade_check(struct throtl_grp *tg)
+{
+ unsigned long now = jiffies;
+
+ if (tg->td->limit_index != LIMIT_LOW)
+ return;
+
+ if (time_after(tg->last_check_time + tg->td->throtl_slice, now))
+ return;
+
+ tg->last_check_time = now;
+
+ if (!time_after_eq(now,
+ __tg_last_low_overflow_time(tg) + tg->td->throtl_slice))
+ return;
+
+ if (throtl_can_upgrade(tg->td, NULL))
+ throtl_upgrade_state(tg->td);
+}
+
+static void throtl_upgrade_state(struct throtl_data *td)
+{
+ struct cgroup_subsys_state *pos_css;
+ struct blkcg_gq *blkg;
+
+ td->limit_index = LIMIT_MAX;
+ td->low_upgrade_time = jiffies;
+ td->scale = 0;
+ rcu_read_lock();
+ blkg_for_each_descendant_post(blkg, pos_css, td->queue->root_blkg) {
+ struct throtl_grp *tg = blkg_to_tg(blkg);
+ struct throtl_service_queue *sq = &tg->service_queue;
+
+ tg->disptime = jiffies - 1;
+ throtl_select_dispatch(sq);
+ throtl_schedule_next_dispatch(sq, false);
+ }
+ rcu_read_unlock();
+ throtl_select_dispatch(&td->service_queue);
+ throtl_schedule_next_dispatch(&td->service_queue, false);
+ queue_work(kthrotld_workqueue, &td->dispatch_work);
+}
+
+static void throtl_downgrade_state(struct throtl_data *td, int new)
+{
+ td->scale /= 2;
+
+ if (td->scale) {
+ td->low_upgrade_time = jiffies - td->scale * td->throtl_slice;
+ return;
+ }
+
+ td->limit_index = new;
+ td->low_downgrade_time = jiffies;
+}
+
+static bool throtl_tg_can_downgrade(struct throtl_grp *tg)
+{
+ struct throtl_data *td = tg->td;
+ unsigned long now = jiffies;
+
+ /*
+ * If cgroup is below low limit, consider downgrade and throttle other
+ * cgroups
+ */
+ if (time_after_eq(now, td->low_upgrade_time + td->throtl_slice) &&
+ time_after_eq(now, tg_last_low_overflow_time(tg) +
+ td->throtl_slice) &&
+ (!throtl_tg_is_idle(tg) ||
+ !list_empty(&tg_to_blkg(tg)->blkcg->css.children)))
+ return true;
+ return false;
+}
+
+static bool throtl_hierarchy_can_downgrade(struct throtl_grp *tg)
+{
+ while (true) {
+ if (!throtl_tg_can_downgrade(tg))
+ return false;
+ tg = sq_to_tg(tg->service_queue.parent_sq);
+ if (!tg || !tg_to_blkg(tg)->parent)
+ break;
+ }
+ return true;
+}
+
+static void throtl_downgrade_check(struct throtl_grp *tg)
+{
+ uint64_t bps;
+ unsigned int iops;
+ unsigned long elapsed_time;
+ unsigned long now = jiffies;
+
+ if (tg->td->limit_index != LIMIT_MAX ||
+ !tg->td->limit_valid[LIMIT_LOW])
+ return;
+ if (!list_empty(&tg_to_blkg(tg)->blkcg->css.children))
+ return;
+ if (time_after(tg->last_check_time + tg->td->throtl_slice, now))
+ return;
+
+ elapsed_time = now - tg->last_check_time;
+ tg->last_check_time = now;
+
+ if (time_before(now, tg_last_low_overflow_time(tg) +
+ tg->td->throtl_slice))
+ return;
+
+ if (tg->bps[READ][LIMIT_LOW]) {
+ bps = tg->last_bytes_disp[READ] * HZ;
+ do_div(bps, elapsed_time);
+ if (bps >= tg->bps[READ][LIMIT_LOW])
+ tg->last_low_overflow_time[READ] = now;
+ }
+
+ if (tg->bps[WRITE][LIMIT_LOW]) {
+ bps = tg->last_bytes_disp[WRITE] * HZ;
+ do_div(bps, elapsed_time);
+ if (bps >= tg->bps[WRITE][LIMIT_LOW])
+ tg->last_low_overflow_time[WRITE] = now;
+ }
+
+ if (tg->iops[READ][LIMIT_LOW]) {
+ iops = tg->last_io_disp[READ] * HZ / elapsed_time;
+ if (iops >= tg->iops[READ][LIMIT_LOW])
+ tg->last_low_overflow_time[READ] = now;
+ }
+
+ if (tg->iops[WRITE][LIMIT_LOW]) {
+ iops = tg->last_io_disp[WRITE] * HZ / elapsed_time;
+ if (iops >= tg->iops[WRITE][LIMIT_LOW])
+ tg->last_low_overflow_time[WRITE] = now;
+ }
+
+ /*
+ * If cgroup is below low limit, consider downgrade and throttle other
+ * cgroups
+ */
+ if (throtl_hierarchy_can_downgrade(tg))
+ throtl_downgrade_state(tg->td, LIMIT_LOW);
+
+ tg->last_bytes_disp[READ] = 0;
+ tg->last_bytes_disp[WRITE] = 0;
+ tg->last_io_disp[READ] = 0;
+ tg->last_io_disp[WRITE] = 0;
+}
+
+static void blk_throtl_update_idletime(struct throtl_grp *tg)
+{
+ unsigned long now = ktime_get_ns() >> 10;
+ unsigned long last_finish_time = tg->last_finish_time;
+
+ if (now <= last_finish_time || last_finish_time == 0 ||
+ last_finish_time == tg->checked_last_finish_time)
+ return;
+
+ tg->avg_idletime = (tg->avg_idletime * 7 + now - last_finish_time) >> 3;
+ tg->checked_last_finish_time = last_finish_time;
+}
+
+#ifdef CONFIG_BLK_DEV_THROTTLING_LOW
+static void throtl_update_latency_buckets(struct throtl_data *td)
+{
+ struct avg_latency_bucket avg_latency[LATENCY_BUCKET_SIZE];
+ int i, cpu;
+ unsigned long last_latency = 0;
+ unsigned long latency;
+
+ if (!blk_queue_nonrot(td->queue))
+ return;
+ if (time_before(jiffies, td->last_calculate_time + HZ))
+ return;
+ td->last_calculate_time = jiffies;
+
+ memset(avg_latency, 0, sizeof(avg_latency));
+ for (i = 0; i < LATENCY_BUCKET_SIZE; i++) {
+ struct latency_bucket *tmp = &td->tmp_buckets[i];
+
+ for_each_possible_cpu(cpu) {
+ struct latency_bucket *bucket;
+
+ /* this isn't race free, but ok in practice */
+ bucket = per_cpu_ptr(td->latency_buckets, cpu);
+ tmp->total_latency += bucket[i].total_latency;
+ tmp->samples += bucket[i].samples;
+ bucket[i].total_latency = 0;
+ bucket[i].samples = 0;
+ }
+
+ if (tmp->samples >= 32) {
+ int samples = tmp->samples;
+
+ latency = tmp->total_latency;
+
+ tmp->total_latency = 0;
+ tmp->samples = 0;
+ latency /= samples;
+ if (latency == 0)
+ continue;
+ avg_latency[i].latency = latency;
+ }
+ }
+
+ for (i = 0; i < LATENCY_BUCKET_SIZE; i++) {
+ if (!avg_latency[i].latency) {
+ if (td->avg_buckets[i].latency < last_latency)
+ td->avg_buckets[i].latency = last_latency;
+ continue;
+ }
+
+ if (!td->avg_buckets[i].valid)
+ latency = avg_latency[i].latency;
+ else
+ latency = (td->avg_buckets[i].latency * 7 +
+ avg_latency[i].latency) >> 3;
+
+ td->avg_buckets[i].latency = max(latency, last_latency);
+ td->avg_buckets[i].valid = true;
+ last_latency = td->avg_buckets[i].latency;
+ }
+}
+#else
+static inline void throtl_update_latency_buckets(struct throtl_data *td)
+{
+}
+#endif
+
+static void blk_throtl_assoc_bio(struct throtl_grp *tg, struct bio *bio)
+{
+#ifdef CONFIG_BLK_DEV_THROTTLING_LOW
+ int ret;
+
+ ret = bio_associate_current(bio);
+ if (ret == 0 || ret == -EBUSY)
+ bio->bi_cg_private = tg;
+ blk_stat_set_issue(&bio->bi_issue_stat, bio_sectors(bio));
+#else
+ bio_associate_current(bio);
+#endif
+}
+
bool blk_throtl_bio(struct request_queue *q, struct blkcg_gq *blkg,
struct bio *bio)
{
@@ -1399,6 +2052,7 @@
struct throtl_service_queue *sq;
bool rw = bio_data_dir(bio);
bool throttled = false;
+ struct throtl_data *td = tg->td;
WARN_ON_ONCE(!rcu_read_lock_held());
@@ -1408,19 +2062,35 @@
spin_lock_irq(q->queue_lock);
+ throtl_update_latency_buckets(td);
+
if (unlikely(blk_queue_bypass(q)))
goto out_unlock;
+ blk_throtl_assoc_bio(tg, bio);
+ blk_throtl_update_idletime(tg);
+
sq = &tg->service_queue;
+again:
while (true) {
+ if (tg->last_low_overflow_time[rw] == 0)
+ tg->last_low_overflow_time[rw] = jiffies;
+ throtl_downgrade_check(tg);
+ throtl_upgrade_check(tg);
/* throtl is FIFO - if bios are already queued, should queue */
if (sq->nr_queued[rw])
break;
/* if above limits, break to queue */
- if (!tg_may_dispatch(tg, bio, NULL))
+ if (!tg_may_dispatch(tg, bio, NULL)) {
+ tg->last_low_overflow_time[rw] = jiffies;
+ if (throtl_can_upgrade(td, tg)) {
+ throtl_upgrade_state(td);
+ goto again;
+ }
break;
+ }
/* within limits, let's charge and dispatch directly */
throtl_charge_bio(tg, bio);
@@ -1453,12 +2123,14 @@
/* out-of-limit, queue to @tg */
throtl_log(sq, "[%c] bio. bdisp=%llu sz=%u bps=%llu iodisp=%u iops=%u queued=%d/%d",
rw == READ ? 'R' : 'W',
- tg->bytes_disp[rw], bio->bi_iter.bi_size, tg->bps[rw],
- tg->io_disp[rw], tg->iops[rw],
+ tg->bytes_disp[rw], bio->bi_iter.bi_size,
+ tg_bps_limit(tg, rw),
+ tg->io_disp[rw], tg_iops_limit(tg, rw),
sq->nr_queued[READ], sq->nr_queued[WRITE]);
- bio_associate_current(bio);
- tg->td->nr_queued[rw]++;
+ tg->last_low_overflow_time[rw] = jiffies;
+
+ td->nr_queued[rw]++;
throtl_add_bio_tg(bio, qn, tg);
throttled = true;
@@ -1483,9 +2155,94 @@
*/
if (!throttled)
bio_clear_flag(bio, BIO_THROTTLED);
+
+#ifdef CONFIG_BLK_DEV_THROTTLING_LOW
+ if (throttled || !td->track_bio_latency)
+ bio->bi_issue_stat.stat |= SKIP_LATENCY;
+#endif
return throttled;
}
+#ifdef CONFIG_BLK_DEV_THROTTLING_LOW
+static void throtl_track_latency(struct throtl_data *td, sector_t size,
+ int op, unsigned long time)
+{
+ struct latency_bucket *latency;
+ int index;
+
+ if (!td || td->limit_index != LIMIT_LOW || op != REQ_OP_READ ||
+ !blk_queue_nonrot(td->queue))
+ return;
+
+ index = request_bucket_index(size);
+
+ latency = get_cpu_ptr(td->latency_buckets);
+ latency[index].total_latency += time;
+ latency[index].samples++;
+ put_cpu_ptr(td->latency_buckets);
+}
+
+void blk_throtl_stat_add(struct request *rq, u64 time_ns)
+{
+ struct request_queue *q = rq->q;
+ struct throtl_data *td = q->td;
+
+ throtl_track_latency(td, blk_stat_size(&rq->issue_stat),
+ req_op(rq), time_ns >> 10);
+}
+
+void blk_throtl_bio_endio(struct bio *bio)
+{
+ struct throtl_grp *tg;
+ u64 finish_time_ns;
+ unsigned long finish_time;
+ unsigned long start_time;
+ unsigned long lat;
+
+ tg = bio->bi_cg_private;
+ if (!tg)
+ return;
+ bio->bi_cg_private = NULL;
+
+ finish_time_ns = ktime_get_ns();
+ tg->last_finish_time = finish_time_ns >> 10;
+
+ start_time = blk_stat_time(&bio->bi_issue_stat) >> 10;
+ finish_time = __blk_stat_time(finish_time_ns) >> 10;
+ if (!start_time || finish_time <= start_time)
+ return;
+
+ lat = finish_time - start_time;
+ /* this is only for bio based driver */
+ if (!(bio->bi_issue_stat.stat & SKIP_LATENCY))
+ throtl_track_latency(tg->td, blk_stat_size(&bio->bi_issue_stat),
+ bio_op(bio), lat);
+
+ if (tg->latency_target) {
+ int bucket;
+ unsigned int threshold;
+
+ bucket = request_bucket_index(
+ blk_stat_size(&bio->bi_issue_stat));
+ threshold = tg->td->avg_buckets[bucket].latency +
+ tg->latency_target;
+ if (lat > threshold)
+ tg->bad_bio_cnt++;
+ /*
+ * Not race free, could get wrong count, which means cgroups
+ * will be throttled
+ */
+ tg->bio_cnt++;
+ }
+
+ if (time_after(jiffies, tg->bio_cnt_reset_time) || tg->bio_cnt > 1024) {
+ tg->bio_cnt_reset_time = tg->td->throtl_slice + jiffies;
+ tg->bio_cnt /= 2;
+ tg->bad_bio_cnt /= 2;
+ }
+}
+#endif
+
/*
* Dispatch all bios from all children tg's queued on @parent_sq. On
* return, @parent_sq is guaranteed to not have any active children tg's
@@ -1558,6 +2315,12 @@
td = kzalloc_node(sizeof(*td), GFP_KERNEL, q->node);
if (!td)
return -ENOMEM;
+ td->latency_buckets = __alloc_percpu(sizeof(struct latency_bucket) *
+ LATENCY_BUCKET_SIZE, __alignof__(u64));
+ if (!td->latency_buckets) {
+ kfree(td);
+ return -ENOMEM;
+ }
INIT_WORK(&td->dispatch_work, blk_throtl_dispatch_work_fn);
throtl_service_queue_init(&td->service_queue);
@@ -1565,10 +2328,17 @@
q->td = td;
td->queue = q;
+ td->limit_valid[LIMIT_MAX] = true;
+ td->limit_index = LIMIT_MAX;
+ td->low_upgrade_time = jiffies;
+ td->low_downgrade_time = jiffies;
+
/* activate policy */
ret = blkcg_activate_policy(q, &blkcg_policy_throtl);
- if (ret)
+ if (ret) {
+ free_percpu(td->latency_buckets);
kfree(td);
+ }
return ret;
}
@@ -1577,9 +2347,74 @@
BUG_ON(!q->td);
throtl_shutdown_wq(q);
blkcg_deactivate_policy(q, &blkcg_policy_throtl);
+ free_percpu(q->td->latency_buckets);
kfree(q->td);
}
+void blk_throtl_register_queue(struct request_queue *q)
+{
+ struct throtl_data *td;
+ struct cgroup_subsys_state *pos_css;
+ struct blkcg_gq *blkg;
+
+ td = q->td;
+ BUG_ON(!td);
+
+ if (blk_queue_nonrot(q)) {
+ td->throtl_slice = DFL_THROTL_SLICE_SSD;
+ td->dft_idletime_threshold = DFL_IDLE_THRESHOLD_SSD;
+ } else {
+ td->throtl_slice = DFL_THROTL_SLICE_HD;
+ td->dft_idletime_threshold = DFL_IDLE_THRESHOLD_HD;
+ }
+#ifndef CONFIG_BLK_DEV_THROTTLING_LOW
+ /* if no low limit, use previous default */
+ td->throtl_slice = DFL_THROTL_SLICE_HD;
+#endif
+
+ td->track_bio_latency = !q->mq_ops && !q->request_fn;
+ if (!td->track_bio_latency)
+ blk_stat_enable_accounting(q);
+
+ /*
+ * some tg are created before queue is fully initialized, eg, nonrot
+ * isn't initialized yet
+ */
+ rcu_read_lock();
+ blkg_for_each_descendant_post(blkg, pos_css, q->root_blkg) {
+ struct throtl_grp *tg = blkg_to_tg(blkg);
+
+ tg->idletime_threshold = td->dft_idletime_threshold;
+ }
+ rcu_read_unlock();
+}
+
+#ifdef CONFIG_BLK_DEV_THROTTLING_LOW
+ssize_t blk_throtl_sample_time_show(struct request_queue *q, char *page)
+{
+ if (!q->td)
+ return -EINVAL;
+ return sprintf(page, "%u\n", jiffies_to_msecs(q->td->throtl_slice));
+}
+
+ssize_t blk_throtl_sample_time_store(struct request_queue *q,
+ const char *page, size_t count)
+{
+ unsigned long v;
+ unsigned long t;
+
+ if (!q->td)
+ return -EINVAL;
+ if (kstrtoul(page, 10, &v))
+ return -EINVAL;
+ t = msecs_to_jiffies(v);
+ if (t == 0 || t > MAX_THROTL_SLICE)
+ return -EINVAL;
+ q->td->throtl_slice = t;
+ return count;
+}
+#endif
+
static int __init throtl_init(void)
{
kthrotld_workqueue = alloc_workqueue("kthrotld", WQ_MEM_RECLAIM, 0);
diff --git a/block/blk-timeout.c b/block/blk-timeout.c
index a30441a..cbff183 100644
--- a/block/blk-timeout.c
+++ b/block/blk-timeout.c
@@ -89,7 +89,6 @@
ret = q->rq_timed_out_fn(req);
switch (ret) {
case BLK_EH_HANDLED:
- /* Can we use req->errors here? */
__blk_complete_request(req);
break;
case BLK_EH_RESET_TIMER:
diff --git a/block/blk-wbt.c b/block/blk-wbt.c
index 1aedb1f..17676f4 100644
--- a/block/blk-wbt.c
+++ b/block/blk-wbt.c
@@ -255,8 +255,8 @@
* that it's writes impacting us, and not just some sole read on
* a device that is in a lower power state.
*/
- return stat[BLK_STAT_READ].nr_samples >= 1 &&
- stat[BLK_STAT_WRITE].nr_samples >= RWB_MIN_WRITE_SAMPLES;
+ return (stat[READ].nr_samples >= 1 &&
+ stat[WRITE].nr_samples >= RWB_MIN_WRITE_SAMPLES);
}
static u64 rwb_sync_issue_lat(struct rq_wb *rwb)
@@ -277,7 +277,7 @@
LAT_EXCEEDED,
};
-static int __latency_exceeded(struct rq_wb *rwb, struct blk_rq_stat *stat)
+static int latency_exceeded(struct rq_wb *rwb, struct blk_rq_stat *stat)
{
struct backing_dev_info *bdi = rwb->queue->backing_dev_info;
u64 thislat;
@@ -293,7 +293,7 @@
*/
thislat = rwb_sync_issue_lat(rwb);
if (thislat > rwb->cur_win_nsec ||
- (thislat > rwb->min_lat_nsec && !stat[BLK_STAT_READ].nr_samples)) {
+ (thislat > rwb->min_lat_nsec && !stat[READ].nr_samples)) {
trace_wbt_lat(bdi, thislat);
return LAT_EXCEEDED;
}
@@ -308,8 +308,8 @@
* waited or still has writes in flights, consider us doing
* just writes as well.
*/
- if ((stat[BLK_STAT_WRITE].nr_samples && blk_stat_is_current(stat)) ||
- wb_recent_wait(rwb) || wbt_inflight(rwb))
+ if (stat[WRITE].nr_samples || wb_recent_wait(rwb) ||
+ wbt_inflight(rwb))
return LAT_UNKNOWN_WRITES;
return LAT_UNKNOWN;
}
@@ -317,8 +317,8 @@
/*
* If the 'min' latency exceeds our target, step down.
*/
- if (stat[BLK_STAT_READ].min > rwb->min_lat_nsec) {
- trace_wbt_lat(bdi, stat[BLK_STAT_READ].min);
+ if (stat[READ].min > rwb->min_lat_nsec) {
+ trace_wbt_lat(bdi, stat[READ].min);
trace_wbt_stat(bdi, stat);
return LAT_EXCEEDED;
}
@@ -329,14 +329,6 @@
return LAT_OK;
}
-static int latency_exceeded(struct rq_wb *rwb)
-{
- struct blk_rq_stat stat[2];
-
- blk_queue_stat_get(rwb->queue, stat);
- return __latency_exceeded(rwb, stat);
-}
-
static void rwb_trace_step(struct rq_wb *rwb, const char *msg)
{
struct backing_dev_info *bdi = rwb->queue->backing_dev_info;
@@ -355,7 +347,6 @@
rwb->scale_step--;
rwb->unknown_cnt = 0;
- blk_stat_clear(rwb->queue);
rwb->scaled_max = calc_wb_limits(rwb);
@@ -385,15 +376,12 @@
rwb->scaled_max = false;
rwb->unknown_cnt = 0;
- blk_stat_clear(rwb->queue);
calc_wb_limits(rwb);
rwb_trace_step(rwb, "step down");
}
static void rwb_arm_timer(struct rq_wb *rwb)
{
- unsigned long expires;
-
if (rwb->scale_step > 0) {
/*
* We should speed this up, using some variant of a fast
@@ -411,17 +399,16 @@
rwb->cur_win_nsec = rwb->win_nsec;
}
- expires = jiffies + nsecs_to_jiffies(rwb->cur_win_nsec);
- mod_timer(&rwb->window_timer, expires);
+ blk_stat_activate_nsecs(rwb->cb, rwb->cur_win_nsec);
}
-static void wb_timer_fn(unsigned long data)
+static void wb_timer_fn(struct blk_stat_callback *cb)
{
- struct rq_wb *rwb = (struct rq_wb *) data;
+ struct rq_wb *rwb = cb->data;
unsigned int inflight = wbt_inflight(rwb);
int status;
- status = latency_exceeded(rwb);
+ status = latency_exceeded(rwb, cb->stat);
trace_wbt_timer(rwb->queue->backing_dev_info, status, rwb->scale_step,
inflight);
@@ -614,7 +601,7 @@
__wbt_wait(rwb, bio->bi_opf, lock);
- if (!timer_pending(&rwb->window_timer))
+ if (!blk_stat_is_active(rwb->cb))
rwb_arm_timer(rwb);
if (current_is_kswapd())
@@ -666,22 +653,37 @@
rwb->wc = write_cache_on;
}
- /*
- * Disable wbt, if enabled by default. Only called from CFQ, if we have
- * cgroups enabled
+/*
+ * Disable wbt, if enabled by default. Only called from CFQ.
*/
void wbt_disable_default(struct request_queue *q)
{
struct rq_wb *rwb = q->rq_wb;
- if (rwb && rwb->enable_state == WBT_STATE_ON_DEFAULT) {
- del_timer_sync(&rwb->window_timer);
- rwb->win_nsec = rwb->min_lat_nsec = 0;
- wbt_update_limits(rwb);
- }
+ if (rwb && rwb->enable_state == WBT_STATE_ON_DEFAULT)
+ wbt_exit(q);
}
EXPORT_SYMBOL_GPL(wbt_disable_default);
+/*
+ * Enable wbt if defaults are configured that way
+ */
+void wbt_enable_default(struct request_queue *q)
+{
+ /* Throttling already enabled? */
+ if (q->rq_wb)
+ return;
+
+ /* Queue not registered? Maybe shutting down... */
+ if (!test_bit(QUEUE_FLAG_REGISTERED, &q->queue_flags))
+ return;
+
+ if ((q->mq_ops && IS_ENABLED(CONFIG_BLK_WBT_MQ)) ||
+ (q->request_fn && IS_ENABLED(CONFIG_BLK_WBT_SQ)))
+ wbt_init(q);
+}
+EXPORT_SYMBOL_GPL(wbt_enable_default);
+
u64 wbt_default_latency_nsec(struct request_queue *q)
{
/*
@@ -694,29 +696,33 @@
return 75000000ULL;
}
+static int wbt_data_dir(const struct request *rq)
+{
+ return rq_data_dir(rq);
+}
+
int wbt_init(struct request_queue *q)
{
struct rq_wb *rwb;
int i;
- /*
- * For now, we depend on the stats window being larger than
- * our monitoring window. Ensure that this isn't inadvertently
- * violated.
- */
- BUILD_BUG_ON(RWB_WINDOW_NSEC > BLK_STAT_NSEC);
BUILD_BUG_ON(WBT_NR_BITS > BLK_STAT_RES_BITS);
rwb = kzalloc(sizeof(*rwb), GFP_KERNEL);
if (!rwb)
return -ENOMEM;
+ rwb->cb = blk_stat_alloc_callback(wb_timer_fn, wbt_data_dir, 2, rwb);
+ if (!rwb->cb) {
+ kfree(rwb);
+ return -ENOMEM;
+ }
+
for (i = 0; i < WBT_NUM_RWQ; i++) {
atomic_set(&rwb->rq_wait[i].inflight, 0);
init_waitqueue_head(&rwb->rq_wait[i].wait);
}
- setup_timer(&rwb->window_timer, wb_timer_fn, (unsigned long) rwb);
rwb->wc = 1;
rwb->queue_depth = RWB_DEF_DEPTH;
rwb->last_comp = rwb->last_issue = jiffies;
@@ -726,10 +732,10 @@
wbt_update_limits(rwb);
/*
- * Assign rwb, and turn on stats tracking for this queue
+ * Assign rwb and add the stats callback.
*/
q->rq_wb = rwb;
- blk_stat_enable(q);
+ blk_stat_add_callback(q, rwb->cb);
rwb->min_lat_nsec = wbt_default_latency_nsec(q);
@@ -744,7 +750,8 @@
struct rq_wb *rwb = q->rq_wb;
if (rwb) {
- del_timer_sync(&rwb->window_timer);
+ blk_stat_remove_callback(q, rwb->cb);
+ blk_stat_free_callback(rwb->cb);
q->rq_wb = NULL;
kfree(rwb);
}
diff --git a/block/blk-wbt.h b/block/blk-wbt.h
index 65f1de5..df6de50 100644
--- a/block/blk-wbt.h
+++ b/block/blk-wbt.h
@@ -32,27 +32,27 @@
static inline void wbt_clear_state(struct blk_issue_stat *stat)
{
- stat->time &= BLK_STAT_TIME_MASK;
+ stat->stat &= ~BLK_STAT_RES_MASK;
}
static inline enum wbt_flags wbt_stat_to_mask(struct blk_issue_stat *stat)
{
- return (stat->time & BLK_STAT_MASK) >> BLK_STAT_SHIFT;
+ return (stat->stat & BLK_STAT_RES_MASK) >> BLK_STAT_RES_SHIFT;
}
static inline void wbt_track(struct blk_issue_stat *stat, enum wbt_flags wb_acct)
{
- stat->time |= ((u64) wb_acct) << BLK_STAT_SHIFT;
+ stat->stat |= ((u64) wb_acct) << BLK_STAT_RES_SHIFT;
}
static inline bool wbt_is_tracked(struct blk_issue_stat *stat)
{
- return (stat->time >> BLK_STAT_SHIFT) & WBT_TRACKED;
+ return (stat->stat >> BLK_STAT_RES_SHIFT) & WBT_TRACKED;
}
static inline bool wbt_is_read(struct blk_issue_stat *stat)
{
- return (stat->time >> BLK_STAT_SHIFT) & WBT_READ;
+ return (stat->stat >> BLK_STAT_RES_SHIFT) & WBT_READ;
}
struct rq_wait {
@@ -81,7 +81,7 @@
u64 win_nsec; /* default window size */
u64 cur_win_nsec; /* current window size */
- struct timer_list window_timer;
+ struct blk_stat_callback *cb;
s64 sync_issue;
void *sync_cookie;
@@ -117,6 +117,7 @@
void wbt_requeue(struct rq_wb *, struct blk_issue_stat *);
void wbt_issue(struct rq_wb *, struct blk_issue_stat *);
void wbt_disable_default(struct request_queue *);
+void wbt_enable_default(struct request_queue *);
void wbt_set_queue_depth(struct rq_wb *, unsigned int);
void wbt_set_write_cache(struct rq_wb *, bool);
@@ -155,6 +156,9 @@
static inline void wbt_disable_default(struct request_queue *q)
{
}
+static inline void wbt_enable_default(struct request_queue *q)
+{
+}
static inline void wbt_set_queue_depth(struct rq_wb *rwb, unsigned int depth)
{
}
diff --git a/block/blk.h b/block/blk.h
index d1ea4bd9..2ed7022 100644
--- a/block/blk.h
+++ b/block/blk.h
@@ -60,15 +60,12 @@
int blk_init_rl(struct request_list *rl, struct request_queue *q,
gfp_t gfp_mask);
void blk_exit_rl(struct request_list *rl);
-void init_request_from_bio(struct request *req, struct bio *bio);
void blk_rq_bio_prep(struct request_queue *q, struct request *rq,
struct bio *bio);
void blk_queue_bypass_start(struct request_queue *q);
void blk_queue_bypass_end(struct request_queue *q);
void blk_dequeue_request(struct request *rq);
void __blk_queue_free_tags(struct request_queue *q);
-bool __blk_end_bidi_request(struct request *rq, int error,
- unsigned int nr_bytes, unsigned int bidi_bytes);
void blk_freeze_queue(struct request_queue *q);
static inline void blk_queue_enter_live(struct request_queue *q)
@@ -319,10 +316,22 @@
extern void blk_throtl_drain(struct request_queue *q);
extern int blk_throtl_init(struct request_queue *q);
extern void blk_throtl_exit(struct request_queue *q);
+extern void blk_throtl_register_queue(struct request_queue *q);
#else /* CONFIG_BLK_DEV_THROTTLING */
static inline void blk_throtl_drain(struct request_queue *q) { }
static inline int blk_throtl_init(struct request_queue *q) { return 0; }
static inline void blk_throtl_exit(struct request_queue *q) { }
+static inline void blk_throtl_register_queue(struct request_queue *q) { }
#endif /* CONFIG_BLK_DEV_THROTTLING */
+#ifdef CONFIG_BLK_DEV_THROTTLING_LOW
+extern ssize_t blk_throtl_sample_time_show(struct request_queue *q, char *page);
+extern ssize_t blk_throtl_sample_time_store(struct request_queue *q,
+ const char *page, size_t count);
+extern void blk_throtl_bio_endio(struct bio *bio);
+extern void blk_throtl_stat_add(struct request *rq, u64 time);
+#else
+static inline void blk_throtl_bio_endio(struct bio *bio) { }
+static inline void blk_throtl_stat_add(struct request *rq, u64 time) { }
+#endif
#endif /* BLK_INTERNAL_H */
diff --git a/block/bsg-lib.c b/block/bsg-lib.c
index cd15f9d..0a23dbb 100644
--- a/block/bsg-lib.c
+++ b/block/bsg-lib.c
@@ -37,7 +37,7 @@
struct bsg_job *job = container_of(kref, struct bsg_job, kref);
struct request *rq = job->req;
- blk_end_request_all(rq, rq->errors);
+ blk_end_request_all(rq, scsi_req(rq)->result);
put_device(job->dev); /* release reference for the request */
@@ -74,7 +74,7 @@
struct scsi_request *rq = scsi_req(req);
int err;
- err = job->req->errors = result;
+ err = scsi_req(job->req)->result = result;
if (err < 0)
/* we're only returning the result field in the reply */
rq->sense_len = sizeof(u32);
@@ -177,7 +177,7 @@
* @q: request queue to manage
*
* On error the create_bsg_job function should return a -Exyz error value
- * that will be set to the req->errors.
+ * that will be set to ->result.
*
* Drivers/subsys should pass this to the queue init function.
*/
@@ -201,7 +201,7 @@
ret = bsg_create_job(dev, req);
if (ret) {
- req->errors = ret;
+ scsi_req(req)->result = ret;
blk_end_request_all(req, ret);
spin_lock_irq(q->queue_lock);
continue;
diff --git a/block/bsg.c b/block/bsg.c
index 74835db..6fd0854 100644
--- a/block/bsg.c
+++ b/block/bsg.c
@@ -391,13 +391,13 @@
struct scsi_request *req = scsi_req(rq);
int ret = 0;
- dprintk("rq %p bio %p 0x%x\n", rq, bio, rq->errors);
+ dprintk("rq %p bio %p 0x%x\n", rq, bio, req->result);
/*
* fill in all the output members
*/
- hdr->device_status = rq->errors & 0xff;
- hdr->transport_status = host_byte(rq->errors);
- hdr->driver_status = driver_byte(rq->errors);
+ hdr->device_status = req->result & 0xff;
+ hdr->transport_status = host_byte(req->result);
+ hdr->driver_status = driver_byte(req->result);
hdr->info = 0;
if (hdr->device_status || hdr->transport_status || hdr->driver_status)
hdr->info |= SG_INFO_CHECK;
@@ -431,8 +431,8 @@
* just a protocol response (i.e. non negative), that gets
* processed above.
*/
- if (!ret && rq->errors < 0)
- ret = rq->errors;
+ if (!ret && req->result < 0)
+ ret = req->result;
blk_rq_unmap_user(bio);
scsi_req_free_cmd(req);
@@ -650,7 +650,7 @@
dprintk("%s: write %zd bytes\n", bd->name, count);
- if (unlikely(segment_eq(get_fs(), KERNEL_DS)))
+ if (unlikely(uaccess_kernel()))
return -EINVAL;
bsg_set_block(bd, file);
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 440b95e..da69b07 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -3761,16 +3761,14 @@
}
#ifdef CONFIG_CFQ_GROUP_IOSCHED
-static bool check_blkcg_changed(struct cfq_io_cq *cic, struct bio *bio)
+static void check_blkcg_changed(struct cfq_io_cq *cic, struct bio *bio)
{
struct cfq_data *cfqd = cic_to_cfqd(cic);
struct cfq_queue *cfqq;
uint64_t serial_nr;
- bool nonroot_cg;
rcu_read_lock();
serial_nr = bio_blkcg(bio)->css.serial_nr;
- nonroot_cg = bio_blkcg(bio) != &blkcg_root;
rcu_read_unlock();
/*
@@ -3778,7 +3776,7 @@
* spuriously on a newly created cic but there's no harm.
*/
if (unlikely(!cfqd) || likely(cic->blkcg_serial_nr == serial_nr))
- return nonroot_cg;
+ return;
/*
* Drop reference to queues. New queues will be assigned in new
@@ -3799,12 +3797,10 @@
}
cic->blkcg_serial_nr = serial_nr;
- return nonroot_cg;
}
#else
-static inline bool check_blkcg_changed(struct cfq_io_cq *cic, struct bio *bio)
+static inline void check_blkcg_changed(struct cfq_io_cq *cic, struct bio *bio)
{
- return false;
}
#endif /* CONFIG_CFQ_GROUP_IOSCHED */
@@ -4449,12 +4445,11 @@
const int rw = rq_data_dir(rq);
const bool is_sync = rq_is_sync(rq);
struct cfq_queue *cfqq;
- bool disable_wbt;
spin_lock_irq(q->queue_lock);
check_ioprio_changed(cic, bio);
- disable_wbt = check_blkcg_changed(cic, bio);
+ check_blkcg_changed(cic, bio);
new_queue:
cfqq = cic_to_cfqq(cic, is_sync);
if (!cfqq || cfqq == &cfqd->oom_cfqq) {
@@ -4491,9 +4486,6 @@
rq->elv.priv[1] = cfqq->cfqg;
spin_unlock_irq(q->queue_lock);
- if (disable_wbt)
- wbt_disable_default(q);
-
return 0;
}
@@ -4706,6 +4698,7 @@
*/
if (blk_queue_nonrot(q))
cfqd->cfq_slice_idle = 0;
+ wbt_disable_default(q);
}
/*
diff --git a/block/compat_ioctl.c b/block/compat_ioctl.c
index 570021a..04325b8 100644
--- a/block/compat_ioctl.c
+++ b/block/compat_ioctl.c
@@ -685,7 +685,7 @@
case BLKALIGNOFF:
return compat_put_int(arg, bdev_alignment_offset(bdev));
case BLKDISCARDZEROES:
- return compat_put_uint(arg, bdev_discard_zeroes_data(bdev));
+ return compat_put_uint(arg, 0);
case BLKFLSBUF:
case BLKROSET:
case BLKDISCARD:
diff --git a/block/elevator.c b/block/elevator.c
index 4d9084a..bf11e70 100644
--- a/block/elevator.c
+++ b/block/elevator.c
@@ -41,6 +41,7 @@
#include "blk.h"
#include "blk-mq-sched.h"
+#include "blk-wbt.h"
static DEFINE_SPINLOCK(elv_list_lock);
static LIST_HEAD(elv_list);
@@ -877,6 +878,8 @@
kobject_uevent(&e->kobj, KOBJ_REMOVE);
kobject_del(&e->kobj);
e->registered = 0;
+ /* Re-enable throttling in case elevator disabled it */
+ wbt_enable_default(q);
}
}
EXPORT_SYMBOL(elv_unregister_queue);
diff --git a/block/genhd.c b/block/genhd.c
index a9c516a..d252d29 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -271,16 +271,17 @@
/**
* register_blkdev - register a new block device
*
- * @major: the requested major device number [1..255]. If @major=0, try to
+ * @major: the requested major device number [1..255]. If @major = 0, try to
* allocate any unused major number.
* @name: the name of the new block device as a zero terminated string
*
* The @name must be unique within the system.
*
- * The return value depends on the @major input parameter.
+ * The return value depends on the @major input parameter:
+ *
* - if a major device number was requested in range [1..255] then the
* function returns zero on success, or a negative error code
- * - if any unused major number was requested with @major=0 parameter
+ * - if any unused major number was requested with @major = 0 parameter
* then the return value is the allocated major number in range
* [1..255] or a negative error code otherwise
*/
@@ -1060,8 +1061,19 @@
NULL
};
+static umode_t disk_visible(struct kobject *kobj, struct attribute *a, int n)
+{
+ struct device *dev = container_of(kobj, typeof(*dev), kobj);
+ struct gendisk *disk = dev_to_disk(dev);
+
+ if (a == &dev_attr_badblocks.attr && !disk->bb)
+ return 0;
+ return a->mode;
+}
+
static struct attribute_group disk_attr_group = {
.attrs = disk_attrs,
+ .is_visible = disk_visible,
};
static const struct attribute_group *disk_attr_groups[] = {
@@ -1352,7 +1364,7 @@
owner = disk->fops->owner;
if (owner && !try_module_get(owner))
return NULL;
- kobj = kobject_get(&disk_to_dev(disk)->kobj);
+ kobj = kobject_get_unless_zero(&disk_to_dev(disk)->kobj);
if (kobj == NULL) {
module_put(owner);
return NULL;
diff --git a/block/ioctl.c b/block/ioctl.c
index 7b88820..0de02ee 100644
--- a/block/ioctl.c
+++ b/block/ioctl.c
@@ -255,7 +255,7 @@
truncate_inode_pages_range(mapping, start, end);
return blkdev_issue_zeroout(bdev, start >> 9, len >> 9, GFP_KERNEL,
- false);
+ BLKDEV_ZERO_NOUNMAP);
}
static int put_ushort(unsigned long arg, unsigned short val)
@@ -547,7 +547,7 @@
case BLKALIGNOFF:
return put_int(arg, bdev_alignment_offset(bdev));
case BLKDISCARDZEROES:
- return put_uint(arg, bdev_discard_zeroes_data(bdev));
+ return put_uint(arg, 0);
case BLKSECTGET:
max_sectors = min_t(unsigned int, USHRT_MAX,
queue_max_sectors(bdev_get_queue(bdev)));
diff --git a/block/ioprio.c b/block/ioprio.c
index 0c47a00..4b120c9 100644
--- a/block/ioprio.c
+++ b/block/ioprio.c
@@ -163,22 +163,12 @@
int ioprio_best(unsigned short aprio, unsigned short bprio)
{
- unsigned short aclass;
- unsigned short bclass;
-
if (!ioprio_valid(aprio))
aprio = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_BE, IOPRIO_NORM);
if (!ioprio_valid(bprio))
bprio = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_BE, IOPRIO_NORM);
- aclass = IOPRIO_PRIO_CLASS(aprio);
- bclass = IOPRIO_PRIO_CLASS(bprio);
- if (aclass == bclass)
- return min(aprio, bprio);
- if (aclass > bclass)
- return bprio;
- else
- return aprio;
+ return min(aprio, bprio);
}
SYSCALL_DEFINE2(ioprio_get, int, which, int, who)
diff --git a/block/kyber-iosched.c b/block/kyber-iosched.c
new file mode 100644
index 0000000..3b0090b
--- /dev/null
+++ b/block/kyber-iosched.c
@@ -0,0 +1,719 @@
+/*
+ * The Kyber I/O scheduler. Controls latency by throttling queue depths using
+ * scalable techniques.
+ *
+ * Copyright (C) 2017 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ */
+
+#include <linux/kernel.h>
+#include <linux/blkdev.h>
+#include <linux/blk-mq.h>
+#include <linux/elevator.h>
+#include <linux/module.h>
+#include <linux/sbitmap.h>
+
+#include "blk.h"
+#include "blk-mq.h"
+#include "blk-mq-sched.h"
+#include "blk-mq-tag.h"
+#include "blk-stat.h"
+
+/* Scheduling domains. */
+enum {
+ KYBER_READ,
+ KYBER_SYNC_WRITE,
+ KYBER_OTHER, /* Async writes, discard, etc. */
+ KYBER_NUM_DOMAINS,
+};
+
+enum {
+ KYBER_MIN_DEPTH = 256,
+
+ /*
+ * In order to prevent starvation of synchronous requests by a flood of
+ * asynchronous requests, we reserve 25% of requests for synchronous
+ * operations.
+ */
+ KYBER_ASYNC_PERCENT = 75,
+};
+
+/*
+ * Initial device-wide depths for each scheduling domain.
+ *
+ * Even for fast devices with lots of tags like NVMe, you can saturate
+ * the device with only a fraction of the maximum possible queue depth.
+ * So, we cap these to a reasonable value.
+ */
+static const unsigned int kyber_depth[] = {
+ [KYBER_READ] = 256,
+ [KYBER_SYNC_WRITE] = 128,
+ [KYBER_OTHER] = 64,
+};
+
+/*
+ * Scheduling domain batch sizes. We favor reads.
+ */
+static const unsigned int kyber_batch_size[] = {
+ [KYBER_READ] = 16,
+ [KYBER_SYNC_WRITE] = 8,
+ [KYBER_OTHER] = 8,
+};
+
+struct kyber_queue_data {
+ struct request_queue *q;
+
+ struct blk_stat_callback *cb;
+
+ /*
+ * The device is divided into multiple scheduling domains based on the
+ * request type. Each domain has a fixed number of in-flight requests of
+ * that type device-wide, limited by these tokens.
+ */
+ struct sbitmap_queue domain_tokens[KYBER_NUM_DOMAINS];
+
+ /*
+ * Async request percentage, converted to per-word depth for
+ * sbitmap_get_shallow().
+ */
+ unsigned int async_depth;
+
+ /* Target latencies in nanoseconds. */
+ u64 read_lat_nsec, write_lat_nsec;
+};
+
+struct kyber_hctx_data {
+ spinlock_t lock;
+ struct list_head rqs[KYBER_NUM_DOMAINS];
+ unsigned int cur_domain;
+ unsigned int batching;
+ wait_queue_t domain_wait[KYBER_NUM_DOMAINS];
+ atomic_t wait_index[KYBER_NUM_DOMAINS];
+};
+
+static int rq_sched_domain(const struct request *rq)
+{
+ unsigned int op = rq->cmd_flags;
+
+ if ((op & REQ_OP_MASK) == REQ_OP_READ)
+ return KYBER_READ;
+ else if ((op & REQ_OP_MASK) == REQ_OP_WRITE && op_is_sync(op))
+ return KYBER_SYNC_WRITE;
+ else
+ return KYBER_OTHER;
+}
+
+enum {
+ NONE = 0,
+ GOOD = 1,
+ GREAT = 2,
+ BAD = -1,
+ AWFUL = -2,
+};
+
+#define IS_GOOD(status) ((status) > 0)
+#define IS_BAD(status) ((status) < 0)
+
+static int kyber_lat_status(struct blk_stat_callback *cb,
+ unsigned int sched_domain, u64 target)
+{
+ u64 latency;
+
+ if (!cb->stat[sched_domain].nr_samples)
+ return NONE;
+
+ latency = cb->stat[sched_domain].mean;
+ if (latency >= 2 * target)
+ return AWFUL;
+ else if (latency > target)
+ return BAD;
+ else if (latency <= target / 2)
+ return GREAT;
+ else /* (latency <= target) */
+ return GOOD;
+}
+
+/*
+ * Adjust the read or synchronous write depth given the status of reads and
+ * writes. The goal is that the latencies of the two domains are fair (i.e., if
+ * one is good, then the other is good).
+ */
+static void kyber_adjust_rw_depth(struct kyber_queue_data *kqd,
+ unsigned int sched_domain, int this_status,
+ int other_status)
+{
+ unsigned int orig_depth, depth;
+
+ /*
+ * If this domain had no samples, or reads and writes are both good or
+ * both bad, don't adjust the depth.
+ */
+ if (this_status == NONE ||
+ (IS_GOOD(this_status) && IS_GOOD(other_status)) ||
+ (IS_BAD(this_status) && IS_BAD(other_status)))
+ return;
+
+ orig_depth = depth = kqd->domain_tokens[sched_domain].sb.depth;
+
+ if (other_status == NONE) {
+ depth++;
+ } else {
+ switch (this_status) {
+ case GOOD:
+ if (other_status == AWFUL)
+ depth -= max(depth / 4, 1U);
+ else
+ depth -= max(depth / 8, 1U);
+ break;
+ case GREAT:
+ if (other_status == AWFUL)
+ depth /= 2;
+ else
+ depth -= max(depth / 4, 1U);
+ break;
+ case BAD:
+ depth++;
+ break;
+ case AWFUL:
+ if (other_status == GREAT)
+ depth += 2;
+ else
+ depth++;
+ break;
+ }
+ }
+
+ depth = clamp(depth, 1U, kyber_depth[sched_domain]);
+ if (depth != orig_depth)
+ sbitmap_queue_resize(&kqd->domain_tokens[sched_domain], depth);
+}
+
+/*
+ * Adjust the depth of other requests given the status of reads and synchronous
+ * writes. As long as either domain is doing fine, we don't throttle, but if
+ * both domains are doing badly, we throttle heavily.
+ */
+static void kyber_adjust_other_depth(struct kyber_queue_data *kqd,
+ int read_status, int write_status,
+ bool have_samples)
+{
+ unsigned int orig_depth, depth;
+ int status;
+
+ orig_depth = depth = kqd->domain_tokens[KYBER_OTHER].sb.depth;
+
+ if (read_status == NONE && write_status == NONE) {
+ depth += 2;
+ } else if (have_samples) {
+ if (read_status == NONE)
+ status = write_status;
+ else if (write_status == NONE)
+ status = read_status;
+ else
+ status = max(read_status, write_status);
+ switch (status) {
+ case GREAT:
+ depth += 2;
+ break;
+ case GOOD:
+ depth++;
+ break;
+ case BAD:
+ depth -= max(depth / 4, 1U);
+ break;
+ case AWFUL:
+ depth /= 2;
+ break;
+ }
+ }
+
+ depth = clamp(depth, 1U, kyber_depth[KYBER_OTHER]);
+ if (depth != orig_depth)
+ sbitmap_queue_resize(&kqd->domain_tokens[KYBER_OTHER], depth);
+}
+
+/*
+ * Apply heuristics for limiting queue depths based on gathered latency
+ * statistics.
+ */
+static void kyber_stat_timer_fn(struct blk_stat_callback *cb)
+{
+ struct kyber_queue_data *kqd = cb->data;
+ int read_status, write_status;
+
+ read_status = kyber_lat_status(cb, KYBER_READ, kqd->read_lat_nsec);
+ write_status = kyber_lat_status(cb, KYBER_SYNC_WRITE, kqd->write_lat_nsec);
+
+ kyber_adjust_rw_depth(kqd, KYBER_READ, read_status, write_status);
+ kyber_adjust_rw_depth(kqd, KYBER_SYNC_WRITE, write_status, read_status);
+ kyber_adjust_other_depth(kqd, read_status, write_status,
+ cb->stat[KYBER_OTHER].nr_samples != 0);
+
+ /*
+ * Continue monitoring latencies if we aren't hitting the targets or
+ * we're still throttling other requests.
+ */
+ if (!blk_stat_is_active(kqd->cb) &&
+ ((IS_BAD(read_status) || IS_BAD(write_status) ||
+ kqd->domain_tokens[KYBER_OTHER].sb.depth < kyber_depth[KYBER_OTHER])))
+ blk_stat_activate_msecs(kqd->cb, 100);
+}
+
+static unsigned int kyber_sched_tags_shift(struct kyber_queue_data *kqd)
+{
+ /*
+ * All of the hardware queues have the same depth, so we can just grab
+ * the shift of the first one.
+ */
+ return kqd->q->queue_hw_ctx[0]->sched_tags->bitmap_tags.sb.shift;
+}
+
+static struct kyber_queue_data *kyber_queue_data_alloc(struct request_queue *q)
+{
+ struct kyber_queue_data *kqd;
+ unsigned int max_tokens;
+ unsigned int shift;
+ int ret = -ENOMEM;
+ int i;
+
+ kqd = kmalloc_node(sizeof(*kqd), GFP_KERNEL, q->node);
+ if (!kqd)
+ goto err;
+ kqd->q = q;
+
+ kqd->cb = blk_stat_alloc_callback(kyber_stat_timer_fn, rq_sched_domain,
+ KYBER_NUM_DOMAINS, kqd);
+ if (!kqd->cb)
+ goto err_kqd;
+
+ /*
+ * The maximum number of tokens for any scheduling domain is at least
+ * the queue depth of a single hardware queue. If the hardware doesn't
+ * have many tags, still provide a reasonable number.
+ */
+ max_tokens = max_t(unsigned int, q->tag_set->queue_depth,
+ KYBER_MIN_DEPTH);
+ for (i = 0; i < KYBER_NUM_DOMAINS; i++) {
+ WARN_ON(!kyber_depth[i]);
+ WARN_ON(!kyber_batch_size[i]);
+ ret = sbitmap_queue_init_node(&kqd->domain_tokens[i],
+ max_tokens, -1, false, GFP_KERNEL,
+ q->node);
+ if (ret) {
+ while (--i >= 0)
+ sbitmap_queue_free(&kqd->domain_tokens[i]);
+ goto err_cb;
+ }
+ sbitmap_queue_resize(&kqd->domain_tokens[i], kyber_depth[i]);
+ }
+
+ shift = kyber_sched_tags_shift(kqd);
+ kqd->async_depth = (1U << shift) * KYBER_ASYNC_PERCENT / 100U;
+
+ kqd->read_lat_nsec = 2000000ULL;
+ kqd->write_lat_nsec = 10000000ULL;
+
+ return kqd;
+
+err_cb:
+ blk_stat_free_callback(kqd->cb);
+err_kqd:
+ kfree(kqd);
+err:
+ return ERR_PTR(ret);
+}
+
+static int kyber_init_sched(struct request_queue *q, struct elevator_type *e)
+{
+ struct kyber_queue_data *kqd;
+ struct elevator_queue *eq;
+
+ eq = elevator_alloc(q, e);
+ if (!eq)
+ return -ENOMEM;
+
+ kqd = kyber_queue_data_alloc(q);
+ if (IS_ERR(kqd)) {
+ kobject_put(&eq->kobj);
+ return PTR_ERR(kqd);
+ }
+
+ eq->elevator_data = kqd;
+ q->elevator = eq;
+
+ blk_stat_add_callback(q, kqd->cb);
+
+ return 0;
+}
+
+static void kyber_exit_sched(struct elevator_queue *e)
+{
+ struct kyber_queue_data *kqd = e->elevator_data;
+ struct request_queue *q = kqd->q;
+ int i;
+
+ blk_stat_remove_callback(q, kqd->cb);
+
+ for (i = 0; i < KYBER_NUM_DOMAINS; i++)
+ sbitmap_queue_free(&kqd->domain_tokens[i]);
+ blk_stat_free_callback(kqd->cb);
+ kfree(kqd);
+}
+
+static int kyber_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx)
+{
+ struct kyber_hctx_data *khd;
+ int i;
+
+ khd = kmalloc_node(sizeof(*khd), GFP_KERNEL, hctx->numa_node);
+ if (!khd)
+ return -ENOMEM;
+
+ spin_lock_init(&khd->lock);
+
+ for (i = 0; i < KYBER_NUM_DOMAINS; i++) {
+ INIT_LIST_HEAD(&khd->rqs[i]);
+ INIT_LIST_HEAD(&khd->domain_wait[i].task_list);
+ atomic_set(&khd->wait_index[i], 0);
+ }
+
+ khd->cur_domain = 0;
+ khd->batching = 0;
+
+ hctx->sched_data = khd;
+
+ return 0;
+}
+
+static void kyber_exit_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx)
+{
+ kfree(hctx->sched_data);
+}
+
+static int rq_get_domain_token(struct request *rq)
+{
+ return (long)rq->elv.priv[0];
+}
+
+static void rq_set_domain_token(struct request *rq, int token)
+{
+ rq->elv.priv[0] = (void *)(long)token;
+}
+
+static void rq_clear_domain_token(struct kyber_queue_data *kqd,
+ struct request *rq)
+{
+ unsigned int sched_domain;
+ int nr;
+
+ nr = rq_get_domain_token(rq);
+ if (nr != -1) {
+ sched_domain = rq_sched_domain(rq);
+ sbitmap_queue_clear(&kqd->domain_tokens[sched_domain], nr,
+ rq->mq_ctx->cpu);
+ }
+}
+
+static struct request *kyber_get_request(struct request_queue *q,
+ unsigned int op,
+ struct blk_mq_alloc_data *data)
+{
+ struct kyber_queue_data *kqd = q->elevator->elevator_data;
+ struct request *rq;
+
+ /*
+ * We use the scheduler tags as per-hardware queue queueing tokens.
+ * Async requests can be limited at this stage.
+ */
+ if (!op_is_sync(op))
+ data->shallow_depth = kqd->async_depth;
+
+ rq = __blk_mq_alloc_request(data, op);
+ if (rq)
+ rq_set_domain_token(rq, -1);
+ return rq;
+}
+
+static void kyber_put_request(struct request *rq)
+{
+ struct request_queue *q = rq->q;
+ struct kyber_queue_data *kqd = q->elevator->elevator_data;
+
+ rq_clear_domain_token(kqd, rq);
+ blk_mq_finish_request(rq);
+}
+
+static void kyber_completed_request(struct request *rq)
+{
+ struct request_queue *q = rq->q;
+ struct kyber_queue_data *kqd = q->elevator->elevator_data;
+ unsigned int sched_domain;
+ u64 now, latency, target;
+
+ /*
+ * Check if this request met our latency goal. If not, quickly gather
+ * some statistics and start throttling.
+ */
+ sched_domain = rq_sched_domain(rq);
+ switch (sched_domain) {
+ case KYBER_READ:
+ target = kqd->read_lat_nsec;
+ break;
+ case KYBER_SYNC_WRITE:
+ target = kqd->write_lat_nsec;
+ break;
+ default:
+ return;
+ }
+
+ /* If we are already monitoring latencies, don't check again. */
+ if (blk_stat_is_active(kqd->cb))
+ return;
+
+ now = __blk_stat_time(ktime_to_ns(ktime_get()));
+ if (now < blk_stat_time(&rq->issue_stat))
+ return;
+
+ latency = now - blk_stat_time(&rq->issue_stat);
+
+ if (latency > target)
+ blk_stat_activate_msecs(kqd->cb, 10);
+}
+
+static void kyber_flush_busy_ctxs(struct kyber_hctx_data *khd,
+ struct blk_mq_hw_ctx *hctx)
+{
+ LIST_HEAD(rq_list);
+ struct request *rq, *next;
+
+ blk_mq_flush_busy_ctxs(hctx, &rq_list);
+ list_for_each_entry_safe(rq, next, &rq_list, queuelist) {
+ unsigned int sched_domain;
+
+ sched_domain = rq_sched_domain(rq);
+ list_move_tail(&rq->queuelist, &khd->rqs[sched_domain]);
+ }
+}
+
+static int kyber_domain_wake(wait_queue_t *wait, unsigned mode, int flags,
+ void *key)
+{
+ struct blk_mq_hw_ctx *hctx = READ_ONCE(wait->private);
+
+ list_del_init(&wait->task_list);
+ blk_mq_run_hw_queue(hctx, true);
+ return 1;
+}
+
+static int kyber_get_domain_token(struct kyber_queue_data *kqd,
+ struct kyber_hctx_data *khd,
+ struct blk_mq_hw_ctx *hctx)
+{
+ unsigned int sched_domain = khd->cur_domain;
+ struct sbitmap_queue *domain_tokens = &kqd->domain_tokens[sched_domain];
+ wait_queue_t *wait = &khd->domain_wait[sched_domain];
+ struct sbq_wait_state *ws;
+ int nr;
+
+ nr = __sbitmap_queue_get(domain_tokens);
+ if (nr >= 0)
+ return nr;
+
+ /*
+ * If we failed to get a domain token, make sure the hardware queue is
+ * run when one becomes available. Note that this is serialized on
+ * khd->lock, but we still need to be careful about the waker.
+ */
+ if (list_empty_careful(&wait->task_list)) {
+ init_waitqueue_func_entry(wait, kyber_domain_wake);
+ wait->private = hctx;
+ ws = sbq_wait_ptr(domain_tokens,
+ &khd->wait_index[sched_domain]);
+ add_wait_queue(&ws->wait, wait);
+
+ /*
+ * Try again in case a token was freed before we got on the wait
+ * queue.
+ */
+ nr = __sbitmap_queue_get(domain_tokens);
+ }
+ return nr;
+}
+
+static struct request *
+kyber_dispatch_cur_domain(struct kyber_queue_data *kqd,
+ struct kyber_hctx_data *khd,
+ struct blk_mq_hw_ctx *hctx,
+ bool *flushed)
+{
+ struct list_head *rqs;
+ struct request *rq;
+ int nr;
+
+ rqs = &khd->rqs[khd->cur_domain];
+ rq = list_first_entry_or_null(rqs, struct request, queuelist);
+
+ /*
+ * If there wasn't already a pending request and we haven't flushed the
+ * software queues yet, flush the software queues and check again.
+ */
+ if (!rq && !*flushed) {
+ kyber_flush_busy_ctxs(khd, hctx);
+ *flushed = true;
+ rq = list_first_entry_or_null(rqs, struct request, queuelist);
+ }
+
+ if (rq) {
+ nr = kyber_get_domain_token(kqd, khd, hctx);
+ if (nr >= 0) {
+ khd->batching++;
+ rq_set_domain_token(rq, nr);
+ list_del_init(&rq->queuelist);
+ return rq;
+ }
+ }
+
+ /* There were either no pending requests or no tokens. */
+ return NULL;
+}
+
+static struct request *kyber_dispatch_request(struct blk_mq_hw_ctx *hctx)
+{
+ struct kyber_queue_data *kqd = hctx->queue->elevator->elevator_data;
+ struct kyber_hctx_data *khd = hctx->sched_data;
+ bool flushed = false;
+ struct request *rq;
+ int i;
+
+ spin_lock(&khd->lock);
+
+ /*
+ * First, if we are still entitled to batch, try to dispatch a request
+ * from the batch.
+ */
+ if (khd->batching < kyber_batch_size[khd->cur_domain]) {
+ rq = kyber_dispatch_cur_domain(kqd, khd, hctx, &flushed);
+ if (rq)
+ goto out;
+ }
+
+ /*
+ * Either,
+ * 1. We were no longer entitled to a batch.
+ * 2. The domain we were batching didn't have any requests.
+ * 3. The domain we were batching was out of tokens.
+ *
+ * Start another batch. Note that this wraps back around to the original
+ * domain if no other domains have requests or tokens.
+ */
+ khd->batching = 0;
+ for (i = 0; i < KYBER_NUM_DOMAINS; i++) {
+ if (khd->cur_domain == KYBER_NUM_DOMAINS - 1)
+ khd->cur_domain = 0;
+ else
+ khd->cur_domain++;
+
+ rq = kyber_dispatch_cur_domain(kqd, khd, hctx, &flushed);
+ if (rq)
+ goto out;
+ }
+
+ rq = NULL;
+out:
+ spin_unlock(&khd->lock);
+ return rq;
+}
+
+static bool kyber_has_work(struct blk_mq_hw_ctx *hctx)
+{
+ struct kyber_hctx_data *khd = hctx->sched_data;
+ int i;
+
+ for (i = 0; i < KYBER_NUM_DOMAINS; i++) {
+ if (!list_empty_careful(&khd->rqs[i]))
+ return true;
+ }
+ return false;
+}
+
+#define KYBER_LAT_SHOW_STORE(op) \
+static ssize_t kyber_##op##_lat_show(struct elevator_queue *e, \
+ char *page) \
+{ \
+ struct kyber_queue_data *kqd = e->elevator_data; \
+ \
+ return sprintf(page, "%llu\n", kqd->op##_lat_nsec); \
+} \
+ \
+static ssize_t kyber_##op##_lat_store(struct elevator_queue *e, \
+ const char *page, size_t count) \
+{ \
+ struct kyber_queue_data *kqd = e->elevator_data; \
+ unsigned long long nsec; \
+ int ret; \
+ \
+ ret = kstrtoull(page, 10, &nsec); \
+ if (ret) \
+ return ret; \
+ \
+ kqd->op##_lat_nsec = nsec; \
+ \
+ return count; \
+}
+KYBER_LAT_SHOW_STORE(read);
+KYBER_LAT_SHOW_STORE(write);
+#undef KYBER_LAT_SHOW_STORE
+
+#define KYBER_LAT_ATTR(op) __ATTR(op##_lat_nsec, 0644, kyber_##op##_lat_show, kyber_##op##_lat_store)
+static struct elv_fs_entry kyber_sched_attrs[] = {
+ KYBER_LAT_ATTR(read),
+ KYBER_LAT_ATTR(write),
+ __ATTR_NULL
+};
+#undef KYBER_LAT_ATTR
+
+static struct elevator_type kyber_sched = {
+ .ops.mq = {
+ .init_sched = kyber_init_sched,
+ .exit_sched = kyber_exit_sched,
+ .init_hctx = kyber_init_hctx,
+ .exit_hctx = kyber_exit_hctx,
+ .get_request = kyber_get_request,
+ .put_request = kyber_put_request,
+ .completed_request = kyber_completed_request,
+ .dispatch_request = kyber_dispatch_request,
+ .has_work = kyber_has_work,
+ },
+ .uses_mq = true,
+ .elevator_attrs = kyber_sched_attrs,
+ .elevator_name = "kyber",
+ .elevator_owner = THIS_MODULE,
+};
+
+static int __init kyber_init(void)
+{
+ return elv_register(&kyber_sched);
+}
+
+static void __exit kyber_exit(void)
+{
+ elv_unregister(&kyber_sched);
+}
+
+module_init(kyber_init);
+module_exit(kyber_exit);
+
+MODULE_AUTHOR("Omar Sandoval");
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Kyber I/O scheduler");
diff --git a/block/partition-generic.c b/block/partition-generic.c
index 7afb990..0171a2f 100644
--- a/block/partition-generic.c
+++ b/block/partition-generic.c
@@ -497,7 +497,6 @@
if (disk->fops->revalidate_disk)
disk->fops->revalidate_disk(disk);
- blk_integrity_revalidate(disk);
check_disk_size_change(disk, bdev);
bdev->bd_invalidated = 0;
if (!get_capacity(disk) || !(state = check_partition(disk, bdev)))
diff --git a/block/scsi_ioctl.c b/block/scsi_ioctl.c
index 2a2fc76..4a294a5 100644
--- a/block/scsi_ioctl.c
+++ b/block/scsi_ioctl.c
@@ -262,11 +262,11 @@
/*
* fill in all the output members
*/
- hdr->status = rq->errors & 0xff;
- hdr->masked_status = status_byte(rq->errors);
- hdr->msg_status = msg_byte(rq->errors);
- hdr->host_status = host_byte(rq->errors);
- hdr->driver_status = driver_byte(rq->errors);
+ hdr->status = req->result & 0xff;
+ hdr->masked_status = status_byte(req->result);
+ hdr->msg_status = msg_byte(req->result);
+ hdr->host_status = host_byte(req->result);
+ hdr->driver_status = driver_byte(req->result);
hdr->info = 0;
if (hdr->masked_status || hdr->host_status || hdr->driver_status)
hdr->info |= SG_INFO_CHECK;
@@ -362,7 +362,7 @@
goto out_free_cdb;
bio = rq->bio;
- rq->retries = 0;
+ req->retries = 0;
start_time = jiffies;
@@ -476,13 +476,13 @@
goto error;
/* default. possible overriden later */
- rq->retries = 5;
+ req->retries = 5;
switch (opcode) {
case SEND_DIAGNOSTIC:
case FORMAT_UNIT:
rq->timeout = FORMAT_UNIT_TIMEOUT;
- rq->retries = 1;
+ req->retries = 1;
break;
case START_STOP:
rq->timeout = START_STOP_TIMEOUT;
@@ -495,7 +495,7 @@
break;
case READ_DEFECT_DATA:
rq->timeout = READ_DEFECT_DATA_TIMEOUT;
- rq->retries = 1;
+ req->retries = 1;
break;
default:
rq->timeout = BLK_DEFAULT_SG_TIMEOUT;
@@ -509,7 +509,7 @@
blk_execute_rq(q, disk, rq, 0);
- err = rq->errors & 0xff; /* only 8 bit SCSI status */
+ err = req->result & 0xff; /* only 8 bit SCSI status */
if (err) {
if (req->sense_len && req->sense) {
bytes = (OMAX_SB_LEN > req->sense_len) ?
@@ -547,7 +547,8 @@
scsi_req(rq)->cmd[0] = cmd;
scsi_req(rq)->cmd[4] = data;
scsi_req(rq)->cmd_len = 6;
- err = blk_execute_rq(q, bd_disk, rq, 0);
+ blk_execute_rq(q, bd_disk, rq, 0);
+ err = scsi_req(rq)->result ? -EIO : 0;
blk_put_request(rq);
return err;
diff --git a/block/sed-opal.c b/block/sed-opal.c
index 14035f8..9b30ae5 100644
--- a/block/sed-opal.c
+++ b/block/sed-opal.c
@@ -275,8 +275,8 @@
u8 flags = tper->supported_features;
if (!(flags & TPER_SYNC_SUPPORTED)) {
- pr_err("TPer sync not supported. flags = %d\n",
- tper->supported_features);
+ pr_debug("TPer sync not supported. flags = %d\n",
+ tper->supported_features);
return false;
}
@@ -289,7 +289,7 @@
u32 nlo = be32_to_cpu(sum->num_locking_objects);
if (nlo == 0) {
- pr_err("Need at least one locking object.\n");
+ pr_debug("Need at least one locking object.\n");
return false;
}
@@ -385,9 +385,9 @@
error = step->fn(dev, step->data);
if (error) {
- pr_err("Error on step function: %d with error %d: %s\n",
- state, error,
- opal_error_to_human(error));
+ pr_debug("Error on step function: %d with error %d: %s\n",
+ state, error,
+ opal_error_to_human(error));
/* For each OPAL command we do a discovery0 then we
* start some sort of session.
@@ -419,8 +419,8 @@
print_buffer(dev->resp, hlen);
if (hlen > IO_BUFFER_LENGTH - sizeof(*hdr)) {
- pr_warn("Discovery length overflows buffer (%zu+%u)/%u\n",
- sizeof(*hdr), hlen, IO_BUFFER_LENGTH);
+ pr_debug("Discovery length overflows buffer (%zu+%u)/%u\n",
+ sizeof(*hdr), hlen, IO_BUFFER_LENGTH);
return -EFAULT;
}
@@ -503,7 +503,7 @@
if (*err)
return;
if (cmd->pos >= IO_BUFFER_LENGTH - 1) {
- pr_err("Error adding u8: end of buffer.\n");
+ pr_debug("Error adding u8: end of buffer.\n");
*err = -ERANGE;
return;
}
@@ -553,7 +553,7 @@
len = DIV_ROUND_UP(msb, 4);
if (cmd->pos >= IO_BUFFER_LENGTH - len - 1) {
- pr_err("Error adding u64: end of buffer.\n");
+ pr_debug("Error adding u64: end of buffer.\n");
*err = -ERANGE;
return;
}
@@ -579,7 +579,7 @@
}
if (len >= IO_BUFFER_LENGTH - cmd->pos - header_len) {
- pr_err("Error adding bytestring: end of buffer.\n");
+ pr_debug("Error adding bytestring: end of buffer.\n");
*err = -ERANGE;
return;
}
@@ -597,7 +597,7 @@
static int build_locking_range(u8 *buffer, size_t length, u8 lr)
{
if (length > OPAL_UID_LENGTH) {
- pr_err("Can't build locking range. Length OOB\n");
+ pr_debug("Can't build locking range. Length OOB\n");
return -ERANGE;
}
@@ -614,7 +614,7 @@
static int build_locking_user(u8 *buffer, size_t length, u8 lr)
{
if (length > OPAL_UID_LENGTH) {
- pr_err("Can't build locking range user, Length OOB\n");
+ pr_debug("Can't build locking range user, Length OOB\n");
return -ERANGE;
}
@@ -648,7 +648,7 @@
add_token_u8(&err, cmd, OPAL_ENDLIST);
if (err) {
- pr_err("Error finalizing command.\n");
+ pr_debug("Error finalizing command.\n");
return -EFAULT;
}
@@ -660,7 +660,7 @@
hdr->subpkt.length = cpu_to_be32(cmd->pos - sizeof(*hdr));
while (cmd->pos % 4) {
if (cmd->pos >= IO_BUFFER_LENGTH) {
- pr_err("Error: Buffer overrun\n");
+ pr_debug("Error: Buffer overrun\n");
return -ERANGE;
}
cmd->cmd[cmd->pos++] = 0;
@@ -679,14 +679,14 @@
const struct opal_resp_tok *tok;
if (n >= resp->num) {
- pr_err("Token number doesn't exist: %d, resp: %d\n",
- n, resp->num);
+ pr_debug("Token number doesn't exist: %d, resp: %d\n",
+ n, resp->num);
return ERR_PTR(-EINVAL);
}
tok = &resp->toks[n];
if (tok->len == 0) {
- pr_err("Token length must be non-zero\n");
+ pr_debug("Token length must be non-zero\n");
return ERR_PTR(-EINVAL);
}
@@ -727,7 +727,7 @@
tok->type = OPAL_DTA_TOKENID_UINT;
if (tok->len > 9) {
- pr_warn("uint64 with more than 8 bytes\n");
+ pr_debug("uint64 with more than 8 bytes\n");
return -EINVAL;
}
for (i = tok->len - 1; i > 0; i--) {
@@ -814,8 +814,8 @@
if (clen == 0 || plen == 0 || slen == 0 ||
slen > IO_BUFFER_LENGTH - sizeof(*hdr)) {
- pr_err("Bad header length. cp: %u, pkt: %u, subpkt: %u\n",
- clen, plen, slen);
+ pr_debug("Bad header length. cp: %u, pkt: %u, subpkt: %u\n",
+ clen, plen, slen);
print_buffer(pos, sizeof(*hdr));
return -EINVAL;
}
@@ -848,7 +848,7 @@
}
if (num_entries == 0) {
- pr_err("Couldn't parse response.\n");
+ pr_debug("Couldn't parse response.\n");
return -EINVAL;
}
resp->num = num_entries;
@@ -861,18 +861,18 @@
{
*store = NULL;
if (!resp) {
- pr_err("Response is NULL\n");
+ pr_debug("Response is NULL\n");
return 0;
}
if (n > resp->num) {
- pr_err("Response has %d tokens. Can't access %d\n",
- resp->num, n);
+ pr_debug("Response has %d tokens. Can't access %d\n",
+ resp->num, n);
return 0;
}
if (resp->toks[n].type != OPAL_DTA_TOKENID_BYTESTRING) {
- pr_err("Token is not a byte string!\n");
+ pr_debug("Token is not a byte string!\n");
return 0;
}
@@ -883,26 +883,26 @@
static u64 response_get_u64(const struct parsed_resp *resp, int n)
{
if (!resp) {
- pr_err("Response is NULL\n");
+ pr_debug("Response is NULL\n");
return 0;
}
if (n > resp->num) {
- pr_err("Response has %d tokens. Can't access %d\n",
- resp->num, n);
+ pr_debug("Response has %d tokens. Can't access %d\n",
+ resp->num, n);
return 0;
}
if (resp->toks[n].type != OPAL_DTA_TOKENID_UINT) {
- pr_err("Token is not unsigned it: %d\n",
- resp->toks[n].type);
+ pr_debug("Token is not unsigned it: %d\n",
+ resp->toks[n].type);
return 0;
}
if (!(resp->toks[n].width == OPAL_WIDTH_TINY ||
resp->toks[n].width == OPAL_WIDTH_SHORT)) {
- pr_err("Atom is not short or tiny: %d\n",
- resp->toks[n].width);
+ pr_debug("Atom is not short or tiny: %d\n",
+ resp->toks[n].width);
return 0;
}
@@ -949,7 +949,7 @@
error = response_parse(dev->resp, IO_BUFFER_LENGTH, &dev->parsed);
if (error) {
- pr_err("Couldn't parse response.\n");
+ pr_debug("Couldn't parse response.\n");
return error;
}
@@ -975,7 +975,7 @@
tsn = response_get_u64(&dev->parsed, 5);
if (hsn == 0 && tsn == 0) {
- pr_err("Couldn't authenticate session\n");
+ pr_debug("Couldn't authenticate session\n");
return -EPERM;
}
@@ -1012,7 +1012,7 @@
ret = cmd_finalize(dev, dev->hsn, dev->tsn);
if (ret) {
- pr_err("Error finalizing command buffer: %d\n", ret);
+ pr_debug("Error finalizing command buffer: %d\n", ret);
return ret;
}
@@ -1041,7 +1041,7 @@
add_token_u8(&err, dev, OPAL_ENDLIST);
if (err) {
- pr_err("Error building gen key command\n");
+ pr_debug("Error building gen key command\n");
return err;
}
@@ -1059,8 +1059,8 @@
return error;
keylen = response_get_string(&dev->parsed, 4, &activekey);
if (!activekey) {
- pr_err("%s: Couldn't extract the Activekey from the response\n",
- __func__);
+ pr_debug("%s: Couldn't extract the Activekey from the response\n",
+ __func__);
return OPAL_INVAL_PARAM;
}
dev->prev_data = kmemdup(activekey, keylen, GFP_KERNEL);
@@ -1103,7 +1103,7 @@
add_token_u8(&err, dev, OPAL_ENDLIST);
add_token_u8(&err, dev, OPAL_ENDLIST);
if (err) {
- pr_err("Error building get active key command\n");
+ pr_debug("Error building get active key command\n");
return err;
}
@@ -1159,7 +1159,7 @@
err = generic_lr_enable_disable(dev, uid, !!setup->RLE, !!setup->WLE,
0, 0);
if (err)
- pr_err("Failed to create enable global lr command\n");
+ pr_debug("Failed to create enable global lr command\n");
return err;
}
@@ -1217,7 +1217,7 @@
}
if (err) {
- pr_err("Error building Setup Locking range command.\n");
+ pr_debug("Error building Setup Locking range command.\n");
return err;
}
@@ -1234,11 +1234,8 @@
u32 hsn;
int err = 0;
- if (key == NULL && auth != OPAL_ANYBODY_UID) {
- pr_err("%s: Attempted to open ADMIN_SP Session without a Host" \
- "Challenge, and not as the Anybody UID\n", __func__);
+ if (key == NULL && auth != OPAL_ANYBODY_UID)
return OPAL_INVAL_PARAM;
- }
clear_opal_cmd(dev);
@@ -1273,12 +1270,12 @@
add_token_u8(&err, dev, OPAL_ENDLIST);
break;
default:
- pr_err("Cannot start Admin SP session with auth %d\n", auth);
+ pr_debug("Cannot start Admin SP session with auth %d\n", auth);
return OPAL_INVAL_PARAM;
}
if (err) {
- pr_err("Error building start adminsp session command.\n");
+ pr_debug("Error building start adminsp session command.\n");
return err;
}
@@ -1369,7 +1366,7 @@
add_token_u8(&err, dev, OPAL_ENDLIST);
if (err) {
- pr_err("Error building STARTSESSION command.\n");
+ pr_debug("Error building STARTSESSION command.\n");
return err;
}
@@ -1391,7 +1388,7 @@
add_token_u8(&err, dev, OPAL_STARTLIST);
add_token_u8(&err, dev, OPAL_ENDLIST);
if (err) {
- pr_err("Error building REVERT TPER command.\n");
+ pr_debug("Error building REVERT TPER command.\n");
return err;
}
@@ -1426,7 +1423,7 @@
add_token_u8(&err, dev, OPAL_ENDLIST);
if (err) {
- pr_err("Error building Activate UserN command.\n");
+ pr_debug("Error building Activate UserN command.\n");
return err;
}
@@ -1453,7 +1450,7 @@
add_token_u8(&err, dev, OPAL_ENDLIST);
if (err) {
- pr_err("Error building Erase Locking Range Command.\n");
+ pr_debug("Error building Erase Locking Range Command.\n");
return err;
}
return finalize_and_send(dev, parse_and_check_status);
@@ -1484,7 +1481,7 @@
add_token_u8(&err, dev, OPAL_ENDLIST);
if (err) {
- pr_err("Error Building set MBR Done command\n");
+ pr_debug("Error Building set MBR Done command\n");
return err;
}
@@ -1516,7 +1513,7 @@
add_token_u8(&err, dev, OPAL_ENDLIST);
if (err) {
- pr_err("Error Building set MBR done command\n");
+ pr_debug("Error Building set MBR done command\n");
return err;
}
@@ -1567,7 +1564,7 @@
if (generic_pw_cmd(usr->opal_key.key, usr->opal_key.key_len,
cpin_uid, dev)) {
- pr_err("Error building set password command.\n");
+ pr_debug("Error building set password command.\n");
return -ERANGE;
}
@@ -1582,7 +1579,7 @@
memcpy(cpin_uid, opaluid[OPAL_C_PIN_SID], OPAL_UID_LENGTH);
if (generic_pw_cmd(key->key, key->key_len, cpin_uid, dev)) {
- pr_err("Error building Set SID cpin\n");
+ pr_debug("Error building Set SID cpin\n");
return -ERANGE;
}
return finalize_and_send(dev, parse_and_check_status);
@@ -1657,7 +1654,7 @@
add_token_u8(&err, dev, OPAL_ENDLIST);
if (err) {
- pr_err("Error building add user to locking range command.\n");
+ pr_debug("Error building add user to locking range command.\n");
return err;
}
@@ -1691,7 +1688,7 @@
/* vars are initalized to locked */
break;
default:
- pr_err("Tried to set an invalid locking state... returning to uland\n");
+ pr_debug("Tried to set an invalid locking state... returning to uland\n");
return OPAL_INVAL_PARAM;
}
@@ -1718,7 +1715,7 @@
add_token_u8(&err, dev, OPAL_ENDLIST);
if (err) {
- pr_err("Error building SET command.\n");
+ pr_debug("Error building SET command.\n");
return err;
}
return finalize_and_send(dev, parse_and_check_status);
@@ -1752,14 +1749,14 @@
/* vars are initalized to locked */
break;
default:
- pr_err("Tried to set an invalid locking state.\n");
+ pr_debug("Tried to set an invalid locking state.\n");
return OPAL_INVAL_PARAM;
}
ret = generic_lr_enable_disable(dev, lr_buffer, 1, 1,
read_locked, write_locked);
if (ret < 0) {
- pr_err("Error building SET command.\n");
+ pr_debug("Error building SET command.\n");
return ret;
}
return finalize_and_send(dev, parse_and_check_status);
@@ -1811,7 +1808,7 @@
}
if (err) {
- pr_err("Error building Activate LockingSP command.\n");
+ pr_debug("Error building Activate LockingSP command.\n");
return err;
}
@@ -1831,7 +1828,7 @@
/* 0x08 is Manufacured Inactive */
/* 0x09 is Manufactured */
if (lc_status != OPAL_MANUFACTURED_INACTIVE) {
- pr_err("Couldn't determine the status of the Lifcycle state\n");
+ pr_debug("Couldn't determine the status of the Lifecycle state\n");
return -ENODEV;
}
@@ -1868,7 +1865,7 @@
add_token_u8(&err, dev, OPAL_ENDLIST);
if (err) {
- pr_err("Error Building GET Lifecycle Status command\n");
+ pr_debug("Error Building GET Lifecycle Status command\n");
return err;
}
@@ -1887,7 +1884,7 @@
strlen = response_get_string(&dev->parsed, 4, &msid_pin);
if (!msid_pin) {
- pr_err("%s: Couldn't extract PIN from response\n", __func__);
+ pr_debug("%s: Couldn't extract PIN from response\n", __func__);
return OPAL_INVAL_PARAM;
}
@@ -1929,7 +1926,7 @@
add_token_u8(&err, dev, OPAL_ENDLIST);
if (err) {
- pr_err("Error building Get MSID CPIN PIN command.\n");
+ pr_debug("Error building Get MSID CPIN PIN command.\n");
return err;
}
@@ -2124,18 +2121,18 @@
if (lk_unlk->l_state != OPAL_RO &&
lk_unlk->l_state != OPAL_RW) {
- pr_err("Locking state was not RO or RW\n");
+ pr_debug("Locking state was not RO or RW\n");
return -EINVAL;
}
if (lk_unlk->session.who < OPAL_USER1 ||
lk_unlk->session.who > OPAL_USER9) {
- pr_err("Authority was not within the range of users: %d\n",
- lk_unlk->session.who);
+ pr_debug("Authority was not within the range of users: %d\n",
+ lk_unlk->session.who);
return -EINVAL;
}
if (lk_unlk->session.sum) {
- pr_err("%s not supported in sum. Use setup locking range\n",
- __func__);
+ pr_debug("%s not supported in sum. Use setup locking range\n",
+ __func__);
return -EINVAL;
}
@@ -2312,7 +2309,7 @@
/* We can't activate Admin1 it's active as manufactured */
if (opal_session->who < OPAL_USER1 ||
opal_session->who > OPAL_USER9) {
- pr_err("Who was not a valid user: %d\n", opal_session->who);
+ pr_debug("Who was not a valid user: %d\n", opal_session->who);
return -EINVAL;
}
@@ -2343,9 +2340,9 @@
ret = __opal_lock_unlock(dev, &suspend->unlk);
if (ret) {
- pr_warn("Failed to unlock LR %hhu with sum %d\n",
- suspend->unlk.session.opal_key.lr,
- suspend->unlk.session.sum);
+ pr_debug("Failed to unlock LR %hhu with sum %d\n",
+ suspend->unlk.session.opal_key.lr,
+ suspend->unlk.session.sum);
was_failure = true;
}
}
@@ -2363,10 +2360,8 @@
return -EACCES;
if (!dev)
return -ENOTSUPP;
- if (!dev->supported) {
- pr_err("Not supported\n");
+ if (!dev->supported)
return -ENOTSUPP;
- }
p = memdup_user(arg, _IOC_SIZE(cmd));
if (IS_ERR(p))
@@ -2410,7 +2405,7 @@
ret = opal_secure_erase_locking_range(dev, p);
break;
default:
- pr_warn("No such Opal Ioctl %u\n", cmd);
+ break;
}
kfree(p);
diff --git a/block/t10-pi.c b/block/t10-pi.c
index 2c97912..680c6d6 100644
--- a/block/t10-pi.c
+++ b/block/t10-pi.c
@@ -160,28 +160,28 @@
return t10_pi_verify(iter, t10_pi_ip_fn, 3);
}
-struct blk_integrity_profile t10_pi_type1_crc = {
+const struct blk_integrity_profile t10_pi_type1_crc = {
.name = "T10-DIF-TYPE1-CRC",
.generate_fn = t10_pi_type1_generate_crc,
.verify_fn = t10_pi_type1_verify_crc,
};
EXPORT_SYMBOL(t10_pi_type1_crc);
-struct blk_integrity_profile t10_pi_type1_ip = {
+const struct blk_integrity_profile t10_pi_type1_ip = {
.name = "T10-DIF-TYPE1-IP",
.generate_fn = t10_pi_type1_generate_ip,
.verify_fn = t10_pi_type1_verify_ip,
};
EXPORT_SYMBOL(t10_pi_type1_ip);
-struct blk_integrity_profile t10_pi_type3_crc = {
+const struct blk_integrity_profile t10_pi_type3_crc = {
.name = "T10-DIF-TYPE3-CRC",
.generate_fn = t10_pi_type3_generate_crc,
.verify_fn = t10_pi_type3_verify_crc,
};
EXPORT_SYMBOL(t10_pi_type3_crc);
-struct blk_integrity_profile t10_pi_type3_ip = {
+const struct blk_integrity_profile t10_pi_type3_ip = {
.name = "T10-DIF-TYPE3-IP",
.generate_fn = t10_pi_type3_generate_ip,
.verify_fn = t10_pi_type3_verify_ip,
diff --git a/crypto/Kconfig b/crypto/Kconfig
index f37e9cc..aac4bc90 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -374,7 +374,6 @@
tristate "XTS support"
select CRYPTO_BLKCIPHER
select CRYPTO_MANAGER
- select CRYPTO_GF128MUL
select CRYPTO_ECB
help
XTS: IEEE1619/D16 narrow block cipher use with aes-xts-plain,
@@ -513,6 +512,23 @@
'crct10dif-plcmul' module, which is faster when computing the
crct10dif checksum as compared with the generic table implementation.
+config CRYPTO_CRCT10DIF_VPMSUM
+ tristate "CRC32T10DIF powerpc64 hardware acceleration"
+ depends on PPC64 && ALTIVEC && CRC_T10DIF
+ select CRYPTO_HASH
+ help
+ CRC10T10DIF algorithm implemented using vector polynomial
+ multiply-sum (vpmsum) instructions, introduced in POWER8. Enable on
+ POWER8 and newer processors for improved performance.
+
+config CRYPTO_VPMSUM_TESTER
+ tristate "Powerpc64 vpmsum hardware acceleration tester"
+ depends on CRYPTO_CRCT10DIF_VPMSUM && CRYPTO_CRC32C_VPMSUM
+ help
+ Stress test for CRC32c and CRC-T10DIF algorithms implemented with
+ POWER8 vpmsum instructions.
+ Unless you are testing these algorithms, you don't need this.
+
config CRYPTO_GHASH
tristate "GHASH digest algorithm"
select CRYPTO_GF128MUL
diff --git a/crypto/acompress.c b/crypto/acompress.c
index 47d1162..1544b7c 100644
--- a/crypto/acompress.c
+++ b/crypto/acompress.c
@@ -166,5 +166,34 @@
}
EXPORT_SYMBOL_GPL(crypto_unregister_acomp);
+int crypto_register_acomps(struct acomp_alg *algs, int count)
+{
+ int i, ret;
+
+ for (i = 0; i < count; i++) {
+ ret = crypto_register_acomp(&algs[i]);
+ if (ret)
+ goto err;
+ }
+
+ return 0;
+
+err:
+ for (--i; i >= 0; --i)
+ crypto_unregister_acomp(&algs[i]);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(crypto_register_acomps);
+
+void crypto_unregister_acomps(struct acomp_alg *algs, int count)
+{
+ int i;
+
+ for (i = count - 1; i >= 0; --i)
+ crypto_unregister_acomp(&algs[i]);
+}
+EXPORT_SYMBOL_GPL(crypto_unregister_acomps);
+
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Asynchronous compression type");
diff --git a/crypto/af_alg.c b/crypto/af_alg.c
index 690deca..3556d8e 100644
--- a/crypto/af_alg.c
+++ b/crypto/af_alg.c
@@ -160,11 +160,11 @@
if (sock->state == SS_CONNECTED)
return -EINVAL;
- if (addr_len != sizeof(*sa))
+ if (addr_len < sizeof(*sa))
return -EINVAL;
sa->salg_type[sizeof(sa->salg_type) - 1] = 0;
- sa->salg_name[sizeof(sa->salg_name) - 1] = 0;
+ sa->salg_name[sizeof(sa->salg_name) + addr_len - sizeof(*sa) - 1] = 0;
type = alg_get_type(sa->salg_type);
if (IS_ERR(type) && PTR_ERR(type) == -ENOENT) {
diff --git a/crypto/algapi.c b/crypto/algapi.c
index 6b52e8f..9eed4ef 100644
--- a/crypto/algapi.c
+++ b/crypto/algapi.c
@@ -963,11 +963,11 @@
u32 c;
if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
- !((unsigned long)b & (__alignof__(*b) - 1)))
+ IS_ALIGNED((unsigned long)b, __alignof__(*b)))
for (; size >= 4; size -= 4) {
c = be32_to_cpu(*--b) + 1;
*b = cpu_to_be32(c);
- if (c)
+ if (likely(c))
return;
}
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index ef59d99..8af664f 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -45,6 +45,11 @@
char iv[];
};
+struct aead_tfm {
+ struct crypto_aead *aead;
+ bool has_key;
+};
+
struct aead_ctx {
struct aead_sg_list tsgl;
struct aead_async_rsgl first_rsgl;
@@ -723,24 +728,146 @@
.poll = aead_poll,
};
+static int aead_check_key(struct socket *sock)
+{
+ int err = 0;
+ struct sock *psk;
+ struct alg_sock *pask;
+ struct aead_tfm *tfm;
+ struct sock *sk = sock->sk;
+ struct alg_sock *ask = alg_sk(sk);
+
+ lock_sock(sk);
+ if (ask->refcnt)
+ goto unlock_child;
+
+ psk = ask->parent;
+ pask = alg_sk(ask->parent);
+ tfm = pask->private;
+
+ err = -ENOKEY;
+ lock_sock_nested(psk, SINGLE_DEPTH_NESTING);
+ if (!tfm->has_key)
+ goto unlock;
+
+ if (!pask->refcnt++)
+ sock_hold(psk);
+
+ ask->refcnt = 1;
+ sock_put(psk);
+
+ err = 0;
+
+unlock:
+ release_sock(psk);
+unlock_child:
+ release_sock(sk);
+
+ return err;
+}
+
+static int aead_sendmsg_nokey(struct socket *sock, struct msghdr *msg,
+ size_t size)
+{
+ int err;
+
+ err = aead_check_key(sock);
+ if (err)
+ return err;
+
+ return aead_sendmsg(sock, msg, size);
+}
+
+static ssize_t aead_sendpage_nokey(struct socket *sock, struct page *page,
+ int offset, size_t size, int flags)
+{
+ int err;
+
+ err = aead_check_key(sock);
+ if (err)
+ return err;
+
+ return aead_sendpage(sock, page, offset, size, flags);
+}
+
+static int aead_recvmsg_nokey(struct socket *sock, struct msghdr *msg,
+ size_t ignored, int flags)
+{
+ int err;
+
+ err = aead_check_key(sock);
+ if (err)
+ return err;
+
+ return aead_recvmsg(sock, msg, ignored, flags);
+}
+
+static struct proto_ops algif_aead_ops_nokey = {
+ .family = PF_ALG,
+
+ .connect = sock_no_connect,
+ .socketpair = sock_no_socketpair,
+ .getname = sock_no_getname,
+ .ioctl = sock_no_ioctl,
+ .listen = sock_no_listen,
+ .shutdown = sock_no_shutdown,
+ .getsockopt = sock_no_getsockopt,
+ .mmap = sock_no_mmap,
+ .bind = sock_no_bind,
+ .accept = sock_no_accept,
+ .setsockopt = sock_no_setsockopt,
+
+ .release = af_alg_release,
+ .sendmsg = aead_sendmsg_nokey,
+ .sendpage = aead_sendpage_nokey,
+ .recvmsg = aead_recvmsg_nokey,
+ .poll = aead_poll,
+};
+
static void *aead_bind(const char *name, u32 type, u32 mask)
{
- return crypto_alloc_aead(name, type, mask);
+ struct aead_tfm *tfm;
+ struct crypto_aead *aead;
+
+ tfm = kzalloc(sizeof(*tfm), GFP_KERNEL);
+ if (!tfm)
+ return ERR_PTR(-ENOMEM);
+
+ aead = crypto_alloc_aead(name, type, mask);
+ if (IS_ERR(aead)) {
+ kfree(tfm);
+ return ERR_CAST(aead);
+ }
+
+ tfm->aead = aead;
+
+ return tfm;
}
static void aead_release(void *private)
{
- crypto_free_aead(private);
+ struct aead_tfm *tfm = private;
+
+ crypto_free_aead(tfm->aead);
+ kfree(tfm);
}
static int aead_setauthsize(void *private, unsigned int authsize)
{
- return crypto_aead_setauthsize(private, authsize);
+ struct aead_tfm *tfm = private;
+
+ return crypto_aead_setauthsize(tfm->aead, authsize);
}
static int aead_setkey(void *private, const u8 *key, unsigned int keylen)
{
- return crypto_aead_setkey(private, key, keylen);
+ struct aead_tfm *tfm = private;
+ int err;
+
+ err = crypto_aead_setkey(tfm->aead, key, keylen);
+ tfm->has_key = !err;
+
+ return err;
}
static void aead_sock_destruct(struct sock *sk)
@@ -757,12 +884,14 @@
af_alg_release_parent(sk);
}
-static int aead_accept_parent(void *private, struct sock *sk)
+static int aead_accept_parent_nokey(void *private, struct sock *sk)
{
struct aead_ctx *ctx;
struct alg_sock *ask = alg_sk(sk);
- unsigned int len = sizeof(*ctx) + crypto_aead_reqsize(private);
- unsigned int ivlen = crypto_aead_ivsize(private);
+ struct aead_tfm *tfm = private;
+ struct crypto_aead *aead = tfm->aead;
+ unsigned int len = sizeof(*ctx) + crypto_aead_reqsize(aead);
+ unsigned int ivlen = crypto_aead_ivsize(aead);
ctx = sock_kmalloc(sk, len, GFP_KERNEL);
if (!ctx)
@@ -789,7 +918,7 @@
ask->private = ctx;
- aead_request_set_tfm(&ctx->aead_req, private);
+ aead_request_set_tfm(&ctx->aead_req, aead);
aead_request_set_callback(&ctx->aead_req, CRYPTO_TFM_REQ_MAY_BACKLOG,
af_alg_complete, &ctx->completion);
@@ -798,13 +927,25 @@
return 0;
}
+static int aead_accept_parent(void *private, struct sock *sk)
+{
+ struct aead_tfm *tfm = private;
+
+ if (!tfm->has_key)
+ return -ENOKEY;
+
+ return aead_accept_parent_nokey(private, sk);
+}
+
static const struct af_alg_type algif_type_aead = {
.bind = aead_bind,
.release = aead_release,
.setkey = aead_setkey,
.setauthsize = aead_setauthsize,
.accept = aead_accept_parent,
+ .accept_nokey = aead_accept_parent_nokey,
.ops = &algif_aead_ops,
+ .ops_nokey = &algif_aead_ops_nokey,
.name = "aead",
.owner = THIS_MODULE
};
diff --git a/crypto/cbc.c b/crypto/cbc.c
index bc160a3..b761b1f 100644
--- a/crypto/cbc.c
+++ b/crypto/cbc.c
@@ -10,6 +10,7 @@
*
*/
+#include <crypto/algapi.h>
#include <crypto/cbc.h>
#include <crypto/internal/skcipher.h>
#include <linux/err.h>
@@ -108,8 +109,10 @@
static int crypto_cbc_create(struct crypto_template *tmpl, struct rtattr **tb)
{
struct skcipher_instance *inst;
+ struct crypto_attr_type *algt;
struct crypto_spawn *spawn;
struct crypto_alg *alg;
+ u32 mask;
int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_SKCIPHER);
@@ -120,8 +123,16 @@
if (!inst)
return -ENOMEM;
- alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER,
- CRYPTO_ALG_TYPE_MASK);
+ algt = crypto_get_attr_type(tb);
+ err = PTR_ERR(algt);
+ if (IS_ERR(algt))
+ goto err_free_inst;
+
+ mask = CRYPTO_ALG_TYPE_MASK |
+ crypto_requires_off(algt->type, algt->mask,
+ CRYPTO_ALG_NEED_FALLBACK);
+
+ alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER, mask);
err = PTR_ERR(alg);
if (IS_ERR(alg))
goto err_free_inst;
diff --git a/crypto/crypto_user.c b/crypto/crypto_user.c
index a90404a..0dbe2be7 100644
--- a/crypto/crypto_user.c
+++ b/crypto/crypto_user.c
@@ -83,7 +83,7 @@
{
struct crypto_report_cipher rcipher;
- strncpy(rcipher.type, "cipher", sizeof(rcipher.type));
+ strlcpy(rcipher.type, "cipher", sizeof(rcipher.type));
rcipher.blocksize = alg->cra_blocksize;
rcipher.min_keysize = alg->cra_cipher.cia_min_keysize;
@@ -102,7 +102,7 @@
{
struct crypto_report_comp rcomp;
- strncpy(rcomp.type, "compression", sizeof(rcomp.type));
+ strlcpy(rcomp.type, "compression", sizeof(rcomp.type));
if (nla_put(skb, CRYPTOCFGA_REPORT_COMPRESS,
sizeof(struct crypto_report_comp), &rcomp))
goto nla_put_failure;
@@ -116,7 +116,7 @@
{
struct crypto_report_acomp racomp;
- strncpy(racomp.type, "acomp", sizeof(racomp.type));
+ strlcpy(racomp.type, "acomp", sizeof(racomp.type));
if (nla_put(skb, CRYPTOCFGA_REPORT_ACOMP,
sizeof(struct crypto_report_acomp), &racomp))
@@ -131,7 +131,7 @@
{
struct crypto_report_akcipher rakcipher;
- strncpy(rakcipher.type, "akcipher", sizeof(rakcipher.type));
+ strlcpy(rakcipher.type, "akcipher", sizeof(rakcipher.type));
if (nla_put(skb, CRYPTOCFGA_REPORT_AKCIPHER,
sizeof(struct crypto_report_akcipher), &rakcipher))
@@ -146,7 +146,7 @@
{
struct crypto_report_kpp rkpp;
- strncpy(rkpp.type, "kpp", sizeof(rkpp.type));
+ strlcpy(rkpp.type, "kpp", sizeof(rkpp.type));
if (nla_put(skb, CRYPTOCFGA_REPORT_KPP,
sizeof(struct crypto_report_kpp), &rkpp))
@@ -160,10 +160,10 @@
static int crypto_report_one(struct crypto_alg *alg,
struct crypto_user_alg *ualg, struct sk_buff *skb)
{
- strncpy(ualg->cru_name, alg->cra_name, sizeof(ualg->cru_name));
- strncpy(ualg->cru_driver_name, alg->cra_driver_name,
+ strlcpy(ualg->cru_name, alg->cra_name, sizeof(ualg->cru_name));
+ strlcpy(ualg->cru_driver_name, alg->cra_driver_name,
sizeof(ualg->cru_driver_name));
- strncpy(ualg->cru_module_name, module_name(alg->cra_module),
+ strlcpy(ualg->cru_module_name, module_name(alg->cra_module),
sizeof(ualg->cru_module_name));
ualg->cru_type = 0;
@@ -176,7 +176,7 @@
if (alg->cra_flags & CRYPTO_ALG_LARVAL) {
struct crypto_report_larval rl;
- strncpy(rl.type, "larval", sizeof(rl.type));
+ strlcpy(rl.type, "larval", sizeof(rl.type));
if (nla_put(skb, CRYPTOCFGA_REPORT_LARVAL,
sizeof(struct crypto_report_larval), &rl))
goto nla_put_failure;
@@ -483,7 +483,8 @@
[CRYPTO_MSG_DELRNG - CRYPTO_MSG_BASE] = { .doit = crypto_del_rng },
};
-static int crypto_user_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int crypto_user_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct nlattr *attrs[CRYPTOCFGA_MAX+1];
const struct crypto_link *link;
@@ -522,7 +523,7 @@
}
err = nlmsg_parse(nlh, crypto_msg_min[type], attrs, CRYPTOCFGA_MAX,
- crypto_policy);
+ crypto_policy, extack);
if (err < 0)
return err;
diff --git a/crypto/ctr.c b/crypto/ctr.c
index a4f4a89..477d922 100644
--- a/crypto/ctr.c
+++ b/crypto/ctr.c
@@ -181,15 +181,24 @@
static struct crypto_instance *crypto_ctr_alloc(struct rtattr **tb)
{
struct crypto_instance *inst;
+ struct crypto_attr_type *algt;
struct crypto_alg *alg;
+ u32 mask;
int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_BLKCIPHER);
if (err)
return ERR_PTR(err);
- alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER,
- CRYPTO_ALG_TYPE_MASK);
+ algt = crypto_get_attr_type(tb);
+ if (IS_ERR(algt))
+ return ERR_CAST(algt);
+
+ mask = CRYPTO_ALG_TYPE_MASK |
+ crypto_requires_off(algt->type, algt->mask,
+ CRYPTO_ALG_NEED_FALLBACK);
+
+ alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, mask);
if (IS_ERR(alg))
return ERR_CAST(alg);
@@ -350,6 +359,8 @@
struct skcipher_alg *alg;
struct crypto_skcipher_spawn *spawn;
const char *cipher_name;
+ u32 mask;
+
int err;
algt = crypto_get_attr_type(tb);
@@ -367,12 +378,14 @@
if (!inst)
return -ENOMEM;
+ mask = crypto_requires_sync(algt->type, algt->mask) |
+ crypto_requires_off(algt->type, algt->mask,
+ CRYPTO_ALG_NEED_FALLBACK);
+
spawn = skcipher_instance_ctx(inst);
crypto_set_skcipher_spawn(spawn, skcipher_crypto_instance(inst));
- err = crypto_grab_skcipher(spawn, cipher_name, 0,
- crypto_requires_sync(algt->type,
- algt->mask));
+ err = crypto_grab_skcipher(spawn, cipher_name, 0, mask);
if (err)
goto err_free_inst;
diff --git a/crypto/deflate.c b/crypto/deflate.c
index f942cb3..94ec3b3 100644
--- a/crypto/deflate.c
+++ b/crypto/deflate.c
@@ -43,20 +43,24 @@
struct z_stream_s decomp_stream;
};
-static int deflate_comp_init(struct deflate_ctx *ctx)
+static int deflate_comp_init(struct deflate_ctx *ctx, int format)
{
int ret = 0;
struct z_stream_s *stream = &ctx->comp_stream;
stream->workspace = vzalloc(zlib_deflate_workspacesize(
- -DEFLATE_DEF_WINBITS, DEFLATE_DEF_MEMLEVEL));
+ MAX_WBITS, MAX_MEM_LEVEL));
if (!stream->workspace) {
ret = -ENOMEM;
goto out;
}
- ret = zlib_deflateInit2(stream, DEFLATE_DEF_LEVEL, Z_DEFLATED,
- -DEFLATE_DEF_WINBITS, DEFLATE_DEF_MEMLEVEL,
- Z_DEFAULT_STRATEGY);
+ if (format)
+ ret = zlib_deflateInit(stream, 3);
+ else
+ ret = zlib_deflateInit2(stream, DEFLATE_DEF_LEVEL, Z_DEFLATED,
+ -DEFLATE_DEF_WINBITS,
+ DEFLATE_DEF_MEMLEVEL,
+ Z_DEFAULT_STRATEGY);
if (ret != Z_OK) {
ret = -EINVAL;
goto out_free;
@@ -68,7 +72,7 @@
goto out;
}
-static int deflate_decomp_init(struct deflate_ctx *ctx)
+static int deflate_decomp_init(struct deflate_ctx *ctx, int format)
{
int ret = 0;
struct z_stream_s *stream = &ctx->decomp_stream;
@@ -78,7 +82,10 @@
ret = -ENOMEM;
goto out;
}
- ret = zlib_inflateInit2(stream, -DEFLATE_DEF_WINBITS);
+ if (format)
+ ret = zlib_inflateInit(stream);
+ else
+ ret = zlib_inflateInit2(stream, -DEFLATE_DEF_WINBITS);
if (ret != Z_OK) {
ret = -EINVAL;
goto out_free;
@@ -102,21 +109,21 @@
vfree(ctx->decomp_stream.workspace);
}
-static int __deflate_init(void *ctx)
+static int __deflate_init(void *ctx, int format)
{
int ret;
- ret = deflate_comp_init(ctx);
+ ret = deflate_comp_init(ctx, format);
if (ret)
goto out;
- ret = deflate_decomp_init(ctx);
+ ret = deflate_decomp_init(ctx, format);
if (ret)
deflate_comp_exit(ctx);
out:
return ret;
}
-static void *deflate_alloc_ctx(struct crypto_scomp *tfm)
+static void *gen_deflate_alloc_ctx(struct crypto_scomp *tfm, int format)
{
struct deflate_ctx *ctx;
int ret;
@@ -125,7 +132,7 @@
if (!ctx)
return ERR_PTR(-ENOMEM);
- ret = __deflate_init(ctx);
+ ret = __deflate_init(ctx, format);
if (ret) {
kfree(ctx);
return ERR_PTR(ret);
@@ -134,11 +141,21 @@
return ctx;
}
+static void *deflate_alloc_ctx(struct crypto_scomp *tfm)
+{
+ return gen_deflate_alloc_ctx(tfm, 0);
+}
+
+static void *zlib_deflate_alloc_ctx(struct crypto_scomp *tfm)
+{
+ return gen_deflate_alloc_ctx(tfm, 1);
+}
+
static int deflate_init(struct crypto_tfm *tfm)
{
struct deflate_ctx *ctx = crypto_tfm_ctx(tfm);
- return __deflate_init(ctx);
+ return __deflate_init(ctx, 0);
}
static void __deflate_exit(void *ctx)
@@ -272,7 +289,7 @@
.coa_decompress = deflate_decompress } }
};
-static struct scomp_alg scomp = {
+static struct scomp_alg scomp[] = { {
.alloc_ctx = deflate_alloc_ctx,
.free_ctx = deflate_free_ctx,
.compress = deflate_scompress,
@@ -282,7 +299,17 @@
.cra_driver_name = "deflate-scomp",
.cra_module = THIS_MODULE,
}
-};
+}, {
+ .alloc_ctx = zlib_deflate_alloc_ctx,
+ .free_ctx = deflate_free_ctx,
+ .compress = deflate_scompress,
+ .decompress = deflate_sdecompress,
+ .base = {
+ .cra_name = "zlib-deflate",
+ .cra_driver_name = "zlib-deflate-scomp",
+ .cra_module = THIS_MODULE,
+ }
+} };
static int __init deflate_mod_init(void)
{
@@ -292,7 +319,7 @@
if (ret)
return ret;
- ret = crypto_register_scomp(&scomp);
+ ret = crypto_register_scomps(scomp, ARRAY_SIZE(scomp));
if (ret) {
crypto_unregister_alg(&alg);
return ret;
@@ -304,7 +331,7 @@
static void __exit deflate_mod_fini(void)
{
crypto_unregister_alg(&alg);
- crypto_unregister_scomp(&scomp);
+ crypto_unregister_scomps(scomp, ARRAY_SIZE(scomp));
}
module_init(deflate_mod_init);
diff --git a/crypto/dh.c b/crypto/dh.c
index ddcb528..87e3542 100644
--- a/crypto/dh.c
+++ b/crypto/dh.c
@@ -79,7 +79,8 @@
return 0;
}
-static int dh_set_secret(struct crypto_kpp *tfm, void *buf, unsigned int len)
+static int dh_set_secret(struct crypto_kpp *tfm, const void *buf,
+ unsigned int len)
{
struct dh_ctx *ctx = dh_get_ctx(tfm);
struct dh params;
diff --git a/crypto/drbg.c b/crypto/drbg.c
index 8a4d98b..fa749f4 100644
--- a/crypto/drbg.c
+++ b/crypto/drbg.c
@@ -1749,17 +1749,16 @@
u8 *inbuf, u32 inlen,
u8 *outbuf, u32 outlen)
{
- struct scatterlist sg_in;
+ struct scatterlist sg_in, sg_out;
int ret;
sg_init_one(&sg_in, inbuf, inlen);
+ sg_init_one(&sg_out, drbg->outscratchpad, DRBG_OUTSCRATCHLEN);
while (outlen) {
u32 cryptlen = min3(inlen, outlen, (u32)DRBG_OUTSCRATCHLEN);
- struct scatterlist sg_out;
/* Output buffer may not be valid for SGL, use scratchpad */
- sg_init_one(&sg_out, drbg->outscratchpad, cryptlen);
skcipher_request_set_crypt(drbg->ctr_req, &sg_in, &sg_out,
cryptlen, drbg->V);
ret = crypto_skcipher_encrypt(drbg->ctr_req);
diff --git a/crypto/ecdh.c b/crypto/ecdh.c
index 3de2898..63ca337 100644
--- a/crypto/ecdh.c
+++ b/crypto/ecdh.c
@@ -38,7 +38,8 @@
}
}
-static int ecdh_set_secret(struct crypto_kpp *tfm, void *buf, unsigned int len)
+static int ecdh_set_secret(struct crypto_kpp *tfm, const void *buf,
+ unsigned int len)
{
struct ecdh_ctx *ctx = ecdh_get_ctx(tfm);
struct ecdh params;
diff --git a/crypto/gf128mul.c b/crypto/gf128mul.c
index 72015fe..dc01212 100644
--- a/crypto/gf128mul.c
+++ b/crypto/gf128mul.c
@@ -44,7 +44,7 @@
---------------------------------------------------------------------------
Issue 31/01/2006
- This file provides fast multiplication in GF(128) as required by several
+ This file provides fast multiplication in GF(2^128) as required by several
cryptographic authentication modes
*/
@@ -88,76 +88,59 @@
q(0xf8), q(0xf9), q(0xfa), q(0xfb), q(0xfc), q(0xfd), q(0xfe), q(0xff) \
}
-/* Given the value i in 0..255 as the byte overflow when a field element
- in GHASH is multiplied by x^8, this function will return the values that
- are generated in the lo 16-bit word of the field value by applying the
- modular polynomial. The values lo_byte and hi_byte are returned via the
- macro xp_fun(lo_byte, hi_byte) so that the values can be assembled into
- memory as required by a suitable definition of this macro operating on
- the table above
-*/
-
-#define xx(p, q) 0x##p##q
-
-#define xda_bbe(i) ( \
- (i & 0x80 ? xx(43, 80) : 0) ^ (i & 0x40 ? xx(21, c0) : 0) ^ \
- (i & 0x20 ? xx(10, e0) : 0) ^ (i & 0x10 ? xx(08, 70) : 0) ^ \
- (i & 0x08 ? xx(04, 38) : 0) ^ (i & 0x04 ? xx(02, 1c) : 0) ^ \
- (i & 0x02 ? xx(01, 0e) : 0) ^ (i & 0x01 ? xx(00, 87) : 0) \
-)
-
-#define xda_lle(i) ( \
- (i & 0x80 ? xx(e1, 00) : 0) ^ (i & 0x40 ? xx(70, 80) : 0) ^ \
- (i & 0x20 ? xx(38, 40) : 0) ^ (i & 0x10 ? xx(1c, 20) : 0) ^ \
- (i & 0x08 ? xx(0e, 10) : 0) ^ (i & 0x04 ? xx(07, 08) : 0) ^ \
- (i & 0x02 ? xx(03, 84) : 0) ^ (i & 0x01 ? xx(01, c2) : 0) \
-)
-
-static const u16 gf128mul_table_lle[256] = gf128mul_dat(xda_lle);
-static const u16 gf128mul_table_bbe[256] = gf128mul_dat(xda_bbe);
-
-/* These functions multiply a field element by x, by x^4 and by x^8
- * in the polynomial field representation. It uses 32-bit word operations
- * to gain speed but compensates for machine endianess and hence works
- * correctly on both styles of machine.
+/*
+ * Given a value i in 0..255 as the byte overflow when a field element
+ * in GF(2^128) is multiplied by x^8, the following macro returns the
+ * 16-bit value that must be XOR-ed into the low-degree end of the
+ * product to reduce it modulo the polynomial x^128 + x^7 + x^2 + x + 1.
+ *
+ * There are two versions of the macro, and hence two tables: one for
+ * the "be" convention where the highest-order bit is the coefficient of
+ * the highest-degree polynomial term, and one for the "le" convention
+ * where the highest-order bit is the coefficient of the lowest-degree
+ * polynomial term. In both cases the values are stored in CPU byte
+ * endianness such that the coefficients are ordered consistently across
+ * bytes, i.e. in the "be" table bits 15..0 of the stored value
+ * correspond to the coefficients of x^15..x^0, and in the "le" table
+ * bits 15..0 correspond to the coefficients of x^0..x^15.
+ *
+ * Therefore, provided that the appropriate byte endianness conversions
+ * are done by the multiplication functions (and these must be in place
+ * anyway to support both little endian and big endian CPUs), the "be"
+ * table can be used for multiplications of both "bbe" and "ble"
+ * elements, and the "le" table can be used for multiplications of both
+ * "lle" and "lbe" elements.
*/
-static void gf128mul_x_lle(be128 *r, const be128 *x)
-{
- u64 a = be64_to_cpu(x->a);
- u64 b = be64_to_cpu(x->b);
- u64 _tt = gf128mul_table_lle[(b << 7) & 0xff];
+#define xda_be(i) ( \
+ (i & 0x80 ? 0x4380 : 0) ^ (i & 0x40 ? 0x21c0 : 0) ^ \
+ (i & 0x20 ? 0x10e0 : 0) ^ (i & 0x10 ? 0x0870 : 0) ^ \
+ (i & 0x08 ? 0x0438 : 0) ^ (i & 0x04 ? 0x021c : 0) ^ \
+ (i & 0x02 ? 0x010e : 0) ^ (i & 0x01 ? 0x0087 : 0) \
+)
- r->b = cpu_to_be64((b >> 1) | (a << 63));
- r->a = cpu_to_be64((a >> 1) ^ (_tt << 48));
-}
+#define xda_le(i) ( \
+ (i & 0x80 ? 0xe100 : 0) ^ (i & 0x40 ? 0x7080 : 0) ^ \
+ (i & 0x20 ? 0x3840 : 0) ^ (i & 0x10 ? 0x1c20 : 0) ^ \
+ (i & 0x08 ? 0x0e10 : 0) ^ (i & 0x04 ? 0x0708 : 0) ^ \
+ (i & 0x02 ? 0x0384 : 0) ^ (i & 0x01 ? 0x01c2 : 0) \
+)
-static void gf128mul_x_bbe(be128 *r, const be128 *x)
-{
- u64 a = be64_to_cpu(x->a);
- u64 b = be64_to_cpu(x->b);
- u64 _tt = gf128mul_table_bbe[a >> 63];
+static const u16 gf128mul_table_le[256] = gf128mul_dat(xda_le);
+static const u16 gf128mul_table_be[256] = gf128mul_dat(xda_be);
- r->a = cpu_to_be64((a << 1) | (b >> 63));
- r->b = cpu_to_be64((b << 1) ^ _tt);
-}
-
-void gf128mul_x_ble(be128 *r, const be128 *x)
-{
- u64 a = le64_to_cpu(x->a);
- u64 b = le64_to_cpu(x->b);
- u64 _tt = gf128mul_table_bbe[b >> 63];
-
- r->a = cpu_to_le64((a << 1) ^ _tt);
- r->b = cpu_to_le64((b << 1) | (a >> 63));
-}
-EXPORT_SYMBOL(gf128mul_x_ble);
+/*
+ * The following functions multiply a field element by x^8 in
+ * the polynomial field representation. They use 64-bit word operations
+ * to gain speed but compensate for machine endianness and hence work
+ * correctly on both styles of machine.
+ */
static void gf128mul_x8_lle(be128 *x)
{
u64 a = be64_to_cpu(x->a);
u64 b = be64_to_cpu(x->b);
- u64 _tt = gf128mul_table_lle[b & 0xff];
+ u64 _tt = gf128mul_table_le[b & 0xff];
x->b = cpu_to_be64((b >> 8) | (a << 56));
x->a = cpu_to_be64((a >> 8) ^ (_tt << 48));
@@ -167,7 +150,7 @@
{
u64 a = be64_to_cpu(x->a);
u64 b = be64_to_cpu(x->b);
- u64 _tt = gf128mul_table_bbe[a >> 56];
+ u64 _tt = gf128mul_table_be[a >> 56];
x->a = cpu_to_be64((a << 8) | (b >> 56));
x->b = cpu_to_be64((b << 8) ^ _tt);
@@ -251,7 +234,7 @@
/* This version uses 64k bytes of table space.
A 16 byte buffer has to be multiplied by a 16 byte key
- value in GF(128). If we consider a GF(128) value in
+ value in GF(2^128). If we consider a GF(2^128) value in
the buffer's lowest byte, we can construct a table of
the 256 16 byte values that result from the 256 values
of this byte. This requires 4096 bytes. But we also
@@ -315,7 +298,7 @@
}
EXPORT_SYMBOL(gf128mul_free_64k);
-void gf128mul_64k_bbe(be128 *a, struct gf128mul_64k *t)
+void gf128mul_64k_bbe(be128 *a, const struct gf128mul_64k *t)
{
u8 *ap = (u8 *)a;
be128 r[1];
@@ -330,7 +313,7 @@
/* This version uses 4k bytes of table space.
A 16 byte buffer has to be multiplied by a 16 byte key
- value in GF(128). If we consider a GF(128) value in a
+ value in GF(2^128). If we consider a GF(2^128) value in a
single byte, we can construct a table of the 256 16 byte
values that result from the 256 values of this byte.
This requires 4096 bytes. If we take the highest byte in
@@ -388,7 +371,7 @@
}
EXPORT_SYMBOL(gf128mul_init_4k_bbe);
-void gf128mul_4k_lle(be128 *a, struct gf128mul_4k *t)
+void gf128mul_4k_lle(be128 *a, const struct gf128mul_4k *t)
{
u8 *ap = (u8 *)a;
be128 r[1];
@@ -403,7 +386,7 @@
}
EXPORT_SYMBOL(gf128mul_4k_lle);
-void gf128mul_4k_bbe(be128 *a, struct gf128mul_4k *t)
+void gf128mul_4k_bbe(be128 *a, const struct gf128mul_4k *t)
{
u8 *ap = (u8 *)a;
be128 r[1];
diff --git a/crypto/lz4.c b/crypto/lz4.c
index 71eff9b..2ce2660 100644
--- a/crypto/lz4.c
+++ b/crypto/lz4.c
@@ -97,7 +97,7 @@
int out_len = LZ4_decompress_safe(src, dst, slen, *dlen);
if (out_len < 0)
- return out_len;
+ return -EINVAL;
*dlen = out_len;
return 0;
diff --git a/crypto/lz4hc.c b/crypto/lz4hc.c
index 03a34a8..2be14f0 100644
--- a/crypto/lz4hc.c
+++ b/crypto/lz4hc.c
@@ -98,7 +98,7 @@
int out_len = LZ4_decompress_safe(src, dst, slen, *dlen);
if (out_len < 0)
- return out_len;
+ return -EINVAL;
*dlen = out_len;
return 0;
diff --git a/crypto/md5.c b/crypto/md5.c
index 2355a7c..f7ae1a4 100644
--- a/crypto/md5.c
+++ b/crypto/md5.c
@@ -21,9 +21,11 @@
#include <linux/module.h>
#include <linux/string.h>
#include <linux/types.h>
-#include <linux/cryptohash.h>
#include <asm/byteorder.h>
+#define MD5_DIGEST_WORDS 4
+#define MD5_MESSAGE_BYTES 64
+
const u8 md5_zero_message_hash[MD5_DIGEST_SIZE] = {
0xd4, 0x1d, 0x8c, 0xd9, 0x8f, 0x00, 0xb2, 0x04,
0xe9, 0x80, 0x09, 0x98, 0xec, 0xf8, 0x42, 0x7e,
@@ -47,6 +49,97 @@
}
}
+#define F1(x, y, z) (z ^ (x & (y ^ z)))
+#define F2(x, y, z) F1(z, x, y)
+#define F3(x, y, z) (x ^ y ^ z)
+#define F4(x, y, z) (y ^ (x | ~z))
+
+#define MD5STEP(f, w, x, y, z, in, s) \
+ (w += f(x, y, z) + in, w = (w<<s | w>>(32-s)) + x)
+
+static void md5_transform(__u32 *hash, __u32 const *in)
+{
+ u32 a, b, c, d;
+
+ a = hash[0];
+ b = hash[1];
+ c = hash[2];
+ d = hash[3];
+
+ MD5STEP(F1, a, b, c, d, in[0] + 0xd76aa478, 7);
+ MD5STEP(F1, d, a, b, c, in[1] + 0xe8c7b756, 12);
+ MD5STEP(F1, c, d, a, b, in[2] + 0x242070db, 17);
+ MD5STEP(F1, b, c, d, a, in[3] + 0xc1bdceee, 22);
+ MD5STEP(F1, a, b, c, d, in[4] + 0xf57c0faf, 7);
+ MD5STEP(F1, d, a, b, c, in[5] + 0x4787c62a, 12);
+ MD5STEP(F1, c, d, a, b, in[6] + 0xa8304613, 17);
+ MD5STEP(F1, b, c, d, a, in[7] + 0xfd469501, 22);
+ MD5STEP(F1, a, b, c, d, in[8] + 0x698098d8, 7);
+ MD5STEP(F1, d, a, b, c, in[9] + 0x8b44f7af, 12);
+ MD5STEP(F1, c, d, a, b, in[10] + 0xffff5bb1, 17);
+ MD5STEP(F1, b, c, d, a, in[11] + 0x895cd7be, 22);
+ MD5STEP(F1, a, b, c, d, in[12] + 0x6b901122, 7);
+ MD5STEP(F1, d, a, b, c, in[13] + 0xfd987193, 12);
+ MD5STEP(F1, c, d, a, b, in[14] + 0xa679438e, 17);
+ MD5STEP(F1, b, c, d, a, in[15] + 0x49b40821, 22);
+
+ MD5STEP(F2, a, b, c, d, in[1] + 0xf61e2562, 5);
+ MD5STEP(F2, d, a, b, c, in[6] + 0xc040b340, 9);
+ MD5STEP(F2, c, d, a, b, in[11] + 0x265e5a51, 14);
+ MD5STEP(F2, b, c, d, a, in[0] + 0xe9b6c7aa, 20);
+ MD5STEP(F2, a, b, c, d, in[5] + 0xd62f105d, 5);
+ MD5STEP(F2, d, a, b, c, in[10] + 0x02441453, 9);
+ MD5STEP(F2, c, d, a, b, in[15] + 0xd8a1e681, 14);
+ MD5STEP(F2, b, c, d, a, in[4] + 0xe7d3fbc8, 20);
+ MD5STEP(F2, a, b, c, d, in[9] + 0x21e1cde6, 5);
+ MD5STEP(F2, d, a, b, c, in[14] + 0xc33707d6, 9);
+ MD5STEP(F2, c, d, a, b, in[3] + 0xf4d50d87, 14);
+ MD5STEP(F2, b, c, d, a, in[8] + 0x455a14ed, 20);
+ MD5STEP(F2, a, b, c, d, in[13] + 0xa9e3e905, 5);
+ MD5STEP(F2, d, a, b, c, in[2] + 0xfcefa3f8, 9);
+ MD5STEP(F2, c, d, a, b, in[7] + 0x676f02d9, 14);
+ MD5STEP(F2, b, c, d, a, in[12] + 0x8d2a4c8a, 20);
+
+ MD5STEP(F3, a, b, c, d, in[5] + 0xfffa3942, 4);
+ MD5STEP(F3, d, a, b, c, in[8] + 0x8771f681, 11);
+ MD5STEP(F3, c, d, a, b, in[11] + 0x6d9d6122, 16);
+ MD5STEP(F3, b, c, d, a, in[14] + 0xfde5380c, 23);
+ MD5STEP(F3, a, b, c, d, in[1] + 0xa4beea44, 4);
+ MD5STEP(F3, d, a, b, c, in[4] + 0x4bdecfa9, 11);
+ MD5STEP(F3, c, d, a, b, in[7] + 0xf6bb4b60, 16);
+ MD5STEP(F3, b, c, d, a, in[10] + 0xbebfbc70, 23);
+ MD5STEP(F3, a, b, c, d, in[13] + 0x289b7ec6, 4);
+ MD5STEP(F3, d, a, b, c, in[0] + 0xeaa127fa, 11);
+ MD5STEP(F3, c, d, a, b, in[3] + 0xd4ef3085, 16);
+ MD5STEP(F3, b, c, d, a, in[6] + 0x04881d05, 23);
+ MD5STEP(F3, a, b, c, d, in[9] + 0xd9d4d039, 4);
+ MD5STEP(F3, d, a, b, c, in[12] + 0xe6db99e5, 11);
+ MD5STEP(F3, c, d, a, b, in[15] + 0x1fa27cf8, 16);
+ MD5STEP(F3, b, c, d, a, in[2] + 0xc4ac5665, 23);
+
+ MD5STEP(F4, a, b, c, d, in[0] + 0xf4292244, 6);
+ MD5STEP(F4, d, a, b, c, in[7] + 0x432aff97, 10);
+ MD5STEP(F4, c, d, a, b, in[14] + 0xab9423a7, 15);
+ MD5STEP(F4, b, c, d, a, in[5] + 0xfc93a039, 21);
+ MD5STEP(F4, a, b, c, d, in[12] + 0x655b59c3, 6);
+ MD5STEP(F4, d, a, b, c, in[3] + 0x8f0ccc92, 10);
+ MD5STEP(F4, c, d, a, b, in[10] + 0xffeff47d, 15);
+ MD5STEP(F4, b, c, d, a, in[1] + 0x85845dd1, 21);
+ MD5STEP(F4, a, b, c, d, in[8] + 0x6fa87e4f, 6);
+ MD5STEP(F4, d, a, b, c, in[15] + 0xfe2ce6e0, 10);
+ MD5STEP(F4, c, d, a, b, in[6] + 0xa3014314, 15);
+ MD5STEP(F4, b, c, d, a, in[13] + 0x4e0811a1, 21);
+ MD5STEP(F4, a, b, c, d, in[4] + 0xf7537e82, 6);
+ MD5STEP(F4, d, a, b, c, in[11] + 0xbd3af235, 10);
+ MD5STEP(F4, c, d, a, b, in[2] + 0x2ad7d2bb, 15);
+ MD5STEP(F4, b, c, d, a, in[9] + 0xeb86d391, 21);
+
+ hash[0] += a;
+ hash[1] += b;
+ hash[2] += c;
+ hash[3] += d;
+}
+
static inline void md5_transform_helper(struct md5_state *ctx)
{
le32_to_cpu_array(ctx->block, sizeof(ctx->block) / sizeof(u32));
diff --git a/crypto/scompress.c b/crypto/scompress.c
index 6b048b3..ae1d3cf 100644
--- a/crypto/scompress.c
+++ b/crypto/scompress.c
@@ -353,5 +353,34 @@
}
EXPORT_SYMBOL_GPL(crypto_unregister_scomp);
+int crypto_register_scomps(struct scomp_alg *algs, int count)
+{
+ int i, ret;
+
+ for (i = 0; i < count; i++) {
+ ret = crypto_register_scomp(&algs[i]);
+ if (ret)
+ goto err;
+ }
+
+ return 0;
+
+err:
+ for (--i; i >= 0; --i)
+ crypto_unregister_scomp(&algs[i]);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(crypto_register_scomps);
+
+void crypto_unregister_scomps(struct scomp_alg *algs, int count)
+{
+ int i;
+
+ for (i = count - 1; i >= 0; --i)
+ crypto_unregister_scomp(&algs[i]);
+}
+EXPORT_SYMBOL_GPL(crypto_unregister_scomps);
+
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Synchronous compression type");
diff --git a/crypto/testmgr.c b/crypto/testmgr.c
index f9c378a..6f5f3ed 100644
--- a/crypto/testmgr.c
+++ b/crypto/testmgr.c
@@ -83,47 +83,47 @@
struct aead_test_suite {
struct {
- struct aead_testvec *vecs;
+ const struct aead_testvec *vecs;
unsigned int count;
} enc, dec;
};
struct cipher_test_suite {
struct {
- struct cipher_testvec *vecs;
+ const struct cipher_testvec *vecs;
unsigned int count;
} enc, dec;
};
struct comp_test_suite {
struct {
- struct comp_testvec *vecs;
+ const struct comp_testvec *vecs;
unsigned int count;
} comp, decomp;
};
struct hash_test_suite {
- struct hash_testvec *vecs;
+ const struct hash_testvec *vecs;
unsigned int count;
};
struct cprng_test_suite {
- struct cprng_testvec *vecs;
+ const struct cprng_testvec *vecs;
unsigned int count;
};
struct drbg_test_suite {
- struct drbg_testvec *vecs;
+ const struct drbg_testvec *vecs;
unsigned int count;
};
struct akcipher_test_suite {
- struct akcipher_testvec *vecs;
+ const struct akcipher_testvec *vecs;
unsigned int count;
};
struct kpp_test_suite {
- struct kpp_testvec *vecs;
+ const struct kpp_testvec *vecs;
unsigned int count;
};
@@ -145,7 +145,8 @@
} suite;
};
-static unsigned int IDX[8] = { IDX1, IDX2, IDX3, IDX4, IDX5, IDX6, IDX7, IDX8 };
+static const unsigned int IDX[8] = {
+ IDX1, IDX2, IDX3, IDX4, IDX5, IDX6, IDX7, IDX8 };
static void hexdump(unsigned char *buf, unsigned int len)
{
@@ -203,7 +204,7 @@
}
static int ahash_partial_update(struct ahash_request **preq,
- struct crypto_ahash *tfm, struct hash_testvec *template,
+ struct crypto_ahash *tfm, const struct hash_testvec *template,
void *hash_buff, int k, int temp, struct scatterlist *sg,
const char *algo, char *result, struct tcrypt_result *tresult)
{
@@ -260,9 +261,9 @@
return ret;
}
-static int __test_hash(struct crypto_ahash *tfm, struct hash_testvec *template,
- unsigned int tcount, bool use_digest,
- const int align_offset)
+static int __test_hash(struct crypto_ahash *tfm,
+ const struct hash_testvec *template, unsigned int tcount,
+ bool use_digest, const int align_offset)
{
const char *algo = crypto_tfm_alg_driver_name(crypto_ahash_tfm(tfm));
size_t digest_size = crypto_ahash_digestsize(tfm);
@@ -538,7 +539,8 @@
return ret;
}
-static int test_hash(struct crypto_ahash *tfm, struct hash_testvec *template,
+static int test_hash(struct crypto_ahash *tfm,
+ const struct hash_testvec *template,
unsigned int tcount, bool use_digest)
{
unsigned int alignmask;
@@ -566,7 +568,7 @@
}
static int __test_aead(struct crypto_aead *tfm, int enc,
- struct aead_testvec *template, unsigned int tcount,
+ const struct aead_testvec *template, unsigned int tcount,
const bool diff_dst, const int align_offset)
{
const char *algo = crypto_tfm_alg_driver_name(crypto_aead_tfm(tfm));
@@ -957,7 +959,7 @@
}
static int test_aead(struct crypto_aead *tfm, int enc,
- struct aead_testvec *template, unsigned int tcount)
+ const struct aead_testvec *template, unsigned int tcount)
{
unsigned int alignmask;
int ret;
@@ -990,7 +992,8 @@
}
static int test_cipher(struct crypto_cipher *tfm, int enc,
- struct cipher_testvec *template, unsigned int tcount)
+ const struct cipher_testvec *template,
+ unsigned int tcount)
{
const char *algo = crypto_tfm_alg_driver_name(crypto_cipher_tfm(tfm));
unsigned int i, j, k;
@@ -1068,7 +1071,8 @@
}
static int __test_skcipher(struct crypto_skcipher *tfm, int enc,
- struct cipher_testvec *template, unsigned int tcount,
+ const struct cipher_testvec *template,
+ unsigned int tcount,
const bool diff_dst, const int align_offset)
{
const char *algo =
@@ -1332,7 +1336,8 @@
}
static int test_skcipher(struct crypto_skcipher *tfm, int enc,
- struct cipher_testvec *template, unsigned int tcount)
+ const struct cipher_testvec *template,
+ unsigned int tcount)
{
unsigned int alignmask;
int ret;
@@ -1364,8 +1369,10 @@
return 0;
}
-static int test_comp(struct crypto_comp *tfm, struct comp_testvec *ctemplate,
- struct comp_testvec *dtemplate, int ctcount, int dtcount)
+static int test_comp(struct crypto_comp *tfm,
+ const struct comp_testvec *ctemplate,
+ const struct comp_testvec *dtemplate,
+ int ctcount, int dtcount)
{
const char *algo = crypto_tfm_alg_driver_name(crypto_comp_tfm(tfm));
unsigned int i;
@@ -1444,12 +1451,14 @@
return ret;
}
-static int test_acomp(struct crypto_acomp *tfm, struct comp_testvec *ctemplate,
- struct comp_testvec *dtemplate, int ctcount, int dtcount)
+static int test_acomp(struct crypto_acomp *tfm,
+ const struct comp_testvec *ctemplate,
+ const struct comp_testvec *dtemplate,
+ int ctcount, int dtcount)
{
const char *algo = crypto_tfm_alg_driver_name(crypto_acomp_tfm(tfm));
unsigned int i;
- char *output;
+ char *output, *decomp_out;
int ret;
struct scatterlist src, dst;
struct acomp_req *req;
@@ -1459,6 +1468,12 @@
if (!output)
return -ENOMEM;
+ decomp_out = kmalloc(COMP_BUF_SIZE, GFP_KERNEL);
+ if (!decomp_out) {
+ kfree(output);
+ return -ENOMEM;
+ }
+
for (i = 0; i < ctcount; i++) {
unsigned int dlen = COMP_BUF_SIZE;
int ilen = ctemplate[i].inlen;
@@ -1497,7 +1512,23 @@
goto out;
}
- if (req->dlen != ctemplate[i].outlen) {
+ ilen = req->dlen;
+ dlen = COMP_BUF_SIZE;
+ sg_init_one(&src, output, ilen);
+ sg_init_one(&dst, decomp_out, dlen);
+ init_completion(&result.completion);
+ acomp_request_set_params(req, &src, &dst, ilen, dlen);
+
+ ret = wait_async_op(&result, crypto_acomp_decompress(req));
+ if (ret) {
+ pr_err("alg: acomp: compression failed on test %d for %s: ret=%d\n",
+ i + 1, algo, -ret);
+ kfree(input_vec);
+ acomp_request_free(req);
+ goto out;
+ }
+
+ if (req->dlen != ctemplate[i].inlen) {
pr_err("alg: acomp: Compression test %d failed for %s: output len = %d\n",
i + 1, algo, req->dlen);
ret = -EINVAL;
@@ -1506,7 +1537,7 @@
goto out;
}
- if (memcmp(output, ctemplate[i].output, req->dlen)) {
+ if (memcmp(input_vec, decomp_out, req->dlen)) {
pr_err("alg: acomp: Compression test %d failed for %s\n",
i + 1, algo);
hexdump(output, req->dlen);
@@ -1584,11 +1615,13 @@
ret = 0;
out:
+ kfree(decomp_out);
kfree(output);
return ret;
}
-static int test_cprng(struct crypto_rng *tfm, struct cprng_testvec *template,
+static int test_cprng(struct crypto_rng *tfm,
+ const struct cprng_testvec *template,
unsigned int tcount)
{
const char *algo = crypto_tfm_alg_driver_name(crypto_rng_tfm(tfm));
@@ -1865,7 +1898,7 @@
}
-static int drbg_cavs_test(struct drbg_testvec *test, int pr,
+static int drbg_cavs_test(const struct drbg_testvec *test, int pr,
const char *driver, u32 type, u32 mask)
{
int ret = -EAGAIN;
@@ -1939,7 +1972,7 @@
int err = 0;
int pr = 0;
int i = 0;
- struct drbg_testvec *template = desc->suite.drbg.vecs;
+ const struct drbg_testvec *template = desc->suite.drbg.vecs;
unsigned int tcount = desc->suite.drbg.count;
if (0 == memcmp(driver, "drbg_pr_", 8))
@@ -1958,7 +1991,7 @@
}
-static int do_test_kpp(struct crypto_kpp *tfm, struct kpp_testvec *vec,
+static int do_test_kpp(struct crypto_kpp *tfm, const struct kpp_testvec *vec,
const char *alg)
{
struct kpp_request *req;
@@ -2050,7 +2083,7 @@
}
static int test_kpp(struct crypto_kpp *tfm, const char *alg,
- struct kpp_testvec *vecs, unsigned int tcount)
+ const struct kpp_testvec *vecs, unsigned int tcount)
{
int ret, i;
@@ -2086,7 +2119,7 @@
}
static int test_akcipher_one(struct crypto_akcipher *tfm,
- struct akcipher_testvec *vecs)
+ const struct akcipher_testvec *vecs)
{
char *xbuf[XBUFSIZE];
struct akcipher_request *req;
@@ -2206,7 +2239,8 @@
}
static int test_akcipher(struct crypto_akcipher *tfm, const char *alg,
- struct akcipher_testvec *vecs, unsigned int tcount)
+ const struct akcipher_testvec *vecs,
+ unsigned int tcount)
{
const char *algo =
crypto_tfm_alg_driver_name(crypto_akcipher_tfm(tfm));
@@ -2634,6 +2668,7 @@
}, {
.alg = "ctr(des3_ede)",
.test = alg_test_skcipher,
+ .fips_allowed = 1,
.suite = {
.cipher = {
.enc = __VECS(des3_ede_ctr_enc_tv_template),
@@ -2875,6 +2910,7 @@
}, {
.alg = "ecb(cipher_null)",
.test = alg_test_null,
+ .fips_allowed = 1,
}, {
.alg = "ecb(des)",
.test = alg_test_skcipher,
@@ -3477,6 +3513,16 @@
.dec = __VECS(tf_xts_dec_tv_template)
}
}
+ }, {
+ .alg = "zlib-deflate",
+ .test = alg_test_comp,
+ .fips_allowed = 1,
+ .suite = {
+ .comp = {
+ .comp = __VECS(zlib_deflate_comp_tv_template),
+ .decomp = __VECS(zlib_deflate_decomp_tv_template)
+ }
+ }
}
};
diff --git a/crypto/testmgr.h b/crypto/testmgr.h
index 03f4731..4293573 100644
--- a/crypto/testmgr.h
+++ b/crypto/testmgr.h
@@ -34,9 +34,9 @@
struct hash_testvec {
/* only used with keyed hash algorithms */
- char *key;
- char *plaintext;
- char *digest;
+ const char *key;
+ const char *plaintext;
+ const char *digest;
unsigned char tap[MAX_TAP];
unsigned short psize;
unsigned char np;
@@ -63,11 +63,11 @@
*/
struct cipher_testvec {
- char *key;
- char *iv;
- char *iv_out;
- char *input;
- char *result;
+ const char *key;
+ const char *iv;
+ const char *iv_out;
+ const char *input;
+ const char *result;
unsigned short tap[MAX_TAP];
int np;
unsigned char also_non_np;
@@ -80,11 +80,11 @@
};
struct aead_testvec {
- char *key;
- char *iv;
- char *input;
- char *assoc;
- char *result;
+ const char *key;
+ const char *iv;
+ const char *input;
+ const char *assoc;
+ const char *result;
unsigned char tap[MAX_TAP];
unsigned char atap[MAX_TAP];
int np;
@@ -99,10 +99,10 @@
};
struct cprng_testvec {
- char *key;
- char *dt;
- char *v;
- char *result;
+ const char *key;
+ const char *dt;
+ const char *v;
+ const char *result;
unsigned char klen;
unsigned short dtlen;
unsigned short vlen;
@@ -111,24 +111,24 @@
};
struct drbg_testvec {
- unsigned char *entropy;
+ const unsigned char *entropy;
size_t entropylen;
- unsigned char *entpra;
- unsigned char *entprb;
+ const unsigned char *entpra;
+ const unsigned char *entprb;
size_t entprlen;
- unsigned char *addtla;
- unsigned char *addtlb;
+ const unsigned char *addtla;
+ const unsigned char *addtlb;
size_t addtllen;
- unsigned char *pers;
+ const unsigned char *pers;
size_t perslen;
- unsigned char *expected;
+ const unsigned char *expected;
size_t expectedlen;
};
struct akcipher_testvec {
- unsigned char *key;
- unsigned char *m;
- unsigned char *c;
+ const unsigned char *key;
+ const unsigned char *m;
+ const unsigned char *c;
unsigned int key_len;
unsigned int m_size;
unsigned int c_size;
@@ -136,22 +136,22 @@
};
struct kpp_testvec {
- unsigned char *secret;
- unsigned char *b_public;
- unsigned char *expected_a_public;
- unsigned char *expected_ss;
+ const unsigned char *secret;
+ const unsigned char *b_public;
+ const unsigned char *expected_a_public;
+ const unsigned char *expected_ss;
unsigned short secret_size;
unsigned short b_public_size;
unsigned short expected_a_public_size;
unsigned short expected_ss_size;
};
-static char zeroed_string[48];
+static const char zeroed_string[48];
/*
* RSA test vectors. Borrowed from openSSL.
*/
-static struct akcipher_testvec rsa_tv_template[] = {
+static const struct akcipher_testvec rsa_tv_template[] = {
{
#ifndef CONFIG_CRYPTO_FIPS
.key =
@@ -538,7 +538,7 @@
}
};
-struct kpp_testvec dh_tv_template[] = {
+static const struct kpp_testvec dh_tv_template[] = {
{
.secret =
#ifdef __LITTLE_ENDIAN
@@ -755,7 +755,7 @@
}
};
-struct kpp_testvec ecdh_tv_template[] = {
+static const struct kpp_testvec ecdh_tv_template[] = {
{
#ifndef CONFIG_CRYPTO_FIPS
.secret =
@@ -846,7 +846,7 @@
/*
* MD4 test vectors from RFC1320
*/
-static struct hash_testvec md4_tv_template [] = {
+static const struct hash_testvec md4_tv_template[] = {
{
.plaintext = "",
.digest = "\x31\xd6\xcf\xe0\xd1\x6a\xe9\x31"
@@ -887,7 +887,7 @@
},
};
-static struct hash_testvec sha3_224_tv_template[] = {
+static const struct hash_testvec sha3_224_tv_template[] = {
{
.plaintext = "",
.digest = "\x6b\x4e\x03\x42\x36\x67\xdb\xb7"
@@ -912,7 +912,7 @@
},
};
-static struct hash_testvec sha3_256_tv_template[] = {
+static const struct hash_testvec sha3_256_tv_template[] = {
{
.plaintext = "",
.digest = "\xa7\xff\xc6\xf8\xbf\x1e\xd7\x66"
@@ -938,7 +938,7 @@
};
-static struct hash_testvec sha3_384_tv_template[] = {
+static const struct hash_testvec sha3_384_tv_template[] = {
{
.plaintext = "",
.digest = "\x0c\x63\xa7\x5b\x84\x5e\x4f\x7d"
@@ -970,7 +970,7 @@
};
-static struct hash_testvec sha3_512_tv_template[] = {
+static const struct hash_testvec sha3_512_tv_template[] = {
{
.plaintext = "",
.digest = "\xa6\x9f\x73\xcc\xa2\x3a\x9a\xc5"
@@ -1011,7 +1011,7 @@
/*
* MD5 test vectors from RFC1321
*/
-static struct hash_testvec md5_tv_template[] = {
+static const struct hash_testvec md5_tv_template[] = {
{
.digest = "\xd4\x1d\x8c\xd9\x8f\x00\xb2\x04"
"\xe9\x80\x09\x98\xec\xf8\x42\x7e",
@@ -1055,7 +1055,7 @@
/*
* RIPEMD-128 test vectors from ISO/IEC 10118-3:2004(E)
*/
-static struct hash_testvec rmd128_tv_template[] = {
+static const struct hash_testvec rmd128_tv_template[] = {
{
.digest = "\xcd\xf2\x62\x13\xa1\x50\xdc\x3e"
"\xcb\x61\x0f\x18\xf6\xb3\x8b\x46",
@@ -1117,7 +1117,7 @@
/*
* RIPEMD-160 test vectors from ISO/IEC 10118-3:2004(E)
*/
-static struct hash_testvec rmd160_tv_template[] = {
+static const struct hash_testvec rmd160_tv_template[] = {
{
.digest = "\x9c\x11\x85\xa5\xc5\xe9\xfc\x54\x61\x28"
"\x08\x97\x7e\xe8\xf5\x48\xb2\x25\x8d\x31",
@@ -1179,7 +1179,7 @@
/*
* RIPEMD-256 test vectors
*/
-static struct hash_testvec rmd256_tv_template[] = {
+static const struct hash_testvec rmd256_tv_template[] = {
{
.digest = "\x02\xba\x4c\x4e\x5f\x8e\xcd\x18"
"\x77\xfc\x52\xd6\x4d\x30\xe3\x7a"
@@ -1245,7 +1245,7 @@
/*
* RIPEMD-320 test vectors
*/
-static struct hash_testvec rmd320_tv_template[] = {
+static const struct hash_testvec rmd320_tv_template[] = {
{
.digest = "\x22\xd6\x5d\x56\x61\x53\x6c\xdc\x75\xc1"
"\xfd\xf5\xc6\xde\x7b\x41\xb9\xf2\x73\x25"
@@ -1308,7 +1308,7 @@
}
};
-static struct hash_testvec crct10dif_tv_template[] = {
+static const struct hash_testvec crct10dif_tv_template[] = {
{
.plaintext = "abc",
.psize = 3,
@@ -1358,7 +1358,7 @@
* SHA1 test vectors from from FIPS PUB 180-1
* Long vector from CAVS 5.0
*/
-static struct hash_testvec sha1_tv_template[] = {
+static const struct hash_testvec sha1_tv_template[] = {
{
.plaintext = "",
.psize = 0,
@@ -1548,7 +1548,7 @@
/*
* SHA224 test vectors from from FIPS PUB 180-2
*/
-static struct hash_testvec sha224_tv_template[] = {
+static const struct hash_testvec sha224_tv_template[] = {
{
.plaintext = "",
.psize = 0,
@@ -1720,7 +1720,7 @@
/*
* SHA256 test vectors from from NIST
*/
-static struct hash_testvec sha256_tv_template[] = {
+static const struct hash_testvec sha256_tv_template[] = {
{
.plaintext = "",
.psize = 0,
@@ -1891,7 +1891,7 @@
/*
* SHA384 test vectors from from NIST and kerneli
*/
-static struct hash_testvec sha384_tv_template[] = {
+static const struct hash_testvec sha384_tv_template[] = {
{
.plaintext = "",
.psize = 0,
@@ -2083,7 +2083,7 @@
/*
* SHA512 test vectors from from NIST and kerneli
*/
-static struct hash_testvec sha512_tv_template[] = {
+static const struct hash_testvec sha512_tv_template[] = {
{
.plaintext = "",
.psize = 0,
@@ -2290,7 +2290,7 @@
* by Vincent Rijmen and Paulo S. L. M. Barreto as part of the NESSIE
* submission
*/
-static struct hash_testvec wp512_tv_template[] = {
+static const struct hash_testvec wp512_tv_template[] = {
{
.plaintext = "",
.psize = 0,
@@ -2386,7 +2386,7 @@
},
};
-static struct hash_testvec wp384_tv_template[] = {
+static const struct hash_testvec wp384_tv_template[] = {
{
.plaintext = "",
.psize = 0,
@@ -2466,7 +2466,7 @@
},
};
-static struct hash_testvec wp256_tv_template[] = {
+static const struct hash_testvec wp256_tv_template[] = {
{
.plaintext = "",
.psize = 0,
@@ -2533,7 +2533,7 @@
/*
* TIGER test vectors from Tiger website
*/
-static struct hash_testvec tgr192_tv_template[] = {
+static const struct hash_testvec tgr192_tv_template[] = {
{
.plaintext = "",
.psize = 0,
@@ -2576,7 +2576,7 @@
},
};
-static struct hash_testvec tgr160_tv_template[] = {
+static const struct hash_testvec tgr160_tv_template[] = {
{
.plaintext = "",
.psize = 0,
@@ -2619,7 +2619,7 @@
},
};
-static struct hash_testvec tgr128_tv_template[] = {
+static const struct hash_testvec tgr128_tv_template[] = {
{
.plaintext = "",
.psize = 0,
@@ -2656,7 +2656,7 @@
},
};
-static struct hash_testvec ghash_tv_template[] =
+static const struct hash_testvec ghash_tv_template[] =
{
{
.key = "\xdf\xa6\xbf\x4d\xed\x81\xdb\x03"
@@ -2771,7 +2771,7 @@
* HMAC-MD5 test vectors from RFC2202
* (These need to be fixed to not use strlen).
*/
-static struct hash_testvec hmac_md5_tv_template[] =
+static const struct hash_testvec hmac_md5_tv_template[] =
{
{
.key = "\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b",
@@ -2851,7 +2851,7 @@
/*
* HMAC-RIPEMD128 test vectors from RFC2286
*/
-static struct hash_testvec hmac_rmd128_tv_template[] = {
+static const struct hash_testvec hmac_rmd128_tv_template[] = {
{
.key = "\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b",
.ksize = 16,
@@ -2930,7 +2930,7 @@
/*
* HMAC-RIPEMD160 test vectors from RFC2286
*/
-static struct hash_testvec hmac_rmd160_tv_template[] = {
+static const struct hash_testvec hmac_rmd160_tv_template[] = {
{
.key = "\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b",
.ksize = 20,
@@ -3009,7 +3009,7 @@
/*
* HMAC-SHA1 test vectors from RFC2202
*/
-static struct hash_testvec hmac_sha1_tv_template[] = {
+static const struct hash_testvec hmac_sha1_tv_template[] = {
{
.key = "\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b",
.ksize = 20,
@@ -3090,7 +3090,7 @@
/*
* SHA224 HMAC test vectors from RFC4231
*/
-static struct hash_testvec hmac_sha224_tv_template[] = {
+static const struct hash_testvec hmac_sha224_tv_template[] = {
{
.key = "\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
"\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
@@ -3203,7 +3203,7 @@
* HMAC-SHA256 test vectors from
* draft-ietf-ipsec-ciph-sha-256-01.txt
*/
-static struct hash_testvec hmac_sha256_tv_template[] = {
+static const struct hash_testvec hmac_sha256_tv_template[] = {
{
.key = "\x01\x02\x03\x04\x05\x06\x07\x08"
"\x09\x0a\x0b\x0c\x0d\x0e\x0f\x10"
@@ -3338,7 +3338,7 @@
},
};
-static struct hash_testvec aes_cmac128_tv_template[] = {
+static const struct hash_testvec aes_cmac128_tv_template[] = {
{ /* From NIST Special Publication 800-38B, AES-128 */
.key = "\x2b\x7e\x15\x16\x28\xae\xd2\xa6"
"\xab\xf7\x15\x88\x09\xcf\x4f\x3c",
@@ -3413,7 +3413,7 @@
}
};
-static struct hash_testvec aes_cbcmac_tv_template[] = {
+static const struct hash_testvec aes_cbcmac_tv_template[] = {
{
.key = "\x2b\x7e\x15\x16\x28\xae\xd2\xa6"
"\xab\xf7\x15\x88\x09\xcf\x4f\x3c",
@@ -3473,7 +3473,7 @@
}
};
-static struct hash_testvec des3_ede_cmac64_tv_template[] = {
+static const struct hash_testvec des3_ede_cmac64_tv_template[] = {
/*
* From NIST Special Publication 800-38B, Three Key TDEA
* Corrected test vectors from:
@@ -3519,7 +3519,7 @@
}
};
-static struct hash_testvec aes_xcbc128_tv_template[] = {
+static const struct hash_testvec aes_xcbc128_tv_template[] = {
{
.key = "\x00\x01\x02\x03\x04\x05\x06\x07"
"\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f",
@@ -3585,35 +3585,35 @@
}
};
-static char vmac_string1[128] = {'\x01', '\x01', '\x01', '\x01',
- '\x02', '\x03', '\x02', '\x02',
- '\x02', '\x04', '\x01', '\x07',
- '\x04', '\x01', '\x04', '\x03',};
-static char vmac_string2[128] = {'a', 'b', 'c',};
-static char vmac_string3[128] = {'a', 'b', 'c', 'a', 'b', 'c',
- 'a', 'b', 'c', 'a', 'b', 'c',
- 'a', 'b', 'c', 'a', 'b', 'c',
- 'a', 'b', 'c', 'a', 'b', 'c',
- 'a', 'b', 'c', 'a', 'b', 'c',
- 'a', 'b', 'c', 'a', 'b', 'c',
- 'a', 'b', 'c', 'a', 'b', 'c',
- 'a', 'b', 'c', 'a', 'b', 'c',
- };
+static const char vmac_string1[128] = {'\x01', '\x01', '\x01', '\x01',
+ '\x02', '\x03', '\x02', '\x02',
+ '\x02', '\x04', '\x01', '\x07',
+ '\x04', '\x01', '\x04', '\x03',};
+static const char vmac_string2[128] = {'a', 'b', 'c',};
+static const char vmac_string3[128] = {'a', 'b', 'c', 'a', 'b', 'c',
+ 'a', 'b', 'c', 'a', 'b', 'c',
+ 'a', 'b', 'c', 'a', 'b', 'c',
+ 'a', 'b', 'c', 'a', 'b', 'c',
+ 'a', 'b', 'c', 'a', 'b', 'c',
+ 'a', 'b', 'c', 'a', 'b', 'c',
+ 'a', 'b', 'c', 'a', 'b', 'c',
+ 'a', 'b', 'c', 'a', 'b', 'c',
+ };
-static char vmac_string4[17] = {'b', 'c', 'e', 'f',
- 'i', 'j', 'l', 'm',
- 'o', 'p', 'r', 's',
- 't', 'u', 'w', 'x', 'z'};
+static const char vmac_string4[17] = {'b', 'c', 'e', 'f',
+ 'i', 'j', 'l', 'm',
+ 'o', 'p', 'r', 's',
+ 't', 'u', 'w', 'x', 'z'};
-static char vmac_string5[127] = {'r', 'm', 'b', 't', 'c',
- 'o', 'l', 'k', ']', '%',
- '9', '2', '7', '!', 'A'};
+static const char vmac_string5[127] = {'r', 'm', 'b', 't', 'c',
+ 'o', 'l', 'k', ']', '%',
+ '9', '2', '7', '!', 'A'};
-static char vmac_string6[129] = {'p', 't', '*', '7', 'l',
- 'i', '!', '#', 'w', '0',
- 'z', '/', '4', 'A', 'n'};
+static const char vmac_string6[129] = {'p', 't', '*', '7', 'l',
+ 'i', '!', '#', 'w', '0',
+ 'z', '/', '4', 'A', 'n'};
-static struct hash_testvec aes_vmac128_tv_template[] = {
+static const struct hash_testvec aes_vmac128_tv_template[] = {
{
.key = "\x00\x01\x02\x03\x04\x05\x06\x07"
"\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f",
@@ -3691,7 +3691,7 @@
* SHA384 HMAC test vectors from RFC4231
*/
-static struct hash_testvec hmac_sha384_tv_template[] = {
+static const struct hash_testvec hmac_sha384_tv_template[] = {
{
.key = "\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
"\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
@@ -3789,7 +3789,7 @@
* SHA512 HMAC test vectors from RFC4231
*/
-static struct hash_testvec hmac_sha512_tv_template[] = {
+static const struct hash_testvec hmac_sha512_tv_template[] = {
{
.key = "\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
"\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
@@ -3894,7 +3894,7 @@
},
};
-static struct hash_testvec hmac_sha3_224_tv_template[] = {
+static const struct hash_testvec hmac_sha3_224_tv_template[] = {
{
.key = "\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
"\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
@@ -3983,7 +3983,7 @@
},
};
-static struct hash_testvec hmac_sha3_256_tv_template[] = {
+static const struct hash_testvec hmac_sha3_256_tv_template[] = {
{
.key = "\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
"\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
@@ -4072,7 +4072,7 @@
},
};
-static struct hash_testvec hmac_sha3_384_tv_template[] = {
+static const struct hash_testvec hmac_sha3_384_tv_template[] = {
{
.key = "\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
"\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
@@ -4169,7 +4169,7 @@
},
};
-static struct hash_testvec hmac_sha3_512_tv_template[] = {
+static const struct hash_testvec hmac_sha3_512_tv_template[] = {
{
.key = "\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
"\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
@@ -4278,7 +4278,7 @@
* Poly1305 test vectors from RFC7539 A.3.
*/
-static struct hash_testvec poly1305_tv_template[] = {
+static const struct hash_testvec poly1305_tv_template[] = {
{ /* Test Vector #1 */
.plaintext = "\x00\x00\x00\x00\x00\x00\x00\x00"
"\x00\x00\x00\x00\x00\x00\x00\x00"
@@ -4523,7 +4523,7 @@
/*
* DES test vectors.
*/
-static struct cipher_testvec des_enc_tv_template[] = {
+static const struct cipher_testvec des_enc_tv_template[] = {
{ /* From Applied Cryptography */
.key = "\x01\x23\x45\x67\x89\xab\xcd\xef",
.klen = 8,
@@ -4697,7 +4697,7 @@
},
};
-static struct cipher_testvec des_dec_tv_template[] = {
+static const struct cipher_testvec des_dec_tv_template[] = {
{ /* From Applied Cryptography */
.key = "\x01\x23\x45\x67\x89\xab\xcd\xef",
.klen = 8,
@@ -4807,7 +4807,7 @@
},
};
-static struct cipher_testvec des_cbc_enc_tv_template[] = {
+static const struct cipher_testvec des_cbc_enc_tv_template[] = {
{ /* From OpenSSL */
.key = "\x01\x23\x45\x67\x89\xab\xcd\xef",
.klen = 8,
@@ -4933,7 +4933,7 @@
},
};
-static struct cipher_testvec des_cbc_dec_tv_template[] = {
+static const struct cipher_testvec des_cbc_dec_tv_template[] = {
{ /* FIPS Pub 81 */
.key = "\x01\x23\x45\x67\x89\xab\xcd\xef",
.klen = 8,
@@ -5042,7 +5042,7 @@
},
};
-static struct cipher_testvec des_ctr_enc_tv_template[] = {
+static const struct cipher_testvec des_ctr_enc_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55",
.klen = 8,
@@ -5188,7 +5188,7 @@
},
};
-static struct cipher_testvec des_ctr_dec_tv_template[] = {
+static const struct cipher_testvec des_ctr_dec_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\xC9\x83\xA6\xC9\xEC\x0F\x32\x55",
.klen = 8,
@@ -5334,7 +5334,7 @@
},
};
-static struct cipher_testvec des3_ede_enc_tv_template[] = {
+static const struct cipher_testvec des3_ede_enc_tv_template[] = {
{ /* These are from openssl */
.key = "\x01\x23\x45\x67\x89\xab\xcd\xef"
"\x55\x55\x55\x55\x55\x55\x55\x55"
@@ -5499,7 +5499,7 @@
},
};
-static struct cipher_testvec des3_ede_dec_tv_template[] = {
+static const struct cipher_testvec des3_ede_dec_tv_template[] = {
{ /* These are from openssl */
.key = "\x01\x23\x45\x67\x89\xab\xcd\xef"
"\x55\x55\x55\x55\x55\x55\x55\x55"
@@ -5664,7 +5664,7 @@
},
};
-static struct cipher_testvec des3_ede_cbc_enc_tv_template[] = {
+static const struct cipher_testvec des3_ede_cbc_enc_tv_template[] = {
{ /* Generated from openssl */
.key = "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24"
"\x44\x4D\x99\x5A\x12\xD6\x40\xC0"
@@ -5844,7 +5844,7 @@
},
};
-static struct cipher_testvec des3_ede_cbc_dec_tv_template[] = {
+static const struct cipher_testvec des3_ede_cbc_dec_tv_template[] = {
{ /* Generated from openssl */
.key = "\xE9\xC0\xFF\x2E\x76\x0B\x64\x24"
"\x44\x4D\x99\x5A\x12\xD6\x40\xC0"
@@ -6024,7 +6024,7 @@
},
};
-static struct cipher_testvec des3_ede_ctr_enc_tv_template[] = {
+static const struct cipher_testvec des3_ede_ctr_enc_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\x9C\xD6\xF3\x9C\xB9\x5A\x67\x00"
"\x5A\x67\x00\x2D\xCE\xEB\x2D\xCE"
@@ -6302,7 +6302,7 @@
},
};
-static struct cipher_testvec des3_ede_ctr_dec_tv_template[] = {
+static const struct cipher_testvec des3_ede_ctr_dec_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\x9C\xD6\xF3\x9C\xB9\x5A\x67\x00"
"\x5A\x67\x00\x2D\xCE\xEB\x2D\xCE"
@@ -6583,7 +6583,7 @@
/*
* Blowfish test vectors.
*/
-static struct cipher_testvec bf_enc_tv_template[] = {
+static const struct cipher_testvec bf_enc_tv_template[] = {
{ /* DES test vectors from OpenSSL */
.key = "\x00\x00\x00\x00\x00\x00\x00\x00",
.klen = 8,
@@ -6775,7 +6775,7 @@
},
};
-static struct cipher_testvec bf_dec_tv_template[] = {
+static const struct cipher_testvec bf_dec_tv_template[] = {
{ /* DES test vectors from OpenSSL */
.key = "\x00\x00\x00\x00\x00\x00\x00\x00",
.klen = 8,
@@ -6967,7 +6967,7 @@
},
};
-static struct cipher_testvec bf_cbc_enc_tv_template[] = {
+static const struct cipher_testvec bf_cbc_enc_tv_template[] = {
{ /* From OpenSSL */
.key = "\x01\x23\x45\x67\x89\xab\xcd\xef"
"\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87",
@@ -7124,7 +7124,7 @@
},
};
-static struct cipher_testvec bf_cbc_dec_tv_template[] = {
+static const struct cipher_testvec bf_cbc_dec_tv_template[] = {
{ /* From OpenSSL */
.key = "\x01\x23\x45\x67\x89\xab\xcd\xef"
"\xf0\xe1\xd2\xc3\xb4\xa5\x96\x87",
@@ -7281,7 +7281,7 @@
},
};
-static struct cipher_testvec bf_ctr_enc_tv_template[] = {
+static const struct cipher_testvec bf_ctr_enc_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -7693,7 +7693,7 @@
},
};
-static struct cipher_testvec bf_ctr_dec_tv_template[] = {
+static const struct cipher_testvec bf_ctr_dec_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -8108,7 +8108,7 @@
/*
* Twofish test vectors.
*/
-static struct cipher_testvec tf_enc_tv_template[] = {
+static const struct cipher_testvec tf_enc_tv_template[] = {
{
.key = zeroed_string,
.klen = 16,
@@ -8276,7 +8276,7 @@
},
};
-static struct cipher_testvec tf_dec_tv_template[] = {
+static const struct cipher_testvec tf_dec_tv_template[] = {
{
.key = zeroed_string,
.klen = 16,
@@ -8444,7 +8444,7 @@
},
};
-static struct cipher_testvec tf_cbc_enc_tv_template[] = {
+static const struct cipher_testvec tf_cbc_enc_tv_template[] = {
{ /* Generated with Nettle */
.key = zeroed_string,
.klen = 16,
@@ -8627,7 +8627,7 @@
},
};
-static struct cipher_testvec tf_cbc_dec_tv_template[] = {
+static const struct cipher_testvec tf_cbc_dec_tv_template[] = {
{ /* Reverse of the first four above */
.key = zeroed_string,
.klen = 16,
@@ -8810,7 +8810,7 @@
},
};
-static struct cipher_testvec tf_ctr_enc_tv_template[] = {
+static const struct cipher_testvec tf_ctr_enc_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -9221,7 +9221,7 @@
},
};
-static struct cipher_testvec tf_ctr_dec_tv_template[] = {
+static const struct cipher_testvec tf_ctr_dec_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -9632,7 +9632,7 @@
},
};
-static struct cipher_testvec tf_lrw_enc_tv_template[] = {
+static const struct cipher_testvec tf_lrw_enc_tv_template[] = {
/* Generated from AES-LRW test vectors */
{
.key = "\x45\x62\xac\x25\xf8\x28\x17\x6d"
@@ -9884,7 +9884,7 @@
},
};
-static struct cipher_testvec tf_lrw_dec_tv_template[] = {
+static const struct cipher_testvec tf_lrw_dec_tv_template[] = {
/* Generated from AES-LRW test vectors */
/* same as enc vectors with input and result reversed */
{
@@ -10137,7 +10137,7 @@
},
};
-static struct cipher_testvec tf_xts_enc_tv_template[] = {
+static const struct cipher_testvec tf_xts_enc_tv_template[] = {
/* Generated from AES-XTS test vectors */
{
.key = "\x00\x00\x00\x00\x00\x00\x00\x00"
@@ -10479,7 +10479,7 @@
},
};
-static struct cipher_testvec tf_xts_dec_tv_template[] = {
+static const struct cipher_testvec tf_xts_dec_tv_template[] = {
/* Generated from AES-XTS test vectors */
/* same as enc vectors with input and result reversed */
{
@@ -10826,7 +10826,7 @@
* Serpent test vectors. These are backwards because Serpent writes
* octet sequences in right-to-left mode.
*/
-static struct cipher_testvec serpent_enc_tv_template[] = {
+static const struct cipher_testvec serpent_enc_tv_template[] = {
{
.input = "\x00\x01\x02\x03\x04\x05\x06\x07"
"\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f",
@@ -11002,7 +11002,7 @@
},
};
-static struct cipher_testvec tnepres_enc_tv_template[] = {
+static const struct cipher_testvec tnepres_enc_tv_template[] = {
{ /* KeySize=128, PT=0, I=1 */
.input = "\x00\x00\x00\x00\x00\x00\x00\x00"
"\x00\x00\x00\x00\x00\x00\x00\x00",
@@ -11052,7 +11052,7 @@
};
-static struct cipher_testvec serpent_dec_tv_template[] = {
+static const struct cipher_testvec serpent_dec_tv_template[] = {
{
.input = "\x12\x07\xfc\xce\x9b\xd0\xd6\x47"
"\x6a\xe9\x8f\xbe\xd1\x43\xa0\xe2",
@@ -11228,7 +11228,7 @@
},
};
-static struct cipher_testvec tnepres_dec_tv_template[] = {
+static const struct cipher_testvec tnepres_dec_tv_template[] = {
{
.input = "\x41\xcc\x6b\x31\x59\x31\x45\x97"
"\x6d\x6f\xbb\x38\x4b\x37\x21\x28",
@@ -11269,7 +11269,7 @@
},
};
-static struct cipher_testvec serpent_cbc_enc_tv_template[] = {
+static const struct cipher_testvec serpent_cbc_enc_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -11410,7 +11410,7 @@
},
};
-static struct cipher_testvec serpent_cbc_dec_tv_template[] = {
+static const struct cipher_testvec serpent_cbc_dec_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -11551,7 +11551,7 @@
},
};
-static struct cipher_testvec serpent_ctr_enc_tv_template[] = {
+static const struct cipher_testvec serpent_ctr_enc_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -11962,7 +11962,7 @@
},
};
-static struct cipher_testvec serpent_ctr_dec_tv_template[] = {
+static const struct cipher_testvec serpent_ctr_dec_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -12373,7 +12373,7 @@
},
};
-static struct cipher_testvec serpent_lrw_enc_tv_template[] = {
+static const struct cipher_testvec serpent_lrw_enc_tv_template[] = {
/* Generated from AES-LRW test vectors */
{
.key = "\x45\x62\xac\x25\xf8\x28\x17\x6d"
@@ -12625,7 +12625,7 @@
},
};
-static struct cipher_testvec serpent_lrw_dec_tv_template[] = {
+static const struct cipher_testvec serpent_lrw_dec_tv_template[] = {
/* Generated from AES-LRW test vectors */
/* same as enc vectors with input and result reversed */
{
@@ -12878,7 +12878,7 @@
},
};
-static struct cipher_testvec serpent_xts_enc_tv_template[] = {
+static const struct cipher_testvec serpent_xts_enc_tv_template[] = {
/* Generated from AES-XTS test vectors */
{
.key = "\x00\x00\x00\x00\x00\x00\x00\x00"
@@ -13220,7 +13220,7 @@
},
};
-static struct cipher_testvec serpent_xts_dec_tv_template[] = {
+static const struct cipher_testvec serpent_xts_dec_tv_template[] = {
/* Generated from AES-XTS test vectors */
/* same as enc vectors with input and result reversed */
{
@@ -13564,7 +13564,7 @@
};
/* Cast6 test vectors from RFC 2612 */
-static struct cipher_testvec cast6_enc_tv_template[] = {
+static const struct cipher_testvec cast6_enc_tv_template[] = {
{
.key = "\x23\x42\xbb\x9e\xfa\x38\x54\x2c"
"\x0a\xf7\x56\x47\xf2\x9f\x61\x5d",
@@ -13735,7 +13735,7 @@
},
};
-static struct cipher_testvec cast6_dec_tv_template[] = {
+static const struct cipher_testvec cast6_dec_tv_template[] = {
{
.key = "\x23\x42\xbb\x9e\xfa\x38\x54\x2c"
"\x0a\xf7\x56\x47\xf2\x9f\x61\x5d",
@@ -13906,7 +13906,7 @@
},
};
-static struct cipher_testvec cast6_cbc_enc_tv_template[] = {
+static const struct cipher_testvec cast6_cbc_enc_tv_template[] = {
{ /* Generated from TF test vectors */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -14047,7 +14047,7 @@
},
};
-static struct cipher_testvec cast6_cbc_dec_tv_template[] = {
+static const struct cipher_testvec cast6_cbc_dec_tv_template[] = {
{ /* Generated from TF test vectors */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -14188,7 +14188,7 @@
},
};
-static struct cipher_testvec cast6_ctr_enc_tv_template[] = {
+static const struct cipher_testvec cast6_ctr_enc_tv_template[] = {
{ /* Generated from TF test vectors */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -14345,7 +14345,7 @@
},
};
-static struct cipher_testvec cast6_ctr_dec_tv_template[] = {
+static const struct cipher_testvec cast6_ctr_dec_tv_template[] = {
{ /* Generated from TF test vectors */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -14502,7 +14502,7 @@
},
};
-static struct cipher_testvec cast6_lrw_enc_tv_template[] = {
+static const struct cipher_testvec cast6_lrw_enc_tv_template[] = {
{ /* Generated from TF test vectors */
.key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c"
"\x23\x84\xcb\x1c\x77\xd6\x19\x5d"
@@ -14649,7 +14649,7 @@
},
};
-static struct cipher_testvec cast6_lrw_dec_tv_template[] = {
+static const struct cipher_testvec cast6_lrw_dec_tv_template[] = {
{ /* Generated from TF test vectors */
.key = "\xf8\xd4\x76\xff\xd6\x46\xee\x6c"
"\x23\x84\xcb\x1c\x77\xd6\x19\x5d"
@@ -14796,7 +14796,7 @@
},
};
-static struct cipher_testvec cast6_xts_enc_tv_template[] = {
+static const struct cipher_testvec cast6_xts_enc_tv_template[] = {
{ /* Generated from TF test vectors */
.key = "\x27\x18\x28\x18\x28\x45\x90\x45"
"\x23\x53\x60\x28\x74\x71\x35\x26"
@@ -14945,7 +14945,7 @@
},
};
-static struct cipher_testvec cast6_xts_dec_tv_template[] = {
+static const struct cipher_testvec cast6_xts_dec_tv_template[] = {
{ /* Generated from TF test vectors */
.key = "\x27\x18\x28\x18\x28\x45\x90\x45"
"\x23\x53\x60\x28\x74\x71\x35\x26"
@@ -15098,7 +15098,7 @@
/*
* AES test vectors.
*/
-static struct cipher_testvec aes_enc_tv_template[] = {
+static const struct cipher_testvec aes_enc_tv_template[] = {
{ /* From FIPS-197 */
.key = "\x00\x01\x02\x03\x04\x05\x06\x07"
"\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f",
@@ -15270,7 +15270,7 @@
},
};
-static struct cipher_testvec aes_dec_tv_template[] = {
+static const struct cipher_testvec aes_dec_tv_template[] = {
{ /* From FIPS-197 */
.key = "\x00\x01\x02\x03\x04\x05\x06\x07"
"\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f",
@@ -15442,7 +15442,7 @@
},
};
-static struct cipher_testvec aes_cbc_enc_tv_template[] = {
+static const struct cipher_testvec aes_cbc_enc_tv_template[] = {
{ /* From RFC 3602 */
.key = "\x06\xa9\x21\x40\x36\xb8\xa1\x5b"
"\x51\x2e\x03\xd5\x34\x12\x00\x06",
@@ -15664,7 +15664,7 @@
},
};
-static struct cipher_testvec aes_cbc_dec_tv_template[] = {
+static const struct cipher_testvec aes_cbc_dec_tv_template[] = {
{ /* From RFC 3602 */
.key = "\x06\xa9\x21\x40\x36\xb8\xa1\x5b"
"\x51\x2e\x03\xd5\x34\x12\x00\x06",
@@ -15886,7 +15886,7 @@
},
};
-static struct aead_testvec hmac_md5_ecb_cipher_null_enc_tv_template[] = {
+static const struct aead_testvec hmac_md5_ecb_cipher_null_enc_tv_template[] = {
{ /* Input data from RFC 2410 Case 1 */
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -15928,7 +15928,7 @@
},
};
-static struct aead_testvec hmac_md5_ecb_cipher_null_dec_tv_template[] = {
+static const struct aead_testvec hmac_md5_ecb_cipher_null_dec_tv_template[] = {
{
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -15970,7 +15970,7 @@
},
};
-static struct aead_testvec hmac_sha1_aes_cbc_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha1_aes_cbc_enc_tv_temp[] = {
{ /* RFC 3602 Case 1 */
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -16239,7 +16239,7 @@
},
};
-static struct aead_testvec hmac_sha1_ecb_cipher_null_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha1_ecb_cipher_null_enc_tv_temp[] = {
{ /* Input data from RFC 2410 Case 1 */
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -16285,7 +16285,7 @@
},
};
-static struct aead_testvec hmac_sha1_ecb_cipher_null_dec_tv_temp[] = {
+static const struct aead_testvec hmac_sha1_ecb_cipher_null_dec_tv_temp[] = {
{
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -16331,7 +16331,7 @@
},
};
-static struct aead_testvec hmac_sha256_aes_cbc_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha256_aes_cbc_enc_tv_temp[] = {
{ /* RFC 3602 Case 1 */
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -16614,7 +16614,7 @@
},
};
-static struct aead_testvec hmac_sha512_aes_cbc_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha512_aes_cbc_enc_tv_temp[] = {
{ /* RFC 3602 Case 1 */
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -16953,7 +16953,7 @@
},
};
-static struct aead_testvec hmac_sha1_des_cbc_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha1_des_cbc_enc_tv_temp[] = {
{ /*Generated with cryptopp*/
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -17012,7 +17012,7 @@
},
};
-static struct aead_testvec hmac_sha224_des_cbc_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha224_des_cbc_enc_tv_temp[] = {
{ /*Generated with cryptopp*/
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -17071,7 +17071,7 @@
},
};
-static struct aead_testvec hmac_sha256_des_cbc_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha256_des_cbc_enc_tv_temp[] = {
{ /*Generated with cryptopp*/
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -17132,7 +17132,7 @@
},
};
-static struct aead_testvec hmac_sha384_des_cbc_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha384_des_cbc_enc_tv_temp[] = {
{ /*Generated with cryptopp*/
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -17197,7 +17197,7 @@
},
};
-static struct aead_testvec hmac_sha512_des_cbc_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha512_des_cbc_enc_tv_temp[] = {
{ /*Generated with cryptopp*/
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -17266,7 +17266,7 @@
},
};
-static struct aead_testvec hmac_sha1_des3_ede_cbc_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha1_des3_ede_cbc_enc_tv_temp[] = {
{ /*Generated with cryptopp*/
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -17327,7 +17327,7 @@
},
};
-static struct aead_testvec hmac_sha224_des3_ede_cbc_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha224_des3_ede_cbc_enc_tv_temp[] = {
{ /*Generated with cryptopp*/
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -17388,7 +17388,7 @@
},
};
-static struct aead_testvec hmac_sha256_des3_ede_cbc_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha256_des3_ede_cbc_enc_tv_temp[] = {
{ /*Generated with cryptopp*/
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -17451,7 +17451,7 @@
},
};
-static struct aead_testvec hmac_sha384_des3_ede_cbc_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha384_des3_ede_cbc_enc_tv_temp[] = {
{ /*Generated with cryptopp*/
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -17518,7 +17518,7 @@
},
};
-static struct aead_testvec hmac_sha512_des3_ede_cbc_enc_tv_temp[] = {
+static const struct aead_testvec hmac_sha512_des3_ede_cbc_enc_tv_temp[] = {
{ /*Generated with cryptopp*/
#ifdef __LITTLE_ENDIAN
.key = "\x08\x00" /* rta length */
@@ -17589,7 +17589,7 @@
},
};
-static struct cipher_testvec aes_lrw_enc_tv_template[] = {
+static const struct cipher_testvec aes_lrw_enc_tv_template[] = {
/* from http://grouper.ieee.org/groups/1619/email/pdf00017.pdf */
{ /* LRW-32-AES 1 */
.key = "\x45\x62\xac\x25\xf8\x28\x17\x6d"
@@ -17842,7 +17842,7 @@
}
};
-static struct cipher_testvec aes_lrw_dec_tv_template[] = {
+static const struct cipher_testvec aes_lrw_dec_tv_template[] = {
/* from http://grouper.ieee.org/groups/1619/email/pdf00017.pdf */
/* same as enc vectors with input and result reversed */
{ /* LRW-32-AES 1 */
@@ -18096,7 +18096,7 @@
}
};
-static struct cipher_testvec aes_xts_enc_tv_template[] = {
+static const struct cipher_testvec aes_xts_enc_tv_template[] = {
/* http://grouper.ieee.org/groups/1619/email/pdf00086.pdf */
{ /* XTS-AES 1 */
.key = "\x00\x00\x00\x00\x00\x00\x00\x00"
@@ -18439,7 +18439,7 @@
}
};
-static struct cipher_testvec aes_xts_dec_tv_template[] = {
+static const struct cipher_testvec aes_xts_dec_tv_template[] = {
/* http://grouper.ieee.org/groups/1619/email/pdf00086.pdf */
{ /* XTS-AES 1 */
.key = "\x00\x00\x00\x00\x00\x00\x00\x00"
@@ -18783,7 +18783,7 @@
};
-static struct cipher_testvec aes_ctr_enc_tv_template[] = {
+static const struct cipher_testvec aes_ctr_enc_tv_template[] = {
{ /* From NIST Special Publication 800-38A, Appendix F.5 */
.key = "\x2b\x7e\x15\x16\x28\xae\xd2\xa6"
"\xab\xf7\x15\x88\x09\xcf\x4f\x3c",
@@ -19138,7 +19138,7 @@
},
};
-static struct cipher_testvec aes_ctr_dec_tv_template[] = {
+static const struct cipher_testvec aes_ctr_dec_tv_template[] = {
{ /* From NIST Special Publication 800-38A, Appendix F.5 */
.key = "\x2b\x7e\x15\x16\x28\xae\xd2\xa6"
"\xab\xf7\x15\x88\x09\xcf\x4f\x3c",
@@ -19493,7 +19493,7 @@
},
};
-static struct cipher_testvec aes_ctr_rfc3686_enc_tv_template[] = {
+static const struct cipher_testvec aes_ctr_rfc3686_enc_tv_template[] = {
{ /* From RFC 3686 */
.key = "\xae\x68\x52\xf8\x12\x10\x67\xcc"
"\x4b\xf7\xa5\x76\x55\x77\xf3\x9e"
@@ -20625,7 +20625,7 @@
},
};
-static struct cipher_testvec aes_ctr_rfc3686_dec_tv_template[] = {
+static const struct cipher_testvec aes_ctr_rfc3686_dec_tv_template[] = {
{ /* From RFC 3686 */
.key = "\xae\x68\x52\xf8\x12\x10\x67\xcc"
"\x4b\xf7\xa5\x76\x55\x77\xf3\x9e"
@@ -20716,7 +20716,7 @@
},
};
-static struct cipher_testvec aes_ofb_enc_tv_template[] = {
+static const struct cipher_testvec aes_ofb_enc_tv_template[] = {
/* From NIST Special Publication 800-38A, Appendix F.5 */
{
.key = "\x2b\x7e\x15\x16\x28\xae\xd2\xa6"
@@ -20745,7 +20745,7 @@
}
};
-static struct cipher_testvec aes_ofb_dec_tv_template[] = {
+static const struct cipher_testvec aes_ofb_dec_tv_template[] = {
/* From NIST Special Publication 800-38A, Appendix F.5 */
{
.key = "\x2b\x7e\x15\x16\x28\xae\xd2\xa6"
@@ -20774,7 +20774,7 @@
}
};
-static struct aead_testvec aes_gcm_enc_tv_template[] = {
+static const struct aead_testvec aes_gcm_enc_tv_template[] = {
{ /* From McGrew & Viega - http://citeseer.ist.psu.edu/656989.html */
.key = zeroed_string,
.klen = 16,
@@ -20934,7 +20934,7 @@
}
};
-static struct aead_testvec aes_gcm_dec_tv_template[] = {
+static const struct aead_testvec aes_gcm_dec_tv_template[] = {
{ /* From McGrew & Viega - http://citeseer.ist.psu.edu/656989.html */
.key = zeroed_string,
.klen = 32,
@@ -21136,7 +21136,7 @@
}
};
-static struct aead_testvec aes_gcm_rfc4106_enc_tv_template[] = {
+static const struct aead_testvec aes_gcm_rfc4106_enc_tv_template[] = {
{ /* Generated using Crypto++ */
.key = zeroed_string,
.klen = 20,
@@ -21749,7 +21749,7 @@
}
};
-static struct aead_testvec aes_gcm_rfc4106_dec_tv_template[] = {
+static const struct aead_testvec aes_gcm_rfc4106_dec_tv_template[] = {
{ /* Generated using Crypto++ */
.key = zeroed_string,
.klen = 20,
@@ -22363,7 +22363,7 @@
}
};
-static struct aead_testvec aes_gcm_rfc4543_enc_tv_template[] = {
+static const struct aead_testvec aes_gcm_rfc4543_enc_tv_template[] = {
{ /* From draft-mcgrew-gcm-test-01 */
.key = "\x4c\x80\xcd\xef\xbb\x5d\x10\xda"
"\x90\x6a\xc7\x3c\x36\x13\xa6\x34"
@@ -22394,7 +22394,7 @@
}
};
-static struct aead_testvec aes_gcm_rfc4543_dec_tv_template[] = {
+static const struct aead_testvec aes_gcm_rfc4543_dec_tv_template[] = {
{ /* From draft-mcgrew-gcm-test-01 */
.key = "\x4c\x80\xcd\xef\xbb\x5d\x10\xda"
"\x90\x6a\xc7\x3c\x36\x13\xa6\x34"
@@ -22453,7 +22453,7 @@
},
};
-static struct aead_testvec aes_ccm_enc_tv_template[] = {
+static const struct aead_testvec aes_ccm_enc_tv_template[] = {
{ /* From RFC 3610 */
.key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7"
"\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf",
@@ -22737,7 +22737,7 @@
}
};
-static struct aead_testvec aes_ccm_dec_tv_template[] = {
+static const struct aead_testvec aes_ccm_dec_tv_template[] = {
{ /* From RFC 3610 */
.key = "\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7"
"\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf",
@@ -23069,7 +23069,7 @@
* These vectors are copied/generated from the ones for rfc4106 with
* the key truncated by one byte..
*/
-static struct aead_testvec aes_ccm_rfc4309_enc_tv_template[] = {
+static const struct aead_testvec aes_ccm_rfc4309_enc_tv_template[] = {
{ /* Generated using Crypto++ */
.key = zeroed_string,
.klen = 19,
@@ -23682,7 +23682,7 @@
}
};
-static struct aead_testvec aes_ccm_rfc4309_dec_tv_template[] = {
+static const struct aead_testvec aes_ccm_rfc4309_dec_tv_template[] = {
{ /* Generated using Crypto++ */
.key = zeroed_string,
.klen = 19,
@@ -24298,7 +24298,7 @@
/*
* ChaCha20-Poly1305 AEAD test vectors from RFC7539 2.8.2./A.5.
*/
-static struct aead_testvec rfc7539_enc_tv_template[] = {
+static const struct aead_testvec rfc7539_enc_tv_template[] = {
{
.key = "\x80\x81\x82\x83\x84\x85\x86\x87"
"\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f"
@@ -24430,7 +24430,7 @@
},
};
-static struct aead_testvec rfc7539_dec_tv_template[] = {
+static const struct aead_testvec rfc7539_dec_tv_template[] = {
{
.key = "\x80\x81\x82\x83\x84\x85\x86\x87"
"\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f"
@@ -24565,7 +24565,7 @@
/*
* draft-irtf-cfrg-chacha20-poly1305
*/
-static struct aead_testvec rfc7539esp_enc_tv_template[] = {
+static const struct aead_testvec rfc7539esp_enc_tv_template[] = {
{
.key = "\x1c\x92\x40\xa5\xeb\x55\xd3\x8a"
"\xf3\x33\x88\x86\x04\xf6\xb5\xf0"
@@ -24653,7 +24653,7 @@
},
};
-static struct aead_testvec rfc7539esp_dec_tv_template[] = {
+static const struct aead_testvec rfc7539esp_dec_tv_template[] = {
{
.key = "\x1c\x92\x40\xa5\xeb\x55\xd3\x8a"
"\xf3\x33\x88\x86\x04\xf6\xb5\xf0"
@@ -24749,7 +24749,7 @@
* semiblock of the ciphertext from the test vector. For decryption, iv is
* the first semiblock of the ciphertext.
*/
-static struct cipher_testvec aes_kw_enc_tv_template[] = {
+static const struct cipher_testvec aes_kw_enc_tv_template[] = {
{
.key = "\x75\x75\xda\x3a\x93\x60\x7c\xc2"
"\xbf\xd8\xce\xc7\xaa\xdf\xd9\xa6",
@@ -24764,7 +24764,7 @@
},
};
-static struct cipher_testvec aes_kw_dec_tv_template[] = {
+static const struct cipher_testvec aes_kw_dec_tv_template[] = {
{
.key = "\x80\xaa\x99\x73\x27\xa4\x80\x6b"
"\x6a\x7a\x41\xa5\x2b\x86\xc3\x71"
@@ -24787,7 +24787,7 @@
* http://csrc.nist.gov/groups/STM/cavp/documents/rng/RNGVS.pdf
* Only AES-128 is supported at this time.
*/
-static struct cprng_testvec ansi_cprng_aes_tv_template[] = {
+static const struct cprng_testvec ansi_cprng_aes_tv_template[] = {
{
.key = "\xf3\xb1\x66\x6d\x13\x60\x72\x42"
"\xed\x06\x1c\xab\xb8\xd4\x62\x02",
@@ -24883,7 +24883,7 @@
* (Hash, HMAC, CTR) are tested with all permutations of use cases (w/ and
* w/o personalization string, w/ and w/o additional input string).
*/
-static struct drbg_testvec drbg_pr_sha256_tv_template[] = {
+static const struct drbg_testvec drbg_pr_sha256_tv_template[] = {
{
.entropy = (unsigned char *)
"\x72\x88\x4c\xcd\x6c\x85\x57\x70\xf7\x0b\x8b\x86"
@@ -25041,7 +25041,7 @@
},
};
-static struct drbg_testvec drbg_pr_hmac_sha256_tv_template[] = {
+static const struct drbg_testvec drbg_pr_hmac_sha256_tv_template[] = {
{
.entropy = (unsigned char *)
"\x99\x69\xe5\x4b\x47\x03\xff\x31\x78\x5b\x87\x9a"
@@ -25199,7 +25199,7 @@
},
};
-static struct drbg_testvec drbg_pr_ctr_aes128_tv_template[] = {
+static const struct drbg_testvec drbg_pr_ctr_aes128_tv_template[] = {
{
.entropy = (unsigned char *)
"\xd1\x44\xc6\x61\x81\x6d\xca\x9d\x15\x28\x8a\x42"
@@ -25323,7 +25323,7 @@
* (Hash, HMAC, CTR) are tested with all permutations of use cases (w/ and
* w/o personalization string, w/ and w/o additional input string).
*/
-static struct drbg_testvec drbg_nopr_sha256_tv_template[] = {
+static const struct drbg_testvec drbg_nopr_sha256_tv_template[] = {
{
.entropy = (unsigned char *)
"\xa6\x5a\xd0\xf3\x45\xdb\x4e\x0e\xff\xe8\x75\xc3"
@@ -25445,7 +25445,7 @@
},
};
-static struct drbg_testvec drbg_nopr_hmac_sha256_tv_template[] = {
+static const struct drbg_testvec drbg_nopr_hmac_sha256_tv_template[] = {
{
.entropy = (unsigned char *)
"\xca\x85\x19\x11\x34\x93\x84\xbf\xfe\x89\xde\x1c"
@@ -25567,7 +25567,7 @@
},
};
-static struct drbg_testvec drbg_nopr_ctr_aes192_tv_template[] = {
+static const struct drbg_testvec drbg_nopr_ctr_aes192_tv_template[] = {
{
.entropy = (unsigned char *)
"\xc3\x5c\x2f\xa2\xa8\x9d\x52\xa1\x1f\xa3\x2a\xa9"
@@ -25591,7 +25591,7 @@
},
};
-static struct drbg_testvec drbg_nopr_ctr_aes256_tv_template[] = {
+static const struct drbg_testvec drbg_nopr_ctr_aes256_tv_template[] = {
{
.entropy = (unsigned char *)
"\x36\x40\x19\x40\xfa\x8b\x1f\xba\x91\xa1\x66\x1f"
@@ -25615,7 +25615,7 @@
},
};
-static struct drbg_testvec drbg_nopr_ctr_aes128_tv_template[] = {
+static const struct drbg_testvec drbg_nopr_ctr_aes128_tv_template[] = {
{
.entropy = (unsigned char *)
"\x87\xe1\xc5\x32\x99\x7f\x57\xa3\x5c\x28\x6d\xe8"
@@ -25704,7 +25704,7 @@
};
/* Cast5 test vectors from RFC 2144 */
-static struct cipher_testvec cast5_enc_tv_template[] = {
+static const struct cipher_testvec cast5_enc_tv_template[] = {
{
.key = "\x01\x23\x45\x67\x12\x34\x56\x78"
"\x23\x45\x67\x89\x34\x56\x78\x9a",
@@ -25865,7 +25865,7 @@
},
};
-static struct cipher_testvec cast5_dec_tv_template[] = {
+static const struct cipher_testvec cast5_dec_tv_template[] = {
{
.key = "\x01\x23\x45\x67\x12\x34\x56\x78"
"\x23\x45\x67\x89\x34\x56\x78\x9a",
@@ -26026,7 +26026,7 @@
},
};
-static struct cipher_testvec cast5_cbc_enc_tv_template[] = {
+static const struct cipher_testvec cast5_cbc_enc_tv_template[] = {
{ /* Generated from TF test vectors */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A",
@@ -26164,7 +26164,7 @@
},
};
-static struct cipher_testvec cast5_cbc_dec_tv_template[] = {
+static const struct cipher_testvec cast5_cbc_dec_tv_template[] = {
{ /* Generated from TF test vectors */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A",
@@ -26302,7 +26302,7 @@
},
};
-static struct cipher_testvec cast5_ctr_enc_tv_template[] = {
+static const struct cipher_testvec cast5_ctr_enc_tv_template[] = {
{ /* Generated from TF test vectors */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A",
@@ -26453,7 +26453,7 @@
},
};
-static struct cipher_testvec cast5_ctr_dec_tv_template[] = {
+static const struct cipher_testvec cast5_ctr_dec_tv_template[] = {
{ /* Generated from TF test vectors */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A",
@@ -26607,7 +26607,7 @@
/*
* ARC4 test vectors from OpenSSL
*/
-static struct cipher_testvec arc4_enc_tv_template[] = {
+static const struct cipher_testvec arc4_enc_tv_template[] = {
{
.key = "\x01\x23\x45\x67\x89\xab\xcd\xef",
.klen = 8,
@@ -26673,7 +26673,7 @@
},
};
-static struct cipher_testvec arc4_dec_tv_template[] = {
+static const struct cipher_testvec arc4_dec_tv_template[] = {
{
.key = "\x01\x23\x45\x67\x89\xab\xcd\xef",
.klen = 8,
@@ -26742,7 +26742,7 @@
/*
* TEA test vectors
*/
-static struct cipher_testvec tea_enc_tv_template[] = {
+static const struct cipher_testvec tea_enc_tv_template[] = {
{
.key = zeroed_string,
.klen = 16,
@@ -26785,7 +26785,7 @@
}
};
-static struct cipher_testvec tea_dec_tv_template[] = {
+static const struct cipher_testvec tea_dec_tv_template[] = {
{
.key = zeroed_string,
.klen = 16,
@@ -26831,7 +26831,7 @@
/*
* XTEA test vectors
*/
-static struct cipher_testvec xtea_enc_tv_template[] = {
+static const struct cipher_testvec xtea_enc_tv_template[] = {
{
.key = zeroed_string,
.klen = 16,
@@ -26874,7 +26874,7 @@
}
};
-static struct cipher_testvec xtea_dec_tv_template[] = {
+static const struct cipher_testvec xtea_dec_tv_template[] = {
{
.key = zeroed_string,
.klen = 16,
@@ -26920,7 +26920,7 @@
/*
* KHAZAD test vectors.
*/
-static struct cipher_testvec khazad_enc_tv_template[] = {
+static const struct cipher_testvec khazad_enc_tv_template[] = {
{
.key = "\x80\x00\x00\x00\x00\x00\x00\x00"
"\x00\x00\x00\x00\x00\x00\x00\x00",
@@ -26966,7 +26966,7 @@
},
};
-static struct cipher_testvec khazad_dec_tv_template[] = {
+static const struct cipher_testvec khazad_dec_tv_template[] = {
{
.key = "\x80\x00\x00\x00\x00\x00\x00\x00"
"\x00\x00\x00\x00\x00\x00\x00\x00",
@@ -27016,7 +27016,7 @@
* Anubis test vectors.
*/
-static struct cipher_testvec anubis_enc_tv_template[] = {
+static const struct cipher_testvec anubis_enc_tv_template[] = {
{
.key = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe"
"\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe",
@@ -27079,7 +27079,7 @@
},
};
-static struct cipher_testvec anubis_dec_tv_template[] = {
+static const struct cipher_testvec anubis_dec_tv_template[] = {
{
.key = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe"
"\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe",
@@ -27142,7 +27142,7 @@
},
};
-static struct cipher_testvec anubis_cbc_enc_tv_template[] = {
+static const struct cipher_testvec anubis_cbc_enc_tv_template[] = {
{
.key = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe"
"\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe",
@@ -27177,7 +27177,7 @@
},
};
-static struct cipher_testvec anubis_cbc_dec_tv_template[] = {
+static const struct cipher_testvec anubis_cbc_dec_tv_template[] = {
{
.key = "\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe"
"\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe",
@@ -27215,7 +27215,7 @@
/*
* XETA test vectors
*/
-static struct cipher_testvec xeta_enc_tv_template[] = {
+static const struct cipher_testvec xeta_enc_tv_template[] = {
{
.key = zeroed_string,
.klen = 16,
@@ -27258,7 +27258,7 @@
}
};
-static struct cipher_testvec xeta_dec_tv_template[] = {
+static const struct cipher_testvec xeta_dec_tv_template[] = {
{
.key = zeroed_string,
.klen = 16,
@@ -27304,7 +27304,7 @@
/*
* FCrypt test vectors
*/
-static struct cipher_testvec fcrypt_pcbc_enc_tv_template[] = {
+static const struct cipher_testvec fcrypt_pcbc_enc_tv_template[] = {
{ /* http://www.openafs.org/pipermail/openafs-devel/2000-December/005320.html */
.key = "\x00\x00\x00\x00\x00\x00\x00\x00",
.klen = 8,
@@ -27365,7 +27365,7 @@
}
};
-static struct cipher_testvec fcrypt_pcbc_dec_tv_template[] = {
+static const struct cipher_testvec fcrypt_pcbc_dec_tv_template[] = {
{ /* http://www.openafs.org/pipermail/openafs-devel/2000-December/005320.html */
.key = "\x00\x00\x00\x00\x00\x00\x00\x00",
.klen = 8,
@@ -27429,7 +27429,7 @@
/*
* CAMELLIA test vectors.
*/
-static struct cipher_testvec camellia_enc_tv_template[] = {
+static const struct cipher_testvec camellia_enc_tv_template[] = {
{
.key = "\x01\x23\x45\x67\x89\xab\xcd\xef"
"\xfe\xdc\xba\x98\x76\x54\x32\x10",
@@ -27729,7 +27729,7 @@
},
};
-static struct cipher_testvec camellia_dec_tv_template[] = {
+static const struct cipher_testvec camellia_dec_tv_template[] = {
{
.key = "\x01\x23\x45\x67\x89\xab\xcd\xef"
"\xfe\xdc\xba\x98\x76\x54\x32\x10",
@@ -28029,7 +28029,7 @@
},
};
-static struct cipher_testvec camellia_cbc_enc_tv_template[] = {
+static const struct cipher_testvec camellia_cbc_enc_tv_template[] = {
{
.key = "\x06\xa9\x21\x40\x36\xb8\xa1\x5b"
"\x51\x2e\x03\xd5\x34\x12\x00\x06",
@@ -28325,7 +28325,7 @@
},
};
-static struct cipher_testvec camellia_cbc_dec_tv_template[] = {
+static const struct cipher_testvec camellia_cbc_dec_tv_template[] = {
{
.key = "\x06\xa9\x21\x40\x36\xb8\xa1\x5b"
"\x51\x2e\x03\xd5\x34\x12\x00\x06",
@@ -28621,7 +28621,7 @@
},
};
-static struct cipher_testvec camellia_ctr_enc_tv_template[] = {
+static const struct cipher_testvec camellia_ctr_enc_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -29288,7 +29288,7 @@
},
};
-static struct cipher_testvec camellia_ctr_dec_tv_template[] = {
+static const struct cipher_testvec camellia_ctr_dec_tv_template[] = {
{ /* Generated with Crypto++ */
.key = "\x85\x62\x3F\x1C\xF9\xD6\x1C\xF9"
"\xD6\xB3\x90\x6D\x4A\x90\x6D\x4A"
@@ -29955,7 +29955,7 @@
},
};
-static struct cipher_testvec camellia_lrw_enc_tv_template[] = {
+static const struct cipher_testvec camellia_lrw_enc_tv_template[] = {
/* Generated from AES-LRW test vectors */
{
.key = "\x45\x62\xac\x25\xf8\x28\x17\x6d"
@@ -30207,7 +30207,7 @@
},
};
-static struct cipher_testvec camellia_lrw_dec_tv_template[] = {
+static const struct cipher_testvec camellia_lrw_dec_tv_template[] = {
/* Generated from AES-LRW test vectors */
/* same as enc vectors with input and result reversed */
{
@@ -30460,7 +30460,7 @@
},
};
-static struct cipher_testvec camellia_xts_enc_tv_template[] = {
+static const struct cipher_testvec camellia_xts_enc_tv_template[] = {
/* Generated from AES-XTS test vectors */
{
.key = "\x00\x00\x00\x00\x00\x00\x00\x00"
@@ -30802,7 +30802,7 @@
},
};
-static struct cipher_testvec camellia_xts_dec_tv_template[] = {
+static const struct cipher_testvec camellia_xts_dec_tv_template[] = {
/* Generated from AES-XTS test vectors */
/* same as enc vectors with input and result reversed */
{
@@ -31148,7 +31148,7 @@
/*
* SEED test vectors
*/
-static struct cipher_testvec seed_enc_tv_template[] = {
+static const struct cipher_testvec seed_enc_tv_template[] = {
{
.key = zeroed_string,
.klen = 16,
@@ -31190,7 +31190,7 @@
}
};
-static struct cipher_testvec seed_dec_tv_template[] = {
+static const struct cipher_testvec seed_dec_tv_template[] = {
{
.key = zeroed_string,
.klen = 16,
@@ -31232,7 +31232,7 @@
}
};
-static struct cipher_testvec salsa20_stream_enc_tv_template[] = {
+static const struct cipher_testvec salsa20_stream_enc_tv_template[] = {
/*
* Testvectors from verified.test-vectors submitted to ECRYPT.
* They are truncated to size 39, 64, 111, 129 to test a variety
@@ -32401,7 +32401,7 @@
},
};
-static struct cipher_testvec chacha20_enc_tv_template[] = {
+static const struct cipher_testvec chacha20_enc_tv_template[] = {
{ /* RFC7539 A.2. Test Vector #1 */
.key = "\x00\x00\x00\x00\x00\x00\x00\x00"
"\x00\x00\x00\x00\x00\x00\x00\x00"
@@ -32912,7 +32912,7 @@
/*
* CTS (Cipher Text Stealing) mode tests
*/
-static struct cipher_testvec cts_mode_enc_tv_template[] = {
+static const struct cipher_testvec cts_mode_enc_tv_template[] = {
{ /* from rfc3962 */
.klen = 16,
.key = "\x63\x68\x69\x63\x6b\x65\x6e\x20"
@@ -33014,7 +33014,7 @@
}
};
-static struct cipher_testvec cts_mode_dec_tv_template[] = {
+static const struct cipher_testvec cts_mode_dec_tv_template[] = {
{ /* from rfc3962 */
.klen = 16,
.key = "\x63\x68\x69\x63\x6b\x65\x6e\x20"
@@ -33132,7 +33132,7 @@
* Params: winbits=-11, Z_DEFAULT_COMPRESSION, MAX_MEM_LEVEL.
*/
-static struct comp_testvec deflate_comp_tv_template[] = {
+static const struct comp_testvec deflate_comp_tv_template[] = {
{
.inlen = 70,
.outlen = 38,
@@ -33168,7 +33168,7 @@
},
};
-static struct comp_testvec deflate_decomp_tv_template[] = {
+static const struct comp_testvec deflate_decomp_tv_template[] = {
{
.inlen = 122,
.outlen = 191,
@@ -33204,10 +33204,85 @@
},
};
+static const struct comp_testvec zlib_deflate_comp_tv_template[] = {
+ {
+ .inlen = 70,
+ .outlen = 44,
+ .input = "Join us now and share the software "
+ "Join us now and share the software ",
+ .output = "\x78\x5e\xf3\xca\xcf\xcc\x53\x28"
+ "\x2d\x56\xc8\xcb\x2f\x57\x48\xcc"
+ "\x4b\x51\x28\xce\x48\x2c\x4a\x55"
+ "\x28\xc9\x48\x55\x28\xce\x4f\x2b"
+ "\x29\x07\x71\xbc\x08\x2b\x01\x00"
+ "\x7c\x65\x19\x3d",
+ }, {
+ .inlen = 191,
+ .outlen = 129,
+ .input = "This document describes a compression method based on the DEFLATE"
+ "compression algorithm. This document defines the application of "
+ "the DEFLATE algorithm to the IP Payload Compression Protocol.",
+ .output = "\x78\x5e\x5d\xce\x41\x0a\xc3\x30"
+ "\x0c\x04\xc0\xaf\xec\x0b\xf2\x87"
+ "\xd2\xa6\x50\xe8\xc1\x07\x7f\x40"
+ "\xb1\x95\x5a\x60\x5b\xc6\x56\x0f"
+ "\xfd\x7d\x93\x1e\x42\xe8\x51\xec"
+ "\xee\x20\x9f\x64\x20\x6a\x78\x17"
+ "\xae\x86\xc8\x23\x74\x59\x78\x80"
+ "\x10\xb4\xb4\xce\x63\x88\x56\x14"
+ "\xb6\xa4\x11\x0b\x0d\x8e\xd8\x6e"
+ "\x4b\x8c\xdb\x7c\x7f\x5e\xfc\x7c"
+ "\xae\x51\x7e\x69\x17\x4b\x65\x02"
+ "\xfc\x1f\xbc\x4a\xdd\xd8\x7d\x48"
+ "\xad\x65\x09\x64\x3b\xac\xeb\xd9"
+ "\xc2\x01\xc0\xf4\x17\x3c\x1c\x1c"
+ "\x7d\xb2\x52\xc4\xf5\xf4\x8f\xeb"
+ "\x6a\x1a\x34\x4f\x5f\x2e\x32\x45"
+ "\x4e",
+ },
+};
+
+static const struct comp_testvec zlib_deflate_decomp_tv_template[] = {
+ {
+ .inlen = 128,
+ .outlen = 191,
+ .input = "\x78\x9c\x5d\x8d\x31\x0e\xc2\x30"
+ "\x10\x04\xbf\xb2\x2f\xc8\x1f\x10"
+ "\x04\x09\x89\xc2\x85\x3f\x70\xb1"
+ "\x2f\xf8\x24\xdb\x67\xd9\x47\xc1"
+ "\xef\x49\x68\x12\x51\xae\x76\x67"
+ "\xd6\x27\x19\x88\x1a\xde\x85\xab"
+ "\x21\xf2\x08\x5d\x16\x1e\x20\x04"
+ "\x2d\xad\xf3\x18\xa2\x15\x85\x2d"
+ "\x69\xc4\x42\x83\x23\xb6\x6c\x89"
+ "\x71\x9b\xef\xcf\x8b\x9f\xcf\x33"
+ "\xca\x2f\xed\x62\xa9\x4c\x80\xff"
+ "\x13\xaf\x52\x37\xed\x0e\x52\x6b"
+ "\x59\x02\xd9\x4e\xe8\x7a\x76\x1d"
+ "\x02\x98\xfe\x8a\x87\x83\xa3\x4f"
+ "\x56\x8a\xb8\x9e\x8e\x5c\x57\xd3"
+ "\xa0\x79\xfa\x02\x2e\x32\x45\x4e",
+ .output = "This document describes a compression method based on the DEFLATE"
+ "compression algorithm. This document defines the application of "
+ "the DEFLATE algorithm to the IP Payload Compression Protocol.",
+ }, {
+ .inlen = 44,
+ .outlen = 70,
+ .input = "\x78\x9c\xf3\xca\xcf\xcc\x53\x28"
+ "\x2d\x56\xc8\xcb\x2f\x57\x48\xcc"
+ "\x4b\x51\x28\xce\x48\x2c\x4a\x55"
+ "\x28\xc9\x48\x55\x28\xce\x4f\x2b"
+ "\x29\x07\x71\xbc\x08\x2b\x01\x00"
+ "\x7c\x65\x19\x3d",
+ .output = "Join us now and share the software "
+ "Join us now and share the software ",
+ },
+};
+
/*
* LZO test vectors (null-terminated strings).
*/
-static struct comp_testvec lzo_comp_tv_template[] = {
+static const struct comp_testvec lzo_comp_tv_template[] = {
{
.inlen = 70,
.outlen = 57,
@@ -33247,7 +33322,7 @@
},
};
-static struct comp_testvec lzo_decomp_tv_template[] = {
+static const struct comp_testvec lzo_decomp_tv_template[] = {
{
.inlen = 133,
.outlen = 159,
@@ -33290,7 +33365,7 @@
*/
#define MICHAEL_MIC_TEST_VECTORS 6
-static struct hash_testvec michael_mic_tv_template[] = {
+static const struct hash_testvec michael_mic_tv_template[] = {
{
.key = "\x00\x00\x00\x00\x00\x00\x00\x00",
.ksize = 8,
@@ -33338,7 +33413,7 @@
/*
* CRC32 test vectors
*/
-static struct hash_testvec crc32_tv_template[] = {
+static const struct hash_testvec crc32_tv_template[] = {
{
.key = "\x87\xa9\xcb\xed",
.ksize = 4,
@@ -33770,7 +33845,7 @@
/*
* CRC32C test vectors
*/
-static struct hash_testvec crc32c_tv_template[] = {
+static const struct hash_testvec crc32c_tv_template[] = {
{
.psize = 0,
.digest = "\x00\x00\x00\x00",
@@ -34206,7 +34281,7 @@
/*
* Blakcifn CRC test vectors
*/
-static struct hash_testvec bfin_crc_tv_template[] = {
+static const struct hash_testvec bfin_crc_tv_template[] = {
{
.psize = 0,
.digest = "\x00\x00\x00\x00",
@@ -34291,7 +34366,7 @@
};
-static struct comp_testvec lz4_comp_tv_template[] = {
+static const struct comp_testvec lz4_comp_tv_template[] = {
{
.inlen = 255,
.outlen = 218,
@@ -34322,7 +34397,7 @@
},
};
-static struct comp_testvec lz4_decomp_tv_template[] = {
+static const struct comp_testvec lz4_decomp_tv_template[] = {
{
.inlen = 218,
.outlen = 255,
@@ -34352,7 +34427,7 @@
},
};
-static struct comp_testvec lz4hc_comp_tv_template[] = {
+static const struct comp_testvec lz4hc_comp_tv_template[] = {
{
.inlen = 255,
.outlen = 216,
@@ -34383,7 +34458,7 @@
},
};
-static struct comp_testvec lz4hc_decomp_tv_template[] = {
+static const struct comp_testvec lz4hc_decomp_tv_template[] = {
{
.inlen = 216,
.outlen = 255,
diff --git a/crypto/xts.c b/crypto/xts.c
index 89ace5e..d86c11a 100644
--- a/crypto/xts.c
+++ b/crypto/xts.c
@@ -39,11 +39,11 @@
};
struct rctx {
- be128 buf[XTS_BUFFER_SIZE / sizeof(be128)];
+ le128 buf[XTS_BUFFER_SIZE / sizeof(le128)];
- be128 t;
+ le128 t;
- be128 *ext;
+ le128 *ext;
struct scatterlist srcbuf[2];
struct scatterlist dstbuf[2];
@@ -99,7 +99,7 @@
static int post_crypt(struct skcipher_request *req)
{
struct rctx *rctx = skcipher_request_ctx(req);
- be128 *buf = rctx->ext ?: rctx->buf;
+ le128 *buf = rctx->ext ?: rctx->buf;
struct skcipher_request *subreq;
const int bs = XTS_BLOCK_SIZE;
struct skcipher_walk w;
@@ -112,12 +112,12 @@
while (w.nbytes) {
unsigned int avail = w.nbytes;
- be128 *wdst;
+ le128 *wdst;
wdst = w.dst.virt.addr;
do {
- be128_xor(wdst, buf++, wdst);
+ le128_xor(wdst, buf++, wdst);
wdst++;
} while ((avail -= bs) >= bs);
@@ -150,7 +150,7 @@
static int pre_crypt(struct skcipher_request *req)
{
struct rctx *rctx = skcipher_request_ctx(req);
- be128 *buf = rctx->ext ?: rctx->buf;
+ le128 *buf = rctx->ext ?: rctx->buf;
struct skcipher_request *subreq;
const int bs = XTS_BLOCK_SIZE;
struct skcipher_walk w;
@@ -174,15 +174,15 @@
while (w.nbytes) {
unsigned int avail = w.nbytes;
- be128 *wsrc;
- be128 *wdst;
+ le128 *wsrc;
+ le128 *wdst;
wsrc = w.src.virt.addr;
wdst = w.dst.virt.addr;
do {
*buf++ = rctx->t;
- be128_xor(wdst++, &rctx->t, wsrc++);
+ le128_xor(wdst++, &rctx->t, wsrc++);
gf128mul_x_ble(&rctx->t, &rctx->t);
} while ((avail -= bs) >= bs);
@@ -369,8 +369,8 @@
const unsigned int max_blks = req->tbuflen / bsize;
struct blkcipher_walk walk;
unsigned int nblocks;
- be128 *src, *dst, *t;
- be128 *t_buf = req->tbuf;
+ le128 *src, *dst, *t;
+ le128 *t_buf = req->tbuf;
int err, i;
BUG_ON(max_blks < 1);
@@ -383,8 +383,8 @@
return err;
nblocks = min(nbytes / bsize, max_blks);
- src = (be128 *)walk.src.virt.addr;
- dst = (be128 *)walk.dst.virt.addr;
+ src = (le128 *)walk.src.virt.addr;
+ dst = (le128 *)walk.dst.virt.addr;
/* calculate first value of T */
req->tweak_fn(req->tweak_ctx, (u8 *)&t_buf[0], walk.iv);
@@ -400,7 +400,7 @@
t = &t_buf[i];
/* PP <- T xor P */
- be128_xor(dst + i, t, src + i);
+ le128_xor(dst + i, t, src + i);
}
/* CC <- E(Key2,PP) */
@@ -409,7 +409,7 @@
/* C <- T xor CC */
for (i = 0; i < nblocks; i++)
- be128_xor(dst + i, dst + i, &t_buf[i]);
+ le128_xor(dst + i, dst + i, &t_buf[i]);
src += nblocks;
dst += nblocks;
@@ -417,7 +417,7 @@
nblocks = min(nbytes / bsize, max_blks);
} while (nblocks > 0);
- *(be128 *)walk.iv = *t;
+ *(le128 *)walk.iv = *t;
err = blkcipher_walk_done(desc, &walk, nbytes);
nbytes = walk.nbytes;
@@ -425,8 +425,8 @@
break;
nblocks = min(nbytes / bsize, max_blks);
- src = (be128 *)walk.src.virt.addr;
- dst = (be128 *)walk.dst.virt.addr;
+ src = (le128 *)walk.src.virt.addr;
+ dst = (le128 *)walk.dst.virt.addr;
}
return err;
diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
index 83e5f7e..18f3036 100644
--- a/drivers/acpi/Kconfig
+++ b/drivers/acpi/Kconfig
@@ -256,7 +256,7 @@
config ACPI_IPMI
tristate "IPMI"
- depends on IPMI_SI
+ depends on IPMI_HANDLER
default n
help
This driver enables the ACPI to access the BMC controller. And it
@@ -440,7 +440,7 @@
config ACPI_BGRT
bool "Boottime Graphics Resource Table support"
- depends on EFI && X86
+ depends on EFI && (X86 || ARM64)
help
This driver adds support for exposing the ACPI Boottime Graphics
Resource Table, which allows the operating system to obtain
@@ -469,9 +469,8 @@
config ACPI_EXTLOG
tristate "Extended Error Log support"
- depends on X86_MCE && X86_LOCAL_APIC
+ depends on X86_MCE && X86_LOCAL_APIC && EDAC
select UEFI_CPER
- select RAS
default n
help
Certain usages such as Predictive Failure Analysis (PFA) require
@@ -506,7 +505,7 @@
config XPOWER_PMIC_OPREGION
bool "ACPI operation region support for XPower AXP288 PMIC"
- depends on AXP288_ADC = y
+ depends on MFD_AXP20X_I2C
help
This config adds ACPI operation region support for XPower AXP288 PMIC.
@@ -516,6 +515,12 @@
help
This config adds ACPI operation region support for BXT WhiskeyCove PMIC.
+config CHT_WC_PMIC_OPREGION
+ bool "ACPI operation region support for CHT Whiskey Cove PMIC"
+ depends on INTEL_SOC_PMIC_CHTWC
+ help
+ This config adds ACPI operation region support for CHT Whiskey Cove PMIC.
+
endif
config ACPI_CONFIGFS
diff --git a/drivers/acpi/Makefile b/drivers/acpi/Makefile
index d94f92f..d78065c 100644
--- a/drivers/acpi/Makefile
+++ b/drivers/acpi/Makefile
@@ -101,6 +101,7 @@
obj-$(CONFIG_CRC_PMIC_OPREGION) += pmic/intel_pmic_crc.o
obj-$(CONFIG_XPOWER_PMIC_OPREGION) += pmic/intel_pmic_xpower.o
obj-$(CONFIG_BXT_WC_PMIC_OPREGION) += pmic/intel_pmic_bxtwc.o
+obj-$(CONFIG_CHT_WC_PMIC_OPREGION) += pmic/intel_pmic_chtwc.o
obj-$(CONFIG_ACPI_CONFIGFS) += acpi_configfs.o
diff --git a/drivers/acpi/ac.c b/drivers/acpi/ac.c
index f71b756..8f52483 100644
--- a/drivers/acpi/ac.c
+++ b/drivers/acpi/ac.c
@@ -57,12 +57,23 @@
static int acpi_ac_remove(struct acpi_device *device);
static void acpi_ac_notify(struct acpi_device *device, u32 event);
+struct acpi_ac_bl {
+ const char *hid;
+ int hrv;
+};
+
static const struct acpi_device_id ac_device_ids[] = {
{"ACPI0003", 0},
{"", 0},
};
MODULE_DEVICE_TABLE(acpi, ac_device_ids);
+/* Lists of PMIC ACPI HIDs with an (often better) native charger driver */
+static const struct acpi_ac_bl acpi_ac_blacklist[] = {
+ { "INT33F4", -1 }, /* X-Powers AXP288 PMIC */
+ { "INT34D3", 3 }, /* Intel Cherrytrail Whiskey Cove PMIC */
+};
+
#ifdef CONFIG_PM_SLEEP
static int acpi_ac_resume(struct device *dev);
#endif
@@ -424,11 +435,20 @@
static int __init acpi_ac_init(void)
{
+ unsigned int i;
int result;
if (acpi_disabled)
return -ENODEV;
+ for (i = 0; i < ARRAY_SIZE(acpi_ac_blacklist); i++)
+ if (acpi_dev_present(acpi_ac_blacklist[i].hid, "1",
+ acpi_ac_blacklist[i].hrv)) {
+ pr_info(PREFIX "AC: found native %s PMIC, not loading\n",
+ acpi_ac_blacklist[i].hid);
+ return -ENODEV;
+ }
+
#ifdef CONFIG_ACPI_PROCFS_POWER
acpi_ac_dir = acpi_lock_ac_dir();
if (!acpi_ac_dir)
diff --git a/drivers/acpi/acpi_extlog.c b/drivers/acpi/acpi_extlog.c
index a15270a..502ea4dc 100644
--- a/drivers/acpi/acpi_extlog.c
+++ b/drivers/acpi/acpi_extlog.c
@@ -229,7 +229,7 @@
if (!(cap & MCG_ELOG_P) || !extlog_get_l1addr())
return -ENODEV;
- if (get_edac_report_status() == EDAC_REPORTING_FORCE) {
+ if (edac_get_report_status() == EDAC_REPORTING_FORCE) {
pr_warn("Not loading eMCA, error reporting force-enabled through EDAC.\n");
return -EPERM;
}
@@ -285,8 +285,8 @@
* eMCA event report method has higher priority than EDAC method,
* unless EDAC event report method is mandatory.
*/
- old_edac_report_status = get_edac_report_status();
- set_edac_report_status(EDAC_REPORTING_DISABLED);
+ old_edac_report_status = edac_get_report_status();
+ edac_set_report_status(EDAC_REPORTING_DISABLED);
mce_register_decode_chain(&extlog_mce_dec);
/* enable OS to be involved to take over management from BIOS */
((struct extlog_l1_head *)extlog_l1_addr)->flags |= FLAG_OS_OPTIN;
@@ -308,7 +308,7 @@
static void __exit extlog_exit(void)
{
- set_edac_report_status(old_edac_report_status);
+ edac_set_report_status(old_edac_report_status);
mce_unregister_decode_chain(&extlog_mce_dec);
((struct extlog_l1_head *)extlog_l1_addr)->flags &= ~FLAG_OS_OPTIN;
if (extlog_l1_addr)
diff --git a/drivers/acpi/acpi_ipmi.c b/drivers/acpi/acpi_ipmi.c
index 747c2ba..1b64419 100644
--- a/drivers/acpi/acpi_ipmi.c
+++ b/drivers/acpi/acpi_ipmi.c
@@ -429,8 +429,7 @@
if (msg->recv_type == IPMI_RESPONSE_RECV_TYPE &&
msg->msg.data_len == 1) {
if (msg->msg.data[0] == IPMI_TIMEOUT_COMPLETION_CODE) {
- dev_WARN_ONCE(dev, true,
- "Unexpected response (timeout).\n");
+ dev_dbg_once(dev, "Unexpected response (timeout).\n");
tx_msg->msg_done = ACPI_IPMI_TIMEOUT;
}
goto out_comp;
diff --git a/drivers/acpi/acpi_platform.c b/drivers/acpi/acpi_platform.c
index 03250e1f..88cd949 100644
--- a/drivers/acpi/acpi_platform.c
+++ b/drivers/acpi/acpi_platform.c
@@ -121,11 +121,14 @@
if (IS_ERR(pdev))
dev_err(&adev->dev, "platform device creation failed: %ld\n",
PTR_ERR(pdev));
- else
+ else {
+ set_dev_node(&pdev->dev, acpi_get_node(adev->handle));
dev_dbg(&adev->dev, "created platform device %s\n",
dev_name(&pdev->dev));
+ }
kfree(resources);
+
return pdev;
}
EXPORT_SYMBOL_GPL(acpi_create_platform_device);
diff --git a/drivers/acpi/acpi_processor.c b/drivers/acpi/acpi_processor.c
index 0143135..f098e25 100644
--- a/drivers/acpi/acpi_processor.c
+++ b/drivers/acpi/acpi_processor.c
@@ -388,11 +388,6 @@
if (result) /* Processor is not physically present or unavailable */
return 0;
-#ifdef CONFIG_SMP
- if (pr->id >= setup_max_cpus && pr->id != 0)
- return 0;
-#endif
-
BUG_ON(pr->id >= nr_cpu_ids);
/*
diff --git a/drivers/acpi/acpi_video.c b/drivers/acpi/acpi_video.c
index d00bc0e..e88fe36 100644
--- a/drivers/acpi/acpi_video.c
+++ b/drivers/acpi/acpi_video.c
@@ -73,6 +73,10 @@
MODULE_PARM_DESC(report_key_events,
"0: none, 1: output changes, 2: brightness changes, 3: all");
+/*
+ * Whether the struct acpi_video_device_attrib::device_id_scheme bit should be
+ * assumed even if not actually set.
+ */
static bool device_id_scheme = false;
module_param(device_id_scheme, bool, 0444);
@@ -88,6 +92,18 @@
static void acpi_video_bus_notify(struct acpi_device *device, u32 event);
void acpi_video_detect_exit(void);
+/*
+ * Indices in the _BCL method response: the first two items are special,
+ * the rest are all supported levels.
+ *
+ * See page 575 of the ACPI spec 3.0
+ */
+enum acpi_video_level_idx {
+ ACPI_VIDEO_AC_LEVEL, /* level when machine has full power */
+ ACPI_VIDEO_BATTERY_LEVEL, /* level when machine is on batteries */
+ ACPI_VIDEO_FIRST_LEVEL, /* actual supported levels begin here */
+};
+
static const struct acpi_device_id video_device_ids[] = {
{ACPI_VIDEO_HID, 0},
{"", 0},
@@ -132,7 +148,15 @@
the VGA device. */
u32 pipe_id:3; /* For VGA multiple-head devices. */
u32 reserved:10; /* Must be 0 */
- u32 device_id_scheme:1; /* Device ID Scheme */
+
+ /*
+ * The device ID might not actually follow the scheme described by this
+ * struct acpi_video_device_attrib. If it does, then this bit
+ * device_id_scheme is set; otherwise, other fields should be ignored.
+ *
+ * (but also see the global flag device_id_scheme)
+ */
+ u32 device_id_scheme:1;
};
struct acpi_video_enumerated_device {
@@ -217,20 +241,16 @@
if (acpi_video_device_lcd_get_level_current(vd, &cur_level, false))
return -EINVAL;
- for (i = 2; i < vd->brightness->count; i++) {
+ for (i = ACPI_VIDEO_FIRST_LEVEL; i < vd->brightness->count; i++) {
if (vd->brightness->levels[i] == cur_level)
- /*
- * The first two entries are special - see page 575
- * of the ACPI spec 3.0
- */
- return i - 2;
+ return i - ACPI_VIDEO_FIRST_LEVEL;
}
return 0;
}
static int acpi_video_set_brightness(struct backlight_device *bd)
{
- int request_level = bd->props.brightness + 2;
+ int request_level = bd->props.brightness + ACPI_VIDEO_FIRST_LEVEL;
struct acpi_video_device *vd = bl_get_data(bd);
cancel_delayed_work(&vd->switch_brightness_work);
@@ -244,18 +264,18 @@
};
/* thermal cooling device callbacks */
-static int video_get_max_state(struct thermal_cooling_device *cooling_dev, unsigned
- long *state)
+static int video_get_max_state(struct thermal_cooling_device *cooling_dev,
+ unsigned long *state)
{
struct acpi_device *device = cooling_dev->devdata;
struct acpi_video_device *video = acpi_driver_data(device);
- *state = video->brightness->count - 3;
+ *state = video->brightness->count - ACPI_VIDEO_FIRST_LEVEL - 1;
return 0;
}
-static int video_get_cur_state(struct thermal_cooling_device *cooling_dev, unsigned
- long *state)
+static int video_get_cur_state(struct thermal_cooling_device *cooling_dev,
+ unsigned long *state)
{
struct acpi_device *device = cooling_dev->devdata;
struct acpi_video_device *video = acpi_driver_data(device);
@@ -264,7 +284,8 @@
if (acpi_video_device_lcd_get_level_current(video, &level, false))
return -EINVAL;
- for (offset = 2; offset < video->brightness->count; offset++)
+ for (offset = ACPI_VIDEO_FIRST_LEVEL; offset < video->brightness->count;
+ offset++)
if (level == video->brightness->levels[offset]) {
*state = video->brightness->count - offset - 1;
return 0;
@@ -280,7 +301,7 @@
struct acpi_video_device *video = acpi_driver_data(device);
int level;
- if (state >= video->brightness->count - 2)
+ if (state >= video->brightness->count - ACPI_VIDEO_FIRST_LEVEL)
return -EINVAL;
state = video->brightness->count - state;
@@ -345,10 +366,12 @@
}
device->brightness->curr = level;
- for (state = 2; state < device->brightness->count; state++)
+ for (state = ACPI_VIDEO_FIRST_LEVEL; state < device->brightness->count;
+ state++)
if (level == device->brightness->levels[state]) {
if (device->backlight)
- device->backlight->props.brightness = state - 2;
+ device->backlight->props.brightness =
+ state - ACPI_VIDEO_FIRST_LEVEL;
return 0;
}
@@ -530,14 +553,16 @@
if (device->brightness->flags._BQC_use_index) {
/*
- * _BQC returns an index that doesn't account for
- * the first 2 items with special meaning, so we need
- * to compensate for that by offsetting ourselves
+ * _BQC returns an index that doesn't account for the first 2
+ * items with special meaning (see enum acpi_video_level_idx),
+ * so we need to compensate for that by offsetting ourselves
*/
if (device->brightness->flags._BCL_reversed)
- bqc_value = device->brightness->count - 3 - bqc_value;
+ bqc_value = device->brightness->count -
+ ACPI_VIDEO_FIRST_LEVEL - 1 - bqc_value;
- level = device->brightness->levels[bqc_value + 2];
+ level = device->brightness->levels[bqc_value +
+ ACPI_VIDEO_FIRST_LEVEL];
} else {
level = bqc_value;
}
@@ -571,7 +596,8 @@
*level = acpi_video_bqc_value_to_level(device, *level);
- for (i = 2; i < device->brightness->count; i++)
+ for (i = ACPI_VIDEO_FIRST_LEVEL;
+ i < device->brightness->count; i++)
if (device->brightness->levels[i] == *level) {
device->brightness->curr = *level;
return 0;
@@ -714,9 +740,37 @@
/*
* Some systems always report current brightness level as maximum
- * through _BQC, we need to test another value for them.
+ * through _BQC, we need to test another value for them. However,
+ * there is a subtlety:
+ *
+ * If the _BCL package ordering is descending, the first level
+ * (br->levels[2]) is likely to be 0, and if the number of levels
+ * matches the number of steps, we might confuse a returned level to
+ * mean the index.
+ *
+ * For example:
+ *
+ * current_level = max_level = 100
+ * test_level = 0
+ * returned level = 100
+ *
+ * In this case 100 means the level, not the index, and _BCM failed.
+ * Still, if the _BCL package ordering is descending, the index of
+ * level 0 is also 100, so we assume _BQC is indexed, when it's not.
+ *
+ * This causes all _BQC calls to return bogus values causing weird
+ * behavior from the user's perspective. For example:
+ *
+ * xbacklight -set 10; xbacklight -set 20;
+ *
+ * would flash to 90% and then slowly down to the desired level (20).
+ *
+ * The solution is simple; test anything other than the first level
+ * (e.g. 1).
*/
- test_level = current_level == max_level ? br->levels[3] : max_level;
+ test_level = current_level == max_level
+ ? br->levels[ACPI_VIDEO_FIRST_LEVEL + 1]
+ : max_level;
result = acpi_video_device_lcd_set_level(device, test_level);
if (result)
@@ -730,8 +784,8 @@
/* buggy _BQC found, need to find out if it uses index */
if (level < br->count) {
if (br->flags._BCL_reversed)
- level = br->count - 3 - level;
- if (br->levels[level + 2] == test_level)
+ level = br->count - ACPI_VIDEO_FIRST_LEVEL - 1 - level;
+ if (br->levels[level + ACPI_VIDEO_FIRST_LEVEL] == test_level)
br->flags._BQC_use_index = 1;
}
@@ -761,7 +815,7 @@
goto out;
}
- if (obj->package.count < 2) {
+ if (obj->package.count < ACPI_VIDEO_FIRST_LEVEL) {
result = -EINVAL;
goto out;
}
@@ -773,8 +827,13 @@
goto out;
}
- br->levels = kmalloc((obj->package.count + 2) * sizeof *(br->levels),
- GFP_KERNEL);
+ /*
+ * Note that we have to reserve 2 extra items (ACPI_VIDEO_FIRST_LEVEL),
+ * in order to account for buggy BIOS which don't export the first two
+ * special levels (see below)
+ */
+ br->levels = kmalloc((obj->package.count + ACPI_VIDEO_FIRST_LEVEL) *
+ sizeof(*br->levels), GFP_KERNEL);
if (!br->levels) {
result = -ENOMEM;
goto out_free;
@@ -788,7 +847,8 @@
}
value = (u32) o->integer.value;
/* Skip duplicate entries */
- if (count > 2 && br->levels[count - 1] == value)
+ if (count > ACPI_VIDEO_FIRST_LEVEL
+ && br->levels[count - 1] == value)
continue;
br->levels[count] = value;
@@ -804,27 +864,30 @@
* In this case, the first two elements in _BCL packages
* are also supported brightness levels that OS should take care of.
*/
- for (i = 2; i < count; i++) {
- if (br->levels[i] == br->levels[0])
+ for (i = ACPI_VIDEO_FIRST_LEVEL; i < count; i++) {
+ if (br->levels[i] == br->levels[ACPI_VIDEO_AC_LEVEL])
level_ac_battery++;
- if (br->levels[i] == br->levels[1])
+ if (br->levels[i] == br->levels[ACPI_VIDEO_BATTERY_LEVEL])
level_ac_battery++;
}
- if (level_ac_battery < 2) {
- level_ac_battery = 2 - level_ac_battery;
+ if (level_ac_battery < ACPI_VIDEO_FIRST_LEVEL) {
+ level_ac_battery = ACPI_VIDEO_FIRST_LEVEL - level_ac_battery;
br->flags._BCL_no_ac_battery_levels = 1;
- for (i = (count - 1 + level_ac_battery); i >= 2; i--)
+ for (i = (count - 1 + level_ac_battery);
+ i >= ACPI_VIDEO_FIRST_LEVEL; i--)
br->levels[i] = br->levels[i - level_ac_battery];
count += level_ac_battery;
- } else if (level_ac_battery > 2)
+ } else if (level_ac_battery > ACPI_VIDEO_FIRST_LEVEL)
ACPI_ERROR((AE_INFO, "Too many duplicates in _BCL package"));
/* Check if the _BCL package is in a reversed order */
- if (max_level == br->levels[2]) {
+ if (max_level == br->levels[ACPI_VIDEO_FIRST_LEVEL]) {
br->flags._BCL_reversed = 1;
- sort(&br->levels[2], count - 2, sizeof(br->levels[2]),
- acpi_video_cmp_level, NULL);
+ sort(&br->levels[ACPI_VIDEO_FIRST_LEVEL],
+ count - ACPI_VIDEO_FIRST_LEVEL,
+ sizeof(br->levels[ACPI_VIDEO_FIRST_LEVEL]),
+ acpi_video_cmp_level, NULL);
} else if (max_level != br->levels[count - 1])
ACPI_ERROR((AE_INFO,
"Found unordered _BCL package"));
@@ -894,7 +957,7 @@
* level_old is invalid (no matter whether it's a level
* or an index). Set the backlight to max_level in this case.
*/
- for (i = 2; i < br->count; i++)
+ for (i = ACPI_VIDEO_FIRST_LEVEL; i < br->count; i++)
if (level == br->levels[i])
break;
if (i == br->count || !level)
@@ -906,7 +969,8 @@
goto out_free_levels;
ACPI_DEBUG_PRINT((ACPI_DB_INFO,
- "found %d brightness levels\n", br->count - 2));
+ "found %d brightness levels\n",
+ br->count - ACPI_VIDEO_FIRST_LEVEL));
return 0;
out_free_levels:
@@ -1297,7 +1361,7 @@
max = max_below = 0;
min = min_above = 255;
/* Find closest level to level_current */
- for (i = 2; i < device->brightness->count; i++) {
+ for (i = ACPI_VIDEO_FIRST_LEVEL; i < device->brightness->count; i++) {
l = device->brightness->levels[i];
if (abs(l - level_current) < abs(delta)) {
delta = l - level_current;
@@ -1307,7 +1371,7 @@
}
/* Ajust level_current to closest available level */
level_current += delta;
- for (i = 2; i < device->brightness->count; i++) {
+ for (i = ACPI_VIDEO_FIRST_LEVEL; i < device->brightness->count; i++) {
l = device->brightness->levels[i];
if (l < min)
min = l;
@@ -1680,7 +1744,8 @@
memset(&props, 0, sizeof(struct backlight_properties));
props.type = BACKLIGHT_FIRMWARE;
- props.max_brightness = device->brightness->count - 3;
+ props.max_brightness =
+ device->brightness->count - ACPI_VIDEO_FIRST_LEVEL - 1;
device->backlight = backlight_device_register(name,
parent,
device,
diff --git a/drivers/acpi/apei/erst.c b/drivers/acpi/apei/erst.c
index ec4f507..7207e5f 100644
--- a/drivers/acpi/apei/erst.c
+++ b/drivers/acpi/apei/erst.c
@@ -925,15 +925,9 @@
static int erst_open_pstore(struct pstore_info *psi);
static int erst_close_pstore(struct pstore_info *psi);
-static ssize_t erst_reader(u64 *id, enum pstore_type_id *type, int *count,
- struct timespec *time, char **buf,
- bool *compressed, ssize_t *ecc_notice_size,
- struct pstore_info *psi);
-static int erst_writer(enum pstore_type_id type, enum kmsg_dump_reason reason,
- u64 *id, unsigned int part, int count, bool compressed,
- size_t size, struct pstore_info *psi);
-static int erst_clearer(enum pstore_type_id type, u64 id, int count,
- struct timespec time, struct pstore_info *psi);
+static ssize_t erst_reader(struct pstore_record *record);
+static int erst_writer(struct pstore_record *record);
+static int erst_clearer(struct pstore_record *record);
static struct pstore_info erst_info = {
.owner = THIS_MODULE,
@@ -986,10 +980,7 @@
return 0;
}
-static ssize_t erst_reader(u64 *id, enum pstore_type_id *type, int *count,
- struct timespec *time, char **buf,
- bool *compressed, ssize_t *ecc_notice_size,
- struct pstore_info *psi)
+static ssize_t erst_reader(struct pstore_record *record)
{
int rc;
ssize_t len = 0;
@@ -1027,42 +1018,40 @@
if (uuid_le_cmp(rcd->hdr.creator_id, CPER_CREATOR_PSTORE) != 0)
goto skip;
- *buf = kmalloc(len, GFP_KERNEL);
- if (*buf == NULL) {
+ record->buf = kmalloc(len, GFP_KERNEL);
+ if (record->buf == NULL) {
rc = -ENOMEM;
goto out;
}
- memcpy(*buf, rcd->data, len - sizeof(*rcd));
- *id = record_id;
- *compressed = false;
- *ecc_notice_size = 0;
+ memcpy(record->buf, rcd->data, len - sizeof(*rcd));
+ record->id = record_id;
+ record->compressed = false;
+ record->ecc_notice_size = 0;
if (uuid_le_cmp(rcd->sec_hdr.section_type,
CPER_SECTION_TYPE_DMESG_Z) == 0) {
- *type = PSTORE_TYPE_DMESG;
- *compressed = true;
+ record->type = PSTORE_TYPE_DMESG;
+ record->compressed = true;
} else if (uuid_le_cmp(rcd->sec_hdr.section_type,
CPER_SECTION_TYPE_DMESG) == 0)
- *type = PSTORE_TYPE_DMESG;
+ record->type = PSTORE_TYPE_DMESG;
else if (uuid_le_cmp(rcd->sec_hdr.section_type,
CPER_SECTION_TYPE_MCE) == 0)
- *type = PSTORE_TYPE_MCE;
+ record->type = PSTORE_TYPE_MCE;
else
- *type = PSTORE_TYPE_UNKNOWN;
+ record->type = PSTORE_TYPE_UNKNOWN;
if (rcd->hdr.validation_bits & CPER_VALID_TIMESTAMP)
- time->tv_sec = rcd->hdr.timestamp;
+ record->time.tv_sec = rcd->hdr.timestamp;
else
- time->tv_sec = 0;
- time->tv_nsec = 0;
+ record->time.tv_sec = 0;
+ record->time.tv_nsec = 0;
out:
kfree(rcd);
return (rc < 0) ? rc : (len - sizeof(*rcd));
}
-static int erst_writer(enum pstore_type_id type, enum kmsg_dump_reason reason,
- u64 *id, unsigned int part, int count, bool compressed,
- size_t size, struct pstore_info *psi)
+static int erst_writer(struct pstore_record *record)
{
struct cper_pstore_record *rcd = (struct cper_pstore_record *)
(erst_info.buf - sizeof(*rcd));
@@ -1077,21 +1066,21 @@
/* timestamp valid. platform_id, partition_id are invalid */
rcd->hdr.validation_bits = CPER_VALID_TIMESTAMP;
rcd->hdr.timestamp = get_seconds();
- rcd->hdr.record_length = sizeof(*rcd) + size;
+ rcd->hdr.record_length = sizeof(*rcd) + record->size;
rcd->hdr.creator_id = CPER_CREATOR_PSTORE;
rcd->hdr.notification_type = CPER_NOTIFY_MCE;
rcd->hdr.record_id = cper_next_record_id();
rcd->hdr.flags = CPER_HW_ERROR_FLAGS_PREVERR;
rcd->sec_hdr.section_offset = sizeof(*rcd);
- rcd->sec_hdr.section_length = size;
+ rcd->sec_hdr.section_length = record->size;
rcd->sec_hdr.revision = CPER_SEC_REV;
/* fru_id and fru_text is invalid */
rcd->sec_hdr.validation_bits = 0;
rcd->sec_hdr.flags = CPER_SEC_PRIMARY;
- switch (type) {
+ switch (record->type) {
case PSTORE_TYPE_DMESG:
- if (compressed)
+ if (record->compressed)
rcd->sec_hdr.section_type = CPER_SECTION_TYPE_DMESG_Z;
else
rcd->sec_hdr.section_type = CPER_SECTION_TYPE_DMESG;
@@ -1105,15 +1094,14 @@
rcd->sec_hdr.section_severity = CPER_SEV_FATAL;
ret = erst_write(&rcd->hdr);
- *id = rcd->hdr.record_id;
+ record->id = rcd->hdr.record_id;
return ret;
}
-static int erst_clearer(enum pstore_type_id type, u64 id, int count,
- struct timespec time, struct pstore_info *psi)
+static int erst_clearer(struct pstore_record *record)
{
- return erst_clear(id);
+ return erst_clear(record->id);
}
static int __init erst_init(void)
diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c
index 79b3c9c..d0855c0 100644
--- a/drivers/acpi/apei/ghes.c
+++ b/drivers/acpi/apei/ghes.c
@@ -1005,9 +1005,8 @@
switch (generic->notify.type) {
case ACPI_HEST_NOTIFY_POLLED:
- ghes->timer.function = ghes_poll_func;
- ghes->timer.data = (unsigned long)ghes;
- init_timer_deferrable(&ghes->timer);
+ setup_deferrable_timer(&ghes->timer, ghes_poll_func,
+ (unsigned long)ghes);
ghes_add_timer(ghes);
break;
case ACPI_HEST_NOTIFY_EXTERNAL:
diff --git a/drivers/acpi/arm64/Kconfig b/drivers/acpi/arm64/Kconfig
index 4616da4..5a6f80f 100644
--- a/drivers/acpi/arm64/Kconfig
+++ b/drivers/acpi/arm64/Kconfig
@@ -4,3 +4,6 @@
config ACPI_IORT
bool
+
+config ACPI_GTDT
+ bool
diff --git a/drivers/acpi/arm64/Makefile b/drivers/acpi/arm64/Makefile
index 72331f2..1017def 100644
--- a/drivers/acpi/arm64/Makefile
+++ b/drivers/acpi/arm64/Makefile
@@ -1 +1,2 @@
obj-$(CONFIG_ACPI_IORT) += iort.o
+obj-$(CONFIG_ACPI_GTDT) += gtdt.o
diff --git a/drivers/acpi/arm64/gtdt.c b/drivers/acpi/arm64/gtdt.c
new file mode 100644
index 0000000..597a737
--- /dev/null
+++ b/drivers/acpi/arm64/gtdt.c
@@ -0,0 +1,417 @@
+/*
+ * ARM Specific GTDT table Support
+ *
+ * Copyright (C) 2016, Linaro Ltd.
+ * Author: Daniel Lezcano <daniel.lezcano@linaro.org>
+ * Fu Wei <fu.wei@linaro.org>
+ * Hanjun Guo <hanjun.guo@linaro.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/acpi.h>
+#include <linux/init.h>
+#include <linux/irqdomain.h>
+#include <linux/kernel.h>
+#include <linux/platform_device.h>
+
+#include <clocksource/arm_arch_timer.h>
+
+#undef pr_fmt
+#define pr_fmt(fmt) "ACPI GTDT: " fmt
+
+/**
+ * struct acpi_gtdt_descriptor - Store the key info of GTDT for all functions
+ * @gtdt: The pointer to the struct acpi_table_gtdt of GTDT table.
+ * @gtdt_end: The pointer to the end of GTDT table.
+ * @platform_timer: The pointer to the start of Platform Timer Structure
+ *
+ * The struct store the key info of GTDT table, it should be initialized by
+ * acpi_gtdt_init.
+ */
+struct acpi_gtdt_descriptor {
+ struct acpi_table_gtdt *gtdt;
+ void *gtdt_end;
+ void *platform_timer;
+};
+
+static struct acpi_gtdt_descriptor acpi_gtdt_desc __initdata;
+
+static inline void *next_platform_timer(void *platform_timer)
+{
+ struct acpi_gtdt_header *gh = platform_timer;
+
+ platform_timer += gh->length;
+ if (platform_timer < acpi_gtdt_desc.gtdt_end)
+ return platform_timer;
+
+ return NULL;
+}
+
+#define for_each_platform_timer(_g) \
+ for (_g = acpi_gtdt_desc.platform_timer; _g; \
+ _g = next_platform_timer(_g))
+
+static inline bool is_timer_block(void *platform_timer)
+{
+ struct acpi_gtdt_header *gh = platform_timer;
+
+ return gh->type == ACPI_GTDT_TYPE_TIMER_BLOCK;
+}
+
+static inline bool is_non_secure_watchdog(void *platform_timer)
+{
+ struct acpi_gtdt_header *gh = platform_timer;
+ struct acpi_gtdt_watchdog *wd = platform_timer;
+
+ if (gh->type != ACPI_GTDT_TYPE_WATCHDOG)
+ return false;
+
+ return !(wd->timer_flags & ACPI_GTDT_WATCHDOG_SECURE);
+}
+
+static int __init map_gt_gsi(u32 interrupt, u32 flags)
+{
+ int trigger, polarity;
+
+ trigger = (flags & ACPI_GTDT_INTERRUPT_MODE) ? ACPI_EDGE_SENSITIVE
+ : ACPI_LEVEL_SENSITIVE;
+
+ polarity = (flags & ACPI_GTDT_INTERRUPT_POLARITY) ? ACPI_ACTIVE_LOW
+ : ACPI_ACTIVE_HIGH;
+
+ return acpi_register_gsi(NULL, interrupt, trigger, polarity);
+}
+
+/**
+ * acpi_gtdt_map_ppi() - Map the PPIs of per-cpu arch_timer.
+ * @type: the type of PPI.
+ *
+ * Note: Secure state is not managed by the kernel on ARM64 systems.
+ * So we only handle the non-secure timer PPIs,
+ * ARCH_TIMER_PHYS_SECURE_PPI is treated as invalid type.
+ *
+ * Return: the mapped PPI value, 0 if error.
+ */
+int __init acpi_gtdt_map_ppi(int type)
+{
+ struct acpi_table_gtdt *gtdt = acpi_gtdt_desc.gtdt;
+
+ switch (type) {
+ case ARCH_TIMER_PHYS_NONSECURE_PPI:
+ return map_gt_gsi(gtdt->non_secure_el1_interrupt,
+ gtdt->non_secure_el1_flags);
+ case ARCH_TIMER_VIRT_PPI:
+ return map_gt_gsi(gtdt->virtual_timer_interrupt,
+ gtdt->virtual_timer_flags);
+
+ case ARCH_TIMER_HYP_PPI:
+ return map_gt_gsi(gtdt->non_secure_el2_interrupt,
+ gtdt->non_secure_el2_flags);
+ default:
+ pr_err("Failed to map timer interrupt: invalid type.\n");
+ }
+
+ return 0;
+}
+
+/**
+ * acpi_gtdt_c3stop() - Got c3stop info from GTDT according to the type of PPI.
+ * @type: the type of PPI.
+ *
+ * Return: true if the timer HW state is lost when a CPU enters an idle state,
+ * false otherwise
+ */
+bool __init acpi_gtdt_c3stop(int type)
+{
+ struct acpi_table_gtdt *gtdt = acpi_gtdt_desc.gtdt;
+
+ switch (type) {
+ case ARCH_TIMER_PHYS_NONSECURE_PPI:
+ return !(gtdt->non_secure_el1_flags & ACPI_GTDT_ALWAYS_ON);
+
+ case ARCH_TIMER_VIRT_PPI:
+ return !(gtdt->virtual_timer_flags & ACPI_GTDT_ALWAYS_ON);
+
+ case ARCH_TIMER_HYP_PPI:
+ return !(gtdt->non_secure_el2_flags & ACPI_GTDT_ALWAYS_ON);
+
+ default:
+ pr_err("Failed to get c3stop info: invalid type.\n");
+ }
+
+ return false;
+}
+
+/**
+ * acpi_gtdt_init() - Get the info of GTDT table to prepare for further init.
+ * @table: The pointer to GTDT table.
+ * @platform_timer_count: It points to a integer variable which is used
+ * for storing the number of platform timers.
+ * This pointer could be NULL, if the caller
+ * doesn't need this info.
+ *
+ * Return: 0 if success, -EINVAL if error.
+ */
+int __init acpi_gtdt_init(struct acpi_table_header *table,
+ int *platform_timer_count)
+{
+ void *platform_timer;
+ struct acpi_table_gtdt *gtdt;
+
+ gtdt = container_of(table, struct acpi_table_gtdt, header);
+ acpi_gtdt_desc.gtdt = gtdt;
+ acpi_gtdt_desc.gtdt_end = (void *)table + table->length;
+ acpi_gtdt_desc.platform_timer = NULL;
+ if (platform_timer_count)
+ *platform_timer_count = 0;
+
+ if (table->revision < 2) {
+ pr_warn("Revision:%d doesn't support Platform Timers.\n",
+ table->revision);
+ return 0;
+ }
+
+ if (!gtdt->platform_timer_count) {
+ pr_debug("No Platform Timer.\n");
+ return 0;
+ }
+
+ platform_timer = (void *)gtdt + gtdt->platform_timer_offset;
+ if (platform_timer < (void *)table + sizeof(struct acpi_table_gtdt)) {
+ pr_err(FW_BUG "invalid timer data.\n");
+ return -EINVAL;
+ }
+ acpi_gtdt_desc.platform_timer = platform_timer;
+ if (platform_timer_count)
+ *platform_timer_count = gtdt->platform_timer_count;
+
+ return 0;
+}
+
+static int __init gtdt_parse_timer_block(struct acpi_gtdt_timer_block *block,
+ struct arch_timer_mem *timer_mem)
+{
+ int i;
+ struct arch_timer_mem_frame *frame;
+ struct acpi_gtdt_timer_entry *gtdt_frame;
+
+ if (!block->timer_count) {
+ pr_err(FW_BUG "GT block present, but frame count is zero.");
+ return -ENODEV;
+ }
+
+ if (block->timer_count > ARCH_TIMER_MEM_MAX_FRAMES) {
+ pr_err(FW_BUG "GT block lists %d frames, ACPI spec only allows 8\n",
+ block->timer_count);
+ return -EINVAL;
+ }
+
+ timer_mem->cntctlbase = (phys_addr_t)block->block_address;
+ /*
+ * The CNTCTLBase frame is 4KB (register offsets 0x000 - 0xFFC).
+ * See ARM DDI 0487A.k_iss10775, page I1-5129, Table I1-3
+ * "CNTCTLBase memory map".
+ */
+ timer_mem->size = SZ_4K;
+
+ gtdt_frame = (void *)block + block->timer_offset;
+ if (gtdt_frame + block->timer_count != (void *)block + block->header.length)
+ return -EINVAL;
+
+ /*
+ * Get the GT timer Frame data for every GT Block Timer
+ */
+ for (i = 0; i < block->timer_count; i++, gtdt_frame++) {
+ if (gtdt_frame->common_flags & ACPI_GTDT_GT_IS_SECURE_TIMER)
+ continue;
+ if (gtdt_frame->frame_number >= ARCH_TIMER_MEM_MAX_FRAMES ||
+ !gtdt_frame->base_address || !gtdt_frame->timer_interrupt)
+ goto error;
+
+ frame = &timer_mem->frame[gtdt_frame->frame_number];
+
+ /* duplicate frame */
+ if (frame->valid)
+ goto error;
+
+ frame->phys_irq = map_gt_gsi(gtdt_frame->timer_interrupt,
+ gtdt_frame->timer_flags);
+ if (frame->phys_irq <= 0) {
+ pr_warn("failed to map physical timer irq in frame %d.\n",
+ gtdt_frame->frame_number);
+ goto error;
+ }
+
+ if (gtdt_frame->virtual_timer_interrupt) {
+ frame->virt_irq =
+ map_gt_gsi(gtdt_frame->virtual_timer_interrupt,
+ gtdt_frame->virtual_timer_flags);
+ if (frame->virt_irq <= 0) {
+ pr_warn("failed to map virtual timer irq in frame %d.\n",
+ gtdt_frame->frame_number);
+ goto error;
+ }
+ } else {
+ pr_debug("virtual timer in frame %d not implemented.\n",
+ gtdt_frame->frame_number);
+ }
+
+ frame->cntbase = gtdt_frame->base_address;
+ /*
+ * The CNTBaseN frame is 4KB (register offsets 0x000 - 0xFFC).
+ * See ARM DDI 0487A.k_iss10775, page I1-5130, Table I1-4
+ * "CNTBaseN memory map".
+ */
+ frame->size = SZ_4K;
+ frame->valid = true;
+ }
+
+ return 0;
+
+error:
+ do {
+ if (gtdt_frame->common_flags & ACPI_GTDT_GT_IS_SECURE_TIMER ||
+ gtdt_frame->frame_number >= ARCH_TIMER_MEM_MAX_FRAMES)
+ continue;
+
+ frame = &timer_mem->frame[gtdt_frame->frame_number];
+
+ if (frame->phys_irq > 0)
+ acpi_unregister_gsi(gtdt_frame->timer_interrupt);
+ frame->phys_irq = 0;
+
+ if (frame->virt_irq > 0)
+ acpi_unregister_gsi(gtdt_frame->virtual_timer_interrupt);
+ frame->virt_irq = 0;
+ } while (i-- >= 0 && gtdt_frame--);
+
+ return -EINVAL;
+}
+
+/**
+ * acpi_arch_timer_mem_init() - Get the info of all GT blocks in GTDT table.
+ * @timer_mem: The pointer to the array of struct arch_timer_mem for returning
+ * the result of parsing. The element number of this array should
+ * be platform_timer_count(the total number of platform timers).
+ * @timer_count: It points to a integer variable which is used for storing the
+ * number of GT blocks we have parsed.
+ *
+ * Return: 0 if success, -EINVAL/-ENODEV if error.
+ */
+int __init acpi_arch_timer_mem_init(struct arch_timer_mem *timer_mem,
+ int *timer_count)
+{
+ int ret;
+ void *platform_timer;
+
+ *timer_count = 0;
+ for_each_platform_timer(platform_timer) {
+ if (is_timer_block(platform_timer)) {
+ ret = gtdt_parse_timer_block(platform_timer, timer_mem);
+ if (ret)
+ return ret;
+ timer_mem++;
+ (*timer_count)++;
+ }
+ }
+
+ if (*timer_count)
+ pr_info("found %d memory-mapped timer block(s).\n",
+ *timer_count);
+
+ return 0;
+}
+
+/*
+ * Initialize a SBSA generic Watchdog platform device info from GTDT
+ */
+static int __init gtdt_import_sbsa_gwdt(struct acpi_gtdt_watchdog *wd,
+ int index)
+{
+ struct platform_device *pdev;
+ int irq = map_gt_gsi(wd->timer_interrupt, wd->timer_flags);
+
+ /*
+ * According to SBSA specification the size of refresh and control
+ * frames of SBSA Generic Watchdog is SZ_4K(Offset 0x000 – 0xFFF).
+ */
+ struct resource res[] = {
+ DEFINE_RES_MEM(wd->control_frame_address, SZ_4K),
+ DEFINE_RES_MEM(wd->refresh_frame_address, SZ_4K),
+ DEFINE_RES_IRQ(irq),
+ };
+ int nr_res = ARRAY_SIZE(res);
+
+ pr_debug("found a Watchdog (0x%llx/0x%llx gsi:%u flags:0x%x).\n",
+ wd->refresh_frame_address, wd->control_frame_address,
+ wd->timer_interrupt, wd->timer_flags);
+
+ if (!(wd->refresh_frame_address && wd->control_frame_address)) {
+ pr_err(FW_BUG "failed to get the Watchdog base address.\n");
+ acpi_unregister_gsi(wd->timer_interrupt);
+ return -EINVAL;
+ }
+
+ if (irq <= 0) {
+ pr_warn("failed to map the Watchdog interrupt.\n");
+ nr_res--;
+ }
+
+ /*
+ * Add a platform device named "sbsa-gwdt" to match the platform driver.
+ * "sbsa-gwdt": SBSA(Server Base System Architecture) Generic Watchdog
+ * The platform driver can get device info below by matching this name.
+ */
+ pdev = platform_device_register_simple("sbsa-gwdt", index, res, nr_res);
+ if (IS_ERR(pdev)) {
+ acpi_unregister_gsi(wd->timer_interrupt);
+ return PTR_ERR(pdev);
+ }
+
+ return 0;
+}
+
+static int __init gtdt_sbsa_gwdt_init(void)
+{
+ void *platform_timer;
+ struct acpi_table_header *table;
+ int ret, timer_count, gwdt_count = 0;
+
+ if (acpi_disabled)
+ return 0;
+
+ if (ACPI_FAILURE(acpi_get_table(ACPI_SIG_GTDT, 0, &table)))
+ return -EINVAL;
+
+ /*
+ * Note: Even though the global variable acpi_gtdt_desc has been
+ * initialized by acpi_gtdt_init() while initializing the arch timers,
+ * when we call this function to get SBSA watchdogs info from GTDT, the
+ * pointers stashed in it are stale (since they are early temporary
+ * mappings carried out before acpi_permanent_mmap is set) and we need
+ * to re-initialize them with permanent mapped pointer values to let the
+ * GTDT parsing possible.
+ */
+ ret = acpi_gtdt_init(table, &timer_count);
+ if (ret || !timer_count)
+ return ret;
+
+ for_each_platform_timer(platform_timer) {
+ if (is_non_secure_watchdog(platform_timer)) {
+ ret = gtdt_import_sbsa_gwdt(platform_timer, gwdt_count);
+ if (ret)
+ break;
+ gwdt_count++;
+ }
+ }
+
+ if (gwdt_count)
+ pr_info("found %d SBSA generic Watchdog(s).\n", gwdt_count);
+
+ return ret;
+}
+
+device_initcall(gtdt_sbsa_gwdt_init);
diff --git a/drivers/acpi/battery.c b/drivers/acpi/battery.c
index 4ef1e46..d42eeef 100644
--- a/drivers/acpi/battery.c
+++ b/drivers/acpi/battery.c
@@ -67,6 +67,7 @@
MODULE_LICENSE("GPL");
static async_cookie_t async_cookie;
+static bool battery_driver_registered;
static int battery_bix_broken_package;
static int battery_notification_delay_ms;
static unsigned int cache_time = 1000;
@@ -93,6 +94,11 @@
MODULE_DEVICE_TABLE(acpi, battery_device_ids);
+/* Lists of PMIC ACPI HIDs with an (often better) native battery driver */
+static const char * const acpi_battery_blacklist[] = {
+ "INT33F4", /* X-Powers AXP288 PMIC */
+};
+
enum {
ACPI_BATTERY_ALARM_PRESENT,
ACPI_BATTERY_XINFO_PRESENT,
@@ -1315,8 +1321,17 @@
static void __init acpi_battery_init_async(void *unused, async_cookie_t cookie)
{
+ unsigned int i;
int result;
+ for (i = 0; i < ARRAY_SIZE(acpi_battery_blacklist); i++)
+ if (acpi_dev_present(acpi_battery_blacklist[i], "1", -1)) {
+ pr_info(PREFIX ACPI_BATTERY_DEVICE_NAME
+ ": found native %s PMIC, not loading\n",
+ acpi_battery_blacklist[i]);
+ return;
+ }
+
dmi_check_system(bat_dmi_table);
#ifdef CONFIG_ACPI_PROCFS_POWER
@@ -1329,6 +1344,7 @@
if (result < 0)
acpi_unlock_battery_dir(acpi_battery_dir);
#endif
+ battery_driver_registered = (result == 0);
}
static int __init acpi_battery_init(void)
@@ -1343,9 +1359,11 @@
static void __exit acpi_battery_exit(void)
{
async_synchronize_cookie(async_cookie + 1);
- acpi_bus_unregister_driver(&acpi_battery_driver);
+ if (battery_driver_registered)
+ acpi_bus_unregister_driver(&acpi_battery_driver);
#ifdef CONFIG_ACPI_PROCFS_POWER
- acpi_unlock_battery_dir(acpi_battery_dir);
+ if (acpi_battery_dir)
+ acpi_unlock_battery_dir(acpi_battery_dir);
#endif
}
diff --git a/drivers/acpi/bgrt.c b/drivers/acpi/bgrt.c
index ca28aa57..df1c629 100644
--- a/drivers/acpi/bgrt.c
+++ b/drivers/acpi/bgrt.c
@@ -81,6 +81,12 @@
.bin_attrs = bgrt_bin_attributes,
};
+int __init acpi_parse_bgrt(struct acpi_table_header *table)
+{
+ efi_bgrt_init(table);
+ return 0;
+}
+
static int __init bgrt_init(void)
{
int ret;
diff --git a/drivers/acpi/blacklist.c b/drivers/acpi/blacklist.c
index 4421f7c..bb542ac 100644
--- a/drivers/acpi/blacklist.c
+++ b/drivers/acpi/blacklist.c
@@ -188,6 +188,14 @@
DMI_MATCH(DMI_PRODUCT_NAME, "Latitude 3350"),
},
},
+ {
+ .callback = dmi_enable_rev_override,
+ .ident = "DELL Inspiron 7537",
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc."),
+ DMI_MATCH(DMI_PRODUCT_NAME, "Inspiron 7537"),
+ },
+ },
#endif
{}
};
diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi/cppc_acpi.c
index 3ca0729..6cbe603 100644
--- a/drivers/acpi/cppc_acpi.c
+++ b/drivers/acpi/cppc_acpi.c
@@ -132,49 +132,54 @@
#define to_cpc_desc(a) container_of(a, struct cpc_desc, kobj)
+#define show_cppc_data(access_fn, struct_name, member_name) \
+ static ssize_t show_##member_name(struct kobject *kobj, \
+ struct attribute *attr, char *buf) \
+ { \
+ struct cpc_desc *cpc_ptr = to_cpc_desc(kobj); \
+ struct struct_name st_name = {0}; \
+ int ret; \
+ \
+ ret = access_fn(cpc_ptr->cpu_id, &st_name); \
+ if (ret) \
+ return ret; \
+ \
+ return scnprintf(buf, PAGE_SIZE, "%llu\n", \
+ (u64)st_name.member_name); \
+ } \
+ define_one_cppc_ro(member_name)
+
+show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, highest_perf);
+show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, lowest_perf);
+show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, nominal_perf);
+show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, lowest_nonlinear_perf);
+show_cppc_data(cppc_get_perf_ctrs, cppc_perf_fb_ctrs, reference_perf);
+show_cppc_data(cppc_get_perf_ctrs, cppc_perf_fb_ctrs, wraparound_time);
+
static ssize_t show_feedback_ctrs(struct kobject *kobj,
struct attribute *attr, char *buf)
{
struct cpc_desc *cpc_ptr = to_cpc_desc(kobj);
struct cppc_perf_fb_ctrs fb_ctrs = {0};
+ int ret;
- cppc_get_perf_ctrs(cpc_ptr->cpu_id, &fb_ctrs);
+ ret = cppc_get_perf_ctrs(cpc_ptr->cpu_id, &fb_ctrs);
+ if (ret)
+ return ret;
return scnprintf(buf, PAGE_SIZE, "ref:%llu del:%llu\n",
fb_ctrs.reference, fb_ctrs.delivered);
}
define_one_cppc_ro(feedback_ctrs);
-static ssize_t show_reference_perf(struct kobject *kobj,
- struct attribute *attr, char *buf)
-{
- struct cpc_desc *cpc_ptr = to_cpc_desc(kobj);
- struct cppc_perf_fb_ctrs fb_ctrs = {0};
-
- cppc_get_perf_ctrs(cpc_ptr->cpu_id, &fb_ctrs);
-
- return scnprintf(buf, PAGE_SIZE, "%llu\n",
- fb_ctrs.reference_perf);
-}
-define_one_cppc_ro(reference_perf);
-
-static ssize_t show_wraparound_time(struct kobject *kobj,
- struct attribute *attr, char *buf)
-{
- struct cpc_desc *cpc_ptr = to_cpc_desc(kobj);
- struct cppc_perf_fb_ctrs fb_ctrs = {0};
-
- cppc_get_perf_ctrs(cpc_ptr->cpu_id, &fb_ctrs);
-
- return scnprintf(buf, PAGE_SIZE, "%llu\n", fb_ctrs.ctr_wrap_time);
-
-}
-define_one_cppc_ro(wraparound_time);
-
static struct attribute *cppc_attrs[] = {
&feedback_ctrs.attr,
&reference_perf.attr,
&wraparound_time.attr,
+ &highest_perf.attr,
+ &lowest_perf.attr,
+ &lowest_nonlinear_perf.attr,
+ &nominal_perf.attr,
NULL
};
@@ -972,9 +977,9 @@
int cppc_get_perf_caps(int cpunum, struct cppc_perf_caps *perf_caps)
{
struct cpc_desc *cpc_desc = per_cpu(cpc_desc_ptr, cpunum);
- struct cpc_register_resource *highest_reg, *lowest_reg, *ref_perf,
- *nom_perf;
- u64 high, low, nom;
+ struct cpc_register_resource *highest_reg, *lowest_reg,
+ *lowest_non_linear_reg, *nominal_reg;
+ u64 high, low, nom, min_nonlinear;
int ret = 0, regs_in_pcc = 0;
if (!cpc_desc) {
@@ -984,12 +989,12 @@
highest_reg = &cpc_desc->cpc_regs[HIGHEST_PERF];
lowest_reg = &cpc_desc->cpc_regs[LOWEST_PERF];
- ref_perf = &cpc_desc->cpc_regs[REFERENCE_PERF];
- nom_perf = &cpc_desc->cpc_regs[NOMINAL_PERF];
+ lowest_non_linear_reg = &cpc_desc->cpc_regs[LOW_NON_LINEAR_PERF];
+ nominal_reg = &cpc_desc->cpc_regs[NOMINAL_PERF];
/* Are any of the regs PCC ?*/
if (CPC_IN_PCC(highest_reg) || CPC_IN_PCC(lowest_reg) ||
- CPC_IN_PCC(ref_perf) || CPC_IN_PCC(nom_perf)) {
+ CPC_IN_PCC(lowest_non_linear_reg) || CPC_IN_PCC(nominal_reg)) {
regs_in_pcc = 1;
down_write(&pcc_data.pcc_lock);
/* Ring doorbell once to update PCC subspace */
@@ -1005,10 +1010,13 @@
cpc_read(cpunum, lowest_reg, &low);
perf_caps->lowest_perf = low;
- cpc_read(cpunum, nom_perf, &nom);
+ cpc_read(cpunum, nominal_reg, &nom);
perf_caps->nominal_perf = nom;
- if (!high || !low || !nom)
+ cpc_read(cpunum, lowest_non_linear_reg, &min_nonlinear);
+ perf_caps->lowest_nonlinear_perf = min_nonlinear;
+
+ if (!high || !low || !nom || !min_nonlinear)
ret = -EFAULT;
out_err:
@@ -1083,7 +1091,7 @@
perf_fb_ctrs->delivered = delivered;
perf_fb_ctrs->reference = reference;
perf_fb_ctrs->reference_perf = ref_perf;
- perf_fb_ctrs->ctr_wrap_time = ctr_wrap_time;
+ perf_fb_ctrs->wraparound_time = ctr_wrap_time;
out_err:
if (regs_in_pcc)
up_write(&pcc_data.pcc_lock);
diff --git a/drivers/acpi/internal.h b/drivers/acpi/internal.h
index f159001..66229ff 100644
--- a/drivers/acpi/internal.h
+++ b/drivers/acpi/internal.h
@@ -65,8 +65,6 @@
#endif
int acpi_rev_override_setup(char *str);
-extern bool acpi_force_hot_remove;
-
void acpi_sysfs_add_hotplug_profile(struct acpi_hotplug_profile *hotplug,
const char *name);
int acpi_scan_add_handler_with_hotplug(struct acpi_scan_handler *handler,
diff --git a/drivers/acpi/pmic/intel_pmic_chtwc.c b/drivers/acpi/pmic/intel_pmic_chtwc.c
new file mode 100644
index 0000000..85636d7
--- /dev/null
+++ b/drivers/acpi/pmic/intel_pmic_chtwc.c
@@ -0,0 +1,280 @@
+/*
+ * Intel CHT Whiskey Cove PMIC operation region driver
+ * Copyright (C) 2017 Hans de Goede <hdegoede@redhat.com>
+ *
+ * Based on various non upstream patches to support the CHT Whiskey Cove PMIC:
+ * Copyright (C) 2013-2015 Intel Corporation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/acpi.h>
+#include <linux/init.h>
+#include <linux/mfd/intel_soc_pmic.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include "intel_pmic.h"
+
+#define CHT_WC_V1P05A_CTRL 0x6e3b
+#define CHT_WC_V1P15_CTRL 0x6e3c
+#define CHT_WC_V1P05A_VSEL 0x6e3d
+#define CHT_WC_V1P15_VSEL 0x6e3e
+#define CHT_WC_V1P8A_CTRL 0x6e56
+#define CHT_WC_V1P8SX_CTRL 0x6e57
+#define CHT_WC_VDDQ_CTRL 0x6e58
+#define CHT_WC_V1P2A_CTRL 0x6e59
+#define CHT_WC_V1P2SX_CTRL 0x6e5a
+#define CHT_WC_V1P8A_VSEL 0x6e5b
+#define CHT_WC_VDDQ_VSEL 0x6e5c
+#define CHT_WC_V2P8SX_CTRL 0x6e5d
+#define CHT_WC_V3P3A_CTRL 0x6e5e
+#define CHT_WC_V3P3SD_CTRL 0x6e5f
+#define CHT_WC_VSDIO_CTRL 0x6e67
+#define CHT_WC_V3P3A_VSEL 0x6e68
+#define CHT_WC_VPROG1A_CTRL 0x6e90
+#define CHT_WC_VPROG1B_CTRL 0x6e91
+#define CHT_WC_VPROG1F_CTRL 0x6e95
+#define CHT_WC_VPROG2D_CTRL 0x6e99
+#define CHT_WC_VPROG3A_CTRL 0x6e9a
+#define CHT_WC_VPROG3B_CTRL 0x6e9b
+#define CHT_WC_VPROG4A_CTRL 0x6e9c
+#define CHT_WC_VPROG4B_CTRL 0x6e9d
+#define CHT_WC_VPROG4C_CTRL 0x6e9e
+#define CHT_WC_VPROG4D_CTRL 0x6e9f
+#define CHT_WC_VPROG5A_CTRL 0x6ea0
+#define CHT_WC_VPROG5B_CTRL 0x6ea1
+#define CHT_WC_VPROG6A_CTRL 0x6ea2
+#define CHT_WC_VPROG6B_CTRL 0x6ea3
+#define CHT_WC_VPROG1A_VSEL 0x6ec0
+#define CHT_WC_VPROG1B_VSEL 0x6ec1
+#define CHT_WC_V1P8SX_VSEL 0x6ec2
+#define CHT_WC_V1P2SX_VSEL 0x6ec3
+#define CHT_WC_V1P2A_VSEL 0x6ec4
+#define CHT_WC_VPROG1F_VSEL 0x6ec5
+#define CHT_WC_VSDIO_VSEL 0x6ec6
+#define CHT_WC_V2P8SX_VSEL 0x6ec7
+#define CHT_WC_V3P3SD_VSEL 0x6ec8
+#define CHT_WC_VPROG2D_VSEL 0x6ec9
+#define CHT_WC_VPROG3A_VSEL 0x6eca
+#define CHT_WC_VPROG3B_VSEL 0x6ecb
+#define CHT_WC_VPROG4A_VSEL 0x6ecc
+#define CHT_WC_VPROG4B_VSEL 0x6ecd
+#define CHT_WC_VPROG4C_VSEL 0x6ece
+#define CHT_WC_VPROG4D_VSEL 0x6ecf
+#define CHT_WC_VPROG5A_VSEL 0x6ed0
+#define CHT_WC_VPROG5B_VSEL 0x6ed1
+#define CHT_WC_VPROG6A_VSEL 0x6ed2
+#define CHT_WC_VPROG6B_VSEL 0x6ed3
+
+/*
+ * Regulator support is based on the non upstream patch:
+ * "regulator: whiskey_cove: implements Whiskey Cove pmic VRF support"
+ * https://github.com/intel-aero/meta-intel-aero/blob/master/recipes-kernel/linux/linux-yocto/0019-regulator-whiskey_cove-implements-WhiskeyCove-pmic-V.patch
+ */
+static struct pmic_table power_table[] = {
+ {
+ .address = 0x0,
+ .reg = CHT_WC_V1P8A_CTRL,
+ .bit = 0x01,
+ }, /* V18A */
+ {
+ .address = 0x04,
+ .reg = CHT_WC_V1P8SX_CTRL,
+ .bit = 0x07,
+ }, /* V18X */
+ {
+ .address = 0x08,
+ .reg = CHT_WC_VDDQ_CTRL,
+ .bit = 0x01,
+ }, /* VDDQ */
+ {
+ .address = 0x0c,
+ .reg = CHT_WC_V1P2A_CTRL,
+ .bit = 0x07,
+ }, /* V12A */
+ {
+ .address = 0x10,
+ .reg = CHT_WC_V1P2SX_CTRL,
+ .bit = 0x07,
+ }, /* V12X */
+ {
+ .address = 0x14,
+ .reg = CHT_WC_V2P8SX_CTRL,
+ .bit = 0x07,
+ }, /* V28X */
+ {
+ .address = 0x18,
+ .reg = CHT_WC_V3P3A_CTRL,
+ .bit = 0x01,
+ }, /* V33A */
+ {
+ .address = 0x1c,
+ .reg = CHT_WC_V3P3SD_CTRL,
+ .bit = 0x07,
+ }, /* V3SD */
+ {
+ .address = 0x20,
+ .reg = CHT_WC_VSDIO_CTRL,
+ .bit = 0x07,
+ }, /* VSD */
+/* {
+ .address = 0x24,
+ .reg = ??,
+ .bit = ??,
+ }, ** VSW2 */
+/* {
+ .address = 0x28,
+ .reg = ??,
+ .bit = ??,
+ }, ** VSW1 */
+/* {
+ .address = 0x2c,
+ .reg = ??,
+ .bit = ??,
+ }, ** VUPY */
+/* {
+ .address = 0x30,
+ .reg = ??,
+ .bit = ??,
+ }, ** VRSO */
+ {
+ .address = 0x34,
+ .reg = CHT_WC_VPROG1A_CTRL,
+ .bit = 0x07,
+ }, /* VP1A */
+ {
+ .address = 0x38,
+ .reg = CHT_WC_VPROG1B_CTRL,
+ .bit = 0x07,
+ }, /* VP1B */
+ {
+ .address = 0x3c,
+ .reg = CHT_WC_VPROG1F_CTRL,
+ .bit = 0x07,
+ }, /* VP1F */
+ {
+ .address = 0x40,
+ .reg = CHT_WC_VPROG2D_CTRL,
+ .bit = 0x07,
+ }, /* VP2D */
+ {
+ .address = 0x44,
+ .reg = CHT_WC_VPROG3A_CTRL,
+ .bit = 0x07,
+ }, /* VP3A */
+ {
+ .address = 0x48,
+ .reg = CHT_WC_VPROG3B_CTRL,
+ .bit = 0x07,
+ }, /* VP3B */
+ {
+ .address = 0x4c,
+ .reg = CHT_WC_VPROG4A_CTRL,
+ .bit = 0x07,
+ }, /* VP4A */
+ {
+ .address = 0x50,
+ .reg = CHT_WC_VPROG4B_CTRL,
+ .bit = 0x07,
+ }, /* VP4B */
+ {
+ .address = 0x54,
+ .reg = CHT_WC_VPROG4C_CTRL,
+ .bit = 0x07,
+ }, /* VP4C */
+ {
+ .address = 0x58,
+ .reg = CHT_WC_VPROG4D_CTRL,
+ .bit = 0x07,
+ }, /* VP4D */
+ {
+ .address = 0x5c,
+ .reg = CHT_WC_VPROG5A_CTRL,
+ .bit = 0x07,
+ }, /* VP5A */
+ {
+ .address = 0x60,
+ .reg = CHT_WC_VPROG5B_CTRL,
+ .bit = 0x07,
+ }, /* VP5B */
+ {
+ .address = 0x64,
+ .reg = CHT_WC_VPROG6A_CTRL,
+ .bit = 0x07,
+ }, /* VP6A */
+ {
+ .address = 0x68,
+ .reg = CHT_WC_VPROG6B_CTRL,
+ .bit = 0x07,
+ }, /* VP6B */
+/* {
+ .address = 0x6c,
+ .reg = ??,
+ .bit = ??,
+ } ** VP7A */
+};
+
+static int intel_cht_wc_pmic_get_power(struct regmap *regmap, int reg,
+ int bit, u64 *value)
+{
+ int data;
+
+ if (regmap_read(regmap, reg, &data))
+ return -EIO;
+
+ *value = (data & bit) ? 1 : 0;
+ return 0;
+}
+
+static int intel_cht_wc_pmic_update_power(struct regmap *regmap, int reg,
+ int bitmask, bool on)
+{
+ return regmap_update_bits(regmap, reg, bitmask, on ? 1 : 0);
+}
+
+/*
+ * The thermal table and ops are empty, we do not support the Thermal opregion
+ * (DPTF) due to lacking documentation.
+ */
+static struct intel_pmic_opregion_data intel_cht_wc_pmic_opregion_data = {
+ .get_power = intel_cht_wc_pmic_get_power,
+ .update_power = intel_cht_wc_pmic_update_power,
+ .power_table = power_table,
+ .power_table_count = ARRAY_SIZE(power_table),
+};
+
+static int intel_cht_wc_pmic_opregion_probe(struct platform_device *pdev)
+{
+ struct intel_soc_pmic *pmic = dev_get_drvdata(pdev->dev.parent);
+
+ return intel_pmic_install_opregion_handler(&pdev->dev,
+ ACPI_HANDLE(pdev->dev.parent),
+ pmic->regmap,
+ &intel_cht_wc_pmic_opregion_data);
+}
+
+static struct platform_device_id cht_wc_opregion_id_table[] = {
+ { .name = "cht_wcove_region" },
+ {},
+};
+MODULE_DEVICE_TABLE(platform, cht_wc_opregion_id_table);
+
+static struct platform_driver intel_cht_wc_pmic_opregion_driver = {
+ .probe = intel_cht_wc_pmic_opregion_probe,
+ .driver = {
+ .name = "cht_whiskey_cove_pmic",
+ },
+ .id_table = cht_wc_opregion_id_table,
+};
+module_platform_driver(intel_cht_wc_pmic_opregion_driver);
+
+MODULE_DESCRIPTION("Intel CHT Whiskey Cove PMIC operation region driver");
+MODULE_AUTHOR("Hans de Goede <hdegoede@redhat.com>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/acpi/pmic/intel_pmic_xpower.c b/drivers/acpi/pmic/intel_pmic_xpower.c
index e6e991a..55f5111 100644
--- a/drivers/acpi/pmic/intel_pmic_xpower.c
+++ b/drivers/acpi/pmic/intel_pmic_xpower.c
@@ -18,7 +18,6 @@
#include <linux/mfd/axp20x.h>
#include <linux/regmap.h>
#include <linux/platform_device.h>
-#include <linux/iio/consumer.h>
#include "intel_pmic.h"
#define XPOWER_GPADC_LOW 0x5b
@@ -186,28 +185,16 @@
* @regmap: regmap of the PMIC device
* @reg: register to get the reading
*
- * We could get the sensor value by manipulating the HW regs here, but since
- * the axp288 IIO driver may also access the same regs at the same time, the
- * APIs provided by IIO subsystem are used here instead to avoid problems. As
- * a result, the two passed in params are of no actual use.
- *
* Return a positive value on success, errno on failure.
*/
static int intel_xpower_pmic_get_raw_temp(struct regmap *regmap, int reg)
{
- struct iio_channel *gpadc_chan;
- int ret, val;
+ u8 buf[2];
- gpadc_chan = iio_channel_get(NULL, "axp288-system-temp");
- if (IS_ERR_OR_NULL(gpadc_chan))
- return -EACCES;
+ if (regmap_bulk_read(regmap, AXP288_GP_ADC_H, buf, 2))
+ return -EIO;
- ret = iio_read_channel_raw(gpadc_chan, &val);
- if (ret < 0)
- val = ret;
-
- iio_channel_release(gpadc_chan);
- return val;
+ return (buf[0] << 4) + ((buf[1] >> 4) & 0x0F);
}
static struct intel_pmic_opregion_data intel_xpower_pmic_opregion_data = {
diff --git a/drivers/acpi/processor_driver.c b/drivers/acpi/processor_driver.c
index 9d5f0c7..8697a82 100644
--- a/drivers/acpi/processor_driver.c
+++ b/drivers/acpi/processor_driver.c
@@ -251,6 +251,9 @@
if (ACPI_SUCCESS(status))
return 0;
+ result = -ENODEV;
+ acpi_pss_perf_exit(pr, device);
+
err_power_exit:
acpi_processor_power_exit(pr);
return result;
@@ -259,11 +262,16 @@
static int acpi_processor_start(struct device *dev)
{
struct acpi_device *device = ACPI_COMPANION(dev);
+ int ret;
if (!device)
return -ENODEV;
- return __acpi_processor_start(device);
+ /* Protect against concurrent CPU hotplug operations */
+ get_online_cpus();
+ ret = __acpi_processor_start(device);
+ put_online_cpus();
+ return ret;
}
static int acpi_processor_stop(struct device *dev)
diff --git a/drivers/acpi/processor_throttling.c b/drivers/acpi/processor_throttling.c
index a12f96c..3de34633 100644
--- a/drivers/acpi/processor_throttling.c
+++ b/drivers/acpi/processor_throttling.c
@@ -62,8 +62,8 @@
#define THROTTLING_POSTCHANGE (2)
static int acpi_processor_get_throttling(struct acpi_processor *pr);
-int acpi_processor_set_throttling(struct acpi_processor *pr,
- int state, bool force);
+static int __acpi_processor_set_throttling(struct acpi_processor *pr,
+ int state, bool force, bool direct);
static int acpi_processor_update_tsd_coord(void)
{
@@ -891,7 +891,8 @@
ACPI_DEBUG_PRINT((ACPI_DB_INFO,
"Invalid throttling state, reset\n"));
state = 0;
- ret = acpi_processor_set_throttling(pr, state, true);
+ ret = __acpi_processor_set_throttling(pr, state, true,
+ true);
if (ret)
return ret;
}
@@ -901,36 +902,31 @@
return 0;
}
+static long __acpi_processor_get_throttling(void *data)
+{
+ struct acpi_processor *pr = data;
+
+ return pr->throttling.acpi_processor_get_throttling(pr);
+}
+
static int acpi_processor_get_throttling(struct acpi_processor *pr)
{
- cpumask_var_t saved_mask;
- int ret;
-
if (!pr)
return -EINVAL;
if (!pr->flags.throttling)
return -ENODEV;
- if (!alloc_cpumask_var(&saved_mask, GFP_KERNEL))
- return -ENOMEM;
-
/*
- * Migrate task to the cpu pointed by pr.
+ * This is either called from the CPU hotplug callback of
+ * processor_driver or via the ACPI probe function. In the latter
+ * case the CPU is not guaranteed to be online. Both call sites are
+ * protected against CPU hotplug.
*/
- cpumask_copy(saved_mask, ¤t->cpus_allowed);
- /* FIXME: use work_on_cpu() */
- if (set_cpus_allowed_ptr(current, cpumask_of(pr->id))) {
- /* Can't migrate to the target pr->id CPU. Exit */
- free_cpumask_var(saved_mask);
+ if (!cpu_online(pr->id))
return -ENODEV;
- }
- ret = pr->throttling.acpi_processor_get_throttling(pr);
- /* restore the previous state */
- set_cpus_allowed_ptr(current, saved_mask);
- free_cpumask_var(saved_mask);
- return ret;
+ return work_on_cpu(pr->id, __acpi_processor_get_throttling, pr);
}
static int acpi_processor_get_fadt_info(struct acpi_processor *pr)
@@ -1080,8 +1076,15 @@
arg->target_state, arg->force);
}
-int acpi_processor_set_throttling(struct acpi_processor *pr,
- int state, bool force)
+static int call_on_cpu(int cpu, long (*fn)(void *), void *arg, bool direct)
+{
+ if (direct)
+ return fn(arg);
+ return work_on_cpu(cpu, fn, arg);
+}
+
+static int __acpi_processor_set_throttling(struct acpi_processor *pr,
+ int state, bool force, bool direct)
{
int ret = 0;
unsigned int i;
@@ -1130,7 +1133,8 @@
arg.pr = pr;
arg.target_state = state;
arg.force = force;
- ret = work_on_cpu(pr->id, acpi_processor_throttling_fn, &arg);
+ ret = call_on_cpu(pr->id, acpi_processor_throttling_fn, &arg,
+ direct);
} else {
/*
* When the T-state coordination is SW_ALL or HW_ALL,
@@ -1163,8 +1167,8 @@
arg.pr = match_pr;
arg.target_state = state;
arg.force = force;
- ret = work_on_cpu(pr->id, acpi_processor_throttling_fn,
- &arg);
+ ret = call_on_cpu(pr->id, acpi_processor_throttling_fn,
+ &arg, direct);
}
}
/*
@@ -1182,6 +1186,12 @@
return ret;
}
+int acpi_processor_set_throttling(struct acpi_processor *pr, int state,
+ bool force)
+{
+ return __acpi_processor_set_throttling(pr, state, force, false);
+}
+
int acpi_processor_get_throttling_info(struct acpi_processor *pr)
{
int result = 0;
diff --git a/drivers/acpi/property.c b/drivers/acpi/property.c
index 3afddcd..9364398 100644
--- a/drivers/acpi/property.c
+++ b/drivers/acpi/property.c
@@ -37,14 +37,16 @@
static bool acpi_enumerate_nondev_subnodes(acpi_handle scope,
const union acpi_object *desc,
- struct acpi_device_data *data);
+ struct acpi_device_data *data,
+ struct fwnode_handle *parent);
static bool acpi_extract_properties(const union acpi_object *desc,
struct acpi_device_data *data);
static bool acpi_nondev_subnode_extract(const union acpi_object *desc,
acpi_handle handle,
const union acpi_object *link,
- struct list_head *list)
+ struct list_head *list,
+ struct fwnode_handle *parent)
{
struct acpi_data_node *dn;
bool result;
@@ -55,6 +57,7 @@
dn->name = link->package.elements[0].string.pointer;
dn->fwnode.type = FWNODE_ACPI_DATA;
+ dn->parent = parent;
INIT_LIST_HEAD(&dn->data.subnodes);
result = acpi_extract_properties(desc, &dn->data);
@@ -71,9 +74,11 @@
*/
status = acpi_get_parent(handle, &scope);
if (ACPI_SUCCESS(status)
- && acpi_enumerate_nondev_subnodes(scope, desc, &dn->data))
+ && acpi_enumerate_nondev_subnodes(scope, desc, &dn->data,
+ &dn->fwnode))
result = true;
- } else if (acpi_enumerate_nondev_subnodes(NULL, desc, &dn->data)) {
+ } else if (acpi_enumerate_nondev_subnodes(NULL, desc, &dn->data,
+ &dn->fwnode)) {
result = true;
}
@@ -91,7 +96,8 @@
static bool acpi_nondev_subnode_data_ok(acpi_handle handle,
const union acpi_object *link,
- struct list_head *list)
+ struct list_head *list,
+ struct fwnode_handle *parent)
{
struct acpi_buffer buf = { ACPI_ALLOCATE_BUFFER };
acpi_status status;
@@ -101,7 +107,8 @@
if (ACPI_FAILURE(status))
return false;
- if (acpi_nondev_subnode_extract(buf.pointer, handle, link, list))
+ if (acpi_nondev_subnode_extract(buf.pointer, handle, link, list,
+ parent))
return true;
ACPI_FREE(buf.pointer);
@@ -110,7 +117,8 @@
static bool acpi_nondev_subnode_ok(acpi_handle scope,
const union acpi_object *link,
- struct list_head *list)
+ struct list_head *list,
+ struct fwnode_handle *parent)
{
acpi_handle handle;
acpi_status status;
@@ -123,12 +131,13 @@
if (ACPI_FAILURE(status))
return false;
- return acpi_nondev_subnode_data_ok(handle, link, list);
+ return acpi_nondev_subnode_data_ok(handle, link, list, parent);
}
static int acpi_add_nondev_subnodes(acpi_handle scope,
const union acpi_object *links,
- struct list_head *list)
+ struct list_head *list,
+ struct fwnode_handle *parent)
{
bool ret = false;
int i;
@@ -150,15 +159,18 @@
/* The second one may be a string, a reference or a package. */
switch (link->package.elements[1].type) {
case ACPI_TYPE_STRING:
- result = acpi_nondev_subnode_ok(scope, link, list);
+ result = acpi_nondev_subnode_ok(scope, link, list,
+ parent);
break;
case ACPI_TYPE_LOCAL_REFERENCE:
handle = link->package.elements[1].reference.handle;
- result = acpi_nondev_subnode_data_ok(handle, link, list);
+ result = acpi_nondev_subnode_data_ok(handle, link, list,
+ parent);
break;
case ACPI_TYPE_PACKAGE:
desc = &link->package.elements[1];
- result = acpi_nondev_subnode_extract(desc, NULL, link, list);
+ result = acpi_nondev_subnode_extract(desc, NULL, link,
+ list, parent);
break;
default:
result = false;
@@ -172,7 +184,8 @@
static bool acpi_enumerate_nondev_subnodes(acpi_handle scope,
const union acpi_object *desc,
- struct acpi_device_data *data)
+ struct acpi_device_data *data,
+ struct fwnode_handle *parent)
{
int i;
@@ -194,7 +207,8 @@
if (memcmp(uuid->buffer.pointer, ads_uuid, sizeof(ads_uuid)))
continue;
- return acpi_add_nondev_subnodes(scope, links, &data->subnodes);
+ return acpi_add_nondev_subnodes(scope, links, &data->subnodes,
+ parent);
}
return false;
@@ -345,7 +359,8 @@
if (acpi_of)
acpi_init_of_compatible(adev);
}
- if (acpi_enumerate_nondev_subnodes(adev->handle, buf.pointer, &adev->data))
+ if (acpi_enumerate_nondev_subnodes(adev->handle, buf.pointer,
+ &adev->data, acpi_fwnode_handle(adev)))
adev->data.pointer = buf.pointer;
if (!adev->data.pointer) {
@@ -699,6 +714,8 @@
return ret;
*(char **)val = obj->string.pointer;
+
+ return 1;
} else {
ret = -EINVAL;
}
@@ -708,7 +725,15 @@
int acpi_dev_prop_read_single(struct acpi_device *adev, const char *propname,
enum dev_prop_type proptype, void *val)
{
- return adev ? acpi_data_prop_read_single(&adev->data, propname, proptype, val) : -EINVAL;
+ int ret;
+
+ if (!adev)
+ return -EINVAL;
+
+ ret = acpi_data_prop_read_single(&adev->data, propname, proptype, val);
+ if (ret < 0 || proptype != ACPI_TYPE_STRING)
+ return ret;
+ return 0;
}
static int acpi_copy_property_array_u8(const union acpi_object *items, u8 *val,
@@ -784,7 +809,7 @@
val[i] = items[i].string.pointer;
}
- return 0;
+ return nval;
}
static int acpi_data_prop_read(struct acpi_device_data *data,
@@ -798,7 +823,7 @@
if (val && nval == 1) {
ret = acpi_data_prop_read_single(data, propname, proptype, val);
- if (!ret)
+ if (ret >= 0)
return ret;
}
@@ -809,7 +834,7 @@
if (!val)
return obj->package.count;
- if (nval > obj->package.count)
+ if (proptype != DEV_PROP_STRING && nval > obj->package.count)
return -EOVERFLOW;
else if (nval <= 0)
return -EINVAL;
@@ -830,7 +855,9 @@
ret = acpi_copy_property_array_u64(items, (u64 *)val, nval);
break;
case DEV_PROP_STRING:
- ret = acpi_copy_property_array_string(items, (char **)val, nval);
+ ret = acpi_copy_property_array_string(
+ items, (char **)val,
+ min_t(u32, nval, obj->package.count));
break;
default:
ret = -EINVAL;
@@ -865,21 +892,22 @@
}
/**
- * acpi_get_next_subnode - Return the next child node handle for a device.
- * @dev: Device to find the next child node for.
+ * acpi_get_next_subnode - Return the next child node handle for a fwnode
+ * @fwnode: Firmware node to find the next child node for.
* @child: Handle to one of the device's child nodes or a null handle.
*/
-struct fwnode_handle *acpi_get_next_subnode(struct device *dev,
+struct fwnode_handle *acpi_get_next_subnode(struct fwnode_handle *fwnode,
struct fwnode_handle *child)
{
- struct acpi_device *adev = ACPI_COMPANION(dev);
+ struct acpi_device *adev = to_acpi_device_node(fwnode);
struct list_head *head, *next;
- if (!adev)
- return NULL;
-
if (!child || child->type == FWNODE_ACPI) {
- head = &adev->children;
+ if (adev)
+ head = &adev->children;
+ else
+ goto nondev;
+
if (list_empty(head))
goto nondev;
@@ -888,7 +916,6 @@
next = adev->node.next;
if (next == head) {
child = NULL;
- adev = ACPI_COMPANION(dev);
goto nondev;
}
adev = list_entry(next, struct acpi_device, node);
@@ -900,9 +927,16 @@
nondev:
if (!child || child->type == FWNODE_ACPI_DATA) {
+ struct acpi_data_node *data = to_acpi_data_node(fwnode);
struct acpi_data_node *dn;
- head = &adev->data.subnodes;
+ if (adev)
+ head = &adev->data.subnodes;
+ else if (data)
+ head = &data->data.subnodes;
+ else
+ return NULL;
+
if (list_empty(head))
return NULL;
@@ -920,3 +954,168 @@
}
return NULL;
}
+
+/**
+ * acpi_node_get_parent - Return parent fwnode of this fwnode
+ * @fwnode: Firmware node whose parent to get
+ *
+ * Returns parent node of an ACPI device or data firmware node or %NULL if
+ * not available.
+ */
+struct fwnode_handle *acpi_node_get_parent(struct fwnode_handle *fwnode)
+{
+ if (is_acpi_data_node(fwnode)) {
+ /* All data nodes have parent pointer so just return that */
+ return to_acpi_data_node(fwnode)->parent;
+ } else if (is_acpi_device_node(fwnode)) {
+ acpi_handle handle, parent_handle;
+
+ handle = to_acpi_device_node(fwnode)->handle;
+ if (ACPI_SUCCESS(acpi_get_parent(handle, &parent_handle))) {
+ struct acpi_device *adev;
+
+ if (!acpi_bus_get_device(parent_handle, &adev))
+ return acpi_fwnode_handle(adev);
+ }
+ }
+
+ return NULL;
+}
+
+/**
+ * acpi_graph_get_next_endpoint - Get next endpoint ACPI firmware node
+ * @fwnode: Pointer to the parent firmware node
+ * @prev: Previous endpoint node or %NULL to get the first
+ *
+ * Looks up next endpoint ACPI firmware node below a given @fwnode. Returns
+ * %NULL if there is no next endpoint, ERR_PTR() in case of error. In case
+ * of success the next endpoint is returned.
+ */
+struct fwnode_handle *acpi_graph_get_next_endpoint(struct fwnode_handle *fwnode,
+ struct fwnode_handle *prev)
+{
+ struct fwnode_handle *port = NULL;
+ struct fwnode_handle *endpoint;
+
+ if (!prev) {
+ do {
+ port = fwnode_get_next_child_node(fwnode, port);
+ /* Ports must have port property */
+ if (fwnode_property_present(port, "port"))
+ break;
+ } while (port);
+ } else {
+ port = fwnode_get_parent(prev);
+ }
+
+ if (!port)
+ return NULL;
+
+ endpoint = fwnode_get_next_child_node(port, prev);
+ while (!endpoint) {
+ port = fwnode_get_next_child_node(fwnode, port);
+ if (!port)
+ break;
+ if (fwnode_property_present(port, "port"))
+ endpoint = fwnode_get_next_child_node(port, NULL);
+ }
+
+ if (endpoint) {
+ /* Endpoints must have "endpoint" property */
+ if (!fwnode_property_present(endpoint, "endpoint"))
+ return ERR_PTR(-EPROTO);
+ }
+
+ return endpoint;
+}
+
+/**
+ * acpi_graph_get_child_prop_value - Return a child with a given property value
+ * @fwnode: device fwnode
+ * @prop_name: The name of the property to look for
+ * @val: the desired property value
+ *
+ * Return the port node corresponding to a given port number. Returns
+ * the child node on success, NULL otherwise.
+ */
+static struct fwnode_handle *acpi_graph_get_child_prop_value(
+ struct fwnode_handle *fwnode, const char *prop_name, unsigned int val)
+{
+ struct fwnode_handle *child;
+
+ fwnode_for_each_child_node(fwnode, child) {
+ u32 nr;
+
+ if (!fwnode_property_read_u32(fwnode, prop_name, &nr))
+ continue;
+
+ if (val == nr)
+ return child;
+ }
+
+ return NULL;
+}
+
+
+/**
+ * acpi_graph_get_remote_enpoint - Parses and returns remote end of an endpoint
+ * @fwnode: Endpoint firmware node pointing to a remote device
+ * @parent: Firmware node of remote port parent is filled here if not %NULL
+ * @port: Firmware node of remote port is filled here if not %NULL
+ * @endpoint: Firmware node of remote endpoint is filled here if not %NULL
+ *
+ * Function parses remote end of ACPI firmware remote endpoint and fills in
+ * fields requested by the caller. Returns %0 in case of success and
+ * negative errno otherwise.
+ */
+int acpi_graph_get_remote_endpoint(struct fwnode_handle *fwnode,
+ struct fwnode_handle **parent,
+ struct fwnode_handle **port,
+ struct fwnode_handle **endpoint)
+{
+ unsigned int port_nr, endpoint_nr;
+ struct acpi_reference_args args;
+ int ret;
+
+ memset(&args, 0, sizeof(args));
+ ret = acpi_node_get_property_reference(fwnode, "remote-endpoint", 0,
+ &args);
+ if (ret)
+ return ret;
+
+ /*
+ * Always require two arguments with the reference: port and
+ * endpoint indices.
+ */
+ if (args.nargs != 2)
+ return -EPROTO;
+
+ fwnode = acpi_fwnode_handle(args.adev);
+ port_nr = args.args[0];
+ endpoint_nr = args.args[1];
+
+ if (parent)
+ *parent = fwnode;
+
+ if (!port && !endpoint)
+ return 0;
+
+ fwnode = acpi_graph_get_child_prop_value(fwnode, "port", port_nr);
+ if (!fwnode)
+ return -EPROTO;
+
+ if (port)
+ *port = fwnode;
+
+ if (!endpoint)
+ return 0;
+
+ fwnode = acpi_graph_get_child_prop_value(fwnode, "endpoint",
+ endpoint_nr);
+ if (!fwnode)
+ return -EPROTO;
+
+ *endpoint = fwnode;
+
+ return 0;
+}
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 2433569..c269310 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -30,12 +30,6 @@
#define INVALID_ACPI_HANDLE ((acpi_handle)empty_zero_page)
-/*
- * If set, devices will be hot-removed even if they cannot be put offline
- * gracefully (from the kernel's standpoint).
- */
-bool acpi_force_hot_remove;
-
static const char *dummy_hid = "device";
static LIST_HEAD(acpi_dep_list);
@@ -170,9 +164,6 @@
pn->put_online = false;
}
ret = device_offline(pn->dev);
- if (acpi_force_hot_remove)
- continue;
-
if (ret >= 0) {
pn->put_online = !ret;
} else {
@@ -241,11 +232,11 @@
acpi_walk_namespace(ACPI_TYPE_ANY, handle, ACPI_UINT32_MAX,
NULL, acpi_bus_offline, (void *)true,
(void **)&errdev);
- if (!errdev || acpi_force_hot_remove)
+ if (!errdev)
acpi_bus_offline(handle, 0, (void *)true,
(void **)&errdev);
- if (errdev && !acpi_force_hot_remove) {
+ if (errdev) {
dev_warn(errdev, "Offline failed.\n");
acpi_bus_online(handle, 0, NULL, NULL);
acpi_walk_namespace(ACPI_TYPE_ANY, handle,
@@ -263,8 +254,7 @@
unsigned long long sta;
acpi_status status;
- if (device->handler && device->handler->hotplug.demand_offline
- && !acpi_force_hot_remove) {
+ if (device->handler && device->handler->hotplug.demand_offline) {
if (!acpi_scan_is_offline(device, true))
return -EBUSY;
} else {
@@ -1850,6 +1840,8 @@
device->flags.power_manageable = 0;
device->flags.initialized = true;
+ } else if (device->flags.visited) {
+ goto ok;
}
ret = acpi_scan_attach_handler(device);
@@ -1866,10 +1858,10 @@
if (ret < 0)
return;
- if (ret > 0 || !device->pnp.type.platform_id)
- acpi_device_set_enumerated(device);
- else
+ if (device->pnp.type.platform_id)
acpi_default_enumeration(device);
+ else
+ acpi_device_set_enumerated(device);
ok:
list_for_each_entry(child, &device->children, node)
diff --git a/drivers/acpi/sysfs.c b/drivers/acpi/sysfs.c
index cf05ae9..1b5ee1e 100644
--- a/drivers/acpi/sysfs.c
+++ b/drivers/acpi/sysfs.c
@@ -921,7 +921,7 @@
static ssize_t force_remove_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
- return sprintf(buf, "%d\n", !!acpi_force_hot_remove);
+ return sprintf(buf, "%d\n", 0);
}
static ssize_t force_remove_store(struct kobject *kobj,
@@ -935,9 +935,10 @@
if (ret < 0)
return ret;
- lock_device_hotplug();
- acpi_force_hot_remove = val;
- unlock_device_hotplug();
+ if (val) {
+ pr_err("Enabling force_remove is not supported anymore. Please report to linux-acpi@vger.kernel.org if you depend on this functionality\n");
+ return -EINVAL;
+ }
return size;
}
diff --git a/drivers/acpi/tables.c b/drivers/acpi/tables.c
index 2604189d..ff42539 100644
--- a/drivers/acpi/tables.c
+++ b/drivers/acpi/tables.c
@@ -311,22 +311,6 @@
}
int __init
-acpi_parse_entries(char *id,
- unsigned long table_size,
- acpi_tbl_entry_handler handler,
- struct acpi_table_header *table_header,
- int entry_id, unsigned int max_entries)
-{
- struct acpi_subtable_proc proc = {
- .id = entry_id,
- .handler = handler,
- };
-
- return acpi_parse_entries_array(id, table_size, table_header,
- &proc, 1, max_entries);
-}
-
-int __init
acpi_table_parse_entries_array(char *id,
unsigned long table_size,
struct acpi_subtable_proc *proc, int proc_num,
@@ -556,7 +540,7 @@
* But it's not enough on X86 because ioremap will
* complain later (used by acpi_os_map_memory) that the pages
* that should get mapped are not marked "reserved".
- * Both memblock_reserve and e820_add_region (via arch_reserve_mem_area)
+ * Both memblock_reserve and e820__range_add (via arch_reserve_mem_area)
* works fine.
*/
memblock_reserve(acpi_tables_addr, all_tables_size);
diff --git a/drivers/acpi/utils.c b/drivers/acpi/utils.c
index 22c0995..27d0dcf 100644
--- a/drivers/acpi/utils.c
+++ b/drivers/acpi/utils.c
@@ -736,6 +736,72 @@
}
EXPORT_SYMBOL(acpi_dev_found);
+struct acpi_dev_present_info {
+ struct acpi_device_id hid[2];
+ const char *uid;
+ s64 hrv;
+};
+
+static int acpi_dev_present_cb(struct device *dev, void *data)
+{
+ struct acpi_device *adev = to_acpi_device(dev);
+ struct acpi_dev_present_info *match = data;
+ unsigned long long hrv;
+ acpi_status status;
+
+ if (acpi_match_device_ids(adev, match->hid))
+ return 0;
+
+ if (match->uid && (!adev->pnp.unique_id ||
+ strcmp(adev->pnp.unique_id, match->uid)))
+ return 0;
+
+ if (match->hrv == -1)
+ return 1;
+
+ status = acpi_evaluate_integer(adev->handle, "_HRV", NULL, &hrv);
+ if (ACPI_FAILURE(status))
+ return 0;
+
+ return hrv == match->hrv;
+}
+
+/**
+ * acpi_dev_present - Detect that a given ACPI device is present
+ * @hid: Hardware ID of the device.
+ * @uid: Unique ID of the device, pass NULL to not check _UID
+ * @hrv: Hardware Revision of the device, pass -1 to not check _HRV
+ *
+ * Return %true if a matching device was present at the moment of invocation.
+ * Note that if the device is pluggable, it may since have disappeared.
+ *
+ * Note that unlike acpi_dev_found() this function checks the status
+ * of the device. So for devices which are present in the dsdt, but
+ * which are disabled (their _STA callback returns 0) this function
+ * will return false.
+ *
+ * For this function to work, acpi_bus_scan() must have been executed
+ * which happens in the subsys_initcall() subsection. Hence, do not
+ * call from a subsys_initcall() or earlier (use acpi_get_devices()
+ * instead). Calling from module_init() is fine (which is synonymous
+ * with device_initcall()).
+ */
+bool acpi_dev_present(const char *hid, const char *uid, s64 hrv)
+{
+ struct acpi_dev_present_info match = {};
+ struct device *dev;
+
+ strlcpy(match.hid[0].id, hid, sizeof(match.hid[0].id));
+ match.uid = uid;
+ match.hrv = hrv;
+
+ dev = bus_find_device(&acpi_bus_type, NULL, &match,
+ acpi_dev_present_cb);
+
+ return !!dev;
+}
+EXPORT_SYMBOL(acpi_dev_present);
+
/*
* acpi_backlight= handling, this is done here rather then in video_detect.c
* because __setup cannot be used in modules.
diff --git a/drivers/ata/Kconfig b/drivers/ata/Kconfig
index 70b57d2..ff6cb9e 100644
--- a/drivers/ata/Kconfig
+++ b/drivers/ata/Kconfig
@@ -14,7 +14,6 @@
tristate "Serial ATA and Parallel ATA drivers (libata)"
depends on HAS_IOMEM
depends on BLOCK
- depends on !(M32R || S390) || BROKEN
select SCSI
select GLOB
---help---
@@ -118,6 +117,15 @@
If unsure, say N.
+config AHCI_DM816
+ tristate "DaVinci DM816 AHCI SATA support"
+ depends on ARCH_OMAP2PLUS
+ help
+ This option enables support for the DaVinci DM816 SoC's
+ onboard AHCI SATA controller.
+
+ If unsure, say N.
+
config AHCI_ST
tristate "ST AHCI SATA support"
depends on ARCH_STI
@@ -885,14 +893,6 @@
If unsure, say N.
-config PATA_AT91
- tristate "PATA support for AT91SAM9260"
- depends on ARM && SOC_AT91SAM9
- help
- This option enables support for IDE devices on the Atmel AT91SAM9260 SoC.
-
- If unsure, say N.
-
config PATA_CMD640_PCI
tristate "CMD640 PCI PATA support (Experimental)"
depends on PCI
diff --git a/drivers/ata/Makefile b/drivers/ata/Makefile
index 89a0a19..3048cc1 100644
--- a/drivers/ata/Makefile
+++ b/drivers/ata/Makefile
@@ -14,6 +14,7 @@
obj-$(CONFIG_AHCI_BRCM) += ahci_brcm.o libahci.o libahci_platform.o
obj-$(CONFIG_AHCI_CEVA) += ahci_ceva.o libahci.o libahci_platform.o
obj-$(CONFIG_AHCI_DA850) += ahci_da850.o libahci.o libahci_platform.o
+obj-$(CONFIG_AHCI_DM816) += ahci_dm816.o libahci.o libahci_platform.o
obj-$(CONFIG_AHCI_IMX) += ahci_imx.o libahci.o libahci_platform.o
obj-$(CONFIG_AHCI_MVEBU) += ahci_mvebu.o libahci.o libahci_platform.o
obj-$(CONFIG_AHCI_OCTEON) += ahci_octeon.o
@@ -91,7 +92,6 @@
# SFF PIO only
obj-$(CONFIG_PATA_AT32) += pata_at32.o
-obj-$(CONFIG_PATA_AT91) += pata_at91.o
obj-$(CONFIG_PATA_CMD640_PCI) += pata_cmd640.o
obj-$(CONFIG_PATA_FALCON) += pata_falcon.o
obj-$(CONFIG_PATA_ISAPNP) += pata_isapnp.o
diff --git a/drivers/ata/ahci_dm816.c b/drivers/ata/ahci_dm816.c
new file mode 100644
index 0000000..fbd827c
--- /dev/null
+++ b/drivers/ata/ahci_dm816.c
@@ -0,0 +1,200 @@
+/*
+ * DaVinci DM816 AHCI SATA platform driver
+ *
+ * Copyright (C) 2017 BayLibre SAS
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2, or (at your option)
+ * any later version.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/device.h>
+#include <linux/pm.h>
+#include <linux/platform_device.h>
+#include <linux/libata.h>
+#include <linux/ahci_platform.h>
+
+#include "ahci.h"
+
+#define AHCI_DM816_DRV_NAME "ahci-dm816"
+
+#define AHCI_DM816_PHY_ENPLL(x) ((x) << 0)
+#define AHCI_DM816_PHY_MPY(x) ((x) << 1)
+#define AHCI_DM816_PHY_LOS(x) ((x) << 12)
+#define AHCI_DM816_PHY_RXCDR(x) ((x) << 13)
+#define AHCI_DM816_PHY_RXEQ(x) ((x) << 16)
+#define AHCI_DM816_PHY_TXSWING(x) ((x) << 23)
+
+#define AHCI_DM816_P0PHYCR_REG 0x178
+#define AHCI_DM816_P1PHYCR_REG 0x1f8
+
+#define AHCI_DM816_PLL_OUT 1500000000LU
+
+static const unsigned long pll_mpy_table[] = {
+ 400, 500, 600, 800, 825, 1000, 1200,
+ 1250, 1500, 1600, 1650, 2000, 2200, 2500
+};
+
+static int ahci_dm816_get_mpy_bits(unsigned long refclk_rate)
+{
+ unsigned long pll_multiplier;
+ int i;
+
+ /*
+ * We need to determine the value of the multiplier (MPY) bits.
+ * In order to include the 8.25 multiplier we need to first divide
+ * the refclk rate by 100.
+ */
+ pll_multiplier = AHCI_DM816_PLL_OUT / (refclk_rate / 100);
+
+ for (i = 0; i < ARRAY_SIZE(pll_mpy_table); i++) {
+ if (pll_mpy_table[i] == pll_multiplier)
+ return i;
+ }
+
+ /*
+ * We should have divided evenly - if not, return an invalid
+ * value.
+ */
+ return -1;
+}
+
+static int ahci_dm816_phy_init(struct ahci_host_priv *hpriv, struct device *dev)
+{
+ unsigned long refclk_rate;
+ int mpy;
+ u32 val;
+
+ /*
+ * We should have been supplied two clocks: the functional and
+ * keep-alive clock and the external reference clock. We need the
+ * rate of the latter to calculate the correct value of MPY bits.
+ */
+ if (!hpriv->clks[1]) {
+ dev_err(dev, "reference clock not supplied\n");
+ return -EINVAL;
+ }
+
+ refclk_rate = clk_get_rate(hpriv->clks[1]);
+ if ((refclk_rate % 100) != 0) {
+ dev_err(dev, "reference clock rate must be divisible by 100\n");
+ return -EINVAL;
+ }
+
+ mpy = ahci_dm816_get_mpy_bits(refclk_rate);
+ if (mpy < 0) {
+ dev_err(dev, "can't calculate the MPY bits value\n");
+ return -EINVAL;
+ }
+
+ /* Enable the PHY and configure the first HBA port. */
+ val = AHCI_DM816_PHY_MPY(mpy) | AHCI_DM816_PHY_LOS(1) |
+ AHCI_DM816_PHY_RXCDR(4) | AHCI_DM816_PHY_RXEQ(1) |
+ AHCI_DM816_PHY_TXSWING(3) | AHCI_DM816_PHY_ENPLL(1);
+ writel(val, hpriv->mmio + AHCI_DM816_P0PHYCR_REG);
+
+ /* Configure the second HBA port. */
+ val = AHCI_DM816_PHY_LOS(1) | AHCI_DM816_PHY_RXCDR(4) |
+ AHCI_DM816_PHY_RXEQ(1) | AHCI_DM816_PHY_TXSWING(3);
+ writel(val, hpriv->mmio + AHCI_DM816_P1PHYCR_REG);
+
+ return 0;
+}
+
+static int ahci_dm816_softreset(struct ata_link *link,
+ unsigned int *class, unsigned long deadline)
+{
+ int pmp, ret;
+
+ pmp = sata_srst_pmp(link);
+
+ /*
+ * There's an issue with the SATA controller on DM816 SoC: if we
+ * enable Port Multiplier support, but the drive is connected directly
+ * to the board, it can't be detected. As a workaround: if PMP is
+ * enabled, we first call ahci_do_softreset() and pass it the result of
+ * sata_srst_pmp(). If this call fails, we retry with pmp = 0.
+ */
+ ret = ahci_do_softreset(link, class, pmp, deadline, ahci_check_ready);
+ if (pmp && ret == -EBUSY)
+ return ahci_do_softreset(link, class, 0,
+ deadline, ahci_check_ready);
+
+ return ret;
+}
+
+static struct ata_port_operations ahci_dm816_port_ops = {
+ .inherits = &ahci_platform_ops,
+ .softreset = ahci_dm816_softreset,
+};
+
+static const struct ata_port_info ahci_dm816_port_info = {
+ .flags = AHCI_FLAG_COMMON,
+ .pio_mask = ATA_PIO4,
+ .udma_mask = ATA_UDMA6,
+ .port_ops = &ahci_dm816_port_ops,
+};
+
+static struct scsi_host_template ahci_dm816_platform_sht = {
+ AHCI_SHT(AHCI_DM816_DRV_NAME),
+};
+
+static int ahci_dm816_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct ahci_host_priv *hpriv;
+ int rc;
+
+ hpriv = ahci_platform_get_resources(pdev);
+ if (IS_ERR(hpriv))
+ return PTR_ERR(hpriv);
+
+ rc = ahci_platform_enable_resources(hpriv);
+ if (rc)
+ return rc;
+
+ rc = ahci_dm816_phy_init(hpriv, dev);
+ if (rc)
+ goto disable_resources;
+
+ rc = ahci_platform_init_host(pdev, hpriv,
+ &ahci_dm816_port_info,
+ &ahci_dm816_platform_sht);
+ if (rc)
+ goto disable_resources;
+
+ return 0;
+
+disable_resources:
+ ahci_platform_disable_resources(hpriv);
+
+ return rc;
+}
+
+static SIMPLE_DEV_PM_OPS(ahci_dm816_pm_ops,
+ ahci_platform_suspend,
+ ahci_platform_resume);
+
+static const struct of_device_id ahci_dm816_of_match[] = {
+ { .compatible = "ti,dm816-ahci", },
+ { },
+};
+MODULE_DEVICE_TABLE(of, ahci_dm816_of_match);
+
+static struct platform_driver ahci_dm816_driver = {
+ .probe = ahci_dm816_probe,
+ .remove = ata_platform_remove_one,
+ .driver = {
+ .name = AHCI_DM816_DRV_NAME,
+ .of_match_table = ahci_dm816_of_match,
+ .pm = &ahci_dm816_pm_ops,
+ },
+};
+module_platform_driver(ahci_dm816_driver);
+
+MODULE_DESCRIPTION("DaVinci DM816 AHCI SATA platform driver");
+MODULE_AUTHOR("Bartosz Golaszewski <bgolaszewski@baylibre.com>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/ata/ahci_octeon.c b/drivers/ata/ahci_octeon.c
index ea865fe..5a44e08 100644
--- a/drivers/ata/ahci_octeon.c
+++ b/drivers/ata/ahci_octeon.c
@@ -38,11 +38,6 @@
int ret;
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!res) {
- dev_err(&pdev->dev, "Platform resource[0] is missing\n");
- return -ENODEV;
- }
-
base = devm_ioremap_resource(&pdev->dev, res);
if (IS_ERR(base))
return PTR_ERR(base);
diff --git a/drivers/ata/libata-core.c b/drivers/ata/libata-core.c
index ca75823..2d83b8c 100644
--- a/drivers/ata/libata-core.c
+++ b/drivers/ata/libata-core.c
@@ -4910,7 +4910,7 @@
* LOCKING:
* spin_lock_irqsave(host lock)
*/
-void ata_sg_clean(struct ata_queued_cmd *qc)
+static void ata_sg_clean(struct ata_queued_cmd *qc)
{
struct ata_port *ap = qc->ap;
struct scatterlist *sg = qc->sg;
@@ -5902,9 +5902,9 @@
INIT_LIST_HEAD(&ap->eh_done_q);
init_waitqueue_head(&ap->eh_wait_q);
init_completion(&ap->park_req_pending);
- init_timer_deferrable(&ap->fastdrain_timer);
- ap->fastdrain_timer.function = ata_eh_fastdrain_timerfn;
- ap->fastdrain_timer.data = (unsigned long)ap;
+ setup_deferrable_timer(&ap->fastdrain_timer,
+ ata_eh_fastdrain_timerfn,
+ (unsigned long)ap);
ap->cbl = ATA_CBL_NONE;
diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
index 1ac7074..49ba983 100644
--- a/drivers/ata/libata-scsi.c
+++ b/drivers/ata/libata-scsi.c
@@ -3393,46 +3393,6 @@
}
/**
- * ata_format_dsm_trim_descr() - SATL Write Same to ATA SCT Write Same
- * @cmd: SCSI command being translated
- * @lba: Starting sector
- * @num: Number of sectors to be zero'd.
- *
- * Rewrite the WRITE SAME payload to be an SCT Write Same formatted
- * descriptor.
- * NOTE: Writes a pattern (0's) in the foreground.
- *
- * Return: Number of bytes copied into sglist.
- */
-static size_t ata_format_sct_write_same(struct scsi_cmnd *cmd, u64 lba, u64 num)
-{
- struct scsi_device *sdp = cmd->device;
- size_t len = sdp->sector_size;
- size_t r;
- u16 *buf;
- unsigned long flags;
-
- spin_lock_irqsave(&ata_scsi_rbuf_lock, flags);
- buf = ((void *)ata_scsi_rbuf);
-
- put_unaligned_le16(0x0002, &buf[0]); /* SCT_ACT_WRITE_SAME */
- put_unaligned_le16(0x0101, &buf[1]); /* WRITE PTRN FG */
- put_unaligned_le64(lba, &buf[2]);
- put_unaligned_le64(num, &buf[6]);
- put_unaligned_le32(0u, &buf[10]); /* pattern */
-
- WARN_ON(len > ATA_SCSI_RBUF_SIZE);
-
- if (len > ATA_SCSI_RBUF_SIZE)
- len = ATA_SCSI_RBUF_SIZE;
-
- r = sg_copy_from_buffer(scsi_sglist(cmd), scsi_sg_count(cmd), buf, len);
- spin_unlock_irqrestore(&ata_scsi_rbuf_lock, flags);
-
- return r;
-}
-
-/**
* ata_scsi_write_same_xlat() - SATL Write Same to ATA SCT Write Same
* @qc: Command to be translated
*
@@ -3462,32 +3422,31 @@
if (unlikely(!dev->dma_mode))
goto invalid_opcode;
+ /*
+ * We only allow sending this command through the block layer,
+ * as it modifies the DATA OUT buffer, which would corrupt user
+ * memory for SG_IO commands.
+ */
+ if (unlikely(blk_rq_is_passthrough(scmd->request)))
+ goto invalid_opcode;
+
if (unlikely(scmd->cmd_len < 16)) {
fp = 15;
goto invalid_fld;
}
scsi_16_lba_len(cdb, &block, &n_block);
- if (unmap) {
- /* If trim is not enabled the cmd is invalid. */
- if ((dev->horkage & ATA_HORKAGE_NOTRIM) ||
- !ata_id_has_trim(dev->id)) {
- fp = 1;
- bp = 3;
- goto invalid_fld;
- }
- /* If the request is too large the cmd is invalid */
- if (n_block > 0xffff * trmax) {
- fp = 2;
- goto invalid_fld;
- }
- } else {
- /* If write same is not available the cmd is invalid */
- if (!ata_id_sct_write_same(dev->id)) {
- fp = 1;
- bp = 3;
- goto invalid_fld;
- }
+ if (!unmap ||
+ (dev->horkage & ATA_HORKAGE_NOTRIM) ||
+ !ata_id_has_trim(dev->id)) {
+ fp = 1;
+ bp = 3;
+ goto invalid_fld;
+ }
+ /* If the request is too large the cmd is invalid */
+ if (n_block > 0xffff * trmax) {
+ fp = 2;
+ goto invalid_fld;
}
/*
@@ -3502,49 +3461,28 @@
* For DATA SET MANAGEMENT TRIM in ACS-2 nsect (aka count)
* is defined as number of 512 byte blocks to be transferred.
*/
- if (unmap) {
- size = ata_format_dsm_trim_descr(scmd, trmax, block, n_block);
- if (size != len)
- goto invalid_param_len;
- if (ata_ncq_enabled(dev) && ata_fpdma_dsm_supported(dev)) {
- /* Newer devices support queued TRIM commands */
- tf->protocol = ATA_PROT_NCQ;
- tf->command = ATA_CMD_FPDMA_SEND;
- tf->hob_nsect = ATA_SUBCMD_FPDMA_SEND_DSM & 0x1f;
- tf->nsect = qc->tag << 3;
- tf->hob_feature = (size / 512) >> 8;
- tf->feature = size / 512;
+ size = ata_format_dsm_trim_descr(scmd, trmax, block, n_block);
+ if (size != len)
+ goto invalid_param_len;
- tf->auxiliary = 1;
- } else {
- tf->protocol = ATA_PROT_DMA;
- tf->hob_feature = 0;
- tf->feature = ATA_DSM_TRIM;
- tf->hob_nsect = (size / 512) >> 8;
- tf->nsect = size / 512;
- tf->command = ATA_CMD_DSM;
- }
+ if (ata_ncq_enabled(dev) && ata_fpdma_dsm_supported(dev)) {
+ /* Newer devices support queued TRIM commands */
+ tf->protocol = ATA_PROT_NCQ;
+ tf->command = ATA_CMD_FPDMA_SEND;
+ tf->hob_nsect = ATA_SUBCMD_FPDMA_SEND_DSM & 0x1f;
+ tf->nsect = qc->tag << 3;
+ tf->hob_feature = (size / 512) >> 8;
+ tf->feature = size / 512;
+
+ tf->auxiliary = 1;
} else {
- size = ata_format_sct_write_same(scmd, block, n_block);
- if (size != len)
- goto invalid_param_len;
-
- tf->hob_feature = 0;
- tf->feature = 0;
- tf->hob_nsect = 0;
- tf->nsect = 1;
- tf->lbah = 0;
- tf->lbam = 0;
- tf->lbal = ATA_CMD_STANDBYNOW1;
- tf->hob_lbah = 0;
- tf->hob_lbam = 0;
- tf->hob_lbal = 0;
- tf->device = ATA_CMD_STANDBYNOW1;
tf->protocol = ATA_PROT_DMA;
- tf->command = ATA_CMD_WRITE_LOG_DMA_EXT;
- if (unlikely(dev->flags & ATA_DFLAG_PIO))
- tf->command = ATA_CMD_WRITE_LOG_EXT;
+ tf->hob_feature = 0;
+ tf->feature = ATA_DSM_TRIM;
+ tf->hob_nsect = (size / 512) >> 8;
+ tf->nsect = size / 512;
+ tf->command = ATA_CMD_DSM;
}
tf->flags |= ATA_TFLAG_ISADDR | ATA_TFLAG_DEVICE | ATA_TFLAG_LBA48 |
@@ -3619,10 +3557,6 @@
case START_STOP:
supported = 3;
break;
- case WRITE_SAME_16:
- if (!ata_id_sct_write_same(dev->id))
- break;
- /* fallthrough: if SCT ... only enable for ZBC */
case ZBC_IN:
case ZBC_OUT:
if (ata_id_zoned_cap(dev->id) ||
diff --git a/drivers/ata/pata_at91.c b/drivers/ata/pata_at91.c
deleted file mode 100644
index fd5b34f..0000000
--- a/drivers/ata/pata_at91.c
+++ /dev/null
@@ -1,503 +0,0 @@
-/*
- * PATA driver for AT91SAM9260 Static Memory Controller
- * with CompactFlash interface in True IDE mode
- *
- * Copyright (C) 2009 Matyukevich Sergey
- * 2011 Igor Plyatov
- *
- * Based on:
- * * generic platform driver by Paul Mundt: drivers/ata/pata_platform.c
- * * pata_at32 driver by Kristoffer Nyborg Gregertsen
- * * at91_ide driver by Stanislaw Gruszka
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2
- * as published by the Free Software Foundation.
- *
- */
-
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/blkdev.h>
-#include <linux/gfp.h>
-#include <scsi/scsi_host.h>
-#include <linux/ata.h>
-#include <linux/clk.h>
-#include <linux/libata.h>
-#include <linux/mfd/syscon.h>
-#include <linux/mfd/syscon/atmel-smc.h>
-#include <linux/platform_device.h>
-#include <linux/ata_platform.h>
-#include <linux/platform_data/atmel.h>
-#include <linux/regmap.h>
-#include <linux/gpio.h>
-
-#define DRV_NAME "pata_at91"
-#define DRV_VERSION "0.3"
-
-#define CF_IDE_OFFSET 0x00c00000
-#define CF_ALT_IDE_OFFSET 0x00e00000
-#define CF_IDE_RES_SIZE 0x08
-#define CS_PULSE_MAXIMUM 319
-#define ER_SMC_CALC 1
-#define ER_SMC_RECALC 2
-
-struct at91_ide_info {
- unsigned long mode;
- unsigned int cs;
- struct clk *mck;
- void __iomem *ide_addr;
- void __iomem *alt_addr;
-};
-
-/**
- * struct smc_range - range of valid values for SMC register.
- */
-struct smc_range {
- int min;
- int max;
-};
-
-struct regmap *smc;
-
-struct at91sam9_smc_generic_fields {
- struct regmap_field *setup;
- struct regmap_field *pulse;
- struct regmap_field *cycle;
- struct regmap_field *mode;
-} fields;
-
-/**
- * adjust_smc_value - adjust value for one of SMC registers.
- * @value: adjusted value
- * @range: array of SMC ranges with valid values
- * @size: SMC ranges array size
- *
- * This returns the difference between input and output value or negative
- * in case of invalid input value.
- * If negative returned, then output value = maximal possible from ranges.
- */
-static int adjust_smc_value(int *value, struct smc_range *range, int size)
-{
- int maximum = (range + size - 1)->max;
- int remainder;
-
- do {
- if (*value < range->min) {
- remainder = range->min - *value;
- *value = range->min; /* nearest valid value */
- return remainder;
- } else if ((range->min <= *value) && (*value <= range->max))
- return 0;
-
- range++;
- } while (--size);
- *value = maximum;
-
- return -1; /* invalid value */
-}
-
-/**
- * calc_smc_vals - calculate SMC register values
- * @dev: ATA device
- * @setup: SMC_SETUP register value
- * @pulse: SMC_PULSE register value
- * @cycle: SMC_CYCLE register value
- *
- * This returns negative in case of invalid values for SMC registers:
- * -ER_SMC_RECALC - recalculation required for SMC values,
- * -ER_SMC_CALC - calculation failed (invalid input values).
- *
- * SMC use special coding scheme, see "Coding and Range of Timing
- * Parameters" table from AT91SAM9 datasheets.
- *
- * SMC_SETUP = 128*setup[5] + setup[4:0]
- * SMC_PULSE = 256*pulse[6] + pulse[5:0]
- * SMC_CYCLE = 256*cycle[8:7] + cycle[6:0]
- */
-static int calc_smc_vals(struct device *dev,
- int *setup, int *pulse, int *cycle, int *cs_pulse)
-{
- int ret_val;
- int err = 0;
- struct smc_range range_setup[] = { /* SMC_SETUP valid values */
- {.min = 0, .max = 31}, /* first range */
- {.min = 128, .max = 159} /* second range */
- };
- struct smc_range range_pulse[] = { /* SMC_PULSE valid values */
- {.min = 0, .max = 63}, /* first range */
- {.min = 256, .max = 319} /* second range */
- };
- struct smc_range range_cycle[] = { /* SMC_CYCLE valid values */
- {.min = 0, .max = 127}, /* first range */
- {.min = 256, .max = 383}, /* second range */
- {.min = 512, .max = 639}, /* third range */
- {.min = 768, .max = 895} /* fourth range */
- };
-
- ret_val = adjust_smc_value(setup, range_setup, ARRAY_SIZE(range_setup));
- if (ret_val < 0)
- dev_warn(dev, "maximal SMC Setup value\n");
- else
- *cycle += ret_val;
-
- ret_val = adjust_smc_value(pulse, range_pulse, ARRAY_SIZE(range_pulse));
- if (ret_val < 0)
- dev_warn(dev, "maximal SMC Pulse value\n");
- else
- *cycle += ret_val;
-
- ret_val = adjust_smc_value(cycle, range_cycle, ARRAY_SIZE(range_cycle));
- if (ret_val < 0)
- dev_warn(dev, "maximal SMC Cycle value\n");
-
- *cs_pulse = *cycle;
- if (*cs_pulse > CS_PULSE_MAXIMUM) {
- dev_err(dev, "unable to calculate valid SMC settings\n");
- return -ER_SMC_CALC;
- }
-
- ret_val = adjust_smc_value(cs_pulse, range_pulse,
- ARRAY_SIZE(range_pulse));
- if (ret_val < 0) {
- dev_warn(dev, "maximal SMC CS Pulse value\n");
- } else if (ret_val != 0) {
- *cycle = *cs_pulse;
- dev_warn(dev, "SMC Cycle extended\n");
- err = -ER_SMC_RECALC;
- }
-
- return err;
-}
-
-/**
- * to_smc_format - convert values into SMC format
- * @setup: SETUP value of SMC Setup Register
- * @pulse: PULSE value of SMC Pulse Register
- * @cycle: CYCLE value of SMC Cycle Register
- * @cs_pulse: NCS_PULSE value of SMC Pulse Register
- */
-static void to_smc_format(int *setup, int *pulse, int *cycle, int *cs_pulse)
-{
- *setup = (*setup & 0x1f) | ((*setup & 0x80) >> 2);
- *pulse = (*pulse & 0x3f) | ((*pulse & 0x100) >> 2);
- *cycle = (*cycle & 0x7f) | ((*cycle & 0x300) >> 1);
- *cs_pulse = (*cs_pulse & 0x3f) | ((*cs_pulse & 0x100) >> 2);
-}
-
-static unsigned long calc_mck_cycles(unsigned long ns, unsigned long mck_hz)
-{
- unsigned long mul;
-
- /*
- * cycles = x [nsec] * f [Hz] / 10^9 [ns in sec] =
- * x * (f / 1_000_000_000) =
- * x * ((f * 65536) / 1_000_000_000) / 65536 =
- * x * (((f / 10_000) * 65536) / 100_000) / 65536 =
- */
-
- mul = (mck_hz / 10000) << 16;
- mul /= 100000;
-
- return (ns * mul + 65536) >> 16; /* rounding */
-}
-
-/**
- * set_smc_timing - SMC timings setup.
- * @dev: device
- * @info: AT91 IDE info
- * @ata: ATA timings
- *
- * Its assumed that write timings are same as read timings,
- * cs_setup = 0 and cs_pulse = cycle.
- */
-static void set_smc_timing(struct device *dev, struct ata_device *adev,
- struct at91_ide_info *info, const struct ata_timing *ata)
-{
- int ret = 0;
- int use_iordy;
- unsigned int t6z; /* data tristate time in ns */
- unsigned int cycle; /* SMC Cycle width in MCK ticks */
- unsigned int setup; /* SMC Setup width in MCK ticks */
- unsigned int pulse; /* CFIOR and CFIOW pulse width in MCK ticks */
- unsigned int cs_pulse; /* CS4 or CS5 pulse width in MCK ticks*/
- unsigned int tdf_cycles; /* SMC TDF MCK ticks */
- unsigned long mck_hz; /* MCK frequency in Hz */
-
- t6z = (ata->mode < XFER_PIO_5) ? 30 : 20;
- mck_hz = clk_get_rate(info->mck);
- cycle = calc_mck_cycles(ata->cyc8b, mck_hz);
- setup = calc_mck_cycles(ata->setup, mck_hz);
- pulse = calc_mck_cycles(ata->act8b, mck_hz);
- tdf_cycles = calc_mck_cycles(t6z, mck_hz);
-
- do {
- ret = calc_smc_vals(dev, &setup, &pulse, &cycle, &cs_pulse);
- } while (ret == -ER_SMC_RECALC);
-
- if (ret == -ER_SMC_CALC)
- dev_err(dev, "Interface may not operate correctly\n");
-
- dev_dbg(dev, "SMC Setup=%u, Pulse=%u, Cycle=%u, CS Pulse=%u\n",
- setup, pulse, cycle, cs_pulse);
- to_smc_format(&setup, &pulse, &cycle, &cs_pulse);
- /* disable or enable waiting for IORDY signal */
- use_iordy = ata_pio_need_iordy(adev);
- if (use_iordy)
- info->mode |= AT91_SMC_EXNWMODE_READY;
-
- if (tdf_cycles > 15) {
- tdf_cycles = 15;
- dev_warn(dev, "maximal SMC TDF Cycles value\n");
- }
-
- dev_dbg(dev, "Use IORDY=%u, TDF Cycles=%u\n", use_iordy, tdf_cycles);
-
- regmap_fields_write(fields.setup, info->cs,
- AT91SAM9_SMC_NRDSETUP(setup) |
- AT91SAM9_SMC_NWESETUP(setup) |
- AT91SAM9_SMC_NCS_NRDSETUP(0) |
- AT91SAM9_SMC_NCS_WRSETUP(0));
- regmap_fields_write(fields.pulse, info->cs,
- AT91SAM9_SMC_NRDPULSE(pulse) |
- AT91SAM9_SMC_NWEPULSE(pulse) |
- AT91SAM9_SMC_NCS_NRDPULSE(cs_pulse) |
- AT91SAM9_SMC_NCS_WRPULSE(cs_pulse));
- regmap_fields_write(fields.cycle, info->cs,
- AT91SAM9_SMC_NRDCYCLE(cycle) |
- AT91SAM9_SMC_NWECYCLE(cycle));
- regmap_fields_write(fields.mode, info->cs, info->mode |
- AT91_SMC_TDF_(tdf_cycles));
-}
-
-static void pata_at91_set_piomode(struct ata_port *ap, struct ata_device *adev)
-{
- struct at91_ide_info *info = ap->host->private_data;
- struct ata_timing timing;
- int ret;
-
- /* Compute ATA timing and set it to SMC */
- ret = ata_timing_compute(adev, adev->pio_mode, &timing, 1000, 0);
- if (ret) {
- dev_warn(ap->dev, "Failed to compute ATA timing %d, "
- "set PIO_0 timing\n", ret);
- timing = *ata_timing_find_mode(XFER_PIO_0);
- }
- set_smc_timing(ap->dev, adev, info, &timing);
-}
-
-static unsigned int pata_at91_data_xfer_noirq(struct ata_queued_cmd *qc,
- unsigned char *buf, unsigned int buflen, int rw)
-{
- struct at91_ide_info *info = qc->dev->link->ap->host->private_data;
- unsigned int consumed;
- unsigned int mode;
- unsigned long flags;
-
- local_irq_save(flags);
- regmap_fields_read(fields.mode, info->cs, &mode);
-
- /* set 16bit mode before writing data */
- regmap_fields_write(fields.mode, info->cs, (mode & ~AT91_SMC_DBW) |
- AT91_SMC_DBW_16);
-
- consumed = ata_sff_data_xfer(qc, buf, buflen, rw);
-
- /* restore 8bit mode after data is written */
- regmap_fields_write(fields.mode, info->cs, (mode & ~AT91_SMC_DBW) |
- AT91_SMC_DBW_8);
-
- local_irq_restore(flags);
- return consumed;
-}
-
-static struct scsi_host_template pata_at91_sht = {
- ATA_PIO_SHT(DRV_NAME),
-};
-
-static struct ata_port_operations pata_at91_port_ops = {
- .inherits = &ata_sff_port_ops,
-
- .sff_data_xfer = pata_at91_data_xfer_noirq,
- .set_piomode = pata_at91_set_piomode,
- .cable_detect = ata_cable_40wire,
-};
-
-static int at91sam9_smc_fields_init(struct device *dev)
-{
- struct reg_field field = REG_FIELD(0, 0, 31);
-
- field.id_size = 8;
- field.id_offset = AT91SAM9_SMC_GENERIC_BLK_SZ;
-
- field.reg = AT91SAM9_SMC_SETUP(AT91SAM9_SMC_GENERIC);
- fields.setup = devm_regmap_field_alloc(dev, smc, field);
- if (IS_ERR(fields.setup))
- return PTR_ERR(fields.setup);
-
- field.reg = AT91SAM9_SMC_PULSE(AT91SAM9_SMC_GENERIC);
- fields.pulse = devm_regmap_field_alloc(dev, smc, field);
- if (IS_ERR(fields.pulse))
- return PTR_ERR(fields.pulse);
-
- field.reg = AT91SAM9_SMC_CYCLE(AT91SAM9_SMC_GENERIC);
- fields.cycle = devm_regmap_field_alloc(dev, smc, field);
- if (IS_ERR(fields.cycle))
- return PTR_ERR(fields.cycle);
-
- field.reg = AT91SAM9_SMC_MODE(AT91SAM9_SMC_GENERIC);
- fields.mode = devm_regmap_field_alloc(dev, smc, field);
-
- return PTR_ERR_OR_ZERO(fields.mode);
-}
-
-static int pata_at91_probe(struct platform_device *pdev)
-{
- struct at91_cf_data *board = dev_get_platdata(&pdev->dev);
- struct device *dev = &pdev->dev;
- struct at91_ide_info *info;
- struct resource *mem_res;
- struct ata_host *host;
- struct ata_port *ap;
-
- int irq_flags = 0;
- int irq = 0;
- int ret;
-
- /* get platform resources: IO/CTL memories and irq/rst pins */
-
- if (pdev->num_resources != 1) {
- dev_err(&pdev->dev, "invalid number of resources\n");
- return -EINVAL;
- }
-
- mem_res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
-
- if (!mem_res) {
- dev_err(dev, "failed to get mem resource\n");
- return -EINVAL;
- }
-
- irq = board->irq_pin;
-
- smc = syscon_regmap_lookup_by_phandle(pdev->dev.of_node, "atmel,smc");
- if (IS_ERR(smc))
- return PTR_ERR(smc);
-
- ret = at91sam9_smc_fields_init(dev);
- if (ret < 0)
- return ret;
-
- /* init ata host */
-
- host = ata_host_alloc(dev, 1);
-
- if (!host)
- return -ENOMEM;
-
- ap = host->ports[0];
- ap->ops = &pata_at91_port_ops;
- ap->flags |= ATA_FLAG_SLAVE_POSS;
- ap->pio_mask = ATA_PIO4;
-
- if (!gpio_is_valid(irq)) {
- ap->flags |= ATA_FLAG_PIO_POLLING;
- ata_port_desc(ap, "no IRQ, using PIO polling");
- }
-
- info = devm_kzalloc(dev, sizeof(*info), GFP_KERNEL);
-
- if (!info) {
- dev_err(dev, "failed to allocate memory for private data\n");
- return -ENOMEM;
- }
-
- info->mck = clk_get(NULL, "mck");
-
- if (IS_ERR(info->mck)) {
- dev_err(dev, "failed to get access to mck clock\n");
- return -ENODEV;
- }
-
- info->cs = board->chipselect;
- info->mode = AT91_SMC_READMODE | AT91_SMC_WRITEMODE |
- AT91_SMC_EXNWMODE_READY | AT91_SMC_BAT_SELECT |
- AT91_SMC_DBW_8 | AT91_SMC_TDF_(0);
-
- info->ide_addr = devm_ioremap(dev,
- mem_res->start + CF_IDE_OFFSET, CF_IDE_RES_SIZE);
-
- if (!info->ide_addr) {
- dev_err(dev, "failed to map IO base\n");
- ret = -ENOMEM;
- goto err_put;
- }
-
- info->alt_addr = devm_ioremap(dev,
- mem_res->start + CF_ALT_IDE_OFFSET, CF_IDE_RES_SIZE);
-
- if (!info->alt_addr) {
- dev_err(dev, "failed to map CTL base\n");
- ret = -ENOMEM;
- goto err_put;
- }
-
- ap->ioaddr.cmd_addr = info->ide_addr;
- ap->ioaddr.ctl_addr = info->alt_addr + 0x06;
- ap->ioaddr.altstatus_addr = ap->ioaddr.ctl_addr;
-
- ata_sff_std_ports(&ap->ioaddr);
-
- ata_port_desc(ap, "mmio cmd 0x%llx ctl 0x%llx",
- (unsigned long long)mem_res->start + CF_IDE_OFFSET,
- (unsigned long long)mem_res->start + CF_ALT_IDE_OFFSET);
-
- host->private_data = info;
-
- ret = ata_host_activate(host, gpio_is_valid(irq) ? gpio_to_irq(irq) : 0,
- gpio_is_valid(irq) ? ata_sff_interrupt : NULL,
- irq_flags, &pata_at91_sht);
- if (ret)
- goto err_put;
-
- return 0;
-
-err_put:
- clk_put(info->mck);
- return ret;
-}
-
-static int pata_at91_remove(struct platform_device *pdev)
-{
- struct ata_host *host = platform_get_drvdata(pdev);
- struct at91_ide_info *info;
-
- if (!host)
- return 0;
- info = host->private_data;
-
- ata_host_detach(host);
-
- if (!info)
- return 0;
-
- clk_put(info->mck);
-
- return 0;
-}
-
-static struct platform_driver pata_at91_driver = {
- .probe = pata_at91_probe,
- .remove = pata_at91_remove,
- .driver = {
- .name = DRV_NAME,
- },
-};
-
-module_platform_driver(pata_at91_driver);
-
-MODULE_LICENSE("GPL");
-MODULE_DESCRIPTION("Driver for CF in True IDE mode on AT91SAM9260 SoC");
-MODULE_AUTHOR("Matyukevich Sergey");
-MODULE_VERSION(DRV_VERSION);
-
diff --git a/drivers/ata/pata_macio.c b/drivers/ata/pata_macio.c
index e347e7a..0adcb40 100644
--- a/drivers/ata/pata_macio.c
+++ b/drivers/ata/pata_macio.c
@@ -1328,7 +1328,7 @@
}
#endif /* CONFIG_PM_SLEEP */
-static struct of_device_id pata_macio_match[] =
+static const struct of_device_id pata_macio_match[] =
{
{
.name = "IDE",
diff --git a/drivers/ata/pata_mpc52xx.c b/drivers/ata/pata_mpc52xx.c
index 252ba27..9730125 100644
--- a/drivers/ata/pata_mpc52xx.c
+++ b/drivers/ata/pata_mpc52xx.c
@@ -847,7 +847,7 @@
}
#endif
-static struct of_device_id mpc52xx_ata_of_match[] = {
+static const struct of_device_id mpc52xx_ata_of_match[] = {
{ .compatible = "fsl,mpc5200-ata", },
{ .compatible = "mpc5200-ata", },
{},
diff --git a/drivers/ata/pata_of_platform.c b/drivers/ata/pata_of_platform.c
index 201a32d..01161c1 100644
--- a/drivers/ata/pata_of_platform.c
+++ b/drivers/ata/pata_of_platform.c
@@ -67,7 +67,7 @@
reg_shift, pio_mask, &pata_platform_sht);
}
-static struct of_device_id pata_of_platform_match[] = {
+static const struct of_device_id pata_of_platform_match[] = {
{ .compatible = "ata-generic", },
{ },
};
diff --git a/drivers/ata/sata_fsl.c b/drivers/ata/sata_fsl.c
index a723ae92..01734d5 100644
--- a/drivers/ata/sata_fsl.c
+++ b/drivers/ata/sata_fsl.c
@@ -1612,7 +1612,7 @@
}
#endif
-static struct of_device_id fsl_sata_match[] = {
+static const struct of_device_id fsl_sata_match[] = {
{
.compatible = "fsl,pq-sata",
},
diff --git a/drivers/ata/sata_mv.c b/drivers/ata/sata_mv.c
index 00ce26d..b66bcda 100644
--- a/drivers/ata/sata_mv.c
+++ b/drivers/ata/sata_mv.c
@@ -4286,7 +4286,7 @@
#endif
#ifdef CONFIG_OF
-static struct of_device_id mv_sata_dt_ids[] = {
+static const struct of_device_id mv_sata_dt_ids[] = {
{ .compatible = "marvell,armada-370-sata", },
{ .compatible = "marvell,orion-sata", },
{},
diff --git a/drivers/atm/ambassador.c b/drivers/atm/ambassador.c
index 4a61079..906705e 100644
--- a/drivers/atm/ambassador.c
+++ b/drivers/atm/ambassador.c
@@ -2267,9 +2267,8 @@
dev->atm_dev->ci_range.vpi_bits = NUM_VPI_BITS;
dev->atm_dev->ci_range.vci_bits = NUM_VCI_BITS;
- init_timer(&dev->housekeeping);
- dev->housekeeping.function = do_housekeeping;
- dev->housekeeping.data = (unsigned long) dev;
+ setup_timer(&dev->housekeeping, do_housekeeping,
+ (unsigned long)dev);
mod_timer(&dev->housekeeping, jiffies);
// enable host interrupts
diff --git a/drivers/base/platform-msi.c b/drivers/base/platform-msi.c
index 0fc7c4d..d35e9a2 100644
--- a/drivers/base/platform-msi.c
+++ b/drivers/base/platform-msi.c
@@ -345,8 +345,7 @@
data->host_data = host_data;
domain = irq_domain_create_hierarchy(dev->msi_domain, 0, nvec,
- of_node_to_fwnode(dev->of_node),
- ops, data);
+ dev->fwnode, ops, data);
if (!domain)
goto free_priv;
diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index e697dec..ad19642 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -121,7 +121,9 @@
#define genpd_lock_interruptible(p) p->lock_ops->lock_interruptible(p)
#define genpd_unlock(p) p->lock_ops->unlock(p)
+#define genpd_status_on(genpd) (genpd->status == GPD_STATE_ACTIVE)
#define genpd_is_irq_safe(genpd) (genpd->flags & GENPD_FLAG_IRQ_SAFE)
+#define genpd_is_always_on(genpd) (genpd->flags & GENPD_FLAG_ALWAYS_ON)
static inline bool irq_safe_dev_in_no_sleep_domain(struct device *dev,
struct generic_pm_domain *genpd)
@@ -130,8 +132,12 @@
ret = pm_runtime_is_irq_safe(dev) && !genpd_is_irq_safe(genpd);
- /* Warn once if IRQ safe dev in no sleep domain */
- if (ret)
+ /*
+ * Warn once if an IRQ safe device is attached to a no sleep domain, as
+ * to indicate a suboptimal configuration for PM. For an always on
+ * domain this isn't case, thus don't warn.
+ */
+ if (ret && !genpd_is_always_on(genpd))
dev_warn_once(dev, "PM domain %s will not be powered off\n",
genpd->name);
@@ -296,11 +302,15 @@
* (1) The domain is already in the "power off" state.
* (2) System suspend is in progress.
*/
- if (genpd->status == GPD_STATE_POWER_OFF
- || genpd->prepared_count > 0)
+ if (!genpd_status_on(genpd) || genpd->prepared_count > 0)
return 0;
- if (atomic_read(&genpd->sd_count) > 0)
+ /*
+ * Abort power off for the PM domain in the following situations:
+ * (1) The domain is configured as always on.
+ * (2) When the domain has a subdomain being powered on.
+ */
+ if (genpd_is_always_on(genpd) || atomic_read(&genpd->sd_count) > 0)
return -EBUSY;
list_for_each_entry(pdd, &genpd->dev_list, list_node) {
@@ -373,7 +383,7 @@
struct gpd_link *link;
int ret = 0;
- if (genpd->status == GPD_STATE_ACTIVE)
+ if (genpd_status_on(genpd))
return 0;
/*
@@ -752,7 +762,7 @@
{
struct gpd_link *link;
- if (genpd->status == GPD_STATE_POWER_OFF)
+ if (!genpd_status_on(genpd) || genpd_is_always_on(genpd))
return;
if (genpd->suspended_count != genpd->device_count
@@ -761,7 +771,8 @@
/* Choose the deepest state when suspending */
genpd->state_idx = genpd->state_count - 1;
- _genpd_power_off(genpd, false);
+ if (_genpd_power_off(genpd, false))
+ return;
genpd->status = GPD_STATE_POWER_OFF;
@@ -793,7 +804,7 @@
{
struct gpd_link *link;
- if (genpd->status == GPD_STATE_ACTIVE)
+ if (genpd_status_on(genpd))
return;
list_for_each_entry(link, &genpd->slave_links, slave_node) {
@@ -1329,8 +1340,7 @@
genpd_lock(subdomain);
genpd_lock_nested(genpd, SINGLE_DEPTH_NESTING);
- if (genpd->status == GPD_STATE_POWER_OFF
- && subdomain->status != GPD_STATE_POWER_OFF) {
+ if (!genpd_status_on(genpd) && genpd_status_on(subdomain)) {
ret = -EINVAL;
goto out;
}
@@ -1346,7 +1356,7 @@
list_add_tail(&link->master_node, &genpd->master_links);
link->slave = subdomain;
list_add_tail(&link->slave_node, &subdomain->slave_links);
- if (subdomain->status != GPD_STATE_POWER_OFF)
+ if (genpd_status_on(subdomain))
genpd_sd_counter_inc(genpd);
out:
@@ -1406,7 +1416,7 @@
list_del(&link->master_node);
list_del(&link->slave_node);
kfree(link);
- if (subdomain->status != GPD_STATE_POWER_OFF)
+ if (genpd_status_on(subdomain))
genpd_sd_counter_dec(genpd);
ret = 0;
@@ -1492,6 +1502,10 @@
genpd->dev_ops.start = pm_clk_resume;
}
+ /* Always-on domains must be powered on at initialization. */
+ if (genpd_is_always_on(genpd) && !genpd_status_on(genpd))
+ return -EINVAL;
+
/* Use only one "off" state if there were no states declared */
if (genpd->state_count == 0) {
ret = genpd_set_default_power_state(genpd);
@@ -1700,12 +1714,12 @@
mutex_lock(&gpd_list_lock);
- if (pm_genpd_present(genpd))
+ if (pm_genpd_present(genpd)) {
ret = genpd_add_provider(np, genpd_xlate_simple, genpd);
-
- if (!ret) {
- genpd->provider = &np->fwnode;
- genpd->has_provider = true;
+ if (!ret) {
+ genpd->provider = &np->fwnode;
+ genpd->has_provider = true;
+ }
}
mutex_unlock(&gpd_list_lock);
@@ -2079,11 +2093,6 @@
int err;
u32 residency;
u32 entry_latency, exit_latency;
- const struct of_device_id *match_id;
-
- match_id = of_match_node(idle_state_match, state_node);
- if (!match_id)
- return -EINVAL;
err = of_property_read_u32(state_node, "entry-latency-us",
&entry_latency);
@@ -2132,6 +2141,7 @@
int err, ret;
int count;
struct of_phandle_iterator it;
+ const struct of_device_id *match_id;
count = of_count_phandle_with_args(dn, "domain-idle-states", NULL);
if (count <= 0)
@@ -2144,6 +2154,9 @@
/* Loop over the phandles until all the requested entry is found */
of_for_each_phandle(&it, err, dn, "domain-idle-states", NULL, 0) {
np = it.node;
+ match_id = of_match_node(idle_state_match, np);
+ if (!match_id)
+ continue;
ret = genpd_parse_state(&st[i++], np);
if (ret) {
pr_err
@@ -2155,8 +2168,11 @@
}
}
- *n = count;
- *states = st;
+ *n = i;
+ if (!i)
+ kfree(st);
+ else
+ *states = st;
return 0;
}
@@ -2221,7 +2237,7 @@
if (WARN_ON(genpd->status >= ARRAY_SIZE(status_lookup)))
goto exit;
- if (genpd->status == GPD_STATE_POWER_OFF)
+ if (!genpd_status_on(genpd))
snprintf(state, sizeof(state), "%s-%u",
status_lookup[genpd->status], genpd->state_idx);
else
diff --git a/drivers/base/property.c b/drivers/base/property.c
index c458c63..149de31 100644
--- a/drivers/base/property.c
+++ b/drivers/base/property.c
@@ -15,6 +15,7 @@
#include <linux/kernel.h>
#include <linux/of.h>
#include <linux/of_address.h>
+#include <linux/of_graph.h>
#include <linux/property.h>
#include <linux/etherdevice.h>
#include <linux/phy.h>
@@ -146,47 +147,45 @@
const char *propname,
const char **strings, size_t nval)
{
+ const struct property_entry *prop;
const void *pointer;
- size_t length = nval * sizeof(*strings);
+ size_t array_len, length;
+
+ /* Find out the array length. */
+ prop = pset_prop_get(pset, propname);
+ if (!prop)
+ return -EINVAL;
+
+ if (!prop->is_array)
+ /* The array length for a non-array string property is 1. */
+ array_len = 1;
+ else
+ /* Find the length of an array. */
+ array_len = pset_prop_count_elems_of_size(pset, propname,
+ sizeof(const char *));
+
+ /* Return how many there are if strings is NULL. */
+ if (!strings)
+ return array_len;
+
+ array_len = min(nval, array_len);
+ length = array_len * sizeof(*strings);
pointer = pset_prop_find(pset, propname, length);
if (IS_ERR(pointer))
return PTR_ERR(pointer);
memcpy(strings, pointer, length);
- return 0;
+
+ return array_len;
}
-static int pset_prop_read_string(struct property_set *pset,
- const char *propname, const char **strings)
-{
- const struct property_entry *prop;
- const char * const *pointer;
-
- prop = pset_prop_get(pset, propname);
- if (!prop)
- return -EINVAL;
- if (!prop->is_string)
- return -EILSEQ;
- if (prop->is_array) {
- pointer = prop->pointer.str;
- if (!pointer)
- return -ENODATA;
- } else {
- pointer = &prop->value.str;
- if (*pointer && strnlen(*pointer, prop->length) >= prop->length)
- return -EILSEQ;
- }
-
- *strings = *pointer;
- return 0;
-}
-
-static inline struct fwnode_handle *dev_fwnode(struct device *dev)
+struct fwnode_handle *dev_fwnode(struct device *dev)
{
return IS_ENABLED(CONFIG_OF) && dev->of_node ?
&dev->of_node->fwnode : dev->fwnode;
}
+EXPORT_SYMBOL_GPL(dev_fwnode);
/**
* device_property_present - check if a property of a device is present
@@ -340,8 +339,8 @@
* Function reads an array of string properties with @propname from the device
* firmware description and stores them to @val if found.
*
- * Return: number of values if @val was %NULL,
- * %0 if the property was found (success),
+ * Return: number of values read on success if @val is non-NULL,
+ * number of values available on success if @val is NULL,
* %-EINVAL if given arguments are not valid,
* %-ENODATA if the property does not have a value,
* %-EPROTO or %-EILSEQ if the property is not an array of strings,
@@ -553,25 +552,8 @@
return acpi_node_prop_read(fwnode, propname, DEV_PROP_STRING,
val, nval);
else if (is_pset_node(fwnode))
- return val ?
- pset_prop_read_string_array(to_pset_node(fwnode),
- propname, val, nval) :
- pset_prop_count_elems_of_size(to_pset_node(fwnode),
- propname,
- sizeof(const char *));
- return -ENXIO;
-}
-
-static int __fwnode_property_read_string(struct fwnode_handle *fwnode,
- const char *propname, const char **val)
-{
- if (is_of_node(fwnode))
- return of_property_read_string(to_of_node(fwnode), propname, val);
- else if (is_acpi_node(fwnode))
- return acpi_node_prop_read(fwnode, propname, DEV_PROP_STRING,
- val, 1);
- else if (is_pset_node(fwnode))
- return pset_prop_read_string(to_pset_node(fwnode), propname, val);
+ return pset_prop_read_string_array(to_pset_node(fwnode),
+ propname, val, nval);
return -ENXIO;
}
@@ -585,11 +567,11 @@
* Read an string list property @propname from the given firmware node and store
* them to @val if found.
*
- * Return: number of values if @val was %NULL,
- * %0 if the property was found (success),
+ * Return: number of values read on success if @val is non-NULL,
+ * number of values available on success if @val is NULL,
* %-EINVAL if given arguments are not valid,
* %-ENODATA if the property does not have a value,
- * %-EPROTO if the property is not an array of strings,
+ * %-EPROTO or %-EILSEQ if the property is not an array of strings,
* %-EOVERFLOW if the size of the property is not as expected,
* %-ENXIO if no suitable firmware interface is present.
*/
@@ -626,14 +608,9 @@
int fwnode_property_read_string(struct fwnode_handle *fwnode,
const char *propname, const char **val)
{
- int ret;
+ int ret = fwnode_property_read_string_array(fwnode, propname, val, 1);
- ret = __fwnode_property_read_string(fwnode, propname, val);
- if (ret == -EINVAL && !IS_ERR_OR_NULL(fwnode) &&
- !IS_ERR_OR_NULL(fwnode->secondary))
- ret = __fwnode_property_read_string(fwnode->secondary,
- propname, val);
- return ret;
+ return ret < 0 ? ret : 0;
}
EXPORT_SYMBOL_GPL(fwnode_property_read_string);
@@ -932,6 +909,75 @@
EXPORT_SYMBOL_GPL(device_add_properties);
/**
+ * fwnode_get_next_parent - Iterate to the node's parent
+ * @fwnode: Firmware whose parent is retrieved
+ *
+ * This is like fwnode_get_parent() except that it drops the refcount
+ * on the passed node, making it suitable for iterating through a
+ * node's parents.
+ *
+ * Returns a node pointer with refcount incremented, use
+ * fwnode_handle_node() on it when done.
+ */
+struct fwnode_handle *fwnode_get_next_parent(struct fwnode_handle *fwnode)
+{
+ struct fwnode_handle *parent = fwnode_get_parent(fwnode);
+
+ fwnode_handle_put(fwnode);
+
+ return parent;
+}
+EXPORT_SYMBOL_GPL(fwnode_get_next_parent);
+
+/**
+ * fwnode_get_parent - Return parent firwmare node
+ * @fwnode: Firmware whose parent is retrieved
+ *
+ * Return parent firmware node of the given node if possible or %NULL if no
+ * parent was available.
+ */
+struct fwnode_handle *fwnode_get_parent(struct fwnode_handle *fwnode)
+{
+ struct fwnode_handle *parent = NULL;
+
+ if (is_of_node(fwnode)) {
+ struct device_node *node;
+
+ node = of_get_parent(to_of_node(fwnode));
+ if (node)
+ parent = &node->fwnode;
+ } else if (is_acpi_node(fwnode)) {
+ parent = acpi_node_get_parent(fwnode);
+ }
+
+ return parent;
+}
+EXPORT_SYMBOL_GPL(fwnode_get_parent);
+
+/**
+ * fwnode_get_next_child_node - Return the next child node handle for a node
+ * @fwnode: Firmware node to find the next child node for.
+ * @child: Handle to one of the node's child nodes or a %NULL handle.
+ */
+struct fwnode_handle *fwnode_get_next_child_node(struct fwnode_handle *fwnode,
+ struct fwnode_handle *child)
+{
+ if (is_of_node(fwnode)) {
+ struct device_node *node;
+
+ node = of_get_next_available_child(to_of_node(fwnode),
+ to_of_node(child));
+ if (node)
+ return &node->fwnode;
+ } else if (is_acpi_node(fwnode)) {
+ return acpi_get_next_subnode(fwnode, child);
+ }
+
+ return NULL;
+}
+EXPORT_SYMBOL_GPL(fwnode_get_next_child_node);
+
+/**
* device_get_next_child_node - Return the next child node handle for a device
* @dev: Device to find the next child node for.
* @child: Handle to one of the device's child nodes or a null handle.
@@ -939,34 +985,33 @@
struct fwnode_handle *device_get_next_child_node(struct device *dev,
struct fwnode_handle *child)
{
- if (IS_ENABLED(CONFIG_OF) && dev->of_node) {
- struct device_node *node;
+ struct acpi_device *adev = ACPI_COMPANION(dev);
+ struct fwnode_handle *fwnode = NULL;
- node = of_get_next_available_child(dev->of_node, to_of_node(child));
- if (node)
- return &node->fwnode;
- } else if (IS_ENABLED(CONFIG_ACPI)) {
- return acpi_get_next_subnode(dev, child);
- }
- return NULL;
+ if (dev->of_node)
+ fwnode = &dev->of_node->fwnode;
+ else if (adev)
+ fwnode = acpi_fwnode_handle(adev);
+
+ return fwnode_get_next_child_node(fwnode, child);
}
EXPORT_SYMBOL_GPL(device_get_next_child_node);
/**
- * device_get_named_child_node - Return first matching named child node handle
- * @dev: Device to find the named child node for.
+ * fwnode_get_named_child_node - Return first matching named child node handle
+ * @fwnode: Firmware node to find the named child node for.
* @childname: String to match child node name against.
*/
-struct fwnode_handle *device_get_named_child_node(struct device *dev,
+struct fwnode_handle *fwnode_get_named_child_node(struct fwnode_handle *fwnode,
const char *childname)
{
struct fwnode_handle *child;
/*
- * Find first matching named child node of this device.
+ * Find first matching named child node of this fwnode.
* For ACPI this will be a data only sub-node.
*/
- device_for_each_child_node(dev, child) {
+ fwnode_for_each_child_node(fwnode, child) {
if (is_of_node(child)) {
if (!of_node_cmp(to_of_node(child)->name, childname))
return child;
@@ -978,9 +1023,32 @@
return NULL;
}
+EXPORT_SYMBOL_GPL(fwnode_get_named_child_node);
+
+/**
+ * device_get_named_child_node - Return first matching named child node handle
+ * @dev: Device to find the named child node for.
+ * @childname: String to match child node name against.
+ */
+struct fwnode_handle *device_get_named_child_node(struct device *dev,
+ const char *childname)
+{
+ return fwnode_get_named_child_node(dev_fwnode(dev), childname);
+}
EXPORT_SYMBOL_GPL(device_get_named_child_node);
/**
+ * fwnode_handle_get - Obtain a reference to a device node
+ * @fwnode: Pointer to the device node to obtain the reference to.
+ */
+void fwnode_handle_get(struct fwnode_handle *fwnode)
+{
+ if (is_of_node(fwnode))
+ of_node_get(to_of_node(fwnode));
+}
+EXPORT_SYMBOL_GPL(fwnode_handle_get);
+
+/**
* fwnode_handle_put - Drop reference to a device node
* @fwnode: Pointer to the device node to drop the reference to.
*
@@ -1117,3 +1185,157 @@
return device_get_mac_addr(dev, "address", addr, alen);
}
EXPORT_SYMBOL(device_get_mac_address);
+
+/**
+ * device_graph_get_next_endpoint - Get next endpoint firmware node
+ * @fwnode: Pointer to the parent firmware node
+ * @prev: Previous endpoint node or %NULL to get the first
+ *
+ * Returns an endpoint firmware node pointer or %NULL if no more endpoints
+ * are available.
+ */
+struct fwnode_handle *
+fwnode_graph_get_next_endpoint(struct fwnode_handle *fwnode,
+ struct fwnode_handle *prev)
+{
+ struct fwnode_handle *endpoint = NULL;
+
+ if (is_of_node(fwnode)) {
+ struct device_node *node;
+
+ node = of_graph_get_next_endpoint(to_of_node(fwnode),
+ to_of_node(prev));
+
+ if (node)
+ endpoint = &node->fwnode;
+ } else if (is_acpi_node(fwnode)) {
+ endpoint = acpi_graph_get_next_endpoint(fwnode, prev);
+ if (IS_ERR(endpoint))
+ endpoint = NULL;
+ }
+
+ return endpoint;
+
+}
+EXPORT_SYMBOL_GPL(fwnode_graph_get_next_endpoint);
+
+/**
+ * fwnode_graph_get_remote_port_parent - Return fwnode of a remote device
+ * @fwnode: Endpoint firmware node pointing to the remote endpoint
+ *
+ * Extracts firmware node of a remote device the @fwnode points to.
+ */
+struct fwnode_handle *
+fwnode_graph_get_remote_port_parent(struct fwnode_handle *fwnode)
+{
+ struct fwnode_handle *parent = NULL;
+
+ if (is_of_node(fwnode)) {
+ struct device_node *node;
+
+ node = of_graph_get_remote_port_parent(to_of_node(fwnode));
+ if (node)
+ parent = &node->fwnode;
+ } else if (is_acpi_node(fwnode)) {
+ int ret;
+
+ ret = acpi_graph_get_remote_endpoint(fwnode, &parent, NULL,
+ NULL);
+ if (ret)
+ return NULL;
+ }
+
+ return parent;
+}
+EXPORT_SYMBOL_GPL(fwnode_graph_get_remote_port_parent);
+
+/**
+ * fwnode_graph_get_remote_port - Return fwnode of a remote port
+ * @fwnode: Endpoint firmware node pointing to the remote endpoint
+ *
+ * Extracts firmware node of a remote port the @fwnode points to.
+ */
+struct fwnode_handle *fwnode_graph_get_remote_port(struct fwnode_handle *fwnode)
+{
+ struct fwnode_handle *port = NULL;
+
+ if (is_of_node(fwnode)) {
+ struct device_node *node;
+
+ node = of_graph_get_remote_port(to_of_node(fwnode));
+ if (node)
+ port = &node->fwnode;
+ } else if (is_acpi_node(fwnode)) {
+ int ret;
+
+ ret = acpi_graph_get_remote_endpoint(fwnode, NULL, &port, NULL);
+ if (ret)
+ return NULL;
+ }
+
+ return port;
+}
+EXPORT_SYMBOL_GPL(fwnode_graph_get_remote_port);
+
+/**
+ * fwnode_graph_get_remote_endpoint - Return fwnode of a remote endpoint
+ * @fwnode: Endpoint firmware node pointing to the remote endpoint
+ *
+ * Extracts firmware node of a remote endpoint the @fwnode points to.
+ */
+struct fwnode_handle *
+fwnode_graph_get_remote_endpoint(struct fwnode_handle *fwnode)
+{
+ struct fwnode_handle *endpoint = NULL;
+
+ if (is_of_node(fwnode)) {
+ struct device_node *node;
+
+ node = of_parse_phandle(to_of_node(fwnode), "remote-endpoint",
+ 0);
+ if (node)
+ endpoint = &node->fwnode;
+ } else if (is_acpi_node(fwnode)) {
+ int ret;
+
+ ret = acpi_graph_get_remote_endpoint(fwnode, NULL, NULL,
+ &endpoint);
+ if (ret)
+ return NULL;
+ }
+
+ return endpoint;
+}
+EXPORT_SYMBOL_GPL(fwnode_graph_get_remote_endpoint);
+
+/**
+ * fwnode_graph_parse_endpoint - parse common endpoint node properties
+ * @fwnode: pointer to endpoint fwnode_handle
+ * @endpoint: pointer to the fwnode endpoint data structure
+ *
+ * Parse @fwnode representing a graph endpoint node and store the
+ * information in @endpoint. The caller must hold a reference to
+ * @fwnode.
+ */
+int fwnode_graph_parse_endpoint(struct fwnode_handle *fwnode,
+ struct fwnode_endpoint *endpoint)
+{
+ struct fwnode_handle *port_fwnode = fwnode_get_parent(fwnode);
+
+ memset(endpoint, 0, sizeof(*endpoint));
+
+ endpoint->local_fwnode = fwnode;
+
+ if (is_acpi_node(port_fwnode)) {
+ fwnode_property_read_u32(port_fwnode, "port", &endpoint->port);
+ fwnode_property_read_u32(fwnode, "endpoint", &endpoint->id);
+ } else {
+ fwnode_property_read_u32(port_fwnode, "reg", &endpoint->port);
+ fwnode_property_read_u32(fwnode, "reg", &endpoint->id);
+ }
+
+ fwnode_handle_put(port_fwnode);
+
+ return 0;
+}
+EXPORT_SYMBOL(fwnode_graph_parse_endpoint);
diff --git a/drivers/bcma/driver_gpio.c b/drivers/bcma/driver_gpio.c
index 771a2a2..7bde8d7 100644
--- a/drivers/bcma/driver_gpio.c
+++ b/drivers/bcma/driver_gpio.c
@@ -185,8 +185,7 @@
chip->owner = THIS_MODULE;
chip->parent = bcma_bus_get_host_dev(bus);
#if IS_BUILTIN(CONFIG_OF)
- if (cc->core->bus->hosttype == BCMA_HOSTTYPE_SOC)
- chip->of_node = cc->core->dev.of_node;
+ chip->of_node = cc->core->dev.of_node;
#endif
switch (bus->chipinfo.id) {
case BCMA_CHIP_ID_BCM4707:
diff --git a/drivers/bcma/main.c b/drivers/bcma/main.c
index 12da68e..e6986c7 100644
--- a/drivers/bcma/main.c
+++ b/drivers/bcma/main.c
@@ -201,9 +201,6 @@
{
struct device_node *node;
- if (!IS_ENABLED(CONFIG_OF_IRQ))
- return;
-
node = bcma_of_find_child_device(parent, core);
if (node)
core->dev.of_node = node;
@@ -242,19 +239,18 @@
core->dev.release = bcma_release_core_dev;
core->dev.bus = &bcma_bus_type;
dev_set_name(&core->dev, "bcma%d:%d", bus->num, core->core_index);
+ core->dev.parent = bcma_bus_get_host_dev(bus);
+ if (core->dev.parent)
+ bcma_of_fill_device(core->dev.parent, core);
switch (bus->hosttype) {
case BCMA_HOSTTYPE_PCI:
- core->dev.parent = &bus->host_pci->dev;
core->dma_dev = &bus->host_pci->dev;
core->irq = bus->host_pci->irq;
break;
case BCMA_HOSTTYPE_SOC:
if (IS_ENABLED(CONFIG_OF) && bus->host_pdev) {
core->dma_dev = &bus->host_pdev->dev;
- core->dev.parent = &bus->host_pdev->dev;
- if (core->dev.parent)
- bcma_of_fill_device(core->dev.parent, core);
} else {
core->dev.dma_mask = &core->dev.coherent_dma_mask;
core->dma_dev = &core->dev;
diff --git a/drivers/block/Kconfig b/drivers/block/Kconfig
index f744de7..d545abb 100644
--- a/drivers/block/Kconfig
+++ b/drivers/block/Kconfig
@@ -219,7 +219,7 @@
To use the loop device, you need the losetup utility, found in the
util-linux package, see
- <ftp://ftp.kernel.org/pub/linux/utils/util-linux/>.
+ <https://www.kernel.org/pub/linux/utils/util-linux/>.
The loop device driver can also be used to "hide" a file system in
a disk partition, floppy, or regular file, either using encryption
@@ -312,22 +312,6 @@
Use device /dev/skd$N amd /dev/skd$Np$M.
-config BLK_DEV_OSD
- tristate "OSD object-as-blkdev support"
- depends on SCSI_OSD_ULD
- ---help---
- Saying Y or M here will allow the exporting of a single SCSI
- OSD (object-based storage) object as a Linux block device.
-
- For example, if you create a 2G object on an OSD device,
- you can then use this module to present that 2G object as
- a Linux block device.
-
- To compile this driver as a module, choose M here: the
- module will be called osdblk.
-
- If unsure, say N.
-
config BLK_DEV_SX8
tristate "Promise SATA SX8 support"
depends on PCI
@@ -434,23 +418,6 @@
This driver provides Support for ATA over Ethernet block
devices like the Coraid EtherDrive (R) Storage Blade.
-config MG_DISK
- tristate "mGine mflash, gflash support"
- depends on ARM && GPIOLIB
- help
- mGine mFlash(gFlash) block device driver
-
-config MG_DISK_RES
- int "Size of reserved area before MBR"
- depends on MG_DISK
- default 0
- help
- Define size of reserved area that usually used for boot. Unit is KB.
- All of the block device operation will be taken this value as start
- offset
- Examples:
- 1024 => 1 MB
-
config SUNVDC
tristate "Sun Virtual Disk Client support"
depends on SUN_LDOMS
@@ -512,19 +479,7 @@
Enable support for SCSI passthrough (e.g. the SG_IO ioctl) on
virtio-blk devices. This is only supported for the legacy
virtio protocol and not enabled by default by any hypervisor.
- Your probably want to virtio-scsi instead.
-
-config BLK_DEV_HD
- bool "Very old hard disk (MFM/RLL/IDE) driver"
- depends on HAVE_IDE
- depends on !ARM || ARCH_RPC || BROKEN
- help
- This is a very old hard disk driver that lacks the enhanced
- functionality of the newer ones.
-
- It is required for systems with ancient MFM/RLL/ESDI drives.
-
- If unsure, say N.
+ You probably want to use virtio-scsi instead.
config BLK_DEV_RBD
tristate "Rados block device (RBD)"
diff --git a/drivers/block/Makefile b/drivers/block/Makefile
index 1e9661e..ec8c368 100644
--- a/drivers/block/Makefile
+++ b/drivers/block/Makefile
@@ -19,10 +19,8 @@
obj-$(CONFIG_BLK_DEV_DAC960) += DAC960.o
obj-$(CONFIG_XILINX_SYSACE) += xsysace.o
obj-$(CONFIG_CDROM_PKTCDVD) += pktcdvd.o
-obj-$(CONFIG_MG_DISK) += mg_disk.o
obj-$(CONFIG_SUNVDC) += sunvdc.o
obj-$(CONFIG_BLK_DEV_SKD) += skd.o
-obj-$(CONFIG_BLK_DEV_OSD) += osdblk.o
obj-$(CONFIG_BLK_DEV_UMEM) += umem.o
obj-$(CONFIG_BLK_DEV_NBD) += nbd.o
@@ -30,7 +28,6 @@
obj-$(CONFIG_VIRTIO_BLK) += virtio_blk.o
obj-$(CONFIG_BLK_DEV_SX8) += sx8.o
-obj-$(CONFIG_BLK_DEV_HD) += hd.o
obj-$(CONFIG_XEN_BLKDEV_FRONTEND) += xen-blkfront.o
obj-$(CONFIG_XEN_BLKDEV_BACKEND) += xen-blkback/
diff --git a/drivers/block/ataflop.c b/drivers/block/ataflop.c
index 2104b1b..fa69ecd 100644
--- a/drivers/block/ataflop.c
+++ b/drivers/block/ataflop.c
@@ -617,12 +617,12 @@
if (!fd_request)
return;
- fd_request->errors++;
- if (fd_request->errors >= MAX_ERRORS) {
+ fd_request->error_count++;
+ if (fd_request->error_count >= MAX_ERRORS) {
printk(KERN_ERR "fd%d: too many errors.\n", SelectedDrive );
fd_end_request_cur(-EIO);
}
- else if (fd_request->errors == RECALIBRATE_ERRORS) {
+ else if (fd_request->error_count == RECALIBRATE_ERRORS) {
printk(KERN_WARNING "fd%d: recalibrating\n", SelectedDrive );
if (SelectedDrive != -1)
SUD.track = -1;
@@ -1386,7 +1386,7 @@
ReqData = ReqBuffer + 512 * ReqCnt;
if (UseTrackbuffer)
- read_track = (ReqCmd == READ && fd_request->errors == 0);
+ read_track = (ReqCmd == READ && fd_request->error_count == 0);
else
read_track = 0;
@@ -1409,8 +1409,10 @@
fdc_queue = 0;
if (q) {
rq = blk_fetch_request(q);
- if (rq)
+ if (rq) {
+ rq->error_count = 0;
break;
+ }
}
} while (fdc_queue != old_pos);
diff --git a/drivers/block/brd.c b/drivers/block/brd.c
index 3adc32a..4ec84d5 100644
--- a/drivers/block/brd.c
+++ b/drivers/block/brd.c
@@ -134,28 +134,6 @@
return page;
}
-static void brd_free_page(struct brd_device *brd, sector_t sector)
-{
- struct page *page;
- pgoff_t idx;
-
- spin_lock(&brd->brd_lock);
- idx = sector >> PAGE_SECTORS_SHIFT;
- page = radix_tree_delete(&brd->brd_pages, idx);
- spin_unlock(&brd->brd_lock);
- if (page)
- __free_page(page);
-}
-
-static void brd_zero_page(struct brd_device *brd, sector_t sector)
-{
- struct page *page;
-
- page = brd_lookup_page(brd, sector);
- if (page)
- clear_highpage(page);
-}
-
/*
* Free all backing store pages and radix tree. This must only be called when
* there are no other users of the device.
@@ -212,24 +190,6 @@
return 0;
}
-static void discard_from_brd(struct brd_device *brd,
- sector_t sector, size_t n)
-{
- while (n >= PAGE_SIZE) {
- /*
- * Don't want to actually discard pages here because
- * re-allocating the pages can result in writeback
- * deadlocks under heavy load.
- */
- if (0)
- brd_free_page(brd, sector);
- else
- brd_zero_page(brd, sector);
- sector += PAGE_SIZE >> SECTOR_SHIFT;
- n -= PAGE_SIZE;
- }
-}
-
/*
* Copy n bytes from src to the brd starting at sector. Does not sleep.
*/
@@ -338,14 +298,6 @@
if (bio_end_sector(bio) > get_capacity(bdev->bd_disk))
goto io_error;
- if (unlikely(bio_op(bio) == REQ_OP_DISCARD)) {
- if (sector & ((PAGE_SIZE >> SECTOR_SHIFT) - 1) ||
- bio->bi_iter.bi_size & ~PAGE_MASK)
- goto io_error;
- discard_from_brd(brd, sector, bio->bi_iter.bi_size);
- goto out;
- }
-
bio_for_each_segment(bvec, bio, iter) {
unsigned int len = bvec.bv_len;
int err;
@@ -357,7 +309,6 @@
sector += len >> SECTOR_SHIFT;
}
-out:
bio_endio(bio);
return BLK_QC_T_NONE;
io_error:
@@ -464,11 +415,6 @@
* is harmless)
*/
blk_queue_physical_block_size(brd->brd_queue, PAGE_SIZE);
-
- brd->brd_queue->limits.discard_granularity = PAGE_SIZE;
- blk_queue_max_discard_sectors(brd->brd_queue, UINT_MAX);
- brd->brd_queue->limits.discard_zeroes_data = 1;
- queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, brd->brd_queue);
#ifdef CONFIG_BLK_DEV_RAM_DAX
queue_flag_set_unlocked(QUEUE_FLAG_DAX, brd->brd_queue);
#endif
diff --git a/drivers/block/cciss.c b/drivers/block/cciss.c
index 8e1a455..cd37550 100644
--- a/drivers/block/cciss.c
+++ b/drivers/block/cciss.c
@@ -1864,8 +1864,7 @@
/* set the residual count for pc requests */
if (blk_rq_is_passthrough(rq))
scsi_req(rq)->resid_len = c->err_info->ResidualCnt;
-
- blk_end_request_all(rq, (rq->errors == 0) ? 0 : -EIO);
+ blk_end_request_all(rq, scsi_req(rq)->result ? -EIO : 0);
spin_lock_irqsave(&h->lock, flags);
cmd_free(h, c);
@@ -3140,18 +3139,19 @@
{
int retry_cmd = 0;
struct request *rq = cmd->rq;
+ struct scsi_request *sreq = scsi_req(rq);
- rq->errors = 0;
+ sreq->result = 0;
if (timeout)
- rq->errors = make_status_bytes(0, 0, 0, DRIVER_TIMEOUT);
+ sreq->result = make_status_bytes(0, 0, 0, DRIVER_TIMEOUT);
if (cmd->err_info->CommandStatus == 0) /* no error has occurred */
goto after_error_processing;
switch (cmd->err_info->CommandStatus) {
case CMD_TARGET_STATUS:
- rq->errors = evaluate_target_status(h, cmd, &retry_cmd);
+ sreq->result = evaluate_target_status(h, cmd, &retry_cmd);
break;
case CMD_DATA_UNDERRUN:
if (!blk_rq_is_passthrough(cmd->rq)) {
@@ -3169,7 +3169,7 @@
case CMD_INVALID:
dev_warn(&h->pdev->dev, "cciss: cmd %p is "
"reported invalid\n", cmd);
- rq->errors = make_status_bytes(SAM_STAT_GOOD,
+ sreq->result = make_status_bytes(SAM_STAT_GOOD,
cmd->err_info->CommandStatus, DRIVER_OK,
blk_rq_is_passthrough(cmd->rq) ?
DID_PASSTHROUGH : DID_ERROR);
@@ -3177,7 +3177,7 @@
case CMD_PROTOCOL_ERR:
dev_warn(&h->pdev->dev, "cciss: cmd %p has "
"protocol error\n", cmd);
- rq->errors = make_status_bytes(SAM_STAT_GOOD,
+ sreq->result = make_status_bytes(SAM_STAT_GOOD,
cmd->err_info->CommandStatus, DRIVER_OK,
blk_rq_is_passthrough(cmd->rq) ?
DID_PASSTHROUGH : DID_ERROR);
@@ -3185,7 +3185,7 @@
case CMD_HARDWARE_ERR:
dev_warn(&h->pdev->dev, "cciss: cmd %p had "
" hardware error\n", cmd);
- rq->errors = make_status_bytes(SAM_STAT_GOOD,
+ sreq->result = make_status_bytes(SAM_STAT_GOOD,
cmd->err_info->CommandStatus, DRIVER_OK,
blk_rq_is_passthrough(cmd->rq) ?
DID_PASSTHROUGH : DID_ERROR);
@@ -3193,7 +3193,7 @@
case CMD_CONNECTION_LOST:
dev_warn(&h->pdev->dev, "cciss: cmd %p had "
"connection lost\n", cmd);
- rq->errors = make_status_bytes(SAM_STAT_GOOD,
+ sreq->result = make_status_bytes(SAM_STAT_GOOD,
cmd->err_info->CommandStatus, DRIVER_OK,
blk_rq_is_passthrough(cmd->rq) ?
DID_PASSTHROUGH : DID_ERROR);
@@ -3201,7 +3201,7 @@
case CMD_ABORTED:
dev_warn(&h->pdev->dev, "cciss: cmd %p was "
"aborted\n", cmd);
- rq->errors = make_status_bytes(SAM_STAT_GOOD,
+ sreq->result = make_status_bytes(SAM_STAT_GOOD,
cmd->err_info->CommandStatus, DRIVER_OK,
blk_rq_is_passthrough(cmd->rq) ?
DID_PASSTHROUGH : DID_ABORT);
@@ -3209,7 +3209,7 @@
case CMD_ABORT_FAILED:
dev_warn(&h->pdev->dev, "cciss: cmd %p reports "
"abort failed\n", cmd);
- rq->errors = make_status_bytes(SAM_STAT_GOOD,
+ sreq->result = make_status_bytes(SAM_STAT_GOOD,
cmd->err_info->CommandStatus, DRIVER_OK,
blk_rq_is_passthrough(cmd->rq) ?
DID_PASSTHROUGH : DID_ERROR);
@@ -3224,21 +3224,21 @@
} else
dev_warn(&h->pdev->dev,
"%p retried too many times\n", cmd);
- rq->errors = make_status_bytes(SAM_STAT_GOOD,
+ sreq->result = make_status_bytes(SAM_STAT_GOOD,
cmd->err_info->CommandStatus, DRIVER_OK,
blk_rq_is_passthrough(cmd->rq) ?
DID_PASSTHROUGH : DID_ABORT);
break;
case CMD_TIMEOUT:
dev_warn(&h->pdev->dev, "cmd %p timedout\n", cmd);
- rq->errors = make_status_bytes(SAM_STAT_GOOD,
+ sreq->result = make_status_bytes(SAM_STAT_GOOD,
cmd->err_info->CommandStatus, DRIVER_OK,
blk_rq_is_passthrough(cmd->rq) ?
DID_PASSTHROUGH : DID_ERROR);
break;
case CMD_UNABORTABLE:
dev_warn(&h->pdev->dev, "cmd %p unabortable\n", cmd);
- rq->errors = make_status_bytes(SAM_STAT_GOOD,
+ sreq->result = make_status_bytes(SAM_STAT_GOOD,
cmd->err_info->CommandStatus, DRIVER_OK,
blk_rq_is_passthrough(cmd->rq) ?
DID_PASSTHROUGH : DID_ERROR);
@@ -3247,7 +3247,7 @@
dev_warn(&h->pdev->dev, "cmd %p returned "
"unknown status %x\n", cmd,
cmd->err_info->CommandStatus);
- rq->errors = make_status_bytes(SAM_STAT_GOOD,
+ sreq->result = make_status_bytes(SAM_STAT_GOOD,
cmd->err_info->CommandStatus, DRIVER_OK,
blk_rq_is_passthrough(cmd->rq) ?
DID_PASSTHROUGH : DID_ERROR);
@@ -3380,9 +3380,9 @@
if (dma_mapping_error(&h->pdev->dev, temp64.val)) {
dev_warn(&h->pdev->dev,
"%s: error mapping page for DMA\n", __func__);
- creq->errors = make_status_bytes(SAM_STAT_GOOD,
- 0, DRIVER_OK,
- DID_SOFT_ERROR);
+ scsi_req(creq)->result =
+ make_status_bytes(SAM_STAT_GOOD, 0, DRIVER_OK,
+ DID_SOFT_ERROR);
cmd_free(h, c);
return;
}
@@ -3395,9 +3395,9 @@
if (cciss_map_sg_chain_block(h, c, h->cmd_sg_list[c->cmdindex],
(seg - (h->max_cmd_sgentries - 1)) *
sizeof(SGDescriptor_struct))) {
- creq->errors = make_status_bytes(SAM_STAT_GOOD,
- 0, DRIVER_OK,
- DID_SOFT_ERROR);
+ scsi_req(creq)->result =
+ make_status_bytes(SAM_STAT_GOOD, 0, DRIVER_OK,
+ DID_SOFT_ERROR);
cmd_free(h, c);
return;
}
diff --git a/drivers/block/drbd/drbd_debugfs.c b/drivers/block/drbd/drbd_debugfs.c
index de5c3ee..494837e 100644
--- a/drivers/block/drbd/drbd_debugfs.c
+++ b/drivers/block/drbd/drbd_debugfs.c
@@ -236,9 +236,6 @@
seq_print_rq_state_bit(m, f & EE_CALL_AL_COMPLETE_IO, &sep, "in-AL");
seq_print_rq_state_bit(m, f & EE_SEND_WRITE_ACK, &sep, "C");
seq_print_rq_state_bit(m, f & EE_MAY_SET_IN_SYNC, &sep, "set-in-sync");
-
- if (f & EE_IS_TRIM)
- __seq_print_rq_state_bit(m, f & EE_IS_TRIM_USE_ZEROOUT, &sep, "zero-out", "trim");
seq_print_rq_state_bit(m, f & EE_WRITE_SAME, &sep, "write-same");
seq_putc(m, '\n');
}
diff --git a/drivers/block/drbd/drbd_int.h b/drivers/block/drbd/drbd_int.h
index 724d1c5..d5da45b 100644
--- a/drivers/block/drbd/drbd_int.h
+++ b/drivers/block/drbd/drbd_int.h
@@ -437,9 +437,6 @@
/* is this a TRIM aka REQ_DISCARD? */
__EE_IS_TRIM,
- /* our lower level cannot handle trim,
- * and we want to fall back to zeroout instead */
- __EE_IS_TRIM_USE_ZEROOUT,
/* In case a barrier failed,
* we need to resubmit without the barrier flag. */
@@ -482,7 +479,6 @@
#define EE_CALL_AL_COMPLETE_IO (1<<__EE_CALL_AL_COMPLETE_IO)
#define EE_MAY_SET_IN_SYNC (1<<__EE_MAY_SET_IN_SYNC)
#define EE_IS_TRIM (1<<__EE_IS_TRIM)
-#define EE_IS_TRIM_USE_ZEROOUT (1<<__EE_IS_TRIM_USE_ZEROOUT)
#define EE_RESUBMITTED (1<<__EE_RESUBMITTED)
#define EE_WAS_ERROR (1<<__EE_WAS_ERROR)
#define EE_HAS_DIGEST (1<<__EE_HAS_DIGEST)
@@ -1561,8 +1557,6 @@
extern void drbd_endio_write_sec_final(struct drbd_peer_request *peer_req);
/* drbd_receiver.c */
-extern int drbd_issue_discard_or_zero_out(struct drbd_device *device,
- sector_t start, unsigned int nr_sectors, bool discard);
extern int drbd_receiver(struct drbd_thread *thi);
extern int drbd_ack_receiver(struct drbd_thread *thi);
extern void drbd_send_ping_wf(struct work_struct *ws);
diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c
index 92c60cb..84455c3 100644
--- a/drivers/block/drbd/drbd_main.c
+++ b/drivers/block/drbd/drbd_main.c
@@ -931,7 +931,6 @@
p->qlim->io_min = cpu_to_be32(queue_io_min(q));
p->qlim->io_opt = cpu_to_be32(queue_io_opt(q));
p->qlim->discard_enabled = blk_queue_discard(q);
- p->qlim->discard_zeroes_data = queue_discard_zeroes_data(q);
p->qlim->write_same_capable = !!q->limits.max_write_same_sectors;
} else {
q = device->rq_queue;
@@ -941,7 +940,6 @@
p->qlim->io_min = cpu_to_be32(queue_io_min(q));
p->qlim->io_opt = cpu_to_be32(queue_io_opt(q));
p->qlim->discard_enabled = 0;
- p->qlim->discard_zeroes_data = 0;
p->qlim->write_same_capable = 0;
}
}
@@ -1668,7 +1666,8 @@
(bio->bi_opf & REQ_FUA ? DP_FUA : 0) |
(bio->bi_opf & REQ_PREFLUSH ? DP_FLUSH : 0) |
(bio_op(bio) == REQ_OP_WRITE_SAME ? DP_WSAME : 0) |
- (bio_op(bio) == REQ_OP_DISCARD ? DP_DISCARD : 0);
+ (bio_op(bio) == REQ_OP_DISCARD ? DP_DISCARD : 0) |
+ (bio_op(bio) == REQ_OP_WRITE_ZEROES ? DP_DISCARD : 0);
else
return bio->bi_opf & REQ_SYNC ? DP_RW_SYNC : 0;
}
diff --git a/drivers/block/drbd/drbd_nl.c b/drivers/block/drbd/drbd_nl.c
index 908c704..02255a0 100644
--- a/drivers/block/drbd/drbd_nl.c
+++ b/drivers/block/drbd/drbd_nl.c
@@ -1199,10 +1199,6 @@
struct drbd_connection *connection = first_peer_device(device)->connection;
bool can_do = b ? blk_queue_discard(b) : true;
- if (can_do && b && !b->limits.discard_zeroes_data && !discard_zeroes_if_aligned) {
- can_do = false;
- drbd_info(device, "discard_zeroes_data=0 and discard_zeroes_if_aligned=no: disabling discards\n");
- }
if (can_do && connection->cstate >= C_CONNECTED && !(connection->agreed_features & DRBD_FF_TRIM)) {
can_do = false;
drbd_info(connection, "peer DRBD too old, does not support TRIM: disabling discards\n");
@@ -1217,10 +1213,12 @@
blk_queue_discard_granularity(q, 512);
q->limits.max_discard_sectors = drbd_max_discard_sectors(connection);
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, q);
+ q->limits.max_write_zeroes_sectors = drbd_max_discard_sectors(connection);
} else {
queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD, q);
blk_queue_discard_granularity(q, 0);
q->limits.max_discard_sectors = 0;
+ q->limits.max_write_zeroes_sectors = 0;
}
}
@@ -1482,8 +1480,7 @@
if (disk_conf->al_extents > drbd_al_extents_max(nbc))
disk_conf->al_extents = drbd_al_extents_max(nbc);
- if (!blk_queue_discard(q)
- || (!q->limits.discard_zeroes_data && !disk_conf->discard_zeroes_if_aligned)) {
+ if (!blk_queue_discard(q)) {
if (disk_conf->rs_discard_granularity) {
disk_conf->rs_discard_granularity = 0; /* disable feature */
drbd_info(device, "rs_discard_granularity feature disabled\n");
diff --git a/drivers/block/drbd/drbd_nla.c b/drivers/block/drbd/drbd_nla.c
index b2d4791..6bf806d 100644
--- a/drivers/block/drbd/drbd_nla.c
+++ b/drivers/block/drbd/drbd_nla.c
@@ -34,7 +34,7 @@
err = drbd_nla_check_mandatory(maxtype, nla);
if (!err)
- err = nla_parse_nested(tb, maxtype, nla, policy);
+ err = nla_parse_nested(tb, maxtype, nla, policy, NULL);
return err;
}
diff --git a/drivers/block/drbd/drbd_receiver.c b/drivers/block/drbd/drbd_receiver.c
index aa6bf96..1b0a2be 100644
--- a/drivers/block/drbd/drbd_receiver.c
+++ b/drivers/block/drbd/drbd_receiver.c
@@ -1448,105 +1448,14 @@
drbd_info(resource, "Method to ensure write ordering: %s\n", write_ordering_str[resource->write_ordering]);
}
-/*
- * We *may* ignore the discard-zeroes-data setting, if so configured.
- *
- * Assumption is that it "discard_zeroes_data=0" is only because the backend
- * may ignore partial unaligned discards.
- *
- * LVM/DM thin as of at least
- * LVM version: 2.02.115(2)-RHEL7 (2015-01-28)
- * Library version: 1.02.93-RHEL7 (2015-01-28)
- * Driver version: 4.29.0
- * still behaves this way.
- *
- * For unaligned (wrt. alignment and granularity) or too small discards,
- * we zero-out the initial (and/or) trailing unaligned partial chunks,
- * but discard all the aligned full chunks.
- *
- * At least for LVM/DM thin, the result is effectively "discard_zeroes_data=1".
- */
-int drbd_issue_discard_or_zero_out(struct drbd_device *device, sector_t start, unsigned int nr_sectors, bool discard)
-{
- struct block_device *bdev = device->ldev->backing_bdev;
- struct request_queue *q = bdev_get_queue(bdev);
- sector_t tmp, nr;
- unsigned int max_discard_sectors, granularity;
- int alignment;
- int err = 0;
-
- if (!discard)
- goto zero_out;
-
- /* Zero-sector (unknown) and one-sector granularities are the same. */
- granularity = max(q->limits.discard_granularity >> 9, 1U);
- alignment = (bdev_discard_alignment(bdev) >> 9) % granularity;
-
- max_discard_sectors = min(q->limits.max_discard_sectors, (1U << 22));
- max_discard_sectors -= max_discard_sectors % granularity;
- if (unlikely(!max_discard_sectors))
- goto zero_out;
-
- if (nr_sectors < granularity)
- goto zero_out;
-
- tmp = start;
- if (sector_div(tmp, granularity) != alignment) {
- if (nr_sectors < 2*granularity)
- goto zero_out;
- /* start + gran - (start + gran - align) % gran */
- tmp = start + granularity - alignment;
- tmp = start + granularity - sector_div(tmp, granularity);
-
- nr = tmp - start;
- err |= blkdev_issue_zeroout(bdev, start, nr, GFP_NOIO, 0);
- nr_sectors -= nr;
- start = tmp;
- }
- while (nr_sectors >= granularity) {
- nr = min_t(sector_t, nr_sectors, max_discard_sectors);
- err |= blkdev_issue_discard(bdev, start, nr, GFP_NOIO, 0);
- nr_sectors -= nr;
- start += nr;
- }
- zero_out:
- if (nr_sectors) {
- err |= blkdev_issue_zeroout(bdev, start, nr_sectors, GFP_NOIO, 0);
- }
- return err != 0;
-}
-
-static bool can_do_reliable_discards(struct drbd_device *device)
-{
- struct request_queue *q = bdev_get_queue(device->ldev->backing_bdev);
- struct disk_conf *dc;
- bool can_do;
-
- if (!blk_queue_discard(q))
- return false;
-
- if (q->limits.discard_zeroes_data)
- return true;
-
- rcu_read_lock();
- dc = rcu_dereference(device->ldev->disk_conf);
- can_do = dc->discard_zeroes_if_aligned;
- rcu_read_unlock();
- return can_do;
-}
-
static void drbd_issue_peer_discard(struct drbd_device *device, struct drbd_peer_request *peer_req)
{
- /* If the backend cannot discard, or does not guarantee
- * read-back zeroes in discarded ranges, we fall back to
- * zero-out. Unless configuration specifically requested
- * otherwise. */
- if (!can_do_reliable_discards(device))
- peer_req->flags |= EE_IS_TRIM_USE_ZEROOUT;
+ struct block_device *bdev = device->ldev->backing_bdev;
- if (drbd_issue_discard_or_zero_out(device, peer_req->i.sector,
- peer_req->i.size >> 9, !(peer_req->flags & EE_IS_TRIM_USE_ZEROOUT)))
+ if (blkdev_issue_zeroout(bdev, peer_req->i.sector, peer_req->i.size >> 9,
+ GFP_NOIO, 0))
peer_req->flags |= EE_WAS_ERROR;
+
drbd_endio_write_sec_final(peer_req);
}
@@ -2376,7 +2285,7 @@
static unsigned long wire_flags_to_bio_op(u32 dpf)
{
if (dpf & DP_DISCARD)
- return REQ_OP_DISCARD;
+ return REQ_OP_WRITE_ZEROES;
else
return REQ_OP_WRITE;
}
@@ -2567,7 +2476,7 @@
op_flags = wire_flags_to_bio_flags(dp_flags);
if (pi->cmd == P_TRIM) {
D_ASSERT(peer_device, peer_req->i.size > 0);
- D_ASSERT(peer_device, op == REQ_OP_DISCARD);
+ D_ASSERT(peer_device, op == REQ_OP_WRITE_ZEROES);
D_ASSERT(peer_device, peer_req->pages == NULL);
} else if (peer_req->pages == NULL) {
D_ASSERT(device, peer_req->i.size == 0);
@@ -4880,7 +4789,7 @@
if (get_ldev(device)) {
struct drbd_peer_request *peer_req;
- const int op = REQ_OP_DISCARD;
+ const int op = REQ_OP_WRITE_ZEROES;
peer_req = drbd_alloc_peer_req(peer_device, ID_SYNCER, sector,
size, 0, GFP_NOIO);
diff --git a/drivers/block/drbd/drbd_req.c b/drivers/block/drbd/drbd_req.c
index 652114a..b5730e1 100644
--- a/drivers/block/drbd/drbd_req.c
+++ b/drivers/block/drbd/drbd_req.c
@@ -59,6 +59,7 @@
drbd_req_make_private_bio(req, bio_src);
req->rq_state = (bio_data_dir(bio_src) == WRITE ? RQ_WRITE : 0)
| (bio_op(bio_src) == REQ_OP_WRITE_SAME ? RQ_WSAME : 0)
+ | (bio_op(bio_src) == REQ_OP_WRITE_ZEROES ? RQ_UNMAP : 0)
| (bio_op(bio_src) == REQ_OP_DISCARD ? RQ_UNMAP : 0);
req->device = device;
req->master_bio = bio_src;
@@ -1148,10 +1149,10 @@
static void drbd_process_discard_req(struct drbd_request *req)
{
- int err = drbd_issue_discard_or_zero_out(req->device,
- req->i.sector, req->i.size >> 9, true);
+ struct block_device *bdev = req->device->ldev->backing_bdev;
- if (err)
+ if (blkdev_issue_zeroout(bdev, req->i.sector, req->i.size >> 9,
+ GFP_NOIO, 0))
req->private_bio->bi_error = -EIO;
bio_endio(req->private_bio);
}
@@ -1180,7 +1181,8 @@
if (get_ldev(device)) {
if (drbd_insert_fault(device, type))
bio_io_error(bio);
- else if (bio_op(bio) == REQ_OP_DISCARD)
+ else if (bio_op(bio) == REQ_OP_WRITE_ZEROES ||
+ bio_op(bio) == REQ_OP_DISCARD)
drbd_process_discard_req(req);
else
generic_make_request(bio);
@@ -1234,7 +1236,8 @@
_drbd_start_io_acct(device, req);
/* process discards always from our submitter thread */
- if (bio_op(bio) & REQ_OP_DISCARD)
+ if ((bio_op(bio) & REQ_OP_WRITE_ZEROES) ||
+ (bio_op(bio) & REQ_OP_DISCARD))
goto queue_for_submitter_thread;
if (rw == WRITE && req->private_bio && req->i.size
diff --git a/drivers/block/drbd/drbd_worker.c b/drivers/block/drbd/drbd_worker.c
index 3bff33f..1afcb4e 100644
--- a/drivers/block/drbd/drbd_worker.c
+++ b/drivers/block/drbd/drbd_worker.c
@@ -174,7 +174,8 @@
struct drbd_peer_request *peer_req = bio->bi_private;
struct drbd_device *device = peer_req->peer_device->device;
bool is_write = bio_data_dir(bio) == WRITE;
- bool is_discard = !!(bio_op(bio) == REQ_OP_DISCARD);
+ bool is_discard = bio_op(bio) == REQ_OP_WRITE_ZEROES ||
+ bio_op(bio) == REQ_OP_DISCARD;
if (bio->bi_error && __ratelimit(&drbd_ratelimit_state))
drbd_warn(device, "%s: error=%d s=%llus\n",
@@ -249,6 +250,7 @@
/* to avoid recursion in __req_mod */
if (unlikely(bio->bi_error)) {
switch (bio_op(bio)) {
+ case REQ_OP_WRITE_ZEROES:
case REQ_OP_DISCARD:
if (bio->bi_error == -EOPNOTSUPP)
what = DISCARD_COMPLETED_NOTSUPP;
diff --git a/drivers/block/floppy.c b/drivers/block/floppy.c
index 45b4384..60d4c76 100644
--- a/drivers/block/floppy.c
+++ b/drivers/block/floppy.c
@@ -2805,8 +2805,10 @@
fdc_queue = 0;
if (q) {
current_req = blk_fetch_request(q);
- if (current_req)
+ if (current_req) {
+ current_req->error_count = 0;
break;
+ }
}
} while (fdc_queue != old_pos);
@@ -2866,7 +2868,7 @@
_floppy = floppy_type + DP->autodetect[DRS->probed_format];
} else
probing = 0;
- errors = &(current_req->errors);
+ errors = &(current_req->error_count);
tmp = make_raw_rw_request();
if (tmp < 2) {
request_done(tmp);
@@ -4207,9 +4209,7 @@
disks[drive]->fops = &floppy_fops;
sprintf(disks[drive]->disk_name, "fd%d", drive);
- init_timer(&motor_off_timer[drive]);
- motor_off_timer[drive].data = drive;
- motor_off_timer[drive].function = motor_off_callback;
+ setup_timer(&motor_off_timer[drive], motor_off_callback, drive);
}
err = register_blkdev(FLOPPY_MAJOR, "fd");
diff --git a/drivers/block/hd.c b/drivers/block/hd.c
deleted file mode 100644
index 6043648..0000000
--- a/drivers/block/hd.c
+++ /dev/null
@@ -1,803 +0,0 @@
-/*
- * Copyright (C) 1991, 1992 Linus Torvalds
- *
- * This is the low-level hd interrupt support. It traverses the
- * request-list, using interrupts to jump between functions. As
- * all the functions are called within interrupts, we may not
- * sleep. Special care is recommended.
- *
- * modified by Drew Eckhardt to check nr of hd's from the CMOS.
- *
- * Thanks to Branko Lankester, lankeste@fwi.uva.nl, who found a bug
- * in the early extended-partition checks and added DM partitions
- *
- * IRQ-unmask, drive-id, multiple-mode, support for ">16 heads",
- * and general streamlining by Mark Lord.
- *
- * Removed 99% of above. Use Mark's ide driver for those options.
- * This is now a lightweight ST-506 driver. (Paul Gortmaker)
- *
- * Modified 1995 Russell King for ARM processor.
- *
- * Bugfix: max_sectors must be <= 255 or the wheels tend to come
- * off in a hurry once you queue things up - Paul G. 02/2001
- */
-
-/* Uncomment the following if you want verbose error reports. */
-/* #define VERBOSE_ERRORS */
-
-#include <linux/blkdev.h>
-#include <linux/errno.h>
-#include <linux/signal.h>
-#include <linux/interrupt.h>
-#include <linux/timer.h>
-#include <linux/fs.h>
-#include <linux/kernel.h>
-#include <linux/genhd.h>
-#include <linux/string.h>
-#include <linux/ioport.h>
-#include <linux/init.h>
-#include <linux/blkpg.h>
-#include <linux/ata.h>
-#include <linux/hdreg.h>
-
-#define HD_IRQ 14
-
-#define REALLY_SLOW_IO
-#include <asm/io.h>
-#include <linux/uaccess.h>
-
-#ifdef __arm__
-#undef HD_IRQ
-#endif
-#include <asm/irq.h>
-#ifdef __arm__
-#define HD_IRQ IRQ_HARDDISK
-#endif
-
-/* Hd controller regster ports */
-
-#define HD_DATA 0x1f0 /* _CTL when writing */
-#define HD_ERROR 0x1f1 /* see err-bits */
-#define HD_NSECTOR 0x1f2 /* nr of sectors to read/write */
-#define HD_SECTOR 0x1f3 /* starting sector */
-#define HD_LCYL 0x1f4 /* starting cylinder */
-#define HD_HCYL 0x1f5 /* high byte of starting cyl */
-#define HD_CURRENT 0x1f6 /* 101dhhhh , d=drive, hhhh=head */
-#define HD_STATUS 0x1f7 /* see status-bits */
-#define HD_FEATURE HD_ERROR /* same io address, read=error, write=feature */
-#define HD_PRECOMP HD_FEATURE /* obsolete use of this port - predates IDE */
-#define HD_COMMAND HD_STATUS /* same io address, read=status, write=cmd */
-
-#define HD_CMD 0x3f6 /* used for resets */
-#define HD_ALTSTATUS 0x3f6 /* same as HD_STATUS but doesn't clear irq */
-
-/* Bits of HD_STATUS */
-#define ERR_STAT 0x01
-#define INDEX_STAT 0x02
-#define ECC_STAT 0x04 /* Corrected error */
-#define DRQ_STAT 0x08
-#define SEEK_STAT 0x10
-#define SERVICE_STAT SEEK_STAT
-#define WRERR_STAT 0x20
-#define READY_STAT 0x40
-#define BUSY_STAT 0x80
-
-/* Bits for HD_ERROR */
-#define MARK_ERR 0x01 /* Bad address mark */
-#define TRK0_ERR 0x02 /* couldn't find track 0 */
-#define ABRT_ERR 0x04 /* Command aborted */
-#define MCR_ERR 0x08 /* media change request */
-#define ID_ERR 0x10 /* ID field not found */
-#define MC_ERR 0x20 /* media changed */
-#define ECC_ERR 0x40 /* Uncorrectable ECC error */
-#define BBD_ERR 0x80 /* pre-EIDE meaning: block marked bad */
-#define ICRC_ERR 0x80 /* new meaning: CRC error during transfer */
-
-static DEFINE_SPINLOCK(hd_lock);
-static struct request_queue *hd_queue;
-static struct request *hd_req;
-
-#define TIMEOUT_VALUE (6*HZ)
-#define HD_DELAY 0
-
-#define MAX_ERRORS 16 /* Max read/write errors/sector */
-#define RESET_FREQ 8 /* Reset controller every 8th retry */
-#define RECAL_FREQ 4 /* Recalibrate every 4th retry */
-#define MAX_HD 2
-
-#define STAT_OK (READY_STAT|SEEK_STAT)
-#define OK_STATUS(s) (((s)&(STAT_OK|(BUSY_STAT|WRERR_STAT|ERR_STAT)))==STAT_OK)
-
-static void recal_intr(void);
-static void bad_rw_intr(void);
-
-static int reset;
-static int hd_error;
-
-/*
- * This struct defines the HD's and their types.
- */
-struct hd_i_struct {
- unsigned int head, sect, cyl, wpcom, lzone, ctl;
- int unit;
- int recalibrate;
- int special_op;
-};
-
-#ifdef HD_TYPE
-static struct hd_i_struct hd_info[] = { HD_TYPE };
-static int NR_HD = ARRAY_SIZE(hd_info);
-#else
-static struct hd_i_struct hd_info[MAX_HD];
-static int NR_HD;
-#endif
-
-static struct gendisk *hd_gendisk[MAX_HD];
-
-static struct timer_list device_timer;
-
-#define TIMEOUT_VALUE (6*HZ)
-
-#define SET_TIMER \
- do { \
- mod_timer(&device_timer, jiffies + TIMEOUT_VALUE); \
- } while (0)
-
-static void (*do_hd)(void) = NULL;
-#define SET_HANDLER(x) \
-if ((do_hd = (x)) != NULL) \
- SET_TIMER; \
-else \
- del_timer(&device_timer);
-
-
-#if (HD_DELAY > 0)
-
-#include <linux/i8253.h>
-
-unsigned long last_req;
-
-unsigned long read_timer(void)
-{
- unsigned long t, flags;
- int i;
-
- raw_spin_lock_irqsave(&i8253_lock, flags);
- t = jiffies * 11932;
- outb_p(0, 0x43);
- i = inb_p(0x40);
- i |= inb(0x40) << 8;
- raw_spin_unlock_irqrestore(&i8253_lock, flags);
- return(t - i);
-}
-#endif
-
-static void __init hd_setup(char *str, int *ints)
-{
- int hdind = 0;
-
- if (ints[0] != 3)
- return;
- if (hd_info[0].head != 0)
- hdind = 1;
- hd_info[hdind].head = ints[2];
- hd_info[hdind].sect = ints[3];
- hd_info[hdind].cyl = ints[1];
- hd_info[hdind].wpcom = 0;
- hd_info[hdind].lzone = ints[1];
- hd_info[hdind].ctl = (ints[2] > 8 ? 8 : 0);
- NR_HD = hdind+1;
-}
-
-static bool hd_end_request(int err, unsigned int bytes)
-{
- if (__blk_end_request(hd_req, err, bytes))
- return true;
- hd_req = NULL;
- return false;
-}
-
-static bool hd_end_request_cur(int err)
-{
- return hd_end_request(err, blk_rq_cur_bytes(hd_req));
-}
-
-static void dump_status(const char *msg, unsigned int stat)
-{
- char *name = "hd?";
- if (hd_req)
- name = hd_req->rq_disk->disk_name;
-
-#ifdef VERBOSE_ERRORS
- printk("%s: %s: status=0x%02x { ", name, msg, stat & 0xff);
- if (stat & BUSY_STAT) printk("Busy ");
- if (stat & READY_STAT) printk("DriveReady ");
- if (stat & WRERR_STAT) printk("WriteFault ");
- if (stat & SEEK_STAT) printk("SeekComplete ");
- if (stat & DRQ_STAT) printk("DataRequest ");
- if (stat & ECC_STAT) printk("CorrectedError ");
- if (stat & INDEX_STAT) printk("Index ");
- if (stat & ERR_STAT) printk("Error ");
- printk("}\n");
- if ((stat & ERR_STAT) == 0) {
- hd_error = 0;
- } else {
- hd_error = inb(HD_ERROR);
- printk("%s: %s: error=0x%02x { ", name, msg, hd_error & 0xff);
- if (hd_error & BBD_ERR) printk("BadSector ");
- if (hd_error & ECC_ERR) printk("UncorrectableError ");
- if (hd_error & ID_ERR) printk("SectorIdNotFound ");
- if (hd_error & ABRT_ERR) printk("DriveStatusError ");
- if (hd_error & TRK0_ERR) printk("TrackZeroNotFound ");
- if (hd_error & MARK_ERR) printk("AddrMarkNotFound ");
- printk("}");
- if (hd_error & (BBD_ERR|ECC_ERR|ID_ERR|MARK_ERR)) {
- printk(", CHS=%d/%d/%d", (inb(HD_HCYL)<<8) + inb(HD_LCYL),
- inb(HD_CURRENT) & 0xf, inb(HD_SECTOR));
- if (hd_req)
- printk(", sector=%ld", blk_rq_pos(hd_req));
- }
- printk("\n");
- }
-#else
- printk("%s: %s: status=0x%02x.\n", name, msg, stat & 0xff);
- if ((stat & ERR_STAT) == 0) {
- hd_error = 0;
- } else {
- hd_error = inb(HD_ERROR);
- printk("%s: %s: error=0x%02x.\n", name, msg, hd_error & 0xff);
- }
-#endif
-}
-
-static void check_status(void)
-{
- int i = inb_p(HD_STATUS);
-
- if (!OK_STATUS(i)) {
- dump_status("check_status", i);
- bad_rw_intr();
- }
-}
-
-static int controller_busy(void)
-{
- int retries = 100000;
- unsigned char status;
-
- do {
- status = inb_p(HD_STATUS);
- } while ((status & BUSY_STAT) && --retries);
- return status;
-}
-
-static int status_ok(void)
-{
- unsigned char status = inb_p(HD_STATUS);
-
- if (status & BUSY_STAT)
- return 1; /* Ancient, but does it make sense??? */
- if (status & WRERR_STAT)
- return 0;
- if (!(status & READY_STAT))
- return 0;
- if (!(status & SEEK_STAT))
- return 0;
- return 1;
-}
-
-static int controller_ready(unsigned int drive, unsigned int head)
-{
- int retry = 100;
-
- do {
- if (controller_busy() & BUSY_STAT)
- return 0;
- outb_p(0xA0 | (drive<<4) | head, HD_CURRENT);
- if (status_ok())
- return 1;
- } while (--retry);
- return 0;
-}
-
-static void hd_out(struct hd_i_struct *disk,
- unsigned int nsect,
- unsigned int sect,
- unsigned int head,
- unsigned int cyl,
- unsigned int cmd,
- void (*intr_addr)(void))
-{
- unsigned short port;
-
-#if (HD_DELAY > 0)
- while (read_timer() - last_req < HD_DELAY)
- /* nothing */;
-#endif
- if (reset)
- return;
- if (!controller_ready(disk->unit, head)) {
- reset = 1;
- return;
- }
- SET_HANDLER(intr_addr);
- outb_p(disk->ctl, HD_CMD);
- port = HD_DATA;
- outb_p(disk->wpcom >> 2, ++port);
- outb_p(nsect, ++port);
- outb_p(sect, ++port);
- outb_p(cyl, ++port);
- outb_p(cyl >> 8, ++port);
- outb_p(0xA0 | (disk->unit << 4) | head, ++port);
- outb_p(cmd, ++port);
-}
-
-static void hd_request (void);
-
-static int drive_busy(void)
-{
- unsigned int i;
- unsigned char c;
-
- for (i = 0; i < 500000 ; i++) {
- c = inb_p(HD_STATUS);
- if ((c & (BUSY_STAT | READY_STAT | SEEK_STAT)) == STAT_OK)
- return 0;
- }
- dump_status("reset timed out", c);
- return 1;
-}
-
-static void reset_controller(void)
-{
- int i;
-
- outb_p(4, HD_CMD);
- for (i = 0; i < 1000; i++) barrier();
- outb_p(hd_info[0].ctl & 0x0f, HD_CMD);
- for (i = 0; i < 1000; i++) barrier();
- if (drive_busy())
- printk("hd: controller still busy\n");
- else if ((hd_error = inb(HD_ERROR)) != 1)
- printk("hd: controller reset failed: %02x\n", hd_error);
-}
-
-static void reset_hd(void)
-{
- static int i;
-
-repeat:
- if (reset) {
- reset = 0;
- i = -1;
- reset_controller();
- } else {
- check_status();
- if (reset)
- goto repeat;
- }
- if (++i < NR_HD) {
- struct hd_i_struct *disk = &hd_info[i];
- disk->special_op = disk->recalibrate = 1;
- hd_out(disk, disk->sect, disk->sect, disk->head-1,
- disk->cyl, ATA_CMD_INIT_DEV_PARAMS, &reset_hd);
- if (reset)
- goto repeat;
- } else
- hd_request();
-}
-
-/*
- * Ok, don't know what to do with the unexpected interrupts: on some machines
- * doing a reset and a retry seems to result in an eternal loop. Right now I
- * ignore it, and just set the timeout.
- *
- * On laptops (and "green" PCs), an unexpected interrupt occurs whenever the
- * drive enters "idle", "standby", or "sleep" mode, so if the status looks
- * "good", we just ignore the interrupt completely.
- */
-static void unexpected_hd_interrupt(void)
-{
- unsigned int stat = inb_p(HD_STATUS);
-
- if (stat & (BUSY_STAT|DRQ_STAT|ECC_STAT|ERR_STAT)) {
- dump_status("unexpected interrupt", stat);
- SET_TIMER;
- }
-}
-
-/*
- * bad_rw_intr() now tries to be a bit smarter and does things
- * according to the error returned by the controller.
- * -Mika Liljeberg (liljeber@cs.Helsinki.FI)
- */
-static void bad_rw_intr(void)
-{
- struct request *req = hd_req;
-
- if (req != NULL) {
- struct hd_i_struct *disk = req->rq_disk->private_data;
- if (++req->errors >= MAX_ERRORS || (hd_error & BBD_ERR)) {
- hd_end_request_cur(-EIO);
- disk->special_op = disk->recalibrate = 1;
- } else if (req->errors % RESET_FREQ == 0)
- reset = 1;
- else if ((hd_error & TRK0_ERR) || req->errors % RECAL_FREQ == 0)
- disk->special_op = disk->recalibrate = 1;
- /* Otherwise just retry */
- }
-}
-
-static inline int wait_DRQ(void)
-{
- int retries;
- int stat;
-
- for (retries = 0; retries < 100000; retries++) {
- stat = inb_p(HD_STATUS);
- if (stat & DRQ_STAT)
- return 0;
- }
- dump_status("wait_DRQ", stat);
- return -1;
-}
-
-static void read_intr(void)
-{
- struct request *req;
- int i, retries = 100000;
-
- do {
- i = (unsigned) inb_p(HD_STATUS);
- if (i & BUSY_STAT)
- continue;
- if (!OK_STATUS(i))
- break;
- if (i & DRQ_STAT)
- goto ok_to_read;
- } while (--retries > 0);
- dump_status("read_intr", i);
- bad_rw_intr();
- hd_request();
- return;
-
-ok_to_read:
- req = hd_req;
- insw(HD_DATA, bio_data(req->bio), 256);
-#ifdef DEBUG
- printk("%s: read: sector %ld, remaining = %u, buffer=%p\n",
- req->rq_disk->disk_name, blk_rq_pos(req) + 1,
- blk_rq_sectors(req) - 1, bio_data(req->bio)+512);
-#endif
- if (hd_end_request(0, 512)) {
- SET_HANDLER(&read_intr);
- return;
- }
-
- (void) inb_p(HD_STATUS);
-#if (HD_DELAY > 0)
- last_req = read_timer();
-#endif
- hd_request();
-}
-
-static void write_intr(void)
-{
- struct request *req = hd_req;
- int i;
- int retries = 100000;
-
- do {
- i = (unsigned) inb_p(HD_STATUS);
- if (i & BUSY_STAT)
- continue;
- if (!OK_STATUS(i))
- break;
- if ((blk_rq_sectors(req) <= 1) || (i & DRQ_STAT))
- goto ok_to_write;
- } while (--retries > 0);
- dump_status("write_intr", i);
- bad_rw_intr();
- hd_request();
- return;
-
-ok_to_write:
- if (hd_end_request(0, 512)) {
- SET_HANDLER(&write_intr);
- outsw(HD_DATA, bio_data(req->bio), 256);
- return;
- }
-
-#if (HD_DELAY > 0)
- last_req = read_timer();
-#endif
- hd_request();
-}
-
-static void recal_intr(void)
-{
- check_status();
-#if (HD_DELAY > 0)
- last_req = read_timer();
-#endif
- hd_request();
-}
-
-/*
- * This is another of the error-routines I don't know what to do with. The
- * best idea seems to just set reset, and start all over again.
- */
-static void hd_times_out(unsigned long dummy)
-{
- char *name;
-
- do_hd = NULL;
-
- if (!hd_req)
- return;
-
- spin_lock_irq(hd_queue->queue_lock);
- reset = 1;
- name = hd_req->rq_disk->disk_name;
- printk("%s: timeout\n", name);
- if (++hd_req->errors >= MAX_ERRORS) {
-#ifdef DEBUG
- printk("%s: too many errors\n", name);
-#endif
- hd_end_request_cur(-EIO);
- }
- hd_request();
- spin_unlock_irq(hd_queue->queue_lock);
-}
-
-static int do_special_op(struct hd_i_struct *disk, struct request *req)
-{
- if (disk->recalibrate) {
- disk->recalibrate = 0;
- hd_out(disk, disk->sect, 0, 0, 0, ATA_CMD_RESTORE, &recal_intr);
- return reset;
- }
- if (disk->head > 16) {
- printk("%s: cannot handle device with more than 16 heads - giving up\n", req->rq_disk->disk_name);
- hd_end_request_cur(-EIO);
- }
- disk->special_op = 0;
- return 1;
-}
-
-/*
- * The driver enables interrupts as much as possible. In order to do this,
- * (a) the device-interrupt is disabled before entering hd_request(),
- * and (b) the timeout-interrupt is disabled before the sti().
- *
- * Interrupts are still masked (by default) whenever we are exchanging
- * data/cmds with a drive, because some drives seem to have very poor
- * tolerance for latency during I/O. The IDE driver has support to unmask
- * interrupts for non-broken hardware, so use that driver if required.
- */
-static void hd_request(void)
-{
- unsigned int block, nsect, sec, track, head, cyl;
- struct hd_i_struct *disk;
- struct request *req;
-
- if (do_hd)
- return;
-repeat:
- del_timer(&device_timer);
-
- if (!hd_req) {
- hd_req = blk_fetch_request(hd_queue);
- if (!hd_req) {
- do_hd = NULL;
- return;
- }
- }
- req = hd_req;
-
- if (reset) {
- reset_hd();
- return;
- }
- disk = req->rq_disk->private_data;
- block = blk_rq_pos(req);
- nsect = blk_rq_sectors(req);
- if (block >= get_capacity(req->rq_disk) ||
- ((block+nsect) > get_capacity(req->rq_disk))) {
- printk("%s: bad access: block=%d, count=%d\n",
- req->rq_disk->disk_name, block, nsect);
- hd_end_request_cur(-EIO);
- goto repeat;
- }
-
- if (disk->special_op) {
- if (do_special_op(disk, req))
- goto repeat;
- return;
- }
- sec = block % disk->sect + 1;
- track = block / disk->sect;
- head = track % disk->head;
- cyl = track / disk->head;
-#ifdef DEBUG
- printk("%s: %sing: CHS=%d/%d/%d, sectors=%d, buffer=%p\n",
- req->rq_disk->disk_name,
- req_data_dir(req) == READ ? "read" : "writ",
- cyl, head, sec, nsect, bio_data(req->bio));
-#endif
-
- switch (req_op(req)) {
- case REQ_OP_READ:
- hd_out(disk, nsect, sec, head, cyl, ATA_CMD_PIO_READ,
- &read_intr);
- if (reset)
- goto repeat;
- break;
- case REQ_OP_WRITE:
- hd_out(disk, nsect, sec, head, cyl, ATA_CMD_PIO_WRITE,
- &write_intr);
- if (reset)
- goto repeat;
- if (wait_DRQ()) {
- bad_rw_intr();
- goto repeat;
- }
- outsw(HD_DATA, bio_data(req->bio), 256);
- break;
- default:
- printk("unknown hd-command\n");
- hd_end_request_cur(-EIO);
- break;
- }
-}
-
-static void do_hd_request(struct request_queue *q)
-{
- hd_request();
-}
-
-static int hd_getgeo(struct block_device *bdev, struct hd_geometry *geo)
-{
- struct hd_i_struct *disk = bdev->bd_disk->private_data;
-
- geo->heads = disk->head;
- geo->sectors = disk->sect;
- geo->cylinders = disk->cyl;
- return 0;
-}
-
-/*
- * Releasing a block device means we sync() it, so that it can safely
- * be forgotten about...
- */
-
-static irqreturn_t hd_interrupt(int irq, void *dev_id)
-{
- void (*handler)(void) = do_hd;
-
- spin_lock(hd_queue->queue_lock);
-
- do_hd = NULL;
- del_timer(&device_timer);
- if (!handler)
- handler = unexpected_hd_interrupt;
- handler();
-
- spin_unlock(hd_queue->queue_lock);
-
- return IRQ_HANDLED;
-}
-
-static const struct block_device_operations hd_fops = {
- .getgeo = hd_getgeo,
-};
-
-static int __init hd_init(void)
-{
- int drive;
-
- if (register_blkdev(HD_MAJOR, "hd"))
- return -1;
-
- hd_queue = blk_init_queue(do_hd_request, &hd_lock);
- if (!hd_queue) {
- unregister_blkdev(HD_MAJOR, "hd");
- return -ENOMEM;
- }
-
- blk_queue_max_hw_sectors(hd_queue, 255);
- init_timer(&device_timer);
- device_timer.function = hd_times_out;
- blk_queue_logical_block_size(hd_queue, 512);
-
- if (!NR_HD) {
- /*
- * We don't know anything about the drive. This means
- * that you *MUST* specify the drive parameters to the
- * kernel yourself.
- *
- * If we were on an i386, we used to read this info from
- * the BIOS or CMOS. This doesn't work all that well,
- * since this assumes that this is a primary or secondary
- * drive, and if we're using this legacy driver, it's
- * probably an auxiliary controller added to recover
- * legacy data off an ST-506 drive. Either way, it's
- * definitely safest to have the user explicitly specify
- * the information.
- */
- printk("hd: no drives specified - use hd=cyl,head,sectors"
- " on kernel command line\n");
- goto out;
- }
-
- for (drive = 0 ; drive < NR_HD ; drive++) {
- struct gendisk *disk = alloc_disk(64);
- struct hd_i_struct *p = &hd_info[drive];
- if (!disk)
- goto Enomem;
- disk->major = HD_MAJOR;
- disk->first_minor = drive << 6;
- disk->fops = &hd_fops;
- sprintf(disk->disk_name, "hd%c", 'a'+drive);
- disk->private_data = p;
- set_capacity(disk, p->head * p->sect * p->cyl);
- disk->queue = hd_queue;
- p->unit = drive;
- hd_gendisk[drive] = disk;
- printk("%s: %luMB, CHS=%d/%d/%d\n",
- disk->disk_name, (unsigned long)get_capacity(disk)/2048,
- p->cyl, p->head, p->sect);
- }
-
- if (request_irq(HD_IRQ, hd_interrupt, 0, "hd", NULL)) {
- printk("hd: unable to get IRQ%d for the hard disk driver\n",
- HD_IRQ);
- goto out1;
- }
- if (!request_region(HD_DATA, 8, "hd")) {
- printk(KERN_WARNING "hd: port 0x%x busy\n", HD_DATA);
- goto out2;
- }
- if (!request_region(HD_CMD, 1, "hd(cmd)")) {
- printk(KERN_WARNING "hd: port 0x%x busy\n", HD_CMD);
- goto out3;
- }
-
- /* Let them fly */
- for (drive = 0; drive < NR_HD; drive++)
- add_disk(hd_gendisk[drive]);
-
- return 0;
-
-out3:
- release_region(HD_DATA, 8);
-out2:
- free_irq(HD_IRQ, NULL);
-out1:
- for (drive = 0; drive < NR_HD; drive++)
- put_disk(hd_gendisk[drive]);
- NR_HD = 0;
-out:
- del_timer(&device_timer);
- unregister_blkdev(HD_MAJOR, "hd");
- blk_cleanup_queue(hd_queue);
- return -1;
-Enomem:
- while (drive--)
- put_disk(hd_gendisk[drive]);
- goto out;
-}
-
-static int __init parse_hd_setup(char *line)
-{
- int ints[6];
-
- (void) get_options(line, ARRAY_SIZE(ints), ints);
- hd_setup(NULL, ints);
-
- return 1;
-}
-__setup("hd=", parse_hd_setup);
-
-late_initcall(hd_init);
diff --git a/drivers/block/loop.c b/drivers/block/loop.c
index 0ecb646..994403e 100644
--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -445,32 +445,27 @@
return ret;
}
-static inline void handle_partial_read(struct loop_cmd *cmd, long bytes)
+static void lo_complete_rq(struct request *rq)
{
- if (bytes < 0 || op_is_write(req_op(cmd->rq)))
- return;
+ struct loop_cmd *cmd = blk_mq_rq_to_pdu(rq);
- if (unlikely(bytes < blk_rq_bytes(cmd->rq))) {
+ if (unlikely(req_op(cmd->rq) == REQ_OP_READ && cmd->use_aio &&
+ cmd->ret >= 0 && cmd->ret < blk_rq_bytes(cmd->rq))) {
struct bio *bio = cmd->rq->bio;
- bio_advance(bio, bytes);
+ bio_advance(bio, cmd->ret);
zero_fill_bio(bio);
}
+
+ blk_mq_end_request(rq, cmd->ret < 0 ? -EIO : 0);
}
static void lo_rw_aio_complete(struct kiocb *iocb, long ret, long ret2)
{
struct loop_cmd *cmd = container_of(iocb, struct loop_cmd, iocb);
- struct request *rq = cmd->rq;
- handle_partial_read(cmd, ret);
-
- if (ret > 0)
- ret = 0;
- else if (ret < 0)
- ret = -EIO;
-
- blk_mq_complete_request(rq, ret);
+ cmd->ret = ret;
+ blk_mq_complete_request(cmd->rq);
}
static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd,
@@ -528,6 +523,7 @@
case REQ_OP_FLUSH:
return lo_req_flush(lo, rq);
case REQ_OP_DISCARD:
+ case REQ_OP_WRITE_ZEROES:
return lo_discard(lo, rq, pos);
case REQ_OP_WRITE:
if (lo->transfer)
@@ -826,7 +822,7 @@
q->limits.discard_granularity = 0;
q->limits.discard_alignment = 0;
blk_queue_max_discard_sectors(q, 0);
- q->limits.discard_zeroes_data = 0;
+ blk_queue_max_write_zeroes_sectors(q, 0);
queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD, q);
return;
}
@@ -834,7 +830,7 @@
q->limits.discard_granularity = inode->i_sb->s_blocksize;
q->limits.discard_alignment = 0;
blk_queue_max_discard_sectors(q, UINT_MAX >> 9);
- q->limits.discard_zeroes_data = 1;
+ blk_queue_max_write_zeroes_sectors(q, UINT_MAX >> 9);
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, q);
}
@@ -1660,6 +1656,7 @@
switch (req_op(cmd->rq)) {
case REQ_OP_FLUSH:
case REQ_OP_DISCARD:
+ case REQ_OP_WRITE_ZEROES:
cmd->use_aio = false;
break;
default:
@@ -1686,8 +1683,10 @@
ret = do_req_filebacked(lo, cmd->rq);
failed:
/* complete non-aio request */
- if (!cmd->use_aio || ret)
- blk_mq_complete_request(cmd->rq, ret ? -EIO : 0);
+ if (!cmd->use_aio || ret) {
+ cmd->ret = ret ? -EIO : 0;
+ blk_mq_complete_request(cmd->rq);
+ }
}
static void loop_queue_work(struct kthread_work *work)
@@ -1710,9 +1709,10 @@
return 0;
}
-static struct blk_mq_ops loop_mq_ops = {
+static const struct blk_mq_ops loop_mq_ops = {
.queue_rq = loop_queue_rq,
.init_request = loop_init_request,
+ .complete = lo_complete_rq,
};
static int loop_add(struct loop_device **l, int i)
diff --git a/drivers/block/loop.h b/drivers/block/loop.h
index fb2237c..fecd3f9 100644
--- a/drivers/block/loop.h
+++ b/drivers/block/loop.h
@@ -70,6 +70,7 @@
struct request *rq;
struct list_head list;
bool use_aio; /* use AIO interface to handle I/O */
+ long ret;
struct kiocb iocb;
};
diff --git a/drivers/block/mg_disk.c b/drivers/block/mg_disk.c
deleted file mode 100644
index 286f276..0000000
--- a/drivers/block/mg_disk.c
+++ /dev/null
@@ -1,1112 +0,0 @@
-/*
- * drivers/block/mg_disk.c
- *
- * Support for the mGine m[g]flash IO mode.
- * Based on legacy hd.c
- *
- * (c) 2008 mGine Co.,LTD
- * (c) 2008 unsik Kim <donari75@gmail.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/fs.h>
-#include <linux/blkdev.h>
-#include <linux/hdreg.h>
-#include <linux/ata.h>
-#include <linux/interrupt.h>
-#include <linux/delay.h>
-#include <linux/platform_device.h>
-#include <linux/gpio.h>
-#include <linux/mg_disk.h>
-#include <linux/slab.h>
-
-#define MG_RES_SEC (CONFIG_MG_DISK_RES << 1)
-
-/* name for block device */
-#define MG_DISK_NAME "mgd"
-
-#define MG_DISK_MAJ 0
-#define MG_DISK_MAX_PART 16
-#define MG_SECTOR_SIZE 512
-#define MG_MAX_SECTS 256
-
-/* Register offsets */
-#define MG_BUFF_OFFSET 0x8000
-#define MG_REG_OFFSET 0xC000
-#define MG_REG_FEATURE (MG_REG_OFFSET + 2) /* write case */
-#define MG_REG_ERROR (MG_REG_OFFSET + 2) /* read case */
-#define MG_REG_SECT_CNT (MG_REG_OFFSET + 4)
-#define MG_REG_SECT_NUM (MG_REG_OFFSET + 6)
-#define MG_REG_CYL_LOW (MG_REG_OFFSET + 8)
-#define MG_REG_CYL_HIGH (MG_REG_OFFSET + 0xA)
-#define MG_REG_DRV_HEAD (MG_REG_OFFSET + 0xC)
-#define MG_REG_COMMAND (MG_REG_OFFSET + 0xE) /* write case */
-#define MG_REG_STATUS (MG_REG_OFFSET + 0xE) /* read case */
-#define MG_REG_DRV_CTRL (MG_REG_OFFSET + 0x10)
-#define MG_REG_BURST_CTRL (MG_REG_OFFSET + 0x12)
-
-/* handy status */
-#define MG_STAT_READY (ATA_DRDY | ATA_DSC)
-#define MG_READY_OK(s) (((s) & (MG_STAT_READY | (ATA_BUSY | ATA_DF | \
- ATA_ERR))) == MG_STAT_READY)
-
-/* error code for others */
-#define MG_ERR_NONE 0
-#define MG_ERR_TIMEOUT 0x100
-#define MG_ERR_INIT_STAT 0x101
-#define MG_ERR_TRANSLATION 0x102
-#define MG_ERR_CTRL_RST 0x103
-#define MG_ERR_INV_STAT 0x104
-#define MG_ERR_RSTOUT 0x105
-
-#define MG_MAX_ERRORS 6 /* Max read/write errors */
-
-/* command */
-#define MG_CMD_RD 0x20
-#define MG_CMD_WR 0x30
-#define MG_CMD_SLEEP 0x99
-#define MG_CMD_WAKEUP 0xC3
-#define MG_CMD_ID 0xEC
-#define MG_CMD_WR_CONF 0x3C
-#define MG_CMD_RD_CONF 0x40
-
-/* operation mode */
-#define MG_OP_CASCADE (1 << 0)
-#define MG_OP_CASCADE_SYNC_RD (1 << 1)
-#define MG_OP_CASCADE_SYNC_WR (1 << 2)
-#define MG_OP_INTERLEAVE (1 << 3)
-
-/* synchronous */
-#define MG_BURST_LAT_4 (3 << 4)
-#define MG_BURST_LAT_5 (4 << 4)
-#define MG_BURST_LAT_6 (5 << 4)
-#define MG_BURST_LAT_7 (6 << 4)
-#define MG_BURST_LAT_8 (7 << 4)
-#define MG_BURST_LEN_4 (1 << 1)
-#define MG_BURST_LEN_8 (2 << 1)
-#define MG_BURST_LEN_16 (3 << 1)
-#define MG_BURST_LEN_32 (4 << 1)
-#define MG_BURST_LEN_CONT (0 << 1)
-
-/* timeout value (unit: ms) */
-#define MG_TMAX_CONF_TO_CMD 1
-#define MG_TMAX_WAIT_RD_DRQ 10
-#define MG_TMAX_WAIT_WR_DRQ 500
-#define MG_TMAX_RST_TO_BUSY 10
-#define MG_TMAX_HDRST_TO_RDY 500
-#define MG_TMAX_SWRST_TO_RDY 500
-#define MG_TMAX_RSTOUT 3000
-
-#define MG_DEV_MASK (MG_BOOT_DEV | MG_STORAGE_DEV | MG_STORAGE_DEV_SKIP_RST)
-
-/* main structure for mflash driver */
-struct mg_host {
- struct device *dev;
-
- struct request_queue *breq;
- struct request *req;
- spinlock_t lock;
- struct gendisk *gd;
-
- struct timer_list timer;
- void (*mg_do_intr) (struct mg_host *);
-
- u16 id[ATA_ID_WORDS];
-
- u16 cyls;
- u16 heads;
- u16 sectors;
- u32 n_sectors;
- u32 nres_sectors;
-
- void __iomem *dev_base;
- unsigned int irq;
- unsigned int rst;
- unsigned int rstout;
-
- u32 major;
- u32 error;
-};
-
-/*
- * Debugging macro and defines
- */
-#undef DO_MG_DEBUG
-#ifdef DO_MG_DEBUG
-# define MG_DBG(fmt, args...) \
- printk(KERN_DEBUG "%s:%d "fmt, __func__, __LINE__, ##args)
-#else /* CONFIG_MG_DEBUG */
-# define MG_DBG(fmt, args...) do { } while (0)
-#endif /* CONFIG_MG_DEBUG */
-
-static void mg_request(struct request_queue *);
-
-static bool mg_end_request(struct mg_host *host, int err, unsigned int nr_bytes)
-{
- if (__blk_end_request(host->req, err, nr_bytes))
- return true;
-
- host->req = NULL;
- return false;
-}
-
-static bool mg_end_request_cur(struct mg_host *host, int err)
-{
- return mg_end_request(host, err, blk_rq_cur_bytes(host->req));
-}
-
-static void mg_dump_status(const char *msg, unsigned int stat,
- struct mg_host *host)
-{
- char *name = MG_DISK_NAME;
-
- if (host->req)
- name = host->req->rq_disk->disk_name;
-
- printk(KERN_ERR "%s: %s: status=0x%02x { ", name, msg, stat & 0xff);
- if (stat & ATA_BUSY)
- printk("Busy ");
- if (stat & ATA_DRDY)
- printk("DriveReady ");
- if (stat & ATA_DF)
- printk("WriteFault ");
- if (stat & ATA_DSC)
- printk("SeekComplete ");
- if (stat & ATA_DRQ)
- printk("DataRequest ");
- if (stat & ATA_CORR)
- printk("CorrectedError ");
- if (stat & ATA_ERR)
- printk("Error ");
- printk("}\n");
- if ((stat & ATA_ERR) == 0) {
- host->error = 0;
- } else {
- host->error = inb((unsigned long)host->dev_base + MG_REG_ERROR);
- printk(KERN_ERR "%s: %s: error=0x%02x { ", name, msg,
- host->error & 0xff);
- if (host->error & ATA_BBK)
- printk("BadSector ");
- if (host->error & ATA_UNC)
- printk("UncorrectableError ");
- if (host->error & ATA_IDNF)
- printk("SectorIdNotFound ");
- if (host->error & ATA_ABORTED)
- printk("DriveStatusError ");
- if (host->error & ATA_AMNF)
- printk("AddrMarkNotFound ");
- printk("}");
- if (host->error & (ATA_BBK | ATA_UNC | ATA_IDNF | ATA_AMNF)) {
- if (host->req)
- printk(", sector=%u",
- (unsigned int)blk_rq_pos(host->req));
- }
- printk("\n");
- }
-}
-
-static unsigned int mg_wait(struct mg_host *host, u32 expect, u32 msec)
-{
- u8 status;
- unsigned long expire, cur_jiffies;
- struct mg_drv_data *prv_data = host->dev->platform_data;
-
- host->error = MG_ERR_NONE;
- expire = jiffies + msecs_to_jiffies(msec);
-
- /* These 2 times dummy status read prevents reading invalid
- * status. A very little time (3 times of mflash operating clk)
- * is required for busy bit is set. Use dummy read instead of
- * busy wait, because mflash's PLL is machine dependent.
- */
- if (prv_data->use_polling) {
- status = inb((unsigned long)host->dev_base + MG_REG_STATUS);
- status = inb((unsigned long)host->dev_base + MG_REG_STATUS);
- }
-
- status = inb((unsigned long)host->dev_base + MG_REG_STATUS);
-
- do {
- cur_jiffies = jiffies;
- if (status & ATA_BUSY) {
- if (expect == ATA_BUSY)
- break;
- } else {
- /* Check the error condition! */
- if (status & ATA_ERR) {
- mg_dump_status("mg_wait", status, host);
- break;
- }
-
- if (expect == MG_STAT_READY)
- if (MG_READY_OK(status))
- break;
-
- if (expect == ATA_DRQ)
- if (status & ATA_DRQ)
- break;
- }
- if (!msec) {
- mg_dump_status("not ready", status, host);
- return MG_ERR_INV_STAT;
- }
-
- status = inb((unsigned long)host->dev_base + MG_REG_STATUS);
- } while (time_before(cur_jiffies, expire));
-
- if (time_after_eq(cur_jiffies, expire) && msec)
- host->error = MG_ERR_TIMEOUT;
-
- return host->error;
-}
-
-static unsigned int mg_wait_rstout(u32 rstout, u32 msec)
-{
- unsigned long expire;
-
- expire = jiffies + msecs_to_jiffies(msec);
- while (time_before(jiffies, expire)) {
- if (gpio_get_value(rstout) == 1)
- return MG_ERR_NONE;
- msleep(10);
- }
-
- return MG_ERR_RSTOUT;
-}
-
-static void mg_unexpected_intr(struct mg_host *host)
-{
- u32 status = inb((unsigned long)host->dev_base + MG_REG_STATUS);
-
- mg_dump_status("mg_unexpected_intr", status, host);
-}
-
-static irqreturn_t mg_irq(int irq, void *dev_id)
-{
- struct mg_host *host = dev_id;
- void (*handler)(struct mg_host *) = host->mg_do_intr;
-
- spin_lock(&host->lock);
-
- host->mg_do_intr = NULL;
- del_timer(&host->timer);
- if (!handler)
- handler = mg_unexpected_intr;
- handler(host);
-
- spin_unlock(&host->lock);
-
- return IRQ_HANDLED;
-}
-
-/* local copy of ata_id_string() */
-static void mg_id_string(const u16 *id, unsigned char *s,
- unsigned int ofs, unsigned int len)
-{
- unsigned int c;
-
- BUG_ON(len & 1);
-
- while (len > 0) {
- c = id[ofs] >> 8;
- *s = c;
- s++;
-
- c = id[ofs] & 0xff;
- *s = c;
- s++;
-
- ofs++;
- len -= 2;
- }
-}
-
-/* local copy of ata_id_c_string() */
-static void mg_id_c_string(const u16 *id, unsigned char *s,
- unsigned int ofs, unsigned int len)
-{
- unsigned char *p;
-
- mg_id_string(id, s, ofs, len - 1);
-
- p = s + strnlen(s, len - 1);
- while (p > s && p[-1] == ' ')
- p--;
- *p = '\0';
-}
-
-static int mg_get_disk_id(struct mg_host *host)
-{
- u32 i;
- s32 err;
- const u16 *id = host->id;
- struct mg_drv_data *prv_data = host->dev->platform_data;
- char fwrev[ATA_ID_FW_REV_LEN + 1];
- char model[ATA_ID_PROD_LEN + 1];
- char serial[ATA_ID_SERNO_LEN + 1];
-
- if (!prv_data->use_polling)
- outb(ATA_NIEN, (unsigned long)host->dev_base + MG_REG_DRV_CTRL);
-
- outb(MG_CMD_ID, (unsigned long)host->dev_base + MG_REG_COMMAND);
- err = mg_wait(host, ATA_DRQ, MG_TMAX_WAIT_RD_DRQ);
- if (err)
- return err;
-
- for (i = 0; i < (MG_SECTOR_SIZE >> 1); i++)
- host->id[i] = le16_to_cpu(inw((unsigned long)host->dev_base +
- MG_BUFF_OFFSET + i * 2));
-
- outb(MG_CMD_RD_CONF, (unsigned long)host->dev_base + MG_REG_COMMAND);
- err = mg_wait(host, MG_STAT_READY, MG_TMAX_CONF_TO_CMD);
- if (err)
- return err;
-
- if ((id[ATA_ID_FIELD_VALID] & 1) == 0)
- return MG_ERR_TRANSLATION;
-
- host->n_sectors = ata_id_u32(id, ATA_ID_LBA_CAPACITY);
- host->cyls = id[ATA_ID_CYLS];
- host->heads = id[ATA_ID_HEADS];
- host->sectors = id[ATA_ID_SECTORS];
-
- if (MG_RES_SEC && host->heads && host->sectors) {
- /* modify cyls, n_sectors */
- host->cyls = (host->n_sectors - MG_RES_SEC) /
- host->heads / host->sectors;
- host->nres_sectors = host->n_sectors - host->cyls *
- host->heads * host->sectors;
- host->n_sectors -= host->nres_sectors;
- }
-
- mg_id_c_string(id, fwrev, ATA_ID_FW_REV, sizeof(fwrev));
- mg_id_c_string(id, model, ATA_ID_PROD, sizeof(model));
- mg_id_c_string(id, serial, ATA_ID_SERNO, sizeof(serial));
- printk(KERN_INFO "mg_disk: model: %s\n", model);
- printk(KERN_INFO "mg_disk: firm: %.8s\n", fwrev);
- printk(KERN_INFO "mg_disk: serial: %s\n", serial);
- printk(KERN_INFO "mg_disk: %d + reserved %d sectors\n",
- host->n_sectors, host->nres_sectors);
-
- if (!prv_data->use_polling)
- outb(0, (unsigned long)host->dev_base + MG_REG_DRV_CTRL);
-
- return err;
-}
-
-
-static int mg_disk_init(struct mg_host *host)
-{
- struct mg_drv_data *prv_data = host->dev->platform_data;
- s32 err;
- u8 init_status;
-
- /* hdd rst low */
- gpio_set_value(host->rst, 0);
- err = mg_wait(host, ATA_BUSY, MG_TMAX_RST_TO_BUSY);
- if (err)
- return err;
-
- /* hdd rst high */
- gpio_set_value(host->rst, 1);
- err = mg_wait(host, MG_STAT_READY, MG_TMAX_HDRST_TO_RDY);
- if (err)
- return err;
-
- /* soft reset on */
- outb(ATA_SRST | (prv_data->use_polling ? ATA_NIEN : 0),
- (unsigned long)host->dev_base + MG_REG_DRV_CTRL);
- err = mg_wait(host, ATA_BUSY, MG_TMAX_RST_TO_BUSY);
- if (err)
- return err;
-
- /* soft reset off */
- outb(prv_data->use_polling ? ATA_NIEN : 0,
- (unsigned long)host->dev_base + MG_REG_DRV_CTRL);
- err = mg_wait(host, MG_STAT_READY, MG_TMAX_SWRST_TO_RDY);
- if (err)
- return err;
-
- init_status = inb((unsigned long)host->dev_base + MG_REG_STATUS) & 0xf;
-
- if (init_status == 0xf)
- return MG_ERR_INIT_STAT;
-
- return err;
-}
-
-static void mg_bad_rw_intr(struct mg_host *host)
-{
- if (host->req)
- if (++host->req->errors >= MG_MAX_ERRORS ||
- host->error == MG_ERR_TIMEOUT)
- mg_end_request_cur(host, -EIO);
-}
-
-static unsigned int mg_out(struct mg_host *host,
- unsigned int sect_num,
- unsigned int sect_cnt,
- unsigned int cmd,
- void (*intr_addr)(struct mg_host *))
-{
- struct mg_drv_data *prv_data = host->dev->platform_data;
-
- if (mg_wait(host, MG_STAT_READY, MG_TMAX_CONF_TO_CMD))
- return host->error;
-
- if (!prv_data->use_polling) {
- host->mg_do_intr = intr_addr;
- mod_timer(&host->timer, jiffies + 3 * HZ);
- }
- if (MG_RES_SEC)
- sect_num += MG_RES_SEC;
- outb((u8)sect_cnt, (unsigned long)host->dev_base + MG_REG_SECT_CNT);
- outb((u8)sect_num, (unsigned long)host->dev_base + MG_REG_SECT_NUM);
- outb((u8)(sect_num >> 8), (unsigned long)host->dev_base +
- MG_REG_CYL_LOW);
- outb((u8)(sect_num >> 16), (unsigned long)host->dev_base +
- MG_REG_CYL_HIGH);
- outb((u8)((sect_num >> 24) | ATA_LBA | ATA_DEVICE_OBS),
- (unsigned long)host->dev_base + MG_REG_DRV_HEAD);
- outb(cmd, (unsigned long)host->dev_base + MG_REG_COMMAND);
- return MG_ERR_NONE;
-}
-
-static void mg_read_one(struct mg_host *host, struct request *req)
-{
- u16 *buff = (u16 *)bio_data(req->bio);
- u32 i;
-
- for (i = 0; i < MG_SECTOR_SIZE >> 1; i++)
- *buff++ = inw((unsigned long)host->dev_base + MG_BUFF_OFFSET +
- (i << 1));
-}
-
-static void mg_read(struct request *req)
-{
- struct mg_host *host = req->rq_disk->private_data;
-
- if (mg_out(host, blk_rq_pos(req), blk_rq_sectors(req),
- MG_CMD_RD, NULL) != MG_ERR_NONE)
- mg_bad_rw_intr(host);
-
- MG_DBG("requested %d sects (from %ld), buffer=0x%p\n",
- blk_rq_sectors(req), blk_rq_pos(req), bio_data(req->bio));
-
- do {
- if (mg_wait(host, ATA_DRQ,
- MG_TMAX_WAIT_RD_DRQ) != MG_ERR_NONE) {
- mg_bad_rw_intr(host);
- return;
- }
-
- mg_read_one(host, req);
-
- outb(MG_CMD_RD_CONF, (unsigned long)host->dev_base +
- MG_REG_COMMAND);
- } while (mg_end_request(host, 0, MG_SECTOR_SIZE));
-}
-
-static void mg_write_one(struct mg_host *host, struct request *req)
-{
- u16 *buff = (u16 *)bio_data(req->bio);
- u32 i;
-
- for (i = 0; i < MG_SECTOR_SIZE >> 1; i++)
- outw(*buff++, (unsigned long)host->dev_base + MG_BUFF_OFFSET +
- (i << 1));
-}
-
-static void mg_write(struct request *req)
-{
- struct mg_host *host = req->rq_disk->private_data;
- unsigned int rem = blk_rq_sectors(req);
-
- if (mg_out(host, blk_rq_pos(req), rem,
- MG_CMD_WR, NULL) != MG_ERR_NONE) {
- mg_bad_rw_intr(host);
- return;
- }
-
- MG_DBG("requested %d sects (from %ld), buffer=0x%p\n",
- rem, blk_rq_pos(req), bio_data(req->bio));
-
- if (mg_wait(host, ATA_DRQ,
- MG_TMAX_WAIT_WR_DRQ) != MG_ERR_NONE) {
- mg_bad_rw_intr(host);
- return;
- }
-
- do {
- mg_write_one(host, req);
-
- outb(MG_CMD_WR_CONF, (unsigned long)host->dev_base +
- MG_REG_COMMAND);
-
- rem--;
- if (rem > 1 && mg_wait(host, ATA_DRQ,
- MG_TMAX_WAIT_WR_DRQ) != MG_ERR_NONE) {
- mg_bad_rw_intr(host);
- return;
- } else if (mg_wait(host, MG_STAT_READY,
- MG_TMAX_WAIT_WR_DRQ) != MG_ERR_NONE) {
- mg_bad_rw_intr(host);
- return;
- }
- } while (mg_end_request(host, 0, MG_SECTOR_SIZE));
-}
-
-static void mg_read_intr(struct mg_host *host)
-{
- struct request *req = host->req;
- u32 i;
-
- /* check status */
- do {
- i = inb((unsigned long)host->dev_base + MG_REG_STATUS);
- if (i & ATA_BUSY)
- break;
- if (!MG_READY_OK(i))
- break;
- if (i & ATA_DRQ)
- goto ok_to_read;
- } while (0);
- mg_dump_status("mg_read_intr", i, host);
- mg_bad_rw_intr(host);
- mg_request(host->breq);
- return;
-
-ok_to_read:
- mg_read_one(host, req);
-
- MG_DBG("sector %ld, remaining=%ld, buffer=0x%p\n",
- blk_rq_pos(req), blk_rq_sectors(req) - 1, bio_data(req->bio));
-
- /* send read confirm */
- outb(MG_CMD_RD_CONF, (unsigned long)host->dev_base + MG_REG_COMMAND);
-
- if (mg_end_request(host, 0, MG_SECTOR_SIZE)) {
- /* set handler if read remains */
- host->mg_do_intr = mg_read_intr;
- mod_timer(&host->timer, jiffies + 3 * HZ);
- } else /* goto next request */
- mg_request(host->breq);
-}
-
-static void mg_write_intr(struct mg_host *host)
-{
- struct request *req = host->req;
- u32 i;
- bool rem;
-
- /* check status */
- do {
- i = inb((unsigned long)host->dev_base + MG_REG_STATUS);
- if (i & ATA_BUSY)
- break;
- if (!MG_READY_OK(i))
- break;
- if ((blk_rq_sectors(req) <= 1) || (i & ATA_DRQ))
- goto ok_to_write;
- } while (0);
- mg_dump_status("mg_write_intr", i, host);
- mg_bad_rw_intr(host);
- mg_request(host->breq);
- return;
-
-ok_to_write:
- if ((rem = mg_end_request(host, 0, MG_SECTOR_SIZE))) {
- /* write 1 sector and set handler if remains */
- mg_write_one(host, req);
- MG_DBG("sector %ld, remaining=%ld, buffer=0x%p\n",
- blk_rq_pos(req), blk_rq_sectors(req), bio_data(req->bio));
- host->mg_do_intr = mg_write_intr;
- mod_timer(&host->timer, jiffies + 3 * HZ);
- }
-
- /* send write confirm */
- outb(MG_CMD_WR_CONF, (unsigned long)host->dev_base + MG_REG_COMMAND);
-
- if (!rem)
- mg_request(host->breq);
-}
-
-static void mg_times_out(unsigned long data)
-{
- struct mg_host *host = (struct mg_host *)data;
- char *name;
-
- spin_lock_irq(&host->lock);
-
- if (!host->req)
- goto out_unlock;
-
- host->mg_do_intr = NULL;
-
- name = host->req->rq_disk->disk_name;
- printk(KERN_DEBUG "%s: timeout\n", name);
-
- host->error = MG_ERR_TIMEOUT;
- mg_bad_rw_intr(host);
-
-out_unlock:
- mg_request(host->breq);
- spin_unlock_irq(&host->lock);
-}
-
-static void mg_request_poll(struct request_queue *q)
-{
- struct mg_host *host = q->queuedata;
-
- while (1) {
- if (!host->req) {
- host->req = blk_fetch_request(q);
- if (!host->req)
- break;
- }
-
- switch (req_op(host->req)) {
- case REQ_OP_READ:
- mg_read(host->req);
- break;
- case REQ_OP_WRITE:
- mg_write(host->req);
- break;
- default:
- mg_end_request_cur(host, -EIO);
- break;
- }
- }
-}
-
-static unsigned int mg_issue_req(struct request *req,
- struct mg_host *host,
- unsigned int sect_num,
- unsigned int sect_cnt)
-{
- switch (req_op(host->req)) {
- case REQ_OP_READ:
- if (mg_out(host, sect_num, sect_cnt, MG_CMD_RD, &mg_read_intr)
- != MG_ERR_NONE) {
- mg_bad_rw_intr(host);
- return host->error;
- }
- break;
- case REQ_OP_WRITE:
- /* TODO : handler */
- outb(ATA_NIEN, (unsigned long)host->dev_base + MG_REG_DRV_CTRL);
- if (mg_out(host, sect_num, sect_cnt, MG_CMD_WR, &mg_write_intr)
- != MG_ERR_NONE) {
- mg_bad_rw_intr(host);
- return host->error;
- }
- del_timer(&host->timer);
- mg_wait(host, ATA_DRQ, MG_TMAX_WAIT_WR_DRQ);
- outb(0, (unsigned long)host->dev_base + MG_REG_DRV_CTRL);
- if (host->error) {
- mg_bad_rw_intr(host);
- return host->error;
- }
- mg_write_one(host, req);
- mod_timer(&host->timer, jiffies + 3 * HZ);
- outb(MG_CMD_WR_CONF, (unsigned long)host->dev_base +
- MG_REG_COMMAND);
- break;
- default:
- mg_end_request_cur(host, -EIO);
- break;
- }
- return MG_ERR_NONE;
-}
-
-/* This function also called from IRQ context */
-static void mg_request(struct request_queue *q)
-{
- struct mg_host *host = q->queuedata;
- struct request *req;
- u32 sect_num, sect_cnt;
-
- while (1) {
- if (!host->req) {
- host->req = blk_fetch_request(q);
- if (!host->req)
- break;
- }
- req = host->req;
-
- /* check unwanted request call */
- if (host->mg_do_intr)
- return;
-
- del_timer(&host->timer);
-
- sect_num = blk_rq_pos(req);
- /* deal whole segments */
- sect_cnt = blk_rq_sectors(req);
-
- /* sanity check */
- if (sect_num >= get_capacity(req->rq_disk) ||
- ((sect_num + sect_cnt) >
- get_capacity(req->rq_disk))) {
- printk(KERN_WARNING
- "%s: bad access: sector=%d, count=%d\n",
- req->rq_disk->disk_name,
- sect_num, sect_cnt);
- mg_end_request_cur(host, -EIO);
- continue;
- }
-
- if (!mg_issue_req(req, host, sect_num, sect_cnt))
- return;
- }
-}
-
-static int mg_getgeo(struct block_device *bdev, struct hd_geometry *geo)
-{
- struct mg_host *host = bdev->bd_disk->private_data;
-
- geo->cylinders = (unsigned short)host->cyls;
- geo->heads = (unsigned char)host->heads;
- geo->sectors = (unsigned char)host->sectors;
- return 0;
-}
-
-static const struct block_device_operations mg_disk_ops = {
- .getgeo = mg_getgeo
-};
-
-#ifdef CONFIG_PM_SLEEP
-static int mg_suspend(struct device *dev)
-{
- struct mg_drv_data *prv_data = dev->platform_data;
- struct mg_host *host = prv_data->host;
-
- if (mg_wait(host, MG_STAT_READY, MG_TMAX_CONF_TO_CMD))
- return -EIO;
-
- if (!prv_data->use_polling)
- outb(ATA_NIEN, (unsigned long)host->dev_base + MG_REG_DRV_CTRL);
-
- outb(MG_CMD_SLEEP, (unsigned long)host->dev_base + MG_REG_COMMAND);
- /* wait until mflash deep sleep */
- msleep(1);
-
- if (mg_wait(host, MG_STAT_READY, MG_TMAX_CONF_TO_CMD)) {
- if (!prv_data->use_polling)
- outb(0, (unsigned long)host->dev_base + MG_REG_DRV_CTRL);
- return -EIO;
- }
-
- return 0;
-}
-
-static int mg_resume(struct device *dev)
-{
- struct mg_drv_data *prv_data = dev->platform_data;
- struct mg_host *host = prv_data->host;
-
- if (mg_wait(host, MG_STAT_READY, MG_TMAX_CONF_TO_CMD))
- return -EIO;
-
- outb(MG_CMD_WAKEUP, (unsigned long)host->dev_base + MG_REG_COMMAND);
- /* wait until mflash wakeup */
- msleep(1);
-
- if (mg_wait(host, MG_STAT_READY, MG_TMAX_CONF_TO_CMD))
- return -EIO;
-
- if (!prv_data->use_polling)
- outb(0, (unsigned long)host->dev_base + MG_REG_DRV_CTRL);
-
- return 0;
-}
-#endif
-
-static SIMPLE_DEV_PM_OPS(mg_pm, mg_suspend, mg_resume);
-
-static int mg_probe(struct platform_device *plat_dev)
-{
- struct mg_host *host;
- struct resource *rsc;
- struct mg_drv_data *prv_data = plat_dev->dev.platform_data;
- int err = 0;
-
- if (!prv_data) {
- printk(KERN_ERR "%s:%d fail (no driver_data)\n",
- __func__, __LINE__);
- err = -EINVAL;
- goto probe_err;
- }
-
- /* alloc mg_host */
- host = kzalloc(sizeof(struct mg_host), GFP_KERNEL);
- if (!host) {
- printk(KERN_ERR "%s:%d fail (no memory for mg_host)\n",
- __func__, __LINE__);
- err = -ENOMEM;
- goto probe_err;
- }
- host->major = MG_DISK_MAJ;
-
- /* link each other */
- prv_data->host = host;
- host->dev = &plat_dev->dev;
-
- /* io remap */
- rsc = platform_get_resource(plat_dev, IORESOURCE_MEM, 0);
- if (!rsc) {
- printk(KERN_ERR "%s:%d platform_get_resource fail\n",
- __func__, __LINE__);
- err = -EINVAL;
- goto probe_err_2;
- }
- host->dev_base = ioremap(rsc->start, resource_size(rsc));
- if (!host->dev_base) {
- printk(KERN_ERR "%s:%d ioremap fail\n",
- __func__, __LINE__);
- err = -EIO;
- goto probe_err_2;
- }
- MG_DBG("dev_base = 0x%x\n", (u32)host->dev_base);
-
- /* get reset pin */
- rsc = platform_get_resource_byname(plat_dev, IORESOURCE_IO,
- MG_RST_PIN);
- if (!rsc) {
- printk(KERN_ERR "%s:%d get reset pin fail\n",
- __func__, __LINE__);
- err = -EIO;
- goto probe_err_3;
- }
- host->rst = rsc->start;
-
- /* init rst pin */
- err = gpio_request(host->rst, MG_RST_PIN);
- if (err)
- goto probe_err_3;
- gpio_direction_output(host->rst, 1);
-
- /* reset out pin */
- if (!(prv_data->dev_attr & MG_DEV_MASK)) {
- err = -EINVAL;
- goto probe_err_3a;
- }
-
- if (prv_data->dev_attr != MG_BOOT_DEV) {
- rsc = platform_get_resource_byname(plat_dev, IORESOURCE_IO,
- MG_RSTOUT_PIN);
- if (!rsc) {
- printk(KERN_ERR "%s:%d get reset-out pin fail\n",
- __func__, __LINE__);
- err = -EIO;
- goto probe_err_3a;
- }
- host->rstout = rsc->start;
- err = gpio_request(host->rstout, MG_RSTOUT_PIN);
- if (err)
- goto probe_err_3a;
- gpio_direction_input(host->rstout);
- }
-
- /* disk reset */
- if (prv_data->dev_attr == MG_STORAGE_DEV) {
- /* If POR seq. not yet finished, wait */
- err = mg_wait_rstout(host->rstout, MG_TMAX_RSTOUT);
- if (err)
- goto probe_err_3b;
- err = mg_disk_init(host);
- if (err) {
- printk(KERN_ERR "%s:%d fail (err code : %d)\n",
- __func__, __LINE__, err);
- err = -EIO;
- goto probe_err_3b;
- }
- }
-
- /* get irq resource */
- if (!prv_data->use_polling) {
- host->irq = platform_get_irq(plat_dev, 0);
- if (host->irq == -ENXIO) {
- err = host->irq;
- goto probe_err_3b;
- }
- err = request_irq(host->irq, mg_irq,
- IRQF_TRIGGER_RISING,
- MG_DEV_NAME, host);
- if (err) {
- printk(KERN_ERR "%s:%d fail (request_irq err=%d)\n",
- __func__, __LINE__, err);
- goto probe_err_3b;
- }
-
- }
-
- /* get disk id */
- err = mg_get_disk_id(host);
- if (err) {
- printk(KERN_ERR "%s:%d fail (err code : %d)\n",
- __func__, __LINE__, err);
- err = -EIO;
- goto probe_err_4;
- }
-
- err = register_blkdev(host->major, MG_DISK_NAME);
- if (err < 0) {
- printk(KERN_ERR "%s:%d register_blkdev fail (err code : %d)\n",
- __func__, __LINE__, err);
- goto probe_err_4;
- }
- if (!host->major)
- host->major = err;
-
- spin_lock_init(&host->lock);
-
- if (prv_data->use_polling)
- host->breq = blk_init_queue(mg_request_poll, &host->lock);
- else
- host->breq = blk_init_queue(mg_request, &host->lock);
-
- if (!host->breq) {
- err = -ENOMEM;
- printk(KERN_ERR "%s:%d (blk_init_queue) fail\n",
- __func__, __LINE__);
- goto probe_err_5;
- }
- host->breq->queuedata = host;
-
- /* mflash is random device, thanx for the noop */
- err = elevator_change(host->breq, "noop");
- if (err) {
- printk(KERN_ERR "%s:%d (elevator_init) fail\n",
- __func__, __LINE__);
- goto probe_err_6;
- }
- blk_queue_max_hw_sectors(host->breq, MG_MAX_SECTS);
- blk_queue_logical_block_size(host->breq, MG_SECTOR_SIZE);
-
- init_timer(&host->timer);
- host->timer.function = mg_times_out;
- host->timer.data = (unsigned long)host;
-
- host->gd = alloc_disk(MG_DISK_MAX_PART);
- if (!host->gd) {
- printk(KERN_ERR "%s:%d (alloc_disk) fail\n",
- __func__, __LINE__);
- err = -ENOMEM;
- goto probe_err_7;
- }
- host->gd->major = host->major;
- host->gd->first_minor = 0;
- host->gd->fops = &mg_disk_ops;
- host->gd->queue = host->breq;
- host->gd->private_data = host;
- sprintf(host->gd->disk_name, MG_DISK_NAME"a");
-
- set_capacity(host->gd, host->n_sectors);
-
- add_disk(host->gd);
-
- return err;
-
-probe_err_7:
- del_timer_sync(&host->timer);
-probe_err_6:
- blk_cleanup_queue(host->breq);
-probe_err_5:
- unregister_blkdev(host->major, MG_DISK_NAME);
-probe_err_4:
- if (!prv_data->use_polling)
- free_irq(host->irq, host);
-probe_err_3b:
- gpio_free(host->rstout);
-probe_err_3a:
- gpio_free(host->rst);
-probe_err_3:
- iounmap(host->dev_base);
-probe_err_2:
- kfree(host);
-probe_err:
- return err;
-}
-
-static int mg_remove(struct platform_device *plat_dev)
-{
- struct mg_drv_data *prv_data = plat_dev->dev.platform_data;
- struct mg_host *host = prv_data->host;
- int err = 0;
-
- /* delete timer */
- del_timer_sync(&host->timer);
-
- /* remove disk */
- if (host->gd) {
- del_gendisk(host->gd);
- put_disk(host->gd);
- }
- /* remove queue */
- if (host->breq)
- blk_cleanup_queue(host->breq);
-
- /* unregister blk device */
- unregister_blkdev(host->major, MG_DISK_NAME);
-
- /* free irq */
- if (!prv_data->use_polling)
- free_irq(host->irq, host);
-
- /* free reset-out pin */
- if (prv_data->dev_attr != MG_BOOT_DEV)
- gpio_free(host->rstout);
-
- /* free rst pin */
- if (host->rst)
- gpio_free(host->rst);
-
- /* unmap io */
- if (host->dev_base)
- iounmap(host->dev_base);
-
- /* free mg_host */
- kfree(host);
-
- return err;
-}
-
-static struct platform_driver mg_disk_driver = {
- .probe = mg_probe,
- .remove = mg_remove,
- .driver = {
- .name = MG_DEV_NAME,
- .pm = &mg_pm,
- }
-};
-
-/****************************************************************************
- *
- * Module stuff
- *
- ****************************************************************************/
-
-static int __init mg_init(void)
-{
- printk(KERN_INFO "mGine mflash driver, (c) 2008 mGine Co.\n");
- return platform_driver_register(&mg_disk_driver);
-}
-
-static void __exit mg_exit(void)
-{
- printk(KERN_INFO "mflash driver : bye bye\n");
- platform_driver_unregister(&mg_disk_driver);
-}
-
-module_init(mg_init);
-module_exit(mg_exit);
-
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("unsik Kim <donari75@gmail.com>");
-MODULE_DESCRIPTION("mGine m[g]flash device driver");
diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c
index 1d1dc11..02804cc 100644
--- a/drivers/block/mtip32xx/mtip32xx.c
+++ b/drivers/block/mtip32xx/mtip32xx.c
@@ -169,6 +169,25 @@
return false; /* device present */
}
+/* we have to use runtime tag to setup command header */
+static void mtip_init_cmd_header(struct request *rq)
+{
+ struct driver_data *dd = rq->q->queuedata;
+ struct mtip_cmd *cmd = blk_mq_rq_to_pdu(rq);
+ u32 host_cap_64 = readl(dd->mmio + HOST_CAP) & HOST_CAP_64;
+
+ /* Point the command headers at the command tables. */
+ cmd->command_header = dd->port->command_list +
+ (sizeof(struct mtip_cmd_hdr) * rq->tag);
+ cmd->command_header_dma = dd->port->command_list_dma +
+ (sizeof(struct mtip_cmd_hdr) * rq->tag);
+
+ if (host_cap_64)
+ cmd->command_header->ctbau = __force_bit2int cpu_to_le32((cmd->command_dma >> 16) >> 16);
+
+ cmd->command_header->ctba = __force_bit2int cpu_to_le32(cmd->command_dma & 0xFFFFFFFF);
+}
+
static struct mtip_cmd *mtip_get_int_command(struct driver_data *dd)
{
struct request *rq;
@@ -180,6 +199,9 @@
if (IS_ERR(rq))
return NULL;
+ /* Internal cmd isn't submitted via .queue_rq */
+ mtip_init_cmd_header(rq);
+
return blk_mq_rq_to_pdu(rq);
}
@@ -241,7 +263,8 @@
rq = mtip_rq_from_tag(dd, tag);
- blk_mq_complete_request(rq, status);
+ cmd->status = status;
+ blk_mq_complete_request(rq);
}
/*
@@ -2910,18 +2933,19 @@
if (unlikely(cmd->unaligned))
up(&dd->port->cmd_slot_unal);
- blk_mq_end_request(rq, rq->errors);
+ blk_mq_end_request(rq, cmd->status);
}
static void mtip_abort_cmd(struct request *req, void *data,
bool reserved)
{
+ struct mtip_cmd *cmd = blk_mq_rq_to_pdu(req);
struct driver_data *dd = data;
dbg_printk(MTIP_DRV_NAME " Aborting request, tag = %d\n", req->tag);
clear_bit(req->tag, dd->port->cmds_to_issue);
- req->errors = -EIO;
+ cmd->status = -EIO;
mtip_softirq_done_fn(req);
}
@@ -3807,6 +3831,8 @@
struct request *rq = bd->rq;
int ret;
+ mtip_init_cmd_header(rq);
+
if (unlikely(mtip_check_unal_depth(hctx, rq)))
return BLK_MQ_RQ_QUEUE_BUSY;
@@ -3816,7 +3842,6 @@
if (likely(!ret))
return BLK_MQ_RQ_QUEUE_OK;
- rq->errors = ret;
return BLK_MQ_RQ_QUEUE_ERROR;
}
@@ -3838,7 +3863,6 @@
{
struct driver_data *dd = data;
struct mtip_cmd *cmd = blk_mq_rq_to_pdu(rq);
- u32 host_cap_64 = readl(dd->mmio + HOST_CAP) & HOST_CAP_64;
/*
* For flush requests, request_idx starts at the end of the
@@ -3855,17 +3879,6 @@
memset(cmd->command, 0, CMD_DMA_ALLOC_SZ);
- /* Point the command headers at the command tables. */
- cmd->command_header = dd->port->command_list +
- (sizeof(struct mtip_cmd_hdr) * request_idx);
- cmd->command_header_dma = dd->port->command_list_dma +
- (sizeof(struct mtip_cmd_hdr) * request_idx);
-
- if (host_cap_64)
- cmd->command_header->ctbau = __force_bit2int cpu_to_le32((cmd->command_dma >> 16) >> 16);
-
- cmd->command_header->ctba = __force_bit2int cpu_to_le32(cmd->command_dma & 0xFFFFFFFF);
-
sg_init_table(cmd->sg, MTIP_MAX_SG);
return 0;
}
@@ -3889,7 +3902,7 @@
return BLK_EH_RESET_TIMER;
}
-static struct blk_mq_ops mtip_mq_ops = {
+static const struct blk_mq_ops mtip_mq_ops = {
.queue_rq = mtip_queue_rq,
.init_request = mtip_init_cmd,
.exit_request = mtip_free_cmd,
@@ -4025,7 +4038,6 @@
dd->queue->limits.discard_granularity = 4096;
blk_queue_max_discard_sectors(dd->queue,
MTIP_MAX_TRIM_ENTRY_LEN * MTIP_MAX_TRIM_ENTRIES);
- dd->queue->limits.discard_zeroes_data = 0;
}
/* Set the capacity of the device in 512 byte sectors. */
@@ -4107,9 +4119,11 @@
struct driver_data *dd = (struct driver_data *)data;
struct mtip_cmd *cmd;
- if (likely(!reserv))
- blk_mq_complete_request(rq, -ENODEV);
- else if (test_bit(MTIP_PF_IC_ACTIVE_BIT, &dd->port->flags)) {
+ if (likely(!reserv)) {
+ cmd = blk_mq_rq_to_pdu(rq);
+ cmd->status = -ENODEV;
+ blk_mq_complete_request(rq);
+ } else if (test_bit(MTIP_PF_IC_ACTIVE_BIT, &dd->port->flags)) {
cmd = mtip_cmd_from_tag(dd, MTIP_TAG_INTERNAL);
if (cmd->comp_func)
@@ -4162,7 +4176,7 @@
dev_info(&dd->pdev->dev, "device %s surprise removal\n",
dd->disk->disk_name);
- blk_mq_freeze_queue_start(dd->queue);
+ blk_freeze_queue_start(dd->queue);
blk_mq_stop_hw_queues(dd->queue);
blk_mq_tagset_busy_iter(&dd->tags, mtip_no_dev_cleanup, dd);
diff --git a/drivers/block/mtip32xx/mtip32xx.h b/drivers/block/mtip32xx/mtip32xx.h
index 7617888..57b4152 100644
--- a/drivers/block/mtip32xx/mtip32xx.h
+++ b/drivers/block/mtip32xx/mtip32xx.h
@@ -352,6 +352,7 @@
int retries; /* The number of retries left for this command. */
int direction; /* Data transfer direction */
+ int status;
};
/* Structure used to describe a port. */
diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index d8a2356..9b482ba 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -40,49 +40,82 @@
#include <asm/types.h>
#include <linux/nbd.h>
+#include <linux/nbd-netlink.h>
+#include <net/genetlink.h>
static DEFINE_IDR(nbd_index_idr);
static DEFINE_MUTEX(nbd_index_mutex);
+static int nbd_total_devices = 0;
struct nbd_sock {
struct socket *sock;
struct mutex tx_lock;
struct request *pending;
int sent;
+ bool dead;
+ int fallback_index;
+ int cookie;
+};
+
+struct recv_thread_args {
+ struct work_struct work;
+ struct nbd_device *nbd;
+ int index;
+};
+
+struct link_dead_args {
+ struct work_struct work;
+ int index;
};
#define NBD_TIMEDOUT 0
#define NBD_DISCONNECT_REQUESTED 1
#define NBD_DISCONNECTED 2
-#define NBD_RUNNING 3
+#define NBD_HAS_PID_FILE 3
+#define NBD_HAS_CONFIG_REF 4
+#define NBD_BOUND 5
+#define NBD_DESTROY_ON_DISCONNECT 6
-struct nbd_device {
+struct nbd_config {
u32 flags;
unsigned long runtime_flags;
+ u64 dead_conn_timeout;
+
struct nbd_sock **socks;
- int magic;
-
- struct blk_mq_tag_set tag_set;
-
- struct mutex config_lock;
- struct gendisk *disk;
int num_connections;
+ atomic_t live_connections;
+ wait_queue_head_t conn_wait;
+
atomic_t recv_threads;
wait_queue_head_t recv_wq;
loff_t blksize;
loff_t bytesize;
-
- struct task_struct *task_recv;
- struct task_struct *task_setup;
-
#if IS_ENABLED(CONFIG_DEBUG_FS)
struct dentry *dbg_dir;
#endif
};
+struct nbd_device {
+ struct blk_mq_tag_set tag_set;
+
+ int index;
+ refcount_t config_refs;
+ refcount_t refs;
+ struct nbd_config *config;
+ struct mutex config_lock;
+ struct gendisk *disk;
+
+ struct list_head list;
+ struct task_struct *task_recv;
+ struct task_struct *task_setup;
+};
+
struct nbd_cmd {
struct nbd_device *nbd;
+ int index;
+ int cookie;
struct completion send_complete;
+ int status;
};
#if IS_ENABLED(CONFIG_DEBUG_FS)
@@ -100,18 +133,16 @@
static int nbd_dev_dbg_init(struct nbd_device *nbd);
static void nbd_dev_dbg_close(struct nbd_device *nbd);
-
+static void nbd_config_put(struct nbd_device *nbd);
+static void nbd_connect_reply(struct genl_info *info, int index);
+static int nbd_genl_status(struct sk_buff *skb, struct genl_info *info);
+static void nbd_dead_link_work(struct work_struct *work);
static inline struct device *nbd_to_dev(struct nbd_device *nbd)
{
return disk_to_dev(nbd->disk);
}
-static bool nbd_is_connected(struct nbd_device *nbd)
-{
- return !!nbd->task_recv;
-}
-
static const char *nbdcmd_to_ascii(int cmd)
{
switch (cmd) {
@@ -124,44 +155,104 @@
return "invalid";
}
-static int nbd_size_clear(struct nbd_device *nbd, struct block_device *bdev)
+static ssize_t pid_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
{
- if (bdev->bd_openers <= 1)
- bd_set_size(bdev, 0);
- set_capacity(nbd->disk, 0);
- kobject_uevent(&nbd_to_dev(nbd)->kobj, KOBJ_CHANGE);
+ struct gendisk *disk = dev_to_disk(dev);
+ struct nbd_device *nbd = (struct nbd_device *)disk->private_data;
- return 0;
+ return sprintf(buf, "%d\n", task_pid_nr(nbd->task_recv));
}
-static void nbd_size_update(struct nbd_device *nbd, struct block_device *bdev)
+static struct device_attribute pid_attr = {
+ .attr = { .name = "pid", .mode = S_IRUGO},
+ .show = pid_show,
+};
+
+static void nbd_dev_remove(struct nbd_device *nbd)
{
- blk_queue_logical_block_size(nbd->disk->queue, nbd->blksize);
- blk_queue_physical_block_size(nbd->disk->queue, nbd->blksize);
- bd_set_size(bdev, nbd->bytesize);
- set_capacity(nbd->disk, nbd->bytesize >> 9);
+ struct gendisk *disk = nbd->disk;
+ if (disk) {
+ del_gendisk(disk);
+ blk_cleanup_queue(disk->queue);
+ blk_mq_free_tag_set(&nbd->tag_set);
+ disk->private_data = NULL;
+ put_disk(disk);
+ }
+ kfree(nbd);
+}
+
+static void nbd_put(struct nbd_device *nbd)
+{
+ if (refcount_dec_and_mutex_lock(&nbd->refs,
+ &nbd_index_mutex)) {
+ idr_remove(&nbd_index_idr, nbd->index);
+ mutex_unlock(&nbd_index_mutex);
+ nbd_dev_remove(nbd);
+ }
+}
+
+static int nbd_disconnected(struct nbd_config *config)
+{
+ return test_bit(NBD_DISCONNECTED, &config->runtime_flags) ||
+ test_bit(NBD_DISCONNECT_REQUESTED, &config->runtime_flags);
+}
+
+static void nbd_mark_nsock_dead(struct nbd_device *nbd, struct nbd_sock *nsock,
+ int notify)
+{
+ if (!nsock->dead && notify && !nbd_disconnected(nbd->config)) {
+ struct link_dead_args *args;
+ args = kmalloc(sizeof(struct link_dead_args), GFP_NOIO);
+ if (args) {
+ INIT_WORK(&args->work, nbd_dead_link_work);
+ args->index = nbd->index;
+ queue_work(system_wq, &args->work);
+ }
+ }
+ if (!nsock->dead) {
+ kernel_sock_shutdown(nsock->sock, SHUT_RDWR);
+ atomic_dec(&nbd->config->live_connections);
+ }
+ nsock->dead = true;
+ nsock->pending = NULL;
+ nsock->sent = 0;
+}
+
+static void nbd_size_clear(struct nbd_device *nbd)
+{
+ if (nbd->config->bytesize) {
+ set_capacity(nbd->disk, 0);
+ kobject_uevent(&nbd_to_dev(nbd)->kobj, KOBJ_CHANGE);
+ }
+}
+
+static void nbd_size_update(struct nbd_device *nbd)
+{
+ struct nbd_config *config = nbd->config;
+ blk_queue_logical_block_size(nbd->disk->queue, config->blksize);
+ blk_queue_physical_block_size(nbd->disk->queue, config->blksize);
+ set_capacity(nbd->disk, config->bytesize >> 9);
kobject_uevent(&nbd_to_dev(nbd)->kobj, KOBJ_CHANGE);
}
-static void nbd_size_set(struct nbd_device *nbd, struct block_device *bdev,
- loff_t blocksize, loff_t nr_blocks)
+static void nbd_size_set(struct nbd_device *nbd, loff_t blocksize,
+ loff_t nr_blocks)
{
- nbd->blksize = blocksize;
- nbd->bytesize = blocksize * nr_blocks;
- if (nbd_is_connected(nbd))
- nbd_size_update(nbd, bdev);
+ struct nbd_config *config = nbd->config;
+ config->blksize = blocksize;
+ config->bytesize = blocksize * nr_blocks;
+ nbd_size_update(nbd);
}
-static void nbd_end_request(struct nbd_cmd *cmd)
+static void nbd_complete_rq(struct request *req)
{
- struct nbd_device *nbd = cmd->nbd;
- struct request *req = blk_mq_rq_from_pdu(cmd);
- int error = req->errors ? -EIO : 0;
+ struct nbd_cmd *cmd = blk_mq_rq_to_pdu(req);
- dev_dbg(nbd_to_dev(nbd), "request %p: %s\n", cmd,
- error ? "failed" : "done");
+ dev_dbg(nbd_to_dev(cmd->nbd), "request %p: %s\n", cmd,
+ cmd->status ? "failed" : "done");
- blk_mq_complete_request(req, error);
+ blk_mq_end_request(req, cmd->status);
}
/*
@@ -169,17 +260,18 @@
*/
static void sock_shutdown(struct nbd_device *nbd)
{
+ struct nbd_config *config = nbd->config;
int i;
- if (nbd->num_connections == 0)
+ if (config->num_connections == 0)
return;
- if (test_and_set_bit(NBD_DISCONNECTED, &nbd->runtime_flags))
+ if (test_and_set_bit(NBD_DISCONNECTED, &config->runtime_flags))
return;
- for (i = 0; i < nbd->num_connections; i++) {
- struct nbd_sock *nsock = nbd->socks[i];
+ for (i = 0; i < config->num_connections; i++) {
+ struct nbd_sock *nsock = config->socks[i];
mutex_lock(&nsock->tx_lock);
- kernel_sock_shutdown(nsock->sock, SHUT_RDWR);
+ nbd_mark_nsock_dead(nbd, nsock, 0);
mutex_unlock(&nsock->tx_lock);
}
dev_warn(disk_to_dev(nbd->disk), "shutting down sockets\n");
@@ -190,14 +282,58 @@
{
struct nbd_cmd *cmd = blk_mq_rq_to_pdu(req);
struct nbd_device *nbd = cmd->nbd;
+ struct nbd_config *config;
- dev_err(nbd_to_dev(nbd), "Connection timed out, shutting down connection\n");
- set_bit(NBD_TIMEDOUT, &nbd->runtime_flags);
- req->errors = -EIO;
+ if (!refcount_inc_not_zero(&nbd->config_refs)) {
+ cmd->status = -EIO;
+ return BLK_EH_HANDLED;
+ }
- mutex_lock(&nbd->config_lock);
+ /* If we are waiting on our dead timer then we could get timeout
+ * callbacks for our request. For this we just want to reset the timer
+ * and let the queue side take care of everything.
+ */
+ if (!completion_done(&cmd->send_complete)) {
+ nbd_config_put(nbd);
+ return BLK_EH_RESET_TIMER;
+ }
+ config = nbd->config;
+
+ if (config->num_connections > 1) {
+ dev_err_ratelimited(nbd_to_dev(nbd),
+ "Connection timed out, retrying\n");
+ /*
+ * Hooray we have more connections, requeue this IO, the submit
+ * path will put it on a real connection.
+ */
+ if (config->socks && config->num_connections > 1) {
+ if (cmd->index < config->num_connections) {
+ struct nbd_sock *nsock =
+ config->socks[cmd->index];
+ mutex_lock(&nsock->tx_lock);
+ /* We can have multiple outstanding requests, so
+ * we don't want to mark the nsock dead if we've
+ * already reconnected with a new socket, so
+ * only mark it dead if its the same socket we
+ * were sent out on.
+ */
+ if (cmd->cookie == nsock->cookie)
+ nbd_mark_nsock_dead(nbd, nsock, 1);
+ mutex_unlock(&nsock->tx_lock);
+ }
+ blk_mq_requeue_request(req, true);
+ nbd_config_put(nbd);
+ return BLK_EH_NOT_HANDLED;
+ }
+ } else {
+ dev_err_ratelimited(nbd_to_dev(nbd),
+ "Connection timed out\n");
+ }
+ set_bit(NBD_TIMEDOUT, &config->runtime_flags);
+ cmd->status = -EIO;
sock_shutdown(nbd);
- mutex_unlock(&nbd->config_lock);
+ nbd_config_put(nbd);
+
return BLK_EH_HANDLED;
}
@@ -207,7 +343,8 @@
static int sock_xmit(struct nbd_device *nbd, int index, int send,
struct iov_iter *iter, int msg_flags, int *sent)
{
- struct socket *sock = nbd->socks[index]->sock;
+ struct nbd_config *config = nbd->config;
+ struct socket *sock = config->socks[index]->sock;
int result;
struct msghdr msg;
unsigned long pflags = current->flags;
@@ -244,7 +381,7 @@
*sent += result;
} while (msg_data_left(&msg));
- tsk_restore_flags(current, pflags, PF_MEMALLOC);
+ current_restore_flags(pflags, PF_MEMALLOC);
return result;
}
@@ -253,7 +390,8 @@
static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd, int index)
{
struct request *req = blk_mq_rq_from_pdu(cmd);
- struct nbd_sock *nsock = nbd->socks[index];
+ struct nbd_config *config = nbd->config;
+ struct nbd_sock *nsock = config->socks[index];
int result;
struct nbd_request request = {.magic = htonl(NBD_REQUEST_MAGIC)};
struct kvec iov = {.iov_base = &request, .iov_len = sizeof(request)};
@@ -284,7 +422,7 @@
}
if (rq_data_dir(req) == WRITE &&
- (nbd->flags & NBD_FLAG_READ_ONLY)) {
+ (config->flags & NBD_FLAG_READ_ONLY)) {
dev_err_ratelimited(disk_to_dev(nbd->disk),
"Write on read-only\n");
return -EIO;
@@ -301,6 +439,8 @@
}
iov_iter_advance(&from, sent);
}
+ cmd->index = index;
+ cmd->cookie = nsock->cookie;
request.type = htonl(type);
if (type != NBD_CMD_FLUSH) {
request.from = cpu_to_be64((u64)blk_rq_pos(req) << 9);
@@ -328,7 +468,7 @@
}
dev_err_ratelimited(disk_to_dev(nbd->disk),
"Send control failed (result %d)\n", result);
- return -EIO;
+ return -EAGAIN;
}
send_pages:
if (type != NBD_CMD_WRITE)
@@ -370,7 +510,7 @@
dev_err(disk_to_dev(nbd->disk),
"Send data failed (result %d)\n",
result);
- return -EIO;
+ return -EAGAIN;
}
/*
* The completion might already have come in,
@@ -392,6 +532,7 @@
/* NULL returned = something went wrong, inform userspace */
static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd, int index)
{
+ struct nbd_config *config = nbd->config;
int result;
struct nbd_reply reply;
struct nbd_cmd *cmd;
@@ -405,8 +546,7 @@
iov_iter_kvec(&to, READ | ITER_KVEC, &iov, 1, sizeof(reply));
result = sock_xmit(nbd, index, 0, &to, MSG_WAITALL, NULL);
if (result <= 0) {
- if (!test_bit(NBD_DISCONNECTED, &nbd->runtime_flags) &&
- !test_bit(NBD_DISCONNECT_REQUESTED, &nbd->runtime_flags))
+ if (!nbd_disconnected(config))
dev_err(disk_to_dev(nbd->disk),
"Receive control failed (result %d)\n", result);
return ERR_PTR(result);
@@ -433,7 +573,7 @@
if (ntohl(reply.error)) {
dev_err(disk_to_dev(nbd->disk), "Other side returned error (%d)\n",
ntohl(reply.error));
- req->errors = -EIO;
+ cmd->status = -EIO;
return cmd;
}
@@ -449,8 +589,19 @@
if (result <= 0) {
dev_err(disk_to_dev(nbd->disk), "Receive data failed (result %d)\n",
result);
- req->errors = -EIO;
- return cmd;
+ /*
+ * If we've disconnected or we only have 1
+ * connection then we need to make sure we
+ * complete this request, otherwise error out
+ * and let the timeout stuff handle resubmitting
+ * this request onto another connection.
+ */
+ if (nbd_disconnected(config) ||
+ config->num_connections <= 1) {
+ cmd->status = -EIO;
+ return cmd;
+ }
+ return ERR_PTR(-EIO);
}
dev_dbg(nbd_to_dev(nbd), "request %p: got %d bytes data\n",
cmd, bvec.bv_len);
@@ -462,54 +613,34 @@
return cmd;
}
-static ssize_t pid_show(struct device *dev,
- struct device_attribute *attr, char *buf)
-{
- struct gendisk *disk = dev_to_disk(dev);
- struct nbd_device *nbd = (struct nbd_device *)disk->private_data;
-
- return sprintf(buf, "%d\n", task_pid_nr(nbd->task_recv));
-}
-
-static struct device_attribute pid_attr = {
- .attr = { .name = "pid", .mode = S_IRUGO},
- .show = pid_show,
-};
-
-struct recv_thread_args {
- struct work_struct work;
- struct nbd_device *nbd;
- int index;
-};
-
static void recv_work(struct work_struct *work)
{
struct recv_thread_args *args = container_of(work,
struct recv_thread_args,
work);
struct nbd_device *nbd = args->nbd;
+ struct nbd_config *config = nbd->config;
struct nbd_cmd *cmd;
int ret = 0;
- BUG_ON(nbd->magic != NBD_MAGIC);
while (1) {
cmd = nbd_read_stat(nbd, args->index);
if (IS_ERR(cmd)) {
+ struct nbd_sock *nsock = config->socks[args->index];
+
+ mutex_lock(&nsock->tx_lock);
+ nbd_mark_nsock_dead(nbd, nsock, 1);
+ mutex_unlock(&nsock->tx_lock);
ret = PTR_ERR(cmd);
break;
}
- nbd_end_request(cmd);
+ blk_mq_complete_request(blk_mq_rq_from_pdu(cmd));
}
-
- /*
- * We got an error, shut everybody down if this wasn't the result of a
- * disconnect request.
- */
- if (ret && !test_bit(NBD_DISCONNECT_REQUESTED, &nbd->runtime_flags))
- sock_shutdown(nbd);
- atomic_dec(&nbd->recv_threads);
- wake_up(&nbd->recv_wq);
+ atomic_dec(&config->recv_threads);
+ wake_up(&config->recv_wq);
+ nbd_config_put(nbd);
+ kfree(args);
}
static void nbd_clear_req(struct request *req, void *data, bool reserved)
@@ -519,47 +650,119 @@
if (!blk_mq_request_started(req))
return;
cmd = blk_mq_rq_to_pdu(req);
- req->errors = -EIO;
- nbd_end_request(cmd);
+ cmd->status = -EIO;
+ blk_mq_complete_request(req);
}
static void nbd_clear_que(struct nbd_device *nbd)
{
- BUG_ON(nbd->magic != NBD_MAGIC);
-
+ blk_mq_stop_hw_queues(nbd->disk->queue);
blk_mq_tagset_busy_iter(&nbd->tag_set, nbd_clear_req, NULL);
+ blk_mq_start_hw_queues(nbd->disk->queue);
dev_dbg(disk_to_dev(nbd->disk), "queue cleared\n");
}
+static int find_fallback(struct nbd_device *nbd, int index)
+{
+ struct nbd_config *config = nbd->config;
+ int new_index = -1;
+ struct nbd_sock *nsock = config->socks[index];
+ int fallback = nsock->fallback_index;
+
+ if (test_bit(NBD_DISCONNECTED, &config->runtime_flags))
+ return new_index;
+
+ if (config->num_connections <= 1) {
+ dev_err_ratelimited(disk_to_dev(nbd->disk),
+ "Attempted send on invalid socket\n");
+ return new_index;
+ }
+
+ if (fallback >= 0 && fallback < config->num_connections &&
+ !config->socks[fallback]->dead)
+ return fallback;
+
+ if (nsock->fallback_index < 0 ||
+ nsock->fallback_index >= config->num_connections ||
+ config->socks[nsock->fallback_index]->dead) {
+ int i;
+ for (i = 0; i < config->num_connections; i++) {
+ if (i == index)
+ continue;
+ if (!config->socks[i]->dead) {
+ new_index = i;
+ break;
+ }
+ }
+ nsock->fallback_index = new_index;
+ if (new_index < 0) {
+ dev_err_ratelimited(disk_to_dev(nbd->disk),
+ "Dead connection, failed to find a fallback\n");
+ return new_index;
+ }
+ }
+ new_index = nsock->fallback_index;
+ return new_index;
+}
+
+static int wait_for_reconnect(struct nbd_device *nbd)
+{
+ struct nbd_config *config = nbd->config;
+ if (!config->dead_conn_timeout)
+ return 0;
+ if (test_bit(NBD_DISCONNECTED, &config->runtime_flags))
+ return 0;
+ wait_event_interruptible_timeout(config->conn_wait,
+ atomic_read(&config->live_connections),
+ config->dead_conn_timeout);
+ return atomic_read(&config->live_connections);
+}
static int nbd_handle_cmd(struct nbd_cmd *cmd, int index)
{
struct request *req = blk_mq_rq_from_pdu(cmd);
struct nbd_device *nbd = cmd->nbd;
+ struct nbd_config *config;
struct nbd_sock *nsock;
int ret;
- if (index >= nbd->num_connections) {
+ if (!refcount_inc_not_zero(&nbd->config_refs)) {
+ dev_err_ratelimited(disk_to_dev(nbd->disk),
+ "Socks array is empty\n");
+ return -EINVAL;
+ }
+ config = nbd->config;
+
+ if (index >= config->num_connections) {
dev_err_ratelimited(disk_to_dev(nbd->disk),
"Attempted send on invalid socket\n");
+ nbd_config_put(nbd);
return -EINVAL;
}
-
- if (test_bit(NBD_DISCONNECTED, &nbd->runtime_flags)) {
- dev_err_ratelimited(disk_to_dev(nbd->disk),
- "Attempted send on closed socket\n");
- return -EINVAL;
- }
-
- req->errors = 0;
-
- nsock = nbd->socks[index];
+ cmd->status = 0;
+again:
+ nsock = config->socks[index];
mutex_lock(&nsock->tx_lock);
- if (unlikely(!nsock->sock)) {
+ if (nsock->dead) {
+ int old_index = index;
+ index = find_fallback(nbd, index);
mutex_unlock(&nsock->tx_lock);
- dev_err_ratelimited(disk_to_dev(nbd->disk),
- "Attempted send on closed socket\n");
- return -EINVAL;
+ if (index < 0) {
+ if (wait_for_reconnect(nbd)) {
+ index = old_index;
+ goto again;
+ }
+ /* All the sockets should already be down at this point,
+ * we just want to make sure that DISCONNECTED is set so
+ * any requests that come in that were queue'ed waiting
+ * for the reconnect timer don't trigger the timer again
+ * and instead just error out.
+ */
+ sock_shutdown(nbd);
+ nbd_config_put(nbd);
+ return -EIO;
+ }
+ goto again;
}
/* Handle the case that we have a pending request that was partially
@@ -572,9 +775,21 @@
ret = 0;
goto out;
}
+ /*
+ * Some failures are related to the link going down, so anything that
+ * returns EAGAIN can be retried on a different socket.
+ */
ret = nbd_send_cmd(nbd, cmd, index);
+ if (ret == -EAGAIN) {
+ dev_err_ratelimited(disk_to_dev(nbd->disk),
+ "Request send failed trying another connection\n");
+ nbd_mark_nsock_dead(nbd, nsock, 1);
+ mutex_unlock(&nsock->tx_lock);
+ goto again;
+ }
out:
mutex_unlock(&nsock->tx_lock);
+ nbd_config_put(nbd);
return ret;
}
@@ -611,9 +826,10 @@
return ret;
}
-static int nbd_add_socket(struct nbd_device *nbd, struct block_device *bdev,
- unsigned long arg)
+static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
+ bool netlink)
{
+ struct nbd_config *config = nbd->config;
struct socket *sock;
struct nbd_sock **socks;
struct nbd_sock *nsock;
@@ -623,43 +839,107 @@
if (!sock)
return err;
- if (!nbd->task_setup)
+ if (!netlink && !nbd->task_setup &&
+ !test_bit(NBD_BOUND, &config->runtime_flags))
nbd->task_setup = current;
- if (nbd->task_setup != current) {
+
+ if (!netlink &&
+ (nbd->task_setup != current ||
+ test_bit(NBD_BOUND, &config->runtime_flags))) {
dev_err(disk_to_dev(nbd->disk),
"Device being setup by another task");
- return -EINVAL;
+ sockfd_put(sock);
+ return -EBUSY;
}
- socks = krealloc(nbd->socks, (nbd->num_connections + 1) *
+ socks = krealloc(config->socks, (config->num_connections + 1) *
sizeof(struct nbd_sock *), GFP_KERNEL);
- if (!socks)
+ if (!socks) {
+ sockfd_put(sock);
return -ENOMEM;
+ }
nsock = kzalloc(sizeof(struct nbd_sock), GFP_KERNEL);
- if (!nsock)
+ if (!nsock) {
+ sockfd_put(sock);
return -ENOMEM;
+ }
- nbd->socks = socks;
+ config->socks = socks;
+ nsock->fallback_index = -1;
+ nsock->dead = false;
mutex_init(&nsock->tx_lock);
nsock->sock = sock;
nsock->pending = NULL;
nsock->sent = 0;
- socks[nbd->num_connections++] = nsock;
+ nsock->cookie = 0;
+ socks[config->num_connections++] = nsock;
+ atomic_inc(&config->live_connections);
- if (max_part)
- bdev->bd_invalidated = 1;
return 0;
}
+static int nbd_reconnect_socket(struct nbd_device *nbd, unsigned long arg)
+{
+ struct nbd_config *config = nbd->config;
+ struct socket *sock, *old;
+ struct recv_thread_args *args;
+ int i;
+ int err;
+
+ sock = sockfd_lookup(arg, &err);
+ if (!sock)
+ return err;
+
+ args = kzalloc(sizeof(*args), GFP_KERNEL);
+ if (!args) {
+ sockfd_put(sock);
+ return -ENOMEM;
+ }
+
+ for (i = 0; i < config->num_connections; i++) {
+ struct nbd_sock *nsock = config->socks[i];
+
+ if (!nsock->dead)
+ continue;
+
+ mutex_lock(&nsock->tx_lock);
+ if (!nsock->dead) {
+ mutex_unlock(&nsock->tx_lock);
+ continue;
+ }
+ sk_set_memalloc(sock->sk);
+ atomic_inc(&config->recv_threads);
+ refcount_inc(&nbd->config_refs);
+ old = nsock->sock;
+ nsock->fallback_index = -1;
+ nsock->sock = sock;
+ nsock->dead = false;
+ INIT_WORK(&args->work, recv_work);
+ args->index = i;
+ args->nbd = nbd;
+ nsock->cookie++;
+ mutex_unlock(&nsock->tx_lock);
+ sockfd_put(old);
+
+ /* We take the tx_mutex in an error path in the recv_work, so we
+ * need to queue_work outside of the tx_mutex.
+ */
+ queue_work(recv_workqueue, &args->work);
+
+ atomic_inc(&config->live_connections);
+ wake_up(&config->conn_wait);
+ return 0;
+ }
+ sockfd_put(sock);
+ kfree(args);
+ return -ENOSPC;
+}
+
/* Reset all properties of an NBD device */
static void nbd_reset(struct nbd_device *nbd)
{
- nbd->runtime_flags = 0;
- nbd->blksize = 1024;
- nbd->bytesize = 0;
- set_capacity(nbd->disk, 0);
- nbd->flags = 0;
+ nbd->config = NULL;
nbd->tag_set.timeout = 0;
queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD, nbd->disk->queue);
}
@@ -668,21 +948,23 @@
{
if (bdev->bd_openers > 1)
return;
- set_device_ro(bdev, false);
- bdev->bd_inode->i_size = 0;
+ bd_set_size(bdev, 0);
if (max_part > 0) {
blkdev_reread_part(bdev);
bdev->bd_invalidated = 1;
}
}
-static void nbd_parse_flags(struct nbd_device *nbd, struct block_device *bdev)
+static void nbd_parse_flags(struct nbd_device *nbd)
{
- if (nbd->flags & NBD_FLAG_READ_ONLY)
- set_device_ro(bdev, true);
- if (nbd->flags & NBD_FLAG_SEND_TRIM)
+ struct nbd_config *config = nbd->config;
+ if (config->flags & NBD_FLAG_READ_ONLY)
+ set_disk_ro(nbd->disk, true);
+ else
+ set_disk_ro(nbd->disk, false);
+ if (config->flags & NBD_FLAG_SEND_TRIM)
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, nbd->disk->queue);
- if (nbd->flags & NBD_FLAG_SEND_FLUSH)
+ if (config->flags & NBD_FLAG_SEND_FLUSH)
blk_queue_write_cache(nbd->disk->queue, true, false);
else
blk_queue_write_cache(nbd->disk->queue, false, false);
@@ -690,6 +972,7 @@
static void send_disconnects(struct nbd_device *nbd)
{
+ struct nbd_config *config = nbd->config;
struct nbd_request request = {
.magic = htonl(NBD_REQUEST_MAGIC),
.type = htonl(NBD_CMD_DISC),
@@ -698,7 +981,7 @@
struct iov_iter from;
int i, ret;
- for (i = 0; i < nbd->num_connections; i++) {
+ for (i = 0; i < config->num_connections; i++) {
iov_iter_kvec(&from, WRITE | ITER_KVEC, &iov, 1, sizeof(request));
ret = sock_xmit(nbd, i, 1, &from, 0, NULL);
if (ret <= 0)
@@ -707,145 +990,162 @@
}
}
-static int nbd_disconnect(struct nbd_device *nbd, struct block_device *bdev)
+static int nbd_disconnect(struct nbd_device *nbd)
{
+ struct nbd_config *config = nbd->config;
+
dev_info(disk_to_dev(nbd->disk), "NBD_DISCONNECT\n");
- if (!nbd->socks)
- return -EINVAL;
-
- mutex_unlock(&nbd->config_lock);
- fsync_bdev(bdev);
- mutex_lock(&nbd->config_lock);
-
- /* Check again after getting mutex back. */
- if (!nbd->socks)
- return -EINVAL;
-
if (!test_and_set_bit(NBD_DISCONNECT_REQUESTED,
- &nbd->runtime_flags))
+ &config->runtime_flags))
send_disconnects(nbd);
return 0;
}
-static int nbd_clear_sock(struct nbd_device *nbd, struct block_device *bdev)
+static void nbd_clear_sock(struct nbd_device *nbd)
{
sock_shutdown(nbd);
nbd_clear_que(nbd);
-
- __invalidate_device(bdev, true);
- nbd_bdev_reset(bdev);
- /*
- * We want to give the run thread a chance to wait for everybody
- * to clean up and then do it's own cleanup.
- */
- if (!test_bit(NBD_RUNNING, &nbd->runtime_flags) &&
- nbd->num_connections) {
- int i;
-
- for (i = 0; i < nbd->num_connections; i++) {
- sockfd_put(nbd->socks[i]->sock);
- kfree(nbd->socks[i]);
- }
- kfree(nbd->socks);
- nbd->socks = NULL;
- nbd->num_connections = 0;
- }
nbd->task_setup = NULL;
-
- return 0;
}
-static int nbd_start_device(struct nbd_device *nbd, struct block_device *bdev)
+static void nbd_config_put(struct nbd_device *nbd)
{
- struct recv_thread_args *args;
- int num_connections = nbd->num_connections;
+ if (refcount_dec_and_mutex_lock(&nbd->config_refs,
+ &nbd->config_lock)) {
+ struct nbd_config *config = nbd->config;
+ nbd_dev_dbg_close(nbd);
+ nbd_size_clear(nbd);
+ if (test_and_clear_bit(NBD_HAS_PID_FILE,
+ &config->runtime_flags))
+ device_remove_file(disk_to_dev(nbd->disk), &pid_attr);
+ nbd->task_recv = NULL;
+ nbd_clear_sock(nbd);
+ if (config->num_connections) {
+ int i;
+ for (i = 0; i < config->num_connections; i++) {
+ sockfd_put(config->socks[i]->sock);
+ kfree(config->socks[i]);
+ }
+ kfree(config->socks);
+ }
+ nbd_reset(nbd);
+
+ mutex_unlock(&nbd->config_lock);
+ nbd_put(nbd);
+ module_put(THIS_MODULE);
+ }
+}
+
+static int nbd_start_device(struct nbd_device *nbd)
+{
+ struct nbd_config *config = nbd->config;
+ int num_connections = config->num_connections;
int error = 0, i;
if (nbd->task_recv)
return -EBUSY;
- if (!nbd->socks)
+ if (!config->socks)
return -EINVAL;
if (num_connections > 1 &&
- !(nbd->flags & NBD_FLAG_CAN_MULTI_CONN)) {
+ !(config->flags & NBD_FLAG_CAN_MULTI_CONN)) {
dev_err(disk_to_dev(nbd->disk), "server does not support multiple connections per device.\n");
- error = -EINVAL;
- goto out_err;
+ return -EINVAL;
}
- set_bit(NBD_RUNNING, &nbd->runtime_flags);
- blk_mq_update_nr_hw_queues(&nbd->tag_set, nbd->num_connections);
- args = kcalloc(num_connections, sizeof(*args), GFP_KERNEL);
- if (!args) {
- error = -ENOMEM;
- goto out_err;
- }
+ blk_mq_update_nr_hw_queues(&nbd->tag_set, config->num_connections);
nbd->task_recv = current;
- mutex_unlock(&nbd->config_lock);
- nbd_parse_flags(nbd, bdev);
+ nbd_parse_flags(nbd);
error = device_create_file(disk_to_dev(nbd->disk), &pid_attr);
if (error) {
dev_err(disk_to_dev(nbd->disk), "device_create_file failed!\n");
- goto out_recv;
+ return error;
}
-
- nbd_size_update(nbd, bdev);
+ set_bit(NBD_HAS_PID_FILE, &config->runtime_flags);
nbd_dev_dbg_init(nbd);
for (i = 0; i < num_connections; i++) {
- sk_set_memalloc(nbd->socks[i]->sock->sk);
- atomic_inc(&nbd->recv_threads);
- INIT_WORK(&args[i].work, recv_work);
- args[i].nbd = nbd;
- args[i].index = i;
- queue_work(recv_workqueue, &args[i].work);
+ struct recv_thread_args *args;
+
+ args = kzalloc(sizeof(*args), GFP_KERNEL);
+ if (!args) {
+ sock_shutdown(nbd);
+ return -ENOMEM;
+ }
+ sk_set_memalloc(config->socks[i]->sock->sk);
+ atomic_inc(&config->recv_threads);
+ refcount_inc(&nbd->config_refs);
+ INIT_WORK(&args->work, recv_work);
+ args->nbd = nbd;
+ args->index = i;
+ queue_work(recv_workqueue, &args->work);
}
- wait_event_interruptible(nbd->recv_wq,
- atomic_read(&nbd->recv_threads) == 0);
- for (i = 0; i < num_connections; i++)
- flush_work(&args[i].work);
- nbd_dev_dbg_close(nbd);
- nbd_size_clear(nbd, bdev);
- device_remove_file(disk_to_dev(nbd->disk), &pid_attr);
-out_recv:
- mutex_lock(&nbd->config_lock);
- nbd->task_recv = NULL;
-out_err:
- clear_bit(NBD_RUNNING, &nbd->runtime_flags);
- nbd_clear_sock(nbd, bdev);
-
- /* user requested, ignore socket errors */
- if (test_bit(NBD_DISCONNECT_REQUESTED, &nbd->runtime_flags))
- error = 0;
- if (test_bit(NBD_TIMEDOUT, &nbd->runtime_flags))
- error = -ETIMEDOUT;
-
- nbd_reset(nbd);
return error;
}
+static int nbd_start_device_ioctl(struct nbd_device *nbd, struct block_device *bdev)
+{
+ struct nbd_config *config = nbd->config;
+ int ret;
+
+ ret = nbd_start_device(nbd);
+ if (ret)
+ return ret;
+
+ bd_set_size(bdev, config->bytesize);
+ if (max_part)
+ bdev->bd_invalidated = 1;
+ mutex_unlock(&nbd->config_lock);
+ ret = wait_event_interruptible(config->recv_wq,
+ atomic_read(&config->recv_threads) == 0);
+ if (ret)
+ sock_shutdown(nbd);
+ mutex_lock(&nbd->config_lock);
+ bd_set_size(bdev, 0);
+ /* user requested, ignore socket errors */
+ if (test_bit(NBD_DISCONNECT_REQUESTED, &config->runtime_flags))
+ ret = 0;
+ if (test_bit(NBD_TIMEDOUT, &config->runtime_flags))
+ ret = -ETIMEDOUT;
+ return ret;
+}
+
+static void nbd_clear_sock_ioctl(struct nbd_device *nbd,
+ struct block_device *bdev)
+{
+ sock_shutdown(nbd);
+ kill_bdev(bdev);
+ nbd_bdev_reset(bdev);
+ if (test_and_clear_bit(NBD_HAS_CONFIG_REF,
+ &nbd->config->runtime_flags))
+ nbd_config_put(nbd);
+}
+
/* Must be called with config_lock held */
static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd,
unsigned int cmd, unsigned long arg)
{
+ struct nbd_config *config = nbd->config;
+
switch (cmd) {
case NBD_DISCONNECT:
- return nbd_disconnect(nbd, bdev);
+ return nbd_disconnect(nbd);
case NBD_CLEAR_SOCK:
- return nbd_clear_sock(nbd, bdev);
+ nbd_clear_sock_ioctl(nbd, bdev);
+ return 0;
case NBD_SET_SOCK:
- return nbd_add_socket(nbd, bdev, arg);
+ return nbd_add_socket(nbd, arg, false);
case NBD_SET_BLKSIZE:
- nbd_size_set(nbd, bdev, arg,
- div_s64(nbd->bytesize, arg));
+ nbd_size_set(nbd, arg,
+ div_s64(config->bytesize, arg));
return 0;
case NBD_SET_SIZE:
- nbd_size_set(nbd, bdev, nbd->blksize,
- div_s64(arg, nbd->blksize));
+ nbd_size_set(nbd, config->blksize,
+ div_s64(arg, config->blksize));
return 0;
case NBD_SET_SIZE_BLOCKS:
- nbd_size_set(nbd, bdev, nbd->blksize, arg);
+ nbd_size_set(nbd, config->blksize, arg);
return 0;
case NBD_SET_TIMEOUT:
if (arg) {
@@ -855,10 +1155,10 @@
return 0;
case NBD_SET_FLAGS:
- nbd->flags = arg;
+ config->flags = arg;
return 0;
case NBD_DO_IT:
- return nbd_start_device(nbd, bdev);
+ return nbd_start_device_ioctl(nbd, bdev);
case NBD_CLEAR_QUE:
/*
* This is for compatibility only. The queue is always cleared
@@ -879,23 +1179,92 @@
unsigned int cmd, unsigned long arg)
{
struct nbd_device *nbd = bdev->bd_disk->private_data;
- int error;
+ struct nbd_config *config = nbd->config;
+ int error = -EINVAL;
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
- BUG_ON(nbd->magic != NBD_MAGIC);
-
mutex_lock(&nbd->config_lock);
- error = __nbd_ioctl(bdev, nbd, cmd, arg);
- mutex_unlock(&nbd->config_lock);
+ /* Don't allow ioctl operations on a nbd device that was created with
+ * netlink, unless it's DISCONNECT or CLEAR_SOCK, which are fine.
+ */
+ if (!test_bit(NBD_BOUND, &config->runtime_flags) ||
+ (cmd == NBD_DISCONNECT || cmd == NBD_CLEAR_SOCK))
+ error = __nbd_ioctl(bdev, nbd, cmd, arg);
+ else
+ dev_err(nbd_to_dev(nbd), "Cannot use ioctl interface on a netlink controlled device.\n");
+ mutex_unlock(&nbd->config_lock);
return error;
}
+static struct nbd_config *nbd_alloc_config(void)
+{
+ struct nbd_config *config;
+
+ config = kzalloc(sizeof(struct nbd_config), GFP_NOFS);
+ if (!config)
+ return NULL;
+ atomic_set(&config->recv_threads, 0);
+ init_waitqueue_head(&config->recv_wq);
+ init_waitqueue_head(&config->conn_wait);
+ config->blksize = 1024;
+ atomic_set(&config->live_connections, 0);
+ try_module_get(THIS_MODULE);
+ return config;
+}
+
+static int nbd_open(struct block_device *bdev, fmode_t mode)
+{
+ struct nbd_device *nbd;
+ int ret = 0;
+
+ mutex_lock(&nbd_index_mutex);
+ nbd = bdev->bd_disk->private_data;
+ if (!nbd) {
+ ret = -ENXIO;
+ goto out;
+ }
+ if (!refcount_inc_not_zero(&nbd->refs)) {
+ ret = -ENXIO;
+ goto out;
+ }
+ if (!refcount_inc_not_zero(&nbd->config_refs)) {
+ struct nbd_config *config;
+
+ mutex_lock(&nbd->config_lock);
+ if (refcount_inc_not_zero(&nbd->config_refs)) {
+ mutex_unlock(&nbd->config_lock);
+ goto out;
+ }
+ config = nbd->config = nbd_alloc_config();
+ if (!config) {
+ ret = -ENOMEM;
+ mutex_unlock(&nbd->config_lock);
+ goto out;
+ }
+ refcount_set(&nbd->config_refs, 1);
+ refcount_inc(&nbd->refs);
+ mutex_unlock(&nbd->config_lock);
+ }
+out:
+ mutex_unlock(&nbd_index_mutex);
+ return ret;
+}
+
+static void nbd_release(struct gendisk *disk, fmode_t mode)
+{
+ struct nbd_device *nbd = disk->private_data;
+ nbd_config_put(nbd);
+ nbd_put(nbd);
+}
+
static const struct block_device_operations nbd_fops =
{
.owner = THIS_MODULE,
+ .open = nbd_open,
+ .release = nbd_release,
.ioctl = nbd_ioctl,
.compat_ioctl = nbd_ioctl,
};
@@ -927,7 +1296,7 @@
static int nbd_dbg_flags_show(struct seq_file *s, void *unused)
{
struct nbd_device *nbd = s->private;
- u32 flags = nbd->flags;
+ u32 flags = nbd->config->flags;
seq_printf(s, "Hex: 0x%08x\n\n", flags);
@@ -960,6 +1329,7 @@
static int nbd_dev_dbg_init(struct nbd_device *nbd)
{
struct dentry *dir;
+ struct nbd_config *config = nbd->config;
if (!nbd_dbg_dir)
return -EIO;
@@ -970,12 +1340,12 @@
nbd_name(nbd));
return -EIO;
}
- nbd->dbg_dir = dir;
+ config->dbg_dir = dir;
debugfs_create_file("tasks", 0444, dir, nbd, &nbd_dbg_tasks_ops);
- debugfs_create_u64("size_bytes", 0444, dir, &nbd->bytesize);
+ debugfs_create_u64("size_bytes", 0444, dir, &config->bytesize);
debugfs_create_u32("timeout", 0444, dir, &nbd->tag_set.timeout);
- debugfs_create_u64("blocksize", 0444, dir, &nbd->blksize);
+ debugfs_create_u64("blocksize", 0444, dir, &config->blksize);
debugfs_create_file("flags", 0444, dir, nbd, &nbd_dbg_flags_ops);
return 0;
@@ -983,7 +1353,7 @@
static void nbd_dev_dbg_close(struct nbd_device *nbd)
{
- debugfs_remove_recursive(nbd->dbg_dir);
+ debugfs_remove_recursive(nbd->config->dbg_dir);
}
static int nbd_dbg_init(void)
@@ -1035,25 +1405,13 @@
return 0;
}
-static struct blk_mq_ops nbd_mq_ops = {
+static const struct blk_mq_ops nbd_mq_ops = {
.queue_rq = nbd_queue_rq,
+ .complete = nbd_complete_rq,
.init_request = nbd_init_request,
.timeout = nbd_xmit_timeout,
};
-static void nbd_dev_remove(struct nbd_device *nbd)
-{
- struct gendisk *disk = nbd->disk;
- nbd->magic = 0;
- if (disk) {
- del_gendisk(disk);
- blk_cleanup_queue(disk->queue);
- blk_mq_free_tag_set(&nbd->tag_set);
- put_disk(disk);
- }
- kfree(nbd);
-}
-
static int nbd_dev_add(int index)
{
struct nbd_device *nbd;
@@ -1082,6 +1440,7 @@
if (err < 0)
goto out_free_disk;
+ nbd->index = index;
nbd->disk = disk;
nbd->tag_set.ops = &nbd_mq_ops;
nbd->tag_set.nr_hw_queues = 1;
@@ -1110,20 +1469,23 @@
queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, disk->queue);
disk->queue->limits.discard_granularity = 512;
blk_queue_max_discard_sectors(disk->queue, UINT_MAX);
- disk->queue->limits.discard_zeroes_data = 0;
+ blk_queue_max_segment_size(disk->queue, UINT_MAX);
+ blk_queue_max_segments(disk->queue, USHRT_MAX);
blk_queue_max_hw_sectors(disk->queue, 65536);
disk->queue->limits.max_sectors = 256;
- nbd->magic = NBD_MAGIC;
mutex_init(&nbd->config_lock);
+ refcount_set(&nbd->config_refs, 0);
+ refcount_set(&nbd->refs, 1);
+ INIT_LIST_HEAD(&nbd->list);
disk->major = NBD_MAJOR;
disk->first_minor = index << part_shift;
disk->fops = &nbd_fops;
disk->private_data = nbd;
sprintf(disk->disk_name, "nbd%d", index);
- init_waitqueue_head(&nbd->recv_wq);
nbd_reset(nbd);
add_disk(disk);
+ nbd_total_devices++;
return index;
out_free_tags:
@@ -1138,10 +1500,535 @@
return err;
}
-/*
- * And here should be modules and kernel interface
- * (Just smiley confuses emacs :-)
+static int find_free_cb(int id, void *ptr, void *data)
+{
+ struct nbd_device *nbd = ptr;
+ struct nbd_device **found = data;
+
+ if (!refcount_read(&nbd->config_refs)) {
+ *found = nbd;
+ return 1;
+ }
+ return 0;
+}
+
+/* Netlink interface. */
+static struct nla_policy nbd_attr_policy[NBD_ATTR_MAX + 1] = {
+ [NBD_ATTR_INDEX] = { .type = NLA_U32 },
+ [NBD_ATTR_SIZE_BYTES] = { .type = NLA_U64 },
+ [NBD_ATTR_BLOCK_SIZE_BYTES] = { .type = NLA_U64 },
+ [NBD_ATTR_TIMEOUT] = { .type = NLA_U64 },
+ [NBD_ATTR_SERVER_FLAGS] = { .type = NLA_U64 },
+ [NBD_ATTR_CLIENT_FLAGS] = { .type = NLA_U64 },
+ [NBD_ATTR_SOCKETS] = { .type = NLA_NESTED},
+ [NBD_ATTR_DEAD_CONN_TIMEOUT] = { .type = NLA_U64 },
+ [NBD_ATTR_DEVICE_LIST] = { .type = NLA_NESTED},
+};
+
+static struct nla_policy nbd_sock_policy[NBD_SOCK_MAX + 1] = {
+ [NBD_SOCK_FD] = { .type = NLA_U32 },
+};
+
+/* We don't use this right now since we don't parse the incoming list, but we
+ * still want it here so userspace knows what to expect.
*/
+static struct nla_policy __attribute__((unused))
+nbd_device_policy[NBD_DEVICE_ATTR_MAX + 1] = {
+ [NBD_DEVICE_INDEX] = { .type = NLA_U32 },
+ [NBD_DEVICE_CONNECTED] = { .type = NLA_U8 },
+};
+
+static int nbd_genl_connect(struct sk_buff *skb, struct genl_info *info)
+{
+ struct nbd_device *nbd = NULL;
+ struct nbd_config *config;
+ int index = -1;
+ int ret;
+ bool put_dev = false;
+
+ if (!netlink_capable(skb, CAP_SYS_ADMIN))
+ return -EPERM;
+
+ if (info->attrs[NBD_ATTR_INDEX])
+ index = nla_get_u32(info->attrs[NBD_ATTR_INDEX]);
+ if (!info->attrs[NBD_ATTR_SOCKETS]) {
+ printk(KERN_ERR "nbd: must specify at least one socket\n");
+ return -EINVAL;
+ }
+ if (!info->attrs[NBD_ATTR_SIZE_BYTES]) {
+ printk(KERN_ERR "nbd: must specify a size in bytes for the device\n");
+ return -EINVAL;
+ }
+again:
+ mutex_lock(&nbd_index_mutex);
+ if (index == -1) {
+ ret = idr_for_each(&nbd_index_idr, &find_free_cb, &nbd);
+ if (ret == 0) {
+ int new_index;
+ new_index = nbd_dev_add(-1);
+ if (new_index < 0) {
+ mutex_unlock(&nbd_index_mutex);
+ printk(KERN_ERR "nbd: failed to add new device\n");
+ return ret;
+ }
+ nbd = idr_find(&nbd_index_idr, new_index);
+ }
+ } else {
+ nbd = idr_find(&nbd_index_idr, index);
+ }
+ if (!nbd) {
+ printk(KERN_ERR "nbd: couldn't find device at index %d\n",
+ index);
+ mutex_unlock(&nbd_index_mutex);
+ return -EINVAL;
+ }
+ if (!refcount_inc_not_zero(&nbd->refs)) {
+ mutex_unlock(&nbd_index_mutex);
+ if (index == -1)
+ goto again;
+ printk(KERN_ERR "nbd: device at index %d is going down\n",
+ index);
+ return -EINVAL;
+ }
+ mutex_unlock(&nbd_index_mutex);
+
+ mutex_lock(&nbd->config_lock);
+ if (refcount_read(&nbd->config_refs)) {
+ mutex_unlock(&nbd->config_lock);
+ nbd_put(nbd);
+ if (index == -1)
+ goto again;
+ printk(KERN_ERR "nbd: nbd%d already in use\n", index);
+ return -EBUSY;
+ }
+ if (WARN_ON(nbd->config)) {
+ mutex_unlock(&nbd->config_lock);
+ nbd_put(nbd);
+ return -EINVAL;
+ }
+ config = nbd->config = nbd_alloc_config();
+ if (!nbd->config) {
+ mutex_unlock(&nbd->config_lock);
+ nbd_put(nbd);
+ printk(KERN_ERR "nbd: couldn't allocate config\n");
+ return -ENOMEM;
+ }
+ refcount_set(&nbd->config_refs, 1);
+ set_bit(NBD_BOUND, &config->runtime_flags);
+
+ if (info->attrs[NBD_ATTR_SIZE_BYTES]) {
+ u64 bytes = nla_get_u64(info->attrs[NBD_ATTR_SIZE_BYTES]);
+ nbd_size_set(nbd, config->blksize,
+ div64_u64(bytes, config->blksize));
+ }
+ if (info->attrs[NBD_ATTR_BLOCK_SIZE_BYTES]) {
+ u64 bsize =
+ nla_get_u64(info->attrs[NBD_ATTR_BLOCK_SIZE_BYTES]);
+ nbd_size_set(nbd, bsize, div64_u64(config->bytesize, bsize));
+ }
+ if (info->attrs[NBD_ATTR_TIMEOUT]) {
+ u64 timeout = nla_get_u64(info->attrs[NBD_ATTR_TIMEOUT]);
+ nbd->tag_set.timeout = timeout * HZ;
+ blk_queue_rq_timeout(nbd->disk->queue, timeout * HZ);
+ }
+ if (info->attrs[NBD_ATTR_DEAD_CONN_TIMEOUT]) {
+ config->dead_conn_timeout =
+ nla_get_u64(info->attrs[NBD_ATTR_DEAD_CONN_TIMEOUT]);
+ config->dead_conn_timeout *= HZ;
+ }
+ if (info->attrs[NBD_ATTR_SERVER_FLAGS])
+ config->flags =
+ nla_get_u64(info->attrs[NBD_ATTR_SERVER_FLAGS]);
+ if (info->attrs[NBD_ATTR_CLIENT_FLAGS]) {
+ u64 flags = nla_get_u64(info->attrs[NBD_ATTR_CLIENT_FLAGS]);
+ if (flags & NBD_CFLAG_DESTROY_ON_DISCONNECT) {
+ set_bit(NBD_DESTROY_ON_DISCONNECT,
+ &config->runtime_flags);
+ put_dev = true;
+ }
+ }
+
+ if (info->attrs[NBD_ATTR_SOCKETS]) {
+ struct nlattr *attr;
+ int rem, fd;
+
+ nla_for_each_nested(attr, info->attrs[NBD_ATTR_SOCKETS],
+ rem) {
+ struct nlattr *socks[NBD_SOCK_MAX+1];
+
+ if (nla_type(attr) != NBD_SOCK_ITEM) {
+ printk(KERN_ERR "nbd: socks must be embedded in a SOCK_ITEM attr\n");
+ ret = -EINVAL;
+ goto out;
+ }
+ ret = nla_parse_nested(socks, NBD_SOCK_MAX, attr,
+ nbd_sock_policy, info->extack);
+ if (ret != 0) {
+ printk(KERN_ERR "nbd: error processing sock list\n");
+ ret = -EINVAL;
+ goto out;
+ }
+ if (!socks[NBD_SOCK_FD])
+ continue;
+ fd = (int)nla_get_u32(socks[NBD_SOCK_FD]);
+ ret = nbd_add_socket(nbd, fd, true);
+ if (ret)
+ goto out;
+ }
+ }
+ ret = nbd_start_device(nbd);
+out:
+ mutex_unlock(&nbd->config_lock);
+ if (!ret) {
+ set_bit(NBD_HAS_CONFIG_REF, &config->runtime_flags);
+ refcount_inc(&nbd->config_refs);
+ nbd_connect_reply(info, nbd->index);
+ }
+ nbd_config_put(nbd);
+ if (put_dev)
+ nbd_put(nbd);
+ return ret;
+}
+
+static int nbd_genl_disconnect(struct sk_buff *skb, struct genl_info *info)
+{
+ struct nbd_device *nbd;
+ int index;
+
+ if (!netlink_capable(skb, CAP_SYS_ADMIN))
+ return -EPERM;
+
+ if (!info->attrs[NBD_ATTR_INDEX]) {
+ printk(KERN_ERR "nbd: must specify an index to disconnect\n");
+ return -EINVAL;
+ }
+ index = nla_get_u32(info->attrs[NBD_ATTR_INDEX]);
+ mutex_lock(&nbd_index_mutex);
+ nbd = idr_find(&nbd_index_idr, index);
+ if (!nbd) {
+ mutex_unlock(&nbd_index_mutex);
+ printk(KERN_ERR "nbd: couldn't find device at index %d\n",
+ index);
+ return -EINVAL;
+ }
+ if (!refcount_inc_not_zero(&nbd->refs)) {
+ mutex_unlock(&nbd_index_mutex);
+ printk(KERN_ERR "nbd: device at index %d is going down\n",
+ index);
+ return -EINVAL;
+ }
+ mutex_unlock(&nbd_index_mutex);
+ if (!refcount_inc_not_zero(&nbd->config_refs)) {
+ nbd_put(nbd);
+ return 0;
+ }
+ mutex_lock(&nbd->config_lock);
+ nbd_disconnect(nbd);
+ mutex_unlock(&nbd->config_lock);
+ if (test_and_clear_bit(NBD_HAS_CONFIG_REF,
+ &nbd->config->runtime_flags))
+ nbd_config_put(nbd);
+ nbd_config_put(nbd);
+ nbd_put(nbd);
+ return 0;
+}
+
+static int nbd_genl_reconfigure(struct sk_buff *skb, struct genl_info *info)
+{
+ struct nbd_device *nbd = NULL;
+ struct nbd_config *config;
+ int index;
+ int ret = -EINVAL;
+ bool put_dev = false;
+
+ if (!netlink_capable(skb, CAP_SYS_ADMIN))
+ return -EPERM;
+
+ if (!info->attrs[NBD_ATTR_INDEX]) {
+ printk(KERN_ERR "nbd: must specify a device to reconfigure\n");
+ return -EINVAL;
+ }
+ index = nla_get_u32(info->attrs[NBD_ATTR_INDEX]);
+ mutex_lock(&nbd_index_mutex);
+ nbd = idr_find(&nbd_index_idr, index);
+ if (!nbd) {
+ mutex_unlock(&nbd_index_mutex);
+ printk(KERN_ERR "nbd: couldn't find a device at index %d\n",
+ index);
+ return -EINVAL;
+ }
+ if (!refcount_inc_not_zero(&nbd->refs)) {
+ mutex_unlock(&nbd_index_mutex);
+ printk(KERN_ERR "nbd: device at index %d is going down\n",
+ index);
+ return -EINVAL;
+ }
+ mutex_unlock(&nbd_index_mutex);
+
+ if (!refcount_inc_not_zero(&nbd->config_refs)) {
+ dev_err(nbd_to_dev(nbd),
+ "not configured, cannot reconfigure\n");
+ nbd_put(nbd);
+ return -EINVAL;
+ }
+
+ mutex_lock(&nbd->config_lock);
+ config = nbd->config;
+ if (!test_bit(NBD_BOUND, &config->runtime_flags) ||
+ !nbd->task_recv) {
+ dev_err(nbd_to_dev(nbd),
+ "not configured, cannot reconfigure\n");
+ goto out;
+ }
+
+ if (info->attrs[NBD_ATTR_TIMEOUT]) {
+ u64 timeout = nla_get_u64(info->attrs[NBD_ATTR_TIMEOUT]);
+ nbd->tag_set.timeout = timeout * HZ;
+ blk_queue_rq_timeout(nbd->disk->queue, timeout * HZ);
+ }
+ if (info->attrs[NBD_ATTR_DEAD_CONN_TIMEOUT]) {
+ config->dead_conn_timeout =
+ nla_get_u64(info->attrs[NBD_ATTR_DEAD_CONN_TIMEOUT]);
+ config->dead_conn_timeout *= HZ;
+ }
+ if (info->attrs[NBD_ATTR_CLIENT_FLAGS]) {
+ u64 flags = nla_get_u64(info->attrs[NBD_ATTR_CLIENT_FLAGS]);
+ if (flags & NBD_CFLAG_DESTROY_ON_DISCONNECT) {
+ if (!test_and_set_bit(NBD_DESTROY_ON_DISCONNECT,
+ &config->runtime_flags))
+ put_dev = true;
+ } else {
+ if (test_and_clear_bit(NBD_DESTROY_ON_DISCONNECT,
+ &config->runtime_flags))
+ refcount_inc(&nbd->refs);
+ }
+ }
+
+ if (info->attrs[NBD_ATTR_SOCKETS]) {
+ struct nlattr *attr;
+ int rem, fd;
+
+ nla_for_each_nested(attr, info->attrs[NBD_ATTR_SOCKETS],
+ rem) {
+ struct nlattr *socks[NBD_SOCK_MAX+1];
+
+ if (nla_type(attr) != NBD_SOCK_ITEM) {
+ printk(KERN_ERR "nbd: socks must be embedded in a SOCK_ITEM attr\n");
+ ret = -EINVAL;
+ goto out;
+ }
+ ret = nla_parse_nested(socks, NBD_SOCK_MAX, attr,
+ nbd_sock_policy, info->extack);
+ if (ret != 0) {
+ printk(KERN_ERR "nbd: error processing sock list\n");
+ ret = -EINVAL;
+ goto out;
+ }
+ if (!socks[NBD_SOCK_FD])
+ continue;
+ fd = (int)nla_get_u32(socks[NBD_SOCK_FD]);
+ ret = nbd_reconnect_socket(nbd, fd);
+ if (ret) {
+ if (ret == -ENOSPC)
+ ret = 0;
+ goto out;
+ }
+ dev_info(nbd_to_dev(nbd), "reconnected socket\n");
+ }
+ }
+out:
+ mutex_unlock(&nbd->config_lock);
+ nbd_config_put(nbd);
+ nbd_put(nbd);
+ if (put_dev)
+ nbd_put(nbd);
+ return ret;
+}
+
+static const struct genl_ops nbd_connect_genl_ops[] = {
+ {
+ .cmd = NBD_CMD_CONNECT,
+ .policy = nbd_attr_policy,
+ .doit = nbd_genl_connect,
+ },
+ {
+ .cmd = NBD_CMD_DISCONNECT,
+ .policy = nbd_attr_policy,
+ .doit = nbd_genl_disconnect,
+ },
+ {
+ .cmd = NBD_CMD_RECONFIGURE,
+ .policy = nbd_attr_policy,
+ .doit = nbd_genl_reconfigure,
+ },
+ {
+ .cmd = NBD_CMD_STATUS,
+ .policy = nbd_attr_policy,
+ .doit = nbd_genl_status,
+ },
+};
+
+static const struct genl_multicast_group nbd_mcast_grps[] = {
+ { .name = NBD_GENL_MCAST_GROUP_NAME, },
+};
+
+static struct genl_family nbd_genl_family __ro_after_init = {
+ .hdrsize = 0,
+ .name = NBD_GENL_FAMILY_NAME,
+ .version = NBD_GENL_VERSION,
+ .module = THIS_MODULE,
+ .ops = nbd_connect_genl_ops,
+ .n_ops = ARRAY_SIZE(nbd_connect_genl_ops),
+ .maxattr = NBD_ATTR_MAX,
+ .mcgrps = nbd_mcast_grps,
+ .n_mcgrps = ARRAY_SIZE(nbd_mcast_grps),
+};
+
+static int populate_nbd_status(struct nbd_device *nbd, struct sk_buff *reply)
+{
+ struct nlattr *dev_opt;
+ u8 connected = 0;
+ int ret;
+
+ /* This is a little racey, but for status it's ok. The
+ * reason we don't take a ref here is because we can't
+ * take a ref in the index == -1 case as we would need
+ * to put under the nbd_index_mutex, which could
+ * deadlock if we are configured to remove ourselves
+ * once we're disconnected.
+ */
+ if (refcount_read(&nbd->config_refs))
+ connected = 1;
+ dev_opt = nla_nest_start(reply, NBD_DEVICE_ITEM);
+ if (!dev_opt)
+ return -EMSGSIZE;
+ ret = nla_put_u32(reply, NBD_DEVICE_INDEX, nbd->index);
+ if (ret)
+ return -EMSGSIZE;
+ ret = nla_put_u8(reply, NBD_DEVICE_CONNECTED,
+ connected);
+ if (ret)
+ return -EMSGSIZE;
+ nla_nest_end(reply, dev_opt);
+ return 0;
+}
+
+static int status_cb(int id, void *ptr, void *data)
+{
+ struct nbd_device *nbd = ptr;
+ return populate_nbd_status(nbd, (struct sk_buff *)data);
+}
+
+static int nbd_genl_status(struct sk_buff *skb, struct genl_info *info)
+{
+ struct nlattr *dev_list;
+ struct sk_buff *reply;
+ void *reply_head;
+ size_t msg_size;
+ int index = -1;
+ int ret = -ENOMEM;
+
+ if (info->attrs[NBD_ATTR_INDEX])
+ index = nla_get_u32(info->attrs[NBD_ATTR_INDEX]);
+
+ mutex_lock(&nbd_index_mutex);
+
+ msg_size = nla_total_size(nla_attr_size(sizeof(u32)) +
+ nla_attr_size(sizeof(u8)));
+ msg_size *= (index == -1) ? nbd_total_devices : 1;
+
+ reply = genlmsg_new(msg_size, GFP_KERNEL);
+ if (!reply)
+ goto out;
+ reply_head = genlmsg_put_reply(reply, info, &nbd_genl_family, 0,
+ NBD_CMD_STATUS);
+ if (!reply_head) {
+ nlmsg_free(reply);
+ goto out;
+ }
+
+ dev_list = nla_nest_start(reply, NBD_ATTR_DEVICE_LIST);
+ if (index == -1) {
+ ret = idr_for_each(&nbd_index_idr, &status_cb, reply);
+ if (ret) {
+ nlmsg_free(reply);
+ goto out;
+ }
+ } else {
+ struct nbd_device *nbd;
+ nbd = idr_find(&nbd_index_idr, index);
+ if (nbd) {
+ ret = populate_nbd_status(nbd, reply);
+ if (ret) {
+ nlmsg_free(reply);
+ goto out;
+ }
+ }
+ }
+ nla_nest_end(reply, dev_list);
+ genlmsg_end(reply, reply_head);
+ genlmsg_reply(reply, info);
+ ret = 0;
+out:
+ mutex_unlock(&nbd_index_mutex);
+ return ret;
+}
+
+static void nbd_connect_reply(struct genl_info *info, int index)
+{
+ struct sk_buff *skb;
+ void *msg_head;
+ int ret;
+
+ skb = genlmsg_new(nla_total_size(sizeof(u32)), GFP_KERNEL);
+ if (!skb)
+ return;
+ msg_head = genlmsg_put_reply(skb, info, &nbd_genl_family, 0,
+ NBD_CMD_CONNECT);
+ if (!msg_head) {
+ nlmsg_free(skb);
+ return;
+ }
+ ret = nla_put_u32(skb, NBD_ATTR_INDEX, index);
+ if (ret) {
+ nlmsg_free(skb);
+ return;
+ }
+ genlmsg_end(skb, msg_head);
+ genlmsg_reply(skb, info);
+}
+
+static void nbd_mcast_index(int index)
+{
+ struct sk_buff *skb;
+ void *msg_head;
+ int ret;
+
+ skb = genlmsg_new(nla_total_size(sizeof(u32)), GFP_KERNEL);
+ if (!skb)
+ return;
+ msg_head = genlmsg_put(skb, 0, 0, &nbd_genl_family, 0,
+ NBD_CMD_LINK_DEAD);
+ if (!msg_head) {
+ nlmsg_free(skb);
+ return;
+ }
+ ret = nla_put_u32(skb, NBD_ATTR_INDEX, index);
+ if (ret) {
+ nlmsg_free(skb);
+ return;
+ }
+ genlmsg_end(skb, msg_head);
+ genlmsg_multicast(&nbd_genl_family, skb, 0, 0, GFP_KERNEL);
+}
+
+static void nbd_dead_link_work(struct work_struct *work)
+{
+ struct link_dead_args *args = container_of(work, struct link_dead_args,
+ work);
+ nbd_mcast_index(args->index);
+ kfree(args);
+}
static int __init nbd_init(void)
{
@@ -1184,6 +2071,11 @@
return -EIO;
}
+ if (genl_register_family(&nbd_genl_family)) {
+ unregister_blkdev(NBD_MAJOR, "nbd");
+ destroy_workqueue(recv_workqueue);
+ return -EINVAL;
+ }
nbd_dbg_init();
mutex_lock(&nbd_index_mutex);
@@ -1195,17 +2087,34 @@
static int nbd_exit_cb(int id, void *ptr, void *data)
{
+ struct list_head *list = (struct list_head *)data;
struct nbd_device *nbd = ptr;
- nbd_dev_remove(nbd);
+
+ list_add_tail(&nbd->list, list);
return 0;
}
static void __exit nbd_cleanup(void)
{
+ struct nbd_device *nbd;
+ LIST_HEAD(del_list);
+
nbd_dbg_close();
- idr_for_each(&nbd_index_idr, &nbd_exit_cb, NULL);
+ mutex_lock(&nbd_index_mutex);
+ idr_for_each(&nbd_index_idr, &nbd_exit_cb, &del_list);
+ mutex_unlock(&nbd_index_mutex);
+
+ while (!list_empty(&del_list)) {
+ nbd = list_first_entry(&del_list, struct nbd_device, list);
+ list_del_init(&nbd->list);
+ if (refcount_read(&nbd->refs) != 1)
+ printk(KERN_ERR "nbd: possibly leaking a device\n");
+ nbd_put(nbd);
+ }
+
idr_destroy(&nbd_index_idr);
+ genl_unregister_family(&nbd_genl_family);
destroy_workqueue(recv_workqueue);
unregister_blkdev(NBD_MAJOR, "nbd");
}
diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c
index 6f2e565..d946e1e 100644
--- a/drivers/block/null_blk.c
+++ b/drivers/block/null_blk.c
@@ -117,6 +117,10 @@
module_param(use_lightnvm, bool, S_IRUGO);
MODULE_PARM_DESC(use_lightnvm, "Register as a LightNVM device");
+static bool blocking;
+module_param(blocking, bool, S_IRUGO);
+MODULE_PARM_DESC(blocking, "Register as a blocking blk-mq driver device");
+
static int irqmode = NULL_IRQ_SOFTIRQ;
static int null_set_irqmode(const char *str, const struct kernel_param *kp)
@@ -277,7 +281,7 @@
case NULL_IRQ_SOFTIRQ:
switch (queue_mode) {
case NULL_Q_MQ:
- blk_mq_complete_request(cmd->rq, cmd->rq->errors);
+ blk_mq_complete_request(cmd->rq);
break;
case NULL_Q_RQ:
blk_complete_request(cmd->rq);
@@ -357,6 +361,8 @@
{
struct nullb_cmd *cmd = blk_mq_rq_to_pdu(bd->rq);
+ might_sleep_if(hctx->flags & BLK_MQ_F_BLOCKING);
+
if (irqmode == NULL_IRQ_TIMER) {
hrtimer_init(&cmd->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
cmd->timer.function = null_cmd_timer_expired;
@@ -392,7 +398,7 @@
return 0;
}
-static struct blk_mq_ops null_mq_ops = {
+static const struct blk_mq_ops null_mq_ops = {
.queue_rq = null_queue_rq,
.init_hctx = null_init_hctx,
.complete = null_softirq_done_fn,
@@ -437,14 +443,7 @@
if (IS_ERR(rq))
return -ENOMEM;
- rq->__sector = bio->bi_iter.bi_sector;
- rq->ioprio = bio_prio(bio);
-
- if (bio_has_data(bio))
- rq->nr_phys_segments = bio_phys_segments(q, bio);
-
- rq->__data_len = bio->bi_iter.bi_size;
- rq->bio = rq->biotail = bio;
+ blk_init_request_from_bio(rq, bio);
rq->end_io_data = rqd;
@@ -724,6 +723,9 @@
nullb->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
nullb->tag_set.driver_data = nullb;
+ if (blocking)
+ nullb->tag_set.flags |= BLK_MQ_F_BLOCKING;
+
rv = blk_mq_alloc_tag_set(&nullb->tag_set);
if (rv)
goto out_cleanup_queues;
diff --git a/drivers/block/osdblk.c b/drivers/block/osdblk.c
deleted file mode 100644
index 8127b82..0000000
--- a/drivers/block/osdblk.c
+++ /dev/null
@@ -1,693 +0,0 @@
-
-/*
- osdblk.c -- Export a single SCSI OSD object as a Linux block device
-
-
- Copyright 2009 Red Hat, Inc.
-
- This program is free software; you can redistribute it and/or modify
- it under the terms of the GNU General Public License as published by
- the Free Software Foundation.
-
- This program is distributed in the hope that it will be useful,
- but WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- GNU General Public License for more details.
-
- You should have received a copy of the GNU General Public License
- along with this program; see the file COPYING. If not, write to
- the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.
-
-
- Instructions for use
- --------------------
-
- 1) Map a Linux block device to an existing OSD object.
-
- In this example, we will use partition id 1234, object id 5678,
- OSD device /dev/osd1.
-
- $ echo "1234 5678 /dev/osd1" > /sys/class/osdblk/add
-
-
- 2) List all active blkdev<->object mappings.
-
- In this example, we have performed step #1 twice, creating two blkdevs,
- mapped to two separate OSD objects.
-
- $ cat /sys/class/osdblk/list
- 0 174 1234 5678 /dev/osd1
- 1 179 1994 897123 /dev/osd0
-
- The columns, in order, are:
- - blkdev unique id
- - blkdev assigned major
- - OSD object partition id
- - OSD object id
- - OSD device
-
-
- 3) Remove an active blkdev<->object mapping.
-
- In this example, we remove the mapping with blkdev unique id 1.
-
- $ echo 1 > /sys/class/osdblk/remove
-
-
- NOTE: The actual creation and deletion of OSD objects is outside the scope
- of this driver.
-
- */
-
-#include <linux/kernel.h>
-#include <linux/device.h>
-#include <linux/module.h>
-#include <linux/fs.h>
-#include <linux/slab.h>
-#include <scsi/osd_initiator.h>
-#include <scsi/osd_attributes.h>
-#include <scsi/osd_sec.h>
-#include <scsi/scsi_device.h>
-
-#define DRV_NAME "osdblk"
-#define PFX DRV_NAME ": "
-
-/* #define _OSDBLK_DEBUG */
-#ifdef _OSDBLK_DEBUG
-#define OSDBLK_DEBUG(fmt, a...) \
- printk(KERN_NOTICE "osdblk @%s:%d: " fmt, __func__, __LINE__, ##a)
-#else
-#define OSDBLK_DEBUG(fmt, a...) \
- do { if (0) printk(fmt, ##a); } while (0)
-#endif
-
-MODULE_AUTHOR("Jeff Garzik <jeff@garzik.org>");
-MODULE_DESCRIPTION("block device inside an OSD object osdblk.ko");
-MODULE_LICENSE("GPL");
-
-struct osdblk_device;
-
-enum {
- OSDBLK_MINORS_PER_MAJOR = 256, /* max minors per blkdev */
- OSDBLK_MAX_REQ = 32, /* max parallel requests */
- OSDBLK_OP_TIMEOUT = 4 * 60, /* sync OSD req timeout */
-};
-
-struct osdblk_request {
- struct request *rq; /* blk layer request */
- struct bio *bio; /* cloned bio */
- struct osdblk_device *osdev; /* associated blkdev */
-};
-
-struct osdblk_device {
- int id; /* blkdev unique id */
-
- int major; /* blkdev assigned major */
- struct gendisk *disk; /* blkdev's gendisk and rq */
- struct request_queue *q;
-
- struct osd_dev *osd; /* associated OSD */
-
- char name[32]; /* blkdev name, e.g. osdblk34 */
-
- spinlock_t lock; /* queue lock */
-
- struct osd_obj_id obj; /* OSD partition, obj id */
- uint8_t obj_cred[OSD_CAP_LEN]; /* OSD cred */
-
- struct osdblk_request req[OSDBLK_MAX_REQ]; /* request table */
-
- struct list_head node;
-
- char osd_path[0]; /* OSD device path */
-};
-
-static struct class *class_osdblk; /* /sys/class/osdblk */
-static DEFINE_MUTEX(ctl_mutex); /* Serialize open/close/setup/teardown */
-static LIST_HEAD(osdblkdev_list);
-
-static const struct block_device_operations osdblk_bd_ops = {
- .owner = THIS_MODULE,
-};
-
-static const struct osd_attr g_attr_logical_length = ATTR_DEF(
- OSD_APAGE_OBJECT_INFORMATION, OSD_ATTR_OI_LOGICAL_LENGTH, 8);
-
-static void osdblk_make_credential(u8 cred_a[OSD_CAP_LEN],
- const struct osd_obj_id *obj)
-{
- osd_sec_init_nosec_doall_caps(cred_a, obj, false, true);
-}
-
-/* copied from exofs; move to libosd? */
-/*
- * Perform a synchronous OSD operation. copied from exofs; move to libosd?
- */
-static int osd_sync_op(struct osd_request *or, int timeout, uint8_t *credential)
-{
- int ret;
-
- or->timeout = timeout;
- ret = osd_finalize_request(or, 0, credential, NULL);
- if (ret)
- return ret;
-
- ret = osd_execute_request(or);
-
- /* osd_req_decode_sense(or, ret); */
- return ret;
-}
-
-/*
- * Perform an asynchronous OSD operation. copied from exofs; move to libosd?
- */
-static int osd_async_op(struct osd_request *or, osd_req_done_fn *async_done,
- void *caller_context, u8 *cred)
-{
- int ret;
-
- ret = osd_finalize_request(or, 0, cred, NULL);
- if (ret)
- return ret;
-
- ret = osd_execute_request_async(or, async_done, caller_context);
-
- return ret;
-}
-
-/* copied from exofs; move to libosd? */
-static int extract_attr_from_req(struct osd_request *or, struct osd_attr *attr)
-{
- struct osd_attr cur_attr = {.attr_page = 0}; /* start with zeros */
- void *iter = NULL;
- int nelem;
-
- do {
- nelem = 1;
- osd_req_decode_get_attr_list(or, &cur_attr, &nelem, &iter);
- if ((cur_attr.attr_page == attr->attr_page) &&
- (cur_attr.attr_id == attr->attr_id)) {
- attr->len = cur_attr.len;
- attr->val_ptr = cur_attr.val_ptr;
- return 0;
- }
- } while (iter);
-
- return -EIO;
-}
-
-static int osdblk_get_obj_size(struct osdblk_device *osdev, u64 *size_out)
-{
- struct osd_request *or;
- struct osd_attr attr;
- int ret;
-
- /* start request */
- or = osd_start_request(osdev->osd, GFP_KERNEL);
- if (!or)
- return -ENOMEM;
-
- /* create a get-attributes(length) request */
- osd_req_get_attributes(or, &osdev->obj);
-
- osd_req_add_get_attr_list(or, &g_attr_logical_length, 1);
-
- /* execute op synchronously */
- ret = osd_sync_op(or, OSDBLK_OP_TIMEOUT, osdev->obj_cred);
- if (ret)
- goto out;
-
- /* extract length from returned attribute info */
- attr = g_attr_logical_length;
- ret = extract_attr_from_req(or, &attr);
- if (ret)
- goto out;
-
- *size_out = get_unaligned_be64(attr.val_ptr);
-
-out:
- osd_end_request(or);
- return ret;
-
-}
-
-static void osdblk_osd_complete(struct osd_request *or, void *private)
-{
- struct osdblk_request *orq = private;
- struct osd_sense_info osi;
- int ret = osd_req_decode_sense(or, &osi);
-
- if (ret) {
- ret = -EIO;
- OSDBLK_DEBUG("osdblk_osd_complete with err=%d\n", ret);
- }
-
- /* complete OSD request */
- osd_end_request(or);
-
- /* complete request passed to osdblk by block layer */
- __blk_end_request_all(orq->rq, ret);
-}
-
-static void bio_chain_put(struct bio *chain)
-{
- struct bio *tmp;
-
- while (chain) {
- tmp = chain;
- chain = chain->bi_next;
-
- bio_put(tmp);
- }
-}
-
-static struct bio *bio_chain_clone(struct bio *old_chain, gfp_t gfpmask)
-{
- struct bio *tmp, *new_chain = NULL, *tail = NULL;
-
- while (old_chain) {
- tmp = bio_clone_kmalloc(old_chain, gfpmask);
- if (!tmp)
- goto err_out;
-
- tmp->bi_bdev = NULL;
- gfpmask &= ~__GFP_DIRECT_RECLAIM;
- tmp->bi_next = NULL;
-
- if (!new_chain)
- new_chain = tail = tmp;
- else {
- tail->bi_next = tmp;
- tail = tmp;
- }
-
- old_chain = old_chain->bi_next;
- }
-
- return new_chain;
-
-err_out:
- OSDBLK_DEBUG("bio_chain_clone with err\n");
- bio_chain_put(new_chain);
- return NULL;
-}
-
-static void osdblk_rq_fn(struct request_queue *q)
-{
- struct osdblk_device *osdev = q->queuedata;
-
- while (1) {
- struct request *rq;
- struct osdblk_request *orq;
- struct osd_request *or;
- struct bio *bio;
- bool do_write, do_flush;
-
- /* peek at request from block layer */
- rq = blk_fetch_request(q);
- if (!rq)
- break;
-
- /* deduce our operation (read, write, flush) */
- /* I wish the block layer simplified cmd_type/cmd_flags/cmd[]
- * into a clearly defined set of RPC commands:
- * read, write, flush, scsi command, power mgmt req,
- * driver-specific, etc.
- */
-
- do_flush = (req_op(rq) == REQ_OP_FLUSH);
- do_write = (rq_data_dir(rq) == WRITE);
-
- if (!do_flush) { /* osd_flush does not use a bio */
- /* a bio clone to be passed down to OSD request */
- bio = bio_chain_clone(rq->bio, GFP_ATOMIC);
- if (!bio)
- break;
- } else
- bio = NULL;
-
- /* alloc internal OSD request, for OSD command execution */
- or = osd_start_request(osdev->osd, GFP_ATOMIC);
- if (!or) {
- bio_chain_put(bio);
- OSDBLK_DEBUG("osd_start_request with err\n");
- break;
- }
-
- orq = &osdev->req[rq->tag];
- orq->rq = rq;
- orq->bio = bio;
- orq->osdev = osdev;
-
- /* init OSD command: flush, write or read */
- if (do_flush)
- osd_req_flush_object(or, &osdev->obj,
- OSD_CDB_FLUSH_ALL, 0, 0);
- else if (do_write)
- osd_req_write(or, &osdev->obj, blk_rq_pos(rq) * 512ULL,
- bio, blk_rq_bytes(rq));
- else
- osd_req_read(or, &osdev->obj, blk_rq_pos(rq) * 512ULL,
- bio, blk_rq_bytes(rq));
-
- OSDBLK_DEBUG("%s 0x%x bytes at 0x%llx\n",
- do_flush ? "flush" : do_write ?
- "write" : "read", blk_rq_bytes(rq),
- blk_rq_pos(rq) * 512ULL);
-
- /* begin OSD command execution */
- if (osd_async_op(or, osdblk_osd_complete, orq,
- osdev->obj_cred)) {
- osd_end_request(or);
- blk_requeue_request(q, rq);
- bio_chain_put(bio);
- OSDBLK_DEBUG("osd_execute_request_async with err\n");
- break;
- }
-
- /* remove the special 'flush' marker, now that the command
- * is executing
- */
- rq->special = NULL;
- }
-}
-
-static void osdblk_free_disk(struct osdblk_device *osdev)
-{
- struct gendisk *disk = osdev->disk;
-
- if (!disk)
- return;
-
- if (disk->flags & GENHD_FL_UP)
- del_gendisk(disk);
- if (disk->queue)
- blk_cleanup_queue(disk->queue);
- put_disk(disk);
-}
-
-static int osdblk_init_disk(struct osdblk_device *osdev)
-{
- struct gendisk *disk;
- struct request_queue *q;
- int rc;
- u64 obj_size = 0;
-
- /* contact OSD, request size info about the object being mapped */
- rc = osdblk_get_obj_size(osdev, &obj_size);
- if (rc)
- return rc;
-
- /* create gendisk info */
- disk = alloc_disk(OSDBLK_MINORS_PER_MAJOR);
- if (!disk)
- return -ENOMEM;
-
- sprintf(disk->disk_name, DRV_NAME "%d", osdev->id);
- disk->major = osdev->major;
- disk->first_minor = 0;
- disk->fops = &osdblk_bd_ops;
- disk->private_data = osdev;
-
- /* init rq */
- q = blk_init_queue(osdblk_rq_fn, &osdev->lock);
- if (!q) {
- put_disk(disk);
- return -ENOMEM;
- }
-
- /* switch queue to TCQ mode; allocate tag map */
- rc = blk_queue_init_tags(q, OSDBLK_MAX_REQ, NULL, BLK_TAG_ALLOC_FIFO);
- if (rc) {
- blk_cleanup_queue(q);
- put_disk(disk);
- return rc;
- }
-
- /* Set our limits to the lower device limits, because osdblk cannot
- * sleep when allocating a lower-request and therefore cannot be
- * bouncing.
- */
- blk_queue_stack_limits(q, osd_request_queue(osdev->osd));
-
- blk_queue_prep_rq(q, blk_queue_start_tag);
- blk_queue_write_cache(q, true, false);
-
- disk->queue = q;
-
- q->queuedata = osdev;
-
- osdev->disk = disk;
- osdev->q = q;
-
- /* finally, announce the disk to the world */
- set_capacity(disk, obj_size / 512ULL);
- add_disk(disk);
-
- printk(KERN_INFO "%s: Added of size 0x%llx\n",
- disk->disk_name, (unsigned long long)obj_size);
-
- return 0;
-}
-
-/********************************************************************
- * /sys/class/osdblk/
- * add map OSD object to blkdev
- * remove unmap OSD object
- * list show mappings
- *******************************************************************/
-
-static void class_osdblk_release(struct class *cls)
-{
- kfree(cls);
-}
-
-static ssize_t class_osdblk_list(struct class *c,
- struct class_attribute *attr,
- char *data)
-{
- int n = 0;
- struct list_head *tmp;
-
- mutex_lock_nested(&ctl_mutex, SINGLE_DEPTH_NESTING);
-
- list_for_each(tmp, &osdblkdev_list) {
- struct osdblk_device *osdev;
-
- osdev = list_entry(tmp, struct osdblk_device, node);
-
- n += sprintf(data+n, "%d %d %llu %llu %s\n",
- osdev->id,
- osdev->major,
- osdev->obj.partition,
- osdev->obj.id,
- osdev->osd_path);
- }
-
- mutex_unlock(&ctl_mutex);
- return n;
-}
-
-static ssize_t class_osdblk_add(struct class *c,
- struct class_attribute *attr,
- const char *buf, size_t count)
-{
- struct osdblk_device *osdev;
- ssize_t rc;
- int irc, new_id = 0;
- struct list_head *tmp;
-
- if (!try_module_get(THIS_MODULE))
- return -ENODEV;
-
- /* new osdblk_device object */
- osdev = kzalloc(sizeof(*osdev) + strlen(buf) + 1, GFP_KERNEL);
- if (!osdev) {
- rc = -ENOMEM;
- goto err_out_mod;
- }
-
- /* static osdblk_device initialization */
- spin_lock_init(&osdev->lock);
- INIT_LIST_HEAD(&osdev->node);
-
- /* generate unique id: find highest unique id, add one */
-
- mutex_lock_nested(&ctl_mutex, SINGLE_DEPTH_NESTING);
-
- list_for_each(tmp, &osdblkdev_list) {
- struct osdblk_device *osdev;
-
- osdev = list_entry(tmp, struct osdblk_device, node);
- if (osdev->id > new_id)
- new_id = osdev->id + 1;
- }
-
- osdev->id = new_id;
-
- /* add to global list */
- list_add_tail(&osdev->node, &osdblkdev_list);
-
- mutex_unlock(&ctl_mutex);
-
- /* parse add command */
- if (sscanf(buf, "%llu %llu %s", &osdev->obj.partition, &osdev->obj.id,
- osdev->osd_path) != 3) {
- rc = -EINVAL;
- goto err_out_slot;
- }
-
- /* initialize rest of new object */
- sprintf(osdev->name, DRV_NAME "%d", osdev->id);
-
- /* contact requested OSD */
- osdev->osd = osduld_path_lookup(osdev->osd_path);
- if (IS_ERR(osdev->osd)) {
- rc = PTR_ERR(osdev->osd);
- goto err_out_slot;
- }
-
- /* build OSD credential */
- osdblk_make_credential(osdev->obj_cred, &osdev->obj);
-
- /* register our block device */
- irc = register_blkdev(0, osdev->name);
- if (irc < 0) {
- rc = irc;
- goto err_out_osd;
- }
-
- osdev->major = irc;
-
- /* set up and announce blkdev mapping */
- rc = osdblk_init_disk(osdev);
- if (rc)
- goto err_out_blkdev;
-
- return count;
-
-err_out_blkdev:
- unregister_blkdev(osdev->major, osdev->name);
-err_out_osd:
- osduld_put_device(osdev->osd);
-err_out_slot:
- mutex_lock_nested(&ctl_mutex, SINGLE_DEPTH_NESTING);
- list_del_init(&osdev->node);
- mutex_unlock(&ctl_mutex);
-
- kfree(osdev);
-err_out_mod:
- OSDBLK_DEBUG("Error adding device %s\n", buf);
- module_put(THIS_MODULE);
- return rc;
-}
-
-static ssize_t class_osdblk_remove(struct class *c,
- struct class_attribute *attr,
- const char *buf,
- size_t count)
-{
- struct osdblk_device *osdev = NULL;
- int target_id, rc;
- unsigned long ul;
- struct list_head *tmp;
-
- rc = kstrtoul(buf, 10, &ul);
- if (rc)
- return rc;
-
- /* convert to int; abort if we lost anything in the conversion */
- target_id = (int) ul;
- if (target_id != ul)
- return -EINVAL;
-
- /* remove object from list immediately */
- mutex_lock_nested(&ctl_mutex, SINGLE_DEPTH_NESTING);
-
- list_for_each(tmp, &osdblkdev_list) {
- osdev = list_entry(tmp, struct osdblk_device, node);
- if (osdev->id == target_id) {
- list_del_init(&osdev->node);
- break;
- }
- osdev = NULL;
- }
-
- mutex_unlock(&ctl_mutex);
-
- if (!osdev)
- return -ENOENT;
-
- /* clean up and free blkdev and associated OSD connection */
- osdblk_free_disk(osdev);
- unregister_blkdev(osdev->major, osdev->name);
- osduld_put_device(osdev->osd);
- kfree(osdev);
-
- /* release module ref */
- module_put(THIS_MODULE);
-
- return count;
-}
-
-static struct class_attribute class_osdblk_attrs[] = {
- __ATTR(add, 0200, NULL, class_osdblk_add),
- __ATTR(remove, 0200, NULL, class_osdblk_remove),
- __ATTR(list, 0444, class_osdblk_list, NULL),
- __ATTR_NULL
-};
-
-static int osdblk_sysfs_init(void)
-{
- int ret = 0;
-
- /*
- * create control files in sysfs
- * /sys/class/osdblk/...
- */
- class_osdblk = kzalloc(sizeof(*class_osdblk), GFP_KERNEL);
- if (!class_osdblk)
- return -ENOMEM;
-
- class_osdblk->name = DRV_NAME;
- class_osdblk->owner = THIS_MODULE;
- class_osdblk->class_release = class_osdblk_release;
- class_osdblk->class_attrs = class_osdblk_attrs;
-
- ret = class_register(class_osdblk);
- if (ret) {
- kfree(class_osdblk);
- class_osdblk = NULL;
- printk(PFX "failed to create class osdblk\n");
- return ret;
- }
-
- return 0;
-}
-
-static void osdblk_sysfs_cleanup(void)
-{
- if (class_osdblk)
- class_destroy(class_osdblk);
- class_osdblk = NULL;
-}
-
-static int __init osdblk_init(void)
-{
- int rc;
-
- rc = osdblk_sysfs_init();
- if (rc)
- return rc;
-
- return 0;
-}
-
-static void __exit osdblk_exit(void)
-{
- osdblk_sysfs_cleanup();
-}
-
-module_init(osdblk_init);
-module_exit(osdblk_exit);
-
diff --git a/drivers/block/paride/pcd.c b/drivers/block/paride/pcd.c
index 939641d..b1267ef 100644
--- a/drivers/block/paride/pcd.c
+++ b/drivers/block/paride/pcd.c
@@ -300,6 +300,11 @@
struct gendisk *disk = alloc_disk(1);
if (!disk)
continue;
+ disk->queue = blk_init_queue(do_pcd_request, &pcd_lock);
+ if (!disk->queue) {
+ put_disk(disk);
+ continue;
+ }
cd->disk = disk;
cd->pi = &cd->pia;
cd->present = 0;
@@ -735,18 +740,36 @@
}
/* I/O request processing */
-static struct request_queue *pcd_queue;
+static int pcd_queue;
-static void do_pcd_request(struct request_queue * q)
+static int set_next_request(void)
+{
+ struct pcd_unit *cd;
+ struct request_queue *q;
+ int old_pos = pcd_queue;
+
+ do {
+ cd = &pcd[pcd_queue];
+ q = cd->present ? cd->disk->queue : NULL;
+ if (++pcd_queue == PCD_UNITS)
+ pcd_queue = 0;
+ if (q) {
+ pcd_req = blk_fetch_request(q);
+ if (pcd_req)
+ break;
+ }
+ } while (pcd_queue != old_pos);
+
+ return pcd_req != NULL;
+}
+
+static void pcd_request(void)
{
if (pcd_busy)
return;
while (1) {
- if (!pcd_req) {
- pcd_req = blk_fetch_request(q);
- if (!pcd_req)
- return;
- }
+ if (!pcd_req && !set_next_request())
+ return;
if (rq_data_dir(pcd_req) == READ) {
struct pcd_unit *cd = pcd_req->rq_disk->private_data;
@@ -766,6 +789,11 @@
}
}
+static void do_pcd_request(struct request_queue *q)
+{
+ pcd_request();
+}
+
static inline void next_request(int err)
{
unsigned long saved_flags;
@@ -774,7 +802,7 @@
if (!__blk_end_request_cur(pcd_req, err))
pcd_req = NULL;
pcd_busy = 0;
- do_pcd_request(pcd_queue);
+ pcd_request();
spin_unlock_irqrestore(&pcd_lock, saved_flags);
}
@@ -849,7 +877,7 @@
do_pcd_read();
spin_lock_irqsave(&pcd_lock, saved_flags);
- do_pcd_request(pcd_queue);
+ pcd_request();
spin_unlock_irqrestore(&pcd_lock, saved_flags);
}
@@ -957,19 +985,10 @@
return -EBUSY;
}
- pcd_queue = blk_init_queue(do_pcd_request, &pcd_lock);
- if (!pcd_queue) {
- unregister_blkdev(major, name);
- for (unit = 0, cd = pcd; unit < PCD_UNITS; unit++, cd++)
- put_disk(cd->disk);
- return -ENOMEM;
- }
-
for (unit = 0, cd = pcd; unit < PCD_UNITS; unit++, cd++) {
if (cd->present) {
register_cdrom(&cd->info);
cd->disk->private_data = cd;
- cd->disk->queue = pcd_queue;
add_disk(cd->disk);
}
}
@@ -988,9 +1007,9 @@
pi_release(cd->pi);
unregister_cdrom(&cd->info);
}
+ blk_cleanup_queue(cd->disk->queue);
put_disk(cd->disk);
}
- blk_cleanup_queue(pcd_queue);
unregister_blkdev(major, name);
pi_unregister_driver(par_drv);
}
diff --git a/drivers/block/paride/pd.c b/drivers/block/paride/pd.c
index 9cfd2e0..7d2402f 100644
--- a/drivers/block/paride/pd.c
+++ b/drivers/block/paride/pd.c
@@ -381,12 +381,33 @@
static enum action do_pd_read_drq(void);
static enum action do_pd_write_done(void);
-static struct request_queue *pd_queue;
+static int pd_queue;
static int pd_claimed;
static struct pd_unit *pd_current; /* current request's drive */
static PIA *pi_current; /* current request's PIA */
+static int set_next_request(void)
+{
+ struct gendisk *disk;
+ struct request_queue *q;
+ int old_pos = pd_queue;
+
+ do {
+ disk = pd[pd_queue].gd;
+ q = disk ? disk->queue : NULL;
+ if (++pd_queue == PD_UNITS)
+ pd_queue = 0;
+ if (q) {
+ pd_req = blk_fetch_request(q);
+ if (pd_req)
+ break;
+ }
+ } while (pd_queue != old_pos);
+
+ return pd_req != NULL;
+}
+
static void run_fsm(void)
{
while (1) {
@@ -418,8 +439,7 @@
spin_lock_irqsave(&pd_lock, saved_flags);
if (!__blk_end_request_cur(pd_req,
res == Ok ? 0 : -EIO)) {
- pd_req = blk_fetch_request(pd_queue);
- if (!pd_req)
+ if (!set_next_request())
stop = 1;
}
spin_unlock_irqrestore(&pd_lock, saved_flags);
@@ -719,18 +739,15 @@
enum action (*func)(struct pd_unit *disk))
{
struct request *rq;
- int err = 0;
rq = blk_get_request(disk->gd->queue, REQ_OP_DRV_IN, __GFP_RECLAIM);
if (IS_ERR(rq))
return PTR_ERR(rq);
rq->special = func;
-
- err = blk_execute_rq(disk->gd->queue, disk->gd, rq, 0);
-
+ blk_execute_rq(disk->gd->queue, disk->gd, rq, 0);
blk_put_request(rq);
- return err;
+ return 0;
}
/* kernel glue structures */
@@ -839,7 +856,13 @@
p->first_minor = (disk - pd) << PD_BITS;
disk->gd = p;
p->private_data = disk;
- p->queue = pd_queue;
+ p->queue = blk_init_queue(do_pd_request, &pd_lock);
+ if (!p->queue) {
+ disk->gd = NULL;
+ put_disk(p);
+ return;
+ }
+ blk_queue_max_hw_sectors(p->queue, cluster);
if (disk->drive == -1) {
for (disk->drive = 0; disk->drive <= 1; disk->drive++)
@@ -919,26 +942,18 @@
if (disable)
goto out1;
- pd_queue = blk_init_queue(do_pd_request, &pd_lock);
- if (!pd_queue)
- goto out1;
-
- blk_queue_max_hw_sectors(pd_queue, cluster);
-
if (register_blkdev(major, name))
- goto out2;
+ goto out1;
printk("%s: %s version %s, major %d, cluster %d, nice %d\n",
name, name, PD_VERSION, major, cluster, nice);
if (!pd_detect())
- goto out3;
+ goto out2;
return 0;
-out3:
- unregister_blkdev(major, name);
out2:
- blk_cleanup_queue(pd_queue);
+ unregister_blkdev(major, name);
out1:
return -ENODEV;
}
@@ -953,11 +968,11 @@
if (p) {
disk->gd = NULL;
del_gendisk(p);
+ blk_cleanup_queue(p->queue);
put_disk(p);
pi_release(disk->pi);
}
}
- blk_cleanup_queue(pd_queue);
}
MODULE_LICENSE("GPL");
diff --git a/drivers/block/paride/pf.c b/drivers/block/paride/pf.c
index 14c5d32..f24ca73 100644
--- a/drivers/block/paride/pf.c
+++ b/drivers/block/paride/pf.c
@@ -287,6 +287,12 @@
struct gendisk *disk = alloc_disk(1);
if (!disk)
continue;
+ disk->queue = blk_init_queue(do_pf_request, &pf_spin_lock);
+ if (!disk->queue) {
+ put_disk(disk);
+ return;
+ }
+ blk_queue_max_segments(disk->queue, cluster);
pf->disk = disk;
pf->pi = &pf->pia;
pf->media_status = PF_NM;
@@ -772,7 +778,28 @@
return (((status_reg(pf_current) & (STAT_BUSY | pf_mask)) == pf_mask));
}
-static struct request_queue *pf_queue;
+static int pf_queue;
+
+static int set_next_request(void)
+{
+ struct pf_unit *pf;
+ struct request_queue *q;
+ int old_pos = pf_queue;
+
+ do {
+ pf = &units[pf_queue];
+ q = pf->present ? pf->disk->queue : NULL;
+ if (++pf_queue == PF_UNITS)
+ pf_queue = 0;
+ if (q) {
+ pf_req = blk_fetch_request(q);
+ if (pf_req)
+ break;
+ }
+ } while (pf_queue != old_pos);
+
+ return pf_req != NULL;
+}
static void pf_end_request(int err)
{
@@ -780,16 +807,13 @@
pf_req = NULL;
}
-static void do_pf_request(struct request_queue * q)
+static void pf_request(void)
{
if (pf_busy)
return;
repeat:
- if (!pf_req) {
- pf_req = blk_fetch_request(q);
- if (!pf_req)
- return;
- }
+ if (!pf_req && !set_next_request())
+ return;
pf_current = pf_req->rq_disk->private_data;
pf_block = blk_rq_pos(pf_req);
@@ -817,6 +841,11 @@
}
}
+static void do_pf_request(struct request_queue *q)
+{
+ pf_request();
+}
+
static int pf_next_buf(void)
{
unsigned long saved_flags;
@@ -846,7 +875,7 @@
spin_lock_irqsave(&pf_spin_lock, saved_flags);
pf_end_request(err);
pf_busy = 0;
- do_pf_request(pf_queue);
+ pf_request();
spin_unlock_irqrestore(&pf_spin_lock, saved_flags);
}
@@ -972,15 +1001,6 @@
put_disk(pf->disk);
return -EBUSY;
}
- pf_queue = blk_init_queue(do_pf_request, &pf_spin_lock);
- if (!pf_queue) {
- unregister_blkdev(major, name);
- for (pf = units, unit = 0; unit < PF_UNITS; pf++, unit++)
- put_disk(pf->disk);
- return -ENOMEM;
- }
-
- blk_queue_max_segments(pf_queue, cluster);
for (pf = units, unit = 0; unit < PF_UNITS; pf++, unit++) {
struct gendisk *disk = pf->disk;
@@ -988,7 +1008,6 @@
if (!pf->present)
continue;
disk->private_data = pf;
- disk->queue = pf_queue;
add_disk(disk);
}
return 0;
@@ -1003,10 +1022,10 @@
if (!pf->present)
continue;
del_gendisk(pf->disk);
+ blk_cleanup_queue(pf->disk->queue);
put_disk(pf->disk);
pi_release(pf->pi);
}
- blk_cleanup_queue(pf_queue);
}
MODULE_LICENSE("GPL");
diff --git a/drivers/block/pktcdvd.c b/drivers/block/pktcdvd.c
index 66d846b..205b865 100644
--- a/drivers/block/pktcdvd.c
+++ b/drivers/block/pktcdvd.c
@@ -724,7 +724,7 @@
rq->rq_flags |= RQF_QUIET;
blk_execute_rq(rq->q, pd->bdev->bd_disk, rq, 0);
- if (rq->errors)
+ if (scsi_req(rq)->result)
ret = -EIO;
out:
blk_put_request(rq);
diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 517838b..089ac41 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -4317,7 +4317,7 @@
return 0;
}
-static struct blk_mq_ops rbd_mq_ops = {
+static const struct blk_mq_ops rbd_mq_ops = {
.queue_rq = rbd_queue_rq,
.init_request = rbd_init_request,
};
@@ -4380,7 +4380,6 @@
q->limits.discard_granularity = segment_size;
q->limits.discard_alignment = segment_size;
blk_queue_max_discard_sectors(q, segment_size / SECTOR_SIZE);
- q->limits.discard_zeroes_data = 1;
if (!ceph_test_opt(rbd_dev->rbd_client->client, NOCRC))
q->backing_dev_info->capabilities |= BDI_CAP_STABLE_WRITES;
diff --git a/drivers/block/rsxx/dev.c b/drivers/block/rsxx/dev.c
index f81d70b..9c56636 100644
--- a/drivers/block/rsxx/dev.c
+++ b/drivers/block/rsxx/dev.c
@@ -300,7 +300,6 @@
RSXX_HW_BLK_SIZE >> 9);
card->queue->limits.discard_granularity = RSXX_HW_BLK_SIZE;
card->queue->limits.discard_alignment = RSXX_HW_BLK_SIZE;
- card->queue->limits.discard_zeroes_data = 1;
}
card->queue->queuedata = card;
diff --git a/drivers/block/swim.c b/drivers/block/swim.c
index b5afd49..3064be6 100644
--- a/drivers/block/swim.c
+++ b/drivers/block/swim.c
@@ -211,7 +211,7 @@
struct swim_priv {
struct swim __iomem *base;
spinlock_t lock;
- struct request_queue *queue;
+ int fdc_queue;
int floppy_count;
struct floppy_state unit[FD_MAX_UNIT];
};
@@ -525,12 +525,33 @@
return 0;
}
-static void redo_fd_request(struct request_queue *q)
+static struct request *swim_next_request(struct swim_priv *swd)
{
+ struct request_queue *q;
+ struct request *rq;
+ int old_pos = swd->fdc_queue;
+
+ do {
+ q = swd->unit[swd->fdc_queue].disk->queue;
+ if (++swd->fdc_queue == swd->floppy_count)
+ swd->fdc_queue = 0;
+ if (q) {
+ rq = blk_fetch_request(q);
+ if (rq)
+ return rq;
+ }
+ } while (swd->fdc_queue != old_pos);
+
+ return NULL;
+}
+
+static void do_fd_request(struct request_queue *q)
+{
+ struct swim_priv *swd = q->queuedata;
struct request *req;
struct floppy_state *fs;
- req = blk_fetch_request(q);
+ req = swim_next_request(swd);
while (req) {
int err = -EIO;
@@ -554,15 +575,10 @@
}
done:
if (!__blk_end_request_cur(req, err))
- req = blk_fetch_request(q);
+ req = swim_next_request(swd);
}
}
-static void do_fd_request(struct request_queue *q)
-{
- redo_fd_request(q);
-}
-
static struct floppy_struct floppy_type[4] = {
{ 0, 0, 0, 0, 0, 0x00, 0x00, 0x00, 0x00, NULL }, /* no testing */
{ 720, 9, 1, 80, 0, 0x2A, 0x02, 0xDF, 0x50, NULL }, /* 360KB SS 3.5"*/
@@ -833,22 +849,25 @@
return -EBUSY;
}
+ spin_lock_init(&swd->lock);
+
for (drive = 0; drive < swd->floppy_count; drive++) {
swd->unit[drive].disk = alloc_disk(1);
if (swd->unit[drive].disk == NULL) {
err = -ENOMEM;
goto exit_put_disks;
}
+ swd->unit[drive].disk->queue = blk_init_queue(do_fd_request,
+ &swd->lock);
+ if (!swd->unit[drive].disk->queue) {
+ err = -ENOMEM;
+ put_disk(swd->unit[drive].disk);
+ goto exit_put_disks;
+ }
+ swd->unit[drive].disk->queue->queuedata = swd;
swd->unit[drive].swd = swd;
}
- spin_lock_init(&swd->lock);
- swd->queue = blk_init_queue(do_fd_request, &swd->lock);
- if (!swd->queue) {
- err = -ENOMEM;
- goto exit_put_disks;
- }
-
for (drive = 0; drive < swd->floppy_count; drive++) {
swd->unit[drive].disk->flags = GENHD_FL_REMOVABLE;
swd->unit[drive].disk->major = FLOPPY_MAJOR;
@@ -856,7 +875,6 @@
sprintf(swd->unit[drive].disk->disk_name, "fd%d", drive);
swd->unit[drive].disk->fops = &floppy_fops;
swd->unit[drive].disk->private_data = &swd->unit[drive];
- swd->unit[drive].disk->queue = swd->queue;
set_capacity(swd->unit[drive].disk, 2880);
add_disk(swd->unit[drive].disk);
}
@@ -943,13 +961,12 @@
for (drive = 0; drive < swd->floppy_count; drive++) {
del_gendisk(swd->unit[drive].disk);
+ blk_cleanup_queue(swd->unit[drive].disk->queue);
put_disk(swd->unit[drive].disk);
}
unregister_blkdev(FLOPPY_MAJOR, "fd");
- blk_cleanup_queue(swd->queue);
-
/* eject floppies */
for (drive = 0; drive < swd->floppy_count; drive++)
diff --git a/drivers/block/swim3.c b/drivers/block/swim3.c
index 61b3ffa..ba4809c 100644
--- a/drivers/block/swim3.c
+++ b/drivers/block/swim3.c
@@ -343,8 +343,8 @@
req->rq_disk->disk_name, req->cmd,
(long)blk_rq_pos(req), blk_rq_sectors(req),
bio_data(req->bio));
- swim3_dbg(" errors=%d current_nr_sectors=%u\n",
- req->errors, blk_rq_cur_sectors(req));
+ swim3_dbg(" current_nr_sectors=%u\n",
+ blk_rq_cur_sectors(req));
#endif
if (blk_rq_pos(req) >= fs->total_secs) {
diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 1d4c9f8..f946142 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -111,7 +111,7 @@
return virtqueue_add_sgs(vq, sgs, num_out, num_in, vbr, GFP_ATOMIC);
}
-static inline void virtblk_scsi_reques_done(struct request *req)
+static inline void virtblk_scsi_request_done(struct request *req)
{
struct virtblk_req *vbr = blk_mq_rq_to_pdu(req);
struct virtio_blk *vblk = req->q->queuedata;
@@ -119,7 +119,7 @@
sreq->resid_len = virtio32_to_cpu(vblk->vdev, vbr->in_hdr.residual);
sreq->sense_len = virtio32_to_cpu(vblk->vdev, vbr->in_hdr.sense_len);
- req->errors = virtio32_to_cpu(vblk->vdev, vbr->in_hdr.errors);
+ sreq->result = virtio32_to_cpu(vblk->vdev, vbr->in_hdr.errors);
}
static int virtblk_ioctl(struct block_device *bdev, fmode_t mode,
@@ -144,7 +144,7 @@
{
return -EIO;
}
-static inline void virtblk_scsi_reques_done(struct request *req)
+static inline void virtblk_scsi_request_done(struct request *req)
{
}
#define virtblk_ioctl NULL
@@ -175,19 +175,15 @@
static inline void virtblk_request_done(struct request *req)
{
struct virtblk_req *vbr = blk_mq_rq_to_pdu(req);
- int error = virtblk_result(vbr);
switch (req_op(req)) {
case REQ_OP_SCSI_IN:
case REQ_OP_SCSI_OUT:
- virtblk_scsi_reques_done(req);
- break;
- case REQ_OP_DRV_IN:
- req->errors = (error != 0);
+ virtblk_scsi_request_done(req);
break;
}
- blk_mq_end_request(req, error);
+ blk_mq_end_request(req, virtblk_result(vbr));
}
static void virtblk_done(struct virtqueue *vq)
@@ -205,7 +201,7 @@
while ((vbr = virtqueue_get_buf(vblk->vqs[qid].vq, &len)) != NULL) {
struct request *req = blk_mq_rq_from_pdu(vbr);
- blk_mq_complete_request(req, req->errors);
+ blk_mq_complete_request(req);
req_done = true;
}
if (unlikely(virtqueue_is_broken(vq)))
@@ -310,7 +306,8 @@
if (err)
goto out;
- err = blk_execute_rq(vblk->disk->queue, vblk->disk, req, false);
+ blk_execute_rq(vblk->disk->queue, vblk->disk, req, false);
+ err = virtblk_result(blk_mq_rq_to_pdu(req));
out:
blk_put_request(req);
return err;
@@ -597,7 +594,7 @@
return blk_mq_virtio_map_queues(set, vblk->vdev, 0);
}
-static struct blk_mq_ops virtio_mq_ops = {
+static const struct blk_mq_ops virtio_mq_ops = {
.queue_rq = virtio_queue_rq,
.complete = virtblk_request_done,
.init_request = virtblk_init_request,
diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c
index 5067a0a..3945963 100644
--- a/drivers/block/xen-blkfront.c
+++ b/drivers/block/xen-blkfront.c
@@ -115,6 +115,15 @@
atomic_t pending;
};
+struct blkif_req {
+ int error;
+};
+
+static inline struct blkif_req *blkif_req(struct request *rq)
+{
+ return blk_mq_rq_to_pdu(rq);
+}
+
static DEFINE_MUTEX(blkfront_mutex);
static const struct block_device_operations xlvbd_block_fops;
@@ -907,8 +916,14 @@
return BLK_MQ_RQ_QUEUE_BUSY;
}
-static struct blk_mq_ops blkfront_mq_ops = {
+static void blkif_complete_rq(struct request *rq)
+{
+ blk_mq_end_request(rq, blkif_req(rq)->error);
+}
+
+static const struct blk_mq_ops blkfront_mq_ops = {
.queue_rq = blkif_queue_rq,
+ .complete = blkif_complete_rq,
};
static void blkif_set_queue_limits(struct blkfront_info *info)
@@ -969,7 +984,7 @@
info->tag_set.queue_depth = BLK_RING_SIZE(info);
info->tag_set.numa_node = NUMA_NO_NODE;
info->tag_set.flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_SG_MERGE;
- info->tag_set.cmd_size = 0;
+ info->tag_set.cmd_size = sizeof(struct blkif_req);
info->tag_set.driver_data = info;
if (blk_mq_alloc_tag_set(&info->tag_set))
@@ -1543,7 +1558,6 @@
unsigned long flags;
struct blkfront_ring_info *rinfo = (struct blkfront_ring_info *)dev_id;
struct blkfront_info *info = rinfo->dev_info;
- int error;
if (unlikely(info->connected != BLKIF_STATE_CONNECTED))
return IRQ_HANDLED;
@@ -1587,37 +1601,36 @@
continue;
}
- error = (bret->status == BLKIF_RSP_OKAY) ? 0 : -EIO;
+ blkif_req(req)->error = (bret->status == BLKIF_RSP_OKAY) ? 0 : -EIO;
switch (bret->operation) {
case BLKIF_OP_DISCARD:
if (unlikely(bret->status == BLKIF_RSP_EOPNOTSUPP)) {
struct request_queue *rq = info->rq;
printk(KERN_WARNING "blkfront: %s: %s op failed\n",
info->gd->disk_name, op_name(bret->operation));
- error = -EOPNOTSUPP;
+ blkif_req(req)->error = -EOPNOTSUPP;
info->feature_discard = 0;
info->feature_secdiscard = 0;
queue_flag_clear(QUEUE_FLAG_DISCARD, rq);
queue_flag_clear(QUEUE_FLAG_SECERASE, rq);
}
- blk_mq_complete_request(req, error);
break;
case BLKIF_OP_FLUSH_DISKCACHE:
case BLKIF_OP_WRITE_BARRIER:
if (unlikely(bret->status == BLKIF_RSP_EOPNOTSUPP)) {
printk(KERN_WARNING "blkfront: %s: %s op failed\n",
info->gd->disk_name, op_name(bret->operation));
- error = -EOPNOTSUPP;
+ blkif_req(req)->error = -EOPNOTSUPP;
}
if (unlikely(bret->status == BLKIF_RSP_ERROR &&
rinfo->shadow[id].req.u.rw.nr_segments == 0)) {
printk(KERN_WARNING "blkfront: %s: empty %s op failed\n",
info->gd->disk_name, op_name(bret->operation));
- error = -EOPNOTSUPP;
+ blkif_req(req)->error = -EOPNOTSUPP;
}
- if (unlikely(error)) {
- if (error == -EOPNOTSUPP)
- error = 0;
+ if (unlikely(blkif_req(req)->error)) {
+ if (blkif_req(req)->error == -EOPNOTSUPP)
+ blkif_req(req)->error = 0;
info->feature_fua = 0;
info->feature_flush = 0;
xlvbd_flush(info);
@@ -1629,11 +1642,12 @@
dev_dbg(&info->xbdev->dev, "Bad return from blkdev data "
"request: %x\n", bret->status);
- blk_mq_complete_request(req, error);
break;
default:
BUG();
}
+
+ blk_mq_complete_request(req);
}
rinfo->ring.rsp_cons = i;
@@ -2345,6 +2359,7 @@
unsigned long sector_size;
unsigned int physical_sector_size;
unsigned int binfo;
+ char *envp[] = { "RESIZE=1", NULL };
int err, i;
switch (info->connected) {
@@ -2361,6 +2376,8 @@
sectors);
set_capacity(info->gd, sectors);
revalidate_disk(info->gd);
+ kobject_uevent_env(&disk_to_dev(info->gd)->kobj,
+ KOBJ_CHANGE, envp);
return;
case BLKIF_STATE_SUSPENDED:
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 0c09d42..6fac5fe 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -829,10 +829,14 @@
offset = (bio->bi_iter.bi_sector &
(SECTORS_PER_PAGE - 1)) << SECTOR_SHIFT;
- if (unlikely(bio_op(bio) == REQ_OP_DISCARD)) {
+ switch (bio_op(bio)) {
+ case REQ_OP_DISCARD:
+ case REQ_OP_WRITE_ZEROES:
zram_bio_discard(zram, index, offset, bio);
bio_endio(bio);
return;
+ default:
+ break;
}
bio_for_each_segment(bvec, bio, iter) {
@@ -1192,6 +1196,8 @@
zram->disk->queue->limits.max_sectors = SECTORS_PER_PAGE;
zram->disk->queue->limits.chunk_sectors = 0;
blk_queue_max_discard_sectors(zram->disk->queue, UINT_MAX);
+ queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, zram->disk->queue);
+
/*
* zram_bio_discard() will clear all logical blocks if logical block
* size is identical with physical block size(PAGE_SIZE). But if it is
@@ -1201,10 +1207,7 @@
* zeroed.
*/
if (ZRAM_LOGICAL_BLOCK_SIZE == PAGE_SIZE)
- zram->disk->queue->limits.discard_zeroes_data = 1;
- else
- zram->disk->queue->limits.discard_zeroes_data = 0;
- queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, zram->disk->queue);
+ blk_queue_max_write_zeroes_sectors(zram->disk->queue, UINT_MAX);
add_disk(zram->disk);
diff --git a/drivers/bluetooth/Kconfig b/drivers/bluetooth/Kconfig
index 08e0545..737d93e 100644
--- a/drivers/bluetooth/Kconfig
+++ b/drivers/bluetooth/Kconfig
@@ -76,6 +76,12 @@
Say Y here to compile support for Bluetooth UART devices into the
kernel or say M to compile it as module (hci_uart).
+config BT_HCIUART_SERDEV
+ bool
+ depends on SERIAL_DEV_BUS && BT_HCIUART
+ depends on SERIAL_DEV_BUS=y || SERIAL_DEV_BUS=BT_HCIUART
+ default y
+
config BT_HCIUART_H4
bool "UART (H4) protocol support"
depends on BT_HCIUART
@@ -86,6 +92,18 @@
Say Y here to compile support for HCI UART (H4) protocol.
+config BT_HCIUART_NOKIA
+ tristate "UART Nokia H4+ protocol support"
+ depends on BT_HCIUART
+ depends on BT_HCIUART_SERDEV
+ depends on PM
+ help
+ Nokia H4+ is serial protocol for communication between Bluetooth
+ device and host. This protocol is required for Bluetooth devices
+ with UART interface in Nokia devices.
+
+ Say Y here to compile support for Nokia's H4+ protocol.
+
config BT_HCIUART_BCSP
bool "BCSP protocol support"
depends on BT_HCIUART
@@ -344,7 +362,7 @@
config BT_QCOMSMD
tristate "Qualcomm SMD based HCI support"
- depends on QCOM_SMD || (COMPILE_TEST && QCOM_SMD=n)
+ depends on RPMSG || (COMPILE_TEST && RPMSG=n)
depends on QCOM_WCNSS_CTRL || (COMPILE_TEST && QCOM_WCNSS_CTRL=n)
select BT_QCA
help
diff --git a/drivers/bluetooth/Makefile b/drivers/bluetooth/Makefile
index 8062718..e693ca6 100644
--- a/drivers/bluetooth/Makefile
+++ b/drivers/bluetooth/Makefile
@@ -25,10 +25,13 @@
obj-$(CONFIG_BT_RTL) += btrtl.o
obj-$(CONFIG_BT_QCA) += btqca.o
+obj-$(CONFIG_BT_HCIUART_NOKIA) += hci_nokia.o
+
btmrvl-y := btmrvl_main.o
btmrvl-$(CONFIG_DEBUG_FS) += btmrvl_debugfs.o
hci_uart-y := hci_ldisc.o
+hci_uart-$(CONFIG_BT_HCIUART_SERDEV) += hci_serdev.o
hci_uart-$(CONFIG_BT_HCIUART_H4) += hci_h4.o
hci_uart-$(CONFIG_BT_HCIUART_BCSP) += hci_bcsp.o
hci_uart-$(CONFIG_BT_HCIUART_LL) += hci_ll.o
diff --git a/drivers/bluetooth/bluecard_cs.c b/drivers/bluetooth/bluecard_cs.c
index c0b3b55..007c0a4 100644
--- a/drivers/bluetooth/bluecard_cs.c
+++ b/drivers/bluetooth/bluecard_cs.c
@@ -695,9 +695,8 @@
spin_lock_init(&(info->lock));
- init_timer(&(info->timer));
- info->timer.function = &bluecard_activity_led_timeout;
- info->timer.data = (u_long)info;
+ setup_timer(&(info->timer), &bluecard_activity_led_timeout,
+ (u_long)info);
skb_queue_head_init(&(info->txq));
diff --git a/drivers/bluetooth/btmrvl_sdio.c b/drivers/bluetooth/btmrvl_sdio.c
index 08e01f0..eb794f0 100644
--- a/drivers/bluetooth/btmrvl_sdio.c
+++ b/drivers/bluetooth/btmrvl_sdio.c
@@ -20,6 +20,7 @@
#include <linux/firmware.h>
#include <linux/slab.h>
+#include <linux/suspend.h>
#include <linux/mmc/sdio_ids.h>
#include <linux/mmc/sdio_func.h>
@@ -60,13 +61,15 @@
static irqreturn_t btmrvl_wake_irq_bt(int irq, void *priv)
{
- struct btmrvl_plt_wake_cfg *cfg = priv;
+ struct btmrvl_sdio_card *card = priv;
+ struct btmrvl_plt_wake_cfg *cfg = card->plt_wake_cfg;
- if (cfg->irq_bt >= 0) {
- pr_info("%s: wake by bt", __func__);
- cfg->wake_by_bt = true;
- disable_irq_nosync(irq);
- }
+ pr_info("%s: wake by bt", __func__);
+ cfg->wake_by_bt = true;
+ disable_irq_nosync(irq);
+
+ pm_wakeup_event(&card->func->dev, 0);
+ pm_system_wakeup();
return IRQ_HANDLED;
}
@@ -101,7 +104,7 @@
} else {
ret = devm_request_irq(dev, cfg->irq_bt,
btmrvl_wake_irq_bt,
- 0, "bt_wake", cfg);
+ 0, "bt_wake", card);
if (ret) {
dev_err(dev,
"Failed to request irq_bt %d (%d)\n",
@@ -1574,7 +1577,7 @@
MODULE_SHUTDOWN_REQ);
btmrvl_sdio_disable_host_int(card);
}
- BT_DBG("unregester dev");
+ BT_DBG("unregister dev");
card->priv->surprise_removed = true;
btmrvl_sdio_unregister_dev(card);
btmrvl_remove_card(card->priv);
@@ -1625,6 +1628,13 @@
if (priv->adapter->hs_state != HS_ACTIVATED) {
if (btmrvl_enable_hs(priv)) {
BT_ERR("HS not activated, suspend failed!");
+ /* Disable platform specific wakeup interrupt */
+ if (card->plt_wake_cfg &&
+ card->plt_wake_cfg->irq_bt >= 0) {
+ disable_irq_wake(card->plt_wake_cfg->irq_bt);
+ disable_irq(card->plt_wake_cfg->irq_bt);
+ }
+
priv->adapter->is_suspending = false;
return -EBUSY;
}
@@ -1637,10 +1647,10 @@
if (priv->adapter->hs_state == HS_ACTIVATED) {
BT_DBG("suspend with MMC_PM_KEEP_POWER");
return sdio_set_host_pm_flags(func, MMC_PM_KEEP_POWER);
- } else {
- BT_DBG("suspend without MMC_PM_KEEP_POWER");
- return 0;
}
+
+ BT_DBG("suspend without MMC_PM_KEEP_POWER");
+ return 0;
}
static int btmrvl_sdio_resume(struct device *dev)
diff --git a/drivers/bluetooth/btqcomsmd.c b/drivers/bluetooth/btqcomsmd.c
index 8d4868a..ef730c1 100644
--- a/drivers/bluetooth/btqcomsmd.c
+++ b/drivers/bluetooth/btqcomsmd.c
@@ -14,7 +14,7 @@
#include <linux/module.h>
#include <linux/slab.h>
-#include <linux/soc/qcom/smd.h>
+#include <linux/rpmsg.h>
#include <linux/soc/qcom/wcnss_ctrl.h>
#include <linux/platform_device.h>
@@ -26,8 +26,8 @@
struct btqcomsmd {
struct hci_dev *hdev;
- struct qcom_smd_channel *acl_channel;
- struct qcom_smd_channel *cmd_channel;
+ struct rpmsg_endpoint *acl_channel;
+ struct rpmsg_endpoint *cmd_channel;
};
static int btqcomsmd_recv(struct hci_dev *hdev, unsigned int type,
@@ -48,19 +48,19 @@
return hci_recv_frame(hdev, skb);
}
-static int btqcomsmd_acl_callback(struct qcom_smd_channel *channel,
- const void *data, size_t count)
+static int btqcomsmd_acl_callback(struct rpmsg_device *rpdev, void *data,
+ int count, void *priv, u32 addr)
{
- struct btqcomsmd *btq = qcom_smd_get_drvdata(channel);
+ struct btqcomsmd *btq = priv;
btq->hdev->stat.byte_rx += count;
return btqcomsmd_recv(btq->hdev, HCI_ACLDATA_PKT, data, count);
}
-static int btqcomsmd_cmd_callback(struct qcom_smd_channel *channel,
- const void *data, size_t count)
+static int btqcomsmd_cmd_callback(struct rpmsg_device *rpdev, void *data,
+ int count, void *priv, u32 addr)
{
- struct btqcomsmd *btq = qcom_smd_get_drvdata(channel);
+ struct btqcomsmd *btq = priv;
return btqcomsmd_recv(btq->hdev, HCI_EVENT_PKT, data, count);
}
@@ -72,12 +72,12 @@
switch (hci_skb_pkt_type(skb)) {
case HCI_ACLDATA_PKT:
- ret = qcom_smd_send(btq->acl_channel, skb->data, skb->len);
+ ret = rpmsg_send(btq->acl_channel, skb->data, skb->len);
hdev->stat.acl_tx++;
hdev->stat.byte_tx += skb->len;
break;
case HCI_COMMAND_PKT:
- ret = qcom_smd_send(btq->cmd_channel, skb->data, skb->len);
+ ret = rpmsg_send(btq->cmd_channel, skb->data, skb->len);
hdev->stat.cmd_tx++;
break;
default:
@@ -114,18 +114,15 @@
wcnss = dev_get_drvdata(pdev->dev.parent);
btq->acl_channel = qcom_wcnss_open_channel(wcnss, "APPS_RIVA_BT_ACL",
- btqcomsmd_acl_callback);
+ btqcomsmd_acl_callback, btq);
if (IS_ERR(btq->acl_channel))
return PTR_ERR(btq->acl_channel);
btq->cmd_channel = qcom_wcnss_open_channel(wcnss, "APPS_RIVA_BT_CMD",
- btqcomsmd_cmd_callback);
+ btqcomsmd_cmd_callback, btq);
if (IS_ERR(btq->cmd_channel))
return PTR_ERR(btq->cmd_channel);
- qcom_smd_set_drvdata(btq->acl_channel, btq);
- qcom_smd_set_drvdata(btq->cmd_channel, btq);
-
hdev = hci_alloc_dev();
if (!hdev)
return -ENOMEM;
@@ -158,6 +155,9 @@
hci_unregister_dev(btq->hdev);
hci_free_dev(btq->hdev);
+ rpmsg_destroy_ept(btq->cmd_channel);
+ rpmsg_destroy_ept(btq->acl_channel);
+
return 0;
}
diff --git a/drivers/bluetooth/btrtl.c b/drivers/bluetooth/btrtl.c
index fc9b257..8279094 100644
--- a/drivers/bluetooth/btrtl.c
+++ b/drivers/bluetooth/btrtl.c
@@ -275,11 +275,8 @@
BT_INFO("%s: rtl: loading %s", hdev->name, name);
ret = request_firmware(&fw, name, &hdev->dev);
- if (ret < 0) {
- BT_ERR("%s: Failed to load %s", hdev->name, name);
+ if (ret < 0)
return ret;
- }
-
ret = fw->size;
*buff = kmemdup(fw->data, ret, GFP_KERNEL);
@@ -331,6 +328,7 @@
u8 *cfg_buff = NULL;
u8 *tbuff;
char *cfg_name = NULL;
+ bool config_needed = false;
switch (lmp_subver) {
case RTL_ROM_LMP_8723B:
@@ -344,6 +342,7 @@
break;
case RTL_ROM_LMP_8822B:
cfg_name = "rtl_bt/rtl8822b_config.bin";
+ config_needed = true;
break;
default:
BT_ERR("%s: rtl: no config according to lmp_subver %04x",
@@ -353,8 +352,12 @@
if (cfg_name) {
cfg_sz = rtl_load_config(hdev, cfg_name, &cfg_buff);
- if (cfg_sz < 0)
+ if (cfg_sz < 0) {
cfg_sz = 0;
+ if (config_needed)
+ BT_ERR("Necessary config file %s not found\n",
+ cfg_name);
+ }
} else
cfg_sz = 0;
diff --git a/drivers/bluetooth/btusb.c b/drivers/bluetooth/btusb.c
index 1c8094e..7fa373b 100644
--- a/drivers/bluetooth/btusb.c
+++ b/drivers/bluetooth/btusb.c
@@ -26,6 +26,7 @@
#include <linux/firmware.h>
#include <linux/of_device.h>
#include <linux/of_irq.h>
+#include <linux/suspend.h>
#include <asm/unaligned.h>
#include <net/bluetooth/bluetooth.h>
@@ -262,6 +263,7 @@
{ USB_DEVICE(0x0cf3, 0xe007), .driver_info = BTUSB_QCA_ROME },
{ USB_DEVICE(0x0cf3, 0xe009), .driver_info = BTUSB_QCA_ROME },
{ USB_DEVICE(0x0cf3, 0xe300), .driver_info = BTUSB_QCA_ROME },
+ { USB_DEVICE(0x0cf3, 0xe301), .driver_info = BTUSB_QCA_ROME },
{ USB_DEVICE(0x0cf3, 0xe360), .driver_info = BTUSB_QCA_ROME },
{ USB_DEVICE(0x0489, 0xe092), .driver_info = BTUSB_QCA_ROME },
{ USB_DEVICE(0x04ca, 0x3011), .driver_info = BTUSB_QCA_ROME },
@@ -328,6 +330,7 @@
{ USB_DEVICE(0x1286, 0x204e), .driver_info = BTUSB_MARVELL },
/* Intel Bluetooth devices */
+ { USB_DEVICE(0x8087, 0x0025), .driver_info = BTUSB_INTEL_NEW },
{ USB_DEVICE(0x8087, 0x07da), .driver_info = BTUSB_CSR },
{ USB_DEVICE(0x8087, 0x07dc), .driver_info = BTUSB_INTEL },
{ USB_DEVICE(0x8087, 0x0a2a), .driver_info = BTUSB_INTEL },
@@ -2024,13 +2027,18 @@
return -EINVAL;
}
- /* At the moment the iBT 3.0 hardware variants 0x0b (LnP/SfP)
- * and 0x0c (WsP) are supported by this firmware loading method.
+ /* Check for supported iBT hardware variants of this firmware
+ * loading method.
*
* This check has been put in place to ensure correct forward
* compatibility options when newer hardware variants come along.
*/
- if (ver.hw_variant != 0x0b && ver.hw_variant != 0x0c) {
+ switch (ver.hw_variant) {
+ case 0x0b: /* SfP */
+ case 0x0c: /* WsP */
+ case 0x12: /* ThP */
+ break;
+ default:
BT_ERR("%s: Unsupported Intel hardware variant (%u)",
hdev->name, ver.hw_variant);
return -EINVAL;
@@ -2792,6 +2800,7 @@
struct btusb_data *data = priv;
pm_wakeup_event(&data->udev->dev, 0);
+ pm_system_wakeup();
/* Disable only if not already disabled (keep it balanced) */
if (test_and_clear_bit(BTUSB_OOB_WAKE_ENABLED, &data->flags)) {
diff --git a/drivers/bluetooth/hci_bcm.c b/drivers/bluetooth/hci_bcm.c
index 5262a20..f87bfdf 100644
--- a/drivers/bluetooth/hci_bcm.c
+++ b/drivers/bluetooth/hci_bcm.c
@@ -146,13 +146,13 @@
static int bcm_gpio_set_power(struct bcm_device *dev, bool powered)
{
if (powered && !IS_ERR(dev->clk) && !dev->clk_enabled)
- clk_enable(dev->clk);
+ clk_prepare_enable(dev->clk);
gpiod_set_value(dev->shutdown, powered);
gpiod_set_value(dev->device_wakeup, powered);
if (!powered && !IS_ERR(dev->clk) && dev->clk_enabled)
- clk_disable(dev->clk);
+ clk_disable_unprepare(dev->clk);
dev->clk_enabled = powered;
@@ -287,6 +287,9 @@
hu->priv = bcm;
+ if (!hu->tty->dev)
+ goto out;
+
mutex_lock(&bcm_device_lock);
list_for_each(p, &bcm_device_list) {
struct bcm_device *dev = list_entry(p, struct bcm_device, list);
@@ -307,7 +310,7 @@
}
mutex_unlock(&bcm_device_lock);
-
+out:
return 0;
}
@@ -697,28 +700,14 @@
/* Always tell the ACPI core to skip this resource */
return 1;
}
+#endif /* CONFIG_ACPI */
-static int bcm_acpi_probe(struct bcm_device *dev)
+static int bcm_platform_probe(struct bcm_device *dev)
{
struct platform_device *pdev = dev->pdev;
- LIST_HEAD(resources);
- const struct dmi_system_id *dmi_id;
- const struct acpi_gpio_mapping *gpio_mapping = acpi_bcm_int_last_gpios;
- const struct acpi_device_id *id;
- int ret;
dev->name = dev_name(&pdev->dev);
- /* Retrieve GPIO data */
- id = acpi_match_device(pdev->dev.driver->acpi_match_table, &pdev->dev);
- if (id)
- gpio_mapping = (const struct acpi_gpio_mapping *) id->driver_data;
-
- ret = acpi_dev_add_driver_gpios(ACPI_COMPANION(&pdev->dev),
- gpio_mapping);
- if (ret)
- return ret;
-
dev->clk = devm_clk_get(&pdev->dev, NULL);
dev->device_wakeup = devm_gpiod_get_optional(&pdev->dev,
@@ -755,6 +744,33 @@
return -EINVAL;
}
+ return 0;
+}
+
+#ifdef CONFIG_ACPI
+static int bcm_acpi_probe(struct bcm_device *dev)
+{
+ struct platform_device *pdev = dev->pdev;
+ LIST_HEAD(resources);
+ const struct dmi_system_id *dmi_id;
+ const struct acpi_gpio_mapping *gpio_mapping = acpi_bcm_int_last_gpios;
+ const struct acpi_device_id *id;
+ int ret;
+
+ /* Retrieve GPIO data */
+ id = acpi_match_device(pdev->dev.driver->acpi_match_table, &pdev->dev);
+ if (id)
+ gpio_mapping = (const struct acpi_gpio_mapping *) id->driver_data;
+
+ ret = acpi_dev_add_driver_gpios(ACPI_COMPANION(&pdev->dev),
+ gpio_mapping);
+ if (ret)
+ return ret;
+
+ ret = bcm_platform_probe(dev);
+ if (ret)
+ return ret;
+
/* Retrieve UART ACPI info */
ret = acpi_dev_get_resources(ACPI_COMPANION(&dev->pdev->dev),
&resources, bcm_resource, dev);
@@ -789,7 +805,10 @@
dev->pdev = pdev;
- ret = bcm_acpi_probe(dev);
+ if (has_acpi_companion(&pdev->dev))
+ ret = bcm_acpi_probe(dev);
+ else
+ ret = bcm_platform_probe(dev);
if (ret)
return ret;
diff --git a/drivers/bluetooth/hci_h4.c b/drivers/bluetooth/hci_h4.c
index 635597b6..82e5a32 100644
--- a/drivers/bluetooth/hci_h4.c
+++ b/drivers/bluetooth/hci_h4.c
@@ -171,9 +171,20 @@
const unsigned char *buffer, int count,
const struct h4_recv_pkt *pkts, int pkts_count)
{
+ struct hci_uart *hu = hci_get_drvdata(hdev);
+ u8 alignment = hu->alignment;
+
while (count) {
int i, len;
+ /* remove padding bytes from buffer */
+ for (; hu->padding && count > 0; hu->padding--) {
+ count--;
+ buffer++;
+ }
+ if (!count)
+ break;
+
if (!skb) {
for (i = 0; i < pkts_count; i++) {
if (buffer[0] != (&pkts[i])->type)
@@ -253,11 +264,17 @@
}
if (!dlen) {
+ hu->padding = (skb->len - 1) % alignment;
+ hu->padding = (alignment - hu->padding) % alignment;
+
/* No more data, complete frame */
(&pkts[i])->recv(hdev, skb);
skb = NULL;
}
} else {
+ hu->padding = (skb->len - 1) % alignment;
+ hu->padding = (alignment - hu->padding) % alignment;
+
/* Complete frame */
(&pkts[i])->recv(hdev, skb);
skb = NULL;
diff --git a/drivers/bluetooth/hci_intel.c b/drivers/bluetooth/hci_intel.c
index 9e27128..fa50999 100644
--- a/drivers/bluetooth/hci_intel.c
+++ b/drivers/bluetooth/hci_intel.c
@@ -307,6 +307,9 @@
struct list_head *p;
int err = -ENODEV;
+ if (!hu->tty->dev)
+ return err;
+
mutex_lock(&intel_device_list_lock);
list_for_each(p, &intel_device_list) {
@@ -379,6 +382,9 @@
struct intel_data *intel = container_of(work, struct intel_data,
busy_work);
+ if (!intel->hu->tty->dev)
+ return;
+
/* Link is busy, delay the suspend */
mutex_lock(&intel_device_list_lock);
list_for_each(p, &intel_device_list) {
@@ -601,12 +607,18 @@
return -EINVAL;
}
- /* At the moment only the hardware variant iBT 3.0 (LnP/SfP) is
- * supported by this firmware loading method. This check has been
- * put in place to ensure correct forward compatibility options
- * when newer hardware variants come along.
- */
- if (ver.hw_variant != 0x0b) {
+ /* Check for supported iBT hardware variants of this firmware
+ * loading method.
+ *
+ * This check has been put in place to ensure correct forward
+ * compatibility options when newer hardware variants come along.
+ */
+ switch (ver.hw_variant) {
+ case 0x0b: /* LnP */
+ case 0x0c: /* WsP */
+ case 0x12: /* ThP */
+ break;
+ default:
bt_dev_err(hdev, "Unsupported Intel hardware variant (%u)",
ver.hw_variant);
return -EINVAL;
@@ -699,11 +711,14 @@
/* With this Intel bootloader only the hardware variant and device
* revision information are used to select the right firmware.
*
- * Currently this bootloader support is limited to hardware variant
- * iBT 3.0 (LnP/SfP) which is identified by the value 11 (0x0b).
+ * The firmware filename is ibt-<hw_variant>-<dev_revid>.sfi.
+ *
+ * Currently the supported hardware variants are:
+ * 11 (0x0b) for iBT 3.0 (LnP/SfP)
*/
- snprintf(fwname, sizeof(fwname), "intel/ibt-11-%u.sfi",
- le16_to_cpu(params->dev_revid));
+ snprintf(fwname, sizeof(fwname), "intel/ibt-%u-%u.sfi",
+ le16_to_cpu(ver.hw_variant),
+ le16_to_cpu(params->dev_revid));
err = request_firmware(&fw, fwname, &hdev->dev);
if (err < 0) {
@@ -716,8 +731,9 @@
bt_dev_info(hdev, "Found device firmware: %s", fwname);
/* Save the DDC file name for later */
- snprintf(fwname, sizeof(fwname), "intel/ibt-11-%u.ddc",
- le16_to_cpu(params->dev_revid));
+ snprintf(fwname, sizeof(fwname), "intel/ibt-%u-%u.ddc",
+ le16_to_cpu(ver.hw_variant),
+ le16_to_cpu(params->dev_revid));
kfree_skb(skb);
@@ -889,6 +905,8 @@
list_for_each(p, &intel_device_list) {
struct intel_device *dev = list_entry(p, struct intel_device,
list);
+ if (!hu->tty->dev)
+ break;
if (hu->tty->dev->parent == dev->pdev->dev.parent) {
if (device_may_wakeup(&dev->pdev->dev)) {
set_bit(STATE_LPM_ENABLED, &intel->flags);
@@ -1056,6 +1074,9 @@
BT_DBG("hu %p skb %p", hu, skb);
+ if (!hu->tty->dev)
+ goto out_enqueue;
+
/* Be sure our controller is resumed and potential LPM transaction
* completed before enqueuing any packet.
*/
@@ -1072,7 +1093,7 @@
}
}
mutex_unlock(&intel_device_list_lock);
-
+out_enqueue:
skb_queue_tail(&intel->txq, skb);
return 0;
diff --git a/drivers/bluetooth/hci_ldisc.c b/drivers/bluetooth/hci_ldisc.c
index 9497c46..2edd305 100644
--- a/drivers/bluetooth/hci_ldisc.c
+++ b/drivers/bluetooth/hci_ldisc.c
@@ -113,16 +113,21 @@
{
struct sk_buff *skb = hu->tx_skb;
- if (!skb)
- skb = hu->proto->dequeue(hu);
- else
+ if (!skb) {
+ if (test_bit(HCI_UART_PROTO_READY, &hu->flags))
+ skb = hu->proto->dequeue(hu);
+ } else {
hu->tx_skb = NULL;
+ }
return skb;
}
int hci_uart_tx_wakeup(struct hci_uart *hu)
{
+ if (!test_bit(HCI_UART_PROTO_READY, &hu->flags))
+ return 0;
+
if (test_and_set_bit(HCI_UART_SENDING, &hu->tx_state)) {
set_bit(HCI_UART_TX_WAKEUP, &hu->tx_state);
return 0;
@@ -134,6 +139,7 @@
return 0;
}
+EXPORT_SYMBOL_GPL(hci_uart_tx_wakeup);
static void hci_uart_write_work(struct work_struct *work)
{
@@ -176,6 +182,7 @@
{
struct hci_uart *hu = container_of(work, struct hci_uart, init_ready);
int err;
+ struct hci_dev *hdev;
if (!test_and_clear_bit(HCI_UART_INIT_PENDING, &hu->hdev_flags))
return;
@@ -183,9 +190,12 @@
err = hci_register_dev(hu->hdev);
if (err < 0) {
BT_ERR("Can't register HCI device");
- hci_free_dev(hu->hdev);
+ hdev = hu->hdev;
hu->hdev = NULL;
+ hci_free_dev(hdev);
+ clear_bit(HCI_UART_PROTO_READY, &hu->flags);
hu->proto->close(hu);
+ return;
}
set_bit(HCI_UART_REGISTERED, &hu->flags);
@@ -251,6 +261,9 @@
BT_DBG("%s: type %d len %d", hdev->name, hci_skb_pkt_type(skb),
skb->len);
+ if (!test_bit(HCI_UART_PROTO_READY, &hu->flags))
+ return -EUNATCH;
+
hu->proto->enqueue(hu, skb);
hci_uart_tx_wakeup(hu);
@@ -318,25 +331,6 @@
hu->oper_speed = oper_speed;
}
-void hci_uart_init_tty(struct hci_uart *hu)
-{
- struct tty_struct *tty = hu->tty;
- struct ktermios ktermios;
-
- /* Bring the UART into a known 8 bits no parity hw fc state */
- ktermios = tty->termios;
- ktermios.c_iflag &= ~(IGNBRK | BRKINT | PARMRK | ISTRIP |
- INLCR | IGNCR | ICRNL | IXON);
- ktermios.c_oflag &= ~OPOST;
- ktermios.c_lflag &= ~(ECHO | ECHONL | ICANON | ISIG | IEXTEN);
- ktermios.c_cflag &= ~(CSIZE | PARENB);
- ktermios.c_cflag |= CS8;
- ktermios.c_cflag |= CRTSCTS;
-
- /* tty_set_termios() return not checked as it is always 0 */
- tty_set_termios(tty, &ktermios);
-}
-
void hci_uart_set_baudrate(struct hci_uart *hu, unsigned int speed)
{
struct tty_struct *tty = hu->tty;
@@ -459,6 +453,10 @@
hu->tty = tty;
tty->receive_room = 65536;
+ /* disable alignment support by default */
+ hu->alignment = 1;
+ hu->padding = 0;
+
INIT_WORK(&hu->init_ready, hci_uart_init_work);
INIT_WORK(&hu->write_work, hci_uart_write_work);
@@ -616,6 +614,7 @@
if (hci_register_dev(hdev) < 0) {
BT_ERR("Can't register HCI device");
+ hu->hdev = NULL;
hci_free_dev(hdev);
return -ENODEV;
}
diff --git a/drivers/bluetooth/hci_ll.c b/drivers/bluetooth/hci_ll.c
index 02692fe..adc444f 100644
--- a/drivers/bluetooth/hci_ll.c
+++ b/drivers/bluetooth/hci_ll.c
@@ -34,20 +34,24 @@
#include <linux/sched.h>
#include <linux/types.h>
#include <linux/fcntl.h>
+#include <linux/firmware.h>
#include <linux/interrupt.h>
#include <linux/ptrace.h>
#include <linux/poll.h>
#include <linux/slab.h>
-#include <linux/tty.h>
#include <linux/errno.h>
#include <linux/string.h>
#include <linux/signal.h>
#include <linux/ioctl.h>
+#include <linux/of.h>
+#include <linux/serdev.h>
#include <linux/skbuff.h>
+#include <linux/ti_wilink_st.h>
#include <net/bluetooth/bluetooth.h>
#include <net/bluetooth/hci_core.h>
+#include <linux/gpio/consumer.h>
#include "hci_uart.h"
@@ -76,6 +80,12 @@
u8 cmd;
} __packed;
+struct ll_device {
+ struct hci_uart hu;
+ struct serdev_device *serdev;
+ struct gpio_desc *enable_gpio;
+};
+
struct ll_struct {
unsigned long rx_state;
unsigned long rx_count;
@@ -136,6 +146,9 @@
hu->priv = ll;
+ if (hu->serdev)
+ serdev_device_open(hu->serdev);
+
return 0;
}
@@ -164,6 +177,13 @@
kfree_skb(ll->rx_skb);
+ if (hu->serdev) {
+ struct ll_device *lldev = serdev_device_get_drvdata(hu->serdev);
+ gpiod_set_value_cansleep(lldev->enable_gpio, 0);
+
+ serdev_device_close(hu->serdev);
+ }
+
hu->priv = NULL;
kfree(ll);
@@ -505,9 +525,244 @@
return skb_dequeue(&ll->txq);
}
+#if IS_ENABLED(CONFIG_SERIAL_DEV_BUS)
+static int read_local_version(struct hci_dev *hdev)
+{
+ int err = 0;
+ unsigned short version = 0;
+ struct sk_buff *skb;
+ struct hci_rp_read_local_version *ver;
+
+ skb = __hci_cmd_sync(hdev, HCI_OP_READ_LOCAL_VERSION, 0, NULL, HCI_INIT_TIMEOUT);
+ if (IS_ERR(skb)) {
+ bt_dev_err(hdev, "Reading TI version information failed (%ld)",
+ PTR_ERR(skb));
+ return PTR_ERR(skb);
+ }
+ if (skb->len != sizeof(*ver)) {
+ err = -EILSEQ;
+ goto out;
+ }
+
+ ver = (struct hci_rp_read_local_version *)skb->data;
+ if (le16_to_cpu(ver->manufacturer) != 13) {
+ err = -ENODEV;
+ goto out;
+ }
+
+ version = le16_to_cpu(ver->lmp_subver);
+
+out:
+ if (err) bt_dev_err(hdev, "Failed to read TI version info: %d", err);
+ kfree_skb(skb);
+ return err ? err : version;
+}
+
+/**
+ * download_firmware -
+ * internal function which parses through the .bts firmware
+ * script file intreprets SEND, DELAY actions only as of now
+ */
+static int download_firmware(struct ll_device *lldev)
+{
+ unsigned short chip, min_ver, maj_ver;
+ int version, err, len;
+ unsigned char *ptr, *action_ptr;
+ unsigned char bts_scr_name[40]; /* 40 char long bts scr name? */
+ const struct firmware *fw;
+ struct sk_buff *skb;
+ struct hci_command *cmd;
+
+ version = read_local_version(lldev->hu.hdev);
+ if (version < 0)
+ return version;
+
+ chip = (version & 0x7C00) >> 10;
+ min_ver = (version & 0x007F);
+ maj_ver = (version & 0x0380) >> 7;
+ if (version & 0x8000)
+ maj_ver |= 0x0008;
+
+ snprintf(bts_scr_name, sizeof(bts_scr_name),
+ "ti-connectivity/TIInit_%d.%d.%d.bts",
+ chip, maj_ver, min_ver);
+
+ err = request_firmware(&fw, bts_scr_name, &lldev->serdev->dev);
+ if (err || !fw->data || !fw->size) {
+ bt_dev_err(lldev->hu.hdev, "request_firmware failed(errno %d) for %s",
+ err, bts_scr_name);
+ return -EINVAL;
+ }
+ ptr = (void *)fw->data;
+ len = fw->size;
+ /* bts_header to remove out magic number and
+ * version
+ */
+ ptr += sizeof(struct bts_header);
+ len -= sizeof(struct bts_header);
+
+ while (len > 0 && ptr) {
+ bt_dev_dbg(lldev->hu.hdev, " action size %d, type %d ",
+ ((struct bts_action *)ptr)->size,
+ ((struct bts_action *)ptr)->type);
+
+ action_ptr = &(((struct bts_action *)ptr)->data[0]);
+
+ switch (((struct bts_action *)ptr)->type) {
+ case ACTION_SEND_COMMAND: /* action send */
+ bt_dev_dbg(lldev->hu.hdev, "S");
+ cmd = (struct hci_command *)action_ptr;
+ if (cmd->opcode == 0xff36) {
+ /* ignore remote change
+ * baud rate HCI VS command */
+ bt_dev_warn(lldev->hu.hdev, "change remote baud rate command in firmware");
+ break;
+ }
+ if (cmd->prefix != 1)
+ bt_dev_dbg(lldev->hu.hdev, "command type %d\n", cmd->prefix);
+
+ skb = __hci_cmd_sync(lldev->hu.hdev, cmd->opcode, cmd->plen, &cmd->speed, HCI_INIT_TIMEOUT);
+ if (IS_ERR(skb)) {
+ bt_dev_err(lldev->hu.hdev, "send command failed\n");
+ goto out_rel_fw;
+ }
+ kfree_skb(skb);
+ break;
+ case ACTION_WAIT_EVENT: /* wait */
+ /* no need to wait as command was synchronous */
+ bt_dev_dbg(lldev->hu.hdev, "W");
+ break;
+ case ACTION_DELAY: /* sleep */
+ bt_dev_info(lldev->hu.hdev, "sleep command in scr");
+ mdelay(((struct bts_action_delay *)action_ptr)->msec);
+ break;
+ }
+ len -= (sizeof(struct bts_action) +
+ ((struct bts_action *)ptr)->size);
+ ptr += sizeof(struct bts_action) +
+ ((struct bts_action *)ptr)->size;
+ }
+
+out_rel_fw:
+ /* fw download complete */
+ release_firmware(fw);
+ return err;
+}
+
+static int ll_setup(struct hci_uart *hu)
+{
+ int err, retry = 3;
+ struct ll_device *lldev;
+ struct serdev_device *serdev = hu->serdev;
+ u32 speed;
+
+ if (!serdev)
+ return 0;
+
+ lldev = serdev_device_get_drvdata(serdev);
+
+ serdev_device_set_flow_control(serdev, true);
+
+ do {
+ /* Configure BT_EN to HIGH state */
+ gpiod_set_value_cansleep(lldev->enable_gpio, 0);
+ msleep(5);
+ gpiod_set_value_cansleep(lldev->enable_gpio, 1);
+ msleep(100);
+
+ err = download_firmware(lldev);
+ if (!err)
+ break;
+
+ /* Toggle BT_EN and retry */
+ bt_dev_err(hu->hdev, "download firmware failed, retrying...");
+ } while (retry--);
+
+ if (err)
+ return err;
+
+ /* Operational speed if any */
+ if (hu->oper_speed)
+ speed = hu->oper_speed;
+ else if (hu->proto->oper_speed)
+ speed = hu->proto->oper_speed;
+ else
+ speed = 0;
+
+ if (speed) {
+ struct sk_buff *skb = __hci_cmd_sync(hu->hdev, 0xff36, sizeof(speed), &speed, HCI_INIT_TIMEOUT);
+ if (!IS_ERR(skb)) {
+ kfree_skb(skb);
+ serdev_device_set_baudrate(serdev, speed);
+ }
+ }
+
+ return 0;
+}
+
+static const struct hci_uart_proto llp;
+
+static int hci_ti_probe(struct serdev_device *serdev)
+{
+ struct hci_uart *hu;
+ struct ll_device *lldev;
+ u32 max_speed = 3000000;
+
+ lldev = devm_kzalloc(&serdev->dev, sizeof(struct ll_device), GFP_KERNEL);
+ if (!lldev)
+ return -ENOMEM;
+ hu = &lldev->hu;
+
+ serdev_device_set_drvdata(serdev, lldev);
+ lldev->serdev = hu->serdev = serdev;
+
+ lldev->enable_gpio = devm_gpiod_get_optional(&serdev->dev, "enable", GPIOD_OUT_LOW);
+ if (IS_ERR(lldev->enable_gpio))
+ return PTR_ERR(lldev->enable_gpio);
+
+ of_property_read_u32(serdev->dev.of_node, "max-speed", &max_speed);
+ hci_uart_set_speeds(hu, 115200, max_speed);
+
+ return hci_uart_register_device(hu, &llp);
+}
+
+static void hci_ti_remove(struct serdev_device *serdev)
+{
+ struct ll_device *lldev = serdev_device_get_drvdata(serdev);
+ struct hci_uart *hu = &lldev->hu;
+ struct hci_dev *hdev = hu->hdev;
+
+ cancel_work_sync(&hu->write_work);
+
+ hci_unregister_dev(hdev);
+ hci_free_dev(hdev);
+ hu->proto->close(hu);
+}
+
+static const struct of_device_id hci_ti_of_match[] = {
+ { .compatible = "ti,wl1831-st" },
+ { .compatible = "ti,wl1835-st" },
+ { .compatible = "ti,wl1837-st" },
+ {},
+};
+MODULE_DEVICE_TABLE(of, hci_ti_of_match);
+
+static struct serdev_device_driver hci_ti_drv = {
+ .driver = {
+ .name = "hci-ti",
+ .of_match_table = of_match_ptr(hci_ti_of_match),
+ },
+ .probe = hci_ti_probe,
+ .remove = hci_ti_remove,
+};
+#else
+#define ll_setup NULL
+#endif
+
static const struct hci_uart_proto llp = {
.id = HCI_UART_LL,
.name = "LL",
+ .setup = ll_setup,
.open = ll_open,
.close = ll_close,
.recv = ll_recv,
@@ -518,10 +773,14 @@
int __init ll_init(void)
{
+ serdev_device_driver_register(&hci_ti_drv);
+
return hci_uart_register_proto(&llp);
}
int __exit ll_deinit(void)
{
+ serdev_device_driver_unregister(&hci_ti_drv);
+
return hci_uart_unregister_proto(&llp);
}
diff --git a/drivers/bluetooth/hci_nokia.c b/drivers/bluetooth/hci_nokia.c
new file mode 100644
index 0000000..a7d687d8
--- /dev/null
+++ b/drivers/bluetooth/hci_nokia.c
@@ -0,0 +1,827 @@
+/*
+ * Bluetooth HCI UART H4 driver with Nokia Extensions AKA Nokia H4+
+ *
+ * Copyright (C) 2015 Marcel Holtmann <marcel@holtmann.org>
+ * Copyright (C) 2015-2017 Sebastian Reichel <sre@kernel.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/clk.h>
+#include <linux/errno.h>
+#include <linux/firmware.h>
+#include <linux/gpio/consumer.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/pm_runtime.h>
+#include <linux/serdev.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <linux/types.h>
+#include <linux/unaligned/le_struct.h>
+#include <net/bluetooth/bluetooth.h>
+#include <net/bluetooth/hci_core.h>
+
+#include "hci_uart.h"
+#include "btbcm.h"
+
+#define VERSION "0.1"
+
+#define NOKIA_ID_BCM2048 0x04
+#define NOKIA_ID_TI1271 0x31
+
+#define FIRMWARE_BCM2048 "nokia/bcmfw.bin"
+#define FIRMWARE_TI1271 "nokia/ti1273.bin"
+
+#define HCI_NOKIA_NEG_PKT 0x06
+#define HCI_NOKIA_ALIVE_PKT 0x07
+#define HCI_NOKIA_RADIO_PKT 0x08
+
+#define HCI_NOKIA_NEG_HDR_SIZE 1
+#define HCI_NOKIA_MAX_NEG_SIZE 255
+#define HCI_NOKIA_ALIVE_HDR_SIZE 1
+#define HCI_NOKIA_MAX_ALIVE_SIZE 255
+#define HCI_NOKIA_RADIO_HDR_SIZE 2
+#define HCI_NOKIA_MAX_RADIO_SIZE 255
+
+#define NOKIA_PROTO_PKT 0x44
+#define NOKIA_PROTO_BYTE 0x4c
+
+#define NOKIA_NEG_REQ 0x00
+#define NOKIA_NEG_ACK 0x20
+#define NOKIA_NEG_NAK 0x40
+
+#define H4_TYPE_SIZE 1
+
+#define NOKIA_RECV_ALIVE \
+ .type = HCI_NOKIA_ALIVE_PKT, \
+ .hlen = HCI_NOKIA_ALIVE_HDR_SIZE, \
+ .loff = 0, \
+ .lsize = 1, \
+ .maxlen = HCI_NOKIA_MAX_ALIVE_SIZE \
+
+#define NOKIA_RECV_NEG \
+ .type = HCI_NOKIA_NEG_PKT, \
+ .hlen = HCI_NOKIA_NEG_HDR_SIZE, \
+ .loff = 0, \
+ .lsize = 1, \
+ .maxlen = HCI_NOKIA_MAX_NEG_SIZE \
+
+#define NOKIA_RECV_RADIO \
+ .type = HCI_NOKIA_RADIO_PKT, \
+ .hlen = HCI_NOKIA_RADIO_HDR_SIZE, \
+ .loff = 1, \
+ .lsize = 1, \
+ .maxlen = HCI_NOKIA_MAX_RADIO_SIZE \
+
+struct hci_nokia_neg_hdr {
+ u8 dlen;
+} __packed;
+
+struct hci_nokia_neg_cmd {
+ u8 ack;
+ u16 baud;
+ u16 unused1;
+ u8 proto;
+ u16 sys_clk;
+ u16 unused2;
+} __packed;
+
+#define NOKIA_ALIVE_REQ 0x55
+#define NOKIA_ALIVE_RESP 0xcc
+
+struct hci_nokia_alive_hdr {
+ u8 dlen;
+} __packed;
+
+struct hci_nokia_alive_pkt {
+ u8 mid;
+ u8 unused;
+} __packed;
+
+struct hci_nokia_neg_evt {
+ u8 ack;
+ u16 baud;
+ u16 unused1;
+ u8 proto;
+ u16 sys_clk;
+ u16 unused2;
+ u8 man_id;
+ u8 ver_id;
+} __packed;
+
+#define MAX_BAUD_RATE 3692300
+#define SETUP_BAUD_RATE 921600
+#define INIT_BAUD_RATE 120000
+
+struct hci_nokia_radio_hdr {
+ u8 evt;
+ u8 dlen;
+} __packed;
+
+struct nokia_bt_dev {
+ struct hci_uart hu;
+ struct serdev_device *serdev;
+
+ struct gpio_desc *reset;
+ struct gpio_desc *wakeup_host;
+ struct gpio_desc *wakeup_bt;
+ unsigned long sysclk_speed;
+
+ int wake_irq;
+ struct sk_buff *rx_skb;
+ struct sk_buff_head txq;
+ bdaddr_t bdaddr;
+
+ int init_error;
+ struct completion init_completion;
+
+ u8 man_id;
+ u8 ver_id;
+
+ bool initialized;
+ bool tx_enabled;
+ bool rx_enabled;
+};
+
+static int nokia_enqueue(struct hci_uart *hu, struct sk_buff *skb);
+
+static void nokia_flow_control(struct serdev_device *serdev, bool enable)
+{
+ if (enable) {
+ serdev_device_set_rts(serdev, true);
+ serdev_device_set_flow_control(serdev, true);
+ } else {
+ serdev_device_set_flow_control(serdev, false);
+ serdev_device_set_rts(serdev, false);
+ }
+}
+
+static irqreturn_t wakeup_handler(int irq, void *data)
+{
+ struct nokia_bt_dev *btdev = data;
+ struct device *dev = &btdev->serdev->dev;
+ int wake_state = gpiod_get_value(btdev->wakeup_host);
+
+ if (btdev->rx_enabled == wake_state)
+ return IRQ_HANDLED;
+
+ if (wake_state)
+ pm_runtime_get(dev);
+ else
+ pm_runtime_put(dev);
+
+ btdev->rx_enabled = wake_state;
+
+ return IRQ_HANDLED;
+}
+
+static int nokia_reset(struct hci_uart *hu)
+{
+ struct nokia_bt_dev *btdev = hu->priv;
+ struct device *dev = &btdev->serdev->dev;
+ int err;
+
+ /* reset routine */
+ gpiod_set_value_cansleep(btdev->reset, 1);
+ gpiod_set_value_cansleep(btdev->wakeup_bt, 1);
+
+ msleep(100);
+
+ /* safety check */
+ err = gpiod_get_value_cansleep(btdev->wakeup_host);
+ if (err == 1) {
+ dev_err(dev, "reset: host wakeup not low!");
+ return -EPROTO;
+ }
+
+ /* flush queue */
+ serdev_device_write_flush(btdev->serdev);
+
+ /* init uart */
+ nokia_flow_control(btdev->serdev, false);
+ serdev_device_set_baudrate(btdev->serdev, INIT_BAUD_RATE);
+
+ gpiod_set_value_cansleep(btdev->reset, 0);
+
+ /* wait for cts */
+ err = serdev_device_wait_for_cts(btdev->serdev, true, 200);
+ if (err < 0) {
+ dev_err(dev, "CTS not received: %d", err);
+ return err;
+ }
+
+ nokia_flow_control(btdev->serdev, true);
+
+ return 0;
+}
+
+static int nokia_send_alive_packet(struct hci_uart *hu)
+{
+ struct nokia_bt_dev *btdev = hu->priv;
+ struct device *dev = &btdev->serdev->dev;
+ struct hci_nokia_alive_hdr *hdr;
+ struct hci_nokia_alive_pkt *pkt;
+ struct sk_buff *skb;
+ int len;
+
+ init_completion(&btdev->init_completion);
+
+ len = H4_TYPE_SIZE + sizeof(*hdr) + sizeof(*pkt);
+ skb = bt_skb_alloc(len, GFP_KERNEL);
+ if (!skb)
+ return -ENOMEM;
+
+ hci_skb_pkt_type(skb) = HCI_NOKIA_ALIVE_PKT;
+ memset(skb->data, 0x00, len);
+
+ hdr = (struct hci_nokia_alive_hdr *)skb_put(skb, sizeof(*hdr));
+ hdr->dlen = sizeof(*pkt);
+ pkt = (struct hci_nokia_alive_pkt *)skb_put(skb, sizeof(*pkt));
+ pkt->mid = NOKIA_ALIVE_REQ;
+
+ nokia_enqueue(hu, skb);
+ hci_uart_tx_wakeup(hu);
+
+ dev_dbg(dev, "Alive sent");
+
+ if (!wait_for_completion_interruptible_timeout(&btdev->init_completion,
+ msecs_to_jiffies(1000))) {
+ return -ETIMEDOUT;
+ }
+
+ if (btdev->init_error < 0)
+ return btdev->init_error;
+
+ return 0;
+}
+
+static int nokia_send_negotiation(struct hci_uart *hu)
+{
+ struct nokia_bt_dev *btdev = hu->priv;
+ struct device *dev = &btdev->serdev->dev;
+ struct hci_nokia_neg_cmd *neg_cmd;
+ struct hci_nokia_neg_hdr *neg_hdr;
+ struct sk_buff *skb;
+ int len, err;
+ u16 baud = DIV_ROUND_CLOSEST(btdev->sysclk_speed * 10, SETUP_BAUD_RATE);
+ int sysclk = btdev->sysclk_speed / 1000;
+
+ len = H4_TYPE_SIZE + sizeof(*neg_hdr) + sizeof(*neg_cmd);
+ skb = bt_skb_alloc(len, GFP_KERNEL);
+ if (!skb)
+ return -ENOMEM;
+
+ hci_skb_pkt_type(skb) = HCI_NOKIA_NEG_PKT;
+
+ neg_hdr = (struct hci_nokia_neg_hdr *)skb_put(skb, sizeof(*neg_hdr));
+ neg_hdr->dlen = sizeof(*neg_cmd);
+
+ neg_cmd = (struct hci_nokia_neg_cmd *)skb_put(skb, sizeof(*neg_cmd));
+ neg_cmd->ack = NOKIA_NEG_REQ;
+ neg_cmd->baud = cpu_to_le16(baud);
+ neg_cmd->unused1 = 0x0000;
+ neg_cmd->proto = NOKIA_PROTO_BYTE;
+ neg_cmd->sys_clk = cpu_to_le16(sysclk);
+ neg_cmd->unused2 = 0x0000;
+
+ btdev->init_error = 0;
+ init_completion(&btdev->init_completion);
+
+ nokia_enqueue(hu, skb);
+ hci_uart_tx_wakeup(hu);
+
+ dev_dbg(dev, "Negotiation sent");
+
+ if (!wait_for_completion_interruptible_timeout(&btdev->init_completion,
+ msecs_to_jiffies(10000))) {
+ return -ETIMEDOUT;
+ }
+
+ if (btdev->init_error < 0)
+ return btdev->init_error;
+
+ /* Change to previously negotiated speed. Flow Control
+ * is disabled until bluetooth adapter is ready to avoid
+ * broken bytes being received.
+ */
+ nokia_flow_control(btdev->serdev, false);
+ serdev_device_set_baudrate(btdev->serdev, SETUP_BAUD_RATE);
+ err = serdev_device_wait_for_cts(btdev->serdev, true, 200);
+ if (err < 0) {
+ dev_err(dev, "CTS not received: %d", err);
+ return err;
+ }
+ nokia_flow_control(btdev->serdev, true);
+
+ dev_dbg(dev, "Negotiation successful");
+
+ return 0;
+}
+
+static int nokia_setup_fw(struct hci_uart *hu)
+{
+ struct nokia_bt_dev *btdev = hu->priv;
+ struct device *dev = &btdev->serdev->dev;
+ const char *fwname;
+ const struct firmware *fw;
+ const u8 *fw_ptr;
+ size_t fw_size;
+ int err;
+
+ dev_dbg(dev, "setup firmware");
+
+ if (btdev->man_id == NOKIA_ID_BCM2048) {
+ fwname = FIRMWARE_BCM2048;
+ } else if (btdev->man_id == NOKIA_ID_TI1271) {
+ fwname = FIRMWARE_TI1271;
+ } else {
+ dev_err(dev, "Unsupported bluetooth device!");
+ return -ENODEV;
+ }
+
+ err = request_firmware(&fw, fwname, dev);
+ if (err < 0) {
+ dev_err(dev, "%s: Failed to load Nokia firmware file (%d)",
+ hu->hdev->name, err);
+ return err;
+ }
+
+ fw_ptr = fw->data;
+ fw_size = fw->size;
+
+ while (fw_size >= 4) {
+ u16 pkt_size = get_unaligned_le16(fw_ptr);
+ u8 pkt_type = fw_ptr[2];
+ const struct hci_command_hdr *cmd;
+ u16 opcode;
+ struct sk_buff *skb;
+
+ switch (pkt_type) {
+ case HCI_COMMAND_PKT:
+ cmd = (struct hci_command_hdr *)(fw_ptr + 3);
+ opcode = le16_to_cpu(cmd->opcode);
+
+ skb = __hci_cmd_sync(hu->hdev, opcode, cmd->plen,
+ fw_ptr + 3 + HCI_COMMAND_HDR_SIZE,
+ HCI_INIT_TIMEOUT);
+ if (IS_ERR(skb)) {
+ err = PTR_ERR(skb);
+ dev_err(dev, "%s: FW command %04x failed (%d)",
+ hu->hdev->name, opcode, err);
+ goto done;
+ }
+ kfree_skb(skb);
+ break;
+ case HCI_NOKIA_RADIO_PKT:
+ case HCI_NOKIA_NEG_PKT:
+ case HCI_NOKIA_ALIVE_PKT:
+ break;
+ }
+
+ fw_ptr += pkt_size + 2;
+ fw_size -= pkt_size + 2;
+ }
+
+done:
+ release_firmware(fw);
+ return err;
+}
+
+static int nokia_setup(struct hci_uart *hu)
+{
+ struct nokia_bt_dev *btdev = hu->priv;
+ struct device *dev = &btdev->serdev->dev;
+ int err;
+
+ btdev->initialized = false;
+
+ nokia_flow_control(btdev->serdev, false);
+
+ pm_runtime_get_sync(dev);
+
+ if (btdev->tx_enabled) {
+ gpiod_set_value_cansleep(btdev->wakeup_bt, 0);
+ pm_runtime_put(&btdev->serdev->dev);
+ btdev->tx_enabled = false;
+ }
+
+ dev_dbg(dev, "protocol setup");
+
+ /* 0. reset connection */
+ err = nokia_reset(hu);
+ if (err < 0) {
+ dev_err(dev, "Reset failed: %d", err);
+ goto out;
+ }
+
+ /* 1. negotiate speed etc */
+ err = nokia_send_negotiation(hu);
+ if (err < 0) {
+ dev_err(dev, "Negotiation failed: %d", err);
+ goto out;
+ }
+
+ /* 2. verify correct setup using alive packet */
+ err = nokia_send_alive_packet(hu);
+ if (err < 0) {
+ dev_err(dev, "Alive check failed: %d", err);
+ goto out;
+ }
+
+ /* 3. send firmware */
+ err = nokia_setup_fw(hu);
+ if (err < 0) {
+ dev_err(dev, "Could not setup FW: %d", err);
+ goto out;
+ }
+
+ nokia_flow_control(btdev->serdev, false);
+ serdev_device_set_baudrate(btdev->serdev, MAX_BAUD_RATE);
+ nokia_flow_control(btdev->serdev, true);
+
+ if (btdev->man_id == NOKIA_ID_BCM2048) {
+ hu->hdev->set_bdaddr = btbcm_set_bdaddr;
+ set_bit(HCI_QUIRK_INVALID_BDADDR, &hu->hdev->quirks);
+ dev_dbg(dev, "bcm2048 has invalid bluetooth address!");
+ }
+
+ dev_dbg(dev, "protocol setup done!");
+
+ gpiod_set_value_cansleep(btdev->wakeup_bt, 0);
+ pm_runtime_put(dev);
+ btdev->tx_enabled = false;
+ btdev->initialized = true;
+
+ return 0;
+out:
+ pm_runtime_put(dev);
+
+ return err;
+}
+
+static int nokia_open(struct hci_uart *hu)
+{
+ struct device *dev = &hu->serdev->dev;
+
+ dev_dbg(dev, "protocol open");
+
+ serdev_device_open(hu->serdev);
+
+ pm_runtime_enable(dev);
+
+ return 0;
+}
+
+static int nokia_flush(struct hci_uart *hu)
+{
+ struct nokia_bt_dev *btdev = hu->priv;
+
+ dev_dbg(&btdev->serdev->dev, "flush device");
+
+ skb_queue_purge(&btdev->txq);
+
+ return 0;
+}
+
+static int nokia_close(struct hci_uart *hu)
+{
+ struct nokia_bt_dev *btdev = hu->priv;
+ struct device *dev = &btdev->serdev->dev;
+
+ dev_dbg(dev, "close device");
+
+ btdev->initialized = false;
+
+ skb_queue_purge(&btdev->txq);
+
+ kfree_skb(btdev->rx_skb);
+
+ /* disable module */
+ gpiod_set_value(btdev->reset, 1);
+ gpiod_set_value(btdev->wakeup_bt, 0);
+
+ pm_runtime_disable(&btdev->serdev->dev);
+ serdev_device_close(btdev->serdev);
+
+ return 0;
+}
+
+/* Enqueue frame for transmittion (padding, crc, etc) */
+static int nokia_enqueue(struct hci_uart *hu, struct sk_buff *skb)
+{
+ struct nokia_bt_dev *btdev = hu->priv;
+ int err;
+
+ /* Prepend skb with frame type */
+ memcpy(skb_push(skb, 1), &bt_cb(skb)->pkt_type, 1);
+
+ /* Packets must be word aligned */
+ if (skb->len % 2) {
+ err = skb_pad(skb, 1);
+ if (err)
+ return err;
+ *skb_put(skb, 1) = 0x00;
+ }
+
+ skb_queue_tail(&btdev->txq, skb);
+
+ return 0;
+}
+
+static int nokia_recv_negotiation_packet(struct hci_dev *hdev,
+ struct sk_buff *skb)
+{
+ struct hci_uart *hu = hci_get_drvdata(hdev);
+ struct nokia_bt_dev *btdev = hu->priv;
+ struct device *dev = &btdev->serdev->dev;
+ struct hci_nokia_neg_hdr *hdr;
+ struct hci_nokia_neg_evt *evt;
+ int ret = 0;
+
+ hdr = (struct hci_nokia_neg_hdr *)skb->data;
+ if (hdr->dlen != sizeof(*evt)) {
+ btdev->init_error = -EIO;
+ ret = -EIO;
+ goto finish_neg;
+ }
+
+ evt = (struct hci_nokia_neg_evt *)skb_pull(skb, sizeof(*hdr));
+
+ if (evt->ack != NOKIA_NEG_ACK) {
+ dev_err(dev, "Negotiation received: wrong reply");
+ btdev->init_error = -EINVAL;
+ ret = -EINVAL;
+ goto finish_neg;
+ }
+
+ btdev->man_id = evt->man_id;
+ btdev->ver_id = evt->ver_id;
+
+ dev_dbg(dev, "Negotiation received: baud=%u:clk=%u:manu=%u:vers=%u",
+ evt->baud, evt->sys_clk, evt->man_id, evt->ver_id);
+
+finish_neg:
+ complete(&btdev->init_completion);
+ kfree_skb(skb);
+ return ret;
+}
+
+static int nokia_recv_alive_packet(struct hci_dev *hdev, struct sk_buff *skb)
+{
+ struct hci_uart *hu = hci_get_drvdata(hdev);
+ struct nokia_bt_dev *btdev = hu->priv;
+ struct device *dev = &btdev->serdev->dev;
+ struct hci_nokia_alive_hdr *hdr;
+ struct hci_nokia_alive_pkt *pkt;
+ int ret = 0;
+
+ hdr = (struct hci_nokia_alive_hdr *)skb->data;
+ if (hdr->dlen != sizeof(*pkt)) {
+ dev_err(dev, "Corrupted alive message");
+ btdev->init_error = -EIO;
+ ret = -EIO;
+ goto finish_alive;
+ }
+
+ pkt = (struct hci_nokia_alive_pkt *)skb_pull(skb, sizeof(*hdr));
+
+ if (pkt->mid != NOKIA_ALIVE_RESP) {
+ dev_err(dev, "Alive received: invalid response: 0x%02x!",
+ pkt->mid);
+ btdev->init_error = -EINVAL;
+ ret = -EINVAL;
+ goto finish_alive;
+ }
+
+ dev_dbg(dev, "Alive received");
+
+finish_alive:
+ complete(&btdev->init_completion);
+ kfree_skb(skb);
+ return ret;
+}
+
+static int nokia_recv_radio(struct hci_dev *hdev, struct sk_buff *skb)
+{
+ /* Packets received on the dedicated radio channel are
+ * HCI events and so feed them back into the core.
+ */
+ hci_skb_pkt_type(skb) = HCI_EVENT_PKT;
+ return hci_recv_frame(hdev, skb);
+}
+
+/* Recv data */
+static const struct h4_recv_pkt nokia_recv_pkts[] = {
+ { H4_RECV_ACL, .recv = hci_recv_frame },
+ { H4_RECV_SCO, .recv = hci_recv_frame },
+ { H4_RECV_EVENT, .recv = hci_recv_frame },
+ { NOKIA_RECV_ALIVE, .recv = nokia_recv_alive_packet },
+ { NOKIA_RECV_NEG, .recv = nokia_recv_negotiation_packet },
+ { NOKIA_RECV_RADIO, .recv = nokia_recv_radio },
+};
+
+static int nokia_recv(struct hci_uart *hu, const void *data, int count)
+{
+ struct nokia_bt_dev *btdev = hu->priv;
+ struct device *dev = &btdev->serdev->dev;
+ int err;
+
+ if (!test_bit(HCI_UART_REGISTERED, &hu->flags))
+ return -EUNATCH;
+
+ btdev->rx_skb = h4_recv_buf(hu->hdev, btdev->rx_skb, data, count,
+ nokia_recv_pkts, ARRAY_SIZE(nokia_recv_pkts));
+ if (IS_ERR(btdev->rx_skb)) {
+ err = PTR_ERR(btdev->rx_skb);
+ dev_err(dev, "Frame reassembly failed (%d)", err);
+ btdev->rx_skb = NULL;
+ return err;
+ }
+
+ return count;
+}
+
+static struct sk_buff *nokia_dequeue(struct hci_uart *hu)
+{
+ struct nokia_bt_dev *btdev = hu->priv;
+ struct device *dev = &btdev->serdev->dev;
+ struct sk_buff *result = skb_dequeue(&btdev->txq);
+
+ if (!btdev->initialized)
+ return result;
+
+ if (btdev->tx_enabled == !!result)
+ return result;
+
+ if (result) {
+ pm_runtime_get_sync(dev);
+ gpiod_set_value_cansleep(btdev->wakeup_bt, 1);
+ } else {
+ serdev_device_wait_until_sent(btdev->serdev, 0);
+ gpiod_set_value_cansleep(btdev->wakeup_bt, 0);
+ pm_runtime_put(dev);
+ }
+
+ btdev->tx_enabled = !!result;
+
+ return result;
+}
+
+static const struct hci_uart_proto nokia_proto = {
+ .id = HCI_UART_NOKIA,
+ .name = "Nokia",
+ .open = nokia_open,
+ .close = nokia_close,
+ .recv = nokia_recv,
+ .enqueue = nokia_enqueue,
+ .dequeue = nokia_dequeue,
+ .flush = nokia_flush,
+ .setup = nokia_setup,
+ .manufacturer = 1,
+};
+
+static int nokia_bluetooth_serdev_probe(struct serdev_device *serdev)
+{
+ struct device *dev = &serdev->dev;
+ struct nokia_bt_dev *btdev;
+ struct clk *sysclk;
+ int err = 0;
+
+ btdev = devm_kzalloc(dev, sizeof(*btdev), GFP_KERNEL);
+ if (!btdev)
+ return -ENOMEM;
+
+ btdev->hu.serdev = btdev->serdev = serdev;
+ serdev_device_set_drvdata(serdev, btdev);
+
+ btdev->reset = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
+ if (IS_ERR(btdev->reset)) {
+ err = PTR_ERR(btdev->reset);
+ dev_err(dev, "could not get reset gpio: %d", err);
+ return err;
+ }
+
+ btdev->wakeup_host = devm_gpiod_get(dev, "host-wakeup", GPIOD_IN);
+ if (IS_ERR(btdev->wakeup_host)) {
+ err = PTR_ERR(btdev->wakeup_host);
+ dev_err(dev, "could not get host wakeup gpio: %d", err);
+ return err;
+ }
+
+ btdev->wake_irq = gpiod_to_irq(btdev->wakeup_host);
+
+ err = devm_request_threaded_irq(dev, btdev->wake_irq, NULL,
+ wakeup_handler,
+ IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING | IRQF_ONESHOT,
+ "wakeup", btdev);
+ if (err) {
+ dev_err(dev, "could request wakeup irq: %d", err);
+ return err;
+ }
+
+ btdev->wakeup_bt = devm_gpiod_get(dev, "bluetooth-wakeup",
+ GPIOD_OUT_LOW);
+ if (IS_ERR(btdev->wakeup_bt)) {
+ err = PTR_ERR(btdev->wakeup_bt);
+ dev_err(dev, "could not get BT wakeup gpio: %d", err);
+ return err;
+ }
+
+ sysclk = devm_clk_get(dev, "sysclk");
+ if (IS_ERR(sysclk)) {
+ err = PTR_ERR(sysclk);
+ dev_err(dev, "could not get sysclk: %d", err);
+ return err;
+ }
+
+ clk_prepare_enable(sysclk);
+ btdev->sysclk_speed = clk_get_rate(sysclk);
+ clk_disable_unprepare(sysclk);
+
+ skb_queue_head_init(&btdev->txq);
+
+ btdev->hu.priv = btdev;
+ btdev->hu.alignment = 2; /* Nokia H4+ is word aligned */
+
+ err = hci_uart_register_device(&btdev->hu, &nokia_proto);
+ if (err) {
+ dev_err(dev, "could not register bluetooth uart: %d", err);
+ return err;
+ }
+
+ return 0;
+}
+
+static void nokia_bluetooth_serdev_remove(struct serdev_device *serdev)
+{
+ struct nokia_bt_dev *btdev = serdev_device_get_drvdata(serdev);
+ struct hci_uart *hu = &btdev->hu;
+ struct hci_dev *hdev = hu->hdev;
+
+ cancel_work_sync(&hu->write_work);
+
+ hci_unregister_dev(hdev);
+ hci_free_dev(hdev);
+ hu->proto->close(hu);
+
+ pm_runtime_disable(&btdev->serdev->dev);
+}
+
+static int nokia_bluetooth_runtime_suspend(struct device *dev)
+{
+ struct serdev_device *serdev = to_serdev_device(dev);
+
+ nokia_flow_control(serdev, false);
+ return 0;
+}
+
+static int nokia_bluetooth_runtime_resume(struct device *dev)
+{
+ struct serdev_device *serdev = to_serdev_device(dev);
+
+ nokia_flow_control(serdev, true);
+ return 0;
+}
+
+static const struct dev_pm_ops nokia_bluetooth_pm_ops = {
+ SET_RUNTIME_PM_OPS(nokia_bluetooth_runtime_suspend,
+ nokia_bluetooth_runtime_resume,
+ NULL)
+};
+
+#ifdef CONFIG_OF
+static const struct of_device_id nokia_bluetooth_of_match[] = {
+ { .compatible = "nokia,h4p-bluetooth", },
+ {},
+};
+MODULE_DEVICE_TABLE(of, nokia_bluetooth_of_match);
+#endif
+
+static struct serdev_device_driver nokia_bluetooth_serdev_driver = {
+ .probe = nokia_bluetooth_serdev_probe,
+ .remove = nokia_bluetooth_serdev_remove,
+ .driver = {
+ .name = "nokia-bluetooth",
+ .pm = &nokia_bluetooth_pm_ops,
+ .of_match_table = of_match_ptr(nokia_bluetooth_of_match),
+ },
+};
+
+module_serdev_device_driver(nokia_bluetooth_serdev_driver);
+
+MODULE_AUTHOR("Sebastian Reichel <sre@kernel.org>");
+MODULE_DESCRIPTION("Bluetooth HCI UART Nokia H4+ driver ver " VERSION);
+MODULE_VERSION(VERSION);
+MODULE_LICENSE("GPL");
diff --git a/drivers/bluetooth/hci_serdev.c b/drivers/bluetooth/hci_serdev.c
new file mode 100644
index 0000000..7de0edc
--- /dev/null
+++ b/drivers/bluetooth/hci_serdev.c
@@ -0,0 +1,356 @@
+/*
+ * Bluetooth HCI serdev driver lib
+ *
+ * Copyright (C) 2017 Linaro, Ltd., Rob Herring <robh@kernel.org>
+ *
+ * Based on hci_ldisc.c:
+ *
+ * Copyright (C) 2000-2001 Qualcomm Incorporated
+ * Copyright (C) 2002-2003 Maxim Krasnyansky <maxk@qualcomm.com>
+ * Copyright (C) 2004-2005 Marcel Holtmann <marcel@holtmann.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/serdev.h>
+#include <linux/skbuff.h>
+
+#include <net/bluetooth/bluetooth.h>
+#include <net/bluetooth/hci_core.h>
+
+#include "hci_uart.h"
+
+struct serdev_device_ops hci_serdev_client_ops;
+
+static inline void hci_uart_tx_complete(struct hci_uart *hu, int pkt_type)
+{
+ struct hci_dev *hdev = hu->hdev;
+
+ /* Update HCI stat counters */
+ switch (pkt_type) {
+ case HCI_COMMAND_PKT:
+ hdev->stat.cmd_tx++;
+ break;
+
+ case HCI_ACLDATA_PKT:
+ hdev->stat.acl_tx++;
+ break;
+
+ case HCI_SCODATA_PKT:
+ hdev->stat.sco_tx++;
+ break;
+ }
+}
+
+static inline struct sk_buff *hci_uart_dequeue(struct hci_uart *hu)
+{
+ struct sk_buff *skb = hu->tx_skb;
+
+ if (!skb)
+ skb = hu->proto->dequeue(hu);
+ else
+ hu->tx_skb = NULL;
+
+ return skb;
+}
+
+static void hci_uart_write_work(struct work_struct *work)
+{
+ struct hci_uart *hu = container_of(work, struct hci_uart, write_work);
+ struct serdev_device *serdev = hu->serdev;
+ struct hci_dev *hdev = hu->hdev;
+ struct sk_buff *skb;
+
+ /* REVISIT:
+ * should we cope with bad skbs or ->write() returning an error value?
+ */
+ do {
+ clear_bit(HCI_UART_TX_WAKEUP, &hu->tx_state);
+
+ while ((skb = hci_uart_dequeue(hu))) {
+ int len;
+
+ len = serdev_device_write_buf(serdev,
+ skb->data, skb->len);
+ hdev->stat.byte_tx += len;
+
+ skb_pull(skb, len);
+ if (skb->len) {
+ hu->tx_skb = skb;
+ break;
+ }
+
+ hci_uart_tx_complete(hu, hci_skb_pkt_type(skb));
+ kfree_skb(skb);
+ }
+ } while(test_bit(HCI_UART_TX_WAKEUP, &hu->tx_state));
+
+ clear_bit(HCI_UART_SENDING, &hu->tx_state);
+}
+
+/* ------- Interface to HCI layer ------ */
+
+/* Initialize device */
+static int hci_uart_open(struct hci_dev *hdev)
+{
+ BT_DBG("%s %p", hdev->name, hdev);
+
+ return 0;
+}
+
+/* Reset device */
+static int hci_uart_flush(struct hci_dev *hdev)
+{
+ struct hci_uart *hu = hci_get_drvdata(hdev);
+
+ BT_DBG("hdev %p serdev %p", hdev, hu->serdev);
+
+ if (hu->tx_skb) {
+ kfree_skb(hu->tx_skb); hu->tx_skb = NULL;
+ }
+
+ /* Flush any pending characters in the driver and discipline. */
+ serdev_device_write_flush(hu->serdev);
+
+ if (test_bit(HCI_UART_PROTO_READY, &hu->flags))
+ hu->proto->flush(hu);
+
+ return 0;
+}
+
+/* Close device */
+static int hci_uart_close(struct hci_dev *hdev)
+{
+ BT_DBG("hdev %p", hdev);
+
+ hci_uart_flush(hdev);
+ hdev->flush = NULL;
+
+ return 0;
+}
+
+/* Send frames from HCI layer */
+static int hci_uart_send_frame(struct hci_dev *hdev, struct sk_buff *skb)
+{
+ struct hci_uart *hu = hci_get_drvdata(hdev);
+
+ BT_DBG("%s: type %d len %d", hdev->name, hci_skb_pkt_type(skb),
+ skb->len);
+
+ hu->proto->enqueue(hu, skb);
+
+ hci_uart_tx_wakeup(hu);
+
+ return 0;
+}
+
+static int hci_uart_setup(struct hci_dev *hdev)
+{
+ struct hci_uart *hu = hci_get_drvdata(hdev);
+ struct hci_rp_read_local_version *ver;
+ struct sk_buff *skb;
+ unsigned int speed;
+ int err;
+
+ /* Init speed if any */
+ if (hu->init_speed)
+ speed = hu->init_speed;
+ else if (hu->proto->init_speed)
+ speed = hu->proto->init_speed;
+ else
+ speed = 0;
+
+ if (speed)
+ serdev_device_set_baudrate(hu->serdev, speed);
+
+ /* Operational speed if any */
+ if (hu->oper_speed)
+ speed = hu->oper_speed;
+ else if (hu->proto->oper_speed)
+ speed = hu->proto->oper_speed;
+ else
+ speed = 0;
+
+ if (hu->proto->set_baudrate && speed) {
+ err = hu->proto->set_baudrate(hu, speed);
+ if (err)
+ BT_ERR("%s: failed to set baudrate", hdev->name);
+ else
+ serdev_device_set_baudrate(hu->serdev, speed);
+ }
+
+ if (hu->proto->setup)
+ return hu->proto->setup(hu);
+
+ if (!test_bit(HCI_UART_VND_DETECT, &hu->hdev_flags))
+ return 0;
+
+ skb = __hci_cmd_sync(hdev, HCI_OP_READ_LOCAL_VERSION, 0, NULL,
+ HCI_INIT_TIMEOUT);
+ if (IS_ERR(skb)) {
+ BT_ERR("%s: Reading local version information failed (%ld)",
+ hdev->name, PTR_ERR(skb));
+ return 0;
+ }
+
+ if (skb->len != sizeof(*ver)) {
+ BT_ERR("%s: Event length mismatch for version information",
+ hdev->name);
+ }
+
+ kfree_skb(skb);
+ return 0;
+}
+
+/** hci_uart_write_wakeup - transmit buffer wakeup
+ * @serdev: serial device
+ *
+ * This function is called by the serdev framework when it accepts
+ * more data being sent.
+ */
+static void hci_uart_write_wakeup(struct serdev_device *serdev)
+{
+ struct hci_uart *hu = serdev_device_get_drvdata(serdev);
+
+ BT_DBG("");
+
+ if (!hu || serdev != hu->serdev) {
+ WARN_ON(1);
+ return;
+ }
+
+ if (test_bit(HCI_UART_PROTO_READY, &hu->flags))
+ hci_uart_tx_wakeup(hu);
+}
+
+/** hci_uart_receive_buf - receive buffer wakeup
+ * @serdev: serial device
+ * @data: pointer to received data
+ * @count: count of received data in bytes
+ *
+ * This function is called by the serdev framework when it received data
+ * in the RX buffer.
+ *
+ * Return: number of processed bytes
+ */
+static int hci_uart_receive_buf(struct serdev_device *serdev, const u8 *data,
+ size_t count)
+{
+ struct hci_uart *hu = serdev_device_get_drvdata(serdev);
+
+ if (!hu || serdev != hu->serdev) {
+ WARN_ON(1);
+ return 0;
+ }
+
+ if (!test_bit(HCI_UART_PROTO_READY, &hu->flags))
+ return 0;
+
+ /* It does not need a lock here as it is already protected by a mutex in
+ * tty caller
+ */
+ hu->proto->recv(hu, data, count);
+
+ if (hu->hdev)
+ hu->hdev->stat.byte_rx += count;
+
+ return count;
+}
+
+struct serdev_device_ops hci_serdev_client_ops = {
+ .receive_buf = hci_uart_receive_buf,
+ .write_wakeup = hci_uart_write_wakeup,
+};
+
+int hci_uart_register_device(struct hci_uart *hu,
+ const struct hci_uart_proto *p)
+{
+ int err;
+ struct hci_dev *hdev;
+
+ BT_DBG("");
+
+ serdev_device_set_client_ops(hu->serdev, &hci_serdev_client_ops);
+
+ err = p->open(hu);
+ if (err)
+ return err;
+
+ hu->proto = p;
+ set_bit(HCI_UART_PROTO_READY, &hu->flags);
+
+ /* Initialize and register HCI device */
+ hdev = hci_alloc_dev();
+ if (!hdev) {
+ BT_ERR("Can't allocate HCI device");
+ err = -ENOMEM;
+ goto err_alloc;
+ }
+
+ hu->hdev = hdev;
+
+ hdev->bus = HCI_UART;
+ hci_set_drvdata(hdev, hu);
+
+ INIT_WORK(&hu->write_work, hci_uart_write_work);
+
+ /* Only when vendor specific setup callback is provided, consider
+ * the manufacturer information valid. This avoids filling in the
+ * value for Ericsson when nothing is specified.
+ */
+ if (hu->proto->setup)
+ hdev->manufacturer = hu->proto->manufacturer;
+
+ hdev->open = hci_uart_open;
+ hdev->close = hci_uart_close;
+ hdev->flush = hci_uart_flush;
+ hdev->send = hci_uart_send_frame;
+ hdev->setup = hci_uart_setup;
+ SET_HCIDEV_DEV(hdev, &hu->serdev->dev);
+
+ if (test_bit(HCI_UART_RAW_DEVICE, &hu->hdev_flags))
+ set_bit(HCI_QUIRK_RAW_DEVICE, &hdev->quirks);
+
+ if (test_bit(HCI_UART_EXT_CONFIG, &hu->hdev_flags))
+ set_bit(HCI_QUIRK_EXTERNAL_CONFIG, &hdev->quirks);
+
+ if (!test_bit(HCI_UART_RESET_ON_INIT, &hu->hdev_flags))
+ set_bit(HCI_QUIRK_RESET_ON_CLOSE, &hdev->quirks);
+
+ if (test_bit(HCI_UART_CREATE_AMP, &hu->hdev_flags))
+ hdev->dev_type = HCI_AMP;
+ else
+ hdev->dev_type = HCI_PRIMARY;
+
+ if (test_bit(HCI_UART_INIT_PENDING, &hu->hdev_flags))
+ return 0;
+
+ if (hci_register_dev(hdev) < 0) {
+ BT_ERR("Can't register HCI device");
+ err = -ENODEV;
+ goto err_register;
+ }
+
+ set_bit(HCI_UART_REGISTERED, &hu->flags);
+
+ return 0;
+
+err_register:
+ hci_free_dev(hdev);
+err_alloc:
+ clear_bit(HCI_UART_PROTO_READY, &hu->flags);
+ p->close(hu);
+ return err;
+}
+EXPORT_SYMBOL_GPL(hci_uart_register_device);
diff --git a/drivers/bluetooth/hci_uart.h b/drivers/bluetooth/hci_uart.h
index 0701395..2b05e55 100644
--- a/drivers/bluetooth/hci_uart.h
+++ b/drivers/bluetooth/hci_uart.h
@@ -58,6 +58,7 @@
#define HCI_UART_VND_DETECT 5
struct hci_uart;
+struct serdev_device;
struct hci_uart_proto {
unsigned int id;
@@ -77,6 +78,7 @@
struct hci_uart {
struct tty_struct *tty;
+ struct serdev_device *serdev;
struct hci_dev *hdev;
unsigned long flags;
unsigned long hdev_flags;
@@ -92,6 +94,9 @@
unsigned int init_speed;
unsigned int oper_speed;
+
+ u8 alignment;
+ u8 padding;
};
/* HCI_UART proto flag bits */
@@ -105,9 +110,10 @@
int hci_uart_register_proto(const struct hci_uart_proto *p);
int hci_uart_unregister_proto(const struct hci_uart_proto *p);
+int hci_uart_register_device(struct hci_uart *hu, const struct hci_uart_proto *p);
+
int hci_uart_tx_wakeup(struct hci_uart *hu);
int hci_uart_init_ready(struct hci_uart *hu);
-void hci_uart_init_tty(struct hci_uart *hu);
void hci_uart_set_baudrate(struct hci_uart *hu, unsigned int speed);
void hci_uart_set_flow_control(struct hci_uart *hu, bool enable);
void hci_uart_set_speeds(struct hci_uart *hu, unsigned int init_speed,
diff --git a/drivers/cdrom/cdrom.c b/drivers/cdrom/cdrom.c
index 8773964..76c952f 100644
--- a/drivers/cdrom/cdrom.c
+++ b/drivers/cdrom/cdrom.c
@@ -2218,7 +2218,8 @@
rq->timeout = 60 * HZ;
bio = rq->bio;
- if (blk_execute_rq(q, cdi->disk, rq, 0)) {
+ blk_execute_rq(q, cdi->disk, rq, 0);
+ if (scsi_req(rq)->result) {
struct request_sense *s = req->sense;
ret = -EIO;
cdi->last_sense = s->sense_key;
diff --git a/drivers/char/agp/amd64-agp.c b/drivers/char/agp/amd64-agp.c
index 0ef3500..c99cd19 100644
--- a/drivers/char/agp/amd64-agp.c
+++ b/drivers/char/agp/amd64-agp.c
@@ -14,7 +14,7 @@
#include <linux/agp_backend.h>
#include <linux/mmzone.h>
#include <asm/page.h> /* PAGE_SIZE */
-#include <asm/e820.h>
+#include <asm/e820/api.h>
#include <asm/amd_nb.h>
#include <asm/gart.h>
#include "agp.h"
diff --git a/drivers/char/hw_random/Kconfig b/drivers/char/hw_random/Kconfig
index 0cafe08..1b223c3 100644
--- a/drivers/char/hw_random/Kconfig
+++ b/drivers/char/hw_random/Kconfig
@@ -294,20 +294,6 @@
If unsure, say Y.
-config HW_RANDOM_EXYNOS
- tristate "EXYNOS HW random number generator support"
- depends on ARCH_EXYNOS || COMPILE_TEST
- depends on HAS_IOMEM
- default HW_RANDOM
- ---help---
- This driver provides kernel-side support for the Random Number
- Generator hardware found on EXYNOS SOCs.
-
- To compile this driver as a module, choose M here: the
- module will be called exynos-rng.
-
- If unsure, say Y.
-
config HW_RANDOM_TPM
tristate "TPM HW Random Number Generator support"
depends on TCG_TPM
@@ -423,6 +409,34 @@
If unsure, say Y.
+config HW_RANDOM_MTK
+ tristate "Mediatek Random Number Generator support"
+ depends on HW_RANDOM
+ depends on ARCH_MEDIATEK || COMPILE_TEST
+ default y
+ ---help---
+ This driver provides kernel-side support for the Random Number
+ Generator hardware found on Mediatek SoCs.
+
+ To compile this driver as a module, choose M here. the
+ module will be called mtk-rng.
+
+ If unsure, say Y.
+
+config HW_RANDOM_S390
+ tristate "S390 True Random Number Generator support"
+ depends on S390
+ default HW_RANDOM
+ ---help---
+ This driver provides kernel-side support for the True
+ Random Number Generator available as CPACF extension
+ on modern s390 hardware platforms.
+
+ To compile this driver as a module, choose M here: the
+ module will be called s390-trng.
+
+ If unsure, say Y.
+
endif # HW_RANDOM
config UML_RANDOM
diff --git a/drivers/char/hw_random/Makefile b/drivers/char/hw_random/Makefile
index 5f52b1e..b085975 100644
--- a/drivers/char/hw_random/Makefile
+++ b/drivers/char/hw_random/Makefile
@@ -24,7 +24,6 @@
obj-$(CONFIG_HW_RANDOM_NOMADIK) += nomadik-rng.o
obj-$(CONFIG_HW_RANDOM_PSERIES) += pseries-rng.o
obj-$(CONFIG_HW_RANDOM_POWERNV) += powernv-rng.o
-obj-$(CONFIG_HW_RANDOM_EXYNOS) += exynos-rng.o
obj-$(CONFIG_HW_RANDOM_HISI) += hisi-rng.o
obj-$(CONFIG_HW_RANDOM_TPM) += tpm-rng.o
obj-$(CONFIG_HW_RANDOM_BCM2835) += bcm2835-rng.o
@@ -36,3 +35,5 @@
obj-$(CONFIG_HW_RANDOM_PIC32) += pic32-rng.o
obj-$(CONFIG_HW_RANDOM_MESON) += meson-rng.o
obj-$(CONFIG_HW_RANDOM_CAVIUM) += cavium-rng.o cavium-rng-vf.o
+obj-$(CONFIG_HW_RANDOM_MTK) += mtk-rng.o
+obj-$(CONFIG_HW_RANDOM_S390) += s390-trng.o
diff --git a/drivers/char/hw_random/exynos-rng.c b/drivers/char/hw_random/exynos-rng.c
deleted file mode 100644
index 23d3585..0000000
--- a/drivers/char/hw_random/exynos-rng.c
+++ /dev/null
@@ -1,231 +0,0 @@
-/*
- * exynos-rng.c - Random Number Generator driver for the exynos
- *
- * Copyright (C) 2012 Samsung Electronics
- * Jonghwa Lee <jonghwa3.lee@samsung.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation;
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
- *
- */
-
-#include <linux/hw_random.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/io.h>
-#include <linux/platform_device.h>
-#include <linux/clk.h>
-#include <linux/pm_runtime.h>
-#include <linux/err.h>
-
-#define EXYNOS_PRNG_STATUS_OFFSET 0x10
-#define EXYNOS_PRNG_SEED_OFFSET 0x140
-#define EXYNOS_PRNG_OUT1_OFFSET 0x160
-#define SEED_SETTING_DONE BIT(1)
-#define PRNG_START 0x18
-#define PRNG_DONE BIT(5)
-#define EXYNOS_AUTOSUSPEND_DELAY 100
-
-struct exynos_rng {
- struct device *dev;
- struct hwrng rng;
- void __iomem *mem;
- struct clk *clk;
-};
-
-static u32 exynos_rng_readl(struct exynos_rng *rng, u32 offset)
-{
- return readl_relaxed(rng->mem + offset);
-}
-
-static void exynos_rng_writel(struct exynos_rng *rng, u32 val, u32 offset)
-{
- writel_relaxed(val, rng->mem + offset);
-}
-
-static int exynos_rng_configure(struct exynos_rng *exynos_rng)
-{
- int i;
- int ret = 0;
-
- for (i = 0 ; i < 5 ; i++)
- exynos_rng_writel(exynos_rng, jiffies,
- EXYNOS_PRNG_SEED_OFFSET + 4*i);
-
- if (!(exynos_rng_readl(exynos_rng, EXYNOS_PRNG_STATUS_OFFSET)
- & SEED_SETTING_DONE))
- ret = -EIO;
-
- return ret;
-}
-
-static int exynos_init(struct hwrng *rng)
-{
- struct exynos_rng *exynos_rng = container_of(rng,
- struct exynos_rng, rng);
- int ret = 0;
-
- pm_runtime_get_sync(exynos_rng->dev);
- ret = exynos_rng_configure(exynos_rng);
- pm_runtime_mark_last_busy(exynos_rng->dev);
- pm_runtime_put_autosuspend(exynos_rng->dev);
-
- return ret;
-}
-
-static int exynos_read(struct hwrng *rng, void *buf,
- size_t max, bool wait)
-{
- struct exynos_rng *exynos_rng = container_of(rng,
- struct exynos_rng, rng);
- u32 *data = buf;
- int retry = 100;
- int ret = 4;
-
- pm_runtime_get_sync(exynos_rng->dev);
-
- exynos_rng_writel(exynos_rng, PRNG_START, 0);
-
- while (!(exynos_rng_readl(exynos_rng,
- EXYNOS_PRNG_STATUS_OFFSET) & PRNG_DONE) && --retry)
- cpu_relax();
- if (!retry) {
- ret = -ETIMEDOUT;
- goto out;
- }
-
- exynos_rng_writel(exynos_rng, PRNG_DONE, EXYNOS_PRNG_STATUS_OFFSET);
-
- *data = exynos_rng_readl(exynos_rng, EXYNOS_PRNG_OUT1_OFFSET);
-
-out:
- pm_runtime_mark_last_busy(exynos_rng->dev);
- pm_runtime_put_sync_autosuspend(exynos_rng->dev);
-
- return ret;
-}
-
-static int exynos_rng_probe(struct platform_device *pdev)
-{
- struct exynos_rng *exynos_rng;
- struct resource *res;
- int ret;
-
- exynos_rng = devm_kzalloc(&pdev->dev, sizeof(struct exynos_rng),
- GFP_KERNEL);
- if (!exynos_rng)
- return -ENOMEM;
-
- exynos_rng->dev = &pdev->dev;
- exynos_rng->rng.name = "exynos";
- exynos_rng->rng.init = exynos_init;
- exynos_rng->rng.read = exynos_read;
- exynos_rng->clk = devm_clk_get(&pdev->dev, "secss");
- if (IS_ERR(exynos_rng->clk)) {
- dev_err(&pdev->dev, "Couldn't get clock.\n");
- return -ENOENT;
- }
-
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- exynos_rng->mem = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(exynos_rng->mem))
- return PTR_ERR(exynos_rng->mem);
-
- platform_set_drvdata(pdev, exynos_rng);
-
- pm_runtime_set_autosuspend_delay(&pdev->dev, EXYNOS_AUTOSUSPEND_DELAY);
- pm_runtime_use_autosuspend(&pdev->dev);
- pm_runtime_enable(&pdev->dev);
-
- ret = devm_hwrng_register(&pdev->dev, &exynos_rng->rng);
- if (ret) {
- pm_runtime_dont_use_autosuspend(&pdev->dev);
- pm_runtime_disable(&pdev->dev);
- }
-
- return ret;
-}
-
-static int exynos_rng_remove(struct platform_device *pdev)
-{
- pm_runtime_dont_use_autosuspend(&pdev->dev);
- pm_runtime_disable(&pdev->dev);
-
- return 0;
-}
-
-static int __maybe_unused exynos_rng_runtime_suspend(struct device *dev)
-{
- struct platform_device *pdev = to_platform_device(dev);
- struct exynos_rng *exynos_rng = platform_get_drvdata(pdev);
-
- clk_disable_unprepare(exynos_rng->clk);
-
- return 0;
-}
-
-static int __maybe_unused exynos_rng_runtime_resume(struct device *dev)
-{
- struct platform_device *pdev = to_platform_device(dev);
- struct exynos_rng *exynos_rng = platform_get_drvdata(pdev);
-
- return clk_prepare_enable(exynos_rng->clk);
-}
-
-static int __maybe_unused exynos_rng_suspend(struct device *dev)
-{
- return pm_runtime_force_suspend(dev);
-}
-
-static int __maybe_unused exynos_rng_resume(struct device *dev)
-{
- struct platform_device *pdev = to_platform_device(dev);
- struct exynos_rng *exynos_rng = platform_get_drvdata(pdev);
- int ret;
-
- ret = pm_runtime_force_resume(dev);
- if (ret)
- return ret;
-
- return exynos_rng_configure(exynos_rng);
-}
-
-static const struct dev_pm_ops exynos_rng_pm_ops = {
- SET_SYSTEM_SLEEP_PM_OPS(exynos_rng_suspend, exynos_rng_resume)
- SET_RUNTIME_PM_OPS(exynos_rng_runtime_suspend,
- exynos_rng_runtime_resume, NULL)
-};
-
-static const struct of_device_id exynos_rng_dt_match[] = {
- {
- .compatible = "samsung,exynos4-rng",
- },
- { },
-};
-MODULE_DEVICE_TABLE(of, exynos_rng_dt_match);
-
-static struct platform_driver exynos_rng_driver = {
- .driver = {
- .name = "exynos-rng",
- .pm = &exynos_rng_pm_ops,
- .of_match_table = exynos_rng_dt_match,
- },
- .probe = exynos_rng_probe,
- .remove = exynos_rng_remove,
-};
-
-module_platform_driver(exynos_rng_driver);
-
-MODULE_DESCRIPTION("EXYNOS 4 H/W Random Number Generator driver");
-MODULE_AUTHOR("Jonghwa Lee <jonghwa3.lee@samsung.com>");
-MODULE_LICENSE("GPL");
diff --git a/drivers/char/hw_random/meson-rng.c b/drivers/char/hw_random/meson-rng.c
index 119d698..2e23be8 100644
--- a/drivers/char/hw_random/meson-rng.c
+++ b/drivers/char/hw_random/meson-rng.c
@@ -62,6 +62,7 @@
#include <linux/slab.h>
#include <linux/types.h>
#include <linux/of.h>
+#include <linux/clk.h>
#define RNG_DATA 0x00
@@ -69,6 +70,7 @@
void __iomem *base;
struct platform_device *pdev;
struct hwrng rng;
+ struct clk *core_clk;
};
static int meson_rng_read(struct hwrng *rng, void *buf, size_t max, bool wait)
@@ -81,11 +83,17 @@
return sizeof(u32);
}
+static void meson_rng_clk_disable(void *data)
+{
+ clk_disable_unprepare(data);
+}
+
static int meson_rng_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct meson_rng_data *data;
struct resource *res;
+ int ret;
data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
if (!data)
@@ -98,6 +106,20 @@
if (IS_ERR(data->base))
return PTR_ERR(data->base);
+ data->core_clk = devm_clk_get(dev, "core");
+ if (IS_ERR(data->core_clk))
+ data->core_clk = NULL;
+
+ if (data->core_clk) {
+ ret = clk_prepare_enable(data->core_clk);
+ if (ret)
+ return ret;
+ ret = devm_add_action_or_reset(dev, meson_rng_clk_disable,
+ data->core_clk);
+ if (ret)
+ return ret;
+ }
+
data->rng.name = pdev->name;
data->rng.read = meson_rng_read;
diff --git a/drivers/char/hw_random/mtk-rng.c b/drivers/char/hw_random/mtk-rng.c
new file mode 100644
index 0000000..df8eb54
--- /dev/null
+++ b/drivers/char/hw_random/mtk-rng.c
@@ -0,0 +1,168 @@
+/*
+ * Driver for Mediatek Hardware Random Number Generator
+ *
+ * Copyright (C) 2017 Sean Wang <sean.wang@mediatek.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of
+ * the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+#define MTK_RNG_DEV KBUILD_MODNAME
+
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/err.h>
+#include <linux/hw_random.h>
+#include <linux/io.h>
+#include <linux/iopoll.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+
+#define USEC_POLL 2
+#define TIMEOUT_POLL 20
+
+#define RNG_CTRL 0x00
+#define RNG_EN BIT(0)
+#define RNG_READY BIT(31)
+
+#define RNG_DATA 0x08
+
+#define to_mtk_rng(p) container_of(p, struct mtk_rng, rng)
+
+struct mtk_rng {
+ void __iomem *base;
+ struct clk *clk;
+ struct hwrng rng;
+};
+
+static int mtk_rng_init(struct hwrng *rng)
+{
+ struct mtk_rng *priv = to_mtk_rng(rng);
+ u32 val;
+ int err;
+
+ err = clk_prepare_enable(priv->clk);
+ if (err)
+ return err;
+
+ val = readl(priv->base + RNG_CTRL);
+ val |= RNG_EN;
+ writel(val, priv->base + RNG_CTRL);
+
+ return 0;
+}
+
+static void mtk_rng_cleanup(struct hwrng *rng)
+{
+ struct mtk_rng *priv = to_mtk_rng(rng);
+ u32 val;
+
+ val = readl(priv->base + RNG_CTRL);
+ val &= ~RNG_EN;
+ writel(val, priv->base + RNG_CTRL);
+
+ clk_disable_unprepare(priv->clk);
+}
+
+static bool mtk_rng_wait_ready(struct hwrng *rng, bool wait)
+{
+ struct mtk_rng *priv = to_mtk_rng(rng);
+ int ready;
+
+ ready = readl(priv->base + RNG_CTRL) & RNG_READY;
+ if (!ready && wait)
+ readl_poll_timeout_atomic(priv->base + RNG_CTRL, ready,
+ ready & RNG_READY, USEC_POLL,
+ TIMEOUT_POLL);
+ return !!ready;
+}
+
+static int mtk_rng_read(struct hwrng *rng, void *buf, size_t max, bool wait)
+{
+ struct mtk_rng *priv = to_mtk_rng(rng);
+ int retval = 0;
+
+ while (max >= sizeof(u32)) {
+ if (!mtk_rng_wait_ready(rng, wait))
+ break;
+
+ *(u32 *)buf = readl(priv->base + RNG_DATA);
+ retval += sizeof(u32);
+ buf += sizeof(u32);
+ max -= sizeof(u32);
+ }
+
+ return retval || !wait ? retval : -EIO;
+}
+
+static int mtk_rng_probe(struct platform_device *pdev)
+{
+ struct resource *res;
+ int ret;
+ struct mtk_rng *priv;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!res) {
+ dev_err(&pdev->dev, "no iomem resource\n");
+ return -ENXIO;
+ }
+
+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ priv->rng.name = pdev->name;
+ priv->rng.init = mtk_rng_init;
+ priv->rng.cleanup = mtk_rng_cleanup;
+ priv->rng.read = mtk_rng_read;
+
+ priv->clk = devm_clk_get(&pdev->dev, "rng");
+ if (IS_ERR(priv->clk)) {
+ ret = PTR_ERR(priv->clk);
+ dev_err(&pdev->dev, "no clock for device: %d\n", ret);
+ return ret;
+ }
+
+ priv->base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(priv->base))
+ return PTR_ERR(priv->base);
+
+ ret = devm_hwrng_register(&pdev->dev, &priv->rng);
+ if (ret) {
+ dev_err(&pdev->dev, "failed to register rng device: %d\n",
+ ret);
+ return ret;
+ }
+
+ dev_info(&pdev->dev, "registered RNG driver\n");
+
+ return 0;
+}
+
+static const struct of_device_id mtk_rng_match[] = {
+ { .compatible = "mediatek,mt7623-rng" },
+ {},
+};
+MODULE_DEVICE_TABLE(of, mtk_rng_match);
+
+static struct platform_driver mtk_rng_driver = {
+ .probe = mtk_rng_probe,
+ .driver = {
+ .name = MTK_RNG_DEV,
+ .of_match_table = mtk_rng_match,
+ },
+};
+
+module_platform_driver(mtk_rng_driver);
+
+MODULE_DESCRIPTION("Mediatek Random Number Generator Driver");
+MODULE_AUTHOR("Sean Wang <sean.wang@mediatek.com>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/char/hw_random/n2-drv.c b/drivers/char/hw_random/n2-drv.c
index 31cbdbb..92dd4e9 100644
--- a/drivers/char/hw_random/n2-drv.c
+++ b/drivers/char/hw_random/n2-drv.c
@@ -748,9 +748,7 @@
dev_info(&op->dev, "Registered RNG HVAPI major %lu minor %lu\n",
np->hvapi_major, np->hvapi_minor);
-
- np->units = devm_kzalloc(&op->dev,
- sizeof(struct n2rng_unit) * np->num_units,
+ np->units = devm_kcalloc(&op->dev, np->num_units, sizeof(*np->units),
GFP_KERNEL);
err = -ENOMEM;
if (!np->units)
diff --git a/drivers/char/hw_random/omap-rng.c b/drivers/char/hw_random/omap-rng.c
index b1ad125..74d11ae 100644
--- a/drivers/char/hw_random/omap-rng.c
+++ b/drivers/char/hw_random/omap-rng.c
@@ -398,16 +398,6 @@
return err;
}
- priv->clk = devm_clk_get(&pdev->dev, NULL);
- if (IS_ERR(priv->clk) && PTR_ERR(priv->clk) == -EPROBE_DEFER)
- return -EPROBE_DEFER;
- if (!IS_ERR(priv->clk)) {
- err = clk_prepare_enable(priv->clk);
- if (err)
- dev_err(&pdev->dev, "unable to enable the clk, "
- "err = %d\n", err);
- }
-
/*
* On OMAP4, enabling the shutdown_oflo interrupt is
* done in the interrupt mask register. There is no
@@ -478,6 +468,18 @@
goto err_ioremap;
}
+ priv->clk = devm_clk_get(&pdev->dev, NULL);
+ if (IS_ERR(priv->clk) && PTR_ERR(priv->clk) == -EPROBE_DEFER)
+ return -EPROBE_DEFER;
+ if (!IS_ERR(priv->clk)) {
+ ret = clk_prepare_enable(priv->clk);
+ if (ret) {
+ dev_err(&pdev->dev,
+ "Unable to enable the clk: %d\n", ret);
+ goto err_register;
+ }
+ }
+
ret = (dev->of_node) ? of_get_omap_rng_device_details(priv, pdev) :
get_omap_rng_device_details(priv);
if (ret)
diff --git a/drivers/char/hw_random/s390-trng.c b/drivers/char/hw_random/s390-trng.c
new file mode 100644
index 0000000..aca48e8
--- /dev/null
+++ b/drivers/char/hw_random/s390-trng.c
@@ -0,0 +1,268 @@
+/*
+ * s390 TRNG device driver
+ *
+ * Driver for the TRNG (true random number generation) command
+ * available via CPACF extension MSA 7 on the s390 arch.
+
+ * Copyright IBM Corp. 2017
+ * Author(s): Harald Freudenberger <freude@de.ibm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License (version 2 only)
+ * as published by the Free Software Foundation.
+ *
+ */
+
+#define KMSG_COMPONENT "trng"
+#define pr_fmt(fmt) KMSG_COMPONENT ": " fmt
+
+#include <linux/hw_random.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/cpufeature.h>
+#include <linux/miscdevice.h>
+#include <linux/debugfs.h>
+#include <linux/atomic.h>
+#include <linux/random.h>
+#include <linux/sched/signal.h>
+#include <asm/debug.h>
+#include <asm/cpacf.h>
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("IBM Corporation");
+MODULE_DESCRIPTION("s390 CPACF TRNG device driver");
+
+
+/* trng related debug feature things */
+
+static debug_info_t *debug_info;
+
+#define DEBUG_DBG(...) debug_sprintf_event(debug_info, 6, ##__VA_ARGS__)
+#define DEBUG_INFO(...) debug_sprintf_event(debug_info, 5, ##__VA_ARGS__)
+#define DEBUG_WARN(...) debug_sprintf_event(debug_info, 4, ##__VA_ARGS__)
+#define DEBUG_ERR(...) debug_sprintf_event(debug_info, 3, ##__VA_ARGS__)
+
+
+/* trng helpers */
+
+static atomic64_t trng_dev_counter = ATOMIC64_INIT(0);
+static atomic64_t trng_hwrng_counter = ATOMIC64_INIT(0);
+
+
+/* file io functions */
+
+static int trng_open(struct inode *inode, struct file *file)
+{
+ return nonseekable_open(inode, file);
+}
+
+static ssize_t trng_read(struct file *file, char __user *ubuf,
+ size_t nbytes, loff_t *ppos)
+{
+ u8 buf[32];
+ u8 *p = buf;
+ unsigned int n;
+ ssize_t ret = 0;
+
+ /*
+ * use buf for requests <= sizeof(buf),
+ * otherwise allocate one page and fetch
+ * pagewise.
+ */
+
+ if (nbytes > sizeof(buf)) {
+ p = (u8 *) __get_free_page(GFP_KERNEL);
+ if (!p)
+ return -ENOMEM;
+ }
+
+ while (nbytes) {
+ if (need_resched()) {
+ if (signal_pending(current)) {
+ if (ret == 0)
+ ret = -ERESTARTSYS;
+ break;
+ }
+ schedule();
+ }
+ n = nbytes > PAGE_SIZE ? PAGE_SIZE : nbytes;
+ cpacf_trng(NULL, 0, p, n);
+ atomic64_add(n, &trng_dev_counter);
+ if (copy_to_user(ubuf, p, n)) {
+ ret = -EFAULT;
+ break;
+ }
+ nbytes -= n;
+ ubuf += n;
+ ret += n;
+ }
+
+ if (p != buf)
+ free_page((unsigned long) p);
+
+ DEBUG_DBG("trng_read()=%zd\n", ret);
+ return ret;
+}
+
+
+/* sysfs */
+
+static ssize_t trng_counter_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ u64 dev_counter = atomic64_read(&trng_dev_counter);
+ u64 hwrng_counter = atomic64_read(&trng_hwrng_counter);
+#if IS_ENABLED(CONFIG_ARCH_RANDOM)
+ u64 arch_counter = atomic64_read(&s390_arch_random_counter);
+
+ return snprintf(buf, PAGE_SIZE,
+ "trng: %llu\n"
+ "hwrng: %llu\n"
+ "arch: %llu\n"
+ "total: %llu\n",
+ dev_counter, hwrng_counter, arch_counter,
+ dev_counter + hwrng_counter + arch_counter);
+#else
+ return snprintf(buf, PAGE_SIZE,
+ "trng: %llu\n"
+ "hwrng: %llu\n"
+ "total: %llu\n",
+ dev_counter, hwrng_counter,
+ dev_counter + hwrng_counter);
+#endif
+}
+static DEVICE_ATTR(byte_counter, 0444, trng_counter_show, NULL);
+
+static struct attribute *trng_dev_attrs[] = {
+ &dev_attr_byte_counter.attr,
+ NULL
+};
+
+static const struct attribute_group trng_dev_attr_group = {
+ .attrs = trng_dev_attrs
+};
+
+static const struct attribute_group *trng_dev_attr_groups[] = {
+ &trng_dev_attr_group,
+ NULL
+};
+
+static const struct file_operations trng_fops = {
+ .owner = THIS_MODULE,
+ .open = &trng_open,
+ .release = NULL,
+ .read = &trng_read,
+ .llseek = noop_llseek,
+};
+
+static struct miscdevice trng_dev = {
+ .name = "trng",
+ .minor = MISC_DYNAMIC_MINOR,
+ .mode = 0444,
+ .fops = &trng_fops,
+ .groups = trng_dev_attr_groups,
+};
+
+
+/* hwrng_register */
+
+static inline void _trng_hwrng_read(u8 *buf, size_t len)
+{
+ cpacf_trng(NULL, 0, buf, len);
+ atomic64_add(len, &trng_hwrng_counter);
+}
+
+static int trng_hwrng_data_read(struct hwrng *rng, u32 *data)
+{
+ size_t len = sizeof(*data);
+
+ _trng_hwrng_read((u8 *) data, len);
+
+ DEBUG_DBG("trng_hwrng_data_read()=%zu\n", len);
+
+ return len;
+}
+
+static int trng_hwrng_read(struct hwrng *rng, void *data, size_t max, bool wait)
+{
+ size_t len = max <= PAGE_SIZE ? max : PAGE_SIZE;
+
+ _trng_hwrng_read((u8 *) data, len);
+
+ DEBUG_DBG("trng_hwrng_read()=%zu\n", len);
+
+ return len;
+}
+
+/*
+ * hwrng register struct
+ * The trng is suppost to have 100% entropy, and thus
+ * we register with a very high quality value.
+ */
+static struct hwrng trng_hwrng_dev = {
+ .name = "s390-trng",
+ .data_read = trng_hwrng_data_read,
+ .read = trng_hwrng_read,
+ .quality = 999,
+};
+
+
+/* init and exit */
+
+static void __init trng_debug_init(void)
+{
+ debug_info = debug_register("trng", 1, 1, 4 * sizeof(long));
+ debug_register_view(debug_info, &debug_sprintf_view);
+ debug_set_level(debug_info, 3);
+}
+
+static void trng_debug_exit(void)
+{
+ debug_unregister(debug_info);
+}
+
+static int __init trng_init(void)
+{
+ int ret;
+
+ trng_debug_init();
+
+ /* check if subfunction CPACF_PRNO_TRNG is available */
+ if (!cpacf_query_func(CPACF_PRNO, CPACF_PRNO_TRNG)) {
+ DEBUG_INFO("trng_init CPACF_PRNO_TRNG not available\n");
+ ret = -ENODEV;
+ goto out_dbg;
+ }
+
+ ret = misc_register(&trng_dev);
+ if (ret) {
+ DEBUG_WARN("trng_init misc_register() failed rc=%d\n", ret);
+ goto out_dbg;
+ }
+
+ ret = hwrng_register(&trng_hwrng_dev);
+ if (ret) {
+ DEBUG_WARN("trng_init hwrng_register() failed rc=%d\n", ret);
+ goto out_misc;
+ }
+
+ DEBUG_DBG("trng_init successful\n");
+
+ return 0;
+
+out_misc:
+ misc_deregister(&trng_dev);
+out_dbg:
+ trng_debug_exit();
+ return ret;
+}
+
+static void __exit trng_exit(void)
+{
+ hwrng_unregister(&trng_hwrng_dev);
+ misc_deregister(&trng_dev);
+ trng_debug_exit();
+}
+
+module_cpu_feature_match(MSA, trng_init);
+module_exit(trng_exit);
diff --git a/drivers/char/hw_random/timeriomem-rng.c b/drivers/char/hw_random/timeriomem-rng.c
index cf37db2..a0faa5f 100644
--- a/drivers/char/hw_random/timeriomem-rng.c
+++ b/drivers/char/hw_random/timeriomem-rng.c
@@ -20,84 +20,100 @@
* TODO: add support for reading sizes other than 32bits and masking
*/
-#include <linux/module.h>
-#include <linux/kernel.h>
-#include <linux/platform_device.h>
-#include <linux/of.h>
+#include <linux/completion.h>
+#include <linux/delay.h>
+#include <linux/hrtimer.h>
#include <linux/hw_random.h>
#include <linux/io.h>
+#include <linux/ktime.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
#include <linux/slab.h>
+#include <linux/time.h>
#include <linux/timeriomem-rng.h>
-#include <linux/jiffies.h>
-#include <linux/sched.h>
-#include <linux/timer.h>
-#include <linux/completion.h>
-struct timeriomem_rng_private_data {
+struct timeriomem_rng_private {
void __iomem *io_base;
- unsigned int expires;
- unsigned int period;
+ ktime_t period;
unsigned int present:1;
- struct timer_list timer;
+ struct hrtimer timer;
struct completion completion;
- struct hwrng timeriomem_rng_ops;
+ struct hwrng rng_ops;
};
-#define to_rng_priv(rng) \
- ((struct timeriomem_rng_private_data *)rng->priv)
-
-/*
- * have data return 1, however return 0 if we have nothing
- */
-static int timeriomem_rng_data_present(struct hwrng *rng, int wait)
+static int timeriomem_rng_read(struct hwrng *hwrng, void *data,
+ size_t max, bool wait)
{
- struct timeriomem_rng_private_data *priv = to_rng_priv(rng);
+ struct timeriomem_rng_private *priv =
+ container_of(hwrng, struct timeriomem_rng_private, rng_ops);
+ int retval = 0;
+ int period_us = ktime_to_us(priv->period);
- if (!wait || priv->present)
- return priv->present;
+ /*
+ * The RNG provides 32-bits per read. Ensure there is enough space for
+ * at minimum one read.
+ */
+ if (max < sizeof(u32))
+ return 0;
+
+ /*
+ * There may not have been enough time for new data to be generated
+ * since the last request. If the caller doesn't want to wait, let them
+ * bail out. Otherwise, wait for the completion. If the new data has
+ * already been generated, the completion should already be available.
+ */
+ if (!wait && !priv->present)
+ return 0;
wait_for_completion(&priv->completion);
- return 1;
-}
+ do {
+ /*
+ * After the first read, all additional reads will need to wait
+ * for the RNG to generate new data. Since the period can have
+ * a wide range of values (1us to 1s have been observed), allow
+ * for 1% tolerance in the sleep time rather than a fixed value.
+ */
+ if (retval > 0)
+ usleep_range(period_us,
+ period_us + min(1, period_us / 100));
-static int timeriomem_rng_data_read(struct hwrng *rng, u32 *data)
-{
- struct timeriomem_rng_private_data *priv = to_rng_priv(rng);
- unsigned long cur;
- s32 delay;
+ *(u32 *)data = readl(priv->io_base);
+ retval += sizeof(u32);
+ data += sizeof(u32);
+ max -= sizeof(u32);
+ } while (wait && max > sizeof(u32));
- *data = readl(priv->io_base);
-
- cur = jiffies;
-
- delay = cur - priv->expires;
- delay = priv->period - (delay % priv->period);
-
- priv->expires = cur + delay;
+ /*
+ * Block any new callers until the RNG has had time to generate new
+ * data.
+ */
priv->present = 0;
-
reinit_completion(&priv->completion);
- mod_timer(&priv->timer, priv->expires);
+ hrtimer_forward_now(&priv->timer, priv->period);
+ hrtimer_restart(&priv->timer);
- return 4;
+ return retval;
}
-static void timeriomem_rng_trigger(unsigned long data)
+static enum hrtimer_restart timeriomem_rng_trigger(struct hrtimer *timer)
{
- struct timeriomem_rng_private_data *priv
- = (struct timeriomem_rng_private_data *)data;
+ struct timeriomem_rng_private *priv
+ = container_of(timer, struct timeriomem_rng_private, timer);
priv->present = 1;
complete(&priv->completion);
+
+ return HRTIMER_NORESTART;
}
static int timeriomem_rng_probe(struct platform_device *pdev)
{
struct timeriomem_rng_data *pdata = pdev->dev.platform_data;
- struct timeriomem_rng_private_data *priv;
+ struct timeriomem_rng_private *priv;
struct resource *res;
int err = 0;
int period;
@@ -119,7 +135,7 @@
/* Allocate memory for the device structure (and zero it) */
priv = devm_kzalloc(&pdev->dev,
- sizeof(struct timeriomem_rng_private_data), GFP_KERNEL);
+ sizeof(struct timeriomem_rng_private), GFP_KERNEL);
if (!priv)
return -ENOMEM;
@@ -139,54 +155,41 @@
period = pdata->period;
}
- priv->period = usecs_to_jiffies(period);
- if (priv->period < 1) {
- dev_err(&pdev->dev, "period is less than one jiffy\n");
- return -EINVAL;
- }
-
- priv->expires = jiffies;
- priv->present = 1;
-
+ priv->period = ns_to_ktime(period * NSEC_PER_USEC);
init_completion(&priv->completion);
- complete(&priv->completion);
+ hrtimer_init(&priv->timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
+ priv->timer.function = timeriomem_rng_trigger;
- setup_timer(&priv->timer, timeriomem_rng_trigger, (unsigned long)priv);
-
- priv->timeriomem_rng_ops.name = dev_name(&pdev->dev);
- priv->timeriomem_rng_ops.data_present = timeriomem_rng_data_present;
- priv->timeriomem_rng_ops.data_read = timeriomem_rng_data_read;
- priv->timeriomem_rng_ops.priv = (unsigned long)priv;
+ priv->rng_ops.name = dev_name(&pdev->dev);
+ priv->rng_ops.read = timeriomem_rng_read;
priv->io_base = devm_ioremap_resource(&pdev->dev, res);
if (IS_ERR(priv->io_base)) {
- err = PTR_ERR(priv->io_base);
- goto out_timer;
+ return PTR_ERR(priv->io_base);
}
- err = hwrng_register(&priv->timeriomem_rng_ops);
+ /* Assume random data is already available. */
+ priv->present = 1;
+ complete(&priv->completion);
+
+ err = hwrng_register(&priv->rng_ops);
if (err) {
dev_err(&pdev->dev, "problem registering\n");
- goto out_timer;
+ return err;
}
dev_info(&pdev->dev, "32bits from 0x%p @ %dus\n",
priv->io_base, period);
return 0;
-
-out_timer:
- del_timer_sync(&priv->timer);
- return err;
}
static int timeriomem_rng_remove(struct platform_device *pdev)
{
- struct timeriomem_rng_private_data *priv = platform_get_drvdata(pdev);
+ struct timeriomem_rng_private *priv = platform_get_drvdata(pdev);
- hwrng_unregister(&priv->timeriomem_rng_ops);
-
- del_timer_sync(&priv->timer);
+ hwrng_unregister(&priv->rng_ops);
+ hrtimer_cancel(&priv->timer);
return 0;
}
diff --git a/drivers/char/ipmi/bt-bmc.c b/drivers/char/ipmi/bt-bmc.c
index d6f5d9e..70d434b 100644
--- a/drivers/char/ipmi/bt-bmc.c
+++ b/drivers/char/ipmi/bt-bmc.c
@@ -523,6 +523,7 @@
static const struct of_device_id bt_bmc_match[] = {
{ .compatible = "aspeed,ast2400-ibt-bmc" },
+ { .compatible = "aspeed,ast2500-ibt-bmc" },
{ },
};
diff --git a/drivers/char/ipmi/ipmi_si_intf.c b/drivers/char/ipmi/ipmi_si_intf.c
index 2a7c425..b2b618f 100644
--- a/drivers/char/ipmi/ipmi_si_intf.c
+++ b/drivers/char/ipmi/ipmi_si_intf.c
@@ -1954,7 +1954,9 @@
kfree(info);
goto out;
}
+ mutex_lock(&smi_infos_lock);
rv = try_smi_init(info);
+ mutex_unlock(&smi_infos_lock);
if (rv) {
cleanup_one_si(info);
goto out;
@@ -2042,8 +2044,10 @@
info->slave_addr = slave_addrs[i];
if (!add_smi(info)) {
+ mutex_lock(&smi_infos_lock);
if (try_smi_init(info))
cleanup_one_si(info);
+ mutex_unlock(&smi_infos_lock);
ret = 0;
} else {
kfree(info);
@@ -3492,6 +3496,11 @@
return rv;
}
+/*
+ * Try to start up an interface. Must be called with smi_infos_lock
+ * held, primarily to keep smi_num consistent, we only one to do these
+ * one at a time.
+ */
static int try_smi_init(struct smi_info *new_smi)
{
int rv = 0;
@@ -3524,9 +3533,12 @@
goto out_err;
}
+ new_smi->intf_num = smi_num;
+
/* Do this early so it's available for logs. */
if (!new_smi->dev) {
- init_name = kasprintf(GFP_KERNEL, "ipmi_si.%d", 0);
+ init_name = kasprintf(GFP_KERNEL, "ipmi_si.%d",
+ new_smi->intf_num);
/*
* If we don't already have a device from something
@@ -3593,8 +3605,6 @@
new_smi->interrupt_disabled = true;
atomic_set(&new_smi->need_watch, 0);
- new_smi->intf_num = smi_num;
- smi_num++;
rv = try_enable_event_buffer(new_smi);
if (rv == 0)
@@ -3661,6 +3671,9 @@
goto out_err_stop_timer;
}
+ /* Don't increment till we know we have succeeded. */
+ smi_num++;
+
dev_info(new_smi->dev, "IPMI %s interface initialized\n",
si_to_str[new_smi->si_type]);
diff --git a/drivers/char/ipmi/ipmi_ssif.c b/drivers/char/ipmi/ipmi_ssif.c
index cca6e5b..0b22a9b 100644
--- a/drivers/char/ipmi/ipmi_ssif.c
+++ b/drivers/char/ipmi/ipmi_ssif.c
@@ -891,6 +891,7 @@
* for details on the intricacies of this.
*/
int left;
+ unsigned char *data_to_send;
ssif_inc_stat(ssif_info, sent_messages_parts);
@@ -899,6 +900,7 @@
left = 32;
/* Length byte. */
ssif_info->multi_data[ssif_info->multi_pos] = left;
+ data_to_send = ssif_info->multi_data + ssif_info->multi_pos;
ssif_info->multi_pos += left;
if (left < 32)
/*
@@ -912,7 +914,7 @@
rv = ssif_i2c_send(ssif_info, msg_written_handler,
I2C_SMBUS_WRITE,
SSIF_IPMI_MULTI_PART_REQUEST_MIDDLE,
- ssif_info->multi_data + ssif_info->multi_pos,
+ data_to_send,
I2C_SMBUS_BLOCK_DATA);
if (rv < 0) {
/* request failed, just return the error. */
@@ -1642,9 +1644,8 @@
spin_lock_init(&ssif_info->lock);
ssif_info->ssif_state = SSIF_NORMAL;
- init_timer(&ssif_info->retry_timer);
- ssif_info->retry_timer.data = (unsigned long) ssif_info;
- ssif_info->retry_timer.function = retry_timeout;
+ setup_timer(&ssif_info->retry_timer, retry_timeout,
+ (unsigned long)ssif_info);
for (i = 0; i < SSIF_NUM_STATS; i++)
atomic_set(&ssif_info->stats[i], 0);
diff --git a/drivers/char/ipmi/ipmi_watchdog.c b/drivers/char/ipmi/ipmi_watchdog.c
index 5ca24d9..d165af8 100644
--- a/drivers/char/ipmi/ipmi_watchdog.c
+++ b/drivers/char/ipmi/ipmi_watchdog.c
@@ -516,7 +516,7 @@
msg.cmd = IPMI_WDOG_RESET_TIMER;
msg.data = NULL;
msg.data_len = 0;
- atomic_add(2, &panic_done_count);
+ atomic_add(1, &panic_done_count);
rv = ipmi_request_supply_msgs(watchdog_user,
(struct ipmi_addr *) &addr,
0,
@@ -526,7 +526,7 @@
&panic_halt_heartbeat_recv_msg,
1);
if (rv)
- atomic_sub(2, &panic_done_count);
+ atomic_sub(1, &panic_done_count);
}
static struct ipmi_smi_msg panic_halt_smi_msg = {
@@ -550,12 +550,12 @@
/* Wait for the messages to be free. */
while (atomic_read(&panic_done_count) != 0)
ipmi_poll_interface(watchdog_user);
- atomic_add(2, &panic_done_count);
+ atomic_add(1, &panic_done_count);
rv = i_ipmi_set_timeout(&panic_halt_smi_msg,
&panic_halt_recv_msg,
&send_heartbeat_now);
if (rv) {
- atomic_sub(2, &panic_done_count);
+ atomic_sub(1, &panic_done_count);
printk(KERN_WARNING PFX
"Unable to extend the watchdog timeout.");
} else {
diff --git a/drivers/char/mmtimer.c b/drivers/char/mmtimer.c
index b708c85..0e7fcb0 100644
--- a/drivers/char/mmtimer.c
+++ b/drivers/char/mmtimer.c
@@ -478,18 +478,18 @@
static struct timespec sgi_clock_offset;
static int sgi_clock_period;
-static int sgi_clock_get(clockid_t clockid, struct timespec *tp)
+static int sgi_clock_get(clockid_t clockid, struct timespec64 *tp)
{
u64 nsec;
nsec = rtc_time() * sgi_clock_period
+ sgi_clock_offset.tv_nsec;
- *tp = ns_to_timespec(nsec);
+ *tp = ns_to_timespec64(nsec);
tp->tv_sec += sgi_clock_offset.tv_sec;
return 0;
};
-static int sgi_clock_set(const clockid_t clockid, const struct timespec *tp)
+static int sgi_clock_set(const clockid_t clockid, const struct timespec64 *tp)
{
u64 nsec;
@@ -657,7 +657,7 @@
}
/* Assumption: it_lock is already held with irq's disabled */
-static void sgi_timer_get(struct k_itimer *timr, struct itimerspec *cur_setting)
+static void sgi_timer_get(struct k_itimer *timr, struct itimerspec64 *cur_setting)
{
if (timr->it.mmtimer.clock == TIMER_OFF) {
@@ -668,14 +668,14 @@
return;
}
- cur_setting->it_interval = ns_to_timespec(timr->it.mmtimer.incr * sgi_clock_period);
- cur_setting->it_value = ns_to_timespec((timr->it.mmtimer.expires - rtc_time()) * sgi_clock_period);
+ cur_setting->it_interval = ns_to_timespec64(timr->it.mmtimer.incr * sgi_clock_period);
+ cur_setting->it_value = ns_to_timespec64((timr->it.mmtimer.expires - rtc_time()) * sgi_clock_period);
}
static int sgi_timer_set(struct k_itimer *timr, int flags,
- struct itimerspec * new_setting,
- struct itimerspec * old_setting)
+ struct itimerspec64 *new_setting,
+ struct itimerspec64 *old_setting)
{
unsigned long when, period, irqflags;
int err = 0;
@@ -687,8 +687,8 @@
sgi_timer_get(timr, old_setting);
sgi_timer_del(timr);
- when = timespec_to_ns(&new_setting->it_value);
- period = timespec_to_ns(&new_setting->it_interval);
+ when = timespec64_to_ns(&new_setting->it_value);
+ period = timespec64_to_ns(&new_setting->it_interval);
if (when == 0)
/* Clear timer */
@@ -699,11 +699,11 @@
return -ENOMEM;
if (flags & TIMER_ABSTIME) {
- struct timespec n;
+ struct timespec64 n;
unsigned long now;
- getnstimeofday(&n);
- now = timespec_to_ns(&n);
+ getnstimeofday64(&n);
+ now = timespec64_to_ns(&n);
if (when > now)
when -= now;
else
@@ -765,7 +765,7 @@
return err;
}
-static int sgi_clock_getres(const clockid_t which_clock, struct timespec *tp)
+static int sgi_clock_getres(const clockid_t which_clock, struct timespec64 *tp)
{
tp->tv_sec = 0;
tp->tv_nsec = sgi_clock_period;
diff --git a/drivers/clk/meson/gxbb.h b/drivers/clk/meson/gxbb.h
index 8ee2022..cbd62e4 100644
--- a/drivers/clk/meson/gxbb.h
+++ b/drivers/clk/meson/gxbb.h
@@ -193,7 +193,7 @@
/* CLKID_I2C */
/* #define CLKID_SAR_ADC */
#define CLKID_SMART_CARD 24
-#define CLKID_RNG0 25
+/* CLKID_RNG0 */
#define CLKID_UART0 26
#define CLKID_SDHC 27
#define CLKID_STREAM 28
diff --git a/drivers/clk/mvebu/ap806-system-controller.c b/drivers/clk/mvebu/ap806-system-controller.c
index f177021..8155bac 100644
--- a/drivers/clk/mvebu/ap806-system-controller.c
+++ b/drivers/clk/mvebu/ap806-system-controller.c
@@ -23,7 +23,7 @@
#define AP806_SAR_REG 0x400
#define AP806_SAR_CLKFREQ_MODE_MASK 0x1f
-#define AP806_CLK_NUM 4
+#define AP806_CLK_NUM 5
static struct clk *ap806_clks[AP806_CLK_NUM];
@@ -135,6 +135,23 @@
goto fail3;
}
+ /* eMMC Clock is fixed clock divided by 3 */
+ if (of_property_read_string_index(np, "clock-output-names",
+ 4, &name)) {
+ ap806_clk_data.clk_num--;
+ dev_warn(&pdev->dev,
+ "eMMC clock missing: update the device tree!\n");
+ } else {
+ ap806_clks[4] = clk_register_fixed_factor(NULL, name,
+ fixedclk_name,
+ 0, 1, 3);
+ if (IS_ERR(ap806_clks[4])) {
+ ret = PTR_ERR(ap806_clks[4]);
+ goto fail4;
+ }
+ }
+
+ of_clk_add_provider(np, of_clk_src_onecell_get, &ap806_clk_data);
ret = of_clk_add_provider(np, of_clk_src_onecell_get, &ap806_clk_data);
if (ret)
goto fail_clk_add;
@@ -142,6 +159,8 @@
return 0;
fail_clk_add:
+ clk_unregister_fixed_factor(ap806_clks[4]);
+fail4:
clk_unregister_fixed_factor(ap806_clks[3]);
fail3:
clk_unregister_fixed_rate(ap806_clks[2]);
diff --git a/drivers/clk/sunxi-ng/Kconfig b/drivers/clk/sunxi-ng/Kconfig
index 1c23573..a077ab6 100644
--- a/drivers/clk/sunxi-ng/Kconfig
+++ b/drivers/clk/sunxi-ng/Kconfig
@@ -16,7 +16,7 @@
bool
config SUNXI_CCU_GATE
- bool
+ def_bool y
config SUNXI_CCU_MUX
bool
diff --git a/drivers/clocksource/Kconfig b/drivers/clocksource/Kconfig
index 3356ab8..545d541 100644
--- a/drivers/clocksource/Kconfig
+++ b/drivers/clocksource/Kconfig
@@ -67,20 +67,22 @@
select DW_APB_TIMER
select CLKSRC_OF
-config GEMINI_TIMER
- bool "Cortina Gemini timer driver" if COMPILE_TEST
+config FTTMR010_TIMER
+ bool "Faraday Technology timer driver" if COMPILE_TEST
depends on GENERIC_CLOCKEVENTS
depends on HAS_IOMEM
select CLKSRC_MMIO
select CLKSRC_OF
select MFD_SYSCON
help
- Enables support for the Gemini timer
+ Enables support for the Faraday Technology timer block
+ FTTMR010.
config ROCKCHIP_TIMER
bool "Rockchip timer driver" if COMPILE_TEST
depends on ARM || ARM64
select CLKSRC_OF
+ select CLKSRC_MMIO
help
Enables the support for the rockchip timer driver.
@@ -366,6 +368,17 @@
161010101. The workaround will be active if the hisilicon,erratum-161010101
property is found in the timer node.
+config ARM64_ERRATUM_858921
+ bool "Workaround for Cortex-A73 erratum 858921"
+ default y
+ select ARM_ARCH_TIMER_OOL_WORKAROUND
+ depends on ARM_ARCH_TIMER && ARM64
+ help
+ This option enables a workaround applicable to Cortex-A73
+ (all versions), whose counter may return incorrect values.
+ The workaround will be dynamically enabled when an affected
+ core is detected.
+
config ARM_GLOBAL_TIMER
bool "Support for the ARM global timer" if COMPILE_TEST
select CLKSRC_OF if OF
diff --git a/drivers/clocksource/Makefile b/drivers/clocksource/Makefile
index d227d13..2b5b56a 100644
--- a/drivers/clocksource/Makefile
+++ b/drivers/clocksource/Makefile
@@ -17,7 +17,7 @@
obj-$(CONFIG_DIGICOLOR_TIMER) += timer-digicolor.o
obj-$(CONFIG_DW_APB_TIMER) += dw_apb_timer.o
obj-$(CONFIG_DW_APB_TIMER_OF) += dw_apb_timer_of.o
-obj-$(CONFIG_GEMINI_TIMER) += timer-gemini.o
+obj-$(CONFIG_FTTMR010_TIMER) += timer-fttmr010.o
obj-$(CONFIG_ROCKCHIP_TIMER) += rockchip_timer.o
obj-$(CONFIG_CLKSRC_NOMADIK_MTU) += nomadik-mtu.o
obj-$(CONFIG_CLKSRC_DBX500_PRCMU) += clksrc-dbx500-prcmu.o
diff --git a/drivers/clocksource/arc_timer.c b/drivers/clocksource/arc_timer.c
index 7517f95..2164973 100644
--- a/drivers/clocksource/arc_timer.c
+++ b/drivers/clocksource/arc_timer.c
@@ -37,7 +37,7 @@
clk = of_clk_get(node, 0);
if (IS_ERR(clk)) {
- pr_err("timer missing clk");
+ pr_err("timer missing clk\n");
return PTR_ERR(clk);
}
@@ -89,7 +89,7 @@
READ_BCR(ARC_REG_MCIP_BCR, mp);
if (!mp.gfrc) {
- pr_warn("Global-64-bit-Ctr clocksource not detected");
+ pr_warn("Global-64-bit-Ctr clocksource not detected\n");
return -ENXIO;
}
@@ -140,13 +140,13 @@
READ_BCR(ARC_REG_TIMERS_BCR, timer);
if (!timer.rtc) {
- pr_warn("Local-64-bit-Ctr clocksource not detected");
+ pr_warn("Local-64-bit-Ctr clocksource not detected\n");
return -ENXIO;
}
/* Local to CPU hence not usable in SMP */
if (IS_ENABLED(CONFIG_SMP)) {
- pr_warn("Local-64-bit-Ctr not usable in SMP");
+ pr_warn("Local-64-bit-Ctr not usable in SMP\n");
return -EINVAL;
}
@@ -290,13 +290,13 @@
arc_timer_irq = irq_of_parse_and_map(node, 0);
if (arc_timer_irq <= 0) {
- pr_err("clockevent: missing irq");
+ pr_err("clockevent: missing irq\n");
return -EINVAL;
}
ret = arc_get_timer_clk(node);
if (ret) {
- pr_err("clockevent: missing clk");
+ pr_err("clockevent: missing clk\n");
return ret;
}
@@ -313,7 +313,7 @@
arc_timer_starting_cpu,
arc_timer_dying_cpu);
if (ret) {
- pr_err("Failed to setup hotplug state");
+ pr_err("Failed to setup hotplug state\n");
return ret;
}
return 0;
diff --git a/drivers/clocksource/arm_arch_timer.c b/drivers/clocksource/arm_arch_timer.c
index 7a8a411..a1fb918 100644
--- a/drivers/clocksource/arm_arch_timer.c
+++ b/drivers/clocksource/arm_arch_timer.c
@@ -33,6 +33,9 @@
#include <clocksource/arm_arch_timer.h>
+#undef pr_fmt
+#define pr_fmt(fmt) "arch_timer: " fmt
+
#define CNTTIDR 0x08
#define CNTTIDR_VIRT(n) (BIT(1) << ((n) * 4))
@@ -52,8 +55,6 @@
#define CNTV_TVAL 0x38
#define CNTV_CTL 0x3c
-#define ARCH_CP15_TIMER BIT(0)
-#define ARCH_MEM_TIMER BIT(1)
static unsigned arch_timers_present __initdata;
static void __iomem *arch_counter_base;
@@ -66,23 +67,15 @@
#define to_arch_timer(e) container_of(e, struct arch_timer, evt)
static u32 arch_timer_rate;
-
-enum ppi_nr {
- PHYS_SECURE_PPI,
- PHYS_NONSECURE_PPI,
- VIRT_PPI,
- HYP_PPI,
- MAX_TIMER_PPI
-};
-
-static int arch_timer_ppi[MAX_TIMER_PPI];
+static int arch_timer_ppi[ARCH_TIMER_MAX_TIMER_PPI];
static struct clock_event_device __percpu *arch_timer_evt;
-static enum ppi_nr arch_timer_uses_ppi = VIRT_PPI;
+static enum arch_timer_ppi_nr arch_timer_uses_ppi = ARCH_TIMER_VIRT_PPI;
static bool arch_timer_c3stop;
static bool arch_timer_mem_use_virtual;
static bool arch_counter_suspend_stop;
+static bool vdso_default = true;
static bool evtstrm_enable = IS_ENABLED(CONFIG_ARM_ARCH_TIMER_EVTSTREAM);
@@ -96,6 +89,105 @@
* Architected system timer support.
*/
+static __always_inline
+void arch_timer_reg_write(int access, enum arch_timer_reg reg, u32 val,
+ struct clock_event_device *clk)
+{
+ if (access == ARCH_TIMER_MEM_PHYS_ACCESS) {
+ struct arch_timer *timer = to_arch_timer(clk);
+ switch (reg) {
+ case ARCH_TIMER_REG_CTRL:
+ writel_relaxed(val, timer->base + CNTP_CTL);
+ break;
+ case ARCH_TIMER_REG_TVAL:
+ writel_relaxed(val, timer->base + CNTP_TVAL);
+ break;
+ }
+ } else if (access == ARCH_TIMER_MEM_VIRT_ACCESS) {
+ struct arch_timer *timer = to_arch_timer(clk);
+ switch (reg) {
+ case ARCH_TIMER_REG_CTRL:
+ writel_relaxed(val, timer->base + CNTV_CTL);
+ break;
+ case ARCH_TIMER_REG_TVAL:
+ writel_relaxed(val, timer->base + CNTV_TVAL);
+ break;
+ }
+ } else {
+ arch_timer_reg_write_cp15(access, reg, val);
+ }
+}
+
+static __always_inline
+u32 arch_timer_reg_read(int access, enum arch_timer_reg reg,
+ struct clock_event_device *clk)
+{
+ u32 val;
+
+ if (access == ARCH_TIMER_MEM_PHYS_ACCESS) {
+ struct arch_timer *timer = to_arch_timer(clk);
+ switch (reg) {
+ case ARCH_TIMER_REG_CTRL:
+ val = readl_relaxed(timer->base + CNTP_CTL);
+ break;
+ case ARCH_TIMER_REG_TVAL:
+ val = readl_relaxed(timer->base + CNTP_TVAL);
+ break;
+ }
+ } else if (access == ARCH_TIMER_MEM_VIRT_ACCESS) {
+ struct arch_timer *timer = to_arch_timer(clk);
+ switch (reg) {
+ case ARCH_TIMER_REG_CTRL:
+ val = readl_relaxed(timer->base + CNTV_CTL);
+ break;
+ case ARCH_TIMER_REG_TVAL:
+ val = readl_relaxed(timer->base + CNTV_TVAL);
+ break;
+ }
+ } else {
+ val = arch_timer_reg_read_cp15(access, reg);
+ }
+
+ return val;
+}
+
+/*
+ * Default to cp15 based access because arm64 uses this function for
+ * sched_clock() before DT is probed and the cp15 method is guaranteed
+ * to exist on arm64. arm doesn't use this before DT is probed so even
+ * if we don't have the cp15 accessors we won't have a problem.
+ */
+u64 (*arch_timer_read_counter)(void) = arch_counter_get_cntvct;
+
+static u64 arch_counter_read(struct clocksource *cs)
+{
+ return arch_timer_read_counter();
+}
+
+static u64 arch_counter_read_cc(const struct cyclecounter *cc)
+{
+ return arch_timer_read_counter();
+}
+
+static struct clocksource clocksource_counter = {
+ .name = "arch_sys_counter",
+ .rating = 400,
+ .read = arch_counter_read,
+ .mask = CLOCKSOURCE_MASK(56),
+ .flags = CLOCK_SOURCE_IS_CONTINUOUS,
+};
+
+static struct cyclecounter cyclecounter __ro_after_init = {
+ .read = arch_counter_read_cc,
+ .mask = CLOCKSOURCE_MASK(56),
+};
+
+struct ate_acpi_oem_info {
+ char oem_id[ACPI_OEM_ID_SIZE + 1];
+ char oem_table_id[ACPI_OEM_TABLE_ID_SIZE + 1];
+ u32 oem_revision;
+};
+
#ifdef CONFIG_FSL_ERRATUM_A008585
/*
* The number of retries is an arbitrary value well beyond the highest number
@@ -170,97 +262,289 @@
{
return __hisi_161010101_read_reg(cntvct_el0);
}
+
+static struct ate_acpi_oem_info hisi_161010101_oem_info[] = {
+ /*
+ * Note that trailing spaces are required to properly match
+ * the OEM table information.
+ */
+ {
+ .oem_id = "HISI ",
+ .oem_table_id = "HIP05 ",
+ .oem_revision = 0,
+ },
+ {
+ .oem_id = "HISI ",
+ .oem_table_id = "HIP06 ",
+ .oem_revision = 0,
+ },
+ {
+ .oem_id = "HISI ",
+ .oem_table_id = "HIP07 ",
+ .oem_revision = 0,
+ },
+ { /* Sentinel indicating the end of the OEM array */ },
+};
+#endif
+
+#ifdef CONFIG_ARM64_ERRATUM_858921
+static u64 notrace arm64_858921_read_cntvct_el0(void)
+{
+ u64 old, new;
+
+ old = read_sysreg(cntvct_el0);
+ new = read_sysreg(cntvct_el0);
+ return (((old ^ new) >> 32) & 1) ? old : new;
+}
#endif
#ifdef CONFIG_ARM_ARCH_TIMER_OOL_WORKAROUND
-const struct arch_timer_erratum_workaround *timer_unstable_counter_workaround = NULL;
+DEFINE_PER_CPU(const struct arch_timer_erratum_workaround *,
+ timer_unstable_counter_workaround);
EXPORT_SYMBOL_GPL(timer_unstable_counter_workaround);
DEFINE_STATIC_KEY_FALSE(arch_timer_read_ool_enabled);
EXPORT_SYMBOL_GPL(arch_timer_read_ool_enabled);
+static void erratum_set_next_event_tval_generic(const int access, unsigned long evt,
+ struct clock_event_device *clk)
+{
+ unsigned long ctrl;
+ u64 cval = evt + arch_counter_get_cntvct();
+
+ ctrl = arch_timer_reg_read(access, ARCH_TIMER_REG_CTRL, clk);
+ ctrl |= ARCH_TIMER_CTRL_ENABLE;
+ ctrl &= ~ARCH_TIMER_CTRL_IT_MASK;
+
+ if (access == ARCH_TIMER_PHYS_ACCESS)
+ write_sysreg(cval, cntp_cval_el0);
+ else
+ write_sysreg(cval, cntv_cval_el0);
+
+ arch_timer_reg_write(access, ARCH_TIMER_REG_CTRL, ctrl, clk);
+}
+
+static __maybe_unused int erratum_set_next_event_tval_virt(unsigned long evt,
+ struct clock_event_device *clk)
+{
+ erratum_set_next_event_tval_generic(ARCH_TIMER_VIRT_ACCESS, evt, clk);
+ return 0;
+}
+
+static __maybe_unused int erratum_set_next_event_tval_phys(unsigned long evt,
+ struct clock_event_device *clk)
+{
+ erratum_set_next_event_tval_generic(ARCH_TIMER_PHYS_ACCESS, evt, clk);
+ return 0;
+}
+
static const struct arch_timer_erratum_workaround ool_workarounds[] = {
#ifdef CONFIG_FSL_ERRATUM_A008585
{
+ .match_type = ate_match_dt,
.id = "fsl,erratum-a008585",
+ .desc = "Freescale erratum a005858",
.read_cntp_tval_el0 = fsl_a008585_read_cntp_tval_el0,
.read_cntv_tval_el0 = fsl_a008585_read_cntv_tval_el0,
.read_cntvct_el0 = fsl_a008585_read_cntvct_el0,
+ .set_next_event_phys = erratum_set_next_event_tval_phys,
+ .set_next_event_virt = erratum_set_next_event_tval_virt,
},
#endif
#ifdef CONFIG_HISILICON_ERRATUM_161010101
{
+ .match_type = ate_match_dt,
.id = "hisilicon,erratum-161010101",
+ .desc = "HiSilicon erratum 161010101",
.read_cntp_tval_el0 = hisi_161010101_read_cntp_tval_el0,
.read_cntv_tval_el0 = hisi_161010101_read_cntv_tval_el0,
.read_cntvct_el0 = hisi_161010101_read_cntvct_el0,
+ .set_next_event_phys = erratum_set_next_event_tval_phys,
+ .set_next_event_virt = erratum_set_next_event_tval_virt,
+ },
+ {
+ .match_type = ate_match_acpi_oem_info,
+ .id = hisi_161010101_oem_info,
+ .desc = "HiSilicon erratum 161010101",
+ .read_cntp_tval_el0 = hisi_161010101_read_cntp_tval_el0,
+ .read_cntv_tval_el0 = hisi_161010101_read_cntv_tval_el0,
+ .read_cntvct_el0 = hisi_161010101_read_cntvct_el0,
+ .set_next_event_phys = erratum_set_next_event_tval_phys,
+ .set_next_event_virt = erratum_set_next_event_tval_virt,
+ },
+#endif
+#ifdef CONFIG_ARM64_ERRATUM_858921
+ {
+ .match_type = ate_match_local_cap_id,
+ .id = (void *)ARM64_WORKAROUND_858921,
+ .desc = "ARM erratum 858921",
+ .read_cntvct_el0 = arm64_858921_read_cntvct_el0,
},
#endif
};
+
+typedef bool (*ate_match_fn_t)(const struct arch_timer_erratum_workaround *,
+ const void *);
+
+static
+bool arch_timer_check_dt_erratum(const struct arch_timer_erratum_workaround *wa,
+ const void *arg)
+{
+ const struct device_node *np = arg;
+
+ return of_property_read_bool(np, wa->id);
+}
+
+static
+bool arch_timer_check_local_cap_erratum(const struct arch_timer_erratum_workaround *wa,
+ const void *arg)
+{
+ return this_cpu_has_cap((uintptr_t)wa->id);
+}
+
+
+static
+bool arch_timer_check_acpi_oem_erratum(const struct arch_timer_erratum_workaround *wa,
+ const void *arg)
+{
+ static const struct ate_acpi_oem_info empty_oem_info = {};
+ const struct ate_acpi_oem_info *info = wa->id;
+ const struct acpi_table_header *table = arg;
+
+ /* Iterate over the ACPI OEM info array, looking for a match */
+ while (memcmp(info, &empty_oem_info, sizeof(*info))) {
+ if (!memcmp(info->oem_id, table->oem_id, ACPI_OEM_ID_SIZE) &&
+ !memcmp(info->oem_table_id, table->oem_table_id, ACPI_OEM_TABLE_ID_SIZE) &&
+ info->oem_revision == table->oem_revision)
+ return true;
+
+ info++;
+ }
+
+ return false;
+}
+
+static const struct arch_timer_erratum_workaround *
+arch_timer_iterate_errata(enum arch_timer_erratum_match_type type,
+ ate_match_fn_t match_fn,
+ void *arg)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(ool_workarounds); i++) {
+ if (ool_workarounds[i].match_type != type)
+ continue;
+
+ if (match_fn(&ool_workarounds[i], arg))
+ return &ool_workarounds[i];
+ }
+
+ return NULL;
+}
+
+static
+void arch_timer_enable_workaround(const struct arch_timer_erratum_workaround *wa,
+ bool local)
+{
+ int i;
+
+ if (local) {
+ __this_cpu_write(timer_unstable_counter_workaround, wa);
+ } else {
+ for_each_possible_cpu(i)
+ per_cpu(timer_unstable_counter_workaround, i) = wa;
+ }
+
+ static_branch_enable(&arch_timer_read_ool_enabled);
+
+ /*
+ * Don't use the vdso fastpath if errata require using the
+ * out-of-line counter accessor. We may change our mind pretty
+ * late in the game (with a per-CPU erratum, for example), so
+ * change both the default value and the vdso itself.
+ */
+ if (wa->read_cntvct_el0) {
+ clocksource_counter.archdata.vdso_direct = false;
+ vdso_default = false;
+ }
+}
+
+static void arch_timer_check_ool_workaround(enum arch_timer_erratum_match_type type,
+ void *arg)
+{
+ const struct arch_timer_erratum_workaround *wa;
+ ate_match_fn_t match_fn = NULL;
+ bool local = false;
+
+ switch (type) {
+ case ate_match_dt:
+ match_fn = arch_timer_check_dt_erratum;
+ break;
+ case ate_match_local_cap_id:
+ match_fn = arch_timer_check_local_cap_erratum;
+ local = true;
+ break;
+ case ate_match_acpi_oem_info:
+ match_fn = arch_timer_check_acpi_oem_erratum;
+ break;
+ default:
+ WARN_ON(1);
+ return;
+ }
+
+ wa = arch_timer_iterate_errata(type, match_fn, arg);
+ if (!wa)
+ return;
+
+ if (needs_unstable_timer_counter_workaround()) {
+ const struct arch_timer_erratum_workaround *__wa;
+ __wa = __this_cpu_read(timer_unstable_counter_workaround);
+ if (__wa && wa != __wa)
+ pr_warn("Can't enable workaround for %s (clashes with %s\n)",
+ wa->desc, __wa->desc);
+
+ if (__wa)
+ return;
+ }
+
+ arch_timer_enable_workaround(wa, local);
+ pr_info("Enabling %s workaround for %s\n",
+ local ? "local" : "global", wa->desc);
+}
+
+#define erratum_handler(fn, r, ...) \
+({ \
+ bool __val; \
+ if (needs_unstable_timer_counter_workaround()) { \
+ const struct arch_timer_erratum_workaround *__wa; \
+ __wa = __this_cpu_read(timer_unstable_counter_workaround); \
+ if (__wa && __wa->fn) { \
+ r = __wa->fn(__VA_ARGS__); \
+ __val = true; \
+ } else { \
+ __val = false; \
+ } \
+ } else { \
+ __val = false; \
+ } \
+ __val; \
+})
+
+static bool arch_timer_this_cpu_has_cntvct_wa(void)
+{
+ const struct arch_timer_erratum_workaround *wa;
+
+ wa = __this_cpu_read(timer_unstable_counter_workaround);
+ return wa && wa->read_cntvct_el0;
+}
+#else
+#define arch_timer_check_ool_workaround(t,a) do { } while(0)
+#define erratum_set_next_event_tval_virt(...) ({BUG(); 0;})
+#define erratum_set_next_event_tval_phys(...) ({BUG(); 0;})
+#define erratum_handler(fn, r, ...) ({false;})
+#define arch_timer_this_cpu_has_cntvct_wa() ({false;})
#endif /* CONFIG_ARM_ARCH_TIMER_OOL_WORKAROUND */
-static __always_inline
-void arch_timer_reg_write(int access, enum arch_timer_reg reg, u32 val,
- struct clock_event_device *clk)
-{
- if (access == ARCH_TIMER_MEM_PHYS_ACCESS) {
- struct arch_timer *timer = to_arch_timer(clk);
- switch (reg) {
- case ARCH_TIMER_REG_CTRL:
- writel_relaxed(val, timer->base + CNTP_CTL);
- break;
- case ARCH_TIMER_REG_TVAL:
- writel_relaxed(val, timer->base + CNTP_TVAL);
- break;
- }
- } else if (access == ARCH_TIMER_MEM_VIRT_ACCESS) {
- struct arch_timer *timer = to_arch_timer(clk);
- switch (reg) {
- case ARCH_TIMER_REG_CTRL:
- writel_relaxed(val, timer->base + CNTV_CTL);
- break;
- case ARCH_TIMER_REG_TVAL:
- writel_relaxed(val, timer->base + CNTV_TVAL);
- break;
- }
- } else {
- arch_timer_reg_write_cp15(access, reg, val);
- }
-}
-
-static __always_inline
-u32 arch_timer_reg_read(int access, enum arch_timer_reg reg,
- struct clock_event_device *clk)
-{
- u32 val;
-
- if (access == ARCH_TIMER_MEM_PHYS_ACCESS) {
- struct arch_timer *timer = to_arch_timer(clk);
- switch (reg) {
- case ARCH_TIMER_REG_CTRL:
- val = readl_relaxed(timer->base + CNTP_CTL);
- break;
- case ARCH_TIMER_REG_TVAL:
- val = readl_relaxed(timer->base + CNTP_TVAL);
- break;
- }
- } else if (access == ARCH_TIMER_MEM_VIRT_ACCESS) {
- struct arch_timer *timer = to_arch_timer(clk);
- switch (reg) {
- case ARCH_TIMER_REG_CTRL:
- val = readl_relaxed(timer->base + CNTV_CTL);
- break;
- case ARCH_TIMER_REG_TVAL:
- val = readl_relaxed(timer->base + CNTV_TVAL);
- break;
- }
- } else {
- val = arch_timer_reg_read_cp15(access, reg);
- }
-
- return val;
-}
-
static __always_inline irqreturn_t timer_handler(const int access,
struct clock_event_device *evt)
{
@@ -348,43 +632,14 @@
arch_timer_reg_write(access, ARCH_TIMER_REG_CTRL, ctrl, clk);
}
-#ifdef CONFIG_ARM_ARCH_TIMER_OOL_WORKAROUND
-static __always_inline void erratum_set_next_event_generic(const int access,
- unsigned long evt, struct clock_event_device *clk)
-{
- unsigned long ctrl;
- u64 cval = evt + arch_counter_get_cntvct();
-
- ctrl = arch_timer_reg_read(access, ARCH_TIMER_REG_CTRL, clk);
- ctrl |= ARCH_TIMER_CTRL_ENABLE;
- ctrl &= ~ARCH_TIMER_CTRL_IT_MASK;
-
- if (access == ARCH_TIMER_PHYS_ACCESS)
- write_sysreg(cval, cntp_cval_el0);
- else if (access == ARCH_TIMER_VIRT_ACCESS)
- write_sysreg(cval, cntv_cval_el0);
-
- arch_timer_reg_write(access, ARCH_TIMER_REG_CTRL, ctrl, clk);
-}
-
-static int erratum_set_next_event_virt(unsigned long evt,
- struct clock_event_device *clk)
-{
- erratum_set_next_event_generic(ARCH_TIMER_VIRT_ACCESS, evt, clk);
- return 0;
-}
-
-static int erratum_set_next_event_phys(unsigned long evt,
- struct clock_event_device *clk)
-{
- erratum_set_next_event_generic(ARCH_TIMER_PHYS_ACCESS, evt, clk);
- return 0;
-}
-#endif /* CONFIG_ARM_ARCH_TIMER_OOL_WORKAROUND */
-
static int arch_timer_set_next_event_virt(unsigned long evt,
struct clock_event_device *clk)
{
+ int ret;
+
+ if (erratum_handler(set_next_event_virt, ret, evt, clk))
+ return ret;
+
set_next_event(ARCH_TIMER_VIRT_ACCESS, evt, clk);
return 0;
}
@@ -392,6 +647,11 @@
static int arch_timer_set_next_event_phys(unsigned long evt,
struct clock_event_device *clk)
{
+ int ret;
+
+ if (erratum_handler(set_next_event_phys, ret, evt, clk))
+ return ret;
+
set_next_event(ARCH_TIMER_PHYS_ACCESS, evt, clk);
return 0;
}
@@ -410,25 +670,12 @@
return 0;
}
-static void erratum_workaround_set_sne(struct clock_event_device *clk)
-{
-#ifdef CONFIG_ARM_ARCH_TIMER_OOL_WORKAROUND
- if (!static_branch_unlikely(&arch_timer_read_ool_enabled))
- return;
-
- if (arch_timer_uses_ppi == VIRT_PPI)
- clk->set_next_event = erratum_set_next_event_virt;
- else
- clk->set_next_event = erratum_set_next_event_phys;
-#endif
-}
-
static void __arch_timer_setup(unsigned type,
struct clock_event_device *clk)
{
clk->features = CLOCK_EVT_FEAT_ONESHOT;
- if (type == ARCH_CP15_TIMER) {
+ if (type == ARCH_TIMER_TYPE_CP15) {
if (arch_timer_c3stop)
clk->features |= CLOCK_EVT_FEAT_C3STOP;
clk->name = "arch_sys_timer";
@@ -436,14 +683,14 @@
clk->cpumask = cpumask_of(smp_processor_id());
clk->irq = arch_timer_ppi[arch_timer_uses_ppi];
switch (arch_timer_uses_ppi) {
- case VIRT_PPI:
+ case ARCH_TIMER_VIRT_PPI:
clk->set_state_shutdown = arch_timer_shutdown_virt;
clk->set_state_oneshot_stopped = arch_timer_shutdown_virt;
clk->set_next_event = arch_timer_set_next_event_virt;
break;
- case PHYS_SECURE_PPI:
- case PHYS_NONSECURE_PPI:
- case HYP_PPI:
+ case ARCH_TIMER_PHYS_SECURE_PPI:
+ case ARCH_TIMER_PHYS_NONSECURE_PPI:
+ case ARCH_TIMER_HYP_PPI:
clk->set_state_shutdown = arch_timer_shutdown_phys;
clk->set_state_oneshot_stopped = arch_timer_shutdown_phys;
clk->set_next_event = arch_timer_set_next_event_phys;
@@ -452,7 +699,7 @@
BUG();
}
- erratum_workaround_set_sne(clk);
+ arch_timer_check_ool_workaround(ate_match_local_cap_id, NULL);
} else {
clk->features |= CLOCK_EVT_FEAT_DYNIRQ;
clk->name = "arch_mem_timer";
@@ -508,23 +755,31 @@
{
u32 cntkctl = arch_timer_get_cntkctl();
- /* Disable user access to the timers and the physical counter */
+ /* Disable user access to the timers and both counters */
/* Also disable virtual event stream */
cntkctl &= ~(ARCH_TIMER_USR_PT_ACCESS_EN
| ARCH_TIMER_USR_VT_ACCESS_EN
+ | ARCH_TIMER_USR_VCT_ACCESS_EN
| ARCH_TIMER_VIRT_EVT_EN
| ARCH_TIMER_USR_PCT_ACCESS_EN);
- /* Enable user access to the virtual counter */
- cntkctl |= ARCH_TIMER_USR_VCT_ACCESS_EN;
+ /*
+ * Enable user access to the virtual counter if it doesn't
+ * need to be workaround. The vdso may have been already
+ * disabled though.
+ */
+ if (arch_timer_this_cpu_has_cntvct_wa())
+ pr_info("CPU%d: Trapping CNTVCT access\n", smp_processor_id());
+ else
+ cntkctl |= ARCH_TIMER_USR_VCT_ACCESS_EN;
arch_timer_set_cntkctl(cntkctl);
}
static bool arch_timer_has_nonsecure_ppi(void)
{
- return (arch_timer_uses_ppi == PHYS_SECURE_PPI &&
- arch_timer_ppi[PHYS_NONSECURE_PPI]);
+ return (arch_timer_uses_ppi == ARCH_TIMER_PHYS_SECURE_PPI &&
+ arch_timer_ppi[ARCH_TIMER_PHYS_NONSECURE_PPI]);
}
static u32 check_ppi_trigger(int irq)
@@ -545,14 +800,15 @@
struct clock_event_device *clk = this_cpu_ptr(arch_timer_evt);
u32 flags;
- __arch_timer_setup(ARCH_CP15_TIMER, clk);
+ __arch_timer_setup(ARCH_TIMER_TYPE_CP15, clk);
flags = check_ppi_trigger(arch_timer_ppi[arch_timer_uses_ppi]);
enable_percpu_irq(arch_timer_ppi[arch_timer_uses_ppi], flags);
if (arch_timer_has_nonsecure_ppi()) {
- flags = check_ppi_trigger(arch_timer_ppi[PHYS_NONSECURE_PPI]);
- enable_percpu_irq(arch_timer_ppi[PHYS_NONSECURE_PPI], flags);
+ flags = check_ppi_trigger(arch_timer_ppi[ARCH_TIMER_PHYS_NONSECURE_PPI]);
+ enable_percpu_irq(arch_timer_ppi[ARCH_TIMER_PHYS_NONSECURE_PPI],
+ flags);
}
arch_counter_set_user_access();
@@ -562,43 +818,39 @@
return 0;
}
-static void
-arch_timer_detect_rate(void __iomem *cntbase, struct device_node *np)
+/*
+ * For historical reasons, when probing with DT we use whichever (non-zero)
+ * rate was probed first, and don't verify that others match. If the first node
+ * probed has a clock-frequency property, this overrides the HW register.
+ */
+static void arch_timer_of_configure_rate(u32 rate, struct device_node *np)
{
/* Who has more than one independent system counter? */
if (arch_timer_rate)
return;
- /*
- * Try to determine the frequency from the device tree or CNTFRQ,
- * if ACPI is enabled, get the frequency from CNTFRQ ONLY.
- */
- if (!acpi_disabled ||
- of_property_read_u32(np, "clock-frequency", &arch_timer_rate)) {
- if (cntbase)
- arch_timer_rate = readl_relaxed(cntbase + CNTFRQ);
- else
- arch_timer_rate = arch_timer_get_cntfrq();
- }
+ if (of_property_read_u32(np, "clock-frequency", &arch_timer_rate))
+ arch_timer_rate = rate;
/* Check the timer frequency. */
if (arch_timer_rate == 0)
- pr_warn("Architected timer frequency not available\n");
+ pr_warn("frequency not available\n");
}
static void arch_timer_banner(unsigned type)
{
- pr_info("Architected %s%s%s timer(s) running at %lu.%02luMHz (%s%s%s).\n",
- type & ARCH_CP15_TIMER ? "cp15" : "",
- type == (ARCH_CP15_TIMER | ARCH_MEM_TIMER) ? " and " : "",
- type & ARCH_MEM_TIMER ? "mmio" : "",
- (unsigned long)arch_timer_rate / 1000000,
- (unsigned long)(arch_timer_rate / 10000) % 100,
- type & ARCH_CP15_TIMER ?
- (arch_timer_uses_ppi == VIRT_PPI) ? "virt" : "phys" :
+ pr_info("%s%s%s timer(s) running at %lu.%02luMHz (%s%s%s).\n",
+ type & ARCH_TIMER_TYPE_CP15 ? "cp15" : "",
+ type == (ARCH_TIMER_TYPE_CP15 | ARCH_TIMER_TYPE_MEM) ?
+ " and " : "",
+ type & ARCH_TIMER_TYPE_MEM ? "mmio" : "",
+ (unsigned long)arch_timer_rate / 1000000,
+ (unsigned long)(arch_timer_rate / 10000) % 100,
+ type & ARCH_TIMER_TYPE_CP15 ?
+ (arch_timer_uses_ppi == ARCH_TIMER_VIRT_PPI) ? "virt" : "phys" :
"",
- type == (ARCH_CP15_TIMER | ARCH_MEM_TIMER) ? "/" : "",
- type & ARCH_MEM_TIMER ?
+ type == (ARCH_TIMER_TYPE_CP15 | ARCH_TIMER_TYPE_MEM) ? "/" : "",
+ type & ARCH_TIMER_TYPE_MEM ?
arch_timer_mem_use_virtual ? "virt" : "phys" :
"");
}
@@ -621,37 +873,6 @@
return ((u64) vct_hi << 32) | vct_lo;
}
-/*
- * Default to cp15 based access because arm64 uses this function for
- * sched_clock() before DT is probed and the cp15 method is guaranteed
- * to exist on arm64. arm doesn't use this before DT is probed so even
- * if we don't have the cp15 accessors we won't have a problem.
- */
-u64 (*arch_timer_read_counter)(void) = arch_counter_get_cntvct;
-
-static u64 arch_counter_read(struct clocksource *cs)
-{
- return arch_timer_read_counter();
-}
-
-static u64 arch_counter_read_cc(const struct cyclecounter *cc)
-{
- return arch_timer_read_counter();
-}
-
-static struct clocksource clocksource_counter = {
- .name = "arch_sys_counter",
- .rating = 400,
- .read = arch_counter_read,
- .mask = CLOCKSOURCE_MASK(56),
- .flags = CLOCK_SOURCE_IS_CONTINUOUS,
-};
-
-static struct cyclecounter cyclecounter __ro_after_init = {
- .read = arch_counter_read_cc,
- .mask = CLOCKSOURCE_MASK(56),
-};
-
static struct arch_timer_kvm_info arch_timer_kvm_info;
struct arch_timer_kvm_info *arch_timer_get_kvm_info(void)
@@ -664,22 +885,14 @@
u64 start_count;
/* Register the CP15 based counter if we have one */
- if (type & ARCH_CP15_TIMER) {
- if (IS_ENABLED(CONFIG_ARM64) || arch_timer_uses_ppi == VIRT_PPI)
+ if (type & ARCH_TIMER_TYPE_CP15) {
+ if (IS_ENABLED(CONFIG_ARM64) ||
+ arch_timer_uses_ppi == ARCH_TIMER_VIRT_PPI)
arch_timer_read_counter = arch_counter_get_cntvct;
else
arch_timer_read_counter = arch_counter_get_cntpct;
- clocksource_counter.archdata.vdso_direct = true;
-
-#ifdef CONFIG_ARM_ARCH_TIMER_OOL_WORKAROUND
- /*
- * Don't use the vdso fastpath if errata require using
- * the out-of-line counter accessor.
- */
- if (static_branch_unlikely(&arch_timer_read_ool_enabled))
- clocksource_counter.archdata.vdso_direct = false;
-#endif
+ clocksource_counter.archdata.vdso_direct = vdso_default;
} else {
arch_timer_read_counter = arch_counter_get_cntvct_mem;
}
@@ -699,12 +912,11 @@
static void arch_timer_stop(struct clock_event_device *clk)
{
- pr_debug("arch_timer_teardown disable IRQ%d cpu #%d\n",
- clk->irq, smp_processor_id());
+ pr_debug("disable IRQ%d cpu #%d\n", clk->irq, smp_processor_id());
disable_percpu_irq(arch_timer_ppi[arch_timer_uses_ppi]);
if (arch_timer_has_nonsecure_ppi())
- disable_percpu_irq(arch_timer_ppi[PHYS_NONSECURE_PPI]);
+ disable_percpu_irq(arch_timer_ppi[ARCH_TIMER_PHYS_NONSECURE_PPI]);
clk->set_state_shutdown(clk);
}
@@ -718,14 +930,14 @@
}
#ifdef CONFIG_CPU_PM
-static unsigned int saved_cntkctl;
+static DEFINE_PER_CPU(unsigned long, saved_cntkctl);
static int arch_timer_cpu_pm_notify(struct notifier_block *self,
unsigned long action, void *hcpu)
{
if (action == CPU_PM_ENTER)
- saved_cntkctl = arch_timer_get_cntkctl();
+ __this_cpu_write(saved_cntkctl, arch_timer_get_cntkctl());
else if (action == CPU_PM_ENTER_FAILED || action == CPU_PM_EXIT)
- arch_timer_set_cntkctl(saved_cntkctl);
+ arch_timer_set_cntkctl(__this_cpu_read(saved_cntkctl));
return NOTIFY_OK;
}
@@ -767,24 +979,24 @@
ppi = arch_timer_ppi[arch_timer_uses_ppi];
switch (arch_timer_uses_ppi) {
- case VIRT_PPI:
+ case ARCH_TIMER_VIRT_PPI:
err = request_percpu_irq(ppi, arch_timer_handler_virt,
"arch_timer", arch_timer_evt);
break;
- case PHYS_SECURE_PPI:
- case PHYS_NONSECURE_PPI:
+ case ARCH_TIMER_PHYS_SECURE_PPI:
+ case ARCH_TIMER_PHYS_NONSECURE_PPI:
err = request_percpu_irq(ppi, arch_timer_handler_phys,
"arch_timer", arch_timer_evt);
- if (!err && arch_timer_ppi[PHYS_NONSECURE_PPI]) {
- ppi = arch_timer_ppi[PHYS_NONSECURE_PPI];
+ if (!err && arch_timer_has_nonsecure_ppi()) {
+ ppi = arch_timer_ppi[ARCH_TIMER_PHYS_NONSECURE_PPI];
err = request_percpu_irq(ppi, arch_timer_handler_phys,
"arch_timer", arch_timer_evt);
if (err)
- free_percpu_irq(arch_timer_ppi[PHYS_SECURE_PPI],
+ free_percpu_irq(arch_timer_ppi[ARCH_TIMER_PHYS_SECURE_PPI],
arch_timer_evt);
}
break;
- case HYP_PPI:
+ case ARCH_TIMER_HYP_PPI:
err = request_percpu_irq(ppi, arch_timer_handler_phys,
"arch_timer", arch_timer_evt);
break;
@@ -793,8 +1005,7 @@
}
if (err) {
- pr_err("arch_timer: can't register interrupt %d (%d)\n",
- ppi, err);
+ pr_err("can't register interrupt %d (%d)\n", ppi, err);
goto out_free;
}
@@ -817,7 +1028,7 @@
out_unreg_notify:
free_percpu_irq(arch_timer_ppi[arch_timer_uses_ppi], arch_timer_evt);
if (arch_timer_has_nonsecure_ppi())
- free_percpu_irq(arch_timer_ppi[PHYS_NONSECURE_PPI],
+ free_percpu_irq(arch_timer_ppi[ARCH_TIMER_PHYS_NONSECURE_PPI],
arch_timer_evt);
out_free:
@@ -838,7 +1049,7 @@
t->base = base;
t->evt.irq = irq;
- __arch_timer_setup(ARCH_MEM_TIMER, &t->evt);
+ __arch_timer_setup(ARCH_TIMER_TYPE_MEM, &t->evt);
if (arch_timer_mem_use_virtual)
func = arch_timer_handler_virt_mem;
@@ -847,7 +1058,7 @@
ret = request_irq(irq, func, IRQF_TIMER, "arch_mem_timer", &t->evt);
if (ret) {
- pr_err("arch_timer: Failed to request mem timer irq\n");
+ pr_err("Failed to request mem timer irq\n");
kfree(t);
}
@@ -865,15 +1076,28 @@
{},
};
-static bool __init
-arch_timer_needs_probing(int type, const struct of_device_id *matches)
+static bool __init arch_timer_needs_of_probing(void)
{
struct device_node *dn;
bool needs_probing = false;
+ unsigned int mask = ARCH_TIMER_TYPE_CP15 | ARCH_TIMER_TYPE_MEM;
- dn = of_find_matching_node(NULL, matches);
- if (dn && of_device_is_available(dn) && !(arch_timers_present & type))
+ /* We have two timers, and both device-tree nodes are probed. */
+ if ((arch_timers_present & mask) == mask)
+ return false;
+
+ /*
+ * Only one type of timer is probed,
+ * check if we have another type of timer node in device-tree.
+ */
+ if (arch_timers_present & ARCH_TIMER_TYPE_CP15)
+ dn = of_find_matching_node(NULL, arch_timer_mem_of_match);
+ else
+ dn = of_find_matching_node(NULL, arch_timer_of_match);
+
+ if (dn && of_device_is_available(dn))
needs_probing = true;
+
of_node_put(dn);
return needs_probing;
@@ -881,96 +1105,66 @@
static int __init arch_timer_common_init(void)
{
- unsigned mask = ARCH_CP15_TIMER | ARCH_MEM_TIMER;
-
- /* Wait until both nodes are probed if we have two timers */
- if ((arch_timers_present & mask) != mask) {
- if (arch_timer_needs_probing(ARCH_MEM_TIMER, arch_timer_mem_of_match))
- return 0;
- if (arch_timer_needs_probing(ARCH_CP15_TIMER, arch_timer_of_match))
- return 0;
- }
-
arch_timer_banner(arch_timers_present);
arch_counter_register(arch_timers_present);
return arch_timer_arch_init();
}
-static int __init arch_timer_init(void)
+/**
+ * arch_timer_select_ppi() - Select suitable PPI for the current system.
+ *
+ * If HYP mode is available, we know that the physical timer
+ * has been configured to be accessible from PL1. Use it, so
+ * that a guest can use the virtual timer instead.
+ *
+ * On ARMv8.1 with VH extensions, the kernel runs in HYP. VHE
+ * accesses to CNTP_*_EL1 registers are silently redirected to
+ * their CNTHP_*_EL2 counterparts, and use a different PPI
+ * number.
+ *
+ * If no interrupt provided for virtual timer, we'll have to
+ * stick to the physical timer. It'd better be accessible...
+ * For arm64 we never use the secure interrupt.
+ *
+ * Return: a suitable PPI type for the current system.
+ */
+static enum arch_timer_ppi_nr __init arch_timer_select_ppi(void)
{
- int ret;
- /*
- * If HYP mode is available, we know that the physical timer
- * has been configured to be accessible from PL1. Use it, so
- * that a guest can use the virtual timer instead.
- *
- * If no interrupt provided for virtual timer, we'll have to
- * stick to the physical timer. It'd better be accessible...
- *
- * On ARMv8.1 with VH extensions, the kernel runs in HYP. VHE
- * accesses to CNTP_*_EL1 registers are silently redirected to
- * their CNTHP_*_EL2 counterparts, and use a different PPI
- * number.
- */
- if (is_hyp_mode_available() || !arch_timer_ppi[VIRT_PPI]) {
- bool has_ppi;
+ if (is_kernel_in_hyp_mode())
+ return ARCH_TIMER_HYP_PPI;
- if (is_kernel_in_hyp_mode()) {
- arch_timer_uses_ppi = HYP_PPI;
- has_ppi = !!arch_timer_ppi[HYP_PPI];
- } else {
- arch_timer_uses_ppi = PHYS_SECURE_PPI;
- has_ppi = (!!arch_timer_ppi[PHYS_SECURE_PPI] ||
- !!arch_timer_ppi[PHYS_NONSECURE_PPI]);
- }
+ if (!is_hyp_mode_available() && arch_timer_ppi[ARCH_TIMER_VIRT_PPI])
+ return ARCH_TIMER_VIRT_PPI;
- if (!has_ppi) {
- pr_warn("arch_timer: No interrupt available, giving up\n");
- return -EINVAL;
- }
- }
+ if (IS_ENABLED(CONFIG_ARM64))
+ return ARCH_TIMER_PHYS_NONSECURE_PPI;
- ret = arch_timer_register();
- if (ret)
- return ret;
-
- ret = arch_timer_common_init();
- if (ret)
- return ret;
-
- arch_timer_kvm_info.virtual_irq = arch_timer_ppi[VIRT_PPI];
-
- return 0;
+ return ARCH_TIMER_PHYS_SECURE_PPI;
}
static int __init arch_timer_of_init(struct device_node *np)
{
- int i;
+ int i, ret;
+ u32 rate;
- if (arch_timers_present & ARCH_CP15_TIMER) {
- pr_warn("arch_timer: multiple nodes in dt, skipping\n");
+ if (arch_timers_present & ARCH_TIMER_TYPE_CP15) {
+ pr_warn("multiple nodes in dt, skipping\n");
return 0;
}
- arch_timers_present |= ARCH_CP15_TIMER;
- for (i = PHYS_SECURE_PPI; i < MAX_TIMER_PPI; i++)
+ arch_timers_present |= ARCH_TIMER_TYPE_CP15;
+ for (i = ARCH_TIMER_PHYS_SECURE_PPI; i < ARCH_TIMER_MAX_TIMER_PPI; i++)
arch_timer_ppi[i] = irq_of_parse_and_map(np, i);
- arch_timer_detect_rate(NULL, np);
+ arch_timer_kvm_info.virtual_irq = arch_timer_ppi[ARCH_TIMER_VIRT_PPI];
+
+ rate = arch_timer_get_cntfrq();
+ arch_timer_of_configure_rate(rate, np);
arch_timer_c3stop = !of_property_read_bool(np, "always-on");
-#ifdef CONFIG_ARM_ARCH_TIMER_OOL_WORKAROUND
- for (i = 0; i < ARRAY_SIZE(ool_workarounds); i++) {
- if (of_property_read_bool(np, ool_workarounds[i].id)) {
- timer_unstable_counter_workaround = &ool_workarounds[i];
- static_branch_enable(&arch_timer_read_ool_enabled);
- pr_info("arch_timer: Enabling workaround for %s\n",
- timer_unstable_counter_workaround->id);
- break;
- }
- }
-#endif
+ /* Check for globally applicable workarounds */
+ arch_timer_check_ool_workaround(ate_match_dt, np);
/*
* If we cannot rely on firmware initializing the timer registers then
@@ -978,29 +1172,63 @@
*/
if (IS_ENABLED(CONFIG_ARM) &&
of_property_read_bool(np, "arm,cpu-registers-not-fw-configured"))
- arch_timer_uses_ppi = PHYS_SECURE_PPI;
+ arch_timer_uses_ppi = ARCH_TIMER_PHYS_SECURE_PPI;
+ else
+ arch_timer_uses_ppi = arch_timer_select_ppi();
+
+ if (!arch_timer_ppi[arch_timer_uses_ppi]) {
+ pr_err("No interrupt available, giving up\n");
+ return -EINVAL;
+ }
/* On some systems, the counter stops ticking when in suspend. */
arch_counter_suspend_stop = of_property_read_bool(np,
"arm,no-tick-in-suspend");
- return arch_timer_init();
+ ret = arch_timer_register();
+ if (ret)
+ return ret;
+
+ if (arch_timer_needs_of_probing())
+ return 0;
+
+ return arch_timer_common_init();
}
CLOCKSOURCE_OF_DECLARE(armv7_arch_timer, "arm,armv7-timer", arch_timer_of_init);
CLOCKSOURCE_OF_DECLARE(armv8_arch_timer, "arm,armv8-timer", arch_timer_of_init);
-static int __init arch_timer_mem_init(struct device_node *np)
+static u32 __init
+arch_timer_mem_frame_get_cntfrq(struct arch_timer_mem_frame *frame)
{
- struct device_node *frame, *best_frame = NULL;
- void __iomem *cntctlbase, *base;
- unsigned int irq, ret = -EINVAL;
- u32 cnttidr;
+ void __iomem *base;
+ u32 rate;
- arch_timers_present |= ARCH_MEM_TIMER;
- cntctlbase = of_iomap(np, 0);
+ base = ioremap(frame->cntbase, frame->size);
+ if (!base) {
+ pr_err("Unable to map frame @ %pa\n", &frame->cntbase);
+ return 0;
+ }
+
+ rate = readl_relaxed(frame + CNTFRQ);
+
+ iounmap(frame);
+
+ return rate;
+}
+
+static struct arch_timer_mem_frame * __init
+arch_timer_mem_find_best_frame(struct arch_timer_mem *timer_mem)
+{
+ struct arch_timer_mem_frame *frame, *best_frame = NULL;
+ void __iomem *cntctlbase;
+ u32 cnttidr;
+ int i;
+
+ cntctlbase = ioremap(timer_mem->cntctlbase, timer_mem->size);
if (!cntctlbase) {
- pr_err("arch_timer: Can't find CNTCTLBase\n");
- return -ENXIO;
+ pr_err("Can't map CNTCTLBase @ %pa\n",
+ &timer_mem->cntctlbase);
+ return NULL;
}
cnttidr = readl_relaxed(cntctlbase + CNTTIDR);
@@ -1009,25 +1237,20 @@
* Try to find a virtual capable frame. Otherwise fall back to a
* physical capable frame.
*/
- for_each_available_child_of_node(np, frame) {
- int n;
- u32 cntacr;
+ for (i = 0; i < ARCH_TIMER_MEM_MAX_FRAMES; i++) {
+ u32 cntacr = CNTACR_RFRQ | CNTACR_RWPT | CNTACR_RPCT |
+ CNTACR_RWVT | CNTACR_RVOFF | CNTACR_RVCT;
- if (of_property_read_u32(frame, "frame-number", &n)) {
- pr_err("arch_timer: Missing frame-number\n");
- of_node_put(frame);
- goto out;
- }
+ frame = &timer_mem->frame[i];
+ if (!frame->valid)
+ continue;
/* Try enabling everything, and see what sticks */
- cntacr = CNTACR_RFRQ | CNTACR_RWPT | CNTACR_RPCT |
- CNTACR_RWVT | CNTACR_RVOFF | CNTACR_RVCT;
- writel_relaxed(cntacr, cntctlbase + CNTACR(n));
- cntacr = readl_relaxed(cntctlbase + CNTACR(n));
+ writel_relaxed(cntacr, cntctlbase + CNTACR(i));
+ cntacr = readl_relaxed(cntctlbase + CNTACR(i));
- if ((cnttidr & CNTTIDR_VIRT(n)) &&
+ if ((cnttidr & CNTTIDR_VIRT(i)) &&
!(~cntacr & (CNTACR_RWVT | CNTACR_RVCT))) {
- of_node_put(best_frame);
best_frame = frame;
arch_timer_mem_use_virtual = true;
break;
@@ -1036,99 +1259,262 @@
if (~cntacr & (CNTACR_RWPT | CNTACR_RPCT))
continue;
- of_node_put(best_frame);
- best_frame = of_node_get(frame);
+ best_frame = frame;
}
- ret= -ENXIO;
- base = arch_counter_base = of_io_request_and_map(best_frame, 0,
- "arch_mem_timer");
- if (IS_ERR(base)) {
- pr_err("arch_timer: Can't map frame's registers\n");
- goto out;
- }
+ iounmap(cntctlbase);
+
+ if (!best_frame)
+ pr_err("Unable to find a suitable frame in timer @ %pa\n",
+ &timer_mem->cntctlbase);
+
+ return frame;
+}
+
+static int __init
+arch_timer_mem_frame_register(struct arch_timer_mem_frame *frame)
+{
+ void __iomem *base;
+ int ret, irq = 0;
if (arch_timer_mem_use_virtual)
- irq = irq_of_parse_and_map(best_frame, 1);
+ irq = frame->virt_irq;
else
- irq = irq_of_parse_and_map(best_frame, 0);
+ irq = frame->phys_irq;
- ret = -EINVAL;
if (!irq) {
- pr_err("arch_timer: Frame missing %s irq",
+ pr_err("Frame missing %s irq.\n",
arch_timer_mem_use_virtual ? "virt" : "phys");
- goto out;
- }
-
- arch_timer_detect_rate(base, np);
- ret = arch_timer_mem_register(base, irq);
- if (ret)
- goto out;
-
- return arch_timer_common_init();
-out:
- iounmap(cntctlbase);
- of_node_put(best_frame);
- return ret;
-}
-CLOCKSOURCE_OF_DECLARE(armv7_arch_timer_mem, "arm,armv7-timer-mem",
- arch_timer_mem_init);
-
-#ifdef CONFIG_ACPI
-static int __init map_generic_timer_interrupt(u32 interrupt, u32 flags)
-{
- int trigger, polarity;
-
- if (!interrupt)
- return 0;
-
- trigger = (flags & ACPI_GTDT_INTERRUPT_MODE) ? ACPI_EDGE_SENSITIVE
- : ACPI_LEVEL_SENSITIVE;
-
- polarity = (flags & ACPI_GTDT_INTERRUPT_POLARITY) ? ACPI_ACTIVE_LOW
- : ACPI_ACTIVE_HIGH;
-
- return acpi_register_gsi(NULL, interrupt, trigger, polarity);
-}
-
-/* Initialize per-processor generic timer */
-static int __init arch_timer_acpi_init(struct acpi_table_header *table)
-{
- struct acpi_table_gtdt *gtdt;
-
- if (arch_timers_present & ARCH_CP15_TIMER) {
- pr_warn("arch_timer: already initialized, skipping\n");
return -EINVAL;
}
- gtdt = container_of(table, struct acpi_table_gtdt, header);
+ if (!request_mem_region(frame->cntbase, frame->size,
+ "arch_mem_timer"))
+ return -EBUSY;
- arch_timers_present |= ARCH_CP15_TIMER;
+ base = ioremap(frame->cntbase, frame->size);
+ if (!base) {
+ pr_err("Can't map frame's registers\n");
+ return -ENXIO;
+ }
- arch_timer_ppi[PHYS_SECURE_PPI] =
- map_generic_timer_interrupt(gtdt->secure_el1_interrupt,
- gtdt->secure_el1_flags);
+ ret = arch_timer_mem_register(base, irq);
+ if (ret) {
+ iounmap(base);
+ return ret;
+ }
- arch_timer_ppi[PHYS_NONSECURE_PPI] =
- map_generic_timer_interrupt(gtdt->non_secure_el1_interrupt,
- gtdt->non_secure_el1_flags);
+ arch_counter_base = base;
+ arch_timers_present |= ARCH_TIMER_TYPE_MEM;
- arch_timer_ppi[VIRT_PPI] =
- map_generic_timer_interrupt(gtdt->virtual_timer_interrupt,
- gtdt->virtual_timer_flags);
+ return 0;
+}
- arch_timer_ppi[HYP_PPI] =
- map_generic_timer_interrupt(gtdt->non_secure_el2_interrupt,
- gtdt->non_secure_el2_flags);
+static int __init arch_timer_mem_of_init(struct device_node *np)
+{
+ struct arch_timer_mem *timer_mem;
+ struct arch_timer_mem_frame *frame;
+ struct device_node *frame_node;
+ struct resource res;
+ int ret = -EINVAL;
+ u32 rate;
- /* Get the frequency from CNTFRQ */
- arch_timer_detect_rate(NULL, NULL);
+ timer_mem = kzalloc(sizeof(*timer_mem), GFP_KERNEL);
+ if (!timer_mem)
+ return -ENOMEM;
+
+ if (of_address_to_resource(np, 0, &res))
+ goto out;
+ timer_mem->cntctlbase = res.start;
+ timer_mem->size = resource_size(&res);
+
+ for_each_available_child_of_node(np, frame_node) {
+ u32 n;
+ struct arch_timer_mem_frame *frame;
+
+ if (of_property_read_u32(frame_node, "frame-number", &n)) {
+ pr_err(FW_BUG "Missing frame-number.\n");
+ of_node_put(frame_node);
+ goto out;
+ }
+ if (n >= ARCH_TIMER_MEM_MAX_FRAMES) {
+ pr_err(FW_BUG "Wrong frame-number, only 0-%u are permitted.\n",
+ ARCH_TIMER_MEM_MAX_FRAMES - 1);
+ of_node_put(frame_node);
+ goto out;
+ }
+ frame = &timer_mem->frame[n];
+
+ if (frame->valid) {
+ pr_err(FW_BUG "Duplicated frame-number.\n");
+ of_node_put(frame_node);
+ goto out;
+ }
+
+ if (of_address_to_resource(frame_node, 0, &res)) {
+ of_node_put(frame_node);
+ goto out;
+ }
+ frame->cntbase = res.start;
+ frame->size = resource_size(&res);
+
+ frame->virt_irq = irq_of_parse_and_map(frame_node,
+ ARCH_TIMER_VIRT_SPI);
+ frame->phys_irq = irq_of_parse_and_map(frame_node,
+ ARCH_TIMER_PHYS_SPI);
+
+ frame->valid = true;
+ }
+
+ frame = arch_timer_mem_find_best_frame(timer_mem);
+ if (!frame) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ rate = arch_timer_mem_frame_get_cntfrq(frame);
+ arch_timer_of_configure_rate(rate, np);
+
+ ret = arch_timer_mem_frame_register(frame);
+ if (!ret && !arch_timer_needs_of_probing())
+ ret = arch_timer_common_init();
+out:
+ kfree(timer_mem);
+ return ret;
+}
+CLOCKSOURCE_OF_DECLARE(armv7_arch_timer_mem, "arm,armv7-timer-mem",
+ arch_timer_mem_of_init);
+
+#ifdef CONFIG_ACPI_GTDT
+static int __init
+arch_timer_mem_verify_cntfrq(struct arch_timer_mem *timer_mem)
+{
+ struct arch_timer_mem_frame *frame;
+ u32 rate;
+ int i;
+
+ for (i = 0; i < ARCH_TIMER_MEM_MAX_FRAMES; i++) {
+ frame = &timer_mem->frame[i];
+
+ if (!frame->valid)
+ continue;
+
+ rate = arch_timer_mem_frame_get_cntfrq(frame);
+ if (rate == arch_timer_rate)
+ continue;
+
+ pr_err(FW_BUG "CNTFRQ mismatch: frame @ %pa: (0x%08lx), CPU: (0x%08lx)\n",
+ &frame->cntbase,
+ (unsigned long)rate, (unsigned long)arch_timer_rate);
+
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int __init arch_timer_mem_acpi_init(int platform_timer_count)
+{
+ struct arch_timer_mem *timers, *timer;
+ struct arch_timer_mem_frame *frame;
+ int timer_count, i, ret = 0;
+
+ timers = kcalloc(platform_timer_count, sizeof(*timers),
+ GFP_KERNEL);
+ if (!timers)
+ return -ENOMEM;
+
+ ret = acpi_arch_timer_mem_init(timers, &timer_count);
+ if (ret || !timer_count)
+ goto out;
+
+ for (i = 0; i < timer_count; i++) {
+ ret = arch_timer_mem_verify_cntfrq(&timers[i]);
+ if (ret) {
+ pr_err("Disabling MMIO timers due to CNTFRQ mismatch\n");
+ goto out;
+ }
+ }
+
+ /*
+ * While unlikely, it's theoretically possible that none of the frames
+ * in a timer expose the combination of feature we want.
+ */
+ for (i = i; i < timer_count; i++) {
+ timer = &timers[i];
+
+ frame = arch_timer_mem_find_best_frame(timer);
+ if (frame)
+ break;
+ }
+
+ if (frame)
+ ret = arch_timer_mem_frame_register(frame);
+out:
+ kfree(timers);
+ return ret;
+}
+
+/* Initialize per-processor generic timer and memory-mapped timer(if present) */
+static int __init arch_timer_acpi_init(struct acpi_table_header *table)
+{
+ int ret, platform_timer_count;
+
+ if (arch_timers_present & ARCH_TIMER_TYPE_CP15) {
+ pr_warn("already initialized, skipping\n");
+ return -EINVAL;
+ }
+
+ arch_timers_present |= ARCH_TIMER_TYPE_CP15;
+
+ ret = acpi_gtdt_init(table, &platform_timer_count);
+ if (ret) {
+ pr_err("Failed to init GTDT table.\n");
+ return ret;
+ }
+
+ arch_timer_ppi[ARCH_TIMER_PHYS_NONSECURE_PPI] =
+ acpi_gtdt_map_ppi(ARCH_TIMER_PHYS_NONSECURE_PPI);
+
+ arch_timer_ppi[ARCH_TIMER_VIRT_PPI] =
+ acpi_gtdt_map_ppi(ARCH_TIMER_VIRT_PPI);
+
+ arch_timer_ppi[ARCH_TIMER_HYP_PPI] =
+ acpi_gtdt_map_ppi(ARCH_TIMER_HYP_PPI);
+
+ arch_timer_kvm_info.virtual_irq = arch_timer_ppi[ARCH_TIMER_VIRT_PPI];
+
+ /*
+ * When probing via ACPI, we have no mechanism to override the sysreg
+ * CNTFRQ value. This *must* be correct.
+ */
+ arch_timer_rate = arch_timer_get_cntfrq();
+ if (!arch_timer_rate) {
+ pr_err(FW_BUG "frequency not available.\n");
+ return -EINVAL;
+ }
+
+ arch_timer_uses_ppi = arch_timer_select_ppi();
+ if (!arch_timer_ppi[arch_timer_uses_ppi]) {
+ pr_err("No interrupt available, giving up\n");
+ return -EINVAL;
+ }
/* Always-on capability */
- arch_timer_c3stop = !(gtdt->non_secure_el1_flags & ACPI_GTDT_ALWAYS_ON);
+ arch_timer_c3stop = acpi_gtdt_c3stop(arch_timer_uses_ppi);
- arch_timer_init();
- return 0;
+ /* Check for globally applicable workarounds */
+ arch_timer_check_ool_workaround(ate_match_acpi_oem_info, table);
+
+ ret = arch_timer_register();
+ if (ret)
+ return ret;
+
+ if (platform_timer_count &&
+ arch_timer_mem_acpi_init(platform_timer_count))
+ pr_err("Failed to initialize memory-mapped timer.\n");
+
+ return arch_timer_common_init();
}
CLOCKSOURCE_ACPI_DECLARE(arch_timer, ACPI_SIG_GTDT, arch_timer_acpi_init);
#endif
diff --git a/drivers/clocksource/asm9260_timer.c b/drivers/clocksource/asm9260_timer.c
index 1ba871b..c678083 100644
--- a/drivers/clocksource/asm9260_timer.c
+++ b/drivers/clocksource/asm9260_timer.c
@@ -193,7 +193,7 @@
priv.base = of_io_request_and_map(np, 0, np->name);
if (IS_ERR(priv.base)) {
- pr_err("%s: unable to map resource", np->name);
+ pr_err("%s: unable to map resource\n", np->name);
return PTR_ERR(priv.base);
}
diff --git a/drivers/clocksource/bcm2835_timer.c b/drivers/clocksource/bcm2835_timer.c
index f2f29d2..dce4430 100644
--- a/drivers/clocksource/bcm2835_timer.c
+++ b/drivers/clocksource/bcm2835_timer.c
@@ -89,13 +89,13 @@
base = of_iomap(node, 0);
if (!base) {
- pr_err("Can't remap registers");
+ pr_err("Can't remap registers\n");
return -ENXIO;
}
ret = of_property_read_u32(node, "clock-frequency", &freq);
if (ret) {
- pr_err("Can't read clock-frequency");
+ pr_err("Can't read clock-frequency\n");
goto err_iounmap;
}
@@ -107,7 +107,7 @@
irq = irq_of_parse_and_map(node, DEFAULT_TIMER);
if (irq <= 0) {
- pr_err("Can't parse IRQ");
+ pr_err("Can't parse IRQ\n");
ret = -EINVAL;
goto err_iounmap;
}
diff --git a/drivers/clocksource/bcm_kona_timer.c b/drivers/clocksource/bcm_kona_timer.c
index 92f6e4d..fda5e147 100644
--- a/drivers/clocksource/bcm_kona_timer.c
+++ b/drivers/clocksource/bcm_kona_timer.c
@@ -179,7 +179,7 @@
} else if (!of_property_read_u32(node, "clock-frequency", &freq)) {
arch_timer_rate = freq;
} else {
- pr_err("Kona Timer v1 unable to determine clock-frequency");
+ pr_err("Kona Timer v1 unable to determine clock-frequency\n");
return -EINVAL;
}
diff --git a/drivers/clocksource/clksrc-probe.c b/drivers/clocksource/clksrc-probe.c
index bc62be9..ac701ff 100644
--- a/drivers/clocksource/clksrc-probe.c
+++ b/drivers/clocksource/clksrc-probe.c
@@ -40,7 +40,7 @@
ret = init_func_ret(np);
if (ret) {
- pr_err("Failed to initialize '%s': %d",
+ pr_err("Failed to initialize '%s': %d\n",
of_node_full_name(np), ret);
continue;
}
diff --git a/drivers/clocksource/dw_apb_timer.c b/drivers/clocksource/dw_apb_timer.c
index 63e4f55..1f5f734 100644
--- a/drivers/clocksource/dw_apb_timer.c
+++ b/drivers/clocksource/dw_apb_timer.c
@@ -101,7 +101,7 @@
struct dw_apb_clock_event_device *dw_ced = ced_to_dw_apb_ced(evt);
if (!evt->event_handler) {
- pr_info("Spurious APBT timer interrupt %d", irq);
+ pr_info("Spurious APBT timer interrupt %d\n", irq);
return IRQ_NONE;
}
@@ -257,7 +257,9 @@
clockevents_calc_mult_shift(&dw_ced->ced, freq, APBT_MIN_PERIOD);
dw_ced->ced.max_delta_ns = clockevent_delta2ns(0x7fffffff,
&dw_ced->ced);
+ dw_ced->ced.max_delta_ticks = 0x7fffffff;
dw_ced->ced.min_delta_ns = clockevent_delta2ns(5000, &dw_ced->ced);
+ dw_ced->ced.min_delta_ticks = 5000;
dw_ced->ced.cpumask = cpumask_of(cpu);
dw_ced->ced.features = CLOCK_EVT_FEAT_PERIODIC |
CLOCK_EVT_FEAT_ONESHOT | CLOCK_EVT_FEAT_DYNIRQ;
diff --git a/drivers/clocksource/em_sti.c b/drivers/clocksource/em_sti.c
index aff87df..bc48cbf 100644
--- a/drivers/clocksource/em_sti.c
+++ b/drivers/clocksource/em_sti.c
@@ -78,15 +78,12 @@
int ret;
/* enable clock */
- ret = clk_prepare_enable(p->clk);
+ ret = clk_enable(p->clk);
if (ret) {
dev_err(&p->pdev->dev, "cannot enable clock\n");
return ret;
}
- /* configure channel, periodic mode and maximum timeout */
- p->rate = clk_get_rate(p->clk);
-
/* reset the counter */
em_sti_write(p, STI_SET_H, 0x40000000);
em_sti_write(p, STI_SET_L, 0x00000000);
@@ -107,7 +104,7 @@
em_sti_write(p, STI_INTENCLR, 3);
/* stop clock */
- clk_disable_unprepare(p->clk);
+ clk_disable(p->clk);
}
static u64 em_sti_count(struct em_sti_priv *p)
@@ -205,13 +202,9 @@
static int em_sti_clocksource_enable(struct clocksource *cs)
{
- int ret;
struct em_sti_priv *p = cs_to_em_sti(cs);
- ret = em_sti_start(p, USER_CLOCKSOURCE);
- if (!ret)
- __clocksource_update_freq_hz(cs, p->rate);
- return ret;
+ return em_sti_start(p, USER_CLOCKSOURCE);
}
static void em_sti_clocksource_disable(struct clocksource *cs)
@@ -240,8 +233,7 @@
dev_info(&p->pdev->dev, "used as clock source\n");
- /* Register with dummy 1 Hz value, gets updated in ->enable() */
- clocksource_register_hz(cs, 1);
+ clocksource_register_hz(cs, p->rate);
return 0;
}
@@ -263,7 +255,6 @@
dev_info(&p->pdev->dev, "used for oneshot clock events\n");
em_sti_start(p, USER_CLOCKEVENT);
- clockevents_config(&p->ced, p->rate);
return 0;
}
@@ -294,8 +285,7 @@
dev_info(&p->pdev->dev, "used for clock events\n");
- /* Register with dummy 1 Hz value, gets updated in ->set_state_oneshot() */
- clockevents_config_and_register(ced, 1, 2, 0xffffffff);
+ clockevents_config_and_register(ced, p->rate, 2, 0xffffffff);
}
static int em_sti_probe(struct platform_device *pdev)
@@ -303,6 +293,7 @@
struct em_sti_priv *p;
struct resource *res;
int irq;
+ int ret;
p = devm_kzalloc(&pdev->dev, sizeof(*p), GFP_KERNEL);
if (p == NULL)
@@ -323,6 +314,13 @@
if (IS_ERR(p->base))
return PTR_ERR(p->base);
+ if (devm_request_irq(&pdev->dev, irq, em_sti_interrupt,
+ IRQF_TIMER | IRQF_IRQPOLL | IRQF_NOBALANCING,
+ dev_name(&pdev->dev), p)) {
+ dev_err(&pdev->dev, "failed to request low IRQ\n");
+ return -ENOENT;
+ }
+
/* get hold of clock */
p->clk = devm_clk_get(&pdev->dev, "sclk");
if (IS_ERR(p->clk)) {
@@ -330,13 +328,21 @@
return PTR_ERR(p->clk);
}
- if (devm_request_irq(&pdev->dev, irq, em_sti_interrupt,
- IRQF_TIMER | IRQF_IRQPOLL | IRQF_NOBALANCING,
- dev_name(&pdev->dev), p)) {
- dev_err(&pdev->dev, "failed to request low IRQ\n");
- return -ENOENT;
+ ret = clk_prepare(p->clk);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "cannot prepare clock\n");
+ return ret;
}
+ ret = clk_enable(p->clk);
+ if (ret < 0) {
+ dev_err(&p->pdev->dev, "cannot enable clock\n");
+ clk_unprepare(p->clk);
+ return ret;
+ }
+ p->rate = clk_get_rate(p->clk);
+ clk_disable(p->clk);
+
raw_spin_lock_init(&p->lock);
em_sti_register_clockevent(p);
em_sti_register_clocksource(p);
diff --git a/drivers/clocksource/h8300_timer8.c b/drivers/clocksource/h8300_timer8.c
index 546bb18..804c489 100644
--- a/drivers/clocksource/h8300_timer8.c
+++ b/drivers/clocksource/h8300_timer8.c
@@ -101,15 +101,7 @@
static void timer8_clock_event_start(struct timer8_priv *p, unsigned long delta)
{
- struct clock_event_device *ced = &p->ced;
-
timer8_start(p);
-
- ced->shift = 32;
- ced->mult = div_sc(p->rate, NSEC_PER_SEC, ced->shift);
- ced->max_delta_ns = clockevent_delta2ns(0xffff, ced);
- ced->min_delta_ns = clockevent_delta2ns(0x0001, ced);
-
timer8_set_next(p, delta);
}
diff --git a/drivers/clocksource/meson6_timer.c b/drivers/clocksource/meson6_timer.c
index 52af591..39d21f6 100644
--- a/drivers/clocksource/meson6_timer.c
+++ b/drivers/clocksource/meson6_timer.c
@@ -133,13 +133,13 @@
timer_base = of_io_request_and_map(node, 0, "meson6-timer");
if (IS_ERR(timer_base)) {
- pr_err("Can't map registers");
+ pr_err("Can't map registers\n");
return -ENXIO;
}
irq = irq_of_parse_and_map(node, 0);
if (irq <= 0) {
- pr_err("Can't parse IRQ");
+ pr_err("Can't parse IRQ\n");
return -EINVAL;
}
diff --git a/drivers/clocksource/metag_generic.c b/drivers/clocksource/metag_generic.c
index 6fcf965..3e5fa2f 100644
--- a/drivers/clocksource/metag_generic.c
+++ b/drivers/clocksource/metag_generic.c
@@ -114,7 +114,9 @@
clk->mult = div_sc(hwtimer_freq, NSEC_PER_SEC, clk->shift);
clk->max_delta_ns = clockevent_delta2ns(0x7fffffff, clk);
+ clk->max_delta_ticks = 0x7fffffff;
clk->min_delta_ns = clockevent_delta2ns(0xf, clk);
+ clk->min_delta_ticks = 0xf;
clk->cpumask = cpumask_of(cpu);
clockevents_register_device(clk);
diff --git a/drivers/clocksource/mips-gic-timer.c b/drivers/clocksource/mips-gic-timer.c
index d9ef7a6..3f52ee2 100644
--- a/drivers/clocksource/mips-gic-timer.c
+++ b/drivers/clocksource/mips-gic-timer.c
@@ -154,19 +154,6 @@
return ret;
}
-void __init gic_clocksource_init(unsigned int frequency)
-{
- gic_frequency = frequency;
- gic_timer_irq = MIPS_GIC_IRQ_BASE +
- GIC_LOCAL_TO_HWIRQ(GIC_LOCAL_INT_COMPARE);
-
- __gic_clocksource_init();
- gic_clockevent_init();
-
- /* And finally start the counter */
- gic_start_count();
-}
-
static int __init gic_clocksource_of_init(struct device_node *node)
{
struct clk *clk;
@@ -174,7 +161,7 @@
if (!gic_present || !node->parent ||
!of_device_is_compatible(node->parent, "mti,gic")) {
- pr_warn("No DT definition for the mips gic driver");
+ pr_warn("No DT definition for the mips gic driver\n");
return -ENXIO;
}
diff --git a/drivers/clocksource/nomadik-mtu.c b/drivers/clocksource/nomadik-mtu.c
index 3c124d1..7d44de3 100644
--- a/drivers/clocksource/nomadik-mtu.c
+++ b/drivers/clocksource/nomadik-mtu.c
@@ -260,25 +260,25 @@
base = of_iomap(node, 0);
if (!base) {
- pr_err("Can't remap registers");
+ pr_err("Can't remap registers\n");
return -ENXIO;
}
pclk = of_clk_get_by_name(node, "apb_pclk");
if (IS_ERR(pclk)) {
- pr_err("could not get apb_pclk");
+ pr_err("could not get apb_pclk\n");
return PTR_ERR(pclk);
}
clk = of_clk_get_by_name(node, "timclk");
if (IS_ERR(clk)) {
- pr_err("could not get timclk");
+ pr_err("could not get timclk\n");
return PTR_ERR(clk);
}
irq = irq_of_parse_and_map(node, 0);
if (irq <= 0) {
- pr_err("Can't parse IRQ");
+ pr_err("Can't parse IRQ\n");
return -EINVAL;
}
diff --git a/drivers/clocksource/numachip.c b/drivers/clocksource/numachip.c
index 4e0f11f..6a20dc8 100644
--- a/drivers/clocksource/numachip.c
+++ b/drivers/clocksource/numachip.c
@@ -51,7 +51,9 @@
.mult = 1,
.shift = 0,
.min_delta_ns = 1250,
+ .min_delta_ticks = 1250,
.max_delta_ns = LONG_MAX,
+ .max_delta_ticks = LONG_MAX,
};
static void numachip_timer_interrupt(void)
diff --git a/drivers/clocksource/pxa_timer.c b/drivers/clocksource/pxa_timer.c
index 1c24de2..a10fa667 100644
--- a/drivers/clocksource/pxa_timer.c
+++ b/drivers/clocksource/pxa_timer.c
@@ -166,14 +166,14 @@
ret = setup_irq(irq, &pxa_ost0_irq);
if (ret) {
- pr_err("Failed to setup irq");
+ pr_err("Failed to setup irq\n");
return ret;
}
ret = clocksource_mmio_init(timer_base + OSCR, "oscr0", clock_tick_rate, 200,
32, clocksource_mmio_readl_up);
if (ret) {
- pr_err("Failed to init clocksource");
+ pr_err("Failed to init clocksource\n");
return ret;
}
@@ -203,7 +203,7 @@
ret = clk_prepare_enable(clk);
if (ret) {
- pr_crit("Failed to prepare clock");
+ pr_crit("Failed to prepare clock\n");
return ret;
}
diff --git a/drivers/clocksource/rockchip_timer.c b/drivers/clocksource/rockchip_timer.c
index 23e267a..49c02be 100644
--- a/drivers/clocksource/rockchip_timer.c
+++ b/drivers/clocksource/rockchip_timer.c
@@ -11,6 +11,8 @@
#include <linux/clockchips.h>
#include <linux/init.h>
#include <linux/interrupt.h>
+#include <linux/sched_clock.h>
+#include <linux/slab.h>
#include <linux/of.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
@@ -19,6 +21,8 @@
#define TIMER_LOAD_COUNT0 0x00
#define TIMER_LOAD_COUNT1 0x04
+#define TIMER_CURRENT_VALUE0 0x08
+#define TIMER_CURRENT_VALUE1 0x0C
#define TIMER_CONTROL_REG3288 0x10
#define TIMER_CONTROL_REG3399 0x1c
#define TIMER_INT_STATUS 0x18
@@ -29,103 +33,118 @@
#define TIMER_MODE_USER_DEFINED_COUNT (1 << 1)
#define TIMER_INT_UNMASK (1 << 2)
-struct bc_timer {
- struct clock_event_device ce;
+struct rk_timer {
void __iomem *base;
void __iomem *ctrl;
+ struct clk *clk;
+ struct clk *pclk;
u32 freq;
+ int irq;
};
-static struct bc_timer bc_timer;
+struct rk_clkevt {
+ struct clock_event_device ce;
+ struct rk_timer timer;
+};
-static inline struct bc_timer *rk_timer(struct clock_event_device *ce)
+static struct rk_clkevt *rk_clkevt;
+static struct rk_timer *rk_clksrc;
+
+static inline struct rk_timer *rk_timer(struct clock_event_device *ce)
{
- return container_of(ce, struct bc_timer, ce);
+ return &container_of(ce, struct rk_clkevt, ce)->timer;
}
-static inline void __iomem *rk_base(struct clock_event_device *ce)
+static inline void rk_timer_disable(struct rk_timer *timer)
{
- return rk_timer(ce)->base;
+ writel_relaxed(TIMER_DISABLE, timer->ctrl);
}
-static inline void __iomem *rk_ctrl(struct clock_event_device *ce)
+static inline void rk_timer_enable(struct rk_timer *timer, u32 flags)
{
- return rk_timer(ce)->ctrl;
-}
-
-static inline void rk_timer_disable(struct clock_event_device *ce)
-{
- writel_relaxed(TIMER_DISABLE, rk_ctrl(ce));
-}
-
-static inline void rk_timer_enable(struct clock_event_device *ce, u32 flags)
-{
- writel_relaxed(TIMER_ENABLE | TIMER_INT_UNMASK | flags,
- rk_ctrl(ce));
+ writel_relaxed(TIMER_ENABLE | flags, timer->ctrl);
}
static void rk_timer_update_counter(unsigned long cycles,
- struct clock_event_device *ce)
+ struct rk_timer *timer)
{
- writel_relaxed(cycles, rk_base(ce) + TIMER_LOAD_COUNT0);
- writel_relaxed(0, rk_base(ce) + TIMER_LOAD_COUNT1);
+ writel_relaxed(cycles, timer->base + TIMER_LOAD_COUNT0);
+ writel_relaxed(0, timer->base + TIMER_LOAD_COUNT1);
}
-static void rk_timer_interrupt_clear(struct clock_event_device *ce)
+static void rk_timer_interrupt_clear(struct rk_timer *timer)
{
- writel_relaxed(1, rk_base(ce) + TIMER_INT_STATUS);
+ writel_relaxed(1, timer->base + TIMER_INT_STATUS);
}
static inline int rk_timer_set_next_event(unsigned long cycles,
struct clock_event_device *ce)
{
- rk_timer_disable(ce);
- rk_timer_update_counter(cycles, ce);
- rk_timer_enable(ce, TIMER_MODE_USER_DEFINED_COUNT);
+ struct rk_timer *timer = rk_timer(ce);
+
+ rk_timer_disable(timer);
+ rk_timer_update_counter(cycles, timer);
+ rk_timer_enable(timer, TIMER_MODE_USER_DEFINED_COUNT |
+ TIMER_INT_UNMASK);
return 0;
}
static int rk_timer_shutdown(struct clock_event_device *ce)
{
- rk_timer_disable(ce);
+ struct rk_timer *timer = rk_timer(ce);
+
+ rk_timer_disable(timer);
return 0;
}
static int rk_timer_set_periodic(struct clock_event_device *ce)
{
- rk_timer_disable(ce);
- rk_timer_update_counter(rk_timer(ce)->freq / HZ - 1, ce);
- rk_timer_enable(ce, TIMER_MODE_FREE_RUNNING);
+ struct rk_timer *timer = rk_timer(ce);
+
+ rk_timer_disable(timer);
+ rk_timer_update_counter(timer->freq / HZ - 1, timer);
+ rk_timer_enable(timer, TIMER_MODE_FREE_RUNNING | TIMER_INT_UNMASK);
return 0;
}
static irqreturn_t rk_timer_interrupt(int irq, void *dev_id)
{
struct clock_event_device *ce = dev_id;
+ struct rk_timer *timer = rk_timer(ce);
- rk_timer_interrupt_clear(ce);
+ rk_timer_interrupt_clear(timer);
if (clockevent_state_oneshot(ce))
- rk_timer_disable(ce);
+ rk_timer_disable(timer);
ce->event_handler(ce);
return IRQ_HANDLED;
}
-static int __init rk_timer_init(struct device_node *np, u32 ctrl_reg)
+static u64 notrace rk_timer_sched_read(void)
{
- struct clock_event_device *ce = &bc_timer.ce;
+ return ~readl_relaxed(rk_clksrc->base + TIMER_CURRENT_VALUE0);
+}
+
+static int __init
+rk_timer_probe(struct rk_timer *timer, struct device_node *np)
+{
struct clk *timer_clk;
struct clk *pclk;
int ret = -EINVAL, irq;
+ u32 ctrl_reg = TIMER_CONTROL_REG3288;
- bc_timer.base = of_iomap(np, 0);
- if (!bc_timer.base) {
+ timer->base = of_iomap(np, 0);
+ if (!timer->base) {
pr_err("Failed to get base address for '%s'\n", TIMER_NAME);
return -ENXIO;
}
- bc_timer.ctrl = bc_timer.base + ctrl_reg;
+
+ if (of_device_is_compatible(np, "rockchip,rk3399-timer"))
+ ctrl_reg = TIMER_CONTROL_REG3399;
+
+ timer->ctrl = timer->base + ctrl_reg;
pclk = of_clk_get_by_name(np, "pclk");
if (IS_ERR(pclk)) {
@@ -139,6 +158,7 @@
pr_err("Failed to enable pclk for '%s'\n", TIMER_NAME);
goto out_unmap;
}
+ timer->pclk = pclk;
timer_clk = of_clk_get_by_name(np, "timer");
if (IS_ERR(timer_clk)) {
@@ -152,8 +172,9 @@
pr_err("Failed to enable timer clock\n");
goto out_timer_clk;
}
+ timer->clk = timer_clk;
- bc_timer.freq = clk_get_rate(timer_clk);
+ timer->freq = clk_get_rate(timer_clk);
irq = irq_of_parse_and_map(np, 0);
if (!irq) {
@@ -161,28 +182,10 @@
pr_err("Failed to map interrupts for '%s'\n", TIMER_NAME);
goto out_irq;
}
+ timer->irq = irq;
- ce->name = TIMER_NAME;
- ce->features = CLOCK_EVT_FEAT_PERIODIC | CLOCK_EVT_FEAT_ONESHOT |
- CLOCK_EVT_FEAT_DYNIRQ;
- ce->set_next_event = rk_timer_set_next_event;
- ce->set_state_shutdown = rk_timer_shutdown;
- ce->set_state_periodic = rk_timer_set_periodic;
- ce->irq = irq;
- ce->cpumask = cpu_possible_mask;
- ce->rating = 250;
-
- rk_timer_interrupt_clear(ce);
- rk_timer_disable(ce);
-
- ret = request_irq(irq, rk_timer_interrupt, IRQF_TIMER, TIMER_NAME, ce);
- if (ret) {
- pr_err("Failed to initialize '%s': %d\n", TIMER_NAME, ret);
- goto out_irq;
- }
-
- clockevents_config_and_register(ce, bc_timer.freq, 1, UINT_MAX);
-
+ rk_timer_interrupt_clear(timer);
+ rk_timer_disable(timer);
return 0;
out_irq:
@@ -190,22 +193,115 @@
out_timer_clk:
clk_disable_unprepare(pclk);
out_unmap:
- iounmap(bc_timer.base);
+ iounmap(timer->base);
return ret;
}
-static int __init rk3288_timer_init(struct device_node *np)
+static void __init rk_timer_cleanup(struct rk_timer *timer)
{
- return rk_timer_init(np, TIMER_CONTROL_REG3288);
+ clk_disable_unprepare(timer->clk);
+ clk_disable_unprepare(timer->pclk);
+ iounmap(timer->base);
}
-static int __init rk3399_timer_init(struct device_node *np)
+static int __init rk_clkevt_init(struct device_node *np)
{
- return rk_timer_init(np, TIMER_CONTROL_REG3399);
+ struct clock_event_device *ce;
+ int ret = -EINVAL;
+
+ rk_clkevt = kzalloc(sizeof(struct rk_clkevt), GFP_KERNEL);
+ if (!rk_clkevt) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ ret = rk_timer_probe(&rk_clkevt->timer, np);
+ if (ret)
+ goto out_probe;
+
+ ce = &rk_clkevt->ce;
+ ce->name = TIMER_NAME;
+ ce->features = CLOCK_EVT_FEAT_PERIODIC | CLOCK_EVT_FEAT_ONESHOT |
+ CLOCK_EVT_FEAT_DYNIRQ;
+ ce->set_next_event = rk_timer_set_next_event;
+ ce->set_state_shutdown = rk_timer_shutdown;
+ ce->set_state_periodic = rk_timer_set_periodic;
+ ce->irq = rk_clkevt->timer.irq;
+ ce->cpumask = cpu_possible_mask;
+ ce->rating = 250;
+
+ ret = request_irq(rk_clkevt->timer.irq, rk_timer_interrupt, IRQF_TIMER,
+ TIMER_NAME, ce);
+ if (ret) {
+ pr_err("Failed to initialize '%s': %d\n",
+ TIMER_NAME, ret);
+ goto out_irq;
+ }
+
+ clockevents_config_and_register(&rk_clkevt->ce,
+ rk_clkevt->timer.freq, 1, UINT_MAX);
+ return 0;
+
+out_irq:
+ rk_timer_cleanup(&rk_clkevt->timer);
+out_probe:
+ kfree(rk_clkevt);
+out:
+ /* Leave rk_clkevt not NULL to prevent future init */
+ rk_clkevt = ERR_PTR(ret);
+ return ret;
}
-CLOCKSOURCE_OF_DECLARE(rk3288_timer, "rockchip,rk3288-timer",
- rk3288_timer_init);
-CLOCKSOURCE_OF_DECLARE(rk3399_timer, "rockchip,rk3399-timer",
- rk3399_timer_init);
+static int __init rk_clksrc_init(struct device_node *np)
+{
+ int ret = -EINVAL;
+
+ rk_clksrc = kzalloc(sizeof(struct rk_timer), GFP_KERNEL);
+ if (!rk_clksrc) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ ret = rk_timer_probe(rk_clksrc, np);
+ if (ret)
+ goto out_probe;
+
+ rk_timer_update_counter(UINT_MAX, rk_clksrc);
+ rk_timer_enable(rk_clksrc, 0);
+
+ ret = clocksource_mmio_init(rk_clksrc->base + TIMER_CURRENT_VALUE0,
+ TIMER_NAME, rk_clksrc->freq, 250, 32,
+ clocksource_mmio_readl_down);
+ if (ret) {
+ pr_err("Failed to register clocksource");
+ goto out_clocksource;
+ }
+
+ sched_clock_register(rk_timer_sched_read, 32, rk_clksrc->freq);
+ return 0;
+
+out_clocksource:
+ rk_timer_cleanup(rk_clksrc);
+out_probe:
+ kfree(rk_clksrc);
+out:
+ /* Leave rk_clksrc not NULL to prevent future init */
+ rk_clksrc = ERR_PTR(ret);
+ return ret;
+}
+
+static int __init rk_timer_init(struct device_node *np)
+{
+ if (!rk_clkevt)
+ return rk_clkevt_init(np);
+
+ if (!rk_clksrc)
+ return rk_clksrc_init(np);
+
+ pr_err("Too many timer definitions for '%s'\n", TIMER_NAME);
+ return -EINVAL;
+}
+
+CLOCKSOURCE_OF_DECLARE(rk3288_timer, "rockchip,rk3288-timer", rk_timer_init);
+CLOCKSOURCE_OF_DECLARE(rk3399_timer, "rockchip,rk3399-timer", rk_timer_init);
diff --git a/drivers/clocksource/samsung_pwm_timer.c b/drivers/clocksource/samsung_pwm_timer.c
index 0093ece..a68e653 100644
--- a/drivers/clocksource/samsung_pwm_timer.c
+++ b/drivers/clocksource/samsung_pwm_timer.c
@@ -385,7 +385,7 @@
mask = ~pwm.variant.output_mask & ((1 << SAMSUNG_PWM_NUM) - 1);
channel = fls(mask) - 1;
if (channel < 0) {
- pr_crit("failed to find PWM channel for clocksource");
+ pr_crit("failed to find PWM channel for clocksource\n");
return -EINVAL;
}
pwm.source_id = channel;
@@ -393,7 +393,7 @@
mask &= ~(1 << channel);
channel = fls(mask) - 1;
if (channel < 0) {
- pr_crit("failed to find PWM channel for clock event");
+ pr_crit("failed to find PWM channel for clock event\n");
return -EINVAL;
}
pwm.event_id = channel;
@@ -448,7 +448,7 @@
pwm.timerclk = of_clk_get_by_name(np, "timers");
if (IS_ERR(pwm.timerclk)) {
- pr_crit("failed to get timers clock for timer");
+ pr_crit("failed to get timers clock for timer\n");
return PTR_ERR(pwm.timerclk);
}
diff --git a/drivers/clocksource/sh_cmt.c b/drivers/clocksource/sh_cmt.c
index 28757ed..e09e8bf 100644
--- a/drivers/clocksource/sh_cmt.c
+++ b/drivers/clocksource/sh_cmt.c
@@ -103,7 +103,6 @@
unsigned long match_value;
unsigned long next_match_value;
unsigned long max_match_value;
- unsigned long rate;
raw_spinlock_t lock;
struct clock_event_device ced;
struct clocksource cs;
@@ -118,6 +117,7 @@
void __iomem *mapbase;
struct clk *clk;
+ unsigned long rate;
raw_spinlock_t lock; /* Protect the shared start/stop register */
@@ -320,7 +320,7 @@
raw_spin_unlock_irqrestore(&ch->cmt->lock, flags);
}
-static int sh_cmt_enable(struct sh_cmt_channel *ch, unsigned long *rate)
+static int sh_cmt_enable(struct sh_cmt_channel *ch)
{
int k, ret;
@@ -340,11 +340,9 @@
/* configure channel, periodic mode and maximum timeout */
if (ch->cmt->info->width == 16) {
- *rate = clk_get_rate(ch->cmt->clk) / 512;
sh_cmt_write_cmcsr(ch, SH_CMT16_CMCSR_CMIE |
SH_CMT16_CMCSR_CKS512);
} else {
- *rate = clk_get_rate(ch->cmt->clk) / 8;
sh_cmt_write_cmcsr(ch, SH_CMT32_CMCSR_CMM |
SH_CMT32_CMCSR_CMTOUT_IE |
SH_CMT32_CMCSR_CMR_IRQ |
@@ -572,7 +570,7 @@
raw_spin_lock_irqsave(&ch->lock, flags);
if (!(ch->flags & (FLAG_CLOCKEVENT | FLAG_CLOCKSOURCE)))
- ret = sh_cmt_enable(ch, &ch->rate);
+ ret = sh_cmt_enable(ch);
if (ret)
goto out;
@@ -640,10 +638,9 @@
ch->total_cycles = 0;
ret = sh_cmt_start(ch, FLAG_CLOCKSOURCE);
- if (!ret) {
- __clocksource_update_freq_hz(cs, ch->rate);
+ if (!ret)
ch->cs_enabled = true;
- }
+
return ret;
}
@@ -697,8 +694,7 @@
dev_info(&ch->cmt->pdev->dev, "ch%u: used as clock source\n",
ch->index);
- /* Register with dummy 1 Hz value, gets updated in ->enable() */
- clocksource_register_hz(cs, 1);
+ clocksource_register_hz(cs, ch->cmt->rate);
return 0;
}
@@ -709,19 +705,10 @@
static void sh_cmt_clock_event_start(struct sh_cmt_channel *ch, int periodic)
{
- struct clock_event_device *ced = &ch->ced;
-
sh_cmt_start(ch, FLAG_CLOCKEVENT);
- /* TODO: calculate good shift from rate and counter bit width */
-
- ced->shift = 32;
- ced->mult = div_sc(ch->rate, NSEC_PER_SEC, ced->shift);
- ced->max_delta_ns = clockevent_delta2ns(ch->max_match_value, ced);
- ced->min_delta_ns = clockevent_delta2ns(0x1f, ced);
-
if (periodic)
- sh_cmt_set_next(ch, ((ch->rate + HZ/2) / HZ) - 1);
+ sh_cmt_set_next(ch, ((ch->cmt->rate + HZ/2) / HZ) - 1);
else
sh_cmt_set_next(ch, ch->max_match_value);
}
@@ -824,6 +811,14 @@
ced->suspend = sh_cmt_clock_event_suspend;
ced->resume = sh_cmt_clock_event_resume;
+ /* TODO: calculate good shift from rate and counter bit width */
+ ced->shift = 32;
+ ced->mult = div_sc(ch->cmt->rate, NSEC_PER_SEC, ced->shift);
+ ced->max_delta_ns = clockevent_delta2ns(ch->max_match_value, ced);
+ ced->max_delta_ticks = ch->max_match_value;
+ ced->min_delta_ns = clockevent_delta2ns(0x1f, ced);
+ ced->min_delta_ticks = 0x1f;
+
dev_info(&ch->cmt->pdev->dev, "ch%u: used for clock events\n",
ch->index);
clockevents_register_device(ced);
@@ -996,6 +991,18 @@
if (ret < 0)
goto err_clk_put;
+ /* Determine clock rate. */
+ ret = clk_enable(cmt->clk);
+ if (ret < 0)
+ goto err_clk_unprepare;
+
+ if (cmt->info->width == 16)
+ cmt->rate = clk_get_rate(cmt->clk) / 512;
+ else
+ cmt->rate = clk_get_rate(cmt->clk) / 8;
+
+ clk_disable(cmt->clk);
+
/* Map the memory resource(s). */
ret = sh_cmt_map_memory(cmt);
if (ret < 0)
diff --git a/drivers/clocksource/sh_tmu.c b/drivers/clocksource/sh_tmu.c
index 1fbf2aa..31d8816 100644
--- a/drivers/clocksource/sh_tmu.c
+++ b/drivers/clocksource/sh_tmu.c
@@ -46,7 +46,6 @@
void __iomem *base;
int irq;
- unsigned long rate;
unsigned long periodic;
struct clock_event_device ced;
struct clocksource cs;
@@ -59,6 +58,7 @@
void __iomem *mapbase;
struct clk *clk;
+ unsigned long rate;
enum sh_tmu_model model;
@@ -165,7 +165,6 @@
sh_tmu_write(ch, TCNT, 0xffffffff);
/* configure channel to parent clock / 4, irq off */
- ch->rate = clk_get_rate(ch->tmu->clk) / 4;
sh_tmu_write(ch, TCR, TCR_TPSC_CLK4);
/* enable channel */
@@ -271,10 +270,8 @@
return 0;
ret = sh_tmu_enable(ch);
- if (!ret) {
- __clocksource_update_freq_hz(cs, ch->rate);
+ if (!ret)
ch->cs_enabled = true;
- }
return ret;
}
@@ -334,8 +331,7 @@
dev_info(&ch->tmu->pdev->dev, "ch%u: used as clock source\n",
ch->index);
- /* Register with dummy 1 Hz value, gets updated in ->enable() */
- clocksource_register_hz(cs, 1);
+ clocksource_register_hz(cs, ch->tmu->rate);
return 0;
}
@@ -346,14 +342,10 @@
static void sh_tmu_clock_event_start(struct sh_tmu_channel *ch, int periodic)
{
- struct clock_event_device *ced = &ch->ced;
-
sh_tmu_enable(ch);
- clockevents_config(ced, ch->rate);
-
if (periodic) {
- ch->periodic = (ch->rate + HZ/2) / HZ;
+ ch->periodic = (ch->tmu->rate + HZ/2) / HZ;
sh_tmu_set_next(ch, ch->periodic, 1);
}
}
@@ -435,7 +427,7 @@
dev_info(&ch->tmu->pdev->dev, "ch%u: used for clock events\n",
ch->index);
- clockevents_config_and_register(ced, 1, 0x300, 0xffffffff);
+ clockevents_config_and_register(ced, ch->tmu->rate, 0x300, 0xffffffff);
ret = request_irq(ch->irq, sh_tmu_interrupt,
IRQF_TIMER | IRQF_IRQPOLL | IRQF_NOBALANCING,
@@ -561,6 +553,14 @@
if (ret < 0)
goto err_clk_put;
+ /* Determine clock rate. */
+ ret = clk_enable(tmu->clk);
+ if (ret < 0)
+ goto err_clk_unprepare;
+
+ tmu->rate = clk_get_rate(tmu->clk) / 4;
+ clk_disable(tmu->clk);
+
/* Map the memory resource. */
ret = sh_tmu_map_memory(tmu);
if (ret < 0) {
diff --git a/drivers/clocksource/sun4i_timer.c b/drivers/clocksource/sun4i_timer.c
index c83452c..4452d5c 100644
--- a/drivers/clocksource/sun4i_timer.c
+++ b/drivers/clocksource/sun4i_timer.c
@@ -159,25 +159,25 @@
timer_base = of_iomap(node, 0);
if (!timer_base) {
- pr_crit("Can't map registers");
+ pr_crit("Can't map registers\n");
return -ENXIO;
}
irq = irq_of_parse_and_map(node, 0);
if (irq <= 0) {
- pr_crit("Can't parse IRQ");
+ pr_crit("Can't parse IRQ\n");
return -EINVAL;
}
clk = of_clk_get(node, 0);
if (IS_ERR(clk)) {
- pr_crit("Can't get timer clock");
+ pr_crit("Can't get timer clock\n");
return PTR_ERR(clk);
}
ret = clk_prepare_enable(clk);
if (ret) {
- pr_err("Failed to prepare clock");
+ pr_err("Failed to prepare clock\n");
return ret;
}
@@ -200,7 +200,7 @@
ret = clocksource_mmio_init(timer_base + TIMER_CNTVAL_REG(1), node->name,
rate, 350, 32, clocksource_mmio_readl_down);
if (ret) {
- pr_err("Failed to register clocksource");
+ pr_err("Failed to register clocksource\n");
return ret;
}
diff --git a/drivers/clocksource/tegra20_timer.c b/drivers/clocksource/tegra20_timer.c
index f960891..b9990b9 100644
--- a/drivers/clocksource/tegra20_timer.c
+++ b/drivers/clocksource/tegra20_timer.c
@@ -245,7 +245,7 @@
rtc_base = of_iomap(np, 0);
if (!rtc_base) {
- pr_err("Can't map RTC registers");
+ pr_err("Can't map RTC registers\n");
return -ENXIO;
}
diff --git a/drivers/clocksource/time-armada-370-xp.c b/drivers/clocksource/time-armada-370-xp.c
index 4440aef..aea4380 100644
--- a/drivers/clocksource/time-armada-370-xp.c
+++ b/drivers/clocksource/time-armada-370-xp.c
@@ -247,13 +247,13 @@
timer_base = of_iomap(np, 0);
if (!timer_base) {
- pr_err("Failed to iomap");
+ pr_err("Failed to iomap\n");
return -ENXIO;
}
local_base = of_iomap(np, 1);
if (!local_base) {
- pr_err("Failed to iomap");
+ pr_err("Failed to iomap\n");
return -ENXIO;
}
@@ -298,7 +298,7 @@
"armada_370_xp_clocksource",
timer_clk, 300, 32, clocksource_mmio_readl_down);
if (res) {
- pr_err("Failed to initialize clocksource mmio");
+ pr_err("Failed to initialize clocksource mmio\n");
return res;
}
@@ -315,7 +315,7 @@
armada_370_xp_evt);
/* Immediately configure the timer on the boot CPU */
if (res) {
- pr_err("Failed to request percpu irq");
+ pr_err("Failed to request percpu irq\n");
return res;
}
@@ -324,7 +324,7 @@
armada_370_xp_timer_starting_cpu,
armada_370_xp_timer_dying_cpu);
if (res) {
- pr_err("Failed to setup hotplug state and timer");
+ pr_err("Failed to setup hotplug state and timer\n");
return res;
}
@@ -339,7 +339,7 @@
int ret;
if (IS_ERR(clk)) {
- pr_err("Failed to get clock");
+ pr_err("Failed to get clock\n");
return PTR_ERR(clk);
}
@@ -375,7 +375,7 @@
/* Must have at least a clock */
if (IS_ERR(clk)) {
- pr_err("Failed to get clock");
+ pr_err("Failed to get clock\n");
return PTR_ERR(clk);
}
@@ -399,7 +399,7 @@
clk = of_clk_get(np, 0);
if (IS_ERR(clk)) {
- pr_err("Failed to get clock");
+ pr_err("Failed to get clock\n");
return PTR_ERR(clk);
}
diff --git a/drivers/clocksource/time-efm32.c b/drivers/clocksource/time-efm32.c
index 5ac344b..ce0f97b 100644
--- a/drivers/clocksource/time-efm32.c
+++ b/drivers/clocksource/time-efm32.c
@@ -235,7 +235,7 @@
ret = setup_irq(irq, &efm32_clock_event_irq);
if (ret) {
- pr_err("Failed setup irq");
+ pr_err("Failed setup irq\n");
goto err_setup_irq;
}
diff --git a/drivers/clocksource/time-orion.c b/drivers/clocksource/time-orion.c
index a28f496..b9b97f6 100644
--- a/drivers/clocksource/time-orion.c
+++ b/drivers/clocksource/time-orion.c
@@ -15,6 +15,7 @@
#include <linux/bitops.h>
#include <linux/clk.h>
#include <linux/clockchips.h>
+#include <linux/delay.h>
#include <linux/interrupt.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
@@ -36,6 +37,21 @@
static void __iomem *timer_base;
+static unsigned long notrace orion_read_timer(void)
+{
+ return ~readl(timer_base + TIMER0_VAL);
+}
+
+static struct delay_timer orion_delay_timer = {
+ .read_current_timer = orion_read_timer,
+};
+
+static void orion_delay_timer_init(unsigned long rate)
+{
+ orion_delay_timer.freq = rate;
+ register_current_timer_delay(&orion_delay_timer);
+}
+
/*
* Free-running clocksource handling.
*/
@@ -106,6 +122,7 @@
static int __init orion_timer_init(struct device_node *np)
{
+ unsigned long rate;
struct clk *clk;
int irq, ret;
@@ -124,7 +141,7 @@
ret = clk_prepare_enable(clk);
if (ret) {
- pr_err("Failed to prepare clock");
+ pr_err("Failed to prepare clock\n");
return ret;
}
@@ -135,6 +152,8 @@
return -EINVAL;
}
+ rate = clk_get_rate(clk);
+
/* setup timer0 as free-running clocksource */
writel(~0, timer_base + TIMER0_VAL);
writel(~0, timer_base + TIMER0_RELOAD);
@@ -142,15 +161,15 @@
TIMER0_RELOAD_EN | TIMER0_EN,
TIMER0_RELOAD_EN | TIMER0_EN);
- ret = clocksource_mmio_init(timer_base + TIMER0_VAL, "orion_clocksource",
- clk_get_rate(clk), 300, 32,
+ ret = clocksource_mmio_init(timer_base + TIMER0_VAL,
+ "orion_clocksource", rate, 300, 32,
clocksource_mmio_readl_down);
if (ret) {
- pr_err("Failed to initialize mmio timer");
+ pr_err("Failed to initialize mmio timer\n");
return ret;
}
- sched_clock_register(orion_read_sched_clock, 32, clk_get_rate(clk));
+ sched_clock_register(orion_read_sched_clock, 32, rate);
/* setup timer1 as clockevent timer */
ret = setup_irq(irq, &orion_clkevt_irq);
@@ -162,9 +181,12 @@
ticks_per_jiffy = (clk_get_rate(clk) + HZ/2) / HZ;
orion_clkevt.cpumask = cpumask_of(0);
orion_clkevt.irq = irq;
- clockevents_config_and_register(&orion_clkevt, clk_get_rate(clk),
+ clockevents_config_and_register(&orion_clkevt, rate,
ORION_ONESHOT_MIN, ORION_ONESHOT_MAX);
+
+ orion_delay_timer_init(rate);
+
return 0;
}
CLOCKSOURCE_OF_DECLARE(orion_timer, "marvell,orion-timer", orion_timer_init);
diff --git a/drivers/clocksource/timer-atlas7.c b/drivers/clocksource/timer-atlas7.c
index 3d8a181..50300ee 100644
--- a/drivers/clocksource/timer-atlas7.c
+++ b/drivers/clocksource/timer-atlas7.c
@@ -192,7 +192,9 @@
ce->set_next_event = sirfsoc_timer_set_next_event;
clockevents_calc_mult_shift(ce, atlas7_timer_rate, 60);
ce->max_delta_ns = clockevent_delta2ns(-2, ce);
+ ce->max_delta_ticks = (unsigned long)-2;
ce->min_delta_ns = clockevent_delta2ns(2, ce);
+ ce->min_delta_ticks = 2;
ce->cpumask = cpumask_of(cpu);
action->dev_id = ce;
diff --git a/drivers/clocksource/timer-atmel-pit.c b/drivers/clocksource/timer-atmel-pit.c
index c0b5df3..cc11235 100644
--- a/drivers/clocksource/timer-atmel-pit.c
+++ b/drivers/clocksource/timer-atmel-pit.c
@@ -226,7 +226,7 @@
ret = clocksource_register_hz(&data->clksrc, pit_rate);
if (ret) {
- pr_err("Failed to register clocksource");
+ pr_err("Failed to register clocksource\n");
return ret;
}
diff --git a/drivers/clocksource/timer-digicolor.c b/drivers/clocksource/timer-digicolor.c
index e9f50d2..94a161e 100644
--- a/drivers/clocksource/timer-digicolor.c
+++ b/drivers/clocksource/timer-digicolor.c
@@ -161,19 +161,19 @@
*/
dc_timer_dev.base = of_iomap(node, 0);
if (!dc_timer_dev.base) {
- pr_err("Can't map registers");
+ pr_err("Can't map registers\n");
return -ENXIO;
}
irq = irq_of_parse_and_map(node, dc_timer_dev.timer_id);
if (irq <= 0) {
- pr_err("Can't parse IRQ");
+ pr_err("Can't parse IRQ\n");
return -EINVAL;
}
clk = of_clk_get(node, 0);
if (IS_ERR(clk)) {
- pr_err("Can't get timer clock");
+ pr_err("Can't get timer clock\n");
return PTR_ERR(clk);
}
clk_prepare_enable(clk);
diff --git a/drivers/clocksource/timer-gemini.c b/drivers/clocksource/timer-fttmr010.c
similarity index 72%
rename from drivers/clocksource/timer-gemini.c
rename to drivers/clocksource/timer-fttmr010.c
index dda27b7..b4a6f1e 100644
--- a/drivers/clocksource/timer-gemini.c
+++ b/drivers/clocksource/timer-fttmr010.c
@@ -1,5 +1,5 @@
/*
- * Gemini timer driver
+ * Faraday Technology FTTMR010 timer driver
* Copyright (C) 2017 Linus Walleij <linus.walleij@linaro.org>
*
* Based on a rewrite of arch/arm/mach-gemini/timer.c:
@@ -16,17 +16,7 @@
#include <linux/clockchips.h>
#include <linux/clocksource.h>
#include <linux/sched_clock.h>
-
-/*
- * Relevant registers in the global syscon
- */
-#define GLOBAL_STATUS 0x04
-#define CPU_AHB_RATIO_MASK (0x3 << 18)
-#define CPU_AHB_1_1 (0x0 << 18)
-#define CPU_AHB_3_2 (0x1 << 18)
-#define CPU_AHB_24_13 (0x2 << 18)
-#define CPU_AHB_2_1 (0x3 << 18)
-#define REG_TO_AHB_SPEED(reg) ((((reg) >> 15) & 0x7) * 10 + 130)
+#include <linux/clk.h>
/*
* Register definitions for the timers
@@ -77,12 +67,12 @@
static unsigned int tick_rate;
static void __iomem *base;
-static u64 notrace gemini_read_sched_clock(void)
+static u64 notrace fttmr010_read_sched_clock(void)
{
return readl(base + TIMER3_COUNT);
}
-static int gemini_timer_set_next_event(unsigned long cycles,
+static int fttmr010_timer_set_next_event(unsigned long cycles,
struct clock_event_device *evt)
{
u32 cr;
@@ -96,7 +86,7 @@
return 0;
}
-static int gemini_timer_shutdown(struct clock_event_device *evt)
+static int fttmr010_timer_shutdown(struct clock_event_device *evt)
{
u32 cr;
@@ -127,7 +117,7 @@
return 0;
}
-static int gemini_timer_set_periodic(struct clock_event_device *evt)
+static int fttmr010_timer_set_periodic(struct clock_event_device *evt)
{
u32 period = DIV_ROUND_CLOSEST(tick_rate, HZ);
u32 cr;
@@ -158,54 +148,40 @@
}
/* Use TIMER1 as clock event */
-static struct clock_event_device gemini_clockevent = {
+static struct clock_event_device fttmr010_clockevent = {
.name = "TIMER1",
/* Reasonably fast and accurate clock event */
.rating = 300,
.shift = 32,
.features = CLOCK_EVT_FEAT_PERIODIC |
CLOCK_EVT_FEAT_ONESHOT,
- .set_next_event = gemini_timer_set_next_event,
- .set_state_shutdown = gemini_timer_shutdown,
- .set_state_periodic = gemini_timer_set_periodic,
- .set_state_oneshot = gemini_timer_shutdown,
- .tick_resume = gemini_timer_shutdown,
+ .set_next_event = fttmr010_timer_set_next_event,
+ .set_state_shutdown = fttmr010_timer_shutdown,
+ .set_state_periodic = fttmr010_timer_set_periodic,
+ .set_state_oneshot = fttmr010_timer_shutdown,
+ .tick_resume = fttmr010_timer_shutdown,
};
/*
* IRQ handler for the timer
*/
-static irqreturn_t gemini_timer_interrupt(int irq, void *dev_id)
+static irqreturn_t fttmr010_timer_interrupt(int irq, void *dev_id)
{
- struct clock_event_device *evt = &gemini_clockevent;
+ struct clock_event_device *evt = &fttmr010_clockevent;
evt->event_handler(evt);
return IRQ_HANDLED;
}
-static struct irqaction gemini_timer_irq = {
- .name = "Gemini Timer Tick",
+static struct irqaction fttmr010_timer_irq = {
+ .name = "Faraday FTTMR010 Timer Tick",
.flags = IRQF_TIMER,
- .handler = gemini_timer_interrupt,
+ .handler = fttmr010_timer_interrupt,
};
-static int __init gemini_timer_of_init(struct device_node *np)
+static int __init fttmr010_timer_common_init(struct device_node *np)
{
- static struct regmap *map;
int irq;
- int ret;
- u32 val;
-
- map = syscon_regmap_lookup_by_phandle(np, "syscon");
- if (IS_ERR(map)) {
- pr_err("Can't get regmap for syscon handle");
- return -ENODEV;
- }
- ret = regmap_read(map, GLOBAL_STATUS, &val);
- if (ret) {
- pr_err("Can't read syscon status register");
- return -ENXIO;
- }
base = of_iomap(np, 0);
if (!base) {
@@ -219,26 +195,6 @@
return -EINVAL;
}
- tick_rate = REG_TO_AHB_SPEED(val) * 1000000;
- printk(KERN_INFO "Bus: %dMHz", tick_rate / 1000000);
-
- tick_rate /= 6; /* APB bus run AHB*(1/6) */
-
- switch (val & CPU_AHB_RATIO_MASK) {
- case CPU_AHB_1_1:
- printk(KERN_CONT "(1/1)\n");
- break;
- case CPU_AHB_3_2:
- printk(KERN_CONT "(3/2)\n");
- break;
- case CPU_AHB_24_13:
- printk(KERN_CONT "(24/13)\n");
- break;
- case CPU_AHB_2_1:
- printk(KERN_CONT "(2/1)\n");
- break;
- }
-
/*
* Reset the interrupt mask and status
*/
@@ -255,9 +211,9 @@
writel(0, base + TIMER3_MATCH1);
writel(0, base + TIMER3_MATCH2);
clocksource_mmio_init(base + TIMER3_COUNT,
- "gemini_clocksource", tick_rate,
+ "fttmr010_clocksource", tick_rate,
300, 32, clocksource_mmio_readl_up);
- sched_clock_register(gemini_read_sched_clock, 32, tick_rate);
+ sched_clock_register(fttmr010_read_sched_clock, 32, tick_rate);
/*
* Setup clockevent timer (interrupt-driven.)
@@ -266,12 +222,82 @@
writel(0, base + TIMER1_LOAD);
writel(0, base + TIMER1_MATCH1);
writel(0, base + TIMER1_MATCH2);
- setup_irq(irq, &gemini_timer_irq);
- gemini_clockevent.cpumask = cpumask_of(0);
- clockevents_config_and_register(&gemini_clockevent, tick_rate,
+ setup_irq(irq, &fttmr010_timer_irq);
+ fttmr010_clockevent.cpumask = cpumask_of(0);
+ clockevents_config_and_register(&fttmr010_clockevent, tick_rate,
1, 0xffffffff);
return 0;
}
-CLOCKSOURCE_OF_DECLARE(nomadik_mtu, "cortina,gemini-timer",
- gemini_timer_of_init);
+
+static int __init fttmr010_timer_of_init(struct device_node *np)
+{
+ /*
+ * These implementations require a clock reference.
+ * FIXME: we currently only support clocking using PCLK
+ * and using EXTCLK is not supported in the driver.
+ */
+ struct clk *clk;
+
+ clk = of_clk_get_by_name(np, "PCLK");
+ if (IS_ERR(clk)) {
+ pr_err("could not get PCLK");
+ return PTR_ERR(clk);
+ }
+ tick_rate = clk_get_rate(clk);
+
+ return fttmr010_timer_common_init(np);
+}
+CLOCKSOURCE_OF_DECLARE(fttmr010, "faraday,fttmr010", fttmr010_timer_of_init);
+
+/*
+ * Gemini-specific: relevant registers in the global syscon
+ */
+#define GLOBAL_STATUS 0x04
+#define CPU_AHB_RATIO_MASK (0x3 << 18)
+#define CPU_AHB_1_1 (0x0 << 18)
+#define CPU_AHB_3_2 (0x1 << 18)
+#define CPU_AHB_24_13 (0x2 << 18)
+#define CPU_AHB_2_1 (0x3 << 18)
+#define REG_TO_AHB_SPEED(reg) ((((reg) >> 15) & 0x7) * 10 + 130)
+
+static int __init gemini_timer_of_init(struct device_node *np)
+{
+ static struct regmap *map;
+ int ret;
+ u32 val;
+
+ map = syscon_regmap_lookup_by_phandle(np, "syscon");
+ if (IS_ERR(map)) {
+ pr_err("Can't get regmap for syscon handle\n");
+ return -ENODEV;
+ }
+ ret = regmap_read(map, GLOBAL_STATUS, &val);
+ if (ret) {
+ pr_err("Can't read syscon status register\n");
+ return -ENXIO;
+ }
+
+ tick_rate = REG_TO_AHB_SPEED(val) * 1000000;
+ pr_info("Bus: %dMHz ", tick_rate / 1000000);
+
+ tick_rate /= 6; /* APB bus run AHB*(1/6) */
+
+ switch (val & CPU_AHB_RATIO_MASK) {
+ case CPU_AHB_1_1:
+ pr_cont("(1/1)\n");
+ break;
+ case CPU_AHB_3_2:
+ pr_cont("(3/2)\n");
+ break;
+ case CPU_AHB_24_13:
+ pr_cont("(24/13)\n");
+ break;
+ case CPU_AHB_2_1:
+ pr_cont("(2/1)\n");
+ break;
+ }
+
+ return fttmr010_timer_common_init(np);
+}
+CLOCKSOURCE_OF_DECLARE(gemini, "cortina,gemini-timer", gemini_timer_of_init);
diff --git a/drivers/clocksource/timer-integrator-ap.c b/drivers/clocksource/timer-integrator-ap.c
index df6e672..04ad306 100644
--- a/drivers/clocksource/timer-integrator-ap.c
+++ b/drivers/clocksource/timer-integrator-ap.c
@@ -200,7 +200,7 @@
err = of_property_read_string(of_aliases,
"arm,timer-primary", &path);
if (err) {
- pr_warn("Failed to read property");
+ pr_warn("Failed to read property\n");
return err;
}
@@ -209,7 +209,7 @@
err = of_property_read_string(of_aliases,
"arm,timer-secondary", &path);
if (err) {
- pr_warn("Failed to read property");
+ pr_warn("Failed to read property\n");
return err;
}
diff --git a/drivers/clocksource/timer-nps.c b/drivers/clocksource/timer-nps.c
index da1f798..e74ea17 100644
--- a/drivers/clocksource/timer-nps.c
+++ b/drivers/clocksource/timer-nps.c
@@ -55,7 +55,7 @@
*clk = of_clk_get(node, 0);
ret = PTR_ERR_OR_ZERO(*clk);
if (ret) {
- pr_err("timer missing clk");
+ pr_err("timer missing clk\n");
return ret;
}
@@ -247,7 +247,7 @@
nps_timer0_irq = irq_of_parse_and_map(node, 0);
if (nps_timer0_irq <= 0) {
- pr_err("clockevent: missing irq");
+ pr_err("clockevent: missing irq\n");
return -EINVAL;
}
@@ -270,7 +270,7 @@
nps_timer_starting_cpu,
nps_timer_dying_cpu);
if (ret) {
- pr_err("Failed to setup hotplug state");
+ pr_err("Failed to setup hotplug state\n");
clk_disable_unprepare(clk);
free_percpu_irq(nps_timer0_irq, &nps_clockevent_device);
return ret;
diff --git a/drivers/clocksource/timer-prima2.c b/drivers/clocksource/timer-prima2.c
index bfa981a..b4122ed 100644
--- a/drivers/clocksource/timer-prima2.c
+++ b/drivers/clocksource/timer-prima2.c
@@ -196,20 +196,20 @@
clk = of_clk_get(np, 0);
if (IS_ERR(clk)) {
- pr_err("Failed to get clock");
+ pr_err("Failed to get clock\n");
return PTR_ERR(clk);
}
ret = clk_prepare_enable(clk);
if (ret) {
- pr_err("Failed to enable clock");
+ pr_err("Failed to enable clock\n");
return ret;
}
rate = clk_get_rate(clk);
if (rate < PRIMA2_CLOCK_FREQ || rate % PRIMA2_CLOCK_FREQ) {
- pr_err("Invalid clock rate");
+ pr_err("Invalid clock rate\n");
return -EINVAL;
}
@@ -229,7 +229,7 @@
ret = clocksource_register_hz(&sirfsoc_clocksource, PRIMA2_CLOCK_FREQ);
if (ret) {
- pr_err("Failed to register clocksource");
+ pr_err("Failed to register clocksource\n");
return ret;
}
@@ -237,7 +237,7 @@
ret = setup_irq(sirfsoc_timer_irq.irq, &sirfsoc_timer_irq);
if (ret) {
- pr_err("Failed to setup irq");
+ pr_err("Failed to setup irq\n");
return ret;
}
diff --git a/drivers/clocksource/timer-sp804.c b/drivers/clocksource/timer-sp804.c
index d078633..2d575a8c 100644
--- a/drivers/clocksource/timer-sp804.c
+++ b/drivers/clocksource/timer-sp804.c
@@ -299,13 +299,13 @@
base = of_iomap(np, 0);
if (!base) {
- pr_err("Failed to iomap");
+ pr_err("Failed to iomap\n");
return -ENXIO;
}
clk = of_clk_get(np, 0);
if (IS_ERR(clk)) {
- pr_err("Failed to get clock");
+ pr_err("Failed to get clock\n");
return PTR_ERR(clk);
}
diff --git a/drivers/clocksource/timer-sun5i.c b/drivers/clocksource/timer-sun5i.c
index a3e662b..2e9c830 100644
--- a/drivers/clocksource/timer-sun5i.c
+++ b/drivers/clocksource/timer-sun5i.c
@@ -332,19 +332,19 @@
timer_base = of_io_request_and_map(node, 0, of_node_full_name(node));
if (IS_ERR(timer_base)) {
- pr_err("Can't map registers");
+ pr_err("Can't map registers\n");
return PTR_ERR(timer_base);;
}
irq = irq_of_parse_and_map(node, 0);
if (irq <= 0) {
- pr_err("Can't parse IRQ");
+ pr_err("Can't parse IRQ\n");
return -EINVAL;
}
clk = of_clk_get(node, 0);
if (IS_ERR(clk)) {
- pr_err("Can't get timer clock");
+ pr_err("Can't get timer clock\n");
return PTR_ERR(clk);
}
diff --git a/drivers/clocksource/vf_pit_timer.c b/drivers/clocksource/vf_pit_timer.c
index 55d8d84..e0849e2 100644
--- a/drivers/clocksource/vf_pit_timer.c
+++ b/drivers/clocksource/vf_pit_timer.c
@@ -165,7 +165,7 @@
timer_base = of_iomap(np, 0);
if (!timer_base) {
- pr_err("Failed to iomap");
+ pr_err("Failed to iomap\n");
return -ENXIO;
}
diff --git a/drivers/cpufreq/Kconfig.arm b/drivers/cpufreq/Kconfig.arm
index 74fa5c5..74ed7e9 100644
--- a/drivers/cpufreq/Kconfig.arm
+++ b/drivers/cpufreq/Kconfig.arm
@@ -247,6 +247,12 @@
help
This adds the CPUFreq driver support for Tegra124 SOCs.
+config ARM_TEGRA186_CPUFREQ
+ tristate "Tegra186 CPUFreq support"
+ depends on ARCH_TEGRA && TEGRA_BPMP
+ help
+ This adds the CPUFreq driver support for Tegra186 SOCs.
+
config ARM_TI_CPUFREQ
bool "Texas Instruments CPUFreq support"
depends on ARCH_OMAP2PLUS
diff --git a/drivers/cpufreq/Makefile b/drivers/cpufreq/Makefile
index 9f5a804..b7e78f0 100644
--- a/drivers/cpufreq/Makefile
+++ b/drivers/cpufreq/Makefile
@@ -77,6 +77,7 @@
obj-$(CONFIG_ARM_STI_CPUFREQ) += sti-cpufreq.o
obj-$(CONFIG_ARM_TEGRA20_CPUFREQ) += tegra20-cpufreq.o
obj-$(CONFIG_ARM_TEGRA124_CPUFREQ) += tegra124-cpufreq.o
+obj-$(CONFIG_ARM_TEGRA186_CPUFREQ) += tegra186-cpufreq.o
obj-$(CONFIG_ARM_TI_CPUFREQ) += ti-cpufreq.o
obj-$(CONFIG_ARM_VEXPRESS_SPC_CPUFREQ) += vexpress-spc-cpufreq.o
obj-$(CONFIG_ACPI_CPPC_CPUFREQ) += cppc_cpufreq.o
diff --git a/drivers/cpufreq/dbx500-cpufreq.c b/drivers/cpufreq/dbx500-cpufreq.c
index 5c3ec1d..3575b82 100644
--- a/drivers/cpufreq/dbx500-cpufreq.c
+++ b/drivers/cpufreq/dbx500-cpufreq.c
@@ -11,6 +11,7 @@
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/cpufreq.h>
+#include <linux/cpu_cooling.h>
#include <linux/delay.h>
#include <linux/slab.h>
#include <linux/platform_device.h>
@@ -18,6 +19,7 @@
static struct cpufreq_frequency_table *freq_table;
static struct clk *armss_clk;
+static struct thermal_cooling_device *cdev;
static int dbx500_cpufreq_target(struct cpufreq_policy *policy,
unsigned int index)
@@ -32,6 +34,22 @@
return cpufreq_generic_init(policy, freq_table, 20 * 1000);
}
+static int dbx500_cpufreq_exit(struct cpufreq_policy *policy)
+{
+ if (!IS_ERR(cdev))
+ cpufreq_cooling_unregister(cdev);
+ return 0;
+}
+
+static void dbx500_cpufreq_ready(struct cpufreq_policy *policy)
+{
+ cdev = cpufreq_cooling_register(policy->cpus);
+ if (IS_ERR(cdev))
+ pr_err("Failed to register cooling device %ld\n", PTR_ERR(cdev));
+ else
+ pr_info("Cooling device registered: %s\n", cdev->type);
+}
+
static struct cpufreq_driver dbx500_cpufreq_driver = {
.flags = CPUFREQ_STICKY | CPUFREQ_CONST_LOOPS |
CPUFREQ_NEED_INITIAL_FREQ_CHECK,
@@ -39,6 +57,8 @@
.target_index = dbx500_cpufreq_target,
.get = cpufreq_generic_get,
.init = dbx500_cpufreq_init,
+ .exit = dbx500_cpufreq_exit,
+ .ready = dbx500_cpufreq_ready,
.name = "DBX500",
.attr = cpufreq_generic_attr,
};
diff --git a/drivers/cpufreq/ia64-acpi-cpufreq.c b/drivers/cpufreq/ia64-acpi-cpufreq.c
index e28a31a..a757c0a 100644
--- a/drivers/cpufreq/ia64-acpi-cpufreq.c
+++ b/drivers/cpufreq/ia64-acpi-cpufreq.c
@@ -34,6 +34,11 @@
unsigned int resume;
};
+struct cpufreq_acpi_req {
+ unsigned int cpu;
+ unsigned int state;
+};
+
static struct cpufreq_acpi_io *acpi_io_data[NR_CPUS];
static struct cpufreq_driver acpi_cpufreq_driver;
@@ -83,8 +88,7 @@
static unsigned
extract_clock (
struct cpufreq_acpi_io *data,
- unsigned value,
- unsigned int cpu)
+ unsigned value)
{
unsigned long i;
@@ -98,60 +102,43 @@
}
-static unsigned int
+static long
processor_get_freq (
- struct cpufreq_acpi_io *data,
- unsigned int cpu)
+ void *arg)
{
- int ret = 0;
- u32 value = 0;
- cpumask_t saved_mask;
- unsigned long clock_freq;
+ struct cpufreq_acpi_req *req = arg;
+ unsigned int cpu = req->cpu;
+ struct cpufreq_acpi_io *data = acpi_io_data[cpu];
+ u32 value;
+ int ret;
pr_debug("processor_get_freq\n");
-
- saved_mask = current->cpus_allowed;
- set_cpus_allowed_ptr(current, cpumask_of(cpu));
if (smp_processor_id() != cpu)
- goto migrate_end;
+ return -EAGAIN;
/* processor_get_pstate gets the instantaneous frequency */
ret = processor_get_pstate(&value);
-
if (ret) {
- set_cpus_allowed_ptr(current, &saved_mask);
pr_warn("get performance failed with error %d\n", ret);
- ret = 0;
- goto migrate_end;
+ return ret;
}
- clock_freq = extract_clock(data, value, cpu);
- ret = (clock_freq*1000);
-
-migrate_end:
- set_cpus_allowed_ptr(current, &saved_mask);
- return ret;
+ return 1000 * extract_clock(data, value);
}
-static int
+static long
processor_set_freq (
- struct cpufreq_acpi_io *data,
- struct cpufreq_policy *policy,
- int state)
+ void *arg)
{
- int ret = 0;
- u32 value = 0;
- cpumask_t saved_mask;
- int retval;
+ struct cpufreq_acpi_req *req = arg;
+ unsigned int cpu = req->cpu;
+ struct cpufreq_acpi_io *data = acpi_io_data[cpu];
+ int ret, state = req->state;
+ u32 value;
pr_debug("processor_set_freq\n");
-
- saved_mask = current->cpus_allowed;
- set_cpus_allowed_ptr(current, cpumask_of(policy->cpu));
- if (smp_processor_id() != policy->cpu) {
- retval = -EAGAIN;
- goto migrate_end;
- }
+ if (smp_processor_id() != cpu)
+ return -EAGAIN;
if (state == data->acpi_data.state) {
if (unlikely(data->resume)) {
@@ -159,8 +146,7 @@
data->resume = 0;
} else {
pr_debug("Already at target state (P%d)\n", state);
- retval = 0;
- goto migrate_end;
+ return 0;
}
}
@@ -171,7 +157,6 @@
* First we write the target state's 'control' value to the
* control_register.
*/
-
value = (u32) data->acpi_data.states[state].control;
pr_debug("Transitioning to state: 0x%08x\n", value);
@@ -179,17 +164,11 @@
ret = processor_set_pstate(value);
if (ret) {
pr_warn("Transition failed with error %d\n", ret);
- retval = -ENODEV;
- goto migrate_end;
+ return -ENODEV;
}
data->acpi_data.state = state;
-
- retval = 0;
-
-migrate_end:
- set_cpus_allowed_ptr(current, &saved_mask);
- return (retval);
+ return 0;
}
@@ -197,11 +176,13 @@
acpi_cpufreq_get (
unsigned int cpu)
{
- struct cpufreq_acpi_io *data = acpi_io_data[cpu];
+ struct cpufreq_acpi_req req;
+ long ret;
- pr_debug("acpi_cpufreq_get\n");
+ req.cpu = cpu;
+ ret = work_on_cpu(cpu, processor_get_freq, &req);
- return processor_get_freq(data, cpu);
+ return ret > 0 ? (unsigned int) ret : 0;
}
@@ -210,7 +191,12 @@
struct cpufreq_policy *policy,
unsigned int index)
{
- return processor_set_freq(acpi_io_data[policy->cpu], policy, index);
+ struct cpufreq_acpi_req req;
+
+ req.cpu = policy->cpu;
+ req.state = index;
+
+ return work_on_cpu(req.cpu, processor_set_freq, &req);
}
static int
diff --git a/drivers/cpufreq/imx6q-cpufreq.c b/drivers/cpufreq/imx6q-cpufreq.c
index 7719b02..9c13f09 100644
--- a/drivers/cpufreq/imx6q-cpufreq.c
+++ b/drivers/cpufreq/imx6q-cpufreq.c
@@ -161,8 +161,13 @@
static int imx6q_cpufreq_init(struct cpufreq_policy *policy)
{
+ int ret;
+
policy->clk = arm_clk;
- return cpufreq_generic_init(policy, freq_table, transition_latency);
+ ret = cpufreq_generic_init(policy, freq_table, transition_latency);
+ policy->suspend_freq = policy->max;
+
+ return ret;
}
static struct cpufreq_driver imx6q_cpufreq_driver = {
@@ -173,6 +178,7 @@
.init = imx6q_cpufreq_init,
.name = "imx6q-cpufreq",
.attr = cpufreq_generic_attr,
+ .suspend = cpufreq_generic_suspend,
};
static int imx6q_cpufreq_probe(struct platform_device *pdev)
@@ -222,6 +228,13 @@
arm_reg = regulator_get(cpu_dev, "arm");
pu_reg = regulator_get_optional(cpu_dev, "pu");
soc_reg = regulator_get(cpu_dev, "soc");
+ if (PTR_ERR(arm_reg) == -EPROBE_DEFER ||
+ PTR_ERR(soc_reg) == -EPROBE_DEFER ||
+ PTR_ERR(pu_reg) == -EPROBE_DEFER) {
+ ret = -EPROBE_DEFER;
+ dev_dbg(cpu_dev, "regulators not ready, defer\n");
+ goto put_reg;
+ }
if (IS_ERR(arm_reg) || IS_ERR(soc_reg)) {
dev_err(cpu_dev, "failed to get regulators\n");
ret = -ENOENT;
@@ -255,7 +268,7 @@
ret = dev_pm_opp_init_cpufreq_table(cpu_dev, &freq_table);
if (ret) {
dev_err(cpu_dev, "failed to init cpufreq table: %d\n", ret);
- goto put_reg;
+ goto out_free_opp;
}
/* Make imx6_soc_volt array's size same as arm opp number */
diff --git a/drivers/cpufreq/intel_pstate.c b/drivers/cpufreq/intel_pstate.c
index 283491f..b7de5bd 100644
--- a/drivers/cpufreq/intel_pstate.c
+++ b/drivers/cpufreq/intel_pstate.c
@@ -37,7 +37,11 @@
#include <asm/cpufeature.h>
#include <asm/intel-family.h>
+#define INTEL_PSTATE_DEFAULT_SAMPLING_INTERVAL (10 * NSEC_PER_MSEC)
+#define INTEL_PSTATE_HWP_SAMPLING_INTERVAL (50 * NSEC_PER_MSEC)
+
#define INTEL_CPUFREQ_TRANSITION_LATENCY 20000
+#define INTEL_CPUFREQ_TRANSITION_DELAY 500
#ifdef CONFIG_ACPI
#include <acpi/processor.h>
@@ -74,6 +78,11 @@
return ret;
}
+static inline int32_t percent_fp(int percent)
+{
+ return div_fp(percent, 100);
+}
+
static inline u64 mul_ext_fp(u64 x, u64 y)
{
return (x * y) >> EXT_FRAC_BITS;
@@ -186,45 +195,22 @@
};
/**
- * struct perf_limits - Store user and policy limits
- * @no_turbo: User requested turbo state from intel_pstate sysfs
- * @turbo_disabled: Platform turbo status either from msr
- * MSR_IA32_MISC_ENABLE or when maximum available pstate
- * matches the maximum turbo pstate
- * @max_perf_pct: Effective maximum performance limit in percentage, this
- * is minimum of either limits enforced by cpufreq policy
- * or limits from user set limits via intel_pstate sysfs
- * @min_perf_pct: Effective minimum performance limit in percentage, this
- * is maximum of either limits enforced by cpufreq policy
- * or limits from user set limits via intel_pstate sysfs
- * @max_perf: This is a scaled value between 0 to 255 for max_perf_pct
- * This value is used to limit max pstate
- * @min_perf: This is a scaled value between 0 to 255 for min_perf_pct
- * This value is used to limit min pstate
- * @max_policy_pct: The maximum performance in percentage enforced by
- * cpufreq setpolicy interface
- * @max_sysfs_pct: The maximum performance in percentage enforced by
- * intel pstate sysfs interface, unused when per cpu
- * controls are enforced
- * @min_policy_pct: The minimum performance in percentage enforced by
- * cpufreq setpolicy interface
- * @min_sysfs_pct: The minimum performance in percentage enforced by
- * intel pstate sysfs interface, unused when per cpu
- * controls are enforced
- *
- * Storage for user and policy defined limits.
+ * struct global_params - Global parameters, mostly tunable via sysfs.
+ * @no_turbo: Whether or not to use turbo P-states.
+ * @turbo_disabled: Whethet or not turbo P-states are available at all,
+ * based on the MSR_IA32_MISC_ENABLE value and whether or
+ * not the maximum reported turbo P-state is different from
+ * the maximum reported non-turbo one.
+ * @min_perf_pct: Minimum capacity limit in percent of the maximum turbo
+ * P-state capacity.
+ * @max_perf_pct: Maximum capacity limit in percent of the maximum turbo
+ * P-state capacity.
*/
-struct perf_limits {
- int no_turbo;
- int turbo_disabled;
+struct global_params {
+ bool no_turbo;
+ bool turbo_disabled;
int max_perf_pct;
int min_perf_pct;
- int32_t max_perf;
- int32_t min_perf;
- int max_policy_pct;
- int max_sysfs_pct;
- int min_policy_pct;
- int min_sysfs_pct;
};
/**
@@ -245,9 +231,10 @@
* @prev_cummulative_iowait: IO Wait time difference from last and
* current sample
* @sample: Storage for storing last Sample data
- * @perf_limits: Pointer to perf_limit unique to this CPU
- * Not all field in the structure are applicable
- * when per cpu controls are enforced
+ * @min_perf: Minimum capacity limit as a fraction of the maximum
+ * turbo P-state capacity.
+ * @max_perf: Maximum capacity limit as a fraction of the maximum
+ * turbo P-state capacity.
* @acpi_perf_data: Stores ACPI perf information read from _PSS
* @valid_pss_table: Set to true for valid ACPI _PSS entries found
* @epp_powersave: Last saved HWP energy performance preference
@@ -279,7 +266,8 @@
u64 prev_tsc;
u64 prev_cummulative_iowait;
struct sample sample;
- struct perf_limits *perf_limits;
+ int32_t min_perf;
+ int32_t max_perf;
#ifdef CONFIG_ACPI
struct acpi_processor_performance acpi_perf_data;
bool valid_pss_table;
@@ -324,7 +312,7 @@
* @get_scaling: Callback to get frequency scaling factor
* @get_val: Callback to convert P state to actual MSR write value
* @get_vid: Callback to get VID data for Atom platforms
- * @get_target_pstate: Callback to a function to calculate next P state to use
+ * @update_util: Active mode utilization update callback.
*
* Core and Atom CPU models have different way to get P State limits. This
* structure is used to store those callbacks.
@@ -337,43 +325,31 @@
int (*get_scaling)(void);
u64 (*get_val)(struct cpudata*, int pstate);
void (*get_vid)(struct cpudata *);
- int32_t (*get_target_pstate)(struct cpudata *);
+ void (*update_util)(struct update_util_data *data, u64 time,
+ unsigned int flags);
};
-/**
- * struct cpu_defaults- Per CPU model default config data
- * @pid_policy: PID config data
- * @funcs: Callback function data
- */
-struct cpu_defaults {
- struct pstate_adjust_policy pid_policy;
- struct pstate_funcs funcs;
-};
-
-static inline int32_t get_target_pstate_use_performance(struct cpudata *cpu);
-static inline int32_t get_target_pstate_use_cpu_load(struct cpudata *cpu);
-
-static struct pstate_adjust_policy pid_params __read_mostly;
static struct pstate_funcs pstate_funcs __read_mostly;
+static struct pstate_adjust_policy pid_params __read_mostly = {
+ .sample_rate_ms = 10,
+ .sample_rate_ns = 10 * NSEC_PER_MSEC,
+ .deadband = 0,
+ .setpoint = 97,
+ .p_gain_pct = 20,
+ .d_gain_pct = 0,
+ .i_gain_pct = 0,
+};
+
static int hwp_active __read_mostly;
static bool per_cpu_limits __read_mostly;
-static bool driver_registered __read_mostly;
+static struct cpufreq_driver *intel_pstate_driver __read_mostly;
#ifdef CONFIG_ACPI
static bool acpi_ppc;
#endif
-static struct perf_limits global;
-
-static void intel_pstate_init_limits(struct perf_limits *limits)
-{
- memset(limits, 0, sizeof(*limits));
- limits->max_perf_pct = 100;
- limits->max_perf = int_ext_tofp(1);
- limits->max_policy_pct = 100;
- limits->max_sysfs_pct = 100;
-}
+static struct global_params global;
static DEFINE_MUTEX(intel_pstate_driver_lock);
static DEFINE_MUTEX(intel_pstate_limits_lock);
@@ -530,29 +506,6 @@
}
#endif
-static inline void pid_reset(struct _pid *pid, int setpoint, int busy,
- int deadband, int integral) {
- pid->setpoint = int_tofp(setpoint);
- pid->deadband = int_tofp(deadband);
- pid->integral = int_tofp(integral);
- pid->last_err = int_tofp(setpoint) - int_tofp(busy);
-}
-
-static inline void pid_p_gain_set(struct _pid *pid, int percent)
-{
- pid->p_gain = div_fp(percent, 100);
-}
-
-static inline void pid_i_gain_set(struct _pid *pid, int percent)
-{
- pid->i_gain = div_fp(percent, 100);
-}
-
-static inline void pid_d_gain_set(struct _pid *pid, int percent)
-{
- pid->d_gain = div_fp(percent, 100);
-}
-
static signed int pid_calc(struct _pid *pid, int32_t busy)
{
signed int result;
@@ -590,23 +543,17 @@
return (signed int)fp_toint(result);
}
-static inline void intel_pstate_busy_pid_reset(struct cpudata *cpu)
+static inline void intel_pstate_pid_reset(struct cpudata *cpu)
{
- pid_p_gain_set(&cpu->pid, pid_params.p_gain_pct);
- pid_d_gain_set(&cpu->pid, pid_params.d_gain_pct);
- pid_i_gain_set(&cpu->pid, pid_params.i_gain_pct);
+ struct _pid *pid = &cpu->pid;
- pid_reset(&cpu->pid, pid_params.setpoint, 100, pid_params.deadband, 0);
-}
-
-static inline void intel_pstate_reset_all_pid(void)
-{
- unsigned int cpu;
-
- for_each_online_cpu(cpu) {
- if (all_cpu_data[cpu])
- intel_pstate_busy_pid_reset(all_cpu_data[cpu]);
- }
+ pid->p_gain = percent_fp(pid_params.p_gain_pct);
+ pid->d_gain = percent_fp(pid_params.d_gain_pct);
+ pid->i_gain = percent_fp(pid_params.i_gain_pct);
+ pid->setpoint = int_tofp(pid_params.setpoint);
+ pid->last_err = pid->setpoint - int_tofp(100);
+ pid->deadband = int_tofp(pid_params.deadband);
+ pid->integral = 0;
}
static inline void update_turbo_state(void)
@@ -621,6 +568,14 @@
cpu->pstate.max_pstate == cpu->pstate.turbo_pstate);
}
+static int min_perf_pct_min(void)
+{
+ struct cpudata *cpu = all_cpu_data[0];
+
+ return DIV_ROUND_UP(cpu->pstate.min_pstate * 100,
+ cpu->pstate.turbo_pstate);
+}
+
static s16 intel_pstate_get_epb(struct cpudata *cpu_data)
{
u64 epb;
@@ -838,96 +793,80 @@
NULL,
};
-static void intel_pstate_hwp_set(struct cpufreq_policy *policy)
+static void intel_pstate_hwp_set(unsigned int cpu)
{
- int min, hw_min, max, hw_max, cpu;
- struct perf_limits *perf_limits = &global;
+ struct cpudata *cpu_data = all_cpu_data[cpu];
+ int min, hw_min, max, hw_max;
u64 value, cap;
+ s16 epp;
- for_each_cpu(cpu, policy->cpus) {
- struct cpudata *cpu_data = all_cpu_data[cpu];
- s16 epp;
+ rdmsrl_on_cpu(cpu, MSR_HWP_CAPABILITIES, &cap);
+ hw_min = HWP_LOWEST_PERF(cap);
+ if (global.no_turbo)
+ hw_max = HWP_GUARANTEED_PERF(cap);
+ else
+ hw_max = HWP_HIGHEST_PERF(cap);
- if (per_cpu_limits)
- perf_limits = all_cpu_data[cpu]->perf_limits;
+ max = fp_ext_toint(hw_max * cpu_data->max_perf);
+ if (cpu_data->policy == CPUFREQ_POLICY_PERFORMANCE)
+ min = max;
+ else
+ min = fp_ext_toint(hw_max * cpu_data->min_perf);
- rdmsrl_on_cpu(cpu, MSR_HWP_CAPABILITIES, &cap);
- hw_min = HWP_LOWEST_PERF(cap);
- if (global.no_turbo)
- hw_max = HWP_GUARANTEED_PERF(cap);
- else
- hw_max = HWP_HIGHEST_PERF(cap);
+ rdmsrl_on_cpu(cpu, MSR_HWP_REQUEST, &value);
- max = fp_ext_toint(hw_max * perf_limits->max_perf);
- if (cpu_data->policy == CPUFREQ_POLICY_PERFORMANCE)
- min = max;
- else
- min = fp_ext_toint(hw_max * perf_limits->min_perf);
+ value &= ~HWP_MIN_PERF(~0L);
+ value |= HWP_MIN_PERF(min);
- rdmsrl_on_cpu(cpu, MSR_HWP_REQUEST, &value);
+ value &= ~HWP_MAX_PERF(~0L);
+ value |= HWP_MAX_PERF(max);
- value &= ~HWP_MIN_PERF(~0L);
- value |= HWP_MIN_PERF(min);
+ if (cpu_data->epp_policy == cpu_data->policy)
+ goto skip_epp;
- value &= ~HWP_MAX_PERF(~0L);
- value |= HWP_MAX_PERF(max);
+ cpu_data->epp_policy = cpu_data->policy;
- if (cpu_data->epp_policy == cpu_data->policy)
+ if (cpu_data->epp_saved >= 0) {
+ epp = cpu_data->epp_saved;
+ cpu_data->epp_saved = -EINVAL;
+ goto update_epp;
+ }
+
+ if (cpu_data->policy == CPUFREQ_POLICY_PERFORMANCE) {
+ epp = intel_pstate_get_epp(cpu_data, value);
+ cpu_data->epp_powersave = epp;
+ /* If EPP read was failed, then don't try to write */
+ if (epp < 0)
goto skip_epp;
- cpu_data->epp_policy = cpu_data->policy;
+ epp = 0;
+ } else {
+ /* skip setting EPP, when saved value is invalid */
+ if (cpu_data->epp_powersave < 0)
+ goto skip_epp;
- if (cpu_data->epp_saved >= 0) {
- epp = cpu_data->epp_saved;
- cpu_data->epp_saved = -EINVAL;
- goto update_epp;
- }
+ /*
+ * No need to restore EPP when it is not zero. This
+ * means:
+ * - Policy is not changed
+ * - user has manually changed
+ * - Error reading EPB
+ */
+ epp = intel_pstate_get_epp(cpu_data, value);
+ if (epp)
+ goto skip_epp;
- if (cpu_data->policy == CPUFREQ_POLICY_PERFORMANCE) {
- epp = intel_pstate_get_epp(cpu_data, value);
- cpu_data->epp_powersave = epp;
- /* If EPP read was failed, then don't try to write */
- if (epp < 0)
- goto skip_epp;
-
-
- epp = 0;
- } else {
- /* skip setting EPP, when saved value is invalid */
- if (cpu_data->epp_powersave < 0)
- goto skip_epp;
-
- /*
- * No need to restore EPP when it is not zero. This
- * means:
- * - Policy is not changed
- * - user has manually changed
- * - Error reading EPB
- */
- epp = intel_pstate_get_epp(cpu_data, value);
- if (epp)
- goto skip_epp;
-
- epp = cpu_data->epp_powersave;
- }
-update_epp:
- if (static_cpu_has(X86_FEATURE_HWP_EPP)) {
- value &= ~GENMASK_ULL(31, 24);
- value |= (u64)epp << 24;
- } else {
- intel_pstate_set_epb(cpu, epp);
- }
-skip_epp:
- wrmsrl_on_cpu(cpu, MSR_HWP_REQUEST, value);
+ epp = cpu_data->epp_powersave;
}
-}
-
-static int intel_pstate_hwp_set_policy(struct cpufreq_policy *policy)
-{
- if (hwp_active)
- intel_pstate_hwp_set(policy);
-
- return 0;
+update_epp:
+ if (static_cpu_has(X86_FEATURE_HWP_EPP)) {
+ value &= ~GENMASK_ULL(31, 24);
+ value |= (u64)epp << 24;
+ } else {
+ intel_pstate_set_epb(cpu, epp);
+ }
+skip_epp:
+ wrmsrl_on_cpu(cpu, MSR_HWP_REQUEST, value);
}
static int intel_pstate_hwp_save_state(struct cpufreq_policy *policy)
@@ -944,20 +883,17 @@
static int intel_pstate_resume(struct cpufreq_policy *policy)
{
- int ret;
-
if (!hwp_active)
return 0;
mutex_lock(&intel_pstate_limits_lock);
all_cpu_data[policy->cpu]->epp_policy = 0;
-
- ret = intel_pstate_hwp_set_policy(policy);
+ intel_pstate_hwp_set(policy->cpu);
mutex_unlock(&intel_pstate_limits_lock);
- return ret;
+ return 0;
}
static void intel_pstate_update_policies(void)
@@ -971,9 +907,14 @@
/************************** debugfs begin ************************/
static int pid_param_set(void *data, u64 val)
{
+ unsigned int cpu;
+
*(u32 *)data = val;
pid_params.sample_rate_ns = pid_params.sample_rate_ms * NSEC_PER_MSEC;
- intel_pstate_reset_all_pid();
+ for_each_possible_cpu(cpu)
+ if (all_cpu_data[cpu])
+ intel_pstate_pid_reset(all_cpu_data[cpu]);
+
return 0;
}
@@ -1084,7 +1025,7 @@
mutex_lock(&intel_pstate_driver_lock);
- if (!driver_registered) {
+ if (!intel_pstate_driver) {
mutex_unlock(&intel_pstate_driver_lock);
return -EAGAIN;
}
@@ -1109,7 +1050,7 @@
mutex_lock(&intel_pstate_driver_lock);
- if (!driver_registered) {
+ if (!intel_pstate_driver) {
mutex_unlock(&intel_pstate_driver_lock);
return -EAGAIN;
}
@@ -1129,7 +1070,7 @@
mutex_lock(&intel_pstate_driver_lock);
- if (!driver_registered) {
+ if (!intel_pstate_driver) {
mutex_unlock(&intel_pstate_driver_lock);
return -EAGAIN;
}
@@ -1157,7 +1098,7 @@
mutex_lock(&intel_pstate_driver_lock);
- if (!driver_registered) {
+ if (!intel_pstate_driver) {
mutex_unlock(&intel_pstate_driver_lock);
return -EAGAIN;
}
@@ -1174,6 +1115,15 @@
global.no_turbo = clamp_t(int, input, 0, 1);
+ if (global.no_turbo) {
+ struct cpudata *cpu = all_cpu_data[0];
+ int pct = cpu->pstate.max_pstate * 100 / cpu->pstate.turbo_pstate;
+
+ /* Squash the global minimum into the permitted range. */
+ if (global.min_perf_pct > pct)
+ global.min_perf_pct = pct;
+ }
+
mutex_unlock(&intel_pstate_limits_lock);
intel_pstate_update_policies();
@@ -1195,18 +1145,14 @@
mutex_lock(&intel_pstate_driver_lock);
- if (!driver_registered) {
+ if (!intel_pstate_driver) {
mutex_unlock(&intel_pstate_driver_lock);
return -EAGAIN;
}
mutex_lock(&intel_pstate_limits_lock);
- global.max_sysfs_pct = clamp_t(int, input, 0 , 100);
- global.max_perf_pct = min(global.max_policy_pct, global.max_sysfs_pct);
- global.max_perf_pct = max(global.min_policy_pct, global.max_perf_pct);
- global.max_perf_pct = max(global.min_perf_pct, global.max_perf_pct);
- global.max_perf = percent_ext_fp(global.max_perf_pct);
+ global.max_perf_pct = clamp_t(int, input, global.min_perf_pct, 100);
mutex_unlock(&intel_pstate_limits_lock);
@@ -1229,18 +1175,15 @@
mutex_lock(&intel_pstate_driver_lock);
- if (!driver_registered) {
+ if (!intel_pstate_driver) {
mutex_unlock(&intel_pstate_driver_lock);
return -EAGAIN;
}
mutex_lock(&intel_pstate_limits_lock);
- global.min_sysfs_pct = clamp_t(int, input, 0 , 100);
- global.min_perf_pct = max(global.min_policy_pct, global.min_sysfs_pct);
- global.min_perf_pct = min(global.max_policy_pct, global.min_perf_pct);
- global.min_perf_pct = min(global.max_perf_pct, global.min_perf_pct);
- global.min_perf = percent_ext_fp(global.min_perf_pct);
+ global.min_perf_pct = clamp_t(int, input,
+ min_perf_pct_min(), global.max_perf_pct);
mutex_unlock(&intel_pstate_limits_lock);
@@ -1554,132 +1497,10 @@
return ret;
}
-static struct cpu_defaults core_params = {
- .pid_policy = {
- .sample_rate_ms = 10,
- .deadband = 0,
- .setpoint = 97,
- .p_gain_pct = 20,
- .d_gain_pct = 0,
- .i_gain_pct = 0,
- },
- .funcs = {
- .get_max = core_get_max_pstate,
- .get_max_physical = core_get_max_pstate_physical,
- .get_min = core_get_min_pstate,
- .get_turbo = core_get_turbo_pstate,
- .get_scaling = core_get_scaling,
- .get_val = core_get_val,
- .get_target_pstate = get_target_pstate_use_performance,
- },
-};
-
-static const struct cpu_defaults silvermont_params = {
- .pid_policy = {
- .sample_rate_ms = 10,
- .deadband = 0,
- .setpoint = 60,
- .p_gain_pct = 14,
- .d_gain_pct = 0,
- .i_gain_pct = 4,
- },
- .funcs = {
- .get_max = atom_get_max_pstate,
- .get_max_physical = atom_get_max_pstate,
- .get_min = atom_get_min_pstate,
- .get_turbo = atom_get_turbo_pstate,
- .get_val = atom_get_val,
- .get_scaling = silvermont_get_scaling,
- .get_vid = atom_get_vid,
- .get_target_pstate = get_target_pstate_use_cpu_load,
- },
-};
-
-static const struct cpu_defaults airmont_params = {
- .pid_policy = {
- .sample_rate_ms = 10,
- .deadband = 0,
- .setpoint = 60,
- .p_gain_pct = 14,
- .d_gain_pct = 0,
- .i_gain_pct = 4,
- },
- .funcs = {
- .get_max = atom_get_max_pstate,
- .get_max_physical = atom_get_max_pstate,
- .get_min = atom_get_min_pstate,
- .get_turbo = atom_get_turbo_pstate,
- .get_val = atom_get_val,
- .get_scaling = airmont_get_scaling,
- .get_vid = atom_get_vid,
- .get_target_pstate = get_target_pstate_use_cpu_load,
- },
-};
-
-static const struct cpu_defaults knl_params = {
- .pid_policy = {
- .sample_rate_ms = 10,
- .deadband = 0,
- .setpoint = 97,
- .p_gain_pct = 20,
- .d_gain_pct = 0,
- .i_gain_pct = 0,
- },
- .funcs = {
- .get_max = core_get_max_pstate,
- .get_max_physical = core_get_max_pstate_physical,
- .get_min = core_get_min_pstate,
- .get_turbo = knl_get_turbo_pstate,
- .get_scaling = core_get_scaling,
- .get_val = core_get_val,
- .get_target_pstate = get_target_pstate_use_performance,
- },
-};
-
-static const struct cpu_defaults bxt_params = {
- .pid_policy = {
- .sample_rate_ms = 10,
- .deadband = 0,
- .setpoint = 60,
- .p_gain_pct = 14,
- .d_gain_pct = 0,
- .i_gain_pct = 4,
- },
- .funcs = {
- .get_max = core_get_max_pstate,
- .get_max_physical = core_get_max_pstate_physical,
- .get_min = core_get_min_pstate,
- .get_turbo = core_get_turbo_pstate,
- .get_scaling = core_get_scaling,
- .get_val = core_get_val,
- .get_target_pstate = get_target_pstate_use_cpu_load,
- },
-};
-
-static void intel_pstate_get_min_max(struct cpudata *cpu, int *min, int *max)
+static int intel_pstate_get_base_pstate(struct cpudata *cpu)
{
- int max_perf = cpu->pstate.turbo_pstate;
- int max_perf_adj;
- int min_perf;
- struct perf_limits *perf_limits = &global;
-
- if (global.no_turbo || global.turbo_disabled)
- max_perf = cpu->pstate.max_pstate;
-
- if (per_cpu_limits)
- perf_limits = cpu->perf_limits;
-
- /*
- * performance can be limited by user through sysfs, by cpufreq
- * policy, or by cpu specific default values determined through
- * experimentation.
- */
- max_perf_adj = fp_ext_toint(max_perf * perf_limits->max_perf);
- *max = clamp_t(int, max_perf_adj,
- cpu->pstate.min_pstate, cpu->pstate.turbo_pstate);
-
- min_perf = fp_ext_toint(max_perf * perf_limits->min_perf);
- *min = clamp_t(int, min_perf, cpu->pstate.min_pstate, max_perf);
+ return global.no_turbo || global.turbo_disabled ?
+ cpu->pstate.max_pstate : cpu->pstate.turbo_pstate;
}
static void intel_pstate_set_pstate(struct cpudata *cpu, int pstate)
@@ -1702,11 +1523,13 @@
static void intel_pstate_max_within_limits(struct cpudata *cpu)
{
- int min_pstate, max_pstate;
+ int pstate;
update_turbo_state();
- intel_pstate_get_min_max(cpu, &min_pstate, &max_pstate);
- intel_pstate_set_pstate(cpu, max_pstate);
+ pstate = intel_pstate_get_base_pstate(cpu);
+ pstate = max(cpu->pstate.min_pstate,
+ fp_ext_toint(pstate * cpu->max_perf));
+ intel_pstate_set_pstate(cpu, pstate);
}
static void intel_pstate_get_cpu_pstates(struct cpudata *cpu)
@@ -1767,7 +1590,11 @@
* that sample.time will always be reset before setting the utilization
* update hook and make the caller skip the sample then.
*/
- return !!cpu->last_sample_time;
+ if (cpu->last_sample_time) {
+ intel_pstate_calc_avg_perf(cpu);
+ return true;
+ }
+ return false;
}
static inline int32_t get_avg_frequency(struct cpudata *cpu)
@@ -1788,6 +1615,9 @@
int32_t busy_frac, boost;
int target, avg_pstate;
+ if (cpu->policy == CPUFREQ_POLICY_PERFORMANCE)
+ return cpu->pstate.turbo_pstate;
+
busy_frac = div_fp(sample->mperf, sample->tsc);
boost = cpu->iowait_boost;
@@ -1824,6 +1654,9 @@
int32_t perf_scaled, max_pstate, current_pstate, sample_ratio;
u64 duration_ns;
+ if (cpu->policy == CPUFREQ_POLICY_PERFORMANCE)
+ return cpu->pstate.turbo_pstate;
+
/*
* perf_scaled is the ratio of the average P-state during the last
* sampling period to the P-state requested last time (in percent).
@@ -1858,11 +1691,13 @@
static int intel_pstate_prepare_request(struct cpudata *cpu, int pstate)
{
- int max_perf, min_perf;
+ int max_pstate = intel_pstate_get_base_pstate(cpu);
+ int min_pstate;
- intel_pstate_get_min_max(cpu, &min_perf, &max_perf);
- pstate = clamp_t(int, pstate, min_perf, max_perf);
- return pstate;
+ min_pstate = max(cpu->pstate.min_pstate,
+ fp_ext_toint(max_pstate * cpu->min_perf));
+ max_pstate = max(min_pstate, fp_ext_toint(max_pstate * cpu->max_perf));
+ return clamp_t(int, pstate, min_pstate, max_pstate);
}
static void intel_pstate_update_pstate(struct cpudata *cpu, int pstate)
@@ -1874,16 +1709,11 @@
wrmsrl(MSR_IA32_PERF_CTL, pstate_funcs.get_val(cpu, pstate));
}
-static inline void intel_pstate_adjust_busy_pstate(struct cpudata *cpu)
+static void intel_pstate_adjust_pstate(struct cpudata *cpu, int target_pstate)
{
- int from, target_pstate;
+ int from = cpu->pstate.current_pstate;
struct sample *sample;
- from = cpu->pstate.current_pstate;
-
- target_pstate = cpu->policy == CPUFREQ_POLICY_PERFORMANCE ?
- cpu->pstate.turbo_pstate : pstate_funcs.get_target_pstate(cpu);
-
update_turbo_state();
target_pstate = intel_pstate_prepare_request(cpu, target_pstate);
@@ -1902,76 +1732,155 @@
fp_toint(cpu->iowait_boost * 100));
}
+static void intel_pstate_update_util_hwp(struct update_util_data *data,
+ u64 time, unsigned int flags)
+{
+ struct cpudata *cpu = container_of(data, struct cpudata, update_util);
+ u64 delta_ns = time - cpu->sample.time;
+
+ if ((s64)delta_ns >= INTEL_PSTATE_HWP_SAMPLING_INTERVAL)
+ intel_pstate_sample(cpu, time);
+}
+
+static void intel_pstate_update_util_pid(struct update_util_data *data,
+ u64 time, unsigned int flags)
+{
+ struct cpudata *cpu = container_of(data, struct cpudata, update_util);
+ u64 delta_ns = time - cpu->sample.time;
+
+ if ((s64)delta_ns < pid_params.sample_rate_ns)
+ return;
+
+ if (intel_pstate_sample(cpu, time)) {
+ int target_pstate;
+
+ target_pstate = get_target_pstate_use_performance(cpu);
+ intel_pstate_adjust_pstate(cpu, target_pstate);
+ }
+}
+
static void intel_pstate_update_util(struct update_util_data *data, u64 time,
unsigned int flags)
{
struct cpudata *cpu = container_of(data, struct cpudata, update_util);
u64 delta_ns;
- if (pstate_funcs.get_target_pstate == get_target_pstate_use_cpu_load) {
- if (flags & SCHED_CPUFREQ_IOWAIT) {
- cpu->iowait_boost = int_tofp(1);
- } else if (cpu->iowait_boost) {
- /* Clear iowait_boost if the CPU may have been idle. */
- delta_ns = time - cpu->last_update;
- if (delta_ns > TICK_NSEC)
- cpu->iowait_boost = 0;
- }
- cpu->last_update = time;
+ if (flags & SCHED_CPUFREQ_IOWAIT) {
+ cpu->iowait_boost = int_tofp(1);
+ } else if (cpu->iowait_boost) {
+ /* Clear iowait_boost if the CPU may have been idle. */
+ delta_ns = time - cpu->last_update;
+ if (delta_ns > TICK_NSEC)
+ cpu->iowait_boost = 0;
}
-
+ cpu->last_update = time;
delta_ns = time - cpu->sample.time;
- if ((s64)delta_ns >= pid_params.sample_rate_ns) {
- bool sample_taken = intel_pstate_sample(cpu, time);
+ if ((s64)delta_ns < INTEL_PSTATE_DEFAULT_SAMPLING_INTERVAL)
+ return;
- if (sample_taken) {
- intel_pstate_calc_avg_perf(cpu);
- if (!hwp_active)
- intel_pstate_adjust_busy_pstate(cpu);
- }
+ if (intel_pstate_sample(cpu, time)) {
+ int target_pstate;
+
+ target_pstate = get_target_pstate_use_cpu_load(cpu);
+ intel_pstate_adjust_pstate(cpu, target_pstate);
}
}
+static struct pstate_funcs core_funcs = {
+ .get_max = core_get_max_pstate,
+ .get_max_physical = core_get_max_pstate_physical,
+ .get_min = core_get_min_pstate,
+ .get_turbo = core_get_turbo_pstate,
+ .get_scaling = core_get_scaling,
+ .get_val = core_get_val,
+ .update_util = intel_pstate_update_util_pid,
+};
+
+static const struct pstate_funcs silvermont_funcs = {
+ .get_max = atom_get_max_pstate,
+ .get_max_physical = atom_get_max_pstate,
+ .get_min = atom_get_min_pstate,
+ .get_turbo = atom_get_turbo_pstate,
+ .get_val = atom_get_val,
+ .get_scaling = silvermont_get_scaling,
+ .get_vid = atom_get_vid,
+ .update_util = intel_pstate_update_util,
+};
+
+static const struct pstate_funcs airmont_funcs = {
+ .get_max = atom_get_max_pstate,
+ .get_max_physical = atom_get_max_pstate,
+ .get_min = atom_get_min_pstate,
+ .get_turbo = atom_get_turbo_pstate,
+ .get_val = atom_get_val,
+ .get_scaling = airmont_get_scaling,
+ .get_vid = atom_get_vid,
+ .update_util = intel_pstate_update_util,
+};
+
+static const struct pstate_funcs knl_funcs = {
+ .get_max = core_get_max_pstate,
+ .get_max_physical = core_get_max_pstate_physical,
+ .get_min = core_get_min_pstate,
+ .get_turbo = knl_get_turbo_pstate,
+ .get_scaling = core_get_scaling,
+ .get_val = core_get_val,
+ .update_util = intel_pstate_update_util_pid,
+};
+
+static const struct pstate_funcs bxt_funcs = {
+ .get_max = core_get_max_pstate,
+ .get_max_physical = core_get_max_pstate_physical,
+ .get_min = core_get_min_pstate,
+ .get_turbo = core_get_turbo_pstate,
+ .get_scaling = core_get_scaling,
+ .get_val = core_get_val,
+ .update_util = intel_pstate_update_util,
+};
+
#define ICPU(model, policy) \
{ X86_VENDOR_INTEL, 6, model, X86_FEATURE_APERFMPERF,\
(unsigned long)&policy }
static const struct x86_cpu_id intel_pstate_cpu_ids[] = {
- ICPU(INTEL_FAM6_SANDYBRIDGE, core_params),
- ICPU(INTEL_FAM6_SANDYBRIDGE_X, core_params),
- ICPU(INTEL_FAM6_ATOM_SILVERMONT1, silvermont_params),
- ICPU(INTEL_FAM6_IVYBRIDGE, core_params),
- ICPU(INTEL_FAM6_HASWELL_CORE, core_params),
- ICPU(INTEL_FAM6_BROADWELL_CORE, core_params),
- ICPU(INTEL_FAM6_IVYBRIDGE_X, core_params),
- ICPU(INTEL_FAM6_HASWELL_X, core_params),
- ICPU(INTEL_FAM6_HASWELL_ULT, core_params),
- ICPU(INTEL_FAM6_HASWELL_GT3E, core_params),
- ICPU(INTEL_FAM6_BROADWELL_GT3E, core_params),
- ICPU(INTEL_FAM6_ATOM_AIRMONT, airmont_params),
- ICPU(INTEL_FAM6_SKYLAKE_MOBILE, core_params),
- ICPU(INTEL_FAM6_BROADWELL_X, core_params),
- ICPU(INTEL_FAM6_SKYLAKE_DESKTOP, core_params),
- ICPU(INTEL_FAM6_BROADWELL_XEON_D, core_params),
- ICPU(INTEL_FAM6_XEON_PHI_KNL, knl_params),
- ICPU(INTEL_FAM6_XEON_PHI_KNM, knl_params),
- ICPU(INTEL_FAM6_ATOM_GOLDMONT, bxt_params),
+ ICPU(INTEL_FAM6_SANDYBRIDGE, core_funcs),
+ ICPU(INTEL_FAM6_SANDYBRIDGE_X, core_funcs),
+ ICPU(INTEL_FAM6_ATOM_SILVERMONT1, silvermont_funcs),
+ ICPU(INTEL_FAM6_IVYBRIDGE, core_funcs),
+ ICPU(INTEL_FAM6_HASWELL_CORE, core_funcs),
+ ICPU(INTEL_FAM6_BROADWELL_CORE, core_funcs),
+ ICPU(INTEL_FAM6_IVYBRIDGE_X, core_funcs),
+ ICPU(INTEL_FAM6_HASWELL_X, core_funcs),
+ ICPU(INTEL_FAM6_HASWELL_ULT, core_funcs),
+ ICPU(INTEL_FAM6_HASWELL_GT3E, core_funcs),
+ ICPU(INTEL_FAM6_BROADWELL_GT3E, core_funcs),
+ ICPU(INTEL_FAM6_ATOM_AIRMONT, airmont_funcs),
+ ICPU(INTEL_FAM6_SKYLAKE_MOBILE, core_funcs),
+ ICPU(INTEL_FAM6_BROADWELL_X, core_funcs),
+ ICPU(INTEL_FAM6_SKYLAKE_DESKTOP, core_funcs),
+ ICPU(INTEL_FAM6_BROADWELL_XEON_D, core_funcs),
+ ICPU(INTEL_FAM6_XEON_PHI_KNL, knl_funcs),
+ ICPU(INTEL_FAM6_XEON_PHI_KNM, knl_funcs),
+ ICPU(INTEL_FAM6_ATOM_GOLDMONT, bxt_funcs),
+ ICPU(INTEL_FAM6_ATOM_GEMINI_LAKE, bxt_funcs),
{}
};
MODULE_DEVICE_TABLE(x86cpu, intel_pstate_cpu_ids);
static const struct x86_cpu_id intel_pstate_cpu_oob_ids[] __initconst = {
- ICPU(INTEL_FAM6_BROADWELL_XEON_D, core_params),
- ICPU(INTEL_FAM6_BROADWELL_X, core_params),
- ICPU(INTEL_FAM6_SKYLAKE_X, core_params),
+ ICPU(INTEL_FAM6_BROADWELL_XEON_D, core_funcs),
+ ICPU(INTEL_FAM6_BROADWELL_X, core_funcs),
+ ICPU(INTEL_FAM6_SKYLAKE_X, core_funcs),
{}
};
static const struct x86_cpu_id intel_pstate_cpu_ee_disable_ids[] = {
- ICPU(INTEL_FAM6_KABYLAKE_DESKTOP, core_params),
+ ICPU(INTEL_FAM6_KABYLAKE_DESKTOP, core_funcs),
{}
};
+static bool pid_in_use(void);
+
static int intel_pstate_init_cpu(unsigned int cpunum)
{
struct cpudata *cpu;
@@ -1979,18 +1888,11 @@
cpu = all_cpu_data[cpunum];
if (!cpu) {
- unsigned int size = sizeof(struct cpudata);
-
- if (per_cpu_limits)
- size += sizeof(struct perf_limits);
-
- cpu = kzalloc(size, GFP_KERNEL);
+ cpu = kzalloc(sizeof(*cpu), GFP_KERNEL);
if (!cpu)
return -ENOMEM;
all_cpu_data[cpunum] = cpu;
- if (per_cpu_limits)
- cpu->perf_limits = (struct perf_limits *)(cpu + 1);
cpu->epp_default = -EINVAL;
cpu->epp_powersave = -EINVAL;
@@ -2009,14 +1911,12 @@
intel_pstate_disable_ee(cpunum);
intel_pstate_hwp_enable(cpu);
- pid_params.sample_rate_ms = 50;
- pid_params.sample_rate_ns = 50 * NSEC_PER_MSEC;
+ } else if (pid_in_use()) {
+ intel_pstate_pid_reset(cpu);
}
intel_pstate_get_cpu_pstates(cpu);
- intel_pstate_busy_pid_reset(cpu);
-
pr_debug("controlling: cpu %d\n", cpunum);
return 0;
@@ -2039,7 +1939,7 @@
/* Prevent intel_pstate_update_util() from using stale data. */
cpu->sample.time = 0;
cpufreq_add_update_util_hook(cpu_num, &cpu->update_util,
- intel_pstate_update_util);
+ pstate_funcs.update_util);
cpu->update_util_set = true;
}
@@ -2055,46 +1955,68 @@
synchronize_sched();
}
-static void intel_pstate_update_perf_limits(struct cpufreq_policy *policy,
- struct perf_limits *limits)
+static int intel_pstate_get_max_freq(struct cpudata *cpu)
{
+ return global.turbo_disabled || global.no_turbo ?
+ cpu->pstate.max_freq : cpu->pstate.turbo_freq;
+}
+
+static void intel_pstate_update_perf_limits(struct cpufreq_policy *policy,
+ struct cpudata *cpu)
+{
+ int max_freq = intel_pstate_get_max_freq(cpu);
int32_t max_policy_perf, min_policy_perf;
- max_policy_perf = div_ext_fp(policy->max, policy->cpuinfo.max_freq);
+ max_policy_perf = div_ext_fp(policy->max, max_freq);
max_policy_perf = clamp_t(int32_t, max_policy_perf, 0, int_ext_tofp(1));
if (policy->max == policy->min) {
min_policy_perf = max_policy_perf;
} else {
- min_policy_perf = div_ext_fp(policy->min,
- policy->cpuinfo.max_freq);
+ min_policy_perf = div_ext_fp(policy->min, max_freq);
min_policy_perf = clamp_t(int32_t, min_policy_perf,
0, max_policy_perf);
}
/* Normalize user input to [min_perf, max_perf] */
- limits->min_perf = max(min_policy_perf,
- percent_ext_fp(limits->min_sysfs_pct));
- limits->min_perf = min(limits->min_perf, max_policy_perf);
- limits->max_perf = min(max_policy_perf,
- percent_ext_fp(limits->max_sysfs_pct));
- limits->max_perf = max(min_policy_perf, limits->max_perf);
+ if (per_cpu_limits) {
+ cpu->min_perf = min_policy_perf;
+ cpu->max_perf = max_policy_perf;
+ } else {
+ int32_t global_min, global_max;
- /* Make sure min_perf <= max_perf */
- limits->min_perf = min(limits->min_perf, limits->max_perf);
+ /* Global limits are in percent of the maximum turbo P-state. */
+ global_max = percent_ext_fp(global.max_perf_pct);
+ global_min = percent_ext_fp(global.min_perf_pct);
+ if (max_freq != cpu->pstate.turbo_freq) {
+ int32_t turbo_factor;
- limits->max_perf = round_up(limits->max_perf, EXT_FRAC_BITS);
- limits->min_perf = round_up(limits->min_perf, EXT_FRAC_BITS);
- limits->max_perf_pct = fp_ext_toint(limits->max_perf * 100);
- limits->min_perf_pct = fp_ext_toint(limits->min_perf * 100);
+ turbo_factor = div_ext_fp(cpu->pstate.turbo_pstate,
+ cpu->pstate.max_pstate);
+ global_min = mul_ext_fp(global_min, turbo_factor);
+ global_max = mul_ext_fp(global_max, turbo_factor);
+ }
+ global_min = clamp_t(int32_t, global_min, 0, global_max);
+
+ cpu->min_perf = max(min_policy_perf, global_min);
+ cpu->min_perf = min(cpu->min_perf, max_policy_perf);
+ cpu->max_perf = min(max_policy_perf, global_max);
+ cpu->max_perf = max(min_policy_perf, cpu->max_perf);
+
+ /* Make sure min_perf <= max_perf */
+ cpu->min_perf = min(cpu->min_perf, cpu->max_perf);
+ }
+
+ cpu->max_perf = round_up(cpu->max_perf, EXT_FRAC_BITS);
+ cpu->min_perf = round_up(cpu->min_perf, EXT_FRAC_BITS);
pr_debug("cpu:%d max_perf_pct:%d min_perf_pct:%d\n", policy->cpu,
- limits->max_perf_pct, limits->min_perf_pct);
+ fp_ext_toint(cpu->max_perf * 100),
+ fp_ext_toint(cpu->min_perf * 100));
}
static int intel_pstate_set_policy(struct cpufreq_policy *policy)
{
struct cpudata *cpu;
- struct perf_limits *perf_limits = &global;
if (!policy->cpuinfo.max_freq)
return -ENODEV;
@@ -2105,19 +2027,9 @@
cpu = all_cpu_data[policy->cpu];
cpu->policy = policy->policy;
- if (cpu->pstate.max_pstate_physical > cpu->pstate.max_pstate &&
- policy->max < policy->cpuinfo.max_freq &&
- policy->max > cpu->pstate.max_pstate * cpu->pstate.scaling) {
- pr_debug("policy->max > max non turbo frequency\n");
- policy->max = policy->cpuinfo.max_freq;
- }
-
- if (per_cpu_limits)
- perf_limits = cpu->perf_limits;
-
mutex_lock(&intel_pstate_limits_lock);
- intel_pstate_update_perf_limits(policy, perf_limits);
+ intel_pstate_update_perf_limits(policy, cpu);
if (cpu->policy == CPUFREQ_POLICY_PERFORMANCE) {
/*
@@ -2130,38 +2042,38 @@
intel_pstate_set_update_util_hook(policy->cpu);
- intel_pstate_hwp_set_policy(policy);
+ if (hwp_active)
+ intel_pstate_hwp_set(policy->cpu);
mutex_unlock(&intel_pstate_limits_lock);
return 0;
}
+static void intel_pstate_adjust_policy_max(struct cpufreq_policy *policy,
+ struct cpudata *cpu)
+{
+ if (cpu->pstate.max_pstate_physical > cpu->pstate.max_pstate &&
+ policy->max < policy->cpuinfo.max_freq &&
+ policy->max > cpu->pstate.max_freq) {
+ pr_debug("policy->max > max non turbo frequency\n");
+ policy->max = policy->cpuinfo.max_freq;
+ }
+}
+
static int intel_pstate_verify_policy(struct cpufreq_policy *policy)
{
struct cpudata *cpu = all_cpu_data[policy->cpu];
update_turbo_state();
- policy->cpuinfo.max_freq = global.turbo_disabled || global.no_turbo ?
- cpu->pstate.max_freq :
- cpu->pstate.turbo_freq;
-
- cpufreq_verify_within_cpu_limits(policy);
+ cpufreq_verify_within_limits(policy, policy->cpuinfo.min_freq,
+ intel_pstate_get_max_freq(cpu));
if (policy->policy != CPUFREQ_POLICY_POWERSAVE &&
policy->policy != CPUFREQ_POLICY_PERFORMANCE)
return -EINVAL;
- /* When per-CPU limits are used, sysfs limits are not used */
- if (!per_cpu_limits) {
- unsigned int max_freq, min_freq;
-
- max_freq = policy->cpuinfo.max_freq *
- global.max_sysfs_pct / 100;
- min_freq = policy->cpuinfo.max_freq *
- global.min_sysfs_pct / 100;
- cpufreq_verify_within_limits(policy, min_freq, max_freq);
- }
+ intel_pstate_adjust_policy_max(policy, cpu);
return 0;
}
@@ -2202,8 +2114,8 @@
cpu = all_cpu_data[policy->cpu];
- if (per_cpu_limits)
- intel_pstate_init_limits(cpu->perf_limits);
+ cpu->max_perf = int_ext_tofp(1);
+ cpu->min_perf = 0;
policy->min = cpu->pstate.min_pstate * cpu->pstate.scaling;
policy->max = cpu->pstate.turbo_pstate * cpu->pstate.scaling;
@@ -2257,10 +2169,12 @@
struct cpudata *cpu = all_cpu_data[policy->cpu];
update_turbo_state();
- policy->cpuinfo.max_freq = global.no_turbo || global.turbo_disabled ?
- cpu->pstate.max_freq : cpu->pstate.turbo_freq;
+ cpufreq_verify_within_limits(policy, policy->cpuinfo.min_freq,
+ intel_pstate_get_max_freq(cpu));
- cpufreq_verify_within_cpu_limits(policy);
+ intel_pstate_adjust_policy_max(policy, cpu);
+
+ intel_pstate_update_perf_limits(policy, cpu);
return 0;
}
@@ -2324,6 +2238,7 @@
return ret;
policy->cpuinfo.transition_latency = INTEL_CPUFREQ_TRANSITION_LATENCY;
+ policy->transition_delay_us = INTEL_CPUFREQ_TRANSITION_DELAY;
/* This reflects the intel_pstate_get_cpu_pstates() setting. */
policy->cur = policy->cpuinfo.min_freq;
@@ -2341,7 +2256,13 @@
.name = "intel_cpufreq",
};
-static struct cpufreq_driver *intel_pstate_driver = &intel_pstate;
+static struct cpufreq_driver *default_driver = &intel_pstate;
+
+static bool pid_in_use(void)
+{
+ return intel_pstate_driver == &intel_pstate &&
+ pstate_funcs.update_util == intel_pstate_update_util_pid;
+}
static void intel_pstate_driver_cleanup(void)
{
@@ -2358,26 +2279,26 @@
}
}
put_online_cpus();
+ intel_pstate_driver = NULL;
}
-static int intel_pstate_register_driver(void)
+static int intel_pstate_register_driver(struct cpufreq_driver *driver)
{
int ret;
- intel_pstate_init_limits(&global);
+ memset(&global, 0, sizeof(global));
+ global.max_perf_pct = 100;
+ intel_pstate_driver = driver;
ret = cpufreq_register_driver(intel_pstate_driver);
if (ret) {
intel_pstate_driver_cleanup();
return ret;
}
- mutex_lock(&intel_pstate_limits_lock);
- driver_registered = true;
- mutex_unlock(&intel_pstate_limits_lock);
+ global.min_perf_pct = min_perf_pct_min();
- if (intel_pstate_driver == &intel_pstate && !hwp_active &&
- pstate_funcs.get_target_pstate != get_target_pstate_use_cpu_load)
+ if (pid_in_use())
intel_pstate_debug_expose_params();
return 0;
@@ -2388,14 +2309,9 @@
if (hwp_active)
return -EBUSY;
- if (intel_pstate_driver == &intel_pstate && !hwp_active &&
- pstate_funcs.get_target_pstate != get_target_pstate_use_cpu_load)
+ if (pid_in_use())
intel_pstate_debug_hide_params();
- mutex_lock(&intel_pstate_limits_lock);
- driver_registered = false;
- mutex_unlock(&intel_pstate_limits_lock);
-
cpufreq_unregister_driver(intel_pstate_driver);
intel_pstate_driver_cleanup();
@@ -2404,7 +2320,7 @@
static ssize_t intel_pstate_show_status(char *buf)
{
- if (!driver_registered)
+ if (!intel_pstate_driver)
return sprintf(buf, "off\n");
return sprintf(buf, "%s\n", intel_pstate_driver == &intel_pstate ?
@@ -2416,11 +2332,11 @@
int ret;
if (size == 3 && !strncmp(buf, "off", size))
- return driver_registered ?
+ return intel_pstate_driver ?
intel_pstate_unregister_driver() : -EINVAL;
if (size == 6 && !strncmp(buf, "active", size)) {
- if (driver_registered) {
+ if (intel_pstate_driver) {
if (intel_pstate_driver == &intel_pstate)
return 0;
@@ -2429,13 +2345,12 @@
return ret;
}
- intel_pstate_driver = &intel_pstate;
- return intel_pstate_register_driver();
+ return intel_pstate_register_driver(&intel_pstate);
}
if (size == 7 && !strncmp(buf, "passive", size)) {
- if (driver_registered) {
- if (intel_pstate_driver != &intel_pstate)
+ if (intel_pstate_driver) {
+ if (intel_pstate_driver == &intel_cpufreq)
return 0;
ret = intel_pstate_unregister_driver();
@@ -2443,8 +2358,7 @@
return ret;
}
- intel_pstate_driver = &intel_cpufreq;
- return intel_pstate_register_driver();
+ return intel_pstate_register_driver(&intel_cpufreq);
}
return -EINVAL;
@@ -2465,23 +2379,17 @@
return 0;
}
-static void __init copy_pid_params(struct pstate_adjust_policy *policy)
-{
- pid_params.sample_rate_ms = policy->sample_rate_ms;
- pid_params.sample_rate_ns = pid_params.sample_rate_ms * NSEC_PER_MSEC;
- pid_params.p_gain_pct = policy->p_gain_pct;
- pid_params.i_gain_pct = policy->i_gain_pct;
- pid_params.d_gain_pct = policy->d_gain_pct;
- pid_params.deadband = policy->deadband;
- pid_params.setpoint = policy->setpoint;
-}
-
#ifdef CONFIG_ACPI
static void intel_pstate_use_acpi_profile(void)
{
- if (acpi_gbl_FADT.preferred_profile == PM_MOBILE)
- pstate_funcs.get_target_pstate =
- get_target_pstate_use_cpu_load;
+ switch (acpi_gbl_FADT.preferred_profile) {
+ case PM_MOBILE:
+ case PM_TABLET:
+ case PM_APPLIANCE_PC:
+ case PM_DESKTOP:
+ case PM_WORKSTATION:
+ pstate_funcs.update_util = intel_pstate_update_util;
+ }
}
#else
static void intel_pstate_use_acpi_profile(void)
@@ -2498,7 +2406,7 @@
pstate_funcs.get_scaling = funcs->get_scaling;
pstate_funcs.get_val = funcs->get_val;
pstate_funcs.get_vid = funcs->get_vid;
- pstate_funcs.get_target_pstate = funcs->get_target_pstate;
+ pstate_funcs.update_util = funcs->update_util;
intel_pstate_use_acpi_profile();
}
@@ -2637,29 +2545,31 @@
static int __init intel_pstate_init(void)
{
- const struct x86_cpu_id *id;
- struct cpu_defaults *cpu_def;
- int rc = 0;
+ int rc;
if (no_load)
return -ENODEV;
- if (x86_match_cpu(hwp_support_ids) && !no_hwp) {
- copy_cpu_funcs(&core_params.funcs);
- hwp_active++;
- intel_pstate.attr = hwp_cpufreq_attrs;
- goto hwp_cpu_matched;
+ if (x86_match_cpu(hwp_support_ids)) {
+ copy_cpu_funcs(&core_funcs);
+ if (no_hwp) {
+ pstate_funcs.update_util = intel_pstate_update_util;
+ } else {
+ hwp_active++;
+ intel_pstate.attr = hwp_cpufreq_attrs;
+ pstate_funcs.update_util = intel_pstate_update_util_hwp;
+ goto hwp_cpu_matched;
+ }
+ } else {
+ const struct x86_cpu_id *id;
+
+ id = x86_match_cpu(intel_pstate_cpu_ids);
+ if (!id)
+ return -ENODEV;
+
+ copy_cpu_funcs((struct pstate_funcs *)id->driver_data);
}
- id = x86_match_cpu(intel_pstate_cpu_ids);
- if (!id)
- return -ENODEV;
-
- cpu_def = (struct cpu_defaults *)id->driver_data;
-
- copy_pid_params(&cpu_def->pid_policy);
- copy_cpu_funcs(&cpu_def->funcs);
-
if (intel_pstate_msrs_not_valid())
return -ENODEV;
@@ -2685,7 +2595,7 @@
intel_pstate_sysfs_expose_params();
mutex_lock(&intel_pstate_driver_lock);
- rc = intel_pstate_register_driver();
+ rc = intel_pstate_register_driver(default_driver);
mutex_unlock(&intel_pstate_driver_lock);
if (rc)
return rc;
@@ -2706,7 +2616,7 @@
no_load = 1;
} else if (!strcmp(str, "passive")) {
pr_info("Passive mode enabled\n");
- intel_pstate_driver = &intel_cpufreq;
+ default_driver = &intel_cpufreq;
no_hwp = 1;
}
if (!strcmp(str, "no_hwp")) {
diff --git a/drivers/cpufreq/mt8173-cpufreq.c b/drivers/cpufreq/mt8173-cpufreq.c
index ab25b123..fd1886f 100644
--- a/drivers/cpufreq/mt8173-cpufreq.c
+++ b/drivers/cpufreq/mt8173-cpufreq.c
@@ -573,14 +573,33 @@
.probe = mt8173_cpufreq_probe,
};
-static int mt8173_cpufreq_driver_init(void)
+/* List of machines supported by this driver */
+static const struct of_device_id mt8173_cpufreq_machines[] __initconst = {
+ { .compatible = "mediatek,mt817x", },
+ { .compatible = "mediatek,mt8173", },
+ { .compatible = "mediatek,mt8176", },
+
+ { }
+};
+
+static int __init mt8173_cpufreq_driver_init(void)
{
+ struct device_node *np;
+ const struct of_device_id *match;
struct platform_device *pdev;
int err;
- if (!of_machine_is_compatible("mediatek,mt8173"))
+ np = of_find_node_by_path("/");
+ if (!np)
return -ENODEV;
+ match = of_match_node(mt8173_cpufreq_machines, np);
+ of_node_put(np);
+ if (!match) {
+ pr_warn("Machine is not compatible with mt8173-cpufreq\n");
+ return -ENODEV;
+ }
+
err = platform_driver_register(&mt8173_cpufreq_platdrv);
if (err)
return err;
diff --git a/drivers/cpufreq/qoriq-cpufreq.c b/drivers/cpufreq/qoriq-cpufreq.c
index bfec1bc..e2ea433 100644
--- a/drivers/cpufreq/qoriq-cpufreq.c
+++ b/drivers/cpufreq/qoriq-cpufreq.c
@@ -52,17 +52,27 @@
{
struct device_node *soc;
u32 sysfreq;
+ struct clk *pltclk;
+ int ret;
+ /* get platform freq by searching bus-frequency property */
soc = of_find_node_by_type(NULL, "soc");
- if (!soc)
- return 0;
+ if (soc) {
+ ret = of_property_read_u32(soc, "bus-frequency", &sysfreq);
+ of_node_put(soc);
+ if (!ret)
+ return sysfreq;
+ }
- if (of_property_read_u32(soc, "bus-frequency", &sysfreq))
- sysfreq = 0;
+ /* get platform freq by its clock name */
+ pltclk = clk_get(NULL, "cg-pll0-div1");
+ if (IS_ERR(pltclk)) {
+ pr_err("%s: can't get bus frequency %ld\n",
+ __func__, PTR_ERR(pltclk));
+ return PTR_ERR(pltclk);
+ }
- of_node_put(soc);
-
- return sysfreq;
+ return clk_get_rate(pltclk);
}
static struct clk *cpu_to_clk(int cpu)
diff --git a/drivers/cpufreq/sh-cpufreq.c b/drivers/cpufreq/sh-cpufreq.c
index 86628e2..719c3d9 100644
--- a/drivers/cpufreq/sh-cpufreq.c
+++ b/drivers/cpufreq/sh-cpufreq.c
@@ -30,11 +30,51 @@
static DEFINE_PER_CPU(struct clk, sh_cpuclk);
+struct cpufreq_target {
+ struct cpufreq_policy *policy;
+ unsigned int freq;
+};
+
static unsigned int sh_cpufreq_get(unsigned int cpu)
{
return (clk_get_rate(&per_cpu(sh_cpuclk, cpu)) + 500) / 1000;
}
+static long __sh_cpufreq_target(void *arg)
+{
+ struct cpufreq_target *target = arg;
+ struct cpufreq_policy *policy = target->policy;
+ int cpu = policy->cpu;
+ struct clk *cpuclk = &per_cpu(sh_cpuclk, cpu);
+ struct cpufreq_freqs freqs;
+ struct device *dev;
+ long freq;
+
+ if (smp_processor_id() != cpu)
+ return -ENODEV;
+
+ dev = get_cpu_device(cpu);
+
+ /* Convert target_freq from kHz to Hz */
+ freq = clk_round_rate(cpuclk, target->freq * 1000);
+
+ if (freq < (policy->min * 1000) || freq > (policy->max * 1000))
+ return -EINVAL;
+
+ dev_dbg(dev, "requested frequency %u Hz\n", target->freq * 1000);
+
+ freqs.old = sh_cpufreq_get(cpu);
+ freqs.new = (freq + 500) / 1000;
+ freqs.flags = 0;
+
+ cpufreq_freq_transition_begin(target->policy, &freqs);
+ clk_set_rate(cpuclk, freq);
+ cpufreq_freq_transition_end(target->policy, &freqs, 0);
+
+ dev_dbg(dev, "set frequency %lu Hz\n", freq);
+ return 0;
+}
+
/*
* Here we notify other drivers of the proposed change and the final change.
*/
@@ -42,40 +82,9 @@
unsigned int target_freq,
unsigned int relation)
{
- unsigned int cpu = policy->cpu;
- struct clk *cpuclk = &per_cpu(sh_cpuclk, cpu);
- cpumask_t cpus_allowed;
- struct cpufreq_freqs freqs;
- struct device *dev;
- long freq;
+ struct cpufreq_target data = { .policy = policy, .freq = target_freq };
- cpus_allowed = current->cpus_allowed;
- set_cpus_allowed_ptr(current, cpumask_of(cpu));
-
- BUG_ON(smp_processor_id() != cpu);
-
- dev = get_cpu_device(cpu);
-
- /* Convert target_freq from kHz to Hz */
- freq = clk_round_rate(cpuclk, target_freq * 1000);
-
- if (freq < (policy->min * 1000) || freq > (policy->max * 1000))
- return -EINVAL;
-
- dev_dbg(dev, "requested frequency %u Hz\n", target_freq * 1000);
-
- freqs.old = sh_cpufreq_get(cpu);
- freqs.new = (freq + 500) / 1000;
- freqs.flags = 0;
-
- cpufreq_freq_transition_begin(policy, &freqs);
- set_cpus_allowed_ptr(current, &cpus_allowed);
- clk_set_rate(cpuclk, freq);
- cpufreq_freq_transition_end(policy, &freqs, 0);
-
- dev_dbg(dev, "set frequency %lu Hz\n", freq);
-
- return 0;
+ return work_on_cpu(policy->cpu, __sh_cpufreq_target, &data);
}
static int sh_cpufreq_verify(struct cpufreq_policy *policy)
diff --git a/drivers/cpufreq/sparc-us2e-cpufreq.c b/drivers/cpufreq/sparc-us2e-cpufreq.c
index 35ddb6d..90f33ef 100644
--- a/drivers/cpufreq/sparc-us2e-cpufreq.c
+++ b/drivers/cpufreq/sparc-us2e-cpufreq.c
@@ -118,10 +118,6 @@
unsigned long clock_tick,
unsigned long old_divisor, unsigned long divisor)
{
- unsigned long flags;
-
- local_irq_save(flags);
-
estar &= ~ESTAR_MODE_DIV_MASK;
/* This is based upon the state transition diagram in the IIe manual. */
@@ -152,8 +148,6 @@
} else {
BUG();
}
-
- local_irq_restore(flags);
}
static unsigned long index_to_estar_mode(unsigned int index)
@@ -229,48 +223,51 @@
return ret;
}
+static void __us2e_freq_get(void *arg)
+{
+ unsigned long *estar = arg;
+
+ *estar = read_hbreg(HBIRD_ESTAR_MODE_ADDR);
+}
+
static unsigned int us2e_freq_get(unsigned int cpu)
{
- cpumask_t cpus_allowed;
unsigned long clock_tick, estar;
- cpumask_copy(&cpus_allowed, ¤t->cpus_allowed);
- set_cpus_allowed_ptr(current, cpumask_of(cpu));
-
clock_tick = sparc64_get_clock_tick(cpu) / 1000;
- estar = read_hbreg(HBIRD_ESTAR_MODE_ADDR);
-
- set_cpus_allowed_ptr(current, &cpus_allowed);
+ if (smp_call_function_single(cpu, __us2e_freq_get, &estar, 1))
+ return 0;
return clock_tick / estar_to_divisor(estar);
}
-static int us2e_freq_target(struct cpufreq_policy *policy, unsigned int index)
+static void __us2e_freq_target(void *arg)
{
- unsigned int cpu = policy->cpu;
+ unsigned int cpu = smp_processor_id();
+ unsigned int *index = arg;
unsigned long new_bits, new_freq;
unsigned long clock_tick, divisor, old_divisor, estar;
- cpumask_t cpus_allowed;
-
- cpumask_copy(&cpus_allowed, ¤t->cpus_allowed);
- set_cpus_allowed_ptr(current, cpumask_of(cpu));
new_freq = clock_tick = sparc64_get_clock_tick(cpu) / 1000;
- new_bits = index_to_estar_mode(index);
- divisor = index_to_divisor(index);
+ new_bits = index_to_estar_mode(*index);
+ divisor = index_to_divisor(*index);
new_freq /= divisor;
estar = read_hbreg(HBIRD_ESTAR_MODE_ADDR);
old_divisor = estar_to_divisor(estar);
- if (old_divisor != divisor)
+ if (old_divisor != divisor) {
us2e_transition(estar, new_bits, clock_tick * 1000,
old_divisor, divisor);
+ }
+}
- set_cpus_allowed_ptr(current, &cpus_allowed);
+static int us2e_freq_target(struct cpufreq_policy *policy, unsigned int index)
+{
+ unsigned int cpu = policy->cpu;
- return 0;
+ return smp_call_function_single(cpu, __us2e_freq_target, &index, 1);
}
static int __init us2e_freq_cpu_init(struct cpufreq_policy *policy)
diff --git a/drivers/cpufreq/sparc-us3-cpufreq.c b/drivers/cpufreq/sparc-us3-cpufreq.c
index a8d86a4..30645b0 100644
--- a/drivers/cpufreq/sparc-us3-cpufreq.c
+++ b/drivers/cpufreq/sparc-us3-cpufreq.c
@@ -35,22 +35,28 @@
#define SAFARI_CFG_DIV_32 0x0000000080000000UL
#define SAFARI_CFG_DIV_MASK 0x00000000C0000000UL
-static unsigned long read_safari_cfg(void)
+static void read_safari_cfg(void *arg)
{
- unsigned long ret;
+ unsigned long ret, *val = arg;
__asm__ __volatile__("ldxa [%%g0] %1, %0"
: "=&r" (ret)
: "i" (ASI_SAFARI_CONFIG));
- return ret;
+ *val = ret;
}
-static void write_safari_cfg(unsigned long val)
+static void update_safari_cfg(void *arg)
{
+ unsigned long reg, *new_bits = arg;
+
+ read_safari_cfg(®);
+ reg &= ~SAFARI_CFG_DIV_MASK;
+ reg |= *new_bits;
+
__asm__ __volatile__("stxa %0, [%%g0] %1\n\t"
"membar #Sync"
: /* no outputs */
- : "r" (val), "i" (ASI_SAFARI_CONFIG)
+ : "r" (reg), "i" (ASI_SAFARI_CONFIG)
: "memory");
}
@@ -78,29 +84,17 @@
static unsigned int us3_freq_get(unsigned int cpu)
{
- cpumask_t cpus_allowed;
unsigned long reg;
- unsigned int ret;
- cpumask_copy(&cpus_allowed, ¤t->cpus_allowed);
- set_cpus_allowed_ptr(current, cpumask_of(cpu));
-
- reg = read_safari_cfg();
- ret = get_current_freq(cpu, reg);
-
- set_cpus_allowed_ptr(current, &cpus_allowed);
-
- return ret;
+ if (smp_call_function_single(cpu, read_safari_cfg, ®, 1))
+ return 0;
+ return get_current_freq(cpu, reg);
}
static int us3_freq_target(struct cpufreq_policy *policy, unsigned int index)
{
unsigned int cpu = policy->cpu;
- unsigned long new_bits, new_freq, reg;
- cpumask_t cpus_allowed;
-
- cpumask_copy(&cpus_allowed, ¤t->cpus_allowed);
- set_cpus_allowed_ptr(current, cpumask_of(cpu));
+ unsigned long new_bits, new_freq;
new_freq = sparc64_get_clock_tick(cpu) / 1000;
switch (index) {
@@ -121,15 +115,7 @@
BUG();
}
- reg = read_safari_cfg();
-
- reg &= ~SAFARI_CFG_DIV_MASK;
- reg |= new_bits;
- write_safari_cfg(reg);
-
- set_cpus_allowed_ptr(current, &cpus_allowed);
-
- return 0;
+ return smp_call_function_single(cpu, update_safari_cfg, &new_bits, 1);
}
static int __init us3_freq_cpu_init(struct cpufreq_policy *policy)
diff --git a/drivers/cpufreq/tegra186-cpufreq.c b/drivers/cpufreq/tegra186-cpufreq.c
new file mode 100644
index 0000000..fe78753
--- /dev/null
+++ b/drivers/cpufreq/tegra186-cpufreq.c
@@ -0,0 +1,275 @@
+/*
+ * Copyright (c) 2017, NVIDIA CORPORATION. All rights reserved
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/cpufreq.h>
+#include <linux/dma-mapping.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+
+#include <soc/tegra/bpmp.h>
+#include <soc/tegra/bpmp-abi.h>
+
+#define EDVD_CORE_VOLT_FREQ(core) (0x20 + (core) * 0x4)
+#define EDVD_CORE_VOLT_FREQ_F_SHIFT 0
+#define EDVD_CORE_VOLT_FREQ_V_SHIFT 16
+
+struct tegra186_cpufreq_cluster_info {
+ unsigned long offset;
+ int cpus[4];
+ unsigned int bpmp_cluster_id;
+};
+
+#define NO_CPU -1
+static const struct tegra186_cpufreq_cluster_info tegra186_clusters[] = {
+ /* Denver cluster */
+ {
+ .offset = SZ_64K * 7,
+ .cpus = { 1, 2, NO_CPU, NO_CPU },
+ .bpmp_cluster_id = 0,
+ },
+ /* A57 cluster */
+ {
+ .offset = SZ_64K * 6,
+ .cpus = { 0, 3, 4, 5 },
+ .bpmp_cluster_id = 1,
+ },
+};
+
+struct tegra186_cpufreq_cluster {
+ const struct tegra186_cpufreq_cluster_info *info;
+ struct cpufreq_frequency_table *table;
+};
+
+struct tegra186_cpufreq_data {
+ void __iomem *regs;
+
+ size_t num_clusters;
+ struct tegra186_cpufreq_cluster *clusters;
+};
+
+static int tegra186_cpufreq_init(struct cpufreq_policy *policy)
+{
+ struct tegra186_cpufreq_data *data = cpufreq_get_driver_data();
+ unsigned int i;
+
+ for (i = 0; i < data->num_clusters; i++) {
+ struct tegra186_cpufreq_cluster *cluster = &data->clusters[i];
+ const struct tegra186_cpufreq_cluster_info *info =
+ cluster->info;
+ int core;
+
+ for (core = 0; core < ARRAY_SIZE(info->cpus); core++) {
+ if (info->cpus[core] == policy->cpu)
+ break;
+ }
+ if (core == ARRAY_SIZE(info->cpus))
+ continue;
+
+ policy->driver_data =
+ data->regs + info->offset + EDVD_CORE_VOLT_FREQ(core);
+ cpufreq_table_validate_and_show(policy, cluster->table);
+ }
+
+ policy->cpuinfo.transition_latency = 300 * 1000;
+
+ return 0;
+}
+
+static int tegra186_cpufreq_set_target(struct cpufreq_policy *policy,
+ unsigned int index)
+{
+ struct cpufreq_frequency_table *tbl = policy->freq_table + index;
+ void __iomem *edvd_reg = policy->driver_data;
+ u32 edvd_val = tbl->driver_data;
+
+ writel(edvd_val, edvd_reg);
+
+ return 0;
+}
+
+static struct cpufreq_driver tegra186_cpufreq_driver = {
+ .name = "tegra186",
+ .flags = CPUFREQ_STICKY | CPUFREQ_HAVE_GOVERNOR_PER_POLICY,
+ .verify = cpufreq_generic_frequency_table_verify,
+ .target_index = tegra186_cpufreq_set_target,
+ .init = tegra186_cpufreq_init,
+ .attr = cpufreq_generic_attr,
+};
+
+static struct cpufreq_frequency_table *init_vhint_table(
+ struct platform_device *pdev, struct tegra_bpmp *bpmp,
+ unsigned int cluster_id)
+{
+ struct cpufreq_frequency_table *table;
+ struct mrq_cpu_vhint_request req;
+ struct tegra_bpmp_message msg;
+ struct cpu_vhint_data *data;
+ int err, i, j, num_rates = 0;
+ dma_addr_t phys;
+ void *virt;
+
+ virt = dma_alloc_coherent(bpmp->dev, sizeof(*data), &phys,
+ GFP_KERNEL | GFP_DMA32);
+ if (!virt)
+ return ERR_PTR(-ENOMEM);
+
+ data = (struct cpu_vhint_data *)virt;
+
+ memset(&req, 0, sizeof(req));
+ req.addr = phys;
+ req.cluster_id = cluster_id;
+
+ memset(&msg, 0, sizeof(msg));
+ msg.mrq = MRQ_CPU_VHINT;
+ msg.tx.data = &req;
+ msg.tx.size = sizeof(req);
+
+ err = tegra_bpmp_transfer(bpmp, &msg);
+ if (err) {
+ table = ERR_PTR(err);
+ goto free;
+ }
+
+ for (i = data->vfloor; i <= data->vceil; i++) {
+ u16 ndiv = data->ndiv[i];
+
+ if (ndiv < data->ndiv_min || ndiv > data->ndiv_max)
+ continue;
+
+ /* Only store lowest voltage index for each rate */
+ if (i > 0 && ndiv == data->ndiv[i - 1])
+ continue;
+
+ num_rates++;
+ }
+
+ table = devm_kcalloc(&pdev->dev, num_rates + 1, sizeof(*table),
+ GFP_KERNEL);
+ if (!table) {
+ table = ERR_PTR(-ENOMEM);
+ goto free;
+ }
+
+ for (i = data->vfloor, j = 0; i <= data->vceil; i++) {
+ struct cpufreq_frequency_table *point;
+ u16 ndiv = data->ndiv[i];
+ u32 edvd_val = 0;
+
+ if (ndiv < data->ndiv_min || ndiv > data->ndiv_max)
+ continue;
+
+ /* Only store lowest voltage index for each rate */
+ if (i > 0 && ndiv == data->ndiv[i - 1])
+ continue;
+
+ edvd_val |= i << EDVD_CORE_VOLT_FREQ_V_SHIFT;
+ edvd_val |= ndiv << EDVD_CORE_VOLT_FREQ_F_SHIFT;
+
+ point = &table[j++];
+ point->driver_data = edvd_val;
+ point->frequency = data->ref_clk_hz * ndiv / data->pdiv /
+ data->mdiv / 1000;
+ }
+
+ table[j].frequency = CPUFREQ_TABLE_END;
+
+free:
+ dma_free_coherent(bpmp->dev, sizeof(*data), virt, phys);
+
+ return table;
+}
+
+static int tegra186_cpufreq_probe(struct platform_device *pdev)
+{
+ struct tegra186_cpufreq_data *data;
+ struct tegra_bpmp *bpmp;
+ struct resource *res;
+ unsigned int i = 0, err;
+
+ data = devm_kzalloc(&pdev->dev, sizeof(*data), GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+
+ data->clusters = devm_kcalloc(&pdev->dev, ARRAY_SIZE(tegra186_clusters),
+ sizeof(*data->clusters), GFP_KERNEL);
+ if (!data->clusters)
+ return -ENOMEM;
+
+ data->num_clusters = ARRAY_SIZE(tegra186_clusters);
+
+ bpmp = tegra_bpmp_get(&pdev->dev);
+ if (IS_ERR(bpmp))
+ return PTR_ERR(bpmp);
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ data->regs = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(data->regs)) {
+ err = PTR_ERR(data->regs);
+ goto put_bpmp;
+ }
+
+ for (i = 0; i < data->num_clusters; i++) {
+ struct tegra186_cpufreq_cluster *cluster = &data->clusters[i];
+
+ cluster->info = &tegra186_clusters[i];
+ cluster->table = init_vhint_table(
+ pdev, bpmp, cluster->info->bpmp_cluster_id);
+ if (IS_ERR(cluster->table)) {
+ err = PTR_ERR(cluster->table);
+ goto put_bpmp;
+ }
+ }
+
+ tegra_bpmp_put(bpmp);
+
+ tegra186_cpufreq_driver.driver_data = data;
+
+ err = cpufreq_register_driver(&tegra186_cpufreq_driver);
+ if (err)
+ return err;
+
+ return 0;
+
+put_bpmp:
+ tegra_bpmp_put(bpmp);
+
+ return err;
+}
+
+static int tegra186_cpufreq_remove(struct platform_device *pdev)
+{
+ cpufreq_unregister_driver(&tegra186_cpufreq_driver);
+
+ return 0;
+}
+
+static const struct of_device_id tegra186_cpufreq_of_match[] = {
+ { .compatible = "nvidia,tegra186-ccplex-cluster", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, tegra186_cpufreq_of_match);
+
+static struct platform_driver tegra186_cpufreq_platform_driver = {
+ .driver = {
+ .name = "tegra186-cpufreq",
+ .of_match_table = tegra186_cpufreq_of_match,
+ },
+ .probe = tegra186_cpufreq_probe,
+ .remove = tegra186_cpufreq_remove,
+};
+module_platform_driver(tegra186_cpufreq_platform_driver);
+
+MODULE_AUTHOR("Mikko Perttunen <mperttunen@nvidia.com>");
+MODULE_DESCRIPTION("NVIDIA Tegra186 cpufreq driver");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/cpuidle/cpuidle-cps.c b/drivers/cpuidle/cpuidle-cps.c
index 926ba98..12b9145 100644
--- a/drivers/cpuidle/cpuidle-cps.c
+++ b/drivers/cpuidle/cpuidle-cps.c
@@ -118,7 +118,7 @@
static int __init cps_cpuidle_init(void)
{
- int err, cpu, core, i;
+ int err, cpu, i;
struct cpuidle_device *device;
/* Detect supported states */
@@ -160,7 +160,6 @@
}
for_each_possible_cpu(cpu) {
- core = cpu_data[cpu].core;
device = &per_cpu(cpuidle_dev, cpu);
device->cpu = cpu;
#ifdef CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED
diff --git a/drivers/cpuidle/cpuidle-powernv.c b/drivers/cpuidle/cpuidle-powernv.c
index cda8f62..12409a5 100644
--- a/drivers/cpuidle/cpuidle-powernv.c
+++ b/drivers/cpuidle/cpuidle-powernv.c
@@ -56,10 +56,9 @@
snooze_exit_time = get_tb() + snooze_timeout;
ppc64_runlatch_off();
+ HMT_very_low();
while (!need_resched()) {
- HMT_low();
- HMT_very_low();
- if (snooze_timeout_en && get_tb() > snooze_exit_time)
+ if (likely(snooze_timeout_en) && get_tb() > snooze_exit_time)
break;
}
@@ -215,11 +214,25 @@
stop_psscr_table[index].mask = psscr_mask;
}
+/*
+ * Returns 0 if prop1_len == prop2_len. Else returns -1
+ */
+static inline int validate_dt_prop_sizes(const char *prop1, int prop1_len,
+ const char *prop2, int prop2_len)
+{
+ if (prop1_len == prop2_len)
+ return 0;
+
+ pr_warn("cpuidle-powernv: array sizes don't match for %s and %s\n",
+ prop1, prop2);
+ return -1;
+}
+
static int powernv_add_idle_states(void)
{
struct device_node *power_mgt;
int nr_idle_states = 1; /* Snooze */
- int dt_idle_states;
+ int dt_idle_states, count;
u32 latency_ns[CPUIDLE_STATE_MAX];
u32 residency_ns[CPUIDLE_STATE_MAX];
u32 flags[CPUIDLE_STATE_MAX];
@@ -244,6 +257,21 @@
goto out;
}
+ count = of_property_count_u32_elems(power_mgt,
+ "ibm,cpu-idle-state-latencies-ns");
+
+ if (validate_dt_prop_sizes("ibm,cpu-idle-state-flags", dt_idle_states,
+ "ibm,cpu-idle-state-latencies-ns",
+ count) != 0)
+ goto out;
+
+ count = of_property_count_strings(power_mgt,
+ "ibm,cpu-idle-state-names");
+ if (validate_dt_prop_sizes("ibm,cpu-idle-state-flags", dt_idle_states,
+ "ibm,cpu-idle-state-names",
+ count) != 0)
+ goto out;
+
/*
* Since snooze is used as first idle state, max idle states allowed is
* CPUIDLE_STATE_MAX -1
@@ -278,6 +306,22 @@
has_stop_states = (flags[0] &
(OPAL_PM_STOP_INST_FAST | OPAL_PM_STOP_INST_DEEP));
if (has_stop_states) {
+ count = of_property_count_u64_elems(power_mgt,
+ "ibm,cpu-idle-state-psscr");
+ if (validate_dt_prop_sizes("ibm,cpu-idle-state-flags",
+ dt_idle_states,
+ "ibm,cpu-idle-state-psscr",
+ count) != 0)
+ goto out;
+
+ count = of_property_count_u64_elems(power_mgt,
+ "ibm,cpu-idle-state-psscr-mask");
+ if (validate_dt_prop_sizes("ibm,cpu-idle-state-flags",
+ dt_idle_states,
+ "ibm,cpu-idle-state-psscr-mask",
+ count) != 0)
+ goto out;
+
if (of_property_read_u64_array(power_mgt,
"ibm,cpu-idle-state-psscr", psscr_val, dt_idle_states)) {
pr_warn("cpuidle-powernv: missing ibm,cpu-idle-state-psscr in DT\n");
@@ -292,8 +336,21 @@
}
}
- rc = of_property_read_u32_array(power_mgt,
- "ibm,cpu-idle-state-residency-ns", residency_ns, dt_idle_states);
+ count = of_property_count_u32_elems(power_mgt,
+ "ibm,cpu-idle-state-residency-ns");
+
+ if (count < 0) {
+ rc = count;
+ } else if (validate_dt_prop_sizes("ibm,cpu-idle-state-flags",
+ dt_idle_states,
+ "ibm,cpu-idle-state-residency-ns",
+ count) != 0) {
+ goto out;
+ } else {
+ rc = of_property_read_u32_array(power_mgt,
+ "ibm,cpu-idle-state-residency-ns",
+ residency_ns, dt_idle_states);
+ }
for (i = 0; i < dt_idle_states; i++) {
unsigned int exit_latency, target_residency;
diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig
index 473d312..fb1e60f 100644
--- a/drivers/crypto/Kconfig
+++ b/drivers/crypto/Kconfig
@@ -388,6 +388,21 @@
This option enables support for the Security Controller (SCC)
found in Freescale i.MX25 chips.
+config CRYPTO_DEV_EXYNOS_RNG
+ tristate "EXYNOS HW pseudo random number generator support"
+ depends on ARCH_EXYNOS || COMPILE_TEST
+ depends on HAS_IOMEM
+ select CRYPTO_RNG
+ ---help---
+ This driver provides kernel-side support through the
+ cryptographic API for the pseudo random number generator hardware
+ found on Exynos SoCs.
+
+ To compile this driver as a module, choose M here: the
+ module will be called exynos-rng.
+
+ If unsure, say Y.
+
config CRYPTO_DEV_S5P
tristate "Support for Samsung S5PV210/Exynos crypto accelerator"
depends on ARCH_S5PV210 || ARCH_EXYNOS || COMPILE_TEST
@@ -515,6 +530,13 @@
source "drivers/crypto/qat/Kconfig"
source "drivers/crypto/cavium/cpt/Kconfig"
+config CRYPTO_DEV_CAVIUM_ZIP
+ tristate "Cavium ZIP driver"
+ depends on PCI && 64BIT && (ARM64 || COMPILE_TEST)
+ ---help---
+ Select this option if you want to enable compression/decompression
+ acceleration on Cavium's ARM based SoCs
+
config CRYPTO_DEV_QCE
tristate "Qualcomm crypto engine accelerator"
depends on (ARCH_QCOM || COMPILE_TEST) && HAS_DMA && HAS_IOMEM
@@ -619,4 +641,6 @@
Secure Processing Unit (SPU). The SPU driver registers ablkcipher,
ahash, and aead algorithms with the kernel cryptographic API.
+source "drivers/crypto/stm32/Kconfig"
+
endif # CRYPTO_HW
diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile
index 7396094..463f335 100644
--- a/drivers/crypto/Makefile
+++ b/drivers/crypto/Makefile
@@ -2,9 +2,11 @@
obj-$(CONFIG_CRYPTO_DEV_ATMEL_SHA) += atmel-sha.o
obj-$(CONFIG_CRYPTO_DEV_ATMEL_TDES) += atmel-tdes.o
obj-$(CONFIG_CRYPTO_DEV_BFIN_CRC) += bfin_crc.o
+obj-$(CONFIG_CRYPTO_DEV_CAVIUM_ZIP) += cavium/
obj-$(CONFIG_CRYPTO_DEV_CCP) += ccp/
obj-$(CONFIG_CRYPTO_DEV_CHELSIO) += chelsio/
obj-$(CONFIG_CRYPTO_DEV_CPT) += cavium/cpt/
+obj-$(CONFIG_CRYPTO_DEV_EXYNOS_RNG) += exynos-rng.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM) += caam/
obj-$(CONFIG_CRYPTO_DEV_GEODE) += geode-aes.o
obj-$(CONFIG_CRYPTO_DEV_HIFN_795X) += hifn_795x.o
@@ -30,6 +32,7 @@
obj-$(CONFIG_CRYPTO_DEV_ROCKCHIP) += rockchip/
obj-$(CONFIG_CRYPTO_DEV_S5P) += s5p-sss.o
obj-$(CONFIG_CRYPTO_DEV_SAHARA) += sahara.o
+obj-$(CONFIG_CRYPTO_DEV_STM32) += stm32/
obj-$(CONFIG_CRYPTO_DEV_SUN4I_SS) += sunxi-ss/
obj-$(CONFIG_CRYPTO_DEV_TALITOS) += talitos.o
obj-$(CONFIG_CRYPTO_DEV_UX500) += ux500/
diff --git a/drivers/crypto/amcc/crypto4xx_core.c b/drivers/crypto/amcc/crypto4xx_core.c
index d10b4ae..fdc83a2 100644
--- a/drivers/crypto/amcc/crypto4xx_core.c
+++ b/drivers/crypto/amcc/crypto4xx_core.c
@@ -50,7 +50,7 @@
static void crypto4xx_hw_init(struct crypto4xx_device *dev)
{
union ce_ring_size ring_size;
- union ce_ring_contol ring_ctrl;
+ union ce_ring_control ring_ctrl;
union ce_part_ring_size part_ring_size;
union ce_io_threshold io_threshold;
u32 rand_num;
diff --git a/drivers/crypto/amcc/crypto4xx_reg_def.h b/drivers/crypto/amcc/crypto4xx_reg_def.h
index 46fe57c..279b872 100644
--- a/drivers/crypto/amcc/crypto4xx_reg_def.h
+++ b/drivers/crypto/amcc/crypto4xx_reg_def.h
@@ -180,7 +180,7 @@
} __attribute__((packed));
#define CRYPTO4XX_RING_CONTROL_OFFSET 0x54
-union ce_ring_contol {
+union ce_ring_control {
struct {
u32 continuous:1;
u32 rsv:5;
diff --git a/drivers/crypto/bcm/util.c b/drivers/crypto/bcm/util.c
index 0502f46..430c557 100644
--- a/drivers/crypto/bcm/util.c
+++ b/drivers/crypto/bcm/util.c
@@ -312,7 +312,7 @@
}
rc = crypto_shash_final(&sdesc->shash, result);
if (rc)
- pr_err("%s: Could not genereate %s hash", __func__, name);
+ pr_err("%s: Could not generate %s hash", __func__, name);
do_shash_err:
crypto_free_shash(hash);
diff --git a/drivers/crypto/caam/Kconfig b/drivers/crypto/caam/Kconfig
index bc0d356..e36aeac 100644
--- a/drivers/crypto/caam/Kconfig
+++ b/drivers/crypto/caam/Kconfig
@@ -87,6 +87,23 @@
To compile this as a module, choose M here: the module
will be called caamalg.
+config CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI
+ tristate "Queue Interface as Crypto API backend"
+ depends on CRYPTO_DEV_FSL_CAAM_JR && FSL_DPAA && NET
+ default y
+ select CRYPTO_AUTHENC
+ select CRYPTO_BLKCIPHER
+ help
+ Selecting this will use CAAM Queue Interface (QI) for sending
+ & receiving crypto jobs to/from CAAM. This gives better performance
+ than job ring interface when the number of cores are more than the
+ number of job rings assigned to the kernel. The number of portals
+ assigned to the kernel should also be more than the number of
+ job rings.
+
+ To compile this as a module, choose M here: the module
+ will be called caamalg_qi.
+
config CRYPTO_DEV_FSL_CAAM_AHASH_API
tristate "Register hash algorithm implementations with Crypto API"
depends on CRYPTO_DEV_FSL_CAAM_JR
@@ -136,4 +153,5 @@
information in the CAAM driver.
config CRYPTO_DEV_FSL_CAAM_CRYPTO_API_DESC
- def_tristate CRYPTO_DEV_FSL_CAAM_CRYPTO_API
+ def_tristate (CRYPTO_DEV_FSL_CAAM_CRYPTO_API || \
+ CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI)
diff --git a/drivers/crypto/caam/Makefile b/drivers/crypto/caam/Makefile
index 6554742..9e2e988 100644
--- a/drivers/crypto/caam/Makefile
+++ b/drivers/crypto/caam/Makefile
@@ -8,6 +8,7 @@
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM) += caam.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_JR) += caam_jr.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API) += caamalg.o
+obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI) += caamalg_qi.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_DESC) += caamalg_desc.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API) += caamhash.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_RNG_API) += caamrng.o
@@ -16,3 +17,7 @@
caam-objs := ctrl.o
caam_jr-objs := jr.o key_gen.o error.o
caam_pkc-y := caampkc.o pkc_desc.o
+ifneq ($(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI),)
+ ccflags-y += -DCONFIG_CAAM_QI
+ caam-objs += qi.o
+endif
diff --git a/drivers/crypto/caam/caamalg.c b/drivers/crypto/caam/caamalg.c
index 9bc80eb..398807d 100644
--- a/drivers/crypto/caam/caamalg.c
+++ b/drivers/crypto/caam/caamalg.c
@@ -266,8 +266,9 @@
/* aead_encrypt shared descriptor */
desc = ctx->sh_desc_enc;
- cnstr_shdsc_aead_encap(desc, &ctx->cdata, &ctx->adata, ctx->authsize,
- is_rfc3686, nonce, ctx1_iv_off);
+ cnstr_shdsc_aead_encap(desc, &ctx->cdata, &ctx->adata, ivsize,
+ ctx->authsize, is_rfc3686, nonce, ctx1_iv_off,
+ false);
dma_sync_single_for_device(jrdev, ctx->sh_desc_enc_dma,
desc_bytes(desc), DMA_TO_DEVICE);
@@ -299,7 +300,7 @@
desc = ctx->sh_desc_dec;
cnstr_shdsc_aead_decap(desc, &ctx->cdata, &ctx->adata, ivsize,
ctx->authsize, alg->caam.geniv, is_rfc3686,
- nonce, ctx1_iv_off);
+ nonce, ctx1_iv_off, false);
dma_sync_single_for_device(jrdev, ctx->sh_desc_dec_dma,
desc_bytes(desc), DMA_TO_DEVICE);
@@ -333,7 +334,7 @@
desc = ctx->sh_desc_enc;
cnstr_shdsc_aead_givencap(desc, &ctx->cdata, &ctx->adata, ivsize,
ctx->authsize, is_rfc3686, nonce,
- ctx1_iv_off);
+ ctx1_iv_off, false);
dma_sync_single_for_device(jrdev, ctx->sh_desc_enc_dma,
desc_bytes(desc), DMA_TO_DEVICE);
diff --git a/drivers/crypto/caam/caamalg_desc.c b/drivers/crypto/caam/caamalg_desc.c
index f3f48c1..6f9c7ec 100644
--- a/drivers/crypto/caam/caamalg_desc.c
+++ b/drivers/crypto/caam/caamalg_desc.c
@@ -265,17 +265,19 @@
* split key is to be used, the size of the split key itself is
* specified. Valid algorithm values - one of OP_ALG_ALGSEL_{MD5, SHA1,
* SHA224, SHA256, SHA384, SHA512} ANDed with OP_ALG_AAI_HMAC_PRECOMP.
+ * @ivsize: initialization vector size
* @icvsize: integrity check value (ICV) size (truncated or full)
* @is_rfc3686: true when ctr(aes) is wrapped by rfc3686 template
* @nonce: pointer to rfc3686 nonce
* @ctx1_iv_off: IV offset in CONTEXT1 register
+ * @is_qi: true when called from caam/qi
*
* Note: Requires an MDHA split key.
*/
void cnstr_shdsc_aead_encap(u32 * const desc, struct alginfo *cdata,
- struct alginfo *adata, unsigned int icvsize,
- const bool is_rfc3686, u32 *nonce,
- const u32 ctx1_iv_off)
+ struct alginfo *adata, unsigned int ivsize,
+ unsigned int icvsize, const bool is_rfc3686,
+ u32 *nonce, const u32 ctx1_iv_off, const bool is_qi)
{
/* Note: Context registers are saved. */
init_sh_desc_key_aead(desc, cdata, adata, is_rfc3686, nonce);
@@ -284,6 +286,25 @@
append_operation(desc, adata->algtype | OP_ALG_AS_INITFINAL |
OP_ALG_ENCRYPT);
+ if (is_qi) {
+ u32 *wait_load_cmd;
+
+ /* REG3 = assoclen */
+ append_seq_load(desc, 4, LDST_CLASS_DECO |
+ LDST_SRCDST_WORD_DECO_MATH3 |
+ (4 << LDST_OFFSET_SHIFT));
+
+ wait_load_cmd = append_jump(desc, JUMP_JSL | JUMP_TEST_ALL |
+ JUMP_COND_CALM | JUMP_COND_NCP |
+ JUMP_COND_NOP | JUMP_COND_NIP |
+ JUMP_COND_NIFP);
+ set_jump_tgt_here(desc, wait_load_cmd);
+
+ append_seq_load(desc, ivsize, LDST_CLASS_1_CCB |
+ LDST_SRCDST_BYTE_CONTEXT |
+ (ctx1_iv_off << LDST_OFFSET_SHIFT));
+ }
+
/* Read and write assoclen bytes */
append_math_add(desc, VARSEQINLEN, ZERO, REG3, CAAM_CMD_SZ);
append_math_add(desc, VARSEQOUTLEN, ZERO, REG3, CAAM_CMD_SZ);
@@ -338,6 +359,7 @@
* @is_rfc3686: true when ctr(aes) is wrapped by rfc3686 template
* @nonce: pointer to rfc3686 nonce
* @ctx1_iv_off: IV offset in CONTEXT1 register
+ * @is_qi: true when called from caam/qi
*
* Note: Requires an MDHA split key.
*/
@@ -345,7 +367,7 @@
struct alginfo *adata, unsigned int ivsize,
unsigned int icvsize, const bool geniv,
const bool is_rfc3686, u32 *nonce,
- const u32 ctx1_iv_off)
+ const u32 ctx1_iv_off, const bool is_qi)
{
/* Note: Context registers are saved. */
init_sh_desc_key_aead(desc, cdata, adata, is_rfc3686, nonce);
@@ -354,6 +376,26 @@
append_operation(desc, adata->algtype | OP_ALG_AS_INITFINAL |
OP_ALG_DECRYPT | OP_ALG_ICV_ON);
+ if (is_qi) {
+ u32 *wait_load_cmd;
+
+ /* REG3 = assoclen */
+ append_seq_load(desc, 4, LDST_CLASS_DECO |
+ LDST_SRCDST_WORD_DECO_MATH3 |
+ (4 << LDST_OFFSET_SHIFT));
+
+ wait_load_cmd = append_jump(desc, JUMP_JSL | JUMP_TEST_ALL |
+ JUMP_COND_CALM | JUMP_COND_NCP |
+ JUMP_COND_NOP | JUMP_COND_NIP |
+ JUMP_COND_NIFP);
+ set_jump_tgt_here(desc, wait_load_cmd);
+
+ if (!geniv)
+ append_seq_load(desc, ivsize, LDST_CLASS_1_CCB |
+ LDST_SRCDST_BYTE_CONTEXT |
+ (ctx1_iv_off << LDST_OFFSET_SHIFT));
+ }
+
/* Read and write assoclen bytes */
append_math_add(desc, VARSEQINLEN, ZERO, REG3, CAAM_CMD_SZ);
if (geniv)
@@ -423,21 +465,44 @@
* @is_rfc3686: true when ctr(aes) is wrapped by rfc3686 template
* @nonce: pointer to rfc3686 nonce
* @ctx1_iv_off: IV offset in CONTEXT1 register
+ * @is_qi: true when called from caam/qi
*
* Note: Requires an MDHA split key.
*/
void cnstr_shdsc_aead_givencap(u32 * const desc, struct alginfo *cdata,
struct alginfo *adata, unsigned int ivsize,
unsigned int icvsize, const bool is_rfc3686,
- u32 *nonce, const u32 ctx1_iv_off)
+ u32 *nonce, const u32 ctx1_iv_off,
+ const bool is_qi)
{
u32 geniv, moveiv;
/* Note: Context registers are saved. */
init_sh_desc_key_aead(desc, cdata, adata, is_rfc3686, nonce);
- if (is_rfc3686)
+ if (is_qi) {
+ u32 *wait_load_cmd;
+
+ /* REG3 = assoclen */
+ append_seq_load(desc, 4, LDST_CLASS_DECO |
+ LDST_SRCDST_WORD_DECO_MATH3 |
+ (4 << LDST_OFFSET_SHIFT));
+
+ wait_load_cmd = append_jump(desc, JUMP_JSL | JUMP_TEST_ALL |
+ JUMP_COND_CALM | JUMP_COND_NCP |
+ JUMP_COND_NOP | JUMP_COND_NIP |
+ JUMP_COND_NIFP);
+ set_jump_tgt_here(desc, wait_load_cmd);
+ }
+
+ if (is_rfc3686) {
+ if (is_qi)
+ append_seq_load(desc, ivsize, LDST_CLASS_1_CCB |
+ LDST_SRCDST_BYTE_CONTEXT |
+ (ctx1_iv_off << LDST_OFFSET_SHIFT));
+
goto copy_iv;
+ }
/* Generate IV */
geniv = NFIFOENTRY_STYPE_PAD | NFIFOENTRY_DEST_DECO |
diff --git a/drivers/crypto/caam/caamalg_desc.h b/drivers/crypto/caam/caamalg_desc.h
index 9555173..8731e4a 100644
--- a/drivers/crypto/caam/caamalg_desc.h
+++ b/drivers/crypto/caam/caamalg_desc.h
@@ -12,6 +12,9 @@
#define DESC_AEAD_ENC_LEN (DESC_AEAD_BASE + 11 * CAAM_CMD_SZ)
#define DESC_AEAD_DEC_LEN (DESC_AEAD_BASE + 15 * CAAM_CMD_SZ)
#define DESC_AEAD_GIVENC_LEN (DESC_AEAD_ENC_LEN + 7 * CAAM_CMD_SZ)
+#define DESC_QI_AEAD_ENC_LEN (DESC_AEAD_ENC_LEN + 3 * CAAM_CMD_SZ)
+#define DESC_QI_AEAD_DEC_LEN (DESC_AEAD_DEC_LEN + 3 * CAAM_CMD_SZ)
+#define DESC_QI_AEAD_GIVENC_LEN (DESC_AEAD_GIVENC_LEN + 3 * CAAM_CMD_SZ)
/* Note: Nonce is counted in cdata.keylen */
#define DESC_AEAD_CTR_RFC3686_LEN (4 * CAAM_CMD_SZ)
@@ -45,20 +48,22 @@
unsigned int icvsize);
void cnstr_shdsc_aead_encap(u32 * const desc, struct alginfo *cdata,
- struct alginfo *adata, unsigned int icvsize,
- const bool is_rfc3686, u32 *nonce,
- const u32 ctx1_iv_off);
+ struct alginfo *adata, unsigned int ivsize,
+ unsigned int icvsize, const bool is_rfc3686,
+ u32 *nonce, const u32 ctx1_iv_off,
+ const bool is_qi);
void cnstr_shdsc_aead_decap(u32 * const desc, struct alginfo *cdata,
struct alginfo *adata, unsigned int ivsize,
unsigned int icvsize, const bool geniv,
const bool is_rfc3686, u32 *nonce,
- const u32 ctx1_iv_off);
+ const u32 ctx1_iv_off, const bool is_qi);
void cnstr_shdsc_aead_givencap(u32 * const desc, struct alginfo *cdata,
struct alginfo *adata, unsigned int ivsize,
unsigned int icvsize, const bool is_rfc3686,
- u32 *nonce, const u32 ctx1_iv_off);
+ u32 *nonce, const u32 ctx1_iv_off,
+ const bool is_qi);
void cnstr_shdsc_gcm_encap(u32 * const desc, struct alginfo *cdata,
unsigned int icvsize);
diff --git a/drivers/crypto/caam/caamalg_qi.c b/drivers/crypto/caam/caamalg_qi.c
new file mode 100644
index 0000000..ea0e5b8
--- /dev/null
+++ b/drivers/crypto/caam/caamalg_qi.c
@@ -0,0 +1,2387 @@
+/*
+ * Freescale FSL CAAM support for crypto API over QI backend.
+ * Based on caamalg.c
+ *
+ * Copyright 2013-2016 Freescale Semiconductor, Inc.
+ * Copyright 2016-2017 NXP
+ */
+
+#include "compat.h"
+
+#include "regs.h"
+#include "intern.h"
+#include "desc_constr.h"
+#include "error.h"
+#include "sg_sw_sec4.h"
+#include "sg_sw_qm.h"
+#include "key_gen.h"
+#include "qi.h"
+#include "jr.h"
+#include "caamalg_desc.h"
+
+/*
+ * crypto alg
+ */
+#define CAAM_CRA_PRIORITY 2000
+/* max key is sum of AES_MAX_KEY_SIZE, max split key size */
+#define CAAM_MAX_KEY_SIZE (AES_MAX_KEY_SIZE + \
+ SHA512_DIGEST_SIZE * 2)
+
+#define DESC_MAX_USED_BYTES (DESC_QI_AEAD_GIVENC_LEN + \
+ CAAM_MAX_KEY_SIZE)
+#define DESC_MAX_USED_LEN (DESC_MAX_USED_BYTES / CAAM_CMD_SZ)
+
+struct caam_alg_entry {
+ int class1_alg_type;
+ int class2_alg_type;
+ bool rfc3686;
+ bool geniv;
+};
+
+struct caam_aead_alg {
+ struct aead_alg aead;
+ struct caam_alg_entry caam;
+ bool registered;
+};
+
+/*
+ * per-session context
+ */
+struct caam_ctx {
+ struct device *jrdev;
+ u32 sh_desc_enc[DESC_MAX_USED_LEN];
+ u32 sh_desc_dec[DESC_MAX_USED_LEN];
+ u32 sh_desc_givenc[DESC_MAX_USED_LEN];
+ u8 key[CAAM_MAX_KEY_SIZE];
+ dma_addr_t key_dma;
+ struct alginfo adata;
+ struct alginfo cdata;
+ unsigned int authsize;
+ struct device *qidev;
+ spinlock_t lock; /* Protects multiple init of driver context */
+ struct caam_drv_ctx *drv_ctx[NUM_OP];
+};
+
+static int aead_set_sh_desc(struct crypto_aead *aead)
+{
+ struct caam_aead_alg *alg = container_of(crypto_aead_alg(aead),
+ typeof(*alg), aead);
+ struct caam_ctx *ctx = crypto_aead_ctx(aead);
+ unsigned int ivsize = crypto_aead_ivsize(aead);
+ u32 ctx1_iv_off = 0;
+ u32 *nonce = NULL;
+ unsigned int data_len[2];
+ u32 inl_mask;
+ const bool ctr_mode = ((ctx->cdata.algtype & OP_ALG_AAI_MASK) ==
+ OP_ALG_AAI_CTR_MOD128);
+ const bool is_rfc3686 = alg->caam.rfc3686;
+
+ if (!ctx->cdata.keylen || !ctx->authsize)
+ return 0;
+
+ /*
+ * AES-CTR needs to load IV in CONTEXT1 reg
+ * at an offset of 128bits (16bytes)
+ * CONTEXT1[255:128] = IV
+ */
+ if (ctr_mode)
+ ctx1_iv_off = 16;
+
+ /*
+ * RFC3686 specific:
+ * CONTEXT1[255:128] = {NONCE, IV, COUNTER}
+ */
+ if (is_rfc3686) {
+ ctx1_iv_off = 16 + CTR_RFC3686_NONCE_SIZE;
+ nonce = (u32 *)((void *)ctx->key + ctx->adata.keylen_pad +
+ ctx->cdata.keylen - CTR_RFC3686_NONCE_SIZE);
+ }
+
+ data_len[0] = ctx->adata.keylen_pad;
+ data_len[1] = ctx->cdata.keylen;
+
+ if (alg->caam.geniv)
+ goto skip_enc;
+
+ /* aead_encrypt shared descriptor */
+ if (desc_inline_query(DESC_QI_AEAD_ENC_LEN +
+ (is_rfc3686 ? DESC_AEAD_CTR_RFC3686_LEN : 0),
+ DESC_JOB_IO_LEN, data_len, &inl_mask,
+ ARRAY_SIZE(data_len)) < 0)
+ return -EINVAL;
+
+ if (inl_mask & 1)
+ ctx->adata.key_virt = ctx->key;
+ else
+ ctx->adata.key_dma = ctx->key_dma;
+
+ if (inl_mask & 2)
+ ctx->cdata.key_virt = ctx->key + ctx->adata.keylen_pad;
+ else
+ ctx->cdata.key_dma = ctx->key_dma + ctx->adata.keylen_pad;
+
+ ctx->adata.key_inline = !!(inl_mask & 1);
+ ctx->cdata.key_inline = !!(inl_mask & 2);
+
+ cnstr_shdsc_aead_encap(ctx->sh_desc_enc, &ctx->cdata, &ctx->adata,
+ ivsize, ctx->authsize, is_rfc3686, nonce,
+ ctx1_iv_off, true);
+
+skip_enc:
+ /* aead_decrypt shared descriptor */
+ if (desc_inline_query(DESC_QI_AEAD_DEC_LEN +
+ (is_rfc3686 ? DESC_AEAD_CTR_RFC3686_LEN : 0),
+ DESC_JOB_IO_LEN, data_len, &inl_mask,
+ ARRAY_SIZE(data_len)) < 0)
+ return -EINVAL;
+
+ if (inl_mask & 1)
+ ctx->adata.key_virt = ctx->key;
+ else
+ ctx->adata.key_dma = ctx->key_dma;
+
+ if (inl_mask & 2)
+ ctx->cdata.key_virt = ctx->key + ctx->adata.keylen_pad;
+ else
+ ctx->cdata.key_dma = ctx->key_dma + ctx->adata.keylen_pad;
+
+ ctx->adata.key_inline = !!(inl_mask & 1);
+ ctx->cdata.key_inline = !!(inl_mask & 2);
+
+ cnstr_shdsc_aead_decap(ctx->sh_desc_dec, &ctx->cdata, &ctx->adata,
+ ivsize, ctx->authsize, alg->caam.geniv,
+ is_rfc3686, nonce, ctx1_iv_off, true);
+
+ if (!alg->caam.geniv)
+ goto skip_givenc;
+
+ /* aead_givencrypt shared descriptor */
+ if (desc_inline_query(DESC_QI_AEAD_GIVENC_LEN +
+ (is_rfc3686 ? DESC_AEAD_CTR_RFC3686_LEN : 0),
+ DESC_JOB_IO_LEN, data_len, &inl_mask,
+ ARRAY_SIZE(data_len)) < 0)
+ return -EINVAL;
+
+ if (inl_mask & 1)
+ ctx->adata.key_virt = ctx->key;
+ else
+ ctx->adata.key_dma = ctx->key_dma;
+
+ if (inl_mask & 2)
+ ctx->cdata.key_virt = ctx->key + ctx->adata.keylen_pad;
+ else
+ ctx->cdata.key_dma = ctx->key_dma + ctx->adata.keylen_pad;
+
+ ctx->adata.key_inline = !!(inl_mask & 1);
+ ctx->cdata.key_inline = !!(inl_mask & 2);
+
+ cnstr_shdsc_aead_givencap(ctx->sh_desc_enc, &ctx->cdata, &ctx->adata,
+ ivsize, ctx->authsize, is_rfc3686, nonce,
+ ctx1_iv_off, true);
+
+skip_givenc:
+ return 0;
+}
+
+static int aead_setauthsize(struct crypto_aead *authenc, unsigned int authsize)
+{
+ struct caam_ctx *ctx = crypto_aead_ctx(authenc);
+
+ ctx->authsize = authsize;
+ aead_set_sh_desc(authenc);
+
+ return 0;
+}
+
+static int aead_setkey(struct crypto_aead *aead, const u8 *key,
+ unsigned int keylen)
+{
+ struct caam_ctx *ctx = crypto_aead_ctx(aead);
+ struct device *jrdev = ctx->jrdev;
+ struct crypto_authenc_keys keys;
+ int ret = 0;
+
+ if (crypto_authenc_extractkeys(&keys, key, keylen) != 0)
+ goto badkey;
+
+#ifdef DEBUG
+ dev_err(jrdev, "keylen %d enckeylen %d authkeylen %d\n",
+ keys.authkeylen + keys.enckeylen, keys.enckeylen,
+ keys.authkeylen);
+ print_hex_dump(KERN_ERR, "key in @" __stringify(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, key, keylen, 1);
+#endif
+
+ ret = gen_split_key(jrdev, ctx->key, &ctx->adata, keys.authkey,
+ keys.authkeylen, CAAM_MAX_KEY_SIZE -
+ keys.enckeylen);
+ if (ret)
+ goto badkey;
+
+ /* postpend encryption key to auth split key */
+ memcpy(ctx->key + ctx->adata.keylen_pad, keys.enckey, keys.enckeylen);
+ dma_sync_single_for_device(jrdev, ctx->key_dma, ctx->adata.keylen_pad +
+ keys.enckeylen, DMA_TO_DEVICE);
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ctx.key@" __stringify(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, ctx->key,
+ ctx->adata.keylen_pad + keys.enckeylen, 1);
+#endif
+
+ ctx->cdata.keylen = keys.enckeylen;
+
+ ret = aead_set_sh_desc(aead);
+ if (ret)
+ goto badkey;
+
+ /* Now update the driver contexts with the new shared descriptor */
+ if (ctx->drv_ctx[ENCRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[ENCRYPT],
+ ctx->sh_desc_enc);
+ if (ret) {
+ dev_err(jrdev, "driver enc context update failed\n");
+ goto badkey;
+ }
+ }
+
+ if (ctx->drv_ctx[DECRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[DECRYPT],
+ ctx->sh_desc_dec);
+ if (ret) {
+ dev_err(jrdev, "driver dec context update failed\n");
+ goto badkey;
+ }
+ }
+
+ return ret;
+badkey:
+ crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN);
+ return -EINVAL;
+}
+
+static int ablkcipher_setkey(struct crypto_ablkcipher *ablkcipher,
+ const u8 *key, unsigned int keylen)
+{
+ struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher);
+ struct crypto_tfm *tfm = crypto_ablkcipher_tfm(ablkcipher);
+ const char *alg_name = crypto_tfm_alg_name(tfm);
+ struct device *jrdev = ctx->jrdev;
+ unsigned int ivsize = crypto_ablkcipher_ivsize(ablkcipher);
+ u32 ctx1_iv_off = 0;
+ const bool ctr_mode = ((ctx->cdata.algtype & OP_ALG_AAI_MASK) ==
+ OP_ALG_AAI_CTR_MOD128);
+ const bool is_rfc3686 = (ctr_mode && strstr(alg_name, "rfc3686"));
+ int ret = 0;
+
+ memcpy(ctx->key, key, keylen);
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "key in @" __stringify(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, key, keylen, 1);
+#endif
+ /*
+ * AES-CTR needs to load IV in CONTEXT1 reg
+ * at an offset of 128bits (16bytes)
+ * CONTEXT1[255:128] = IV
+ */
+ if (ctr_mode)
+ ctx1_iv_off = 16;
+
+ /*
+ * RFC3686 specific:
+ * | CONTEXT1[255:128] = {NONCE, IV, COUNTER}
+ * | *key = {KEY, NONCE}
+ */
+ if (is_rfc3686) {
+ ctx1_iv_off = 16 + CTR_RFC3686_NONCE_SIZE;
+ keylen -= CTR_RFC3686_NONCE_SIZE;
+ }
+
+ dma_sync_single_for_device(jrdev, ctx->key_dma, keylen, DMA_TO_DEVICE);
+ ctx->cdata.keylen = keylen;
+ ctx->cdata.key_virt = ctx->key;
+ ctx->cdata.key_inline = true;
+
+ /* ablkcipher encrypt, decrypt, givencrypt shared descriptors */
+ cnstr_shdsc_ablkcipher_encap(ctx->sh_desc_enc, &ctx->cdata, ivsize,
+ is_rfc3686, ctx1_iv_off);
+ cnstr_shdsc_ablkcipher_decap(ctx->sh_desc_dec, &ctx->cdata, ivsize,
+ is_rfc3686, ctx1_iv_off);
+ cnstr_shdsc_ablkcipher_givencap(ctx->sh_desc_givenc, &ctx->cdata,
+ ivsize, is_rfc3686, ctx1_iv_off);
+
+ /* Now update the driver contexts with the new shared descriptor */
+ if (ctx->drv_ctx[ENCRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[ENCRYPT],
+ ctx->sh_desc_enc);
+ if (ret) {
+ dev_err(jrdev, "driver enc context update failed\n");
+ goto badkey;
+ }
+ }
+
+ if (ctx->drv_ctx[DECRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[DECRYPT],
+ ctx->sh_desc_dec);
+ if (ret) {
+ dev_err(jrdev, "driver dec context update failed\n");
+ goto badkey;
+ }
+ }
+
+ if (ctx->drv_ctx[GIVENCRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[GIVENCRYPT],
+ ctx->sh_desc_givenc);
+ if (ret) {
+ dev_err(jrdev, "driver givenc context update failed\n");
+ goto badkey;
+ }
+ }
+
+ return ret;
+badkey:
+ crypto_ablkcipher_set_flags(ablkcipher, CRYPTO_TFM_RES_BAD_KEY_LEN);
+ return -EINVAL;
+}
+
+static int xts_ablkcipher_setkey(struct crypto_ablkcipher *ablkcipher,
+ const u8 *key, unsigned int keylen)
+{
+ struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher);
+ struct device *jrdev = ctx->jrdev;
+ int ret = 0;
+
+ if (keylen != 2 * AES_MIN_KEY_SIZE && keylen != 2 * AES_MAX_KEY_SIZE) {
+ crypto_ablkcipher_set_flags(ablkcipher,
+ CRYPTO_TFM_RES_BAD_KEY_LEN);
+ dev_err(jrdev, "key size mismatch\n");
+ return -EINVAL;
+ }
+
+ memcpy(ctx->key, key, keylen);
+ dma_sync_single_for_device(jrdev, ctx->key_dma, keylen, DMA_TO_DEVICE);
+ ctx->cdata.keylen = keylen;
+ ctx->cdata.key_virt = ctx->key;
+ ctx->cdata.key_inline = true;
+
+ /* xts ablkcipher encrypt, decrypt shared descriptors */
+ cnstr_shdsc_xts_ablkcipher_encap(ctx->sh_desc_enc, &ctx->cdata);
+ cnstr_shdsc_xts_ablkcipher_decap(ctx->sh_desc_dec, &ctx->cdata);
+
+ /* Now update the driver contexts with the new shared descriptor */
+ if (ctx->drv_ctx[ENCRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[ENCRYPT],
+ ctx->sh_desc_enc);
+ if (ret) {
+ dev_err(jrdev, "driver enc context update failed\n");
+ goto badkey;
+ }
+ }
+
+ if (ctx->drv_ctx[DECRYPT]) {
+ ret = caam_drv_ctx_update(ctx->drv_ctx[DECRYPT],
+ ctx->sh_desc_dec);
+ if (ret) {
+ dev_err(jrdev, "driver dec context update failed\n");
+ goto badkey;
+ }
+ }
+
+ return ret;
+badkey:
+ crypto_ablkcipher_set_flags(ablkcipher, CRYPTO_TFM_RES_BAD_KEY_LEN);
+ return 0;
+}
+
+/*
+ * aead_edesc - s/w-extended aead descriptor
+ * @src_nents: number of segments in input scatterlist
+ * @dst_nents: number of segments in output scatterlist
+ * @iv_dma: dma address of iv for checking continuity and link table
+ * @qm_sg_bytes: length of dma mapped h/w link table
+ * @qm_sg_dma: bus physical mapped address of h/w link table
+ * @assoclen_dma: bus physical mapped address of req->assoclen
+ * @drv_req: driver-specific request structure
+ * @sgt: the h/w link table
+ */
+struct aead_edesc {
+ int src_nents;
+ int dst_nents;
+ dma_addr_t iv_dma;
+ int qm_sg_bytes;
+ dma_addr_t qm_sg_dma;
+ dma_addr_t assoclen_dma;
+ struct caam_drv_req drv_req;
+ struct qm_sg_entry sgt[0];
+};
+
+/*
+ * ablkcipher_edesc - s/w-extended ablkcipher descriptor
+ * @src_nents: number of segments in input scatterlist
+ * @dst_nents: number of segments in output scatterlist
+ * @iv_dma: dma address of iv for checking continuity and link table
+ * @qm_sg_bytes: length of dma mapped h/w link table
+ * @qm_sg_dma: bus physical mapped address of h/w link table
+ * @drv_req: driver-specific request structure
+ * @sgt: the h/w link table
+ */
+struct ablkcipher_edesc {
+ int src_nents;
+ int dst_nents;
+ dma_addr_t iv_dma;
+ int qm_sg_bytes;
+ dma_addr_t qm_sg_dma;
+ struct caam_drv_req drv_req;
+ struct qm_sg_entry sgt[0];
+};
+
+static struct caam_drv_ctx *get_drv_ctx(struct caam_ctx *ctx,
+ enum optype type)
+{
+ /*
+ * This function is called on the fast path with values of 'type'
+ * known at compile time. Invalid arguments are not expected and
+ * thus no checks are made.
+ */
+ struct caam_drv_ctx *drv_ctx = ctx->drv_ctx[type];
+ u32 *desc;
+
+ if (unlikely(!drv_ctx)) {
+ spin_lock(&ctx->lock);
+
+ /* Read again to check if some other core init drv_ctx */
+ drv_ctx = ctx->drv_ctx[type];
+ if (!drv_ctx) {
+ int cpu;
+
+ if (type == ENCRYPT)
+ desc = ctx->sh_desc_enc;
+ else if (type == DECRYPT)
+ desc = ctx->sh_desc_dec;
+ else /* (type == GIVENCRYPT) */
+ desc = ctx->sh_desc_givenc;
+
+ cpu = smp_processor_id();
+ drv_ctx = caam_drv_ctx_init(ctx->qidev, &cpu, desc);
+ if (likely(!IS_ERR_OR_NULL(drv_ctx)))
+ drv_ctx->op_type = type;
+
+ ctx->drv_ctx[type] = drv_ctx;
+ }
+
+ spin_unlock(&ctx->lock);
+ }
+
+ return drv_ctx;
+}
+
+static void caam_unmap(struct device *dev, struct scatterlist *src,
+ struct scatterlist *dst, int src_nents,
+ int dst_nents, dma_addr_t iv_dma, int ivsize,
+ enum optype op_type, dma_addr_t qm_sg_dma,
+ int qm_sg_bytes)
+{
+ if (dst != src) {
+ if (src_nents)
+ dma_unmap_sg(dev, src, src_nents, DMA_TO_DEVICE);
+ dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE);
+ } else {
+ dma_unmap_sg(dev, src, src_nents, DMA_BIDIRECTIONAL);
+ }
+
+ if (iv_dma)
+ dma_unmap_single(dev, iv_dma, ivsize,
+ op_type == GIVENCRYPT ? DMA_FROM_DEVICE :
+ DMA_TO_DEVICE);
+ if (qm_sg_bytes)
+ dma_unmap_single(dev, qm_sg_dma, qm_sg_bytes, DMA_TO_DEVICE);
+}
+
+static void aead_unmap(struct device *dev,
+ struct aead_edesc *edesc,
+ struct aead_request *req)
+{
+ struct crypto_aead *aead = crypto_aead_reqtfm(req);
+ int ivsize = crypto_aead_ivsize(aead);
+
+ caam_unmap(dev, req->src, req->dst, edesc->src_nents, edesc->dst_nents,
+ edesc->iv_dma, ivsize, edesc->drv_req.drv_ctx->op_type,
+ edesc->qm_sg_dma, edesc->qm_sg_bytes);
+ dma_unmap_single(dev, edesc->assoclen_dma, 4, DMA_TO_DEVICE);
+}
+
+static void ablkcipher_unmap(struct device *dev,
+ struct ablkcipher_edesc *edesc,
+ struct ablkcipher_request *req)
+{
+ struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req);
+ int ivsize = crypto_ablkcipher_ivsize(ablkcipher);
+
+ caam_unmap(dev, req->src, req->dst, edesc->src_nents, edesc->dst_nents,
+ edesc->iv_dma, ivsize, edesc->drv_req.drv_ctx->op_type,
+ edesc->qm_sg_dma, edesc->qm_sg_bytes);
+}
+
+static void aead_done(struct caam_drv_req *drv_req, u32 status)
+{
+ struct device *qidev;
+ struct aead_edesc *edesc;
+ struct aead_request *aead_req = drv_req->app_ctx;
+ struct crypto_aead *aead = crypto_aead_reqtfm(aead_req);
+ struct caam_ctx *caam_ctx = crypto_aead_ctx(aead);
+ int ecode = 0;
+
+ qidev = caam_ctx->qidev;
+
+ if (unlikely(status)) {
+ caam_jr_strstatus(qidev, status);
+ ecode = -EIO;
+ }
+
+ edesc = container_of(drv_req, typeof(*edesc), drv_req);
+ aead_unmap(qidev, edesc, aead_req);
+
+ aead_request_complete(aead_req, ecode);
+ qi_cache_free(edesc);
+}
+
+/*
+ * allocate and map the aead extended descriptor
+ */
+static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
+ bool encrypt)
+{
+ struct crypto_aead *aead = crypto_aead_reqtfm(req);
+ struct caam_ctx *ctx = crypto_aead_ctx(aead);
+ struct caam_aead_alg *alg = container_of(crypto_aead_alg(aead),
+ typeof(*alg), aead);
+ struct device *qidev = ctx->qidev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ? GFP_KERNEL : GFP_ATOMIC;
+ int src_nents, mapped_src_nents, dst_nents = 0, mapped_dst_nents = 0;
+ struct aead_edesc *edesc;
+ dma_addr_t qm_sg_dma, iv_dma = 0;
+ int ivsize = 0;
+ unsigned int authsize = ctx->authsize;
+ int qm_sg_index = 0, qm_sg_ents = 0, qm_sg_bytes;
+ int in_len, out_len;
+ struct qm_sg_entry *sg_table, *fd_sgt;
+ struct caam_drv_ctx *drv_ctx;
+ enum optype op_type = encrypt ? ENCRYPT : DECRYPT;
+
+ drv_ctx = get_drv_ctx(ctx, op_type);
+ if (unlikely(IS_ERR_OR_NULL(drv_ctx)))
+ return (struct aead_edesc *)drv_ctx;
+
+ /* allocate space for base edesc and hw desc commands, link tables */
+ edesc = qi_cache_alloc(GFP_DMA | flags);
+ if (unlikely(!edesc)) {
+ dev_err(qidev, "could not allocate extended descriptor\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ if (likely(req->src == req->dst)) {
+ src_nents = sg_nents_for_len(req->src, req->assoclen +
+ req->cryptlen +
+ (encrypt ? authsize : 0));
+ if (unlikely(src_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in src S/G\n",
+ req->assoclen + req->cryptlen +
+ (encrypt ? authsize : 0));
+ qi_cache_free(edesc);
+ return ERR_PTR(src_nents);
+ }
+
+ mapped_src_nents = dma_map_sg(qidev, req->src, src_nents,
+ DMA_BIDIRECTIONAL);
+ if (unlikely(!mapped_src_nents)) {
+ dev_err(qidev, "unable to map source\n");
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+ } else {
+ src_nents = sg_nents_for_len(req->src, req->assoclen +
+ req->cryptlen);
+ if (unlikely(src_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in src S/G\n",
+ req->assoclen + req->cryptlen);
+ qi_cache_free(edesc);
+ return ERR_PTR(src_nents);
+ }
+
+ dst_nents = sg_nents_for_len(req->dst, req->assoclen +
+ req->cryptlen +
+ (encrypt ? authsize :
+ (-authsize)));
+ if (unlikely(dst_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in dst S/G\n",
+ req->assoclen + req->cryptlen +
+ (encrypt ? authsize : (-authsize)));
+ qi_cache_free(edesc);
+ return ERR_PTR(dst_nents);
+ }
+
+ if (src_nents) {
+ mapped_src_nents = dma_map_sg(qidev, req->src,
+ src_nents, DMA_TO_DEVICE);
+ if (unlikely(!mapped_src_nents)) {
+ dev_err(qidev, "unable to map source\n");
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+ } else {
+ mapped_src_nents = 0;
+ }
+
+ mapped_dst_nents = dma_map_sg(qidev, req->dst, dst_nents,
+ DMA_FROM_DEVICE);
+ if (unlikely(!mapped_dst_nents)) {
+ dev_err(qidev, "unable to map destination\n");
+ dma_unmap_sg(qidev, req->src, src_nents, DMA_TO_DEVICE);
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+ }
+
+ if ((alg->caam.rfc3686 && encrypt) || !alg->caam.geniv) {
+ ivsize = crypto_aead_ivsize(aead);
+ iv_dma = dma_map_single(qidev, req->iv, ivsize, DMA_TO_DEVICE);
+ if (dma_mapping_error(qidev, iv_dma)) {
+ dev_err(qidev, "unable to map IV\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents,
+ dst_nents, 0, 0, op_type, 0, 0);
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+ }
+
+ /*
+ * Create S/G table: req->assoclen, [IV,] req->src [, req->dst].
+ * Input is not contiguous.
+ */
+ qm_sg_ents = 1 + !!ivsize + mapped_src_nents +
+ (mapped_dst_nents > 1 ? mapped_dst_nents : 0);
+ sg_table = &edesc->sgt[0];
+ qm_sg_bytes = qm_sg_ents * sizeof(*sg_table);
+
+ edesc->src_nents = src_nents;
+ edesc->dst_nents = dst_nents;
+ edesc->iv_dma = iv_dma;
+ edesc->drv_req.app_ctx = req;
+ edesc->drv_req.cbk = aead_done;
+ edesc->drv_req.drv_ctx = drv_ctx;
+
+ edesc->assoclen_dma = dma_map_single(qidev, &req->assoclen, 4,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(qidev, edesc->assoclen_dma)) {
+ dev_err(qidev, "unable to map assoclen\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents,
+ iv_dma, ivsize, op_type, 0, 0);
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ dma_to_qm_sg_one(sg_table, edesc->assoclen_dma, 4, 0);
+ qm_sg_index++;
+ if (ivsize) {
+ dma_to_qm_sg_one(sg_table + qm_sg_index, iv_dma, ivsize, 0);
+ qm_sg_index++;
+ }
+ sg_to_qm_sg_last(req->src, mapped_src_nents, sg_table + qm_sg_index, 0);
+ qm_sg_index += mapped_src_nents;
+
+ if (mapped_dst_nents > 1)
+ sg_to_qm_sg_last(req->dst, mapped_dst_nents, sg_table +
+ qm_sg_index, 0);
+
+ qm_sg_dma = dma_map_single(qidev, sg_table, qm_sg_bytes, DMA_TO_DEVICE);
+ if (dma_mapping_error(qidev, qm_sg_dma)) {
+ dev_err(qidev, "unable to map S/G table\n");
+ dma_unmap_single(qidev, edesc->assoclen_dma, 4, DMA_TO_DEVICE);
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents,
+ iv_dma, ivsize, op_type, 0, 0);
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ edesc->qm_sg_dma = qm_sg_dma;
+ edesc->qm_sg_bytes = qm_sg_bytes;
+
+ out_len = req->assoclen + req->cryptlen +
+ (encrypt ? ctx->authsize : (-ctx->authsize));
+ in_len = 4 + ivsize + req->assoclen + req->cryptlen;
+
+ fd_sgt = &edesc->drv_req.fd_sgt[0];
+ dma_to_qm_sg_one_last_ext(&fd_sgt[1], qm_sg_dma, in_len, 0);
+
+ if (req->dst == req->src) {
+ if (mapped_src_nents == 1)
+ dma_to_qm_sg_one(&fd_sgt[0], sg_dma_address(req->src),
+ out_len, 0);
+ else
+ dma_to_qm_sg_one_ext(&fd_sgt[0], qm_sg_dma +
+ (1 + !!ivsize) * sizeof(*sg_table),
+ out_len, 0);
+ } else if (mapped_dst_nents == 1) {
+ dma_to_qm_sg_one(&fd_sgt[0], sg_dma_address(req->dst), out_len,
+ 0);
+ } else {
+ dma_to_qm_sg_one_ext(&fd_sgt[0], qm_sg_dma + sizeof(*sg_table) *
+ qm_sg_index, out_len, 0);
+ }
+
+ return edesc;
+}
+
+static inline int aead_crypt(struct aead_request *req, bool encrypt)
+{
+ struct aead_edesc *edesc;
+ struct crypto_aead *aead = crypto_aead_reqtfm(req);
+ struct caam_ctx *ctx = crypto_aead_ctx(aead);
+ int ret;
+
+ if (unlikely(caam_congested))
+ return -EAGAIN;
+
+ /* allocate extended descriptor */
+ edesc = aead_edesc_alloc(req, encrypt);
+ if (IS_ERR_OR_NULL(edesc))
+ return PTR_ERR(edesc);
+
+ /* Create and submit job descriptor */
+ ret = caam_qi_enqueue(ctx->qidev, &edesc->drv_req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ } else {
+ aead_unmap(ctx->qidev, edesc, req);
+ qi_cache_free(edesc);
+ }
+
+ return ret;
+}
+
+static int aead_encrypt(struct aead_request *req)
+{
+ return aead_crypt(req, true);
+}
+
+static int aead_decrypt(struct aead_request *req)
+{
+ return aead_crypt(req, false);
+}
+
+static void ablkcipher_done(struct caam_drv_req *drv_req, u32 status)
+{
+ struct ablkcipher_edesc *edesc;
+ struct ablkcipher_request *req = drv_req->app_ctx;
+ struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req);
+ struct caam_ctx *caam_ctx = crypto_ablkcipher_ctx(ablkcipher);
+ struct device *qidev = caam_ctx->qidev;
+#ifdef DEBUG
+ int ivsize = crypto_ablkcipher_ivsize(ablkcipher);
+
+ dev_err(qidev, "%s %d: status 0x%x\n", __func__, __LINE__, status);
+#endif
+
+ edesc = container_of(drv_req, typeof(*edesc), drv_req);
+
+ if (status)
+ caam_jr_strstatus(qidev, status);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "dstiv @" __stringify(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, req->info,
+ edesc->src_nents > 1 ? 100 : ivsize, 1);
+ dbg_dump_sg(KERN_ERR, "dst @" __stringify(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, req->dst,
+ edesc->dst_nents > 1 ? 100 : req->nbytes, 1);
+#endif
+
+ ablkcipher_unmap(qidev, edesc, req);
+ qi_cache_free(edesc);
+
+ ablkcipher_request_complete(req, status);
+}
+
+static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request
+ *req, bool encrypt)
+{
+ struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req);
+ struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher);
+ struct device *qidev = ctx->qidev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ?
+ GFP_KERNEL : GFP_ATOMIC;
+ int src_nents, mapped_src_nents, dst_nents = 0, mapped_dst_nents = 0;
+ struct ablkcipher_edesc *edesc;
+ dma_addr_t iv_dma;
+ bool in_contig;
+ int ivsize = crypto_ablkcipher_ivsize(ablkcipher);
+ int dst_sg_idx, qm_sg_ents;
+ struct qm_sg_entry *sg_table, *fd_sgt;
+ struct caam_drv_ctx *drv_ctx;
+ enum optype op_type = encrypt ? ENCRYPT : DECRYPT;
+
+ drv_ctx = get_drv_ctx(ctx, op_type);
+ if (unlikely(IS_ERR_OR_NULL(drv_ctx)))
+ return (struct ablkcipher_edesc *)drv_ctx;
+
+ src_nents = sg_nents_for_len(req->src, req->nbytes);
+ if (unlikely(src_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in src S/G\n",
+ req->nbytes);
+ return ERR_PTR(src_nents);
+ }
+
+ if (unlikely(req->src != req->dst)) {
+ dst_nents = sg_nents_for_len(req->dst, req->nbytes);
+ if (unlikely(dst_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in dst S/G\n",
+ req->nbytes);
+ return ERR_PTR(dst_nents);
+ }
+
+ mapped_src_nents = dma_map_sg(qidev, req->src, src_nents,
+ DMA_TO_DEVICE);
+ if (unlikely(!mapped_src_nents)) {
+ dev_err(qidev, "unable to map source\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ mapped_dst_nents = dma_map_sg(qidev, req->dst, dst_nents,
+ DMA_FROM_DEVICE);
+ if (unlikely(!mapped_dst_nents)) {
+ dev_err(qidev, "unable to map destination\n");
+ dma_unmap_sg(qidev, req->src, src_nents, DMA_TO_DEVICE);
+ return ERR_PTR(-ENOMEM);
+ }
+ } else {
+ mapped_src_nents = dma_map_sg(qidev, req->src, src_nents,
+ DMA_BIDIRECTIONAL);
+ if (unlikely(!mapped_src_nents)) {
+ dev_err(qidev, "unable to map source\n");
+ return ERR_PTR(-ENOMEM);
+ }
+ }
+
+ iv_dma = dma_map_single(qidev, req->info, ivsize, DMA_TO_DEVICE);
+ if (dma_mapping_error(qidev, iv_dma)) {
+ dev_err(qidev, "unable to map IV\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, 0,
+ 0, 0, 0, 0);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ if (mapped_src_nents == 1 &&
+ iv_dma + ivsize == sg_dma_address(req->src)) {
+ in_contig = true;
+ qm_sg_ents = 0;
+ } else {
+ in_contig = false;
+ qm_sg_ents = 1 + mapped_src_nents;
+ }
+ dst_sg_idx = qm_sg_ents;
+
+ /* allocate space for base edesc and link tables */
+ edesc = qi_cache_alloc(GFP_DMA | flags);
+ if (unlikely(!edesc)) {
+ dev_err(qidev, "could not allocate extended descriptor\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents,
+ iv_dma, ivsize, op_type, 0, 0);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ edesc->src_nents = src_nents;
+ edesc->dst_nents = dst_nents;
+ edesc->iv_dma = iv_dma;
+ qm_sg_ents += mapped_dst_nents > 1 ? mapped_dst_nents : 0;
+ sg_table = &edesc->sgt[0];
+ edesc->qm_sg_bytes = qm_sg_ents * sizeof(*sg_table);
+ edesc->drv_req.app_ctx = req;
+ edesc->drv_req.cbk = ablkcipher_done;
+ edesc->drv_req.drv_ctx = drv_ctx;
+
+ if (!in_contig) {
+ dma_to_qm_sg_one(sg_table, iv_dma, ivsize, 0);
+ sg_to_qm_sg_last(req->src, mapped_src_nents, sg_table + 1, 0);
+ }
+
+ if (mapped_dst_nents > 1)
+ sg_to_qm_sg_last(req->dst, mapped_dst_nents, sg_table +
+ dst_sg_idx, 0);
+
+ edesc->qm_sg_dma = dma_map_single(qidev, sg_table, edesc->qm_sg_bytes,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(qidev, edesc->qm_sg_dma)) {
+ dev_err(qidev, "unable to map S/G table\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents,
+ iv_dma, ivsize, op_type, 0, 0);
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ fd_sgt = &edesc->drv_req.fd_sgt[0];
+
+ if (!in_contig)
+ dma_to_qm_sg_one_last_ext(&fd_sgt[1], edesc->qm_sg_dma,
+ ivsize + req->nbytes, 0);
+ else
+ dma_to_qm_sg_one_last(&fd_sgt[1], iv_dma, ivsize + req->nbytes,
+ 0);
+
+ if (req->src == req->dst) {
+ if (!in_contig)
+ dma_to_qm_sg_one_ext(&fd_sgt[0], edesc->qm_sg_dma +
+ sizeof(*sg_table), req->nbytes, 0);
+ else
+ dma_to_qm_sg_one(&fd_sgt[0], sg_dma_address(req->src),
+ req->nbytes, 0);
+ } else if (mapped_dst_nents > 1) {
+ dma_to_qm_sg_one_ext(&fd_sgt[0], edesc->qm_sg_dma + dst_sg_idx *
+ sizeof(*sg_table), req->nbytes, 0);
+ } else {
+ dma_to_qm_sg_one(&fd_sgt[0], sg_dma_address(req->dst),
+ req->nbytes, 0);
+ }
+
+ return edesc;
+}
+
+static struct ablkcipher_edesc *ablkcipher_giv_edesc_alloc(
+ struct skcipher_givcrypt_request *creq)
+{
+ struct ablkcipher_request *req = &creq->creq;
+ struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req);
+ struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher);
+ struct device *qidev = ctx->qidev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ?
+ GFP_KERNEL : GFP_ATOMIC;
+ int src_nents, mapped_src_nents, dst_nents, mapped_dst_nents;
+ struct ablkcipher_edesc *edesc;
+ dma_addr_t iv_dma;
+ bool out_contig;
+ int ivsize = crypto_ablkcipher_ivsize(ablkcipher);
+ struct qm_sg_entry *sg_table, *fd_sgt;
+ int dst_sg_idx, qm_sg_ents;
+ struct caam_drv_ctx *drv_ctx;
+
+ drv_ctx = get_drv_ctx(ctx, GIVENCRYPT);
+ if (unlikely(IS_ERR_OR_NULL(drv_ctx)))
+ return (struct ablkcipher_edesc *)drv_ctx;
+
+ src_nents = sg_nents_for_len(req->src, req->nbytes);
+ if (unlikely(src_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in src S/G\n",
+ req->nbytes);
+ return ERR_PTR(src_nents);
+ }
+
+ if (unlikely(req->src != req->dst)) {
+ dst_nents = sg_nents_for_len(req->dst, req->nbytes);
+ if (unlikely(dst_nents < 0)) {
+ dev_err(qidev, "Insufficient bytes (%d) in dst S/G\n",
+ req->nbytes);
+ return ERR_PTR(dst_nents);
+ }
+
+ mapped_src_nents = dma_map_sg(qidev, req->src, src_nents,
+ DMA_TO_DEVICE);
+ if (unlikely(!mapped_src_nents)) {
+ dev_err(qidev, "unable to map source\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ mapped_dst_nents = dma_map_sg(qidev, req->dst, dst_nents,
+ DMA_FROM_DEVICE);
+ if (unlikely(!mapped_dst_nents)) {
+ dev_err(qidev, "unable to map destination\n");
+ dma_unmap_sg(qidev, req->src, src_nents, DMA_TO_DEVICE);
+ return ERR_PTR(-ENOMEM);
+ }
+ } else {
+ mapped_src_nents = dma_map_sg(qidev, req->src, src_nents,
+ DMA_BIDIRECTIONAL);
+ if (unlikely(!mapped_src_nents)) {
+ dev_err(qidev, "unable to map source\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ dst_nents = src_nents;
+ mapped_dst_nents = src_nents;
+ }
+
+ iv_dma = dma_map_single(qidev, creq->giv, ivsize, DMA_FROM_DEVICE);
+ if (dma_mapping_error(qidev, iv_dma)) {
+ dev_err(qidev, "unable to map IV\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents, 0,
+ 0, 0, 0, 0);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ qm_sg_ents = mapped_src_nents > 1 ? mapped_src_nents : 0;
+ dst_sg_idx = qm_sg_ents;
+ if (mapped_dst_nents == 1 &&
+ iv_dma + ivsize == sg_dma_address(req->dst)) {
+ out_contig = true;
+ } else {
+ out_contig = false;
+ qm_sg_ents += 1 + mapped_dst_nents;
+ }
+
+ /* allocate space for base edesc and link tables */
+ edesc = qi_cache_alloc(GFP_DMA | flags);
+ if (!edesc) {
+ dev_err(qidev, "could not allocate extended descriptor\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents,
+ iv_dma, ivsize, GIVENCRYPT, 0, 0);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ edesc->src_nents = src_nents;
+ edesc->dst_nents = dst_nents;
+ edesc->iv_dma = iv_dma;
+ sg_table = &edesc->sgt[0];
+ edesc->qm_sg_bytes = qm_sg_ents * sizeof(*sg_table);
+ edesc->drv_req.app_ctx = req;
+ edesc->drv_req.cbk = ablkcipher_done;
+ edesc->drv_req.drv_ctx = drv_ctx;
+
+ if (mapped_src_nents > 1)
+ sg_to_qm_sg_last(req->src, mapped_src_nents, sg_table, 0);
+
+ if (!out_contig) {
+ dma_to_qm_sg_one(sg_table + dst_sg_idx, iv_dma, ivsize, 0);
+ sg_to_qm_sg_last(req->dst, mapped_dst_nents, sg_table +
+ dst_sg_idx + 1, 0);
+ }
+
+ edesc->qm_sg_dma = dma_map_single(qidev, sg_table, edesc->qm_sg_bytes,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(qidev, edesc->qm_sg_dma)) {
+ dev_err(qidev, "unable to map S/G table\n");
+ caam_unmap(qidev, req->src, req->dst, src_nents, dst_nents,
+ iv_dma, ivsize, GIVENCRYPT, 0, 0);
+ qi_cache_free(edesc);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ fd_sgt = &edesc->drv_req.fd_sgt[0];
+
+ if (mapped_src_nents > 1)
+ dma_to_qm_sg_one_ext(&fd_sgt[1], edesc->qm_sg_dma, req->nbytes,
+ 0);
+ else
+ dma_to_qm_sg_one(&fd_sgt[1], sg_dma_address(req->src),
+ req->nbytes, 0);
+
+ if (!out_contig)
+ dma_to_qm_sg_one_ext(&fd_sgt[0], edesc->qm_sg_dma + dst_sg_idx *
+ sizeof(*sg_table), ivsize + req->nbytes,
+ 0);
+ else
+ dma_to_qm_sg_one(&fd_sgt[0], sg_dma_address(req->dst),
+ ivsize + req->nbytes, 0);
+
+ return edesc;
+}
+
+static inline int ablkcipher_crypt(struct ablkcipher_request *req, bool encrypt)
+{
+ struct ablkcipher_edesc *edesc;
+ struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req);
+ struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher);
+ int ret;
+
+ if (unlikely(caam_congested))
+ return -EAGAIN;
+
+ /* allocate extended descriptor */
+ edesc = ablkcipher_edesc_alloc(req, encrypt);
+ if (IS_ERR(edesc))
+ return PTR_ERR(edesc);
+
+ ret = caam_qi_enqueue(ctx->qidev, &edesc->drv_req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ } else {
+ ablkcipher_unmap(ctx->qidev, edesc, req);
+ qi_cache_free(edesc);
+ }
+
+ return ret;
+}
+
+static int ablkcipher_encrypt(struct ablkcipher_request *req)
+{
+ return ablkcipher_crypt(req, true);
+}
+
+static int ablkcipher_decrypt(struct ablkcipher_request *req)
+{
+ return ablkcipher_crypt(req, false);
+}
+
+static int ablkcipher_givencrypt(struct skcipher_givcrypt_request *creq)
+{
+ struct ablkcipher_request *req = &creq->creq;
+ struct ablkcipher_edesc *edesc;
+ struct crypto_ablkcipher *ablkcipher = crypto_ablkcipher_reqtfm(req);
+ struct caam_ctx *ctx = crypto_ablkcipher_ctx(ablkcipher);
+ int ret;
+
+ if (unlikely(caam_congested))
+ return -EAGAIN;
+
+ /* allocate extended descriptor */
+ edesc = ablkcipher_giv_edesc_alloc(creq);
+ if (IS_ERR(edesc))
+ return PTR_ERR(edesc);
+
+ ret = caam_qi_enqueue(ctx->qidev, &edesc->drv_req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ } else {
+ ablkcipher_unmap(ctx->qidev, edesc, req);
+ qi_cache_free(edesc);
+ }
+
+ return ret;
+}
+
+#define template_ablkcipher template_u.ablkcipher
+struct caam_alg_template {
+ char name[CRYPTO_MAX_ALG_NAME];
+ char driver_name[CRYPTO_MAX_ALG_NAME];
+ unsigned int blocksize;
+ u32 type;
+ union {
+ struct ablkcipher_alg ablkcipher;
+ } template_u;
+ u32 class1_alg_type;
+ u32 class2_alg_type;
+};
+
+static struct caam_alg_template driver_algs[] = {
+ /* ablkcipher descriptor */
+ {
+ .name = "cbc(aes)",
+ .driver_name = "cbc-aes-caam-qi",
+ .blocksize = AES_BLOCK_SIZE,
+ .type = CRYPTO_ALG_TYPE_GIVCIPHER,
+ .template_ablkcipher = {
+ .setkey = ablkcipher_setkey,
+ .encrypt = ablkcipher_encrypt,
+ .decrypt = ablkcipher_decrypt,
+ .givencrypt = ablkcipher_givencrypt,
+ .geniv = "<built-in>",
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ },
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ },
+ {
+ .name = "cbc(des3_ede)",
+ .driver_name = "cbc-3des-caam-qi",
+ .blocksize = DES3_EDE_BLOCK_SIZE,
+ .type = CRYPTO_ALG_TYPE_GIVCIPHER,
+ .template_ablkcipher = {
+ .setkey = ablkcipher_setkey,
+ .encrypt = ablkcipher_encrypt,
+ .decrypt = ablkcipher_decrypt,
+ .givencrypt = ablkcipher_givencrypt,
+ .geniv = "<built-in>",
+ .min_keysize = DES3_EDE_KEY_SIZE,
+ .max_keysize = DES3_EDE_KEY_SIZE,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ },
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ },
+ {
+ .name = "cbc(des)",
+ .driver_name = "cbc-des-caam-qi",
+ .blocksize = DES_BLOCK_SIZE,
+ .type = CRYPTO_ALG_TYPE_GIVCIPHER,
+ .template_ablkcipher = {
+ .setkey = ablkcipher_setkey,
+ .encrypt = ablkcipher_encrypt,
+ .decrypt = ablkcipher_decrypt,
+ .givencrypt = ablkcipher_givencrypt,
+ .geniv = "<built-in>",
+ .min_keysize = DES_KEY_SIZE,
+ .max_keysize = DES_KEY_SIZE,
+ .ivsize = DES_BLOCK_SIZE,
+ },
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ },
+ {
+ .name = "ctr(aes)",
+ .driver_name = "ctr-aes-caam-qi",
+ .blocksize = 1,
+ .type = CRYPTO_ALG_TYPE_ABLKCIPHER,
+ .template_ablkcipher = {
+ .setkey = ablkcipher_setkey,
+ .encrypt = ablkcipher_encrypt,
+ .decrypt = ablkcipher_decrypt,
+ .geniv = "chainiv",
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ },
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CTR_MOD128,
+ },
+ {
+ .name = "rfc3686(ctr(aes))",
+ .driver_name = "rfc3686-ctr-aes-caam-qi",
+ .blocksize = 1,
+ .type = CRYPTO_ALG_TYPE_GIVCIPHER,
+ .template_ablkcipher = {
+ .setkey = ablkcipher_setkey,
+ .encrypt = ablkcipher_encrypt,
+ .decrypt = ablkcipher_decrypt,
+ .givencrypt = ablkcipher_givencrypt,
+ .geniv = "<built-in>",
+ .min_keysize = AES_MIN_KEY_SIZE +
+ CTR_RFC3686_NONCE_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE +
+ CTR_RFC3686_NONCE_SIZE,
+ .ivsize = CTR_RFC3686_IV_SIZE,
+ },
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CTR_MOD128,
+ },
+ {
+ .name = "xts(aes)",
+ .driver_name = "xts-aes-caam-qi",
+ .blocksize = AES_BLOCK_SIZE,
+ .type = CRYPTO_ALG_TYPE_ABLKCIPHER,
+ .template_ablkcipher = {
+ .setkey = xts_ablkcipher_setkey,
+ .encrypt = ablkcipher_encrypt,
+ .decrypt = ablkcipher_decrypt,
+ .geniv = "eseqiv",
+ .min_keysize = 2 * AES_MIN_KEY_SIZE,
+ .max_keysize = 2 * AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ },
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_XTS,
+ },
+};
+
+static struct caam_aead_alg driver_aeads[] = {
+ /* single-pass ipsec_esp descriptor */
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(md5),cbc(aes))",
+ .cra_driver_name = "authenc-hmac-md5-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = MD5_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_MD5 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(md5),"
+ "cbc(aes)))",
+ .cra_driver_name = "echainiv-authenc-hmac-md5-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = MD5_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_MD5 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha1),cbc(aes))",
+ .cra_driver_name = "authenc-hmac-sha1-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA1_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA1 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha1),"
+ "cbc(aes)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha1-cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA1_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA1 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha224),cbc(aes))",
+ .cra_driver_name = "authenc-hmac-sha224-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA224_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA224 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha224),"
+ "cbc(aes)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha224-cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA224_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA224 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha256),cbc(aes))",
+ .cra_driver_name = "authenc-hmac-sha256-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA256_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA256 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha256),"
+ "cbc(aes)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha256-cbc-aes-"
+ "caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA256_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA256 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha384),cbc(aes))",
+ .cra_driver_name = "authenc-hmac-sha384-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA384_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA384 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha384),"
+ "cbc(aes)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha384-cbc-aes-"
+ "caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA384_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA384 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha512),cbc(aes))",
+ .cra_driver_name = "authenc-hmac-sha512-"
+ "cbc-aes-caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA512_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA512 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha512),"
+ "cbc(aes)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha512-cbc-aes-"
+ "caam-qi",
+ .cra_blocksize = AES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = AES_BLOCK_SIZE,
+ .maxauthsize = SHA512_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA512 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(md5),cbc(des3_ede))",
+ .cra_driver_name = "authenc-hmac-md5-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = MD5_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_MD5 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(md5),"
+ "cbc(des3_ede)))",
+ .cra_driver_name = "echainiv-authenc-hmac-md5-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = MD5_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_MD5 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha1),"
+ "cbc(des3_ede))",
+ .cra_driver_name = "authenc-hmac-sha1-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA1_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA1 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha1),"
+ "cbc(des3_ede)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha1-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA1_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA1 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha224),"
+ "cbc(des3_ede))",
+ .cra_driver_name = "authenc-hmac-sha224-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA224_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA224 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha224),"
+ "cbc(des3_ede)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha224-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA224_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA224 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha256),"
+ "cbc(des3_ede))",
+ .cra_driver_name = "authenc-hmac-sha256-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA256_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA256 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha256),"
+ "cbc(des3_ede)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha256-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA256_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA256 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha384),"
+ "cbc(des3_ede))",
+ .cra_driver_name = "authenc-hmac-sha384-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA384_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA384 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha384),"
+ "cbc(des3_ede)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha384-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA384_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA384 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha512),"
+ "cbc(des3_ede))",
+ .cra_driver_name = "authenc-hmac-sha512-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA512_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA512 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha512),"
+ "cbc(des3_ede)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha512-"
+ "cbc-des3_ede-caam-qi",
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .maxauthsize = SHA512_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA512 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(md5),cbc(des))",
+ .cra_driver_name = "authenc-hmac-md5-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = MD5_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_MD5 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(md5),"
+ "cbc(des)))",
+ .cra_driver_name = "echainiv-authenc-hmac-md5-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = MD5_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_MD5 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha1),cbc(des))",
+ .cra_driver_name = "authenc-hmac-sha1-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA1_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA1 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha1),"
+ "cbc(des)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha1-cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA1_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA1 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha224),cbc(des))",
+ .cra_driver_name = "authenc-hmac-sha224-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA224_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA224 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha224),"
+ "cbc(des)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha224-cbc-des-"
+ "caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA224_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA224 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha256),cbc(des))",
+ .cra_driver_name = "authenc-hmac-sha256-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA256_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA256 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha256),"
+ "cbc(des)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha256-cbc-desi-"
+ "caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA256_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA256 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha384),cbc(des))",
+ .cra_driver_name = "authenc-hmac-sha384-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA384_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA384 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ },
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha384),"
+ "cbc(des)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha384-cbc-des-"
+ "caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA384_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA384 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "authenc(hmac(sha512),cbc(des))",
+ .cra_driver_name = "authenc-hmac-sha512-"
+ "cbc-des-caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA512_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA512 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ }
+ },
+ {
+ .aead = {
+ .base = {
+ .cra_name = "echainiv(authenc(hmac(sha512),"
+ "cbc(des)))",
+ .cra_driver_name = "echainiv-authenc-"
+ "hmac-sha512-cbc-des-"
+ "caam-qi",
+ .cra_blocksize = DES_BLOCK_SIZE,
+ },
+ .setkey = aead_setkey,
+ .setauthsize = aead_setauthsize,
+ .encrypt = aead_encrypt,
+ .decrypt = aead_decrypt,
+ .ivsize = DES_BLOCK_SIZE,
+ .maxauthsize = SHA512_DIGEST_SIZE,
+ },
+ .caam = {
+ .class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_CBC,
+ .class2_alg_type = OP_ALG_ALGSEL_SHA512 |
+ OP_ALG_AAI_HMAC_PRECOMP,
+ .geniv = true,
+ }
+ },
+};
+
+struct caam_crypto_alg {
+ struct list_head entry;
+ struct crypto_alg crypto_alg;
+ struct caam_alg_entry caam;
+};
+
+static int caam_init_common(struct caam_ctx *ctx, struct caam_alg_entry *caam)
+{
+ struct caam_drv_private *priv;
+
+ /*
+ * distribute tfms across job rings to ensure in-order
+ * crypto request processing per tfm
+ */
+ ctx->jrdev = caam_jr_alloc();
+ if (IS_ERR(ctx->jrdev)) {
+ pr_err("Job Ring Device allocation for transform failed\n");
+ return PTR_ERR(ctx->jrdev);
+ }
+
+ ctx->key_dma = dma_map_single(ctx->jrdev, ctx->key, sizeof(ctx->key),
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(ctx->jrdev, ctx->key_dma)) {
+ dev_err(ctx->jrdev, "unable to map key\n");
+ caam_jr_free(ctx->jrdev);
+ return -ENOMEM;
+ }
+
+ /* copy descriptor header template value */
+ ctx->cdata.algtype = OP_TYPE_CLASS1_ALG | caam->class1_alg_type;
+ ctx->adata.algtype = OP_TYPE_CLASS2_ALG | caam->class2_alg_type;
+
+ priv = dev_get_drvdata(ctx->jrdev->parent);
+ ctx->qidev = priv->qidev;
+
+ spin_lock_init(&ctx->lock);
+ ctx->drv_ctx[ENCRYPT] = NULL;
+ ctx->drv_ctx[DECRYPT] = NULL;
+ ctx->drv_ctx[GIVENCRYPT] = NULL;
+
+ return 0;
+}
+
+static int caam_cra_init(struct crypto_tfm *tfm)
+{
+ struct crypto_alg *alg = tfm->__crt_alg;
+ struct caam_crypto_alg *caam_alg = container_of(alg, typeof(*caam_alg),
+ crypto_alg);
+ struct caam_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ return caam_init_common(ctx, &caam_alg->caam);
+}
+
+static int caam_aead_init(struct crypto_aead *tfm)
+{
+ struct aead_alg *alg = crypto_aead_alg(tfm);
+ struct caam_aead_alg *caam_alg = container_of(alg, typeof(*caam_alg),
+ aead);
+ struct caam_ctx *ctx = crypto_aead_ctx(tfm);
+
+ return caam_init_common(ctx, &caam_alg->caam);
+}
+
+static void caam_exit_common(struct caam_ctx *ctx)
+{
+ caam_drv_ctx_rel(ctx->drv_ctx[ENCRYPT]);
+ caam_drv_ctx_rel(ctx->drv_ctx[DECRYPT]);
+ caam_drv_ctx_rel(ctx->drv_ctx[GIVENCRYPT]);
+
+ dma_unmap_single(ctx->jrdev, ctx->key_dma, sizeof(ctx->key),
+ DMA_TO_DEVICE);
+
+ caam_jr_free(ctx->jrdev);
+}
+
+static void caam_cra_exit(struct crypto_tfm *tfm)
+{
+ caam_exit_common(crypto_tfm_ctx(tfm));
+}
+
+static void caam_aead_exit(struct crypto_aead *tfm)
+{
+ caam_exit_common(crypto_aead_ctx(tfm));
+}
+
+static struct list_head alg_list;
+static void __exit caam_qi_algapi_exit(void)
+{
+ struct caam_crypto_alg *t_alg, *n;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(driver_aeads); i++) {
+ struct caam_aead_alg *t_alg = driver_aeads + i;
+
+ if (t_alg->registered)
+ crypto_unregister_aead(&t_alg->aead);
+ }
+
+ if (!alg_list.next)
+ return;
+
+ list_for_each_entry_safe(t_alg, n, &alg_list, entry) {
+ crypto_unregister_alg(&t_alg->crypto_alg);
+ list_del(&t_alg->entry);
+ kfree(t_alg);
+ }
+}
+
+static struct caam_crypto_alg *caam_alg_alloc(struct caam_alg_template
+ *template)
+{
+ struct caam_crypto_alg *t_alg;
+ struct crypto_alg *alg;
+
+ t_alg = kzalloc(sizeof(*t_alg), GFP_KERNEL);
+ if (!t_alg)
+ return ERR_PTR(-ENOMEM);
+
+ alg = &t_alg->crypto_alg;
+
+ snprintf(alg->cra_name, CRYPTO_MAX_ALG_NAME, "%s", template->name);
+ snprintf(alg->cra_driver_name, CRYPTO_MAX_ALG_NAME, "%s",
+ template->driver_name);
+ alg->cra_module = THIS_MODULE;
+ alg->cra_init = caam_cra_init;
+ alg->cra_exit = caam_cra_exit;
+ alg->cra_priority = CAAM_CRA_PRIORITY;
+ alg->cra_blocksize = template->blocksize;
+ alg->cra_alignmask = 0;
+ alg->cra_ctxsize = sizeof(struct caam_ctx);
+ alg->cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_KERN_DRIVER_ONLY |
+ template->type;
+ switch (template->type) {
+ case CRYPTO_ALG_TYPE_GIVCIPHER:
+ alg->cra_type = &crypto_givcipher_type;
+ alg->cra_ablkcipher = template->template_ablkcipher;
+ break;
+ case CRYPTO_ALG_TYPE_ABLKCIPHER:
+ alg->cra_type = &crypto_ablkcipher_type;
+ alg->cra_ablkcipher = template->template_ablkcipher;
+ break;
+ }
+
+ t_alg->caam.class1_alg_type = template->class1_alg_type;
+ t_alg->caam.class2_alg_type = template->class2_alg_type;
+
+ return t_alg;
+}
+
+static void caam_aead_alg_init(struct caam_aead_alg *t_alg)
+{
+ struct aead_alg *alg = &t_alg->aead;
+
+ alg->base.cra_module = THIS_MODULE;
+ alg->base.cra_priority = CAAM_CRA_PRIORITY;
+ alg->base.cra_ctxsize = sizeof(struct caam_ctx);
+ alg->base.cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_KERN_DRIVER_ONLY;
+
+ alg->init = caam_aead_init;
+ alg->exit = caam_aead_exit;
+}
+
+static int __init caam_qi_algapi_init(void)
+{
+ struct device_node *dev_node;
+ struct platform_device *pdev;
+ struct device *ctrldev;
+ struct caam_drv_private *priv;
+ int i = 0, err = 0;
+ u32 cha_vid, cha_inst, des_inst, aes_inst, md_inst;
+ unsigned int md_limit = SHA512_DIGEST_SIZE;
+ bool registered = false;
+
+ dev_node = of_find_compatible_node(NULL, NULL, "fsl,sec-v4.0");
+ if (!dev_node) {
+ dev_node = of_find_compatible_node(NULL, NULL, "fsl,sec4.0");
+ if (!dev_node)
+ return -ENODEV;
+ }
+
+ pdev = of_find_device_by_node(dev_node);
+ of_node_put(dev_node);
+ if (!pdev)
+ return -ENODEV;
+
+ ctrldev = &pdev->dev;
+ priv = dev_get_drvdata(ctrldev);
+
+ /*
+ * If priv is NULL, it's probably because the caam driver wasn't
+ * properly initialized (e.g. RNG4 init failed). Thus, bail out here.
+ */
+ if (!priv || !priv->qi_present)
+ return -ENODEV;
+
+ INIT_LIST_HEAD(&alg_list);
+
+ /*
+ * Register crypto algorithms the device supports.
+ * First, detect presence and attributes of DES, AES, and MD blocks.
+ */
+ cha_vid = rd_reg32(&priv->ctrl->perfmon.cha_id_ls);
+ cha_inst = rd_reg32(&priv->ctrl->perfmon.cha_num_ls);
+ des_inst = (cha_inst & CHA_ID_LS_DES_MASK) >> CHA_ID_LS_DES_SHIFT;
+ aes_inst = (cha_inst & CHA_ID_LS_AES_MASK) >> CHA_ID_LS_AES_SHIFT;
+ md_inst = (cha_inst & CHA_ID_LS_MD_MASK) >> CHA_ID_LS_MD_SHIFT;
+
+ /* If MD is present, limit digest size based on LP256 */
+ if (md_inst && ((cha_vid & CHA_ID_LS_MD_MASK) == CHA_ID_LS_MD_LP256))
+ md_limit = SHA256_DIGEST_SIZE;
+
+ for (i = 0; i < ARRAY_SIZE(driver_algs); i++) {
+ struct caam_crypto_alg *t_alg;
+ struct caam_alg_template *alg = driver_algs + i;
+ u32 alg_sel = alg->class1_alg_type & OP_ALG_ALGSEL_MASK;
+
+ /* Skip DES algorithms if not supported by device */
+ if (!des_inst &&
+ ((alg_sel == OP_ALG_ALGSEL_3DES) ||
+ (alg_sel == OP_ALG_ALGSEL_DES)))
+ continue;
+
+ /* Skip AES algorithms if not supported by device */
+ if (!aes_inst && (alg_sel == OP_ALG_ALGSEL_AES))
+ continue;
+
+ t_alg = caam_alg_alloc(alg);
+ if (IS_ERR(t_alg)) {
+ err = PTR_ERR(t_alg);
+ dev_warn(priv->qidev, "%s alg allocation failed\n",
+ alg->driver_name);
+ continue;
+ }
+
+ err = crypto_register_alg(&t_alg->crypto_alg);
+ if (err) {
+ dev_warn(priv->qidev, "%s alg registration failed\n",
+ t_alg->crypto_alg.cra_driver_name);
+ kfree(t_alg);
+ continue;
+ }
+
+ list_add_tail(&t_alg->entry, &alg_list);
+ registered = true;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(driver_aeads); i++) {
+ struct caam_aead_alg *t_alg = driver_aeads + i;
+ u32 c1_alg_sel = t_alg->caam.class1_alg_type &
+ OP_ALG_ALGSEL_MASK;
+ u32 c2_alg_sel = t_alg->caam.class2_alg_type &
+ OP_ALG_ALGSEL_MASK;
+ u32 alg_aai = t_alg->caam.class1_alg_type & OP_ALG_AAI_MASK;
+
+ /* Skip DES algorithms if not supported by device */
+ if (!des_inst &&
+ ((c1_alg_sel == OP_ALG_ALGSEL_3DES) ||
+ (c1_alg_sel == OP_ALG_ALGSEL_DES)))
+ continue;
+
+ /* Skip AES algorithms if not supported by device */
+ if (!aes_inst && (c1_alg_sel == OP_ALG_ALGSEL_AES))
+ continue;
+
+ /*
+ * Check support for AES algorithms not available
+ * on LP devices.
+ */
+ if (((cha_vid & CHA_ID_LS_AES_MASK) == CHA_ID_LS_AES_LP) &&
+ (alg_aai == OP_ALG_AAI_GCM))
+ continue;
+
+ /*
+ * Skip algorithms requiring message digests
+ * if MD or MD size is not supported by device.
+ */
+ if (c2_alg_sel &&
+ (!md_inst || (t_alg->aead.maxauthsize > md_limit)))
+ continue;
+
+ caam_aead_alg_init(t_alg);
+
+ err = crypto_register_aead(&t_alg->aead);
+ if (err) {
+ pr_warn("%s alg registration failed\n",
+ t_alg->aead.base.cra_driver_name);
+ continue;
+ }
+
+ t_alg->registered = true;
+ registered = true;
+ }
+
+ if (registered)
+ dev_info(priv->qidev, "algorithms registered in /proc/crypto\n");
+
+ return err;
+}
+
+module_init(caam_qi_algapi_init);
+module_exit(caam_qi_algapi_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Support for crypto API using CAAM-QI backend");
+MODULE_AUTHOR("Freescale Semiconductor");
diff --git a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c
index 5d7f73d..dd353e3 100644
--- a/drivers/crypto/caam/ctrl.c
+++ b/drivers/crypto/caam/ctrl.c
@@ -18,6 +18,10 @@
bool caam_little_end;
EXPORT_SYMBOL(caam_little_end);
+#ifdef CONFIG_CAAM_QI
+#include "qi.h"
+#endif
+
/*
* i.MX targets tend to have clock control subsystems that can
* enable/disable clocking to our device.
@@ -310,6 +314,11 @@
/* Remove platform devices under the crypto node */
of_platform_depopulate(ctrldev);
+#ifdef CONFIG_CAAM_QI
+ if (ctrlpriv->qidev)
+ caam_qi_shutdown(ctrlpriv->qidev);
+#endif
+
/* De-initialize RNG state handles initialized by this driver. */
if (ctrlpriv->rng4_sh_init)
deinstantiate_rng(ctrldev, ctrlpriv->rng4_sh_init);
@@ -400,23 +409,6 @@
}
EXPORT_SYMBOL(caam_get_era);
-#ifdef CONFIG_DEBUG_FS
-static int caam_debugfs_u64_get(void *data, u64 *val)
-{
- *val = caam64_to_cpu(*(u64 *)data);
- return 0;
-}
-
-static int caam_debugfs_u32_get(void *data, u64 *val)
-{
- *val = caam32_to_cpu(*(u32 *)data);
- return 0;
-}
-
-DEFINE_SIMPLE_ATTRIBUTE(caam_fops_u32_ro, caam_debugfs_u32_get, NULL, "%llu\n");
-DEFINE_SIMPLE_ATTRIBUTE(caam_fops_u64_ro, caam_debugfs_u64_get, NULL, "%llu\n");
-#endif
-
static const struct of_device_id caam_match[] = {
{
.compatible = "fsl,sec-v4.0",
@@ -613,6 +605,18 @@
goto iounmap_ctrl;
}
+#ifdef CONFIG_DEBUG_FS
+ /*
+ * FIXME: needs better naming distinction, as some amalgamation of
+ * "caam" and nprop->full_name. The OF name isn't distinctive,
+ * but does separate instances
+ */
+ perfmon = (struct caam_perfmon __force *)&ctrl->perfmon;
+
+ ctrlpriv->dfs_root = debugfs_create_dir(dev_name(dev), NULL);
+ ctrlpriv->ctl = debugfs_create_dir("ctl", ctrlpriv->dfs_root);
+#endif
+
ring = 0;
for_each_available_child_of_node(nprop, np)
if (of_device_is_compatible(np, "fsl,sec-v4.0-job-ring") ||
@@ -637,6 +641,13 @@
);
/* This is all that's required to physically enable QI */
wr_reg32(&ctrlpriv->qi->qi_control_lo, QICTL_DQEN);
+
+ /* If QMAN driver is present, init CAAM-QI backend */
+#ifdef CONFIG_CAAM_QI
+ ret = caam_qi_init(pdev);
+ if (ret)
+ dev_err(dev, "caam qi i/f init failed: %d\n", ret);
+#endif
}
/* If no QI and no rings specified, quit and go home */
@@ -724,17 +735,6 @@
ctrlpriv->total_jobrs, ctrlpriv->qi_present);
#ifdef CONFIG_DEBUG_FS
- /*
- * FIXME: needs better naming distinction, as some amalgamation of
- * "caam" and nprop->full_name. The OF name isn't distinctive,
- * but does separate instances
- */
- perfmon = (struct caam_perfmon __force *)&ctrl->perfmon;
-
- ctrlpriv->dfs_root = debugfs_create_dir(dev_name(dev), NULL);
- ctrlpriv->ctl = debugfs_create_dir("ctl", ctrlpriv->dfs_root);
-
- /* Controller-level - performance monitor counters */
ctrlpriv->ctl_rq_dequeued =
debugfs_create_file("rq_dequeued",
@@ -817,6 +817,9 @@
return 0;
caam_remove:
+#ifdef CONFIG_DEBUG_FS
+ debugfs_remove_recursive(ctrlpriv->dfs_root);
+#endif
caam_remove(pdev);
return ret;
diff --git a/drivers/crypto/caam/desc_constr.h b/drivers/crypto/caam/desc_constr.h
index b9c8d98..d8e83ca 100644
--- a/drivers/crypto/caam/desc_constr.h
+++ b/drivers/crypto/caam/desc_constr.h
@@ -4,6 +4,9 @@
* Copyright 2008-2012 Freescale Semiconductor, Inc.
*/
+#ifndef DESC_CONSTR_H
+#define DESC_CONSTR_H
+
#include "desc.h"
#include "regs.h"
@@ -491,3 +494,5 @@
return (rem_bytes >= 0) ? 0 : -1;
}
+
+#endif /* DESC_CONSTR_H */
diff --git a/drivers/crypto/caam/intern.h b/drivers/crypto/caam/intern.h
index dbed8ba..85b6c58 100644
--- a/drivers/crypto/caam/intern.h
+++ b/drivers/crypto/caam/intern.h
@@ -66,6 +66,9 @@
struct caam_drv_private {
struct device *dev;
+#ifdef CONFIG_CAAM_QI
+ struct device *qidev;
+#endif
struct platform_device *pdev;
/* Physical-presence section */
@@ -109,9 +112,30 @@
struct debugfs_blob_wrapper ctl_kek_wrap, ctl_tkek_wrap, ctl_tdsk_wrap;
struct dentry *ctl_kek, *ctl_tkek, *ctl_tdsk;
+#ifdef CONFIG_CAAM_QI
+ struct dentry *qi_congested;
+#endif
#endif
};
void caam_jr_algapi_init(struct device *dev);
void caam_jr_algapi_remove(struct device *dev);
+
+#ifdef CONFIG_DEBUG_FS
+static int caam_debugfs_u64_get(void *data, u64 *val)
+{
+ *val = caam64_to_cpu(*(u64 *)data);
+ return 0;
+}
+
+static int caam_debugfs_u32_get(void *data, u64 *val)
+{
+ *val = caam32_to_cpu(*(u32 *)data);
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(caam_fops_u32_ro, caam_debugfs_u32_get, NULL, "%llu\n");
+DEFINE_SIMPLE_ATTRIBUTE(caam_fops_u64_ro, caam_debugfs_u64_get, NULL, "%llu\n");
+#endif
+
#endif /* INTERN_H */
diff --git a/drivers/crypto/caam/qi.c b/drivers/crypto/caam/qi.c
new file mode 100644
index 0000000..1990ed4
--- /dev/null
+++ b/drivers/crypto/caam/qi.c
@@ -0,0 +1,805 @@
+/*
+ * CAAM/SEC 4.x QI transport/backend driver
+ * Queue Interface backend functionality
+ *
+ * Copyright 2013-2016 Freescale Semiconductor, Inc.
+ * Copyright 2016-2017 NXP
+ */
+
+#include <linux/cpumask.h>
+#include <linux/kthread.h>
+#include <soc/fsl/qman.h>
+
+#include "regs.h"
+#include "qi.h"
+#include "desc.h"
+#include "intern.h"
+#include "desc_constr.h"
+
+#define PREHDR_RSLS_SHIFT 31
+
+/*
+ * Use a reasonable backlog of frames (per CPU) as congestion threshold,
+ * so that resources used by the in-flight buffers do not become a memory hog.
+ */
+#define MAX_RSP_FQ_BACKLOG_PER_CPU 256
+
+/* Length of a single buffer in the QI driver memory cache */
+#define CAAM_QI_MEMCACHE_SIZE 512
+
+#define CAAM_QI_ENQUEUE_RETRIES 10000
+
+#define CAAM_NAPI_WEIGHT 63
+
+/*
+ * caam_napi - struct holding CAAM NAPI-related params
+ * @irqtask: IRQ task for QI backend
+ * @p: QMan portal
+ */
+struct caam_napi {
+ struct napi_struct irqtask;
+ struct qman_portal *p;
+};
+
+/*
+ * caam_qi_pcpu_priv - percpu private data structure to main list of pending
+ * responses expected on each cpu.
+ * @caam_napi: CAAM NAPI params
+ * @net_dev: netdev used by NAPI
+ * @rsp_fq: response FQ from CAAM
+ */
+struct caam_qi_pcpu_priv {
+ struct caam_napi caam_napi;
+ struct net_device net_dev;
+ struct qman_fq *rsp_fq;
+} ____cacheline_aligned;
+
+static DEFINE_PER_CPU(struct caam_qi_pcpu_priv, pcpu_qipriv);
+
+/*
+ * caam_qi_priv - CAAM QI backend private params
+ * @cgr: QMan congestion group
+ * @qi_pdev: platform device for QI backend
+ */
+struct caam_qi_priv {
+ struct qman_cgr cgr;
+ struct platform_device *qi_pdev;
+};
+
+static struct caam_qi_priv qipriv ____cacheline_aligned;
+
+/*
+ * This is written by only one core - the one that initialized the CGR - and
+ * read by multiple cores (all the others).
+ */
+bool caam_congested __read_mostly;
+EXPORT_SYMBOL(caam_congested);
+
+#ifdef CONFIG_DEBUG_FS
+/*
+ * This is a counter for the number of times the congestion group (where all
+ * the request and response queueus are) reached congestion. Incremented
+ * each time the congestion callback is called with congested == true.
+ */
+static u64 times_congested;
+#endif
+
+/*
+ * CPU from where the module initialised. This is required because QMan driver
+ * requires CGRs to be removed from same CPU from where they were originally
+ * allocated.
+ */
+static int mod_init_cpu;
+
+/*
+ * This is a a cache of buffers, from which the users of CAAM QI driver
+ * can allocate short (CAAM_QI_MEMCACHE_SIZE) buffers. It's faster than
+ * doing malloc on the hotpath.
+ * NOTE: A more elegant solution would be to have some headroom in the frames
+ * being processed. This could be added by the dpaa-ethernet driver.
+ * This would pose a problem for userspace application processing which
+ * cannot know of this limitation. So for now, this will work.
+ * NOTE: The memcache is SMP-safe. No need to handle spinlocks in-here
+ */
+static struct kmem_cache *qi_cache;
+
+int caam_qi_enqueue(struct device *qidev, struct caam_drv_req *req)
+{
+ struct qm_fd fd;
+ dma_addr_t addr;
+ int ret;
+ int num_retries = 0;
+
+ qm_fd_clear_fd(&fd);
+ qm_fd_set_compound(&fd, qm_sg_entry_get_len(&req->fd_sgt[1]));
+
+ addr = dma_map_single(qidev, req->fd_sgt, sizeof(req->fd_sgt),
+ DMA_BIDIRECTIONAL);
+ if (dma_mapping_error(qidev, addr)) {
+ dev_err(qidev, "DMA mapping error for QI enqueue request\n");
+ return -EIO;
+ }
+ qm_fd_addr_set64(&fd, addr);
+
+ do {
+ ret = qman_enqueue(req->drv_ctx->req_fq, &fd);
+ if (likely(!ret))
+ return 0;
+
+ if (ret != -EBUSY)
+ break;
+ num_retries++;
+ } while (num_retries < CAAM_QI_ENQUEUE_RETRIES);
+
+ dev_err(qidev, "qman_enqueue failed: %d\n", ret);
+
+ return ret;
+}
+EXPORT_SYMBOL(caam_qi_enqueue);
+
+static void caam_fq_ern_cb(struct qman_portal *qm, struct qman_fq *fq,
+ const union qm_mr_entry *msg)
+{
+ const struct qm_fd *fd;
+ struct caam_drv_req *drv_req;
+ struct device *qidev = &(raw_cpu_ptr(&pcpu_qipriv)->net_dev.dev);
+
+ fd = &msg->ern.fd;
+
+ if (qm_fd_get_format(fd) != qm_fd_compound) {
+ dev_err(qidev, "Non-compound FD from CAAM\n");
+ return;
+ }
+
+ drv_req = (struct caam_drv_req *)phys_to_virt(qm_fd_addr_get64(fd));
+ if (!drv_req) {
+ dev_err(qidev,
+ "Can't find original request for CAAM response\n");
+ return;
+ }
+
+ dma_unmap_single(drv_req->drv_ctx->qidev, qm_fd_addr(fd),
+ sizeof(drv_req->fd_sgt), DMA_BIDIRECTIONAL);
+
+ drv_req->cbk(drv_req, -EIO);
+}
+
+static struct qman_fq *create_caam_req_fq(struct device *qidev,
+ struct qman_fq *rsp_fq,
+ dma_addr_t hwdesc,
+ int fq_sched_flag)
+{
+ int ret;
+ struct qman_fq *req_fq;
+ struct qm_mcc_initfq opts;
+
+ req_fq = kzalloc(sizeof(*req_fq), GFP_ATOMIC);
+ if (!req_fq)
+ return ERR_PTR(-ENOMEM);
+
+ req_fq->cb.ern = caam_fq_ern_cb;
+ req_fq->cb.fqs = NULL;
+
+ ret = qman_create_fq(0, QMAN_FQ_FLAG_DYNAMIC_FQID |
+ QMAN_FQ_FLAG_TO_DCPORTAL, req_fq);
+ if (ret) {
+ dev_err(qidev, "Failed to create session req FQ\n");
+ goto create_req_fq_fail;
+ }
+
+ memset(&opts, 0, sizeof(opts));
+ opts.we_mask = cpu_to_be16(QM_INITFQ_WE_FQCTRL | QM_INITFQ_WE_DESTWQ |
+ QM_INITFQ_WE_CONTEXTB |
+ QM_INITFQ_WE_CONTEXTA | QM_INITFQ_WE_CGID);
+ opts.fqd.fq_ctrl = cpu_to_be16(QM_FQCTRL_CPCSTASH | QM_FQCTRL_CGE);
+ qm_fqd_set_destwq(&opts.fqd, qm_channel_caam, 2);
+ opts.fqd.context_b = cpu_to_be32(qman_fq_fqid(rsp_fq));
+ qm_fqd_context_a_set64(&opts.fqd, hwdesc);
+ opts.fqd.cgid = qipriv.cgr.cgrid;
+
+ ret = qman_init_fq(req_fq, fq_sched_flag, &opts);
+ if (ret) {
+ dev_err(qidev, "Failed to init session req FQ\n");
+ goto init_req_fq_fail;
+ }
+
+ dev_info(qidev, "Allocated request FQ %u for CPU %u\n", req_fq->fqid,
+ smp_processor_id());
+ return req_fq;
+
+init_req_fq_fail:
+ qman_destroy_fq(req_fq);
+create_req_fq_fail:
+ kfree(req_fq);
+ return ERR_PTR(ret);
+}
+
+static int empty_retired_fq(struct device *qidev, struct qman_fq *fq)
+{
+ int ret;
+
+ ret = qman_volatile_dequeue(fq, QMAN_VOLATILE_FLAG_WAIT_INT |
+ QMAN_VOLATILE_FLAG_FINISH,
+ QM_VDQCR_PRECEDENCE_VDQCR |
+ QM_VDQCR_NUMFRAMES_TILLEMPTY);
+ if (ret) {
+ dev_err(qidev, "Volatile dequeue fail for FQ: %u\n", fq->fqid);
+ return ret;
+ }
+
+ do {
+ struct qman_portal *p;
+
+ p = qman_get_affine_portal(smp_processor_id());
+ qman_p_poll_dqrr(p, 16);
+ } while (fq->flags & QMAN_FQ_STATE_NE);
+
+ return 0;
+}
+
+static int kill_fq(struct device *qidev, struct qman_fq *fq)
+{
+ u32 flags;
+ int ret;
+
+ ret = qman_retire_fq(fq, &flags);
+ if (ret < 0) {
+ dev_err(qidev, "qman_retire_fq failed: %d\n", ret);
+ return ret;
+ }
+
+ if (!ret)
+ goto empty_fq;
+
+ /* Async FQ retirement condition */
+ if (ret == 1) {
+ /* Retry till FQ gets in retired state */
+ do {
+ msleep(20);
+ } while (fq->state != qman_fq_state_retired);
+
+ WARN_ON(fq->flags & QMAN_FQ_STATE_BLOCKOOS);
+ WARN_ON(fq->flags & QMAN_FQ_STATE_ORL);
+ }
+
+empty_fq:
+ if (fq->flags & QMAN_FQ_STATE_NE) {
+ ret = empty_retired_fq(qidev, fq);
+ if (ret) {
+ dev_err(qidev, "empty_retired_fq fail for FQ: %u\n",
+ fq->fqid);
+ return ret;
+ }
+ }
+
+ ret = qman_oos_fq(fq);
+ if (ret)
+ dev_err(qidev, "OOS of FQID: %u failed\n", fq->fqid);
+
+ qman_destroy_fq(fq);
+
+ return ret;
+}
+
+static int empty_caam_fq(struct qman_fq *fq)
+{
+ int ret;
+ struct qm_mcr_queryfq_np np;
+
+ /* Wait till the older CAAM FQ get empty */
+ do {
+ ret = qman_query_fq_np(fq, &np);
+ if (ret)
+ return ret;
+
+ if (!qm_mcr_np_get(&np, frm_cnt))
+ break;
+
+ msleep(20);
+ } while (1);
+
+ /*
+ * Give extra time for pending jobs from this FQ in holding tanks
+ * to get processed
+ */
+ msleep(20);
+ return 0;
+}
+
+int caam_drv_ctx_update(struct caam_drv_ctx *drv_ctx, u32 *sh_desc)
+{
+ int ret;
+ u32 num_words;
+ struct qman_fq *new_fq, *old_fq;
+ struct device *qidev = drv_ctx->qidev;
+
+ num_words = desc_len(sh_desc);
+ if (num_words > MAX_SDLEN) {
+ dev_err(qidev, "Invalid descriptor len: %d words\n", num_words);
+ return -EINVAL;
+ }
+
+ /* Note down older req FQ */
+ old_fq = drv_ctx->req_fq;
+
+ /* Create a new req FQ in parked state */
+ new_fq = create_caam_req_fq(drv_ctx->qidev, drv_ctx->rsp_fq,
+ drv_ctx->context_a, 0);
+ if (unlikely(IS_ERR_OR_NULL(new_fq))) {
+ dev_err(qidev, "FQ allocation for shdesc update failed\n");
+ return PTR_ERR(new_fq);
+ }
+
+ /* Hook up new FQ to context so that new requests keep queuing */
+ drv_ctx->req_fq = new_fq;
+
+ /* Empty and remove the older FQ */
+ ret = empty_caam_fq(old_fq);
+ if (ret) {
+ dev_err(qidev, "Old CAAM FQ empty failed: %d\n", ret);
+
+ /* We can revert to older FQ */
+ drv_ctx->req_fq = old_fq;
+
+ if (kill_fq(qidev, new_fq))
+ dev_warn(qidev, "New CAAM FQ: %u kill failed\n",
+ new_fq->fqid);
+
+ return ret;
+ }
+
+ /*
+ * Re-initialise pre-header. Set RSLS and SDLEN.
+ * Update the shared descriptor for driver context.
+ */
+ drv_ctx->prehdr[0] = cpu_to_caam32((1 << PREHDR_RSLS_SHIFT) |
+ num_words);
+ memcpy(drv_ctx->sh_desc, sh_desc, desc_bytes(sh_desc));
+ dma_sync_single_for_device(qidev, drv_ctx->context_a,
+ sizeof(drv_ctx->sh_desc) +
+ sizeof(drv_ctx->prehdr),
+ DMA_BIDIRECTIONAL);
+
+ /* Put the new FQ in scheduled state */
+ ret = qman_schedule_fq(new_fq);
+ if (ret) {
+ dev_err(qidev, "Fail to sched new CAAM FQ, ecode = %d\n", ret);
+
+ /*
+ * We can kill new FQ and revert to old FQ.
+ * Since the desc is already modified, it is success case
+ */
+
+ drv_ctx->req_fq = old_fq;
+
+ if (kill_fq(qidev, new_fq))
+ dev_warn(qidev, "New CAAM FQ: %u kill failed\n",
+ new_fq->fqid);
+ } else if (kill_fq(qidev, old_fq)) {
+ dev_warn(qidev, "Old CAAM FQ: %u kill failed\n", old_fq->fqid);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(caam_drv_ctx_update);
+
+struct caam_drv_ctx *caam_drv_ctx_init(struct device *qidev,
+ int *cpu,
+ u32 *sh_desc)
+{
+ size_t size;
+ u32 num_words;
+ dma_addr_t hwdesc;
+ struct caam_drv_ctx *drv_ctx;
+ const cpumask_t *cpus = qman_affine_cpus();
+ static DEFINE_PER_CPU(int, last_cpu);
+
+ num_words = desc_len(sh_desc);
+ if (num_words > MAX_SDLEN) {
+ dev_err(qidev, "Invalid descriptor len: %d words\n",
+ num_words);
+ return ERR_PTR(-EINVAL);
+ }
+
+ drv_ctx = kzalloc(sizeof(*drv_ctx), GFP_ATOMIC);
+ if (!drv_ctx)
+ return ERR_PTR(-ENOMEM);
+
+ /*
+ * Initialise pre-header - set RSLS and SDLEN - and shared descriptor
+ * and dma-map them.
+ */
+ drv_ctx->prehdr[0] = cpu_to_caam32((1 << PREHDR_RSLS_SHIFT) |
+ num_words);
+ memcpy(drv_ctx->sh_desc, sh_desc, desc_bytes(sh_desc));
+ size = sizeof(drv_ctx->prehdr) + sizeof(drv_ctx->sh_desc);
+ hwdesc = dma_map_single(qidev, drv_ctx->prehdr, size,
+ DMA_BIDIRECTIONAL);
+ if (dma_mapping_error(qidev, hwdesc)) {
+ dev_err(qidev, "DMA map error for preheader + shdesc\n");
+ kfree(drv_ctx);
+ return ERR_PTR(-ENOMEM);
+ }
+ drv_ctx->context_a = hwdesc;
+
+ /* If given CPU does not own the portal, choose another one that does */
+ if (!cpumask_test_cpu(*cpu, cpus)) {
+ int *pcpu = &get_cpu_var(last_cpu);
+
+ *pcpu = cpumask_next(*pcpu, cpus);
+ if (*pcpu >= nr_cpu_ids)
+ *pcpu = cpumask_first(cpus);
+ *cpu = *pcpu;
+
+ put_cpu_var(last_cpu);
+ }
+ drv_ctx->cpu = *cpu;
+
+ /* Find response FQ hooked with this CPU */
+ drv_ctx->rsp_fq = per_cpu(pcpu_qipriv.rsp_fq, drv_ctx->cpu);
+
+ /* Attach request FQ */
+ drv_ctx->req_fq = create_caam_req_fq(qidev, drv_ctx->rsp_fq, hwdesc,
+ QMAN_INITFQ_FLAG_SCHED);
+ if (unlikely(IS_ERR_OR_NULL(drv_ctx->req_fq))) {
+ dev_err(qidev, "create_caam_req_fq failed\n");
+ dma_unmap_single(qidev, hwdesc, size, DMA_BIDIRECTIONAL);
+ kfree(drv_ctx);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ drv_ctx->qidev = qidev;
+ return drv_ctx;
+}
+EXPORT_SYMBOL(caam_drv_ctx_init);
+
+void *qi_cache_alloc(gfp_t flags)
+{
+ return kmem_cache_alloc(qi_cache, flags);
+}
+EXPORT_SYMBOL(qi_cache_alloc);
+
+void qi_cache_free(void *obj)
+{
+ kmem_cache_free(qi_cache, obj);
+}
+EXPORT_SYMBOL(qi_cache_free);
+
+static int caam_qi_poll(struct napi_struct *napi, int budget)
+{
+ struct caam_napi *np = container_of(napi, struct caam_napi, irqtask);
+
+ int cleaned = qman_p_poll_dqrr(np->p, budget);
+
+ if (cleaned < budget) {
+ napi_complete(napi);
+ qman_p_irqsource_add(np->p, QM_PIRQ_DQRI);
+ }
+
+ return cleaned;
+}
+
+void caam_drv_ctx_rel(struct caam_drv_ctx *drv_ctx)
+{
+ if (IS_ERR_OR_NULL(drv_ctx))
+ return;
+
+ /* Remove request FQ */
+ if (kill_fq(drv_ctx->qidev, drv_ctx->req_fq))
+ dev_err(drv_ctx->qidev, "Crypto session req FQ kill failed\n");
+
+ dma_unmap_single(drv_ctx->qidev, drv_ctx->context_a,
+ sizeof(drv_ctx->sh_desc) + sizeof(drv_ctx->prehdr),
+ DMA_BIDIRECTIONAL);
+ kfree(drv_ctx);
+}
+EXPORT_SYMBOL(caam_drv_ctx_rel);
+
+int caam_qi_shutdown(struct device *qidev)
+{
+ int i, ret;
+ struct caam_qi_priv *priv = dev_get_drvdata(qidev);
+ const cpumask_t *cpus = qman_affine_cpus();
+ struct cpumask old_cpumask = current->cpus_allowed;
+
+ for_each_cpu(i, cpus) {
+ struct napi_struct *irqtask;
+
+ irqtask = &per_cpu_ptr(&pcpu_qipriv.caam_napi, i)->irqtask;
+ napi_disable(irqtask);
+ netif_napi_del(irqtask);
+
+ if (kill_fq(qidev, per_cpu(pcpu_qipriv.rsp_fq, i)))
+ dev_err(qidev, "Rsp FQ kill failed, cpu: %d\n", i);
+ kfree(per_cpu(pcpu_qipriv.rsp_fq, i));
+ }
+
+ /*
+ * QMan driver requires CGRs to be deleted from same CPU from where they
+ * were instantiated. Hence we get the module removal execute from the
+ * same CPU from where it was originally inserted.
+ */
+ set_cpus_allowed_ptr(current, get_cpu_mask(mod_init_cpu));
+
+ ret = qman_delete_cgr(&priv->cgr);
+ if (ret)
+ dev_err(qidev, "Deletion of CGR failed: %d\n", ret);
+ else
+ qman_release_cgrid(priv->cgr.cgrid);
+
+ kmem_cache_destroy(qi_cache);
+
+ /* Now that we're done with the CGRs, restore the cpus allowed mask */
+ set_cpus_allowed_ptr(current, &old_cpumask);
+
+ platform_device_unregister(priv->qi_pdev);
+ return ret;
+}
+
+static void cgr_cb(struct qman_portal *qm, struct qman_cgr *cgr, int congested)
+{
+ caam_congested = congested;
+
+ if (congested) {
+#ifdef CONFIG_DEBUG_FS
+ times_congested++;
+#endif
+ pr_debug_ratelimited("CAAM entered congestion\n");
+
+ } else {
+ pr_debug_ratelimited("CAAM exited congestion\n");
+ }
+}
+
+static int caam_qi_napi_schedule(struct qman_portal *p, struct caam_napi *np)
+{
+ /*
+ * In case of threaded ISR, for RT kernels in_irq() does not return
+ * appropriate value, so use in_serving_softirq to distinguish between
+ * softirq and irq contexts.
+ */
+ if (unlikely(in_irq() || !in_serving_softirq())) {
+ /* Disable QMan IRQ source and invoke NAPI */
+ qman_p_irqsource_remove(p, QM_PIRQ_DQRI);
+ np->p = p;
+ napi_schedule(&np->irqtask);
+ return 1;
+ }
+ return 0;
+}
+
+static enum qman_cb_dqrr_result caam_rsp_fq_dqrr_cb(struct qman_portal *p,
+ struct qman_fq *rsp_fq,
+ const struct qm_dqrr_entry *dqrr)
+{
+ struct caam_napi *caam_napi = raw_cpu_ptr(&pcpu_qipriv.caam_napi);
+ struct caam_drv_req *drv_req;
+ const struct qm_fd *fd;
+ struct device *qidev = &(raw_cpu_ptr(&pcpu_qipriv)->net_dev.dev);
+ u32 status;
+
+ if (caam_qi_napi_schedule(p, caam_napi))
+ return qman_cb_dqrr_stop;
+
+ fd = &dqrr->fd;
+ status = be32_to_cpu(fd->status);
+ if (unlikely(status))
+ dev_err(qidev, "Error: %#x in CAAM response FD\n", status);
+
+ if (unlikely(qm_fd_get_format(fd) != qm_fd_compound)) {
+ dev_err(qidev, "Non-compound FD from CAAM\n");
+ return qman_cb_dqrr_consume;
+ }
+
+ drv_req = (struct caam_drv_req *)phys_to_virt(qm_fd_addr_get64(fd));
+ if (unlikely(!drv_req)) {
+ dev_err(qidev,
+ "Can't find original request for caam response\n");
+ return qman_cb_dqrr_consume;
+ }
+
+ dma_unmap_single(drv_req->drv_ctx->qidev, qm_fd_addr(fd),
+ sizeof(drv_req->fd_sgt), DMA_BIDIRECTIONAL);
+
+ drv_req->cbk(drv_req, status);
+ return qman_cb_dqrr_consume;
+}
+
+static int alloc_rsp_fq_cpu(struct device *qidev, unsigned int cpu)
+{
+ struct qm_mcc_initfq opts;
+ struct qman_fq *fq;
+ int ret;
+
+ fq = kzalloc(sizeof(*fq), GFP_KERNEL | GFP_DMA);
+ if (!fq)
+ return -ENOMEM;
+
+ fq->cb.dqrr = caam_rsp_fq_dqrr_cb;
+
+ ret = qman_create_fq(0, QMAN_FQ_FLAG_NO_ENQUEUE |
+ QMAN_FQ_FLAG_DYNAMIC_FQID, fq);
+ if (ret) {
+ dev_err(qidev, "Rsp FQ create failed\n");
+ kfree(fq);
+ return -ENODEV;
+ }
+
+ memset(&opts, 0, sizeof(opts));
+ opts.we_mask = cpu_to_be16(QM_INITFQ_WE_FQCTRL | QM_INITFQ_WE_DESTWQ |
+ QM_INITFQ_WE_CONTEXTB |
+ QM_INITFQ_WE_CONTEXTA | QM_INITFQ_WE_CGID);
+ opts.fqd.fq_ctrl = cpu_to_be16(QM_FQCTRL_CTXASTASHING |
+ QM_FQCTRL_CPCSTASH | QM_FQCTRL_CGE);
+ qm_fqd_set_destwq(&opts.fqd, qman_affine_channel(cpu), 3);
+ opts.fqd.cgid = qipriv.cgr.cgrid;
+ opts.fqd.context_a.stashing.exclusive = QM_STASHING_EXCL_CTX |
+ QM_STASHING_EXCL_DATA;
+ qm_fqd_set_stashing(&opts.fqd, 0, 1, 1);
+
+ ret = qman_init_fq(fq, QMAN_INITFQ_FLAG_SCHED, &opts);
+ if (ret) {
+ dev_err(qidev, "Rsp FQ init failed\n");
+ kfree(fq);
+ return -ENODEV;
+ }
+
+ per_cpu(pcpu_qipriv.rsp_fq, cpu) = fq;
+
+ dev_info(qidev, "Allocated response FQ %u for CPU %u", fq->fqid, cpu);
+ return 0;
+}
+
+static int init_cgr(struct device *qidev)
+{
+ int ret;
+ struct qm_mcc_initcgr opts;
+ const u64 cpus = *(u64 *)qman_affine_cpus();
+ const int num_cpus = hweight64(cpus);
+ const u64 val = num_cpus * MAX_RSP_FQ_BACKLOG_PER_CPU;
+
+ ret = qman_alloc_cgrid(&qipriv.cgr.cgrid);
+ if (ret) {
+ dev_err(qidev, "CGR alloc failed for rsp FQs: %d\n", ret);
+ return ret;
+ }
+
+ qipriv.cgr.cb = cgr_cb;
+ memset(&opts, 0, sizeof(opts));
+ opts.we_mask = cpu_to_be16(QM_CGR_WE_CSCN_EN | QM_CGR_WE_CS_THRES |
+ QM_CGR_WE_MODE);
+ opts.cgr.cscn_en = QM_CGR_EN;
+ opts.cgr.mode = QMAN_CGR_MODE_FRAME;
+ qm_cgr_cs_thres_set64(&opts.cgr.cs_thres, val, 1);
+
+ ret = qman_create_cgr(&qipriv.cgr, QMAN_CGR_FLAG_USE_INIT, &opts);
+ if (ret) {
+ dev_err(qidev, "Error %d creating CAAM CGRID: %u\n", ret,
+ qipriv.cgr.cgrid);
+ return ret;
+ }
+
+ dev_info(qidev, "Congestion threshold set to %llu\n", val);
+ return 0;
+}
+
+static int alloc_rsp_fqs(struct device *qidev)
+{
+ int ret, i;
+ const cpumask_t *cpus = qman_affine_cpus();
+
+ /*Now create response FQs*/
+ for_each_cpu(i, cpus) {
+ ret = alloc_rsp_fq_cpu(qidev, i);
+ if (ret) {
+ dev_err(qidev, "CAAM rsp FQ alloc failed, cpu: %u", i);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static void free_rsp_fqs(void)
+{
+ int i;
+ const cpumask_t *cpus = qman_affine_cpus();
+
+ for_each_cpu(i, cpus)
+ kfree(per_cpu(pcpu_qipriv.rsp_fq, i));
+}
+
+int caam_qi_init(struct platform_device *caam_pdev)
+{
+ int err, i;
+ struct platform_device *qi_pdev;
+ struct device *ctrldev = &caam_pdev->dev, *qidev;
+ struct caam_drv_private *ctrlpriv;
+ const cpumask_t *cpus = qman_affine_cpus();
+ struct cpumask old_cpumask = current->cpus_allowed;
+ static struct platform_device_info qi_pdev_info = {
+ .name = "caam_qi",
+ .id = PLATFORM_DEVID_NONE
+ };
+
+ /*
+ * QMAN requires CGRs to be removed from same CPU+portal from where it
+ * was originally allocated. Hence we need to note down the
+ * initialisation CPU and use the same CPU for module exit.
+ * We select the first CPU to from the list of portal owning CPUs.
+ * Then we pin module init to this CPU.
+ */
+ mod_init_cpu = cpumask_first(cpus);
+ set_cpus_allowed_ptr(current, get_cpu_mask(mod_init_cpu));
+
+ qi_pdev_info.parent = ctrldev;
+ qi_pdev_info.dma_mask = dma_get_mask(ctrldev);
+ qi_pdev = platform_device_register_full(&qi_pdev_info);
+ if (IS_ERR(qi_pdev))
+ return PTR_ERR(qi_pdev);
+
+ ctrlpriv = dev_get_drvdata(ctrldev);
+ qidev = &qi_pdev->dev;
+
+ qipriv.qi_pdev = qi_pdev;
+ dev_set_drvdata(qidev, &qipriv);
+
+ /* Initialize the congestion detection */
+ err = init_cgr(qidev);
+ if (err) {
+ dev_err(qidev, "CGR initialization failed: %d\n", err);
+ platform_device_unregister(qi_pdev);
+ return err;
+ }
+
+ /* Initialise response FQs */
+ err = alloc_rsp_fqs(qidev);
+ if (err) {
+ dev_err(qidev, "Can't allocate CAAM response FQs: %d\n", err);
+ free_rsp_fqs();
+ platform_device_unregister(qi_pdev);
+ return err;
+ }
+
+ /*
+ * Enable the NAPI contexts on each of the core which has an affine
+ * portal.
+ */
+ for_each_cpu(i, cpus) {
+ struct caam_qi_pcpu_priv *priv = per_cpu_ptr(&pcpu_qipriv, i);
+ struct caam_napi *caam_napi = &priv->caam_napi;
+ struct napi_struct *irqtask = &caam_napi->irqtask;
+ struct net_device *net_dev = &priv->net_dev;
+
+ net_dev->dev = *qidev;
+ INIT_LIST_HEAD(&net_dev->napi_list);
+
+ netif_napi_add(net_dev, irqtask, caam_qi_poll,
+ CAAM_NAPI_WEIGHT);
+
+ napi_enable(irqtask);
+ }
+
+ /* Hook up QI device to parent controlling caam device */
+ ctrlpriv->qidev = qidev;
+
+ qi_cache = kmem_cache_create("caamqicache", CAAM_QI_MEMCACHE_SIZE, 0,
+ SLAB_CACHE_DMA, NULL);
+ if (!qi_cache) {
+ dev_err(qidev, "Can't allocate CAAM cache\n");
+ free_rsp_fqs();
+ platform_device_unregister(qi_pdev);
+ return -ENOMEM;
+ }
+
+ /* Done with the CGRs; restore the cpus allowed mask */
+ set_cpus_allowed_ptr(current, &old_cpumask);
+#ifdef CONFIG_DEBUG_FS
+ ctrlpriv->qi_congested = debugfs_create_file("qi_congested", 0444,
+ ctrlpriv->ctl,
+ ×_congested,
+ &caam_fops_u64_ro);
+#endif
+ dev_info(qidev, "Linux CAAM Queue I/F driver initialised\n");
+ return 0;
+}
diff --git a/drivers/crypto/caam/qi.h b/drivers/crypto/caam/qi.h
new file mode 100644
index 0000000..33b0433
--- /dev/null
+++ b/drivers/crypto/caam/qi.h
@@ -0,0 +1,201 @@
+/*
+ * Public definitions for the CAAM/QI (Queue Interface) backend.
+ *
+ * Copyright 2013-2016 Freescale Semiconductor, Inc.
+ * Copyright 2016-2017 NXP
+ */
+
+#ifndef __QI_H__
+#define __QI_H__
+
+#include <soc/fsl/qman.h>
+#include "compat.h"
+#include "desc.h"
+#include "desc_constr.h"
+
+/*
+ * CAAM hardware constructs a job descriptor which points to a shared descriptor
+ * (as pointed by context_a of to-CAAM FQ).
+ * When the job descriptor is executed by DECO, the whole job descriptor
+ * together with shared descriptor gets loaded in DECO buffer, which is
+ * 64 words (each 32-bit) long.
+ *
+ * The job descriptor constructed by CAAM hardware has the following layout:
+ *
+ * HEADER (1 word)
+ * Shdesc ptr (1 or 2 words)
+ * SEQ_OUT_PTR (1 word)
+ * Out ptr (1 or 2 words)
+ * Out length (1 word)
+ * SEQ_IN_PTR (1 word)
+ * In ptr (1 or 2 words)
+ * In length (1 word)
+ *
+ * The shdesc ptr is used to fetch shared descriptor contents into DECO buffer.
+ *
+ * Apart from shdesc contents, the total number of words that get loaded in DECO
+ * buffer are '8' or '11'. The remaining words in DECO buffer can be used for
+ * storing shared descriptor.
+ */
+#define MAX_SDLEN ((CAAM_DESC_BYTES_MAX - DESC_JOB_IO_LEN) / CAAM_CMD_SZ)
+
+extern bool caam_congested __read_mostly;
+
+/*
+ * This is the request structure the driver application should fill while
+ * submitting a job to driver.
+ */
+struct caam_drv_req;
+
+/*
+ * caam_qi_cbk - application's callback function invoked by the driver when the
+ * request has been successfully processed.
+ * @drv_req: original request that was submitted
+ * @status: completion status of request (0 - success, non-zero - error code)
+ */
+typedef void (*caam_qi_cbk)(struct caam_drv_req *drv_req, u32 status);
+
+enum optype {
+ ENCRYPT,
+ DECRYPT,
+ GIVENCRYPT,
+ NUM_OP
+};
+
+/**
+ * caam_drv_ctx - CAAM/QI backend driver context
+ *
+ * The jobs are processed by the driver against a driver context.
+ * With every cryptographic context, a driver context is attached.
+ * The driver context contains data for private use by driver.
+ * For the applications, this is an opaque structure.
+ *
+ * @prehdr: preheader placed before shrd desc
+ * @sh_desc: shared descriptor
+ * @context_a: shared descriptor dma address
+ * @req_fq: to-CAAM request frame queue
+ * @rsp_fq: from-CAAM response frame queue
+ * @cpu: cpu on which to receive CAAM response
+ * @op_type: operation type
+ * @qidev: device pointer for CAAM/QI backend
+ */
+struct caam_drv_ctx {
+ u32 prehdr[2];
+ u32 sh_desc[MAX_SDLEN];
+ dma_addr_t context_a;
+ struct qman_fq *req_fq;
+ struct qman_fq *rsp_fq;
+ int cpu;
+ enum optype op_type;
+ struct device *qidev;
+} ____cacheline_aligned;
+
+/**
+ * caam_drv_req - The request structure the driver application should fill while
+ * submitting a job to driver.
+ * @fd_sgt: QMan S/G pointing to output (fd_sgt[0]) and input (fd_sgt[1])
+ * buffers.
+ * @cbk: callback function to invoke when job is completed
+ * @app_ctx: arbitrary context attached with request by the application
+ *
+ * The fields mentioned below should not be used by application.
+ * These are for private use by driver.
+ *
+ * @hdr__: linked list header to maintain list of outstanding requests to CAAM
+ * @hwaddr: DMA address for the S/G table.
+ */
+struct caam_drv_req {
+ struct qm_sg_entry fd_sgt[2];
+ struct caam_drv_ctx *drv_ctx;
+ caam_qi_cbk cbk;
+ void *app_ctx;
+} ____cacheline_aligned;
+
+/**
+ * caam_drv_ctx_init - Initialise a CAAM/QI driver context
+ *
+ * A CAAM/QI driver context must be attached with each cryptographic context.
+ * This function allocates memory for CAAM/QI context and returns a handle to
+ * the application. This handle must be submitted along with each enqueue
+ * request to the driver by the application.
+ *
+ * @cpu: CPU where the application prefers to the driver to receive CAAM
+ * responses. The request completion callback would be issued from this
+ * CPU.
+ * @sh_desc: shared descriptor pointer to be attached with CAAM/QI driver
+ * context.
+ *
+ * Returns a driver context on success or negative error code on failure.
+ */
+struct caam_drv_ctx *caam_drv_ctx_init(struct device *qidev, int *cpu,
+ u32 *sh_desc);
+
+/**
+ * caam_qi_enqueue - Submit a request to QI backend driver.
+ *
+ * The request structure must be properly filled as described above.
+ *
+ * @qidev: device pointer for QI backend
+ * @req: CAAM QI request structure
+ *
+ * Returns 0 on success or negative error code on failure.
+ */
+int caam_qi_enqueue(struct device *qidev, struct caam_drv_req *req);
+
+/**
+ * caam_drv_ctx_busy - Check if there are too many jobs pending with CAAM
+ * or too many CAAM responses are pending to be processed.
+ * @drv_ctx: driver context for which job is to be submitted
+ *
+ * Returns caam congestion status 'true/false'
+ */
+bool caam_drv_ctx_busy(struct caam_drv_ctx *drv_ctx);
+
+/**
+ * caam_drv_ctx_update - Update QI driver context
+ *
+ * Invoked when shared descriptor is required to be change in driver context.
+ *
+ * @drv_ctx: driver context to be updated
+ * @sh_desc: new shared descriptor pointer to be updated in QI driver context
+ *
+ * Returns 0 on success or negative error code on failure.
+ */
+int caam_drv_ctx_update(struct caam_drv_ctx *drv_ctx, u32 *sh_desc);
+
+/**
+ * caam_drv_ctx_rel - Release a QI driver context
+ * @drv_ctx: context to be released
+ */
+void caam_drv_ctx_rel(struct caam_drv_ctx *drv_ctx);
+
+int caam_qi_init(struct platform_device *pdev);
+int caam_qi_shutdown(struct device *dev);
+
+/**
+ * qi_cache_alloc - Allocate buffers from CAAM-QI cache
+ *
+ * Invoked when a user of the CAAM-QI (i.e. caamalg-qi) needs data which has
+ * to be allocated on the hotpath. Instead of using malloc, one can use the
+ * services of the CAAM QI memory cache (backed by kmem_cache). The buffers
+ * will have a size of 256B, which is sufficient for hosting 16 SG entries.
+ *
+ * @flags: flags that would be used for the equivalent malloc(..) call
+ *
+ * Returns a pointer to a retrieved buffer on success or NULL on failure.
+ */
+void *qi_cache_alloc(gfp_t flags);
+
+/**
+ * qi_cache_free - Frees buffers allocated from CAAM-QI cache
+ *
+ * Invoked when a user of the CAAM-QI (i.e. caamalg-qi) no longer needs
+ * the buffer previously allocated by a qi_cache_alloc call.
+ * No checking is being done, the call is a passthrough call to
+ * kmem_cache_free(...)
+ *
+ * @obj: object previously allocated using qi_cache_alloc()
+ */
+void qi_cache_free(void *obj);
+
+#endif /* __QI_H__ */
diff --git a/drivers/crypto/caam/sg_sw_qm.h b/drivers/crypto/caam/sg_sw_qm.h
new file mode 100644
index 0000000..d000b4df
--- /dev/null
+++ b/drivers/crypto/caam/sg_sw_qm.h
@@ -0,0 +1,108 @@
+/*
+ * Copyright 2013-2016 Freescale Semiconductor, Inc.
+ * Copyright 2016-2017 NXP
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ * * Neither the name of Freescale Semiconductor nor the
+ * names of its contributors may be used to endorse or promote products
+ * derived from this software without specific prior written permission.
+ *
+ *
+ * ALTERNATIVELY, this software may be distributed under the terms of the
+ * GNU General Public License ("GPL") as published by the Free Software
+ * Foundation, either version 2 of that License or (at your option) any
+ * later version.
+ *
+ * THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+ * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ * DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY
+ * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+ * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+ * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef __SG_SW_QM_H
+#define __SG_SW_QM_H
+
+#include <soc/fsl/qman.h>
+#include "regs.h"
+
+static inline void __dma_to_qm_sg(struct qm_sg_entry *qm_sg_ptr, dma_addr_t dma,
+ u16 offset)
+{
+ qm_sg_entry_set64(qm_sg_ptr, dma);
+ qm_sg_ptr->__reserved2 = 0;
+ qm_sg_ptr->bpid = 0;
+ qm_sg_ptr->offset = cpu_to_be16(offset & QM_SG_OFF_MASK);
+}
+
+static inline void dma_to_qm_sg_one(struct qm_sg_entry *qm_sg_ptr,
+ dma_addr_t dma, u32 len, u16 offset)
+{
+ __dma_to_qm_sg(qm_sg_ptr, dma, offset);
+ qm_sg_entry_set_len(qm_sg_ptr, len);
+}
+
+static inline void dma_to_qm_sg_one_last(struct qm_sg_entry *qm_sg_ptr,
+ dma_addr_t dma, u32 len, u16 offset)
+{
+ __dma_to_qm_sg(qm_sg_ptr, dma, offset);
+ qm_sg_entry_set_f(qm_sg_ptr, len);
+}
+
+static inline void dma_to_qm_sg_one_ext(struct qm_sg_entry *qm_sg_ptr,
+ dma_addr_t dma, u32 len, u16 offset)
+{
+ __dma_to_qm_sg(qm_sg_ptr, dma, offset);
+ qm_sg_ptr->cfg = cpu_to_be32(QM_SG_EXT | (len & QM_SG_LEN_MASK));
+}
+
+static inline void dma_to_qm_sg_one_last_ext(struct qm_sg_entry *qm_sg_ptr,
+ dma_addr_t dma, u32 len,
+ u16 offset)
+{
+ __dma_to_qm_sg(qm_sg_ptr, dma, offset);
+ qm_sg_ptr->cfg = cpu_to_be32(QM_SG_EXT | QM_SG_FIN |
+ (len & QM_SG_LEN_MASK));
+}
+
+/*
+ * convert scatterlist to h/w link table format
+ * but does not have final bit; instead, returns last entry
+ */
+static inline struct qm_sg_entry *
+sg_to_qm_sg(struct scatterlist *sg, int sg_count,
+ struct qm_sg_entry *qm_sg_ptr, u16 offset)
+{
+ while (sg_count && sg) {
+ dma_to_qm_sg_one(qm_sg_ptr, sg_dma_address(sg),
+ sg_dma_len(sg), offset);
+ qm_sg_ptr++;
+ sg = sg_next(sg);
+ sg_count--;
+ }
+ return qm_sg_ptr - 1;
+}
+
+/*
+ * convert scatterlist to h/w link table format
+ * scatterlist must have been previously dma mapped
+ */
+static inline void sg_to_qm_sg_last(struct scatterlist *sg, int sg_count,
+ struct qm_sg_entry *qm_sg_ptr, u16 offset)
+{
+ qm_sg_ptr = sg_to_qm_sg(sg, sg_count, qm_sg_ptr, offset);
+ qm_sg_entry_set_f(qm_sg_ptr, qm_sg_entry_get_len(qm_sg_ptr));
+}
+
+#endif /* __SG_SW_QM_H */
diff --git a/drivers/crypto/cavium/Makefile b/drivers/crypto/cavium/Makefile
new file mode 100644
index 0000000..641268b
--- /dev/null
+++ b/drivers/crypto/cavium/Makefile
@@ -0,0 +1,4 @@
+#
+# Makefile for Cavium crypto device drivers
+#
+obj-$(CONFIG_CRYPTO_DEV_CAVIUM_ZIP) += zip/
diff --git a/drivers/crypto/cavium/zip/Makefile b/drivers/crypto/cavium/zip/Makefile
new file mode 100644
index 0000000..b2f3baaf
--- /dev/null
+++ b/drivers/crypto/cavium/zip/Makefile
@@ -0,0 +1,11 @@
+#
+# Makefile for Cavium's ZIP Driver.
+#
+
+obj-$(CONFIG_CRYPTO_DEV_CAVIUM_ZIP) += thunderx_zip.o
+thunderx_zip-y := zip_main.o \
+ zip_device.o \
+ zip_crypto.o \
+ zip_mem.o \
+ zip_deflate.o \
+ zip_inflate.o
diff --git a/drivers/crypto/cavium/zip/common.h b/drivers/crypto/cavium/zip/common.h
new file mode 100644
index 0000000..dc451e0
--- /dev/null
+++ b/drivers/crypto/cavium/zip/common.h
@@ -0,0 +1,202 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __COMMON_H__
+#define __COMMON_H__
+
+#include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/seq_file.h>
+#include <linux/string.h>
+#include <linux/types.h>
+#include <linux/version.h>
+
+/* Device specific zlib function definitions */
+#include "zip_device.h"
+
+/* ZIP device definitions */
+#include "zip_main.h"
+
+/* ZIP memory allocation/deallocation related definitions */
+#include "zip_mem.h"
+
+/* Device specific structure definitions */
+#include "zip_regs.h"
+
+#define ZIP_ERROR -1
+
+#define ZIP_FLUSH_FINISH 4
+
+#define RAW_FORMAT 0 /* for rawpipe */
+#define ZLIB_FORMAT 1 /* for zpipe */
+#define GZIP_FORMAT 2 /* for gzpipe */
+#define LZS_FORMAT 3 /* for lzspipe */
+
+/* Max number of ZIP devices supported */
+#define MAX_ZIP_DEVICES 2
+
+/* Configures the number of zip queues to be used */
+#define ZIP_NUM_QUEUES 2
+
+#define DYNAMIC_STOP_EXCESS 1024
+
+/* Maximum buffer sizes in direct mode */
+#define MAX_INPUT_BUFFER_SIZE (64 * 1024)
+#define MAX_OUTPUT_BUFFER_SIZE (64 * 1024)
+
+/**
+ * struct zip_operation - common data structure for comp and decomp operations
+ * @input: Next input byte is read from here
+ * @output: Next output byte written here
+ * @ctx_addr: Inflate context buffer address
+ * @history: Pointer to the history buffer
+ * @input_len: Number of bytes available at next_in
+ * @input_total_len: Total number of input bytes read
+ * @output_len: Remaining free space at next_out
+ * @output_total_len: Total number of bytes output so far
+ * @csum: Checksum value of the uncompressed data
+ * @flush: Flush flag
+ * @format: Format (depends on stream's wrap)
+ * @speed: Speed depends on stream's level
+ * @ccode: Compression code ( stream's strategy)
+ * @lzs_flag: Flag for LZS support
+ * @begin_file: Beginning of file indication for inflate
+ * @history_len: Size of the history data
+ * @end_file: Ending of the file indication for inflate
+ * @compcode: Completion status of the ZIP invocation
+ * @bytes_read: Input bytes read in current instruction
+ * @bits_processed: Total bits processed for entire file
+ * @sizeofptr: To distinguish between ILP32 and LP64
+ * @sizeofzops: Optional just for padding
+ *
+ * This structure is used to maintain the required meta data for the
+ * comp and decomp operations.
+ */
+struct zip_operation {
+ u8 *input;
+ u8 *output;
+ u64 ctx_addr;
+ u64 history;
+
+ u32 input_len;
+ u32 input_total_len;
+
+ u32 output_len;
+ u32 output_total_len;
+
+ u32 csum;
+ u32 flush;
+
+ u32 format;
+ u32 speed;
+ u32 ccode;
+ u32 lzs_flag;
+
+ u32 begin_file;
+ u32 history_len;
+
+ u32 end_file;
+ u32 compcode;
+ u32 bytes_read;
+ u32 bits_processed;
+
+ u32 sizeofptr;
+ u32 sizeofzops;
+};
+
+/* error messages */
+#define zip_err(fmt, args...) pr_err("ZIP ERR:%s():%d: " \
+ fmt "\n", __func__, __LINE__, ## args)
+
+#ifdef MSG_ENABLE
+/* Enable all messages */
+#define zip_msg(fmt, args...) pr_info("ZIP_MSG:" fmt "\n", ## args)
+#else
+#define zip_msg(fmt, args...)
+#endif
+
+#if defined(ZIP_DEBUG_ENABLE) && defined(MSG_ENABLE)
+
+#ifdef DEBUG_LEVEL
+
+#define FILE_NAME (strrchr(__FILE__, '/') ? strrchr(__FILE__, '/') + 1 : \
+ strrchr(__FILE__, '\\') ? strrchr(__FILE__, '\\') + 1 : __FILE__)
+
+#if DEBUG_LEVEL >= 4
+
+#define zip_dbg(fmt, args...) pr_info("ZIP DBG: %s: %s() : %d: " \
+ fmt "\n", FILE_NAME, __func__, __LINE__, ## args)
+
+#elif DEBUG_LEVEL >= 3
+
+#define zip_dbg(fmt, args...) pr_info("ZIP DBG: %s: %s() : %d: " \
+ fmt "\n", FILE_NAME, __func__, __LINE__, ## args)
+
+#elif DEBUG_LEVEL >= 2
+
+#define zip_dbg(fmt, args...) pr_info("ZIP DBG: %s() : %d: " \
+ fmt "\n", __func__, __LINE__, ## args)
+
+#else
+
+#define zip_dbg(fmt, args...) pr_info("ZIP DBG:" fmt "\n", ## args)
+
+#endif /* DEBUG LEVEL >=4 */
+
+#else
+
+#define zip_dbg(fmt, args...) pr_info("ZIP DBG:" fmt "\n", ## args)
+
+#endif /* DEBUG_LEVEL */
+#else
+
+#define zip_dbg(fmt, args...)
+
+#endif /* ZIP_DEBUG_ENABLE && MSG_ENABLE*/
+
+#endif
diff --git a/drivers/crypto/cavium/zip/zip_crypto.c b/drivers/crypto/cavium/zip/zip_crypto.c
new file mode 100644
index 0000000..8df4d26
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_crypto.c
@@ -0,0 +1,313 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#include "zip_crypto.h"
+
+static void zip_static_init_zip_ops(struct zip_operation *zip_ops,
+ int lzs_flag)
+{
+ zip_ops->flush = ZIP_FLUSH_FINISH;
+
+ /* equivalent to level 6 of opensource zlib */
+ zip_ops->speed = 1;
+
+ if (!lzs_flag) {
+ zip_ops->ccode = 0; /* Auto Huffman */
+ zip_ops->lzs_flag = 0;
+ zip_ops->format = ZLIB_FORMAT;
+ } else {
+ zip_ops->ccode = 3; /* LZS Encoding */
+ zip_ops->lzs_flag = 1;
+ zip_ops->format = LZS_FORMAT;
+ }
+ zip_ops->begin_file = 1;
+ zip_ops->history_len = 0;
+ zip_ops->end_file = 1;
+ zip_ops->compcode = 0;
+ zip_ops->csum = 1; /* Adler checksum desired */
+}
+
+int zip_ctx_init(struct zip_kernel_ctx *zip_ctx, int lzs_flag)
+{
+ struct zip_operation *comp_ctx = &zip_ctx->zip_comp;
+ struct zip_operation *decomp_ctx = &zip_ctx->zip_decomp;
+
+ zip_static_init_zip_ops(comp_ctx, lzs_flag);
+ zip_static_init_zip_ops(decomp_ctx, lzs_flag);
+
+ comp_ctx->input = zip_data_buf_alloc(MAX_INPUT_BUFFER_SIZE);
+ if (!comp_ctx->input)
+ return -ENOMEM;
+
+ comp_ctx->output = zip_data_buf_alloc(MAX_OUTPUT_BUFFER_SIZE);
+ if (!comp_ctx->output)
+ goto err_comp_input;
+
+ decomp_ctx->input = zip_data_buf_alloc(MAX_INPUT_BUFFER_SIZE);
+ if (!decomp_ctx->input)
+ goto err_comp_output;
+
+ decomp_ctx->output = zip_data_buf_alloc(MAX_OUTPUT_BUFFER_SIZE);
+ if (!decomp_ctx->output)
+ goto err_decomp_input;
+
+ return 0;
+
+err_decomp_input:
+ zip_data_buf_free(decomp_ctx->input, MAX_INPUT_BUFFER_SIZE);
+
+err_comp_output:
+ zip_data_buf_free(comp_ctx->output, MAX_OUTPUT_BUFFER_SIZE);
+
+err_comp_input:
+ zip_data_buf_free(comp_ctx->input, MAX_INPUT_BUFFER_SIZE);
+
+ return -ENOMEM;
+}
+
+void zip_ctx_exit(struct zip_kernel_ctx *zip_ctx)
+{
+ struct zip_operation *comp_ctx = &zip_ctx->zip_comp;
+ struct zip_operation *dec_ctx = &zip_ctx->zip_decomp;
+
+ zip_data_buf_free(comp_ctx->input, MAX_INPUT_BUFFER_SIZE);
+ zip_data_buf_free(comp_ctx->output, MAX_OUTPUT_BUFFER_SIZE);
+
+ zip_data_buf_free(dec_ctx->input, MAX_INPUT_BUFFER_SIZE);
+ zip_data_buf_free(dec_ctx->output, MAX_OUTPUT_BUFFER_SIZE);
+}
+
+int zip_compress(const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen,
+ struct zip_kernel_ctx *zip_ctx)
+{
+ struct zip_operation *zip_ops = NULL;
+ struct zip_state zip_state;
+ struct zip_device *zip = NULL;
+ int ret;
+
+ if (!zip_ctx || !src || !dst || !dlen)
+ return -ENOMEM;
+
+ zip = zip_get_device(zip_get_node_id());
+ if (!zip)
+ return -ENODEV;
+
+ memset(&zip_state, 0, sizeof(struct zip_state));
+ zip_ops = &zip_ctx->zip_comp;
+
+ zip_ops->input_len = slen;
+ zip_ops->output_len = *dlen;
+ memcpy(zip_ops->input, src, slen);
+
+ ret = zip_deflate(zip_ops, &zip_state, zip);
+
+ if (!ret) {
+ *dlen = zip_ops->output_len;
+ memcpy(dst, zip_ops->output, *dlen);
+ }
+
+ return ret;
+}
+
+int zip_decompress(const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen,
+ struct zip_kernel_ctx *zip_ctx)
+{
+ struct zip_operation *zip_ops = NULL;
+ struct zip_state zip_state;
+ struct zip_device *zip = NULL;
+ int ret;
+
+ if (!zip_ctx || !src || !dst || !dlen)
+ return -ENOMEM;
+
+ zip = zip_get_device(zip_get_node_id());
+ if (!zip)
+ return -ENODEV;
+
+ memset(&zip_state, 0, sizeof(struct zip_state));
+ zip_ops = &zip_ctx->zip_decomp;
+ memcpy(zip_ops->input, src, slen);
+
+ /* Work around for a bug in zlib which needs an extra bytes sometimes */
+ if (zip_ops->ccode != 3) /* Not LZS Encoding */
+ zip_ops->input[slen++] = 0;
+
+ zip_ops->input_len = slen;
+ zip_ops->output_len = *dlen;
+
+ ret = zip_inflate(zip_ops, &zip_state, zip);
+
+ if (!ret) {
+ *dlen = zip_ops->output_len;
+ memcpy(dst, zip_ops->output, *dlen);
+ }
+
+ return ret;
+}
+
+/* Legacy Compress framework start */
+int zip_alloc_comp_ctx_deflate(struct crypto_tfm *tfm)
+{
+ int ret;
+ struct zip_kernel_ctx *zip_ctx = crypto_tfm_ctx(tfm);
+
+ ret = zip_ctx_init(zip_ctx, 0);
+
+ return ret;
+}
+
+int zip_alloc_comp_ctx_lzs(struct crypto_tfm *tfm)
+{
+ int ret;
+ struct zip_kernel_ctx *zip_ctx = crypto_tfm_ctx(tfm);
+
+ ret = zip_ctx_init(zip_ctx, 1);
+
+ return ret;
+}
+
+void zip_free_comp_ctx(struct crypto_tfm *tfm)
+{
+ struct zip_kernel_ctx *zip_ctx = crypto_tfm_ctx(tfm);
+
+ zip_ctx_exit(zip_ctx);
+}
+
+int zip_comp_compress(struct crypto_tfm *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen)
+{
+ int ret;
+ struct zip_kernel_ctx *zip_ctx = crypto_tfm_ctx(tfm);
+
+ ret = zip_compress(src, slen, dst, dlen, zip_ctx);
+
+ return ret;
+}
+
+int zip_comp_decompress(struct crypto_tfm *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen)
+{
+ int ret;
+ struct zip_kernel_ctx *zip_ctx = crypto_tfm_ctx(tfm);
+
+ ret = zip_decompress(src, slen, dst, dlen, zip_ctx);
+
+ return ret;
+} /* Legacy compress framework end */
+
+/* SCOMP framework start */
+void *zip_alloc_scomp_ctx_deflate(struct crypto_scomp *tfm)
+{
+ int ret;
+ struct zip_kernel_ctx *zip_ctx;
+
+ zip_ctx = kzalloc(sizeof(*zip_ctx), GFP_KERNEL);
+ if (!zip_ctx)
+ return ERR_PTR(-ENOMEM);
+
+ ret = zip_ctx_init(zip_ctx, 0);
+
+ if (ret) {
+ kzfree(zip_ctx);
+ return ERR_PTR(ret);
+ }
+
+ return zip_ctx;
+}
+
+void *zip_alloc_scomp_ctx_lzs(struct crypto_scomp *tfm)
+{
+ int ret;
+ struct zip_kernel_ctx *zip_ctx;
+
+ zip_ctx = kzalloc(sizeof(*zip_ctx), GFP_KERNEL);
+ if (!zip_ctx)
+ return ERR_PTR(-ENOMEM);
+
+ ret = zip_ctx_init(zip_ctx, 1);
+
+ if (ret) {
+ kzfree(zip_ctx);
+ return ERR_PTR(ret);
+ }
+
+ return zip_ctx;
+}
+
+void zip_free_scomp_ctx(struct crypto_scomp *tfm, void *ctx)
+{
+ struct zip_kernel_ctx *zip_ctx = ctx;
+
+ zip_ctx_exit(zip_ctx);
+ kzfree(zip_ctx);
+}
+
+int zip_scomp_compress(struct crypto_scomp *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen, void *ctx)
+{
+ int ret;
+ struct zip_kernel_ctx *zip_ctx = ctx;
+
+ ret = zip_compress(src, slen, dst, dlen, zip_ctx);
+
+ return ret;
+}
+
+int zip_scomp_decompress(struct crypto_scomp *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen, void *ctx)
+{
+ int ret;
+ struct zip_kernel_ctx *zip_ctx = ctx;
+
+ ret = zip_decompress(src, slen, dst, dlen, zip_ctx);
+
+ return ret;
+} /* SCOMP framework end */
diff --git a/drivers/crypto/cavium/zip/zip_crypto.h b/drivers/crypto/cavium/zip/zip_crypto.h
new file mode 100644
index 0000000..b59ddfc
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_crypto.h
@@ -0,0 +1,79 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_CRYPTO_H__
+#define __ZIP_CRYPTO_H__
+
+#include <linux/crypto.h>
+#include <crypto/internal/scompress.h>
+#include "common.h"
+#include "zip_deflate.h"
+#include "zip_inflate.h"
+
+struct zip_kernel_ctx {
+ struct zip_operation zip_comp;
+ struct zip_operation zip_decomp;
+};
+
+int zip_alloc_comp_ctx_deflate(struct crypto_tfm *tfm);
+int zip_alloc_comp_ctx_lzs(struct crypto_tfm *tfm);
+void zip_free_comp_ctx(struct crypto_tfm *tfm);
+int zip_comp_compress(struct crypto_tfm *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen);
+int zip_comp_decompress(struct crypto_tfm *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen);
+
+void *zip_alloc_scomp_ctx_deflate(struct crypto_scomp *tfm);
+void *zip_alloc_scomp_ctx_lzs(struct crypto_scomp *tfm);
+void zip_free_scomp_ctx(struct crypto_scomp *tfm, void *zip_ctx);
+int zip_scomp_compress(struct crypto_scomp *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen, void *ctx);
+int zip_scomp_decompress(struct crypto_scomp *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen, void *ctx);
+#endif
diff --git a/drivers/crypto/cavium/zip/zip_deflate.c b/drivers/crypto/cavium/zip/zip_deflate.c
new file mode 100644
index 0000000..9a944b8
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_deflate.c
@@ -0,0 +1,200 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#include <linux/delay.h>
+#include <linux/sched.h>
+
+#include "common.h"
+#include "zip_deflate.h"
+
+/* Prepares the deflate zip command */
+static int prepare_zip_command(struct zip_operation *zip_ops,
+ struct zip_state *s, union zip_inst_s *zip_cmd)
+{
+ union zip_zres_s *result_ptr = &s->result;
+
+ memset(zip_cmd, 0, sizeof(s->zip_cmd));
+ memset(result_ptr, 0, sizeof(s->result));
+
+ /* IWORD #0 */
+ /* History gather */
+ zip_cmd->s.hg = 0;
+ /* compression enable = 1 for deflate */
+ zip_cmd->s.ce = 1;
+ /* sf (sync flush) */
+ zip_cmd->s.sf = 1;
+ /* ef (end of file) */
+ if (zip_ops->flush == ZIP_FLUSH_FINISH) {
+ zip_cmd->s.ef = 1;
+ zip_cmd->s.sf = 0;
+ }
+
+ zip_cmd->s.cc = zip_ops->ccode;
+ /* ss (compression speed/storage) */
+ zip_cmd->s.ss = zip_ops->speed;
+
+ /* IWORD #1 */
+ /* adler checksum */
+ zip_cmd->s.adlercrc32 = zip_ops->csum;
+ zip_cmd->s.historylength = zip_ops->history_len;
+ zip_cmd->s.dg = 0;
+
+ /* IWORD # 6 and 7 - compression input/history pointer */
+ zip_cmd->s.inp_ptr_addr.s.addr = __pa(zip_ops->input);
+ zip_cmd->s.inp_ptr_ctl.s.length = (zip_ops->input_len +
+ zip_ops->history_len);
+ zip_cmd->s.ds = 0;
+
+ /* IWORD # 8 and 9 - Output pointer */
+ zip_cmd->s.out_ptr_addr.s.addr = __pa(zip_ops->output);
+ zip_cmd->s.out_ptr_ctl.s.length = zip_ops->output_len;
+ /* maximum number of output-stream bytes that can be written */
+ zip_cmd->s.totaloutputlength = zip_ops->output_len;
+
+ /* IWORD # 10 and 11 - Result pointer */
+ zip_cmd->s.res_ptr_addr.s.addr = __pa(result_ptr);
+ /* Clearing completion code */
+ result_ptr->s.compcode = 0;
+
+ return 0;
+}
+
+/**
+ * zip_deflate - API to offload deflate operation to hardware
+ * @zip_ops: Pointer to zip operation structure
+ * @s: Pointer to the structure representing zip state
+ * @zip_dev: Pointer to zip device structure
+ *
+ * This function prepares the zip deflate command and submits it to the zip
+ * engine for processing.
+ *
+ * Return: 0 if successful or error code
+ */
+int zip_deflate(struct zip_operation *zip_ops, struct zip_state *s,
+ struct zip_device *zip_dev)
+{
+ union zip_inst_s *zip_cmd = &s->zip_cmd;
+ union zip_zres_s *result_ptr = &s->result;
+ u32 queue;
+
+ /* Prepares zip command based on the input parameters */
+ prepare_zip_command(zip_ops, s, zip_cmd);
+
+ atomic64_add(zip_ops->input_len, &zip_dev->stats.comp_in_bytes);
+ /* Loads zip command into command queues and rings door bell */
+ queue = zip_load_instr(zip_cmd, zip_dev);
+
+ /* Stats update for compression requests submitted */
+ atomic64_inc(&zip_dev->stats.comp_req_submit);
+
+ while (!result_ptr->s.compcode)
+ continue;
+
+ /* Stats update for compression requests completed */
+ atomic64_inc(&zip_dev->stats.comp_req_complete);
+
+ zip_ops->compcode = result_ptr->s.compcode;
+ switch (zip_ops->compcode) {
+ case ZIP_CMD_NOTDONE:
+ zip_dbg("Zip instruction not yet completed");
+ return ZIP_ERROR;
+
+ case ZIP_CMD_SUCCESS:
+ zip_dbg("Zip instruction completed successfully");
+ zip_update_cmd_bufs(zip_dev, queue);
+ break;
+
+ case ZIP_CMD_DTRUNC:
+ zip_dbg("Output Truncate error");
+ /* Returning ZIP_ERROR to avoid copy to user */
+ return ZIP_ERROR;
+
+ default:
+ zip_err("Zip instruction failed. Code:%d", zip_ops->compcode);
+ return ZIP_ERROR;
+ }
+
+ /* Update the CRC depending on the format */
+ switch (zip_ops->format) {
+ case RAW_FORMAT:
+ zip_dbg("RAW Format: %d ", zip_ops->format);
+ /* Get checksum from engine, need to feed it again */
+ zip_ops->csum = result_ptr->s.adler32;
+ break;
+
+ case ZLIB_FORMAT:
+ zip_dbg("ZLIB Format: %d ", zip_ops->format);
+ zip_ops->csum = result_ptr->s.adler32;
+ break;
+
+ case GZIP_FORMAT:
+ zip_dbg("GZIP Format: %d ", zip_ops->format);
+ zip_ops->csum = result_ptr->s.crc32;
+ break;
+
+ case LZS_FORMAT:
+ zip_dbg("LZS Format: %d ", zip_ops->format);
+ break;
+
+ default:
+ zip_err("Unknown Format:%d\n", zip_ops->format);
+ }
+
+ atomic64_add(result_ptr->s.totalbyteswritten,
+ &zip_dev->stats.comp_out_bytes);
+
+ /* Update output_len */
+ if (zip_ops->output_len < result_ptr->s.totalbyteswritten) {
+ /* Dynamic stop && strm->output_len < zipconstants[onfsize] */
+ zip_err("output_len (%d) < total bytes written(%d)\n",
+ zip_ops->output_len, result_ptr->s.totalbyteswritten);
+ zip_ops->output_len = 0;
+
+ } else {
+ zip_ops->output_len = result_ptr->s.totalbyteswritten;
+ }
+
+ return 0;
+}
diff --git a/drivers/crypto/cavium/zip/zip_deflate.h b/drivers/crypto/cavium/zip/zip_deflate.h
new file mode 100644
index 0000000..1d32e76
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_deflate.h
@@ -0,0 +1,62 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_DEFLATE_H__
+#define __ZIP_DEFLATE_H__
+
+/**
+ * zip_deflate - API to offload deflate operation to hardware
+ * @zip_ops: Pointer to zip operation structure
+ * @s: Pointer to the structure representing zip state
+ * @zip_dev: Pointer to the structure representing zip device
+ *
+ * This function prepares the zip deflate command and submits it to the zip
+ * engine by ringing the doorbell.
+ *
+ * Return: 0 if successful or error code
+ */
+int zip_deflate(struct zip_operation *zip_ops, struct zip_state *s,
+ struct zip_device *zip_dev);
+#endif
diff --git a/drivers/crypto/cavium/zip/zip_device.c b/drivers/crypto/cavium/zip/zip_device.c
new file mode 100644
index 0000000..ccf21fb
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_device.c
@@ -0,0 +1,202 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#include "common.h"
+#include "zip_deflate.h"
+
+/**
+ * zip_cmd_queue_consumed - Calculates the space consumed in the command queue.
+ *
+ * @zip_dev: Pointer to zip device structure
+ * @queue: Queue number
+ *
+ * Return: Bytes consumed in the command queue buffer.
+ */
+static inline u32 zip_cmd_queue_consumed(struct zip_device *zip_dev, int queue)
+{
+ return ((zip_dev->iq[queue].sw_head - zip_dev->iq[queue].sw_tail) *
+ sizeof(u64 *));
+}
+
+/**
+ * zip_load_instr - Submits the instruction into the ZIP command queue
+ * @instr: Pointer to the instruction to be submitted
+ * @zip_dev: Pointer to ZIP device structure to which the instruction is to
+ * be submitted
+ *
+ * This function copies the ZIP instruction to the command queue and rings the
+ * doorbell to notify the engine of the instruction submission. The command
+ * queue is maintained in a circular fashion. When there is space for exactly
+ * one instruction in the queue, next chunk pointer of the queue is made to
+ * point to the head of the queue, thus maintaining a circular queue.
+ *
+ * Return: Queue number to which the instruction was submitted
+ */
+u32 zip_load_instr(union zip_inst_s *instr,
+ struct zip_device *zip_dev)
+{
+ union zip_quex_doorbell dbell;
+ u32 queue = 0;
+ u32 consumed = 0;
+ u64 *ncb_ptr = NULL;
+ union zip_nptr_s ncp;
+
+ /*
+ * Distribute the instructions between the enabled queues based on
+ * the CPU id.
+ */
+ if (smp_processor_id() % 2 == 0)
+ queue = 0;
+ else
+ queue = 1;
+
+ zip_dbg("CPU Core: %d Queue number:%d", smp_processor_id(), queue);
+
+ /* Take cmd buffer lock */
+ spin_lock(&zip_dev->iq[queue].lock);
+
+ /*
+ * Command Queue implementation
+ * 1. If there is place for new instructions, push the cmd at sw_head.
+ * 2. If there is place for exactly one instruction, push the new cmd
+ * at the sw_head. Make sw_head point to the sw_tail to make it
+ * circular. Write sw_head's physical address to the "Next-Chunk
+ * Buffer Ptr" to make it cmd_hw_tail.
+ * 3. Ring the door bell.
+ */
+ zip_dbg("sw_head : %lx", zip_dev->iq[queue].sw_head);
+ zip_dbg("sw_tail : %lx", zip_dev->iq[queue].sw_tail);
+
+ consumed = zip_cmd_queue_consumed(zip_dev, queue);
+ /* Check if there is space to push just one cmd */
+ if ((consumed + 128) == (ZIP_CMD_QBUF_SIZE - 8)) {
+ zip_dbg("Cmd queue space available for single command");
+ /* Space for one cmd, pust it and make it circular queue */
+ memcpy((u8 *)zip_dev->iq[queue].sw_head, (u8 *)instr,
+ sizeof(union zip_inst_s));
+ zip_dev->iq[queue].sw_head += 16; /* 16 64_bit words = 128B */
+
+ /* Now, point the "Next-Chunk Buffer Ptr" to sw_head */
+ ncb_ptr = zip_dev->iq[queue].sw_head;
+
+ zip_dbg("ncb addr :0x%lx sw_head addr :0x%lx",
+ ncb_ptr, zip_dev->iq[queue].sw_head - 16);
+
+ /* Using Circular command queue */
+ zip_dev->iq[queue].sw_head = zip_dev->iq[queue].sw_tail;
+ /* Mark this buffer for free */
+ zip_dev->iq[queue].free_flag = 1;
+
+ /* Write new chunk buffer address at "Next-Chunk Buffer Ptr" */
+ ncp.u_reg64 = 0ull;
+ ncp.s.addr = __pa(zip_dev->iq[queue].sw_head);
+ *ncb_ptr = ncp.u_reg64;
+ zip_dbg("*ncb_ptr :0x%lx sw_head[phys] :0x%lx",
+ *ncb_ptr, __pa(zip_dev->iq[queue].sw_head));
+
+ zip_dev->iq[queue].pend_cnt++;
+
+ } else {
+ zip_dbg("Enough space is available for commands");
+ /* Push this cmd to cmd queue buffer */
+ memcpy((u8 *)zip_dev->iq[queue].sw_head, (u8 *)instr,
+ sizeof(union zip_inst_s));
+ zip_dev->iq[queue].sw_head += 16; /* 16 64_bit words = 128B */
+
+ zip_dev->iq[queue].pend_cnt++;
+ }
+ zip_dbg("sw_head :0x%lx sw_tail :0x%lx hw_tail :0x%lx",
+ zip_dev->iq[queue].sw_head, zip_dev->iq[queue].sw_tail,
+ zip_dev->iq[queue].hw_tail);
+
+ zip_dbg(" Pushed the new cmd : pend_cnt : %d",
+ zip_dev->iq[queue].pend_cnt);
+
+ /* Ring the doorbell */
+ dbell.u_reg64 = 0ull;
+ dbell.s.dbell_cnt = 1;
+ zip_reg_write(dbell.u_reg64,
+ (zip_dev->reg_base + ZIP_QUEX_DOORBELL(queue)));
+
+ /* Unlock cmd buffer lock */
+ spin_unlock(&zip_dev->iq[queue].lock);
+
+ return queue;
+}
+
+/**
+ * zip_update_cmd_bufs - Updates the queue statistics after posting the
+ * instruction
+ * @zip_dev: Pointer to zip device structure
+ * @queue: Queue number
+ */
+void zip_update_cmd_bufs(struct zip_device *zip_dev, u32 queue)
+{
+ /* Take cmd buffer lock */
+ spin_lock(&zip_dev->iq[queue].lock);
+
+ /* Check if the previous buffer can be freed */
+ if (zip_dev->iq[queue].free_flag == 1) {
+ zip_dbg("Free flag. Free cmd buffer, adjust sw head and tail");
+ /* Reset the free flag */
+ zip_dev->iq[queue].free_flag = 0;
+
+ /* Point the hw_tail to start of the new chunk buffer */
+ zip_dev->iq[queue].hw_tail = zip_dev->iq[queue].sw_head;
+ } else {
+ zip_dbg("Free flag not set. increment hw tail");
+ zip_dev->iq[queue].hw_tail += 16; /* 16 64_bit words = 128B */
+ }
+
+ zip_dev->iq[queue].done_cnt++;
+ zip_dev->iq[queue].pend_cnt--;
+
+ zip_dbg("sw_head :0x%lx sw_tail :0x%lx hw_tail :0x%lx",
+ zip_dev->iq[queue].sw_head, zip_dev->iq[queue].sw_tail,
+ zip_dev->iq[queue].hw_tail);
+ zip_dbg(" Got CC : pend_cnt : %d\n", zip_dev->iq[queue].pend_cnt);
+
+ spin_unlock(&zip_dev->iq[queue].lock);
+}
diff --git a/drivers/crypto/cavium/zip/zip_device.h b/drivers/crypto/cavium/zip/zip_device.h
new file mode 100644
index 0000000..9e18b3b
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_device.h
@@ -0,0 +1,108 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_DEVICE_H__
+#define __ZIP_DEVICE_H__
+
+#include <linux/types.h>
+#include "zip_main.h"
+
+struct sg_info {
+ /*
+ * Pointer to the input data when scatter_gather == 0 and
+ * pointer to the input gather list buffer when scatter_gather == 1
+ */
+ union zip_zptr_s *gather;
+
+ /*
+ * Pointer to the output data when scatter_gather == 0 and
+ * pointer to the output scatter list buffer when scatter_gather == 1
+ */
+ union zip_zptr_s *scatter;
+
+ /*
+ * Holds size of the output buffer pointed by scatter list
+ * when scatter_gather == 1
+ */
+ u64 scatter_buf_size;
+
+ /* for gather data */
+ u64 gather_enable;
+
+ /* for scatter data */
+ u64 scatter_enable;
+
+ /* Number of gather list pointers for gather data */
+ u32 gbuf_cnt;
+
+ /* Number of scatter list pointers for scatter data */
+ u32 sbuf_cnt;
+
+ /* Buffers allocation state */
+ u8 alloc_state;
+};
+
+/**
+ * struct zip_state - Structure representing the required information related
+ * to a command
+ * @zip_cmd: Pointer to zip instruction structure
+ * @result: Pointer to zip result structure
+ * @ctx: Context pointer for inflate
+ * @history: Decompression history pointer
+ * @sginfo: Scatter-gather info structure
+ */
+struct zip_state {
+ union zip_inst_s zip_cmd;
+ union zip_zres_s result;
+ union zip_zptr_s *ctx;
+ union zip_zptr_s *history;
+ struct sg_info sginfo;
+};
+
+#define ZIP_CONTEXT_SIZE 2048
+#define ZIP_INFLATE_HISTORY_SIZE 32768
+#define ZIP_DEFLATE_HISTORY_SIZE 32768
+
+#endif
diff --git a/drivers/crypto/cavium/zip/zip_inflate.c b/drivers/crypto/cavium/zip/zip_inflate.c
new file mode 100644
index 0000000..50cbdd8
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_inflate.c
@@ -0,0 +1,223 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#include <linux/delay.h>
+#include <linux/sched.h>
+
+#include "common.h"
+#include "zip_inflate.h"
+
+static int prepare_inflate_zcmd(struct zip_operation *zip_ops,
+ struct zip_state *s, union zip_inst_s *zip_cmd)
+{
+ union zip_zres_s *result_ptr = &s->result;
+
+ memset(zip_cmd, 0, sizeof(s->zip_cmd));
+ memset(result_ptr, 0, sizeof(s->result));
+
+ /* IWORD#0 */
+
+ /* Decompression History Gather list - no gather list */
+ zip_cmd->s.hg = 0;
+ /* For decompression, CE must be 0x0. */
+ zip_cmd->s.ce = 0;
+ /* For decompression, SS must be 0x0. */
+ zip_cmd->s.ss = 0;
+ /* For decompression, SF should always be set. */
+ zip_cmd->s.sf = 1;
+
+ /* Begin File */
+ if (zip_ops->begin_file == 0)
+ zip_cmd->s.bf = 0;
+ else
+ zip_cmd->s.bf = 1;
+
+ zip_cmd->s.ef = 1;
+ /* 0: for Deflate decompression, 3: for LZS decompression */
+ zip_cmd->s.cc = zip_ops->ccode;
+
+ /* IWORD #1*/
+
+ /* adler checksum */
+ zip_cmd->s.adlercrc32 = zip_ops->csum;
+
+ /*
+ * HISTORYLENGTH must be 0x0 for any ZIP decompress operation.
+ * History data is added to a decompression operation via IWORD3.
+ */
+ zip_cmd->s.historylength = 0;
+ zip_cmd->s.ds = 0;
+
+ /* IWORD # 8 and 9 - Output pointer */
+ zip_cmd->s.out_ptr_addr.s.addr = __pa(zip_ops->output);
+ zip_cmd->s.out_ptr_ctl.s.length = zip_ops->output_len;
+
+ /* Maximum number of output-stream bytes that can be written */
+ zip_cmd->s.totaloutputlength = zip_ops->output_len;
+
+ zip_dbg("Data Direct Input case ");
+
+ /* IWORD # 6 and 7 - input pointer */
+ zip_cmd->s.dg = 0;
+ zip_cmd->s.inp_ptr_addr.s.addr = __pa((u8 *)zip_ops->input);
+ zip_cmd->s.inp_ptr_ctl.s.length = zip_ops->input_len;
+
+ /* IWORD # 10 and 11 - Result pointer */
+ zip_cmd->s.res_ptr_addr.s.addr = __pa(result_ptr);
+
+ /* Clearing completion code */
+ result_ptr->s.compcode = 0;
+
+ /* Returning 0 for time being.*/
+ return 0;
+}
+
+/**
+ * zip_inflate - API to offload inflate operation to hardware
+ * @zip_ops: Pointer to zip operation structure
+ * @s: Pointer to the structure representing zip state
+ * @zip_dev: Pointer to zip device structure
+ *
+ * This function prepares the zip inflate command and submits it to the zip
+ * engine for processing.
+ *
+ * Return: 0 if successful or error code
+ */
+int zip_inflate(struct zip_operation *zip_ops, struct zip_state *s,
+ struct zip_device *zip_dev)
+{
+ union zip_inst_s *zip_cmd = &s->zip_cmd;
+ union zip_zres_s *result_ptr = &s->result;
+ u32 queue;
+
+ /* Prepare inflate zip command */
+ prepare_inflate_zcmd(zip_ops, s, zip_cmd);
+
+ atomic64_add(zip_ops->input_len, &zip_dev->stats.decomp_in_bytes);
+
+ /* Load inflate command to zip queue and ring the doorbell */
+ queue = zip_load_instr(zip_cmd, zip_dev);
+
+ /* Decompression requests submitted stats update */
+ atomic64_inc(&zip_dev->stats.decomp_req_submit);
+
+ while (!result_ptr->s.compcode)
+ continue;
+
+ /* Decompression requests completed stats update */
+ atomic64_inc(&zip_dev->stats.decomp_req_complete);
+
+ zip_ops->compcode = result_ptr->s.compcode;
+ switch (zip_ops->compcode) {
+ case ZIP_CMD_NOTDONE:
+ zip_dbg("Zip Instruction not yet completed\n");
+ return ZIP_ERROR;
+
+ case ZIP_CMD_SUCCESS:
+ zip_dbg("Zip Instruction completed successfully\n");
+ break;
+
+ case ZIP_CMD_DYNAMIC_STOP:
+ zip_dbg(" Dynamic stop Initiated\n");
+ break;
+
+ default:
+ zip_dbg("Instruction failed. Code = %d\n", zip_ops->compcode);
+ atomic64_inc(&zip_dev->stats.decomp_bad_reqs);
+ zip_update_cmd_bufs(zip_dev, queue);
+ return ZIP_ERROR;
+ }
+
+ zip_update_cmd_bufs(zip_dev, queue);
+
+ if ((zip_ops->ccode == 3) && (zip_ops->flush == 4) &&
+ (zip_ops->compcode != ZIP_CMD_DYNAMIC_STOP))
+ result_ptr->s.ef = 1;
+
+ zip_ops->csum = result_ptr->s.adler32;
+
+ atomic64_add(result_ptr->s.totalbyteswritten,
+ &zip_dev->stats.decomp_out_bytes);
+
+ if (zip_ops->output_len < result_ptr->s.totalbyteswritten) {
+ zip_err("output_len (%d) < total bytes written (%d)\n",
+ zip_ops->output_len, result_ptr->s.totalbyteswritten);
+ zip_ops->output_len = 0;
+ } else {
+ zip_ops->output_len = result_ptr->s.totalbyteswritten;
+ }
+
+ zip_ops->bytes_read = result_ptr->s.totalbytesread;
+ zip_ops->bits_processed = result_ptr->s.totalbitsprocessed;
+ zip_ops->end_file = result_ptr->s.ef;
+ if (zip_ops->end_file) {
+ switch (zip_ops->format) {
+ case RAW_FORMAT:
+ zip_dbg("RAW Format: %d ", zip_ops->format);
+ /* Get checksum from engine */
+ zip_ops->csum = result_ptr->s.adler32;
+ break;
+
+ case ZLIB_FORMAT:
+ zip_dbg("ZLIB Format: %d ", zip_ops->format);
+ zip_ops->csum = result_ptr->s.adler32;
+ break;
+
+ case GZIP_FORMAT:
+ zip_dbg("GZIP Format: %d ", zip_ops->format);
+ zip_ops->csum = result_ptr->s.crc32;
+ break;
+
+ case LZS_FORMAT:
+ zip_dbg("LZS Format: %d ", zip_ops->format);
+ break;
+
+ default:
+ zip_err("Format error:%d\n", zip_ops->format);
+ }
+ }
+
+ return 0;
+}
diff --git a/drivers/crypto/cavium/zip/zip_inflate.h b/drivers/crypto/cavium/zip/zip_inflate.h
new file mode 100644
index 0000000..6b20f17
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_inflate.h
@@ -0,0 +1,62 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_INFLATE_H__
+#define __ZIP_INFLATE_H__
+
+/**
+ * zip_inflate - API to offload inflate operation to hardware
+ * @zip_ops: Pointer to zip operation structure
+ * @s: Pointer to the structure representing zip state
+ * @zip_dev: Pointer to the structure representing zip device
+ *
+ * This function prepares the zip inflate command and submits it to the zip
+ * engine for processing.
+ *
+ * Return: 0 if successful or error code
+ */
+int zip_inflate(struct zip_operation *zip_ops, struct zip_state *s,
+ struct zip_device *zip_dev);
+#endif
diff --git a/drivers/crypto/cavium/zip/zip_main.c b/drivers/crypto/cavium/zip/zip_main.c
new file mode 100644
index 0000000..1cd8aa48
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_main.c
@@ -0,0 +1,729 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#include "common.h"
+#include "zip_crypto.h"
+
+#define DRV_NAME "ThunderX-ZIP"
+
+static struct zip_device *zip_dev[MAX_ZIP_DEVICES];
+
+static const struct pci_device_id zip_id_table[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVICE_ID_THUNDERX_ZIP) },
+ { 0, }
+};
+
+void zip_reg_write(u64 val, u64 __iomem *addr)
+{
+ writeq(val, addr);
+}
+
+u64 zip_reg_read(u64 __iomem *addr)
+{
+ return readq(addr);
+}
+
+/*
+ * Allocates new ZIP device structure
+ * Returns zip_device pointer or NULL if cannot allocate memory for zip_device
+ */
+static struct zip_device *zip_alloc_device(struct pci_dev *pdev)
+{
+ struct zip_device *zip = NULL;
+ int idx;
+
+ for (idx = 0; idx < MAX_ZIP_DEVICES; idx++) {
+ if (!zip_dev[idx])
+ break;
+ }
+
+ /* To ensure that the index is within the limit */
+ if (idx < MAX_ZIP_DEVICES)
+ zip = devm_kzalloc(&pdev->dev, sizeof(*zip), GFP_KERNEL);
+
+ if (!zip)
+ return NULL;
+
+ zip_dev[idx] = zip;
+ zip->index = idx;
+ return zip;
+}
+
+/**
+ * zip_get_device - Get ZIP device based on node id of cpu
+ *
+ * @node: Node id of the current cpu
+ * Return: Pointer to Zip device structure
+ */
+struct zip_device *zip_get_device(int node)
+{
+ if ((node < MAX_ZIP_DEVICES) && (node >= 0))
+ return zip_dev[node];
+
+ zip_err("ZIP device not found for node id %d\n", node);
+ return NULL;
+}
+
+/**
+ * zip_get_node_id - Get the node id of the current cpu
+ *
+ * Return: Node id of the current cpu
+ */
+int zip_get_node_id(void)
+{
+ return cpu_to_node(smp_processor_id());
+}
+
+/* Initializes the ZIP h/w sub-system */
+static int zip_init_hw(struct zip_device *zip)
+{
+ union zip_cmd_ctl cmd_ctl;
+ union zip_constants constants;
+ union zip_que_ena que_ena;
+ union zip_quex_map que_map;
+ union zip_que_pri que_pri;
+
+ union zip_quex_sbuf_addr que_sbuf_addr;
+ union zip_quex_sbuf_ctl que_sbuf_ctl;
+
+ int q = 0;
+
+ /* Enable the ZIP Engine(Core) Clock */
+ cmd_ctl.u_reg64 = zip_reg_read(zip->reg_base + ZIP_CMD_CTL);
+ cmd_ctl.s.forceclk = 1;
+ zip_reg_write(cmd_ctl.u_reg64 & 0xFF, (zip->reg_base + ZIP_CMD_CTL));
+
+ zip_msg("ZIP_CMD_CTL : 0x%016llx",
+ zip_reg_read(zip->reg_base + ZIP_CMD_CTL));
+
+ constants.u_reg64 = zip_reg_read(zip->reg_base + ZIP_CONSTANTS);
+ zip->depth = constants.s.depth;
+ zip->onfsize = constants.s.onfsize;
+ zip->ctxsize = constants.s.ctxsize;
+
+ zip_msg("depth: 0x%016llx , onfsize : 0x%016llx , ctxsize : 0x%016llx",
+ zip->depth, zip->onfsize, zip->ctxsize);
+
+ /*
+ * Program ZIP_QUE(0..7)_SBUF_ADDR and ZIP_QUE(0..7)_SBUF_CTL to
+ * have the correct buffer pointer and size configured for each
+ * instruction queue.
+ */
+ for (q = 0; q < ZIP_NUM_QUEUES; q++) {
+ que_sbuf_ctl.u_reg64 = 0ull;
+ que_sbuf_ctl.s.size = (ZIP_CMD_QBUF_SIZE / sizeof(u64));
+ que_sbuf_ctl.s.inst_be = 0;
+ que_sbuf_ctl.s.stream_id = 0;
+ zip_reg_write(que_sbuf_ctl.u_reg64,
+ (zip->reg_base + ZIP_QUEX_SBUF_CTL(q)));
+
+ zip_msg("QUEX_SBUF_CTL[%d]: 0x%016llx", q,
+ zip_reg_read(zip->reg_base + ZIP_QUEX_SBUF_CTL(q)));
+ }
+
+ for (q = 0; q < ZIP_NUM_QUEUES; q++) {
+ memset(&zip->iq[q], 0x0, sizeof(struct zip_iq));
+
+ spin_lock_init(&zip->iq[q].lock);
+
+ if (zip_cmd_qbuf_alloc(zip, q)) {
+ while (q != 0) {
+ q--;
+ zip_cmd_qbuf_free(zip, q);
+ }
+ return -ENOMEM;
+ }
+
+ /* Initialize tail ptr to head */
+ zip->iq[q].sw_tail = zip->iq[q].sw_head;
+ zip->iq[q].hw_tail = zip->iq[q].sw_head;
+
+ /* Write the physical addr to register */
+ que_sbuf_addr.u_reg64 = 0ull;
+ que_sbuf_addr.s.ptr = (__pa(zip->iq[q].sw_head) >>
+ ZIP_128B_ALIGN);
+
+ zip_msg("QUE[%d]_PTR(PHYS): 0x%016llx", q,
+ (u64)que_sbuf_addr.s.ptr);
+
+ zip_reg_write(que_sbuf_addr.u_reg64,
+ (zip->reg_base + ZIP_QUEX_SBUF_ADDR(q)));
+
+ zip_msg("QUEX_SBUF_ADDR[%d]: 0x%016llx", q,
+ zip_reg_read(zip->reg_base + ZIP_QUEX_SBUF_ADDR(q)));
+
+ zip_dbg("sw_head :0x%lx sw_tail :0x%lx hw_tail :0x%lx",
+ zip->iq[q].sw_head, zip->iq[q].sw_tail,
+ zip->iq[q].hw_tail);
+ zip_dbg("sw_head phy addr : 0x%lx", que_sbuf_addr.s.ptr);
+ }
+
+ /*
+ * Queue-to-ZIP core mapping
+ * If a queue is not mapped to a particular core, it is equivalent to
+ * the ZIP core being disabled.
+ */
+ que_ena.u_reg64 = 0x0ull;
+ /* Enabling queues based on ZIP_NUM_QUEUES */
+ for (q = 0; q < ZIP_NUM_QUEUES; q++)
+ que_ena.s.ena |= (0x1 << q);
+ zip_reg_write(que_ena.u_reg64, (zip->reg_base + ZIP_QUE_ENA));
+
+ zip_msg("QUE_ENA : 0x%016llx",
+ zip_reg_read(zip->reg_base + ZIP_QUE_ENA));
+
+ for (q = 0; q < ZIP_NUM_QUEUES; q++) {
+ que_map.u_reg64 = 0ull;
+ /* Mapping each queue to two ZIP cores */
+ que_map.s.zce = 0x3;
+ zip_reg_write(que_map.u_reg64,
+ (zip->reg_base + ZIP_QUEX_MAP(q)));
+
+ zip_msg("QUE_MAP(%d) : 0x%016llx", q,
+ zip_reg_read(zip->reg_base + ZIP_QUEX_MAP(q)));
+ }
+
+ que_pri.u_reg64 = 0ull;
+ for (q = 0; q < ZIP_NUM_QUEUES; q++)
+ que_pri.s.pri |= (0x1 << q); /* Higher Priority RR */
+ zip_reg_write(que_pri.u_reg64, (zip->reg_base + ZIP_QUE_PRI));
+
+ zip_msg("QUE_PRI %016llx", zip_reg_read(zip->reg_base + ZIP_QUE_PRI));
+
+ return 0;
+}
+
+static int zip_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
+{
+ struct device *dev = &pdev->dev;
+ struct zip_device *zip = NULL;
+ int err;
+
+ zip = zip_alloc_device(pdev);
+ if (!zip)
+ return -ENOMEM;
+
+ dev_info(dev, "Found ZIP device %d %x:%x on Node %d\n", zip->index,
+ pdev->vendor, pdev->device, dev_to_node(dev));
+
+ pci_set_drvdata(pdev, zip);
+ zip->pdev = pdev;
+
+ err = pci_enable_device(pdev);
+ if (err) {
+ dev_err(dev, "Failed to enable PCI device");
+ goto err_free_device;
+ }
+
+ err = pci_request_regions(pdev, DRV_NAME);
+ if (err) {
+ dev_err(dev, "PCI request regions failed 0x%x", err);
+ goto err_disable_device;
+ }
+
+ err = pci_set_dma_mask(pdev, DMA_BIT_MASK(48));
+ if (err) {
+ dev_err(dev, "Unable to get usable DMA configuration\n");
+ goto err_release_regions;
+ }
+
+ err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(48));
+ if (err) {
+ dev_err(dev, "Unable to get 48-bit DMA for allocations\n");
+ goto err_release_regions;
+ }
+
+ /* MAP configuration registers */
+ zip->reg_base = pci_ioremap_bar(pdev, PCI_CFG_ZIP_PF_BAR0);
+ if (!zip->reg_base) {
+ dev_err(dev, "ZIP: Cannot map BAR0 CSR memory space, aborting");
+ err = -ENOMEM;
+ goto err_release_regions;
+ }
+
+ /* Initialize ZIP Hardware */
+ err = zip_init_hw(zip);
+ if (err)
+ goto err_release_regions;
+
+ return 0;
+
+err_release_regions:
+ if (zip->reg_base)
+ iounmap(zip->reg_base);
+ pci_release_regions(pdev);
+
+err_disable_device:
+ pci_disable_device(pdev);
+
+err_free_device:
+ pci_set_drvdata(pdev, NULL);
+
+ /* Remove zip_dev from zip_device list, free the zip_device memory */
+ zip_dev[zip->index] = NULL;
+ devm_kfree(dev, zip);
+
+ return err;
+}
+
+static void zip_remove(struct pci_dev *pdev)
+{
+ struct zip_device *zip = pci_get_drvdata(pdev);
+ union zip_cmd_ctl cmd_ctl;
+ int q = 0;
+
+ if (!zip)
+ return;
+
+ if (zip->reg_base) {
+ cmd_ctl.u_reg64 = 0x0ull;
+ cmd_ctl.s.reset = 1; /* Forces ZIP cores to do reset */
+ zip_reg_write(cmd_ctl.u_reg64, (zip->reg_base + ZIP_CMD_CTL));
+ iounmap(zip->reg_base);
+ }
+
+ pci_release_regions(pdev);
+ pci_disable_device(pdev);
+
+ /*
+ * Free Command Queue buffers. This free should be called for all
+ * the enabled Queues.
+ */
+ for (q = 0; q < ZIP_NUM_QUEUES; q++)
+ zip_cmd_qbuf_free(zip, q);
+
+ pci_set_drvdata(pdev, NULL);
+ /* remove zip device from zip device list */
+ zip_dev[zip->index] = NULL;
+}
+
+/* PCI Sub-System Interface */
+static struct pci_driver zip_driver = {
+ .name = DRV_NAME,
+ .id_table = zip_id_table,
+ .probe = zip_probe,
+ .remove = zip_remove,
+};
+
+/* Kernel Crypto Subsystem Interface */
+
+static struct crypto_alg zip_comp_deflate = {
+ .cra_name = "deflate",
+ .cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
+ .cra_ctxsize = sizeof(struct zip_kernel_ctx),
+ .cra_priority = 300,
+ .cra_module = THIS_MODULE,
+ .cra_init = zip_alloc_comp_ctx_deflate,
+ .cra_exit = zip_free_comp_ctx,
+ .cra_u = { .compress = {
+ .coa_compress = zip_comp_compress,
+ .coa_decompress = zip_comp_decompress
+ } }
+};
+
+static struct crypto_alg zip_comp_lzs = {
+ .cra_name = "lzs",
+ .cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
+ .cra_ctxsize = sizeof(struct zip_kernel_ctx),
+ .cra_priority = 300,
+ .cra_module = THIS_MODULE,
+ .cra_init = zip_alloc_comp_ctx_lzs,
+ .cra_exit = zip_free_comp_ctx,
+ .cra_u = { .compress = {
+ .coa_compress = zip_comp_compress,
+ .coa_decompress = zip_comp_decompress
+ } }
+};
+
+static struct scomp_alg zip_scomp_deflate = {
+ .alloc_ctx = zip_alloc_scomp_ctx_deflate,
+ .free_ctx = zip_free_scomp_ctx,
+ .compress = zip_scomp_compress,
+ .decompress = zip_scomp_decompress,
+ .base = {
+ .cra_name = "deflate",
+ .cra_driver_name = "deflate-scomp",
+ .cra_module = THIS_MODULE,
+ .cra_priority = 300,
+ }
+};
+
+static struct scomp_alg zip_scomp_lzs = {
+ .alloc_ctx = zip_alloc_scomp_ctx_lzs,
+ .free_ctx = zip_free_scomp_ctx,
+ .compress = zip_scomp_compress,
+ .decompress = zip_scomp_decompress,
+ .base = {
+ .cra_name = "lzs",
+ .cra_driver_name = "lzs-scomp",
+ .cra_module = THIS_MODULE,
+ .cra_priority = 300,
+ }
+};
+
+static int zip_register_compression_device(void)
+{
+ int ret;
+
+ ret = crypto_register_alg(&zip_comp_deflate);
+ if (ret < 0) {
+ zip_err("Deflate algorithm registration failed\n");
+ return ret;
+ }
+
+ ret = crypto_register_alg(&zip_comp_lzs);
+ if (ret < 0) {
+ zip_err("LZS algorithm registration failed\n");
+ goto err_unregister_alg_deflate;
+ }
+
+ ret = crypto_register_scomp(&zip_scomp_deflate);
+ if (ret < 0) {
+ zip_err("Deflate scomp algorithm registration failed\n");
+ goto err_unregister_alg_lzs;
+ }
+
+ ret = crypto_register_scomp(&zip_scomp_lzs);
+ if (ret < 0) {
+ zip_err("LZS scomp algorithm registration failed\n");
+ goto err_unregister_scomp_deflate;
+ }
+
+ return ret;
+
+err_unregister_scomp_deflate:
+ crypto_unregister_scomp(&zip_scomp_deflate);
+err_unregister_alg_lzs:
+ crypto_unregister_alg(&zip_comp_lzs);
+err_unregister_alg_deflate:
+ crypto_unregister_alg(&zip_comp_deflate);
+
+ return ret;
+}
+
+static void zip_unregister_compression_device(void)
+{
+ crypto_unregister_alg(&zip_comp_deflate);
+ crypto_unregister_alg(&zip_comp_lzs);
+ crypto_unregister_scomp(&zip_scomp_deflate);
+ crypto_unregister_scomp(&zip_scomp_lzs);
+}
+
+/*
+ * debugfs functions
+ */
+#ifdef CONFIG_DEBUG_FS
+#include <linux/debugfs.h>
+
+/* Displays ZIP device statistics */
+static int zip_show_stats(struct seq_file *s, void *unused)
+{
+ u64 val = 0ull;
+ u64 avg_chunk = 0ull, avg_cr = 0ull;
+ u32 q = 0;
+
+ int index = 0;
+ struct zip_device *zip;
+ struct zip_stats *st;
+
+ for (index = 0; index < MAX_ZIP_DEVICES; index++) {
+ if (zip_dev[index]) {
+ zip = zip_dev[index];
+ st = &zip->stats;
+
+ /* Get all the pending requests */
+ for (q = 0; q < ZIP_NUM_QUEUES; q++) {
+ val = zip_reg_read((zip->reg_base +
+ ZIP_DBG_COREX_STA(q)));
+ val = (val >> 32);
+ val = val & 0xffffff;
+ atomic64_add(val, &st->pending_req);
+ }
+
+ avg_chunk = (atomic64_read(&st->comp_in_bytes) /
+ atomic64_read(&st->comp_req_complete));
+ avg_cr = (atomic64_read(&st->comp_in_bytes) /
+ atomic64_read(&st->comp_out_bytes));
+ seq_printf(s, " ZIP Device %d Stats\n"
+ "-----------------------------------\n"
+ "Comp Req Submitted : \t%lld\n"
+ "Comp Req Completed : \t%lld\n"
+ "Compress In Bytes : \t%lld\n"
+ "Compressed Out Bytes : \t%lld\n"
+ "Average Chunk size : \t%llu\n"
+ "Average Compression ratio : \t%llu\n"
+ "Decomp Req Submitted : \t%lld\n"
+ "Decomp Req Completed : \t%lld\n"
+ "Decompress In Bytes : \t%lld\n"
+ "Decompressed Out Bytes : \t%lld\n"
+ "Decompress Bad requests : \t%lld\n"
+ "Pending Req : \t%lld\n"
+ "---------------------------------\n",
+ index,
+ (u64)atomic64_read(&st->comp_req_submit),
+ (u64)atomic64_read(&st->comp_req_complete),
+ (u64)atomic64_read(&st->comp_in_bytes),
+ (u64)atomic64_read(&st->comp_out_bytes),
+ avg_chunk,
+ avg_cr,
+ (u64)atomic64_read(&st->decomp_req_submit),
+ (u64)atomic64_read(&st->decomp_req_complete),
+ (u64)atomic64_read(&st->decomp_in_bytes),
+ (u64)atomic64_read(&st->decomp_out_bytes),
+ (u64)atomic64_read(&st->decomp_bad_reqs),
+ (u64)atomic64_read(&st->pending_req));
+
+ /* Reset pending requests count */
+ atomic64_set(&st->pending_req, 0);
+ }
+ }
+ return 0;
+}
+
+/* Clears stats data */
+static int zip_clear_stats(struct seq_file *s, void *unused)
+{
+ int index = 0;
+
+ for (index = 0; index < MAX_ZIP_DEVICES; index++) {
+ if (zip_dev[index]) {
+ memset(&zip_dev[index]->stats, 0,
+ sizeof(struct zip_stats));
+ seq_printf(s, "Cleared stats for zip %d\n", index);
+ }
+ }
+
+ return 0;
+}
+
+static struct zip_registers zipregs[64] = {
+ {"ZIP_CMD_CTL ", 0x0000ull},
+ {"ZIP_THROTTLE ", 0x0010ull},
+ {"ZIP_CONSTANTS ", 0x00A0ull},
+ {"ZIP_QUE0_MAP ", 0x1400ull},
+ {"ZIP_QUE1_MAP ", 0x1408ull},
+ {"ZIP_QUE_ENA ", 0x0500ull},
+ {"ZIP_QUE_PRI ", 0x0508ull},
+ {"ZIP_QUE0_DONE ", 0x2000ull},
+ {"ZIP_QUE1_DONE ", 0x2008ull},
+ {"ZIP_QUE0_DOORBELL ", 0x4000ull},
+ {"ZIP_QUE1_DOORBELL ", 0x4008ull},
+ {"ZIP_QUE0_SBUF_ADDR ", 0x1000ull},
+ {"ZIP_QUE1_SBUF_ADDR ", 0x1008ull},
+ {"ZIP_QUE0_SBUF_CTL ", 0x1200ull},
+ {"ZIP_QUE1_SBUF_CTL ", 0x1208ull},
+ { NULL, 0}
+};
+
+/* Prints registers' contents */
+static int zip_print_regs(struct seq_file *s, void *unused)
+{
+ u64 val = 0;
+ int i = 0, index = 0;
+
+ for (index = 0; index < MAX_ZIP_DEVICES; index++) {
+ if (zip_dev[index]) {
+ seq_printf(s, "--------------------------------\n"
+ " ZIP Device %d Registers\n"
+ "--------------------------------\n",
+ index);
+
+ i = 0;
+
+ while (zipregs[i].reg_name) {
+ val = zip_reg_read((zip_dev[index]->reg_base +
+ zipregs[i].reg_offset));
+ seq_printf(s, "%s: 0x%016llx\n",
+ zipregs[i].reg_name, val);
+ i++;
+ }
+ }
+ }
+ return 0;
+}
+
+static int zip_stats_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, zip_show_stats, NULL);
+}
+
+static const struct file_operations zip_stats_fops = {
+ .owner = THIS_MODULE,
+ .open = zip_stats_open,
+ .read = seq_read,
+};
+
+static int zip_clear_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, zip_clear_stats, NULL);
+}
+
+static const struct file_operations zip_clear_fops = {
+ .owner = THIS_MODULE,
+ .open = zip_clear_open,
+ .read = seq_read,
+};
+
+static int zip_regs_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, zip_print_regs, NULL);
+}
+
+static const struct file_operations zip_regs_fops = {
+ .owner = THIS_MODULE,
+ .open = zip_regs_open,
+ .read = seq_read,
+};
+
+/* Root directory for thunderx_zip debugfs entry */
+static struct dentry *zip_debugfs_root;
+
+static int __init zip_debugfs_init(void)
+{
+ struct dentry *zip_stats, *zip_clear, *zip_regs;
+
+ if (!debugfs_initialized())
+ return -ENODEV;
+
+ zip_debugfs_root = debugfs_create_dir("thunderx_zip", NULL);
+ if (!zip_debugfs_root)
+ return -ENOMEM;
+
+ /* Creating files for entries inside thunderx_zip directory */
+ zip_stats = debugfs_create_file("zip_stats", 0444,
+ zip_debugfs_root,
+ NULL, &zip_stats_fops);
+ if (!zip_stats)
+ goto failed_to_create;
+
+ zip_clear = debugfs_create_file("zip_clear", 0444,
+ zip_debugfs_root,
+ NULL, &zip_clear_fops);
+ if (!zip_clear)
+ goto failed_to_create;
+
+ zip_regs = debugfs_create_file("zip_regs", 0444,
+ zip_debugfs_root,
+ NULL, &zip_regs_fops);
+ if (!zip_regs)
+ goto failed_to_create;
+
+ return 0;
+
+failed_to_create:
+ debugfs_remove_recursive(zip_debugfs_root);
+ return -ENOENT;
+}
+
+static void __exit zip_debugfs_exit(void)
+{
+ debugfs_remove_recursive(zip_debugfs_root);
+}
+
+#else
+static int __init zip_debugfs_init(void)
+{
+ return 0;
+}
+
+static void __exit zip_debugfs_exit(void) { }
+
+#endif
+/* debugfs - end */
+
+static int __init zip_init_module(void)
+{
+ int ret;
+
+ zip_msg("%s\n", DRV_NAME);
+
+ ret = pci_register_driver(&zip_driver);
+ if (ret < 0) {
+ zip_err("ZIP: pci_register_driver() failed\n");
+ return ret;
+ }
+
+ /* Register with the Kernel Crypto Interface */
+ ret = zip_register_compression_device();
+ if (ret < 0) {
+ zip_err("ZIP: Kernel Crypto Registration failed\n");
+ goto err_pci_unregister;
+ }
+
+ /* comp-decomp statistics are handled with debugfs interface */
+ ret = zip_debugfs_init();
+ if (ret < 0) {
+ zip_err("ZIP: debugfs initialization failed\n");
+ goto err_crypto_unregister;
+ }
+
+ return ret;
+
+err_crypto_unregister:
+ zip_unregister_compression_device();
+
+err_pci_unregister:
+ pci_unregister_driver(&zip_driver);
+ return ret;
+}
+
+static void __exit zip_cleanup_module(void)
+{
+ zip_debugfs_exit();
+
+ /* Unregister from the kernel crypto interface */
+ zip_unregister_compression_device();
+
+ /* Unregister this driver for pci zip devices */
+ pci_unregister_driver(&zip_driver);
+}
+
+module_init(zip_init_module);
+module_exit(zip_cleanup_module);
+
+MODULE_AUTHOR("Cavium Inc");
+MODULE_DESCRIPTION("Cavium Inc ThunderX ZIP Driver");
+MODULE_LICENSE("GPL v2");
+MODULE_DEVICE_TABLE(pci, zip_id_table);
diff --git a/drivers/crypto/cavium/zip/zip_main.h b/drivers/crypto/cavium/zip/zip_main.h
new file mode 100644
index 0000000..64e051f
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_main.h
@@ -0,0 +1,121 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_MAIN_H__
+#define __ZIP_MAIN_H__
+
+#include "zip_device.h"
+#include "zip_regs.h"
+
+/* PCI device IDs */
+#define PCI_DEVICE_ID_THUNDERX_ZIP 0xA01A
+
+/* ZIP device BARs */
+#define PCI_CFG_ZIP_PF_BAR0 0 /* Base addr for normal regs */
+
+/* Maximum available zip queues */
+#define ZIP_MAX_NUM_QUEUES 8
+
+#define ZIP_128B_ALIGN 7
+
+/* Command queue buffer size */
+#define ZIP_CMD_QBUF_SIZE (8064 + 8)
+
+struct zip_registers {
+ char *reg_name;
+ u64 reg_offset;
+};
+
+/* ZIP Compression - Decompression stats */
+struct zip_stats {
+ atomic64_t comp_req_submit;
+ atomic64_t comp_req_complete;
+ atomic64_t decomp_req_submit;
+ atomic64_t decomp_req_complete;
+ atomic64_t pending_req;
+ atomic64_t comp_in_bytes;
+ atomic64_t comp_out_bytes;
+ atomic64_t decomp_in_bytes;
+ atomic64_t decomp_out_bytes;
+ atomic64_t decomp_bad_reqs;
+};
+
+/* ZIP Instruction Queue */
+struct zip_iq {
+ u64 *sw_head;
+ u64 *sw_tail;
+ u64 *hw_tail;
+ u64 done_cnt;
+ u64 pend_cnt;
+ u64 free_flag;
+
+ /* ZIP IQ lock */
+ spinlock_t lock;
+};
+
+/* ZIP Device */
+struct zip_device {
+ u32 index;
+ void __iomem *reg_base;
+ struct pci_dev *pdev;
+
+ /* Different ZIP Constants */
+ u64 depth;
+ u64 onfsize;
+ u64 ctxsize;
+
+ struct zip_iq iq[ZIP_MAX_NUM_QUEUES];
+ struct zip_stats stats;
+};
+
+/* Prototypes */
+struct zip_device *zip_get_device(int node_id);
+int zip_get_node_id(void);
+void zip_reg_write(u64 val, u64 __iomem *addr);
+u64 zip_reg_read(u64 __iomem *addr);
+void zip_update_cmd_bufs(struct zip_device *zip_dev, u32 queue);
+u32 zip_load_instr(union zip_inst_s *instr, struct zip_device *zip_dev);
+
+#endif /* ZIP_MAIN_H */
diff --git a/drivers/crypto/cavium/zip/zip_mem.c b/drivers/crypto/cavium/zip/zip_mem.c
new file mode 100644
index 0000000..b3e0843
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_mem.c
@@ -0,0 +1,114 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#include <linux/types.h>
+#include <linux/vmalloc.h>
+
+#include "common.h"
+
+/**
+ * zip_cmd_qbuf_alloc - Allocates a cmd buffer for ZIP Instruction Queue
+ * @zip: Pointer to zip device structure
+ * @q: Queue number to allocate bufffer to
+ * Return: 0 if successful, -ENOMEM otherwise
+ */
+int zip_cmd_qbuf_alloc(struct zip_device *zip, int q)
+{
+ zip->iq[q].sw_head = (u64 *)__get_free_pages((GFP_KERNEL | GFP_DMA),
+ get_order(ZIP_CMD_QBUF_SIZE));
+
+ if (!zip->iq[q].sw_head)
+ return -ENOMEM;
+
+ memset(zip->iq[q].sw_head, 0, ZIP_CMD_QBUF_SIZE);
+
+ zip_dbg("cmd_qbuf_alloc[%d] Success : %p\n", q, zip->iq[q].sw_head);
+ return 0;
+}
+
+/**
+ * zip_cmd_qbuf_free - Frees the cmd Queue buffer
+ * @zip: Pointer to zip device structure
+ * @q: Queue number to free buffer of
+ */
+void zip_cmd_qbuf_free(struct zip_device *zip, int q)
+{
+ zip_dbg("Freeing cmd_qbuf 0x%lx\n", zip->iq[q].sw_tail);
+
+ free_pages((u64)zip->iq[q].sw_tail, get_order(ZIP_CMD_QBUF_SIZE));
+}
+
+/**
+ * zip_data_buf_alloc - Allocates memory for a data bufffer
+ * @size: Size of the buffer to allocate
+ * Returns: Pointer to the buffer allocated
+ */
+u8 *zip_data_buf_alloc(u64 size)
+{
+ u8 *ptr;
+
+ ptr = (u8 *)__get_free_pages((GFP_KERNEL | GFP_DMA),
+ get_order(size));
+
+ if (!ptr)
+ return NULL;
+
+ memset(ptr, 0, size);
+
+ zip_dbg("Data buffer allocation success\n");
+ return ptr;
+}
+
+/**
+ * zip_data_buf_free - Frees the memory of a data buffer
+ * @ptr: Pointer to the buffer
+ * @size: Buffer size
+ */
+void zip_data_buf_free(u8 *ptr, u64 size)
+{
+ zip_dbg("Freeing data buffer 0x%lx\n", ptr);
+
+ free_pages((u64)ptr, get_order(size));
+}
diff --git a/drivers/crypto/cavium/zip/zip_mem.h b/drivers/crypto/cavium/zip/zip_mem.h
new file mode 100644
index 0000000..f8f2f08
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_mem.h
@@ -0,0 +1,78 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_MEM_H__
+#define __ZIP_MEM_H__
+
+/**
+ * zip_cmd_qbuf_free - Frees the cmd Queue buffer
+ * @zip: Pointer to zip device structure
+ * @q: Queue nmber to free buffer of
+ */
+void zip_cmd_qbuf_free(struct zip_device *zip, int q);
+
+/**
+ * zip_cmd_qbuf_alloc - Allocates a Chunk/cmd buffer for ZIP Inst(cmd) Queue
+ * @zip: Pointer to zip device structure
+ * @q: Queue number to allocate bufffer to
+ * Return: 0 if successful, 1 otherwise
+ */
+int zip_cmd_qbuf_alloc(struct zip_device *zip, int q);
+
+/**
+ * zip_data_buf_alloc - Allocates memory for a data bufffer
+ * @size: Size of the buffer to allocate
+ * Returns: Pointer to the buffer allocated
+ */
+u8 *zip_data_buf_alloc(u64 size);
+
+/**
+ * zip_data_buf_free - Frees the memory of a data buffer
+ * @ptr: Pointer to the buffer
+ * @size: Buffer size
+ */
+void zip_data_buf_free(u8 *ptr, u64 size);
+
+#endif
diff --git a/drivers/crypto/cavium/zip/zip_regs.h b/drivers/crypto/cavium/zip/zip_regs.h
new file mode 100644
index 0000000..d0be682
--- /dev/null
+++ b/drivers/crypto/cavium/zip/zip_regs.h
@@ -0,0 +1,1347 @@
+/***********************license start************************************
+ * Copyright (c) 2003-2017 Cavium, Inc.
+ * All rights reserved.
+ *
+ * License: one of 'Cavium License' or 'GNU General Public License Version 2'
+ *
+ * This file is provided under the terms of the Cavium License (see below)
+ * or under the terms of GNU General Public License, Version 2, as
+ * published by the Free Software Foundation. When using or redistributing
+ * this file, you may do so under either license.
+ *
+ * Cavium License: Redistribution and use in source and binary forms, with
+ * or without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ *
+ * * Neither the name of Cavium Inc. nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * This Software, including technical data, may be subject to U.S. export
+ * control laws, including the U.S. Export Administration Act and its
+ * associated regulations, and may be subject to export or import
+ * regulations in other countries.
+ *
+ * TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS"
+ * AND WITH ALL FAULTS AND CAVIUM INC. MAKES NO PROMISES, REPRESENTATIONS
+ * OR WARRANTIES, EITHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH
+ * RESPECT TO THE SOFTWARE, INCLUDING ITS CONDITION, ITS CONFORMITY TO ANY
+ * REPRESENTATION OR DESCRIPTION, OR THE EXISTENCE OF ANY LATENT OR PATENT
+ * DEFECTS, AND CAVIUM SPECIFICALLY DISCLAIMS ALL IMPLIED (IF ANY)
+ * WARRANTIES OF TITLE, MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR A
+ * PARTICULAR PURPOSE, LACK OF VIRUSES, ACCURACY OR COMPLETENESS, QUIET
+ * ENJOYMENT, QUIET POSSESSION OR CORRESPONDENCE TO DESCRIPTION. THE
+ * ENTIRE RISK ARISING OUT OF USE OR PERFORMANCE OF THE SOFTWARE LIES
+ * WITH YOU.
+ ***********************license end**************************************/
+
+#ifndef __ZIP_REGS_H__
+#define __ZIP_REGS_H__
+
+/*
+ * Configuration and status register (CSR) address and type definitions for
+ * Cavium ZIP.
+ */
+
+#include <linux/kern_levels.h>
+
+/* ZIP invocation result completion status codes */
+#define ZIP_CMD_NOTDONE 0x0
+
+/* Successful completion. */
+#define ZIP_CMD_SUCCESS 0x1
+
+/* Output truncated */
+#define ZIP_CMD_DTRUNC 0x2
+
+/* Dynamic Stop */
+#define ZIP_CMD_DYNAMIC_STOP 0x3
+
+/* Uncompress ran out of input data when IWORD0[EF] was set */
+#define ZIP_CMD_ITRUNC 0x4
+
+/* Uncompress found the reserved block type 3 */
+#define ZIP_CMD_RBLOCK 0x5
+
+/*
+ * Uncompress found LEN != ZIP_CMD_NLEN in an uncompressed block in the input.
+ */
+#define ZIP_CMD_NLEN 0x6
+
+/* Uncompress found a bad code in the main Huffman codes. */
+#define ZIP_CMD_BADCODE 0x7
+
+/* Uncompress found a bad code in the 19 Huffman codes encoding lengths. */
+#define ZIP_CMD_BADCODE2 0x8
+
+/* Compress found a zero-length input. */
+#define ZIP_CMD_ZERO_LEN 0x9
+
+/* The compress or decompress encountered an internal parity error. */
+#define ZIP_CMD_PARITY 0xA
+
+/*
+ * Uncompress found a string identifier that precedes the uncompressed data and
+ * decompression history.
+ */
+#define ZIP_CMD_FATAL 0xB
+
+/**
+ * enum zip_int_vec_e - ZIP MSI-X Vector Enumeration, enumerates the MSI-X
+ * interrupt vectors.
+ */
+enum zip_int_vec_e {
+ ZIP_INT_VEC_E_ECCE = 0x10,
+ ZIP_INT_VEC_E_FIFE = 0x11,
+ ZIP_INT_VEC_E_QUE0_DONE = 0x0,
+ ZIP_INT_VEC_E_QUE0_ERR = 0x8,
+ ZIP_INT_VEC_E_QUE1_DONE = 0x1,
+ ZIP_INT_VEC_E_QUE1_ERR = 0x9,
+ ZIP_INT_VEC_E_QUE2_DONE = 0x2,
+ ZIP_INT_VEC_E_QUE2_ERR = 0xa,
+ ZIP_INT_VEC_E_QUE3_DONE = 0x3,
+ ZIP_INT_VEC_E_QUE3_ERR = 0xb,
+ ZIP_INT_VEC_E_QUE4_DONE = 0x4,
+ ZIP_INT_VEC_E_QUE4_ERR = 0xc,
+ ZIP_INT_VEC_E_QUE5_DONE = 0x5,
+ ZIP_INT_VEC_E_QUE5_ERR = 0xd,
+ ZIP_INT_VEC_E_QUE6_DONE = 0x6,
+ ZIP_INT_VEC_E_QUE6_ERR = 0xe,
+ ZIP_INT_VEC_E_QUE7_DONE = 0x7,
+ ZIP_INT_VEC_E_QUE7_ERR = 0xf,
+ ZIP_INT_VEC_E_ENUM_LAST = 0x12,
+};
+
+/**
+ * union zip_zptr_addr_s - ZIP Generic Pointer Structure for ADDR.
+ *
+ * It is the generic format of pointers in ZIP_INST_S.
+ */
+union zip_zptr_addr_s {
+ u64 u_reg64;
+ struct {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_49_63 : 15;
+ u64 addr : 49;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 addr : 49;
+ u64 reserved_49_63 : 15;
+#endif
+ } s;
+
+};
+
+/**
+ * union zip_zptr_ctl_s - ZIP Generic Pointer Structure for CTL.
+ *
+ * It is the generic format of pointers in ZIP_INST_S.
+ */
+union zip_zptr_ctl_s {
+ u64 u_reg64;
+ struct {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_112_127 : 16;
+ u64 length : 16;
+ u64 reserved_67_95 : 29;
+ u64 fw : 1;
+ u64 nc : 1;
+ u64 data_be : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 data_be : 1;
+ u64 nc : 1;
+ u64 fw : 1;
+ u64 reserved_67_95 : 29;
+ u64 length : 16;
+ u64 reserved_112_127 : 16;
+#endif
+ } s;
+};
+
+/**
+ * union zip_inst_s - ZIP Instruction Structure.
+ * Each ZIP instruction has 16 words (they are called IWORD0 to IWORD15 within
+ * the structure).
+ */
+union zip_inst_s {
+ u64 u_reg64[16];
+ struct {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 doneint : 1;
+ u64 reserved_56_62 : 7;
+ u64 totaloutputlength : 24;
+ u64 reserved_27_31 : 5;
+ u64 exn : 3;
+ u64 reserved_23_23 : 1;
+ u64 exbits : 7;
+ u64 reserved_12_15 : 4;
+ u64 sf : 1;
+ u64 ss : 2;
+ u64 cc : 2;
+ u64 ef : 1;
+ u64 bf : 1;
+ u64 ce : 1;
+ u64 reserved_3_3 : 1;
+ u64 ds : 1;
+ u64 dg : 1;
+ u64 hg : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 hg : 1;
+ u64 dg : 1;
+ u64 ds : 1;
+ u64 reserved_3_3 : 1;
+ u64 ce : 1;
+ u64 bf : 1;
+ u64 ef : 1;
+ u64 cc : 2;
+ u64 ss : 2;
+ u64 sf : 1;
+ u64 reserved_12_15 : 4;
+ u64 exbits : 7;
+ u64 reserved_23_23 : 1;
+ u64 exn : 3;
+ u64 reserved_27_31 : 5;
+ u64 totaloutputlength : 24;
+ u64 reserved_56_62 : 7;
+ u64 doneint : 1;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 historylength : 16;
+ u64 reserved_96_111 : 16;
+ u64 adlercrc32 : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 adlercrc32 : 32;
+ u64 reserved_96_111 : 16;
+ u64 historylength : 16;
+#endif
+ union zip_zptr_addr_s ctx_ptr_addr;
+ union zip_zptr_ctl_s ctx_ptr_ctl;
+ union zip_zptr_addr_s his_ptr_addr;
+ union zip_zptr_ctl_s his_ptr_ctl;
+ union zip_zptr_addr_s inp_ptr_addr;
+ union zip_zptr_ctl_s inp_ptr_ctl;
+ union zip_zptr_addr_s out_ptr_addr;
+ union zip_zptr_ctl_s out_ptr_ctl;
+ union zip_zptr_addr_s res_ptr_addr;
+ union zip_zptr_ctl_s res_ptr_ctl;
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_817_831 : 15;
+ u64 wq_ptr : 49;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 wq_ptr : 49;
+ u64 reserved_817_831 : 15;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_882_895 : 14;
+ u64 tt : 2;
+ u64 reserved_874_879 : 6;
+ u64 grp : 10;
+ u64 tag : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 tag : 32;
+ u64 grp : 10;
+ u64 reserved_874_879 : 6;
+ u64 tt : 2;
+ u64 reserved_882_895 : 14;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_896_959 : 64;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 reserved_896_959 : 64;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_960_1023 : 64;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 reserved_960_1023 : 64;
+#endif
+ } s;
+};
+
+/**
+ * union zip_nptr_s - ZIP Instruction Next-Chunk-Buffer Pointer (NPTR)
+ * Structure
+ *
+ * ZIP_NPTR structure is used to chain all the zip instruction buffers
+ * together. ZIP instruction buffers are managed (allocated and released) by
+ * the software.
+ */
+union zip_nptr_s {
+ u64 u_reg64;
+ struct {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_49_63 : 15;
+ u64 addr : 49;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 addr : 49;
+ u64 reserved_49_63 : 15;
+#endif
+ } s;
+};
+
+/**
+ * union zip_zptr_s - ZIP Generic Pointer Structure.
+ *
+ * It is the generic format of pointers in ZIP_INST_S.
+ */
+union zip_zptr_s {
+ u64 u_reg64[2];
+ struct {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_49_63 : 15;
+ u64 addr : 49;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 addr : 49;
+ u64 reserved_49_63 : 15;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_112_127 : 16;
+ u64 length : 16;
+ u64 reserved_67_95 : 29;
+ u64 fw : 1;
+ u64 nc : 1;
+ u64 data_be : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 data_be : 1;
+ u64 nc : 1;
+ u64 fw : 1;
+ u64 reserved_67_95 : 29;
+ u64 length : 16;
+ u64 reserved_112_127 : 16;
+#endif
+ } s;
+};
+
+/**
+ * union zip_zres_s - ZIP Result Structure
+ *
+ * The ZIP coprocessor writes the result structure after it completes the
+ * invocation. The result structure is exactly 24 bytes, and each invocation of
+ * the ZIP coprocessor produces exactly one result structure.
+ */
+union zip_zres_s {
+ u64 u_reg64[3];
+ struct {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 crc32 : 32;
+ u64 adler32 : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 adler32 : 32;
+ u64 crc32 : 32;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 totalbyteswritten : 32;
+ u64 totalbytesread : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 totalbytesread : 32;
+ u64 totalbyteswritten : 32;
+#endif
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 totalbitsprocessed : 32;
+ u64 doneint : 1;
+ u64 reserved_155_158 : 4;
+ u64 exn : 3;
+ u64 reserved_151_151 : 1;
+ u64 exbits : 7;
+ u64 reserved_137_143 : 7;
+ u64 ef : 1;
+
+ volatile u64 compcode : 8;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+
+ volatile u64 compcode : 8;
+ u64 ef : 1;
+ u64 reserved_137_143 : 7;
+ u64 exbits : 7;
+ u64 reserved_151_151 : 1;
+ u64 exn : 3;
+ u64 reserved_155_158 : 4;
+ u64 doneint : 1;
+ u64 totalbitsprocessed : 32;
+#endif
+ } s;
+};
+
+/**
+ * union zip_cmd_ctl - Structure representing the register that controls
+ * clock and reset.
+ */
+union zip_cmd_ctl {
+ u64 u_reg64;
+ struct zip_cmd_ctl_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_2_63 : 62;
+ u64 forceclk : 1;
+ u64 reset : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 reset : 1;
+ u64 forceclk : 1;
+ u64 reserved_2_63 : 62;
+#endif
+ } s;
+};
+
+#define ZIP_CMD_CTL 0x0ull
+
+/**
+ * union zip_constants - Data structure representing the register that contains
+ * all of the current implementation-related parameters of the zip core in this
+ * chip.
+ */
+union zip_constants {
+ u64 u_reg64;
+ struct zip_constants_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 nexec : 8;
+ u64 reserved_49_55 : 7;
+ u64 syncflush_capable : 1;
+ u64 depth : 16;
+ u64 onfsize : 12;
+ u64 ctxsize : 12;
+ u64 reserved_1_7 : 7;
+ u64 disabled : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 disabled : 1;
+ u64 reserved_1_7 : 7;
+ u64 ctxsize : 12;
+ u64 onfsize : 12;
+ u64 depth : 16;
+ u64 syncflush_capable : 1;
+ u64 reserved_49_55 : 7;
+ u64 nexec : 8;
+#endif
+ } s;
+};
+
+#define ZIP_CONSTANTS 0x00A0ull
+
+/**
+ * union zip_corex_bist_status - Represents registers which have the BIST
+ * status of memories in zip cores.
+ *
+ * Each bit is the BIST result of an individual memory
+ * (per bit, 0 = pass and 1 = fail).
+ */
+union zip_corex_bist_status {
+ u64 u_reg64;
+ struct zip_corex_bist_status_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_53_63 : 11;
+ u64 bstatus : 53;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 bstatus : 53;
+ u64 reserved_53_63 : 11;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_COREX_BIST_STATUS(u64 param1)
+{
+ if (((param1 <= 1)))
+ return 0x0520ull + (param1 & 1) * 0x8ull;
+ pr_err("ZIP_COREX_BIST_STATUS: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_ctl_bist_status - Represents register that has the BIST status of
+ * memories in ZIP_CTL (instruction buffer, G/S pointer FIFO, input data
+ * buffer, output data buffers).
+ *
+ * Each bit is the BIST result of an individual memory
+ * (per bit, 0 = pass and 1 = fail).
+ */
+union zip_ctl_bist_status {
+ u64 u_reg64;
+ struct zip_ctl_bist_status_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_9_63 : 55;
+ u64 bstatus : 9;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 bstatus : 9;
+ u64 reserved_9_63 : 55;
+#endif
+ } s;
+};
+
+#define ZIP_CTL_BIST_STATUS 0x0510ull
+
+/**
+ * union zip_ctl_cfg - Represents the register that controls the behavior of
+ * the ZIP DMA engines.
+ *
+ * It is recommended to keep default values for normal operation. Changing the
+ * values of the fields may be useful for diagnostics.
+ */
+union zip_ctl_cfg {
+ u64 u_reg64;
+ struct zip_ctl_cfg_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_52_63 : 12;
+ u64 ildf : 4;
+ u64 reserved_36_47 : 12;
+ u64 drtf : 4;
+ u64 reserved_27_31 : 5;
+ u64 stcf : 3;
+ u64 reserved_19_23 : 5;
+ u64 ldf : 3;
+ u64 reserved_2_15 : 14;
+ u64 busy : 1;
+ u64 reserved_0_0 : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 reserved_0_0 : 1;
+ u64 busy : 1;
+ u64 reserved_2_15 : 14;
+ u64 ldf : 3;
+ u64 reserved_19_23 : 5;
+ u64 stcf : 3;
+ u64 reserved_27_31 : 5;
+ u64 drtf : 4;
+ u64 reserved_36_47 : 12;
+ u64 ildf : 4;
+ u64 reserved_52_63 : 12;
+#endif
+ } s;
+};
+
+#define ZIP_CTL_CFG 0x0560ull
+
+/**
+ * union zip_dbg_corex_inst - Represents the registers that reflect the status
+ * of the current instruction that the ZIP core is executing or has executed.
+ *
+ * These registers are only for debug use.
+ */
+union zip_dbg_corex_inst {
+ u64 u_reg64;
+ struct zip_dbg_corex_inst_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 busy : 1;
+ u64 reserved_35_62 : 28;
+ u64 qid : 3;
+ u64 iid : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 iid : 32;
+ u64 qid : 3;
+ u64 reserved_35_62 : 28;
+ u64 busy : 1;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_DBG_COREX_INST(u64 param1)
+{
+ if (((param1 <= 1)))
+ return 0x0640ull + (param1 & 1) * 0x8ull;
+ pr_err("ZIP_DBG_COREX_INST: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_dbg_corex_sta - Represents registers that reflect the status of
+ * the zip cores.
+ *
+ * They are for debug use only.
+ */
+union zip_dbg_corex_sta {
+ u64 u_reg64;
+ struct zip_dbg_corex_sta_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 busy : 1;
+ u64 reserved_37_62 : 26;
+ u64 ist : 5;
+ u64 nie : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 nie : 32;
+ u64 ist : 5;
+ u64 reserved_37_62 : 26;
+ u64 busy : 1;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_DBG_COREX_STA(u64 param1)
+{
+ if (((param1 <= 1)))
+ return 0x0680ull + (param1 & 1) * 0x8ull;
+ pr_err("ZIP_DBG_COREX_STA: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_dbg_quex_sta - Represets registers that reflect status of the zip
+ * instruction queues.
+ *
+ * They are for debug use only.
+ */
+union zip_dbg_quex_sta {
+ u64 u_reg64;
+ struct zip_dbg_quex_sta_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 busy : 1;
+ u64 reserved_56_62 : 7;
+ u64 rqwc : 24;
+ u64 nii : 32;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 nii : 32;
+ u64 rqwc : 24;
+ u64 reserved_56_62 : 7;
+ u64 busy : 1;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_DBG_QUEX_STA(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x1800ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_DBG_QUEX_STA: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_ecc_ctl - Represents the register that enables ECC for each
+ * individual internal memory that requires ECC.
+ *
+ * For debug purpose, it can also flip one or two bits in the ECC data.
+ */
+union zip_ecc_ctl {
+ u64 u_reg64;
+ struct zip_ecc_ctl_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_19_63 : 45;
+ u64 vmem_cdis : 1;
+ u64 vmem_fs : 2;
+ u64 reserved_15_15 : 1;
+ u64 idf1_cdis : 1;
+ u64 idf1_fs : 2;
+ u64 reserved_11_11 : 1;
+ u64 idf0_cdis : 1;
+ u64 idf0_fs : 2;
+ u64 reserved_7_7 : 1;
+ u64 gspf_cdis : 1;
+ u64 gspf_fs : 2;
+ u64 reserved_3_3 : 1;
+ u64 iqf_cdis : 1;
+ u64 iqf_fs : 2;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 iqf_fs : 2;
+ u64 iqf_cdis : 1;
+ u64 reserved_3_3 : 1;
+ u64 gspf_fs : 2;
+ u64 gspf_cdis : 1;
+ u64 reserved_7_7 : 1;
+ u64 idf0_fs : 2;
+ u64 idf0_cdis : 1;
+ u64 reserved_11_11 : 1;
+ u64 idf1_fs : 2;
+ u64 idf1_cdis : 1;
+ u64 reserved_15_15 : 1;
+ u64 vmem_fs : 2;
+ u64 vmem_cdis : 1;
+ u64 reserved_19_63 : 45;
+#endif
+ } s;
+};
+
+#define ZIP_ECC_CTL 0x0568ull
+
+/* NCB - zip_ecce_ena_w1c */
+union zip_ecce_ena_w1c {
+ u64 u_reg64;
+ struct zip_ecce_ena_w1c_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_37_63 : 27;
+ u64 dbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 sbe : 5;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 sbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 dbe : 5;
+ u64 reserved_37_63 : 27;
+#endif
+ } s;
+};
+
+#define ZIP_ECCE_ENA_W1C 0x0598ull
+
+/* NCB - zip_ecce_ena_w1s */
+union zip_ecce_ena_w1s {
+ u64 u_reg64;
+ struct zip_ecce_ena_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_37_63 : 27;
+ u64 dbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 sbe : 5;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 sbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 dbe : 5;
+ u64 reserved_37_63 : 27;
+#endif
+ } s;
+};
+
+#define ZIP_ECCE_ENA_W1S 0x0590ull
+
+/**
+ * union zip_ecce_int - Represents the register that contains the status of the
+ * ECC interrupt sources.
+ */
+union zip_ecce_int {
+ u64 u_reg64;
+ struct zip_ecce_int_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_37_63 : 27;
+ u64 dbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 sbe : 5;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 sbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 dbe : 5;
+ u64 reserved_37_63 : 27;
+#endif
+ } s;
+};
+
+#define ZIP_ECCE_INT 0x0580ull
+
+/* NCB - zip_ecce_int_w1s */
+union zip_ecce_int_w1s {
+ u64 u_reg64;
+ struct zip_ecce_int_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_37_63 : 27;
+ u64 dbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 sbe : 5;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 sbe : 5;
+ u64 reserved_5_31 : 27;
+ u64 dbe : 5;
+ u64 reserved_37_63 : 27;
+#endif
+ } s;
+};
+
+#define ZIP_ECCE_INT_W1S 0x0588ull
+
+/* NCB - zip_fife_ena_w1c */
+union zip_fife_ena_w1c {
+ u64 u_reg64;
+ struct zip_fife_ena_w1c_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_42_63 : 22;
+ u64 asserts : 42;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 asserts : 42;
+ u64 reserved_42_63 : 22;
+#endif
+ } s;
+};
+
+#define ZIP_FIFE_ENA_W1C 0x0090ull
+
+/* NCB - zip_fife_ena_w1s */
+union zip_fife_ena_w1s {
+ u64 u_reg64;
+ struct zip_fife_ena_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_42_63 : 22;
+ u64 asserts : 42;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 asserts : 42;
+ u64 reserved_42_63 : 22;
+#endif
+ } s;
+};
+
+#define ZIP_FIFE_ENA_W1S 0x0088ull
+
+/* NCB - zip_fife_int */
+union zip_fife_int {
+ u64 u_reg64;
+ struct zip_fife_int_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_42_63 : 22;
+ u64 asserts : 42;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 asserts : 42;
+ u64 reserved_42_63 : 22;
+#endif
+ } s;
+};
+
+#define ZIP_FIFE_INT 0x0078ull
+
+/* NCB - zip_fife_int_w1s */
+union zip_fife_int_w1s {
+ u64 u_reg64;
+ struct zip_fife_int_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_42_63 : 22;
+ u64 asserts : 42;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 asserts : 42;
+ u64 reserved_42_63 : 22;
+#endif
+ } s;
+};
+
+#define ZIP_FIFE_INT_W1S 0x0080ull
+
+/**
+ * union zip_msix_pbax - Represents the register that is the MSI-X PBA table
+ *
+ * The bit number is indexed by the ZIP_INT_VEC_E enumeration.
+ */
+union zip_msix_pbax {
+ u64 u_reg64;
+ struct zip_msix_pbax_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 pend : 64;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 pend : 64;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_MSIX_PBAX(u64 param1)
+{
+ if (((param1 == 0)))
+ return 0x0000838000FF0000ull;
+ pr_err("ZIP_MSIX_PBAX: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_msix_vecx_addr - Represents the register that is the MSI-X vector
+ * table, indexed by the ZIP_INT_VEC_E enumeration.
+ */
+union zip_msix_vecx_addr {
+ u64 u_reg64;
+ struct zip_msix_vecx_addr_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_49_63 : 15;
+ u64 addr : 47;
+ u64 reserved_1_1 : 1;
+ u64 secvec : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 secvec : 1;
+ u64 reserved_1_1 : 1;
+ u64 addr : 47;
+ u64 reserved_49_63 : 15;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_MSIX_VECX_ADDR(u64 param1)
+{
+ if (((param1 <= 17)))
+ return 0x0000838000F00000ull + (param1 & 31) * 0x10ull;
+ pr_err("ZIP_MSIX_VECX_ADDR: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_msix_vecx_ctl - Represents the register that is the MSI-X vector
+ * table, indexed by the ZIP_INT_VEC_E enumeration.
+ */
+union zip_msix_vecx_ctl {
+ u64 u_reg64;
+ struct zip_msix_vecx_ctl_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_33_63 : 31;
+ u64 mask : 1;
+ u64 reserved_20_31 : 12;
+ u64 data : 20;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 data : 20;
+ u64 reserved_20_31 : 12;
+ u64 mask : 1;
+ u64 reserved_33_63 : 31;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_MSIX_VECX_CTL(u64 param1)
+{
+ if (((param1 <= 17)))
+ return 0x0000838000F00008ull + (param1 & 31) * 0x10ull;
+ pr_err("ZIP_MSIX_VECX_CTL: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_done - Represents the registers that contain the per-queue
+ * instruction done count.
+ */
+union zip_quex_done {
+ u64 u_reg64;
+ struct zip_quex_done_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_20_63 : 44;
+ u64 done : 20;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 done : 20;
+ u64 reserved_20_63 : 44;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_DONE(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x2000ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_DONE: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_done_ack - Represents the registers on write to which will
+ * decrement the per-queue instructiona done count.
+ */
+union zip_quex_done_ack {
+ u64 u_reg64;
+ struct zip_quex_done_ack_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_20_63 : 44;
+ u64 done_ack : 20;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 done_ack : 20;
+ u64 reserved_20_63 : 44;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_DONE_ACK(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x2200ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_DONE_ACK: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_done_ena_w1c - Represents the register which when written
+ * 1 to will disable the DONEINT interrupt for the queue.
+ */
+union zip_quex_done_ena_w1c {
+ u64 u_reg64;
+ struct zip_quex_done_ena_w1c_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_1_63 : 63;
+ u64 done_ena : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 done_ena : 1;
+ u64 reserved_1_63 : 63;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_DONE_ENA_W1C(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x2600ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_DONE_ENA_W1C: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_done_ena_w1s - Represents the register that when written 1 to
+ * will enable the DONEINT interrupt for the queue.
+ */
+union zip_quex_done_ena_w1s {
+ u64 u_reg64;
+ struct zip_quex_done_ena_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_1_63 : 63;
+ u64 done_ena : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 done_ena : 1;
+ u64 reserved_1_63 : 63;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_DONE_ENA_W1S(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x2400ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_DONE_ENA_W1S: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_done_wait - Represents the register that specifies the per
+ * queue interrupt coalescing settings.
+ */
+union zip_quex_done_wait {
+ u64 u_reg64;
+ struct zip_quex_done_wait_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_48_63 : 16;
+ u64 time_wait : 16;
+ u64 reserved_20_31 : 12;
+ u64 num_wait : 20;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 num_wait : 20;
+ u64 reserved_20_31 : 12;
+ u64 time_wait : 16;
+ u64 reserved_48_63 : 16;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_DONE_WAIT(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x2800ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_DONE_WAIT: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_doorbell - Represents doorbell registers for the ZIP
+ * instruction queues.
+ */
+union zip_quex_doorbell {
+ u64 u_reg64;
+ struct zip_quex_doorbell_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_20_63 : 44;
+ u64 dbell_cnt : 20;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 dbell_cnt : 20;
+ u64 reserved_20_63 : 44;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_DOORBELL(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x4000ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_DOORBELL: %llu\n", param1);
+ return 0;
+}
+
+union zip_quex_err_ena_w1c {
+ u64 u_reg64;
+ struct zip_quex_err_ena_w1c_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_5_63 : 59;
+ u64 mdbe : 1;
+ u64 nwrp : 1;
+ u64 nrrp : 1;
+ u64 irde : 1;
+ u64 dovf : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 dovf : 1;
+ u64 irde : 1;
+ u64 nrrp : 1;
+ u64 nwrp : 1;
+ u64 mdbe : 1;
+ u64 reserved_5_63 : 59;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_ERR_ENA_W1C(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x3600ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_ERR_ENA_W1C: %llu\n", param1);
+ return 0;
+}
+
+union zip_quex_err_ena_w1s {
+ u64 u_reg64;
+ struct zip_quex_err_ena_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_5_63 : 59;
+ u64 mdbe : 1;
+ u64 nwrp : 1;
+ u64 nrrp : 1;
+ u64 irde : 1;
+ u64 dovf : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 dovf : 1;
+ u64 irde : 1;
+ u64 nrrp : 1;
+ u64 nwrp : 1;
+ u64 mdbe : 1;
+ u64 reserved_5_63 : 59;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_ERR_ENA_W1S(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x3400ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_ERR_ENA_W1S: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_err_int - Represents registers that contain the per-queue
+ * error interrupts.
+ */
+union zip_quex_err_int {
+ u64 u_reg64;
+ struct zip_quex_err_int_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_5_63 : 59;
+ u64 mdbe : 1;
+ u64 nwrp : 1;
+ u64 nrrp : 1;
+ u64 irde : 1;
+ u64 dovf : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 dovf : 1;
+ u64 irde : 1;
+ u64 nrrp : 1;
+ u64 nwrp : 1;
+ u64 mdbe : 1;
+ u64 reserved_5_63 : 59;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_ERR_INT(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x3000ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_ERR_INT: %llu\n", param1);
+ return 0;
+}
+
+/* NCB - zip_que#_err_int_w1s */
+union zip_quex_err_int_w1s {
+ u64 u_reg64;
+ struct zip_quex_err_int_w1s_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_5_63 : 59;
+ u64 mdbe : 1;
+ u64 nwrp : 1;
+ u64 nrrp : 1;
+ u64 irde : 1;
+ u64 dovf : 1;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 dovf : 1;
+ u64 irde : 1;
+ u64 nrrp : 1;
+ u64 nwrp : 1;
+ u64 mdbe : 1;
+ u64 reserved_5_63 : 59;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_ERR_INT_W1S(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x3200ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_ERR_INT_W1S: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_gcfg - Represents the registers that reflect status of the
+ * zip instruction queues,debug use only.
+ */
+union zip_quex_gcfg {
+ u64 u_reg64;
+ struct zip_quex_gcfg_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_4_63 : 60;
+ u64 iqb_ldwb : 1;
+ u64 cbw_sty : 1;
+ u64 l2ld_cmd : 2;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 l2ld_cmd : 2;
+ u64 cbw_sty : 1;
+ u64 iqb_ldwb : 1;
+ u64 reserved_4_63 : 60;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_GCFG(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x1A00ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_GCFG: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_map - Represents the registers that control how each
+ * instruction queue maps to zip cores.
+ */
+union zip_quex_map {
+ u64 u_reg64;
+ struct zip_quex_map_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_2_63 : 62;
+ u64 zce : 2;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 zce : 2;
+ u64 reserved_2_63 : 62;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_MAP(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x1400ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_MAP: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_sbuf_addr - Represents the registers that set the buffer
+ * parameters for the instruction queues.
+ *
+ * When quiescent (i.e. outstanding doorbell count is 0), it is safe to rewrite
+ * this register to effectively reset the command buffer state machine.
+ * These registers must be programmed after SW programs the corresponding
+ * ZIP_QUE(0..7)_SBUF_CTL.
+ */
+union zip_quex_sbuf_addr {
+ u64 u_reg64;
+ struct zip_quex_sbuf_addr_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_49_63 : 15;
+ u64 ptr : 42;
+ u64 off : 7;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 off : 7;
+ u64 ptr : 42;
+ u64 reserved_49_63 : 15;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_SBUF_ADDR(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x1000ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_SBUF_ADDR: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_quex_sbuf_ctl - Represents the registers that set the buffer
+ * parameters for the instruction queues.
+ *
+ * When quiescent (i.e. outstanding doorbell count is 0), it is safe to rewrite
+ * this register to effectively reset the command buffer state machine.
+ * These registers must be programmed before SW programs the corresponding
+ * ZIP_QUE(0..7)_SBUF_ADDR.
+ */
+union zip_quex_sbuf_ctl {
+ u64 u_reg64;
+ struct zip_quex_sbuf_ctl_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_45_63 : 19;
+ u64 size : 13;
+ u64 inst_be : 1;
+ u64 reserved_24_30 : 7;
+ u64 stream_id : 8;
+ u64 reserved_12_15 : 4;
+ u64 aura : 12;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 aura : 12;
+ u64 reserved_12_15 : 4;
+ u64 stream_id : 8;
+ u64 reserved_24_30 : 7;
+ u64 inst_be : 1;
+ u64 size : 13;
+ u64 reserved_45_63 : 19;
+#endif
+ } s;
+};
+
+static inline u64 ZIP_QUEX_SBUF_CTL(u64 param1)
+{
+ if (((param1 <= 7)))
+ return 0x1200ull + (param1 & 7) * 0x8ull;
+ pr_err("ZIP_QUEX_SBUF_CTL: %llu\n", param1);
+ return 0;
+}
+
+/**
+ * union zip_que_ena - Represents queue enable register
+ *
+ * If a queue is disabled, ZIP_CTL stops fetching instructions from the queue.
+ */
+union zip_que_ena {
+ u64 u_reg64;
+ struct zip_que_ena_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_8_63 : 56;
+ u64 ena : 8;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 ena : 8;
+ u64 reserved_8_63 : 56;
+#endif
+ } s;
+};
+
+#define ZIP_QUE_ENA 0x0500ull
+
+/**
+ * union zip_que_pri - Represents the register that defines the priority
+ * between instruction queues.
+ */
+union zip_que_pri {
+ u64 u_reg64;
+ struct zip_que_pri_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_8_63 : 56;
+ u64 pri : 8;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 pri : 8;
+ u64 reserved_8_63 : 56;
+#endif
+ } s;
+};
+
+#define ZIP_QUE_PRI 0x0508ull
+
+/**
+ * union zip_throttle - Represents the register that controls the maximum
+ * number of in-flight X2I data fetch transactions.
+ *
+ * Writing 0 to this register causes the ZIP module to temporarily suspend NCB
+ * accesses; it is not recommended for normal operation, but may be useful for
+ * diagnostics.
+ */
+union zip_throttle {
+ u64 u_reg64;
+ struct zip_throttle_s {
+#if defined(__BIG_ENDIAN_BITFIELD)
+ u64 reserved_6_63 : 58;
+ u64 ld_infl : 6;
+#elif defined(__LITTLE_ENDIAN_BITFIELD)
+ u64 ld_infl : 6;
+ u64 reserved_6_63 : 58;
+#endif
+ } s;
+};
+
+#define ZIP_THROTTLE 0x0010ull
+
+#endif /* _CSRS_ZIP__ */
diff --git a/drivers/crypto/ccp/Makefile b/drivers/crypto/ccp/Makefile
index 346ceb8..60919a3 100644
--- a/drivers/crypto/ccp/Makefile
+++ b/drivers/crypto/ccp/Makefile
@@ -12,4 +12,6 @@
ccp-crypto-aes.o \
ccp-crypto-aes-cmac.o \
ccp-crypto-aes-xts.o \
+ ccp-crypto-aes-galois.o \
+ ccp-crypto-des3.o \
ccp-crypto-sha.o
diff --git a/drivers/crypto/ccp/ccp-crypto-aes-galois.c b/drivers/crypto/ccp/ccp-crypto-aes-galois.c
new file mode 100644
index 0000000..38ee6f3
--- /dev/null
+++ b/drivers/crypto/ccp/ccp-crypto-aes-galois.c
@@ -0,0 +1,252 @@
+/*
+ * AMD Cryptographic Coprocessor (CCP) AES GCM crypto API support
+ *
+ * Copyright (C) 2016 Advanced Micro Devices, Inc.
+ *
+ * Author: Gary R Hook <gary.hook@amd.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/module.h>
+#include <linux/sched.h>
+#include <linux/delay.h>
+#include <linux/scatterlist.h>
+#include <linux/crypto.h>
+#include <crypto/internal/aead.h>
+#include <crypto/algapi.h>
+#include <crypto/aes.h>
+#include <crypto/ctr.h>
+#include <crypto/scatterwalk.h>
+#include <linux/delay.h>
+
+#include "ccp-crypto.h"
+
+#define AES_GCM_IVSIZE 12
+
+static int ccp_aes_gcm_complete(struct crypto_async_request *async_req, int ret)
+{
+ return ret;
+}
+
+static int ccp_aes_gcm_setkey(struct crypto_aead *tfm, const u8 *key,
+ unsigned int key_len)
+{
+ struct ccp_ctx *ctx = crypto_aead_ctx(tfm);
+
+ switch (key_len) {
+ case AES_KEYSIZE_128:
+ ctx->u.aes.type = CCP_AES_TYPE_128;
+ break;
+ case AES_KEYSIZE_192:
+ ctx->u.aes.type = CCP_AES_TYPE_192;
+ break;
+ case AES_KEYSIZE_256:
+ ctx->u.aes.type = CCP_AES_TYPE_256;
+ break;
+ default:
+ crypto_aead_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN);
+ return -EINVAL;
+ }
+
+ ctx->u.aes.mode = CCP_AES_MODE_GCM;
+ ctx->u.aes.key_len = key_len;
+
+ memcpy(ctx->u.aes.key, key, key_len);
+ sg_init_one(&ctx->u.aes.key_sg, ctx->u.aes.key, key_len);
+
+ return 0;
+}
+
+static int ccp_aes_gcm_setauthsize(struct crypto_aead *tfm,
+ unsigned int authsize)
+{
+ return 0;
+}
+
+static int ccp_aes_gcm_crypt(struct aead_request *req, bool encrypt)
+{
+ struct crypto_aead *tfm = crypto_aead_reqtfm(req);
+ struct ccp_ctx *ctx = crypto_aead_ctx(tfm);
+ struct ccp_aes_req_ctx *rctx = aead_request_ctx(req);
+ struct scatterlist *iv_sg = NULL;
+ unsigned int iv_len = 0;
+ int i;
+ int ret = 0;
+
+ if (!ctx->u.aes.key_len)
+ return -EINVAL;
+
+ if (ctx->u.aes.mode != CCP_AES_MODE_GCM)
+ return -EINVAL;
+
+ if (!req->iv)
+ return -EINVAL;
+
+ /*
+ * 5 parts:
+ * plaintext/ciphertext input
+ * AAD
+ * key
+ * IV
+ * Destination+tag buffer
+ */
+
+ /* Prepare the IV: 12 bytes + an integer (counter) */
+ memcpy(rctx->iv, req->iv, AES_GCM_IVSIZE);
+ for (i = 0; i < 3; i++)
+ rctx->iv[i + AES_GCM_IVSIZE] = 0;
+ rctx->iv[AES_BLOCK_SIZE - 1] = 1;
+
+ /* Set up a scatterlist for the IV */
+ iv_sg = &rctx->iv_sg;
+ iv_len = AES_BLOCK_SIZE;
+ sg_init_one(iv_sg, rctx->iv, iv_len);
+
+ /* The AAD + plaintext are concatenated in the src buffer */
+ memset(&rctx->cmd, 0, sizeof(rctx->cmd));
+ INIT_LIST_HEAD(&rctx->cmd.entry);
+ rctx->cmd.engine = CCP_ENGINE_AES;
+ rctx->cmd.u.aes.type = ctx->u.aes.type;
+ rctx->cmd.u.aes.mode = ctx->u.aes.mode;
+ rctx->cmd.u.aes.action = encrypt;
+ rctx->cmd.u.aes.key = &ctx->u.aes.key_sg;
+ rctx->cmd.u.aes.key_len = ctx->u.aes.key_len;
+ rctx->cmd.u.aes.iv = iv_sg;
+ rctx->cmd.u.aes.iv_len = iv_len;
+ rctx->cmd.u.aes.src = req->src;
+ rctx->cmd.u.aes.src_len = req->cryptlen;
+ rctx->cmd.u.aes.aad_len = req->assoclen;
+
+ /* The cipher text + the tag are in the dst buffer */
+ rctx->cmd.u.aes.dst = req->dst;
+
+ ret = ccp_crypto_enqueue_request(&req->base, &rctx->cmd);
+
+ return ret;
+}
+
+static int ccp_aes_gcm_encrypt(struct aead_request *req)
+{
+ return ccp_aes_gcm_crypt(req, CCP_AES_ACTION_ENCRYPT);
+}
+
+static int ccp_aes_gcm_decrypt(struct aead_request *req)
+{
+ return ccp_aes_gcm_crypt(req, CCP_AES_ACTION_DECRYPT);
+}
+
+static int ccp_aes_gcm_cra_init(struct crypto_aead *tfm)
+{
+ struct ccp_ctx *ctx = crypto_aead_ctx(tfm);
+
+ ctx->complete = ccp_aes_gcm_complete;
+ ctx->u.aes.key_len = 0;
+
+ crypto_aead_set_reqsize(tfm, sizeof(struct ccp_aes_req_ctx));
+
+ return 0;
+}
+
+static void ccp_aes_gcm_cra_exit(struct crypto_tfm *tfm)
+{
+}
+
+static struct aead_alg ccp_aes_gcm_defaults = {
+ .setkey = ccp_aes_gcm_setkey,
+ .setauthsize = ccp_aes_gcm_setauthsize,
+ .encrypt = ccp_aes_gcm_encrypt,
+ .decrypt = ccp_aes_gcm_decrypt,
+ .init = ccp_aes_gcm_cra_init,
+ .ivsize = AES_GCM_IVSIZE,
+ .maxauthsize = AES_BLOCK_SIZE,
+ .base = {
+ .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER |
+ CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_KERN_DRIVER_ONLY |
+ CRYPTO_ALG_NEED_FALLBACK,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct ccp_ctx),
+ .cra_priority = CCP_CRA_PRIORITY,
+ .cra_type = &crypto_ablkcipher_type,
+ .cra_exit = ccp_aes_gcm_cra_exit,
+ .cra_module = THIS_MODULE,
+ },
+};
+
+struct ccp_aes_aead_def {
+ enum ccp_aes_mode mode;
+ unsigned int version;
+ const char *name;
+ const char *driver_name;
+ unsigned int blocksize;
+ unsigned int ivsize;
+ struct aead_alg *alg_defaults;
+};
+
+static struct ccp_aes_aead_def aes_aead_algs[] = {
+ {
+ .mode = CCP_AES_MODE_GHASH,
+ .version = CCP_VERSION(5, 0),
+ .name = "gcm(aes)",
+ .driver_name = "gcm-aes-ccp",
+ .blocksize = 1,
+ .ivsize = AES_BLOCK_SIZE,
+ .alg_defaults = &ccp_aes_gcm_defaults,
+ },
+};
+
+static int ccp_register_aes_aead(struct list_head *head,
+ const struct ccp_aes_aead_def *def)
+{
+ struct ccp_crypto_aead *ccp_aead;
+ struct aead_alg *alg;
+ int ret;
+
+ ccp_aead = kzalloc(sizeof(*ccp_aead), GFP_KERNEL);
+ if (!ccp_aead)
+ return -ENOMEM;
+
+ INIT_LIST_HEAD(&ccp_aead->entry);
+
+ ccp_aead->mode = def->mode;
+
+ /* Copy the defaults and override as necessary */
+ alg = &ccp_aead->alg;
+ *alg = *def->alg_defaults;
+ snprintf(alg->base.cra_name, CRYPTO_MAX_ALG_NAME, "%s", def->name);
+ snprintf(alg->base.cra_driver_name, CRYPTO_MAX_ALG_NAME, "%s",
+ def->driver_name);
+ alg->base.cra_blocksize = def->blocksize;
+ alg->base.cra_ablkcipher.ivsize = def->ivsize;
+
+ ret = crypto_register_aead(alg);
+ if (ret) {
+ pr_err("%s ablkcipher algorithm registration error (%d)\n",
+ alg->base.cra_name, ret);
+ kfree(ccp_aead);
+ return ret;
+ }
+
+ list_add(&ccp_aead->entry, head);
+
+ return 0;
+}
+
+int ccp_register_aes_aeads(struct list_head *head)
+{
+ int i, ret;
+ unsigned int ccpversion = ccp_version();
+
+ for (i = 0; i < ARRAY_SIZE(aes_aead_algs); i++) {
+ if (aes_aead_algs[i].version > ccpversion)
+ continue;
+ ret = ccp_register_aes_aead(head, &aes_aead_algs[i]);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
diff --git a/drivers/crypto/ccp/ccp-crypto-des3.c b/drivers/crypto/ccp/ccp-crypto-des3.c
new file mode 100644
index 0000000..5af7347
--- /dev/null
+++ b/drivers/crypto/ccp/ccp-crypto-des3.c
@@ -0,0 +1,254 @@
+/*
+ * AMD Cryptographic Coprocessor (CCP) DES3 crypto API support
+ *
+ * Copyright (C) 2016 Advanced Micro Devices, Inc.
+ *
+ * Author: Gary R Hook <ghook@amd.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/module.h>
+#include <linux/sched.h>
+#include <linux/delay.h>
+#include <linux/scatterlist.h>
+#include <linux/crypto.h>
+#include <crypto/algapi.h>
+#include <crypto/scatterwalk.h>
+#include <crypto/des.h>
+
+#include "ccp-crypto.h"
+
+static int ccp_des3_complete(struct crypto_async_request *async_req, int ret)
+{
+ struct ablkcipher_request *req = ablkcipher_request_cast(async_req);
+ struct ccp_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
+ struct ccp_des3_req_ctx *rctx = ablkcipher_request_ctx(req);
+
+ if (ret)
+ return ret;
+
+ if (ctx->u.des3.mode != CCP_DES3_MODE_ECB)
+ memcpy(req->info, rctx->iv, DES3_EDE_BLOCK_SIZE);
+
+ return 0;
+}
+
+static int ccp_des3_setkey(struct crypto_ablkcipher *tfm, const u8 *key,
+ unsigned int key_len)
+{
+ struct ccp_ctx *ctx = crypto_tfm_ctx(crypto_ablkcipher_tfm(tfm));
+ struct ccp_crypto_ablkcipher_alg *alg =
+ ccp_crypto_ablkcipher_alg(crypto_ablkcipher_tfm(tfm));
+ u32 *flags = &tfm->base.crt_flags;
+
+
+ /* From des_generic.c:
+ *
+ * RFC2451:
+ * If the first two or last two independent 64-bit keys are
+ * equal (k1 == k2 or k2 == k3), then the DES3 operation is simply the
+ * same as DES. Implementers MUST reject keys that exhibit this
+ * property.
+ */
+ const u32 *K = (const u32 *)key;
+
+ if (unlikely(!((K[0] ^ K[2]) | (K[1] ^ K[3])) ||
+ !((K[2] ^ K[4]) | (K[3] ^ K[5]))) &&
+ (*flags & CRYPTO_TFM_REQ_WEAK_KEY)) {
+ *flags |= CRYPTO_TFM_RES_WEAK_KEY;
+ return -EINVAL;
+ }
+
+ /* It's not clear that there is any support for a keysize of 112.
+ * If needed, the caller should make K1 == K3
+ */
+ ctx->u.des3.type = CCP_DES3_TYPE_168;
+ ctx->u.des3.mode = alg->mode;
+ ctx->u.des3.key_len = key_len;
+
+ memcpy(ctx->u.des3.key, key, key_len);
+ sg_init_one(&ctx->u.des3.key_sg, ctx->u.des3.key, key_len);
+
+ return 0;
+}
+
+static int ccp_des3_crypt(struct ablkcipher_request *req, bool encrypt)
+{
+ struct ccp_ctx *ctx = crypto_tfm_ctx(req->base.tfm);
+ struct ccp_des3_req_ctx *rctx = ablkcipher_request_ctx(req);
+ struct scatterlist *iv_sg = NULL;
+ unsigned int iv_len = 0;
+ int ret;
+
+ if (!ctx->u.des3.key_len)
+ return -EINVAL;
+
+ if (((ctx->u.des3.mode == CCP_DES3_MODE_ECB) ||
+ (ctx->u.des3.mode == CCP_DES3_MODE_CBC)) &&
+ (req->nbytes & (DES3_EDE_BLOCK_SIZE - 1)))
+ return -EINVAL;
+
+ if (ctx->u.des3.mode != CCP_DES3_MODE_ECB) {
+ if (!req->info)
+ return -EINVAL;
+
+ memcpy(rctx->iv, req->info, DES3_EDE_BLOCK_SIZE);
+ iv_sg = &rctx->iv_sg;
+ iv_len = DES3_EDE_BLOCK_SIZE;
+ sg_init_one(iv_sg, rctx->iv, iv_len);
+ }
+
+ memset(&rctx->cmd, 0, sizeof(rctx->cmd));
+ INIT_LIST_HEAD(&rctx->cmd.entry);
+ rctx->cmd.engine = CCP_ENGINE_DES3;
+ rctx->cmd.u.des3.type = ctx->u.des3.type;
+ rctx->cmd.u.des3.mode = ctx->u.des3.mode;
+ rctx->cmd.u.des3.action = (encrypt)
+ ? CCP_DES3_ACTION_ENCRYPT
+ : CCP_DES3_ACTION_DECRYPT;
+ rctx->cmd.u.des3.key = &ctx->u.des3.key_sg;
+ rctx->cmd.u.des3.key_len = ctx->u.des3.key_len;
+ rctx->cmd.u.des3.iv = iv_sg;
+ rctx->cmd.u.des3.iv_len = iv_len;
+ rctx->cmd.u.des3.src = req->src;
+ rctx->cmd.u.des3.src_len = req->nbytes;
+ rctx->cmd.u.des3.dst = req->dst;
+
+ ret = ccp_crypto_enqueue_request(&req->base, &rctx->cmd);
+
+ return ret;
+}
+
+static int ccp_des3_encrypt(struct ablkcipher_request *req)
+{
+ return ccp_des3_crypt(req, true);
+}
+
+static int ccp_des3_decrypt(struct ablkcipher_request *req)
+{
+ return ccp_des3_crypt(req, false);
+}
+
+static int ccp_des3_cra_init(struct crypto_tfm *tfm)
+{
+ struct ccp_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ ctx->complete = ccp_des3_complete;
+ ctx->u.des3.key_len = 0;
+
+ tfm->crt_ablkcipher.reqsize = sizeof(struct ccp_des3_req_ctx);
+
+ return 0;
+}
+
+static void ccp_des3_cra_exit(struct crypto_tfm *tfm)
+{
+}
+
+static struct crypto_alg ccp_des3_defaults = {
+ .cra_flags = CRYPTO_ALG_TYPE_ABLKCIPHER |
+ CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_KERN_DRIVER_ONLY |
+ CRYPTO_ALG_NEED_FALLBACK,
+ .cra_blocksize = DES3_EDE_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct ccp_ctx),
+ .cra_priority = CCP_CRA_PRIORITY,
+ .cra_type = &crypto_ablkcipher_type,
+ .cra_init = ccp_des3_cra_init,
+ .cra_exit = ccp_des3_cra_exit,
+ .cra_module = THIS_MODULE,
+ .cra_ablkcipher = {
+ .setkey = ccp_des3_setkey,
+ .encrypt = ccp_des3_encrypt,
+ .decrypt = ccp_des3_decrypt,
+ .min_keysize = DES3_EDE_KEY_SIZE,
+ .max_keysize = DES3_EDE_KEY_SIZE,
+ },
+};
+
+struct ccp_des3_def {
+ enum ccp_des3_mode mode;
+ unsigned int version;
+ const char *name;
+ const char *driver_name;
+ unsigned int blocksize;
+ unsigned int ivsize;
+ struct crypto_alg *alg_defaults;
+};
+
+static struct ccp_des3_def des3_algs[] = {
+ {
+ .mode = CCP_DES3_MODE_ECB,
+ .version = CCP_VERSION(5, 0),
+ .name = "ecb(des3_ede)",
+ .driver_name = "ecb-des3-ccp",
+ .blocksize = DES3_EDE_BLOCK_SIZE,
+ .ivsize = 0,
+ .alg_defaults = &ccp_des3_defaults,
+ },
+ {
+ .mode = CCP_DES3_MODE_CBC,
+ .version = CCP_VERSION(5, 0),
+ .name = "cbc(des3_ede)",
+ .driver_name = "cbc-des3-ccp",
+ .blocksize = DES3_EDE_BLOCK_SIZE,
+ .ivsize = DES3_EDE_BLOCK_SIZE,
+ .alg_defaults = &ccp_des3_defaults,
+ },
+};
+
+static int ccp_register_des3_alg(struct list_head *head,
+ const struct ccp_des3_def *def)
+{
+ struct ccp_crypto_ablkcipher_alg *ccp_alg;
+ struct crypto_alg *alg;
+ int ret;
+
+ ccp_alg = kzalloc(sizeof(*ccp_alg), GFP_KERNEL);
+ if (!ccp_alg)
+ return -ENOMEM;
+
+ INIT_LIST_HEAD(&ccp_alg->entry);
+
+ ccp_alg->mode = def->mode;
+
+ /* Copy the defaults and override as necessary */
+ alg = &ccp_alg->alg;
+ *alg = *def->alg_defaults;
+ snprintf(alg->cra_name, CRYPTO_MAX_ALG_NAME, "%s", def->name);
+ snprintf(alg->cra_driver_name, CRYPTO_MAX_ALG_NAME, "%s",
+ def->driver_name);
+ alg->cra_blocksize = def->blocksize;
+ alg->cra_ablkcipher.ivsize = def->ivsize;
+
+ ret = crypto_register_alg(alg);
+ if (ret) {
+ pr_err("%s ablkcipher algorithm registration error (%d)\n",
+ alg->cra_name, ret);
+ kfree(ccp_alg);
+ return ret;
+ }
+
+ list_add(&ccp_alg->entry, head);
+
+ return 0;
+}
+
+int ccp_register_des3_algs(struct list_head *head)
+{
+ int i, ret;
+ unsigned int ccpversion = ccp_version();
+
+ for (i = 0; i < ARRAY_SIZE(des3_algs); i++) {
+ if (des3_algs[i].version > ccpversion)
+ continue;
+ ret = ccp_register_des3_alg(head, &des3_algs[i]);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
diff --git a/drivers/crypto/ccp/ccp-crypto-main.c b/drivers/crypto/ccp/ccp-crypto-main.c
index e0380e5..8dccbdd 100644
--- a/drivers/crypto/ccp/ccp-crypto-main.c
+++ b/drivers/crypto/ccp/ccp-crypto-main.c
@@ -33,9 +33,14 @@
module_param(sha_disable, uint, 0444);
MODULE_PARM_DESC(sha_disable, "Disable use of SHA - any non-zero value");
+static unsigned int des3_disable;
+module_param(des3_disable, uint, 0444);
+MODULE_PARM_DESC(des3_disable, "Disable use of 3DES - any non-zero value");
+
/* List heads for the supported algorithms */
static LIST_HEAD(hash_algs);
static LIST_HEAD(cipher_algs);
+static LIST_HEAD(aead_algs);
/* For any tfm, requests for that tfm must be returned on the order
* received. With multiple queues available, the CCP can process more
@@ -335,6 +340,16 @@
ret = ccp_register_aes_xts_algs(&cipher_algs);
if (ret)
return ret;
+
+ ret = ccp_register_aes_aeads(&aead_algs);
+ if (ret)
+ return ret;
+ }
+
+ if (!des3_disable) {
+ ret = ccp_register_des3_algs(&cipher_algs);
+ if (ret)
+ return ret;
}
if (!sha_disable) {
@@ -350,6 +365,7 @@
{
struct ccp_crypto_ahash_alg *ahash_alg, *ahash_tmp;
struct ccp_crypto_ablkcipher_alg *ablk_alg, *ablk_tmp;
+ struct ccp_crypto_aead *aead_alg, *aead_tmp;
list_for_each_entry_safe(ahash_alg, ahash_tmp, &hash_algs, entry) {
crypto_unregister_ahash(&ahash_alg->alg);
@@ -362,6 +378,12 @@
list_del(&ablk_alg->entry);
kfree(ablk_alg);
}
+
+ list_for_each_entry_safe(aead_alg, aead_tmp, &aead_algs, entry) {
+ crypto_unregister_aead(&aead_alg->alg);
+ list_del(&aead_alg->entry);
+ kfree(aead_alg);
+ }
}
static int ccp_crypto_init(void)
diff --git a/drivers/crypto/ccp/ccp-crypto-sha.c b/drivers/crypto/ccp/ccp-crypto-sha.c
index 84a652b..6b46eea 100644
--- a/drivers/crypto/ccp/ccp-crypto-sha.c
+++ b/drivers/crypto/ccp/ccp-crypto-sha.c
@@ -146,6 +146,12 @@
case CCP_SHA_TYPE_256:
rctx->cmd.u.sha.ctx_len = SHA256_DIGEST_SIZE;
break;
+ case CCP_SHA_TYPE_384:
+ rctx->cmd.u.sha.ctx_len = SHA384_DIGEST_SIZE;
+ break;
+ case CCP_SHA_TYPE_512:
+ rctx->cmd.u.sha.ctx_len = SHA512_DIGEST_SIZE;
+ break;
default:
/* Should never get here */
break;
@@ -393,6 +399,22 @@
.digest_size = SHA256_DIGEST_SIZE,
.block_size = SHA256_BLOCK_SIZE,
},
+ {
+ .version = CCP_VERSION(5, 0),
+ .name = "sha384",
+ .drv_name = "sha384-ccp",
+ .type = CCP_SHA_TYPE_384,
+ .digest_size = SHA384_DIGEST_SIZE,
+ .block_size = SHA384_BLOCK_SIZE,
+ },
+ {
+ .version = CCP_VERSION(5, 0),
+ .name = "sha512",
+ .drv_name = "sha512-ccp",
+ .type = CCP_SHA_TYPE_512,
+ .digest_size = SHA512_DIGEST_SIZE,
+ .block_size = SHA512_BLOCK_SIZE,
+ },
};
static int ccp_register_hmac_alg(struct list_head *head,
diff --git a/drivers/crypto/ccp/ccp-crypto.h b/drivers/crypto/ccp/ccp-crypto.h
index 8335b32..dd5bf15 100644
--- a/drivers/crypto/ccp/ccp-crypto.h
+++ b/drivers/crypto/ccp/ccp-crypto.h
@@ -19,10 +19,14 @@
#include <linux/ccp.h>
#include <crypto/algapi.h>
#include <crypto/aes.h>
+#include <crypto/internal/aead.h>
+#include <crypto/aead.h>
#include <crypto/ctr.h>
#include <crypto/hash.h>
#include <crypto/sha.h>
+#define CCP_LOG_LEVEL KERN_INFO
+
#define CCP_CRA_PRIORITY 300
struct ccp_crypto_ablkcipher_alg {
@@ -33,6 +37,14 @@
struct crypto_alg alg;
};
+struct ccp_crypto_aead {
+ struct list_head entry;
+
+ u32 mode;
+
+ struct aead_alg alg;
+};
+
struct ccp_crypto_ahash_alg {
struct list_head entry;
@@ -95,6 +107,9 @@
struct scatterlist iv_sg;
u8 iv[AES_BLOCK_SIZE];
+ struct scatterlist tag_sg;
+ u8 tag[AES_BLOCK_SIZE];
+
/* Fields used for RFC3686 requests */
u8 *rfc3686_info;
u8 rfc3686_iv[AES_BLOCK_SIZE];
@@ -137,9 +152,29 @@
u8 buf[AES_BLOCK_SIZE];
};
-/***** SHA related defines *****/
-#define MAX_SHA_CONTEXT_SIZE SHA256_DIGEST_SIZE
-#define MAX_SHA_BLOCK_SIZE SHA256_BLOCK_SIZE
+/***** 3DES related defines *****/
+struct ccp_des3_ctx {
+ enum ccp_engine engine;
+ enum ccp_des3_type type;
+ enum ccp_des3_mode mode;
+
+ struct scatterlist key_sg;
+ unsigned int key_len;
+ u8 key[AES_MAX_KEY_SIZE];
+};
+
+struct ccp_des3_req_ctx {
+ struct scatterlist iv_sg;
+ u8 iv[AES_BLOCK_SIZE];
+
+ struct ccp_cmd cmd;
+};
+
+/* SHA-related defines
+ * These values must be large enough to accommodate any variant
+ */
+#define MAX_SHA_CONTEXT_SIZE SHA512_DIGEST_SIZE
+#define MAX_SHA_BLOCK_SIZE SHA512_BLOCK_SIZE
struct ccp_sha_ctx {
struct scatterlist opad_sg;
@@ -199,6 +234,7 @@
union {
struct ccp_aes_ctx aes;
struct ccp_sha_ctx sha;
+ struct ccp_des3_ctx des3;
} u;
};
@@ -210,6 +246,8 @@
int ccp_register_aes_algs(struct list_head *head);
int ccp_register_aes_cmac_algs(struct list_head *head);
int ccp_register_aes_xts_algs(struct list_head *head);
+int ccp_register_aes_aeads(struct list_head *head);
int ccp_register_sha_algs(struct list_head *head);
+int ccp_register_des3_algs(struct list_head *head);
#endif
diff --git a/drivers/crypto/ccp/ccp-dev-v3.c b/drivers/crypto/ccp/ccp-dev-v3.c
index 7bc0998..367c2e3 100644
--- a/drivers/crypto/ccp/ccp-dev-v3.c
+++ b/drivers/crypto/ccp/ccp-dev-v3.c
@@ -315,17 +315,73 @@
return ccp_do_cmd(op, cr, ARRAY_SIZE(cr));
}
+static void ccp_disable_queue_interrupts(struct ccp_device *ccp)
+{
+ iowrite32(0x00, ccp->io_regs + IRQ_MASK_REG);
+}
+
+static void ccp_enable_queue_interrupts(struct ccp_device *ccp)
+{
+ iowrite32(ccp->qim, ccp->io_regs + IRQ_MASK_REG);
+}
+
+static void ccp_irq_bh(unsigned long data)
+{
+ struct ccp_device *ccp = (struct ccp_device *)data;
+ struct ccp_cmd_queue *cmd_q;
+ u32 q_int, status;
+ unsigned int i;
+
+ status = ioread32(ccp->io_regs + IRQ_STATUS_REG);
+
+ for (i = 0; i < ccp->cmd_q_count; i++) {
+ cmd_q = &ccp->cmd_q[i];
+
+ q_int = status & (cmd_q->int_ok | cmd_q->int_err);
+ if (q_int) {
+ cmd_q->int_status = status;
+ cmd_q->q_status = ioread32(cmd_q->reg_status);
+ cmd_q->q_int_status = ioread32(cmd_q->reg_int_status);
+
+ /* On error, only save the first error value */
+ if ((q_int & cmd_q->int_err) && !cmd_q->cmd_error)
+ cmd_q->cmd_error = CMD_Q_ERROR(cmd_q->q_status);
+
+ cmd_q->int_rcvd = 1;
+
+ /* Acknowledge the interrupt and wake the kthread */
+ iowrite32(q_int, ccp->io_regs + IRQ_STATUS_REG);
+ wake_up_interruptible(&cmd_q->int_queue);
+ }
+ }
+ ccp_enable_queue_interrupts(ccp);
+}
+
+static irqreturn_t ccp_irq_handler(int irq, void *data)
+{
+ struct device *dev = data;
+ struct ccp_device *ccp = dev_get_drvdata(dev);
+
+ ccp_disable_queue_interrupts(ccp);
+ if (ccp->use_tasklet)
+ tasklet_schedule(&ccp->irq_tasklet);
+ else
+ ccp_irq_bh((unsigned long)ccp);
+
+ return IRQ_HANDLED;
+}
+
static int ccp_init(struct ccp_device *ccp)
{
struct device *dev = ccp->dev;
struct ccp_cmd_queue *cmd_q;
struct dma_pool *dma_pool;
char dma_pool_name[MAX_DMAPOOL_NAME_LEN];
- unsigned int qmr, qim, i;
+ unsigned int qmr, i;
int ret;
/* Find available queues */
- qim = 0;
+ ccp->qim = 0;
qmr = ioread32(ccp->io_regs + Q_MASK_REG);
for (i = 0; i < MAX_HW_QUEUES; i++) {
if (!(qmr & (1 << i)))
@@ -370,7 +426,7 @@
init_waitqueue_head(&cmd_q->int_queue);
/* Build queue interrupt mask (two interrupts per queue) */
- qim |= cmd_q->int_ok | cmd_q->int_err;
+ ccp->qim |= cmd_q->int_ok | cmd_q->int_err;
#ifdef CONFIG_ARM64
/* For arm64 set the recommended queue cache settings */
@@ -388,14 +444,14 @@
dev_notice(dev, "%u command queues available\n", ccp->cmd_q_count);
/* Disable and clear interrupts until ready */
- iowrite32(0x00, ccp->io_regs + IRQ_MASK_REG);
+ ccp_disable_queue_interrupts(ccp);
for (i = 0; i < ccp->cmd_q_count; i++) {
cmd_q = &ccp->cmd_q[i];
ioread32(cmd_q->reg_int_status);
ioread32(cmd_q->reg_status);
}
- iowrite32(qim, ccp->io_regs + IRQ_STATUS_REG);
+ iowrite32(ccp->qim, ccp->io_regs + IRQ_STATUS_REG);
/* Request an irq */
ret = ccp->get_irq(ccp);
@@ -404,6 +460,11 @@
goto e_pool;
}
+ /* Initialize the ISR tasklet? */
+ if (ccp->use_tasklet)
+ tasklet_init(&ccp->irq_tasklet, ccp_irq_bh,
+ (unsigned long)ccp);
+
dev_dbg(dev, "Starting threads...\n");
/* Create a kthread for each queue */
for (i = 0; i < ccp->cmd_q_count; i++) {
@@ -426,7 +487,7 @@
dev_dbg(dev, "Enabling interrupts...\n");
/* Enable interrupts */
- iowrite32(qim, ccp->io_regs + IRQ_MASK_REG);
+ ccp_enable_queue_interrupts(ccp);
dev_dbg(dev, "Registering device...\n");
ccp_add_device(ccp);
@@ -463,7 +524,7 @@
{
struct ccp_cmd_queue *cmd_q;
struct ccp_cmd *cmd;
- unsigned int qim, i;
+ unsigned int i;
/* Unregister the DMA engine */
ccp_dmaengine_unregister(ccp);
@@ -474,22 +535,15 @@
/* Remove this device from the list of available units */
ccp_del_device(ccp);
- /* Build queue interrupt mask (two interrupt masks per queue) */
- qim = 0;
- for (i = 0; i < ccp->cmd_q_count; i++) {
- cmd_q = &ccp->cmd_q[i];
- qim |= cmd_q->int_ok | cmd_q->int_err;
- }
-
/* Disable and clear interrupts */
- iowrite32(0x00, ccp->io_regs + IRQ_MASK_REG);
+ ccp_disable_queue_interrupts(ccp);
for (i = 0; i < ccp->cmd_q_count; i++) {
cmd_q = &ccp->cmd_q[i];
ioread32(cmd_q->reg_int_status);
ioread32(cmd_q->reg_status);
}
- iowrite32(qim, ccp->io_regs + IRQ_STATUS_REG);
+ iowrite32(ccp->qim, ccp->io_regs + IRQ_STATUS_REG);
/* Stop the queue kthreads */
for (i = 0; i < ccp->cmd_q_count; i++)
@@ -516,43 +570,10 @@
}
}
-static irqreturn_t ccp_irq_handler(int irq, void *data)
-{
- struct device *dev = data;
- struct ccp_device *ccp = dev_get_drvdata(dev);
- struct ccp_cmd_queue *cmd_q;
- u32 q_int, status;
- unsigned int i;
-
- status = ioread32(ccp->io_regs + IRQ_STATUS_REG);
-
- for (i = 0; i < ccp->cmd_q_count; i++) {
- cmd_q = &ccp->cmd_q[i];
-
- q_int = status & (cmd_q->int_ok | cmd_q->int_err);
- if (q_int) {
- cmd_q->int_status = status;
- cmd_q->q_status = ioread32(cmd_q->reg_status);
- cmd_q->q_int_status = ioread32(cmd_q->reg_int_status);
-
- /* On error, only save the first error value */
- if ((q_int & cmd_q->int_err) && !cmd_q->cmd_error)
- cmd_q->cmd_error = CMD_Q_ERROR(cmd_q->q_status);
-
- cmd_q->int_rcvd = 1;
-
- /* Acknowledge the interrupt and wake the kthread */
- iowrite32(q_int, ccp->io_regs + IRQ_STATUS_REG);
- wake_up_interruptible(&cmd_q->int_queue);
- }
- }
-
- return IRQ_HANDLED;
-}
-
static const struct ccp_actions ccp3_actions = {
.aes = ccp_perform_aes,
.xts_aes = ccp_perform_xts_aes,
+ .des3 = NULL,
.sha = ccp_perform_sha,
.rsa = ccp_perform_rsa,
.passthru = ccp_perform_passthru,
diff --git a/drivers/crypto/ccp/ccp-dev-v5.c b/drivers/crypto/ccp/ccp-dev-v5.c
index fc08b4e..ccbe32d 100644
--- a/drivers/crypto/ccp/ccp-dev-v5.c
+++ b/drivers/crypto/ccp/ccp-dev-v5.c
@@ -108,6 +108,12 @@
u16 type:2;
} aes_xts;
struct {
+ u16 size:7;
+ u16 encrypt:1;
+ u16 mode:5;
+ u16 type:2;
+ } des3;
+ struct {
u16 rsvd1:10;
u16 type:4;
u16 rsvd2:1;
@@ -139,6 +145,10 @@
#define CCP_AES_TYPE(p) ((p)->aes.type)
#define CCP_XTS_SIZE(p) ((p)->aes_xts.size)
#define CCP_XTS_ENCRYPT(p) ((p)->aes_xts.encrypt)
+#define CCP_DES3_SIZE(p) ((p)->des3.size)
+#define CCP_DES3_ENCRYPT(p) ((p)->des3.encrypt)
+#define CCP_DES3_MODE(p) ((p)->des3.mode)
+#define CCP_DES3_TYPE(p) ((p)->des3.type)
#define CCP_SHA_TYPE(p) ((p)->sha.type)
#define CCP_RSA_SIZE(p) ((p)->rsa.size)
#define CCP_PT_BYTESWAP(p) ((p)->pt.byteswap)
@@ -388,6 +398,47 @@
return ccp5_do_cmd(&desc, op->cmd_q);
}
+static int ccp5_perform_des3(struct ccp_op *op)
+{
+ struct ccp5_desc desc;
+ union ccp_function function;
+ u32 key_addr = op->sb_key * LSB_ITEM_SIZE;
+
+ /* Zero out all the fields of the command desc */
+ memset(&desc, 0, sizeof(struct ccp5_desc));
+
+ CCP5_CMD_ENGINE(&desc) = CCP_ENGINE_DES3;
+
+ CCP5_CMD_SOC(&desc) = op->soc;
+ CCP5_CMD_IOC(&desc) = 1;
+ CCP5_CMD_INIT(&desc) = op->init;
+ CCP5_CMD_EOM(&desc) = op->eom;
+ CCP5_CMD_PROT(&desc) = 0;
+
+ function.raw = 0;
+ CCP_DES3_ENCRYPT(&function) = op->u.des3.action;
+ CCP_DES3_MODE(&function) = op->u.des3.mode;
+ CCP_DES3_TYPE(&function) = op->u.des3.type;
+ CCP5_CMD_FUNCTION(&desc) = function.raw;
+
+ CCP5_CMD_LEN(&desc) = op->src.u.dma.length;
+
+ CCP5_CMD_SRC_LO(&desc) = ccp_addr_lo(&op->src.u.dma);
+ CCP5_CMD_SRC_HI(&desc) = ccp_addr_hi(&op->src.u.dma);
+ CCP5_CMD_SRC_MEM(&desc) = CCP_MEMTYPE_SYSTEM;
+
+ CCP5_CMD_DST_LO(&desc) = ccp_addr_lo(&op->dst.u.dma);
+ CCP5_CMD_DST_HI(&desc) = ccp_addr_hi(&op->dst.u.dma);
+ CCP5_CMD_DST_MEM(&desc) = CCP_MEMTYPE_SYSTEM;
+
+ CCP5_CMD_KEY_LO(&desc) = lower_32_bits(key_addr);
+ CCP5_CMD_KEY_HI(&desc) = 0;
+ CCP5_CMD_KEY_MEM(&desc) = CCP_MEMTYPE_SB;
+ CCP5_CMD_LSB_ID(&desc) = op->sb_ctx;
+
+ return ccp5_do_cmd(&desc, op->cmd_q);
+}
+
static int ccp5_perform_rsa(struct ccp_op *op)
{
struct ccp5_desc desc;
@@ -435,6 +486,7 @@
struct ccp_dma_info *saddr = &op->src.u.dma;
struct ccp_dma_info *daddr = &op->dst.u.dma;
+
memset(&desc, 0, Q_DESC_SIZE);
CCP5_CMD_ENGINE(&desc) = CCP_ENGINE_PASSTHRU;
@@ -653,6 +705,65 @@
return rc;
}
+static void ccp5_disable_queue_interrupts(struct ccp_device *ccp)
+{
+ unsigned int i;
+
+ for (i = 0; i < ccp->cmd_q_count; i++)
+ iowrite32(0x0, ccp->cmd_q[i].reg_int_enable);
+}
+
+static void ccp5_enable_queue_interrupts(struct ccp_device *ccp)
+{
+ unsigned int i;
+
+ for (i = 0; i < ccp->cmd_q_count; i++)
+ iowrite32(SUPPORTED_INTERRUPTS, ccp->cmd_q[i].reg_int_enable);
+}
+
+static void ccp5_irq_bh(unsigned long data)
+{
+ struct ccp_device *ccp = (struct ccp_device *)data;
+ u32 status;
+ unsigned int i;
+
+ for (i = 0; i < ccp->cmd_q_count; i++) {
+ struct ccp_cmd_queue *cmd_q = &ccp->cmd_q[i];
+
+ status = ioread32(cmd_q->reg_interrupt_status);
+
+ if (status) {
+ cmd_q->int_status = status;
+ cmd_q->q_status = ioread32(cmd_q->reg_status);
+ cmd_q->q_int_status = ioread32(cmd_q->reg_int_status);
+
+ /* On error, only save the first error value */
+ if ((status & INT_ERROR) && !cmd_q->cmd_error)
+ cmd_q->cmd_error = CMD_Q_ERROR(cmd_q->q_status);
+
+ cmd_q->int_rcvd = 1;
+
+ /* Acknowledge the interrupt and wake the kthread */
+ iowrite32(status, cmd_q->reg_interrupt_status);
+ wake_up_interruptible(&cmd_q->int_queue);
+ }
+ }
+ ccp5_enable_queue_interrupts(ccp);
+}
+
+static irqreturn_t ccp5_irq_handler(int irq, void *data)
+{
+ struct device *dev = data;
+ struct ccp_device *ccp = dev_get_drvdata(dev);
+
+ ccp5_disable_queue_interrupts(ccp);
+ if (ccp->use_tasklet)
+ tasklet_schedule(&ccp->irq_tasklet);
+ else
+ ccp5_irq_bh((unsigned long)ccp);
+ return IRQ_HANDLED;
+}
+
static int ccp5_init(struct ccp_device *ccp)
{
struct device *dev = ccp->dev;
@@ -729,6 +840,7 @@
dev_dbg(dev, "queue #%u available\n", i);
}
+
if (ccp->cmd_q_count == 0) {
dev_notice(dev, "no command queues available\n");
ret = -EIO;
@@ -736,19 +848,18 @@
}
/* Turn off the queues and disable interrupts until ready */
+ ccp5_disable_queue_interrupts(ccp);
for (i = 0; i < ccp->cmd_q_count; i++) {
cmd_q = &ccp->cmd_q[i];
cmd_q->qcontrol = 0; /* Start with nothing */
iowrite32(cmd_q->qcontrol, cmd_q->reg_control);
- /* Disable the interrupts */
- iowrite32(0x00, cmd_q->reg_int_enable);
ioread32(cmd_q->reg_int_status);
ioread32(cmd_q->reg_status);
- /* Clear the interrupts */
- iowrite32(ALL_INTERRUPTS, cmd_q->reg_interrupt_status);
+ /* Clear the interrupt status */
+ iowrite32(SUPPORTED_INTERRUPTS, cmd_q->reg_interrupt_status);
}
dev_dbg(dev, "Requesting an IRQ...\n");
@@ -758,6 +869,10 @@
dev_err(dev, "unable to allocate an IRQ\n");
goto e_pool;
}
+ /* Initialize the ISR tasklet */
+ if (ccp->use_tasklet)
+ tasklet_init(&ccp->irq_tasklet, ccp5_irq_bh,
+ (unsigned long)ccp);
dev_dbg(dev, "Loading LSB map...\n");
/* Copy the private LSB mask to the public registers */
@@ -826,11 +941,7 @@
}
dev_dbg(dev, "Enabling interrupts...\n");
- /* Enable interrupts */
- for (i = 0; i < ccp->cmd_q_count; i++) {
- cmd_q = &ccp->cmd_q[i];
- iowrite32(ALL_INTERRUPTS, cmd_q->reg_int_enable);
- }
+ ccp5_enable_queue_interrupts(ccp);
dev_dbg(dev, "Registering device...\n");
/* Put this on the unit list to make it available */
@@ -882,17 +993,15 @@
ccp_del_device(ccp);
/* Disable and clear interrupts */
+ ccp5_disable_queue_interrupts(ccp);
for (i = 0; i < ccp->cmd_q_count; i++) {
cmd_q = &ccp->cmd_q[i];
/* Turn off the run bit */
iowrite32(cmd_q->qcontrol & ~CMD5_Q_RUN, cmd_q->reg_control);
- /* Disable the interrupts */
- iowrite32(ALL_INTERRUPTS, cmd_q->reg_interrupt_status);
-
/* Clear the interrupt status */
- iowrite32(0x00, cmd_q->reg_int_enable);
+ iowrite32(SUPPORTED_INTERRUPTS, cmd_q->reg_interrupt_status);
ioread32(cmd_q->reg_int_status);
ioread32(cmd_q->reg_status);
}
@@ -925,38 +1034,6 @@
}
}
-static irqreturn_t ccp5_irq_handler(int irq, void *data)
-{
- struct device *dev = data;
- struct ccp_device *ccp = dev_get_drvdata(dev);
- u32 status;
- unsigned int i;
-
- for (i = 0; i < ccp->cmd_q_count; i++) {
- struct ccp_cmd_queue *cmd_q = &ccp->cmd_q[i];
-
- status = ioread32(cmd_q->reg_interrupt_status);
-
- if (status) {
- cmd_q->int_status = status;
- cmd_q->q_status = ioread32(cmd_q->reg_status);
- cmd_q->q_int_status = ioread32(cmd_q->reg_int_status);
-
- /* On error, only save the first error value */
- if ((status & INT_ERROR) && !cmd_q->cmd_error)
- cmd_q->cmd_error = CMD_Q_ERROR(cmd_q->q_status);
-
- cmd_q->int_rcvd = 1;
-
- /* Acknowledge the interrupt and wake the kthread */
- iowrite32(ALL_INTERRUPTS, cmd_q->reg_interrupt_status);
- wake_up_interruptible(&cmd_q->int_queue);
- }
- }
-
- return IRQ_HANDLED;
-}
-
static void ccp5_config(struct ccp_device *ccp)
{
/* Public side */
@@ -994,6 +1071,7 @@
.aes = ccp5_perform_aes,
.xts_aes = ccp5_perform_xts_aes,
.sha = ccp5_perform_sha,
+ .des3 = ccp5_perform_des3,
.rsa = ccp5_perform_rsa,
.passthru = ccp5_perform_passthru,
.ecc = ccp5_perform_ecc,
diff --git a/drivers/crypto/ccp/ccp-dev.h b/drivers/crypto/ccp/ccp-dev.h
index aa36f3f..0cb09d0 100644
--- a/drivers/crypto/ccp/ccp-dev.h
+++ b/drivers/crypto/ccp/ccp-dev.h
@@ -109,9 +109,8 @@
#define INT_COMPLETION 0x1
#define INT_ERROR 0x2
#define INT_QUEUE_STOPPED 0x4
-#define ALL_INTERRUPTS (INT_COMPLETION| \
- INT_ERROR| \
- INT_QUEUE_STOPPED)
+#define INT_EMPTY_QUEUE 0x8
+#define SUPPORTED_INTERRUPTS (INT_COMPLETION | INT_ERROR)
#define LSB_REGION_WIDTH 5
#define MAX_LSB_CNT 8
@@ -194,6 +193,9 @@
#define CCP_XTS_AES_KEY_SB_COUNT 1
#define CCP_XTS_AES_CTX_SB_COUNT 1
+#define CCP_DES3_KEY_SB_COUNT 1
+#define CCP_DES3_CTX_SB_COUNT 1
+
#define CCP_SHA_SB_COUNT 1
#define CCP_RSA_MAX_WIDTH 4096
@@ -337,7 +339,10 @@
void *dev_specific;
int (*get_irq)(struct ccp_device *ccp);
void (*free_irq)(struct ccp_device *ccp);
+ unsigned int qim;
unsigned int irq;
+ bool use_tasklet;
+ struct tasklet_struct irq_tasklet;
/* I/O area used for device communication. The register mapping
* starts at an offset into the mapped bar.
@@ -424,33 +429,33 @@
};
#define CCP_MEMTYPE_LSB CCP_MEMTYPE_KSB
+
struct ccp_dma_info {
dma_addr_t address;
unsigned int offset;
unsigned int length;
enum dma_data_direction dir;
-};
+} __packed __aligned(4);
struct ccp_dm_workarea {
struct device *dev;
struct dma_pool *dma_pool;
- unsigned int length;
u8 *address;
struct ccp_dma_info dma;
+ unsigned int length;
};
struct ccp_sg_workarea {
struct scatterlist *sg;
int nents;
+ unsigned int sg_used;
struct scatterlist *dma_sg;
struct device *dma_dev;
unsigned int dma_count;
enum dma_data_direction dma_dir;
- unsigned int sg_used;
-
u64 bytes_left;
};
@@ -479,6 +484,12 @@
enum ccp_xts_aes_unit_size unit_size;
};
+struct ccp_des3_op {
+ enum ccp_des3_type type;
+ enum ccp_des3_mode mode;
+ enum ccp_des3_action action;
+};
+
struct ccp_sha_op {
enum ccp_sha_type type;
u64 msg_bits;
@@ -516,6 +527,7 @@
union {
struct ccp_aes_op aes;
struct ccp_xts_aes_op xts;
+ struct ccp_des3_op des3;
struct ccp_sha_op sha;
struct ccp_rsa_op rsa;
struct ccp_passthru_op passthru;
@@ -624,13 +636,13 @@
struct ccp_actions {
int (*aes)(struct ccp_op *);
int (*xts_aes)(struct ccp_op *);
+ int (*des3)(struct ccp_op *);
int (*sha)(struct ccp_op *);
int (*rsa)(struct ccp_op *);
int (*passthru)(struct ccp_op *);
int (*ecc)(struct ccp_op *);
u32 (*sballoc)(struct ccp_cmd_queue *, unsigned int);
- void (*sbfree)(struct ccp_cmd_queue *, unsigned int,
- unsigned int);
+ void (*sbfree)(struct ccp_cmd_queue *, unsigned int, unsigned int);
unsigned int (*get_free_slots)(struct ccp_cmd_queue *);
int (*init)(struct ccp_device *);
void (*destroy)(struct ccp_device *);
diff --git a/drivers/crypto/ccp/ccp-ops.c b/drivers/crypto/ccp/ccp-ops.c
index f1396c3..c0dfdac 100644
--- a/drivers/crypto/ccp/ccp-ops.c
+++ b/drivers/crypto/ccp/ccp-ops.c
@@ -16,6 +16,7 @@
#include <linux/pci.h>
#include <linux/interrupt.h>
#include <crypto/scatterwalk.h>
+#include <crypto/des.h>
#include <linux/ccp.h>
#include "ccp-dev.h"
@@ -41,6 +42,20 @@
cpu_to_be32(SHA256_H6), cpu_to_be32(SHA256_H7),
};
+static const __be64 ccp_sha384_init[SHA512_DIGEST_SIZE / sizeof(__be64)] = {
+ cpu_to_be64(SHA384_H0), cpu_to_be64(SHA384_H1),
+ cpu_to_be64(SHA384_H2), cpu_to_be64(SHA384_H3),
+ cpu_to_be64(SHA384_H4), cpu_to_be64(SHA384_H5),
+ cpu_to_be64(SHA384_H6), cpu_to_be64(SHA384_H7),
+};
+
+static const __be64 ccp_sha512_init[SHA512_DIGEST_SIZE / sizeof(__be64)] = {
+ cpu_to_be64(SHA512_H0), cpu_to_be64(SHA512_H1),
+ cpu_to_be64(SHA512_H2), cpu_to_be64(SHA512_H3),
+ cpu_to_be64(SHA512_H4), cpu_to_be64(SHA512_H5),
+ cpu_to_be64(SHA512_H6), cpu_to_be64(SHA512_H7),
+};
+
#define CCP_NEW_JOBID(ccp) ((ccp->vdata->version == CCP_VERSION(3, 0)) ? \
ccp_gen_jobid(ccp) : 0)
@@ -586,6 +601,255 @@
return ret;
}
+static int ccp_run_aes_gcm_cmd(struct ccp_cmd_queue *cmd_q,
+ struct ccp_cmd *cmd)
+{
+ struct ccp_aes_engine *aes = &cmd->u.aes;
+ struct ccp_dm_workarea key, ctx, final_wa, tag;
+ struct ccp_data src, dst;
+ struct ccp_data aad;
+ struct ccp_op op;
+
+ unsigned long long *final;
+ unsigned int dm_offset;
+ unsigned int ilen;
+ bool in_place = true; /* Default value */
+ int ret;
+
+ struct scatterlist *p_inp, sg_inp[2];
+ struct scatterlist *p_tag, sg_tag[2];
+ struct scatterlist *p_outp, sg_outp[2];
+ struct scatterlist *p_aad;
+
+ if (!aes->iv)
+ return -EINVAL;
+
+ if (!((aes->key_len == AES_KEYSIZE_128) ||
+ (aes->key_len == AES_KEYSIZE_192) ||
+ (aes->key_len == AES_KEYSIZE_256)))
+ return -EINVAL;
+
+ if (!aes->key) /* Gotta have a key SGL */
+ return -EINVAL;
+
+ /* First, decompose the source buffer into AAD & PT,
+ * and the destination buffer into AAD, CT & tag, or
+ * the input into CT & tag.
+ * It is expected that the input and output SGs will
+ * be valid, even if the AAD and input lengths are 0.
+ */
+ p_aad = aes->src;
+ p_inp = scatterwalk_ffwd(sg_inp, aes->src, aes->aad_len);
+ p_outp = scatterwalk_ffwd(sg_outp, aes->dst, aes->aad_len);
+ if (aes->action == CCP_AES_ACTION_ENCRYPT) {
+ ilen = aes->src_len;
+ p_tag = scatterwalk_ffwd(sg_tag, p_outp, ilen);
+ } else {
+ /* Input length for decryption includes tag */
+ ilen = aes->src_len - AES_BLOCK_SIZE;
+ p_tag = scatterwalk_ffwd(sg_tag, p_inp, ilen);
+ }
+
+ memset(&op, 0, sizeof(op));
+ op.cmd_q = cmd_q;
+ op.jobid = CCP_NEW_JOBID(cmd_q->ccp);
+ op.sb_key = cmd_q->sb_key; /* Pre-allocated */
+ op.sb_ctx = cmd_q->sb_ctx; /* Pre-allocated */
+ op.init = 1;
+ op.u.aes.type = aes->type;
+
+ /* Copy the key to the LSB */
+ ret = ccp_init_dm_workarea(&key, cmd_q,
+ CCP_AES_CTX_SB_COUNT * CCP_SB_BYTES,
+ DMA_TO_DEVICE);
+ if (ret)
+ return ret;
+
+ dm_offset = CCP_SB_BYTES - aes->key_len;
+ ccp_set_dm_area(&key, dm_offset, aes->key, 0, aes->key_len);
+ ret = ccp_copy_to_sb(cmd_q, &key, op.jobid, op.sb_key,
+ CCP_PASSTHRU_BYTESWAP_256BIT);
+ if (ret) {
+ cmd->engine_error = cmd_q->cmd_error;
+ goto e_key;
+ }
+
+ /* Copy the context (IV) to the LSB.
+ * There is an assumption here that the IV is 96 bits in length, plus
+ * a nonce of 32 bits. If no IV is present, use a zeroed buffer.
+ */
+ ret = ccp_init_dm_workarea(&ctx, cmd_q,
+ CCP_AES_CTX_SB_COUNT * CCP_SB_BYTES,
+ DMA_BIDIRECTIONAL);
+ if (ret)
+ goto e_key;
+
+ dm_offset = CCP_AES_CTX_SB_COUNT * CCP_SB_BYTES - aes->iv_len;
+ ccp_set_dm_area(&ctx, dm_offset, aes->iv, 0, aes->iv_len);
+
+ ret = ccp_copy_to_sb(cmd_q, &ctx, op.jobid, op.sb_ctx,
+ CCP_PASSTHRU_BYTESWAP_256BIT);
+ if (ret) {
+ cmd->engine_error = cmd_q->cmd_error;
+ goto e_ctx;
+ }
+
+ op.init = 1;
+ if (aes->aad_len > 0) {
+ /* Step 1: Run a GHASH over the Additional Authenticated Data */
+ ret = ccp_init_data(&aad, cmd_q, p_aad, aes->aad_len,
+ AES_BLOCK_SIZE,
+ DMA_TO_DEVICE);
+ if (ret)
+ goto e_ctx;
+
+ op.u.aes.mode = CCP_AES_MODE_GHASH;
+ op.u.aes.action = CCP_AES_GHASHAAD;
+
+ while (aad.sg_wa.bytes_left) {
+ ccp_prepare_data(&aad, NULL, &op, AES_BLOCK_SIZE, true);
+
+ ret = cmd_q->ccp->vdata->perform->aes(&op);
+ if (ret) {
+ cmd->engine_error = cmd_q->cmd_error;
+ goto e_aad;
+ }
+
+ ccp_process_data(&aad, NULL, &op);
+ op.init = 0;
+ }
+ }
+
+ op.u.aes.mode = CCP_AES_MODE_GCTR;
+ op.u.aes.action = aes->action;
+
+ if (ilen > 0) {
+ /* Step 2: Run a GCTR over the plaintext */
+ in_place = (sg_virt(p_inp) == sg_virt(p_outp)) ? true : false;
+
+ ret = ccp_init_data(&src, cmd_q, p_inp, ilen,
+ AES_BLOCK_SIZE,
+ in_place ? DMA_BIDIRECTIONAL
+ : DMA_TO_DEVICE);
+ if (ret)
+ goto e_ctx;
+
+ if (in_place) {
+ dst = src;
+ } else {
+ ret = ccp_init_data(&dst, cmd_q, p_outp, ilen,
+ AES_BLOCK_SIZE, DMA_FROM_DEVICE);
+ if (ret)
+ goto e_src;
+ }
+
+ op.soc = 0;
+ op.eom = 0;
+ op.init = 1;
+ while (src.sg_wa.bytes_left) {
+ ccp_prepare_data(&src, &dst, &op, AES_BLOCK_SIZE, true);
+ if (!src.sg_wa.bytes_left) {
+ unsigned int nbytes = aes->src_len
+ % AES_BLOCK_SIZE;
+
+ if (nbytes) {
+ op.eom = 1;
+ op.u.aes.size = (nbytes * 8) - 1;
+ }
+ }
+
+ ret = cmd_q->ccp->vdata->perform->aes(&op);
+ if (ret) {
+ cmd->engine_error = cmd_q->cmd_error;
+ goto e_dst;
+ }
+
+ ccp_process_data(&src, &dst, &op);
+ op.init = 0;
+ }
+ }
+
+ /* Step 3: Update the IV portion of the context with the original IV */
+ ret = ccp_copy_from_sb(cmd_q, &ctx, op.jobid, op.sb_ctx,
+ CCP_PASSTHRU_BYTESWAP_256BIT);
+ if (ret) {
+ cmd->engine_error = cmd_q->cmd_error;
+ goto e_dst;
+ }
+
+ ccp_set_dm_area(&ctx, dm_offset, aes->iv, 0, aes->iv_len);
+
+ ret = ccp_copy_to_sb(cmd_q, &ctx, op.jobid, op.sb_ctx,
+ CCP_PASSTHRU_BYTESWAP_256BIT);
+ if (ret) {
+ cmd->engine_error = cmd_q->cmd_error;
+ goto e_dst;
+ }
+
+ /* Step 4: Concatenate the lengths of the AAD and source, and
+ * hash that 16 byte buffer.
+ */
+ ret = ccp_init_dm_workarea(&final_wa, cmd_q, AES_BLOCK_SIZE,
+ DMA_BIDIRECTIONAL);
+ if (ret)
+ goto e_dst;
+ final = (unsigned long long *) final_wa.address;
+ final[0] = cpu_to_be64(aes->aad_len * 8);
+ final[1] = cpu_to_be64(ilen * 8);
+
+ op.u.aes.mode = CCP_AES_MODE_GHASH;
+ op.u.aes.action = CCP_AES_GHASHFINAL;
+ op.src.type = CCP_MEMTYPE_SYSTEM;
+ op.src.u.dma.address = final_wa.dma.address;
+ op.src.u.dma.length = AES_BLOCK_SIZE;
+ op.dst.type = CCP_MEMTYPE_SYSTEM;
+ op.dst.u.dma.address = final_wa.dma.address;
+ op.dst.u.dma.length = AES_BLOCK_SIZE;
+ op.eom = 1;
+ op.u.aes.size = 0;
+ ret = cmd_q->ccp->vdata->perform->aes(&op);
+ if (ret)
+ goto e_dst;
+
+ if (aes->action == CCP_AES_ACTION_ENCRYPT) {
+ /* Put the ciphered tag after the ciphertext. */
+ ccp_get_dm_area(&final_wa, 0, p_tag, 0, AES_BLOCK_SIZE);
+ } else {
+ /* Does this ciphered tag match the input? */
+ ret = ccp_init_dm_workarea(&tag, cmd_q, AES_BLOCK_SIZE,
+ DMA_BIDIRECTIONAL);
+ if (ret)
+ goto e_tag;
+ ccp_set_dm_area(&tag, 0, p_tag, 0, AES_BLOCK_SIZE);
+
+ ret = memcmp(tag.address, final_wa.address, AES_BLOCK_SIZE);
+ ccp_dm_free(&tag);
+ }
+
+e_tag:
+ ccp_dm_free(&final_wa);
+
+e_dst:
+ if (aes->src_len && !in_place)
+ ccp_free_data(&dst, cmd_q);
+
+e_src:
+ if (aes->src_len)
+ ccp_free_data(&src, cmd_q);
+
+e_aad:
+ if (aes->aad_len)
+ ccp_free_data(&aad, cmd_q);
+
+e_ctx:
+ ccp_dm_free(&ctx);
+
+e_key:
+ ccp_dm_free(&key);
+
+ return ret;
+}
+
static int ccp_run_aes_cmd(struct ccp_cmd_queue *cmd_q, struct ccp_cmd *cmd)
{
struct ccp_aes_engine *aes = &cmd->u.aes;
@@ -599,6 +863,9 @@
if (aes->mode == CCP_AES_MODE_CMAC)
return ccp_run_aes_cmac_cmd(cmd_q, cmd);
+ if (aes->mode == CCP_AES_MODE_GCM)
+ return ccp_run_aes_gcm_cmd(cmd_q, cmd);
+
if (!((aes->key_len == AES_KEYSIZE_128) ||
(aes->key_len == AES_KEYSIZE_192) ||
(aes->key_len == AES_KEYSIZE_256)))
@@ -925,6 +1192,200 @@
return ret;
}
+static int ccp_run_des3_cmd(struct ccp_cmd_queue *cmd_q, struct ccp_cmd *cmd)
+{
+ struct ccp_des3_engine *des3 = &cmd->u.des3;
+
+ struct ccp_dm_workarea key, ctx;
+ struct ccp_data src, dst;
+ struct ccp_op op;
+ unsigned int dm_offset;
+ unsigned int len_singlekey;
+ bool in_place = false;
+ int ret;
+
+ /* Error checks */
+ if (!cmd_q->ccp->vdata->perform->des3)
+ return -EINVAL;
+
+ if (des3->key_len != DES3_EDE_KEY_SIZE)
+ return -EINVAL;
+
+ if (((des3->mode == CCP_DES3_MODE_ECB) ||
+ (des3->mode == CCP_DES3_MODE_CBC)) &&
+ (des3->src_len & (DES3_EDE_BLOCK_SIZE - 1)))
+ return -EINVAL;
+
+ if (!des3->key || !des3->src || !des3->dst)
+ return -EINVAL;
+
+ if (des3->mode != CCP_DES3_MODE_ECB) {
+ if (des3->iv_len != DES3_EDE_BLOCK_SIZE)
+ return -EINVAL;
+
+ if (!des3->iv)
+ return -EINVAL;
+ }
+
+ ret = -EIO;
+ /* Zero out all the fields of the command desc */
+ memset(&op, 0, sizeof(op));
+
+ /* Set up the Function field */
+ op.cmd_q = cmd_q;
+ op.jobid = CCP_NEW_JOBID(cmd_q->ccp);
+ op.sb_key = cmd_q->sb_key;
+
+ op.init = (des3->mode == CCP_DES3_MODE_ECB) ? 0 : 1;
+ op.u.des3.type = des3->type;
+ op.u.des3.mode = des3->mode;
+ op.u.des3.action = des3->action;
+
+ /*
+ * All supported key sizes fit in a single (32-byte) KSB entry and
+ * (like AES) must be in little endian format. Use the 256-bit byte
+ * swap passthru option to convert from big endian to little endian.
+ */
+ ret = ccp_init_dm_workarea(&key, cmd_q,
+ CCP_DES3_KEY_SB_COUNT * CCP_SB_BYTES,
+ DMA_TO_DEVICE);
+ if (ret)
+ return ret;
+
+ /*
+ * The contents of the key triplet are in the reverse order of what
+ * is required by the engine. Copy the 3 pieces individually to put
+ * them where they belong.
+ */
+ dm_offset = CCP_SB_BYTES - des3->key_len; /* Basic offset */
+
+ len_singlekey = des3->key_len / 3;
+ ccp_set_dm_area(&key, dm_offset + 2 * len_singlekey,
+ des3->key, 0, len_singlekey);
+ ccp_set_dm_area(&key, dm_offset + len_singlekey,
+ des3->key, len_singlekey, len_singlekey);
+ ccp_set_dm_area(&key, dm_offset,
+ des3->key, 2 * len_singlekey, len_singlekey);
+
+ /* Copy the key to the SB */
+ ret = ccp_copy_to_sb(cmd_q, &key, op.jobid, op.sb_key,
+ CCP_PASSTHRU_BYTESWAP_256BIT);
+ if (ret) {
+ cmd->engine_error = cmd_q->cmd_error;
+ goto e_key;
+ }
+
+ /*
+ * The DES3 context fits in a single (32-byte) KSB entry and
+ * must be in little endian format. Use the 256-bit byte swap
+ * passthru option to convert from big endian to little endian.
+ */
+ if (des3->mode != CCP_DES3_MODE_ECB) {
+ u32 load_mode;
+
+ op.sb_ctx = cmd_q->sb_ctx;
+
+ ret = ccp_init_dm_workarea(&ctx, cmd_q,
+ CCP_DES3_CTX_SB_COUNT * CCP_SB_BYTES,
+ DMA_BIDIRECTIONAL);
+ if (ret)
+ goto e_key;
+
+ /* Load the context into the LSB */
+ dm_offset = CCP_SB_BYTES - des3->iv_len;
+ ccp_set_dm_area(&ctx, dm_offset, des3->iv, 0, des3->iv_len);
+
+ if (cmd_q->ccp->vdata->version == CCP_VERSION(3, 0))
+ load_mode = CCP_PASSTHRU_BYTESWAP_NOOP;
+ else
+ load_mode = CCP_PASSTHRU_BYTESWAP_256BIT;
+ ret = ccp_copy_to_sb(cmd_q, &ctx, op.jobid, op.sb_ctx,
+ load_mode);
+ if (ret) {
+ cmd->engine_error = cmd_q->cmd_error;
+ goto e_ctx;
+ }
+ }
+
+ /*
+ * Prepare the input and output data workareas. For in-place
+ * operations we need to set the dma direction to BIDIRECTIONAL
+ * and copy the src workarea to the dst workarea.
+ */
+ if (sg_virt(des3->src) == sg_virt(des3->dst))
+ in_place = true;
+
+ ret = ccp_init_data(&src, cmd_q, des3->src, des3->src_len,
+ DES3_EDE_BLOCK_SIZE,
+ in_place ? DMA_BIDIRECTIONAL : DMA_TO_DEVICE);
+ if (ret)
+ goto e_ctx;
+
+ if (in_place)
+ dst = src;
+ else {
+ ret = ccp_init_data(&dst, cmd_q, des3->dst, des3->src_len,
+ DES3_EDE_BLOCK_SIZE, DMA_FROM_DEVICE);
+ if (ret)
+ goto e_src;
+ }
+
+ /* Send data to the CCP DES3 engine */
+ while (src.sg_wa.bytes_left) {
+ ccp_prepare_data(&src, &dst, &op, DES3_EDE_BLOCK_SIZE, true);
+ if (!src.sg_wa.bytes_left) {
+ op.eom = 1;
+
+ /* Since we don't retrieve the context in ECB mode
+ * we have to wait for the operation to complete
+ * on the last piece of data
+ */
+ op.soc = 0;
+ }
+
+ ret = cmd_q->ccp->vdata->perform->des3(&op);
+ if (ret) {
+ cmd->engine_error = cmd_q->cmd_error;
+ goto e_dst;
+ }
+
+ ccp_process_data(&src, &dst, &op);
+ }
+
+ if (des3->mode != CCP_DES3_MODE_ECB) {
+ /* Retrieve the context and make BE */
+ ret = ccp_copy_from_sb(cmd_q, &ctx, op.jobid, op.sb_ctx,
+ CCP_PASSTHRU_BYTESWAP_256BIT);
+ if (ret) {
+ cmd->engine_error = cmd_q->cmd_error;
+ goto e_dst;
+ }
+
+ /* ...but we only need the last DES3_EDE_BLOCK_SIZE bytes */
+ if (cmd_q->ccp->vdata->version == CCP_VERSION(3, 0))
+ dm_offset = CCP_SB_BYTES - des3->iv_len;
+ else
+ dm_offset = 0;
+ ccp_get_dm_area(&ctx, dm_offset, des3->iv, 0,
+ DES3_EDE_BLOCK_SIZE);
+ }
+e_dst:
+ if (!in_place)
+ ccp_free_data(&dst, cmd_q);
+
+e_src:
+ ccp_free_data(&src, cmd_q);
+
+e_ctx:
+ if (des3->mode != CCP_DES3_MODE_ECB)
+ ccp_dm_free(&ctx);
+
+e_key:
+ ccp_dm_free(&key);
+
+ return ret;
+}
+
static int ccp_run_sha_cmd(struct ccp_cmd_queue *cmd_q, struct ccp_cmd *cmd)
{
struct ccp_sha_engine *sha = &cmd->u.sha;
@@ -955,6 +1416,18 @@
return -EINVAL;
block_size = SHA256_BLOCK_SIZE;
break;
+ case CCP_SHA_TYPE_384:
+ if (cmd_q->ccp->vdata->version < CCP_VERSION(4, 0)
+ || sha->ctx_len < SHA384_DIGEST_SIZE)
+ return -EINVAL;
+ block_size = SHA384_BLOCK_SIZE;
+ break;
+ case CCP_SHA_TYPE_512:
+ if (cmd_q->ccp->vdata->version < CCP_VERSION(4, 0)
+ || sha->ctx_len < SHA512_DIGEST_SIZE)
+ return -EINVAL;
+ block_size = SHA512_BLOCK_SIZE;
+ break;
default:
return -EINVAL;
}
@@ -1042,6 +1515,21 @@
sb_count = 1;
ooffset = ioffset = 0;
break;
+ case CCP_SHA_TYPE_384:
+ digest_size = SHA384_DIGEST_SIZE;
+ init = (void *) ccp_sha384_init;
+ ctx_size = SHA512_DIGEST_SIZE;
+ sb_count = 2;
+ ioffset = 0;
+ ooffset = 2 * CCP_SB_BYTES - SHA384_DIGEST_SIZE;
+ break;
+ case CCP_SHA_TYPE_512:
+ digest_size = SHA512_DIGEST_SIZE;
+ init = (void *) ccp_sha512_init;
+ ctx_size = SHA512_DIGEST_SIZE;
+ sb_count = 2;
+ ooffset = ioffset = 0;
+ break;
default:
ret = -EINVAL;
goto e_data;
@@ -1060,6 +1548,11 @@
op.u.sha.type = sha->type;
op.u.sha.msg_bits = sha->msg_bits;
+ /* For SHA1/224/256 the context fits in a single (32-byte) SB entry;
+ * SHA384/512 require 2 adjacent SB slots, with the right half in the
+ * first slot, and the left half in the second. Each portion must then
+ * be in little endian format: use the 256-bit byte swap option.
+ */
ret = ccp_init_dm_workarea(&ctx, cmd_q, sb_count * CCP_SB_BYTES,
DMA_BIDIRECTIONAL);
if (ret)
@@ -1071,6 +1564,13 @@
case CCP_SHA_TYPE_256:
memcpy(ctx.address + ioffset, init, ctx_size);
break;
+ case CCP_SHA_TYPE_384:
+ case CCP_SHA_TYPE_512:
+ memcpy(ctx.address + ctx_size / 2, init,
+ ctx_size / 2);
+ memcpy(ctx.address, init + ctx_size / 2,
+ ctx_size / 2);
+ break;
default:
ret = -EINVAL;
goto e_ctx;
@@ -1137,6 +1637,15 @@
sha->ctx, 0,
digest_size);
break;
+ case CCP_SHA_TYPE_384:
+ case CCP_SHA_TYPE_512:
+ ccp_get_dm_area(&ctx, 0,
+ sha->ctx, LSB_ITEM_SIZE - ooffset,
+ LSB_ITEM_SIZE);
+ ccp_get_dm_area(&ctx, LSB_ITEM_SIZE + ooffset,
+ sha->ctx, 0,
+ LSB_ITEM_SIZE - ooffset);
+ break;
default:
ret = -EINVAL;
goto e_ctx;
@@ -1174,6 +1683,16 @@
ctx.address + ooffset,
digest_size);
break;
+ case CCP_SHA_TYPE_384:
+ case CCP_SHA_TYPE_512:
+ memcpy(hmac_buf + block_size,
+ ctx.address + LSB_ITEM_SIZE + ooffset,
+ LSB_ITEM_SIZE);
+ memcpy(hmac_buf + block_size +
+ (LSB_ITEM_SIZE - ooffset),
+ ctx.address,
+ LSB_ITEM_SIZE);
+ break;
default:
ret = -EINVAL;
goto e_ctx;
@@ -1831,6 +2350,9 @@
case CCP_ENGINE_XTS_AES_128:
ret = ccp_run_xts_aes_cmd(cmd_q, cmd);
break;
+ case CCP_ENGINE_DES3:
+ ret = ccp_run_des3_cmd(cmd_q, cmd);
+ break;
case CCP_ENGINE_SHA:
ret = ccp_run_sha_cmd(cmd_q, cmd);
break;
diff --git a/drivers/crypto/ccp/ccp-pci.c b/drivers/crypto/ccp/ccp-pci.c
index 28a9996..e880d4cf4 100644
--- a/drivers/crypto/ccp/ccp-pci.c
+++ b/drivers/crypto/ccp/ccp-pci.c
@@ -69,6 +69,7 @@
goto e_irq;
}
}
+ ccp->use_tasklet = true;
return 0;
@@ -100,6 +101,7 @@
dev_notice(dev, "unable to allocate MSI IRQ (%d)\n", ret);
goto e_msi;
}
+ ccp->use_tasklet = true;
return 0;
diff --git a/drivers/crypto/chelsio/chcr_algo.c b/drivers/crypto/chelsio/chcr_algo.c
index 41bc7f4..f00e0d8 100644
--- a/drivers/crypto/chelsio/chcr_algo.c
+++ b/drivers/crypto/chelsio/chcr_algo.c
@@ -294,7 +294,7 @@
static struct crypto_shash *chcr_alloc_shash(unsigned int ds)
{
- struct crypto_shash *base_hash = NULL;
+ struct crypto_shash *base_hash = ERR_PTR(-EINVAL);
switch (ds) {
case SHA1_DIGEST_SIZE:
@@ -522,7 +522,7 @@
{
struct uld_ctx *u_ctx = ULD_CTX(ctx);
int iv_loc = IV_DSGL;
- int qid = u_ctx->lldi.rxq_ids[ctx->tx_channel_id];
+ int qid = u_ctx->lldi.rxq_ids[ctx->rx_qidx];
unsigned int immdatalen = 0, nr_frags = 0;
if (is_ofld_imm(skb)) {
@@ -543,7 +543,7 @@
chcr_req->wreq.cookie = cpu_to_be64((uintptr_t)req);
chcr_req->wreq.rx_chid_to_rx_q_id =
FILL_WR_RX_Q_ID(ctx->dev->rx_channel_id, qid,
- is_iv ? iv_loc : IV_NOP, ctx->tx_channel_id);
+ is_iv ? iv_loc : IV_NOP, ctx->tx_qidx);
chcr_req->ulptx.cmd_dest = FILL_ULPTX_CMD_DEST(ctx->dev->tx_channel_id,
qid);
@@ -721,19 +721,19 @@
struct sk_buff *skb;
if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0],
- ctx->tx_channel_id))) {
+ ctx->tx_qidx))) {
if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG))
return -EBUSY;
}
- skb = create_cipher_wr(req, u_ctx->lldi.rxq_ids[ctx->tx_channel_id],
+ skb = create_cipher_wr(req, u_ctx->lldi.rxq_ids[ctx->rx_qidx],
CHCR_ENCRYPT_OP);
if (IS_ERR(skb)) {
pr_err("chcr : %s : Failed to form WR. No memory\n", __func__);
return PTR_ERR(skb);
}
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
}
@@ -746,19 +746,19 @@
struct sk_buff *skb;
if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0],
- ctx->tx_channel_id))) {
+ ctx->tx_qidx))) {
if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG))
return -EBUSY;
}
- skb = create_cipher_wr(req, u_ctx->lldi.rxq_ids[0],
+ skb = create_cipher_wr(req, u_ctx->lldi.rxq_ids[ctx->rx_qidx],
CHCR_DECRYPT_OP);
if (IS_ERR(skb)) {
pr_err("chcr : %s : Failed to form WR. No memory\n", __func__);
return PTR_ERR(skb);
}
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
}
@@ -766,7 +766,9 @@
static int chcr_device_init(struct chcr_context *ctx)
{
struct uld_ctx *u_ctx;
+ struct adapter *adap;
unsigned int id;
+ int txq_perchan, txq_idx, ntxq;
int err = 0, rxq_perchan, rxq_idx;
id = smp_processor_id();
@@ -777,11 +779,18 @@
goto out;
}
u_ctx = ULD_CTX(ctx);
+ adap = padap(ctx->dev);
+ ntxq = min_not_zero((unsigned int)u_ctx->lldi.nrxq,
+ adap->vres.ncrypto_fc);
rxq_perchan = u_ctx->lldi.nrxq / u_ctx->lldi.nchan;
+ txq_perchan = ntxq / u_ctx->lldi.nchan;
rxq_idx = ctx->dev->tx_channel_id * rxq_perchan;
rxq_idx += id % rxq_perchan;
+ txq_idx = ctx->dev->tx_channel_id * txq_perchan;
+ txq_idx += id % txq_perchan;
spin_lock(&ctx->dev->lock_chcr_dev);
- ctx->tx_channel_id = rxq_idx;
+ ctx->rx_qidx = rxq_idx;
+ ctx->tx_qidx = txq_idx;
ctx->dev->tx_channel_id = !ctx->dev->tx_channel_id;
ctx->dev->rx_channel_id = 0;
spin_unlock(&ctx->dev->lock_chcr_dev);
@@ -935,7 +944,7 @@
u_ctx = ULD_CTX(ctx);
if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0],
- ctx->tx_channel_id))) {
+ ctx->tx_qidx))) {
if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG))
return -EBUSY;
}
@@ -975,7 +984,7 @@
}
req_ctx->reqlen = remainder;
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
@@ -1028,7 +1037,7 @@
return -ENOMEM;
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
}
@@ -1047,7 +1056,7 @@
u_ctx = ULD_CTX(ctx);
if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0],
- ctx->tx_channel_id))) {
+ ctx->tx_qidx))) {
if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG))
return -EBUSY;
}
@@ -1079,7 +1088,7 @@
return -ENOMEM;
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
@@ -1100,7 +1109,7 @@
u_ctx = ULD_CTX(ctx);
if (unlikely(cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0],
- ctx->tx_channel_id))) {
+ ctx->tx_qidx))) {
if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG))
return -EBUSY;
}
@@ -1130,7 +1139,7 @@
return -ENOMEM;
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
}
@@ -1334,20 +1343,36 @@
return crypto_skcipher_encrypt(skreq);
}
-
-static unsigned char get_hmac(unsigned int authsize)
+static int chcr_aead_need_fallback(struct aead_request *req, int src_nent,
+ int aadmax, int wrlen,
+ unsigned short op_type)
{
- switch (authsize) {
- case ICV_8:
- return CHCR_SCMD_HMAC_CTRL_PL1;
- case ICV_10:
- return CHCR_SCMD_HMAC_CTRL_TRUNC_RFC4366;
- case ICV_12:
- return CHCR_SCMD_HMAC_CTRL_IPSEC_96BIT;
- }
- return CHCR_SCMD_HMAC_CTRL_NO_TRUNC;
+ unsigned int authsize = crypto_aead_authsize(crypto_aead_reqtfm(req));
+
+ if (((req->cryptlen - (op_type ? authsize : 0)) == 0) ||
+ (req->assoclen > aadmax) ||
+ (src_nent > MAX_SKB_FRAGS) ||
+ (wrlen > MAX_WR_SIZE))
+ return 1;
+ return 0;
}
+static int chcr_aead_fallback(struct aead_request *req, unsigned short op_type)
+{
+ struct crypto_aead *tfm = crypto_aead_reqtfm(req);
+ struct chcr_context *ctx = crypto_aead_ctx(tfm);
+ struct chcr_aead_ctx *aeadctx = AEAD_CTX(ctx);
+ struct aead_request *subreq = aead_request_ctx(req);
+
+ aead_request_set_tfm(subreq, aeadctx->sw_cipher);
+ aead_request_set_callback(subreq, req->base.flags,
+ req->base.complete, req->base.data);
+ aead_request_set_crypt(subreq, req->src, req->dst, req->cryptlen,
+ req->iv);
+ aead_request_set_ad(subreq, req->assoclen);
+ return op_type ? crypto_aead_decrypt(subreq) :
+ crypto_aead_encrypt(subreq);
+}
static struct sk_buff *create_authenc_wr(struct aead_request *req,
unsigned short qid,
@@ -1371,7 +1396,7 @@
unsigned short stop_offset = 0;
unsigned int assoclen = req->assoclen;
unsigned int authsize = crypto_aead_authsize(tfm);
- int err = 0;
+ int err = -EINVAL, src_nent;
int null = 0;
gfp_t flags = req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP ? GFP_KERNEL :
GFP_ATOMIC;
@@ -1381,8 +1406,8 @@
if (op_type && req->cryptlen < crypto_aead_authsize(tfm))
goto err;
-
- if (sg_nents_for_len(req->src, req->assoclen + req->cryptlen) < 0)
+ src_nent = sg_nents_for_len(req->src, req->assoclen + req->cryptlen);
+ if (src_nent < 0)
goto err;
src = scatterwalk_ffwd(reqctx->srcffwd, req->src, req->assoclen);
reqctx->dst = src;
@@ -1400,7 +1425,7 @@
}
reqctx->dst_nents = sg_nents_for_len(reqctx->dst, req->cryptlen +
(op_type ? -authsize : authsize));
- if (reqctx->dst_nents <= 0) {
+ if (reqctx->dst_nents < 0) {
pr_err("AUTHENC:Invalid Destination sg entries\n");
goto err;
}
@@ -1408,6 +1433,12 @@
kctx_len = (ntohl(KEY_CONTEXT_CTX_LEN_V(aeadctx->key_ctx_hdr)) << 4)
- sizeof(chcr_req->key_ctx);
transhdr_len = CIPHER_TRANSHDR_SIZE(kctx_len, dst_size);
+ if (chcr_aead_need_fallback(req, src_nent + MIN_AUTH_SG,
+ T6_MAX_AAD_SIZE,
+ transhdr_len + (sgl_len(src_nent + MIN_AUTH_SG) * 8),
+ op_type)) {
+ return ERR_PTR(chcr_aead_fallback(req, op_type));
+ }
skb = alloc_skb((transhdr_len + sizeof(struct sge_opaque_hdr)), flags);
if (!skb)
goto err;
@@ -1489,24 +1520,6 @@
return ERR_PTR(-EINVAL);
}
-static void aes_gcm_empty_pld_pad(struct scatterlist *sg,
- unsigned short offset)
-{
- struct page *spage;
- unsigned char *addr;
-
- spage = sg_page(sg);
- get_page(spage); /* so that it is not freed by NIC */
-#ifdef KMAP_ATOMIC_ARGS
- addr = kmap_atomic(spage, KM_SOFTIRQ0);
-#else
- addr = kmap_atomic(spage);
-#endif
- memset(addr + sg->offset, 0, offset + 1);
-
- kunmap_atomic(addr);
-}
-
static int set_msg_len(u8 *block, unsigned int msglen, int csize)
{
__be32 data;
@@ -1570,11 +1583,6 @@
struct chcr_aead_reqctx *reqctx = aead_request_ctx(req);
int rc = 0;
- if (req->assoclen > T5_MAX_AAD_SIZE) {
- pr_err("CCM: Unsupported AAD data. It should be < %d\n",
- T5_MAX_AAD_SIZE);
- return -EINVAL;
- }
if (sub_type == CRYPTO_ALG_SUB_TYPE_AEAD_RFC4309) {
reqctx->iv[0] = 3;
memcpy(reqctx->iv + 1, &aeadctx->salt[0], 3);
@@ -1600,13 +1608,13 @@
struct chcr_context *chcrctx)
{
struct crypto_aead *tfm = crypto_aead_reqtfm(req);
+ struct chcr_aead_ctx *aeadctx = AEAD_CTX(crypto_aead_ctx(tfm));
unsigned int ivsize = AES_BLOCK_SIZE;
unsigned int cipher_mode = CHCR_SCMD_CIPHER_MODE_AES_CCM;
unsigned int mac_mode = CHCR_SCMD_AUTH_MODE_CBCMAC;
unsigned int c_id = chcrctx->dev->rx_channel_id;
unsigned int ccm_xtra;
unsigned char tag_offset = 0, auth_offset = 0;
- unsigned char hmac_ctrl = get_hmac(crypto_aead_authsize(tfm));
unsigned int assoclen;
if (get_aead_subtype(tfm) == CRYPTO_ALG_SUB_TYPE_AEAD_RFC4309)
@@ -1642,8 +1650,8 @@
crypto_aead_authsize(tfm));
sec_cpl->seqno_numivs = FILL_SEC_CPL_SCMD0_SEQNO(op_type,
(op_type == CHCR_ENCRYPT_OP) ? 0 : 1,
- cipher_mode, mac_mode, hmac_ctrl,
- ivsize >> 1);
+ cipher_mode, mac_mode,
+ aeadctx->hmac_ctrl, ivsize >> 1);
sec_cpl->ivgen_hdrlen = FILL_SEC_CPL_IVGEN_HDRLEN(0, 0, 1, 0,
1, dst_size);
@@ -1719,16 +1727,17 @@
unsigned int dst_size = 0, kctx_len;
unsigned int sub_type;
unsigned int authsize = crypto_aead_authsize(tfm);
- int err = 0;
+ int err = -EINVAL, src_nent;
gfp_t flags = req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP ? GFP_KERNEL :
GFP_ATOMIC;
if (op_type && req->cryptlen < crypto_aead_authsize(tfm))
goto err;
-
- if (sg_nents_for_len(req->src, req->assoclen + req->cryptlen) < 0)
+ src_nent = sg_nents_for_len(req->src, req->assoclen + req->cryptlen);
+ if (src_nent < 0)
goto err;
+
sub_type = get_aead_subtype(tfm);
src = scatterwalk_ffwd(reqctx->srcffwd, req->src, req->assoclen);
reqctx->dst = src;
@@ -1744,7 +1753,7 @@
}
reqctx->dst_nents = sg_nents_for_len(reqctx->dst, req->cryptlen +
(op_type ? -authsize : authsize));
- if (reqctx->dst_nents <= 0) {
+ if (reqctx->dst_nents < 0) {
pr_err("CCM:Invalid Destination sg entries\n");
goto err;
}
@@ -1756,6 +1765,13 @@
dst_size = get_space_for_phys_dsgl(reqctx->dst_nents);
kctx_len = ((DIV_ROUND_UP(aeadctx->enckey_len, 16)) << 4) * 2;
transhdr_len = CIPHER_TRANSHDR_SIZE(kctx_len, dst_size);
+ if (chcr_aead_need_fallback(req, src_nent + MIN_CCM_SG,
+ T6_MAX_AAD_SIZE - 18,
+ transhdr_len + (sgl_len(src_nent + MIN_CCM_SG) * 8),
+ op_type)) {
+ return ERR_PTR(chcr_aead_fallback(req, op_type));
+ }
+
skb = alloc_skb((transhdr_len + sizeof(struct sge_opaque_hdr)), flags);
if (!skb)
@@ -1820,8 +1836,7 @@
unsigned char tag_offset = 0;
unsigned int crypt_len = 0;
unsigned int authsize = crypto_aead_authsize(tfm);
- unsigned char hmac_ctrl = get_hmac(authsize);
- int err = 0;
+ int err = -EINVAL, src_nent;
gfp_t flags = req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP ? GFP_KERNEL :
GFP_ATOMIC;
@@ -1831,8 +1846,8 @@
if (op_type && req->cryptlen < crypto_aead_authsize(tfm))
goto err;
-
- if (sg_nents_for_len(req->src, req->assoclen + req->cryptlen) < 0)
+ src_nent = sg_nents_for_len(req->src, req->assoclen + req->cryptlen);
+ if (src_nent < 0)
goto err;
src = scatterwalk_ffwd(reqctx->srcffwd, req->src, req->assoclen);
@@ -1854,7 +1869,7 @@
crypt_len = req->cryptlen;
reqctx->dst_nents = sg_nents_for_len(reqctx->dst, req->cryptlen +
(op_type ? -authsize : authsize));
- if (reqctx->dst_nents <= 0) {
+ if (reqctx->dst_nents < 0) {
pr_err("GCM:Invalid Destination sg entries\n");
goto err;
}
@@ -1864,6 +1879,12 @@
kctx_len = ((DIV_ROUND_UP(aeadctx->enckey_len, 16)) << 4) +
AEAD_H_SIZE;
transhdr_len = CIPHER_TRANSHDR_SIZE(kctx_len, dst_size);
+ if (chcr_aead_need_fallback(req, src_nent + MIN_GCM_SG,
+ T6_MAX_AAD_SIZE,
+ transhdr_len + (sgl_len(src_nent + MIN_GCM_SG) * 8),
+ op_type)) {
+ return ERR_PTR(chcr_aead_fallback(req, op_type));
+ }
skb = alloc_skb((transhdr_len + sizeof(struct sge_opaque_hdr)), flags);
if (!skb)
goto err;
@@ -1881,11 +1902,11 @@
chcr_req->sec_cpl.op_ivinsrtofst = FILL_SEC_CPL_OP_IVINSR(
ctx->dev->rx_channel_id, 2, (ivsize ?
(req->assoclen + 1) : 0));
- chcr_req->sec_cpl.pldlen = htonl(req->assoclen + ivsize + crypt_len);
+ chcr_req->sec_cpl.pldlen =
+ htonl(req->assoclen + ivsize + req->cryptlen);
chcr_req->sec_cpl.aadstart_cipherstop_hi = FILL_SEC_CPL_CIPHERSTOP_HI(
req->assoclen ? 1 : 0, req->assoclen,
req->assoclen + ivsize + 1, 0);
- if (req->cryptlen) {
chcr_req->sec_cpl.cipherstop_lo_authinsert =
FILL_SEC_CPL_AUTHINSERT(0, req->assoclen + ivsize + 1,
tag_offset, tag_offset);
@@ -1893,17 +1914,8 @@
FILL_SEC_CPL_SCMD0_SEQNO(op_type, (op_type ==
CHCR_ENCRYPT_OP) ? 1 : 0,
CHCR_SCMD_CIPHER_MODE_AES_GCM,
- CHCR_SCMD_AUTH_MODE_GHASH, hmac_ctrl,
- ivsize >> 1);
- } else {
- chcr_req->sec_cpl.cipherstop_lo_authinsert =
- FILL_SEC_CPL_AUTHINSERT(0, 0, 0, 0);
- chcr_req->sec_cpl.seqno_numivs =
- FILL_SEC_CPL_SCMD0_SEQNO(op_type,
- (op_type == CHCR_ENCRYPT_OP) ?
- 1 : 0, CHCR_SCMD_CIPHER_MODE_AES_CBC,
- 0, 0, ivsize >> 1);
- }
+ CHCR_SCMD_AUTH_MODE_GHASH,
+ aeadctx->hmac_ctrl, ivsize >> 1);
chcr_req->sec_cpl.ivgen_hdrlen = FILL_SEC_CPL_IVGEN_HDRLEN(0, 0, 1,
0, 1, dst_size);
chcr_req->key_ctx.ctx_hdr = aeadctx->key_ctx_hdr;
@@ -1936,15 +1948,7 @@
write_sg_to_skb(skb, &frags, req->src, req->assoclen);
write_buffer_to_skb(skb, &frags, reqctx->iv, ivsize);
-
- if (req->cryptlen) {
- write_sg_to_skb(skb, &frags, src, req->cryptlen);
- } else {
- aes_gcm_empty_pld_pad(req->dst, authsize - 1);
- write_sg_to_skb(skb, &frags, reqctx->dst, crypt_len);
-
- }
-
+ write_sg_to_skb(skb, &frags, src, req->cryptlen);
create_wreq(ctx, chcr_req, req, skb, kctx_len, size, 1,
sizeof(struct cpl_rx_phys_dsgl) + dst_size);
reqctx->skb = skb;
@@ -1965,8 +1969,15 @@
{
struct chcr_context *ctx = crypto_aead_ctx(tfm);
struct chcr_aead_ctx *aeadctx = AEAD_CTX(ctx);
+ struct aead_alg *alg = crypto_aead_alg(tfm);
- crypto_aead_set_reqsize(tfm, sizeof(struct chcr_aead_reqctx));
+ aeadctx->sw_cipher = crypto_alloc_aead(alg->base.cra_name, 0,
+ CRYPTO_ALG_NEED_FALLBACK);
+ if (IS_ERR(aeadctx->sw_cipher))
+ return PTR_ERR(aeadctx->sw_cipher);
+ crypto_aead_set_reqsize(tfm, max(sizeof(struct chcr_aead_reqctx),
+ sizeof(struct aead_request) +
+ crypto_aead_reqsize(aeadctx->sw_cipher)));
aeadctx->null = crypto_get_default_null_skcipher();
if (IS_ERR(aeadctx->null))
return PTR_ERR(aeadctx->null);
@@ -1975,7 +1986,11 @@
static void chcr_aead_cra_exit(struct crypto_aead *tfm)
{
+ struct chcr_context *ctx = crypto_aead_ctx(tfm);
+ struct chcr_aead_ctx *aeadctx = AEAD_CTX(ctx);
+
crypto_put_default_null_skcipher();
+ crypto_free_aead(aeadctx->sw_cipher);
}
static int chcr_authenc_null_setauthsize(struct crypto_aead *tfm,
@@ -1985,7 +2000,7 @@
aeadctx->hmac_ctrl = CHCR_SCMD_HMAC_CTRL_NOP;
aeadctx->mayverify = VERIFY_HW;
- return 0;
+ return crypto_aead_setauthsize(aeadctx->sw_cipher, authsize);
}
static int chcr_authenc_setauthsize(struct crypto_aead *tfm,
unsigned int authsize)
@@ -2022,7 +2037,7 @@
aeadctx->hmac_ctrl = CHCR_SCMD_HMAC_CTRL_NO_TRUNC;
aeadctx->mayverify = VERIFY_SW;
}
- return 0;
+ return crypto_aead_setauthsize(aeadctx->sw_cipher, authsize);
}
@@ -2062,7 +2077,7 @@
CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
}
- return 0;
+ return crypto_aead_setauthsize(aeadctx->sw_cipher, authsize);
}
static int chcr_4106_4309_setauthsize(struct crypto_aead *tfm,
@@ -2088,7 +2103,7 @@
CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
}
- return 0;
+ return crypto_aead_setauthsize(aeadctx->sw_cipher, authsize);
}
static int chcr_ccm_setauthsize(struct crypto_aead *tfm,
@@ -2130,10 +2145,10 @@
CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
}
- return 0;
+ return crypto_aead_setauthsize(aeadctx->sw_cipher, authsize);
}
-static int chcr_aead_ccm_setkey(struct crypto_aead *aead,
+static int chcr_ccm_common_setkey(struct crypto_aead *aead,
const u8 *key,
unsigned int keylen)
{
@@ -2142,8 +2157,6 @@
unsigned char ck_size, mk_size;
int key_ctx_size = 0;
- memcpy(aeadctx->key, key, keylen);
- aeadctx->enckey_len = keylen;
key_ctx_size = sizeof(struct _key_ctx) +
((DIV_ROUND_UP(keylen, 16)) << 4) * 2;
if (keylen == AES_KEYSIZE_128) {
@@ -2163,9 +2176,32 @@
}
aeadctx->key_ctx_hdr = FILL_KEY_CTX_HDR(ck_size, mk_size, 0, 0,
key_ctx_size >> 4);
+ memcpy(aeadctx->key, key, keylen);
+ aeadctx->enckey_len = keylen;
+
return 0;
}
+static int chcr_aead_ccm_setkey(struct crypto_aead *aead,
+ const u8 *key,
+ unsigned int keylen)
+{
+ struct chcr_context *ctx = crypto_aead_ctx(aead);
+ struct chcr_aead_ctx *aeadctx = AEAD_CTX(ctx);
+ int error;
+
+ crypto_aead_clear_flags(aeadctx->sw_cipher, CRYPTO_TFM_REQ_MASK);
+ crypto_aead_set_flags(aeadctx->sw_cipher, crypto_aead_get_flags(aead) &
+ CRYPTO_TFM_REQ_MASK);
+ error = crypto_aead_setkey(aeadctx->sw_cipher, key, keylen);
+ crypto_aead_clear_flags(aead, CRYPTO_TFM_RES_MASK);
+ crypto_aead_set_flags(aead, crypto_aead_get_flags(aeadctx->sw_cipher) &
+ CRYPTO_TFM_RES_MASK);
+ if (error)
+ return error;
+ return chcr_ccm_common_setkey(aead, key, keylen);
+}
+
static int chcr_aead_rfc4309_setkey(struct crypto_aead *aead, const u8 *key,
unsigned int keylen)
{
@@ -2180,7 +2216,7 @@
}
keylen -= 3;
memcpy(aeadctx->salt, key + keylen, 3);
- return chcr_aead_ccm_setkey(aead, key, keylen);
+ return chcr_ccm_common_setkey(aead, key, keylen);
}
static int chcr_gcm_setkey(struct crypto_aead *aead, const u8 *key,
@@ -2193,6 +2229,17 @@
unsigned int ck_size;
int ret = 0, key_ctx_size = 0;
+ aeadctx->enckey_len = 0;
+ crypto_aead_clear_flags(aeadctx->sw_cipher, CRYPTO_TFM_REQ_MASK);
+ crypto_aead_set_flags(aeadctx->sw_cipher, crypto_aead_get_flags(aead)
+ & CRYPTO_TFM_REQ_MASK);
+ ret = crypto_aead_setkey(aeadctx->sw_cipher, key, keylen);
+ crypto_aead_clear_flags(aead, CRYPTO_TFM_RES_MASK);
+ crypto_aead_set_flags(aead, crypto_aead_get_flags(aeadctx->sw_cipher) &
+ CRYPTO_TFM_RES_MASK);
+ if (ret)
+ goto out;
+
if (get_aead_subtype(aead) == CRYPTO_ALG_SUB_TYPE_AEAD_RFC4106 &&
keylen > 3) {
keylen -= 4; /* nonce/salt is present in the last 4 bytes */
@@ -2207,8 +2254,7 @@
} else {
crypto_tfm_set_flags((struct crypto_tfm *)aead,
CRYPTO_TFM_RES_BAD_KEY_LEN);
- aeadctx->enckey_len = 0;
- pr_err("GCM: Invalid key length %d", keylen);
+ pr_err("GCM: Invalid key length %d\n", keylen);
ret = -EINVAL;
goto out;
}
@@ -2259,11 +2305,21 @@
int err = 0, i, key_ctx_len = 0;
unsigned char ck_size = 0;
unsigned char pad[CHCR_HASH_MAX_BLOCK_SIZE_128] = { 0 };
- struct crypto_shash *base_hash = NULL;
+ struct crypto_shash *base_hash = ERR_PTR(-EINVAL);
struct algo_param param;
int align;
u8 *o_ptr = NULL;
+ crypto_aead_clear_flags(aeadctx->sw_cipher, CRYPTO_TFM_REQ_MASK);
+ crypto_aead_set_flags(aeadctx->sw_cipher, crypto_aead_get_flags(authenc)
+ & CRYPTO_TFM_REQ_MASK);
+ err = crypto_aead_setkey(aeadctx->sw_cipher, key, keylen);
+ crypto_aead_clear_flags(authenc, CRYPTO_TFM_RES_MASK);
+ crypto_aead_set_flags(authenc, crypto_aead_get_flags(aeadctx->sw_cipher)
+ & CRYPTO_TFM_RES_MASK);
+ if (err)
+ goto out;
+
if (crypto_authenc_extractkeys(&keys, key, keylen) != 0) {
crypto_aead_set_flags(authenc, CRYPTO_TFM_RES_BAD_KEY_LEN);
goto out;
@@ -2296,7 +2352,8 @@
base_hash = chcr_alloc_shash(max_authsize);
if (IS_ERR(base_hash)) {
pr_err("chcr : Base driver cannot be loaded\n");
- goto out;
+ aeadctx->enckey_len = 0;
+ return -EINVAL;
}
{
SHASH_DESC_ON_STACK(shash, base_hash);
@@ -2351,7 +2408,7 @@
}
out:
aeadctx->enckey_len = 0;
- if (base_hash)
+ if (!IS_ERR(base_hash))
chcr_free_shash(base_hash);
return -EINVAL;
}
@@ -2363,11 +2420,21 @@
struct chcr_aead_ctx *aeadctx = AEAD_CTX(ctx);
struct chcr_authenc_ctx *actx = AUTHENC_CTX(aeadctx);
struct crypto_authenc_keys keys;
-
+ int err;
/* it contains auth and cipher key both*/
int key_ctx_len = 0;
unsigned char ck_size = 0;
+ crypto_aead_clear_flags(aeadctx->sw_cipher, CRYPTO_TFM_REQ_MASK);
+ crypto_aead_set_flags(aeadctx->sw_cipher, crypto_aead_get_flags(authenc)
+ & CRYPTO_TFM_REQ_MASK);
+ err = crypto_aead_setkey(aeadctx->sw_cipher, key, keylen);
+ crypto_aead_clear_flags(authenc, CRYPTO_TFM_RES_MASK);
+ crypto_aead_set_flags(authenc, crypto_aead_get_flags(aeadctx->sw_cipher)
+ & CRYPTO_TFM_RES_MASK);
+ if (err)
+ goto out;
+
if (crypto_authenc_extractkeys(&keys, key, keylen) != 0) {
crypto_aead_set_flags(authenc, CRYPTO_TFM_RES_BAD_KEY_LEN);
goto out;
@@ -2465,22 +2532,20 @@
}
u_ctx = ULD_CTX(ctx);
if (cxgb4_is_crypto_q_full(u_ctx->lldi.ports[0],
- ctx->tx_channel_id)) {
+ ctx->tx_qidx)) {
if (!(req->base.flags & CRYPTO_TFM_REQ_MAY_BACKLOG))
return -EBUSY;
}
/* Form a WR from req */
- skb = create_wr_fn(req, u_ctx->lldi.rxq_ids[ctx->tx_channel_id], size,
+ skb = create_wr_fn(req, u_ctx->lldi.rxq_ids[ctx->rx_qidx], size,
op_type);
- if (IS_ERR(skb) || skb == NULL) {
- pr_err("chcr : %s : failed to form WR. No memory\n", __func__);
+ if (IS_ERR(skb) || !skb)
return PTR_ERR(skb);
- }
skb->dev = u_ctx->lldi.ports[0];
- set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_channel_id);
+ set_wr_txq(skb, CPL_PRIORITY_DATA, ctx->tx_qidx);
chcr_send_wr(skb);
return -EINPROGRESS;
}
@@ -2673,6 +2738,7 @@
.cra_name = "gcm(aes)",
.cra_driver_name = "gcm-aes-chcr",
.cra_blocksize = 1,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_gcm_ctx),
@@ -2691,6 +2757,7 @@
.cra_name = "rfc4106(gcm(aes))",
.cra_driver_name = "rfc4106-gcm-aes-chcr",
.cra_blocksize = 1,
+ .cra_priority = CHCR_AEAD_PRIORITY + 1,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_gcm_ctx),
@@ -2710,6 +2777,7 @@
.cra_name = "ccm(aes)",
.cra_driver_name = "ccm-aes-chcr",
.cra_blocksize = 1,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx),
@@ -2728,6 +2796,7 @@
.cra_name = "rfc4309(ccm(aes))",
.cra_driver_name = "rfc4309-ccm-aes-chcr",
.cra_blocksize = 1,
+ .cra_priority = CHCR_AEAD_PRIORITY + 1,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx),
@@ -2747,6 +2816,7 @@
.cra_driver_name =
"authenc-hmac-sha1-cbc-aes-chcr",
.cra_blocksize = AES_BLOCK_SIZE,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_authenc_ctx),
@@ -2768,6 +2838,7 @@
.cra_driver_name =
"authenc-hmac-sha256-cbc-aes-chcr",
.cra_blocksize = AES_BLOCK_SIZE,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_authenc_ctx),
@@ -2788,6 +2859,7 @@
.cra_driver_name =
"authenc-hmac-sha224-cbc-aes-chcr",
.cra_blocksize = AES_BLOCK_SIZE,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_authenc_ctx),
@@ -2807,6 +2879,7 @@
.cra_driver_name =
"authenc-hmac-sha384-cbc-aes-chcr",
.cra_blocksize = AES_BLOCK_SIZE,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_authenc_ctx),
@@ -2827,6 +2900,7 @@
.cra_driver_name =
"authenc-hmac-sha512-cbc-aes-chcr",
.cra_blocksize = AES_BLOCK_SIZE,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_authenc_ctx),
@@ -2847,6 +2921,7 @@
.cra_driver_name =
"authenc-digest_null-cbc-aes-chcr",
.cra_blocksize = AES_BLOCK_SIZE,
+ .cra_priority = CHCR_AEAD_PRIORITY,
.cra_ctxsize = sizeof(struct chcr_context) +
sizeof(struct chcr_aead_ctx) +
sizeof(struct chcr_authenc_ctx),
@@ -2915,10 +2990,9 @@
name = driver_algs[i].alg.crypto.cra_driver_name;
break;
case CRYPTO_ALG_TYPE_AEAD:
- driver_algs[i].alg.aead.base.cra_priority =
- CHCR_CRA_PRIORITY;
driver_algs[i].alg.aead.base.cra_flags =
- CRYPTO_ALG_TYPE_AEAD | CRYPTO_ALG_ASYNC;
+ CRYPTO_ALG_TYPE_AEAD | CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_NEED_FALLBACK;
driver_algs[i].alg.aead.encrypt = chcr_aead_encrypt;
driver_algs[i].alg.aead.decrypt = chcr_aead_decrypt;
driver_algs[i].alg.aead.init = chcr_aead_cra_init;
diff --git a/drivers/crypto/chelsio/chcr_algo.h b/drivers/crypto/chelsio/chcr_algo.h
index ba38bae..751d06a 100644
--- a/drivers/crypto/chelsio/chcr_algo.h
+++ b/drivers/crypto/chelsio/chcr_algo.h
@@ -218,6 +218,10 @@
#define MAX_NK 8
#define CRYPTO_MAX_IMM_TX_PKT_LEN 256
+#define MAX_WR_SIZE 512
+#define MIN_AUTH_SG 2 /*IV + AAD*/
+#define MIN_GCM_SG 2 /* IV + AAD*/
+#define MIN_CCM_SG 3 /*IV+AAD+B0*/
struct algo_param {
unsigned int auth_mode;
diff --git a/drivers/crypto/chelsio/chcr_core.h b/drivers/crypto/chelsio/chcr_core.h
index 79da22b..cd0c35a1 100644
--- a/drivers/crypto/chelsio/chcr_core.h
+++ b/drivers/crypto/chelsio/chcr_core.h
@@ -54,6 +54,8 @@
#define CHK_MAC_ERR_BIT(x) (((x) >> MAC_ERROR_BIT) & 1)
#define MAX_SALT 4
+#define padap(dev) pci_get_drvdata(dev->u_ctx->lldi.pdev)
+
struct uld_ctx;
struct _key_ctx {
diff --git a/drivers/crypto/chelsio/chcr_crypto.h b/drivers/crypto/chelsio/chcr_crypto.h
index 81cfd0b..5b2fabb 100644
--- a/drivers/crypto/chelsio/chcr_crypto.h
+++ b/drivers/crypto/chelsio/chcr_crypto.h
@@ -41,15 +41,15 @@
#define CCM_B0_SIZE 16
#define CCM_AAD_FIELD_SIZE 2
-#define T5_MAX_AAD_SIZE 512
+#define T6_MAX_AAD_SIZE 511
/* Define following if h/w is not dropping the AAD and IV data before
* giving the processed data
*/
-#define CHCR_CRA_PRIORITY 3000
-
+#define CHCR_CRA_PRIORITY 500
+#define CHCR_AEAD_PRIORITY 6000
#define CHCR_AES_MAX_KEY_LEN (2 * (AES_MAX_KEY_SIZE)) /* consider xts */
#define CHCR_MAX_CRYPTO_IV_LEN 16 /* AES IV len */
@@ -188,6 +188,7 @@
__be32 key_ctx_hdr;
unsigned int enckey_len;
struct crypto_skcipher *null;
+ struct crypto_aead *sw_cipher;
u8 salt[MAX_SALT];
u8 key[CHCR_AES_MAX_KEY_LEN];
u16 hmac_ctrl;
@@ -211,7 +212,8 @@
struct chcr_context {
struct chcr_dev *dev;
- unsigned char tx_channel_id;
+ unsigned char tx_qidx;
+ unsigned char rx_qidx;
struct __crypto_ctx crypto_ctx[0];
};
diff --git a/drivers/crypto/exynos-rng.c b/drivers/crypto/exynos-rng.c
new file mode 100644
index 0000000..451620b
--- /dev/null
+++ b/drivers/crypto/exynos-rng.c
@@ -0,0 +1,389 @@
+/*
+ * exynos-rng.c - Random Number Generator driver for the Exynos
+ *
+ * Copyright (c) 2017 Krzysztof Kozlowski <krzk@kernel.org>
+ *
+ * Loosely based on old driver from drivers/char/hw_random/exynos-rng.c:
+ * Copyright (C) 2012 Samsung Electronics
+ * Jonghwa Lee <jonghwa3.lee@samsung.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation;
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/clk.h>
+#include <linux/crypto.h>
+#include <linux/err.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+
+#include <crypto/internal/rng.h>
+
+#define EXYNOS_RNG_CONTROL 0x0
+#define EXYNOS_RNG_STATUS 0x10
+#define EXYNOS_RNG_SEED_BASE 0x140
+#define EXYNOS_RNG_SEED(n) (EXYNOS_RNG_SEED_BASE + (n * 0x4))
+#define EXYNOS_RNG_OUT_BASE 0x160
+#define EXYNOS_RNG_OUT(n) (EXYNOS_RNG_OUT_BASE + (n * 0x4))
+
+/* EXYNOS_RNG_CONTROL bit fields */
+#define EXYNOS_RNG_CONTROL_START 0x18
+/* EXYNOS_RNG_STATUS bit fields */
+#define EXYNOS_RNG_STATUS_SEED_SETTING_DONE BIT(1)
+#define EXYNOS_RNG_STATUS_RNG_DONE BIT(5)
+
+/* Five seed and output registers, each 4 bytes */
+#define EXYNOS_RNG_SEED_REGS 5
+#define EXYNOS_RNG_SEED_SIZE (EXYNOS_RNG_SEED_REGS * 4)
+
+/*
+ * Driver re-seeds itself with generated random numbers to increase
+ * the randomness.
+ *
+ * Time for next re-seed in ms.
+ */
+#define EXYNOS_RNG_RESEED_TIME 100
+/*
+ * In polling mode, do not wait infinitely for the engine to finish the work.
+ */
+#define EXYNOS_RNG_WAIT_RETRIES 100
+
+/* Context for crypto */
+struct exynos_rng_ctx {
+ struct exynos_rng_dev *rng;
+};
+
+/* Device associated memory */
+struct exynos_rng_dev {
+ struct device *dev;
+ void __iomem *mem;
+ struct clk *clk;
+ /* Generated numbers stored for seeding during resume */
+ u8 seed_save[EXYNOS_RNG_SEED_SIZE];
+ unsigned int seed_save_len;
+ /* Time of last seeding in jiffies */
+ unsigned long last_seeding;
+};
+
+static struct exynos_rng_dev *exynos_rng_dev;
+
+static u32 exynos_rng_readl(struct exynos_rng_dev *rng, u32 offset)
+{
+ return readl_relaxed(rng->mem + offset);
+}
+
+static void exynos_rng_writel(struct exynos_rng_dev *rng, u32 val, u32 offset)
+{
+ writel_relaxed(val, rng->mem + offset);
+}
+
+static int exynos_rng_set_seed(struct exynos_rng_dev *rng,
+ const u8 *seed, unsigned int slen)
+{
+ u32 val;
+ int i;
+
+ /* Round seed length because loop iterates over full register size */
+ slen = ALIGN_DOWN(slen, 4);
+
+ if (slen < EXYNOS_RNG_SEED_SIZE)
+ return -EINVAL;
+
+ for (i = 0; i < slen ; i += 4) {
+ unsigned int seed_reg = (i / 4) % EXYNOS_RNG_SEED_REGS;
+
+ val = seed[i] << 24;
+ val |= seed[i + 1] << 16;
+ val |= seed[i + 2] << 8;
+ val |= seed[i + 3] << 0;
+
+ exynos_rng_writel(rng, val, EXYNOS_RNG_SEED(seed_reg));
+ }
+
+ val = exynos_rng_readl(rng, EXYNOS_RNG_STATUS);
+ if (!(val & EXYNOS_RNG_STATUS_SEED_SETTING_DONE)) {
+ dev_warn(rng->dev, "Seed setting not finished\n");
+ return -EIO;
+ }
+
+ rng->last_seeding = jiffies;
+
+ return 0;
+}
+
+/*
+ * Read from output registers and put the data under 'dst' array,
+ * up to dlen bytes.
+ *
+ * Returns number of bytes actually stored in 'dst' (dlen
+ * or EXYNOS_RNG_SEED_SIZE).
+ */
+static unsigned int exynos_rng_copy_random(struct exynos_rng_dev *rng,
+ u8 *dst, unsigned int dlen)
+{
+ unsigned int cnt = 0;
+ int i, j;
+ u32 val;
+
+ for (j = 0; j < EXYNOS_RNG_SEED_REGS; j++) {
+ val = exynos_rng_readl(rng, EXYNOS_RNG_OUT(j));
+
+ for (i = 0; i < 4; i++) {
+ dst[cnt] = val & 0xff;
+ val >>= 8;
+ if (++cnt >= dlen)
+ return cnt;
+ }
+ }
+
+ return cnt;
+}
+
+/*
+ * Start the engine and poll for finish. Then read from output registers
+ * filling the 'dst' buffer up to 'dlen' bytes or up to size of generated
+ * random data (EXYNOS_RNG_SEED_SIZE).
+ *
+ * On success: return 0 and store number of read bytes under 'read' address.
+ * On error: return -ERRNO.
+ */
+static int exynos_rng_get_random(struct exynos_rng_dev *rng,
+ u8 *dst, unsigned int dlen,
+ unsigned int *read)
+{
+ int retry = EXYNOS_RNG_WAIT_RETRIES;
+
+ exynos_rng_writel(rng, EXYNOS_RNG_CONTROL_START,
+ EXYNOS_RNG_CONTROL);
+
+ while (!(exynos_rng_readl(rng,
+ EXYNOS_RNG_STATUS) & EXYNOS_RNG_STATUS_RNG_DONE) && --retry)
+ cpu_relax();
+
+ if (!retry)
+ return -ETIMEDOUT;
+
+ /* Clear status bit */
+ exynos_rng_writel(rng, EXYNOS_RNG_STATUS_RNG_DONE,
+ EXYNOS_RNG_STATUS);
+ *read = exynos_rng_copy_random(rng, dst, dlen);
+
+ return 0;
+}
+
+/* Re-seed itself from time to time */
+static void exynos_rng_reseed(struct exynos_rng_dev *rng)
+{
+ unsigned long next_seeding = rng->last_seeding + \
+ msecs_to_jiffies(EXYNOS_RNG_RESEED_TIME);
+ unsigned long now = jiffies;
+ unsigned int read = 0;
+ u8 seed[EXYNOS_RNG_SEED_SIZE];
+
+ if (time_before(now, next_seeding))
+ return;
+
+ if (exynos_rng_get_random(rng, seed, sizeof(seed), &read))
+ return;
+
+ exynos_rng_set_seed(rng, seed, read);
+}
+
+static int exynos_rng_generate(struct crypto_rng *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int dlen)
+{
+ struct exynos_rng_ctx *ctx = crypto_rng_ctx(tfm);
+ struct exynos_rng_dev *rng = ctx->rng;
+ unsigned int read = 0;
+ int ret;
+
+ ret = clk_prepare_enable(rng->clk);
+ if (ret)
+ return ret;
+
+ do {
+ ret = exynos_rng_get_random(rng, dst, dlen, &read);
+ if (ret)
+ break;
+
+ dlen -= read;
+ dst += read;
+
+ exynos_rng_reseed(rng);
+ } while (dlen > 0);
+
+ clk_disable_unprepare(rng->clk);
+
+ return ret;
+}
+
+static int exynos_rng_seed(struct crypto_rng *tfm, const u8 *seed,
+ unsigned int slen)
+{
+ struct exynos_rng_ctx *ctx = crypto_rng_ctx(tfm);
+ struct exynos_rng_dev *rng = ctx->rng;
+ int ret;
+
+ ret = clk_prepare_enable(rng->clk);
+ if (ret)
+ return ret;
+
+ ret = exynos_rng_set_seed(ctx->rng, seed, slen);
+
+ clk_disable_unprepare(rng->clk);
+
+ return ret;
+}
+
+static int exynos_rng_kcapi_init(struct crypto_tfm *tfm)
+{
+ struct exynos_rng_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ ctx->rng = exynos_rng_dev;
+
+ return 0;
+}
+
+static struct rng_alg exynos_rng_alg = {
+ .generate = exynos_rng_generate,
+ .seed = exynos_rng_seed,
+ .seedsize = EXYNOS_RNG_SEED_SIZE,
+ .base = {
+ .cra_name = "stdrng",
+ .cra_driver_name = "exynos_rng",
+ .cra_priority = 100,
+ .cra_ctxsize = sizeof(struct exynos_rng_ctx),
+ .cra_module = THIS_MODULE,
+ .cra_init = exynos_rng_kcapi_init,
+ }
+};
+
+static int exynos_rng_probe(struct platform_device *pdev)
+{
+ struct exynos_rng_dev *rng;
+ struct resource *res;
+ int ret;
+
+ if (exynos_rng_dev)
+ return -EEXIST;
+
+ rng = devm_kzalloc(&pdev->dev, sizeof(*rng), GFP_KERNEL);
+ if (!rng)
+ return -ENOMEM;
+
+ rng->dev = &pdev->dev;
+ rng->clk = devm_clk_get(&pdev->dev, "secss");
+ if (IS_ERR(rng->clk)) {
+ dev_err(&pdev->dev, "Couldn't get clock.\n");
+ return PTR_ERR(rng->clk);
+ }
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ rng->mem = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(rng->mem))
+ return PTR_ERR(rng->mem);
+
+ platform_set_drvdata(pdev, rng);
+
+ exynos_rng_dev = rng;
+
+ ret = crypto_register_rng(&exynos_rng_alg);
+ if (ret) {
+ dev_err(&pdev->dev,
+ "Couldn't register rng crypto alg: %d\n", ret);
+ exynos_rng_dev = NULL;
+ }
+
+ return ret;
+}
+
+static int exynos_rng_remove(struct platform_device *pdev)
+{
+ crypto_unregister_rng(&exynos_rng_alg);
+
+ exynos_rng_dev = NULL;
+
+ return 0;
+}
+
+static int __maybe_unused exynos_rng_suspend(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct exynos_rng_dev *rng = platform_get_drvdata(pdev);
+ int ret;
+
+ /* If we were never seeded then after resume it will be the same */
+ if (!rng->last_seeding)
+ return 0;
+
+ rng->seed_save_len = 0;
+ ret = clk_prepare_enable(rng->clk);
+ if (ret)
+ return ret;
+
+ /* Get new random numbers and store them for seeding on resume. */
+ exynos_rng_get_random(rng, rng->seed_save, sizeof(rng->seed_save),
+ &(rng->seed_save_len));
+ dev_dbg(rng->dev, "Stored %u bytes for seeding on system resume\n",
+ rng->seed_save_len);
+
+ clk_disable_unprepare(rng->clk);
+
+ return 0;
+}
+
+static int __maybe_unused exynos_rng_resume(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct exynos_rng_dev *rng = platform_get_drvdata(pdev);
+ int ret;
+
+ /* Never seeded so nothing to do */
+ if (!rng->last_seeding)
+ return 0;
+
+ ret = clk_prepare_enable(rng->clk);
+ if (ret)
+ return ret;
+
+ ret = exynos_rng_set_seed(rng, rng->seed_save, rng->seed_save_len);
+
+ clk_disable_unprepare(rng->clk);
+
+ return ret;
+}
+
+static SIMPLE_DEV_PM_OPS(exynos_rng_pm_ops, exynos_rng_suspend,
+ exynos_rng_resume);
+
+static const struct of_device_id exynos_rng_dt_match[] = {
+ {
+ .compatible = "samsung,exynos4-rng",
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, exynos_rng_dt_match);
+
+static struct platform_driver exynos_rng_driver = {
+ .driver = {
+ .name = "exynos-rng",
+ .pm = &exynos_rng_pm_ops,
+ .of_match_table = exynos_rng_dt_match,
+ },
+ .probe = exynos_rng_probe,
+ .remove = exynos_rng_remove,
+};
+
+module_platform_driver(exynos_rng_driver);
+
+MODULE_DESCRIPTION("Exynos H/W Random Number Generator driver");
+MODULE_AUTHOR("Krzysztof Kozlowski <krzk@kernel.org>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/crypto/ixp4xx_crypto.c b/drivers/crypto/ixp4xx_crypto.c
index 7868765..771dd26 100644
--- a/drivers/crypto/ixp4xx_crypto.c
+++ b/drivers/crypto/ixp4xx_crypto.c
@@ -806,7 +806,7 @@
void *ptr;
nbytes -= len;
- ptr = page_address(sg_page(sg)) + sg->offset;
+ ptr = sg_virt(sg);
next_buf = dma_pool_alloc(buffer_pool, flags, &next_buf_phys);
if (!next_buf) {
buf = NULL;
diff --git a/drivers/crypto/mediatek/mtk-aes.c b/drivers/crypto/mediatek/mtk-aes.c
index 3a47cdb..9e845e8 100644
--- a/drivers/crypto/mediatek/mtk-aes.c
+++ b/drivers/crypto/mediatek/mtk-aes.c
@@ -19,13 +19,10 @@
#define AES_BUF_ORDER 2
#define AES_BUF_SIZE ((PAGE_SIZE << AES_BUF_ORDER) \
& ~(AES_BLOCK_SIZE - 1))
+#define AES_MAX_STATE_BUF_SIZE SIZE_IN_WORDS(AES_KEYSIZE_256 + \
+ AES_BLOCK_SIZE * 2)
+#define AES_MAX_CT_SIZE 6
-/* AES command token size */
-#define AES_CT_SIZE_ECB 2
-#define AES_CT_SIZE_CBC 3
-#define AES_CT_SIZE_CTR 3
-#define AES_CT_SIZE_GCM_OUT 5
-#define AES_CT_SIZE_GCM_IN 6
#define AES_CT_CTRL_HDR cpu_to_le32(0x00220000)
/* AES-CBC/ECB/CTR command token */
@@ -50,6 +47,8 @@
#define AES_TFM_128BITS cpu_to_le32(0xb << 16)
#define AES_TFM_192BITS cpu_to_le32(0xd << 16)
#define AES_TFM_256BITS cpu_to_le32(0xf << 16)
+#define AES_TFM_GHASH_DIGEST cpu_to_le32(0x2 << 21)
+#define AES_TFM_GHASH cpu_to_le32(0x4 << 23)
/* AES transform information word 1 fields */
#define AES_TFM_ECB cpu_to_le32(0x0 << 0)
#define AES_TFM_CBC cpu_to_le32(0x1 << 0)
@@ -59,10 +58,9 @@
#define AES_TFM_FULL_IV cpu_to_le32(0xf << 5) /* using IV 0-3 */
#define AES_TFM_IV_CTR_MODE cpu_to_le32(0x1 << 10)
#define AES_TFM_ENC_HASH cpu_to_le32(0x1 << 17)
-#define AES_TFM_GHASH_DIG cpu_to_le32(0x2 << 21)
-#define AES_TFM_GHASH cpu_to_le32(0x4 << 23)
/* AES flags */
+#define AES_FLAGS_CIPHER_MSK GENMASK(2, 0)
#define AES_FLAGS_ECB BIT(0)
#define AES_FLAGS_CBC BIT(1)
#define AES_FLAGS_CTR BIT(2)
@@ -70,19 +68,15 @@
#define AES_FLAGS_ENCRYPT BIT(4)
#define AES_FLAGS_BUSY BIT(5)
+#define AES_AUTH_TAG_ERR cpu_to_le32(BIT(26))
+
/**
- * Command token(CT) is a set of hardware instructions that
- * are used to control engine's processing flow of AES.
+ * mtk_aes_info - hardware information of AES
+ * @cmd: command token, hardware instruction
+ * @tfm: transform state of cipher algorithm.
+ * @state: contains keys and initial vectors.
*
- * Transform information(TFM) is used to define AES state and
- * contains all keys and initial vectors.
- *
- * The engine requires CT and TFM to do:
- * - Commands decoding and control of the engine's data path.
- * - Coordinating hardware data fetch and store operations.
- * - Result token construction and output.
- *
- * Memory map of GCM's TFM:
+ * Memory layout of GCM buffer:
* /-----------\
* | AES KEY | 128/196/256 bits
* |-----------|
@@ -90,14 +84,16 @@
* |-----------|
* | IVs | 4 * 4 bytes
* \-----------/
+ *
+ * The engine requires all these info to do:
+ * - Commands decoding and control of the engine's data path.
+ * - Coordinating hardware data fetch and store operations.
+ * - Result token construction and output.
*/
-struct mtk_aes_ct {
- __le32 cmd[AES_CT_SIZE_GCM_IN];
-};
-
-struct mtk_aes_tfm {
- __le32 ctrl[2];
- __le32 state[SIZE_IN_WORDS(AES_KEYSIZE_256 + AES_BLOCK_SIZE * 2)];
+struct mtk_aes_info {
+ __le32 cmd[AES_MAX_CT_SIZE];
+ __le32 tfm[2];
+ __le32 state[AES_MAX_STATE_BUF_SIZE];
};
struct mtk_aes_reqctx {
@@ -107,11 +103,12 @@
struct mtk_aes_base_ctx {
struct mtk_cryp *cryp;
u32 keylen;
+ __le32 keymode;
+
mtk_aes_fn start;
- struct mtk_aes_ct ct;
+ struct mtk_aes_info info;
dma_addr_t ct_dma;
- struct mtk_aes_tfm tfm;
dma_addr_t tfm_dma;
__le32 ct_hdr;
@@ -248,6 +245,33 @@
sg->length += dma->remainder;
}
+static inline void mtk_aes_write_state_le(__le32 *dst, const u32 *src, u32 size)
+{
+ int i;
+
+ for (i = 0; i < SIZE_IN_WORDS(size); i++)
+ dst[i] = cpu_to_le32(src[i]);
+}
+
+static inline void mtk_aes_write_state_be(__be32 *dst, const u32 *src, u32 size)
+{
+ int i;
+
+ for (i = 0; i < SIZE_IN_WORDS(size); i++)
+ dst[i] = cpu_to_be32(src[i]);
+}
+
+static inline int mtk_aes_complete(struct mtk_cryp *cryp,
+ struct mtk_aes_rec *aes,
+ int err)
+{
+ aes->flags &= ~AES_FLAGS_BUSY;
+ aes->areq->complete(aes->areq, err);
+ /* Handle new request */
+ tasklet_schedule(&aes->queue_task);
+ return err;
+}
+
/*
* Write descriptors for processing. This will configure the engine, load
* the transform information and then start the packet processing.
@@ -262,7 +286,7 @@
/* Write command descriptors */
for (nents = 0; nents < slen; ++nents, ssg = sg_next(ssg)) {
- cmd = ring->cmd_base + ring->cmd_pos;
+ cmd = ring->cmd_next;
cmd->hdr = MTK_DESC_BUF_LEN(ssg->length);
cmd->buf = cpu_to_le32(sg_dma_address(ssg));
@@ -274,25 +298,30 @@
cmd->tfm = cpu_to_le32(aes->ctx->tfm_dma);
}
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
+ /* Shift ring buffer and check boundary */
+ if (++ring->cmd_next == ring->cmd_base + MTK_DESC_NUM)
+ ring->cmd_next = ring->cmd_base;
}
cmd->hdr |= MTK_DESC_LAST;
/* Prepare result descriptors */
for (nents = 0; nents < dlen; ++nents, dsg = sg_next(dsg)) {
- res = ring->res_base + ring->res_pos;
+ res = ring->res_next;
res->hdr = MTK_DESC_BUF_LEN(dsg->length);
res->buf = cpu_to_le32(sg_dma_address(dsg));
if (nents == 0)
res->hdr |= MTK_DESC_FIRST;
- if (++ring->res_pos == MTK_DESC_NUM)
- ring->res_pos = 0;
+ /* Shift ring buffer and check boundary */
+ if (++ring->res_next == ring->res_base + MTK_DESC_NUM)
+ ring->res_next = ring->res_base;
}
res->hdr |= MTK_DESC_LAST;
+ /* Pointer to current result descriptor */
+ ring->res_prev = res;
+
/* Prepare enough space for authenticated tag */
if (aes->flags & AES_FLAGS_GCM)
res->hdr += AES_BLOCK_SIZE;
@@ -313,9 +342,7 @@
{
struct mtk_aes_base_ctx *ctx = aes->ctx;
- dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(ctx->ct),
- DMA_TO_DEVICE);
- dma_unmap_single(cryp->dev, ctx->tfm_dma, sizeof(ctx->tfm),
+ dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(ctx->info),
DMA_TO_DEVICE);
if (aes->src.sg == aes->dst.sg) {
@@ -346,16 +373,14 @@
static int mtk_aes_map(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
{
struct mtk_aes_base_ctx *ctx = aes->ctx;
+ struct mtk_aes_info *info = &ctx->info;
- ctx->ct_dma = dma_map_single(cryp->dev, &ctx->ct, sizeof(ctx->ct),
+ ctx->ct_dma = dma_map_single(cryp->dev, info, sizeof(*info),
DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(cryp->dev, ctx->ct_dma)))
- return -EINVAL;
+ goto exit;
- ctx->tfm_dma = dma_map_single(cryp->dev, &ctx->tfm, sizeof(ctx->tfm),
- DMA_TO_DEVICE);
- if (unlikely(dma_mapping_error(cryp->dev, ctx->tfm_dma)))
- goto tfm_map_err;
+ ctx->tfm_dma = ctx->ct_dma + sizeof(info->cmd);
if (aes->src.sg == aes->dst.sg) {
aes->src.sg_len = dma_map_sg(cryp->dev, aes->src.sg,
@@ -382,13 +407,9 @@
return mtk_aes_xmit(cryp, aes);
sg_map_err:
- dma_unmap_single(cryp->dev, ctx->tfm_dma, sizeof(ctx->tfm),
- DMA_TO_DEVICE);
-tfm_map_err:
- dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(ctx->ct),
- DMA_TO_DEVICE);
-
- return -EINVAL;
+ dma_unmap_single(cryp->dev, ctx->ct_dma, sizeof(*info), DMA_TO_DEVICE);
+exit:
+ return mtk_aes_complete(cryp, aes, -EINVAL);
}
/* Initialize transform information of CBC/ECB/CTR mode */
@@ -397,50 +418,43 @@
{
struct ablkcipher_request *req = ablkcipher_request_cast(aes->areq);
struct mtk_aes_base_ctx *ctx = aes->ctx;
+ struct mtk_aes_info *info = &ctx->info;
+ u32 cnt = 0;
ctx->ct_hdr = AES_CT_CTRL_HDR | cpu_to_le32(len);
- ctx->ct.cmd[0] = AES_CMD0 | cpu_to_le32(len);
- ctx->ct.cmd[1] = AES_CMD1;
+ info->cmd[cnt++] = AES_CMD0 | cpu_to_le32(len);
+ info->cmd[cnt++] = AES_CMD1;
+ info->tfm[0] = AES_TFM_SIZE(ctx->keylen) | ctx->keymode;
if (aes->flags & AES_FLAGS_ENCRYPT)
- ctx->tfm.ctrl[0] = AES_TFM_BASIC_OUT;
+ info->tfm[0] |= AES_TFM_BASIC_OUT;
else
- ctx->tfm.ctrl[0] = AES_TFM_BASIC_IN;
+ info->tfm[0] |= AES_TFM_BASIC_IN;
- if (ctx->keylen == SIZE_IN_WORDS(AES_KEYSIZE_128))
- ctx->tfm.ctrl[0] |= AES_TFM_128BITS;
- else if (ctx->keylen == SIZE_IN_WORDS(AES_KEYSIZE_256))
- ctx->tfm.ctrl[0] |= AES_TFM_256BITS;
- else
- ctx->tfm.ctrl[0] |= AES_TFM_192BITS;
+ switch (aes->flags & AES_FLAGS_CIPHER_MSK) {
+ case AES_FLAGS_CBC:
+ info->tfm[1] = AES_TFM_CBC;
+ break;
+ case AES_FLAGS_ECB:
+ info->tfm[1] = AES_TFM_ECB;
+ goto ecb;
+ case AES_FLAGS_CTR:
+ info->tfm[1] = AES_TFM_CTR_LOAD;
+ goto ctr;
- if (aes->flags & AES_FLAGS_CBC) {
- const u32 *iv = (const u32 *)req->info;
- u32 *iv_state = ctx->tfm.state + ctx->keylen;
- int i;
-
- ctx->tfm.ctrl[0] |= AES_TFM_SIZE(ctx->keylen +
- SIZE_IN_WORDS(AES_BLOCK_SIZE));
- ctx->tfm.ctrl[1] = AES_TFM_CBC | AES_TFM_FULL_IV;
-
- for (i = 0; i < SIZE_IN_WORDS(AES_BLOCK_SIZE); i++)
- iv_state[i] = cpu_to_le32(iv[i]);
-
- ctx->ct.cmd[2] = AES_CMD2;
- ctx->ct_size = AES_CT_SIZE_CBC;
- } else if (aes->flags & AES_FLAGS_ECB) {
- ctx->tfm.ctrl[0] |= AES_TFM_SIZE(ctx->keylen);
- ctx->tfm.ctrl[1] = AES_TFM_ECB;
-
- ctx->ct_size = AES_CT_SIZE_ECB;
- } else if (aes->flags & AES_FLAGS_CTR) {
- ctx->tfm.ctrl[0] |= AES_TFM_SIZE(ctx->keylen +
- SIZE_IN_WORDS(AES_BLOCK_SIZE));
- ctx->tfm.ctrl[1] = AES_TFM_CTR_LOAD | AES_TFM_FULL_IV;
-
- ctx->ct.cmd[2] = AES_CMD2;
- ctx->ct_size = AES_CT_SIZE_CTR;
+ default:
+ /* Should not happen... */
+ return;
}
+
+ mtk_aes_write_state_le(info->state + ctx->keylen, req->info,
+ AES_BLOCK_SIZE);
+ctr:
+ info->tfm[0] += AES_TFM_SIZE(SIZE_IN_WORDS(AES_BLOCK_SIZE));
+ info->tfm[1] |= AES_TFM_FULL_IV;
+ info->cmd[cnt++] = AES_CMD2;
+ecb:
+ ctx->ct_size = cnt;
}
static int mtk_aes_dma(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
@@ -465,7 +479,7 @@
padlen = mtk_aes_padlen(len);
if (len + padlen > AES_BUF_SIZE)
- return -ENOMEM;
+ return mtk_aes_complete(cryp, aes, -ENOMEM);
if (!src_aligned) {
sg_copy_to_buffer(src, sg_nents(src), aes->buf, len);
@@ -525,13 +539,10 @@
return ctx->start(cryp, aes);
}
-static int mtk_aes_complete(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
+static int mtk_aes_transfer_complete(struct mtk_cryp *cryp,
+ struct mtk_aes_rec *aes)
{
- aes->flags &= ~AES_FLAGS_BUSY;
- aes->areq->complete(aes->areq, 0);
-
- /* Handle new request */
- return mtk_aes_handle_queue(cryp, aes->id, NULL);
+ return mtk_aes_complete(cryp, aes, 0);
}
static int mtk_aes_start(struct mtk_cryp *cryp, struct mtk_aes_rec *aes)
@@ -540,7 +551,7 @@
struct mtk_aes_reqctx *rctx = ablkcipher_request_ctx(req);
mtk_aes_set_mode(aes, rctx);
- aes->resume = mtk_aes_complete;
+ aes->resume = mtk_aes_transfer_complete;
return mtk_aes_dma(cryp, aes, req->src, req->dst, req->nbytes);
}
@@ -557,15 +568,14 @@
struct mtk_aes_ctr_ctx *cctx = mtk_aes_ctr_ctx_cast(ctx);
struct ablkcipher_request *req = ablkcipher_request_cast(aes->areq);
struct scatterlist *src, *dst;
- int i;
- u32 start, end, ctr, blocks, *iv_state;
+ u32 start, end, ctr, blocks;
size_t datalen;
bool fragmented = false;
/* Check for transfer completion. */
cctx->offset += aes->total;
if (cctx->offset >= req->nbytes)
- return mtk_aes_complete(cryp, aes);
+ return mtk_aes_transfer_complete(cryp, aes);
/* Compute data length. */
datalen = req->nbytes - cctx->offset;
@@ -587,9 +597,8 @@
scatterwalk_ffwd(cctx->dst, req->dst, cctx->offset));
/* Write IVs into transform state buffer. */
- iv_state = ctx->tfm.state + ctx->keylen;
- for (i = 0; i < SIZE_IN_WORDS(AES_BLOCK_SIZE); i++)
- iv_state[i] = cpu_to_le32(cctx->iv[i]);
+ mtk_aes_write_state_le(ctx->info.state + ctx->keylen, cctx->iv,
+ AES_BLOCK_SIZE);
if (unlikely(fragmented)) {
/*
@@ -599,7 +608,6 @@
cctx->iv[3] = cpu_to_be32(ctr);
crypto_inc((u8 *)cctx->iv, AES_BLOCK_SIZE);
}
- aes->resume = mtk_aes_ctr_transfer;
return mtk_aes_dma(cryp, aes, src, dst, datalen);
}
@@ -615,6 +623,7 @@
memcpy(cctx->iv, req->info, AES_BLOCK_SIZE);
cctx->offset = 0;
aes->total = 0;
+ aes->resume = mtk_aes_ctr_transfer;
return mtk_aes_ctr_transfer(cryp, aes);
}
@@ -624,21 +633,25 @@
const u8 *key, u32 keylen)
{
struct mtk_aes_base_ctx *ctx = crypto_ablkcipher_ctx(tfm);
- const u32 *aes_key = (const u32 *)key;
- u32 *key_state = ctx->tfm.state;
- int i;
- if (keylen != AES_KEYSIZE_128 &&
- keylen != AES_KEYSIZE_192 &&
- keylen != AES_KEYSIZE_256) {
+ switch (keylen) {
+ case AES_KEYSIZE_128:
+ ctx->keymode = AES_TFM_128BITS;
+ break;
+ case AES_KEYSIZE_192:
+ ctx->keymode = AES_TFM_192BITS;
+ break;
+ case AES_KEYSIZE_256:
+ ctx->keymode = AES_TFM_256BITS;
+ break;
+
+ default:
crypto_ablkcipher_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
}
ctx->keylen = SIZE_IN_WORDS(keylen);
-
- for (i = 0; i < ctx->keylen; i++)
- key_state[i] = cpu_to_le32(aes_key[i]);
+ mtk_aes_write_state_le(ctx->info.state, (const u32 *)key, keylen);
return 0;
}
@@ -789,6 +802,19 @@
return container_of(ctx, struct mtk_aes_gcm_ctx, base);
}
+/*
+ * Engine will verify and compare tag automatically, so we just need
+ * to check returned status which stored in the result descriptor.
+ */
+static int mtk_aes_gcm_tag_verify(struct mtk_cryp *cryp,
+ struct mtk_aes_rec *aes)
+{
+ u32 status = cryp->ring[aes->id]->res_prev->ct;
+
+ return mtk_aes_complete(cryp, aes, (status & AES_AUTH_TAG_ERR) ?
+ -EBADMSG : 0);
+}
+
/* Initialize transform information of GCM mode */
static void mtk_aes_gcm_info_init(struct mtk_cryp *cryp,
struct mtk_aes_rec *aes,
@@ -797,45 +823,35 @@
struct aead_request *req = aead_request_cast(aes->areq);
struct mtk_aes_base_ctx *ctx = aes->ctx;
struct mtk_aes_gcm_ctx *gctx = mtk_aes_gcm_ctx_cast(ctx);
- const u32 *iv = (const u32 *)req->iv;
- u32 *iv_state = ctx->tfm.state + ctx->keylen +
- SIZE_IN_WORDS(AES_BLOCK_SIZE);
+ struct mtk_aes_info *info = &ctx->info;
u32 ivsize = crypto_aead_ivsize(crypto_aead_reqtfm(req));
- int i;
+ u32 cnt = 0;
ctx->ct_hdr = AES_CT_CTRL_HDR | len;
- ctx->ct.cmd[0] = AES_GCM_CMD0 | cpu_to_le32(req->assoclen);
- ctx->ct.cmd[1] = AES_GCM_CMD1 | cpu_to_le32(req->assoclen);
- ctx->ct.cmd[2] = AES_GCM_CMD2;
- ctx->ct.cmd[3] = AES_GCM_CMD3 | cpu_to_le32(gctx->textlen);
+ info->cmd[cnt++] = AES_GCM_CMD0 | cpu_to_le32(req->assoclen);
+ info->cmd[cnt++] = AES_GCM_CMD1 | cpu_to_le32(req->assoclen);
+ info->cmd[cnt++] = AES_GCM_CMD2;
+ info->cmd[cnt++] = AES_GCM_CMD3 | cpu_to_le32(gctx->textlen);
if (aes->flags & AES_FLAGS_ENCRYPT) {
- ctx->ct.cmd[4] = AES_GCM_CMD4 | cpu_to_le32(gctx->authsize);
- ctx->ct_size = AES_CT_SIZE_GCM_OUT;
- ctx->tfm.ctrl[0] = AES_TFM_GCM_OUT;
+ info->cmd[cnt++] = AES_GCM_CMD4 | cpu_to_le32(gctx->authsize);
+ info->tfm[0] = AES_TFM_GCM_OUT;
} else {
- ctx->ct.cmd[4] = AES_GCM_CMD5 | cpu_to_le32(gctx->authsize);
- ctx->ct.cmd[5] = AES_GCM_CMD6 | cpu_to_le32(gctx->authsize);
- ctx->ct_size = AES_CT_SIZE_GCM_IN;
- ctx->tfm.ctrl[0] = AES_TFM_GCM_IN;
+ info->cmd[cnt++] = AES_GCM_CMD5 | cpu_to_le32(gctx->authsize);
+ info->cmd[cnt++] = AES_GCM_CMD6 | cpu_to_le32(gctx->authsize);
+ info->tfm[0] = AES_TFM_GCM_IN;
}
+ ctx->ct_size = cnt;
- if (ctx->keylen == SIZE_IN_WORDS(AES_KEYSIZE_128))
- ctx->tfm.ctrl[0] |= AES_TFM_128BITS;
- else if (ctx->keylen == SIZE_IN_WORDS(AES_KEYSIZE_256))
- ctx->tfm.ctrl[0] |= AES_TFM_256BITS;
- else
- ctx->tfm.ctrl[0] |= AES_TFM_192BITS;
+ info->tfm[0] |= AES_TFM_GHASH_DIGEST | AES_TFM_GHASH | AES_TFM_SIZE(
+ ctx->keylen + SIZE_IN_WORDS(AES_BLOCK_SIZE + ivsize)) |
+ ctx->keymode;
+ info->tfm[1] = AES_TFM_CTR_INIT | AES_TFM_IV_CTR_MODE | AES_TFM_3IV |
+ AES_TFM_ENC_HASH;
- ctx->tfm.ctrl[0] |= AES_TFM_GHASH_DIG | AES_TFM_GHASH |
- AES_TFM_SIZE(ctx->keylen + SIZE_IN_WORDS(
- AES_BLOCK_SIZE + ivsize));
- ctx->tfm.ctrl[1] = AES_TFM_CTR_INIT | AES_TFM_IV_CTR_MODE |
- AES_TFM_3IV | AES_TFM_ENC_HASH;
-
- for (i = 0; i < SIZE_IN_WORDS(ivsize); i++)
- iv_state[i] = cpu_to_le32(iv[i]);
+ mtk_aes_write_state_le(info->state + ctx->keylen + SIZE_IN_WORDS(
+ AES_BLOCK_SIZE), (const u32 *)req->iv, ivsize);
}
static int mtk_aes_gcm_dma(struct mtk_cryp *cryp, struct mtk_aes_rec *aes,
@@ -856,7 +872,7 @@
if (!src_aligned || !dst_aligned) {
if (aes->total > AES_BUF_SIZE)
- return -ENOMEM;
+ return mtk_aes_complete(cryp, aes, -ENOMEM);
if (!src_aligned) {
sg_copy_to_buffer(src, sg_nents(src), aes->buf, len);
@@ -892,6 +908,8 @@
if (aes->flags & AES_FLAGS_ENCRYPT) {
u32 tag[4];
+
+ aes->resume = mtk_aes_transfer_complete;
/* Compute total process length. */
aes->total = len + gctx->authsize;
/* Compute text length. */
@@ -899,10 +917,10 @@
/* Hardware will append authenticated tag to output buffer */
scatterwalk_map_and_copy(tag, req->dst, len, gctx->authsize, 1);
} else {
+ aes->resume = mtk_aes_gcm_tag_verify;
aes->total = len;
gctx->textlen = req->cryptlen - gctx->authsize;
}
- aes->resume = mtk_aes_complete;
return mtk_aes_gcm_dma(cryp, aes, req->src, req->dst, len);
}
@@ -915,7 +933,7 @@
rctx->mode = AES_FLAGS_GCM | mode;
return mtk_aes_handle_queue(ctx->cryp, !!(mode & AES_FLAGS_ENCRYPT),
- &req->base);
+ &req->base);
}
static void mtk_gcm_setkey_done(struct crypto_async_request *req, int err)
@@ -949,24 +967,26 @@
struct scatterlist sg[1];
struct skcipher_request req;
} *data;
- const u32 *aes_key;
- u32 *key_state, *hash_state;
- int err, i;
+ int err;
- if (keylen != AES_KEYSIZE_256 &&
- keylen != AES_KEYSIZE_192 &&
- keylen != AES_KEYSIZE_128) {
+ switch (keylen) {
+ case AES_KEYSIZE_128:
+ ctx->keymode = AES_TFM_128BITS;
+ break;
+ case AES_KEYSIZE_192:
+ ctx->keymode = AES_TFM_192BITS;
+ break;
+ case AES_KEYSIZE_256:
+ ctx->keymode = AES_TFM_256BITS;
+ break;
+
+ default:
crypto_aead_set_flags(aead, CRYPTO_TFM_RES_BAD_KEY_LEN);
return -EINVAL;
}
- key_state = ctx->tfm.state;
- aes_key = (u32 *)key;
ctx->keylen = SIZE_IN_WORDS(keylen);
- for (i = 0; i < ctx->keylen; i++)
- ctx->tfm.state[i] = cpu_to_le32(aes_key[i]);
-
/* Same as crypto_gcm_setkey() from crypto/gcm.c */
crypto_skcipher_clear_flags(ctr, CRYPTO_TFM_REQ_MASK);
crypto_skcipher_set_flags(ctr, crypto_aead_get_flags(aead) &
@@ -1001,10 +1021,11 @@
if (err)
goto out;
- hash_state = key_state + ctx->keylen;
-
- for (i = 0; i < 4; i++)
- hash_state[i] = cpu_to_be32(data->hash[i]);
+ /* Write key into state buffer */
+ mtk_aes_write_state_le(ctx->info.state, (const u32 *)key, keylen);
+ /* Write key(H) into state buffer */
+ mtk_aes_write_state_be(ctx->info.state + ctx->keylen, data->hash,
+ AES_BLOCK_SIZE);
out:
kzfree(data);
return err;
@@ -1092,58 +1113,36 @@
},
};
-static void mtk_aes_enc_task(unsigned long data)
+static void mtk_aes_queue_task(unsigned long data)
{
- struct mtk_cryp *cryp = (struct mtk_cryp *)data;
- struct mtk_aes_rec *aes = cryp->aes[0];
+ struct mtk_aes_rec *aes = (struct mtk_aes_rec *)data;
+
+ mtk_aes_handle_queue(aes->cryp, aes->id, NULL);
+}
+
+static void mtk_aes_done_task(unsigned long data)
+{
+ struct mtk_aes_rec *aes = (struct mtk_aes_rec *)data;
+ struct mtk_cryp *cryp = aes->cryp;
mtk_aes_unmap(cryp, aes);
aes->resume(cryp, aes);
}
-static void mtk_aes_dec_task(unsigned long data)
+static irqreturn_t mtk_aes_irq(int irq, void *dev_id)
{
- struct mtk_cryp *cryp = (struct mtk_cryp *)data;
- struct mtk_aes_rec *aes = cryp->aes[1];
+ struct mtk_aes_rec *aes = (struct mtk_aes_rec *)dev_id;
+ struct mtk_cryp *cryp = aes->cryp;
+ u32 val = mtk_aes_read(cryp, RDR_STAT(aes->id));
- mtk_aes_unmap(cryp, aes);
- aes->resume(cryp, aes);
-}
-
-static irqreturn_t mtk_aes_enc_irq(int irq, void *dev_id)
-{
- struct mtk_cryp *cryp = (struct mtk_cryp *)dev_id;
- struct mtk_aes_rec *aes = cryp->aes[0];
- u32 val = mtk_aes_read(cryp, RDR_STAT(RING0));
-
- mtk_aes_write(cryp, RDR_STAT(RING0), val);
+ mtk_aes_write(cryp, RDR_STAT(aes->id), val);
if (likely(AES_FLAGS_BUSY & aes->flags)) {
- mtk_aes_write(cryp, RDR_PROC_COUNT(RING0), MTK_CNT_RST);
- mtk_aes_write(cryp, RDR_THRESH(RING0),
+ mtk_aes_write(cryp, RDR_PROC_COUNT(aes->id), MTK_CNT_RST);
+ mtk_aes_write(cryp, RDR_THRESH(aes->id),
MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
- tasklet_schedule(&aes->task);
- } else {
- dev_warn(cryp->dev, "AES interrupt when no active requests.\n");
- }
- return IRQ_HANDLED;
-}
-
-static irqreturn_t mtk_aes_dec_irq(int irq, void *dev_id)
-{
- struct mtk_cryp *cryp = (struct mtk_cryp *)dev_id;
- struct mtk_aes_rec *aes = cryp->aes[1];
- u32 val = mtk_aes_read(cryp, RDR_STAT(RING1));
-
- mtk_aes_write(cryp, RDR_STAT(RING1), val);
-
- if (likely(AES_FLAGS_BUSY & aes->flags)) {
- mtk_aes_write(cryp, RDR_PROC_COUNT(RING1), MTK_CNT_RST);
- mtk_aes_write(cryp, RDR_THRESH(RING1),
- MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
-
- tasklet_schedule(&aes->task);
+ tasklet_schedule(&aes->done_task);
} else {
dev_warn(cryp->dev, "AES interrupt when no active requests.\n");
}
@@ -1171,14 +1170,20 @@
if (!aes[i]->buf)
goto err_cleanup;
- aes[i]->id = i;
+ aes[i]->cryp = cryp;
spin_lock_init(&aes[i]->lock);
crypto_init_queue(&aes[i]->queue, AES_QUEUE_SIZE);
+
+ tasklet_init(&aes[i]->queue_task, mtk_aes_queue_task,
+ (unsigned long)aes[i]);
+ tasklet_init(&aes[i]->done_task, mtk_aes_done_task,
+ (unsigned long)aes[i]);
}
- tasklet_init(&aes[0]->task, mtk_aes_enc_task, (unsigned long)cryp);
- tasklet_init(&aes[1]->task, mtk_aes_dec_task, (unsigned long)cryp);
+ /* Link to ring0 and ring1 respectively */
+ aes[0]->id = MTK_RING0;
+ aes[1]->id = MTK_RING1;
return 0;
@@ -1196,7 +1201,9 @@
int i;
for (i = 0; i < MTK_REC_NUM; i++) {
- tasklet_kill(&cryp->aes[i]->task);
+ tasklet_kill(&cryp->aes[i]->done_task);
+ tasklet_kill(&cryp->aes[i]->queue_task);
+
free_page((unsigned long)cryp->aes[i]->buf);
kfree(cryp->aes[i]);
}
@@ -1246,25 +1253,23 @@
if (ret)
goto err_record;
- /* Ring0 is use by encryption record */
- ret = devm_request_irq(cryp->dev, cryp->irq[RING0], mtk_aes_enc_irq,
- IRQF_TRIGGER_LOW, "mtk-aes", cryp);
+ ret = devm_request_irq(cryp->dev, cryp->irq[MTK_RING0], mtk_aes_irq,
+ 0, "mtk-aes", cryp->aes[0]);
if (ret) {
- dev_err(cryp->dev, "unable to request AES encryption irq.\n");
+ dev_err(cryp->dev, "unable to request AES irq.\n");
goto err_res;
}
- /* Ring1 is use by decryption record */
- ret = devm_request_irq(cryp->dev, cryp->irq[RING1], mtk_aes_dec_irq,
- IRQF_TRIGGER_LOW, "mtk-aes", cryp);
+ ret = devm_request_irq(cryp->dev, cryp->irq[MTK_RING1], mtk_aes_irq,
+ 0, "mtk-aes", cryp->aes[1]);
if (ret) {
- dev_err(cryp->dev, "unable to request AES decryption irq.\n");
+ dev_err(cryp->dev, "unable to request AES irq.\n");
goto err_res;
}
/* Enable ring0 and ring1 interrupt */
- mtk_aes_write(cryp, AIC_ENABLE_SET(RING0), MTK_IRQ_RDR0);
- mtk_aes_write(cryp, AIC_ENABLE_SET(RING1), MTK_IRQ_RDR1);
+ mtk_aes_write(cryp, AIC_ENABLE_SET(MTK_RING0), MTK_IRQ_RDR0);
+ mtk_aes_write(cryp, AIC_ENABLE_SET(MTK_RING1), MTK_IRQ_RDR1);
spin_lock(&mtk_aes.lock);
list_add_tail(&cryp->aes_list, &mtk_aes.dev_list);
diff --git a/drivers/crypto/mediatek/mtk-platform.c b/drivers/crypto/mediatek/mtk-platform.c
index a9c713d..b6ecc28 100644
--- a/drivers/crypto/mediatek/mtk-platform.c
+++ b/drivers/crypto/mediatek/mtk-platform.c
@@ -334,7 +334,7 @@
/* Enable the 4 rings for the packet engines. */
mtk_desc_ring_link(cryp, 0xf);
- for (i = 0; i < RING_MAX; i++) {
+ for (i = 0; i < MTK_RING_MAX; i++) {
mtk_cmd_desc_ring_setup(cryp, i, &cap);
mtk_res_desc_ring_setup(cryp, i, &cap);
}
@@ -359,7 +359,7 @@
{
u32 val;
- if (hw == RING_MAX)
+ if (hw == MTK_RING_MAX)
val = readl(cryp->base + AIC_G_VERSION);
else
val = readl(cryp->base + AIC_VERSION(hw));
@@ -368,7 +368,7 @@
if (val != MTK_AIC_VER11 && val != MTK_AIC_VER12)
return -ENXIO;
- if (hw == RING_MAX)
+ if (hw == MTK_RING_MAX)
val = readl(cryp->base + AIC_G_OPTIONS);
else
val = readl(cryp->base + AIC_OPTIONS(hw));
@@ -389,7 +389,7 @@
return err;
/* Disable all interrupts and set initial configuration */
- if (hw == RING_MAX) {
+ if (hw == MTK_RING_MAX) {
writel(0, cryp->base + AIC_G_ENABLE_CTRL);
writel(0, cryp->base + AIC_G_POL_CTRL);
writel(0, cryp->base + AIC_G_TYPE_CTRL);
@@ -431,7 +431,7 @@
{
int i;
- for (i = 0; i < RING_MAX; i++) {
+ for (i = 0; i < MTK_RING_MAX; i++) {
dma_free_coherent(cryp->dev, MTK_DESC_RING_SZ,
cryp->ring[i]->res_base,
cryp->ring[i]->res_dma);
@@ -447,7 +447,7 @@
struct mtk_ring **ring = cryp->ring;
int i, err = ENOMEM;
- for (i = 0; i < RING_MAX; i++) {
+ for (i = 0; i < MTK_RING_MAX; i++) {
ring[i] = kzalloc(sizeof(**ring), GFP_KERNEL);
if (!ring[i])
goto err_cleanup;
@@ -465,6 +465,9 @@
GFP_KERNEL);
if (!ring[i]->res_base)
goto err_cleanup;
+
+ ring[i]->cmd_next = ring[i]->cmd_base;
+ ring[i]->res_next = ring[i]->res_base;
}
return 0;
diff --git a/drivers/crypto/mediatek/mtk-platform.h b/drivers/crypto/mediatek/mtk-platform.h
index ed6d871..303c152 100644
--- a/drivers/crypto/mediatek/mtk-platform.h
+++ b/drivers/crypto/mediatek/mtk-platform.h
@@ -38,14 +38,14 @@
* Ring 2/3 are used by SHA.
*/
enum {
- RING0 = 0,
- RING1,
- RING2,
- RING3,
- RING_MAX,
+ MTK_RING0,
+ MTK_RING1,
+ MTK_RING2,
+ MTK_RING3,
+ MTK_RING_MAX
};
-#define MTK_REC_NUM (RING_MAX / 2)
+#define MTK_REC_NUM (MTK_RING_MAX / 2)
#define MTK_IRQ_NUM 5
/**
@@ -84,11 +84,12 @@
/**
* struct mtk_ring - Descriptor ring
* @cmd_base: pointer to command descriptor ring base
+ * @cmd_next: pointer to the next command descriptor
* @cmd_dma: DMA address of command descriptor ring
- * @cmd_pos: current position in the command descriptor ring
* @res_base: pointer to result descriptor ring base
+ * @res_next: pointer to the next result descriptor
+ * @res_prev: pointer to the previous result descriptor
* @res_dma: DMA address of result descriptor ring
- * @res_pos: current position in the result descriptor ring
*
* A descriptor ring is a circular buffer that is used to manage
* one or more descriptors. There are two type of descriptor rings;
@@ -96,11 +97,12 @@
*/
struct mtk_ring {
struct mtk_desc *cmd_base;
+ struct mtk_desc *cmd_next;
dma_addr_t cmd_dma;
- u32 cmd_pos;
struct mtk_desc *res_base;
+ struct mtk_desc *res_next;
+ struct mtk_desc *res_prev;
dma_addr_t res_dma;
- u32 res_pos;
};
/**
@@ -125,9 +127,11 @@
/**
* struct mtk_aes_rec - AES operation record
+ * @cryp: pointer to Cryptographic device
* @queue: crypto request queue
* @areq: pointer to async request
- * @task: the tasklet is use in AES interrupt
+ * @done_task: the tasklet is use in AES interrupt
+ * @queue_task: the tasklet is used to dequeue request
* @ctx: pointer to current context
* @src: the structure that holds source sg list info
* @dst: the structure that holds destination sg list info
@@ -136,16 +140,18 @@
* @resume: pointer to resume function
* @total: request buffer length
* @buf: pointer to page buffer
- * @id: record identification
+ * @id: the current use of ring
* @flags: it's describing AES operation state
* @lock: the async queue lock
*
* Structure used to record AES execution state.
*/
struct mtk_aes_rec {
+ struct mtk_cryp *cryp;
struct crypto_queue queue;
struct crypto_async_request *areq;
- struct tasklet_struct task;
+ struct tasklet_struct done_task;
+ struct tasklet_struct queue_task;
struct mtk_aes_base_ctx *ctx;
struct mtk_aes_dma src;
struct mtk_aes_dma dst;
@@ -166,19 +172,23 @@
/**
* struct mtk_sha_rec - SHA operation record
+ * @cryp: pointer to Cryptographic device
* @queue: crypto request queue
* @req: pointer to ahash request
- * @task: the tasklet is use in SHA interrupt
- * @id: record identification
+ * @done_task: the tasklet is use in SHA interrupt
+ * @queue_task: the tasklet is used to dequeue request
+ * @id: the current use of ring
* @flags: it's describing SHA operation state
- * @lock: the ablkcipher queue lock
+ * @lock: the async queue lock
*
* Structure used to record SHA execution state.
*/
struct mtk_sha_rec {
+ struct mtk_cryp *cryp;
struct crypto_queue queue;
struct ahash_request *req;
- struct tasklet_struct task;
+ struct tasklet_struct done_task;
+ struct tasklet_struct queue_task;
u8 id;
unsigned long flags;
@@ -193,13 +203,11 @@
* @clk_ethif: pointer to ethif clock
* @clk_cryp: pointer to crypto clock
* @irq: global system and rings IRQ
- * @ring: pointer to execution state of AES
- * @aes: pointer to execution state of SHA
- * @sha: each execution record map to a ring
+ * @ring: pointer to descriptor rings
+ * @aes: pointer to operation record of AES
+ * @sha: pointer to operation record of SHA
* @aes_list: device list of AES
* @sha_list: device list of SHA
- * @tmp: pointer to temporary buffer for internal use
- * @tmp_dma: DMA address of temporary buffer
* @rec: it's used to select SHA record for tfm
*
* Structure storing cryptographic device information.
@@ -211,15 +219,13 @@
struct clk *clk_cryp;
int irq[MTK_IRQ_NUM];
- struct mtk_ring *ring[RING_MAX];
+ struct mtk_ring *ring[MTK_RING_MAX];
struct mtk_aes_rec *aes[MTK_REC_NUM];
struct mtk_sha_rec *sha[MTK_REC_NUM];
struct list_head aes_list;
struct list_head sha_list;
- void *tmp;
- dma_addr_t tmp_dma;
bool rec;
};
diff --git a/drivers/crypto/mediatek/mtk-sha.c b/drivers/crypto/mediatek/mtk-sha.c
index 55e3805..2226f12 100644
--- a/drivers/crypto/mediatek/mtk-sha.c
+++ b/drivers/crypto/mediatek/mtk-sha.c
@@ -17,13 +17,13 @@
#define SHA_ALIGN_MSK (sizeof(u32) - 1)
#define SHA_QUEUE_SIZE 512
-#define SHA_TMP_BUF_SIZE 512
#define SHA_BUF_SIZE ((u32)PAGE_SIZE)
#define SHA_OP_UPDATE 1
#define SHA_OP_FINAL 2
#define SHA_DATA_LEN_MSK cpu_to_le32(GENMASK(16, 0))
+#define SHA_MAX_DIGEST_BUF_SIZE 32
/* SHA command token */
#define SHA_CT_SIZE 5
@@ -34,7 +34,6 @@
/* SHA transform information */
#define SHA_TFM_HASH cpu_to_le32(0x2 << 0)
-#define SHA_TFM_INNER_DIG cpu_to_le32(0x1 << 21)
#define SHA_TFM_SIZE(x) cpu_to_le32((x) << 8)
#define SHA_TFM_START cpu_to_le32(0x1 << 4)
#define SHA_TFM_CONTINUE cpu_to_le32(0x1 << 5)
@@ -61,31 +60,17 @@
#define SHA_FLAGS_PAD BIT(10)
/**
- * mtk_sha_ct is a set of hardware instructions(command token)
- * that are used to control engine's processing flow of SHA,
- * and it contains the first two words of transform state.
- */
-struct mtk_sha_ct {
- __le32 ctrl[2];
- __le32 cmd[3];
-};
-
-/**
- * mtk_sha_tfm is used to define SHA transform state
- * and store result digest that produced by engine.
- */
-struct mtk_sha_tfm {
- __le32 ctrl[2];
- __le32 digest[SIZE_IN_WORDS(SHA512_DIGEST_SIZE)];
-};
-
-/**
- * mtk_sha_info consists of command token and transform state
- * of SHA, its role is similar to mtk_aes_info.
+ * mtk_sha_info - hardware information of AES
+ * @cmd: command token, hardware instruction
+ * @tfm: transform state of cipher algorithm.
+ * @state: contains keys and initial vectors.
+ *
*/
struct mtk_sha_info {
- struct mtk_sha_ct ct;
- struct mtk_sha_tfm tfm;
+ __le32 ctrl[2];
+ __le32 cmd[3];
+ __le32 tfm[2];
+ __le32 digest[SHA_MAX_DIGEST_BUF_SIZE];
};
struct mtk_sha_reqctx {
@@ -94,7 +79,6 @@
unsigned long op;
u64 digcnt;
- bool start;
size_t bufcnt;
dma_addr_t dma_addr;
@@ -153,6 +137,21 @@
writel_relaxed(value, cryp->base + offset);
}
+static inline void mtk_sha_ring_shift(struct mtk_ring *ring,
+ struct mtk_desc **cmd_curr,
+ struct mtk_desc **res_curr,
+ int *count)
+{
+ *cmd_curr = ring->cmd_next++;
+ *res_curr = ring->res_next++;
+ (*count)++;
+
+ if (ring->cmd_next == ring->cmd_base + MTK_DESC_NUM) {
+ ring->cmd_next = ring->cmd_base;
+ ring->res_next = ring->res_base;
+ }
+}
+
static struct mtk_cryp *mtk_sha_find_dev(struct mtk_sha_ctx *tctx)
{
struct mtk_cryp *cryp = NULL;
@@ -251,7 +250,9 @@
bits[1] = cpu_to_be64(size << 3);
bits[0] = cpu_to_be64(size >> 61);
- if (ctx->flags & (SHA_FLAGS_SHA384 | SHA_FLAGS_SHA512)) {
+ switch (ctx->flags & SHA_FLAGS_ALGO_MSK) {
+ case SHA_FLAGS_SHA384:
+ case SHA_FLAGS_SHA512:
index = ctx->bufcnt & 0x7f;
padlen = (index < 112) ? (112 - index) : ((128 + 112) - index);
*(ctx->buffer + ctx->bufcnt) = 0x80;
@@ -259,7 +260,9 @@
memcpy(ctx->buffer + ctx->bufcnt + padlen, bits, 16);
ctx->bufcnt += padlen + 16;
ctx->flags |= SHA_FLAGS_PAD;
- } else {
+ break;
+
+ default:
index = ctx->bufcnt & 0x3f;
padlen = (index < 56) ? (56 - index) : ((64 + 56) - index);
*(ctx->buffer + ctx->bufcnt) = 0x80;
@@ -267,36 +270,35 @@
memcpy(ctx->buffer + ctx->bufcnt + padlen, &bits[1], 8);
ctx->bufcnt += padlen + 8;
ctx->flags |= SHA_FLAGS_PAD;
+ break;
}
}
/* Initialize basic transform information of SHA */
static void mtk_sha_info_init(struct mtk_sha_reqctx *ctx)
{
- struct mtk_sha_ct *ct = &ctx->info.ct;
- struct mtk_sha_tfm *tfm = &ctx->info.tfm;
+ struct mtk_sha_info *info = &ctx->info;
ctx->ct_hdr = SHA_CT_CTRL_HDR;
ctx->ct_size = SHA_CT_SIZE;
- tfm->ctrl[0] = SHA_TFM_HASH | SHA_TFM_INNER_DIG |
- SHA_TFM_SIZE(SIZE_IN_WORDS(ctx->ds));
+ info->tfm[0] = SHA_TFM_HASH | SHA_TFM_SIZE(SIZE_IN_WORDS(ctx->ds));
switch (ctx->flags & SHA_FLAGS_ALGO_MSK) {
case SHA_FLAGS_SHA1:
- tfm->ctrl[0] |= SHA_TFM_SHA1;
+ info->tfm[0] |= SHA_TFM_SHA1;
break;
case SHA_FLAGS_SHA224:
- tfm->ctrl[0] |= SHA_TFM_SHA224;
+ info->tfm[0] |= SHA_TFM_SHA224;
break;
case SHA_FLAGS_SHA256:
- tfm->ctrl[0] |= SHA_TFM_SHA256;
+ info->tfm[0] |= SHA_TFM_SHA256;
break;
case SHA_FLAGS_SHA384:
- tfm->ctrl[0] |= SHA_TFM_SHA384;
+ info->tfm[0] |= SHA_TFM_SHA384;
break;
case SHA_FLAGS_SHA512:
- tfm->ctrl[0] |= SHA_TFM_SHA512;
+ info->tfm[0] |= SHA_TFM_SHA512;
break;
default:
@@ -304,13 +306,13 @@
return;
}
- tfm->ctrl[1] = SHA_TFM_HASH_STORE;
- ct->ctrl[0] = tfm->ctrl[0] | SHA_TFM_CONTINUE | SHA_TFM_START;
- ct->ctrl[1] = tfm->ctrl[1];
+ info->tfm[1] = SHA_TFM_HASH_STORE;
+ info->ctrl[0] = info->tfm[0] | SHA_TFM_CONTINUE | SHA_TFM_START;
+ info->ctrl[1] = info->tfm[1];
- ct->cmd[0] = SHA_CMD0;
- ct->cmd[1] = SHA_CMD1;
- ct->cmd[2] = SHA_CMD2 | SHA_TFM_DIGEST(SIZE_IN_WORDS(ctx->ds));
+ info->cmd[0] = SHA_CMD0;
+ info->cmd[1] = SHA_CMD1;
+ info->cmd[2] = SHA_CMD2 | SHA_TFM_DIGEST(SIZE_IN_WORDS(ctx->ds));
}
/*
@@ -319,23 +321,21 @@
*/
static int mtk_sha_info_update(struct mtk_cryp *cryp,
struct mtk_sha_rec *sha,
- size_t len)
+ size_t len1, size_t len2)
{
struct mtk_sha_reqctx *ctx = ahash_request_ctx(sha->req);
struct mtk_sha_info *info = &ctx->info;
- struct mtk_sha_ct *ct = &info->ct;
-
- if (ctx->start)
- ctx->start = false;
- else
- ct->ctrl[0] &= ~SHA_TFM_START;
ctx->ct_hdr &= ~SHA_DATA_LEN_MSK;
- ctx->ct_hdr |= cpu_to_le32(len);
- ct->cmd[0] &= ~SHA_DATA_LEN_MSK;
- ct->cmd[0] |= cpu_to_le32(len);
+ ctx->ct_hdr |= cpu_to_le32(len1 + len2);
+ info->cmd[0] &= ~SHA_DATA_LEN_MSK;
+ info->cmd[0] |= cpu_to_le32(len1 + len2);
- ctx->digcnt += len;
+ /* Setting SHA_TFM_START only for the first iteration */
+ if (ctx->digcnt)
+ info->ctrl[0] &= ~SHA_TFM_START;
+
+ ctx->digcnt += len1;
ctx->ct_dma = dma_map_single(cryp->dev, info, sizeof(*info),
DMA_BIDIRECTIONAL);
@@ -343,7 +343,8 @@
dev_err(cryp->dev, "dma %zu bytes error\n", sizeof(*info));
return -EINVAL;
}
- ctx->tfm_dma = ctx->ct_dma + sizeof(*ct);
+
+ ctx->tfm_dma = ctx->ct_dma + sizeof(info->ctrl) + sizeof(info->cmd);
return 0;
}
@@ -408,7 +409,6 @@
ctx->bufcnt = 0;
ctx->digcnt = 0;
ctx->buffer = tctx->buf;
- ctx->start = true;
if (tctx->flags & SHA_FLAGS_HMAC) {
struct mtk_sha_hmac_ctx *bctx = tctx->base;
@@ -422,89 +422,39 @@
}
static int mtk_sha_xmit(struct mtk_cryp *cryp, struct mtk_sha_rec *sha,
- dma_addr_t addr, size_t len)
+ dma_addr_t addr1, size_t len1,
+ dma_addr_t addr2, size_t len2)
{
struct mtk_sha_reqctx *ctx = ahash_request_ctx(sha->req);
struct mtk_ring *ring = cryp->ring[sha->id];
- struct mtk_desc *cmd = ring->cmd_base + ring->cmd_pos;
- struct mtk_desc *res = ring->res_base + ring->res_pos;
- int err;
+ struct mtk_desc *cmd, *res;
+ int err, count = 0;
- err = mtk_sha_info_update(cryp, sha, len);
+ err = mtk_sha_info_update(cryp, sha, len1, len2);
if (err)
return err;
/* Fill in the command/result descriptors */
- res->hdr = MTK_DESC_FIRST | MTK_DESC_LAST | MTK_DESC_BUF_LEN(len);
- res->buf = cpu_to_le32(cryp->tmp_dma);
+ mtk_sha_ring_shift(ring, &cmd, &res, &count);
- cmd->hdr = MTK_DESC_FIRST | MTK_DESC_LAST | MTK_DESC_BUF_LEN(len) |
+ res->hdr = MTK_DESC_FIRST | MTK_DESC_BUF_LEN(len1);
+ cmd->hdr = MTK_DESC_FIRST | MTK_DESC_BUF_LEN(len1) |
MTK_DESC_CT_LEN(ctx->ct_size);
-
- cmd->buf = cpu_to_le32(addr);
+ cmd->buf = cpu_to_le32(addr1);
cmd->ct = cpu_to_le32(ctx->ct_dma);
cmd->ct_hdr = ctx->ct_hdr;
cmd->tfm = cpu_to_le32(ctx->tfm_dma);
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
+ if (len2) {
+ mtk_sha_ring_shift(ring, &cmd, &res, &count);
- ring->res_pos = ring->cmd_pos;
- /*
- * Make sure that all changes to the DMA ring are done before we
- * start engine.
- */
- wmb();
- /* Start DMA transfer */
- mtk_sha_write(cryp, RDR_PREP_COUNT(sha->id), MTK_DESC_CNT(1));
- mtk_sha_write(cryp, CDR_PREP_COUNT(sha->id), MTK_DESC_CNT(1));
+ res->hdr = MTK_DESC_BUF_LEN(len2);
+ cmd->hdr = MTK_DESC_BUF_LEN(len2);
+ cmd->buf = cpu_to_le32(addr2);
+ }
- return -EINPROGRESS;
-}
-
-static int mtk_sha_xmit2(struct mtk_cryp *cryp,
- struct mtk_sha_rec *sha,
- struct mtk_sha_reqctx *ctx,
- size_t len1, size_t len2)
-{
- struct mtk_ring *ring = cryp->ring[sha->id];
- struct mtk_desc *cmd = ring->cmd_base + ring->cmd_pos;
- struct mtk_desc *res = ring->res_base + ring->res_pos;
- int err;
-
- err = mtk_sha_info_update(cryp, sha, len1 + len2);
- if (err)
- return err;
-
- /* Fill in the command/result descriptors */
- res->hdr = MTK_DESC_BUF_LEN(len1) | MTK_DESC_FIRST;
- res->buf = cpu_to_le32(cryp->tmp_dma);
-
- cmd->hdr = MTK_DESC_BUF_LEN(len1) | MTK_DESC_FIRST |
- MTK_DESC_CT_LEN(ctx->ct_size);
- cmd->buf = cpu_to_le32(sg_dma_address(ctx->sg));
- cmd->ct = cpu_to_le32(ctx->ct_dma);
- cmd->ct_hdr = ctx->ct_hdr;
- cmd->tfm = cpu_to_le32(ctx->tfm_dma);
-
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
-
- ring->res_pos = ring->cmd_pos;
-
- cmd = ring->cmd_base + ring->cmd_pos;
- res = ring->res_base + ring->res_pos;
-
- res->hdr = MTK_DESC_BUF_LEN(len2) | MTK_DESC_LAST;
- res->buf = cpu_to_le32(cryp->tmp_dma);
-
- cmd->hdr = MTK_DESC_BUF_LEN(len2) | MTK_DESC_LAST;
- cmd->buf = cpu_to_le32(ctx->dma_addr);
-
- if (++ring->cmd_pos == MTK_DESC_NUM)
- ring->cmd_pos = 0;
-
- ring->res_pos = ring->cmd_pos;
+ cmd->hdr |= MTK_DESC_LAST;
+ res->hdr |= MTK_DESC_LAST;
/*
* Make sure that all changes to the DMA ring are done before we
@@ -512,8 +462,8 @@
*/
wmb();
/* Start DMA transfer */
- mtk_sha_write(cryp, RDR_PREP_COUNT(sha->id), MTK_DESC_CNT(2));
- mtk_sha_write(cryp, CDR_PREP_COUNT(sha->id), MTK_DESC_CNT(2));
+ mtk_sha_write(cryp, RDR_PREP_COUNT(sha->id), MTK_DESC_CNT(count));
+ mtk_sha_write(cryp, CDR_PREP_COUNT(sha->id), MTK_DESC_CNT(count));
return -EINPROGRESS;
}
@@ -532,7 +482,7 @@
ctx->flags &= ~SHA_FLAGS_SG;
- return mtk_sha_xmit(cryp, sha, ctx->dma_addr, count);
+ return mtk_sha_xmit(cryp, sha, ctx->dma_addr, count, 0, 0);
}
static int mtk_sha_update_slow(struct mtk_cryp *cryp,
@@ -625,7 +575,8 @@
if (len == 0) {
ctx->flags &= ~SHA_FLAGS_SG;
- return mtk_sha_xmit(cryp, sha, ctx->dma_addr, count);
+ return mtk_sha_xmit(cryp, sha, ctx->dma_addr,
+ count, 0, 0);
} else {
ctx->sg = sg;
@@ -635,7 +586,8 @@
}
ctx->flags |= SHA_FLAGS_SG;
- return mtk_sha_xmit2(cryp, sha, ctx, len, count);
+ return mtk_sha_xmit(cryp, sha, sg_dma_address(ctx->sg),
+ len, ctx->dma_addr, count);
}
}
@@ -646,7 +598,8 @@
ctx->flags |= SHA_FLAGS_SG;
- return mtk_sha_xmit(cryp, sha, sg_dma_address(ctx->sg), len);
+ return mtk_sha_xmit(cryp, sha, sg_dma_address(ctx->sg),
+ len, 0, 0);
}
static int mtk_sha_final_req(struct mtk_cryp *cryp,
@@ -668,7 +621,7 @@
static int mtk_sha_finish(struct ahash_request *req)
{
struct mtk_sha_reqctx *ctx = ahash_request_ctx(req);
- u32 *digest = ctx->info.tfm.digest;
+ __le32 *digest = ctx->info.digest;
u32 *result = (u32 *)req->result;
int i;
@@ -694,7 +647,7 @@
sha->req->base.complete(&sha->req->base, err);
/* Handle new request */
- mtk_sha_handle_queue(cryp, sha->id - RING2, NULL);
+ tasklet_schedule(&sha->queue_task);
}
static int mtk_sha_handle_queue(struct mtk_cryp *cryp, u8 id,
@@ -1216,60 +1169,38 @@
},
};
-static void mtk_sha_task0(unsigned long data)
+static void mtk_sha_queue_task(unsigned long data)
{
- struct mtk_cryp *cryp = (struct mtk_cryp *)data;
- struct mtk_sha_rec *sha = cryp->sha[0];
+ struct mtk_sha_rec *sha = (struct mtk_sha_rec *)data;
+
+ mtk_sha_handle_queue(sha->cryp, sha->id - MTK_RING2, NULL);
+}
+
+static void mtk_sha_done_task(unsigned long data)
+{
+ struct mtk_sha_rec *sha = (struct mtk_sha_rec *)data;
+ struct mtk_cryp *cryp = sha->cryp;
mtk_sha_unmap(cryp, sha);
mtk_sha_complete(cryp, sha);
}
-static void mtk_sha_task1(unsigned long data)
+static irqreturn_t mtk_sha_irq(int irq, void *dev_id)
{
- struct mtk_cryp *cryp = (struct mtk_cryp *)data;
- struct mtk_sha_rec *sha = cryp->sha[1];
+ struct mtk_sha_rec *sha = (struct mtk_sha_rec *)dev_id;
+ struct mtk_cryp *cryp = sha->cryp;
+ u32 val = mtk_sha_read(cryp, RDR_STAT(sha->id));
- mtk_sha_unmap(cryp, sha);
- mtk_sha_complete(cryp, sha);
-}
-
-static irqreturn_t mtk_sha_ring2_irq(int irq, void *dev_id)
-{
- struct mtk_cryp *cryp = (struct mtk_cryp *)dev_id;
- struct mtk_sha_rec *sha = cryp->sha[0];
- u32 val = mtk_sha_read(cryp, RDR_STAT(RING2));
-
- mtk_sha_write(cryp, RDR_STAT(RING2), val);
+ mtk_sha_write(cryp, RDR_STAT(sha->id), val);
if (likely((SHA_FLAGS_BUSY & sha->flags))) {
- mtk_sha_write(cryp, RDR_PROC_COUNT(RING2), MTK_CNT_RST);
- mtk_sha_write(cryp, RDR_THRESH(RING2),
+ mtk_sha_write(cryp, RDR_PROC_COUNT(sha->id), MTK_CNT_RST);
+ mtk_sha_write(cryp, RDR_THRESH(sha->id),
MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
- tasklet_schedule(&sha->task);
+ tasklet_schedule(&sha->done_task);
} else {
- dev_warn(cryp->dev, "AES interrupt when no active requests.\n");
- }
- return IRQ_HANDLED;
-}
-
-static irqreturn_t mtk_sha_ring3_irq(int irq, void *dev_id)
-{
- struct mtk_cryp *cryp = (struct mtk_cryp *)dev_id;
- struct mtk_sha_rec *sha = cryp->sha[1];
- u32 val = mtk_sha_read(cryp, RDR_STAT(RING3));
-
- mtk_sha_write(cryp, RDR_STAT(RING3), val);
-
- if (likely((SHA_FLAGS_BUSY & sha->flags))) {
- mtk_sha_write(cryp, RDR_PROC_COUNT(RING3), MTK_CNT_RST);
- mtk_sha_write(cryp, RDR_THRESH(RING3),
- MTK_RDR_PROC_THRESH | MTK_RDR_PROC_MODE);
-
- tasklet_schedule(&sha->task);
- } else {
- dev_warn(cryp->dev, "AES interrupt when no active requests.\n");
+ dev_warn(cryp->dev, "SHA interrupt when no active requests.\n");
}
return IRQ_HANDLED;
}
@@ -1288,14 +1219,20 @@
if (!sha[i])
goto err_cleanup;
- sha[i]->id = i + RING2;
+ sha[i]->cryp = cryp;
spin_lock_init(&sha[i]->lock);
crypto_init_queue(&sha[i]->queue, SHA_QUEUE_SIZE);
+
+ tasklet_init(&sha[i]->queue_task, mtk_sha_queue_task,
+ (unsigned long)sha[i]);
+ tasklet_init(&sha[i]->done_task, mtk_sha_done_task,
+ (unsigned long)sha[i]);
}
- tasklet_init(&sha[0]->task, mtk_sha_task0, (unsigned long)cryp);
- tasklet_init(&sha[1]->task, mtk_sha_task1, (unsigned long)cryp);
+ /* Link to ring2 and ring3 respectively */
+ sha[0]->id = MTK_RING2;
+ sha[1]->id = MTK_RING3;
cryp->rec = 1;
@@ -1312,7 +1249,9 @@
int i;
for (i = 0; i < MTK_REC_NUM; i++) {
- tasklet_kill(&cryp->sha[i]->task);
+ tasklet_kill(&cryp->sha[i]->done_task);
+ tasklet_kill(&cryp->sha[i]->queue_task);
+
kfree(cryp->sha[i]);
}
}
@@ -1368,35 +1307,23 @@
if (err)
goto err_record;
- /* Ring2 is use by SHA record0 */
- err = devm_request_irq(cryp->dev, cryp->irq[RING2],
- mtk_sha_ring2_irq, IRQF_TRIGGER_LOW,
- "mtk-sha", cryp);
+ err = devm_request_irq(cryp->dev, cryp->irq[MTK_RING2], mtk_sha_irq,
+ 0, "mtk-sha", cryp->sha[0]);
if (err) {
dev_err(cryp->dev, "unable to request sha irq0.\n");
goto err_res;
}
- /* Ring3 is use by SHA record1 */
- err = devm_request_irq(cryp->dev, cryp->irq[RING3],
- mtk_sha_ring3_irq, IRQF_TRIGGER_LOW,
- "mtk-sha", cryp);
+ err = devm_request_irq(cryp->dev, cryp->irq[MTK_RING3], mtk_sha_irq,
+ 0, "mtk-sha", cryp->sha[1]);
if (err) {
dev_err(cryp->dev, "unable to request sha irq1.\n");
goto err_res;
}
/* Enable ring2 and ring3 interrupt for hash */
- mtk_sha_write(cryp, AIC_ENABLE_SET(RING2), MTK_IRQ_RDR2);
- mtk_sha_write(cryp, AIC_ENABLE_SET(RING3), MTK_IRQ_RDR3);
-
- cryp->tmp = dma_alloc_coherent(cryp->dev, SHA_TMP_BUF_SIZE,
- &cryp->tmp_dma, GFP_KERNEL);
- if (!cryp->tmp) {
- dev_err(cryp->dev, "unable to allocate tmp buffer.\n");
- err = -EINVAL;
- goto err_res;
- }
+ mtk_sha_write(cryp, AIC_ENABLE_SET(MTK_RING2), MTK_IRQ_RDR2);
+ mtk_sha_write(cryp, AIC_ENABLE_SET(MTK_RING3), MTK_IRQ_RDR3);
spin_lock(&mtk_sha.lock);
list_add_tail(&cryp->sha_list, &mtk_sha.dev_list);
@@ -1412,8 +1339,6 @@
spin_lock(&mtk_sha.lock);
list_del(&cryp->sha_list);
spin_unlock(&mtk_sha.lock);
- dma_free_coherent(cryp->dev, SHA_TMP_BUF_SIZE,
- cryp->tmp, cryp->tmp_dma);
err_res:
mtk_sha_record_free(cryp);
err_record:
@@ -1429,7 +1354,5 @@
spin_unlock(&mtk_sha.lock);
mtk_sha_unregister_algs();
- dma_free_coherent(cryp->dev, SHA_TMP_BUF_SIZE,
- cryp->tmp, cryp->tmp_dma);
mtk_sha_record_free(cryp);
}
diff --git a/drivers/crypto/n2_core.c b/drivers/crypto/n2_core.c
index c5aac25..4ecb77a 100644
--- a/drivers/crypto/n2_core.c
+++ b/drivers/crypto/n2_core.c
@@ -65,6 +65,11 @@
struct list_head list;
};
+struct spu_qreg {
+ struct spu_queue *queue;
+ unsigned long type;
+};
+
static struct spu_queue **cpu_to_cwq;
static struct spu_queue **cpu_to_mau;
@@ -1631,31 +1636,27 @@
kmem_cache_destroy(queue_cache[HV_NCS_QTYPE_CWQ - 1]);
}
-static int spu_queue_register(struct spu_queue *p, unsigned long q_type)
+static long spu_queue_register_workfn(void *arg)
{
- cpumask_var_t old_allowed;
+ struct spu_qreg *qr = arg;
+ struct spu_queue *p = qr->queue;
+ unsigned long q_type = qr->type;
unsigned long hv_ret;
- if (cpumask_empty(&p->sharing))
- return -EINVAL;
-
- if (!alloc_cpumask_var(&old_allowed, GFP_KERNEL))
- return -ENOMEM;
-
- cpumask_copy(old_allowed, ¤t->cpus_allowed);
-
- set_cpus_allowed_ptr(current, &p->sharing);
-
hv_ret = sun4v_ncs_qconf(q_type, __pa(p->q),
CWQ_NUM_ENTRIES, &p->qhandle);
if (!hv_ret)
sun4v_ncs_sethead_marker(p->qhandle, 0);
- set_cpus_allowed_ptr(current, old_allowed);
+ return hv_ret ? -EINVAL : 0;
+}
- free_cpumask_var(old_allowed);
+static int spu_queue_register(struct spu_queue *p, unsigned long q_type)
+{
+ int cpu = cpumask_any_and(&p->sharing, cpu_online_mask);
+ struct spu_qreg qr = { .queue = p, .type = q_type };
- return (hv_ret ? -EINVAL : 0);
+ return work_on_cpu_safe(cpu, spu_queue_register_workfn, &qr);
}
static int spu_queue_setup(struct spu_queue *p)
diff --git a/drivers/crypto/qat/qat_common/qat_asym_algs.c b/drivers/crypto/qat/qat_common/qat_asym_algs.c
index 0d35dca..2aab80b 100644
--- a/drivers/crypto/qat/qat_common/qat_asym_algs.c
+++ b/drivers/crypto/qat/qat_common/qat_asym_algs.c
@@ -491,7 +491,7 @@
ctx->g2 = false;
}
-static int qat_dh_set_secret(struct crypto_kpp *tfm, void *buf,
+static int qat_dh_set_secret(struct crypto_kpp *tfm, const void *buf,
unsigned int len)
{
struct qat_dh_ctx *ctx = kpp_tfm_ctx(tfm);
diff --git a/drivers/crypto/s5p-sss.c b/drivers/crypto/s5p-sss.c
index 1b9da3d..7ac657f 100644
--- a/drivers/crypto/s5p-sss.c
+++ b/drivers/crypto/s5p-sss.c
@@ -170,6 +170,32 @@
int keylen;
};
+/**
+ * struct s5p_aes_dev - Crypto device state container
+ * @dev: Associated device
+ * @clk: Clock for accessing hardware
+ * @ioaddr: Mapped IO memory region
+ * @aes_ioaddr: Per-varian offset for AES block IO memory
+ * @irq_fc: Feed control interrupt line
+ * @req: Crypto request currently handled by the device
+ * @ctx: Configuration for currently handled crypto request
+ * @sg_src: Scatter list with source data for currently handled block
+ * in device. This is DMA-mapped into device.
+ * @sg_dst: Scatter list with destination data for currently handled block
+ * in device. This is DMA-mapped into device.
+ * @sg_src_cpy: In case of unaligned access, copied scatter list
+ * with source data.
+ * @sg_dst_cpy: In case of unaligned access, copied scatter list
+ * with destination data.
+ * @tasklet: New request scheduling jib
+ * @queue: Crypto queue
+ * @busy: Indicates whether the device is currently handling some request
+ * thus it uses some of the fields from this state, like:
+ * req, ctx, sg_src/dst (and copies). This essentially
+ * protects against concurrent access to these fields.
+ * @lock: Lock for protecting both access to device hardware registers
+ * and fields related to current request (including the busy field).
+ */
struct s5p_aes_dev {
struct device *dev;
struct clk *clk;
@@ -182,7 +208,6 @@
struct scatterlist *sg_src;
struct scatterlist *sg_dst;
- /* In case of unaligned access: */
struct scatterlist *sg_src_cpy;
struct scatterlist *sg_dst_cpy;
@@ -190,8 +215,6 @@
struct crypto_queue queue;
bool busy;
spinlock_t lock;
-
- struct samsung_aes_variant *variant;
};
static struct s5p_aes_dev *s5p_dev;
@@ -287,7 +310,6 @@
static void s5p_aes_complete(struct s5p_aes_dev *dev, int err)
{
dev->req->base.complete(&dev->req->base, err);
- dev->busy = false;
}
static void s5p_unset_outdata(struct s5p_aes_dev *dev)
@@ -462,7 +484,7 @@
spin_unlock_irqrestore(&dev->lock, flags);
s5p_aes_complete(dev, 0);
- dev->busy = true;
+ /* Device is still busy */
tasklet_schedule(&dev->tasklet);
} else {
/*
@@ -483,6 +505,7 @@
error:
s5p_sg_done(dev);
+ dev->busy = false;
spin_unlock_irqrestore(&dev->lock, flags);
s5p_aes_complete(dev, err);
@@ -634,6 +657,7 @@
indata_error:
s5p_sg_done(dev);
+ dev->busy = false;
spin_unlock_irqrestore(&dev->lock, flags);
s5p_aes_complete(dev, err);
}
@@ -851,7 +875,6 @@
}
pdata->busy = false;
- pdata->variant = variant;
pdata->dev = dev;
platform_set_drvdata(pdev, pdata);
s5p_dev = pdata;
diff --git a/drivers/crypto/stm32/Kconfig b/drivers/crypto/stm32/Kconfig
new file mode 100644
index 0000000..09b4ec8
--- /dev/null
+++ b/drivers/crypto/stm32/Kconfig
@@ -0,0 +1,7 @@
+config CRYPTO_DEV_STM32
+ tristate "Support for STM32 crypto accelerators"
+ depends on ARCH_STM32
+ select CRYPTO_HASH
+ help
+ This enables support for the CRC32 hw accelerator which can be found
+ on STMicroelectronis STM32 SOC.
diff --git a/drivers/crypto/stm32/Makefile b/drivers/crypto/stm32/Makefile
new file mode 100644
index 0000000..73b4c6e
--- /dev/null
+++ b/drivers/crypto/stm32/Makefile
@@ -0,0 +1,2 @@
+obj-$(CONFIG_CRYPTO_DEV_STM32) += stm32_cryp.o
+stm32_cryp-objs := stm32_crc32.o
diff --git a/drivers/crypto/stm32/stm32_crc32.c b/drivers/crypto/stm32/stm32_crc32.c
new file mode 100644
index 0000000..ec83b1e
--- /dev/null
+++ b/drivers/crypto/stm32/stm32_crc32.c
@@ -0,0 +1,324 @@
+/*
+ * Copyright (C) STMicroelectronics SA 2017
+ * Author: Fabien Dessenne <fabien.dessenne@st.com>
+ * License terms: GNU General Public License (GPL), version 2
+ */
+
+#include <linux/bitrev.h>
+#include <linux/clk.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+
+#include <crypto/internal/hash.h>
+
+#include <asm/unaligned.h>
+
+#define DRIVER_NAME "stm32-crc32"
+#define CHKSUM_DIGEST_SIZE 4
+#define CHKSUM_BLOCK_SIZE 1
+
+/* Registers */
+#define CRC_DR 0x00000000
+#define CRC_CR 0x00000008
+#define CRC_INIT 0x00000010
+#define CRC_POL 0x00000014
+
+/* Registers values */
+#define CRC_CR_RESET BIT(0)
+#define CRC_CR_REVERSE (BIT(7) | BIT(6) | BIT(5))
+#define CRC_INIT_DEFAULT 0xFFFFFFFF
+
+/* Polynomial reversed */
+#define POLY_CRC32 0xEDB88320
+#define POLY_CRC32C 0x82F63B78
+
+struct stm32_crc {
+ struct list_head list;
+ struct device *dev;
+ void __iomem *regs;
+ struct clk *clk;
+ u8 pending_data[sizeof(u32)];
+ size_t nb_pending_bytes;
+};
+
+struct stm32_crc_list {
+ struct list_head dev_list;
+ spinlock_t lock; /* protect dev_list */
+};
+
+static struct stm32_crc_list crc_list = {
+ .dev_list = LIST_HEAD_INIT(crc_list.dev_list),
+ .lock = __SPIN_LOCK_UNLOCKED(crc_list.lock),
+};
+
+struct stm32_crc_ctx {
+ u32 key;
+ u32 poly;
+};
+
+struct stm32_crc_desc_ctx {
+ u32 partial; /* crc32c: partial in first 4 bytes of that struct */
+ struct stm32_crc *crc;
+};
+
+static int stm32_crc32_cra_init(struct crypto_tfm *tfm)
+{
+ struct stm32_crc_ctx *mctx = crypto_tfm_ctx(tfm);
+
+ mctx->key = CRC_INIT_DEFAULT;
+ mctx->poly = POLY_CRC32;
+ return 0;
+}
+
+static int stm32_crc32c_cra_init(struct crypto_tfm *tfm)
+{
+ struct stm32_crc_ctx *mctx = crypto_tfm_ctx(tfm);
+
+ mctx->key = CRC_INIT_DEFAULT;
+ mctx->poly = POLY_CRC32C;
+ return 0;
+}
+
+static int stm32_crc_setkey(struct crypto_shash *tfm, const u8 *key,
+ unsigned int keylen)
+{
+ struct stm32_crc_ctx *mctx = crypto_shash_ctx(tfm);
+
+ if (keylen != sizeof(u32)) {
+ crypto_shash_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN);
+ return -EINVAL;
+ }
+
+ mctx->key = get_unaligned_le32(key);
+ return 0;
+}
+
+static int stm32_crc_init(struct shash_desc *desc)
+{
+ struct stm32_crc_desc_ctx *ctx = shash_desc_ctx(desc);
+ struct stm32_crc_ctx *mctx = crypto_shash_ctx(desc->tfm);
+ struct stm32_crc *crc;
+
+ spin_lock_bh(&crc_list.lock);
+ list_for_each_entry(crc, &crc_list.dev_list, list) {
+ ctx->crc = crc;
+ break;
+ }
+ spin_unlock_bh(&crc_list.lock);
+
+ /* Reset, set key, poly and configure in bit reverse mode */
+ writel(bitrev32(mctx->key), ctx->crc->regs + CRC_INIT);
+ writel(bitrev32(mctx->poly), ctx->crc->regs + CRC_POL);
+ writel(CRC_CR_RESET | CRC_CR_REVERSE, ctx->crc->regs + CRC_CR);
+
+ /* Store partial result */
+ ctx->partial = readl(ctx->crc->regs + CRC_DR);
+ ctx->crc->nb_pending_bytes = 0;
+
+ return 0;
+}
+
+static int stm32_crc_update(struct shash_desc *desc, const u8 *d8,
+ unsigned int length)
+{
+ struct stm32_crc_desc_ctx *ctx = shash_desc_ctx(desc);
+ struct stm32_crc *crc = ctx->crc;
+ u32 *d32;
+ unsigned int i;
+
+ if (unlikely(crc->nb_pending_bytes)) {
+ while (crc->nb_pending_bytes != sizeof(u32) && length) {
+ /* Fill in pending data */
+ crc->pending_data[crc->nb_pending_bytes++] = *(d8++);
+ length--;
+ }
+
+ if (crc->nb_pending_bytes == sizeof(u32)) {
+ /* Process completed pending data */
+ writel(*(u32 *)crc->pending_data, crc->regs + CRC_DR);
+ crc->nb_pending_bytes = 0;
+ }
+ }
+
+ d32 = (u32 *)d8;
+ for (i = 0; i < length >> 2; i++)
+ /* Process 32 bits data */
+ writel(*(d32++), crc->regs + CRC_DR);
+
+ /* Store partial result */
+ ctx->partial = readl(crc->regs + CRC_DR);
+
+ /* Check for pending data (non 32 bits) */
+ length &= 3;
+ if (likely(!length))
+ return 0;
+
+ if ((crc->nb_pending_bytes + length) >= sizeof(u32)) {
+ /* Shall not happen */
+ dev_err(crc->dev, "Pending data overflow\n");
+ return -EINVAL;
+ }
+
+ d8 = (const u8 *)d32;
+ for (i = 0; i < length; i++)
+ /* Store pending data */
+ crc->pending_data[crc->nb_pending_bytes++] = *(d8++);
+
+ return 0;
+}
+
+static int stm32_crc_final(struct shash_desc *desc, u8 *out)
+{
+ struct stm32_crc_desc_ctx *ctx = shash_desc_ctx(desc);
+ struct stm32_crc_ctx *mctx = crypto_shash_ctx(desc->tfm);
+
+ /* Send computed CRC */
+ put_unaligned_le32(mctx->poly == POLY_CRC32C ?
+ ~ctx->partial : ctx->partial, out);
+
+ return 0;
+}
+
+static int stm32_crc_finup(struct shash_desc *desc, const u8 *data,
+ unsigned int length, u8 *out)
+{
+ return stm32_crc_update(desc, data, length) ?:
+ stm32_crc_final(desc, out);
+}
+
+static int stm32_crc_digest(struct shash_desc *desc, const u8 *data,
+ unsigned int length, u8 *out)
+{
+ return stm32_crc_init(desc) ?: stm32_crc_finup(desc, data, length, out);
+}
+
+static struct shash_alg algs[] = {
+ /* CRC-32 */
+ {
+ .setkey = stm32_crc_setkey,
+ .init = stm32_crc_init,
+ .update = stm32_crc_update,
+ .final = stm32_crc_final,
+ .finup = stm32_crc_finup,
+ .digest = stm32_crc_digest,
+ .descsize = sizeof(struct stm32_crc_desc_ctx),
+ .digestsize = CHKSUM_DIGEST_SIZE,
+ .base = {
+ .cra_name = "crc32",
+ .cra_driver_name = DRIVER_NAME,
+ .cra_priority = 200,
+ .cra_blocksize = CHKSUM_BLOCK_SIZE,
+ .cra_alignmask = 3,
+ .cra_ctxsize = sizeof(struct stm32_crc_ctx),
+ .cra_module = THIS_MODULE,
+ .cra_init = stm32_crc32_cra_init,
+ }
+ },
+ /* CRC-32Castagnoli */
+ {
+ .setkey = stm32_crc_setkey,
+ .init = stm32_crc_init,
+ .update = stm32_crc_update,
+ .final = stm32_crc_final,
+ .finup = stm32_crc_finup,
+ .digest = stm32_crc_digest,
+ .descsize = sizeof(struct stm32_crc_desc_ctx),
+ .digestsize = CHKSUM_DIGEST_SIZE,
+ .base = {
+ .cra_name = "crc32c",
+ .cra_driver_name = DRIVER_NAME,
+ .cra_priority = 200,
+ .cra_blocksize = CHKSUM_BLOCK_SIZE,
+ .cra_alignmask = 3,
+ .cra_ctxsize = sizeof(struct stm32_crc_ctx),
+ .cra_module = THIS_MODULE,
+ .cra_init = stm32_crc32c_cra_init,
+ }
+ }
+};
+
+static int stm32_crc_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct stm32_crc *crc;
+ struct resource *res;
+ int ret;
+
+ crc = devm_kzalloc(dev, sizeof(*crc), GFP_KERNEL);
+ if (!crc)
+ return -ENOMEM;
+
+ crc->dev = dev;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ crc->regs = devm_ioremap_resource(dev, res);
+ if (IS_ERR(crc->regs)) {
+ dev_err(dev, "Cannot map CRC IO\n");
+ return PTR_ERR(crc->regs);
+ }
+
+ crc->clk = devm_clk_get(dev, NULL);
+ if (IS_ERR(crc->clk)) {
+ dev_err(dev, "Could not get clock\n");
+ return PTR_ERR(crc->clk);
+ }
+
+ ret = clk_prepare_enable(crc->clk);
+ if (ret) {
+ dev_err(crc->dev, "Failed to enable clock\n");
+ return ret;
+ }
+
+ platform_set_drvdata(pdev, crc);
+
+ spin_lock(&crc_list.lock);
+ list_add(&crc->list, &crc_list.dev_list);
+ spin_unlock(&crc_list.lock);
+
+ ret = crypto_register_shashes(algs, ARRAY_SIZE(algs));
+ if (ret) {
+ dev_err(dev, "Failed to register\n");
+ clk_disable_unprepare(crc->clk);
+ return ret;
+ }
+
+ dev_info(dev, "Initialized\n");
+
+ return 0;
+}
+
+static int stm32_crc_remove(struct platform_device *pdev)
+{
+ struct stm32_crc *crc = platform_get_drvdata(pdev);
+
+ spin_lock(&crc_list.lock);
+ list_del(&crc->list);
+ spin_unlock(&crc_list.lock);
+
+ crypto_unregister_shash(algs);
+
+ clk_disable_unprepare(crc->clk);
+
+ return 0;
+}
+
+static const struct of_device_id stm32_dt_ids[] = {
+ { .compatible = "st,stm32f7-crc", },
+ {},
+};
+MODULE_DEVICE_TABLE(of, stm32_dt_ids);
+
+static struct platform_driver stm32_crc_driver = {
+ .probe = stm32_crc_probe,
+ .remove = stm32_crc_remove,
+ .driver = {
+ .name = DRIVER_NAME,
+ .of_match_table = stm32_dt_ids,
+ },
+};
+
+module_platform_driver(stm32_crc_driver);
+
+MODULE_AUTHOR("Fabien Dessenne <fabien.dessenne@st.com>");
+MODULE_DESCRIPTION("STMicrolectronics STM32 CRC32 hardware driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/dax/pmem.c b/drivers/dax/pmem.c
index 033f49b3..cb0d742 100644
--- a/drivers/dax/pmem.c
+++ b/drivers/dax/pmem.c
@@ -43,6 +43,7 @@
struct dax_pmem *dax_pmem = to_dax_pmem(ref);
dev_dbg(dax_pmem->dev, "%s\n", __func__);
+ wait_for_completion(&dax_pmem->cmp);
percpu_ref_exit(ref);
}
@@ -53,7 +54,6 @@
dev_dbg(dax_pmem->dev, "%s\n", __func__);
percpu_ref_kill(ref);
- wait_for_completion(&dax_pmem->cmp);
}
static int dax_pmem_probe(struct device *dev)
diff --git a/drivers/devfreq/governor.h b/drivers/devfreq/governor.h
index 71576b8..a4f2fa1 100644
--- a/drivers/devfreq/governor.h
+++ b/drivers/devfreq/governor.h
@@ -25,6 +25,35 @@
#define DEVFREQ_GOV_SUSPEND 0x4
#define DEVFREQ_GOV_RESUME 0x5
+/**
+ * struct devfreq_governor - Devfreq policy governor
+ * @node: list node - contains registered devfreq governors
+ * @name: Governor's name
+ * @immutable: Immutable flag for governor. If the value is 1,
+ * this govenror is never changeable to other governor.
+ * @get_target_freq: Returns desired operating frequency for the device.
+ * Basically, get_target_freq will run
+ * devfreq_dev_profile.get_dev_status() to get the
+ * status of the device (load = busy_time / total_time).
+ * If no_central_polling is set, this callback is called
+ * only with update_devfreq() notified by OPP.
+ * @event_handler: Callback for devfreq core framework to notify events
+ * to governors. Events include per device governor
+ * init and exit, opp changes out of devfreq, suspend
+ * and resume of per device devfreq during device idle.
+ *
+ * Note that the callbacks are called with devfreq->lock locked by devfreq.
+ */
+struct devfreq_governor {
+ struct list_head node;
+
+ const char name[DEVFREQ_NAME_LEN];
+ const unsigned int immutable;
+ int (*get_target_freq)(struct devfreq *this, unsigned long *freq);
+ int (*event_handler)(struct devfreq *devfreq,
+ unsigned int event, void *data);
+};
+
/* Caution: devfreq->lock must be locked before calling update_devfreq */
extern int update_devfreq(struct devfreq *devfreq);
diff --git a/drivers/edac/Kconfig b/drivers/edac/Kconfig
index 4773f28..96afb2a 100644
--- a/drivers/edac/Kconfig
+++ b/drivers/edac/Kconfig
@@ -10,26 +10,16 @@
bool
menuconfig EDAC
- bool "EDAC (Error Detection And Correction) reporting"
- depends on HAS_IOMEM && EDAC_SUPPORT
+ tristate "EDAC (Error Detection And Correction) reporting"
+ depends on HAS_IOMEM && EDAC_SUPPORT && RAS
help
- EDAC is designed to report errors in the core system.
- These are low-level errors that are reported in the CPU or
- supporting chipset or other subsystems:
+ EDAC is a subsystem along with hardware-specific drivers designed to
+ report hardware errors. These are low-level errors that are reported
+ in the CPU or supporting chipset or other subsystems:
memory errors, cache errors, PCI errors, thermal throttling, etc..
If unsure, select 'Y'.
- If this code is reporting problems on your system, please
- see the EDAC project web pages for more information at:
-
- <http://bluesmoke.sourceforge.net/>
-
- and:
-
- <http://buttersideup.com/edacwiki>
-
- There is also a mailing list for the EDAC project, which can
- be found via the sourceforge page.
+ The mailing list for the EDAC project is linux-edac@vger.kernel.org.
if EDAC
@@ -62,21 +52,9 @@
which occur really early upon boot, before the module infrastructure
has been initialized.
-config EDAC_MM_EDAC
- tristate "Main Memory EDAC (Error Detection And Correction) reporting"
- select RAS
- help
- Some systems are able to detect and correct errors in main
- memory. EDAC can report statistics on memory error
- detection and correction (EDAC - or commonly referred to ECC
- errors). EDAC will also try to decode where these errors
- occurred so that a particular failing memory module can be
- replaced. If unsure, select 'Y'.
-
config EDAC_GHES
bool "Output ACPI APEI/GHES BIOS detected errors via EDAC"
- depends on ACPI_APEI_GHES && (EDAC_MM_EDAC=y)
- default y
+ depends on ACPI_APEI_GHES && (EDAC=y)
help
Not all machines support hardware-driven error report. Some of those
provide a BIOS-driven error report mechanism via ACPI, using the
@@ -98,7 +76,7 @@
config EDAC_AMD64
tristate "AMD64 (Opteron, Athlon64)"
- depends on EDAC_MM_EDAC && AMD_NB && EDAC_DECODE_MCE
+ depends on AMD_NB && EDAC_DECODE_MCE
help
Support for error detection and correction of DRAM ECC errors on
the AMD64 families (>= K8) of memory controllers.
@@ -124,28 +102,28 @@
config EDAC_AMD76X
tristate "AMD 76x (760, 762, 768)"
- depends on EDAC_MM_EDAC && PCI && X86_32
+ depends on PCI && X86_32
help
Support for error detection and correction on the AMD 76x
series of chipsets used with the Athlon processor.
config EDAC_E7XXX
tristate "Intel e7xxx (e7205, e7500, e7501, e7505)"
- depends on EDAC_MM_EDAC && PCI && X86_32
+ depends on PCI && X86_32
help
Support for error detection and correction on the Intel
E7205, E7500, E7501 and E7505 server chipsets.
config EDAC_E752X
tristate "Intel e752x (e7520, e7525, e7320) and 3100"
- depends on EDAC_MM_EDAC && PCI && X86
+ depends on PCI && X86
help
Support for error detection and correction on the Intel
E7520, E7525, E7320 server chipsets.
config EDAC_I82443BXGX
tristate "Intel 82443BX/GX (440BX/GX)"
- depends on EDAC_MM_EDAC && PCI && X86_32
+ depends on PCI && X86_32
depends on BROKEN
help
Support for error detection and correction on the Intel
@@ -153,56 +131,56 @@
config EDAC_I82875P
tristate "Intel 82875p (D82875P, E7210)"
- depends on EDAC_MM_EDAC && PCI && X86_32
+ depends on PCI && X86_32
help
Support for error detection and correction on the Intel
DP82785P and E7210 server chipsets.
config EDAC_I82975X
tristate "Intel 82975x (D82975x)"
- depends on EDAC_MM_EDAC && PCI && X86
+ depends on PCI && X86
help
Support for error detection and correction on the Intel
DP82975x server chipsets.
config EDAC_I3000
tristate "Intel 3000/3010"
- depends on EDAC_MM_EDAC && PCI && X86
+ depends on PCI && X86
help
Support for error detection and correction on the Intel
3000 and 3010 server chipsets.
config EDAC_I3200
tristate "Intel 3200"
- depends on EDAC_MM_EDAC && PCI && X86
+ depends on PCI && X86
help
Support for error detection and correction on the Intel
3200 and 3210 server chipsets.
config EDAC_IE31200
tristate "Intel e312xx"
- depends on EDAC_MM_EDAC && PCI && X86
+ depends on PCI && X86
help
Support for error detection and correction on the Intel
E3-1200 based DRAM controllers.
config EDAC_X38
tristate "Intel X38"
- depends on EDAC_MM_EDAC && PCI && X86
+ depends on PCI && X86
help
Support for error detection and correction on the Intel
X38 server chipsets.
config EDAC_I5400
tristate "Intel 5400 (Seaburg) chipsets"
- depends on EDAC_MM_EDAC && PCI && X86
+ depends on PCI && X86
help
Support for error detection and correction the Intel
i5400 MCH chipset (Seaburg).
config EDAC_I7CORE
tristate "Intel i7 Core (Nehalem) processors"
- depends on EDAC_MM_EDAC && PCI && X86 && X86_MCE_INTEL
+ depends on PCI && X86 && X86_MCE_INTEL
help
Support for error detection and correction the Intel
i7 Core (Nehalem) Integrated Memory Controller that exists on
@@ -211,58 +189,56 @@
config EDAC_I82860
tristate "Intel 82860"
- depends on EDAC_MM_EDAC && PCI && X86_32
+ depends on PCI && X86_32
help
Support for error detection and correction on the Intel
82860 chipset.
config EDAC_R82600
tristate "Radisys 82600 embedded chipset"
- depends on EDAC_MM_EDAC && PCI && X86_32
+ depends on PCI && X86_32
help
Support for error detection and correction on the Radisys
82600 embedded chipset.
config EDAC_I5000
tristate "Intel Greencreek/Blackford chipset"
- depends on EDAC_MM_EDAC && X86 && PCI
+ depends on X86 && PCI
help
Support for error detection and correction the Intel
Greekcreek/Blackford chipsets.
config EDAC_I5100
tristate "Intel San Clemente MCH"
- depends on EDAC_MM_EDAC && X86 && PCI
+ depends on X86 && PCI
help
Support for error detection and correction the Intel
San Clemente MCH.
config EDAC_I7300
tristate "Intel Clarksboro MCH"
- depends on EDAC_MM_EDAC && X86 && PCI
+ depends on X86 && PCI
help
Support for error detection and correction the Intel
Clarksboro MCH (Intel 7300 chipset).
config EDAC_SBRIDGE
tristate "Intel Sandy-Bridge/Ivy-Bridge/Haswell Integrated MC"
- depends on EDAC_MM_EDAC && PCI && X86_64 && X86_MCE_INTEL
- depends on PCI_MMCONFIG
+ depends on PCI && X86_64 && X86_MCE_INTEL && PCI_MMCONFIG
help
Support for error detection and correction the Intel
Sandy Bridge, Ivy Bridge and Haswell Integrated Memory Controllers.
config EDAC_SKX
tristate "Intel Skylake server Integrated MC"
- depends on EDAC_MM_EDAC && PCI && X86_64 && X86_MCE_INTEL
- depends on PCI_MMCONFIG
+ depends on PCI && X86_64 && X86_MCE_INTEL && PCI_MMCONFIG
help
Support for error detection and correction the Intel
Skylake server Integrated Memory Controllers.
config EDAC_PND2
tristate "Intel Pondicherry2"
- depends on EDAC_MM_EDAC && PCI && X86_64 && X86_MCE_INTEL
+ depends on PCI && X86_64 && X86_MCE_INTEL
help
Support for error detection and correction on the Intel
Pondicherry2 Integrated Memory Controller. This SoC IP is
@@ -271,36 +247,35 @@
config EDAC_MPC85XX
tristate "Freescale MPC83xx / MPC85xx"
- depends on EDAC_MM_EDAC && FSL_SOC
+ depends on FSL_SOC
help
Support for error detection and correction on the Freescale
MPC8349, MPC8560, MPC8540, MPC8548, T4240
config EDAC_LAYERSCAPE
tristate "Freescale Layerscape DDR"
- depends on EDAC_MM_EDAC && ARCH_LAYERSCAPE
+ depends on ARCH_LAYERSCAPE
help
Support for error detection and correction on Freescale memory
controllers on Layerscape SoCs.
config EDAC_MV64X60
tristate "Marvell MV64x60"
- depends on EDAC_MM_EDAC && MV64X60
+ depends on MV64X60
help
Support for error detection and correction on the Marvell
MV64360 and MV64460 chipsets.
config EDAC_PASEMI
tristate "PA Semi PWRficient"
- depends on EDAC_MM_EDAC && PCI
- depends on PPC_PASEMI
+ depends on PPC_PASEMI && PCI
help
Support for error detection and correction on PA Semi
PWRficient.
config EDAC_CELL
tristate "Cell Broadband Engine memory controller"
- depends on EDAC_MM_EDAC && PPC_CELL_COMMON
+ depends on PPC_CELL_COMMON
help
Support for error detection and correction on the
Cell Broadband Engine internal memory controller
@@ -308,7 +283,7 @@
config EDAC_PPC4XX
tristate "PPC4xx IBM DDR2 Memory Controller"
- depends on EDAC_MM_EDAC && 4xx
+ depends on 4xx
help
This enables support for EDAC on the ECC memory used
with the IBM DDR2 memory controller found in various
@@ -317,7 +292,7 @@
config EDAC_AMD8131
tristate "AMD8131 HyperTransport PCI-X Tunnel"
- depends on EDAC_MM_EDAC && PCI && PPC_MAPLE
+ depends on PCI && PPC_MAPLE
help
Support for error detection and correction on the
AMD8131 HyperTransport PCI-X Tunnel chip.
@@ -326,7 +301,7 @@
config EDAC_AMD8111
tristate "AMD8111 HyperTransport I/O Hub"
- depends on EDAC_MM_EDAC && PCI && PPC_MAPLE
+ depends on PCI && PPC_MAPLE
help
Support for error detection and correction on the
AMD8111 HyperTransport I/O Hub chip.
@@ -335,7 +310,7 @@
config EDAC_CPC925
tristate "IBM CPC925 Memory Controller (PPC970FX)"
- depends on EDAC_MM_EDAC && PPC64
+ depends on PPC64
help
Support for error detection and correction on the
IBM CPC925 Bridge and Memory Controller, which is
@@ -344,7 +319,7 @@
config EDAC_TILE
tristate "Tilera Memory Controller"
- depends on EDAC_MM_EDAC && TILE
+ depends on TILE
default y
help
Support for error detection and correction on the
@@ -352,49 +327,59 @@
config EDAC_HIGHBANK_MC
tristate "Highbank Memory Controller"
- depends on EDAC_MM_EDAC && ARCH_HIGHBANK
+ depends on ARCH_HIGHBANK
help
Support for error detection and correction on the
Calxeda Highbank memory controller.
config EDAC_HIGHBANK_L2
tristate "Highbank L2 Cache"
- depends on EDAC_MM_EDAC && ARCH_HIGHBANK
+ depends on ARCH_HIGHBANK
help
Support for error detection and correction on the
Calxeda Highbank memory controller.
config EDAC_OCTEON_PC
tristate "Cavium Octeon Primary Caches"
- depends on EDAC_MM_EDAC && CPU_CAVIUM_OCTEON
+ depends on CPU_CAVIUM_OCTEON
help
Support for error detection and correction on the primary caches of
the cnMIPS cores of Cavium Octeon family SOCs.
config EDAC_OCTEON_L2C
tristate "Cavium Octeon Secondary Caches (L2C)"
- depends on EDAC_MM_EDAC && CAVIUM_OCTEON_SOC
+ depends on CAVIUM_OCTEON_SOC
help
Support for error detection and correction on the
Cavium Octeon family of SOCs.
config EDAC_OCTEON_LMC
tristate "Cavium Octeon DRAM Memory Controller (LMC)"
- depends on EDAC_MM_EDAC && CAVIUM_OCTEON_SOC
+ depends on CAVIUM_OCTEON_SOC
help
Support for error detection and correction on the
Cavium Octeon family of SOCs.
config EDAC_OCTEON_PCI
tristate "Cavium Octeon PCI Controller"
- depends on EDAC_MM_EDAC && PCI && CAVIUM_OCTEON_SOC
+ depends on PCI && CAVIUM_OCTEON_SOC
help
Support for error detection and correction on the
Cavium Octeon family of SOCs.
+config EDAC_THUNDERX
+ tristate "Cavium ThunderX EDAC"
+ depends on ARM64
+ depends on PCI
+ help
+ Support for error detection and correction on the
+ Cavium ThunderX memory controllers (LMC), Cache
+ Coherent Processor Interconnect (CCPI) and L2 cache
+ blocks (TAD, CBC, MCI).
+
config EDAC_ALTERA
bool "Altera SOCFPGA ECC"
- depends on EDAC_MM_EDAC=y && ARCH_SOCFPGA
+ depends on EDAC=y && ARCH_SOCFPGA
help
Support for error detection and correction on the
Altera SOCs. This must be selected for SDRAM ECC.
@@ -460,14 +445,14 @@
config EDAC_SYNOPSYS
tristate "Synopsys DDR Memory Controller"
- depends on EDAC_MM_EDAC && ARCH_ZYNQ
+ depends on ARCH_ZYNQ
help
Support for error detection and correction on the Synopsys DDR
memory controller.
config EDAC_XGENE
tristate "APM X-Gene SoC"
- depends on EDAC_MM_EDAC && (ARM64 || COMPILE_TEST)
+ depends on (ARM64 || COMPILE_TEST)
help
Support for error detection and correction on the
APM X-Gene family of SOCs.
diff --git a/drivers/edac/Makefile b/drivers/edac/Makefile
index 587107e..0fd9ffa 100644
--- a/drivers/edac/Makefile
+++ b/drivers/edac/Makefile
@@ -6,8 +6,7 @@
# GNU General Public License.
#
-obj-$(CONFIG_EDAC) := edac_stub.o
-obj-$(CONFIG_EDAC_MM_EDAC) += edac_core.o
+obj-$(CONFIG_EDAC) := edac_core.o
edac_core-y := edac_mc.o edac_device.o edac_mc_sysfs.o
edac_core-y += edac_module.o edac_device_sysfs.o wq.o
@@ -67,13 +66,14 @@
obj-$(CONFIG_EDAC_TILE) += tile_edac.o
-obj-$(CONFIG_EDAC_HIGHBANK_MC) += highbank_mc_edac.o
-obj-$(CONFIG_EDAC_HIGHBANK_L2) += highbank_l2_edac.o
+obj-$(CONFIG_EDAC_HIGHBANK_MC) += highbank_mc_edac.o
+obj-$(CONFIG_EDAC_HIGHBANK_L2) += highbank_l2_edac.o
obj-$(CONFIG_EDAC_OCTEON_PC) += octeon_edac-pc.o
obj-$(CONFIG_EDAC_OCTEON_L2C) += octeon_edac-l2c.o
obj-$(CONFIG_EDAC_OCTEON_LMC) += octeon_edac-lmc.o
obj-$(CONFIG_EDAC_OCTEON_PCI) += octeon_edac-pci.o
+obj-$(CONFIG_EDAC_THUNDERX) += thunderx_edac.o
obj-$(CONFIG_EDAC_ALTERA) += altera_edac.o
obj-$(CONFIG_EDAC_SYNOPSYS) += synopsys_edac.o
diff --git a/drivers/edac/altera_edac.c b/drivers/edac/altera_edac.c
index c5a5b91..7717b09 100644
--- a/drivers/edac/altera_edac.c
+++ b/drivers/edac/altera_edac.c
@@ -1023,13 +1023,23 @@
return ret;
}
+static int socfpga_is_a10(void)
+{
+ return of_machine_is_compatible("altr,socfpga-arria10");
+}
+
static int validate_parent_available(struct device_node *np);
static const struct of_device_id altr_edac_a10_device_of_match[];
static int __init __maybe_unused altr_init_a10_ecc_device_type(char *compat)
{
int irq;
- struct device_node *child, *np = of_find_compatible_node(NULL, NULL,
- "altr,socfpga-a10-ecc-manager");
+ struct device_node *child, *np;
+
+ if (!socfpga_is_a10())
+ return -ENODEV;
+
+ np = of_find_compatible_node(NULL, NULL,
+ "altr,socfpga-a10-ecc-manager");
if (!np) {
edac_printk(KERN_ERR, EDAC_DEVICE, "ECC Manager not found\n");
return -ENODEV;
@@ -1545,8 +1555,12 @@
static int __init socfpga_init_sdmmc_ecc(void)
{
int rc = -ENODEV;
- struct device_node *child = of_find_compatible_node(NULL, NULL,
- "altr,socfpga-sdmmc-ecc");
+ struct device_node *child;
+
+ if (!socfpga_is_a10())
+ return -ENODEV;
+
+ child = of_find_compatible_node(NULL, NULL, "altr,socfpga-sdmmc-ecc");
if (!child) {
edac_printk(KERN_WARNING, EDAC_DEVICE, "SDMMC node not found\n");
return -ENODEV;
diff --git a/drivers/edac/edac_mc.c b/drivers/edac/edac_mc.c
index e5573c5..4800721 100644
--- a/drivers/edac/edac_mc.c
+++ b/drivers/edac/edac_mc.c
@@ -40,6 +40,11 @@
#define edac_atomic_scrub(va, size) do { } while (0)
#endif
+int edac_op_state = EDAC_OPSTATE_INVAL;
+EXPORT_SYMBOL_GPL(edac_op_state);
+
+static int edac_report = EDAC_REPORTING_ENABLED;
+
/* lock to memory controller's control array */
static DEFINE_MUTEX(mem_ctls_mutex);
static LIST_HEAD(mc_devices);
@@ -52,6 +57,65 @@
static struct bus_type mc_bus[EDAC_MAX_MCS];
+int edac_get_report_status(void)
+{
+ return edac_report;
+}
+EXPORT_SYMBOL_GPL(edac_get_report_status);
+
+void edac_set_report_status(int new)
+{
+ if (new == EDAC_REPORTING_ENABLED ||
+ new == EDAC_REPORTING_DISABLED ||
+ new == EDAC_REPORTING_FORCE)
+ edac_report = new;
+}
+EXPORT_SYMBOL_GPL(edac_set_report_status);
+
+static int edac_report_set(const char *str, const struct kernel_param *kp)
+{
+ if (!str)
+ return -EINVAL;
+
+ if (!strncmp(str, "on", 2))
+ edac_report = EDAC_REPORTING_ENABLED;
+ else if (!strncmp(str, "off", 3))
+ edac_report = EDAC_REPORTING_DISABLED;
+ else if (!strncmp(str, "force", 5))
+ edac_report = EDAC_REPORTING_FORCE;
+
+ return 0;
+}
+
+static int edac_report_get(char *buffer, const struct kernel_param *kp)
+{
+ int ret = 0;
+
+ switch (edac_report) {
+ case EDAC_REPORTING_ENABLED:
+ ret = sprintf(buffer, "on");
+ break;
+ case EDAC_REPORTING_DISABLED:
+ ret = sprintf(buffer, "off");
+ break;
+ case EDAC_REPORTING_FORCE:
+ ret = sprintf(buffer, "force");
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static const struct kernel_param_ops edac_report_ops = {
+ .set = edac_report_set,
+ .get = edac_report_get,
+};
+
+module_param_cb(edac_report, &edac_report_ops, &edac_report, 0644);
+
unsigned edac_dimm_info_location(struct dimm_info *dimm, char *buf,
unsigned len)
{
@@ -505,22 +569,6 @@
EXPORT_SYMBOL_GPL(find_mci_by_dev);
/*
- * handler for EDAC to check if NMI type handler has asserted interrupt
- */
-static int edac_mc_assert_error_check_and_clear(void)
-{
- int old_state;
-
- if (edac_op_state == EDAC_OPSTATE_POLL)
- return 1;
-
- old_state = edac_err_assert;
- edac_err_assert = 0;
-
- return old_state;
-}
-
-/*
* edac_mc_workq_function
* performs the operation scheduled by a workq request
*/
@@ -536,7 +584,7 @@
return;
}
- if (edac_mc_assert_error_check_and_clear())
+ if (edac_op_state == EDAC_OPSTATE_POLL)
mci->edac_check(mci);
mutex_unlock(&mem_ctls_mutex);
@@ -601,7 +649,6 @@
}
list_add_tail_rcu(&mci->link, insert_before);
- atomic_inc(&edac_handlers);
return 0;
fail0:
@@ -619,7 +666,6 @@
static int del_mc_from_global_list(struct mem_ctl_info *mci)
{
- int handlers = atomic_dec_return(&edac_handlers);
list_del_rcu(&mci->link);
/* these are for safe removal of devices from global list while
@@ -628,7 +674,7 @@
synchronize_rcu();
INIT_LIST_HEAD(&mci->link);
- return handlers;
+ return list_empty(&mc_devices);
}
struct mem_ctl_info *edac_mc_find(int idx)
@@ -763,7 +809,7 @@
/* mark MCI offline: */
mci->op_state = OP_OFFLINE;
- if (!del_mc_from_global_list(mci))
+ if (del_mc_from_global_list(mci))
edac_mc_owner = NULL;
mutex_unlock(&mem_ctls_mutex);
@@ -1195,10 +1241,13 @@
/* Report the error via the trace interface */
grain_bits = fls_long(e->grain) + 1;
- trace_mc_event(type, e->msg, e->label, e->error_count,
- mci->mc_idx, e->top_layer, e->mid_layer, e->low_layer,
- (e->page_frame_number << PAGE_SHIFT) | e->offset_in_page,
- grain_bits, e->syndrome, e->other_detail);
+
+ if (IS_ENABLED(CONFIG_RAS))
+ trace_mc_event(type, e->msg, e->label, e->error_count,
+ mci->mc_idx, e->top_layer, e->mid_layer,
+ e->low_layer,
+ (e->page_frame_number << PAGE_SHIFT) | e->offset_in_page,
+ grain_bits, e->syndrome, e->other_detail);
edac_raw_mc_handle_error(type, mci, e);
}
diff --git a/drivers/edac/edac_stub.c b/drivers/edac/edac_stub.c
deleted file mode 100644
index 952e411..0000000
--- a/drivers/edac/edac_stub.c
+++ /dev/null
@@ -1,68 +0,0 @@
-/*
- * common EDAC components that must be in kernel
- *
- * Author: Dave Jiang <djiang@mvista.com>
- *
- * 2007 (c) MontaVista Software, Inc.
- * 2010 (c) Advanced Micro Devices Inc.
- * Borislav Petkov <bp@alien8.de>
- *
- * This file is licensed under the terms of the GNU General Public
- * License version 2. This program is licensed "as is" without any
- * warranty of any kind, whether express or implied.
- *
- */
-#include <linux/module.h>
-#include <linux/edac.h>
-#include <linux/atomic.h>
-#include <linux/device.h>
-
-int edac_op_state = EDAC_OPSTATE_INVAL;
-EXPORT_SYMBOL_GPL(edac_op_state);
-
-atomic_t edac_handlers = ATOMIC_INIT(0);
-EXPORT_SYMBOL_GPL(edac_handlers);
-
-int edac_err_assert = 0;
-EXPORT_SYMBOL_GPL(edac_err_assert);
-
-int edac_report_status = EDAC_REPORTING_ENABLED;
-EXPORT_SYMBOL_GPL(edac_report_status);
-
-static int __init edac_report_setup(char *str)
-{
- if (!str)
- return -EINVAL;
-
- if (!strncmp(str, "on", 2))
- set_edac_report_status(EDAC_REPORTING_ENABLED);
- else if (!strncmp(str, "off", 3))
- set_edac_report_status(EDAC_REPORTING_DISABLED);
- else if (!strncmp(str, "force", 5))
- set_edac_report_status(EDAC_REPORTING_FORCE);
-
- return 0;
-}
-__setup("edac_report=", edac_report_setup);
-
-/*
- * called to determine if there is an EDAC driver interested in
- * knowing an event (such as NMI) occurred
- */
-int edac_handler_set(void)
-{
- if (edac_op_state == EDAC_OPSTATE_POLL)
- return 0;
-
- return atomic_read(&edac_handlers);
-}
-EXPORT_SYMBOL_GPL(edac_handler_set);
-
-/*
- * handler for NMI type of interrupts to assert error
- */
-void edac_atomic_assert_error(void)
-{
- edac_err_assert++;
-}
-EXPORT_SYMBOL_GPL(edac_atomic_assert_error);
diff --git a/drivers/edac/pnd2_edac.c b/drivers/edac/pnd2_edac.c
index 928e0db..1cad5a9 100644
--- a/drivers/edac/pnd2_edac.c
+++ b/drivers/edac/pnd2_edac.c
@@ -1349,7 +1349,7 @@
struct dram_addr daddr;
char *type;
- if (get_edac_report_status() == EDAC_REPORTING_DISABLED)
+ if (edac_get_report_status() == EDAC_REPORTING_DISABLED)
return NOTIFY_DONE;
mci = pnd2_mci;
diff --git a/drivers/edac/sb_edac.c b/drivers/edac/sb_edac.c
index a65ea44..ea21cb6 100644
--- a/drivers/edac/sb_edac.c
+++ b/drivers/edac/sb_edac.c
@@ -3075,7 +3075,7 @@
struct sbridge_pvt *pvt;
char *type;
- if (get_edac_report_status() == EDAC_REPORTING_DISABLED)
+ if (edac_get_report_status() == EDAC_REPORTING_DISABLED)
return NOTIFY_DONE;
mci = get_mci_for_node_id(mce->socketid);
@@ -3441,7 +3441,7 @@
if (rc >= 0) {
mce_register_decode_chain(&sbridge_mce_dec);
- if (get_edac_report_status() == EDAC_REPORTING_DISABLED)
+ if (edac_get_report_status() == EDAC_REPORTING_DISABLED)
sbridge_printk(KERN_WARNING, "Loading driver, error reporting disabled.\n");
return 0;
}
diff --git a/drivers/edac/skx_edac.c b/drivers/edac/skx_edac.c
index 1159dba..64bef6c9 100644
--- a/drivers/edac/skx_edac.c
+++ b/drivers/edac/skx_edac.c
@@ -971,7 +971,7 @@
struct mem_ctl_info *mci;
char *type;
- if (get_edac_report_status() == EDAC_REPORTING_DISABLED)
+ if (edac_get_report_status() == EDAC_REPORTING_DISABLED)
return NOTIFY_DONE;
/* ignore unless this is memory related with an address */
diff --git a/drivers/edac/thunderx_edac.c b/drivers/edac/thunderx_edac.c
new file mode 100644
index 0000000..86d585c
--- /dev/null
+++ b/drivers/edac/thunderx_edac.c
@@ -0,0 +1,2174 @@
+/*
+ * Cavium ThunderX memory controller kernel module
+ *
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License. See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright Cavium, Inc. (C) 2015-2017. All rights reserved.
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/edac.h>
+#include <linux/interrupt.h>
+#include <linux/string.h>
+#include <linux/stop_machine.h>
+#include <linux/delay.h>
+#include <linux/sizes.h>
+#include <linux/atomic.h>
+#include <linux/bitfield.h>
+#include <linux/circ_buf.h>
+
+#include <asm/page.h>
+
+#include "edac_module.h"
+
+#define phys_to_pfn(phys) (PFN_DOWN(phys))
+
+#define THUNDERX_NODE GENMASK(45, 44)
+
+enum {
+ ERR_CORRECTED = 1,
+ ERR_UNCORRECTED = 2,
+ ERR_UNKNOWN = 3,
+};
+
+#define MAX_SYNDROME_REGS 4
+
+struct error_syndrome {
+ u64 reg[MAX_SYNDROME_REGS];
+};
+
+struct error_descr {
+ int type;
+ u64 mask;
+ char *descr;
+};
+
+static void decode_register(char *str, size_t size,
+ const struct error_descr *descr,
+ const uint64_t reg)
+{
+ int ret = 0;
+
+ while (descr->type && descr->mask && descr->descr) {
+ if (reg & descr->mask) {
+ ret = snprintf(str, size, "\n\t%s, %s",
+ descr->type == ERR_CORRECTED ?
+ "Corrected" : "Uncorrected",
+ descr->descr);
+ str += ret;
+ size -= ret;
+ }
+ descr++;
+ }
+}
+
+static unsigned long get_bits(unsigned long data, int pos, int width)
+{
+ return (data >> pos) & ((1 << width) - 1);
+}
+
+#define L2C_CTL 0x87E080800000
+#define L2C_CTL_DISIDXALIAS BIT(0)
+
+#define PCI_DEVICE_ID_THUNDER_LMC 0xa022
+
+#define LMC_FADR 0x20
+#define LMC_FADR_FDIMM(x) ((x >> 37) & 0x1)
+#define LMC_FADR_FBUNK(x) ((x >> 36) & 0x1)
+#define LMC_FADR_FBANK(x) ((x >> 32) & 0xf)
+#define LMC_FADR_FROW(x) ((x >> 14) & 0xffff)
+#define LMC_FADR_FCOL(x) ((x >> 0) & 0x1fff)
+
+#define LMC_NXM_FADR 0x28
+#define LMC_ECC_SYND 0x38
+
+#define LMC_ECC_PARITY_TEST 0x108
+
+#define LMC_INT_W1S 0x150
+
+#define LMC_INT_ENA_W1C 0x158
+#define LMC_INT_ENA_W1S 0x160
+
+#define LMC_CONFIG 0x188
+
+#define LMC_CONFIG_BG2 BIT(62)
+#define LMC_CONFIG_RANK_ENA BIT(42)
+#define LMC_CONFIG_PBANK_LSB(x) (((x) >> 5) & 0xF)
+#define LMC_CONFIG_ROW_LSB(x) (((x) >> 2) & 0x7)
+
+#define LMC_CONTROL 0x190
+#define LMC_CONTROL_XOR_BANK BIT(16)
+
+#define LMC_INT 0x1F0
+
+#define LMC_INT_DDR_ERR BIT(11)
+#define LMC_INT_DED_ERR (0xFUL << 5)
+#define LMC_INT_SEC_ERR (0xFUL << 1)
+#define LMC_INT_NXM_WR_MASK BIT(0)
+
+#define LMC_DDR_PLL_CTL 0x258
+#define LMC_DDR_PLL_CTL_DDR4 BIT(29)
+
+#define LMC_FADR_SCRAMBLED 0x330
+
+#define LMC_INT_UE (LMC_INT_DDR_ERR | LMC_INT_DED_ERR | \
+ LMC_INT_NXM_WR_MASK)
+
+#define LMC_INT_CE (LMC_INT_SEC_ERR)
+
+static const struct error_descr lmc_errors[] = {
+ {
+ .type = ERR_CORRECTED,
+ .mask = LMC_INT_SEC_ERR,
+ .descr = "Single-bit ECC error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = LMC_INT_DDR_ERR,
+ .descr = "DDR chip error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = LMC_INT_DED_ERR,
+ .descr = "Double-bit ECC error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = LMC_INT_NXM_WR_MASK,
+ .descr = "Non-existent memory write",
+ },
+ {0, 0, NULL},
+};
+
+#define LMC_INT_EN_DDR_ERROR_ALERT_ENA BIT(5)
+#define LMC_INT_EN_DLCRAM_DED_ERR BIT(4)
+#define LMC_INT_EN_DLCRAM_SEC_ERR BIT(3)
+#define LMC_INT_INTR_DED_ENA BIT(2)
+#define LMC_INT_INTR_SEC_ENA BIT(1)
+#define LMC_INT_INTR_NXM_WR_ENA BIT(0)
+
+#define LMC_INT_ENA_ALL GENMASK(5, 0)
+
+#define LMC_DDR_PLL_CTL 0x258
+#define LMC_DDR_PLL_CTL_DDR4 BIT(29)
+
+#define LMC_CONTROL 0x190
+#define LMC_CONTROL_RDIMM BIT(0)
+
+#define LMC_SCRAM_FADR 0x330
+
+#define LMC_CHAR_MASK0 0x228
+#define LMC_CHAR_MASK2 0x238
+
+#define RING_ENTRIES 8
+
+struct debugfs_entry {
+ const char *name;
+ umode_t mode;
+ const struct file_operations fops;
+};
+
+struct lmc_err_ctx {
+ u64 reg_int;
+ u64 reg_fadr;
+ u64 reg_nxm_fadr;
+ u64 reg_scram_fadr;
+ u64 reg_ecc_synd;
+};
+
+struct thunderx_lmc {
+ void __iomem *regs;
+ struct pci_dev *pdev;
+ struct msix_entry msix_ent;
+
+ atomic_t ecc_int;
+
+ u64 mask0;
+ u64 mask2;
+ u64 parity_test;
+ u64 node;
+
+ int xbits;
+ int bank_width;
+ int pbank_lsb;
+ int dimm_lsb;
+ int rank_lsb;
+ int bank_lsb;
+ int row_lsb;
+ int col_hi_lsb;
+
+ int xor_bank;
+ int l2c_alias;
+
+ struct page *mem;
+
+ struct lmc_err_ctx err_ctx[RING_ENTRIES];
+ unsigned long ring_head;
+ unsigned long ring_tail;
+};
+
+#define ring_pos(pos, size) ((pos) & (size - 1))
+
+#define DEBUGFS_STRUCT(_name, _mode, _write, _read) \
+static struct debugfs_entry debugfs_##_name = { \
+ .name = __stringify(_name), \
+ .mode = VERIFY_OCTAL_PERMISSIONS(_mode), \
+ .fops = { \
+ .open = simple_open, \
+ .write = _write, \
+ .read = _read, \
+ .llseek = generic_file_llseek, \
+ }, \
+}
+
+#define DEBUGFS_FIELD_ATTR(_type, _field) \
+static ssize_t thunderx_##_type##_##_field##_read(struct file *file, \
+ char __user *data, \
+ size_t count, loff_t *ppos) \
+{ \
+ struct thunderx_##_type *pdata = file->private_data; \
+ char buf[20]; \
+ \
+ snprintf(buf, count, "0x%016llx", pdata->_field); \
+ return simple_read_from_buffer(data, count, ppos, \
+ buf, sizeof(buf)); \
+} \
+ \
+static ssize_t thunderx_##_type##_##_field##_write(struct file *file, \
+ const char __user *data, \
+ size_t count, loff_t *ppos) \
+{ \
+ struct thunderx_##_type *pdata = file->private_data; \
+ int res; \
+ \
+ res = kstrtoull_from_user(data, count, 0, &pdata->_field); \
+ \
+ return res ? res : count; \
+} \
+ \
+DEBUGFS_STRUCT(_field, 0600, \
+ thunderx_##_type##_##_field##_write, \
+ thunderx_##_type##_##_field##_read) \
+
+#define DEBUGFS_REG_ATTR(_type, _name, _reg) \
+static ssize_t thunderx_##_type##_##_name##_read(struct file *file, \
+ char __user *data, \
+ size_t count, loff_t *ppos) \
+{ \
+ struct thunderx_##_type *pdata = file->private_data; \
+ char buf[20]; \
+ \
+ sprintf(buf, "0x%016llx", readq(pdata->regs + _reg)); \
+ return simple_read_from_buffer(data, count, ppos, \
+ buf, sizeof(buf)); \
+} \
+ \
+static ssize_t thunderx_##_type##_##_name##_write(struct file *file, \
+ const char __user *data, \
+ size_t count, loff_t *ppos) \
+{ \
+ struct thunderx_##_type *pdata = file->private_data; \
+ u64 val; \
+ int res; \
+ \
+ res = kstrtoull_from_user(data, count, 0, &val); \
+ \
+ if (!res) { \
+ writeq(val, pdata->regs + _reg); \
+ res = count; \
+ } \
+ \
+ return res; \
+} \
+ \
+DEBUGFS_STRUCT(_name, 0600, \
+ thunderx_##_type##_##_name##_write, \
+ thunderx_##_type##_##_name##_read)
+
+#define LMC_DEBUGFS_ENT(_field) DEBUGFS_FIELD_ATTR(lmc, _field)
+
+/*
+ * To get an ECC error injected, the following steps are needed:
+ * - Setup the ECC injection by writing the appropriate parameters:
+ * echo <bit mask value> > /sys/kernel/debug/<device number>/ecc_mask0
+ * echo <bit mask value> > /sys/kernel/debug/<device number>/ecc_mask2
+ * echo 0x802 > /sys/kernel/debug/<device number>/ecc_parity_test
+ * - Do the actual injection:
+ * echo 1 > /sys/kernel/debug/<device number>/inject_ecc
+ */
+static ssize_t thunderx_lmc_inject_int_write(struct file *file,
+ const char __user *data,
+ size_t count, loff_t *ppos)
+{
+ struct thunderx_lmc *lmc = file->private_data;
+ u64 val;
+ int res;
+
+ res = kstrtoull_from_user(data, count, 0, &val);
+
+ if (!res) {
+ /* Trigger the interrupt */
+ writeq(val, lmc->regs + LMC_INT_W1S);
+ res = count;
+ }
+
+ return res;
+}
+
+static ssize_t thunderx_lmc_int_read(struct file *file,
+ char __user *data,
+ size_t count, loff_t *ppos)
+{
+ struct thunderx_lmc *lmc = file->private_data;
+ char buf[20];
+ u64 lmc_int = readq(lmc->regs + LMC_INT);
+
+ snprintf(buf, sizeof(buf), "0x%016llx", lmc_int);
+ return simple_read_from_buffer(data, count, ppos, buf, sizeof(buf));
+}
+
+#define TEST_PATTERN 0xa5
+
+static int inject_ecc_fn(void *arg)
+{
+ struct thunderx_lmc *lmc = arg;
+ uintptr_t addr, phys;
+ unsigned int cline_size = cache_line_size();
+ const unsigned int lines = PAGE_SIZE / cline_size;
+ unsigned int i, cl_idx;
+
+ addr = (uintptr_t)page_address(lmc->mem);
+ phys = (uintptr_t)page_to_phys(lmc->mem);
+
+ cl_idx = (phys & 0x7f) >> 4;
+ lmc->parity_test &= ~(7ULL << 8);
+ lmc->parity_test |= (cl_idx << 8);
+
+ writeq(lmc->mask0, lmc->regs + LMC_CHAR_MASK0);
+ writeq(lmc->mask2, lmc->regs + LMC_CHAR_MASK2);
+ writeq(lmc->parity_test, lmc->regs + LMC_ECC_PARITY_TEST);
+
+ readq(lmc->regs + LMC_CHAR_MASK0);
+ readq(lmc->regs + LMC_CHAR_MASK2);
+ readq(lmc->regs + LMC_ECC_PARITY_TEST);
+
+ for (i = 0; i < lines; i++) {
+ memset((void *)addr, TEST_PATTERN, cline_size);
+ barrier();
+
+ /*
+ * Flush L1 cachelines to the PoC (L2).
+ * This will cause cacheline eviction to the L2.
+ */
+ asm volatile("dc civac, %0\n"
+ "dsb sy\n"
+ : : "r"(addr + i * cline_size));
+ }
+
+ for (i = 0; i < lines; i++) {
+ /*
+ * Flush L2 cachelines to the DRAM.
+ * This will cause cacheline eviction to the DRAM
+ * and ECC corruption according to the masks set.
+ */
+ __asm__ volatile("sys #0,c11,C1,#2, %0\n"
+ : : "r"(phys + i * cline_size));
+ }
+
+ for (i = 0; i < lines; i++) {
+ /*
+ * Invalidate L2 cachelines.
+ * The subsequent load will cause cacheline fetch
+ * from the DRAM and an error interrupt
+ */
+ __asm__ volatile("sys #0,c11,C1,#1, %0"
+ : : "r"(phys + i * cline_size));
+ }
+
+ for (i = 0; i < lines; i++) {
+ /*
+ * Invalidate L1 cachelines.
+ * The subsequent load will cause cacheline fetch
+ * from the L2 and/or DRAM
+ */
+ asm volatile("dc ivac, %0\n"
+ "dsb sy\n"
+ : : "r"(addr + i * cline_size));
+ }
+
+ return 0;
+}
+
+static ssize_t thunderx_lmc_inject_ecc_write(struct file *file,
+ const char __user *data,
+ size_t count, loff_t *ppos)
+{
+ struct thunderx_lmc *lmc = file->private_data;
+
+ unsigned int cline_size = cache_line_size();
+
+ u8 tmp[cline_size];
+ void __iomem *addr;
+ unsigned int offs, timeout = 100000;
+
+ atomic_set(&lmc->ecc_int, 0);
+
+ lmc->mem = alloc_pages_node(lmc->node, GFP_KERNEL, 0);
+
+ if (!lmc->mem)
+ return -ENOMEM;
+
+ addr = page_address(lmc->mem);
+
+ while (!atomic_read(&lmc->ecc_int) && timeout--) {
+ stop_machine(inject_ecc_fn, lmc, NULL);
+
+ for (offs = 0; offs < PAGE_SIZE; offs += sizeof(tmp)) {
+ /*
+ * Do a load from the previously rigged location
+ * This should generate an error interrupt.
+ */
+ memcpy(tmp, addr + offs, cline_size);
+ asm volatile("dsb ld\n");
+ }
+ }
+
+ __free_pages(lmc->mem, 0);
+
+ return count;
+}
+
+LMC_DEBUGFS_ENT(mask0);
+LMC_DEBUGFS_ENT(mask2);
+LMC_DEBUGFS_ENT(parity_test);
+
+DEBUGFS_STRUCT(inject_int, 0200, thunderx_lmc_inject_int_write, NULL);
+DEBUGFS_STRUCT(inject_ecc, 0200, thunderx_lmc_inject_ecc_write, NULL);
+DEBUGFS_STRUCT(int_w1c, 0400, NULL, thunderx_lmc_int_read);
+
+struct debugfs_entry *lmc_dfs_ents[] = {
+ &debugfs_mask0,
+ &debugfs_mask2,
+ &debugfs_parity_test,
+ &debugfs_inject_ecc,
+ &debugfs_inject_int,
+ &debugfs_int_w1c,
+};
+
+static int thunderx_create_debugfs_nodes(struct dentry *parent,
+ struct debugfs_entry *attrs[],
+ void *data,
+ size_t num)
+{
+ int i;
+ struct dentry *ent;
+
+ if (!IS_ENABLED(CONFIG_EDAC_DEBUG))
+ return 0;
+
+ if (!parent)
+ return -ENOENT;
+
+ for (i = 0; i < num; i++) {
+ ent = edac_debugfs_create_file(attrs[i]->name, attrs[i]->mode,
+ parent, data, &attrs[i]->fops);
+
+ if (!ent)
+ break;
+ }
+
+ return i;
+}
+
+static phys_addr_t thunderx_faddr_to_phys(u64 faddr, struct thunderx_lmc *lmc)
+{
+ phys_addr_t addr = 0;
+ int bank, xbits;
+
+ addr |= lmc->node << 40;
+ addr |= LMC_FADR_FDIMM(faddr) << lmc->dimm_lsb;
+ addr |= LMC_FADR_FBUNK(faddr) << lmc->rank_lsb;
+ addr |= LMC_FADR_FROW(faddr) << lmc->row_lsb;
+ addr |= (LMC_FADR_FCOL(faddr) >> 4) << lmc->col_hi_lsb;
+
+ bank = LMC_FADR_FBANK(faddr) << lmc->bank_lsb;
+
+ if (lmc->xor_bank)
+ bank ^= get_bits(addr, 12 + lmc->xbits, lmc->bank_width);
+
+ addr |= bank << lmc->bank_lsb;
+
+ xbits = PCI_FUNC(lmc->pdev->devfn);
+
+ if (lmc->l2c_alias)
+ xbits ^= get_bits(addr, 20, lmc->xbits) ^
+ get_bits(addr, 12, lmc->xbits);
+
+ addr |= xbits << 7;
+
+ return addr;
+}
+
+static unsigned int thunderx_get_num_lmcs(unsigned int node)
+{
+ unsigned int number = 0;
+ struct pci_dev *pdev = NULL;
+
+ do {
+ pdev = pci_get_device(PCI_VENDOR_ID_CAVIUM,
+ PCI_DEVICE_ID_THUNDER_LMC,
+ pdev);
+ if (pdev) {
+#ifdef CONFIG_NUMA
+ if (pdev->dev.numa_node == node)
+ number++;
+#else
+ number++;
+#endif
+ }
+ } while (pdev);
+
+ return number;
+}
+
+#define LMC_MESSAGE_SIZE 120
+#define LMC_OTHER_SIZE (50 * ARRAY_SIZE(lmc_errors))
+
+static irqreturn_t thunderx_lmc_err_isr(int irq, void *dev_id)
+{
+ struct mem_ctl_info *mci = dev_id;
+ struct thunderx_lmc *lmc = mci->pvt_info;
+
+ unsigned long head = ring_pos(lmc->ring_head, ARRAY_SIZE(lmc->err_ctx));
+ struct lmc_err_ctx *ctx = &lmc->err_ctx[head];
+
+ writeq(0, lmc->regs + LMC_CHAR_MASK0);
+ writeq(0, lmc->regs + LMC_CHAR_MASK2);
+ writeq(0x2, lmc->regs + LMC_ECC_PARITY_TEST);
+
+ ctx->reg_int = readq(lmc->regs + LMC_INT);
+ ctx->reg_fadr = readq(lmc->regs + LMC_FADR);
+ ctx->reg_nxm_fadr = readq(lmc->regs + LMC_NXM_FADR);
+ ctx->reg_scram_fadr = readq(lmc->regs + LMC_SCRAM_FADR);
+ ctx->reg_ecc_synd = readq(lmc->regs + LMC_ECC_SYND);
+
+ lmc->ring_head++;
+
+ atomic_set(&lmc->ecc_int, 1);
+
+ /* Clear the interrupt */
+ writeq(ctx->reg_int, lmc->regs + LMC_INT);
+
+ return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t thunderx_lmc_threaded_isr(int irq, void *dev_id)
+{
+ struct mem_ctl_info *mci = dev_id;
+ struct thunderx_lmc *lmc = mci->pvt_info;
+ phys_addr_t phys_addr;
+
+ unsigned long tail;
+ struct lmc_err_ctx *ctx;
+
+ irqreturn_t ret = IRQ_NONE;
+
+ char *msg;
+ char *other;
+
+ msg = kmalloc(LMC_MESSAGE_SIZE, GFP_KERNEL);
+ other = kmalloc(LMC_OTHER_SIZE, GFP_KERNEL);
+
+ if (!msg || !other)
+ goto err_free;
+
+ while (CIRC_CNT(lmc->ring_head, lmc->ring_tail,
+ ARRAY_SIZE(lmc->err_ctx))) {
+ tail = ring_pos(lmc->ring_tail, ARRAY_SIZE(lmc->err_ctx));
+
+ ctx = &lmc->err_ctx[tail];
+
+ dev_dbg(&lmc->pdev->dev, "LMC_INT: %016llx\n",
+ ctx->reg_int);
+ dev_dbg(&lmc->pdev->dev, "LMC_FADR: %016llx\n",
+ ctx->reg_fadr);
+ dev_dbg(&lmc->pdev->dev, "LMC_NXM_FADR: %016llx\n",
+ ctx->reg_nxm_fadr);
+ dev_dbg(&lmc->pdev->dev, "LMC_SCRAM_FADR: %016llx\n",
+ ctx->reg_scram_fadr);
+ dev_dbg(&lmc->pdev->dev, "LMC_ECC_SYND: %016llx\n",
+ ctx->reg_ecc_synd);
+
+ snprintf(msg, LMC_MESSAGE_SIZE,
+ "DIMM %lld rank %lld bank %lld row %lld col %lld",
+ LMC_FADR_FDIMM(ctx->reg_scram_fadr),
+ LMC_FADR_FBUNK(ctx->reg_scram_fadr),
+ LMC_FADR_FBANK(ctx->reg_scram_fadr),
+ LMC_FADR_FROW(ctx->reg_scram_fadr),
+ LMC_FADR_FCOL(ctx->reg_scram_fadr));
+
+ decode_register(other, LMC_OTHER_SIZE, lmc_errors,
+ ctx->reg_int);
+
+ phys_addr = thunderx_faddr_to_phys(ctx->reg_fadr, lmc);
+
+ if (ctx->reg_int & LMC_INT_UE)
+ edac_mc_handle_error(HW_EVENT_ERR_UNCORRECTED, mci, 1,
+ phys_to_pfn(phys_addr),
+ offset_in_page(phys_addr),
+ 0, -1, -1, -1, msg, other);
+ else if (ctx->reg_int & LMC_INT_CE)
+ edac_mc_handle_error(HW_EVENT_ERR_CORRECTED, mci, 1,
+ phys_to_pfn(phys_addr),
+ offset_in_page(phys_addr),
+ 0, -1, -1, -1, msg, other);
+
+ lmc->ring_tail++;
+ }
+
+ ret = IRQ_HANDLED;
+
+err_free:
+ kfree(msg);
+ kfree(other);
+
+ return ret;
+}
+
+#ifdef CONFIG_PM
+static int thunderx_lmc_suspend(struct pci_dev *pdev, pm_message_t state)
+{
+ pci_save_state(pdev);
+ pci_disable_device(pdev);
+
+ pci_set_power_state(pdev, pci_choose_state(pdev, state));
+
+ return 0;
+}
+
+static int thunderx_lmc_resume(struct pci_dev *pdev)
+{
+ pci_set_power_state(pdev, PCI_D0);
+ pci_enable_wake(pdev, PCI_D0, 0);
+ pci_restore_state(pdev);
+
+ return 0;
+}
+#endif
+
+static const struct pci_device_id thunderx_lmc_pci_tbl[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVICE_ID_THUNDER_LMC) },
+ { 0, },
+};
+
+static inline int pci_dev_to_mc_idx(struct pci_dev *pdev)
+{
+ int node = dev_to_node(&pdev->dev);
+ int ret = PCI_FUNC(pdev->devfn);
+
+ ret += max(node, 0) << 3;
+
+ return ret;
+}
+
+static int thunderx_lmc_probe(struct pci_dev *pdev,
+ const struct pci_device_id *id)
+{
+ struct thunderx_lmc *lmc;
+ struct edac_mc_layer layer;
+ struct mem_ctl_info *mci;
+ u64 lmc_control, lmc_ddr_pll_ctl, lmc_config;
+ int ret;
+ u64 lmc_int;
+ void *l2c_ioaddr;
+
+ layer.type = EDAC_MC_LAYER_SLOT;
+ layer.size = 2;
+ layer.is_virt_csrow = false;
+
+ ret = pcim_enable_device(pdev);
+ if (ret) {
+ dev_err(&pdev->dev, "Cannot enable PCI device: %d\n", ret);
+ return ret;
+ }
+
+ ret = pcim_iomap_regions(pdev, BIT(0), "thunderx_lmc");
+ if (ret) {
+ dev_err(&pdev->dev, "Cannot map PCI resources: %d\n", ret);
+ return ret;
+ }
+
+ mci = edac_mc_alloc(pci_dev_to_mc_idx(pdev), 1, &layer,
+ sizeof(struct thunderx_lmc));
+ if (!mci)
+ return -ENOMEM;
+
+ mci->pdev = &pdev->dev;
+ lmc = mci->pvt_info;
+
+ pci_set_drvdata(pdev, mci);
+
+ lmc->regs = pcim_iomap_table(pdev)[0];
+
+ lmc_control = readq(lmc->regs + LMC_CONTROL);
+ lmc_ddr_pll_ctl = readq(lmc->regs + LMC_DDR_PLL_CTL);
+ lmc_config = readq(lmc->regs + LMC_CONFIG);
+
+ if (lmc_control & LMC_CONTROL_RDIMM) {
+ mci->mtype_cap = FIELD_GET(LMC_DDR_PLL_CTL_DDR4,
+ lmc_ddr_pll_ctl) ?
+ MEM_RDDR4 : MEM_RDDR3;
+ } else {
+ mci->mtype_cap = FIELD_GET(LMC_DDR_PLL_CTL_DDR4,
+ lmc_ddr_pll_ctl) ?
+ MEM_DDR4 : MEM_DDR3;
+ }
+
+ mci->edac_ctl_cap = EDAC_FLAG_NONE | EDAC_FLAG_SECDED;
+ mci->edac_cap = EDAC_FLAG_SECDED;
+
+ mci->mod_name = "thunderx-lmc";
+ mci->mod_ver = "1";
+ mci->ctl_name = "thunderx-lmc";
+ mci->dev_name = dev_name(&pdev->dev);
+ mci->scrub_mode = SCRUB_NONE;
+
+ lmc->pdev = pdev;
+ lmc->msix_ent.entry = 0;
+
+ lmc->ring_head = 0;
+ lmc->ring_tail = 0;
+
+ ret = pci_enable_msix_exact(pdev, &lmc->msix_ent, 1);
+ if (ret) {
+ dev_err(&pdev->dev, "Cannot enable interrupt: %d\n", ret);
+ goto err_free;
+ }
+
+ ret = devm_request_threaded_irq(&pdev->dev, lmc->msix_ent.vector,
+ thunderx_lmc_err_isr,
+ thunderx_lmc_threaded_isr, 0,
+ "[EDAC] ThunderX LMC", mci);
+ if (ret) {
+ dev_err(&pdev->dev, "Cannot set ISR: %d\n", ret);
+ goto err_free;
+ }
+
+ lmc->node = FIELD_GET(THUNDERX_NODE, pci_resource_start(pdev, 0));
+
+ lmc->xbits = thunderx_get_num_lmcs(lmc->node) >> 1;
+ lmc->bank_width = (FIELD_GET(LMC_DDR_PLL_CTL_DDR4, lmc_ddr_pll_ctl) &&
+ FIELD_GET(LMC_CONFIG_BG2, lmc_config)) ? 4 : 3;
+
+ lmc->pbank_lsb = (lmc_config >> 5) & 0xf;
+ lmc->dimm_lsb = 28 + lmc->pbank_lsb + lmc->xbits;
+ lmc->rank_lsb = lmc->dimm_lsb;
+ lmc->rank_lsb -= FIELD_GET(LMC_CONFIG_RANK_ENA, lmc_config) ? 1 : 0;
+ lmc->bank_lsb = 7 + lmc->xbits;
+ lmc->row_lsb = 14 + LMC_CONFIG_ROW_LSB(lmc_config) + lmc->xbits;
+
+ lmc->col_hi_lsb = lmc->bank_lsb + lmc->bank_width;
+
+ lmc->xor_bank = lmc_control & LMC_CONTROL_XOR_BANK;
+
+ l2c_ioaddr = ioremap(L2C_CTL | FIELD_PREP(THUNDERX_NODE, lmc->node),
+ PAGE_SIZE);
+
+ if (!l2c_ioaddr) {
+ dev_err(&pdev->dev, "Cannot map L2C_CTL\n");
+ goto err_free;
+ }
+
+ lmc->l2c_alias = !(readq(l2c_ioaddr) & L2C_CTL_DISIDXALIAS);
+
+ iounmap(l2c_ioaddr);
+
+ ret = edac_mc_add_mc(mci);
+ if (ret) {
+ dev_err(&pdev->dev, "Cannot add the MC: %d\n", ret);
+ goto err_free;
+ }
+
+ lmc_int = readq(lmc->regs + LMC_INT);
+ writeq(lmc_int, lmc->regs + LMC_INT);
+
+ writeq(LMC_INT_ENA_ALL, lmc->regs + LMC_INT_ENA_W1S);
+
+ if (IS_ENABLED(CONFIG_EDAC_DEBUG)) {
+ ret = thunderx_create_debugfs_nodes(mci->debugfs,
+ lmc_dfs_ents,
+ lmc,
+ ARRAY_SIZE(lmc_dfs_ents));
+
+ if (ret != ARRAY_SIZE(lmc_dfs_ents)) {
+ dev_warn(&pdev->dev, "Error creating debugfs entries: %d%s\n",
+ ret, ret >= 0 ? " created" : "");
+ }
+ }
+
+ return 0;
+
+err_free:
+ pci_set_drvdata(pdev, NULL);
+ edac_mc_free(mci);
+
+ return ret;
+}
+
+static void thunderx_lmc_remove(struct pci_dev *pdev)
+{
+ struct mem_ctl_info *mci = pci_get_drvdata(pdev);
+ struct thunderx_lmc *lmc = mci->pvt_info;
+
+ writeq(LMC_INT_ENA_ALL, lmc->regs + LMC_INT_ENA_W1C);
+
+ edac_mc_del_mc(&pdev->dev);
+ edac_mc_free(mci);
+}
+
+MODULE_DEVICE_TABLE(pci, thunderx_lmc_pci_tbl);
+
+static struct pci_driver thunderx_lmc_driver = {
+ .name = "thunderx_lmc_edac",
+ .probe = thunderx_lmc_probe,
+ .remove = thunderx_lmc_remove,
+#ifdef CONFIG_PM
+ .suspend = thunderx_lmc_suspend,
+ .resume = thunderx_lmc_resume,
+#endif
+ .id_table = thunderx_lmc_pci_tbl,
+};
+
+/*---------------------- OCX driver ---------------------------------*/
+
+#define PCI_DEVICE_ID_THUNDER_OCX 0xa013
+
+#define OCX_LINK_INTS 3
+#define OCX_INTS (OCX_LINK_INTS + 1)
+#define OCX_RX_LANES 24
+#define OCX_RX_LANE_STATS 15
+
+#define OCX_COM_INT 0x100
+#define OCX_COM_INT_W1S 0x108
+#define OCX_COM_INT_ENA_W1S 0x110
+#define OCX_COM_INT_ENA_W1C 0x118
+
+#define OCX_COM_IO_BADID BIT(54)
+#define OCX_COM_MEM_BADID BIT(53)
+#define OCX_COM_COPR_BADID BIT(52)
+#define OCX_COM_WIN_REQ_BADID BIT(51)
+#define OCX_COM_WIN_REQ_TOUT BIT(50)
+#define OCX_COM_RX_LANE GENMASK(23, 0)
+
+#define OCX_COM_INT_CE (OCX_COM_IO_BADID | \
+ OCX_COM_MEM_BADID | \
+ OCX_COM_COPR_BADID | \
+ OCX_COM_WIN_REQ_BADID | \
+ OCX_COM_WIN_REQ_TOUT)
+
+static const struct error_descr ocx_com_errors[] = {
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_COM_IO_BADID,
+ .descr = "Invalid IO transaction node ID",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_COM_MEM_BADID,
+ .descr = "Invalid memory transaction node ID",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_COM_COPR_BADID,
+ .descr = "Invalid coprocessor transaction node ID",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_COM_WIN_REQ_BADID,
+ .descr = "Invalid SLI transaction node ID",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_COM_WIN_REQ_TOUT,
+ .descr = "Window/core request timeout",
+ },
+ {0, 0, NULL},
+};
+
+#define OCX_COM_LINKX_INT(x) (0x120 + (x) * 8)
+#define OCX_COM_LINKX_INT_W1S(x) (0x140 + (x) * 8)
+#define OCX_COM_LINKX_INT_ENA_W1S(x) (0x160 + (x) * 8)
+#define OCX_COM_LINKX_INT_ENA_W1C(x) (0x180 + (x) * 8)
+
+#define OCX_COM_LINK_BAD_WORD BIT(13)
+#define OCX_COM_LINK_ALIGN_FAIL BIT(12)
+#define OCX_COM_LINK_ALIGN_DONE BIT(11)
+#define OCX_COM_LINK_UP BIT(10)
+#define OCX_COM_LINK_STOP BIT(9)
+#define OCX_COM_LINK_BLK_ERR BIT(8)
+#define OCX_COM_LINK_REINIT BIT(7)
+#define OCX_COM_LINK_LNK_DATA BIT(6)
+#define OCX_COM_LINK_RXFIFO_DBE BIT(5)
+#define OCX_COM_LINK_RXFIFO_SBE BIT(4)
+#define OCX_COM_LINK_TXFIFO_DBE BIT(3)
+#define OCX_COM_LINK_TXFIFO_SBE BIT(2)
+#define OCX_COM_LINK_REPLAY_DBE BIT(1)
+#define OCX_COM_LINK_REPLAY_SBE BIT(0)
+
+static const struct error_descr ocx_com_link_errors[] = {
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_COM_LINK_REPLAY_SBE,
+ .descr = "Replay buffer single-bit error",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_COM_LINK_TXFIFO_SBE,
+ .descr = "TX FIFO single-bit error",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_COM_LINK_RXFIFO_SBE,
+ .descr = "RX FIFO single-bit error",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_COM_LINK_BLK_ERR,
+ .descr = "Block code error",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_COM_LINK_ALIGN_FAIL,
+ .descr = "Link alignment failure",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_COM_LINK_BAD_WORD,
+ .descr = "Bad code word",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = OCX_COM_LINK_REPLAY_DBE,
+ .descr = "Replay buffer double-bit error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = OCX_COM_LINK_TXFIFO_DBE,
+ .descr = "TX FIFO double-bit error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = OCX_COM_LINK_RXFIFO_DBE,
+ .descr = "RX FIFO double-bit error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = OCX_COM_LINK_STOP,
+ .descr = "Link stopped",
+ },
+ {0, 0, NULL},
+};
+
+#define OCX_COM_LINK_INT_UE (OCX_COM_LINK_REPLAY_DBE | \
+ OCX_COM_LINK_TXFIFO_DBE | \
+ OCX_COM_LINK_RXFIFO_DBE | \
+ OCX_COM_LINK_STOP)
+
+#define OCX_COM_LINK_INT_CE (OCX_COM_LINK_REPLAY_SBE | \
+ OCX_COM_LINK_TXFIFO_SBE | \
+ OCX_COM_LINK_RXFIFO_SBE | \
+ OCX_COM_LINK_BLK_ERR | \
+ OCX_COM_LINK_ALIGN_FAIL | \
+ OCX_COM_LINK_BAD_WORD)
+
+#define OCX_LNE_INT(x) (0x8018 + (x) * 0x100)
+#define OCX_LNE_INT_EN(x) (0x8020 + (x) * 0x100)
+#define OCX_LNE_BAD_CNT(x) (0x8028 + (x) * 0x100)
+#define OCX_LNE_CFG(x) (0x8000 + (x) * 0x100)
+#define OCX_LNE_STAT(x, y) (0x8040 + (x) * 0x100 + (y) * 8)
+
+#define OCX_LNE_CFG_RX_BDRY_LOCK_DIS BIT(8)
+#define OCX_LNE_CFG_RX_STAT_WRAP_DIS BIT(2)
+#define OCX_LNE_CFG_RX_STAT_RDCLR BIT(1)
+#define OCX_LNE_CFG_RX_STAT_ENA BIT(0)
+
+
+#define OCX_LANE_BAD_64B67B BIT(8)
+#define OCX_LANE_DSKEW_FIFO_OVFL BIT(5)
+#define OCX_LANE_SCRM_SYNC_LOSS BIT(4)
+#define OCX_LANE_UKWN_CNTL_WORD BIT(3)
+#define OCX_LANE_CRC32_ERR BIT(2)
+#define OCX_LANE_BDRY_SYNC_LOSS BIT(1)
+#define OCX_LANE_SERDES_LOCK_LOSS BIT(0)
+
+#define OCX_COM_LANE_INT_UE (0)
+#define OCX_COM_LANE_INT_CE (OCX_LANE_SERDES_LOCK_LOSS | \
+ OCX_LANE_BDRY_SYNC_LOSS | \
+ OCX_LANE_CRC32_ERR | \
+ OCX_LANE_UKWN_CNTL_WORD | \
+ OCX_LANE_SCRM_SYNC_LOSS | \
+ OCX_LANE_DSKEW_FIFO_OVFL | \
+ OCX_LANE_BAD_64B67B)
+
+static const struct error_descr ocx_lane_errors[] = {
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_LANE_SERDES_LOCK_LOSS,
+ .descr = "RX SerDes lock lost",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_LANE_BDRY_SYNC_LOSS,
+ .descr = "RX word boundary lost",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_LANE_CRC32_ERR,
+ .descr = "CRC32 error",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_LANE_UKWN_CNTL_WORD,
+ .descr = "Unknown control word",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_LANE_SCRM_SYNC_LOSS,
+ .descr = "Scrambler synchronization lost",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_LANE_DSKEW_FIFO_OVFL,
+ .descr = "RX deskew FIFO overflow",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = OCX_LANE_BAD_64B67B,
+ .descr = "Bad 64B/67B codeword",
+ },
+ {0, 0, NULL},
+};
+
+#define OCX_LNE_INT_ENA_ALL (GENMASK(9, 8) | GENMASK(6, 0))
+#define OCX_COM_INT_ENA_ALL (GENMASK(54, 50) | GENMASK(23, 0))
+#define OCX_COM_LINKX_INT_ENA_ALL (GENMASK(13, 12) | \
+ GENMASK(9, 7) | GENMASK(5, 0))
+
+#define OCX_TLKX_ECC_CTL(x) (0x10018 + (x) * 0x2000)
+#define OCX_RLKX_ECC_CTL(x) (0x18018 + (x) * 0x2000)
+
+struct ocx_com_err_ctx {
+ u64 reg_com_int;
+ u64 reg_lane_int[OCX_RX_LANES];
+ u64 reg_lane_stat11[OCX_RX_LANES];
+};
+
+struct ocx_link_err_ctx {
+ u64 reg_com_link_int;
+ int link;
+};
+
+struct thunderx_ocx {
+ void __iomem *regs;
+ int com_link;
+ struct pci_dev *pdev;
+ struct edac_device_ctl_info *edac_dev;
+
+ struct dentry *debugfs;
+ struct msix_entry msix_ent[OCX_INTS];
+
+ struct ocx_com_err_ctx com_err_ctx[RING_ENTRIES];
+ struct ocx_link_err_ctx link_err_ctx[RING_ENTRIES];
+
+ unsigned long com_ring_head;
+ unsigned long com_ring_tail;
+
+ unsigned long link_ring_head;
+ unsigned long link_ring_tail;
+};
+
+#define OCX_MESSAGE_SIZE SZ_1K
+#define OCX_OTHER_SIZE (50 * ARRAY_SIZE(ocx_com_link_errors))
+
+/* This handler is threaded */
+static irqreturn_t thunderx_ocx_com_isr(int irq, void *irq_id)
+{
+ struct msix_entry *msix = irq_id;
+ struct thunderx_ocx *ocx = container_of(msix, struct thunderx_ocx,
+ msix_ent[msix->entry]);
+
+ int lane;
+ unsigned long head = ring_pos(ocx->com_ring_head,
+ ARRAY_SIZE(ocx->com_err_ctx));
+ struct ocx_com_err_ctx *ctx = &ocx->com_err_ctx[head];
+
+ ctx->reg_com_int = readq(ocx->regs + OCX_COM_INT);
+
+ for (lane = 0; lane < OCX_RX_LANES; lane++) {
+ ctx->reg_lane_int[lane] =
+ readq(ocx->regs + OCX_LNE_INT(lane));
+ ctx->reg_lane_stat11[lane] =
+ readq(ocx->regs + OCX_LNE_STAT(lane, 11));
+
+ writeq(ctx->reg_lane_int[lane], ocx->regs + OCX_LNE_INT(lane));
+ }
+
+ writeq(ctx->reg_com_int, ocx->regs + OCX_COM_INT);
+
+ ocx->com_ring_head++;
+
+ return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t thunderx_ocx_com_threaded_isr(int irq, void *irq_id)
+{
+ struct msix_entry *msix = irq_id;
+ struct thunderx_ocx *ocx = container_of(msix, struct thunderx_ocx,
+ msix_ent[msix->entry]);
+
+ irqreturn_t ret = IRQ_NONE;
+
+ unsigned long tail;
+ struct ocx_com_err_ctx *ctx;
+ int lane;
+ char *msg;
+ char *other;
+
+ msg = kmalloc(OCX_MESSAGE_SIZE, GFP_KERNEL);
+ other = kmalloc(OCX_OTHER_SIZE, GFP_KERNEL);
+
+ if (!msg || !other)
+ goto err_free;
+
+ while (CIRC_CNT(ocx->com_ring_head, ocx->com_ring_tail,
+ ARRAY_SIZE(ocx->com_err_ctx))) {
+ tail = ring_pos(ocx->com_ring_tail,
+ ARRAY_SIZE(ocx->com_err_ctx));
+ ctx = &ocx->com_err_ctx[tail];
+
+ snprintf(msg, OCX_MESSAGE_SIZE, "%s: OCX_COM_INT: %016llx",
+ ocx->edac_dev->ctl_name, ctx->reg_com_int);
+
+ decode_register(other, OCX_OTHER_SIZE,
+ ocx_com_errors, ctx->reg_com_int);
+
+ strncat(msg, other, OCX_MESSAGE_SIZE);
+
+ for (lane = 0; lane < OCX_RX_LANES; lane++)
+ if (ctx->reg_com_int & BIT(lane)) {
+ snprintf(other, OCX_OTHER_SIZE,
+ "\n\tOCX_LNE_INT[%02d]: %016llx OCX_LNE_STAT11[%02d]: %016llx",
+ lane, ctx->reg_lane_int[lane],
+ lane, ctx->reg_lane_stat11[lane]);
+
+ strncat(msg, other, OCX_MESSAGE_SIZE);
+
+ decode_register(other, OCX_OTHER_SIZE,
+ ocx_lane_errors,
+ ctx->reg_lane_int[lane]);
+ strncat(msg, other, OCX_MESSAGE_SIZE);
+ }
+
+ if (ctx->reg_com_int & OCX_COM_INT_CE)
+ edac_device_handle_ce(ocx->edac_dev, 0, 0, msg);
+
+ ocx->com_ring_tail++;
+ }
+
+ ret = IRQ_HANDLED;
+
+err_free:
+ kfree(other);
+ kfree(msg);
+
+ return ret;
+}
+
+static irqreturn_t thunderx_ocx_lnk_isr(int irq, void *irq_id)
+{
+ struct msix_entry *msix = irq_id;
+ struct thunderx_ocx *ocx = container_of(msix, struct thunderx_ocx,
+ msix_ent[msix->entry]);
+ unsigned long head = ring_pos(ocx->link_ring_head,
+ ARRAY_SIZE(ocx->link_err_ctx));
+ struct ocx_link_err_ctx *ctx = &ocx->link_err_ctx[head];
+
+ ctx->link = msix->entry;
+ ctx->reg_com_link_int = readq(ocx->regs + OCX_COM_LINKX_INT(ctx->link));
+
+ writeq(ctx->reg_com_link_int, ocx->regs + OCX_COM_LINKX_INT(ctx->link));
+
+ ocx->link_ring_head++;
+
+ return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t thunderx_ocx_lnk_threaded_isr(int irq, void *irq_id)
+{
+ struct msix_entry *msix = irq_id;
+ struct thunderx_ocx *ocx = container_of(msix, struct thunderx_ocx,
+ msix_ent[msix->entry]);
+ irqreturn_t ret = IRQ_NONE;
+ unsigned long tail;
+ struct ocx_link_err_ctx *ctx;
+
+ char *msg;
+ char *other;
+
+ msg = kmalloc(OCX_MESSAGE_SIZE, GFP_KERNEL);
+ other = kmalloc(OCX_OTHER_SIZE, GFP_KERNEL);
+
+ if (!msg || !other)
+ goto err_free;
+
+ while (CIRC_CNT(ocx->link_ring_head, ocx->link_ring_tail,
+ ARRAY_SIZE(ocx->link_err_ctx))) {
+ tail = ring_pos(ocx->link_ring_head,
+ ARRAY_SIZE(ocx->link_err_ctx));
+
+ ctx = &ocx->link_err_ctx[tail];
+
+ snprintf(msg, OCX_MESSAGE_SIZE,
+ "%s: OCX_COM_LINK_INT[%d]: %016llx",
+ ocx->edac_dev->ctl_name,
+ ctx->link, ctx->reg_com_link_int);
+
+ decode_register(other, OCX_OTHER_SIZE,
+ ocx_com_link_errors, ctx->reg_com_link_int);
+
+ strncat(msg, other, OCX_MESSAGE_SIZE);
+
+ if (ctx->reg_com_link_int & OCX_COM_LINK_INT_UE)
+ edac_device_handle_ue(ocx->edac_dev, 0, 0, msg);
+ else if (ctx->reg_com_link_int & OCX_COM_LINK_INT_CE)
+ edac_device_handle_ce(ocx->edac_dev, 0, 0, msg);
+
+ ocx->link_ring_tail++;
+ }
+
+ ret = IRQ_HANDLED;
+err_free:
+ kfree(other);
+ kfree(msg);
+
+ return ret;
+}
+
+#define OCX_DEBUGFS_ATTR(_name, _reg) DEBUGFS_REG_ATTR(ocx, _name, _reg)
+
+OCX_DEBUGFS_ATTR(tlk0_ecc_ctl, OCX_TLKX_ECC_CTL(0));
+OCX_DEBUGFS_ATTR(tlk1_ecc_ctl, OCX_TLKX_ECC_CTL(1));
+OCX_DEBUGFS_ATTR(tlk2_ecc_ctl, OCX_TLKX_ECC_CTL(2));
+
+OCX_DEBUGFS_ATTR(rlk0_ecc_ctl, OCX_RLKX_ECC_CTL(0));
+OCX_DEBUGFS_ATTR(rlk1_ecc_ctl, OCX_RLKX_ECC_CTL(1));
+OCX_DEBUGFS_ATTR(rlk2_ecc_ctl, OCX_RLKX_ECC_CTL(2));
+
+OCX_DEBUGFS_ATTR(com_link0_int, OCX_COM_LINKX_INT_W1S(0));
+OCX_DEBUGFS_ATTR(com_link1_int, OCX_COM_LINKX_INT_W1S(1));
+OCX_DEBUGFS_ATTR(com_link2_int, OCX_COM_LINKX_INT_W1S(2));
+
+OCX_DEBUGFS_ATTR(lne00_badcnt, OCX_LNE_BAD_CNT(0));
+OCX_DEBUGFS_ATTR(lne01_badcnt, OCX_LNE_BAD_CNT(1));
+OCX_DEBUGFS_ATTR(lne02_badcnt, OCX_LNE_BAD_CNT(2));
+OCX_DEBUGFS_ATTR(lne03_badcnt, OCX_LNE_BAD_CNT(3));
+OCX_DEBUGFS_ATTR(lne04_badcnt, OCX_LNE_BAD_CNT(4));
+OCX_DEBUGFS_ATTR(lne05_badcnt, OCX_LNE_BAD_CNT(5));
+OCX_DEBUGFS_ATTR(lne06_badcnt, OCX_LNE_BAD_CNT(6));
+OCX_DEBUGFS_ATTR(lne07_badcnt, OCX_LNE_BAD_CNT(7));
+
+OCX_DEBUGFS_ATTR(lne08_badcnt, OCX_LNE_BAD_CNT(8));
+OCX_DEBUGFS_ATTR(lne09_badcnt, OCX_LNE_BAD_CNT(9));
+OCX_DEBUGFS_ATTR(lne10_badcnt, OCX_LNE_BAD_CNT(10));
+OCX_DEBUGFS_ATTR(lne11_badcnt, OCX_LNE_BAD_CNT(11));
+OCX_DEBUGFS_ATTR(lne12_badcnt, OCX_LNE_BAD_CNT(12));
+OCX_DEBUGFS_ATTR(lne13_badcnt, OCX_LNE_BAD_CNT(13));
+OCX_DEBUGFS_ATTR(lne14_badcnt, OCX_LNE_BAD_CNT(14));
+OCX_DEBUGFS_ATTR(lne15_badcnt, OCX_LNE_BAD_CNT(15));
+
+OCX_DEBUGFS_ATTR(lne16_badcnt, OCX_LNE_BAD_CNT(16));
+OCX_DEBUGFS_ATTR(lne17_badcnt, OCX_LNE_BAD_CNT(17));
+OCX_DEBUGFS_ATTR(lne18_badcnt, OCX_LNE_BAD_CNT(18));
+OCX_DEBUGFS_ATTR(lne19_badcnt, OCX_LNE_BAD_CNT(19));
+OCX_DEBUGFS_ATTR(lne20_badcnt, OCX_LNE_BAD_CNT(20));
+OCX_DEBUGFS_ATTR(lne21_badcnt, OCX_LNE_BAD_CNT(21));
+OCX_DEBUGFS_ATTR(lne22_badcnt, OCX_LNE_BAD_CNT(22));
+OCX_DEBUGFS_ATTR(lne23_badcnt, OCX_LNE_BAD_CNT(23));
+
+OCX_DEBUGFS_ATTR(com_int, OCX_COM_INT_W1S);
+
+struct debugfs_entry *ocx_dfs_ents[] = {
+ &debugfs_tlk0_ecc_ctl,
+ &debugfs_tlk1_ecc_ctl,
+ &debugfs_tlk2_ecc_ctl,
+
+ &debugfs_rlk0_ecc_ctl,
+ &debugfs_rlk1_ecc_ctl,
+ &debugfs_rlk2_ecc_ctl,
+
+ &debugfs_com_link0_int,
+ &debugfs_com_link1_int,
+ &debugfs_com_link2_int,
+
+ &debugfs_lne00_badcnt,
+ &debugfs_lne01_badcnt,
+ &debugfs_lne02_badcnt,
+ &debugfs_lne03_badcnt,
+ &debugfs_lne04_badcnt,
+ &debugfs_lne05_badcnt,
+ &debugfs_lne06_badcnt,
+ &debugfs_lne07_badcnt,
+ &debugfs_lne08_badcnt,
+ &debugfs_lne09_badcnt,
+ &debugfs_lne10_badcnt,
+ &debugfs_lne11_badcnt,
+ &debugfs_lne12_badcnt,
+ &debugfs_lne13_badcnt,
+ &debugfs_lne14_badcnt,
+ &debugfs_lne15_badcnt,
+ &debugfs_lne16_badcnt,
+ &debugfs_lne17_badcnt,
+ &debugfs_lne18_badcnt,
+ &debugfs_lne19_badcnt,
+ &debugfs_lne20_badcnt,
+ &debugfs_lne21_badcnt,
+ &debugfs_lne22_badcnt,
+ &debugfs_lne23_badcnt,
+
+ &debugfs_com_int,
+};
+
+static const struct pci_device_id thunderx_ocx_pci_tbl[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVICE_ID_THUNDER_OCX) },
+ { 0, },
+};
+
+static void thunderx_ocx_clearstats(struct thunderx_ocx *ocx)
+{
+ int lane, stat, cfg;
+
+ for (lane = 0; lane < OCX_RX_LANES; lane++) {
+ cfg = readq(ocx->regs + OCX_LNE_CFG(lane));
+ cfg |= OCX_LNE_CFG_RX_STAT_RDCLR;
+ cfg &= ~OCX_LNE_CFG_RX_STAT_ENA;
+ writeq(cfg, ocx->regs + OCX_LNE_CFG(lane));
+
+ for (stat = 0; stat < OCX_RX_LANE_STATS; stat++)
+ readq(ocx->regs + OCX_LNE_STAT(lane, stat));
+ }
+}
+
+static int thunderx_ocx_probe(struct pci_dev *pdev,
+ const struct pci_device_id *id)
+{
+ struct thunderx_ocx *ocx;
+ struct edac_device_ctl_info *edac_dev;
+ char name[32];
+ int idx;
+ int i;
+ int ret;
+ u64 reg;
+
+ ret = pcim_enable_device(pdev);
+ if (ret) {
+ dev_err(&pdev->dev, "Cannot enable PCI device: %d\n", ret);
+ return ret;
+ }
+
+ ret = pcim_iomap_regions(pdev, BIT(0), "thunderx_ocx");
+ if (ret) {
+ dev_err(&pdev->dev, "Cannot map PCI resources: %d\n", ret);
+ return ret;
+ }
+
+ idx = edac_device_alloc_index();
+ snprintf(name, sizeof(name), "OCX%d", idx);
+ edac_dev = edac_device_alloc_ctl_info(sizeof(struct thunderx_ocx),
+ name, 1, "CCPI", 1,
+ 0, NULL, 0, idx);
+ if (!edac_dev) {
+ dev_err(&pdev->dev, "Cannot allocate EDAC device: %d\n", ret);
+ return -ENOMEM;
+ }
+ ocx = edac_dev->pvt_info;
+ ocx->edac_dev = edac_dev;
+ ocx->com_ring_head = 0;
+ ocx->com_ring_tail = 0;
+ ocx->link_ring_head = 0;
+ ocx->link_ring_tail = 0;
+
+ ocx->regs = pcim_iomap_table(pdev)[0];
+ if (!ocx->regs) {
+ dev_err(&pdev->dev, "Cannot map PCI resources: %d\n", ret);
+ ret = -ENODEV;
+ goto err_free;
+ }
+
+ ocx->pdev = pdev;
+
+ for (i = 0; i < OCX_INTS; i++) {
+ ocx->msix_ent[i].entry = i;
+ ocx->msix_ent[i].vector = 0;
+ }
+
+ ret = pci_enable_msix_exact(pdev, ocx->msix_ent, OCX_INTS);
+ if (ret) {
+ dev_err(&pdev->dev, "Cannot enable interrupt: %d\n", ret);
+ goto err_free;
+ }
+
+ for (i = 0; i < OCX_INTS; i++) {
+ ret = devm_request_threaded_irq(&pdev->dev,
+ ocx->msix_ent[i].vector,
+ (i == 3) ?
+ thunderx_ocx_com_isr :
+ thunderx_ocx_lnk_isr,
+ (i == 3) ?
+ thunderx_ocx_com_threaded_isr :
+ thunderx_ocx_lnk_threaded_isr,
+ 0, "[EDAC] ThunderX OCX",
+ &ocx->msix_ent[i]);
+ if (ret)
+ goto err_free;
+ }
+
+ edac_dev->dev = &pdev->dev;
+ edac_dev->dev_name = dev_name(&pdev->dev);
+ edac_dev->mod_name = "thunderx-ocx";
+ edac_dev->ctl_name = "thunderx-ocx";
+
+ ret = edac_device_add_device(edac_dev);
+ if (ret) {
+ dev_err(&pdev->dev, "Cannot add EDAC device: %d\n", ret);
+ goto err_free;
+ }
+
+ if (IS_ENABLED(CONFIG_EDAC_DEBUG)) {
+ ocx->debugfs = edac_debugfs_create_dir(pdev->dev.kobj.name);
+
+ ret = thunderx_create_debugfs_nodes(ocx->debugfs,
+ ocx_dfs_ents,
+ ocx,
+ ARRAY_SIZE(ocx_dfs_ents));
+ if (ret != ARRAY_SIZE(ocx_dfs_ents)) {
+ dev_warn(&pdev->dev, "Error creating debugfs entries: %d%s\n",
+ ret, ret >= 0 ? " created" : "");
+ }
+ }
+
+ pci_set_drvdata(pdev, edac_dev);
+
+ thunderx_ocx_clearstats(ocx);
+
+ for (i = 0; i < OCX_RX_LANES; i++) {
+ writeq(OCX_LNE_INT_ENA_ALL,
+ ocx->regs + OCX_LNE_INT_EN(i));
+
+ reg = readq(ocx->regs + OCX_LNE_INT(i));
+ writeq(reg, ocx->regs + OCX_LNE_INT(i));
+
+ }
+
+ for (i = 0; i < OCX_LINK_INTS; i++) {
+ reg = readq(ocx->regs + OCX_COM_LINKX_INT(i));
+ writeq(reg, ocx->regs + OCX_COM_LINKX_INT(i));
+
+ writeq(OCX_COM_LINKX_INT_ENA_ALL,
+ ocx->regs + OCX_COM_LINKX_INT_ENA_W1S(i));
+ }
+
+ reg = readq(ocx->regs + OCX_COM_INT);
+ writeq(reg, ocx->regs + OCX_COM_INT);
+
+ writeq(OCX_COM_INT_ENA_ALL, ocx->regs + OCX_COM_INT_ENA_W1S);
+
+ return 0;
+err_free:
+ edac_device_free_ctl_info(edac_dev);
+
+ return ret;
+}
+
+static void thunderx_ocx_remove(struct pci_dev *pdev)
+{
+ struct edac_device_ctl_info *edac_dev = pci_get_drvdata(pdev);
+ struct thunderx_ocx *ocx = edac_dev->pvt_info;
+ int i;
+
+ writeq(OCX_COM_INT_ENA_ALL, ocx->regs + OCX_COM_INT_ENA_W1C);
+
+ for (i = 0; i < OCX_INTS; i++) {
+ writeq(OCX_COM_LINKX_INT_ENA_ALL,
+ ocx->regs + OCX_COM_LINKX_INT_ENA_W1C(i));
+ }
+
+ edac_debugfs_remove_recursive(ocx->debugfs);
+
+ edac_device_del_device(&pdev->dev);
+ edac_device_free_ctl_info(edac_dev);
+}
+
+MODULE_DEVICE_TABLE(pci, thunderx_ocx_pci_tbl);
+
+static struct pci_driver thunderx_ocx_driver = {
+ .name = "thunderx_ocx_edac",
+ .probe = thunderx_ocx_probe,
+ .remove = thunderx_ocx_remove,
+ .id_table = thunderx_ocx_pci_tbl,
+};
+
+/*---------------------- L2C driver ---------------------------------*/
+
+#define PCI_DEVICE_ID_THUNDER_L2C_TAD 0xa02e
+#define PCI_DEVICE_ID_THUNDER_L2C_CBC 0xa02f
+#define PCI_DEVICE_ID_THUNDER_L2C_MCI 0xa030
+
+#define L2C_TAD_INT_W1C 0x40000
+#define L2C_TAD_INT_W1S 0x40008
+
+#define L2C_TAD_INT_ENA_W1C 0x40020
+#define L2C_TAD_INT_ENA_W1S 0x40028
+
+
+#define L2C_TAD_INT_L2DDBE BIT(1)
+#define L2C_TAD_INT_SBFSBE BIT(2)
+#define L2C_TAD_INT_SBFDBE BIT(3)
+#define L2C_TAD_INT_FBFSBE BIT(4)
+#define L2C_TAD_INT_FBFDBE BIT(5)
+#define L2C_TAD_INT_TAGDBE BIT(9)
+#define L2C_TAD_INT_RDDISLMC BIT(15)
+#define L2C_TAD_INT_WRDISLMC BIT(16)
+#define L2C_TAD_INT_LFBTO BIT(17)
+#define L2C_TAD_INT_GSYNCTO BIT(18)
+#define L2C_TAD_INT_RTGSBE BIT(32)
+#define L2C_TAD_INT_RTGDBE BIT(33)
+#define L2C_TAD_INT_RDDISOCI BIT(34)
+#define L2C_TAD_INT_WRDISOCI BIT(35)
+
+#define L2C_TAD_INT_ECC (L2C_TAD_INT_L2DDBE | \
+ L2C_TAD_INT_SBFSBE | L2C_TAD_INT_SBFDBE | \
+ L2C_TAD_INT_FBFSBE | L2C_TAD_INT_FBFDBE)
+
+#define L2C_TAD_INT_CE (L2C_TAD_INT_SBFSBE | \
+ L2C_TAD_INT_FBFSBE)
+
+#define L2C_TAD_INT_UE (L2C_TAD_INT_L2DDBE | \
+ L2C_TAD_INT_SBFDBE | \
+ L2C_TAD_INT_FBFDBE | \
+ L2C_TAD_INT_TAGDBE | \
+ L2C_TAD_INT_RTGDBE | \
+ L2C_TAD_INT_WRDISOCI | \
+ L2C_TAD_INT_RDDISOCI | \
+ L2C_TAD_INT_WRDISLMC | \
+ L2C_TAD_INT_RDDISLMC | \
+ L2C_TAD_INT_LFBTO | \
+ L2C_TAD_INT_GSYNCTO)
+
+static const struct error_descr l2_tad_errors[] = {
+ {
+ .type = ERR_CORRECTED,
+ .mask = L2C_TAD_INT_SBFSBE,
+ .descr = "SBF single-bit error",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = L2C_TAD_INT_FBFSBE,
+ .descr = "FBF single-bit error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_TAD_INT_L2DDBE,
+ .descr = "L2D double-bit error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_TAD_INT_SBFDBE,
+ .descr = "SBF double-bit error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_TAD_INT_FBFDBE,
+ .descr = "FBF double-bit error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_TAD_INT_TAGDBE,
+ .descr = "TAG double-bit error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_TAD_INT_RTGDBE,
+ .descr = "RTG double-bit error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_TAD_INT_WRDISOCI,
+ .descr = "Write to a disabled CCPI",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_TAD_INT_RDDISOCI,
+ .descr = "Read from a disabled CCPI",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_TAD_INT_WRDISLMC,
+ .descr = "Write to a disabled LMC",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_TAD_INT_RDDISLMC,
+ .descr = "Read from a disabled LMC",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_TAD_INT_LFBTO,
+ .descr = "LFB entry timeout",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_TAD_INT_GSYNCTO,
+ .descr = "Global sync CCPI timeout",
+ },
+ {0, 0, NULL},
+};
+
+#define L2C_TAD_INT_TAG (L2C_TAD_INT_TAGDBE)
+
+#define L2C_TAD_INT_RTG (L2C_TAD_INT_RTGDBE)
+
+#define L2C_TAD_INT_DISLMC (L2C_TAD_INT_WRDISLMC | L2C_TAD_INT_RDDISLMC)
+
+#define L2C_TAD_INT_DISOCI (L2C_TAD_INT_WRDISOCI | L2C_TAD_INT_RDDISOCI)
+
+#define L2C_TAD_INT_ENA_ALL (L2C_TAD_INT_ECC | L2C_TAD_INT_TAG | \
+ L2C_TAD_INT_RTG | \
+ L2C_TAD_INT_DISLMC | L2C_TAD_INT_DISOCI | \
+ L2C_TAD_INT_LFBTO)
+
+#define L2C_TAD_TIMETWO 0x50000
+#define L2C_TAD_TIMEOUT 0x50100
+#define L2C_TAD_ERR 0x60000
+#define L2C_TAD_TQD_ERR 0x60100
+#define L2C_TAD_TTG_ERR 0x60200
+
+
+#define L2C_CBC_INT_W1C 0x60000
+
+#define L2C_CBC_INT_RSDSBE BIT(0)
+#define L2C_CBC_INT_RSDDBE BIT(1)
+
+#define L2C_CBC_INT_RSD (L2C_CBC_INT_RSDSBE | L2C_CBC_INT_RSDDBE)
+
+#define L2C_CBC_INT_MIBSBE BIT(4)
+#define L2C_CBC_INT_MIBDBE BIT(5)
+
+#define L2C_CBC_INT_MIB (L2C_CBC_INT_MIBSBE | L2C_CBC_INT_MIBDBE)
+
+#define L2C_CBC_INT_IORDDISOCI BIT(6)
+#define L2C_CBC_INT_IOWRDISOCI BIT(7)
+
+#define L2C_CBC_INT_IODISOCI (L2C_CBC_INT_IORDDISOCI | \
+ L2C_CBC_INT_IOWRDISOCI)
+
+#define L2C_CBC_INT_CE (L2C_CBC_INT_RSDSBE | L2C_CBC_INT_MIBSBE)
+#define L2C_CBC_INT_UE (L2C_CBC_INT_RSDDBE | L2C_CBC_INT_MIBDBE)
+
+
+static const struct error_descr l2_cbc_errors[] = {
+ {
+ .type = ERR_CORRECTED,
+ .mask = L2C_CBC_INT_RSDSBE,
+ .descr = "RSD single-bit error",
+ },
+ {
+ .type = ERR_CORRECTED,
+ .mask = L2C_CBC_INT_MIBSBE,
+ .descr = "MIB single-bit error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_CBC_INT_RSDDBE,
+ .descr = "RSD double-bit error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_CBC_INT_MIBDBE,
+ .descr = "MIB double-bit error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_CBC_INT_IORDDISOCI,
+ .descr = "Read from a disabled CCPI",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_CBC_INT_IOWRDISOCI,
+ .descr = "Write to a disabled CCPI",
+ },
+ {0, 0, NULL},
+};
+
+#define L2C_CBC_INT_W1S 0x60008
+#define L2C_CBC_INT_ENA_W1C 0x60020
+
+#define L2C_CBC_INT_ENA_ALL (L2C_CBC_INT_RSD | L2C_CBC_INT_MIB | \
+ L2C_CBC_INT_IODISOCI)
+
+#define L2C_CBC_INT_ENA_W1S 0x60028
+
+#define L2C_CBC_IODISOCIERR 0x80008
+#define L2C_CBC_IOCERR 0x80010
+#define L2C_CBC_RSDERR 0x80018
+#define L2C_CBC_MIBERR 0x80020
+
+
+#define L2C_MCI_INT_W1C 0x0
+
+#define L2C_MCI_INT_VBFSBE BIT(0)
+#define L2C_MCI_INT_VBFDBE BIT(1)
+
+static const struct error_descr l2_mci_errors[] = {
+ {
+ .type = ERR_CORRECTED,
+ .mask = L2C_MCI_INT_VBFSBE,
+ .descr = "VBF single-bit error",
+ },
+ {
+ .type = ERR_UNCORRECTED,
+ .mask = L2C_MCI_INT_VBFDBE,
+ .descr = "VBF double-bit error",
+ },
+ {0, 0, NULL},
+};
+
+#define L2C_MCI_INT_W1S 0x8
+#define L2C_MCI_INT_ENA_W1C 0x20
+
+#define L2C_MCI_INT_ENA_ALL (L2C_MCI_INT_VBFSBE | L2C_MCI_INT_VBFDBE)
+
+#define L2C_MCI_INT_ENA_W1S 0x28
+
+#define L2C_MCI_ERR 0x10000
+
+#define L2C_MESSAGE_SIZE SZ_1K
+#define L2C_OTHER_SIZE (50 * ARRAY_SIZE(l2_tad_errors))
+
+struct l2c_err_ctx {
+ char *reg_ext_name;
+ u64 reg_int;
+ u64 reg_ext;
+};
+
+struct thunderx_l2c {
+ void __iomem *regs;
+ struct pci_dev *pdev;
+ struct edac_device_ctl_info *edac_dev;
+
+ struct dentry *debugfs;
+
+ int index;
+
+ struct msix_entry msix_ent;
+
+ struct l2c_err_ctx err_ctx[RING_ENTRIES];
+ unsigned long ring_head;
+ unsigned long ring_tail;
+};
+
+static irqreturn_t thunderx_l2c_tad_isr(int irq, void *irq_id)
+{
+ struct msix_entry *msix = irq_id;
+ struct thunderx_l2c *tad = container_of(msix, struct thunderx_l2c,
+ msix_ent);
+
+ unsigned long head = ring_pos(tad->ring_head, ARRAY_SIZE(tad->err_ctx));
+ struct l2c_err_ctx *ctx = &tad->err_ctx[head];
+
+ ctx->reg_int = readq(tad->regs + L2C_TAD_INT_W1C);
+
+ if (ctx->reg_int & L2C_TAD_INT_ECC) {
+ ctx->reg_ext_name = "TQD_ERR";
+ ctx->reg_ext = readq(tad->regs + L2C_TAD_TQD_ERR);
+ } else if (ctx->reg_int & L2C_TAD_INT_TAG) {
+ ctx->reg_ext_name = "TTG_ERR";
+ ctx->reg_ext = readq(tad->regs + L2C_TAD_TTG_ERR);
+ } else if (ctx->reg_int & L2C_TAD_INT_LFBTO) {
+ ctx->reg_ext_name = "TIMEOUT";
+ ctx->reg_ext = readq(tad->regs + L2C_TAD_TIMEOUT);
+ } else if (ctx->reg_int & L2C_TAD_INT_DISOCI) {
+ ctx->reg_ext_name = "ERR";
+ ctx->reg_ext = readq(tad->regs + L2C_TAD_ERR);
+ }
+
+ writeq(ctx->reg_int, tad->regs + L2C_TAD_INT_W1C);
+
+ tad->ring_head++;
+
+ return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t thunderx_l2c_cbc_isr(int irq, void *irq_id)
+{
+ struct msix_entry *msix = irq_id;
+ struct thunderx_l2c *cbc = container_of(msix, struct thunderx_l2c,
+ msix_ent);
+
+ unsigned long head = ring_pos(cbc->ring_head, ARRAY_SIZE(cbc->err_ctx));
+ struct l2c_err_ctx *ctx = &cbc->err_ctx[head];
+
+ ctx->reg_int = readq(cbc->regs + L2C_CBC_INT_W1C);
+
+ if (ctx->reg_int & L2C_CBC_INT_RSD) {
+ ctx->reg_ext_name = "RSDERR";
+ ctx->reg_ext = readq(cbc->regs + L2C_CBC_RSDERR);
+ } else if (ctx->reg_int & L2C_CBC_INT_MIB) {
+ ctx->reg_ext_name = "MIBERR";
+ ctx->reg_ext = readq(cbc->regs + L2C_CBC_MIBERR);
+ } else if (ctx->reg_int & L2C_CBC_INT_IODISOCI) {
+ ctx->reg_ext_name = "IODISOCIERR";
+ ctx->reg_ext = readq(cbc->regs + L2C_CBC_IODISOCIERR);
+ }
+
+ writeq(ctx->reg_int, cbc->regs + L2C_CBC_INT_W1C);
+
+ cbc->ring_head++;
+
+ return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t thunderx_l2c_mci_isr(int irq, void *irq_id)
+{
+ struct msix_entry *msix = irq_id;
+ struct thunderx_l2c *mci = container_of(msix, struct thunderx_l2c,
+ msix_ent);
+
+ unsigned long head = ring_pos(mci->ring_head, ARRAY_SIZE(mci->err_ctx));
+ struct l2c_err_ctx *ctx = &mci->err_ctx[head];
+
+ ctx->reg_int = readq(mci->regs + L2C_MCI_INT_W1C);
+ ctx->reg_ext = readq(mci->regs + L2C_MCI_ERR);
+
+ writeq(ctx->reg_int, mci->regs + L2C_MCI_INT_W1C);
+
+ ctx->reg_ext_name = "ERR";
+
+ mci->ring_head++;
+
+ return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t thunderx_l2c_threaded_isr(int irq, void *irq_id)
+{
+ struct msix_entry *msix = irq_id;
+ struct thunderx_l2c *l2c = container_of(msix, struct thunderx_l2c,
+ msix_ent);
+
+ unsigned long tail = ring_pos(l2c->ring_tail, ARRAY_SIZE(l2c->err_ctx));
+ struct l2c_err_ctx *ctx = &l2c->err_ctx[tail];
+ irqreturn_t ret = IRQ_NONE;
+
+ u64 mask_ue, mask_ce;
+ const struct error_descr *l2_errors;
+ char *reg_int_name;
+
+ char *msg;
+ char *other;
+
+ msg = kmalloc(OCX_MESSAGE_SIZE, GFP_KERNEL);
+ other = kmalloc(OCX_OTHER_SIZE, GFP_KERNEL);
+
+ if (!msg || !other)
+ goto err_free;
+
+ switch (l2c->pdev->device) {
+ case PCI_DEVICE_ID_THUNDER_L2C_TAD:
+ reg_int_name = "L2C_TAD_INT";
+ mask_ue = L2C_TAD_INT_UE;
+ mask_ce = L2C_TAD_INT_CE;
+ l2_errors = l2_tad_errors;
+ break;
+ case PCI_DEVICE_ID_THUNDER_L2C_CBC:
+ reg_int_name = "L2C_CBC_INT";
+ mask_ue = L2C_CBC_INT_UE;
+ mask_ce = L2C_CBC_INT_CE;
+ l2_errors = l2_cbc_errors;
+ break;
+ case PCI_DEVICE_ID_THUNDER_L2C_MCI:
+ reg_int_name = "L2C_MCI_INT";
+ mask_ue = L2C_MCI_INT_VBFDBE;
+ mask_ce = L2C_MCI_INT_VBFSBE;
+ l2_errors = l2_mci_errors;
+ break;
+ default:
+ dev_err(&l2c->pdev->dev, "Unsupported device: %04x\n",
+ l2c->pdev->device);
+ return IRQ_NONE;
+ }
+
+ while (CIRC_CNT(l2c->ring_head, l2c->ring_tail,
+ ARRAY_SIZE(l2c->err_ctx))) {
+ snprintf(msg, L2C_MESSAGE_SIZE,
+ "%s: %s: %016llx, %s: %016llx",
+ l2c->edac_dev->ctl_name, reg_int_name, ctx->reg_int,
+ ctx->reg_ext_name, ctx->reg_ext);
+
+ decode_register(other, L2C_OTHER_SIZE, l2_errors, ctx->reg_int);
+
+ strncat(msg, other, L2C_MESSAGE_SIZE);
+
+ if (ctx->reg_int & mask_ue)
+ edac_device_handle_ue(l2c->edac_dev, 0, 0, msg);
+ else if (ctx->reg_int & mask_ce)
+ edac_device_handle_ce(l2c->edac_dev, 0, 0, msg);
+
+ l2c->ring_tail++;
+ }
+
+ return IRQ_HANDLED;
+
+err_free:
+ kfree(other);
+ kfree(msg);
+
+ return ret;
+}
+
+#define L2C_DEBUGFS_ATTR(_name, _reg) DEBUGFS_REG_ATTR(l2c, _name, _reg)
+
+L2C_DEBUGFS_ATTR(tad_int, L2C_TAD_INT_W1S);
+
+struct debugfs_entry *l2c_tad_dfs_ents[] = {
+ &debugfs_tad_int,
+};
+
+L2C_DEBUGFS_ATTR(cbc_int, L2C_CBC_INT_W1S);
+
+struct debugfs_entry *l2c_cbc_dfs_ents[] = {
+ &debugfs_cbc_int,
+};
+
+L2C_DEBUGFS_ATTR(mci_int, L2C_MCI_INT_W1S);
+
+struct debugfs_entry *l2c_mci_dfs_ents[] = {
+ &debugfs_mci_int,
+};
+
+static const struct pci_device_id thunderx_l2c_pci_tbl[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVICE_ID_THUNDER_L2C_TAD), },
+ { PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVICE_ID_THUNDER_L2C_CBC), },
+ { PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, PCI_DEVICE_ID_THUNDER_L2C_MCI), },
+ { 0, },
+};
+
+static int thunderx_l2c_probe(struct pci_dev *pdev,
+ const struct pci_device_id *id)
+{
+ struct thunderx_l2c *l2c;
+ struct edac_device_ctl_info *edac_dev;
+ struct debugfs_entry **l2c_devattr;
+ size_t dfs_entries;
+ irqreturn_t (*thunderx_l2c_isr)(int, void *) = NULL;
+ char name[32];
+ const char *fmt;
+ u64 reg_en_offs, reg_en_mask;
+ int idx;
+ int ret;
+
+ ret = pcim_enable_device(pdev);
+ if (ret) {
+ dev_err(&pdev->dev, "Cannot enable PCI device: %d\n", ret);
+ return ret;
+ }
+
+ ret = pcim_iomap_regions(pdev, BIT(0), "thunderx_l2c");
+ if (ret) {
+ dev_err(&pdev->dev, "Cannot map PCI resources: %d\n", ret);
+ return ret;
+ }
+
+ switch (pdev->device) {
+ case PCI_DEVICE_ID_THUNDER_L2C_TAD:
+ thunderx_l2c_isr = thunderx_l2c_tad_isr;
+ l2c_devattr = l2c_tad_dfs_ents;
+ dfs_entries = ARRAY_SIZE(l2c_tad_dfs_ents);
+ fmt = "L2C-TAD%d";
+ reg_en_offs = L2C_TAD_INT_ENA_W1S;
+ reg_en_mask = L2C_TAD_INT_ENA_ALL;
+ break;
+ case PCI_DEVICE_ID_THUNDER_L2C_CBC:
+ thunderx_l2c_isr = thunderx_l2c_cbc_isr;
+ l2c_devattr = l2c_cbc_dfs_ents;
+ dfs_entries = ARRAY_SIZE(l2c_cbc_dfs_ents);
+ fmt = "L2C-CBC%d";
+ reg_en_offs = L2C_CBC_INT_ENA_W1S;
+ reg_en_mask = L2C_CBC_INT_ENA_ALL;
+ break;
+ case PCI_DEVICE_ID_THUNDER_L2C_MCI:
+ thunderx_l2c_isr = thunderx_l2c_mci_isr;
+ l2c_devattr = l2c_mci_dfs_ents;
+ dfs_entries = ARRAY_SIZE(l2c_mci_dfs_ents);
+ fmt = "L2C-MCI%d";
+ reg_en_offs = L2C_MCI_INT_ENA_W1S;
+ reg_en_mask = L2C_MCI_INT_ENA_ALL;
+ break;
+ default:
+ //Should never ever get here
+ dev_err(&pdev->dev, "Unsupported PCI device: %04x\n",
+ pdev->device);
+ return -EINVAL;
+ }
+
+ idx = edac_device_alloc_index();
+ snprintf(name, sizeof(name), fmt, idx);
+
+ edac_dev = edac_device_alloc_ctl_info(sizeof(struct thunderx_l2c),
+ name, 1, "L2C", 1, 0,
+ NULL, 0, idx);
+ if (!edac_dev) {
+ dev_err(&pdev->dev, "Cannot allocate EDAC device\n");
+ return -ENOMEM;
+ }
+
+ l2c = edac_dev->pvt_info;
+ l2c->edac_dev = edac_dev;
+
+ l2c->regs = pcim_iomap_table(pdev)[0];
+ if (!l2c->regs) {
+ dev_err(&pdev->dev, "Cannot map PCI resources\n");
+ ret = -ENODEV;
+ goto err_free;
+ }
+
+ l2c->pdev = pdev;
+
+ l2c->ring_head = 0;
+ l2c->ring_tail = 0;
+
+ l2c->msix_ent.entry = 0;
+ l2c->msix_ent.vector = 0;
+
+ ret = pci_enable_msix_exact(pdev, &l2c->msix_ent, 1);
+ if (ret) {
+ dev_err(&pdev->dev, "Cannot enable interrupt: %d\n", ret);
+ goto err_free;
+ }
+
+ ret = devm_request_threaded_irq(&pdev->dev, l2c->msix_ent.vector,
+ thunderx_l2c_isr,
+ thunderx_l2c_threaded_isr,
+ 0, "[EDAC] ThunderX L2C",
+ &l2c->msix_ent);
+ if (ret)
+ goto err_free;
+
+ edac_dev->dev = &pdev->dev;
+ edac_dev->dev_name = dev_name(&pdev->dev);
+ edac_dev->mod_name = "thunderx-l2c";
+ edac_dev->ctl_name = "thunderx-l2c";
+
+ ret = edac_device_add_device(edac_dev);
+ if (ret) {
+ dev_err(&pdev->dev, "Cannot add EDAC device: %d\n", ret);
+ goto err_free;
+ }
+
+ if (IS_ENABLED(CONFIG_EDAC_DEBUG)) {
+ l2c->debugfs = edac_debugfs_create_dir(pdev->dev.kobj.name);
+
+ thunderx_create_debugfs_nodes(l2c->debugfs, l2c_devattr,
+ l2c, dfs_entries);
+
+ if (ret != dfs_entries) {
+ dev_warn(&pdev->dev, "Error creating debugfs entries: %d%s\n",
+ ret, ret >= 0 ? " created" : "");
+ }
+ }
+
+ pci_set_drvdata(pdev, edac_dev);
+
+ writeq(reg_en_mask, l2c->regs + reg_en_offs);
+
+ return 0;
+
+err_free:
+ edac_device_free_ctl_info(edac_dev);
+
+ return ret;
+}
+
+static void thunderx_l2c_remove(struct pci_dev *pdev)
+{
+ struct edac_device_ctl_info *edac_dev = pci_get_drvdata(pdev);
+ struct thunderx_l2c *l2c = edac_dev->pvt_info;
+
+ switch (pdev->device) {
+ case PCI_DEVICE_ID_THUNDER_L2C_TAD:
+ writeq(L2C_TAD_INT_ENA_ALL, l2c->regs + L2C_TAD_INT_ENA_W1C);
+ break;
+ case PCI_DEVICE_ID_THUNDER_L2C_CBC:
+ writeq(L2C_CBC_INT_ENA_ALL, l2c->regs + L2C_CBC_INT_ENA_W1C);
+ break;
+ case PCI_DEVICE_ID_THUNDER_L2C_MCI:
+ writeq(L2C_MCI_INT_ENA_ALL, l2c->regs + L2C_MCI_INT_ENA_W1C);
+ break;
+ }
+
+ edac_debugfs_remove_recursive(l2c->debugfs);
+
+ edac_device_del_device(&pdev->dev);
+ edac_device_free_ctl_info(edac_dev);
+}
+
+MODULE_DEVICE_TABLE(pci, thunderx_l2c_pci_tbl);
+
+static struct pci_driver thunderx_l2c_driver = {
+ .name = "thunderx_l2c_edac",
+ .probe = thunderx_l2c_probe,
+ .remove = thunderx_l2c_remove,
+ .id_table = thunderx_l2c_pci_tbl,
+};
+
+static int __init thunderx_edac_init(void)
+{
+ int rc = 0;
+
+ rc = pci_register_driver(&thunderx_lmc_driver);
+ if (rc)
+ return rc;
+
+ rc = pci_register_driver(&thunderx_ocx_driver);
+ if (rc)
+ goto err_lmc;
+
+ rc = pci_register_driver(&thunderx_l2c_driver);
+ if (rc)
+ goto err_ocx;
+
+ return rc;
+err_ocx:
+ pci_unregister_driver(&thunderx_ocx_driver);
+err_lmc:
+ pci_unregister_driver(&thunderx_lmc_driver);
+
+ return rc;
+}
+
+static void __exit thunderx_edac_exit(void)
+{
+ pci_unregister_driver(&thunderx_l2c_driver);
+ pci_unregister_driver(&thunderx_ocx_driver);
+ pci_unregister_driver(&thunderx_lmc_driver);
+
+}
+
+module_init(thunderx_edac_init);
+module_exit(thunderx_edac_exit);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Cavium, Inc.");
+MODULE_DESCRIPTION("EDAC Driver for Cavium ThunderX");
diff --git a/drivers/extcon/devres.c b/drivers/extcon/devres.c
index b40eb18..186fd73 100644
--- a/drivers/extcon/devres.c
+++ b/drivers/extcon/devres.c
@@ -50,6 +50,13 @@
extcon_unregister_notifier(this->edev, this->id, this->nb);
}
+static void devm_extcon_dev_notifier_all_unreg(struct device *dev, void *res)
+{
+ struct extcon_dev_notifier_devres *this = res;
+
+ extcon_unregister_notifier_all(this->edev, this->nb);
+}
+
/**
* devm_extcon_dev_allocate - Allocate managed extcon device
* @dev: device owning the extcon device being created
@@ -214,3 +221,57 @@
devm_extcon_dev_match, edev));
}
EXPORT_SYMBOL(devm_extcon_unregister_notifier);
+
+/**
+ * devm_extcon_register_notifier_all()
+ * - Resource-managed extcon_register_notifier_all()
+ * @dev: device to allocate extcon device
+ * @edev: the extcon device that has the external connecotr.
+ * @nb: a notifier block to be registered.
+ *
+ * This function manages automatically the notifier of extcon device using
+ * device resource management and simplify the control of unregistering
+ * the notifier of extcon device. To get more information, refer that function.
+ *
+ * Returns 0 if success or negaive error number if failure.
+ */
+int devm_extcon_register_notifier_all(struct device *dev, struct extcon_dev *edev,
+ struct notifier_block *nb)
+{
+ struct extcon_dev_notifier_devres *ptr;
+ int ret;
+
+ ptr = devres_alloc(devm_extcon_dev_notifier_all_unreg, sizeof(*ptr),
+ GFP_KERNEL);
+ if (!ptr)
+ return -ENOMEM;
+
+ ret = extcon_register_notifier_all(edev, nb);
+ if (ret) {
+ devres_free(ptr);
+ return ret;
+ }
+
+ ptr->edev = edev;
+ ptr->nb = nb;
+ devres_add(dev, ptr);
+
+ return 0;
+}
+EXPORT_SYMBOL(devm_extcon_register_notifier_all);
+
+/**
+ * devm_extcon_unregister_notifier_all()
+ * - Resource-managed extcon_unregister_notifier_all()
+ * @dev: device to allocate extcon device
+ * @edev: the extcon device that has the external connecotr.
+ * @nb: a notifier block to be registered.
+ */
+void devm_extcon_unregister_notifier_all(struct device *dev,
+ struct extcon_dev *edev,
+ struct notifier_block *nb)
+{
+ WARN_ON(devres_release(dev, devm_extcon_dev_notifier_all_unreg,
+ devm_extcon_dev_match, edev));
+}
+EXPORT_SYMBOL(devm_extcon_unregister_notifier_all);
diff --git a/drivers/extcon/extcon.c b/drivers/extcon/extcon.c
index 09ac5e7..e775054 100644
--- a/drivers/extcon/extcon.c
+++ b/drivers/extcon/extcon.c
@@ -448,8 +448,19 @@
spin_lock_irqsave(&edev->lock, flags);
state = !!(edev->state & BIT(index));
+
+ /*
+ * Call functions in a raw notifier chain for the specific one
+ * external connector.
+ */
raw_notifier_call_chain(&edev->nh[index], state, edev);
+ /*
+ * Call functions in a raw notifier chain for the all supported
+ * external connectors.
+ */
+ raw_notifier_call_chain(&edev->nh_all, state, edev);
+
/* This could be in interrupt handler */
prop_buf = (char *)get_zeroed_page(GFP_ATOMIC);
if (!prop_buf) {
@@ -954,6 +965,59 @@
}
EXPORT_SYMBOL_GPL(extcon_unregister_notifier);
+/**
+ * extcon_register_notifier_all() - Register a notifier block for all connectors
+ * @edev: the extcon device that has the external connecotr.
+ * @nb: a notifier block to be registered.
+ *
+ * This fucntion registers a notifier block in order to receive the state
+ * change of all supported external connectors from extcon device.
+ * And The second parameter given to the callback of nb (val) is
+ * the current state and third parameter is the edev pointer.
+ *
+ * Returns 0 if success or error number if fail
+ */
+int extcon_register_notifier_all(struct extcon_dev *edev,
+ struct notifier_block *nb)
+{
+ unsigned long flags;
+ int ret;
+
+ if (!edev || !nb)
+ return -EINVAL;
+
+ spin_lock_irqsave(&edev->lock, flags);
+ ret = raw_notifier_chain_register(&edev->nh_all, nb);
+ spin_unlock_irqrestore(&edev->lock, flags);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(extcon_register_notifier_all);
+
+/**
+ * extcon_unregister_notifier_all() - Unregister a notifier block from extcon.
+ * @edev: the extcon device that has the external connecotr.
+ * @nb: a notifier block to be registered.
+ *
+ * Returns 0 if success or error number if fail
+ */
+int extcon_unregister_notifier_all(struct extcon_dev *edev,
+ struct notifier_block *nb)
+{
+ unsigned long flags;
+ int ret;
+
+ if (!edev || !nb)
+ return -EINVAL;
+
+ spin_lock_irqsave(&edev->lock, flags);
+ ret = raw_notifier_chain_unregister(&edev->nh_all, nb);
+ spin_unlock_irqrestore(&edev->lock, flags);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(extcon_unregister_notifier_all);
+
static struct attribute *extcon_attrs[] = {
&dev_attr_state.attr,
&dev_attr_name.attr,
@@ -1212,6 +1276,8 @@
for (index = 0; index < edev->max_supported; index++)
RAW_INIT_NOTIFIER_HEAD(&edev->nh[index]);
+ RAW_INIT_NOTIFIER_HEAD(&edev->nh_all);
+
dev_set_drvdata(&edev->dev, edev);
edev->state = 0;
diff --git a/drivers/extcon/extcon.h b/drivers/extcon/extcon.h
index 993ddcc..dddddcf 100644
--- a/drivers/extcon/extcon.h
+++ b/drivers/extcon/extcon.h
@@ -21,6 +21,8 @@
* @dev: Device of this extcon.
* @state: Attach/detach state of this extcon. Do not provide at
* register-time.
+ * @nh_all: Notifier for the state change events for all supported
+ * external connectors from this extcon.
* @nh: Notifier for the state change events from this extcon
* @entry: To support list of extcon devices so that users can
* search for extcon devices based on the extcon name.
@@ -43,6 +45,7 @@
/* Internal data. Please do not set. */
struct device dev;
+ struct raw_notifier_head nh_all;
struct raw_notifier_head *nh;
struct list_head entry;
int max_supported;
diff --git a/drivers/firmware/efi/Makefile b/drivers/firmware/efi/Makefile
index ad67342..0329d31 100644
--- a/drivers/firmware/efi/Makefile
+++ b/drivers/firmware/efi/Makefile
@@ -9,6 +9,7 @@
#
KASAN_SANITIZE_runtime-wrappers.o := n
+obj-$(CONFIG_ACPI_BGRT) += efi-bgrt.o
obj-$(CONFIG_EFI) += efi.o vars.o reboot.o memattr.o
obj-$(CONFIG_EFI) += capsule.o memmap.o
obj-$(CONFIG_EFI_VARS) += efivars.o
diff --git a/arch/x86/platform/efi/efi-bgrt.c b/drivers/firmware/efi/efi-bgrt.c
similarity index 100%
rename from arch/x86/platform/efi/efi-bgrt.c
rename to drivers/firmware/efi/efi-bgrt.c
diff --git a/drivers/firmware/efi/efi-pstore.c b/drivers/firmware/efi/efi-pstore.c
index f402ba2..ed3137c 100644
--- a/drivers/firmware/efi/efi-pstore.c
+++ b/drivers/firmware/efi/efi-pstore.c
@@ -28,26 +28,16 @@
return 0;
}
-struct pstore_read_data {
- u64 *id;
- enum pstore_type_id *type;
- int *count;
- struct timespec *timespec;
- bool *compressed;
- ssize_t *ecc_notice_size;
- char **buf;
-};
-
static inline u64 generic_id(unsigned long timestamp,
unsigned int part, int count)
{
return ((u64) timestamp * 100 + part) * 1000 + count;
}
-static int efi_pstore_read_func(struct efivar_entry *entry, void *data)
+static int efi_pstore_read_func(struct efivar_entry *entry,
+ struct pstore_record *record)
{
efi_guid_t vendor = LINUX_EFI_CRASH_GUID;
- struct pstore_read_data *cb_data = data;
char name[DUMP_NAME_LEN], data_type;
int i;
int cnt;
@@ -61,37 +51,37 @@
name[i] = entry->var.VariableName[i];
if (sscanf(name, "dump-type%u-%u-%d-%lu-%c",
- cb_data->type, &part, &cnt, &time, &data_type) == 5) {
- *cb_data->id = generic_id(time, part, cnt);
- *cb_data->count = cnt;
- cb_data->timespec->tv_sec = time;
- cb_data->timespec->tv_nsec = 0;
+ &record->type, &part, &cnt, &time, &data_type) == 5) {
+ record->id = generic_id(time, part, cnt);
+ record->count = cnt;
+ record->time.tv_sec = time;
+ record->time.tv_nsec = 0;
if (data_type == 'C')
- *cb_data->compressed = true;
+ record->compressed = true;
else
- *cb_data->compressed = false;
- *cb_data->ecc_notice_size = 0;
+ record->compressed = false;
+ record->ecc_notice_size = 0;
} else if (sscanf(name, "dump-type%u-%u-%d-%lu",
- cb_data->type, &part, &cnt, &time) == 4) {
- *cb_data->id = generic_id(time, part, cnt);
- *cb_data->count = cnt;
- cb_data->timespec->tv_sec = time;
- cb_data->timespec->tv_nsec = 0;
- *cb_data->compressed = false;
- *cb_data->ecc_notice_size = 0;
+ &record->type, &part, &cnt, &time) == 4) {
+ record->id = generic_id(time, part, cnt);
+ record->count = cnt;
+ record->time.tv_sec = time;
+ record->time.tv_nsec = 0;
+ record->compressed = false;
+ record->ecc_notice_size = 0;
} else if (sscanf(name, "dump-type%u-%u-%lu",
- cb_data->type, &part, &time) == 3) {
+ &record->type, &part, &time) == 3) {
/*
* Check if an old format,
* which doesn't support holding
* multiple logs, remains.
*/
- *cb_data->id = generic_id(time, part, 0);
- *cb_data->count = 0;
- cb_data->timespec->tv_sec = time;
- cb_data->timespec->tv_nsec = 0;
- *cb_data->compressed = false;
- *cb_data->ecc_notice_size = 0;
+ record->id = generic_id(time, part, 0);
+ record->count = 0;
+ record->time.tv_sec = time;
+ record->time.tv_nsec = 0;
+ record->compressed = false;
+ record->ecc_notice_size = 0;
} else
return 0;
@@ -99,7 +89,7 @@
__efivar_entry_get(entry, &entry->var.Attributes,
&entry->var.DataSize, entry->var.Data);
size = entry->var.DataSize;
- memcpy(*cb_data->buf, entry->var.Data,
+ memcpy(record->buf, entry->var.Data,
(size_t)min_t(unsigned long, EFIVARS_DATA_SIZE_MAX, size));
return size;
@@ -164,7 +154,7 @@
/**
* efi_pstore_sysfs_entry_iter
*
- * @data: function-specific data to pass to callback
+ * @record: pstore record to pass to callback
* @pos: entry to begin iterating from
*
* You MUST call efivar_enter_iter_begin() before this function, and
@@ -175,7 +165,8 @@
* the next entry of the last one passed to efi_pstore_read_func().
* To begin iterating from the beginning of the list @pos must be %NULL.
*/
-static int efi_pstore_sysfs_entry_iter(void *data, struct efivar_entry **pos)
+static int efi_pstore_sysfs_entry_iter(struct pstore_record *record,
+ struct efivar_entry **pos)
{
struct efivar_entry *entry, *n;
struct list_head *head = &efivar_sysfs_list;
@@ -186,7 +177,7 @@
list_for_each_entry_safe(entry, n, head, list) {
efi_pstore_scan_sysfs_enter(entry, n, head);
- size = efi_pstore_read_func(entry, data);
+ size = efi_pstore_read_func(entry, record);
ret = efi_pstore_scan_sysfs_exit(entry, n, head,
size < 0);
if (ret)
@@ -201,7 +192,7 @@
list_for_each_entry_safe_from((*pos), n, head, list) {
efi_pstore_scan_sysfs_enter((*pos), n, head);
- size = efi_pstore_read_func((*pos), data);
+ size = efi_pstore_read_func((*pos), record);
ret = efi_pstore_scan_sysfs_exit((*pos), n, head, size < 0);
if (ret)
return ret;
@@ -225,71 +216,57 @@
* size < 0: Failed to get data of entry logging via efi_pstore_write(),
* and pstore will stop reading entry.
*/
-static ssize_t efi_pstore_read(u64 *id, enum pstore_type_id *type,
- int *count, struct timespec *timespec,
- char **buf, bool *compressed,
- ssize_t *ecc_notice_size,
- struct pstore_info *psi)
+static ssize_t efi_pstore_read(struct pstore_record *record)
{
- struct pstore_read_data data;
+ struct efivar_entry *entry = (struct efivar_entry *)record->psi->data;
ssize_t size;
- data.id = id;
- data.type = type;
- data.count = count;
- data.timespec = timespec;
- data.compressed = compressed;
- data.ecc_notice_size = ecc_notice_size;
- data.buf = buf;
-
- *data.buf = kzalloc(EFIVARS_DATA_SIZE_MAX, GFP_KERNEL);
- if (!*data.buf)
+ record->buf = kzalloc(EFIVARS_DATA_SIZE_MAX, GFP_KERNEL);
+ if (!record->buf)
return -ENOMEM;
if (efivar_entry_iter_begin()) {
- kfree(*data.buf);
- return -EINTR;
+ size = -EINTR;
+ goto out;
}
- size = efi_pstore_sysfs_entry_iter(&data,
- (struct efivar_entry **)&psi->data);
+ size = efi_pstore_sysfs_entry_iter(record, &entry);
efivar_entry_iter_end();
- if (size <= 0)
- kfree(*data.buf);
+
+out:
+ if (size <= 0) {
+ kfree(record->buf);
+ record->buf = NULL;
+ }
return size;
}
-static int efi_pstore_write(enum pstore_type_id type,
- enum kmsg_dump_reason reason, u64 *id,
- unsigned int part, int count, bool compressed, size_t size,
- struct pstore_info *psi)
+static int efi_pstore_write(struct pstore_record *record)
{
char name[DUMP_NAME_LEN];
efi_char16_t efi_name[DUMP_NAME_LEN];
efi_guid_t vendor = LINUX_EFI_CRASH_GUID;
int i, ret = 0;
- sprintf(name, "dump-type%u-%u-%d-%lu-%c", type, part, count,
- get_seconds(), compressed ? 'C' : 'D');
+ snprintf(name, sizeof(name), "dump-type%u-%u-%d-%lu-%c",
+ record->type, record->part, record->count,
+ get_seconds(), record->compressed ? 'C' : 'D');
for (i = 0; i < DUMP_NAME_LEN; i++)
efi_name[i] = name[i];
- efivar_entry_set_safe(efi_name, vendor, PSTORE_EFI_ATTRIBUTES,
- !pstore_cannot_block_path(reason),
- size, psi->buf);
+ ret = efivar_entry_set_safe(efi_name, vendor, PSTORE_EFI_ATTRIBUTES,
+ !pstore_cannot_block_path(record->reason),
+ record->size, record->psi->buf);
- if (reason == KMSG_DUMP_OOPS)
+ if (record->reason == KMSG_DUMP_OOPS)
efivar_run_worker();
- *id = part;
+ record->id = record->part;
return ret;
};
struct pstore_erase_data {
- u64 id;
- enum pstore_type_id type;
- int count;
- struct timespec time;
+ struct pstore_record *record;
efi_char16_t *name;
};
@@ -315,8 +292,9 @@
* Check if an old format, which doesn't support
* holding multiple logs, remains.
*/
- sprintf(name_old, "dump-type%u-%u-%lu", ed->type,
- (unsigned int)ed->id, ed->time.tv_sec);
+ snprintf(name_old, sizeof(name_old), "dump-type%u-%u-%lu",
+ ed->record->type, (unsigned int)ed->record->id,
+ ed->record->time.tv_sec);
for (i = 0; i < DUMP_NAME_LEN; i++)
efi_name_old[i] = name_old[i];
@@ -341,8 +319,7 @@
return 1;
}
-static int efi_pstore_erase(enum pstore_type_id type, u64 id, int count,
- struct timespec time, struct pstore_info *psi)
+static int efi_pstore_erase(struct pstore_record *record)
{
struct pstore_erase_data edata;
struct efivar_entry *entry = NULL;
@@ -351,17 +328,16 @@
int found, i;
unsigned int part;
- do_div(id, 1000);
- part = do_div(id, 100);
- sprintf(name, "dump-type%u-%u-%d-%lu", type, part, count, time.tv_sec);
+ do_div(record->id, 1000);
+ part = do_div(record->id, 100);
+ snprintf(name, sizeof(name), "dump-type%u-%u-%d-%lu",
+ record->type, record->part, record->count,
+ record->time.tv_sec);
for (i = 0; i < DUMP_NAME_LEN; i++)
efi_name[i] = name[i];
- edata.id = part;
- edata.type = type;
- edata.count = count;
- edata.time = time;
+ edata.record = record;
edata.name = efi_name;
if (efivar_entry_iter_begin())
diff --git a/drivers/firmware/efi/libstub/arm-stub.c b/drivers/firmware/efi/libstub/arm-stub.c
index d4056c6..8181ac1 100644
--- a/drivers/firmware/efi/libstub/arm-stub.c
+++ b/drivers/firmware/efi/libstub/arm-stub.c
@@ -18,7 +18,27 @@
#include "efistub.h"
-bool __nokaslr;
+/*
+ * This is the base address at which to start allocating virtual memory ranges
+ * for UEFI Runtime Services. This is in the low TTBR0 range so that we can use
+ * any allocation we choose, and eliminate the risk of a conflict after kexec.
+ * The value chosen is the largest non-zero power of 2 suitable for this purpose
+ * both on 32-bit and 64-bit ARM CPUs, to maximize the likelihood that it can
+ * be mapped efficiently.
+ * Since 32-bit ARM could potentially execute with a 1G/3G user/kernel split,
+ * map everything below 1 GB. (512 MB is a reasonable upper bound for the
+ * entire footprint of the UEFI runtime services memory regions)
+ */
+#define EFI_RT_VIRTUAL_BASE SZ_512M
+#define EFI_RT_VIRTUAL_SIZE SZ_512M
+
+#ifdef CONFIG_ARM64
+# define EFI_RT_VIRTUAL_LIMIT TASK_SIZE_64
+#else
+# define EFI_RT_VIRTUAL_LIMIT TASK_SIZE
+#endif
+
+static u64 virtmap_base = EFI_RT_VIRTUAL_BASE;
efi_status_t efi_open_volume(efi_system_table_t *sys_table_arg,
void *__image, void **__fh)
@@ -118,8 +138,6 @@
if (sys_table->hdr.signature != EFI_SYSTEM_TABLE_SIGNATURE)
goto fail;
- pr_efi(sys_table, "Booting Linux Kernel...\n");
-
status = check_platform_features(sys_table);
if (status != EFI_SUCCESS)
goto fail;
@@ -153,17 +171,15 @@
goto fail;
}
- /* check whether 'nokaslr' was passed on the command line */
- if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) {
- static const u8 default_cmdline[] = CONFIG_CMDLINE;
- const u8 *str, *cmdline = cmdline_ptr;
+ if (IS_ENABLED(CONFIG_CMDLINE_EXTEND) ||
+ IS_ENABLED(CONFIG_CMDLINE_FORCE) ||
+ cmdline_size == 0)
+ efi_parse_options(CONFIG_CMDLINE);
- if (IS_ENABLED(CONFIG_CMDLINE_FORCE))
- cmdline = default_cmdline;
- str = strstr(cmdline, "nokaslr");
- if (str == cmdline || (str > cmdline && *(str - 1) == ' '))
- __nokaslr = true;
- }
+ if (!IS_ENABLED(CONFIG_CMDLINE_FORCE) && cmdline_size > 0)
+ efi_parse_options(cmdline_ptr);
+
+ pr_efi(sys_table, "Booting Linux Kernel...\n");
si = setup_graphics(sys_table);
@@ -176,10 +192,6 @@
goto fail_free_cmdline;
}
- status = efi_parse_options(cmdline_ptr);
- if (status != EFI_SUCCESS)
- pr_efi_err(sys_table, "Failed to parse EFI cmdline options\n");
-
secure_boot = efi_get_secureboot(sys_table);
/*
@@ -213,8 +225,9 @@
if (!fdt_addr)
pr_efi(sys_table, "Generating empty DTB\n");
- status = handle_cmdline_files(sys_table, image, cmdline_ptr,
- "initrd=", dram_base + SZ_512M,
+ status = handle_cmdline_files(sys_table, image, cmdline_ptr, "initrd=",
+ efi_get_max_initrd_addr(dram_base,
+ *image_addr),
(unsigned long *)&initrd_addr,
(unsigned long *)&initrd_size);
if (status != EFI_SUCCESS)
@@ -222,9 +235,29 @@
efi_random_get_seed(sys_table);
+ if (!nokaslr()) {
+ /*
+ * Randomize the base of the UEFI runtime services region.
+ * Preserve the 2 MB alignment of the region by taking a
+ * shift of 21 bit positions into account when scaling
+ * the headroom value using a 32-bit random value.
+ */
+ static const u64 headroom = EFI_RT_VIRTUAL_LIMIT -
+ EFI_RT_VIRTUAL_BASE -
+ EFI_RT_VIRTUAL_SIZE;
+ u32 rnd;
+
+ status = efi_get_random_bytes(sys_table, sizeof(rnd),
+ (u8 *)&rnd);
+ if (status == EFI_SUCCESS) {
+ virtmap_base = EFI_RT_VIRTUAL_BASE +
+ (((headroom >> 21) * rnd) >> (32 - 21));
+ }
+ }
+
new_fdt_addr = fdt_addr;
status = allocate_new_fdt_and_exit_boot(sys_table, handle,
- &new_fdt_addr, dram_base + MAX_FDT_OFFSET,
+ &new_fdt_addr, efi_get_max_fdt_addr(dram_base),
initrd_addr, initrd_size, cmdline_ptr,
fdt_addr, fdt_size);
@@ -251,18 +284,6 @@
return EFI_ERROR;
}
-/*
- * This is the base address at which to start allocating virtual memory ranges
- * for UEFI Runtime Services. This is in the low TTBR0 range so that we can use
- * any allocation we choose, and eliminate the risk of a conflict after kexec.
- * The value chosen is the largest non-zero power of 2 suitable for this purpose
- * both on 32-bit and 64-bit ARM CPUs, to maximize the likelihood that it can
- * be mapped efficiently.
- * Since 32-bit ARM could potentially execute with a 1G/3G user/kernel split,
- * map everything below 1 GB.
- */
-#define EFI_RT_VIRTUAL_BASE SZ_512M
-
static int cmp_mem_desc(const void *l, const void *r)
{
const efi_memory_desc_t *left = l, *right = r;
@@ -312,7 +333,7 @@
unsigned long desc_size, efi_memory_desc_t *runtime_map,
int *count)
{
- u64 efi_virt_base = EFI_RT_VIRTUAL_BASE;
+ u64 efi_virt_base = virtmap_base;
efi_memory_desc_t *in, *prev = NULL, *out = runtime_map;
int l;
diff --git a/drivers/firmware/efi/libstub/arm32-stub.c b/drivers/firmware/efi/libstub/arm32-stub.c
index e1f0b28..becbda4 100644
--- a/drivers/firmware/efi/libstub/arm32-stub.c
+++ b/drivers/firmware/efi/libstub/arm32-stub.c
@@ -9,6 +9,8 @@
#include <linux/efi.h>
#include <asm/efi.h>
+#include "efistub.h"
+
efi_status_t check_platform_features(efi_system_table_t *sys_table_arg)
{
int block;
@@ -63,27 +65,25 @@
efi_call_early(free_pool, si);
}
-efi_status_t handle_kernel_image(efi_system_table_t *sys_table,
- unsigned long *image_addr,
- unsigned long *image_size,
- unsigned long *reserve_addr,
- unsigned long *reserve_size,
- unsigned long dram_base,
- efi_loaded_image_t *image)
+static efi_status_t reserve_kernel_base(efi_system_table_t *sys_table_arg,
+ unsigned long dram_base,
+ unsigned long *reserve_addr,
+ unsigned long *reserve_size)
{
- unsigned long nr_pages;
- efi_status_t status;
- /* Use alloc_addr to tranlsate between types */
efi_physical_addr_t alloc_addr;
+ efi_memory_desc_t *memory_map;
+ unsigned long nr_pages, map_size, desc_size, buff_size;
+ efi_status_t status;
+ unsigned long l;
- /*
- * Verify that the DRAM base address is compatible with the ARM
- * boot protocol, which determines the base of DRAM by masking
- * off the low 27 bits of the address at which the zImage is
- * loaded. These assumptions are made by the decompressor,
- * before any memory map is available.
- */
- dram_base = round_up(dram_base, SZ_128M);
+ struct efi_boot_memmap map = {
+ .map = &memory_map,
+ .map_size = &map_size,
+ .desc_size = &desc_size,
+ .desc_ver = NULL,
+ .key_ptr = NULL,
+ .buff_size = &buff_size,
+ };
/*
* Reserve memory for the uncompressed kernel image. This is
@@ -94,18 +94,128 @@
* Do this very early, as prints can cause memory allocations
* that may conflict with this.
*/
- alloc_addr = dram_base;
- *reserve_size = MAX_UNCOMP_KERNEL_SIZE;
- nr_pages = round_up(*reserve_size, EFI_PAGE_SIZE) / EFI_PAGE_SIZE;
- status = sys_table->boottime->allocate_pages(EFI_ALLOCATE_ADDRESS,
- EFI_LOADER_DATA,
- nr_pages, &alloc_addr);
+ alloc_addr = dram_base + MAX_UNCOMP_KERNEL_SIZE;
+ nr_pages = MAX_UNCOMP_KERNEL_SIZE / EFI_PAGE_SIZE;
+ status = efi_call_early(allocate_pages, EFI_ALLOCATE_MAX_ADDRESS,
+ EFI_BOOT_SERVICES_DATA, nr_pages, &alloc_addr);
+ if (status == EFI_SUCCESS) {
+ if (alloc_addr == dram_base) {
+ *reserve_addr = alloc_addr;
+ *reserve_size = MAX_UNCOMP_KERNEL_SIZE;
+ return EFI_SUCCESS;
+ }
+ /*
+ * If we end up here, the allocation succeeded but starts below
+ * dram_base. This can only occur if the real base of DRAM is
+ * not a multiple of 128 MB, in which case dram_base will have
+ * been rounded up. Since this implies that a part of the region
+ * was already occupied, we need to fall through to the code
+ * below to ensure that the existing allocations don't conflict.
+ * For this reason, we use EFI_BOOT_SERVICES_DATA above and not
+ * EFI_LOADER_DATA, which we wouldn't able to distinguish from
+ * allocations that we want to disallow.
+ */
+ }
+
+ /*
+ * If the allocation above failed, we may still be able to proceed:
+ * if the only allocations in the region are of types that will be
+ * released to the OS after ExitBootServices(), the decompressor can
+ * safely overwrite them.
+ */
+ status = efi_get_memory_map(sys_table_arg, &map);
if (status != EFI_SUCCESS) {
- *reserve_size = 0;
+ pr_efi_err(sys_table_arg,
+ "reserve_kernel_base(): Unable to retrieve memory map.\n");
+ return status;
+ }
+
+ for (l = 0; l < map_size; l += desc_size) {
+ efi_memory_desc_t *desc;
+ u64 start, end;
+
+ desc = (void *)memory_map + l;
+ start = desc->phys_addr;
+ end = start + desc->num_pages * EFI_PAGE_SIZE;
+
+ /* Skip if entry does not intersect with region */
+ if (start >= dram_base + MAX_UNCOMP_KERNEL_SIZE ||
+ end <= dram_base)
+ continue;
+
+ switch (desc->type) {
+ case EFI_BOOT_SERVICES_CODE:
+ case EFI_BOOT_SERVICES_DATA:
+ /* Ignore types that are released to the OS anyway */
+ continue;
+
+ case EFI_CONVENTIONAL_MEMORY:
+ /*
+ * Reserve the intersection between this entry and the
+ * region.
+ */
+ start = max(start, (u64)dram_base);
+ end = min(end, (u64)dram_base + MAX_UNCOMP_KERNEL_SIZE);
+
+ status = efi_call_early(allocate_pages,
+ EFI_ALLOCATE_ADDRESS,
+ EFI_LOADER_DATA,
+ (end - start) / EFI_PAGE_SIZE,
+ &start);
+ if (status != EFI_SUCCESS) {
+ pr_efi_err(sys_table_arg,
+ "reserve_kernel_base(): alloc failed.\n");
+ goto out;
+ }
+ break;
+
+ case EFI_LOADER_CODE:
+ case EFI_LOADER_DATA:
+ /*
+ * These regions may be released and reallocated for
+ * another purpose (including EFI_RUNTIME_SERVICE_DATA)
+ * at any time during the execution of the OS loader,
+ * so we cannot consider them as safe.
+ */
+ default:
+ /*
+ * Treat any other allocation in the region as unsafe */
+ status = EFI_OUT_OF_RESOURCES;
+ goto out;
+ }
+ }
+
+ status = EFI_SUCCESS;
+out:
+ efi_call_early(free_pool, memory_map);
+ return status;
+}
+
+efi_status_t handle_kernel_image(efi_system_table_t *sys_table,
+ unsigned long *image_addr,
+ unsigned long *image_size,
+ unsigned long *reserve_addr,
+ unsigned long *reserve_size,
+ unsigned long dram_base,
+ efi_loaded_image_t *image)
+{
+ efi_status_t status;
+
+ /*
+ * Verify that the DRAM base address is compatible with the ARM
+ * boot protocol, which determines the base of DRAM by masking
+ * off the low 27 bits of the address at which the zImage is
+ * loaded. These assumptions are made by the decompressor,
+ * before any memory map is available.
+ */
+ dram_base = round_up(dram_base, SZ_128M);
+
+ status = reserve_kernel_base(sys_table, dram_base, reserve_addr,
+ reserve_size);
+ if (status != EFI_SUCCESS) {
pr_efi_err(sys_table, "Unable to allocate memory for uncompressed kernel.\n");
return status;
}
- *reserve_addr = alloc_addr;
/*
* Relocate the zImage, so that it appears in the lowest 128 MB
diff --git a/drivers/firmware/efi/libstub/arm64-stub.c b/drivers/firmware/efi/libstub/arm64-stub.c
index eae693e..b4c2589 100644
--- a/drivers/firmware/efi/libstub/arm64-stub.c
+++ b/drivers/firmware/efi/libstub/arm64-stub.c
@@ -16,8 +16,6 @@
#include "efistub.h"
-extern bool __nokaslr;
-
efi_status_t check_platform_features(efi_system_table_t *sys_table_arg)
{
u64 tg;
@@ -52,7 +50,7 @@
u64 phys_seed = 0;
if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) {
- if (!__nokaslr) {
+ if (!nokaslr()) {
status = efi_get_random_bytes(sys_table_arg,
sizeof(phys_seed),
(u8 *)&phys_seed);
diff --git a/drivers/firmware/efi/libstub/efi-stub-helper.c b/drivers/firmware/efi/libstub/efi-stub-helper.c
index 919822b..b018436 100644
--- a/drivers/firmware/efi/libstub/efi-stub-helper.c
+++ b/drivers/firmware/efi/libstub/efi-stub-helper.c
@@ -32,6 +32,18 @@
static unsigned long __chunk_size = EFI_READ_CHUNK_SIZE;
+static int __section(.data) __nokaslr;
+static int __section(.data) __quiet;
+
+int __pure nokaslr(void)
+{
+ return __nokaslr;
+}
+int __pure is_quiet(void)
+{
+ return __quiet;
+}
+
#define EFI_MMAP_NR_SLACK_SLOTS 8
struct file_info {
@@ -409,17 +421,17 @@
* environments, first in the early boot environment of the EFI boot
* stub, and subsequently during the kernel boot.
*/
-efi_status_t efi_parse_options(char *cmdline)
+efi_status_t efi_parse_options(char const *cmdline)
{
char *str;
- /*
- * Currently, the only efi= option we look for is 'nochunk', which
- * is intended to work around known issues on certain x86 UEFI
- * versions. So ignore for now on other architectures.
- */
- if (!IS_ENABLED(CONFIG_X86))
- return EFI_SUCCESS;
+ str = strstr(cmdline, "nokaslr");
+ if (str == cmdline || (str && str > cmdline && *(str - 1) == ' '))
+ __nokaslr = 1;
+
+ str = strstr(cmdline, "quiet");
+ if (str == cmdline || (str && str > cmdline && *(str - 1) == ' '))
+ __quiet = 1;
/*
* If no EFI parameters were specified on the cmdline we've got
@@ -436,14 +448,14 @@
* Remember, because efi= is also used by the kernel we need to
* skip over arguments we don't understand.
*/
- while (*str) {
+ while (*str && *str != ' ') {
if (!strncmp(str, "nochunk", 7)) {
str += strlen("nochunk");
__chunk_size = -1UL;
}
/* Group words together, delimited by "," */
- while (*str && *str != ',')
+ while (*str && *str != ' ' && *str != ',')
str++;
if (*str == ',')
diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/libstub/efistub.h
index 71c4d0e..83f268c 100644
--- a/drivers/firmware/efi/libstub/efistub.h
+++ b/drivers/firmware/efi/libstub/efistub.h
@@ -24,6 +24,15 @@
#define EFI_ALLOC_ALIGN EFI_PAGE_SIZE
#endif
+extern int __pure nokaslr(void);
+extern int __pure is_quiet(void);
+
+#define pr_efi(sys_table, msg) do { \
+ if (!is_quiet()) efi_printk(sys_table, "EFI stub: "msg); \
+} while (0)
+
+#define pr_efi_err(sys_table, msg) efi_printk(sys_table, "EFI stub: ERROR: "msg)
+
void efi_char16_printk(efi_system_table_t *, efi_char16_t *);
efi_status_t efi_open_volume(efi_system_table_t *sys_table_arg, void *__image,
diff --git a/drivers/firmware/efi/libstub/fdt.c b/drivers/firmware/efi/libstub/fdt.c
index 260c4b4..41f457b 100644
--- a/drivers/firmware/efi/libstub/fdt.c
+++ b/drivers/firmware/efi/libstub/fdt.c
@@ -206,6 +206,10 @@
return update_fdt_memmap(p->new_fdt_addr, map);
}
+#ifndef MAX_FDT_SIZE
+#define MAX_FDT_SIZE SZ_2M
+#endif
+
/*
* Allocate memory for a new FDT, then add EFI, commandline, and
* initrd related fields to the FDT. This routine increases the
@@ -233,7 +237,6 @@
u32 desc_ver;
unsigned long mmap_key;
efi_memory_desc_t *memory_map, *runtime_map;
- unsigned long new_fdt_size;
efi_status_t status;
int runtime_entry_count = 0;
struct efi_boot_memmap map;
@@ -262,41 +265,29 @@
"Exiting boot services and installing virtual address map...\n");
map.map = &memory_map;
+ status = efi_high_alloc(sys_table, MAX_FDT_SIZE, EFI_FDT_ALIGN,
+ new_fdt_addr, max_addr);
+ if (status != EFI_SUCCESS) {
+ pr_efi_err(sys_table,
+ "Unable to allocate memory for new device tree.\n");
+ goto fail;
+ }
+
/*
- * Estimate size of new FDT, and allocate memory for it. We
- * will allocate a bigger buffer if this ends up being too
- * small, so a rough guess is OK here.
+ * Now that we have done our final memory allocation (and free)
+ * we can get the memory map key needed for exit_boot_services().
*/
- new_fdt_size = fdt_size + EFI_PAGE_SIZE;
- while (1) {
- status = efi_high_alloc(sys_table, new_fdt_size, EFI_FDT_ALIGN,
- new_fdt_addr, max_addr);
- if (status != EFI_SUCCESS) {
- pr_efi_err(sys_table, "Unable to allocate memory for new device tree.\n");
- goto fail;
- }
+ status = efi_get_memory_map(sys_table, &map);
+ if (status != EFI_SUCCESS)
+ goto fail_free_new_fdt;
- status = update_fdt(sys_table,
- (void *)fdt_addr, fdt_size,
- (void *)*new_fdt_addr, new_fdt_size,
- cmdline_ptr, initrd_addr, initrd_size);
+ status = update_fdt(sys_table, (void *)fdt_addr, fdt_size,
+ (void *)*new_fdt_addr, MAX_FDT_SIZE, cmdline_ptr,
+ initrd_addr, initrd_size);
- /* Succeeding the first time is the expected case. */
- if (status == EFI_SUCCESS)
- break;
-
- if (status == EFI_BUFFER_TOO_SMALL) {
- /*
- * We need to allocate more space for the new
- * device tree, so free existing buffer that is
- * too small.
- */
- efi_free(sys_table, new_fdt_size, *new_fdt_addr);
- new_fdt_size += EFI_PAGE_SIZE;
- } else {
- pr_efi_err(sys_table, "Unable to construct new device tree.\n");
- goto fail_free_new_fdt;
- }
+ if (status != EFI_SUCCESS) {
+ pr_efi_err(sys_table, "Unable to construct new device tree.\n");
+ goto fail_free_new_fdt;
}
priv.runtime_map = runtime_map;
@@ -340,7 +331,7 @@
pr_efi_err(sys_table, "Exit boot services failed.\n");
fail_free_new_fdt:
- efi_free(sys_table, new_fdt_size, *new_fdt_addr);
+ efi_free(sys_table, MAX_FDT_SIZE, *new_fdt_addr);
fail:
sys_table->boottime->free_pool(runtime_map);
diff --git a/drivers/firmware/efi/libstub/secureboot.c b/drivers/firmware/efi/libstub/secureboot.c
index 5da36e5..8c34d50 100644
--- a/drivers/firmware/efi/libstub/secureboot.c
+++ b/drivers/firmware/efi/libstub/secureboot.c
@@ -12,6 +12,8 @@
#include <linux/efi.h>
#include <asm/efi.h>
+#include "efistub.h"
+
/* BIOS variables */
static const efi_guid_t efi_variable_guid = EFI_GLOBAL_VARIABLE_GUID;
static const efi_char16_t const efi_SecureBoot_name[] = {
diff --git a/drivers/gpu/drm/sti/sti_compositor.c b/drivers/gpu/drm/sti/sti_compositor.c
index f62041f..11d4e88 100644
--- a/drivers/gpu/drm/sti/sti_compositor.c
+++ b/drivers/gpu/drm/sti/sti_compositor.c
@@ -89,7 +89,7 @@
/* Nothing to do, wait for the second round */
break;
default:
- DRM_ERROR("Unknow subdev compoment type\n");
+ DRM_ERROR("Unknown subdev component type\n");
return 1;
}
}
diff --git a/drivers/gpu/drm/udl/udl_fb.c b/drivers/gpu/drm/udl/udl_fb.c
index 8e8d60e..b1f0d52 100644
--- a/drivers/gpu/drm/udl/udl_fb.c
+++ b/drivers/gpu/drm/udl/udl_fb.c
@@ -37,7 +37,7 @@
};
#define DL_ALIGN_UP(x, a) ALIGN(x, a)
-#define DL_ALIGN_DOWN(x, a) ALIGN(x-(a-1), a)
+#define DL_ALIGN_DOWN(x, a) ALIGN_DOWN(x, a)
/** Read the red component (0..255) of a 32 bpp colour. */
#define DLO_RGB_GETRED(col) (uint8_t)((col) & 0xFF)
diff --git a/drivers/hid/Kconfig b/drivers/hid/Kconfig
index 8c54cb8..fe40e5e 100644
--- a/drivers/hid/Kconfig
+++ b/drivers/hid/Kconfig
@@ -98,6 +98,18 @@
---help---
Support for A4 tech X5 and WOP-35 / Trust 450L mice.
+config HID_ACCUTOUCH
+ tristate "Accutouch touch device"
+ depends on USB_HID
+ ---help---
+ This selects a driver for the Accutouch 2216 touch controller.
+
+ The driver works around a problem in the reported device capabilities
+ which causes userspace to detect the device as a mouse rather than
+ a touchscreen.
+
+ Say Y here if you have a Accutouch 2216 touch controller.
+
config HID_ACRUX
tristate "ACRUX game controller support"
depends on HID
@@ -136,13 +148,16 @@
config HID_ASUS
tristate "Asus"
- depends on I2C_HID
+ depends on LEDS_CLASS
---help---
- Support for Asus notebook built-in keyboard and touchpad via i2c.
+ Support for Asus notebook built-in keyboard and touchpad via i2c, and
+ the Asus Republic of Gamers laptop keyboard special keys.
Supported devices:
- EeeBook X205TA
- VivoBook E200HA
+ - GL553V series
+ - GL753V series
config HID_AUREAL
tristate "Aureal"
@@ -215,7 +230,8 @@
config HID_CP2112
tristate "Silicon Labs CP2112 HID USB-to-SMBus Bridge support"
- depends on USB_HID && I2C && GPIOLIB && GPIOLIB_IRQCHIP
+ depends on USB_HID && I2C && GPIOLIB
+ select GPIOLIB_IRQCHIP
---help---
Support for Silicon Labs CP2112 HID USB to SMBus Master Bridge.
This is a HID device driver which registers as an i2c adapter
@@ -441,6 +457,7 @@
config HID_LOGITECH_HIDPP
tristate "Logitech HID++ devices support"
depends on HID_LOGITECH
+ select POWER_SUPPLY
---help---
Support for Logitech devices relyingon the HID++ Logitech specification
@@ -581,6 +598,12 @@
To compile this driver as a module, choose M here: the
module will be called hid-multitouch.
+config HID_NTI
+ tristate "NTI keyboard adapters"
+ ---help---
+ Support for the "extra" Sun keyboard keys on keyboards attached
+ through Network Technologies USB-SUN keyboard adapters.
+
config HID_NTRIG
tristate "N-Trig touch screen"
depends on USB_HID
diff --git a/drivers/hid/Makefile b/drivers/hid/Makefile
index 4d111f2..fef027b 100644
--- a/drivers/hid/Makefile
+++ b/drivers/hid/Makefile
@@ -21,6 +21,7 @@
hid-wiimote-$(CONFIG_DEBUG_FS) += hid-wiimote-debug.o
obj-$(CONFIG_HID_A4TECH) += hid-a4tech.o
+obj-$(CONFIG_HID_ACCUTOUCH) += hid-accutouch.o
obj-$(CONFIG_HID_ALPS) += hid-alps.o
obj-$(CONFIG_HID_ACRUX) += hid-axff.o
obj-$(CONFIG_HID_APPLE) += hid-apple.o
@@ -62,6 +63,7 @@
obj-$(CONFIG_HID_MICROSOFT) += hid-microsoft.o
obj-$(CONFIG_HID_MONTEREY) += hid-monterey.o
obj-$(CONFIG_HID_MULTITOUCH) += hid-multitouch.o
+obj-$(CONFIG_HID_NTI) += hid-nti.o
obj-$(CONFIG_HID_NTRIG) += hid-ntrig.o
obj-$(CONFIG_HID_ORTEK) += hid-ortek.o
obj-$(CONFIG_HID_PRODIKEYS) += hid-prodikeys.o
diff --git a/drivers/hid/hid-accutouch.c b/drivers/hid/hid-accutouch.c
new file mode 100644
index 0000000..4e28716
--- /dev/null
+++ b/drivers/hid/hid-accutouch.c
@@ -0,0 +1,52 @@
+/*
+ * HID driver for Elo Accutouch touchscreens
+ *
+ * Copyright (c) 2016, Collabora Ltd.
+ * Copyright (c) 2016, General Electric Company
+ *
+ * based on hid-penmount.c
+ * Copyright (c) 2014 Christian Gmeiner <christian.gmeiner <at> gmail.com>
+ */
+
+/*
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ */
+
+#include <linux/hid.h>
+#include <linux/module.h>
+#include "hid-ids.h"
+
+static int accutouch_input_mapping(struct hid_device *hdev,
+ struct hid_input *hi,
+ struct hid_field *field,
+ struct hid_usage *usage,
+ unsigned long **bit, int *max)
+{
+ if ((usage->hid & HID_USAGE_PAGE) == HID_UP_BUTTON) {
+ hid_map_usage(hi, usage, bit, max, EV_KEY, BTN_TOUCH);
+ return 1;
+ }
+
+ return 0;
+}
+
+static const struct hid_device_id accutouch_devices[] = {
+ { HID_USB_DEVICE(USB_VENDOR_ID_ELO, USB_DEVICE_ID_ELO_ACCUTOUCH_2216) },
+ { }
+};
+MODULE_DEVICE_TABLE(hid, accutouch_devices);
+
+static struct hid_driver accutouch_driver = {
+ .name = "hid-accutouch",
+ .id_table = accutouch_devices,
+ .input_mapping = accutouch_input_mapping,
+};
+
+module_hid_driver(accutouch_driver);
+
+MODULE_AUTHOR("Martyn Welch <martyn.welch@collabora.co.uk");
+MODULE_DESCRIPTION("Elo Accutouch HID TouchScreen driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/hid/hid-asus.c b/drivers/hid/hid-asus.c
index 70b12f8..16df6cc9 100644
--- a/drivers/hid/hid-asus.c
+++ b/drivers/hid/hid-asus.c
@@ -40,8 +40,12 @@
#define FEATURE_REPORT_ID 0x0d
#define INPUT_REPORT_ID 0x5d
+#define FEATURE_KBD_REPORT_ID 0x5a
#define INPUT_REPORT_SIZE 28
+#define FEATURE_KBD_REPORT_SIZE 16
+
+#define SUPPORT_KBD_BACKLIGHT BIT(0)
#define MAX_CONTACTS 5
@@ -63,18 +67,31 @@
#define QUIRK_NO_INIT_REPORTS BIT(1)
#define QUIRK_SKIP_INPUT_MAPPING BIT(2)
#define QUIRK_IS_MULTITOUCH BIT(3)
+#define QUIRK_NO_CONSUMER_USAGES BIT(4)
+#define QUIRK_USE_KBD_BACKLIGHT BIT(5)
-#define KEYBOARD_QUIRKS (QUIRK_FIX_NOTEBOOK_REPORT | \
- QUIRK_NO_INIT_REPORTS)
-#define TOUCHPAD_QUIRKS (QUIRK_NO_INIT_REPORTS | \
+#define I2C_KEYBOARD_QUIRKS (QUIRK_FIX_NOTEBOOK_REPORT | \
+ QUIRK_NO_INIT_REPORTS | \
+ QUIRK_NO_CONSUMER_USAGES)
+#define I2C_TOUCHPAD_QUIRKS (QUIRK_NO_INIT_REPORTS | \
QUIRK_SKIP_INPUT_MAPPING | \
QUIRK_IS_MULTITOUCH)
#define TRKID_SGN ((TRKID_MAX + 1) >> 1)
+struct asus_kbd_leds {
+ struct led_classdev cdev;
+ struct hid_device *hdev;
+ struct work_struct work;
+ unsigned int brightness;
+ bool removed;
+};
+
struct asus_drvdata {
unsigned long quirks;
struct input_dev *input;
+ struct asus_kbd_leds *kbd_backlight;
+ bool enable_backlight;
};
static void asus_report_contact_down(struct input_dev *input,
@@ -169,6 +186,148 @@
return 0;
}
+static int asus_kbd_set_report(struct hid_device *hdev, u8 *buf, size_t buf_size)
+{
+ unsigned char *dmabuf;
+ int ret;
+
+ dmabuf = kmemdup(buf, buf_size, GFP_KERNEL);
+ if (!dmabuf)
+ return -ENOMEM;
+
+ ret = hid_hw_raw_request(hdev, FEATURE_KBD_REPORT_ID, dmabuf,
+ buf_size, HID_FEATURE_REPORT,
+ HID_REQ_SET_REPORT);
+ kfree(dmabuf);
+
+ return ret;
+}
+
+static int asus_kbd_init(struct hid_device *hdev)
+{
+ u8 buf[] = { FEATURE_KBD_REPORT_ID, 0x41, 0x53, 0x55, 0x53, 0x20, 0x54,
+ 0x65, 0x63, 0x68, 0x2e, 0x49, 0x6e, 0x63, 0x2e, 0x00 };
+ int ret;
+
+ ret = asus_kbd_set_report(hdev, buf, sizeof(buf));
+ if (ret < 0)
+ hid_err(hdev, "Asus failed to send init command: %d\n", ret);
+
+ return ret;
+}
+
+static int asus_kbd_get_functions(struct hid_device *hdev,
+ unsigned char *kbd_func)
+{
+ u8 buf[] = { FEATURE_KBD_REPORT_ID, 0x05, 0x20, 0x31, 0x00, 0x08 };
+ u8 *readbuf;
+ int ret;
+
+ ret = asus_kbd_set_report(hdev, buf, sizeof(buf));
+ if (ret < 0) {
+ hid_err(hdev, "Asus failed to send configuration command: %d\n", ret);
+ return ret;
+ }
+
+ readbuf = kzalloc(FEATURE_KBD_REPORT_SIZE, GFP_KERNEL);
+ if (!readbuf)
+ return -ENOMEM;
+
+ ret = hid_hw_raw_request(hdev, FEATURE_KBD_REPORT_ID, readbuf,
+ FEATURE_KBD_REPORT_SIZE, HID_FEATURE_REPORT,
+ HID_REQ_GET_REPORT);
+ if (ret < 0) {
+ hid_err(hdev, "Asus failed to request functions: %d\n", ret);
+ kfree(readbuf);
+ return ret;
+ }
+
+ *kbd_func = readbuf[6];
+
+ kfree(readbuf);
+ return ret;
+}
+
+static void asus_kbd_backlight_set(struct led_classdev *led_cdev,
+ enum led_brightness brightness)
+{
+ struct asus_kbd_leds *led = container_of(led_cdev, struct asus_kbd_leds,
+ cdev);
+ if (led->brightness == brightness)
+ return;
+
+ led->brightness = brightness;
+ schedule_work(&led->work);
+}
+
+static enum led_brightness asus_kbd_backlight_get(struct led_classdev *led_cdev)
+{
+ struct asus_kbd_leds *led = container_of(led_cdev, struct asus_kbd_leds,
+ cdev);
+
+ return led->brightness;
+}
+
+static void asus_kbd_backlight_work(struct work_struct *work)
+{
+ struct asus_kbd_leds *led = container_of(work, struct asus_kbd_leds, work);
+ u8 buf[] = { FEATURE_KBD_REPORT_ID, 0xba, 0xc5, 0xc4, 0x00 };
+ int ret;
+
+ if (led->removed)
+ return;
+
+ buf[4] = led->brightness;
+
+ ret = asus_kbd_set_report(led->hdev, buf, sizeof(buf));
+ if (ret < 0)
+ hid_err(led->hdev, "Asus failed to set keyboard backlight: %d\n", ret);
+}
+
+static int asus_kbd_register_leds(struct hid_device *hdev)
+{
+ struct asus_drvdata *drvdata = hid_get_drvdata(hdev);
+ unsigned char kbd_func;
+ int ret;
+
+ /* Initialize keyboard */
+ ret = asus_kbd_init(hdev);
+ if (ret < 0)
+ return ret;
+
+ /* Get keyboard functions */
+ ret = asus_kbd_get_functions(hdev, &kbd_func);
+ if (ret < 0)
+ return ret;
+
+ /* Check for backlight support */
+ if (!(kbd_func & SUPPORT_KBD_BACKLIGHT))
+ return -ENODEV;
+
+ drvdata->kbd_backlight = devm_kzalloc(&hdev->dev,
+ sizeof(struct asus_kbd_leds),
+ GFP_KERNEL);
+ if (!drvdata->kbd_backlight)
+ return -ENOMEM;
+
+ drvdata->kbd_backlight->removed = false;
+ drvdata->kbd_backlight->brightness = 0;
+ drvdata->kbd_backlight->hdev = hdev;
+ drvdata->kbd_backlight->cdev.name = "asus::kbd_backlight";
+ drvdata->kbd_backlight->cdev.max_brightness = 3;
+ drvdata->kbd_backlight->cdev.brightness_set = asus_kbd_backlight_set;
+ drvdata->kbd_backlight->cdev.brightness_get = asus_kbd_backlight_get;
+ INIT_WORK(&drvdata->kbd_backlight->work, asus_kbd_backlight_work);
+
+ ret = devm_led_classdev_register(&hdev->dev, &drvdata->kbd_backlight->cdev);
+ if (ret < 0) {
+ /* No need to have this still around */
+ devm_kfree(&hdev->dev, drvdata->kbd_backlight);
+ }
+
+ return ret;
+}
+
static int asus_input_configured(struct hid_device *hdev, struct hid_input *hi)
{
struct input_dev *input = hi->input;
@@ -196,9 +355,14 @@
drvdata->input = input;
+ if (drvdata->enable_backlight && asus_kbd_register_leds(hdev))
+ hid_warn(hdev, "Failed to initialize backlight.\n");
+
return 0;
}
+#define asus_map_key_clear(c) hid_map_usage_clear(hi, usage, bit, \
+ max, EV_KEY, (c))
static int asus_input_mapping(struct hid_device *hdev,
struct hid_input *hi, struct hid_field *field,
struct hid_usage *usage, unsigned long **bit,
@@ -213,6 +377,65 @@
return -1;
}
+ /* ASUS-specific keyboard hotkeys */
+ if ((usage->hid & HID_USAGE_PAGE) == 0xff310000) {
+ set_bit(EV_REP, hi->input->evbit);
+ switch (usage->hid & HID_USAGE) {
+ case 0x10: asus_map_key_clear(KEY_BRIGHTNESSDOWN); break;
+ case 0x20: asus_map_key_clear(KEY_BRIGHTNESSUP); break;
+ case 0x35: asus_map_key_clear(KEY_DISPLAY_OFF); break;
+ case 0x6c: asus_map_key_clear(KEY_SLEEP); break;
+ case 0x82: asus_map_key_clear(KEY_CAMERA); break;
+ case 0x88: asus_map_key_clear(KEY_RFKILL); break;
+ case 0xb5: asus_map_key_clear(KEY_CALC); break;
+ case 0xc4: asus_map_key_clear(KEY_KBDILLUMUP); break;
+ case 0xc5: asus_map_key_clear(KEY_KBDILLUMDOWN); break;
+
+ /* ASUS touchpad toggle */
+ case 0x6b: asus_map_key_clear(KEY_F21); break;
+
+ /* ROG key */
+ case 0x38: asus_map_key_clear(KEY_PROG1); break;
+
+ /* Fn+C ASUS Splendid */
+ case 0xba: asus_map_key_clear(KEY_PROG2); break;
+
+ /* Fn+Space Power4Gear Hybrid */
+ case 0x5c: asus_map_key_clear(KEY_PROG3); break;
+
+ default:
+ /* ASUS lazily declares 256 usages, ignore the rest,
+ * as some make the keyboard appear as a pointer device. */
+ return -1;
+ }
+
+ /*
+ * Check and enable backlight only on devices with UsagePage ==
+ * 0xff31 to avoid initializing the keyboard firmware multiple
+ * times on devices with multiple HID descriptors but same
+ * PID/VID.
+ */
+ if (drvdata->quirks & QUIRK_USE_KBD_BACKLIGHT)
+ drvdata->enable_backlight = true;
+
+ return 1;
+ }
+
+ if (drvdata->quirks & QUIRK_NO_CONSUMER_USAGES &&
+ (usage->hid & HID_USAGE_PAGE) == HID_UP_CONSUMER) {
+ switch (usage->hid & HID_USAGE) {
+ case 0xe2: /* Mute */
+ case 0xe9: /* Volume up */
+ case 0xea: /* Volume down */
+ return 0;
+ default:
+ /* Ignore dummy Consumer usages which make the
+ * keyboard incorrectly appear as a pointer device.
+ */
+ return -1;
+ }
+ }
+
return 0;
}
@@ -305,6 +528,16 @@
return ret;
}
+static void asus_remove(struct hid_device *hdev)
+{
+ struct asus_drvdata *drvdata = hid_get_drvdata(hdev);
+
+ if (drvdata->kbd_backlight) {
+ drvdata->kbd_backlight->removed = true;
+ cancel_work_sync(&drvdata->kbd_backlight->work);
+ }
+}
+
static __u8 *asus_report_fixup(struct hid_device *hdev, __u8 *rdesc,
unsigned int *rsize)
{
@@ -320,9 +553,13 @@
static const struct hid_device_id asus_devices[] = {
{ HID_I2C_DEVICE(USB_VENDOR_ID_ASUSTEK,
- USB_DEVICE_ID_ASUSTEK_NOTEBOOK_KEYBOARD), KEYBOARD_QUIRKS},
+ USB_DEVICE_ID_ASUSTEK_I2C_KEYBOARD), I2C_KEYBOARD_QUIRKS},
{ HID_I2C_DEVICE(USB_VENDOR_ID_ASUSTEK,
- USB_DEVICE_ID_ASUSTEK_TOUCHPAD), TOUCHPAD_QUIRKS },
+ USB_DEVICE_ID_ASUSTEK_I2C_TOUCHPAD), I2C_TOUCHPAD_QUIRKS },
+ { HID_USB_DEVICE(USB_VENDOR_ID_ASUSTEK,
+ USB_DEVICE_ID_ASUSTEK_ROG_KEYBOARD1) },
+ { HID_USB_DEVICE(USB_VENDOR_ID_ASUSTEK,
+ USB_DEVICE_ID_ASUSTEK_ROG_KEYBOARD2), QUIRK_USE_KBD_BACKLIGHT },
{ }
};
MODULE_DEVICE_TABLE(hid, asus_devices);
@@ -332,6 +569,7 @@
.id_table = asus_devices,
.report_fixup = asus_report_fixup,
.probe = asus_probe,
+ .remove = asus_remove,
.input_mapping = asus_input_mapping,
.input_configured = asus_input_configured,
#ifdef CONFIG_PM
diff --git a/drivers/hid/hid-core.c b/drivers/hid/hid-core.c
index d162f0d..37084b6 100644
--- a/drivers/hid/hid-core.c
+++ b/drivers/hid/hid-core.c
@@ -1694,7 +1694,7 @@
len += sprintf(buf + len, "input");
if (hdev->claimed & HID_CLAIMED_HIDDEV)
len += sprintf(buf + len, "%shiddev%d", len ? "," : "",
- hdev->minor);
+ ((struct hiddev *)hdev->hiddev)->minor);
if (hdev->claimed & HID_CLAIMED_HIDRAW)
len += sprintf(buf + len, "%shidraw%d", len ? "," : "",
((struct hidraw *)hdev->hidraw)->minor);
@@ -1851,8 +1851,10 @@
{ HID_USB_DEVICE(USB_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_MAGIC_KEYBOARD_ANSI) },
{ HID_USB_DEVICE(USB_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_FOUNTAIN_TP_ONLY) },
{ HID_USB_DEVICE(USB_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_GEYSER1_TP_ONLY) },
- { HID_I2C_DEVICE(USB_VENDOR_ID_ASUSTEK, USB_DEVICE_ID_ASUSTEK_NOTEBOOK_KEYBOARD) },
- { HID_I2C_DEVICE(USB_VENDOR_ID_ASUSTEK, USB_DEVICE_ID_ASUSTEK_TOUCHPAD) },
+ { HID_I2C_DEVICE(USB_VENDOR_ID_ASUSTEK, USB_DEVICE_ID_ASUSTEK_I2C_KEYBOARD) },
+ { HID_I2C_DEVICE(USB_VENDOR_ID_ASUSTEK, USB_DEVICE_ID_ASUSTEK_I2C_TOUCHPAD) },
+ { HID_USB_DEVICE(USB_VENDOR_ID_ASUSTEK, USB_DEVICE_ID_ASUSTEK_ROG_KEYBOARD1) },
+ { HID_USB_DEVICE(USB_VENDOR_ID_ASUSTEK, USB_DEVICE_ID_ASUSTEK_ROG_KEYBOARD2) },
{ HID_USB_DEVICE(USB_VENDOR_ID_AUREAL, USB_DEVICE_ID_AUREAL_W01RN) },
{ HID_USB_DEVICE(USB_VENDOR_ID_BELKIN, USB_DEVICE_ID_FLIP_KVM) },
{ HID_USB_DEVICE(USB_VENDOR_ID_BETOP_2185BFM, 0x2208) },
@@ -1891,6 +1893,7 @@
{ HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_ELECOM, USB_DEVICE_ID_ELECOM_BM084) },
{ HID_USB_DEVICE(USB_VENDOR_ID_ELO, 0x0009) },
{ HID_USB_DEVICE(USB_VENDOR_ID_ELO, 0x0030) },
+ { HID_USB_DEVICE(USB_VENDOR_ID_ELO, USB_DEVICE_ID_ELO_ACCUTOUCH_2216) },
{ HID_USB_DEVICE(USB_VENDOR_ID_EMS, USB_DEVICE_ID_EMS_TRIO_LINKER_PLUS_II) },
{ HID_USB_DEVICE(USB_VENDOR_ID_EZKEY, USB_DEVICE_ID_BTC_8193) },
{ HID_USB_DEVICE(USB_VENDOR_ID_GAMERON, USB_DEVICE_ID_GAMERON_DUAL_PSX_ADAPTOR) },
@@ -1991,6 +1994,7 @@
{ HID_USB_DEVICE(USB_VENDOR_ID_MICROSOFT, USB_DEVICE_ID_MS_POWER_COVER) },
{ HID_USB_DEVICE(USB_VENDOR_ID_MONTEREY, USB_DEVICE_ID_GENIUS_KB29E) },
{ HID_USB_DEVICE(USB_VENDOR_ID_MSI, USB_DEVICE_ID_MSI_GT683R_LED_PANEL) },
+ { HID_USB_DEVICE(USB_VENDOR_ID_NTI, USB_DEVICE_ID_USB_SUN) },
{ HID_USB_DEVICE(USB_VENDOR_ID_NTRIG, USB_DEVICE_ID_NTRIG_TOUCH_SCREEN) },
{ HID_USB_DEVICE(USB_VENDOR_ID_NTRIG, USB_DEVICE_ID_NTRIG_TOUCH_SCREEN_1) },
{ HID_USB_DEVICE(USB_VENDOR_ID_NTRIG, USB_DEVICE_ID_NTRIG_TOUCH_SCREEN_2) },
diff --git a/drivers/hid/hid-cp2112.c b/drivers/hid/hid-cp2112.c
index b22d0f8..078026f 100644
--- a/drivers/hid/hid-cp2112.c
+++ b/drivers/hid/hid-cp2112.c
@@ -27,6 +27,7 @@
#include <linux/gpio.h>
#include <linux/gpio/driver.h>
#include <linux/hid.h>
+#include <linux/hidraw.h>
#include <linux/i2c.h>
#include <linux/module.h>
#include <linux/nls.h>
@@ -1297,7 +1298,8 @@
dev->adap.algo_data = dev;
dev->adap.dev.parent = &hdev->dev;
snprintf(dev->adap.name, sizeof(dev->adap.name),
- "CP2112 SMBus Bridge on hiddev%d", hdev->minor);
+ "CP2112 SMBus Bridge on hidraw%d",
+ ((struct hidraw *)hdev->hidraw)->minor);
dev->hwversion = buf[2];
init_waitqueue_head(&dev->wait);
diff --git a/drivers/hid/hid-debug.c b/drivers/hid/hid-debug.c
index c6c9c51..5271db5 100644
--- a/drivers/hid/hid-debug.c
+++ b/drivers/hid/hid-debug.c
@@ -140,9 +140,11 @@
{0, 0x03, "LightPen"},
{0, 0x04, "TouchScreen"},
{0, 0x05, "TouchPad"},
+ {0, 0x0e, "DeviceConfiguration"},
{0, 0x20, "Stylus"},
{0, 0x21, "Puck"},
{0, 0x22, "Finger"},
+ {0, 0x23, "DeviceSettings"},
{0, 0x30, "TipPressure"},
{0, 0x31, "BarrelPressure"},
{0, 0x32, "InRange"},
diff --git a/drivers/hid/hid-ids.h b/drivers/hid/hid-ids.h
index b26c030..643390b 100644
--- a/drivers/hid/hid-ids.h
+++ b/drivers/hid/hid-ids.h
@@ -173,8 +173,10 @@
#define USB_VENDOR_ID_ASUSTEK 0x0b05
#define USB_DEVICE_ID_ASUSTEK_LCM 0x1726
#define USB_DEVICE_ID_ASUSTEK_LCM2 0x175b
-#define USB_DEVICE_ID_ASUSTEK_NOTEBOOK_KEYBOARD 0x8585
-#define USB_DEVICE_ID_ASUSTEK_TOUCHPAD 0x0101
+#define USB_DEVICE_ID_ASUSTEK_I2C_KEYBOARD 0x8585
+#define USB_DEVICE_ID_ASUSTEK_I2C_TOUCHPAD 0x0101
+#define USB_DEVICE_ID_ASUSTEK_ROG_KEYBOARD1 0x1854
+#define USB_DEVICE_ID_ASUSTEK_ROG_KEYBOARD2 0x1837
#define USB_VENDOR_ID_ATEN 0x0557
#define USB_DEVICE_ID_ATEN_UC100KM 0x2004
@@ -184,6 +186,7 @@
#define USB_DEVICE_ID_ATEN_4PORTKVMC 0x2208
#define USB_DEVICE_ID_ATEN_CS682 0x2213
#define USB_DEVICE_ID_ATEN_CS692 0x8021
+#define USB_DEVICE_ID_ATEN_CS1758 0x2220
#define USB_VENDOR_ID_ATMEL 0x03eb
#define USB_DEVICE_ID_ATMEL_MULTITOUCH 0x211c
@@ -366,6 +369,7 @@
#define USB_VENDOR_ID_ELO 0x04E7
#define USB_DEVICE_ID_ELO_TS2515 0x0022
#define USB_DEVICE_ID_ELO_TS2700 0x0020
+#define USB_DEVICE_ID_ELO_ACCUTOUCH_2216 0x0050
#define USB_VENDOR_ID_EMS 0x2006
#define USB_DEVICE_ID_EMS_TRIO_LINKER_PLUS_II 0x0118
@@ -548,6 +552,9 @@
#define USB_VENDOR_ID_IRTOUCHSYSTEMS 0x6615
#define USB_DEVICE_ID_IRTOUCH_INFRARED_USB 0x0070
+#define USB_VENDOR_ID_INNOMEDIA 0x1292
+#define USB_DEVICE_ID_INNEX_GENESIS_ATARI 0x4745
+
#define USB_VENDOR_ID_ITE 0x048d
#define USB_DEVICE_ID_ITE_LENOVO_YOGA 0x8386
#define USB_DEVICE_ID_ITE_LENOVO_YOGA2 0x8350
@@ -771,6 +778,9 @@
#define USB_DEVICE_ID_NOVATEK_PCT 0x0600
#define USB_DEVICE_ID_NOVATEK_MOUSE 0x1602
+#define USB_VENDOR_ID_NTI 0x0757
+#define USB_DEVICE_ID_USB_SUN 0x0a00
+
#define USB_VENDOR_ID_NTRIG 0x1b96
#define USB_DEVICE_ID_NTRIG_TOUCH_SCREEN 0x0001
#define USB_DEVICE_ID_NTRIG_TOUCH_SCREEN_1 0x0003
diff --git a/drivers/hid/hid-input.c b/drivers/hid/hid-input.c
index d05f903c..a1ebdd7 100644
--- a/drivers/hid/hid-input.c
+++ b/drivers/hid/hid-input.c
@@ -1150,18 +1150,26 @@
/*
* Ignore out-of-range values as per HID specification,
- * section 5.10 and 6.2.25.
+ * section 5.10 and 6.2.25, when NULL state bit is present.
+ * When it's not, clamp the value to match Microsoft's input
+ * driver as mentioned in "Required HID usages for digitizers":
+ * https://msdn.microsoft.com/en-us/library/windows/hardware/dn672278(v=vs.85).asp
*
* The logical_minimum < logical_maximum check is done so that we
* don't unintentionally discard values sent by devices which
* don't specify logical min and max.
*/
if ((field->flags & HID_MAIN_ITEM_VARIABLE) &&
- (field->logical_minimum < field->logical_maximum) &&
- (value < field->logical_minimum ||
- value > field->logical_maximum)) {
- dbg_hid("Ignoring out-of-range value %x\n", value);
- return;
+ (field->logical_minimum < field->logical_maximum)) {
+ if (field->flags & HID_MAIN_ITEM_NULL_STATE &&
+ (value < field->logical_minimum ||
+ value > field->logical_maximum)) {
+ dbg_hid("Ignoring out-of-range value %x\n", value);
+ return;
+ }
+ value = clamp(value,
+ field->logical_minimum,
+ field->logical_maximum);
}
/*
diff --git a/drivers/hid/hid-logitech-dj.c b/drivers/hid/hid-logitech-dj.c
index 5bc6d80..826fa1e 100644
--- a/drivers/hid/hid-logitech-dj.c
+++ b/drivers/hid/hid-logitech-dj.c
@@ -692,8 +692,12 @@
dbg_hid("%s:%s\n", __func__, hid->phys);
}
-static u8 unifying_name_query[] = {0x10, 0xff, 0x83, 0xb5, 0x40, 0x00, 0x00};
-static u8 unifying_name_answer[] = {0x11, 0xff, 0x83, 0xb5};
+/*
+ * Register 0xB5 is "pairing information". It is solely intended for the
+ * receiver, so do not overwrite the device index.
+ */
+static u8 unifying_pairing_query[] = {0x10, 0xff, 0x83, 0xb5};
+static u8 unifying_pairing_answer[] = {0x11, 0xff, 0x83, 0xb5};
static int logi_dj_ll_raw_request(struct hid_device *hid,
unsigned char reportnum, __u8 *buf,
@@ -712,9 +716,9 @@
/* special case where we should not overwrite
* the device_index */
- if (count == 7 && !memcmp(buf, unifying_name_query,
- sizeof(unifying_name_query)))
- buf[4] |= djdev->device_index - 1;
+ if (count == 7 && !memcmp(buf, unifying_pairing_query,
+ sizeof(unifying_pairing_query)))
+ buf[4] = (buf[4] & 0xf0) | (djdev->device_index - 1);
else
buf[1] = djdev->device_index;
return hid_hw_raw_request(djrcv_dev->hdev, reportnum, buf,
@@ -911,9 +915,8 @@
/* special case were the device wants to know its unifying
* name */
if (size == HIDPP_REPORT_LONG_LENGTH &&
- !memcmp(data, unifying_name_answer,
- sizeof(unifying_name_answer)) &&
- ((data[4] & 0xF0) == 0x40))
+ !memcmp(data, unifying_pairing_answer,
+ sizeof(unifying_pairing_answer)))
device_index = (data[4] & 0x0F) + 1;
else
return false;
diff --git a/drivers/hid/hid-logitech-hidpp.c b/drivers/hid/hid-logitech-hidpp.c
index 2e2515a..41b3946 100644
--- a/drivers/hid/hid-logitech-hidpp.c
+++ b/drivers/hid/hid-logitech-hidpp.c
@@ -56,15 +56,21 @@
#define HIDPP_QUIRK_CLASS_M560 BIT(1)
#define HIDPP_QUIRK_CLASS_K400 BIT(2)
#define HIDPP_QUIRK_CLASS_G920 BIT(3)
+#define HIDPP_QUIRK_CLASS_K750 BIT(4)
/* bits 2..20 are reserved for classes */
-#define HIDPP_QUIRK_CONNECT_EVENTS BIT(21)
+/* #define HIDPP_QUIRK_CONNECT_EVENTS BIT(21) disabled */
#define HIDPP_QUIRK_WTP_PHYSICAL_BUTTONS BIT(22)
#define HIDPP_QUIRK_NO_HIDINPUT BIT(23)
#define HIDPP_QUIRK_FORCE_OUTPUT_REPORTS BIT(24)
+#define HIDPP_QUIRK_UNIFYING BIT(25)
-#define HIDPP_QUIRK_DELAYED_INIT (HIDPP_QUIRK_NO_HIDINPUT | \
- HIDPP_QUIRK_CONNECT_EVENTS)
+#define HIDPP_QUIRK_DELAYED_INIT HIDPP_QUIRK_NO_HIDINPUT
+
+#define HIDPP_CAPABILITY_HIDPP10_BATTERY BIT(0)
+#define HIDPP_CAPABILITY_HIDPP20_BATTERY BIT(1)
+#define HIDPP_CAPABILITY_BATTERY_MILEAGE BIT(2)
+#define HIDPP_CAPABILITY_BATTERY_LEVEL_STATUS BIT(3)
/*
* There are two hidpp protocols in use, the first version hidpp10 is known
@@ -110,6 +116,18 @@
};
} __packed;
+struct hidpp_battery {
+ u8 feature_index;
+ u8 solar_feature_index;
+ struct power_supply_desc desc;
+ struct power_supply *ps;
+ char name[64];
+ int status;
+ int capacity;
+ int level;
+ bool online;
+};
+
struct hidpp_device {
struct hid_device *hid_dev;
struct mutex send_mutex;
@@ -128,8 +146,10 @@
struct input_dev *delayed_input;
unsigned long quirks;
-};
+ unsigned long capabilities;
+ struct hidpp_battery battery;
+};
/* HID++ 1.0 error codes */
#define HIDPP_ERROR 0x8f
@@ -380,15 +400,220 @@
#define HIDPP_SET_LONG_REGISTER 0x82
#define HIDPP_GET_LONG_REGISTER 0x83
-#define HIDPP_REG_PAIRING_INFORMATION 0xB5
-#define DEVICE_NAME 0x40
+#define HIDPP_REG_GENERAL 0x00
-static char *hidpp_get_unifying_name(struct hidpp_device *hidpp_dev)
+static int hidpp10_enable_battery_reporting(struct hidpp_device *hidpp_dev)
{
struct hidpp_report response;
int ret;
- /* hid-logitech-dj is in charge of setting the right device index */
- u8 params[1] = { DEVICE_NAME };
+ u8 params[3] = { 0 };
+
+ ret = hidpp_send_rap_command_sync(hidpp_dev,
+ REPORT_ID_HIDPP_SHORT,
+ HIDPP_GET_REGISTER,
+ HIDPP_REG_GENERAL,
+ NULL, 0, &response);
+ if (ret)
+ return ret;
+
+ memcpy(params, response.rap.params, 3);
+
+ /* Set the battery bit */
+ params[0] |= BIT(4);
+
+ return hidpp_send_rap_command_sync(hidpp_dev,
+ REPORT_ID_HIDPP_SHORT,
+ HIDPP_SET_REGISTER,
+ HIDPP_REG_GENERAL,
+ params, 3, &response);
+}
+
+#define HIDPP_REG_BATTERY_STATUS 0x07
+
+static int hidpp10_battery_status_map_level(u8 param)
+{
+ int level;
+
+ switch (param) {
+ case 1 ... 2:
+ level = POWER_SUPPLY_CAPACITY_LEVEL_CRITICAL;
+ break;
+ case 3 ... 4:
+ level = POWER_SUPPLY_CAPACITY_LEVEL_LOW;
+ break;
+ case 5 ... 6:
+ level = POWER_SUPPLY_CAPACITY_LEVEL_NORMAL;
+ break;
+ case 7:
+ level = POWER_SUPPLY_CAPACITY_LEVEL_HIGH;
+ break;
+ default:
+ level = POWER_SUPPLY_CAPACITY_LEVEL_UNKNOWN;
+ }
+
+ return level;
+}
+
+static int hidpp10_battery_status_map_status(u8 param)
+{
+ int status;
+
+ switch (param) {
+ case 0x00:
+ /* discharging (in use) */
+ status = POWER_SUPPLY_STATUS_DISCHARGING;
+ break;
+ case 0x21: /* (standard) charging */
+ case 0x24: /* fast charging */
+ case 0x25: /* slow charging */
+ status = POWER_SUPPLY_STATUS_CHARGING;
+ break;
+ case 0x26: /* topping charge */
+ case 0x22: /* charge complete */
+ status = POWER_SUPPLY_STATUS_FULL;
+ break;
+ case 0x20: /* unknown */
+ status = POWER_SUPPLY_STATUS_UNKNOWN;
+ break;
+ /*
+ * 0x01...0x1F = reserved (not charging)
+ * 0x23 = charging error
+ * 0x27..0xff = reserved
+ */
+ default:
+ status = POWER_SUPPLY_STATUS_NOT_CHARGING;
+ break;
+ }
+
+ return status;
+}
+
+static int hidpp10_query_battery_status(struct hidpp_device *hidpp)
+{
+ struct hidpp_report response;
+ int ret, status;
+
+ ret = hidpp_send_rap_command_sync(hidpp,
+ REPORT_ID_HIDPP_SHORT,
+ HIDPP_GET_REGISTER,
+ HIDPP_REG_BATTERY_STATUS,
+ NULL, 0, &response);
+ if (ret)
+ return ret;
+
+ hidpp->battery.level =
+ hidpp10_battery_status_map_level(response.rap.params[0]);
+ status = hidpp10_battery_status_map_status(response.rap.params[1]);
+ hidpp->battery.status = status;
+ /* the capacity is only available when discharging or full */
+ hidpp->battery.online = status == POWER_SUPPLY_STATUS_DISCHARGING ||
+ status == POWER_SUPPLY_STATUS_FULL;
+
+ return 0;
+}
+
+#define HIDPP_REG_BATTERY_MILEAGE 0x0D
+
+static int hidpp10_battery_mileage_map_status(u8 param)
+{
+ int status;
+
+ switch (param >> 6) {
+ case 0x00:
+ /* discharging (in use) */
+ status = POWER_SUPPLY_STATUS_DISCHARGING;
+ break;
+ case 0x01: /* charging */
+ status = POWER_SUPPLY_STATUS_CHARGING;
+ break;
+ case 0x02: /* charge complete */
+ status = POWER_SUPPLY_STATUS_FULL;
+ break;
+ /*
+ * 0x03 = charging error
+ */
+ default:
+ status = POWER_SUPPLY_STATUS_NOT_CHARGING;
+ break;
+ }
+
+ return status;
+}
+
+static int hidpp10_query_battery_mileage(struct hidpp_device *hidpp)
+{
+ struct hidpp_report response;
+ int ret, status;
+
+ ret = hidpp_send_rap_command_sync(hidpp,
+ REPORT_ID_HIDPP_SHORT,
+ HIDPP_GET_REGISTER,
+ HIDPP_REG_BATTERY_MILEAGE,
+ NULL, 0, &response);
+ if (ret)
+ return ret;
+
+ hidpp->battery.capacity = response.rap.params[0];
+ status = hidpp10_battery_mileage_map_status(response.rap.params[2]);
+ hidpp->battery.status = status;
+ /* the capacity is only available when discharging or full */
+ hidpp->battery.online = status == POWER_SUPPLY_STATUS_DISCHARGING ||
+ status == POWER_SUPPLY_STATUS_FULL;
+
+ return 0;
+}
+
+static int hidpp10_battery_event(struct hidpp_device *hidpp, u8 *data, int size)
+{
+ struct hidpp_report *report = (struct hidpp_report *)data;
+ int status, capacity, level;
+ bool changed;
+
+ if (report->report_id != REPORT_ID_HIDPP_SHORT)
+ return 0;
+
+ switch (report->rap.sub_id) {
+ case HIDPP_REG_BATTERY_STATUS:
+ capacity = hidpp->battery.capacity;
+ level = hidpp10_battery_status_map_level(report->rawbytes[1]);
+ status = hidpp10_battery_status_map_status(report->rawbytes[2]);
+ break;
+ case HIDPP_REG_BATTERY_MILEAGE:
+ capacity = report->rap.params[0];
+ level = hidpp->battery.level;
+ status = hidpp10_battery_mileage_map_status(report->rawbytes[3]);
+ break;
+ default:
+ return 0;
+ }
+
+ changed = capacity != hidpp->battery.capacity ||
+ level != hidpp->battery.level ||
+ status != hidpp->battery.status;
+
+ /* the capacity is only available when discharging or full */
+ hidpp->battery.online = status == POWER_SUPPLY_STATUS_DISCHARGING ||
+ status == POWER_SUPPLY_STATUS_FULL;
+
+ if (changed) {
+ hidpp->battery.level = level;
+ hidpp->battery.status = status;
+ if (hidpp->battery.ps)
+ power_supply_changed(hidpp->battery.ps);
+ }
+
+ return 0;
+}
+
+#define HIDPP_REG_PAIRING_INFORMATION 0xB5
+#define HIDPP_EXTENDED_PAIRING 0x30
+#define HIDPP_DEVICE_NAME 0x40
+
+static char *hidpp_unifying_get_name(struct hidpp_device *hidpp_dev)
+{
+ struct hidpp_report response;
+ int ret;
+ u8 params[1] = { HIDPP_DEVICE_NAME };
char *name;
int len;
@@ -417,6 +642,54 @@
return name;
}
+static int hidpp_unifying_get_serial(struct hidpp_device *hidpp, u32 *serial)
+{
+ struct hidpp_report response;
+ int ret;
+ u8 params[1] = { HIDPP_EXTENDED_PAIRING };
+
+ ret = hidpp_send_rap_command_sync(hidpp,
+ REPORT_ID_HIDPP_SHORT,
+ HIDPP_GET_LONG_REGISTER,
+ HIDPP_REG_PAIRING_INFORMATION,
+ params, 1, &response);
+ if (ret)
+ return ret;
+
+ /*
+ * We don't care about LE or BE, we will output it as a string
+ * with %4phD, so we need to keep the order.
+ */
+ *serial = *((u32 *)&response.rap.params[1]);
+ return 0;
+}
+
+static int hidpp_unifying_init(struct hidpp_device *hidpp)
+{
+ struct hid_device *hdev = hidpp->hid_dev;
+ const char *name;
+ u32 serial;
+ int ret;
+
+ ret = hidpp_unifying_get_serial(hidpp, &serial);
+ if (ret)
+ return ret;
+
+ snprintf(hdev->uniq, sizeof(hdev->uniq), "%04x-%4phD",
+ hdev->product, &serial);
+ dbg_hid("HID++ Unifying: Got serial: %s\n", hdev->uniq);
+
+ name = hidpp_unifying_get_name(hidpp);
+ if (!name)
+ return -EIO;
+
+ snprintf(hdev->name, sizeof(hdev->name), "%s", name);
+ dbg_hid("HID++ Unifying: Got name: %s\n", name);
+
+ kfree(name);
+ return 0;
+}
+
/* -------------------------------------------------------------------------- */
/* 0x0000: Root */
/* -------------------------------------------------------------------------- */
@@ -441,6 +714,9 @@
if (ret)
return ret;
+ if (response.fap.params[0] == 0)
+ return -ENOENT;
+
*feature_index = response.fap.params[0];
*feature_type = response.fap.params[1];
@@ -607,6 +883,355 @@
}
/* -------------------------------------------------------------------------- */
+/* 0x1000: Battery level status */
+/* -------------------------------------------------------------------------- */
+
+#define HIDPP_PAGE_BATTERY_LEVEL_STATUS 0x1000
+
+#define CMD_BATTERY_LEVEL_STATUS_GET_BATTERY_LEVEL_STATUS 0x00
+#define CMD_BATTERY_LEVEL_STATUS_GET_BATTERY_CAPABILITY 0x10
+
+#define EVENT_BATTERY_LEVEL_STATUS_BROADCAST 0x00
+
+#define FLAG_BATTERY_LEVEL_DISABLE_OSD BIT(0)
+#define FLAG_BATTERY_LEVEL_MILEAGE BIT(1)
+#define FLAG_BATTERY_LEVEL_RECHARGEABLE BIT(2)
+
+static int hidpp_map_battery_level(int capacity)
+{
+ if (capacity < 11)
+ return POWER_SUPPLY_CAPACITY_LEVEL_CRITICAL;
+ else if (capacity < 31)
+ return POWER_SUPPLY_CAPACITY_LEVEL_LOW;
+ else if (capacity < 81)
+ return POWER_SUPPLY_CAPACITY_LEVEL_NORMAL;
+ return POWER_SUPPLY_CAPACITY_LEVEL_FULL;
+}
+
+static int hidpp20_batterylevel_map_status_capacity(u8 data[3], int *capacity,
+ int *next_capacity,
+ int *level)
+{
+ int status;
+
+ *capacity = data[0];
+ *next_capacity = data[1];
+ *level = POWER_SUPPLY_CAPACITY_LEVEL_UNKNOWN;
+
+ /* When discharging, we can rely on the device reported capacity.
+ * For all other states the device reports 0 (unknown).
+ */
+ switch (data[2]) {
+ case 0: /* discharging (in use) */
+ status = POWER_SUPPLY_STATUS_DISCHARGING;
+ *level = hidpp_map_battery_level(*capacity);
+ break;
+ case 1: /* recharging */
+ status = POWER_SUPPLY_STATUS_CHARGING;
+ break;
+ case 2: /* charge in final stage */
+ status = POWER_SUPPLY_STATUS_CHARGING;
+ break;
+ case 3: /* charge complete */
+ status = POWER_SUPPLY_STATUS_FULL;
+ *level = POWER_SUPPLY_CAPACITY_LEVEL_FULL;
+ *capacity = 100;
+ break;
+ case 4: /* recharging below optimal speed */
+ status = POWER_SUPPLY_STATUS_CHARGING;
+ break;
+ /* 5 = invalid battery type
+ 6 = thermal error
+ 7 = other charging error */
+ default:
+ status = POWER_SUPPLY_STATUS_NOT_CHARGING;
+ break;
+ }
+
+ return status;
+}
+
+static int hidpp20_batterylevel_get_battery_capacity(struct hidpp_device *hidpp,
+ u8 feature_index,
+ int *status,
+ int *capacity,
+ int *next_capacity,
+ int *level)
+{
+ struct hidpp_report response;
+ int ret;
+ u8 *params = (u8 *)response.fap.params;
+
+ ret = hidpp_send_fap_command_sync(hidpp, feature_index,
+ CMD_BATTERY_LEVEL_STATUS_GET_BATTERY_LEVEL_STATUS,
+ NULL, 0, &response);
+ if (ret > 0) {
+ hid_err(hidpp->hid_dev, "%s: received protocol error 0x%02x\n",
+ __func__, ret);
+ return -EPROTO;
+ }
+ if (ret)
+ return ret;
+
+ *status = hidpp20_batterylevel_map_status_capacity(params, capacity,
+ next_capacity,
+ level);
+
+ return 0;
+}
+
+static int hidpp20_batterylevel_get_battery_info(struct hidpp_device *hidpp,
+ u8 feature_index)
+{
+ struct hidpp_report response;
+ int ret;
+ u8 *params = (u8 *)response.fap.params;
+ unsigned int level_count, flags;
+
+ ret = hidpp_send_fap_command_sync(hidpp, feature_index,
+ CMD_BATTERY_LEVEL_STATUS_GET_BATTERY_CAPABILITY,
+ NULL, 0, &response);
+ if (ret > 0) {
+ hid_err(hidpp->hid_dev, "%s: received protocol error 0x%02x\n",
+ __func__, ret);
+ return -EPROTO;
+ }
+ if (ret)
+ return ret;
+
+ level_count = params[0];
+ flags = params[1];
+
+ if (level_count < 10 || !(flags & FLAG_BATTERY_LEVEL_MILEAGE))
+ hidpp->capabilities |= HIDPP_CAPABILITY_BATTERY_LEVEL_STATUS;
+ else
+ hidpp->capabilities |= HIDPP_CAPABILITY_BATTERY_MILEAGE;
+
+ return 0;
+}
+
+static int hidpp20_query_battery_info(struct hidpp_device *hidpp)
+{
+ u8 feature_type;
+ int ret;
+ int status, capacity, next_capacity, level;
+
+ if (hidpp->battery.feature_index == 0xff) {
+ ret = hidpp_root_get_feature(hidpp,
+ HIDPP_PAGE_BATTERY_LEVEL_STATUS,
+ &hidpp->battery.feature_index,
+ &feature_type);
+ if (ret)
+ return ret;
+ }
+
+ ret = hidpp20_batterylevel_get_battery_capacity(hidpp,
+ hidpp->battery.feature_index,
+ &status, &capacity,
+ &next_capacity, &level);
+ if (ret)
+ return ret;
+
+ ret = hidpp20_batterylevel_get_battery_info(hidpp,
+ hidpp->battery.feature_index);
+ if (ret)
+ return ret;
+
+ hidpp->battery.status = status;
+ hidpp->battery.capacity = capacity;
+ hidpp->battery.level = level;
+ /* the capacity is only available when discharging or full */
+ hidpp->battery.online = status == POWER_SUPPLY_STATUS_DISCHARGING ||
+ status == POWER_SUPPLY_STATUS_FULL;
+
+ return 0;
+}
+
+static int hidpp20_battery_event(struct hidpp_device *hidpp,
+ u8 *data, int size)
+{
+ struct hidpp_report *report = (struct hidpp_report *)data;
+ int status, capacity, next_capacity, level;
+ bool changed;
+
+ if (report->fap.feature_index != hidpp->battery.feature_index ||
+ report->fap.funcindex_clientid != EVENT_BATTERY_LEVEL_STATUS_BROADCAST)
+ return 0;
+
+ status = hidpp20_batterylevel_map_status_capacity(report->fap.params,
+ &capacity,
+ &next_capacity,
+ &level);
+
+ /* the capacity is only available when discharging or full */
+ hidpp->battery.online = status == POWER_SUPPLY_STATUS_DISCHARGING ||
+ status == POWER_SUPPLY_STATUS_FULL;
+
+ changed = capacity != hidpp->battery.capacity ||
+ level != hidpp->battery.level ||
+ status != hidpp->battery.status;
+
+ if (changed) {
+ hidpp->battery.level = level;
+ hidpp->battery.capacity = capacity;
+ hidpp->battery.status = status;
+ if (hidpp->battery.ps)
+ power_supply_changed(hidpp->battery.ps);
+ }
+
+ return 0;
+}
+
+static enum power_supply_property hidpp_battery_props[] = {
+ POWER_SUPPLY_PROP_ONLINE,
+ POWER_SUPPLY_PROP_STATUS,
+ POWER_SUPPLY_PROP_SCOPE,
+ POWER_SUPPLY_PROP_MODEL_NAME,
+ POWER_SUPPLY_PROP_MANUFACTURER,
+ POWER_SUPPLY_PROP_SERIAL_NUMBER,
+ 0, /* placeholder for POWER_SUPPLY_PROP_CAPACITY, */
+ 0, /* placeholder for POWER_SUPPLY_PROP_CAPACITY_LEVEL, */
+};
+
+static int hidpp_battery_get_property(struct power_supply *psy,
+ enum power_supply_property psp,
+ union power_supply_propval *val)
+{
+ struct hidpp_device *hidpp = power_supply_get_drvdata(psy);
+ int ret = 0;
+
+ switch(psp) {
+ case POWER_SUPPLY_PROP_STATUS:
+ val->intval = hidpp->battery.status;
+ break;
+ case POWER_SUPPLY_PROP_CAPACITY:
+ val->intval = hidpp->battery.capacity;
+ break;
+ case POWER_SUPPLY_PROP_CAPACITY_LEVEL:
+ val->intval = hidpp->battery.level;
+ break;
+ case POWER_SUPPLY_PROP_SCOPE:
+ val->intval = POWER_SUPPLY_SCOPE_DEVICE;
+ break;
+ case POWER_SUPPLY_PROP_ONLINE:
+ val->intval = hidpp->battery.online;
+ break;
+ case POWER_SUPPLY_PROP_MODEL_NAME:
+ if (!strncmp(hidpp->name, "Logitech ", 9))
+ val->strval = hidpp->name + 9;
+ else
+ val->strval = hidpp->name;
+ break;
+ case POWER_SUPPLY_PROP_MANUFACTURER:
+ val->strval = "Logitech";
+ break;
+ case POWER_SUPPLY_PROP_SERIAL_NUMBER:
+ val->strval = hidpp->hid_dev->uniq;
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+/* -------------------------------------------------------------------------- */
+/* 0x4301: Solar Keyboard */
+/* -------------------------------------------------------------------------- */
+
+#define HIDPP_PAGE_SOLAR_KEYBOARD 0x4301
+
+#define CMD_SOLAR_SET_LIGHT_MEASURE 0x00
+
+#define EVENT_SOLAR_BATTERY_BROADCAST 0x00
+#define EVENT_SOLAR_BATTERY_LIGHT_MEASURE 0x10
+#define EVENT_SOLAR_CHECK_LIGHT_BUTTON 0x20
+
+static int hidpp_solar_request_battery_event(struct hidpp_device *hidpp)
+{
+ struct hidpp_report response;
+ u8 params[2] = { 1, 1 };
+ u8 feature_type;
+ int ret;
+
+ if (hidpp->battery.feature_index == 0xff) {
+ ret = hidpp_root_get_feature(hidpp,
+ HIDPP_PAGE_SOLAR_KEYBOARD,
+ &hidpp->battery.solar_feature_index,
+ &feature_type);
+ if (ret)
+ return ret;
+ }
+
+ ret = hidpp_send_fap_command_sync(hidpp,
+ hidpp->battery.solar_feature_index,
+ CMD_SOLAR_SET_LIGHT_MEASURE,
+ params, 2, &response);
+ if (ret > 0) {
+ hid_err(hidpp->hid_dev, "%s: received protocol error 0x%02x\n",
+ __func__, ret);
+ return -EPROTO;
+ }
+ if (ret)
+ return ret;
+
+ hidpp->capabilities |= HIDPP_CAPABILITY_BATTERY_MILEAGE;
+
+ return 0;
+}
+
+static int hidpp_solar_battery_event(struct hidpp_device *hidpp,
+ u8 *data, int size)
+{
+ struct hidpp_report *report = (struct hidpp_report *)data;
+ int capacity, lux, status;
+ u8 function;
+
+ function = report->fap.funcindex_clientid;
+
+
+ if (report->fap.feature_index != hidpp->battery.solar_feature_index ||
+ !(function == EVENT_SOLAR_BATTERY_BROADCAST ||
+ function == EVENT_SOLAR_BATTERY_LIGHT_MEASURE ||
+ function == EVENT_SOLAR_CHECK_LIGHT_BUTTON))
+ return 0;
+
+ capacity = report->fap.params[0];
+
+ switch (function) {
+ case EVENT_SOLAR_BATTERY_LIGHT_MEASURE:
+ lux = (report->fap.params[1] << 8) | report->fap.params[2];
+ if (lux > 200)
+ status = POWER_SUPPLY_STATUS_CHARGING;
+ else
+ status = POWER_SUPPLY_STATUS_DISCHARGING;
+ break;
+ case EVENT_SOLAR_CHECK_LIGHT_BUTTON:
+ default:
+ if (capacity < hidpp->battery.capacity)
+ status = POWER_SUPPLY_STATUS_DISCHARGING;
+ else
+ status = POWER_SUPPLY_STATUS_CHARGING;
+
+ }
+
+ if (capacity == 100)
+ status = POWER_SUPPLY_STATUS_FULL;
+
+ hidpp->battery.online = true;
+ if (capacity != hidpp->battery.capacity ||
+ status != hidpp->battery.status) {
+ hidpp->battery.capacity = capacity;
+ hidpp->battery.status = status;
+ if (hidpp->battery.ps)
+ power_supply_changed(hidpp->battery.ps);
+ }
+
+ return 0;
+}
+
+/* -------------------------------------------------------------------------- */
/* 0x6010: Touchpad FW items */
/* -------------------------------------------------------------------------- */
@@ -1599,9 +2224,6 @@
struct wtp_data *wd = hidpp->private_data;
int ret;
- if (!connected)
- return 0;
-
if (!wd->x_size) {
ret = wtp_get_config(hidpp);
if (ret) {
@@ -1669,9 +2291,6 @@
hidpp_dev = hid_get_drvdata(hdev);
- if (!connected)
- return -ENODEV;
-
return hidpp_send_rap_command_sync(
hidpp_dev,
REPORT_ID_HIDPP_SHORT,
@@ -1875,9 +2494,6 @@
{
struct hidpp_device *hidpp = hid_get_drvdata(hdev);
- if (!connected)
- return 0;
-
if (!disable_tap_to_click)
return 0;
@@ -1974,6 +2590,7 @@
struct hidpp_report *question = hidpp->send_receive_buf;
struct hidpp_report *answer = hidpp->send_receive_buf;
struct hidpp_report *report = (struct hidpp_report *)data;
+ int ret;
/*
* If the mutex is locked then we have a pending answer from a
@@ -2002,12 +2619,26 @@
if (unlikely(hidpp_report_is_connect_event(report))) {
atomic_set(&hidpp->connected,
!(report->rap.params[0] & (1 << 6)));
- if ((hidpp->quirks & HIDPP_QUIRK_CONNECT_EVENTS) &&
- (schedule_work(&hidpp->work) == 0))
+ if (schedule_work(&hidpp->work) == 0)
dbg_hid("%s: connect event already queued\n", __func__);
return 1;
}
+ if (hidpp->capabilities & HIDPP_CAPABILITY_HIDPP20_BATTERY) {
+ ret = hidpp20_battery_event(hidpp, data, size);
+ if (ret != 0)
+ return ret;
+ ret = hidpp_solar_battery_event(hidpp, data, size);
+ if (ret != 0)
+ return ret;
+ }
+
+ if (hidpp->capabilities & HIDPP_CAPABILITY_HIDPP10_BATTERY) {
+ ret = hidpp10_battery_event(hidpp, data, size);
+ if (ret != 0)
+ return ret;
+ }
+
return 0;
}
@@ -2058,20 +2689,90 @@
return 0;
}
-static void hidpp_overwrite_name(struct hid_device *hdev, bool use_unifying)
+static int hidpp_initialize_battery(struct hidpp_device *hidpp)
+{
+ static atomic_t battery_no = ATOMIC_INIT(0);
+ struct power_supply_config cfg = { .drv_data = hidpp };
+ struct power_supply_desc *desc = &hidpp->battery.desc;
+ enum power_supply_property *battery_props;
+ struct hidpp_battery *battery;
+ unsigned int num_battery_props;
+ unsigned long n;
+ int ret;
+
+ if (hidpp->battery.ps)
+ return 0;
+
+ hidpp->battery.feature_index = 0xff;
+ hidpp->battery.solar_feature_index = 0xff;
+
+ if (hidpp->protocol_major >= 2) {
+ if (hidpp->quirks & HIDPP_QUIRK_CLASS_K750)
+ ret = hidpp_solar_request_battery_event(hidpp);
+ else
+ ret = hidpp20_query_battery_info(hidpp);
+
+ if (ret)
+ return ret;
+ hidpp->capabilities |= HIDPP_CAPABILITY_HIDPP20_BATTERY;
+ } else {
+ ret = hidpp10_query_battery_status(hidpp);
+ if (ret) {
+ ret = hidpp10_query_battery_mileage(hidpp);
+ if (ret)
+ return -ENOENT;
+ hidpp->capabilities |= HIDPP_CAPABILITY_BATTERY_MILEAGE;
+ } else {
+ hidpp->capabilities |= HIDPP_CAPABILITY_BATTERY_LEVEL_STATUS;
+ }
+ hidpp->capabilities |= HIDPP_CAPABILITY_HIDPP10_BATTERY;
+ }
+
+ battery_props = devm_kmemdup(&hidpp->hid_dev->dev,
+ hidpp_battery_props,
+ sizeof(hidpp_battery_props),
+ GFP_KERNEL);
+ num_battery_props = ARRAY_SIZE(hidpp_battery_props) - 2;
+
+ if (hidpp->capabilities & HIDPP_CAPABILITY_BATTERY_MILEAGE)
+ battery_props[num_battery_props++] =
+ POWER_SUPPLY_PROP_CAPACITY;
+
+ if (hidpp->capabilities & HIDPP_CAPABILITY_BATTERY_LEVEL_STATUS)
+ battery_props[num_battery_props++] =
+ POWER_SUPPLY_PROP_CAPACITY_LEVEL;
+
+ battery = &hidpp->battery;
+
+ n = atomic_inc_return(&battery_no) - 1;
+ desc->properties = battery_props;
+ desc->num_properties = num_battery_props;
+ desc->get_property = hidpp_battery_get_property;
+ sprintf(battery->name, "hidpp_battery_%ld", n);
+ desc->name = battery->name;
+ desc->type = POWER_SUPPLY_TYPE_BATTERY;
+ desc->use_for_apm = 0;
+
+ battery->ps = devm_power_supply_register(&hidpp->hid_dev->dev,
+ &battery->desc,
+ &cfg);
+ if (IS_ERR(battery->ps))
+ return PTR_ERR(battery->ps);
+
+ power_supply_powers(battery->ps, &hidpp->hid_dev->dev);
+
+ return ret;
+}
+
+static void hidpp_overwrite_name(struct hid_device *hdev)
{
struct hidpp_device *hidpp = hid_get_drvdata(hdev);
char *name;
- if (use_unifying)
- /*
- * the device is connected through an Unifying receiver, and
- * might not be already connected.
- * Ask the receiver for its name.
- */
- name = hidpp_get_unifying_name(hidpp);
- else
- name = hidpp_get_device_name(hidpp);
+ if (hidpp->protocol_major < 2)
+ return;
+
+ name = hidpp_get_device_name(hidpp);
if (!name) {
hid_err(hdev, "unable to retrieve the name of the device");
@@ -2129,6 +2830,16 @@
struct input_dev *input;
char *name, *devm_name;
+ if (!connected) {
+ if (hidpp->battery.ps) {
+ hidpp->battery.online = false;
+ hidpp->battery.status = POWER_SUPPLY_STATUS_UNKNOWN;
+ hidpp->battery.level = POWER_SUPPLY_CAPACITY_LEVEL_UNKNOWN;
+ power_supply_changed(hidpp->battery.ps);
+ }
+ return;
+ }
+
if (hidpp->quirks & HIDPP_QUIRK_CLASS_WTP) {
ret = wtp_connect(hdev, connected);
if (ret)
@@ -2143,9 +2854,6 @@
return;
}
- if (!connected || hidpp->delayed_input)
- return;
-
/* the device is already connected, we can ask for its name and
* protocol */
if (!hidpp->protocol_major) {
@@ -2158,11 +2866,7 @@
hidpp->protocol_major, hidpp->protocol_minor);
}
- if (!(hidpp->quirks & HIDPP_QUIRK_NO_HIDINPUT))
- /* if HID created the input nodes for us, we can stop now */
- return;
-
- if (!hidpp->name || hidpp->name == hdev->name) {
+ if (hidpp->name == hdev->name && hidpp->protocol_major >= 2) {
name = hidpp_get_device_name(hidpp);
if (!name) {
hid_err(hdev,
@@ -2178,6 +2882,25 @@
hidpp->name = devm_name;
}
+ hidpp_initialize_battery(hidpp);
+
+ /* forward current battery state */
+ if (hidpp->capabilities & HIDPP_CAPABILITY_HIDPP10_BATTERY) {
+ hidpp10_enable_battery_reporting(hidpp);
+ if (hidpp->capabilities & HIDPP_CAPABILITY_BATTERY_MILEAGE)
+ hidpp10_query_battery_mileage(hidpp);
+ else
+ hidpp10_query_battery_status(hidpp);
+ } else if (hidpp->capabilities & HIDPP_CAPABILITY_HIDPP20_BATTERY) {
+ hidpp20_query_battery_info(hidpp);
+ }
+ if (hidpp->battery.ps)
+ power_supply_changed(hidpp->battery.ps);
+
+ if (!(hidpp->quirks & HIDPP_QUIRK_NO_HIDINPUT) || hidpp->delayed_input)
+ /* if the input nodes are already created, we can stop now */
+ return;
+
input = hidpp_allocate_input(hdev);
if (!input) {
hid_err(hdev, "cannot allocate new input device: %d\n", ret);
@@ -2193,6 +2916,17 @@
hidpp->delayed_input = input;
}
+static DEVICE_ATTR(builtin_power_supply, 0000, NULL, NULL);
+
+static struct attribute *sysfs_attrs[] = {
+ &dev_attr_builtin_power_supply.attr,
+ NULL
+};
+
+static struct attribute_group ps_attribute_group = {
+ .attrs = sysfs_attrs
+};
+
static int hidpp_probe(struct hid_device *hdev, const struct hid_device_id *id)
{
struct hidpp_device *hidpp;
@@ -2211,9 +2945,11 @@
hidpp->quirks = id->driver_data;
+ if (id->group == HID_GROUP_LOGITECH_DJ_DEVICE)
+ hidpp->quirks |= HIDPP_QUIRK_UNIFYING;
+
if (disable_raw_mode) {
hidpp->quirks &= ~HIDPP_QUIRK_CLASS_WTP;
- hidpp->quirks &= ~HIDPP_QUIRK_CONNECT_EVENTS;
hidpp->quirks &= ~HIDPP_QUIRK_NO_HIDINPUT;
}
@@ -2235,6 +2971,12 @@
mutex_init(&hidpp->send_mutex);
init_waitqueue_head(&hidpp->wait);
+ /* indicates we are handling the battery properties in the kernel */
+ ret = sysfs_create_group(&hdev->dev.kobj, &ps_attribute_group);
+ if (ret)
+ hid_warn(hdev, "Cannot allocate sysfs group for %s\n",
+ hdev->name);
+
ret = hid_parse(hdev);
if (ret) {
hid_err(hdev, "%s:parse failed\n", __func__);
@@ -2263,8 +3005,12 @@
/* Allow incoming packets */
hid_device_io_start(hdev);
+ if (hidpp->quirks & HIDPP_QUIRK_UNIFYING)
+ hidpp_unifying_init(hidpp);
+
connected = hidpp_is_connected(hidpp);
- if (id->group != HID_GROUP_LOGITECH_DJ_DEVICE) {
+ atomic_set(&hidpp->connected, connected);
+ if (!(hidpp->quirks & HIDPP_QUIRK_UNIFYING)) {
if (!connected) {
ret = -ENODEV;
hid_err(hdev, "Device not connected");
@@ -2273,10 +3019,9 @@
hid_info(hdev, "HID++ %u.%u device connected.\n",
hidpp->protocol_major, hidpp->protocol_minor);
- }
- hidpp_overwrite_name(hdev, id->group == HID_GROUP_LOGITECH_DJ_DEVICE);
- atomic_set(&hidpp->connected, connected);
+ hidpp_overwrite_name(hdev);
+ }
if (connected && (hidpp->quirks & HIDPP_QUIRK_CLASS_WTP)) {
ret = wtp_get_config(hidpp);
@@ -2299,12 +3044,10 @@
}
}
- if (hidpp->quirks & HIDPP_QUIRK_CONNECT_EVENTS) {
- /* Allow incoming packets */
- hid_device_io_start(hdev);
+ /* Allow incoming packets */
+ hid_device_io_start(hdev);
- hidpp_connect_event(hidpp);
- }
+ hidpp_connect_event(hidpp);
return ret;
@@ -2316,6 +3059,7 @@
}
hid_hw_start_fail:
hid_parse_fail:
+ sysfs_remove_group(&hdev->dev.kobj, &ps_attribute_group);
cancel_work_sync(&hidpp->work);
mutex_destroy(&hidpp->send_mutex);
allocate_fail:
@@ -2327,6 +3071,8 @@
{
struct hidpp_device *hidpp = hid_get_drvdata(hdev);
+ sysfs_remove_group(&hdev->dev.kobj, &ps_attribute_group);
+
if (hidpp->quirks & HIDPP_QUIRK_CLASS_G920) {
hidpp_ff_deinit(hdev);
hid_hw_close(hdev);
@@ -2357,7 +3103,11 @@
{ /* Keyboard logitech K400 */
HID_DEVICE(BUS_USB, HID_GROUP_LOGITECH_DJ_DEVICE,
USB_VENDOR_ID_LOGITECH, 0x4024),
- .driver_data = HIDPP_QUIRK_CONNECT_EVENTS | HIDPP_QUIRK_CLASS_K400 },
+ .driver_data = HIDPP_QUIRK_CLASS_K400 },
+ { /* Solar Keyboard Logitech K750 */
+ HID_DEVICE(BUS_USB, HID_GROUP_LOGITECH_DJ_DEVICE,
+ USB_VENDOR_ID_LOGITECH, 0x4002),
+ .driver_data = HIDPP_QUIRK_CLASS_K750 },
{ HID_DEVICE(BUS_USB, HID_GROUP_LOGITECH_DJ_DEVICE,
USB_VENDOR_ID_LOGITECH, HID_ANY_ID)},
diff --git a/drivers/hid/hid-multitouch.c b/drivers/hid/hid-multitouch.c
index 6926474..24d5b6d 100644
--- a/drivers/hid/hid-multitouch.c
+++ b/drivers/hid/hid-multitouch.c
@@ -69,6 +69,7 @@
#define MT_QUIRK_CONTACT_CNT_ACCURATE (1 << 12)
#define MT_QUIRK_FORCE_GET_FEATURE (1 << 13)
#define MT_QUIRK_FIX_CONST_CONTACT_ID (1 << 14)
+#define MT_QUIRK_TOUCH_SIZE_SCALING (1 << 15)
#define MT_INPUTMODE_TOUCHSCREEN 0x02
#define MT_INPUTMODE_TOUCHPAD 0x03
@@ -222,7 +223,8 @@
*/
{ .name = MT_CLS_3M,
.quirks = MT_QUIRK_VALID_IS_CONFIDENCE |
- MT_QUIRK_SLOT_IS_CONTACTID,
+ MT_QUIRK_SLOT_IS_CONTACTID |
+ MT_QUIRK_TOUCH_SIZE_SCALING,
.sn_move = 2048,
.sn_width = 128,
.sn_height = 128,
@@ -658,9 +660,17 @@
if (active) {
/* this finger is in proximity of the sensor */
int wide = (s->w > s->h);
- /* divided by two to match visual scale of touch */
- int major = max(s->w, s->h) >> 1;
- int minor = min(s->w, s->h) >> 1;
+ int major = max(s->w, s->h);
+ int minor = min(s->w, s->h);
+
+ /*
+ * divided by two to match visual scale of touch
+ * for devices with this quirk
+ */
+ if (td->mtclass.quirks & MT_QUIRK_TOUCH_SIZE_SCALING) {
+ major = major >> 1;
+ minor = minor >> 1;
+ }
input_event(input, EV_ABS, ABS_MT_POSITION_X, s->x);
input_event(input, EV_ABS, ABS_MT_POSITION_Y, s->y);
diff --git a/drivers/hid/hid-nti.c b/drivers/hid/hid-nti.c
new file mode 100644
index 0000000..5bb827b
--- /dev/null
+++ b/drivers/hid/hid-nti.c
@@ -0,0 +1,59 @@
+/*
+ * USB HID quirks support for Network Technologies, Inc. "USB-SUN" USB
+ * adapter for pre-USB Sun keyboards
+ *
+ * Copyright (c) 2011 Google, Inc.
+ *
+ * Based on HID apple driver by
+ * Copyright (c) 1999 Andreas Gal
+ * Copyright (c) 2000-2005 Vojtech Pavlik <vojtech@suse.cz>
+ * Copyright (c) 2005 Michael Haboustak <mike-@cinci.rr.com> for Concept2, Inc
+ * Copyright (c) 2006-2007 Jiri Kosina
+ * Copyright (c) 2008 Jiri Slaby <jirislaby@gmail.com>
+ */
+
+/*
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ */
+
+#include <linux/device.h>
+#include <linux/input.h>
+#include <linux/hid.h>
+#include <linux/module.h>
+
+#include "hid-ids.h"
+
+MODULE_AUTHOR("Jonathan Klabunde Tomer <jktomer@google.com>");
+MODULE_DESCRIPTION("HID driver for Network Technologies USB-SUN keyboard adapter");
+
+/*
+ * NTI Sun keyboard adapter has wrong logical maximum in report descriptor
+ */
+static __u8 *nti_usbsun_report_fixup(struct hid_device *hdev, __u8 *rdesc,
+ unsigned int *rsize)
+{
+ if (*rsize >= 60 && rdesc[53] == 0x65 && rdesc[59] == 0x65) {
+ hid_info(hdev, "fixing up NTI USB-SUN keyboard adapter report descriptor\n");
+ rdesc[53] = rdesc[59] = 0xe7;
+ }
+ return rdesc;
+}
+
+static const struct hid_device_id nti_devices[] = {
+ { HID_USB_DEVICE(USB_VENDOR_ID_NTI, USB_DEVICE_ID_USB_SUN) },
+ { }
+};
+MODULE_DEVICE_TABLE(hid, nti_devices);
+
+static struct hid_driver nti_driver = {
+ .name = "nti",
+ .id_table = nti_devices,
+ .report_fixup = nti_usbsun_report_fixup
+};
+
+module_hid_driver(nti_driver);
+
+MODULE_LICENSE("GPL");
diff --git a/drivers/hid/hid-picolcd_debugfs.c b/drivers/hid/hid-picolcd_debugfs.c
index 3c13af6..3e0feb4 100644
--- a/drivers/hid/hid-picolcd_debugfs.c
+++ b/drivers/hid/hid-picolcd_debugfs.c
@@ -736,7 +736,7 @@
}
break;
case REPORT_MEMORY:
- /* Data buffer in response to REPORT_READ_MEMORY or REPORT_WRTIE_MEMORY */
+ /* Data buffer in response to REPORT_READ_MEMORY or REPORT_WRITE_MEMORY */
snprintf(buff, BUFF_SZ, "report %s (%d, size=%d)\n",
"REPORT_MEMORY", report->id, size-1);
hid_debug_event(hdev, buff);
diff --git a/drivers/hid/hid-sony.c b/drivers/hid/hid-sony.c
index 740996f..d03203a 100644
--- a/drivers/hid/hid-sony.c
+++ b/drivers/hid/hid-sony.c
@@ -48,19 +48,21 @@
#define PS3REMOTE BIT(4)
#define DUALSHOCK4_CONTROLLER_USB BIT(5)
#define DUALSHOCK4_CONTROLLER_BT BIT(6)
-#define MOTION_CONTROLLER_USB BIT(7)
-#define MOTION_CONTROLLER_BT BIT(8)
-#define NAVIGATION_CONTROLLER_USB BIT(9)
-#define NAVIGATION_CONTROLLER_BT BIT(10)
-#define SINO_LITE_CONTROLLER BIT(11)
-#define FUTUREMAX_DANCE_MAT BIT(12)
+#define DUALSHOCK4_DONGLE BIT(7)
+#define MOTION_CONTROLLER_USB BIT(8)
+#define MOTION_CONTROLLER_BT BIT(9)
+#define NAVIGATION_CONTROLLER_USB BIT(10)
+#define NAVIGATION_CONTROLLER_BT BIT(11)
+#define SINO_LITE_CONTROLLER BIT(12)
+#define FUTUREMAX_DANCE_MAT BIT(13)
#define SIXAXIS_CONTROLLER (SIXAXIS_CONTROLLER_USB | SIXAXIS_CONTROLLER_BT)
#define MOTION_CONTROLLER (MOTION_CONTROLLER_USB | MOTION_CONTROLLER_BT)
#define NAVIGATION_CONTROLLER (NAVIGATION_CONTROLLER_USB |\
NAVIGATION_CONTROLLER_BT)
#define DUALSHOCK4_CONTROLLER (DUALSHOCK4_CONTROLLER_USB |\
- DUALSHOCK4_CONTROLLER_BT)
+ DUALSHOCK4_CONTROLLER_BT | \
+ DUALSHOCK4_DONGLE)
#define SONY_LED_SUPPORT (SIXAXIS_CONTROLLER | BUZZ_CONTROLLER |\
DUALSHOCK4_CONTROLLER | MOTION_CONTROLLER |\
NAVIGATION_CONTROLLER)
@@ -73,89 +75,6 @@
#define MAX_LEDS 4
-/*
- * The Sixaxis reports both digital and analog values for each button on the
- * controller except for Start, Select and the PS button. The controller ends
- * up reporting 27 axes which causes them to spill over into the multi-touch
- * axis values. Additionally, the controller only has 20 actual, physical axes
- * so there are several unused axes in between the used ones.
- */
-static u8 sixaxis_rdesc[] = {
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x04, /* Usage (Joystick), */
- 0xA1, 0x01, /* Collection (Application), */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0x01, /* Report ID (1), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x01, /* Report Count (1), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x81, 0x03, /* Input (Constant, Variable), */
- 0x75, 0x01, /* Report Size (1), */
- 0x95, 0x13, /* Report Count (19), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x01, /* Logical Maximum (1), */
- 0x35, 0x00, /* Physical Minimum (0), */
- 0x45, 0x01, /* Physical Maximum (1), */
- 0x05, 0x09, /* Usage Page (Button), */
- 0x19, 0x01, /* Usage Minimum (01h), */
- 0x29, 0x13, /* Usage Maximum (13h), */
- 0x81, 0x02, /* Input (Variable), */
- 0x75, 0x01, /* Report Size (1), */
- 0x95, 0x0D, /* Report Count (13), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x81, 0x03, /* Input (Constant, Variable), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xA1, 0x00, /* Collection (Physical), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x04, /* Report Count (4), */
- 0x35, 0x00, /* Physical Minimum (0), */
- 0x46, 0xFF, 0x00, /* Physical Maximum (255), */
- 0x09, 0x30, /* Usage (X), */
- 0x09, 0x31, /* Usage (Y), */
- 0x09, 0x32, /* Usage (Z), */
- 0x09, 0x35, /* Usage (Rz), */
- 0x81, 0x02, /* Input (Variable), */
- 0xC0, /* End Collection, */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x95, 0x13, /* Report Count (19), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0x81, 0x02, /* Input (Variable), */
- 0x95, 0x0C, /* Report Count (12), */
- 0x81, 0x01, /* Input (Constant), */
- 0x75, 0x10, /* Report Size (16), */
- 0x95, 0x04, /* Report Count (4), */
- 0x26, 0xFF, 0x03, /* Logical Maximum (1023), */
- 0x46, 0xFF, 0x03, /* Physical Maximum (1023), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0x81, 0x02, /* Input (Variable), */
- 0xC0, /* End Collection, */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0x02, /* Report ID (2), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0xEE, /* Report ID (238), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0xEF, /* Report ID (239), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xC0 /* End Collection */
-};
/* PS/3 Motion controller */
static u8 motion_rdesc[] = {
@@ -254,567 +173,6 @@
0xC0 /* End Collection */
};
-/* PS/3 Navigation controller */
-static u8 navigation_rdesc[] = {
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x04, /* Usage (Joystick), */
- 0xA1, 0x01, /* Collection (Application), */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0x01, /* Report ID (1), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x01, /* Report Count (1), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x81, 0x03, /* Input (Constant, Variable), */
- 0x75, 0x01, /* Report Size (1), */
- 0x95, 0x13, /* Report Count (19), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x01, /* Logical Maximum (1), */
- 0x35, 0x00, /* Physical Minimum (0), */
- 0x45, 0x01, /* Physical Maximum (1), */
- 0x05, 0x09, /* Usage Page (Button), */
- 0x19, 0x01, /* Usage Minimum (01h), */
- 0x29, 0x13, /* Usage Maximum (13h), */
- 0x81, 0x02, /* Input (Variable), */
- 0x75, 0x01, /* Report Size (1), */
- 0x95, 0x0D, /* Report Count (13), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x81, 0x03, /* Input (Constant, Variable), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xA1, 0x00, /* Collection (Physical), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x02, /* Report Count (2), */
- 0x35, 0x00, /* Physical Minimum (0), */
- 0x46, 0xFF, 0x00, /* Physical Maximum (255), */
- 0x09, 0x30, /* Usage (X), */
- 0x09, 0x31, /* Usage (Y), */
- 0x81, 0x02, /* Input (Variable), */
- 0xC0, /* End Collection, */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x95, 0x06, /* Report Count (6), */
- 0x81, 0x03, /* Input (Constant, Variable), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x05, /* Report Count (5), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x95, 0x01, /* Report Count (1), */
- 0x81, 0x02, /* Input (Variable), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x95, 0x01, /* Report Count (1), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x95, 0x1E, /* Report Count (24), */
- 0x81, 0x02, /* Input (Variable), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0x91, 0x02, /* Output (Variable), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0x02, /* Report ID (2), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0xEE, /* Report ID (238), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0xEF, /* Report ID (239), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xC0 /* End Collection */
-};
-
-/*
- * The default descriptor doesn't provide mapping for the accelerometers
- * or orientation sensors. This fixed descriptor maps the accelerometers
- * to usage values 0x40, 0x41 and 0x42 and maps the orientation sensors
- * to usage values 0x43, 0x44 and 0x45.
- */
-static u8 dualshock4_usb_rdesc[] = {
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x05, /* Usage (Gamepad), */
- 0xA1, 0x01, /* Collection (Application), */
- 0x85, 0x01, /* Report ID (1), */
- 0x09, 0x30, /* Usage (X), */
- 0x09, 0x31, /* Usage (Y), */
- 0x09, 0x32, /* Usage (Z), */
- 0x09, 0x35, /* Usage (Rz), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x04, /* Report Count (4), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x39, /* Usage (Hat Switch), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x07, /* Logical Maximum (7), */
- 0x35, 0x00, /* Physical Minimum (0), */
- 0x46, 0x3B, 0x01, /* Physical Maximum (315), */
- 0x65, 0x14, /* Unit (Degrees), */
- 0x75, 0x04, /* Report Size (4), */
- 0x95, 0x01, /* Report Count (1), */
- 0x81, 0x42, /* Input (Variable, Null State), */
- 0x65, 0x00, /* Unit, */
- 0x05, 0x09, /* Usage Page (Button), */
- 0x19, 0x01, /* Usage Minimum (01h), */
- 0x29, 0x0D, /* Usage Maximum (0Dh), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x01, /* Logical Maximum (1), */
- 0x75, 0x01, /* Report Size (1), */
- 0x95, 0x0E, /* Report Count (14), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x09, 0x20, /* Usage (20h), */
- 0x75, 0x06, /* Report Size (6), */
- 0x95, 0x01, /* Report Count (1), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x3F, /* Logical Maximum (63), */
- 0x81, 0x02, /* Input (Variable), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x33, /* Usage (Rx), */
- 0x09, 0x34, /* Usage (Ry), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x02, /* Report Count (2), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x09, 0x21, /* Usage (21h), */
- 0x95, 0x03, /* Report Count (3), */
- 0x81, 0x02, /* Input (Variable), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x19, 0x40, /* Usage Minimum (40h), */
- 0x29, 0x42, /* Usage Maximum (42h), */
- 0x16, 0x00, 0x80, /* Logical Minimum (-32768), */
- 0x26, 0xFF, 0x7F, /* Logical Maximum (32767), */
- 0x75, 0x10, /* Report Size (16), */
- 0x95, 0x03, /* Report Count (3), */
- 0x81, 0x02, /* Input (Variable), */
- 0x19, 0x43, /* Usage Minimum (43h), */
- 0x29, 0x45, /* Usage Maximum (45h), */
- 0x16, 0x00, 0x80, /* Logical Minimum (-32768), */
- 0x26, 0xFF, 0x7F, /* Logical Maximum (32767), */
- 0x95, 0x03, /* Report Count (3), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x09, 0x21, /* Usage (21h), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x27, /* Report Count (39), */
- 0x81, 0x02, /* Input (Variable), */
- 0x85, 0x05, /* Report ID (5), */
- 0x09, 0x22, /* Usage (22h), */
- 0x95, 0x1F, /* Report Count (31), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x04, /* Report ID (4), */
- 0x09, 0x23, /* Usage (23h), */
- 0x95, 0x24, /* Report Count (36), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x02, /* Report ID (2), */
- 0x09, 0x24, /* Usage (24h), */
- 0x95, 0x24, /* Report Count (36), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x08, /* Report ID (8), */
- 0x09, 0x25, /* Usage (25h), */
- 0x95, 0x03, /* Report Count (3), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x10, /* Report ID (16), */
- 0x09, 0x26, /* Usage (26h), */
- 0x95, 0x04, /* Report Count (4), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x11, /* Report ID (17), */
- 0x09, 0x27, /* Usage (27h), */
- 0x95, 0x02, /* Report Count (2), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x12, /* Report ID (18), */
- 0x06, 0x02, 0xFF, /* Usage Page (FF02h), */
- 0x09, 0x21, /* Usage (21h), */
- 0x95, 0x0F, /* Report Count (15), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x13, /* Report ID (19), */
- 0x09, 0x22, /* Usage (22h), */
- 0x95, 0x16, /* Report Count (22), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x14, /* Report ID (20), */
- 0x06, 0x05, 0xFF, /* Usage Page (FF05h), */
- 0x09, 0x20, /* Usage (20h), */
- 0x95, 0x10, /* Report Count (16), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x15, /* Report ID (21), */
- 0x09, 0x21, /* Usage (21h), */
- 0x95, 0x2C, /* Report Count (44), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x06, 0x80, 0xFF, /* Usage Page (FF80h), */
- 0x85, 0x80, /* Report ID (128), */
- 0x09, 0x20, /* Usage (20h), */
- 0x95, 0x06, /* Report Count (6), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x81, /* Report ID (129), */
- 0x09, 0x21, /* Usage (21h), */
- 0x95, 0x06, /* Report Count (6), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x82, /* Report ID (130), */
- 0x09, 0x22, /* Usage (22h), */
- 0x95, 0x05, /* Report Count (5), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x83, /* Report ID (131), */
- 0x09, 0x23, /* Usage (23h), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x84, /* Report ID (132), */
- 0x09, 0x24, /* Usage (24h), */
- 0x95, 0x04, /* Report Count (4), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x85, /* Report ID (133), */
- 0x09, 0x25, /* Usage (25h), */
- 0x95, 0x06, /* Report Count (6), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x86, /* Report ID (134), */
- 0x09, 0x26, /* Usage (26h), */
- 0x95, 0x06, /* Report Count (6), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x87, /* Report ID (135), */
- 0x09, 0x27, /* Usage (27h), */
- 0x95, 0x23, /* Report Count (35), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x88, /* Report ID (136), */
- 0x09, 0x28, /* Usage (28h), */
- 0x95, 0x22, /* Report Count (34), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x89, /* Report ID (137), */
- 0x09, 0x29, /* Usage (29h), */
- 0x95, 0x02, /* Report Count (2), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x90, /* Report ID (144), */
- 0x09, 0x30, /* Usage (30h), */
- 0x95, 0x05, /* Report Count (5), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x91, /* Report ID (145), */
- 0x09, 0x31, /* Usage (31h), */
- 0x95, 0x03, /* Report Count (3), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x92, /* Report ID (146), */
- 0x09, 0x32, /* Usage (32h), */
- 0x95, 0x03, /* Report Count (3), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x93, /* Report ID (147), */
- 0x09, 0x33, /* Usage (33h), */
- 0x95, 0x0C, /* Report Count (12), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA0, /* Report ID (160), */
- 0x09, 0x40, /* Usage (40h), */
- 0x95, 0x06, /* Report Count (6), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA1, /* Report ID (161), */
- 0x09, 0x41, /* Usage (41h), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA2, /* Report ID (162), */
- 0x09, 0x42, /* Usage (42h), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA3, /* Report ID (163), */
- 0x09, 0x43, /* Usage (43h), */
- 0x95, 0x30, /* Report Count (48), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA4, /* Report ID (164), */
- 0x09, 0x44, /* Usage (44h), */
- 0x95, 0x0D, /* Report Count (13), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA5, /* Report ID (165), */
- 0x09, 0x45, /* Usage (45h), */
- 0x95, 0x15, /* Report Count (21), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA6, /* Report ID (166), */
- 0x09, 0x46, /* Usage (46h), */
- 0x95, 0x15, /* Report Count (21), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xF0, /* Report ID (240), */
- 0x09, 0x47, /* Usage (47h), */
- 0x95, 0x3F, /* Report Count (63), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xF1, /* Report ID (241), */
- 0x09, 0x48, /* Usage (48h), */
- 0x95, 0x3F, /* Report Count (63), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xF2, /* Report ID (242), */
- 0x09, 0x49, /* Usage (49h), */
- 0x95, 0x0F, /* Report Count (15), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA7, /* Report ID (167), */
- 0x09, 0x4A, /* Usage (4Ah), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA8, /* Report ID (168), */
- 0x09, 0x4B, /* Usage (4Bh), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA9, /* Report ID (169), */
- 0x09, 0x4C, /* Usage (4Ch), */
- 0x95, 0x08, /* Report Count (8), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xAA, /* Report ID (170), */
- 0x09, 0x4E, /* Usage (4Eh), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xAB, /* Report ID (171), */
- 0x09, 0x4F, /* Usage (4Fh), */
- 0x95, 0x39, /* Report Count (57), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xAC, /* Report ID (172), */
- 0x09, 0x50, /* Usage (50h), */
- 0x95, 0x39, /* Report Count (57), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xAD, /* Report ID (173), */
- 0x09, 0x51, /* Usage (51h), */
- 0x95, 0x0B, /* Report Count (11), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xAE, /* Report ID (174), */
- 0x09, 0x52, /* Usage (52h), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xAF, /* Report ID (175), */
- 0x09, 0x53, /* Usage (53h), */
- 0x95, 0x02, /* Report Count (2), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xB0, /* Report ID (176), */
- 0x09, 0x54, /* Usage (54h), */
- 0x95, 0x3F, /* Report Count (63), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0 /* End Collection */
-};
-
-/*
- * The default behavior of the Dualshock 4 is to send reports using report
- * type 1 when running over Bluetooth. However, when feature report 2 is
- * requested during the controller initialization it starts sending input
- * reports in report 17. Since report 17 is undefined in the default HID
- * descriptor the button and axis definitions must be moved to report 17 or
- * the HID layer won't process the received input.
- */
-static u8 dualshock4_bt_rdesc[] = {
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x05, /* Usage (Gamepad), */
- 0xA1, 0x01, /* Collection (Application), */
- 0x85, 0x01, /* Report ID (1), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x0A, /* Report Count (9), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x04, 0xFF, /* Usage Page (FF04h), */
- 0x85, 0x02, /* Report ID (2), */
- 0x09, 0x24, /* Usage (24h), */
- 0x95, 0x24, /* Report Count (36), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA3, /* Report ID (163), */
- 0x09, 0x25, /* Usage (25h), */
- 0x95, 0x30, /* Report Count (48), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x05, /* Report ID (5), */
- 0x09, 0x26, /* Usage (26h), */
- 0x95, 0x28, /* Report Count (40), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x06, /* Report ID (6), */
- 0x09, 0x27, /* Usage (27h), */
- 0x95, 0x34, /* Report Count (52), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x07, /* Report ID (7), */
- 0x09, 0x28, /* Usage (28h), */
- 0x95, 0x30, /* Report Count (48), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x08, /* Report ID (8), */
- 0x09, 0x29, /* Usage (29h), */
- 0x95, 0x2F, /* Report Count (47), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x06, 0x03, 0xFF, /* Usage Page (FF03h), */
- 0x85, 0x03, /* Report ID (3), */
- 0x09, 0x21, /* Usage (21h), */
- 0x95, 0x26, /* Report Count (38), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x04, /* Report ID (4), */
- 0x09, 0x22, /* Usage (22h), */
- 0x95, 0x2E, /* Report Count (46), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xF0, /* Report ID (240), */
- 0x09, 0x47, /* Usage (47h), */
- 0x95, 0x3F, /* Report Count (63), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xF1, /* Report ID (241), */
- 0x09, 0x48, /* Usage (48h), */
- 0x95, 0x3F, /* Report Count (63), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xF2, /* Report ID (242), */
- 0x09, 0x49, /* Usage (49h), */
- 0x95, 0x0F, /* Report Count (15), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x11, /* Report ID (17), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x09, 0x20, /* Usage (20h), */
- 0x95, 0x02, /* Report Count (2), */
- 0x81, 0x02, /* Input (Variable), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x30, /* Usage (X), */
- 0x09, 0x31, /* Usage (Y), */
- 0x09, 0x32, /* Usage (Z), */
- 0x09, 0x35, /* Usage (Rz), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x04, /* Report Count (4), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x39, /* Usage (Hat Switch), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x07, /* Logical Maximum (7), */
- 0x75, 0x04, /* Report Size (4), */
- 0x95, 0x01, /* Report Count (1), */
- 0x81, 0x42, /* Input (Variable, Null State), */
- 0x05, 0x09, /* Usage Page (Button), */
- 0x19, 0x01, /* Usage Minimum (01h), */
- 0x29, 0x0D, /* Usage Maximum (0Dh), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x01, /* Logical Maximum (1), */
- 0x75, 0x01, /* Report Size (1), */
- 0x95, 0x0E, /* Report Count (14), */
- 0x81, 0x02, /* Input (Variable), */
- 0x75, 0x06, /* Report Size (6), */
- 0x95, 0x01, /* Report Count (1), */
- 0x81, 0x01, /* Input (Constant), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x33, /* Usage (Rx), */
- 0x09, 0x34, /* Usage (Ry), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x02, /* Report Count (2), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x09, 0x20, /* Usage (20h), */
- 0x95, 0x03, /* Report Count (3), */
- 0x81, 0x02, /* Input (Variable), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x19, 0x40, /* Usage Minimum (40h), */
- 0x29, 0x42, /* Usage Maximum (42h), */
- 0x16, 0x00, 0x80, /* Logical Minimum (-32768), */
- 0x26, 0xFF, 0x7F, /* Logical Maximum (32767), */
- 0x75, 0x10, /* Report Size (16), */
- 0x95, 0x03, /* Report Count (3), */
- 0x81, 0x02, /* Input (Variable), */
- 0x19, 0x43, /* Usage Minimum (43h), */
- 0x29, 0x45, /* Usage Maximum (45h), */
- 0x16, 0x00, 0x80, /* Logical Minimum (-32768), */
- 0x26, 0xFF, 0x7F, /* Logical Maximum (32767), */
- 0x95, 0x03, /* Report Count (3), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x09, 0x20, /* Usage (20h), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x31, /* Report Count (51), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x21, /* Usage (21h), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x4D, /* Report Count (77), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x12, /* Report ID (18), */
- 0x09, 0x22, /* Usage (22h), */
- 0x95, 0x8D, /* Report Count (141), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x23, /* Usage (23h), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x13, /* Report ID (19), */
- 0x09, 0x24, /* Usage (24h), */
- 0x95, 0xCD, /* Report Count (205), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x25, /* Usage (25h), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x14, /* Report ID (20), */
- 0x09, 0x26, /* Usage (26h), */
- 0x96, 0x0D, 0x01, /* Report Count (269), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x27, /* Usage (27h), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x15, /* Report ID (21), */
- 0x09, 0x28, /* Usage (28h), */
- 0x96, 0x4D, 0x01, /* Report Count (333), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x29, /* Usage (29h), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x16, /* Report ID (22), */
- 0x09, 0x2A, /* Usage (2Ah), */
- 0x96, 0x8D, 0x01, /* Report Count (397), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x2B, /* Usage (2Bh), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x17, /* Report ID (23), */
- 0x09, 0x2C, /* Usage (2Ch), */
- 0x96, 0xCD, 0x01, /* Report Count (461), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x2D, /* Usage (2Dh), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x18, /* Report ID (24), */
- 0x09, 0x2E, /* Usage (2Eh), */
- 0x96, 0x0D, 0x02, /* Report Count (525), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x2F, /* Usage (2Fh), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x19, /* Report ID (25), */
- 0x09, 0x30, /* Usage (30h), */
- 0x96, 0x22, 0x02, /* Report Count (546), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x31, /* Usage (31h), */
- 0x91, 0x02, /* Output (Variable), */
- 0x06, 0x80, 0xFF, /* Usage Page (FF80h), */
- 0x85, 0x82, /* Report ID (130), */
- 0x09, 0x22, /* Usage (22h), */
- 0x95, 0x3F, /* Report Count (63), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x83, /* Report ID (131), */
- 0x09, 0x23, /* Usage (23h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x84, /* Report ID (132), */
- 0x09, 0x24, /* Usage (24h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x90, /* Report ID (144), */
- 0x09, 0x30, /* Usage (30h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x91, /* Report ID (145), */
- 0x09, 0x31, /* Usage (31h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x92, /* Report ID (146), */
- 0x09, 0x32, /* Usage (32h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x93, /* Report ID (147), */
- 0x09, 0x33, /* Usage (33h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA0, /* Report ID (160), */
- 0x09, 0x40, /* Usage (40h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA4, /* Report ID (164), */
- 0x09, 0x44, /* Usage (44h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0 /* End Collection */
-};
-
static u8 ps3remote_rdesc[] = {
0x05, 0x01, /* GUsagePage Generic Desktop */
0x09, 0x05, /* LUsage 0x05 [Game Pad] */
@@ -977,6 +335,67 @@
[20] = BTN_TRIGGER_HAPPY20,
};
+/* The Navigation controller is a partial DS3 and uses the same HID report
+ * and hence the same keymap indices, however not not all axes/buttons
+ * are physically present. We use the same axis and button mapping as
+ * the DS3, which uses the Linux gamepad spec.
+ */
+static const unsigned int navigation_absmap[] = {
+ [0x30] = ABS_X,
+ [0x31] = ABS_Y,
+ [0x33] = ABS_Z, /* L2 */
+};
+
+/* Buttons not physically available on the device, but still available
+ * in the reports are explicitly set to 0 for documentation purposes.
+ */
+static const unsigned int navigation_keymap[] = {
+ [0x01] = 0, /* Select */
+ [0x02] = BTN_THUMBL, /* L3 */
+ [0x03] = 0, /* R3 */
+ [0x04] = 0, /* Start */
+ [0x05] = BTN_DPAD_UP, /* Up */
+ [0x06] = BTN_DPAD_RIGHT, /* Right */
+ [0x07] = BTN_DPAD_DOWN, /* Down */
+ [0x08] = BTN_DPAD_LEFT, /* Left */
+ [0x09] = BTN_TL2, /* L2 */
+ [0x0a] = 0, /* R2 */
+ [0x0b] = BTN_TL, /* L1 */
+ [0x0c] = 0, /* R1 */
+ [0x0d] = BTN_NORTH, /* Triangle */
+ [0x0e] = BTN_EAST, /* Circle */
+ [0x0f] = BTN_SOUTH, /* Cross */
+ [0x10] = BTN_WEST, /* Square */
+ [0x11] = BTN_MODE, /* PS */
+};
+
+static const unsigned int sixaxis_absmap[] = {
+ [0x30] = ABS_X,
+ [0x31] = ABS_Y,
+ [0x32] = ABS_RX, /* right stick X */
+ [0x35] = ABS_RY, /* right stick Y */
+};
+
+static const unsigned int sixaxis_keymap[] = {
+ [0x01] = BTN_SELECT, /* Select */
+ [0x02] = BTN_THUMBL, /* L3 */
+ [0x03] = BTN_THUMBR, /* R3 */
+ [0x04] = BTN_START, /* Start */
+ [0x05] = BTN_DPAD_UP, /* Up */
+ [0x06] = BTN_DPAD_RIGHT, /* Right */
+ [0x07] = BTN_DPAD_DOWN, /* Down */
+ [0x08] = BTN_DPAD_LEFT, /* Left */
+ [0x09] = BTN_TL2, /* L2 */
+ [0x0a] = BTN_TR2, /* R2 */
+ [0x0b] = BTN_TL, /* L1 */
+ [0x0c] = BTN_TR, /* R1 */
+ [0x0d] = BTN_NORTH, /* Triangle */
+ [0x0e] = BTN_EAST, /* Circle */
+ [0x0f] = BTN_SOUTH, /* Cross */
+ [0x10] = BTN_WEST, /* Square */
+ [0x11] = BTN_MODE, /* PS */
+};
+
static const unsigned int ds4_absmap[] = {
[0x30] = ABS_X,
[0x31] = ABS_Y,
@@ -1002,6 +421,10 @@
[0xd] = BTN_MODE, /* PS */
};
+static const struct {int x; int y; } ds4_hat_mapping[] = {
+ {0, -1}, {1, -1}, {1, 0}, {1, 1}, {0, 1}, {-1, 1}, {-1, 0}, {-1, -1},
+ {0, 0}
+};
static enum power_supply_property sony_battery_props[] = {
POWER_SUPPLY_PROP_PRESENT,
@@ -1048,6 +471,7 @@
};
#define DS4_FEATURE_REPORT_0x02_SIZE 37
+#define DS4_FEATURE_REPORT_0x05_SIZE 41
#define DS4_FEATURE_REPORT_0x81_SIZE 7
#define DS4_INPUT_REPORT_0x11_SIZE 78
#define DS4_OUTPUT_REPORT_0x05_SIZE 32
@@ -1059,23 +483,62 @@
/* Offsets relative to USB input report (0x1). Bluetooth (0x11) requires an
* additional +2.
*/
+#define DS4_INPUT_REPORT_AXIS_OFFSET 1
#define DS4_INPUT_REPORT_BUTTON_OFFSET 5
+#define DS4_INPUT_REPORT_TIMESTAMP_OFFSET 10
+#define DS4_INPUT_REPORT_GYRO_X_OFFSET 13
#define DS4_INPUT_REPORT_BATTERY_OFFSET 30
#define DS4_INPUT_REPORT_TOUCHPAD_OFFSET 33
+#define SENSOR_SUFFIX " Motion Sensors"
#define DS4_TOUCHPAD_SUFFIX " Touchpad"
+/* Default to 4ms poll interval, which is same as USB (not adjustable). */
+#define DS4_BT_DEFAULT_POLL_INTERVAL_MS 4
+#define DS4_BT_MAX_POLL_INTERVAL_MS 62
+#define DS4_GYRO_RES_PER_DEG_S 1024
+#define DS4_ACC_RES_PER_G 8192
+
+#define SIXAXIS_INPUT_REPORT_ACC_X_OFFSET 41
+#define SIXAXIS_ACC_RES_PER_G 113
+
static DEFINE_SPINLOCK(sony_dev_list_lock);
static LIST_HEAD(sony_device_list);
static DEFINE_IDA(sony_device_id_allocator);
+/* Used for calibration of DS4 accelerometer and gyro. */
+struct ds4_calibration_data {
+ int abs_code;
+ short bias;
+ /* Calibration requires scaling against a sensitivity value, which is a
+ * float. Store sensitivity as a fraction to limit floating point
+ * calculations until final calibration.
+ */
+ int sens_numer;
+ int sens_denom;
+};
+
+enum ds4_dongle_state {
+ DONGLE_DISCONNECTED,
+ DONGLE_CALIBRATING,
+ DONGLE_CONNECTED,
+ DONGLE_DISABLED
+};
+
+enum sony_worker {
+ SONY_WORKER_STATE,
+ SONY_WORKER_HOTPLUG
+};
+
struct sony_sc {
spinlock_t lock;
struct list_head list_node;
struct hid_device *hdev;
struct input_dev *touchpad;
+ struct input_dev *sensor_dev;
struct led_classdev *leds[MAX_LEDS];
unsigned long quirks;
+ struct work_struct hotplug_worker;
struct work_struct state_worker;
void (*send_output_report)(struct sony_sc *);
struct power_supply *battery;
@@ -1089,34 +552,82 @@
#endif
u8 mac_address[6];
- u8 worker_initialized;
+ u8 hotplug_worker_initialized;
+ u8 state_worker_initialized;
u8 defer_initialization;
u8 cable_state;
u8 battery_charging;
u8 battery_capacity;
u8 led_state[MAX_LEDS];
- u8 resume_led_state[MAX_LEDS];
u8 led_delay_on[MAX_LEDS];
u8 led_delay_off[MAX_LEDS];
u8 led_count;
- bool ds4_dongle_connected;
+
+ bool timestamp_initialized;
+ u16 prev_timestamp;
+ unsigned int timestamp_us;
+
+ u8 ds4_bt_poll_interval;
+ enum ds4_dongle_state ds4_dongle_state;
+ /* DS4 calibration data */
+ struct ds4_calibration_data ds4_calib_data[6];
};
static void sony_set_leds(struct sony_sc *sc);
-static inline void sony_schedule_work(struct sony_sc *sc)
+static inline void sony_schedule_work(struct sony_sc *sc,
+ enum sony_worker which)
{
- if (!sc->defer_initialization)
- schedule_work(&sc->state_worker);
+ switch (which) {
+ case SONY_WORKER_STATE:
+ if (!sc->defer_initialization)
+ schedule_work(&sc->state_worker);
+ break;
+ case SONY_WORKER_HOTPLUG:
+ if (sc->hotplug_worker_initialized)
+ schedule_work(&sc->hotplug_worker);
+ break;
+ }
}
-static u8 *sixaxis_fixup(struct hid_device *hdev, u8 *rdesc,
- unsigned int *rsize)
+static ssize_t ds4_show_poll_interval(struct device *dev,
+ struct device_attribute
+ *attr, char *buf)
{
- *rsize = sizeof(sixaxis_rdesc);
- return sixaxis_rdesc;
+ struct hid_device *hdev = to_hid_device(dev);
+ struct sony_sc *sc = hid_get_drvdata(hdev);
+
+ return snprintf(buf, PAGE_SIZE, "%i\n", sc->ds4_bt_poll_interval);
}
+static ssize_t ds4_store_poll_interval(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct hid_device *hdev = to_hid_device(dev);
+ struct sony_sc *sc = hid_get_drvdata(hdev);
+ unsigned long flags;
+ u8 interval;
+
+ if (kstrtou8(buf, 0, &interval))
+ return -EINVAL;
+
+ if (interval > DS4_BT_MAX_POLL_INTERVAL_MS)
+ return -EINVAL;
+
+ spin_lock_irqsave(&sc->lock, flags);
+ sc->ds4_bt_poll_interval = interval;
+ spin_unlock_irqrestore(&sc->lock, flags);
+
+ sony_schedule_work(sc, SONY_WORKER_STATE);
+
+ return count;
+}
+
+static DEVICE_ATTR(bt_poll_interval, 0644, ds4_show_poll_interval,
+ ds4_store_poll_interval);
+
+
static u8 *motion_fixup(struct hid_device *hdev, u8 *rdesc,
unsigned int *rsize)
{
@@ -1124,13 +635,6 @@
return motion_rdesc;
}
-static u8 *navigation_fixup(struct hid_device *hdev, u8 *rdesc,
- unsigned int *rsize)
-{
- *rsize = sizeof(navigation_rdesc);
- return navigation_rdesc;
-}
-
static u8 *ps3remote_fixup(struct hid_device *hdev, u8 *rdesc,
unsigned int *rsize)
{
@@ -1172,6 +676,102 @@
return 1;
}
+static int navigation_mapping(struct hid_device *hdev, struct hid_input *hi,
+ struct hid_field *field, struct hid_usage *usage,
+ unsigned long **bit, int *max)
+{
+ if ((usage->hid & HID_USAGE_PAGE) == HID_UP_BUTTON) {
+ unsigned int key = usage->hid & HID_USAGE;
+
+ if (key >= ARRAY_SIZE(sixaxis_keymap))
+ return -1;
+
+ key = navigation_keymap[key];
+ if (!key)
+ return -1;
+
+ hid_map_usage_clear(hi, usage, bit, max, EV_KEY, key);
+ return 1;
+ } else if (usage->hid == HID_GD_POINTER) {
+ /* See comment in sixaxis_mapping, basically the L2 (and R2)
+ * triggers are reported through GD Pointer.
+ * In addition we ignore any analog button 'axes' and only
+ * support digital buttons.
+ */
+ switch (usage->usage_index) {
+ case 8: /* L2 */
+ usage->hid = HID_GD_Z;
+ break;
+ default:
+ return -1;
+ }
+
+ hid_map_usage_clear(hi, usage, bit, max, EV_ABS, usage->hid & 0xf);
+ return 1;
+ } else if ((usage->hid & HID_USAGE_PAGE) == HID_UP_GENDESK) {
+ unsigned int abs = usage->hid & HID_USAGE;
+
+ if (abs >= ARRAY_SIZE(navigation_absmap))
+ return -1;
+
+ abs = navigation_absmap[abs];
+
+ hid_map_usage_clear(hi, usage, bit, max, EV_ABS, abs);
+ return 1;
+ }
+
+ return -1;
+}
+
+
+static int sixaxis_mapping(struct hid_device *hdev, struct hid_input *hi,
+ struct hid_field *field, struct hid_usage *usage,
+ unsigned long **bit, int *max)
+{
+ if ((usage->hid & HID_USAGE_PAGE) == HID_UP_BUTTON) {
+ unsigned int key = usage->hid & HID_USAGE;
+
+ if (key >= ARRAY_SIZE(sixaxis_keymap))
+ return -1;
+
+ key = sixaxis_keymap[key];
+ hid_map_usage_clear(hi, usage, bit, max, EV_KEY, key);
+ return 1;
+ } else if (usage->hid == HID_GD_POINTER) {
+ /* The DS3 provides analog values for most buttons and even
+ * for HAT axes through GD Pointer. L2 and R2 are reported
+ * among these as well instead of as GD Z / RZ. Remap L2
+ * and R2 and ignore other analog 'button axes' as there is
+ * no good way for reporting them.
+ */
+ switch (usage->usage_index) {
+ case 8: /* L2 */
+ usage->hid = HID_GD_Z;
+ break;
+ case 9: /* R2 */
+ usage->hid = HID_GD_RZ;
+ break;
+ default:
+ return -1;
+ }
+
+ hid_map_usage_clear(hi, usage, bit, max, EV_ABS, usage->hid & 0xf);
+ return 1;
+ } else if ((usage->hid & HID_USAGE_PAGE) == HID_UP_GENDESK) {
+ unsigned int abs = usage->hid & HID_USAGE;
+
+ if (abs >= ARRAY_SIZE(sixaxis_absmap))
+ return -1;
+
+ abs = sixaxis_absmap[abs];
+
+ hid_map_usage_clear(hi, usage, bit, max, EV_ABS, abs);
+ return 1;
+ }
+
+ return -1;
+}
+
static int ds4_mapping(struct hid_device *hdev, struct hid_input *hi,
struct hid_field *field, struct hid_usage *usage,
unsigned long **bit, int *max)
@@ -1227,30 +827,9 @@
rdesc[55] = 0x06;
}
- /*
- * The default Dualshock 4 USB descriptor doesn't assign
- * the gyroscope values to corresponding axes so we need a
- * modified one.
- */
- if (sc->quirks & DUALSHOCK4_CONTROLLER_USB) {
- hid_info(hdev, "Using modified Dualshock 4 report descriptor with gyroscope axes\n");
- rdesc = dualshock4_usb_rdesc;
- *rsize = sizeof(dualshock4_usb_rdesc);
- } else if (sc->quirks & DUALSHOCK4_CONTROLLER_BT) {
- hid_info(hdev, "Using modified Dualshock 4 Bluetooth report descriptor\n");
- rdesc = dualshock4_bt_rdesc;
- *rsize = sizeof(dualshock4_bt_rdesc);
- }
-
- if (sc->quirks & SIXAXIS_CONTROLLER)
- return sixaxis_fixup(hdev, rdesc, rsize);
-
if (sc->quirks & MOTION_CONTROLLER)
return motion_fixup(hdev, rdesc, rsize);
- if (sc->quirks & NAVIGATION_CONTROLLER)
- return navigation_fixup(hdev, rdesc, rsize);
-
if (sc->quirks & PS3REMOTE)
return ps3remote_fixup(hdev, rdesc, rsize);
@@ -1288,22 +867,132 @@
sc->battery_capacity = battery_capacity;
sc->battery_charging = battery_charging;
spin_unlock_irqrestore(&sc->lock, flags);
+
+ if (sc->quirks & SIXAXIS_CONTROLLER) {
+ int val;
+
+ offset = SIXAXIS_INPUT_REPORT_ACC_X_OFFSET;
+ val = ((rd[offset+1] << 8) | rd[offset]) - 511;
+ input_report_abs(sc->sensor_dev, ABS_X, val);
+
+ /* Y and Z are swapped and inversed */
+ val = 511 - ((rd[offset+5] << 8) | rd[offset+4]);
+ input_report_abs(sc->sensor_dev, ABS_Y, val);
+
+ val = 511 - ((rd[offset+3] << 8) | rd[offset+2]);
+ input_report_abs(sc->sensor_dev, ABS_Z, val);
+
+ input_sync(sc->sensor_dev);
+ }
}
static void dualshock4_parse_report(struct sony_sc *sc, u8 *rd, int size)
{
+ struct hid_input *hidinput = list_entry(sc->hdev->inputs.next,
+ struct hid_input, list);
+ struct input_dev *input_dev = hidinput->input;
unsigned long flags;
int n, m, offset, num_touch_data, max_touch_data;
u8 cable_state, battery_capacity, battery_charging;
+ u16 timestamp;
/* When using Bluetooth the header is 2 bytes longer, so skip these. */
- int data_offset = (sc->quirks & DUALSHOCK4_CONTROLLER_USB) ? 0 : 2;
+ int data_offset = (sc->quirks & DUALSHOCK4_CONTROLLER_BT) ? 2 : 0;
/* Second bit of third button byte is for the touchpad button. */
offset = data_offset + DS4_INPUT_REPORT_BUTTON_OFFSET;
input_report_key(sc->touchpad, BTN_LEFT, rd[offset+2] & 0x2);
/*
+ * The default behavior of the Dualshock 4 is to send reports using
+ * report type 1 when running over Bluetooth. However, when feature
+ * report 2 is requested during the controller initialization it starts
+ * sending input reports in report 17. Since report 17 is undefined
+ * in the default HID descriptor, the HID layer won't generate events.
+ * While it is possible (and this was done before) to fixup the HID
+ * descriptor to add this mapping, it was better to do this manually.
+ * The reason is there were various pieces software both open and closed
+ * source, relying on the descriptors to be the same across various
+ * operating systems. If the descriptors wouldn't match some
+ * applications e.g. games on Wine would not be able to function due
+ * to different descriptors, which such applications are not parsing.
+ */
+ if (rd[0] == 17) {
+ int value;
+
+ offset = data_offset + DS4_INPUT_REPORT_AXIS_OFFSET;
+ input_report_abs(input_dev, ABS_X, rd[offset]);
+ input_report_abs(input_dev, ABS_Y, rd[offset+1]);
+ input_report_abs(input_dev, ABS_RX, rd[offset+2]);
+ input_report_abs(input_dev, ABS_RY, rd[offset+3]);
+
+ value = rd[offset+4] & 0xf;
+ if (value > 7)
+ value = 8; /* Center 0, 0 */
+ input_report_abs(input_dev, ABS_HAT0X, ds4_hat_mapping[value].x);
+ input_report_abs(input_dev, ABS_HAT0Y, ds4_hat_mapping[value].y);
+
+ input_report_key(input_dev, BTN_WEST, rd[offset+4] & 0x10);
+ input_report_key(input_dev, BTN_SOUTH, rd[offset+4] & 0x20);
+ input_report_key(input_dev, BTN_EAST, rd[offset+4] & 0x40);
+ input_report_key(input_dev, BTN_NORTH, rd[offset+4] & 0x80);
+
+ input_report_key(input_dev, BTN_TL, rd[offset+5] & 0x1);
+ input_report_key(input_dev, BTN_TR, rd[offset+5] & 0x2);
+ input_report_key(input_dev, BTN_TL2, rd[offset+5] & 0x4);
+ input_report_key(input_dev, BTN_TR2, rd[offset+5] & 0x8);
+ input_report_key(input_dev, BTN_SELECT, rd[offset+5] & 0x10);
+ input_report_key(input_dev, BTN_START, rd[offset+5] & 0x20);
+ input_report_key(input_dev, BTN_THUMBL, rd[offset+5] & 0x40);
+ input_report_key(input_dev, BTN_THUMBR, rd[offset+5] & 0x80);
+
+ input_report_key(input_dev, BTN_MODE, rd[offset+6] & 0x1);
+
+ input_report_abs(input_dev, ABS_Z, rd[offset+7]);
+ input_report_abs(input_dev, ABS_RZ, rd[offset+8]);
+
+ input_sync(input_dev);
+ }
+
+ /* Convert timestamp (in 5.33us unit) to timestamp_us */
+ offset = data_offset + DS4_INPUT_REPORT_TIMESTAMP_OFFSET;
+ timestamp = get_unaligned_le16(&rd[offset]);
+ if (!sc->timestamp_initialized) {
+ sc->timestamp_us = ((unsigned int)timestamp * 16) / 3;
+ sc->timestamp_initialized = true;
+ } else {
+ u16 delta;
+
+ if (sc->prev_timestamp > timestamp)
+ delta = (U16_MAX - sc->prev_timestamp + timestamp + 1);
+ else
+ delta = timestamp - sc->prev_timestamp;
+ sc->timestamp_us += (delta * 16) / 3;
+ }
+ sc->prev_timestamp = timestamp;
+ input_event(sc->sensor_dev, EV_MSC, MSC_TIMESTAMP, sc->timestamp_us);
+
+ offset = data_offset + DS4_INPUT_REPORT_GYRO_X_OFFSET;
+ for (n = 0; n < 6; n++) {
+ /* Store data in int for more precision during mult_frac. */
+ int raw_data = (short)((rd[offset+1] << 8) | rd[offset]);
+ struct ds4_calibration_data *calib = &sc->ds4_calib_data[n];
+
+ /* High precision is needed during calibration, but the
+ * calibrated values are within 32-bit.
+ * Note: we swap numerator 'x' and 'numer' in mult_frac for
+ * precision reasons so we don't need 64-bit.
+ */
+ int calib_data = mult_frac(calib->sens_numer,
+ raw_data - calib->bias,
+ calib->sens_denom);
+
+ input_report_abs(sc->sensor_dev, calib->abs_code, calib_data);
+ offset += 2;
+ }
+ input_sync(sc->sensor_dev);
+
+ /*
* The lower 4 bits of byte 30 (or 32 for BT) contain the battery level
* and the 5th bit contains the USB cable state.
*/
@@ -1341,7 +1030,7 @@
* Trackpad data starts 2 bytes later (e.g. 35 for USB).
*/
offset = data_offset + DS4_INPUT_REPORT_TOUCHPAD_OFFSET;
- max_touch_data = (sc->quirks & DUALSHOCK4_CONTROLLER_USB) ? 3 : 4;
+ max_touch_data = (sc->quirks & DUALSHOCK4_CONTROLLER_BT) ? 4 : 3;
if (rd[offset] > 0 && rd[offset] <= max_touch_data)
num_touch_data = rd[offset];
else
@@ -1415,47 +1104,79 @@
} else if ((sc->quirks & NAVIGATION_CONTROLLER) && rd[0] == 0x01 &&
size == 49) {
sixaxis_parse_report(sc, rd, size);
- } else if (((sc->quirks & DUALSHOCK4_CONTROLLER_USB) && rd[0] == 0x01 &&
- size == 64) || ((sc->quirks & DUALSHOCK4_CONTROLLER_BT)
- && rd[0] == 0x11 && size == 78)) {
- if (sc->quirks & DUALSHOCK4_CONTROLLER_BT) {
- /* CRC check */
- u8 bthdr = 0xA1;
- u32 crc;
- u32 report_crc;
+ } else if ((sc->quirks & DUALSHOCK4_CONTROLLER_USB) && rd[0] == 0x01 &&
+ size == 64) {
+ dualshock4_parse_report(sc, rd, size);
+ } else if (((sc->quirks & DUALSHOCK4_CONTROLLER_BT) && rd[0] == 0x11 &&
+ size == 78)) {
+ /* CRC check */
+ u8 bthdr = 0xA1;
+ u32 crc;
+ u32 report_crc;
- crc = crc32_le(0xFFFFFFFF, &bthdr, 1);
- crc = ~crc32_le(crc, rd, DS4_INPUT_REPORT_0x11_SIZE-4);
- report_crc = get_unaligned_le32(&rd[DS4_INPUT_REPORT_0x11_SIZE-4]);
- if (crc != report_crc) {
- hid_dbg(sc->hdev, "DualShock 4 input report's CRC check failed, received crc 0x%0x != 0x%0x\n",
- report_crc, crc);
- return -EILSEQ;
- }
+ crc = crc32_le(0xFFFFFFFF, &bthdr, 1);
+ crc = ~crc32_le(crc, rd, DS4_INPUT_REPORT_0x11_SIZE-4);
+ report_crc = get_unaligned_le32(&rd[DS4_INPUT_REPORT_0x11_SIZE-4]);
+ if (crc != report_crc) {
+ hid_dbg(sc->hdev, "DualShock 4 input report's CRC check failed, received crc 0x%0x != 0x%0x\n",
+ report_crc, crc);
+ return -EILSEQ;
}
+ dualshock4_parse_report(sc, rd, size);
+ } else if ((sc->quirks & DUALSHOCK4_DONGLE) && rd[0] == 0x01 &&
+ size == 64) {
+ unsigned long flags;
+ enum ds4_dongle_state dongle_state;
+
/*
* In the case of a DS4 USB dongle, bit[2] of byte 31 indicates
* if a DS4 is actually connected (indicated by '0').
* For non-dongle, this bit is always 0 (connected).
*/
- if (sc->hdev->vendor == USB_VENDOR_ID_SONY &&
- sc->hdev->product == USB_DEVICE_ID_SONY_PS4_CONTROLLER_DONGLE) {
- bool connected = (rd[31] & 0x04) ? false : true;
+ bool connected = (rd[31] & 0x04) ? false : true;
- if (!sc->ds4_dongle_connected && connected) {
- hid_info(sc->hdev, "DualShock 4 USB dongle: controller connected\n");
- sony_set_leds(sc);
- sc->ds4_dongle_connected = true;
- } else if (sc->ds4_dongle_connected && !connected) {
- hid_info(sc->hdev, "DualShock 4 USB dongle: controller disconnected\n");
- sc->ds4_dongle_connected = false;
- /* Return 0, so hidraw can get the report. */
- return 0;
- } else if (!sc->ds4_dongle_connected) {
- /* Return 0, so hidraw can get the report. */
- return 0;
- }
+ spin_lock_irqsave(&sc->lock, flags);
+ dongle_state = sc->ds4_dongle_state;
+ spin_unlock_irqrestore(&sc->lock, flags);
+
+ /*
+ * The dongle always sends input reports even when no
+ * DS4 is attached. When a DS4 is connected, we need to
+ * obtain calibration data before we can use it.
+ * The code below tracks dongle state and kicks of
+ * calibration when needed and only allows us to process
+ * input if a DS4 is actually connected.
+ */
+ if (dongle_state == DONGLE_DISCONNECTED && connected) {
+ hid_info(sc->hdev, "DualShock 4 USB dongle: controller connected\n");
+ sony_set_leds(sc);
+
+ spin_lock_irqsave(&sc->lock, flags);
+ sc->ds4_dongle_state = DONGLE_CALIBRATING;
+ spin_unlock_irqrestore(&sc->lock, flags);
+
+ sony_schedule_work(sc, SONY_WORKER_HOTPLUG);
+
+ /* Don't process the report since we don't have
+ * calibration data, but let hidraw have it anyway.
+ */
+ return 0;
+ } else if ((dongle_state == DONGLE_CONNECTED ||
+ dongle_state == DONGLE_DISABLED) && !connected) {
+ hid_info(sc->hdev, "DualShock 4 USB dongle: controller disconnected\n");
+
+ spin_lock_irqsave(&sc->lock, flags);
+ sc->ds4_dongle_state = DONGLE_DISCONNECTED;
+ spin_unlock_irqrestore(&sc->lock, flags);
+
+ /* Return 0, so hidraw can get the report. */
+ return 0;
+ } else if (dongle_state == DONGLE_CALIBRATING ||
+ dongle_state == DONGLE_DISABLED ||
+ dongle_state == DONGLE_DISCONNECTED) {
+ /* Return 0, so hidraw can get the report. */
+ return 0;
}
dualshock4_parse_report(sc, rd, size);
@@ -1463,7 +1184,7 @@
if (sc->defer_initialization) {
sc->defer_initialization = 0;
- sony_schedule_work(sc);
+ sony_schedule_work(sc, SONY_WORKER_STATE);
}
return 0;
@@ -1501,10 +1222,16 @@
if (sc->quirks & PS3REMOTE)
return ps3remote_mapping(hdev, hi, field, usage, bit, max);
+ if (sc->quirks & NAVIGATION_CONTROLLER)
+ return navigation_mapping(hdev, hi, field, usage, bit, max);
+
+ if (sc->quirks & SIXAXIS_CONTROLLER)
+ return sixaxis_mapping(hdev, hi, field, usage, bit, max);
if (sc->quirks & DUALSHOCK4_CONTROLLER)
return ds4_mapping(hdev, hi, field, usage, bit, max);
+
/* Let hid-core decide for the others */
return 0;
}
@@ -1541,7 +1268,7 @@
snprintf(name, name_sz, "%s" DS4_TOUCHPAD_SUFFIX, sc->hdev->name);
sc->touchpad->name = name;
- ret = input_mt_init_slots(sc->touchpad, touch_count, 0);
+ ret = input_mt_init_slots(sc->touchpad, touch_count, INPUT_MT_POINTER);
if (ret < 0)
goto err;
@@ -1581,6 +1308,103 @@
sc->touchpad = NULL;
}
+static int sony_register_sensors(struct sony_sc *sc)
+{
+ size_t name_sz;
+ char *name;
+ int ret;
+ int range;
+
+ sc->sensor_dev = input_allocate_device();
+ if (!sc->sensor_dev)
+ return -ENOMEM;
+
+ input_set_drvdata(sc->sensor_dev, sc);
+ sc->sensor_dev->dev.parent = &sc->hdev->dev;
+ sc->sensor_dev->phys = sc->hdev->phys;
+ sc->sensor_dev->uniq = sc->hdev->uniq;
+ sc->sensor_dev->id.bustype = sc->hdev->bus;
+ sc->sensor_dev->id.vendor = sc->hdev->vendor;
+ sc->sensor_dev->id.product = sc->hdev->product;
+ sc->sensor_dev->id.version = sc->hdev->version;
+
+ /* Append a suffix to the controller name as there are various
+ * DS4 compatible non-Sony devices with different names.
+ */
+ name_sz = strlen(sc->hdev->name) + sizeof(SENSOR_SUFFIX);
+ name = kzalloc(name_sz, GFP_KERNEL);
+ if (!name) {
+ ret = -ENOMEM;
+ goto err;
+ }
+ snprintf(name, name_sz, "%s" SENSOR_SUFFIX, sc->hdev->name);
+ sc->sensor_dev->name = name;
+
+ if (sc->quirks & SIXAXIS_CONTROLLER) {
+ /* For the DS3 we only support the accelerometer, which works
+ * quite well even without calibration. The device also has
+ * a 1-axis gyro, but it is very difficult to manage from within
+ * the driver even to get data, the sensor is inaccurate and
+ * the behavior is very different between hardware revisions.
+ */
+ input_set_abs_params(sc->sensor_dev, ABS_X, -512, 511, 4, 0);
+ input_set_abs_params(sc->sensor_dev, ABS_Y, -512, 511, 4, 0);
+ input_set_abs_params(sc->sensor_dev, ABS_Z, -512, 511, 4, 0);
+ input_abs_set_res(sc->sensor_dev, ABS_X, SIXAXIS_ACC_RES_PER_G);
+ input_abs_set_res(sc->sensor_dev, ABS_Y, SIXAXIS_ACC_RES_PER_G);
+ input_abs_set_res(sc->sensor_dev, ABS_Z, SIXAXIS_ACC_RES_PER_G);
+ } else if (sc->quirks & DUALSHOCK4_CONTROLLER) {
+ range = DS4_ACC_RES_PER_G*4;
+ input_set_abs_params(sc->sensor_dev, ABS_X, -range, range, 16, 0);
+ input_set_abs_params(sc->sensor_dev, ABS_Y, -range, range, 16, 0);
+ input_set_abs_params(sc->sensor_dev, ABS_Z, -range, range, 16, 0);
+ input_abs_set_res(sc->sensor_dev, ABS_X, DS4_ACC_RES_PER_G);
+ input_abs_set_res(sc->sensor_dev, ABS_Y, DS4_ACC_RES_PER_G);
+ input_abs_set_res(sc->sensor_dev, ABS_Z, DS4_ACC_RES_PER_G);
+
+ range = DS4_GYRO_RES_PER_DEG_S*2048;
+ input_set_abs_params(sc->sensor_dev, ABS_RX, -range, range, 16, 0);
+ input_set_abs_params(sc->sensor_dev, ABS_RY, -range, range, 16, 0);
+ input_set_abs_params(sc->sensor_dev, ABS_RZ, -range, range, 16, 0);
+ input_abs_set_res(sc->sensor_dev, ABS_RX, DS4_GYRO_RES_PER_DEG_S);
+ input_abs_set_res(sc->sensor_dev, ABS_RY, DS4_GYRO_RES_PER_DEG_S);
+ input_abs_set_res(sc->sensor_dev, ABS_RZ, DS4_GYRO_RES_PER_DEG_S);
+
+ __set_bit(EV_MSC, sc->sensor_dev->evbit);
+ __set_bit(MSC_TIMESTAMP, sc->sensor_dev->mscbit);
+ }
+
+ __set_bit(INPUT_PROP_ACCELEROMETER, sc->sensor_dev->propbit);
+
+ ret = input_register_device(sc->sensor_dev);
+ if (ret < 0)
+ goto err;
+
+ return 0;
+
+err:
+ kfree(sc->sensor_dev->name);
+ sc->sensor_dev->name = NULL;
+
+ input_free_device(sc->sensor_dev);
+ sc->sensor_dev = NULL;
+
+ return ret;
+}
+
+static void sony_unregister_sensors(struct sony_sc *sc)
+{
+ if (!sc->sensor_dev)
+ return;
+
+ kfree(sc->sensor_dev->name);
+ sc->sensor_dev->name = NULL;
+
+ input_unregister_device(sc->sensor_dev);
+ sc->sensor_dev = NULL;
+}
+
+
/*
* Sending HID_REQ_GET_REPORT changes the operation mode of the ps3 controller
* to "operational". Without this, the ps3 controller will not report any
@@ -1646,26 +1470,176 @@
}
/*
- * Requesting feature report 0x02 in Bluetooth mode changes the state of the
- * controller so that it sends full input reports of type 0x11.
+ * Request DS4 calibration data for the motion sensors.
+ * For Bluetooth this also affects the operating mode (see below).
*/
-static int dualshock4_set_operational_bt(struct hid_device *hdev)
+static int dualshock4_get_calibration_data(struct sony_sc *sc)
{
u8 *buf;
int ret;
+ short gyro_pitch_bias, gyro_pitch_plus, gyro_pitch_minus;
+ short gyro_yaw_bias, gyro_yaw_plus, gyro_yaw_minus;
+ short gyro_roll_bias, gyro_roll_plus, gyro_roll_minus;
+ short gyro_speed_plus, gyro_speed_minus;
+ short acc_x_plus, acc_x_minus;
+ short acc_y_plus, acc_y_minus;
+ short acc_z_plus, acc_z_minus;
+ int speed_2x;
+ int range_2g;
- buf = kmalloc(DS4_FEATURE_REPORT_0x02_SIZE, GFP_KERNEL);
- if (!buf)
- return -ENOMEM;
+ /* For Bluetooth we use a different request, which supports CRC.
+ * Note: in Bluetooth mode feature report 0x02 also changes the state
+ * of the controller, so that it sends input reports of type 0x11.
+ */
+ if (sc->quirks & (DUALSHOCK4_CONTROLLER_USB | DUALSHOCK4_DONGLE)) {
+ buf = kmalloc(DS4_FEATURE_REPORT_0x02_SIZE, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
- ret = hid_hw_raw_request(hdev, 0x02, buf, DS4_FEATURE_REPORT_0x02_SIZE,
- HID_FEATURE_REPORT, HID_REQ_GET_REPORT);
+ ret = hid_hw_raw_request(sc->hdev, 0x02, buf,
+ DS4_FEATURE_REPORT_0x02_SIZE,
+ HID_FEATURE_REPORT,
+ HID_REQ_GET_REPORT);
+ if (ret < 0)
+ goto err_stop;
+ } else {
+ u8 bthdr = 0xA3;
+ u32 crc;
+ u32 report_crc;
+ int retries;
+ buf = kmalloc(DS4_FEATURE_REPORT_0x05_SIZE, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ for (retries = 0; retries < 3; retries++) {
+ ret = hid_hw_raw_request(sc->hdev, 0x05, buf,
+ DS4_FEATURE_REPORT_0x05_SIZE,
+ HID_FEATURE_REPORT,
+ HID_REQ_GET_REPORT);
+ if (ret < 0)
+ goto err_stop;
+
+ /* CRC check */
+ crc = crc32_le(0xFFFFFFFF, &bthdr, 1);
+ crc = ~crc32_le(crc, buf, DS4_FEATURE_REPORT_0x05_SIZE-4);
+ report_crc = get_unaligned_le32(&buf[DS4_FEATURE_REPORT_0x05_SIZE-4]);
+ if (crc != report_crc) {
+ hid_warn(sc->hdev, "DualShock 4 calibration report's CRC check failed, received crc 0x%0x != 0x%0x\n",
+ report_crc, crc);
+ if (retries < 2) {
+ hid_warn(sc->hdev, "Retrying DualShock 4 get calibration report request\n");
+ continue;
+ } else {
+ ret = -EILSEQ;
+ goto err_stop;
+ }
+ } else {
+ break;
+ }
+ }
+ }
+
+ gyro_pitch_bias = get_unaligned_le16(&buf[1]);
+ gyro_yaw_bias = get_unaligned_le16(&buf[3]);
+ gyro_roll_bias = get_unaligned_le16(&buf[5]);
+ if (sc->quirks & DUALSHOCK4_CONTROLLER_USB) {
+ gyro_pitch_plus = get_unaligned_le16(&buf[7]);
+ gyro_pitch_minus = get_unaligned_le16(&buf[9]);
+ gyro_yaw_plus = get_unaligned_le16(&buf[11]);
+ gyro_yaw_minus = get_unaligned_le16(&buf[13]);
+ gyro_roll_plus = get_unaligned_le16(&buf[15]);
+ gyro_roll_minus = get_unaligned_le16(&buf[17]);
+ } else {
+ /* BT + Dongle */
+ gyro_pitch_plus = get_unaligned_le16(&buf[7]);
+ gyro_yaw_plus = get_unaligned_le16(&buf[9]);
+ gyro_roll_plus = get_unaligned_le16(&buf[11]);
+ gyro_pitch_minus = get_unaligned_le16(&buf[13]);
+ gyro_yaw_minus = get_unaligned_le16(&buf[15]);
+ gyro_roll_minus = get_unaligned_le16(&buf[17]);
+ }
+ gyro_speed_plus = get_unaligned_le16(&buf[19]);
+ gyro_speed_minus = get_unaligned_le16(&buf[21]);
+ acc_x_plus = get_unaligned_le16(&buf[23]);
+ acc_x_minus = get_unaligned_le16(&buf[25]);
+ acc_y_plus = get_unaligned_le16(&buf[27]);
+ acc_y_minus = get_unaligned_le16(&buf[29]);
+ acc_z_plus = get_unaligned_le16(&buf[31]);
+ acc_z_minus = get_unaligned_le16(&buf[33]);
+
+ /* Set gyroscope calibration and normalization parameters.
+ * Data values will be normalized to 1/DS4_GYRO_RES_PER_DEG_S degree/s.
+ */
+ speed_2x = (gyro_speed_plus + gyro_speed_minus);
+ sc->ds4_calib_data[0].abs_code = ABS_RX;
+ sc->ds4_calib_data[0].bias = gyro_pitch_bias;
+ sc->ds4_calib_data[0].sens_numer = speed_2x*DS4_GYRO_RES_PER_DEG_S;
+ sc->ds4_calib_data[0].sens_denom = gyro_pitch_plus - gyro_pitch_minus;
+
+ sc->ds4_calib_data[1].abs_code = ABS_RY;
+ sc->ds4_calib_data[1].bias = gyro_yaw_bias;
+ sc->ds4_calib_data[1].sens_numer = speed_2x*DS4_GYRO_RES_PER_DEG_S;
+ sc->ds4_calib_data[1].sens_denom = gyro_yaw_plus - gyro_yaw_minus;
+
+ sc->ds4_calib_data[2].abs_code = ABS_RZ;
+ sc->ds4_calib_data[2].bias = gyro_roll_bias;
+ sc->ds4_calib_data[2].sens_numer = speed_2x*DS4_GYRO_RES_PER_DEG_S;
+ sc->ds4_calib_data[2].sens_denom = gyro_roll_plus - gyro_roll_minus;
+
+ /* Set accelerometer calibration and normalization parameters.
+ * Data values will be normalized to 1/DS4_ACC_RES_PER_G G.
+ */
+ range_2g = acc_x_plus - acc_x_minus;
+ sc->ds4_calib_data[3].abs_code = ABS_X;
+ sc->ds4_calib_data[3].bias = acc_x_plus - range_2g / 2;
+ sc->ds4_calib_data[3].sens_numer = 2*DS4_ACC_RES_PER_G;
+ sc->ds4_calib_data[3].sens_denom = range_2g;
+
+ range_2g = acc_y_plus - acc_y_minus;
+ sc->ds4_calib_data[4].abs_code = ABS_Y;
+ sc->ds4_calib_data[4].bias = acc_y_plus - range_2g / 2;
+ sc->ds4_calib_data[4].sens_numer = 2*DS4_ACC_RES_PER_G;
+ sc->ds4_calib_data[4].sens_denom = range_2g;
+
+ range_2g = acc_z_plus - acc_z_minus;
+ sc->ds4_calib_data[5].abs_code = ABS_Z;
+ sc->ds4_calib_data[5].bias = acc_z_plus - range_2g / 2;
+ sc->ds4_calib_data[5].sens_numer = 2*DS4_ACC_RES_PER_G;
+ sc->ds4_calib_data[5].sens_denom = range_2g;
+
+err_stop:
kfree(buf);
-
return ret;
}
+static void dualshock4_calibration_work(struct work_struct *work)
+{
+ struct sony_sc *sc = container_of(work, struct sony_sc, hotplug_worker);
+ unsigned long flags;
+ enum ds4_dongle_state dongle_state;
+ int ret;
+
+ ret = dualshock4_get_calibration_data(sc);
+ if (ret < 0) {
+ /* This call is very unlikely to fail for the dongle. When it
+ * fails we are probably in a very bad state, so mark the
+ * dongle as disabled. We will re-enable the dongle if a new
+ * DS4 hotplug is detect from sony_raw_event as any issues
+ * are likely resolved then (the dongle is quite stupid).
+ */
+ hid_err(sc->hdev, "DualShock 4 USB dongle: calibration failed, disabling device\n");
+ dongle_state = DONGLE_DISABLED;
+ } else {
+ hid_info(sc->hdev, "DualShock 4 USB dongle: calibration completed\n");
+ dongle_state = DONGLE_CONNECTED;
+ }
+
+ spin_lock_irqsave(&sc->lock, flags);
+ sc->ds4_dongle_state = dongle_state;
+ spin_unlock_irqrestore(&sc->lock, flags);
+}
+
static void sixaxis_set_leds_from_id(struct sony_sc *sc)
{
static const u8 sixaxis_leds[10][4] = {
@@ -1696,10 +1670,10 @@
{
/* The first 4 color/index entries match what the PS4 assigns */
static const u8 color_code[7][3] = {
- /* Blue */ { 0x00, 0x00, 0x01 },
- /* Red */ { 0x01, 0x00, 0x00 },
- /* Green */ { 0x00, 0x01, 0x00 },
- /* Pink */ { 0x02, 0x00, 0x01 },
+ /* Blue */ { 0x00, 0x00, 0x40 },
+ /* Red */ { 0x40, 0x00, 0x00 },
+ /* Green */ { 0x00, 0x40, 0x00 },
+ /* Pink */ { 0x20, 0x00, 0x20 },
/* Orange */ { 0x02, 0x01, 0x00 },
/* Teal */ { 0x00, 0x01, 0x01 },
/* White */ { 0x01, 0x01, 0x01 }
@@ -1740,7 +1714,7 @@
static void sony_set_leds(struct sony_sc *sc)
{
if (!(sc->quirks & BUZZ_CONTROLLER))
- sony_schedule_work(sc);
+ sony_schedule_work(sc, SONY_WORKER_STATE);
else
buzz_set_leds(sc);
}
@@ -1851,7 +1825,7 @@
new_off != drv_data->led_delay_off[n]) {
drv_data->led_delay_on[n] = new_on;
drv_data->led_delay_off[n] = new_off;
- sony_schedule_work(drv_data);
+ sony_schedule_work(drv_data, SONY_WORKER_STATE);
}
return 0;
@@ -1964,6 +1938,7 @@
led->name = name;
led->brightness = sc->led_state[n];
led->max_brightness = max_brightness[n];
+ led->flags = LED_CORE_SUSPENDRESUME;
led->brightness_get = sony_led_get_brightness;
led->brightness_set = sony_led_set_brightness;
@@ -2052,26 +2027,24 @@
int offset;
/*
- * NOTE: The buf[1] field of the Bluetooth report controls
- * the Dualshock 4 reporting rate.
- *
- * Known values include:
- *
- * 0x80 - 1000hz (full speed)
- * 0xA0 - 31hz
- * 0xB0 - 20hz
- * 0xD0 - 66hz
+ * NOTE: The lower 6 bits of buf[1] field of the Bluetooth report
+ * control the interval at which Dualshock 4 reports data:
+ * 0x00 - 1ms
+ * 0x01 - 1ms
+ * 0x02 - 2ms
+ * 0x3E - 62ms
+ * 0x3F - disabled
*/
- if (sc->quirks & DUALSHOCK4_CONTROLLER_USB) {
+ if (sc->quirks & (DUALSHOCK4_CONTROLLER_USB | DUALSHOCK4_DONGLE)) {
memset(buf, 0, DS4_OUTPUT_REPORT_0x05_SIZE);
buf[0] = 0x05;
- buf[1] = 0xFF;
+ buf[1] = 0x07; /* blink + LEDs + motor */
offset = 4;
} else {
memset(buf, 0, DS4_OUTPUT_REPORT_0x11_SIZE);
buf[0] = 0x11;
- buf[1] = 0xC0; /* HID + CRC */
- buf[3] = 0x0F;
+ buf[1] = 0xC0 /* HID + CRC */ | sc->ds4_bt_poll_interval;
+ buf[3] = 0x07; /* blink + LEDs + motor */
offset = 6;
}
@@ -2095,7 +2068,7 @@
buf[offset++] = sc->led_delay_on[3];
buf[offset++] = sc->led_delay_off[3];
- if (sc->quirks & DUALSHOCK4_CONTROLLER_USB)
+ if (sc->quirks & (DUALSHOCK4_CONTROLLER_USB | DUALSHOCK4_DONGLE))
hid_hw_output_report(hdev, buf, DS4_OUTPUT_REPORT_0x05_SIZE);
else {
/* CRC generation */
@@ -2152,7 +2125,7 @@
else if (sc->quirks & DUALSHOCK4_CONTROLLER_BT)
sc->output_report_dmabuf = kmalloc(DS4_OUTPUT_REPORT_0x11_SIZE,
GFP_KERNEL);
- else if (sc->quirks & DUALSHOCK4_CONTROLLER_USB)
+ else if (sc->quirks & (DUALSHOCK4_CONTROLLER_USB | DUALSHOCK4_DONGLE))
sc->output_report_dmabuf = kmalloc(DS4_OUTPUT_REPORT_0x05_SIZE,
GFP_KERNEL);
else if (sc->quirks & MOTION_CONTROLLER)
@@ -2180,7 +2153,7 @@
sc->left = effect->u.rumble.strong_magnitude / 256;
sc->right = effect->u.rumble.weak_magnitude / 256;
- sony_schedule_work(sc);
+ sony_schedule_work(sc, SONY_WORKER_STATE);
return 0;
}
@@ -2397,7 +2370,7 @@
hid_warn(sc->hdev, "UNIQ does not contain a MAC address; duplicate check skipped\n");
return 0;
}
- } else if (sc->quirks & DUALSHOCK4_CONTROLLER_USB) {
+ } else if (sc->quirks & (DUALSHOCK4_CONTROLLER_USB | DUALSHOCK4_DONGLE)) {
buf = kmalloc(DS4_FEATURE_REPORT_0x81_SIZE, GFP_KERNEL);
if (!buf)
return -ENOMEM;
@@ -2451,6 +2424,12 @@
*/
for (n = 0; n < 6; n++)
sc->mac_address[5-n] = buf[4+n];
+
+ snprintf(sc->hdev->uniq, sizeof(sc->hdev->uniq),
+ "%02hhx:%02hhx:%02hhx:%02hhx:%02hhx:%02hhx",
+ sc->mac_address[5], sc->mac_address[4],
+ sc->mac_address[3], sc->mac_address[2],
+ sc->mac_address[1], sc->mac_address[0]);
} else {
return 0;
}
@@ -2501,18 +2480,21 @@
{
sc->send_output_report = send_output_report;
- if (!sc->worker_initialized)
+ if (!sc->state_worker_initialized)
INIT_WORK(&sc->state_worker, sony_state_worker);
- sc->worker_initialized = 1;
+ sc->state_worker_initialized = 1;
}
static inline void sony_cancel_work_sync(struct sony_sc *sc)
{
- if (sc->worker_initialized)
+ if (sc->hotplug_worker_initialized)
+ cancel_work_sync(&sc->hotplug_worker);
+ if (sc->state_worker_initialized)
cancel_work_sync(&sc->state_worker);
}
+
static int sony_input_configured(struct hid_device *hdev,
struct hid_input *hidinput)
{
@@ -2526,14 +2508,17 @@
goto err_stop;
}
+ ret = append_dev_id = sony_check_add(sc);
+ if (ret < 0)
+ goto err_stop;
+
ret = sony_allocate_output_report(sc);
if (ret < 0) {
hid_err(hdev, "failed to allocate the output report buffer\n");
goto err_stop;
}
- if ((sc->quirks & SIXAXIS_CONTROLLER_USB) ||
- (sc->quirks & NAVIGATION_CONTROLLER_USB)) {
+ if (sc->quirks & NAVIGATION_CONTROLLER_USB) {
/*
* The Sony Sixaxis does not handle HID Output Reports on the
* Interrupt EP like it could, so we need to force HID Output
@@ -2553,24 +2538,79 @@
hdev->quirks |= HID_QUIRK_NO_OUTPUT_REPORTS_ON_INTR_EP;
hdev->quirks |= HID_QUIRK_SKIP_OUTPUT_REPORT_ID;
sc->defer_initialization = 1;
+
ret = sixaxis_set_operational_usb(hdev);
+ if (ret < 0) {
+ hid_err(hdev, "Failed to set controller into operational mode\n");
+ goto err_stop;
+ }
+
sony_init_output_report(sc, sixaxis_send_output_report);
- } else if ((sc->quirks & SIXAXIS_CONTROLLER_BT) ||
- (sc->quirks & NAVIGATION_CONTROLLER_BT)) {
+ } else if (sc->quirks & NAVIGATION_CONTROLLER_BT) {
+ /*
+ * The Navigation controller wants output reports sent on the ctrl
+ * endpoint when connected via Bluetooth.
+ */
+ hdev->quirks |= HID_QUIRK_NO_OUTPUT_REPORTS_ON_INTR_EP;
+
+ ret = sixaxis_set_operational_bt(hdev);
+ if (ret < 0) {
+ hid_err(hdev, "Failed to set controller into operational mode\n");
+ goto err_stop;
+ }
+
+ sony_init_output_report(sc, sixaxis_send_output_report);
+ } else if (sc->quirks & SIXAXIS_CONTROLLER_USB) {
+ /*
+ * The Sony Sixaxis does not handle HID Output Reports on the
+ * Interrupt EP and the device only becomes active when the
+ * PS button is pressed. See comment for Navigation controller
+ * above for more details.
+ */
+ hdev->quirks |= HID_QUIRK_NO_OUTPUT_REPORTS_ON_INTR_EP;
+ hdev->quirks |= HID_QUIRK_SKIP_OUTPUT_REPORT_ID;
+ sc->defer_initialization = 1;
+
+ ret = sixaxis_set_operational_usb(hdev);
+ if (ret < 0) {
+ hid_err(hdev, "Failed to set controller into operational mode\n");
+ goto err_stop;
+ }
+
+ ret = sony_register_sensors(sc);
+ if (ret) {
+ hid_err(sc->hdev,
+ "Unable to initialize motion sensors: %d\n", ret);
+ goto err_stop;
+ }
+
+ sony_init_output_report(sc, sixaxis_send_output_report);
+ } else if (sc->quirks & SIXAXIS_CONTROLLER_BT) {
/*
* The Sixaxis wants output reports sent on the ctrl endpoint
* when connected via Bluetooth.
*/
hdev->quirks |= HID_QUIRK_NO_OUTPUT_REPORTS_ON_INTR_EP;
+
ret = sixaxis_set_operational_bt(hdev);
+ if (ret < 0) {
+ hid_err(hdev, "Failed to set controller into operational mode\n");
+ goto err_stop;
+ }
+
+ ret = sony_register_sensors(sc);
+ if (ret) {
+ hid_err(sc->hdev,
+ "Unable to initialize motion sensors: %d\n", ret);
+ goto err_stop;
+ }
+
sony_init_output_report(sc, sixaxis_send_output_report);
} else if (sc->quirks & DUALSHOCK4_CONTROLLER) {
- if (sc->quirks & DUALSHOCK4_CONTROLLER_BT) {
- ret = dualshock4_set_operational_bt(hdev);
- if (ret < 0) {
- hid_err(hdev, "failed to set the Dualshock 4 operational mode\n");
- goto err_stop;
- }
+ ret = dualshock4_get_calibration_data(sc);
+ if (ret < 0) {
+ hid_err(hdev, "Failed to get calibration data from Dualshock 4\n");
+ goto err_stop;
}
/*
@@ -2585,6 +2625,28 @@
goto err_stop;
}
+ ret = sony_register_sensors(sc);
+ if (ret) {
+ hid_err(sc->hdev,
+ "Unable to initialize motion sensors: %d\n", ret);
+ goto err_stop;
+ }
+
+ if (sc->quirks & DUALSHOCK4_CONTROLLER_BT) {
+ sc->ds4_bt_poll_interval = DS4_BT_DEFAULT_POLL_INTERVAL_MS;
+ ret = device_create_file(&sc->hdev->dev, &dev_attr_bt_poll_interval);
+ if (ret)
+ hid_warn(sc->hdev,
+ "can't create sysfs bt_poll_interval attribute err: %d\n",
+ ret);
+ }
+
+ if (sc->quirks & DUALSHOCK4_DONGLE) {
+ INIT_WORK(&sc->hotplug_worker, dualshock4_calibration_work);
+ sc->hotplug_worker_initialized = 1;
+ sc->ds4_dongle_state = DONGLE_DISCONNECTED;
+ }
+
sony_init_output_report(sc, dualshock4_send_output_report);
} else if (sc->quirks & MOTION_CONTROLLER) {
sony_init_output_report(sc, motion_send_output_report);
@@ -2592,13 +2654,6 @@
ret = 0;
}
- if (ret < 0)
- goto err_stop;
-
- ret = append_dev_id = sony_check_add(sc);
- if (ret < 0)
- goto err_stop;
-
if (sc->quirks & SONY_LED_SUPPORT) {
ret = sony_leds_init(sc);
if (ret < 0)
@@ -2628,12 +2683,20 @@
err_close:
hid_hw_close(hdev);
err_stop:
+ /* Piggy back on the default ds4_bt_ poll_interval to determine
+ * if we need to remove the file as we don't know for sure if we
+ * executed that logic.
+ */
+ if (sc->ds4_bt_poll_interval)
+ device_remove_file(&sc->hdev->dev, &dev_attr_bt_poll_interval);
if (sc->quirks & SONY_LED_SUPPORT)
sony_leds_remove(sc);
if (sc->quirks & SONY_BATTERY_SUPPORT)
sony_battery_remove(sc);
if (sc->touchpad)
sony_unregister_touchpad(sc);
+ if (sc->sensor_dev)
+ sony_unregister_sensors(sc);
sony_cancel_work_sync(sc);
kfree(sc->output_report_dmabuf);
sony_remove_dev_list(sc);
@@ -2675,13 +2738,13 @@
else if (sc->quirks & SIXAXIS_CONTROLLER)
connect_mask |= HID_CONNECT_HIDDEV_FORCE;
- /* Patch the hw version on DS4 compatible devices, so applications can
+ /* Patch the hw version on DS3/4 compatible devices, so applications can
* distinguish between the default HID mappings and the mappings defined
* by the Linux game controller spec. This is important for the SDL2
* library, which has a game controller database, which uses device ids
* in combination with version as a key.
*/
- if (sc->quirks & DUALSHOCK4_CONTROLLER)
+ if (sc->quirks & (SIXAXIS_CONTROLLER | DUALSHOCK4_CONTROLLER))
hdev->version |= 0x8000;
ret = hid_hw_start(hdev, connect_mask);
@@ -2721,6 +2784,12 @@
if (sc->touchpad)
sony_unregister_touchpad(sc);
+ if (sc->sensor_dev)
+ sony_unregister_sensors(sc);
+
+ if (sc->quirks & DUALSHOCK4_CONTROLLER_BT)
+ device_remove_file(&sc->hdev->dev, &dev_attr_bt_poll_interval);
+
sony_cancel_work_sync(sc);
kfree(sc->output_report_dmabuf);
@@ -2736,47 +2805,32 @@
static int sony_suspend(struct hid_device *hdev, pm_message_t message)
{
- /*
- * On suspend save the current LED state,
- * stop running force-feedback and blank the LEDS.
- */
- if (SONY_LED_SUPPORT || SONY_FF_SUPPORT) {
+#ifdef CONFIG_SONY_FF
+
+ /* On suspend stop any running force-feedback events */
+ if (SONY_FF_SUPPORT) {
struct sony_sc *sc = hid_get_drvdata(hdev);
-#ifdef CONFIG_SONY_FF
sc->left = sc->right = 0;
-#endif
-
- memcpy(sc->resume_led_state, sc->led_state,
- sizeof(sc->resume_led_state));
- memset(sc->led_state, 0, sizeof(sc->led_state));
-
sony_send_output_report(sc);
}
+#endif
return 0;
}
static int sony_resume(struct hid_device *hdev)
{
- /* Restore the state of controller LEDs on resume */
- if (SONY_LED_SUPPORT) {
- struct sony_sc *sc = hid_get_drvdata(hdev);
+ struct sony_sc *sc = hid_get_drvdata(hdev);
- memcpy(sc->led_state, sc->resume_led_state,
- sizeof(sc->led_state));
-
- /*
- * The Sixaxis and navigation controllers on USB need to be
- * reinitialized on resume or they won't behave properly.
- */
- if ((sc->quirks & SIXAXIS_CONTROLLER_USB) ||
- (sc->quirks & NAVIGATION_CONTROLLER_USB)) {
- sixaxis_set_operational_usb(sc->hdev);
- sc->defer_initialization = 1;
- }
-
- sony_set_leds(sc);
+ /*
+ * The Sixaxis and navigation controllers on USB need to be
+ * reinitialized on resume or they won't behave properly.
+ */
+ if ((sc->quirks & SIXAXIS_CONTROLLER_USB) ||
+ (sc->quirks & NAVIGATION_CONTROLLER_USB)) {
+ sixaxis_set_operational_usb(sc->hdev);
+ sc->defer_initialization = 1;
}
return 0;
@@ -2828,7 +2882,7 @@
{ HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_PS4_CONTROLLER_2),
.driver_data = DUALSHOCK4_CONTROLLER_BT },
{ HID_USB_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_PS4_CONTROLLER_DONGLE),
- .driver_data = DUALSHOCK4_CONTROLLER_USB },
+ .driver_data = DUALSHOCK4_DONGLE },
/* Nyko Core Controller for PS3 */
{ HID_USB_DEVICE(USB_VENDOR_ID_SINO_LITE, USB_DEVICE_ID_SINO_LITE_CONTROLLER),
.driver_data = SIXAXIS_CONTROLLER_USB | SINO_LITE_CONTROLLER },
diff --git a/drivers/hid/i2c-hid/i2c-hid.c b/drivers/hid/i2c-hid/i2c-hid.c
index ea3c354..8daa8ce 100644
--- a/drivers/hid/i2c-hid/i2c-hid.c
+++ b/drivers/hid/i2c-hid/i2c-hid.c
@@ -508,66 +508,6 @@
report->device->report_enum[report->type].numbered + 2;
}
-static void i2c_hid_init_report(struct hid_report *report, u8 *buffer,
- size_t bufsize)
-{
- struct hid_device *hid = report->device;
- struct i2c_client *client = hid->driver_data;
- struct i2c_hid *ihid = i2c_get_clientdata(client);
- unsigned int size, ret_size;
-
- size = i2c_hid_get_report_length(report);
- if (i2c_hid_get_report(client,
- report->type == HID_FEATURE_REPORT ? 0x03 : 0x01,
- report->id, buffer, size))
- return;
-
- i2c_hid_dbg(ihid, "report (len=%d): %*ph\n", size, size, buffer);
-
- ret_size = buffer[0] | (buffer[1] << 8);
-
- if (ret_size != size) {
- dev_err(&client->dev, "error in %s size:%d / ret_size:%d\n",
- __func__, size, ret_size);
- return;
- }
-
- /* hid->driver_lock is held as we are in probe function,
- * we just need to setup the input fields, so using
- * hid_report_raw_event is safe. */
- hid_report_raw_event(hid, report->type, buffer + 2, size - 2, 1);
-}
-
-/*
- * Initialize all reports
- */
-static void i2c_hid_init_reports(struct hid_device *hid)
-{
- struct hid_report *report;
- struct i2c_client *client = hid->driver_data;
- struct i2c_hid *ihid = i2c_get_clientdata(client);
- u8 *inbuf = kzalloc(ihid->bufsize, GFP_KERNEL);
-
- if (!inbuf) {
- dev_err(&client->dev, "can not retrieve initial reports\n");
- return;
- }
-
- /*
- * The device must be powered on while we fetch initial reports
- * from it.
- */
- pm_runtime_get_sync(&client->dev);
-
- list_for_each_entry(report,
- &hid->report_enum[HID_FEATURE_REPORT].report_list, list)
- i2c_hid_init_report(report, inbuf, ihid->bufsize);
-
- pm_runtime_put(&client->dev);
-
- kfree(inbuf);
-}
-
/*
* Traverse the supplied list of reports and find the longest
*/
@@ -789,9 +729,6 @@
return ret;
}
- if (!(hid->quirks & HID_QUIRK_NO_INIT_REPORTS))
- i2c_hid_init_reports(hid);
-
return 0;
}
@@ -994,6 +931,11 @@
}
pdata->hid_descriptor_address = val;
+ ret = of_property_read_u32(dev->of_node, "post-power-on-delay-ms",
+ &val);
+ if (!ret)
+ pdata->post_power_delay_ms = val;
+
return 0;
}
@@ -1053,6 +995,24 @@
ihid->pdata = *platform_data;
}
+ ihid->pdata.supply = devm_regulator_get(&client->dev, "vdd");
+ if (IS_ERR(ihid->pdata.supply)) {
+ ret = PTR_ERR(ihid->pdata.supply);
+ if (ret != -EPROBE_DEFER)
+ dev_err(&client->dev, "Failed to get regulator: %d\n",
+ ret);
+ goto err;
+ }
+
+ ret = regulator_enable(ihid->pdata.supply);
+ if (ret < 0) {
+ dev_err(&client->dev, "Failed to enable regulator: %d\n",
+ ret);
+ goto err;
+ }
+ if (ihid->pdata.post_power_delay_ms)
+ msleep(ihid->pdata.post_power_delay_ms);
+
i2c_set_clientdata(client, ihid);
ihid->client = client;
@@ -1068,7 +1028,7 @@
* real computation later. */
ret = i2c_hid_alloc_buffers(ihid, HID_MIN_BUFFER_SIZE);
if (ret < 0)
- goto err;
+ goto err_regulator;
pm_runtime_get_noresume(&client->dev);
pm_runtime_set_active(&client->dev);
@@ -1125,6 +1085,9 @@
pm_runtime_put_noidle(&client->dev);
pm_runtime_disable(&client->dev);
+err_regulator:
+ regulator_disable(ihid->pdata.supply);
+
err:
i2c_hid_free_buffers(ihid);
kfree(ihid);
@@ -1149,6 +1112,8 @@
if (ihid->bufsize)
i2c_hid_free_buffers(ihid);
+ regulator_disable(ihid->pdata.supply);
+
kfree(ihid);
return 0;
@@ -1199,6 +1164,10 @@
else
hid_warn(hid, "Failed to enable irq wake: %d\n",
wake_status);
+ } else {
+ ret = regulator_disable(ihid->pdata.supply);
+ if (ret < 0)
+ hid_warn(hid, "Failed to disable supply: %d\n", ret);
}
return 0;
@@ -1212,7 +1181,13 @@
struct hid_device *hid = ihid->hid;
int wake_status;
- if (device_may_wakeup(&client->dev) && ihid->irq_wake_enabled) {
+ if (!device_may_wakeup(&client->dev)) {
+ ret = regulator_enable(ihid->pdata.supply);
+ if (ret < 0)
+ hid_warn(hid, "Failed to enable supply: %d\n", ret);
+ if (ihid->pdata.post_power_delay_ms)
+ msleep(ihid->pdata.post_power_delay_ms);
+ } else if (ihid->irq_wake_enabled) {
wake_status = disable_irq_wake(client->irq);
if (!wake_status)
ihid->irq_wake_enabled = false;
diff --git a/drivers/hid/usbhid/hid-core.c b/drivers/hid/usbhid/hid-core.c
index 961bc6f..83772fa 100644
--- a/drivers/hid/usbhid/hid-core.c
+++ b/drivers/hid/usbhid/hid-core.c
@@ -52,6 +52,10 @@
module_param_named(mousepoll, hid_mousepoll_interval, uint, 0644);
MODULE_PARM_DESC(mousepoll, "Polling interval of mice");
+static unsigned int hid_jspoll_interval;
+module_param_named(jspoll, hid_jspoll_interval, uint, 0644);
+MODULE_PARM_DESC(jspoll, "Polling interval of joysticks");
+
static unsigned int ignoreled;
module_param_named(ignoreled, ignoreled, uint, 0644);
MODULE_PARM_DESC(ignoreled, "Autosuspend with active leds");
@@ -753,11 +757,9 @@
struct hid_report_enum *report_enum;
int err, ret;
- if (!(hid->quirks & HID_QUIRK_NO_INIT_INPUT_REPORTS)) {
- report_enum = &hid->report_enum[HID_INPUT_REPORT];
- list_for_each_entry(report, &report_enum->report_list, list)
- usbhid_submit_report(hid, report, USB_DIR_IN);
- }
+ report_enum = &hid->report_enum[HID_INPUT_REPORT];
+ list_for_each_entry(report, &report_enum->report_list, list)
+ usbhid_submit_report(hid, report, USB_DIR_IN);
report_enum = &hid->report_enum[HID_FEATURE_REPORT];
list_for_each_entry(report, &report_enum->report_list, list)
@@ -1004,10 +1006,9 @@
return -EINVAL;
}
- if (!(rdesc = kmalloc(rsize, GFP_KERNEL))) {
- dbg_hid("couldn't allocate rdesc memory\n");
+ rdesc = kmalloc(rsize, GFP_KERNEL);
+ if (!rdesc)
return -ENOMEM;
- }
hid_set_idle(dev, interface->desc.bInterfaceNumber, 0, 0);
@@ -1077,13 +1078,21 @@
if (hid->quirks & HID_QUIRK_FULLSPEED_INTERVAL &&
dev->speed == USB_SPEED_HIGH) {
interval = fls(endpoint->bInterval*8);
- printk(KERN_INFO "%s: Fixing fullspeed to highspeed interval: %d -> %d\n",
- hid->name, endpoint->bInterval, interval);
+ pr_info("%s: Fixing fullspeed to highspeed interval: %d -> %d\n",
+ hid->name, endpoint->bInterval, interval);
}
- /* Change the polling interval of mice. */
- if (hid->collection->usage == HID_GD_MOUSE && hid_mousepoll_interval > 0)
- interval = hid_mousepoll_interval;
+ /* Change the polling interval of mice and joysticks. */
+ switch (hid->collection->usage) {
+ case HID_GD_MOUSE:
+ if (hid_mousepoll_interval > 0)
+ interval = hid_mousepoll_interval;
+ break;
+ case HID_GD_JOYSTICK:
+ if (hid_jspoll_interval > 0)
+ interval = hid_jspoll_interval;
+ break;
+ }
ret = -ENOMEM;
if (usb_endpoint_dir_in(endpoint)) {
@@ -1120,9 +1129,6 @@
usbhid->urbctrl->transfer_dma = usbhid->ctrlbuf_dma;
usbhid->urbctrl->transfer_flags |= URB_NO_TRANSFER_DMA_MAP;
- if (!(hid->quirks & HID_QUIRK_NO_INIT_REPORTS))
- usbhid_init_reports(hid);
-
set_bit(HID_STARTED, &usbhid->iofl);
if (hid->quirks & HID_QUIRK_ALWAYS_POLL) {
@@ -1456,10 +1462,9 @@
* the size of the HID report descriptor has not changed.
*/
rdesc = kmalloc(hid->dev_rsize, GFP_KERNEL);
- if (!rdesc) {
- dbg_hid("couldn't allocate rdesc memory (post_reset)\n");
+ if (!rdesc)
return -ENOMEM;
- }
+
status = hid_get_class_descriptor(dev,
interface->desc.bInterfaceNumber,
HID_DT_REPORT, rdesc, hid->dev_rsize);
@@ -1637,7 +1642,7 @@
retval = usb_register(&hid_driver);
if (retval)
goto usb_register_fail;
- printk(KERN_INFO KBUILD_MODNAME ": " DRIVER_DESC "\n");
+ pr_info(KBUILD_MODNAME ": " DRIVER_DESC "\n");
return 0;
usb_register_fail:
diff --git a/drivers/hid/usbhid/hid-quirks.c b/drivers/hid/usbhid/hid-quirks.c
index a69a3c8..6316498 100644
--- a/drivers/hid/usbhid/hid-quirks.c
+++ b/drivers/hid/usbhid/hid-quirks.c
@@ -65,6 +65,7 @@
{ USB_VENDOR_ID_ATEN, USB_DEVICE_ID_ATEN_4PORTKVMC, HID_QUIRK_NOGET },
{ USB_VENDOR_ID_ATEN, USB_DEVICE_ID_ATEN_CS682, HID_QUIRK_NOGET },
{ USB_VENDOR_ID_ATEN, USB_DEVICE_ID_ATEN_CS692, HID_QUIRK_NOGET },
+ { USB_VENDOR_ID_ATEN, USB_DEVICE_ID_ATEN_CS1758, HID_QUIRK_NOGET },
{ USB_VENDOR_ID_CH, USB_DEVICE_ID_CH_FIGHTERSTICK, HID_QUIRK_NOGET },
{ USB_VENDOR_ID_CH, USB_DEVICE_ID_CH_COMBATSTICK, HID_QUIRK_NOGET },
{ USB_VENDOR_ID_CH, USB_DEVICE_ID_CH_FLIGHT_SIM_ECLIPSE_YOKE, HID_QUIRK_NOGET },
@@ -161,10 +162,11 @@
{ USB_VENDOR_ID_SYNAPTICS, USB_DEVICE_ID_SYNAPTICS_HD, HID_QUIRK_NO_INIT_REPORTS },
{ USB_VENDOR_ID_SYNAPTICS, USB_DEVICE_ID_SYNAPTICS_QUAD_HD, HID_QUIRK_NO_INIT_REPORTS },
{ USB_VENDOR_ID_SYNAPTICS, USB_DEVICE_ID_SYNAPTICS_TP_V103, HID_QUIRK_NO_INIT_REPORTS },
- { USB_VENDOR_ID_HOLTEK_ALT, USB_DEVICE_ID_HOLTEK_ALT_KEYBOARD_A096, HID_QUIRK_NO_INIT_INPUT_REPORTS },
+ { USB_VENDOR_ID_HOLTEK_ALT, USB_DEVICE_ID_HOLTEK_ALT_KEYBOARD_A096, HID_QUIRK_NO_INIT_REPORTS },
{ USB_VENDOR_ID_MULTIPLE_1781, USB_DEVICE_ID_RAPHNET_4NES4SNES_OLD, HID_QUIRK_MULTI_INPUT },
{ USB_VENDOR_ID_DRACAL_RAPHNET, USB_DEVICE_ID_RAPHNET_2NES2SNES, HID_QUIRK_MULTI_INPUT },
{ USB_VENDOR_ID_DRACAL_RAPHNET, USB_DEVICE_ID_RAPHNET_4NES4SNES, HID_QUIRK_MULTI_INPUT },
+ { USB_VENDOR_ID_INNOMEDIA, USB_DEVICE_ID_INNEX_GENESIS_ATARI, HID_QUIRK_MULTI_INPUT },
{ 0, 0 }
};
@@ -240,10 +242,8 @@
}
q_new = kmalloc(sizeof(struct quirks_list_struct), GFP_KERNEL);
- if (!q_new) {
- dbg_hid("Could not allocate quirks_list_struct\n");
+ if (!q_new)
return -ENOMEM;
- }
q_new->hid_bl_item.idVendor = idVendor;
q_new->hid_bl_item.idProduct = idProduct;
@@ -309,10 +309,9 @@
&idVendor, &idProduct, &quirks);
if (m != 3 ||
- usbhid_modify_dquirk(idVendor, idProduct, quirks) != 0) {
- printk(KERN_WARNING
- "Could not parse HID quirk module param %s\n",
- quirks_param[n]);
+ usbhid_modify_dquirk(idVendor, idProduct, quirks) != 0) {
+ pr_warn("Could not parse HID quirk module param %s\n",
+ quirks_param[n]);
}
}
diff --git a/drivers/hid/usbhid/hiddev.c b/drivers/hid/usbhid/hiddev.c
index 774bd70..0e06368 100644
--- a/drivers/hid/usbhid/hiddev.c
+++ b/drivers/hid/usbhid/hiddev.c
@@ -47,16 +47,6 @@
#endif
#define HIDDEV_BUFFER_SIZE 2048
-struct hiddev {
- int exist;
- int open;
- struct mutex existancelock;
- wait_queue_head_t wait;
- struct hid_device *hid;
- struct list_head list;
- spinlock_t list_lock;
-};
-
struct hiddev_list {
struct hiddev_usage_ref buffer[HIDDEV_BUFFER_SIZE];
int head;
@@ -690,6 +680,7 @@
case HIDIOCINITREPORT:
usbhid_init_reports(hid);
+ hiddev->initialized = true;
r = 0;
break;
@@ -791,6 +782,10 @@
case HIDIOCGUSAGES:
case HIDIOCSUSAGES:
case HIDIOCGCOLLECTIONINDEX:
+ if (!hiddev->initialized) {
+ usbhid_init_reports(hid);
+ hiddev->initialized = true;
+ }
r = hiddev_ioctl_usage(hiddev, cmd, user_arg);
break;
@@ -911,6 +906,15 @@
kfree(hiddev);
return -1;
}
+
+ /*
+ * If HID_QUIRK_NO_INIT_REPORTS is set, make sure we don't initialize
+ * the reports.
+ */
+ hiddev->initialized = hid->quirks & HID_QUIRK_NO_INIT_REPORTS;
+
+ hiddev->minor = usbhid->intf->minor;
+
return 0;
}
diff --git a/drivers/hid/wacom.h b/drivers/hid/wacom.h
index 38ee212..c7b9ab1 100644
--- a/drivers/hid/wacom.h
+++ b/drivers/hid/wacom.h
@@ -110,6 +110,7 @@
WACOM_WORKER_WIRELESS,
WACOM_WORKER_BATTERY,
WACOM_WORKER_REMOTE,
+ WACOM_WORKER_MODE_CHANGE,
};
struct wacom;
@@ -167,6 +168,7 @@
struct work_struct remote_work;
struct delayed_work init_work;
struct wacom_remote *remote;
+ struct work_struct mode_change_work;
bool generic_has_leds;
struct wacom_leds {
struct wacom_group_leds *groups;
@@ -196,6 +198,9 @@
case WACOM_WORKER_REMOTE:
schedule_work(&wacom->remote_work);
break;
+ case WACOM_WORKER_MODE_CHANGE:
+ schedule_work(&wacom->mode_change_work);
+ break;
}
}
diff --git a/drivers/hid/wacom_sys.c b/drivers/hid/wacom_sys.c
index e2666ef..0022c0d 100644
--- a/drivers/hid/wacom_sys.c
+++ b/drivers/hid/wacom_sys.c
@@ -325,6 +325,13 @@
if (features->type == HID_GENERIC) {
/* Any last-minute generic device setup */
+ if (wacom_wac->has_mode_change) {
+ if (wacom_wac->is_direct_mode)
+ features->device_type |= WACOM_DEVICETYPE_DIRECT;
+ else
+ features->device_type &= ~WACOM_DEVICETYPE_DIRECT;
+ }
+
if (features->touch_max > 1) {
if (features->device_type & WACOM_DEVICETYPE_DIRECT)
input_mt_init_slots(wacom_wac->touch_input,
@@ -2093,8 +2100,10 @@
wacom_wac->shared->touch_input = wacom_wac->touch_input;
}
- if (wacom_wac->has_mute_touch_switch)
+ if (wacom_wac->has_mute_touch_switch) {
wacom_wac->shared->has_mute_touch_switch = true;
+ wacom_wac->shared->is_touch_on = true;
+ }
if (wacom_wac->shared->has_mute_touch_switch &&
wacom_wac->shared->touch_input) {
@@ -2490,6 +2499,46 @@
}
}
+static void wacom_mode_change_work(struct work_struct *work)
+{
+ struct wacom *wacom = container_of(work, struct wacom, mode_change_work);
+ struct wacom_shared *shared = wacom->wacom_wac.shared;
+ struct wacom *wacom1 = NULL;
+ struct wacom *wacom2 = NULL;
+ bool is_direct = wacom->wacom_wac.is_direct_mode;
+ int error = 0;
+
+ if (shared->pen) {
+ wacom1 = hid_get_drvdata(shared->pen);
+ wacom_release_resources(wacom1);
+ hid_hw_stop(wacom1->hdev);
+ wacom1->wacom_wac.has_mode_change = true;
+ wacom1->wacom_wac.is_direct_mode = is_direct;
+ }
+
+ if (shared->touch) {
+ wacom2 = hid_get_drvdata(shared->touch);
+ wacom_release_resources(wacom2);
+ hid_hw_stop(wacom2->hdev);
+ wacom2->wacom_wac.has_mode_change = true;
+ wacom2->wacom_wac.is_direct_mode = is_direct;
+ }
+
+ if (wacom1) {
+ error = wacom_parse_and_register(wacom1, false);
+ if (error)
+ return;
+ }
+
+ if (wacom2) {
+ error = wacom_parse_and_register(wacom2, false);
+ if (error)
+ return;
+ }
+
+ return;
+}
+
static int wacom_probe(struct hid_device *hdev,
const struct hid_device_id *id)
{
@@ -2534,6 +2583,7 @@
INIT_WORK(&wacom->wireless_work, wacom_wireless_work);
INIT_WORK(&wacom->battery_work, wacom_battery_work);
INIT_WORK(&wacom->remote_work, wacom_remote_work);
+ INIT_WORK(&wacom->mode_change_work, wacom_mode_change_work);
/* ask for the report descriptor to be loaded by HID */
error = hid_parse(hdev);
@@ -2576,6 +2626,7 @@
cancel_work_sync(&wacom->wireless_work);
cancel_work_sync(&wacom->battery_work);
cancel_work_sync(&wacom->remote_work);
+ cancel_work_sync(&wacom->mode_change_work);
if (hdev->bus == BUS_BLUETOOTH)
device_remove_file(&hdev->dev, &dev_attr_speed);
diff --git a/drivers/hid/wacom_wac.c b/drivers/hid/wacom_wac.c
index c68ac65..4b225fb 100644
--- a/drivers/hid/wacom_wac.c
+++ b/drivers/hid/wacom_wac.c
@@ -773,131 +773,6 @@
return 0;
}
-static int wacom_remote_irq(struct wacom_wac *wacom_wac, size_t len)
-{
- unsigned char *data = wacom_wac->data;
- struct input_dev *input;
- struct wacom *wacom = container_of(wacom_wac, struct wacom, wacom_wac);
- struct wacom_remote *remote = wacom->remote;
- int bat_charging, bat_percent, touch_ring_mode;
- __u32 serial;
- int i, index = -1;
- unsigned long flags;
-
- if (data[0] != WACOM_REPORT_REMOTE) {
- hid_dbg(wacom->hdev, "%s: received unknown report #%d",
- __func__, data[0]);
- return 0;
- }
-
- serial = data[3] + (data[4] << 8) + (data[5] << 16);
- wacom_wac->id[0] = PAD_DEVICE_ID;
-
- spin_lock_irqsave(&remote->remote_lock, flags);
-
- for (i = 0; i < WACOM_MAX_REMOTES; i++) {
- if (remote->remotes[i].serial == serial) {
- index = i;
- break;
- }
- }
-
- if (index < 0 || !remote->remotes[index].registered)
- goto out;
-
- input = remote->remotes[index].input;
-
- input_report_key(input, BTN_0, (data[9] & 0x01));
- input_report_key(input, BTN_1, (data[9] & 0x02));
- input_report_key(input, BTN_2, (data[9] & 0x04));
- input_report_key(input, BTN_3, (data[9] & 0x08));
- input_report_key(input, BTN_4, (data[9] & 0x10));
- input_report_key(input, BTN_5, (data[9] & 0x20));
- input_report_key(input, BTN_6, (data[9] & 0x40));
- input_report_key(input, BTN_7, (data[9] & 0x80));
-
- input_report_key(input, BTN_8, (data[10] & 0x01));
- input_report_key(input, BTN_9, (data[10] & 0x02));
- input_report_key(input, BTN_A, (data[10] & 0x04));
- input_report_key(input, BTN_B, (data[10] & 0x08));
- input_report_key(input, BTN_C, (data[10] & 0x10));
- input_report_key(input, BTN_X, (data[10] & 0x20));
- input_report_key(input, BTN_Y, (data[10] & 0x40));
- input_report_key(input, BTN_Z, (data[10] & 0x80));
-
- input_report_key(input, BTN_BASE, (data[11] & 0x01));
- input_report_key(input, BTN_BASE2, (data[11] & 0x02));
-
- if (data[12] & 0x80)
- input_report_abs(input, ABS_WHEEL, (data[12] & 0x7f));
- else
- input_report_abs(input, ABS_WHEEL, 0);
-
- bat_percent = data[7] & 0x7f;
- bat_charging = !!(data[7] & 0x80);
-
- if (data[9] | data[10] | (data[11] & 0x03) | data[12])
- input_report_abs(input, ABS_MISC, PAD_DEVICE_ID);
- else
- input_report_abs(input, ABS_MISC, 0);
-
- input_event(input, EV_MSC, MSC_SERIAL, serial);
-
- input_sync(input);
-
- /*Which mode select (LED light) is currently on?*/
- touch_ring_mode = (data[11] & 0xC0) >> 6;
-
- for (i = 0; i < WACOM_MAX_REMOTES; i++) {
- if (remote->remotes[i].serial == serial)
- wacom->led.groups[i].select = touch_ring_mode;
- }
-
- __wacom_notify_battery(&remote->remotes[index].battery, bat_percent,
- bat_charging, 1, bat_charging);
-
-out:
- spin_unlock_irqrestore(&remote->remote_lock, flags);
- return 0;
-}
-
-static void wacom_remote_status_irq(struct wacom_wac *wacom_wac, size_t len)
-{
- struct wacom *wacom = container_of(wacom_wac, struct wacom, wacom_wac);
- unsigned char *data = wacom_wac->data;
- struct wacom_remote *remote = wacom->remote;
- struct wacom_remote_data remote_data;
- unsigned long flags;
- int i, ret;
-
- if (data[0] != WACOM_REPORT_DEVICE_LIST)
- return;
-
- memset(&remote_data, 0, sizeof(struct wacom_remote_data));
-
- for (i = 0; i < WACOM_MAX_REMOTES; i++) {
- int j = i * 6;
- int serial = (data[j+6] << 16) + (data[j+5] << 8) + data[j+4];
- bool connected = data[j+2];
-
- remote_data.remote[i].serial = serial;
- remote_data.remote[i].connected = connected;
- }
-
- spin_lock_irqsave(&remote->remote_lock, flags);
-
- ret = kfifo_in(&remote->remote_fifo, &remote_data, sizeof(remote_data));
- if (ret != sizeof(remote_data)) {
- spin_unlock_irqrestore(&remote->remote_lock, flags);
- hid_err(wacom->hdev, "Can't queue Remote status event.\n");
- return;
- }
-
- spin_unlock_irqrestore(&remote->remote_lock, flags);
-
- wacom_schedule_work(wacom_wac, WACOM_WORKER_REMOTE);
-}
-
static inline bool report_touch_events(struct wacom_wac *wacom)
{
return (touch_arbitration ? !wacom->shared->stylus_in_proximity : 1);
@@ -1116,6 +991,131 @@
return 0;
}
+static int wacom_remote_irq(struct wacom_wac *wacom_wac, size_t len)
+{
+ unsigned char *data = wacom_wac->data;
+ struct input_dev *input;
+ struct wacom *wacom = container_of(wacom_wac, struct wacom, wacom_wac);
+ struct wacom_remote *remote = wacom->remote;
+ int bat_charging, bat_percent, touch_ring_mode;
+ __u32 serial;
+ int i, index = -1;
+ unsigned long flags;
+
+ if (data[0] != WACOM_REPORT_REMOTE) {
+ hid_dbg(wacom->hdev, "%s: received unknown report #%d",
+ __func__, data[0]);
+ return 0;
+ }
+
+ serial = data[3] + (data[4] << 8) + (data[5] << 16);
+ wacom_wac->id[0] = PAD_DEVICE_ID;
+
+ spin_lock_irqsave(&remote->remote_lock, flags);
+
+ for (i = 0; i < WACOM_MAX_REMOTES; i++) {
+ if (remote->remotes[i].serial == serial) {
+ index = i;
+ break;
+ }
+ }
+
+ if (index < 0 || !remote->remotes[index].registered)
+ goto out;
+
+ input = remote->remotes[index].input;
+
+ input_report_key(input, BTN_0, (data[9] & 0x01));
+ input_report_key(input, BTN_1, (data[9] & 0x02));
+ input_report_key(input, BTN_2, (data[9] & 0x04));
+ input_report_key(input, BTN_3, (data[9] & 0x08));
+ input_report_key(input, BTN_4, (data[9] & 0x10));
+ input_report_key(input, BTN_5, (data[9] & 0x20));
+ input_report_key(input, BTN_6, (data[9] & 0x40));
+ input_report_key(input, BTN_7, (data[9] & 0x80));
+
+ input_report_key(input, BTN_8, (data[10] & 0x01));
+ input_report_key(input, BTN_9, (data[10] & 0x02));
+ input_report_key(input, BTN_A, (data[10] & 0x04));
+ input_report_key(input, BTN_B, (data[10] & 0x08));
+ input_report_key(input, BTN_C, (data[10] & 0x10));
+ input_report_key(input, BTN_X, (data[10] & 0x20));
+ input_report_key(input, BTN_Y, (data[10] & 0x40));
+ input_report_key(input, BTN_Z, (data[10] & 0x80));
+
+ input_report_key(input, BTN_BASE, (data[11] & 0x01));
+ input_report_key(input, BTN_BASE2, (data[11] & 0x02));
+
+ if (data[12] & 0x80)
+ input_report_abs(input, ABS_WHEEL, (data[12] & 0x7f));
+ else
+ input_report_abs(input, ABS_WHEEL, 0);
+
+ bat_percent = data[7] & 0x7f;
+ bat_charging = !!(data[7] & 0x80);
+
+ if (data[9] | data[10] | (data[11] & 0x03) | data[12])
+ input_report_abs(input, ABS_MISC, PAD_DEVICE_ID);
+ else
+ input_report_abs(input, ABS_MISC, 0);
+
+ input_event(input, EV_MSC, MSC_SERIAL, serial);
+
+ input_sync(input);
+
+ /*Which mode select (LED light) is currently on?*/
+ touch_ring_mode = (data[11] & 0xC0) >> 6;
+
+ for (i = 0; i < WACOM_MAX_REMOTES; i++) {
+ if (remote->remotes[i].serial == serial)
+ wacom->led.groups[i].select = touch_ring_mode;
+ }
+
+ __wacom_notify_battery(&remote->remotes[index].battery, bat_percent,
+ bat_charging, 1, bat_charging);
+
+out:
+ spin_unlock_irqrestore(&remote->remote_lock, flags);
+ return 0;
+}
+
+static void wacom_remote_status_irq(struct wacom_wac *wacom_wac, size_t len)
+{
+ struct wacom *wacom = container_of(wacom_wac, struct wacom, wacom_wac);
+ unsigned char *data = wacom_wac->data;
+ struct wacom_remote *remote = wacom->remote;
+ struct wacom_remote_data remote_data;
+ unsigned long flags;
+ int i, ret;
+
+ if (data[0] != WACOM_REPORT_DEVICE_LIST)
+ return;
+
+ memset(&remote_data, 0, sizeof(struct wacom_remote_data));
+
+ for (i = 0; i < WACOM_MAX_REMOTES; i++) {
+ int j = i * 6;
+ int serial = (data[j+6] << 16) + (data[j+5] << 8) + data[j+4];
+ bool connected = data[j+2];
+
+ remote_data.remote[i].serial = serial;
+ remote_data.remote[i].connected = connected;
+ }
+
+ spin_lock_irqsave(&remote->remote_lock, flags);
+
+ ret = kfifo_in(&remote->remote_fifo, &remote_data, sizeof(remote_data));
+ if (ret != sizeof(remote_data)) {
+ spin_unlock_irqrestore(&remote->remote_lock, flags);
+ hid_err(wacom->hdev, "Can't queue Remote status event.\n");
+ return;
+ }
+
+ spin_unlock_irqrestore(&remote->remote_lock, flags);
+
+ wacom_schedule_work(wacom_wac, WACOM_WORKER_REMOTE);
+}
+
static int int_dist(int x1, int y1, int x2, int y2)
{
int x = x2 - x1;
@@ -1739,6 +1739,7 @@
features->device_type |= WACOM_DEVICETYPE_PAD;
break;
case WACOM_HID_WD_TOUCHONOFF:
+ case WACOM_HID_WD_MUTE_DEVICE:
/*
* This usage, which is used to mute touch events, comes
* from the pad packet, but is reported on the touch
@@ -1768,6 +1769,26 @@
wacom_map_usage(input, usage, field, EV_ABS, ABS_WHEEL, 0);
features->device_type |= WACOM_DEVICETYPE_PAD;
break;
+ case WACOM_HID_WD_BUTTONCONFIG:
+ wacom_map_usage(input, usage, field, EV_KEY, KEY_BUTTONCONFIG, 0);
+ features->device_type |= WACOM_DEVICETYPE_PAD;
+ break;
+ case WACOM_HID_WD_ONSCREEN_KEYBOARD:
+ wacom_map_usage(input, usage, field, EV_KEY, KEY_ONSCREEN_KEYBOARD, 0);
+ features->device_type |= WACOM_DEVICETYPE_PAD;
+ break;
+ case WACOM_HID_WD_CONTROLPANEL:
+ wacom_map_usage(input, usage, field, EV_KEY, KEY_CONTROLPANEL, 0);
+ features->device_type |= WACOM_DEVICETYPE_PAD;
+ break;
+ case WACOM_HID_WD_MODE_CHANGE:
+ /* do not overwrite previous data */
+ if (!wacom_wac->has_mode_change) {
+ wacom_wac->has_mode_change = true;
+ wacom_wac->is_direct_mode = true;
+ }
+ features->device_type |= WACOM_DEVICETYPE_PAD;
+ break;
}
switch (equivalent_usage & 0xfffffff0) {
@@ -1811,12 +1832,13 @@
struct wacom_features *features = &wacom_wac->features;
unsigned equivalent_usage = wacom_equivalent_usage(usage->hid);
int i;
+ bool is_touch_on = value;
/*
* Avoid reporting this event and setting inrange_state if this usage
* hasn't been mapped.
*/
- if (!usage->type)
+ if (!usage->type && equivalent_usage != WACOM_HID_WD_MODE_CHANGE)
return;
if (wacom_equivalent_usage(field->physical) == HID_DG_TABLETFUNCTIONKEY) {
@@ -1830,14 +1852,28 @@
input_event(input, usage->type, usage->code, 0);
break;
+ case WACOM_HID_WD_MUTE_DEVICE:
+ if (wacom_wac->shared->touch_input && value) {
+ wacom_wac->shared->is_touch_on = !wacom_wac->shared->is_touch_on;
+ is_touch_on = wacom_wac->shared->is_touch_on;
+ }
+
+ /* fall through*/
case WACOM_HID_WD_TOUCHONOFF:
if (wacom_wac->shared->touch_input) {
input_report_switch(wacom_wac->shared->touch_input,
- SW_MUTE_DEVICE, !value);
+ SW_MUTE_DEVICE, !is_touch_on);
input_sync(wacom_wac->shared->touch_input);
}
break;
+ case WACOM_HID_WD_MODE_CHANGE:
+ if (wacom_wac->is_direct_mode != value) {
+ wacom_wac->is_direct_mode = value;
+ wacom_schedule_work(&wacom->wacom_wac, WACOM_WORKER_MODE_CHANGE);
+ }
+ break;
+
case WACOM_HID_WD_BUTTONCENTER:
for (i = 0; i < wacom->led.count; i++)
wacom_update_led(wacom, features->numbered_buttons,
@@ -1845,6 +1881,8 @@
/* fall through*/
default:
input_event(input, usage->type, usage->code, value);
+ if (value)
+ wacom_wac->hid_data.pad_input_event_flag = true;
break;
}
}
@@ -1885,9 +1923,12 @@
bool active = wacom_wac->hid_data.inrange_state != 0;
/* report prox for expresskey events */
- if (wacom_equivalent_usage(report->field[0]->physical) == HID_DG_TABLETFUNCTIONKEY) {
+ if ((wacom_equivalent_usage(report->field[0]->physical) == HID_DG_TABLETFUNCTIONKEY) &&
+ wacom_wac->hid_data.pad_input_event_flag) {
input_event(input, EV_ABS, ABS_MISC, active ? PAD_DEVICE_ID : 0);
input_sync(input);
+ if (!active)
+ wacom_wac->hid_data.pad_input_event_flag = false;
}
}
@@ -2197,6 +2238,13 @@
bool prox = hid_data->tipswitch &&
report_touch_events(wacom_wac);
+ if (wacom_wac->shared->has_mute_touch_switch &&
+ !wacom_wac->shared->is_touch_on) {
+ if (!wacom_wac->shared->touch_down)
+ return;
+ prox = 0;
+ }
+
wacom_wac->hid_data.num_received++;
if (wacom_wac->hid_data.num_received > wacom_wac->hid_data.num_expected)
return;
@@ -4127,7 +4175,7 @@
BAMBOO_PEN, WACOM_INTUOS_RES, WACOM_INTUOS_RES };
static const struct wacom_features wacom_features_0x301 =
{ "Wacom Bamboo One M", 21648, 13530, 1023, 31,
- BAMBOO_PT, WACOM_INTUOS_RES, WACOM_INTUOS_RES };
+ BAMBOO_PEN, WACOM_INTUOS_RES, WACOM_INTUOS_RES };
static const struct wacom_features wacom_features_0x302 =
{ "Wacom Intuos PT S", 15200, 9500, 1023, 31,
INTUOSHT, WACOM_INTUOS_RES, WACOM_INTUOS_RES, .touch_max = 16,
diff --git a/drivers/hid/wacom_wac.h b/drivers/hid/wacom_wac.h
index 857ccee..570d295 100644
--- a/drivers/hid/wacom_wac.h
+++ b/drivers/hid/wacom_wac.h
@@ -120,6 +120,11 @@
#define WACOM_HID_WD_BATTERY_LEVEL (WACOM_HID_UP_WACOMDIGITIZER | 0x043b)
#define WACOM_HID_WD_EXPRESSKEY00 (WACOM_HID_UP_WACOMDIGITIZER | 0x0910)
#define WACOM_HID_WD_EXPRESSKEYCAP00 (WACOM_HID_UP_WACOMDIGITIZER | 0x0950)
+#define WACOM_HID_WD_MODE_CHANGE (WACOM_HID_UP_WACOMDIGITIZER | 0x0980)
+#define WACOM_HID_WD_MUTE_DEVICE (WACOM_HID_UP_WACOMDIGITIZER | 0x0981)
+#define WACOM_HID_WD_CONTROLPANEL (WACOM_HID_UP_WACOMDIGITIZER | 0x0982)
+#define WACOM_HID_WD_ONSCREEN_KEYBOARD (WACOM_HID_UP_WACOMDIGITIZER | 0x0983)
+#define WACOM_HID_WD_BUTTONCONFIG (WACOM_HID_UP_WACOMDIGITIZER | 0x0986)
#define WACOM_HID_WD_BUTTONHOME (WACOM_HID_UP_WACOMDIGITIZER | 0x0990)
#define WACOM_HID_WD_BUTTONUP (WACOM_HID_UP_WACOMDIGITIZER | 0x0991)
#define WACOM_HID_WD_BUTTONDOWN (WACOM_HID_UP_WACOMDIGITIZER | 0x0992)
@@ -270,6 +275,7 @@
struct hid_device *pen;
struct hid_device *touch;
bool has_mute_touch_switch;
+ bool is_touch_on;
};
struct hid_data {
@@ -295,6 +301,7 @@
int bat_charging;
int bat_connected;
int ps_connected;
+ bool pad_input_event_flag;
};
struct wacom_remote_data {
@@ -327,6 +334,9 @@
int mode_value;
struct hid_data hid_data;
bool has_mute_touch_switch;
+ bool has_mode_change;
+ bool is_direct_mode;
+
};
#endif
diff --git a/drivers/hsi/clients/ssi_protocol.c b/drivers/hsi/clients/ssi_protocol.c
index 7ef8196..26b0510 100644
--- a/drivers/hsi/clients/ssi_protocol.c
+++ b/drivers/hsi/clients/ssi_protocol.c
@@ -980,7 +980,7 @@
goto drop;
/* Pad to 32-bits - FIXME: Revisit*/
if ((skb->len & 3) && skb_pad(skb, 4 - (skb->len & 3)))
- goto drop;
+ goto inc_dropped;
/*
* Modem sends Phonet messages over SSI with its own endianess...
@@ -1032,8 +1032,9 @@
drop2:
hsi_free_msg(msg);
drop:
- dev->stats.tx_dropped++;
dev_kfree_skb(skb);
+inc_dropped:
+ dev->stats.tx_dropped++;
return 0;
}
diff --git a/drivers/hv/Kconfig b/drivers/hv/Kconfig
index 0403b51..c29cd53 100644
--- a/drivers/hv/Kconfig
+++ b/drivers/hv/Kconfig
@@ -7,6 +7,9 @@
Select this option to run Linux as a Hyper-V client operating
system.
+config HYPERV_TSCPAGE
+ def_bool HYPERV && X86_64
+
config HYPERV_UTILS
tristate "Microsoft Hyper-V Utilities driver"
depends on HYPERV && CONNECTOR && NLS
diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c
index 87799e8..c3f1a9e 100644
--- a/drivers/hv/ring_buffer.c
+++ b/drivers/hv/ring_buffer.c
@@ -32,6 +32,8 @@
#include "hyperv_vmbus.h"
+#define VMBUS_PKT_TRAILER 8
+
/*
* When we write to the ring buffer, check if the host needs to
* be signaled. Here is the details of this protocol:
@@ -336,6 +338,12 @@
return 0;
}
+static inline void
+init_cached_read_index(struct hv_ring_buffer_info *rbi)
+{
+ rbi->cached_read_index = rbi->ring_buffer->read_index;
+}
+
int hv_ringbuffer_read(struct vmbus_channel *channel,
void *buffer, u32 buflen, u32 *buffer_actual_len,
u64 *requestid, bool raw)
@@ -366,7 +374,8 @@
return ret;
}
- init_cached_read_index(channel);
+ init_cached_read_index(inring_info);
+
next_read_location = hv_get_next_read_location(inring_info);
next_read_location = hv_copyfrom_ringbuffer(inring_info, &desc,
sizeof(desc),
@@ -410,3 +419,86 @@
return ret;
}
+
+/*
+ * Determine number of bytes available in ring buffer after
+ * the current iterator (priv_read_index) location.
+ *
+ * This is similar to hv_get_bytes_to_read but with private
+ * read index instead.
+ */
+static u32 hv_pkt_iter_avail(const struct hv_ring_buffer_info *rbi)
+{
+ u32 priv_read_loc = rbi->priv_read_index;
+ u32 write_loc = READ_ONCE(rbi->ring_buffer->write_index);
+
+ if (write_loc >= priv_read_loc)
+ return write_loc - priv_read_loc;
+ else
+ return (rbi->ring_datasize - priv_read_loc) + write_loc;
+}
+
+/*
+ * Get first vmbus packet from ring buffer after read_index
+ *
+ * If ring buffer is empty, returns NULL and no other action needed.
+ */
+struct vmpacket_descriptor *hv_pkt_iter_first(struct vmbus_channel *channel)
+{
+ struct hv_ring_buffer_info *rbi = &channel->inbound;
+
+ /* set state for later hv_signal_on_read() */
+ init_cached_read_index(rbi);
+
+ if (hv_pkt_iter_avail(rbi) < sizeof(struct vmpacket_descriptor))
+ return NULL;
+
+ return hv_get_ring_buffer(rbi) + rbi->priv_read_index;
+}
+EXPORT_SYMBOL_GPL(hv_pkt_iter_first);
+
+/*
+ * Get next vmbus packet from ring buffer.
+ *
+ * Advances the current location (priv_read_index) and checks for more
+ * data. If the end of the ring buffer is reached, then return NULL.
+ */
+struct vmpacket_descriptor *
+__hv_pkt_iter_next(struct vmbus_channel *channel,
+ const struct vmpacket_descriptor *desc)
+{
+ struct hv_ring_buffer_info *rbi = &channel->inbound;
+ u32 packetlen = desc->len8 << 3;
+ u32 dsize = rbi->ring_datasize;
+
+ /* bump offset to next potential packet */
+ rbi->priv_read_index += packetlen + VMBUS_PKT_TRAILER;
+ if (rbi->priv_read_index >= dsize)
+ rbi->priv_read_index -= dsize;
+
+ /* more data? */
+ if (hv_pkt_iter_avail(rbi) < sizeof(struct vmpacket_descriptor))
+ return NULL;
+ else
+ return hv_get_ring_buffer(rbi) + rbi->priv_read_index;
+}
+EXPORT_SYMBOL_GPL(__hv_pkt_iter_next);
+
+/*
+ * Update host ring buffer after iterating over packets.
+ */
+void hv_pkt_iter_close(struct vmbus_channel *channel)
+{
+ struct hv_ring_buffer_info *rbi = &channel->inbound;
+
+ /*
+ * Make sure all reads are done before we update the read index since
+ * the writer may start writing to the read area once the read index
+ * is updated.
+ */
+ virt_rmb();
+ rbi->ring_buffer->read_index = rbi->priv_read_index;
+
+ hv_signal_on_read(channel);
+}
+EXPORT_SYMBOL_GPL(hv_pkt_iter_close);
diff --git a/drivers/hwmon/Kconfig b/drivers/hwmon/Kconfig
index 0649d53f3..22d5eaf 100644
--- a/drivers/hwmon/Kconfig
+++ b/drivers/hwmon/Kconfig
@@ -341,6 +341,15 @@
This driver can also be built as a module. If so, the module
will be called asb100.
+config SENSORS_ASPEED
+ tristate "ASPEED AST2400/AST2500 PWM and Fan tach driver"
+ help
+ This driver provides support for ASPEED AST2400/AST2500 PWM
+ and Fan Tacho controllers.
+
+ This driver can also be built as a module. If so, the module
+ will be called aspeed_pwm_tacho.
+
config SENSORS_ATXP1
tristate "Attansic ATXP1 VID controller"
depends on I2C
@@ -1643,16 +1652,6 @@
This driver can also be built as a module. If so, the module
will be called tmp421.
-config SENSORS_TWL4030_MADC
- tristate "Texas Instruments TWL4030 MADC Hwmon"
- depends on TWL4030_MADC
- help
- If you say yes here you get hwmon support for triton
- TWL4030-MADC.
-
- This driver can also be built as a module. If so it will be called
- twl4030-madc-hwmon.
-
config SENSORS_VEXPRESS
tristate "Versatile Express"
depends on VEXPRESS_CONFIG
diff --git a/drivers/hwmon/Makefile b/drivers/hwmon/Makefile
index 5509edf..d4641a9 100644
--- a/drivers/hwmon/Makefile
+++ b/drivers/hwmon/Makefile
@@ -46,6 +46,7 @@
obj-$(CONFIG_SENSORS_APPLESMC) += applesmc.o
obj-$(CONFIG_SENSORS_ARM_SCPI) += scpi-hwmon.o
obj-$(CONFIG_SENSORS_ASC7621) += asc7621.o
+obj-$(CONFIG_SENSORS_ASPEED) += aspeed-pwm-tacho.o
obj-$(CONFIG_SENSORS_ATXP1) += atxp1.o
obj-$(CONFIG_SENSORS_CORETEMP) += coretemp.o
obj-$(CONFIG_SENSORS_DA9052_ADC)+= da9052-hwmon.o
@@ -157,7 +158,6 @@
obj-$(CONFIG_SENSORS_TMP108) += tmp108.o
obj-$(CONFIG_SENSORS_TMP401) += tmp401.o
obj-$(CONFIG_SENSORS_TMP421) += tmp421.o
-obj-$(CONFIG_SENSORS_TWL4030_MADC)+= twl4030-madc-hwmon.o
obj-$(CONFIG_SENSORS_VEXPRESS) += vexpress-hwmon.o
obj-$(CONFIG_SENSORS_VIA_CPUTEMP)+= via-cputemp.o
obj-$(CONFIG_SENSORS_VIA686A) += via686a.o
diff --git a/drivers/hwmon/ad7414.c b/drivers/hwmon/ad7414.c
index 763490a..cec227f 100644
--- a/drivers/hwmon/ad7414.c
+++ b/drivers/hwmon/ad7414.c
@@ -217,9 +217,16 @@
};
MODULE_DEVICE_TABLE(i2c, ad7414_id);
+static const struct of_device_id ad7414_of_match[] = {
+ { .compatible = "ad,ad7414" },
+ { },
+};
+MODULE_DEVICE_TABLE(of, ad7414_of_match);
+
static struct i2c_driver ad7414_driver = {
.driver = {
.name = "ad7414",
+ .of_match_table = of_match_ptr(ad7414_of_match),
},
.probe = ad7414_probe,
.id_table = ad7414_id,
diff --git a/drivers/hwmon/adc128d818.c b/drivers/hwmon/adc128d818.c
index bbe3a5c..a557b46 100644
--- a/drivers/hwmon/adc128d818.c
+++ b/drivers/hwmon/adc128d818.c
@@ -546,10 +546,17 @@
};
MODULE_DEVICE_TABLE(i2c, adc128_id);
+static const struct of_device_id adc128_of_match[] = {
+ { .compatible = "ti,adc128d818" },
+ { },
+};
+MODULE_DEVICE_TABLE(of, adc128_of_match);
+
static struct i2c_driver adc128_driver = {
.class = I2C_CLASS_HWMON,
.driver = {
.name = "adc128d818",
+ .of_match_table = of_match_ptr(adc128_of_match),
},
.probe = adc128_probe,
.remove = adc128_remove,
diff --git a/drivers/hwmon/ads1015.c b/drivers/hwmon/ads1015.c
index 2b3105c..5140c27d 100644
--- a/drivers/hwmon/ads1015.c
+++ b/drivers/hwmon/ads1015.c
@@ -31,6 +31,7 @@
#include <linux/hwmon-sysfs.h>
#include <linux/err.h>
#include <linux/mutex.h>
+#include <linux/of_device.h>
#include <linux/of.h>
#include <linux/i2c/ads1015.h>
@@ -268,7 +269,12 @@
GFP_KERNEL);
if (!data)
return -ENOMEM;
- data->id = id->driver_data;
+
+ if (client->dev.of_node)
+ data->id = (enum ads1015_chips)
+ of_device_get_match_data(&client->dev);
+ else
+ data->id = id->driver_data;
i2c_set_clientdata(client, data);
mutex_init(&data->update_lock);
@@ -303,9 +309,23 @@
};
MODULE_DEVICE_TABLE(i2c, ads1015_id);
+static const struct of_device_id ads1015_of_match[] = {
+ {
+ .compatible = "ti,ads1015",
+ .data = (void *)ads1015
+ },
+ {
+ .compatible = "ti,ads1115",
+ .data = (void *)ads1115
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, ads1015_of_match);
+
static struct i2c_driver ads1015_driver = {
.driver = {
.name = "ads1015",
+ .of_match_table = of_match_ptr(ads1015_of_match),
},
.probe = ads1015_probe,
.remove = ads1015_remove,
diff --git a/drivers/hwmon/ads7828.c b/drivers/hwmon/ads7828.c
index ee396ff..898607b 100644
--- a/drivers/hwmon/ads7828.c
+++ b/drivers/hwmon/ads7828.c
@@ -31,9 +31,11 @@
#include <linux/i2c.h>
#include <linux/init.h>
#include <linux/module.h>
+#include <linux/of_device.h>
#include <linux/platform_data/ads7828.h>
#include <linux/regmap.h>
#include <linux/slab.h>
+#include <linux/regulator/consumer.h>
/* The ADS7828 registers */
#define ADS7828_CMD_SD_SE 0x80 /* Single ended inputs */
@@ -118,9 +120,12 @@
struct ads7828_data *data;
struct device *hwmon_dev;
unsigned int vref_mv = ADS7828_INT_VREF_MV;
+ unsigned int vref_uv;
bool diff_input = false;
bool ext_vref = false;
unsigned int regval;
+ enum ads7828_chips chip;
+ struct regulator *reg;
data = devm_kzalloc(dev, sizeof(struct ads7828_data), GFP_KERNEL);
if (!data)
@@ -131,14 +136,32 @@
ext_vref = pdata->ext_vref;
if (ext_vref && pdata->vref_mv)
vref_mv = pdata->vref_mv;
+ } else if (dev->of_node) {
+ diff_input = of_property_read_bool(dev->of_node,
+ "ti,differential-input");
+ reg = devm_regulator_get_optional(dev, "vref");
+ if (!IS_ERR(reg)) {
+ vref_uv = regulator_get_voltage(reg);
+ vref_mv = DIV_ROUND_CLOSEST(vref_uv, 1000);
+ if (vref_mv < ADS7828_EXT_VREF_MV_MIN ||
+ vref_mv > ADS7828_EXT_VREF_MV_MAX)
+ return -EINVAL;
+ ext_vref = true;
+ }
}
+ if (client->dev.of_node)
+ chip = (enum ads7828_chips)
+ of_device_get_match_data(&client->dev);
+ else
+ chip = id->driver_data;
+
/* Bound Vref with min/max values */
vref_mv = clamp_val(vref_mv, ADS7828_EXT_VREF_MV_MIN,
ADS7828_EXT_VREF_MV_MAX);
/* ADS7828 uses 12-bit samples, while ADS7830 is 8-bit */
- if (id->driver_data == ads7828) {
+ if (chip == ads7828) {
data->lsb_resol = DIV_ROUND_CLOSEST(vref_mv * 1000, 4096);
data->regmap = devm_regmap_init_i2c(client,
&ads2828_regmap_config);
@@ -177,9 +200,23 @@
};
MODULE_DEVICE_TABLE(i2c, ads7828_device_ids);
+static const struct of_device_id ads7828_of_match[] = {
+ {
+ .compatible = "ti,ads7828",
+ .data = (void *)ads7828
+ },
+ {
+ .compatible = "ti,ads7830",
+ .data = (void *)ads7830
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, ads7828_of_match);
+
static struct i2c_driver ads7828_driver = {
.driver = {
.name = "ads7828",
+ .of_match_table = of_match_ptr(ads7828_of_match),
},
.id_table = ads7828_device_ids,
diff --git a/drivers/hwmon/adt7475.c b/drivers/hwmon/adt7475.c
index c646670..c803e3c 100644
--- a/drivers/hwmon/adt7475.c
+++ b/drivers/hwmon/adt7475.c
@@ -13,6 +13,7 @@
*/
#include <linux/module.h>
+#include <linux/of_device.h>
#include <linux/init.h>
#include <linux/slab.h>
#include <linux/i2c.h>
@@ -58,6 +59,8 @@
#define REG_VENDID 0x3E
#define REG_DEVID2 0x3F
+#define REG_CONFIG1 0x40
+
#define REG_STATUS1 0x41
#define REG_STATUS2 0x42
@@ -161,6 +164,27 @@
};
MODULE_DEVICE_TABLE(i2c, adt7475_id);
+static const struct of_device_id adt7475_of_match[] = {
+ {
+ .compatible = "adi,adt7473",
+ .data = (void *)adt7473
+ },
+ {
+ .compatible = "adi,adt7475",
+ .data = (void *)adt7475
+ },
+ {
+ .compatible = "adi,adt7476",
+ .data = (void *)adt7476
+ },
+ {
+ .compatible = "adi,adt7490",
+ .data = (void *)adt7490
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, adt7475_of_match);
+
struct adt7475_data {
struct device *hwmon_dev;
struct mutex lock;
@@ -1250,6 +1274,7 @@
static int adt7475_probe(struct i2c_client *client,
const struct i2c_device_id *id)
{
+ enum chips chip;
static const char * const names[] = {
[adt7473] = "ADT7473",
[adt7475] = "ADT7475",
@@ -1268,8 +1293,13 @@
mutex_init(&data->lock);
i2c_set_clientdata(client, data);
+ if (client->dev.of_node)
+ chip = (enum chips)of_device_get_match_data(&client->dev);
+ else
+ chip = id->driver_data;
+
/* Initialize device-specific values */
- switch (id->driver_data) {
+ switch (chip) {
case adt7476:
data->has_voltage = 0x0e; /* in1 to in3 */
revision = adt7475_read(REG_DEVID2) & 0x07;
@@ -1343,6 +1373,17 @@
for (i = 0; i < ADT7475_PWM_COUNT; i++)
adt7475_read_pwm(client, i);
+ /* Start monitoring */
+ switch (chip) {
+ case adt7475:
+ case adt7476:
+ i2c_smbus_write_byte_data(client, REG_CONFIG1,
+ adt7475_read(REG_CONFIG1) | 0x01);
+ break;
+ default:
+ break;
+ }
+
ret = sysfs_create_group(&client->dev.kobj, &adt7475_attr_group);
if (ret)
return ret;
@@ -1428,6 +1469,7 @@
.class = I2C_CLASS_HWMON,
.driver = {
.name = "adt7475",
+ .of_match_table = of_match_ptr(adt7475_of_match),
},
.probe = adt7475_probe,
.remove = adt7475_remove,
diff --git a/drivers/hwmon/aspeed-pwm-tacho.c b/drivers/hwmon/aspeed-pwm-tacho.c
new file mode 100644
index 0000000..48403a2
--- /dev/null
+++ b/drivers/hwmon/aspeed-pwm-tacho.c
@@ -0,0 +1,835 @@
+/*
+ * Copyright (c) 2016 Google, Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 or later as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/clk.h>
+#include <linux/gpio/consumer.h>
+#include <linux/delay.h>
+#include <linux/hwmon.h>
+#include <linux/hwmon-sysfs.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/of_device.h>
+#include <linux/platform_device.h>
+#include <linux/sysfs.h>
+#include <linux/regmap.h>
+
+/* ASPEED PWM & FAN Tach Register Definition */
+#define ASPEED_PTCR_CTRL 0x00
+#define ASPEED_PTCR_CLK_CTRL 0x04
+#define ASPEED_PTCR_DUTY0_CTRL 0x08
+#define ASPEED_PTCR_DUTY1_CTRL 0x0c
+#define ASPEED_PTCR_TYPEM_CTRL 0x10
+#define ASPEED_PTCR_TYPEM_CTRL1 0x14
+#define ASPEED_PTCR_TYPEN_CTRL 0x18
+#define ASPEED_PTCR_TYPEN_CTRL1 0x1c
+#define ASPEED_PTCR_TACH_SOURCE 0x20
+#define ASPEED_PTCR_TRIGGER 0x28
+#define ASPEED_PTCR_RESULT 0x2c
+#define ASPEED_PTCR_INTR_CTRL 0x30
+#define ASPEED_PTCR_INTR_STS 0x34
+#define ASPEED_PTCR_TYPEM_LIMIT 0x38
+#define ASPEED_PTCR_TYPEN_LIMIT 0x3C
+#define ASPEED_PTCR_CTRL_EXT 0x40
+#define ASPEED_PTCR_CLK_CTRL_EXT 0x44
+#define ASPEED_PTCR_DUTY2_CTRL 0x48
+#define ASPEED_PTCR_DUTY3_CTRL 0x4c
+#define ASPEED_PTCR_TYPEO_CTRL 0x50
+#define ASPEED_PTCR_TYPEO_CTRL1 0x54
+#define ASPEED_PTCR_TACH_SOURCE_EXT 0x60
+#define ASPEED_PTCR_TYPEO_LIMIT 0x78
+
+/* ASPEED_PTCR_CTRL : 0x00 - General Control Register */
+#define ASPEED_PTCR_CTRL_SET_PWMD_TYPE_PART1 15
+#define ASPEED_PTCR_CTRL_SET_PWMD_TYPE_PART2 6
+#define ASPEED_PTCR_CTRL_SET_PWMD_TYPE_MASK (BIT(7) | BIT(15))
+
+#define ASPEED_PTCR_CTRL_SET_PWMC_TYPE_PART1 14
+#define ASPEED_PTCR_CTRL_SET_PWMC_TYPE_PART2 5
+#define ASPEED_PTCR_CTRL_SET_PWMC_TYPE_MASK (BIT(6) | BIT(14))
+
+#define ASPEED_PTCR_CTRL_SET_PWMB_TYPE_PART1 13
+#define ASPEED_PTCR_CTRL_SET_PWMB_TYPE_PART2 4
+#define ASPEED_PTCR_CTRL_SET_PWMB_TYPE_MASK (BIT(5) | BIT(13))
+
+#define ASPEED_PTCR_CTRL_SET_PWMA_TYPE_PART1 12
+#define ASPEED_PTCR_CTRL_SET_PWMA_TYPE_PART2 3
+#define ASPEED_PTCR_CTRL_SET_PWMA_TYPE_MASK (BIT(4) | BIT(12))
+
+#define ASPEED_PTCR_CTRL_FAN_NUM_EN(x) BIT(16 + (x))
+
+#define ASPEED_PTCR_CTRL_PWMD_EN BIT(11)
+#define ASPEED_PTCR_CTRL_PWMC_EN BIT(10)
+#define ASPEED_PTCR_CTRL_PWMB_EN BIT(9)
+#define ASPEED_PTCR_CTRL_PWMA_EN BIT(8)
+
+#define ASPEED_PTCR_CTRL_CLK_SRC BIT(1)
+#define ASPEED_PTCR_CTRL_CLK_EN BIT(0)
+
+/* ASPEED_PTCR_CLK_CTRL : 0x04 - Clock Control Register */
+/* TYPE N */
+#define ASPEED_PTCR_CLK_CTRL_TYPEN_MASK GENMASK(31, 16)
+#define ASPEED_PTCR_CLK_CTRL_TYPEN_UNIT 24
+#define ASPEED_PTCR_CLK_CTRL_TYPEN_H 20
+#define ASPEED_PTCR_CLK_CTRL_TYPEN_L 16
+/* TYPE M */
+#define ASPEED_PTCR_CLK_CTRL_TYPEM_MASK GENMASK(15, 0)
+#define ASPEED_PTCR_CLK_CTRL_TYPEM_UNIT 8
+#define ASPEED_PTCR_CLK_CTRL_TYPEM_H 4
+#define ASPEED_PTCR_CLK_CTRL_TYPEM_L 0
+
+/*
+ * ASPEED_PTCR_DUTY_CTRL/1/2/3 : 0x08/0x0C/0x48/0x4C - PWM-FAN duty control
+ * 0/1/2/3 register
+ */
+#define DUTY_CTRL_PWM2_FALL_POINT 24
+#define DUTY_CTRL_PWM2_RISE_POINT 16
+#define DUTY_CTRL_PWM2_RISE_FALL_MASK GENMASK(31, 16)
+#define DUTY_CTRL_PWM1_FALL_POINT 8
+#define DUTY_CTRL_PWM1_RISE_POINT 0
+#define DUTY_CTRL_PWM1_RISE_FALL_MASK GENMASK(15, 0)
+
+/* ASPEED_PTCR_TYPEM_CTRL : 0x10/0x18/0x50 - Type M/N/O Ctrl 0 Register */
+#define TYPE_CTRL_FAN_MASK (GENMASK(5, 1) | GENMASK(31, 16))
+#define TYPE_CTRL_FAN1_MASK GENMASK(31, 0)
+#define TYPE_CTRL_FAN_PERIOD 16
+#define TYPE_CTRL_FAN_MODE 4
+#define TYPE_CTRL_FAN_DIVISION 1
+#define TYPE_CTRL_FAN_TYPE_EN 1
+
+/* ASPEED_PTCR_TACH_SOURCE : 0x20/0x60 - Tach Source Register */
+/* bit [0,1] at 0x20, bit [2] at 0x60 */
+#define TACH_PWM_SOURCE_BIT01(x) ((x) * 2)
+#define TACH_PWM_SOURCE_BIT2(x) ((x) * 2)
+#define TACH_PWM_SOURCE_MASK_BIT01(x) (0x3 << ((x) * 2))
+#define TACH_PWM_SOURCE_MASK_BIT2(x) BIT((x) * 2)
+
+/* ASPEED_PTCR_RESULT : 0x2c - Result Register */
+#define RESULT_STATUS_MASK BIT(31)
+#define RESULT_VALUE_MASK 0xfffff
+
+/* ASPEED_PTCR_CTRL_EXT : 0x40 - General Control Extension #1 Register */
+#define ASPEED_PTCR_CTRL_SET_PWMH_TYPE_PART1 15
+#define ASPEED_PTCR_CTRL_SET_PWMH_TYPE_PART2 6
+#define ASPEED_PTCR_CTRL_SET_PWMH_TYPE_MASK (BIT(7) | BIT(15))
+
+#define ASPEED_PTCR_CTRL_SET_PWMG_TYPE_PART1 14
+#define ASPEED_PTCR_CTRL_SET_PWMG_TYPE_PART2 5
+#define ASPEED_PTCR_CTRL_SET_PWMG_TYPE_MASK (BIT(6) | BIT(14))
+
+#define ASPEED_PTCR_CTRL_SET_PWMF_TYPE_PART1 13
+#define ASPEED_PTCR_CTRL_SET_PWMF_TYPE_PART2 4
+#define ASPEED_PTCR_CTRL_SET_PWMF_TYPE_MASK (BIT(5) | BIT(13))
+
+#define ASPEED_PTCR_CTRL_SET_PWME_TYPE_PART1 12
+#define ASPEED_PTCR_CTRL_SET_PWME_TYPE_PART2 3
+#define ASPEED_PTCR_CTRL_SET_PWME_TYPE_MASK (BIT(4) | BIT(12))
+
+#define ASPEED_PTCR_CTRL_PWMH_EN BIT(11)
+#define ASPEED_PTCR_CTRL_PWMG_EN BIT(10)
+#define ASPEED_PTCR_CTRL_PWMF_EN BIT(9)
+#define ASPEED_PTCR_CTRL_PWME_EN BIT(8)
+
+/* ASPEED_PTCR_CLK_EXT_CTRL : 0x44 - Clock Control Extension #1 Register */
+/* TYPE O */
+#define ASPEED_PTCR_CLK_CTRL_TYPEO_MASK GENMASK(15, 0)
+#define ASPEED_PTCR_CLK_CTRL_TYPEO_UNIT 8
+#define ASPEED_PTCR_CLK_CTRL_TYPEO_H 4
+#define ASPEED_PTCR_CLK_CTRL_TYPEO_L 0
+
+#define PWM_MAX 255
+
+#define M_PWM_DIV_H 0x00
+#define M_PWM_DIV_L 0x05
+#define M_PWM_PERIOD 0x5F
+#define M_TACH_CLK_DIV 0x00
+#define M_TACH_MODE 0x00
+#define M_TACH_UNIT 0x1000
+#define INIT_FAN_CTRL 0xFF
+
+struct aspeed_pwm_tacho_data {
+ struct regmap *regmap;
+ unsigned long clk_freq;
+ bool pwm_present[8];
+ bool fan_tach_present[16];
+ u8 type_pwm_clock_unit[3];
+ u8 type_pwm_clock_division_h[3];
+ u8 type_pwm_clock_division_l[3];
+ u8 type_fan_tach_clock_division[3];
+ u16 type_fan_tach_unit[3];
+ u8 pwm_port_type[8];
+ u8 pwm_port_fan_ctrl[8];
+ u8 fan_tach_ch_source[16];
+ const struct attribute_group *groups[3];
+};
+
+enum type { TYPEM, TYPEN, TYPEO };
+
+struct type_params {
+ u32 l_value;
+ u32 h_value;
+ u32 unit_value;
+ u32 clk_ctrl_mask;
+ u32 clk_ctrl_reg;
+ u32 ctrl_reg;
+ u32 ctrl_reg1;
+};
+
+static const struct type_params type_params[] = {
+ [TYPEM] = {
+ .l_value = ASPEED_PTCR_CLK_CTRL_TYPEM_L,
+ .h_value = ASPEED_PTCR_CLK_CTRL_TYPEM_H,
+ .unit_value = ASPEED_PTCR_CLK_CTRL_TYPEM_UNIT,
+ .clk_ctrl_mask = ASPEED_PTCR_CLK_CTRL_TYPEM_MASK,
+ .clk_ctrl_reg = ASPEED_PTCR_CLK_CTRL,
+ .ctrl_reg = ASPEED_PTCR_TYPEM_CTRL,
+ .ctrl_reg1 = ASPEED_PTCR_TYPEM_CTRL1,
+ },
+ [TYPEN] = {
+ .l_value = ASPEED_PTCR_CLK_CTRL_TYPEN_L,
+ .h_value = ASPEED_PTCR_CLK_CTRL_TYPEN_H,
+ .unit_value = ASPEED_PTCR_CLK_CTRL_TYPEN_UNIT,
+ .clk_ctrl_mask = ASPEED_PTCR_CLK_CTRL_TYPEN_MASK,
+ .clk_ctrl_reg = ASPEED_PTCR_CLK_CTRL,
+ .ctrl_reg = ASPEED_PTCR_TYPEN_CTRL,
+ .ctrl_reg1 = ASPEED_PTCR_TYPEN_CTRL1,
+ },
+ [TYPEO] = {
+ .l_value = ASPEED_PTCR_CLK_CTRL_TYPEO_L,
+ .h_value = ASPEED_PTCR_CLK_CTRL_TYPEO_H,
+ .unit_value = ASPEED_PTCR_CLK_CTRL_TYPEO_UNIT,
+ .clk_ctrl_mask = ASPEED_PTCR_CLK_CTRL_TYPEO_MASK,
+ .clk_ctrl_reg = ASPEED_PTCR_CLK_CTRL_EXT,
+ .ctrl_reg = ASPEED_PTCR_TYPEO_CTRL,
+ .ctrl_reg1 = ASPEED_PTCR_TYPEO_CTRL1,
+ }
+};
+
+enum pwm_port { PWMA, PWMB, PWMC, PWMD, PWME, PWMF, PWMG, PWMH };
+
+struct pwm_port_params {
+ u32 pwm_en;
+ u32 ctrl_reg;
+ u32 type_part1;
+ u32 type_part2;
+ u32 type_mask;
+ u32 duty_ctrl_rise_point;
+ u32 duty_ctrl_fall_point;
+ u32 duty_ctrl_reg;
+ u32 duty_ctrl_rise_fall_mask;
+};
+
+static const struct pwm_port_params pwm_port_params[] = {
+ [PWMA] = {
+ .pwm_en = ASPEED_PTCR_CTRL_PWMA_EN,
+ .ctrl_reg = ASPEED_PTCR_CTRL,
+ .type_part1 = ASPEED_PTCR_CTRL_SET_PWMA_TYPE_PART1,
+ .type_part2 = ASPEED_PTCR_CTRL_SET_PWMA_TYPE_PART2,
+ .type_mask = ASPEED_PTCR_CTRL_SET_PWMA_TYPE_MASK,
+ .duty_ctrl_rise_point = DUTY_CTRL_PWM1_RISE_POINT,
+ .duty_ctrl_fall_point = DUTY_CTRL_PWM1_FALL_POINT,
+ .duty_ctrl_reg = ASPEED_PTCR_DUTY0_CTRL,
+ .duty_ctrl_rise_fall_mask = DUTY_CTRL_PWM1_RISE_FALL_MASK,
+ },
+ [PWMB] = {
+ .pwm_en = ASPEED_PTCR_CTRL_PWMB_EN,
+ .ctrl_reg = ASPEED_PTCR_CTRL,
+ .type_part1 = ASPEED_PTCR_CTRL_SET_PWMB_TYPE_PART1,
+ .type_part2 = ASPEED_PTCR_CTRL_SET_PWMB_TYPE_PART2,
+ .type_mask = ASPEED_PTCR_CTRL_SET_PWMB_TYPE_MASK,
+ .duty_ctrl_rise_point = DUTY_CTRL_PWM2_RISE_POINT,
+ .duty_ctrl_fall_point = DUTY_CTRL_PWM2_FALL_POINT,
+ .duty_ctrl_reg = ASPEED_PTCR_DUTY0_CTRL,
+ .duty_ctrl_rise_fall_mask = DUTY_CTRL_PWM2_RISE_FALL_MASK,
+ },
+ [PWMC] = {
+ .pwm_en = ASPEED_PTCR_CTRL_PWMC_EN,
+ .ctrl_reg = ASPEED_PTCR_CTRL,
+ .type_part1 = ASPEED_PTCR_CTRL_SET_PWMC_TYPE_PART1,
+ .type_part2 = ASPEED_PTCR_CTRL_SET_PWMC_TYPE_PART2,
+ .type_mask = ASPEED_PTCR_CTRL_SET_PWMC_TYPE_MASK,
+ .duty_ctrl_rise_point = DUTY_CTRL_PWM1_RISE_POINT,
+ .duty_ctrl_fall_point = DUTY_CTRL_PWM1_FALL_POINT,
+ .duty_ctrl_reg = ASPEED_PTCR_DUTY1_CTRL,
+ .duty_ctrl_rise_fall_mask = DUTY_CTRL_PWM1_RISE_FALL_MASK,
+ },
+ [PWMD] = {
+ .pwm_en = ASPEED_PTCR_CTRL_PWMD_EN,
+ .ctrl_reg = ASPEED_PTCR_CTRL,
+ .type_part1 = ASPEED_PTCR_CTRL_SET_PWMD_TYPE_PART1,
+ .type_part2 = ASPEED_PTCR_CTRL_SET_PWMD_TYPE_PART2,
+ .type_mask = ASPEED_PTCR_CTRL_SET_PWMD_TYPE_MASK,
+ .duty_ctrl_rise_point = DUTY_CTRL_PWM2_RISE_POINT,
+ .duty_ctrl_fall_point = DUTY_CTRL_PWM2_FALL_POINT,
+ .duty_ctrl_reg = ASPEED_PTCR_DUTY1_CTRL,
+ .duty_ctrl_rise_fall_mask = DUTY_CTRL_PWM2_RISE_FALL_MASK,
+ },
+ [PWME] = {
+ .pwm_en = ASPEED_PTCR_CTRL_PWME_EN,
+ .ctrl_reg = ASPEED_PTCR_CTRL_EXT,
+ .type_part1 = ASPEED_PTCR_CTRL_SET_PWME_TYPE_PART1,
+ .type_part2 = ASPEED_PTCR_CTRL_SET_PWME_TYPE_PART2,
+ .type_mask = ASPEED_PTCR_CTRL_SET_PWME_TYPE_MASK,
+ .duty_ctrl_rise_point = DUTY_CTRL_PWM1_RISE_POINT,
+ .duty_ctrl_fall_point = DUTY_CTRL_PWM1_FALL_POINT,
+ .duty_ctrl_reg = ASPEED_PTCR_DUTY2_CTRL,
+ .duty_ctrl_rise_fall_mask = DUTY_CTRL_PWM1_RISE_FALL_MASK,
+ },
+ [PWMF] = {
+ .pwm_en = ASPEED_PTCR_CTRL_PWMF_EN,
+ .ctrl_reg = ASPEED_PTCR_CTRL_EXT,
+ .type_part1 = ASPEED_PTCR_CTRL_SET_PWMF_TYPE_PART1,
+ .type_part2 = ASPEED_PTCR_CTRL_SET_PWMF_TYPE_PART2,
+ .type_mask = ASPEED_PTCR_CTRL_SET_PWMF_TYPE_MASK,
+ .duty_ctrl_rise_point = DUTY_CTRL_PWM2_RISE_POINT,
+ .duty_ctrl_fall_point = DUTY_CTRL_PWM2_FALL_POINT,
+ .duty_ctrl_reg = ASPEED_PTCR_DUTY2_CTRL,
+ .duty_ctrl_rise_fall_mask = DUTY_CTRL_PWM2_RISE_FALL_MASK,
+ },
+ [PWMG] = {
+ .pwm_en = ASPEED_PTCR_CTRL_PWMG_EN,
+ .ctrl_reg = ASPEED_PTCR_CTRL_EXT,
+ .type_part1 = ASPEED_PTCR_CTRL_SET_PWMG_TYPE_PART1,
+ .type_part2 = ASPEED_PTCR_CTRL_SET_PWMG_TYPE_PART2,
+ .type_mask = ASPEED_PTCR_CTRL_SET_PWMG_TYPE_MASK,
+ .duty_ctrl_rise_point = DUTY_CTRL_PWM1_RISE_POINT,
+ .duty_ctrl_fall_point = DUTY_CTRL_PWM1_FALL_POINT,
+ .duty_ctrl_reg = ASPEED_PTCR_DUTY3_CTRL,
+ .duty_ctrl_rise_fall_mask = DUTY_CTRL_PWM1_RISE_FALL_MASK,
+ },
+ [PWMH] = {
+ .pwm_en = ASPEED_PTCR_CTRL_PWMH_EN,
+ .ctrl_reg = ASPEED_PTCR_CTRL_EXT,
+ .type_part1 = ASPEED_PTCR_CTRL_SET_PWMH_TYPE_PART1,
+ .type_part2 = ASPEED_PTCR_CTRL_SET_PWMH_TYPE_PART2,
+ .type_mask = ASPEED_PTCR_CTRL_SET_PWMH_TYPE_MASK,
+ .duty_ctrl_rise_point = DUTY_CTRL_PWM2_RISE_POINT,
+ .duty_ctrl_fall_point = DUTY_CTRL_PWM2_FALL_POINT,
+ .duty_ctrl_reg = ASPEED_PTCR_DUTY3_CTRL,
+ .duty_ctrl_rise_fall_mask = DUTY_CTRL_PWM2_RISE_FALL_MASK,
+ }
+};
+
+static int regmap_aspeed_pwm_tacho_reg_write(void *context, unsigned int reg,
+ unsigned int val)
+{
+ void __iomem *regs = (void __iomem *)context;
+
+ writel(val, regs + reg);
+ return 0;
+}
+
+static int regmap_aspeed_pwm_tacho_reg_read(void *context, unsigned int reg,
+ unsigned int *val)
+{
+ void __iomem *regs = (void __iomem *)context;
+
+ *val = readl(regs + reg);
+ return 0;
+}
+
+static const struct regmap_config aspeed_pwm_tacho_regmap_config = {
+ .reg_bits = 32,
+ .val_bits = 32,
+ .reg_stride = 4,
+ .max_register = ASPEED_PTCR_TYPEO_LIMIT,
+ .reg_write = regmap_aspeed_pwm_tacho_reg_write,
+ .reg_read = regmap_aspeed_pwm_tacho_reg_read,
+ .fast_io = true,
+};
+
+static void aspeed_set_clock_enable(struct regmap *regmap, bool val)
+{
+ regmap_update_bits(regmap, ASPEED_PTCR_CTRL,
+ ASPEED_PTCR_CTRL_CLK_EN,
+ val ? ASPEED_PTCR_CTRL_CLK_EN : 0);
+}
+
+static void aspeed_set_clock_source(struct regmap *regmap, int val)
+{
+ regmap_update_bits(regmap, ASPEED_PTCR_CTRL,
+ ASPEED_PTCR_CTRL_CLK_SRC,
+ val ? ASPEED_PTCR_CTRL_CLK_SRC : 0);
+}
+
+static void aspeed_set_pwm_clock_values(struct regmap *regmap, u8 type,
+ u8 div_high, u8 div_low, u8 unit)
+{
+ u32 reg_value = ((div_high << type_params[type].h_value) |
+ (div_low << type_params[type].l_value) |
+ (unit << type_params[type].unit_value));
+
+ regmap_update_bits(regmap, type_params[type].clk_ctrl_reg,
+ type_params[type].clk_ctrl_mask, reg_value);
+}
+
+static void aspeed_set_pwm_port_enable(struct regmap *regmap, u8 pwm_port,
+ bool enable)
+{
+ regmap_update_bits(regmap, pwm_port_params[pwm_port].ctrl_reg,
+ pwm_port_params[pwm_port].pwm_en,
+ enable ? pwm_port_params[pwm_port].pwm_en : 0);
+}
+
+static void aspeed_set_pwm_port_type(struct regmap *regmap,
+ u8 pwm_port, u8 type)
+{
+ u32 reg_value = (type & 0x1) << pwm_port_params[pwm_port].type_part1;
+
+ reg_value |= (type & 0x2) << pwm_port_params[pwm_port].type_part2;
+
+ regmap_update_bits(regmap, pwm_port_params[pwm_port].ctrl_reg,
+ pwm_port_params[pwm_port].type_mask, reg_value);
+}
+
+static void aspeed_set_pwm_port_duty_rising_falling(struct regmap *regmap,
+ u8 pwm_port, u8 rising,
+ u8 falling)
+{
+ u32 reg_value = (rising <<
+ pwm_port_params[pwm_port].duty_ctrl_rise_point);
+ reg_value |= (falling <<
+ pwm_port_params[pwm_port].duty_ctrl_fall_point);
+
+ regmap_update_bits(regmap, pwm_port_params[pwm_port].duty_ctrl_reg,
+ pwm_port_params[pwm_port].duty_ctrl_rise_fall_mask,
+ reg_value);
+}
+
+static void aspeed_set_tacho_type_enable(struct regmap *regmap, u8 type,
+ bool enable)
+{
+ regmap_update_bits(regmap, type_params[type].ctrl_reg,
+ TYPE_CTRL_FAN_TYPE_EN,
+ enable ? TYPE_CTRL_FAN_TYPE_EN : 0);
+}
+
+static void aspeed_set_tacho_type_values(struct regmap *regmap, u8 type,
+ u8 mode, u16 unit, u8 division)
+{
+ u32 reg_value = ((mode << TYPE_CTRL_FAN_MODE) |
+ (unit << TYPE_CTRL_FAN_PERIOD) |
+ (division << TYPE_CTRL_FAN_DIVISION));
+
+ regmap_update_bits(regmap, type_params[type].ctrl_reg,
+ TYPE_CTRL_FAN_MASK, reg_value);
+ regmap_update_bits(regmap, type_params[type].ctrl_reg1,
+ TYPE_CTRL_FAN1_MASK, unit << 16);
+}
+
+static void aspeed_set_fan_tach_ch_enable(struct regmap *regmap, u8 fan_tach_ch,
+ bool enable)
+{
+ regmap_update_bits(regmap, ASPEED_PTCR_CTRL,
+ ASPEED_PTCR_CTRL_FAN_NUM_EN(fan_tach_ch),
+ enable ?
+ ASPEED_PTCR_CTRL_FAN_NUM_EN(fan_tach_ch) : 0);
+}
+
+static void aspeed_set_fan_tach_ch_source(struct regmap *regmap, u8 fan_tach_ch,
+ u8 fan_tach_ch_source)
+{
+ u32 reg_value1 = ((fan_tach_ch_source & 0x3) <<
+ TACH_PWM_SOURCE_BIT01(fan_tach_ch));
+ u32 reg_value2 = (((fan_tach_ch_source & 0x4) >> 2) <<
+ TACH_PWM_SOURCE_BIT2(fan_tach_ch));
+
+ regmap_update_bits(regmap, ASPEED_PTCR_TACH_SOURCE,
+ TACH_PWM_SOURCE_MASK_BIT01(fan_tach_ch),
+ reg_value1);
+
+ regmap_update_bits(regmap, ASPEED_PTCR_TACH_SOURCE_EXT,
+ TACH_PWM_SOURCE_MASK_BIT2(fan_tach_ch),
+ reg_value2);
+}
+
+static void aspeed_set_pwm_port_fan_ctrl(struct aspeed_pwm_tacho_data *priv,
+ u8 index, u8 fan_ctrl)
+{
+ u16 period, dc_time_on;
+
+ period = priv->type_pwm_clock_unit[priv->pwm_port_type[index]];
+ period += 1;
+ dc_time_on = (fan_ctrl * period) / PWM_MAX;
+
+ if (dc_time_on == 0) {
+ aspeed_set_pwm_port_enable(priv->regmap, index, false);
+ } else {
+ if (dc_time_on == period)
+ dc_time_on = 0;
+
+ aspeed_set_pwm_port_duty_rising_falling(priv->regmap, index, 0,
+ dc_time_on);
+ aspeed_set_pwm_port_enable(priv->regmap, index, true);
+ }
+}
+
+static u32 aspeed_get_fan_tach_ch_measure_period(struct aspeed_pwm_tacho_data
+ *priv, u8 type)
+{
+ u32 clk;
+ u16 tacho_unit;
+ u8 clk_unit, div_h, div_l, tacho_div;
+
+ clk = priv->clk_freq;
+ clk_unit = priv->type_pwm_clock_unit[type];
+ div_h = priv->type_pwm_clock_division_h[type];
+ div_h = 0x1 << div_h;
+ div_l = priv->type_pwm_clock_division_l[type];
+ if (div_l == 0)
+ div_l = 1;
+ else
+ div_l = div_l * 2;
+
+ tacho_unit = priv->type_fan_tach_unit[type];
+ tacho_div = priv->type_fan_tach_clock_division[type];
+
+ tacho_div = 0x4 << (tacho_div * 2);
+ return clk / (clk_unit * div_h * div_l * tacho_div * tacho_unit);
+}
+
+static u32 aspeed_get_fan_tach_ch_rpm(struct aspeed_pwm_tacho_data *priv,
+ u8 fan_tach_ch)
+{
+ u32 raw_data, tach_div, clk_source, sec, val;
+ u8 fan_tach_ch_source, type;
+
+ regmap_write(priv->regmap, ASPEED_PTCR_TRIGGER, 0);
+ regmap_write(priv->regmap, ASPEED_PTCR_TRIGGER, 0x1 << fan_tach_ch);
+
+ fan_tach_ch_source = priv->fan_tach_ch_source[fan_tach_ch];
+ type = priv->pwm_port_type[fan_tach_ch_source];
+
+ sec = (1000 / aspeed_get_fan_tach_ch_measure_period(priv, type));
+ msleep(sec);
+
+ regmap_read(priv->regmap, ASPEED_PTCR_RESULT, &val);
+ raw_data = val & RESULT_VALUE_MASK;
+ tach_div = priv->type_fan_tach_clock_division[type];
+ tach_div = 0x4 << (tach_div * 2);
+ clk_source = priv->clk_freq;
+
+ if (raw_data == 0)
+ return 0;
+
+ return (clk_source * 60) / (2 * raw_data * tach_div);
+}
+
+static ssize_t set_pwm(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct sensor_device_attribute *sensor_attr = to_sensor_dev_attr(attr);
+ int index = sensor_attr->index;
+ int ret;
+ struct aspeed_pwm_tacho_data *priv = dev_get_drvdata(dev);
+ long fan_ctrl;
+
+ ret = kstrtol(buf, 10, &fan_ctrl);
+ if (ret != 0)
+ return ret;
+
+ if (fan_ctrl < 0 || fan_ctrl > PWM_MAX)
+ return -EINVAL;
+
+ if (priv->pwm_port_fan_ctrl[index] == fan_ctrl)
+ return count;
+
+ priv->pwm_port_fan_ctrl[index] = fan_ctrl;
+ aspeed_set_pwm_port_fan_ctrl(priv, index, fan_ctrl);
+
+ return count;
+}
+
+static ssize_t show_pwm(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct sensor_device_attribute *sensor_attr = to_sensor_dev_attr(attr);
+ int index = sensor_attr->index;
+ struct aspeed_pwm_tacho_data *priv = dev_get_drvdata(dev);
+
+ return sprintf(buf, "%u\n", priv->pwm_port_fan_ctrl[index]);
+}
+
+static ssize_t show_rpm(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct sensor_device_attribute *sensor_attr = to_sensor_dev_attr(attr);
+ int index = sensor_attr->index;
+ u32 rpm;
+ struct aspeed_pwm_tacho_data *priv = dev_get_drvdata(dev);
+
+ rpm = aspeed_get_fan_tach_ch_rpm(priv, index);
+
+ return sprintf(buf, "%u\n", rpm);
+}
+
+static umode_t pwm_is_visible(struct kobject *kobj,
+ struct attribute *a, int index)
+{
+ struct device *dev = container_of(kobj, struct device, kobj);
+ struct aspeed_pwm_tacho_data *priv = dev_get_drvdata(dev);
+
+ if (!priv->pwm_present[index])
+ return 0;
+ return a->mode;
+}
+
+static umode_t fan_dev_is_visible(struct kobject *kobj,
+ struct attribute *a, int index)
+{
+ struct device *dev = container_of(kobj, struct device, kobj);
+ struct aspeed_pwm_tacho_data *priv = dev_get_drvdata(dev);
+
+ if (!priv->fan_tach_present[index])
+ return 0;
+ return a->mode;
+}
+
+static SENSOR_DEVICE_ATTR(pwm0, 0644,
+ show_pwm, set_pwm, 0);
+static SENSOR_DEVICE_ATTR(pwm1, 0644,
+ show_pwm, set_pwm, 1);
+static SENSOR_DEVICE_ATTR(pwm2, 0644,
+ show_pwm, set_pwm, 2);
+static SENSOR_DEVICE_ATTR(pwm3, 0644,
+ show_pwm, set_pwm, 3);
+static SENSOR_DEVICE_ATTR(pwm4, 0644,
+ show_pwm, set_pwm, 4);
+static SENSOR_DEVICE_ATTR(pwm5, 0644,
+ show_pwm, set_pwm, 5);
+static SENSOR_DEVICE_ATTR(pwm6, 0644,
+ show_pwm, set_pwm, 6);
+static SENSOR_DEVICE_ATTR(pwm7, 0644,
+ show_pwm, set_pwm, 7);
+static struct attribute *pwm_dev_attrs[] = {
+ &sensor_dev_attr_pwm0.dev_attr.attr,
+ &sensor_dev_attr_pwm1.dev_attr.attr,
+ &sensor_dev_attr_pwm2.dev_attr.attr,
+ &sensor_dev_attr_pwm3.dev_attr.attr,
+ &sensor_dev_attr_pwm4.dev_attr.attr,
+ &sensor_dev_attr_pwm5.dev_attr.attr,
+ &sensor_dev_attr_pwm6.dev_attr.attr,
+ &sensor_dev_attr_pwm7.dev_attr.attr,
+ NULL,
+};
+
+static const struct attribute_group pwm_dev_group = {
+ .attrs = pwm_dev_attrs,
+ .is_visible = pwm_is_visible,
+};
+
+static SENSOR_DEVICE_ATTR(fan0_input, 0444,
+ show_rpm, NULL, 0);
+static SENSOR_DEVICE_ATTR(fan1_input, 0444,
+ show_rpm, NULL, 1);
+static SENSOR_DEVICE_ATTR(fan2_input, 0444,
+ show_rpm, NULL, 2);
+static SENSOR_DEVICE_ATTR(fan3_input, 0444,
+ show_rpm, NULL, 3);
+static SENSOR_DEVICE_ATTR(fan4_input, 0444,
+ show_rpm, NULL, 4);
+static SENSOR_DEVICE_ATTR(fan5_input, 0444,
+ show_rpm, NULL, 5);
+static SENSOR_DEVICE_ATTR(fan6_input, 0444,
+ show_rpm, NULL, 6);
+static SENSOR_DEVICE_ATTR(fan7_input, 0444,
+ show_rpm, NULL, 7);
+static SENSOR_DEVICE_ATTR(fan8_input, 0444,
+ show_rpm, NULL, 8);
+static SENSOR_DEVICE_ATTR(fan9_input, 0444,
+ show_rpm, NULL, 9);
+static SENSOR_DEVICE_ATTR(fan10_input, 0444,
+ show_rpm, NULL, 10);
+static SENSOR_DEVICE_ATTR(fan11_input, 0444,
+ show_rpm, NULL, 11);
+static SENSOR_DEVICE_ATTR(fan12_input, 0444,
+ show_rpm, NULL, 12);
+static SENSOR_DEVICE_ATTR(fan13_input, 0444,
+ show_rpm, NULL, 13);
+static SENSOR_DEVICE_ATTR(fan14_input, 0444,
+ show_rpm, NULL, 14);
+static SENSOR_DEVICE_ATTR(fan15_input, 0444,
+ show_rpm, NULL, 15);
+static struct attribute *fan_dev_attrs[] = {
+ &sensor_dev_attr_fan0_input.dev_attr.attr,
+ &sensor_dev_attr_fan1_input.dev_attr.attr,
+ &sensor_dev_attr_fan2_input.dev_attr.attr,
+ &sensor_dev_attr_fan3_input.dev_attr.attr,
+ &sensor_dev_attr_fan4_input.dev_attr.attr,
+ &sensor_dev_attr_fan5_input.dev_attr.attr,
+ &sensor_dev_attr_fan6_input.dev_attr.attr,
+ &sensor_dev_attr_fan7_input.dev_attr.attr,
+ &sensor_dev_attr_fan8_input.dev_attr.attr,
+ &sensor_dev_attr_fan9_input.dev_attr.attr,
+ &sensor_dev_attr_fan10_input.dev_attr.attr,
+ &sensor_dev_attr_fan11_input.dev_attr.attr,
+ &sensor_dev_attr_fan12_input.dev_attr.attr,
+ &sensor_dev_attr_fan13_input.dev_attr.attr,
+ &sensor_dev_attr_fan14_input.dev_attr.attr,
+ &sensor_dev_attr_fan15_input.dev_attr.attr,
+ NULL
+};
+
+static const struct attribute_group fan_dev_group = {
+ .attrs = fan_dev_attrs,
+ .is_visible = fan_dev_is_visible,
+};
+
+/*
+ * The clock type is type M :
+ * The PWM frequency = 24MHz / (type M clock division L bit *
+ * type M clock division H bit * (type M PWM period bit + 1))
+ */
+static void aspeed_create_type(struct aspeed_pwm_tacho_data *priv)
+{
+ priv->type_pwm_clock_division_h[TYPEM] = M_PWM_DIV_H;
+ priv->type_pwm_clock_division_l[TYPEM] = M_PWM_DIV_L;
+ priv->type_pwm_clock_unit[TYPEM] = M_PWM_PERIOD;
+ aspeed_set_pwm_clock_values(priv->regmap, TYPEM, M_PWM_DIV_H,
+ M_PWM_DIV_L, M_PWM_PERIOD);
+ aspeed_set_tacho_type_enable(priv->regmap, TYPEM, true);
+ priv->type_fan_tach_clock_division[TYPEM] = M_TACH_CLK_DIV;
+ priv->type_fan_tach_unit[TYPEM] = M_TACH_UNIT;
+ aspeed_set_tacho_type_values(priv->regmap, TYPEM, M_TACH_MODE,
+ M_TACH_UNIT, M_TACH_CLK_DIV);
+}
+
+static void aspeed_create_pwm_port(struct aspeed_pwm_tacho_data *priv,
+ u8 pwm_port)
+{
+ aspeed_set_pwm_port_enable(priv->regmap, pwm_port, true);
+ priv->pwm_present[pwm_port] = true;
+
+ priv->pwm_port_type[pwm_port] = TYPEM;
+ aspeed_set_pwm_port_type(priv->regmap, pwm_port, TYPEM);
+
+ priv->pwm_port_fan_ctrl[pwm_port] = INIT_FAN_CTRL;
+ aspeed_set_pwm_port_fan_ctrl(priv, pwm_port, INIT_FAN_CTRL);
+}
+
+static void aspeed_create_fan_tach_channel(struct aspeed_pwm_tacho_data *priv,
+ u8 *fan_tach_ch,
+ int count,
+ u8 pwm_source)
+{
+ u8 val, index;
+
+ for (val = 0; val < count; val++) {
+ index = fan_tach_ch[val];
+ aspeed_set_fan_tach_ch_enable(priv->regmap, index, true);
+ priv->fan_tach_present[index] = true;
+ priv->fan_tach_ch_source[index] = pwm_source;
+ aspeed_set_fan_tach_ch_source(priv->regmap, index, pwm_source);
+ }
+}
+
+static int aspeed_create_fan(struct device *dev,
+ struct device_node *child,
+ struct aspeed_pwm_tacho_data *priv)
+{
+ u8 *fan_tach_ch;
+ u32 pwm_port;
+ int ret, count;
+
+ ret = of_property_read_u32(child, "reg", &pwm_port);
+ if (ret)
+ return ret;
+ aspeed_create_pwm_port(priv, (u8)pwm_port);
+
+ count = of_property_count_u8_elems(child, "aspeed,fan-tach-ch");
+ if (count < 1)
+ return -EINVAL;
+ fan_tach_ch = devm_kzalloc(dev, sizeof(*fan_tach_ch) * count,
+ GFP_KERNEL);
+ if (!fan_tach_ch)
+ return -ENOMEM;
+ ret = of_property_read_u8_array(child, "aspeed,fan-tach-ch",
+ fan_tach_ch, count);
+ if (ret)
+ return ret;
+ aspeed_create_fan_tach_channel(priv, fan_tach_ch, count, pwm_port);
+
+ return 0;
+}
+
+static int aspeed_pwm_tacho_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct device_node *np, *child;
+ struct aspeed_pwm_tacho_data *priv;
+ void __iomem *regs;
+ struct resource *res;
+ struct device *hwmon;
+ struct clk *clk;
+ int ret;
+
+ np = dev->of_node;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!res)
+ return -ENOENT;
+ regs = devm_ioremap_resource(dev, res);
+ if (IS_ERR(regs))
+ return PTR_ERR(regs);
+ priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+ priv->regmap = devm_regmap_init(dev, NULL, (__force void *)regs,
+ &aspeed_pwm_tacho_regmap_config);
+ if (IS_ERR(priv->regmap))
+ return PTR_ERR(priv->regmap);
+ regmap_write(priv->regmap, ASPEED_PTCR_TACH_SOURCE, 0);
+ regmap_write(priv->regmap, ASPEED_PTCR_TACH_SOURCE_EXT, 0);
+
+ clk = devm_clk_get(dev, NULL);
+ if (IS_ERR(clk))
+ return -ENODEV;
+ priv->clk_freq = clk_get_rate(clk);
+ aspeed_set_clock_enable(priv->regmap, true);
+ aspeed_set_clock_source(priv->regmap, 0);
+
+ aspeed_create_type(priv);
+
+ for_each_child_of_node(np, child) {
+ ret = aspeed_create_fan(dev, child, priv);
+ of_node_put(child);
+ if (ret)
+ return ret;
+ }
+ of_node_put(np);
+
+ priv->groups[0] = &pwm_dev_group;
+ priv->groups[1] = &fan_dev_group;
+ priv->groups[2] = NULL;
+ hwmon = devm_hwmon_device_register_with_groups(dev,
+ "aspeed_pwm_tacho",
+ priv, priv->groups);
+ return PTR_ERR_OR_ZERO(hwmon);
+}
+
+static const struct of_device_id of_pwm_tacho_match_table[] = {
+ { .compatible = "aspeed,ast2400-pwm-tacho", },
+ { .compatible = "aspeed,ast2500-pwm-tacho", },
+ {},
+};
+MODULE_DEVICE_TABLE(of, of_pwm_tacho_match_table);
+
+static struct platform_driver aspeed_pwm_tacho_driver = {
+ .probe = aspeed_pwm_tacho_probe,
+ .driver = {
+ .name = "aspeed_pwm_tacho",
+ .of_match_table = of_pwm_tacho_match_table,
+ },
+};
+
+module_platform_driver(aspeed_pwm_tacho_driver);
+
+MODULE_AUTHOR("Jaghathiswari Rankappagounder Natarajan <jaghu@google.com>");
+MODULE_DESCRIPTION("ASPEED PWM and Fan Tacho device driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/hwmon/dell-smm-hwmon.c b/drivers/hwmon/dell-smm-hwmon.c
index 34704b0..3189246 100644
--- a/drivers/hwmon/dell-smm-hwmon.c
+++ b/drivers/hwmon/dell-smm-hwmon.c
@@ -995,6 +995,13 @@
},
.driver_data = (void *)&i8k_config_data[DELL_XPS],
},
+ {
+ .ident = "Dell XPS 15 9560",
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc."),
+ DMI_MATCH(DMI_PRODUCT_NAME, "XPS 15 9560"),
+ },
+ },
{ }
};
diff --git a/drivers/hwmon/hwmon.c b/drivers/hwmon/hwmon.c
index 28375d5..dd6e17c 100644
--- a/drivers/hwmon/hwmon.c
+++ b/drivers/hwmon/hwmon.c
@@ -186,7 +186,7 @@
char *buf)
{
struct hwmon_device_attribute *hattr = to_hwmon_attr(devattr);
- char *s;
+ const char *s;
int ret;
ret = hattr->ops->read_string(dev, hattr->type, hattr->attr,
diff --git a/drivers/hwmon/ina209.c b/drivers/hwmon/ina209.c
index 5378fde..aa0768c 100644
--- a/drivers/hwmon/ina209.c
+++ b/drivers/hwmon/ina209.c
@@ -117,7 +117,7 @@
case INA209_SHUNT_VOLTAGE_POS_WARN:
case INA209_SHUNT_VOLTAGE_NEG_WARN:
/* LSB=10 uV. Convert to mV. */
- return DIV_ROUND_CLOSEST(val, 100);
+ return DIV_ROUND_CLOSEST((s16)val, 100);
case INA209_BUS_VOLTAGE:
case INA209_BUS_VOLTAGE_MAX_PEAK:
@@ -146,7 +146,7 @@
case INA209_CURRENT:
/* LSB=1 mA (selected). Is in mA */
- return val;
+ return (s16)val;
}
/* programmer goofed */
@@ -608,11 +608,18 @@
};
MODULE_DEVICE_TABLE(i2c, ina209_id);
+static const struct of_device_id ina209_of_match[] = {
+ { .compatible = "ti,ina209" },
+ { },
+};
+MODULE_DEVICE_TABLE(of, ina209_of_match);
+
/* This is the driver that will be inserted */
static struct i2c_driver ina209_driver = {
.class = I2C_CLASS_HWMON,
.driver = {
.name = "ina209",
+ .of_match_table = of_match_ptr(ina209_of_match),
},
.probe = ina209_probe,
.remove = ina209_remove,
diff --git a/drivers/hwmon/ina2xx.c b/drivers/hwmon/ina2xx.c
index b24f1d3..62e38fa 100644
--- a/drivers/hwmon/ina2xx.c
+++ b/drivers/hwmon/ina2xx.c
@@ -34,6 +34,7 @@
#include <linux/hwmon.h>
#include <linux/hwmon-sysfs.h>
#include <linux/jiffies.h>
+#include <linux/of_device.h>
#include <linux/of.h>
#include <linux/delay.h>
#include <linux/util_macros.h>
@@ -424,13 +425,19 @@
struct device *hwmon_dev;
u32 val;
int ret, group = 0;
+ enum ina2xx_ids chip;
+
+ if (client->dev.of_node)
+ chip = (enum ina2xx_ids)of_device_get_match_data(&client->dev);
+ else
+ chip = id->driver_data;
data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
if (!data)
return -ENOMEM;
/* set the device type */
- data->config = &ina2xx_config[id->driver_data];
+ data->config = &ina2xx_config[chip];
if (of_property_read_u32(dev->of_node, "shunt-resistor", &val) < 0) {
struct ina2xx_platform_data *pdata = dev_get_platdata(dev);
@@ -487,9 +494,35 @@
};
MODULE_DEVICE_TABLE(i2c, ina2xx_id);
+static const struct of_device_id ina2xx_of_match[] = {
+ {
+ .compatible = "ti,ina219",
+ .data = (void *)ina219
+ },
+ {
+ .compatible = "ti,ina220",
+ .data = (void *)ina219
+ },
+ {
+ .compatible = "ti,ina226",
+ .data = (void *)ina226
+ },
+ {
+ .compatible = "ti,ina230",
+ .data = (void *)ina226
+ },
+ {
+ .compatible = "ti,ina231",
+ .data = (void *)ina226
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, ina2xx_of_match);
+
static struct i2c_driver ina2xx_driver = {
.driver = {
.name = "ina2xx",
+ .of_match_table = of_match_ptr(ina2xx_of_match),
},
.probe = ina2xx_probe,
.id_table = ina2xx_id,
diff --git a/drivers/hwmon/lm63.c b/drivers/hwmon/lm63.c
index 2e19486..4c17709 100644
--- a/drivers/hwmon/lm63.c
+++ b/drivers/hwmon/lm63.c
@@ -46,6 +46,7 @@
#include <linux/hwmon.h>
#include <linux/err.h>
#include <linux/mutex.h>
+#include <linux/of_device.h>
#include <linux/sysfs.h>
#include <linux/types.h>
@@ -1115,6 +1116,10 @@
mutex_init(&data->update_lock);
/* Set the device type */
+ if (client->dev.of_node)
+ data->kind = (enum chips)of_device_get_match_data(&client->dev);
+ else
+ data->kind = id->driver_data;
data->kind = id->driver_data;
if (data->kind == lm64)
data->temp2_offset = 16000;
@@ -1149,10 +1154,28 @@
};
MODULE_DEVICE_TABLE(i2c, lm63_id);
+static const struct of_device_id lm63_of_match[] = {
+ {
+ .compatible = "national,lm63",
+ .data = (void *)lm63
+ },
+ {
+ .compatible = "national,lm64",
+ .data = (void *)lm64
+ },
+ {
+ .compatible = "national,lm96163",
+ .data = (void *)lm96163
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, lm63_of_match);
+
static struct i2c_driver lm63_driver = {
.class = I2C_CLASS_HWMON,
.driver = {
.name = "lm63",
+ .of_match_table = of_match_ptr(lm63_of_match),
},
.probe = lm63_probe,
.id_table = lm63_id,
diff --git a/drivers/hwmon/lm75.c b/drivers/hwmon/lm75.c
index eff3b24..005ffb5 100644
--- a/drivers/hwmon/lm75.c
+++ b/drivers/hwmon/lm75.c
@@ -26,6 +26,7 @@
#include <linux/hwmon.h>
#include <linux/hwmon-sysfs.h>
#include <linux/err.h>
+#include <linux/of_device.h>
#include <linux/of.h>
#include <linux/regmap.h>
#include "lm75.h"
@@ -273,7 +274,12 @@
int status, err;
u8 set_mask, clr_mask;
int new;
- enum lm75_type kind = id->driver_data;
+ enum lm75_type kind;
+
+ if (client->dev.of_node)
+ kind = (enum lm75_type)of_device_get_match_data(&client->dev);
+ else
+ kind = id->driver_data;
if (!i2c_check_functionality(client->adapter,
I2C_FUNC_SMBUS_BYTE_DATA | I2C_FUNC_SMBUS_WORD_DATA))
@@ -424,6 +430,95 @@
};
MODULE_DEVICE_TABLE(i2c, lm75_ids);
+static const struct of_device_id lm75_of_match[] = {
+ {
+ .compatible = "adi,adt75",
+ .data = (void *)adt75
+ },
+ {
+ .compatible = "dallas,ds1775",
+ .data = (void *)ds1775
+ },
+ {
+ .compatible = "dallas,ds75",
+ .data = (void *)ds75
+ },
+ {
+ .compatible = "dallas,ds7505",
+ .data = (void *)ds7505
+ },
+ {
+ .compatible = "gmt,g751",
+ .data = (void *)g751
+ },
+ {
+ .compatible = "national,lm75",
+ .data = (void *)lm75
+ },
+ {
+ .compatible = "national,lm75a",
+ .data = (void *)lm75a
+ },
+ {
+ .compatible = "national,lm75b",
+ .data = (void *)lm75b
+ },
+ {
+ .compatible = "maxim,max6625",
+ .data = (void *)max6625
+ },
+ {
+ .compatible = "maxim,max6626",
+ .data = (void *)max6626
+ },
+ {
+ .compatible = "maxim,mcp980x",
+ .data = (void *)mcp980x
+ },
+ {
+ .compatible = "st,stds75",
+ .data = (void *)stds75
+ },
+ {
+ .compatible = "microchip,tcn75",
+ .data = (void *)tcn75
+ },
+ {
+ .compatible = "ti,tmp100",
+ .data = (void *)tmp100
+ },
+ {
+ .compatible = "ti,tmp101",
+ .data = (void *)tmp101
+ },
+ {
+ .compatible = "ti,tmp105",
+ .data = (void *)tmp105
+ },
+ {
+ .compatible = "ti,tmp112",
+ .data = (void *)tmp112
+ },
+ {
+ .compatible = "ti,tmp175",
+ .data = (void *)tmp175
+ },
+ {
+ .compatible = "ti,tmp275",
+ .data = (void *)tmp275
+ },
+ {
+ .compatible = "ti,tmp75",
+ .data = (void *)tmp75
+ },
+ {
+ .compatible = "ti,tmp75c",
+ .data = (void *)tmp75c
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, lm75_of_match);
+
#define LM75A_ID 0xA1
/* Return 0 if detection is successful, -ENODEV otherwise */
@@ -560,6 +655,7 @@
.class = I2C_CLASS_HWMON,
.driver = {
.name = "lm75",
+ .of_match_table = of_match_ptr(lm75_of_match),
.pm = LM75_DEV_PM_OPS,
},
.probe = lm75_probe,
diff --git a/drivers/hwmon/lm85.c b/drivers/hwmon/lm85.c
index 691469f..0a32587 100644
--- a/drivers/hwmon/lm85.c
+++ b/drivers/hwmon/lm85.c
@@ -25,6 +25,7 @@
*/
#include <linux/module.h>
+#include <linux/of_device.h>
#include <linux/init.h>
#include <linux/slab.h>
#include <linux/jiffies.h>
@@ -1552,7 +1553,10 @@
return -ENOMEM;
data->client = client;
- data->type = id->driver_data;
+ if (client->dev.of_node)
+ data->type = (enum chips)of_device_get_match_data(&client->dev);
+ else
+ data->type = id->driver_data;
mutex_init(&data->update_lock);
/* Fill in the chip specific driver values */
@@ -1623,10 +1627,60 @@
};
MODULE_DEVICE_TABLE(i2c, lm85_id);
+static const struct of_device_id lm85_of_match[] = {
+ {
+ .compatible = "adi,adm1027",
+ .data = (void *)adm1027
+ },
+ {
+ .compatible = "adi,adt7463",
+ .data = (void *)adt7463
+ },
+ {
+ .compatible = "adi,adt7468",
+ .data = (void *)adt7468
+ },
+ {
+ .compatible = "national,lm85",
+ .data = (void *)lm85
+ },
+ {
+ .compatible = "national,lm85b",
+ .data = (void *)lm85
+ },
+ {
+ .compatible = "national,lm85c",
+ .data = (void *)lm85
+ },
+ {
+ .compatible = "smsc,emc6d100",
+ .data = (void *)emc6d100
+ },
+ {
+ .compatible = "smsc,emc6d101",
+ .data = (void *)emc6d100
+ },
+ {
+ .compatible = "smsc,emc6d102",
+ .data = (void *)emc6d102
+ },
+ {
+ .compatible = "smsc,emc6d103",
+ .data = (void *)emc6d103
+ },
+ {
+ .compatible = "smsc,emc6d103s",
+ .data = (void *)emc6d103s
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, lm85_of_match);
+
static struct i2c_driver lm85_driver = {
.class = I2C_CLASS_HWMON,
.driver = {
.name = "lm85",
+ .of_match_table = of_match_ptr(lm85_of_match),
},
.probe = lm85_probe,
.id_table = lm85_id,
diff --git a/drivers/hwmon/lm87.c b/drivers/hwmon/lm87.c
index e06faf9..b48d307 100644
--- a/drivers/hwmon/lm87.c
+++ b/drivers/hwmon/lm87.c
@@ -66,6 +66,7 @@
#include <linux/hwmon-vid.h>
#include <linux/err.h>
#include <linux/mutex.h>
+#include <linux/regulator/consumer.h>
/*
* Addresses to scan
@@ -74,8 +75,6 @@
static const unsigned short normal_i2c[] = { 0x2c, 0x2d, 0x2e, I2C_CLIENT_END };
-enum chips { lm87, adm1024 };
-
/*
* The LM87 registers
*/
@@ -855,8 +854,26 @@
{
struct lm87_data *data = i2c_get_clientdata(client);
int rc;
+ struct device_node *of_node = client->dev.of_node;
+ u8 val = 0;
+ struct regulator *vcc = NULL;
- if (dev_get_platdata(&client->dev)) {
+ if (of_node) {
+ if (of_property_read_bool(of_node, "has-temp3"))
+ val |= CHAN_TEMP3;
+ if (of_property_read_bool(of_node, "has-in6"))
+ val |= CHAN_NO_FAN(0);
+ if (of_property_read_bool(of_node, "has-in7"))
+ val |= CHAN_NO_FAN(1);
+ vcc = devm_regulator_get_optional(&client->dev, "vcc");
+ if (!IS_ERR(vcc)) {
+ if (regulator_get_voltage(vcc) == 5000000)
+ val |= CHAN_VCC_5V;
+ }
+ data->channel = val;
+ lm87_write_value(client,
+ LM87_REG_CHANNEL_MODE, data->channel);
+ } else if (dev_get_platdata(&client->dev)) {
data->channel = *(u8 *)dev_get_platdata(&client->dev);
lm87_write_value(client,
LM87_REG_CHANNEL_MODE, data->channel);
@@ -962,16 +979,24 @@
*/
static const struct i2c_device_id lm87_id[] = {
- { "lm87", lm87 },
- { "adm1024", adm1024 },
+ { "lm87", 0 },
+ { "adm1024", 0 },
{ }
};
MODULE_DEVICE_TABLE(i2c, lm87_id);
+static const struct of_device_id lm87_of_match[] = {
+ { .compatible = "ti,lm87" },
+ { .compatible = "adi,adm1024" },
+ { },
+};
+MODULE_DEVICE_TABLE(of, lm87_of_match);
+
static struct i2c_driver lm87_driver = {
.class = I2C_CLASS_HWMON,
.driver = {
.name = "lm87",
+ .of_match_table = lm87_of_match,
},
.probe = lm87_probe,
.id_table = lm87_id,
diff --git a/drivers/hwmon/lm90.c b/drivers/hwmon/lm90.c
index aff5297..c2f411c 100644
--- a/drivers/hwmon/lm90.c
+++ b/drivers/hwmon/lm90.c
@@ -92,6 +92,7 @@
#include <linux/hwmon.h>
#include <linux/err.h>
#include <linux/mutex.h>
+#include <linux/of_device.h>
#include <linux/sysfs.h>
#include <linux/interrupt.h>
#include <linux/regulator/consumer.h>
@@ -235,6 +236,99 @@
};
MODULE_DEVICE_TABLE(i2c, lm90_id);
+static const struct of_device_id lm90_of_match[] = {
+ {
+ .compatible = "adi,adm1032",
+ .data = (void *)adm1032
+ },
+ {
+ .compatible = "adi,adt7461",
+ .data = (void *)adt7461
+ },
+ {
+ .compatible = "adi,adt7461a",
+ .data = (void *)adt7461
+ },
+ {
+ .compatible = "gmt,g781",
+ .data = (void *)g781
+ },
+ {
+ .compatible = "national,lm90",
+ .data = (void *)lm90
+ },
+ {
+ .compatible = "national,lm86",
+ .data = (void *)lm86
+ },
+ {
+ .compatible = "national,lm89",
+ .data = (void *)lm86
+ },
+ {
+ .compatible = "national,lm99",
+ .data = (void *)lm99
+ },
+ {
+ .compatible = "dallas,max6646",
+ .data = (void *)max6646
+ },
+ {
+ .compatible = "dallas,max6647",
+ .data = (void *)max6646
+ },
+ {
+ .compatible = "dallas,max6649",
+ .data = (void *)max6646
+ },
+ {
+ .compatible = "dallas,max6657",
+ .data = (void *)max6657
+ },
+ {
+ .compatible = "dallas,max6658",
+ .data = (void *)max6657
+ },
+ {
+ .compatible = "dallas,max6659",
+ .data = (void *)max6659
+ },
+ {
+ .compatible = "dallas,max6680",
+ .data = (void *)max6680
+ },
+ {
+ .compatible = "dallas,max6681",
+ .data = (void *)max6680
+ },
+ {
+ .compatible = "dallas,max6695",
+ .data = (void *)max6696
+ },
+ {
+ .compatible = "dallas,max6696",
+ .data = (void *)max6696
+ },
+ {
+ .compatible = "onnn,nct1008",
+ .data = (void *)adt7461
+ },
+ {
+ .compatible = "winbond,w83l771",
+ .data = (void *)w83l771
+ },
+ {
+ .compatible = "nxp,sa56004",
+ .data = (void *)sa56004
+ },
+ {
+ .compatible = "ti,tmp451",
+ .data = (void *)tmp451
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, lm90_of_match);
+
/*
* chip type specific parameters
*/
@@ -1677,7 +1771,10 @@
mutex_init(&data->update_lock);
/* Set the device type */
- data->kind = id->driver_data;
+ if (client->dev.of_node)
+ data->kind = (enum chips)of_device_get_match_data(&client->dev);
+ else
+ data->kind = id->driver_data;
if (data->kind == adm1032) {
if (!i2c_check_functionality(adapter, I2C_FUNC_SMBUS_BYTE))
client->flags &= ~I2C_CLIENT_PEC;
@@ -1816,6 +1913,7 @@
.class = I2C_CLASS_HWMON,
.driver = {
.name = "lm90",
+ .of_match_table = of_match_ptr(lm90_of_match),
},
.probe = lm90_probe,
.alert = lm90_alert,
diff --git a/drivers/hwmon/lm95245.c b/drivers/hwmon/lm95245.c
index a3bfd88..27cb06d 100644
--- a/drivers/hwmon/lm95245.c
+++ b/drivers/hwmon/lm95245.c
@@ -622,10 +622,18 @@
};
MODULE_DEVICE_TABLE(i2c, lm95245_id);
+static const struct of_device_id lm95245_of_match[] = {
+ { .compatible = "national,lm95235" },
+ { .compatible = "national,lm95245" },
+ { },
+};
+MODULE_DEVICE_TABLE(of, lm95245_of_match);
+
static struct i2c_driver lm95245_driver = {
.class = I2C_CLASS_HWMON,
.driver = {
.name = "lm95245",
+ .of_match_table = of_match_ptr(lm95245_of_match),
},
.probe = lm95245_probe,
.id_table = lm95245_id,
diff --git a/drivers/hwmon/max6697.c b/drivers/hwmon/max6697.c
index f03a717..221fd14 100644
--- a/drivers/hwmon/max6697.c
+++ b/drivers/hwmon/max6697.c
@@ -24,6 +24,7 @@
#include <linux/hwmon-sysfs.h>
#include <linux/err.h>
#include <linux/mutex.h>
+#include <linux/of_device.h>
#include <linux/of.h>
#include <linux/platform_data/max6697.h>
@@ -632,7 +633,10 @@
if (!data)
return -ENOMEM;
- data->type = id->driver_data;
+ if (client->dev.of_node)
+ data->type = (enum chips)of_device_get_match_data(&client->dev);
+ else
+ data->type = id->driver_data;
data->chip = &max6697_chip_data[data->type];
data->client = client;
mutex_init(&data->update_lock);
@@ -662,10 +666,56 @@
};
MODULE_DEVICE_TABLE(i2c, max6697_id);
+static const struct of_device_id max6697_of_match[] = {
+ {
+ .compatible = "maxim,max6581",
+ .data = (void *)max6581
+ },
+ {
+ .compatible = "maxim,max6602",
+ .data = (void *)max6602
+ },
+ {
+ .compatible = "maxim,max6622",
+ .data = (void *)max6622
+ },
+ {
+ .compatible = "maxim,max6636",
+ .data = (void *)max6636
+ },
+ {
+ .compatible = "maxim,max6689",
+ .data = (void *)max6689
+ },
+ {
+ .compatible = "maxim,max6693",
+ .data = (void *)max6693
+ },
+ {
+ .compatible = "maxim,max6694",
+ .data = (void *)max6694
+ },
+ {
+ .compatible = "maxim,max6697",
+ .data = (void *)max6697
+ },
+ {
+ .compatible = "maxim,max6698",
+ .data = (void *)max6698
+ },
+ {
+ .compatible = "maxim,max6699",
+ .data = (void *)max6699
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, max6697_of_match);
+
static struct i2c_driver max6697_driver = {
.class = I2C_CLASS_HWMON,
.driver = {
.name = "max6697",
+ .of_match_table = of_match_ptr(max6697_of_match),
},
.probe = max6697_probe,
.id_table = max6697_id,
diff --git a/drivers/hwmon/pmbus/adm1275.c b/drivers/hwmon/pmbus/adm1275.c
index 4ab5293..00d6995a 100644
--- a/drivers/hwmon/pmbus/adm1275.c
+++ b/drivers/hwmon/pmbus/adm1275.c
@@ -101,8 +101,8 @@
[0] = { 27169, 0, -1 }, /* voltage */
[1] = { 806, 20475, -1 }, /* current, irange25 */
[2] = { 404, 20475, -1 }, /* current, irange50 */
- [3] = { 0, -1, 8549 }, /* power, irange25 */
- [4] = { 0, -1, 4279 }, /* power, irange50 */
+ [3] = { 8549, 0, -1 }, /* power, irange25 */
+ [4] = { 4279, 0, -1 }, /* power, irange50 */
};
static const struct coefficients adm1275_coefficients[] = {
diff --git a/drivers/hwmon/pmbus/ucd9000.c b/drivers/hwmon/pmbus/ucd9000.c
index 3e3aa95..3518f0c 100644
--- a/drivers/hwmon/pmbus/ucd9000.c
+++ b/drivers/hwmon/pmbus/ucd9000.c
@@ -21,6 +21,7 @@
#include <linux/kernel.h>
#include <linux/module.h>
+#include <linux/of_device.h>
#include <linux/init.h>
#include <linux/err.h>
#include <linux/slab.h>
@@ -119,6 +120,35 @@
};
MODULE_DEVICE_TABLE(i2c, ucd9000_id);
+static const struct of_device_id ucd9000_of_match[] = {
+ {
+ .compatible = "ti,ucd9000",
+ .data = (void *)ucd9000
+ },
+ {
+ .compatible = "ti,ucd90120",
+ .data = (void *)ucd90120
+ },
+ {
+ .compatible = "ti,ucd90124",
+ .data = (void *)ucd90124
+ },
+ {
+ .compatible = "ti,ucd90160",
+ .data = (void *)ucd90160
+ },
+ {
+ .compatible = "ti,ucd9090",
+ .data = (void *)ucd9090
+ },
+ {
+ .compatible = "ti,ucd90910",
+ .data = (void *)ucd90910
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, ucd9000_of_match);
+
static int ucd9000_probe(struct i2c_client *client,
const struct i2c_device_id *id)
{
@@ -126,6 +156,7 @@
struct ucd9000_data *data;
struct pmbus_driver_info *info;
const struct i2c_device_id *mid;
+ enum chips chip;
int i, ret;
if (!i2c_check_functionality(client->adapter,
@@ -151,7 +182,12 @@
return -ENODEV;
}
- if (id->driver_data != ucd9000 && id->driver_data != mid->driver_data)
+ if (client->dev.of_node)
+ chip = (enum chips)of_device_get_match_data(&client->dev);
+ else
+ chip = id->driver_data;
+
+ if (chip != ucd9000 && chip != mid->driver_data)
dev_notice(&client->dev,
"Device mismatch: Configured %s, detected %s\n",
id->name, mid->name);
@@ -234,6 +270,7 @@
static struct i2c_driver ucd9000_driver = {
.driver = {
.name = "ucd9000",
+ .of_match_table = of_match_ptr(ucd9000_of_match),
},
.probe = ucd9000_probe,
.remove = pmbus_do_remove,
diff --git a/drivers/hwmon/pmbus/ucd9200.c b/drivers/hwmon/pmbus/ucd9200.c
index 033d6ac..a8712c5 100644
--- a/drivers/hwmon/pmbus/ucd9200.c
+++ b/drivers/hwmon/pmbus/ucd9200.c
@@ -20,6 +20,7 @@
#include <linux/kernel.h>
#include <linux/module.h>
+#include <linux/of_device.h>
#include <linux/init.h>
#include <linux/err.h>
#include <linux/slab.h>
@@ -46,12 +47,50 @@
};
MODULE_DEVICE_TABLE(i2c, ucd9200_id);
+static const struct of_device_id ucd9200_of_match[] = {
+ {
+ .compatible = "ti,cd9200",
+ .data = (void *)ucd9200
+ },
+ {
+ .compatible = "ti,cd9220",
+ .data = (void *)ucd9220
+ },
+ {
+ .compatible = "ti,cd9222",
+ .data = (void *)ucd9222
+ },
+ {
+ .compatible = "ti,cd9224",
+ .data = (void *)ucd9224
+ },
+ {
+ .compatible = "ti,cd9240",
+ .data = (void *)ucd9240
+ },
+ {
+ .compatible = "ti,cd9244",
+ .data = (void *)ucd9244
+ },
+ {
+ .compatible = "ti,cd9246",
+ .data = (void *)ucd9246
+ },
+ {
+ .compatible = "ti,cd9248",
+ .data = (void *)ucd9248
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, ucd9200_of_match);
+
static int ucd9200_probe(struct i2c_client *client,
const struct i2c_device_id *id)
{
u8 block_buffer[I2C_SMBUS_BLOCK_MAX + 1];
struct pmbus_driver_info *info;
const struct i2c_device_id *mid;
+ enum chips chip;
int i, j, ret;
if (!i2c_check_functionality(client->adapter,
@@ -76,7 +115,13 @@
dev_err(&client->dev, "Unsupported device\n");
return -ENODEV;
}
- if (id->driver_data != ucd9200 && id->driver_data != mid->driver_data)
+
+ if (client->dev.of_node)
+ chip = (enum chips)of_device_get_match_data(&client->dev);
+ else
+ chip = id->driver_data;
+
+ if (chip != ucd9200 && chip != mid->driver_data)
dev_notice(&client->dev,
"Device mismatch: Configured %s, detected %s\n",
id->name, mid->name);
@@ -167,6 +212,7 @@
static struct i2c_driver ucd9200_driver = {
.driver = {
.name = "ucd9200",
+ .of_match_table = of_match_ptr(ucd9200_of_match),
},
.probe = ucd9200_probe,
.remove = pmbus_do_remove,
diff --git a/drivers/hwmon/stts751.c b/drivers/hwmon/stts751.c
index 5545068..d56251d 100644
--- a/drivers/hwmon/stts751.c
+++ b/drivers/hwmon/stts751.c
@@ -85,6 +85,12 @@
{ }
};
+static const struct of_device_id stts751_of_match[] = {
+ { .compatible = "stts751" },
+ { },
+};
+MODULE_DEVICE_TABLE(of, stts751_of_match);
+
struct stts751_priv {
struct device *dev;
struct i2c_client *client;
@@ -819,6 +825,7 @@
.class = I2C_CLASS_HWMON,
.driver = {
.name = DEVNAME,
+ .of_match_table = of_match_ptr(stts751_of_match),
},
.probe = stts751_probe,
.id_table = stts751_id,
diff --git a/drivers/hwmon/tmp102.c b/drivers/hwmon/tmp102.c
index 36bba2a..5eafbaa 100644
--- a/drivers/hwmon/tmp102.c
+++ b/drivers/hwmon/tmp102.c
@@ -323,8 +323,15 @@
};
MODULE_DEVICE_TABLE(i2c, tmp102_id);
+static const struct of_device_id tmp102_of_match[] = {
+ { .compatible = "ti,tmp102" },
+ { },
+};
+MODULE_DEVICE_TABLE(of, tmp102_of_match);
+
static struct i2c_driver tmp102_driver = {
.driver.name = DRIVER_NAME,
+ .driver.of_match_table = of_match_ptr(tmp102_of_match),
.driver.pm = &tmp102_dev_pm_ops,
.probe = tmp102_probe,
.id_table = tmp102_id,
diff --git a/drivers/hwmon/tmp103.c b/drivers/hwmon/tmp103.c
index ad571ec..7f85b14 100644
--- a/drivers/hwmon/tmp103.c
+++ b/drivers/hwmon/tmp103.c
@@ -150,8 +150,7 @@
return PTR_ERR_OR_ZERO(hwmon_dev);
}
-#ifdef CONFIG_PM
-static int tmp103_suspend(struct device *dev)
+static int __maybe_unused tmp103_suspend(struct device *dev)
{
struct regmap *regmap = dev_get_drvdata(dev);
@@ -159,7 +158,7 @@
TMP103_CONF_SD_MASK, 0);
}
-static int tmp103_resume(struct device *dev)
+static int __maybe_unused tmp103_resume(struct device *dev)
{
struct regmap *regmap = dev_get_drvdata(dev);
@@ -167,15 +166,7 @@
TMP103_CONF_SD_MASK, TMP103_CONF_SD);
}
-static const struct dev_pm_ops tmp103_dev_pm_ops = {
- .suspend = tmp103_suspend,
- .resume = tmp103_resume,
-};
-
-#define TMP103_DEV_PM_OPS (&tmp103_dev_pm_ops)
-#else
-#define TMP103_DEV_PM_OPS NULL
-#endif /* CONFIG_PM */
+static SIMPLE_DEV_PM_OPS(tmp103_dev_pm_ops, tmp103_suspend, tmp103_resume);
static const struct i2c_device_id tmp103_id[] = {
{ "tmp103", 0 },
@@ -183,10 +174,17 @@
};
MODULE_DEVICE_TABLE(i2c, tmp103_id);
+static const struct of_device_id tmp103_of_match[] = {
+ { .compatible = "ti,tmp103" },
+ { },
+};
+MODULE_DEVICE_TABLE(of, tmp103_of_match);
+
static struct i2c_driver tmp103_driver = {
.driver = {
.name = "tmp103",
- .pm = TMP103_DEV_PM_OPS,
+ .of_match_table = of_match_ptr(tmp103_of_match),
+ .pm = &tmp103_dev_pm_ops,
},
.probe = tmp103_probe,
.id_table = tmp103_id,
diff --git a/drivers/hwmon/tmp421.c b/drivers/hwmon/tmp421.c
index bfb98b9..e363992 100644
--- a/drivers/hwmon/tmp421.c
+++ b/drivers/hwmon/tmp421.c
@@ -29,6 +29,7 @@
#include <linux/hwmon-sysfs.h>
#include <linux/err.h>
#include <linux/mutex.h>
+#include <linux/of_device.h>
#include <linux/sysfs.h>
/* Addresses to scan */
@@ -69,6 +70,31 @@
};
MODULE_DEVICE_TABLE(i2c, tmp421_id);
+static const struct of_device_id tmp421_of_match[] = {
+ {
+ .compatible = "ti,tmp421",
+ .data = (void *)2
+ },
+ {
+ .compatible = "ti,tmp422",
+ .data = (void *)3
+ },
+ {
+ .compatible = "ti,tmp423",
+ .data = (void *)4
+ },
+ {
+ .compatible = "ti,tmp441",
+ .data = (void *)2
+ },
+ {
+ .compatible = "ti,tmp422",
+ .data = (void *)3
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, tmp421_of_match);
+
struct tmp421_data {
struct i2c_client *client;
struct mutex update_lock;
@@ -78,7 +104,7 @@
struct hwmon_chip_info chip;
char valid;
unsigned long last_updated;
- int channels;
+ unsigned long channels;
u8 config;
s16 temp[4];
};
@@ -272,7 +298,11 @@
return -ENOMEM;
mutex_init(&data->update_lock);
- data->channels = id->driver_data;
+ if (client->dev.of_node)
+ data->channels = (unsigned long)
+ of_device_get_match_data(&client->dev);
+ else
+ data->channels = id->driver_data;
data->client = client;
err = tmp421_init_client(client);
@@ -301,6 +331,7 @@
.class = I2C_CLASS_HWMON,
.driver = {
.name = "tmp421",
+ .of_match_table = of_match_ptr(tmp421_of_match),
},
.probe = tmp421_probe,
.id_table = tmp421_id,
diff --git a/drivers/hwmon/twl4030-madc-hwmon.c b/drivers/hwmon/twl4030-madc-hwmon.c
deleted file mode 100644
index b5caf7f..0000000
--- a/drivers/hwmon/twl4030-madc-hwmon.c
+++ /dev/null
@@ -1,118 +0,0 @@
-/*
- *
- * TWL4030 MADC Hwmon driver-This driver monitors the real time
- * conversion of analog signals like battery temperature,
- * battery type, battery level etc. User can ask for the conversion on a
- * particular channel using the sysfs nodes.
- *
- * Copyright (C) 2011 Texas Instruments Incorporated - http://www.ti.com/
- * J Keerthy <j-keerthy@ti.com>
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
- * 02110-1301 USA
- *
- */
-#include <linux/init.h>
-#include <linux/module.h>
-#include <linux/kernel.h>
-#include <linux/i2c/twl.h>
-#include <linux/device.h>
-#include <linux/platform_device.h>
-#include <linux/i2c/twl4030-madc.h>
-#include <linux/hwmon.h>
-#include <linux/hwmon-sysfs.h>
-#include <linux/stddef.h>
-#include <linux/sysfs.h>
-#include <linux/err.h>
-#include <linux/types.h>
-
-/*
- * sysfs hook function
- */
-static ssize_t madc_read(struct device *dev,
- struct device_attribute *devattr, char *buf)
-{
- struct sensor_device_attribute *attr = to_sensor_dev_attr(devattr);
- struct twl4030_madc_request req = {
- .channels = 1 << attr->index,
- .method = TWL4030_MADC_SW2,
- .type = TWL4030_MADC_WAIT,
- };
- long val;
-
- val = twl4030_madc_conversion(&req);
- if (val < 0)
- return val;
-
- return sprintf(buf, "%d\n", req.rbuf[attr->index]);
-}
-
-/* sysfs nodes to read individual channels from user side */
-static SENSOR_DEVICE_ATTR(in0_input, S_IRUGO, madc_read, NULL, 0);
-static SENSOR_DEVICE_ATTR(temp1_input, S_IRUGO, madc_read, NULL, 1);
-static SENSOR_DEVICE_ATTR(in2_input, S_IRUGO, madc_read, NULL, 2);
-static SENSOR_DEVICE_ATTR(in3_input, S_IRUGO, madc_read, NULL, 3);
-static SENSOR_DEVICE_ATTR(in4_input, S_IRUGO, madc_read, NULL, 4);
-static SENSOR_DEVICE_ATTR(in5_input, S_IRUGO, madc_read, NULL, 5);
-static SENSOR_DEVICE_ATTR(in6_input, S_IRUGO, madc_read, NULL, 6);
-static SENSOR_DEVICE_ATTR(in7_input, S_IRUGO, madc_read, NULL, 7);
-static SENSOR_DEVICE_ATTR(in8_input, S_IRUGO, madc_read, NULL, 8);
-static SENSOR_DEVICE_ATTR(in9_input, S_IRUGO, madc_read, NULL, 9);
-static SENSOR_DEVICE_ATTR(curr10_input, S_IRUGO, madc_read, NULL, 10);
-static SENSOR_DEVICE_ATTR(in11_input, S_IRUGO, madc_read, NULL, 11);
-static SENSOR_DEVICE_ATTR(in12_input, S_IRUGO, madc_read, NULL, 12);
-static SENSOR_DEVICE_ATTR(in15_input, S_IRUGO, madc_read, NULL, 15);
-
-static struct attribute *twl4030_madc_attrs[] = {
- &sensor_dev_attr_in0_input.dev_attr.attr,
- &sensor_dev_attr_temp1_input.dev_attr.attr,
- &sensor_dev_attr_in2_input.dev_attr.attr,
- &sensor_dev_attr_in3_input.dev_attr.attr,
- &sensor_dev_attr_in4_input.dev_attr.attr,
- &sensor_dev_attr_in5_input.dev_attr.attr,
- &sensor_dev_attr_in6_input.dev_attr.attr,
- &sensor_dev_attr_in7_input.dev_attr.attr,
- &sensor_dev_attr_in8_input.dev_attr.attr,
- &sensor_dev_attr_in9_input.dev_attr.attr,
- &sensor_dev_attr_curr10_input.dev_attr.attr,
- &sensor_dev_attr_in11_input.dev_attr.attr,
- &sensor_dev_attr_in12_input.dev_attr.attr,
- &sensor_dev_attr_in15_input.dev_attr.attr,
- NULL
-};
-ATTRIBUTE_GROUPS(twl4030_madc);
-
-static int twl4030_madc_hwmon_probe(struct platform_device *pdev)
-{
- struct device *hwmon;
-
- hwmon = devm_hwmon_device_register_with_groups(&pdev->dev,
- "twl4030_madc", NULL,
- twl4030_madc_groups);
- return PTR_ERR_OR_ZERO(hwmon);
-}
-
-static struct platform_driver twl4030_madc_hwmon_driver = {
- .probe = twl4030_madc_hwmon_probe,
- .driver = {
- .name = "twl4030_madc_hwmon",
- },
-};
-
-module_platform_driver(twl4030_madc_hwmon_driver);
-
-MODULE_DESCRIPTION("TWL4030 ADC Hwmon driver");
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR("J Keerthy");
-MODULE_ALIAS("platform:twl4030_madc_hwmon");
diff --git a/drivers/hwmon/w83627ehf.c b/drivers/hwmon/w83627ehf.c
index ab346ed..ad68b6d 100644
--- a/drivers/hwmon/w83627ehf.c
+++ b/drivers/hwmon/w83627ehf.c
@@ -135,11 +135,16 @@
outb(ld, ioreg + 1);
}
-static inline void
+static inline int
superio_enter(int ioreg)
{
+ if (!request_muxed_region(ioreg, 2, DRVNAME))
+ return -EBUSY;
+
outb(0x87, ioreg);
outb(0x87, ioreg);
+
+ return 0;
}
static inline void
@@ -148,6 +153,7 @@
outb(0xaa, ioreg);
outb(0x02, ioreg);
outb(0x02, ioreg + 1);
+ release_region(ioreg, 2);
}
/*
@@ -1970,8 +1976,6 @@
return;
}
- superio_enter(sio_data->sioreg);
-
/* fan4 and fan5 share some pins with the GPIO and serial flash */
if (sio_data->kind == nct6775) {
/* On NCT6775, fan4 shares pins with the fdc interface */
@@ -2013,8 +2017,6 @@
fan4min = fan4pin;
}
- superio_exit(sio_data->sioreg);
-
data->has_fan = data->has_fan_min = 0x03; /* fan1 and fan2 */
data->has_fan |= (fan3pin << 2);
data->has_fan_min |= (fan3pin << 2);
@@ -2352,7 +2354,11 @@
w83627ehf_init_device(data, sio_data->kind);
data->vrm = vid_which_vrm();
- superio_enter(sio_data->sioreg);
+
+ err = superio_enter(sio_data->sioreg);
+ if (err)
+ goto exit_release;
+
/* Read VID value */
if (sio_data->kind == w83667hg || sio_data->kind == w83667hg_b ||
sio_data->kind == nct6775 || sio_data->kind == nct6776) {
@@ -2364,8 +2370,10 @@
superio_select(sio_data->sioreg, W83667HG_LD_VID);
data->vid = superio_inb(sio_data->sioreg, 0xe3);
err = device_create_file(dev, &dev_attr_cpu0_vid);
- if (err)
+ if (err) {
+ superio_exit(sio_data->sioreg);
goto exit_release;
+ }
} else if (sio_data->kind != w83627uhg) {
superio_select(sio_data->sioreg, W83627EHF_LD_HWM);
if (superio_inb(sio_data->sioreg, SIO_REG_VID_CTRL) & 0x80) {
@@ -2401,8 +2409,10 @@
data->vid &= 0x3f;
err = device_create_file(dev, &dev_attr_cpu0_vid);
- if (err)
+ if (err) {
+ superio_exit(sio_data->sioreg);
goto exit_release;
+ }
} else {
dev_info(dev,
"VID pins in output mode, CPU VID not available\n");
@@ -2424,10 +2434,10 @@
pr_info("Enabled fan debounce for chip %s\n", data->name);
}
- superio_exit(sio_data->sioreg);
-
w83627ehf_check_fan_inputs(sio_data, data);
+ superio_exit(sio_data->sioreg);
+
/* Read fan clock dividers immediately */
w83627ehf_update_fan_div_common(dev, data);
@@ -2712,8 +2722,11 @@
u16 val;
const char *sio_name;
+ int err;
- superio_enter(sioaddr);
+ err = superio_enter(sioaddr);
+ if (err)
+ return err;
if (force_id)
val = force_id;
diff --git a/drivers/hwtracing/coresight/coresight-etb10.c b/drivers/hwtracing/coresight/coresight-etb10.c
index d7325c65..979ea6e 100644
--- a/drivers/hwtracing/coresight/coresight-etb10.c
+++ b/drivers/hwtracing/coresight/coresight-etb10.c
@@ -321,7 +321,7 @@
static unsigned long etb_reset_buffer(struct coresight_device *csdev,
struct perf_output_handle *handle,
- void *sink_config, bool *lost)
+ void *sink_config)
{
unsigned long size = 0;
struct cs_buffers *buf = sink_config;
@@ -343,7 +343,6 @@
* resetting parameters here and squaring off with the ring
* buffer API in the tracer PMU is fine.
*/
- *lost = !!local_xchg(&buf->lost, 0);
size = local_xchg(&buf->data_size, 0);
}
@@ -385,7 +384,7 @@
(unsigned long)write_ptr);
write_ptr &= ~(ETB_FRAME_SIZE_WORDS - 1);
- local_inc(&buf->lost);
+ perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED);
}
/*
@@ -396,7 +395,7 @@
*/
status = readl_relaxed(drvdata->base + ETB_STATUS_REG);
if (status & ETB_STATUS_RAM_FULL) {
- local_inc(&buf->lost);
+ perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED);
to_read = capacity;
read_ptr = write_ptr;
} else {
@@ -429,7 +428,7 @@
if (read_ptr > (drvdata->buffer_depth - 1))
read_ptr -= drvdata->buffer_depth;
/* let the decoder know we've skipped ahead */
- local_inc(&buf->lost);
+ perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED);
}
/* finally tell HW where we want to start reading from */
diff --git a/drivers/hwtracing/coresight/coresight-etm-perf.c b/drivers/hwtracing/coresight/coresight-etm-perf.c
index 26cfac3..288a423 100644
--- a/drivers/hwtracing/coresight/coresight-etm-perf.c
+++ b/drivers/hwtracing/coresight/coresight-etm-perf.c
@@ -302,7 +302,8 @@
return;
fail_end_stop:
- perf_aux_output_end(handle, 0, true);
+ perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED);
+ perf_aux_output_end(handle, 0);
fail:
event->hw.state = PERF_HES_STOPPED;
goto out;
@@ -310,7 +311,6 @@
static void etm_event_stop(struct perf_event *event, int mode)
{
- bool lost;
int cpu = smp_processor_id();
unsigned long size;
struct coresight_device *sink, *csdev = per_cpu(csdev_src, cpu);
@@ -348,10 +348,9 @@
return;
size = sink_ops(sink)->reset_buffer(sink, handle,
- event_data->snk_config,
- &lost);
+ event_data->snk_config);
- perf_aux_output_end(handle, size, lost);
+ perf_aux_output_end(handle, size);
}
/* Disabling the path make its elements available to other sessions */
diff --git a/drivers/hwtracing/coresight/coresight-priv.h b/drivers/hwtracing/coresight/coresight-priv.h
index ef9d8e9..5f662d8 100644
--- a/drivers/hwtracing/coresight/coresight-priv.h
+++ b/drivers/hwtracing/coresight/coresight-priv.h
@@ -76,7 +76,6 @@
* @nr_pages: max number of pages granted to us
* @offset: offset within the current buffer
* @data_size: how much we collected in this run
- * @lost: other than zero if we had a HW buffer wrap around
* @snapshot: is this run in snapshot mode
* @data_pages: a handle the ring buffer
*/
@@ -85,7 +84,6 @@
unsigned int nr_pages;
unsigned long offset;
local_t data_size;
- local_t lost;
bool snapshot;
void **data_pages;
};
diff --git a/drivers/hwtracing/coresight/coresight-tmc-etf.c b/drivers/hwtracing/coresight/coresight-tmc-etf.c
index 1549436..aec61a6 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etf.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etf.c
@@ -329,7 +329,7 @@
static unsigned long tmc_reset_etf_buffer(struct coresight_device *csdev,
struct perf_output_handle *handle,
- void *sink_config, bool *lost)
+ void *sink_config)
{
long size = 0;
struct cs_buffers *buf = sink_config;
@@ -350,7 +350,6 @@
* resetting parameters here and squaring off with the ring
* buffer API in the tracer PMU is fine.
*/
- *lost = !!local_xchg(&buf->lost, 0);
size = local_xchg(&buf->data_size, 0);
}
@@ -389,7 +388,7 @@
*/
status = readl_relaxed(drvdata->base + TMC_STS);
if (status & TMC_STS_FULL) {
- local_inc(&buf->lost);
+ perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED);
to_read = drvdata->size;
} else {
to_read = CIRC_CNT(write_ptr, read_ptr, drvdata->size);
@@ -434,7 +433,7 @@
read_ptr -= drvdata->size;
/* Tell the HW */
writel_relaxed(read_ptr, drvdata->base + TMC_RRP);
- local_inc(&buf->lost);
+ perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED);
}
cur = buf->cur;
diff --git a/drivers/ide/ide-atapi.c b/drivers/ide/ide-atapi.c
index feb3006..5901937 100644
--- a/drivers/ide/ide-atapi.c
+++ b/drivers/ide/ide-atapi.c
@@ -107,7 +107,8 @@
memcpy(scsi_req(rq)->cmd, pc->c, 12);
if (drive->media == ide_tape)
scsi_req(rq)->cmd[13] = REQ_IDETAPE_PC1;
- error = blk_execute_rq(drive->queue, disk, rq, 0);
+ blk_execute_rq(drive->queue, disk, rq, 0);
+ error = scsi_req(rq)->result ? -EIO : 0;
put_req:
blk_put_request(rq);
return error;
@@ -454,7 +455,7 @@
debug_log("%s: I/O error\n", drive->name);
if (drive->media != ide_tape)
- pc->rq->errors++;
+ scsi_req(pc->rq)->result++;
if (scsi_req(rq)->cmd[0] == REQUEST_SENSE) {
printk(KERN_ERR PFX "%s: I/O error in request "
@@ -488,13 +489,13 @@
drive->failed_pc = NULL;
if (ata_misc_request(rq)) {
- rq->errors = 0;
+ scsi_req(rq)->result = 0;
error = 0;
} else {
if (blk_rq_is_passthrough(rq) && uptodate <= 0) {
- if (rq->errors == 0)
- rq->errors = -EIO;
+ if (scsi_req(rq)->result == 0)
+ scsi_req(rq)->result = -EIO;
}
error = uptodate ? 0 : -EIO;
diff --git a/drivers/ide/ide-cd.c b/drivers/ide/ide-cd.c
index 74f1b7d..07e5ff3 100644
--- a/drivers/ide/ide-cd.c
+++ b/drivers/ide/ide-cd.c
@@ -247,10 +247,10 @@
struct cdrom_info *info = drive->driver_data;
- if (!rq->errors)
+ if (!scsi_req(rq)->result)
info->write_timeout = jiffies + ATAPI_WAIT_WRITE_BUSY;
- rq->errors = 1;
+ scsi_req(rq)->result = 1;
if (time_after(jiffies, info->write_timeout))
return 0;
@@ -294,8 +294,8 @@
}
/* if we have an error, pass CHECK_CONDITION as the SCSI status byte */
- if (blk_rq_is_scsi(rq) && !rq->errors)
- rq->errors = SAM_STAT_CHECK_CONDITION;
+ if (blk_rq_is_scsi(rq) && !scsi_req(rq)->result)
+ scsi_req(rq)->result = SAM_STAT_CHECK_CONDITION;
if (blk_noretry_request(rq))
do_end_request = 1;
@@ -325,7 +325,7 @@
* Arrange to retry the request but be sure to give up if we've
* retried too many times.
*/
- if (++rq->errors > ERROR_MAX)
+ if (++scsi_req(rq)->result > ERROR_MAX)
do_end_request = 1;
break;
case ILLEGAL_REQUEST:
@@ -372,7 +372,7 @@
/* go to the default handler for other errors */
ide_error(drive, "cdrom_decode_status", stat);
return 1;
- } else if (++rq->errors > ERROR_MAX)
+ } else if (++scsi_req(rq)->result > ERROR_MAX)
/* we've racked up too many retries, abort */
do_end_request = 1;
}
@@ -452,7 +452,8 @@
}
}
- error = blk_execute_rq(drive->queue, info->disk, rq, 0);
+ blk_execute_rq(drive->queue, info->disk, rq, 0);
+ error = scsi_req(rq)->result ? -EIO : 0;
if (buffer)
*bufflen = scsi_req(rq)->resid_len;
@@ -683,8 +684,8 @@
if (cmd->nleft == 0)
uptodate = 1;
} else {
- if (uptodate <= 0 && rq->errors == 0)
- rq->errors = -EIO;
+ if (uptodate <= 0 && scsi_req(rq)->result == 0)
+ scsi_req(rq)->result = -EIO;
}
if (uptodate == 0 && rq->bio)
@@ -1379,7 +1380,7 @@
* appropriate action
*/
if (c[0] == MODE_SENSE || c[0] == MODE_SELECT) {
- rq->errors = ILLEGAL_REQUEST;
+ scsi_req(rq)->result = ILLEGAL_REQUEST;
return BLKPREP_KILL;
}
diff --git a/drivers/ide/ide-cd_ioctl.c b/drivers/ide/ide-cd_ioctl.c
index 9fcefbc..55cd736 100644
--- a/drivers/ide/ide-cd_ioctl.c
+++ b/drivers/ide/ide-cd_ioctl.c
@@ -307,7 +307,8 @@
scsi_req_init(rq);
ide_req(rq)->type = ATA_PRIV_MISC;
rq->rq_flags = RQF_QUIET;
- ret = blk_execute_rq(drive->queue, cd->disk, rq, 0);
+ blk_execute_rq(drive->queue, cd->disk, rq, 0);
+ ret = scsi_req(rq)->result ? -EIO : 0;
blk_put_request(rq);
/*
* A reset will unlock the door. If it was previously locked,
diff --git a/drivers/ide/ide-devsets.c b/drivers/ide/ide-devsets.c
index a45dda5..9b69c32 100644
--- a/drivers/ide/ide-devsets.c
+++ b/drivers/ide/ide-devsets.c
@@ -173,8 +173,8 @@
*(int *)&scsi_req(rq)->cmd[1] = arg;
rq->special = setting->set;
- if (blk_execute_rq(q, NULL, rq, 0))
- ret = rq->errors;
+ blk_execute_rq(q, NULL, rq, 0);
+ ret = scsi_req(rq)->result;
blk_put_request(rq);
return ret;
@@ -186,7 +186,7 @@
err = setfunc(drive, *(int *)&scsi_req(rq)->cmd[1]);
if (err)
- rq->errors = err;
- ide_complete_rq(drive, err, blk_rq_bytes(rq));
+ scsi_req(rq)->result = err;
+ ide_complete_rq(drive, 0, blk_rq_bytes(rq));
return ide_stopped;
}
diff --git a/drivers/ide/ide-disk.c b/drivers/ide/ide-disk.c
index 1861597..7c06237 100644
--- a/drivers/ide/ide-disk.c
+++ b/drivers/ide/ide-disk.c
@@ -470,7 +470,6 @@
static int set_multcount(ide_drive_t *drive, int arg)
{
struct request *rq;
- int error;
if (arg < 0 || arg > (drive->id[ATA_ID_MAX_MULTSECT] & 0xff))
return -EINVAL;
@@ -484,7 +483,7 @@
drive->mult_req = arg;
drive->special_flags |= IDE_SFLAG_SET_MULTMODE;
- error = blk_execute_rq(drive->queue, NULL, rq, 0);
+ blk_execute_rq(drive->queue, NULL, rq, 0);
blk_put_request(rq);
return (drive->mult_count == arg) ? 0 : -EIO;
diff --git a/drivers/ide/ide-dma.c b/drivers/ide/ide-dma.c
index 17a65ac..51c8122 100644
--- a/drivers/ide/ide-dma.c
+++ b/drivers/ide/ide-dma.c
@@ -490,7 +490,7 @@
* make sure request is sane
*/
if (hwif->rq)
- hwif->rq->errors = 0;
+ scsi_req(hwif->rq)->result = 0;
return ret;
}
diff --git a/drivers/ide/ide-eh.c b/drivers/ide/ide-eh.c
index cf3af68..4b7ffd7 100644
--- a/drivers/ide/ide-eh.c
+++ b/drivers/ide/ide-eh.c
@@ -12,7 +12,7 @@
if ((stat & ATA_BUSY) ||
((stat & ATA_DF) && (drive->dev_flags & IDE_DFLAG_NOWERR) == 0)) {
/* other bits are useless when BUSY */
- rq->errors |= ERROR_RESET;
+ scsi_req(rq)->result |= ERROR_RESET;
} else if (stat & ATA_ERR) {
/* err has different meaning on cdrom and tape */
if (err == ATA_ABORTED) {
@@ -25,10 +25,10 @@
drive->crc_count++;
} else if (err & (ATA_BBK | ATA_UNC)) {
/* retries won't help these */
- rq->errors = ERROR_MAX;
+ scsi_req(rq)->result = ERROR_MAX;
} else if (err & ATA_TRK0NF) {
/* help it find track zero */
- rq->errors |= ERROR_RECAL;
+ scsi_req(rq)->result |= ERROR_RECAL;
}
}
@@ -39,23 +39,23 @@
ide_pad_transfer(drive, READ, nsect * SECTOR_SIZE);
}
- if (rq->errors >= ERROR_MAX || blk_noretry_request(rq)) {
+ if (scsi_req(rq)->result >= ERROR_MAX || blk_noretry_request(rq)) {
ide_kill_rq(drive, rq);
return ide_stopped;
}
if (hwif->tp_ops->read_status(hwif) & (ATA_BUSY | ATA_DRQ))
- rq->errors |= ERROR_RESET;
+ scsi_req(rq)->result |= ERROR_RESET;
- if ((rq->errors & ERROR_RESET) == ERROR_RESET) {
- ++rq->errors;
+ if ((scsi_req(rq)->result & ERROR_RESET) == ERROR_RESET) {
+ ++scsi_req(rq)->result;
return ide_do_reset(drive);
}
- if ((rq->errors & ERROR_RECAL) == ERROR_RECAL)
+ if ((scsi_req(rq)->result & ERROR_RECAL) == ERROR_RECAL)
drive->special_flags |= IDE_SFLAG_RECALIBRATE;
- ++rq->errors;
+ ++scsi_req(rq)->result;
return ide_stopped;
}
@@ -68,7 +68,7 @@
if ((stat & ATA_BUSY) ||
((stat & ATA_DF) && (drive->dev_flags & IDE_DFLAG_NOWERR) == 0)) {
/* other bits are useless when BUSY */
- rq->errors |= ERROR_RESET;
+ scsi_req(rq)->result |= ERROR_RESET;
} else {
/* add decoding error stuff */
}
@@ -77,14 +77,14 @@
/* force an abort */
hwif->tp_ops->exec_command(hwif, ATA_CMD_IDLEIMMEDIATE);
- if (rq->errors >= ERROR_MAX) {
+ if (scsi_req(rq)->result >= ERROR_MAX) {
ide_kill_rq(drive, rq);
} else {
- if ((rq->errors & ERROR_RESET) == ERROR_RESET) {
- ++rq->errors;
+ if ((scsi_req(rq)->result & ERROR_RESET) == ERROR_RESET) {
+ ++scsi_req(rq)->result;
return ide_do_reset(drive);
}
- ++rq->errors;
+ ++scsi_req(rq)->result;
}
return ide_stopped;
@@ -130,11 +130,11 @@
if (cmd)
ide_complete_cmd(drive, cmd, stat, err);
} else if (ata_pm_request(rq)) {
- rq->errors = 1;
+ scsi_req(rq)->result = 1;
ide_complete_pm_rq(drive, rq);
return ide_stopped;
}
- rq->errors = err;
+ scsi_req(rq)->result = err;
ide_complete_rq(drive, err ? -EIO : 0, blk_rq_bytes(rq));
return ide_stopped;
}
@@ -149,8 +149,8 @@
if (rq && ata_misc_request(rq) &&
scsi_req(rq)->cmd[0] == REQ_DRIVE_RESET) {
- if (err <= 0 && rq->errors == 0)
- rq->errors = -EIO;
+ if (err <= 0 && scsi_req(rq)->result == 0)
+ scsi_req(rq)->result = -EIO;
ide_complete_rq(drive, err ? err : 0, blk_rq_bytes(rq));
}
}
diff --git a/drivers/ide/ide-floppy.c b/drivers/ide/ide-floppy.c
index a69e801..8ac6048 100644
--- a/drivers/ide/ide-floppy.c
+++ b/drivers/ide/ide-floppy.c
@@ -98,7 +98,7 @@
}
if (ata_misc_request(rq))
- rq->errors = uptodate ? 0 : IDE_DRV_ERROR_GENERAL;
+ scsi_req(rq)->result = uptodate ? 0 : IDE_DRV_ERROR_GENERAL;
return uptodate;
}
@@ -239,7 +239,7 @@
? rq->rq_disk->disk_name
: "dev?"));
- if (rq->errors >= ERROR_MAX) {
+ if (scsi_req(rq)->result >= ERROR_MAX) {
if (drive->failed_pc) {
ide_floppy_report_error(floppy, drive->failed_pc);
drive->failed_pc = NULL;
@@ -247,7 +247,7 @@
printk(KERN_ERR PFX "%s: I/O error\n", drive->name);
if (ata_misc_request(rq)) {
- rq->errors = 0;
+ scsi_req(rq)->result = 0;
ide_complete_rq(drive, 0, blk_rq_bytes(rq));
return ide_stopped;
} else
@@ -301,8 +301,8 @@
return ide_floppy_issue_pc(drive, &cmd, pc);
out_end:
drive->failed_pc = NULL;
- if (blk_rq_is_passthrough(rq) && rq->errors == 0)
- rq->errors = -EIO;
+ if (blk_rq_is_passthrough(rq) && scsi_req(rq)->result == 0)
+ scsi_req(rq)->result = -EIO;
ide_complete_rq(drive, -EIO, blk_rq_bytes(rq));
return ide_stopped;
}
diff --git a/drivers/ide/ide-io.c b/drivers/ide/ide-io.c
index 043b1fb9..45b3f41 100644
--- a/drivers/ide/ide-io.c
+++ b/drivers/ide/ide-io.c
@@ -141,12 +141,12 @@
drive->failed_pc = NULL;
if ((media == ide_floppy || media == ide_tape) && drv_req) {
- rq->errors = 0;
+ scsi_req(rq)->result = 0;
} else {
if (media == ide_tape)
- rq->errors = IDE_DRV_ERROR_GENERAL;
- else if (blk_rq_is_passthrough(rq) && rq->errors == 0)
- rq->errors = -EIO;
+ scsi_req(rq)->result = IDE_DRV_ERROR_GENERAL;
+ else if (blk_rq_is_passthrough(rq) && scsi_req(rq)->result == 0)
+ scsi_req(rq)->result = -EIO;
}
ide_complete_rq(drive, -EIO, blk_rq_bytes(rq));
@@ -271,7 +271,7 @@
#ifdef DEBUG
printk("%s: DRIVE_CMD (null)\n", drive->name);
#endif
- rq->errors = 0;
+ scsi_req(rq)->result = 0;
ide_complete_rq(drive, 0, blk_rq_bytes(rq));
return ide_stopped;
diff --git a/drivers/ide/ide-ioctls.c b/drivers/ide/ide-ioctls.c
index 248a3e0..8c0d172 100644
--- a/drivers/ide/ide-ioctls.c
+++ b/drivers/ide/ide-ioctls.c
@@ -128,7 +128,8 @@
rq = blk_get_request(drive->queue, REQ_OP_DRV_IN, __GFP_RECLAIM);
scsi_req_init(rq);
ide_req(rq)->type = ATA_PRIV_TASKFILE;
- err = blk_execute_rq(drive->queue, NULL, rq, 0);
+ blk_execute_rq(drive->queue, NULL, rq, 0);
+ err = scsi_req(rq)->result ? -EIO : 0;
blk_put_request(rq);
return err;
@@ -227,8 +228,8 @@
ide_req(rq)->type = ATA_PRIV_MISC;
scsi_req(rq)->cmd_len = 1;
scsi_req(rq)->cmd[0] = REQ_DRIVE_RESET;
- if (blk_execute_rq(drive->queue, NULL, rq, 1))
- ret = rq->errors;
+ blk_execute_rq(drive->queue, NULL, rq, 1);
+ ret = scsi_req(rq)->result;
blk_put_request(rq);
return ret;
}
diff --git a/drivers/ide/ide-park.c b/drivers/ide/ide-park.c
index 101aed9..94e3107 100644
--- a/drivers/ide/ide-park.c
+++ b/drivers/ide/ide-park.c
@@ -37,7 +37,8 @@
scsi_req(rq)->cmd_len = 1;
ide_req(rq)->type = ATA_PRIV_MISC;
rq->special = &timeout;
- rc = blk_execute_rq(q, NULL, rq, 1);
+ blk_execute_rq(q, NULL, rq, 1);
+ rc = scsi_req(rq)->result ? -EIO : 0;
blk_put_request(rq);
if (rc)
goto out;
diff --git a/drivers/ide/ide-pm.c b/drivers/ide/ide-pm.c
index ec951be..0977fc1 100644
--- a/drivers/ide/ide-pm.c
+++ b/drivers/ide/ide-pm.c
@@ -27,7 +27,8 @@
mesg.event = PM_EVENT_FREEZE;
rqpm.pm_state = mesg.event;
- ret = blk_execute_rq(drive->queue, NULL, rq, 0);
+ blk_execute_rq(drive->queue, NULL, rq, 0);
+ ret = scsi_req(rq)->result ? -EIO : 0;
blk_put_request(rq);
if (ret == 0 && ide_port_acpi(hwif)) {
@@ -55,8 +56,8 @@
spin_lock_irq(q->queue_lock);
if (unlikely(blk_queue_dying(q))) {
rq->rq_flags |= RQF_QUIET;
- rq->errors = -ENXIO;
- __blk_end_request_all(rq, rq->errors);
+ scsi_req(rq)->result = -ENXIO;
+ __blk_end_request_all(rq, 0);
spin_unlock_irq(q->queue_lock);
return -ENXIO;
}
@@ -66,7 +67,7 @@
wait_for_completion_io(&wait);
- return rq->errors ? -EIO : 0;
+ return scsi_req(rq)->result ? -EIO : 0;
}
int generic_ide_resume(struct device *dev)
diff --git a/drivers/ide/ide-tape.c b/drivers/ide/ide-tape.c
index d8a552b..a0651f9 100644
--- a/drivers/ide/ide-tape.c
+++ b/drivers/ide/ide-tape.c
@@ -366,7 +366,7 @@
err = pc->error;
}
}
- rq->errors = err;
+ scsi_req(rq)->result = err;
return uptodate;
}
@@ -879,7 +879,7 @@
tape->valid = 0;
ret = size;
- if (rq->errors == IDE_DRV_ERROR_GENERAL)
+ if (scsi_req(rq)->result == IDE_DRV_ERROR_GENERAL)
ret = -EIO;
out_put:
blk_put_request(rq);
diff --git a/drivers/ide/ide-taskfile.c b/drivers/ide/ide-taskfile.c
index 4c0007c..d71199d 100644
--- a/drivers/ide/ide-taskfile.c
+++ b/drivers/ide/ide-taskfile.c
@@ -287,7 +287,7 @@
u8 saved_io_32bit = drive->io_32bit;
if (cmd->tf_flags & IDE_TFLAG_FS)
- cmd->rq->errors = 0;
+ scsi_req(cmd->rq)->result = 0;
if (cmd->tf_flags & IDE_TFLAG_IO_16BIT)
drive->io_32bit = 0;
@@ -329,7 +329,7 @@
u8 set_xfer = !!(cmd->tf_flags & IDE_TFLAG_SET_XFER);
ide_complete_cmd(drive, cmd, stat, err);
- rq->errors = err;
+ scsi_req(rq)->result = err;
if (err == 0 && set_xfer) {
ide_set_xfer_rate(drive, nsect);
@@ -452,8 +452,8 @@
rq->special = cmd;
cmd->rq = rq;
- error = blk_execute_rq(drive->queue, NULL, rq, 0);
-
+ blk_execute_rq(drive->queue, NULL, rq, 0);
+ error = scsi_req(rq)->result ? -EIO : 0;
put_req:
blk_put_request(rq);
return error;
diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c
index 0f58f46..329d08c 100644
--- a/drivers/infiniband/core/addr.c
+++ b/drivers/infiniband/core/addr.c
@@ -88,7 +88,7 @@
return false;
ret = nla_parse(tb, LS_NLA_TYPE_MAX - 1, nlmsg_data(nlh),
- nlmsg_len(nlh), ib_nl_addr_policy);
+ nlmsg_len(nlh), ib_nl_addr_policy, NULL);
if (ret)
return false;
diff --git a/drivers/infiniband/core/iwpm_util.c b/drivers/infiniband/core/iwpm_util.c
index 3ef51a9..f13870e 100644
--- a/drivers/infiniband/core/iwpm_util.c
+++ b/drivers/infiniband/core/iwpm_util.c
@@ -472,12 +472,14 @@
int ret;
const char *err_str = "";
- ret = nlmsg_validate(cb->nlh, nlh_len, policy_max-1, nlmsg_policy);
+ ret = nlmsg_validate(cb->nlh, nlh_len, policy_max - 1, nlmsg_policy,
+ NULL);
if (ret) {
err_str = "Invalid attribute";
goto parse_nlmsg_error;
}
- ret = nlmsg_parse(cb->nlh, nlh_len, nltb, policy_max-1, nlmsg_policy);
+ ret = nlmsg_parse(cb->nlh, nlh_len, nltb, policy_max - 1,
+ nlmsg_policy, NULL);
if (ret) {
err_str = "Unable to parse the nlmsg";
goto parse_nlmsg_error;
diff --git a/drivers/infiniband/core/netlink.c b/drivers/infiniband/core/netlink.c
index 10469b0..b784055 100644
--- a/drivers/infiniband/core/netlink.c
+++ b/drivers/infiniband/core/netlink.c
@@ -146,7 +146,8 @@
}
EXPORT_SYMBOL(ibnl_put_attr);
-static int ibnl_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int ibnl_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct ibnl_client *client;
int type = nlh->nlmsg_type;
@@ -209,7 +210,7 @@
if (nlh->nlmsg_flags & NLM_F_REQUEST)
return;
- ibnl_rcv_msg(skb, nlh);
+ ibnl_rcv_msg(skb, nlh, NULL);
msglen = NLMSG_ALIGN(nlh->nlmsg_len);
if (msglen > skb->len)
diff --git a/drivers/infiniband/core/sa_query.c b/drivers/infiniband/core/sa_query.c
index 81b742c..ceae153 100644
--- a/drivers/infiniband/core/sa_query.c
+++ b/drivers/infiniband/core/sa_query.c
@@ -808,7 +808,7 @@
return -EPERM;
ret = nla_parse(tb, LS_NLA_TYPE_MAX - 1, nlmsg_data(nlh),
- nlmsg_len(nlh), ib_nl_policy);
+ nlmsg_len(nlh), ib_nl_policy, NULL);
attr = (const struct nlattr *)tb[LS_NLA_TYPE_TIMEOUT];
if (ret || !attr)
goto settimeout_out;
@@ -860,7 +860,7 @@
return 0;
ret = nla_parse(tb, LS_NLA_TYPE_MAX - 1, nlmsg_data(nlh),
- nlmsg_len(nlh), ib_nl_policy);
+ nlmsg_len(nlh), ib_nl_policy, NULL);
if (ret)
return 0;
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 3cd064b..ce8ba61 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -729,16 +729,6 @@
return container_of(ibmw, struct mlx5_ib_mw, ibmw);
}
-struct mlx5_ib_ah {
- struct ib_ah ibah;
- struct mlx5_av av;
-};
-
-static inline struct mlx5_ib_ah *to_mah(struct ib_ah *ibah)
-{
- return container_of(ibah, struct mlx5_ib_ah, ibah);
-}
-
int mlx5_ib_db_map_user(struct mlx5_ib_ucontext *context, unsigned long virt,
struct mlx5_db *db);
void mlx5_ib_db_unmap_user(struct mlx5_ib_ucontext *context, struct mlx5_db *db);
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index ad8a263..ed63201 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -897,6 +897,7 @@
if (init_attr->create_flags & ~(IB_QP_CREATE_SIGNATURE_EN |
IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK |
IB_QP_CREATE_IPOIB_UD_LSO |
+ IB_QP_CREATE_NETIF_QP |
mlx5_ib_create_qp_sqpn_qp1()))
return -EINVAL;
diff --git a/drivers/infiniband/hw/nes/nes.h b/drivers/infiniband/hw/nes/nes.h
index 85acd08..3f9e56e 100644
--- a/drivers/infiniband/hw/nes/nes.h
+++ b/drivers/infiniband/hw/nes/nes.h
@@ -36,6 +36,7 @@
#include <linux/netdevice.h>
#include <linux/inetdevice.h>
+#include <linux/interrupt.h>
#include <linux/spinlock.h>
#include <linux/kernel.h>
#include <linux/delay.h>
diff --git a/drivers/infiniband/hw/qedr/main.c b/drivers/infiniband/hw/qedr/main.c
index b9b47e5..ced0461 100644
--- a/drivers/infiniband/hw/qedr/main.c
+++ b/drivers/infiniband/hw/qedr/main.c
@@ -587,9 +587,8 @@
#define EVENT_TYPE_CQ 1
#define EVENT_TYPE_QP 2
struct qedr_dev *dev = (struct qedr_dev *)context;
- union event_ring_data *data = fw_handle;
- u64 roce_handle64 = ((u64)data->roce_handle.hi << 32) +
- data->roce_handle.lo;
+ struct regpair *async_handle = (struct regpair *)fw_handle;
+ u64 roce_handle64 = ((u64) async_handle->hi << 32) + async_handle->lo;
u8 event_type = EVENT_TYPE_NOT_DEFINED;
struct ib_event event;
struct ib_cq *ibcq;
diff --git a/drivers/infiniband/hw/qedr/qedr.h b/drivers/infiniband/hw/qedr/qedr.h
index bb32e47..5cb9195 100644
--- a/drivers/infiniband/hw/qedr/qedr.h
+++ b/drivers/infiniband/hw/qedr/qedr.h
@@ -38,7 +38,8 @@
#include <linux/qed/qed_chain.h>
#include <linux/qed/qed_roce_if.h>
#include <linux/qed/qede_roce.h>
-#include "qedr_hsi.h"
+#include <linux/qed/roce_common.h>
+#include "qedr_hsi_rdma.h"
#define QEDR_MODULE_VERSION "8.10.10.0"
#define QEDR_NODE_DESC "QLogic 579xx RoCE HCA"
diff --git a/drivers/infiniband/hw/qedr/qedr_cm.c b/drivers/infiniband/hw/qedr/qedr_cm.c
index 6996328..a6280ce 100644
--- a/drivers/infiniband/hw/qedr/qedr_cm.c
+++ b/drivers/infiniband/hw/qedr/qedr_cm.c
@@ -43,14 +43,11 @@
#include <rdma/ib_addr.h>
#include <rdma/ib_cache.h>
-#include "qedr_hsi.h"
#include <linux/qed/qed_if.h>
#include <linux/qed/qed_roce_if.h>
#include "qedr.h"
-#include "qedr_hsi.h"
#include "verbs.h"
#include <rdma/qedr-abi.h>
-#include "qedr_hsi.h"
#include "qedr_cm.h"
void qedr_inc_sw_gsi_cons(struct qedr_qp_hwq_info *info)
diff --git a/drivers/infiniband/hw/qedr/verbs.c b/drivers/infiniband/hw/qedr/verbs.c
index 6b3bb32..2091902 100644
--- a/drivers/infiniband/hw/qedr/verbs.c
+++ b/drivers/infiniband/hw/qedr/verbs.c
@@ -43,7 +43,8 @@
#include <rdma/ib_addr.h>
#include <rdma/ib_cache.h>
-#include "qedr_hsi.h"
+#include <linux/qed/common_hsi.h>
+#include "qedr_hsi_rdma.h"
#include <linux/qed/qed_if.h>
#include "qedr.h"
#include "verbs.h"
diff --git a/drivers/input/serio/i8042-x86ia64io.h b/drivers/input/serio/i8042-x86ia64io.h
index 312bd6c..09720d9 100644
--- a/drivers/input/serio/i8042-x86ia64io.h
+++ b/drivers/input/serio/i8042-x86ia64io.h
@@ -620,6 +620,13 @@
DMI_MATCH(DMI_PRODUCT_NAME, "20046"),
},
},
+ {
+ /* Clevo P650RS, 650RP6, Sager NP8152-S, and others */
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Notebook"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "P65xRP"),
+ },
+ },
{ }
};
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 37e204f..6ee3a25 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -327,6 +327,14 @@
help
Support for the IOMMU API for s390 PCI devices.
+config S390_CCW_IOMMU
+ bool "S390 CCW IOMMU Support"
+ depends on S390 && CCW
+ select IOMMU_API
+ help
+ Enables bits of IOMMU API required by VFIO. The iommu_ops
+ is not implemented as it is not necessary for VFIO.
+
config MTK_IOMMU
bool "MTK IOMMU Support"
depends on ARM || ARM64
diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
index b17536d6..63cacf5 100644
--- a/drivers/iommu/amd_iommu.c
+++ b/drivers/iommu/amd_iommu.c
@@ -1234,7 +1234,7 @@
build_inv_iommu_pages(&cmd, address, size, domain->id, pde);
- for (i = 0; i < amd_iommus_present; ++i) {
+ for (i = 0; i < amd_iommu_get_num_iommus(); ++i) {
if (!domain->dev_iommu[i])
continue;
@@ -1278,7 +1278,7 @@
{
int i;
- for (i = 0; i < amd_iommus_present; ++i) {
+ for (i = 0; i < amd_iommu_get_num_iommus(); ++i) {
if (domain && !domain->dev_iommu[i])
continue;
@@ -3363,7 +3363,7 @@
* IOMMU TLB needs to be flushed before Device TLB to
* prevent device TLB refill from IOMMU TLB
*/
- for (i = 0; i < amd_iommus_present; ++i) {
+ for (i = 0; i < amd_iommu_get_num_iommus(); ++i) {
if (domain->dev_iommu[i] == 0)
continue;
diff --git a/drivers/iommu/amd_iommu_init.c b/drivers/iommu/amd_iommu_init.c
index 6130278..5a11328 100644
--- a/drivers/iommu/amd_iommu_init.c
+++ b/drivers/iommu/amd_iommu_init.c
@@ -167,7 +167,9 @@
/* Array to assign indices to IOMMUs*/
struct amd_iommu *amd_iommus[MAX_IOMMUS];
-int amd_iommus_present;
+
+/* Number of IOMMUs present in the system */
+static int amd_iommus_present;
/* IOMMUs have a non-present cache? */
bool amd_iommu_np_cache __read_mostly;
@@ -254,10 +256,6 @@
static int __init iommu_go_to_state(enum iommu_init_state state);
static void init_device_table_dma(void);
-static int iommu_pc_get_set_reg_val(struct amd_iommu *iommu,
- u8 bank, u8 cntr, u8 fxn,
- u64 *value, bool is_write);
-
static inline void update_last_devid(u16 devid)
{
if (devid > amd_iommu_last_bdf)
@@ -272,6 +270,11 @@
return 1UL << shift;
}
+int amd_iommu_get_num_iommus(void)
+{
+ return amd_iommus_present;
+}
+
/* Access to l1 and l2 indexed register spaces */
static u32 iommu_read_l1(struct amd_iommu *iommu, u16 l1, u8 address)
@@ -1336,7 +1339,7 @@
/* Add IOMMU to internal data structures */
list_add_tail(&iommu->list, &amd_iommu_list);
- iommu->index = amd_iommus_present++;
+ iommu->index = amd_iommus_present++;
if (unlikely(iommu->index >= MAX_IOMMUS)) {
WARN(1, "AMD-Vi: System has more IOMMUs than supported by this driver\n");
@@ -1477,6 +1480,8 @@
return 0;
}
+static int iommu_pc_get_set_reg(struct amd_iommu *iommu, u8 bank, u8 cntr,
+ u8 fxn, u64 *value, bool is_write);
static void init_iommu_perf_ctr(struct amd_iommu *iommu)
{
@@ -1488,8 +1493,8 @@
amd_iommu_pc_present = true;
/* Check if the performance counters can be written to */
- if ((0 != iommu_pc_get_set_reg_val(iommu, 0, 0, 0, &val, true)) ||
- (0 != iommu_pc_get_set_reg_val(iommu, 0, 0, 0, &val2, false)) ||
+ if ((iommu_pc_get_set_reg(iommu, 0, 0, 0, &val, true)) ||
+ (iommu_pc_get_set_reg(iommu, 0, 0, 0, &val2, false)) ||
(val != val2)) {
pr_err("AMD-Vi: Unable to write to IOMMU perf counter.\n");
amd_iommu_pc_present = false;
@@ -2711,6 +2716,18 @@
}
EXPORT_SYMBOL(amd_iommu_v2_supported);
+struct amd_iommu *get_amd_iommu(unsigned int idx)
+{
+ unsigned int i = 0;
+ struct amd_iommu *iommu;
+
+ for_each_iommu(iommu)
+ if (i++ == idx)
+ return iommu;
+ return NULL;
+}
+EXPORT_SYMBOL(get_amd_iommu);
+
/****************************************************************************
*
* IOMMU EFR Performance Counter support functionality. This code allows
@@ -2718,17 +2735,14 @@
*
****************************************************************************/
-u8 amd_iommu_pc_get_max_banks(u16 devid)
+u8 amd_iommu_pc_get_max_banks(unsigned int idx)
{
- struct amd_iommu *iommu;
- u8 ret = 0;
+ struct amd_iommu *iommu = get_amd_iommu(idx);
- /* locate the iommu governing the devid */
- iommu = amd_iommu_rlookup_table[devid];
if (iommu)
- ret = iommu->max_banks;
+ return iommu->max_banks;
- return ret;
+ return 0;
}
EXPORT_SYMBOL(amd_iommu_pc_get_max_banks);
@@ -2738,62 +2752,69 @@
}
EXPORT_SYMBOL(amd_iommu_pc_supported);
-u8 amd_iommu_pc_get_max_counters(u16 devid)
+u8 amd_iommu_pc_get_max_counters(unsigned int idx)
{
- struct amd_iommu *iommu;
- u8 ret = 0;
+ struct amd_iommu *iommu = get_amd_iommu(idx);
- /* locate the iommu governing the devid */
- iommu = amd_iommu_rlookup_table[devid];
if (iommu)
- ret = iommu->max_counters;
+ return iommu->max_counters;
- return ret;
+ return 0;
}
EXPORT_SYMBOL(amd_iommu_pc_get_max_counters);
-static int iommu_pc_get_set_reg_val(struct amd_iommu *iommu,
- u8 bank, u8 cntr, u8 fxn,
- u64 *value, bool is_write)
+static int iommu_pc_get_set_reg(struct amd_iommu *iommu, u8 bank, u8 cntr,
+ u8 fxn, u64 *value, bool is_write)
{
u32 offset;
u32 max_offset_lim;
- /* Check for valid iommu and pc register indexing */
- if (WARN_ON((fxn > 0x28) || (fxn & 7)))
+ /* Make sure the IOMMU PC resource is available */
+ if (!amd_iommu_pc_present)
return -ENODEV;
- offset = (u32)(((0x40|bank) << 12) | (cntr << 8) | fxn);
+ /* Check for valid iommu and pc register indexing */
+ if (WARN_ON(!iommu || (fxn > 0x28) || (fxn & 7)))
+ return -ENODEV;
+
+ offset = (u32)(((0x40 | bank) << 12) | (cntr << 8) | fxn);
/* Limit the offset to the hw defined mmio region aperture */
- max_offset_lim = (u32)(((0x40|iommu->max_banks) << 12) |
+ max_offset_lim = (u32)(((0x40 | iommu->max_banks) << 12) |
(iommu->max_counters << 8) | 0x28);
if ((offset < MMIO_CNTR_REG_OFFSET) ||
(offset > max_offset_lim))
return -EINVAL;
if (is_write) {
- writel((u32)*value, iommu->mmio_base + offset);
- writel((*value >> 32), iommu->mmio_base + offset + 4);
+ u64 val = *value & GENMASK_ULL(47, 0);
+
+ writel((u32)val, iommu->mmio_base + offset);
+ writel((val >> 32), iommu->mmio_base + offset + 4);
} else {
*value = readl(iommu->mmio_base + offset + 4);
*value <<= 32;
- *value = readl(iommu->mmio_base + offset);
+ *value |= readl(iommu->mmio_base + offset);
+ *value &= GENMASK_ULL(47, 0);
}
return 0;
}
-EXPORT_SYMBOL(amd_iommu_pc_get_set_reg_val);
-int amd_iommu_pc_get_set_reg_val(u16 devid, u8 bank, u8 cntr, u8 fxn,
- u64 *value, bool is_write)
+int amd_iommu_pc_get_reg(struct amd_iommu *iommu, u8 bank, u8 cntr, u8 fxn, u64 *value)
{
- struct amd_iommu *iommu = amd_iommu_rlookup_table[devid];
+ if (!iommu)
+ return -EINVAL;
- /* Make sure the IOMMU PC resource is available */
- if (!amd_iommu_pc_present || iommu == NULL)
- return -ENODEV;
-
- return iommu_pc_get_set_reg_val(iommu, bank, cntr, fxn,
- value, is_write);
+ return iommu_pc_get_set_reg(iommu, bank, cntr, fxn, value, false);
}
+EXPORT_SYMBOL(amd_iommu_pc_get_reg);
+
+int amd_iommu_pc_set_reg(struct amd_iommu *iommu, u8 bank, u8 cntr, u8 fxn, u64 *value)
+{
+ if (!iommu)
+ return -EINVAL;
+
+ return iommu_pc_get_set_reg(iommu, bank, cntr, fxn, value, true);
+}
+EXPORT_SYMBOL(amd_iommu_pc_set_reg);
diff --git a/drivers/iommu/amd_iommu_proto.h b/drivers/iommu/amd_iommu_proto.h
index 7eb60c1..466260f 100644
--- a/drivers/iommu/amd_iommu_proto.h
+++ b/drivers/iommu/amd_iommu_proto.h
@@ -21,6 +21,7 @@
#include "amd_iommu_types.h"
+extern int amd_iommu_get_num_iommus(void);
extern int amd_iommu_init_dma_ops(void);
extern int amd_iommu_init_passthrough(void);
extern irqreturn_t amd_iommu_int_thread(int irq, void *data);
@@ -56,13 +57,6 @@
extern int amd_iommu_domain_clear_gcr3(struct iommu_domain *dom, int pasid);
extern struct iommu_domain *amd_iommu_get_v2_domain(struct pci_dev *pdev);
-/* IOMMU Performance Counter functions */
-extern bool amd_iommu_pc_supported(void);
-extern u8 amd_iommu_pc_get_max_banks(u16 devid);
-extern u8 amd_iommu_pc_get_max_counters(u16 devid);
-extern int amd_iommu_pc_get_set_reg_val(u16 devid, u8 bank, u8 cntr, u8 fxn,
- u64 *value, bool is_write);
-
#ifdef CONFIG_IRQ_REMAP
extern int amd_iommu_create_irq_domain(struct amd_iommu *iommu);
#else
diff --git a/drivers/iommu/amd_iommu_types.h b/drivers/iommu/amd_iommu_types.h
index 003f3ce..4de8f41 100644
--- a/drivers/iommu/amd_iommu_types.h
+++ b/drivers/iommu/amd_iommu_types.h
@@ -611,9 +611,6 @@
*/
extern struct amd_iommu *amd_iommus[MAX_IOMMUS];
-/* Number of IOMMUs present in the system */
-extern int amd_iommus_present;
-
/*
* Declarations for the global list of all protection domains
*/
diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
index 8162121..595d0c9 100644
--- a/drivers/irqchip/Kconfig
+++ b/drivers/irqchip/Kconfig
@@ -115,6 +115,12 @@
select GENERIC_IRQ_CHIP
select IRQ_DOMAIN
+config FARADAY_FTINTC010
+ bool
+ select IRQ_DOMAIN
+ select MULTI_IRQ_HANDLER
+ select SPARSE_IRQ
+
config HISILICON_IRQ_MBIGEN
bool
select ARM_GIC_V3
diff --git a/drivers/irqchip/Makefile b/drivers/irqchip/Makefile
index 152bc40..b64c59b 100644
--- a/drivers/irqchip/Makefile
+++ b/drivers/irqchip/Makefile
@@ -6,7 +6,7 @@
obj-$(CONFIG_ARCH_BCM2835) += irq-bcm2835.o
obj-$(CONFIG_ARCH_BCM2835) += irq-bcm2836.o
obj-$(CONFIG_ARCH_EXYNOS) += exynos-combiner.o
-obj-$(CONFIG_ARCH_GEMINI) += irq-gemini.o
+obj-$(CONFIG_FARADAY_FTINTC010) += irq-ftintc010.o
obj-$(CONFIG_ARCH_HIP04) += irq-hip04.o
obj-$(CONFIG_ARCH_LPC32XX) += irq-lpc32xx.o
obj-$(CONFIG_ARCH_MMP) += irq-mmp.o
@@ -16,7 +16,6 @@
obj-$(CONFIG_DW_APB_ICTL) += irq-dw-apb-ictl.o
obj-$(CONFIG_METAG) += irq-metag-ext.o
obj-$(CONFIG_METAG_PERFCOUNTER_IRQS) += irq-metag.o
-obj-$(CONFIG_ARCH_MOXART) += irq-moxart.o
obj-$(CONFIG_CLPS711X_IRQCHIP) += irq-clps711x.o
obj-$(CONFIG_OR1K_PIC) += irq-or1k-pic.o
obj-$(CONFIG_ORION_IRQCHIP) += irq-orion.o
@@ -62,7 +61,7 @@
obj-$(CONFIG_BRCMSTB_L2_IRQ) += irq-brcmstb-l2.o
obj-$(CONFIG_KEYSTONE_IRQ) += irq-keystone.o
obj-$(CONFIG_MIPS_GIC) += irq-mips-gic.o
-obj-$(CONFIG_ARCH_MEDIATEK) += irq-mtk-sysirq.o
+obj-$(CONFIG_ARCH_MEDIATEK) += irq-mtk-sysirq.o irq-mtk-cirq.o
obj-$(CONFIG_ARCH_DIGICOLOR) += irq-digicolor.o
obj-$(CONFIG_RENESAS_H8300H_INTC) += irq-renesas-h8300h.o
obj-$(CONFIG_RENESAS_H8S_INTC) += irq-renesas-h8s.o
diff --git a/drivers/irqchip/irq-atmel-aic5.c b/drivers/irqchip/irq-atmel-aic5.c
index 2a624d8..c04ee9a 100644
--- a/drivers/irqchip/irq-atmel-aic5.c
+++ b/drivers/irqchip/irq-atmel-aic5.c
@@ -150,6 +150,8 @@
}
#ifdef CONFIG_PM
+static u32 *smr_cache;
+
static void aic5_suspend(struct irq_data *d)
{
struct irq_domain *domain = d->domain;
@@ -159,6 +161,12 @@
int i;
u32 mask;
+ if (smr_cache)
+ for (i = 0; i < domain->revmap_size; i++) {
+ irq_reg_writel(bgc, i, AT91_AIC5_SSR);
+ smr_cache[i] = irq_reg_readl(bgc, AT91_AIC5_SMR);
+ }
+
irq_gc_lock(bgc);
for (i = 0; i < dgc->irqs_per_chip; i++) {
mask = 1 << i;
@@ -184,9 +192,21 @@
u32 mask;
irq_gc_lock(bgc);
+
+ if (smr_cache) {
+ irq_reg_writel(bgc, 0xffffffff, AT91_AIC5_SPU);
+ for (i = 0; i < domain->revmap_size; i++) {
+ irq_reg_writel(bgc, i, AT91_AIC5_SSR);
+ irq_reg_writel(bgc, i, AT91_AIC5_SVR);
+ irq_reg_writel(bgc, smr_cache[i], AT91_AIC5_SMR);
+ }
+ }
+
for (i = 0; i < dgc->irqs_per_chip; i++) {
mask = 1 << i;
- if ((mask & gc->mask_cache) == (mask & gc->wake_active))
+
+ if (!smr_cache &&
+ ((mask & gc->mask_cache) == (mask & gc->wake_active)))
continue;
irq_reg_writel(bgc, i + gc->irq_base, AT91_AIC5_SSR);
@@ -342,6 +362,13 @@
static int __init sama5d2_aic5_of_init(struct device_node *node,
struct device_node *parent)
{
+#ifdef CONFIG_PM
+ smr_cache = kcalloc(DIV_ROUND_UP(NR_SAMA5D2_IRQS, 32) * 32,
+ sizeof(*smr_cache), GFP_KERNEL);
+ if (!smr_cache)
+ return -ENOMEM;
+#endif
+
return aic5_of_init(node, parent, NR_SAMA5D2_IRQS);
}
IRQCHIP_DECLARE(sama5d2_aic5, "atmel,sama5d2-aic", sama5d2_aic5_of_init);
diff --git a/drivers/irqchip/irq-ftintc010.c b/drivers/irqchip/irq-ftintc010.c
new file mode 100644
index 0000000..cd2dc8b
--- /dev/null
+++ b/drivers/irqchip/irq-ftintc010.c
@@ -0,0 +1,194 @@
+/*
+ * irqchip for the Faraday Technology FTINTC010 Copyright (C) 2017 Linus
+ * Walleij <linus.walleij@linaro.org>
+ *
+ * Based on arch/arm/mach-gemini/irq.c
+ * Copyright (C) 2001-2006 Storlink, Corp.
+ * Copyright (C) 2008-2009 Paulius Zaleckas <paulius.zaleckas@gmail.com>
+ */
+#include <linux/bitops.h>
+#include <linux/irq.h>
+#include <linux/io.h>
+#include <linux/irqchip.h>
+#include <linux/irqchip/versatile-fpga.h>
+#include <linux/irqdomain.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/of_irq.h>
+#include <linux/cpu.h>
+
+#include <asm/exception.h>
+#include <asm/mach/irq.h>
+
+#define FT010_NUM_IRQS 32
+
+#define FT010_IRQ_SOURCE(base_addr) (base_addr + 0x00)
+#define FT010_IRQ_MASK(base_addr) (base_addr + 0x04)
+#define FT010_IRQ_CLEAR(base_addr) (base_addr + 0x08)
+/* Selects level- or edge-triggered */
+#define FT010_IRQ_MODE(base_addr) (base_addr + 0x0C)
+/* Selects active low/high or falling/rising edge */
+#define FT010_IRQ_POLARITY(base_addr) (base_addr + 0x10)
+#define FT010_IRQ_STATUS(base_addr) (base_addr + 0x14)
+#define FT010_FIQ_SOURCE(base_addr) (base_addr + 0x20)
+#define FT010_FIQ_MASK(base_addr) (base_addr + 0x24)
+#define FT010_FIQ_CLEAR(base_addr) (base_addr + 0x28)
+#define FT010_FIQ_MODE(base_addr) (base_addr + 0x2C)
+#define FT010_FIQ_POLARITY(base_addr) (base_addr + 0x30)
+#define FT010_FIQ_STATUS(base_addr) (base_addr + 0x34)
+
+/**
+ * struct ft010_irq_data - irq data container for the Faraday IRQ controller
+ * @base: memory offset in virtual memory
+ * @chip: chip container for this instance
+ * @domain: IRQ domain for this instance
+ */
+struct ft010_irq_data {
+ void __iomem *base;
+ struct irq_chip chip;
+ struct irq_domain *domain;
+};
+
+static void ft010_irq_mask(struct irq_data *d)
+{
+ struct ft010_irq_data *f = irq_data_get_irq_chip_data(d);
+ unsigned int mask;
+
+ mask = readl(FT010_IRQ_MASK(f->base));
+ mask &= ~BIT(irqd_to_hwirq(d));
+ writel(mask, FT010_IRQ_MASK(f->base));
+}
+
+static void ft010_irq_unmask(struct irq_data *d)
+{
+ struct ft010_irq_data *f = irq_data_get_irq_chip_data(d);
+ unsigned int mask;
+
+ mask = readl(FT010_IRQ_MASK(f->base));
+ mask |= BIT(irqd_to_hwirq(d));
+ writel(mask, FT010_IRQ_MASK(f->base));
+}
+
+static void ft010_irq_ack(struct irq_data *d)
+{
+ struct ft010_irq_data *f = irq_data_get_irq_chip_data(d);
+
+ writel(BIT(irqd_to_hwirq(d)), FT010_IRQ_CLEAR(f->base));
+}
+
+static int ft010_irq_set_type(struct irq_data *d, unsigned int trigger)
+{
+ struct ft010_irq_data *f = irq_data_get_irq_chip_data(d);
+ int offset = irqd_to_hwirq(d);
+ u32 mode, polarity;
+
+ mode = readl(FT010_IRQ_MODE(f->base));
+ polarity = readl(FT010_IRQ_POLARITY(f->base));
+
+ if (trigger & (IRQ_TYPE_LEVEL_LOW)) {
+ irq_set_handler_locked(d, handle_level_irq);
+ mode &= ~BIT(offset);
+ polarity |= BIT(offset);
+ } else if (trigger & (IRQ_TYPE_LEVEL_HIGH)) {
+ irq_set_handler_locked(d, handle_level_irq);
+ mode &= ~BIT(offset);
+ polarity &= ~BIT(offset);
+ } else if (trigger & IRQ_TYPE_EDGE_FALLING) {
+ irq_set_handler_locked(d, handle_edge_irq);
+ mode |= BIT(offset);
+ polarity |= BIT(offset);
+ } else if (trigger & IRQ_TYPE_EDGE_RISING) {
+ irq_set_handler_locked(d, handle_edge_irq);
+ mode |= BIT(offset);
+ polarity &= ~BIT(offset);
+ } else {
+ irq_set_handler_locked(d, handle_bad_irq);
+ pr_warn("Faraday IRQ: no supported trigger selected for line %d\n",
+ offset);
+ }
+
+ writel(mode, FT010_IRQ_MODE(f->base));
+ writel(polarity, FT010_IRQ_POLARITY(f->base));
+
+ return 0;
+}
+
+static struct irq_chip ft010_irq_chip = {
+ .name = "FTINTC010",
+ .irq_ack = ft010_irq_ack,
+ .irq_mask = ft010_irq_mask,
+ .irq_unmask = ft010_irq_unmask,
+ .irq_set_type = ft010_irq_set_type,
+};
+
+/* Local static for the IRQ entry call */
+static struct ft010_irq_data firq;
+
+asmlinkage void __exception_irq_entry ft010_irqchip_handle_irq(struct pt_regs *regs)
+{
+ struct ft010_irq_data *f = &firq;
+ int irq;
+ u32 status;
+
+ while ((status = readl(FT010_IRQ_STATUS(f->base)))) {
+ irq = ffs(status) - 1;
+ handle_domain_irq(f->domain, irq, regs);
+ }
+}
+
+static int ft010_irqdomain_map(struct irq_domain *d, unsigned int irq,
+ irq_hw_number_t hwirq)
+{
+ struct ft010_irq_data *f = d->host_data;
+
+ irq_set_chip_data(irq, f);
+ /* All IRQs should set up their type, flags as bad by default */
+ irq_set_chip_and_handler(irq, &ft010_irq_chip, handle_bad_irq);
+ irq_set_probe(irq);
+
+ return 0;
+}
+
+static void ft010_irqdomain_unmap(struct irq_domain *d, unsigned int irq)
+{
+ irq_set_chip_and_handler(irq, NULL, NULL);
+ irq_set_chip_data(irq, NULL);
+}
+
+static const struct irq_domain_ops ft010_irqdomain_ops = {
+ .map = ft010_irqdomain_map,
+ .unmap = ft010_irqdomain_unmap,
+ .xlate = irq_domain_xlate_onetwocell,
+};
+
+int __init ft010_of_init_irq(struct device_node *node,
+ struct device_node *parent)
+{
+ struct ft010_irq_data *f = &firq;
+
+ /*
+ * Disable the idle handler by default since it is buggy
+ * For more info see arch/arm/mach-gemini/idle.c
+ */
+ cpu_idle_poll_ctrl(true);
+
+ f->base = of_iomap(node, 0);
+ WARN(!f->base, "unable to map gemini irq registers\n");
+
+ /* Disable all interrupts */
+ writel(0, FT010_IRQ_MASK(f->base));
+ writel(0, FT010_FIQ_MASK(f->base));
+
+ f->domain = irq_domain_add_simple(node, FT010_NUM_IRQS, 0,
+ &ft010_irqdomain_ops, f);
+ set_handle_irq(ft010_irqchip_handle_irq);
+
+ return 0;
+}
+IRQCHIP_DECLARE(faraday, "faraday,ftintc010",
+ ft010_of_init_irq);
+IRQCHIP_DECLARE(gemini, "cortina,gemini-interrupt-controller",
+ ft010_of_init_irq);
+IRQCHIP_DECLARE(moxa, "moxa,moxart-ic",
+ ft010_of_init_irq);
diff --git a/drivers/irqchip/irq-gemini.c b/drivers/irqchip/irq-gemini.c
deleted file mode 100644
index 495224c..0000000
--- a/drivers/irqchip/irq-gemini.c
+++ /dev/null
@@ -1,185 +0,0 @@
-/*
- * irqchip for the Cortina Systems Gemini Copyright (C) 2017 Linus
- * Walleij <linus.walleij@linaro.org>
- *
- * Based on arch/arm/mach-gemini/irq.c
- * Copyright (C) 2001-2006 Storlink, Corp.
- * Copyright (C) 2008-2009 Paulius Zaleckas <paulius.zaleckas@teltonika.lt>
- */
-#include <linux/bitops.h>
-#include <linux/irq.h>
-#include <linux/io.h>
-#include <linux/irqchip.h>
-#include <linux/irqchip/versatile-fpga.h>
-#include <linux/irqdomain.h>
-#include <linux/module.h>
-#include <linux/of.h>
-#include <linux/of_address.h>
-#include <linux/of_irq.h>
-#include <linux/cpu.h>
-
-#include <asm/exception.h>
-#include <asm/mach/irq.h>
-
-#define GEMINI_NUM_IRQS 32
-
-#define GEMINI_IRQ_SOURCE(base_addr) (base_addr + 0x00)
-#define GEMINI_IRQ_MASK(base_addr) (base_addr + 0x04)
-#define GEMINI_IRQ_CLEAR(base_addr) (base_addr + 0x08)
-#define GEMINI_IRQ_MODE(base_addr) (base_addr + 0x0C)
-#define GEMINI_IRQ_POLARITY(base_addr) (base_addr + 0x10)
-#define GEMINI_IRQ_STATUS(base_addr) (base_addr + 0x14)
-#define GEMINI_FIQ_SOURCE(base_addr) (base_addr + 0x20)
-#define GEMINI_FIQ_MASK(base_addr) (base_addr + 0x24)
-#define GEMINI_FIQ_CLEAR(base_addr) (base_addr + 0x28)
-#define GEMINI_FIQ_MODE(base_addr) (base_addr + 0x2C)
-#define GEMINI_FIQ_POLARITY(base_addr) (base_addr + 0x30)
-#define GEMINI_FIQ_STATUS(base_addr) (base_addr + 0x34)
-
-/**
- * struct gemini_irq_data - irq data container for the Gemini IRQ controller
- * @base: memory offset in virtual memory
- * @chip: chip container for this instance
- * @domain: IRQ domain for this instance
- */
-struct gemini_irq_data {
- void __iomem *base;
- struct irq_chip chip;
- struct irq_domain *domain;
-};
-
-static void gemini_irq_mask(struct irq_data *d)
-{
- struct gemini_irq_data *g = irq_data_get_irq_chip_data(d);
- unsigned int mask;
-
- mask = readl(GEMINI_IRQ_MASK(g->base));
- mask &= ~BIT(irqd_to_hwirq(d));
- writel(mask, GEMINI_IRQ_MASK(g->base));
-}
-
-static void gemini_irq_unmask(struct irq_data *d)
-{
- struct gemini_irq_data *g = irq_data_get_irq_chip_data(d);
- unsigned int mask;
-
- mask = readl(GEMINI_IRQ_MASK(g->base));
- mask |= BIT(irqd_to_hwirq(d));
- writel(mask, GEMINI_IRQ_MASK(g->base));
-}
-
-static void gemini_irq_ack(struct irq_data *d)
-{
- struct gemini_irq_data *g = irq_data_get_irq_chip_data(d);
-
- writel(BIT(irqd_to_hwirq(d)), GEMINI_IRQ_CLEAR(g->base));
-}
-
-static int gemini_irq_set_type(struct irq_data *d, unsigned int trigger)
-{
- struct gemini_irq_data *g = irq_data_get_irq_chip_data(d);
- int offset = irqd_to_hwirq(d);
- u32 mode, polarity;
-
- mode = readl(GEMINI_IRQ_MODE(g->base));
- polarity = readl(GEMINI_IRQ_POLARITY(g->base));
-
- if (trigger & (IRQ_TYPE_LEVEL_HIGH)) {
- irq_set_handler_locked(d, handle_level_irq);
- /* Disable edge detection */
- mode &= ~BIT(offset);
- polarity &= ~BIT(offset);
- } else if (trigger & IRQ_TYPE_EDGE_RISING) {
- irq_set_handler_locked(d, handle_edge_irq);
- mode |= BIT(offset);
- polarity |= BIT(offset);
- } else if (trigger & IRQ_TYPE_EDGE_FALLING) {
- irq_set_handler_locked(d, handle_edge_irq);
- mode |= BIT(offset);
- polarity &= ~BIT(offset);
- } else {
- irq_set_handler_locked(d, handle_bad_irq);
- pr_warn("GEMINI IRQ: no supported trigger selected for line %d\n",
- offset);
- }
-
- writel(mode, GEMINI_IRQ_MODE(g->base));
- writel(polarity, GEMINI_IRQ_POLARITY(g->base));
-
- return 0;
-}
-
-static struct irq_chip gemini_irq_chip = {
- .name = "GEMINI",
- .irq_ack = gemini_irq_ack,
- .irq_mask = gemini_irq_mask,
- .irq_unmask = gemini_irq_unmask,
- .irq_set_type = gemini_irq_set_type,
-};
-
-/* Local static for the IRQ entry call */
-static struct gemini_irq_data girq;
-
-asmlinkage void __exception_irq_entry gemini_irqchip_handle_irq(struct pt_regs *regs)
-{
- struct gemini_irq_data *g = &girq;
- int irq;
- u32 status;
-
- while ((status = readl(GEMINI_IRQ_STATUS(g->base)))) {
- irq = ffs(status) - 1;
- handle_domain_irq(g->domain, irq, regs);
- }
-}
-
-static int gemini_irqdomain_map(struct irq_domain *d, unsigned int irq,
- irq_hw_number_t hwirq)
-{
- struct gemini_irq_data *g = d->host_data;
-
- irq_set_chip_data(irq, g);
- /* All IRQs should set up their type, flags as bad by default */
- irq_set_chip_and_handler(irq, &gemini_irq_chip, handle_bad_irq);
- irq_set_probe(irq);
-
- return 0;
-}
-
-static void gemini_irqdomain_unmap(struct irq_domain *d, unsigned int irq)
-{
- irq_set_chip_and_handler(irq, NULL, NULL);
- irq_set_chip_data(irq, NULL);
-}
-
-static const struct irq_domain_ops gemini_irqdomain_ops = {
- .map = gemini_irqdomain_map,
- .unmap = gemini_irqdomain_unmap,
- .xlate = irq_domain_xlate_onetwocell,
-};
-
-int __init gemini_of_init_irq(struct device_node *node,
- struct device_node *parent)
-{
- struct gemini_irq_data *g = &girq;
-
- /*
- * Disable the idle handler by default since it is buggy
- * For more info see arch/arm/mach-gemini/idle.c
- */
- cpu_idle_poll_ctrl(true);
-
- g->base = of_iomap(node, 0);
- WARN(!g->base, "unable to map gemini irq registers\n");
-
- /* Disable all interrupts */
- writel(0, GEMINI_IRQ_MASK(g->base));
- writel(0, GEMINI_FIQ_MASK(g->base));
-
- g->domain = irq_domain_add_simple(node, GEMINI_NUM_IRQS, 0,
- &gemini_irqdomain_ops, g);
- set_handle_irq(gemini_irqchip_handle_irq);
-
- return 0;
-}
-IRQCHIP_DECLARE(gemini, "cortina,gemini-interrupt-controller",
- gemini_of_init_irq);
diff --git a/drivers/irqchip/irq-gic-v3-its-platform-msi.c b/drivers/irqchip/irq-gic-v3-its-platform-msi.c
index 470b4aa..9e9dda3 100644
--- a/drivers/irqchip/irq-gic-v3-its-platform-msi.c
+++ b/drivers/irqchip/irq-gic-v3-its-platform-msi.c
@@ -15,6 +15,7 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
+#include <linux/acpi_iort.h>
#include <linux/device.h>
#include <linux/msi.h>
#include <linux/of.h>
@@ -24,15 +25,11 @@
.name = "ITS-pMSI",
};
-static int its_pmsi_prepare(struct irq_domain *domain, struct device *dev,
- int nvec, msi_alloc_info_t *info)
+static int of_pmsi_get_dev_id(struct irq_domain *domain, struct device *dev,
+ u32 *dev_id)
{
- struct msi_domain_info *msi_info;
- u32 dev_id;
int ret, index = 0;
- msi_info = msi_get_domain_info(domain->parent);
-
/* Suck the DeviceID out of the msi-parent property */
do {
struct of_phandle_args args;
@@ -43,11 +40,32 @@
if (args.np == irq_domain_get_of_node(domain)) {
if (WARN_ON(args.args_count != 1))
return -EINVAL;
- dev_id = args.args[0];
+ *dev_id = args.args[0];
break;
}
} while (!ret);
+ return ret;
+}
+
+int __weak iort_pmsi_get_dev_id(struct device *dev, u32 *dev_id)
+{
+ return -1;
+}
+
+static int its_pmsi_prepare(struct irq_domain *domain, struct device *dev,
+ int nvec, msi_alloc_info_t *info)
+{
+ struct msi_domain_info *msi_info;
+ u32 dev_id;
+ int ret;
+
+ msi_info = msi_get_domain_info(domain->parent);
+
+ if (dev->of_node)
+ ret = of_pmsi_get_dev_id(domain, dev, &dev_id);
+ else
+ ret = iort_pmsi_get_dev_id(dev, &dev_id);
if (ret)
return ret;
@@ -73,34 +91,79 @@
{},
};
-static int __init its_pmsi_init(void)
+static int __init its_pmsi_init_one(struct fwnode_handle *fwnode,
+ const char *name)
+{
+ struct irq_domain *parent;
+
+ parent = irq_find_matching_fwnode(fwnode, DOMAIN_BUS_NEXUS);
+ if (!parent || !msi_get_domain_info(parent)) {
+ pr_err("%s: unable to locate ITS domain\n", name);
+ return -ENXIO;
+ }
+
+ if (!platform_msi_create_irq_domain(fwnode, &its_pmsi_domain_info,
+ parent)) {
+ pr_err("%s: unable to create platform domain\n", name);
+ return -ENXIO;
+ }
+
+ pr_info("Platform MSI: %s domain created\n", name);
+ return 0;
+}
+
+#ifdef CONFIG_ACPI
+static int __init
+its_pmsi_parse_madt(struct acpi_subtable_header *header,
+ const unsigned long end)
+{
+ struct acpi_madt_generic_translator *its_entry;
+ struct fwnode_handle *domain_handle;
+ const char *node_name;
+ int err = -ENXIO;
+
+ its_entry = (struct acpi_madt_generic_translator *)header;
+ node_name = kasprintf(GFP_KERNEL, "ITS@0x%lx",
+ (long)its_entry->base_address);
+ domain_handle = iort_find_domain_token(its_entry->translation_id);
+ if (!domain_handle) {
+ pr_err("%s: Unable to locate ITS domain handle\n", node_name);
+ goto out;
+ }
+
+ err = its_pmsi_init_one(domain_handle, node_name);
+
+out:
+ kfree(node_name);
+ return err;
+}
+
+static void __init its_pmsi_acpi_init(void)
+{
+ acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_TRANSLATOR,
+ its_pmsi_parse_madt, 0);
+}
+#else
+static inline void its_pmsi_acpi_init(void) { }
+#endif
+
+static void __init its_pmsi_of_init(void)
{
struct device_node *np;
- struct irq_domain *parent;
for (np = of_find_matching_node(NULL, its_device_id); np;
np = of_find_matching_node(np, its_device_id)) {
if (!of_property_read_bool(np, "msi-controller"))
continue;
- parent = irq_find_matching_host(np, DOMAIN_BUS_NEXUS);
- if (!parent || !msi_get_domain_info(parent)) {
- pr_err("%s: unable to locate ITS domain\n",
- np->full_name);
- continue;
- }
-
- if (!platform_msi_create_irq_domain(of_node_to_fwnode(np),
- &its_pmsi_domain_info,
- parent)) {
- pr_err("%s: unable to create platform domain\n",
- np->full_name);
- continue;
- }
-
- pr_info("Platform MSI: %s domain created\n", np->full_name);
+ its_pmsi_init_one(of_node_to_fwnode(np), np->full_name);
}
+}
+static int __init its_pmsi_init(void)
+{
+ its_pmsi_of_init();
+ its_pmsi_acpi_init();
return 0;
}
early_initcall(its_pmsi_init);
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index f77f840..45ea1933 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -16,13 +16,13 @@
*/
#include <linux/acpi.h>
+#include <linux/acpi_iort.h>
#include <linux/bitmap.h>
#include <linux/cpu.h>
#include <linux/delay.h>
#include <linux/dma-iommu.h>
#include <linux/interrupt.h>
#include <linux/irqdomain.h>
-#include <linux/acpi_iort.h>
#include <linux/log2.h>
#include <linux/mm.h>
#include <linux/msi.h>
diff --git a/drivers/irqchip/irq-imx-gpcv2.c b/drivers/irqchip/irq-imx-gpcv2.c
index 2d203b4..9463f35 100644
--- a/drivers/irqchip/irq-imx-gpcv2.c
+++ b/drivers/irqchip/irq-imx-gpcv2.c
@@ -268,6 +268,11 @@
imx_gpcv2_instance = cd;
register_syscore_ops(&imx_gpcv2_syscore_ops);
+ /*
+ * Clear the OF_POPULATED flag set in of_irq_init so that
+ * later the GPC power domain driver will not be skipped.
+ */
+ of_node_clear_flag(node, OF_POPULATED);
return 0;
}
diff --git a/drivers/irqchip/irq-mbigen.c b/drivers/irqchip/irq-mbigen.c
index 03b79b0..d2306c8 100644
--- a/drivers/irqchip/irq-mbigen.c
+++ b/drivers/irqchip/irq-mbigen.c
@@ -16,6 +16,7 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
+#include <linux/acpi.h>
#include <linux/interrupt.h>
#include <linux/irqchip.h>
#include <linux/module.h>
@@ -180,7 +181,7 @@
unsigned long *hwirq,
unsigned int *type)
{
- if (is_of_node(fwspec->fwnode)) {
+ if (is_of_node(fwspec->fwnode) || is_acpi_device_node(fwspec->fwnode)) {
if (fwspec->param_count != 2)
return -EINVAL;
@@ -236,27 +237,15 @@
.free = irq_domain_free_irqs_common,
};
-static int mbigen_device_probe(struct platform_device *pdev)
+static int mbigen_of_create_domain(struct platform_device *pdev,
+ struct mbigen_device *mgn_chip)
{
- struct mbigen_device *mgn_chip;
+ struct device *parent;
struct platform_device *child;
struct irq_domain *domain;
struct device_node *np;
- struct device *parent;
- struct resource *res;
u32 num_pins;
- mgn_chip = devm_kzalloc(&pdev->dev, sizeof(*mgn_chip), GFP_KERNEL);
- if (!mgn_chip)
- return -ENOMEM;
-
- mgn_chip->pdev = pdev;
-
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- mgn_chip->base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(mgn_chip->base))
- return PTR_ERR(mgn_chip->base);
-
for_each_child_of_node(pdev->dev.of_node, np) {
if (!of_property_read_bool(np, "interrupt-controller"))
continue;
@@ -280,6 +269,91 @@
return -ENOMEM;
}
+ return 0;
+}
+
+#ifdef CONFIG_ACPI
+static int mbigen_acpi_create_domain(struct platform_device *pdev,
+ struct mbigen_device *mgn_chip)
+{
+ struct irq_domain *domain;
+ u32 num_pins = 0;
+ int ret;
+
+ /*
+ * "num-pins" is the total number of interrupt pins implemented in
+ * this mbigen instance, and mbigen is an interrupt controller
+ * connected to ITS converting wired interrupts into MSI, so we
+ * use "num-pins" to alloc MSI vectors which are needed by client
+ * devices connected to it.
+ *
+ * Here is the DSDT device node used for mbigen in firmware:
+ * Device(MBI0) {
+ * Name(_HID, "HISI0152")
+ * Name(_UID, Zero)
+ * Name(_CRS, ResourceTemplate() {
+ * Memory32Fixed(ReadWrite, 0xa0080000, 0x10000)
+ * })
+ *
+ * Name(_DSD, Package () {
+ * ToUUID("daffd814-6eba-4d8c-8a91-bc9bbf4aa301"),
+ * Package () {
+ * Package () {"num-pins", 378}
+ * }
+ * })
+ * }
+ */
+ ret = device_property_read_u32(&pdev->dev, "num-pins", &num_pins);
+ if (ret || num_pins == 0)
+ return -EINVAL;
+
+ domain = platform_msi_create_device_domain(&pdev->dev, num_pins,
+ mbigen_write_msg,
+ &mbigen_domain_ops,
+ mgn_chip);
+ if (!domain)
+ return -ENOMEM;
+
+ return 0;
+}
+#else
+static inline int mbigen_acpi_create_domain(struct platform_device *pdev,
+ struct mbigen_device *mgn_chip)
+{
+ return -ENODEV;
+}
+#endif
+
+static int mbigen_device_probe(struct platform_device *pdev)
+{
+ struct mbigen_device *mgn_chip;
+ struct resource *res;
+ int err;
+
+ mgn_chip = devm_kzalloc(&pdev->dev, sizeof(*mgn_chip), GFP_KERNEL);
+ if (!mgn_chip)
+ return -ENOMEM;
+
+ mgn_chip->pdev = pdev;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ mgn_chip->base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(mgn_chip->base))
+ return PTR_ERR(mgn_chip->base);
+
+ if (IS_ENABLED(CONFIG_OF) && pdev->dev.of_node)
+ err = mbigen_of_create_domain(pdev, mgn_chip);
+ else if (ACPI_COMPANION(&pdev->dev))
+ err = mbigen_acpi_create_domain(pdev, mgn_chip);
+ else
+ err = -EINVAL;
+
+ if (err) {
+ dev_err(&pdev->dev, "Failed to create mbi-gen@%p irqdomain",
+ mgn_chip->base);
+ return err;
+ }
+
platform_set_drvdata(pdev, mgn_chip);
return 0;
}
@@ -290,11 +364,17 @@
};
MODULE_DEVICE_TABLE(of, mbigen_of_match);
+static const struct acpi_device_id mbigen_acpi_match[] = {
+ { "HISI0152", 0 },
+ {}
+};
+MODULE_DEVICE_TABLE(acpi, mbigen_acpi_match);
+
static struct platform_driver mbigen_platform_driver = {
.driver = {
.name = "Hisilicon MBIGEN-V2",
- .owner = THIS_MODULE,
.of_match_table = mbigen_of_match,
+ .acpi_match_table = ACPI_PTR(mbigen_acpi_match),
},
.probe = mbigen_device_probe,
};
diff --git a/drivers/irqchip/irq-mips-gic.c b/drivers/irqchip/irq-mips-gic.c
index cd20df1..eb7fbe1 100644
--- a/drivers/irqchip/irq-mips-gic.c
+++ b/drivers/irqchip/irq-mips-gic.c
@@ -29,25 +29,12 @@
DECLARE_BITMAP(pcpu_mask, GIC_MAX_INTRS);
};
-struct gic_irq_spec {
- enum {
- GIC_DEVICE,
- GIC_IPI
- } type;
-
- union {
- struct cpumask *ipimask;
- unsigned int hwirq;
- };
-};
-
static unsigned long __gic_base_addr;
static void __iomem *gic_base;
static struct gic_pcpu_mask pcpu_masks[NR_CPUS];
static DEFINE_SPINLOCK(gic_lock);
static struct irq_domain *gic_irq_domain;
-static struct irq_domain *gic_dev_domain;
static struct irq_domain *gic_ipi_domain;
static int gic_shared_intrs;
static int gic_vpes;
@@ -55,6 +42,7 @@
static unsigned int timer_cpu_pin;
static struct irq_chip gic_level_irq_controller, gic_edge_irq_controller;
DECLARE_BITMAP(ipi_resrv, GIC_MAX_INTRS);
+DECLARE_BITMAP(ipi_available, GIC_MAX_INTRS);
static void __gic_irq_dispatch(void);
@@ -693,132 +681,7 @@
return 0;
}
-static int gic_setup_dev_chip(struct irq_domain *d, unsigned int virq,
- unsigned int hwirq)
-{
- struct irq_chip *chip;
- int err;
-
- if (hwirq >= GIC_SHARED_HWIRQ_BASE) {
- err = irq_domain_set_hwirq_and_chip(d, virq, hwirq,
- &gic_level_irq_controller,
- NULL);
- } else {
- switch (GIC_HWIRQ_TO_LOCAL(hwirq)) {
- case GIC_LOCAL_INT_TIMER:
- case GIC_LOCAL_INT_PERFCTR:
- case GIC_LOCAL_INT_FDC:
- /*
- * HACK: These are all really percpu interrupts, but
- * the rest of the MIPS kernel code does not use the
- * percpu IRQ API for them.
- */
- chip = &gic_all_vpes_local_irq_controller;
- irq_set_handler(virq, handle_percpu_irq);
- break;
-
- default:
- chip = &gic_local_irq_controller;
- irq_set_handler(virq, handle_percpu_devid_irq);
- irq_set_percpu_devid(virq);
- break;
- }
-
- err = irq_domain_set_hwirq_and_chip(d, virq, hwirq,
- chip, NULL);
- }
-
- return err;
-}
-
-static int gic_irq_domain_alloc(struct irq_domain *d, unsigned int virq,
- unsigned int nr_irqs, void *arg)
-{
- struct gic_irq_spec *spec = arg;
- irq_hw_number_t hwirq, base_hwirq;
- int cpu, ret, i;
-
- if (spec->type == GIC_DEVICE) {
- /* verify that shared irqs don't conflict with an IPI irq */
- if ((spec->hwirq >= GIC_SHARED_HWIRQ_BASE) &&
- test_bit(GIC_HWIRQ_TO_SHARED(spec->hwirq), ipi_resrv))
- return -EBUSY;
-
- return gic_setup_dev_chip(d, virq, spec->hwirq);
- } else {
- base_hwirq = find_first_bit(ipi_resrv, gic_shared_intrs);
- if (base_hwirq == gic_shared_intrs) {
- return -ENOMEM;
- }
-
- /* check that we have enough space */
- for (i = base_hwirq; i < nr_irqs; i++) {
- if (!test_bit(i, ipi_resrv))
- return -EBUSY;
- }
- bitmap_clear(ipi_resrv, base_hwirq, nr_irqs);
-
- /* map the hwirq for each cpu consecutively */
- i = 0;
- for_each_cpu(cpu, spec->ipimask) {
- hwirq = GIC_SHARED_TO_HWIRQ(base_hwirq + i);
-
- ret = irq_domain_set_hwirq_and_chip(d, virq + i, hwirq,
- &gic_level_irq_controller,
- NULL);
- if (ret)
- goto error;
-
- irq_set_handler(virq + i, handle_level_irq);
-
- ret = gic_shared_irq_domain_map(d, virq + i, hwirq, cpu);
- if (ret)
- goto error;
-
- i++;
- }
-
- /*
- * tell the parent about the base hwirq we allocated so it can
- * set its own domain data
- */
- spec->hwirq = base_hwirq;
- }
-
- return 0;
-error:
- bitmap_set(ipi_resrv, base_hwirq, nr_irqs);
- return ret;
-}
-
-void gic_irq_domain_free(struct irq_domain *d, unsigned int virq,
- unsigned int nr_irqs)
-{
- irq_hw_number_t base_hwirq;
- struct irq_data *data;
-
- data = irq_get_irq_data(virq);
- if (!data)
- return;
-
- base_hwirq = GIC_HWIRQ_TO_SHARED(irqd_to_hwirq(data));
- bitmap_set(ipi_resrv, base_hwirq, nr_irqs);
-}
-
-int gic_irq_domain_match(struct irq_domain *d, struct device_node *node,
- enum irq_domain_bus_token bus_token)
-{
- /* this domain should'nt be accessed directly */
- return 0;
-}
-
-static const struct irq_domain_ops gic_irq_domain_ops = {
- .alloc = gic_irq_domain_alloc,
- .free = gic_irq_domain_free,
- .match = gic_irq_domain_match,
-};
-
-static int gic_dev_domain_xlate(struct irq_domain *d, struct device_node *ctrlr,
+static int gic_irq_domain_xlate(struct irq_domain *d, struct device_node *ctrlr,
const u32 *intspec, unsigned int intsize,
irq_hw_number_t *out_hwirq,
unsigned int *out_type)
@@ -837,58 +700,82 @@
return 0;
}
-static int gic_dev_domain_alloc(struct irq_domain *d, unsigned int virq,
+static int gic_irq_domain_map(struct irq_domain *d, unsigned int virq,
+ irq_hw_number_t hwirq)
+{
+ int err;
+
+ if (hwirq >= GIC_SHARED_HWIRQ_BASE) {
+ /* verify that shared irqs don't conflict with an IPI irq */
+ if (test_bit(GIC_HWIRQ_TO_SHARED(hwirq), ipi_resrv))
+ return -EBUSY;
+
+ err = irq_domain_set_hwirq_and_chip(d, virq, hwirq,
+ &gic_level_irq_controller,
+ NULL);
+ if (err)
+ return err;
+
+ return gic_shared_irq_domain_map(d, virq, hwirq, 0);
+ }
+
+ switch (GIC_HWIRQ_TO_LOCAL(hwirq)) {
+ case GIC_LOCAL_INT_TIMER:
+ case GIC_LOCAL_INT_PERFCTR:
+ case GIC_LOCAL_INT_FDC:
+ /*
+ * HACK: These are all really percpu interrupts, but
+ * the rest of the MIPS kernel code does not use the
+ * percpu IRQ API for them.
+ */
+ err = irq_domain_set_hwirq_and_chip(d, virq, hwirq,
+ &gic_all_vpes_local_irq_controller,
+ NULL);
+ if (err)
+ return err;
+
+ irq_set_handler(virq, handle_percpu_irq);
+ break;
+
+ default:
+ err = irq_domain_set_hwirq_and_chip(d, virq, hwirq,
+ &gic_local_irq_controller,
+ NULL);
+ if (err)
+ return err;
+
+ irq_set_handler(virq, handle_percpu_devid_irq);
+ irq_set_percpu_devid(virq);
+ break;
+ }
+
+ return gic_local_irq_domain_map(d, virq, hwirq);
+}
+
+static int gic_irq_domain_alloc(struct irq_domain *d, unsigned int virq,
unsigned int nr_irqs, void *arg)
{
struct irq_fwspec *fwspec = arg;
- struct gic_irq_spec spec = {
- .type = GIC_DEVICE,
- };
- int i, ret;
+ irq_hw_number_t hwirq;
if (fwspec->param[0] == GIC_SHARED)
- spec.hwirq = GIC_SHARED_TO_HWIRQ(fwspec->param[1]);
+ hwirq = GIC_SHARED_TO_HWIRQ(fwspec->param[1]);
else
- spec.hwirq = GIC_LOCAL_TO_HWIRQ(fwspec->param[1]);
+ hwirq = GIC_LOCAL_TO_HWIRQ(fwspec->param[1]);
- ret = irq_domain_alloc_irqs_parent(d, virq, nr_irqs, &spec);
- if (ret)
- return ret;
-
- for (i = 0; i < nr_irqs; i++) {
- ret = gic_setup_dev_chip(d, virq + i, spec.hwirq + i);
- if (ret)
- goto error;
- }
-
- return 0;
-
-error:
- irq_domain_free_irqs_parent(d, virq, nr_irqs);
- return ret;
+ return gic_irq_domain_map(d, virq, hwirq);
}
-void gic_dev_domain_free(struct irq_domain *d, unsigned int virq,
+void gic_irq_domain_free(struct irq_domain *d, unsigned int virq,
unsigned int nr_irqs)
{
- /* no real allocation is done for dev irqs, so no need to free anything */
- return;
}
-static void gic_dev_domain_activate(struct irq_domain *domain,
- struct irq_data *d)
-{
- if (GIC_HWIRQ_TO_LOCAL(d->hwirq) < GIC_NUM_LOCAL_INTRS)
- gic_local_irq_domain_map(domain, d->irq, d->hwirq);
- else
- gic_shared_irq_domain_map(domain, d->irq, d->hwirq, 0);
-}
-
-static struct irq_domain_ops gic_dev_domain_ops = {
- .xlate = gic_dev_domain_xlate,
- .alloc = gic_dev_domain_alloc,
- .free = gic_dev_domain_free,
- .activate = gic_dev_domain_activate,
+static const struct irq_domain_ops gic_irq_domain_ops = {
+ .xlate = gic_irq_domain_xlate,
+ .alloc = gic_irq_domain_alloc,
+ .free = gic_irq_domain_free,
+ .map = gic_irq_domain_map,
};
static int gic_ipi_domain_xlate(struct irq_domain *d, struct device_node *ctrlr,
@@ -910,20 +797,32 @@
unsigned int nr_irqs, void *arg)
{
struct cpumask *ipimask = arg;
- struct gic_irq_spec spec = {
- .type = GIC_IPI,
- .ipimask = ipimask
- };
- int ret, i;
+ irq_hw_number_t hwirq, base_hwirq;
+ int cpu, ret, i;
- ret = irq_domain_alloc_irqs_parent(d, virq, nr_irqs, &spec);
- if (ret)
- return ret;
+ base_hwirq = find_first_bit(ipi_available, gic_shared_intrs);
+ if (base_hwirq == gic_shared_intrs)
+ return -ENOMEM;
- /* the parent should have set spec.hwirq to the base_hwirq it allocated */
- for (i = 0; i < nr_irqs; i++) {
- ret = irq_domain_set_hwirq_and_chip(d, virq + i,
- GIC_SHARED_TO_HWIRQ(spec.hwirq + i),
+ /* check that we have enough space */
+ for (i = base_hwirq; i < nr_irqs; i++) {
+ if (!test_bit(i, ipi_available))
+ return -EBUSY;
+ }
+ bitmap_clear(ipi_available, base_hwirq, nr_irqs);
+
+ /* map the hwirq for each cpu consecutively */
+ i = 0;
+ for_each_cpu(cpu, ipimask) {
+ hwirq = GIC_SHARED_TO_HWIRQ(base_hwirq + i);
+
+ ret = irq_domain_set_hwirq_and_chip(d, virq + i, hwirq,
+ &gic_edge_irq_controller,
+ NULL);
+ if (ret)
+ goto error;
+
+ ret = irq_domain_set_hwirq_and_chip(d->parent, virq + i, hwirq,
&gic_edge_irq_controller,
NULL);
if (ret)
@@ -932,18 +831,32 @@
ret = irq_set_irq_type(virq + i, IRQ_TYPE_EDGE_RISING);
if (ret)
goto error;
+
+ ret = gic_shared_irq_domain_map(d, virq + i, hwirq, cpu);
+ if (ret)
+ goto error;
+
+ i++;
}
return 0;
error:
- irq_domain_free_irqs_parent(d, virq, nr_irqs);
+ bitmap_set(ipi_available, base_hwirq, nr_irqs);
return ret;
}
void gic_ipi_domain_free(struct irq_domain *d, unsigned int virq,
unsigned int nr_irqs)
{
- irq_domain_free_irqs_parent(d, virq, nr_irqs);
+ irq_hw_number_t base_hwirq;
+ struct irq_data *data;
+
+ data = irq_get_irq_data(virq);
+ if (!data)
+ return;
+
+ base_hwirq = GIC_HWIRQ_TO_SHARED(irqd_to_hwirq(data));
+ bitmap_set(ipi_available, base_hwirq, nr_irqs);
}
int gic_ipi_domain_match(struct irq_domain *d, struct device_node *node,
@@ -968,38 +881,6 @@
.match = gic_ipi_domain_match,
};
-static void __init gic_map_single_int(struct device_node *node,
- unsigned int irq)
-{
- unsigned int linux_irq;
- struct irq_fwspec local_int_fwspec = {
- .fwnode = &node->fwnode,
- .param_count = 3,
- .param = {
- [0] = GIC_LOCAL,
- [1] = irq,
- [2] = IRQ_TYPE_NONE,
- },
- };
-
- if (!gic_local_irq_is_routable(irq))
- return;
-
- linux_irq = irq_create_fwspec_mapping(&local_int_fwspec);
- WARN_ON(!linux_irq);
-}
-
-static void __init gic_map_interrupts(struct device_node *node)
-{
- gic_map_single_int(node, GIC_LOCAL_INT_WD);
- gic_map_single_int(node, GIC_LOCAL_INT_COMPARE);
- gic_map_single_int(node, GIC_LOCAL_INT_TIMER);
- gic_map_single_int(node, GIC_LOCAL_INT_PERFCTR);
- gic_map_single_int(node, GIC_LOCAL_INT_SWINT0);
- gic_map_single_int(node, GIC_LOCAL_INT_SWINT1);
- gic_map_single_int(node, GIC_LOCAL_INT_FDC);
-}
-
static void __init __gic_init(unsigned long gic_base_addr,
unsigned long gic_addrspace_size,
unsigned int cpu_vec, unsigned int irqbase,
@@ -1071,13 +952,6 @@
panic("Failed to add GIC IRQ domain");
gic_irq_domain->name = "mips-gic-irq";
- gic_dev_domain = irq_domain_add_hierarchy(gic_irq_domain, 0,
- GIC_NUM_LOCAL_INTRS + gic_shared_intrs,
- node, &gic_dev_domain_ops, NULL);
- if (!gic_dev_domain)
- panic("Failed to add GIC DEV domain");
- gic_dev_domain->name = "mips-gic-dev";
-
gic_ipi_domain = irq_domain_add_hierarchy(gic_irq_domain,
IRQ_DOMAIN_FLAG_IPI_PER_CPU,
GIC_NUM_LOCAL_INTRS + gic_shared_intrs,
@@ -1098,8 +972,8 @@
2 * gic_vpes);
}
+ bitmap_copy(ipi_available, ipi_resrv, GIC_MAX_INTRS);
gic_basic_init();
- gic_map_interrupts(node);
}
void __init gic_init(unsigned long gic_base_addr,
diff --git a/drivers/irqchip/irq-moxart.c b/drivers/irqchip/irq-moxart.c
deleted file mode 100644
index a24b06a..0000000
--- a/drivers/irqchip/irq-moxart.c
+++ /dev/null
@@ -1,116 +0,0 @@
-/*
- * MOXA ART SoCs IRQ chip driver.
- *
- * Copyright (C) 2013 Jonas Jensen
- *
- * Jonas Jensen <jonas.jensen@gmail.com>
- *
- * This file is licensed under the terms of the GNU General Public
- * License version 2. This program is licensed "as is" without any
- * warranty of any kind, whether express or implied.
- */
-
-#include <linux/io.h>
-#include <linux/irq.h>
-#include <linux/irqchip.h>
-#include <linux/of.h>
-#include <linux/of_address.h>
-#include <linux/of_irq.h>
-#include <linux/irqdomain.h>
-
-#include <asm/exception.h>
-
-#define IRQ_SOURCE_REG 0
-#define IRQ_MASK_REG 0x04
-#define IRQ_CLEAR_REG 0x08
-#define IRQ_MODE_REG 0x0c
-#define IRQ_LEVEL_REG 0x10
-#define IRQ_STATUS_REG 0x14
-
-#define FIQ_SOURCE_REG 0x20
-#define FIQ_MASK_REG 0x24
-#define FIQ_CLEAR_REG 0x28
-#define FIQ_MODE_REG 0x2c
-#define FIQ_LEVEL_REG 0x30
-#define FIQ_STATUS_REG 0x34
-
-
-struct moxart_irq_data {
- void __iomem *base;
- struct irq_domain *domain;
- unsigned int interrupt_mask;
-};
-
-static struct moxart_irq_data intc;
-
-static void __exception_irq_entry handle_irq(struct pt_regs *regs)
-{
- u32 irqstat;
- int hwirq;
-
- irqstat = readl(intc.base + IRQ_STATUS_REG);
-
- while (irqstat) {
- hwirq = ffs(irqstat) - 1;
- handle_IRQ(irq_linear_revmap(intc.domain, hwirq), regs);
- irqstat &= ~(1 << hwirq);
- }
-}
-
-static int __init moxart_of_intc_init(struct device_node *node,
- struct device_node *parent)
-{
- unsigned int clr = IRQ_NOREQUEST | IRQ_NOPROBE | IRQ_NOAUTOEN;
- int ret;
- struct irq_chip_generic *gc;
-
- intc.base = of_iomap(node, 0);
- if (!intc.base) {
- pr_err("%s: unable to map IC registers\n",
- node->full_name);
- return -EINVAL;
- }
-
- intc.domain = irq_domain_add_linear(node, 32, &irq_generic_chip_ops,
- intc.base);
- if (!intc.domain) {
- pr_err("%s: unable to create IRQ domain\n", node->full_name);
- return -EINVAL;
- }
-
- ret = irq_alloc_domain_generic_chips(intc.domain, 32, 1,
- "MOXARTINTC", handle_edge_irq,
- clr, 0, IRQ_GC_INIT_MASK_CACHE);
- if (ret) {
- pr_err("%s: could not allocate generic chip\n",
- node->full_name);
- irq_domain_remove(intc.domain);
- return -EINVAL;
- }
-
- ret = of_property_read_u32(node, "interrupt-mask",
- &intc.interrupt_mask);
- if (ret)
- pr_err("%s: could not read interrupt-mask DT property\n",
- node->full_name);
-
- gc = irq_get_domain_generic_chip(intc.domain, 0);
-
- gc->reg_base = intc.base;
- gc->chip_types[0].regs.mask = IRQ_MASK_REG;
- gc->chip_types[0].regs.ack = IRQ_CLEAR_REG;
- gc->chip_types[0].chip.irq_ack = irq_gc_ack_set_bit;
- gc->chip_types[0].chip.irq_mask = irq_gc_mask_clr_bit;
- gc->chip_types[0].chip.irq_unmask = irq_gc_mask_set_bit;
-
- writel(0, intc.base + IRQ_MASK_REG);
- writel(0xffffffff, intc.base + IRQ_CLEAR_REG);
-
- writel(intc.interrupt_mask, intc.base + IRQ_MODE_REG);
- writel(intc.interrupt_mask, intc.base + IRQ_LEVEL_REG);
-
- set_handle_irq(handle_irq);
-
- return 0;
-}
-IRQCHIP_DECLARE(moxa_moxart_ic, "moxa,moxart-ic", moxart_of_intc_init);
diff --git a/drivers/irqchip/irq-mtk-cirq.c b/drivers/irqchip/irq-mtk-cirq.c
new file mode 100644
index 0000000..18c65c1
--- /dev/null
+++ b/drivers/irqchip/irq-mtk-cirq.c
@@ -0,0 +1,306 @@
+/*
+ * Copyright (c) 2016 MediaTek Inc.
+ * Author: Youlin.Pei <youlin.pei@mediatek.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/irq.h>
+#include <linux/irqchip.h>
+#include <linux/irqdomain.h>
+#include <linux/of.h>
+#include <linux/of_irq.h>
+#include <linux/of_address.h>
+#include <linux/slab.h>
+#include <linux/syscore_ops.h>
+
+#define CIRQ_ACK 0x40
+#define CIRQ_MASK_SET 0xc0
+#define CIRQ_MASK_CLR 0x100
+#define CIRQ_SENS_SET 0x180
+#define CIRQ_SENS_CLR 0x1c0
+#define CIRQ_POL_SET 0x240
+#define CIRQ_POL_CLR 0x280
+#define CIRQ_CONTROL 0x300
+
+#define CIRQ_EN 0x1
+#define CIRQ_EDGE 0x2
+#define CIRQ_FLUSH 0x4
+
+struct mtk_cirq_chip_data {
+ void __iomem *base;
+ unsigned int ext_irq_start;
+ unsigned int ext_irq_end;
+ struct irq_domain *domain;
+};
+
+static struct mtk_cirq_chip_data *cirq_data;
+
+static void mtk_cirq_write_mask(struct irq_data *data, unsigned int offset)
+{
+ struct mtk_cirq_chip_data *chip_data = data->chip_data;
+ unsigned int cirq_num = data->hwirq;
+ u32 mask = 1 << (cirq_num % 32);
+
+ writel_relaxed(mask, chip_data->base + offset + (cirq_num / 32) * 4);
+}
+
+static void mtk_cirq_mask(struct irq_data *data)
+{
+ mtk_cirq_write_mask(data, CIRQ_MASK_SET);
+ irq_chip_mask_parent(data);
+}
+
+static void mtk_cirq_unmask(struct irq_data *data)
+{
+ mtk_cirq_write_mask(data, CIRQ_MASK_CLR);
+ irq_chip_unmask_parent(data);
+}
+
+static int mtk_cirq_set_type(struct irq_data *data, unsigned int type)
+{
+ int ret;
+
+ switch (type & IRQ_TYPE_SENSE_MASK) {
+ case IRQ_TYPE_EDGE_FALLING:
+ mtk_cirq_write_mask(data, CIRQ_POL_CLR);
+ mtk_cirq_write_mask(data, CIRQ_SENS_CLR);
+ break;
+ case IRQ_TYPE_EDGE_RISING:
+ mtk_cirq_write_mask(data, CIRQ_POL_SET);
+ mtk_cirq_write_mask(data, CIRQ_SENS_CLR);
+ break;
+ case IRQ_TYPE_LEVEL_LOW:
+ mtk_cirq_write_mask(data, CIRQ_POL_CLR);
+ mtk_cirq_write_mask(data, CIRQ_SENS_SET);
+ break;
+ case IRQ_TYPE_LEVEL_HIGH:
+ mtk_cirq_write_mask(data, CIRQ_POL_SET);
+ mtk_cirq_write_mask(data, CIRQ_SENS_SET);
+ break;
+ default:
+ break;
+ }
+
+ data = data->parent_data;
+ ret = data->chip->irq_set_type(data, type);
+ return ret;
+}
+
+static struct irq_chip mtk_cirq_chip = {
+ .name = "MT_CIRQ",
+ .irq_mask = mtk_cirq_mask,
+ .irq_unmask = mtk_cirq_unmask,
+ .irq_eoi = irq_chip_eoi_parent,
+ .irq_set_type = mtk_cirq_set_type,
+ .irq_retrigger = irq_chip_retrigger_hierarchy,
+#ifdef CONFIG_SMP
+ .irq_set_affinity = irq_chip_set_affinity_parent,
+#endif
+};
+
+static int mtk_cirq_domain_translate(struct irq_domain *d,
+ struct irq_fwspec *fwspec,
+ unsigned long *hwirq,
+ unsigned int *type)
+{
+ if (is_of_node(fwspec->fwnode)) {
+ if (fwspec->param_count != 3)
+ return -EINVAL;
+
+ /* No PPI should point to this domain */
+ if (fwspec->param[0] != 0)
+ return -EINVAL;
+
+ /* cirq support irq number check */
+ if (fwspec->param[1] < cirq_data->ext_irq_start ||
+ fwspec->param[1] > cirq_data->ext_irq_end)
+ return -EINVAL;
+
+ *hwirq = fwspec->param[1] - cirq_data->ext_irq_start;
+ *type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK;
+ return 0;
+ }
+
+ return -EINVAL;
+}
+
+static int mtk_cirq_domain_alloc(struct irq_domain *domain, unsigned int virq,
+ unsigned int nr_irqs, void *arg)
+{
+ int ret;
+ irq_hw_number_t hwirq;
+ unsigned int type;
+ struct irq_fwspec *fwspec = arg;
+ struct irq_fwspec parent_fwspec = *fwspec;
+
+ ret = mtk_cirq_domain_translate(domain, fwspec, &hwirq, &type);
+ if (ret)
+ return ret;
+
+ if (WARN_ON(nr_irqs != 1))
+ return -EINVAL;
+
+ irq_domain_set_hwirq_and_chip(domain, virq, hwirq,
+ &mtk_cirq_chip,
+ domain->host_data);
+
+ parent_fwspec.fwnode = domain->parent->fwnode;
+ return irq_domain_alloc_irqs_parent(domain, virq, nr_irqs,
+ &parent_fwspec);
+}
+
+static const struct irq_domain_ops cirq_domain_ops = {
+ .translate = mtk_cirq_domain_translate,
+ .alloc = mtk_cirq_domain_alloc,
+ .free = irq_domain_free_irqs_common,
+};
+
+#ifdef CONFIG_PM_SLEEP
+static int mtk_cirq_suspend(void)
+{
+ u32 value, mask;
+ unsigned int irq, hwirq_num;
+ bool pending, masked;
+ int i, pendret, maskret;
+
+ /*
+ * When external interrupts happened, CIRQ will record the status
+ * even CIRQ is not enabled. When execute flush command, CIRQ will
+ * resend the signals according to the status. So if don't clear the
+ * status, CIRQ will resend the wrong signals.
+ *
+ * arch_suspend_disable_irqs() will be called before CIRQ suspend
+ * callback. If clear all the status simply, the external interrupts
+ * which happened between arch_suspend_disable_irqs and CIRQ suspend
+ * callback will be lost. Using following steps to avoid this issue;
+ *
+ * - Iterate over all the CIRQ supported interrupts;
+ * - For each interrupt, inspect its pending and masked status at GIC
+ * level;
+ * - If pending and unmasked, it happened between
+ * arch_suspend_disable_irqs and CIRQ suspend callback, don't ACK
+ * it. Otherwise, ACK it.
+ */
+ hwirq_num = cirq_data->ext_irq_end - cirq_data->ext_irq_start + 1;
+ for (i = 0; i < hwirq_num; i++) {
+ irq = irq_find_mapping(cirq_data->domain, i);
+ if (irq) {
+ pendret = irq_get_irqchip_state(irq,
+ IRQCHIP_STATE_PENDING,
+ &pending);
+
+ maskret = irq_get_irqchip_state(irq,
+ IRQCHIP_STATE_MASKED,
+ &masked);
+
+ if (pendret == 0 && maskret == 0 &&
+ (pending && !masked))
+ continue;
+ }
+
+ mask = 1 << (i % 32);
+ writel_relaxed(mask, cirq_data->base + CIRQ_ACK + (i / 32) * 4);
+ }
+
+ /* set edge_only mode, record edge-triggerd interrupts */
+ /* enable cirq */
+ value = readl_relaxed(cirq_data->base + CIRQ_CONTROL);
+ value |= (CIRQ_EDGE | CIRQ_EN);
+ writel_relaxed(value, cirq_data->base + CIRQ_CONTROL);
+
+ return 0;
+}
+
+static void mtk_cirq_resume(void)
+{
+ u32 value;
+
+ /* flush recored interrupts, will send signals to parent controller */
+ value = readl_relaxed(cirq_data->base + CIRQ_CONTROL);
+ writel_relaxed(value | CIRQ_FLUSH, cirq_data->base + CIRQ_CONTROL);
+
+ /* disable cirq */
+ value = readl_relaxed(cirq_data->base + CIRQ_CONTROL);
+ value &= ~(CIRQ_EDGE | CIRQ_EN);
+ writel_relaxed(value, cirq_data->base + CIRQ_CONTROL);
+}
+
+static struct syscore_ops mtk_cirq_syscore_ops = {
+ .suspend = mtk_cirq_suspend,
+ .resume = mtk_cirq_resume,
+};
+
+static void mtk_cirq_syscore_init(void)
+{
+ register_syscore_ops(&mtk_cirq_syscore_ops);
+}
+#else
+static inline void mtk_cirq_syscore_init(void) {}
+#endif
+
+static int __init mtk_cirq_of_init(struct device_node *node,
+ struct device_node *parent)
+{
+ struct irq_domain *domain, *domain_parent;
+ unsigned int irq_num;
+ int ret;
+
+ domain_parent = irq_find_host(parent);
+ if (!domain_parent) {
+ pr_err("mtk_cirq: interrupt-parent not found\n");
+ return -EINVAL;
+ }
+
+ cirq_data = kzalloc(sizeof(*cirq_data), GFP_KERNEL);
+ if (!cirq_data)
+ return -ENOMEM;
+
+ cirq_data->base = of_iomap(node, 0);
+ if (!cirq_data->base) {
+ pr_err("mtk_cirq: unable to map cirq register\n");
+ ret = -ENXIO;
+ goto out_free;
+ }
+
+ ret = of_property_read_u32_index(node, "mediatek,ext-irq-range", 0,
+ &cirq_data->ext_irq_start);
+ if (ret)
+ goto out_unmap;
+
+ ret = of_property_read_u32_index(node, "mediatek,ext-irq-range", 1,
+ &cirq_data->ext_irq_end);
+ if (ret)
+ goto out_unmap;
+
+ irq_num = cirq_data->ext_irq_end - cirq_data->ext_irq_start + 1;
+ domain = irq_domain_add_hierarchy(domain_parent, 0,
+ irq_num, node,
+ &cirq_domain_ops, cirq_data);
+ if (!domain) {
+ ret = -ENOMEM;
+ goto out_unmap;
+ }
+ cirq_data->domain = domain;
+
+ mtk_cirq_syscore_init();
+
+ return 0;
+
+out_unmap:
+ iounmap(cirq_data->base);
+out_free:
+ kfree(cirq_data);
+ return ret;
+}
+
+IRQCHIP_DECLARE(mtk_cirq, "mediatek,mtk-cirq", mtk_cirq_of_init);
diff --git a/drivers/irqchip/irq-mtk-sysirq.c b/drivers/irqchip/irq-mtk-sysirq.c
index 63ac73b..eeac512 100644
--- a/drivers/irqchip/irq-mtk-sysirq.c
+++ b/drivers/irqchip/irq-mtk-sysirq.c
@@ -24,22 +24,29 @@
struct mtk_sysirq_chip_data {
spinlock_t lock;
- void __iomem *intpol_base;
+ u32 nr_intpol_bases;
+ void __iomem **intpol_bases;
+ u32 *intpol_words;
+ u8 *intpol_idx;
+ u16 *which_word;
};
static int mtk_sysirq_set_type(struct irq_data *data, unsigned int type)
{
irq_hw_number_t hwirq = data->hwirq;
struct mtk_sysirq_chip_data *chip_data = data->chip_data;
+ u8 intpol_idx = chip_data->intpol_idx[hwirq];
+ void __iomem *base;
u32 offset, reg_index, value;
unsigned long flags;
int ret;
+ base = chip_data->intpol_bases[intpol_idx];
+ reg_index = chip_data->which_word[hwirq];
offset = hwirq & 0x1f;
- reg_index = hwirq >> 5;
spin_lock_irqsave(&chip_data->lock, flags);
- value = readl_relaxed(chip_data->intpol_base + reg_index * 4);
+ value = readl_relaxed(base + reg_index * 4);
if (type == IRQ_TYPE_LEVEL_LOW || type == IRQ_TYPE_EDGE_FALLING) {
if (type == IRQ_TYPE_LEVEL_LOW)
type = IRQ_TYPE_LEVEL_HIGH;
@@ -49,7 +56,8 @@
} else {
value &= ~(1 << offset);
}
- writel(value, chip_data->intpol_base + reg_index * 4);
+
+ writel_relaxed(value, base + reg_index * 4);
data = data->parent_data;
ret = data->chip->irq_set_type(data, type);
@@ -124,8 +132,7 @@
{
struct irq_domain *domain, *domain_parent;
struct mtk_sysirq_chip_data *chip_data;
- int ret, size, intpol_num;
- struct resource res;
+ int ret, size, intpol_num = 0, nr_intpol_bases = 0, i = 0;
domain_parent = irq_find_host(parent);
if (!domain_parent) {
@@ -133,36 +140,103 @@
return -EINVAL;
}
- ret = of_address_to_resource(node, 0, &res);
- if (ret)
- return ret;
-
chip_data = kzalloc(sizeof(*chip_data), GFP_KERNEL);
if (!chip_data)
return -ENOMEM;
- size = resource_size(&res);
- intpol_num = size * 8;
- chip_data->intpol_base = ioremap(res.start, size);
- if (!chip_data->intpol_base) {
- pr_err("mtk_sysirq: unable to map sysirq register\n");
- ret = -ENXIO;
- goto out_free;
+ while (of_get_address(node, i++, NULL, NULL))
+ nr_intpol_bases++;
+
+ if (nr_intpol_bases == 0) {
+ pr_err("mtk_sysirq: base address not specified\n");
+ ret = -EINVAL;
+ goto out_free_chip;
+ }
+
+ chip_data->intpol_words = kcalloc(nr_intpol_bases,
+ sizeof(*chip_data->intpol_words),
+ GFP_KERNEL);
+ if (!chip_data->intpol_words) {
+ ret = -ENOMEM;
+ goto out_free_chip;
+ }
+
+ chip_data->intpol_bases = kcalloc(nr_intpol_bases,
+ sizeof(*chip_data->intpol_bases),
+ GFP_KERNEL);
+ if (!chip_data->intpol_bases) {
+ ret = -ENOMEM;
+ goto out_free_intpol_words;
+ }
+
+ for (i = 0; i < nr_intpol_bases; i++) {
+ struct resource res;
+
+ ret = of_address_to_resource(node, i, &res);
+ size = resource_size(&res);
+ intpol_num += size * 8;
+ chip_data->intpol_words[i] = size / 4;
+ chip_data->intpol_bases[i] = of_iomap(node, i);
+ if (ret || !chip_data->intpol_bases[i]) {
+ pr_err("%s: couldn't map region %d\n",
+ node->full_name, i);
+ ret = -ENODEV;
+ goto out_free_intpol;
+ }
+ }
+
+ chip_data->intpol_idx = kcalloc(intpol_num,
+ sizeof(*chip_data->intpol_idx),
+ GFP_KERNEL);
+ if (!chip_data->intpol_idx) {
+ ret = -ENOMEM;
+ goto out_free_intpol;
+ }
+
+ chip_data->which_word = kcalloc(intpol_num,
+ sizeof(*chip_data->which_word),
+ GFP_KERNEL);
+ if (!chip_data->which_word) {
+ ret = -ENOMEM;
+ goto out_free_intpol_idx;
+ }
+
+ /*
+ * assign an index of the intpol_bases for each irq
+ * to set it fast later
+ */
+ for (i = 0; i < intpol_num ; i++) {
+ u32 word = i / 32, j;
+
+ for (j = 0; word >= chip_data->intpol_words[j] ; j++)
+ word -= chip_data->intpol_words[j];
+
+ chip_data->intpol_idx[i] = j;
+ chip_data->which_word[i] = word;
}
domain = irq_domain_add_hierarchy(domain_parent, 0, intpol_num, node,
&sysirq_domain_ops, chip_data);
if (!domain) {
ret = -ENOMEM;
- goto out_unmap;
+ goto out_free_which_word;
}
spin_lock_init(&chip_data->lock);
return 0;
-out_unmap:
- iounmap(chip_data->intpol_base);
-out_free:
+out_free_which_word:
+ kfree(chip_data->which_word);
+out_free_intpol_idx:
+ kfree(chip_data->intpol_idx);
+out_free_intpol:
+ for (i = 0; i < nr_intpol_bases; i++)
+ if (chip_data->intpol_bases[i])
+ iounmap(chip_data->intpol_bases[i]);
+ kfree(chip_data->intpol_bases);
+out_free_intpol_words:
+ kfree(chip_data->intpol_words);
+out_free_chip:
kfree(chip_data);
return ret;
}
diff --git a/drivers/isdn/divert/isdn_divert.c b/drivers/isdn/divert/isdn_divert.c
index 50749a7..060d357 100644
--- a/drivers/isdn/divert/isdn_divert.c
+++ b/drivers/isdn/divert/isdn_divert.c
@@ -157,10 +157,8 @@
/* allocate mem for information struct */
if (!(cs = kmalloc(sizeof(struct call_struc), GFP_ATOMIC)))
return (-ENOMEM); /* no memory */
- init_timer(&cs->timer);
+ setup_timer(&cs->timer, deflect_timer_expire, (ulong)cs);
cs->info[0] = '\0';
- cs->timer.function = deflect_timer_expire;
- cs->timer.data = (ulong) cs; /* pointer to own structure */
cs->ics.driver = drvid;
cs->ics.command = ISDN_CMD_PROT_IO; /* protocol specific io */
cs->ics.arg = DSS1_CMD_INVOKE; /* invoke supplementary service */
@@ -452,10 +450,9 @@
return (0); /* no external deflection needed */
if (!(cs = kmalloc(sizeof(struct call_struc), GFP_ATOMIC)))
return (0); /* no memory */
- init_timer(&cs->timer);
+ setup_timer(&cs->timer, deflect_timer_expire,
+ (ulong)cs);
cs->info[0] = '\0';
- cs->timer.function = deflect_timer_expire;
- cs->timer.data = (ulong) cs; /* pointer to own structure */
cs->ics = *ic; /* copy incoming data */
if (!cs->ics.parm.setup.phone[0]) strcpy(cs->ics.parm.setup.phone, "0");
diff --git a/drivers/isdn/hardware/eicon/divasi.c b/drivers/isdn/hardware/eicon/divasi.c
index cb88090..c610495 100644
--- a/drivers/isdn/hardware/eicon/divasi.c
+++ b/drivers/isdn/hardware/eicon/divasi.c
@@ -300,9 +300,8 @@
p_os = (diva_um_idi_os_context_t *) diva_um_id_get_os_context(e);
init_waitqueue_head(&p_os->read_wait);
init_waitqueue_head(&p_os->close_wait);
- init_timer(&p_os->diva_timer_id);
- p_os->diva_timer_id.function = (void *) diva_um_timer_function;
- p_os->diva_timer_id.data = (unsigned long) p_os;
+ setup_timer(&p_os->diva_timer_id, (void *)diva_um_timer_function,
+ (unsigned long)p_os);
p_os->aborted = 0;
p_os->adapter_nr = adapter_nr;
return (1);
diff --git a/drivers/isdn/hardware/mISDN/Kconfig b/drivers/isdn/hardware/mISDN/Kconfig
index 09df54f..fda912b 100644
--- a/drivers/isdn/hardware/mISDN/Kconfig
+++ b/drivers/isdn/hardware/mISDN/Kconfig
@@ -13,7 +13,7 @@
config MISDN_HFCMULTI
tristate "Support for HFC multiport cards (HFC-4S/8S/E1)"
- depends on PCI || 8xx
+ depends on PCI || CPM1
depends on MISDN
help
Enable support for cards with Cologne Chip AG's HFC multiport
@@ -27,8 +27,8 @@
bool "Support for XHFC embedded board in HFC multiport driver"
depends on MISDN
depends on MISDN_HFCMULTI
- depends on 8xx
- default 8xx
+ depends on CPM1
+ default CPM1
help
Enable support for the XHFC embedded solution from Speech Design.
diff --git a/drivers/isdn/hardware/mISDN/hfc_multi_8xx.h b/drivers/isdn/hardware/mISDN/hfc_multi_8xx.h
index 0eafe9f..8a25474 100644
--- a/drivers/isdn/hardware/mISDN/hfc_multi_8xx.h
+++ b/drivers/isdn/hardware/mISDN/hfc_multi_8xx.h
@@ -6,7 +6,7 @@
*
*/
-#include <asm/8xx_immap.h>
+#include <asm/cpm1.h>
/* Change this to the value used by your board */
#ifndef IMAP_ADDR
diff --git a/drivers/isdn/hardware/mISDN/hfcmulti.c b/drivers/isdn/hardware/mISDN/hfcmulti.c
index 480c2d7..961c07e 100644
--- a/drivers/isdn/hardware/mISDN/hfcmulti.c
+++ b/drivers/isdn/hardware/mISDN/hfcmulti.c
@@ -3878,9 +3878,8 @@
if (hc->dnum[pt]) {
mode_hfcmulti(hc, dch->slot, dch->dev.D.protocol,
-1, 0, -1, 0);
- dch->timer.function = (void *) hfcmulti_dbusy_timer;
- dch->timer.data = (long) dch;
- init_timer(&dch->timer);
+ setup_timer(&dch->timer, (void *)hfcmulti_dbusy_timer,
+ (long)dch);
}
for (i = 1; i <= 31; i++) {
if (!((1 << i) & hc->bmask[pt])) /* skip unused chan */
@@ -3986,9 +3985,8 @@
hc->chan[i].slot_rx = -1;
hc->chan[i].conf = -1;
mode_hfcmulti(hc, i, dch->dev.D.protocol, -1, 0, -1, 0);
- dch->timer.function = (void *) hfcmulti_dbusy_timer;
- dch->timer.data = (long) dch;
- init_timer(&dch->timer);
+ setup_timer(&dch->timer, (void *)hfcmulti_dbusy_timer,
+ (long)dch);
hc->chan[i - 2].slot_tx = -1;
hc->chan[i - 2].slot_rx = -1;
hc->chan[i - 2].conf = -1;
diff --git a/drivers/isdn/hardware/mISDN/hfcpci.c b/drivers/isdn/hardware/mISDN/hfcpci.c
index ff48da6..5dc246d 100644
--- a/drivers/isdn/hardware/mISDN/hfcpci.c
+++ b/drivers/isdn/hardware/mISDN/hfcpci.c
@@ -1717,9 +1717,8 @@
inithfcpci(struct hfc_pci *hc)
{
printk(KERN_DEBUG "inithfcpci: entered\n");
- hc->dch.timer.function = (void *) hfcpci_dbusy_timer;
- hc->dch.timer.data = (long) &hc->dch;
- init_timer(&hc->dch.timer);
+ setup_timer(&hc->dch.timer, (void *)hfcpci_dbusy_timer,
+ (long)&hc->dch);
hc->chanlimit = 2;
mode_hfcpci(&hc->bch[0], 1, -1);
mode_hfcpci(&hc->bch[1], 2, -1);
@@ -2044,9 +2043,7 @@
Write_hfc(hc, HFCPCI_INT_M1, hc->hw.int_m1);
/* At this point the needed PCI config is done */
/* fifos are still not enabled */
- hc->hw.timer.function = (void *) hfcpci_Timer;
- hc->hw.timer.data = (long) hc;
- init_timer(&hc->hw.timer);
+ setup_timer(&hc->hw.timer, (void *)hfcpci_Timer, (long)hc);
/* default PCM master */
test_and_set_bit(HFC_CFG_MASTER, &hc->cfg);
return 0;
diff --git a/drivers/isdn/hardware/mISDN/mISDNipac.c b/drivers/isdn/hardware/mISDN/mISDNipac.c
index 77dec28..6742b0d 100644
--- a/drivers/isdn/hardware/mISDN/mISDNipac.c
+++ b/drivers/isdn/hardware/mISDN/mISDNipac.c
@@ -796,9 +796,8 @@
}
isac->mon_tx = NULL;
isac->mon_rx = NULL;
- isac->dch.timer.function = (void *) dbusy_timer_handler;
- isac->dch.timer.data = (long)isac;
- init_timer(&isac->dch.timer);
+ setup_timer(&isac->dch.timer, (void *)dbusy_timer_handler,
+ (long)isac);
isac->mocr = 0xaa;
if (isac->type & IPAC_TYPE_ISACX) {
/* Disable all IRQ */
diff --git a/drivers/isdn/hardware/mISDN/mISDNisar.c b/drivers/isdn/hardware/mISDN/mISDNisar.c
index feafa91..5b07859 100644
--- a/drivers/isdn/hardware/mISDN/mISDNisar.c
+++ b/drivers/isdn/hardware/mISDN/mISDNisar.c
@@ -1635,13 +1635,11 @@
}
if (isar->version != 1)
return -EINVAL;
- isar->ch[0].ftimer.function = &ftimer_handler;
- isar->ch[0].ftimer.data = (long)&isar->ch[0];
- init_timer(&isar->ch[0].ftimer);
+ setup_timer(&isar->ch[0].ftimer, &ftimer_handler,
+ (long)&isar->ch[0]);
test_and_set_bit(FLG_INITIALIZED, &isar->ch[0].bch.Flags);
- isar->ch[1].ftimer.function = &ftimer_handler;
- isar->ch[1].ftimer.data = (long)&isar->ch[1];
- init_timer(&isar->ch[1].ftimer);
+ setup_timer(&isar->ch[1].ftimer, &ftimer_handler,
+ (long)&isar->ch[1]);
test_and_set_bit(FLG_INITIALIZED, &isar->ch[1].bch.Flags);
return 0;
}
diff --git a/drivers/isdn/hardware/mISDN/w6692.c b/drivers/isdn/hardware/mISDN/w6692.c
index 3b067ea..3052c83 100644
--- a/drivers/isdn/hardware/mISDN/w6692.c
+++ b/drivers/isdn/hardware/mISDN/w6692.c
@@ -852,9 +852,8 @@
{
u8 val;
- card->dch.timer.function = (void *)dbusy_timer_handler;
- card->dch.timer.data = (u_long)&card->dch;
- init_timer(&card->dch.timer);
+ setup_timer(&card->dch.timer, (void *)dbusy_timer_handler,
+ (u_long)&card->dch);
w6692_mode(&card->bc[0], ISDN_P_NONE);
w6692_mode(&card->bc[1], ISDN_P_NONE);
WriteW6692(card, W_D_CTL, 0x00);
diff --git a/drivers/isdn/hisax/amd7930_fn.c b/drivers/isdn/hisax/amd7930_fn.c
index 36817e0..3a4c2f9 100644
--- a/drivers/isdn/hisax/amd7930_fn.c
+++ b/drivers/isdn/hisax/amd7930_fn.c
@@ -789,7 +789,5 @@
void setup_Amd7930(struct IsdnCardState *cs)
{
INIT_WORK(&cs->tqueue, Amd7930_bh);
- cs->dbusytimer.function = (void *) dbusy_timer_handler;
- cs->dbusytimer.data = (long) cs;
- init_timer(&cs->dbusytimer);
+ setup_timer(&cs->dbusytimer, (void *)dbusy_timer_handler, (long)cs);
}
diff --git a/drivers/isdn/hisax/arcofi.c b/drivers/isdn/hisax/arcofi.c
index 29ec2df..9826bad 100644
--- a/drivers/isdn/hisax/arcofi.c
+++ b/drivers/isdn/hisax/arcofi.c
@@ -125,9 +125,7 @@
void
init_arcofi(struct IsdnCardState *cs) {
- cs->dc.isac.arcofitimer.function = (void *) arcofi_timer;
- cs->dc.isac.arcofitimer.data = (long) cs;
- init_timer(&cs->dc.isac.arcofitimer);
+ setup_timer(&cs->dc.isac.arcofitimer, (void *)arcofi_timer, (long)cs);
init_waitqueue_head(&cs->dc.isac.arcofi_wait);
test_and_set_bit(HW_ARCOFI, &cs->HW_Flags);
}
diff --git a/drivers/isdn/hisax/diva.c b/drivers/isdn/hisax/diva.c
index 4fc90de..079336e 100644
--- a/drivers/isdn/hisax/diva.c
+++ b/drivers/isdn/hisax/diva.c
@@ -976,9 +976,8 @@
printk(KERN_INFO "Diva: IPACX Design Id: %x\n",
MemReadISAC_IPACX(cs, IPACX_ID) & 0x3F);
} else { /* DIVA 2.0 */
- cs->hw.diva.tl.function = (void *) diva_led_handler;
- cs->hw.diva.tl.data = (long) cs;
- init_timer(&cs->hw.diva.tl);
+ setup_timer(&cs->hw.diva.tl, (void *)diva_led_handler,
+ (long)cs);
cs->readisac = &ReadISAC;
cs->writeisac = &WriteISAC;
cs->readisacfifo = &ReadISACfifo;
diff --git a/drivers/isdn/hisax/elsa.c b/drivers/isdn/hisax/elsa.c
index d8ef64d..03bc5d5 100644
--- a/drivers/isdn/hisax/elsa.c
+++ b/drivers/isdn/hisax/elsa.c
@@ -1147,9 +1147,7 @@
init_arcofi(cs);
#endif
setup_isac(cs);
- cs->hw.elsa.tl.function = (void *) elsa_led_handler;
- cs->hw.elsa.tl.data = (long) cs;
- init_timer(&cs->hw.elsa.tl);
+ setup_timer(&cs->hw.elsa.tl, (void *)elsa_led_handler, (long)cs);
/* Teste Timer */
if (cs->hw.elsa.timer) {
byteout(cs->hw.elsa.trig, 0xff);
diff --git a/drivers/isdn/hisax/fsm.c b/drivers/isdn/hisax/fsm.c
index c7a9471..d63266f 100644
--- a/drivers/isdn/hisax/fsm.c
+++ b/drivers/isdn/hisax/fsm.c
@@ -98,13 +98,11 @@
FsmInitTimer(struct FsmInst *fi, struct FsmTimer *ft)
{
ft->fi = fi;
- ft->tl.function = (void *) FsmExpireTimer;
- ft->tl.data = (long) ft;
#if FSM_TIMER_DEBUG
if (ft->fi->debug)
ft->fi->printdebug(ft->fi, "FsmInitTimer %lx", (long) ft);
#endif
- init_timer(&ft->tl);
+ setup_timer(&ft->tl, (void *)FsmExpireTimer, (long)ft);
}
void
diff --git a/drivers/isdn/hisax/hfc4s8s_l1.c b/drivers/isdn/hisax/hfc4s8s_l1.c
index e034ed8..90f051c 100644
--- a/drivers/isdn/hisax/hfc4s8s_l1.c
+++ b/drivers/isdn/hisax/hfc4s8s_l1.c
@@ -1396,9 +1396,8 @@
l1p = hw->l1 + i;
spin_lock_init(&l1p->lock);
l1p->hw = hw;
- l1p->l1_timer.function = (void *) hfc_l1_timer;
- l1p->l1_timer.data = (long) (l1p);
- init_timer(&l1p->l1_timer);
+ setup_timer(&l1p->l1_timer, (void *)hfc_l1_timer,
+ (long)(l1p));
l1p->st_num = i;
skb_queue_head_init(&l1p->d_tx_queue);
l1p->d_if.ifc.priv = hw->l1 + i;
diff --git a/drivers/isdn/hisax/hfc_2bds0.c b/drivers/isdn/hisax/hfc_2bds0.c
index a756e5c..ad8597a 100644
--- a/drivers/isdn/hisax/hfc_2bds0.c
+++ b/drivers/isdn/hisax/hfc_2bds0.c
@@ -1073,8 +1073,6 @@
cs->writeisacfifo = &dummyf;
cs->BC_Read_Reg = &ReadReg;
cs->BC_Write_Reg = &WriteReg;
- cs->dbusytimer.function = (void *) hfc_dbusy_timer;
- cs->dbusytimer.data = (long) cs;
- init_timer(&cs->dbusytimer);
+ setup_timer(&cs->dbusytimer, (void *)hfc_dbusy_timer, (long)cs);
INIT_WORK(&cs->tqueue, hfcd_bh);
}
diff --git a/drivers/isdn/hisax/hfc_pci.c b/drivers/isdn/hisax/hfc_pci.c
index 90449e1..f9ca35c 100644
--- a/drivers/isdn/hisax/hfc_pci.c
+++ b/drivers/isdn/hisax/hfc_pci.c
@@ -1582,9 +1582,7 @@
cs->bcs[1].BC_SetStack = setstack_2b;
cs->bcs[0].BC_Close = close_hfcpci;
cs->bcs[1].BC_Close = close_hfcpci;
- cs->dbusytimer.function = (void *) hfcpci_dbusy_timer;
- cs->dbusytimer.data = (long) cs;
- init_timer(&cs->dbusytimer);
+ setup_timer(&cs->dbusytimer, (void *)hfcpci_dbusy_timer, (long)cs);
mode_hfcpci(cs->bcs, 0, 0);
mode_hfcpci(cs->bcs + 1, 0, 1);
}
@@ -1746,9 +1744,7 @@
cs->BC_Write_Reg = NULL;
cs->irq_func = &hfcpci_interrupt;
cs->irq_flags |= IRQF_SHARED;
- cs->hw.hfcpci.timer.function = (void *) hfcpci_Timer;
- cs->hw.hfcpci.timer.data = (long) cs;
- init_timer(&cs->hw.hfcpci.timer);
+ setup_timer(&cs->hw.hfcpci.timer, (void *)hfcpci_Timer, (long)cs);
cs->cardmsg = &hfcpci_card_msg;
cs->auxcmd = &hfcpci_auxcmd;
diff --git a/drivers/isdn/hisax/hfc_sx.c b/drivers/isdn/hisax/hfc_sx.c
index 13b2151..3aef8e1 100644
--- a/drivers/isdn/hisax/hfc_sx.c
+++ b/drivers/isdn/hisax/hfc_sx.c
@@ -1495,9 +1495,7 @@
} else
return (0); /* no valid card type */
- cs->dbusytimer.function = (void *) hfcsx_dbusy_timer;
- cs->dbusytimer.data = (long) cs;
- init_timer(&cs->dbusytimer);
+ setup_timer(&cs->dbusytimer, (void *)hfcsx_dbusy_timer, (long)cs);
INIT_WORK(&cs->tqueue, hfcsx_bh);
cs->readisac = NULL;
cs->writeisac = NULL;
@@ -1507,11 +1505,9 @@
cs->BC_Write_Reg = NULL;
cs->irq_func = &hfcsx_interrupt;
- cs->hw.hfcsx.timer.function = (void *) hfcsx_Timer;
- cs->hw.hfcsx.timer.data = (long) cs;
cs->hw.hfcsx.b_fifo_size = 0; /* fifo size still unknown */
cs->hw.hfcsx.cirm = ccd_sp_irqtab[cs->irq & 0xF]; /* RAM not evaluated */
- init_timer(&cs->hw.hfcsx.timer);
+ setup_timer(&cs->hw.hfcsx.timer, (void *)hfcsx_Timer, (long)cs);
reset_hfcsx(cs);
cs->cardmsg = &hfcsx_card_msg;
diff --git a/drivers/isdn/hisax/hfc_usb.c b/drivers/isdn/hisax/hfc_usb.c
index 678bd52..6dbd1f1 100644
--- a/drivers/isdn/hisax/hfc_usb.c
+++ b/drivers/isdn/hisax/hfc_usb.c
@@ -1165,14 +1165,10 @@
hfc->old_led_state = 0;
/* init the t3 timer */
- init_timer(&hfc->t3_timer);
- hfc->t3_timer.data = (long) hfc;
- hfc->t3_timer.function = (void *) l1_timer_expire_t3;
+ setup_timer(&hfc->t3_timer, (void *)l1_timer_expire_t3, (long)hfc);
/* init the t4 timer */
- init_timer(&hfc->t4_timer);
- hfc->t4_timer.data = (long) hfc;
- hfc->t4_timer.function = (void *) l1_timer_expire_t4;
+ setup_timer(&hfc->t4_timer, (void *)l1_timer_expire_t4, (long)hfc);
/* init the background machinery for control requests */
hfc->ctrl_read.bRequestType = 0xc0;
diff --git a/drivers/isdn/hisax/hfcscard.c b/drivers/isdn/hisax/hfcscard.c
index 394da64..4672870 100644
--- a/drivers/isdn/hisax/hfcscard.c
+++ b/drivers/isdn/hisax/hfcscard.c
@@ -253,9 +253,7 @@
outb(0x57, cs->hw.hfcD.addr | 1);
}
set_cs_func(cs);
- cs->hw.hfcD.timer.function = (void *) hfcs_Timer;
- cs->hw.hfcD.timer.data = (long) cs;
- init_timer(&cs->hw.hfcD.timer);
+ setup_timer(&cs->hw.hfcD.timer, (void *)hfcs_Timer, (long)cs);
cs->cardmsg = &hfcs_card_msg;
cs->irq_func = &hfcs_interrupt;
return (1);
diff --git a/drivers/isdn/hisax/icc.c b/drivers/isdn/hisax/icc.c
index 96d1df0..c7c3797 100644
--- a/drivers/isdn/hisax/icc.c
+++ b/drivers/isdn/hisax/icc.c
@@ -676,7 +676,5 @@
void setup_icc(struct IsdnCardState *cs)
{
INIT_WORK(&cs->tqueue, icc_bh);
- cs->dbusytimer.function = (void *) dbusy_timer_handler;
- cs->dbusytimer.data = (long) cs;
- init_timer(&cs->dbusytimer);
+ setup_timer(&cs->dbusytimer, (void *)dbusy_timer_handler, (long)cs);
}
diff --git a/drivers/isdn/hisax/ipacx.c b/drivers/isdn/hisax/ipacx.c
index 9cc26b4..43effe7 100644
--- a/drivers/isdn/hisax/ipacx.c
+++ b/drivers/isdn/hisax/ipacx.c
@@ -424,9 +424,7 @@
cs->setstack_d = dch_setstack;
- cs->dbusytimer.function = (void *) dbusy_timer_handler;
- cs->dbusytimer.data = (long) cs;
- init_timer(&cs->dbusytimer);
+ setup_timer(&cs->dbusytimer, (void *)dbusy_timer_handler, (long)cs);
cs->writeisac(cs, IPACX_TR_CONF0, 0x00); // clear LDD
cs->writeisac(cs, IPACX_TR_CONF2, 0x00); // enable transmitter
diff --git a/drivers/isdn/hisax/isac.c b/drivers/isdn/hisax/isac.c
index df7e05c..4273b45 100644
--- a/drivers/isdn/hisax/isac.c
+++ b/drivers/isdn/hisax/isac.c
@@ -677,7 +677,5 @@
void setup_isac(struct IsdnCardState *cs)
{
INIT_WORK(&cs->tqueue, isac_bh);
- cs->dbusytimer.function = (void *) dbusy_timer_handler;
- cs->dbusytimer.data = (long) cs;
- init_timer(&cs->dbusytimer);
+ setup_timer(&cs->dbusytimer, (void *)dbusy_timer_handler, (long)cs);
}
diff --git a/drivers/isdn/hisax/isar.c b/drivers/isdn/hisax/isar.c
index f4956c7..0dc60b2 100644
--- a/drivers/isdn/hisax/isar.c
+++ b/drivers/isdn/hisax/isar.c
@@ -1902,10 +1902,8 @@
cs->bcs[1].BC_SetStack = setstack_isar;
cs->bcs[0].BC_Close = close_isarstate;
cs->bcs[1].BC_Close = close_isarstate;
- cs->bcs[0].hw.isar.ftimer.function = (void *) ftimer_handler;
- cs->bcs[0].hw.isar.ftimer.data = (long) &cs->bcs[0];
- init_timer(&cs->bcs[0].hw.isar.ftimer);
- cs->bcs[1].hw.isar.ftimer.function = (void *) ftimer_handler;
- cs->bcs[1].hw.isar.ftimer.data = (long) &cs->bcs[1];
- init_timer(&cs->bcs[1].hw.isar.ftimer);
+ setup_timer(&cs->bcs[0].hw.isar.ftimer, (void *)ftimer_handler,
+ (long)&cs->bcs[0]);
+ setup_timer(&cs->bcs[1].hw.isar.ftimer, (void *)ftimer_handler,
+ (long)&cs->bcs[1]);
}
diff --git a/drivers/isdn/hisax/isdnl3.c b/drivers/isdn/hisax/isdnl3.c
index c754706..569ce52 100644
--- a/drivers/isdn/hisax/isdnl3.c
+++ b/drivers/isdn/hisax/isdnl3.c
@@ -169,9 +169,7 @@
L3InitTimer(struct l3_process *pc, struct L3Timer *t)
{
t->pc = pc;
- t->tl.function = (void *) L3ExpireTimer;
- t->tl.data = (long) t;
- init_timer(&t->tl);
+ setup_timer(&t->tl, (void *)L3ExpireTimer, (long)t);
}
void
diff --git a/drivers/isdn/hisax/teleint.c b/drivers/isdn/hisax/teleint.c
index bf64754..950399f 100644
--- a/drivers/isdn/hisax/teleint.c
+++ b/drivers/isdn/hisax/teleint.c
@@ -278,9 +278,7 @@
cs->bcs[0].hw.hfc.send = NULL;
cs->bcs[1].hw.hfc.send = NULL;
cs->hw.hfc.fifosize = 7 * 1024 + 512;
- cs->hw.hfc.timer.function = (void *) TeleInt_Timer;
- cs->hw.hfc.timer.data = (long) cs;
- init_timer(&cs->hw.hfc.timer);
+ setup_timer(&cs->hw.hfc.timer, (void *)TeleInt_Timer, (long)cs);
if (!request_region(cs->hw.hfc.addr, 2, "TeleInt isdn")) {
printk(KERN_WARNING
"HiSax: TeleInt config port %x-%x already in use\n",
diff --git a/drivers/isdn/hisax/w6692.c b/drivers/isdn/hisax/w6692.c
index a858955..c99f0ec 100644
--- a/drivers/isdn/hisax/w6692.c
+++ b/drivers/isdn/hisax/w6692.c
@@ -901,9 +901,8 @@
if (part & 1) {
cs->setstack_d = setstack_W6692;
cs->DC_Close = DC_Close_W6692;
- cs->dbusytimer.function = (void *) dbusy_timer_handler;
- cs->dbusytimer.data = (long) cs;
- init_timer(&cs->dbusytimer);
+ setup_timer(&cs->dbusytimer, (void *)dbusy_timer_handler,
+ (long)cs);
resetW6692(cs);
ph_command(cs, W_L1CMD_RST);
cs->dc.w6692.ph_state = W_L1CMD_RST;
diff --git a/drivers/isdn/i4l/isdn_ppp.c b/drivers/isdn/i4l/isdn_ppp.c
index 9c1e8ad..d07dd51 100644
--- a/drivers/isdn/i4l/isdn_ppp.c
+++ b/drivers/isdn/i4l/isdn_ppp.c
@@ -2370,9 +2370,8 @@
rs->state = CCPResetIdle;
rs->is = is;
rs->id = id;
- init_timer(&rs->timer);
- rs->timer.data = (unsigned long)rs;
- rs->timer.function = isdn_ppp_ccp_timer_callback;
+ setup_timer(&rs->timer, isdn_ppp_ccp_timer_callback,
+ (unsigned long)rs);
is->reset->rs[id] = rs;
}
return rs;
diff --git a/drivers/isdn/i4l/isdn_tty.c b/drivers/isdn/i4l/isdn_tty.c
index 1b16955..ddd8207 100644
--- a/drivers/isdn/i4l/isdn_tty.c
+++ b/drivers/isdn/i4l/isdn_tty.c
@@ -1812,9 +1812,8 @@
info->isdn_channel = -1;
info->drv_index = -1;
info->xmit_size = ISDN_SERIAL_XMIT_SIZE;
- init_timer(&info->nc_timer);
- info->nc_timer.function = isdn_tty_modem_do_ncarrier;
- info->nc_timer.data = (unsigned long) info;
+ setup_timer(&info->nc_timer, isdn_tty_modem_do_ncarrier,
+ (unsigned long)info);
skb_queue_head_init(&info->xmit_queue);
#ifdef CONFIG_ISDN_AUDIO
skb_queue_head_init(&info->dtmf_queue);
diff --git a/drivers/isdn/mISDN/dsp_core.c b/drivers/isdn/mISDN/dsp_core.c
index 9b85295..880e9d3 100644
--- a/drivers/isdn/mISDN/dsp_core.c
+++ b/drivers/isdn/mISDN/dsp_core.c
@@ -1092,9 +1092,7 @@
ndsp->pcm_bank_tx = -1;
ndsp->hfc_conf = -1; /* current conference number */
/* set tone timer */
- ndsp->tone.tl.function = (void *)dsp_tone_timeout;
- ndsp->tone.tl.data = (long) ndsp;
- init_timer(&ndsp->tone.tl);
+ setup_timer(&ndsp->tone.tl, (void *)dsp_tone_timeout, (long)ndsp);
if (dtmfthreshold < 20 || dtmfthreshold > 500)
dtmfthreshold = 200;
diff --git a/drivers/isdn/mISDN/fsm.c b/drivers/isdn/mISDN/fsm.c
index 26477d4..78fc5d5 100644
--- a/drivers/isdn/mISDN/fsm.c
+++ b/drivers/isdn/mISDN/fsm.c
@@ -110,13 +110,11 @@
mISDN_FsmInitTimer(struct FsmInst *fi, struct FsmTimer *ft)
{
ft->fi = fi;
- ft->tl.function = (void *) FsmExpireTimer;
- ft->tl.data = (long) ft;
#if FSM_TIMER_DEBUG
if (ft->fi->debug)
ft->fi->printdebug(ft->fi, "mISDN_FsmInitTimer %lx", (long) ft);
#endif
- init_timer(&ft->tl);
+ setup_timer(&ft->tl, (void *)FsmExpireTimer, (long)ft);
}
EXPORT_SYMBOL(mISDN_FsmInitTimer);
diff --git a/drivers/isdn/mISDN/l1oip_core.c b/drivers/isdn/mISDN/l1oip_core.c
index 6ceca7d..6be2041 100644
--- a/drivers/isdn/mISDN/l1oip_core.c
+++ b/drivers/isdn/mISDN/l1oip_core.c
@@ -1443,9 +1443,7 @@
hc->keep_tl.expires = jiffies + 2 * HZ; /* two seconds first time */
add_timer(&hc->keep_tl);
- hc->timeout_tl.function = (void *)l1oip_timeout;
- hc->timeout_tl.data = (ulong)hc;
- init_timer(&hc->timeout_tl);
+ setup_timer(&hc->timeout_tl, (void *)l1oip_timeout, (ulong)hc);
hc->timeout_on = 0; /* state that we have timer off */
return 0;
diff --git a/drivers/leds/Kconfig b/drivers/leds/Kconfig
index 275f467..6c29998 100644
--- a/drivers/leds/Kconfig
+++ b/drivers/leds/Kconfig
@@ -76,6 +76,15 @@
This option enables support for LEDs connected to the BCM6358
LED HW controller accessed via MMIO registers.
+config LEDS_CPCAP
+ tristate "LED Support for Motorola CPCAP"
+ depends on LEDS_CLASS
+ depends on MFD_CPCAP
+ depends on OF
+ help
+ This option enables support for LEDs offered by Motorola's
+ CPCAP PMIC.
+
config LEDS_LM3530
tristate "LCD Backlight driver for LM3530"
depends on LEDS_CLASS
@@ -126,6 +135,14 @@
This option enables support for the so called "User LED" of
Mikrotik's Routerboard 532.
+config LEDS_MT6323
+ tristate "LED Support for Mediatek MT6323 PMIC"
+ depends on LEDS_CLASS
+ depends on MFD_MT6397
+ help
+ This option enables support for on-chip LED drivers found on
+ Mediatek MT6323 PMIC.
+
config LEDS_S3C24XX
tristate "LED Support for Samsung S3C24XX GPIO LEDs"
depends on LEDS_CLASS
@@ -241,7 +258,6 @@
tristate "LED Support for TI LP3952 2 channel LED driver"
depends on LEDS_CLASS
depends on I2C
- depends on ACPI
depends on GPIOLIB
select REGMAP_I2C
help
@@ -463,15 +479,6 @@
To compile this driver as a module, choose M here: the module will
be called leds-adp5520.
-config LEDS_DELL_NETBOOKS
- tristate "External LED on Dell Business Netbooks"
- depends on LEDS_CLASS
- depends on X86 && ACPI_WMI
- depends on DELL_SMBIOS
- help
- This adds support for the Latitude 2100 and similar
- notebooks that have an external LED.
-
config LEDS_MC13783
tristate "LED Support for MC13XXX PMIC"
depends on LEDS_CLASS
diff --git a/drivers/leds/Makefile b/drivers/leds/Makefile
index 6b82737..45f1339 100644
--- a/drivers/leds/Makefile
+++ b/drivers/leds/Makefile
@@ -11,6 +11,7 @@
obj-$(CONFIG_LEDS_BCM6328) += leds-bcm6328.o
obj-$(CONFIG_LEDS_BCM6358) += leds-bcm6358.o
obj-$(CONFIG_LEDS_BD2802) += leds-bd2802.o
+obj-$(CONFIG_LEDS_CPCAP) += leds-cpcap.o
obj-$(CONFIG_LEDS_LOCOMO) += leds-locomo.o
obj-$(CONFIG_LEDS_LM3530) += leds-lm3530.o
obj-$(CONFIG_LEDS_LM3533) += leds-lm3533.o
@@ -52,7 +53,6 @@
obj-$(CONFIG_LEDS_INTEL_SS4200) += leds-ss4200.o
obj-$(CONFIG_LEDS_LT3593) += leds-lt3593.o
obj-$(CONFIG_LEDS_ADP5520) += leds-adp5520.o
-obj-$(CONFIG_LEDS_DELL_NETBOOKS) += dell-led.o
obj-$(CONFIG_LEDS_MC13783) += leds-mc13783.o
obj-$(CONFIG_LEDS_NS2) += leds-ns2.o
obj-$(CONFIG_LEDS_NETXBIG) += leds-netxbig.o
@@ -72,6 +72,7 @@
obj-$(CONFIG_LEDS_PM8058) += leds-pm8058.o
obj-$(CONFIG_LEDS_MLXCPLD) += leds-mlxcpld.o
obj-$(CONFIG_LEDS_NIC78BX) += leds-nic78bx.o
+obj-$(CONFIG_LEDS_MT6323) += leds-mt6323.o
# LED SPI Drivers
obj-$(CONFIG_LEDS_DAC124S085) += leds-dac124s085.o
diff --git a/drivers/leds/dell-led.c b/drivers/leds/dell-led.c
deleted file mode 100644
index b3d6e9c..0000000
--- a/drivers/leds/dell-led.c
+++ /dev/null
@@ -1,261 +0,0 @@
-/*
- * dell_led.c - Dell LED Driver
- *
- * Copyright (C) 2010 Dell Inc.
- * Louis Davis <louis_davis@dell.com>
- * Jim Dailey <jim_dailey@dell.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as
- * published by the Free Software Foundation.
- *
- */
-
-#include <linux/acpi.h>
-#include <linux/leds.h>
-#include <linux/slab.h>
-#include <linux/module.h>
-#include <linux/dmi.h>
-#include <linux/dell-led.h>
-#include "../platform/x86/dell-smbios.h"
-
-MODULE_AUTHOR("Louis Davis/Jim Dailey");
-MODULE_DESCRIPTION("Dell LED Control Driver");
-MODULE_LICENSE("GPL");
-
-#define DELL_LED_BIOS_GUID "F6E4FE6E-909D-47cb-8BAB-C9F6F2F8D396"
-#define DELL_APP_GUID "A80593CE-A997-11DA-B012-B622A1EF5492"
-MODULE_ALIAS("wmi:" DELL_LED_BIOS_GUID);
-
-/* Error Result Codes: */
-#define INVALID_DEVICE_ID 250
-#define INVALID_PARAMETER 251
-#define INVALID_BUFFER 252
-#define INTERFACE_ERROR 253
-#define UNSUPPORTED_COMMAND 254
-#define UNSPECIFIED_ERROR 255
-
-/* Device ID */
-#define DEVICE_ID_PANEL_BACK 1
-
-/* LED Commands */
-#define CMD_LED_ON 16
-#define CMD_LED_OFF 17
-#define CMD_LED_BLINK 18
-
-#define GLOBAL_MIC_MUTE_ENABLE 0x364
-#define GLOBAL_MIC_MUTE_DISABLE 0x365
-
-static int dell_micmute_led_set(int state)
-{
- struct calling_interface_buffer *buffer;
- struct calling_interface_token *token;
-
- if (!wmi_has_guid(DELL_APP_GUID))
- return -ENODEV;
-
- if (state == 0)
- token = dell_smbios_find_token(GLOBAL_MIC_MUTE_DISABLE);
- else if (state == 1)
- token = dell_smbios_find_token(GLOBAL_MIC_MUTE_ENABLE);
- else
- return -EINVAL;
-
- if (!token)
- return -ENODEV;
-
- buffer = dell_smbios_get_buffer();
- buffer->input[0] = token->location;
- buffer->input[1] = token->value;
- dell_smbios_send_request(1, 0);
- dell_smbios_release_buffer();
-
- return state;
-}
-
-int dell_app_wmi_led_set(int whichled, int on)
-{
- int state = 0;
-
- switch (whichled) {
- case DELL_LED_MICMUTE:
- state = dell_micmute_led_set(on);
- break;
- default:
- pr_warn("led type %x is not supported\n", whichled);
- break;
- }
-
- return state;
-}
-EXPORT_SYMBOL_GPL(dell_app_wmi_led_set);
-
-struct bios_args {
- unsigned char length;
- unsigned char result_code;
- unsigned char device_id;
- unsigned char command;
- unsigned char on_time;
- unsigned char off_time;
-};
-
-static int dell_led_perform_fn(u8 length,
- u8 result_code,
- u8 device_id,
- u8 command,
- u8 on_time,
- u8 off_time)
-{
- struct bios_args *bios_return;
- u8 return_code;
- union acpi_object *obj;
- struct acpi_buffer output = { ACPI_ALLOCATE_BUFFER, NULL };
- struct acpi_buffer input;
- acpi_status status;
-
- struct bios_args args;
- args.length = length;
- args.result_code = result_code;
- args.device_id = device_id;
- args.command = command;
- args.on_time = on_time;
- args.off_time = off_time;
-
- input.length = sizeof(struct bios_args);
- input.pointer = &args;
-
- status = wmi_evaluate_method(DELL_LED_BIOS_GUID,
- 1,
- 1,
- &input,
- &output);
-
- if (ACPI_FAILURE(status))
- return status;
-
- obj = output.pointer;
-
- if (!obj)
- return -EINVAL;
- else if (obj->type != ACPI_TYPE_BUFFER) {
- kfree(obj);
- return -EINVAL;
- }
-
- bios_return = ((struct bios_args *)obj->buffer.pointer);
- return_code = bios_return->result_code;
-
- kfree(obj);
-
- return return_code;
-}
-
-static int led_on(void)
-{
- return dell_led_perform_fn(3, /* Length of command */
- INTERFACE_ERROR, /* Init to INTERFACE_ERROR */
- DEVICE_ID_PANEL_BACK, /* Device ID */
- CMD_LED_ON, /* Command */
- 0, /* not used */
- 0); /* not used */
-}
-
-static int led_off(void)
-{
- return dell_led_perform_fn(3, /* Length of command */
- INTERFACE_ERROR, /* Init to INTERFACE_ERROR */
- DEVICE_ID_PANEL_BACK, /* Device ID */
- CMD_LED_OFF, /* Command */
- 0, /* not used */
- 0); /* not used */
-}
-
-static int led_blink(unsigned char on_eighths,
- unsigned char off_eighths)
-{
- return dell_led_perform_fn(5, /* Length of command */
- INTERFACE_ERROR, /* Init to INTERFACE_ERROR */
- DEVICE_ID_PANEL_BACK, /* Device ID */
- CMD_LED_BLINK, /* Command */
- on_eighths, /* blink on in eigths of a second */
- off_eighths); /* blink off in eights of a second */
-}
-
-static void dell_led_set(struct led_classdev *led_cdev,
- enum led_brightness value)
-{
- if (value == LED_OFF)
- led_off();
- else
- led_on();
-}
-
-static int dell_led_blink(struct led_classdev *led_cdev,
- unsigned long *delay_on,
- unsigned long *delay_off)
-{
- unsigned long on_eighths;
- unsigned long off_eighths;
-
- /* The Dell LED delay is based on 125ms intervals.
- Need to round up to next interval. */
-
- on_eighths = (*delay_on + 124) / 125;
- if (0 == on_eighths)
- on_eighths = 1;
- if (on_eighths > 255)
- on_eighths = 255;
- *delay_on = on_eighths * 125;
-
- off_eighths = (*delay_off + 124) / 125;
- if (0 == off_eighths)
- off_eighths = 1;
- if (off_eighths > 255)
- off_eighths = 255;
- *delay_off = off_eighths * 125;
-
- led_blink(on_eighths, off_eighths);
-
- return 0;
-}
-
-static struct led_classdev dell_led = {
- .name = "dell::lid",
- .brightness = LED_OFF,
- .max_brightness = 1,
- .brightness_set = dell_led_set,
- .blink_set = dell_led_blink,
- .flags = LED_CORE_SUSPENDRESUME,
-};
-
-static int __init dell_led_init(void)
-{
- int error = 0;
-
- if (!wmi_has_guid(DELL_LED_BIOS_GUID) && !wmi_has_guid(DELL_APP_GUID))
- return -ENODEV;
-
- if (wmi_has_guid(DELL_LED_BIOS_GUID)) {
- error = led_off();
- if (error != 0)
- return -ENODEV;
-
- error = led_classdev_register(NULL, &dell_led);
- }
-
- return error;
-}
-
-static void __exit dell_led_exit(void)
-{
- int error = 0;
-
- if (wmi_has_guid(DELL_LED_BIOS_GUID)) {
- error = led_off();
- if (error == 0)
- led_classdev_unregister(&dell_led);
- }
-}
-
-module_init(dell_led_init);
-module_exit(dell_led_exit);
diff --git a/drivers/leds/led-class.c b/drivers/leds/led-class.c
index f2b0a80..b0e2d55 100644
--- a/drivers/leds/led-class.c
+++ b/drivers/leds/led-class.c
@@ -244,11 +244,14 @@
}
/**
- * led_classdev_register - register a new object of led_classdev class.
- * @parent: The device to register.
+ * of_led_classdev_register - register a new object of led_classdev class.
+ *
+ * @parent: parent of LED device
* @led_cdev: the led_classdev structure for this device.
+ * @np: DT node describing this LED
*/
-int led_classdev_register(struct device *parent, struct led_classdev *led_cdev)
+int of_led_classdev_register(struct device *parent, struct device_node *np,
+ struct led_classdev *led_cdev)
{
char name[LED_MAX_NAME_SIZE];
int ret;
@@ -261,6 +264,7 @@
led_cdev, led_cdev->groups, "%s", name);
if (IS_ERR(led_cdev->dev))
return PTR_ERR(led_cdev->dev);
+ led_cdev->dev->of_node = np;
if (ret)
dev_warn(parent, "Led %s renamed to %s due to name collision",
@@ -303,7 +307,7 @@
return 0;
}
-EXPORT_SYMBOL_GPL(led_classdev_register);
+EXPORT_SYMBOL_GPL(of_led_classdev_register);
/**
* led_classdev_unregister - unregisters a object of led_properties class.
@@ -348,12 +352,14 @@
}
/**
- * devm_led_classdev_register - resource managed led_classdev_register()
- * @parent: The device to register.
+ * devm_of_led_classdev_register - resource managed led_classdev_register()
+ *
+ * @parent: parent of LED device
* @led_cdev: the led_classdev structure for this device.
*/
-int devm_led_classdev_register(struct device *parent,
- struct led_classdev *led_cdev)
+int devm_of_led_classdev_register(struct device *parent,
+ struct device_node *np,
+ struct led_classdev *led_cdev)
{
struct led_classdev **dr;
int rc;
@@ -362,7 +368,7 @@
if (!dr)
return -ENOMEM;
- rc = led_classdev_register(parent, led_cdev);
+ rc = of_led_classdev_register(parent, np, led_cdev);
if (rc) {
devres_free(dr);
return rc;
@@ -373,7 +379,7 @@
return 0;
}
-EXPORT_SYMBOL_GPL(devm_led_classdev_register);
+EXPORT_SYMBOL_GPL(devm_of_led_classdev_register);
static int devm_led_classdev_match(struct device *dev, void *res, void *data)
{
diff --git a/drivers/leds/leds-cpcap.c b/drivers/leds/leds-cpcap.c
new file mode 100644
index 0000000..f0f28c4
--- /dev/null
+++ b/drivers/leds/leds-cpcap.c
@@ -0,0 +1,239 @@
+/*
+ * Copyright (c) 2017 Sebastian Reichel <sre@kernel.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 or
+ * later as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/leds.h>
+#include <linux/mfd/motorola-cpcap.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/of_device.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <linux/regulator/consumer.h>
+
+#define CPCAP_LED_NO_CURRENT 0x0001
+
+struct cpcap_led_info {
+ u16 reg;
+ u16 mask;
+ u16 limit;
+ u16 init_mask;
+ u16 init_val;
+};
+
+static const struct cpcap_led_info cpcap_led_red = {
+ .reg = CPCAP_REG_REDC,
+ .mask = 0x03FF,
+ .limit = 31,
+};
+
+static const struct cpcap_led_info cpcap_led_green = {
+ .reg = CPCAP_REG_GREENC,
+ .mask = 0x03FF,
+ .limit = 31,
+};
+
+static const struct cpcap_led_info cpcap_led_blue = {
+ .reg = CPCAP_REG_BLUEC,
+ .mask = 0x03FF,
+ .limit = 31,
+};
+
+/* aux display light */
+static const struct cpcap_led_info cpcap_led_adl = {
+ .reg = CPCAP_REG_ADLC,
+ .mask = 0x000F,
+ .limit = 1,
+ .init_mask = 0x7FFF,
+ .init_val = 0x5FF0,
+};
+
+/* camera privacy led */
+static const struct cpcap_led_info cpcap_led_cp = {
+ .reg = CPCAP_REG_CLEDC,
+ .mask = 0x0007,
+ .limit = 1,
+ .init_mask = 0x03FF,
+ .init_val = 0x0008,
+};
+
+struct cpcap_led {
+ struct led_classdev led;
+ const struct cpcap_led_info *info;
+ struct device *dev;
+ struct regmap *regmap;
+ struct mutex update_lock;
+ struct regulator *vdd;
+ bool powered;
+
+ u32 current_limit;
+};
+
+static u16 cpcap_led_val(u8 current_limit, u8 duty_cycle)
+{
+ current_limit &= 0x1f; /* 5 bit */
+ duty_cycle &= 0x0f; /* 4 bit */
+
+ return current_limit << 4 | duty_cycle;
+}
+
+static int cpcap_led_set_power(struct cpcap_led *led, bool status)
+{
+ int err;
+
+ if (status == led->powered)
+ return 0;
+
+ if (status)
+ err = regulator_enable(led->vdd);
+ else
+ err = regulator_disable(led->vdd);
+
+ if (err) {
+ dev_err(led->dev, "regulator failure: %d", err);
+ return err;
+ }
+
+ led->powered = status;
+
+ return 0;
+}
+
+static int cpcap_led_set(struct led_classdev *ledc, enum led_brightness value)
+{
+ struct cpcap_led *led = container_of(ledc, struct cpcap_led, led);
+ int brightness;
+ int err;
+
+ mutex_lock(&led->update_lock);
+
+ if (value > LED_OFF) {
+ err = cpcap_led_set_power(led, true);
+ if (err)
+ goto exit;
+ }
+
+ if (value == LED_OFF) {
+ /* Avoid HW issue by turning off current before duty cycle */
+ err = regmap_update_bits(led->regmap,
+ led->info->reg, led->info->mask, CPCAP_LED_NO_CURRENT);
+ if (err) {
+ dev_err(led->dev, "regmap failed: %d", err);
+ goto exit;
+ }
+
+ brightness = cpcap_led_val(value, LED_OFF);
+ } else {
+ brightness = cpcap_led_val(value, LED_ON);
+ }
+
+ err = regmap_update_bits(led->regmap, led->info->reg, led->info->mask,
+ brightness);
+ if (err) {
+ dev_err(led->dev, "regmap failed: %d", err);
+ goto exit;
+ }
+
+ if (value == LED_OFF) {
+ err = cpcap_led_set_power(led, false);
+ if (err)
+ goto exit;
+ }
+
+exit:
+ mutex_unlock(&led->update_lock);
+ return err;
+}
+
+static const struct of_device_id cpcap_led_of_match[] = {
+ { .compatible = "motorola,cpcap-led-red", .data = &cpcap_led_red },
+ { .compatible = "motorola,cpcap-led-green", .data = &cpcap_led_green },
+ { .compatible = "motorola,cpcap-led-blue", .data = &cpcap_led_blue },
+ { .compatible = "motorola,cpcap-led-adl", .data = &cpcap_led_adl },
+ { .compatible = "motorola,cpcap-led-cp", .data = &cpcap_led_cp },
+ {},
+};
+MODULE_DEVICE_TABLE(of, cpcap_led_of_match);
+
+static int cpcap_led_probe(struct platform_device *pdev)
+{
+ const struct of_device_id *match;
+ struct cpcap_led *led;
+ int err;
+
+ match = of_match_device(of_match_ptr(cpcap_led_of_match), &pdev->dev);
+ if (!match || !match->data)
+ return -EINVAL;
+
+ led = devm_kzalloc(&pdev->dev, sizeof(*led), GFP_KERNEL);
+ if (!led)
+ return -ENOMEM;
+ platform_set_drvdata(pdev, led);
+ led->info = match->data;
+ led->dev = &pdev->dev;
+
+ if (led->info->reg == 0x0000) {
+ dev_err(led->dev, "Unsupported LED");
+ return -ENODEV;
+ }
+
+ led->regmap = dev_get_regmap(pdev->dev.parent, NULL);
+ if (!led->regmap)
+ return -ENODEV;
+
+ led->vdd = devm_regulator_get(&pdev->dev, "vdd");
+ if (IS_ERR(led->vdd)) {
+ err = PTR_ERR(led->vdd);
+ dev_err(led->dev, "Couldn't get regulator: %d", err);
+ return err;
+ }
+
+ err = device_property_read_string(&pdev->dev, "label", &led->led.name);
+ if (err) {
+ dev_err(led->dev, "Couldn't read LED label: %d", err);
+ return err;
+ }
+
+ if (led->info->init_mask) {
+ err = regmap_update_bits(led->regmap, led->info->reg,
+ led->info->init_mask, led->info->init_val);
+ if (err) {
+ dev_err(led->dev, "regmap failed: %d", err);
+ return err;
+ }
+ }
+
+ mutex_init(&led->update_lock);
+
+ led->led.max_brightness = led->info->limit;
+ led->led.brightness_set_blocking = cpcap_led_set;
+ err = devm_led_classdev_register(&pdev->dev, &led->led);
+ if (err) {
+ dev_err(led->dev, "Couldn't register LED: %d", err);
+ return err;
+ }
+
+ return 0;
+}
+
+static struct platform_driver cpcap_led_driver = {
+ .probe = cpcap_led_probe,
+ .driver = {
+ .name = "cpcap-led",
+ .of_match_table = cpcap_led_of_match,
+ },
+};
+module_platform_driver(cpcap_led_driver);
+
+MODULE_DESCRIPTION("CPCAP LED driver");
+MODULE_AUTHOR("Sebastian Reichel <sre@kernel.org>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/leds/leds-gpio.c b/drivers/leds/leds-gpio.c
index 066fc75..e753ba9 100644
--- a/drivers/leds/leds-gpio.c
+++ b/drivers/leds/leds-gpio.c
@@ -77,7 +77,7 @@
static int create_gpio_led(const struct gpio_led *template,
struct gpio_led_data *led_dat, struct device *parent,
- gpio_blink_set_t blink_set)
+ struct device_node *np, gpio_blink_set_t blink_set)
{
int ret, state;
@@ -139,7 +139,7 @@
if (ret < 0)
return ret;
- return devm_led_classdev_register(parent, &led_dat->cdev);
+ return devm_of_led_classdev_register(parent, np, &led_dat->cdev);
}
struct gpio_leds_priv {
@@ -208,7 +208,7 @@
if (fwnode_property_present(child, "panic-indicator"))
led.panic_indicator = 1;
- ret = create_gpio_led(&led, led_dat, dev, NULL);
+ ret = create_gpio_led(&led, led_dat, dev, np, NULL);
if (ret < 0) {
fwnode_handle_put(child);
return ERR_PTR(ret);
@@ -242,9 +242,9 @@
priv->num_leds = pdata->num_leds;
for (i = 0; i < priv->num_leds; i++) {
- ret = create_gpio_led(&pdata->leds[i],
- &priv->leds[i],
- &pdev->dev, pdata->gpio_blink_set);
+ ret = create_gpio_led(&pdata->leds[i], &priv->leds[i],
+ &pdev->dev, NULL,
+ pdata->gpio_blink_set);
if (ret < 0)
return ret;
}
diff --git a/drivers/leds/leds-lp3952.c b/drivers/leds/leds-lp3952.c
index 4847e89..847f7f2 100644
--- a/drivers/leds/leds-lp3952.c
+++ b/drivers/leds/leds-lp3952.c
@@ -10,7 +10,6 @@
*
*/
-#include <linux/acpi.h>
#include <linux/delay.h>
#include <linux/gpio.h>
#include <linux/i2c.h>
@@ -103,10 +102,11 @@
const char *str;
ret = device_property_read_string(dev, label, &str);
- if (!ret)
- strncpy(dest, str, LP3952_LABEL_MAX_LEN);
+ if (ret)
+ return ret;
- return ret;
+ strncpy(dest, str, LP3952_LABEL_MAX_LEN);
+ return 0;
}
static int lp3952_register_led_classdev(struct lp3952_led_array *priv)
@@ -276,19 +276,9 @@
};
MODULE_DEVICE_TABLE(i2c, lp3952_id);
-#ifdef CONFIG_ACPI
-static const struct acpi_device_id lp3952_acpi_match[] = {
- {"TXNW3952", 0},
- {}
-};
-
-MODULE_DEVICE_TABLE(acpi, lp3952_acpi_match);
-#endif
-
static struct i2c_driver lp3952_i2c_driver = {
.driver = {
.name = LP3952_NAME,
- .acpi_match_table = ACPI_PTR(lp3952_acpi_match),
},
.probe = lp3952_probe,
.remove = lp3952_remove,
diff --git a/drivers/leds/leds-mt6323.c b/drivers/leds/leds-mt6323.c
new file mode 100644
index 0000000..8893c74
--- /dev/null
+++ b/drivers/leds/leds-mt6323.c
@@ -0,0 +1,502 @@
+/*
+ * LED driver for Mediatek MT6323 PMIC
+ *
+ * Copyright (C) 2017 Sean Wang <sean.wang@mediatek.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of
+ * the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+#include <linux/kernel.h>
+#include <linux/leds.h>
+#include <linux/mfd/mt6323/registers.h>
+#include <linux/mfd/mt6397/core.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+
+/*
+ * Register field for MT6323_TOP_CKPDN0 to enable
+ * 32K clock common for LED device.
+ */
+#define MT6323_RG_DRV_32K_CK_PDN BIT(11)
+#define MT6323_RG_DRV_32K_CK_PDN_MASK BIT(11)
+
+/*
+ * Register field for MT6323_TOP_CKPDN2 to enable
+ * individual clock for LED device.
+ */
+#define MT6323_RG_ISINK_CK_PDN(i) BIT(i)
+#define MT6323_RG_ISINK_CK_PDN_MASK(i) BIT(i)
+
+/*
+ * Register field for MT6323_TOP_CKCON1 to select
+ * clock source.
+ */
+#define MT6323_RG_ISINK_CK_SEL_MASK(i) (BIT(10) << (i))
+
+/*
+ * Register for MT6323_ISINK_CON0 to setup the
+ * duty cycle of the blink.
+ */
+#define MT6323_ISINK_CON0(i) (MT6323_ISINK0_CON0 + 0x8 * (i))
+#define MT6323_ISINK_DIM_DUTY_MASK (0x1f << 8)
+#define MT6323_ISINK_DIM_DUTY(i) (((i) << 8) & \
+ MT6323_ISINK_DIM_DUTY_MASK)
+
+/* Register to setup the period of the blink. */
+#define MT6323_ISINK_CON1(i) (MT6323_ISINK0_CON1 + 0x8 * (i))
+#define MT6323_ISINK_DIM_FSEL_MASK (0xffff)
+#define MT6323_ISINK_DIM_FSEL(i) ((i) & MT6323_ISINK_DIM_FSEL_MASK)
+
+/* Register to control the brightness. */
+#define MT6323_ISINK_CON2(i) (MT6323_ISINK0_CON2 + 0x8 * (i))
+#define MT6323_ISINK_CH_STEP_SHIFT 12
+#define MT6323_ISINK_CH_STEP_MASK (0x7 << 12)
+#define MT6323_ISINK_CH_STEP(i) (((i) << 12) & \
+ MT6323_ISINK_CH_STEP_MASK)
+#define MT6323_ISINK_SFSTR0_TC_MASK (0x3 << 1)
+#define MT6323_ISINK_SFSTR0_TC(i) (((i) << 1) & \
+ MT6323_ISINK_SFSTR0_TC_MASK)
+#define MT6323_ISINK_SFSTR0_EN_MASK BIT(0)
+#define MT6323_ISINK_SFSTR0_EN BIT(0)
+
+/* Register to LED channel enablement. */
+#define MT6323_ISINK_CH_EN_MASK(i) BIT(i)
+#define MT6323_ISINK_CH_EN(i) BIT(i)
+
+#define MT6323_MAX_PERIOD 10000
+#define MT6323_MAX_LEDS 4
+#define MT6323_MAX_BRIGHTNESS 6
+#define MT6323_UNIT_DUTY 3125
+#define MT6323_CAL_HW_DUTY(o, p) DIV_ROUND_CLOSEST((o) * 100000ul,\
+ (p) * MT6323_UNIT_DUTY)
+
+struct mt6323_leds;
+
+/**
+ * struct mt6323_led - state container for the LED device
+ * @id: the identifier in MT6323 LED device
+ * @parent: the pointer to MT6323 LED controller
+ * @cdev: LED class device for this LED device
+ * @current_brightness: current state of the LED device
+ */
+struct mt6323_led {
+ int id;
+ struct mt6323_leds *parent;
+ struct led_classdev cdev;
+ enum led_brightness current_brightness;
+};
+
+/**
+ * struct mt6323_leds - state container for holding LED controller
+ * of the driver
+ * @dev: the device pointer
+ * @hw: the underlying hardware providing shared
+ * bus for the register operations
+ * @lock: the lock among process context
+ * @led: the array that contains the state of individual
+ * LED device
+ */
+struct mt6323_leds {
+ struct device *dev;
+ struct mt6397_chip *hw;
+ /* protect among process context */
+ struct mutex lock;
+ struct mt6323_led *led[MT6323_MAX_LEDS];
+};
+
+static int mt6323_led_hw_brightness(struct led_classdev *cdev,
+ enum led_brightness brightness)
+{
+ struct mt6323_led *led = container_of(cdev, struct mt6323_led, cdev);
+ struct mt6323_leds *leds = led->parent;
+ struct regmap *regmap = leds->hw->regmap;
+ u32 con2_mask = 0, con2_val = 0;
+ int ret;
+
+ /*
+ * Setup current output for the corresponding
+ * brightness level.
+ */
+ con2_mask |= MT6323_ISINK_CH_STEP_MASK |
+ MT6323_ISINK_SFSTR0_TC_MASK |
+ MT6323_ISINK_SFSTR0_EN_MASK;
+ con2_val |= MT6323_ISINK_CH_STEP(brightness - 1) |
+ MT6323_ISINK_SFSTR0_TC(2) |
+ MT6323_ISINK_SFSTR0_EN;
+
+ ret = regmap_update_bits(regmap, MT6323_ISINK_CON2(led->id),
+ con2_mask, con2_val);
+ return ret;
+}
+
+static int mt6323_led_hw_off(struct led_classdev *cdev)
+{
+ struct mt6323_led *led = container_of(cdev, struct mt6323_led, cdev);
+ struct mt6323_leds *leds = led->parent;
+ struct regmap *regmap = leds->hw->regmap;
+ unsigned int status;
+ int ret;
+
+ status = MT6323_ISINK_CH_EN(led->id);
+ ret = regmap_update_bits(regmap, MT6323_ISINK_EN_CTRL,
+ MT6323_ISINK_CH_EN_MASK(led->id), ~status);
+ if (ret < 0)
+ return ret;
+
+ usleep_range(100, 300);
+ ret = regmap_update_bits(regmap, MT6323_TOP_CKPDN2,
+ MT6323_RG_ISINK_CK_PDN_MASK(led->id),
+ MT6323_RG_ISINK_CK_PDN(led->id));
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+static enum led_brightness
+mt6323_get_led_hw_brightness(struct led_classdev *cdev)
+{
+ struct mt6323_led *led = container_of(cdev, struct mt6323_led, cdev);
+ struct mt6323_leds *leds = led->parent;
+ struct regmap *regmap = leds->hw->regmap;
+ unsigned int status;
+ int ret;
+
+ ret = regmap_read(regmap, MT6323_TOP_CKPDN2, &status);
+ if (ret < 0)
+ return ret;
+
+ if (status & MT6323_RG_ISINK_CK_PDN_MASK(led->id))
+ return 0;
+
+ ret = regmap_read(regmap, MT6323_ISINK_EN_CTRL, &status);
+ if (ret < 0)
+ return ret;
+
+ if (!(status & MT6323_ISINK_CH_EN(led->id)))
+ return 0;
+
+ ret = regmap_read(regmap, MT6323_ISINK_CON2(led->id), &status);
+ if (ret < 0)
+ return ret;
+
+ return ((status & MT6323_ISINK_CH_STEP_MASK)
+ >> MT6323_ISINK_CH_STEP_SHIFT) + 1;
+}
+
+static int mt6323_led_hw_on(struct led_classdev *cdev,
+ enum led_brightness brightness)
+{
+ struct mt6323_led *led = container_of(cdev, struct mt6323_led, cdev);
+ struct mt6323_leds *leds = led->parent;
+ struct regmap *regmap = leds->hw->regmap;
+ unsigned int status;
+ int ret;
+
+ /*
+ * Setup required clock source, enable the corresponding
+ * clock and channel and let work with continuous blink as
+ * the default.
+ */
+ ret = regmap_update_bits(regmap, MT6323_TOP_CKCON1,
+ MT6323_RG_ISINK_CK_SEL_MASK(led->id), 0);
+ if (ret < 0)
+ return ret;
+
+ status = MT6323_RG_ISINK_CK_PDN(led->id);
+ ret = regmap_update_bits(regmap, MT6323_TOP_CKPDN2,
+ MT6323_RG_ISINK_CK_PDN_MASK(led->id),
+ ~status);
+ if (ret < 0)
+ return ret;
+
+ usleep_range(100, 300);
+
+ ret = regmap_update_bits(regmap, MT6323_ISINK_EN_CTRL,
+ MT6323_ISINK_CH_EN_MASK(led->id),
+ MT6323_ISINK_CH_EN(led->id));
+ if (ret < 0)
+ return ret;
+
+ ret = mt6323_led_hw_brightness(cdev, brightness);
+ if (ret < 0)
+ return ret;
+
+ ret = regmap_update_bits(regmap, MT6323_ISINK_CON0(led->id),
+ MT6323_ISINK_DIM_DUTY_MASK,
+ MT6323_ISINK_DIM_DUTY(31));
+ if (ret < 0)
+ return ret;
+
+ ret = regmap_update_bits(regmap, MT6323_ISINK_CON1(led->id),
+ MT6323_ISINK_DIM_FSEL_MASK,
+ MT6323_ISINK_DIM_FSEL(1000));
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+static int mt6323_led_set_blink(struct led_classdev *cdev,
+ unsigned long *delay_on,
+ unsigned long *delay_off)
+{
+ struct mt6323_led *led = container_of(cdev, struct mt6323_led, cdev);
+ struct mt6323_leds *leds = led->parent;
+ struct regmap *regmap = leds->hw->regmap;
+ unsigned long period;
+ u8 duty_hw;
+ int ret;
+
+ /*
+ * Units are in ms, if over the hardware able
+ * to support, fallback into software blink
+ */
+ period = *delay_on + *delay_off;
+
+ if (period > MT6323_MAX_PERIOD)
+ return -EINVAL;
+
+ /*
+ * LED subsystem requires a default user
+ * friendly blink pattern for the LED so using
+ * 1Hz duty cycle 50% here if without specific
+ * value delay_on and delay off being assigned.
+ */
+ if (!*delay_on && !*delay_off) {
+ *delay_on = 500;
+ *delay_off = 500;
+ }
+
+ /*
+ * Calculate duty_hw based on the percentage of period during
+ * which the led is ON.
+ */
+ duty_hw = MT6323_CAL_HW_DUTY(*delay_on, period);
+
+ /* hardware doesn't support zero duty cycle. */
+ if (!duty_hw)
+ return -EINVAL;
+
+ mutex_lock(&leds->lock);
+ /*
+ * Set max_brightness as the software blink behavior
+ * when no blink brightness.
+ */
+ if (!led->current_brightness) {
+ ret = mt6323_led_hw_on(cdev, cdev->max_brightness);
+ if (ret < 0)
+ goto out;
+ led->current_brightness = cdev->max_brightness;
+ }
+
+ ret = regmap_update_bits(regmap, MT6323_ISINK_CON0(led->id),
+ MT6323_ISINK_DIM_DUTY_MASK,
+ MT6323_ISINK_DIM_DUTY(duty_hw - 1));
+ if (ret < 0)
+ goto out;
+
+ ret = regmap_update_bits(regmap, MT6323_ISINK_CON1(led->id),
+ MT6323_ISINK_DIM_FSEL_MASK,
+ MT6323_ISINK_DIM_FSEL(period - 1));
+out:
+ mutex_unlock(&leds->lock);
+
+ return ret;
+}
+
+static int mt6323_led_set_brightness(struct led_classdev *cdev,
+ enum led_brightness brightness)
+{
+ struct mt6323_led *led = container_of(cdev, struct mt6323_led, cdev);
+ struct mt6323_leds *leds = led->parent;
+ int ret;
+
+ mutex_lock(&leds->lock);
+
+ if (!led->current_brightness && brightness) {
+ ret = mt6323_led_hw_on(cdev, brightness);
+ if (ret < 0)
+ goto out;
+ } else if (brightness) {
+ ret = mt6323_led_hw_brightness(cdev, brightness);
+ if (ret < 0)
+ goto out;
+ } else {
+ ret = mt6323_led_hw_off(cdev);
+ if (ret < 0)
+ goto out;
+ }
+
+ led->current_brightness = brightness;
+out:
+ mutex_unlock(&leds->lock);
+
+ return ret;
+}
+
+static int mt6323_led_set_dt_default(struct led_classdev *cdev,
+ struct device_node *np)
+{
+ struct mt6323_led *led = container_of(cdev, struct mt6323_led, cdev);
+ const char *state;
+ int ret = 0;
+
+ led->cdev.name = of_get_property(np, "label", NULL) ? : np->name;
+ led->cdev.default_trigger = of_get_property(np,
+ "linux,default-trigger",
+ NULL);
+
+ state = of_get_property(np, "default-state", NULL);
+ if (state) {
+ if (!strcmp(state, "keep")) {
+ ret = mt6323_get_led_hw_brightness(cdev);
+ if (ret < 0)
+ return ret;
+ led->current_brightness = ret;
+ ret = 0;
+ } else if (!strcmp(state, "on")) {
+ ret =
+ mt6323_led_set_brightness(cdev, cdev->max_brightness);
+ } else {
+ ret = mt6323_led_set_brightness(cdev, LED_OFF);
+ }
+ }
+
+ return ret;
+}
+
+static int mt6323_led_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct device_node *np = pdev->dev.of_node;
+ struct device_node *child;
+ struct mt6397_chip *hw = dev_get_drvdata(pdev->dev.parent);
+ struct mt6323_leds *leds;
+ struct mt6323_led *led;
+ int ret;
+ unsigned int status;
+ u32 reg;
+
+ leds = devm_kzalloc(dev, sizeof(*leds), GFP_KERNEL);
+ if (!leds)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, leds);
+ leds->dev = dev;
+
+ /*
+ * leds->hw points to the underlying bus for the register
+ * controlled.
+ */
+ leds->hw = hw;
+ mutex_init(&leds->lock);
+
+ status = MT6323_RG_DRV_32K_CK_PDN;
+ ret = regmap_update_bits(leds->hw->regmap, MT6323_TOP_CKPDN0,
+ MT6323_RG_DRV_32K_CK_PDN_MASK, ~status);
+ if (ret < 0) {
+ dev_err(leds->dev,
+ "Failed to update MT6323_TOP_CKPDN0 Register\n");
+ return ret;
+ }
+
+ for_each_available_child_of_node(np, child) {
+ ret = of_property_read_u32(child, "reg", ®);
+ if (ret) {
+ dev_err(dev, "Failed to read led 'reg' property\n");
+ goto put_child_node;
+ }
+
+ if (reg >= MT6323_MAX_LEDS || leds->led[reg]) {
+ dev_err(dev, "Invalid led reg %u\n", reg);
+ ret = -EINVAL;
+ goto put_child_node;
+ }
+
+ led = devm_kzalloc(dev, sizeof(*led), GFP_KERNEL);
+ if (!led) {
+ ret = -ENOMEM;
+ goto put_child_node;
+ }
+
+ leds->led[reg] = led;
+ leds->led[reg]->id = reg;
+ leds->led[reg]->cdev.max_brightness = MT6323_MAX_BRIGHTNESS;
+ leds->led[reg]->cdev.brightness_set_blocking =
+ mt6323_led_set_brightness;
+ leds->led[reg]->cdev.blink_set = mt6323_led_set_blink;
+ leds->led[reg]->cdev.brightness_get =
+ mt6323_get_led_hw_brightness;
+ leds->led[reg]->parent = leds;
+
+ ret = mt6323_led_set_dt_default(&leds->led[reg]->cdev, child);
+ if (ret < 0) {
+ dev_err(leds->dev,
+ "Failed to LED set default from devicetree\n");
+ goto put_child_node;
+ }
+
+ ret = devm_led_classdev_register(dev, &leds->led[reg]->cdev);
+ if (ret) {
+ dev_err(&pdev->dev, "Failed to register LED: %d\n",
+ ret);
+ goto put_child_node;
+ }
+ leds->led[reg]->cdev.dev->of_node = child;
+ }
+
+ return 0;
+
+put_child_node:
+ of_node_put(child);
+ return ret;
+}
+
+static int mt6323_led_remove(struct platform_device *pdev)
+{
+ struct mt6323_leds *leds = platform_get_drvdata(pdev);
+ int i;
+
+ /* Turn the LEDs off on driver removal. */
+ for (i = 0 ; leds->led[i] ; i++)
+ mt6323_led_hw_off(&leds->led[i]->cdev);
+
+ regmap_update_bits(leds->hw->regmap, MT6323_TOP_CKPDN0,
+ MT6323_RG_DRV_32K_CK_PDN_MASK,
+ MT6323_RG_DRV_32K_CK_PDN);
+
+ mutex_destroy(&leds->lock);
+
+ return 0;
+}
+
+static const struct of_device_id mt6323_led_dt_match[] = {
+ { .compatible = "mediatek,mt6323-led" },
+ {},
+};
+MODULE_DEVICE_TABLE(of, mt6323_led_dt_match);
+
+static struct platform_driver mt6323_led_driver = {
+ .probe = mt6323_led_probe,
+ .remove = mt6323_led_remove,
+ .driver = {
+ .name = "mt6323-led",
+ .of_match_table = mt6323_led_dt_match,
+ },
+};
+
+module_platform_driver(mt6323_led_driver);
+
+MODULE_DESCRIPTION("LED driver for Mediatek MT6323 PMIC");
+MODULE_AUTHOR("Sean Wang <sean.wang@mediatek.com>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/leds/leds-pca9532.c b/drivers/leds/leds-pca9532.c
index 06e6310..7fea18b 100644
--- a/drivers/leds/leds-pca9532.c
+++ b/drivers/leds/leds-pca9532.c
@@ -254,6 +254,21 @@
mutex_unlock(&data->update_lock);
}
+static enum pca9532_state pca9532_getled(struct pca9532_led *led)
+{
+ struct i2c_client *client = led->client;
+ struct pca9532_data *data = i2c_get_clientdata(client);
+ u8 maxleds = data->chip_info->num_leds;
+ char reg;
+ enum pca9532_state ret;
+
+ mutex_lock(&data->update_lock);
+ reg = i2c_smbus_read_byte_data(client, LED_REG(maxleds, led->id));
+ ret = reg >> LED_NUM(led->id)/2;
+ mutex_unlock(&data->update_lock);
+ return ret;
+}
+
#ifdef CONFIG_LEDS_PCA9532_GPIO
static int pca9532_gpio_request_pin(struct gpio_chip *gc, unsigned offset)
{
@@ -366,7 +381,10 @@
gpios++;
break;
case PCA9532_TYPE_LED:
- led->state = pled->state;
+ if (pled->state == PCA9532_KEEP)
+ led->state = pca9532_getled(led);
+ else
+ led->state = pled->state;
led->name = pled->name;
led->ldev.name = led->name;
led->ldev.default_trigger = pled->default_trigger;
@@ -456,6 +474,7 @@
const struct of_device_id *match;
int devid, maxleds;
int i = 0;
+ const char *state;
match = of_match_device(of_pca9532_leds_match, dev);
if (!match)
@@ -475,6 +494,12 @@
of_property_read_u32(child, "type", &pdata->leds[i].type);
of_property_read_string(child, "linux,default-trigger",
&pdata->leds[i].default_trigger);
+ if (!of_property_read_string(child, "default-state", &state)) {
+ if (!strcmp(state, "on"))
+ pdata->leds[i].state = PCA9532_ON;
+ else if (!strcmp(state, "keep"))
+ pdata->leds[i].state = PCA9532_KEEP;
+ }
if (++i >= maxleds) {
of_node_put(child);
break;
diff --git a/drivers/leds/trigger/ledtrig-cpu.c b/drivers/leds/trigger/ledtrig-cpu.c
index a418964..66a6260 100644
--- a/drivers/leds/trigger/ledtrig-cpu.c
+++ b/drivers/leds/trigger/ledtrig-cpu.c
@@ -31,12 +31,16 @@
#define MAX_NAME_LEN 8
struct led_trigger_cpu {
+ bool is_active;
char name[MAX_NAME_LEN];
struct led_trigger *_trig;
};
static DEFINE_PER_CPU(struct led_trigger_cpu, cpu_trig);
+static struct led_trigger *trig_cpu_all;
+static atomic_t num_active_cpus = ATOMIC_INIT(0);
+
/**
* ledtrig_cpu - emit a CPU event as a trigger
* @evt: CPU event to be emitted
@@ -47,26 +51,46 @@
void ledtrig_cpu(enum cpu_led_event ledevt)
{
struct led_trigger_cpu *trig = this_cpu_ptr(&cpu_trig);
+ bool is_active = trig->is_active;
/* Locate the correct CPU LED */
switch (ledevt) {
case CPU_LED_IDLE_END:
case CPU_LED_START:
/* Will turn the LED on, max brightness */
- led_trigger_event(trig->_trig, LED_FULL);
+ is_active = true;
break;
case CPU_LED_IDLE_START:
case CPU_LED_STOP:
case CPU_LED_HALTED:
/* Will turn the LED off */
- led_trigger_event(trig->_trig, LED_OFF);
+ is_active = false;
break;
default:
/* Will leave the LED as it is */
break;
}
+
+ if (is_active != trig->is_active) {
+ unsigned int active_cpus;
+ unsigned int total_cpus;
+
+ /* Update trigger state */
+ trig->is_active = is_active;
+ atomic_add(is_active ? 1 : -1, &num_active_cpus);
+ active_cpus = atomic_read(&num_active_cpus);
+ total_cpus = num_present_cpus();
+
+ led_trigger_event(trig->_trig,
+ is_active ? LED_FULL : LED_OFF);
+
+
+ led_trigger_event(trig_cpu_all,
+ DIV_ROUND_UP(LED_FULL * active_cpus, total_cpus));
+
+ }
}
EXPORT_SYMBOL(ledtrig_cpu);
@@ -113,6 +137,11 @@
BUILD_BUG_ON(CONFIG_NR_CPUS > 9999);
/*
+ * Registering a trigger for all CPUs.
+ */
+ led_trigger_register_simple("cpu", &trig_cpu_all);
+
+ /*
* Registering CPU led trigger for each CPU core here
* ignores CPU hotplug, but after this CPU hotplug works
* fine with this trigger.
diff --git a/drivers/lguest/x86/core.c b/drivers/lguest/x86/core.c
index d71f632..b4f79b9 100644
--- a/drivers/lguest/x86/core.c
+++ b/drivers/lguest/x86/core.c
@@ -504,7 +504,7 @@
* byte, not the size, hence the "-1").
*/
state->host_gdt_desc.size = GDT_SIZE-1;
- state->host_gdt_desc.address = (long)get_cpu_gdt_table(i);
+ state->host_gdt_desc.address = (long)get_cpu_gdt_rw(i);
/*
* All CPUs on the Host use the same Interrupt Descriptor
@@ -554,8 +554,8 @@
* The Host needs to be able to use the LGUEST segments on this
* CPU, too, so put them in the Host GDT.
*/
- get_cpu_gdt_table(i)[GDT_ENTRY_LGUEST_CS] = FULL_EXEC_SEGMENT;
- get_cpu_gdt_table(i)[GDT_ENTRY_LGUEST_DS] = FULL_SEGMENT;
+ get_cpu_gdt_rw(i)[GDT_ENTRY_LGUEST_CS] = FULL_EXEC_SEGMENT;
+ get_cpu_gdt_rw(i)[GDT_ENTRY_LGUEST_DS] = FULL_SEGMENT;
}
/*
diff --git a/drivers/lightnvm/Kconfig b/drivers/lightnvm/Kconfig
index 0527141..ead61a9 100644
--- a/drivers/lightnvm/Kconfig
+++ b/drivers/lightnvm/Kconfig
@@ -33,4 +33,13 @@
host. The target is implemented using a linear mapping table and
cost-based garbage collection. It is optimized for 4K IO sizes.
+config NVM_PBLK
+ tristate "Physical Block Device Open-Channel SSD target"
+ ---help---
+ Allows an open-channel SSD to be exposed as a block device to the
+ host. The target assumes the device exposes raw flash and must be
+ explicitly managed by the host.
+
+ Please note the disk format is considered EXPERIMENTAL for now.
+
endif # NVM
diff --git a/drivers/lightnvm/Makefile b/drivers/lightnvm/Makefile
index b2a39e2..82d1a11 100644
--- a/drivers/lightnvm/Makefile
+++ b/drivers/lightnvm/Makefile
@@ -4,3 +4,8 @@
obj-$(CONFIG_NVM) := core.o
obj-$(CONFIG_NVM_RRPC) += rrpc.o
+obj-$(CONFIG_NVM_PBLK) += pblk.o
+pblk-y := pblk-init.o pblk-core.o pblk-rb.o \
+ pblk-write.o pblk-cache.o pblk-read.o \
+ pblk-gc.o pblk-recovery.o pblk-map.o \
+ pblk-rl.o pblk-sysfs.o
diff --git a/drivers/lightnvm/core.c b/drivers/lightnvm/core.c
index 5262ba6..54a06c3 100644
--- a/drivers/lightnvm/core.c
+++ b/drivers/lightnvm/core.c
@@ -89,7 +89,7 @@
WARN_ON(!test_and_clear_bit(i, dev->lun_map));
}
-static void nvm_remove_tgt_dev(struct nvm_tgt_dev *tgt_dev)
+static void nvm_remove_tgt_dev(struct nvm_tgt_dev *tgt_dev, int clear)
{
struct nvm_dev *dev = tgt_dev->parent;
struct nvm_dev_map *dev_map = tgt_dev->map;
@@ -100,11 +100,14 @@
int *lun_offs = ch_map->lun_offs;
int ch = i + ch_map->ch_off;
- for (j = 0; j < ch_map->nr_luns; j++) {
- int lun = j + lun_offs[j];
- int lunid = (ch * dev->geo.luns_per_chnl) + lun;
+ if (clear) {
+ for (j = 0; j < ch_map->nr_luns; j++) {
+ int lun = j + lun_offs[j];
+ int lunid = (ch * dev->geo.luns_per_chnl) + lun;
- WARN_ON(!test_and_clear_bit(lunid, dev->lun_map));
+ WARN_ON(!test_and_clear_bit(lunid,
+ dev->lun_map));
+ }
}
kfree(ch_map->lun_offs);
@@ -232,6 +235,7 @@
struct nvm_target *t;
struct nvm_tgt_dev *tgt_dev;
void *targetdata;
+ int ret;
tt = nvm_find_target_type(create->tgttype, 1);
if (!tt) {
@@ -252,34 +256,43 @@
return -ENOMEM;
t = kmalloc(sizeof(struct nvm_target), GFP_KERNEL);
- if (!t)
+ if (!t) {
+ ret = -ENOMEM;
goto err_reserve;
+ }
tgt_dev = nvm_create_tgt_dev(dev, s->lun_begin, s->lun_end);
if (!tgt_dev) {
pr_err("nvm: could not create target device\n");
+ ret = -ENOMEM;
goto err_t;
}
- tqueue = blk_alloc_queue_node(GFP_KERNEL, dev->q->node);
- if (!tqueue)
+ tdisk = alloc_disk(0);
+ if (!tdisk) {
+ ret = -ENOMEM;
goto err_dev;
+ }
+
+ tqueue = blk_alloc_queue_node(GFP_KERNEL, dev->q->node);
+ if (!tqueue) {
+ ret = -ENOMEM;
+ goto err_disk;
+ }
blk_queue_make_request(tqueue, tt->make_rq);
- tdisk = alloc_disk(0);
- if (!tdisk)
- goto err_queue;
-
- sprintf(tdisk->disk_name, "%s", create->tgtname);
+ strlcpy(tdisk->disk_name, create->tgtname, sizeof(tdisk->disk_name));
tdisk->flags = GENHD_FL_EXT_DEVT;
tdisk->major = 0;
tdisk->first_minor = 0;
tdisk->fops = &nvm_fops;
tdisk->queue = tqueue;
- targetdata = tt->init(tgt_dev, tdisk);
- if (IS_ERR(targetdata))
+ targetdata = tt->init(tgt_dev, tdisk, create->flags);
+ if (IS_ERR(targetdata)) {
+ ret = PTR_ERR(targetdata);
goto err_init;
+ }
tdisk->private_data = targetdata;
tqueue->queuedata = targetdata;
@@ -289,8 +302,10 @@
set_capacity(tdisk, tt->capacity(targetdata));
add_disk(tdisk);
- if (tt->sysfs_init && tt->sysfs_init(tdisk))
+ if (tt->sysfs_init && tt->sysfs_init(tdisk)) {
+ ret = -ENOMEM;
goto err_sysfs;
+ }
t->type = tt;
t->disk = tdisk;
@@ -305,16 +320,17 @@
if (tt->exit)
tt->exit(targetdata);
err_init:
- put_disk(tdisk);
-err_queue:
blk_cleanup_queue(tqueue);
+ tdisk->queue = NULL;
+err_disk:
+ put_disk(tdisk);
err_dev:
- nvm_remove_tgt_dev(tgt_dev);
+ nvm_remove_tgt_dev(tgt_dev, 0);
err_t:
kfree(t);
err_reserve:
nvm_release_luns_err(dev, s->lun_begin, s->lun_end);
- return -ENOMEM;
+ return ret;
}
static void __nvm_remove_target(struct nvm_target *t)
@@ -332,7 +348,7 @@
if (tt->exit)
tt->exit(tdisk->private_data);
- nvm_remove_tgt_dev(t->dev);
+ nvm_remove_tgt_dev(t->dev, 1);
put_disk(tdisk);
list_del(&t->list);
@@ -411,6 +427,18 @@
return -ENOMEM;
}
+static void nvm_unregister_map(struct nvm_dev *dev)
+{
+ struct nvm_dev_map *rmap = dev->rmap;
+ int i;
+
+ for (i = 0; i < dev->geo.nr_chnls; i++)
+ kfree(rmap->chnls[i].lun_offs);
+
+ kfree(rmap->chnls);
+ kfree(rmap);
+}
+
static void nvm_map_to_dev(struct nvm_tgt_dev *tgt_dev, struct ppa_addr *p)
{
struct nvm_dev_map *dev_map = tgt_dev->map;
@@ -486,7 +514,6 @@
int *lun_roffs;
struct ppa_addr gaddr;
u64 pba = le64_to_cpu(entries[i]);
- int off;
u64 diff;
if (!pba)
@@ -496,8 +523,6 @@
ch_rmap = &dev_rmap->chnls[gaddr.g.ch];
lun_roffs = ch_rmap->lun_offs;
- off = gaddr.g.ch * geo->luns_per_chnl + gaddr.g.lun;
-
diff = ((ch_rmap->ch_off * geo->luns_per_chnl) +
(lun_roffs[gaddr.g.lun])) * geo->sec_per_lun;
@@ -590,11 +615,11 @@
memset(&rqd, 0, sizeof(struct nvm_rq));
- nvm_set_rqd_ppalist(dev, &rqd, ppas, nr_ppas, 1);
+ nvm_set_rqd_ppalist(tgt_dev, &rqd, ppas, nr_ppas, 1);
nvm_rq_tgt_to_dev(tgt_dev, &rqd);
ret = dev->ops->set_bb_tbl(dev, &rqd.ppa_addr, rqd.nr_ppas, type);
- nvm_free_rqd_ppalist(dev, &rqd);
+ nvm_free_rqd_ppalist(tgt_dev, &rqd);
if (ret) {
pr_err("nvm: failed bb mark\n");
return -EINVAL;
@@ -626,34 +651,45 @@
}
EXPORT_SYMBOL(nvm_submit_io);
-int nvm_erase_blk(struct nvm_tgt_dev *tgt_dev, struct ppa_addr *ppas, int flags)
+static void nvm_end_io_sync(struct nvm_rq *rqd)
{
- struct nvm_dev *dev = tgt_dev->parent;
+ struct completion *waiting = rqd->private;
+
+ complete(waiting);
+}
+
+int nvm_erase_sync(struct nvm_tgt_dev *tgt_dev, struct ppa_addr *ppas,
+ int nr_ppas)
+{
+ struct nvm_geo *geo = &tgt_dev->geo;
struct nvm_rq rqd;
int ret;
-
- if (!dev->ops->erase_block)
- return 0;
-
- nvm_map_to_dev(tgt_dev, ppas);
+ DECLARE_COMPLETION_ONSTACK(wait);
memset(&rqd, 0, sizeof(struct nvm_rq));
- ret = nvm_set_rqd_ppalist(dev, &rqd, ppas, 1, 1);
+ rqd.opcode = NVM_OP_ERASE;
+ rqd.end_io = nvm_end_io_sync;
+ rqd.private = &wait;
+ rqd.flags = geo->plane_mode >> 1;
+
+ ret = nvm_set_rqd_ppalist(tgt_dev, &rqd, ppas, nr_ppas, 1);
if (ret)
return ret;
- nvm_rq_tgt_to_dev(tgt_dev, &rqd);
+ ret = nvm_submit_io(tgt_dev, &rqd);
+ if (ret) {
+ pr_err("rrpr: erase I/O submission failed: %d\n", ret);
+ goto free_ppa_list;
+ }
+ wait_for_completion_io(&wait);
- rqd.flags = flags;
-
- ret = dev->ops->erase_block(dev, &rqd);
-
- nvm_free_rqd_ppalist(dev, &rqd);
+free_ppa_list:
+ nvm_free_rqd_ppalist(tgt_dev, &rqd);
return ret;
}
-EXPORT_SYMBOL(nvm_erase_blk);
+EXPORT_SYMBOL(nvm_erase_sync);
int nvm_get_l2p_tbl(struct nvm_tgt_dev *tgt_dev, u64 slba, u32 nlb,
nvm_l2p_update_fn *update_l2p, void *priv)
@@ -732,10 +768,11 @@
}
EXPORT_SYMBOL(nvm_put_area);
-int nvm_set_rqd_ppalist(struct nvm_dev *dev, struct nvm_rq *rqd,
+int nvm_set_rqd_ppalist(struct nvm_tgt_dev *tgt_dev, struct nvm_rq *rqd,
const struct ppa_addr *ppas, int nr_ppas, int vblk)
{
- struct nvm_geo *geo = &dev->geo;
+ struct nvm_dev *dev = tgt_dev->parent;
+ struct nvm_geo *geo = &tgt_dev->geo;
int i, plane_cnt, pl_idx;
struct ppa_addr ppa;
@@ -773,12 +810,12 @@
}
EXPORT_SYMBOL(nvm_set_rqd_ppalist);
-void nvm_free_rqd_ppalist(struct nvm_dev *dev, struct nvm_rq *rqd)
+void nvm_free_rqd_ppalist(struct nvm_tgt_dev *tgt_dev, struct nvm_rq *rqd)
{
if (!rqd->ppa_list)
return;
- nvm_dev_dma_free(dev, rqd->ppa_list, rqd->dma_ppa_list);
+ nvm_dev_dma_free(tgt_dev->parent, rqd->ppa_list, rqd->dma_ppa_list);
}
EXPORT_SYMBOL(nvm_free_rqd_ppalist);
@@ -972,7 +1009,7 @@
return ret;
}
-void nvm_free(struct nvm_dev *dev)
+static void nvm_free(struct nvm_dev *dev)
{
if (!dev)
return;
@@ -980,7 +1017,7 @@
if (dev->dma_pool)
dev->ops->destroy_dma_pool(dev->dma_pool);
- kfree(dev->rmap);
+ nvm_unregister_map(dev);
kfree(dev->lptbl);
kfree(dev->lun_map);
kfree(dev);
@@ -1174,13 +1211,13 @@
list_for_each_entry(dev, &nvm_devices, devices) {
struct nvm_ioctl_device_info *info = &devices->info[i];
- sprintf(info->devname, "%s", dev->name);
+ strlcpy(info->devname, dev->name, sizeof(info->devname));
/* kept for compatibility */
info->bmversion[0] = 1;
info->bmversion[1] = 0;
info->bmversion[2] = 0;
- sprintf(info->bmname, "%s", "gennvm");
+ strlcpy(info->bmname, "gennvm", sizeof(info->bmname));
i++;
if (i > 31) {
@@ -1217,8 +1254,16 @@
create.tgtname[DISK_NAME_LEN - 1] = '\0';
if (create.flags != 0) {
- pr_err("nvm: no flags supported\n");
- return -EINVAL;
+ __u32 flags = create.flags;
+
+ /* Check for valid flags */
+ if (flags & NVM_TARGET_FACTORY)
+ flags &= ~NVM_TARGET_FACTORY;
+
+ if (flags) {
+ pr_err("nvm: flag not supported\n");
+ return -EINVAL;
+ }
}
return __nvm_configure_create(&create);
diff --git a/drivers/lightnvm/pblk-cache.c b/drivers/lightnvm/pblk-cache.c
new file mode 100644
index 0000000..59bcea8
--- /dev/null
+++ b/drivers/lightnvm/pblk-cache.c
@@ -0,0 +1,114 @@
+/*
+ * Copyright (C) 2016 CNEX Labs
+ * Initial release: Javier Gonzalez <javier@cnexlabs.com>
+ * Matias Bjorling <matias@cnexlabs.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * pblk-cache.c - pblk's write cache
+ */
+
+#include "pblk.h"
+
+int pblk_write_to_cache(struct pblk *pblk, struct bio *bio, unsigned long flags)
+{
+ struct pblk_w_ctx w_ctx;
+ sector_t lba = pblk_get_lba(bio);
+ unsigned int bpos, pos;
+ int nr_entries = pblk_get_secs(bio);
+ int i, ret;
+
+ /* Update the write buffer head (mem) with the entries that we can
+ * write. The write in itself cannot fail, so there is no need to
+ * rollback from here on.
+ */
+retry:
+ ret = pblk_rb_may_write_user(&pblk->rwb, bio, nr_entries, &bpos);
+ if (ret == NVM_IO_REQUEUE) {
+ io_schedule();
+ goto retry;
+ }
+
+ if (unlikely(!bio_has_data(bio)))
+ goto out;
+
+ w_ctx.flags = flags;
+ pblk_ppa_set_empty(&w_ctx.ppa);
+
+ for (i = 0; i < nr_entries; i++) {
+ void *data = bio_data(bio);
+
+ w_ctx.lba = lba + i;
+
+ pos = pblk_rb_wrap_pos(&pblk->rwb, bpos + i);
+ pblk_rb_write_entry_user(&pblk->rwb, data, w_ctx, pos);
+
+ bio_advance(bio, PBLK_EXPOSED_PAGE_SIZE);
+ }
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_add(nr_entries, &pblk->inflight_writes);
+ atomic_long_add(nr_entries, &pblk->req_writes);
+#endif
+
+out:
+ pblk_write_should_kick(pblk);
+ return ret;
+}
+
+/*
+ * On GC the incoming lbas are not necessarily sequential. Also, some of the
+ * lbas might not be valid entries, which are marked as empty by the GC thread
+ */
+int pblk_write_gc_to_cache(struct pblk *pblk, void *data, u64 *lba_list,
+ unsigned int nr_entries, unsigned int nr_rec_entries,
+ struct pblk_line *gc_line, unsigned long flags)
+{
+ struct pblk_w_ctx w_ctx;
+ unsigned int bpos, pos;
+ int i, valid_entries;
+
+ /* Update the write buffer head (mem) with the entries that we can
+ * write. The write in itself cannot fail, so there is no need to
+ * rollback from here on.
+ */
+retry:
+ if (!pblk_rb_may_write_gc(&pblk->rwb, nr_rec_entries, &bpos)) {
+ io_schedule();
+ goto retry;
+ }
+
+ w_ctx.flags = flags;
+ pblk_ppa_set_empty(&w_ctx.ppa);
+
+ for (i = 0, valid_entries = 0; i < nr_entries; i++) {
+ if (lba_list[i] == ADDR_EMPTY)
+ continue;
+
+ w_ctx.lba = lba_list[i];
+
+ pos = pblk_rb_wrap_pos(&pblk->rwb, bpos + valid_entries);
+ pblk_rb_write_entry_gc(&pblk->rwb, data, w_ctx, gc_line, pos);
+
+ data += PBLK_EXPOSED_PAGE_SIZE;
+ valid_entries++;
+ }
+
+ WARN_ONCE(nr_rec_entries != valid_entries,
+ "pblk: inconsistent GC write\n");
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_add(valid_entries, &pblk->inflight_writes);
+ atomic_long_add(valid_entries, &pblk->recov_gc_writes);
+#endif
+
+ pblk_write_should_kick(pblk);
+ return NVM_IO_OK;
+}
diff --git a/drivers/lightnvm/pblk-core.c b/drivers/lightnvm/pblk-core.c
new file mode 100644
index 0000000..5e44768
--- /dev/null
+++ b/drivers/lightnvm/pblk-core.c
@@ -0,0 +1,1667 @@
+/*
+ * Copyright (C) 2016 CNEX Labs
+ * Initial release: Javier Gonzalez <javier@cnexlabs.com>
+ * Matias Bjorling <matias@cnexlabs.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * pblk-core.c - pblk's core functionality
+ *
+ */
+
+#include "pblk.h"
+#include <linux/time.h>
+
+static void pblk_mark_bb(struct pblk *pblk, struct pblk_line *line,
+ struct ppa_addr *ppa)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ int pos = pblk_dev_ppa_to_pos(geo, *ppa);
+
+ pr_debug("pblk: erase failed: line:%d, pos:%d\n", line->id, pos);
+ atomic_long_inc(&pblk->erase_failed);
+
+ atomic_dec(&line->blk_in_line);
+ if (test_and_set_bit(pos, line->blk_bitmap))
+ pr_err("pblk: attempted to erase bb: line:%d, pos:%d\n",
+ line->id, pos);
+
+ pblk_line_run_ws(pblk, NULL, ppa, pblk_line_mark_bb);
+}
+
+static void __pblk_end_io_erase(struct pblk *pblk, struct nvm_rq *rqd)
+{
+ struct pblk_line *line;
+
+ line = &pblk->lines[pblk_dev_ppa_to_line(rqd->ppa_addr)];
+ atomic_dec(&line->left_seblks);
+
+ if (rqd->error) {
+ struct ppa_addr *ppa;
+
+ ppa = kmalloc(sizeof(struct ppa_addr), GFP_ATOMIC);
+ if (!ppa)
+ return;
+
+ *ppa = rqd->ppa_addr;
+ pblk_mark_bb(pblk, line, ppa);
+ }
+}
+
+/* Erase completion assumes that only one block is erased at the time */
+static void pblk_end_io_erase(struct nvm_rq *rqd)
+{
+ struct pblk *pblk = rqd->private;
+
+ up(&pblk->erase_sem);
+ __pblk_end_io_erase(pblk, rqd);
+ mempool_free(rqd, pblk->r_rq_pool);
+}
+
+static void __pblk_map_invalidate(struct pblk *pblk, struct pblk_line *line,
+ u64 paddr)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct list_head *move_list = NULL;
+
+ /* Lines being reclaimed (GC'ed) cannot be invalidated. Before the L2P
+ * table is modified with reclaimed sectors, a check is done to endure
+ * that newer updates are not overwritten.
+ */
+ spin_lock(&line->lock);
+ if (line->state == PBLK_LINESTATE_GC ||
+ line->state == PBLK_LINESTATE_FREE) {
+ spin_unlock(&line->lock);
+ return;
+ }
+
+ if (test_and_set_bit(paddr, line->invalid_bitmap)) {
+ WARN_ONCE(1, "pblk: double invalidate\n");
+ spin_unlock(&line->lock);
+ return;
+ }
+ line->vsc--;
+
+ if (line->state == PBLK_LINESTATE_CLOSED)
+ move_list = pblk_line_gc_list(pblk, line);
+ spin_unlock(&line->lock);
+
+ if (move_list) {
+ spin_lock(&l_mg->gc_lock);
+ spin_lock(&line->lock);
+ /* Prevent moving a line that has just been chosen for GC */
+ if (line->state == PBLK_LINESTATE_GC ||
+ line->state == PBLK_LINESTATE_FREE) {
+ spin_unlock(&line->lock);
+ spin_unlock(&l_mg->gc_lock);
+ return;
+ }
+ spin_unlock(&line->lock);
+
+ list_move_tail(&line->list, move_list);
+ spin_unlock(&l_mg->gc_lock);
+ }
+}
+
+void pblk_map_invalidate(struct pblk *pblk, struct ppa_addr ppa)
+{
+ struct pblk_line *line;
+ u64 paddr;
+ int line_id;
+
+#ifdef CONFIG_NVM_DEBUG
+ /* Callers must ensure that the ppa points to a device address */
+ BUG_ON(pblk_addr_in_cache(ppa));
+ BUG_ON(pblk_ppa_empty(ppa));
+#endif
+
+ line_id = pblk_tgt_ppa_to_line(ppa);
+ line = &pblk->lines[line_id];
+ paddr = pblk_dev_ppa_to_line_addr(pblk, ppa);
+
+ __pblk_map_invalidate(pblk, line, paddr);
+}
+
+void pblk_map_pad_invalidate(struct pblk *pblk, struct pblk_line *line,
+ u64 paddr)
+{
+ __pblk_map_invalidate(pblk, line, paddr);
+
+ pblk_rb_sync_init(&pblk->rwb, NULL);
+ line->left_ssecs--;
+ if (!line->left_ssecs)
+ pblk_line_run_ws(pblk, line, NULL, pblk_line_close_ws);
+ pblk_rb_sync_end(&pblk->rwb, NULL);
+}
+
+static void pblk_invalidate_range(struct pblk *pblk, sector_t slba,
+ unsigned int nr_secs)
+{
+ sector_t lba;
+
+ spin_lock(&pblk->trans_lock);
+ for (lba = slba; lba < slba + nr_secs; lba++) {
+ struct ppa_addr ppa;
+
+ ppa = pblk_trans_map_get(pblk, lba);
+
+ if (!pblk_addr_in_cache(ppa) && !pblk_ppa_empty(ppa))
+ pblk_map_invalidate(pblk, ppa);
+
+ pblk_ppa_set_empty(&ppa);
+ pblk_trans_map_set(pblk, lba, ppa);
+ }
+ spin_unlock(&pblk->trans_lock);
+}
+
+struct nvm_rq *pblk_alloc_rqd(struct pblk *pblk, int rw)
+{
+ mempool_t *pool;
+ struct nvm_rq *rqd;
+ int rq_size;
+
+ if (rw == WRITE) {
+ pool = pblk->w_rq_pool;
+ rq_size = pblk_w_rq_size;
+ } else {
+ pool = pblk->r_rq_pool;
+ rq_size = pblk_r_rq_size;
+ }
+
+ rqd = mempool_alloc(pool, GFP_KERNEL);
+ memset(rqd, 0, rq_size);
+
+ return rqd;
+}
+
+void pblk_free_rqd(struct pblk *pblk, struct nvm_rq *rqd, int rw)
+{
+ mempool_t *pool;
+
+ if (rw == WRITE)
+ pool = pblk->w_rq_pool;
+ else
+ pool = pblk->r_rq_pool;
+
+ mempool_free(rqd, pool);
+}
+
+void pblk_bio_free_pages(struct pblk *pblk, struct bio *bio, int off,
+ int nr_pages)
+{
+ struct bio_vec bv;
+ int i;
+
+ WARN_ON(off + nr_pages != bio->bi_vcnt);
+
+ bio_advance(bio, off * PBLK_EXPOSED_PAGE_SIZE);
+ for (i = off; i < nr_pages + off; i++) {
+ bv = bio->bi_io_vec[i];
+ mempool_free(bv.bv_page, pblk->page_pool);
+ }
+}
+
+int pblk_bio_add_pages(struct pblk *pblk, struct bio *bio, gfp_t flags,
+ int nr_pages)
+{
+ struct request_queue *q = pblk->dev->q;
+ struct page *page;
+ int i, ret;
+
+ for (i = 0; i < nr_pages; i++) {
+ page = mempool_alloc(pblk->page_pool, flags);
+ if (!page)
+ goto err;
+
+ ret = bio_add_pc_page(q, bio, page, PBLK_EXPOSED_PAGE_SIZE, 0);
+ if (ret != PBLK_EXPOSED_PAGE_SIZE) {
+ pr_err("pblk: could not add page to bio\n");
+ mempool_free(page, pblk->page_pool);
+ goto err;
+ }
+ }
+
+ return 0;
+err:
+ pblk_bio_free_pages(pblk, bio, 0, i - 1);
+ return -1;
+}
+
+static void pblk_write_kick(struct pblk *pblk)
+{
+ wake_up_process(pblk->writer_ts);
+ mod_timer(&pblk->wtimer, jiffies + msecs_to_jiffies(1000));
+}
+
+void pblk_write_timer_fn(unsigned long data)
+{
+ struct pblk *pblk = (struct pblk *)data;
+
+ /* kick the write thread every tick to flush outstanding data */
+ pblk_write_kick(pblk);
+}
+
+void pblk_write_should_kick(struct pblk *pblk)
+{
+ unsigned int secs_avail = pblk_rb_read_count(&pblk->rwb);
+
+ if (secs_avail >= pblk->min_write_pgs)
+ pblk_write_kick(pblk);
+}
+
+void pblk_end_bio_sync(struct bio *bio)
+{
+ struct completion *waiting = bio->bi_private;
+
+ complete(waiting);
+}
+
+void pblk_end_io_sync(struct nvm_rq *rqd)
+{
+ struct completion *waiting = rqd->private;
+
+ complete(waiting);
+}
+
+void pblk_flush_writer(struct pblk *pblk)
+{
+ struct bio *bio;
+ int ret;
+ DECLARE_COMPLETION_ONSTACK(wait);
+
+ bio = bio_alloc(GFP_KERNEL, 1);
+ if (!bio)
+ return;
+
+ bio->bi_iter.bi_sector = 0; /* internal bio */
+ bio_set_op_attrs(bio, REQ_OP_WRITE, REQ_OP_FLUSH);
+ bio->bi_private = &wait;
+ bio->bi_end_io = pblk_end_bio_sync;
+
+ ret = pblk_write_to_cache(pblk, bio, 0);
+ if (ret == NVM_IO_OK) {
+ if (!wait_for_completion_io_timeout(&wait,
+ msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
+ pr_err("pblk: flush cache timed out\n");
+ }
+ } else if (ret != NVM_IO_DONE) {
+ pr_err("pblk: tear down bio failed\n");
+ }
+
+ if (bio->bi_error)
+ pr_err("pblk: flush sync write failed (%u)\n", bio->bi_error);
+
+ bio_put(bio);
+}
+
+struct list_head *pblk_line_gc_list(struct pblk *pblk, struct pblk_line *line)
+{
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct list_head *move_list = NULL;
+
+ if (!line->vsc) {
+ if (line->gc_group != PBLK_LINEGC_FULL) {
+ line->gc_group = PBLK_LINEGC_FULL;
+ move_list = &l_mg->gc_full_list;
+ }
+ } else if (line->vsc < lm->mid_thrs) {
+ if (line->gc_group != PBLK_LINEGC_HIGH) {
+ line->gc_group = PBLK_LINEGC_HIGH;
+ move_list = &l_mg->gc_high_list;
+ }
+ } else if (line->vsc < lm->high_thrs) {
+ if (line->gc_group != PBLK_LINEGC_MID) {
+ line->gc_group = PBLK_LINEGC_MID;
+ move_list = &l_mg->gc_mid_list;
+ }
+ } else if (line->vsc < line->sec_in_line) {
+ if (line->gc_group != PBLK_LINEGC_LOW) {
+ line->gc_group = PBLK_LINEGC_LOW;
+ move_list = &l_mg->gc_low_list;
+ }
+ } else if (line->vsc == line->sec_in_line) {
+ if (line->gc_group != PBLK_LINEGC_EMPTY) {
+ line->gc_group = PBLK_LINEGC_EMPTY;
+ move_list = &l_mg->gc_empty_list;
+ }
+ } else {
+ line->state = PBLK_LINESTATE_CORRUPT;
+ line->gc_group = PBLK_LINEGC_NONE;
+ move_list = &l_mg->corrupt_list;
+ pr_err("pblk: corrupted vsc for line %d, vsc:%d (%d/%d/%d)\n",
+ line->id, line->vsc,
+ line->sec_in_line,
+ lm->high_thrs, lm->mid_thrs);
+ }
+
+ return move_list;
+}
+
+void pblk_discard(struct pblk *pblk, struct bio *bio)
+{
+ sector_t slba = pblk_get_lba(bio);
+ sector_t nr_secs = pblk_get_secs(bio);
+
+ pblk_invalidate_range(pblk, slba, nr_secs);
+}
+
+struct ppa_addr pblk_get_lba_map(struct pblk *pblk, sector_t lba)
+{
+ struct ppa_addr ppa;
+
+ spin_lock(&pblk->trans_lock);
+ ppa = pblk_trans_map_get(pblk, lba);
+ spin_unlock(&pblk->trans_lock);
+
+ return ppa;
+}
+
+void pblk_log_write_err(struct pblk *pblk, struct nvm_rq *rqd)
+{
+ atomic_long_inc(&pblk->write_failed);
+#ifdef CONFIG_NVM_DEBUG
+ pblk_print_failed_rqd(pblk, rqd, rqd->error);
+#endif
+}
+
+void pblk_log_read_err(struct pblk *pblk, struct nvm_rq *rqd)
+{
+ /* Empty page read is not necessarily an error (e.g., L2P recovery) */
+ if (rqd->error == NVM_RSP_ERR_EMPTYPAGE) {
+ atomic_long_inc(&pblk->read_empty);
+ return;
+ }
+
+ switch (rqd->error) {
+ case NVM_RSP_WARN_HIGHECC:
+ atomic_long_inc(&pblk->read_high_ecc);
+ break;
+ case NVM_RSP_ERR_FAILECC:
+ case NVM_RSP_ERR_FAILCRC:
+ atomic_long_inc(&pblk->read_failed);
+ break;
+ default:
+ pr_err("pblk: unknown read error:%d\n", rqd->error);
+ }
+#ifdef CONFIG_NVM_DEBUG
+ pblk_print_failed_rqd(pblk, rqd, rqd->error);
+#endif
+}
+
+int pblk_submit_io(struct pblk *pblk, struct nvm_rq *rqd)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+
+#ifdef CONFIG_NVM_DEBUG
+ struct ppa_addr *ppa_list;
+
+ ppa_list = (rqd->nr_ppas > 1) ? rqd->ppa_list : &rqd->ppa_addr;
+ if (pblk_boundary_ppa_checks(dev, ppa_list, rqd->nr_ppas)) {
+ WARN_ON(1);
+ return -EINVAL;
+ }
+
+ if (rqd->opcode == NVM_OP_PWRITE) {
+ struct pblk_line *line;
+ struct ppa_addr ppa;
+ int i;
+
+ for (i = 0; i < rqd->nr_ppas; i++) {
+ ppa = ppa_list[i];
+ line = &pblk->lines[pblk_dev_ppa_to_line(ppa)];
+
+ spin_lock(&line->lock);
+ if (line->state != PBLK_LINESTATE_OPEN) {
+ pr_err("pblk: bad ppa: line:%d,state:%d\n",
+ line->id, line->state);
+ WARN_ON(1);
+ spin_unlock(&line->lock);
+ return -EINVAL;
+ }
+ spin_unlock(&line->lock);
+ }
+ }
+#endif
+ return nvm_submit_io(dev, rqd);
+}
+
+struct bio *pblk_bio_map_addr(struct pblk *pblk, void *data,
+ unsigned int nr_secs, unsigned int len,
+ gfp_t gfp_mask)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ void *kaddr = data;
+ struct page *page;
+ struct bio *bio;
+ int i, ret;
+
+ if (l_mg->emeta_alloc_type == PBLK_KMALLOC_META)
+ return bio_map_kern(dev->q, kaddr, len, gfp_mask);
+
+ bio = bio_kmalloc(gfp_mask, nr_secs);
+ if (!bio)
+ return ERR_PTR(-ENOMEM);
+
+ for (i = 0; i < nr_secs; i++) {
+ page = vmalloc_to_page(kaddr);
+ if (!page) {
+ pr_err("pblk: could not map vmalloc bio\n");
+ bio_put(bio);
+ bio = ERR_PTR(-ENOMEM);
+ goto out;
+ }
+
+ ret = bio_add_pc_page(dev->q, bio, page, PAGE_SIZE, 0);
+ if (ret != PAGE_SIZE) {
+ pr_err("pblk: could not add page to bio\n");
+ bio_put(bio);
+ bio = ERR_PTR(-ENOMEM);
+ goto out;
+ }
+
+ kaddr += PAGE_SIZE;
+ }
+out:
+ return bio;
+}
+
+int pblk_calc_secs(struct pblk *pblk, unsigned long secs_avail,
+ unsigned long secs_to_flush)
+{
+ int max = pblk->max_write_pgs;
+ int min = pblk->min_write_pgs;
+ int secs_to_sync = 0;
+
+ if (secs_avail >= max)
+ secs_to_sync = max;
+ else if (secs_avail >= min)
+ secs_to_sync = min * (secs_avail / min);
+ else if (secs_to_flush)
+ secs_to_sync = min;
+
+ return secs_to_sync;
+}
+
+static u64 __pblk_alloc_page(struct pblk *pblk, struct pblk_line *line,
+ int nr_secs)
+{
+ u64 addr;
+ int i;
+
+ /* logic error: ppa out-of-bounds. Prevent generating bad address */
+ if (line->cur_sec + nr_secs > pblk->lm.sec_per_line) {
+ WARN(1, "pblk: page allocation out of bounds\n");
+ nr_secs = pblk->lm.sec_per_line - line->cur_sec;
+ }
+
+ line->cur_sec = addr = find_next_zero_bit(line->map_bitmap,
+ pblk->lm.sec_per_line, line->cur_sec);
+ for (i = 0; i < nr_secs; i++, line->cur_sec++)
+ WARN_ON(test_and_set_bit(line->cur_sec, line->map_bitmap));
+
+ return addr;
+}
+
+u64 pblk_alloc_page(struct pblk *pblk, struct pblk_line *line, int nr_secs)
+{
+ u64 addr;
+
+ /* Lock needed in case a write fails and a recovery needs to remap
+ * failed write buffer entries
+ */
+ spin_lock(&line->lock);
+ addr = __pblk_alloc_page(pblk, line, nr_secs);
+ line->left_msecs -= nr_secs;
+ WARN(line->left_msecs < 0, "pblk: page allocation out of bounds\n");
+ spin_unlock(&line->lock);
+
+ return addr;
+}
+
+/*
+ * Submit emeta to one LUN in the raid line at the time to avoid a deadlock when
+ * taking the per LUN semaphore.
+ */
+static int pblk_line_submit_emeta_io(struct pblk *pblk, struct pblk_line *line,
+ u64 paddr, int dir)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct bio *bio;
+ struct nvm_rq rqd;
+ struct ppa_addr *ppa_list;
+ dma_addr_t dma_ppa_list;
+ void *emeta = line->emeta;
+ int min = pblk->min_write_pgs;
+ int left_ppas = lm->emeta_sec;
+ int id = line->id;
+ int rq_ppas, rq_len;
+ int cmd_op, bio_op;
+ int flags;
+ int i, j;
+ int ret;
+ DECLARE_COMPLETION_ONSTACK(wait);
+
+ if (dir == WRITE) {
+ bio_op = REQ_OP_WRITE;
+ cmd_op = NVM_OP_PWRITE;
+ flags = pblk_set_progr_mode(pblk, WRITE);
+ } else if (dir == READ) {
+ bio_op = REQ_OP_READ;
+ cmd_op = NVM_OP_PREAD;
+ flags = pblk_set_read_mode(pblk);
+ } else
+ return -EINVAL;
+
+ ppa_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL, &dma_ppa_list);
+ if (!ppa_list)
+ return -ENOMEM;
+
+next_rq:
+ memset(&rqd, 0, sizeof(struct nvm_rq));
+
+ rq_ppas = pblk_calc_secs(pblk, left_ppas, 0);
+ rq_len = rq_ppas * geo->sec_size;
+
+ bio = pblk_bio_map_addr(pblk, emeta, rq_ppas, rq_len, GFP_KERNEL);
+ if (IS_ERR(bio)) {
+ ret = PTR_ERR(bio);
+ goto free_rqd_dma;
+ }
+
+ bio->bi_iter.bi_sector = 0; /* internal bio */
+ bio_set_op_attrs(bio, bio_op, 0);
+
+ rqd.bio = bio;
+ rqd.opcode = cmd_op;
+ rqd.flags = flags;
+ rqd.nr_ppas = rq_ppas;
+ rqd.ppa_list = ppa_list;
+ rqd.dma_ppa_list = dma_ppa_list;
+ rqd.end_io = pblk_end_io_sync;
+ rqd.private = &wait;
+
+ if (dir == WRITE) {
+ for (i = 0; i < rqd.nr_ppas; ) {
+ spin_lock(&line->lock);
+ paddr = __pblk_alloc_page(pblk, line, min);
+ spin_unlock(&line->lock);
+ for (j = 0; j < min; j++, i++, paddr++)
+ rqd.ppa_list[i] =
+ addr_to_gen_ppa(pblk, paddr, id);
+ }
+ } else {
+ for (i = 0; i < rqd.nr_ppas; ) {
+ struct ppa_addr ppa = addr_to_gen_ppa(pblk, paddr, id);
+ int pos = pblk_dev_ppa_to_pos(geo, ppa);
+
+ while (test_bit(pos, line->blk_bitmap)) {
+ paddr += min;
+ if (pblk_boundary_paddr_checks(pblk, paddr)) {
+ pr_err("pblk: corrupt emeta line:%d\n",
+ line->id);
+ bio_put(bio);
+ ret = -EINTR;
+ goto free_rqd_dma;
+ }
+
+ ppa = addr_to_gen_ppa(pblk, paddr, id);
+ pos = pblk_dev_ppa_to_pos(geo, ppa);
+ }
+
+ if (pblk_boundary_paddr_checks(pblk, paddr + min)) {
+ pr_err("pblk: corrupt emeta line:%d\n",
+ line->id);
+ bio_put(bio);
+ ret = -EINTR;
+ goto free_rqd_dma;
+ }
+
+ for (j = 0; j < min; j++, i++, paddr++)
+ rqd.ppa_list[i] =
+ addr_to_gen_ppa(pblk, paddr, line->id);
+ }
+ }
+
+ ret = pblk_submit_io(pblk, &rqd);
+ if (ret) {
+ pr_err("pblk: emeta I/O submission failed: %d\n", ret);
+ bio_put(bio);
+ goto free_rqd_dma;
+ }
+
+ if (!wait_for_completion_io_timeout(&wait,
+ msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
+ pr_err("pblk: emeta I/O timed out\n");
+ }
+ reinit_completion(&wait);
+
+ bio_put(bio);
+
+ if (rqd.error) {
+ if (dir == WRITE)
+ pblk_log_write_err(pblk, &rqd);
+ else
+ pblk_log_read_err(pblk, &rqd);
+ }
+
+ emeta += rq_len;
+ left_ppas -= rq_ppas;
+ if (left_ppas)
+ goto next_rq;
+free_rqd_dma:
+ nvm_dev_dma_free(dev->parent, ppa_list, dma_ppa_list);
+ return ret;
+}
+
+u64 pblk_line_smeta_start(struct pblk *pblk, struct pblk_line *line)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line_meta *lm = &pblk->lm;
+ int bit;
+
+ /* This usually only happens on bad lines */
+ bit = find_first_zero_bit(line->blk_bitmap, lm->blk_per_line);
+ if (bit >= lm->blk_per_line)
+ return -1;
+
+ return bit * geo->sec_per_pl;
+}
+
+static int pblk_line_submit_smeta_io(struct pblk *pblk, struct pblk_line *line,
+ u64 paddr, int dir)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct bio *bio;
+ struct nvm_rq rqd;
+ __le64 *lba_list = NULL;
+ int i, ret;
+ int cmd_op, bio_op;
+ int flags;
+ DECLARE_COMPLETION_ONSTACK(wait);
+
+ if (dir == WRITE) {
+ bio_op = REQ_OP_WRITE;
+ cmd_op = NVM_OP_PWRITE;
+ flags = pblk_set_progr_mode(pblk, WRITE);
+ lba_list = pblk_line_emeta_to_lbas(line->emeta);
+ } else if (dir == READ) {
+ bio_op = REQ_OP_READ;
+ cmd_op = NVM_OP_PREAD;
+ flags = pblk_set_read_mode(pblk);
+ } else
+ return -EINVAL;
+
+ memset(&rqd, 0, sizeof(struct nvm_rq));
+
+ rqd.ppa_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL,
+ &rqd.dma_ppa_list);
+ if (!rqd.ppa_list)
+ return -ENOMEM;
+
+ bio = bio_map_kern(dev->q, line->smeta, lm->smeta_len, GFP_KERNEL);
+ if (IS_ERR(bio)) {
+ ret = PTR_ERR(bio);
+ goto free_ppa_list;
+ }
+
+ bio->bi_iter.bi_sector = 0; /* internal bio */
+ bio_set_op_attrs(bio, bio_op, 0);
+
+ rqd.bio = bio;
+ rqd.opcode = cmd_op;
+ rqd.flags = flags;
+ rqd.nr_ppas = lm->smeta_sec;
+ rqd.end_io = pblk_end_io_sync;
+ rqd.private = &wait;
+
+ for (i = 0; i < lm->smeta_sec; i++, paddr++) {
+ rqd.ppa_list[i] = addr_to_gen_ppa(pblk, paddr, line->id);
+ if (dir == WRITE)
+ lba_list[paddr] = cpu_to_le64(ADDR_EMPTY);
+ }
+
+ /*
+ * This I/O is sent by the write thread when a line is replace. Since
+ * the write thread is the only one sending write and erase commands,
+ * there is no need to take the LUN semaphore.
+ */
+ ret = pblk_submit_io(pblk, &rqd);
+ if (ret) {
+ pr_err("pblk: smeta I/O submission failed: %d\n", ret);
+ bio_put(bio);
+ goto free_ppa_list;
+ }
+
+ if (!wait_for_completion_io_timeout(&wait,
+ msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
+ pr_err("pblk: smeta I/O timed out\n");
+ }
+
+ if (rqd.error) {
+ if (dir == WRITE)
+ pblk_log_write_err(pblk, &rqd);
+ else
+ pblk_log_read_err(pblk, &rqd);
+ }
+
+free_ppa_list:
+ nvm_dev_dma_free(dev->parent, rqd.ppa_list, rqd.dma_ppa_list);
+
+ return ret;
+}
+
+int pblk_line_read_smeta(struct pblk *pblk, struct pblk_line *line)
+{
+ u64 bpaddr = pblk_line_smeta_start(pblk, line);
+
+ return pblk_line_submit_smeta_io(pblk, line, bpaddr, READ);
+}
+
+int pblk_line_read_emeta(struct pblk *pblk, struct pblk_line *line)
+{
+ return pblk_line_submit_emeta_io(pblk, line, line->emeta_ssec, READ);
+}
+
+static void pblk_setup_e_rq(struct pblk *pblk, struct nvm_rq *rqd,
+ struct ppa_addr ppa)
+{
+ rqd->opcode = NVM_OP_ERASE;
+ rqd->ppa_addr = ppa;
+ rqd->nr_ppas = 1;
+ rqd->flags = pblk_set_progr_mode(pblk, ERASE);
+ rqd->bio = NULL;
+}
+
+static int pblk_blk_erase_sync(struct pblk *pblk, struct ppa_addr ppa)
+{
+ struct nvm_rq rqd;
+ int ret;
+ DECLARE_COMPLETION_ONSTACK(wait);
+
+ memset(&rqd, 0, sizeof(struct nvm_rq));
+
+ pblk_setup_e_rq(pblk, &rqd, ppa);
+
+ rqd.end_io = pblk_end_io_sync;
+ rqd.private = &wait;
+
+ /* The write thread schedules erases so that it minimizes disturbances
+ * with writes. Thus, there is no need to take the LUN semaphore.
+ */
+ ret = pblk_submit_io(pblk, &rqd);
+ if (ret) {
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+
+ pr_err("pblk: could not sync erase line:%d,blk:%d\n",
+ pblk_dev_ppa_to_line(ppa),
+ pblk_dev_ppa_to_pos(geo, ppa));
+
+ rqd.error = ret;
+ goto out;
+ }
+
+ if (!wait_for_completion_io_timeout(&wait,
+ msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
+ pr_err("pblk: sync erase timed out\n");
+ }
+
+out:
+ rqd.private = pblk;
+ __pblk_end_io_erase(pblk, &rqd);
+
+ return 0;
+}
+
+int pblk_line_erase(struct pblk *pblk, struct pblk_line *line)
+{
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct ppa_addr ppa;
+ int bit = -1;
+
+ /* Erase only good blocks, one at a time */
+ do {
+ spin_lock(&line->lock);
+ bit = find_next_zero_bit(line->erase_bitmap, lm->blk_per_line,
+ bit + 1);
+ if (bit >= lm->blk_per_line) {
+ spin_unlock(&line->lock);
+ break;
+ }
+
+ ppa = pblk->luns[bit].bppa; /* set ch and lun */
+ ppa.g.blk = line->id;
+
+ atomic_dec(&line->left_eblks);
+ WARN_ON(test_and_set_bit(bit, line->erase_bitmap));
+ spin_unlock(&line->lock);
+
+ if (pblk_blk_erase_sync(pblk, ppa)) {
+ pr_err("pblk: failed to erase line %d\n", line->id);
+ return -ENOMEM;
+ }
+ } while (1);
+
+ return 0;
+}
+
+/* For now lines are always assumed full lines. Thus, smeta former and current
+ * lun bitmaps are omitted.
+ */
+static int pblk_line_set_metadata(struct pblk *pblk, struct pblk_line *line,
+ struct pblk_line *cur)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct line_smeta *smeta = line->smeta;
+ struct line_emeta *emeta = line->emeta;
+ int nr_blk_line;
+
+ /* After erasing the line, new bad blocks might appear and we risk
+ * having an invalid line
+ */
+ nr_blk_line = lm->blk_per_line -
+ bitmap_weight(line->blk_bitmap, lm->blk_per_line);
+ if (nr_blk_line < lm->min_blk_line) {
+ spin_lock(&l_mg->free_lock);
+ spin_lock(&line->lock);
+ line->state = PBLK_LINESTATE_BAD;
+ spin_unlock(&line->lock);
+
+ list_add_tail(&line->list, &l_mg->bad_list);
+ spin_unlock(&l_mg->free_lock);
+
+ pr_debug("pblk: line %d is bad\n", line->id);
+
+ return 0;
+ }
+
+ /* Run-time metadata */
+ line->lun_bitmap = ((void *)(smeta)) + sizeof(struct line_smeta);
+
+ /* Mark LUNs allocated in this line (all for now) */
+ bitmap_set(line->lun_bitmap, 0, lm->lun_bitmap_len);
+
+ smeta->header.identifier = cpu_to_le32(PBLK_MAGIC);
+ memcpy(smeta->header.uuid, pblk->instance_uuid, 16);
+ smeta->header.id = cpu_to_le32(line->id);
+ smeta->header.type = cpu_to_le16(line->type);
+ smeta->header.version = cpu_to_le16(1);
+
+ /* Start metadata */
+ smeta->seq_nr = cpu_to_le64(line->seq_nr);
+ smeta->window_wr_lun = cpu_to_le32(geo->nr_luns);
+
+ /* Fill metadata among lines */
+ if (cur) {
+ memcpy(line->lun_bitmap, cur->lun_bitmap, lm->lun_bitmap_len);
+ smeta->prev_id = cpu_to_le32(cur->id);
+ cur->emeta->next_id = cpu_to_le32(line->id);
+ } else {
+ smeta->prev_id = cpu_to_le32(PBLK_LINE_EMPTY);
+ }
+
+ /* All smeta must be set at this point */
+ smeta->header.crc = cpu_to_le32(pblk_calc_meta_header_crc(pblk, smeta));
+ smeta->crc = cpu_to_le32(pblk_calc_smeta_crc(pblk, smeta));
+
+ /* End metadata */
+ memcpy(&emeta->header, &smeta->header, sizeof(struct line_header));
+ emeta->seq_nr = cpu_to_le64(line->seq_nr);
+ emeta->nr_lbas = cpu_to_le64(line->sec_in_line);
+ emeta->nr_valid_lbas = cpu_to_le64(0);
+ emeta->next_id = cpu_to_le32(PBLK_LINE_EMPTY);
+ emeta->crc = cpu_to_le32(0);
+ emeta->prev_id = smeta->prev_id;
+
+ return 1;
+}
+
+/* For now lines are always assumed full lines. Thus, smeta former and current
+ * lun bitmaps are omitted.
+ */
+static int pblk_line_init_bb(struct pblk *pblk, struct pblk_line *line,
+ int init)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ int nr_bb = 0;
+ u64 off;
+ int bit = -1;
+
+ line->sec_in_line = lm->sec_per_line;
+
+ /* Capture bad block information on line mapping bitmaps */
+ while ((bit = find_next_bit(line->blk_bitmap, lm->blk_per_line,
+ bit + 1)) < lm->blk_per_line) {
+ off = bit * geo->sec_per_pl;
+ bitmap_shift_left(l_mg->bb_aux, l_mg->bb_template, off,
+ lm->sec_per_line);
+ bitmap_or(line->map_bitmap, line->map_bitmap, l_mg->bb_aux,
+ lm->sec_per_line);
+ line->sec_in_line -= geo->sec_per_blk;
+ if (bit >= lm->emeta_bb)
+ nr_bb++;
+ }
+
+ /* Mark smeta metadata sectors as bad sectors */
+ bit = find_first_zero_bit(line->blk_bitmap, lm->blk_per_line);
+ off = bit * geo->sec_per_pl;
+retry_smeta:
+ bitmap_set(line->map_bitmap, off, lm->smeta_sec);
+ line->sec_in_line -= lm->smeta_sec;
+ line->smeta_ssec = off;
+ line->cur_sec = off + lm->smeta_sec;
+
+ if (init && pblk_line_submit_smeta_io(pblk, line, off, WRITE)) {
+ pr_debug("pblk: line smeta I/O failed. Retry\n");
+ off += geo->sec_per_pl;
+ goto retry_smeta;
+ }
+
+ bitmap_copy(line->invalid_bitmap, line->map_bitmap, lm->sec_per_line);
+
+ /* Mark emeta metadata sectors as bad sectors. We need to consider bad
+ * blocks to make sure that there are enough sectors to store emeta
+ */
+ bit = lm->sec_per_line;
+ off = lm->sec_per_line - lm->emeta_sec;
+ bitmap_set(line->invalid_bitmap, off, lm->emeta_sec);
+ while (nr_bb) {
+ off -= geo->sec_per_pl;
+ if (!test_bit(off, line->invalid_bitmap)) {
+ bitmap_set(line->invalid_bitmap, off, geo->sec_per_pl);
+ nr_bb--;
+ }
+ }
+
+ line->sec_in_line -= lm->emeta_sec;
+ line->emeta_ssec = off;
+ line->vsc = line->left_ssecs = line->left_msecs = line->sec_in_line;
+
+ if (lm->sec_per_line - line->sec_in_line !=
+ bitmap_weight(line->invalid_bitmap, lm->sec_per_line)) {
+ spin_lock(&line->lock);
+ line->state = PBLK_LINESTATE_BAD;
+ spin_unlock(&line->lock);
+
+ list_add_tail(&line->list, &l_mg->bad_list);
+ pr_err("pblk: unexpected line %d is bad\n", line->id);
+
+ return 0;
+ }
+
+ return 1;
+}
+
+static int pblk_line_prepare(struct pblk *pblk, struct pblk_line *line)
+{
+ struct pblk_line_meta *lm = &pblk->lm;
+ int blk_in_line = atomic_read(&line->blk_in_line);
+
+ line->map_bitmap = mempool_alloc(pblk->line_meta_pool, GFP_ATOMIC);
+ if (!line->map_bitmap)
+ return -ENOMEM;
+ memset(line->map_bitmap, 0, lm->sec_bitmap_len);
+
+ /* invalid_bitmap is special since it is used when line is closed. No
+ * need to zeroized; it will be initialized using bb info form
+ * map_bitmap
+ */
+ line->invalid_bitmap = mempool_alloc(pblk->line_meta_pool, GFP_ATOMIC);
+ if (!line->invalid_bitmap) {
+ mempool_free(line->map_bitmap, pblk->line_meta_pool);
+ return -ENOMEM;
+ }
+
+ spin_lock(&line->lock);
+ if (line->state != PBLK_LINESTATE_FREE) {
+ spin_unlock(&line->lock);
+ WARN(1, "pblk: corrupted line state\n");
+ return -EINTR;
+ }
+ line->state = PBLK_LINESTATE_OPEN;
+
+ atomic_set(&line->left_eblks, blk_in_line);
+ atomic_set(&line->left_seblks, blk_in_line);
+ spin_unlock(&line->lock);
+
+ /* Bad blocks do not need to be erased */
+ bitmap_copy(line->erase_bitmap, line->blk_bitmap, lm->blk_per_line);
+
+ kref_init(&line->ref);
+
+ return 0;
+}
+
+int pblk_line_recov_alloc(struct pblk *pblk, struct pblk_line *line)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ int ret;
+
+ spin_lock(&l_mg->free_lock);
+ l_mg->data_line = line;
+ list_del(&line->list);
+
+ ret = pblk_line_prepare(pblk, line);
+ if (ret) {
+ list_add(&line->list, &l_mg->free_list);
+ spin_unlock(&l_mg->free_lock);
+ return ret;
+ }
+ spin_unlock(&l_mg->free_lock);
+
+ pblk_rl_free_lines_dec(&pblk->rl, line);
+
+ if (!pblk_line_init_bb(pblk, line, 0)) {
+ list_add(&line->list, &l_mg->free_list);
+ return -EINTR;
+ }
+
+ return 0;
+}
+
+void pblk_line_recov_close(struct pblk *pblk, struct pblk_line *line)
+{
+ mempool_free(line->map_bitmap, pblk->line_meta_pool);
+ line->map_bitmap = NULL;
+ line->smeta = NULL;
+ line->emeta = NULL;
+}
+
+struct pblk_line *pblk_line_get(struct pblk *pblk)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_line *line = NULL;
+ int bit;
+
+ lockdep_assert_held(&l_mg->free_lock);
+
+retry_get:
+ if (list_empty(&l_mg->free_list)) {
+ pr_err("pblk: no free lines\n");
+ goto out;
+ }
+
+ line = list_first_entry(&l_mg->free_list, struct pblk_line, list);
+ list_del(&line->list);
+ l_mg->nr_free_lines--;
+
+ bit = find_first_zero_bit(line->blk_bitmap, lm->blk_per_line);
+ if (unlikely(bit >= lm->blk_per_line)) {
+ spin_lock(&line->lock);
+ line->state = PBLK_LINESTATE_BAD;
+ spin_unlock(&line->lock);
+
+ list_add_tail(&line->list, &l_mg->bad_list);
+
+ pr_debug("pblk: line %d is bad\n", line->id);
+ goto retry_get;
+ }
+
+ if (pblk_line_prepare(pblk, line)) {
+ pr_err("pblk: failed to prepare line %d\n", line->id);
+ list_add(&line->list, &l_mg->free_list);
+ return NULL;
+ }
+
+out:
+ return line;
+}
+
+static struct pblk_line *pblk_line_retry(struct pblk *pblk,
+ struct pblk_line *line)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line *retry_line;
+
+ spin_lock(&l_mg->free_lock);
+ retry_line = pblk_line_get(pblk);
+ if (!retry_line) {
+ l_mg->data_line = NULL;
+ spin_unlock(&l_mg->free_lock);
+ return NULL;
+ }
+
+ retry_line->smeta = line->smeta;
+ retry_line->emeta = line->emeta;
+ retry_line->meta_line = line->meta_line;
+
+ pblk_line_free(pblk, line);
+ l_mg->data_line = retry_line;
+ spin_unlock(&l_mg->free_lock);
+
+ if (pblk_line_erase(pblk, retry_line)) {
+ spin_lock(&l_mg->free_lock);
+ l_mg->data_line = NULL;
+ spin_unlock(&l_mg->free_lock);
+ return NULL;
+ }
+
+ pblk_rl_free_lines_dec(&pblk->rl, retry_line);
+
+ return retry_line;
+}
+
+struct pblk_line *pblk_line_get_first_data(struct pblk *pblk)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line *line;
+ int meta_line;
+ int is_next = 0;
+
+ spin_lock(&l_mg->free_lock);
+ line = pblk_line_get(pblk);
+ if (!line) {
+ spin_unlock(&l_mg->free_lock);
+ return NULL;
+ }
+
+ line->seq_nr = l_mg->d_seq_nr++;
+ line->type = PBLK_LINETYPE_DATA;
+ l_mg->data_line = line;
+
+ meta_line = find_first_zero_bit(&l_mg->meta_bitmap, PBLK_DATA_LINES);
+ set_bit(meta_line, &l_mg->meta_bitmap);
+ line->smeta = l_mg->sline_meta[meta_line].meta;
+ line->emeta = l_mg->eline_meta[meta_line].meta;
+ line->meta_line = meta_line;
+
+ /* Allocate next line for preparation */
+ l_mg->data_next = pblk_line_get(pblk);
+ if (l_mg->data_next) {
+ l_mg->data_next->seq_nr = l_mg->d_seq_nr++;
+ l_mg->data_next->type = PBLK_LINETYPE_DATA;
+ is_next = 1;
+ }
+ spin_unlock(&l_mg->free_lock);
+
+ pblk_rl_free_lines_dec(&pblk->rl, line);
+ if (is_next)
+ pblk_rl_free_lines_dec(&pblk->rl, l_mg->data_next);
+
+ if (pblk_line_erase(pblk, line))
+ return NULL;
+
+retry_setup:
+ if (!pblk_line_set_metadata(pblk, line, NULL)) {
+ line = pblk_line_retry(pblk, line);
+ if (!line)
+ return NULL;
+
+ goto retry_setup;
+ }
+
+ if (!pblk_line_init_bb(pblk, line, 1)) {
+ line = pblk_line_retry(pblk, line);
+ if (!line)
+ return NULL;
+
+ goto retry_setup;
+ }
+
+ return line;
+}
+
+struct pblk_line *pblk_line_replace_data(struct pblk *pblk)
+{
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line *cur, *new;
+ unsigned int left_seblks;
+ int meta_line;
+ int is_next = 0;
+
+ cur = l_mg->data_line;
+ new = l_mg->data_next;
+ if (!new)
+ return NULL;
+ l_mg->data_line = new;
+
+retry_line:
+ left_seblks = atomic_read(&new->left_seblks);
+ if (left_seblks) {
+ /* If line is not fully erased, erase it */
+ if (atomic_read(&new->left_eblks)) {
+ if (pblk_line_erase(pblk, new))
+ return NULL;
+ } else {
+ io_schedule();
+ }
+ goto retry_line;
+ }
+
+ spin_lock(&l_mg->free_lock);
+ /* Allocate next line for preparation */
+ l_mg->data_next = pblk_line_get(pblk);
+ if (l_mg->data_next) {
+ l_mg->data_next->seq_nr = l_mg->d_seq_nr++;
+ l_mg->data_next->type = PBLK_LINETYPE_DATA;
+ is_next = 1;
+ }
+
+retry_meta:
+ meta_line = find_first_zero_bit(&l_mg->meta_bitmap, PBLK_DATA_LINES);
+ if (meta_line == PBLK_DATA_LINES) {
+ spin_unlock(&l_mg->free_lock);
+ io_schedule();
+ spin_lock(&l_mg->free_lock);
+ goto retry_meta;
+ }
+
+ set_bit(meta_line, &l_mg->meta_bitmap);
+ new->smeta = l_mg->sline_meta[meta_line].meta;
+ new->emeta = l_mg->eline_meta[meta_line].meta;
+ new->meta_line = meta_line;
+
+ memset(new->smeta, 0, lm->smeta_len);
+ memset(new->emeta, 0, lm->emeta_len);
+ spin_unlock(&l_mg->free_lock);
+
+ if (is_next)
+ pblk_rl_free_lines_dec(&pblk->rl, l_mg->data_next);
+
+retry_setup:
+ if (!pblk_line_set_metadata(pblk, new, cur)) {
+ new = pblk_line_retry(pblk, new);
+ if (!new)
+ return NULL;
+
+ goto retry_setup;
+ }
+
+ if (!pblk_line_init_bb(pblk, new, 1)) {
+ new = pblk_line_retry(pblk, new);
+ if (!new)
+ return NULL;
+
+ goto retry_setup;
+ }
+
+ return new;
+}
+
+void pblk_line_free(struct pblk *pblk, struct pblk_line *line)
+{
+ if (line->map_bitmap)
+ mempool_free(line->map_bitmap, pblk->line_meta_pool);
+ if (line->invalid_bitmap)
+ mempool_free(line->invalid_bitmap, pblk->line_meta_pool);
+
+ line->map_bitmap = NULL;
+ line->invalid_bitmap = NULL;
+ line->smeta = NULL;
+ line->emeta = NULL;
+}
+
+void pblk_line_put(struct kref *ref)
+{
+ struct pblk_line *line = container_of(ref, struct pblk_line, ref);
+ struct pblk *pblk = line->pblk;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+
+ spin_lock(&line->lock);
+ WARN_ON(line->state != PBLK_LINESTATE_GC);
+ line->state = PBLK_LINESTATE_FREE;
+ line->gc_group = PBLK_LINEGC_NONE;
+ pblk_line_free(pblk, line);
+ spin_unlock(&line->lock);
+
+ spin_lock(&l_mg->free_lock);
+ list_add_tail(&line->list, &l_mg->free_list);
+ l_mg->nr_free_lines++;
+ spin_unlock(&l_mg->free_lock);
+
+ pblk_rl_free_lines_inc(&pblk->rl, line);
+}
+
+int pblk_blk_erase_async(struct pblk *pblk, struct ppa_addr ppa)
+{
+ struct nvm_rq *rqd;
+ int err;
+
+ rqd = mempool_alloc(pblk->r_rq_pool, GFP_KERNEL);
+ memset(rqd, 0, pblk_r_rq_size);
+
+ pblk_setup_e_rq(pblk, rqd, ppa);
+
+ rqd->end_io = pblk_end_io_erase;
+ rqd->private = pblk;
+
+ /* The write thread schedules erases so that it minimizes disturbances
+ * with writes. Thus, there is no need to take the LUN semaphore.
+ */
+ err = pblk_submit_io(pblk, rqd);
+ if (err) {
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+
+ pr_err("pblk: could not async erase line:%d,blk:%d\n",
+ pblk_dev_ppa_to_line(ppa),
+ pblk_dev_ppa_to_pos(geo, ppa));
+ }
+
+ return err;
+}
+
+struct pblk_line *pblk_line_get_data(struct pblk *pblk)
+{
+ return pblk->l_mg.data_line;
+}
+
+struct pblk_line *pblk_line_get_data_next(struct pblk *pblk)
+{
+ return pblk->l_mg.data_next;
+}
+
+int pblk_line_is_full(struct pblk_line *line)
+{
+ return (line->left_msecs == 0);
+}
+
+void pblk_line_close(struct pblk *pblk, struct pblk_line *line)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct list_head *move_list;
+
+ line->emeta->crc = cpu_to_le32(pblk_calc_emeta_crc(pblk, line->emeta));
+
+ if (pblk_line_submit_emeta_io(pblk, line, line->cur_sec, WRITE))
+ pr_err("pblk: line %d close I/O failed\n", line->id);
+
+ WARN(!bitmap_full(line->map_bitmap, line->sec_in_line),
+ "pblk: corrupt closed line %d\n", line->id);
+
+ spin_lock(&l_mg->free_lock);
+ WARN_ON(!test_and_clear_bit(line->meta_line, &l_mg->meta_bitmap));
+ spin_unlock(&l_mg->free_lock);
+
+ spin_lock(&l_mg->gc_lock);
+ spin_lock(&line->lock);
+ WARN_ON(line->state != PBLK_LINESTATE_OPEN);
+ line->state = PBLK_LINESTATE_CLOSED;
+ move_list = pblk_line_gc_list(pblk, line);
+
+ list_add_tail(&line->list, move_list);
+
+ mempool_free(line->map_bitmap, pblk->line_meta_pool);
+ line->map_bitmap = NULL;
+ line->smeta = NULL;
+ line->emeta = NULL;
+
+ spin_unlock(&line->lock);
+ spin_unlock(&l_mg->gc_lock);
+}
+
+void pblk_line_close_ws(struct work_struct *work)
+{
+ struct pblk_line_ws *line_ws = container_of(work, struct pblk_line_ws,
+ ws);
+ struct pblk *pblk = line_ws->pblk;
+ struct pblk_line *line = line_ws->line;
+
+ pblk_line_close(pblk, line);
+ mempool_free(line_ws, pblk->line_ws_pool);
+}
+
+void pblk_line_mark_bb(struct work_struct *work)
+{
+ struct pblk_line_ws *line_ws = container_of(work, struct pblk_line_ws,
+ ws);
+ struct pblk *pblk = line_ws->pblk;
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct ppa_addr *ppa = line_ws->priv;
+ int ret;
+
+ ret = nvm_set_tgt_bb_tbl(dev, ppa, 1, NVM_BLK_T_GRWN_BAD);
+ if (ret) {
+ struct pblk_line *line;
+ int pos;
+
+ line = &pblk->lines[pblk_dev_ppa_to_line(*ppa)];
+ pos = pblk_dev_ppa_to_pos(&dev->geo, *ppa);
+
+ pr_err("pblk: failed to mark bb, line:%d, pos:%d\n",
+ line->id, pos);
+ }
+
+ kfree(ppa);
+ mempool_free(line_ws, pblk->line_ws_pool);
+}
+
+void pblk_line_run_ws(struct pblk *pblk, struct pblk_line *line, void *priv,
+ void (*work)(struct work_struct *))
+{
+ struct pblk_line_ws *line_ws;
+
+ line_ws = mempool_alloc(pblk->line_ws_pool, GFP_ATOMIC);
+ if (!line_ws)
+ return;
+
+ line_ws->pblk = pblk;
+ line_ws->line = line;
+ line_ws->priv = priv;
+
+ INIT_WORK(&line_ws->ws, work);
+ queue_work(pblk->kw_wq, &line_ws->ws);
+}
+
+void pblk_down_rq(struct pblk *pblk, struct ppa_addr *ppa_list, int nr_ppas,
+ unsigned long *lun_bitmap)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_lun *rlun;
+ int lun_id = ppa_list[0].g.ch * geo->luns_per_chnl + ppa_list[0].g.lun;
+ int ret;
+
+ /*
+ * Only send one inflight I/O per LUN. Since we map at a page
+ * granurality, all ppas in the I/O will map to the same LUN
+ */
+#ifdef CONFIG_NVM_DEBUG
+ int i;
+
+ for (i = 1; i < nr_ppas; i++)
+ WARN_ON(ppa_list[0].g.lun != ppa_list[i].g.lun ||
+ ppa_list[0].g.ch != ppa_list[i].g.ch);
+#endif
+ /* If the LUN has been locked for this same request, do no attempt to
+ * lock it again
+ */
+ if (test_and_set_bit(lun_id, lun_bitmap))
+ return;
+
+ rlun = &pblk->luns[lun_id];
+ ret = down_timeout(&rlun->wr_sem, msecs_to_jiffies(5000));
+ if (ret) {
+ switch (ret) {
+ case -ETIME:
+ pr_err("pblk: lun semaphore timed out\n");
+ break;
+ case -EINTR:
+ pr_err("pblk: lun semaphore timed out\n");
+ break;
+ }
+ }
+}
+
+void pblk_up_rq(struct pblk *pblk, struct ppa_addr *ppa_list, int nr_ppas,
+ unsigned long *lun_bitmap)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_lun *rlun;
+ int nr_luns = geo->nr_luns;
+ int bit = -1;
+
+ while ((bit = find_next_bit(lun_bitmap, nr_luns, bit + 1)) < nr_luns) {
+ rlun = &pblk->luns[bit];
+ up(&rlun->wr_sem);
+ }
+
+ kfree(lun_bitmap);
+}
+
+void pblk_update_map(struct pblk *pblk, sector_t lba, struct ppa_addr ppa)
+{
+ struct ppa_addr l2p_ppa;
+
+ /* logic error: lba out-of-bounds. Ignore update */
+ if (!(lba < pblk->rl.nr_secs)) {
+ WARN(1, "pblk: corrupted L2P map request\n");
+ return;
+ }
+
+ spin_lock(&pblk->trans_lock);
+ l2p_ppa = pblk_trans_map_get(pblk, lba);
+
+ if (!pblk_addr_in_cache(l2p_ppa) && !pblk_ppa_empty(l2p_ppa))
+ pblk_map_invalidate(pblk, l2p_ppa);
+
+ pblk_trans_map_set(pblk, lba, ppa);
+ spin_unlock(&pblk->trans_lock);
+}
+
+void pblk_update_map_cache(struct pblk *pblk, sector_t lba, struct ppa_addr ppa)
+{
+#ifdef CONFIG_NVM_DEBUG
+ /* Callers must ensure that the ppa points to a cache address */
+ BUG_ON(!pblk_addr_in_cache(ppa));
+ BUG_ON(pblk_rb_pos_oob(&pblk->rwb, pblk_addr_to_cacheline(ppa)));
+#endif
+
+ pblk_update_map(pblk, lba, ppa);
+}
+
+int pblk_update_map_gc(struct pblk *pblk, sector_t lba, struct ppa_addr ppa,
+ struct pblk_line *gc_line)
+{
+ struct ppa_addr l2p_ppa;
+ int ret = 1;
+
+#ifdef CONFIG_NVM_DEBUG
+ /* Callers must ensure that the ppa points to a cache address */
+ BUG_ON(!pblk_addr_in_cache(ppa));
+ BUG_ON(pblk_rb_pos_oob(&pblk->rwb, pblk_addr_to_cacheline(ppa)));
+#endif
+
+ /* logic error: lba out-of-bounds. Ignore update */
+ if (!(lba < pblk->rl.nr_secs)) {
+ WARN(1, "pblk: corrupted L2P map request\n");
+ return 0;
+ }
+
+ spin_lock(&pblk->trans_lock);
+ l2p_ppa = pblk_trans_map_get(pblk, lba);
+
+ /* Prevent updated entries to be overwritten by GC */
+ if (pblk_addr_in_cache(l2p_ppa) || pblk_ppa_empty(l2p_ppa) ||
+ pblk_tgt_ppa_to_line(l2p_ppa) != gc_line->id) {
+ ret = 0;
+ goto out;
+ }
+
+ pblk_trans_map_set(pblk, lba, ppa);
+out:
+ spin_unlock(&pblk->trans_lock);
+ return ret;
+}
+
+void pblk_update_map_dev(struct pblk *pblk, sector_t lba, struct ppa_addr ppa,
+ struct ppa_addr entry_line)
+{
+ struct ppa_addr l2p_line;
+
+#ifdef CONFIG_NVM_DEBUG
+ /* Callers must ensure that the ppa points to a device address */
+ BUG_ON(pblk_addr_in_cache(ppa));
+#endif
+ /* Invalidate and discard padded entries */
+ if (lba == ADDR_EMPTY) {
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_inc(&pblk->padded_wb);
+#endif
+ pblk_map_invalidate(pblk, ppa);
+ return;
+ }
+
+ /* logic error: lba out-of-bounds. Ignore update */
+ if (!(lba < pblk->rl.nr_secs)) {
+ WARN(1, "pblk: corrupted L2P map request\n");
+ return;
+ }
+
+ spin_lock(&pblk->trans_lock);
+ l2p_line = pblk_trans_map_get(pblk, lba);
+
+ /* Do not update L2P if the cacheline has been updated. In this case,
+ * the mapped ppa must be invalidated
+ */
+ if (l2p_line.ppa != entry_line.ppa) {
+ if (!pblk_ppa_empty(ppa))
+ pblk_map_invalidate(pblk, ppa);
+ goto out;
+ }
+
+#ifdef CONFIG_NVM_DEBUG
+ WARN_ON(!pblk_addr_in_cache(l2p_line) && !pblk_ppa_empty(l2p_line));
+#endif
+
+ pblk_trans_map_set(pblk, lba, ppa);
+out:
+ spin_unlock(&pblk->trans_lock);
+}
+
+void pblk_lookup_l2p_seq(struct pblk *pblk, struct ppa_addr *ppas,
+ sector_t blba, int nr_secs)
+{
+ int i;
+
+ spin_lock(&pblk->trans_lock);
+ for (i = 0; i < nr_secs; i++)
+ ppas[i] = pblk_trans_map_get(pblk, blba + i);
+ spin_unlock(&pblk->trans_lock);
+}
+
+void pblk_lookup_l2p_rand(struct pblk *pblk, struct ppa_addr *ppas,
+ u64 *lba_list, int nr_secs)
+{
+ sector_t lba;
+ int i;
+
+ spin_lock(&pblk->trans_lock);
+ for (i = 0; i < nr_secs; i++) {
+ lba = lba_list[i];
+ if (lba == ADDR_EMPTY) {
+ ppas[i].ppa = ADDR_EMPTY;
+ } else {
+ /* logic error: lba out-of-bounds. Ignore update */
+ if (!(lba < pblk->rl.nr_secs)) {
+ WARN(1, "pblk: corrupted L2P map request\n");
+ continue;
+ }
+ ppas[i] = pblk_trans_map_get(pblk, lba);
+ }
+ }
+ spin_unlock(&pblk->trans_lock);
+}
diff --git a/drivers/lightnvm/pblk-gc.c b/drivers/lightnvm/pblk-gc.c
new file mode 100644
index 0000000..eaf479c
--- /dev/null
+++ b/drivers/lightnvm/pblk-gc.c
@@ -0,0 +1,555 @@
+/*
+ * Copyright (C) 2016 CNEX Labs
+ * Initial release: Javier Gonzalez <javier@cnexlabs.com>
+ * Matias Bjorling <matias@cnexlabs.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * pblk-gc.c - pblk's garbage collector
+ */
+
+#include "pblk.h"
+#include <linux/delay.h>
+
+static void pblk_gc_free_gc_rq(struct pblk_gc_rq *gc_rq)
+{
+ kfree(gc_rq->data);
+ kfree(gc_rq->lba_list);
+ kfree(gc_rq);
+}
+
+static int pblk_gc_write(struct pblk *pblk)
+{
+ struct pblk_gc *gc = &pblk->gc;
+ struct pblk_gc_rq *gc_rq, *tgc_rq;
+ LIST_HEAD(w_list);
+
+ spin_lock(&gc->w_lock);
+ if (list_empty(&gc->w_list)) {
+ spin_unlock(&gc->w_lock);
+ return 1;
+ }
+
+ list_for_each_entry_safe(gc_rq, tgc_rq, &gc->w_list, list) {
+ list_move_tail(&gc_rq->list, &w_list);
+ gc->w_entries--;
+ }
+ spin_unlock(&gc->w_lock);
+
+ list_for_each_entry_safe(gc_rq, tgc_rq, &w_list, list) {
+ pblk_write_gc_to_cache(pblk, gc_rq->data, gc_rq->lba_list,
+ gc_rq->nr_secs, gc_rq->secs_to_gc,
+ gc_rq->line, PBLK_IOTYPE_GC);
+
+ kref_put(&gc_rq->line->ref, pblk_line_put);
+
+ list_del(&gc_rq->list);
+ pblk_gc_free_gc_rq(gc_rq);
+ }
+
+ return 0;
+}
+
+static void pblk_gc_writer_kick(struct pblk_gc *gc)
+{
+ wake_up_process(gc->gc_writer_ts);
+}
+
+/*
+ * Responsible for managing all memory related to a gc request. Also in case of
+ * failure
+ */
+static int pblk_gc_move_valid_secs(struct pblk *pblk, struct pblk_line *line,
+ u64 *lba_list, unsigned int nr_secs)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_gc *gc = &pblk->gc;
+ struct pblk_gc_rq *gc_rq;
+ void *data;
+ unsigned int secs_to_gc;
+ int ret = NVM_IO_OK;
+
+ data = kmalloc(nr_secs * geo->sec_size, GFP_KERNEL);
+ if (!data) {
+ ret = NVM_IO_ERR;
+ goto free_lba_list;
+ }
+
+ /* Read from GC victim block */
+ if (pblk_submit_read_gc(pblk, lba_list, data, nr_secs,
+ &secs_to_gc, line)) {
+ ret = NVM_IO_ERR;
+ goto free_data;
+ }
+
+ if (!secs_to_gc)
+ goto free_data;
+
+ gc_rq = kmalloc(sizeof(struct pblk_gc_rq), GFP_KERNEL);
+ if (!gc_rq) {
+ ret = NVM_IO_ERR;
+ goto free_data;
+ }
+
+ gc_rq->line = line;
+ gc_rq->data = data;
+ gc_rq->lba_list = lba_list;
+ gc_rq->nr_secs = nr_secs;
+ gc_rq->secs_to_gc = secs_to_gc;
+
+ kref_get(&line->ref);
+
+retry:
+ spin_lock(&gc->w_lock);
+ if (gc->w_entries > 256) {
+ spin_unlock(&gc->w_lock);
+ usleep_range(256, 1024);
+ goto retry;
+ }
+ gc->w_entries++;
+ list_add_tail(&gc_rq->list, &gc->w_list);
+ spin_unlock(&gc->w_lock);
+
+ pblk_gc_writer_kick(&pblk->gc);
+
+ return NVM_IO_OK;
+
+free_data:
+ kfree(data);
+free_lba_list:
+ kfree(lba_list);
+
+ return ret;
+}
+
+static void pblk_put_line_back(struct pblk *pblk, struct pblk_line *line)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct list_head *move_list;
+
+ spin_lock(&line->lock);
+ WARN_ON(line->state != PBLK_LINESTATE_GC);
+ line->state = PBLK_LINESTATE_CLOSED;
+ move_list = pblk_line_gc_list(pblk, line);
+ spin_unlock(&line->lock);
+
+ if (move_list) {
+ spin_lock(&l_mg->gc_lock);
+ list_add_tail(&line->list, move_list);
+ spin_unlock(&l_mg->gc_lock);
+ }
+}
+
+static void pblk_gc_line_ws(struct work_struct *work)
+{
+ struct pblk_line_ws *line_ws = container_of(work, struct pblk_line_ws,
+ ws);
+ struct pblk *pblk = line_ws->pblk;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line *line = line_ws->line;
+ struct pblk_line_meta *lm = &pblk->lm;
+ __le64 *lba_list = line_ws->priv;
+ u64 *gc_list;
+ int sec_left;
+ int nr_ppas, bit;
+ int put_line = 1;
+
+ pr_debug("pblk: line '%d' being reclaimed for GC\n", line->id);
+
+ spin_lock(&line->lock);
+ sec_left = line->vsc;
+ if (!sec_left) {
+ /* Lines are erased before being used (l_mg->data_/log_next) */
+ spin_unlock(&line->lock);
+ goto out;
+ }
+ spin_unlock(&line->lock);
+
+ if (sec_left < 0) {
+ pr_err("pblk: corrupted GC line (%d)\n", line->id);
+ put_line = 0;
+ pblk_put_line_back(pblk, line);
+ goto out;
+ }
+
+ bit = -1;
+next_rq:
+ gc_list = kmalloc_array(pblk->max_write_pgs, sizeof(u64), GFP_KERNEL);
+ if (!gc_list) {
+ put_line = 0;
+ pblk_put_line_back(pblk, line);
+ goto out;
+ }
+
+ nr_ppas = 0;
+ do {
+ bit = find_next_zero_bit(line->invalid_bitmap, lm->sec_per_line,
+ bit + 1);
+ if (bit > line->emeta_ssec)
+ break;
+
+ gc_list[nr_ppas++] = le64_to_cpu(lba_list[bit]);
+ } while (nr_ppas < pblk->max_write_pgs);
+
+ if (unlikely(!nr_ppas)) {
+ kfree(gc_list);
+ goto out;
+ }
+
+ if (pblk_gc_move_valid_secs(pblk, line, gc_list, nr_ppas)) {
+ pr_err("pblk: could not GC all sectors: line:%d (%d/%d/%d)\n",
+ line->id, line->vsc,
+ nr_ppas, nr_ppas);
+ put_line = 0;
+ pblk_put_line_back(pblk, line);
+ goto out;
+ }
+
+ sec_left -= nr_ppas;
+ if (sec_left > 0)
+ goto next_rq;
+
+out:
+ pblk_mfree(line->emeta, l_mg->emeta_alloc_type);
+ mempool_free(line_ws, pblk->line_ws_pool);
+ atomic_dec(&pblk->gc.inflight_gc);
+ if (put_line)
+ kref_put(&line->ref, pblk_line_put);
+}
+
+static int pblk_gc_line(struct pblk *pblk, struct pblk_line *line)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_line_ws *line_ws;
+ __le64 *lba_list;
+ int ret;
+
+ line_ws = mempool_alloc(pblk->line_ws_pool, GFP_KERNEL);
+ line->emeta = pblk_malloc(lm->emeta_len, l_mg->emeta_alloc_type,
+ GFP_KERNEL);
+ if (!line->emeta) {
+ pr_err("pblk: cannot use GC emeta\n");
+ goto fail_free_ws;
+ }
+
+ ret = pblk_line_read_emeta(pblk, line);
+ if (ret) {
+ pr_err("pblk: line %d read emeta failed (%d)\n", line->id, ret);
+ goto fail_free_emeta;
+ }
+
+ /* If this read fails, it means that emeta is corrupted. For now, leave
+ * the line untouched. TODO: Implement a recovery routine that scans and
+ * moves all sectors on the line.
+ */
+ lba_list = pblk_recov_get_lba_list(pblk, line->emeta);
+ if (!lba_list) {
+ pr_err("pblk: could not interpret emeta (line %d)\n", line->id);
+ goto fail_free_emeta;
+ }
+
+ line_ws->pblk = pblk;
+ line_ws->line = line;
+ line_ws->priv = lba_list;
+
+ INIT_WORK(&line_ws->ws, pblk_gc_line_ws);
+ queue_work(pblk->gc.gc_reader_wq, &line_ws->ws);
+
+ return 0;
+
+fail_free_emeta:
+ pblk_mfree(line->emeta, l_mg->emeta_alloc_type);
+fail_free_ws:
+ mempool_free(line_ws, pblk->line_ws_pool);
+ pblk_put_line_back(pblk, line);
+
+ return 1;
+}
+
+static void pblk_gc_lines(struct pblk *pblk, struct list_head *gc_list)
+{
+ struct pblk_line *line, *tline;
+
+ list_for_each_entry_safe(line, tline, gc_list, list) {
+ if (pblk_gc_line(pblk, line))
+ pr_err("pblk: failed to GC line %d\n", line->id);
+ list_del(&line->list);
+ }
+}
+
+/*
+ * Lines with no valid sectors will be returned to the free list immediately. If
+ * GC is activated - either because the free block count is under the determined
+ * threshold, or because it is being forced from user space - only lines with a
+ * high count of invalid sectors will be recycled.
+ */
+static void pblk_gc_run(struct pblk *pblk)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_gc *gc = &pblk->gc;
+ struct pblk_line *line, *tline;
+ unsigned int nr_blocks_free, nr_blocks_need;
+ struct list_head *group_list;
+ int run_gc, gc_group = 0;
+ int prev_gc = 0;
+ int inflight_gc = atomic_read(&gc->inflight_gc);
+ LIST_HEAD(gc_list);
+
+ spin_lock(&l_mg->gc_lock);
+ list_for_each_entry_safe(line, tline, &l_mg->gc_full_list, list) {
+ spin_lock(&line->lock);
+ WARN_ON(line->state != PBLK_LINESTATE_CLOSED);
+ line->state = PBLK_LINESTATE_GC;
+ spin_unlock(&line->lock);
+
+ list_del(&line->list);
+ kref_put(&line->ref, pblk_line_put);
+ }
+ spin_unlock(&l_mg->gc_lock);
+
+ nr_blocks_need = pblk_rl_gc_thrs(&pblk->rl);
+ nr_blocks_free = pblk_rl_nr_free_blks(&pblk->rl);
+ run_gc = (nr_blocks_need > nr_blocks_free || gc->gc_forced);
+
+next_gc_group:
+ group_list = l_mg->gc_lists[gc_group++];
+ spin_lock(&l_mg->gc_lock);
+ while (run_gc && !list_empty(group_list)) {
+ /* No need to queue up more GC lines than we can handle */
+ if (!run_gc || inflight_gc > gc->gc_jobs_active) {
+ spin_unlock(&l_mg->gc_lock);
+ pblk_gc_lines(pblk, &gc_list);
+ return;
+ }
+
+ line = list_first_entry(group_list, struct pblk_line, list);
+ nr_blocks_free += atomic_read(&line->blk_in_line);
+
+ spin_lock(&line->lock);
+ WARN_ON(line->state != PBLK_LINESTATE_CLOSED);
+ line->state = PBLK_LINESTATE_GC;
+ list_move_tail(&line->list, &gc_list);
+ atomic_inc(&gc->inflight_gc);
+ inflight_gc++;
+ spin_unlock(&line->lock);
+
+ prev_gc = 1;
+ run_gc = (nr_blocks_need > nr_blocks_free || gc->gc_forced);
+ }
+ spin_unlock(&l_mg->gc_lock);
+
+ pblk_gc_lines(pblk, &gc_list);
+
+ if (!prev_gc && pblk->rl.rb_state > gc_group &&
+ gc_group < PBLK_NR_GC_LISTS)
+ goto next_gc_group;
+}
+
+
+static void pblk_gc_kick(struct pblk *pblk)
+{
+ struct pblk_gc *gc = &pblk->gc;
+
+ wake_up_process(gc->gc_ts);
+ pblk_gc_writer_kick(gc);
+ mod_timer(&gc->gc_timer, jiffies + msecs_to_jiffies(GC_TIME_MSECS));
+}
+
+static void pblk_gc_timer(unsigned long data)
+{
+ struct pblk *pblk = (struct pblk *)data;
+
+ pblk_gc_kick(pblk);
+}
+
+static int pblk_gc_ts(void *data)
+{
+ struct pblk *pblk = data;
+
+ while (!kthread_should_stop()) {
+ pblk_gc_run(pblk);
+ set_current_state(TASK_INTERRUPTIBLE);
+ io_schedule();
+ }
+
+ return 0;
+}
+
+static int pblk_gc_writer_ts(void *data)
+{
+ struct pblk *pblk = data;
+
+ while (!kthread_should_stop()) {
+ if (!pblk_gc_write(pblk))
+ continue;
+ set_current_state(TASK_INTERRUPTIBLE);
+ io_schedule();
+ }
+
+ return 0;
+}
+
+static void pblk_gc_start(struct pblk *pblk)
+{
+ pblk->gc.gc_active = 1;
+
+ pr_debug("pblk: gc start\n");
+}
+
+int pblk_gc_status(struct pblk *pblk)
+{
+ struct pblk_gc *gc = &pblk->gc;
+ int ret;
+
+ spin_lock(&gc->lock);
+ ret = gc->gc_active;
+ spin_unlock(&gc->lock);
+
+ return ret;
+}
+
+static void __pblk_gc_should_start(struct pblk *pblk)
+{
+ struct pblk_gc *gc = &pblk->gc;
+
+ lockdep_assert_held(&gc->lock);
+
+ if (gc->gc_enabled && !gc->gc_active)
+ pblk_gc_start(pblk);
+}
+
+void pblk_gc_should_start(struct pblk *pblk)
+{
+ struct pblk_gc *gc = &pblk->gc;
+
+ spin_lock(&gc->lock);
+ __pblk_gc_should_start(pblk);
+ spin_unlock(&gc->lock);
+}
+
+/*
+ * If flush_wq == 1 then no lock should be held by the caller since
+ * flush_workqueue can sleep
+ */
+static void pblk_gc_stop(struct pblk *pblk, int flush_wq)
+{
+ spin_lock(&pblk->gc.lock);
+ pblk->gc.gc_active = 0;
+ spin_unlock(&pblk->gc.lock);
+
+ pr_debug("pblk: gc stop\n");
+}
+
+void pblk_gc_should_stop(struct pblk *pblk)
+{
+ struct pblk_gc *gc = &pblk->gc;
+
+ if (gc->gc_active && !gc->gc_forced)
+ pblk_gc_stop(pblk, 0);
+}
+
+void pblk_gc_sysfs_state_show(struct pblk *pblk, int *gc_enabled,
+ int *gc_active)
+{
+ struct pblk_gc *gc = &pblk->gc;
+
+ spin_lock(&gc->lock);
+ *gc_enabled = gc->gc_enabled;
+ *gc_active = gc->gc_active;
+ spin_unlock(&gc->lock);
+}
+
+void pblk_gc_sysfs_force(struct pblk *pblk, int force)
+{
+ struct pblk_gc *gc = &pblk->gc;
+ int rsv = 0;
+
+ spin_lock(&gc->lock);
+ if (force) {
+ gc->gc_enabled = 1;
+ rsv = 64;
+ }
+ pblk_rl_set_gc_rsc(&pblk->rl, rsv);
+ gc->gc_forced = force;
+ __pblk_gc_should_start(pblk);
+ spin_unlock(&gc->lock);
+}
+
+int pblk_gc_init(struct pblk *pblk)
+{
+ struct pblk_gc *gc = &pblk->gc;
+ int ret;
+
+ gc->gc_ts = kthread_create(pblk_gc_ts, pblk, "pblk-gc-ts");
+ if (IS_ERR(gc->gc_ts)) {
+ pr_err("pblk: could not allocate GC main kthread\n");
+ return PTR_ERR(gc->gc_ts);
+ }
+
+ gc->gc_writer_ts = kthread_create(pblk_gc_writer_ts, pblk,
+ "pblk-gc-writer-ts");
+ if (IS_ERR(gc->gc_writer_ts)) {
+ pr_err("pblk: could not allocate GC writer kthread\n");
+ ret = PTR_ERR(gc->gc_writer_ts);
+ goto fail_free_main_kthread;
+ }
+
+ setup_timer(&gc->gc_timer, pblk_gc_timer, (unsigned long)pblk);
+ mod_timer(&gc->gc_timer, jiffies + msecs_to_jiffies(GC_TIME_MSECS));
+
+ gc->gc_active = 0;
+ gc->gc_forced = 0;
+ gc->gc_enabled = 1;
+ gc->gc_jobs_active = 8;
+ gc->w_entries = 0;
+ atomic_set(&gc->inflight_gc, 0);
+
+ gc->gc_reader_wq = alloc_workqueue("pblk-gc-reader-wq",
+ WQ_MEM_RECLAIM | WQ_UNBOUND, gc->gc_jobs_active);
+ if (!gc->gc_reader_wq) {
+ pr_err("pblk: could not allocate GC reader workqueue\n");
+ ret = -ENOMEM;
+ goto fail_free_writer_kthread;
+ }
+
+ spin_lock_init(&gc->lock);
+ spin_lock_init(&gc->w_lock);
+ INIT_LIST_HEAD(&gc->w_list);
+
+ return 0;
+
+fail_free_writer_kthread:
+ kthread_stop(gc->gc_writer_ts);
+fail_free_main_kthread:
+ kthread_stop(gc->gc_ts);
+
+ return ret;
+}
+
+void pblk_gc_exit(struct pblk *pblk)
+{
+ struct pblk_gc *gc = &pblk->gc;
+
+ flush_workqueue(gc->gc_reader_wq);
+
+ del_timer(&gc->gc_timer);
+ pblk_gc_stop(pblk, 1);
+
+ if (gc->gc_ts)
+ kthread_stop(gc->gc_ts);
+
+ if (pblk->gc.gc_reader_wq)
+ destroy_workqueue(pblk->gc.gc_reader_wq);
+
+ if (gc->gc_writer_ts)
+ kthread_stop(gc->gc_writer_ts);
+}
diff --git a/drivers/lightnvm/pblk-init.c b/drivers/lightnvm/pblk-init.c
new file mode 100644
index 0000000..ae8cd6d
--- /dev/null
+++ b/drivers/lightnvm/pblk-init.c
@@ -0,0 +1,962 @@
+/*
+ * Copyright (C) 2015 IT University of Copenhagen (rrpc.c)
+ * Copyright (C) 2016 CNEX Labs
+ * Initial release: Javier Gonzalez <javier@cnexlabs.com>
+ * Matias Bjorling <matias@cnexlabs.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Implementation of a physical block-device target for Open-channel SSDs.
+ *
+ * pblk-init.c - pblk's initialization.
+ */
+
+#include "pblk.h"
+
+static struct kmem_cache *pblk_blk_ws_cache, *pblk_rec_cache, *pblk_r_rq_cache,
+ *pblk_w_rq_cache, *pblk_line_meta_cache;
+static DECLARE_RWSEM(pblk_lock);
+
+static int pblk_rw_io(struct request_queue *q, struct pblk *pblk,
+ struct bio *bio)
+{
+ int ret;
+
+ /* Read requests must be <= 256kb due to NVMe's 64 bit completion bitmap
+ * constraint. Writes can be of arbitrary size.
+ */
+ if (bio_data_dir(bio) == READ) {
+ blk_queue_split(q, &bio, q->bio_split);
+ ret = pblk_submit_read(pblk, bio);
+ if (ret == NVM_IO_DONE && bio_flagged(bio, BIO_CLONED))
+ bio_put(bio);
+
+ return ret;
+ }
+
+ /* Prevent deadlock in the case of a modest LUN configuration and large
+ * user I/Os. Unless stalled, the rate limiter leaves at least 256KB
+ * available for user I/O.
+ */
+ if (unlikely(pblk_get_secs(bio) >= pblk_rl_sysfs_rate_show(&pblk->rl)))
+ blk_queue_split(q, &bio, q->bio_split);
+
+ return pblk_write_to_cache(pblk, bio, PBLK_IOTYPE_USER);
+}
+
+static blk_qc_t pblk_make_rq(struct request_queue *q, struct bio *bio)
+{
+ struct pblk *pblk = q->queuedata;
+
+ if (bio_op(bio) == REQ_OP_DISCARD) {
+ pblk_discard(pblk, bio);
+ if (!(bio->bi_opf & REQ_PREFLUSH)) {
+ bio_endio(bio);
+ return BLK_QC_T_NONE;
+ }
+ }
+
+ switch (pblk_rw_io(q, pblk, bio)) {
+ case NVM_IO_ERR:
+ bio_io_error(bio);
+ break;
+ case NVM_IO_DONE:
+ bio_endio(bio);
+ break;
+ }
+
+ return BLK_QC_T_NONE;
+}
+
+static void pblk_l2p_free(struct pblk *pblk)
+{
+ vfree(pblk->trans_map);
+}
+
+static int pblk_l2p_init(struct pblk *pblk)
+{
+ sector_t i;
+ struct ppa_addr ppa;
+ int entry_size = 8;
+
+ if (pblk->ppaf_bitsize < 32)
+ entry_size = 4;
+
+ pblk->trans_map = vmalloc(entry_size * pblk->rl.nr_secs);
+ if (!pblk->trans_map)
+ return -ENOMEM;
+
+ pblk_ppa_set_empty(&ppa);
+
+ for (i = 0; i < pblk->rl.nr_secs; i++)
+ pblk_trans_map_set(pblk, i, ppa);
+
+ return 0;
+}
+
+static void pblk_rwb_free(struct pblk *pblk)
+{
+ if (pblk_rb_tear_down_check(&pblk->rwb))
+ pr_err("pblk: write buffer error on tear down\n");
+
+ pblk_rb_data_free(&pblk->rwb);
+ vfree(pblk_rb_entries_ref(&pblk->rwb));
+}
+
+static int pblk_rwb_init(struct pblk *pblk)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_rb_entry *entries;
+ unsigned long nr_entries;
+ unsigned int power_size, power_seg_sz;
+
+ nr_entries = pblk_rb_calculate_size(pblk->pgs_in_buffer);
+
+ entries = vzalloc(nr_entries * sizeof(struct pblk_rb_entry));
+ if (!entries)
+ return -ENOMEM;
+
+ power_size = get_count_order(nr_entries);
+ power_seg_sz = get_count_order(geo->sec_size);
+
+ return pblk_rb_init(&pblk->rwb, entries, power_size, power_seg_sz);
+}
+
+/* Minimum pages needed within a lun */
+#define PAGE_POOL_SIZE 16
+#define ADDR_POOL_SIZE 64
+
+static int pblk_set_ppaf(struct pblk *pblk)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct nvm_addr_format ppaf = geo->ppaf;
+ int power_len;
+
+ /* Re-calculate channel and lun format to adapt to configuration */
+ power_len = get_count_order(geo->nr_chnls);
+ if (1 << power_len != geo->nr_chnls) {
+ pr_err("pblk: supports only power-of-two channel config.\n");
+ return -EINVAL;
+ }
+ ppaf.ch_len = power_len;
+
+ power_len = get_count_order(geo->luns_per_chnl);
+ if (1 << power_len != geo->luns_per_chnl) {
+ pr_err("pblk: supports only power-of-two LUN config.\n");
+ return -EINVAL;
+ }
+ ppaf.lun_len = power_len;
+
+ pblk->ppaf.sec_offset = 0;
+ pblk->ppaf.pln_offset = ppaf.sect_len;
+ pblk->ppaf.ch_offset = pblk->ppaf.pln_offset + ppaf.pln_len;
+ pblk->ppaf.lun_offset = pblk->ppaf.ch_offset + ppaf.ch_len;
+ pblk->ppaf.pg_offset = pblk->ppaf.lun_offset + ppaf.lun_len;
+ pblk->ppaf.blk_offset = pblk->ppaf.pg_offset + ppaf.pg_len;
+ pblk->ppaf.sec_mask = (1ULL << ppaf.sect_len) - 1;
+ pblk->ppaf.pln_mask = ((1ULL << ppaf.pln_len) - 1) <<
+ pblk->ppaf.pln_offset;
+ pblk->ppaf.ch_mask = ((1ULL << ppaf.ch_len) - 1) <<
+ pblk->ppaf.ch_offset;
+ pblk->ppaf.lun_mask = ((1ULL << ppaf.lun_len) - 1) <<
+ pblk->ppaf.lun_offset;
+ pblk->ppaf.pg_mask = ((1ULL << ppaf.pg_len) - 1) <<
+ pblk->ppaf.pg_offset;
+ pblk->ppaf.blk_mask = ((1ULL << ppaf.blk_len) - 1) <<
+ pblk->ppaf.blk_offset;
+
+ pblk->ppaf_bitsize = pblk->ppaf.blk_offset + ppaf.blk_len;
+
+ return 0;
+}
+
+static int pblk_init_global_caches(struct pblk *pblk)
+{
+ char cache_name[PBLK_CACHE_NAME_LEN];
+
+ down_write(&pblk_lock);
+ pblk_blk_ws_cache = kmem_cache_create("pblk_blk_ws",
+ sizeof(struct pblk_line_ws), 0, 0, NULL);
+ if (!pblk_blk_ws_cache) {
+ up_write(&pblk_lock);
+ return -ENOMEM;
+ }
+
+ pblk_rec_cache = kmem_cache_create("pblk_rec",
+ sizeof(struct pblk_rec_ctx), 0, 0, NULL);
+ if (!pblk_rec_cache) {
+ kmem_cache_destroy(pblk_blk_ws_cache);
+ up_write(&pblk_lock);
+ return -ENOMEM;
+ }
+
+ pblk_r_rq_cache = kmem_cache_create("pblk_r_rq", pblk_r_rq_size,
+ 0, 0, NULL);
+ if (!pblk_r_rq_cache) {
+ kmem_cache_destroy(pblk_blk_ws_cache);
+ kmem_cache_destroy(pblk_rec_cache);
+ up_write(&pblk_lock);
+ return -ENOMEM;
+ }
+
+ pblk_w_rq_cache = kmem_cache_create("pblk_w_rq", pblk_w_rq_size,
+ 0, 0, NULL);
+ if (!pblk_w_rq_cache) {
+ kmem_cache_destroy(pblk_blk_ws_cache);
+ kmem_cache_destroy(pblk_rec_cache);
+ kmem_cache_destroy(pblk_r_rq_cache);
+ up_write(&pblk_lock);
+ return -ENOMEM;
+ }
+
+ snprintf(cache_name, sizeof(cache_name), "pblk_line_m_%s",
+ pblk->disk->disk_name);
+ pblk_line_meta_cache = kmem_cache_create(cache_name,
+ pblk->lm.sec_bitmap_len, 0, 0, NULL);
+ if (!pblk_line_meta_cache) {
+ kmem_cache_destroy(pblk_blk_ws_cache);
+ kmem_cache_destroy(pblk_rec_cache);
+ kmem_cache_destroy(pblk_r_rq_cache);
+ kmem_cache_destroy(pblk_w_rq_cache);
+ up_write(&pblk_lock);
+ return -ENOMEM;
+ }
+ up_write(&pblk_lock);
+
+ return 0;
+}
+
+static int pblk_core_init(struct pblk *pblk)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ int max_write_ppas;
+ int mod;
+
+ pblk->min_write_pgs = geo->sec_per_pl * (geo->sec_size / PAGE_SIZE);
+ max_write_ppas = pblk->min_write_pgs * geo->nr_luns;
+ pblk->max_write_pgs = (max_write_ppas < nvm_max_phys_sects(dev)) ?
+ max_write_ppas : nvm_max_phys_sects(dev);
+ pblk->pgs_in_buffer = NVM_MEM_PAGE_WRITE * geo->sec_per_pg *
+ geo->nr_planes * geo->nr_luns;
+
+ if (pblk->max_write_pgs > PBLK_MAX_REQ_ADDRS) {
+ pr_err("pblk: cannot support device max_phys_sect\n");
+ return -EINVAL;
+ }
+
+ div_u64_rem(geo->sec_per_blk, pblk->min_write_pgs, &mod);
+ if (mod) {
+ pr_err("pblk: bad configuration of sectors/pages\n");
+ return -EINVAL;
+ }
+
+ if (pblk_init_global_caches(pblk))
+ return -ENOMEM;
+
+ pblk->page_pool = mempool_create_page_pool(PAGE_POOL_SIZE, 0);
+ if (!pblk->page_pool)
+ return -ENOMEM;
+
+ pblk->line_ws_pool = mempool_create_slab_pool(geo->nr_luns,
+ pblk_blk_ws_cache);
+ if (!pblk->line_ws_pool)
+ goto free_page_pool;
+
+ pblk->rec_pool = mempool_create_slab_pool(geo->nr_luns, pblk_rec_cache);
+ if (!pblk->rec_pool)
+ goto free_blk_ws_pool;
+
+ pblk->r_rq_pool = mempool_create_slab_pool(64, pblk_r_rq_cache);
+ if (!pblk->r_rq_pool)
+ goto free_rec_pool;
+
+ pblk->w_rq_pool = mempool_create_slab_pool(64, pblk_w_rq_cache);
+ if (!pblk->w_rq_pool)
+ goto free_r_rq_pool;
+
+ pblk->line_meta_pool =
+ mempool_create_slab_pool(16, pblk_line_meta_cache);
+ if (!pblk->line_meta_pool)
+ goto free_w_rq_pool;
+
+ pblk->kw_wq = alloc_workqueue("pblk-aux-wq",
+ WQ_MEM_RECLAIM | WQ_UNBOUND, 1);
+ if (!pblk->kw_wq)
+ goto free_line_meta_pool;
+
+ if (pblk_set_ppaf(pblk))
+ goto free_kw_wq;
+
+ if (pblk_rwb_init(pblk))
+ goto free_kw_wq;
+
+ INIT_LIST_HEAD(&pblk->compl_list);
+ return 0;
+
+free_kw_wq:
+ destroy_workqueue(pblk->kw_wq);
+free_line_meta_pool:
+ mempool_destroy(pblk->line_meta_pool);
+free_w_rq_pool:
+ mempool_destroy(pblk->w_rq_pool);
+free_r_rq_pool:
+ mempool_destroy(pblk->r_rq_pool);
+free_rec_pool:
+ mempool_destroy(pblk->rec_pool);
+free_blk_ws_pool:
+ mempool_destroy(pblk->line_ws_pool);
+free_page_pool:
+ mempool_destroy(pblk->page_pool);
+ return -ENOMEM;
+}
+
+static void pblk_core_free(struct pblk *pblk)
+{
+ if (pblk->kw_wq)
+ destroy_workqueue(pblk->kw_wq);
+
+ mempool_destroy(pblk->page_pool);
+ mempool_destroy(pblk->line_ws_pool);
+ mempool_destroy(pblk->rec_pool);
+ mempool_destroy(pblk->r_rq_pool);
+ mempool_destroy(pblk->w_rq_pool);
+ mempool_destroy(pblk->line_meta_pool);
+
+ kmem_cache_destroy(pblk_blk_ws_cache);
+ kmem_cache_destroy(pblk_rec_cache);
+ kmem_cache_destroy(pblk_r_rq_cache);
+ kmem_cache_destroy(pblk_w_rq_cache);
+ kmem_cache_destroy(pblk_line_meta_cache);
+}
+
+static void pblk_luns_free(struct pblk *pblk)
+{
+ kfree(pblk->luns);
+}
+
+static void pblk_lines_free(struct pblk *pblk)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line *line;
+ int i;
+
+ spin_lock(&l_mg->free_lock);
+ for (i = 0; i < l_mg->nr_lines; i++) {
+ line = &pblk->lines[i];
+
+ pblk_line_free(pblk, line);
+ kfree(line->blk_bitmap);
+ kfree(line->erase_bitmap);
+ }
+ spin_unlock(&l_mg->free_lock);
+}
+
+static void pblk_line_meta_free(struct pblk *pblk)
+{
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ int i;
+
+ kfree(l_mg->bb_template);
+ kfree(l_mg->bb_aux);
+
+ for (i = 0; i < PBLK_DATA_LINES; i++) {
+ pblk_mfree(l_mg->sline_meta[i].meta, l_mg->smeta_alloc_type);
+ pblk_mfree(l_mg->eline_meta[i].meta, l_mg->emeta_alloc_type);
+ }
+
+ kfree(pblk->lines);
+}
+
+static int pblk_bb_discovery(struct nvm_tgt_dev *dev, struct pblk_lun *rlun)
+{
+ struct nvm_geo *geo = &dev->geo;
+ struct ppa_addr ppa;
+ u8 *blks;
+ int nr_blks, ret;
+
+ nr_blks = geo->blks_per_lun * geo->plane_mode;
+ blks = kmalloc(nr_blks, GFP_KERNEL);
+ if (!blks)
+ return -ENOMEM;
+
+ ppa.ppa = 0;
+ ppa.g.ch = rlun->bppa.g.ch;
+ ppa.g.lun = rlun->bppa.g.lun;
+
+ ret = nvm_get_tgt_bb_tbl(dev, ppa, blks);
+ if (ret)
+ goto out;
+
+ nr_blks = nvm_bb_tbl_fold(dev->parent, blks, nr_blks);
+ if (nr_blks < 0) {
+ ret = nr_blks;
+ goto out;
+ }
+
+ rlun->bb_list = blks;
+
+ return 0;
+out:
+ kfree(blks);
+ return ret;
+}
+
+static int pblk_bb_line(struct pblk *pblk, struct pblk_line *line)
+{
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_lun *rlun;
+ int bb_cnt = 0;
+ int i;
+
+ line->blk_bitmap = kzalloc(lm->blk_bitmap_len, GFP_KERNEL);
+ if (!line->blk_bitmap)
+ return -ENOMEM;
+
+ line->erase_bitmap = kzalloc(lm->blk_bitmap_len, GFP_KERNEL);
+ if (!line->erase_bitmap) {
+ kfree(line->blk_bitmap);
+ return -ENOMEM;
+ }
+
+ for (i = 0; i < lm->blk_per_line; i++) {
+ rlun = &pblk->luns[i];
+ if (rlun->bb_list[line->id] == NVM_BLK_T_FREE)
+ continue;
+
+ set_bit(i, line->blk_bitmap);
+ bb_cnt++;
+ }
+
+ return bb_cnt;
+}
+
+static int pblk_luns_init(struct pblk *pblk, struct ppa_addr *luns)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_lun *rlun;
+ int i, ret;
+
+ /* TODO: Implement unbalanced LUN support */
+ if (geo->luns_per_chnl < 0) {
+ pr_err("pblk: unbalanced LUN config.\n");
+ return -EINVAL;
+ }
+
+ pblk->luns = kcalloc(geo->nr_luns, sizeof(struct pblk_lun), GFP_KERNEL);
+ if (!pblk->luns)
+ return -ENOMEM;
+
+ for (i = 0; i < geo->nr_luns; i++) {
+ /* Stripe across channels */
+ int ch = i % geo->nr_chnls;
+ int lun_raw = i / geo->nr_chnls;
+ int lunid = lun_raw + ch * geo->luns_per_chnl;
+
+ rlun = &pblk->luns[i];
+ rlun->bppa = luns[lunid];
+
+ sema_init(&rlun->wr_sem, 1);
+
+ ret = pblk_bb_discovery(dev, rlun);
+ if (ret) {
+ while (--i >= 0)
+ kfree(pblk->luns[i].bb_list);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int pblk_lines_configure(struct pblk *pblk, int flags)
+{
+ struct pblk_line *line = NULL;
+ int ret = 0;
+
+ if (!(flags & NVM_TARGET_FACTORY)) {
+ line = pblk_recov_l2p(pblk);
+ if (IS_ERR(line)) {
+ pr_err("pblk: could not recover l2p table\n");
+ ret = -EFAULT;
+ }
+ }
+
+ if (!line) {
+ /* Configure next line for user data */
+ line = pblk_line_get_first_data(pblk);
+ if (!line) {
+ pr_err("pblk: line list corrupted\n");
+ ret = -EFAULT;
+ }
+ }
+
+ return ret;
+}
+
+/* See comment over struct line_emeta definition */
+static unsigned int calc_emeta_len(struct pblk *pblk, struct pblk_line_meta *lm)
+{
+ return (sizeof(struct line_emeta) +
+ ((lm->sec_per_line - lm->emeta_sec) * sizeof(u64)) +
+ (pblk->l_mg.nr_lines * sizeof(u32)) +
+ lm->blk_bitmap_len);
+}
+
+static void pblk_set_provision(struct pblk *pblk, long nr_free_blks)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ sector_t provisioned;
+
+ pblk->over_pct = 20;
+
+ provisioned = nr_free_blks;
+ provisioned *= (100 - pblk->over_pct);
+ sector_div(provisioned, 100);
+
+ /* Internally pblk manages all free blocks, but all calculations based
+ * on user capacity consider only provisioned blocks
+ */
+ pblk->rl.total_blocks = nr_free_blks;
+ pblk->rl.nr_secs = nr_free_blks * geo->sec_per_blk;
+ pblk->capacity = provisioned * geo->sec_per_blk;
+ atomic_set(&pblk->rl.free_blocks, nr_free_blks);
+}
+
+static int pblk_lines_init(struct pblk *pblk)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_line *line;
+ unsigned int smeta_len, emeta_len;
+ long nr_bad_blks, nr_meta_blks, nr_free_blks;
+ int bb_distance;
+ int i;
+ int ret;
+
+ lm->sec_per_line = geo->sec_per_blk * geo->nr_luns;
+ lm->blk_per_line = geo->nr_luns;
+ lm->blk_bitmap_len = BITS_TO_LONGS(geo->nr_luns) * sizeof(long);
+ lm->sec_bitmap_len = BITS_TO_LONGS(lm->sec_per_line) * sizeof(long);
+ lm->lun_bitmap_len = BITS_TO_LONGS(geo->nr_luns) * sizeof(long);
+ lm->high_thrs = lm->sec_per_line / 2;
+ lm->mid_thrs = lm->sec_per_line / 4;
+
+ /* Calculate necessary pages for smeta. See comment over struct
+ * line_smeta definition
+ */
+ lm->smeta_len = sizeof(struct line_smeta) +
+ PBLK_LINE_NR_LUN_BITMAP * lm->lun_bitmap_len;
+
+ i = 1;
+add_smeta_page:
+ lm->smeta_sec = i * geo->sec_per_pl;
+ lm->smeta_len = lm->smeta_sec * geo->sec_size;
+
+ smeta_len = sizeof(struct line_smeta) +
+ PBLK_LINE_NR_LUN_BITMAP * lm->lun_bitmap_len;
+ if (smeta_len > lm->smeta_len) {
+ i++;
+ goto add_smeta_page;
+ }
+
+ /* Calculate necessary pages for emeta. See comment over struct
+ * line_emeta definition
+ */
+ i = 1;
+add_emeta_page:
+ lm->emeta_sec = i * geo->sec_per_pl;
+ lm->emeta_len = lm->emeta_sec * geo->sec_size;
+
+ emeta_len = calc_emeta_len(pblk, lm);
+ if (emeta_len > lm->emeta_len) {
+ i++;
+ goto add_emeta_page;
+ }
+ lm->emeta_bb = geo->nr_luns - i;
+
+ nr_meta_blks = (lm->smeta_sec + lm->emeta_sec +
+ (geo->sec_per_blk / 2)) / geo->sec_per_blk;
+ lm->min_blk_line = nr_meta_blks + 1;
+
+ l_mg->nr_lines = geo->blks_per_lun;
+ l_mg->log_line = l_mg->data_line = NULL;
+ l_mg->l_seq_nr = l_mg->d_seq_nr = 0;
+ l_mg->nr_free_lines = 0;
+ bitmap_zero(&l_mg->meta_bitmap, PBLK_DATA_LINES);
+
+ /* smeta is always small enough to fit on a kmalloc memory allocation,
+ * emeta depends on the number of LUNs allocated to the pblk instance
+ */
+ l_mg->smeta_alloc_type = PBLK_KMALLOC_META;
+ for (i = 0; i < PBLK_DATA_LINES; i++) {
+ l_mg->sline_meta[i].meta = kmalloc(lm->smeta_len, GFP_KERNEL);
+ if (!l_mg->sline_meta[i].meta)
+ while (--i >= 0) {
+ kfree(l_mg->sline_meta[i].meta);
+ ret = -ENOMEM;
+ goto fail;
+ }
+ }
+
+ if (lm->emeta_len > KMALLOC_MAX_CACHE_SIZE) {
+ l_mg->emeta_alloc_type = PBLK_VMALLOC_META;
+
+ for (i = 0; i < PBLK_DATA_LINES; i++) {
+ l_mg->eline_meta[i].meta = vmalloc(lm->emeta_len);
+ if (!l_mg->eline_meta[i].meta)
+ while (--i >= 0) {
+ vfree(l_mg->eline_meta[i].meta);
+ ret = -ENOMEM;
+ goto fail;
+ }
+ }
+ } else {
+ l_mg->emeta_alloc_type = PBLK_KMALLOC_META;
+
+ for (i = 0; i < PBLK_DATA_LINES; i++) {
+ l_mg->eline_meta[i].meta =
+ kmalloc(lm->emeta_len, GFP_KERNEL);
+ if (!l_mg->eline_meta[i].meta)
+ while (--i >= 0) {
+ kfree(l_mg->eline_meta[i].meta);
+ ret = -ENOMEM;
+ goto fail;
+ }
+ }
+ }
+
+ l_mg->bb_template = kzalloc(lm->sec_bitmap_len, GFP_KERNEL);
+ if (!l_mg->bb_template) {
+ ret = -ENOMEM;
+ goto fail_free_meta;
+ }
+
+ l_mg->bb_aux = kzalloc(lm->sec_bitmap_len, GFP_KERNEL);
+ if (!l_mg->bb_aux) {
+ ret = -ENOMEM;
+ goto fail_free_bb_template;
+ }
+
+ bb_distance = (geo->nr_luns) * geo->sec_per_pl;
+ for (i = 0; i < lm->sec_per_line; i += bb_distance)
+ bitmap_set(l_mg->bb_template, i, geo->sec_per_pl);
+
+ INIT_LIST_HEAD(&l_mg->free_list);
+ INIT_LIST_HEAD(&l_mg->corrupt_list);
+ INIT_LIST_HEAD(&l_mg->bad_list);
+ INIT_LIST_HEAD(&l_mg->gc_full_list);
+ INIT_LIST_HEAD(&l_mg->gc_high_list);
+ INIT_LIST_HEAD(&l_mg->gc_mid_list);
+ INIT_LIST_HEAD(&l_mg->gc_low_list);
+ INIT_LIST_HEAD(&l_mg->gc_empty_list);
+
+ l_mg->gc_lists[0] = &l_mg->gc_high_list;
+ l_mg->gc_lists[1] = &l_mg->gc_mid_list;
+ l_mg->gc_lists[2] = &l_mg->gc_low_list;
+
+ spin_lock_init(&l_mg->free_lock);
+ spin_lock_init(&l_mg->gc_lock);
+
+ pblk->lines = kcalloc(l_mg->nr_lines, sizeof(struct pblk_line),
+ GFP_KERNEL);
+ if (!pblk->lines) {
+ ret = -ENOMEM;
+ goto fail_free_bb_aux;
+ }
+
+ nr_free_blks = 0;
+ for (i = 0; i < l_mg->nr_lines; i++) {
+ int blk_in_line;
+
+ line = &pblk->lines[i];
+
+ line->pblk = pblk;
+ line->id = i;
+ line->type = PBLK_LINETYPE_FREE;
+ line->state = PBLK_LINESTATE_FREE;
+ line->gc_group = PBLK_LINEGC_NONE;
+ spin_lock_init(&line->lock);
+
+ nr_bad_blks = pblk_bb_line(pblk, line);
+ if (nr_bad_blks < 0 || nr_bad_blks > lm->blk_per_line) {
+ ret = -EINVAL;
+ goto fail_free_lines;
+ }
+
+ blk_in_line = lm->blk_per_line - nr_bad_blks;
+ if (blk_in_line < lm->min_blk_line) {
+ line->state = PBLK_LINESTATE_BAD;
+ list_add_tail(&line->list, &l_mg->bad_list);
+ continue;
+ }
+
+ nr_free_blks += blk_in_line;
+ atomic_set(&line->blk_in_line, blk_in_line);
+
+ l_mg->nr_free_lines++;
+ list_add_tail(&line->list, &l_mg->free_list);
+ }
+
+ pblk_set_provision(pblk, nr_free_blks);
+
+ sema_init(&pblk->erase_sem, 1);
+
+ /* Cleanup per-LUN bad block lists - managed within lines on run-time */
+ for (i = 0; i < geo->nr_luns; i++)
+ kfree(pblk->luns[i].bb_list);
+
+ return 0;
+fail_free_lines:
+ kfree(pblk->lines);
+fail_free_bb_aux:
+ kfree(l_mg->bb_aux);
+fail_free_bb_template:
+ kfree(l_mg->bb_template);
+fail_free_meta:
+ for (i = 0; i < PBLK_DATA_LINES; i++) {
+ pblk_mfree(l_mg->sline_meta[i].meta, l_mg->smeta_alloc_type);
+ pblk_mfree(l_mg->eline_meta[i].meta, l_mg->emeta_alloc_type);
+ }
+fail:
+ for (i = 0; i < geo->nr_luns; i++)
+ kfree(pblk->luns[i].bb_list);
+
+ return ret;
+}
+
+static int pblk_writer_init(struct pblk *pblk)
+{
+ setup_timer(&pblk->wtimer, pblk_write_timer_fn, (unsigned long)pblk);
+ mod_timer(&pblk->wtimer, jiffies + msecs_to_jiffies(100));
+
+ pblk->writer_ts = kthread_create(pblk_write_ts, pblk, "pblk-writer-t");
+ if (IS_ERR(pblk->writer_ts)) {
+ pr_err("pblk: could not allocate writer kthread\n");
+ return PTR_ERR(pblk->writer_ts);
+ }
+
+ return 0;
+}
+
+static void pblk_writer_stop(struct pblk *pblk)
+{
+ if (pblk->writer_ts)
+ kthread_stop(pblk->writer_ts);
+ del_timer(&pblk->wtimer);
+}
+
+static void pblk_free(struct pblk *pblk)
+{
+ pblk_luns_free(pblk);
+ pblk_lines_free(pblk);
+ pblk_line_meta_free(pblk);
+ pblk_core_free(pblk);
+ pblk_l2p_free(pblk);
+
+ kfree(pblk);
+}
+
+static void pblk_tear_down(struct pblk *pblk)
+{
+ pblk_flush_writer(pblk);
+ pblk_writer_stop(pblk);
+ pblk_rb_sync_l2p(&pblk->rwb);
+ pblk_recov_pad(pblk);
+ pblk_rwb_free(pblk);
+ pblk_rl_free(&pblk->rl);
+
+ pr_debug("pblk: consistent tear down\n");
+}
+
+static void pblk_exit(void *private)
+{
+ struct pblk *pblk = private;
+
+ down_write(&pblk_lock);
+ pblk_gc_exit(pblk);
+ pblk_tear_down(pblk);
+ pblk_free(pblk);
+ up_write(&pblk_lock);
+}
+
+static sector_t pblk_capacity(void *private)
+{
+ struct pblk *pblk = private;
+
+ return pblk->capacity * NR_PHY_IN_LOG;
+}
+
+static void *pblk_init(struct nvm_tgt_dev *dev, struct gendisk *tdisk,
+ int flags)
+{
+ struct nvm_geo *geo = &dev->geo;
+ struct request_queue *bqueue = dev->q;
+ struct request_queue *tqueue = tdisk->queue;
+ struct pblk *pblk;
+ int ret;
+
+ if (dev->identity.dom & NVM_RSP_L2P) {
+ pr_err("pblk: device-side L2P table not supported. (%x)\n",
+ dev->identity.dom);
+ return ERR_PTR(-EINVAL);
+ }
+
+ pblk = kzalloc(sizeof(struct pblk), GFP_KERNEL);
+ if (!pblk)
+ return ERR_PTR(-ENOMEM);
+
+ pblk->dev = dev;
+ pblk->disk = tdisk;
+
+ spin_lock_init(&pblk->trans_lock);
+ spin_lock_init(&pblk->lock);
+
+ if (flags & NVM_TARGET_FACTORY)
+ pblk_setup_uuid(pblk);
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_set(&pblk->inflight_writes, 0);
+ atomic_long_set(&pblk->padded_writes, 0);
+ atomic_long_set(&pblk->padded_wb, 0);
+ atomic_long_set(&pblk->nr_flush, 0);
+ atomic_long_set(&pblk->req_writes, 0);
+ atomic_long_set(&pblk->sub_writes, 0);
+ atomic_long_set(&pblk->sync_writes, 0);
+ atomic_long_set(&pblk->compl_writes, 0);
+ atomic_long_set(&pblk->inflight_reads, 0);
+ atomic_long_set(&pblk->sync_reads, 0);
+ atomic_long_set(&pblk->recov_writes, 0);
+ atomic_long_set(&pblk->recov_writes, 0);
+ atomic_long_set(&pblk->recov_gc_writes, 0);
+#endif
+
+ atomic_long_set(&pblk->read_failed, 0);
+ atomic_long_set(&pblk->read_empty, 0);
+ atomic_long_set(&pblk->read_high_ecc, 0);
+ atomic_long_set(&pblk->read_failed_gc, 0);
+ atomic_long_set(&pblk->write_failed, 0);
+ atomic_long_set(&pblk->erase_failed, 0);
+
+ ret = pblk_luns_init(pblk, dev->luns);
+ if (ret) {
+ pr_err("pblk: could not initialize luns\n");
+ goto fail;
+ }
+
+ ret = pblk_lines_init(pblk);
+ if (ret) {
+ pr_err("pblk: could not initialize lines\n");
+ goto fail_free_luns;
+ }
+
+ ret = pblk_core_init(pblk);
+ if (ret) {
+ pr_err("pblk: could not initialize core\n");
+ goto fail_free_line_meta;
+ }
+
+ ret = pblk_l2p_init(pblk);
+ if (ret) {
+ pr_err("pblk: could not initialize maps\n");
+ goto fail_free_core;
+ }
+
+ ret = pblk_lines_configure(pblk, flags);
+ if (ret) {
+ pr_err("pblk: could not configure lines\n");
+ goto fail_free_l2p;
+ }
+
+ ret = pblk_writer_init(pblk);
+ if (ret) {
+ pr_err("pblk: could not initialize write thread\n");
+ goto fail_free_lines;
+ }
+
+ ret = pblk_gc_init(pblk);
+ if (ret) {
+ pr_err("pblk: could not initialize gc\n");
+ goto fail_stop_writer;
+ }
+
+ /* inherit the size from the underlying device */
+ blk_queue_logical_block_size(tqueue, queue_physical_block_size(bqueue));
+ blk_queue_max_hw_sectors(tqueue, queue_max_hw_sectors(bqueue));
+
+ blk_queue_write_cache(tqueue, true, false);
+
+ tqueue->limits.discard_granularity = geo->pgs_per_blk * geo->pfpg_size;
+ tqueue->limits.discard_alignment = 0;
+ blk_queue_max_discard_sectors(tqueue, UINT_MAX >> 9);
+ queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, tqueue);
+
+ pr_info("pblk init: luns:%u, lines:%d, secs:%llu, buf entries:%u\n",
+ geo->nr_luns, pblk->l_mg.nr_lines,
+ (unsigned long long)pblk->rl.nr_secs,
+ pblk->rwb.nr_entries);
+
+ wake_up_process(pblk->writer_ts);
+ return pblk;
+
+fail_stop_writer:
+ pblk_writer_stop(pblk);
+fail_free_lines:
+ pblk_lines_free(pblk);
+fail_free_l2p:
+ pblk_l2p_free(pblk);
+fail_free_core:
+ pblk_core_free(pblk);
+fail_free_line_meta:
+ pblk_line_meta_free(pblk);
+fail_free_luns:
+ pblk_luns_free(pblk);
+fail:
+ kfree(pblk);
+ return ERR_PTR(ret);
+}
+
+/* physical block device target */
+static struct nvm_tgt_type tt_pblk = {
+ .name = "pblk",
+ .version = {1, 0, 0},
+
+ .make_rq = pblk_make_rq,
+ .capacity = pblk_capacity,
+
+ .init = pblk_init,
+ .exit = pblk_exit,
+
+ .sysfs_init = pblk_sysfs_init,
+ .sysfs_exit = pblk_sysfs_exit,
+};
+
+static int __init pblk_module_init(void)
+{
+ return nvm_register_tgt_type(&tt_pblk);
+}
+
+static void pblk_module_exit(void)
+{
+ nvm_unregister_tgt_type(&tt_pblk);
+}
+
+module_init(pblk_module_init);
+module_exit(pblk_module_exit);
+MODULE_AUTHOR("Javier Gonzalez <javier@cnexlabs.com>");
+MODULE_AUTHOR("Matias Bjorling <matias@cnexlabs.com>");
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Physical Block-Device for Open-Channel SSDs");
diff --git a/drivers/lightnvm/pblk-map.c b/drivers/lightnvm/pblk-map.c
new file mode 100644
index 0000000..17c1695
--- /dev/null
+++ b/drivers/lightnvm/pblk-map.c
@@ -0,0 +1,136 @@
+/*
+ * Copyright (C) 2016 CNEX Labs
+ * Initial release: Javier Gonzalez <javier@cnexlabs.com>
+ * Matias Bjorling <matias@cnexlabs.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * pblk-map.c - pblk's lba-ppa mapping strategy
+ *
+ */
+
+#include "pblk.h"
+
+static void pblk_map_page_data(struct pblk *pblk, unsigned int sentry,
+ struct ppa_addr *ppa_list,
+ unsigned long *lun_bitmap,
+ struct pblk_sec_meta *meta_list,
+ unsigned int valid_secs)
+{
+ struct pblk_line *line = pblk_line_get_data(pblk);
+ struct line_emeta *emeta = line->emeta;
+ struct pblk_w_ctx *w_ctx;
+ __le64 *lba_list = pblk_line_emeta_to_lbas(emeta);
+ u64 paddr;
+ int nr_secs = pblk->min_write_pgs;
+ int i;
+
+ paddr = pblk_alloc_page(pblk, line, nr_secs);
+
+ for (i = 0; i < nr_secs; i++, paddr++) {
+ /* ppa to be sent to the device */
+ ppa_list[i] = addr_to_gen_ppa(pblk, paddr, line->id);
+
+ /* Write context for target bio completion on write buffer. Note
+ * that the write buffer is protected by the sync backpointer,
+ * and a single writer thread have access to each specific entry
+ * at a time. Thus, it is safe to modify the context for the
+ * entry we are setting up for submission without taking any
+ * lock or memory barrier.
+ */
+ if (i < valid_secs) {
+ kref_get(&line->ref);
+ w_ctx = pblk_rb_w_ctx(&pblk->rwb, sentry + i);
+ w_ctx->ppa = ppa_list[i];
+ meta_list[i].lba = cpu_to_le64(w_ctx->lba);
+ lba_list[paddr] = cpu_to_le64(w_ctx->lba);
+ le64_add_cpu(&line->emeta->nr_valid_lbas, 1);
+ } else {
+ meta_list[i].lba = cpu_to_le64(ADDR_EMPTY);
+ lba_list[paddr] = cpu_to_le64(ADDR_EMPTY);
+ pblk_map_pad_invalidate(pblk, line, paddr);
+ }
+ }
+
+ if (pblk_line_is_full(line)) {
+ line = pblk_line_replace_data(pblk);
+ if (!line)
+ return;
+ }
+
+ pblk_down_rq(pblk, ppa_list, nr_secs, lun_bitmap);
+}
+
+void pblk_map_rq(struct pblk *pblk, struct nvm_rq *rqd, unsigned int sentry,
+ unsigned long *lun_bitmap, unsigned int valid_secs,
+ unsigned int off)
+{
+ struct pblk_sec_meta *meta_list = rqd->meta_list;
+ unsigned int map_secs;
+ int min = pblk->min_write_pgs;
+ int i;
+
+ for (i = off; i < rqd->nr_ppas; i += min) {
+ map_secs = (i + min > valid_secs) ? (valid_secs % min) : min;
+ pblk_map_page_data(pblk, sentry + i, &rqd->ppa_list[i],
+ lun_bitmap, &meta_list[i], map_secs);
+ }
+}
+
+/* only if erase_ppa is set, acquire erase semaphore */
+void pblk_map_erase_rq(struct pblk *pblk, struct nvm_rq *rqd,
+ unsigned int sentry, unsigned long *lun_bitmap,
+ unsigned int valid_secs, struct ppa_addr *erase_ppa)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line *e_line = pblk_line_get_data_next(pblk);
+ struct pblk_sec_meta *meta_list = rqd->meta_list;
+ unsigned int map_secs;
+ int min = pblk->min_write_pgs;
+ int i, erase_lun;
+
+ for (i = 0; i < rqd->nr_ppas; i += min) {
+ map_secs = (i + min > valid_secs) ? (valid_secs % min) : min;
+ pblk_map_page_data(pblk, sentry + i, &rqd->ppa_list[i],
+ lun_bitmap, &meta_list[i], map_secs);
+
+ erase_lun = rqd->ppa_list[i].g.lun * geo->nr_chnls +
+ rqd->ppa_list[i].g.ch;
+
+ if (!test_bit(erase_lun, e_line->erase_bitmap)) {
+ if (down_trylock(&pblk->erase_sem))
+ continue;
+
+ set_bit(erase_lun, e_line->erase_bitmap);
+ atomic_dec(&e_line->left_eblks);
+ *erase_ppa = rqd->ppa_list[i];
+ erase_ppa->g.blk = e_line->id;
+
+ /* Avoid evaluating e_line->left_eblks */
+ return pblk_map_rq(pblk, rqd, sentry, lun_bitmap,
+ valid_secs, i + min);
+ }
+ }
+
+ /* Erase blocks that are bad in this line but might not be in next */
+ if (unlikely(ppa_empty(*erase_ppa))) {
+ struct pblk_line_meta *lm = &pblk->lm;
+
+ i = find_first_zero_bit(e_line->erase_bitmap, lm->blk_per_line);
+ if (i == lm->blk_per_line)
+ return;
+
+ set_bit(i, e_line->erase_bitmap);
+ atomic_dec(&e_line->left_eblks);
+ *erase_ppa = pblk->luns[i].bppa; /* set ch and lun */
+ erase_ppa->g.blk = e_line->id;
+ }
+}
diff --git a/drivers/lightnvm/pblk-rb.c b/drivers/lightnvm/pblk-rb.c
new file mode 100644
index 0000000..045384d
--- /dev/null
+++ b/drivers/lightnvm/pblk-rb.c
@@ -0,0 +1,852 @@
+/*
+ * Copyright (C) 2016 CNEX Labs
+ * Initial release: Javier Gonzalez <javier@cnexlabs.com>
+ *
+ * Based upon the circular ringbuffer.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * pblk-rb.c - pblk's write buffer
+ */
+
+#include <linux/circ_buf.h>
+
+#include "pblk.h"
+
+static DECLARE_RWSEM(pblk_rb_lock);
+
+void pblk_rb_data_free(struct pblk_rb *rb)
+{
+ struct pblk_rb_pages *p, *t;
+
+ down_write(&pblk_rb_lock);
+ list_for_each_entry_safe(p, t, &rb->pages, list) {
+ free_pages((unsigned long)page_address(p->pages), p->order);
+ list_del(&p->list);
+ kfree(p);
+ }
+ up_write(&pblk_rb_lock);
+}
+
+/*
+ * Initialize ring buffer. The data and metadata buffers must be previously
+ * allocated and their size must be a power of two
+ * (Documentation/circular-buffers.txt)
+ */
+int pblk_rb_init(struct pblk_rb *rb, struct pblk_rb_entry *rb_entry_base,
+ unsigned int power_size, unsigned int power_seg_sz)
+{
+ struct pblk *pblk = container_of(rb, struct pblk, rwb);
+ unsigned int init_entry = 0;
+ unsigned int alloc_order = power_size;
+ unsigned int max_order = MAX_ORDER - 1;
+ unsigned int order, iter;
+
+ down_write(&pblk_rb_lock);
+ rb->entries = rb_entry_base;
+ rb->seg_size = (1 << power_seg_sz);
+ rb->nr_entries = (1 << power_size);
+ rb->mem = rb->subm = rb->sync = rb->l2p_update = 0;
+ rb->sync_point = EMPTY_ENTRY;
+
+ spin_lock_init(&rb->w_lock);
+ spin_lock_init(&rb->s_lock);
+
+ INIT_LIST_HEAD(&rb->pages);
+
+ if (alloc_order >= max_order) {
+ order = max_order;
+ iter = (1 << (alloc_order - max_order));
+ } else {
+ order = alloc_order;
+ iter = 1;
+ }
+
+ do {
+ struct pblk_rb_entry *entry;
+ struct pblk_rb_pages *page_set;
+ void *kaddr;
+ unsigned long set_size;
+ int i;
+
+ page_set = kmalloc(sizeof(struct pblk_rb_pages), GFP_KERNEL);
+ if (!page_set) {
+ up_write(&pblk_rb_lock);
+ return -ENOMEM;
+ }
+
+ page_set->order = order;
+ page_set->pages = alloc_pages(GFP_KERNEL, order);
+ if (!page_set->pages) {
+ kfree(page_set);
+ pblk_rb_data_free(rb);
+ up_write(&pblk_rb_lock);
+ return -ENOMEM;
+ }
+ kaddr = page_address(page_set->pages);
+
+ entry = &rb->entries[init_entry];
+ entry->data = kaddr;
+ entry->cacheline = pblk_cacheline_to_addr(init_entry++);
+ entry->w_ctx.flags = PBLK_WRITABLE_ENTRY;
+
+ set_size = (1 << order);
+ for (i = 1; i < set_size; i++) {
+ entry = &rb->entries[init_entry];
+ entry->cacheline = pblk_cacheline_to_addr(init_entry++);
+ entry->data = kaddr + (i * rb->seg_size);
+ entry->w_ctx.flags = PBLK_WRITABLE_ENTRY;
+ bio_list_init(&entry->w_ctx.bios);
+ }
+
+ list_add_tail(&page_set->list, &rb->pages);
+ iter--;
+ } while (iter > 0);
+ up_write(&pblk_rb_lock);
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_set(&rb->inflight_sync_point, 0);
+#endif
+
+ /*
+ * Initialize rate-limiter, which controls access to the write buffer
+ * but user and GC I/O
+ */
+ pblk_rl_init(&pblk->rl, rb->nr_entries);
+
+ return 0;
+}
+
+/*
+ * pblk_rb_calculate_size -- calculate the size of the write buffer
+ */
+unsigned int pblk_rb_calculate_size(unsigned int nr_entries)
+{
+ /* Alloc a write buffer that can at least fit 128 entries */
+ return (1 << max(get_count_order(nr_entries), 7));
+}
+
+void *pblk_rb_entries_ref(struct pblk_rb *rb)
+{
+ return rb->entries;
+}
+
+static void clean_wctx(struct pblk_w_ctx *w_ctx)
+{
+ int flags;
+
+try:
+ flags = READ_ONCE(w_ctx->flags);
+ if (!(flags & PBLK_SUBMITTED_ENTRY))
+ goto try;
+
+ /* Release flags on context. Protect from writes and reads */
+ smp_store_release(&w_ctx->flags, PBLK_WRITABLE_ENTRY);
+ pblk_ppa_set_empty(&w_ctx->ppa);
+}
+
+#define pblk_rb_ring_count(head, tail, size) CIRC_CNT(head, tail, size)
+#define pblk_rb_ring_space(rb, head, tail, size) \
+ (CIRC_SPACE(head, tail, size))
+
+/*
+ * Buffer space is calculated with respect to the back pointer signaling
+ * synchronized entries to the media.
+ */
+static unsigned int pblk_rb_space(struct pblk_rb *rb)
+{
+ unsigned int mem = READ_ONCE(rb->mem);
+ unsigned int sync = READ_ONCE(rb->sync);
+
+ return pblk_rb_ring_space(rb, mem, sync, rb->nr_entries);
+}
+
+/*
+ * Buffer count is calculated with respect to the submission entry signaling the
+ * entries that are available to send to the media
+ */
+unsigned int pblk_rb_read_count(struct pblk_rb *rb)
+{
+ unsigned int mem = READ_ONCE(rb->mem);
+ unsigned int subm = READ_ONCE(rb->subm);
+
+ return pblk_rb_ring_count(mem, subm, rb->nr_entries);
+}
+
+unsigned int pblk_rb_read_commit(struct pblk_rb *rb, unsigned int nr_entries)
+{
+ unsigned int subm;
+
+ subm = READ_ONCE(rb->subm);
+ /* Commit read means updating submission pointer */
+ smp_store_release(&rb->subm,
+ (subm + nr_entries) & (rb->nr_entries - 1));
+
+ return subm;
+}
+
+static int __pblk_rb_update_l2p(struct pblk_rb *rb, unsigned int *l2p_upd,
+ unsigned int to_update)
+{
+ struct pblk *pblk = container_of(rb, struct pblk, rwb);
+ struct pblk_line *line;
+ struct pblk_rb_entry *entry;
+ struct pblk_w_ctx *w_ctx;
+ unsigned int i;
+
+ for (i = 0; i < to_update; i++) {
+ entry = &rb->entries[*l2p_upd];
+ w_ctx = &entry->w_ctx;
+
+ pblk_update_map_dev(pblk, w_ctx->lba, w_ctx->ppa,
+ entry->cacheline);
+
+ line = &pblk->lines[pblk_tgt_ppa_to_line(w_ctx->ppa)];
+ kref_put(&line->ref, pblk_line_put);
+ clean_wctx(w_ctx);
+ *l2p_upd = (*l2p_upd + 1) & (rb->nr_entries - 1);
+ }
+
+ return 0;
+}
+
+/*
+ * When we move the l2p_update pointer, we update the l2p table - lookups will
+ * point to the physical address instead of to the cacheline in the write buffer
+ * from this moment on.
+ */
+static int pblk_rb_update_l2p(struct pblk_rb *rb, unsigned int nr_entries,
+ unsigned int mem, unsigned int sync)
+{
+ unsigned int space, count;
+ int ret = 0;
+
+ lockdep_assert_held(&rb->w_lock);
+
+ /* Update l2p only as buffer entries are being overwritten */
+ space = pblk_rb_ring_space(rb, mem, rb->l2p_update, rb->nr_entries);
+ if (space > nr_entries)
+ goto out;
+
+ count = nr_entries - space;
+ /* l2p_update used exclusively under rb->w_lock */
+ ret = __pblk_rb_update_l2p(rb, &rb->l2p_update, count);
+
+out:
+ return ret;
+}
+
+/*
+ * Update the l2p entry for all sectors stored on the write buffer. This means
+ * that all future lookups to the l2p table will point to a device address, not
+ * to the cacheline in the write buffer.
+ */
+void pblk_rb_sync_l2p(struct pblk_rb *rb)
+{
+ unsigned int sync;
+ unsigned int to_update;
+
+ spin_lock(&rb->w_lock);
+
+ /* Protect from reads and writes */
+ sync = smp_load_acquire(&rb->sync);
+
+ to_update = pblk_rb_ring_count(sync, rb->l2p_update, rb->nr_entries);
+ __pblk_rb_update_l2p(rb, &rb->l2p_update, to_update);
+
+ spin_unlock(&rb->w_lock);
+}
+
+/*
+ * Write @nr_entries to ring buffer from @data buffer if there is enough space.
+ * Typically, 4KB data chunks coming from a bio will be copied to the ring
+ * buffer, thus the write will fail if not all incoming data can be copied.
+ *
+ */
+static void __pblk_rb_write_entry(struct pblk_rb *rb, void *data,
+ struct pblk_w_ctx w_ctx,
+ struct pblk_rb_entry *entry)
+{
+ memcpy(entry->data, data, rb->seg_size);
+
+ entry->w_ctx.lba = w_ctx.lba;
+ entry->w_ctx.ppa = w_ctx.ppa;
+}
+
+void pblk_rb_write_entry_user(struct pblk_rb *rb, void *data,
+ struct pblk_w_ctx w_ctx, unsigned int ring_pos)
+{
+ struct pblk *pblk = container_of(rb, struct pblk, rwb);
+ struct pblk_rb_entry *entry;
+ int flags;
+
+ entry = &rb->entries[ring_pos];
+ flags = READ_ONCE(entry->w_ctx.flags);
+#ifdef CONFIG_NVM_DEBUG
+ /* Caller must guarantee that the entry is free */
+ BUG_ON(!(flags & PBLK_WRITABLE_ENTRY));
+#endif
+
+ __pblk_rb_write_entry(rb, data, w_ctx, entry);
+
+ pblk_update_map_cache(pblk, w_ctx.lba, entry->cacheline);
+ flags = w_ctx.flags | PBLK_WRITTEN_DATA;
+
+ /* Release flags on write context. Protect from writes */
+ smp_store_release(&entry->w_ctx.flags, flags);
+}
+
+void pblk_rb_write_entry_gc(struct pblk_rb *rb, void *data,
+ struct pblk_w_ctx w_ctx, struct pblk_line *gc_line,
+ unsigned int ring_pos)
+{
+ struct pblk *pblk = container_of(rb, struct pblk, rwb);
+ struct pblk_rb_entry *entry;
+ int flags;
+
+ entry = &rb->entries[ring_pos];
+ flags = READ_ONCE(entry->w_ctx.flags);
+#ifdef CONFIG_NVM_DEBUG
+ /* Caller must guarantee that the entry is free */
+ BUG_ON(!(flags & PBLK_WRITABLE_ENTRY));
+#endif
+
+ __pblk_rb_write_entry(rb, data, w_ctx, entry);
+
+ if (!pblk_update_map_gc(pblk, w_ctx.lba, entry->cacheline, gc_line))
+ entry->w_ctx.lba = ADDR_EMPTY;
+
+ flags = w_ctx.flags | PBLK_WRITTEN_DATA;
+
+ /* Release flags on write context. Protect from writes */
+ smp_store_release(&entry->w_ctx.flags, flags);
+}
+
+static int pblk_rb_sync_point_set(struct pblk_rb *rb, struct bio *bio,
+ unsigned int pos)
+{
+ struct pblk_rb_entry *entry;
+ unsigned int subm, sync_point;
+ int flags;
+
+ subm = READ_ONCE(rb->subm);
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_inc(&rb->inflight_sync_point);
+#endif
+
+ if (pos == subm)
+ return 0;
+
+ sync_point = (pos == 0) ? (rb->nr_entries - 1) : (pos - 1);
+ entry = &rb->entries[sync_point];
+
+ flags = READ_ONCE(entry->w_ctx.flags);
+ flags |= PBLK_FLUSH_ENTRY;
+
+ /* Release flags on context. Protect from writes */
+ smp_store_release(&entry->w_ctx.flags, flags);
+
+ /* Protect syncs */
+ smp_store_release(&rb->sync_point, sync_point);
+
+ spin_lock_irq(&rb->s_lock);
+ bio_list_add(&entry->w_ctx.bios, bio);
+ spin_unlock_irq(&rb->s_lock);
+
+ return 1;
+}
+
+static int __pblk_rb_may_write(struct pblk_rb *rb, unsigned int nr_entries,
+ unsigned int *pos)
+{
+ unsigned int mem;
+ unsigned int sync;
+
+ sync = READ_ONCE(rb->sync);
+ mem = READ_ONCE(rb->mem);
+
+ if (pblk_rb_ring_space(rb, mem, sync, rb->nr_entries) < nr_entries)
+ return 0;
+
+ if (pblk_rb_update_l2p(rb, nr_entries, mem, sync))
+ return 0;
+
+ *pos = mem;
+
+ return 1;
+}
+
+static int pblk_rb_may_write(struct pblk_rb *rb, unsigned int nr_entries,
+ unsigned int *pos)
+{
+ if (!__pblk_rb_may_write(rb, nr_entries, pos))
+ return 0;
+
+ /* Protect from read count */
+ smp_store_release(&rb->mem, (*pos + nr_entries) & (rb->nr_entries - 1));
+ return 1;
+}
+
+static int pblk_rb_may_write_flush(struct pblk_rb *rb, unsigned int nr_entries,
+ unsigned int *pos, struct bio *bio,
+ int *io_ret)
+{
+ unsigned int mem;
+
+ if (!__pblk_rb_may_write(rb, nr_entries, pos))
+ return 0;
+
+ mem = (*pos + nr_entries) & (rb->nr_entries - 1);
+ *io_ret = NVM_IO_DONE;
+
+ if (bio->bi_opf & REQ_PREFLUSH) {
+ struct pblk *pblk = container_of(rb, struct pblk, rwb);
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_inc(&pblk->nr_flush);
+#endif
+ if (pblk_rb_sync_point_set(&pblk->rwb, bio, mem))
+ *io_ret = NVM_IO_OK;
+ }
+
+ /* Protect from read count */
+ smp_store_release(&rb->mem, mem);
+ return 1;
+}
+
+/*
+ * Atomically check that (i) there is space on the write buffer for the
+ * incoming I/O, and (ii) the current I/O type has enough budget in the write
+ * buffer (rate-limiter).
+ */
+int pblk_rb_may_write_user(struct pblk_rb *rb, struct bio *bio,
+ unsigned int nr_entries, unsigned int *pos)
+{
+ struct pblk *pblk = container_of(rb, struct pblk, rwb);
+ int flush_done;
+
+ spin_lock(&rb->w_lock);
+ if (!pblk_rl_user_may_insert(&pblk->rl, nr_entries)) {
+ spin_unlock(&rb->w_lock);
+ return NVM_IO_REQUEUE;
+ }
+
+ if (!pblk_rb_may_write_flush(rb, nr_entries, pos, bio, &flush_done)) {
+ spin_unlock(&rb->w_lock);
+ return NVM_IO_REQUEUE;
+ }
+
+ pblk_rl_user_in(&pblk->rl, nr_entries);
+ spin_unlock(&rb->w_lock);
+
+ return flush_done;
+}
+
+/*
+ * Look at pblk_rb_may_write_user comment
+ */
+int pblk_rb_may_write_gc(struct pblk_rb *rb, unsigned int nr_entries,
+ unsigned int *pos)
+{
+ struct pblk *pblk = container_of(rb, struct pblk, rwb);
+
+ spin_lock(&rb->w_lock);
+ if (!pblk_rl_gc_may_insert(&pblk->rl, nr_entries)) {
+ spin_unlock(&rb->w_lock);
+ return 0;
+ }
+
+ if (!pblk_rb_may_write(rb, nr_entries, pos)) {
+ spin_unlock(&rb->w_lock);
+ return 0;
+ }
+
+ pblk_rl_gc_in(&pblk->rl, nr_entries);
+ spin_unlock(&rb->w_lock);
+
+ return 1;
+}
+
+/*
+ * The caller of this function must ensure that the backpointer will not
+ * overwrite the entries passed on the list.
+ */
+unsigned int pblk_rb_read_to_bio_list(struct pblk_rb *rb, struct bio *bio,
+ struct list_head *list,
+ unsigned int max)
+{
+ struct pblk_rb_entry *entry, *tentry;
+ struct page *page;
+ unsigned int read = 0;
+ int ret;
+
+ list_for_each_entry_safe(entry, tentry, list, index) {
+ if (read > max) {
+ pr_err("pblk: too many entries on list\n");
+ goto out;
+ }
+
+ page = virt_to_page(entry->data);
+ if (!page) {
+ pr_err("pblk: could not allocate write bio page\n");
+ goto out;
+ }
+
+ ret = bio_add_page(bio, page, rb->seg_size, 0);
+ if (ret != rb->seg_size) {
+ pr_err("pblk: could not add page to write bio\n");
+ goto out;
+ }
+
+ list_del(&entry->index);
+ read++;
+ }
+
+out:
+ return read;
+}
+
+/*
+ * Read available entries on rb and add them to the given bio. To avoid a memory
+ * copy, a page reference to the write buffer is used to be added to the bio.
+ *
+ * This function is used by the write thread to form the write bio that will
+ * persist data on the write buffer to the media.
+ */
+unsigned int pblk_rb_read_to_bio(struct pblk_rb *rb, struct bio *bio,
+ struct pblk_c_ctx *c_ctx,
+ unsigned int pos,
+ unsigned int nr_entries,
+ unsigned int count)
+{
+ struct pblk *pblk = container_of(rb, struct pblk, rwb);
+ struct pblk_rb_entry *entry;
+ struct page *page;
+ unsigned int pad = 0, read = 0, to_read = nr_entries;
+ unsigned int user_io = 0, gc_io = 0;
+ unsigned int i;
+ int flags;
+ int ret;
+
+ if (count < nr_entries) {
+ pad = nr_entries - count;
+ to_read = count;
+ }
+
+ c_ctx->sentry = pos;
+ c_ctx->nr_valid = to_read;
+ c_ctx->nr_padded = pad;
+
+ for (i = 0; i < to_read; i++) {
+ entry = &rb->entries[pos];
+
+ /* A write has been allowed into the buffer, but data is still
+ * being copied to it. It is ok to busy wait.
+ */
+try:
+ flags = READ_ONCE(entry->w_ctx.flags);
+ if (!(flags & PBLK_WRITTEN_DATA))
+ goto try;
+
+ if (flags & PBLK_IOTYPE_USER)
+ user_io++;
+ else if (flags & PBLK_IOTYPE_GC)
+ gc_io++;
+ else
+ WARN(1, "pblk: unknown IO type\n");
+
+ page = virt_to_page(entry->data);
+ if (!page) {
+ pr_err("pblk: could not allocate write bio page\n");
+ flags &= ~PBLK_WRITTEN_DATA;
+ flags |= PBLK_SUBMITTED_ENTRY;
+ /* Release flags on context. Protect from writes */
+ smp_store_release(&entry->w_ctx.flags, flags);
+ goto out;
+ }
+
+ ret = bio_add_page(bio, page, rb->seg_size, 0);
+ if (ret != rb->seg_size) {
+ pr_err("pblk: could not add page to write bio\n");
+ flags &= ~PBLK_WRITTEN_DATA;
+ flags |= PBLK_SUBMITTED_ENTRY;
+ /* Release flags on context. Protect from writes */
+ smp_store_release(&entry->w_ctx.flags, flags);
+ goto out;
+ }
+
+ if (flags & PBLK_FLUSH_ENTRY) {
+ unsigned int sync_point;
+
+ sync_point = READ_ONCE(rb->sync_point);
+ if (sync_point == pos) {
+ /* Protect syncs */
+ smp_store_release(&rb->sync_point, EMPTY_ENTRY);
+ }
+
+ flags &= ~PBLK_FLUSH_ENTRY;
+#ifdef CONFIG_NVM_DEBUG
+ atomic_dec(&rb->inflight_sync_point);
+#endif
+ }
+
+ flags &= ~PBLK_WRITTEN_DATA;
+ flags |= PBLK_SUBMITTED_ENTRY;
+
+ /* Release flags on context. Protect from writes */
+ smp_store_release(&entry->w_ctx.flags, flags);
+
+ pos = (pos + 1) & (rb->nr_entries - 1);
+ }
+
+ read = to_read;
+ pblk_rl_out(&pblk->rl, user_io, gc_io);
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_add(pad, &((struct pblk *)
+ (container_of(rb, struct pblk, rwb)))->padded_writes);
+#endif
+out:
+ return read;
+}
+
+/*
+ * Copy to bio only if the lba matches the one on the given cache entry.
+ * Otherwise, it means that the entry has been overwritten, and the bio should
+ * be directed to disk.
+ */
+int pblk_rb_copy_to_bio(struct pblk_rb *rb, struct bio *bio, sector_t lba,
+ u64 pos, int bio_iter)
+{
+ struct pblk_rb_entry *entry;
+ struct pblk_w_ctx *w_ctx;
+ void *data;
+ int flags;
+ int ret = 1;
+
+ spin_lock(&rb->w_lock);
+
+#ifdef CONFIG_NVM_DEBUG
+ /* Caller must ensure that the access will not cause an overflow */
+ BUG_ON(pos >= rb->nr_entries);
+#endif
+ entry = &rb->entries[pos];
+ w_ctx = &entry->w_ctx;
+ flags = READ_ONCE(w_ctx->flags);
+
+ /* Check if the entry has been overwritten or is scheduled to be */
+ if (w_ctx->lba != lba || flags & PBLK_WRITABLE_ENTRY) {
+ ret = 0;
+ goto out;
+ }
+
+ /* Only advance the bio if it hasn't been advanced already. If advanced,
+ * this bio is at least a partial bio (i.e., it has partially been
+ * filled with data from the cache). If part of the data resides on the
+ * media, we will read later on
+ */
+ if (unlikely(!bio->bi_iter.bi_idx))
+ bio_advance(bio, bio_iter * PBLK_EXPOSED_PAGE_SIZE);
+
+ data = bio_data(bio);
+ memcpy(data, entry->data, rb->seg_size);
+
+out:
+ spin_unlock(&rb->w_lock);
+ return ret;
+}
+
+struct pblk_w_ctx *pblk_rb_w_ctx(struct pblk_rb *rb, unsigned int pos)
+{
+ unsigned int entry = pos & (rb->nr_entries - 1);
+
+ return &rb->entries[entry].w_ctx;
+}
+
+unsigned int pblk_rb_sync_init(struct pblk_rb *rb, unsigned long *flags)
+ __acquires(&rb->s_lock)
+{
+ if (flags)
+ spin_lock_irqsave(&rb->s_lock, *flags);
+ else
+ spin_lock_irq(&rb->s_lock);
+
+ return rb->sync;
+}
+
+void pblk_rb_sync_end(struct pblk_rb *rb, unsigned long *flags)
+ __releases(&rb->s_lock)
+{
+ lockdep_assert_held(&rb->s_lock);
+
+ if (flags)
+ spin_unlock_irqrestore(&rb->s_lock, *flags);
+ else
+ spin_unlock_irq(&rb->s_lock);
+}
+
+unsigned int pblk_rb_sync_advance(struct pblk_rb *rb, unsigned int nr_entries)
+{
+ unsigned int sync;
+ unsigned int i;
+
+ lockdep_assert_held(&rb->s_lock);
+
+ sync = READ_ONCE(rb->sync);
+
+ for (i = 0; i < nr_entries; i++)
+ sync = (sync + 1) & (rb->nr_entries - 1);
+
+ /* Protect from counts */
+ smp_store_release(&rb->sync, sync);
+
+ return sync;
+}
+
+unsigned int pblk_rb_sync_point_count(struct pblk_rb *rb)
+{
+ unsigned int subm, sync_point;
+ unsigned int count;
+
+ /* Protect syncs */
+ sync_point = smp_load_acquire(&rb->sync_point);
+ if (sync_point == EMPTY_ENTRY)
+ return 0;
+
+ subm = READ_ONCE(rb->subm);
+
+ /* The sync point itself counts as a sector to sync */
+ count = pblk_rb_ring_count(sync_point, subm, rb->nr_entries) + 1;
+
+ return count;
+}
+
+/*
+ * Scan from the current position of the sync pointer to find the entry that
+ * corresponds to the given ppa. This is necessary since write requests can be
+ * completed out of order. The assumption is that the ppa is close to the sync
+ * pointer thus the search will not take long.
+ *
+ * The caller of this function must guarantee that the sync pointer will no
+ * reach the entry while it is using the metadata associated with it. With this
+ * assumption in mind, there is no need to take the sync lock.
+ */
+struct pblk_rb_entry *pblk_rb_sync_scan_entry(struct pblk_rb *rb,
+ struct ppa_addr *ppa)
+{
+ unsigned int sync, subm, count;
+ unsigned int i;
+
+ sync = READ_ONCE(rb->sync);
+ subm = READ_ONCE(rb->subm);
+ count = pblk_rb_ring_count(subm, sync, rb->nr_entries);
+
+ for (i = 0; i < count; i++)
+ sync = (sync + 1) & (rb->nr_entries - 1);
+
+ return NULL;
+}
+
+int pblk_rb_tear_down_check(struct pblk_rb *rb)
+{
+ struct pblk_rb_entry *entry;
+ int i;
+ int ret = 0;
+
+ spin_lock(&rb->w_lock);
+ spin_lock_irq(&rb->s_lock);
+
+ if ((rb->mem == rb->subm) && (rb->subm == rb->sync) &&
+ (rb->sync == rb->l2p_update) &&
+ (rb->sync_point == EMPTY_ENTRY)) {
+ goto out;
+ }
+
+ if (!rb->entries) {
+ ret = 1;
+ goto out;
+ }
+
+ for (i = 0; i < rb->nr_entries; i++) {
+ entry = &rb->entries[i];
+
+ if (!entry->data) {
+ ret = 1;
+ goto out;
+ }
+ }
+
+out:
+ spin_unlock(&rb->w_lock);
+ spin_unlock_irq(&rb->s_lock);
+
+ return ret;
+}
+
+unsigned int pblk_rb_wrap_pos(struct pblk_rb *rb, unsigned int pos)
+{
+ return (pos & (rb->nr_entries - 1));
+}
+
+int pblk_rb_pos_oob(struct pblk_rb *rb, u64 pos)
+{
+ return (pos >= rb->nr_entries);
+}
+
+ssize_t pblk_rb_sysfs(struct pblk_rb *rb, char *buf)
+{
+ struct pblk *pblk = container_of(rb, struct pblk, rwb);
+ struct pblk_c_ctx *c;
+ ssize_t offset;
+ int queued_entries = 0;
+
+ spin_lock_irq(&rb->s_lock);
+ list_for_each_entry(c, &pblk->compl_list, list)
+ queued_entries++;
+ spin_unlock_irq(&rb->s_lock);
+
+ if (rb->sync_point != EMPTY_ENTRY)
+ offset = scnprintf(buf, PAGE_SIZE,
+ "%u\t%u\t%u\t%u\t%u\t%u\t%u - %u/%u/%u - %d\n",
+ rb->nr_entries,
+ rb->mem,
+ rb->subm,
+ rb->sync,
+ rb->l2p_update,
+#ifdef CONFIG_NVM_DEBUG
+ atomic_read(&rb->inflight_sync_point),
+#else
+ 0,
+#endif
+ rb->sync_point,
+ pblk_rb_read_count(rb),
+ pblk_rb_space(rb),
+ pblk_rb_sync_point_count(rb),
+ queued_entries);
+ else
+ offset = scnprintf(buf, PAGE_SIZE,
+ "%u\t%u\t%u\t%u\t%u\t%u\tNULL - %u/%u/%u - %d\n",
+ rb->nr_entries,
+ rb->mem,
+ rb->subm,
+ rb->sync,
+ rb->l2p_update,
+#ifdef CONFIG_NVM_DEBUG
+ atomic_read(&rb->inflight_sync_point),
+#else
+ 0,
+#endif
+ pblk_rb_read_count(rb),
+ pblk_rb_space(rb),
+ pblk_rb_sync_point_count(rb),
+ queued_entries);
+
+ return offset;
+}
diff --git a/drivers/lightnvm/pblk-read.c b/drivers/lightnvm/pblk-read.c
new file mode 100644
index 0000000..4a12f14
--- /dev/null
+++ b/drivers/lightnvm/pblk-read.c
@@ -0,0 +1,529 @@
+/*
+ * Copyright (C) 2016 CNEX Labs
+ * Initial release: Javier Gonzalez <javier@cnexlabs.com>
+ * Matias Bjorling <matias@cnexlabs.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * pblk-read.c - pblk's read path
+ */
+
+#include "pblk.h"
+
+/*
+ * There is no guarantee that the value read from cache has not been updated and
+ * resides at another location in the cache. We guarantee though that if the
+ * value is read from the cache, it belongs to the mapped lba. In order to
+ * guarantee and order between writes and reads are ordered, a flush must be
+ * issued.
+ */
+static int pblk_read_from_cache(struct pblk *pblk, struct bio *bio,
+ sector_t lba, struct ppa_addr ppa,
+ int bio_iter)
+{
+#ifdef CONFIG_NVM_DEBUG
+ /* Callers must ensure that the ppa points to a cache address */
+ BUG_ON(pblk_ppa_empty(ppa));
+ BUG_ON(!pblk_addr_in_cache(ppa));
+#endif
+
+ return pblk_rb_copy_to_bio(&pblk->rwb, bio, lba,
+ pblk_addr_to_cacheline(ppa), bio_iter);
+}
+
+static void pblk_read_ppalist_rq(struct pblk *pblk, struct nvm_rq *rqd,
+ unsigned long *read_bitmap)
+{
+ struct bio *bio = rqd->bio;
+ struct ppa_addr ppas[PBLK_MAX_REQ_ADDRS];
+ sector_t blba = pblk_get_lba(bio);
+ int nr_secs = rqd->nr_ppas;
+ int advanced_bio = 0;
+ int i, j = 0;
+
+ /* logic error: lba out-of-bounds. Ignore read request */
+ if (blba + nr_secs >= pblk->rl.nr_secs) {
+ WARN(1, "pblk: read lbas out of bounds\n");
+ return;
+ }
+
+ pblk_lookup_l2p_seq(pblk, ppas, blba, nr_secs);
+
+ for (i = 0; i < nr_secs; i++) {
+ struct ppa_addr p = ppas[i];
+ sector_t lba = blba + i;
+
+retry:
+ if (pblk_ppa_empty(p)) {
+ WARN_ON(test_and_set_bit(i, read_bitmap));
+ continue;
+ }
+
+ /* Try to read from write buffer. The address is later checked
+ * on the write buffer to prevent retrieving overwritten data.
+ */
+ if (pblk_addr_in_cache(p)) {
+ if (!pblk_read_from_cache(pblk, bio, lba, p, i)) {
+ pblk_lookup_l2p_seq(pblk, &p, lba, 1);
+ goto retry;
+ }
+ WARN_ON(test_and_set_bit(i, read_bitmap));
+ advanced_bio = 1;
+ } else {
+ /* Read from media non-cached sectors */
+ rqd->ppa_list[j++] = p;
+ }
+
+ if (advanced_bio)
+ bio_advance(bio, PBLK_EXPOSED_PAGE_SIZE);
+ }
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_add(nr_secs, &pblk->inflight_reads);
+#endif
+}
+
+static int pblk_submit_read_io(struct pblk *pblk, struct nvm_rq *rqd)
+{
+ int err;
+
+ rqd->flags = pblk_set_read_mode(pblk);
+
+ err = pblk_submit_io(pblk, rqd);
+ if (err)
+ return NVM_IO_ERR;
+
+ return NVM_IO_OK;
+}
+
+static void pblk_end_io_read(struct nvm_rq *rqd)
+{
+ struct pblk *pblk = rqd->private;
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct pblk_r_ctx *r_ctx = nvm_rq_to_pdu(rqd);
+ struct bio *bio = rqd->bio;
+
+ if (rqd->error)
+ pblk_log_read_err(pblk, rqd);
+#ifdef CONFIG_NVM_DEBUG
+ else
+ WARN_ONCE(bio->bi_error, "pblk: corrupted read error\n");
+#endif
+
+ if (rqd->nr_ppas > 1)
+ nvm_dev_dma_free(dev->parent, rqd->ppa_list, rqd->dma_ppa_list);
+
+ bio_put(bio);
+ if (r_ctx->orig_bio) {
+#ifdef CONFIG_NVM_DEBUG
+ WARN_ONCE(r_ctx->orig_bio->bi_error,
+ "pblk: corrupted read bio\n");
+#endif
+ bio_endio(r_ctx->orig_bio);
+ bio_put(r_ctx->orig_bio);
+ }
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_add(rqd->nr_ppas, &pblk->sync_reads);
+ atomic_long_sub(rqd->nr_ppas, &pblk->inflight_reads);
+#endif
+
+ pblk_free_rqd(pblk, rqd, READ);
+}
+
+static int pblk_fill_partial_read_bio(struct pblk *pblk, struct nvm_rq *rqd,
+ unsigned int bio_init_idx,
+ unsigned long *read_bitmap)
+{
+ struct bio *new_bio, *bio = rqd->bio;
+ struct bio_vec src_bv, dst_bv;
+ void *ppa_ptr = NULL;
+ void *src_p, *dst_p;
+ dma_addr_t dma_ppa_list = 0;
+ int nr_secs = rqd->nr_ppas;
+ int nr_holes = nr_secs - bitmap_weight(read_bitmap, nr_secs);
+ int i, ret, hole;
+ DECLARE_COMPLETION_ONSTACK(wait);
+
+ new_bio = bio_alloc(GFP_KERNEL, nr_holes);
+ if (!new_bio) {
+ pr_err("pblk: could not alloc read bio\n");
+ return NVM_IO_ERR;
+ }
+
+ if (pblk_bio_add_pages(pblk, new_bio, GFP_KERNEL, nr_holes))
+ goto err;
+
+ if (nr_holes != new_bio->bi_vcnt) {
+ pr_err("pblk: malformed bio\n");
+ goto err;
+ }
+
+ new_bio->bi_iter.bi_sector = 0; /* internal bio */
+ bio_set_op_attrs(new_bio, REQ_OP_READ, 0);
+ new_bio->bi_private = &wait;
+ new_bio->bi_end_io = pblk_end_bio_sync;
+
+ rqd->bio = new_bio;
+ rqd->nr_ppas = nr_holes;
+ rqd->end_io = NULL;
+
+ if (unlikely(nr_secs > 1 && nr_holes == 1)) {
+ ppa_ptr = rqd->ppa_list;
+ dma_ppa_list = rqd->dma_ppa_list;
+ rqd->ppa_addr = rqd->ppa_list[0];
+ }
+
+ ret = pblk_submit_read_io(pblk, rqd);
+ if (ret) {
+ bio_put(rqd->bio);
+ pr_err("pblk: read IO submission failed\n");
+ goto err;
+ }
+
+ if (!wait_for_completion_io_timeout(&wait,
+ msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
+ pr_err("pblk: partial read I/O timed out\n");
+ }
+
+ if (rqd->error) {
+ atomic_long_inc(&pblk->read_failed);
+#ifdef CONFIG_NVM_DEBUG
+ pblk_print_failed_rqd(pblk, rqd, rqd->error);
+#endif
+ }
+
+ if (unlikely(nr_secs > 1 && nr_holes == 1)) {
+ rqd->ppa_list = ppa_ptr;
+ rqd->dma_ppa_list = dma_ppa_list;
+ }
+
+ /* Fill the holes in the original bio */
+ i = 0;
+ hole = find_first_zero_bit(read_bitmap, nr_secs);
+ do {
+ src_bv = new_bio->bi_io_vec[i++];
+ dst_bv = bio->bi_io_vec[bio_init_idx + hole];
+
+ src_p = kmap_atomic(src_bv.bv_page);
+ dst_p = kmap_atomic(dst_bv.bv_page);
+
+ memcpy(dst_p + dst_bv.bv_offset,
+ src_p + src_bv.bv_offset,
+ PBLK_EXPOSED_PAGE_SIZE);
+
+ kunmap_atomic(src_p);
+ kunmap_atomic(dst_p);
+
+ mempool_free(src_bv.bv_page, pblk->page_pool);
+
+ hole = find_next_zero_bit(read_bitmap, nr_secs, hole + 1);
+ } while (hole < nr_secs);
+
+ bio_put(new_bio);
+
+ /* Complete the original bio and associated request */
+ rqd->bio = bio;
+ rqd->nr_ppas = nr_secs;
+ rqd->private = pblk;
+
+ bio_endio(bio);
+ pblk_end_io_read(rqd);
+ return NVM_IO_OK;
+
+err:
+ /* Free allocated pages in new bio */
+ pblk_bio_free_pages(pblk, bio, 0, new_bio->bi_vcnt);
+ rqd->private = pblk;
+ pblk_end_io_read(rqd);
+ return NVM_IO_ERR;
+}
+
+static void pblk_read_rq(struct pblk *pblk, struct nvm_rq *rqd,
+ unsigned long *read_bitmap)
+{
+ struct bio *bio = rqd->bio;
+ struct ppa_addr ppa;
+ sector_t lba = pblk_get_lba(bio);
+
+ /* logic error: lba out-of-bounds. Ignore read request */
+ if (lba >= pblk->rl.nr_secs) {
+ WARN(1, "pblk: read lba out of bounds\n");
+ return;
+ }
+
+ pblk_lookup_l2p_seq(pblk, &ppa, lba, 1);
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_inc(&pblk->inflight_reads);
+#endif
+
+retry:
+ if (pblk_ppa_empty(ppa)) {
+ WARN_ON(test_and_set_bit(0, read_bitmap));
+ return;
+ }
+
+ /* Try to read from write buffer. The address is later checked on the
+ * write buffer to prevent retrieving overwritten data.
+ */
+ if (pblk_addr_in_cache(ppa)) {
+ if (!pblk_read_from_cache(pblk, bio, lba, ppa, 0)) {
+ pblk_lookup_l2p_seq(pblk, &ppa, lba, 1);
+ goto retry;
+ }
+ WARN_ON(test_and_set_bit(0, read_bitmap));
+ } else {
+ rqd->ppa_addr = ppa;
+ }
+}
+
+int pblk_submit_read(struct pblk *pblk, struct bio *bio)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ unsigned int nr_secs = pblk_get_secs(bio);
+ struct nvm_rq *rqd;
+ unsigned long read_bitmap; /* Max 64 ppas per request */
+ unsigned int bio_init_idx;
+ int ret = NVM_IO_ERR;
+
+ if (nr_secs > PBLK_MAX_REQ_ADDRS)
+ return NVM_IO_ERR;
+
+ bitmap_zero(&read_bitmap, nr_secs);
+
+ rqd = pblk_alloc_rqd(pblk, READ);
+ if (IS_ERR(rqd)) {
+ pr_err_ratelimited("pblk: not able to alloc rqd");
+ return NVM_IO_ERR;
+ }
+
+ rqd->opcode = NVM_OP_PREAD;
+ rqd->bio = bio;
+ rqd->nr_ppas = nr_secs;
+ rqd->private = pblk;
+ rqd->end_io = pblk_end_io_read;
+
+ /* Save the index for this bio's start. This is needed in case
+ * we need to fill a partial read.
+ */
+ bio_init_idx = pblk_get_bi_idx(bio);
+
+ if (nr_secs > 1) {
+ rqd->ppa_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL,
+ &rqd->dma_ppa_list);
+ if (!rqd->ppa_list) {
+ pr_err("pblk: not able to allocate ppa list\n");
+ goto fail_rqd_free;
+ }
+
+ pblk_read_ppalist_rq(pblk, rqd, &read_bitmap);
+ } else {
+ pblk_read_rq(pblk, rqd, &read_bitmap);
+ }
+
+ bio_get(bio);
+ if (bitmap_full(&read_bitmap, nr_secs)) {
+ bio_endio(bio);
+ pblk_end_io_read(rqd);
+ return NVM_IO_OK;
+ }
+
+ /* All sectors are to be read from the device */
+ if (bitmap_empty(&read_bitmap, rqd->nr_ppas)) {
+ struct bio *int_bio = NULL;
+ struct pblk_r_ctx *r_ctx = nvm_rq_to_pdu(rqd);
+
+ /* Clone read bio to deal with read errors internally */
+ int_bio = bio_clone_bioset(bio, GFP_KERNEL, fs_bio_set);
+ if (!int_bio) {
+ pr_err("pblk: could not clone read bio\n");
+ return NVM_IO_ERR;
+ }
+
+ rqd->bio = int_bio;
+ r_ctx->orig_bio = bio;
+
+ ret = pblk_submit_read_io(pblk, rqd);
+ if (ret) {
+ pr_err("pblk: read IO submission failed\n");
+ if (int_bio)
+ bio_put(int_bio);
+ return ret;
+ }
+
+ return NVM_IO_OK;
+ }
+
+ /* The read bio request could be partially filled by the write buffer,
+ * but there are some holes that need to be read from the drive.
+ */
+ ret = pblk_fill_partial_read_bio(pblk, rqd, bio_init_idx, &read_bitmap);
+ if (ret) {
+ pr_err("pblk: failed to perform partial read\n");
+ return ret;
+ }
+
+ return NVM_IO_OK;
+
+fail_rqd_free:
+ pblk_free_rqd(pblk, rqd, READ);
+ return ret;
+}
+
+static int read_ppalist_rq_gc(struct pblk *pblk, struct nvm_rq *rqd,
+ struct pblk_line *line, u64 *lba_list,
+ unsigned int nr_secs)
+{
+ struct ppa_addr ppas[PBLK_MAX_REQ_ADDRS];
+ int valid_secs = 0;
+ int i;
+
+ pblk_lookup_l2p_rand(pblk, ppas, lba_list, nr_secs);
+
+ for (i = 0; i < nr_secs; i++) {
+ if (pblk_addr_in_cache(ppas[i]) || ppas[i].g.blk != line->id ||
+ pblk_ppa_empty(ppas[i])) {
+ lba_list[i] = ADDR_EMPTY;
+ continue;
+ }
+
+ rqd->ppa_list[valid_secs++] = ppas[i];
+ }
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_add(valid_secs, &pblk->inflight_reads);
+#endif
+ return valid_secs;
+}
+
+static int read_rq_gc(struct pblk *pblk, struct nvm_rq *rqd,
+ struct pblk_line *line, sector_t lba)
+{
+ struct ppa_addr ppa;
+ int valid_secs = 0;
+
+ if (lba == ADDR_EMPTY)
+ goto out;
+
+ /* logic error: lba out-of-bounds */
+ if (lba >= pblk->rl.nr_secs) {
+ WARN(1, "pblk: read lba out of bounds\n");
+ goto out;
+ }
+
+ spin_lock(&pblk->trans_lock);
+ ppa = pblk_trans_map_get(pblk, lba);
+ spin_unlock(&pblk->trans_lock);
+
+ /* Ignore updated values until the moment */
+ if (pblk_addr_in_cache(ppa) || ppa.g.blk != line->id ||
+ pblk_ppa_empty(ppa))
+ goto out;
+
+ rqd->ppa_addr = ppa;
+ valid_secs = 1;
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_inc(&pblk->inflight_reads);
+#endif
+
+out:
+ return valid_secs;
+}
+
+int pblk_submit_read_gc(struct pblk *pblk, u64 *lba_list, void *data,
+ unsigned int nr_secs, unsigned int *secs_to_gc,
+ struct pblk_line *line)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct request_queue *q = dev->q;
+ struct bio *bio;
+ struct nvm_rq rqd;
+ int ret, data_len;
+ DECLARE_COMPLETION_ONSTACK(wait);
+
+ memset(&rqd, 0, sizeof(struct nvm_rq));
+
+ if (nr_secs > 1) {
+ rqd.ppa_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL,
+ &rqd.dma_ppa_list);
+ if (!rqd.ppa_list)
+ return NVM_IO_ERR;
+
+ *secs_to_gc = read_ppalist_rq_gc(pblk, &rqd, line, lba_list,
+ nr_secs);
+ if (*secs_to_gc == 1) {
+ struct ppa_addr ppa;
+
+ ppa = rqd.ppa_list[0];
+ nvm_dev_dma_free(dev->parent, rqd.ppa_list,
+ rqd.dma_ppa_list);
+ rqd.ppa_addr = ppa;
+ }
+ } else {
+ *secs_to_gc = read_rq_gc(pblk, &rqd, line, lba_list[0]);
+ }
+
+ if (!(*secs_to_gc))
+ goto out;
+
+ data_len = (*secs_to_gc) * geo->sec_size;
+ bio = bio_map_kern(q, data, data_len, GFP_KERNEL);
+ if (IS_ERR(bio)) {
+ pr_err("pblk: could not allocate GC bio (%lu)\n", PTR_ERR(bio));
+ goto err_free_dma;
+ }
+
+ bio->bi_iter.bi_sector = 0; /* internal bio */
+ bio_set_op_attrs(bio, REQ_OP_READ, 0);
+
+ rqd.opcode = NVM_OP_PREAD;
+ rqd.end_io = pblk_end_io_sync;
+ rqd.private = &wait;
+ rqd.nr_ppas = *secs_to_gc;
+ rqd.bio = bio;
+
+ ret = pblk_submit_read_io(pblk, &rqd);
+ if (ret) {
+ bio_endio(bio);
+ pr_err("pblk: GC read request failed\n");
+ goto err_free_dma;
+ }
+
+ if (!wait_for_completion_io_timeout(&wait,
+ msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
+ pr_err("pblk: GC read I/O timed out\n");
+ }
+
+ if (rqd.error) {
+ atomic_long_inc(&pblk->read_failed_gc);
+#ifdef CONFIG_NVM_DEBUG
+ pblk_print_failed_rqd(pblk, &rqd, rqd.error);
+#endif
+ }
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_add(*secs_to_gc, &pblk->sync_reads);
+ atomic_long_add(*secs_to_gc, &pblk->recov_gc_reads);
+ atomic_long_sub(*secs_to_gc, &pblk->inflight_reads);
+#endif
+
+out:
+ if (rqd.nr_ppas > 1)
+ nvm_dev_dma_free(dev->parent, rqd.ppa_list, rqd.dma_ppa_list);
+ return NVM_IO_OK;
+
+err_free_dma:
+ if (rqd.nr_ppas > 1)
+ nvm_dev_dma_free(dev->parent, rqd.ppa_list, rqd.dma_ppa_list);
+ return NVM_IO_ERR;
+}
diff --git a/drivers/lightnvm/pblk-recovery.c b/drivers/lightnvm/pblk-recovery.c
new file mode 100644
index 0000000..f8f8508
--- /dev/null
+++ b/drivers/lightnvm/pblk-recovery.c
@@ -0,0 +1,998 @@
+/*
+ * Copyright (C) 2016 CNEX Labs
+ * Initial: Javier Gonzalez <javier@cnexlabs.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * pblk-recovery.c - pblk's recovery path
+ */
+
+#include "pblk.h"
+
+void pblk_submit_rec(struct work_struct *work)
+{
+ struct pblk_rec_ctx *recovery =
+ container_of(work, struct pblk_rec_ctx, ws_rec);
+ struct pblk *pblk = recovery->pblk;
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_rq *rqd = recovery->rqd;
+ struct pblk_c_ctx *c_ctx = nvm_rq_to_pdu(rqd);
+ int max_secs = nvm_max_phys_sects(dev);
+ struct bio *bio;
+ unsigned int nr_rec_secs;
+ unsigned int pgs_read;
+ int ret;
+
+ nr_rec_secs = bitmap_weight((unsigned long int *)&rqd->ppa_status,
+ max_secs);
+
+ bio = bio_alloc(GFP_KERNEL, nr_rec_secs);
+ if (!bio) {
+ pr_err("pblk: not able to create recovery bio\n");
+ return;
+ }
+
+ bio->bi_iter.bi_sector = 0;
+ bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
+ rqd->bio = bio;
+ rqd->nr_ppas = nr_rec_secs;
+
+ pgs_read = pblk_rb_read_to_bio_list(&pblk->rwb, bio, &recovery->failed,
+ nr_rec_secs);
+ if (pgs_read != nr_rec_secs) {
+ pr_err("pblk: could not read recovery entries\n");
+ goto err;
+ }
+
+ if (pblk_setup_w_rec_rq(pblk, rqd, c_ctx)) {
+ pr_err("pblk: could not setup recovery request\n");
+ goto err;
+ }
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_add(nr_rec_secs, &pblk->recov_writes);
+#endif
+
+ ret = pblk_submit_io(pblk, rqd);
+ if (ret) {
+ pr_err("pblk: I/O submission failed: %d\n", ret);
+ goto err;
+ }
+
+ mempool_free(recovery, pblk->rec_pool);
+ return;
+
+err:
+ bio_put(bio);
+ pblk_free_rqd(pblk, rqd, WRITE);
+}
+
+int pblk_recov_setup_rq(struct pblk *pblk, struct pblk_c_ctx *c_ctx,
+ struct pblk_rec_ctx *recovery, u64 *comp_bits,
+ unsigned int comp)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ int max_secs = nvm_max_phys_sects(dev);
+ struct nvm_rq *rec_rqd;
+ struct pblk_c_ctx *rec_ctx;
+ int nr_entries = c_ctx->nr_valid + c_ctx->nr_padded;
+
+ rec_rqd = pblk_alloc_rqd(pblk, WRITE);
+ if (IS_ERR(rec_rqd)) {
+ pr_err("pblk: could not create recovery req.\n");
+ return -ENOMEM;
+ }
+
+ rec_ctx = nvm_rq_to_pdu(rec_rqd);
+
+ /* Copy completion bitmap, but exclude the first X completed entries */
+ bitmap_shift_right((unsigned long int *)&rec_rqd->ppa_status,
+ (unsigned long int *)comp_bits,
+ comp, max_secs);
+
+ /* Save the context for the entries that need to be re-written and
+ * update current context with the completed entries.
+ */
+ rec_ctx->sentry = pblk_rb_wrap_pos(&pblk->rwb, c_ctx->sentry + comp);
+ if (comp >= c_ctx->nr_valid) {
+ rec_ctx->nr_valid = 0;
+ rec_ctx->nr_padded = nr_entries - comp;
+
+ c_ctx->nr_padded = comp - c_ctx->nr_valid;
+ } else {
+ rec_ctx->nr_valid = c_ctx->nr_valid - comp;
+ rec_ctx->nr_padded = c_ctx->nr_padded;
+
+ c_ctx->nr_valid = comp;
+ c_ctx->nr_padded = 0;
+ }
+
+ recovery->rqd = rec_rqd;
+ recovery->pblk = pblk;
+
+ return 0;
+}
+
+__le64 *pblk_recov_get_lba_list(struct pblk *pblk, struct line_emeta *emeta)
+{
+ u32 crc;
+
+ crc = pblk_calc_emeta_crc(pblk, emeta);
+ if (le32_to_cpu(emeta->crc) != crc)
+ return NULL;
+
+ if (le32_to_cpu(emeta->header.identifier) != PBLK_MAGIC)
+ return NULL;
+
+ return pblk_line_emeta_to_lbas(emeta);
+}
+
+static int pblk_recov_l2p_from_emeta(struct pblk *pblk, struct pblk_line *line)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct line_emeta *emeta = line->emeta;
+ __le64 *lba_list;
+ int data_start;
+ int nr_data_lbas, nr_valid_lbas, nr_lbas = 0;
+ int i;
+
+ lba_list = pblk_recov_get_lba_list(pblk, emeta);
+ if (!lba_list)
+ return 1;
+
+ data_start = pblk_line_smeta_start(pblk, line) + lm->smeta_sec;
+ nr_data_lbas = lm->sec_per_line - lm->emeta_sec;
+ nr_valid_lbas = le64_to_cpu(emeta->nr_valid_lbas);
+
+ for (i = data_start; i < nr_data_lbas && nr_lbas < nr_valid_lbas; i++) {
+ struct ppa_addr ppa;
+ int pos;
+
+ ppa = addr_to_pblk_ppa(pblk, i, line->id);
+ pos = pblk_ppa_to_pos(geo, ppa);
+
+ /* Do not update bad blocks */
+ if (test_bit(pos, line->blk_bitmap))
+ continue;
+
+ if (le64_to_cpu(lba_list[i]) == ADDR_EMPTY) {
+ spin_lock(&line->lock);
+ if (test_and_set_bit(i, line->invalid_bitmap))
+ WARN_ONCE(1, "pblk: rec. double invalidate:\n");
+ else
+ line->vsc--;
+ spin_unlock(&line->lock);
+
+ continue;
+ }
+
+ pblk_update_map(pblk, le64_to_cpu(lba_list[i]), ppa);
+ nr_lbas++;
+ }
+
+ if (nr_valid_lbas != nr_lbas)
+ pr_err("pblk: line %d - inconsistent lba list(%llu/%d)\n",
+ line->id, line->emeta->nr_valid_lbas, nr_lbas);
+
+ line->left_msecs = 0;
+
+ return 0;
+}
+
+static int pblk_calc_sec_in_line(struct pblk *pblk, struct pblk_line *line)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line_meta *lm = &pblk->lm;
+ int nr_bb = bitmap_weight(line->blk_bitmap, lm->blk_per_line);
+
+ return lm->sec_per_line - lm->smeta_sec - lm->emeta_sec -
+ nr_bb * geo->sec_per_blk;
+}
+
+struct pblk_recov_alloc {
+ struct ppa_addr *ppa_list;
+ struct pblk_sec_meta *meta_list;
+ struct nvm_rq *rqd;
+ void *data;
+ dma_addr_t dma_ppa_list;
+ dma_addr_t dma_meta_list;
+};
+
+static int pblk_recov_read_oob(struct pblk *pblk, struct pblk_line *line,
+ struct pblk_recov_alloc p, u64 r_ptr)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct ppa_addr *ppa_list;
+ struct pblk_sec_meta *meta_list;
+ struct nvm_rq *rqd;
+ struct bio *bio;
+ void *data;
+ dma_addr_t dma_ppa_list, dma_meta_list;
+ u64 r_ptr_int;
+ int left_ppas;
+ int rq_ppas, rq_len;
+ int i, j;
+ int ret = 0;
+ DECLARE_COMPLETION_ONSTACK(wait);
+
+ ppa_list = p.ppa_list;
+ meta_list = p.meta_list;
+ rqd = p.rqd;
+ data = p.data;
+ dma_ppa_list = p.dma_ppa_list;
+ dma_meta_list = p.dma_meta_list;
+
+ left_ppas = line->cur_sec - r_ptr;
+ if (!left_ppas)
+ return 0;
+
+ r_ptr_int = r_ptr;
+
+next_read_rq:
+ memset(rqd, 0, pblk_r_rq_size);
+
+ rq_ppas = pblk_calc_secs(pblk, left_ppas, 0);
+ if (!rq_ppas)
+ rq_ppas = pblk->min_write_pgs;
+ rq_len = rq_ppas * geo->sec_size;
+
+ bio = bio_map_kern(dev->q, data, rq_len, GFP_KERNEL);
+ if (IS_ERR(bio))
+ return PTR_ERR(bio);
+
+ bio->bi_iter.bi_sector = 0; /* internal bio */
+ bio_set_op_attrs(bio, REQ_OP_READ, 0);
+
+ rqd->bio = bio;
+ rqd->opcode = NVM_OP_PREAD;
+ rqd->flags = pblk_set_read_mode(pblk);
+ rqd->meta_list = meta_list;
+ rqd->nr_ppas = rq_ppas;
+ rqd->ppa_list = ppa_list;
+ rqd->dma_ppa_list = dma_ppa_list;
+ rqd->dma_meta_list = dma_meta_list;
+ rqd->end_io = pblk_end_io_sync;
+ rqd->private = &wait;
+
+ for (i = 0; i < rqd->nr_ppas; ) {
+ struct ppa_addr ppa;
+ int pos;
+
+ ppa = addr_to_gen_ppa(pblk, r_ptr_int, line->id);
+ pos = pblk_dev_ppa_to_pos(geo, ppa);
+
+ while (test_bit(pos, line->blk_bitmap)) {
+ r_ptr_int += pblk->min_write_pgs;
+ ppa = addr_to_gen_ppa(pblk, r_ptr_int, line->id);
+ pos = pblk_dev_ppa_to_pos(geo, ppa);
+ }
+
+ for (j = 0; j < pblk->min_write_pgs; j++, i++, r_ptr_int++)
+ rqd->ppa_list[i] =
+ addr_to_gen_ppa(pblk, r_ptr_int, line->id);
+ }
+
+ /* If read fails, more padding is needed */
+ ret = pblk_submit_io(pblk, rqd);
+ if (ret) {
+ pr_err("pblk: I/O submission failed: %d\n", ret);
+ return ret;
+ }
+
+ if (!wait_for_completion_io_timeout(&wait,
+ msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
+ pr_err("pblk: L2P recovery read timed out\n");
+ return -EINTR;
+ }
+
+ reinit_completion(&wait);
+
+ /* At this point, the read should not fail. If it does, it is a problem
+ * we cannot recover from here. Need FTL log.
+ */
+ if (rqd->error) {
+ pr_err("pblk: L2P recovery failed (%d)\n", rqd->error);
+ return -EINTR;
+ }
+
+ for (i = 0; i < rqd->nr_ppas; i++) {
+ u64 lba = le64_to_cpu(meta_list[i].lba);
+
+ if (lba == ADDR_EMPTY || lba > pblk->rl.nr_secs)
+ continue;
+
+ pblk_update_map(pblk, lba, rqd->ppa_list[i]);
+ }
+
+ left_ppas -= rq_ppas;
+ if (left_ppas > 0)
+ goto next_read_rq;
+
+ return 0;
+}
+
+static int pblk_recov_pad_oob(struct pblk *pblk, struct pblk_line *line,
+ struct pblk_recov_alloc p, int left_ppas)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct ppa_addr *ppa_list;
+ struct pblk_sec_meta *meta_list;
+ struct nvm_rq *rqd;
+ struct bio *bio;
+ void *data;
+ dma_addr_t dma_ppa_list, dma_meta_list;
+ __le64 *lba_list = pblk_line_emeta_to_lbas(line->emeta);
+ u64 w_ptr = line->cur_sec;
+ int left_line_ppas = line->left_msecs;
+ int rq_ppas, rq_len;
+ int i, j;
+ int ret = 0;
+ DECLARE_COMPLETION_ONSTACK(wait);
+
+ ppa_list = p.ppa_list;
+ meta_list = p.meta_list;
+ rqd = p.rqd;
+ data = p.data;
+ dma_ppa_list = p.dma_ppa_list;
+ dma_meta_list = p.dma_meta_list;
+
+next_pad_rq:
+ rq_ppas = pblk_calc_secs(pblk, left_ppas, 0);
+ if (!rq_ppas)
+ rq_ppas = pblk->min_write_pgs;
+ rq_len = rq_ppas * geo->sec_size;
+
+ bio = bio_map_kern(dev->q, data, rq_len, GFP_KERNEL);
+ if (IS_ERR(bio))
+ return PTR_ERR(bio);
+
+ bio->bi_iter.bi_sector = 0; /* internal bio */
+ bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
+
+ memset(rqd, 0, pblk_r_rq_size);
+
+ rqd->bio = bio;
+ rqd->opcode = NVM_OP_PWRITE;
+ rqd->flags = pblk_set_progr_mode(pblk, WRITE);
+ rqd->meta_list = meta_list;
+ rqd->nr_ppas = rq_ppas;
+ rqd->ppa_list = ppa_list;
+ rqd->dma_ppa_list = dma_ppa_list;
+ rqd->dma_meta_list = dma_meta_list;
+ rqd->end_io = pblk_end_io_sync;
+ rqd->private = &wait;
+
+ for (i = 0; i < rqd->nr_ppas; ) {
+ struct ppa_addr ppa;
+ int pos;
+
+ w_ptr = pblk_alloc_page(pblk, line, pblk->min_write_pgs);
+ ppa = addr_to_pblk_ppa(pblk, w_ptr, line->id);
+ pos = pblk_ppa_to_pos(geo, ppa);
+
+ while (test_bit(pos, line->blk_bitmap)) {
+ w_ptr += pblk->min_write_pgs;
+ ppa = addr_to_pblk_ppa(pblk, w_ptr, line->id);
+ pos = pblk_ppa_to_pos(geo, ppa);
+ }
+
+ for (j = 0; j < pblk->min_write_pgs; j++, i++, w_ptr++) {
+ struct ppa_addr dev_ppa;
+
+ dev_ppa = addr_to_gen_ppa(pblk, w_ptr, line->id);
+
+ pblk_map_invalidate(pblk, dev_ppa);
+ meta_list[i].lba = cpu_to_le64(ADDR_EMPTY);
+ lba_list[w_ptr] = cpu_to_le64(ADDR_EMPTY);
+ rqd->ppa_list[i] = dev_ppa;
+ }
+ }
+
+ ret = pblk_submit_io(pblk, rqd);
+ if (ret) {
+ pr_err("pblk: I/O submission failed: %d\n", ret);
+ return ret;
+ }
+
+ if (!wait_for_completion_io_timeout(&wait,
+ msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
+ pr_err("pblk: L2P recovery write timed out\n");
+ }
+ reinit_completion(&wait);
+
+ left_line_ppas -= rq_ppas;
+ left_ppas -= rq_ppas;
+ if (left_ppas > 0 && left_line_ppas)
+ goto next_pad_rq;
+
+ return 0;
+}
+
+/* When this function is called, it means that not all upper pages have been
+ * written in a page that contains valid data. In order to recover this data, we
+ * first find the write pointer on the device, then we pad all necessary
+ * sectors, and finally attempt to read the valid data
+ */
+static int pblk_recov_scan_all_oob(struct pblk *pblk, struct pblk_line *line,
+ struct pblk_recov_alloc p)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct ppa_addr *ppa_list;
+ struct pblk_sec_meta *meta_list;
+ struct nvm_rq *rqd;
+ struct bio *bio;
+ void *data;
+ dma_addr_t dma_ppa_list, dma_meta_list;
+ u64 w_ptr = 0, r_ptr;
+ int rq_ppas, rq_len;
+ int i, j;
+ int ret = 0;
+ int rec_round;
+ int left_ppas = pblk_calc_sec_in_line(pblk, line) - line->cur_sec;
+ DECLARE_COMPLETION_ONSTACK(wait);
+
+ ppa_list = p.ppa_list;
+ meta_list = p.meta_list;
+ rqd = p.rqd;
+ data = p.data;
+ dma_ppa_list = p.dma_ppa_list;
+ dma_meta_list = p.dma_meta_list;
+
+ /* we could recover up until the line write pointer */
+ r_ptr = line->cur_sec;
+ rec_round = 0;
+
+next_rq:
+ memset(rqd, 0, pblk_r_rq_size);
+
+ rq_ppas = pblk_calc_secs(pblk, left_ppas, 0);
+ if (!rq_ppas)
+ rq_ppas = pblk->min_write_pgs;
+ rq_len = rq_ppas * geo->sec_size;
+
+ bio = bio_map_kern(dev->q, data, rq_len, GFP_KERNEL);
+ if (IS_ERR(bio))
+ return PTR_ERR(bio);
+
+ bio->bi_iter.bi_sector = 0; /* internal bio */
+ bio_set_op_attrs(bio, REQ_OP_READ, 0);
+
+ rqd->bio = bio;
+ rqd->opcode = NVM_OP_PREAD;
+ rqd->flags = pblk_set_read_mode(pblk);
+ rqd->meta_list = meta_list;
+ rqd->nr_ppas = rq_ppas;
+ rqd->ppa_list = ppa_list;
+ rqd->dma_ppa_list = dma_ppa_list;
+ rqd->dma_meta_list = dma_meta_list;
+ rqd->end_io = pblk_end_io_sync;
+ rqd->private = &wait;
+
+ for (i = 0; i < rqd->nr_ppas; ) {
+ struct ppa_addr ppa;
+ int pos;
+
+ w_ptr = pblk_alloc_page(pblk, line, pblk->min_write_pgs);
+ ppa = addr_to_gen_ppa(pblk, w_ptr, line->id);
+ pos = pblk_dev_ppa_to_pos(geo, ppa);
+
+ while (test_bit(pos, line->blk_bitmap)) {
+ w_ptr += pblk->min_write_pgs;
+ ppa = addr_to_gen_ppa(pblk, w_ptr, line->id);
+ pos = pblk_dev_ppa_to_pos(geo, ppa);
+ }
+
+ for (j = 0; j < pblk->min_write_pgs; j++, i++, w_ptr++)
+ rqd->ppa_list[i] =
+ addr_to_gen_ppa(pblk, w_ptr, line->id);
+ }
+
+ ret = pblk_submit_io(pblk, rqd);
+ if (ret) {
+ pr_err("pblk: I/O submission failed: %d\n", ret);
+ return ret;
+ }
+
+ if (!wait_for_completion_io_timeout(&wait,
+ msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
+ pr_err("pblk: L2P recovery read timed out\n");
+ }
+ reinit_completion(&wait);
+
+ /* This should not happen since the read failed during normal recovery,
+ * but the media works funny sometimes...
+ */
+ if (!rec_round++ && !rqd->error) {
+ rec_round = 0;
+ for (i = 0; i < rqd->nr_ppas; i++, r_ptr++) {
+ u64 lba = le64_to_cpu(meta_list[i].lba);
+
+ if (lba == ADDR_EMPTY || lba > pblk->rl.nr_secs)
+ continue;
+
+ pblk_update_map(pblk, lba, rqd->ppa_list[i]);
+ }
+ }
+
+ /* Reached the end of the written line */
+ if (rqd->error == NVM_RSP_ERR_EMPTYPAGE) {
+ int pad_secs, nr_error_bits, bit;
+ int ret;
+
+ bit = find_first_bit((void *)&rqd->ppa_status, rqd->nr_ppas);
+ nr_error_bits = rqd->nr_ppas - bit;
+
+ /* Roll back failed sectors */
+ line->cur_sec -= nr_error_bits;
+ line->left_msecs += nr_error_bits;
+ bitmap_clear(line->map_bitmap, line->cur_sec, nr_error_bits);
+
+ pad_secs = pblk_pad_distance(pblk);
+ if (pad_secs > line->left_msecs)
+ pad_secs = line->left_msecs;
+
+ ret = pblk_recov_pad_oob(pblk, line, p, pad_secs);
+ if (ret)
+ pr_err("pblk: OOB padding failed (err:%d)\n", ret);
+
+ ret = pblk_recov_read_oob(pblk, line, p, r_ptr);
+ if (ret)
+ pr_err("pblk: OOB read failed (err:%d)\n", ret);
+
+ line->left_ssecs = line->left_msecs;
+ left_ppas = 0;
+ }
+
+ left_ppas -= rq_ppas;
+ if (left_ppas > 0)
+ goto next_rq;
+
+ return ret;
+}
+
+static int pblk_recov_scan_oob(struct pblk *pblk, struct pblk_line *line,
+ struct pblk_recov_alloc p, int *done)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct ppa_addr *ppa_list;
+ struct pblk_sec_meta *meta_list;
+ struct nvm_rq *rqd;
+ struct bio *bio;
+ void *data;
+ dma_addr_t dma_ppa_list, dma_meta_list;
+ u64 paddr;
+ int rq_ppas, rq_len;
+ int i, j;
+ int ret = 0;
+ int left_ppas = pblk_calc_sec_in_line(pblk, line);
+ DECLARE_COMPLETION_ONSTACK(wait);
+
+ ppa_list = p.ppa_list;
+ meta_list = p.meta_list;
+ rqd = p.rqd;
+ data = p.data;
+ dma_ppa_list = p.dma_ppa_list;
+ dma_meta_list = p.dma_meta_list;
+
+ *done = 1;
+
+next_rq:
+ memset(rqd, 0, pblk_r_rq_size);
+
+ rq_ppas = pblk_calc_secs(pblk, left_ppas, 0);
+ if (!rq_ppas)
+ rq_ppas = pblk->min_write_pgs;
+ rq_len = rq_ppas * geo->sec_size;
+
+ bio = bio_map_kern(dev->q, data, rq_len, GFP_KERNEL);
+ if (IS_ERR(bio))
+ return PTR_ERR(bio);
+
+ bio->bi_iter.bi_sector = 0; /* internal bio */
+ bio_set_op_attrs(bio, REQ_OP_READ, 0);
+
+ rqd->bio = bio;
+ rqd->opcode = NVM_OP_PREAD;
+ rqd->flags = pblk_set_read_mode(pblk);
+ rqd->meta_list = meta_list;
+ rqd->nr_ppas = rq_ppas;
+ rqd->ppa_list = ppa_list;
+ rqd->dma_ppa_list = dma_ppa_list;
+ rqd->dma_meta_list = dma_meta_list;
+ rqd->end_io = pblk_end_io_sync;
+ rqd->private = &wait;
+
+ for (i = 0; i < rqd->nr_ppas; ) {
+ struct ppa_addr ppa;
+ int pos;
+
+ paddr = pblk_alloc_page(pblk, line, pblk->min_write_pgs);
+ ppa = addr_to_gen_ppa(pblk, paddr, line->id);
+ pos = pblk_dev_ppa_to_pos(geo, ppa);
+
+ while (test_bit(pos, line->blk_bitmap)) {
+ paddr += pblk->min_write_pgs;
+ ppa = addr_to_gen_ppa(pblk, paddr, line->id);
+ pos = pblk_dev_ppa_to_pos(geo, ppa);
+ }
+
+ for (j = 0; j < pblk->min_write_pgs; j++, i++, paddr++)
+ rqd->ppa_list[i] =
+ addr_to_gen_ppa(pblk, paddr, line->id);
+ }
+
+ ret = pblk_submit_io(pblk, rqd);
+ if (ret) {
+ pr_err("pblk: I/O submission failed: %d\n", ret);
+ bio_put(bio);
+ return ret;
+ }
+
+ if (!wait_for_completion_io_timeout(&wait,
+ msecs_to_jiffies(PBLK_COMMAND_TIMEOUT_MS))) {
+ pr_err("pblk: L2P recovery read timed out\n");
+ }
+ reinit_completion(&wait);
+
+ /* Reached the end of the written line */
+ if (rqd->error) {
+ int nr_error_bits, bit;
+
+ bit = find_first_bit((void *)&rqd->ppa_status, rqd->nr_ppas);
+ nr_error_bits = rqd->nr_ppas - bit;
+
+ /* Roll back failed sectors */
+ line->cur_sec -= nr_error_bits;
+ line->left_msecs += nr_error_bits;
+ line->left_ssecs = line->left_msecs;
+ bitmap_clear(line->map_bitmap, line->cur_sec, nr_error_bits);
+
+ left_ppas = 0;
+ rqd->nr_ppas = bit;
+
+ if (rqd->error != NVM_RSP_ERR_EMPTYPAGE)
+ *done = 0;
+ }
+
+ for (i = 0; i < rqd->nr_ppas; i++) {
+ u64 lba = le64_to_cpu(meta_list[i].lba);
+
+ if (lba == ADDR_EMPTY || lba > pblk->rl.nr_secs)
+ continue;
+
+ pblk_update_map(pblk, lba, rqd->ppa_list[i]);
+ }
+
+ left_ppas -= rq_ppas;
+ if (left_ppas > 0)
+ goto next_rq;
+
+ return ret;
+}
+
+/* Scan line for lbas on out of bound area */
+static int pblk_recov_l2p_from_oob(struct pblk *pblk, struct pblk_line *line)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct nvm_rq *rqd;
+ struct ppa_addr *ppa_list;
+ struct pblk_sec_meta *meta_list;
+ struct pblk_recov_alloc p;
+ void *data;
+ dma_addr_t dma_ppa_list, dma_meta_list;
+ int done, ret = 0;
+
+ rqd = pblk_alloc_rqd(pblk, READ);
+ if (IS_ERR(rqd))
+ return PTR_ERR(rqd);
+
+ meta_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL, &dma_meta_list);
+ if (!meta_list) {
+ ret = -ENOMEM;
+ goto free_rqd;
+ }
+
+ ppa_list = (void *)(meta_list) + pblk_dma_meta_size;
+ dma_ppa_list = dma_meta_list + pblk_dma_meta_size;
+
+ data = kcalloc(pblk->max_write_pgs, geo->sec_size, GFP_KERNEL);
+ if (!data) {
+ ret = -ENOMEM;
+ goto free_meta_list;
+ }
+
+ p.ppa_list = ppa_list;
+ p.meta_list = meta_list;
+ p.rqd = rqd;
+ p.data = data;
+ p.dma_ppa_list = dma_ppa_list;
+ p.dma_meta_list = dma_meta_list;
+
+ ret = pblk_recov_scan_oob(pblk, line, p, &done);
+ if (ret) {
+ pr_err("pblk: could not recover L2P from OOB\n");
+ goto out;
+ }
+
+ if (!done) {
+ ret = pblk_recov_scan_all_oob(pblk, line, p);
+ if (ret) {
+ pr_err("pblk: could not recover L2P from OOB\n");
+ goto out;
+ }
+ }
+
+ if (pblk_line_is_full(line))
+ pblk_line_recov_close(pblk, line);
+
+out:
+ kfree(data);
+free_meta_list:
+ nvm_dev_dma_free(dev->parent, meta_list, dma_meta_list);
+free_rqd:
+ pblk_free_rqd(pblk, rqd, READ);
+
+ return ret;
+}
+
+/* Insert lines ordered by sequence number (seq_num) on list */
+static void pblk_recov_line_add_ordered(struct list_head *head,
+ struct pblk_line *line)
+{
+ struct pblk_line *t = NULL;
+
+ list_for_each_entry(t, head, list)
+ if (t->seq_nr > line->seq_nr)
+ break;
+
+ __list_add(&line->list, t->list.prev, &t->list);
+}
+
+struct pblk_line *pblk_recov_l2p(struct pblk *pblk)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line *line, *tline, *data_line = NULL;
+ struct line_smeta *smeta;
+ struct line_emeta *emeta;
+ int found_lines = 0, recovered_lines = 0, open_lines = 0;
+ int is_next = 0;
+ int meta_line;
+ int i, valid_uuid = 0;
+ LIST_HEAD(recov_list);
+
+ /* TODO: Implement FTL snapshot */
+
+ /* Scan recovery - takes place when FTL snapshot fails */
+ spin_lock(&l_mg->free_lock);
+ meta_line = find_first_zero_bit(&l_mg->meta_bitmap, PBLK_DATA_LINES);
+ set_bit(meta_line, &l_mg->meta_bitmap);
+ smeta = l_mg->sline_meta[meta_line].meta;
+ emeta = l_mg->eline_meta[meta_line].meta;
+ spin_unlock(&l_mg->free_lock);
+
+ /* Order data lines using their sequence number */
+ for (i = 0; i < l_mg->nr_lines; i++) {
+ u32 crc;
+
+ line = &pblk->lines[i];
+
+ memset(smeta, 0, lm->smeta_len);
+ line->smeta = smeta;
+ line->lun_bitmap = ((void *)(smeta)) +
+ sizeof(struct line_smeta);
+
+ /* Lines that cannot be read are assumed as not written here */
+ if (pblk_line_read_smeta(pblk, line))
+ continue;
+
+ crc = pblk_calc_smeta_crc(pblk, smeta);
+ if (le32_to_cpu(smeta->crc) != crc)
+ continue;
+
+ if (le32_to_cpu(smeta->header.identifier) != PBLK_MAGIC)
+ continue;
+
+ if (le16_to_cpu(smeta->header.version) != 1) {
+ pr_err("pblk: found incompatible line version %u\n",
+ smeta->header.version);
+ return ERR_PTR(-EINVAL);
+ }
+
+ /* The first valid instance uuid is used for initialization */
+ if (!valid_uuid) {
+ memcpy(pblk->instance_uuid, smeta->header.uuid, 16);
+ valid_uuid = 1;
+ }
+
+ if (memcmp(pblk->instance_uuid, smeta->header.uuid, 16)) {
+ pr_debug("pblk: ignore line %u due to uuid mismatch\n",
+ i);
+ continue;
+ }
+
+ /* Update line metadata */
+ spin_lock(&line->lock);
+ line->id = le32_to_cpu(line->smeta->header.id);
+ line->type = le16_to_cpu(line->smeta->header.type);
+ line->seq_nr = le64_to_cpu(line->smeta->seq_nr);
+ spin_unlock(&line->lock);
+
+ /* Update general metadata */
+ spin_lock(&l_mg->free_lock);
+ if (line->seq_nr >= l_mg->d_seq_nr)
+ l_mg->d_seq_nr = line->seq_nr + 1;
+ l_mg->nr_free_lines--;
+ spin_unlock(&l_mg->free_lock);
+
+ if (pblk_line_recov_alloc(pblk, line))
+ goto out;
+
+ pblk_recov_line_add_ordered(&recov_list, line);
+ found_lines++;
+ pr_debug("pblk: recovering data line %d, seq:%llu\n",
+ line->id, smeta->seq_nr);
+ }
+
+ if (!found_lines) {
+ pblk_setup_uuid(pblk);
+
+ spin_lock(&l_mg->free_lock);
+ WARN_ON_ONCE(!test_and_clear_bit(meta_line,
+ &l_mg->meta_bitmap));
+ spin_unlock(&l_mg->free_lock);
+
+ goto out;
+ }
+
+ /* Verify closed blocks and recover this portion of L2P table*/
+ list_for_each_entry_safe(line, tline, &recov_list, list) {
+ int off, nr_bb;
+
+ recovered_lines++;
+ /* Calculate where emeta starts based on the line bb */
+ off = lm->sec_per_line - lm->emeta_sec;
+ nr_bb = bitmap_weight(line->blk_bitmap, lm->blk_per_line);
+ off -= nr_bb * geo->sec_per_pl;
+
+ memset(emeta, 0, lm->emeta_len);
+ line->emeta = emeta;
+ line->emeta_ssec = off;
+
+ if (pblk_line_read_emeta(pblk, line)) {
+ pblk_recov_l2p_from_oob(pblk, line);
+ goto next;
+ }
+
+ if (pblk_recov_l2p_from_emeta(pblk, line))
+ pblk_recov_l2p_from_oob(pblk, line);
+
+next:
+ if (pblk_line_is_full(line)) {
+ struct list_head *move_list;
+
+ spin_lock(&line->lock);
+ line->state = PBLK_LINESTATE_CLOSED;
+ move_list = pblk_line_gc_list(pblk, line);
+ spin_unlock(&line->lock);
+
+ spin_lock(&l_mg->gc_lock);
+ list_move_tail(&line->list, move_list);
+ spin_unlock(&l_mg->gc_lock);
+
+ mempool_free(line->map_bitmap, pblk->line_meta_pool);
+ line->map_bitmap = NULL;
+ line->smeta = NULL;
+ line->emeta = NULL;
+ } else {
+ if (open_lines > 1)
+ pr_err("pblk: failed to recover L2P\n");
+
+ open_lines++;
+ line->meta_line = meta_line;
+ data_line = line;
+ }
+ }
+
+ spin_lock(&l_mg->free_lock);
+ if (!open_lines) {
+ WARN_ON_ONCE(!test_and_clear_bit(meta_line,
+ &l_mg->meta_bitmap));
+ pblk_line_replace_data(pblk);
+ } else {
+ /* Allocate next line for preparation */
+ l_mg->data_next = pblk_line_get(pblk);
+ if (l_mg->data_next) {
+ l_mg->data_next->seq_nr = l_mg->d_seq_nr++;
+ l_mg->data_next->type = PBLK_LINETYPE_DATA;
+ is_next = 1;
+ }
+ }
+ spin_unlock(&l_mg->free_lock);
+
+ if (is_next) {
+ pblk_line_erase(pblk, l_mg->data_next);
+ pblk_rl_free_lines_dec(&pblk->rl, l_mg->data_next);
+ }
+
+out:
+ if (found_lines != recovered_lines)
+ pr_err("pblk: failed to recover all found lines %d/%d\n",
+ found_lines, recovered_lines);
+
+ return data_line;
+}
+
+/*
+ * Pad until smeta can be read on current data line
+ */
+void pblk_recov_pad(struct pblk *pblk)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line *line;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct nvm_rq *rqd;
+ struct pblk_recov_alloc p;
+ struct ppa_addr *ppa_list;
+ struct pblk_sec_meta *meta_list;
+ void *data;
+ dma_addr_t dma_ppa_list, dma_meta_list;
+
+ spin_lock(&l_mg->free_lock);
+ line = l_mg->data_line;
+ spin_unlock(&l_mg->free_lock);
+
+ rqd = pblk_alloc_rqd(pblk, READ);
+ if (IS_ERR(rqd))
+ return;
+
+ meta_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL, &dma_meta_list);
+ if (!meta_list)
+ goto free_rqd;
+
+ ppa_list = (void *)(meta_list) + pblk_dma_meta_size;
+ dma_ppa_list = dma_meta_list + pblk_dma_meta_size;
+
+ data = kcalloc(pblk->max_write_pgs, geo->sec_size, GFP_KERNEL);
+ if (!data)
+ goto free_meta_list;
+
+ p.ppa_list = ppa_list;
+ p.meta_list = meta_list;
+ p.rqd = rqd;
+ p.data = data;
+ p.dma_ppa_list = dma_ppa_list;
+ p.dma_meta_list = dma_meta_list;
+
+ if (pblk_recov_pad_oob(pblk, line, p, line->left_msecs)) {
+ pr_err("pblk: Tear down padding failed\n");
+ goto free_data;
+ }
+
+ pblk_line_close(pblk, line);
+
+free_data:
+ kfree(data);
+free_meta_list:
+ nvm_dev_dma_free(dev->parent, meta_list, dma_meta_list);
+free_rqd:
+ pblk_free_rqd(pblk, rqd, READ);
+}
diff --git a/drivers/lightnvm/pblk-rl.c b/drivers/lightnvm/pblk-rl.c
new file mode 100644
index 0000000..ab7cbb1
--- /dev/null
+++ b/drivers/lightnvm/pblk-rl.c
@@ -0,0 +1,184 @@
+/*
+ * Copyright (C) 2016 CNEX Labs
+ * Initial release: Javier Gonzalez <javier@cnexlabs.com>
+ * Matias Bjorling <matias@cnexlabs.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * pblk-rl.c - pblk's rate limiter for user I/O
+ *
+ */
+
+#include "pblk.h"
+
+static void pblk_rl_kick_u_timer(struct pblk_rl *rl)
+{
+ mod_timer(&rl->u_timer, jiffies + msecs_to_jiffies(5000));
+}
+
+int pblk_rl_user_may_insert(struct pblk_rl *rl, int nr_entries)
+{
+ int rb_user_cnt = atomic_read(&rl->rb_user_cnt);
+
+ return (!(rb_user_cnt + nr_entries > rl->rb_user_max));
+}
+
+int pblk_rl_gc_may_insert(struct pblk_rl *rl, int nr_entries)
+{
+ int rb_gc_cnt = atomic_read(&rl->rb_gc_cnt);
+ int rb_user_active;
+
+ /* If there is no user I/O let GC take over space on the write buffer */
+ rb_user_active = READ_ONCE(rl->rb_user_active);
+ return (!(rb_gc_cnt + nr_entries > rl->rb_gc_max && rb_user_active));
+}
+
+void pblk_rl_user_in(struct pblk_rl *rl, int nr_entries)
+{
+ atomic_add(nr_entries, &rl->rb_user_cnt);
+
+ /* Release user I/O state. Protect from GC */
+ smp_store_release(&rl->rb_user_active, 1);
+ pblk_rl_kick_u_timer(rl);
+}
+
+void pblk_rl_gc_in(struct pblk_rl *rl, int nr_entries)
+{
+ atomic_add(nr_entries, &rl->rb_gc_cnt);
+}
+
+void pblk_rl_out(struct pblk_rl *rl, int nr_user, int nr_gc)
+{
+ atomic_sub(nr_user, &rl->rb_user_cnt);
+ atomic_sub(nr_gc, &rl->rb_gc_cnt);
+}
+
+unsigned long pblk_rl_nr_free_blks(struct pblk_rl *rl)
+{
+ return atomic_read(&rl->free_blocks);
+}
+
+/*
+ * We check for (i) the number of free blocks in the current LUN and (ii) the
+ * total number of free blocks in the pblk instance. This is to even out the
+ * number of free blocks on each LUN when GC kicks in.
+ *
+ * Only the total number of free blocks is used to configure the rate limiter.
+ */
+static int pblk_rl_update_rates(struct pblk_rl *rl, unsigned long max)
+{
+ unsigned long free_blocks = pblk_rl_nr_free_blks(rl);
+
+ if (free_blocks >= rl->high) {
+ rl->rb_user_max = max - rl->rb_gc_rsv;
+ rl->rb_gc_max = rl->rb_gc_rsv;
+ rl->rb_state = PBLK_RL_HIGH;
+ } else if (free_blocks < rl->high) {
+ int shift = rl->high_pw - rl->rb_windows_pw;
+ int user_windows = free_blocks >> shift;
+ int user_max = user_windows << PBLK_MAX_REQ_ADDRS_PW;
+ int gc_max;
+
+ rl->rb_user_max = user_max;
+ gc_max = max - rl->rb_user_max;
+ rl->rb_gc_max = max(gc_max, rl->rb_gc_rsv);
+
+ if (free_blocks > rl->low)
+ rl->rb_state = PBLK_RL_MID;
+ else
+ rl->rb_state = PBLK_RL_LOW;
+ }
+
+ return rl->rb_state;
+}
+
+void pblk_rl_set_gc_rsc(struct pblk_rl *rl, int rsv)
+{
+ rl->rb_gc_rsv = rl->rb_gc_max = rsv;
+}
+
+void pblk_rl_free_lines_inc(struct pblk_rl *rl, struct pblk_line *line)
+{
+ struct pblk *pblk = container_of(rl, struct pblk, rl);
+ int blk_in_line = atomic_read(&line->blk_in_line);
+ int ret;
+
+ atomic_add(blk_in_line, &rl->free_blocks);
+ /* Rates will not change that often - no need to lock update */
+ ret = pblk_rl_update_rates(rl, rl->rb_budget);
+
+ if (ret == (PBLK_RL_MID | PBLK_RL_LOW))
+ pblk_gc_should_start(pblk);
+ else
+ pblk_gc_should_stop(pblk);
+}
+
+void pblk_rl_free_lines_dec(struct pblk_rl *rl, struct pblk_line *line)
+{
+ struct pblk *pblk = container_of(rl, struct pblk, rl);
+ int blk_in_line = atomic_read(&line->blk_in_line);
+ int ret;
+
+ atomic_sub(blk_in_line, &rl->free_blocks);
+
+ /* Rates will not change that often - no need to lock update */
+ ret = pblk_rl_update_rates(rl, rl->rb_budget);
+ if (ret == (PBLK_RL_MID | PBLK_RL_LOW))
+ pblk_gc_should_start(pblk);
+ else
+ pblk_gc_should_stop(pblk);
+}
+
+int pblk_rl_gc_thrs(struct pblk_rl *rl)
+{
+ return rl->high;
+}
+
+int pblk_rl_sysfs_rate_show(struct pblk_rl *rl)
+{
+ return rl->rb_user_max;
+}
+
+static void pblk_rl_u_timer(unsigned long data)
+{
+ struct pblk_rl *rl = (struct pblk_rl *)data;
+
+ /* Release user I/O state. Protect from GC */
+ smp_store_release(&rl->rb_user_active, 0);
+}
+
+void pblk_rl_free(struct pblk_rl *rl)
+{
+ del_timer(&rl->u_timer);
+}
+
+void pblk_rl_init(struct pblk_rl *rl, int budget)
+{
+ unsigned int rb_windows;
+
+ rl->high = rl->total_blocks / PBLK_USER_HIGH_THRS;
+ rl->low = rl->total_blocks / PBLK_USER_LOW_THRS;
+ rl->high_pw = get_count_order(rl->high);
+
+ /* This will always be a power-of-2 */
+ rb_windows = budget / PBLK_MAX_REQ_ADDRS;
+ rl->rb_windows_pw = get_count_order(rb_windows) + 1;
+
+ /* To start with, all buffer is available to user I/O writers */
+ rl->rb_budget = budget;
+ rl->rb_user_max = budget;
+ atomic_set(&rl->rb_user_cnt, 0);
+ rl->rb_gc_max = 0;
+ rl->rb_state = PBLK_RL_HIGH;
+ atomic_set(&rl->rb_gc_cnt, 0);
+
+ setup_timer(&rl->u_timer, pblk_rl_u_timer, (unsigned long)rl);
+ rl->rb_user_active = 0;
+}
diff --git a/drivers/lightnvm/pblk-sysfs.c b/drivers/lightnvm/pblk-sysfs.c
new file mode 100644
index 0000000..f0af1d1
--- /dev/null
+++ b/drivers/lightnvm/pblk-sysfs.c
@@ -0,0 +1,507 @@
+/*
+ * Copyright (C) 2016 CNEX Labs
+ * Initial release: Javier Gonzalez <javier@cnexlabs.com>
+ * Matias Bjorling <matias@cnexlabs.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Implementation of a physical block-device target for Open-channel SSDs.
+ *
+ * pblk-sysfs.c - pblk's sysfs
+ *
+ */
+
+#include "pblk.h"
+
+static ssize_t pblk_sysfs_luns_show(struct pblk *pblk, char *page)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_lun *rlun;
+ ssize_t sz = 0;
+ int i;
+
+ for (i = 0; i < geo->nr_luns; i++) {
+ int active = 1;
+
+ rlun = &pblk->luns[i];
+ if (!down_trylock(&rlun->wr_sem)) {
+ active = 0;
+ up(&rlun->wr_sem);
+ }
+ sz += snprintf(page + sz, PAGE_SIZE - sz,
+ "pblk: pos:%d, ch:%d, lun:%d - %d\n",
+ i,
+ rlun->bppa.g.ch,
+ rlun->bppa.g.lun,
+ active);
+ }
+
+ return sz;
+}
+
+static ssize_t pblk_sysfs_rate_limiter(struct pblk *pblk, char *page)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ int free_blocks, total_blocks;
+ int rb_user_max, rb_user_cnt;
+ int rb_gc_max, rb_gc_rsv, rb_gc_cnt, rb_budget, rb_state;
+
+ free_blocks = atomic_read(&pblk->rl.free_blocks);
+ rb_user_max = pblk->rl.rb_user_max;
+ rb_user_cnt = atomic_read(&pblk->rl.rb_user_cnt);
+ rb_gc_max = pblk->rl.rb_gc_max;
+ rb_gc_rsv = pblk->rl.rb_gc_rsv;
+ rb_gc_cnt = atomic_read(&pblk->rl.rb_gc_cnt);
+ rb_budget = pblk->rl.rb_budget;
+ rb_state = pblk->rl.rb_state;
+
+ total_blocks = geo->blks_per_lun * geo->nr_luns;
+
+ return snprintf(page, PAGE_SIZE,
+ "u:%u/%u,gc:%u/%u/%u(%u/%u)(stop:<%u,full:>%u,free:%d/%d)-%d\n",
+ rb_user_cnt,
+ rb_user_max,
+ rb_gc_cnt,
+ rb_gc_max,
+ rb_gc_rsv,
+ rb_state,
+ rb_budget,
+ pblk->rl.low,
+ pblk->rl.high,
+ free_blocks,
+ total_blocks,
+ READ_ONCE(pblk->rl.rb_user_active));
+}
+
+static ssize_t pblk_sysfs_gc_state_show(struct pblk *pblk, char *page)
+{
+ int gc_enabled, gc_active;
+
+ pblk_gc_sysfs_state_show(pblk, &gc_enabled, &gc_active);
+ return snprintf(page, PAGE_SIZE, "gc_enabled=%d, gc_active=%d\n",
+ gc_enabled, gc_active);
+}
+
+static ssize_t pblk_sysfs_stats(struct pblk *pblk, char *page)
+{
+ ssize_t sz;
+
+ sz = snprintf(page, PAGE_SIZE,
+ "read_failed=%lu, read_high_ecc=%lu, read_empty=%lu, read_failed_gc=%lu, write_failed=%lu, erase_failed=%lu\n",
+ atomic_long_read(&pblk->read_failed),
+ atomic_long_read(&pblk->read_high_ecc),
+ atomic_long_read(&pblk->read_empty),
+ atomic_long_read(&pblk->read_failed_gc),
+ atomic_long_read(&pblk->write_failed),
+ atomic_long_read(&pblk->erase_failed));
+
+ return sz;
+}
+
+static ssize_t pblk_sysfs_write_buffer(struct pblk *pblk, char *page)
+{
+ return pblk_rb_sysfs(&pblk->rwb, page);
+}
+
+static ssize_t pblk_sysfs_ppaf(struct pblk *pblk, char *page)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ ssize_t sz = 0;
+
+ sz = snprintf(page, PAGE_SIZE - sz,
+ "g:(b:%d)blk:%d/%d,pg:%d/%d,lun:%d/%d,ch:%d/%d,pl:%d/%d,sec:%d/%d\n",
+ pblk->ppaf_bitsize,
+ pblk->ppaf.blk_offset, geo->ppaf.blk_len,
+ pblk->ppaf.pg_offset, geo->ppaf.pg_len,
+ pblk->ppaf.lun_offset, geo->ppaf.lun_len,
+ pblk->ppaf.ch_offset, geo->ppaf.ch_len,
+ pblk->ppaf.pln_offset, geo->ppaf.pln_len,
+ pblk->ppaf.sec_offset, geo->ppaf.sect_len);
+
+ sz += snprintf(page + sz, PAGE_SIZE - sz,
+ "d:blk:%d/%d,pg:%d/%d,lun:%d/%d,ch:%d/%d,pl:%d/%d,sec:%d/%d\n",
+ geo->ppaf.blk_offset, geo->ppaf.blk_len,
+ geo->ppaf.pg_offset, geo->ppaf.pg_len,
+ geo->ppaf.lun_offset, geo->ppaf.lun_len,
+ geo->ppaf.ch_offset, geo->ppaf.ch_len,
+ geo->ppaf.pln_offset, geo->ppaf.pln_len,
+ geo->ppaf.sect_offset, geo->ppaf.sect_len);
+
+ return sz;
+}
+
+static ssize_t pblk_sysfs_lines(struct pblk *pblk, char *page)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_line_mgmt *l_mg = &pblk->l_mg;
+ struct pblk_line *line;
+ ssize_t sz = 0;
+ int nr_free_lines;
+ int cur_data, cur_log;
+ int free_line_cnt = 0, closed_line_cnt = 0;
+ int d_line_cnt = 0, l_line_cnt = 0;
+ int gc_full = 0, gc_high = 0, gc_mid = 0, gc_low = 0, gc_empty = 0;
+ int free = 0, bad = 0, cor = 0;
+ int msecs = 0, ssecs = 0, cur_sec = 0, vsc = 0, sec_in_line = 0;
+ int map_weight = 0, meta_weight = 0;
+
+ spin_lock(&l_mg->free_lock);
+ cur_data = (l_mg->data_line) ? l_mg->data_line->id : -1;
+ cur_log = (l_mg->log_line) ? l_mg->log_line->id : -1;
+ nr_free_lines = l_mg->nr_free_lines;
+
+ list_for_each_entry(line, &l_mg->free_list, list)
+ free_line_cnt++;
+ spin_unlock(&l_mg->free_lock);
+
+ spin_lock(&l_mg->gc_lock);
+ list_for_each_entry(line, &l_mg->gc_full_list, list) {
+ if (line->type == PBLK_LINETYPE_DATA)
+ d_line_cnt++;
+ else if (line->type == PBLK_LINETYPE_LOG)
+ l_line_cnt++;
+ closed_line_cnt++;
+ gc_full++;
+ }
+
+ list_for_each_entry(line, &l_mg->gc_high_list, list) {
+ if (line->type == PBLK_LINETYPE_DATA)
+ d_line_cnt++;
+ else if (line->type == PBLK_LINETYPE_LOG)
+ l_line_cnt++;
+ closed_line_cnt++;
+ gc_high++;
+ }
+
+ list_for_each_entry(line, &l_mg->gc_mid_list, list) {
+ if (line->type == PBLK_LINETYPE_DATA)
+ d_line_cnt++;
+ else if (line->type == PBLK_LINETYPE_LOG)
+ l_line_cnt++;
+ closed_line_cnt++;
+ gc_mid++;
+ }
+
+ list_for_each_entry(line, &l_mg->gc_low_list, list) {
+ if (line->type == PBLK_LINETYPE_DATA)
+ d_line_cnt++;
+ else if (line->type == PBLK_LINETYPE_LOG)
+ l_line_cnt++;
+ closed_line_cnt++;
+ gc_low++;
+ }
+
+ list_for_each_entry(line, &l_mg->gc_empty_list, list) {
+ if (line->type == PBLK_LINETYPE_DATA)
+ d_line_cnt++;
+ else if (line->type == PBLK_LINETYPE_LOG)
+ l_line_cnt++;
+ closed_line_cnt++;
+ gc_empty++;
+ }
+
+ list_for_each_entry(line, &l_mg->free_list, list)
+ free++;
+ list_for_each_entry(line, &l_mg->bad_list, list)
+ bad++;
+ list_for_each_entry(line, &l_mg->corrupt_list, list)
+ cor++;
+ spin_unlock(&l_mg->gc_lock);
+
+ spin_lock(&l_mg->free_lock);
+ if (l_mg->data_line) {
+ cur_sec = l_mg->data_line->cur_sec;
+ msecs = l_mg->data_line->left_msecs;
+ ssecs = l_mg->data_line->left_ssecs;
+ vsc = l_mg->data_line->vsc;
+ sec_in_line = l_mg->data_line->sec_in_line;
+ meta_weight = bitmap_weight(&l_mg->meta_bitmap,
+ PBLK_DATA_LINES);
+ map_weight = bitmap_weight(l_mg->data_line->map_bitmap,
+ lm->sec_per_line);
+ }
+ spin_unlock(&l_mg->free_lock);
+
+ if (nr_free_lines != free_line_cnt)
+ pr_err("pblk: corrupted free line list\n");
+
+ sz = snprintf(page, PAGE_SIZE - sz,
+ "line: nluns:%d, nblks:%d, nsecs:%d\n",
+ geo->nr_luns, lm->blk_per_line, lm->sec_per_line);
+
+ sz += snprintf(page + sz, PAGE_SIZE - sz,
+ "lines:d:%d,l:%d-f:%d(%d),b:%d,co:%d,c:%d(d:%d,l:%d)t:%d\n",
+ cur_data, cur_log,
+ free, nr_free_lines, bad, cor,
+ closed_line_cnt,
+ d_line_cnt, l_line_cnt,
+ l_mg->nr_lines);
+
+ sz += snprintf(page + sz, PAGE_SIZE - sz,
+ "GC: full:%d, high:%d, mid:%d, low:%d, empty:%d, queue:%d\n",
+ gc_full, gc_high, gc_mid, gc_low, gc_empty,
+ atomic_read(&pblk->gc.inflight_gc));
+
+ sz += snprintf(page + sz, PAGE_SIZE - sz,
+ "data (%d) cur:%d, left:%d/%d, vsc:%d, s:%d, map:%d/%d (%d)\n",
+ cur_data, cur_sec, msecs, ssecs, vsc, sec_in_line,
+ map_weight, lm->sec_per_line, meta_weight);
+
+ return sz;
+}
+
+static ssize_t pblk_sysfs_lines_info(struct pblk *pblk, char *page)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ struct pblk_line_meta *lm = &pblk->lm;
+ ssize_t sz = 0;
+
+ sz = snprintf(page, PAGE_SIZE - sz,
+ "smeta - len:%d, secs:%d\n",
+ lm->smeta_len, lm->smeta_sec);
+ sz += snprintf(page + sz, PAGE_SIZE - sz,
+ "emeta - len:%d, sec:%d, bb_start:%d\n",
+ lm->emeta_len, lm->emeta_sec,
+ lm->emeta_bb);
+ sz += snprintf(page + sz, PAGE_SIZE - sz,
+ "bitmap lengths: sec:%d, blk:%d, lun:%d\n",
+ lm->sec_bitmap_len,
+ lm->blk_bitmap_len,
+ lm->lun_bitmap_len);
+ sz += snprintf(page + sz, PAGE_SIZE - sz,
+ "blk_line:%d, sec_line:%d, sec_blk:%d\n",
+ lm->blk_per_line,
+ lm->sec_per_line,
+ geo->sec_per_blk);
+
+ return sz;
+}
+
+#ifdef CONFIG_NVM_DEBUG
+static ssize_t pblk_sysfs_stats_debug(struct pblk *pblk, char *page)
+{
+ return snprintf(page, PAGE_SIZE,
+ "%lu\t%lu\t%lu\t%lu\t%lu\t%lu\t%lu\t%lu\t%lu\t%lu\t%lu\t%lu\t%lu\n",
+ atomic_long_read(&pblk->inflight_writes),
+ atomic_long_read(&pblk->inflight_reads),
+ atomic_long_read(&pblk->req_writes),
+ atomic_long_read(&pblk->nr_flush),
+ atomic_long_read(&pblk->padded_writes),
+ atomic_long_read(&pblk->padded_wb),
+ atomic_long_read(&pblk->sub_writes),
+ atomic_long_read(&pblk->sync_writes),
+ atomic_long_read(&pblk->compl_writes),
+ atomic_long_read(&pblk->recov_writes),
+ atomic_long_read(&pblk->recov_gc_writes),
+ atomic_long_read(&pblk->recov_gc_reads),
+ atomic_long_read(&pblk->sync_reads));
+}
+#endif
+
+static ssize_t pblk_sysfs_rate_store(struct pblk *pblk, const char *page,
+ size_t len)
+{
+ struct pblk_gc *gc = &pblk->gc;
+ size_t c_len;
+ int value;
+
+ c_len = strcspn(page, "\n");
+ if (c_len >= len)
+ return -EINVAL;
+
+ if (kstrtouint(page, 0, &value))
+ return -EINVAL;
+
+ spin_lock(&gc->lock);
+ pblk_rl_set_gc_rsc(&pblk->rl, value);
+ spin_unlock(&gc->lock);
+
+ return len;
+}
+
+static ssize_t pblk_sysfs_gc_force(struct pblk *pblk, const char *page,
+ size_t len)
+{
+ size_t c_len;
+ int force;
+
+ c_len = strcspn(page, "\n");
+ if (c_len >= len)
+ return -EINVAL;
+
+ if (kstrtouint(page, 0, &force))
+ return -EINVAL;
+
+ if (force < 0 || force > 1)
+ return -EINVAL;
+
+ pblk_gc_sysfs_force(pblk, force);
+
+ return len;
+}
+
+static struct attribute sys_write_luns = {
+ .name = "write_luns",
+ .mode = 0444,
+};
+
+static struct attribute sys_rate_limiter_attr = {
+ .name = "rate_limiter",
+ .mode = 0444,
+};
+
+static struct attribute sys_gc_state = {
+ .name = "gc_state",
+ .mode = 0444,
+};
+
+static struct attribute sys_errors_attr = {
+ .name = "errors",
+ .mode = 0444,
+};
+
+static struct attribute sys_rb_attr = {
+ .name = "write_buffer",
+ .mode = 0444,
+};
+
+static struct attribute sys_stats_ppaf_attr = {
+ .name = "ppa_format",
+ .mode = 0444,
+};
+
+static struct attribute sys_lines_attr = {
+ .name = "lines",
+ .mode = 0444,
+};
+
+static struct attribute sys_lines_info_attr = {
+ .name = "lines_info",
+ .mode = 0444,
+};
+
+static struct attribute sys_gc_force = {
+ .name = "gc_force",
+ .mode = 0200,
+};
+
+static struct attribute sys_gc_rl_max = {
+ .name = "gc_rl_max",
+ .mode = 0200,
+};
+
+#ifdef CONFIG_NVM_DEBUG
+static struct attribute sys_stats_debug_attr = {
+ .name = "stats",
+ .mode = 0444,
+};
+#endif
+
+static struct attribute *pblk_attrs[] = {
+ &sys_write_luns,
+ &sys_rate_limiter_attr,
+ &sys_errors_attr,
+ &sys_gc_state,
+ &sys_gc_force,
+ &sys_gc_rl_max,
+ &sys_rb_attr,
+ &sys_stats_ppaf_attr,
+ &sys_lines_attr,
+ &sys_lines_info_attr,
+#ifdef CONFIG_NVM_DEBUG
+ &sys_stats_debug_attr,
+#endif
+ NULL,
+};
+
+static ssize_t pblk_sysfs_show(struct kobject *kobj, struct attribute *attr,
+ char *buf)
+{
+ struct pblk *pblk = container_of(kobj, struct pblk, kobj);
+
+ if (strcmp(attr->name, "rate_limiter") == 0)
+ return pblk_sysfs_rate_limiter(pblk, buf);
+ else if (strcmp(attr->name, "write_luns") == 0)
+ return pblk_sysfs_luns_show(pblk, buf);
+ else if (strcmp(attr->name, "gc_state") == 0)
+ return pblk_sysfs_gc_state_show(pblk, buf);
+ else if (strcmp(attr->name, "errors") == 0)
+ return pblk_sysfs_stats(pblk, buf);
+ else if (strcmp(attr->name, "write_buffer") == 0)
+ return pblk_sysfs_write_buffer(pblk, buf);
+ else if (strcmp(attr->name, "ppa_format") == 0)
+ return pblk_sysfs_ppaf(pblk, buf);
+ else if (strcmp(attr->name, "lines") == 0)
+ return pblk_sysfs_lines(pblk, buf);
+ else if (strcmp(attr->name, "lines_info") == 0)
+ return pblk_sysfs_lines_info(pblk, buf);
+#ifdef CONFIG_NVM_DEBUG
+ else if (strcmp(attr->name, "stats") == 0)
+ return pblk_sysfs_stats_debug(pblk, buf);
+#endif
+ return 0;
+}
+
+static ssize_t pblk_sysfs_store(struct kobject *kobj, struct attribute *attr,
+ const char *buf, size_t len)
+{
+ struct pblk *pblk = container_of(kobj, struct pblk, kobj);
+
+ if (strcmp(attr->name, "gc_rl_max") == 0)
+ return pblk_sysfs_rate_store(pblk, buf, len);
+ else if (strcmp(attr->name, "gc_force") == 0)
+ return pblk_sysfs_gc_force(pblk, buf, len);
+
+ return 0;
+}
+
+static const struct sysfs_ops pblk_sysfs_ops = {
+ .show = pblk_sysfs_show,
+ .store = pblk_sysfs_store,
+};
+
+static struct kobj_type pblk_ktype = {
+ .sysfs_ops = &pblk_sysfs_ops,
+ .default_attrs = pblk_attrs,
+};
+
+int pblk_sysfs_init(struct gendisk *tdisk)
+{
+ struct pblk *pblk = tdisk->private_data;
+ struct device *parent_dev = disk_to_dev(pblk->disk);
+ int ret;
+
+ ret = kobject_init_and_add(&pblk->kobj, &pblk_ktype,
+ kobject_get(&parent_dev->kobj),
+ "%s", "pblk");
+ if (ret) {
+ pr_err("pblk: could not register %s/pblk\n",
+ tdisk->disk_name);
+ return ret;
+ }
+
+ kobject_uevent(&pblk->kobj, KOBJ_ADD);
+ return 0;
+}
+
+void pblk_sysfs_exit(struct gendisk *tdisk)
+{
+ struct pblk *pblk = tdisk->private_data;
+
+ kobject_uevent(&pblk->kobj, KOBJ_REMOVE);
+ kobject_del(&pblk->kobj);
+ kobject_put(&pblk->kobj);
+}
diff --git a/drivers/lightnvm/pblk-write.c b/drivers/lightnvm/pblk-write.c
new file mode 100644
index 0000000..aef6fd7
--- /dev/null
+++ b/drivers/lightnvm/pblk-write.c
@@ -0,0 +1,414 @@
+/*
+ * Copyright (C) 2016 CNEX Labs
+ * Initial release: Javier Gonzalez <javier@cnexlabs.com>
+ * Matias Bjorling <matias@cnexlabs.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * pblk-write.c - pblk's write path from write buffer to media
+ */
+
+#include "pblk.h"
+
+static void pblk_sync_line(struct pblk *pblk, struct pblk_line *line)
+{
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_inc(&pblk->sync_writes);
+#endif
+
+ /* Counter protected by rb sync lock */
+ line->left_ssecs--;
+ if (!line->left_ssecs)
+ pblk_line_run_ws(pblk, line, NULL, pblk_line_close_ws);
+}
+
+static unsigned long pblk_end_w_bio(struct pblk *pblk, struct nvm_rq *rqd,
+ struct pblk_c_ctx *c_ctx)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct bio *original_bio;
+ unsigned long ret;
+ int i;
+
+ for (i = 0; i < c_ctx->nr_valid; i++) {
+ struct pblk_w_ctx *w_ctx;
+ struct ppa_addr p;
+ struct pblk_line *line;
+
+ w_ctx = pblk_rb_w_ctx(&pblk->rwb, c_ctx->sentry + i);
+
+ p = rqd->ppa_list[i];
+ line = &pblk->lines[pblk_dev_ppa_to_line(p)];
+ pblk_sync_line(pblk, line);
+
+ while ((original_bio = bio_list_pop(&w_ctx->bios)))
+ bio_endio(original_bio);
+ }
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_add(c_ctx->nr_valid, &pblk->compl_writes);
+#endif
+
+ ret = pblk_rb_sync_advance(&pblk->rwb, c_ctx->nr_valid);
+
+ if (rqd->meta_list)
+ nvm_dev_dma_free(dev->parent, rqd->meta_list,
+ rqd->dma_meta_list);
+
+ bio_put(rqd->bio);
+ pblk_free_rqd(pblk, rqd, WRITE);
+
+ return ret;
+}
+
+static unsigned long pblk_end_queued_w_bio(struct pblk *pblk,
+ struct nvm_rq *rqd,
+ struct pblk_c_ctx *c_ctx)
+{
+ list_del(&c_ctx->list);
+ return pblk_end_w_bio(pblk, rqd, c_ctx);
+}
+
+static void pblk_complete_write(struct pblk *pblk, struct nvm_rq *rqd,
+ struct pblk_c_ctx *c_ctx)
+{
+ struct pblk_c_ctx *c, *r;
+ unsigned long flags;
+ unsigned long pos;
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_sub(c_ctx->nr_valid, &pblk->inflight_writes);
+#endif
+
+ pblk_up_rq(pblk, rqd->ppa_list, rqd->nr_ppas, c_ctx->lun_bitmap);
+
+ pos = pblk_rb_sync_init(&pblk->rwb, &flags);
+ if (pos == c_ctx->sentry) {
+ pos = pblk_end_w_bio(pblk, rqd, c_ctx);
+
+retry:
+ list_for_each_entry_safe(c, r, &pblk->compl_list, list) {
+ rqd = nvm_rq_from_c_ctx(c);
+ if (c->sentry == pos) {
+ pos = pblk_end_queued_w_bio(pblk, rqd, c);
+ goto retry;
+ }
+ }
+ } else {
+ WARN_ON(nvm_rq_from_c_ctx(c_ctx) != rqd);
+ list_add_tail(&c_ctx->list, &pblk->compl_list);
+ }
+ pblk_rb_sync_end(&pblk->rwb, &flags);
+}
+
+/* When a write fails, we are not sure whether the block has grown bad or a page
+ * range is more susceptible to write errors. If a high number of pages fail, we
+ * assume that the block is bad and we mark it accordingly. In all cases, we
+ * remap and resubmit the failed entries as fast as possible; if a flush is
+ * waiting on a completion, the whole stack would stall otherwise.
+ */
+static void pblk_end_w_fail(struct pblk *pblk, struct nvm_rq *rqd)
+{
+ void *comp_bits = &rqd->ppa_status;
+ struct pblk_c_ctx *c_ctx = nvm_rq_to_pdu(rqd);
+ struct pblk_rec_ctx *recovery;
+ struct ppa_addr *ppa_list = rqd->ppa_list;
+ int nr_ppas = rqd->nr_ppas;
+ unsigned int c_entries;
+ int bit, ret;
+
+ if (unlikely(nr_ppas == 1))
+ ppa_list = &rqd->ppa_addr;
+
+ recovery = mempool_alloc(pblk->rec_pool, GFP_ATOMIC);
+ if (!recovery) {
+ pr_err("pblk: could not allocate recovery context\n");
+ return;
+ }
+ INIT_LIST_HEAD(&recovery->failed);
+
+ bit = -1;
+ while ((bit = find_next_bit(comp_bits, nr_ppas, bit + 1)) < nr_ppas) {
+ struct pblk_rb_entry *entry;
+ struct ppa_addr ppa;
+
+ /* Logic error */
+ if (bit > c_ctx->nr_valid) {
+ WARN_ONCE(1, "pblk: corrupted write request\n");
+ mempool_free(recovery, pblk->rec_pool);
+ goto out;
+ }
+
+ ppa = ppa_list[bit];
+ entry = pblk_rb_sync_scan_entry(&pblk->rwb, &ppa);
+ if (!entry) {
+ pr_err("pblk: could not scan entry on write failure\n");
+ mempool_free(recovery, pblk->rec_pool);
+ goto out;
+ }
+
+ /* The list is filled first and emptied afterwards. No need for
+ * protecting it with a lock
+ */
+ list_add_tail(&entry->index, &recovery->failed);
+ }
+
+ c_entries = find_first_bit(comp_bits, nr_ppas);
+ ret = pblk_recov_setup_rq(pblk, c_ctx, recovery, comp_bits, c_entries);
+ if (ret) {
+ pr_err("pblk: could not recover from write failure\n");
+ mempool_free(recovery, pblk->rec_pool);
+ goto out;
+ }
+
+ INIT_WORK(&recovery->ws_rec, pblk_submit_rec);
+ queue_work(pblk->kw_wq, &recovery->ws_rec);
+
+out:
+ pblk_complete_write(pblk, rqd, c_ctx);
+}
+
+static void pblk_end_io_write(struct nvm_rq *rqd)
+{
+ struct pblk *pblk = rqd->private;
+ struct pblk_c_ctx *c_ctx = nvm_rq_to_pdu(rqd);
+
+ if (rqd->error) {
+ pblk_log_write_err(pblk, rqd);
+ return pblk_end_w_fail(pblk, rqd);
+ }
+#ifdef CONFIG_NVM_DEBUG
+ else
+ WARN_ONCE(rqd->bio->bi_error, "pblk: corrupted write error\n");
+#endif
+
+ pblk_complete_write(pblk, rqd, c_ctx);
+}
+
+static int pblk_alloc_w_rq(struct pblk *pblk, struct nvm_rq *rqd,
+ unsigned int nr_secs)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+
+ /* Setup write request */
+ rqd->opcode = NVM_OP_PWRITE;
+ rqd->nr_ppas = nr_secs;
+ rqd->flags = pblk_set_progr_mode(pblk, WRITE);
+ rqd->private = pblk;
+ rqd->end_io = pblk_end_io_write;
+
+ rqd->meta_list = nvm_dev_dma_alloc(dev->parent, GFP_KERNEL,
+ &rqd->dma_meta_list);
+ if (!rqd->meta_list)
+ return -ENOMEM;
+
+ if (unlikely(nr_secs == 1))
+ return 0;
+
+ rqd->ppa_list = rqd->meta_list + pblk_dma_meta_size;
+ rqd->dma_ppa_list = rqd->dma_meta_list + pblk_dma_meta_size;
+
+ return 0;
+}
+
+static int pblk_setup_w_rq(struct pblk *pblk, struct nvm_rq *rqd,
+ struct pblk_c_ctx *c_ctx)
+{
+ struct pblk_line_meta *lm = &pblk->lm;
+ struct pblk_line *e_line = pblk_line_get_data_next(pblk);
+ struct ppa_addr erase_ppa;
+ unsigned int valid = c_ctx->nr_valid;
+ unsigned int padded = c_ctx->nr_padded;
+ unsigned int nr_secs = valid + padded;
+ unsigned long *lun_bitmap;
+ int ret = 0;
+
+ lun_bitmap = kzalloc(lm->lun_bitmap_len, GFP_KERNEL);
+ if (!lun_bitmap) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ c_ctx->lun_bitmap = lun_bitmap;
+
+ ret = pblk_alloc_w_rq(pblk, rqd, nr_secs);
+ if (ret) {
+ kfree(lun_bitmap);
+ goto out;
+ }
+
+ ppa_set_empty(&erase_ppa);
+ if (likely(!e_line || !atomic_read(&e_line->left_eblks)))
+ pblk_map_rq(pblk, rqd, c_ctx->sentry, lun_bitmap, valid, 0);
+ else
+ pblk_map_erase_rq(pblk, rqd, c_ctx->sentry, lun_bitmap,
+ valid, &erase_ppa);
+
+out:
+ if (unlikely(e_line && !ppa_empty(erase_ppa))) {
+ if (pblk_blk_erase_async(pblk, erase_ppa)) {
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ int bit;
+
+ atomic_inc(&e_line->left_eblks);
+ bit = erase_ppa.g.lun * geo->nr_chnls + erase_ppa.g.ch;
+ WARN_ON(!test_and_clear_bit(bit, e_line->erase_bitmap));
+ up(&pblk->erase_sem);
+ }
+ }
+
+ return ret;
+}
+
+int pblk_setup_w_rec_rq(struct pblk *pblk, struct nvm_rq *rqd,
+ struct pblk_c_ctx *c_ctx)
+{
+ struct pblk_line_meta *lm = &pblk->lm;
+ unsigned long *lun_bitmap;
+ int ret;
+
+ lun_bitmap = kzalloc(lm->lun_bitmap_len, GFP_KERNEL);
+ if (!lun_bitmap)
+ return -ENOMEM;
+
+ c_ctx->lun_bitmap = lun_bitmap;
+
+ ret = pblk_alloc_w_rq(pblk, rqd, rqd->nr_ppas);
+ if (ret)
+ return ret;
+
+ pblk_map_rq(pblk, rqd, c_ctx->sentry, lun_bitmap, c_ctx->nr_valid, 0);
+
+ rqd->ppa_status = (u64)0;
+ rqd->flags = pblk_set_progr_mode(pblk, WRITE);
+
+ return ret;
+}
+
+static int pblk_calc_secs_to_sync(struct pblk *pblk, unsigned int secs_avail,
+ unsigned int secs_to_flush)
+{
+ int secs_to_sync;
+
+ secs_to_sync = pblk_calc_secs(pblk, secs_avail, secs_to_flush);
+
+#ifdef CONFIG_NVM_DEBUG
+ if ((!secs_to_sync && secs_to_flush)
+ || (secs_to_sync < 0)
+ || (secs_to_sync > secs_avail && !secs_to_flush)) {
+ pr_err("pblk: bad sector calculation (a:%d,s:%d,f:%d)\n",
+ secs_avail, secs_to_sync, secs_to_flush);
+ }
+#endif
+
+ return secs_to_sync;
+}
+
+static int pblk_submit_write(struct pblk *pblk)
+{
+ struct bio *bio;
+ struct nvm_rq *rqd;
+ struct pblk_c_ctx *c_ctx;
+ unsigned int pgs_read;
+ unsigned int secs_avail, secs_to_sync, secs_to_com;
+ unsigned int secs_to_flush;
+ unsigned long pos;
+ int err;
+
+ /* If there are no sectors in the cache, flushes (bios without data)
+ * will be cleared on the cache threads
+ */
+ secs_avail = pblk_rb_read_count(&pblk->rwb);
+ if (!secs_avail)
+ return 1;
+
+ secs_to_flush = pblk_rb_sync_point_count(&pblk->rwb);
+ if (!secs_to_flush && secs_avail < pblk->min_write_pgs)
+ return 1;
+
+ rqd = pblk_alloc_rqd(pblk, WRITE);
+ if (IS_ERR(rqd)) {
+ pr_err("pblk: cannot allocate write req.\n");
+ return 1;
+ }
+ c_ctx = nvm_rq_to_pdu(rqd);
+
+ bio = bio_alloc(GFP_KERNEL, pblk->max_write_pgs);
+ if (!bio) {
+ pr_err("pblk: cannot allocate write bio\n");
+ goto fail_free_rqd;
+ }
+ bio->bi_iter.bi_sector = 0; /* internal bio */
+ bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
+ rqd->bio = bio;
+
+ secs_to_sync = pblk_calc_secs_to_sync(pblk, secs_avail, secs_to_flush);
+ if (secs_to_sync > pblk->max_write_pgs) {
+ pr_err("pblk: bad buffer sync calculation\n");
+ goto fail_put_bio;
+ }
+
+ secs_to_com = (secs_to_sync > secs_avail) ? secs_avail : secs_to_sync;
+ pos = pblk_rb_read_commit(&pblk->rwb, secs_to_com);
+
+ pgs_read = pblk_rb_read_to_bio(&pblk->rwb, bio, c_ctx, pos,
+ secs_to_sync, secs_avail);
+ if (!pgs_read) {
+ pr_err("pblk: corrupted write bio\n");
+ goto fail_put_bio;
+ }
+
+ if (c_ctx->nr_padded)
+ if (pblk_bio_add_pages(pblk, bio, GFP_KERNEL, c_ctx->nr_padded))
+ goto fail_put_bio;
+
+ /* Assign lbas to ppas and populate request structure */
+ err = pblk_setup_w_rq(pblk, rqd, c_ctx);
+ if (err) {
+ pr_err("pblk: could not setup write request\n");
+ goto fail_free_bio;
+ }
+
+ err = pblk_submit_io(pblk, rqd);
+ if (err) {
+ pr_err("pblk: I/O submission failed: %d\n", err);
+ goto fail_free_bio;
+ }
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_long_add(secs_to_sync, &pblk->sub_writes);
+#endif
+
+ return 0;
+
+fail_free_bio:
+ if (c_ctx->nr_padded)
+ pblk_bio_free_pages(pblk, bio, secs_to_sync, c_ctx->nr_padded);
+fail_put_bio:
+ bio_put(bio);
+fail_free_rqd:
+ pblk_free_rqd(pblk, rqd, WRITE);
+
+ return 1;
+}
+
+int pblk_write_ts(void *data)
+{
+ struct pblk *pblk = data;
+
+ while (!kthread_should_stop()) {
+ if (!pblk_submit_write(pblk))
+ continue;
+ set_current_state(TASK_INTERRUPTIBLE);
+ io_schedule();
+ }
+
+ return 0;
+}
diff --git a/drivers/lightnvm/pblk.h b/drivers/lightnvm/pblk.h
new file mode 100644
index 0000000..99f3186
--- /dev/null
+++ b/drivers/lightnvm/pblk.h
@@ -0,0 +1,1121 @@
+/*
+ * Copyright (C) 2015 IT University of Copenhagen (rrpc.h)
+ * Copyright (C) 2016 CNEX Labs
+ * Initial release: Matias Bjorling <matias@cnexlabs.com>
+ * Write buffering: Javier Gonzalez <javier@cnexlabs.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License version
+ * 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Implementation of a Physical Block-device target for Open-channel SSDs.
+ *
+ */
+
+#ifndef PBLK_H_
+#define PBLK_H_
+
+#include <linux/blkdev.h>
+#include <linux/blk-mq.h>
+#include <linux/bio.h>
+#include <linux/module.h>
+#include <linux/kthread.h>
+#include <linux/vmalloc.h>
+#include <linux/crc32.h>
+#include <linux/uuid.h>
+
+#include <linux/lightnvm.h>
+
+/* Run only GC if less than 1/X blocks are free */
+#define GC_LIMIT_INVERSE 5
+#define GC_TIME_MSECS 1000
+
+#define PBLK_SECTOR (512)
+#define PBLK_EXPOSED_PAGE_SIZE (4096)
+#define PBLK_MAX_REQ_ADDRS (64)
+#define PBLK_MAX_REQ_ADDRS_PW (6)
+
+#define PBLK_CACHE_NAME_LEN (DISK_NAME_LEN + 16)
+
+#define PBLK_COMMAND_TIMEOUT_MS 30000
+
+/* Max 512 LUNs per device */
+#define PBLK_MAX_LUNS_BITMAP (4)
+
+#define NR_PHY_IN_LOG (PBLK_EXPOSED_PAGE_SIZE / PBLK_SECTOR)
+
+#define pblk_for_each_lun(pblk, rlun, i) \
+ for ((i) = 0, rlun = &(pblk)->luns[0]; \
+ (i) < (pblk)->nr_luns; (i)++, rlun = &(pblk)->luns[(i)])
+
+#define ERASE 2 /* READ = 0, WRITE = 1 */
+
+enum {
+ /* IO Types */
+ PBLK_IOTYPE_USER = 1 << 0,
+ PBLK_IOTYPE_GC = 1 << 1,
+
+ /* Write buffer flags */
+ PBLK_FLUSH_ENTRY = 1 << 2,
+ PBLK_WRITTEN_DATA = 1 << 3,
+ PBLK_SUBMITTED_ENTRY = 1 << 4,
+ PBLK_WRITABLE_ENTRY = 1 << 5,
+};
+
+enum {
+ PBLK_BLK_ST_OPEN = 0x1,
+ PBLK_BLK_ST_CLOSED = 0x2,
+};
+
+/* The number of GC lists and the rate-limiter states go together. This way the
+ * rate-limiter can dictate how much GC is needed based on resource utilization.
+ */
+#define PBLK_NR_GC_LISTS 3
+#define PBLK_MAX_GC_JOBS 32
+
+enum {
+ PBLK_RL_HIGH = 1,
+ PBLK_RL_MID = 2,
+ PBLK_RL_LOW = 3,
+};
+
+struct pblk_sec_meta {
+ u64 reserved;
+ __le64 lba;
+};
+
+#define pblk_dma_meta_size (sizeof(struct pblk_sec_meta) * PBLK_MAX_REQ_ADDRS)
+
+/* write completion context */
+struct pblk_c_ctx {
+ struct list_head list; /* Head for out-of-order completion */
+
+ unsigned long *lun_bitmap; /* Luns used on current request */
+ unsigned int sentry;
+ unsigned int nr_valid;
+ unsigned int nr_padded;
+};
+
+/* Read context */
+struct pblk_r_ctx {
+ struct bio *orig_bio;
+};
+
+/* Recovery context */
+struct pblk_rec_ctx {
+ struct pblk *pblk;
+ struct nvm_rq *rqd;
+ struct list_head failed;
+ struct work_struct ws_rec;
+};
+
+/* Write context */
+struct pblk_w_ctx {
+ struct bio_list bios; /* Original bios - used for completion
+ * in REQ_FUA, REQ_FLUSH case
+ */
+ u64 lba; /* Logic addr. associated with entry */
+ struct ppa_addr ppa; /* Physic addr. associated with entry */
+ int flags; /* Write context flags */
+};
+
+struct pblk_rb_entry {
+ struct ppa_addr cacheline; /* Cacheline for this entry */
+ void *data; /* Pointer to data on this entry */
+ struct pblk_w_ctx w_ctx; /* Context for this entry */
+ struct list_head index; /* List head to enable indexes */
+};
+
+#define EMPTY_ENTRY (~0U)
+
+struct pblk_rb_pages {
+ struct page *pages;
+ int order;
+ struct list_head list;
+};
+
+struct pblk_rb {
+ struct pblk_rb_entry *entries; /* Ring buffer entries */
+ unsigned int mem; /* Write offset - points to next
+ * writable entry in memory
+ */
+ unsigned int subm; /* Read offset - points to last entry
+ * that has been submitted to the media
+ * to be persisted
+ */
+ unsigned int sync; /* Synced - backpointer that signals
+ * the last submitted entry that has
+ * been successfully persisted to media
+ */
+ unsigned int sync_point; /* Sync point - last entry that must be
+ * flushed to the media. Used with
+ * REQ_FLUSH and REQ_FUA
+ */
+ unsigned int l2p_update; /* l2p update point - next entry for
+ * which l2p mapping will be updated to
+ * contain a device ppa address (instead
+ * of a cacheline
+ */
+ unsigned int nr_entries; /* Number of entries in write buffer -
+ * must be a power of two
+ */
+ unsigned int seg_size; /* Size of the data segments being
+ * stored on each entry. Typically this
+ * will be 4KB
+ */
+
+ struct list_head pages; /* List of data pages */
+
+ spinlock_t w_lock; /* Write lock */
+ spinlock_t s_lock; /* Sync lock */
+
+#ifdef CONFIG_NVM_DEBUG
+ atomic_t inflight_sync_point; /* Not served REQ_FLUSH | REQ_FUA */
+#endif
+};
+
+#define PBLK_RECOVERY_SECTORS 16
+
+struct pblk_lun {
+ struct ppa_addr bppa;
+
+ u8 *bb_list; /* Bad block list for LUN. Only used on
+ * bring up. Bad blocks are managed
+ * within lines on run-time.
+ */
+
+ struct semaphore wr_sem;
+};
+
+struct pblk_gc_rq {
+ struct pblk_line *line;
+ void *data;
+ u64 *lba_list;
+ int nr_secs;
+ int secs_to_gc;
+ struct list_head list;
+};
+
+struct pblk_gc {
+ int gc_active;
+ int gc_enabled;
+ int gc_forced;
+ int gc_jobs_active;
+ atomic_t inflight_gc;
+
+ struct task_struct *gc_ts;
+ struct task_struct *gc_writer_ts;
+ struct workqueue_struct *gc_reader_wq;
+ struct timer_list gc_timer;
+
+ int w_entries;
+ struct list_head w_list;
+
+ spinlock_t lock;
+ spinlock_t w_lock;
+};
+
+struct pblk_rl {
+ unsigned int high; /* Upper threshold for rate limiter (free run -
+ * user I/O rate limiter
+ */
+ unsigned int low; /* Lower threshold for rate limiter (user I/O
+ * rate limiter - stall)
+ */
+ unsigned int high_pw; /* High rounded up as a power of 2 */
+
+#define PBLK_USER_HIGH_THRS 2 /* Begin write limit at 50 percent
+ * available blks
+ */
+#define PBLK_USER_LOW_THRS 20 /* Aggressive GC at 5% available blocks */
+
+ int rb_windows_pw; /* Number of rate windows in the write buffer
+ * given as a power-of-2. This guarantees that
+ * when user I/O is being rate limited, there
+ * will be reserved enough space for the GC to
+ * place its payload. A window is of
+ * pblk->max_write_pgs size, which in NVMe is
+ * 64, i.e., 256kb.
+ */
+ int rb_budget; /* Total number of entries available for I/O */
+ int rb_user_max; /* Max buffer entries available for user I/O */
+ atomic_t rb_user_cnt; /* User I/O buffer counter */
+ int rb_gc_max; /* Max buffer entries available for GC I/O */
+ int rb_gc_rsv; /* Reserved buffer entries for GC I/O */
+ int rb_state; /* Rate-limiter current state */
+ atomic_t rb_gc_cnt; /* GC I/O buffer counter */
+
+ int rb_user_active;
+ struct timer_list u_timer;
+
+ unsigned long long nr_secs;
+ unsigned long total_blocks;
+ atomic_t free_blocks;
+};
+
+#define PBLK_LINE_NR_LUN_BITMAP 2
+#define PBLK_LINE_NR_SEC_BITMAP 2
+#define PBLK_LINE_EMPTY (~0U)
+
+enum {
+ /* Line Types */
+ PBLK_LINETYPE_FREE = 0,
+ PBLK_LINETYPE_LOG = 1,
+ PBLK_LINETYPE_DATA = 2,
+
+ /* Line state */
+ PBLK_LINESTATE_FREE = 10,
+ PBLK_LINESTATE_OPEN = 11,
+ PBLK_LINESTATE_CLOSED = 12,
+ PBLK_LINESTATE_GC = 13,
+ PBLK_LINESTATE_BAD = 14,
+ PBLK_LINESTATE_CORRUPT = 15,
+
+ /* GC group */
+ PBLK_LINEGC_NONE = 20,
+ PBLK_LINEGC_EMPTY = 21,
+ PBLK_LINEGC_LOW = 22,
+ PBLK_LINEGC_MID = 23,
+ PBLK_LINEGC_HIGH = 24,
+ PBLK_LINEGC_FULL = 25,
+};
+
+#define PBLK_MAGIC 0x70626c6b /*pblk*/
+
+struct line_header {
+ __le32 crc;
+ __le32 identifier; /* pblk identifier */
+ __u8 uuid[16]; /* instance uuid */
+ __le16 type; /* line type */
+ __le16 version; /* type version */
+ __le32 id; /* line id for current line */
+};
+
+struct line_smeta {
+ struct line_header header;
+
+ __le32 crc; /* Full structure including struct crc */
+ /* Previous line metadata */
+ __le32 prev_id; /* Line id for previous line */
+
+ /* Current line metadata */
+ __le64 seq_nr; /* Sequence number for current line */
+
+ /* Active writers */
+ __le32 window_wr_lun; /* Number of parallel LUNs to write */
+
+ __le32 rsvd[2];
+};
+
+/*
+ * Metadata Layout:
+ * 1. struct pblk_emeta
+ * 2. nr_lbas u64 forming lba list
+ * 3. nr_lines (all) u32 valid sector count (vsc) (~0U: non-alloc line)
+ * 4. nr_luns bits (u64 format) forming line bad block bitmap
+ *
+ * 3. and 4. will be part of FTL log
+ */
+struct line_emeta {
+ struct line_header header;
+
+ __le32 crc; /* Full structure including struct crc */
+
+ /* Previous line metadata */
+ __le32 prev_id; /* Line id for prev line */
+
+ /* Current line metadata */
+ __le64 seq_nr; /* Sequence number for current line */
+
+ /* Active writers */
+ __le32 window_wr_lun; /* Number of parallel LUNs to write */
+
+ /* Bookkeeping for recovery */
+ __le32 next_id; /* Line id for next line */
+ __le64 nr_lbas; /* Number of lbas mapped in line */
+ __le64 nr_valid_lbas; /* Number of valid lbas mapped in line */
+};
+
+struct pblk_line {
+ struct pblk *pblk;
+ unsigned int id; /* Line number corresponds to the
+ * block line
+ */
+ unsigned int seq_nr; /* Unique line sequence number */
+
+ int state; /* PBLK_LINESTATE_X */
+ int type; /* PBLK_LINETYPE_X */
+ int gc_group; /* PBLK_LINEGC_X */
+ struct list_head list; /* Free, GC lists */
+
+ unsigned long *lun_bitmap; /* Bitmap for LUNs mapped in line */
+
+ struct line_smeta *smeta; /* Start metadata */
+ struct line_emeta *emeta; /* End metadata */
+ int meta_line; /* Metadata line id */
+ u64 smeta_ssec; /* Sector where smeta starts */
+ u64 emeta_ssec; /* Sector where emeta starts */
+
+ unsigned int sec_in_line; /* Number of usable secs in line */
+
+ atomic_t blk_in_line; /* Number of good blocks in line */
+ unsigned long *blk_bitmap; /* Bitmap for valid/invalid blocks */
+ unsigned long *erase_bitmap; /* Bitmap for erased blocks */
+
+ unsigned long *map_bitmap; /* Bitmap for mapped sectors in line */
+ unsigned long *invalid_bitmap; /* Bitmap for invalid sectors in line */
+
+ atomic_t left_eblks; /* Blocks left for erasing */
+ atomic_t left_seblks; /* Blocks left for sync erasing */
+
+ int left_msecs; /* Sectors left for mapping */
+ int left_ssecs; /* Sectors left to sync */
+ unsigned int cur_sec; /* Sector map pointer */
+ unsigned int vsc; /* Valid sector count in line */
+
+ struct kref ref; /* Write buffer L2P references */
+
+ spinlock_t lock; /* Necessary for invalid_bitmap only */
+};
+
+#define PBLK_DATA_LINES 4
+
+enum{
+ PBLK_KMALLOC_META = 1,
+ PBLK_VMALLOC_META = 2,
+};
+
+struct pblk_line_metadata {
+ void *meta;
+};
+
+struct pblk_line_mgmt {
+ int nr_lines; /* Total number of full lines */
+ int nr_free_lines; /* Number of full lines in free list */
+
+ /* Free lists - use free_lock */
+ struct list_head free_list; /* Full lines ready to use */
+ struct list_head corrupt_list; /* Full lines corrupted */
+ struct list_head bad_list; /* Full lines bad */
+
+ /* GC lists - use gc_lock */
+ struct list_head *gc_lists[PBLK_NR_GC_LISTS];
+ struct list_head gc_high_list; /* Full lines ready to GC, high isc */
+ struct list_head gc_mid_list; /* Full lines ready to GC, mid isc */
+ struct list_head gc_low_list; /* Full lines ready to GC, low isc */
+
+ struct list_head gc_full_list; /* Full lines ready to GC, no valid */
+ struct list_head gc_empty_list; /* Full lines close, all valid */
+
+ struct pblk_line *log_line; /* Current FTL log line */
+ struct pblk_line *data_line; /* Current data line */
+ struct pblk_line *log_next; /* Next FTL log line */
+ struct pblk_line *data_next; /* Next data line */
+
+ /* Metadata allocation type: VMALLOC | KMALLOC */
+ int smeta_alloc_type;
+ int emeta_alloc_type;
+
+ /* Pre-allocated metadata for data lines */
+ struct pblk_line_metadata sline_meta[PBLK_DATA_LINES];
+ struct pblk_line_metadata eline_meta[PBLK_DATA_LINES];
+ unsigned long meta_bitmap;
+
+ /* Helpers for fast bitmap calculations */
+ unsigned long *bb_template;
+ unsigned long *bb_aux;
+
+ unsigned long d_seq_nr; /* Data line unique sequence number */
+ unsigned long l_seq_nr; /* Log line unique sequence number */
+
+ spinlock_t free_lock;
+ spinlock_t gc_lock;
+};
+
+struct pblk_line_meta {
+ unsigned int smeta_len; /* Total length for smeta */
+ unsigned int smeta_sec; /* Sectors needed for smeta*/
+ unsigned int emeta_len; /* Total length for emeta */
+ unsigned int emeta_sec; /* Sectors needed for emeta*/
+ unsigned int emeta_bb; /* Boundary for bb that affects emeta */
+ unsigned int sec_bitmap_len; /* Length for sector bitmap in line */
+ unsigned int blk_bitmap_len; /* Length for block bitmap in line */
+ unsigned int lun_bitmap_len; /* Length for lun bitmap in line */
+
+ unsigned int blk_per_line; /* Number of blocks in a full line */
+ unsigned int sec_per_line; /* Number of sectors in a line */
+ unsigned int min_blk_line; /* Min. number of good blocks in line */
+
+ unsigned int mid_thrs; /* Threshold for GC mid list */
+ unsigned int high_thrs; /* Threshold for GC high list */
+};
+
+struct pblk_addr_format {
+ u64 ch_mask;
+ u64 lun_mask;
+ u64 pln_mask;
+ u64 blk_mask;
+ u64 pg_mask;
+ u64 sec_mask;
+ u8 ch_offset;
+ u8 lun_offset;
+ u8 pln_offset;
+ u8 blk_offset;
+ u8 pg_offset;
+ u8 sec_offset;
+};
+
+struct pblk {
+ struct nvm_tgt_dev *dev;
+ struct gendisk *disk;
+
+ struct kobject kobj;
+
+ struct pblk_lun *luns;
+
+ struct pblk_line *lines; /* Line array */
+ struct pblk_line_mgmt l_mg; /* Line management */
+ struct pblk_line_meta lm; /* Line metadata */
+
+ int ppaf_bitsize;
+ struct pblk_addr_format ppaf;
+
+ struct pblk_rb rwb;
+
+ int min_write_pgs; /* Minimum amount of pages required by controller */
+ int max_write_pgs; /* Maximum amount of pages supported by controller */
+ int pgs_in_buffer; /* Number of pages that need to be held in buffer to
+ * guarantee successful reads.
+ */
+
+ sector_t capacity; /* Device capacity when bad blocks are subtracted */
+ int over_pct; /* Percentage of device used for over-provisioning */
+
+ /* pblk provisioning values. Used by rate limiter */
+ struct pblk_rl rl;
+
+ struct semaphore erase_sem;
+
+ unsigned char instance_uuid[16];
+#ifdef CONFIG_NVM_DEBUG
+ /* All debug counters apply to 4kb sector I/Os */
+ atomic_long_t inflight_writes; /* Inflight writes (user and gc) */
+ atomic_long_t padded_writes; /* Sectors padded due to flush/fua */
+ atomic_long_t padded_wb; /* Sectors padded in write buffer */
+ atomic_long_t nr_flush; /* Number of flush/fua I/O */
+ atomic_long_t req_writes; /* Sectors stored on write buffer */
+ atomic_long_t sub_writes; /* Sectors submitted from buffer */
+ atomic_long_t sync_writes; /* Sectors synced to media */
+ atomic_long_t compl_writes; /* Sectors completed in write bio */
+ atomic_long_t inflight_reads; /* Inflight sector read requests */
+ atomic_long_t sync_reads; /* Completed sector read requests */
+ atomic_long_t recov_writes; /* Sectors submitted from recovery */
+ atomic_long_t recov_gc_writes; /* Sectors submitted from write GC */
+ atomic_long_t recov_gc_reads; /* Sectors submitted from read GC */
+#endif
+
+ spinlock_t lock;
+
+ atomic_long_t read_failed;
+ atomic_long_t read_empty;
+ atomic_long_t read_high_ecc;
+ atomic_long_t read_failed_gc;
+ atomic_long_t write_failed;
+ atomic_long_t erase_failed;
+
+ struct task_struct *writer_ts;
+
+ /* Simple translation map of logical addresses to physical addresses.
+ * The logical addresses is known by the host system, while the physical
+ * addresses are used when writing to the disk block device.
+ */
+ unsigned char *trans_map;
+ spinlock_t trans_lock;
+
+ struct list_head compl_list;
+
+ mempool_t *page_pool;
+ mempool_t *line_ws_pool;
+ mempool_t *rec_pool;
+ mempool_t *r_rq_pool;
+ mempool_t *w_rq_pool;
+ mempool_t *line_meta_pool;
+
+ struct workqueue_struct *kw_wq;
+ struct timer_list wtimer;
+
+ struct pblk_gc gc;
+};
+
+struct pblk_line_ws {
+ struct pblk *pblk;
+ struct pblk_line *line;
+ void *priv;
+ struct work_struct ws;
+};
+
+#define pblk_r_rq_size (sizeof(struct nvm_rq) + sizeof(struct pblk_r_ctx))
+#define pblk_w_rq_size (sizeof(struct nvm_rq) + sizeof(struct pblk_c_ctx))
+
+/*
+ * pblk ring buffer operations
+ */
+int pblk_rb_init(struct pblk_rb *rb, struct pblk_rb_entry *rb_entry_base,
+ unsigned int power_size, unsigned int power_seg_sz);
+unsigned int pblk_rb_calculate_size(unsigned int nr_entries);
+void *pblk_rb_entries_ref(struct pblk_rb *rb);
+int pblk_rb_may_write_user(struct pblk_rb *rb, struct bio *bio,
+ unsigned int nr_entries, unsigned int *pos);
+int pblk_rb_may_write_gc(struct pblk_rb *rb, unsigned int nr_entries,
+ unsigned int *pos);
+void pblk_rb_write_entry_user(struct pblk_rb *rb, void *data,
+ struct pblk_w_ctx w_ctx, unsigned int pos);
+void pblk_rb_write_entry_gc(struct pblk_rb *rb, void *data,
+ struct pblk_w_ctx w_ctx, struct pblk_line *gc_line,
+ unsigned int pos);
+struct pblk_w_ctx *pblk_rb_w_ctx(struct pblk_rb *rb, unsigned int pos);
+
+void pblk_rb_sync_l2p(struct pblk_rb *rb);
+unsigned int pblk_rb_read_to_bio(struct pblk_rb *rb, struct bio *bio,
+ struct pblk_c_ctx *c_ctx,
+ unsigned int pos,
+ unsigned int nr_entries,
+ unsigned int count);
+unsigned int pblk_rb_read_to_bio_list(struct pblk_rb *rb, struct bio *bio,
+ struct list_head *list,
+ unsigned int max);
+int pblk_rb_copy_to_bio(struct pblk_rb *rb, struct bio *bio, sector_t lba,
+ u64 pos, int bio_iter);
+unsigned int pblk_rb_read_commit(struct pblk_rb *rb, unsigned int entries);
+
+unsigned int pblk_rb_sync_init(struct pblk_rb *rb, unsigned long *flags);
+unsigned int pblk_rb_sync_advance(struct pblk_rb *rb, unsigned int nr_entries);
+struct pblk_rb_entry *pblk_rb_sync_scan_entry(struct pblk_rb *rb,
+ struct ppa_addr *ppa);
+void pblk_rb_sync_end(struct pblk_rb *rb, unsigned long *flags);
+unsigned int pblk_rb_sync_point_count(struct pblk_rb *rb);
+
+unsigned int pblk_rb_read_count(struct pblk_rb *rb);
+unsigned int pblk_rb_wrap_pos(struct pblk_rb *rb, unsigned int pos);
+
+int pblk_rb_tear_down_check(struct pblk_rb *rb);
+int pblk_rb_pos_oob(struct pblk_rb *rb, u64 pos);
+void pblk_rb_data_free(struct pblk_rb *rb);
+ssize_t pblk_rb_sysfs(struct pblk_rb *rb, char *buf);
+
+/*
+ * pblk core
+ */
+struct nvm_rq *pblk_alloc_rqd(struct pblk *pblk, int rw);
+int pblk_setup_w_rec_rq(struct pblk *pblk, struct nvm_rq *rqd,
+ struct pblk_c_ctx *c_ctx);
+void pblk_free_rqd(struct pblk *pblk, struct nvm_rq *rqd, int rw);
+void pblk_flush_writer(struct pblk *pblk);
+struct ppa_addr pblk_get_lba_map(struct pblk *pblk, sector_t lba);
+void pblk_discard(struct pblk *pblk, struct bio *bio);
+void pblk_log_write_err(struct pblk *pblk, struct nvm_rq *rqd);
+void pblk_log_read_err(struct pblk *pblk, struct nvm_rq *rqd);
+int pblk_submit_io(struct pblk *pblk, struct nvm_rq *rqd);
+struct bio *pblk_bio_map_addr(struct pblk *pblk, void *data,
+ unsigned int nr_secs, unsigned int len,
+ gfp_t gfp_mask);
+struct pblk_line *pblk_line_get(struct pblk *pblk);
+struct pblk_line *pblk_line_get_first_data(struct pblk *pblk);
+struct pblk_line *pblk_line_replace_data(struct pblk *pblk);
+int pblk_line_recov_alloc(struct pblk *pblk, struct pblk_line *line);
+void pblk_line_recov_close(struct pblk *pblk, struct pblk_line *line);
+struct pblk_line *pblk_line_get_data(struct pblk *pblk);
+struct pblk_line *pblk_line_get_data_next(struct pblk *pblk);
+int pblk_line_erase(struct pblk *pblk, struct pblk_line *line);
+int pblk_line_is_full(struct pblk_line *line);
+void pblk_line_free(struct pblk *pblk, struct pblk_line *line);
+void pblk_line_close_ws(struct work_struct *work);
+void pblk_line_close(struct pblk *pblk, struct pblk_line *line);
+void pblk_line_mark_bb(struct work_struct *work);
+void pblk_line_run_ws(struct pblk *pblk, struct pblk_line *line, void *priv,
+ void (*work)(struct work_struct *));
+u64 pblk_line_smeta_start(struct pblk *pblk, struct pblk_line *line);
+int pblk_line_read_smeta(struct pblk *pblk, struct pblk_line *line);
+int pblk_line_read_emeta(struct pblk *pblk, struct pblk_line *line);
+int pblk_blk_erase_async(struct pblk *pblk, struct ppa_addr erase_ppa);
+void pblk_line_put(struct kref *ref);
+struct list_head *pblk_line_gc_list(struct pblk *pblk, struct pblk_line *line);
+u64 pblk_alloc_page(struct pblk *pblk, struct pblk_line *line, int nr_secs);
+int pblk_calc_secs(struct pblk *pblk, unsigned long secs_avail,
+ unsigned long secs_to_flush);
+void pblk_down_rq(struct pblk *pblk, struct ppa_addr *ppa_list, int nr_ppas,
+ unsigned long *lun_bitmap);
+void pblk_up_rq(struct pblk *pblk, struct ppa_addr *ppa_list, int nr_ppas,
+ unsigned long *lun_bitmap);
+void pblk_end_bio_sync(struct bio *bio);
+void pblk_end_io_sync(struct nvm_rq *rqd);
+int pblk_bio_add_pages(struct pblk *pblk, struct bio *bio, gfp_t flags,
+ int nr_pages);
+void pblk_map_pad_invalidate(struct pblk *pblk, struct pblk_line *line,
+ u64 paddr);
+void pblk_bio_free_pages(struct pblk *pblk, struct bio *bio, int off,
+ int nr_pages);
+void pblk_map_invalidate(struct pblk *pblk, struct ppa_addr ppa);
+void pblk_update_map(struct pblk *pblk, sector_t lba, struct ppa_addr ppa);
+void pblk_update_map_cache(struct pblk *pblk, sector_t lba,
+ struct ppa_addr ppa);
+void pblk_update_map_dev(struct pblk *pblk, sector_t lba,
+ struct ppa_addr ppa, struct ppa_addr entry_line);
+int pblk_update_map_gc(struct pblk *pblk, sector_t lba, struct ppa_addr ppa,
+ struct pblk_line *gc_line);
+void pblk_lookup_l2p_rand(struct pblk *pblk, struct ppa_addr *ppas,
+ u64 *lba_list, int nr_secs);
+void pblk_lookup_l2p_seq(struct pblk *pblk, struct ppa_addr *ppas,
+ sector_t blba, int nr_secs);
+
+/*
+ * pblk user I/O write path
+ */
+int pblk_write_to_cache(struct pblk *pblk, struct bio *bio,
+ unsigned long flags);
+int pblk_write_gc_to_cache(struct pblk *pblk, void *data, u64 *lba_list,
+ unsigned int nr_entries, unsigned int nr_rec_entries,
+ struct pblk_line *gc_line, unsigned long flags);
+
+/*
+ * pblk map
+ */
+void pblk_map_erase_rq(struct pblk *pblk, struct nvm_rq *rqd,
+ unsigned int sentry, unsigned long *lun_bitmap,
+ unsigned int valid_secs, struct ppa_addr *erase_ppa);
+void pblk_map_rq(struct pblk *pblk, struct nvm_rq *rqd, unsigned int sentry,
+ unsigned long *lun_bitmap, unsigned int valid_secs,
+ unsigned int off);
+
+/*
+ * pblk write thread
+ */
+int pblk_write_ts(void *data);
+void pblk_write_timer_fn(unsigned long data);
+void pblk_write_should_kick(struct pblk *pblk);
+
+/*
+ * pblk read path
+ */
+int pblk_submit_read(struct pblk *pblk, struct bio *bio);
+int pblk_submit_read_gc(struct pblk *pblk, u64 *lba_list, void *data,
+ unsigned int nr_secs, unsigned int *secs_to_gc,
+ struct pblk_line *line);
+/*
+ * pblk recovery
+ */
+void pblk_submit_rec(struct work_struct *work);
+struct pblk_line *pblk_recov_l2p(struct pblk *pblk);
+void pblk_recov_pad(struct pblk *pblk);
+__le64 *pblk_recov_get_lba_list(struct pblk *pblk, struct line_emeta *emeta);
+int pblk_recov_setup_rq(struct pblk *pblk, struct pblk_c_ctx *c_ctx,
+ struct pblk_rec_ctx *recovery, u64 *comp_bits,
+ unsigned int comp);
+
+/*
+ * pblk gc
+ */
+#define PBLK_GC_TRIES 3
+
+int pblk_gc_init(struct pblk *pblk);
+void pblk_gc_exit(struct pblk *pblk);
+void pblk_gc_should_start(struct pblk *pblk);
+void pblk_gc_should_stop(struct pblk *pblk);
+int pblk_gc_status(struct pblk *pblk);
+void pblk_gc_sysfs_state_show(struct pblk *pblk, int *gc_enabled,
+ int *gc_active);
+void pblk_gc_sysfs_force(struct pblk *pblk, int force);
+
+/*
+ * pblk rate limiter
+ */
+void pblk_rl_init(struct pblk_rl *rl, int budget);
+void pblk_rl_free(struct pblk_rl *rl);
+int pblk_rl_gc_thrs(struct pblk_rl *rl);
+unsigned long pblk_rl_nr_free_blks(struct pblk_rl *rl);
+int pblk_rl_user_may_insert(struct pblk_rl *rl, int nr_entries);
+void pblk_rl_user_in(struct pblk_rl *rl, int nr_entries);
+int pblk_rl_gc_may_insert(struct pblk_rl *rl, int nr_entries);
+void pblk_rl_gc_in(struct pblk_rl *rl, int nr_entries);
+void pblk_rl_out(struct pblk_rl *rl, int nr_user, int nr_gc);
+void pblk_rl_set_gc_rsc(struct pblk_rl *rl, int rsv);
+int pblk_rl_sysfs_rate_show(struct pblk_rl *rl);
+void pblk_rl_free_lines_inc(struct pblk_rl *rl, struct pblk_line *line);
+void pblk_rl_free_lines_dec(struct pblk_rl *rl, struct pblk_line *line);
+
+/*
+ * pblk sysfs
+ */
+int pblk_sysfs_init(struct gendisk *tdisk);
+void pblk_sysfs_exit(struct gendisk *tdisk);
+
+static inline void *pblk_malloc(size_t size, int type, gfp_t flags)
+{
+ if (type == PBLK_KMALLOC_META)
+ return kmalloc(size, flags);
+ return vmalloc(size);
+}
+
+static inline void pblk_mfree(void *ptr, int type)
+{
+ if (type == PBLK_KMALLOC_META)
+ kfree(ptr);
+ else
+ vfree(ptr);
+}
+
+static inline struct nvm_rq *nvm_rq_from_c_ctx(void *c_ctx)
+{
+ return c_ctx - sizeof(struct nvm_rq);
+}
+
+static inline void *pblk_line_emeta_to_lbas(struct line_emeta *emeta)
+{
+ return (emeta) + 1;
+}
+
+#define NVM_MEM_PAGE_WRITE (8)
+
+static inline int pblk_pad_distance(struct pblk *pblk)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+
+ return NVM_MEM_PAGE_WRITE * geo->nr_luns * geo->sec_per_pl;
+}
+
+static inline int pblk_dev_ppa_to_line(struct ppa_addr p)
+{
+ return p.g.blk;
+}
+
+static inline int pblk_tgt_ppa_to_line(struct ppa_addr p)
+{
+ return p.g.blk;
+}
+
+static inline int pblk_ppa_to_pos(struct nvm_geo *geo, struct ppa_addr p)
+{
+ return p.g.lun * geo->nr_chnls + p.g.ch;
+}
+
+/* A block within a line corresponds to the lun */
+static inline int pblk_dev_ppa_to_pos(struct nvm_geo *geo, struct ppa_addr p)
+{
+ return p.g.lun * geo->nr_chnls + p.g.ch;
+}
+
+static inline struct ppa_addr pblk_ppa32_to_ppa64(struct pblk *pblk, u32 ppa32)
+{
+ struct ppa_addr ppa64;
+
+ ppa64.ppa = 0;
+
+ if (ppa32 == -1) {
+ ppa64.ppa = ADDR_EMPTY;
+ } else if (ppa32 & (1U << 31)) {
+ ppa64.c.line = ppa32 & ((~0U) >> 1);
+ ppa64.c.is_cached = 1;
+ } else {
+ ppa64.g.blk = (ppa32 & pblk->ppaf.blk_mask) >>
+ pblk->ppaf.blk_offset;
+ ppa64.g.pg = (ppa32 & pblk->ppaf.pg_mask) >>
+ pblk->ppaf.pg_offset;
+ ppa64.g.lun = (ppa32 & pblk->ppaf.lun_mask) >>
+ pblk->ppaf.lun_offset;
+ ppa64.g.ch = (ppa32 & pblk->ppaf.ch_mask) >>
+ pblk->ppaf.ch_offset;
+ ppa64.g.pl = (ppa32 & pblk->ppaf.pln_mask) >>
+ pblk->ppaf.pln_offset;
+ ppa64.g.sec = (ppa32 & pblk->ppaf.sec_mask) >>
+ pblk->ppaf.sec_offset;
+ }
+
+ return ppa64;
+}
+
+static inline struct ppa_addr pblk_trans_map_get(struct pblk *pblk,
+ sector_t lba)
+{
+ struct ppa_addr ppa;
+
+ if (pblk->ppaf_bitsize < 32) {
+ u32 *map = (u32 *)pblk->trans_map;
+
+ ppa = pblk_ppa32_to_ppa64(pblk, map[lba]);
+ } else {
+ struct ppa_addr *map = (struct ppa_addr *)pblk->trans_map;
+
+ ppa = map[lba];
+ }
+
+ return ppa;
+}
+
+static inline u32 pblk_ppa64_to_ppa32(struct pblk *pblk, struct ppa_addr ppa64)
+{
+ u32 ppa32 = 0;
+
+ if (ppa64.ppa == ADDR_EMPTY) {
+ ppa32 = ~0U;
+ } else if (ppa64.c.is_cached) {
+ ppa32 |= ppa64.c.line;
+ ppa32 |= 1U << 31;
+ } else {
+ ppa32 |= ppa64.g.blk << pblk->ppaf.blk_offset;
+ ppa32 |= ppa64.g.pg << pblk->ppaf.pg_offset;
+ ppa32 |= ppa64.g.lun << pblk->ppaf.lun_offset;
+ ppa32 |= ppa64.g.ch << pblk->ppaf.ch_offset;
+ ppa32 |= ppa64.g.pl << pblk->ppaf.pln_offset;
+ ppa32 |= ppa64.g.sec << pblk->ppaf.sec_offset;
+ }
+
+ return ppa32;
+}
+
+static inline void pblk_trans_map_set(struct pblk *pblk, sector_t lba,
+ struct ppa_addr ppa)
+{
+ if (pblk->ppaf_bitsize < 32) {
+ u32 *map = (u32 *)pblk->trans_map;
+
+ map[lba] = pblk_ppa64_to_ppa32(pblk, ppa);
+ } else {
+ u64 *map = (u64 *)pblk->trans_map;
+
+ map[lba] = ppa.ppa;
+ }
+}
+
+static inline u64 pblk_dev_ppa_to_line_addr(struct pblk *pblk,
+ struct ppa_addr p)
+{
+ u64 paddr;
+
+ paddr = 0;
+ paddr |= (u64)p.g.pg << pblk->ppaf.pg_offset;
+ paddr |= (u64)p.g.lun << pblk->ppaf.lun_offset;
+ paddr |= (u64)p.g.ch << pblk->ppaf.ch_offset;
+ paddr |= (u64)p.g.pl << pblk->ppaf.pln_offset;
+ paddr |= (u64)p.g.sec << pblk->ppaf.sec_offset;
+
+ return paddr;
+}
+
+static inline int pblk_ppa_empty(struct ppa_addr ppa_addr)
+{
+ return (ppa_addr.ppa == ADDR_EMPTY);
+}
+
+static inline void pblk_ppa_set_empty(struct ppa_addr *ppa_addr)
+{
+ ppa_addr->ppa = ADDR_EMPTY;
+}
+
+static inline int pblk_addr_in_cache(struct ppa_addr ppa)
+{
+ return (ppa.ppa != ADDR_EMPTY && ppa.c.is_cached);
+}
+
+static inline int pblk_addr_to_cacheline(struct ppa_addr ppa)
+{
+ return ppa.c.line;
+}
+
+static inline struct ppa_addr pblk_cacheline_to_addr(int addr)
+{
+ struct ppa_addr p;
+
+ p.c.line = addr;
+ p.c.is_cached = 1;
+
+ return p;
+}
+
+static inline struct ppa_addr addr_to_gen_ppa(struct pblk *pblk, u64 paddr,
+ u64 line_id)
+{
+ struct ppa_addr ppa;
+
+ ppa.ppa = 0;
+ ppa.g.blk = line_id;
+ ppa.g.pg = (paddr & pblk->ppaf.pg_mask) >> pblk->ppaf.pg_offset;
+ ppa.g.lun = (paddr & pblk->ppaf.lun_mask) >> pblk->ppaf.lun_offset;
+ ppa.g.ch = (paddr & pblk->ppaf.ch_mask) >> pblk->ppaf.ch_offset;
+ ppa.g.pl = (paddr & pblk->ppaf.pln_mask) >> pblk->ppaf.pln_offset;
+ ppa.g.sec = (paddr & pblk->ppaf.sec_mask) >> pblk->ppaf.sec_offset;
+
+ return ppa;
+}
+
+static inline struct ppa_addr addr_to_pblk_ppa(struct pblk *pblk, u64 paddr,
+ u64 line_id)
+{
+ struct ppa_addr ppa;
+
+ ppa = addr_to_gen_ppa(pblk, paddr, line_id);
+
+ return ppa;
+}
+
+static inline u32 pblk_calc_meta_header_crc(struct pblk *pblk,
+ struct line_smeta *smeta)
+{
+ u32 crc = ~(u32)0;
+
+ crc = crc32_le(crc, (unsigned char *)smeta + sizeof(crc),
+ sizeof(struct line_header) - sizeof(crc));
+
+ return crc;
+}
+
+static inline u32 pblk_calc_smeta_crc(struct pblk *pblk,
+ struct line_smeta *smeta)
+{
+ struct pblk_line_meta *lm = &pblk->lm;
+ u32 crc = ~(u32)0;
+
+ crc = crc32_le(crc, (unsigned char *)smeta +
+ sizeof(struct line_header) + sizeof(crc),
+ lm->smeta_len -
+ sizeof(struct line_header) - sizeof(crc));
+
+ return crc;
+}
+
+static inline u32 pblk_calc_emeta_crc(struct pblk *pblk,
+ struct line_emeta *emeta)
+{
+ struct pblk_line_meta *lm = &pblk->lm;
+ u32 crc = ~(u32)0;
+
+ crc = crc32_le(crc, (unsigned char *)emeta +
+ sizeof(struct line_header) + sizeof(crc),
+ lm->emeta_len -
+ sizeof(struct line_header) - sizeof(crc));
+
+ return crc;
+}
+
+static inline int pblk_set_progr_mode(struct pblk *pblk, int type)
+{
+ struct nvm_tgt_dev *dev = pblk->dev;
+ struct nvm_geo *geo = &dev->geo;
+ int flags;
+
+ flags = geo->plane_mode >> 1;
+
+ if (type == WRITE)
+ flags |= NVM_IO_SCRAMBLE_ENABLE;
+
+ return flags;
+}
+
+static inline int pblk_set_read_mode(struct pblk *pblk)
+{
+ return NVM_IO_SNGL_ACCESS | NVM_IO_SUSPEND | NVM_IO_SCRAMBLE_ENABLE;
+}
+
+#ifdef CONFIG_NVM_DEBUG
+static inline void print_ppa(struct ppa_addr *p, char *msg, int error)
+{
+ if (p->c.is_cached) {
+ pr_err("ppa: (%s: %x) cache line: %llu\n",
+ msg, error, (u64)p->c.line);
+ } else {
+ pr_err("ppa: (%s: %x):ch:%d,lun:%d,blk:%d,pg:%d,pl:%d,sec:%d\n",
+ msg, error,
+ p->g.ch, p->g.lun, p->g.blk,
+ p->g.pg, p->g.pl, p->g.sec);
+ }
+}
+
+static inline void pblk_print_failed_rqd(struct pblk *pblk, struct nvm_rq *rqd,
+ int error)
+{
+ int bit = -1;
+
+ if (rqd->nr_ppas == 1) {
+ print_ppa(&rqd->ppa_addr, "rqd", error);
+ return;
+ }
+
+ while ((bit = find_next_bit((void *)&rqd->ppa_status, rqd->nr_ppas,
+ bit + 1)) < rqd->nr_ppas) {
+ print_ppa(&rqd->ppa_list[bit], "rqd", error);
+ }
+
+ pr_err("error:%d, ppa_status:%llx\n", error, rqd->ppa_status);
+}
+#endif
+
+static inline int pblk_boundary_ppa_checks(struct nvm_tgt_dev *tgt_dev,
+ struct ppa_addr *ppas, int nr_ppas)
+{
+ struct nvm_geo *geo = &tgt_dev->geo;
+ struct ppa_addr *ppa;
+ int i;
+
+ for (i = 0; i < nr_ppas; i++) {
+ ppa = &ppas[i];
+
+ if (!ppa->c.is_cached &&
+ ppa->g.ch < geo->nr_chnls &&
+ ppa->g.lun < geo->luns_per_chnl &&
+ ppa->g.pl < geo->nr_planes &&
+ ppa->g.blk < geo->blks_per_lun &&
+ ppa->g.pg < geo->pgs_per_blk &&
+ ppa->g.sec < geo->sec_per_pg)
+ continue;
+
+#ifdef CONFIG_NVM_DEBUG
+ print_ppa(ppa, "boundary", i);
+#endif
+ return 1;
+ }
+ return 0;
+}
+
+static inline int pblk_boundary_paddr_checks(struct pblk *pblk, u64 paddr)
+{
+ struct pblk_line_meta *lm = &pblk->lm;
+
+ if (paddr > lm->sec_per_line)
+ return 1;
+
+ return 0;
+}
+
+static inline unsigned int pblk_get_bi_idx(struct bio *bio)
+{
+ return bio->bi_iter.bi_idx;
+}
+
+static inline sector_t pblk_get_lba(struct bio *bio)
+{
+ return bio->bi_iter.bi_sector / NR_PHY_IN_LOG;
+}
+
+static inline unsigned int pblk_get_secs(struct bio *bio)
+{
+ return bio->bi_iter.bi_size / PBLK_EXPOSED_PAGE_SIZE;
+}
+
+static inline sector_t pblk_get_sector(sector_t lba)
+{
+ return lba * NR_PHY_IN_LOG;
+}
+
+static inline void pblk_setup_uuid(struct pblk *pblk)
+{
+ uuid_le uuid;
+
+ uuid_le_gen(&uuid);
+ memcpy(pblk->instance_uuid, uuid.b, 16);
+}
+#endif /* PBLK_H_ */
diff --git a/drivers/lightnvm/rrpc.c b/drivers/lightnvm/rrpc.c
index e00b1d7..cf0e28a 100644
--- a/drivers/lightnvm/rrpc.c
+++ b/drivers/lightnvm/rrpc.c
@@ -318,10 +318,6 @@
}
page = mempool_alloc(rrpc->page_pool, GFP_NOIO);
- if (!page) {
- bio_put(bio);
- return -ENOMEM;
- }
while ((slot = find_first_zero_bit(rblk->invalid_pages,
nr_sec_per_blk)) < nr_sec_per_blk) {
@@ -414,7 +410,6 @@
struct rrpc *rrpc = gcb->rrpc;
struct rrpc_block *rblk = gcb->rblk;
struct rrpc_lun *rlun = rblk->rlun;
- struct nvm_tgt_dev *dev = rrpc->dev;
struct ppa_addr ppa;
mempool_free(gcb, rrpc->gcb_pool);
@@ -430,7 +425,7 @@
ppa.g.lun = rlun->bppa.g.lun;
ppa.g.blk = rblk->id;
- if (nvm_erase_blk(dev, &ppa, 0))
+ if (nvm_erase_sync(rrpc->dev, &ppa, 1))
goto put_back;
rrpc_put_blk(rrpc, rblk);
@@ -822,7 +817,7 @@
for (i = 0; i < npages; i++) {
/* We assume that mapping occurs at 4KB granularity */
- BUG_ON(!(laddr + i >= 0 && laddr + i < rrpc->nr_sects));
+ BUG_ON(!(laddr + i < rrpc->nr_sects));
gp = &rrpc->trans_map[laddr + i];
if (gp->rblk) {
@@ -851,7 +846,7 @@
if (!is_gc && rrpc_lock_rq(rrpc, bio, rqd))
return NVM_IO_REQUEUE;
- BUG_ON(!(laddr >= 0 && laddr < rrpc->nr_sects));
+ BUG_ON(!(laddr < rrpc->nr_sects));
gp = &rrpc->trans_map[laddr];
if (gp->rblk) {
@@ -1007,11 +1002,6 @@
}
rqd = mempool_alloc(rrpc->rq_pool, GFP_KERNEL);
- if (!rqd) {
- pr_err_ratelimited("rrpc: not able to queue bio.");
- bio_io_error(bio);
- return BLK_QC_T_NONE;
- }
memset(rqd, 0, sizeof(struct nvm_rq));
err = rrpc_submit_io(rrpc, bio, rqd, NVM_IOTYPE_NONE);
@@ -1275,8 +1265,10 @@
}
nr_blks = nvm_bb_tbl_fold(dev->parent, blks, nr_blks);
- if (nr_blks < 0)
- return nr_blks;
+ if (nr_blks < 0) {
+ ret = nr_blks;
+ goto out;
+ }
for (i = 0; i < nr_blks; i++) {
if (blks[i] == NVM_BLK_T_FREE)
@@ -1514,7 +1506,8 @@
static struct nvm_tgt_type tt_rrpc;
-static void *rrpc_init(struct nvm_tgt_dev *dev, struct gendisk *tdisk)
+static void *rrpc_init(struct nvm_tgt_dev *dev, struct gendisk *tdisk,
+ int flags)
{
struct request_queue *bqueue = dev->q;
struct request_queue *tqueue = tdisk->queue;
diff --git a/drivers/macintosh/via-macii.c b/drivers/macintosh/via-macii.c
index 3725f08..415c145 100644
--- a/drivers/macintosh/via-macii.c
+++ b/drivers/macintosh/via-macii.c
@@ -12,7 +12,7 @@
* 1999-08-02 (jmt) - Initial rewrite for Unified ADB.
* 2000-03-29 Tony Mantler <tonym@mac.linux-m68k.org>
* - Big overhaul, should actually work now.
- * 2006-12-31 Finn Thain <fthain@telegraphics.com.au> - Another overhaul.
+ * 2006-12-31 Finn Thain - Another overhaul.
*
* Suggested reading:
* Inside Macintosh, ch. 5 ADB Manager
diff --git a/drivers/mailbox/Kconfig b/drivers/mailbox/Kconfig
index ceff415..ee1a3d9 100644
--- a/drivers/mailbox/Kconfig
+++ b/drivers/mailbox/Kconfig
@@ -144,12 +144,22 @@
want to use the APM X-Gene SLIMpro IPCM support.
config BCM_PDC_MBOX
- tristate "Broadcom PDC Mailbox"
- depends on ARM64 || COMPILE_TEST
+ tristate "Broadcom FlexSparx DMA Mailbox"
+ depends on ARCH_BCM_IPROC || COMPILE_TEST
depends on HAS_DMA
+ help
+ Mailbox implementation for the Broadcom FlexSparx DMA ring manager,
+ which provides access to various offload engines on Broadcom
+ SoCs, including FA2/FA+ on Northstar Plus and PDC on Northstar 2.
+
+config BCM_FLEXRM_MBOX
+ tristate "Broadcom FlexRM Mailbox"
+ depends on ARM64
+ depends on HAS_DMA
+ select GENERIC_MSI_IRQ_DOMAIN
default ARCH_BCM_IPROC
help
- Mailbox implementation for the Broadcom PDC ring manager,
+ Mailbox implementation of the Broadcom FlexRM ring manager,
which provides access to various offload engines on Broadcom
- SoCs. Say Y here if you want to use the Broadcom PDC.
+ SoCs. Say Y here if you want to use the Broadcom FlexRM.
endif
diff --git a/drivers/mailbox/Makefile b/drivers/mailbox/Makefile
index 7dde4f6..e2bcb03 100644
--- a/drivers/mailbox/Makefile
+++ b/drivers/mailbox/Makefile
@@ -30,4 +30,6 @@
obj-$(CONFIG_BCM_PDC_MBOX) += bcm-pdc-mailbox.o
+obj-$(CONFIG_BCM_FLEXRM_MBOX) += bcm-flexrm-mailbox.o
+
obj-$(CONFIG_TEGRA_HSP_MBOX) += tegra-hsp.o
diff --git a/drivers/mailbox/bcm-flexrm-mailbox.c b/drivers/mailbox/bcm-flexrm-mailbox.c
new file mode 100644
index 0000000..da67882
--- /dev/null
+++ b/drivers/mailbox/bcm-flexrm-mailbox.c
@@ -0,0 +1,1595 @@
+/* Broadcom FlexRM Mailbox Driver
+ *
+ * Copyright (C) 2017 Broadcom
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * Each Broadcom FlexSparx4 offload engine is implemented as an
+ * extension to Broadcom FlexRM ring manager. The FlexRM ring
+ * manager provides a set of rings which can be used to submit
+ * work to a FlexSparx4 offload engine.
+ *
+ * This driver creates a mailbox controller using a set of FlexRM
+ * rings where each mailbox channel represents a separate FlexRM ring.
+ */
+
+#include <asm/barrier.h>
+#include <asm/byteorder.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmapool.h>
+#include <linux/err.h>
+#include <linux/idr.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/mailbox_controller.h>
+#include <linux/mailbox_client.h>
+#include <linux/mailbox/brcm-message.h>
+#include <linux/module.h>
+#include <linux/msi.h>
+#include <linux/of_address.h>
+#include <linux/of_irq.h>
+#include <linux/platform_device.h>
+#include <linux/spinlock.h>
+
+/* ====== FlexRM register defines ===== */
+
+/* FlexRM configuration */
+#define RING_REGS_SIZE 0x10000
+#define RING_DESC_SIZE 8
+#define RING_DESC_INDEX(offset) \
+ ((offset) / RING_DESC_SIZE)
+#define RING_DESC_OFFSET(index) \
+ ((index) * RING_DESC_SIZE)
+#define RING_MAX_REQ_COUNT 1024
+#define RING_BD_ALIGN_ORDER 12
+#define RING_BD_ALIGN_CHECK(addr) \
+ (!((addr) & ((0x1 << RING_BD_ALIGN_ORDER) - 1)))
+#define RING_BD_TOGGLE_INVALID(offset) \
+ (((offset) >> RING_BD_ALIGN_ORDER) & 0x1)
+#define RING_BD_TOGGLE_VALID(offset) \
+ (!RING_BD_TOGGLE_INVALID(offset))
+#define RING_BD_DESC_PER_REQ 32
+#define RING_BD_DESC_COUNT \
+ (RING_MAX_REQ_COUNT * RING_BD_DESC_PER_REQ)
+#define RING_BD_SIZE \
+ (RING_BD_DESC_COUNT * RING_DESC_SIZE)
+#define RING_CMPL_ALIGN_ORDER 13
+#define RING_CMPL_DESC_COUNT RING_MAX_REQ_COUNT
+#define RING_CMPL_SIZE \
+ (RING_CMPL_DESC_COUNT * RING_DESC_SIZE)
+#define RING_VER_MAGIC 0x76303031
+
+/* Per-Ring register offsets */
+#define RING_VER 0x000
+#define RING_BD_START_ADDR 0x004
+#define RING_BD_READ_PTR 0x008
+#define RING_BD_WRITE_PTR 0x00c
+#define RING_BD_READ_PTR_DDR_LS 0x010
+#define RING_BD_READ_PTR_DDR_MS 0x014
+#define RING_CMPL_START_ADDR 0x018
+#define RING_CMPL_WRITE_PTR 0x01c
+#define RING_NUM_REQ_RECV_LS 0x020
+#define RING_NUM_REQ_RECV_MS 0x024
+#define RING_NUM_REQ_TRANS_LS 0x028
+#define RING_NUM_REQ_TRANS_MS 0x02c
+#define RING_NUM_REQ_OUTSTAND 0x030
+#define RING_CONTROL 0x034
+#define RING_FLUSH_DONE 0x038
+#define RING_MSI_ADDR_LS 0x03c
+#define RING_MSI_ADDR_MS 0x040
+#define RING_MSI_CONTROL 0x048
+#define RING_BD_READ_PTR_DDR_CONTROL 0x04c
+#define RING_MSI_DATA_VALUE 0x064
+
+/* Register RING_BD_START_ADDR fields */
+#define BD_LAST_UPDATE_HW_SHIFT 28
+#define BD_LAST_UPDATE_HW_MASK 0x1
+#define BD_START_ADDR_VALUE(pa) \
+ ((u32)((((dma_addr_t)(pa)) >> RING_BD_ALIGN_ORDER) & 0x0fffffff))
+#define BD_START_ADDR_DECODE(val) \
+ ((dma_addr_t)((val) & 0x0fffffff) << RING_BD_ALIGN_ORDER)
+
+/* Register RING_CMPL_START_ADDR fields */
+#define CMPL_START_ADDR_VALUE(pa) \
+ ((u32)((((u64)(pa)) >> RING_CMPL_ALIGN_ORDER) & 0x03ffffff))
+
+/* Register RING_CONTROL fields */
+#define CONTROL_MASK_DISABLE_CONTROL 12
+#define CONTROL_FLUSH_SHIFT 5
+#define CONTROL_ACTIVE_SHIFT 4
+#define CONTROL_RATE_ADAPT_MASK 0xf
+#define CONTROL_RATE_DYNAMIC 0x0
+#define CONTROL_RATE_FAST 0x8
+#define CONTROL_RATE_MEDIUM 0x9
+#define CONTROL_RATE_SLOW 0xa
+#define CONTROL_RATE_IDLE 0xb
+
+/* Register RING_FLUSH_DONE fields */
+#define FLUSH_DONE_MASK 0x1
+
+/* Register RING_MSI_CONTROL fields */
+#define MSI_TIMER_VAL_SHIFT 16
+#define MSI_TIMER_VAL_MASK 0xffff
+#define MSI_ENABLE_SHIFT 15
+#define MSI_ENABLE_MASK 0x1
+#define MSI_COUNT_SHIFT 0
+#define MSI_COUNT_MASK 0x3ff
+
+/* Register RING_BD_READ_PTR_DDR_CONTROL fields */
+#define BD_READ_PTR_DDR_TIMER_VAL_SHIFT 16
+#define BD_READ_PTR_DDR_TIMER_VAL_MASK 0xffff
+#define BD_READ_PTR_DDR_ENABLE_SHIFT 15
+#define BD_READ_PTR_DDR_ENABLE_MASK 0x1
+
+/* ====== FlexRM ring descriptor defines ===== */
+
+/* Completion descriptor format */
+#define CMPL_OPAQUE_SHIFT 0
+#define CMPL_OPAQUE_MASK 0xffff
+#define CMPL_ENGINE_STATUS_SHIFT 16
+#define CMPL_ENGINE_STATUS_MASK 0xffff
+#define CMPL_DME_STATUS_SHIFT 32
+#define CMPL_DME_STATUS_MASK 0xffff
+#define CMPL_RM_STATUS_SHIFT 48
+#define CMPL_RM_STATUS_MASK 0xffff
+
+/* Completion DME status code */
+#define DME_STATUS_MEM_COR_ERR BIT(0)
+#define DME_STATUS_MEM_UCOR_ERR BIT(1)
+#define DME_STATUS_FIFO_UNDERFLOW BIT(2)
+#define DME_STATUS_FIFO_OVERFLOW BIT(3)
+#define DME_STATUS_RRESP_ERR BIT(4)
+#define DME_STATUS_BRESP_ERR BIT(5)
+#define DME_STATUS_ERROR_MASK (DME_STATUS_MEM_COR_ERR | \
+ DME_STATUS_MEM_UCOR_ERR | \
+ DME_STATUS_FIFO_UNDERFLOW | \
+ DME_STATUS_FIFO_OVERFLOW | \
+ DME_STATUS_RRESP_ERR | \
+ DME_STATUS_BRESP_ERR)
+
+/* Completion RM status code */
+#define RM_STATUS_CODE_SHIFT 0
+#define RM_STATUS_CODE_MASK 0x3ff
+#define RM_STATUS_CODE_GOOD 0x0
+#define RM_STATUS_CODE_AE_TIMEOUT 0x3ff
+
+/* General descriptor format */
+#define DESC_TYPE_SHIFT 60
+#define DESC_TYPE_MASK 0xf
+#define DESC_PAYLOAD_SHIFT 0
+#define DESC_PAYLOAD_MASK 0x0fffffffffffffff
+
+/* Null descriptor format */
+#define NULL_TYPE 0
+#define NULL_TOGGLE_SHIFT 58
+#define NULL_TOGGLE_MASK 0x1
+
+/* Header descriptor format */
+#define HEADER_TYPE 1
+#define HEADER_TOGGLE_SHIFT 58
+#define HEADER_TOGGLE_MASK 0x1
+#define HEADER_ENDPKT_SHIFT 57
+#define HEADER_ENDPKT_MASK 0x1
+#define HEADER_STARTPKT_SHIFT 56
+#define HEADER_STARTPKT_MASK 0x1
+#define HEADER_BDCOUNT_SHIFT 36
+#define HEADER_BDCOUNT_MASK 0x1f
+#define HEADER_BDCOUNT_MAX HEADER_BDCOUNT_MASK
+#define HEADER_FLAGS_SHIFT 16
+#define HEADER_FLAGS_MASK 0xffff
+#define HEADER_OPAQUE_SHIFT 0
+#define HEADER_OPAQUE_MASK 0xffff
+
+/* Source (SRC) descriptor format */
+#define SRC_TYPE 2
+#define SRC_LENGTH_SHIFT 44
+#define SRC_LENGTH_MASK 0xffff
+#define SRC_ADDR_SHIFT 0
+#define SRC_ADDR_MASK 0x00000fffffffffff
+
+/* Destination (DST) descriptor format */
+#define DST_TYPE 3
+#define DST_LENGTH_SHIFT 44
+#define DST_LENGTH_MASK 0xffff
+#define DST_ADDR_SHIFT 0
+#define DST_ADDR_MASK 0x00000fffffffffff
+
+/* Immediate (IMM) descriptor format */
+#define IMM_TYPE 4
+#define IMM_DATA_SHIFT 0
+#define IMM_DATA_MASK 0x0fffffffffffffff
+
+/* Next pointer (NPTR) descriptor format */
+#define NPTR_TYPE 5
+#define NPTR_TOGGLE_SHIFT 58
+#define NPTR_TOGGLE_MASK 0x1
+#define NPTR_ADDR_SHIFT 0
+#define NPTR_ADDR_MASK 0x00000fffffffffff
+
+/* Mega source (MSRC) descriptor format */
+#define MSRC_TYPE 6
+#define MSRC_LENGTH_SHIFT 44
+#define MSRC_LENGTH_MASK 0xffff
+#define MSRC_ADDR_SHIFT 0
+#define MSRC_ADDR_MASK 0x00000fffffffffff
+
+/* Mega destination (MDST) descriptor format */
+#define MDST_TYPE 7
+#define MDST_LENGTH_SHIFT 44
+#define MDST_LENGTH_MASK 0xffff
+#define MDST_ADDR_SHIFT 0
+#define MDST_ADDR_MASK 0x00000fffffffffff
+
+/* Source with tlast (SRCT) descriptor format */
+#define SRCT_TYPE 8
+#define SRCT_LENGTH_SHIFT 44
+#define SRCT_LENGTH_MASK 0xffff
+#define SRCT_ADDR_SHIFT 0
+#define SRCT_ADDR_MASK 0x00000fffffffffff
+
+/* Destination with tlast (DSTT) descriptor format */
+#define DSTT_TYPE 9
+#define DSTT_LENGTH_SHIFT 44
+#define DSTT_LENGTH_MASK 0xffff
+#define DSTT_ADDR_SHIFT 0
+#define DSTT_ADDR_MASK 0x00000fffffffffff
+
+/* Immediate with tlast (IMMT) descriptor format */
+#define IMMT_TYPE 10
+#define IMMT_DATA_SHIFT 0
+#define IMMT_DATA_MASK 0x0fffffffffffffff
+
+/* Descriptor helper macros */
+#define DESC_DEC(_d, _s, _m) (((_d) >> (_s)) & (_m))
+#define DESC_ENC(_d, _v, _s, _m) \
+ do { \
+ (_d) &= ~((u64)(_m) << (_s)); \
+ (_d) |= (((u64)(_v) & (_m)) << (_s)); \
+ } while (0)
+
+/* ====== FlexRM data structures ===== */
+
+struct flexrm_ring {
+ /* Unprotected members */
+ int num;
+ struct flexrm_mbox *mbox;
+ void __iomem *regs;
+ bool irq_requested;
+ unsigned int irq;
+ unsigned int msi_timer_val;
+ unsigned int msi_count_threshold;
+ struct ida requests_ida;
+ struct brcm_message *requests[RING_MAX_REQ_COUNT];
+ void *bd_base;
+ dma_addr_t bd_dma_base;
+ u32 bd_write_offset;
+ void *cmpl_base;
+ dma_addr_t cmpl_dma_base;
+ /* Protected members */
+ spinlock_t lock;
+ struct brcm_message *last_pending_msg;
+ u32 cmpl_read_offset;
+};
+
+struct flexrm_mbox {
+ struct device *dev;
+ void __iomem *regs;
+ u32 num_rings;
+ struct flexrm_ring *rings;
+ struct dma_pool *bd_pool;
+ struct dma_pool *cmpl_pool;
+ struct mbox_controller controller;
+};
+
+/* ====== FlexRM ring descriptor helper routines ===== */
+
+static u64 flexrm_read_desc(void *desc_ptr)
+{
+ return le64_to_cpu(*((u64 *)desc_ptr));
+}
+
+static void flexrm_write_desc(void *desc_ptr, u64 desc)
+{
+ *((u64 *)desc_ptr) = cpu_to_le64(desc);
+}
+
+static u32 flexrm_cmpl_desc_to_reqid(u64 cmpl_desc)
+{
+ return (u32)(cmpl_desc & CMPL_OPAQUE_MASK);
+}
+
+static int flexrm_cmpl_desc_to_error(u64 cmpl_desc)
+{
+ u32 status;
+
+ status = DESC_DEC(cmpl_desc, CMPL_DME_STATUS_SHIFT,
+ CMPL_DME_STATUS_MASK);
+ if (status & DME_STATUS_ERROR_MASK)
+ return -EIO;
+
+ status = DESC_DEC(cmpl_desc, CMPL_RM_STATUS_SHIFT,
+ CMPL_RM_STATUS_MASK);
+ status &= RM_STATUS_CODE_MASK;
+ if (status == RM_STATUS_CODE_AE_TIMEOUT)
+ return -ETIMEDOUT;
+
+ return 0;
+}
+
+static bool flexrm_is_next_table_desc(void *desc_ptr)
+{
+ u64 desc = flexrm_read_desc(desc_ptr);
+ u32 type = DESC_DEC(desc, DESC_TYPE_SHIFT, DESC_TYPE_MASK);
+
+ return (type == NPTR_TYPE) ? true : false;
+}
+
+static u64 flexrm_next_table_desc(u32 toggle, dma_addr_t next_addr)
+{
+ u64 desc = 0;
+
+ DESC_ENC(desc, NPTR_TYPE, DESC_TYPE_SHIFT, DESC_TYPE_MASK);
+ DESC_ENC(desc, toggle, NPTR_TOGGLE_SHIFT, NPTR_TOGGLE_MASK);
+ DESC_ENC(desc, next_addr, NPTR_ADDR_SHIFT, NPTR_ADDR_MASK);
+
+ return desc;
+}
+
+static u64 flexrm_null_desc(u32 toggle)
+{
+ u64 desc = 0;
+
+ DESC_ENC(desc, NULL_TYPE, DESC_TYPE_SHIFT, DESC_TYPE_MASK);
+ DESC_ENC(desc, toggle, NULL_TOGGLE_SHIFT, NULL_TOGGLE_MASK);
+
+ return desc;
+}
+
+static u32 flexrm_estimate_header_desc_count(u32 nhcnt)
+{
+ u32 hcnt = nhcnt / HEADER_BDCOUNT_MAX;
+
+ if (!(nhcnt % HEADER_BDCOUNT_MAX))
+ hcnt += 1;
+
+ return hcnt;
+}
+
+static void flexrm_flip_header_toogle(void *desc_ptr)
+{
+ u64 desc = flexrm_read_desc(desc_ptr);
+
+ if (desc & ((u64)0x1 << HEADER_TOGGLE_SHIFT))
+ desc &= ~((u64)0x1 << HEADER_TOGGLE_SHIFT);
+ else
+ desc |= ((u64)0x1 << HEADER_TOGGLE_SHIFT);
+
+ flexrm_write_desc(desc_ptr, desc);
+}
+
+static u64 flexrm_header_desc(u32 toggle, u32 startpkt, u32 endpkt,
+ u32 bdcount, u32 flags, u32 opaque)
+{
+ u64 desc = 0;
+
+ DESC_ENC(desc, HEADER_TYPE, DESC_TYPE_SHIFT, DESC_TYPE_MASK);
+ DESC_ENC(desc, toggle, HEADER_TOGGLE_SHIFT, HEADER_TOGGLE_MASK);
+ DESC_ENC(desc, startpkt, HEADER_STARTPKT_SHIFT, HEADER_STARTPKT_MASK);
+ DESC_ENC(desc, endpkt, HEADER_ENDPKT_SHIFT, HEADER_ENDPKT_MASK);
+ DESC_ENC(desc, bdcount, HEADER_BDCOUNT_SHIFT, HEADER_BDCOUNT_MASK);
+ DESC_ENC(desc, flags, HEADER_FLAGS_SHIFT, HEADER_FLAGS_MASK);
+ DESC_ENC(desc, opaque, HEADER_OPAQUE_SHIFT, HEADER_OPAQUE_MASK);
+
+ return desc;
+}
+
+static void flexrm_enqueue_desc(u32 nhpos, u32 nhcnt, u32 reqid,
+ u64 desc, void **desc_ptr, u32 *toggle,
+ void *start_desc, void *end_desc)
+{
+ u64 d;
+ u32 nhavail, _toggle, _startpkt, _endpkt, _bdcount;
+
+ /* Sanity check */
+ if (nhcnt <= nhpos)
+ return;
+
+ /*
+ * Each request or packet start with a HEADER descriptor followed
+ * by one or more non-HEADER descriptors (SRC, SRCT, MSRC, DST,
+ * DSTT, MDST, IMM, and IMMT). The number of non-HEADER descriptors
+ * following a HEADER descriptor is represented by BDCOUNT field
+ * of HEADER descriptor. The max value of BDCOUNT field is 31 which
+ * means we can only have 31 non-HEADER descriptors following one
+ * HEADER descriptor.
+ *
+ * In general use, number of non-HEADER descriptors can easily go
+ * beyond 31. To tackle this situation, we have packet (or request)
+ * extenstion bits (STARTPKT and ENDPKT) in the HEADER descriptor.
+ *
+ * To use packet extension, the first HEADER descriptor of request
+ * (or packet) will have STARTPKT=1 and ENDPKT=0. The intermediate
+ * HEADER descriptors will have STARTPKT=0 and ENDPKT=0. The last
+ * HEADER descriptor will have STARTPKT=0 and ENDPKT=1. Also, the
+ * TOGGLE bit of the first HEADER will be set to invalid state to
+ * ensure that FlexRM does not start fetching descriptors till all
+ * descriptors are enqueued. The user of this function will flip
+ * the TOGGLE bit of first HEADER after all descriptors are
+ * enqueued.
+ */
+
+ if ((nhpos % HEADER_BDCOUNT_MAX == 0) && (nhcnt - nhpos)) {
+ /* Prepare the header descriptor */
+ nhavail = (nhcnt - nhpos);
+ _toggle = (nhpos == 0) ? !(*toggle) : (*toggle);
+ _startpkt = (nhpos == 0) ? 0x1 : 0x0;
+ _endpkt = (nhavail <= HEADER_BDCOUNT_MAX) ? 0x1 : 0x0;
+ _bdcount = (nhavail <= HEADER_BDCOUNT_MAX) ?
+ nhavail : HEADER_BDCOUNT_MAX;
+ if (nhavail <= HEADER_BDCOUNT_MAX)
+ _bdcount = nhavail;
+ else
+ _bdcount = HEADER_BDCOUNT_MAX;
+ d = flexrm_header_desc(_toggle, _startpkt, _endpkt,
+ _bdcount, 0x0, reqid);
+
+ /* Write header descriptor */
+ flexrm_write_desc(*desc_ptr, d);
+
+ /* Point to next descriptor */
+ *desc_ptr += sizeof(desc);
+ if (*desc_ptr == end_desc)
+ *desc_ptr = start_desc;
+
+ /* Skip next pointer descriptors */
+ while (flexrm_is_next_table_desc(*desc_ptr)) {
+ *toggle = (*toggle) ? 0 : 1;
+ *desc_ptr += sizeof(desc);
+ if (*desc_ptr == end_desc)
+ *desc_ptr = start_desc;
+ }
+ }
+
+ /* Write desired descriptor */
+ flexrm_write_desc(*desc_ptr, desc);
+
+ /* Point to next descriptor */
+ *desc_ptr += sizeof(desc);
+ if (*desc_ptr == end_desc)
+ *desc_ptr = start_desc;
+
+ /* Skip next pointer descriptors */
+ while (flexrm_is_next_table_desc(*desc_ptr)) {
+ *toggle = (*toggle) ? 0 : 1;
+ *desc_ptr += sizeof(desc);
+ if (*desc_ptr == end_desc)
+ *desc_ptr = start_desc;
+ }
+}
+
+static u64 flexrm_src_desc(dma_addr_t addr, unsigned int length)
+{
+ u64 desc = 0;
+
+ DESC_ENC(desc, SRC_TYPE, DESC_TYPE_SHIFT, DESC_TYPE_MASK);
+ DESC_ENC(desc, length, SRC_LENGTH_SHIFT, SRC_LENGTH_MASK);
+ DESC_ENC(desc, addr, SRC_ADDR_SHIFT, SRC_ADDR_MASK);
+
+ return desc;
+}
+
+static u64 flexrm_msrc_desc(dma_addr_t addr, unsigned int length_div_16)
+{
+ u64 desc = 0;
+
+ DESC_ENC(desc, MSRC_TYPE, DESC_TYPE_SHIFT, DESC_TYPE_MASK);
+ DESC_ENC(desc, length_div_16, MSRC_LENGTH_SHIFT, MSRC_LENGTH_MASK);
+ DESC_ENC(desc, addr, MSRC_ADDR_SHIFT, MSRC_ADDR_MASK);
+
+ return desc;
+}
+
+static u64 flexrm_dst_desc(dma_addr_t addr, unsigned int length)
+{
+ u64 desc = 0;
+
+ DESC_ENC(desc, DST_TYPE, DESC_TYPE_SHIFT, DESC_TYPE_MASK);
+ DESC_ENC(desc, length, DST_LENGTH_SHIFT, DST_LENGTH_MASK);
+ DESC_ENC(desc, addr, DST_ADDR_SHIFT, DST_ADDR_MASK);
+
+ return desc;
+}
+
+static u64 flexrm_mdst_desc(dma_addr_t addr, unsigned int length_div_16)
+{
+ u64 desc = 0;
+
+ DESC_ENC(desc, MDST_TYPE, DESC_TYPE_SHIFT, DESC_TYPE_MASK);
+ DESC_ENC(desc, length_div_16, MDST_LENGTH_SHIFT, MDST_LENGTH_MASK);
+ DESC_ENC(desc, addr, MDST_ADDR_SHIFT, MDST_ADDR_MASK);
+
+ return desc;
+}
+
+static u64 flexrm_imm_desc(u64 data)
+{
+ u64 desc = 0;
+
+ DESC_ENC(desc, IMM_TYPE, DESC_TYPE_SHIFT, DESC_TYPE_MASK);
+ DESC_ENC(desc, data, IMM_DATA_SHIFT, IMM_DATA_MASK);
+
+ return desc;
+}
+
+static u64 flexrm_srct_desc(dma_addr_t addr, unsigned int length)
+{
+ u64 desc = 0;
+
+ DESC_ENC(desc, SRCT_TYPE, DESC_TYPE_SHIFT, DESC_TYPE_MASK);
+ DESC_ENC(desc, length, SRCT_LENGTH_SHIFT, SRCT_LENGTH_MASK);
+ DESC_ENC(desc, addr, SRCT_ADDR_SHIFT, SRCT_ADDR_MASK);
+
+ return desc;
+}
+
+static u64 flexrm_dstt_desc(dma_addr_t addr, unsigned int length)
+{
+ u64 desc = 0;
+
+ DESC_ENC(desc, DSTT_TYPE, DESC_TYPE_SHIFT, DESC_TYPE_MASK);
+ DESC_ENC(desc, length, DSTT_LENGTH_SHIFT, DSTT_LENGTH_MASK);
+ DESC_ENC(desc, addr, DSTT_ADDR_SHIFT, DSTT_ADDR_MASK);
+
+ return desc;
+}
+
+static u64 flexrm_immt_desc(u64 data)
+{
+ u64 desc = 0;
+
+ DESC_ENC(desc, IMMT_TYPE, DESC_TYPE_SHIFT, DESC_TYPE_MASK);
+ DESC_ENC(desc, data, IMMT_DATA_SHIFT, IMMT_DATA_MASK);
+
+ return desc;
+}
+
+static bool flexrm_spu_sanity_check(struct brcm_message *msg)
+{
+ struct scatterlist *sg;
+
+ if (!msg->spu.src || !msg->spu.dst)
+ return false;
+ for (sg = msg->spu.src; sg; sg = sg_next(sg)) {
+ if (sg->length & 0xf) {
+ if (sg->length > SRC_LENGTH_MASK)
+ return false;
+ } else {
+ if (sg->length > (MSRC_LENGTH_MASK * 16))
+ return false;
+ }
+ }
+ for (sg = msg->spu.dst; sg; sg = sg_next(sg)) {
+ if (sg->length & 0xf) {
+ if (sg->length > DST_LENGTH_MASK)
+ return false;
+ } else {
+ if (sg->length > (MDST_LENGTH_MASK * 16))
+ return false;
+ }
+ }
+
+ return true;
+}
+
+static u32 flexrm_spu_estimate_nonheader_desc_count(struct brcm_message *msg)
+{
+ u32 cnt = 0;
+ unsigned int dst_target = 0;
+ struct scatterlist *src_sg = msg->spu.src, *dst_sg = msg->spu.dst;
+
+ while (src_sg || dst_sg) {
+ if (src_sg) {
+ cnt++;
+ dst_target = src_sg->length;
+ src_sg = sg_next(src_sg);
+ } else
+ dst_target = UINT_MAX;
+
+ while (dst_target && dst_sg) {
+ cnt++;
+ if (dst_sg->length < dst_target)
+ dst_target -= dst_sg->length;
+ else
+ dst_target = 0;
+ dst_sg = sg_next(dst_sg);
+ }
+ }
+
+ return cnt;
+}
+
+static int flexrm_spu_dma_map(struct device *dev, struct brcm_message *msg)
+{
+ int rc;
+
+ rc = dma_map_sg(dev, msg->spu.src, sg_nents(msg->spu.src),
+ DMA_TO_DEVICE);
+ if (rc < 0)
+ return rc;
+
+ rc = dma_map_sg(dev, msg->spu.dst, sg_nents(msg->spu.dst),
+ DMA_FROM_DEVICE);
+ if (rc < 0) {
+ dma_unmap_sg(dev, msg->spu.src, sg_nents(msg->spu.src),
+ DMA_TO_DEVICE);
+ return rc;
+ }
+
+ return 0;
+}
+
+static void flexrm_spu_dma_unmap(struct device *dev, struct brcm_message *msg)
+{
+ dma_unmap_sg(dev, msg->spu.dst, sg_nents(msg->spu.dst),
+ DMA_FROM_DEVICE);
+ dma_unmap_sg(dev, msg->spu.src, sg_nents(msg->spu.src),
+ DMA_TO_DEVICE);
+}
+
+static void *flexrm_spu_write_descs(struct brcm_message *msg, u32 nhcnt,
+ u32 reqid, void *desc_ptr, u32 toggle,
+ void *start_desc, void *end_desc)
+{
+ u64 d;
+ u32 nhpos = 0;
+ void *orig_desc_ptr = desc_ptr;
+ unsigned int dst_target = 0;
+ struct scatterlist *src_sg = msg->spu.src, *dst_sg = msg->spu.dst;
+
+ while (src_sg || dst_sg) {
+ if (src_sg) {
+ if (sg_dma_len(src_sg) & 0xf)
+ d = flexrm_src_desc(sg_dma_address(src_sg),
+ sg_dma_len(src_sg));
+ else
+ d = flexrm_msrc_desc(sg_dma_address(src_sg),
+ sg_dma_len(src_sg)/16);
+ flexrm_enqueue_desc(nhpos, nhcnt, reqid,
+ d, &desc_ptr, &toggle,
+ start_desc, end_desc);
+ nhpos++;
+ dst_target = sg_dma_len(src_sg);
+ src_sg = sg_next(src_sg);
+ } else
+ dst_target = UINT_MAX;
+
+ while (dst_target && dst_sg) {
+ if (sg_dma_len(dst_sg) & 0xf)
+ d = flexrm_dst_desc(sg_dma_address(dst_sg),
+ sg_dma_len(dst_sg));
+ else
+ d = flexrm_mdst_desc(sg_dma_address(dst_sg),
+ sg_dma_len(dst_sg)/16);
+ flexrm_enqueue_desc(nhpos, nhcnt, reqid,
+ d, &desc_ptr, &toggle,
+ start_desc, end_desc);
+ nhpos++;
+ if (sg_dma_len(dst_sg) < dst_target)
+ dst_target -= sg_dma_len(dst_sg);
+ else
+ dst_target = 0;
+ dst_sg = sg_next(dst_sg);
+ }
+ }
+
+ /* Null descriptor with invalid toggle bit */
+ flexrm_write_desc(desc_ptr, flexrm_null_desc(!toggle));
+
+ /* Ensure that descriptors have been written to memory */
+ wmb();
+
+ /* Flip toggle bit in header */
+ flexrm_flip_header_toogle(orig_desc_ptr);
+
+ return desc_ptr;
+}
+
+static bool flexrm_sba_sanity_check(struct brcm_message *msg)
+{
+ u32 i;
+
+ if (!msg->sba.cmds || !msg->sba.cmds_count)
+ return false;
+
+ for (i = 0; i < msg->sba.cmds_count; i++) {
+ if (((msg->sba.cmds[i].flags & BRCM_SBA_CMD_TYPE_B) ||
+ (msg->sba.cmds[i].flags & BRCM_SBA_CMD_TYPE_C)) &&
+ (msg->sba.cmds[i].flags & BRCM_SBA_CMD_HAS_OUTPUT))
+ return false;
+ if ((msg->sba.cmds[i].flags & BRCM_SBA_CMD_TYPE_B) &&
+ (msg->sba.cmds[i].data_len > SRCT_LENGTH_MASK))
+ return false;
+ if ((msg->sba.cmds[i].flags & BRCM_SBA_CMD_TYPE_C) &&
+ (msg->sba.cmds[i].data_len > SRCT_LENGTH_MASK))
+ return false;
+ if ((msg->sba.cmds[i].flags & BRCM_SBA_CMD_HAS_RESP) &&
+ (msg->sba.cmds[i].resp_len > DSTT_LENGTH_MASK))
+ return false;
+ if ((msg->sba.cmds[i].flags & BRCM_SBA_CMD_HAS_OUTPUT) &&
+ (msg->sba.cmds[i].data_len > DSTT_LENGTH_MASK))
+ return false;
+ }
+
+ return true;
+}
+
+static u32 flexrm_sba_estimate_nonheader_desc_count(struct brcm_message *msg)
+{
+ u32 i, cnt;
+
+ cnt = 0;
+ for (i = 0; i < msg->sba.cmds_count; i++) {
+ cnt++;
+
+ if ((msg->sba.cmds[i].flags & BRCM_SBA_CMD_TYPE_B) ||
+ (msg->sba.cmds[i].flags & BRCM_SBA_CMD_TYPE_C))
+ cnt++;
+
+ if (msg->sba.cmds[i].flags & BRCM_SBA_CMD_HAS_RESP)
+ cnt++;
+
+ if (msg->sba.cmds[i].flags & BRCM_SBA_CMD_HAS_OUTPUT)
+ cnt++;
+ }
+
+ return cnt;
+}
+
+static void *flexrm_sba_write_descs(struct brcm_message *msg, u32 nhcnt,
+ u32 reqid, void *desc_ptr, u32 toggle,
+ void *start_desc, void *end_desc)
+{
+ u64 d;
+ u32 i, nhpos = 0;
+ struct brcm_sba_command *c;
+ void *orig_desc_ptr = desc_ptr;
+
+ /* Convert SBA commands into descriptors */
+ for (i = 0; i < msg->sba.cmds_count; i++) {
+ c = &msg->sba.cmds[i];
+
+ if ((c->flags & BRCM_SBA_CMD_HAS_RESP) &&
+ (c->flags & BRCM_SBA_CMD_HAS_OUTPUT)) {
+ /* Destination response descriptor */
+ d = flexrm_dst_desc(c->resp, c->resp_len);
+ flexrm_enqueue_desc(nhpos, nhcnt, reqid,
+ d, &desc_ptr, &toggle,
+ start_desc, end_desc);
+ nhpos++;
+ } else if (c->flags & BRCM_SBA_CMD_HAS_RESP) {
+ /* Destination response with tlast descriptor */
+ d = flexrm_dstt_desc(c->resp, c->resp_len);
+ flexrm_enqueue_desc(nhpos, nhcnt, reqid,
+ d, &desc_ptr, &toggle,
+ start_desc, end_desc);
+ nhpos++;
+ }
+
+ if (c->flags & BRCM_SBA_CMD_HAS_OUTPUT) {
+ /* Destination with tlast descriptor */
+ d = flexrm_dstt_desc(c->data, c->data_len);
+ flexrm_enqueue_desc(nhpos, nhcnt, reqid,
+ d, &desc_ptr, &toggle,
+ start_desc, end_desc);
+ nhpos++;
+ }
+
+ if (c->flags & BRCM_SBA_CMD_TYPE_B) {
+ /* Command as immediate descriptor */
+ d = flexrm_imm_desc(c->cmd);
+ flexrm_enqueue_desc(nhpos, nhcnt, reqid,
+ d, &desc_ptr, &toggle,
+ start_desc, end_desc);
+ nhpos++;
+ } else {
+ /* Command as immediate descriptor with tlast */
+ d = flexrm_immt_desc(c->cmd);
+ flexrm_enqueue_desc(nhpos, nhcnt, reqid,
+ d, &desc_ptr, &toggle,
+ start_desc, end_desc);
+ nhpos++;
+ }
+
+ if ((c->flags & BRCM_SBA_CMD_TYPE_B) ||
+ (c->flags & BRCM_SBA_CMD_TYPE_C)) {
+ /* Source with tlast descriptor */
+ d = flexrm_srct_desc(c->data, c->data_len);
+ flexrm_enqueue_desc(nhpos, nhcnt, reqid,
+ d, &desc_ptr, &toggle,
+ start_desc, end_desc);
+ nhpos++;
+ }
+ }
+
+ /* Null descriptor with invalid toggle bit */
+ flexrm_write_desc(desc_ptr, flexrm_null_desc(!toggle));
+
+ /* Ensure that descriptors have been written to memory */
+ wmb();
+
+ /* Flip toggle bit in header */
+ flexrm_flip_header_toogle(orig_desc_ptr);
+
+ return desc_ptr;
+}
+
+static bool flexrm_sanity_check(struct brcm_message *msg)
+{
+ if (!msg)
+ return false;
+
+ switch (msg->type) {
+ case BRCM_MESSAGE_SPU:
+ return flexrm_spu_sanity_check(msg);
+ case BRCM_MESSAGE_SBA:
+ return flexrm_sba_sanity_check(msg);
+ default:
+ return false;
+ };
+}
+
+static u32 flexrm_estimate_nonheader_desc_count(struct brcm_message *msg)
+{
+ if (!msg)
+ return 0;
+
+ switch (msg->type) {
+ case BRCM_MESSAGE_SPU:
+ return flexrm_spu_estimate_nonheader_desc_count(msg);
+ case BRCM_MESSAGE_SBA:
+ return flexrm_sba_estimate_nonheader_desc_count(msg);
+ default:
+ return 0;
+ };
+}
+
+static int flexrm_dma_map(struct device *dev, struct brcm_message *msg)
+{
+ if (!dev || !msg)
+ return -EINVAL;
+
+ switch (msg->type) {
+ case BRCM_MESSAGE_SPU:
+ return flexrm_spu_dma_map(dev, msg);
+ default:
+ break;
+ }
+
+ return 0;
+}
+
+static void flexrm_dma_unmap(struct device *dev, struct brcm_message *msg)
+{
+ if (!dev || !msg)
+ return;
+
+ switch (msg->type) {
+ case BRCM_MESSAGE_SPU:
+ flexrm_spu_dma_unmap(dev, msg);
+ break;
+ default:
+ break;
+ }
+}
+
+static void *flexrm_write_descs(struct brcm_message *msg, u32 nhcnt,
+ u32 reqid, void *desc_ptr, u32 toggle,
+ void *start_desc, void *end_desc)
+{
+ if (!msg || !desc_ptr || !start_desc || !end_desc)
+ return ERR_PTR(-ENOTSUPP);
+
+ if ((desc_ptr < start_desc) || (end_desc <= desc_ptr))
+ return ERR_PTR(-ERANGE);
+
+ switch (msg->type) {
+ case BRCM_MESSAGE_SPU:
+ return flexrm_spu_write_descs(msg, nhcnt, reqid,
+ desc_ptr, toggle,
+ start_desc, end_desc);
+ case BRCM_MESSAGE_SBA:
+ return flexrm_sba_write_descs(msg, nhcnt, reqid,
+ desc_ptr, toggle,
+ start_desc, end_desc);
+ default:
+ return ERR_PTR(-ENOTSUPP);
+ };
+}
+
+/* ====== FlexRM driver helper routines ===== */
+
+static int flexrm_new_request(struct flexrm_ring *ring,
+ struct brcm_message *batch_msg,
+ struct brcm_message *msg)
+{
+ void *next;
+ unsigned long flags;
+ u32 val, count, nhcnt;
+ u32 read_offset, write_offset;
+ bool exit_cleanup = false;
+ int ret = 0, reqid;
+
+ /* Do sanity check on message */
+ if (!flexrm_sanity_check(msg))
+ return -EIO;
+ msg->error = 0;
+
+ /* If no requests possible then save data pointer and goto done. */
+ reqid = ida_simple_get(&ring->requests_ida, 0,
+ RING_MAX_REQ_COUNT, GFP_KERNEL);
+ if (reqid < 0) {
+ spin_lock_irqsave(&ring->lock, flags);
+ if (batch_msg)
+ ring->last_pending_msg = batch_msg;
+ else
+ ring->last_pending_msg = msg;
+ spin_unlock_irqrestore(&ring->lock, flags);
+ return 0;
+ }
+ ring->requests[reqid] = msg;
+
+ /* Do DMA mappings for the message */
+ ret = flexrm_dma_map(ring->mbox->dev, msg);
+ if (ret < 0) {
+ ring->requests[reqid] = NULL;
+ ida_simple_remove(&ring->requests_ida, reqid);
+ return ret;
+ }
+
+ /* If last_pending_msg is already set then goto done with error */
+ spin_lock_irqsave(&ring->lock, flags);
+ if (ring->last_pending_msg)
+ ret = -ENOSPC;
+ spin_unlock_irqrestore(&ring->lock, flags);
+ if (ret < 0) {
+ dev_warn(ring->mbox->dev, "no space in ring %d\n", ring->num);
+ exit_cleanup = true;
+ goto exit;
+ }
+
+ /* Determine current HW BD read offset */
+ read_offset = readl_relaxed(ring->regs + RING_BD_READ_PTR);
+ val = readl_relaxed(ring->regs + RING_BD_START_ADDR);
+ read_offset *= RING_DESC_SIZE;
+ read_offset += (u32)(BD_START_ADDR_DECODE(val) - ring->bd_dma_base);
+
+ /*
+ * Number required descriptors = number of non-header descriptors +
+ * number of header descriptors +
+ * 1x null descriptor
+ */
+ nhcnt = flexrm_estimate_nonheader_desc_count(msg);
+ count = flexrm_estimate_header_desc_count(nhcnt) + nhcnt + 1;
+
+ /* Check for available descriptor space. */
+ write_offset = ring->bd_write_offset;
+ while (count) {
+ if (!flexrm_is_next_table_desc(ring->bd_base + write_offset))
+ count--;
+ write_offset += RING_DESC_SIZE;
+ if (write_offset == RING_BD_SIZE)
+ write_offset = 0x0;
+ if (write_offset == read_offset)
+ break;
+ }
+ if (count) {
+ spin_lock_irqsave(&ring->lock, flags);
+ if (batch_msg)
+ ring->last_pending_msg = batch_msg;
+ else
+ ring->last_pending_msg = msg;
+ spin_unlock_irqrestore(&ring->lock, flags);
+ ret = 0;
+ exit_cleanup = true;
+ goto exit;
+ }
+
+ /* Write descriptors to ring */
+ next = flexrm_write_descs(msg, nhcnt, reqid,
+ ring->bd_base + ring->bd_write_offset,
+ RING_BD_TOGGLE_VALID(ring->bd_write_offset),
+ ring->bd_base, ring->bd_base + RING_BD_SIZE);
+ if (IS_ERR(next)) {
+ ret = PTR_ERR(next);
+ exit_cleanup = true;
+ goto exit;
+ }
+
+ /* Save ring BD write offset */
+ ring->bd_write_offset = (unsigned long)(next - ring->bd_base);
+
+exit:
+ /* Update error status in message */
+ msg->error = ret;
+
+ /* Cleanup if we failed */
+ if (exit_cleanup) {
+ flexrm_dma_unmap(ring->mbox->dev, msg);
+ ring->requests[reqid] = NULL;
+ ida_simple_remove(&ring->requests_ida, reqid);
+ }
+
+ return ret;
+}
+
+static int flexrm_process_completions(struct flexrm_ring *ring)
+{
+ u64 desc;
+ int err, count = 0;
+ unsigned long flags;
+ struct brcm_message *msg = NULL;
+ u32 reqid, cmpl_read_offset, cmpl_write_offset;
+ struct mbox_chan *chan = &ring->mbox->controller.chans[ring->num];
+
+ spin_lock_irqsave(&ring->lock, flags);
+
+ /* Check last_pending_msg */
+ if (ring->last_pending_msg) {
+ msg = ring->last_pending_msg;
+ ring->last_pending_msg = NULL;
+ }
+
+ /*
+ * Get current completion read and write offset
+ *
+ * Note: We should read completion write pointer atleast once
+ * after we get a MSI interrupt because HW maintains internal
+ * MSI status which will allow next MSI interrupt only after
+ * completion write pointer is read.
+ */
+ cmpl_write_offset = readl_relaxed(ring->regs + RING_CMPL_WRITE_PTR);
+ cmpl_write_offset *= RING_DESC_SIZE;
+ cmpl_read_offset = ring->cmpl_read_offset;
+ ring->cmpl_read_offset = cmpl_write_offset;
+
+ spin_unlock_irqrestore(&ring->lock, flags);
+
+ /* If last_pending_msg was set then queue it back */
+ if (msg)
+ mbox_send_message(chan, msg);
+
+ /* For each completed request notify mailbox clients */
+ reqid = 0;
+ while (cmpl_read_offset != cmpl_write_offset) {
+ /* Dequeue next completion descriptor */
+ desc = *((u64 *)(ring->cmpl_base + cmpl_read_offset));
+
+ /* Next read offset */
+ cmpl_read_offset += RING_DESC_SIZE;
+ if (cmpl_read_offset == RING_CMPL_SIZE)
+ cmpl_read_offset = 0;
+
+ /* Decode error from completion descriptor */
+ err = flexrm_cmpl_desc_to_error(desc);
+ if (err < 0) {
+ dev_warn(ring->mbox->dev,
+ "got completion desc=0x%lx with error %d",
+ (unsigned long)desc, err);
+ }
+
+ /* Determine request id from completion descriptor */
+ reqid = flexrm_cmpl_desc_to_reqid(desc);
+
+ /* Determine message pointer based on reqid */
+ msg = ring->requests[reqid];
+ if (!msg) {
+ dev_warn(ring->mbox->dev,
+ "null msg pointer for completion desc=0x%lx",
+ (unsigned long)desc);
+ continue;
+ }
+
+ /* Release reqid for recycling */
+ ring->requests[reqid] = NULL;
+ ida_simple_remove(&ring->requests_ida, reqid);
+
+ /* Unmap DMA mappings */
+ flexrm_dma_unmap(ring->mbox->dev, msg);
+
+ /* Give-back message to mailbox client */
+ msg->error = err;
+ mbox_chan_received_data(chan, msg);
+
+ /* Increment number of completions processed */
+ count++;
+ }
+
+ return count;
+}
+
+/* ====== FlexRM interrupt handler ===== */
+
+static irqreturn_t flexrm_irq_event(int irq, void *dev_id)
+{
+ /* We only have MSI for completions so just wakeup IRQ thread */
+ /* Ring related errors will be informed via completion descriptors */
+
+ return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t flexrm_irq_thread(int irq, void *dev_id)
+{
+ flexrm_process_completions(dev_id);
+
+ return IRQ_HANDLED;
+}
+
+/* ====== FlexRM mailbox callbacks ===== */
+
+static int flexrm_send_data(struct mbox_chan *chan, void *data)
+{
+ int i, rc;
+ struct flexrm_ring *ring = chan->con_priv;
+ struct brcm_message *msg = data;
+
+ if (msg->type == BRCM_MESSAGE_BATCH) {
+ for (i = msg->batch.msgs_queued;
+ i < msg->batch.msgs_count; i++) {
+ rc = flexrm_new_request(ring, msg,
+ &msg->batch.msgs[i]);
+ if (rc) {
+ msg->error = rc;
+ return rc;
+ }
+ msg->batch.msgs_queued++;
+ }
+ return 0;
+ }
+
+ return flexrm_new_request(ring, NULL, data);
+}
+
+static bool flexrm_peek_data(struct mbox_chan *chan)
+{
+ int cnt = flexrm_process_completions(chan->con_priv);
+
+ return (cnt > 0) ? true : false;
+}
+
+static int flexrm_startup(struct mbox_chan *chan)
+{
+ u64 d;
+ u32 val, off;
+ int ret = 0;
+ dma_addr_t next_addr;
+ struct flexrm_ring *ring = chan->con_priv;
+
+ /* Allocate BD memory */
+ ring->bd_base = dma_pool_alloc(ring->mbox->bd_pool,
+ GFP_KERNEL, &ring->bd_dma_base);
+ if (!ring->bd_base) {
+ dev_err(ring->mbox->dev, "can't allocate BD memory\n");
+ ret = -ENOMEM;
+ goto fail;
+ }
+
+ /* Configure next table pointer entries in BD memory */
+ for (off = 0; off < RING_BD_SIZE; off += RING_DESC_SIZE) {
+ next_addr = off + RING_DESC_SIZE;
+ if (next_addr == RING_BD_SIZE)
+ next_addr = 0;
+ next_addr += ring->bd_dma_base;
+ if (RING_BD_ALIGN_CHECK(next_addr))
+ d = flexrm_next_table_desc(RING_BD_TOGGLE_VALID(off),
+ next_addr);
+ else
+ d = flexrm_null_desc(RING_BD_TOGGLE_INVALID(off));
+ flexrm_write_desc(ring->bd_base + off, d);
+ }
+
+ /* Allocate completion memory */
+ ring->cmpl_base = dma_pool_alloc(ring->mbox->cmpl_pool,
+ GFP_KERNEL, &ring->cmpl_dma_base);
+ if (!ring->cmpl_base) {
+ dev_err(ring->mbox->dev, "can't allocate completion memory\n");
+ ret = -ENOMEM;
+ goto fail_free_bd_memory;
+ }
+ memset(ring->cmpl_base, 0, RING_CMPL_SIZE);
+
+ /* Request IRQ */
+ if (ring->irq == UINT_MAX) {
+ dev_err(ring->mbox->dev, "ring IRQ not available\n");
+ ret = -ENODEV;
+ goto fail_free_cmpl_memory;
+ }
+ ret = request_threaded_irq(ring->irq,
+ flexrm_irq_event,
+ flexrm_irq_thread,
+ 0, dev_name(ring->mbox->dev), ring);
+ if (ret) {
+ dev_err(ring->mbox->dev, "failed to request ring IRQ\n");
+ goto fail_free_cmpl_memory;
+ }
+ ring->irq_requested = true;
+
+ /* Disable/inactivate ring */
+ writel_relaxed(0x0, ring->regs + RING_CONTROL);
+
+ /* Program BD start address */
+ val = BD_START_ADDR_VALUE(ring->bd_dma_base);
+ writel_relaxed(val, ring->regs + RING_BD_START_ADDR);
+
+ /* BD write pointer will be same as HW write pointer */
+ ring->bd_write_offset =
+ readl_relaxed(ring->regs + RING_BD_WRITE_PTR);
+ ring->bd_write_offset *= RING_DESC_SIZE;
+
+ /* Program completion start address */
+ val = CMPL_START_ADDR_VALUE(ring->cmpl_dma_base);
+ writel_relaxed(val, ring->regs + RING_CMPL_START_ADDR);
+
+ /* Ensure last pending message is cleared */
+ ring->last_pending_msg = NULL;
+
+ /* Completion read pointer will be same as HW write pointer */
+ ring->cmpl_read_offset =
+ readl_relaxed(ring->regs + RING_CMPL_WRITE_PTR);
+ ring->cmpl_read_offset *= RING_DESC_SIZE;
+
+ /* Read ring Tx, Rx, and Outstanding counts to clear */
+ readl_relaxed(ring->regs + RING_NUM_REQ_RECV_LS);
+ readl_relaxed(ring->regs + RING_NUM_REQ_RECV_MS);
+ readl_relaxed(ring->regs + RING_NUM_REQ_TRANS_LS);
+ readl_relaxed(ring->regs + RING_NUM_REQ_TRANS_MS);
+ readl_relaxed(ring->regs + RING_NUM_REQ_OUTSTAND);
+
+ /* Configure RING_MSI_CONTROL */
+ val = 0;
+ val |= (ring->msi_timer_val << MSI_TIMER_VAL_SHIFT);
+ val |= BIT(MSI_ENABLE_SHIFT);
+ val |= (ring->msi_count_threshold & MSI_COUNT_MASK) << MSI_COUNT_SHIFT;
+ writel_relaxed(val, ring->regs + RING_MSI_CONTROL);
+
+ /* Enable/activate ring */
+ val = BIT(CONTROL_ACTIVE_SHIFT);
+ writel_relaxed(val, ring->regs + RING_CONTROL);
+
+ return 0;
+
+fail_free_cmpl_memory:
+ dma_pool_free(ring->mbox->cmpl_pool,
+ ring->cmpl_base, ring->cmpl_dma_base);
+ ring->cmpl_base = NULL;
+fail_free_bd_memory:
+ dma_pool_free(ring->mbox->bd_pool,
+ ring->bd_base, ring->bd_dma_base);
+ ring->bd_base = NULL;
+fail:
+ return ret;
+}
+
+static void flexrm_shutdown(struct mbox_chan *chan)
+{
+ u32 reqid;
+ unsigned int timeout;
+ struct brcm_message *msg;
+ struct flexrm_ring *ring = chan->con_priv;
+
+ /* Disable/inactivate ring */
+ writel_relaxed(0x0, ring->regs + RING_CONTROL);
+
+ /* Flush ring with timeout of 1s */
+ timeout = 1000;
+ writel_relaxed(BIT(CONTROL_FLUSH_SHIFT),
+ ring->regs + RING_CONTROL);
+ do {
+ if (readl_relaxed(ring->regs + RING_FLUSH_DONE) &
+ FLUSH_DONE_MASK)
+ break;
+ mdelay(1);
+ } while (timeout--);
+
+ /* Abort all in-flight requests */
+ for (reqid = 0; reqid < RING_MAX_REQ_COUNT; reqid++) {
+ msg = ring->requests[reqid];
+ if (!msg)
+ continue;
+
+ /* Release reqid for recycling */
+ ring->requests[reqid] = NULL;
+ ida_simple_remove(&ring->requests_ida, reqid);
+
+ /* Unmap DMA mappings */
+ flexrm_dma_unmap(ring->mbox->dev, msg);
+
+ /* Give-back message to mailbox client */
+ msg->error = -EIO;
+ mbox_chan_received_data(chan, msg);
+ }
+
+ /* Release IRQ */
+ if (ring->irq_requested) {
+ free_irq(ring->irq, ring);
+ ring->irq_requested = false;
+ }
+
+ /* Free-up completion descriptor ring */
+ if (ring->cmpl_base) {
+ dma_pool_free(ring->mbox->cmpl_pool,
+ ring->cmpl_base, ring->cmpl_dma_base);
+ ring->cmpl_base = NULL;
+ }
+
+ /* Free-up BD descriptor ring */
+ if (ring->bd_base) {
+ dma_pool_free(ring->mbox->bd_pool,
+ ring->bd_base, ring->bd_dma_base);
+ ring->bd_base = NULL;
+ }
+}
+
+static bool flexrm_last_tx_done(struct mbox_chan *chan)
+{
+ bool ret;
+ unsigned long flags;
+ struct flexrm_ring *ring = chan->con_priv;
+
+ spin_lock_irqsave(&ring->lock, flags);
+ ret = (ring->last_pending_msg) ? false : true;
+ spin_unlock_irqrestore(&ring->lock, flags);
+
+ return ret;
+}
+
+static const struct mbox_chan_ops flexrm_mbox_chan_ops = {
+ .send_data = flexrm_send_data,
+ .startup = flexrm_startup,
+ .shutdown = flexrm_shutdown,
+ .last_tx_done = flexrm_last_tx_done,
+ .peek_data = flexrm_peek_data,
+};
+
+static struct mbox_chan *flexrm_mbox_of_xlate(struct mbox_controller *cntlr,
+ const struct of_phandle_args *pa)
+{
+ struct mbox_chan *chan;
+ struct flexrm_ring *ring;
+
+ if (pa->args_count < 3)
+ return ERR_PTR(-EINVAL);
+
+ if (pa->args[0] >= cntlr->num_chans)
+ return ERR_PTR(-ENOENT);
+
+ if (pa->args[1] > MSI_COUNT_MASK)
+ return ERR_PTR(-EINVAL);
+
+ if (pa->args[2] > MSI_TIMER_VAL_MASK)
+ return ERR_PTR(-EINVAL);
+
+ chan = &cntlr->chans[pa->args[0]];
+ ring = chan->con_priv;
+ ring->msi_count_threshold = pa->args[1];
+ ring->msi_timer_val = pa->args[2];
+
+ return chan;
+}
+
+/* ====== FlexRM platform driver ===== */
+
+static void flexrm_mbox_msi_write(struct msi_desc *desc, struct msi_msg *msg)
+{
+ struct device *dev = msi_desc_to_dev(desc);
+ struct flexrm_mbox *mbox = dev_get_drvdata(dev);
+ struct flexrm_ring *ring = &mbox->rings[desc->platform.msi_index];
+
+ /* Configure per-Ring MSI registers */
+ writel_relaxed(msg->address_lo, ring->regs + RING_MSI_ADDR_LS);
+ writel_relaxed(msg->address_hi, ring->regs + RING_MSI_ADDR_MS);
+ writel_relaxed(msg->data, ring->regs + RING_MSI_DATA_VALUE);
+}
+
+static int flexrm_mbox_probe(struct platform_device *pdev)
+{
+ int index, ret = 0;
+ void __iomem *regs;
+ void __iomem *regs_end;
+ struct msi_desc *desc;
+ struct resource *iomem;
+ struct flexrm_ring *ring;
+ struct flexrm_mbox *mbox;
+ struct device *dev = &pdev->dev;
+
+ /* Allocate driver mailbox struct */
+ mbox = devm_kzalloc(dev, sizeof(*mbox), GFP_KERNEL);
+ if (!mbox) {
+ ret = -ENOMEM;
+ goto fail;
+ }
+ mbox->dev = dev;
+ platform_set_drvdata(pdev, mbox);
+
+ /* Get resource for registers */
+ iomem = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!iomem || (resource_size(iomem) < RING_REGS_SIZE)) {
+ ret = -ENODEV;
+ goto fail;
+ }
+
+ /* Map registers of all rings */
+ mbox->regs = devm_ioremap_resource(&pdev->dev, iomem);
+ if (IS_ERR(mbox->regs)) {
+ ret = PTR_ERR(mbox->regs);
+ dev_err(&pdev->dev, "Failed to remap mailbox regs: %d\n", ret);
+ goto fail;
+ }
+ regs_end = mbox->regs + resource_size(iomem);
+
+ /* Scan and count available rings */
+ mbox->num_rings = 0;
+ for (regs = mbox->regs; regs < regs_end; regs += RING_REGS_SIZE) {
+ if (readl_relaxed(regs + RING_VER) == RING_VER_MAGIC)
+ mbox->num_rings++;
+ }
+ if (!mbox->num_rings) {
+ ret = -ENODEV;
+ goto fail;
+ }
+
+ /* Allocate driver ring structs */
+ ring = devm_kcalloc(dev, mbox->num_rings, sizeof(*ring), GFP_KERNEL);
+ if (!ring) {
+ ret = -ENOMEM;
+ goto fail;
+ }
+ mbox->rings = ring;
+
+ /* Initialize members of driver ring structs */
+ regs = mbox->regs;
+ for (index = 0; index < mbox->num_rings; index++) {
+ ring = &mbox->rings[index];
+ ring->num = index;
+ ring->mbox = mbox;
+ while ((regs < regs_end) &&
+ (readl_relaxed(regs + RING_VER) != RING_VER_MAGIC))
+ regs += RING_REGS_SIZE;
+ if (regs_end <= regs) {
+ ret = -ENODEV;
+ goto fail;
+ }
+ ring->regs = regs;
+ regs += RING_REGS_SIZE;
+ ring->irq = UINT_MAX;
+ ring->irq_requested = false;
+ ring->msi_timer_val = MSI_TIMER_VAL_MASK;
+ ring->msi_count_threshold = 0x1;
+ ida_init(&ring->requests_ida);
+ memset(ring->requests, 0, sizeof(ring->requests));
+ ring->bd_base = NULL;
+ ring->bd_dma_base = 0;
+ ring->cmpl_base = NULL;
+ ring->cmpl_dma_base = 0;
+ spin_lock_init(&ring->lock);
+ ring->last_pending_msg = NULL;
+ ring->cmpl_read_offset = 0;
+ }
+
+ /* FlexRM is capable of 40-bit physical addresses only */
+ ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(40));
+ if (ret) {
+ ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
+ if (ret)
+ goto fail;
+ }
+
+ /* Create DMA pool for ring BD memory */
+ mbox->bd_pool = dma_pool_create("bd", dev, RING_BD_SIZE,
+ 1 << RING_BD_ALIGN_ORDER, 0);
+ if (!mbox->bd_pool) {
+ ret = -ENOMEM;
+ goto fail;
+ }
+
+ /* Create DMA pool for ring completion memory */
+ mbox->cmpl_pool = dma_pool_create("cmpl", dev, RING_CMPL_SIZE,
+ 1 << RING_CMPL_ALIGN_ORDER, 0);
+ if (!mbox->cmpl_pool) {
+ ret = -ENOMEM;
+ goto fail_destroy_bd_pool;
+ }
+
+ /* Allocate platform MSIs for each ring */
+ ret = platform_msi_domain_alloc_irqs(dev, mbox->num_rings,
+ flexrm_mbox_msi_write);
+ if (ret)
+ goto fail_destroy_cmpl_pool;
+
+ /* Save alloced IRQ numbers for each ring */
+ for_each_msi_entry(desc, dev) {
+ ring = &mbox->rings[desc->platform.msi_index];
+ ring->irq = desc->irq;
+ }
+
+ /* Initialize mailbox controller */
+ mbox->controller.txdone_irq = false;
+ mbox->controller.txdone_poll = true;
+ mbox->controller.txpoll_period = 1;
+ mbox->controller.ops = &flexrm_mbox_chan_ops;
+ mbox->controller.dev = dev;
+ mbox->controller.num_chans = mbox->num_rings;
+ mbox->controller.of_xlate = flexrm_mbox_of_xlate;
+ mbox->controller.chans = devm_kcalloc(dev, mbox->num_rings,
+ sizeof(*mbox->controller.chans), GFP_KERNEL);
+ if (!mbox->controller.chans) {
+ ret = -ENOMEM;
+ goto fail_free_msis;
+ }
+ for (index = 0; index < mbox->num_rings; index++)
+ mbox->controller.chans[index].con_priv = &mbox->rings[index];
+
+ /* Register mailbox controller */
+ ret = mbox_controller_register(&mbox->controller);
+ if (ret)
+ goto fail_free_msis;
+
+ dev_info(dev, "registered flexrm mailbox with %d channels\n",
+ mbox->controller.num_chans);
+
+ return 0;
+
+fail_free_msis:
+ platform_msi_domain_free_irqs(dev);
+fail_destroy_cmpl_pool:
+ dma_pool_destroy(mbox->cmpl_pool);
+fail_destroy_bd_pool:
+ dma_pool_destroy(mbox->bd_pool);
+fail:
+ return ret;
+}
+
+static int flexrm_mbox_remove(struct platform_device *pdev)
+{
+ int index;
+ struct device *dev = &pdev->dev;
+ struct flexrm_ring *ring;
+ struct flexrm_mbox *mbox = platform_get_drvdata(pdev);
+
+ mbox_controller_unregister(&mbox->controller);
+
+ platform_msi_domain_free_irqs(dev);
+
+ dma_pool_destroy(mbox->cmpl_pool);
+ dma_pool_destroy(mbox->bd_pool);
+
+ for (index = 0; index < mbox->num_rings; index++) {
+ ring = &mbox->rings[index];
+ ida_destroy(&ring->requests_ida);
+ }
+
+ return 0;
+}
+
+static const struct of_device_id flexrm_mbox_of_match[] = {
+ { .compatible = "brcm,iproc-flexrm-mbox", },
+ {},
+};
+MODULE_DEVICE_TABLE(of, flexrm_mbox_of_match);
+
+static struct platform_driver flexrm_mbox_driver = {
+ .driver = {
+ .name = "brcm-flexrm-mbox",
+ .of_match_table = flexrm_mbox_of_match,
+ },
+ .probe = flexrm_mbox_probe,
+ .remove = flexrm_mbox_remove,
+};
+module_platform_driver(flexrm_mbox_driver);
+
+MODULE_AUTHOR("Anup Patel <anup.patel@broadcom.com>");
+MODULE_DESCRIPTION("Broadcom FlexRM mailbox driver");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/mailbox/bcm-pdc-mailbox.c b/drivers/mailbox/bcm-pdc-mailbox.c
index 2aeb034..4fe7be0 100644
--- a/drivers/mailbox/bcm-pdc-mailbox.c
+++ b/drivers/mailbox/bcm-pdc-mailbox.c
@@ -18,7 +18,8 @@
* Broadcom PDC Mailbox Driver
* The PDC provides a ring based programming interface to one or more hardware
* offload engines. For example, the PDC driver works with both SPU-M and SPU2
- * cryptographic offload hardware. In some chips the PDC is referred to as MDE.
+ * cryptographic offload hardware. In some chips the PDC is referred to as MDE,
+ * and in others the FA2/FA+ hardware is used with this PDC driver.
*
* The PDC driver registers with the Linux mailbox framework as a mailbox
* controller, once for each PDC instance. Ring 0 for each PDC is registered as
@@ -108,6 +109,7 @@
#define PDC_INTMASK_OFFSET 0x24
#define PDC_INTSTATUS_OFFSET 0x20
#define PDC_RCVLAZY0_OFFSET (0x30 + 4 * PDC_RINGSET)
+#define FA_RCVLAZY0_OFFSET 0x100
/*
* For SPU2, configure MDE_CKSUM_CONTROL to write 17 bytes of metadata
@@ -162,6 +164,11 @@
/* Maximum size buffer the DMA engine can handle */
#define PDC_DMA_BUF_MAX 16384
+enum pdc_hw {
+ FA_HW, /* FA2/FA+ hardware (i.e. Northstar Plus) */
+ PDC_HW /* PDC/MDE hardware (i.e. Northstar 2, Pegasus) */
+};
+
struct pdc_dma_map {
void *ctx; /* opaque context associated with frame */
};
@@ -211,13 +218,13 @@
u32 gptimer; /* 0x028 */
u32 PAD;
- u32 intrcvlazy_0; /* 0x030 */
- u32 intrcvlazy_1; /* 0x034 */
- u32 intrcvlazy_2; /* 0x038 */
- u32 intrcvlazy_3; /* 0x03c */
+ u32 intrcvlazy_0; /* 0x030 (Only in PDC, not FA2) */
+ u32 intrcvlazy_1; /* 0x034 (Only in PDC, not FA2) */
+ u32 intrcvlazy_2; /* 0x038 (Only in PDC, not FA2) */
+ u32 intrcvlazy_3; /* 0x03c (Only in PDC, not FA2) */
u32 PAD[48];
- u32 removed_intrecvlazy; /* 0x100 */
+ u32 fa_intrecvlazy; /* 0x100 (Only in FA2, not PDC) */
u32 flowctlthresh; /* 0x104 */
u32 wrrthresh; /* 0x108 */
u32 gmac_idle_cnt_thresh; /* 0x10c */
@@ -243,7 +250,7 @@
u32 serdes_status1; /* 0x1b0 */
u32 PAD[11]; /* 0x1b4-1dc */
u32 clk_ctl_st; /* 0x1e0 */
- u32 hw_war; /* 0x1e4 */
+ u32 hw_war; /* 0x1e4 (Only in PDC, not FA2) */
u32 pwrctl; /* 0x1e8 */
u32 PAD[5];
@@ -410,6 +417,9 @@
u32 txnobuf; /* unable to create tx descriptor */
u32 rxnobuf; /* unable to create rx descriptor */
u32 rx_oflow; /* count of rx overflows */
+
+ /* hardware type - FA2 or PDC/MDE */
+ enum pdc_hw hw_type;
};
/* Global variables */
@@ -1396,7 +1406,13 @@
/* interrupt configuration */
iowrite32(PDC_INTMASK, pdcs->pdc_reg_vbase + PDC_INTMASK_OFFSET);
- iowrite32(PDC_LAZY_INT, pdcs->pdc_reg_vbase + PDC_RCVLAZY0_OFFSET);
+
+ if (pdcs->hw_type == FA_HW)
+ iowrite32(PDC_LAZY_INT, pdcs->pdc_reg_vbase +
+ FA_RCVLAZY0_OFFSET);
+ else
+ iowrite32(PDC_LAZY_INT, pdcs->pdc_reg_vbase +
+ PDC_RCVLAZY0_OFFSET);
/* read irq from device tree */
pdcs->pdc_irq = irq_of_parse_and_map(dn, 0);
@@ -1465,6 +1481,17 @@
return 0;
}
+/* Device tree API */
+static const int pdc_hw = PDC_HW;
+static const int fa_hw = FA_HW;
+
+static const struct of_device_id pdc_mbox_of_match[] = {
+ {.compatible = "brcm,iproc-pdc-mbox", .data = &pdc_hw},
+ {.compatible = "brcm,iproc-fa2-mbox", .data = &fa_hw},
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, pdc_mbox_of_match);
+
/**
* pdc_dt_read() - Read application-specific data from device tree.
* @pdev: Platform device
@@ -1481,6 +1508,8 @@
{
struct device *dev = &pdev->dev;
struct device_node *dn = pdev->dev.of_node;
+ const struct of_device_id *match;
+ const int *hw_type;
int err;
err = of_property_read_u32(dn, "brcm,rx-status-len",
@@ -1492,6 +1521,14 @@
pdcs->use_bcm_hdr = of_property_read_bool(dn, "brcm,use-bcm-hdr");
+ pdcs->hw_type = PDC_HW;
+
+ match = of_match_device(of_match_ptr(pdc_mbox_of_match), dev);
+ if (match != NULL) {
+ hw_type = match->data;
+ pdcs->hw_type = *hw_type;
+ }
+
return 0;
}
@@ -1525,7 +1562,7 @@
pdcs->pdc_idx = pdcg.num_spu;
pdcg.num_spu++;
- err = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
+ err = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(39));
if (err) {
dev_warn(dev, "PDC device cannot perform DMA. Error %d.", err);
goto cleanup;
@@ -1611,12 +1648,6 @@
return 0;
}
-static const struct of_device_id pdc_mbox_of_match[] = {
- {.compatible = "brcm,iproc-pdc-mbox"},
- { /* sentinel */ }
-};
-MODULE_DEVICE_TABLE(of, pdc_mbox_of_match);
-
static struct platform_driver pdc_mbox_driver = {
.probe = pdc_probe,
.remove = pdc_remove,
diff --git a/drivers/mailbox/hi6220-mailbox.c b/drivers/mailbox/hi6220-mailbox.c
index 613722d..519376d 100644
--- a/drivers/mailbox/hi6220-mailbox.c
+++ b/drivers/mailbox/hi6220-mailbox.c
@@ -221,7 +221,7 @@
mbox->irq_map_chan[mchan->ack_irq] = NULL;
}
-static struct mbox_chan_ops hi6220_mbox_ops = {
+static const struct mbox_chan_ops hi6220_mbox_ops = {
.send_data = hi6220_mbox_send_data,
.startup = hi6220_mbox_startup,
.shutdown = hi6220_mbox_shutdown,
diff --git a/drivers/mailbox/mailbox-xgene-slimpro.c b/drivers/mailbox/mailbox-xgene-slimpro.c
index dd2afbc..a704016 100644
--- a/drivers/mailbox/mailbox-xgene-slimpro.c
+++ b/drivers/mailbox/mailbox-xgene-slimpro.c
@@ -174,7 +174,7 @@
devm_free_irq(mb_chan->dev, mb_chan->irq, mb_chan);
}
-static struct mbox_chan_ops slimpro_mbox_ops = {
+static const struct mbox_chan_ops slimpro_mbox_ops = {
.send_data = slimpro_mbox_send_data,
.startup = slimpro_mbox_startup,
.shutdown = slimpro_mbox_shutdown,
diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
index 4671f8a..9dfbf7e 100644
--- a/drivers/mailbox/mailbox.c
+++ b/drivers/mailbox/mailbox.c
@@ -103,11 +103,14 @@
/* Submit next message */
msg_submit(chan);
+ if (!mssg)
+ return;
+
/* Notify the client */
- if (mssg && chan->cl->tx_done)
+ if (chan->cl->tx_done)
chan->cl->tx_done(chan->cl, mssg, r);
- if (chan->cl->tx_block)
+ if (r != -ETIME && chan->cl->tx_block)
complete(&chan->tx_complete);
}
@@ -260,7 +263,7 @@
msg_submit(chan);
- if (chan->cl->tx_block && chan->active_req) {
+ if (chan->cl->tx_block) {
unsigned long wait;
int ret;
@@ -271,8 +274,8 @@
ret = wait_for_completion_timeout(&chan->tx_complete, wait);
if (ret == 0) {
- t = -EIO;
- tx_tick(chan, -EIO);
+ t = -ETIME;
+ tx_tick(chan, t);
}
}
@@ -453,6 +456,12 @@
txdone = TXDONE_BY_ACK;
if (txdone == TXDONE_BY_POLL) {
+
+ if (!mbox->ops->last_tx_done) {
+ dev_err(mbox->dev, "last_tx_done method is absent\n");
+ return -EINVAL;
+ }
+
hrtimer_init(&mbox->poll_hrt, CLOCK_MONOTONIC,
HRTIMER_MODE_REL);
mbox->poll_hrt.function = txdone_hrtimer;
diff --git a/drivers/md/Kconfig b/drivers/md/Kconfig
index b7767da..585ff32 100644
--- a/drivers/md/Kconfig
+++ b/drivers/md/Kconfig
@@ -115,7 +115,7 @@
RAID-10 requires mdadm-1.7.0 or later, available at:
- ftp://ftp.kernel.org/pub/linux/utils/raid/mdadm/
+ https://www.kernel.org/pub/linux/utils/raid/mdadm/
If unsure, say Y.
diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
index 9c689b3..975922c8 100644
--- a/drivers/md/dm-cache-target.c
+++ b/drivers/md/dm-cache-target.c
@@ -2773,7 +2773,6 @@
ti->num_discard_bios = 1;
ti->discards_supported = true;
- ti->discard_zeroes_data_unsupported = true;
ti->split_discard_bios = false;
cache->features = ca->features;
diff --git a/drivers/md/dm-core.h b/drivers/md/dm-core.h
index 136fda3..fea5bd5 100644
--- a/drivers/md/dm-core.h
+++ b/drivers/md/dm-core.h
@@ -132,6 +132,7 @@
void dm_init_normal_md_queue(struct mapped_device *md);
int md_in_flight(struct mapped_device *md);
void disable_write_same(struct mapped_device *md);
+void disable_write_zeroes(struct mapped_device *md);
static inline struct completion *dm_get_completion_from_kobject(struct kobject *kobj)
{
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 389a363..ef1d836 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -2030,7 +2030,6 @@
wake_up_process(cc->write_thread);
ti->num_flush_bios = 1;
- ti->discard_zeroes_data_unsupported = true;
return 0;
diff --git a/drivers/md/dm-io.c b/drivers/md/dm-io.c
index 03940bf..3702e50 100644
--- a/drivers/md/dm-io.c
+++ b/drivers/md/dm-io.c
@@ -312,9 +312,12 @@
*/
if (op == REQ_OP_DISCARD)
special_cmd_max_sectors = q->limits.max_discard_sectors;
+ else if (op == REQ_OP_WRITE_ZEROES)
+ special_cmd_max_sectors = q->limits.max_write_zeroes_sectors;
else if (op == REQ_OP_WRITE_SAME)
special_cmd_max_sectors = q->limits.max_write_same_sectors;
- if ((op == REQ_OP_DISCARD || op == REQ_OP_WRITE_SAME) &&
+ if ((op == REQ_OP_DISCARD || op == REQ_OP_WRITE_ZEROES ||
+ op == REQ_OP_WRITE_SAME) &&
special_cmd_max_sectors == 0) {
dec_count(io, region, -EOPNOTSUPP);
return;
@@ -328,11 +331,18 @@
/*
* Allocate a suitably sized-bio.
*/
- if ((op == REQ_OP_DISCARD) || (op == REQ_OP_WRITE_SAME))
+ switch (op) {
+ case REQ_OP_DISCARD:
+ case REQ_OP_WRITE_ZEROES:
+ num_bvecs = 0;
+ break;
+ case REQ_OP_WRITE_SAME:
num_bvecs = 1;
- else
+ break;
+ default:
num_bvecs = min_t(int, BIO_MAX_PAGES,
dm_sector_div_up(remaining, (PAGE_SIZE >> SECTOR_SHIFT)));
+ }
bio = bio_alloc_bioset(GFP_NOIO, num_bvecs, io->client->bios);
bio->bi_iter.bi_sector = where->sector + (where->count - remaining);
@@ -341,7 +351,7 @@
bio_set_op_attrs(bio, op, op_flags);
store_io_and_region_in_bio(bio, io, region);
- if (op == REQ_OP_DISCARD) {
+ if (op == REQ_OP_DISCARD || op == REQ_OP_WRITE_ZEROES) {
num_sectors = min_t(sector_t, special_cmd_max_sectors, remaining);
bio->bi_iter.bi_size = num_sectors << SECTOR_SHIFT;
remaining -= num_sectors;
diff --git a/drivers/md/dm-kcopyd.c b/drivers/md/dm-kcopyd.c
index 9e9d04cb..f858467 100644
--- a/drivers/md/dm-kcopyd.c
+++ b/drivers/md/dm-kcopyd.c
@@ -733,11 +733,11 @@
job->pages = &zero_page_list;
/*
- * Use WRITE SAME to optimize zeroing if all dests support it.
+ * Use WRITE ZEROES to optimize zeroing if all dests support it.
*/
- job->rw = REQ_OP_WRITE_SAME;
+ job->rw = REQ_OP_WRITE_ZEROES;
for (i = 0; i < job->num_dests; i++)
- if (!bdev_write_same(job->dests[i].bdev)) {
+ if (!bdev_write_zeroes_sectors(job->dests[i].bdev)) {
job->rw = WRITE;
break;
}
diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c
index 4788b0b..e17fd44 100644
--- a/drivers/md/dm-linear.c
+++ b/drivers/md/dm-linear.c
@@ -59,6 +59,7 @@
ti->num_flush_bios = 1;
ti->num_discard_bios = 1;
ti->num_write_same_bios = 1;
+ ti->num_write_zeroes_bios = 1;
ti->private = lc;
return 0;
diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
index 7f223db..2950b14 100644
--- a/drivers/md/dm-mpath.c
+++ b/drivers/md/dm-mpath.c
@@ -1103,6 +1103,7 @@
ti->num_flush_bios = 1;
ti->num_discard_bios = 1;
ti->num_write_same_bios = 1;
+ ti->num_write_zeroes_bios = 1;
if (m->queue_mode == DM_TYPE_BIO_BASED)
ti->per_io_data_size = multipath_per_bio_data_size();
else
@@ -1491,7 +1492,7 @@
*/
int r = DM_ENDIO_REQUEUE;
- if (!error && !clone->errors)
+ if (!error)
return 0; /* I/O complete */
if (noretry_error(error))
diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c
index 1e217ba..2dae3e5 100644
--- a/drivers/md/dm-raid.c
+++ b/drivers/md/dm-raid.c
@@ -2813,7 +2813,9 @@
/* Assume discards not supported until after checks below. */
ti->discards_supported = false;
- /* RAID level 4,5,6 require discard_zeroes_data for data integrity! */
+ /*
+ * XXX: RAID level 4,5,6 require zeroing for safety.
+ */
raid456 = (rs->md.level == 4 || rs->md.level == 5 || rs->md.level == 6);
for (i = 0; i < rs->raid_disks; i++) {
@@ -2827,8 +2829,6 @@
return;
if (raid456) {
- if (!q->limits.discard_zeroes_data)
- return;
if (!devices_handle_discard_safely) {
DMERR("raid456 discard support disabled due to discard_zeroes_data uncertainty.");
DMERR("Set dm-raid.devices_handle_discard_safely=Y to override.");
diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c
index 2ddc2d2..a95cbb8 100644
--- a/drivers/md/dm-raid1.c
+++ b/drivers/md/dm-raid1.c
@@ -1124,7 +1124,6 @@
ti->num_flush_bios = 1;
ti->num_discard_bios = 1;
ti->per_io_data_size = sizeof(struct dm_raid1_bio_record);
- ti->discard_zeroes_data_unsupported = true;
ms->kmirrord_wq = alloc_workqueue("kmirrord", WQ_MEM_RECLAIM, 0);
if (!ms->kmirrord_wq) {
diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c
index 0b081d1..bff7e3b 100644
--- a/drivers/md/dm-rq.c
+++ b/drivers/md/dm-rq.c
@@ -298,9 +298,14 @@
r = rq_end_io(tio->ti, clone, error, &tio->info);
}
- if (unlikely(r == -EREMOTEIO && (req_op(clone) == REQ_OP_WRITE_SAME) &&
- !clone->q->limits.max_write_same_sectors))
- disable_write_same(tio->md);
+ if (unlikely(r == -EREMOTEIO)) {
+ if (req_op(clone) == REQ_OP_WRITE_SAME &&
+ !clone->q->limits.max_write_same_sectors)
+ disable_write_same(tio->md);
+ if (req_op(clone) == REQ_OP_WRITE_ZEROES &&
+ !clone->q->limits.max_write_zeroes_sectors)
+ disable_write_zeroes(tio->md);
+ }
if (r <= 0)
/* The target wants to complete the I/O */
@@ -358,7 +363,7 @@
if (!rq->q->mq_ops)
blk_complete_request(rq);
else
- blk_mq_complete_request(rq, error);
+ blk_mq_complete_request(rq);
}
/*
@@ -762,7 +767,7 @@
return BLK_MQ_RQ_QUEUE_OK;
}
-static struct blk_mq_ops dm_mq_ops = {
+static const struct blk_mq_ops dm_mq_ops = {
.queue_rq = dm_mq_queue_rq,
.complete = dm_softirq_done,
.init_request = dm_mq_init_request,
diff --git a/drivers/md/dm-stripe.c b/drivers/md/dm-stripe.c
index 28193a5..5ef49c1 100644
--- a/drivers/md/dm-stripe.c
+++ b/drivers/md/dm-stripe.c
@@ -169,6 +169,7 @@
ti->num_flush_bios = stripes;
ti->num_discard_bios = stripes;
ti->num_write_same_bios = stripes;
+ ti->num_write_zeroes_bios = stripes;
sc->chunk_size = chunk_size;
if (chunk_size & (chunk_size - 1))
@@ -293,6 +294,7 @@
return DM_MAPIO_REMAPPED;
}
if (unlikely(bio_op(bio) == REQ_OP_DISCARD) ||
+ unlikely(bio_op(bio) == REQ_OP_WRITE_ZEROES) ||
unlikely(bio_op(bio) == REQ_OP_WRITE_SAME)) {
target_bio_nr = dm_bio_get_target_bio_nr(bio);
BUG_ON(target_bio_nr >= sc->stripes);
diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
index 3ad16d9..958275a 100644
--- a/drivers/md/dm-table.c
+++ b/drivers/md/dm-table.c
@@ -1449,22 +1449,6 @@
return false;
}
-static bool dm_table_discard_zeroes_data(struct dm_table *t)
-{
- struct dm_target *ti;
- unsigned i = 0;
-
- /* Ensure that all targets supports discard_zeroes_data. */
- while (i < dm_table_get_num_targets(t)) {
- ti = dm_table_get_target(t, i++);
-
- if (ti->discard_zeroes_data_unsupported)
- return false;
- }
-
- return true;
-}
-
static int device_is_nonrot(struct dm_target *ti, struct dm_dev *dev,
sector_t start, sector_t len, void *data)
{
@@ -1533,6 +1517,34 @@
return true;
}
+static int device_not_write_zeroes_capable(struct dm_target *ti, struct dm_dev *dev,
+ sector_t start, sector_t len, void *data)
+{
+ struct request_queue *q = bdev_get_queue(dev->bdev);
+
+ return q && !q->limits.max_write_zeroes_sectors;
+}
+
+static bool dm_table_supports_write_zeroes(struct dm_table *t)
+{
+ struct dm_target *ti;
+ unsigned i = 0;
+
+ while (i < dm_table_get_num_targets(t)) {
+ ti = dm_table_get_target(t, i++);
+
+ if (!ti->num_write_zeroes_bios)
+ return false;
+
+ if (!ti->type->iterate_devices ||
+ ti->type->iterate_devices(ti, device_not_write_zeroes_capable, NULL))
+ return false;
+ }
+
+ return true;
+}
+
+
static int device_discard_capable(struct dm_target *ti, struct dm_dev *dev,
sector_t start, sector_t len, void *data)
{
@@ -1592,9 +1604,6 @@
}
blk_queue_write_cache(q, wc, fua);
- if (!dm_table_discard_zeroes_data(t))
- q->limits.discard_zeroes_data = 0;
-
/* Ensure that all underlying devices are non-rotational. */
if (dm_table_all_devices_attribute(t, device_is_nonrot))
queue_flag_set_unlocked(QUEUE_FLAG_NONROT, q);
@@ -1603,6 +1612,8 @@
if (!dm_table_supports_write_same(t))
q->limits.max_write_same_sectors = 0;
+ if (!dm_table_supports_write_zeroes(t))
+ q->limits.max_write_zeroes_sectors = 0;
if (dm_table_all_devices_attribute(t, queue_supports_sg_merge))
queue_flag_clear_unlocked(QUEUE_FLAG_NO_SG_MERGE, q);
diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c
index 2b266a2..a5f1916 100644
--- a/drivers/md/dm-thin.c
+++ b/drivers/md/dm-thin.c
@@ -3263,7 +3263,6 @@
* them down to the data device. The thin device's discard
* processing will cause mappings to be removed from the btree.
*/
- ti->discard_zeroes_data_unsupported = true;
if (pf.discard_enabled && pf.discard_passdown) {
ti->num_discard_bios = 1;
@@ -4119,7 +4118,6 @@
ti->per_io_data_size = sizeof(struct dm_thin_endio_hook);
/* In case the pool supports discards, pass them on. */
- ti->discard_zeroes_data_unsupported = true;
if (tc->pool->pf.discard_enabled) {
ti->discards_supported = true;
ti->num_discard_bios = 1;
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index dfb7597..8bf3977 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -810,7 +810,6 @@
queue_io(md, bio);
} else {
/* done with normal IO or empty flush */
- trace_block_bio_complete(md->queue, bio, io_error);
bio->bi_error = io_error;
bio_endio(bio);
}
@@ -825,6 +824,14 @@
limits->max_write_same_sectors = 0;
}
+void disable_write_zeroes(struct mapped_device *md)
+{
+ struct queue_limits *limits = dm_get_queue_limits(md);
+
+ /* device doesn't really support WRITE ZEROES, disable it */
+ limits->max_write_zeroes_sectors = 0;
+}
+
static void clone_endio(struct bio *bio)
{
int error = bio->bi_error;
@@ -851,9 +858,14 @@
}
}
- if (unlikely(r == -EREMOTEIO && (bio_op(bio) == REQ_OP_WRITE_SAME) &&
- !bdev_get_queue(bio->bi_bdev)->limits.max_write_same_sectors))
- disable_write_same(md);
+ if (unlikely(r == -EREMOTEIO)) {
+ if (bio_op(bio) == REQ_OP_WRITE_SAME &&
+ !bdev_get_queue(bio->bi_bdev)->limits.max_write_same_sectors)
+ disable_write_same(md);
+ if (bio_op(bio) == REQ_OP_WRITE_ZEROES &&
+ !bdev_get_queue(bio->bi_bdev)->limits.max_write_zeroes_sectors)
+ disable_write_zeroes(md);
+ }
free_tio(tio);
dec_pending(io, error);
@@ -1202,6 +1214,11 @@
return ti->num_write_same_bios;
}
+static unsigned get_num_write_zeroes_bios(struct dm_target *ti)
+{
+ return ti->num_write_zeroes_bios;
+}
+
typedef bool (*is_split_required_fn)(struct dm_target *ti);
static bool is_split_required_for_discard(struct dm_target *ti)
@@ -1256,6 +1273,11 @@
return __send_changing_extent_only(ci, get_num_write_same_bios, NULL);
}
+static int __send_write_zeroes(struct clone_info *ci)
+{
+ return __send_changing_extent_only(ci, get_num_write_zeroes_bios, NULL);
+}
+
/*
* Select the correct strategy for processing a non-flush bio.
*/
@@ -1270,6 +1292,8 @@
return __send_discard(ci);
else if (unlikely(bio_op(bio) == REQ_OP_WRITE_SAME))
return __send_write_same(ci);
+ else if (unlikely(bio_op(bio) == REQ_OP_WRITE_ZEROES))
+ return __send_write_zeroes(ci);
ti = dm_table_find_target(ci->map, ci->sector);
if (!dm_target_is_valid(ti))
diff --git a/drivers/md/linear.c b/drivers/md/linear.c
index 3e38e02..377a8a3 100644
--- a/drivers/md/linear.c
+++ b/drivers/md/linear.c
@@ -293,6 +293,7 @@
split, disk_devt(mddev->gendisk),
bio_sector);
mddev_check_writesame(mddev, split);
+ mddev_check_write_zeroes(mddev, split);
generic_make_request(split);
}
} while (split != bio);
diff --git a/drivers/md/md.h b/drivers/md/md.h
index dde8ecb..1e76d64 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -709,4 +709,11 @@
!bdev_get_queue(bio->bi_bdev)->limits.max_write_same_sectors)
mddev->queue->limits.max_write_same_sectors = 0;
}
+
+static inline void mddev_check_write_zeroes(struct mddev *mddev, struct bio *bio)
+{
+ if (bio_op(bio) == REQ_OP_WRITE_ZEROES &&
+ !bdev_get_queue(bio->bi_bdev)->limits.max_write_zeroes_sectors)
+ mddev->queue->limits.max_write_zeroes_sectors = 0;
+}
#endif /* _MD_MD_H */
diff --git a/drivers/md/multipath.c b/drivers/md/multipath.c
index 79a12b5..e95d521 100644
--- a/drivers/md/multipath.c
+++ b/drivers/md/multipath.c
@@ -139,6 +139,7 @@
mp_bh->bio.bi_end_io = multipath_end_request;
mp_bh->bio.bi_private = mp_bh;
mddev_check_writesame(mddev, &mp_bh->bio);
+ mddev_check_write_zeroes(mddev, &mp_bh->bio);
generic_make_request(&mp_bh->bio);
return;
}
diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
index 93347ca..ce7a6a5 100644
--- a/drivers/md/raid0.c
+++ b/drivers/md/raid0.c
@@ -383,6 +383,7 @@
blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
blk_queue_max_write_same_sectors(mddev->queue, mddev->chunk_sectors);
+ blk_queue_max_write_zeroes_sectors(mddev->queue, mddev->chunk_sectors);
blk_queue_max_discard_sectors(mddev->queue, mddev->chunk_sectors);
blk_queue_io_min(mddev->queue, mddev->chunk_sectors << 9);
@@ -504,6 +505,7 @@
split, disk_devt(mddev->gendisk),
bio_sector);
mddev_check_writesame(mddev, split);
+ mddev_check_write_zeroes(mddev, split);
generic_make_request(split);
}
} while (split != bio);
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index a34f587..b59cc10 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -3177,8 +3177,10 @@
if (IS_ERR(conf))
return PTR_ERR(conf);
- if (mddev->queue)
+ if (mddev->queue) {
blk_queue_max_write_same_sectors(mddev->queue, 0);
+ blk_queue_max_write_zeroes_sectors(mddev->queue, 0);
+ }
rdev_for_each(rdev, mddev) {
if (!mddev->gendisk)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index e89a8d7..28ec3a9 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -3749,6 +3749,7 @@
blk_queue_max_discard_sectors(mddev->queue,
mddev->chunk_sectors);
blk_queue_max_write_same_sectors(mddev->queue, 0);
+ blk_queue_max_write_zeroes_sectors(mddev->queue, 0);
blk_queue_io_min(mddev->queue, chunk_size);
if (conf->geo.raid_disks % conf->geo.near_copies)
blk_queue_io_opt(mddev->queue, chunk_size * conf->geo.raid_disks);
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index ed5cd70..2efdb0d 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5031,8 +5031,6 @@
rdev_dec_pending(rdev, conf->mddev);
if (!error) {
- trace_block_bio_complete(bdev_get_queue(raid_bi->bi_bdev),
- raid_bi, 0);
bio_endio(raid_bi);
if (atomic_dec_and_test(&conf->active_aligned_reads))
wake_up(&conf->wait_for_quiescent);
@@ -7229,7 +7227,6 @@
if (mddev->queue) {
int chunk_size;
- bool discard_supported = true;
/* read-ahead size must cover two whole stripes, which
* is 2 * (datadisks) * chunksize where 'n' is the
* number of raid devices
@@ -7265,48 +7262,32 @@
blk_queue_max_discard_sectors(mddev->queue,
0xfffe * STRIPE_SECTORS);
- /*
- * unaligned part of discard request will be ignored, so can't
- * guarantee discard_zeroes_data
- */
- mddev->queue->limits.discard_zeroes_data = 0;
-
blk_queue_max_write_same_sectors(mddev->queue, 0);
+ blk_queue_max_write_zeroes_sectors(mddev->queue, 0);
rdev_for_each(rdev, mddev) {
disk_stack_limits(mddev->gendisk, rdev->bdev,
rdev->data_offset << 9);
disk_stack_limits(mddev->gendisk, rdev->bdev,
rdev->new_data_offset << 9);
- /*
- * discard_zeroes_data is required, otherwise data
- * could be lost. Consider a scenario: discard a stripe
- * (the stripe could be inconsistent if
- * discard_zeroes_data is 0); write one disk of the
- * stripe (the stripe could be inconsistent again
- * depending on which disks are used to calculate
- * parity); the disk is broken; The stripe data of this
- * disk is lost.
- */
- if (!blk_queue_discard(bdev_get_queue(rdev->bdev)) ||
- !bdev_get_queue(rdev->bdev)->
- limits.discard_zeroes_data)
- discard_supported = false;
- /* Unfortunately, discard_zeroes_data is not currently
- * a guarantee - just a hint. So we only allow DISCARD
- * if the sysadmin has confirmed that only safe devices
- * are in use by setting a module parameter.
- */
- if (!devices_handle_discard_safely) {
- if (discard_supported) {
- pr_info("md/raid456: discard support disabled due to uncertainty.\n");
- pr_info("Set raid456.devices_handle_discard_safely=Y to override.\n");
- }
- discard_supported = false;
- }
}
- if (discard_supported &&
+ /*
+ * zeroing is required, otherwise data
+ * could be lost. Consider a scenario: discard a stripe
+ * (the stripe could be inconsistent if
+ * discard_zeroes_data is 0); write one disk of the
+ * stripe (the stripe could be inconsistent again
+ * depending on which disks are used to calculate
+ * parity); the disk is broken; The stripe data of this
+ * disk is lost.
+ *
+ * We only allow DISCARD if the sysadmin has confirmed that
+ * only safe devices are in use by setting a module parameter.
+ * A better idea might be to turn DISCARD into WRITE_ZEROES
+ * requests, as that is required to be safe.
+ */
+ if (devices_handle_discard_safely &&
mddev->queue->limits.max_discard_sectors >= (stripe >> 9) &&
mddev->queue->limits.discard_granularity >= stripe)
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD,
diff --git a/drivers/media/dvb-frontends/horus3a.c b/drivers/media/dvb-frontends/horus3a.c
index 94bb4f7..68d759c 100644
--- a/drivers/media/dvb-frontends/horus3a.c
+++ b/drivers/media/dvb-frontends/horus3a.c
@@ -404,6 +404,6 @@
}
EXPORT_SYMBOL(horus3a_attach);
-MODULE_DESCRIPTION("Sony HORUS3A sattelite tuner driver");
+MODULE_DESCRIPTION("Sony HORUS3A satellite tuner driver");
MODULE_AUTHOR("Sergey Kozlov <serjk@netup.ru>");
MODULE_LICENSE("GPL");
diff --git a/drivers/media/platform/mtk-vpu/mtk_vpu.c b/drivers/media/platform/mtk-vpu/mtk_vpu.c
index 463b69c..aa44e11 100644
--- a/drivers/media/platform/mtk-vpu/mtk_vpu.c
+++ b/drivers/media/platform/mtk-vpu/mtk_vpu.c
@@ -589,7 +589,7 @@
);
if (ret == 0) {
ret = -ETIME;
- dev_err(dev, "wait vpu initialization timout!\n");
+ dev_err(dev, "wait vpu initialization timeout!\n");
goto OUT_LOAD_FW;
} else if (-ERESTARTSYS == ret) {
dev_err(dev, "wait vpu interrupted by a signal!\n");
diff --git a/drivers/misc/cxl/flash.c b/drivers/misc/cxl/flash.c
index c63d61e..7c61c70 100644
--- a/drivers/misc/cxl/flash.c
+++ b/drivers/misc/cxl/flash.c
@@ -343,7 +343,7 @@
return rc;
}
if (rc == 0) {
- pr_devel("remove curent afu\n");
+ pr_devel("remove current afu\n");
for (afu = 0; afu < adapter->slices; afu++)
cxl_guest_remove_afu(adapter->afu[afu]);
diff --git a/drivers/mmc/core/block.c b/drivers/mmc/core/block.c
index ff3da96..8273b07 100644
--- a/drivers/mmc/core/block.c
+++ b/drivers/mmc/core/block.c
@@ -129,6 +129,13 @@
struct mmc_blk_data *md);
static int get_card_status(struct mmc_card *card, u32 *status, int retries);
+static void mmc_blk_requeue(struct request_queue *q, struct request *req)
+{
+ spin_lock_irq(q->queue_lock);
+ blk_requeue_request(q, req);
+ spin_unlock_irq(q->queue_lock);
+}
+
static struct mmc_blk_data *mmc_blk_get(struct gendisk *disk)
{
struct mmc_blk_data *md;
@@ -721,10 +728,41 @@
#endif
};
+static int mmc_blk_part_switch_pre(struct mmc_card *card,
+ unsigned int part_type)
+{
+ int ret = 0;
+
+ if (part_type == EXT_CSD_PART_CONFIG_ACC_RPMB) {
+ if (card->ext_csd.cmdq_en) {
+ ret = mmc_cmdq_disable(card);
+ if (ret)
+ return ret;
+ }
+ mmc_retune_pause(card->host);
+ }
+
+ return ret;
+}
+
+static int mmc_blk_part_switch_post(struct mmc_card *card,
+ unsigned int part_type)
+{
+ int ret = 0;
+
+ if (part_type == EXT_CSD_PART_CONFIG_ACC_RPMB) {
+ mmc_retune_unpause(card->host);
+ if (card->reenable_cmdq && !card->ext_csd.cmdq_en)
+ ret = mmc_cmdq_enable(card);
+ }
+
+ return ret;
+}
+
static inline int mmc_blk_part_switch(struct mmc_card *card,
struct mmc_blk_data *md)
{
- int ret;
+ int ret = 0;
struct mmc_blk_data *main_md = dev_get_drvdata(&card->dev);
if (main_md->part_curr == md->part_type)
@@ -733,8 +771,9 @@
if (mmc_card_mmc(card)) {
u8 part_config = card->ext_csd.part_config;
- if (md->part_type == EXT_CSD_PART_CONFIG_ACC_RPMB)
- mmc_retune_pause(card->host);
+ ret = mmc_blk_part_switch_pre(card, md->part_type);
+ if (ret)
+ return ret;
part_config &= ~EXT_CSD_PART_CONFIG_ACC_MASK;
part_config |= md->part_type;
@@ -743,19 +782,17 @@
EXT_CSD_PART_CONFIG, part_config,
card->ext_csd.part_time);
if (ret) {
- if (md->part_type == EXT_CSD_PART_CONFIG_ACC_RPMB)
- mmc_retune_unpause(card->host);
+ mmc_blk_part_switch_post(card, md->part_type);
return ret;
}
card->ext_csd.part_config = part_config;
- if (main_md->part_curr == EXT_CSD_PART_CONFIG_ACC_RPMB)
- mmc_retune_unpause(card->host);
+ ret = mmc_blk_part_switch_post(card, main_md->part_curr);
}
main_md->part_curr = md->part_type;
- return 0;
+ return ret;
}
static int mmc_sd_num_wr_blocks(struct mmc_card *card, u32 *written_blocks)
@@ -1272,7 +1309,7 @@
{
if (!(card->ext_csd.rel_param & EXT_CSD_WR_REL_PARAM_EN)) {
/* Legacy mode imposes restrictions on transfers. */
- if (!IS_ALIGNED(brq->cmd.arg, card->ext_csd.rel_sectors))
+ if (!IS_ALIGNED(blk_rq_pos(req), card->ext_csd.rel_sectors))
brq->data.blocks = 1;
if (brq->data.blocks > card->ext_csd.rel_sectors)
@@ -1396,36 +1433,39 @@
return MMC_BLK_SUCCESS;
}
-static void mmc_blk_rw_rq_prep(struct mmc_queue_req *mqrq,
- struct mmc_card *card,
- int disable_multi,
- struct mmc_queue *mq)
+static void mmc_blk_data_prep(struct mmc_queue *mq, struct mmc_queue_req *mqrq,
+ int disable_multi, bool *do_rel_wr,
+ bool *do_data_tag)
{
- u32 readcmd, writecmd;
+ struct mmc_blk_data *md = mq->blkdata;
+ struct mmc_card *card = md->queue.card;
struct mmc_blk_request *brq = &mqrq->brq;
struct request *req = mqrq->req;
- struct mmc_blk_data *md = mq->blkdata;
- bool do_data_tag;
/*
* Reliable writes are used to implement Forced Unit Access and
* are supported only on MMCs.
*/
- bool do_rel_wr = (req->cmd_flags & REQ_FUA) &&
- (rq_data_dir(req) == WRITE) &&
- (md->flags & MMC_BLK_REL_WR);
+ *do_rel_wr = (req->cmd_flags & REQ_FUA) &&
+ rq_data_dir(req) == WRITE &&
+ (md->flags & MMC_BLK_REL_WR);
memset(brq, 0, sizeof(struct mmc_blk_request));
- brq->mrq.cmd = &brq->cmd;
+
brq->mrq.data = &brq->data;
- brq->cmd.arg = blk_rq_pos(req);
- if (!mmc_card_blockaddr(card))
- brq->cmd.arg <<= 9;
- brq->cmd.flags = MMC_RSP_SPI_R1 | MMC_RSP_R1 | MMC_CMD_ADTC;
- brq->data.blksz = 512;
brq->stop.opcode = MMC_STOP_TRANSMISSION;
brq->stop.arg = 0;
+
+ if (rq_data_dir(req) == READ) {
+ brq->data.flags = MMC_DATA_READ;
+ brq->stop.flags = MMC_RSP_SPI_R1 | MMC_RSP_R1 | MMC_CMD_AC;
+ } else {
+ brq->data.flags = MMC_DATA_WRITE;
+ brq->stop.flags = MMC_RSP_SPI_R1B | MMC_RSP_R1B | MMC_CMD_AC;
+ }
+
+ brq->data.blksz = 512;
brq->data.blocks = blk_rq_sectors(req);
/*
@@ -1456,6 +1496,68 @@
brq->data.blocks);
}
+ if (*do_rel_wr)
+ mmc_apply_rel_rw(brq, card, req);
+
+ /*
+ * Data tag is used only during writing meta data to speed
+ * up write and any subsequent read of this meta data
+ */
+ *do_data_tag = card->ext_csd.data_tag_unit_size &&
+ (req->cmd_flags & REQ_META) &&
+ (rq_data_dir(req) == WRITE) &&
+ ((brq->data.blocks * brq->data.blksz) >=
+ card->ext_csd.data_tag_unit_size);
+
+ mmc_set_data_timeout(&brq->data, card);
+
+ brq->data.sg = mqrq->sg;
+ brq->data.sg_len = mmc_queue_map_sg(mq, mqrq);
+
+ /*
+ * Adjust the sg list so it is the same size as the
+ * request.
+ */
+ if (brq->data.blocks != blk_rq_sectors(req)) {
+ int i, data_size = brq->data.blocks << 9;
+ struct scatterlist *sg;
+
+ for_each_sg(brq->data.sg, sg, brq->data.sg_len, i) {
+ data_size -= sg->length;
+ if (data_size <= 0) {
+ sg->length += data_size;
+ i++;
+ break;
+ }
+ }
+ brq->data.sg_len = i;
+ }
+
+ mqrq->areq.mrq = &brq->mrq;
+
+ mmc_queue_bounce_pre(mqrq);
+}
+
+static void mmc_blk_rw_rq_prep(struct mmc_queue_req *mqrq,
+ struct mmc_card *card,
+ int disable_multi,
+ struct mmc_queue *mq)
+{
+ u32 readcmd, writecmd;
+ struct mmc_blk_request *brq = &mqrq->brq;
+ struct request *req = mqrq->req;
+ struct mmc_blk_data *md = mq->blkdata;
+ bool do_rel_wr, do_data_tag;
+
+ mmc_blk_data_prep(mq, mqrq, disable_multi, &do_rel_wr, &do_data_tag);
+
+ brq->mrq.cmd = &brq->cmd;
+
+ brq->cmd.arg = blk_rq_pos(req);
+ if (!mmc_card_blockaddr(card))
+ brq->cmd.arg <<= 9;
+ brq->cmd.flags = MMC_RSP_SPI_R1 | MMC_RSP_R1 | MMC_CMD_ADTC;
+
if (brq->data.blocks > 1 || do_rel_wr) {
/* SPI multiblock writes terminate using a special
* token, not a STOP_TRANSMISSION request.
@@ -1470,32 +1572,7 @@
readcmd = MMC_READ_SINGLE_BLOCK;
writecmd = MMC_WRITE_BLOCK;
}
- if (rq_data_dir(req) == READ) {
- brq->cmd.opcode = readcmd;
- brq->data.flags = MMC_DATA_READ;
- if (brq->mrq.stop)
- brq->stop.flags = MMC_RSP_SPI_R1 | MMC_RSP_R1 |
- MMC_CMD_AC;
- } else {
- brq->cmd.opcode = writecmd;
- brq->data.flags = MMC_DATA_WRITE;
- if (brq->mrq.stop)
- brq->stop.flags = MMC_RSP_SPI_R1B | MMC_RSP_R1B |
- MMC_CMD_AC;
- }
-
- if (do_rel_wr)
- mmc_apply_rel_rw(brq, card, req);
-
- /*
- * Data tag is used only during writing meta data to speed
- * up write and any subsequent read of this meta data
- */
- do_data_tag = (card->ext_csd.data_tag_unit_size) &&
- (req->cmd_flags & REQ_META) &&
- (rq_data_dir(req) == WRITE) &&
- ((brq->data.blocks * brq->data.blksz) >=
- card->ext_csd.data_tag_unit_size);
+ brq->cmd.opcode = rq_data_dir(req) == READ ? readcmd : writecmd;
/*
* Pre-defined multi-block transfers are preferable to
@@ -1526,34 +1603,7 @@
brq->mrq.sbc = &brq->sbc;
}
- mmc_set_data_timeout(&brq->data, card);
-
- brq->data.sg = mqrq->sg;
- brq->data.sg_len = mmc_queue_map_sg(mq, mqrq);
-
- /*
- * Adjust the sg list so it is the same size as the
- * request.
- */
- if (brq->data.blocks != blk_rq_sectors(req)) {
- int i, data_size = brq->data.blocks << 9;
- struct scatterlist *sg;
-
- for_each_sg(brq->data.sg, sg, brq->data.sg_len, i) {
- data_size -= sg->length;
- if (data_size <= 0) {
- sg->length += data_size;
- i++;
- break;
- }
- }
- brq->data.sg_len = i;
- }
-
- mqrq->areq.mrq = &brq->mrq;
mqrq->areq.err_check = mmc_blk_err_check;
-
- mmc_queue_bounce_pre(mqrq);
}
static bool mmc_blk_rw_cmd_err(struct mmc_blk_data *md, struct mmc_card *card,
@@ -1585,11 +1635,14 @@
return req_pending;
}
-static void mmc_blk_rw_cmd_abort(struct mmc_card *card, struct request *req)
+static void mmc_blk_rw_cmd_abort(struct mmc_queue *mq, struct mmc_card *card,
+ struct request *req,
+ struct mmc_queue_req *mqrq)
{
if (mmc_card_removed(card))
req->rq_flags |= RQF_QUIET;
while (blk_end_request(req, -EIO, blk_rq_cur_bytes(req)));
+ mmc_queue_req_free(mq, mqrq);
}
/**
@@ -1597,7 +1650,8 @@
* @mq: the queue with the card and host to restart
* @req: a new request that want to be started after the current one
*/
-static void mmc_blk_rw_try_restart(struct mmc_queue *mq, struct request *req)
+static void mmc_blk_rw_try_restart(struct mmc_queue *mq, struct request *req,
+ struct mmc_queue_req *mqrq)
{
if (!req)
return;
@@ -1608,11 +1662,12 @@
if (mmc_card_removed(mq->card)) {
req->rq_flags |= RQF_QUIET;
blk_end_request_all(req, -EIO);
+ mmc_queue_req_free(mq, mqrq);
return;
}
/* Else proceed and try to restart the current async request */
- mmc_blk_rw_rq_prep(mq->mqrq_cur, mq->card, 0, mq);
- mmc_start_areq(mq->card->host, &mq->mqrq_cur->areq, NULL);
+ mmc_blk_rw_rq_prep(mqrq, mq->card, 0, mq);
+ mmc_start_areq(mq->card->host, &mqrq->areq, NULL);
}
static void mmc_blk_issue_rw_rq(struct mmc_queue *mq, struct request *new_req)
@@ -1622,13 +1677,23 @@
struct mmc_blk_request *brq;
int disable_multi = 0, retry = 0, type, retune_retry_done = 0;
enum mmc_blk_status status;
+ struct mmc_queue_req *mqrq_cur = NULL;
struct mmc_queue_req *mq_rq;
struct request *old_req;
struct mmc_async_req *new_areq;
struct mmc_async_req *old_areq;
bool req_pending = true;
- if (!new_req && !mq->mqrq_prev->req)
+ if (new_req) {
+ mqrq_cur = mmc_queue_req_find(mq, new_req);
+ if (!mqrq_cur) {
+ WARN_ON(1);
+ mmc_blk_requeue(mq->queue, new_req);
+ new_req = NULL;
+ }
+ }
+
+ if (!mq->qcnt)
return;
do {
@@ -1641,12 +1706,12 @@
!IS_ALIGNED(blk_rq_sectors(new_req), 8)) {
pr_err("%s: Transfer size is not 4KB sector size aligned\n",
new_req->rq_disk->disk_name);
- mmc_blk_rw_cmd_abort(card, new_req);
+ mmc_blk_rw_cmd_abort(mq, card, new_req, mqrq_cur);
return;
}
- mmc_blk_rw_rq_prep(mq->mqrq_cur, card, 0, mq);
- new_areq = &mq->mqrq_cur->areq;
+ mmc_blk_rw_rq_prep(mqrq_cur, card, 0, mq);
+ new_areq = &mqrq_cur->areq;
} else
new_areq = NULL;
@@ -1657,8 +1722,6 @@
* and there is nothing more to do until it is
* complete.
*/
- if (status == MMC_BLK_NEW_REQUEST)
- mq->new_request = true;
return;
}
@@ -1691,7 +1754,7 @@
pr_err("%s BUG rq_tot %d d_xfer %d\n",
__func__, blk_rq_bytes(old_req),
brq->data.bytes_xfered);
- mmc_blk_rw_cmd_abort(card, old_req);
+ mmc_blk_rw_cmd_abort(mq, card, old_req, mq_rq);
return;
}
break;
@@ -1699,12 +1762,15 @@
req_pending = mmc_blk_rw_cmd_err(md, card, brq, old_req, req_pending);
if (mmc_blk_reset(md, card->host, type)) {
if (req_pending)
- mmc_blk_rw_cmd_abort(card, old_req);
- mmc_blk_rw_try_restart(mq, new_req);
+ mmc_blk_rw_cmd_abort(mq, card, old_req, mq_rq);
+ else
+ mmc_queue_req_free(mq, mq_rq);
+ mmc_blk_rw_try_restart(mq, new_req, mqrq_cur);
return;
}
if (!req_pending) {
- mmc_blk_rw_try_restart(mq, new_req);
+ mmc_queue_req_free(mq, mq_rq);
+ mmc_blk_rw_try_restart(mq, new_req, mqrq_cur);
return;
}
break;
@@ -1716,8 +1782,8 @@
case MMC_BLK_ABORT:
if (!mmc_blk_reset(md, card->host, type))
break;
- mmc_blk_rw_cmd_abort(card, old_req);
- mmc_blk_rw_try_restart(mq, new_req);
+ mmc_blk_rw_cmd_abort(mq, card, old_req, mq_rq);
+ mmc_blk_rw_try_restart(mq, new_req, mqrq_cur);
return;
case MMC_BLK_DATA_ERR: {
int err;
@@ -1726,8 +1792,8 @@
if (!err)
break;
if (err == -ENODEV) {
- mmc_blk_rw_cmd_abort(card, old_req);
- mmc_blk_rw_try_restart(mq, new_req);
+ mmc_blk_rw_cmd_abort(mq, card, old_req, mq_rq);
+ mmc_blk_rw_try_restart(mq, new_req, mqrq_cur);
return;
}
/* Fall through */
@@ -1748,19 +1814,20 @@
req_pending = blk_end_request(old_req, -EIO,
brq->data.blksz);
if (!req_pending) {
- mmc_blk_rw_try_restart(mq, new_req);
+ mmc_queue_req_free(mq, mq_rq);
+ mmc_blk_rw_try_restart(mq, new_req, mqrq_cur);
return;
}
break;
case MMC_BLK_NOMEDIUM:
- mmc_blk_rw_cmd_abort(card, old_req);
- mmc_blk_rw_try_restart(mq, new_req);
+ mmc_blk_rw_cmd_abort(mq, card, old_req, mq_rq);
+ mmc_blk_rw_try_restart(mq, new_req, mqrq_cur);
return;
default:
pr_err("%s: Unhandled return value (%d)",
old_req->rq_disk->disk_name, status);
- mmc_blk_rw_cmd_abort(card, old_req);
- mmc_blk_rw_try_restart(mq, new_req);
+ mmc_blk_rw_cmd_abort(mq, card, old_req, mq_rq);
+ mmc_blk_rw_try_restart(mq, new_req, mqrq_cur);
return;
}
@@ -1776,6 +1843,8 @@
mq_rq->brq.retune_retry_done = retune_retry_done;
}
} while (req_pending);
+
+ mmc_queue_req_free(mq, mq_rq);
}
void mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req)
@@ -1783,9 +1852,8 @@
int ret;
struct mmc_blk_data *md = mq->blkdata;
struct mmc_card *card = md->queue.card;
- bool req_is_special = mmc_req_is_special(req);
- if (req && !mq->mqrq_prev->req)
+ if (req && !mq->qcnt)
/* claim host only for the first request */
mmc_get_card(card);
@@ -1797,20 +1865,19 @@
goto out;
}
- mq->new_request = false;
if (req && req_op(req) == REQ_OP_DISCARD) {
/* complete ongoing async transfer before issuing discard */
- if (card->host->areq)
+ if (mq->qcnt)
mmc_blk_issue_rw_rq(mq, NULL);
mmc_blk_issue_discard_rq(mq, req);
} else if (req && req_op(req) == REQ_OP_SECURE_ERASE) {
/* complete ongoing async transfer before issuing secure erase*/
- if (card->host->areq)
+ if (mq->qcnt)
mmc_blk_issue_rw_rq(mq, NULL);
mmc_blk_issue_secdiscard_rq(mq, req);
} else if (req && req_op(req) == REQ_OP_FLUSH) {
/* complete ongoing async transfer before issuing flush */
- if (card->host->areq)
+ if (mq->qcnt)
mmc_blk_issue_rw_rq(mq, NULL);
mmc_blk_issue_flush(mq, req);
} else {
@@ -1819,13 +1886,7 @@
}
out:
- if ((!req && !mq->new_request) || req_is_special)
- /*
- * Release host when there are no more requests
- * and after special request(discard, flush) is done.
- * In case sepecial request, there is no reentry to
- * the 'mmc_blk_issue_rq' with 'mqrq_prev->req'.
- */
+ if (!mq->qcnt)
mmc_put_card(card);
}
@@ -2105,6 +2166,7 @@
{
struct mmc_blk_data *md, *part_md;
char cap_str[10];
+ int ret;
/*
* Check that the card supports the command class(es) we need.
@@ -2114,9 +2176,15 @@
mmc_fixup_device(card, mmc_blk_fixups);
+ ret = mmc_queue_alloc_shared_queue(card);
+ if (ret)
+ return ret;
+
md = mmc_blk_alloc(card);
- if (IS_ERR(md))
+ if (IS_ERR(md)) {
+ mmc_queue_free_shared_queue(card);
return PTR_ERR(md);
+ }
string_get_size((u64)get_capacity(md->disk), 512, STRING_UNITS_2,
cap_str, sizeof(cap_str));
@@ -2154,6 +2222,7 @@
out:
mmc_blk_remove_parts(card, md);
mmc_blk_remove_req(md);
+ mmc_queue_free_shared_queue(card);
return 0;
}
@@ -2171,6 +2240,7 @@
pm_runtime_put_noidle(&card->dev);
mmc_blk_remove_req(md);
dev_set_drvdata(&card->dev, NULL);
+ mmc_queue_free_shared_queue(card);
}
static int _mmc_blk_suspend(struct mmc_card *card)
diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
index 926e0fd..82c45dd 100644
--- a/drivers/mmc/core/core.c
+++ b/drivers/mmc/core/core.c
@@ -172,14 +172,16 @@
trace_mmc_request_done(host, mrq);
- if (err && cmd->retries && !mmc_card_removed(host->card)) {
- /*
- * Request starter must handle retries - see
- * mmc_wait_for_req_done().
- */
- if (mrq->done)
- mrq->done(mrq);
- } else {
+ /*
+ * We list various conditions for the command to be considered
+ * properly done:
+ *
+ * - There was no error, OK fine then
+ * - We are not doing some kind of retry
+ * - The card was removed (...so just complete everything no matter
+ * if there are errors or retries)
+ */
+ if (!err || !cmd->retries || mmc_card_removed(host->card)) {
mmc_should_fail_request(host, mrq);
if (!host->ongoing_mrq)
@@ -211,10 +213,13 @@
mrq->stop->resp[0], mrq->stop->resp[1],
mrq->stop->resp[2], mrq->stop->resp[3]);
}
-
- if (mrq->done)
- mrq->done(mrq);
}
+ /*
+ * Request starter must handle retries - see
+ * mmc_wait_for_req_done().
+ */
+ if (mrq->done)
+ mrq->done(mrq);
}
EXPORT_SYMBOL(mmc_request_done);
@@ -234,8 +239,10 @@
/*
* For sdio rw commands we must wait for card busy otherwise some
* sdio devices won't work properly.
+ * And bypass I/O abort, reset and bus suspend operations.
*/
- if (mmc_is_io_op(mrq->cmd->opcode) && host->ops->card_busy) {
+ if (sdio_is_io_busy(mrq->cmd->opcode, mrq->cmd->arg) &&
+ host->ops->card_busy) {
int tries = 500; /* Wait aprox 500ms at maximum */
while (host->ops->card_busy(host) && --tries)
@@ -262,26 +269,19 @@
host->ops->request(host, mrq);
}
-static int mmc_start_request(struct mmc_host *host, struct mmc_request *mrq)
+static void mmc_mrq_pr_debug(struct mmc_host *host, struct mmc_request *mrq)
{
-#ifdef CONFIG_MMC_DEBUG
- unsigned int i, sz;
- struct scatterlist *sg;
-#endif
- mmc_retune_hold(host);
-
- if (mmc_card_removed(host->card))
- return -ENOMEDIUM;
-
if (mrq->sbc) {
pr_debug("<%s: starting CMD%u arg %08x flags %08x>\n",
mmc_hostname(host), mrq->sbc->opcode,
mrq->sbc->arg, mrq->sbc->flags);
}
- pr_debug("%s: starting CMD%u arg %08x flags %08x\n",
- mmc_hostname(host), mrq->cmd->opcode,
- mrq->cmd->arg, mrq->cmd->flags);
+ if (mrq->cmd) {
+ pr_debug("%s: starting CMD%u arg %08x flags %08x\n",
+ mmc_hostname(host), mrq->cmd->opcode, mrq->cmd->arg,
+ mrq->cmd->flags);
+ }
if (mrq->data) {
pr_debug("%s: blksz %d blocks %d flags %08x "
@@ -297,11 +297,20 @@
mmc_hostname(host), mrq->stop->opcode,
mrq->stop->arg, mrq->stop->flags);
}
+}
- WARN_ON(!host->claimed);
+static int mmc_mrq_prep(struct mmc_host *host, struct mmc_request *mrq)
+{
+#ifdef CONFIG_MMC_DEBUG
+ unsigned int i, sz;
+ struct scatterlist *sg;
+#endif
- mrq->cmd->error = 0;
- mrq->cmd->mrq = mrq;
+ if (mrq->cmd) {
+ mrq->cmd->error = 0;
+ mrq->cmd->mrq = mrq;
+ mrq->cmd->data = mrq->data;
+ }
if (mrq->sbc) {
mrq->sbc->error = 0;
mrq->sbc->mrq = mrq;
@@ -318,8 +327,6 @@
if (sz != mrq->data->blocks * mrq->data->blksz)
return -EINVAL;
#endif
-
- mrq->cmd->data = mrq->data;
mrq->data->error = 0;
mrq->data->mrq = mrq;
if (mrq->stop) {
@@ -328,6 +335,27 @@
mrq->stop->mrq = mrq;
}
}
+
+ return 0;
+}
+
+static int mmc_start_request(struct mmc_host *host, struct mmc_request *mrq)
+{
+ int err;
+
+ mmc_retune_hold(host);
+
+ if (mmc_card_removed(host->card))
+ return -ENOMEDIUM;
+
+ mmc_mrq_pr_debug(host, mrq);
+
+ WARN_ON(!host->claimed);
+
+ err = mmc_mrq_prep(host, mrq);
+ if (err)
+ return err;
+
led_trigger_event(host->led, LED_FULL);
__mmc_start_request(host, mrq);
@@ -485,56 +513,6 @@
return err;
}
-/*
- * mmc_wait_for_data_req_done() - wait for request completed
- * @host: MMC host to prepare the command.
- * @mrq: MMC request to wait for
- *
- * Blocks MMC context till host controller will ack end of data request
- * execution or new request notification arrives from the block layer.
- * Handles command retries.
- *
- * Returns enum mmc_blk_status after checking errors.
- */
-static enum mmc_blk_status mmc_wait_for_data_req_done(struct mmc_host *host,
- struct mmc_request *mrq)
-{
- struct mmc_command *cmd;
- struct mmc_context_info *context_info = &host->context_info;
- enum mmc_blk_status status;
-
- while (1) {
- wait_event_interruptible(context_info->wait,
- (context_info->is_done_rcv ||
- context_info->is_new_req));
-
- if (context_info->is_done_rcv) {
- context_info->is_done_rcv = false;
- cmd = mrq->cmd;
-
- if (!cmd->error || !cmd->retries ||
- mmc_card_removed(host->card)) {
- status = host->areq->err_check(host->card,
- host->areq);
- break; /* return status */
- } else {
- mmc_retune_recheck(host);
- pr_info("%s: req failed (CMD%u): %d, retrying...\n",
- mmc_hostname(host),
- cmd->opcode, cmd->error);
- cmd->retries--;
- cmd->error = 0;
- __mmc_start_request(host, mrq);
- continue; /* wait for done/new event again */
- }
- }
-
- return MMC_BLK_NEW_REQUEST;
- }
- mmc_retune_release(host);
- return status;
-}
-
void mmc_wait_for_req_done(struct mmc_host *host, struct mmc_request *mrq)
{
struct mmc_command *cmd;
@@ -639,14 +617,44 @@
*/
static enum mmc_blk_status mmc_finalize_areq(struct mmc_host *host)
{
+ struct mmc_context_info *context_info = &host->context_info;
enum mmc_blk_status status;
if (!host->areq)
return MMC_BLK_SUCCESS;
- status = mmc_wait_for_data_req_done(host, host->areq->mrq);
- if (status == MMC_BLK_NEW_REQUEST)
- return status;
+ while (1) {
+ wait_event_interruptible(context_info->wait,
+ (context_info->is_done_rcv ||
+ context_info->is_new_req));
+
+ if (context_info->is_done_rcv) {
+ struct mmc_command *cmd;
+
+ context_info->is_done_rcv = false;
+ cmd = host->areq->mrq->cmd;
+
+ if (!cmd->error || !cmd->retries ||
+ mmc_card_removed(host->card)) {
+ status = host->areq->err_check(host->card,
+ host->areq);
+ break; /* return status */
+ } else {
+ mmc_retune_recheck(host);
+ pr_info("%s: req failed (CMD%u): %d, retrying...\n",
+ mmc_hostname(host),
+ cmd->opcode, cmd->error);
+ cmd->retries--;
+ cmd->error = 0;
+ __mmc_start_request(host, host->areq->mrq);
+ continue; /* wait for done/new event again */
+ }
+ }
+
+ return MMC_BLK_NEW_REQUEST;
+ }
+
+ mmc_retune_release(host);
/*
* Check BKOPS urgency for each R1 response
@@ -683,7 +691,7 @@
{
enum mmc_blk_status status;
int start_err = 0;
- struct mmc_async_req *data = host->areq;
+ struct mmc_async_req *previous = host->areq;
/* Prepare a new request */
if (areq)
@@ -691,13 +699,12 @@
/* Finalize previous request */
status = mmc_finalize_areq(host);
+ if (ret_stat)
+ *ret_stat = status;
/* The previous request is still going on... */
- if (status == MMC_BLK_NEW_REQUEST) {
- if (ret_stat)
- *ret_stat = status;
+ if (status == MMC_BLK_NEW_REQUEST)
return NULL;
- }
/* Fine so far, start the new request! */
if (status == MMC_BLK_SUCCESS && areq)
@@ -716,9 +723,7 @@
else
host->areq = areq;
- if (ret_stat)
- *ret_stat = status;
- return data;
+ return previous;
}
EXPORT_SYMBOL(mmc_start_areq);
@@ -2555,6 +2560,12 @@
}
EXPORT_SYMBOL(mmc_calc_max_discard);
+bool mmc_card_is_blockaddr(struct mmc_card *card)
+{
+ return card ? mmc_card_blockaddr(card) : false;
+}
+EXPORT_SYMBOL(mmc_card_is_blockaddr);
+
int mmc_set_blocklen(struct mmc_card *card, unsigned int blocklen)
{
struct mmc_command cmd = {};
diff --git a/drivers/mmc/core/mmc.c b/drivers/mmc/core/mmc.c
index b502601..2c87ded 100644
--- a/drivers/mmc/core/mmc.c
+++ b/drivers/mmc/core/mmc.c
@@ -790,6 +790,7 @@
MMC_DEV_ATTR(raw_rpmb_size_mult, "%#x\n", card->ext_csd.raw_rpmb_size_mult);
MMC_DEV_ATTR(rel_sectors, "%#x\n", card->ext_csd.rel_sectors);
MMC_DEV_ATTR(ocr, "%08x\n", card->ocr);
+MMC_DEV_ATTR(cmdq_en, "%d\n", card->ext_csd.cmdq_en);
static ssize_t mmc_fwrev_show(struct device *dev,
struct device_attribute *attr,
@@ -845,6 +846,7 @@
&dev_attr_rel_sectors.attr,
&dev_attr_ocr.attr,
&dev_attr_dsr.attr,
+ &dev_attr_cmdq_en.attr,
NULL,
};
ATTRIBUTE_GROUPS(mmc_std);
@@ -1788,6 +1790,13 @@
}
/*
+ * In some cases (e.g. RPMB or mmc_test), the Command Queue must be
+ * disabled for a time, so a flag is needed to indicate to re-enable the
+ * Command Queue.
+ */
+ card->reenable_cmdq = card->ext_csd.cmdq_en;
+
+ /*
* The mandatory minimum values are defined for packed command.
* read: 5, write: 3
*/
diff --git a/drivers/mmc/core/mmc_ops.c b/drivers/mmc/core/mmc_ops.c
index fe80f26..78f75f0 100644
--- a/drivers/mmc/core/mmc_ops.c
+++ b/drivers/mmc/core/mmc_ops.c
@@ -305,7 +305,7 @@
int mmc_send_csd(struct mmc_card *card, u32 *csd)
{
int ret, i;
- u32 *csd_tmp;
+ __be32 *csd_tmp;
if (!mmc_host_is_spi(card->host))
return mmc_send_cxd_native(card->host, card->rca << 16,
@@ -319,7 +319,7 @@
if (ret)
goto err;
- for (i = 0;i < 4;i++)
+ for (i = 0; i < 4; i++)
csd[i] = be32_to_cpu(csd_tmp[i]);
err:
@@ -330,7 +330,7 @@
int mmc_send_cid(struct mmc_host *host, u32 *cid)
{
int ret, i;
- u32 *cid_tmp;
+ __be32 *cid_tmp;
if (!mmc_host_is_spi(host)) {
if (!host->card)
@@ -347,7 +347,7 @@
if (ret)
goto err;
- for (i = 0;i < 4;i++)
+ for (i = 0; i < 4; i++)
cid[i] = be32_to_cpu(cid_tmp[i]);
err:
@@ -838,3 +838,31 @@
{
return (card && card->csd.mmca_vsn > CSD_SPEC_VER_3);
}
+
+static int mmc_cmdq_switch(struct mmc_card *card, bool enable)
+{
+ u8 val = enable ? EXT_CSD_CMDQ_MODE_ENABLED : 0;
+ int err;
+
+ if (!card->ext_csd.cmdq_support)
+ return -EOPNOTSUPP;
+
+ err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL, EXT_CSD_CMDQ_MODE_EN,
+ val, card->ext_csd.generic_cmd6_time);
+ if (!err)
+ card->ext_csd.cmdq_en = enable;
+
+ return err;
+}
+
+int mmc_cmdq_enable(struct mmc_card *card)
+{
+ return mmc_cmdq_switch(card, true);
+}
+EXPORT_SYMBOL_GPL(mmc_cmdq_enable);
+
+int mmc_cmdq_disable(struct mmc_card *card)
+{
+ return mmc_cmdq_switch(card, false);
+}
+EXPORT_SYMBOL_GPL(mmc_cmdq_disable);
diff --git a/drivers/mmc/core/mmc_ops.h b/drivers/mmc/core/mmc_ops.h
index 74beea8..978bd2e 100644
--- a/drivers/mmc/core/mmc_ops.h
+++ b/drivers/mmc/core/mmc_ops.h
@@ -46,6 +46,8 @@
void mmc_start_bkops(struct mmc_card *card, bool from_exception);
int mmc_can_reset(struct mmc_card *card);
int mmc_flush_cache(struct mmc_card *card);
+int mmc_cmdq_enable(struct mmc_card *card);
+int mmc_cmdq_disable(struct mmc_card *card);
#endif
diff --git a/drivers/mmc/core/mmc_test.c b/drivers/mmc/core/mmc_test.c
index f99ac31..fd1b4b8 100644
--- a/drivers/mmc/core/mmc_test.c
+++ b/drivers/mmc/core/mmc_test.c
@@ -26,6 +26,7 @@
#include "card.h"
#include "host.h"
#include "bus.h"
+#include "mmc_ops.h"
#define RESULT_OK 0
#define RESULT_FAIL 1
@@ -3264,6 +3265,14 @@
if (ret)
return ret;
+ if (card->ext_csd.cmdq_en) {
+ mmc_claim_host(card->host);
+ ret = mmc_cmdq_disable(card);
+ mmc_release_host(card->host);
+ if (ret)
+ return ret;
+ }
+
dev_info(&card->dev, "Card claimed for testing.\n");
return 0;
@@ -3271,6 +3280,11 @@
static void mmc_test_remove(struct mmc_card *card)
{
+ if (card->reenable_cmdq) {
+ mmc_claim_host(card->host);
+ mmc_cmdq_enable(card);
+ mmc_release_host(card->host);
+ }
mmc_test_free_result(card);
mmc_test_free_dbgfs_file(card);
}
diff --git a/drivers/mmc/core/queue.c b/drivers/mmc/core/queue.c
index 493eb10..5c37b6b 100644
--- a/drivers/mmc/core/queue.c
+++ b/drivers/mmc/core/queue.c
@@ -40,6 +40,35 @@
return BLKPREP_OK;
}
+struct mmc_queue_req *mmc_queue_req_find(struct mmc_queue *mq,
+ struct request *req)
+{
+ struct mmc_queue_req *mqrq;
+ int i = ffz(mq->qslots);
+
+ if (i >= mq->qdepth)
+ return NULL;
+
+ mqrq = &mq->mqrq[i];
+ WARN_ON(mqrq->req || mq->qcnt >= mq->qdepth ||
+ test_bit(mqrq->task_id, &mq->qslots));
+ mqrq->req = req;
+ mq->qcnt += 1;
+ __set_bit(mqrq->task_id, &mq->qslots);
+
+ return mqrq;
+}
+
+void mmc_queue_req_free(struct mmc_queue *mq,
+ struct mmc_queue_req *mqrq)
+{
+ WARN_ON(!mqrq->req || mq->qcnt < 1 ||
+ !test_bit(mqrq->task_id, &mq->qslots));
+ mqrq->req = NULL;
+ mq->qcnt -= 1;
+ __clear_bit(mqrq->task_id, &mq->qslots);
+}
+
static int mmc_queue_thread(void *d)
{
struct mmc_queue *mq = d;
@@ -50,7 +79,7 @@
down(&mq->thread_sem);
do {
- struct request *req = NULL;
+ struct request *req;
spin_lock_irq(q->queue_lock);
set_current_state(TASK_INTERRUPTIBLE);
@@ -63,38 +92,17 @@
* Dispatch queue is empty so set flags for
* mmc_request_fn() to wake us up.
*/
- if (mq->mqrq_prev->req)
+ if (mq->qcnt)
cntx->is_waiting_last_req = true;
else
mq->asleep = true;
}
- mq->mqrq_cur->req = req;
spin_unlock_irq(q->queue_lock);
- if (req || mq->mqrq_prev->req) {
- bool req_is_special = mmc_req_is_special(req);
-
+ if (req || mq->qcnt) {
set_current_state(TASK_RUNNING);
mmc_blk_issue_rq(mq, req);
cond_resched();
- if (mq->new_request) {
- mq->new_request = false;
- continue; /* fetch again */
- }
-
- /*
- * Current request becomes previous request
- * and vice versa.
- * In case of special requests, current request
- * has been finished. Do not assign it to previous
- * request.
- */
- if (req_is_special)
- mq->mqrq_cur->req = NULL;
-
- mq->mqrq_prev->brq.mrq.data = NULL;
- mq->mqrq_prev->req = NULL;
- swap(mq->mqrq_prev, mq->mqrq_cur);
} else {
if (kthread_should_stop()) {
set_current_state(TASK_RUNNING);
@@ -141,17 +149,13 @@
wake_up_process(mq->thread);
}
-static struct scatterlist *mmc_alloc_sg(int sg_len, int *err)
+static struct scatterlist *mmc_alloc_sg(int sg_len)
{
struct scatterlist *sg;
sg = kmalloc_array(sg_len, sizeof(*sg), GFP_KERNEL);
- if (!sg)
- *err = -ENOMEM;
- else {
- *err = 0;
+ if (sg)
sg_init_table(sg, sg_len);
- }
return sg;
}
@@ -167,8 +171,6 @@
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, q);
blk_queue_max_discard_sectors(q, max_discard);
- if (card->erased_byte == 0 && !mmc_can_discard(card))
- q->limits.discard_zeroes_data = 1;
q->limits.discard_granularity = card->pref_erase << 9;
/* granularity must not be greater than max. discard */
if (card->pref_erase > max_discard)
@@ -177,62 +179,6 @@
queue_flag_set_unlocked(QUEUE_FLAG_SECERASE, q);
}
-#ifdef CONFIG_MMC_BLOCK_BOUNCE
-static bool mmc_queue_alloc_bounce_bufs(struct mmc_queue *mq,
- unsigned int bouncesz)
-{
- int i;
-
- for (i = 0; i < mq->qdepth; i++) {
- mq->mqrq[i].bounce_buf = kmalloc(bouncesz, GFP_KERNEL);
- if (!mq->mqrq[i].bounce_buf)
- goto out_err;
- }
-
- return true;
-
-out_err:
- while (--i >= 0) {
- kfree(mq->mqrq[i].bounce_buf);
- mq->mqrq[i].bounce_buf = NULL;
- }
- pr_warn("%s: unable to allocate bounce buffers\n",
- mmc_card_name(mq->card));
- return false;
-}
-
-static int mmc_queue_alloc_bounce_sgs(struct mmc_queue *mq,
- unsigned int bouncesz)
-{
- int i, ret;
-
- for (i = 0; i < mq->qdepth; i++) {
- mq->mqrq[i].sg = mmc_alloc_sg(1, &ret);
- if (ret)
- return ret;
-
- mq->mqrq[i].bounce_sg = mmc_alloc_sg(bouncesz / 512, &ret);
- if (ret)
- return ret;
- }
-
- return 0;
-}
-#endif
-
-static int mmc_queue_alloc_sgs(struct mmc_queue *mq, int max_segs)
-{
- int i, ret;
-
- for (i = 0; i < mq->qdepth; i++) {
- mq->mqrq[i].sg = mmc_alloc_sg(max_segs, &ret);
- if (ret)
- return ret;
- }
-
- return 0;
-}
-
static void mmc_queue_req_free_bufs(struct mmc_queue_req *mqrq)
{
kfree(mqrq->bounce_sg);
@@ -245,12 +191,166 @@
mqrq->bounce_buf = NULL;
}
-static void mmc_queue_reqs_free_bufs(struct mmc_queue *mq)
+static void mmc_queue_reqs_free_bufs(struct mmc_queue_req *mqrq, int qdepth)
{
int i;
- for (i = 0; i < mq->qdepth; i++)
- mmc_queue_req_free_bufs(&mq->mqrq[i]);
+ for (i = 0; i < qdepth; i++)
+ mmc_queue_req_free_bufs(&mqrq[i]);
+}
+
+static void mmc_queue_free_mqrqs(struct mmc_queue_req *mqrq, int qdepth)
+{
+ mmc_queue_reqs_free_bufs(mqrq, qdepth);
+ kfree(mqrq);
+}
+
+static struct mmc_queue_req *mmc_queue_alloc_mqrqs(int qdepth)
+{
+ struct mmc_queue_req *mqrq;
+ int i;
+
+ mqrq = kcalloc(qdepth, sizeof(*mqrq), GFP_KERNEL);
+ if (mqrq) {
+ for (i = 0; i < qdepth; i++)
+ mqrq[i].task_id = i;
+ }
+
+ return mqrq;
+}
+
+#ifdef CONFIG_MMC_BLOCK_BOUNCE
+static int mmc_queue_alloc_bounce_bufs(struct mmc_queue_req *mqrq, int qdepth,
+ unsigned int bouncesz)
+{
+ int i;
+
+ for (i = 0; i < qdepth; i++) {
+ mqrq[i].bounce_buf = kmalloc(bouncesz, GFP_KERNEL);
+ if (!mqrq[i].bounce_buf)
+ return -ENOMEM;
+
+ mqrq[i].sg = mmc_alloc_sg(1);
+ if (!mqrq[i].sg)
+ return -ENOMEM;
+
+ mqrq[i].bounce_sg = mmc_alloc_sg(bouncesz / 512);
+ if (!mqrq[i].bounce_sg)
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+static bool mmc_queue_alloc_bounce(struct mmc_queue_req *mqrq, int qdepth,
+ unsigned int bouncesz)
+{
+ int ret;
+
+ ret = mmc_queue_alloc_bounce_bufs(mqrq, qdepth, bouncesz);
+ if (ret)
+ mmc_queue_reqs_free_bufs(mqrq, qdepth);
+
+ return !ret;
+}
+
+static unsigned int mmc_queue_calc_bouncesz(struct mmc_host *host)
+{
+ unsigned int bouncesz = MMC_QUEUE_BOUNCESZ;
+
+ if (host->max_segs != 1)
+ return 0;
+
+ if (bouncesz > host->max_req_size)
+ bouncesz = host->max_req_size;
+ if (bouncesz > host->max_seg_size)
+ bouncesz = host->max_seg_size;
+ if (bouncesz > host->max_blk_count * 512)
+ bouncesz = host->max_blk_count * 512;
+
+ if (bouncesz <= 512)
+ return 0;
+
+ return bouncesz;
+}
+#else
+static inline bool mmc_queue_alloc_bounce(struct mmc_queue_req *mqrq,
+ int qdepth, unsigned int bouncesz)
+{
+ return false;
+}
+
+static unsigned int mmc_queue_calc_bouncesz(struct mmc_host *host)
+{
+ return 0;
+}
+#endif
+
+static int mmc_queue_alloc_sgs(struct mmc_queue_req *mqrq, int qdepth,
+ int max_segs)
+{
+ int i;
+
+ for (i = 0; i < qdepth; i++) {
+ mqrq[i].sg = mmc_alloc_sg(max_segs);
+ if (!mqrq[i].sg)
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+void mmc_queue_free_shared_queue(struct mmc_card *card)
+{
+ if (card->mqrq) {
+ mmc_queue_free_mqrqs(card->mqrq, card->qdepth);
+ card->mqrq = NULL;
+ }
+}
+
+static int __mmc_queue_alloc_shared_queue(struct mmc_card *card, int qdepth)
+{
+ struct mmc_host *host = card->host;
+ struct mmc_queue_req *mqrq;
+ unsigned int bouncesz;
+ int ret = 0;
+
+ if (card->mqrq)
+ return -EINVAL;
+
+ mqrq = mmc_queue_alloc_mqrqs(qdepth);
+ if (!mqrq)
+ return -ENOMEM;
+
+ card->mqrq = mqrq;
+ card->qdepth = qdepth;
+
+ bouncesz = mmc_queue_calc_bouncesz(host);
+
+ if (bouncesz && !mmc_queue_alloc_bounce(mqrq, qdepth, bouncesz)) {
+ bouncesz = 0;
+ pr_warn("%s: unable to allocate bounce buffers\n",
+ mmc_card_name(card));
+ }
+
+ card->bouncesz = bouncesz;
+
+ if (!bouncesz) {
+ ret = mmc_queue_alloc_sgs(mqrq, qdepth, host->max_segs);
+ if (ret)
+ goto out_err;
+ }
+
+ return ret;
+
+out_err:
+ mmc_queue_free_shared_queue(card);
+ return ret;
+}
+
+int mmc_queue_alloc_shared_queue(struct mmc_card *card)
+{
+ return __mmc_queue_alloc_shared_queue(card, 2);
}
/**
@@ -267,7 +367,6 @@
{
struct mmc_host *host = card->host;
u64 limit = BLK_BOUNCE_HIGH;
- bool bounce = false;
int ret = -ENOMEM;
if (mmc_dev(host)->dma_mask && *mmc_dev(host)->dma_mask)
@@ -278,13 +377,8 @@
if (!mq->queue)
return -ENOMEM;
- mq->qdepth = 2;
- mq->mqrq = kcalloc(mq->qdepth, sizeof(struct mmc_queue_req),
- GFP_KERNEL);
- if (!mq->mqrq)
- goto blk_cleanup;
- mq->mqrq_cur = &mq->mqrq[0];
- mq->mqrq_prev = &mq->mqrq[1];
+ mq->mqrq = card->mqrq;
+ mq->qdepth = card->qdepth;
mq->queue->queuedata = mq;
blk_queue_prep_rq(mq->queue, mmc_prep_request);
@@ -293,44 +387,17 @@
if (mmc_can_erase(card))
mmc_queue_setup_discard(mq->queue, card);
-#ifdef CONFIG_MMC_BLOCK_BOUNCE
- if (host->max_segs == 1) {
- unsigned int bouncesz;
-
- bouncesz = MMC_QUEUE_BOUNCESZ;
-
- if (bouncesz > host->max_req_size)
- bouncesz = host->max_req_size;
- if (bouncesz > host->max_seg_size)
- bouncesz = host->max_seg_size;
- if (bouncesz > (host->max_blk_count * 512))
- bouncesz = host->max_blk_count * 512;
-
- if (bouncesz > 512 &&
- mmc_queue_alloc_bounce_bufs(mq, bouncesz)) {
- blk_queue_bounce_limit(mq->queue, BLK_BOUNCE_ANY);
- blk_queue_max_hw_sectors(mq->queue, bouncesz / 512);
- blk_queue_max_segments(mq->queue, bouncesz / 512);
- blk_queue_max_segment_size(mq->queue, bouncesz);
-
- ret = mmc_queue_alloc_bounce_sgs(mq, bouncesz);
- if (ret)
- goto cleanup_queue;
- bounce = true;
- }
- }
-#endif
-
- if (!bounce) {
+ if (card->bouncesz) {
+ blk_queue_bounce_limit(mq->queue, BLK_BOUNCE_ANY);
+ blk_queue_max_hw_sectors(mq->queue, card->bouncesz / 512);
+ blk_queue_max_segments(mq->queue, card->bouncesz / 512);
+ blk_queue_max_segment_size(mq->queue, card->bouncesz);
+ } else {
blk_queue_bounce_limit(mq->queue, limit);
blk_queue_max_hw_sectors(mq->queue,
min(host->max_blk_count, host->max_req_size / 512));
blk_queue_max_segments(mq->queue, host->max_segs);
blk_queue_max_segment_size(mq->queue, host->max_seg_size);
-
- ret = mmc_queue_alloc_sgs(mq, host->max_segs);
- if (ret)
- goto cleanup_queue;
}
sema_init(&mq->thread_sem, 1);
@@ -345,11 +412,8 @@
return 0;
- cleanup_queue:
- mmc_queue_reqs_free_bufs(mq);
- kfree(mq->mqrq);
+cleanup_queue:
mq->mqrq = NULL;
-blk_cleanup:
blk_cleanup_queue(mq->queue);
return ret;
}
@@ -371,10 +435,7 @@
blk_start_queue(q);
spin_unlock_irqrestore(q->queue_lock, flags);
- mmc_queue_reqs_free_bufs(mq);
- kfree(mq->mqrq);
mq->mqrq = NULL;
-
mq->card = NULL;
}
EXPORT_SYMBOL(mmc_cleanup_queue);
diff --git a/drivers/mmc/core/queue.h b/drivers/mmc/core/queue.h
index e298f10..871796c 100644
--- a/drivers/mmc/core/queue.h
+++ b/drivers/mmc/core/queue.h
@@ -34,23 +34,25 @@
struct scatterlist *bounce_sg;
unsigned int bounce_sg_len;
struct mmc_async_req areq;
+ int task_id;
};
struct mmc_queue {
struct mmc_card *card;
struct task_struct *thread;
struct semaphore thread_sem;
- bool new_request;
bool suspended;
bool asleep;
struct mmc_blk_data *blkdata;
struct request_queue *queue;
struct mmc_queue_req *mqrq;
- struct mmc_queue_req *mqrq_cur;
- struct mmc_queue_req *mqrq_prev;
int qdepth;
+ int qcnt;
+ unsigned long qslots;
};
+extern int mmc_queue_alloc_shared_queue(struct mmc_card *card);
+extern void mmc_queue_free_shared_queue(struct mmc_card *card);
extern int mmc_init_queue(struct mmc_queue *, struct mmc_card *, spinlock_t *,
const char *);
extern void mmc_cleanup_queue(struct mmc_queue *);
@@ -64,4 +66,8 @@
extern int mmc_access_rpmb(struct mmc_queue *);
+extern struct mmc_queue_req *mmc_queue_req_find(struct mmc_queue *,
+ struct request *);
+extern void mmc_queue_req_free(struct mmc_queue *, struct mmc_queue_req *);
+
#endif
diff --git a/drivers/mmc/core/sd.c b/drivers/mmc/core/sd.c
index 89531b4..d109634 100644
--- a/drivers/mmc/core/sd.c
+++ b/drivers/mmc/core/sd.c
@@ -225,7 +225,7 @@
static int mmc_read_ssr(struct mmc_card *card)
{
unsigned int au, es, et, eo;
- u32 *raw_ssr;
+ __be32 *raw_ssr;
int i;
if (!(card->csd.cmdclass & CCC_APP_SPEC)) {
@@ -853,7 +853,7 @@
/*
* Fetch SCR from card.
*/
- err = mmc_app_send_scr(card, card->raw_scr);
+ err = mmc_app_send_scr(card);
if (err)
return err;
diff --git a/drivers/mmc/core/sd_ops.c b/drivers/mmc/core/sd_ops.c
index 9d5824a..47056d8 100644
--- a/drivers/mmc/core/sd_ops.c
+++ b/drivers/mmc/core/sd_ops.c
@@ -232,14 +232,14 @@
return 0;
}
-int mmc_app_send_scr(struct mmc_card *card, u32 *scr)
+int mmc_app_send_scr(struct mmc_card *card)
{
int err;
struct mmc_request mrq = {};
struct mmc_command cmd = {};
struct mmc_data data = {};
struct scatterlist sg;
- void *data_buf;
+ __be32 *scr;
/* NOTE: caller guarantees scr is heap-allocated */
@@ -250,8 +250,8 @@
/* dma onto stack is unsafe/nonportable, but callers to this
* routine normally provide temporary on-stack buffers ...
*/
- data_buf = kmalloc(sizeof(card->raw_scr), GFP_KERNEL);
- if (data_buf == NULL)
+ scr = kmalloc(sizeof(card->raw_scr), GFP_KERNEL);
+ if (!scr)
return -ENOMEM;
mrq.cmd = &cmd;
@@ -267,23 +267,22 @@
data.sg = &sg;
data.sg_len = 1;
- sg_init_one(&sg, data_buf, 8);
+ sg_init_one(&sg, scr, 8);
mmc_set_data_timeout(&data, card);
mmc_wait_for_req(card->host, &mrq);
- memcpy(scr, data_buf, sizeof(card->raw_scr));
- kfree(data_buf);
+ card->raw_scr[0] = be32_to_cpu(scr[0]);
+ card->raw_scr[1] = be32_to_cpu(scr[1]);
+
+ kfree(scr);
if (cmd.error)
return cmd.error;
if (data.error)
return data.error;
- scr[0] = be32_to_cpu(scr[0]);
- scr[1] = be32_to_cpu(scr[1]);
-
return 0;
}
diff --git a/drivers/mmc/core/sd_ops.h b/drivers/mmc/core/sd_ops.h
index 784f8e6..0e6c3d5 100644
--- a/drivers/mmc/core/sd_ops.h
+++ b/drivers/mmc/core/sd_ops.h
@@ -22,7 +22,7 @@
int mmc_send_app_op_cond(struct mmc_host *host, u32 ocr, u32 *rocr);
int mmc_send_if_cond(struct mmc_host *host, u32 ocr);
int mmc_send_relative_addr(struct mmc_host *host, unsigned int *rca);
-int mmc_app_send_scr(struct mmc_card *card, u32 *scr);
+int mmc_app_send_scr(struct mmc_card *card);
int mmc_sd_switch(struct mmc_card *card, int mode, int group,
u8 value, u8 *resp);
int mmc_app_sd_status(struct mmc_card *card, void *ssr);
diff --git a/drivers/mmc/core/sdio_io.c b/drivers/mmc/core/sdio_io.c
index 74195d7..d40744b 100644
--- a/drivers/mmc/core/sdio_io.c
+++ b/drivers/mmc/core/sdio_io.c
@@ -373,19 +373,16 @@
u8 val;
if (!func) {
- *err_ret = -EINVAL;
+ if (err_ret)
+ *err_ret = -EINVAL;
return 0xFF;
}
- if (err_ret)
- *err_ret = 0;
-
ret = mmc_io_rw_direct(func->card, 0, func->num, addr, 0, &val);
- if (ret) {
- if (err_ret)
- *err_ret = ret;
+ if (err_ret)
+ *err_ret = ret;
+ if (ret)
return 0xFF;
- }
return val;
}
@@ -407,7 +404,8 @@
int ret;
if (!func) {
- *err_ret = -EINVAL;
+ if (err_ret)
+ *err_ret = -EINVAL;
return;
}
@@ -441,7 +439,7 @@
if (err_ret)
*err_ret = ret;
if (ret)
- val = 0xff;
+ return 0xff;
return val;
}
@@ -529,15 +527,11 @@
{
int ret;
- if (err_ret)
- *err_ret = 0;
-
ret = sdio_memcpy_fromio(func, func->tmpbuf, addr, 2);
- if (ret) {
- if (err_ret)
- *err_ret = ret;
+ if (err_ret)
+ *err_ret = ret;
+ if (ret)
return 0xFFFF;
- }
return le16_to_cpup((__le16 *)func->tmpbuf);
}
@@ -581,15 +575,11 @@
{
int ret;
- if (err_ret)
- *err_ret = 0;
-
ret = sdio_memcpy_fromio(func, func->tmpbuf, addr, 4);
- if (ret) {
- if (err_ret)
- *err_ret = ret;
+ if (err_ret)
+ *err_ret = ret;
+ if (ret)
return 0xFFFFFFFF;
- }
return le32_to_cpup((__le32 *)func->tmpbuf);
}
@@ -635,19 +625,16 @@
unsigned char val;
if (!func) {
- *err_ret = -EINVAL;
+ if (err_ret)
+ *err_ret = -EINVAL;
return 0xFF;
}
- if (err_ret)
- *err_ret = 0;
-
ret = mmc_io_rw_direct(func->card, 0, 0, addr, 0, &val);
- if (ret) {
- if (err_ret)
- *err_ret = ret;
+ if (err_ret)
+ *err_ret = ret;
+ if (ret)
return 0xFF;
- }
return val;
}
@@ -673,7 +660,8 @@
int ret;
if (!func) {
- *err_ret = -EINVAL;
+ if (err_ret)
+ *err_ret = -EINVAL;
return;
}
diff --git a/drivers/mmc/core/sdio_ops.c b/drivers/mmc/core/sdio_ops.c
index 3c0d3ab..abaaba3 100644
--- a/drivers/mmc/core/sdio_ops.c
+++ b/drivers/mmc/core/sdio_ops.c
@@ -152,7 +152,7 @@
data.flags = write ? MMC_DATA_WRITE : MMC_DATA_READ;
left_size = data.blksz * data.blocks;
- nents = (left_size - 1) / seg_size + 1;
+ nents = DIV_ROUND_UP(left_size, seg_size);
if (nents > 1) {
if (sg_alloc_table(&sgtable, nents, GFP_KERNEL))
return -ENOMEM;
@@ -161,10 +161,9 @@
data.sg_len = nents;
for_each_sg(data.sg, sg_ptr, data.sg_len, i) {
- sg_set_page(sg_ptr, virt_to_page(buf + (i * seg_size)),
- min(seg_size, left_size),
- offset_in_page(buf + (i * seg_size)));
- left_size = left_size - seg_size;
+ sg_set_buf(sg_ptr, buf + i * seg_size,
+ min(seg_size, left_size));
+ left_size -= seg_size;
}
} else {
data.sg = &sg;
diff --git a/drivers/mmc/core/sdio_ops.h b/drivers/mmc/core/sdio_ops.h
index bed8a83..ee35cb4 100644
--- a/drivers/mmc/core/sdio_ops.h
+++ b/drivers/mmc/core/sdio_ops.h
@@ -26,9 +26,15 @@
int sdio_reset(struct mmc_host *host);
unsigned int mmc_align_data_size(struct mmc_card *card, unsigned int sz);
-static inline bool mmc_is_io_op(u32 opcode)
+static inline bool sdio_is_io_busy(u32 opcode, u32 arg)
{
- return opcode == SD_IO_RW_DIRECT || opcode == SD_IO_RW_EXTENDED;
+ u32 addr;
+
+ addr = (arg >> 9) & 0x1FFFF;
+
+ return (opcode == SD_IO_RW_EXTENDED ||
+ (opcode == SD_IO_RW_DIRECT &&
+ !(addr == SDIO_CCCR_ABORT || addr == SDIO_CCCR_SUSPEND)));
}
#endif
diff --git a/drivers/mmc/host/Kconfig b/drivers/mmc/host/Kconfig
index f08691a..2db84dd 100644
--- a/drivers/mmc/host/Kconfig
+++ b/drivers/mmc/host/Kconfig
@@ -622,6 +622,27 @@
help
If you say yes here SD-Cards may work on the EZkit.
+config MMC_CAVIUM_OCTEON
+ tristate "Cavium OCTEON SD/MMC Card Interface support"
+ depends on CAVIUM_OCTEON_SOC
+ help
+ This selects Cavium OCTEON SD/MMC card Interface.
+ If you have an OCTEON board with a Multimedia Card slot,
+ say Y or M here.
+
+ If unsure, say N.
+
+config MMC_CAVIUM_THUNDERX
+ tristate "Cavium ThunderX SD/MMC Card Interface support"
+ depends on PCI && 64BIT && (ARM64 || COMPILE_TEST)
+ depends on GPIOLIB
+ depends on OF_ADDRESS
+ help
+ This selects Cavium ThunderX SD/MMC Card Interface.
+ If you have an Cavium ARM64 board with a Multimedia Card slot
+ or builtin eMMC chip say Y or M here. If built as a module
+ the module will be called thunderx_mmc.ko.
+
config MMC_DW
tristate "Synopsys DesignWare Memory Card Interface"
depends on HAS_DMA
@@ -799,6 +820,20 @@
depends on PCI
help
+config MMC_BCM2835
+ tristate "Broadcom BCM2835 SDHOST MMC Controller support"
+ depends on ARCH_BCM2835 || COMPILE_TEST
+ depends on HAS_DMA
+ help
+ This selects the BCM2835 SDHOST MMC controller. If you have
+ a BCM2835 platform with SD or MMC devices, say Y or M here.
+
+ Note that the BCM2835 has two SD controllers: The Arasan
+ sdhci controller (supported by MMC_SDHCI_IPROC) and a custom
+ sdhost controller (supported by this driver).
+
+ If unsure, say N.
+
config MMC_MTK
tristate "MediaTek SD/MMC Card Interface support"
depends on HAS_DMA
@@ -828,3 +863,11 @@
Broadcom STB SoCs.
If unsure, say Y.
+
+config MMC_SDHCI_XENON
+ tristate "Marvell Xenon eMMC/SD/SDIO SDHCI driver"
+ depends on MMC_SDHCI_PLTFM
+ help
+ This selects Marvell Xenon eMMC/SD/SDIO SDHCI.
+ If you have a controller with this interface, say Y or M here.
+ If unsure, say N.
diff --git a/drivers/mmc/host/Makefile b/drivers/mmc/host/Makefile
index 6d548c4..926347c 100644
--- a/drivers/mmc/host/Makefile
+++ b/drivers/mmc/host/Makefile
@@ -42,6 +42,10 @@
obj-$(CONFIG_MMC_CB710) += cb710-mmc.o
obj-$(CONFIG_MMC_VIA_SDMMC) += via-sdmmc.o
obj-$(CONFIG_SDH_BFIN) += bfin_sdh.o
+octeon-mmc-objs := cavium.o cavium-octeon.o
+obj-$(CONFIG_MMC_CAVIUM_OCTEON) += octeon-mmc.o
+thunderx-mmc-objs := cavium.o cavium-thunderx.o
+obj-$(CONFIG_MMC_CAVIUM_THUNDERX) += thunderx-mmc.o
obj-$(CONFIG_MMC_DW) += dw_mmc.o
obj-$(CONFIG_MMC_DW_PLTFM) += dw_mmc-pltfm.o
obj-$(CONFIG_MMC_DW_EXYNOS) += dw_mmc-exynos.o
@@ -59,6 +63,7 @@
obj-$(CONFIG_MMC_SUNXI) += sunxi-mmc.o
obj-$(CONFIG_MMC_USDHI6ROL0) += usdhi6rol0.o
obj-$(CONFIG_MMC_TOSHIBA_PCI) += toshsd.o
+obj-$(CONFIG_MMC_BCM2835) += bcm2835.o
obj-$(CONFIG_MMC_REALTEK_PCI) += rtsx_pci_sdmmc.o
obj-$(CONFIG_MMC_REALTEK_USB) += rtsx_usb_sdmmc.o
@@ -83,3 +88,6 @@
ifeq ($(CONFIG_CB710_DEBUG),y)
CFLAGS-cb710-mmc += -DDEBUG
endif
+
+obj-$(CONFIG_MMC_SDHCI_XENON) += sdhci-xenon-driver.o
+sdhci-xenon-driver-y += sdhci-xenon.o sdhci-xenon-phy.o
diff --git a/drivers/mmc/host/android-goldfish.c b/drivers/mmc/host/android-goldfish.c
index 590a8a4..5b3e1c9 100644
--- a/drivers/mmc/host/android-goldfish.c
+++ b/drivers/mmc/host/android-goldfish.c
@@ -212,10 +212,7 @@
if (host->dma_in_use) {
enum dma_data_direction dma_data_dir;
- if (data->flags & MMC_DATA_WRITE)
- dma_data_dir = DMA_TO_DEVICE;
- else
- dma_data_dir = DMA_FROM_DEVICE;
+ dma_data_dir = mmc_get_dma_dir(data);
if (dma_data_dir == DMA_FROM_DEVICE) {
/*
@@ -390,10 +387,7 @@
*/
sg_len = (data->blocks == 1) ? 1 : data->sg_len;
- if (data->flags & MMC_DATA_WRITE)
- dma_data_dir = DMA_TO_DEVICE;
- else
- dma_data_dir = DMA_FROM_DEVICE;
+ dma_data_dir = mmc_get_dma_dir(data);
host->sg_len = dma_map_sg(mmc_dev(host->mmc), data->sg,
sg_len, dma_data_dir);
diff --git a/drivers/mmc/host/atmel-mci.c b/drivers/mmc/host/atmel-mci.c
index 0ad8ef5..388e4a3 100644
--- a/drivers/mmc/host/atmel-mci.c
+++ b/drivers/mmc/host/atmel-mci.c
@@ -954,8 +954,7 @@
if (data)
dma_unmap_sg(&host->pdev->dev,
data->sg, data->sg_len,
- ((data->flags & MMC_DATA_WRITE)
- ? DMA_TO_DEVICE : DMA_FROM_DEVICE));
+ mmc_get_dma_dir(data));
}
/*
@@ -993,8 +992,7 @@
if (data)
dma_unmap_sg(host->dma.chan->device->dev,
data->sg, data->sg_len,
- ((data->flags & MMC_DATA_WRITE)
- ? DMA_TO_DEVICE : DMA_FROM_DEVICE));
+ mmc_get_dma_dir(data));
}
/*
@@ -1095,7 +1093,6 @@
{
u32 iflags, tmp;
unsigned int sg_len;
- enum dma_data_direction dir;
int i;
data->error = -EINPROGRESS;
@@ -1107,13 +1104,10 @@
/* Enable pdc mode */
atmci_writel(host, ATMCI_MR, host->mode_reg | ATMCI_MR_PDCMODE);
- if (data->flags & MMC_DATA_READ) {
- dir = DMA_FROM_DEVICE;
+ if (data->flags & MMC_DATA_READ)
iflags |= ATMCI_ENDRX | ATMCI_RXBUFF;
- } else {
- dir = DMA_TO_DEVICE;
+ else
iflags |= ATMCI_ENDTX | ATMCI_TXBUFE | ATMCI_BLKE;
- }
/* Set BLKLEN */
tmp = atmci_readl(host, ATMCI_MR);
@@ -1123,7 +1117,8 @@
/* Configure PDC */
host->data_size = data->blocks * data->blksz;
- sg_len = dma_map_sg(&host->pdev->dev, data->sg, data->sg_len, dir);
+ sg_len = dma_map_sg(&host->pdev->dev, data->sg, data->sg_len,
+ mmc_get_dma_dir(data));
if ((!host->caps.has_rwproof)
&& (host->data->flags & MMC_DATA_WRITE)) {
@@ -1135,9 +1130,8 @@
}
if (host->data_size)
- atmci_pdc_set_both_buf(host,
- ((dir == DMA_FROM_DEVICE) ? XFER_RECEIVE : XFER_TRANSMIT));
-
+ atmci_pdc_set_both_buf(host, data->flags & MMC_DATA_READ ?
+ XFER_RECEIVE : XFER_TRANSMIT);
return iflags;
}
@@ -1148,7 +1142,6 @@
struct dma_async_tx_descriptor *desc;
struct scatterlist *sg;
unsigned int i;
- enum dma_data_direction direction;
enum dma_transfer_direction slave_dirn;
unsigned int sglen;
u32 maxburst;
@@ -1186,12 +1179,10 @@
return -ENODEV;
if (data->flags & MMC_DATA_READ) {
- direction = DMA_FROM_DEVICE;
host->dma_conf.direction = slave_dirn = DMA_DEV_TO_MEM;
maxburst = atmci_convert_chksize(host,
host->dma_conf.src_maxburst);
} else {
- direction = DMA_TO_DEVICE;
host->dma_conf.direction = slave_dirn = DMA_MEM_TO_DEV;
maxburst = atmci_convert_chksize(host,
host->dma_conf.dst_maxburst);
@@ -1202,7 +1193,7 @@
ATMCI_DMAEN);
sglen = dma_map_sg(chan->device->dev, data->sg,
- data->sg_len, direction);
+ data->sg_len, mmc_get_dma_dir(data));
dmaengine_slave_config(chan, &host->dma_conf);
desc = dmaengine_prep_slave_sg(chan,
@@ -1217,7 +1208,8 @@
return iflags;
unmap_exit:
- dma_unmap_sg(chan->device->dev, data->sg, data->sg_len, direction);
+ dma_unmap_sg(chan->device->dev, data->sg, data->sg_len,
+ mmc_get_dma_dir(data));
return -ENOMEM;
}
diff --git a/drivers/mmc/host/bcm2835.c b/drivers/mmc/host/bcm2835.c
new file mode 100644
index 0000000..1f343a4
--- /dev/null
+++ b/drivers/mmc/host/bcm2835.c
@@ -0,0 +1,1466 @@
+/*
+ * bcm2835 sdhost driver.
+ *
+ * The 2835 has two SD controllers: The Arasan sdhci controller
+ * (supported by the iproc driver) and a custom sdhost controller
+ * (supported by this driver).
+ *
+ * The sdhci controller supports both sdcard and sdio. The sdhost
+ * controller supports the sdcard only, but has better performance.
+ * Also note that the rpi3 has sdio wifi, so driving the sdcard with
+ * the sdhost controller allows to use the sdhci controller for wifi
+ * support.
+ *
+ * The configuration is done by devicetree via pin muxing. Both
+ * SD controller are available on the same pins (2 pin groups = pin 22
+ * to 27 + pin 48 to 53). So it's possible to use both SD controllers
+ * at the same time with different pin groups.
+ *
+ * Author: Phil Elwell <phil@raspberrypi.org>
+ * Copyright (C) 2015-2016 Raspberry Pi (Trading) Ltd.
+ *
+ * Based on
+ * mmc-bcm2835.c by Gellert Weisz
+ * which is, in turn, based on
+ * sdhci-bcm2708.c by Broadcom
+ * sdhci-bcm2835.c by Stephen Warren and Oleksandr Tymoshenko
+ * sdhci.c and sdhci-pci.c by Pierre Ossman
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/dmaengine.h>
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/highmem.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/iopoll.h>
+#include <linux/module.h>
+#include <linux/of_address.h>
+#include <linux/of_irq.h>
+#include <linux/platform_device.h>
+#include <linux/scatterlist.h>
+#include <linux/time.h>
+#include <linux/workqueue.h>
+
+#include <linux/mmc/host.h>
+#include <linux/mmc/mmc.h>
+#include <linux/mmc/sd.h>
+
+#define SDCMD 0x00 /* Command to SD card - 16 R/W */
+#define SDARG 0x04 /* Argument to SD card - 32 R/W */
+#define SDTOUT 0x08 /* Start value for timeout counter - 32 R/W */
+#define SDCDIV 0x0c /* Start value for clock divider - 11 R/W */
+#define SDRSP0 0x10 /* SD card response (31:0) - 32 R */
+#define SDRSP1 0x14 /* SD card response (63:32) - 32 R */
+#define SDRSP2 0x18 /* SD card response (95:64) - 32 R */
+#define SDRSP3 0x1c /* SD card response (127:96) - 32 R */
+#define SDHSTS 0x20 /* SD host status - 11 R/W */
+#define SDVDD 0x30 /* SD card power control - 1 R/W */
+#define SDEDM 0x34 /* Emergency Debug Mode - 13 R/W */
+#define SDHCFG 0x38 /* Host configuration - 2 R/W */
+#define SDHBCT 0x3c /* Host byte count (debug) - 32 R/W */
+#define SDDATA 0x40 /* Data to/from SD card - 32 R/W */
+#define SDHBLC 0x50 /* Host block count (SDIO/SDHC) - 9 R/W */
+
+#define SDCMD_NEW_FLAG 0x8000
+#define SDCMD_FAIL_FLAG 0x4000
+#define SDCMD_BUSYWAIT 0x800
+#define SDCMD_NO_RESPONSE 0x400
+#define SDCMD_LONG_RESPONSE 0x200
+#define SDCMD_WRITE_CMD 0x80
+#define SDCMD_READ_CMD 0x40
+#define SDCMD_CMD_MASK 0x3f
+
+#define SDCDIV_MAX_CDIV 0x7ff
+
+#define SDHSTS_BUSY_IRPT 0x400
+#define SDHSTS_BLOCK_IRPT 0x200
+#define SDHSTS_SDIO_IRPT 0x100
+#define SDHSTS_REW_TIME_OUT 0x80
+#define SDHSTS_CMD_TIME_OUT 0x40
+#define SDHSTS_CRC16_ERROR 0x20
+#define SDHSTS_CRC7_ERROR 0x10
+#define SDHSTS_FIFO_ERROR 0x08
+/* Reserved */
+/* Reserved */
+#define SDHSTS_DATA_FLAG 0x01
+
+#define SDHSTS_TRANSFER_ERROR_MASK (SDHSTS_CRC7_ERROR | \
+ SDHSTS_CRC16_ERROR | \
+ SDHSTS_REW_TIME_OUT | \
+ SDHSTS_FIFO_ERROR)
+
+#define SDHSTS_ERROR_MASK (SDHSTS_CMD_TIME_OUT | \
+ SDHSTS_TRANSFER_ERROR_MASK)
+
+#define SDHCFG_BUSY_IRPT_EN BIT(10)
+#define SDHCFG_BLOCK_IRPT_EN BIT(8)
+#define SDHCFG_SDIO_IRPT_EN BIT(5)
+#define SDHCFG_DATA_IRPT_EN BIT(4)
+#define SDHCFG_SLOW_CARD BIT(3)
+#define SDHCFG_WIDE_EXT_BUS BIT(2)
+#define SDHCFG_WIDE_INT_BUS BIT(1)
+#define SDHCFG_REL_CMD_LINE BIT(0)
+
+#define SDVDD_POWER_OFF 0
+#define SDVDD_POWER_ON 1
+
+#define SDEDM_FORCE_DATA_MODE BIT(19)
+#define SDEDM_CLOCK_PULSE BIT(20)
+#define SDEDM_BYPASS BIT(21)
+
+#define SDEDM_WRITE_THRESHOLD_SHIFT 9
+#define SDEDM_READ_THRESHOLD_SHIFT 14
+#define SDEDM_THRESHOLD_MASK 0x1f
+
+#define SDEDM_FSM_MASK 0xf
+#define SDEDM_FSM_IDENTMODE 0x0
+#define SDEDM_FSM_DATAMODE 0x1
+#define SDEDM_FSM_READDATA 0x2
+#define SDEDM_FSM_WRITEDATA 0x3
+#define SDEDM_FSM_READWAIT 0x4
+#define SDEDM_FSM_READCRC 0x5
+#define SDEDM_FSM_WRITECRC 0x6
+#define SDEDM_FSM_WRITEWAIT1 0x7
+#define SDEDM_FSM_POWERDOWN 0x8
+#define SDEDM_FSM_POWERUP 0x9
+#define SDEDM_FSM_WRITESTART1 0xa
+#define SDEDM_FSM_WRITESTART2 0xb
+#define SDEDM_FSM_GENPULSES 0xc
+#define SDEDM_FSM_WRITEWAIT2 0xd
+#define SDEDM_FSM_STARTPOWDOWN 0xf
+
+#define SDDATA_FIFO_WORDS 16
+
+#define FIFO_READ_THRESHOLD 4
+#define FIFO_WRITE_THRESHOLD 4
+#define SDDATA_FIFO_PIO_BURST 8
+
+#define PIO_THRESHOLD 1 /* Maximum block count for PIO (0 = always DMA) */
+
+struct bcm2835_host {
+ spinlock_t lock;
+ struct mutex mutex;
+
+ void __iomem *ioaddr;
+ u32 phys_addr;
+
+ struct mmc_host *mmc;
+ struct platform_device *pdev;
+
+ int clock; /* Current clock speed */
+ unsigned int max_clk; /* Max possible freq */
+ struct work_struct dma_work;
+ struct delayed_work timeout_work; /* Timer for timeouts */
+ struct sg_mapping_iter sg_miter; /* SG state for PIO */
+ unsigned int blocks; /* remaining PIO blocks */
+ int irq; /* Device IRQ */
+
+ u32 ns_per_fifo_word;
+
+ /* cached registers */
+ u32 hcfg;
+ u32 cdiv;
+
+ struct mmc_request *mrq; /* Current request */
+ struct mmc_command *cmd; /* Current command */
+ struct mmc_data *data; /* Current data request */
+ bool data_complete:1;/* Data finished before cmd */
+ bool use_busy:1; /* Wait for busy interrupt */
+ bool use_sbc:1; /* Send CMD23 */
+
+ /* for threaded irq handler */
+ bool irq_block;
+ bool irq_busy;
+ bool irq_data;
+
+ /* DMA part */
+ struct dma_chan *dma_chan_rxtx;
+ struct dma_chan *dma_chan;
+ struct dma_slave_config dma_cfg_rx;
+ struct dma_slave_config dma_cfg_tx;
+ struct dma_async_tx_descriptor *dma_desc;
+ u32 dma_dir;
+ u32 drain_words;
+ struct page *drain_page;
+ u32 drain_offset;
+ bool use_dma;
+};
+
+static void bcm2835_dumpcmd(struct bcm2835_host *host, struct mmc_command *cmd,
+ const char *label)
+{
+ struct device *dev = &host->pdev->dev;
+
+ if (!cmd)
+ return;
+
+ dev_dbg(dev, "%c%s op %d arg 0x%x flags 0x%x - resp %08x %08x %08x %08x, err %d\n",
+ (cmd == host->cmd) ? '>' : ' ',
+ label, cmd->opcode, cmd->arg, cmd->flags,
+ cmd->resp[0], cmd->resp[1], cmd->resp[2], cmd->resp[3],
+ cmd->error);
+}
+
+static void bcm2835_dumpregs(struct bcm2835_host *host)
+{
+ struct mmc_request *mrq = host->mrq;
+ struct device *dev = &host->pdev->dev;
+
+ if (mrq) {
+ bcm2835_dumpcmd(host, mrq->sbc, "sbc");
+ bcm2835_dumpcmd(host, mrq->cmd, "cmd");
+ if (mrq->data) {
+ dev_dbg(dev, "data blocks %x blksz %x - err %d\n",
+ mrq->data->blocks,
+ mrq->data->blksz,
+ mrq->data->error);
+ }
+ bcm2835_dumpcmd(host, mrq->stop, "stop");
+ }
+
+ dev_dbg(dev, "=========== REGISTER DUMP ===========\n");
+ dev_dbg(dev, "SDCMD 0x%08x\n", readl(host->ioaddr + SDCMD));
+ dev_dbg(dev, "SDARG 0x%08x\n", readl(host->ioaddr + SDARG));
+ dev_dbg(dev, "SDTOUT 0x%08x\n", readl(host->ioaddr + SDTOUT));
+ dev_dbg(dev, "SDCDIV 0x%08x\n", readl(host->ioaddr + SDCDIV));
+ dev_dbg(dev, "SDRSP0 0x%08x\n", readl(host->ioaddr + SDRSP0));
+ dev_dbg(dev, "SDRSP1 0x%08x\n", readl(host->ioaddr + SDRSP1));
+ dev_dbg(dev, "SDRSP2 0x%08x\n", readl(host->ioaddr + SDRSP2));
+ dev_dbg(dev, "SDRSP3 0x%08x\n", readl(host->ioaddr + SDRSP3));
+ dev_dbg(dev, "SDHSTS 0x%08x\n", readl(host->ioaddr + SDHSTS));
+ dev_dbg(dev, "SDVDD 0x%08x\n", readl(host->ioaddr + SDVDD));
+ dev_dbg(dev, "SDEDM 0x%08x\n", readl(host->ioaddr + SDEDM));
+ dev_dbg(dev, "SDHCFG 0x%08x\n", readl(host->ioaddr + SDHCFG));
+ dev_dbg(dev, "SDHBCT 0x%08x\n", readl(host->ioaddr + SDHBCT));
+ dev_dbg(dev, "SDHBLC 0x%08x\n", readl(host->ioaddr + SDHBLC));
+ dev_dbg(dev, "===========================================\n");
+}
+
+static void bcm2835_reset_internal(struct bcm2835_host *host)
+{
+ u32 temp;
+
+ writel(SDVDD_POWER_OFF, host->ioaddr + SDVDD);
+ writel(0, host->ioaddr + SDCMD);
+ writel(0, host->ioaddr + SDARG);
+ writel(0xf00000, host->ioaddr + SDTOUT);
+ writel(0, host->ioaddr + SDCDIV);
+ writel(0x7f8, host->ioaddr + SDHSTS); /* Write 1s to clear */
+ writel(0, host->ioaddr + SDHCFG);
+ writel(0, host->ioaddr + SDHBCT);
+ writel(0, host->ioaddr + SDHBLC);
+
+ /* Limit fifo usage due to silicon bug */
+ temp = readl(host->ioaddr + SDEDM);
+ temp &= ~((SDEDM_THRESHOLD_MASK << SDEDM_READ_THRESHOLD_SHIFT) |
+ (SDEDM_THRESHOLD_MASK << SDEDM_WRITE_THRESHOLD_SHIFT));
+ temp |= (FIFO_READ_THRESHOLD << SDEDM_READ_THRESHOLD_SHIFT) |
+ (FIFO_WRITE_THRESHOLD << SDEDM_WRITE_THRESHOLD_SHIFT);
+ writel(temp, host->ioaddr + SDEDM);
+ msleep(20);
+ writel(SDVDD_POWER_ON, host->ioaddr + SDVDD);
+ msleep(20);
+ host->clock = 0;
+ writel(host->hcfg, host->ioaddr + SDHCFG);
+ writel(host->cdiv, host->ioaddr + SDCDIV);
+}
+
+static void bcm2835_reset(struct mmc_host *mmc)
+{
+ struct bcm2835_host *host = mmc_priv(mmc);
+
+ if (host->dma_chan)
+ dmaengine_terminate_sync(host->dma_chan);
+ bcm2835_reset_internal(host);
+}
+
+static void bcm2835_finish_command(struct bcm2835_host *host);
+
+static void bcm2835_wait_transfer_complete(struct bcm2835_host *host)
+{
+ int timediff;
+ u32 alternate_idle;
+
+ alternate_idle = (host->mrq->data->flags & MMC_DATA_READ) ?
+ SDEDM_FSM_READWAIT : SDEDM_FSM_WRITESTART1;
+
+ timediff = 0;
+
+ while (1) {
+ u32 edm, fsm;
+
+ edm = readl(host->ioaddr + SDEDM);
+ fsm = edm & SDEDM_FSM_MASK;
+
+ if ((fsm == SDEDM_FSM_IDENTMODE) ||
+ (fsm == SDEDM_FSM_DATAMODE))
+ break;
+ if (fsm == alternate_idle) {
+ writel(edm | SDEDM_FORCE_DATA_MODE,
+ host->ioaddr + SDEDM);
+ break;
+ }
+
+ timediff++;
+ if (timediff == 100000) {
+ dev_err(&host->pdev->dev,
+ "wait_transfer_complete - still waiting after %d retries\n",
+ timediff);
+ bcm2835_dumpregs(host);
+ host->mrq->data->error = -ETIMEDOUT;
+ return;
+ }
+ cpu_relax();
+ }
+}
+
+static void bcm2835_dma_complete(void *param)
+{
+ struct bcm2835_host *host = param;
+
+ schedule_work(&host->dma_work);
+}
+
+static void bcm2835_transfer_block_pio(struct bcm2835_host *host, bool is_read)
+{
+ unsigned long flags;
+ size_t blksize;
+ unsigned long wait_max;
+
+ blksize = host->data->blksz;
+
+ wait_max = jiffies + msecs_to_jiffies(500);
+
+ local_irq_save(flags);
+
+ while (blksize) {
+ int copy_words;
+ u32 hsts = 0;
+ size_t len;
+ u32 *buf;
+
+ if (!sg_miter_next(&host->sg_miter)) {
+ host->data->error = -EINVAL;
+ break;
+ }
+
+ len = min(host->sg_miter.length, blksize);
+ if (len % 4) {
+ host->data->error = -EINVAL;
+ break;
+ }
+
+ blksize -= len;
+ host->sg_miter.consumed = len;
+
+ buf = (u32 *)host->sg_miter.addr;
+
+ copy_words = len / 4;
+
+ while (copy_words) {
+ int burst_words, words;
+ u32 edm;
+
+ burst_words = min(SDDATA_FIFO_PIO_BURST, copy_words);
+ edm = readl(host->ioaddr + SDEDM);
+ if (is_read)
+ words = ((edm >> 4) & 0x1f);
+ else
+ words = SDDATA_FIFO_WORDS - ((edm >> 4) & 0x1f);
+
+ if (words < burst_words) {
+ int fsm_state = (edm & SDEDM_FSM_MASK);
+ struct device *dev = &host->pdev->dev;
+
+ if ((is_read &&
+ (fsm_state != SDEDM_FSM_READDATA &&
+ fsm_state != SDEDM_FSM_READWAIT &&
+ fsm_state != SDEDM_FSM_READCRC)) ||
+ (!is_read &&
+ (fsm_state != SDEDM_FSM_WRITEDATA &&
+ fsm_state != SDEDM_FSM_WRITESTART1 &&
+ fsm_state != SDEDM_FSM_WRITESTART2))) {
+ hsts = readl(host->ioaddr + SDHSTS);
+ dev_err(dev, "fsm %x, hsts %08x\n",
+ fsm_state, hsts);
+ if (hsts & SDHSTS_ERROR_MASK)
+ break;
+ }
+
+ if (time_after(jiffies, wait_max)) {
+ dev_err(dev, "PIO %s timeout - EDM %08x\n",
+ is_read ? "read" : "write",
+ edm);
+ hsts = SDHSTS_REW_TIME_OUT;
+ break;
+ }
+ ndelay((burst_words - words) *
+ host->ns_per_fifo_word);
+ continue;
+ } else if (words > copy_words) {
+ words = copy_words;
+ }
+
+ copy_words -= words;
+
+ while (words) {
+ if (is_read)
+ *(buf++) = readl(host->ioaddr + SDDATA);
+ else
+ writel(*(buf++), host->ioaddr + SDDATA);
+ words--;
+ }
+ }
+
+ if (hsts & SDHSTS_ERROR_MASK)
+ break;
+ }
+
+ sg_miter_stop(&host->sg_miter);
+
+ local_irq_restore(flags);
+}
+
+static void bcm2835_transfer_pio(struct bcm2835_host *host)
+{
+ struct device *dev = &host->pdev->dev;
+ u32 sdhsts;
+ bool is_read;
+
+ is_read = (host->data->flags & MMC_DATA_READ) != 0;
+ bcm2835_transfer_block_pio(host, is_read);
+
+ sdhsts = readl(host->ioaddr + SDHSTS);
+ if (sdhsts & (SDHSTS_CRC16_ERROR |
+ SDHSTS_CRC7_ERROR |
+ SDHSTS_FIFO_ERROR)) {
+ dev_err(dev, "%s transfer error - HSTS %08x\n",
+ is_read ? "read" : "write", sdhsts);
+ host->data->error = -EILSEQ;
+ } else if ((sdhsts & (SDHSTS_CMD_TIME_OUT |
+ SDHSTS_REW_TIME_OUT))) {
+ dev_err(dev, "%s timeout error - HSTS %08x\n",
+ is_read ? "read" : "write", sdhsts);
+ host->data->error = -ETIMEDOUT;
+ }
+}
+
+static
+void bcm2835_prepare_dma(struct bcm2835_host *host, struct mmc_data *data)
+{
+ int len, dir_data, dir_slave;
+ struct dma_async_tx_descriptor *desc = NULL;
+ struct dma_chan *dma_chan;
+
+ dma_chan = host->dma_chan_rxtx;
+ if (data->flags & MMC_DATA_READ) {
+ dir_data = DMA_FROM_DEVICE;
+ dir_slave = DMA_DEV_TO_MEM;
+ } else {
+ dir_data = DMA_TO_DEVICE;
+ dir_slave = DMA_MEM_TO_DEV;
+ }
+
+ /* The block doesn't manage the FIFO DREQs properly for
+ * multi-block transfers, so don't attempt to DMA the final
+ * few words. Unfortunately this requires the final sg entry
+ * to be trimmed. N.B. This code demands that the overspill
+ * is contained in a single sg entry.
+ */
+
+ host->drain_words = 0;
+ if ((data->blocks > 1) && (dir_data == DMA_FROM_DEVICE)) {
+ struct scatterlist *sg;
+ u32 len;
+ int i;
+
+ len = min((u32)(FIFO_READ_THRESHOLD - 1) * 4,
+ (u32)data->blocks * data->blksz);
+
+ for_each_sg(data->sg, sg, data->sg_len, i) {
+ if (sg_is_last(sg)) {
+ WARN_ON(sg->length < len);
+ sg->length -= len;
+ host->drain_page = sg_page(sg);
+ host->drain_offset = sg->offset + sg->length;
+ }
+ }
+ host->drain_words = len / 4;
+ }
+
+ /* The parameters have already been validated, so this will not fail */
+ (void)dmaengine_slave_config(dma_chan,
+ (dir_data == DMA_FROM_DEVICE) ?
+ &host->dma_cfg_rx :
+ &host->dma_cfg_tx);
+
+ len = dma_map_sg(dma_chan->device->dev, data->sg, data->sg_len,
+ dir_data);
+
+ if (len > 0) {
+ desc = dmaengine_prep_slave_sg(dma_chan, data->sg,
+ len, dir_slave,
+ DMA_PREP_INTERRUPT |
+ DMA_CTRL_ACK);
+ }
+
+ if (desc) {
+ desc->callback = bcm2835_dma_complete;
+ desc->callback_param = host;
+ host->dma_desc = desc;
+ host->dma_chan = dma_chan;
+ host->dma_dir = dir_data;
+ }
+}
+
+static void bcm2835_start_dma(struct bcm2835_host *host)
+{
+ dmaengine_submit(host->dma_desc);
+ dma_async_issue_pending(host->dma_chan);
+}
+
+static void bcm2835_set_transfer_irqs(struct bcm2835_host *host)
+{
+ u32 all_irqs = SDHCFG_DATA_IRPT_EN | SDHCFG_BLOCK_IRPT_EN |
+ SDHCFG_BUSY_IRPT_EN;
+
+ if (host->dma_desc) {
+ host->hcfg = (host->hcfg & ~all_irqs) |
+ SDHCFG_BUSY_IRPT_EN;
+ } else {
+ host->hcfg = (host->hcfg & ~all_irqs) |
+ SDHCFG_DATA_IRPT_EN |
+ SDHCFG_BUSY_IRPT_EN;
+ }
+
+ writel(host->hcfg, host->ioaddr + SDHCFG);
+}
+
+static
+void bcm2835_prepare_data(struct bcm2835_host *host, struct mmc_command *cmd)
+{
+ struct mmc_data *data = cmd->data;
+
+ WARN_ON(host->data);
+
+ host->data = data;
+ if (!data)
+ return;
+
+ host->data_complete = false;
+ host->data->bytes_xfered = 0;
+
+ if (!host->dma_desc) {
+ /* Use PIO */
+ int flags = SG_MITER_ATOMIC;
+
+ if (data->flags & MMC_DATA_READ)
+ flags |= SG_MITER_TO_SG;
+ else
+ flags |= SG_MITER_FROM_SG;
+ sg_miter_start(&host->sg_miter, data->sg, data->sg_len, flags);
+ host->blocks = data->blocks;
+ }
+
+ bcm2835_set_transfer_irqs(host);
+
+ writel(data->blksz, host->ioaddr + SDHBCT);
+ writel(data->blocks, host->ioaddr + SDHBLC);
+}
+
+static u32 bcm2835_read_wait_sdcmd(struct bcm2835_host *host, u32 max_ms)
+{
+ struct device *dev = &host->pdev->dev;
+ u32 value;
+ int ret;
+
+ ret = readl_poll_timeout(host->ioaddr + SDCMD, value,
+ !(value & SDCMD_NEW_FLAG), 1, 10);
+ if (ret == -ETIMEDOUT)
+ /* if it takes a while make poll interval bigger */
+ ret = readl_poll_timeout(host->ioaddr + SDCMD, value,
+ !(value & SDCMD_NEW_FLAG),
+ 10, max_ms * 1000);
+ if (ret == -ETIMEDOUT)
+ dev_err(dev, "%s: timeout (%d ms)\n", __func__, max_ms);
+
+ return value;
+}
+
+static void bcm2835_finish_request(struct bcm2835_host *host)
+{
+ struct dma_chan *terminate_chan = NULL;
+ struct mmc_request *mrq;
+
+ cancel_delayed_work(&host->timeout_work);
+
+ mrq = host->mrq;
+
+ host->mrq = NULL;
+ host->cmd = NULL;
+ host->data = NULL;
+
+ host->dma_desc = NULL;
+ terminate_chan = host->dma_chan;
+ host->dma_chan = NULL;
+
+ if (terminate_chan) {
+ int err = dmaengine_terminate_all(terminate_chan);
+
+ if (err)
+ dev_err(&host->pdev->dev,
+ "failed to terminate DMA (%d)\n", err);
+ }
+
+ mmc_request_done(host->mmc, mrq);
+}
+
+static
+bool bcm2835_send_command(struct bcm2835_host *host, struct mmc_command *cmd)
+{
+ struct device *dev = &host->pdev->dev;
+ u32 sdcmd, sdhsts;
+ unsigned long timeout;
+
+ WARN_ON(host->cmd);
+
+ sdcmd = bcm2835_read_wait_sdcmd(host, 100);
+ if (sdcmd & SDCMD_NEW_FLAG) {
+ dev_err(dev, "previous command never completed.\n");
+ bcm2835_dumpregs(host);
+ cmd->error = -EILSEQ;
+ bcm2835_finish_request(host);
+ return false;
+ }
+
+ if (!cmd->data && cmd->busy_timeout > 9000)
+ timeout = DIV_ROUND_UP(cmd->busy_timeout, 1000) * HZ + HZ;
+ else
+ timeout = 10 * HZ;
+ schedule_delayed_work(&host->timeout_work, timeout);
+
+ host->cmd = cmd;
+
+ /* Clear any error flags */
+ sdhsts = readl(host->ioaddr + SDHSTS);
+ if (sdhsts & SDHSTS_ERROR_MASK)
+ writel(sdhsts, host->ioaddr + SDHSTS);
+
+ if ((cmd->flags & MMC_RSP_136) && (cmd->flags & MMC_RSP_BUSY)) {
+ dev_err(dev, "unsupported response type!\n");
+ cmd->error = -EINVAL;
+ bcm2835_finish_request(host);
+ return false;
+ }
+
+ bcm2835_prepare_data(host, cmd);
+
+ writel(cmd->arg, host->ioaddr + SDARG);
+
+ sdcmd = cmd->opcode & SDCMD_CMD_MASK;
+
+ host->use_busy = false;
+ if (!(cmd->flags & MMC_RSP_PRESENT)) {
+ sdcmd |= SDCMD_NO_RESPONSE;
+ } else {
+ if (cmd->flags & MMC_RSP_136)
+ sdcmd |= SDCMD_LONG_RESPONSE;
+ if (cmd->flags & MMC_RSP_BUSY) {
+ sdcmd |= SDCMD_BUSYWAIT;
+ host->use_busy = true;
+ }
+ }
+
+ if (cmd->data) {
+ if (cmd->data->flags & MMC_DATA_WRITE)
+ sdcmd |= SDCMD_WRITE_CMD;
+ if (cmd->data->flags & MMC_DATA_READ)
+ sdcmd |= SDCMD_READ_CMD;
+ }
+
+ writel(sdcmd | SDCMD_NEW_FLAG, host->ioaddr + SDCMD);
+
+ return true;
+}
+
+static void bcm2835_transfer_complete(struct bcm2835_host *host)
+{
+ struct mmc_data *data;
+
+ WARN_ON(!host->data_complete);
+
+ data = host->data;
+ host->data = NULL;
+
+ /* Need to send CMD12 if -
+ * a) open-ended multiblock transfer (no CMD23)
+ * b) error in multiblock transfer
+ */
+ if (host->mrq->stop && (data->error || !host->use_sbc)) {
+ if (bcm2835_send_command(host, host->mrq->stop)) {
+ /* No busy, so poll for completion */
+ if (!host->use_busy)
+ bcm2835_finish_command(host);
+ }
+ } else {
+ bcm2835_wait_transfer_complete(host);
+ bcm2835_finish_request(host);
+ }
+}
+
+static void bcm2835_finish_data(struct bcm2835_host *host)
+{
+ struct device *dev = &host->pdev->dev;
+ struct mmc_data *data;
+
+ data = host->data;
+
+ host->hcfg &= ~(SDHCFG_DATA_IRPT_EN | SDHCFG_BLOCK_IRPT_EN);
+ writel(host->hcfg, host->ioaddr + SDHCFG);
+
+ data->bytes_xfered = data->error ? 0 : (data->blksz * data->blocks);
+
+ host->data_complete = true;
+
+ if (host->cmd) {
+ /* Data managed to finish before the
+ * command completed. Make sure we do
+ * things in the proper order.
+ */
+ dev_dbg(dev, "Finished early - HSTS %08x\n",
+ readl(host->ioaddr + SDHSTS));
+ } else {
+ bcm2835_transfer_complete(host);
+ }
+}
+
+static void bcm2835_finish_command(struct bcm2835_host *host)
+{
+ struct device *dev = &host->pdev->dev;
+ struct mmc_command *cmd = host->cmd;
+ u32 sdcmd;
+
+ sdcmd = bcm2835_read_wait_sdcmd(host, 100);
+
+ /* Check for errors */
+ if (sdcmd & SDCMD_NEW_FLAG) {
+ dev_err(dev, "command never completed.\n");
+ bcm2835_dumpregs(host);
+ host->cmd->error = -EIO;
+ bcm2835_finish_request(host);
+ return;
+ } else if (sdcmd & SDCMD_FAIL_FLAG) {
+ u32 sdhsts = readl(host->ioaddr + SDHSTS);
+
+ /* Clear the errors */
+ writel(SDHSTS_ERROR_MASK, host->ioaddr + SDHSTS);
+
+ if (!(sdhsts & SDHSTS_CRC7_ERROR) ||
+ (host->cmd->opcode != MMC_SEND_OP_COND)) {
+ if (sdhsts & SDHSTS_CMD_TIME_OUT) {
+ host->cmd->error = -ETIMEDOUT;
+ } else {
+ dev_err(dev, "unexpected command %d error\n",
+ host->cmd->opcode);
+ bcm2835_dumpregs(host);
+ host->cmd->error = -EILSEQ;
+ }
+ bcm2835_finish_request(host);
+ return;
+ }
+ }
+
+ if (cmd->flags & MMC_RSP_PRESENT) {
+ if (cmd->flags & MMC_RSP_136) {
+ int i;
+
+ for (i = 0; i < 4; i++) {
+ cmd->resp[3 - i] =
+ readl(host->ioaddr + SDRSP0 + i * 4);
+ }
+ } else {
+ cmd->resp[0] = readl(host->ioaddr + SDRSP0);
+ }
+ }
+
+ if (cmd == host->mrq->sbc) {
+ /* Finished CMD23, now send actual command. */
+ host->cmd = NULL;
+ if (bcm2835_send_command(host, host->mrq->cmd)) {
+ if (host->data && host->dma_desc)
+ /* DMA transfer starts now, PIO starts
+ * after irq
+ */
+ bcm2835_start_dma(host);
+
+ if (!host->use_busy)
+ bcm2835_finish_command(host);
+ }
+ } else if (cmd == host->mrq->stop) {
+ /* Finished CMD12 */
+ bcm2835_finish_request(host);
+ } else {
+ /* Processed actual command. */
+ host->cmd = NULL;
+ if (!host->data)
+ bcm2835_finish_request(host);
+ else if (host->data_complete)
+ bcm2835_transfer_complete(host);
+ }
+}
+
+static void bcm2835_timeout(struct work_struct *work)
+{
+ struct delayed_work *d = to_delayed_work(work);
+ struct bcm2835_host *host =
+ container_of(d, struct bcm2835_host, timeout_work);
+ struct device *dev = &host->pdev->dev;
+
+ mutex_lock(&host->mutex);
+
+ if (host->mrq) {
+ dev_err(dev, "timeout waiting for hardware interrupt.\n");
+ bcm2835_dumpregs(host);
+
+ if (host->data) {
+ host->data->error = -ETIMEDOUT;
+ bcm2835_finish_data(host);
+ } else {
+ if (host->cmd)
+ host->cmd->error = -ETIMEDOUT;
+ else
+ host->mrq->cmd->error = -ETIMEDOUT;
+
+ bcm2835_finish_request(host);
+ }
+ }
+
+ mutex_unlock(&host->mutex);
+}
+
+static bool bcm2835_check_cmd_error(struct bcm2835_host *host, u32 intmask)
+{
+ struct device *dev = &host->pdev->dev;
+
+ if (!(intmask & SDHSTS_ERROR_MASK))
+ return false;
+
+ if (!host->cmd)
+ return true;
+
+ dev_err(dev, "sdhost_busy_irq: intmask %08x\n", intmask);
+ if (intmask & SDHSTS_CRC7_ERROR) {
+ host->cmd->error = -EILSEQ;
+ } else if (intmask & (SDHSTS_CRC16_ERROR |
+ SDHSTS_FIFO_ERROR)) {
+ if (host->mrq->data)
+ host->mrq->data->error = -EILSEQ;
+ else
+ host->cmd->error = -EILSEQ;
+ } else if (intmask & SDHSTS_REW_TIME_OUT) {
+ if (host->mrq->data)
+ host->mrq->data->error = -ETIMEDOUT;
+ else
+ host->cmd->error = -ETIMEDOUT;
+ } else if (intmask & SDHSTS_CMD_TIME_OUT) {
+ host->cmd->error = -ETIMEDOUT;
+ }
+ bcm2835_dumpregs(host);
+ return true;
+}
+
+static void bcm2835_check_data_error(struct bcm2835_host *host, u32 intmask)
+{
+ if (!host->data)
+ return;
+ if (intmask & (SDHSTS_CRC16_ERROR | SDHSTS_FIFO_ERROR))
+ host->data->error = -EILSEQ;
+ if (intmask & SDHSTS_REW_TIME_OUT)
+ host->data->error = -ETIMEDOUT;
+}
+
+static void bcm2835_busy_irq(struct bcm2835_host *host)
+{
+ if (WARN_ON(!host->cmd)) {
+ bcm2835_dumpregs(host);
+ return;
+ }
+
+ if (WARN_ON(!host->use_busy)) {
+ bcm2835_dumpregs(host);
+ return;
+ }
+ host->use_busy = false;
+
+ bcm2835_finish_command(host);
+}
+
+static void bcm2835_data_irq(struct bcm2835_host *host, u32 intmask)
+{
+ /* There are no dedicated data/space available interrupt
+ * status bits, so it is necessary to use the single shared
+ * data/space available FIFO status bits. It is therefore not
+ * an error to get here when there is no data transfer in
+ * progress.
+ */
+ if (!host->data)
+ return;
+
+ bcm2835_check_data_error(host, intmask);
+ if (host->data->error)
+ goto finished;
+
+ if (host->data->flags & MMC_DATA_WRITE) {
+ /* Use the block interrupt for writes after the first block */
+ host->hcfg &= ~(SDHCFG_DATA_IRPT_EN);
+ host->hcfg |= SDHCFG_BLOCK_IRPT_EN;
+ writel(host->hcfg, host->ioaddr + SDHCFG);
+ bcm2835_transfer_pio(host);
+ } else {
+ bcm2835_transfer_pio(host);
+ host->blocks--;
+ if ((host->blocks == 0) || host->data->error)
+ goto finished;
+ }
+ return;
+
+finished:
+ host->hcfg &= ~(SDHCFG_DATA_IRPT_EN | SDHCFG_BLOCK_IRPT_EN);
+ writel(host->hcfg, host->ioaddr + SDHCFG);
+}
+
+static void bcm2835_data_threaded_irq(struct bcm2835_host *host)
+{
+ if (!host->data)
+ return;
+ if ((host->blocks == 0) || host->data->error)
+ bcm2835_finish_data(host);
+}
+
+static void bcm2835_block_irq(struct bcm2835_host *host)
+{
+ if (WARN_ON(!host->data)) {
+ bcm2835_dumpregs(host);
+ return;
+ }
+
+ if (!host->dma_desc) {
+ WARN_ON(!host->blocks);
+ if (host->data->error || (--host->blocks == 0))
+ bcm2835_finish_data(host);
+ else
+ bcm2835_transfer_pio(host);
+ } else if (host->data->flags & MMC_DATA_WRITE) {
+ bcm2835_finish_data(host);
+ }
+}
+
+static irqreturn_t bcm2835_irq(int irq, void *dev_id)
+{
+ irqreturn_t result = IRQ_NONE;
+ struct bcm2835_host *host = dev_id;
+ u32 intmask;
+
+ spin_lock(&host->lock);
+
+ intmask = readl(host->ioaddr + SDHSTS);
+
+ writel(SDHSTS_BUSY_IRPT |
+ SDHSTS_BLOCK_IRPT |
+ SDHSTS_SDIO_IRPT |
+ SDHSTS_DATA_FLAG,
+ host->ioaddr + SDHSTS);
+
+ if (intmask & SDHSTS_BLOCK_IRPT) {
+ bcm2835_check_data_error(host, intmask);
+ host->irq_block = true;
+ result = IRQ_WAKE_THREAD;
+ }
+
+ if (intmask & SDHSTS_BUSY_IRPT) {
+ if (!bcm2835_check_cmd_error(host, intmask)) {
+ host->irq_busy = true;
+ result = IRQ_WAKE_THREAD;
+ } else {
+ result = IRQ_HANDLED;
+ }
+ }
+
+ /* There is no true data interrupt status bit, so it is
+ * necessary to qualify the data flag with the interrupt
+ * enable bit.
+ */
+ if ((intmask & SDHSTS_DATA_FLAG) &&
+ (host->hcfg & SDHCFG_DATA_IRPT_EN)) {
+ bcm2835_data_irq(host, intmask);
+ host->irq_data = true;
+ result = IRQ_WAKE_THREAD;
+ }
+
+ spin_unlock(&host->lock);
+
+ return result;
+}
+
+static irqreturn_t bcm2835_threaded_irq(int irq, void *dev_id)
+{
+ struct bcm2835_host *host = dev_id;
+ unsigned long flags;
+ bool block, busy, data;
+
+ spin_lock_irqsave(&host->lock, flags);
+
+ block = host->irq_block;
+ busy = host->irq_busy;
+ data = host->irq_data;
+ host->irq_block = false;
+ host->irq_busy = false;
+ host->irq_data = false;
+
+ spin_unlock_irqrestore(&host->lock, flags);
+
+ mutex_lock(&host->mutex);
+
+ if (block)
+ bcm2835_block_irq(host);
+ if (busy)
+ bcm2835_busy_irq(host);
+ if (data)
+ bcm2835_data_threaded_irq(host);
+
+ mutex_unlock(&host->mutex);
+
+ return IRQ_HANDLED;
+}
+
+static void bcm2835_dma_complete_work(struct work_struct *work)
+{
+ struct bcm2835_host *host =
+ container_of(work, struct bcm2835_host, dma_work);
+ struct mmc_data *data = host->data;
+
+ mutex_lock(&host->mutex);
+
+ if (host->dma_chan) {
+ dma_unmap_sg(host->dma_chan->device->dev,
+ data->sg, data->sg_len,
+ host->dma_dir);
+
+ host->dma_chan = NULL;
+ }
+
+ if (host->drain_words) {
+ unsigned long flags;
+ void *page;
+ u32 *buf;
+
+ if (host->drain_offset & PAGE_MASK) {
+ host->drain_page += host->drain_offset >> PAGE_SHIFT;
+ host->drain_offset &= ~PAGE_MASK;
+ }
+ local_irq_save(flags);
+ page = kmap_atomic(host->drain_page);
+ buf = page + host->drain_offset;
+
+ while (host->drain_words) {
+ u32 edm = readl(host->ioaddr + SDEDM);
+
+ if ((edm >> 4) & 0x1f)
+ *(buf++) = readl(host->ioaddr + SDDATA);
+ host->drain_words--;
+ }
+
+ kunmap_atomic(page);
+ local_irq_restore(flags);
+ }
+
+ bcm2835_finish_data(host);
+
+ mutex_unlock(&host->mutex);
+}
+
+static void bcm2835_set_clock(struct bcm2835_host *host, unsigned int clock)
+{
+ int div;
+
+ /* The SDCDIV register has 11 bits, and holds (div - 2). But
+ * in data mode the max is 50MHz wihout a minimum, and only
+ * the bottom 3 bits are used. Since the switch over is
+ * automatic (unless we have marked the card as slow...),
+ * chosen values have to make sense in both modes. Ident mode
+ * must be 100-400KHz, so can range check the requested
+ * clock. CMD15 must be used to return to data mode, so this
+ * can be monitored.
+ *
+ * clock 250MHz -> 0->125MHz, 1->83.3MHz, 2->62.5MHz, 3->50.0MHz
+ * 4->41.7MHz, 5->35.7MHz, 6->31.3MHz, 7->27.8MHz
+ *
+ * 623->400KHz/27.8MHz
+ * reset value (507)->491159/50MHz
+ *
+ * BUT, the 3-bit clock divisor in data mode is too small if
+ * the core clock is higher than 250MHz, so instead use the
+ * SLOW_CARD configuration bit to force the use of the ident
+ * clock divisor at all times.
+ */
+
+ if (clock < 100000) {
+ /* Can't stop the clock, but make it as slow as possible
+ * to show willing
+ */
+ host->cdiv = SDCDIV_MAX_CDIV;
+ writel(host->cdiv, host->ioaddr + SDCDIV);
+ return;
+ }
+
+ div = host->max_clk / clock;
+ if (div < 2)
+ div = 2;
+ if ((host->max_clk / div) > clock)
+ div++;
+ div -= 2;
+
+ if (div > SDCDIV_MAX_CDIV)
+ div = SDCDIV_MAX_CDIV;
+
+ clock = host->max_clk / (div + 2);
+ host->mmc->actual_clock = clock;
+
+ /* Calibrate some delays */
+
+ host->ns_per_fifo_word = (1000000000 / clock) *
+ ((host->mmc->caps & MMC_CAP_4_BIT_DATA) ? 8 : 32);
+
+ host->cdiv = div;
+ writel(host->cdiv, host->ioaddr + SDCDIV);
+
+ /* Set the timeout to 500ms */
+ writel(host->mmc->actual_clock / 2, host->ioaddr + SDTOUT);
+}
+
+static void bcm2835_request(struct mmc_host *mmc, struct mmc_request *mrq)
+{
+ struct bcm2835_host *host = mmc_priv(mmc);
+ struct device *dev = &host->pdev->dev;
+ u32 edm, fsm;
+
+ /* Reset the error statuses in case this is a retry */
+ if (mrq->sbc)
+ mrq->sbc->error = 0;
+ if (mrq->cmd)
+ mrq->cmd->error = 0;
+ if (mrq->data)
+ mrq->data->error = 0;
+ if (mrq->stop)
+ mrq->stop->error = 0;
+
+ if (mrq->data && !is_power_of_2(mrq->data->blksz)) {
+ dev_err(dev, "unsupported block size (%d bytes)\n",
+ mrq->data->blksz);
+ mrq->cmd->error = -EINVAL;
+ mmc_request_done(mmc, mrq);
+ return;
+ }
+
+ if (host->use_dma && mrq->data && (mrq->data->blocks > PIO_THRESHOLD))
+ bcm2835_prepare_dma(host, mrq->data);
+
+ mutex_lock(&host->mutex);
+
+ WARN_ON(host->mrq);
+ host->mrq = mrq;
+
+ edm = readl(host->ioaddr + SDEDM);
+ fsm = edm & SDEDM_FSM_MASK;
+
+ if ((fsm != SDEDM_FSM_IDENTMODE) &&
+ (fsm != SDEDM_FSM_DATAMODE)) {
+ dev_err(dev, "previous command (%d) not complete (EDM %08x)\n",
+ readl(host->ioaddr + SDCMD) & SDCMD_CMD_MASK,
+ edm);
+ bcm2835_dumpregs(host);
+ mrq->cmd->error = -EILSEQ;
+ bcm2835_finish_request(host);
+ mutex_unlock(&host->mutex);
+ return;
+ }
+
+ host->use_sbc = !!mrq->sbc && host->mrq->data &&
+ (host->mrq->data->flags & MMC_DATA_READ);
+ if (host->use_sbc) {
+ if (bcm2835_send_command(host, mrq->sbc)) {
+ if (!host->use_busy)
+ bcm2835_finish_command(host);
+ }
+ } else if (bcm2835_send_command(host, mrq->cmd)) {
+ if (host->data && host->dma_desc) {
+ /* DMA transfer starts now, PIO starts after irq */
+ bcm2835_start_dma(host);
+ }
+
+ if (!host->use_busy)
+ bcm2835_finish_command(host);
+ }
+
+ mutex_unlock(&host->mutex);
+}
+
+static void bcm2835_set_ios(struct mmc_host *mmc, struct mmc_ios *ios)
+{
+ struct bcm2835_host *host = mmc_priv(mmc);
+
+ mutex_lock(&host->mutex);
+
+ if (!ios->clock || ios->clock != host->clock) {
+ bcm2835_set_clock(host, ios->clock);
+ host->clock = ios->clock;
+ }
+
+ /* set bus width */
+ host->hcfg &= ~SDHCFG_WIDE_EXT_BUS;
+ if (ios->bus_width == MMC_BUS_WIDTH_4)
+ host->hcfg |= SDHCFG_WIDE_EXT_BUS;
+
+ host->hcfg |= SDHCFG_WIDE_INT_BUS;
+
+ /* Disable clever clock switching, to cope with fast core clocks */
+ host->hcfg |= SDHCFG_SLOW_CARD;
+
+ writel(host->hcfg, host->ioaddr + SDHCFG);
+
+ mutex_unlock(&host->mutex);
+}
+
+static struct mmc_host_ops bcm2835_ops = {
+ .request = bcm2835_request,
+ .set_ios = bcm2835_set_ios,
+ .hw_reset = bcm2835_reset,
+};
+
+static int bcm2835_add_host(struct bcm2835_host *host)
+{
+ struct mmc_host *mmc = host->mmc;
+ struct device *dev = &host->pdev->dev;
+ char pio_limit_string[20];
+ int ret;
+
+ mmc->f_max = host->max_clk;
+ mmc->f_min = host->max_clk / SDCDIV_MAX_CDIV;
+
+ mmc->max_busy_timeout = ~0 / (mmc->f_max / 1000);
+
+ dev_dbg(dev, "f_max %d, f_min %d, max_busy_timeout %d\n",
+ mmc->f_max, mmc->f_min, mmc->max_busy_timeout);
+
+ /* host controller capabilities */
+ mmc->caps |= MMC_CAP_SD_HIGHSPEED | MMC_CAP_MMC_HIGHSPEED |
+ MMC_CAP_NEEDS_POLL | MMC_CAP_HW_RESET | MMC_CAP_ERASE |
+ MMC_CAP_CMD23;
+
+ spin_lock_init(&host->lock);
+ mutex_init(&host->mutex);
+
+ if (IS_ERR_OR_NULL(host->dma_chan_rxtx)) {
+ dev_warn(dev, "unable to initialise DMA channel. Falling back to PIO\n");
+ host->use_dma = false;
+ } else {
+ host->use_dma = true;
+
+ host->dma_cfg_tx.src_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES;
+ host->dma_cfg_tx.dst_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES;
+ host->dma_cfg_tx.slave_id = 13; /* DREQ channel */
+ host->dma_cfg_tx.direction = DMA_MEM_TO_DEV;
+ host->dma_cfg_tx.src_addr = 0;
+ host->dma_cfg_tx.dst_addr = host->phys_addr + SDDATA;
+
+ host->dma_cfg_rx.src_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES;
+ host->dma_cfg_rx.dst_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES;
+ host->dma_cfg_rx.slave_id = 13; /* DREQ channel */
+ host->dma_cfg_rx.direction = DMA_DEV_TO_MEM;
+ host->dma_cfg_rx.src_addr = host->phys_addr + SDDATA;
+ host->dma_cfg_rx.dst_addr = 0;
+
+ if (dmaengine_slave_config(host->dma_chan_rxtx,
+ &host->dma_cfg_tx) != 0 ||
+ dmaengine_slave_config(host->dma_chan_rxtx,
+ &host->dma_cfg_rx) != 0)
+ host->use_dma = false;
+ }
+
+ mmc->max_segs = 128;
+ mmc->max_req_size = 524288;
+ mmc->max_seg_size = mmc->max_req_size;
+ mmc->max_blk_size = 1024;
+ mmc->max_blk_count = 65535;
+
+ /* report supported voltage ranges */
+ mmc->ocr_avail = MMC_VDD_32_33 | MMC_VDD_33_34;
+
+ INIT_WORK(&host->dma_work, bcm2835_dma_complete_work);
+ INIT_DELAYED_WORK(&host->timeout_work, bcm2835_timeout);
+
+ /* Set interrupt enables */
+ host->hcfg = SDHCFG_BUSY_IRPT_EN;
+
+ bcm2835_reset_internal(host);
+
+ ret = request_threaded_irq(host->irq, bcm2835_irq,
+ bcm2835_threaded_irq,
+ 0, mmc_hostname(mmc), host);
+ if (ret) {
+ dev_err(dev, "failed to request IRQ %d: %d\n", host->irq, ret);
+ return ret;
+ }
+
+ ret = mmc_add_host(mmc);
+ if (ret) {
+ free_irq(host->irq, host);
+ return ret;
+ }
+
+ pio_limit_string[0] = '\0';
+ if (host->use_dma && (PIO_THRESHOLD > 0))
+ sprintf(pio_limit_string, " (>%d)", PIO_THRESHOLD);
+ dev_info(dev, "loaded - DMA %s%s\n",
+ host->use_dma ? "enabled" : "disabled", pio_limit_string);
+
+ return 0;
+}
+
+static int bcm2835_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct clk *clk;
+ struct resource *iomem;
+ struct bcm2835_host *host;
+ struct mmc_host *mmc;
+ const __be32 *regaddr_p;
+ int ret;
+
+ dev_dbg(dev, "%s\n", __func__);
+ mmc = mmc_alloc_host(sizeof(*host), dev);
+ if (!mmc)
+ return -ENOMEM;
+
+ mmc->ops = &bcm2835_ops;
+ host = mmc_priv(mmc);
+ host->mmc = mmc;
+ host->pdev = pdev;
+ spin_lock_init(&host->lock);
+
+ iomem = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ host->ioaddr = devm_ioremap_resource(dev, iomem);
+ if (IS_ERR(host->ioaddr)) {
+ ret = PTR_ERR(host->ioaddr);
+ goto err;
+ }
+
+ /* Parse OF address directly to get the physical address for
+ * DMA to our registers.
+ */
+ regaddr_p = of_get_address(pdev->dev.of_node, 0, NULL, NULL);
+ if (!regaddr_p) {
+ dev_err(dev, "Can't get phys address\n");
+ ret = -EINVAL;
+ goto err;
+ }
+
+ host->phys_addr = be32_to_cpup(regaddr_p);
+
+ host->dma_chan = NULL;
+ host->dma_desc = NULL;
+
+ host->dma_chan_rxtx = dma_request_slave_channel(dev, "rx-tx");
+
+ clk = devm_clk_get(dev, NULL);
+ if (IS_ERR(clk)) {
+ ret = PTR_ERR(clk);
+ if (ret != -EPROBE_DEFER)
+ dev_err(dev, "could not get clk: %d\n", ret);
+ goto err;
+ }
+
+ host->max_clk = clk_get_rate(clk);
+
+ host->irq = platform_get_irq(pdev, 0);
+ if (host->irq <= 0) {
+ dev_err(dev, "get IRQ failed\n");
+ ret = -EINVAL;
+ goto err;
+ }
+
+ ret = mmc_of_parse(mmc);
+ if (ret)
+ goto err;
+
+ ret = bcm2835_add_host(host);
+ if (ret)
+ goto err;
+
+ platform_set_drvdata(pdev, host);
+
+ dev_dbg(dev, "%s -> OK\n", __func__);
+
+ return 0;
+
+err:
+ dev_dbg(dev, "%s -> err %d\n", __func__, ret);
+ mmc_free_host(mmc);
+
+ return ret;
+}
+
+static int bcm2835_remove(struct platform_device *pdev)
+{
+ struct bcm2835_host *host = platform_get_drvdata(pdev);
+
+ mmc_remove_host(host->mmc);
+
+ writel(SDVDD_POWER_OFF, host->ioaddr + SDVDD);
+
+ free_irq(host->irq, host);
+
+ cancel_work_sync(&host->dma_work);
+ cancel_delayed_work_sync(&host->timeout_work);
+
+ mmc_free_host(host->mmc);
+ platform_set_drvdata(pdev, NULL);
+
+ return 0;
+}
+
+static const struct of_device_id bcm2835_match[] = {
+ { .compatible = "brcm,bcm2835-sdhost" },
+ { }
+};
+MODULE_DEVICE_TABLE(of, bcm2835_match);
+
+static struct platform_driver bcm2835_driver = {
+ .probe = bcm2835_probe,
+ .remove = bcm2835_remove,
+ .driver = {
+ .name = "sdhost-bcm2835",
+ .of_match_table = bcm2835_match,
+ },
+};
+module_platform_driver(bcm2835_driver);
+
+MODULE_ALIAS("platform:sdhost-bcm2835");
+MODULE_DESCRIPTION("BCM2835 SDHost driver");
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Phil Elwell");
diff --git a/drivers/mmc/host/cavium-octeon.c b/drivers/mmc/host/cavium-octeon.c
new file mode 100644
index 0000000..772d090
--- /dev/null
+++ b/drivers/mmc/host/cavium-octeon.c
@@ -0,0 +1,351 @@
+/*
+ * Driver for MMC and SSD cards for Cavium OCTEON SOCs.
+ *
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License. See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2012-2017 Cavium Inc.
+ */
+#include <linux/dma-mapping.h>
+#include <linux/gpio/consumer.h>
+#include <linux/interrupt.h>
+#include <linux/mmc/mmc.h>
+#include <linux/mmc/slot-gpio.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <asm/octeon/octeon.h>
+#include "cavium.h"
+
+#define CVMX_MIO_BOOT_CTL CVMX_ADD_IO_SEG(0x00011800000000D0ull)
+
+/*
+ * The l2c* functions below are used for the EMMC-17978 workaround.
+ *
+ * Due to a bug in the design of the MMC bus hardware, the 2nd to last
+ * cache block of a DMA read must be locked into the L2 Cache.
+ * Otherwise, data corruption may occur.
+ */
+static inline void *phys_to_ptr(u64 address)
+{
+ return (void *)(address | (1ull << 63)); /* XKPHYS */
+}
+
+/*
+ * Lock a single line into L2. The line is zeroed before locking
+ * to make sure no dram accesses are made.
+ */
+static void l2c_lock_line(u64 addr)
+{
+ char *addr_ptr = phys_to_ptr(addr);
+
+ asm volatile (
+ "cache 31, %[line]" /* Unlock the line */
+ ::[line] "m" (*addr_ptr));
+}
+
+/* Unlock a single line in the L2 cache. */
+static void l2c_unlock_line(u64 addr)
+{
+ char *addr_ptr = phys_to_ptr(addr);
+
+ asm volatile (
+ "cache 23, %[line]" /* Unlock the line */
+ ::[line] "m" (*addr_ptr));
+}
+
+/* Locks a memory region in the L2 cache. */
+static void l2c_lock_mem_region(u64 start, u64 len)
+{
+ u64 end;
+
+ /* Round start/end to cache line boundaries */
+ end = ALIGN(start + len - 1, CVMX_CACHE_LINE_SIZE);
+ start = ALIGN(start, CVMX_CACHE_LINE_SIZE);
+
+ while (start <= end) {
+ l2c_lock_line(start);
+ start += CVMX_CACHE_LINE_SIZE;
+ }
+ asm volatile("sync");
+}
+
+/* Unlock a memory region in the L2 cache. */
+static void l2c_unlock_mem_region(u64 start, u64 len)
+{
+ u64 end;
+
+ /* Round start/end to cache line boundaries */
+ end = ALIGN(start + len - 1, CVMX_CACHE_LINE_SIZE);
+ start = ALIGN(start, CVMX_CACHE_LINE_SIZE);
+
+ while (start <= end) {
+ l2c_unlock_line(start);
+ start += CVMX_CACHE_LINE_SIZE;
+ }
+}
+
+static void octeon_mmc_acquire_bus(struct cvm_mmc_host *host)
+{
+ if (!host->has_ciu3) {
+ down(&octeon_bootbus_sem);
+ /* For CN70XX, switch the MMC controller onto the bus. */
+ if (OCTEON_IS_MODEL(OCTEON_CN70XX))
+ writeq(0, (void __iomem *)CVMX_MIO_BOOT_CTL);
+ } else {
+ down(&host->mmc_serializer);
+ }
+}
+
+static void octeon_mmc_release_bus(struct cvm_mmc_host *host)
+{
+ if (!host->has_ciu3)
+ up(&octeon_bootbus_sem);
+ else
+ up(&host->mmc_serializer);
+}
+
+static void octeon_mmc_int_enable(struct cvm_mmc_host *host, u64 val)
+{
+ writeq(val, host->base + MIO_EMM_INT(host));
+ if (!host->dma_active || (host->dma_active && !host->has_ciu3))
+ writeq(val, host->base + MIO_EMM_INT_EN(host));
+}
+
+static void octeon_mmc_set_shared_power(struct cvm_mmc_host *host, int dir)
+{
+ if (dir == 0)
+ if (!atomic_dec_return(&host->shared_power_users))
+ gpiod_set_value_cansleep(host->global_pwr_gpiod, 0);
+ if (dir == 1)
+ if (atomic_inc_return(&host->shared_power_users) == 1)
+ gpiod_set_value_cansleep(host->global_pwr_gpiod, 1);
+}
+
+static void octeon_mmc_dmar_fixup(struct cvm_mmc_host *host,
+ struct mmc_command *cmd,
+ struct mmc_data *data,
+ u64 addr)
+{
+ if (cmd->opcode != MMC_WRITE_MULTIPLE_BLOCK)
+ return;
+ if (data->blksz * data->blocks <= 1024)
+ return;
+
+ host->n_minus_one = addr + (data->blksz * data->blocks) - 1024;
+ l2c_lock_mem_region(host->n_minus_one, 512);
+}
+
+static void octeon_mmc_dmar_fixup_done(struct cvm_mmc_host *host)
+{
+ if (!host->n_minus_one)
+ return;
+ l2c_unlock_mem_region(host->n_minus_one, 512);
+ host->n_minus_one = 0;
+}
+
+static int octeon_mmc_probe(struct platform_device *pdev)
+{
+ struct device_node *cn, *node = pdev->dev.of_node;
+ struct cvm_mmc_host *host;
+ struct resource *res;
+ void __iomem *base;
+ int mmc_irq[9];
+ int i, ret = 0;
+ u64 val;
+
+ host = devm_kzalloc(&pdev->dev, sizeof(*host), GFP_KERNEL);
+ if (!host)
+ return -ENOMEM;
+
+ spin_lock_init(&host->irq_handler_lock);
+ sema_init(&host->mmc_serializer, 1);
+
+ host->dev = &pdev->dev;
+ host->acquire_bus = octeon_mmc_acquire_bus;
+ host->release_bus = octeon_mmc_release_bus;
+ host->int_enable = octeon_mmc_int_enable;
+ host->set_shared_power = octeon_mmc_set_shared_power;
+ if (OCTEON_IS_MODEL(OCTEON_CN6XXX) ||
+ OCTEON_IS_MODEL(OCTEON_CNF7XXX)) {
+ host->dmar_fixup = octeon_mmc_dmar_fixup;
+ host->dmar_fixup_done = octeon_mmc_dmar_fixup_done;
+ }
+
+ host->sys_freq = octeon_get_io_clock_rate();
+
+ if (of_device_is_compatible(node, "cavium,octeon-7890-mmc")) {
+ host->big_dma_addr = true;
+ host->need_irq_handler_lock = true;
+ host->has_ciu3 = true;
+ host->use_sg = true;
+ /*
+ * First seven are the EMM_INT bits 0..6, then two for
+ * the EMM_DMA_INT bits
+ */
+ for (i = 0; i < 9; i++) {
+ mmc_irq[i] = platform_get_irq(pdev, i);
+ if (mmc_irq[i] < 0)
+ return mmc_irq[i];
+
+ /* work around legacy u-boot device trees */
+ irq_set_irq_type(mmc_irq[i], IRQ_TYPE_EDGE_RISING);
+ }
+ } else {
+ host->big_dma_addr = false;
+ host->need_irq_handler_lock = false;
+ host->has_ciu3 = false;
+ /* First one is EMM second DMA */
+ for (i = 0; i < 2; i++) {
+ mmc_irq[i] = platform_get_irq(pdev, i);
+ if (mmc_irq[i] < 0)
+ return mmc_irq[i];
+ }
+ }
+
+ host->last_slot = -1;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!res) {
+ dev_err(&pdev->dev, "Platform resource[0] is missing\n");
+ return -ENXIO;
+ }
+ base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+ host->base = (void __iomem *)base;
+ host->reg_off = 0;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 1);
+ if (!res) {
+ dev_err(&pdev->dev, "Platform resource[1] is missing\n");
+ return -EINVAL;
+ }
+ base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+ host->dma_base = (void __iomem *)base;
+ /*
+ * To keep the register addresses shared we intentionaly use
+ * a negative offset here, first register used on Octeon therefore
+ * starts at 0x20 (MIO_EMM_DMA_CFG).
+ */
+ host->reg_off_dma = -0x20;
+
+ ret = dma_set_mask(&pdev->dev, DMA_BIT_MASK(64));
+ if (ret)
+ return ret;
+
+ /*
+ * Clear out any pending interrupts that may be left over from
+ * bootloader.
+ */
+ val = readq(host->base + MIO_EMM_INT(host));
+ writeq(val, host->base + MIO_EMM_INT(host));
+
+ if (host->has_ciu3) {
+ /* Only CMD_DONE, DMA_DONE, CMD_ERR, DMA_ERR */
+ for (i = 1; i <= 4; i++) {
+ ret = devm_request_irq(&pdev->dev, mmc_irq[i],
+ cvm_mmc_interrupt,
+ 0, cvm_mmc_irq_names[i], host);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "Error: devm_request_irq %d\n",
+ mmc_irq[i]);
+ return ret;
+ }
+ }
+ } else {
+ ret = devm_request_irq(&pdev->dev, mmc_irq[0],
+ cvm_mmc_interrupt, 0, KBUILD_MODNAME,
+ host);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "Error: devm_request_irq %d\n",
+ mmc_irq[0]);
+ return ret;
+ }
+ }
+
+ host->global_pwr_gpiod = devm_gpiod_get_optional(&pdev->dev,
+ "power-gpios",
+ GPIOD_OUT_HIGH);
+ if (IS_ERR(host->global_pwr_gpiod)) {
+ dev_err(&pdev->dev, "Invalid power GPIO\n");
+ return PTR_ERR(host->global_pwr_gpiod);
+ }
+
+ platform_set_drvdata(pdev, host);
+
+ i = 0;
+ for_each_child_of_node(node, cn) {
+ host->slot_pdev[i] =
+ of_platform_device_create(cn, NULL, &pdev->dev);
+ if (!host->slot_pdev[i]) {
+ i++;
+ continue;
+ }
+ ret = cvm_mmc_of_slot_probe(&host->slot_pdev[i]->dev, host);
+ if (ret) {
+ dev_err(&pdev->dev, "Error populating slots\n");
+ octeon_mmc_set_shared_power(host, 0);
+ return ret;
+ }
+ i++;
+ }
+ return 0;
+}
+
+static int octeon_mmc_remove(struct platform_device *pdev)
+{
+ struct cvm_mmc_host *host = platform_get_drvdata(pdev);
+ u64 dma_cfg;
+ int i;
+
+ for (i = 0; i < CAVIUM_MAX_MMC; i++)
+ if (host->slot[i])
+ cvm_mmc_of_slot_remove(host->slot[i]);
+
+ dma_cfg = readq(host->dma_base + MIO_EMM_DMA_CFG(host));
+ dma_cfg &= ~MIO_EMM_DMA_CFG_EN;
+ writeq(dma_cfg, host->dma_base + MIO_EMM_DMA_CFG(host));
+
+ octeon_mmc_set_shared_power(host, 0);
+ return 0;
+}
+
+static const struct of_device_id octeon_mmc_match[] = {
+ {
+ .compatible = "cavium,octeon-6130-mmc",
+ },
+ {
+ .compatible = "cavium,octeon-7890-mmc",
+ },
+ {},
+};
+MODULE_DEVICE_TABLE(of, octeon_mmc_match);
+
+static struct platform_driver octeon_mmc_driver = {
+ .probe = octeon_mmc_probe,
+ .remove = octeon_mmc_remove,
+ .driver = {
+ .name = KBUILD_MODNAME,
+ .of_match_table = octeon_mmc_match,
+ },
+};
+
+static int __init octeon_mmc_init(void)
+{
+ return platform_driver_register(&octeon_mmc_driver);
+}
+
+static void __exit octeon_mmc_cleanup(void)
+{
+ platform_driver_unregister(&octeon_mmc_driver);
+}
+
+module_init(octeon_mmc_init);
+module_exit(octeon_mmc_cleanup);
+
+MODULE_AUTHOR("Cavium Inc. <support@cavium.com>");
+MODULE_DESCRIPTION("Low-level driver for Cavium OCTEON MMC/SSD card");
+MODULE_LICENSE("GPL");
diff --git a/drivers/mmc/host/cavium-thunderx.c b/drivers/mmc/host/cavium-thunderx.c
new file mode 100644
index 0000000..fe3d772
--- /dev/null
+++ b/drivers/mmc/host/cavium-thunderx.c
@@ -0,0 +1,187 @@
+/*
+ * Driver for MMC and SSD cards for Cavium ThunderX SOCs.
+ *
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License. See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2016 Cavium Inc.
+ */
+#include <linux/dma-mapping.h>
+#include <linux/interrupt.h>
+#include <linux/mmc/mmc.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_platform.h>
+#include <linux/pci.h>
+#include "cavium.h"
+
+static void thunder_mmc_acquire_bus(struct cvm_mmc_host *host)
+{
+ down(&host->mmc_serializer);
+}
+
+static void thunder_mmc_release_bus(struct cvm_mmc_host *host)
+{
+ up(&host->mmc_serializer);
+}
+
+static void thunder_mmc_int_enable(struct cvm_mmc_host *host, u64 val)
+{
+ writeq(val, host->base + MIO_EMM_INT(host));
+ writeq(val, host->base + MIO_EMM_INT_EN_SET(host));
+}
+
+static int thunder_mmc_register_interrupts(struct cvm_mmc_host *host,
+ struct pci_dev *pdev)
+{
+ int nvec, ret, i;
+
+ nvec = pci_alloc_irq_vectors(pdev, 1, 9, PCI_IRQ_MSIX);
+ if (nvec < 0)
+ return nvec;
+
+ /* register interrupts */
+ for (i = 0; i < nvec; i++) {
+ ret = devm_request_irq(&pdev->dev, pci_irq_vector(pdev, i),
+ cvm_mmc_interrupt,
+ 0, cvm_mmc_irq_names[i], host);
+ if (ret)
+ return ret;
+ }
+ return 0;
+}
+
+static int thunder_mmc_probe(struct pci_dev *pdev,
+ const struct pci_device_id *id)
+{
+ struct device_node *node = pdev->dev.of_node;
+ struct device *dev = &pdev->dev;
+ struct device_node *child_node;
+ struct cvm_mmc_host *host;
+ int ret, i = 0;
+
+ host = devm_kzalloc(dev, sizeof(*host), GFP_KERNEL);
+ if (!host)
+ return -ENOMEM;
+
+ pci_set_drvdata(pdev, host);
+ ret = pcim_enable_device(pdev);
+ if (ret)
+ return ret;
+
+ ret = pci_request_regions(pdev, KBUILD_MODNAME);
+ if (ret)
+ return ret;
+
+ host->base = pcim_iomap(pdev, 0, pci_resource_len(pdev, 0));
+ if (!host->base)
+ return -EINVAL;
+
+ /* On ThunderX these are identical */
+ host->dma_base = host->base;
+
+ host->reg_off = 0x2000;
+ host->reg_off_dma = 0x160;
+
+ host->clk = devm_clk_get(dev, NULL);
+ if (IS_ERR(host->clk))
+ return PTR_ERR(host->clk);
+
+ ret = clk_prepare_enable(host->clk);
+ if (ret)
+ return ret;
+ host->sys_freq = clk_get_rate(host->clk);
+
+ spin_lock_init(&host->irq_handler_lock);
+ sema_init(&host->mmc_serializer, 1);
+
+ host->dev = dev;
+ host->acquire_bus = thunder_mmc_acquire_bus;
+ host->release_bus = thunder_mmc_release_bus;
+ host->int_enable = thunder_mmc_int_enable;
+
+ host->use_sg = true;
+ host->big_dma_addr = true;
+ host->need_irq_handler_lock = true;
+ host->last_slot = -1;
+
+ ret = dma_set_mask(dev, DMA_BIT_MASK(48));
+ if (ret)
+ goto error;
+
+ /*
+ * Clear out any pending interrupts that may be left over from
+ * bootloader. Writing 1 to the bits clears them.
+ */
+ writeq(127, host->base + MIO_EMM_INT_EN(host));
+ writeq(3, host->base + MIO_EMM_DMA_INT_ENA_W1C(host));
+ /* Clear DMA FIFO */
+ writeq(BIT_ULL(16), host->base + MIO_EMM_DMA_FIFO_CFG(host));
+
+ ret = thunder_mmc_register_interrupts(host, pdev);
+ if (ret)
+ goto error;
+
+ for_each_child_of_node(node, child_node) {
+ /*
+ * mmc_of_parse and devm* require one device per slot.
+ * Create a dummy device per slot and set the node pointer to
+ * the slot. The easiest way to get this is using
+ * of_platform_device_create.
+ */
+ if (of_device_is_compatible(child_node, "mmc-slot")) {
+ host->slot_pdev[i] = of_platform_device_create(child_node, NULL,
+ &pdev->dev);
+ if (!host->slot_pdev[i])
+ continue;
+
+ ret = cvm_mmc_of_slot_probe(&host->slot_pdev[i]->dev, host);
+ if (ret)
+ goto error;
+ }
+ i++;
+ }
+ dev_info(dev, "probed\n");
+ return 0;
+
+error:
+ clk_disable_unprepare(host->clk);
+ return ret;
+}
+
+static void thunder_mmc_remove(struct pci_dev *pdev)
+{
+ struct cvm_mmc_host *host = pci_get_drvdata(pdev);
+ u64 dma_cfg;
+ int i;
+
+ for (i = 0; i < CAVIUM_MAX_MMC; i++)
+ if (host->slot[i])
+ cvm_mmc_of_slot_remove(host->slot[i]);
+
+ dma_cfg = readq(host->dma_base + MIO_EMM_DMA_CFG(host));
+ dma_cfg &= ~MIO_EMM_DMA_CFG_EN;
+ writeq(dma_cfg, host->dma_base + MIO_EMM_DMA_CFG(host));
+
+ clk_disable_unprepare(host->clk);
+}
+
+static const struct pci_device_id thunder_mmc_id_table[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_CAVIUM, 0xa010) },
+ { 0, } /* end of table */
+};
+
+static struct pci_driver thunder_mmc_driver = {
+ .name = KBUILD_MODNAME,
+ .id_table = thunder_mmc_id_table,
+ .probe = thunder_mmc_probe,
+ .remove = thunder_mmc_remove,
+};
+
+module_pci_driver(thunder_mmc_driver);
+
+MODULE_AUTHOR("Cavium Inc.");
+MODULE_DESCRIPTION("Cavium ThunderX eMMC Driver");
+MODULE_LICENSE("GPL");
+MODULE_DEVICE_TABLE(pci, thunder_mmc_id_table);
diff --git a/drivers/mmc/host/cavium.c b/drivers/mmc/host/cavium.c
new file mode 100644
index 0000000..58b51ba
--- /dev/null
+++ b/drivers/mmc/host/cavium.c
@@ -0,0 +1,1090 @@
+/*
+ * Shared part of driver for MMC/SDHC controller on Cavium OCTEON and
+ * ThunderX SOCs.
+ *
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License. See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2012-2017 Cavium Inc.
+ * Authors:
+ * David Daney <david.daney@cavium.com>
+ * Peter Swain <pswain@cavium.com>
+ * Steven J. Hill <steven.hill@cavium.com>
+ * Jan Glauber <jglauber@cavium.com>
+ */
+#include <linux/bitfield.h>
+#include <linux/delay.h>
+#include <linux/dma-direction.h>
+#include <linux/dma-mapping.h>
+#include <linux/gpio/consumer.h>
+#include <linux/interrupt.h>
+#include <linux/mmc/mmc.h>
+#include <linux/mmc/slot-gpio.h>
+#include <linux/module.h>
+#include <linux/regulator/consumer.h>
+#include <linux/scatterlist.h>
+#include <linux/time.h>
+
+#include "cavium.h"
+
+const char *cvm_mmc_irq_names[] = {
+ "MMC Buffer",
+ "MMC Command",
+ "MMC DMA",
+ "MMC Command Error",
+ "MMC DMA Error",
+ "MMC Switch",
+ "MMC Switch Error",
+ "MMC DMA int Fifo",
+ "MMC DMA int",
+};
+
+/*
+ * The Cavium MMC host hardware assumes that all commands have fixed
+ * command and response types. These are correct if MMC devices are
+ * being used. However, non-MMC devices like SD use command and
+ * response types that are unexpected by the host hardware.
+ *
+ * The command and response types can be overridden by supplying an
+ * XOR value that is applied to the type. We calculate the XOR value
+ * from the values in this table and the flags passed from the MMC
+ * core.
+ */
+static struct cvm_mmc_cr_type cvm_mmc_cr_types[] = {
+ {0, 0}, /* CMD0 */
+ {0, 3}, /* CMD1 */
+ {0, 2}, /* CMD2 */
+ {0, 1}, /* CMD3 */
+ {0, 0}, /* CMD4 */
+ {0, 1}, /* CMD5 */
+ {0, 1}, /* CMD6 */
+ {0, 1}, /* CMD7 */
+ {1, 1}, /* CMD8 */
+ {0, 2}, /* CMD9 */
+ {0, 2}, /* CMD10 */
+ {1, 1}, /* CMD11 */
+ {0, 1}, /* CMD12 */
+ {0, 1}, /* CMD13 */
+ {1, 1}, /* CMD14 */
+ {0, 0}, /* CMD15 */
+ {0, 1}, /* CMD16 */
+ {1, 1}, /* CMD17 */
+ {1, 1}, /* CMD18 */
+ {3, 1}, /* CMD19 */
+ {2, 1}, /* CMD20 */
+ {0, 0}, /* CMD21 */
+ {0, 0}, /* CMD22 */
+ {0, 1}, /* CMD23 */
+ {2, 1}, /* CMD24 */
+ {2, 1}, /* CMD25 */
+ {2, 1}, /* CMD26 */
+ {2, 1}, /* CMD27 */
+ {0, 1}, /* CMD28 */
+ {0, 1}, /* CMD29 */
+ {1, 1}, /* CMD30 */
+ {1, 1}, /* CMD31 */
+ {0, 0}, /* CMD32 */
+ {0, 0}, /* CMD33 */
+ {0, 0}, /* CMD34 */
+ {0, 1}, /* CMD35 */
+ {0, 1}, /* CMD36 */
+ {0, 0}, /* CMD37 */
+ {0, 1}, /* CMD38 */
+ {0, 4}, /* CMD39 */
+ {0, 5}, /* CMD40 */
+ {0, 0}, /* CMD41 */
+ {2, 1}, /* CMD42 */
+ {0, 0}, /* CMD43 */
+ {0, 0}, /* CMD44 */
+ {0, 0}, /* CMD45 */
+ {0, 0}, /* CMD46 */
+ {0, 0}, /* CMD47 */
+ {0, 0}, /* CMD48 */
+ {0, 0}, /* CMD49 */
+ {0, 0}, /* CMD50 */
+ {0, 0}, /* CMD51 */
+ {0, 0}, /* CMD52 */
+ {0, 0}, /* CMD53 */
+ {0, 0}, /* CMD54 */
+ {0, 1}, /* CMD55 */
+ {0xff, 0xff}, /* CMD56 */
+ {0, 0}, /* CMD57 */
+ {0, 0}, /* CMD58 */
+ {0, 0}, /* CMD59 */
+ {0, 0}, /* CMD60 */
+ {0, 0}, /* CMD61 */
+ {0, 0}, /* CMD62 */
+ {0, 0} /* CMD63 */
+};
+
+static struct cvm_mmc_cr_mods cvm_mmc_get_cr_mods(struct mmc_command *cmd)
+{
+ struct cvm_mmc_cr_type *cr;
+ u8 hardware_ctype, hardware_rtype;
+ u8 desired_ctype = 0, desired_rtype = 0;
+ struct cvm_mmc_cr_mods r;
+
+ cr = cvm_mmc_cr_types + (cmd->opcode & 0x3f);
+ hardware_ctype = cr->ctype;
+ hardware_rtype = cr->rtype;
+ if (cmd->opcode == MMC_GEN_CMD)
+ hardware_ctype = (cmd->arg & 1) ? 1 : 2;
+
+ switch (mmc_cmd_type(cmd)) {
+ case MMC_CMD_ADTC:
+ desired_ctype = (cmd->data->flags & MMC_DATA_WRITE) ? 2 : 1;
+ break;
+ case MMC_CMD_AC:
+ case MMC_CMD_BC:
+ case MMC_CMD_BCR:
+ desired_ctype = 0;
+ break;
+ }
+
+ switch (mmc_resp_type(cmd)) {
+ case MMC_RSP_NONE:
+ desired_rtype = 0;
+ break;
+ case MMC_RSP_R1:/* MMC_RSP_R5, MMC_RSP_R6, MMC_RSP_R7 */
+ case MMC_RSP_R1B:
+ desired_rtype = 1;
+ break;
+ case MMC_RSP_R2:
+ desired_rtype = 2;
+ break;
+ case MMC_RSP_R3: /* MMC_RSP_R4 */
+ desired_rtype = 3;
+ break;
+ }
+ r.ctype_xor = desired_ctype ^ hardware_ctype;
+ r.rtype_xor = desired_rtype ^ hardware_rtype;
+ return r;
+}
+
+static void check_switch_errors(struct cvm_mmc_host *host)
+{
+ u64 emm_switch;
+
+ emm_switch = readq(host->base + MIO_EMM_SWITCH(host));
+ if (emm_switch & MIO_EMM_SWITCH_ERR0)
+ dev_err(host->dev, "Switch power class error\n");
+ if (emm_switch & MIO_EMM_SWITCH_ERR1)
+ dev_err(host->dev, "Switch hs timing error\n");
+ if (emm_switch & MIO_EMM_SWITCH_ERR2)
+ dev_err(host->dev, "Switch bus width error\n");
+}
+
+static void clear_bus_id(u64 *reg)
+{
+ u64 bus_id_mask = GENMASK_ULL(61, 60);
+
+ *reg &= ~bus_id_mask;
+}
+
+static void set_bus_id(u64 *reg, int bus_id)
+{
+ clear_bus_id(reg);
+ *reg |= FIELD_PREP(GENMASK(61, 60), bus_id);
+}
+
+static int get_bus_id(u64 reg)
+{
+ return FIELD_GET(GENMASK_ULL(61, 60), reg);
+}
+
+/*
+ * We never set the switch_exe bit since that would interfere
+ * with the commands send by the MMC core.
+ */
+static void do_switch(struct cvm_mmc_host *host, u64 emm_switch)
+{
+ int retries = 100;
+ u64 rsp_sts;
+ int bus_id;
+
+ /*
+ * Modes setting only taken from slot 0. Work around that hardware
+ * issue by first switching to slot 0.
+ */
+ bus_id = get_bus_id(emm_switch);
+ clear_bus_id(&emm_switch);
+ writeq(emm_switch, host->base + MIO_EMM_SWITCH(host));
+
+ set_bus_id(&emm_switch, bus_id);
+ writeq(emm_switch, host->base + MIO_EMM_SWITCH(host));
+
+ /* wait for the switch to finish */
+ do {
+ rsp_sts = readq(host->base + MIO_EMM_RSP_STS(host));
+ if (!(rsp_sts & MIO_EMM_RSP_STS_SWITCH_VAL))
+ break;
+ udelay(10);
+ } while (--retries);
+
+ check_switch_errors(host);
+}
+
+static bool switch_val_changed(struct cvm_mmc_slot *slot, u64 new_val)
+{
+ /* Match BUS_ID, HS_TIMING, BUS_WIDTH, POWER_CLASS, CLK_HI, CLK_LO */
+ u64 match = 0x3001070fffffffffull;
+
+ return (slot->cached_switch & match) != (new_val & match);
+}
+
+static void set_wdog(struct cvm_mmc_slot *slot, unsigned int ns)
+{
+ u64 timeout;
+
+ if (!slot->clock)
+ return;
+
+ if (ns)
+ timeout = (slot->clock * ns) / NSEC_PER_SEC;
+ else
+ timeout = (slot->clock * 850ull) / 1000ull;
+ writeq(timeout, slot->host->base + MIO_EMM_WDOG(slot->host));
+}
+
+static void cvm_mmc_reset_bus(struct cvm_mmc_slot *slot)
+{
+ struct cvm_mmc_host *host = slot->host;
+ u64 emm_switch, wdog;
+
+ emm_switch = readq(slot->host->base + MIO_EMM_SWITCH(host));
+ emm_switch &= ~(MIO_EMM_SWITCH_EXE | MIO_EMM_SWITCH_ERR0 |
+ MIO_EMM_SWITCH_ERR1 | MIO_EMM_SWITCH_ERR2);
+ set_bus_id(&emm_switch, slot->bus_id);
+
+ wdog = readq(slot->host->base + MIO_EMM_WDOG(host));
+ do_switch(slot->host, emm_switch);
+
+ slot->cached_switch = emm_switch;
+
+ msleep(20);
+
+ writeq(wdog, slot->host->base + MIO_EMM_WDOG(host));
+}
+
+/* Switch to another slot if needed */
+static void cvm_mmc_switch_to(struct cvm_mmc_slot *slot)
+{
+ struct cvm_mmc_host *host = slot->host;
+ struct cvm_mmc_slot *old_slot;
+ u64 emm_sample, emm_switch;
+
+ if (slot->bus_id == host->last_slot)
+ return;
+
+ if (host->last_slot >= 0 && host->slot[host->last_slot]) {
+ old_slot = host->slot[host->last_slot];
+ old_slot->cached_switch = readq(host->base + MIO_EMM_SWITCH(host));
+ old_slot->cached_rca = readq(host->base + MIO_EMM_RCA(host));
+ }
+
+ writeq(slot->cached_rca, host->base + MIO_EMM_RCA(host));
+ emm_switch = slot->cached_switch;
+ set_bus_id(&emm_switch, slot->bus_id);
+ do_switch(host, emm_switch);
+
+ emm_sample = FIELD_PREP(MIO_EMM_SAMPLE_CMD_CNT, slot->cmd_cnt) |
+ FIELD_PREP(MIO_EMM_SAMPLE_DAT_CNT, slot->dat_cnt);
+ writeq(emm_sample, host->base + MIO_EMM_SAMPLE(host));
+
+ host->last_slot = slot->bus_id;
+}
+
+static void do_read(struct cvm_mmc_host *host, struct mmc_request *req,
+ u64 dbuf)
+{
+ struct sg_mapping_iter *smi = &host->smi;
+ int data_len = req->data->blocks * req->data->blksz;
+ int bytes_xfered, shift = -1;
+ u64 dat = 0;
+
+ /* Auto inc from offset zero */
+ writeq((0x10000 | (dbuf << 6)), host->base + MIO_EMM_BUF_IDX(host));
+
+ for (bytes_xfered = 0; bytes_xfered < data_len;) {
+ if (smi->consumed >= smi->length) {
+ if (!sg_miter_next(smi))
+ break;
+ smi->consumed = 0;
+ }
+
+ if (shift < 0) {
+ dat = readq(host->base + MIO_EMM_BUF_DAT(host));
+ shift = 56;
+ }
+
+ while (smi->consumed < smi->length && shift >= 0) {
+ ((u8 *)smi->addr)[smi->consumed] = (dat >> shift) & 0xff;
+ bytes_xfered++;
+ smi->consumed++;
+ shift -= 8;
+ }
+ }
+
+ sg_miter_stop(smi);
+ req->data->bytes_xfered = bytes_xfered;
+ req->data->error = 0;
+}
+
+static void do_write(struct mmc_request *req)
+{
+ req->data->bytes_xfered = req->data->blocks * req->data->blksz;
+ req->data->error = 0;
+}
+
+static void set_cmd_response(struct cvm_mmc_host *host, struct mmc_request *req,
+ u64 rsp_sts)
+{
+ u64 rsp_hi, rsp_lo;
+
+ if (!(rsp_sts & MIO_EMM_RSP_STS_RSP_VAL))
+ return;
+
+ rsp_lo = readq(host->base + MIO_EMM_RSP_LO(host));
+
+ switch (FIELD_GET(MIO_EMM_RSP_STS_RSP_TYPE, rsp_sts)) {
+ case 1:
+ case 3:
+ req->cmd->resp[0] = (rsp_lo >> 8) & 0xffffffff;
+ req->cmd->resp[1] = 0;
+ req->cmd->resp[2] = 0;
+ req->cmd->resp[3] = 0;
+ break;
+ case 2:
+ req->cmd->resp[3] = rsp_lo & 0xffffffff;
+ req->cmd->resp[2] = (rsp_lo >> 32) & 0xffffffff;
+ rsp_hi = readq(host->base + MIO_EMM_RSP_HI(host));
+ req->cmd->resp[1] = rsp_hi & 0xffffffff;
+ req->cmd->resp[0] = (rsp_hi >> 32) & 0xffffffff;
+ break;
+ }
+}
+
+static int get_dma_dir(struct mmc_data *data)
+{
+ return (data->flags & MMC_DATA_WRITE) ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
+}
+
+static int finish_dma_single(struct cvm_mmc_host *host, struct mmc_data *data)
+{
+ data->bytes_xfered = data->blocks * data->blksz;
+ data->error = 0;
+ return 1;
+}
+
+static int finish_dma_sg(struct cvm_mmc_host *host, struct mmc_data *data)
+{
+ u64 fifo_cfg;
+ int count;
+
+ /* Check if there are any pending requests left */
+ fifo_cfg = readq(host->dma_base + MIO_EMM_DMA_FIFO_CFG(host));
+ count = FIELD_GET(MIO_EMM_DMA_FIFO_CFG_COUNT, fifo_cfg);
+ if (count)
+ dev_err(host->dev, "%u requests still pending\n", count);
+
+ data->bytes_xfered = data->blocks * data->blksz;
+ data->error = 0;
+
+ /* Clear and disable FIFO */
+ writeq(BIT_ULL(16), host->dma_base + MIO_EMM_DMA_FIFO_CFG(host));
+ dma_unmap_sg(host->dev, data->sg, data->sg_len, get_dma_dir(data));
+ return 1;
+}
+
+static int finish_dma(struct cvm_mmc_host *host, struct mmc_data *data)
+{
+ if (host->use_sg && data->sg_len > 1)
+ return finish_dma_sg(host, data);
+ else
+ return finish_dma_single(host, data);
+}
+
+static int check_status(u64 rsp_sts)
+{
+ if (rsp_sts & MIO_EMM_RSP_STS_RSP_BAD_STS ||
+ rsp_sts & MIO_EMM_RSP_STS_RSP_CRC_ERR ||
+ rsp_sts & MIO_EMM_RSP_STS_BLK_CRC_ERR)
+ return -EILSEQ;
+ if (rsp_sts & MIO_EMM_RSP_STS_RSP_TIMEOUT ||
+ rsp_sts & MIO_EMM_RSP_STS_BLK_TIMEOUT)
+ return -ETIMEDOUT;
+ if (rsp_sts & MIO_EMM_RSP_STS_DBUF_ERR)
+ return -EIO;
+ return 0;
+}
+
+/* Try to clean up failed DMA. */
+static void cleanup_dma(struct cvm_mmc_host *host, u64 rsp_sts)
+{
+ u64 emm_dma;
+
+ emm_dma = readq(host->base + MIO_EMM_DMA(host));
+ emm_dma |= FIELD_PREP(MIO_EMM_DMA_VAL, 1) |
+ FIELD_PREP(MIO_EMM_DMA_DAT_NULL, 1);
+ set_bus_id(&emm_dma, get_bus_id(rsp_sts));
+ writeq(emm_dma, host->base + MIO_EMM_DMA(host));
+}
+
+irqreturn_t cvm_mmc_interrupt(int irq, void *dev_id)
+{
+ struct cvm_mmc_host *host = dev_id;
+ struct mmc_request *req;
+ unsigned long flags = 0;
+ u64 emm_int, rsp_sts;
+ bool host_done;
+
+ if (host->need_irq_handler_lock)
+ spin_lock_irqsave(&host->irq_handler_lock, flags);
+ else
+ __acquire(&host->irq_handler_lock);
+
+ /* Clear interrupt bits (write 1 clears ). */
+ emm_int = readq(host->base + MIO_EMM_INT(host));
+ writeq(emm_int, host->base + MIO_EMM_INT(host));
+
+ if (emm_int & MIO_EMM_INT_SWITCH_ERR)
+ check_switch_errors(host);
+
+ req = host->current_req;
+ if (!req)
+ goto out;
+
+ rsp_sts = readq(host->base + MIO_EMM_RSP_STS(host));
+ /*
+ * dma_val set means DMA is still in progress. Don't touch
+ * the request and wait for the interrupt indicating that
+ * the DMA is finished.
+ */
+ if ((rsp_sts & MIO_EMM_RSP_STS_DMA_VAL) && host->dma_active)
+ goto out;
+
+ if (!host->dma_active && req->data &&
+ (emm_int & MIO_EMM_INT_BUF_DONE)) {
+ unsigned int type = (rsp_sts >> 7) & 3;
+
+ if (type == 1)
+ do_read(host, req, rsp_sts & MIO_EMM_RSP_STS_DBUF);
+ else if (type == 2)
+ do_write(req);
+ }
+
+ host_done = emm_int & MIO_EMM_INT_CMD_DONE ||
+ emm_int & MIO_EMM_INT_DMA_DONE ||
+ emm_int & MIO_EMM_INT_CMD_ERR ||
+ emm_int & MIO_EMM_INT_DMA_ERR;
+
+ if (!(host_done && req->done))
+ goto no_req_done;
+
+ req->cmd->error = check_status(rsp_sts);
+
+ if (host->dma_active && req->data)
+ if (!finish_dma(host, req->data))
+ goto no_req_done;
+
+ set_cmd_response(host, req, rsp_sts);
+ if ((emm_int & MIO_EMM_INT_DMA_ERR) &&
+ (rsp_sts & MIO_EMM_RSP_STS_DMA_PEND))
+ cleanup_dma(host, rsp_sts);
+
+ host->current_req = NULL;
+ req->done(req);
+
+no_req_done:
+ if (host->dmar_fixup_done)
+ host->dmar_fixup_done(host);
+ if (host_done)
+ host->release_bus(host);
+out:
+ if (host->need_irq_handler_lock)
+ spin_unlock_irqrestore(&host->irq_handler_lock, flags);
+ else
+ __release(&host->irq_handler_lock);
+ return IRQ_RETVAL(emm_int != 0);
+}
+
+/*
+ * Program DMA_CFG and if needed DMA_ADR.
+ * Returns 0 on error, DMA address otherwise.
+ */
+static u64 prepare_dma_single(struct cvm_mmc_host *host, struct mmc_data *data)
+{
+ u64 dma_cfg, addr;
+ int count, rw;
+
+ count = dma_map_sg(host->dev, data->sg, data->sg_len,
+ get_dma_dir(data));
+ if (!count)
+ return 0;
+
+ rw = (data->flags & MMC_DATA_WRITE) ? 1 : 0;
+ dma_cfg = FIELD_PREP(MIO_EMM_DMA_CFG_EN, 1) |
+ FIELD_PREP(MIO_EMM_DMA_CFG_RW, rw);
+#ifdef __LITTLE_ENDIAN
+ dma_cfg |= FIELD_PREP(MIO_EMM_DMA_CFG_ENDIAN, 1);
+#endif
+ dma_cfg |= FIELD_PREP(MIO_EMM_DMA_CFG_SIZE,
+ (sg_dma_len(&data->sg[0]) / 8) - 1);
+
+ addr = sg_dma_address(&data->sg[0]);
+ if (!host->big_dma_addr)
+ dma_cfg |= FIELD_PREP(MIO_EMM_DMA_CFG_ADR, addr);
+ writeq(dma_cfg, host->dma_base + MIO_EMM_DMA_CFG(host));
+
+ pr_debug("[%s] sg_dma_len: %u total sg_elem: %d\n",
+ (rw) ? "W" : "R", sg_dma_len(&data->sg[0]), count);
+
+ if (host->big_dma_addr)
+ writeq(addr, host->dma_base + MIO_EMM_DMA_ADR(host));
+ return addr;
+}
+
+/*
+ * Queue complete sg list into the FIFO.
+ * Returns 0 on error, 1 otherwise.
+ */
+static u64 prepare_dma_sg(struct cvm_mmc_host *host, struct mmc_data *data)
+{
+ struct scatterlist *sg;
+ u64 fifo_cmd, addr;
+ int count, i, rw;
+
+ count = dma_map_sg(host->dev, data->sg, data->sg_len,
+ get_dma_dir(data));
+ if (!count)
+ return 0;
+ if (count > 16)
+ goto error;
+
+ /* Enable FIFO by removing CLR bit */
+ writeq(0, host->dma_base + MIO_EMM_DMA_FIFO_CFG(host));
+
+ for_each_sg(data->sg, sg, count, i) {
+ /* Program DMA address */
+ addr = sg_dma_address(sg);
+ if (addr & 7)
+ goto error;
+ writeq(addr, host->dma_base + MIO_EMM_DMA_FIFO_ADR(host));
+
+ /*
+ * If we have scatter-gather support we also have an extra
+ * register for the DMA addr, so no need to check
+ * host->big_dma_addr here.
+ */
+ rw = (data->flags & MMC_DATA_WRITE) ? 1 : 0;
+ fifo_cmd = FIELD_PREP(MIO_EMM_DMA_FIFO_CMD_RW, rw);
+
+ /* enable interrupts on the last element */
+ fifo_cmd |= FIELD_PREP(MIO_EMM_DMA_FIFO_CMD_INTDIS,
+ (i + 1 == count) ? 0 : 1);
+
+#ifdef __LITTLE_ENDIAN
+ fifo_cmd |= FIELD_PREP(MIO_EMM_DMA_FIFO_CMD_ENDIAN, 1);
+#endif
+ fifo_cmd |= FIELD_PREP(MIO_EMM_DMA_FIFO_CMD_SIZE,
+ sg_dma_len(sg) / 8 - 1);
+ /*
+ * The write copies the address and the command to the FIFO
+ * and increments the FIFO's COUNT field.
+ */
+ writeq(fifo_cmd, host->dma_base + MIO_EMM_DMA_FIFO_CMD(host));
+ pr_debug("[%s] sg_dma_len: %u sg_elem: %d/%d\n",
+ (rw) ? "W" : "R", sg_dma_len(sg), i, count);
+ }
+
+ /*
+ * In difference to prepare_dma_single we don't return the
+ * address here, as it would not make sense for scatter-gather.
+ * The dma fixup is only required on models that don't support
+ * scatter-gather, so that is not a problem.
+ */
+ return 1;
+
+error:
+ WARN_ON_ONCE(1);
+ dma_unmap_sg(host->dev, data->sg, data->sg_len, get_dma_dir(data));
+ /* Disable FIFO */
+ writeq(BIT_ULL(16), host->dma_base + MIO_EMM_DMA_FIFO_CFG(host));
+ return 0;
+}
+
+static u64 prepare_dma(struct cvm_mmc_host *host, struct mmc_data *data)
+{
+ if (host->use_sg && data->sg_len > 1)
+ return prepare_dma_sg(host, data);
+ else
+ return prepare_dma_single(host, data);
+}
+
+static u64 prepare_ext_dma(struct mmc_host *mmc, struct mmc_request *mrq)
+{
+ struct cvm_mmc_slot *slot = mmc_priv(mmc);
+ u64 emm_dma;
+
+ emm_dma = FIELD_PREP(MIO_EMM_DMA_VAL, 1) |
+ FIELD_PREP(MIO_EMM_DMA_SECTOR,
+ mmc_card_is_blockaddr(mmc->card) ? 1 : 0) |
+ FIELD_PREP(MIO_EMM_DMA_RW,
+ (mrq->data->flags & MMC_DATA_WRITE) ? 1 : 0) |
+ FIELD_PREP(MIO_EMM_DMA_BLOCK_CNT, mrq->data->blocks) |
+ FIELD_PREP(MIO_EMM_DMA_CARD_ADDR, mrq->cmd->arg);
+ set_bus_id(&emm_dma, slot->bus_id);
+
+ if (mmc_card_mmc(mmc->card) || (mmc_card_sd(mmc->card) &&
+ (mmc->card->scr.cmds & SD_SCR_CMD23_SUPPORT)))
+ emm_dma |= FIELD_PREP(MIO_EMM_DMA_MULTI, 1);
+
+ pr_debug("[%s] blocks: %u multi: %d\n",
+ (emm_dma & MIO_EMM_DMA_RW) ? "W" : "R",
+ mrq->data->blocks, (emm_dma & MIO_EMM_DMA_MULTI) ? 1 : 0);
+ return emm_dma;
+}
+
+static void cvm_mmc_dma_request(struct mmc_host *mmc,
+ struct mmc_request *mrq)
+{
+ struct cvm_mmc_slot *slot = mmc_priv(mmc);
+ struct cvm_mmc_host *host = slot->host;
+ struct mmc_data *data;
+ u64 emm_dma, addr;
+
+ if (!mrq->data || !mrq->data->sg || !mrq->data->sg_len ||
+ !mrq->stop || mrq->stop->opcode != MMC_STOP_TRANSMISSION) {
+ dev_err(&mmc->card->dev,
+ "Error: cmv_mmc_dma_request no data\n");
+ goto error;
+ }
+
+ cvm_mmc_switch_to(slot);
+
+ data = mrq->data;
+ pr_debug("DMA request blocks: %d block_size: %d total_size: %d\n",
+ data->blocks, data->blksz, data->blocks * data->blksz);
+ if (data->timeout_ns)
+ set_wdog(slot, data->timeout_ns);
+
+ WARN_ON(host->current_req);
+ host->current_req = mrq;
+
+ emm_dma = prepare_ext_dma(mmc, mrq);
+ addr = prepare_dma(host, data);
+ if (!addr) {
+ dev_err(host->dev, "prepare_dma failed\n");
+ goto error;
+ }
+
+ host->dma_active = true;
+ host->int_enable(host, MIO_EMM_INT_CMD_ERR | MIO_EMM_INT_DMA_DONE |
+ MIO_EMM_INT_DMA_ERR);
+
+ if (host->dmar_fixup)
+ host->dmar_fixup(host, mrq->cmd, data, addr);
+
+ /*
+ * If we have a valid SD card in the slot, we set the response
+ * bit mask to check for CRC errors and timeouts only.
+ * Otherwise, use the default power reset value.
+ */
+ if (mmc_card_sd(mmc->card))
+ writeq(0x00b00000ull, host->base + MIO_EMM_STS_MASK(host));
+ else
+ writeq(0xe4390080ull, host->base + MIO_EMM_STS_MASK(host));
+ writeq(emm_dma, host->base + MIO_EMM_DMA(host));
+ return;
+
+error:
+ mrq->cmd->error = -EINVAL;
+ if (mrq->done)
+ mrq->done(mrq);
+ host->release_bus(host);
+}
+
+static void do_read_request(struct cvm_mmc_host *host, struct mmc_request *mrq)
+{
+ sg_miter_start(&host->smi, mrq->data->sg, mrq->data->sg_len,
+ SG_MITER_ATOMIC | SG_MITER_TO_SG);
+}
+
+static void do_write_request(struct cvm_mmc_host *host, struct mmc_request *mrq)
+{
+ unsigned int data_len = mrq->data->blocks * mrq->data->blksz;
+ struct sg_mapping_iter *smi = &host->smi;
+ unsigned int bytes_xfered;
+ int shift = 56;
+ u64 dat = 0;
+
+ /* Copy data to the xmit buffer before issuing the command. */
+ sg_miter_start(smi, mrq->data->sg, mrq->data->sg_len, SG_MITER_FROM_SG);
+
+ /* Auto inc from offset zero, dbuf zero */
+ writeq(0x10000ull, host->base + MIO_EMM_BUF_IDX(host));
+
+ for (bytes_xfered = 0; bytes_xfered < data_len;) {
+ if (smi->consumed >= smi->length) {
+ if (!sg_miter_next(smi))
+ break;
+ smi->consumed = 0;
+ }
+
+ while (smi->consumed < smi->length && shift >= 0) {
+ dat |= (u64)((u8 *)smi->addr)[smi->consumed] << shift;
+ bytes_xfered++;
+ smi->consumed++;
+ shift -= 8;
+ }
+
+ if (shift < 0) {
+ writeq(dat, host->base + MIO_EMM_BUF_DAT(host));
+ shift = 56;
+ dat = 0;
+ }
+ }
+ sg_miter_stop(smi);
+}
+
+static void cvm_mmc_request(struct mmc_host *mmc, struct mmc_request *mrq)
+{
+ struct cvm_mmc_slot *slot = mmc_priv(mmc);
+ struct cvm_mmc_host *host = slot->host;
+ struct mmc_command *cmd = mrq->cmd;
+ struct cvm_mmc_cr_mods mods;
+ u64 emm_cmd, rsp_sts;
+ int retries = 100;
+
+ /*
+ * Note about locking:
+ * All MMC devices share the same bus and controller. Allow only a
+ * single user of the bootbus/MMC bus at a time. The lock is acquired
+ * on all entry points from the MMC layer.
+ *
+ * For requests the lock is only released after the completion
+ * interrupt!
+ */
+ host->acquire_bus(host);
+
+ if (cmd->opcode == MMC_READ_MULTIPLE_BLOCK ||
+ cmd->opcode == MMC_WRITE_MULTIPLE_BLOCK)
+ return cvm_mmc_dma_request(mmc, mrq);
+
+ cvm_mmc_switch_to(slot);
+
+ mods = cvm_mmc_get_cr_mods(cmd);
+
+ WARN_ON(host->current_req);
+ host->current_req = mrq;
+
+ if (cmd->data) {
+ if (cmd->data->flags & MMC_DATA_READ)
+ do_read_request(host, mrq);
+ else
+ do_write_request(host, mrq);
+
+ if (cmd->data->timeout_ns)
+ set_wdog(slot, cmd->data->timeout_ns);
+ } else
+ set_wdog(slot, 0);
+
+ host->dma_active = false;
+ host->int_enable(host, MIO_EMM_INT_CMD_DONE | MIO_EMM_INT_CMD_ERR);
+
+ emm_cmd = FIELD_PREP(MIO_EMM_CMD_VAL, 1) |
+ FIELD_PREP(MIO_EMM_CMD_CTYPE_XOR, mods.ctype_xor) |
+ FIELD_PREP(MIO_EMM_CMD_RTYPE_XOR, mods.rtype_xor) |
+ FIELD_PREP(MIO_EMM_CMD_IDX, cmd->opcode) |
+ FIELD_PREP(MIO_EMM_CMD_ARG, cmd->arg);
+ set_bus_id(&emm_cmd, slot->bus_id);
+ if (cmd->data && mmc_cmd_type(cmd) == MMC_CMD_ADTC)
+ emm_cmd |= FIELD_PREP(MIO_EMM_CMD_OFFSET,
+ 64 - ((cmd->data->blocks * cmd->data->blksz) / 8));
+
+ writeq(0, host->base + MIO_EMM_STS_MASK(host));
+
+retry:
+ rsp_sts = readq(host->base + MIO_EMM_RSP_STS(host));
+ if (rsp_sts & MIO_EMM_RSP_STS_DMA_VAL ||
+ rsp_sts & MIO_EMM_RSP_STS_CMD_VAL ||
+ rsp_sts & MIO_EMM_RSP_STS_SWITCH_VAL ||
+ rsp_sts & MIO_EMM_RSP_STS_DMA_PEND) {
+ udelay(10);
+ if (--retries)
+ goto retry;
+ }
+ if (!retries)
+ dev_err(host->dev, "Bad status: %llx before command write\n", rsp_sts);
+ writeq(emm_cmd, host->base + MIO_EMM_CMD(host));
+}
+
+static void cvm_mmc_set_ios(struct mmc_host *mmc, struct mmc_ios *ios)
+{
+ struct cvm_mmc_slot *slot = mmc_priv(mmc);
+ struct cvm_mmc_host *host = slot->host;
+ int clk_period = 0, power_class = 10, bus_width = 0;
+ u64 clock, emm_switch;
+
+ host->acquire_bus(host);
+ cvm_mmc_switch_to(slot);
+
+ /* Set the power state */
+ switch (ios->power_mode) {
+ case MMC_POWER_ON:
+ break;
+
+ case MMC_POWER_OFF:
+ cvm_mmc_reset_bus(slot);
+ if (host->global_pwr_gpiod)
+ host->set_shared_power(host, 0);
+ else
+ mmc_regulator_set_ocr(mmc, mmc->supply.vmmc, 0);
+ break;
+
+ case MMC_POWER_UP:
+ if (host->global_pwr_gpiod)
+ host->set_shared_power(host, 1);
+ else
+ mmc_regulator_set_ocr(mmc, mmc->supply.vmmc, ios->vdd);
+ break;
+ }
+
+ /* Convert bus width to HW definition */
+ switch (ios->bus_width) {
+ case MMC_BUS_WIDTH_8:
+ bus_width = 2;
+ break;
+ case MMC_BUS_WIDTH_4:
+ bus_width = 1;
+ break;
+ case MMC_BUS_WIDTH_1:
+ bus_width = 0;
+ break;
+ }
+
+ /* DDR is available for 4/8 bit bus width */
+ if (ios->bus_width && ios->timing == MMC_TIMING_MMC_DDR52)
+ bus_width |= 4;
+
+ /* Change the clock frequency. */
+ clock = ios->clock;
+ if (clock > 52000000)
+ clock = 52000000;
+ slot->clock = clock;
+
+ if (clock)
+ clk_period = (host->sys_freq + clock - 1) / (2 * clock);
+
+ emm_switch = FIELD_PREP(MIO_EMM_SWITCH_HS_TIMING,
+ (ios->timing == MMC_TIMING_MMC_HS)) |
+ FIELD_PREP(MIO_EMM_SWITCH_BUS_WIDTH, bus_width) |
+ FIELD_PREP(MIO_EMM_SWITCH_POWER_CLASS, power_class) |
+ FIELD_PREP(MIO_EMM_SWITCH_CLK_HI, clk_period) |
+ FIELD_PREP(MIO_EMM_SWITCH_CLK_LO, clk_period);
+ set_bus_id(&emm_switch, slot->bus_id);
+
+ if (!switch_val_changed(slot, emm_switch))
+ goto out;
+
+ set_wdog(slot, 0);
+ do_switch(host, emm_switch);
+ slot->cached_switch = emm_switch;
+out:
+ host->release_bus(host);
+}
+
+static const struct mmc_host_ops cvm_mmc_ops = {
+ .request = cvm_mmc_request,
+ .set_ios = cvm_mmc_set_ios,
+ .get_ro = mmc_gpio_get_ro,
+ .get_cd = mmc_gpio_get_cd,
+};
+
+static void cvm_mmc_set_clock(struct cvm_mmc_slot *slot, unsigned int clock)
+{
+ struct mmc_host *mmc = slot->mmc;
+
+ clock = min(clock, mmc->f_max);
+ clock = max(clock, mmc->f_min);
+ slot->clock = clock;
+}
+
+static int cvm_mmc_init_lowlevel(struct cvm_mmc_slot *slot)
+{
+ struct cvm_mmc_host *host = slot->host;
+ u64 emm_switch;
+
+ /* Enable this bus slot. */
+ host->emm_cfg |= (1ull << slot->bus_id);
+ writeq(host->emm_cfg, slot->host->base + MIO_EMM_CFG(host));
+ udelay(10);
+
+ /* Program initial clock speed and power. */
+ cvm_mmc_set_clock(slot, slot->mmc->f_min);
+ emm_switch = FIELD_PREP(MIO_EMM_SWITCH_POWER_CLASS, 10);
+ emm_switch |= FIELD_PREP(MIO_EMM_SWITCH_CLK_HI,
+ (host->sys_freq / slot->clock) / 2);
+ emm_switch |= FIELD_PREP(MIO_EMM_SWITCH_CLK_LO,
+ (host->sys_freq / slot->clock) / 2);
+
+ /* Make the changes take effect on this bus slot. */
+ set_bus_id(&emm_switch, slot->bus_id);
+ do_switch(host, emm_switch);
+
+ slot->cached_switch = emm_switch;
+
+ /*
+ * Set watchdog timeout value and default reset value
+ * for the mask register. Finally, set the CARD_RCA
+ * bit so that we can get the card address relative
+ * to the CMD register for CMD7 transactions.
+ */
+ set_wdog(slot, 0);
+ writeq(0xe4390080ull, host->base + MIO_EMM_STS_MASK(host));
+ writeq(1, host->base + MIO_EMM_RCA(host));
+ return 0;
+}
+
+static int cvm_mmc_of_parse(struct device *dev, struct cvm_mmc_slot *slot)
+{
+ u32 id, cmd_skew = 0, dat_skew = 0, bus_width = 0;
+ struct device_node *node = dev->of_node;
+ struct mmc_host *mmc = slot->mmc;
+ u64 clock_period;
+ int ret;
+
+ ret = of_property_read_u32(node, "reg", &id);
+ if (ret) {
+ dev_err(dev, "Missing or invalid reg property on %s\n",
+ of_node_full_name(node));
+ return ret;
+ }
+
+ if (id >= CAVIUM_MAX_MMC || slot->host->slot[id]) {
+ dev_err(dev, "Invalid reg property on %s\n",
+ of_node_full_name(node));
+ return -EINVAL;
+ }
+
+ mmc->supply.vmmc = devm_regulator_get_optional(dev, "vmmc");
+ if (IS_ERR(mmc->supply.vmmc)) {
+ if (PTR_ERR(mmc->supply.vmmc) == -EPROBE_DEFER)
+ return -EPROBE_DEFER;
+ /*
+ * Legacy Octeon firmware has no regulator entry, fall-back to
+ * a hard-coded voltage to get a sane OCR.
+ */
+ mmc->ocr_avail = MMC_VDD_32_33 | MMC_VDD_33_34;
+ } else {
+ ret = mmc_regulator_get_ocrmask(mmc->supply.vmmc);
+ if (ret > 0)
+ mmc->ocr_avail = ret;
+ }
+
+ /* Common MMC bindings */
+ ret = mmc_of_parse(mmc);
+ if (ret)
+ return ret;
+
+ /* Set bus width */
+ if (!(mmc->caps & (MMC_CAP_8_BIT_DATA | MMC_CAP_4_BIT_DATA))) {
+ of_property_read_u32(node, "cavium,bus-max-width", &bus_width);
+ if (bus_width == 8)
+ mmc->caps |= MMC_CAP_8_BIT_DATA | MMC_CAP_4_BIT_DATA;
+ else if (bus_width == 4)
+ mmc->caps |= MMC_CAP_4_BIT_DATA;
+ }
+
+ /* Set maximum and minimum frequency */
+ if (!mmc->f_max)
+ of_property_read_u32(node, "spi-max-frequency", &mmc->f_max);
+ if (!mmc->f_max || mmc->f_max > 52000000)
+ mmc->f_max = 52000000;
+ mmc->f_min = 400000;
+
+ /* Sampling register settings, period in picoseconds */
+ clock_period = 1000000000000ull / slot->host->sys_freq;
+ of_property_read_u32(node, "cavium,cmd-clk-skew", &cmd_skew);
+ of_property_read_u32(node, "cavium,dat-clk-skew", &dat_skew);
+ slot->cmd_cnt = (cmd_skew + clock_period / 2) / clock_period;
+ slot->dat_cnt = (dat_skew + clock_period / 2) / clock_period;
+
+ return id;
+}
+
+int cvm_mmc_of_slot_probe(struct device *dev, struct cvm_mmc_host *host)
+{
+ struct cvm_mmc_slot *slot;
+ struct mmc_host *mmc;
+ int ret, id;
+
+ mmc = mmc_alloc_host(sizeof(struct cvm_mmc_slot), dev);
+ if (!mmc)
+ return -ENOMEM;
+
+ slot = mmc_priv(mmc);
+ slot->mmc = mmc;
+ slot->host = host;
+
+ ret = cvm_mmc_of_parse(dev, slot);
+ if (ret < 0)
+ goto error;
+ id = ret;
+
+ /* Set up host parameters */
+ mmc->ops = &cvm_mmc_ops;
+
+ /*
+ * We only have a 3.3v supply, we cannot support any
+ * of the UHS modes. We do support the high speed DDR
+ * modes up to 52MHz.
+ */
+ mmc->caps |= MMC_CAP_MMC_HIGHSPEED | MMC_CAP_SD_HIGHSPEED |
+ MMC_CAP_ERASE | MMC_CAP_CMD23 | MMC_CAP_POWER_OFF_CARD |
+ MMC_CAP_3_3V_DDR;
+
+ if (host->use_sg)
+ mmc->max_segs = 16;
+ else
+ mmc->max_segs = 1;
+
+ /* DMA size field can address up to 8 MB */
+ mmc->max_seg_size = 8 * 1024 * 1024;
+ mmc->max_req_size = mmc->max_seg_size;
+ /* External DMA is in 512 byte blocks */
+ mmc->max_blk_size = 512;
+ /* DMA block count field is 15 bits */
+ mmc->max_blk_count = 32767;
+
+ slot->clock = mmc->f_min;
+ slot->bus_id = id;
+ slot->cached_rca = 1;
+
+ host->acquire_bus(host);
+ host->slot[id] = slot;
+ cvm_mmc_switch_to(slot);
+ cvm_mmc_init_lowlevel(slot);
+ host->release_bus(host);
+
+ ret = mmc_add_host(mmc);
+ if (ret) {
+ dev_err(dev, "mmc_add_host() returned %d\n", ret);
+ slot->host->slot[id] = NULL;
+ goto error;
+ }
+ return 0;
+
+error:
+ mmc_free_host(slot->mmc);
+ return ret;
+}
+
+int cvm_mmc_of_slot_remove(struct cvm_mmc_slot *slot)
+{
+ mmc_remove_host(slot->mmc);
+ slot->host->slot[slot->bus_id] = NULL;
+ mmc_free_host(slot->mmc);
+ return 0;
+}
diff --git a/drivers/mmc/host/cavium.h b/drivers/mmc/host/cavium.h
new file mode 100644
index 0000000..f3eea5e
--- /dev/null
+++ b/drivers/mmc/host/cavium.h
@@ -0,0 +1,215 @@
+/*
+ * Driver for MMC and SSD cards for Cavium OCTEON and ThunderX SOCs.
+ *
+ * This file is subject to the terms and conditions of the GNU General Public
+ * License. See the file "COPYING" in the main directory of this archive
+ * for more details.
+ *
+ * Copyright (C) 2012-2017 Cavium Inc.
+ */
+
+#ifndef _CAVIUM_MMC_H_
+#define _CAVIUM_MMC_H_
+
+#include <linux/bitops.h>
+#include <linux/clk.h>
+#include <linux/gpio/consumer.h>
+#include <linux/io.h>
+#include <linux/mmc/host.h>
+#include <linux/of.h>
+#include <linux/scatterlist.h>
+#include <linux/semaphore.h>
+
+#define CAVIUM_MAX_MMC 4
+
+/* DMA register addresses */
+#define MIO_EMM_DMA_FIFO_CFG(x) (0x00 + x->reg_off_dma)
+#define MIO_EMM_DMA_FIFO_ADR(x) (0x10 + x->reg_off_dma)
+#define MIO_EMM_DMA_FIFO_CMD(x) (0x18 + x->reg_off_dma)
+#define MIO_EMM_DMA_CFG(x) (0x20 + x->reg_off_dma)
+#define MIO_EMM_DMA_ADR(x) (0x28 + x->reg_off_dma)
+#define MIO_EMM_DMA_INT(x) (0x30 + x->reg_off_dma)
+#define MIO_EMM_DMA_INT_W1S(x) (0x38 + x->reg_off_dma)
+#define MIO_EMM_DMA_INT_ENA_W1S(x) (0x40 + x->reg_off_dma)
+#define MIO_EMM_DMA_INT_ENA_W1C(x) (0x48 + x->reg_off_dma)
+
+/* register addresses */
+#define MIO_EMM_CFG(x) (0x00 + x->reg_off)
+#define MIO_EMM_SWITCH(x) (0x48 + x->reg_off)
+#define MIO_EMM_DMA(x) (0x50 + x->reg_off)
+#define MIO_EMM_CMD(x) (0x58 + x->reg_off)
+#define MIO_EMM_RSP_STS(x) (0x60 + x->reg_off)
+#define MIO_EMM_RSP_LO(x) (0x68 + x->reg_off)
+#define MIO_EMM_RSP_HI(x) (0x70 + x->reg_off)
+#define MIO_EMM_INT(x) (0x78 + x->reg_off)
+#define MIO_EMM_INT_EN(x) (0x80 + x->reg_off)
+#define MIO_EMM_WDOG(x) (0x88 + x->reg_off)
+#define MIO_EMM_SAMPLE(x) (0x90 + x->reg_off)
+#define MIO_EMM_STS_MASK(x) (0x98 + x->reg_off)
+#define MIO_EMM_RCA(x) (0xa0 + x->reg_off)
+#define MIO_EMM_INT_EN_SET(x) (0xb0 + x->reg_off)
+#define MIO_EMM_INT_EN_CLR(x) (0xb8 + x->reg_off)
+#define MIO_EMM_BUF_IDX(x) (0xe0 + x->reg_off)
+#define MIO_EMM_BUF_DAT(x) (0xe8 + x->reg_off)
+
+struct cvm_mmc_host {
+ struct device *dev;
+ void __iomem *base;
+ void __iomem *dma_base;
+ int reg_off;
+ int reg_off_dma;
+ u64 emm_cfg;
+ u64 n_minus_one; /* OCTEON II workaround location */
+ int last_slot;
+ struct clk *clk;
+ int sys_freq;
+
+ struct mmc_request *current_req;
+ struct sg_mapping_iter smi;
+ bool dma_active;
+ bool use_sg;
+
+ bool has_ciu3;
+ bool big_dma_addr;
+ bool need_irq_handler_lock;
+ spinlock_t irq_handler_lock;
+ struct semaphore mmc_serializer;
+
+ struct gpio_desc *global_pwr_gpiod;
+ atomic_t shared_power_users;
+
+ struct cvm_mmc_slot *slot[CAVIUM_MAX_MMC];
+ struct platform_device *slot_pdev[CAVIUM_MAX_MMC];
+
+ void (*set_shared_power)(struct cvm_mmc_host *, int);
+ void (*acquire_bus)(struct cvm_mmc_host *);
+ void (*release_bus)(struct cvm_mmc_host *);
+ void (*int_enable)(struct cvm_mmc_host *, u64);
+ /* required on some MIPS models */
+ void (*dmar_fixup)(struct cvm_mmc_host *, struct mmc_command *,
+ struct mmc_data *, u64);
+ void (*dmar_fixup_done)(struct cvm_mmc_host *);
+};
+
+struct cvm_mmc_slot {
+ struct mmc_host *mmc; /* slot-level mmc_core object */
+ struct cvm_mmc_host *host; /* common hw for all slots */
+
+ u64 clock;
+
+ u64 cached_switch;
+ u64 cached_rca;
+
+ unsigned int cmd_cnt; /* sample delay */
+ unsigned int dat_cnt; /* sample delay */
+
+ int bus_id;
+};
+
+struct cvm_mmc_cr_type {
+ u8 ctype;
+ u8 rtype;
+};
+
+struct cvm_mmc_cr_mods {
+ u8 ctype_xor;
+ u8 rtype_xor;
+};
+
+/* Bitfield definitions */
+#define MIO_EMM_DMA_FIFO_CFG_CLR BIT_ULL(16)
+#define MIO_EMM_DMA_FIFO_CFG_INT_LVL GENMASK_ULL(12, 8)
+#define MIO_EMM_DMA_FIFO_CFG_COUNT GENMASK_ULL(4, 0)
+
+#define MIO_EMM_DMA_FIFO_CMD_RW BIT_ULL(62)
+#define MIO_EMM_DMA_FIFO_CMD_INTDIS BIT_ULL(60)
+#define MIO_EMM_DMA_FIFO_CMD_SWAP32 BIT_ULL(59)
+#define MIO_EMM_DMA_FIFO_CMD_SWAP16 BIT_ULL(58)
+#define MIO_EMM_DMA_FIFO_CMD_SWAP8 BIT_ULL(57)
+#define MIO_EMM_DMA_FIFO_CMD_ENDIAN BIT_ULL(56)
+#define MIO_EMM_DMA_FIFO_CMD_SIZE GENMASK_ULL(55, 36)
+
+#define MIO_EMM_CMD_SKIP_BUSY BIT_ULL(62)
+#define MIO_EMM_CMD_BUS_ID GENMASK_ULL(61, 60)
+#define MIO_EMM_CMD_VAL BIT_ULL(59)
+#define MIO_EMM_CMD_DBUF BIT_ULL(55)
+#define MIO_EMM_CMD_OFFSET GENMASK_ULL(54, 49)
+#define MIO_EMM_CMD_CTYPE_XOR GENMASK_ULL(42, 41)
+#define MIO_EMM_CMD_RTYPE_XOR GENMASK_ULL(40, 38)
+#define MIO_EMM_CMD_IDX GENMASK_ULL(37, 32)
+#define MIO_EMM_CMD_ARG GENMASK_ULL(31, 0)
+
+#define MIO_EMM_DMA_SKIP_BUSY BIT_ULL(62)
+#define MIO_EMM_DMA_BUS_ID GENMASK_ULL(61, 60)
+#define MIO_EMM_DMA_VAL BIT_ULL(59)
+#define MIO_EMM_DMA_SECTOR BIT_ULL(58)
+#define MIO_EMM_DMA_DAT_NULL BIT_ULL(57)
+#define MIO_EMM_DMA_THRES GENMASK_ULL(56, 51)
+#define MIO_EMM_DMA_REL_WR BIT_ULL(50)
+#define MIO_EMM_DMA_RW BIT_ULL(49)
+#define MIO_EMM_DMA_MULTI BIT_ULL(48)
+#define MIO_EMM_DMA_BLOCK_CNT GENMASK_ULL(47, 32)
+#define MIO_EMM_DMA_CARD_ADDR GENMASK_ULL(31, 0)
+
+#define MIO_EMM_DMA_CFG_EN BIT_ULL(63)
+#define MIO_EMM_DMA_CFG_RW BIT_ULL(62)
+#define MIO_EMM_DMA_CFG_CLR BIT_ULL(61)
+#define MIO_EMM_DMA_CFG_SWAP32 BIT_ULL(59)
+#define MIO_EMM_DMA_CFG_SWAP16 BIT_ULL(58)
+#define MIO_EMM_DMA_CFG_SWAP8 BIT_ULL(57)
+#define MIO_EMM_DMA_CFG_ENDIAN BIT_ULL(56)
+#define MIO_EMM_DMA_CFG_SIZE GENMASK_ULL(55, 36)
+#define MIO_EMM_DMA_CFG_ADR GENMASK_ULL(35, 0)
+
+#define MIO_EMM_INT_SWITCH_ERR BIT_ULL(6)
+#define MIO_EMM_INT_SWITCH_DONE BIT_ULL(5)
+#define MIO_EMM_INT_DMA_ERR BIT_ULL(4)
+#define MIO_EMM_INT_CMD_ERR BIT_ULL(3)
+#define MIO_EMM_INT_DMA_DONE BIT_ULL(2)
+#define MIO_EMM_INT_CMD_DONE BIT_ULL(1)
+#define MIO_EMM_INT_BUF_DONE BIT_ULL(0)
+
+#define MIO_EMM_RSP_STS_BUS_ID GENMASK_ULL(61, 60)
+#define MIO_EMM_RSP_STS_CMD_VAL BIT_ULL(59)
+#define MIO_EMM_RSP_STS_SWITCH_VAL BIT_ULL(58)
+#define MIO_EMM_RSP_STS_DMA_VAL BIT_ULL(57)
+#define MIO_EMM_RSP_STS_DMA_PEND BIT_ULL(56)
+#define MIO_EMM_RSP_STS_DBUF_ERR BIT_ULL(28)
+#define MIO_EMM_RSP_STS_DBUF BIT_ULL(23)
+#define MIO_EMM_RSP_STS_BLK_TIMEOUT BIT_ULL(22)
+#define MIO_EMM_RSP_STS_BLK_CRC_ERR BIT_ULL(21)
+#define MIO_EMM_RSP_STS_RSP_BUSYBIT BIT_ULL(20)
+#define MIO_EMM_RSP_STS_STP_TIMEOUT BIT_ULL(19)
+#define MIO_EMM_RSP_STS_STP_CRC_ERR BIT_ULL(18)
+#define MIO_EMM_RSP_STS_STP_BAD_STS BIT_ULL(17)
+#define MIO_EMM_RSP_STS_STP_VAL BIT_ULL(16)
+#define MIO_EMM_RSP_STS_RSP_TIMEOUT BIT_ULL(15)
+#define MIO_EMM_RSP_STS_RSP_CRC_ERR BIT_ULL(14)
+#define MIO_EMM_RSP_STS_RSP_BAD_STS BIT_ULL(13)
+#define MIO_EMM_RSP_STS_RSP_VAL BIT_ULL(12)
+#define MIO_EMM_RSP_STS_RSP_TYPE GENMASK_ULL(11, 9)
+#define MIO_EMM_RSP_STS_CMD_TYPE GENMASK_ULL(8, 7)
+#define MIO_EMM_RSP_STS_CMD_IDX GENMASK_ULL(6, 1)
+#define MIO_EMM_RSP_STS_CMD_DONE BIT_ULL(0)
+
+#define MIO_EMM_SAMPLE_CMD_CNT GENMASK_ULL(25, 16)
+#define MIO_EMM_SAMPLE_DAT_CNT GENMASK_ULL(9, 0)
+
+#define MIO_EMM_SWITCH_BUS_ID GENMASK_ULL(61, 60)
+#define MIO_EMM_SWITCH_EXE BIT_ULL(59)
+#define MIO_EMM_SWITCH_ERR0 BIT_ULL(58)
+#define MIO_EMM_SWITCH_ERR1 BIT_ULL(57)
+#define MIO_EMM_SWITCH_ERR2 BIT_ULL(56)
+#define MIO_EMM_SWITCH_HS_TIMING BIT_ULL(48)
+#define MIO_EMM_SWITCH_BUS_WIDTH GENMASK_ULL(42, 40)
+#define MIO_EMM_SWITCH_POWER_CLASS GENMASK_ULL(35, 32)
+#define MIO_EMM_SWITCH_CLK_HI GENMASK_ULL(31, 16)
+#define MIO_EMM_SWITCH_CLK_LO GENMASK_ULL(15, 0)
+
+/* Protoypes */
+irqreturn_t cvm_mmc_interrupt(int irq, void *dev_id);
+int cvm_mmc_of_slot_probe(struct device *dev, struct cvm_mmc_host *host);
+int cvm_mmc_of_slot_remove(struct cvm_mmc_slot *slot);
+extern const char *cvm_mmc_irq_names[];
+
+#endif
diff --git a/drivers/mmc/host/davinci_mmc.c b/drivers/mmc/host/davinci_mmc.c
index 1e2600d..621ce47 100644
--- a/drivers/mmc/host/davinci_mmc.c
+++ b/drivers/mmc/host/davinci_mmc.c
@@ -478,18 +478,14 @@
int ret = 0;
host->sg_len = dma_map_sg(mmc_dev(host->mmc), data->sg, data->sg_len,
- ((data->flags & MMC_DATA_WRITE)
- ? DMA_TO_DEVICE
- : DMA_FROM_DEVICE));
+ mmc_get_dma_dir(data));
/* no individual DMA segment should need a partial FIFO */
for (i = 0; i < host->sg_len; i++) {
if (sg_dma_len(data->sg + i) & mask) {
dma_unmap_sg(mmc_dev(host->mmc),
- data->sg, data->sg_len,
- (data->flags & MMC_DATA_WRITE)
- ? DMA_TO_DEVICE
- : DMA_FROM_DEVICE);
+ data->sg, data->sg_len,
+ mmc_get_dma_dir(data));
return -1;
}
}
@@ -802,9 +798,7 @@
davinci_abort_dma(host);
dma_unmap_sg(mmc_dev(host->mmc), data->sg, data->sg_len,
- (data->flags & MMC_DATA_WRITE)
- ? DMA_TO_DEVICE
- : DMA_FROM_DEVICE);
+ mmc_get_dma_dir(data));
host->do_dma = false;
}
host->data_dir = DAVINCI_MMC_DATADIR_NONE;
diff --git a/drivers/mmc/host/dw_mmc.c b/drivers/mmc/host/dw_mmc.c
index 8718432..e45129f 100644
--- a/drivers/mmc/host/dw_mmc.c
+++ b/drivers/mmc/host/dw_mmc.c
@@ -19,6 +19,7 @@
#include <linux/err.h>
#include <linux/init.h>
#include <linux/interrupt.h>
+#include <linux/iopoll.h>
#include <linux/ioport.h>
#include <linux/module.h>
#include <linux/platform_device.h>
@@ -65,6 +66,8 @@
struct idmac_desc_64addr {
u32 des0; /* Control Descriptor */
+#define IDMAC_OWN_CLR64(x) \
+ !((x) & cpu_to_le32(IDMAC_DES0_OWN))
u32 des1; /* Reserved */
@@ -104,11 +107,6 @@
/* Each descriptor can transfer up to 4KB of data in chained mode */
#define DW_MCI_DESC_DATA_LENGTH 0x1000
-static bool dw_mci_reset(struct dw_mci *host);
-static bool dw_mci_ctrl_reset(struct dw_mci *host, u32 reset);
-static int dw_mci_card_busy(struct mmc_host *mmc);
-static int dw_mci_get_cd(struct mmc_host *mmc);
-
#if defined(CONFIG_DEBUG_FS)
static int dw_mci_req_show(struct seq_file *s, void *v)
{
@@ -232,7 +230,66 @@
}
#endif /* defined(CONFIG_DEBUG_FS) */
-static void mci_send_cmd(struct dw_mci_slot *slot, u32 cmd, u32 arg);
+static bool dw_mci_ctrl_reset(struct dw_mci *host, u32 reset)
+{
+ u32 ctrl;
+
+ ctrl = mci_readl(host, CTRL);
+ ctrl |= reset;
+ mci_writel(host, CTRL, ctrl);
+
+ /* wait till resets clear */
+ if (readl_poll_timeout_atomic(host->regs + SDMMC_CTRL, ctrl,
+ !(ctrl & reset),
+ 1, 500 * USEC_PER_MSEC)) {
+ dev_err(host->dev,
+ "Timeout resetting block (ctrl reset %#x)\n",
+ ctrl & reset);
+ return false;
+ }
+
+ return true;
+}
+
+static void dw_mci_wait_while_busy(struct dw_mci *host, u32 cmd_flags)
+{
+ u32 status;
+
+ /*
+ * Databook says that before issuing a new data transfer command
+ * we need to check to see if the card is busy. Data transfer commands
+ * all have SDMMC_CMD_PRV_DAT_WAIT set, so we'll key off that.
+ *
+ * ...also allow sending for SDMMC_CMD_VOLT_SWITCH where busy is
+ * expected.
+ */
+ if ((cmd_flags & SDMMC_CMD_PRV_DAT_WAIT) &&
+ !(cmd_flags & SDMMC_CMD_VOLT_SWITCH)) {
+ if (readl_poll_timeout_atomic(host->regs + SDMMC_STATUS,
+ status,
+ !(status & SDMMC_STATUS_BUSY),
+ 10, 500 * USEC_PER_MSEC))
+ dev_err(host->dev, "Busy; trying anyway\n");
+ }
+}
+
+static void mci_send_cmd(struct dw_mci_slot *slot, u32 cmd, u32 arg)
+{
+ struct dw_mci *host = slot->host;
+ unsigned int cmd_status = 0;
+
+ mci_writel(host, CMDARG, arg);
+ wmb(); /* drain writebuffer */
+ dw_mci_wait_while_busy(host, cmd);
+ mci_writel(host, CMD, SDMMC_CMD_START | cmd);
+
+ if (readl_poll_timeout_atomic(host->regs + SDMMC_CMD, cmd_status,
+ !(cmd_status & SDMMC_CMD_START),
+ 1, 500 * USEC_PER_MSEC))
+ dev_err(&slot->mmc->class_dev,
+ "Timeout sending command (cmd %#x arg %#x status %#x)\n",
+ cmd, arg, cmd_status);
+}
static u32 dw_mci_prepare_command(struct mmc_host *mmc, struct mmc_command *cmd)
{
@@ -341,31 +398,6 @@
return cmdr;
}
-static void dw_mci_wait_while_busy(struct dw_mci *host, u32 cmd_flags)
-{
- unsigned long timeout = jiffies + msecs_to_jiffies(500);
-
- /*
- * Databook says that before issuing a new data transfer command
- * we need to check to see if the card is busy. Data transfer commands
- * all have SDMMC_CMD_PRV_DAT_WAIT set, so we'll key off that.
- *
- * ...also allow sending for SDMMC_CMD_VOLT_SWITCH where busy is
- * expected.
- */
- if ((cmd_flags & SDMMC_CMD_PRV_DAT_WAIT) &&
- !(cmd_flags & SDMMC_CMD_VOLT_SWITCH)) {
- while (mci_readl(host, STATUS) & SDMMC_STATUS_BUSY) {
- if (time_after(jiffies, timeout)) {
- /* Command will fail; we'll pass error then */
- dev_err(host->dev, "Busy; trying anyway\n");
- break;
- }
- udelay(10);
- }
- }
-}
-
static void dw_mci_start_command(struct dw_mci *host,
struct mmc_command *cmd, u32 cmd_flags)
{
@@ -400,14 +432,6 @@
set_bit(EVENT_XFER_COMPLETE, &host->pending_events);
}
-static int dw_mci_get_dma_dir(struct mmc_data *data)
-{
- if (data->flags & MMC_DATA_WRITE)
- return DMA_TO_DEVICE;
- else
- return DMA_FROM_DEVICE;
-}
-
static void dw_mci_dma_cleanup(struct dw_mci *host)
{
struct mmc_data *data = host->data;
@@ -416,7 +440,7 @@
dma_unmap_sg(host->dev,
data->sg,
data->sg_len,
- dw_mci_get_dma_dir(data));
+ mmc_get_dma_dir(data));
data->host_cookie = COOKIE_UNMAPPED;
}
}
@@ -555,7 +579,7 @@
{
unsigned int desc_len;
struct idmac_desc_64addr *desc_first, *desc_last, *desc;
- unsigned long timeout;
+ u32 val;
int i;
desc_first = desc_last = desc = host->sg_cpu;
@@ -577,12 +601,10 @@
* isn't still owned by IDMAC as IDMAC's write
* ops and CPU's read ops are asynchronous.
*/
- timeout = jiffies + msecs_to_jiffies(100);
- while (readl(&desc->des0) & IDMAC_DES0_OWN) {
- if (time_after(jiffies, timeout))
- goto err_own_bit;
- udelay(10);
- }
+ if (readl_poll_timeout_atomic(&desc->des0, val,
+ !(val & IDMAC_DES0_OWN),
+ 10, 100 * USEC_PER_MSEC))
+ goto err_own_bit;
/*
* Set the OWN bit and disable interrupts
@@ -629,7 +651,7 @@
{
unsigned int desc_len;
struct idmac_desc *desc_first, *desc_last, *desc;
- unsigned long timeout;
+ u32 val;
int i;
desc_first = desc_last = desc = host->sg_cpu;
@@ -651,13 +673,11 @@
* isn't still owned by IDMAC as IDMAC's write
* ops and CPU's read ops are asynchronous.
*/
- timeout = jiffies + msecs_to_jiffies(100);
- while (readl(&desc->des0) &
- cpu_to_le32(IDMAC_DES0_OWN)) {
- if (time_after(jiffies, timeout))
- goto err_own_bit;
- udelay(10);
- }
+ if (readl_poll_timeout_atomic(&desc->des0, val,
+ IDMAC_OWN_CLR64(val),
+ 10,
+ 100 * USEC_PER_MSEC))
+ goto err_own_bit;
/*
* Set the OWN bit and disable interrupts
@@ -876,7 +896,7 @@
sg_len = dma_map_sg(host->dev,
data->sg,
data->sg_len,
- dw_mci_get_dma_dir(data));
+ mmc_get_dma_dir(data));
if (sg_len == 0)
return -EINVAL;
@@ -916,10 +936,51 @@
dma_unmap_sg(slot->host->dev,
data->sg,
data->sg_len,
- dw_mci_get_dma_dir(data));
+ mmc_get_dma_dir(data));
data->host_cookie = COOKIE_UNMAPPED;
}
+static int dw_mci_get_cd(struct mmc_host *mmc)
+{
+ int present;
+ struct dw_mci_slot *slot = mmc_priv(mmc);
+ struct dw_mci *host = slot->host;
+ int gpio_cd = mmc_gpio_get_cd(mmc);
+
+ /* Use platform get_cd function, else try onboard card detect */
+ if (((mmc->caps & MMC_CAP_NEEDS_POLL)
+ || !mmc_card_is_removable(mmc))) {
+ present = 1;
+
+ if (!test_bit(DW_MMC_CARD_PRESENT, &slot->flags)) {
+ if (mmc->caps & MMC_CAP_NEEDS_POLL) {
+ dev_info(&mmc->class_dev,
+ "card is polling.\n");
+ } else {
+ dev_info(&mmc->class_dev,
+ "card is non-removable.\n");
+ }
+ set_bit(DW_MMC_CARD_PRESENT, &slot->flags);
+ }
+
+ return present;
+ } else if (gpio_cd >= 0)
+ present = gpio_cd;
+ else
+ present = (mci_readl(slot->host, CDETECT) & (1 << slot->id))
+ == 0 ? 1 : 0;
+
+ spin_lock_bh(&host->lock);
+ if (present && !test_and_set_bit(DW_MMC_CARD_PRESENT, &slot->flags))
+ dev_dbg(&mmc->class_dev, "card is present\n");
+ else if (!present &&
+ !test_and_clear_bit(DW_MMC_CARD_PRESENT, &slot->flags))
+ dev_dbg(&mmc->class_dev, "card is not present\n");
+ spin_unlock_bh(&host->lock);
+
+ return present;
+}
+
static void dw_mci_adjust_fifoth(struct dw_mci *host, struct mmc_data *data)
{
unsigned int blksz = data->blksz;
@@ -1133,27 +1194,6 @@
}
}
-static void mci_send_cmd(struct dw_mci_slot *slot, u32 cmd, u32 arg)
-{
- struct dw_mci *host = slot->host;
- unsigned long timeout = jiffies + msecs_to_jiffies(500);
- unsigned int cmd_status = 0;
-
- mci_writel(host, CMDARG, arg);
- wmb(); /* drain writebuffer */
- dw_mci_wait_while_busy(host, cmd);
- mci_writel(host, CMD, SDMMC_CMD_START | cmd);
-
- while (time_before(jiffies, timeout)) {
- cmd_status = mci_readl(host, CMD);
- if (!(cmd_status & SDMMC_CMD_START))
- return;
- }
- dev_err(&slot->mmc->class_dev,
- "Timeout sending command (cmd %#x arg %#x status %#x)\n",
- cmd, arg, cmd_status);
-}
-
static void dw_mci_setup_bus(struct dw_mci_slot *slot, bool force_clkinit)
{
struct dw_mci *host = slot->host;
@@ -1534,47 +1574,6 @@
return read_only;
}
-static int dw_mci_get_cd(struct mmc_host *mmc)
-{
- int present;
- struct dw_mci_slot *slot = mmc_priv(mmc);
- struct dw_mci *host = slot->host;
- int gpio_cd = mmc_gpio_get_cd(mmc);
-
- /* Use platform get_cd function, else try onboard card detect */
- if (((mmc->caps & MMC_CAP_NEEDS_POLL)
- || !mmc_card_is_removable(mmc))) {
- present = 1;
-
- if (!test_bit(DW_MMC_CARD_PRESENT, &slot->flags)) {
- if (mmc->caps & MMC_CAP_NEEDS_POLL) {
- dev_info(&mmc->class_dev,
- "card is polling.\n");
- } else {
- dev_info(&mmc->class_dev,
- "card is non-removable.\n");
- }
- set_bit(DW_MMC_CARD_PRESENT, &slot->flags);
- }
-
- return present;
- } else if (gpio_cd >= 0)
- present = gpio_cd;
- else
- present = (mci_readl(slot->host, CDETECT) & (1 << slot->id))
- == 0 ? 1 : 0;
-
- spin_lock_bh(&host->lock);
- if (present && !test_and_set_bit(DW_MMC_CARD_PRESENT, &slot->flags))
- dev_dbg(&mmc->class_dev, "card is present\n");
- else if (!present &&
- !test_and_clear_bit(DW_MMC_CARD_PRESENT, &slot->flags))
- dev_dbg(&mmc->class_dev, "card is not present\n");
- spin_unlock_bh(&host->lock);
-
- return present;
-}
-
static void dw_mci_hw_reset(struct mmc_host *mmc)
{
struct dw_mci_slot *slot = mmc_priv(mmc);
@@ -1688,6 +1687,73 @@
return 0;
}
+static bool dw_mci_reset(struct dw_mci *host)
+{
+ u32 flags = SDMMC_CTRL_RESET | SDMMC_CTRL_FIFO_RESET;
+ bool ret = false;
+ u32 status = 0;
+
+ /*
+ * Resetting generates a block interrupt, hence setting
+ * the scatter-gather pointer to NULL.
+ */
+ if (host->sg) {
+ sg_miter_stop(&host->sg_miter);
+ host->sg = NULL;
+ }
+
+ if (host->use_dma)
+ flags |= SDMMC_CTRL_DMA_RESET;
+
+ if (dw_mci_ctrl_reset(host, flags)) {
+ /*
+ * In all cases we clear the RAWINTS
+ * register to clear any interrupts.
+ */
+ mci_writel(host, RINTSTS, 0xFFFFFFFF);
+
+ if (!host->use_dma) {
+ ret = true;
+ goto ciu_out;
+ }
+
+ /* Wait for dma_req to be cleared */
+ if (readl_poll_timeout_atomic(host->regs + SDMMC_STATUS,
+ status,
+ !(status & SDMMC_STATUS_DMA_REQ),
+ 1, 500 * USEC_PER_MSEC)) {
+ dev_err(host->dev,
+ "%s: Timeout waiting for dma_req to be cleared\n",
+ __func__);
+ goto ciu_out;
+ }
+
+ /* when using DMA next we reset the fifo again */
+ if (!dw_mci_ctrl_reset(host, SDMMC_CTRL_FIFO_RESET))
+ goto ciu_out;
+ } else {
+ /* if the controller reset bit did clear, then set clock regs */
+ if (!(mci_readl(host, CTRL) & SDMMC_CTRL_RESET)) {
+ dev_err(host->dev,
+ "%s: fifo/dma reset bits didn't clear but ciu was reset, doing clock update\n",
+ __func__);
+ goto ciu_out;
+ }
+ }
+
+ if (host->use_dma == TRANS_MODE_IDMAC)
+ /* It is also recommended that we reset and reprogram idmac */
+ dw_mci_idmac_reset(host);
+
+ ret = true;
+
+ciu_out:
+ /* After a CTRL reset we need to have CIU set clock registers */
+ mci_send_cmd(host->cur_slot, SDMMC_CMD_UPD_CLK, 0);
+
+ return ret;
+}
+
static const struct mmc_host_ops dw_mci_ops = {
.request = dw_mci_request,
.pre_req = dw_mci_pre_req,
@@ -2830,99 +2896,6 @@
host->use_dma = TRANS_MODE_PIO;
}
-static bool dw_mci_ctrl_reset(struct dw_mci *host, u32 reset)
-{
- unsigned long timeout = jiffies + msecs_to_jiffies(500);
- u32 ctrl;
-
- ctrl = mci_readl(host, CTRL);
- ctrl |= reset;
- mci_writel(host, CTRL, ctrl);
-
- /* wait till resets clear */
- do {
- ctrl = mci_readl(host, CTRL);
- if (!(ctrl & reset))
- return true;
- } while (time_before(jiffies, timeout));
-
- dev_err(host->dev,
- "Timeout resetting block (ctrl reset %#x)\n",
- ctrl & reset);
-
- return false;
-}
-
-static bool dw_mci_reset(struct dw_mci *host)
-{
- u32 flags = SDMMC_CTRL_RESET | SDMMC_CTRL_FIFO_RESET;
- bool ret = false;
-
- /*
- * Reseting generates a block interrupt, hence setting
- * the scatter-gather pointer to NULL.
- */
- if (host->sg) {
- sg_miter_stop(&host->sg_miter);
- host->sg = NULL;
- }
-
- if (host->use_dma)
- flags |= SDMMC_CTRL_DMA_RESET;
-
- if (dw_mci_ctrl_reset(host, flags)) {
- /*
- * In all cases we clear the RAWINTS register to clear any
- * interrupts.
- */
- mci_writel(host, RINTSTS, 0xFFFFFFFF);
-
- /* if using dma we wait for dma_req to clear */
- if (host->use_dma) {
- unsigned long timeout = jiffies + msecs_to_jiffies(500);
- u32 status;
-
- do {
- status = mci_readl(host, STATUS);
- if (!(status & SDMMC_STATUS_DMA_REQ))
- break;
- cpu_relax();
- } while (time_before(jiffies, timeout));
-
- if (status & SDMMC_STATUS_DMA_REQ) {
- dev_err(host->dev,
- "%s: Timeout waiting for dma_req to clear during reset\n",
- __func__);
- goto ciu_out;
- }
-
- /* when using DMA next we reset the fifo again */
- if (!dw_mci_ctrl_reset(host, SDMMC_CTRL_FIFO_RESET))
- goto ciu_out;
- }
- } else {
- /* if the controller reset bit did clear, then set clock regs */
- if (!(mci_readl(host, CTRL) & SDMMC_CTRL_RESET)) {
- dev_err(host->dev,
- "%s: fifo/dma reset bits didn't clear but ciu was reset, doing clock update\n",
- __func__);
- goto ciu_out;
- }
- }
-
- if (host->use_dma == TRANS_MODE_IDMAC)
- /* It is also recommended that we reset and reprogram idmac */
- dw_mci_idmac_reset(host);
-
- ret = true;
-
-ciu_out:
- /* After a CTRL reset we need to have CIU set clock registers */
- mci_send_cmd(host->cur_slot, SDMMC_CMD_UPD_CLK, 0);
-
- return ret;
-}
-
static void dw_mci_cmd11_timer(unsigned long arg)
{
struct dw_mci *host = (struct dw_mci *)arg;
diff --git a/drivers/mmc/host/jz4740_mmc.c b/drivers/mmc/host/jz4740_mmc.c
index 819ad32..57e254a 100644
--- a/drivers/mmc/host/jz4740_mmc.c
+++ b/drivers/mmc/host/jz4740_mmc.c
@@ -200,11 +200,6 @@
return -ENODEV;
}
-static inline int jz4740_mmc_get_dma_dir(struct mmc_data *data)
-{
- return (data->flags & MMC_DATA_READ) ? DMA_FROM_DEVICE : DMA_TO_DEVICE;
-}
-
static inline struct dma_chan *jz4740_mmc_get_dma_chan(struct jz4740_mmc_host *host,
struct mmc_data *data)
{
@@ -215,7 +210,7 @@
struct mmc_data *data)
{
struct dma_chan *chan = jz4740_mmc_get_dma_chan(host, data);
- enum dma_data_direction dir = jz4740_mmc_get_dma_dir(data);
+ enum dma_data_direction dir = mmc_get_dma_dir(data);
dma_unmap_sg(chan->device->dev, data->sg, data->sg_len, dir);
}
@@ -227,7 +222,7 @@
struct dma_chan *chan)
{
struct jz4740_mmc_host_next *next_data = &host->next_data;
- enum dma_data_direction dir = jz4740_mmc_get_dma_dir(data);
+ enum dma_data_direction dir = mmc_get_dma_dir(data);
int sg_len;
if (!next && data->host_cookie &&
diff --git a/drivers/mmc/host/meson-gx-mmc.c b/drivers/mmc/host/meson-gx-mmc.c
index 5a95978..1842ed3 100644
--- a/drivers/mmc/host/meson-gx-mmc.c
+++ b/drivers/mmc/host/meson-gx-mmc.c
@@ -36,23 +36,21 @@
#include <linux/clk-provider.h>
#include <linux/regulator/consumer.h>
#include <linux/interrupt.h>
+#include <linux/bitfield.h>
#define DRIVER_NAME "meson-gx-mmc"
#define SD_EMMC_CLOCK 0x0
-#define CLK_DIV_SHIFT 0
-#define CLK_DIV_WIDTH 6
-#define CLK_DIV_MASK 0x3f
+#define CLK_DIV_MASK GENMASK(5, 0)
#define CLK_DIV_MAX 63
-#define CLK_SRC_SHIFT 6
-#define CLK_SRC_WIDTH 2
-#define CLK_SRC_MASK 0x3
+#define CLK_SRC_MASK GENMASK(7, 6)
#define CLK_SRC_XTAL 0 /* external crystal */
#define CLK_SRC_XTAL_RATE 24000000
#define CLK_SRC_PLL 1 /* FCLK_DIV2 */
#define CLK_SRC_PLL_RATE 1000000000
-#define CLK_PHASE_SHIFT 8
-#define CLK_PHASE_MASK 0x3
+#define CLK_CORE_PHASE_MASK GENMASK(9, 8)
+#define CLK_TX_PHASE_MASK GENMASK(11, 10)
+#define CLK_RX_PHASE_MASK GENMASK(13, 12)
#define CLK_PHASE_0 0
#define CLK_PHASE_90 1
#define CLK_PHASE_180 2
@@ -65,22 +63,17 @@
#define SD_EMMC_START 0x40
#define START_DESC_INIT BIT(0)
#define START_DESC_BUSY BIT(1)
-#define START_DESC_ADDR_SHIFT 2
-#define START_DESC_ADDR_MASK (~0x3)
+#define START_DESC_ADDR_MASK GENMASK(31, 2)
#define SD_EMMC_CFG 0x44
-#define CFG_BUS_WIDTH_SHIFT 0
-#define CFG_BUS_WIDTH_MASK 0x3
+#define CFG_BUS_WIDTH_MASK GENMASK(1, 0)
#define CFG_BUS_WIDTH_1 0x0
#define CFG_BUS_WIDTH_4 0x1
#define CFG_BUS_WIDTH_8 0x2
#define CFG_DDR BIT(2)
-#define CFG_BLK_LEN_SHIFT 4
-#define CFG_BLK_LEN_MASK 0xf
-#define CFG_RESP_TIMEOUT_SHIFT 8
-#define CFG_RESP_TIMEOUT_MASK 0xf
-#define CFG_RC_CC_SHIFT 12
-#define CFG_RC_CC_MASK 0xf
+#define CFG_BLK_LEN_MASK GENMASK(7, 4)
+#define CFG_RESP_TIMEOUT_MASK GENMASK(11, 8)
+#define CFG_RC_CC_MASK GENMASK(15, 12)
#define CFG_STOP_CLOCK BIT(22)
#define CFG_CLK_ALWAYS_ON BIT(18)
#define CFG_CHK_DS BIT(20)
@@ -90,9 +83,8 @@
#define STATUS_BUSY BIT(31)
#define SD_EMMC_IRQ_EN 0x4c
-#define IRQ_EN_MASK 0x3fff
-#define IRQ_RXD_ERR_SHIFT 0
-#define IRQ_RXD_ERR_MASK 0xff
+#define IRQ_EN_MASK GENMASK(13, 0)
+#define IRQ_RXD_ERR_MASK GENMASK(7, 0)
#define IRQ_TXD_ERR BIT(8)
#define IRQ_DESC_ERR BIT(9)
#define IRQ_RESP_ERR BIT(10)
@@ -116,33 +108,20 @@
#define SD_EMMC_CFG_BLK_SIZE 512 /* internal buffer max: 512 bytes */
#define SD_EMMC_CFG_RESP_TIMEOUT 256 /* in clock cycles */
+#define SD_EMMC_CMD_TIMEOUT 1024 /* in ms */
+#define SD_EMMC_CMD_TIMEOUT_DATA 4096 /* in ms */
#define SD_EMMC_CFG_CMD_GAP 16 /* in clock cycles */
+#define SD_EMMC_DESC_BUF_LEN PAGE_SIZE
+
+#define SD_EMMC_PRE_REQ_DONE BIT(0)
+#define SD_EMMC_DESC_CHAIN_MODE BIT(1)
+
#define MUX_CLK_NUM_PARENTS 2
-struct meson_host {
- struct device *dev;
- struct mmc_host *mmc;
- struct mmc_request *mrq;
- struct mmc_command *cmd;
-
- spinlock_t lock;
- void __iomem *regs;
- int irq;
- u32 ocr_mask;
- struct clk *core_clk;
- struct clk_mux mux;
- struct clk *mux_clk;
- struct clk *mux_parent[MUX_CLK_NUM_PARENTS];
- unsigned long current_clock;
-
- struct clk_divider cfg_div;
- struct clk *cfg_div_clk;
-
- unsigned int bounce_buf_size;
- void *bounce_buf;
- dma_addr_t bounce_dma_addr;
-
- bool vqmmc_enabled;
+struct meson_tuning_params {
+ u8 core_phase;
+ u8 tx_phase;
+ u8 rx_phase;
};
struct sd_emmc_desc {
@@ -151,13 +130,37 @@
u32 cmd_data;
u32 cmd_resp;
};
-#define CMD_CFG_LENGTH_SHIFT 0
-#define CMD_CFG_LENGTH_MASK 0x1ff
+
+struct meson_host {
+ struct device *dev;
+ struct mmc_host *mmc;
+ struct mmc_command *cmd;
+
+ spinlock_t lock;
+ void __iomem *regs;
+ struct clk *core_clk;
+ struct clk_mux mux;
+ struct clk *mux_clk;
+ unsigned long current_clock;
+
+ struct clk_divider cfg_div;
+ struct clk *cfg_div_clk;
+
+ unsigned int bounce_buf_size;
+ void *bounce_buf;
+ dma_addr_t bounce_dma_addr;
+ struct sd_emmc_desc *descs;
+ dma_addr_t descs_dma_addr;
+
+ struct meson_tuning_params tp;
+ bool vqmmc_enabled;
+};
+
+#define CMD_CFG_LENGTH_MASK GENMASK(8, 0)
#define CMD_CFG_BLOCK_MODE BIT(9)
#define CMD_CFG_R1B BIT(10)
#define CMD_CFG_END_OF_CHAIN BIT(11)
-#define CMD_CFG_TIMEOUT_SHIFT 12
-#define CMD_CFG_TIMEOUT_MASK 0xf
+#define CMD_CFG_TIMEOUT_MASK GENMASK(15, 12)
#define CMD_CFG_NO_RESP BIT(16)
#define CMD_CFG_NO_CMD BIT(17)
#define CMD_CFG_DATA_IO BIT(18)
@@ -166,17 +169,99 @@
#define CMD_CFG_RESP_128 BIT(21)
#define CMD_CFG_RESP_NUM BIT(22)
#define CMD_CFG_DATA_NUM BIT(23)
-#define CMD_CFG_CMD_INDEX_SHIFT 24
-#define CMD_CFG_CMD_INDEX_MASK 0x3f
+#define CMD_CFG_CMD_INDEX_MASK GENMASK(29, 24)
#define CMD_CFG_ERROR BIT(30)
#define CMD_CFG_OWNER BIT(31)
-#define CMD_DATA_MASK (~0x3)
+#define CMD_DATA_MASK GENMASK(31, 2)
#define CMD_DATA_BIG_ENDIAN BIT(1)
#define CMD_DATA_SRAM BIT(0)
-#define CMD_RESP_MASK (~0x1)
+#define CMD_RESP_MASK GENMASK(31, 1)
#define CMD_RESP_SRAM BIT(0)
+static unsigned int meson_mmc_get_timeout_msecs(struct mmc_data *data)
+{
+ unsigned int timeout = data->timeout_ns / NSEC_PER_MSEC;
+
+ if (!timeout)
+ return SD_EMMC_CMD_TIMEOUT_DATA;
+
+ timeout = roundup_pow_of_two(timeout);
+
+ return min(timeout, 32768U); /* max. 2^15 ms */
+}
+
+static struct mmc_command *meson_mmc_get_next_command(struct mmc_command *cmd)
+{
+ if (cmd->opcode == MMC_SET_BLOCK_COUNT && !cmd->error)
+ return cmd->mrq->cmd;
+ else if (mmc_op_multi(cmd->opcode) &&
+ (!cmd->mrq->sbc || cmd->error || cmd->data->error))
+ return cmd->mrq->stop;
+ else
+ return NULL;
+}
+
+static void meson_mmc_get_transfer_mode(struct mmc_host *mmc,
+ struct mmc_request *mrq)
+{
+ struct mmc_data *data = mrq->data;
+ struct scatterlist *sg;
+ int i;
+ bool use_desc_chain_mode = true;
+
+ for_each_sg(data->sg, sg, data->sg_len, i)
+ /* check for 8 byte alignment */
+ if (sg->offset & 7) {
+ WARN_ONCE(1, "unaligned scatterlist buffer\n");
+ use_desc_chain_mode = false;
+ break;
+ }
+
+ if (use_desc_chain_mode)
+ data->host_cookie |= SD_EMMC_DESC_CHAIN_MODE;
+}
+
+static inline bool meson_mmc_desc_chain_mode(const struct mmc_data *data)
+{
+ return data->host_cookie & SD_EMMC_DESC_CHAIN_MODE;
+}
+
+static inline bool meson_mmc_bounce_buf_read(const struct mmc_data *data)
+{
+ return data && data->flags & MMC_DATA_READ &&
+ !meson_mmc_desc_chain_mode(data);
+}
+
+static void meson_mmc_pre_req(struct mmc_host *mmc, struct mmc_request *mrq)
+{
+ struct mmc_data *data = mrq->data;
+
+ if (!data)
+ return;
+
+ meson_mmc_get_transfer_mode(mmc, mrq);
+ data->host_cookie |= SD_EMMC_PRE_REQ_DONE;
+
+ if (!meson_mmc_desc_chain_mode(data))
+ return;
+
+ data->sg_count = dma_map_sg(mmc_dev(mmc), data->sg, data->sg_len,
+ mmc_get_dma_dir(data));
+ if (!data->sg_count)
+ dev_err(mmc_dev(mmc), "dma_map_sg failed");
+}
+
+static void meson_mmc_post_req(struct mmc_host *mmc, struct mmc_request *mrq,
+ int err)
+{
+ struct mmc_data *data = mrq->data;
+
+ if (data && meson_mmc_desc_chain_mode(data) && data->sg_count)
+ dma_unmap_sg(mmc_dev(mmc), data->sg, data->sg_len,
+ mmc_get_dma_dir(data));
+}
+
static int meson_mmc_clk_set(struct meson_host *host, unsigned long clk_rate)
{
struct mmc_host *mmc = host->mmc;
@@ -244,26 +329,23 @@
char clk_name[32];
int i, ret = 0;
const char *mux_parent_names[MUX_CLK_NUM_PARENTS];
- unsigned int mux_parent_count = 0;
const char *clk_div_parents[1];
u32 clk_reg, cfg;
/* get the mux parents */
for (i = 0; i < MUX_CLK_NUM_PARENTS; i++) {
+ struct clk *clk;
char name[16];
snprintf(name, sizeof(name), "clkin%d", i);
- host->mux_parent[i] = devm_clk_get(host->dev, name);
- if (IS_ERR(host->mux_parent[i])) {
- ret = PTR_ERR(host->mux_parent[i]);
- if (PTR_ERR(host->mux_parent[i]) != -EPROBE_DEFER)
+ clk = devm_clk_get(host->dev, name);
+ if (IS_ERR(clk)) {
+ if (clk != ERR_PTR(-EPROBE_DEFER))
dev_err(host->dev, "Missing clock %s\n", name);
- host->mux_parent[i] = NULL;
- return ret;
+ return PTR_ERR(clk);
}
- mux_parent_names[i] = __clk_get_name(host->mux_parent[i]);
- mux_parent_count++;
+ mux_parent_names[i] = __clk_get_name(clk);
}
/* create the mux */
@@ -272,10 +354,9 @@
init.ops = &clk_mux_ops;
init.flags = 0;
init.parent_names = mux_parent_names;
- init.num_parents = mux_parent_count;
-
+ init.num_parents = MUX_CLK_NUM_PARENTS;
host->mux.reg = host->regs + SD_EMMC_CLOCK;
- host->mux.shift = CLK_SRC_SHIFT;
+ host->mux.shift = __bf_shf(CLK_SRC_MASK);
host->mux.mask = CLK_SRC_MASK;
host->mux.flags = 0;
host->mux.table = NULL;
@@ -287,7 +368,7 @@
/* create the divider */
snprintf(clk_name, sizeof(clk_name), "%s#div", dev_name(host->dev));
- init.name = devm_kstrdup(host->dev, clk_name, GFP_KERNEL);
+ init.name = clk_name;
init.ops = &clk_divider_ops;
init.flags = CLK_SET_RATE_PARENT;
clk_div_parents[0] = __clk_get_name(host->mux_clk);
@@ -295,8 +376,8 @@
init.num_parents = ARRAY_SIZE(clk_div_parents);
host->cfg_div.reg = host->regs + SD_EMMC_CLOCK;
- host->cfg_div.shift = CLK_DIV_SHIFT;
- host->cfg_div.width = CLK_DIV_WIDTH;
+ host->cfg_div.shift = __bf_shf(CLK_DIV_MASK);
+ host->cfg_div.width = __builtin_popcountl(CLK_DIV_MASK);
host->cfg_div.hw.init = &init;
host->cfg_div.flags = CLK_DIVIDER_ONE_BASED |
CLK_DIVIDER_ROUND_CLOSEST | CLK_DIVIDER_ALLOW_ZERO;
@@ -307,9 +388,11 @@
/* init SD_EMMC_CLOCK to sane defaults w/min clock rate */
clk_reg = 0;
- clk_reg |= CLK_PHASE_180 << CLK_PHASE_SHIFT;
- clk_reg |= CLK_SRC_XTAL << CLK_SRC_SHIFT;
- clk_reg |= CLK_DIV_MAX << CLK_DIV_SHIFT;
+ clk_reg |= FIELD_PREP(CLK_CORE_PHASE_MASK, host->tp.core_phase);
+ clk_reg |= FIELD_PREP(CLK_TX_PHASE_MASK, host->tp.tx_phase);
+ clk_reg |= FIELD_PREP(CLK_RX_PHASE_MASK, host->tp.rx_phase);
+ clk_reg |= FIELD_PREP(CLK_SRC_MASK, CLK_SRC_XTAL);
+ clk_reg |= FIELD_PREP(CLK_DIV_MASK, CLK_DIV_MAX);
clk_reg &= ~CLK_ALWAYS_ON;
writel(clk_reg, host->regs + SD_EMMC_CLOCK);
@@ -327,12 +410,37 @@
host->mmc->f_min = clk_round_rate(host->cfg_div_clk, 400000);
ret = meson_mmc_clk_set(host, host->mmc->f_min);
- if (!ret)
+ if (ret)
clk_disable_unprepare(host->cfg_div_clk);
return ret;
}
+static void meson_mmc_set_tuning_params(struct mmc_host *mmc)
+{
+ struct meson_host *host = mmc_priv(mmc);
+ u32 regval;
+
+ /* stop clock */
+ regval = readl(host->regs + SD_EMMC_CFG);
+ regval |= CFG_STOP_CLOCK;
+ writel(regval, host->regs + SD_EMMC_CFG);
+
+ regval = readl(host->regs + SD_EMMC_CLOCK);
+ regval &= ~CLK_CORE_PHASE_MASK;
+ regval |= FIELD_PREP(CLK_CORE_PHASE_MASK, host->tp.core_phase);
+ regval &= ~CLK_TX_PHASE_MASK;
+ regval |= FIELD_PREP(CLK_TX_PHASE_MASK, host->tp.tx_phase);
+ regval &= ~CLK_RX_PHASE_MASK;
+ regval |= FIELD_PREP(CLK_RX_PHASE_MASK, host->tp.rx_phase);
+ writel(regval, host->regs + SD_EMMC_CLOCK);
+
+ /* start clock */
+ regval = readl(host->regs + SD_EMMC_CFG);
+ regval &= ~CFG_STOP_CLOCK;
+ writel(regval, host->regs + SD_EMMC_CFG);
+}
+
static void meson_mmc_set_ios(struct mmc_host *mmc, struct mmc_ios *ios)
{
struct meson_host *host = mmc_priv(mmc);
@@ -397,17 +505,8 @@
val = readl(host->regs + SD_EMMC_CFG);
orig = val;
- val &= ~(CFG_BUS_WIDTH_MASK << CFG_BUS_WIDTH_SHIFT);
- val |= bus_width << CFG_BUS_WIDTH_SHIFT;
-
- val &= ~(CFG_BLK_LEN_MASK << CFG_BLK_LEN_SHIFT);
- val |= ilog2(SD_EMMC_CFG_BLK_SIZE) << CFG_BLK_LEN_SHIFT;
-
- val &= ~(CFG_RESP_TIMEOUT_MASK << CFG_RESP_TIMEOUT_SHIFT);
- val |= ilog2(SD_EMMC_CFG_RESP_TIMEOUT) << CFG_RESP_TIMEOUT_SHIFT;
-
- val &= ~(CFG_RC_CC_MASK << CFG_RC_CC_SHIFT);
- val |= ilog2(SD_EMMC_CFG_CMD_GAP) << CFG_RC_CC_SHIFT;
+ val &= ~CFG_BUS_WIDTH_MASK;
+ val |= FIELD_PREP(CFG_BUS_WIDTH_MASK, bus_width);
val &= ~CFG_DDR;
if (ios->timing == MMC_TIMING_UHS_DDR50 ||
@@ -419,149 +518,189 @@
if (ios->timing == MMC_TIMING_MMC_HS400)
val |= CFG_CHK_DS;
- writel(val, host->regs + SD_EMMC_CFG);
-
- if (val != orig)
+ if (val != orig) {
+ writel(val, host->regs + SD_EMMC_CFG);
dev_dbg(host->dev, "%s: SD_EMMC_CFG: 0x%08x -> 0x%08x\n",
__func__, orig, val);
+ }
}
-static int meson_mmc_request_done(struct mmc_host *mmc, struct mmc_request *mrq)
+static void meson_mmc_request_done(struct mmc_host *mmc,
+ struct mmc_request *mrq)
{
struct meson_host *host = mmc_priv(mmc);
- WARN_ON(host->mrq != mrq);
-
- host->mrq = NULL;
host->cmd = NULL;
mmc_request_done(host->mmc, mrq);
+}
- return 0;
+static void meson_mmc_set_blksz(struct mmc_host *mmc, unsigned int blksz)
+{
+ struct meson_host *host = mmc_priv(mmc);
+ u32 cfg, blksz_old;
+
+ cfg = readl(host->regs + SD_EMMC_CFG);
+ blksz_old = FIELD_GET(CFG_BLK_LEN_MASK, cfg);
+
+ if (!is_power_of_2(blksz))
+ dev_err(host->dev, "blksz %u is not a power of 2\n", blksz);
+
+ blksz = ilog2(blksz);
+
+ /* check if block-size matches, if not update */
+ if (blksz == blksz_old)
+ return;
+
+ dev_dbg(host->dev, "%s: update blk_len %d -> %d\n", __func__,
+ blksz_old, blksz);
+
+ cfg &= ~CFG_BLK_LEN_MASK;
+ cfg |= FIELD_PREP(CFG_BLK_LEN_MASK, blksz);
+ writel(cfg, host->regs + SD_EMMC_CFG);
+}
+
+static void meson_mmc_set_response_bits(struct mmc_command *cmd, u32 *cmd_cfg)
+{
+ if (cmd->flags & MMC_RSP_PRESENT) {
+ if (cmd->flags & MMC_RSP_136)
+ *cmd_cfg |= CMD_CFG_RESP_128;
+ *cmd_cfg |= CMD_CFG_RESP_NUM;
+
+ if (!(cmd->flags & MMC_RSP_CRC))
+ *cmd_cfg |= CMD_CFG_RESP_NOCRC;
+
+ if (cmd->flags & MMC_RSP_BUSY)
+ *cmd_cfg |= CMD_CFG_R1B;
+ } else {
+ *cmd_cfg |= CMD_CFG_NO_RESP;
+ }
+}
+
+static void meson_mmc_desc_chain_transfer(struct mmc_host *mmc, u32 cmd_cfg)
+{
+ struct meson_host *host = mmc_priv(mmc);
+ struct sd_emmc_desc *desc = host->descs;
+ struct mmc_data *data = host->cmd->data;
+ struct scatterlist *sg;
+ u32 start;
+ int i;
+
+ if (data->flags & MMC_DATA_WRITE)
+ cmd_cfg |= CMD_CFG_DATA_WR;
+
+ if (data->blocks > 1) {
+ cmd_cfg |= CMD_CFG_BLOCK_MODE;
+ meson_mmc_set_blksz(mmc, data->blksz);
+ }
+
+ for_each_sg(data->sg, sg, data->sg_count, i) {
+ unsigned int len = sg_dma_len(sg);
+
+ if (data->blocks > 1)
+ len /= data->blksz;
+
+ desc[i].cmd_cfg = cmd_cfg;
+ desc[i].cmd_cfg |= FIELD_PREP(CMD_CFG_LENGTH_MASK, len);
+ if (i > 0)
+ desc[i].cmd_cfg |= CMD_CFG_NO_CMD;
+ desc[i].cmd_arg = host->cmd->arg;
+ desc[i].cmd_resp = 0;
+ desc[i].cmd_data = sg_dma_address(sg);
+ }
+ desc[data->sg_count - 1].cmd_cfg |= CMD_CFG_END_OF_CHAIN;
+
+ dma_wmb(); /* ensure descriptor is written before kicked */
+ start = host->descs_dma_addr | START_DESC_BUSY;
+ writel(start, host->regs + SD_EMMC_START);
}
static void meson_mmc_start_cmd(struct mmc_host *mmc, struct mmc_command *cmd)
{
struct meson_host *host = mmc_priv(mmc);
- struct sd_emmc_desc *desc, desc_tmp;
- u32 cfg;
- u8 blk_len, cmd_cfg_timeout;
+ struct mmc_data *data = cmd->data;
+ u32 cmd_cfg = 0, cmd_data = 0;
unsigned int xfer_bytes = 0;
/* Setup descriptors */
dma_rmb();
- desc = &desc_tmp;
- memset(desc, 0, sizeof(struct sd_emmc_desc));
-
- desc->cmd_cfg |= (cmd->opcode & CMD_CFG_CMD_INDEX_MASK) <<
- CMD_CFG_CMD_INDEX_SHIFT;
- desc->cmd_cfg |= CMD_CFG_OWNER; /* owned by CPU */
- desc->cmd_arg = cmd->arg;
-
- /* Response */
- if (cmd->flags & MMC_RSP_PRESENT) {
- desc->cmd_cfg &= ~CMD_CFG_NO_RESP;
- if (cmd->flags & MMC_RSP_136)
- desc->cmd_cfg |= CMD_CFG_RESP_128;
- desc->cmd_cfg |= CMD_CFG_RESP_NUM;
- desc->cmd_resp = 0;
-
- if (!(cmd->flags & MMC_RSP_CRC))
- desc->cmd_cfg |= CMD_CFG_RESP_NOCRC;
-
- if (cmd->flags & MMC_RSP_BUSY)
- desc->cmd_cfg |= CMD_CFG_R1B;
- } else {
- desc->cmd_cfg |= CMD_CFG_NO_RESP;
- }
-
- /* data? */
- if (cmd->data) {
- desc->cmd_cfg |= CMD_CFG_DATA_IO;
- if (cmd->data->blocks > 1) {
- desc->cmd_cfg |= CMD_CFG_BLOCK_MODE;
- desc->cmd_cfg |=
- (cmd->data->blocks & CMD_CFG_LENGTH_MASK) <<
- CMD_CFG_LENGTH_SHIFT;
-
- /* check if block-size matches, if not update */
- cfg = readl(host->regs + SD_EMMC_CFG);
- blk_len = cfg & (CFG_BLK_LEN_MASK << CFG_BLK_LEN_SHIFT);
- blk_len >>= CFG_BLK_LEN_SHIFT;
- if (blk_len != ilog2(cmd->data->blksz)) {
- dev_dbg(host->dev, "%s: update blk_len %d -> %d\n",
- __func__, blk_len,
- ilog2(cmd->data->blksz));
- blk_len = ilog2(cmd->data->blksz);
- cfg &= ~(CFG_BLK_LEN_MASK << CFG_BLK_LEN_SHIFT);
- cfg |= blk_len << CFG_BLK_LEN_SHIFT;
- writel(cfg, host->regs + SD_EMMC_CFG);
- }
- } else {
- desc->cmd_cfg &= ~CMD_CFG_BLOCK_MODE;
- desc->cmd_cfg |=
- (cmd->data->blksz & CMD_CFG_LENGTH_MASK) <<
- CMD_CFG_LENGTH_SHIFT;
- }
-
- cmd->data->bytes_xfered = 0;
- xfer_bytes = cmd->data->blksz * cmd->data->blocks;
- if (cmd->data->flags & MMC_DATA_WRITE) {
- desc->cmd_cfg |= CMD_CFG_DATA_WR;
- WARN_ON(xfer_bytes > host->bounce_buf_size);
- sg_copy_to_buffer(cmd->data->sg, cmd->data->sg_len,
- host->bounce_buf, xfer_bytes);
- cmd->data->bytes_xfered = xfer_bytes;
- dma_wmb();
- } else {
- desc->cmd_cfg &= ~CMD_CFG_DATA_WR;
- }
-
- if (xfer_bytes > 0) {
- desc->cmd_cfg &= ~CMD_CFG_DATA_NUM;
- desc->cmd_data = host->bounce_dma_addr & CMD_DATA_MASK;
- } else {
- /* write data to data_addr */
- desc->cmd_cfg |= CMD_CFG_DATA_NUM;
- desc->cmd_data = 0;
- }
-
- cmd_cfg_timeout = 12;
- } else {
- desc->cmd_cfg &= ~CMD_CFG_DATA_IO;
- cmd_cfg_timeout = 10;
- }
- desc->cmd_cfg |= (cmd_cfg_timeout & CMD_CFG_TIMEOUT_MASK) <<
- CMD_CFG_TIMEOUT_SHIFT;
host->cmd = cmd;
+ cmd_cfg |= FIELD_PREP(CMD_CFG_CMD_INDEX_MASK, cmd->opcode);
+ cmd_cfg |= CMD_CFG_OWNER; /* owned by CPU */
+
+ meson_mmc_set_response_bits(cmd, &cmd_cfg);
+
+ /* data? */
+ if (data) {
+ data->bytes_xfered = 0;
+ cmd_cfg |= CMD_CFG_DATA_IO;
+ cmd_cfg |= FIELD_PREP(CMD_CFG_TIMEOUT_MASK,
+ ilog2(meson_mmc_get_timeout_msecs(data)));
+
+ if (meson_mmc_desc_chain_mode(data)) {
+ meson_mmc_desc_chain_transfer(mmc, cmd_cfg);
+ return;
+ }
+
+ if (data->blocks > 1) {
+ cmd_cfg |= CMD_CFG_BLOCK_MODE;
+ cmd_cfg |= FIELD_PREP(CMD_CFG_LENGTH_MASK,
+ data->blocks);
+ meson_mmc_set_blksz(mmc, data->blksz);
+ } else {
+ cmd_cfg |= FIELD_PREP(CMD_CFG_LENGTH_MASK, data->blksz);
+ }
+
+ xfer_bytes = data->blksz * data->blocks;
+ if (data->flags & MMC_DATA_WRITE) {
+ cmd_cfg |= CMD_CFG_DATA_WR;
+ WARN_ON(xfer_bytes > host->bounce_buf_size);
+ sg_copy_to_buffer(data->sg, data->sg_len,
+ host->bounce_buf, xfer_bytes);
+ dma_wmb();
+ }
+
+ cmd_data = host->bounce_dma_addr & CMD_DATA_MASK;
+ } else {
+ cmd_cfg |= FIELD_PREP(CMD_CFG_TIMEOUT_MASK,
+ ilog2(SD_EMMC_CMD_TIMEOUT));
+ }
+
/* Last descriptor */
- desc->cmd_cfg |= CMD_CFG_END_OF_CHAIN;
- writel(desc->cmd_cfg, host->regs + SD_EMMC_CMD_CFG);
- writel(desc->cmd_data, host->regs + SD_EMMC_CMD_DAT);
- writel(desc->cmd_resp, host->regs + SD_EMMC_CMD_RSP);
+ cmd_cfg |= CMD_CFG_END_OF_CHAIN;
+ writel(cmd_cfg, host->regs + SD_EMMC_CMD_CFG);
+ writel(cmd_data, host->regs + SD_EMMC_CMD_DAT);
+ writel(0, host->regs + SD_EMMC_CMD_RSP);
wmb(); /* ensure descriptor is written before kicked */
- writel(desc->cmd_arg, host->regs + SD_EMMC_CMD_ARG);
+ writel(cmd->arg, host->regs + SD_EMMC_CMD_ARG);
}
static void meson_mmc_request(struct mmc_host *mmc, struct mmc_request *mrq)
{
struct meson_host *host = mmc_priv(mmc);
+ bool needs_pre_post_req = mrq->data &&
+ !(mrq->data->host_cookie & SD_EMMC_PRE_REQ_DONE);
- WARN_ON(host->mrq != NULL);
+ if (needs_pre_post_req) {
+ meson_mmc_get_transfer_mode(mmc, mrq);
+ if (!meson_mmc_desc_chain_mode(mrq->data))
+ needs_pre_post_req = false;
+ }
+
+ if (needs_pre_post_req)
+ meson_mmc_pre_req(mmc, mrq);
/* Stop execution */
writel(0, host->regs + SD_EMMC_START);
- host->mrq = mrq;
+ meson_mmc_start_cmd(mmc, mrq->sbc ?: mrq->cmd);
- if (mrq->sbc)
- meson_mmc_start_cmd(mmc, mrq->sbc);
- else
- meson_mmc_start_cmd(mmc, mrq->cmd);
+ if (needs_pre_post_req)
+ meson_mmc_post_req(mmc, mrq, 0);
}
-static int meson_mmc_read_resp(struct mmc_host *mmc, struct mmc_command *cmd)
+static void meson_mmc_read_resp(struct mmc_host *mmc, struct mmc_command *cmd)
{
struct meson_host *host = mmc_priv(mmc);
@@ -573,15 +712,13 @@
} else if (cmd->flags & MMC_RSP_PRESENT) {
cmd->resp[0] = readl(host->regs + SD_EMMC_CMD_RSP);
}
-
- return 0;
}
static irqreturn_t meson_mmc_irq(int irq, void *dev_id)
{
struct meson_host *host = dev_id;
- struct mmc_request *mrq;
struct mmc_command *cmd;
+ struct mmc_data *data;
u32 irq_en, status, raw_status;
irqreturn_t ret = IRQ_HANDLED;
@@ -590,14 +727,11 @@
cmd = host->cmd;
- mrq = host->mrq;
-
- if (WARN_ON(!mrq))
- return IRQ_NONE;
-
if (WARN_ON(!cmd))
return IRQ_NONE;
+ data = cmd->data;
+
spin_lock(&host->lock);
irq_en = readl(host->regs + SD_EMMC_IRQ_EN);
raw_status = readl(host->regs + SD_EMMC_STATUS);
@@ -610,6 +744,8 @@
goto out;
}
+ meson_mmc_read_resp(host->mmc, cmd);
+
cmd->error = 0;
if (status & IRQ_RXD_ERR_MASK) {
dev_dbg(host->dev, "Unhandled IRQ: RXD error\n");
@@ -636,12 +772,16 @@
if (status & IRQ_SDIO)
dev_dbg(host->dev, "Unhandled IRQ: SDIO.\n");
- if (status & (IRQ_END_OF_CHAIN | IRQ_RESP_STATUS))
- ret = IRQ_WAKE_THREAD;
- else {
+ if (status & (IRQ_END_OF_CHAIN | IRQ_RESP_STATUS)) {
+ if (data && !cmd->error)
+ data->bytes_xfered = data->blksz * data->blocks;
+ if (meson_mmc_bounce_buf_read(data) ||
+ meson_mmc_get_next_command(cmd))
+ ret = IRQ_WAKE_THREAD;
+ } else {
dev_warn(host->dev, "Unknown IRQ! status=0x%04x: MMC CMD%u arg=0x%08x flags=0x%08x stop=%d\n",
status, cmd->opcode, cmd->arg,
- cmd->flags, mrq->stop ? 1 : 0);
+ cmd->flags, cmd->mrq->stop ? 1 : 0);
if (cmd->data) {
struct mmc_data *data = cmd->data;
@@ -656,10 +796,8 @@
/* ack all (enabled) interrupts */
writel(status, host->regs + SD_EMMC_STATUS);
- if (ret == IRQ_HANDLED) {
- meson_mmc_read_resp(host->mmc, cmd);
+ if (ret == IRQ_HANDLED)
meson_mmc_request_done(host->mmc, cmd->mrq);
- }
spin_unlock(&host->lock);
return ret;
@@ -668,35 +806,53 @@
static irqreturn_t meson_mmc_irq_thread(int irq, void *dev_id)
{
struct meson_host *host = dev_id;
- struct mmc_request *mrq = host->mrq;
- struct mmc_command *cmd = host->cmd;
+ struct mmc_command *next_cmd, *cmd = host->cmd;
struct mmc_data *data;
unsigned int xfer_bytes;
- if (WARN_ON(!mrq))
- return IRQ_NONE;
-
if (WARN_ON(!cmd))
return IRQ_NONE;
data = cmd->data;
- if (data && data->flags & MMC_DATA_READ) {
+ if (meson_mmc_bounce_buf_read(data)) {
xfer_bytes = data->blksz * data->blocks;
WARN_ON(xfer_bytes > host->bounce_buf_size);
sg_copy_from_buffer(data->sg, data->sg_len,
host->bounce_buf, xfer_bytes);
- data->bytes_xfered = xfer_bytes;
}
- meson_mmc_read_resp(host->mmc, cmd);
- if (!data || !data->stop || mrq->sbc)
- meson_mmc_request_done(host->mmc, mrq);
+ next_cmd = meson_mmc_get_next_command(cmd);
+ if (next_cmd)
+ meson_mmc_start_cmd(host->mmc, next_cmd);
else
- meson_mmc_start_cmd(host->mmc, data->stop);
+ meson_mmc_request_done(host->mmc, cmd->mrq);
return IRQ_HANDLED;
}
+static int meson_mmc_execute_tuning(struct mmc_host *mmc, u32 opcode)
+{
+ struct meson_host *host = mmc_priv(mmc);
+ struct meson_tuning_params tp_old = host->tp;
+ int ret = -EINVAL, i, cmd_error;
+
+ dev_info(mmc_dev(mmc), "(re)tuning...\n");
+
+ for (i = CLK_PHASE_0; i <= CLK_PHASE_270; i++) {
+ host->tp.rx_phase = i;
+ /* exclude the active parameter set if retuning */
+ if (!memcmp(&tp_old, &host->tp, sizeof(tp_old)) &&
+ mmc->doing_retune)
+ continue;
+ meson_mmc_set_tuning_params(mmc);
+ ret = mmc_send_tuning(mmc, opcode, &cmd_error);
+ if (!ret)
+ break;
+ }
+
+ return ret;
+}
+
/*
* NOTE: we only need this until the GPIO/pinctrl driver can handle
* interrupts. For now, the MMC core will use this for polling.
@@ -711,10 +867,25 @@
return status;
}
+static void meson_mmc_cfg_init(struct meson_host *host)
+{
+ u32 cfg = 0;
+
+ cfg |= FIELD_PREP(CFG_RESP_TIMEOUT_MASK,
+ ilog2(SD_EMMC_CFG_RESP_TIMEOUT));
+ cfg |= FIELD_PREP(CFG_RC_CC_MASK, ilog2(SD_EMMC_CFG_CMD_GAP));
+ cfg |= FIELD_PREP(CFG_BLK_LEN_MASK, ilog2(SD_EMMC_CFG_BLK_SIZE));
+
+ writel(cfg, host->regs + SD_EMMC_CFG);
+}
+
static const struct mmc_host_ops meson_mmc_ops = {
.request = meson_mmc_request,
.set_ios = meson_mmc_set_ios,
.get_cd = meson_mmc_get_cd,
+ .pre_req = meson_mmc_pre_req,
+ .post_req = meson_mmc_post_req,
+ .execute_tuning = meson_mmc_execute_tuning,
};
static int meson_mmc_probe(struct platform_device *pdev)
@@ -722,7 +893,7 @@
struct resource *res;
struct meson_host *host;
struct mmc_host *mmc;
- int ret;
+ int ret, irq;
mmc = mmc_alloc_host(sizeof(struct meson_host), &pdev->dev);
if (!mmc)
@@ -754,8 +925,8 @@
goto free_host;
}
- host->irq = platform_get_irq(pdev, 0);
- if (host->irq == 0) {
+ irq = platform_get_irq(pdev, 0);
+ if (!irq) {
dev_err(&pdev->dev, "failed to get interrupt resource.\n");
ret = -EINVAL;
goto free_host;
@@ -771,9 +942,13 @@
if (ret)
goto free_host;
+ host->tp.core_phase = CLK_PHASE_180;
+ host->tp.tx_phase = CLK_PHASE_0;
+ host->tp.rx_phase = CLK_PHASE_0;
+
ret = meson_mmc_clk_init(host);
if (ret)
- goto free_host;
+ goto err_core_clk;
/* Stop execution */
writel(0, host->regs + SD_EMMC_START);
@@ -783,14 +958,20 @@
writel(IRQ_EN_MASK, host->regs + SD_EMMC_STATUS);
writel(IRQ_EN_MASK, host->regs + SD_EMMC_IRQ_EN);
- ret = devm_request_threaded_irq(&pdev->dev, host->irq,
- meson_mmc_irq, meson_mmc_irq_thread,
- IRQF_SHARED, DRIVER_NAME, host);
- if (ret)
- goto free_host;
+ /* set config to sane default */
+ meson_mmc_cfg_init(host);
+ ret = devm_request_threaded_irq(&pdev->dev, irq, meson_mmc_irq,
+ meson_mmc_irq_thread, IRQF_SHARED,
+ NULL, host);
+ if (ret)
+ goto err_div_clk;
+
+ mmc->caps |= MMC_CAP_CMD23;
mmc->max_blk_count = CMD_CFG_LENGTH_MASK;
mmc->max_req_size = mmc->max_blk_count * mmc->max_blk_size;
+ mmc->max_segs = SD_EMMC_DESC_BUF_LEN / sizeof(struct sd_emmc_desc);
+ mmc->max_seg_size = mmc->max_req_size;
/* data bounce buffer */
host->bounce_buf_size = mmc->max_req_size;
@@ -800,7 +981,15 @@
if (host->bounce_buf == NULL) {
dev_err(host->dev, "Unable to map allocate DMA bounce buffer.\n");
ret = -ENOMEM;
- goto free_host;
+ goto err_div_clk;
+ }
+
+ host->descs = dma_alloc_coherent(host->dev, SD_EMMC_DESC_BUF_LEN,
+ &host->descs_dma_addr, GFP_KERNEL);
+ if (!host->descs) {
+ dev_err(host->dev, "Allocating descriptor DMA buffer failed\n");
+ ret = -ENOMEM;
+ goto err_bounce_buf;
}
mmc->ops = &meson_mmc_ops;
@@ -808,9 +997,14 @@
return 0;
-free_host:
+err_bounce_buf:
+ dma_free_coherent(host->dev, host->bounce_buf_size,
+ host->bounce_buf, host->bounce_dma_addr);
+err_div_clk:
clk_disable_unprepare(host->cfg_div_clk);
+err_core_clk:
clk_disable_unprepare(host->core_clk);
+free_host:
mmc_free_host(mmc);
return ret;
}
@@ -819,9 +1013,13 @@
{
struct meson_host *host = dev_get_drvdata(&pdev->dev);
+ mmc_remove_host(host->mmc);
+
/* disable interrupts */
writel(0, host->regs + SD_EMMC_IRQ_EN);
+ dma_free_coherent(host->dev, SD_EMMC_DESC_BUF_LEN,
+ host->descs, host->descs_dma_addr);
dma_free_coherent(host->dev, host->bounce_buf_size,
host->bounce_buf, host->bounce_dma_addr);
diff --git a/drivers/mmc/host/mmc_spi.c b/drivers/mmc/host/mmc_spi.c
index e77d79c..476e53d 100644
--- a/drivers/mmc/host/mmc_spi.c
+++ b/drivers/mmc/host/mmc_spi.c
@@ -888,10 +888,7 @@
u32 clock_rate;
unsigned long timeout;
- if (data->flags & MMC_DATA_READ)
- direction = DMA_FROM_DEVICE;
- else
- direction = DMA_TO_DEVICE;
+ direction = mmc_get_dma_dir(data);
mmc_spi_setup_data_message(host, multiple, direction);
t = &host->t;
diff --git a/drivers/mmc/host/mmci.c b/drivers/mmc/host/mmci.c
index 0c6420b..d1ca2f4 100644
--- a/drivers/mmc/host/mmci.c
+++ b/drivers/mmc/host/mmci.c
@@ -516,17 +516,14 @@
static void mmci_dma_unmap(struct mmci_host *host, struct mmc_data *data)
{
struct dma_chan *chan;
- enum dma_data_direction dir;
- if (data->flags & MMC_DATA_READ) {
- dir = DMA_FROM_DEVICE;
+ if (data->flags & MMC_DATA_READ)
chan = host->dma_rx_channel;
- } else {
- dir = DMA_TO_DEVICE;
+ else
chan = host->dma_tx_channel;
- }
- dma_unmap_sg(chan->device->dev, data->sg, data->sg_len, dir);
+ dma_unmap_sg(chan->device->dev, data->sg, data->sg_len,
+ mmc_get_dma_dir(data));
}
static void mmci_dma_finalize(struct mmci_host *host, struct mmc_data *data)
@@ -589,17 +586,14 @@
struct dma_chan *chan;
struct dma_device *device;
struct dma_async_tx_descriptor *desc;
- enum dma_data_direction buffer_dirn;
int nr_sg;
unsigned long flags = DMA_CTRL_ACK;
if (data->flags & MMC_DATA_READ) {
conf.direction = DMA_DEV_TO_MEM;
- buffer_dirn = DMA_FROM_DEVICE;
chan = host->dma_rx_channel;
} else {
conf.direction = DMA_MEM_TO_DEV;
- buffer_dirn = DMA_TO_DEVICE;
chan = host->dma_tx_channel;
}
@@ -612,7 +606,8 @@
return -EINVAL;
device = chan->device;
- nr_sg = dma_map_sg(device->dev, data->sg, data->sg_len, buffer_dirn);
+ nr_sg = dma_map_sg(device->dev, data->sg, data->sg_len,
+ mmc_get_dma_dir(data));
if (nr_sg == 0)
return -EINVAL;
@@ -631,7 +626,8 @@
return 0;
unmap_exit:
- dma_unmap_sg(device->dev, data->sg, data->sg_len, buffer_dirn);
+ dma_unmap_sg(device->dev, data->sg, data->sg_len,
+ mmc_get_dma_dir(data));
return -ENOMEM;
}
diff --git a/drivers/mmc/host/moxart-mmc.c b/drivers/mmc/host/moxart-mmc.c
index bbad309..d4dc55a 100644
--- a/drivers/mmc/host/moxart-mmc.c
+++ b/drivers/mmc/host/moxart-mmc.c
@@ -256,7 +256,7 @@
static void moxart_transfer_dma(struct mmc_data *data, struct moxart_host *host)
{
- u32 len, dir_data, dir_slave;
+ u32 len, dir_slave;
long dma_time;
struct dma_async_tx_descriptor *desc = NULL;
struct dma_chan *dma_chan;
@@ -266,16 +266,14 @@
if (data->flags & MMC_DATA_WRITE) {
dma_chan = host->dma_chan_tx;
- dir_data = DMA_TO_DEVICE;
dir_slave = DMA_MEM_TO_DEV;
} else {
dma_chan = host->dma_chan_rx;
- dir_data = DMA_FROM_DEVICE;
dir_slave = DMA_DEV_TO_MEM;
}
len = dma_map_sg(dma_chan->device->dev, data->sg,
- data->sg_len, dir_data);
+ data->sg_len, mmc_get_dma_dir(data));
if (len > 0) {
desc = dmaengine_prep_slave_sg(dma_chan, data->sg,
@@ -301,7 +299,7 @@
dma_unmap_sg(dma_chan->device->dev,
data->sg, data->sg_len,
- dir_data);
+ mmc_get_dma_dir(data));
}
diff --git a/drivers/mmc/host/mtk-sd.c b/drivers/mmc/host/mtk-sd.c
index b235d8d..5c1e178 100644
--- a/drivers/mmc/host/mtk-sd.c
+++ b/drivers/mmc/host/mtk-sd.c
@@ -76,6 +76,7 @@
#define MSDC_PATCH_BIT1 0xb4
#define MSDC_PAD_TUNE 0xec
#define PAD_DS_TUNE 0x188
+#define PAD_CMD_TUNE 0x18c
#define EMMC50_CFG0 0x208
/*--------------------------------------------------------------------------*/
@@ -211,13 +212,18 @@
#define MSDC_PATCH_BIT_SPCPUSH (0x1 << 29) /* RW */
#define MSDC_PATCH_BIT_DECRCTMO (0x1 << 30) /* RW */
+#define MSDC_PAD_TUNE_DATWRDLY (0x1f << 0) /* RW */
#define MSDC_PAD_TUNE_DATRRDLY (0x1f << 8) /* RW */
#define MSDC_PAD_TUNE_CMDRDLY (0x1f << 16) /* RW */
+#define MSDC_PAD_TUNE_CMDRRDLY (0x1f << 22) /* RW */
+#define MSDC_PAD_TUNE_CLKTDLY (0x1f << 27) /* RW */
#define PAD_DS_TUNE_DLY1 (0x1f << 2) /* RW */
#define PAD_DS_TUNE_DLY2 (0x1f << 7) /* RW */
#define PAD_DS_TUNE_DLY3 (0x1f << 12) /* RW */
+#define PAD_CMD_TUNE_RX_DLY3 (0x1f << 1) /* RW */
+
#define EMMC50_CFG_PADCMD_LATCHCK (0x1 << 0) /* RW */
#define EMMC50_CFG_CRCSTS_EDGE (0x1 << 3) /* RW */
#define EMMC50_CFG_CFCSTS_SEL (0x1 << 4) /* RW */
@@ -285,12 +291,14 @@
u32 patch_bit0;
u32 patch_bit1;
u32 pad_ds_tune;
+ u32 pad_cmd_tune;
u32 emmc50_cfg0;
};
struct msdc_tune_para {
u32 iocon;
u32 pad_tune;
+ u32 pad_cmd_tune;
};
struct msdc_delay_phase {
@@ -332,6 +340,10 @@
unsigned char timing;
bool vqmmc_enabled;
u32 hs400_ds_delay;
+ u32 hs200_cmd_int_delay; /* cmd internal delay for HS200/SDR104 */
+ u32 hs400_cmd_int_delay; /* cmd internal delay for HS400 */
+ bool hs400_cmd_resp_sel_rising;
+ /* cmd response sample selection for HS400 */
bool hs400_mode; /* current eMMC will run at hs400 mode */
struct msdc_save_para save_para; /* used when gate HCLK */
struct msdc_tune_para def_tune_para; /* default tune setting */
@@ -462,11 +474,9 @@
struct mmc_data *data = mrq->data;
if (!(data->host_cookie & MSDC_PREPARE_FLAG)) {
- bool read = (data->flags & MMC_DATA_READ) != 0;
-
data->host_cookie |= MSDC_PREPARE_FLAG;
data->sg_count = dma_map_sg(host->dev, data->sg, data->sg_len,
- read ? DMA_FROM_DEVICE : DMA_TO_DEVICE);
+ mmc_get_dma_dir(data));
}
}
@@ -478,10 +488,8 @@
return;
if (data->host_cookie & MSDC_PREPARE_FLAG) {
- bool read = (data->flags & MMC_DATA_READ) != 0;
-
dma_unmap_sg(host->dev, data->sg, data->sg_len,
- read ? DMA_FROM_DEVICE : DMA_TO_DEVICE);
+ mmc_get_dma_dir(data));
data->host_cookie &= ~MSDC_PREPARE_FLAG;
}
}
@@ -601,8 +609,14 @@
} else {
writel(host->saved_tune_para.iocon, host->base + MSDC_IOCON);
writel(host->saved_tune_para.pad_tune, host->base + MSDC_PAD_TUNE);
+ writel(host->saved_tune_para.pad_cmd_tune,
+ host->base + PAD_CMD_TUNE);
}
+ if (timing == MMC_TIMING_MMC_HS400)
+ sdr_set_field(host->base + PAD_CMD_TUNE,
+ MSDC_PAD_TUNE_CMDRRDLY,
+ host->hs400_cmd_int_delay);
dev_dbg(host->dev, "sclk: %d, timing: %d\n", host->sclk, timing);
}
@@ -1303,7 +1317,7 @@
len_final = len;
}
start += len ? len : 1;
- if (len >= 8 && start_final < 4)
+ if (len >= 12 && start_final < 4)
break;
}
@@ -1326,36 +1340,67 @@
struct msdc_host *host = mmc_priv(mmc);
u32 rise_delay = 0, fall_delay = 0;
struct msdc_delay_phase final_rise_delay, final_fall_delay = { 0,};
+ struct msdc_delay_phase internal_delay_phase;
u8 final_delay, final_maxlen;
+ u32 internal_delay = 0;
int cmd_err;
- int i;
+ int i, j;
+
+ if (mmc->ios.timing == MMC_TIMING_MMC_HS200 ||
+ mmc->ios.timing == MMC_TIMING_UHS_SDR104)
+ sdr_set_field(host->base + MSDC_PAD_TUNE,
+ MSDC_PAD_TUNE_CMDRRDLY,
+ host->hs200_cmd_int_delay);
sdr_clr_bits(host->base + MSDC_IOCON, MSDC_IOCON_RSPL);
for (i = 0 ; i < PAD_DELAY_MAX; i++) {
sdr_set_field(host->base + MSDC_PAD_TUNE,
MSDC_PAD_TUNE_CMDRDLY, i);
- mmc_send_tuning(mmc, opcode, &cmd_err);
- if (!cmd_err)
- rise_delay |= (1 << i);
+ /*
+ * Using the same parameters, it may sometimes pass the test,
+ * but sometimes it may fail. To make sure the parameters are
+ * more stable, we test each set of parameters 3 times.
+ */
+ for (j = 0; j < 3; j++) {
+ mmc_send_tuning(mmc, opcode, &cmd_err);
+ if (!cmd_err) {
+ rise_delay |= (1 << i);
+ } else {
+ rise_delay &= ~(1 << i);
+ break;
+ }
+ }
}
final_rise_delay = get_best_delay(host, rise_delay);
/* if rising edge has enough margin, then do not scan falling edge */
- if (final_rise_delay.maxlen >= 10 ||
- (final_rise_delay.start == 0 && final_rise_delay.maxlen >= 4))
+ if (final_rise_delay.maxlen >= 12 && final_rise_delay.start < 4)
goto skip_fall;
sdr_set_bits(host->base + MSDC_IOCON, MSDC_IOCON_RSPL);
for (i = 0; i < PAD_DELAY_MAX; i++) {
sdr_set_field(host->base + MSDC_PAD_TUNE,
MSDC_PAD_TUNE_CMDRDLY, i);
- mmc_send_tuning(mmc, opcode, &cmd_err);
- if (!cmd_err)
- fall_delay |= (1 << i);
+ /*
+ * Using the same parameters, it may sometimes pass the test,
+ * but sometimes it may fail. To make sure the parameters are
+ * more stable, we test each set of parameters 3 times.
+ */
+ for (j = 0; j < 3; j++) {
+ mmc_send_tuning(mmc, opcode, &cmd_err);
+ if (!cmd_err) {
+ fall_delay |= (1 << i);
+ } else {
+ fall_delay &= ~(1 << i);
+ break;
+ }
+ }
}
final_fall_delay = get_best_delay(host, fall_delay);
skip_fall:
final_maxlen = max(final_rise_delay.maxlen, final_fall_delay.maxlen);
+ if (final_fall_delay.maxlen >= 12 && final_fall_delay.start < 4)
+ final_maxlen = final_fall_delay.maxlen;
if (final_maxlen == final_rise_delay.maxlen) {
sdr_clr_bits(host->base + MSDC_IOCON, MSDC_IOCON_RSPL);
sdr_set_field(host->base + MSDC_PAD_TUNE, MSDC_PAD_TUNE_CMDRDLY,
@@ -1367,7 +1412,71 @@
final_fall_delay.final_phase);
final_delay = final_fall_delay.final_phase;
}
+ if (host->hs200_cmd_int_delay)
+ goto skip_internal;
+ for (i = 0; i < PAD_DELAY_MAX; i++) {
+ sdr_set_field(host->base + MSDC_PAD_TUNE,
+ MSDC_PAD_TUNE_CMDRRDLY, i);
+ mmc_send_tuning(mmc, opcode, &cmd_err);
+ if (!cmd_err)
+ internal_delay |= (1 << i);
+ }
+ dev_dbg(host->dev, "Final internal delay: 0x%x\n", internal_delay);
+ internal_delay_phase = get_best_delay(host, internal_delay);
+ sdr_set_field(host->base + MSDC_PAD_TUNE, MSDC_PAD_TUNE_CMDRRDLY,
+ internal_delay_phase.final_phase);
+skip_internal:
+ dev_dbg(host->dev, "Final cmd pad delay: %x\n", final_delay);
+ return final_delay == 0xff ? -EIO : 0;
+}
+
+static int hs400_tune_response(struct mmc_host *mmc, u32 opcode)
+{
+ struct msdc_host *host = mmc_priv(mmc);
+ u32 cmd_delay = 0;
+ struct msdc_delay_phase final_cmd_delay = { 0,};
+ u8 final_delay;
+ int cmd_err;
+ int i, j;
+
+ /* select EMMC50 PAD CMD tune */
+ sdr_set_bits(host->base + PAD_CMD_TUNE, BIT(0));
+
+ if (mmc->ios.timing == MMC_TIMING_MMC_HS200 ||
+ mmc->ios.timing == MMC_TIMING_UHS_SDR104)
+ sdr_set_field(host->base + MSDC_PAD_TUNE,
+ MSDC_PAD_TUNE_CMDRRDLY,
+ host->hs200_cmd_int_delay);
+
+ if (host->hs400_cmd_resp_sel_rising)
+ sdr_clr_bits(host->base + MSDC_IOCON, MSDC_IOCON_RSPL);
+ else
+ sdr_set_bits(host->base + MSDC_IOCON, MSDC_IOCON_RSPL);
+ for (i = 0 ; i < PAD_DELAY_MAX; i++) {
+ sdr_set_field(host->base + PAD_CMD_TUNE,
+ PAD_CMD_TUNE_RX_DLY3, i);
+ /*
+ * Using the same parameters, it may sometimes pass the test,
+ * but sometimes it may fail. To make sure the parameters are
+ * more stable, we test each set of parameters 3 times.
+ */
+ for (j = 0; j < 3; j++) {
+ mmc_send_tuning(mmc, opcode, &cmd_err);
+ if (!cmd_err) {
+ cmd_delay |= (1 << i);
+ } else {
+ cmd_delay &= ~(1 << i);
+ break;
+ }
+ }
+ }
+ final_cmd_delay = get_best_delay(host, cmd_delay);
+ sdr_set_field(host->base + PAD_CMD_TUNE, PAD_CMD_TUNE_RX_DLY3,
+ final_cmd_delay.final_phase);
+ final_delay = final_cmd_delay.final_phase;
+
+ dev_dbg(host->dev, "Final cmd pad delay: %x\n", final_delay);
return final_delay == 0xff ? -EIO : 0;
}
@@ -1390,7 +1499,7 @@
}
final_rise_delay = get_best_delay(host, rise_delay);
/* if rising edge has enough margin, then do not scan falling edge */
- if (final_rise_delay.maxlen >= 10 ||
+ if (final_rise_delay.maxlen >= 12 ||
(final_rise_delay.start == 0 && final_rise_delay.maxlen >= 4))
goto skip_fall;
@@ -1423,6 +1532,7 @@
final_delay = final_fall_delay.final_phase;
}
+ dev_dbg(host->dev, "Final data pad delay: %x\n", final_delay);
return final_delay == 0xff ? -EIO : 0;
}
@@ -1431,7 +1541,10 @@
struct msdc_host *host = mmc_priv(mmc);
int ret;
- ret = msdc_tune_response(mmc, opcode);
+ if (host->hs400_mode)
+ ret = hs400_tune_response(mmc, opcode);
+ else
+ ret = msdc_tune_response(mmc, opcode);
if (ret == -EIO) {
dev_err(host->dev, "Tune response fail!\n");
return ret;
@@ -1444,6 +1557,7 @@
host->saved_tune_para.iocon = readl(host->base + MSDC_IOCON);
host->saved_tune_para.pad_tune = readl(host->base + MSDC_PAD_TUNE);
+ host->saved_tune_para.pad_cmd_tune = readl(host->base + PAD_CMD_TUNE);
return ret;
}
@@ -1478,6 +1592,25 @@
.hw_reset = msdc_hw_reset,
};
+static void msdc_of_property_parse(struct platform_device *pdev,
+ struct msdc_host *host)
+{
+ of_property_read_u32(pdev->dev.of_node, "hs400-ds-delay",
+ &host->hs400_ds_delay);
+
+ of_property_read_u32(pdev->dev.of_node, "mediatek,hs200-cmd-int-delay",
+ &host->hs200_cmd_int_delay);
+
+ of_property_read_u32(pdev->dev.of_node, "mediatek,hs400-cmd-int-delay",
+ &host->hs400_cmd_int_delay);
+
+ if (of_property_read_bool(pdev->dev.of_node,
+ "mediatek,hs400-cmd-resp-sel-rising"))
+ host->hs400_cmd_resp_sel_rising = true;
+ else
+ host->hs400_cmd_resp_sel_rising = false;
+}
+
static int msdc_drv_probe(struct platform_device *pdev)
{
struct mmc_host *mmc;
@@ -1549,10 +1682,7 @@
goto host_free;
}
- if (!of_property_read_u32(pdev->dev.of_node, "hs400-ds-delay",
- &host->hs400_ds_delay))
- dev_dbg(&pdev->dev, "hs400-ds-delay: %x\n",
- host->hs400_ds_delay);
+ msdc_of_property_parse(pdev, host);
host->dev = &pdev->dev;
host->mmc = mmc;
@@ -1664,6 +1794,7 @@
host->save_para.patch_bit0 = readl(host->base + MSDC_PATCH_BIT);
host->save_para.patch_bit1 = readl(host->base + MSDC_PATCH_BIT1);
host->save_para.pad_ds_tune = readl(host->base + PAD_DS_TUNE);
+ host->save_para.pad_cmd_tune = readl(host->base + PAD_CMD_TUNE);
host->save_para.emmc50_cfg0 = readl(host->base + EMMC50_CFG0);
}
@@ -1676,6 +1807,7 @@
writel(host->save_para.patch_bit0, host->base + MSDC_PATCH_BIT);
writel(host->save_para.patch_bit1, host->base + MSDC_PATCH_BIT1);
writel(host->save_para.pad_ds_tune, host->base + PAD_DS_TUNE);
+ writel(host->save_para.pad_cmd_tune, host->base + PAD_CMD_TUNE);
writel(host->save_para.emmc50_cfg0, host->base + EMMC50_CFG0);
}
diff --git a/drivers/mmc/host/mvsdio.c b/drivers/mmc/host/mvsdio.c
index 42296e5..58d74b8 100644
--- a/drivers/mmc/host/mvsdio.c
+++ b/drivers/mmc/host/mvsdio.c
@@ -125,10 +125,10 @@
return 1;
} else {
dma_addr_t phys_addr;
- int dma_dir = (data->flags & MMC_DATA_READ) ?
- DMA_FROM_DEVICE : DMA_TO_DEVICE;
- host->sg_frags = dma_map_sg(mmc_dev(host->mmc), data->sg,
- data->sg_len, dma_dir);
+
+ host->sg_frags = dma_map_sg(mmc_dev(host->mmc),
+ data->sg, data->sg_len,
+ mmc_get_dma_dir(data));
phys_addr = sg_dma_address(data->sg);
mvsd_write(MVSD_SYS_ADDR_LOW, (u32)phys_addr & 0xffff);
mvsd_write(MVSD_SYS_ADDR_HI, (u32)phys_addr >> 16);
@@ -294,8 +294,7 @@
host->pio_size = 0;
} else {
dma_unmap_sg(mmc_dev(host->mmc), data->sg, host->sg_frags,
- (data->flags & MMC_DATA_READ) ?
- DMA_FROM_DEVICE : DMA_TO_DEVICE);
+ mmc_get_dma_dir(data));
}
if (err_status & MVSD_ERR_DATA_TIMEOUT)
diff --git a/drivers/mmc/host/omap_hsmmc.c b/drivers/mmc/host/omap_hsmmc.c
index a58bd65..8c39dcc 100644
--- a/drivers/mmc/host/omap_hsmmc.c
+++ b/drivers/mmc/host/omap_hsmmc.c
@@ -935,15 +935,6 @@
OMAP_HSMMC_WRITE(host->base, CMD, cmdreg);
}
-static int
-omap_hsmmc_get_dma_dir(struct omap_hsmmc_host *host, struct mmc_data *data)
-{
- if (data->flags & MMC_DATA_WRITE)
- return DMA_TO_DEVICE;
- else
- return DMA_FROM_DEVICE;
-}
-
static struct dma_chan *omap_hsmmc_get_dma_chan(struct omap_hsmmc_host *host,
struct mmc_data *data)
{
@@ -1055,7 +1046,7 @@
dmaengine_terminate_all(chan);
dma_unmap_sg(chan->device->dev,
host->data->sg, host->data->sg_len,
- omap_hsmmc_get_dma_dir(host, host->data));
+ mmc_get_dma_dir(host->data));
host->data->host_cookie = 0;
}
@@ -1350,7 +1341,7 @@
if (!data->host_cookie)
dma_unmap_sg(chan->device->dev,
data->sg, data->sg_len,
- omap_hsmmc_get_dma_dir(host, data));
+ mmc_get_dma_dir(data));
req_in_progress = host->req_in_progress;
host->dma_ch = -1;
@@ -1383,7 +1374,7 @@
/* Check if next job is already prepared */
if (next || data->host_cookie != host->next_data.cookie) {
dma_len = dma_map_sg(chan->device->dev, data->sg, data->sg_len,
- omap_hsmmc_get_dma_dir(host, data));
+ mmc_get_dma_dir(data));
} else {
dma_len = host->next_data.dma_len;
@@ -1569,7 +1560,7 @@
struct dma_chan *c = omap_hsmmc_get_dma_chan(host, data);
dma_unmap_sg(c->device->dev, data->sg, data->sg_len,
- omap_hsmmc_get_dma_dir(host, data));
+ mmc_get_dma_dir(data));
data->host_cookie = 0;
}
}
@@ -1770,8 +1761,8 @@
*/
if (host->pdata->controller_flags & OMAP_HSMMC_SWAKEUP_MISSING) {
struct pinctrl *p = devm_pinctrl_get(host->dev);
- if (!p) {
- ret = -ENODEV;
+ if (IS_ERR(p)) {
+ ret = PTR_ERR(p);
goto err_free_irq;
}
if (IS_ERR(pinctrl_lookup_state(p, PINCTRL_STATE_DEFAULT))) {
diff --git a/drivers/mmc/host/s3cmci.c b/drivers/mmc/host/s3cmci.c
index 7a173f8..8896bf5 100644
--- a/drivers/mmc/host/s3cmci.c
+++ b/drivers/mmc/host/s3cmci.c
@@ -24,6 +24,10 @@
#include <linux/interrupt.h>
#include <linux/irq.h>
#include <linux/io.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/of_gpio.h>
+#include <linux/mmc/slot-gpio.h>
#include <plat/gpio-cfg.h>
#include <mach/dma.h>
@@ -807,21 +811,6 @@
}
-/*
- * ISR for the CardDetect Pin
-*/
-
-static irqreturn_t s3cmci_irq_cd(int irq, void *dev_id)
-{
- struct s3cmci_host *host = (struct s3cmci_host *)dev_id;
-
- dbg(host, dbg_irq, "card detect\n");
-
- mmc_detect_change(host->mmc, msecs_to_jiffies(500));
-
- return IRQ_HANDLED;
-}
-
static void s3cmci_dma_done_callback(void *arg)
{
struct s3cmci_host *host = arg;
@@ -1104,7 +1093,7 @@
conf.direction = DMA_MEM_TO_DEV;
dma_map_sg(mmc_dev(host->mmc), data->sg, data->sg_len,
- rw ? DMA_TO_DEVICE : DMA_FROM_DEVICE);
+ mmc_get_dma_dir(data));
dmaengine_slave_config(host->dma, &conf);
desc = dmaengine_prep_slave_sg(host->dma, data->sg, data->sg_len,
@@ -1121,7 +1110,7 @@
unmap_exit:
dma_unmap_sg(mmc_dev(host->mmc), data->sg, data->sg_len,
- rw ? DMA_TO_DEVICE : DMA_FROM_DEVICE);
+ mmc_get_dma_dir(data));
return -ENOMEM;
}
@@ -1177,19 +1166,6 @@
s3cmci_enable_irq(host, true);
}
-static int s3cmci_card_present(struct mmc_host *mmc)
-{
- struct s3cmci_host *host = mmc_priv(mmc);
- struct s3c24xx_mci_pdata *pdata = host->pdata;
- int ret;
-
- if (pdata->no_detect)
- return -ENOSYS;
-
- ret = gpio_get_value(pdata->gpio_detect) ? 0 : 1;
- return ret ^ pdata->detect_invert;
-}
-
static void s3cmci_request(struct mmc_host *mmc, struct mmc_request *mrq)
{
struct s3cmci_host *host = mmc_priv(mmc);
@@ -1198,7 +1174,7 @@
host->cmd_is_stop = 0;
host->mrq = mrq;
- if (s3cmci_card_present(mmc) == 0) {
+ if (mmc_gpio_get_cd(mmc) == 0) {
dbg(host, dbg_err, "%s: no medium present\n", __func__);
host->mrq->cmd->error = -ENOMEDIUM;
mmc_request_done(mmc, mrq);
@@ -1242,8 +1218,9 @@
case MMC_POWER_ON:
case MMC_POWER_UP:
/* Configure GPE5...GPE10 pins in SD mode */
- s3c_gpio_cfgall_range(S3C2410_GPE(5), 6, S3C_GPIO_SFN(2),
- S3C_GPIO_PULL_NONE);
+ if (!host->pdev->dev.of_node)
+ s3c_gpio_cfgall_range(S3C2410_GPE(5), 6, S3C_GPIO_SFN(2),
+ S3C_GPIO_PULL_NONE);
if (host->pdata->set_power)
host->pdata->set_power(ios->power_mode, ios->vdd);
@@ -1255,7 +1232,8 @@
case MMC_POWER_OFF:
default:
- gpio_direction_output(S3C2410_GPE(5), 0);
+ if (!host->pdev->dev.of_node)
+ gpio_direction_output(S3C2410_GPE(5), 0);
if (host->is2440)
mci_con |= S3C2440_SDICON_SDRESET;
@@ -1295,21 +1273,6 @@
writel(con, host->base + S3C2410_SDICON);
}
-static int s3cmci_get_ro(struct mmc_host *mmc)
-{
- struct s3cmci_host *host = mmc_priv(mmc);
- struct s3c24xx_mci_pdata *pdata = host->pdata;
- int ret;
-
- if (pdata->no_wprotect)
- return 0;
-
- ret = gpio_get_value(pdata->gpio_wprotect) ? 1 : 0;
- ret ^= pdata->wprotect_invert;
-
- return ret;
-}
-
static void s3cmci_enable_sdio_irq(struct mmc_host *mmc, int enable)
{
struct s3cmci_host *host = mmc_priv(mmc);
@@ -1353,8 +1316,8 @@
static struct mmc_host_ops s3cmci_ops = {
.request = s3cmci_request,
.set_ios = s3cmci_set_ios,
- .get_ro = s3cmci_get_ro,
- .get_cd = s3cmci_card_present,
+ .get_ro = mmc_gpio_get_ro,
+ .get_cd = mmc_gpio_get_cd,
.enable_sdio_irq = s3cmci_enable_sdio_irq,
};
@@ -1545,21 +1508,14 @@
#endif /* CONFIG_DEBUG_FS */
-static int s3cmci_probe(struct platform_device *pdev)
+static int s3cmci_probe_pdata(struct s3cmci_host *host)
{
- struct s3cmci_host *host;
- struct mmc_host *mmc;
- int ret;
- int is2440;
- int i;
+ struct platform_device *pdev = host->pdev;
+ struct mmc_host *mmc = host->mmc;
+ struct s3c24xx_mci_pdata *pdata;
+ int i, ret;
- is2440 = platform_get_device_id(pdev)->driver_data;
-
- mmc = mmc_alloc_host(sizeof(struct s3cmci_host), &pdev->dev);
- if (!mmc) {
- ret = -ENOMEM;
- goto probe_out;
- }
+ host->is2440 = platform_get_device_id(pdev)->driver_data;
for (i = S3C2410_GPE(5); i <= S3C2410_GPE(10); i++) {
ret = gpio_request(i, dev_name(&pdev->dev));
@@ -1569,25 +1525,101 @@
for (i--; i >= S3C2410_GPE(5); i--)
gpio_free(i);
- goto probe_free_host;
+ return ret;
}
}
+ if (!pdev->dev.platform_data)
+ pdev->dev.platform_data = &s3cmci_def_pdata;
+
+ pdata = pdev->dev.platform_data;
+
+ if (pdata->no_wprotect)
+ mmc->caps2 |= MMC_CAP2_NO_WRITE_PROTECT;
+
+ if (pdata->no_detect)
+ mmc->caps |= MMC_CAP_NEEDS_POLL;
+
+ if (pdata->wprotect_invert)
+ mmc->caps2 |= MMC_CAP2_RO_ACTIVE_HIGH;
+
+ if (pdata->detect_invert)
+ mmc->caps2 |= MMC_CAP2_CD_ACTIVE_HIGH;
+
+ if (gpio_is_valid(pdata->gpio_detect)) {
+ ret = mmc_gpio_request_cd(mmc, pdata->gpio_detect, 0);
+ if (ret) {
+ dev_err(&pdev->dev, "error requesting GPIO for CD %d\n",
+ ret);
+ return ret;
+ }
+ }
+
+ if (gpio_is_valid(pdata->gpio_wprotect)) {
+ ret = mmc_gpio_request_ro(mmc, pdata->gpio_wprotect);
+ if (ret) {
+ dev_err(&pdev->dev, "error requesting GPIO for WP %d\n",
+ ret);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int s3cmci_probe_dt(struct s3cmci_host *host)
+{
+ struct platform_device *pdev = host->pdev;
+ struct s3c24xx_mci_pdata *pdata;
+ struct mmc_host *mmc = host->mmc;
+ int ret;
+
+ host->is2440 = (int) of_device_get_match_data(&pdev->dev);
+
+ ret = mmc_of_parse(mmc);
+ if (ret)
+ return ret;
+
+ pdata = devm_kzalloc(&pdev->dev, sizeof(*pdata), GFP_KERNEL);
+ if (!pdata)
+ return -ENOMEM;
+
+ pdev->dev.platform_data = pdata;
+
+ return 0;
+}
+
+static int s3cmci_probe(struct platform_device *pdev)
+{
+ struct s3cmci_host *host;
+ struct mmc_host *mmc;
+ int ret;
+ int i;
+
+ mmc = mmc_alloc_host(sizeof(struct s3cmci_host), &pdev->dev);
+ if (!mmc) {
+ ret = -ENOMEM;
+ goto probe_out;
+ }
+
host = mmc_priv(mmc);
host->mmc = mmc;
host->pdev = pdev;
- host->is2440 = is2440;
+
+ if (pdev->dev.of_node)
+ ret = s3cmci_probe_dt(host);
+ else
+ ret = s3cmci_probe_pdata(host);
+
+ if (ret)
+ goto probe_free_host;
host->pdata = pdev->dev.platform_data;
- if (!host->pdata) {
- pdev->dev.platform_data = &s3cmci_def_pdata;
- host->pdata = &s3cmci_def_pdata;
- }
spin_lock_init(&host->complete_lock);
tasklet_init(&host->pio_tasklet, pio_tasklet, (unsigned long) host);
- if (is2440) {
+ if (host->is2440) {
host->sdiimsk = S3C2440_SDIIMSK;
host->sdidata = S3C2440_SDIDATA;
host->clk_div = 1;
@@ -1645,43 +1677,6 @@
disable_irq(host->irq);
host->irq_state = false;
- if (!host->pdata->no_detect) {
- ret = gpio_request(host->pdata->gpio_detect, "s3cmci detect");
- if (ret) {
- dev_err(&pdev->dev, "failed to get detect gpio\n");
- goto probe_free_irq;
- }
-
- host->irq_cd = gpio_to_irq(host->pdata->gpio_detect);
-
- if (host->irq_cd >= 0) {
- if (request_irq(host->irq_cd, s3cmci_irq_cd,
- IRQF_TRIGGER_RISING |
- IRQF_TRIGGER_FALLING,
- DRIVER_NAME, host)) {
- dev_err(&pdev->dev,
- "can't get card detect irq.\n");
- ret = -ENOENT;
- goto probe_free_gpio_cd;
- }
- } else {
- dev_warn(&pdev->dev,
- "host detect has no irq available\n");
- gpio_direction_input(host->pdata->gpio_detect);
- }
- } else
- host->irq_cd = -1;
-
- if (!host->pdata->no_wprotect) {
- ret = gpio_request(host->pdata->gpio_wprotect, "s3cmci wp");
- if (ret) {
- dev_err(&pdev->dev, "failed to get writeprotect\n");
- goto probe_free_irq_cd;
- }
-
- gpio_direction_input(host->pdata->gpio_wprotect);
- }
-
/* Depending on the dma state, get a DMA channel to use. */
if (s3cmci_host_usedma(host)) {
@@ -1689,7 +1684,7 @@
ret = PTR_ERR_OR_ZERO(host->dma);
if (ret) {
dev_err(&pdev->dev, "cannot get DMA channel.\n");
- goto probe_free_gpio_wp;
+ goto probe_free_irq;
}
}
@@ -1768,18 +1763,6 @@
if (s3cmci_host_usedma(host))
dma_release_channel(host->dma);
- probe_free_gpio_wp:
- if (!host->pdata->no_wprotect)
- gpio_free(host->pdata->gpio_wprotect);
-
- probe_free_gpio_cd:
- if (!host->pdata->no_detect)
- gpio_free(host->pdata->gpio_detect);
-
- probe_free_irq_cd:
- if (host->irq_cd >= 0)
- free_irq(host->irq_cd, host);
-
probe_free_irq:
free_irq(host->irq, host);
@@ -1790,8 +1773,9 @@
release_mem_region(host->mem->start, resource_size(host->mem));
probe_free_gpio:
- for (i = S3C2410_GPE(5); i <= S3C2410_GPE(10); i++)
- gpio_free(i);
+ if (!pdev->dev.of_node)
+ for (i = S3C2410_GPE(5); i <= S3C2410_GPE(10); i++)
+ gpio_free(i);
probe_free_host:
mmc_free_host(mmc);
@@ -1818,7 +1802,6 @@
{
struct mmc_host *mmc = platform_get_drvdata(pdev);
struct s3cmci_host *host = mmc_priv(mmc);
- struct s3c24xx_mci_pdata *pd = host->pdata;
int i;
s3cmci_shutdown(pdev);
@@ -1832,15 +1815,9 @@
free_irq(host->irq, host);
- if (!pd->no_wprotect)
- gpio_free(pd->gpio_wprotect);
-
- if (!pd->no_detect)
- gpio_free(pd->gpio_detect);
-
- for (i = S3C2410_GPE(5); i <= S3C2410_GPE(10); i++)
- gpio_free(i);
-
+ if (!pdev->dev.of_node)
+ for (i = S3C2410_GPE(5); i <= S3C2410_GPE(10); i++)
+ gpio_free(i);
iounmap(host->base);
release_mem_region(host->mem->start, resource_size(host->mem));
@@ -1849,6 +1826,23 @@
return 0;
}
+static const struct of_device_id s3cmci_dt_match[] = {
+ {
+ .compatible = "samsung,s3c2410-sdi",
+ .data = (void *)0,
+ },
+ {
+ .compatible = "samsung,s3c2412-sdi",
+ .data = (void *)1,
+ },
+ {
+ .compatible = "samsung,s3c2440-sdi",
+ .data = (void *)1,
+ },
+ { /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, s3cmci_dt_match);
+
static const struct platform_device_id s3cmci_driver_ids[] = {
{
.name = "s3c2410-sdi",
@@ -1868,6 +1862,7 @@
static struct platform_driver s3cmci_driver = {
.driver = {
.name = "s3c-sdi",
+ .of_match_table = s3cmci_dt_match,
},
.id_table = s3cmci_driver_ids,
.probe = s3cmci_probe,
diff --git a/drivers/mmc/host/sdhci-acpi.c b/drivers/mmc/host/sdhci-acpi.c
index 9dcb704..c6a9a1b 100644
--- a/drivers/mmc/host/sdhci-acpi.c
+++ b/drivers/mmc/host/sdhci-acpi.c
@@ -263,10 +263,8 @@
/* Platform specific code during sd probe slot goes here */
- if (hid && !strcmp(hid, "80865ACA")) {
+ if (hid && !strcmp(hid, "80865ACA"))
host->mmc_host_ops.get_cd = bxt_get_cd;
- host->mmc->caps |= MMC_CAP_AGGRESSIVE_PM;
- }
return 0;
}
@@ -302,7 +300,7 @@
.quirks = SDHCI_QUIRK_NO_ENDATTR_IN_NOPDESC,
.quirks2 = SDHCI_QUIRK2_CARD_ON_NEEDS_BUS_ON |
SDHCI_QUIRK2_STOP_WITH_TC,
- .caps = MMC_CAP_WAIT_WHILE_BUSY,
+ .caps = MMC_CAP_WAIT_WHILE_BUSY | MMC_CAP_AGGRESSIVE_PM,
.probe_slot = sdhci_acpi_sd_probe_slot,
};
@@ -524,8 +522,12 @@
static int sdhci_acpi_suspend(struct device *dev)
{
struct sdhci_acpi_host *c = dev_get_drvdata(dev);
+ struct sdhci_host *host = c->host;
- return sdhci_suspend_host(c->host);
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
+ return sdhci_suspend_host(host);
}
static int sdhci_acpi_resume(struct device *dev)
@@ -544,8 +546,12 @@
static int sdhci_acpi_runtime_suspend(struct device *dev)
{
struct sdhci_acpi_host *c = dev_get_drvdata(dev);
+ struct sdhci_host *host = c->host;
- return sdhci_runtime_suspend_host(c->host);
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
+ return sdhci_runtime_suspend_host(host);
}
static int sdhci_acpi_runtime_resume(struct device *dev)
diff --git a/drivers/mmc/host/sdhci-brcmstb.c b/drivers/mmc/host/sdhci-brcmstb.c
index 159f6f6..242c5dc 100644
--- a/drivers/mmc/host/sdhci-brcmstb.c
+++ b/drivers/mmc/host/sdhci-brcmstb.c
@@ -29,6 +29,9 @@
struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
int res;
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
res = sdhci_suspend_host(host);
if (res)
return res;
diff --git a/drivers/mmc/host/sdhci-cadence.c b/drivers/mmc/host/sdhci-cadence.c
index 316cfec..19d5698 100644
--- a/drivers/mmc/host/sdhci-cadence.c
+++ b/drivers/mmc/host/sdhci-cadence.c
@@ -18,6 +18,7 @@
#include <linux/module.h>
#include <linux/mmc/host.h>
#include <linux/mmc/mmc.h>
+#include <linux/of.h>
#include "sdhci-pltfm.h"
@@ -40,6 +41,7 @@
#define SDHCI_CDNS_HRS06_MODE_MMC_DDR 0x3
#define SDHCI_CDNS_HRS06_MODE_MMC_HS200 0x4
#define SDHCI_CDNS_HRS06_MODE_MMC_HS400 0x5
+#define SDHCI_CDNS_HRS06_MODE_MMC_HS400ES 0x6
/* SRS - Slot Register Set (SDHCI-compatible) */
#define SDHCI_CDNS_SRS_BASE 0x200
@@ -54,6 +56,9 @@
#define SDHCI_CDNS_PHY_DLY_EMMC_LEGACY 0x06
#define SDHCI_CDNS_PHY_DLY_EMMC_SDR 0x07
#define SDHCI_CDNS_PHY_DLY_EMMC_DDR 0x08
+#define SDHCI_CDNS_PHY_DLY_SDCLK 0x0b
+#define SDHCI_CDNS_PHY_DLY_HSMMC 0x0c
+#define SDHCI_CDNS_PHY_DLY_STROBE 0x0d
/*
* The tuned val register is 6 bit-wide, but not the whole of the range is
@@ -64,13 +69,34 @@
struct sdhci_cdns_priv {
void __iomem *hrs_addr;
+ bool enhanced_strobe;
};
-static void sdhci_cdns_write_phy_reg(struct sdhci_cdns_priv *priv,
- u8 addr, u8 data)
+struct sdhci_cdns_phy_cfg {
+ const char *property;
+ u8 addr;
+};
+
+static const struct sdhci_cdns_phy_cfg sdhci_cdns_phy_cfgs[] = {
+ { "cdns,phy-input-delay-sd-highspeed", SDHCI_CDNS_PHY_DLY_SD_HS, },
+ { "cdns,phy-input-delay-legacy", SDHCI_CDNS_PHY_DLY_SD_DEFAULT, },
+ { "cdns,phy-input-delay-sd-uhs-sdr12", SDHCI_CDNS_PHY_DLY_UHS_SDR12, },
+ { "cdns,phy-input-delay-sd-uhs-sdr25", SDHCI_CDNS_PHY_DLY_UHS_SDR25, },
+ { "cdns,phy-input-delay-sd-uhs-sdr50", SDHCI_CDNS_PHY_DLY_UHS_SDR50, },
+ { "cdns,phy-input-delay-sd-uhs-ddr50", SDHCI_CDNS_PHY_DLY_UHS_DDR50, },
+ { "cdns,phy-input-delay-mmc-highspeed", SDHCI_CDNS_PHY_DLY_EMMC_SDR, },
+ { "cdns,phy-input-delay-mmc-ddr", SDHCI_CDNS_PHY_DLY_EMMC_DDR, },
+ { "cdns,phy-dll-delay-sdclk", SDHCI_CDNS_PHY_DLY_SDCLK, },
+ { "cdns,phy-dll-delay-sdclk-hsmmc", SDHCI_CDNS_PHY_DLY_HSMMC, },
+ { "cdns,phy-dll-delay-strobe", SDHCI_CDNS_PHY_DLY_STROBE, },
+};
+
+static int sdhci_cdns_write_phy_reg(struct sdhci_cdns_priv *priv,
+ u8 addr, u8 data)
{
void __iomem *reg = priv->hrs_addr + SDHCI_CDNS_HRS04;
u32 tmp;
+ int ret;
tmp = (data << SDHCI_CDNS_HRS04_WDATA_SHIFT) |
(addr << SDHCI_CDNS_HRS04_ADDR_SHIFT);
@@ -79,17 +105,36 @@
tmp |= SDHCI_CDNS_HRS04_WR;
writel(tmp, reg);
+ ret = readl_poll_timeout(reg, tmp, tmp & SDHCI_CDNS_HRS04_ACK, 0, 10);
+ if (ret)
+ return ret;
+
tmp &= ~SDHCI_CDNS_HRS04_WR;
writel(tmp, reg);
+
+ return 0;
}
-static void sdhci_cdns_phy_init(struct sdhci_cdns_priv *priv)
+static int sdhci_cdns_phy_init(struct device_node *np,
+ struct sdhci_cdns_priv *priv)
{
- sdhci_cdns_write_phy_reg(priv, SDHCI_CDNS_PHY_DLY_SD_HS, 4);
- sdhci_cdns_write_phy_reg(priv, SDHCI_CDNS_PHY_DLY_SD_DEFAULT, 4);
- sdhci_cdns_write_phy_reg(priv, SDHCI_CDNS_PHY_DLY_EMMC_LEGACY, 9);
- sdhci_cdns_write_phy_reg(priv, SDHCI_CDNS_PHY_DLY_EMMC_SDR, 2);
- sdhci_cdns_write_phy_reg(priv, SDHCI_CDNS_PHY_DLY_EMMC_DDR, 3);
+ u32 val;
+ int ret, i;
+
+ for (i = 0; i < ARRAY_SIZE(sdhci_cdns_phy_cfgs); i++) {
+ ret = of_property_read_u32(np, sdhci_cdns_phy_cfgs[i].property,
+ &val);
+ if (ret)
+ continue;
+
+ ret = sdhci_cdns_write_phy_reg(priv,
+ sdhci_cdns_phy_cfgs[i].addr,
+ val);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
}
static inline void *sdhci_cdns_priv(struct sdhci_host *host)
@@ -103,16 +148,35 @@
{
/*
* Cadence's spec says the Timeout Clock Frequency is the same as the
- * Base Clock Frequency. Divide it by 1000 to return a value in kHz.
+ * Base Clock Frequency.
*/
- return host->max_clk / 1000;
+ return host->max_clk;
+}
+
+static void sdhci_cdns_set_emmc_mode(struct sdhci_cdns_priv *priv, u32 mode)
+{
+ u32 tmp;
+
+ /* The speed mode for eMMC is selected by HRS06 register */
+ tmp = readl(priv->hrs_addr + SDHCI_CDNS_HRS06);
+ tmp &= ~SDHCI_CDNS_HRS06_MODE_MASK;
+ tmp |= mode;
+ writel(tmp, priv->hrs_addr + SDHCI_CDNS_HRS06);
+}
+
+static u32 sdhci_cdns_get_emmc_mode(struct sdhci_cdns_priv *priv)
+{
+ u32 tmp;
+
+ tmp = readl(priv->hrs_addr + SDHCI_CDNS_HRS06);
+ return tmp & SDHCI_CDNS_HRS06_MODE_MASK;
}
static void sdhci_cdns_set_uhs_signaling(struct sdhci_host *host,
unsigned int timing)
{
struct sdhci_cdns_priv *priv = sdhci_cdns_priv(host);
- u32 mode, tmp;
+ u32 mode;
switch (timing) {
case MMC_TIMING_MMC_HS:
@@ -125,18 +189,17 @@
mode = SDHCI_CDNS_HRS06_MODE_MMC_HS200;
break;
case MMC_TIMING_MMC_HS400:
- mode = SDHCI_CDNS_HRS06_MODE_MMC_HS400;
+ if (priv->enhanced_strobe)
+ mode = SDHCI_CDNS_HRS06_MODE_MMC_HS400ES;
+ else
+ mode = SDHCI_CDNS_HRS06_MODE_MMC_HS400;
break;
default:
mode = SDHCI_CDNS_HRS06_MODE_SD;
break;
}
- /* The speed mode for eMMC is selected by HRS06 register */
- tmp = readl(priv->hrs_addr + SDHCI_CDNS_HRS06);
- tmp &= ~SDHCI_CDNS_HRS06_MODE_MASK;
- tmp |= mode;
- writel(tmp, priv->hrs_addr + SDHCI_CDNS_HRS06);
+ sdhci_cdns_set_emmc_mode(priv, mode);
/* For SD, fall back to the default handler */
if (mode == SDHCI_CDNS_HRS06_MODE_SD)
@@ -213,6 +276,26 @@
return sdhci_cdns_set_tune_val(host, end_of_streak - max_streak / 2);
}
+static void sdhci_cdns_hs400_enhanced_strobe(struct mmc_host *mmc,
+ struct mmc_ios *ios)
+{
+ struct sdhci_host *host = mmc_priv(mmc);
+ struct sdhci_cdns_priv *priv = sdhci_cdns_priv(host);
+ u32 mode;
+
+ priv->enhanced_strobe = ios->enhanced_strobe;
+
+ mode = sdhci_cdns_get_emmc_mode(priv);
+
+ if (mode == SDHCI_CDNS_HRS06_MODE_MMC_HS400 && ios->enhanced_strobe)
+ sdhci_cdns_set_emmc_mode(priv,
+ SDHCI_CDNS_HRS06_MODE_MMC_HS400ES);
+
+ if (mode == SDHCI_CDNS_HRS06_MODE_MMC_HS400ES && !ios->enhanced_strobe)
+ sdhci_cdns_set_emmc_mode(priv,
+ SDHCI_CDNS_HRS06_MODE_MMC_HS400);
+}
+
static int sdhci_cdns_probe(struct platform_device *pdev)
{
struct sdhci_host *host;
@@ -220,8 +303,9 @@
struct sdhci_cdns_priv *priv;
struct clk *clk;
int ret;
+ struct device *dev = &pdev->dev;
- clk = devm_clk_get(&pdev->dev, NULL);
+ clk = devm_clk_get(dev, NULL);
if (IS_ERR(clk))
return PTR_ERR(clk);
@@ -240,14 +324,21 @@
priv = sdhci_cdns_priv(host);
priv->hrs_addr = host->ioaddr;
+ priv->enhanced_strobe = false;
host->ioaddr += SDHCI_CDNS_SRS_BASE;
host->mmc_host_ops.execute_tuning = sdhci_cdns_execute_tuning;
+ host->mmc_host_ops.hs400_enhanced_strobe =
+ sdhci_cdns_hs400_enhanced_strobe;
+
+ sdhci_get_of_property(pdev);
ret = mmc_of_parse(host->mmc);
if (ret)
goto free;
- sdhci_cdns_phy_init(priv);
+ ret = sdhci_cdns_phy_init(dev->of_node, priv);
+ if (ret)
+ goto free;
ret = sdhci_add_host(host);
if (ret)
diff --git a/drivers/mmc/host/sdhci-esdhc-imx.c b/drivers/mmc/host/sdhci-esdhc-imx.c
index 445fc47..23d8b8a 100644
--- a/drivers/mmc/host/sdhci-esdhc-imx.c
+++ b/drivers/mmc/host/sdhci-esdhc-imx.c
@@ -889,6 +889,28 @@
}
}
+static void esdhc_reset_tuning(struct sdhci_host *host)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct pltfm_imx_data *imx_data = sdhci_pltfm_priv(pltfm_host);
+ u32 ctrl;
+
+ /* Rest the tuning circurt */
+ if (esdhc_is_usdhc(imx_data)) {
+ if (imx_data->socdata->flags & ESDHC_FLAG_MAN_TUNING) {
+ ctrl = readl(host->ioaddr + ESDHC_MIX_CTRL);
+ ctrl &= ~ESDHC_MIX_CTRL_SMPCLK_SEL;
+ ctrl &= ~ESDHC_MIX_CTRL_FBCLK_SEL;
+ writel(ctrl, host->ioaddr + ESDHC_MIX_CTRL);
+ writel(0, host->ioaddr + ESDHC_TUNE_CTRL_STATUS);
+ } else if (imx_data->socdata->flags & ESDHC_FLAG_STD_TUNING) {
+ ctrl = readl(host->ioaddr + SDHCI_ACMD12_ERR);
+ ctrl &= ~ESDHC_MIX_CTRL_SMPCLK_SEL;
+ writel(ctrl, host->ioaddr + SDHCI_ACMD12_ERR);
+ }
+ }
+}
+
static void esdhc_set_uhs_signaling(struct sdhci_host *host, unsigned timing)
{
u32 m;
@@ -932,6 +954,10 @@
host->ops->set_clock(host, host->clock);
esdhc_set_strobe_dll(host);
break;
+ case MMC_TIMING_LEGACY:
+ default:
+ esdhc_reset_tuning(host);
+ break;
}
esdhc_change_pinstate(host, timing);
@@ -1323,6 +1349,9 @@
{
struct sdhci_host *host = dev_get_drvdata(dev);
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
return sdhci_suspend_host(host);
}
@@ -1347,6 +1376,9 @@
ret = sdhci_runtime_suspend_host(host);
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
if (!sdhci_sdio_irq_enabled(host)) {
clk_disable_unprepare(imx_data->clk_per);
clk_disable_unprepare(imx_data->clk_ipg);
diff --git a/drivers/mmc/host/sdhci-esdhc.h b/drivers/mmc/host/sdhci-esdhc.h
index ece8b37..c4bbd74 100644
--- a/drivers/mmc/host/sdhci-esdhc.h
+++ b/drivers/mmc/host/sdhci-esdhc.h
@@ -37,6 +37,7 @@
/* Protocol Control Register */
#define ESDHC_PROCTL 0x28
+#define ESDHC_VOLT_SEL 0x00000400
#define ESDHC_CTRL_4BITBUS (0x1 << 1)
#define ESDHC_CTRL_8BITBUS (0x2 << 1)
#define ESDHC_CTRL_BUSWIDTH_MASK (0x3 << 1)
@@ -52,8 +53,14 @@
#define ESDHC_CLOCK_HCKEN 0x00000002
#define ESDHC_CLOCK_IPGEN 0x00000001
+/* Tuning Block Control Register */
+#define ESDHC_TBCTL 0x120
+#define ESDHC_TB_EN 0x00000004
+
/* Control Register for DMA transfer */
#define ESDHC_DMA_SYSCTL 0x40c
+#define ESDHC_PERIPHERAL_CLK_SEL 0x00080000
+#define ESDHC_FLUSH_ASYNC_FIFO 0x00040000
#define ESDHC_DMA_SNOOP 0x00000040
#endif /* _DRIVERS_MMC_SDHCI_ESDHC_H */
diff --git a/drivers/mmc/host/sdhci-msm.c b/drivers/mmc/host/sdhci-msm.c
index 10cdc84..9d601dc 100644
--- a/drivers/mmc/host/sdhci-msm.c
+++ b/drivers/mmc/host/sdhci-msm.c
@@ -991,12 +991,8 @@
mmc_hostname(host->mmc), host->clock, uhs, ctrl_2);
sdhci_writew(host, ctrl_2, SDHCI_HOST_CONTROL2);
- spin_unlock_irq(&host->lock);
-
if (mmc->ios.timing == MMC_TIMING_MMC_HS400)
sdhci_msm_hs400(host, &mmc->ios);
-
- spin_lock_irq(&host->lock);
}
static void sdhci_msm_voltage_switch(struct sdhci_host *host)
@@ -1089,13 +1085,9 @@
goto out;
}
- spin_unlock_irq(&host->lock);
-
sdhci_msm_hc_select_mode(host);
msm_set_clock_rate_for_bus_mode(host, clock);
-
- spin_lock_irq(&host->lock);
out:
__sdhci_msm_set_clock(host, clock);
}
diff --git a/drivers/mmc/host/sdhci-of-arasan.c b/drivers/mmc/host/sdhci-of-arasan.c
index 1cfd7f9..ea6b36c 100644
--- a/drivers/mmc/host/sdhci-of-arasan.c
+++ b/drivers/mmc/host/sdhci-of-arasan.c
@@ -157,21 +157,6 @@
return ret;
}
-static unsigned int sdhci_arasan_get_timeout_clock(struct sdhci_host *host)
-{
- unsigned long freq;
- struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
-
- /* SDHCI timeout clock is in kHz */
- freq = DIV_ROUND_UP(clk_get_rate(pltfm_host->clk), 1000);
-
- /* or in MHz */
- if (host->caps & SDHCI_TIMEOUT_CLK_UNIT)
- freq = DIV_ROUND_UP(freq, 1000);
-
- return freq;
-}
-
static void sdhci_arasan_set_clock(struct sdhci_host *host, unsigned int clock)
{
struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
@@ -194,9 +179,7 @@
* through low speeds without power cycling.
*/
sdhci_set_clock(host, host->max_clk);
- spin_unlock_irq(&host->lock);
phy_power_on(sdhci_arasan->phy);
- spin_lock_irq(&host->lock);
sdhci_arasan->is_phy_on = true;
/*
@@ -215,18 +198,14 @@
}
if (ctrl_phy && sdhci_arasan->is_phy_on) {
- spin_unlock_irq(&host->lock);
phy_power_off(sdhci_arasan->phy);
- spin_lock_irq(&host->lock);
sdhci_arasan->is_phy_on = false;
}
sdhci_set_clock(host, clock);
if (ctrl_phy) {
- spin_unlock_irq(&host->lock);
phy_power_on(sdhci_arasan->phy);
- spin_lock_irq(&host->lock);
sdhci_arasan->is_phy_on = true;
}
}
@@ -286,7 +265,7 @@
static struct sdhci_ops sdhci_arasan_ops = {
.set_clock = sdhci_arasan_set_clock,
.get_max_clock = sdhci_pltfm_clk_get_max_clock,
- .get_timeout_clock = sdhci_arasan_get_timeout_clock,
+ .get_timeout_clock = sdhci_pltfm_clk_get_max_clock,
.set_bus_width = sdhci_set_bus_width,
.reset = sdhci_arasan_reset,
.set_uhs_signaling = sdhci_set_uhs_signaling,
@@ -315,6 +294,9 @@
struct sdhci_arasan_data *sdhci_arasan = sdhci_pltfm_priv(pltfm_host);
int ret;
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
ret = sdhci_suspend_host(host);
if (ret)
return ret;
diff --git a/drivers/mmc/host/sdhci-of-at91.c b/drivers/mmc/host/sdhci-of-at91.c
index d5430ed..7611fd6 100644
--- a/drivers/mmc/host/sdhci-of-at91.c
+++ b/drivers/mmc/host/sdhci-of-at91.c
@@ -98,9 +98,7 @@
if (!IS_ERR(host->mmc->supply.vmmc)) {
struct mmc_host *mmc = host->mmc;
- spin_unlock_irq(&host->lock);
mmc_regulator_set_ocr(mmc, mmc->supply.vmmc, vdd);
- spin_lock_irq(&host->lock);
}
sdhci_set_power_noreg(host, mode, vdd);
}
@@ -140,6 +138,9 @@
ret = sdhci_runtime_suspend_host(host);
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
clk_disable_unprepare(priv->gck);
clk_disable_unprepare(priv->hclock);
clk_disable_unprepare(priv->mainck);
diff --git a/drivers/mmc/host/sdhci-of-esdhc.c b/drivers/mmc/host/sdhci-of-esdhc.c
index d3aa671..44b016b 100644
--- a/drivers/mmc/host/sdhci-of-esdhc.c
+++ b/drivers/mmc/host/sdhci-of-esdhc.c
@@ -16,9 +16,12 @@
#include <linux/err.h>
#include <linux/io.h>
#include <linux/of.h>
+#include <linux/of_address.h>
#include <linux/delay.h>
#include <linux/module.h>
#include <linux/sys_soc.h>
+#include <linux/clk.h>
+#include <linux/ktime.h>
#include <linux/mmc/host.h>
#include "sdhci-pltfm.h"
#include "sdhci-esdhc.h"
@@ -30,6 +33,7 @@
u8 vendor_ver;
u8 spec_ver;
bool quirk_incorrect_hostver;
+ unsigned int peripheral_clock;
};
/**
@@ -414,15 +418,25 @@
static unsigned int esdhc_of_get_max_clock(struct sdhci_host *host)
{
struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct sdhci_esdhc *esdhc = sdhci_pltfm_priv(pltfm_host);
- return pltfm_host->clock;
+ if (esdhc->peripheral_clock)
+ return esdhc->peripheral_clock;
+ else
+ return pltfm_host->clock;
}
static unsigned int esdhc_of_get_min_clock(struct sdhci_host *host)
{
struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct sdhci_esdhc *esdhc = sdhci_pltfm_priv(pltfm_host);
+ unsigned int clock;
- return pltfm_host->clock / 256 / 16;
+ if (esdhc->peripheral_clock)
+ clock = esdhc->peripheral_clock;
+ else
+ clock = pltfm_host->clock;
+ return clock / 256 / 16;
}
static void esdhc_of_set_clock(struct sdhci_host *host, unsigned int clock)
@@ -431,7 +445,7 @@
struct sdhci_esdhc *esdhc = sdhci_pltfm_priv(pltfm_host);
int pre_div = 1;
int div = 1;
- u32 timeout;
+ ktime_t timeout;
u32 temp;
host->mmc->actual_clock = 0;
@@ -443,6 +457,20 @@
if (esdhc->vendor_ver < VENDOR_V_23)
pre_div = 2;
+ /*
+ * Limit SD clock to 167MHz for ls1046a according to its datasheet
+ */
+ if (clock > 167000000 &&
+ of_find_compatible_node(NULL, NULL, "fsl,ls1046a-esdhc"))
+ clock = 167000000;
+
+ /*
+ * Limit SD clock to 125MHz for ls1012a according to its datasheet
+ */
+ if (clock > 125000000 &&
+ of_find_compatible_node(NULL, NULL, "fsl,ls1012a-esdhc"))
+ clock = 125000000;
+
/* Workaround to reduce the clock frequency for p1010 esdhc */
if (of_find_compatible_node(NULL, NULL, "fsl,p1010-esdhc")) {
if (clock > 20000000)
@@ -475,15 +503,14 @@
sdhci_writel(host, temp, ESDHC_SYSTEM_CONTROL);
/* Wait max 20 ms */
- timeout = 20;
+ timeout = ktime_add_ms(ktime_get(), 20);
while (!(sdhci_readl(host, ESDHC_PRSSTAT) & ESDHC_CLOCK_STABLE)) {
- if (timeout == 0) {
+ if (ktime_after(ktime_get(), timeout)) {
pr_err("%s: Internal clock never stabilised.\n",
mmc_hostname(host->mmc));
return;
}
- timeout--;
- mdelay(1);
+ udelay(10);
}
temp |= ESDHC_CLOCK_SDCLKEN;
@@ -512,6 +539,33 @@
sdhci_writel(host, ctrl, ESDHC_PROCTL);
}
+static void esdhc_clock_enable(struct sdhci_host *host, bool enable)
+{
+ u32 val;
+ ktime_t timeout;
+
+ val = sdhci_readl(host, ESDHC_SYSTEM_CONTROL);
+
+ if (enable)
+ val |= ESDHC_CLOCK_SDCLKEN;
+ else
+ val &= ~ESDHC_CLOCK_SDCLKEN;
+
+ sdhci_writel(host, val, ESDHC_SYSTEM_CONTROL);
+
+ /* Wait max 20 ms */
+ timeout = ktime_add_ms(ktime_get(), 20);
+ val = ESDHC_CLOCK_STABLE;
+ while (!(sdhci_readl(host, ESDHC_PRSSTAT) & val)) {
+ if (ktime_after(ktime_get(), timeout)) {
+ pr_err("%s: Internal clock never stabilised.\n",
+ mmc_hostname(host->mmc));
+ break;
+ }
+ udelay(10);
+ }
+}
+
static void esdhc_reset(struct sdhci_host *host, u8 mask)
{
sdhci_reset(host, mask);
@@ -520,6 +574,95 @@
sdhci_writel(host, host->ier, SDHCI_SIGNAL_ENABLE);
}
+/* The SCFG, Supplemental Configuration Unit, provides SoC specific
+ * configuration and status registers for the device. There is a
+ * SDHC IO VSEL control register on SCFG for some platforms. It's
+ * used to support SDHC IO voltage switching.
+ */
+static const struct of_device_id scfg_device_ids[] = {
+ { .compatible = "fsl,t1040-scfg", },
+ { .compatible = "fsl,ls1012a-scfg", },
+ { .compatible = "fsl,ls1046a-scfg", },
+ {}
+};
+
+/* SDHC IO VSEL control register definition */
+#define SCFG_SDHCIOVSELCR 0x408
+#define SDHCIOVSELCR_TGLEN 0x80000000
+#define SDHCIOVSELCR_VSELVAL 0x60000000
+#define SDHCIOVSELCR_SDHC_VS 0x00000001
+
+static int esdhc_signal_voltage_switch(struct mmc_host *mmc,
+ struct mmc_ios *ios)
+{
+ struct sdhci_host *host = mmc_priv(mmc);
+ struct device_node *scfg_node;
+ void __iomem *scfg_base = NULL;
+ u32 sdhciovselcr;
+ u32 val;
+
+ /*
+ * Signal Voltage Switching is only applicable for Host Controllers
+ * v3.00 and above.
+ */
+ if (host->version < SDHCI_SPEC_300)
+ return 0;
+
+ val = sdhci_readl(host, ESDHC_PROCTL);
+
+ switch (ios->signal_voltage) {
+ case MMC_SIGNAL_VOLTAGE_330:
+ val &= ~ESDHC_VOLT_SEL;
+ sdhci_writel(host, val, ESDHC_PROCTL);
+ return 0;
+ case MMC_SIGNAL_VOLTAGE_180:
+ scfg_node = of_find_matching_node(NULL, scfg_device_ids);
+ if (scfg_node)
+ scfg_base = of_iomap(scfg_node, 0);
+ if (scfg_base) {
+ sdhciovselcr = SDHCIOVSELCR_TGLEN |
+ SDHCIOVSELCR_VSELVAL;
+ iowrite32be(sdhciovselcr,
+ scfg_base + SCFG_SDHCIOVSELCR);
+
+ val |= ESDHC_VOLT_SEL;
+ sdhci_writel(host, val, ESDHC_PROCTL);
+ mdelay(5);
+
+ sdhciovselcr = SDHCIOVSELCR_TGLEN |
+ SDHCIOVSELCR_SDHC_VS;
+ iowrite32be(sdhciovselcr,
+ scfg_base + SCFG_SDHCIOVSELCR);
+ iounmap(scfg_base);
+ } else {
+ val |= ESDHC_VOLT_SEL;
+ sdhci_writel(host, val, ESDHC_PROCTL);
+ }
+ return 0;
+ default:
+ return 0;
+ }
+}
+
+static int esdhc_execute_tuning(struct mmc_host *mmc, u32 opcode)
+{
+ struct sdhci_host *host = mmc_priv(mmc);
+ u32 val;
+
+ /* Use tuning block for tuning procedure */
+ esdhc_clock_enable(host, false);
+ val = sdhci_readl(host, ESDHC_DMA_SYSCTL);
+ val |= ESDHC_FLUSH_ASYNC_FIFO;
+ sdhci_writel(host, val, ESDHC_DMA_SYSCTL);
+
+ val = sdhci_readl(host, ESDHC_TBCTL);
+ val |= ESDHC_TB_EN;
+ sdhci_writel(host, val, ESDHC_TBCTL);
+ esdhc_clock_enable(host, true);
+
+ return sdhci_execute_tuning(mmc, opcode);
+}
+
#ifdef CONFIG_PM_SLEEP
static u32 esdhc_proctl;
static int esdhc_of_suspend(struct device *dev)
@@ -528,6 +671,9 @@
esdhc_proctl = sdhci_readl(host, SDHCI_HOST_CONTROL);
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
return sdhci_suspend_host(host);
}
@@ -610,6 +756,9 @@
{
struct sdhci_pltfm_host *pltfm_host;
struct sdhci_esdhc *esdhc;
+ struct device_node *np;
+ struct clk *clk;
+ u32 val;
u16 host_ver;
pltfm_host = sdhci_priv(host);
@@ -623,6 +772,32 @@
esdhc->quirk_incorrect_hostver = true;
else
esdhc->quirk_incorrect_hostver = false;
+
+ np = pdev->dev.of_node;
+ clk = of_clk_get(np, 0);
+ if (!IS_ERR(clk)) {
+ /*
+ * esdhc->peripheral_clock would be assigned with a value
+ * which is eSDHC base clock when use periperal clock.
+ * For ls1046a, the clock value got by common clk API is
+ * peripheral clock while the eSDHC base clock is 1/2
+ * peripheral clock.
+ */
+ if (of_device_is_compatible(np, "fsl,ls1046a-esdhc"))
+ esdhc->peripheral_clock = clk_get_rate(clk) / 2;
+ else
+ esdhc->peripheral_clock = clk_get_rate(clk);
+
+ clk_put(clk);
+ }
+
+ if (esdhc->peripheral_clock) {
+ esdhc_clock_enable(host, false);
+ val = sdhci_readl(host, ESDHC_DMA_SYSCTL);
+ val |= ESDHC_PERIPHERAL_CLK_SEL;
+ sdhci_writel(host, val, ESDHC_DMA_SYSCTL);
+ esdhc_clock_enable(host, true);
+ }
}
static int sdhci_esdhc_probe(struct platform_device *pdev)
@@ -645,6 +820,11 @@
if (IS_ERR(host))
return PTR_ERR(host);
+ host->mmc_host_ops.start_signal_voltage_switch =
+ esdhc_signal_voltage_switch;
+ host->mmc_host_ops.execute_tuning = esdhc_execute_tuning;
+ host->tuning_delay = 1;
+
esdhc_init(pdev, host);
sdhci_get_of_property(pdev);
diff --git a/drivers/mmc/host/sdhci-pci-core.c b/drivers/mmc/host/sdhci-pci-core.c
index 86560d5..92fc3f7 100644
--- a/drivers/mmc/host/sdhci-pci-core.c
+++ b/drivers/mmc/host/sdhci-pci-core.c
@@ -12,6 +12,7 @@
* - JMicron (hardware and technical support)
*/
+#include <linux/string.h>
#include <linux/delay.h>
#include <linux/highmem.h>
#include <linux/module.h>
@@ -36,10 +37,138 @@
static int sdhci_pci_enable_dma(struct sdhci_host *host);
static void sdhci_pci_set_bus_width(struct sdhci_host *host, int width);
static void sdhci_pci_hw_reset(struct sdhci_host *host);
-static int sdhci_pci_select_drive_strength(struct sdhci_host *host,
- struct mmc_card *card,
- unsigned int max_dtr, int host_drv,
- int card_drv, int *drv_type);
+
+#ifdef CONFIG_PM_SLEEP
+static int __sdhci_pci_suspend_host(struct sdhci_pci_chip *chip)
+{
+ int i, ret;
+
+ for (i = 0; i < chip->num_slots; i++) {
+ struct sdhci_pci_slot *slot = chip->slots[i];
+ struct sdhci_host *host;
+
+ if (!slot)
+ continue;
+
+ host = slot->host;
+
+ if (chip->pm_retune && host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
+ ret = sdhci_suspend_host(host);
+ if (ret)
+ goto err_pci_suspend;
+
+ if (host->mmc->pm_flags & MMC_PM_WAKE_SDIO_IRQ)
+ sdhci_enable_irq_wakeups(host);
+ }
+
+ return 0;
+
+err_pci_suspend:
+ while (--i >= 0)
+ sdhci_resume_host(chip->slots[i]->host);
+ return ret;
+}
+
+static int sdhci_pci_init_wakeup(struct sdhci_pci_chip *chip)
+{
+ mmc_pm_flag_t pm_flags = 0;
+ int i;
+
+ for (i = 0; i < chip->num_slots; i++) {
+ struct sdhci_pci_slot *slot = chip->slots[i];
+
+ if (slot)
+ pm_flags |= slot->host->mmc->pm_flags;
+ }
+
+ return device_init_wakeup(&chip->pdev->dev,
+ (pm_flags & MMC_PM_KEEP_POWER) &&
+ (pm_flags & MMC_PM_WAKE_SDIO_IRQ));
+}
+
+static int sdhci_pci_suspend_host(struct sdhci_pci_chip *chip)
+{
+ int ret;
+
+ ret = __sdhci_pci_suspend_host(chip);
+ if (ret)
+ return ret;
+
+ sdhci_pci_init_wakeup(chip);
+
+ return 0;
+}
+
+int sdhci_pci_resume_host(struct sdhci_pci_chip *chip)
+{
+ struct sdhci_pci_slot *slot;
+ int i, ret;
+
+ for (i = 0; i < chip->num_slots; i++) {
+ slot = chip->slots[i];
+ if (!slot)
+ continue;
+
+ ret = sdhci_resume_host(slot->host);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+#endif
+
+#ifdef CONFIG_PM
+static int sdhci_pci_runtime_suspend_host(struct sdhci_pci_chip *chip)
+{
+ struct sdhci_pci_slot *slot;
+ struct sdhci_host *host;
+ int i, ret;
+
+ for (i = 0; i < chip->num_slots; i++) {
+ slot = chip->slots[i];
+ if (!slot)
+ continue;
+
+ host = slot->host;
+
+ ret = sdhci_runtime_suspend_host(host);
+ if (ret)
+ goto err_pci_runtime_suspend;
+
+ if (chip->rpm_retune &&
+ host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+ }
+
+ return 0;
+
+err_pci_runtime_suspend:
+ while (--i >= 0)
+ sdhci_runtime_resume_host(chip->slots[i]->host);
+ return ret;
+}
+
+static int sdhci_pci_runtime_resume_host(struct sdhci_pci_chip *chip)
+{
+ struct sdhci_pci_slot *slot;
+ int i, ret;
+
+ for (i = 0; i < chip->num_slots; i++) {
+ slot = chip->slots[i];
+ if (!slot)
+ continue;
+
+ ret = sdhci_runtime_resume_host(slot->host);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+#endif
/*****************************************************************************\
* *
@@ -71,14 +200,16 @@
return 0;
}
+#ifdef CONFIG_PM_SLEEP
static int ricoh_mmc_resume(struct sdhci_pci_chip *chip)
{
/* Apply a delay to allow controller to settle */
/* Otherwise it becomes confused if card state changed
during suspend */
msleep(500);
- return 0;
+ return sdhci_pci_resume_host(chip);
}
+#endif
static const struct sdhci_pci_fixes sdhci_ricoh = {
.probe = ricoh_probe,
@@ -89,7 +220,9 @@
static const struct sdhci_pci_fixes sdhci_ricoh_mmc = {
.probe_slot = ricoh_mmc_probe_slot,
+#ifdef CONFIG_PM_SLEEP
.resume = ricoh_mmc_resume,
+#endif
.quirks = SDHCI_QUIRK_32BIT_DMA_ADDR |
SDHCI_QUIRK_CLOCK_BEFORE_RESET |
SDHCI_QUIRK_NO_CARD_NO_RESET |
@@ -259,6 +392,81 @@
.probe_slot = pch_hc_probe_slot,
};
+enum {
+ INTEL_DSM_FNS = 0,
+ INTEL_DSM_DRV_STRENGTH = 9,
+ INTEL_DSM_D3_RETUNE = 10,
+};
+
+struct intel_host {
+ u32 dsm_fns;
+ int drv_strength;
+ bool d3_retune;
+};
+
+const u8 intel_dsm_uuid[] = {
+ 0xA5, 0x3E, 0xC1, 0xF6, 0xCD, 0x65, 0x1F, 0x46,
+ 0xAB, 0x7A, 0x29, 0xF7, 0xE8, 0xD5, 0xBD, 0x61,
+};
+
+static int __intel_dsm(struct intel_host *intel_host, struct device *dev,
+ unsigned int fn, u32 *result)
+{
+ union acpi_object *obj;
+ int err = 0;
+ size_t len;
+
+ obj = acpi_evaluate_dsm(ACPI_HANDLE(dev), intel_dsm_uuid, 0, fn, NULL);
+ if (!obj)
+ return -EOPNOTSUPP;
+
+ if (obj->type != ACPI_TYPE_BUFFER || obj->buffer.length < 1) {
+ err = -EINVAL;
+ goto out;
+ }
+
+ len = min_t(size_t, obj->buffer.length, 4);
+
+ *result = 0;
+ memcpy(result, obj->buffer.pointer, len);
+out:
+ ACPI_FREE(obj);
+
+ return err;
+}
+
+static int intel_dsm(struct intel_host *intel_host, struct device *dev,
+ unsigned int fn, u32 *result)
+{
+ if (fn > 31 || !(intel_host->dsm_fns & (1 << fn)))
+ return -EOPNOTSUPP;
+
+ return __intel_dsm(intel_host, dev, fn, result);
+}
+
+static void intel_dsm_init(struct intel_host *intel_host, struct device *dev,
+ struct mmc_host *mmc)
+{
+ int err;
+ u32 val;
+
+ err = __intel_dsm(intel_host, dev, INTEL_DSM_FNS, &intel_host->dsm_fns);
+ if (err) {
+ pr_debug("%s: DSM not supported, error %d\n",
+ mmc_hostname(mmc), err);
+ return;
+ }
+
+ pr_debug("%s: DSM function mask %#x\n",
+ mmc_hostname(mmc), intel_host->dsm_fns);
+
+ err = intel_dsm(intel_host, dev, INTEL_DSM_DRV_STRENGTH, &val);
+ intel_host->drv_strength = err ? 0 : val;
+
+ err = intel_dsm(intel_host, dev, INTEL_DSM_D3_RETUNE, &val);
+ intel_host->d3_retune = err ? true : !!val;
+}
+
static void sdhci_pci_int_hw_reset(struct sdhci_host *host)
{
u8 reg;
@@ -274,67 +482,15 @@
usleep_range(300, 1000);
}
-static int spt_select_drive_strength(struct sdhci_host *host,
- struct mmc_card *card,
- unsigned int max_dtr,
- int host_drv, int card_drv, int *drv_type)
+static int intel_select_drive_strength(struct mmc_card *card,
+ unsigned int max_dtr, int host_drv,
+ int card_drv, int *drv_type)
{
- int drive_strength;
+ struct sdhci_host *host = mmc_priv(card->host);
+ struct sdhci_pci_slot *slot = sdhci_priv(host);
+ struct intel_host *intel_host = sdhci_pci_priv(slot);
- if (sdhci_pci_spt_drive_strength > 0)
- drive_strength = sdhci_pci_spt_drive_strength & 0xf;
- else
- drive_strength = 0; /* Default 50-ohm */
-
- if ((mmc_driver_type_mask(drive_strength) & card_drv) == 0)
- drive_strength = 0; /* Default 50-ohm */
-
- return drive_strength;
-}
-
-/* Try to read the drive strength from the card */
-static void spt_read_drive_strength(struct sdhci_host *host)
-{
- u32 val, i, t;
- u16 m;
-
- if (sdhci_pci_spt_drive_strength)
- return;
-
- sdhci_pci_spt_drive_strength = -1;
-
- m = sdhci_readw(host, SDHCI_HOST_CONTROL2) & 0x7;
- if (m != 3 && m != 5)
- return;
- val = sdhci_readl(host, SDHCI_PRESENT_STATE);
- if (val & 0x3)
- return;
- sdhci_writel(host, 0x007f0023, SDHCI_INT_ENABLE);
- sdhci_writel(host, 0, SDHCI_SIGNAL_ENABLE);
- sdhci_writew(host, 0x10, SDHCI_TRANSFER_MODE);
- sdhci_writeb(host, 0xe, SDHCI_TIMEOUT_CONTROL);
- sdhci_writew(host, 512, SDHCI_BLOCK_SIZE);
- sdhci_writew(host, 1, SDHCI_BLOCK_COUNT);
- sdhci_writel(host, 0, SDHCI_ARGUMENT);
- sdhci_writew(host, 0x83b, SDHCI_COMMAND);
- for (i = 0; i < 1000; i++) {
- val = sdhci_readl(host, SDHCI_INT_STATUS);
- if (val & 0xffff8000)
- return;
- if (val & 0x20)
- break;
- udelay(1);
- }
- val = sdhci_readl(host, SDHCI_PRESENT_STATE);
- if (!(val & 0x800))
- return;
- for (i = 0; i < 47; i++)
- val = sdhci_readl(host, SDHCI_BUFFER);
- t = val & 0xf00;
- if (t != 0x200 && t != 0x300)
- return;
-
- sdhci_pci_spt_drive_strength = 0x10 | ((val >> 12) & 0xf);
+ return intel_host->drv_strength;
}
static int bxt_get_cd(struct mmc_host *mmc)
@@ -359,8 +515,57 @@
return ret;
}
+#define SDHCI_INTEL_PWR_TIMEOUT_CNT 20
+#define SDHCI_INTEL_PWR_TIMEOUT_UDELAY 100
+
+static void sdhci_intel_set_power(struct sdhci_host *host, unsigned char mode,
+ unsigned short vdd)
+{
+ int cntr;
+ u8 reg;
+
+ sdhci_set_power(host, mode, vdd);
+
+ if (mode == MMC_POWER_OFF)
+ return;
+
+ /*
+ * Bus power might not enable after D3 -> D0 transition due to the
+ * present state not yet having propagated. Retry for up to 2ms.
+ */
+ for (cntr = 0; cntr < SDHCI_INTEL_PWR_TIMEOUT_CNT; cntr++) {
+ reg = sdhci_readb(host, SDHCI_POWER_CONTROL);
+ if (reg & SDHCI_POWER_ON)
+ break;
+ udelay(SDHCI_INTEL_PWR_TIMEOUT_UDELAY);
+ reg |= SDHCI_POWER_ON;
+ sdhci_writeb(host, reg, SDHCI_POWER_CONTROL);
+ }
+}
+
+static const struct sdhci_ops sdhci_intel_byt_ops = {
+ .set_clock = sdhci_set_clock,
+ .set_power = sdhci_intel_set_power,
+ .enable_dma = sdhci_pci_enable_dma,
+ .set_bus_width = sdhci_pci_set_bus_width,
+ .reset = sdhci_reset,
+ .set_uhs_signaling = sdhci_set_uhs_signaling,
+ .hw_reset = sdhci_pci_hw_reset,
+};
+
+static void byt_read_dsm(struct sdhci_pci_slot *slot)
+{
+ struct intel_host *intel_host = sdhci_pci_priv(slot);
+ struct device *dev = &slot->chip->pdev->dev;
+ struct mmc_host *mmc = slot->host->mmc;
+
+ intel_dsm_init(intel_host, dev, mmc);
+ slot->chip->rpm_retune = intel_host->d3_retune;
+}
+
static int byt_emmc_probe_slot(struct sdhci_pci_slot *slot)
{
+ byt_read_dsm(slot);
slot->host->mmc->caps |= MMC_CAP_8_BIT_DATA | MMC_CAP_NONREMOVABLE |
MMC_CAP_HW_RESET | MMC_CAP_1_8V_DDR |
MMC_CAP_CMD_DURING_TFR |
@@ -369,10 +574,8 @@
slot->hw_reset = sdhci_pci_int_hw_reset;
if (slot->chip->pdev->device == PCI_DEVICE_ID_INTEL_BSW_EMMC)
slot->host->timeout_clk = 1000; /* 1000 kHz i.e. 1 MHz */
- if (slot->chip->pdev->device == PCI_DEVICE_ID_INTEL_SPT_EMMC) {
- spt_read_drive_strength(slot->host);
- slot->select_drive_strength = spt_select_drive_strength;
- }
+ slot->host->mmc_host_ops.select_drive_strength =
+ intel_select_drive_strength;
return 0;
}
@@ -405,6 +608,8 @@
{
int err;
+ byt_read_dsm(slot);
+
err = ni_set_max_freq(slot);
if (err)
return err;
@@ -416,6 +621,7 @@
static int byt_sdio_probe_slot(struct sdhci_pci_slot *slot)
{
+ byt_read_dsm(slot);
slot->host->mmc->caps |= MMC_CAP_POWER_OFF_CARD | MMC_CAP_NONREMOVABLE |
MMC_CAP_WAIT_WHILE_BUSY;
return 0;
@@ -423,63 +629,20 @@
static int byt_sd_probe_slot(struct sdhci_pci_slot *slot)
{
- slot->host->mmc->caps |= MMC_CAP_WAIT_WHILE_BUSY;
+ byt_read_dsm(slot);
+ slot->host->mmc->caps |= MMC_CAP_WAIT_WHILE_BUSY |
+ MMC_CAP_AGGRESSIVE_PM;
slot->cd_idx = 0;
slot->cd_override_level = true;
if (slot->chip->pdev->device == PCI_DEVICE_ID_INTEL_BXT_SD ||
slot->chip->pdev->device == PCI_DEVICE_ID_INTEL_BXTM_SD ||
slot->chip->pdev->device == PCI_DEVICE_ID_INTEL_APL_SD ||
- slot->chip->pdev->device == PCI_DEVICE_ID_INTEL_GLK_SD) {
+ slot->chip->pdev->device == PCI_DEVICE_ID_INTEL_GLK_SD)
slot->host->mmc_host_ops.get_cd = bxt_get_cd;
- slot->host->mmc->caps |= MMC_CAP_AGGRESSIVE_PM;
- }
return 0;
}
-#define SDHCI_INTEL_PWR_TIMEOUT_CNT 20
-#define SDHCI_INTEL_PWR_TIMEOUT_UDELAY 100
-
-static void sdhci_intel_set_power(struct sdhci_host *host, unsigned char mode,
- unsigned short vdd)
-{
- int cntr;
- u8 reg;
-
- sdhci_set_power(host, mode, vdd);
-
- if (mode == MMC_POWER_OFF)
- return;
-
- spin_unlock_irq(&host->lock);
-
- /*
- * Bus power might not enable after D3 -> D0 transition due to the
- * present state not yet having propagated. Retry for up to 2ms.
- */
- for (cntr = 0; cntr < SDHCI_INTEL_PWR_TIMEOUT_CNT; cntr++) {
- reg = sdhci_readb(host, SDHCI_POWER_CONTROL);
- if (reg & SDHCI_POWER_ON)
- break;
- udelay(SDHCI_INTEL_PWR_TIMEOUT_UDELAY);
- reg |= SDHCI_POWER_ON;
- sdhci_writeb(host, reg, SDHCI_POWER_CONTROL);
- }
-
- spin_lock_irq(&host->lock);
-}
-
-static const struct sdhci_ops sdhci_intel_byt_ops = {
- .set_clock = sdhci_set_clock,
- .set_power = sdhci_intel_set_power,
- .enable_dma = sdhci_pci_enable_dma,
- .set_bus_width = sdhci_pci_set_bus_width,
- .reset = sdhci_reset,
- .set_uhs_signaling = sdhci_set_uhs_signaling,
- .hw_reset = sdhci_pci_hw_reset,
- .select_drive_strength = sdhci_pci_select_drive_strength,
-};
-
static const struct sdhci_pci_fixes sdhci_intel_byt_emmc = {
.allow_runtime_pm = true,
.probe_slot = byt_emmc_probe_slot,
@@ -488,6 +651,7 @@
SDHCI_QUIRK2_CAPS_BIT63_FOR_HS400 |
SDHCI_QUIRK2_STOP_WITH_TC,
.ops = &sdhci_intel_byt_ops,
+ .priv_size = sizeof(struct intel_host),
};
static const struct sdhci_pci_fixes sdhci_ni_byt_sdio = {
@@ -497,6 +661,7 @@
.allow_runtime_pm = true,
.probe_slot = ni_byt_sdio_probe_slot,
.ops = &sdhci_intel_byt_ops,
+ .priv_size = sizeof(struct intel_host),
};
static const struct sdhci_pci_fixes sdhci_intel_byt_sdio = {
@@ -506,6 +671,7 @@
.allow_runtime_pm = true,
.probe_slot = byt_sdio_probe_slot,
.ops = &sdhci_intel_byt_ops,
+ .priv_size = sizeof(struct intel_host),
};
static const struct sdhci_pci_fixes sdhci_intel_byt_sd = {
@@ -517,6 +683,7 @@
.own_cd_for_runtime_pm = true,
.probe_slot = byt_sd_probe_slot,
.ops = &sdhci_intel_byt_ops,
+ .priv_size = sizeof(struct intel_host),
};
/* Define Host controllers for Intel Merrifield platform */
@@ -719,9 +886,14 @@
jmicron_enable_mmc(slot->host, 0);
}
+#ifdef CONFIG_PM_SLEEP
static int jmicron_suspend(struct sdhci_pci_chip *chip)
{
- int i;
+ int i, ret;
+
+ ret = __sdhci_pci_suspend_host(chip);
+ if (ret)
+ return ret;
if (chip->pdev->device == PCI_DEVICE_ID_JMICRON_JMB38X_MMC ||
chip->pdev->device == PCI_DEVICE_ID_JMICRON_JMB388_ESD) {
@@ -729,6 +901,8 @@
jmicron_enable_mmc(chip->slots[i]->host, 0);
}
+ sdhci_pci_init_wakeup(chip);
+
return 0;
}
@@ -748,15 +922,18 @@
return ret;
}
- return 0;
+ return sdhci_pci_resume_host(chip);
}
+#endif
static const struct sdhci_pci_fixes sdhci_o2 = {
.probe = sdhci_pci_o2_probe,
.quirks = SDHCI_QUIRK_NO_ENDATTR_IN_NOPDESC,
.quirks2 = SDHCI_QUIRK2_CLEAR_TRANSFERMODE_REG_BEFORE_CMD,
.probe_slot = sdhci_pci_o2_probe_slot,
+#ifdef CONFIG_PM_SLEEP
.resume = sdhci_pci_o2_resume,
+#endif
};
static const struct sdhci_pci_fixes sdhci_jmicron = {
@@ -765,8 +942,10 @@
.probe_slot = jmicron_probe_slot,
.remove_slot = jmicron_remove_slot,
+#ifdef CONFIG_PM_SLEEP
.suspend = jmicron_suspend,
.resume = jmicron_resume,
+#endif
};
/* SysKonnect CardBus2SDIO extra registers */
@@ -1617,20 +1796,6 @@
slot->hw_reset(host);
}
-static int sdhci_pci_select_drive_strength(struct sdhci_host *host,
- struct mmc_card *card,
- unsigned int max_dtr, int host_drv,
- int card_drv, int *drv_type)
-{
- struct sdhci_pci_slot *slot = sdhci_priv(host);
-
- if (!slot->select_drive_strength)
- return 0;
-
- return slot->select_drive_strength(host, card, max_dtr, host_drv,
- card_drv, drv_type);
-}
-
static const struct sdhci_ops sdhci_pci_ops = {
.set_clock = sdhci_set_clock,
.enable_dma = sdhci_pci_enable_dma,
@@ -1638,7 +1803,6 @@
.reset = sdhci_reset,
.set_uhs_signaling = sdhci_set_uhs_signaling,
.hw_reset = sdhci_pci_hw_reset,
- .select_drive_strength = sdhci_pci_select_drive_strength,
};
/*****************************************************************************\
@@ -1651,83 +1815,29 @@
static int sdhci_pci_suspend(struct device *dev)
{
struct pci_dev *pdev = to_pci_dev(dev);
- struct sdhci_pci_chip *chip;
- struct sdhci_pci_slot *slot;
- mmc_pm_flag_t slot_pm_flags;
- mmc_pm_flag_t pm_flags = 0;
- int i, ret;
+ struct sdhci_pci_chip *chip = pci_get_drvdata(pdev);
- chip = pci_get_drvdata(pdev);
if (!chip)
return 0;
- for (i = 0; i < chip->num_slots; i++) {
- slot = chip->slots[i];
- if (!slot)
- continue;
+ if (chip->fixes && chip->fixes->suspend)
+ return chip->fixes->suspend(chip);
- ret = sdhci_suspend_host(slot->host);
-
- if (ret)
- goto err_pci_suspend;
-
- slot_pm_flags = slot->host->mmc->pm_flags;
- if (slot_pm_flags & MMC_PM_WAKE_SDIO_IRQ)
- sdhci_enable_irq_wakeups(slot->host);
-
- pm_flags |= slot_pm_flags;
- }
-
- if (chip->fixes && chip->fixes->suspend) {
- ret = chip->fixes->suspend(chip);
- if (ret)
- goto err_pci_suspend;
- }
-
- if (pm_flags & MMC_PM_KEEP_POWER) {
- if (pm_flags & MMC_PM_WAKE_SDIO_IRQ)
- device_init_wakeup(dev, true);
- else
- device_init_wakeup(dev, false);
- } else
- device_init_wakeup(dev, false);
-
- return 0;
-
-err_pci_suspend:
- while (--i >= 0)
- sdhci_resume_host(chip->slots[i]->host);
- return ret;
+ return sdhci_pci_suspend_host(chip);
}
static int sdhci_pci_resume(struct device *dev)
{
struct pci_dev *pdev = to_pci_dev(dev);
- struct sdhci_pci_chip *chip;
- struct sdhci_pci_slot *slot;
- int i, ret;
+ struct sdhci_pci_chip *chip = pci_get_drvdata(pdev);
- chip = pci_get_drvdata(pdev);
if (!chip)
return 0;
- if (chip->fixes && chip->fixes->resume) {
- ret = chip->fixes->resume(chip);
- if (ret)
- return ret;
- }
+ if (chip->fixes && chip->fixes->resume)
+ return chip->fixes->resume(chip);
- for (i = 0; i < chip->num_slots; i++) {
- slot = chip->slots[i];
- if (!slot)
- continue;
-
- ret = sdhci_resume_host(slot->host);
- if (ret)
- return ret;
- }
-
- return 0;
+ return sdhci_pci_resume_host(chip);
}
#endif
@@ -1735,67 +1845,29 @@
static int sdhci_pci_runtime_suspend(struct device *dev)
{
struct pci_dev *pdev = to_pci_dev(dev);
- struct sdhci_pci_chip *chip;
- struct sdhci_pci_slot *slot;
- int i, ret;
+ struct sdhci_pci_chip *chip = pci_get_drvdata(pdev);
- chip = pci_get_drvdata(pdev);
if (!chip)
return 0;
- for (i = 0; i < chip->num_slots; i++) {
- slot = chip->slots[i];
- if (!slot)
- continue;
+ if (chip->fixes && chip->fixes->runtime_suspend)
+ return chip->fixes->runtime_suspend(chip);
- ret = sdhci_runtime_suspend_host(slot->host);
-
- if (ret)
- goto err_pci_runtime_suspend;
- }
-
- if (chip->fixes && chip->fixes->suspend) {
- ret = chip->fixes->suspend(chip);
- if (ret)
- goto err_pci_runtime_suspend;
- }
-
- return 0;
-
-err_pci_runtime_suspend:
- while (--i >= 0)
- sdhci_runtime_resume_host(chip->slots[i]->host);
- return ret;
+ return sdhci_pci_runtime_suspend_host(chip);
}
static int sdhci_pci_runtime_resume(struct device *dev)
{
struct pci_dev *pdev = to_pci_dev(dev);
- struct sdhci_pci_chip *chip;
- struct sdhci_pci_slot *slot;
- int i, ret;
+ struct sdhci_pci_chip *chip = pci_get_drvdata(pdev);
- chip = pci_get_drvdata(pdev);
if (!chip)
return 0;
- if (chip->fixes && chip->fixes->resume) {
- ret = chip->fixes->resume(chip);
- if (ret)
- return ret;
- }
+ if (chip->fixes && chip->fixes->runtime_resume)
+ return chip->fixes->runtime_resume(chip);
- for (i = 0; i < chip->num_slots; i++) {
- slot = chip->slots[i];
- if (!slot)
- continue;
-
- ret = sdhci_runtime_resume_host(slot->host);
- if (ret)
- return ret;
- }
-
- return 0;
+ return sdhci_pci_runtime_resume_host(chip);
}
#endif
@@ -1818,6 +1890,7 @@
struct sdhci_pci_slot *slot;
struct sdhci_host *host;
int ret, bar = first_bar + slotno;
+ size_t priv_size = chip->fixes ? chip->fixes->priv_size : 0;
if (!(pci_resource_flags(pdev, bar) & IORESOURCE_MEM)) {
dev_err(&pdev->dev, "BAR %d is not iomem. Aborting.\n", bar);
@@ -1839,7 +1912,7 @@
return ERR_PTR(-ENODEV);
}
- host = sdhci_alloc_host(&pdev->dev, sizeof(struct sdhci_pci_slot));
+ host = sdhci_alloc_host(&pdev->dev, sizeof(*slot) + priv_size);
if (IS_ERR(host)) {
dev_err(&pdev->dev, "cannot allocate host\n");
return ERR_CAST(host);
@@ -1919,7 +1992,10 @@
}
}
- ret = sdhci_add_host(host);
+ if (chip->fixes && chip->fixes->add_host)
+ ret = chip->fixes->add_host(slot);
+ else
+ ret = sdhci_add_host(host);
if (ret)
goto remove;
@@ -2042,6 +2118,8 @@
chip->allow_runtime_pm = chip->fixes->allow_runtime_pm;
}
chip->num_slots = slots;
+ chip->pm_retune = true;
+ chip->rpm_retune = true;
pci_set_drvdata(pdev, chip);
diff --git a/drivers/mmc/host/sdhci-pci-data.c b/drivers/mmc/host/sdhci-pci-data.c
index 56fddc6..a611217 100644
--- a/drivers/mmc/host/sdhci-pci-data.c
+++ b/drivers/mmc/host/sdhci-pci-data.c
@@ -3,6 +3,3 @@
struct sdhci_pci_data *(*sdhci_pci_get_data)(struct pci_dev *pdev, int slotno);
EXPORT_SYMBOL_GPL(sdhci_pci_get_data);
-
-int sdhci_pci_spt_drive_strength;
-EXPORT_SYMBOL_GPL(sdhci_pci_spt_drive_strength);
diff --git a/drivers/mmc/host/sdhci-pci-o2micro.c b/drivers/mmc/host/sdhci-pci-o2micro.c
index d48f031..14273ca 100644
--- a/drivers/mmc/host/sdhci-pci-o2micro.c
+++ b/drivers/mmc/host/sdhci-pci-o2micro.c
@@ -384,8 +384,10 @@
return 0;
}
+#ifdef CONFIG_PM_SLEEP
int sdhci_pci_o2_resume(struct sdhci_pci_chip *chip)
{
sdhci_pci_o2_probe(chip);
- return 0;
+ return sdhci_pci_resume_host(chip);
}
+#endif
diff --git a/drivers/mmc/host/sdhci-pci.h b/drivers/mmc/host/sdhci-pci.h
index 36f7434..37766d20 100644
--- a/drivers/mmc/host/sdhci-pci.h
+++ b/drivers/mmc/host/sdhci-pci.h
@@ -64,12 +64,20 @@
int (*probe) (struct sdhci_pci_chip *);
int (*probe_slot) (struct sdhci_pci_slot *);
+ int (*add_host) (struct sdhci_pci_slot *);
void (*remove_slot) (struct sdhci_pci_slot *, int);
+#ifdef CONFIG_PM_SLEEP
int (*suspend) (struct sdhci_pci_chip *);
int (*resume) (struct sdhci_pci_chip *);
+#endif
+#ifdef CONFIG_PM
+ int (*runtime_suspend) (struct sdhci_pci_chip *);
+ int (*runtime_resume) (struct sdhci_pci_chip *);
+#endif
const struct sdhci_ops *ops;
+ size_t priv_size;
};
struct sdhci_pci_slot {
@@ -85,10 +93,7 @@
bool cd_override_level;
void (*hw_reset)(struct sdhci_host *host);
- int (*select_drive_strength)(struct sdhci_host *host,
- struct mmc_card *card,
- unsigned int max_dtr, int host_drv,
- int card_drv, int *drv_type);
+ unsigned long private[0] ____cacheline_aligned;
};
struct sdhci_pci_chip {
@@ -97,10 +102,21 @@
unsigned int quirks;
unsigned int quirks2;
bool allow_runtime_pm;
+ bool pm_retune;
+ bool rpm_retune;
const struct sdhci_pci_fixes *fixes;
int num_slots; /* Slots on controller */
struct sdhci_pci_slot *slots[MAX_SLOTS]; /* Pointers to host slots */
};
+static inline void *sdhci_pci_priv(struct sdhci_pci_slot *slot)
+{
+ return (void *)slot->private;
+}
+
+#ifdef CONFIG_PM_SLEEP
+int sdhci_pci_resume_host(struct sdhci_pci_chip *chip);
+#endif
+
#endif /* __SDHCI_PCI_H */
diff --git a/drivers/mmc/host/sdhci-pltfm.c b/drivers/mmc/host/sdhci-pltfm.c
index ad49bfa..e090d8c 100644
--- a/drivers/mmc/host/sdhci-pltfm.c
+++ b/drivers/mmc/host/sdhci-pltfm.c
@@ -213,6 +213,9 @@
{
struct sdhci_host *host = dev_get_drvdata(dev);
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
return sdhci_suspend_host(host);
}
diff --git a/drivers/mmc/host/sdhci-pxav2.c b/drivers/mmc/host/sdhci-pxav2.c
index 347eae2..995083c 100644
--- a/drivers/mmc/host/sdhci-pxav2.c
+++ b/drivers/mmc/host/sdhci-pxav2.c
@@ -185,7 +185,11 @@
goto err_clk_get;
}
pltfm_host->clk = clk;
- clk_prepare_enable(clk);
+ ret = clk_prepare_enable(clk);
+ if (ret) {
+ dev_err(&pdev->dev, "failed to enable io clock\n");
+ goto err_clk_enable;
+ }
host->quirks = SDHCI_QUIRK_BROKEN_ADMA
| SDHCI_QUIRK_BROKEN_TIMEOUT_VAL
@@ -222,12 +226,11 @@
goto err_add_host;
}
- platform_set_drvdata(pdev, host);
-
return 0;
err_add_host:
clk_disable_unprepare(clk);
+err_clk_enable:
clk_put(clk);
err_clk_get:
sdhci_pltfm_free(pdev);
diff --git a/drivers/mmc/host/sdhci-pxav3.c b/drivers/mmc/host/sdhci-pxav3.c
index d0f5c05..f953f35 100644
--- a/drivers/mmc/host/sdhci-pxav3.c
+++ b/drivers/mmc/host/sdhci-pxav3.c
@@ -323,11 +323,8 @@
if (host->pwr == 0)
vdd = 0;
- if (!IS_ERR(mmc->supply.vmmc)) {
- spin_unlock_irq(&host->lock);
+ if (!IS_ERR(mmc->supply.vmmc))
mmc_regulator_set_ocr(mmc, mmc->supply.vmmc, vdd);
- spin_lock_irq(&host->lock);
- }
}
static const struct sdhci_ops pxav3_sdhci_ops = {
@@ -480,8 +477,6 @@
goto err_add_host;
}
- platform_set_drvdata(pdev, host);
-
if (host->mmc->pm_caps & MMC_PM_WAKE_SDIO_IRQ)
device_init_wakeup(&pdev->dev, 1);
@@ -529,6 +524,8 @@
struct sdhci_host *host = dev_get_drvdata(dev);
pm_runtime_get_sync(dev);
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
ret = sdhci_suspend_host(host);
pm_runtime_mark_last_busy(dev);
pm_runtime_put_autosuspend(dev);
@@ -562,6 +559,9 @@
if (ret)
return ret;
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
clk_disable_unprepare(pxa->clk_io);
if (!IS_ERR(pxa->clk_core))
clk_disable_unprepare(pxa->clk_core);
diff --git a/drivers/mmc/host/sdhci-s3c.c b/drivers/mmc/host/sdhci-s3c.c
index 3e5c83d..7c065a7 100644
--- a/drivers/mmc/host/sdhci-s3c.c
+++ b/drivers/mmc/host/sdhci-s3c.c
@@ -190,9 +190,7 @@
* speed possible with selected clock source and skip the division.
*/
if (ourhost->no_divider) {
- spin_unlock_irq(&ourhost->host->lock);
rate = clk_round_rate(clksrc, wanted);
- spin_lock_irq(&ourhost->host->lock);
return wanted - rate;
}
@@ -389,9 +387,7 @@
clk &= ~SDHCI_CLOCK_CARD_EN;
sdhci_writew(host, clk, SDHCI_CLOCK_CONTROL);
- spin_unlock_irq(&host->lock);
ret = clk_set_rate(ourhost->clk_bus[ourhost->cur_clk], clock);
- spin_lock_irq(&host->lock);
if (ret != 0) {
dev_err(dev, "%s: failed to set clock rate %uHz\n",
mmc_hostname(host->mmc), clock);
@@ -743,6 +739,9 @@
{
struct sdhci_host *host = dev_get_drvdata(dev);
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
return sdhci_suspend_host(host);
}
@@ -764,6 +763,9 @@
ret = sdhci_runtime_suspend_host(host);
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
if (ourhost->cur_clk >= 0)
clk_disable_unprepare(ourhost->clk_bus[ourhost->cur_clk]);
clk_disable_unprepare(busclk);
diff --git a/drivers/mmc/host/sdhci-sirf.c b/drivers/mmc/host/sdhci-sirf.c
index 5d06863..c251c6c 100644
--- a/drivers/mmc/host/sdhci-sirf.c
+++ b/drivers/mmc/host/sdhci-sirf.c
@@ -237,6 +237,9 @@
struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
int ret;
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
ret = sdhci_suspend_host(host);
if (ret)
return ret;
diff --git a/drivers/mmc/host/sdhci-spear.c b/drivers/mmc/host/sdhci-spear.c
index 255a896..8c0f884 100644
--- a/drivers/mmc/host/sdhci-spear.c
+++ b/drivers/mmc/host/sdhci-spear.c
@@ -165,6 +165,9 @@
struct spear_sdhci *sdhci = sdhci_priv(host);
int ret;
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
ret = sdhci_suspend_host(host);
if (!ret)
clk_disable(sdhci->clk);
diff --git a/drivers/mmc/host/sdhci-st.c b/drivers/mmc/host/sdhci-st.c
index ed92ce72..68c36c9f 100644
--- a/drivers/mmc/host/sdhci-st.c
+++ b/drivers/mmc/host/sdhci-st.c
@@ -418,8 +418,6 @@
goto err_out;
}
- platform_set_drvdata(pdev, host);
-
host_version = readw_relaxed((host->ioaddr + SDHCI_HOST_VERSION));
dev_info(&pdev->dev, "SDHCI ST Initialised: Host Version: 0x%x Vendor Version 0x%x\n",
@@ -465,8 +463,12 @@
struct sdhci_host *host = dev_get_drvdata(dev);
struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
struct st_mmc_platform_data *pdata = sdhci_pltfm_priv(pltfm_host);
- int ret = sdhci_suspend_host(host);
+ int ret;
+ if (host->tuning_mode != SDHCI_TUNING_MODE_3)
+ mmc_retune_needed(host->mmc);
+
+ ret = sdhci_suspend_host(host);
if (ret)
goto out;
diff --git a/drivers/mmc/host/sdhci-tegra.c b/drivers/mmc/host/sdhci-tegra.c
index 20b6ff5..7f93079 100644
--- a/drivers/mmc/host/sdhci-tegra.c
+++ b/drivers/mmc/host/sdhci-tegra.c
@@ -21,6 +21,7 @@
#include <linux/io.h>
#include <linux/of.h>
#include <linux/of_device.h>
+#include <linux/reset.h>
#include <linux/mmc/card.h>
#include <linux/mmc/host.h>
#include <linux/mmc/mmc.h>
@@ -65,6 +66,8 @@
struct gpio_desc *power_gpio;
bool ddr_signaling;
bool pad_calib_required;
+
+ struct reset_control *rst;
};
static u16 tegra_sdhci_readw(struct sdhci_host *host, int reg)
@@ -431,7 +434,23 @@
.pdata = &sdhci_tegra210_pdata,
};
+static const struct sdhci_pltfm_data sdhci_tegra186_pdata = {
+ .quirks = SDHCI_QUIRK_BROKEN_TIMEOUT_VAL |
+ SDHCI_QUIRK_DATA_TIMEOUT_USES_SDCLK |
+ SDHCI_QUIRK_SINGLE_POWER_WRITE |
+ SDHCI_QUIRK_NO_HISPD_BIT |
+ SDHCI_QUIRK_BROKEN_ADMA_ZEROLEN_DESC |
+ SDHCI_QUIRK_CAP_CLOCK_BASE_BROKEN,
+ .quirks2 = SDHCI_QUIRK2_PRESET_VALUE_BROKEN,
+ .ops = &tegra114_sdhci_ops,
+};
+
+static const struct sdhci_tegra_soc_data soc_data_tegra186 = {
+ .pdata = &sdhci_tegra186_pdata,
+};
+
static const struct of_device_id sdhci_tegra_dt_match[] = {
+ { .compatible = "nvidia,tegra186-sdhci", .data = &soc_data_tegra186 },
{ .compatible = "nvidia,tegra210-sdhci", .data = &soc_data_tegra210 },
{ .compatible = "nvidia,tegra124-sdhci", .data = &soc_data_tegra124 },
{ .compatible = "nvidia,tegra114-sdhci", .data = &soc_data_tegra114 },
@@ -489,6 +508,25 @@
clk_prepare_enable(clk);
pltfm_host->clk = clk;
+ tegra_host->rst = devm_reset_control_get(&pdev->dev, "sdhci");
+ if (IS_ERR(tegra_host->rst)) {
+ rc = PTR_ERR(tegra_host->rst);
+ dev_err(&pdev->dev, "failed to get reset control: %d\n", rc);
+ goto err_rst_get;
+ }
+
+ rc = reset_control_assert(tegra_host->rst);
+ if (rc)
+ goto err_rst_get;
+
+ usleep_range(2000, 4000);
+
+ rc = reset_control_deassert(tegra_host->rst);
+ if (rc)
+ goto err_rst_get;
+
+ usleep_range(2000, 4000);
+
rc = sdhci_add_host(host);
if (rc)
goto err_add_host;
@@ -496,6 +534,8 @@
return 0;
err_add_host:
+ reset_control_assert(tegra_host->rst);
+err_rst_get:
clk_disable_unprepare(pltfm_host->clk);
err_clk_get:
err_power_req:
@@ -504,6 +544,23 @@
return rc;
}
+static int sdhci_tegra_remove(struct platform_device *pdev)
+{
+ struct sdhci_host *host = platform_get_drvdata(pdev);
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct sdhci_tegra *tegra_host = sdhci_pltfm_priv(pltfm_host);
+
+ sdhci_remove_host(host, 0);
+
+ reset_control_assert(tegra_host->rst);
+ usleep_range(2000, 4000);
+ clk_disable_unprepare(pltfm_host->clk);
+
+ sdhci_pltfm_free(pdev);
+
+ return 0;
+}
+
static struct platform_driver sdhci_tegra_driver = {
.driver = {
.name = "sdhci-tegra",
@@ -511,7 +568,7 @@
.pm = &sdhci_pltfm_pmops,
},
.probe = sdhci_tegra_probe,
- .remove = sdhci_pltfm_unregister,
+ .remove = sdhci_tegra_remove,
};
module_platform_driver(sdhci_tegra_driver);
diff --git a/drivers/mmc/host/sdhci-xenon-phy.c b/drivers/mmc/host/sdhci-xenon-phy.c
new file mode 100644
index 0000000..6356781
--- /dev/null
+++ b/drivers/mmc/host/sdhci-xenon-phy.c
@@ -0,0 +1,837 @@
+/*
+ * PHY support for Xenon SDHC
+ *
+ * Copyright (C) 2016 Marvell, All Rights Reserved.
+ *
+ * Author: Hu Ziji <huziji@marvell.com>
+ * Date: 2016-8-24
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation version 2.
+ */
+
+#include <linux/slab.h>
+#include <linux/delay.h>
+#include <linux/ktime.h>
+#include <linux/of_address.h>
+
+#include "sdhci-pltfm.h"
+#include "sdhci-xenon.h"
+
+/* Register base for eMMC PHY 5.0 Version */
+#define XENON_EMMC_5_0_PHY_REG_BASE 0x0160
+/* Register base for eMMC PHY 5.1 Version */
+#define XENON_EMMC_PHY_REG_BASE 0x0170
+
+#define XENON_EMMC_PHY_TIMING_ADJUST XENON_EMMC_PHY_REG_BASE
+#define XENON_EMMC_5_0_PHY_TIMING_ADJUST XENON_EMMC_5_0_PHY_REG_BASE
+#define XENON_TIMING_ADJUST_SLOW_MODE BIT(29)
+#define XENON_TIMING_ADJUST_SDIO_MODE BIT(28)
+#define XENON_SAMPL_INV_QSP_PHASE_SELECT BIT(18)
+#define XENON_SAMPL_INV_QSP_PHASE_SELECT_SHIFT 18
+#define XENON_PHY_INITIALIZAION BIT(31)
+#define XENON_WAIT_CYCLE_BEFORE_USING_MASK 0xF
+#define XENON_WAIT_CYCLE_BEFORE_USING_SHIFT 12
+#define XENON_FC_SYNC_EN_DURATION_MASK 0xF
+#define XENON_FC_SYNC_EN_DURATION_SHIFT 8
+#define XENON_FC_SYNC_RST_EN_DURATION_MASK 0xF
+#define XENON_FC_SYNC_RST_EN_DURATION_SHIFT 4
+#define XENON_FC_SYNC_RST_DURATION_MASK 0xF
+#define XENON_FC_SYNC_RST_DURATION_SHIFT 0
+
+#define XENON_EMMC_PHY_FUNC_CONTROL (XENON_EMMC_PHY_REG_BASE + 0x4)
+#define XENON_EMMC_5_0_PHY_FUNC_CONTROL \
+ (XENON_EMMC_5_0_PHY_REG_BASE + 0x4)
+#define XENON_ASYNC_DDRMODE_MASK BIT(23)
+#define XENON_ASYNC_DDRMODE_SHIFT 23
+#define XENON_CMD_DDR_MODE BIT(16)
+#define XENON_DQ_DDR_MODE_SHIFT 8
+#define XENON_DQ_DDR_MODE_MASK 0xFF
+#define XENON_DQ_ASYNC_MODE BIT(4)
+
+#define XENON_EMMC_PHY_PAD_CONTROL (XENON_EMMC_PHY_REG_BASE + 0x8)
+#define XENON_EMMC_5_0_PHY_PAD_CONTROL \
+ (XENON_EMMC_5_0_PHY_REG_BASE + 0x8)
+#define XENON_REC_EN_SHIFT 24
+#define XENON_REC_EN_MASK 0xF
+#define XENON_FC_DQ_RECEN BIT(24)
+#define XENON_FC_CMD_RECEN BIT(25)
+#define XENON_FC_QSP_RECEN BIT(26)
+#define XENON_FC_QSN_RECEN BIT(27)
+#define XENON_OEN_QSN BIT(28)
+#define XENON_AUTO_RECEN_CTRL BIT(30)
+#define XENON_FC_ALL_CMOS_RECEIVER 0xF000
+
+#define XENON_EMMC5_FC_QSP_PD BIT(18)
+#define XENON_EMMC5_FC_QSP_PU BIT(22)
+#define XENON_EMMC5_FC_CMD_PD BIT(17)
+#define XENON_EMMC5_FC_CMD_PU BIT(21)
+#define XENON_EMMC5_FC_DQ_PD BIT(16)
+#define XENON_EMMC5_FC_DQ_PU BIT(20)
+
+#define XENON_EMMC_PHY_PAD_CONTROL1 (XENON_EMMC_PHY_REG_BASE + 0xC)
+#define XENON_EMMC5_1_FC_QSP_PD BIT(9)
+#define XENON_EMMC5_1_FC_QSP_PU BIT(25)
+#define XENON_EMMC5_1_FC_CMD_PD BIT(8)
+#define XENON_EMMC5_1_FC_CMD_PU BIT(24)
+#define XENON_EMMC5_1_FC_DQ_PD 0xFF
+#define XENON_EMMC5_1_FC_DQ_PU (0xFF << 16)
+
+#define XENON_EMMC_PHY_PAD_CONTROL2 (XENON_EMMC_PHY_REG_BASE + 0x10)
+#define XENON_EMMC_5_0_PHY_PAD_CONTROL2 \
+ (XENON_EMMC_5_0_PHY_REG_BASE + 0xC)
+#define XENON_ZNR_MASK 0x1F
+#define XENON_ZNR_SHIFT 8
+#define XENON_ZPR_MASK 0x1F
+/* Preferred ZNR and ZPR value vary between different boards.
+ * The specific ZNR and ZPR value should be defined here
+ * according to board actual timing.
+ */
+#define XENON_ZNR_DEF_VALUE 0xF
+#define XENON_ZPR_DEF_VALUE 0xF
+
+#define XENON_EMMC_PHY_DLL_CONTROL (XENON_EMMC_PHY_REG_BASE + 0x14)
+#define XENON_EMMC_5_0_PHY_DLL_CONTROL \
+ (XENON_EMMC_5_0_PHY_REG_BASE + 0x10)
+#define XENON_DLL_ENABLE BIT(31)
+#define XENON_DLL_UPDATE_STROBE_5_0 BIT(30)
+#define XENON_DLL_REFCLK_SEL BIT(30)
+#define XENON_DLL_UPDATE BIT(23)
+#define XENON_DLL_PHSEL1_SHIFT 24
+#define XENON_DLL_PHSEL0_SHIFT 16
+#define XENON_DLL_PHASE_MASK 0x3F
+#define XENON_DLL_PHASE_90_DEGREE 0x1F
+#define XENON_DLL_FAST_LOCK BIT(5)
+#define XENON_DLL_GAIN2X BIT(3)
+#define XENON_DLL_BYPASS_EN BIT(0)
+
+#define XENON_EMMC_5_0_PHY_LOGIC_TIMING_ADJUST \
+ (XENON_EMMC_5_0_PHY_REG_BASE + 0x14)
+#define XENON_EMMC_5_0_PHY_LOGIC_TIMING_VALUE 0x5A54
+#define XENON_EMMC_PHY_LOGIC_TIMING_ADJUST (XENON_EMMC_PHY_REG_BASE + 0x18)
+#define XENON_LOGIC_TIMING_VALUE 0x00AA8977
+
+/*
+ * List offset of PHY registers and some special register values
+ * in eMMC PHY 5.0 or eMMC PHY 5.1
+ */
+struct xenon_emmc_phy_regs {
+ /* Offset of Timing Adjust register */
+ u16 timing_adj;
+ /* Offset of Func Control register */
+ u16 func_ctrl;
+ /* Offset of Pad Control register */
+ u16 pad_ctrl;
+ /* Offset of Pad Control register 2 */
+ u16 pad_ctrl2;
+ /* Offset of DLL Control register */
+ u16 dll_ctrl;
+ /* Offset of Logic Timing Adjust register */
+ u16 logic_timing_adj;
+ /* DLL Update Enable bit */
+ u32 dll_update;
+ /* value in Logic Timing Adjustment register */
+ u32 logic_timing_val;
+};
+
+static const char * const phy_types[] = {
+ "emmc 5.0 phy",
+ "emmc 5.1 phy"
+};
+
+enum xenon_phy_type_enum {
+ EMMC_5_0_PHY,
+ EMMC_5_1_PHY,
+ NR_PHY_TYPES
+};
+
+enum soc_pad_ctrl_type {
+ SOC_PAD_SD,
+ SOC_PAD_FIXED_1_8V,
+};
+
+struct soc_pad_ctrl {
+ /* Register address of SoC PHY PAD ctrl */
+ void __iomem *reg;
+ /* SoC PHY PAD ctrl type */
+ enum soc_pad_ctrl_type pad_type;
+ /* SoC specific operation to set SoC PHY PAD */
+ void (*set_soc_pad)(struct sdhci_host *host,
+ unsigned char signal_voltage);
+};
+
+static struct xenon_emmc_phy_regs xenon_emmc_5_0_phy_regs = {
+ .timing_adj = XENON_EMMC_5_0_PHY_TIMING_ADJUST,
+ .func_ctrl = XENON_EMMC_5_0_PHY_FUNC_CONTROL,
+ .pad_ctrl = XENON_EMMC_5_0_PHY_PAD_CONTROL,
+ .pad_ctrl2 = XENON_EMMC_5_0_PHY_PAD_CONTROL2,
+ .dll_ctrl = XENON_EMMC_5_0_PHY_DLL_CONTROL,
+ .logic_timing_adj = XENON_EMMC_5_0_PHY_LOGIC_TIMING_ADJUST,
+ .dll_update = XENON_DLL_UPDATE_STROBE_5_0,
+ .logic_timing_val = XENON_EMMC_5_0_PHY_LOGIC_TIMING_VALUE,
+};
+
+static struct xenon_emmc_phy_regs xenon_emmc_5_1_phy_regs = {
+ .timing_adj = XENON_EMMC_PHY_TIMING_ADJUST,
+ .func_ctrl = XENON_EMMC_PHY_FUNC_CONTROL,
+ .pad_ctrl = XENON_EMMC_PHY_PAD_CONTROL,
+ .pad_ctrl2 = XENON_EMMC_PHY_PAD_CONTROL2,
+ .dll_ctrl = XENON_EMMC_PHY_DLL_CONTROL,
+ .logic_timing_adj = XENON_EMMC_PHY_LOGIC_TIMING_ADJUST,
+ .dll_update = XENON_DLL_UPDATE,
+ .logic_timing_val = XENON_LOGIC_TIMING_VALUE,
+};
+
+/*
+ * eMMC PHY configuration and operations
+ */
+struct xenon_emmc_phy_params {
+ bool slow_mode;
+
+ u8 znr;
+ u8 zpr;
+
+ /* Nr of consecutive Sampling Points of a Valid Sampling Window */
+ u8 nr_tun_times;
+ /* Divider for calculating Tuning Step */
+ u8 tun_step_divider;
+
+ struct soc_pad_ctrl pad_ctrl;
+};
+
+static int xenon_alloc_emmc_phy(struct sdhci_host *host)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ struct xenon_emmc_phy_params *params;
+
+ params = devm_kzalloc(mmc_dev(host->mmc), sizeof(*params), GFP_KERNEL);
+ if (!params)
+ return -ENOMEM;
+
+ priv->phy_params = params;
+ if (priv->phy_type == EMMC_5_0_PHY)
+ priv->emmc_phy_regs = &xenon_emmc_5_0_phy_regs;
+ else
+ priv->emmc_phy_regs = &xenon_emmc_5_1_phy_regs;
+
+ return 0;
+}
+
+/*
+ * eMMC 5.0/5.1 PHY init/re-init.
+ * eMMC PHY init should be executed after:
+ * 1. SDCLK frequency changes.
+ * 2. SDCLK is stopped and re-enabled.
+ * 3. config in emmc_phy_regs->timing_adj and emmc_phy_regs->func_ctrl
+ * are changed
+ */
+static int xenon_emmc_phy_init(struct sdhci_host *host)
+{
+ u32 reg;
+ u32 wait, clock;
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ struct xenon_emmc_phy_regs *phy_regs = priv->emmc_phy_regs;
+
+ reg = sdhci_readl(host, phy_regs->timing_adj);
+ reg |= XENON_PHY_INITIALIZAION;
+ sdhci_writel(host, reg, phy_regs->timing_adj);
+
+ /* Add duration of FC_SYNC_RST */
+ wait = ((reg >> XENON_FC_SYNC_RST_DURATION_SHIFT) &
+ XENON_FC_SYNC_RST_DURATION_MASK);
+ /* Add interval between FC_SYNC_EN and FC_SYNC_RST */
+ wait += ((reg >> XENON_FC_SYNC_RST_EN_DURATION_SHIFT) &
+ XENON_FC_SYNC_RST_EN_DURATION_MASK);
+ /* Add duration of asserting FC_SYNC_EN */
+ wait += ((reg >> XENON_FC_SYNC_EN_DURATION_SHIFT) &
+ XENON_FC_SYNC_EN_DURATION_MASK);
+ /* Add duration of waiting for PHY */
+ wait += ((reg >> XENON_WAIT_CYCLE_BEFORE_USING_SHIFT) &
+ XENON_WAIT_CYCLE_BEFORE_USING_MASK);
+ /* 4 additional bus clock and 4 AXI bus clock are required */
+ wait += 8;
+ wait <<= 20;
+
+ clock = host->clock;
+ if (!clock)
+ /* Use the possibly slowest bus frequency value */
+ clock = XENON_LOWEST_SDCLK_FREQ;
+ /* get the wait time */
+ wait /= clock;
+ wait++;
+ /* wait for host eMMC PHY init completes */
+ udelay(wait);
+
+ reg = sdhci_readl(host, phy_regs->timing_adj);
+ reg &= XENON_PHY_INITIALIZAION;
+ if (reg) {
+ dev_err(mmc_dev(host->mmc), "eMMC PHY init cannot complete after %d us\n",
+ wait);
+ return -ETIMEDOUT;
+ }
+
+ return 0;
+}
+
+#define ARMADA_3700_SOC_PAD_1_8V 0x1
+#define ARMADA_3700_SOC_PAD_3_3V 0x0
+
+static void armada_3700_soc_pad_voltage_set(struct sdhci_host *host,
+ unsigned char signal_voltage)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ struct xenon_emmc_phy_params *params = priv->phy_params;
+
+ if (params->pad_ctrl.pad_type == SOC_PAD_FIXED_1_8V) {
+ writel(ARMADA_3700_SOC_PAD_1_8V, params->pad_ctrl.reg);
+ } else if (params->pad_ctrl.pad_type == SOC_PAD_SD) {
+ if (signal_voltage == MMC_SIGNAL_VOLTAGE_180)
+ writel(ARMADA_3700_SOC_PAD_1_8V, params->pad_ctrl.reg);
+ else if (signal_voltage == MMC_SIGNAL_VOLTAGE_330)
+ writel(ARMADA_3700_SOC_PAD_3_3V, params->pad_ctrl.reg);
+ }
+}
+
+/*
+ * Set SoC PHY voltage PAD control register,
+ * according to the operation voltage on PAD.
+ * The detailed operation depends on SoC implementation.
+ */
+static void xenon_emmc_phy_set_soc_pad(struct sdhci_host *host,
+ unsigned char signal_voltage)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ struct xenon_emmc_phy_params *params = priv->phy_params;
+
+ if (!params->pad_ctrl.reg)
+ return;
+
+ if (params->pad_ctrl.set_soc_pad)
+ params->pad_ctrl.set_soc_pad(host, signal_voltage);
+}
+
+/*
+ * Enable eMMC PHY HW DLL
+ * DLL should be enabled and stable before HS200/SDR104 tuning,
+ * and before HS400 data strobe setting.
+ */
+static int xenon_emmc_phy_enable_dll(struct sdhci_host *host)
+{
+ u32 reg;
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ struct xenon_emmc_phy_regs *phy_regs = priv->emmc_phy_regs;
+ ktime_t timeout;
+
+ if (WARN_ON(host->clock <= MMC_HIGH_52_MAX_DTR))
+ return -EINVAL;
+
+ reg = sdhci_readl(host, phy_regs->dll_ctrl);
+ if (reg & XENON_DLL_ENABLE)
+ return 0;
+
+ /* Enable DLL */
+ reg = sdhci_readl(host, phy_regs->dll_ctrl);
+ reg |= (XENON_DLL_ENABLE | XENON_DLL_FAST_LOCK);
+
+ /*
+ * Set Phase as 90 degree, which is most common value.
+ * Might set another value if necessary.
+ * The granularity is 1 degree.
+ */
+ reg &= ~((XENON_DLL_PHASE_MASK << XENON_DLL_PHSEL0_SHIFT) |
+ (XENON_DLL_PHASE_MASK << XENON_DLL_PHSEL1_SHIFT));
+ reg |= ((XENON_DLL_PHASE_90_DEGREE << XENON_DLL_PHSEL0_SHIFT) |
+ (XENON_DLL_PHASE_90_DEGREE << XENON_DLL_PHSEL1_SHIFT));
+
+ reg &= ~XENON_DLL_BYPASS_EN;
+ reg |= phy_regs->dll_update;
+ if (priv->phy_type == EMMC_5_1_PHY)
+ reg &= ~XENON_DLL_REFCLK_SEL;
+ sdhci_writel(host, reg, phy_regs->dll_ctrl);
+
+ /* Wait max 32 ms */
+ timeout = ktime_add_ms(ktime_get(), 32);
+ while (!(sdhci_readw(host, XENON_SLOT_EXT_PRESENT_STATE) &
+ XENON_DLL_LOCK_STATE)) {
+ if (ktime_after(ktime_get(), timeout)) {
+ dev_err(mmc_dev(host->mmc), "Wait for DLL Lock time-out\n");
+ return -ETIMEDOUT;
+ }
+ udelay(100);
+ }
+ return 0;
+}
+
+/*
+ * Config to eMMC PHY to prepare for tuning.
+ * Enable HW DLL and set the TUNING_STEP
+ */
+static int xenon_emmc_phy_config_tuning(struct sdhci_host *host)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ struct xenon_emmc_phy_params *params = priv->phy_params;
+ u32 reg, tuning_step;
+ int ret;
+
+ if (host->clock <= MMC_HIGH_52_MAX_DTR)
+ return -EINVAL;
+
+ ret = xenon_emmc_phy_enable_dll(host);
+ if (ret)
+ return ret;
+
+ /* Achieve TUNING_STEP with HW DLL help */
+ reg = sdhci_readl(host, XENON_SLOT_DLL_CUR_DLY_VAL);
+ tuning_step = reg / params->tun_step_divider;
+ if (unlikely(tuning_step > XENON_TUNING_STEP_MASK)) {
+ dev_warn(mmc_dev(host->mmc),
+ "HS200 TUNING_STEP %d is larger than MAX value\n",
+ tuning_step);
+ tuning_step = XENON_TUNING_STEP_MASK;
+ }
+
+ /* Set TUNING_STEP for later tuning */
+ reg = sdhci_readl(host, XENON_SLOT_OP_STATUS_CTRL);
+ reg &= ~(XENON_TUN_CONSECUTIVE_TIMES_MASK <<
+ XENON_TUN_CONSECUTIVE_TIMES_SHIFT);
+ reg |= (params->nr_tun_times << XENON_TUN_CONSECUTIVE_TIMES_SHIFT);
+ reg &= ~(XENON_TUNING_STEP_MASK << XENON_TUNING_STEP_SHIFT);
+ reg |= (tuning_step << XENON_TUNING_STEP_SHIFT);
+ sdhci_writel(host, reg, XENON_SLOT_OP_STATUS_CTRL);
+
+ return 0;
+}
+
+static void xenon_emmc_phy_disable_data_strobe(struct sdhci_host *host)
+{
+ u32 reg;
+
+ /* Disable SDHC Data Strobe */
+ reg = sdhci_readl(host, XENON_SLOT_EMMC_CTRL);
+ reg &= ~XENON_ENABLE_DATA_STROBE;
+ sdhci_writel(host, reg, XENON_SLOT_EMMC_CTRL);
+}
+
+/* Set HS400 Data Strobe */
+static void xenon_emmc_phy_strobe_delay_adj(struct sdhci_host *host)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ u32 reg;
+
+ if (WARN_ON(host->timing != MMC_TIMING_MMC_HS400))
+ return;
+
+ if (host->clock <= MMC_HIGH_52_MAX_DTR)
+ return;
+
+ dev_dbg(mmc_dev(host->mmc), "starts HS400 strobe delay adjustment\n");
+
+ xenon_emmc_phy_enable_dll(host);
+
+ /* Enable SDHC Data Strobe */
+ reg = sdhci_readl(host, XENON_SLOT_EMMC_CTRL);
+ reg |= XENON_ENABLE_DATA_STROBE;
+ sdhci_writel(host, reg, XENON_SLOT_EMMC_CTRL);
+
+ /* Set Data Strobe Pull down */
+ if (priv->phy_type == EMMC_5_0_PHY) {
+ reg = sdhci_readl(host, XENON_EMMC_5_0_PHY_PAD_CONTROL);
+ reg |= XENON_EMMC5_FC_QSP_PD;
+ reg &= ~XENON_EMMC5_FC_QSP_PU;
+ sdhci_writel(host, reg, XENON_EMMC_5_0_PHY_PAD_CONTROL);
+ } else {
+ reg = sdhci_readl(host, XENON_EMMC_PHY_PAD_CONTROL1);
+ reg |= XENON_EMMC5_1_FC_QSP_PD;
+ reg &= ~XENON_EMMC5_1_FC_QSP_PU;
+ sdhci_writel(host, reg, XENON_EMMC_PHY_PAD_CONTROL1);
+ }
+}
+
+/*
+ * If eMMC PHY Slow Mode is required in lower speed mode (SDCLK < 55MHz)
+ * in SDR mode, enable Slow Mode to bypass eMMC PHY.
+ * SDIO slower SDR mode also requires Slow Mode.
+ *
+ * If Slow Mode is enabled, return true.
+ * Otherwise, return false.
+ */
+static bool xenon_emmc_phy_slow_mode(struct sdhci_host *host,
+ unsigned char timing)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ struct xenon_emmc_phy_params *params = priv->phy_params;
+ struct xenon_emmc_phy_regs *phy_regs = priv->emmc_phy_regs;
+ u32 reg;
+ int ret;
+
+ if (host->clock > MMC_HIGH_52_MAX_DTR)
+ return false;
+
+ reg = sdhci_readl(host, phy_regs->timing_adj);
+ /* When in slower SDR mode, enable Slow Mode for SDIO
+ * or when Slow Mode flag is set
+ */
+ switch (timing) {
+ case MMC_TIMING_LEGACY:
+ /*
+ * If Slow Mode is required, enable Slow Mode by default
+ * in early init phase to avoid any potential issue.
+ */
+ if (params->slow_mode) {
+ reg |= XENON_TIMING_ADJUST_SLOW_MODE;
+ ret = true;
+ } else {
+ reg &= ~XENON_TIMING_ADJUST_SLOW_MODE;
+ ret = false;
+ }
+ break;
+ case MMC_TIMING_UHS_SDR25:
+ case MMC_TIMING_UHS_SDR12:
+ case MMC_TIMING_SD_HS:
+ case MMC_TIMING_MMC_HS:
+ if ((priv->init_card_type == MMC_TYPE_SDIO) ||
+ params->slow_mode) {
+ reg |= XENON_TIMING_ADJUST_SLOW_MODE;
+ ret = true;
+ break;
+ }
+ default:
+ reg &= ~XENON_TIMING_ADJUST_SLOW_MODE;
+ ret = false;
+ }
+
+ sdhci_writel(host, reg, phy_regs->timing_adj);
+ return ret;
+}
+
+/*
+ * Set-up eMMC 5.0/5.1 PHY.
+ * Specific configuration depends on the current speed mode in use.
+ */
+static void xenon_emmc_phy_set(struct sdhci_host *host,
+ unsigned char timing)
+{
+ u32 reg;
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ struct xenon_emmc_phy_params *params = priv->phy_params;
+ struct xenon_emmc_phy_regs *phy_regs = priv->emmc_phy_regs;
+
+ dev_dbg(mmc_dev(host->mmc), "eMMC PHY setting starts\n");
+
+ /* Setup pad, set bit[28] and bits[26:24] */
+ reg = sdhci_readl(host, phy_regs->pad_ctrl);
+ reg |= (XENON_FC_DQ_RECEN | XENON_FC_CMD_RECEN |
+ XENON_FC_QSP_RECEN | XENON_OEN_QSN);
+ /* All FC_XX_RECEIVCE should be set as CMOS Type */
+ reg |= XENON_FC_ALL_CMOS_RECEIVER;
+ sdhci_writel(host, reg, phy_regs->pad_ctrl);
+
+ /* Set CMD and DQ Pull Up */
+ if (priv->phy_type == EMMC_5_0_PHY) {
+ reg = sdhci_readl(host, XENON_EMMC_5_0_PHY_PAD_CONTROL);
+ reg |= (XENON_EMMC5_FC_CMD_PU | XENON_EMMC5_FC_DQ_PU);
+ reg &= ~(XENON_EMMC5_FC_CMD_PD | XENON_EMMC5_FC_DQ_PD);
+ sdhci_writel(host, reg, XENON_EMMC_5_0_PHY_PAD_CONTROL);
+ } else {
+ reg = sdhci_readl(host, XENON_EMMC_PHY_PAD_CONTROL1);
+ reg |= (XENON_EMMC5_1_FC_CMD_PU | XENON_EMMC5_1_FC_DQ_PU);
+ reg &= ~(XENON_EMMC5_1_FC_CMD_PD | XENON_EMMC5_1_FC_DQ_PD);
+ sdhci_writel(host, reg, XENON_EMMC_PHY_PAD_CONTROL1);
+ }
+
+ if (timing == MMC_TIMING_LEGACY) {
+ xenon_emmc_phy_slow_mode(host, timing);
+ goto phy_init;
+ }
+
+ /*
+ * If SDIO card, set SDIO Mode
+ * Otherwise, clear SDIO Mode
+ */
+ reg = sdhci_readl(host, phy_regs->timing_adj);
+ if (priv->init_card_type == MMC_TYPE_SDIO)
+ reg |= XENON_TIMING_ADJUST_SDIO_MODE;
+ else
+ reg &= ~XENON_TIMING_ADJUST_SDIO_MODE;
+ sdhci_writel(host, reg, phy_regs->timing_adj);
+
+ if (xenon_emmc_phy_slow_mode(host, timing))
+ goto phy_init;
+
+ /*
+ * Set preferred ZNR and ZPR value
+ * The ZNR and ZPR value vary between different boards.
+ * Define them both in sdhci-xenon-emmc-phy.h.
+ */
+ reg = sdhci_readl(host, phy_regs->pad_ctrl2);
+ reg &= ~((XENON_ZNR_MASK << XENON_ZNR_SHIFT) | XENON_ZPR_MASK);
+ reg |= ((params->znr << XENON_ZNR_SHIFT) | params->zpr);
+ sdhci_writel(host, reg, phy_regs->pad_ctrl2);
+
+ /*
+ * When setting EMMC_PHY_FUNC_CONTROL register,
+ * SD clock should be disabled
+ */
+ reg = sdhci_readl(host, SDHCI_CLOCK_CONTROL);
+ reg &= ~SDHCI_CLOCK_CARD_EN;
+ sdhci_writew(host, reg, SDHCI_CLOCK_CONTROL);
+
+ reg = sdhci_readl(host, phy_regs->func_ctrl);
+ switch (timing) {
+ case MMC_TIMING_MMC_HS400:
+ reg |= (XENON_DQ_DDR_MODE_MASK << XENON_DQ_DDR_MODE_SHIFT) |
+ XENON_CMD_DDR_MODE;
+ reg &= ~XENON_DQ_ASYNC_MODE;
+ break;
+ case MMC_TIMING_UHS_DDR50:
+ case MMC_TIMING_MMC_DDR52:
+ reg |= (XENON_DQ_DDR_MODE_MASK << XENON_DQ_DDR_MODE_SHIFT) |
+ XENON_CMD_DDR_MODE | XENON_DQ_ASYNC_MODE;
+ break;
+ default:
+ reg &= ~((XENON_DQ_DDR_MODE_MASK << XENON_DQ_DDR_MODE_SHIFT) |
+ XENON_CMD_DDR_MODE);
+ reg |= XENON_DQ_ASYNC_MODE;
+ }
+ sdhci_writel(host, reg, phy_regs->func_ctrl);
+
+ /* Enable bus clock */
+ reg = sdhci_readl(host, SDHCI_CLOCK_CONTROL);
+ reg |= SDHCI_CLOCK_CARD_EN;
+ sdhci_writew(host, reg, SDHCI_CLOCK_CONTROL);
+
+ if (timing == MMC_TIMING_MMC_HS400)
+ /* Hardware team recommend a value for HS400 */
+ sdhci_writel(host, phy_regs->logic_timing_val,
+ phy_regs->logic_timing_adj);
+ else
+ xenon_emmc_phy_disable_data_strobe(host);
+
+phy_init:
+ xenon_emmc_phy_init(host);
+
+ dev_dbg(mmc_dev(host->mmc), "eMMC PHY setting completes\n");
+}
+
+static int get_dt_pad_ctrl_data(struct sdhci_host *host,
+ struct device_node *np,
+ struct xenon_emmc_phy_params *params)
+{
+ int ret = 0;
+ const char *name;
+ struct resource iomem;
+
+ if (of_device_is_compatible(np, "marvell,armada-3700-sdhci"))
+ params->pad_ctrl.set_soc_pad = armada_3700_soc_pad_voltage_set;
+ else
+ return 0;
+
+ if (of_address_to_resource(np, 1, &iomem)) {
+ dev_err(mmc_dev(host->mmc), "Unable to find SoC PAD ctrl register address for %s\n",
+ np->name);
+ return -EINVAL;
+ }
+
+ params->pad_ctrl.reg = devm_ioremap_resource(mmc_dev(host->mmc),
+ &iomem);
+ if (IS_ERR(params->pad_ctrl.reg))
+ return PTR_ERR(params->pad_ctrl.reg);
+
+ ret = of_property_read_string(np, "marvell,pad-type", &name);
+ if (ret) {
+ dev_err(mmc_dev(host->mmc), "Unable to determine SoC PHY PAD ctrl type\n");
+ return ret;
+ }
+ if (!strcmp(name, "sd")) {
+ params->pad_ctrl.pad_type = SOC_PAD_SD;
+ } else if (!strcmp(name, "fixed-1-8v")) {
+ params->pad_ctrl.pad_type = SOC_PAD_FIXED_1_8V;
+ } else {
+ dev_err(mmc_dev(host->mmc), "Unsupported SoC PHY PAD ctrl type %s\n",
+ name);
+ return -EINVAL;
+ }
+
+ return ret;
+}
+
+static int xenon_emmc_phy_parse_param_dt(struct sdhci_host *host,
+ struct device_node *np,
+ struct xenon_emmc_phy_params *params)
+{
+ u32 value;
+
+ params->slow_mode = false;
+ if (of_property_read_bool(np, "marvell,xenon-phy-slow-mode"))
+ params->slow_mode = true;
+
+ params->znr = XENON_ZNR_DEF_VALUE;
+ if (!of_property_read_u32(np, "marvell,xenon-phy-znr", &value))
+ params->znr = value & XENON_ZNR_MASK;
+
+ params->zpr = XENON_ZPR_DEF_VALUE;
+ if (!of_property_read_u32(np, "marvell,xenon-phy-zpr", &value))
+ params->zpr = value & XENON_ZPR_MASK;
+
+ params->nr_tun_times = XENON_TUN_CONSECUTIVE_TIMES;
+ if (!of_property_read_u32(np, "marvell,xenon-phy-nr-success-tun",
+ &value))
+ params->nr_tun_times = value & XENON_TUN_CONSECUTIVE_TIMES_MASK;
+
+ params->tun_step_divider = XENON_TUNING_STEP_DIVIDER;
+ if (!of_property_read_u32(np, "marvell,xenon-phy-tun-step-divider",
+ &value))
+ params->tun_step_divider = value & 0xFF;
+
+ return get_dt_pad_ctrl_data(host, np, params);
+}
+
+/* Set SoC PHY Voltage PAD */
+void xenon_soc_pad_ctrl(struct sdhci_host *host,
+ unsigned char signal_voltage)
+{
+ xenon_emmc_phy_set_soc_pad(host, signal_voltage);
+}
+
+/*
+ * Setting PHY when card is working in High Speed Mode.
+ * HS400 set data strobe line.
+ * HS200/SDR104 set tuning config to prepare for tuning.
+ */
+static int xenon_hs_delay_adj(struct sdhci_host *host)
+{
+ int ret = 0;
+
+ if (WARN_ON(host->clock <= XENON_DEFAULT_SDCLK_FREQ))
+ return -EINVAL;
+
+ switch (host->timing) {
+ case MMC_TIMING_MMC_HS400:
+ xenon_emmc_phy_strobe_delay_adj(host);
+ return 0;
+ case MMC_TIMING_MMC_HS200:
+ case MMC_TIMING_UHS_SDR104:
+ return xenon_emmc_phy_config_tuning(host);
+ case MMC_TIMING_MMC_DDR52:
+ case MMC_TIMING_UHS_DDR50:
+ /*
+ * DDR Mode requires driver to scan Sampling Fixed Delay Line,
+ * to find out a perfect operation sampling point.
+ * It is hard to implement such a scan in host driver
+ * since initiating commands by host driver is not safe.
+ * Thus so far just keep PHY Sampling Fixed Delay in
+ * default value of DDR mode.
+ *
+ * If any timing issue occurs in DDR mode on Marvell products,
+ * please contact maintainer for internal support in Marvell.
+ */
+ dev_warn_once(mmc_dev(host->mmc), "Timing issue might occur in DDR mode\n");
+ return 0;
+ }
+
+ return ret;
+}
+
+/*
+ * Adjust PHY setting.
+ * PHY setting should be adjusted when SDCLK frequency, Bus Width
+ * or Speed Mode is changed.
+ * Additional config are required when card is working in High Speed mode,
+ * after leaving Legacy Mode.
+ */
+int xenon_phy_adj(struct sdhci_host *host, struct mmc_ios *ios)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ int ret = 0;
+
+ if (!host->clock) {
+ priv->clock = 0;
+ return 0;
+ }
+
+ /*
+ * The timing, frequency or bus width is changed,
+ * better to set eMMC PHY based on current setting
+ * and adjust Xenon SDHC delay.
+ */
+ if ((host->clock == priv->clock) &&
+ (ios->bus_width == priv->bus_width) &&
+ (ios->timing == priv->timing))
+ return 0;
+
+ xenon_emmc_phy_set(host, ios->timing);
+
+ /* Update the record */
+ priv->bus_width = ios->bus_width;
+
+ priv->timing = ios->timing;
+ priv->clock = host->clock;
+
+ /* Legacy mode is a special case */
+ if (ios->timing == MMC_TIMING_LEGACY)
+ return 0;
+
+ if (host->clock > XENON_DEFAULT_SDCLK_FREQ)
+ ret = xenon_hs_delay_adj(host);
+ return ret;
+}
+
+void xenon_clean_phy(struct sdhci_host *host)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+
+ kfree(priv->phy_params);
+}
+
+static int xenon_add_phy(struct device_node *np, struct sdhci_host *host,
+ const char *phy_name)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ int i, ret;
+
+ for (i = 0; i < NR_PHY_TYPES; i++) {
+ if (!strcmp(phy_name, phy_types[i])) {
+ priv->phy_type = i;
+ break;
+ }
+ }
+ if (i == NR_PHY_TYPES) {
+ dev_err(mmc_dev(host->mmc),
+ "Unable to determine PHY name %s. Use default eMMC 5.1 PHY\n",
+ phy_name);
+ priv->phy_type = EMMC_5_1_PHY;
+ }
+
+ ret = xenon_alloc_emmc_phy(host);
+ if (ret)
+ return ret;
+
+ ret = xenon_emmc_phy_parse_param_dt(host, np, priv->phy_params);
+ if (ret)
+ xenon_clean_phy(host);
+
+ return ret;
+}
+
+int xenon_phy_parse_dt(struct device_node *np, struct sdhci_host *host)
+{
+ const char *phy_type = NULL;
+
+ if (!of_property_read_string(np, "marvell,xenon-phy-type", &phy_type))
+ return xenon_add_phy(np, host, phy_type);
+
+ return xenon_add_phy(np, host, "emmc 5.1 phy");
+}
diff --git a/drivers/mmc/host/sdhci-xenon.c b/drivers/mmc/host/sdhci-xenon.c
new file mode 100644
index 0000000..6724665
--- /dev/null
+++ b/drivers/mmc/host/sdhci-xenon.c
@@ -0,0 +1,548 @@
+/*
+ * Driver for Marvell Xenon SDHC as a platform device
+ *
+ * Copyright (C) 2016 Marvell, All Rights Reserved.
+ *
+ * Author: Hu Ziji <huziji@marvell.com>
+ * Date: 2016-8-24
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation version 2.
+ *
+ * Inspired by Jisheng Zhang <jszhang@marvell.com>
+ * Special thanks to Video BG4 project team.
+ */
+
+#include <linux/delay.h>
+#include <linux/ktime.h>
+#include <linux/module.h>
+#include <linux/of.h>
+
+#include "sdhci-pltfm.h"
+#include "sdhci-xenon.h"
+
+static int xenon_enable_internal_clk(struct sdhci_host *host)
+{
+ u32 reg;
+ ktime_t timeout;
+
+ reg = sdhci_readl(host, SDHCI_CLOCK_CONTROL);
+ reg |= SDHCI_CLOCK_INT_EN;
+ sdhci_writel(host, reg, SDHCI_CLOCK_CONTROL);
+ /* Wait max 20 ms */
+ timeout = ktime_add_ms(ktime_get(), 20);
+ while (!((reg = sdhci_readw(host, SDHCI_CLOCK_CONTROL))
+ & SDHCI_CLOCK_INT_STABLE)) {
+ if (ktime_after(ktime_get(), timeout)) {
+ dev_err(mmc_dev(host->mmc), "Internal clock never stabilised.\n");
+ return -ETIMEDOUT;
+ }
+ usleep_range(900, 1100);
+ }
+
+ return 0;
+}
+
+/* Set SDCLK-off-while-idle */
+static void xenon_set_sdclk_off_idle(struct sdhci_host *host,
+ unsigned char sdhc_id, bool enable)
+{
+ u32 reg;
+ u32 mask;
+
+ reg = sdhci_readl(host, XENON_SYS_OP_CTRL);
+ /* Get the bit shift basing on the SDHC index */
+ mask = (0x1 << (XENON_SDCLK_IDLEOFF_ENABLE_SHIFT + sdhc_id));
+ if (enable)
+ reg |= mask;
+ else
+ reg &= ~mask;
+
+ sdhci_writel(host, reg, XENON_SYS_OP_CTRL);
+}
+
+/* Enable/Disable the Auto Clock Gating function */
+static void xenon_set_acg(struct sdhci_host *host, bool enable)
+{
+ u32 reg;
+
+ reg = sdhci_readl(host, XENON_SYS_OP_CTRL);
+ if (enable)
+ reg &= ~XENON_AUTO_CLKGATE_DISABLE_MASK;
+ else
+ reg |= XENON_AUTO_CLKGATE_DISABLE_MASK;
+ sdhci_writel(host, reg, XENON_SYS_OP_CTRL);
+}
+
+/* Enable this SDHC */
+static void xenon_enable_sdhc(struct sdhci_host *host,
+ unsigned char sdhc_id)
+{
+ u32 reg;
+
+ reg = sdhci_readl(host, XENON_SYS_OP_CTRL);
+ reg |= (BIT(sdhc_id) << XENON_SLOT_ENABLE_SHIFT);
+ sdhci_writel(host, reg, XENON_SYS_OP_CTRL);
+
+ host->mmc->caps |= MMC_CAP_WAIT_WHILE_BUSY;
+ /*
+ * Force to clear BUS_TEST to
+ * skip bus_test_pre and bus_test_post
+ */
+ host->mmc->caps &= ~MMC_CAP_BUS_WIDTH_TEST;
+}
+
+/* Disable this SDHC */
+static void xenon_disable_sdhc(struct sdhci_host *host,
+ unsigned char sdhc_id)
+{
+ u32 reg;
+
+ reg = sdhci_readl(host, XENON_SYS_OP_CTRL);
+ reg &= ~(BIT(sdhc_id) << XENON_SLOT_ENABLE_SHIFT);
+ sdhci_writel(host, reg, XENON_SYS_OP_CTRL);
+}
+
+/* Enable Parallel Transfer Mode */
+static void xenon_enable_sdhc_parallel_tran(struct sdhci_host *host,
+ unsigned char sdhc_id)
+{
+ u32 reg;
+
+ reg = sdhci_readl(host, XENON_SYS_EXT_OP_CTRL);
+ reg |= BIT(sdhc_id);
+ sdhci_writel(host, reg, XENON_SYS_EXT_OP_CTRL);
+}
+
+/* Mask command conflict error */
+static void xenon_mask_cmd_conflict_err(struct sdhci_host *host)
+{
+ u32 reg;
+
+ reg = sdhci_readl(host, XENON_SYS_EXT_OP_CTRL);
+ reg |= XENON_MASK_CMD_CONFLICT_ERR;
+ sdhci_writel(host, reg, XENON_SYS_EXT_OP_CTRL);
+}
+
+static void xenon_retune_setup(struct sdhci_host *host)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ u32 reg;
+
+ /* Disable the Re-Tuning Request functionality */
+ reg = sdhci_readl(host, XENON_SLOT_RETUNING_REQ_CTRL);
+ reg &= ~XENON_RETUNING_COMPATIBLE;
+ sdhci_writel(host, reg, XENON_SLOT_RETUNING_REQ_CTRL);
+
+ /* Disable the Re-tuning Interrupt */
+ reg = sdhci_readl(host, SDHCI_SIGNAL_ENABLE);
+ reg &= ~SDHCI_INT_RETUNE;
+ sdhci_writel(host, reg, SDHCI_SIGNAL_ENABLE);
+ reg = sdhci_readl(host, SDHCI_INT_ENABLE);
+ reg &= ~SDHCI_INT_RETUNE;
+ sdhci_writel(host, reg, SDHCI_INT_ENABLE);
+
+ /* Force to use Tuning Mode 1 */
+ host->tuning_mode = SDHCI_TUNING_MODE_1;
+ /* Set re-tuning period */
+ host->tuning_count = 1 << (priv->tuning_count - 1);
+}
+
+/*
+ * Operations inside struct sdhci_ops
+ */
+/* Recover the Register Setting cleared during SOFTWARE_RESET_ALL */
+static void xenon_reset_exit(struct sdhci_host *host,
+ unsigned char sdhc_id, u8 mask)
+{
+ /* Only SOFTWARE RESET ALL will clear the register setting */
+ if (!(mask & SDHCI_RESET_ALL))
+ return;
+
+ /* Disable tuning request and auto-retuning again */
+ xenon_retune_setup(host);
+
+ xenon_set_acg(host, true);
+
+ xenon_set_sdclk_off_idle(host, sdhc_id, false);
+
+ xenon_mask_cmd_conflict_err(host);
+}
+
+static void xenon_reset(struct sdhci_host *host, u8 mask)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+
+ sdhci_reset(host, mask);
+ xenon_reset_exit(host, priv->sdhc_id, mask);
+}
+
+/*
+ * Xenon defines different values for HS200 and HS400
+ * in Host_Control_2
+ */
+static void xenon_set_uhs_signaling(struct sdhci_host *host,
+ unsigned int timing)
+{
+ u16 ctrl_2;
+
+ ctrl_2 = sdhci_readw(host, SDHCI_HOST_CONTROL2);
+ /* Select Bus Speed Mode for host */
+ ctrl_2 &= ~SDHCI_CTRL_UHS_MASK;
+ if (timing == MMC_TIMING_MMC_HS200)
+ ctrl_2 |= XENON_CTRL_HS200;
+ else if (timing == MMC_TIMING_UHS_SDR104)
+ ctrl_2 |= SDHCI_CTRL_UHS_SDR104;
+ else if (timing == MMC_TIMING_UHS_SDR12)
+ ctrl_2 |= SDHCI_CTRL_UHS_SDR12;
+ else if (timing == MMC_TIMING_UHS_SDR25)
+ ctrl_2 |= SDHCI_CTRL_UHS_SDR25;
+ else if (timing == MMC_TIMING_UHS_SDR50)
+ ctrl_2 |= SDHCI_CTRL_UHS_SDR50;
+ else if ((timing == MMC_TIMING_UHS_DDR50) ||
+ (timing == MMC_TIMING_MMC_DDR52))
+ ctrl_2 |= SDHCI_CTRL_UHS_DDR50;
+ else if (timing == MMC_TIMING_MMC_HS400)
+ ctrl_2 |= XENON_CTRL_HS400;
+ sdhci_writew(host, ctrl_2, SDHCI_HOST_CONTROL2);
+}
+
+static const struct sdhci_ops sdhci_xenon_ops = {
+ .set_clock = sdhci_set_clock,
+ .set_bus_width = sdhci_set_bus_width,
+ .reset = xenon_reset,
+ .set_uhs_signaling = xenon_set_uhs_signaling,
+ .get_max_clock = sdhci_pltfm_clk_get_max_clock,
+};
+
+static const struct sdhci_pltfm_data sdhci_xenon_pdata = {
+ .ops = &sdhci_xenon_ops,
+ .quirks = SDHCI_QUIRK_NO_ENDATTR_IN_NOPDESC |
+ SDHCI_QUIRK_NO_SIMULT_VDD_AND_POWER |
+ SDHCI_QUIRK_CAP_CLOCK_BASE_BROKEN,
+};
+
+/*
+ * Xenon Specific Operations in mmc_host_ops
+ */
+static void xenon_set_ios(struct mmc_host *mmc, struct mmc_ios *ios)
+{
+ struct sdhci_host *host = mmc_priv(mmc);
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ u32 reg;
+
+ /*
+ * HS400/HS200/eMMC HS doesn't have Preset Value register.
+ * However, sdhci_set_ios will read HS400/HS200 Preset register.
+ * Disable Preset Value register for HS400/HS200.
+ * eMMC HS with preset_enabled set will trigger a bug in
+ * get_preset_value().
+ */
+ if ((ios->timing == MMC_TIMING_MMC_HS400) ||
+ (ios->timing == MMC_TIMING_MMC_HS200) ||
+ (ios->timing == MMC_TIMING_MMC_HS)) {
+ host->preset_enabled = false;
+ host->quirks2 |= SDHCI_QUIRK2_PRESET_VALUE_BROKEN;
+ host->flags &= ~SDHCI_PV_ENABLED;
+
+ reg = sdhci_readw(host, SDHCI_HOST_CONTROL2);
+ reg &= ~SDHCI_CTRL_PRESET_VAL_ENABLE;
+ sdhci_writew(host, reg, SDHCI_HOST_CONTROL2);
+ } else {
+ host->quirks2 &= ~SDHCI_QUIRK2_PRESET_VALUE_BROKEN;
+ }
+
+ sdhci_set_ios(mmc, ios);
+ xenon_phy_adj(host, ios);
+
+ if (host->clock > XENON_DEFAULT_SDCLK_FREQ)
+ xenon_set_sdclk_off_idle(host, priv->sdhc_id, true);
+}
+
+static int xenon_start_signal_voltage_switch(struct mmc_host *mmc,
+ struct mmc_ios *ios)
+{
+ struct sdhci_host *host = mmc_priv(mmc);
+
+ /*
+ * Before SD/SDIO set signal voltage, SD bus clock should be
+ * disabled. However, sdhci_set_clock will also disable the Internal
+ * clock in mmc_set_signal_voltage().
+ * If Internal clock is disabled, the 3.3V/1.8V bit can not be updated.
+ * Thus here manually enable internal clock.
+ *
+ * After switch completes, it is unnecessary to disable internal clock,
+ * since keeping internal clock active obeys SD spec.
+ */
+ xenon_enable_internal_clk(host);
+
+ xenon_soc_pad_ctrl(host, ios->signal_voltage);
+
+ /*
+ * If Vqmmc is fixed on platform, vqmmc regulator should be unavailable.
+ * Thus SDHCI_CTRL_VDD_180 bit might not work then.
+ * Skip the standard voltage switch to avoid any issue.
+ */
+ if (PTR_ERR(mmc->supply.vqmmc) == -ENODEV)
+ return 0;
+
+ return sdhci_start_signal_voltage_switch(mmc, ios);
+}
+
+/*
+ * Update card type.
+ * priv->init_card_type will be used in PHY timing adjustment.
+ */
+static void xenon_init_card(struct mmc_host *mmc, struct mmc_card *card)
+{
+ struct sdhci_host *host = mmc_priv(mmc);
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+
+ /* Update card type*/
+ priv->init_card_type = card->type;
+}
+
+static int xenon_execute_tuning(struct mmc_host *mmc, u32 opcode)
+{
+ struct sdhci_host *host = mmc_priv(mmc);
+
+ if (host->timing == MMC_TIMING_UHS_DDR50)
+ return 0;
+
+ /*
+ * Currently force Xenon driver back to support mode 1 only,
+ * even though Xenon might claim to support mode 2 or mode 3.
+ * It requires more time to test mode 2/mode 3 on more platforms.
+ */
+ if (host->tuning_mode != SDHCI_TUNING_MODE_1)
+ xenon_retune_setup(host);
+
+ return sdhci_execute_tuning(mmc, opcode);
+}
+
+static void xenon_enable_sdio_irq(struct mmc_host *mmc, int enable)
+{
+ struct sdhci_host *host = mmc_priv(mmc);
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ u32 reg;
+ u8 sdhc_id = priv->sdhc_id;
+
+ sdhci_enable_sdio_irq(mmc, enable);
+
+ if (enable) {
+ /*
+ * Set SDIO Card Inserted indication
+ * to enable detecting SDIO async irq.
+ */
+ reg = sdhci_readl(host, XENON_SYS_CFG_INFO);
+ reg |= (1 << (sdhc_id + XENON_SLOT_TYPE_SDIO_SHIFT));
+ sdhci_writel(host, reg, XENON_SYS_CFG_INFO);
+ } else {
+ /* Clear SDIO Card Inserted indication */
+ reg = sdhci_readl(host, XENON_SYS_CFG_INFO);
+ reg &= ~(1 << (sdhc_id + XENON_SLOT_TYPE_SDIO_SHIFT));
+ sdhci_writel(host, reg, XENON_SYS_CFG_INFO);
+ }
+}
+
+static void xenon_replace_mmc_host_ops(struct sdhci_host *host)
+{
+ host->mmc_host_ops.set_ios = xenon_set_ios;
+ host->mmc_host_ops.start_signal_voltage_switch =
+ xenon_start_signal_voltage_switch;
+ host->mmc_host_ops.init_card = xenon_init_card;
+ host->mmc_host_ops.execute_tuning = xenon_execute_tuning;
+ host->mmc_host_ops.enable_sdio_irq = xenon_enable_sdio_irq;
+}
+
+/*
+ * Parse Xenon specific DT properties:
+ * sdhc-id: the index of current SDHC.
+ * Refer to XENON_SYS_CFG_INFO register
+ * tun-count: the interval between re-tuning
+ */
+static int xenon_probe_dt(struct platform_device *pdev)
+{
+ struct device_node *np = pdev->dev.of_node;
+ struct sdhci_host *host = platform_get_drvdata(pdev);
+ struct mmc_host *mmc = host->mmc;
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ u32 sdhc_id, nr_sdhc;
+ u32 tuning_count;
+
+ /* Disable HS200 on Armada AP806 */
+ if (of_device_is_compatible(np, "marvell,armada-ap806-sdhci"))
+ host->quirks2 |= SDHCI_QUIRK2_BROKEN_HS200;
+
+ sdhc_id = 0x0;
+ if (!of_property_read_u32(np, "marvell,xenon-sdhc-id", &sdhc_id)) {
+ nr_sdhc = sdhci_readl(host, XENON_SYS_CFG_INFO);
+ nr_sdhc &= XENON_NR_SUPPORTED_SLOT_MASK;
+ if (unlikely(sdhc_id > nr_sdhc)) {
+ dev_err(mmc_dev(mmc), "SDHC Index %d exceeds Number of SDHCs %d\n",
+ sdhc_id, nr_sdhc);
+ return -EINVAL;
+ }
+ }
+ priv->sdhc_id = sdhc_id;
+
+ tuning_count = XENON_DEF_TUNING_COUNT;
+ if (!of_property_read_u32(np, "marvell,xenon-tun-count",
+ &tuning_count)) {
+ if (unlikely(tuning_count >= XENON_TMR_RETUN_NO_PRESENT)) {
+ dev_err(mmc_dev(mmc), "Wrong Re-tuning Count. Set default value %d\n",
+ XENON_DEF_TUNING_COUNT);
+ tuning_count = XENON_DEF_TUNING_COUNT;
+ }
+ }
+ priv->tuning_count = tuning_count;
+
+ return xenon_phy_parse_dt(np, host);
+}
+
+static int xenon_sdhc_prepare(struct sdhci_host *host)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ u8 sdhc_id = priv->sdhc_id;
+
+ /* Enable SDHC */
+ xenon_enable_sdhc(host, sdhc_id);
+
+ /* Enable ACG */
+ xenon_set_acg(host, true);
+
+ /* Enable Parallel Transfer Mode */
+ xenon_enable_sdhc_parallel_tran(host, sdhc_id);
+
+ /* Disable SDCLK-Off-While-Idle before card init */
+ xenon_set_sdclk_off_idle(host, sdhc_id, false);
+
+ xenon_mask_cmd_conflict_err(host);
+
+ return 0;
+}
+
+static void xenon_sdhc_unprepare(struct sdhci_host *host)
+{
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
+ u8 sdhc_id = priv->sdhc_id;
+
+ /* disable SDHC */
+ xenon_disable_sdhc(host, sdhc_id);
+}
+
+static int xenon_probe(struct platform_device *pdev)
+{
+ struct sdhci_pltfm_host *pltfm_host;
+ struct sdhci_host *host;
+ struct xenon_priv *priv;
+ int err;
+
+ host = sdhci_pltfm_init(pdev, &sdhci_xenon_pdata,
+ sizeof(struct xenon_priv));
+ if (IS_ERR(host))
+ return PTR_ERR(host);
+
+ pltfm_host = sdhci_priv(host);
+ priv = sdhci_pltfm_priv(pltfm_host);
+
+ /*
+ * Link Xenon specific mmc_host_ops function,
+ * to replace standard ones in sdhci_ops.
+ */
+ xenon_replace_mmc_host_ops(host);
+
+ pltfm_host->clk = devm_clk_get(&pdev->dev, "core");
+ if (IS_ERR(pltfm_host->clk)) {
+ err = PTR_ERR(pltfm_host->clk);
+ dev_err(&pdev->dev, "Failed to setup input clk: %d\n", err);
+ goto free_pltfm;
+ }
+ err = clk_prepare_enable(pltfm_host->clk);
+ if (err)
+ goto free_pltfm;
+
+ err = mmc_of_parse(host->mmc);
+ if (err)
+ goto err_clk;
+
+ sdhci_get_of_property(pdev);
+
+ xenon_set_acg(host, false);
+
+ /* Xenon specific dt parse */
+ err = xenon_probe_dt(pdev);
+ if (err)
+ goto err_clk;
+
+ err = xenon_sdhc_prepare(host);
+ if (err)
+ goto clean_phy_param;
+
+ err = sdhci_add_host(host);
+ if (err)
+ goto remove_sdhc;
+
+ return 0;
+
+remove_sdhc:
+ xenon_sdhc_unprepare(host);
+clean_phy_param:
+ xenon_clean_phy(host);
+err_clk:
+ clk_disable_unprepare(pltfm_host->clk);
+free_pltfm:
+ sdhci_pltfm_free(pdev);
+ return err;
+}
+
+static int xenon_remove(struct platform_device *pdev)
+{
+ struct sdhci_host *host = platform_get_drvdata(pdev);
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+
+ xenon_clean_phy(host);
+
+ sdhci_remove_host(host, 0);
+
+ xenon_sdhc_unprepare(host);
+
+ clk_disable_unprepare(pltfm_host->clk);
+
+ sdhci_pltfm_free(pdev);
+
+ return 0;
+}
+
+static const struct of_device_id sdhci_xenon_dt_ids[] = {
+ { .compatible = "marvell,armada-ap806-sdhci",},
+ { .compatible = "marvell,armada-cp110-sdhci",},
+ { .compatible = "marvell,armada-3700-sdhci",},
+ {}
+};
+MODULE_DEVICE_TABLE(of, sdhci_xenon_dt_ids);
+
+static struct platform_driver sdhci_xenon_driver = {
+ .driver = {
+ .name = "xenon-sdhci",
+ .of_match_table = sdhci_xenon_dt_ids,
+ .pm = &sdhci_pltfm_pmops,
+ },
+ .probe = xenon_probe,
+ .remove = xenon_remove,
+};
+
+module_platform_driver(sdhci_xenon_driver);
+
+MODULE_DESCRIPTION("SDHCI platform driver for Marvell Xenon SDHC");
+MODULE_AUTHOR("Hu Ziji <huziji@marvell.com>");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/mmc/host/sdhci-xenon.h b/drivers/mmc/host/sdhci-xenon.h
new file mode 100644
index 0000000..6e6523e
--- /dev/null
+++ b/drivers/mmc/host/sdhci-xenon.h
@@ -0,0 +1,101 @@
+/*
+ * Copyright (C) 2016 Marvell, All Rights Reserved.
+ *
+ * Author: Hu Ziji <huziji@marvell.com>
+ * Date: 2016-8-24
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation version 2.
+ */
+#ifndef SDHCI_XENON_H_
+#define SDHCI_XENON_H_
+
+/* Register Offset of Xenon SDHC self-defined register */
+#define XENON_SYS_CFG_INFO 0x0104
+#define XENON_SLOT_TYPE_SDIO_SHIFT 24
+#define XENON_NR_SUPPORTED_SLOT_MASK 0x7
+
+#define XENON_SYS_OP_CTRL 0x0108
+#define XENON_AUTO_CLKGATE_DISABLE_MASK BIT(20)
+#define XENON_SDCLK_IDLEOFF_ENABLE_SHIFT 8
+#define XENON_SLOT_ENABLE_SHIFT 0
+
+#define XENON_SYS_EXT_OP_CTRL 0x010C
+#define XENON_MASK_CMD_CONFLICT_ERR BIT(8)
+
+#define XENON_SLOT_OP_STATUS_CTRL 0x0128
+#define XENON_TUN_CONSECUTIVE_TIMES_SHIFT 16
+#define XENON_TUN_CONSECUTIVE_TIMES_MASK 0x7
+#define XENON_TUN_CONSECUTIVE_TIMES 0x4
+#define XENON_TUNING_STEP_SHIFT 12
+#define XENON_TUNING_STEP_MASK 0xF
+#define XENON_TUNING_STEP_DIVIDER BIT(6)
+
+#define XENON_SLOT_EMMC_CTRL 0x0130
+#define XENON_ENABLE_DATA_STROBE BIT(24)
+
+#define XENON_SLOT_RETUNING_REQ_CTRL 0x0144
+/* retuning compatible */
+#define XENON_RETUNING_COMPATIBLE 0x1
+
+#define XENON_SLOT_EXT_PRESENT_STATE 0x014C
+#define XENON_DLL_LOCK_STATE 0x1
+
+#define XENON_SLOT_DLL_CUR_DLY_VAL 0x0150
+
+/* Tuning Parameter */
+#define XENON_TMR_RETUN_NO_PRESENT 0xF
+#define XENON_DEF_TUNING_COUNT 0x9
+
+#define XENON_DEFAULT_SDCLK_FREQ 400000
+#define XENON_LOWEST_SDCLK_FREQ 100000
+
+/* Xenon specific Mode Select value */
+#define XENON_CTRL_HS200 0x5
+#define XENON_CTRL_HS400 0x6
+
+struct xenon_priv {
+ unsigned char tuning_count;
+ /* idx of SDHC */
+ u8 sdhc_id;
+
+ /*
+ * eMMC/SD/SDIO require different register settings.
+ * Xenon driver has to recognize card type
+ * before mmc_host->card is not available.
+ * This field records the card type during init.
+ * It is updated in xenon_init_card().
+ *
+ * It is only valid during initialization after it is updated.
+ * Do not access this variable in normal transfers after
+ * initialization completes.
+ */
+ unsigned int init_card_type;
+
+ /*
+ * The bus_width, timing, and clock fields in below
+ * record the current ios setting of Xenon SDHC.
+ * Driver will adjust PHY setting if any change to
+ * ios affects PHY timing.
+ */
+ unsigned char bus_width;
+ unsigned char timing;
+ unsigned int clock;
+
+ int phy_type;
+ /*
+ * Contains board-specific PHY parameters
+ * passed from device tree.
+ */
+ void *phy_params;
+ struct xenon_emmc_phy_regs *emmc_phy_regs;
+};
+
+int xenon_phy_adj(struct sdhci_host *host, struct mmc_ios *ios);
+void xenon_clean_phy(struct sdhci_host *host);
+int xenon_phy_parse_dt(struct device_node *np,
+ struct sdhci_host *host);
+void xenon_soc_pad_ctrl(struct sdhci_host *host,
+ unsigned char signal_voltage);
+#endif
diff --git a/drivers/mmc/host/sdhci.c b/drivers/mmc/host/sdhci.c
index 63bc33a..ecd0d43 100644
--- a/drivers/mmc/host/sdhci.c
+++ b/drivers/mmc/host/sdhci.c
@@ -14,6 +14,7 @@
*/
#include <linux/delay.h>
+#include <linux/ktime.h>
#include <linux/highmem.h>
#include <linux/io.h>
#include <linux/module.h>
@@ -37,7 +38,10 @@
#define DRIVER_NAME "sdhci"
#define DBG(f, x...) \
- pr_debug(DRIVER_NAME " [%s()]: " f, __func__,## x)
+ pr_debug("%s: " DRIVER_NAME ": " f, mmc_hostname(host->mmc), ## x)
+
+#define SDHCI_DUMP(f, x...) \
+ pr_err("%s: " DRIVER_NAME ": " f, mmc_hostname(host->mmc), ## x)
#define MAX_TUNING_LOOP 40
@@ -48,61 +52,68 @@
static void sdhci_enable_preset_value(struct sdhci_host *host, bool enable);
-static void sdhci_dumpregs(struct sdhci_host *host)
+void sdhci_dumpregs(struct sdhci_host *host)
{
- pr_err(DRIVER_NAME ": =========== REGISTER DUMP (%s)===========\n",
- mmc_hostname(host->mmc));
+ SDHCI_DUMP("============ SDHCI REGISTER DUMP ===========\n");
- pr_err(DRIVER_NAME ": Sys addr: 0x%08x | Version: 0x%08x\n",
- sdhci_readl(host, SDHCI_DMA_ADDRESS),
- sdhci_readw(host, SDHCI_HOST_VERSION));
- pr_err(DRIVER_NAME ": Blk size: 0x%08x | Blk cnt: 0x%08x\n",
- sdhci_readw(host, SDHCI_BLOCK_SIZE),
- sdhci_readw(host, SDHCI_BLOCK_COUNT));
- pr_err(DRIVER_NAME ": Argument: 0x%08x | Trn mode: 0x%08x\n",
- sdhci_readl(host, SDHCI_ARGUMENT),
- sdhci_readw(host, SDHCI_TRANSFER_MODE));
- pr_err(DRIVER_NAME ": Present: 0x%08x | Host ctl: 0x%08x\n",
- sdhci_readl(host, SDHCI_PRESENT_STATE),
- sdhci_readb(host, SDHCI_HOST_CONTROL));
- pr_err(DRIVER_NAME ": Power: 0x%08x | Blk gap: 0x%08x\n",
- sdhci_readb(host, SDHCI_POWER_CONTROL),
- sdhci_readb(host, SDHCI_BLOCK_GAP_CONTROL));
- pr_err(DRIVER_NAME ": Wake-up: 0x%08x | Clock: 0x%08x\n",
- sdhci_readb(host, SDHCI_WAKE_UP_CONTROL),
- sdhci_readw(host, SDHCI_CLOCK_CONTROL));
- pr_err(DRIVER_NAME ": Timeout: 0x%08x | Int stat: 0x%08x\n",
- sdhci_readb(host, SDHCI_TIMEOUT_CONTROL),
- sdhci_readl(host, SDHCI_INT_STATUS));
- pr_err(DRIVER_NAME ": Int enab: 0x%08x | Sig enab: 0x%08x\n",
- sdhci_readl(host, SDHCI_INT_ENABLE),
- sdhci_readl(host, SDHCI_SIGNAL_ENABLE));
- pr_err(DRIVER_NAME ": AC12 err: 0x%08x | Slot int: 0x%08x\n",
- sdhci_readw(host, SDHCI_ACMD12_ERR),
- sdhci_readw(host, SDHCI_SLOT_INT_STATUS));
- pr_err(DRIVER_NAME ": Caps: 0x%08x | Caps_1: 0x%08x\n",
- sdhci_readl(host, SDHCI_CAPABILITIES),
- sdhci_readl(host, SDHCI_CAPABILITIES_1));
- pr_err(DRIVER_NAME ": Cmd: 0x%08x | Max curr: 0x%08x\n",
- sdhci_readw(host, SDHCI_COMMAND),
- sdhci_readl(host, SDHCI_MAX_CURRENT));
- pr_err(DRIVER_NAME ": Host ctl2: 0x%08x\n",
- sdhci_readw(host, SDHCI_HOST_CONTROL2));
+ SDHCI_DUMP("Sys addr: 0x%08x | Version: 0x%08x\n",
+ sdhci_readl(host, SDHCI_DMA_ADDRESS),
+ sdhci_readw(host, SDHCI_HOST_VERSION));
+ SDHCI_DUMP("Blk size: 0x%08x | Blk cnt: 0x%08x\n",
+ sdhci_readw(host, SDHCI_BLOCK_SIZE),
+ sdhci_readw(host, SDHCI_BLOCK_COUNT));
+ SDHCI_DUMP("Argument: 0x%08x | Trn mode: 0x%08x\n",
+ sdhci_readl(host, SDHCI_ARGUMENT),
+ sdhci_readw(host, SDHCI_TRANSFER_MODE));
+ SDHCI_DUMP("Present: 0x%08x | Host ctl: 0x%08x\n",
+ sdhci_readl(host, SDHCI_PRESENT_STATE),
+ sdhci_readb(host, SDHCI_HOST_CONTROL));
+ SDHCI_DUMP("Power: 0x%08x | Blk gap: 0x%08x\n",
+ sdhci_readb(host, SDHCI_POWER_CONTROL),
+ sdhci_readb(host, SDHCI_BLOCK_GAP_CONTROL));
+ SDHCI_DUMP("Wake-up: 0x%08x | Clock: 0x%08x\n",
+ sdhci_readb(host, SDHCI_WAKE_UP_CONTROL),
+ sdhci_readw(host, SDHCI_CLOCK_CONTROL));
+ SDHCI_DUMP("Timeout: 0x%08x | Int stat: 0x%08x\n",
+ sdhci_readb(host, SDHCI_TIMEOUT_CONTROL),
+ sdhci_readl(host, SDHCI_INT_STATUS));
+ SDHCI_DUMP("Int enab: 0x%08x | Sig enab: 0x%08x\n",
+ sdhci_readl(host, SDHCI_INT_ENABLE),
+ sdhci_readl(host, SDHCI_SIGNAL_ENABLE));
+ SDHCI_DUMP("AC12 err: 0x%08x | Slot int: 0x%08x\n",
+ sdhci_readw(host, SDHCI_ACMD12_ERR),
+ sdhci_readw(host, SDHCI_SLOT_INT_STATUS));
+ SDHCI_DUMP("Caps: 0x%08x | Caps_1: 0x%08x\n",
+ sdhci_readl(host, SDHCI_CAPABILITIES),
+ sdhci_readl(host, SDHCI_CAPABILITIES_1));
+ SDHCI_DUMP("Cmd: 0x%08x | Max curr: 0x%08x\n",
+ sdhci_readw(host, SDHCI_COMMAND),
+ sdhci_readl(host, SDHCI_MAX_CURRENT));
+ SDHCI_DUMP("Resp[0]: 0x%08x | Resp[1]: 0x%08x\n",
+ sdhci_readl(host, SDHCI_RESPONSE),
+ sdhci_readl(host, SDHCI_RESPONSE + 4));
+ SDHCI_DUMP("Resp[2]: 0x%08x | Resp[3]: 0x%08x\n",
+ sdhci_readl(host, SDHCI_RESPONSE + 8),
+ sdhci_readl(host, SDHCI_RESPONSE + 12));
+ SDHCI_DUMP("Host ctl2: 0x%08x\n",
+ sdhci_readw(host, SDHCI_HOST_CONTROL2));
if (host->flags & SDHCI_USE_ADMA) {
- if (host->flags & SDHCI_USE_64_BIT_DMA)
- pr_err(DRIVER_NAME ": ADMA Err: 0x%08x | ADMA Ptr: 0x%08x%08x\n",
- readl(host->ioaddr + SDHCI_ADMA_ERROR),
- readl(host->ioaddr + SDHCI_ADMA_ADDRESS_HI),
- readl(host->ioaddr + SDHCI_ADMA_ADDRESS));
- else
- pr_err(DRIVER_NAME ": ADMA Err: 0x%08x | ADMA Ptr: 0x%08x\n",
- readl(host->ioaddr + SDHCI_ADMA_ERROR),
- readl(host->ioaddr + SDHCI_ADMA_ADDRESS));
+ if (host->flags & SDHCI_USE_64_BIT_DMA) {
+ SDHCI_DUMP("ADMA Err: 0x%08x | ADMA Ptr: 0x%08x%08x\n",
+ sdhci_readl(host, SDHCI_ADMA_ERROR),
+ sdhci_readl(host, SDHCI_ADMA_ADDRESS_HI),
+ sdhci_readl(host, SDHCI_ADMA_ADDRESS));
+ } else {
+ SDHCI_DUMP("ADMA Err: 0x%08x | ADMA Ptr: 0x%08x\n",
+ sdhci_readl(host, SDHCI_ADMA_ERROR),
+ sdhci_readl(host, SDHCI_ADMA_ADDRESS));
+ }
}
- pr_err(DRIVER_NAME ": ===========================================\n");
+ SDHCI_DUMP("============================================\n");
}
+EXPORT_SYMBOL_GPL(sdhci_dumpregs);
/*****************************************************************************\
* *
@@ -165,7 +176,7 @@
void sdhci_reset(struct sdhci_host *host, u8 mask)
{
- unsigned long timeout;
+ ktime_t timeout;
sdhci_writeb(host, mask, SDHCI_SOFTWARE_RESET);
@@ -177,18 +188,17 @@
}
/* Wait max 100 ms */
- timeout = 100;
+ timeout = ktime_add_ms(ktime_get(), 100);
/* hw clears the bit when it's done */
while (sdhci_readb(host, SDHCI_SOFTWARE_RESET) & mask) {
- if (timeout == 0) {
+ if (ktime_after(ktime_get(), timeout)) {
pr_err("%s: Reset 0x%x never completed.\n",
mmc_hostname(host->mmc), (int)mask);
sdhci_dumpregs(host);
return;
}
- timeout--;
- mdelay(1);
+ udelay(10);
}
}
EXPORT_SYMBOL_GPL(sdhci_reset);
@@ -215,15 +225,8 @@
}
}
-static void sdhci_init(struct sdhci_host *host, int soft)
+static void sdhci_set_default_irqs(struct sdhci_host *host)
{
- struct mmc_host *mmc = host->mmc;
-
- if (soft)
- sdhci_do_reset(host, SDHCI_RESET_CMD|SDHCI_RESET_DATA);
- else
- sdhci_do_reset(host, SDHCI_RESET_ALL);
-
host->ier = SDHCI_INT_BUS_POWER | SDHCI_INT_DATA_END_BIT |
SDHCI_INT_DATA_CRC | SDHCI_INT_DATA_TIMEOUT |
SDHCI_INT_INDEX | SDHCI_INT_END_BIT | SDHCI_INT_CRC |
@@ -236,6 +239,20 @@
sdhci_writel(host, host->ier, SDHCI_INT_ENABLE);
sdhci_writel(host, host->ier, SDHCI_SIGNAL_ENABLE);
+}
+
+static void sdhci_init(struct sdhci_host *host, int soft)
+{
+ struct mmc_host *mmc = host->mmc;
+
+ if (soft)
+ sdhci_do_reset(host, SDHCI_RESET_CMD | SDHCI_RESET_DATA);
+ else
+ sdhci_do_reset(host, SDHCI_RESET_ALL);
+
+ sdhci_set_default_irqs(host);
+
+ host->cqe_on = false;
if (soft) {
/* force clock reconfiguration */
@@ -485,8 +502,7 @@
return data->sg_count;
sg_count = dma_map_sg(mmc_dev(host->mmc), data->sg, data->sg_len,
- data->flags & MMC_DATA_WRITE ?
- DMA_TO_DEVICE : DMA_FROM_DEVICE);
+ mmc_get_dma_dir(data));
if (sg_count == 0)
return -ENOSPC;
@@ -715,8 +731,8 @@
}
if (count >= 0xF) {
- DBG("%s: Too large timeout 0x%x requested for CMD%d!\n",
- mmc_hostname(host->mmc), count, cmd->opcode);
+ DBG("Too large timeout 0x%x requested for CMD%d!\n",
+ count, cmd->opcode);
count = 0xE;
}
@@ -1346,25 +1362,22 @@
void sdhci_enable_clk(struct sdhci_host *host, u16 clk)
{
- unsigned long timeout;
+ ktime_t timeout;
clk |= SDHCI_CLOCK_INT_EN;
sdhci_writew(host, clk, SDHCI_CLOCK_CONTROL);
/* Wait max 20 ms */
- timeout = 20;
+ timeout = ktime_add_ms(ktime_get(), 20);
while (!((clk = sdhci_readw(host, SDHCI_CLOCK_CONTROL))
& SDHCI_CLOCK_INT_STABLE)) {
- if (timeout == 0) {
+ if (ktime_after(ktime_get(), timeout)) {
pr_err("%s: Internal clock never stabilised.\n",
mmc_hostname(host->mmc));
sdhci_dumpregs(host);
return;
}
- timeout--;
- spin_unlock_irq(&host->lock);
- usleep_range(900, 1100);
- spin_lock_irq(&host->lock);
+ udelay(10);
}
clk |= SDHCI_CLOCK_CARD_EN;
@@ -1393,9 +1406,7 @@
{
struct mmc_host *mmc = host->mmc;
- spin_unlock_irq(&host->lock);
mmc_regulator_set_ocr(mmc, mmc->supply.vmmc, vdd);
- spin_lock_irq(&host->lock);
if (mode != MMC_POWER_OFF)
sdhci_writeb(host, SDHCI_POWER_ON, SDHCI_POWER_CONTROL);
@@ -1572,19 +1583,15 @@
}
EXPORT_SYMBOL_GPL(sdhci_set_uhs_signaling);
-static void sdhci_set_ios(struct mmc_host *mmc, struct mmc_ios *ios)
+void sdhci_set_ios(struct mmc_host *mmc, struct mmc_ios *ios)
{
struct sdhci_host *host = mmc_priv(mmc);
- unsigned long flags;
u8 ctrl;
if (ios->power_mode == MMC_POWER_UNDEFINED)
return;
- spin_lock_irqsave(&host->lock, flags);
-
if (host->flags & SDHCI_DEVICE_DEAD) {
- spin_unlock_irqrestore(&host->lock, flags);
if (!IS_ERR(mmc->supply.vmmc) &&
ios->power_mode == MMC_POWER_OFF)
mmc_regulator_set_ocr(mmc, mmc->supply.vmmc, 0);
@@ -1730,8 +1737,8 @@
sdhci_do_reset(host, SDHCI_RESET_CMD | SDHCI_RESET_DATA);
mmiowb();
- spin_unlock_irqrestore(&host->lock, flags);
}
+EXPORT_SYMBOL_GPL(sdhci_set_ios);
static int sdhci_get_cd(struct mmc_host *mmc)
{
@@ -1825,7 +1832,7 @@
}
}
-static void sdhci_enable_sdio_irq(struct mmc_host *mmc, int enable)
+void sdhci_enable_sdio_irq(struct mmc_host *mmc, int enable)
{
struct sdhci_host *host = mmc_priv(mmc);
unsigned long flags;
@@ -1845,9 +1852,10 @@
if (!enable)
pm_runtime_put_noidle(host->mmc->parent);
}
+EXPORT_SYMBOL_GPL(sdhci_enable_sdio_irq);
-static int sdhci_start_signal_voltage_switch(struct mmc_host *mmc,
- struct mmc_ios *ios)
+int sdhci_start_signal_voltage_switch(struct mmc_host *mmc,
+ struct mmc_ios *ios)
{
struct sdhci_host *host = mmc_priv(mmc);
u16 ctrl;
@@ -1939,6 +1947,7 @@
return 0;
}
}
+EXPORT_SYMBOL_GPL(sdhci_start_signal_voltage_switch);
static int sdhci_card_busy(struct mmc_host *mmc)
{
@@ -2003,8 +2012,7 @@
sdhci_writew(host, ctrl, SDHCI_HOST_CONTROL2);
}
-static void sdhci_abort_tuning(struct sdhci_host *host, u32 opcode,
- unsigned long flags)
+static void sdhci_abort_tuning(struct sdhci_host *host, u32 opcode)
{
sdhci_reset_tuning(host);
@@ -2013,9 +2021,7 @@
sdhci_end_tuning(host);
- spin_unlock_irqrestore(&host->lock, flags);
mmc_abort_tuning(host->mmc, opcode);
- spin_lock_irqsave(&host->lock, flags);
}
/*
@@ -2025,12 +2031,14 @@
* interrupt setup is different to other commands and there is no timeout
* interrupt so special handling is needed.
*/
-static void sdhci_send_tuning(struct sdhci_host *host, u32 opcode,
- unsigned long flags)
+static void sdhci_send_tuning(struct sdhci_host *host, u32 opcode)
{
struct mmc_host *mmc = host->mmc;
struct mmc_command cmd = {};
struct mmc_request mrq = {};
+ unsigned long flags;
+
+ spin_lock_irqsave(&host->lock, flags);
cmd.opcode = opcode;
cmd.flags = MMC_RSP_R1 | MMC_CMD_ADTC;
@@ -2064,17 +2072,16 @@
host->tuning_done = 0;
+ mmiowb();
spin_unlock_irqrestore(&host->lock, flags);
/* Wait for Buffer Read Ready interrupt */
wait_event_timeout(host->buf_ready_int, (host->tuning_done == 1),
msecs_to_jiffies(50));
- spin_lock_irqsave(&host->lock, flags);
}
-static void __sdhci_execute_tuning(struct sdhci_host *host, u32 opcode,
- unsigned long flags)
+static void __sdhci_execute_tuning(struct sdhci_host *host, u32 opcode)
{
int i;
@@ -2085,12 +2092,12 @@
for (i = 0; i < MAX_TUNING_LOOP; i++) {
u16 ctrl;
- sdhci_send_tuning(host, opcode, flags);
+ sdhci_send_tuning(host, opcode);
if (!host->tuning_done) {
pr_info("%s: Tuning timeout, falling back to fixed sampling clock\n",
mmc_hostname(host->mmc));
- sdhci_abort_tuning(host, opcode, flags);
+ sdhci_abort_tuning(host, opcode);
return;
}
@@ -2101,9 +2108,9 @@
break;
}
- /* eMMC spec does not require a delay between tuning cycles */
- if (opcode == MMC_SEND_TUNING_BLOCK)
- mdelay(1);
+ /* Spec does not require a delay between tuning cycles */
+ if (host->tuning_delay > 0)
+ mdelay(host->tuning_delay);
}
pr_info("%s: Tuning failed, falling back to fixed sampling clock\n",
@@ -2115,12 +2122,9 @@
{
struct sdhci_host *host = mmc_priv(mmc);
int err = 0;
- unsigned long flags;
unsigned int tuning_count = 0;
bool hs400_tuning;
- spin_lock_irqsave(&host->lock, flags);
-
hs400_tuning = host->flags & SDHCI_HS400_TUNING;
if (host->tuning_mode == SDHCI_TUNING_MODE_1)
@@ -2137,7 +2141,7 @@
/* HS400 tuning is done in HS200 mode */
case MMC_TIMING_MMC_HS400:
err = -EINVAL;
- goto out_unlock;
+ goto out;
case MMC_TIMING_MMC_HS200:
/*
@@ -2158,44 +2162,31 @@
/* FALLTHROUGH */
default:
- goto out_unlock;
+ goto out;
}
if (host->ops->platform_execute_tuning) {
- spin_unlock_irqrestore(&host->lock, flags);
err = host->ops->platform_execute_tuning(host, opcode);
- spin_lock_irqsave(&host->lock, flags);
- goto out_unlock;
+ goto out;
}
host->mmc->retune_period = tuning_count;
+ if (host->tuning_delay < 0)
+ host->tuning_delay = opcode == MMC_SEND_TUNING_BLOCK;
+
sdhci_start_tuning(host);
- __sdhci_execute_tuning(host, opcode, flags);
+ __sdhci_execute_tuning(host, opcode);
sdhci_end_tuning(host);
-out_unlock:
+out:
host->flags &= ~SDHCI_HS400_TUNING;
- spin_unlock_irqrestore(&host->lock, flags);
return err;
}
EXPORT_SYMBOL_GPL(sdhci_execute_tuning);
-static int sdhci_select_drive_strength(struct mmc_card *card,
- unsigned int max_dtr, int host_drv,
- int card_drv, int *drv_type)
-{
- struct sdhci_host *host = mmc_priv(card->host);
-
- if (!host->ops->select_drive_strength)
- return 0;
-
- return host->ops->select_drive_strength(host, card, max_dtr, host_drv,
- card_drv, drv_type);
-}
-
static void sdhci_enable_preset_value(struct sdhci_host *host, bool enable)
{
/* Host Controller v3.00 defines preset value registers */
@@ -2233,8 +2224,7 @@
if (data->host_cookie != COOKIE_UNMAPPED)
dma_unmap_sg(mmc_dev(host->mmc), data->sg, data->sg_len,
- data->flags & MMC_DATA_WRITE ?
- DMA_TO_DEVICE : DMA_FROM_DEVICE);
+ mmc_get_dma_dir(data));
data->host_cookie = COOKIE_UNMAPPED;
}
@@ -2309,7 +2299,6 @@
.start_signal_voltage_switch = sdhci_start_signal_voltage_switch,
.prepare_hs400_tuning = sdhci_prepare_hs400_tuning,
.execute_tuning = sdhci_execute_tuning,
- .select_drive_strength = sdhci_select_drive_strength,
.card_event = sdhci_card_event,
.card_busy = sdhci_card_busy,
};
@@ -2351,8 +2340,7 @@
if (data && data->host_cookie == COOKIE_MAPPED) {
dma_unmap_sg(mmc_dev(host->mmc), data->sg, data->sg_len,
- (data->flags & MMC_DATA_READ) ?
- DMA_FROM_DEVICE : DMA_TO_DEVICE);
+ mmc_get_dma_dir(data));
data->host_cookie = COOKIE_UNMAPPED;
}
}
@@ -2517,7 +2505,6 @@
#ifdef CONFIG_MMC_DEBUG
static void sdhci_adma_show_error(struct sdhci_host *host)
{
- const char *name = mmc_hostname(host->mmc);
void *desc = host->adma_table;
sdhci_dumpregs(host);
@@ -2526,14 +2513,14 @@
struct sdhci_adma2_64_desc *dma_desc = desc;
if (host->flags & SDHCI_USE_64_BIT_DMA)
- DBG("%s: %p: DMA 0x%08x%08x, LEN 0x%04x, Attr=0x%02x\n",
- name, desc, le32_to_cpu(dma_desc->addr_hi),
+ DBG("%p: DMA 0x%08x%08x, LEN 0x%04x, Attr=0x%02x\n",
+ desc, le32_to_cpu(dma_desc->addr_hi),
le32_to_cpu(dma_desc->addr_lo),
le16_to_cpu(dma_desc->len),
le16_to_cpu(dma_desc->cmd));
else
- DBG("%s: %p: DMA 0x%08x, LEN 0x%04x, Attr=0x%02x\n",
- name, desc, le32_to_cpu(dma_desc->addr_lo),
+ DBG("%p: DMA 0x%08x, LEN 0x%04x, Attr=0x%02x\n",
+ desc, le32_to_cpu(dma_desc->addr_lo),
le16_to_cpu(dma_desc->len),
le16_to_cpu(dma_desc->cmd));
@@ -2649,10 +2636,8 @@
~(SDHCI_DEFAULT_BOUNDARY_SIZE - 1)) +
SDHCI_DEFAULT_BOUNDARY_SIZE;
host->data->bytes_xfered = dmanow - dmastart;
- DBG("%s: DMA base 0x%08x, transferred 0x%06x bytes,"
- " next 0x%08x\n",
- mmc_hostname(host->mmc), dmastart,
- host->data->bytes_xfered, dmanow);
+ DBG("DMA base 0x%08x, transferred 0x%06x bytes, next 0x%08x\n",
+ dmastart, host->data->bytes_xfered, dmanow);
sdhci_writel(host, dmanow, SDHCI_DMA_ADDRESS);
}
@@ -2692,14 +2677,19 @@
}
do {
+ DBG("IRQ status 0x%08x\n", intmask);
+
+ if (host->ops->irq) {
+ intmask = host->ops->irq(host, intmask);
+ if (!intmask)
+ goto cont;
+ }
+
/* Clear selected interrupts. */
mask = intmask & (SDHCI_INT_CMD_MASK | SDHCI_INT_DATA_MASK |
SDHCI_INT_BUS_POWER);
sdhci_writel(host, mask, SDHCI_INT_STATUS);
- DBG("*** %s got interrupt: 0x%08x\n",
- mmc_hostname(host->mmc), intmask);
-
if (intmask & (SDHCI_INT_CARD_INSERT | SDHCI_INT_CARD_REMOVE)) {
u32 present = sdhci_readl(host, SDHCI_PRESENT_STATE) &
SDHCI_CARD_PRESENT;
@@ -2759,7 +2749,7 @@
unexpected |= intmask;
sdhci_writel(host, intmask, SDHCI_INT_STATUS);
}
-
+cont:
if (result == IRQ_NONE)
result = IRQ_HANDLED;
@@ -2858,8 +2848,6 @@
sdhci_disable_card_detection(host);
mmc_retune_timer_stop(host->mmc);
- if (host->tuning_mode != SDHCI_TUNING_MODE_3)
- mmc_retune_needed(host->mmc);
if (!device_may_wakeup(mmc_dev(host->mmc))) {
host->ier = 0;
@@ -2920,8 +2908,6 @@
unsigned long flags;
mmc_retune_timer_stop(host->mmc);
- if (host->tuning_mode != SDHCI_TUNING_MODE_3)
- mmc_retune_needed(host->mmc);
spin_lock_irqsave(&host->lock, flags);
host->ier &= SDHCI_INT_CARD_INT;
@@ -2992,6 +2978,119 @@
/*****************************************************************************\
* *
+ * Command Queue Engine (CQE) helpers *
+ * *
+\*****************************************************************************/
+
+void sdhci_cqe_enable(struct mmc_host *mmc)
+{
+ struct sdhci_host *host = mmc_priv(mmc);
+ unsigned long flags;
+ u8 ctrl;
+
+ spin_lock_irqsave(&host->lock, flags);
+
+ ctrl = sdhci_readb(host, SDHCI_HOST_CONTROL);
+ ctrl &= ~SDHCI_CTRL_DMA_MASK;
+ if (host->flags & SDHCI_USE_64_BIT_DMA)
+ ctrl |= SDHCI_CTRL_ADMA64;
+ else
+ ctrl |= SDHCI_CTRL_ADMA32;
+ sdhci_writeb(host, ctrl, SDHCI_HOST_CONTROL);
+
+ sdhci_writew(host, SDHCI_MAKE_BLKSZ(SDHCI_DEFAULT_BOUNDARY_ARG, 512),
+ SDHCI_BLOCK_SIZE);
+
+ /* Set maximum timeout */
+ sdhci_writeb(host, 0xE, SDHCI_TIMEOUT_CONTROL);
+
+ host->ier = host->cqe_ier;
+
+ sdhci_writel(host, host->ier, SDHCI_INT_ENABLE);
+ sdhci_writel(host, host->ier, SDHCI_SIGNAL_ENABLE);
+
+ host->cqe_on = true;
+
+ pr_debug("%s: sdhci: CQE on, IRQ mask %#x, IRQ status %#x\n",
+ mmc_hostname(mmc), host->ier,
+ sdhci_readl(host, SDHCI_INT_STATUS));
+
+ mmiowb();
+ spin_unlock_irqrestore(&host->lock, flags);
+}
+EXPORT_SYMBOL_GPL(sdhci_cqe_enable);
+
+void sdhci_cqe_disable(struct mmc_host *mmc, bool recovery)
+{
+ struct sdhci_host *host = mmc_priv(mmc);
+ unsigned long flags;
+
+ spin_lock_irqsave(&host->lock, flags);
+
+ sdhci_set_default_irqs(host);
+
+ host->cqe_on = false;
+
+ if (recovery) {
+ sdhci_do_reset(host, SDHCI_RESET_CMD);
+ sdhci_do_reset(host, SDHCI_RESET_DATA);
+ }
+
+ pr_debug("%s: sdhci: CQE off, IRQ mask %#x, IRQ status %#x\n",
+ mmc_hostname(mmc), host->ier,
+ sdhci_readl(host, SDHCI_INT_STATUS));
+
+ mmiowb();
+ spin_unlock_irqrestore(&host->lock, flags);
+}
+EXPORT_SYMBOL_GPL(sdhci_cqe_disable);
+
+bool sdhci_cqe_irq(struct sdhci_host *host, u32 intmask, int *cmd_error,
+ int *data_error)
+{
+ u32 mask;
+
+ if (!host->cqe_on)
+ return false;
+
+ if (intmask & (SDHCI_INT_INDEX | SDHCI_INT_END_BIT | SDHCI_INT_CRC))
+ *cmd_error = -EILSEQ;
+ else if (intmask & SDHCI_INT_TIMEOUT)
+ *cmd_error = -ETIMEDOUT;
+ else
+ *cmd_error = 0;
+
+ if (intmask & (SDHCI_INT_DATA_END_BIT | SDHCI_INT_DATA_CRC))
+ *data_error = -EILSEQ;
+ else if (intmask & SDHCI_INT_DATA_TIMEOUT)
+ *data_error = -ETIMEDOUT;
+ else if (intmask & SDHCI_INT_ADMA_ERROR)
+ *data_error = -EIO;
+ else
+ *data_error = 0;
+
+ /* Clear selected interrupts. */
+ mask = intmask & host->cqe_ier;
+ sdhci_writel(host, mask, SDHCI_INT_STATUS);
+
+ if (intmask & SDHCI_INT_BUS_POWER)
+ pr_err("%s: Card is consuming too much power!\n",
+ mmc_hostname(host->mmc));
+
+ intmask &= ~(host->cqe_ier | SDHCI_INT_ERROR);
+ if (intmask) {
+ sdhci_writel(host, intmask, SDHCI_INT_STATUS);
+ pr_err("%s: CQE: Unexpected interrupt 0x%08x.\n",
+ mmc_hostname(host->mmc), intmask);
+ sdhci_dumpregs(host);
+ }
+
+ return true;
+}
+EXPORT_SYMBOL_GPL(sdhci_cqe_irq);
+
+/*****************************************************************************\
+ * *
* Device allocation/registration *
* *
\*****************************************************************************/
@@ -3015,6 +3114,11 @@
host->flags = SDHCI_SIGNALING_330;
+ host->cqe_ier = SDHCI_CQE_INT_MASK;
+ host->cqe_err_ier = SDHCI_CQE_INT_ERR_MASK;
+
+ host->tuning_delay = -1;
+
return host;
}
@@ -3297,20 +3401,22 @@
if (!(host->quirks & SDHCI_QUIRK_DATA_TIMEOUT_USES_SDCLK)) {
host->timeout_clk = (host->caps & SDHCI_TIMEOUT_CLK_MASK) >>
SDHCI_TIMEOUT_CLK_SHIFT;
+
+ if (host->caps & SDHCI_TIMEOUT_CLK_UNIT)
+ host->timeout_clk *= 1000;
+
if (host->timeout_clk == 0) {
- if (host->ops->get_timeout_clock) {
- host->timeout_clk =
- host->ops->get_timeout_clock(host);
- } else {
+ if (!host->ops->get_timeout_clock) {
pr_err("%s: Hardware doesn't specify timeout clock frequency.\n",
mmc_hostname(mmc));
ret = -ENODEV;
goto undma;
}
- }
- if (host->caps & SDHCI_TIMEOUT_CLK_UNIT)
- host->timeout_clk *= 1000;
+ host->timeout_clk =
+ DIV_ROUND_UP(host->ops->get_timeout_clock(host),
+ 1000);
+ }
if (override_timeout_clk)
host->timeout_clk = override_timeout_clk;
@@ -3332,9 +3438,9 @@
!(host->flags & SDHCI_USE_SDMA)) &&
!(host->quirks2 & SDHCI_QUIRK2_ACMD23_BROKEN)) {
host->flags |= SDHCI_AUTO_CMD23;
- DBG("%s: Auto-CMD23 available\n", mmc_hostname(mmc));
+ DBG("Auto-CMD23 available\n");
} else {
- DBG("%s: Auto-CMD23 unavailable\n", mmc_hostname(mmc));
+ DBG("Auto-CMD23 unavailable\n");
}
/*
@@ -3598,6 +3704,22 @@
}
EXPORT_SYMBOL_GPL(sdhci_setup_host);
+void sdhci_cleanup_host(struct sdhci_host *host)
+{
+ struct mmc_host *mmc = host->mmc;
+
+ if (!IS_ERR(mmc->supply.vqmmc))
+ regulator_disable(mmc->supply.vqmmc);
+
+ if (host->align_buffer)
+ dma_free_coherent(mmc_dev(mmc), host->align_buffer_sz +
+ host->adma_table_sz, host->align_buffer,
+ host->align_addr);
+ host->adma_table = NULL;
+ host->align_buffer = NULL;
+}
+EXPORT_SYMBOL_GPL(sdhci_cleanup_host);
+
int __sdhci_add_host(struct sdhci_host *host)
{
struct mmc_host *mmc = host->mmc;
@@ -3662,16 +3784,6 @@
untasklet:
tasklet_kill(&host->finish_tasklet);
- if (!IS_ERR(mmc->supply.vqmmc))
- regulator_disable(mmc->supply.vqmmc);
-
- if (host->align_buffer)
- dma_free_coherent(mmc_dev(mmc), host->align_buffer_sz +
- host->adma_table_sz, host->align_buffer,
- host->align_addr);
- host->adma_table = NULL;
- host->align_buffer = NULL;
-
return ret;
}
EXPORT_SYMBOL_GPL(__sdhci_add_host);
@@ -3684,7 +3796,16 @@
if (ret)
return ret;
- return __sdhci_add_host(host);
+ ret = __sdhci_add_host(host);
+ if (ret)
+ goto cleanup;
+
+ return 0;
+
+cleanup:
+ sdhci_cleanup_host(host);
+
+ return ret;
}
EXPORT_SYMBOL_GPL(sdhci_add_host);
diff --git a/drivers/mmc/host/sdhci.h b/drivers/mmc/host/sdhci.h
index edf3adf..0469fa1 100644
--- a/drivers/mmc/host/sdhci.h
+++ b/drivers/mmc/host/sdhci.h
@@ -134,6 +134,7 @@
#define SDHCI_INT_CARD_REMOVE 0x00000080
#define SDHCI_INT_CARD_INT 0x00000100
#define SDHCI_INT_RETUNE 0x00001000
+#define SDHCI_INT_CQE 0x00004000
#define SDHCI_INT_ERROR 0x00008000
#define SDHCI_INT_TIMEOUT 0x00010000
#define SDHCI_INT_CRC 0x00020000
@@ -158,6 +159,13 @@
SDHCI_INT_BLK_GAP)
#define SDHCI_INT_ALL_MASK ((unsigned int)-1)
+#define SDHCI_CQE_INT_ERR_MASK ( \
+ SDHCI_INT_ADMA_ERROR | SDHCI_INT_BUS_POWER | SDHCI_INT_DATA_END_BIT | \
+ SDHCI_INT_DATA_CRC | SDHCI_INT_DATA_TIMEOUT | SDHCI_INT_INDEX | \
+ SDHCI_INT_END_BIT | SDHCI_INT_CRC | SDHCI_INT_TIMEOUT)
+
+#define SDHCI_CQE_INT_MASK (SDHCI_CQE_INT_ERR_MASK | SDHCI_INT_CQE)
+
#define SDHCI_ACMD12_ERR 0x3C
#define SDHCI_HOST_CONTROL2 0x3E
@@ -518,6 +526,10 @@
/* cached registers */
u32 ier;
+ bool cqe_on; /* CQE is operating */
+ u32 cqe_ier; /* CQE interrupt mask */
+ u32 cqe_err_ier; /* CQE error interrupt mask */
+
wait_queue_head_t buf_ready_int; /* Waitqueue for Buffer Read Ready interrupt */
unsigned int tuning_done; /* Condition flag set when CMD19 succeeds */
@@ -526,6 +538,8 @@
#define SDHCI_TUNING_MODE_1 0
#define SDHCI_TUNING_MODE_2 1
#define SDHCI_TUNING_MODE_3 2
+ /* Delay (ms) between tuning commands */
+ int tuning_delay;
unsigned long private[0] ____cacheline_aligned;
};
@@ -544,9 +558,12 @@
void (*set_power)(struct sdhci_host *host, unsigned char mode,
unsigned short vdd);
+ u32 (*irq)(struct sdhci_host *host, u32 intmask);
+
int (*enable_dma)(struct sdhci_host *host);
unsigned int (*get_max_clock)(struct sdhci_host *host);
unsigned int (*get_min_clock)(struct sdhci_host *host);
+ /* get_timeout_clock should return clk rate in unit of Hz */
unsigned int (*get_timeout_clock)(struct sdhci_host *host);
unsigned int (*get_max_timeout_count)(struct sdhci_host *host);
void (*set_timeout)(struct sdhci_host *host,
@@ -562,10 +579,6 @@
void (*adma_workaround)(struct sdhci_host *host, u32 intmask);
void (*card_event)(struct sdhci_host *host);
void (*voltage_switch)(struct sdhci_host *host);
- int (*select_drive_strength)(struct sdhci_host *host,
- struct mmc_card *card,
- unsigned int max_dtr, int host_drv,
- int card_drv, int *drv_type);
};
#ifdef CONFIG_MMC_SDHCI_IO_ACCESSORS
@@ -652,24 +665,23 @@
#endif /* CONFIG_MMC_SDHCI_IO_ACCESSORS */
-extern struct sdhci_host *sdhci_alloc_host(struct device *dev,
- size_t priv_size);
-extern void sdhci_free_host(struct sdhci_host *host);
+struct sdhci_host *sdhci_alloc_host(struct device *dev, size_t priv_size);
+void sdhci_free_host(struct sdhci_host *host);
static inline void *sdhci_priv(struct sdhci_host *host)
{
return host->private;
}
-extern void sdhci_card_detect(struct sdhci_host *host);
-extern void __sdhci_read_caps(struct sdhci_host *host, u16 *ver, u32 *caps,
- u32 *caps1);
-extern int sdhci_setup_host(struct sdhci_host *host);
-extern int __sdhci_add_host(struct sdhci_host *host);
-extern int sdhci_add_host(struct sdhci_host *host);
-extern void sdhci_remove_host(struct sdhci_host *host, int dead);
-extern void sdhci_send_command(struct sdhci_host *host,
- struct mmc_command *cmd);
+void sdhci_card_detect(struct sdhci_host *host);
+void __sdhci_read_caps(struct sdhci_host *host, u16 *ver, u32 *caps,
+ u32 *caps1);
+int sdhci_setup_host(struct sdhci_host *host);
+void sdhci_cleanup_host(struct sdhci_host *host);
+int __sdhci_add_host(struct sdhci_host *host);
+int sdhci_add_host(struct sdhci_host *host);
+void sdhci_remove_host(struct sdhci_host *host, int dead);
+void sdhci_send_command(struct sdhci_host *host, struct mmc_command *cmd);
static inline void sdhci_read_caps(struct sdhci_host *host)
{
@@ -693,13 +705,24 @@
void sdhci_reset(struct sdhci_host *host, u8 mask);
void sdhci_set_uhs_signaling(struct sdhci_host *host, unsigned timing);
int sdhci_execute_tuning(struct mmc_host *mmc, u32 opcode);
+void sdhci_set_ios(struct mmc_host *mmc, struct mmc_ios *ios);
+int sdhci_start_signal_voltage_switch(struct mmc_host *mmc,
+ struct mmc_ios *ios);
+void sdhci_enable_sdio_irq(struct mmc_host *mmc, int enable);
#ifdef CONFIG_PM
-extern int sdhci_suspend_host(struct sdhci_host *host);
-extern int sdhci_resume_host(struct sdhci_host *host);
-extern void sdhci_enable_irq_wakeups(struct sdhci_host *host);
-extern int sdhci_runtime_suspend_host(struct sdhci_host *host);
-extern int sdhci_runtime_resume_host(struct sdhci_host *host);
+int sdhci_suspend_host(struct sdhci_host *host);
+int sdhci_resume_host(struct sdhci_host *host);
+void sdhci_enable_irq_wakeups(struct sdhci_host *host);
+int sdhci_runtime_suspend_host(struct sdhci_host *host);
+int sdhci_runtime_resume_host(struct sdhci_host *host);
#endif
+void sdhci_cqe_enable(struct mmc_host *mmc);
+void sdhci_cqe_disable(struct mmc_host *mmc, bool recovery);
+bool sdhci_cqe_irq(struct sdhci_host *host, u32 intmask, int *cmd_error,
+ int *data_error);
+
+void sdhci_dumpregs(struct sdhci_host *host);
+
#endif /* __SDHCI_HW_H */
diff --git a/drivers/mmc/host/sunxi-mmc.c b/drivers/mmc/host/sunxi-mmc.c
index 6ffcd28..d6fa221 100644
--- a/drivers/mmc/host/sunxi-mmc.c
+++ b/drivers/mmc/host/sunxi-mmc.c
@@ -385,14 +385,6 @@
wmb();
}
-static enum dma_data_direction sunxi_mmc_get_dma_dir(struct mmc_data *data)
-{
- if (data->flags & MMC_DATA_WRITE)
- return DMA_TO_DEVICE;
- else
- return DMA_FROM_DEVICE;
-}
-
static int sunxi_mmc_map_dma(struct sunxi_mmc_host *host,
struct mmc_data *data)
{
@@ -400,7 +392,7 @@
struct scatterlist *sg;
dma_len = dma_map_sg(mmc_dev(host->mmc), data->sg, data->sg_len,
- sunxi_mmc_get_dma_dir(data));
+ mmc_get_dma_dir(data));
if (dma_len == 0) {
dev_err(mmc_dev(host->mmc), "dma_map_sg failed\n");
return -ENOMEM;
@@ -489,7 +481,7 @@
cmd->opcode == SD_IO_RW_DIRECT))
return;
- dev_err(mmc_dev(host->mmc),
+ dev_dbg(mmc_dev(host->mmc),
"smc %d err, cmd %d,%s%s%s%s%s%s%s%s%s%s !!\n",
host->mmc->index, cmd->opcode,
data ? (data->flags & MMC_DATA_WRITE ? " WR" : " RD") : "",
@@ -551,7 +543,7 @@
rval |= SDXC_FIFO_RESET;
mmc_writel(host, REG_GCTRL, rval);
dma_unmap_sg(mmc_dev(host->mmc), data->sg, data->sg_len,
- sunxi_mmc_get_dma_dir(data));
+ mmc_get_dma_dir(data));
}
mmc_writel(host, REG_RINTR, 0xffff);
@@ -1022,7 +1014,7 @@
if (data)
dma_unmap_sg(mmc_dev(mmc), data->sg, data->sg_len,
- sunxi_mmc_get_dma_dir(data));
+ mmc_get_dma_dir(data));
dev_err(mmc_dev(mmc), "request already pending\n");
mrq->cmd->error = -EBUSY;
diff --git a/drivers/mmc/host/tmio_mmc.h b/drivers/mmc/host/tmio_mmc.h
index 2b349d4..d0edb57 100644
--- a/drivers/mmc/host/tmio_mmc.h
+++ b/drivers/mmc/host/tmio_mmc.h
@@ -50,7 +50,11 @@
#define CTL_CLK_AND_WAIT_CTL 0x138
#define CTL_RESET_SDIO 0x1e0
-/* Definitions for values the CTRL_STATUS register can take. */
+/* Definitions for values the CTL_STOP_INTERNAL_ACTION register can take */
+#define TMIO_STOP_STP BIT(0)
+#define TMIO_STOP_SEC BIT(8)
+
+/* Definitions for values the CTL_STATUS register can take */
#define TMIO_STAT_CMDRESPEND BIT(0)
#define TMIO_STAT_DATAEND BIT(2)
#define TMIO_STAT_CARD_REMOVE BIT(3)
@@ -61,7 +65,7 @@
#define TMIO_STAT_CARD_INSERT_A BIT(9)
#define TMIO_STAT_SIGSTATE_A BIT(10)
-/* These belong technically to CTRL_STATUS2, but the driver merges them */
+/* These belong technically to CTL_STATUS2, but the driver merges them */
#define TMIO_STAT_CMD_IDX_ERR BIT(16)
#define TMIO_STAT_CRCFAIL BIT(17)
#define TMIO_STAT_STOPBIT_ERR BIT(18)
@@ -85,7 +89,7 @@
#define TMIO_BBS 512 /* Boot block size */
-/* Definitions for values the CTRL_SDIO_STATUS register can take. */
+/* Definitions for values the CTL_SDIO_STATUS register can take */
#define TMIO_SDIO_STAT_IOIRQ 0x0001
#define TMIO_SDIO_STAT_EXPUB52 0x4000
#define TMIO_SDIO_STAT_EXWT 0x8000
@@ -137,7 +141,7 @@
bool force_pio;
struct dma_chan *chan_rx;
struct dma_chan *chan_tx;
- struct tasklet_struct dma_complete;
+ struct completion dma_dataend;
struct tasklet_struct dma_issue;
struct scatterlist bounce_sg;
u8 *bounce_buf;
diff --git a/drivers/mmc/host/tmio_mmc_dma.c b/drivers/mmc/host/tmio_mmc_dma.c
index fa8a936..e2093db 100644
--- a/drivers/mmc/host/tmio_mmc_dma.c
+++ b/drivers/mmc/host/tmio_mmc_dma.c
@@ -43,6 +43,34 @@
tmio_mmc_enable_dma(host, true);
}
+static void tmio_mmc_dma_callback(void *arg)
+{
+ struct tmio_mmc_host *host = arg;
+
+ spin_lock_irq(&host->lock);
+
+ if (!host->data)
+ goto out;
+
+ if (host->data->flags & MMC_DATA_READ)
+ dma_unmap_sg(host->chan_rx->device->dev,
+ host->sg_ptr, host->sg_len,
+ DMA_FROM_DEVICE);
+ else
+ dma_unmap_sg(host->chan_tx->device->dev,
+ host->sg_ptr, host->sg_len,
+ DMA_TO_DEVICE);
+
+ spin_unlock_irq(&host->lock);
+
+ wait_for_completion(&host->dma_dataend);
+
+ spin_lock_irq(&host->lock);
+ tmio_mmc_do_data_irq(host);
+out:
+ spin_unlock_irq(&host->lock);
+}
+
static void tmio_mmc_start_dma_rx(struct tmio_mmc_host *host)
{
struct scatterlist *sg = host->sg_ptr, *sg_tmp;
@@ -88,6 +116,10 @@
DMA_DEV_TO_MEM, DMA_CTRL_ACK);
if (desc) {
+ reinit_completion(&host->dma_dataend);
+ desc->callback = tmio_mmc_dma_callback;
+ desc->callback_param = host;
+
cookie = dmaengine_submit(desc);
if (cookie < 0) {
desc = NULL;
@@ -162,6 +194,10 @@
DMA_MEM_TO_DEV, DMA_CTRL_ACK);
if (desc) {
+ reinit_completion(&host->dma_dataend);
+ desc->callback = tmio_mmc_dma_callback;
+ desc->callback_param = host;
+
cookie = dmaengine_submit(desc);
if (cookie < 0) {
desc = NULL;
@@ -221,29 +257,6 @@
dma_async_issue_pending(chan);
}
-static void tmio_mmc_tasklet_fn(unsigned long arg)
-{
- struct tmio_mmc_host *host = (struct tmio_mmc_host *)arg;
-
- spin_lock_irq(&host->lock);
-
- if (!host->data)
- goto out;
-
- if (host->data->flags & MMC_DATA_READ)
- dma_unmap_sg(host->chan_rx->device->dev,
- host->sg_ptr, host->sg_len,
- DMA_FROM_DEVICE);
- else
- dma_unmap_sg(host->chan_tx->device->dev,
- host->sg_ptr, host->sg_len,
- DMA_TO_DEVICE);
-
- tmio_mmc_do_data_irq(host);
-out:
- spin_unlock_irq(&host->lock);
-}
-
void tmio_mmc_request_dma(struct tmio_mmc_host *host, struct tmio_mmc_data *pdata)
{
/* We can only either use DMA for both Tx and Rx or not use it at all */
@@ -306,7 +319,7 @@
if (!host->bounce_buf)
goto ebouncebuf;
- tasklet_init(&host->dma_complete, tmio_mmc_tasklet_fn, (unsigned long)host);
+ init_completion(&host->dma_dataend);
tasklet_init(&host->dma_issue, tmio_mmc_issue_tasklet_fn, (unsigned long)host);
}
diff --git a/drivers/mmc/host/tmio_mmc_pio.c b/drivers/mmc/host/tmio_mmc_pio.c
index 6b789a7..a2d92f1 100644
--- a/drivers/mmc/host/tmio_mmc_pio.c
+++ b/drivers/mmc/host/tmio_mmc_pio.c
@@ -340,7 +340,7 @@
/* CMD12 is handled by hardware */
if (cmd->opcode == MMC_STOP_TRANSMISSION && !cmd->arg) {
- sd_ctrl_write16(host, CTL_STOP_INTERNAL_ACTION, 0x001);
+ sd_ctrl_write16(host, CTL_STOP_INTERNAL_ACTION, TMIO_STOP_STP);
return 0;
}
@@ -367,7 +367,7 @@
if (data) {
c |= DATA_PRESENT;
if (data->blocks > 1) {
- sd_ctrl_write16(host, CTL_STOP_INTERNAL_ACTION, 0x100);
+ sd_ctrl_write16(host, CTL_STOP_INTERNAL_ACTION, TMIO_STOP_SEC);
c |= TRANSFER_MULTI;
/*
@@ -553,10 +553,14 @@
}
if (stop) {
- if (stop->opcode == MMC_STOP_TRANSMISSION && !stop->arg)
- sd_ctrl_write16(host, CTL_STOP_INTERNAL_ACTION, 0x000);
- else
- BUG();
+ if (stop->opcode != MMC_STOP_TRANSMISSION || stop->arg)
+ dev_err(&host->pdev->dev, "unsupported stop: CMD%u,0x%x. We did CMD12,0\n",
+ stop->opcode, stop->arg);
+
+ /* fill in response from auto CMD12 */
+ stop->resp[0] = sd_ctrl_read16_and_16_as_32(host, CTL_RESPONSE);
+
+ sd_ctrl_write16(host, CTL_STOP_INTERNAL_ACTION, 0);
}
schedule_work(&host->done);
@@ -596,11 +600,11 @@
if (done) {
tmio_mmc_disable_mmc_irqs(host, TMIO_STAT_DATAEND);
- tasklet_schedule(&host->dma_complete);
+ complete(&host->dma_dataend);
}
} else if (host->chan_rx && (data->flags & MMC_DATA_READ) && !host->force_pio) {
tmio_mmc_disable_mmc_irqs(host, TMIO_STAT_DATAEND);
- tasklet_schedule(&host->dma_complete);
+ complete(&host->dma_dataend);
} else {
tmio_mmc_do_data_irq(host);
tmio_mmc_disable_mmc_irqs(host, TMIO_MASK_READOP | TMIO_MASK_WRITEOP);
@@ -811,16 +815,14 @@
struct tmio_mmc_host *host = mmc_priv(mmc);
int i, ret = 0;
- if (!host->tap_num) {
- if (!host->init_tuning || !host->select_tuning)
- /* Tuning is not supported */
- goto out;
+ if (!host->init_tuning || !host->select_tuning)
+ /* Tuning is not supported */
+ goto out;
- host->tap_num = host->init_tuning(host);
- if (!host->tap_num)
- /* Tuning is not supported */
- goto out;
- }
+ host->tap_num = host->init_tuning(host);
+ if (!host->tap_num)
+ /* Tuning is not supported */
+ goto out;
if (host->tap_num * 2 >= sizeof(host->taps) * BITS_PER_BYTE) {
dev_warn_once(&host->pdev->dev,
diff --git a/drivers/mtd/mtdcore.c b/drivers/mtd/mtdcore.c
index 66a9ded..1517da3 100644
--- a/drivers/mtd/mtdcore.c
+++ b/drivers/mtd/mtdcore.c
@@ -46,7 +46,7 @@
#include "mtdcore.h"
-static struct backing_dev_info *mtd_bdi;
+struct backing_dev_info *mtd_bdi;
#ifdef CONFIG_PM_SLEEP
@@ -496,11 +496,9 @@
* mtd_device_parse_register() multiple times on the same master MTD,
* especially with CONFIG_MTD_PARTITIONED_MASTER=y.
*/
- if (WARN_ONCE(mtd->backing_dev_info, "MTD already registered\n"))
+ if (WARN_ONCE(mtd->dev.type, "MTD already registered\n"))
return -EEXIST;
- mtd->backing_dev_info = mtd_bdi;
-
BUG_ON(mtd->writesize == 0);
mutex_lock(&mtd_table_mutex);
@@ -1775,13 +1773,18 @@
struct backing_dev_info *bdi;
int ret;
- bdi = kzalloc(sizeof(*bdi), GFP_KERNEL);
+ bdi = bdi_alloc(GFP_KERNEL);
if (!bdi)
return ERR_PTR(-ENOMEM);
- ret = bdi_setup_and_register(bdi, name);
+ bdi->name = name;
+ /*
+ * We put '-0' suffix to the name to get the same name format as we
+ * used to get. Since this is called only once, we get a unique name.
+ */
+ ret = bdi_register(bdi, "%.28s-0", name);
if (ret)
- kfree(bdi);
+ bdi_put(bdi);
return ret ? ERR_PTR(ret) : bdi;
}
@@ -1813,8 +1816,7 @@
out_procfs:
if (proc_mtd)
remove_proc_entry("mtd", NULL);
- bdi_destroy(mtd_bdi);
- kfree(mtd_bdi);
+ bdi_put(mtd_bdi);
err_bdi:
class_unregister(&mtd_class);
err_reg:
@@ -1828,8 +1830,7 @@
if (proc_mtd)
remove_proc_entry("mtd", NULL);
class_unregister(&mtd_class);
- bdi_destroy(mtd_bdi);
- kfree(mtd_bdi);
+ bdi_put(mtd_bdi);
idr_destroy(&mtd_idr);
}
diff --git a/drivers/mtd/mtdsuper.c b/drivers/mtd/mtdsuper.c
index 20c02a3..e43fea8 100644
--- a/drivers/mtd/mtdsuper.c
+++ b/drivers/mtd/mtdsuper.c
@@ -18,6 +18,7 @@
#include <linux/ctype.h>
#include <linux/slab.h>
#include <linux/major.h>
+#include <linux/backing-dev.h>
/*
* compare superblocks to see if they're equivalent
@@ -38,6 +39,8 @@
return 0;
}
+extern struct backing_dev_info *mtd_bdi;
+
/*
* mark the superblock by the MTD device it is using
* - set the device number to be the correct MTD block device for pesuperstence
@@ -49,7 +52,8 @@
sb->s_mtd = mtd;
sb->s_dev = MKDEV(MTD_BLOCK_MAJOR, mtd->index);
- sb->s_bdi = mtd->backing_dev_info;
+ sb->s_bdi = bdi_get(mtd_bdi);
+
return 0;
}
diff --git a/drivers/mtd/ubi/Kconfig b/drivers/mtd/ubi/Kconfig
index f0855ce..43d131f 100644
--- a/drivers/mtd/ubi/Kconfig
+++ b/drivers/mtd/ubi/Kconfig
@@ -2,7 +2,7 @@
tristate "Enable UBI - Unsorted block images"
select CRC32
help
- UBI is a software layer above MTD layer which admits of LVM-like
+ UBI is a software layer above MTD layer which admits use of LVM-like
logical volumes on top of MTD devices, hides some complexities of
flash chips like wear and bad blocks and provides some other useful
capabilities. Please, consult the MTD web site for more details
diff --git a/drivers/mtd/ubi/block.c b/drivers/mtd/ubi/block.c
index c80869e..51f2be8 100644
--- a/drivers/mtd/ubi/block.c
+++ b/drivers/mtd/ubi/block.c
@@ -347,7 +347,7 @@
return 0;
}
-static struct blk_mq_ops ubiblock_mq_ops = {
+static const struct blk_mq_ops ubiblock_mq_ops = {
.queue_rq = ubiblock_queue_rq,
.init_request = ubiblock_init_request,
};
diff --git a/drivers/mtd/ubi/io.c b/drivers/mtd/ubi/io.c
index b6fb8f9..8290432 100644
--- a/drivers/mtd/ubi/io.c
+++ b/drivers/mtd/ubi/io.c
@@ -45,7 +45,7 @@
* About minimal I/O units. In general, UBI assumes flash device model where
* there is only one minimal I/O unit size. E.g., in case of NOR flash it is 1,
* in case of NAND flash it is a NAND page, etc. This is reported by MTD in the
- * @ubi->mtd->writesize field. But as an exception, UBI admits of using another
+ * @ubi->mtd->writesize field. But as an exception, UBI admits use of another
* (smaller) minimal I/O unit size for EC and VID headers to make it possible
* to do different optimizations.
*
diff --git a/drivers/mtd/ubi/ubi.h b/drivers/mtd/ubi/ubi.h
index 697dbcb..5fe6265 100644
--- a/drivers/mtd/ubi/ubi.h
+++ b/drivers/mtd/ubi/ubi.h
@@ -544,8 +544,7 @@
* @vid_hdr_aloffset: starting offset of the VID header aligned to
* @hdrs_min_io_size
* @vid_hdr_shift: contains @vid_hdr_offset - @vid_hdr_aloffset
- * @bad_allowed: whether the MTD device admits of bad physical eraseblocks or
- * not
+ * @bad_allowed: whether the MTD device admits bad physical eraseblocks or not
* @nor_flash: non-zero if working on top of NOR flash
* @max_write_size: maximum amount of bytes the underlying flash can write at a
* time (MTD write buffer size)
diff --git a/drivers/mtd/ubi/upd.c b/drivers/mtd/ubi/upd.c
index 0134ba3..3971256 100644
--- a/drivers/mtd/ubi/upd.c
+++ b/drivers/mtd/ubi/upd.c
@@ -148,11 +148,11 @@
return err;
}
- if (bytes == 0) {
- err = ubi_wl_flush(ubi, UBI_ALL, UBI_ALL);
- if (err)
- return err;
+ err = ubi_wl_flush(ubi, UBI_ALL, UBI_ALL);
+ if (err)
+ return err;
+ if (bytes == 0) {
err = clear_update_marker(ubi, vol, 0);
if (err)
return err;
diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig
index 100fbdc..83a1616 100644
--- a/drivers/net/Kconfig
+++ b/drivers/net/Kconfig
@@ -355,6 +355,14 @@
This option enables the support for mapping interfaces into VRF's. The
support enables VRF devices.
+config VSOCKMON
+ tristate "Virtual vsock monitoring device"
+ depends on VHOST_VSOCK
+ ---help---
+ This option enables a monitoring net device for vsock sockets. It is
+ mostly intended for developers or support to debug vsock issues. If
+ unsure, say N.
+
endif # NET_CORE
config SUNGEM_PHY
diff --git a/drivers/net/Makefile b/drivers/net/Makefile
index 98ed4d9..b2f6556 100644
--- a/drivers/net/Makefile
+++ b/drivers/net/Makefile
@@ -18,7 +18,7 @@
obj-$(CONFIG_MDIO) += mdio.o
obj-$(CONFIG_NET) += Space.o loopback.o
obj-$(CONFIG_NETCONSOLE) += netconsole.o
-obj-$(CONFIG_PHYLIB) += phy/
+obj-y += phy/
obj-$(CONFIG_RIONET) += rionet.o
obj-$(CONFIG_NET_TEAM) += team/
obj-$(CONFIG_TUN) += tun.o
@@ -30,6 +30,7 @@
obj-$(CONFIG_GTP) += gtp.o
obj-$(CONFIG_NLMON) += nlmon.o
obj-$(CONFIG_NET_VRF) += vrf.o
+obj-$(CONFIG_VSOCKMON) += vsockmon.o
#
# Networking Drivers
diff --git a/drivers/net/bonding/bond_3ad.c b/drivers/net/bonding/bond_3ad.c
index edc70ff..c5fd425 100644
--- a/drivers/net/bonding/bond_3ad.c
+++ b/drivers/net/bonding/bond_3ad.c
@@ -92,6 +92,7 @@
AD_LINK_SPEED_2500MBPS,
AD_LINK_SPEED_10000MBPS,
AD_LINK_SPEED_20000MBPS,
+ AD_LINK_SPEED_25000MBPS,
AD_LINK_SPEED_40000MBPS,
AD_LINK_SPEED_56000MBPS,
AD_LINK_SPEED_100000MBPS,
@@ -260,6 +261,7 @@
* %AD_LINK_SPEED_2500MBPS,
* %AD_LINK_SPEED_10000MBPS
* %AD_LINK_SPEED_20000MBPS
+ * %AD_LINK_SPEED_25000MBPS
* %AD_LINK_SPEED_40000MBPS
* %AD_LINK_SPEED_56000MBPS
* %AD_LINK_SPEED_100000MBPS
@@ -302,6 +304,10 @@
speed = AD_LINK_SPEED_20000MBPS;
break;
+ case SPEED_25000:
+ speed = AD_LINK_SPEED_25000MBPS;
+ break;
+
case SPEED_40000:
speed = AD_LINK_SPEED_40000MBPS;
break;
@@ -707,6 +713,9 @@
case AD_LINK_SPEED_20000MBPS:
bandwidth = nports * 20000;
break;
+ case AD_LINK_SPEED_25000MBPS:
+ bandwidth = nports * 25000;
+ break;
case AD_LINK_SPEED_40000MBPS:
bandwidth = nports * 40000;
break;
@@ -1052,8 +1061,7 @@
port->sm_rx_state = AD_RX_INITIALIZE;
port->sm_vars |= AD_PORT_CHURNED;
/* check if port is not enabled */
- } else if (!(port->sm_vars & AD_PORT_BEGIN)
- && !port->is_enabled && !(port->sm_vars & AD_PORT_MOVED))
+ } else if (!(port->sm_vars & AD_PORT_BEGIN) && !port->is_enabled)
port->sm_rx_state = AD_RX_PORT_DISABLED;
/* check if new lacpdu arrived */
else if (lacpdu && ((port->sm_rx_state == AD_RX_EXPIRED) ||
@@ -1081,11 +1089,8 @@
/* if no lacpdu arrived and no timer is on */
switch (port->sm_rx_state) {
case AD_RX_PORT_DISABLED:
- if (port->sm_vars & AD_PORT_MOVED)
- port->sm_rx_state = AD_RX_INITIALIZE;
- else if (port->is_enabled
- && (port->sm_vars
- & AD_PORT_LACP_ENABLED))
+ if (port->is_enabled &&
+ (port->sm_vars & AD_PORT_LACP_ENABLED))
port->sm_rx_state = AD_RX_EXPIRED;
else if (port->is_enabled
&& ((port->sm_vars
@@ -1115,7 +1120,6 @@
port->sm_vars &= ~AD_PORT_SELECTED;
__record_default(port);
port->actor_oper_port_state &= ~AD_STATE_EXPIRED;
- port->sm_vars &= ~AD_PORT_MOVED;
port->sm_rx_state = AD_RX_PORT_DISABLED;
/* Fall Through */
@@ -2442,9 +2446,9 @@
spin_lock_bh(&slave->bond->mode_lock);
ad_update_actor_keys(port, false);
+ spin_unlock_bh(&slave->bond->mode_lock);
netdev_dbg(slave->bond->dev, "Port %d slave %s changed speed/duplex\n",
port->actor_port_number, slave->dev->name);
- spin_unlock_bh(&slave->bond->mode_lock);
}
/**
@@ -2488,12 +2492,12 @@
agg = __get_first_agg(port);
ad_agg_selection_logic(agg, &dummy);
+ spin_unlock_bh(&slave->bond->mode_lock);
+
netdev_dbg(slave->bond->dev, "Port %d changed link status to %s\n",
port->actor_port_number,
link == BOND_LINK_UP ? "UP" : "DOWN");
- spin_unlock_bh(&slave->bond->mode_lock);
-
/* RTNL is held and mode_lock is released so it's safe
* to update slave_array here.
*/
diff --git a/drivers/net/bonding/bond_alb.c b/drivers/net/bonding/bond_alb.c
index c80b023..7d7a3ce 100644
--- a/drivers/net/bonding/bond_alb.c
+++ b/drivers/net/bonding/bond_alb.c
@@ -687,7 +687,8 @@
/* the arp must be sent on the selected rx channel */
tx_slave = rlb_choose_channel(skb, bond);
if (tx_slave)
- ether_addr_copy(arp->mac_src, tx_slave->dev->dev_addr);
+ bond_hw_addr_copy(arp->mac_src, tx_slave->dev->dev_addr,
+ tx_slave->dev->addr_len);
netdev_dbg(bond->dev, "Server sent ARP Reply packet\n");
} else if (arp->op_code == htons(ARPOP_REQUEST)) {
/* Create an entry in the rx_hashtbl for this client as a
@@ -1017,22 +1018,23 @@
rcu_read_unlock();
}
-static int alb_set_slave_mac_addr(struct slave *slave, u8 addr[])
+static int alb_set_slave_mac_addr(struct slave *slave, u8 addr[],
+ unsigned int len)
{
struct net_device *dev = slave->dev;
- struct sockaddr s_addr;
+ struct sockaddr_storage ss;
if (BOND_MODE(slave->bond) == BOND_MODE_TLB) {
- memcpy(dev->dev_addr, addr, dev->addr_len);
+ memcpy(dev->dev_addr, addr, len);
return 0;
}
/* for rlb each slave must have a unique hw mac addresses so that
* each slave will receive packets destined to a different mac
*/
- memcpy(s_addr.sa_data, addr, dev->addr_len);
- s_addr.sa_family = dev->type;
- if (dev_set_mac_address(dev, &s_addr)) {
+ memcpy(ss.__data, addr, len);
+ ss.ss_family = dev->type;
+ if (dev_set_mac_address(dev, (struct sockaddr *)&ss)) {
netdev_err(slave->bond->dev, "dev_set_mac_address of dev %s failed! ALB mode requires that the base driver support setting the hw address also when the network device's interface is open\n",
dev->name);
return -EOPNOTSUPP;
@@ -1046,11 +1048,14 @@
*/
static void alb_swap_mac_addr(struct slave *slave1, struct slave *slave2)
{
- u8 tmp_mac_addr[ETH_ALEN];
+ u8 tmp_mac_addr[MAX_ADDR_LEN];
- ether_addr_copy(tmp_mac_addr, slave1->dev->dev_addr);
- alb_set_slave_mac_addr(slave1, slave2->dev->dev_addr);
- alb_set_slave_mac_addr(slave2, tmp_mac_addr);
+ bond_hw_addr_copy(tmp_mac_addr, slave1->dev->dev_addr,
+ slave1->dev->addr_len);
+ alb_set_slave_mac_addr(slave1, slave2->dev->dev_addr,
+ slave2->dev->addr_len);
+ alb_set_slave_mac_addr(slave2, tmp_mac_addr,
+ slave1->dev->addr_len);
}
@@ -1177,7 +1182,8 @@
/* Try setting slave mac to bond address and fall-through
* to code handling that situation below...
*/
- alb_set_slave_mac_addr(slave, bond->dev->dev_addr);
+ alb_set_slave_mac_addr(slave, bond->dev->dev_addr,
+ bond->dev->addr_len);
}
/* The slave's address is equal to the address of the bond.
@@ -1202,7 +1208,8 @@
}
if (free_mac_slave) {
- alb_set_slave_mac_addr(slave, free_mac_slave->perm_hwaddr);
+ alb_set_slave_mac_addr(slave, free_mac_slave->perm_hwaddr,
+ free_mac_slave->dev->addr_len);
netdev_warn(bond->dev, "the hw address of slave %s is in use by the bond; giving it the hw address of %s\n",
slave->dev->name, free_mac_slave->dev->name);
@@ -1234,8 +1241,8 @@
{
struct slave *slave, *rollback_slave;
struct list_head *iter;
- struct sockaddr sa;
- char tmp_addr[ETH_ALEN];
+ struct sockaddr_storage ss;
+ char tmp_addr[MAX_ADDR_LEN];
int res;
if (bond->alb_info.rlb_enabled)
@@ -1243,12 +1250,14 @@
bond_for_each_slave(bond, slave, iter) {
/* save net_device's current hw address */
- ether_addr_copy(tmp_addr, slave->dev->dev_addr);
+ bond_hw_addr_copy(tmp_addr, slave->dev->dev_addr,
+ slave->dev->addr_len);
res = dev_set_mac_address(slave->dev, addr);
/* restore net_device's hw address */
- ether_addr_copy(slave->dev->dev_addr, tmp_addr);
+ bond_hw_addr_copy(slave->dev->dev_addr, tmp_addr,
+ slave->dev->addr_len);
if (res)
goto unwind;
@@ -1257,16 +1266,19 @@
return 0;
unwind:
- memcpy(sa.sa_data, bond->dev->dev_addr, bond->dev->addr_len);
- sa.sa_family = bond->dev->type;
+ memcpy(ss.__data, bond->dev->dev_addr, bond->dev->addr_len);
+ ss.ss_family = bond->dev->type;
/* unwind from head to the slave that failed */
bond_for_each_slave(bond, rollback_slave, iter) {
if (rollback_slave == slave)
break;
- ether_addr_copy(tmp_addr, rollback_slave->dev->dev_addr);
- dev_set_mac_address(rollback_slave->dev, &sa);
- ether_addr_copy(rollback_slave->dev->dev_addr, tmp_addr);
+ bond_hw_addr_copy(tmp_addr, rollback_slave->dev->dev_addr,
+ rollback_slave->dev->addr_len);
+ dev_set_mac_address(rollback_slave->dev,
+ (struct sockaddr *)&ss);
+ bond_hw_addr_copy(rollback_slave->dev->dev_addr, tmp_addr,
+ rollback_slave->dev->addr_len);
}
return res;
@@ -1582,7 +1594,8 @@
{
int res;
- res = alb_set_slave_mac_addr(slave, slave->perm_hwaddr);
+ res = alb_set_slave_mac_addr(slave, slave->perm_hwaddr,
+ slave->dev->addr_len);
if (res)
return res;
@@ -1696,17 +1709,20 @@
* and thus filter bond->dev_addr's packets, so force bond's mac
*/
if (BOND_MODE(bond) == BOND_MODE_TLB) {
- struct sockaddr sa;
- u8 tmp_addr[ETH_ALEN];
+ struct sockaddr_storage ss;
+ u8 tmp_addr[MAX_ADDR_LEN];
- ether_addr_copy(tmp_addr, new_slave->dev->dev_addr);
+ bond_hw_addr_copy(tmp_addr, new_slave->dev->dev_addr,
+ new_slave->dev->addr_len);
- memcpy(sa.sa_data, bond->dev->dev_addr, bond->dev->addr_len);
- sa.sa_family = bond->dev->type;
+ bond_hw_addr_copy(ss.__data, bond->dev->dev_addr,
+ bond->dev->addr_len);
+ ss.ss_family = bond->dev->type;
/* we don't care if it can't change its mac, best effort */
- dev_set_mac_address(new_slave->dev, &sa);
+ dev_set_mac_address(new_slave->dev, (struct sockaddr *)&ss);
- ether_addr_copy(new_slave->dev->dev_addr, tmp_addr);
+ bond_hw_addr_copy(new_slave->dev->dev_addr, tmp_addr,
+ new_slave->dev->addr_len);
}
/* curr_active_slave must be set before calling alb_swap_mac_addr */
@@ -1716,7 +1732,8 @@
alb_fasten_mac_swap(bond, swap_slave, new_slave);
} else {
/* set the new_slave to the bond mac address */
- alb_set_slave_mac_addr(new_slave, bond->dev->dev_addr);
+ alb_set_slave_mac_addr(new_slave, bond->dev->dev_addr,
+ bond->dev->addr_len);
alb_send_learning_packets(new_slave, bond->dev->dev_addr,
false);
}
@@ -1726,19 +1743,19 @@
int bond_alb_set_mac_address(struct net_device *bond_dev, void *addr)
{
struct bonding *bond = netdev_priv(bond_dev);
- struct sockaddr *sa = addr;
+ struct sockaddr_storage *ss = addr;
struct slave *curr_active;
struct slave *swap_slave;
int res;
- if (!is_valid_ether_addr(sa->sa_data))
+ if (!is_valid_ether_addr(ss->__data))
return -EADDRNOTAVAIL;
res = alb_set_mac_address(bond, addr);
if (res)
return res;
- memcpy(bond_dev->dev_addr, sa->sa_data, bond_dev->addr_len);
+ bond_hw_addr_copy(bond_dev->dev_addr, ss->__data, bond_dev->addr_len);
/* If there is no curr_active_slave there is nothing else to do.
* Otherwise we'll need to pass the new address to it and handle
@@ -1754,7 +1771,8 @@
alb_swap_mac_addr(swap_slave, curr_active);
alb_fasten_mac_swap(bond, swap_slave, curr_active);
} else {
- alb_set_slave_mac_addr(curr_active, bond_dev->dev_addr);
+ alb_set_slave_mac_addr(curr_active, bond_dev->dev_addr,
+ bond_dev->addr_len);
alb_send_learning_packets(curr_active,
bond_dev->dev_addr, false);
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index 8a4ba8b..2be7880 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -201,12 +201,6 @@
unsigned int bond_net_id __read_mostly;
-static __be32 arp_target[BOND_MAX_ARP_TARGETS];
-static int arp_ip_count;
-static int bond_mode = BOND_MODE_ROUNDROBIN;
-static int xmit_hashtype = BOND_XMIT_POLICY_LAYER2;
-static int lacp_fast;
-
/*-------------------------- Forward declarations ---------------------------*/
static int bond_init(struct net_device *bond_dev);
@@ -371,9 +365,10 @@
/* Get link speed and duplex from the slave's base driver
* using ethtool. If for some reason the call fails or the
* values are invalid, set speed and duplex to -1,
- * and return.
+ * and return. Return 1 if speed or duplex settings are
+ * UNKNOWN; 0 otherwise.
*/
-static void bond_update_speed_duplex(struct slave *slave)
+static int bond_update_speed_duplex(struct slave *slave)
{
struct net_device *slave_dev = slave->dev;
struct ethtool_link_ksettings ecmd;
@@ -384,23 +379,21 @@
res = __ethtool_get_link_ksettings(slave_dev, &ecmd);
if (res < 0)
- return;
-
+ return 1;
if (ecmd.base.speed == 0 || ecmd.base.speed == ((__u32)-1))
- return;
-
+ return 1;
switch (ecmd.base.duplex) {
case DUPLEX_FULL:
case DUPLEX_HALF:
break;
default:
- return;
+ return 1;
}
slave->speed = ecmd.base.speed;
slave->duplex = ecmd.base.duplex;
- return;
+ return 0;
}
const char *bond_slave_link_status(s8 link)
@@ -652,8 +645,8 @@
struct slave *new_active,
struct slave *old_active)
{
- u8 tmp_mac[ETH_ALEN];
- struct sockaddr saddr;
+ u8 tmp_mac[MAX_ADDR_LEN];
+ struct sockaddr_storage ss;
int rv;
switch (bond->params.fail_over_mac) {
@@ -673,16 +666,20 @@
old_active = bond_get_old_active(bond, new_active);
if (old_active) {
- ether_addr_copy(tmp_mac, new_active->dev->dev_addr);
- ether_addr_copy(saddr.sa_data,
- old_active->dev->dev_addr);
- saddr.sa_family = new_active->dev->type;
+ bond_hw_addr_copy(tmp_mac, new_active->dev->dev_addr,
+ new_active->dev->addr_len);
+ bond_hw_addr_copy(ss.__data,
+ old_active->dev->dev_addr,
+ old_active->dev->addr_len);
+ ss.ss_family = new_active->dev->type;
} else {
- ether_addr_copy(saddr.sa_data, bond->dev->dev_addr);
- saddr.sa_family = bond->dev->type;
+ bond_hw_addr_copy(ss.__data, bond->dev->dev_addr,
+ bond->dev->addr_len);
+ ss.ss_family = bond->dev->type;
}
- rv = dev_set_mac_address(new_active->dev, &saddr);
+ rv = dev_set_mac_address(new_active->dev,
+ (struct sockaddr *)&ss);
if (rv) {
netdev_err(bond->dev, "Error %d setting MAC of slave %s\n",
-rv, new_active->dev->name);
@@ -692,10 +689,12 @@
if (!old_active)
goto out;
- ether_addr_copy(saddr.sa_data, tmp_mac);
- saddr.sa_family = old_active->dev->type;
+ bond_hw_addr_copy(ss.__data, tmp_mac,
+ new_active->dev->addr_len);
+ ss.ss_family = old_active->dev->type;
- rv = dev_set_mac_address(old_active->dev, &saddr);
+ rv = dev_set_mac_address(old_active->dev,
+ (struct sockaddr *)&ss);
if (rv)
netdev_err(bond->dev, "Error %d setting MAC of slave %s\n",
-rv, new_active->dev->name);
@@ -1104,11 +1103,11 @@
gso_max_size = min(gso_max_size, slave->dev->gso_max_size);
gso_max_segs = min(gso_max_segs, slave->dev->gso_max_segs);
}
+ bond_dev->hard_header_len = max_hard_header_len;
done:
bond_dev->vlan_features = vlan_features;
bond_dev->hw_enc_features = enc_features | NETIF_F_GSO_ENCAP_ALL;
- bond_dev->hard_header_len = max_hard_header_len;
bond_dev->gso_max_segs = gso_max_segs;
netif_set_gso_max_size(bond_dev, gso_max_size);
@@ -1177,6 +1176,9 @@
}
}
+ /* don't change skb->dev for link-local packets */
+ if (is_link_local_ether_addr(eth_hdr(skb)->h_dest))
+ return RX_HANDLER_PASS;
if (bond_should_deliver_exact_match(skb, slave, bond))
return RX_HANDLER_EXACT;
@@ -1191,7 +1193,8 @@
kfree_skb(skb);
return RX_HANDLER_CONSUMED;
}
- ether_addr_copy(eth_hdr(skb)->h_dest, bond->dev->dev_addr);
+ bond_hw_addr_copy(eth_hdr(skb)->h_dest, bond->dev->dev_addr,
+ bond->dev->addr_len);
}
return ret;
@@ -1330,7 +1333,7 @@
struct bonding *bond = netdev_priv(bond_dev);
const struct net_device_ops *slave_ops = slave_dev->netdev_ops;
struct slave *new_slave = NULL, *prev_slave;
- struct sockaddr addr;
+ struct sockaddr_storage ss;
int link_reporting;
int res = 0, i;
@@ -1481,16 +1484,17 @@
* that need it, and for restoring it upon release, and then
* set it to the master's address
*/
- ether_addr_copy(new_slave->perm_hwaddr, slave_dev->dev_addr);
+ bond_hw_addr_copy(new_slave->perm_hwaddr, slave_dev->dev_addr,
+ slave_dev->addr_len);
if (!bond->params.fail_over_mac ||
BOND_MODE(bond) != BOND_MODE_ACTIVEBACKUP) {
/* Set slave to master's mac address. The application already
* set the master's mac address to that of the first slave
*/
- memcpy(addr.sa_data, bond_dev->dev_addr, bond_dev->addr_len);
- addr.sa_family = slave_dev->type;
- res = dev_set_mac_address(slave_dev, &addr);
+ memcpy(ss.__data, bond_dev->dev_addr, bond_dev->addr_len);
+ ss.ss_family = slave_dev->type;
+ res = dev_set_mac_address(slave_dev, (struct sockaddr *)&ss);
if (res) {
netdev_dbg(bond_dev, "Error %d calling set_mac_address\n", res);
goto err_restore_mtu;
@@ -1565,7 +1569,8 @@
new_slave->delay = 0;
new_slave->link_failure_count = 0;
- bond_update_speed_duplex(new_slave);
+ if (bond_update_speed_duplex(new_slave))
+ new_slave->link = BOND_LINK_DOWN;
new_slave->last_rx = jiffies -
(msecs_to_jiffies(bond->params.arp_interval) + 1);
@@ -1773,9 +1778,10 @@
* MAC if this slave's MAC is in use by the bond, or at
* least print a warning.
*/
- ether_addr_copy(addr.sa_data, new_slave->perm_hwaddr);
- addr.sa_family = slave_dev->type;
- dev_set_mac_address(slave_dev, &addr);
+ bond_hw_addr_copy(ss.__data, new_slave->perm_hwaddr,
+ new_slave->dev->addr_len);
+ ss.ss_family = slave_dev->type;
+ dev_set_mac_address(slave_dev, (struct sockaddr *)&ss);
}
err_restore_mtu:
@@ -1818,7 +1824,7 @@
{
struct bonding *bond = netdev_priv(bond_dev);
struct slave *slave, *oldcurrent;
- struct sockaddr addr;
+ struct sockaddr_storage ss;
int old_flags = bond_dev->flags;
netdev_features_t old_features = bond_dev->features;
@@ -1953,9 +1959,10 @@
if (bond->params.fail_over_mac != BOND_FOM_ACTIVE ||
BOND_MODE(bond) != BOND_MODE_ACTIVEBACKUP) {
/* restore original ("permanent") mac address */
- ether_addr_copy(addr.sa_data, slave->perm_hwaddr);
- addr.sa_family = slave_dev->type;
- dev_set_mac_address(slave_dev, &addr);
+ bond_hw_addr_copy(ss.__data, slave->perm_hwaddr,
+ slave->dev->addr_len);
+ ss.ss_family = slave_dev->type;
+ dev_set_mac_address(slave_dev, (struct sockaddr *)&ss);
}
dev_set_mtu(slave_dev, slave->original_mtu);
@@ -2039,8 +2046,7 @@
if (link_state)
continue;
- bond_set_slave_link_state(slave, BOND_LINK_FAIL,
- BOND_SLAVE_NOTIFY_LATER);
+ bond_propose_link_state(slave, BOND_LINK_FAIL);
slave->delay = bond->params.downdelay;
if (slave->delay) {
netdev_info(bond->dev, "link status down for %sinterface %s, disabling it in %d ms\n",
@@ -2055,13 +2061,13 @@
case BOND_LINK_FAIL:
if (link_state) {
/* recovered before downdelay expired */
- bond_set_slave_link_state(slave, BOND_LINK_UP,
- BOND_SLAVE_NOTIFY_LATER);
+ bond_propose_link_state(slave, BOND_LINK_UP);
slave->last_link_up = jiffies;
netdev_info(bond->dev, "link status up again after %d ms for interface %s\n",
(bond->params.downdelay - slave->delay) *
bond->params.miimon,
slave->dev->name);
+ commit++;
continue;
}
@@ -2078,8 +2084,7 @@
if (!link_state)
continue;
- bond_set_slave_link_state(slave, BOND_LINK_BACK,
- BOND_SLAVE_NOTIFY_LATER);
+ bond_propose_link_state(slave, BOND_LINK_BACK);
slave->delay = bond->params.updelay;
if (slave->delay) {
@@ -2092,14 +2097,12 @@
/*FALLTHRU*/
case BOND_LINK_BACK:
if (!link_state) {
- bond_set_slave_link_state(slave,
- BOND_LINK_DOWN,
- BOND_SLAVE_NOTIFY_LATER);
+ bond_propose_link_state(slave, BOND_LINK_DOWN);
netdev_info(bond->dev, "link status down again after %d ms for interface %s\n",
(bond->params.updelay - slave->delay) *
bond->params.miimon,
slave->dev->name);
-
+ commit++;
continue;
}
@@ -2132,7 +2135,13 @@
continue;
case BOND_LINK_UP:
- bond_update_speed_duplex(slave);
+ if (bond_update_speed_duplex(slave)) {
+ slave->link = BOND_LINK_DOWN;
+ netdev_warn(bond->dev,
+ "failed to get link speed/duplex for %s\n",
+ slave->dev->name);
+ continue;
+ }
bond_set_slave_link_state(slave, BOND_LINK_UP,
BOND_SLAVE_NOTIFY_NOW);
slave->last_link_up = jiffies;
@@ -2231,6 +2240,8 @@
mii_work.work);
bool should_notify_peers = false;
unsigned long delay;
+ struct slave *slave;
+ struct list_head *iter;
delay = msecs_to_jiffies(bond->params.miimon);
@@ -2251,6 +2262,9 @@
goto re_arm;
}
+ bond_for_each_slave(bond, slave, iter) {
+ bond_commit_link_state(slave, BOND_SLAVE_NOTIFY_LATER);
+ }
bond_miimon_commit(bond);
rtnl_unlock(); /* might sleep, hold no other locks */
@@ -2575,10 +2589,8 @@
* arp is transmitted to generate traffic. see activebackup_arp_monitor for
* arp monitoring in active backup mode.
*/
-static void bond_loadbalance_arp_mon(struct work_struct *work)
+static void bond_loadbalance_arp_mon(struct bonding *bond)
{
- struct bonding *bond = container_of(work, struct bonding,
- arp_work.work);
struct slave *slave, *oldcurrent;
struct list_head *iter;
int do_failover = 0, slave_state_changed = 0;
@@ -2916,10 +2928,8 @@
return should_notify_rtnl;
}
-static void bond_activebackup_arp_mon(struct work_struct *work)
+static void bond_activebackup_arp_mon(struct bonding *bond)
{
- struct bonding *bond = container_of(work, struct bonding,
- arp_work.work);
bool should_notify_peers = false;
bool should_notify_rtnl = false;
int delta_in_ticks;
@@ -2972,6 +2982,17 @@
}
}
+static void bond_arp_monitor(struct work_struct *work)
+{
+ struct bonding *bond = container_of(work, struct bonding,
+ arp_work.work);
+
+ if (BOND_MODE(bond) == BOND_MODE_ACTIVEBACKUP)
+ bond_activebackup_arp_mon(bond);
+ else
+ bond_loadbalance_arp_mon(bond);
+}
+
/*-------------------------- netdev event handling --------------------------*/
/* Change device name */
@@ -3222,16 +3243,13 @@
/*-------------------------- Device entry points ----------------------------*/
-static void bond_work_init_all(struct bonding *bond)
+void bond_work_init_all(struct bonding *bond)
{
INIT_DELAYED_WORK(&bond->mcast_work,
bond_resend_igmp_join_requests_delayed);
INIT_DELAYED_WORK(&bond->alb_work, bond_alb_monitor);
INIT_DELAYED_WORK(&bond->mii_work, bond_mii_monitor);
- if (BOND_MODE(bond) == BOND_MODE_ACTIVEBACKUP)
- INIT_DELAYED_WORK(&bond->arp_work, bond_activebackup_arp_mon);
- else
- INIT_DELAYED_WORK(&bond->arp_work, bond_loadbalance_arp_mon);
+ INIT_DELAYED_WORK(&bond->arp_work, bond_arp_monitor);
INIT_DELAYED_WORK(&bond->ad_work, bond_3ad_state_machine_handler);
INIT_DELAYED_WORK(&bond->slave_arr_work, bond_slave_arr_handler);
}
@@ -3266,8 +3284,6 @@
}
}
- bond_work_init_all(bond);
-
if (bond_is_lb(bond)) {
/* bond_alb_initialize must be called before the timer
* is started.
@@ -3327,12 +3343,17 @@
for (i = 0; i < sizeof(*_res) / sizeof(u64); i++) {
u64 nv = new[i];
u64 ov = old[i];
+ s64 delta = nv - ov;
/* detects if this particular field is 32bit only */
if (((nv | ov) >> 32) == 0)
- res[i] += (u32)nv - (u32)ov;
- else
- res[i] += nv - ov;
+ delta = (s64)(s32)((u32)nv - (u32)ov);
+
+ /* filter anomalies, some drivers reset their stats
+ * at down/up events.
+ */
+ if (delta > 0)
+ res[i] += delta;
}
}
@@ -3619,7 +3640,7 @@
{
struct bonding *bond = netdev_priv(bond_dev);
struct slave *slave, *rollback_slave;
- struct sockaddr *sa = addr, tmp_sa;
+ struct sockaddr_storage *ss = addr, tmp_ss;
struct list_head *iter;
int res = 0;
@@ -3636,7 +3657,7 @@
BOND_MODE(bond) == BOND_MODE_ACTIVEBACKUP)
return 0;
- if (!is_valid_ether_addr(sa->sa_data))
+ if (!is_valid_ether_addr(ss->__data))
return -EADDRNOTAVAIL;
bond_for_each_slave(bond, slave, iter) {
@@ -3655,12 +3676,12 @@
}
/* success */
- memcpy(bond_dev->dev_addr, sa->sa_data, bond_dev->addr_len);
+ memcpy(bond_dev->dev_addr, ss->__data, bond_dev->addr_len);
return 0;
unwind:
- memcpy(tmp_sa.sa_data, bond_dev->dev_addr, bond_dev->addr_len);
- tmp_sa.sa_family = bond_dev->type;
+ memcpy(tmp_ss.__data, bond_dev->dev_addr, bond_dev->addr_len);
+ tmp_ss.ss_family = bond_dev->type;
/* unwind from head to the slave that failed */
bond_for_each_slave(bond, rollback_slave, iter) {
@@ -3669,7 +3690,8 @@
if (rollback_slave == slave)
break;
- tmp_res = dev_set_mac_address(rollback_slave->dev, &tmp_sa);
+ tmp_res = dev_set_mac_address(rollback_slave->dev,
+ (struct sockaddr *)&tmp_ss);
if (tmp_res) {
netdev_dbg(bond_dev, "unwind err %d dev %s\n",
tmp_res, rollback_slave->dev->name);
@@ -4252,6 +4274,12 @@
int arp_all_targets_value;
u16 ad_actor_sys_prio = 0;
u16 ad_user_port_key = 0;
+ __be32 arp_target[BOND_MAX_ARP_TARGETS];
+ int arp_ip_count;
+ int bond_mode = BOND_MODE_ROUNDROBIN;
+ int xmit_hashtype = BOND_XMIT_POLICY_LAYER2;
+ int lacp_fast = 0;
+ int tlb_dynamic_lb = 0;
/* Convert string parameters. */
if (mode) {
@@ -4564,6 +4592,17 @@
}
ad_user_port_key = valptr->value;
+ if (bond_mode == BOND_MODE_TLB) {
+ bond_opt_initstr(&newval, "default");
+ valptr = bond_opt_parse(bond_opt_get(BOND_OPT_TLB_DYNAMIC_LB),
+ &newval);
+ if (!valptr) {
+ pr_err("Error: No tlb_dynamic_lb default value");
+ return -EINVAL;
+ }
+ tlb_dynamic_lb = valptr->value;
+ }
+
if (lp_interval == 0) {
pr_warn("Warning: ip_interval must be between 1 and %d, so it was reset to %d\n",
INT_MAX, BOND_ALB_DEFAULT_LP_INTERVAL);
@@ -4591,7 +4630,7 @@
params->min_links = min_links;
params->lp_interval = lp_interval;
params->packets_per_slave = packets_per_slave;
- params->tlb_dynamic_lb = 1; /* Default value */
+ params->tlb_dynamic_lb = tlb_dynamic_lb;
params->ad_actor_sys_prio = ad_actor_sys_prio;
eth_zero_addr(params->ad_actor_system);
params->ad_user_port_key = ad_user_port_key;
@@ -4687,6 +4726,8 @@
netif_carrier_off(bond_dev);
+ bond_work_init_all(bond);
+
rtnl_unlock();
if (res < 0)
bond_destructor(bond_dev);
diff --git a/drivers/net/bonding/bond_netlink.c b/drivers/net/bonding/bond_netlink.c
index b8df0f5..c502c13 100644
--- a/drivers/net/bonding/bond_netlink.c
+++ b/drivers/net/bonding/bond_netlink.c
@@ -449,6 +449,11 @@
err = register_netdevice(bond_dev);
netif_carrier_off(bond_dev);
+ if (!err) {
+ struct bonding *bond = netdev_priv(bond_dev);
+
+ bond_work_init_all(bond);
+ }
return err;
}
diff --git a/drivers/net/bonding/bond_procfs.c b/drivers/net/bonding/bond_procfs.c
index f514fe5..d8d4ada 100644
--- a/drivers/net/bonding/bond_procfs.c
+++ b/drivers/net/bonding/bond_procfs.c
@@ -183,7 +183,8 @@
seq_printf(seq, "Link Failure Count: %u\n",
slave->link_failure_count);
- seq_printf(seq, "Permanent HW addr: %pM\n", slave->perm_hwaddr);
+ seq_printf(seq, "Permanent HW addr: %*phC\n",
+ slave->dev->addr_len, slave->perm_hwaddr);
seq_printf(seq, "Slave queue ID: %d\n", slave->queue_id);
if (BOND_MODE(bond) == BOND_MODE_8023AD) {
diff --git a/drivers/net/can/Kconfig b/drivers/net/can/Kconfig
index 22570ea..ac4ff39 100644
--- a/drivers/net/can/Kconfig
+++ b/drivers/net/can/Kconfig
@@ -9,6 +9,24 @@
This driver can also be built as a module. If so, the module
will be called vcan.
+config CAN_VXCAN
+ tristate "Virtual CAN Tunnel (vxcan)"
+ ---help---
+ Similar to the virtual ethernet driver veth, vxcan implements a
+ local CAN traffic tunnel between two virtual CAN network devices.
+ When creating a vxcan, two vxcan devices are created as pair.
+ When one end receives the packet it appears on its pair and vice
+ versa. The vxcan can be used for cross namespace communication.
+
+ In opposite to vcan loopback devices the vxcan only forwards CAN
+ frames to its pair and does *not* provide a local echo of sent
+ CAN frames. To disable a potential echo in af_can.c the vxcan driver
+ announces IFF_ECHO in the interface flags. To have a clean start
+ in each namespace the CAN GW hop counter is set to zero.
+
+ This driver can also be built as a module. If so, the module
+ will be called vxcan.
+
config CAN_SLCAN
tristate "Serial / USB serial CAN Adaptors (slcan)"
depends on TTY
@@ -142,6 +160,7 @@
source "drivers/net/can/ifi_canfd/Kconfig"
source "drivers/net/can/m_can/Kconfig"
source "drivers/net/can/mscan/Kconfig"
+source "drivers/net/can/peak_canfd/Kconfig"
source "drivers/net/can/rcar/Kconfig"
source "drivers/net/can/sja1000/Kconfig"
source "drivers/net/can/softing/Kconfig"
diff --git a/drivers/net/can/Makefile b/drivers/net/can/Makefile
index 0da4f2f..4aabbee 100644
--- a/drivers/net/can/Makefile
+++ b/drivers/net/can/Makefile
@@ -3,6 +3,7 @@
#
obj-$(CONFIG_CAN_VCAN) += vcan.o
+obj-$(CONFIG_CAN_VXCAN) += vxcan.o
obj-$(CONFIG_CAN_SLCAN) += slcan.o
obj-$(CONFIG_CAN_DEV) += can-dev.o
@@ -26,6 +27,7 @@
obj-$(CONFIG_CAN_JANZ_ICAN3) += janz-ican3.o
obj-$(CONFIG_CAN_MSCAN) += mscan/
obj-$(CONFIG_CAN_M_CAN) += m_can/
+obj-$(CONFIG_CAN_PEAK_PCIEFD) += peak_canfd/
obj-$(CONFIG_CAN_SJA1000) += sja1000/
obj-$(CONFIG_CAN_SUN4I) += sun4i_can.o
obj-$(CONFIG_CAN_TI_HECC) += ti_hecc.o
diff --git a/drivers/net/can/m_can/m_can.c b/drivers/net/can/m_can/m_can.c
index 7a6554e..bf8fdae 100644
--- a/drivers/net/can/m_can/m_can.c
+++ b/drivers/net/can/m_can/m_can.c
@@ -23,7 +23,7 @@
#include <linux/of.h>
#include <linux/of_device.h>
#include <linux/platform_device.h>
-
+#include <linux/iopoll.h>
#include <linux/can/dev.h>
/* napi related */
@@ -37,17 +37,19 @@
M_CAN_CREL = 0x0,
M_CAN_ENDN = 0x4,
M_CAN_CUST = 0x8,
- M_CAN_FBTP = 0xc,
+ M_CAN_DBTP = 0xc,
M_CAN_TEST = 0x10,
M_CAN_RWD = 0x14,
M_CAN_CCCR = 0x18,
- M_CAN_BTP = 0x1c,
+ M_CAN_NBTP = 0x1c,
M_CAN_TSCC = 0x20,
M_CAN_TSCV = 0x24,
M_CAN_TOCC = 0x28,
M_CAN_TOCV = 0x2c,
M_CAN_ECR = 0x40,
M_CAN_PSR = 0x44,
+/* TDCR Register only available for version >=3.1.x */
+ M_CAN_TDCR = 0x48,
M_CAN_IR = 0x50,
M_CAN_IE = 0x54,
M_CAN_ILS = 0x58,
@@ -105,21 +107,29 @@
MRAM_CFG_NUM,
};
-/* Fast Bit Timing & Prescaler Register (FBTP) */
-#define FBTR_FBRP_MASK 0x1f
-#define FBTR_FBRP_SHIFT 16
-#define FBTR_FTSEG1_SHIFT 8
-#define FBTR_FTSEG1_MASK (0xf << FBTR_FTSEG1_SHIFT)
-#define FBTR_FTSEG2_SHIFT 4
-#define FBTR_FTSEG2_MASK (0x7 << FBTR_FTSEG2_SHIFT)
-#define FBTR_FSJW_SHIFT 0
-#define FBTR_FSJW_MASK 0x3
+/* Core Release Register (CREL) */
+#define CREL_REL_SHIFT 28
+#define CREL_REL_MASK (0xF << CREL_REL_SHIFT)
+#define CREL_STEP_SHIFT 24
+#define CREL_STEP_MASK (0xF << CREL_STEP_SHIFT)
+#define CREL_SUBSTEP_SHIFT 20
+#define CREL_SUBSTEP_MASK (0xF << CREL_SUBSTEP_SHIFT)
+
+/* Data Bit Timing & Prescaler Register (DBTP) */
+#define DBTP_TDC BIT(23)
+#define DBTP_DBRP_SHIFT 16
+#define DBTP_DBRP_MASK (0x1f << DBTP_DBRP_SHIFT)
+#define DBTP_DTSEG1_SHIFT 8
+#define DBTP_DTSEG1_MASK (0x1f << DBTP_DTSEG1_SHIFT)
+#define DBTP_DTSEG2_SHIFT 4
+#define DBTP_DTSEG2_MASK (0xf << DBTP_DTSEG2_SHIFT)
+#define DBTP_DSJW_SHIFT 0
+#define DBTP_DSJW_MASK (0xf << DBTP_DSJW_SHIFT)
/* Test Register (TEST) */
-#define TEST_LBCK BIT(4)
+#define TEST_LBCK BIT(4)
/* CC Control Register(CCCR) */
-#define CCCR_TEST BIT(7)
#define CCCR_CMR_MASK 0x3
#define CCCR_CMR_SHIFT 10
#define CCCR_CMR_CANFD 0x1
@@ -130,21 +140,32 @@
#define CCCR_CME_CAN 0
#define CCCR_CME_CANFD 0x1
#define CCCR_CME_CANFD_BRS 0x2
+#define CCCR_TXP BIT(14)
#define CCCR_TEST BIT(7)
#define CCCR_MON BIT(5)
+#define CCCR_CSR BIT(4)
+#define CCCR_CSA BIT(3)
+#define CCCR_ASM BIT(2)
#define CCCR_CCE BIT(1)
#define CCCR_INIT BIT(0)
#define CCCR_CANFD 0x10
+/* for version >=3.1.x */
+#define CCCR_EFBI BIT(13)
+#define CCCR_PXHD BIT(12)
+#define CCCR_BRSE BIT(9)
+#define CCCR_FDOE BIT(8)
+/* only for version >=3.2.x */
+#define CCCR_NISO BIT(15)
-/* Bit Timing & Prescaler Register (BTP) */
-#define BTR_BRP_MASK 0x3ff
-#define BTR_BRP_SHIFT 16
-#define BTR_TSEG1_SHIFT 8
-#define BTR_TSEG1_MASK (0x3f << BTR_TSEG1_SHIFT)
-#define BTR_TSEG2_SHIFT 4
-#define BTR_TSEG2_MASK (0xf << BTR_TSEG2_SHIFT)
-#define BTR_SJW_SHIFT 0
-#define BTR_SJW_MASK 0xf
+/* Nominal Bit Timing & Prescaler Register (NBTP) */
+#define NBTP_NSJW_SHIFT 25
+#define NBTP_NSJW_MASK (0x7f << NBTP_NSJW_SHIFT)
+#define NBTP_NBRP_SHIFT 16
+#define NBTP_NBRP_MASK (0x1ff << NBTP_NBRP_SHIFT)
+#define NBTP_NTSEG1_SHIFT 8
+#define NBTP_NTSEG1_MASK (0xff << NBTP_NTSEG1_SHIFT)
+#define NBTP_NTSEG2_SHIFT 0
+#define NBTP_NTSEG2_MASK (0x7f << NBTP_NTSEG2_SHIFT)
/* Error Counter Register(ECR) */
#define ECR_RP BIT(15)
@@ -161,6 +182,13 @@
/* Interrupt Register(IR) */
#define IR_ALL_INT 0xffffffff
+
+/* Renamed bits for versions > 3.1.x */
+#define IR_ARA BIT(29)
+#define IR_PED BIT(28)
+#define IR_PEA BIT(27)
+
+/* Bits for version 3.0.x */
#define IR_STE BIT(31)
#define IR_FOE BIT(30)
#define IR_ACKE BIT(29)
@@ -194,33 +222,40 @@
#define IR_RF0W BIT(1)
#define IR_RF0N BIT(0)
#define IR_ERR_STATE (IR_BO | IR_EW | IR_EP)
-#define IR_ERR_LEC (IR_STE | IR_FOE | IR_ACKE | IR_BE | IR_CRCE)
-#define IR_ERR_BUS (IR_ERR_LEC | IR_WDI | IR_ELO | IR_BEU | \
+
+/* Interrupts for version 3.0.x */
+#define IR_ERR_LEC_30X (IR_STE | IR_FOE | IR_ACKE | IR_BE | IR_CRCE)
+#define IR_ERR_BUS_30X (IR_ERR_LEC_30X | IR_WDI | IR_ELO | IR_BEU | \
IR_BEC | IR_TOO | IR_MRAF | IR_TSW | IR_TEFL | \
IR_RF1L | IR_RF0L)
-#define IR_ERR_ALL (IR_ERR_STATE | IR_ERR_BUS)
+#define IR_ERR_ALL_30X (IR_ERR_STATE | IR_ERR_BUS_30X)
+/* Interrupts for version >= 3.1.x */
+#define IR_ERR_LEC_31X (IR_PED | IR_PEA)
+#define IR_ERR_BUS_31X (IR_ERR_LEC_31X | IR_WDI | IR_ELO | IR_BEU | \
+ IR_BEC | IR_TOO | IR_MRAF | IR_TSW | IR_TEFL | \
+ IR_RF1L | IR_RF0L)
+#define IR_ERR_ALL_31X (IR_ERR_STATE | IR_ERR_BUS_31X)
/* Interrupt Line Select (ILS) */
#define ILS_ALL_INT0 0x0
#define ILS_ALL_INT1 0xFFFFFFFF
/* Interrupt Line Enable (ILE) */
-#define ILE_EINT0 BIT(0)
#define ILE_EINT1 BIT(1)
+#define ILE_EINT0 BIT(0)
/* Rx FIFO 0/1 Configuration (RXF0C/RXF1C) */
-#define RXFC_FWM_OFF 24
-#define RXFC_FWM_MASK 0x7f
-#define RXFC_FWM_1 (1 << RXFC_FWM_OFF)
-#define RXFC_FS_OFF 16
-#define RXFC_FS_MASK 0x7f
+#define RXFC_FWM_SHIFT 24
+#define RXFC_FWM_MASK (0x7f < RXFC_FWM_SHIFT)
+#define RXFC_FS_SHIFT 16
+#define RXFC_FS_MASK (0x7f << RXFC_FS_SHIFT)
/* Rx FIFO 0/1 Status (RXF0S/RXF1S) */
#define RXFS_RFL BIT(25)
#define RXFS_FF BIT(24)
-#define RXFS_FPI_OFF 16
+#define RXFS_FPI_SHIFT 16
#define RXFS_FPI_MASK 0x3f0000
-#define RXFS_FGI_OFF 8
+#define RXFS_FGI_SHIFT 8
#define RXFS_FGI_MASK 0x3f00
#define RXFS_FFL_MASK 0x7f
@@ -229,23 +264,46 @@
#define M_CAN_RXESC_64BYTES 0x777
/* Tx Buffer Configuration(TXBC) */
-#define TXBC_NDTB_OFF 16
-#define TXBC_NDTB_MASK 0x3f
+#define TXBC_NDTB_SHIFT 16
+#define TXBC_NDTB_MASK (0x3f << TXBC_NDTB_SHIFT)
+#define TXBC_TFQS_SHIFT 24
+#define TXBC_TFQS_MASK (0x3f << TXBC_TFQS_SHIFT)
+
+/* Tx FIFO/Queue Status (TXFQS) */
+#define TXFQS_TFQF BIT(21)
+#define TXFQS_TFQPI_SHIFT 16
+#define TXFQS_TFQPI_MASK (0x1f << TXFQS_TFQPI_SHIFT)
+#define TXFQS_TFGI_SHIFT 8
+#define TXFQS_TFGI_MASK (0x1f << TXFQS_TFGI_SHIFT)
+#define TXFQS_TFFL_SHIFT 0
+#define TXFQS_TFFL_MASK (0x3f << TXFQS_TFFL_SHIFT)
/* Tx Buffer Element Size Configuration(TXESC) */
#define TXESC_TBDS_8BYTES 0x0
#define TXESC_TBDS_64BYTES 0x7
-/* Tx Event FIFO Con.guration (TXEFC) */
-#define TXEFC_EFS_OFF 16
-#define TXEFC_EFS_MASK 0x3f
+/* Tx Event FIFO Configuration (TXEFC) */
+#define TXEFC_EFS_SHIFT 16
+#define TXEFC_EFS_MASK (0x3f << TXEFC_EFS_SHIFT)
+
+/* Tx Event FIFO Status (TXEFS) */
+#define TXEFS_TEFL BIT(25)
+#define TXEFS_EFF BIT(24)
+#define TXEFS_EFGI_SHIFT 8
+#define TXEFS_EFGI_MASK (0x1f << TXEFS_EFGI_SHIFT)
+#define TXEFS_EFFL_SHIFT 0
+#define TXEFS_EFFL_MASK (0x3f << TXEFS_EFFL_SHIFT)
+
+/* Tx Event FIFO Acknowledge (TXEFA) */
+#define TXEFA_EFAI_SHIFT 0
+#define TXEFA_EFAI_MASK (0x1f << TXEFA_EFAI_SHIFT)
/* Message RAM Configuration (in bytes) */
#define SIDF_ELEMENT_SIZE 4
#define XIDF_ELEMENT_SIZE 8
#define RXF0_ELEMENT_SIZE 72
#define RXF1_ELEMENT_SIZE 72
-#define RXB_ELEMENT_SIZE 16
+#define RXB_ELEMENT_SIZE 72
#define TXE_ELEMENT_SIZE 8
#define TXB_ELEMENT_SIZE 72
@@ -261,13 +319,25 @@
#define RX_BUF_RTR BIT(29)
/* R1 */
#define RX_BUF_ANMF BIT(31)
-#define RX_BUF_EDL BIT(21)
+#define RX_BUF_FDF BIT(21)
#define RX_BUF_BRS BIT(20)
/* Tx Buffer Element */
-/* R0 */
+/* T0 */
+#define TX_BUF_ESI BIT(31)
#define TX_BUF_XTD BIT(30)
#define TX_BUF_RTR BIT(29)
+/* T1 */
+#define TX_BUF_EFC BIT(23)
+#define TX_BUF_FDF BIT(21)
+#define TX_BUF_BRS BIT(20)
+#define TX_BUF_MM_SHIFT 24
+#define TX_BUF_MM_MASK (0xff << TX_BUF_MM_SHIFT)
+
+/* Tx event FIFO Element */
+/* E1 */
+#define TX_EVENT_MM_SHIFT TX_BUF_MM_SHIFT
+#define TX_EVENT_MM_MASK (0xff << TX_EVENT_MM_SHIFT)
/* address offset and element number for each FIFO/Buffer in the Message RAM */
struct mram_cfg {
@@ -285,6 +355,7 @@
struct clk *cclk;
void __iomem *base;
u32 irqstatus;
+ int version;
/* message ram configuration */
void __iomem *mram_base;
@@ -316,6 +387,18 @@
fpi * TXB_ELEMENT_SIZE + offset);
}
+static inline u32 m_can_txe_fifo_read(const struct m_can_priv *priv,
+ u32 fgi,
+ u32 offset) {
+ return readl(priv->mram_base + priv->mcfg[MRAM_TXE].off +
+ fgi * TXE_ELEMENT_SIZE + offset);
+}
+
+static inline bool m_can_tx_fifo_full(const struct m_can_priv *priv)
+{
+ return !!(m_can_read(priv, M_CAN_TXFQS) & TXFQS_TFQF);
+}
+
static inline void m_can_config_endisable(const struct m_can_priv *priv,
bool enable)
{
@@ -349,7 +432,8 @@
static inline void m_can_enable_all_interrupts(const struct m_can_priv *priv)
{
- m_can_write(priv, M_CAN_ILE, ILE_EINT0 | ILE_EINT1);
+ /* Only interrupt line 0 is used in this driver */
+ m_can_write(priv, M_CAN_ILE, ILE_EINT0);
}
static inline void m_can_disable_all_interrupts(const struct m_can_priv *priv)
@@ -367,9 +451,9 @@
int i;
/* calculate the fifo get index for where to read data */
- fgi = (rxfs & RXFS_FGI_MASK) >> RXFS_FGI_OFF;
+ fgi = (rxfs & RXFS_FGI_MASK) >> RXFS_FGI_SHIFT;
dlc = m_can_fifo_read(priv, fgi, M_CAN_FIFO_DLC);
- if (dlc & RX_BUF_EDL)
+ if (dlc & RX_BUF_FDF)
skb = alloc_canfd_skb(dev, &cf);
else
skb = alloc_can_skb(dev, (struct can_frame **)&cf);
@@ -378,7 +462,7 @@
return;
}
- if (dlc & RX_BUF_EDL)
+ if (dlc & RX_BUF_FDF)
cf->len = can_dlc2len((dlc >> 16) & 0x0F);
else
cf->len = get_can_dlc((dlc >> 16) & 0x0F);
@@ -394,7 +478,7 @@
netdev_dbg(dev, "ESI Error\n");
}
- if (!(dlc & RX_BUF_EDL) && (id & RX_BUF_RTR)) {
+ if (!(dlc & RX_BUF_FDF) && (id & RX_BUF_RTR)) {
cf->can_id |= CAN_RTR_FLAG;
} else {
if (dlc & RX_BUF_BRS)
@@ -532,7 +616,7 @@
ecr = m_can_read(priv, M_CAN_ECR);
bec->rxerr = (ecr & ECR_REC_MASK) >> ECR_REC_SHIFT;
- bec->txerr = ecr & ECR_TEC_MASK;
+ bec->txerr = (ecr & ECR_TEC_MASK) >> ECR_TEC_SHIFT;
return 0;
}
@@ -723,7 +807,7 @@
if (irqstatus & IR_ERR_STATE)
work_done += m_can_handle_state_errors(dev, psr);
- if (irqstatus & IR_ERR_BUS)
+ if (irqstatus & IR_ERR_BUS_30X)
work_done += m_can_handle_bus_errors(dev, irqstatus, psr);
if (irqstatus & IR_RF0N)
@@ -738,6 +822,44 @@
return work_done;
}
+static void m_can_echo_tx_event(struct net_device *dev)
+{
+ u32 txe_count = 0;
+ u32 m_can_txefs;
+ u32 fgi = 0;
+ int i = 0;
+ unsigned int msg_mark;
+
+ struct m_can_priv *priv = netdev_priv(dev);
+ struct net_device_stats *stats = &dev->stats;
+
+ /* read tx event fifo status */
+ m_can_txefs = m_can_read(priv, M_CAN_TXEFS);
+
+ /* Get Tx Event fifo element count */
+ txe_count = (m_can_txefs & TXEFS_EFFL_MASK)
+ >> TXEFS_EFFL_SHIFT;
+
+ /* Get and process all sent elements */
+ for (i = 0; i < txe_count; i++) {
+ /* retrieve get index */
+ fgi = (m_can_read(priv, M_CAN_TXEFS) & TXEFS_EFGI_MASK)
+ >> TXEFS_EFGI_SHIFT;
+
+ /* get message marker */
+ msg_mark = (m_can_txe_fifo_read(priv, fgi, 4) &
+ TX_EVENT_MM_MASK) >> TX_EVENT_MM_SHIFT;
+
+ /* ack txe element */
+ m_can_write(priv, M_CAN_TXEFA, (TXEFA_EFAI_MASK &
+ (fgi << TXEFA_EFAI_SHIFT)));
+
+ /* update stats */
+ stats->tx_bytes += can_get_echo_skb(dev, msg_mark);
+ stats->tx_packets++;
+ }
+}
+
static irqreturn_t m_can_isr(int irq, void *dev_id)
{
struct net_device *dev = (struct net_device *)dev_id;
@@ -758,24 +880,35 @@
* - state change IRQ
* - bus error IRQ and bus error reporting
*/
- if ((ir & IR_RF0N) || (ir & IR_ERR_ALL)) {
+ if ((ir & IR_RF0N) || (ir & IR_ERR_ALL_30X)) {
priv->irqstatus = ir;
m_can_disable_all_interrupts(priv);
napi_schedule(&priv->napi);
}
- /* transmission complete interrupt */
- if (ir & IR_TC) {
- stats->tx_bytes += can_get_echo_skb(dev, 0);
- stats->tx_packets++;
- can_led_event(dev, CAN_LED_EVENT_TX);
- netif_wake_queue(dev);
+ if (priv->version == 30) {
+ if (ir & IR_TC) {
+ /* Transmission Complete Interrupt*/
+ stats->tx_bytes += can_get_echo_skb(dev, 0);
+ stats->tx_packets++;
+ can_led_event(dev, CAN_LED_EVENT_TX);
+ netif_wake_queue(dev);
+ }
+ } else {
+ if (ir & IR_TEFN) {
+ /* New TX FIFO Element arrived */
+ m_can_echo_tx_event(dev);
+ can_led_event(dev, CAN_LED_EVENT_TX);
+ if (netif_queue_stopped(dev) &&
+ !m_can_tx_fifo_full(priv))
+ netif_wake_queue(dev);
+ }
}
return IRQ_HANDLED;
}
-static const struct can_bittiming_const m_can_bittiming_const = {
+static const struct can_bittiming_const m_can_bittiming_const_30X = {
.name = KBUILD_MODNAME,
.tseg1_min = 2, /* Time segment 1 = prop_seg + phase_seg1 */
.tseg1_max = 64,
@@ -787,7 +920,7 @@
.brp_inc = 1,
};
-static const struct can_bittiming_const m_can_data_bittiming_const = {
+static const struct can_bittiming_const m_can_data_bittiming_const_30X = {
.name = KBUILD_MODNAME,
.tseg1_min = 2, /* Time segment 1 = prop_seg + phase_seg1 */
.tseg1_max = 16,
@@ -799,6 +932,30 @@
.brp_inc = 1,
};
+static const struct can_bittiming_const m_can_bittiming_const_31X = {
+ .name = KBUILD_MODNAME,
+ .tseg1_min = 2, /* Time segment 1 = prop_seg + phase_seg1 */
+ .tseg1_max = 256,
+ .tseg2_min = 1, /* Time segment 2 = phase_seg2 */
+ .tseg2_max = 128,
+ .sjw_max = 128,
+ .brp_min = 1,
+ .brp_max = 512,
+ .brp_inc = 1,
+};
+
+static const struct can_bittiming_const m_can_data_bittiming_const_31X = {
+ .name = KBUILD_MODNAME,
+ .tseg1_min = 1, /* Time segment 1 = prop_seg + phase_seg1 */
+ .tseg1_max = 32,
+ .tseg2_min = 1, /* Time segment 2 = phase_seg2 */
+ .tseg2_max = 16,
+ .sjw_max = 16,
+ .brp_min = 1,
+ .brp_max = 32,
+ .brp_inc = 1,
+};
+
static int m_can_set_bittiming(struct net_device *dev)
{
struct m_can_priv *priv = netdev_priv(dev);
@@ -811,19 +968,19 @@
sjw = bt->sjw - 1;
tseg1 = bt->prop_seg + bt->phase_seg1 - 1;
tseg2 = bt->phase_seg2 - 1;
- reg_btp = (brp << BTR_BRP_SHIFT) | (sjw << BTR_SJW_SHIFT) |
- (tseg1 << BTR_TSEG1_SHIFT) | (tseg2 << BTR_TSEG2_SHIFT);
- m_can_write(priv, M_CAN_BTP, reg_btp);
+ reg_btp = (brp << NBTP_NBRP_SHIFT) | (sjw << NBTP_NSJW_SHIFT) |
+ (tseg1 << NBTP_NTSEG1_SHIFT) | (tseg2 << NBTP_NTSEG2_SHIFT);
+ m_can_write(priv, M_CAN_NBTP, reg_btp);
if (priv->can.ctrlmode & CAN_CTRLMODE_FD) {
brp = dbt->brp - 1;
sjw = dbt->sjw - 1;
tseg1 = dbt->prop_seg + dbt->phase_seg1 - 1;
tseg2 = dbt->phase_seg2 - 1;
- reg_btp = (brp << FBTR_FBRP_SHIFT) | (sjw << FBTR_FSJW_SHIFT) |
- (tseg1 << FBTR_FTSEG1_SHIFT) |
- (tseg2 << FBTR_FTSEG2_SHIFT);
- m_can_write(priv, M_CAN_FBTP, reg_btp);
+ reg_btp = (brp << DBTP_DBRP_SHIFT) | (sjw << DBTP_DSJW_SHIFT) |
+ (tseg1 << DBTP_DTSEG1_SHIFT) |
+ (tseg2 << DBTP_DTSEG2_SHIFT);
+ m_can_write(priv, M_CAN_DBTP, reg_btp);
}
return 0;
@@ -834,6 +991,7 @@
* - configure rx fifo
* - accept non-matching frame into fifo 0
* - configure tx buffer
+ * - >= v3.1.x: TX FIFO is used
* - configure mode
* - setup bittiming
*/
@@ -850,49 +1008,89 @@
/* Accept Non-matching Frames Into FIFO 0 */
m_can_write(priv, M_CAN_GFC, 0x0);
- /* only support one Tx Buffer currently */
- m_can_write(priv, M_CAN_TXBC, (1 << TXBC_NDTB_OFF) |
- priv->mcfg[MRAM_TXB].off);
+ if (priv->version == 30) {
+ /* only support one Tx Buffer currently */
+ m_can_write(priv, M_CAN_TXBC, (1 << TXBC_NDTB_SHIFT) |
+ priv->mcfg[MRAM_TXB].off);
+ } else {
+ /* TX FIFO is used for newer IP Core versions */
+ m_can_write(priv, M_CAN_TXBC,
+ (priv->mcfg[MRAM_TXB].num << TXBC_TFQS_SHIFT) |
+ (priv->mcfg[MRAM_TXB].off));
+ }
/* support 64 bytes payload */
m_can_write(priv, M_CAN_TXESC, TXESC_TBDS_64BYTES);
- m_can_write(priv, M_CAN_TXEFC, (1 << TXEFC_EFS_OFF) |
- priv->mcfg[MRAM_TXE].off);
+ /* TX Event FIFO */
+ if (priv->version == 30) {
+ m_can_write(priv, M_CAN_TXEFC, (1 << TXEFC_EFS_SHIFT) |
+ priv->mcfg[MRAM_TXE].off);
+ } else {
+ /* Full TX Event FIFO is used */
+ m_can_write(priv, M_CAN_TXEFC,
+ ((priv->mcfg[MRAM_TXE].num << TXEFC_EFS_SHIFT)
+ & TXEFC_EFS_MASK) |
+ priv->mcfg[MRAM_TXE].off);
+ }
/* rx fifo configuration, blocking mode, fifo size 1 */
m_can_write(priv, M_CAN_RXF0C,
- (priv->mcfg[MRAM_RXF0].num << RXFC_FS_OFF) |
- RXFC_FWM_1 | priv->mcfg[MRAM_RXF0].off);
+ (priv->mcfg[MRAM_RXF0].num << RXFC_FS_SHIFT) |
+ priv->mcfg[MRAM_RXF0].off);
m_can_write(priv, M_CAN_RXF1C,
- (priv->mcfg[MRAM_RXF1].num << RXFC_FS_OFF) |
- RXFC_FWM_1 | priv->mcfg[MRAM_RXF1].off);
+ (priv->mcfg[MRAM_RXF1].num << RXFC_FS_SHIFT) |
+ priv->mcfg[MRAM_RXF1].off);
cccr = m_can_read(priv, M_CAN_CCCR);
- cccr &= ~(CCCR_TEST | CCCR_MON | (CCCR_CMR_MASK << CCCR_CMR_SHIFT) |
- (CCCR_CME_MASK << CCCR_CME_SHIFT));
test = m_can_read(priv, M_CAN_TEST);
test &= ~TEST_LBCK;
+ if (priv->version == 30) {
+ /* Version 3.0.x */
- if (priv->can.ctrlmode & CAN_CTRLMODE_LISTENONLY)
- cccr |= CCCR_MON;
+ cccr &= ~(CCCR_TEST | CCCR_MON |
+ (CCCR_CMR_MASK << CCCR_CMR_SHIFT) |
+ (CCCR_CME_MASK << CCCR_CME_SHIFT));
+ if (priv->can.ctrlmode & CAN_CTRLMODE_FD)
+ cccr |= CCCR_CME_CANFD_BRS << CCCR_CME_SHIFT;
+
+ } else {
+ /* Version 3.1.x or 3.2.x */
+ cccr &= ~(CCCR_TEST | CCCR_MON | CCCR_BRSE | CCCR_FDOE);
+
+ /* Only 3.2.x has NISO Bit implemented */
+ if (priv->can.ctrlmode & CAN_CTRLMODE_FD_NON_ISO)
+ cccr |= CCCR_NISO;
+
+ if (priv->can.ctrlmode & CAN_CTRLMODE_FD)
+ cccr |= (CCCR_BRSE | CCCR_FDOE);
+ }
+
+ /* Loopback Mode */
if (priv->can.ctrlmode & CAN_CTRLMODE_LOOPBACK) {
- cccr |= CCCR_TEST;
+ cccr |= CCCR_TEST | CCCR_MON;
test |= TEST_LBCK;
}
- if (priv->can.ctrlmode & CAN_CTRLMODE_FD)
- cccr |= CCCR_CME_CANFD_BRS << CCCR_CME_SHIFT;
+ /* Enable Monitoring (all versions) */
+ if (priv->can.ctrlmode & CAN_CTRLMODE_LISTENONLY)
+ cccr |= CCCR_MON;
+ /* Write config */
m_can_write(priv, M_CAN_CCCR, cccr);
m_can_write(priv, M_CAN_TEST, test);
- /* enable interrupts */
+ /* Enable interrupts */
m_can_write(priv, M_CAN_IR, IR_ALL_INT);
if (!(priv->can.ctrlmode & CAN_CTRLMODE_BERR_REPORTING))
- m_can_write(priv, M_CAN_IE, IR_ALL_INT & ~IR_ERR_LEC);
+ if (priv->version == 30)
+ m_can_write(priv, M_CAN_IE, IR_ALL_INT &
+ ~(IR_ERR_LEC_30X));
+ else
+ m_can_write(priv, M_CAN_IE, IR_ALL_INT &
+ ~(IR_ERR_LEC_31X));
else
m_can_write(priv, M_CAN_IE, IR_ALL_INT);
@@ -936,33 +1134,140 @@
free_candev(dev);
}
-static struct net_device *alloc_m_can_dev(void)
+/* Checks core release number of M_CAN
+ * returns 0 if an unsupported device is detected
+ * else it returns the release and step coded as:
+ * return value = 10 * <release> + 1 * <step>
+ */
+static int m_can_check_core_release(void __iomem *m_can_base)
+{
+ u32 crel_reg;
+ u8 rel;
+ u8 step;
+ int res;
+ struct m_can_priv temp_priv = {
+ .base = m_can_base
+ };
+
+ /* Read Core Release Version and split into version number
+ * Example: Version 3.2.1 => rel = 3; step = 2; substep = 1;
+ */
+ crel_reg = m_can_read(&temp_priv, M_CAN_CREL);
+ rel = (u8)((crel_reg & CREL_REL_MASK) >> CREL_REL_SHIFT);
+ step = (u8)((crel_reg & CREL_STEP_MASK) >> CREL_STEP_SHIFT);
+
+ if (rel == 3) {
+ /* M_CAN v3.x.y: create return value */
+ res = 30 + step;
+ } else {
+ /* Unsupported M_CAN version */
+ res = 0;
+ }
+
+ return res;
+}
+
+/* Selectable Non ISO support only in version 3.2.x
+ * This function checks if the bit is writable.
+ */
+static bool m_can_niso_supported(const struct m_can_priv *priv)
+{
+ u32 cccr_reg, cccr_poll;
+ int niso_timeout;
+
+ m_can_config_endisable(priv, true);
+ cccr_reg = m_can_read(priv, M_CAN_CCCR);
+ cccr_reg |= CCCR_NISO;
+ m_can_write(priv, M_CAN_CCCR, cccr_reg);
+
+ niso_timeout = readl_poll_timeout((priv->base + M_CAN_CCCR), cccr_poll,
+ (cccr_poll == cccr_reg), 0, 10);
+
+ /* Clear NISO */
+ cccr_reg &= ~(CCCR_NISO);
+ m_can_write(priv, M_CAN_CCCR, cccr_reg);
+
+ m_can_config_endisable(priv, false);
+
+ /* return false if time out (-ETIMEDOUT), else return true */
+ return !niso_timeout;
+}
+
+static struct net_device *alloc_m_can_dev(struct platform_device *pdev,
+ void __iomem *addr, u32 tx_fifo_size)
{
struct net_device *dev;
struct m_can_priv *priv;
+ int m_can_version;
+ unsigned int echo_buffer_count;
- dev = alloc_candev(sizeof(*priv), 1);
- if (!dev)
- return NULL;
+ m_can_version = m_can_check_core_release(addr);
+ /* return if unsupported version */
+ if (!m_can_version) {
+ dev = NULL;
+ goto return_dev;
+ }
+ /* If version < 3.1.x, then only one echo buffer is used */
+ echo_buffer_count = ((m_can_version == 30)
+ ? 1U
+ : (unsigned int)tx_fifo_size);
+
+ dev = alloc_candev(sizeof(*priv), echo_buffer_count);
+ if (!dev) {
+ dev = NULL;
+ goto return_dev;
+ }
priv = netdev_priv(dev);
netif_napi_add(dev, &priv->napi, m_can_poll, M_CAN_NAPI_WEIGHT);
+ /* Shared properties of all M_CAN versions */
+ priv->version = m_can_version;
priv->dev = dev;
- priv->can.bittiming_const = &m_can_bittiming_const;
- priv->can.data_bittiming_const = &m_can_data_bittiming_const;
+ priv->base = addr;
priv->can.do_set_mode = m_can_set_mode;
priv->can.do_get_berr_counter = m_can_get_berr_counter;
- /* CAN_CTRLMODE_FD_NON_ISO is fixed with M_CAN IP v3.0.1 */
- can_set_static_ctrlmode(dev, CAN_CTRLMODE_FD_NON_ISO);
-
- /* CAN_CTRLMODE_FD_NON_ISO can not be changed with M_CAN IP v3.0.1 */
+ /* Set M_CAN supported operations */
priv->can.ctrlmode_supported = CAN_CTRLMODE_LOOPBACK |
CAN_CTRLMODE_LISTENONLY |
CAN_CTRLMODE_BERR_REPORTING |
CAN_CTRLMODE_FD;
+ /* Set properties depending on M_CAN version */
+ switch (priv->version) {
+ case 30:
+ /* CAN_CTRLMODE_FD_NON_ISO is fixed with M_CAN IP v3.0.x */
+ can_set_static_ctrlmode(dev, CAN_CTRLMODE_FD_NON_ISO);
+ priv->can.bittiming_const = &m_can_bittiming_const_30X;
+ priv->can.data_bittiming_const =
+ &m_can_data_bittiming_const_30X;
+ break;
+ case 31:
+ /* CAN_CTRLMODE_FD_NON_ISO is fixed with M_CAN IP v3.1.x */
+ can_set_static_ctrlmode(dev, CAN_CTRLMODE_FD_NON_ISO);
+ priv->can.bittiming_const = &m_can_bittiming_const_31X;
+ priv->can.data_bittiming_const =
+ &m_can_data_bittiming_const_31X;
+ break;
+ case 32:
+ priv->can.bittiming_const = &m_can_bittiming_const_31X;
+ priv->can.data_bittiming_const =
+ &m_can_data_bittiming_const_31X;
+ priv->can.ctrlmode_supported |= (m_can_niso_supported(priv)
+ ? CAN_CTRLMODE_FD_NON_ISO
+ : 0);
+ break;
+ default:
+ /* Unsupported device: free candev */
+ free_m_can_dev(dev);
+ dev_err(&pdev->dev, "Unsupported version number: %2d",
+ priv->version);
+ dev = NULL;
+ break;
+ }
+
+return_dev:
return dev;
}
@@ -1040,19 +1345,34 @@
return 0;
}
+static int m_can_next_echo_skb_occupied(struct net_device *dev, int putidx)
+{
+ struct m_can_priv *priv = netdev_priv(dev);
+ /*get wrap around for loopback skb index */
+ unsigned int wrap = priv->can.echo_skb_max;
+ int next_idx;
+
+ /* calculate next index */
+ next_idx = (++putidx >= wrap ? 0 : putidx);
+
+ /* check if occupied */
+ return !!priv->can.echo_skb[next_idx];
+}
+
static netdev_tx_t m_can_start_xmit(struct sk_buff *skb,
struct net_device *dev)
{
struct m_can_priv *priv = netdev_priv(dev);
struct canfd_frame *cf = (struct canfd_frame *)skb->data;
- u32 id, cccr;
+ u32 id, cccr, fdflags;
int i;
+ int putidx;
if (can_dropped_invalid_skb(dev, skb))
return NETDEV_TX_OK;
- netif_stop_queue(dev);
-
+ /* Generate ID field for TX buffer Element */
+ /* Common to all supported M_CAN versions */
if (cf->can_id & CAN_EFF_FLAG) {
id = cf->can_id & CAN_EFF_MASK;
id |= TX_BUF_XTD;
@@ -1063,33 +1383,93 @@
if (cf->can_id & CAN_RTR_FLAG)
id |= TX_BUF_RTR;
- /* message ram configuration */
- m_can_fifo_write(priv, 0, M_CAN_FIFO_ID, id);
- m_can_fifo_write(priv, 0, M_CAN_FIFO_DLC, can_len2dlc(cf->len) << 16);
+ if (priv->version == 30) {
+ netif_stop_queue(dev);
- for (i = 0; i < cf->len; i += 4)
- m_can_fifo_write(priv, 0, M_CAN_FIFO_DATA(i / 4),
- *(u32 *)(cf->data + i));
+ /* message ram configuration */
+ m_can_fifo_write(priv, 0, M_CAN_FIFO_ID, id);
+ m_can_fifo_write(priv, 0, M_CAN_FIFO_DLC,
+ can_len2dlc(cf->len) << 16);
- can_put_echo_skb(skb, dev, 0);
+ for (i = 0; i < cf->len; i += 4)
+ m_can_fifo_write(priv, 0,
+ M_CAN_FIFO_DATA(i / 4),
+ *(u32 *)(cf->data + i));
- if (priv->can.ctrlmode & CAN_CTRLMODE_FD) {
- cccr = m_can_read(priv, M_CAN_CCCR);
- cccr &= ~(CCCR_CMR_MASK << CCCR_CMR_SHIFT);
- if (can_is_canfd_skb(skb)) {
- if (cf->flags & CANFD_BRS)
- cccr |= CCCR_CMR_CANFD_BRS << CCCR_CMR_SHIFT;
- else
- cccr |= CCCR_CMR_CANFD << CCCR_CMR_SHIFT;
- } else {
- cccr |= CCCR_CMR_CAN << CCCR_CMR_SHIFT;
+ can_put_echo_skb(skb, dev, 0);
+
+ if (priv->can.ctrlmode & CAN_CTRLMODE_FD) {
+ cccr = m_can_read(priv, M_CAN_CCCR);
+ cccr &= ~(CCCR_CMR_MASK << CCCR_CMR_SHIFT);
+ if (can_is_canfd_skb(skb)) {
+ if (cf->flags & CANFD_BRS)
+ cccr |= CCCR_CMR_CANFD_BRS <<
+ CCCR_CMR_SHIFT;
+ else
+ cccr |= CCCR_CMR_CANFD <<
+ CCCR_CMR_SHIFT;
+ } else {
+ cccr |= CCCR_CMR_CAN << CCCR_CMR_SHIFT;
+ }
+ m_can_write(priv, M_CAN_CCCR, cccr);
}
- m_can_write(priv, M_CAN_CCCR, cccr);
- }
+ m_can_write(priv, M_CAN_TXBTIE, 0x1);
+ m_can_write(priv, M_CAN_TXBAR, 0x1);
+ /* End of xmit function for version 3.0.x */
+ } else {
+ /* Transmit routine for version >= v3.1.x */
- /* enable first TX buffer to start transfer */
- m_can_write(priv, M_CAN_TXBTIE, 0x1);
- m_can_write(priv, M_CAN_TXBAR, 0x1);
+ /* Check if FIFO full */
+ if (m_can_tx_fifo_full(priv)) {
+ /* This shouldn't happen */
+ netif_stop_queue(dev);
+ netdev_warn(dev,
+ "TX queue active although FIFO is full.");
+ return NETDEV_TX_BUSY;
+ }
+
+ /* get put index for frame */
+ putidx = ((m_can_read(priv, M_CAN_TXFQS) & TXFQS_TFQPI_MASK)
+ >> TXFQS_TFQPI_SHIFT);
+ /* Write ID Field to FIFO Element */
+ m_can_fifo_write(priv, putidx, M_CAN_FIFO_ID, id);
+
+ /* get CAN FD configuration of frame */
+ fdflags = 0;
+ if (can_is_canfd_skb(skb)) {
+ fdflags |= TX_BUF_FDF;
+ if (cf->flags & CANFD_BRS)
+ fdflags |= TX_BUF_BRS;
+ }
+
+ /* Construct DLC Field. Also contains CAN-FD configuration
+ * use put index of fifo as message marker
+ * it is used in TX interrupt for
+ * sending the correct echo frame
+ */
+ m_can_fifo_write(priv, putidx, M_CAN_FIFO_DLC,
+ ((putidx << TX_BUF_MM_SHIFT) &
+ TX_BUF_MM_MASK) |
+ (can_len2dlc(cf->len) << 16) |
+ fdflags | TX_BUF_EFC);
+
+ for (i = 0; i < cf->len; i += 4)
+ m_can_fifo_write(priv, putidx, M_CAN_FIFO_DATA(i / 4),
+ *(u32 *)(cf->data + i));
+
+ /* Push loopback echo.
+ * Will be looped back on TX interrupt based on message marker
+ */
+ can_put_echo_skb(skb, dev, putidx);
+
+ /* Enable TX FIFO element to start transfer */
+ m_can_write(priv, M_CAN_TXBAR, (1 << putidx));
+
+ /* stop network queue if fifo full */
+ if (m_can_tx_fifo_full(priv) ||
+ m_can_next_echo_skb_occupied(dev, putidx))
+ netif_stop_queue(dev);
+ }
return NETDEV_TX_OK;
}
@@ -1109,55 +1489,37 @@
return register_candev(dev);
}
-static int m_can_of_parse_mram(struct platform_device *pdev,
- struct m_can_priv *priv)
+static void m_can_of_parse_mram(struct m_can_priv *priv,
+ const u32 *mram_config_vals)
{
- struct device_node *np = pdev->dev.of_node;
- struct resource *res;
- void __iomem *addr;
- u32 out_val[MRAM_CFG_LEN];
- int i, start, end, ret;
+ int i, start, end;
- /* message ram could be shared */
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "message_ram");
- if (!res)
- return -ENODEV;
-
- addr = devm_ioremap(&pdev->dev, res->start, resource_size(res));
- if (!addr)
- return -ENOMEM;
-
- /* get message ram configuration */
- ret = of_property_read_u32_array(np, "bosch,mram-cfg",
- out_val, sizeof(out_val) / 4);
- if (ret) {
- dev_err(&pdev->dev, "can not get message ram configuration\n");
- return -ENODEV;
- }
-
- priv->mram_base = addr;
- priv->mcfg[MRAM_SIDF].off = out_val[0];
- priv->mcfg[MRAM_SIDF].num = out_val[1];
+ priv->mcfg[MRAM_SIDF].off = mram_config_vals[0];
+ priv->mcfg[MRAM_SIDF].num = mram_config_vals[1];
priv->mcfg[MRAM_XIDF].off = priv->mcfg[MRAM_SIDF].off +
priv->mcfg[MRAM_SIDF].num * SIDF_ELEMENT_SIZE;
- priv->mcfg[MRAM_XIDF].num = out_val[2];
+ priv->mcfg[MRAM_XIDF].num = mram_config_vals[2];
priv->mcfg[MRAM_RXF0].off = priv->mcfg[MRAM_XIDF].off +
priv->mcfg[MRAM_XIDF].num * XIDF_ELEMENT_SIZE;
- priv->mcfg[MRAM_RXF0].num = out_val[3] & RXFC_FS_MASK;
+ priv->mcfg[MRAM_RXF0].num = mram_config_vals[3] &
+ (RXFC_FS_MASK >> RXFC_FS_SHIFT);
priv->mcfg[MRAM_RXF1].off = priv->mcfg[MRAM_RXF0].off +
priv->mcfg[MRAM_RXF0].num * RXF0_ELEMENT_SIZE;
- priv->mcfg[MRAM_RXF1].num = out_val[4] & RXFC_FS_MASK;
+ priv->mcfg[MRAM_RXF1].num = mram_config_vals[4] &
+ (RXFC_FS_MASK >> RXFC_FS_SHIFT);
priv->mcfg[MRAM_RXB].off = priv->mcfg[MRAM_RXF1].off +
priv->mcfg[MRAM_RXF1].num * RXF1_ELEMENT_SIZE;
- priv->mcfg[MRAM_RXB].num = out_val[5];
+ priv->mcfg[MRAM_RXB].num = mram_config_vals[5];
priv->mcfg[MRAM_TXE].off = priv->mcfg[MRAM_RXB].off +
priv->mcfg[MRAM_RXB].num * RXB_ELEMENT_SIZE;
- priv->mcfg[MRAM_TXE].num = out_val[6];
+ priv->mcfg[MRAM_TXE].num = mram_config_vals[6];
priv->mcfg[MRAM_TXB].off = priv->mcfg[MRAM_TXE].off +
priv->mcfg[MRAM_TXE].num * TXE_ELEMENT_SIZE;
- priv->mcfg[MRAM_TXB].num = out_val[7] & TXBC_NDTB_MASK;
+ priv->mcfg[MRAM_TXB].num = mram_config_vals[7] &
+ (TXBC_NDTB_MASK >> TXBC_NDTB_SHIFT);
- dev_dbg(&pdev->dev, "mram_base %p sidf 0x%x %d xidf 0x%x %d rxf0 0x%x %d rxf1 0x%x %d rxb 0x%x %d txe 0x%x %d txb 0x%x %d\n",
+ dev_dbg(priv->device,
+ "mram_base %p sidf 0x%x %d xidf 0x%x %d rxf0 0x%x %d rxf1 0x%x %d rxb 0x%x %d txe 0x%x %d txb 0x%x %d\n",
priv->mram_base,
priv->mcfg[MRAM_SIDF].off, priv->mcfg[MRAM_SIDF].num,
priv->mcfg[MRAM_XIDF].off, priv->mcfg[MRAM_XIDF].num,
@@ -1176,7 +1538,6 @@
for (i = start; i < end; i += 4)
writel(0x0, priv->mram_base + i);
- return 0;
}
static int m_can_plat_probe(struct platform_device *pdev)
@@ -1185,38 +1546,86 @@
struct m_can_priv *priv;
struct resource *res;
void __iomem *addr;
+ void __iomem *mram_addr;
struct clk *hclk, *cclk;
int irq, ret;
+ struct device_node *np;
+ u32 mram_config_vals[MRAM_CFG_LEN];
+ u32 tx_fifo_size;
+
+ np = pdev->dev.of_node;
hclk = devm_clk_get(&pdev->dev, "hclk");
cclk = devm_clk_get(&pdev->dev, "cclk");
+
if (IS_ERR(hclk) || IS_ERR(cclk)) {
- dev_err(&pdev->dev, "no clock find\n");
- return -ENODEV;
+ dev_err(&pdev->dev, "no clock found\n");
+ ret = -ENODEV;
+ goto failed_ret;
}
+ /* Enable clocks. Necessary to read Core Release in order to determine
+ * M_CAN version
+ */
+ ret = clk_prepare_enable(hclk);
+ if (ret)
+ goto disable_hclk_ret;
+
+ ret = clk_prepare_enable(cclk);
+ if (ret)
+ goto disable_cclk_ret;
+
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "m_can");
addr = devm_ioremap_resource(&pdev->dev, res);
irq = platform_get_irq_byname(pdev, "int0");
- if (IS_ERR(addr) || irq < 0)
- return -EINVAL;
+
+ if (IS_ERR(addr) || irq < 0) {
+ ret = -EINVAL;
+ goto disable_cclk_ret;
+ }
+
+ /* message ram could be shared */
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "message_ram");
+ if (!res) {
+ ret = -ENODEV;
+ goto disable_cclk_ret;
+ }
+
+ mram_addr = devm_ioremap(&pdev->dev, res->start, resource_size(res));
+ if (!mram_addr) {
+ ret = -ENOMEM;
+ goto disable_cclk_ret;
+ }
+
+ /* get message ram configuration */
+ ret = of_property_read_u32_array(np, "bosch,mram-cfg",
+ mram_config_vals,
+ sizeof(mram_config_vals) / 4);
+ if (ret) {
+ dev_err(&pdev->dev, "Could not get Message RAM configuration.");
+ goto disable_cclk_ret;
+ }
+
+ /* Get TX FIFO size
+ * Defines the total amount of echo buffers for loopback
+ */
+ tx_fifo_size = mram_config_vals[7];
/* allocate the m_can device */
- dev = alloc_m_can_dev();
- if (!dev)
- return -ENOMEM;
-
+ dev = alloc_m_can_dev(pdev, addr, tx_fifo_size);
+ if (!dev) {
+ ret = -ENOMEM;
+ goto disable_cclk_ret;
+ }
priv = netdev_priv(dev);
dev->irq = irq;
- priv->base = addr;
priv->device = &pdev->dev;
priv->hclk = hclk;
priv->cclk = cclk;
priv->can.clock.freq = clk_get_rate(cclk);
+ priv->mram_base = mram_addr;
- ret = m_can_of_parse_mram(pdev, priv);
- if (ret)
- goto failed_free_dev;
+ m_can_of_parse_mram(priv, mram_config_vals);
platform_set_drvdata(pdev, dev);
SET_NETDEV_DEV(dev, &pdev->dev);
@@ -1230,13 +1639,22 @@
devm_can_led_init(dev);
- dev_info(&pdev->dev, "%s device registered (regs=%p, irq=%d)\n",
- KBUILD_MODNAME, priv->base, dev->irq);
+ dev_info(&pdev->dev, "%s device registered (irq=%d, version=%d)\n",
+ KBUILD_MODNAME, dev->irq, priv->version);
- return 0;
+ /* Probe finished
+ * Stop clocks. They will be reactivated once the M_CAN device is opened
+ */
+
+ goto disable_cclk_ret;
failed_free_dev:
free_m_can_dev(dev);
+disable_cclk_ret:
+ clk_disable_unprepare(cclk);
+disable_hclk_ret:
+ clk_disable_unprepare(hclk);
+failed_ret:
return ret;
}
diff --git a/drivers/net/can/peak_canfd/Kconfig b/drivers/net/can/peak_canfd/Kconfig
new file mode 100644
index 0000000..84b3097
--- /dev/null
+++ b/drivers/net/can/peak_canfd/Kconfig
@@ -0,0 +1,13 @@
+config CAN_PEAK_PCIEFD
+ depends on PCI
+ tristate "PEAK-System PCAN-PCIe FD cards"
+ ---help---
+ This driver adds support for the PEAK-System PCI Express FD
+ CAN-FD cards family.
+ These 1x or 2x CAN-FD channels cards offer CAN 2.0 a/b as well as
+ CAN-FD access to the CAN bus. Besides the nominal bitrate of up to
+ 1 Mbit/s, the data bytes of CAN-FD frames can be transmitted with
+ up to 12 Mbit/s. A galvanic isolation of the CAN ports protects the
+ electronics of the card and the respective computer against
+ disturbances of up to 500 Volts. The PCAN-PCI Express FD can be
+ operated with ambient temperatures in a range of -40 to +85 °C.
diff --git a/drivers/net/can/peak_canfd/Makefile b/drivers/net/can/peak_canfd/Makefile
new file mode 100644
index 0000000..3dc7a6a
--- /dev/null
+++ b/drivers/net/can/peak_canfd/Makefile
@@ -0,0 +1,5 @@
+#
+# Makefile for the PEAK-System CAN-FD IP module drivers
+#
+obj-$(CONFIG_CAN_PEAK_PCIEFD) += peak_pciefd.o
+peak_pciefd-y := peak_pciefd_main.o peak_canfd.o
diff --git a/drivers/net/can/peak_canfd/peak_canfd.c b/drivers/net/can/peak_canfd/peak_canfd.c
new file mode 100644
index 0000000..0d57be5
--- /dev/null
+++ b/drivers/net/can/peak_canfd/peak_canfd.c
@@ -0,0 +1,801 @@
+/*
+ * Copyright (C) 2007, 2011 Wolfgang Grandegger <wg@grandegger.com>
+ * Copyright (C) 2012 Stephane Grosjean <s.grosjean@peak-system.com>
+ *
+ * Copyright (C) 2016 PEAK System-Technik GmbH
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the version 2 of the GNU General Public License
+ * as published by the Free Software Foundation
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/can.h>
+#include <linux/can/dev.h>
+
+#include "peak_canfd_user.h"
+
+/* internal IP core cache size (used as default echo skbs max number) */
+#define PCANFD_ECHO_SKB_MAX 24
+
+/* bittiming ranges of the PEAK-System PC CAN-FD interfaces */
+static const struct can_bittiming_const peak_canfd_nominal_const = {
+ .name = "peak_canfd",
+ .tseg1_min = 1,
+ .tseg1_max = (1 << PUCAN_TSLOW_TSGEG1_BITS),
+ .tseg2_min = 1,
+ .tseg2_max = (1 << PUCAN_TSLOW_TSGEG2_BITS),
+ .sjw_max = (1 << PUCAN_TSLOW_SJW_BITS),
+ .brp_min = 1,
+ .brp_max = (1 << PUCAN_TSLOW_BRP_BITS),
+ .brp_inc = 1,
+};
+
+static const struct can_bittiming_const peak_canfd_data_const = {
+ .name = "peak_canfd",
+ .tseg1_min = 1,
+ .tseg1_max = (1 << PUCAN_TFAST_TSGEG1_BITS),
+ .tseg2_min = 1,
+ .tseg2_max = (1 << PUCAN_TFAST_TSGEG2_BITS),
+ .sjw_max = (1 << PUCAN_TFAST_SJW_BITS),
+ .brp_min = 1,
+ .brp_max = (1 << PUCAN_TFAST_BRP_BITS),
+ .brp_inc = 1,
+};
+
+static struct peak_canfd_priv *pucan_init_cmd(struct peak_canfd_priv *priv)
+{
+ priv->cmd_len = 0;
+ return priv;
+}
+
+static void *pucan_add_cmd(struct peak_canfd_priv *priv, int cmd_op)
+{
+ struct pucan_command *cmd;
+
+ if (priv->cmd_len + sizeof(*cmd) > priv->cmd_maxlen)
+ return NULL;
+
+ cmd = priv->cmd_buffer + priv->cmd_len;
+
+ /* reset all unused bit to default */
+ memset(cmd, 0, sizeof(*cmd));
+
+ cmd->opcode_channel = pucan_cmd_opcode_channel(priv->index, cmd_op);
+ priv->cmd_len += sizeof(*cmd);
+
+ return cmd;
+}
+
+static int pucan_write_cmd(struct peak_canfd_priv *priv)
+{
+ int err;
+
+ if (priv->pre_cmd) {
+ err = priv->pre_cmd(priv);
+ if (err)
+ return err;
+ }
+
+ err = priv->write_cmd(priv);
+ if (err)
+ return err;
+
+ if (priv->post_cmd)
+ err = priv->post_cmd(priv);
+
+ return err;
+}
+
+/* uCAN commands interface functions */
+static int pucan_set_reset_mode(struct peak_canfd_priv *priv)
+{
+ pucan_add_cmd(pucan_init_cmd(priv), PUCAN_CMD_RESET_MODE);
+ return pucan_write_cmd(priv);
+}
+
+static int pucan_set_normal_mode(struct peak_canfd_priv *priv)
+{
+ int err;
+
+ pucan_add_cmd(pucan_init_cmd(priv), PUCAN_CMD_NORMAL_MODE);
+ err = pucan_write_cmd(priv);
+ if (!err)
+ priv->can.state = CAN_STATE_ERROR_ACTIVE;
+
+ return err;
+}
+
+static int pucan_set_listen_only_mode(struct peak_canfd_priv *priv)
+{
+ int err;
+
+ pucan_add_cmd(pucan_init_cmd(priv), PUCAN_CMD_LISTEN_ONLY_MODE);
+ err = pucan_write_cmd(priv);
+ if (!err)
+ priv->can.state = CAN_STATE_ERROR_ACTIVE;
+
+ return err;
+}
+
+static int pucan_set_timing_slow(struct peak_canfd_priv *priv,
+ const struct can_bittiming *pbt)
+{
+ struct pucan_timing_slow *cmd;
+
+ cmd = pucan_add_cmd(pucan_init_cmd(priv), PUCAN_CMD_TIMING_SLOW);
+
+ cmd->sjw_t = PUCAN_TSLOW_SJW_T(pbt->sjw - 1,
+ priv->can.ctrlmode & CAN_CTRLMODE_3_SAMPLES);
+ cmd->tseg1 = PUCAN_TSLOW_TSEG1(pbt->prop_seg + pbt->phase_seg1 - 1);
+ cmd->tseg2 = PUCAN_TSLOW_TSEG2(pbt->phase_seg2 - 1);
+ cmd->brp = cpu_to_le16(PUCAN_TSLOW_BRP(pbt->brp - 1));
+
+ cmd->ewl = 96; /* default */
+
+ netdev_dbg(priv->ndev,
+ "nominal: brp=%u tseg1=%u tseg2=%u sjw=%u\n",
+ le16_to_cpu(cmd->brp), cmd->tseg1, cmd->tseg2, cmd->sjw_t);
+
+ return pucan_write_cmd(priv);
+}
+
+static int pucan_set_timing_fast(struct peak_canfd_priv *priv,
+ const struct can_bittiming *pbt)
+{
+ struct pucan_timing_fast *cmd;
+
+ cmd = pucan_add_cmd(pucan_init_cmd(priv), PUCAN_CMD_TIMING_FAST);
+
+ cmd->sjw = PUCAN_TFAST_SJW(pbt->sjw - 1);
+ cmd->tseg1 = PUCAN_TFAST_TSEG1(pbt->prop_seg + pbt->phase_seg1 - 1);
+ cmd->tseg2 = PUCAN_TFAST_TSEG2(pbt->phase_seg2 - 1);
+ cmd->brp = cpu_to_le16(PUCAN_TFAST_BRP(pbt->brp - 1));
+
+ netdev_dbg(priv->ndev,
+ "data: brp=%u tseg1=%u tseg2=%u sjw=%u\n",
+ le16_to_cpu(cmd->brp), cmd->tseg1, cmd->tseg2, cmd->sjw);
+
+ return pucan_write_cmd(priv);
+}
+
+static int pucan_set_std_filter(struct peak_canfd_priv *priv, u8 row, u32 mask)
+{
+ struct pucan_std_filter *cmd;
+
+ cmd = pucan_add_cmd(pucan_init_cmd(priv), PUCAN_CMD_SET_STD_FILTER);
+
+ /* all the 11-bits CAN ID values are represented by one bit in a
+ * 64 rows array of 32 bits: the upper 6 bits of the CAN ID select the
+ * row while the lowest 5 bits select the bit in that row.
+ *
+ * bit filter
+ * 1 passed
+ * 0 discarded
+ */
+
+ /* select the row */
+ cmd->idx = row;
+
+ /* set/unset bits in the row */
+ cmd->mask = cpu_to_le32(mask);
+
+ return pucan_write_cmd(priv);
+}
+
+static int pucan_tx_abort(struct peak_canfd_priv *priv, u16 flags)
+{
+ struct pucan_tx_abort *cmd;
+
+ cmd = pucan_add_cmd(pucan_init_cmd(priv), PUCAN_CMD_TX_ABORT);
+
+ cmd->flags = cpu_to_le16(flags);
+
+ return pucan_write_cmd(priv);
+}
+
+static int pucan_clr_err_counters(struct peak_canfd_priv *priv)
+{
+ struct pucan_wr_err_cnt *cmd;
+
+ cmd = pucan_add_cmd(pucan_init_cmd(priv), PUCAN_CMD_WR_ERR_CNT);
+
+ cmd->sel_mask = cpu_to_le16(PUCAN_WRERRCNT_TE | PUCAN_WRERRCNT_RE);
+ cmd->tx_counter = 0;
+ cmd->rx_counter = 0;
+
+ return pucan_write_cmd(priv);
+}
+
+static int pucan_set_options(struct peak_canfd_priv *priv, u16 opt_mask)
+{
+ struct pucan_options *cmd;
+
+ cmd = pucan_add_cmd(pucan_init_cmd(priv), PUCAN_CMD_SET_EN_OPTION);
+
+ cmd->options = cpu_to_le16(opt_mask);
+
+ return pucan_write_cmd(priv);
+}
+
+static int pucan_clr_options(struct peak_canfd_priv *priv, u16 opt_mask)
+{
+ struct pucan_options *cmd;
+
+ cmd = pucan_add_cmd(pucan_init_cmd(priv), PUCAN_CMD_CLR_DIS_OPTION);
+
+ cmd->options = cpu_to_le16(opt_mask);
+
+ return pucan_write_cmd(priv);
+}
+
+static int pucan_setup_rx_barrier(struct peak_canfd_priv *priv)
+{
+ pucan_add_cmd(pucan_init_cmd(priv), PUCAN_CMD_RX_BARRIER);
+
+ return pucan_write_cmd(priv);
+}
+
+/* handle the reception of one CAN frame */
+static int pucan_handle_can_rx(struct peak_canfd_priv *priv,
+ struct pucan_rx_msg *msg)
+{
+ struct net_device_stats *stats = &priv->ndev->stats;
+ struct canfd_frame *cf;
+ struct sk_buff *skb;
+ const u16 rx_msg_flags = le16_to_cpu(msg->flags);
+ u8 cf_len;
+
+ if (rx_msg_flags & PUCAN_MSG_EXT_DATA_LEN)
+ cf_len = can_dlc2len(get_canfd_dlc(pucan_msg_get_dlc(msg)));
+ else
+ cf_len = get_can_dlc(pucan_msg_get_dlc(msg));
+
+ /* if this frame is an echo, */
+ if ((rx_msg_flags & PUCAN_MSG_LOOPED_BACK) &&
+ !(rx_msg_flags & PUCAN_MSG_SELF_RECEIVE)) {
+ int n;
+ unsigned long flags;
+
+ spin_lock_irqsave(&priv->echo_lock, flags);
+ n = can_get_echo_skb(priv->ndev, msg->client);
+ spin_unlock_irqrestore(&priv->echo_lock, flags);
+
+ /* count bytes of the echo instead of skb */
+ stats->tx_bytes += cf_len;
+ stats->tx_packets++;
+
+ if (n) {
+ /* restart tx queue only if a slot is free */
+ netif_wake_queue(priv->ndev);
+ }
+
+ return 0;
+ }
+
+ /* otherwise, it should be pushed into rx fifo */
+ if (rx_msg_flags & PUCAN_MSG_EXT_DATA_LEN) {
+ /* CANFD frame case */
+ skb = alloc_canfd_skb(priv->ndev, &cf);
+ if (!skb)
+ return -ENOMEM;
+
+ if (rx_msg_flags & PUCAN_MSG_BITRATE_SWITCH)
+ cf->flags |= CANFD_BRS;
+
+ if (rx_msg_flags & PUCAN_MSG_ERROR_STATE_IND)
+ cf->flags |= CANFD_ESI;
+ } else {
+ /* CAN 2.0 frame case */
+ skb = alloc_can_skb(priv->ndev, (struct can_frame **)&cf);
+ if (!skb)
+ return -ENOMEM;
+ }
+
+ cf->can_id = le32_to_cpu(msg->can_id);
+ cf->len = cf_len;
+
+ if (rx_msg_flags & PUCAN_MSG_EXT_ID)
+ cf->can_id |= CAN_EFF_FLAG;
+
+ if (rx_msg_flags & PUCAN_MSG_RTR)
+ cf->can_id |= CAN_RTR_FLAG;
+ else
+ memcpy(cf->data, msg->d, cf->len);
+
+ stats->rx_bytes += cf->len;
+ stats->rx_packets++;
+
+ netif_rx(skb);
+
+ return 0;
+}
+
+/* handle rx/tx error counters notification */
+static int pucan_handle_error(struct peak_canfd_priv *priv,
+ struct pucan_error_msg *msg)
+{
+ priv->bec.txerr = msg->tx_err_cnt;
+ priv->bec.rxerr = msg->rx_err_cnt;
+
+ return 0;
+}
+
+/* handle status notification */
+static int pucan_handle_status(struct peak_canfd_priv *priv,
+ struct pucan_status_msg *msg)
+{
+ struct net_device *ndev = priv->ndev;
+ struct net_device_stats *stats = &ndev->stats;
+ struct can_frame *cf;
+ struct sk_buff *skb;
+
+ /* this STATUS is the CNF of the RX_BARRIER: Tx path can be setup */
+ if (pucan_status_is_rx_barrier(msg)) {
+ unsigned long flags;
+
+ if (priv->enable_tx_path) {
+ int err = priv->enable_tx_path(priv);
+
+ if (err)
+ return err;
+ }
+
+ /* restart network queue only if echo skb array is free */
+ spin_lock_irqsave(&priv->echo_lock, flags);
+
+ if (!priv->can.echo_skb[priv->echo_idx]) {
+ spin_unlock_irqrestore(&priv->echo_lock, flags);
+
+ netif_wake_queue(ndev);
+ } else {
+ spin_unlock_irqrestore(&priv->echo_lock, flags);
+ }
+
+ return 0;
+ }
+
+ skb = alloc_can_err_skb(ndev, &cf);
+
+ /* test state error bits according to their priority */
+ if (pucan_status_is_busoff(msg)) {
+ netdev_dbg(ndev, "Bus-off entry status\n");
+ priv->can.state = CAN_STATE_BUS_OFF;
+ priv->can.can_stats.bus_off++;
+ can_bus_off(ndev);
+ if (skb)
+ cf->can_id |= CAN_ERR_BUSOFF;
+
+ } else if (pucan_status_is_passive(msg)) {
+ netdev_dbg(ndev, "Error passive status\n");
+ priv->can.state = CAN_STATE_ERROR_PASSIVE;
+ priv->can.can_stats.error_passive++;
+ if (skb) {
+ cf->can_id |= CAN_ERR_CRTL;
+ cf->data[1] = (priv->bec.txerr > priv->bec.rxerr) ?
+ CAN_ERR_CRTL_TX_PASSIVE :
+ CAN_ERR_CRTL_RX_PASSIVE;
+ cf->data[6] = priv->bec.txerr;
+ cf->data[7] = priv->bec.rxerr;
+ }
+
+ } else if (pucan_status_is_warning(msg)) {
+ netdev_dbg(ndev, "Error warning status\n");
+ priv->can.state = CAN_STATE_ERROR_WARNING;
+ priv->can.can_stats.error_warning++;
+ if (skb) {
+ cf->can_id |= CAN_ERR_CRTL;
+ cf->data[1] = (priv->bec.txerr > priv->bec.rxerr) ?
+ CAN_ERR_CRTL_TX_WARNING :
+ CAN_ERR_CRTL_RX_WARNING;
+ cf->data[6] = priv->bec.txerr;
+ cf->data[7] = priv->bec.rxerr;
+ }
+
+ } else if (priv->can.state != CAN_STATE_ERROR_ACTIVE) {
+ /* back to ERROR_ACTIVE */
+ netdev_dbg(ndev, "Error active status\n");
+ can_change_state(ndev, cf, CAN_STATE_ERROR_ACTIVE,
+ CAN_STATE_ERROR_ACTIVE);
+ } else {
+ dev_kfree_skb(skb);
+ return 0;
+ }
+
+ if (!skb) {
+ stats->rx_dropped++;
+ return -ENOMEM;
+ }
+
+ stats->rx_packets++;
+ stats->rx_bytes += cf->can_dlc;
+ netif_rx(skb);
+
+ return 0;
+}
+
+/* handle uCAN Rx overflow notification */
+static int pucan_handle_cache_critical(struct peak_canfd_priv *priv)
+{
+ struct net_device_stats *stats = &priv->ndev->stats;
+ struct can_frame *cf;
+ struct sk_buff *skb;
+
+ stats->rx_over_errors++;
+ stats->rx_errors++;
+
+ skb = alloc_can_err_skb(priv->ndev, &cf);
+ if (!skb) {
+ stats->rx_dropped++;
+ return -ENOMEM;
+ }
+
+ cf->can_id |= CAN_ERR_CRTL;
+ cf->data[1] = CAN_ERR_CRTL_RX_OVERFLOW;
+
+ cf->data[6] = priv->bec.txerr;
+ cf->data[7] = priv->bec.rxerr;
+
+ stats->rx_bytes += cf->can_dlc;
+ stats->rx_packets++;
+ netif_rx(skb);
+
+ return 0;
+}
+
+/* handle a single uCAN message */
+int peak_canfd_handle_msg(struct peak_canfd_priv *priv,
+ struct pucan_rx_msg *msg)
+{
+ u16 msg_type = le16_to_cpu(msg->type);
+ int msg_size = le16_to_cpu(msg->size);
+ int err;
+
+ if (!msg_size || !msg_type) {
+ /* null packet found: end of list */
+ goto exit;
+ }
+
+ switch (msg_type) {
+ case PUCAN_MSG_CAN_RX:
+ err = pucan_handle_can_rx(priv, (struct pucan_rx_msg *)msg);
+ break;
+ case PUCAN_MSG_ERROR:
+ err = pucan_handle_error(priv, (struct pucan_error_msg *)msg);
+ break;
+ case PUCAN_MSG_STATUS:
+ err = pucan_handle_status(priv, (struct pucan_status_msg *)msg);
+ break;
+ case PUCAN_MSG_CACHE_CRITICAL:
+ err = pucan_handle_cache_critical(priv);
+ break;
+ default:
+ err = 0;
+ }
+
+ if (err < 0)
+ return err;
+
+exit:
+ return msg_size;
+}
+
+/* handle a list of rx_count messages from rx_msg memory address */
+int peak_canfd_handle_msgs_list(struct peak_canfd_priv *priv,
+ struct pucan_rx_msg *msg_list, int msg_count)
+{
+ void *msg_ptr = msg_list;
+ int i, msg_size;
+
+ for (i = 0; i < msg_count; i++) {
+ msg_size = peak_canfd_handle_msg(priv, msg_ptr);
+
+ /* a null packet can be found at the end of a list */
+ if (msg_size <= 0)
+ break;
+
+ msg_ptr += msg_size;
+ }
+
+ if (msg_size < 0)
+ return msg_size;
+
+ return i;
+}
+
+static int peak_canfd_start(struct peak_canfd_priv *priv)
+{
+ int err;
+
+ err = pucan_clr_err_counters(priv);
+ if (err)
+ goto err_exit;
+
+ priv->echo_idx = 0;
+
+ priv->bec.txerr = 0;
+ priv->bec.rxerr = 0;
+
+ if (priv->can.ctrlmode & CAN_CTRLMODE_LISTENONLY)
+ err = pucan_set_listen_only_mode(priv);
+ else
+ err = pucan_set_normal_mode(priv);
+
+err_exit:
+ return err;
+}
+
+static void peak_canfd_stop(struct peak_canfd_priv *priv)
+{
+ int err;
+
+ /* go back to RESET mode */
+ err = pucan_set_reset_mode(priv);
+ if (err) {
+ netdev_err(priv->ndev, "channel %u reset failed\n",
+ priv->index);
+ } else {
+ /* abort last Tx (MUST be done in RESET mode only!) */
+ pucan_tx_abort(priv, PUCAN_TX_ABORT_FLUSH);
+ }
+}
+
+static int peak_canfd_set_mode(struct net_device *ndev, enum can_mode mode)
+{
+ struct peak_canfd_priv *priv = netdev_priv(ndev);
+
+ switch (mode) {
+ case CAN_MODE_START:
+ peak_canfd_start(priv);
+ netif_wake_queue(ndev);
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
+static int peak_canfd_get_berr_counter(const struct net_device *ndev,
+ struct can_berr_counter *bec)
+{
+ struct peak_canfd_priv *priv = netdev_priv(ndev);
+
+ *bec = priv->bec;
+ return 0;
+}
+
+static int peak_canfd_open(struct net_device *ndev)
+{
+ struct peak_canfd_priv *priv = netdev_priv(ndev);
+ int i, err = 0;
+
+ err = open_candev(ndev);
+ if (err) {
+ netdev_err(ndev, "open_candev() failed, error %d\n", err);
+ goto err_exit;
+ }
+
+ err = pucan_set_reset_mode(priv);
+ if (err)
+ goto err_close;
+
+ if (priv->can.ctrlmode & CAN_CTRLMODE_FD) {
+ if (priv->can.ctrlmode & CAN_CTRLMODE_FD_NON_ISO)
+ err = pucan_clr_options(priv, PUCAN_OPTION_CANDFDISO);
+ else
+ err = pucan_set_options(priv, PUCAN_OPTION_CANDFDISO);
+
+ if (err)
+ goto err_close;
+ }
+
+ /* set option: get rx/tx error counters */
+ err = pucan_set_options(priv, PUCAN_OPTION_ERROR);
+ if (err)
+ goto err_close;
+
+ /* accept all standard CAN ID */
+ for (i = 0; i <= PUCAN_FLTSTD_ROW_IDX_MAX; i++)
+ pucan_set_std_filter(priv, i, 0xffffffff);
+
+ err = peak_canfd_start(priv);
+ if (err)
+ goto err_close;
+
+ /* receiving the RB status says when Tx path is ready */
+ err = pucan_setup_rx_barrier(priv);
+ if (!err)
+ goto err_exit;
+
+err_close:
+ close_candev(ndev);
+err_exit:
+ return err;
+}
+
+static int peak_canfd_set_bittiming(struct net_device *ndev)
+{
+ struct peak_canfd_priv *priv = netdev_priv(ndev);
+
+ return pucan_set_timing_slow(priv, &priv->can.bittiming);
+}
+
+static int peak_canfd_set_data_bittiming(struct net_device *ndev)
+{
+ struct peak_canfd_priv *priv = netdev_priv(ndev);
+
+ return pucan_set_timing_fast(priv, &priv->can.data_bittiming);
+}
+
+static int peak_canfd_close(struct net_device *ndev)
+{
+ struct peak_canfd_priv *priv = netdev_priv(ndev);
+
+ netif_stop_queue(ndev);
+ peak_canfd_stop(priv);
+ close_candev(ndev);
+
+ return 0;
+}
+
+static netdev_tx_t peak_canfd_start_xmit(struct sk_buff *skb,
+ struct net_device *ndev)
+{
+ struct peak_canfd_priv *priv = netdev_priv(ndev);
+ struct net_device_stats *stats = &ndev->stats;
+ struct canfd_frame *cf = (struct canfd_frame *)skb->data;
+ struct pucan_tx_msg *msg;
+ u16 msg_size, msg_flags;
+ unsigned long flags;
+ bool should_stop_tx_queue;
+ int room_left;
+ u8 can_dlc;
+
+ if (can_dropped_invalid_skb(ndev, skb))
+ return NETDEV_TX_OK;
+
+ msg_size = ALIGN(sizeof(*msg) + cf->len, 4);
+ msg = priv->alloc_tx_msg(priv, msg_size, &room_left);
+
+ /* should never happen except under bus-off condition and (auto-)restart
+ * mechanism
+ */
+ if (!msg) {
+ stats->tx_dropped++;
+ netif_stop_queue(ndev);
+ return NETDEV_TX_BUSY;
+ }
+
+ msg->size = cpu_to_le16(msg_size);
+ msg->type = cpu_to_le16(PUCAN_MSG_CAN_TX);
+ msg_flags = 0;
+
+ if (cf->can_id & CAN_EFF_FLAG) {
+ msg_flags |= PUCAN_MSG_EXT_ID;
+ msg->can_id = cpu_to_le32(cf->can_id & CAN_EFF_MASK);
+ } else {
+ msg->can_id = cpu_to_le32(cf->can_id & CAN_SFF_MASK);
+ }
+
+ if (can_is_canfd_skb(skb)) {
+ /* CAN FD frame format */
+ can_dlc = can_len2dlc(cf->len);
+
+ msg_flags |= PUCAN_MSG_EXT_DATA_LEN;
+
+ if (cf->flags & CANFD_BRS)
+ msg_flags |= PUCAN_MSG_BITRATE_SWITCH;
+
+ if (cf->flags & CANFD_ESI)
+ msg_flags |= PUCAN_MSG_ERROR_STATE_IND;
+ } else {
+ /* CAN 2.0 frame format */
+ can_dlc = cf->len;
+
+ if (cf->can_id & CAN_RTR_FLAG)
+ msg_flags |= PUCAN_MSG_RTR;
+ }
+
+ /* always ask loopback for echo management */
+ msg_flags |= PUCAN_MSG_LOOPED_BACK;
+
+ /* set driver specific bit to differentiate with application loopback */
+ if (priv->can.ctrlmode & CAN_CTRLMODE_LOOPBACK)
+ msg_flags |= PUCAN_MSG_SELF_RECEIVE;
+
+ msg->flags = cpu_to_le16(msg_flags);
+ msg->channel_dlc = PUCAN_MSG_CHANNEL_DLC(priv->index, can_dlc);
+ memcpy(msg->d, cf->data, cf->len);
+
+ /* struct msg client field is used as an index in the echo skbs ring */
+ msg->client = priv->echo_idx;
+
+ spin_lock_irqsave(&priv->echo_lock, flags);
+
+ /* prepare and save echo skb in internal slot */
+ can_put_echo_skb(skb, ndev, priv->echo_idx);
+
+ /* move echo index to the next slot */
+ priv->echo_idx = (priv->echo_idx + 1) % priv->can.echo_skb_max;
+
+ /* if next slot is not free, stop network queue (no slot free in echo
+ * skb ring means that the controller did not write these frames on
+ * the bus: no need to continue).
+ */
+ should_stop_tx_queue = !!(priv->can.echo_skb[priv->echo_idx]);
+
+ spin_unlock_irqrestore(&priv->echo_lock, flags);
+
+ /* write the skb on the interface */
+ priv->write_tx_msg(priv, msg);
+
+ /* stop network tx queue if not enough room to save one more msg too */
+ if (priv->can.ctrlmode & CAN_CTRLMODE_FD)
+ should_stop_tx_queue |= (room_left <
+ (sizeof(*msg) + CANFD_MAX_DLEN));
+ else
+ should_stop_tx_queue |= (room_left <
+ (sizeof(*msg) + CAN_MAX_DLEN));
+
+ if (should_stop_tx_queue)
+ netif_stop_queue(ndev);
+
+ return NETDEV_TX_OK;
+}
+
+static const struct net_device_ops peak_canfd_netdev_ops = {
+ .ndo_open = peak_canfd_open,
+ .ndo_stop = peak_canfd_close,
+ .ndo_start_xmit = peak_canfd_start_xmit,
+ .ndo_change_mtu = can_change_mtu,
+};
+
+struct net_device *alloc_peak_canfd_dev(int sizeof_priv, int index,
+ int echo_skb_max)
+{
+ struct net_device *ndev;
+ struct peak_canfd_priv *priv;
+
+ /* we DO support local echo */
+ if (echo_skb_max < 0)
+ echo_skb_max = PCANFD_ECHO_SKB_MAX;
+
+ /* allocate the candev object */
+ ndev = alloc_candev(sizeof_priv, echo_skb_max);
+ if (!ndev)
+ return NULL;
+
+ priv = netdev_priv(ndev);
+
+ /* complete now socket-can initialization side */
+ priv->can.state = CAN_STATE_STOPPED;
+ priv->can.bittiming_const = &peak_canfd_nominal_const;
+ priv->can.data_bittiming_const = &peak_canfd_data_const;
+
+ priv->can.do_set_mode = peak_canfd_set_mode;
+ priv->can.do_get_berr_counter = peak_canfd_get_berr_counter;
+ priv->can.do_set_bittiming = peak_canfd_set_bittiming;
+ priv->can.do_set_data_bittiming = peak_canfd_set_data_bittiming;
+ priv->can.ctrlmode_supported = CAN_CTRLMODE_LOOPBACK |
+ CAN_CTRLMODE_LISTENONLY |
+ CAN_CTRLMODE_3_SAMPLES |
+ CAN_CTRLMODE_FD |
+ CAN_CTRLMODE_FD_NON_ISO |
+ CAN_CTRLMODE_BERR_REPORTING;
+
+ priv->ndev = ndev;
+ priv->index = index;
+ priv->cmd_len = 0;
+ spin_lock_init(&priv->echo_lock);
+
+ ndev->flags |= IFF_ECHO;
+ ndev->netdev_ops = &peak_canfd_netdev_ops;
+ ndev->dev_id = index;
+
+ return ndev;
+}
diff --git a/drivers/net/can/peak_canfd/peak_canfd_user.h b/drivers/net/can/peak_canfd/peak_canfd_user.h
new file mode 100644
index 0000000..bf6de47
--- /dev/null
+++ b/drivers/net/can/peak_canfd/peak_canfd_user.h
@@ -0,0 +1,55 @@
+/*
+ * CAN driver for PEAK System micro-CAN based adapters
+ *
+ * Copyright (C) 2003-2011 PEAK System-Technik GmbH
+ * Copyright (C) 2011-2013 Stephane Grosjean <s.grosjean@peak-system.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published
+ * by the Free Software Foundation; version 2 of the License.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#ifndef PEAK_CANFD_USER_H
+#define PEAK_CANFD_USER_H
+
+#include <linux/can/dev/peak_canfd.h>
+
+#define PCANFD_ECHO_SKB_DEF -1
+
+/* data structure private to each uCAN interface */
+struct peak_canfd_priv {
+ struct can_priv can; /* socket-can private data */
+ struct net_device *ndev; /* network device */
+ int index; /* channel index */
+
+ struct can_berr_counter bec; /* rx/tx err counters */
+
+ int echo_idx; /* echo skb free slot index */
+ spinlock_t echo_lock;
+
+ int cmd_len;
+ void *cmd_buffer;
+ int cmd_maxlen;
+
+ int (*pre_cmd)(struct peak_canfd_priv *priv);
+ int (*write_cmd)(struct peak_canfd_priv *priv);
+ int (*post_cmd)(struct peak_canfd_priv *priv);
+
+ int (*enable_tx_path)(struct peak_canfd_priv *priv);
+ void *(*alloc_tx_msg)(struct peak_canfd_priv *priv, u16 msg_size,
+ int *room_left);
+ int (*write_tx_msg)(struct peak_canfd_priv *priv,
+ struct pucan_tx_msg *msg);
+};
+
+struct net_device *alloc_peak_canfd_dev(int sizeof_priv, int index,
+ int echo_skb_max);
+int peak_canfd_handle_msg(struct peak_canfd_priv *priv,
+ struct pucan_rx_msg *msg);
+int peak_canfd_handle_msgs_list(struct peak_canfd_priv *priv,
+ struct pucan_rx_msg *rx_msg, int rx_count);
+#endif
diff --git a/drivers/net/can/peak_canfd/peak_pciefd_main.c b/drivers/net/can/peak_canfd/peak_pciefd_main.c
new file mode 100644
index 0000000..51c2d18
--- /dev/null
+++ b/drivers/net/can/peak_canfd/peak_pciefd_main.c
@@ -0,0 +1,842 @@
+/*
+ * Copyright (C) 2007, 2011 Wolfgang Grandegger <wg@grandegger.com>
+ * Copyright (C) 2012 Stephane Grosjean <s.grosjean@peak-system.com>
+ *
+ * Derived from the PCAN project file driver/src/pcan_pci.c:
+ *
+ * Copyright (C) 2001-2006 PEAK System-Technik GmbH
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the version 2 of the GNU General Public License
+ * as published by the Free Software Foundation
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/interrupt.h>
+#include <linux/netdevice.h>
+#include <linux/delay.h>
+#include <linux/pci.h>
+#include <linux/io.h>
+#include <linux/can.h>
+#include <linux/can/dev.h>
+
+#include "peak_canfd_user.h"
+
+MODULE_AUTHOR("Stephane Grosjean <s.grosjean@peak-system.com>");
+MODULE_DESCRIPTION("Socket-CAN driver for PEAK PCAN PCIe FD family cards");
+MODULE_SUPPORTED_DEVICE("PEAK PCAN PCIe FD CAN cards");
+MODULE_LICENSE("GPL v2");
+
+#define PCIEFD_DRV_NAME "peak_pciefd"
+
+#define PEAK_PCI_VENDOR_ID 0x001c /* The PCI device and vendor IDs */
+#define PEAK_PCIEFD_ID 0x0013 /* for PCIe slot cards */
+
+/* PEAK PCIe board access description */
+#define PCIEFD_BAR0_SIZE (64 * 1024)
+#define PCIEFD_RX_DMA_SIZE (4 * 1024)
+#define PCIEFD_TX_DMA_SIZE (4 * 1024)
+
+#define PCIEFD_TX_PAGE_SIZE (2 * 1024)
+
+/* System Control Registers */
+#define PCIEFD_REG_SYS_CTL_SET 0x0000 /* set bits */
+#define PCIEFD_REG_SYS_CTL_CLR 0x0004 /* clear bits */
+
+/* Version info registers */
+#define PCIEFD_REG_SYS_VER1 0x0040 /* version reg #1 */
+#define PCIEFD_REG_SYS_VER2 0x0044 /* version reg #2 */
+
+/* System Control Registers Bits */
+#define PCIEFD_SYS_CTL_TS_RST 0x00000001 /* timestamp clock */
+#define PCIEFD_SYS_CTL_CLK_EN 0x00000002 /* system clock */
+
+/* CAN-FD channel addresses */
+#define PCIEFD_CANX_OFF(c) (((c) + 1) * 0x1000)
+
+#define PCIEFD_ECHO_SKB_MAX PCANFD_ECHO_SKB_DEF
+
+/* CAN-FD channel registers */
+#define PCIEFD_REG_CAN_MISC 0x0000 /* Misc. control */
+#define PCIEFD_REG_CAN_CLK_SEL 0x0008 /* Clock selector */
+#define PCIEFD_REG_CAN_CMD_PORT_L 0x0010 /* 64-bits command port */
+#define PCIEFD_REG_CAN_CMD_PORT_H 0x0014
+#define PCIEFD_REG_CAN_TX_REQ_ACC 0x0020 /* Tx request accumulator */
+#define PCIEFD_REG_CAN_TX_CTL_SET 0x0030 /* Tx control set register */
+#define PCIEFD_REG_CAN_TX_CTL_CLR 0x0038 /* Tx control clear register */
+#define PCIEFD_REG_CAN_TX_DMA_ADDR_L 0x0040 /* 64-bits addr for Tx DMA */
+#define PCIEFD_REG_CAN_TX_DMA_ADDR_H 0x0044
+#define PCIEFD_REG_CAN_RX_CTL_SET 0x0050 /* Rx control set register */
+#define PCIEFD_REG_CAN_RX_CTL_CLR 0x0058 /* Rx control clear register */
+#define PCIEFD_REG_CAN_RX_CTL_WRT 0x0060 /* Rx control write register */
+#define PCIEFD_REG_CAN_RX_CTL_ACK 0x0068 /* Rx control ACK register */
+#define PCIEFD_REG_CAN_RX_DMA_ADDR_L 0x0070 /* 64-bits addr for Rx DMA */
+#define PCIEFD_REG_CAN_RX_DMA_ADDR_H 0x0074
+
+/* CAN-FD channel misc register bits */
+#define CANFD_MISC_TS_RST 0x00000001 /* timestamp cnt rst */
+
+/* CAN-FD channel Clock SELector Source & DIVider */
+#define CANFD_CLK_SEL_DIV_MASK 0x00000007
+#define CANFD_CLK_SEL_DIV_60MHZ 0x00000000 /* SRC=240MHz only */
+#define CANFD_CLK_SEL_DIV_40MHZ 0x00000001 /* SRC=240MHz only */
+#define CANFD_CLK_SEL_DIV_30MHZ 0x00000002 /* SRC=240MHz only */
+#define CANFD_CLK_SEL_DIV_24MHZ 0x00000003 /* SRC=240MHz only */
+#define CANFD_CLK_SEL_DIV_20MHZ 0x00000004 /* SRC=240MHz only */
+
+#define CANFD_CLK_SEL_SRC_MASK 0x00000008 /* 0=80MHz, 1=240MHz */
+#define CANFD_CLK_SEL_SRC_240MHZ 0x00000008
+#define CANFD_CLK_SEL_SRC_80MHZ (~CANFD_CLK_SEL_SRC_240MHZ & \
+ CANFD_CLK_SEL_SRC_MASK)
+
+#define CANFD_CLK_SEL_20MHZ (CANFD_CLK_SEL_SRC_240MHZ |\
+ CANFD_CLK_SEL_DIV_20MHZ)
+#define CANFD_CLK_SEL_24MHZ (CANFD_CLK_SEL_SRC_240MHZ |\
+ CANFD_CLK_SEL_DIV_24MHZ)
+#define CANFD_CLK_SEL_30MHZ (CANFD_CLK_SEL_SRC_240MHZ |\
+ CANFD_CLK_SEL_DIV_30MHZ)
+#define CANFD_CLK_SEL_40MHZ (CANFD_CLK_SEL_SRC_240MHZ |\
+ CANFD_CLK_SEL_DIV_40MHZ)
+#define CANFD_CLK_SEL_60MHZ (CANFD_CLK_SEL_SRC_240MHZ |\
+ CANFD_CLK_SEL_DIV_60MHZ)
+#define CANFD_CLK_SEL_80MHZ (CANFD_CLK_SEL_SRC_80MHZ)
+
+/* CAN-FD channel Rx/Tx control register bits */
+#define CANFD_CTL_UNC_BIT 0x00010000 /* Uncached DMA mem */
+#define CANFD_CTL_RST_BIT 0x00020000 /* reset DMA action */
+#define CANFD_CTL_IEN_BIT 0x00040000 /* IRQ enable */
+
+/* Rx IRQ Count and Time Limits */
+#define CANFD_CTL_IRQ_CL_DEF 16 /* Rx msg max nb per IRQ in Rx DMA */
+#define CANFD_CTL_IRQ_TL_DEF 10 /* Time before IRQ if < CL (x100 µs) */
+
+#define CANFD_OPTIONS_SET (CANFD_OPTION_ERROR | CANFD_OPTION_BUSLOAD)
+
+/* Tx anticipation window (link logical address should be aligned on 2K
+ * boundary)
+ */
+#define PCIEFD_TX_PAGE_COUNT (PCIEFD_TX_DMA_SIZE / PCIEFD_TX_PAGE_SIZE)
+
+#define CANFD_MSG_LNK_TX 0x1001 /* Tx msgs link */
+
+/* 32-bits IRQ status fields, heading Rx DMA area */
+static inline int pciefd_irq_tag(u32 irq_status)
+{
+ return irq_status & 0x0000000f;
+}
+
+static inline int pciefd_irq_rx_cnt(u32 irq_status)
+{
+ return (irq_status & 0x000007f0) >> 4;
+}
+
+static inline int pciefd_irq_is_lnk(u32 irq_status)
+{
+ return irq_status & 0x00010000;
+}
+
+/* Rx record */
+struct pciefd_rx_dma {
+ __le32 irq_status;
+ __le32 sys_time_low;
+ __le32 sys_time_high;
+ struct pucan_rx_msg msg[0];
+} __packed __aligned(4);
+
+/* Tx Link record */
+struct pciefd_tx_link {
+ __le16 size;
+ __le16 type;
+ __le32 laddr_lo;
+ __le32 laddr_hi;
+} __packed __aligned(4);
+
+/* Tx page descriptor */
+struct pciefd_page {
+ void *vbase; /* page virtual address */
+ dma_addr_t lbase; /* page logical address */
+ u32 offset;
+ u32 size;
+};
+
+#define CANFD_IRQ_SET 0x00000001
+#define CANFD_TX_PATH_SET 0x00000002
+
+/* CAN-FD channel object */
+struct pciefd_board;
+struct pciefd_can {
+ struct peak_canfd_priv ucan; /* must be the first member */
+ void __iomem *reg_base; /* channel config base addr */
+ struct pciefd_board *board; /* reverse link */
+
+ struct pucan_command pucan_cmd; /* command buffer */
+
+ dma_addr_t rx_dma_laddr; /* DMA virtual and logical addr */
+ void *rx_dma_vaddr; /* for Rx and Tx areas */
+ dma_addr_t tx_dma_laddr;
+ void *tx_dma_vaddr;
+
+ struct pciefd_page tx_pages[PCIEFD_TX_PAGE_COUNT];
+ u16 tx_pages_free; /* free Tx pages counter */
+ u16 tx_page_index; /* current page used for Tx */
+ spinlock_t tx_lock;
+
+ u32 irq_status;
+ u32 irq_tag; /* next irq tag */
+};
+
+/* PEAK-PCIe FD board object */
+struct pciefd_board {
+ void __iomem *reg_base;
+ struct pci_dev *pci_dev;
+ int can_count;
+ spinlock_t cmd_lock; /* 64-bits cmds must be atomic */
+ struct pciefd_can *can[0]; /* array of network devices */
+};
+
+/* supported device ids. */
+static const struct pci_device_id peak_pciefd_tbl[] = {
+ {PEAK_PCI_VENDOR_ID, PEAK_PCIEFD_ID, PCI_ANY_ID, PCI_ANY_ID,},
+ {0,}
+};
+
+MODULE_DEVICE_TABLE(pci, peak_pciefd_tbl);
+
+/* read a 32 bits value from a SYS block register */
+static inline u32 pciefd_sys_readreg(const struct pciefd_board *priv, u16 reg)
+{
+ return readl(priv->reg_base + reg);
+}
+
+/* write a 32 bits value into a SYS block register */
+static inline void pciefd_sys_writereg(const struct pciefd_board *priv,
+ u32 val, u16 reg)
+{
+ writel(val, priv->reg_base + reg);
+}
+
+/* read a 32 bits value from CAN-FD block register */
+static inline u32 pciefd_can_readreg(const struct pciefd_can *priv, u16 reg)
+{
+ return readl(priv->reg_base + reg);
+}
+
+/* write a 32 bits value into a CAN-FD block register */
+static inline void pciefd_can_writereg(const struct pciefd_can *priv,
+ u32 val, u16 reg)
+{
+ writel(val, priv->reg_base + reg);
+}
+
+/* give a channel logical Rx DMA address to the board */
+static void pciefd_can_setup_rx_dma(struct pciefd_can *priv)
+{
+#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
+ const u32 dma_addr_h = (u32)(priv->rx_dma_laddr >> 32);
+#else
+ const u32 dma_addr_h = 0;
+#endif
+
+ /* (DMA must be reset for Rx) */
+ pciefd_can_writereg(priv, CANFD_CTL_RST_BIT, PCIEFD_REG_CAN_RX_CTL_SET);
+
+ /* write the logical address of the Rx DMA area for this channel */
+ pciefd_can_writereg(priv, (u32)priv->rx_dma_laddr,
+ PCIEFD_REG_CAN_RX_DMA_ADDR_L);
+ pciefd_can_writereg(priv, dma_addr_h, PCIEFD_REG_CAN_RX_DMA_ADDR_H);
+
+ /* also indicates that Rx DMA is cacheable */
+ pciefd_can_writereg(priv, CANFD_CTL_UNC_BIT, PCIEFD_REG_CAN_RX_CTL_CLR);
+}
+
+/* clear channel logical Rx DMA address from the board */
+static void pciefd_can_clear_rx_dma(struct pciefd_can *priv)
+{
+ /* DMA must be reset for Rx */
+ pciefd_can_writereg(priv, CANFD_CTL_RST_BIT, PCIEFD_REG_CAN_RX_CTL_SET);
+
+ /* clear the logical address of the Rx DMA area for this channel */
+ pciefd_can_writereg(priv, 0, PCIEFD_REG_CAN_RX_DMA_ADDR_L);
+ pciefd_can_writereg(priv, 0, PCIEFD_REG_CAN_RX_DMA_ADDR_H);
+}
+
+/* give a channel logical Tx DMA address to the board */
+static void pciefd_can_setup_tx_dma(struct pciefd_can *priv)
+{
+#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
+ const u32 dma_addr_h = (u32)(priv->tx_dma_laddr >> 32);
+#else
+ const u32 dma_addr_h = 0;
+#endif
+
+ /* (DMA must be reset for Tx) */
+ pciefd_can_writereg(priv, CANFD_CTL_RST_BIT, PCIEFD_REG_CAN_TX_CTL_SET);
+
+ /* write the logical address of the Tx DMA area for this channel */
+ pciefd_can_writereg(priv, (u32)priv->tx_dma_laddr,
+ PCIEFD_REG_CAN_TX_DMA_ADDR_L);
+ pciefd_can_writereg(priv, dma_addr_h, PCIEFD_REG_CAN_TX_DMA_ADDR_H);
+
+ /* also indicates that Tx DMA is cacheable */
+ pciefd_can_writereg(priv, CANFD_CTL_UNC_BIT, PCIEFD_REG_CAN_TX_CTL_CLR);
+}
+
+/* clear channel logical Tx DMA address from the board */
+static void pciefd_can_clear_tx_dma(struct pciefd_can *priv)
+{
+ /* DMA must be reset for Tx */
+ pciefd_can_writereg(priv, CANFD_CTL_RST_BIT, PCIEFD_REG_CAN_TX_CTL_SET);
+
+ /* clear the logical address of the Tx DMA area for this channel */
+ pciefd_can_writereg(priv, 0, PCIEFD_REG_CAN_TX_DMA_ADDR_L);
+ pciefd_can_writereg(priv, 0, PCIEFD_REG_CAN_TX_DMA_ADDR_H);
+}
+
+static void pciefd_can_ack_rx_dma(struct pciefd_can *priv)
+{
+ /* read value of current IRQ tag and inc it for next one */
+ priv->irq_tag = le32_to_cpu(*(__le32 *)priv->rx_dma_vaddr);
+ priv->irq_tag++;
+ priv->irq_tag &= 0xf;
+
+ /* write the next IRQ tag for this CAN */
+ pciefd_can_writereg(priv, priv->irq_tag, PCIEFD_REG_CAN_RX_CTL_ACK);
+}
+
+/* IRQ handler */
+static irqreturn_t pciefd_irq_handler(int irq, void *arg)
+{
+ struct pciefd_can *priv = arg;
+ struct pciefd_rx_dma *rx_dma = priv->rx_dma_vaddr;
+
+ /* INTA mode only to sync with PCIe transaction */
+ if (!pci_dev_msi_enabled(priv->board->pci_dev))
+ (void)pciefd_sys_readreg(priv->board, PCIEFD_REG_SYS_VER1);
+
+ /* read IRQ status from the first 32-bits of the Rx DMA area */
+ priv->irq_status = le32_to_cpu(rx_dma->irq_status);
+
+ /* check if this (shared) IRQ is for this CAN */
+ if (pciefd_irq_tag(priv->irq_status) != priv->irq_tag)
+ return IRQ_NONE;
+
+ /* handle rx messages (if any) */
+ peak_canfd_handle_msgs_list(&priv->ucan,
+ rx_dma->msg,
+ pciefd_irq_rx_cnt(priv->irq_status));
+
+ /* handle tx link interrupt (if any) */
+ if (pciefd_irq_is_lnk(priv->irq_status)) {
+ unsigned long flags;
+
+ spin_lock_irqsave(&priv->tx_lock, flags);
+ priv->tx_pages_free++;
+ spin_unlock_irqrestore(&priv->tx_lock, flags);
+
+ /* wake producer up */
+ netif_wake_queue(priv->ucan.ndev);
+ }
+
+ /* re-enable Rx DMA transfer for this CAN */
+ pciefd_can_ack_rx_dma(priv);
+
+ return IRQ_HANDLED;
+}
+
+static int pciefd_enable_tx_path(struct peak_canfd_priv *ucan)
+{
+ struct pciefd_can *priv = (struct pciefd_can *)ucan;
+ int i;
+
+ /* initialize the Tx pages descriptors */
+ priv->tx_pages_free = PCIEFD_TX_PAGE_COUNT - 1;
+ priv->tx_page_index = 0;
+
+ priv->tx_pages[0].vbase = priv->tx_dma_vaddr;
+ priv->tx_pages[0].lbase = priv->tx_dma_laddr;
+
+ for (i = 0; i < PCIEFD_TX_PAGE_COUNT; i++) {
+ priv->tx_pages[i].offset = 0;
+ priv->tx_pages[i].size = PCIEFD_TX_PAGE_SIZE -
+ sizeof(struct pciefd_tx_link);
+ if (i) {
+ priv->tx_pages[i].vbase =
+ priv->tx_pages[i - 1].vbase +
+ PCIEFD_TX_PAGE_SIZE;
+ priv->tx_pages[i].lbase =
+ priv->tx_pages[i - 1].lbase +
+ PCIEFD_TX_PAGE_SIZE;
+ }
+ }
+
+ /* setup Tx DMA addresses into IP core */
+ pciefd_can_setup_tx_dma(priv);
+
+ /* start (TX_RST=0) Tx Path */
+ pciefd_can_writereg(priv, CANFD_CTL_RST_BIT, PCIEFD_REG_CAN_TX_CTL_CLR);
+
+ return 0;
+}
+
+/* board specific CANFD command pre-processing */
+static int pciefd_pre_cmd(struct peak_canfd_priv *ucan)
+{
+ struct pciefd_can *priv = (struct pciefd_can *)ucan;
+ u16 cmd = pucan_cmd_get_opcode(&priv->pucan_cmd);
+ int err;
+
+ /* pre-process command */
+ switch (cmd) {
+ case PUCAN_CMD_NORMAL_MODE:
+ case PUCAN_CMD_LISTEN_ONLY_MODE:
+
+ if (ucan->can.state == CAN_STATE_BUS_OFF)
+ break;
+
+ /* going into operational mode: setup IRQ handler */
+ err = request_irq(priv->board->pci_dev->irq,
+ pciefd_irq_handler,
+ IRQF_SHARED,
+ PCIEFD_DRV_NAME,
+ priv);
+ if (err)
+ return err;
+
+ /* setup Rx DMA address */
+ pciefd_can_setup_rx_dma(priv);
+
+ /* setup max count of msgs per IRQ */
+ pciefd_can_writereg(priv, (CANFD_CTL_IRQ_TL_DEF) << 8 |
+ CANFD_CTL_IRQ_CL_DEF,
+ PCIEFD_REG_CAN_RX_CTL_WRT);
+
+ /* clear DMA RST for Rx (Rx start) */
+ pciefd_can_writereg(priv, CANFD_CTL_RST_BIT,
+ PCIEFD_REG_CAN_RX_CTL_CLR);
+
+ /* reset timestamps */
+ pciefd_can_writereg(priv, !CANFD_MISC_TS_RST,
+ PCIEFD_REG_CAN_MISC);
+
+ /* do an initial ACK */
+ pciefd_can_ack_rx_dma(priv);
+
+ /* enable IRQ for this CAN after having set next irq_tag */
+ pciefd_can_writereg(priv, CANFD_CTL_IEN_BIT,
+ PCIEFD_REG_CAN_RX_CTL_SET);
+
+ /* Tx path will be setup as soon as RX_BARRIER is received */
+ break;
+ default:
+ break;
+ }
+
+ return 0;
+}
+
+/* write a command */
+static int pciefd_write_cmd(struct peak_canfd_priv *ucan)
+{
+ struct pciefd_can *priv = (struct pciefd_can *)ucan;
+ unsigned long flags;
+
+ /* 64-bits command is atomic */
+ spin_lock_irqsave(&priv->board->cmd_lock, flags);
+
+ pciefd_can_writereg(priv, *(u32 *)ucan->cmd_buffer,
+ PCIEFD_REG_CAN_CMD_PORT_L);
+ pciefd_can_writereg(priv, *(u32 *)(ucan->cmd_buffer + 4),
+ PCIEFD_REG_CAN_CMD_PORT_H);
+
+ spin_unlock_irqrestore(&priv->board->cmd_lock, flags);
+
+ return 0;
+}
+
+/* board specific CANFD command post-processing */
+static int pciefd_post_cmd(struct peak_canfd_priv *ucan)
+{
+ struct pciefd_can *priv = (struct pciefd_can *)ucan;
+ u16 cmd = pucan_cmd_get_opcode(&priv->pucan_cmd);
+
+ switch (cmd) {
+ case PUCAN_CMD_RESET_MODE:
+
+ if (ucan->can.state == CAN_STATE_STOPPED)
+ break;
+
+ /* controller now in reset mode: */
+
+ /* stop and reset DMA addresses in Tx/Rx engines */
+ pciefd_can_clear_tx_dma(priv);
+ pciefd_can_clear_rx_dma(priv);
+
+ /* disable IRQ for this CAN */
+ pciefd_can_writereg(priv, CANFD_CTL_IEN_BIT,
+ PCIEFD_REG_CAN_RX_CTL_CLR);
+
+ free_irq(priv->board->pci_dev->irq, priv);
+
+ ucan->can.state = CAN_STATE_STOPPED;
+
+ break;
+ }
+
+ return 0;
+}
+
+static void *pciefd_alloc_tx_msg(struct peak_canfd_priv *ucan, u16 msg_size,
+ int *room_left)
+{
+ struct pciefd_can *priv = (struct pciefd_can *)ucan;
+ struct pciefd_page *page = priv->tx_pages + priv->tx_page_index;
+ unsigned long flags;
+ void *msg;
+
+ spin_lock_irqsave(&priv->tx_lock, flags);
+
+ if (page->offset + msg_size > page->size) {
+ struct pciefd_tx_link *lk;
+
+ /* not enough space in this page: try another one */
+ if (!priv->tx_pages_free) {
+ spin_unlock_irqrestore(&priv->tx_lock, flags);
+
+ /* Tx overflow */
+ return NULL;
+ }
+
+ priv->tx_pages_free--;
+
+ /* keep address of the very last free slot of current page */
+ lk = page->vbase + page->offset;
+
+ /* next, move on a new free page */
+ priv->tx_page_index = (priv->tx_page_index + 1) %
+ PCIEFD_TX_PAGE_COUNT;
+ page = priv->tx_pages + priv->tx_page_index;
+
+ /* put link record to this new page at the end of prev one */
+ lk->size = cpu_to_le16(sizeof(*lk));
+ lk->type = cpu_to_le16(CANFD_MSG_LNK_TX);
+ lk->laddr_lo = cpu_to_le32(page->lbase);
+
+#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
+ lk->laddr_hi = cpu_to_le32(page->lbase >> 32);
+#else
+ lk->laddr_hi = 0;
+#endif
+ /* next msgs will be put from the begininng of this new page */
+ page->offset = 0;
+ }
+
+ *room_left = priv->tx_pages_free * page->size;
+
+ spin_unlock_irqrestore(&priv->tx_lock, flags);
+
+ msg = page->vbase + page->offset;
+
+ /* give back room left in the tx ring */
+ *room_left += page->size - (page->offset + msg_size);
+
+ return msg;
+}
+
+static int pciefd_write_tx_msg(struct peak_canfd_priv *ucan,
+ struct pucan_tx_msg *msg)
+{
+ struct pciefd_can *priv = (struct pciefd_can *)ucan;
+ struct pciefd_page *page = priv->tx_pages + priv->tx_page_index;
+
+ /* this slot is now reserved for writing the frame */
+ page->offset += le16_to_cpu(msg->size);
+
+ /* tell the board a frame has been written in Tx DMA area */
+ pciefd_can_writereg(priv, 1, PCIEFD_REG_CAN_TX_REQ_ACC);
+
+ return 0;
+}
+
+/* probe for CAN-FD channel #pciefd_board->can_count */
+static int pciefd_can_probe(struct pciefd_board *pciefd)
+{
+ struct net_device *ndev;
+ struct pciefd_can *priv;
+ u32 clk;
+ int err;
+
+ /* allocate the candev object with default isize of echo skbs ring */
+ ndev = alloc_peak_canfd_dev(sizeof(*priv), pciefd->can_count,
+ PCIEFD_ECHO_SKB_MAX);
+ if (!ndev) {
+ dev_err(&pciefd->pci_dev->dev,
+ "failed to alloc candev object\n");
+ goto failure;
+ }
+
+ priv = netdev_priv(ndev);
+
+ /* fill-in candev private object: */
+
+ /* setup PCIe-FD own callbacks */
+ priv->ucan.pre_cmd = pciefd_pre_cmd;
+ priv->ucan.write_cmd = pciefd_write_cmd;
+ priv->ucan.post_cmd = pciefd_post_cmd;
+ priv->ucan.enable_tx_path = pciefd_enable_tx_path;
+ priv->ucan.alloc_tx_msg = pciefd_alloc_tx_msg;
+ priv->ucan.write_tx_msg = pciefd_write_tx_msg;
+
+ /* setup PCIe-FD own command buffer */
+ priv->ucan.cmd_buffer = &priv->pucan_cmd;
+ priv->ucan.cmd_maxlen = sizeof(priv->pucan_cmd);
+
+ priv->board = pciefd;
+
+ /* CAN config regs block address */
+ priv->reg_base = pciefd->reg_base + PCIEFD_CANX_OFF(priv->ucan.index);
+
+ /* allocate non-cacheable DMA'able 4KB memory area for Rx */
+ priv->rx_dma_vaddr = dmam_alloc_coherent(&pciefd->pci_dev->dev,
+ PCIEFD_RX_DMA_SIZE,
+ &priv->rx_dma_laddr,
+ GFP_KERNEL);
+ if (!priv->rx_dma_vaddr) {
+ dev_err(&pciefd->pci_dev->dev,
+ "Rx dmam_alloc_coherent(%u) failure\n",
+ PCIEFD_RX_DMA_SIZE);
+ goto err_free_candev;
+ }
+
+ /* allocate non-cacheable DMA'able 4KB memory area for Tx */
+ priv->tx_dma_vaddr = dmam_alloc_coherent(&pciefd->pci_dev->dev,
+ PCIEFD_TX_DMA_SIZE,
+ &priv->tx_dma_laddr,
+ GFP_KERNEL);
+ if (!priv->tx_dma_vaddr) {
+ dev_err(&pciefd->pci_dev->dev,
+ "Tx dmaim_alloc_coherent(%u) failure\n",
+ PCIEFD_TX_DMA_SIZE);
+ goto err_free_candev;
+ }
+
+ /* CAN clock in RST mode */
+ pciefd_can_writereg(priv, CANFD_MISC_TS_RST, PCIEFD_REG_CAN_MISC);
+
+ /* read current clock value */
+ clk = pciefd_can_readreg(priv, PCIEFD_REG_CAN_CLK_SEL);
+ switch (clk) {
+ case CANFD_CLK_SEL_20MHZ:
+ priv->ucan.can.clock.freq = 20 * 1000 * 1000;
+ break;
+ case CANFD_CLK_SEL_24MHZ:
+ priv->ucan.can.clock.freq = 24 * 1000 * 1000;
+ break;
+ case CANFD_CLK_SEL_30MHZ:
+ priv->ucan.can.clock.freq = 30 * 1000 * 1000;
+ break;
+ case CANFD_CLK_SEL_40MHZ:
+ priv->ucan.can.clock.freq = 40 * 1000 * 1000;
+ break;
+ case CANFD_CLK_SEL_60MHZ:
+ priv->ucan.can.clock.freq = 60 * 1000 * 1000;
+ break;
+ default:
+ pciefd_can_writereg(priv, CANFD_CLK_SEL_80MHZ,
+ PCIEFD_REG_CAN_CLK_SEL);
+
+ /* fallthough */
+ case CANFD_CLK_SEL_80MHZ:
+ priv->ucan.can.clock.freq = 80 * 1000 * 1000;
+ break;
+ }
+
+ ndev->irq = pciefd->pci_dev->irq;
+
+ SET_NETDEV_DEV(ndev, &pciefd->pci_dev->dev);
+
+ err = register_candev(ndev);
+ if (err) {
+ dev_err(&pciefd->pci_dev->dev,
+ "couldn't register CAN device: %d\n", err);
+ goto err_free_candev;
+ }
+
+ spin_lock_init(&priv->tx_lock);
+
+ /* save the object address in the board structure */
+ pciefd->can[pciefd->can_count] = priv;
+
+ dev_info(&pciefd->pci_dev->dev, "%s at reg_base=0x%p irq=%d\n",
+ ndev->name, priv->reg_base, pciefd->pci_dev->irq);
+
+ return 0;
+
+err_free_candev:
+ free_candev(ndev);
+
+failure:
+ return -ENOMEM;
+}
+
+/* remove a CAN-FD channel by releasing all of its resources */
+static void pciefd_can_remove(struct pciefd_can *priv)
+{
+ /* unregister (close) the can device to go back to RST mode first */
+ unregister_candev(priv->ucan.ndev);
+
+ /* finally, free the candev object */
+ free_candev(priv->ucan.ndev);
+}
+
+/* remove all CAN-FD channels by releasing their own resources */
+static void pciefd_can_remove_all(struct pciefd_board *pciefd)
+{
+ while (pciefd->can_count > 0)
+ pciefd_can_remove(pciefd->can[--pciefd->can_count]);
+}
+
+/* probe for the entire device */
+static int peak_pciefd_probe(struct pci_dev *pdev,
+ const struct pci_device_id *ent)
+{
+ struct pciefd_board *pciefd;
+ int err, can_count;
+ u16 sub_sys_id;
+ u8 hw_ver_major;
+ u8 hw_ver_minor;
+ u8 hw_ver_sub;
+ u32 v2;
+
+ err = pci_enable_device(pdev);
+ if (err)
+ return err;
+ err = pci_request_regions(pdev, PCIEFD_DRV_NAME);
+ if (err)
+ goto err_disable_pci;
+
+ /* the number of channels depends on sub-system id */
+ err = pci_read_config_word(pdev, PCI_SUBSYSTEM_ID, &sub_sys_id);
+ if (err)
+ goto err_release_regions;
+
+ dev_dbg(&pdev->dev, "probing device %04x:%04x:%04x\n",
+ pdev->vendor, pdev->device, sub_sys_id);
+
+ if (sub_sys_id >= 0x0012)
+ can_count = 4;
+ else if (sub_sys_id >= 0x0010)
+ can_count = 3;
+ else if (sub_sys_id >= 0x0004)
+ can_count = 2;
+ else
+ can_count = 1;
+
+ /* allocate board structure object */
+ pciefd = devm_kzalloc(&pdev->dev, sizeof(*pciefd) +
+ can_count * sizeof(*pciefd->can),
+ GFP_KERNEL);
+ if (!pciefd) {
+ err = -ENOMEM;
+ goto err_release_regions;
+ }
+
+ /* initialize the board structure */
+ pciefd->pci_dev = pdev;
+ spin_lock_init(&pciefd->cmd_lock);
+
+ /* save the PCI BAR0 virtual address for further system regs access */
+ pciefd->reg_base = pci_iomap(pdev, 0, PCIEFD_BAR0_SIZE);
+ if (!pciefd->reg_base) {
+ dev_err(&pdev->dev, "failed to map PCI resource #0\n");
+ err = -ENOMEM;
+ goto err_release_regions;
+ }
+
+ /* read the firmware version number */
+ v2 = pciefd_sys_readreg(pciefd, PCIEFD_REG_SYS_VER2);
+
+ hw_ver_major = (v2 & 0x0000f000) >> 12;
+ hw_ver_minor = (v2 & 0x00000f00) >> 8;
+ hw_ver_sub = (v2 & 0x000000f0) >> 4;
+
+ dev_info(&pdev->dev,
+ "%ux CAN-FD PCAN-PCIe FPGA v%u.%u.%u:\n", can_count,
+ hw_ver_major, hw_ver_minor, hw_ver_sub);
+
+ /* stop system clock */
+ pciefd_sys_writereg(pciefd, PCIEFD_SYS_CTL_CLK_EN,
+ PCIEFD_REG_SYS_CTL_CLR);
+
+ pci_set_master(pdev);
+
+ /* create now the corresponding channels objects */
+ while (pciefd->can_count < can_count) {
+ err = pciefd_can_probe(pciefd);
+ if (err)
+ goto err_free_canfd;
+
+ pciefd->can_count++;
+ }
+
+ /* set system timestamps counter in RST mode */
+ pciefd_sys_writereg(pciefd, PCIEFD_SYS_CTL_TS_RST,
+ PCIEFD_REG_SYS_CTL_SET);
+
+ /* wait a bit (read cycle) */
+ (void)pciefd_sys_readreg(pciefd, PCIEFD_REG_SYS_VER1);
+
+ /* free all clocks */
+ pciefd_sys_writereg(pciefd, PCIEFD_SYS_CTL_TS_RST,
+ PCIEFD_REG_SYS_CTL_CLR);
+
+ /* start system clock */
+ pciefd_sys_writereg(pciefd, PCIEFD_SYS_CTL_CLK_EN,
+ PCIEFD_REG_SYS_CTL_SET);
+
+ /* remember the board structure address in the device user data */
+ pci_set_drvdata(pdev, pciefd);
+
+ return 0;
+
+err_free_canfd:
+ pciefd_can_remove_all(pciefd);
+
+ pci_iounmap(pdev, pciefd->reg_base);
+
+err_release_regions:
+ pci_release_regions(pdev);
+
+err_disable_pci:
+ pci_disable_device(pdev);
+
+ return err;
+}
+
+/* free the board structure object, as well as its resources: */
+static void peak_pciefd_remove(struct pci_dev *pdev)
+{
+ struct pciefd_board *pciefd = pci_get_drvdata(pdev);
+
+ /* release CAN-FD channels resources */
+ pciefd_can_remove_all(pciefd);
+
+ pci_iounmap(pdev, pciefd->reg_base);
+
+ pci_release_regions(pdev);
+ pci_disable_device(pdev);
+}
+
+static struct pci_driver peak_pciefd_driver = {
+ .name = PCIEFD_DRV_NAME,
+ .id_table = peak_pciefd_tbl,
+ .probe = peak_pciefd_probe,
+ .remove = peak_pciefd_remove,
+};
+
+module_pci_driver(peak_pciefd_driver);
diff --git a/drivers/net/can/spi/Kconfig b/drivers/net/can/spi/Kconfig
index 148cae5..8f2e0dd 100644
--- a/drivers/net/can/spi/Kconfig
+++ b/drivers/net/can/spi/Kconfig
@@ -1,6 +1,12 @@
menu "CAN SPI interfaces"
depends on SPI
+config CAN_HI311X
+ tristate "Holt HI311x SPI CAN controllers"
+ depends on CAN_DEV && SPI && HAS_DMA
+ ---help---
+ Driver for the Holt HI311x SPI CAN controllers.
+
config CAN_MCP251X
tristate "Microchip MCP251x SPI CAN controllers"
depends on HAS_DMA
diff --git a/drivers/net/can/spi/Makefile b/drivers/net/can/spi/Makefile
index 0e86040..f59fa37 100644
--- a/drivers/net/can/spi/Makefile
+++ b/drivers/net/can/spi/Makefile
@@ -3,4 +3,5 @@
#
+obj-$(CONFIG_CAN_HI311X) += hi311x.o
obj-$(CONFIG_CAN_MCP251X) += mcp251x.o
diff --git a/drivers/net/can/spi/hi311x.c b/drivers/net/can/spi/hi311x.c
new file mode 100644
index 0000000..5590c55
--- /dev/null
+++ b/drivers/net/can/spi/hi311x.c
@@ -0,0 +1,1076 @@
+/* CAN bus driver for Holt HI3110 CAN Controller with SPI Interface
+ *
+ * Copyright(C) Timesys Corporation 2016
+ *
+ * Based on Microchip 251x CAN Controller (mcp251x) Linux kernel driver
+ * Copyright 2009 Christian Pellegrin EVOL S.r.l.
+ * Copyright 2007 Raymarine UK, Ltd. All Rights Reserved.
+ * Copyright 2006 Arcom Control Systems Ltd.
+ *
+ * Based on CAN bus driver for the CCAN controller written by
+ * - Sascha Hauer, Marc Kleine-Budde, Pengutronix
+ * - Simon Kallweit, intefo AG
+ * Copyright 2007
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/can/core.h>
+#include <linux/can/dev.h>
+#include <linux/can/led.h>
+#include <linux/clk.h>
+#include <linux/completion.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/freezer.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/netdevice.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/platform_device.h>
+#include <linux/regulator/consumer.h>
+#include <linux/slab.h>
+#include <linux/spi/spi.h>
+#include <linux/uaccess.h>
+
+#define HI3110_MASTER_RESET 0x56
+#define HI3110_READ_CTRL0 0xD2
+#define HI3110_READ_CTRL1 0xD4
+#define HI3110_READ_STATF 0xE2
+#define HI3110_WRITE_CTRL0 0x14
+#define HI3110_WRITE_CTRL1 0x16
+#define HI3110_WRITE_INTE 0x1C
+#define HI3110_WRITE_BTR0 0x18
+#define HI3110_WRITE_BTR1 0x1A
+#define HI3110_READ_BTR0 0xD6
+#define HI3110_READ_BTR1 0xD8
+#define HI3110_READ_INTF 0xDE
+#define HI3110_READ_ERR 0xDC
+#define HI3110_READ_FIFO_WOTIME 0x48
+#define HI3110_WRITE_FIFO 0x12
+#define HI3110_READ_MESSTAT 0xDA
+#define HI3110_READ_REC 0xEA
+#define HI3110_READ_TEC 0xEC
+
+#define HI3110_CTRL0_MODE_MASK (7 << 5)
+#define HI3110_CTRL0_NORMAL_MODE (0 << 5)
+#define HI3110_CTRL0_LOOPBACK_MODE (1 << 5)
+#define HI3110_CTRL0_MONITOR_MODE (2 << 5)
+#define HI3110_CTRL0_SLEEP_MODE (3 << 5)
+#define HI3110_CTRL0_INIT_MODE (4 << 5)
+
+#define HI3110_CTRL1_TXEN BIT(7)
+
+#define HI3110_INT_RXTMP BIT(7)
+#define HI3110_INT_RXFIFO BIT(6)
+#define HI3110_INT_TXCPLT BIT(5)
+#define HI3110_INT_BUSERR BIT(4)
+#define HI3110_INT_MCHG BIT(3)
+#define HI3110_INT_WAKEUP BIT(2)
+#define HI3110_INT_F1MESS BIT(1)
+#define HI3110_INT_F0MESS BIT(0)
+
+#define HI3110_ERR_BUSOFF BIT(7)
+#define HI3110_ERR_TXERRP BIT(6)
+#define HI3110_ERR_RXERRP BIT(5)
+#define HI3110_ERR_BITERR BIT(4)
+#define HI3110_ERR_FRMERR BIT(3)
+#define HI3110_ERR_CRCERR BIT(2)
+#define HI3110_ERR_ACKERR BIT(1)
+#define HI3110_ERR_STUFERR BIT(0)
+#define HI3110_ERR_PROTOCOL_MASK (0x1F)
+#define HI3110_ERR_PASSIVE_MASK (0x60)
+
+#define HI3110_STAT_RXFMTY BIT(1)
+#define HI3110_STAT_BUSOFF BIT(2)
+#define HI3110_STAT_ERRP BIT(3)
+#define HI3110_STAT_ERRW BIT(4)
+
+#define HI3110_BTR0_SJW_SHIFT 6
+#define HI3110_BTR0_BRP_SHIFT 0
+
+#define HI3110_BTR1_SAMP_3PERBIT (1 << 7)
+#define HI3110_BTR1_SAMP_1PERBIT (0 << 7)
+#define HI3110_BTR1_TSEG2_SHIFT 4
+#define HI3110_BTR1_TSEG1_SHIFT 0
+
+#define HI3110_FIFO_WOTIME_TAG_OFF 0
+#define HI3110_FIFO_WOTIME_ID_OFF 1
+#define HI3110_FIFO_WOTIME_DLC_OFF 5
+#define HI3110_FIFO_WOTIME_DAT_OFF 6
+
+#define HI3110_FIFO_WOTIME_TAG_IDE BIT(7)
+#define HI3110_FIFO_WOTIME_ID_RTR BIT(0)
+
+#define HI3110_FIFO_TAG_OFF 0
+#define HI3110_FIFO_ID_OFF 1
+#define HI3110_FIFO_STD_DLC_OFF 3
+#define HI3110_FIFO_STD_DATA_OFF 4
+#define HI3110_FIFO_EXT_DLC_OFF 5
+#define HI3110_FIFO_EXT_DATA_OFF 6
+
+#define HI3110_CAN_MAX_DATA_LEN 8
+#define HI3110_RX_BUF_LEN 15
+#define HI3110_TX_STD_BUF_LEN 12
+#define HI3110_TX_EXT_BUF_LEN 14
+#define HI3110_CAN_FRAME_MAX_BITS 128
+#define HI3110_EFF_FLAGS 0x18 /* IDE + SRR */
+
+#define HI3110_TX_ECHO_SKB_MAX 1
+
+#define HI3110_OST_DELAY_MS (10)
+
+#define DEVICE_NAME "hi3110"
+
+static int hi3110_enable_dma = 1; /* Enable SPI DMA. Default: 1 (On) */
+module_param(hi3110_enable_dma, int, 0444);
+MODULE_PARM_DESC(hi3110_enable_dma, "Enable SPI DMA. Default: 1 (On)");
+
+static const struct can_bittiming_const hi3110_bittiming_const = {
+ .name = DEVICE_NAME,
+ .tseg1_min = 2,
+ .tseg1_max = 16,
+ .tseg2_min = 2,
+ .tseg2_max = 8,
+ .sjw_max = 4,
+ .brp_min = 1,
+ .brp_max = 64,
+ .brp_inc = 1,
+};
+
+enum hi3110_model {
+ CAN_HI3110_HI3110 = 0x3110,
+};
+
+struct hi3110_priv {
+ struct can_priv can;
+ struct net_device *net;
+ struct spi_device *spi;
+ enum hi3110_model model;
+
+ struct mutex hi3110_lock; /* SPI device lock */
+
+ u8 *spi_tx_buf;
+ u8 *spi_rx_buf;
+ dma_addr_t spi_tx_dma;
+ dma_addr_t spi_rx_dma;
+
+ struct sk_buff *tx_skb;
+ int tx_len;
+
+ struct workqueue_struct *wq;
+ struct work_struct tx_work;
+ struct work_struct restart_work;
+
+ int force_quit;
+ int after_suspend;
+#define HI3110_AFTER_SUSPEND_UP 1
+#define HI3110_AFTER_SUSPEND_DOWN 2
+#define HI3110_AFTER_SUSPEND_POWER 4
+#define HI3110_AFTER_SUSPEND_RESTART 8
+ int restart_tx;
+ struct regulator *power;
+ struct regulator *transceiver;
+ struct clk *clk;
+};
+
+static void hi3110_clean(struct net_device *net)
+{
+ struct hi3110_priv *priv = netdev_priv(net);
+
+ if (priv->tx_skb || priv->tx_len)
+ net->stats.tx_errors++;
+ if (priv->tx_skb)
+ dev_kfree_skb(priv->tx_skb);
+ if (priv->tx_len)
+ can_free_echo_skb(priv->net, 0);
+ priv->tx_skb = NULL;
+ priv->tx_len = 0;
+}
+
+/* Note about handling of error return of hi3110_spi_trans: accessing
+ * registers via SPI is not really different conceptually than using
+ * normal I/O assembler instructions, although it's much more
+ * complicated from a practical POV. So it's not advisable to always
+ * check the return value of this function. Imagine that every
+ * read{b,l}, write{b,l} and friends would be bracketed in "if ( < 0)
+ * error();", it would be a great mess (well there are some situation
+ * when exception handling C++ like could be useful after all). So we
+ * just check that transfers are OK at the beginning of our
+ * conversation with the chip and to avoid doing really nasty things
+ * (like injecting bogus packets in the network stack).
+ */
+static int hi3110_spi_trans(struct spi_device *spi, int len)
+{
+ struct hi3110_priv *priv = spi_get_drvdata(spi);
+ struct spi_transfer t = {
+ .tx_buf = priv->spi_tx_buf,
+ .rx_buf = priv->spi_rx_buf,
+ .len = len,
+ .cs_change = 0,
+ };
+ struct spi_message m;
+ int ret;
+
+ spi_message_init(&m);
+
+ if (hi3110_enable_dma) {
+ t.tx_dma = priv->spi_tx_dma;
+ t.rx_dma = priv->spi_rx_dma;
+ m.is_dma_mapped = 1;
+ }
+
+ spi_message_add_tail(&t, &m);
+
+ ret = spi_sync(spi, &m);
+
+ if (ret)
+ dev_err(&spi->dev, "spi transfer failed: ret = %d\n", ret);
+ return ret;
+}
+
+static u8 hi3110_cmd(struct spi_device *spi, u8 command)
+{
+ struct hi3110_priv *priv = spi_get_drvdata(spi);
+
+ priv->spi_tx_buf[0] = command;
+ dev_dbg(&spi->dev, "hi3110_cmd: %02X\n", command);
+
+ return hi3110_spi_trans(spi, 1);
+}
+
+static u8 hi3110_read(struct spi_device *spi, u8 command)
+{
+ struct hi3110_priv *priv = spi_get_drvdata(spi);
+ u8 val = 0;
+
+ priv->spi_tx_buf[0] = command;
+ hi3110_spi_trans(spi, 2);
+ val = priv->spi_rx_buf[1];
+
+ return val;
+}
+
+static void hi3110_write(struct spi_device *spi, u8 reg, u8 val)
+{
+ struct hi3110_priv *priv = spi_get_drvdata(spi);
+
+ priv->spi_tx_buf[0] = reg;
+ priv->spi_tx_buf[1] = val;
+ hi3110_spi_trans(spi, 2);
+}
+
+static void hi3110_hw_tx_frame(struct spi_device *spi, u8 *buf, int len)
+{
+ struct hi3110_priv *priv = spi_get_drvdata(spi);
+
+ priv->spi_tx_buf[0] = HI3110_WRITE_FIFO;
+ memcpy(priv->spi_tx_buf + 1, buf, len);
+ hi3110_spi_trans(spi, len + 1);
+}
+
+static void hi3110_hw_tx(struct spi_device *spi, struct can_frame *frame)
+{
+ u8 buf[HI3110_TX_EXT_BUF_LEN];
+
+ buf[HI3110_FIFO_TAG_OFF] = 0;
+
+ if (frame->can_id & CAN_EFF_FLAG) {
+ /* Extended frame */
+ buf[HI3110_FIFO_ID_OFF] = (frame->can_id & CAN_EFF_MASK) >> 21;
+ buf[HI3110_FIFO_ID_OFF + 1] =
+ (((frame->can_id & CAN_EFF_MASK) >> 13) & 0xe0) |
+ HI3110_EFF_FLAGS |
+ (((frame->can_id & CAN_EFF_MASK) >> 15) & 0x07);
+ buf[HI3110_FIFO_ID_OFF + 2] =
+ (frame->can_id & CAN_EFF_MASK) >> 7;
+ buf[HI3110_FIFO_ID_OFF + 3] =
+ ((frame->can_id & CAN_EFF_MASK) << 1) |
+ ((frame->can_id & CAN_RTR_FLAG) ? 1 : 0);
+
+ buf[HI3110_FIFO_EXT_DLC_OFF] = frame->can_dlc;
+
+ memcpy(buf + HI3110_FIFO_EXT_DATA_OFF,
+ frame->data, frame->can_dlc);
+
+ hi3110_hw_tx_frame(spi, buf, HI3110_TX_EXT_BUF_LEN -
+ (HI3110_CAN_MAX_DATA_LEN - frame->can_dlc));
+ } else {
+ /* Standard frame */
+ buf[HI3110_FIFO_ID_OFF] = (frame->can_id & CAN_SFF_MASK) >> 3;
+ buf[HI3110_FIFO_ID_OFF + 1] =
+ ((frame->can_id & CAN_SFF_MASK) << 5) |
+ ((frame->can_id & CAN_RTR_FLAG) ? (1 << 4) : 0);
+
+ buf[HI3110_FIFO_STD_DLC_OFF] = frame->can_dlc;
+
+ memcpy(buf + HI3110_FIFO_STD_DATA_OFF,
+ frame->data, frame->can_dlc);
+
+ hi3110_hw_tx_frame(spi, buf, HI3110_TX_STD_BUF_LEN -
+ (HI3110_CAN_MAX_DATA_LEN - frame->can_dlc));
+ }
+}
+
+static void hi3110_hw_rx_frame(struct spi_device *spi, u8 *buf)
+{
+ struct hi3110_priv *priv = spi_get_drvdata(spi);
+
+ priv->spi_tx_buf[0] = HI3110_READ_FIFO_WOTIME;
+ hi3110_spi_trans(spi, HI3110_RX_BUF_LEN);
+ memcpy(buf, priv->spi_rx_buf + 1, HI3110_RX_BUF_LEN - 1);
+}
+
+static void hi3110_hw_rx(struct spi_device *spi)
+{
+ struct hi3110_priv *priv = spi_get_drvdata(spi);
+ struct sk_buff *skb;
+ struct can_frame *frame;
+ u8 buf[HI3110_RX_BUF_LEN - 1];
+
+ skb = alloc_can_skb(priv->net, &frame);
+ if (!skb) {
+ priv->net->stats.rx_dropped++;
+ return;
+ }
+
+ hi3110_hw_rx_frame(spi, buf);
+ if (buf[HI3110_FIFO_WOTIME_TAG_OFF] & HI3110_FIFO_WOTIME_TAG_IDE) {
+ /* IDE is recessive (1), indicating extended 29-bit frame */
+ frame->can_id = CAN_EFF_FLAG;
+ frame->can_id |=
+ (buf[HI3110_FIFO_WOTIME_ID_OFF] << 21) |
+ (((buf[HI3110_FIFO_WOTIME_ID_OFF + 1] & 0xE0) >> 5) << 18) |
+ ((buf[HI3110_FIFO_WOTIME_ID_OFF + 1] & 0x07) << 15) |
+ (buf[HI3110_FIFO_WOTIME_ID_OFF + 2] << 7) |
+ (buf[HI3110_FIFO_WOTIME_ID_OFF + 3] >> 1);
+ } else {
+ /* IDE is dominant (0), frame indicating standard 11-bit */
+ frame->can_id =
+ (buf[HI3110_FIFO_WOTIME_ID_OFF] << 3) |
+ ((buf[HI3110_FIFO_WOTIME_ID_OFF + 1] & 0xE0) >> 5);
+ }
+
+ /* Data length */
+ frame->can_dlc = get_can_dlc(buf[HI3110_FIFO_WOTIME_DLC_OFF] & 0x0F);
+
+ if (buf[HI3110_FIFO_WOTIME_ID_OFF + 3] & HI3110_FIFO_WOTIME_ID_RTR)
+ frame->can_id |= CAN_RTR_FLAG;
+ else
+ memcpy(frame->data, buf + HI3110_FIFO_WOTIME_DAT_OFF,
+ frame->can_dlc);
+
+ priv->net->stats.rx_packets++;
+ priv->net->stats.rx_bytes += frame->can_dlc;
+
+ can_led_event(priv->net, CAN_LED_EVENT_RX);
+
+ netif_rx_ni(skb);
+}
+
+static void hi3110_hw_sleep(struct spi_device *spi)
+{
+ hi3110_write(spi, HI3110_WRITE_CTRL0, HI3110_CTRL0_SLEEP_MODE);
+}
+
+static netdev_tx_t hi3110_hard_start_xmit(struct sk_buff *skb,
+ struct net_device *net)
+{
+ struct hi3110_priv *priv = netdev_priv(net);
+ struct spi_device *spi = priv->spi;
+
+ if (priv->tx_skb || priv->tx_len) {
+ dev_err(&spi->dev, "hard_xmit called while tx busy\n");
+ return NETDEV_TX_BUSY;
+ }
+
+ if (can_dropped_invalid_skb(net, skb))
+ return NETDEV_TX_OK;
+
+ netif_stop_queue(net);
+ priv->tx_skb = skb;
+ queue_work(priv->wq, &priv->tx_work);
+
+ return NETDEV_TX_OK;
+}
+
+static int hi3110_do_set_mode(struct net_device *net, enum can_mode mode)
+{
+ struct hi3110_priv *priv = netdev_priv(net);
+
+ switch (mode) {
+ case CAN_MODE_START:
+ hi3110_clean(net);
+ /* We have to delay work since SPI I/O may sleep */
+ priv->can.state = CAN_STATE_ERROR_ACTIVE;
+ priv->restart_tx = 1;
+ if (priv->can.restart_ms == 0)
+ priv->after_suspend = HI3110_AFTER_SUSPEND_RESTART;
+ queue_work(priv->wq, &priv->restart_work);
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
+static int hi3110_get_berr_counter(const struct net_device *net,
+ struct can_berr_counter *bec)
+{
+ struct hi3110_priv *priv = netdev_priv(net);
+ struct spi_device *spi = priv->spi;
+
+ bec->txerr = hi3110_read(spi, HI3110_READ_TEC);
+ bec->rxerr = hi3110_read(spi, HI3110_READ_REC);
+
+ return 0;
+}
+
+static int hi3110_set_normal_mode(struct spi_device *spi)
+{
+ struct hi3110_priv *priv = spi_get_drvdata(spi);
+ u8 reg = 0;
+
+ hi3110_write(spi, HI3110_WRITE_INTE, HI3110_INT_BUSERR |
+ HI3110_INT_RXFIFO | HI3110_INT_TXCPLT);
+
+ /* Enable TX */
+ hi3110_write(spi, HI3110_WRITE_CTRL1, HI3110_CTRL1_TXEN);
+
+ if (priv->can.ctrlmode & CAN_CTRLMODE_LOOPBACK)
+ reg = HI3110_CTRL0_LOOPBACK_MODE;
+ else if (priv->can.ctrlmode & CAN_CTRLMODE_LISTENONLY)
+ reg = HI3110_CTRL0_MONITOR_MODE;
+ else
+ reg = HI3110_CTRL0_NORMAL_MODE;
+
+ hi3110_write(spi, HI3110_WRITE_CTRL0, reg);
+
+ /* Wait for the device to enter the mode */
+ mdelay(HI3110_OST_DELAY_MS);
+ reg = hi3110_read(spi, HI3110_READ_CTRL0);
+ if ((reg & HI3110_CTRL0_MODE_MASK) != reg)
+ return -EBUSY;
+
+ priv->can.state = CAN_STATE_ERROR_ACTIVE;
+ return 0;
+}
+
+static int hi3110_do_set_bittiming(struct net_device *net)
+{
+ struct hi3110_priv *priv = netdev_priv(net);
+ struct can_bittiming *bt = &priv->can.bittiming;
+ struct spi_device *spi = priv->spi;
+
+ hi3110_write(spi, HI3110_WRITE_BTR0,
+ ((bt->sjw - 1) << HI3110_BTR0_SJW_SHIFT) |
+ ((bt->brp - 1) << HI3110_BTR0_BRP_SHIFT));
+
+ hi3110_write(spi, HI3110_WRITE_BTR1,
+ (priv->can.ctrlmode &
+ CAN_CTRLMODE_3_SAMPLES ?
+ HI3110_BTR1_SAMP_3PERBIT : HI3110_BTR1_SAMP_1PERBIT) |
+ ((bt->phase_seg1 + bt->prop_seg - 1)
+ << HI3110_BTR1_TSEG1_SHIFT) |
+ ((bt->phase_seg2 - 1) << HI3110_BTR1_TSEG2_SHIFT));
+
+ dev_dbg(&spi->dev, "BT: 0x%02x 0x%02x\n",
+ hi3110_read(spi, HI3110_READ_BTR0),
+ hi3110_read(spi, HI3110_READ_BTR1));
+
+ return 0;
+}
+
+static int hi3110_setup(struct net_device *net)
+{
+ hi3110_do_set_bittiming(net);
+ return 0;
+}
+
+static int hi3110_hw_reset(struct spi_device *spi)
+{
+ u8 reg;
+ int ret;
+
+ /* Wait for oscillator startup timer after power up */
+ mdelay(HI3110_OST_DELAY_MS);
+
+ ret = hi3110_cmd(spi, HI3110_MASTER_RESET);
+ if (ret)
+ return ret;
+
+ /* Wait for oscillator startup timer after reset */
+ mdelay(HI3110_OST_DELAY_MS);
+
+ reg = hi3110_read(spi, HI3110_READ_CTRL0);
+ if ((reg & HI3110_CTRL0_MODE_MASK) != HI3110_CTRL0_INIT_MODE)
+ return -ENODEV;
+
+ /* As per the datasheet it appears the error flags are
+ * not cleared on reset. Explicitly clear them by performing a read
+ */
+ hi3110_read(spi, HI3110_READ_ERR);
+
+ return 0;
+}
+
+static int hi3110_hw_probe(struct spi_device *spi)
+{
+ u8 statf;
+
+ hi3110_hw_reset(spi);
+
+ /* Confirm correct operation by checking against reset values
+ * in datasheet
+ */
+ statf = hi3110_read(spi, HI3110_READ_STATF);
+
+ dev_dbg(&spi->dev, "statf: %02X\n", statf);
+
+ if (statf != 0x82)
+ return -ENODEV;
+
+ return 0;
+}
+
+static int hi3110_power_enable(struct regulator *reg, int enable)
+{
+ if (IS_ERR_OR_NULL(reg))
+ return 0;
+
+ if (enable)
+ return regulator_enable(reg);
+ else
+ return regulator_disable(reg);
+}
+
+static int hi3110_stop(struct net_device *net)
+{
+ struct hi3110_priv *priv = netdev_priv(net);
+ struct spi_device *spi = priv->spi;
+
+ close_candev(net);
+
+ priv->force_quit = 1;
+ free_irq(spi->irq, priv);
+ destroy_workqueue(priv->wq);
+ priv->wq = NULL;
+
+ mutex_lock(&priv->hi3110_lock);
+
+ /* Disable transmit, interrupts and clear flags */
+ hi3110_write(spi, HI3110_WRITE_CTRL1, 0x0);
+ hi3110_write(spi, HI3110_WRITE_INTE, 0x0);
+ hi3110_read(spi, HI3110_READ_INTF);
+
+ hi3110_clean(net);
+
+ hi3110_hw_sleep(spi);
+
+ hi3110_power_enable(priv->transceiver, 0);
+
+ priv->can.state = CAN_STATE_STOPPED;
+
+ mutex_unlock(&priv->hi3110_lock);
+
+ can_led_event(net, CAN_LED_EVENT_STOP);
+
+ return 0;
+}
+
+static void hi3110_tx_work_handler(struct work_struct *ws)
+{
+ struct hi3110_priv *priv = container_of(ws, struct hi3110_priv,
+ tx_work);
+ struct spi_device *spi = priv->spi;
+ struct net_device *net = priv->net;
+ struct can_frame *frame;
+
+ mutex_lock(&priv->hi3110_lock);
+ if (priv->tx_skb) {
+ if (priv->can.state == CAN_STATE_BUS_OFF) {
+ hi3110_clean(net);
+ } else {
+ frame = (struct can_frame *)priv->tx_skb->data;
+ hi3110_hw_tx(spi, frame);
+ priv->tx_len = 1 + frame->can_dlc;
+ can_put_echo_skb(priv->tx_skb, net, 0);
+ priv->tx_skb = NULL;
+ }
+ }
+ mutex_unlock(&priv->hi3110_lock);
+}
+
+static void hi3110_restart_work_handler(struct work_struct *ws)
+{
+ struct hi3110_priv *priv = container_of(ws, struct hi3110_priv,
+ restart_work);
+ struct spi_device *spi = priv->spi;
+ struct net_device *net = priv->net;
+
+ mutex_lock(&priv->hi3110_lock);
+ if (priv->after_suspend) {
+ hi3110_hw_reset(spi);
+ hi3110_setup(net);
+ if (priv->after_suspend & HI3110_AFTER_SUSPEND_RESTART) {
+ hi3110_set_normal_mode(spi);
+ } else if (priv->after_suspend & HI3110_AFTER_SUSPEND_UP) {
+ netif_device_attach(net);
+ hi3110_clean(net);
+ hi3110_set_normal_mode(spi);
+ netif_wake_queue(net);
+ } else {
+ hi3110_hw_sleep(spi);
+ }
+ priv->after_suspend = 0;
+ priv->force_quit = 0;
+ }
+
+ if (priv->restart_tx) {
+ priv->restart_tx = 0;
+ hi3110_hw_reset(spi);
+ hi3110_setup(net);
+ hi3110_clean(net);
+ hi3110_set_normal_mode(spi);
+ netif_wake_queue(net);
+ }
+ mutex_unlock(&priv->hi3110_lock);
+}
+
+static irqreturn_t hi3110_can_ist(int irq, void *dev_id)
+{
+ struct hi3110_priv *priv = dev_id;
+ struct spi_device *spi = priv->spi;
+ struct net_device *net = priv->net;
+
+ mutex_lock(&priv->hi3110_lock);
+
+ while (!priv->force_quit) {
+ enum can_state new_state;
+ u8 intf, eflag, statf;
+
+ while (!(HI3110_STAT_RXFMTY &
+ (statf = hi3110_read(spi, HI3110_READ_STATF)))) {
+ hi3110_hw_rx(spi);
+ }
+
+ intf = hi3110_read(spi, HI3110_READ_INTF);
+ eflag = hi3110_read(spi, HI3110_READ_ERR);
+ /* Update can state */
+ if (eflag & HI3110_ERR_BUSOFF)
+ new_state = CAN_STATE_BUS_OFF;
+ else if (eflag & HI3110_ERR_PASSIVE_MASK)
+ new_state = CAN_STATE_ERROR_PASSIVE;
+ else if (statf & HI3110_STAT_ERRW)
+ new_state = CAN_STATE_ERROR_WARNING;
+ else
+ new_state = CAN_STATE_ERROR_ACTIVE;
+
+ if (new_state != priv->can.state) {
+ struct can_frame *cf;
+ struct sk_buff *skb;
+ enum can_state rx_state, tx_state;
+ u8 rxerr, txerr;
+
+ skb = alloc_can_err_skb(net, &cf);
+ if (!skb)
+ break;
+
+ txerr = hi3110_read(spi, HI3110_READ_TEC);
+ rxerr = hi3110_read(spi, HI3110_READ_REC);
+ cf->data[6] = txerr;
+ cf->data[7] = rxerr;
+ tx_state = txerr >= rxerr ? new_state : 0;
+ rx_state = txerr <= rxerr ? new_state : 0;
+ can_change_state(net, cf, tx_state, rx_state);
+ netif_rx_ni(skb);
+
+ if (new_state == CAN_STATE_BUS_OFF) {
+ can_bus_off(net);
+ if (priv->can.restart_ms == 0) {
+ priv->force_quit = 1;
+ hi3110_hw_sleep(spi);
+ break;
+ }
+ }
+ }
+
+ /* Update bus errors */
+ if ((intf & HI3110_INT_BUSERR) &&
+ (priv->can.ctrlmode & CAN_CTRLMODE_BERR_REPORTING)) {
+ struct can_frame *cf;
+ struct sk_buff *skb;
+
+ /* Check for protocol errors */
+ if (eflag & HI3110_ERR_PROTOCOL_MASK) {
+ skb = alloc_can_err_skb(net, &cf);
+ if (!skb)
+ break;
+
+ cf->can_id |= CAN_ERR_PROT | CAN_ERR_BUSERROR;
+ priv->can.can_stats.bus_error++;
+ priv->net->stats.rx_errors++;
+ if (eflag & HI3110_ERR_BITERR)
+ cf->data[2] |= CAN_ERR_PROT_BIT;
+ else if (eflag & HI3110_ERR_FRMERR)
+ cf->data[2] |= CAN_ERR_PROT_FORM;
+ else if (eflag & HI3110_ERR_STUFERR)
+ cf->data[2] |= CAN_ERR_PROT_STUFF;
+ else if (eflag & HI3110_ERR_CRCERR)
+ cf->data[3] |= CAN_ERR_PROT_LOC_CRC_SEQ;
+ else if (eflag & HI3110_ERR_ACKERR)
+ cf->data[3] |= CAN_ERR_PROT_LOC_ACK;
+
+ cf->data[6] = hi3110_read(spi, HI3110_READ_TEC);
+ cf->data[7] = hi3110_read(spi, HI3110_READ_REC);
+ netdev_dbg(priv->net, "Bus Error\n");
+ netif_rx_ni(skb);
+ }
+ }
+
+ if (intf == 0)
+ break;
+
+ if (intf & HI3110_INT_TXCPLT) {
+ net->stats.tx_packets++;
+ net->stats.tx_bytes += priv->tx_len - 1;
+ can_led_event(net, CAN_LED_EVENT_TX);
+ if (priv->tx_len) {
+ can_get_echo_skb(net, 0);
+ priv->tx_len = 0;
+ }
+ netif_wake_queue(net);
+ }
+ }
+ mutex_unlock(&priv->hi3110_lock);
+ return IRQ_HANDLED;
+}
+
+static int hi3110_open(struct net_device *net)
+{
+ struct hi3110_priv *priv = netdev_priv(net);
+ struct spi_device *spi = priv->spi;
+ unsigned long flags = IRQF_ONESHOT | IRQF_TRIGGER_RISING;
+ int ret;
+
+ ret = open_candev(net);
+ if (ret)
+ return ret;
+
+ mutex_lock(&priv->hi3110_lock);
+ hi3110_power_enable(priv->transceiver, 1);
+
+ priv->force_quit = 0;
+ priv->tx_skb = NULL;
+ priv->tx_len = 0;
+
+ ret = request_threaded_irq(spi->irq, NULL, hi3110_can_ist,
+ flags, DEVICE_NAME, priv);
+ if (ret) {
+ dev_err(&spi->dev, "failed to acquire irq %d\n", spi->irq);
+ goto out_close;
+ }
+
+ priv->wq = alloc_workqueue("hi3110_wq", WQ_FREEZABLE | WQ_MEM_RECLAIM,
+ 0);
+ if (!priv->wq) {
+ ret = -ENOMEM;
+ goto out_free_irq;
+ }
+ INIT_WORK(&priv->tx_work, hi3110_tx_work_handler);
+ INIT_WORK(&priv->restart_work, hi3110_restart_work_handler);
+
+ ret = hi3110_hw_reset(spi);
+ if (ret)
+ goto out_free_wq;
+
+ ret = hi3110_setup(net);
+ if (ret)
+ goto out_free_wq;
+
+ ret = hi3110_set_normal_mode(spi);
+ if (ret)
+ goto out_free_wq;
+
+ can_led_event(net, CAN_LED_EVENT_OPEN);
+ netif_wake_queue(net);
+ mutex_unlock(&priv->hi3110_lock);
+
+ return 0;
+
+ out_free_wq:
+ destroy_workqueue(priv->wq);
+ out_free_irq:
+ free_irq(spi->irq, priv);
+ hi3110_hw_sleep(spi);
+ out_close:
+ hi3110_power_enable(priv->transceiver, 0);
+ close_candev(net);
+ mutex_unlock(&priv->hi3110_lock);
+ return ret;
+}
+
+static const struct net_device_ops hi3110_netdev_ops = {
+ .ndo_open = hi3110_open,
+ .ndo_stop = hi3110_stop,
+ .ndo_start_xmit = hi3110_hard_start_xmit,
+};
+
+static const struct of_device_id hi3110_of_match[] = {
+ {
+ .compatible = "holt,hi3110",
+ .data = (void *)CAN_HI3110_HI3110,
+ },
+ { }
+};
+MODULE_DEVICE_TABLE(of, hi3110_of_match);
+
+static const struct spi_device_id hi3110_id_table[] = {
+ {
+ .name = "hi3110",
+ .driver_data = (kernel_ulong_t)CAN_HI3110_HI3110,
+ },
+ { }
+};
+MODULE_DEVICE_TABLE(spi, hi3110_id_table);
+
+static int hi3110_can_probe(struct spi_device *spi)
+{
+ const struct of_device_id *of_id = of_match_device(hi3110_of_match,
+ &spi->dev);
+ struct net_device *net;
+ struct hi3110_priv *priv;
+ struct clk *clk;
+ int freq, ret;
+
+ clk = devm_clk_get(&spi->dev, NULL);
+ if (IS_ERR(clk)) {
+ dev_err(&spi->dev, "no CAN clock source defined\n");
+ return PTR_ERR(clk);
+ }
+ freq = clk_get_rate(clk);
+
+ /* Sanity check */
+ if (freq > 40000000)
+ return -ERANGE;
+
+ /* Allocate can/net device */
+ net = alloc_candev(sizeof(struct hi3110_priv), HI3110_TX_ECHO_SKB_MAX);
+ if (!net)
+ return -ENOMEM;
+
+ if (!IS_ERR(clk)) {
+ ret = clk_prepare_enable(clk);
+ if (ret)
+ goto out_free;
+ }
+
+ net->netdev_ops = &hi3110_netdev_ops;
+ net->flags |= IFF_ECHO;
+
+ priv = netdev_priv(net);
+ priv->can.bittiming_const = &hi3110_bittiming_const;
+ priv->can.do_set_mode = hi3110_do_set_mode;
+ priv->can.do_get_berr_counter = hi3110_get_berr_counter;
+ priv->can.clock.freq = freq / 2;
+ priv->can.ctrlmode_supported = CAN_CTRLMODE_3_SAMPLES |
+ CAN_CTRLMODE_LOOPBACK |
+ CAN_CTRLMODE_LISTENONLY |
+ CAN_CTRLMODE_BERR_REPORTING;
+
+ if (of_id)
+ priv->model = (enum hi3110_model)of_id->data;
+ else
+ priv->model = spi_get_device_id(spi)->driver_data;
+ priv->net = net;
+ priv->clk = clk;
+
+ spi_set_drvdata(spi, priv);
+
+ /* Configure the SPI bus */
+ spi->bits_per_word = 8;
+ ret = spi_setup(spi);
+ if (ret)
+ goto out_clk;
+
+ priv->power = devm_regulator_get_optional(&spi->dev, "vdd");
+ priv->transceiver = devm_regulator_get_optional(&spi->dev, "xceiver");
+ if ((PTR_ERR(priv->power) == -EPROBE_DEFER) ||
+ (PTR_ERR(priv->transceiver) == -EPROBE_DEFER)) {
+ ret = -EPROBE_DEFER;
+ goto out_clk;
+ }
+
+ ret = hi3110_power_enable(priv->power, 1);
+ if (ret)
+ goto out_clk;
+
+ priv->spi = spi;
+ mutex_init(&priv->hi3110_lock);
+
+ /* If requested, allocate DMA buffers */
+ if (hi3110_enable_dma) {
+ spi->dev.coherent_dma_mask = ~0;
+
+ /* Minimum coherent DMA allocation is PAGE_SIZE, so allocate
+ * that much and share it between Tx and Rx DMA buffers.
+ */
+ priv->spi_tx_buf = dmam_alloc_coherent(&spi->dev,
+ PAGE_SIZE,
+ &priv->spi_tx_dma,
+ GFP_DMA);
+
+ if (priv->spi_tx_buf) {
+ priv->spi_rx_buf = (priv->spi_tx_buf + (PAGE_SIZE / 2));
+ priv->spi_rx_dma = (dma_addr_t)(priv->spi_tx_dma +
+ (PAGE_SIZE / 2));
+ } else {
+ /* Fall back to non-DMA */
+ hi3110_enable_dma = 0;
+ }
+ }
+
+ /* Allocate non-DMA buffers */
+ if (!hi3110_enable_dma) {
+ priv->spi_tx_buf = devm_kzalloc(&spi->dev, HI3110_RX_BUF_LEN,
+ GFP_KERNEL);
+ if (!priv->spi_tx_buf) {
+ ret = -ENOMEM;
+ goto error_probe;
+ }
+ priv->spi_rx_buf = devm_kzalloc(&spi->dev, HI3110_RX_BUF_LEN,
+ GFP_KERNEL);
+
+ if (!priv->spi_rx_buf) {
+ ret = -ENOMEM;
+ goto error_probe;
+ }
+ }
+
+ SET_NETDEV_DEV(net, &spi->dev);
+
+ ret = hi3110_hw_probe(spi);
+ if (ret) {
+ if (ret == -ENODEV)
+ dev_err(&spi->dev, "Cannot initialize %x. Wrong wiring?\n",
+ priv->model);
+ goto error_probe;
+ }
+ hi3110_hw_sleep(spi);
+
+ ret = register_candev(net);
+ if (ret)
+ goto error_probe;
+
+ devm_can_led_init(net);
+ netdev_info(net, "%x successfully initialized.\n", priv->model);
+
+ return 0;
+
+ error_probe:
+ hi3110_power_enable(priv->power, 0);
+
+ out_clk:
+ if (!IS_ERR(clk))
+ clk_disable_unprepare(clk);
+
+ out_free:
+ free_candev(net);
+
+ dev_err(&spi->dev, "Probe failed, err=%d\n", -ret);
+ return ret;
+}
+
+static int hi3110_can_remove(struct spi_device *spi)
+{
+ struct hi3110_priv *priv = spi_get_drvdata(spi);
+ struct net_device *net = priv->net;
+
+ unregister_candev(net);
+
+ hi3110_power_enable(priv->power, 0);
+
+ if (!IS_ERR(priv->clk))
+ clk_disable_unprepare(priv->clk);
+
+ free_candev(net);
+
+ return 0;
+}
+
+static int __maybe_unused hi3110_can_suspend(struct device *dev)
+{
+ struct spi_device *spi = to_spi_device(dev);
+ struct hi3110_priv *priv = spi_get_drvdata(spi);
+ struct net_device *net = priv->net;
+
+ priv->force_quit = 1;
+ disable_irq(spi->irq);
+
+ /* Note: at this point neither IST nor workqueues are running.
+ * open/stop cannot be called anyway so locking is not needed
+ */
+ if (netif_running(net)) {
+ netif_device_detach(net);
+
+ hi3110_hw_sleep(spi);
+ hi3110_power_enable(priv->transceiver, 0);
+ priv->after_suspend = HI3110_AFTER_SUSPEND_UP;
+ } else {
+ priv->after_suspend = HI3110_AFTER_SUSPEND_DOWN;
+ }
+
+ if (!IS_ERR_OR_NULL(priv->power)) {
+ regulator_disable(priv->power);
+ priv->after_suspend |= HI3110_AFTER_SUSPEND_POWER;
+ }
+
+ return 0;
+}
+
+static int __maybe_unused hi3110_can_resume(struct device *dev)
+{
+ struct spi_device *spi = to_spi_device(dev);
+ struct hi3110_priv *priv = spi_get_drvdata(spi);
+
+ if (priv->after_suspend & HI3110_AFTER_SUSPEND_POWER)
+ hi3110_power_enable(priv->power, 1);
+
+ if (priv->after_suspend & HI3110_AFTER_SUSPEND_UP) {
+ hi3110_power_enable(priv->transceiver, 1);
+ queue_work(priv->wq, &priv->restart_work);
+ } else {
+ priv->after_suspend = 0;
+ }
+
+ priv->force_quit = 0;
+ enable_irq(spi->irq);
+ return 0;
+}
+
+static SIMPLE_DEV_PM_OPS(hi3110_can_pm_ops, hi3110_can_suspend, hi3110_can_resume);
+
+static struct spi_driver hi3110_can_driver = {
+ .driver = {
+ .name = DEVICE_NAME,
+ .of_match_table = hi3110_of_match,
+ .pm = &hi3110_can_pm_ops,
+ },
+ .id_table = hi3110_id_table,
+ .probe = hi3110_can_probe,
+ .remove = hi3110_can_remove,
+};
+
+module_spi_driver(hi3110_can_driver);
+
+MODULE_AUTHOR("Akshay Bhat <akshay.bhat@timesys.com>");
+MODULE_AUTHOR("Casey Fitzpatrick <casey.fitzpatrick@timesys.com>");
+MODULE_DESCRIPTION("Holt HI-3110 CAN driver");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/net/can/ti_hecc.c b/drivers/net/can/ti_hecc.c
index 6749b18..4d49414 100644
--- a/drivers/net/can/ti_hecc.c
+++ b/drivers/net/can/ti_hecc.c
@@ -17,25 +17,6 @@
*
*/
-/*
- * Your platform definitions should specify module ram offsets and interrupt
- * number to use as follows:
- *
- * static struct ti_hecc_platform_data am3517_evm_hecc_pdata = {
- * .scc_hecc_offset = 0,
- * .scc_ram_offset = 0x3000,
- * .hecc_ram_offset = 0x3000,
- * .mbx_offset = 0x2000,
- * .int_line = 0,
- * .revision = 1,
- * .transceiver_switch = hecc_phy_control,
- * };
- *
- * Please see include/linux/can/platform/ti_hecc.h for description of
- * above fields.
- *
- */
-
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/types.h>
@@ -46,11 +27,13 @@
#include <linux/platform_device.h>
#include <linux/clk.h>
#include <linux/io.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/regulator/consumer.h>
#include <linux/can/dev.h>
#include <linux/can/error.h>
#include <linux/can/led.h>
-#include <linux/can/platform/ti_hecc.h>
#define DRV_NAME "ti_hecc"
#define HECC_MODULE_VERSION "0.7"
@@ -214,15 +197,14 @@
struct net_device *ndev;
struct clk *clk;
void __iomem *base;
- u32 scc_ram_offset;
- u32 hecc_ram_offset;
- u32 mbx_offset;
- u32 int_line;
+ void __iomem *hecc_ram;
+ void __iomem *mbx;
+ bool use_hecc1int;
spinlock_t mbx_lock; /* CANME register needs protection */
u32 tx_head;
u32 tx_tail;
u32 rx_next;
- void (*transceiver_switch)(int);
+ struct regulator *reg_xceiver;
};
static inline int get_tx_head_mb(struct ti_hecc_priv *priv)
@@ -242,20 +224,18 @@
static inline void hecc_write_lam(struct ti_hecc_priv *priv, u32 mbxno, u32 val)
{
- __raw_writel(val, priv->base + priv->hecc_ram_offset + mbxno * 4);
+ __raw_writel(val, priv->hecc_ram + mbxno * 4);
}
static inline void hecc_write_mbx(struct ti_hecc_priv *priv, u32 mbxno,
u32 reg, u32 val)
{
- __raw_writel(val, priv->base + priv->mbx_offset + mbxno * 0x10 +
- reg);
+ __raw_writel(val, priv->mbx + mbxno * 0x10 + reg);
}
static inline u32 hecc_read_mbx(struct ti_hecc_priv *priv, u32 mbxno, u32 reg)
{
- return __raw_readl(priv->base + priv->mbx_offset + mbxno * 0x10 +
- reg);
+ return __raw_readl(priv->mbx + mbxno * 0x10 + reg);
}
static inline void hecc_write(struct ti_hecc_priv *priv, u32 reg, u32 val)
@@ -311,11 +291,16 @@
return 0;
}
-static void ti_hecc_transceiver_switch(const struct ti_hecc_priv *priv,
- int on)
+static int ti_hecc_transceiver_switch(const struct ti_hecc_priv *priv,
+ int on)
{
- if (priv->transceiver_switch)
- priv->transceiver_switch(on);
+ if (!priv->reg_xceiver)
+ return 0;
+
+ if (on)
+ return regulator_enable(priv->reg_xceiver);
+ else
+ return regulator_disable(priv->reg_xceiver);
}
static void ti_hecc_reset(struct net_device *ndev)
@@ -409,7 +394,7 @@
/* Prevent message over-write & Enable interrupts */
hecc_write(priv, HECC_CANOPC, HECC_SET_REG);
- if (priv->int_line) {
+ if (priv->use_hecc1int) {
hecc_write(priv, HECC_CANMIL, HECC_SET_REG);
hecc_write(priv, HECC_CANGIM, HECC_CANGIM_DEF_MASK |
HECC_CANGIM_I1EN | HECC_CANGIM_SIL);
@@ -760,7 +745,7 @@
unsigned long ack, flags;
int_status = hecc_read(priv,
- (priv->int_line) ? HECC_CANGIF1 : HECC_CANGIF0);
+ (priv->use_hecc1int) ? HECC_CANGIF1 : HECC_CANGIF0);
if (!int_status)
return IRQ_NONE;
@@ -806,7 +791,7 @@
}
/* clear all interrupt conditions - read back to avoid spurious ints */
- if (priv->int_line) {
+ if (priv->use_hecc1int) {
hecc_write(priv, HECC_CANGIF1, HECC_SET_REG);
int_status = hecc_read(priv, HECC_CANGIF1);
} else {
@@ -872,58 +857,87 @@
.ndo_change_mtu = can_change_mtu,
};
+static const struct of_device_id ti_hecc_dt_ids[] = {
+ {
+ .compatible = "ti,am3517-hecc",
+ },
+ { }
+};
+MODULE_DEVICE_TABLE(of, ti_hecc_dt_ids);
+
static int ti_hecc_probe(struct platform_device *pdev)
{
struct net_device *ndev = (struct net_device *)0;
struct ti_hecc_priv *priv;
- struct ti_hecc_platform_data *pdata;
- struct resource *mem, *irq;
- void __iomem *addr;
+ struct device_node *np = pdev->dev.of_node;
+ struct resource *res, *irq;
+ struct regulator *reg_xceiver;
int err = -ENODEV;
- pdata = dev_get_platdata(&pdev->dev);
- if (!pdata) {
- dev_err(&pdev->dev, "No platform data\n");
- goto probe_exit;
+ if (!IS_ENABLED(CONFIG_OF) || !np)
+ return -EINVAL;
+
+ reg_xceiver = devm_regulator_get(&pdev->dev, "xceiver");
+ if (PTR_ERR(reg_xceiver) == -EPROBE_DEFER)
+ return -EPROBE_DEFER;
+ else if (IS_ERR(reg_xceiver))
+ reg_xceiver = NULL;
+
+ ndev = alloc_candev(sizeof(struct ti_hecc_priv), HECC_MAX_TX_MBOX);
+ if (!ndev) {
+ dev_err(&pdev->dev, "alloc_candev failed\n");
+ return -ENOMEM;
+ }
+ priv = netdev_priv(ndev);
+
+ /* handle hecc memory */
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "hecc");
+ if (!res) {
+ dev_err(&pdev->dev, "can't get IORESOURCE_MEM hecc\n");
+ return -EINVAL;
}
- mem = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!mem) {
- dev_err(&pdev->dev, "No mem resources\n");
- goto probe_exit;
+ priv->base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(priv->base)) {
+ dev_err(&pdev->dev, "hecc ioremap failed\n");
+ return PTR_ERR(priv->base);
}
+
+ /* handle hecc-ram memory */
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "hecc-ram");
+ if (!res) {
+ dev_err(&pdev->dev, "can't get IORESOURCE_MEM hecc-ram\n");
+ return -EINVAL;
+ }
+
+ priv->hecc_ram = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(priv->hecc_ram)) {
+ dev_err(&pdev->dev, "hecc-ram ioremap failed\n");
+ return PTR_ERR(priv->hecc_ram);
+ }
+
+ /* handle mbx memory */
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "mbx");
+ if (!res) {
+ dev_err(&pdev->dev, "can't get IORESOURCE_MEM mbx\n");
+ return -EINVAL;
+ }
+
+ priv->mbx = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(priv->mbx)) {
+ dev_err(&pdev->dev, "mbx ioremap failed\n");
+ return PTR_ERR(priv->mbx);
+ }
+
irq = platform_get_resource(pdev, IORESOURCE_IRQ, 0);
if (!irq) {
dev_err(&pdev->dev, "No irq resource\n");
goto probe_exit;
}
- if (!request_mem_region(mem->start, resource_size(mem), pdev->name)) {
- dev_err(&pdev->dev, "HECC region already claimed\n");
- err = -EBUSY;
- goto probe_exit;
- }
- addr = ioremap(mem->start, resource_size(mem));
- if (!addr) {
- dev_err(&pdev->dev, "ioremap failed\n");
- err = -ENOMEM;
- goto probe_exit_free_region;
- }
- ndev = alloc_candev(sizeof(struct ti_hecc_priv), HECC_MAX_TX_MBOX);
- if (!ndev) {
- dev_err(&pdev->dev, "alloc_candev failed\n");
- err = -ENOMEM;
- goto probe_exit_iounmap;
- }
-
- priv = netdev_priv(ndev);
priv->ndev = ndev;
- priv->base = addr;
- priv->scc_ram_offset = pdata->scc_ram_offset;
- priv->hecc_ram_offset = pdata->hecc_ram_offset;
- priv->mbx_offset = pdata->mbx_offset;
- priv->int_line = pdata->int_line;
- priv->transceiver_switch = pdata->transceiver_switch;
+ priv->reg_xceiver = reg_xceiver;
+ priv->use_hecc1int = of_property_read_bool(np, "ti,use-hecc1int");
priv->can.bittiming_const = &ti_hecc_bittiming_const;
priv->can.do_set_mode = ti_hecc_do_set_mode;
@@ -971,32 +985,23 @@
clk_put(priv->clk);
probe_exit_candev:
free_candev(ndev);
-probe_exit_iounmap:
- iounmap(addr);
-probe_exit_free_region:
- release_mem_region(mem->start, resource_size(mem));
probe_exit:
return err;
}
static int ti_hecc_remove(struct platform_device *pdev)
{
- struct resource *res;
struct net_device *ndev = platform_get_drvdata(pdev);
struct ti_hecc_priv *priv = netdev_priv(ndev);
unregister_candev(ndev);
clk_disable_unprepare(priv->clk);
clk_put(priv->clk);
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- iounmap(priv->base);
- release_mem_region(res->start, resource_size(res));
free_candev(ndev);
return 0;
}
-
#ifdef CONFIG_PM
static int ti_hecc_suspend(struct platform_device *pdev, pm_message_t state)
{
@@ -1045,6 +1050,7 @@
static struct platform_driver ti_hecc_driver = {
.driver = {
.name = DRV_NAME,
+ .of_match_table = ti_hecc_dt_ids,
},
.probe = ti_hecc_probe,
.remove = ti_hecc_remove,
diff --git a/drivers/net/can/usb/Kconfig b/drivers/net/can/usb/Kconfig
index 8483a40..c36f4bd 100644
--- a/drivers/net/can/usb/Kconfig
+++ b/drivers/net/can/usb/Kconfig
@@ -72,6 +72,8 @@
PCAN-USB Pro dual CAN 2.0b channels USB adapter
PCAN-USB FD single CAN-FD channel USB adapter
PCAN-USB Pro FD dual CAN-FD channels USB adapter
+ PCAN-Chip USB CAN-FD to USB stamp module
+ PCAN-USB X6 6 CAN-FD channels USB adapter
(see also http://www.peak-system.com).
@@ -81,4 +83,10 @@
This driver supports the USB2CAN interface
from 8 devices (http://www.8devices.com).
+config CAN_MCBA_USB
+ tristate "Microchip CAN BUS Analyzer interface"
+ ---help---
+ This driver supports the CAN BUS Analyzer interface
+ from Microchip (http://www.microchip.com/development-tools/).
+
endmenu
diff --git a/drivers/net/can/usb/Makefile b/drivers/net/can/usb/Makefile
index a64cf98..164453f 100644
--- a/drivers/net/can/usb/Makefile
+++ b/drivers/net/can/usb/Makefile
@@ -8,3 +8,4 @@
obj-$(CONFIG_CAN_KVASER_USB) += kvaser_usb.o
obj-$(CONFIG_CAN_PEAK_USB) += peak_usb/
obj-$(CONFIG_CAN_8DEV_USB) += usb_8dev.o
+obj-$(CONFIG_CAN_MCBA_USB) += mcba_usb.o
diff --git a/drivers/net/can/usb/gs_usb.c b/drivers/net/can/usb/gs_usb.c
index 300349f..eecee7f 100644
--- a/drivers/net/can/usb/gs_usb.c
+++ b/drivers/net/can/usb/gs_usb.c
@@ -739,13 +739,18 @@
static int gs_usb_set_identify(struct net_device *netdev, bool do_identify)
{
struct gs_can *dev = netdev_priv(netdev);
- struct gs_identify_mode imode;
+ struct gs_identify_mode *imode;
int rc;
+ imode = kmalloc(sizeof(*imode), GFP_KERNEL);
+
+ if (!imode)
+ return -ENOMEM;
+
if (do_identify)
- imode.mode = GS_CAN_IDENTIFY_ON;
+ imode->mode = GS_CAN_IDENTIFY_ON;
else
- imode.mode = GS_CAN_IDENTIFY_OFF;
+ imode->mode = GS_CAN_IDENTIFY_OFF;
rc = usb_control_msg(interface_to_usbdev(dev->iface),
usb_sndctrlpipe(interface_to_usbdev(dev->iface),
@@ -755,10 +760,12 @@
USB_RECIP_INTERFACE,
dev->channel,
0,
- &imode,
- sizeof(imode),
+ imode,
+ sizeof(*imode),
100);
+ kfree(imode);
+
return (rc > 0) ? 0 : rc;
}
diff --git a/drivers/net/can/usb/mcba_usb.c b/drivers/net/can/usb/mcba_usb.c
new file mode 100644
index 0000000..7f02725
--- /dev/null
+++ b/drivers/net/can/usb/mcba_usb.c
@@ -0,0 +1,904 @@
+/* SocketCAN driver for Microchip CAN BUS Analyzer Tool
+ *
+ * Copyright (C) 2017 Mobica Limited
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published
+ * by the Free Software Foundation; version 2 of the License.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program.
+ *
+ * This driver is inspired by the 4.6.2 version of net/can/usb/usb_8dev.c
+ */
+
+#include <asm/unaligned.h>
+#include <linux/can.h>
+#include <linux/can/dev.h>
+#include <linux/can/error.h>
+#include <linux/can/led.h>
+#include <linux/module.h>
+#include <linux/netdevice.h>
+#include <linux/signal.h>
+#include <linux/slab.h>
+#include <linux/usb.h>
+
+/* vendor and product id */
+#define MCBA_MODULE_NAME "mcba_usb"
+#define MCBA_VENDOR_ID 0x04d8
+#define MCBA_PRODUCT_ID 0x0a30
+
+/* driver constants */
+#define MCBA_MAX_RX_URBS 20
+#define MCBA_MAX_TX_URBS 20
+#define MCBA_CTX_FREE MCBA_MAX_TX_URBS
+
+/* RX buffer must be bigger than msg size since at the
+ * beggining USB messages are stacked.
+ */
+#define MCBA_USB_RX_BUFF_SIZE 64
+#define MCBA_USB_TX_BUFF_SIZE (sizeof(struct mcba_usb_msg))
+
+/* MCBA endpoint numbers */
+#define MCBA_USB_EP_IN 1
+#define MCBA_USB_EP_OUT 1
+
+/* Microchip command id */
+#define MBCA_CMD_RECEIVE_MESSAGE 0xE3
+#define MBCA_CMD_I_AM_ALIVE_FROM_CAN 0xF5
+#define MBCA_CMD_I_AM_ALIVE_FROM_USB 0xF7
+#define MBCA_CMD_CHANGE_BIT_RATE 0xA1
+#define MBCA_CMD_TRANSMIT_MESSAGE_EV 0xA3
+#define MBCA_CMD_SETUP_TERMINATION_RESISTANCE 0xA8
+#define MBCA_CMD_READ_FW_VERSION 0xA9
+#define MBCA_CMD_NOTHING_TO_SEND 0xFF
+#define MBCA_CMD_TRANSMIT_MESSAGE_RSP 0xE2
+
+#define MCBA_VER_REQ_USB 1
+#define MCBA_VER_REQ_CAN 2
+
+#define MCBA_SIDL_EXID_MASK 0x8
+#define MCBA_DLC_MASK 0xf
+#define MCBA_DLC_RTR_MASK 0x40
+
+#define MCBA_CAN_STATE_WRN_TH 95
+#define MCBA_CAN_STATE_ERR_PSV_TH 127
+
+#define MCBA_TERMINATION_DISABLED CAN_TERMINATION_DISABLED
+#define MCBA_TERMINATION_ENABLED 120
+
+struct mcba_usb_ctx {
+ struct mcba_priv *priv;
+ u32 ndx;
+ u8 dlc;
+ bool can;
+};
+
+/* Structure to hold all of our device specific stuff */
+struct mcba_priv {
+ struct can_priv can; /* must be the first member */
+ struct sk_buff *echo_skb[MCBA_MAX_TX_URBS];
+ struct mcba_usb_ctx tx_context[MCBA_MAX_TX_URBS];
+ struct usb_device *udev;
+ struct net_device *netdev;
+ struct usb_anchor tx_submitted;
+ struct usb_anchor rx_submitted;
+ struct can_berr_counter bec;
+ bool usb_ka_first_pass;
+ bool can_ka_first_pass;
+ bool can_speed_check;
+ atomic_t free_ctx_cnt;
+};
+
+/* CAN frame */
+struct __packed mcba_usb_msg_can {
+ u8 cmd_id;
+ __be16 eid;
+ __be16 sid;
+ u8 dlc;
+ u8 data[8];
+ u8 timestamp[4];
+ u8 checksum;
+};
+
+/* command frame */
+struct __packed mcba_usb_msg {
+ u8 cmd_id;
+ u8 unused[18];
+};
+
+struct __packed mcba_usb_msg_ka_usb {
+ u8 cmd_id;
+ u8 termination_state;
+ u8 soft_ver_major;
+ u8 soft_ver_minor;
+ u8 unused[15];
+};
+
+struct __packed mcba_usb_msg_ka_can {
+ u8 cmd_id;
+ u8 tx_err_cnt;
+ u8 rx_err_cnt;
+ u8 rx_buff_ovfl;
+ u8 tx_bus_off;
+ __be16 can_bitrate;
+ __le16 rx_lost;
+ u8 can_stat;
+ u8 soft_ver_major;
+ u8 soft_ver_minor;
+ u8 debug_mode;
+ u8 test_complete;
+ u8 test_result;
+ u8 unused[4];
+};
+
+struct __packed mcba_usb_msg_change_bitrate {
+ u8 cmd_id;
+ __be16 bitrate;
+ u8 unused[16];
+};
+
+struct __packed mcba_usb_msg_termination {
+ u8 cmd_id;
+ u8 termination;
+ u8 unused[17];
+};
+
+struct __packed mcba_usb_msg_fw_ver {
+ u8 cmd_id;
+ u8 pic;
+ u8 unused[17];
+};
+
+static const struct usb_device_id mcba_usb_table[] = {
+ { USB_DEVICE(MCBA_VENDOR_ID, MCBA_PRODUCT_ID) },
+ {} /* Terminating entry */
+};
+
+MODULE_DEVICE_TABLE(usb, mcba_usb_table);
+
+static const u16 mcba_termination[] = { MCBA_TERMINATION_DISABLED,
+ MCBA_TERMINATION_ENABLED };
+
+static const u32 mcba_bitrate[] = { 20000, 33333, 50000, 80000, 83333,
+ 100000, 125000, 150000, 175000, 200000,
+ 225000, 250000, 275000, 300000, 500000,
+ 625000, 800000, 1000000 };
+
+static inline void mcba_init_ctx(struct mcba_priv *priv)
+{
+ int i = 0;
+
+ for (i = 0; i < MCBA_MAX_TX_URBS; i++) {
+ priv->tx_context[i].ndx = MCBA_CTX_FREE;
+ priv->tx_context[i].priv = priv;
+ }
+
+ atomic_set(&priv->free_ctx_cnt, ARRAY_SIZE(priv->tx_context));
+}
+
+static inline struct mcba_usb_ctx *mcba_usb_get_free_ctx(struct mcba_priv *priv,
+ struct can_frame *cf)
+{
+ int i = 0;
+ struct mcba_usb_ctx *ctx = NULL;
+
+ for (i = 0; i < MCBA_MAX_TX_URBS; i++) {
+ if (priv->tx_context[i].ndx == MCBA_CTX_FREE) {
+ ctx = &priv->tx_context[i];
+ ctx->ndx = i;
+
+ if (cf) {
+ ctx->can = true;
+ ctx->dlc = cf->can_dlc;
+ } else {
+ ctx->can = false;
+ ctx->dlc = 0;
+ }
+
+ atomic_dec(&priv->free_ctx_cnt);
+ break;
+ }
+ }
+
+ if (!atomic_read(&priv->free_ctx_cnt))
+ /* That was the last free ctx. Slow down tx path */
+ netif_stop_queue(priv->netdev);
+
+ return ctx;
+}
+
+/* mcba_usb_free_ctx and mcba_usb_get_free_ctx are executed by different
+ * threads. The order of execution in below function is important.
+ */
+static inline void mcba_usb_free_ctx(struct mcba_usb_ctx *ctx)
+{
+ /* Increase number of free ctxs before freeing ctx */
+ atomic_inc(&ctx->priv->free_ctx_cnt);
+
+ ctx->ndx = MCBA_CTX_FREE;
+
+ /* Wake up the queue once ctx is marked free */
+ netif_wake_queue(ctx->priv->netdev);
+}
+
+static void mcba_usb_write_bulk_callback(struct urb *urb)
+{
+ struct mcba_usb_ctx *ctx = urb->context;
+ struct net_device *netdev;
+
+ WARN_ON(!ctx);
+
+ netdev = ctx->priv->netdev;
+
+ /* free up our allocated buffer */
+ usb_free_coherent(urb->dev, urb->transfer_buffer_length,
+ urb->transfer_buffer, urb->transfer_dma);
+
+ if (ctx->can) {
+ if (!netif_device_present(netdev))
+ return;
+
+ netdev->stats.tx_packets++;
+ netdev->stats.tx_bytes += ctx->dlc;
+
+ can_led_event(netdev, CAN_LED_EVENT_TX);
+ can_get_echo_skb(netdev, ctx->ndx);
+ }
+
+ if (urb->status)
+ netdev_info(netdev, "Tx URB aborted (%d)\n", urb->status);
+
+ /* Release the context */
+ mcba_usb_free_ctx(ctx);
+}
+
+/* Send data to device */
+static netdev_tx_t mcba_usb_xmit(struct mcba_priv *priv,
+ struct mcba_usb_msg *usb_msg,
+ struct mcba_usb_ctx *ctx)
+{
+ struct urb *urb;
+ u8 *buf;
+ int err;
+
+ /* create a URB, and a buffer for it, and copy the data to the URB */
+ urb = usb_alloc_urb(0, GFP_ATOMIC);
+ if (!urb)
+ return -ENOMEM;
+
+ buf = usb_alloc_coherent(priv->udev, MCBA_USB_TX_BUFF_SIZE, GFP_ATOMIC,
+ &urb->transfer_dma);
+ if (!buf) {
+ err = -ENOMEM;
+ goto nomembuf;
+ }
+
+ memcpy(buf, usb_msg, MCBA_USB_TX_BUFF_SIZE);
+
+ usb_fill_bulk_urb(urb, priv->udev,
+ usb_sndbulkpipe(priv->udev, MCBA_USB_EP_OUT), buf,
+ MCBA_USB_TX_BUFF_SIZE, mcba_usb_write_bulk_callback,
+ ctx);
+
+ urb->transfer_flags |= URB_NO_TRANSFER_DMA_MAP;
+ usb_anchor_urb(urb, &priv->tx_submitted);
+
+ err = usb_submit_urb(urb, GFP_ATOMIC);
+ if (unlikely(err))
+ goto failed;
+
+ /* Release our reference to this URB, the USB core will eventually free
+ * it entirely.
+ */
+ usb_free_urb(urb);
+
+ return 0;
+
+failed:
+ usb_unanchor_urb(urb);
+ usb_free_coherent(priv->udev, MCBA_USB_TX_BUFF_SIZE, buf,
+ urb->transfer_dma);
+
+ if (err == -ENODEV)
+ netif_device_detach(priv->netdev);
+ else
+ netdev_warn(priv->netdev, "failed tx_urb %d\n", err);
+
+nomembuf:
+ usb_free_urb(urb);
+
+ return err;
+}
+
+/* Send data to device */
+static netdev_tx_t mcba_usb_start_xmit(struct sk_buff *skb,
+ struct net_device *netdev)
+{
+ struct mcba_priv *priv = netdev_priv(netdev);
+ struct can_frame *cf = (struct can_frame *)skb->data;
+ struct mcba_usb_ctx *ctx = NULL;
+ struct net_device_stats *stats = &priv->netdev->stats;
+ u16 sid;
+ int err;
+ struct mcba_usb_msg_can usb_msg = {
+ .cmd_id = MBCA_CMD_TRANSMIT_MESSAGE_EV
+ };
+
+ if (can_dropped_invalid_skb(netdev, skb))
+ return NETDEV_TX_OK;
+
+ ctx = mcba_usb_get_free_ctx(priv, cf);
+ if (!ctx)
+ return NETDEV_TX_BUSY;
+
+ can_put_echo_skb(skb, priv->netdev, ctx->ndx);
+
+ if (cf->can_id & CAN_EFF_FLAG) {
+ /* SIDH | SIDL | EIDH | EIDL
+ * 28 - 21 | 20 19 18 x x x 17 16 | 15 - 8 | 7 - 0
+ */
+ sid = MCBA_SIDL_EXID_MASK;
+ /* store 28-18 bits */
+ sid |= (cf->can_id & 0x1ffc0000) >> 13;
+ /* store 17-16 bits */
+ sid |= (cf->can_id & 0x30000) >> 16;
+ put_unaligned_be16(sid, &usb_msg.sid);
+
+ /* store 15-0 bits */
+ put_unaligned_be16(cf->can_id & 0xffff, &usb_msg.eid);
+ } else {
+ /* SIDH | SIDL
+ * 10 - 3 | 2 1 0 x x x x x
+ */
+ put_unaligned_be16((cf->can_id & CAN_SFF_MASK) << 5,
+ &usb_msg.sid);
+ usb_msg.eid = 0;
+ }
+
+ usb_msg.dlc = cf->can_dlc;
+
+ memcpy(usb_msg.data, cf->data, usb_msg.dlc);
+
+ if (cf->can_id & CAN_RTR_FLAG)
+ usb_msg.dlc |= MCBA_DLC_RTR_MASK;
+
+ err = mcba_usb_xmit(priv, (struct mcba_usb_msg *)&usb_msg, ctx);
+ if (err)
+ goto xmit_failed;
+
+ return NETDEV_TX_OK;
+
+xmit_failed:
+ can_free_echo_skb(priv->netdev, ctx->ndx);
+ mcba_usb_free_ctx(ctx);
+ dev_kfree_skb(skb);
+ stats->tx_dropped++;
+
+ return NETDEV_TX_OK;
+}
+
+/* Send cmd to device */
+static void mcba_usb_xmit_cmd(struct mcba_priv *priv,
+ struct mcba_usb_msg *usb_msg)
+{
+ struct mcba_usb_ctx *ctx = NULL;
+ int err;
+
+ ctx = mcba_usb_get_free_ctx(priv, NULL);
+ if (!ctx) {
+ netdev_err(priv->netdev,
+ "Lack of free ctx. Sending (%d) cmd aborted",
+ usb_msg->cmd_id);
+
+ return;
+ }
+
+ err = mcba_usb_xmit(priv, usb_msg, ctx);
+ if (err)
+ netdev_err(priv->netdev, "Failed to send cmd (%d)",
+ usb_msg->cmd_id);
+}
+
+static void mcba_usb_xmit_change_bitrate(struct mcba_priv *priv, u16 bitrate)
+{
+ struct mcba_usb_msg_change_bitrate usb_msg = {
+ .cmd_id = MBCA_CMD_CHANGE_BIT_RATE
+ };
+
+ put_unaligned_be16(bitrate, &usb_msg.bitrate);
+
+ mcba_usb_xmit_cmd(priv, (struct mcba_usb_msg *)&usb_msg);
+}
+
+static void mcba_usb_xmit_read_fw_ver(struct mcba_priv *priv, u8 pic)
+{
+ struct mcba_usb_msg_fw_ver usb_msg = {
+ .cmd_id = MBCA_CMD_READ_FW_VERSION,
+ .pic = pic
+ };
+
+ mcba_usb_xmit_cmd(priv, (struct mcba_usb_msg *)&usb_msg);
+}
+
+static void mcba_usb_process_can(struct mcba_priv *priv,
+ struct mcba_usb_msg_can *msg)
+{
+ struct can_frame *cf;
+ struct sk_buff *skb;
+ struct net_device_stats *stats = &priv->netdev->stats;
+ u16 sid;
+
+ skb = alloc_can_skb(priv->netdev, &cf);
+ if (!skb)
+ return;
+
+ sid = get_unaligned_be16(&msg->sid);
+
+ if (sid & MCBA_SIDL_EXID_MASK) {
+ /* SIDH | SIDL | EIDH | EIDL
+ * 28 - 21 | 20 19 18 x x x 17 16 | 15 - 8 | 7 - 0
+ */
+ cf->can_id = CAN_EFF_FLAG;
+
+ /* store 28-18 bits */
+ cf->can_id |= (sid & 0xffe0) << 13;
+ /* store 17-16 bits */
+ cf->can_id |= (sid & 3) << 16;
+ /* store 15-0 bits */
+ cf->can_id |= get_unaligned_be16(&msg->eid);
+ } else {
+ /* SIDH | SIDL
+ * 10 - 3 | 2 1 0 x x x x x
+ */
+ cf->can_id = (sid & 0xffe0) >> 5;
+ }
+
+ if (msg->dlc & MCBA_DLC_RTR_MASK)
+ cf->can_id |= CAN_RTR_FLAG;
+
+ cf->can_dlc = get_can_dlc(msg->dlc & MCBA_DLC_MASK);
+
+ memcpy(cf->data, msg->data, cf->can_dlc);
+
+ stats->rx_packets++;
+ stats->rx_bytes += cf->can_dlc;
+
+ can_led_event(priv->netdev, CAN_LED_EVENT_RX);
+ netif_rx(skb);
+}
+
+static void mcba_usb_process_ka_usb(struct mcba_priv *priv,
+ struct mcba_usb_msg_ka_usb *msg)
+{
+ if (unlikely(priv->usb_ka_first_pass)) {
+ netdev_info(priv->netdev, "PIC USB version %hhu.%hhu\n",
+ msg->soft_ver_major, msg->soft_ver_minor);
+
+ priv->usb_ka_first_pass = false;
+ }
+
+ if (msg->termination_state)
+ priv->can.termination = MCBA_TERMINATION_ENABLED;
+ else
+ priv->can.termination = MCBA_TERMINATION_DISABLED;
+}
+
+static u32 convert_can2host_bitrate(struct mcba_usb_msg_ka_can *msg)
+{
+ const u32 bitrate = get_unaligned_be16(&msg->can_bitrate);
+
+ if ((bitrate == 33) || (bitrate == 83))
+ return bitrate * 1000 + 333;
+ else
+ return bitrate * 1000;
+}
+
+static void mcba_usb_process_ka_can(struct mcba_priv *priv,
+ struct mcba_usb_msg_ka_can *msg)
+{
+ if (unlikely(priv->can_ka_first_pass)) {
+ netdev_info(priv->netdev, "PIC CAN version %hhu.%hhu\n",
+ msg->soft_ver_major, msg->soft_ver_minor);
+
+ priv->can_ka_first_pass = false;
+ }
+
+ if (unlikely(priv->can_speed_check)) {
+ const u32 bitrate = convert_can2host_bitrate(msg);
+
+ priv->can_speed_check = false;
+
+ if (bitrate != priv->can.bittiming.bitrate)
+ netdev_err(
+ priv->netdev,
+ "Wrong bitrate reported by the device (%u). Expected %u",
+ bitrate, priv->can.bittiming.bitrate);
+ }
+
+ priv->bec.txerr = msg->tx_err_cnt;
+ priv->bec.rxerr = msg->rx_err_cnt;
+
+ if (msg->tx_bus_off)
+ priv->can.state = CAN_STATE_BUS_OFF;
+
+ else if ((priv->bec.txerr > MCBA_CAN_STATE_ERR_PSV_TH) ||
+ (priv->bec.rxerr > MCBA_CAN_STATE_ERR_PSV_TH))
+ priv->can.state = CAN_STATE_ERROR_PASSIVE;
+
+ else if ((priv->bec.txerr > MCBA_CAN_STATE_WRN_TH) ||
+ (priv->bec.rxerr > MCBA_CAN_STATE_WRN_TH))
+ priv->can.state = CAN_STATE_ERROR_WARNING;
+}
+
+static void mcba_usb_process_rx(struct mcba_priv *priv,
+ struct mcba_usb_msg *msg)
+{
+ switch (msg->cmd_id) {
+ case MBCA_CMD_I_AM_ALIVE_FROM_CAN:
+ mcba_usb_process_ka_can(priv,
+ (struct mcba_usb_msg_ka_can *)msg);
+ break;
+
+ case MBCA_CMD_I_AM_ALIVE_FROM_USB:
+ mcba_usb_process_ka_usb(priv,
+ (struct mcba_usb_msg_ka_usb *)msg);
+ break;
+
+ case MBCA_CMD_RECEIVE_MESSAGE:
+ mcba_usb_process_can(priv, (struct mcba_usb_msg_can *)msg);
+ break;
+
+ case MBCA_CMD_NOTHING_TO_SEND:
+ /* Side effect of communication between PIC_USB and PIC_CAN.
+ * PIC_CAN is telling us that it has nothing to send
+ */
+ break;
+
+ case MBCA_CMD_TRANSMIT_MESSAGE_RSP:
+ /* Transmission response from the device containing timestamp */
+ break;
+
+ default:
+ netdev_warn(priv->netdev, "Unsupported msg (0x%hhX)",
+ msg->cmd_id);
+ break;
+ }
+}
+
+/* Callback for reading data from device
+ *
+ * Check urb status, call read function and resubmit urb read operation.
+ */
+static void mcba_usb_read_bulk_callback(struct urb *urb)
+{
+ struct mcba_priv *priv = urb->context;
+ struct net_device *netdev;
+ int retval;
+ int pos = 0;
+
+ netdev = priv->netdev;
+
+ if (!netif_device_present(netdev))
+ return;
+
+ switch (urb->status) {
+ case 0: /* success */
+ break;
+
+ case -ENOENT:
+ case -ESHUTDOWN:
+ return;
+
+ default:
+ netdev_info(netdev, "Rx URB aborted (%d)\n", urb->status);
+
+ goto resubmit_urb;
+ }
+
+ while (pos < urb->actual_length) {
+ struct mcba_usb_msg *msg;
+
+ if (pos + sizeof(struct mcba_usb_msg) > urb->actual_length) {
+ netdev_err(priv->netdev, "format error\n");
+ break;
+ }
+
+ msg = (struct mcba_usb_msg *)(urb->transfer_buffer + pos);
+ mcba_usb_process_rx(priv, msg);
+
+ pos += sizeof(struct mcba_usb_msg);
+ }
+
+resubmit_urb:
+
+ usb_fill_bulk_urb(urb, priv->udev,
+ usb_rcvbulkpipe(priv->udev, MCBA_USB_EP_OUT),
+ urb->transfer_buffer, MCBA_USB_RX_BUFF_SIZE,
+ mcba_usb_read_bulk_callback, priv);
+
+ retval = usb_submit_urb(urb, GFP_ATOMIC);
+
+ if (retval == -ENODEV)
+ netif_device_detach(netdev);
+ else if (retval)
+ netdev_err(netdev, "failed resubmitting read bulk urb: %d\n",
+ retval);
+}
+
+/* Start USB device */
+static int mcba_usb_start(struct mcba_priv *priv)
+{
+ struct net_device *netdev = priv->netdev;
+ int err, i;
+
+ mcba_init_ctx(priv);
+
+ for (i = 0; i < MCBA_MAX_RX_URBS; i++) {
+ struct urb *urb = NULL;
+ u8 *buf;
+
+ /* create a URB, and a buffer for it */
+ urb = usb_alloc_urb(0, GFP_KERNEL);
+ if (!urb) {
+ err = -ENOMEM;
+ break;
+ }
+
+ buf = usb_alloc_coherent(priv->udev, MCBA_USB_RX_BUFF_SIZE,
+ GFP_KERNEL, &urb->transfer_dma);
+ if (!buf) {
+ netdev_err(netdev, "No memory left for USB buffer\n");
+ usb_free_urb(urb);
+ err = -ENOMEM;
+ break;
+ }
+
+ usb_fill_bulk_urb(urb, priv->udev,
+ usb_rcvbulkpipe(priv->udev, MCBA_USB_EP_IN),
+ buf, MCBA_USB_RX_BUFF_SIZE,
+ mcba_usb_read_bulk_callback, priv);
+ urb->transfer_flags |= URB_NO_TRANSFER_DMA_MAP;
+ usb_anchor_urb(urb, &priv->rx_submitted);
+
+ err = usb_submit_urb(urb, GFP_KERNEL);
+ if (err) {
+ usb_unanchor_urb(urb);
+ usb_free_coherent(priv->udev, MCBA_USB_RX_BUFF_SIZE,
+ buf, urb->transfer_dma);
+ usb_free_urb(urb);
+ break;
+ }
+
+ /* Drop reference, USB core will take care of freeing it */
+ usb_free_urb(urb);
+ }
+
+ /* Did we submit any URBs */
+ if (i == 0) {
+ netdev_warn(netdev, "couldn't setup read URBs\n");
+ return err;
+ }
+
+ /* Warn if we've couldn't transmit all the URBs */
+ if (i < MCBA_MAX_RX_URBS)
+ netdev_warn(netdev, "rx performance may be slow\n");
+
+ mcba_usb_xmit_read_fw_ver(priv, MCBA_VER_REQ_USB);
+ mcba_usb_xmit_read_fw_ver(priv, MCBA_VER_REQ_CAN);
+
+ return err;
+}
+
+/* Open USB device */
+static int mcba_usb_open(struct net_device *netdev)
+{
+ struct mcba_priv *priv = netdev_priv(netdev);
+ int err;
+
+ /* common open */
+ err = open_candev(netdev);
+ if (err)
+ return err;
+
+ priv->can_speed_check = true;
+ priv->can.state = CAN_STATE_ERROR_ACTIVE;
+
+ can_led_event(netdev, CAN_LED_EVENT_OPEN);
+ netif_start_queue(netdev);
+
+ return 0;
+}
+
+static void mcba_urb_unlink(struct mcba_priv *priv)
+{
+ usb_kill_anchored_urbs(&priv->rx_submitted);
+ usb_kill_anchored_urbs(&priv->tx_submitted);
+}
+
+/* Close USB device */
+static int mcba_usb_close(struct net_device *netdev)
+{
+ struct mcba_priv *priv = netdev_priv(netdev);
+
+ priv->can.state = CAN_STATE_STOPPED;
+
+ netif_stop_queue(netdev);
+
+ /* Stop polling */
+ mcba_urb_unlink(priv);
+
+ close_candev(netdev);
+ can_led_event(netdev, CAN_LED_EVENT_STOP);
+
+ return 0;
+}
+
+/* Set network device mode
+ *
+ * Maybe we should leave this function empty, because the device
+ * set mode variable with open command.
+ */
+static int mcba_net_set_mode(struct net_device *netdev, enum can_mode mode)
+{
+ return 0;
+}
+
+static int mcba_net_get_berr_counter(const struct net_device *netdev,
+ struct can_berr_counter *bec)
+{
+ struct mcba_priv *priv = netdev_priv(netdev);
+
+ bec->txerr = priv->bec.txerr;
+ bec->rxerr = priv->bec.rxerr;
+
+ return 0;
+}
+
+static const struct net_device_ops mcba_netdev_ops = {
+ .ndo_open = mcba_usb_open,
+ .ndo_stop = mcba_usb_close,
+ .ndo_start_xmit = mcba_usb_start_xmit,
+};
+
+/* Microchip CANBUS has hardcoded bittiming values by default.
+ * This function sends request via USB to change the speed and align bittiming
+ * values for presentation purposes only
+ */
+static int mcba_net_set_bittiming(struct net_device *netdev)
+{
+ struct mcba_priv *priv = netdev_priv(netdev);
+ const u16 bitrate_kbps = priv->can.bittiming.bitrate / 1000;
+
+ mcba_usb_xmit_change_bitrate(priv, bitrate_kbps);
+
+ return 0;
+}
+
+static int mcba_set_termination(struct net_device *netdev, u16 term)
+{
+ struct mcba_priv *priv = netdev_priv(netdev);
+ struct mcba_usb_msg_termination usb_msg = {
+ .cmd_id = MBCA_CMD_SETUP_TERMINATION_RESISTANCE
+ };
+
+ if (term == MCBA_TERMINATION_ENABLED)
+ usb_msg.termination = 1;
+ else
+ usb_msg.termination = 0;
+
+ mcba_usb_xmit_cmd(priv, (struct mcba_usb_msg *)&usb_msg);
+
+ return 0;
+}
+
+static int mcba_usb_probe(struct usb_interface *intf,
+ const struct usb_device_id *id)
+{
+ struct net_device *netdev;
+ struct mcba_priv *priv;
+ int err = -ENOMEM;
+ struct usb_device *usbdev = interface_to_usbdev(intf);
+
+ netdev = alloc_candev(sizeof(struct mcba_priv), MCBA_MAX_TX_URBS);
+ if (!netdev) {
+ dev_err(&intf->dev, "Couldn't alloc candev\n");
+ return -ENOMEM;
+ }
+
+ priv = netdev_priv(netdev);
+
+ priv->udev = usbdev;
+ priv->netdev = netdev;
+ priv->usb_ka_first_pass = true;
+ priv->can_ka_first_pass = true;
+ priv->can_speed_check = false;
+
+ init_usb_anchor(&priv->rx_submitted);
+ init_usb_anchor(&priv->tx_submitted);
+
+ usb_set_intfdata(intf, priv);
+
+ /* Init CAN device */
+ priv->can.state = CAN_STATE_STOPPED;
+ priv->can.termination_const = mcba_termination;
+ priv->can.termination_const_cnt = ARRAY_SIZE(mcba_termination);
+ priv->can.bitrate_const = mcba_bitrate;
+ priv->can.bitrate_const_cnt = ARRAY_SIZE(mcba_bitrate);
+
+ priv->can.do_set_termination = mcba_set_termination;
+ priv->can.do_set_mode = mcba_net_set_mode;
+ priv->can.do_get_berr_counter = mcba_net_get_berr_counter;
+ priv->can.do_set_bittiming = mcba_net_set_bittiming;
+
+ netdev->netdev_ops = &mcba_netdev_ops;
+
+ netdev->flags |= IFF_ECHO; /* we support local echo */
+
+ SET_NETDEV_DEV(netdev, &intf->dev);
+
+ err = register_candev(netdev);
+ if (err) {
+ netdev_err(netdev, "couldn't register CAN device: %d\n", err);
+
+ goto cleanup_free_candev;
+ }
+
+ devm_can_led_init(netdev);
+
+ /* Start USB dev only if we have successfully registered CAN device */
+ err = mcba_usb_start(priv);
+ if (err) {
+ if (err == -ENODEV)
+ netif_device_detach(priv->netdev);
+
+ netdev_warn(netdev, "couldn't start device: %d\n", err);
+
+ goto cleanup_unregister_candev;
+ }
+
+ dev_info(&intf->dev, "Microchip CAN BUS analizer connected\n");
+
+ return 0;
+
+cleanup_unregister_candev:
+ unregister_candev(priv->netdev);
+
+cleanup_free_candev:
+ free_candev(netdev);
+
+ return err;
+}
+
+/* Called by the usb core when driver is unloaded or device is removed */
+static void mcba_usb_disconnect(struct usb_interface *intf)
+{
+ struct mcba_priv *priv = usb_get_intfdata(intf);
+
+ usb_set_intfdata(intf, NULL);
+
+ netdev_info(priv->netdev, "device disconnected\n");
+
+ unregister_candev(priv->netdev);
+ free_candev(priv->netdev);
+
+ mcba_urb_unlink(priv);
+}
+
+static struct usb_driver mcba_usb_driver = {
+ .name = MCBA_MODULE_NAME,
+ .probe = mcba_usb_probe,
+ .disconnect = mcba_usb_disconnect,
+ .id_table = mcba_usb_table,
+};
+
+module_usb_driver(mcba_usb_driver);
+
+MODULE_AUTHOR("Remigiusz Kołłątaj <remigiusz.kollataj@mobica.com>");
+MODULE_DESCRIPTION("SocketCAN driver for Microchip CAN BUS Analyzer Tool");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/net/can/usb/peak_usb/pcan_usb_core.c b/drivers/net/can/usb/peak_usb/pcan_usb_core.c
index 0b0302a..57913db 100644
--- a/drivers/net/can/usb/peak_usb/pcan_usb_core.c
+++ b/drivers/net/can/usb/peak_usb/pcan_usb_core.c
@@ -39,6 +39,7 @@
{USB_DEVICE(PCAN_USB_VENDOR_ID, PCAN_USBPRO_PRODUCT_ID)},
{USB_DEVICE(PCAN_USB_VENDOR_ID, PCAN_USBFD_PRODUCT_ID)},
{USB_DEVICE(PCAN_USB_VENDOR_ID, PCAN_USBPROFD_PRODUCT_ID)},
+ {USB_DEVICE(PCAN_USB_VENDOR_ID, PCAN_USBCHIP_PRODUCT_ID)},
{USB_DEVICE(PCAN_USB_VENDOR_ID, PCAN_USBX6_PRODUCT_ID)},
{} /* Terminating entry */
};
@@ -51,6 +52,7 @@
&pcan_usb_pro,
&pcan_usb_fd,
&pcan_usb_pro_fd,
+ &pcan_usb_chip,
&pcan_usb_x6,
};
diff --git a/drivers/net/can/usb/peak_usb/pcan_usb_core.h b/drivers/net/can/usb/peak_usb/pcan_usb_core.h
index 3cbfb06..c01316ca 100644
--- a/drivers/net/can/usb/peak_usb/pcan_usb_core.h
+++ b/drivers/net/can/usb/peak_usb/pcan_usb_core.h
@@ -27,6 +27,7 @@
#define PCAN_USBPRO_PRODUCT_ID 0x000d
#define PCAN_USBPROFD_PRODUCT_ID 0x0011
#define PCAN_USBFD_PRODUCT_ID 0x0012
+#define PCAN_USBCHIP_PRODUCT_ID 0x0013
#define PCAN_USBX6_PRODUCT_ID 0x0014
#define PCAN_USB_DRIVER_NAME "peak_usb"
@@ -90,6 +91,7 @@
extern const struct peak_usb_adapter pcan_usb;
extern const struct peak_usb_adapter pcan_usb_pro;
extern const struct peak_usb_adapter pcan_usb_fd;
+extern const struct peak_usb_adapter pcan_usb_chip;
extern const struct peak_usb_adapter pcan_usb_pro_fd;
extern const struct peak_usb_adapter pcan_usb_x6;
diff --git a/drivers/net/can/usb/peak_usb/pcan_usb_fd.c b/drivers/net/can/usb/peak_usb/pcan_usb_fd.c
index 3047325..7ccdc3e 100644
--- a/drivers/net/can/usb/peak_usb/pcan_usb_fd.c
+++ b/drivers/net/can/usb/peak_usb/pcan_usb_fd.c
@@ -19,10 +19,10 @@
#include <linux/can.h>
#include <linux/can/dev.h>
#include <linux/can/error.h>
+#include <linux/can/dev/peak_canfd.h>
#include "pcan_usb_core.h"
#include "pcan_usb_pro.h"
-#include "pcan_ucan.h"
MODULE_SUPPORTED_DEVICE("PEAK-System PCAN-USB FD adapter");
MODULE_SUPPORTED_DEVICE("PEAK-System PCAN-USB Pro FD adapter");
@@ -238,7 +238,7 @@
/* 1st, reset error counters: */
prc = (struct pucan_wr_err_cnt *)pc;
- prc->opcode_channel = pucan_cmd_opcode_channel(dev,
+ prc->opcode_channel = pucan_cmd_opcode_channel(dev->ctrl_idx,
PUCAN_CMD_WR_ERR_CNT);
/* select both counters */
@@ -257,9 +257,10 @@
puo->opcode_channel =
(dev->can.ctrlmode & CAN_CTRLMODE_FD_NON_ISO) ?
- pucan_cmd_opcode_channel(dev,
+ pucan_cmd_opcode_channel(dev->ctrl_idx,
PUCAN_CMD_CLR_DIS_OPTION) :
- pucan_cmd_opcode_channel(dev, PUCAN_CMD_SET_EN_OPTION);
+ pucan_cmd_opcode_channel(dev->ctrl_idx,
+ PUCAN_CMD_SET_EN_OPTION);
puo->options = cpu_to_le16(PUCAN_OPTION_CANDFDISO);
@@ -274,7 +275,7 @@
/* next, go back to operational mode */
cmd = (struct pucan_command *)pc;
- cmd->opcode_channel = pucan_cmd_opcode_channel(dev,
+ cmd->opcode_channel = pucan_cmd_opcode_channel(dev->ctrl_idx,
(dev->can.ctrlmode & CAN_CTRLMODE_LISTENONLY) ?
PUCAN_CMD_LISTEN_ONLY_MODE :
PUCAN_CMD_NORMAL_MODE);
@@ -296,7 +297,7 @@
struct pucan_command *cmd = (struct pucan_command *)pc;
/* build cmd to go back to reset mode */
- cmd->opcode_channel = pucan_cmd_opcode_channel(dev,
+ cmd->opcode_channel = pucan_cmd_opcode_channel(dev->ctrl_idx,
PUCAN_CMD_RESET_MODE);
l = sizeof(struct pucan_command);
}
@@ -332,7 +333,7 @@
}
for (i = idx; i < n; i++, cmd++) {
- cmd->opcode_channel = pucan_cmd_opcode_channel(dev,
+ cmd->opcode_channel = pucan_cmd_opcode_channel(dev->ctrl_idx,
PUCAN_CMD_FILTER_STD);
cmd->idx = cpu_to_le16(i);
cmd->mask = cpu_to_le32(mask);
@@ -352,7 +353,7 @@
{
struct pcan_ufd_options *cmd = pcan_usb_fd_cmd_buffer(dev);
- cmd->opcode_channel = pucan_cmd_opcode_channel(dev,
+ cmd->opcode_channel = pucan_cmd_opcode_channel(dev->ctrl_idx,
(onoff) ? PUCAN_CMD_SET_EN_OPTION :
PUCAN_CMD_CLR_DIS_OPTION);
@@ -368,7 +369,7 @@
{
struct pcan_ufd_led *cmd = pcan_usb_fd_cmd_buffer(dev);
- cmd->opcode_channel = pucan_cmd_opcode_channel(dev,
+ cmd->opcode_channel = pucan_cmd_opcode_channel(dev->ctrl_idx,
PCAN_UFD_CMD_LED_SET);
cmd->mode = led_mode;
@@ -382,7 +383,7 @@
{
struct pcan_ufd_clock *cmd = pcan_usb_fd_cmd_buffer(dev);
- cmd->opcode_channel = pucan_cmd_opcode_channel(dev,
+ cmd->opcode_channel = pucan_cmd_opcode_channel(dev->ctrl_idx,
PCAN_UFD_CMD_CLK_SET);
cmd->mode = clk_mode;
@@ -396,7 +397,7 @@
{
struct pucan_timing_slow *cmd = pcan_usb_fd_cmd_buffer(dev);
- cmd->opcode_channel = pucan_cmd_opcode_channel(dev,
+ cmd->opcode_channel = pucan_cmd_opcode_channel(dev->ctrl_idx,
PUCAN_CMD_TIMING_SLOW);
cmd->sjw_t = PUCAN_TSLOW_SJW_T(bt->sjw - 1,
dev->can.ctrlmode & CAN_CTRLMODE_3_SAMPLES);
@@ -417,7 +418,7 @@
{
struct pucan_timing_fast *cmd = pcan_usb_fd_cmd_buffer(dev);
- cmd->opcode_channel = pucan_cmd_opcode_channel(dev,
+ cmd->opcode_channel = pucan_cmd_opcode_channel(dev->ctrl_idx,
PUCAN_CMD_TIMING_FAST);
cmd->sjw = PUCAN_TFAST_SJW(bt->sjw - 1);
cmd->tseg2 = PUCAN_TFAST_TSEG2(bt->phase_seg2 - 1);
@@ -1061,6 +1062,78 @@
.do_get_berr_counter = pcan_usb_fd_get_berr_counter,
};
+/* describes the PCAN-CHIP USB */
+static const struct can_bittiming_const pcan_usb_chip_const = {
+ .name = "pcan_chip_usb",
+ .tseg1_min = 1,
+ .tseg1_max = (1 << PUCAN_TSLOW_TSGEG1_BITS),
+ .tseg2_min = 1,
+ .tseg2_max = (1 << PUCAN_TSLOW_TSGEG2_BITS),
+ .sjw_max = (1 << PUCAN_TSLOW_SJW_BITS),
+ .brp_min = 1,
+ .brp_max = (1 << PUCAN_TSLOW_BRP_BITS),
+ .brp_inc = 1,
+};
+
+static const struct can_bittiming_const pcan_usb_chip_data_const = {
+ .name = "pcan_chip_usb",
+ .tseg1_min = 1,
+ .tseg1_max = (1 << PUCAN_TFAST_TSGEG1_BITS),
+ .tseg2_min = 1,
+ .tseg2_max = (1 << PUCAN_TFAST_TSGEG2_BITS),
+ .sjw_max = (1 << PUCAN_TFAST_SJW_BITS),
+ .brp_min = 1,
+ .brp_max = (1 << PUCAN_TFAST_BRP_BITS),
+ .brp_inc = 1,
+};
+
+const struct peak_usb_adapter pcan_usb_chip = {
+ .name = "PCAN-Chip USB",
+ .device_id = PCAN_USBCHIP_PRODUCT_ID,
+ .ctrl_count = PCAN_USBFD_CHANNEL_COUNT,
+ .ctrlmode_supported = CAN_CTRLMODE_FD |
+ CAN_CTRLMODE_3_SAMPLES | CAN_CTRLMODE_LISTENONLY,
+ .clock = {
+ .freq = PCAN_UFD_CRYSTAL_HZ,
+ },
+ .bittiming_const = &pcan_usb_chip_const,
+ .data_bittiming_const = &pcan_usb_chip_data_const,
+
+ /* size of device private data */
+ .sizeof_dev_private = sizeof(struct pcan_usb_fd_device),
+
+ /* timestamps usage */
+ .ts_used_bits = 32,
+ .ts_period = 1000000, /* calibration period in ts. */
+ .us_per_ts_scale = 1, /* us = (ts * scale) >> shift */
+ .us_per_ts_shift = 0,
+
+ /* give here messages in/out endpoints */
+ .ep_msg_in = PCAN_USBPRO_EP_MSGIN,
+ .ep_msg_out = {PCAN_USBPRO_EP_MSGOUT_0},
+
+ /* size of rx/tx usb buffers */
+ .rx_buffer_size = PCAN_UFD_RX_BUFFER_SIZE,
+ .tx_buffer_size = PCAN_UFD_TX_BUFFER_SIZE,
+
+ /* device callbacks */
+ .intf_probe = pcan_usb_pro_probe, /* same as PCAN-USB Pro */
+ .dev_init = pcan_usb_fd_init,
+
+ .dev_exit = pcan_usb_fd_exit,
+ .dev_free = pcan_usb_fd_free,
+ .dev_set_bus = pcan_usb_fd_set_bus,
+ .dev_set_bittiming = pcan_usb_fd_set_bittiming_slow,
+ .dev_set_data_bittiming = pcan_usb_fd_set_bittiming_fast,
+ .dev_decode_buf = pcan_usb_fd_decode_buf,
+ .dev_start = pcan_usb_fd_start,
+ .dev_stop = pcan_usb_fd_stop,
+ .dev_restart_async = pcan_usb_fd_restart_async,
+ .dev_encode_msg = pcan_usb_fd_encode_msg,
+
+ .do_get_berr_counter = pcan_usb_fd_get_berr_counter,
+};
+
/* describes the PCAN-USB Pro FD adapter */
static const struct can_bittiming_const pcan_usb_pro_fd_const = {
.name = "pcan_usb_pro_fd",
diff --git a/drivers/net/can/vcan.c b/drivers/net/can/vcan.c
index 674f367..facca33 100644
--- a/drivers/net/can/vcan.c
+++ b/drivers/net/can/vcan.c
@@ -1,7 +1,7 @@
/*
* vcan.c - Virtual CAN interface
*
- * Copyright (c) 2002-2007 Volkswagen Group Electronic Research
+ * Copyright (c) 2002-2017 Volkswagen Group Electronic Research
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -50,9 +50,12 @@
#include <linux/slab.h>
#include <net/rtnetlink.h>
+#define DRV_NAME "vcan"
+
MODULE_DESCRIPTION("virtual CAN interface");
MODULE_LICENSE("Dual BSD/GPL");
MODULE_AUTHOR("Urs Thuermann <urs.thuermann@volkswagen.de>");
+MODULE_ALIAS_RTNL_LINK(DRV_NAME);
/*
@@ -164,7 +167,7 @@
}
static struct rtnl_link_ops vcan_link_ops __read_mostly = {
- .kind = "vcan",
+ .kind = DRV_NAME,
.setup = vcan_setup,
};
diff --git a/drivers/net/can/vxcan.c b/drivers/net/can/vxcan.c
new file mode 100644
index 0000000..7fbb247
--- /dev/null
+++ b/drivers/net/can/vxcan.c
@@ -0,0 +1,316 @@
+/*
+ * vxcan.c - Virtual CAN Tunnel for cross namespace communication
+ *
+ * This code is derived from drivers/net/can/vcan.c for the virtual CAN
+ * specific parts and from drivers/net/veth.c to implement the netlink API
+ * for network interface pairs in a common and established way.
+ *
+ * Copyright (c) 2017 Oliver Hartkopp <socketcan@hartkopp.net>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the version 2 of the GNU General Public License
+ * as published by the Free Software Foundation
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/netdevice.h>
+#include <linux/if_arp.h>
+#include <linux/if_ether.h>
+#include <linux/can.h>
+#include <linux/can/dev.h>
+#include <linux/can/skb.h>
+#include <linux/can/vxcan.h>
+#include <linux/slab.h>
+#include <net/rtnetlink.h>
+
+#define DRV_NAME "vxcan"
+
+MODULE_DESCRIPTION("Virtual CAN Tunnel");
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Oliver Hartkopp <socketcan@hartkopp.net>");
+MODULE_ALIAS_RTNL_LINK(DRV_NAME);
+
+struct vxcan_priv {
+ struct net_device __rcu *peer;
+};
+
+static netdev_tx_t vxcan_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+ struct vxcan_priv *priv = netdev_priv(dev);
+ struct net_device *peer;
+ struct canfd_frame *cfd = (struct canfd_frame *)skb->data;
+ struct net_device_stats *peerstats, *srcstats = &dev->stats;
+
+ if (can_dropped_invalid_skb(dev, skb))
+ return NETDEV_TX_OK;
+
+ rcu_read_lock();
+ peer = rcu_dereference(priv->peer);
+ if (unlikely(!peer)) {
+ kfree_skb(skb);
+ dev->stats.tx_dropped++;
+ goto out_unlock;
+ }
+
+ skb = can_create_echo_skb(skb);
+ if (!skb)
+ goto out_unlock;
+
+ /* reset CAN GW hop counter */
+ skb->csum_start = 0;
+ skb->pkt_type = PACKET_BROADCAST;
+ skb->dev = peer;
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
+
+ if (netif_rx_ni(skb) == NET_RX_SUCCESS) {
+ srcstats->tx_packets++;
+ srcstats->tx_bytes += cfd->len;
+ peerstats = &peer->stats;
+ peerstats->rx_packets++;
+ peerstats->rx_bytes += cfd->len;
+ }
+
+out_unlock:
+ rcu_read_unlock();
+ return NETDEV_TX_OK;
+}
+
+
+static int vxcan_open(struct net_device *dev)
+{
+ struct vxcan_priv *priv = netdev_priv(dev);
+ struct net_device *peer = rtnl_dereference(priv->peer);
+
+ if (!peer)
+ return -ENOTCONN;
+
+ if (peer->flags & IFF_UP) {
+ netif_carrier_on(dev);
+ netif_carrier_on(peer);
+ }
+ return 0;
+}
+
+static int vxcan_close(struct net_device *dev)
+{
+ struct vxcan_priv *priv = netdev_priv(dev);
+ struct net_device *peer = rtnl_dereference(priv->peer);
+
+ netif_carrier_off(dev);
+ if (peer)
+ netif_carrier_off(peer);
+
+ return 0;
+}
+
+static int vxcan_get_iflink(const struct net_device *dev)
+{
+ struct vxcan_priv *priv = netdev_priv(dev);
+ struct net_device *peer;
+ int iflink;
+
+ rcu_read_lock();
+ peer = rcu_dereference(priv->peer);
+ iflink = peer ? peer->ifindex : 0;
+ rcu_read_unlock();
+
+ return iflink;
+}
+
+static int vxcan_change_mtu(struct net_device *dev, int new_mtu)
+{
+ /* Do not allow changing the MTU while running */
+ if (dev->flags & IFF_UP)
+ return -EBUSY;
+
+ if (new_mtu != CAN_MTU && new_mtu != CANFD_MTU)
+ return -EINVAL;
+
+ dev->mtu = new_mtu;
+ return 0;
+}
+
+static const struct net_device_ops vxcan_netdev_ops = {
+ .ndo_open = vxcan_open,
+ .ndo_stop = vxcan_close,
+ .ndo_start_xmit = vxcan_xmit,
+ .ndo_get_iflink = vxcan_get_iflink,
+ .ndo_change_mtu = vxcan_change_mtu,
+};
+
+static void vxcan_setup(struct net_device *dev)
+{
+ dev->type = ARPHRD_CAN;
+ dev->mtu = CAN_MTU;
+ dev->hard_header_len = 0;
+ dev->addr_len = 0;
+ dev->tx_queue_len = 0;
+ dev->flags = (IFF_NOARP|IFF_ECHO);
+ dev->netdev_ops = &vxcan_netdev_ops;
+ dev->destructor = free_netdev;
+}
+
+/* forward declaration for rtnl_create_link() */
+static struct rtnl_link_ops vxcan_link_ops;
+
+static int vxcan_newlink(struct net *net, struct net_device *dev,
+ struct nlattr *tb[], struct nlattr *data[])
+{
+ struct vxcan_priv *priv;
+ struct net_device *peer;
+ struct net *peer_net;
+
+ struct nlattr *peer_tb[IFLA_MAX + 1], **tbp = tb;
+ char ifname[IFNAMSIZ];
+ unsigned char name_assign_type;
+ struct ifinfomsg *ifmp = NULL;
+ int err;
+
+ /* register peer device */
+ if (data && data[VXCAN_INFO_PEER]) {
+ struct nlattr *nla_peer;
+
+ nla_peer = data[VXCAN_INFO_PEER];
+ ifmp = nla_data(nla_peer);
+ err = rtnl_nla_parse_ifla(peer_tb,
+ nla_data(nla_peer) +
+ sizeof(struct ifinfomsg),
+ nla_len(nla_peer) -
+ sizeof(struct ifinfomsg),
+ NULL);
+ if (err < 0)
+ return err;
+
+ tbp = peer_tb;
+ }
+
+ if (tbp[IFLA_IFNAME]) {
+ nla_strlcpy(ifname, tbp[IFLA_IFNAME], IFNAMSIZ);
+ name_assign_type = NET_NAME_USER;
+ } else {
+ snprintf(ifname, IFNAMSIZ, DRV_NAME "%%d");
+ name_assign_type = NET_NAME_ENUM;
+ }
+
+ peer_net = rtnl_link_get_net(net, tbp);
+ if (IS_ERR(peer_net))
+ return PTR_ERR(peer_net);
+
+ peer = rtnl_create_link(peer_net, ifname, name_assign_type,
+ &vxcan_link_ops, tbp);
+ if (IS_ERR(peer)) {
+ put_net(peer_net);
+ return PTR_ERR(peer);
+ }
+
+ if (ifmp && dev->ifindex)
+ peer->ifindex = ifmp->ifi_index;
+
+ err = register_netdevice(peer);
+ put_net(peer_net);
+ peer_net = NULL;
+ if (err < 0) {
+ free_netdev(peer);
+ return err;
+ }
+
+ netif_carrier_off(peer);
+
+ err = rtnl_configure_link(peer, ifmp);
+ if (err < 0) {
+ unregister_netdevice(peer);
+ return err;
+ }
+
+ /* register first device */
+ if (tb[IFLA_IFNAME])
+ nla_strlcpy(dev->name, tb[IFLA_IFNAME], IFNAMSIZ);
+ else
+ snprintf(dev->name, IFNAMSIZ, DRV_NAME "%%d");
+
+ err = register_netdevice(dev);
+ if (err < 0) {
+ unregister_netdevice(peer);
+ return err;
+ }
+
+ netif_carrier_off(dev);
+
+ /* cross link the device pair */
+ priv = netdev_priv(dev);
+ rcu_assign_pointer(priv->peer, peer);
+
+ priv = netdev_priv(peer);
+ rcu_assign_pointer(priv->peer, dev);
+
+ return 0;
+}
+
+static void vxcan_dellink(struct net_device *dev, struct list_head *head)
+{
+ struct vxcan_priv *priv;
+ struct net_device *peer;
+
+ priv = netdev_priv(dev);
+ peer = rtnl_dereference(priv->peer);
+
+ /* Note : dellink() is called from default_device_exit_batch(),
+ * before a rcu_synchronize() point. The devices are guaranteed
+ * not being freed before one RCU grace period.
+ */
+ RCU_INIT_POINTER(priv->peer, NULL);
+ unregister_netdevice_queue(dev, head);
+
+ if (peer) {
+ priv = netdev_priv(peer);
+ RCU_INIT_POINTER(priv->peer, NULL);
+ unregister_netdevice_queue(peer, head);
+ }
+}
+
+static const struct nla_policy vxcan_policy[VXCAN_INFO_MAX + 1] = {
+ [VXCAN_INFO_PEER] = { .len = sizeof(struct ifinfomsg) },
+};
+
+static struct net *vxcan_get_link_net(const struct net_device *dev)
+{
+ struct vxcan_priv *priv = netdev_priv(dev);
+ struct net_device *peer = rtnl_dereference(priv->peer);
+
+ return peer ? dev_net(peer) : dev_net(dev);
+}
+
+static struct rtnl_link_ops vxcan_link_ops = {
+ .kind = DRV_NAME,
+ .priv_size = sizeof(struct vxcan_priv),
+ .setup = vxcan_setup,
+ .newlink = vxcan_newlink,
+ .dellink = vxcan_dellink,
+ .policy = vxcan_policy,
+ .maxtype = VXCAN_INFO_MAX,
+ .get_link_net = vxcan_get_link_net,
+};
+
+static __init int vxcan_init(void)
+{
+ pr_info("vxcan: Virtual CAN Tunnel driver\n");
+
+ return rtnl_link_register(&vxcan_link_ops);
+}
+
+static __exit void vxcan_exit(void)
+{
+ rtnl_link_unregister(&vxcan_link_ops);
+}
+
+module_init(vxcan_init);
+module_exit(vxcan_exit);
diff --git a/drivers/net/cris/eth_v10.c b/drivers/net/cris/eth_v10.c
index 91c876a..da02041 100644
--- a/drivers/net/cris/eth_v10.c
+++ b/drivers/net/cris/eth_v10.c
@@ -1412,31 +1412,39 @@
return rc;
}
-static int e100_get_settings(struct net_device *dev,
- struct ethtool_cmd *cmd)
+static int e100_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct net_local *np = netdev_priv(dev);
+ u32 supported;
int err;
spin_lock_irq(&np->lock);
- err = mii_ethtool_gset(&np->mii_if, cmd);
+ err = mii_ethtool_get_link_ksettings(&np->mii_if, cmd);
spin_unlock_irq(&np->lock);
/* The PHY may support 1000baseT, but the Etrax100 does not. */
- cmd->supported &= ~(SUPPORTED_1000baseT_Half
- | SUPPORTED_1000baseT_Full);
+ ethtool_convert_link_mode_to_legacy_u32(&supported,
+ cmd->link_modes.supported);
+
+ supported &= ~(SUPPORTED_1000baseT_Half | SUPPORTED_1000baseT_Full);
+
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+
return err;
}
-static int e100_set_settings(struct net_device *dev,
- struct ethtool_cmd *ecmd)
+static int e100_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *ecmd)
{
- if (ecmd->autoneg == AUTONEG_ENABLE) {
+ if (ecmd->base.autoneg == AUTONEG_ENABLE) {
e100_set_duplex(dev, autoneg);
e100_set_speed(dev, 0);
} else {
- e100_set_duplex(dev, ecmd->duplex == DUPLEX_HALF ? half : full);
- e100_set_speed(dev, ecmd->speed == SPEED_10 ? 10: 100);
+ e100_set_duplex(dev, ecmd->base.duplex == DUPLEX_HALF ?
+ half : full);
+ e100_set_speed(dev, ecmd->base.speed == SPEED_10 ? 10 : 100);
}
return 0;
@@ -1459,11 +1467,11 @@
}
static const struct ethtool_ops e100_ethtool_ops = {
- .get_settings = e100_get_settings,
- .set_settings = e100_set_settings,
.get_drvinfo = e100_get_drvinfo,
.nway_reset = e100_nway_reset,
.get_link = ethtool_op_get_link,
+ .get_link_ksettings = e100_get_link_ksettings,
+ .set_link_ksettings = e100_set_link_ksettings,
};
static int
diff --git a/drivers/net/dsa/Kconfig b/drivers/net/dsa/Kconfig
index 0659846..862ee22 100644
--- a/drivers/net/dsa/Kconfig
+++ b/drivers/net/dsa/Kconfig
@@ -11,7 +11,7 @@
config NET_DSA_BCM_SF2
tristate "Broadcom Starfighter 2 Ethernet switch support"
- depends on HAS_IOMEM && NET_DSA
+ depends on HAS_IOMEM && NET_DSA && OF_MDIO
select NET_DSA_TAG_BRCM
select FIXED_PHY
select BCM7XXX_PHY
@@ -34,4 +34,44 @@
This enables support for the Qualcomm Atheros QCA8K Ethernet
switch chips.
+config NET_DSA_LOOP
+ tristate "DSA mock-up Ethernet switch chip support"
+ depends on NET_DSA
+ select FIXED_PHY
+ ---help---
+ This enables support for a fake mock-up switch chip which
+ exercises the DSA APIs.
+
+config NET_DSA_MT7530
+ tristate "Mediatek MT7530 Ethernet switch support"
+ depends on NET_DSA
+ select NET_DSA_TAG_MTK
+ ---help---
+ This enables support for the Mediatek MT7530 Ethernet switch
+ chip.
+
+config NET_DSA_SMSC_LAN9303
+ tristate
+ select NET_DSA_TAG_LAN9303
+ ---help---
+ This enables support for the SMSC/Microchip LAN9303 3 port ethernet
+ switch chips.
+
+config NET_DSA_SMSC_LAN9303_I2C
+ tristate "SMSC/Microchip LAN9303 3-ports 10/100 ethernet switch in I2C managed mode"
+ depends on NET_DSA && I2C
+ select NET_DSA_SMSC_LAN9303
+ select REGMAP_I2C
+ ---help---
+ Enable access functions if the SMSC/Microchip LAN9303 is configured
+ for I2C managed mode.
+
+config NET_DSA_SMSC_LAN9303_MDIO
+ tristate "SMSC/Microchip LAN9303 3-ports 10/100 ethernet switch in MDIO managed mode"
+ depends on NET_DSA
+ select NET_DSA_SMSC_LAN9303
+ ---help---
+ Enable access functions if the SMSC/Microchip LAN9303 is configured
+ for MDIO managed mode.
+
endmenu
diff --git a/drivers/net/dsa/Makefile b/drivers/net/dsa/Makefile
index a3c9416..edd6303 100644
--- a/drivers/net/dsa/Makefile
+++ b/drivers/net/dsa/Makefile
@@ -2,6 +2,10 @@
obj-$(CONFIG_NET_DSA_BCM_SF2) += bcm-sf2.o
bcm-sf2-objs := bcm_sf2.o bcm_sf2_cfp.o
obj-$(CONFIG_NET_DSA_QCA8K) += qca8k.o
-
+obj-$(CONFIG_NET_DSA_MT7530) += mt7530.o
+obj-$(CONFIG_NET_DSA_SMSC_LAN9303) += lan9303-core.o
+obj-$(CONFIG_NET_DSA_SMSC_LAN9303_I2C) += lan9303_i2c.o
+obj-$(CONFIG_NET_DSA_SMSC_LAN9303_MDIO) += lan9303_mdio.o
obj-y += b53/
obj-y += mv88e6xxx/
+obj-$(CONFIG_NET_DSA_LOOP) += dsa_loop.o dsa_loop_bdinfo.o
diff --git a/drivers/net/dsa/b53/b53_common.c b/drivers/net/dsa/b53/b53_common.c
index 8cf4801..fa0eece2 100644
--- a/drivers/net/dsa/b53/b53_common.c
+++ b/drivers/net/dsa/b53/b53_common.c
@@ -326,6 +326,7 @@
static void b53_set_forwarding(struct b53_device *dev, int enable)
{
+ struct dsa_switch *ds = dev->ds;
u8 mgmt;
b53_read8(dev, B53_CTRL_PAGE, B53_SWITCH_MODE, &mgmt);
@@ -336,6 +337,15 @@
mgmt &= ~SM_SW_FWD_EN;
b53_write8(dev, B53_CTRL_PAGE, B53_SWITCH_MODE, mgmt);
+
+ /* Include IMP port in dumb forwarding mode when no tagging protocol is
+ * set
+ */
+ if (ds->ops->get_tag_protocol(ds) == DSA_TAG_PROTO_NONE) {
+ b53_read8(dev, B53_CTRL_PAGE, B53_SWITCH_CTRL, &mgmt);
+ mgmt |= B53_MII_DUMB_FWDG_EN;
+ b53_write8(dev, B53_CTRL_PAGE, B53_SWITCH_CTRL, mgmt);
+ }
}
static void b53_enable_vlan(struct b53_device *dev, bool enable)
@@ -598,7 +608,8 @@
static int b53_switch_reset(struct b53_device *dev)
{
- u8 mgmt;
+ unsigned int timeout = 1000;
+ u8 mgmt, reg;
b53_switch_reset_gpio(dev);
@@ -607,6 +618,28 @@
b53_write8(dev, B53_CTRL_PAGE, B53_SOFTRESET, 0x00);
}
+ /* This is specific to 58xx devices here, do not use is58xx() which
+ * covers the larger Starfigther 2 family, including 7445/7278 which
+ * still use this driver as a library and need to perform the reset
+ * earlier.
+ */
+ if (dev->chip_id == BCM58XX_DEVICE_ID) {
+ b53_read8(dev, B53_CTRL_PAGE, B53_SOFTRESET, ®);
+ reg |= SW_RST | EN_SW_RST | EN_CH_RST;
+ b53_write8(dev, B53_CTRL_PAGE, B53_SOFTRESET, reg);
+
+ do {
+ b53_read8(dev, B53_CTRL_PAGE, B53_SOFTRESET, ®);
+ if (!(reg & SW_RST))
+ break;
+
+ usleep_range(1000, 2000);
+ } while (timeout-- > 0);
+
+ if (timeout == 0)
+ return -ETIMEDOUT;
+ }
+
b53_read8(dev, B53_CTRL_PAGE, B53_SWITCH_MODE, &mgmt);
if (!(mgmt & SM_SW_FWD_EN)) {
@@ -1731,7 +1764,7 @@
.vlans = 4096,
.enabled_ports = 0x1ff,
.arl_entries = 4,
- .cpu_port = B53_CPU_PORT_25,
+ .cpu_port = B53_CPU_PORT,
.vta_regs = B53_VTA_REGS,
.duplex_reg = B53_DUPLEX_STAT_GE,
.jumbo_pm_reg = B53_JUMBO_PORT_MASK,
diff --git a/drivers/net/dsa/b53/b53_regs.h b/drivers/net/dsa/b53/b53_regs.h
index 9fd24c4..e5c86d4 100644
--- a/drivers/net/dsa/b53/b53_regs.h
+++ b/drivers/net/dsa/b53/b53_regs.h
@@ -104,6 +104,10 @@
#define B53_UC_FWD_EN BIT(6)
#define B53_MC_FWD_EN BIT(7)
+/* Switch control (8 bit) */
+#define B53_SWITCH_CTRL 0x22
+#define B53_MII_DUMB_FWDG_EN BIT(6)
+
/* (16 bit) */
#define B53_UC_FLOOD_MASK 0x32
#define B53_MC_FLOOD_MASK 0x34
@@ -139,6 +143,7 @@
/* Software reset register (8 bit) */
#define B53_SOFTRESET 0x79
#define SW_RST BIT(7)
+#define EN_CH_RST BIT(6)
#define EN_SW_RST BIT(4)
/* Fast Aging Control register (8 bit) */
diff --git a/drivers/net/dsa/bcm_sf2_cfp.c b/drivers/net/dsa/bcm_sf2_cfp.c
index 346dd9a..2fb32d6 100644
--- a/drivers/net/dsa/bcm_sf2_cfp.c
+++ b/drivers/net/dsa/bcm_sf2_cfp.c
@@ -10,10 +10,11 @@
*/
#include <linux/list.h>
-#include <net/dsa.h>
#include <linux/ethtool.h>
#include <linux/if_ether.h>
#include <linux/in.h>
+#include <linux/netdevice.h>
+#include <net/dsa.h>
#include <linux/bitmap.h>
#include "bcm_sf2.h"
diff --git a/drivers/net/dsa/dsa_loop.c b/drivers/net/dsa/dsa_loop.c
new file mode 100644
index 0000000..f0fc4de
--- /dev/null
+++ b/drivers/net/dsa/dsa_loop.c
@@ -0,0 +1,328 @@
+/*
+ * Distributed Switch Architecture loopback driver
+ *
+ * Copyright (C) 2016, Florian Fainelli <f.fainelli@gmail.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/platform_device.h>
+#include <linux/netdevice.h>
+#include <linux/phy.h>
+#include <linux/phy_fixed.h>
+#include <linux/export.h>
+#include <linux/workqueue.h>
+#include <linux/module.h>
+#include <linux/if_bridge.h>
+#include <net/switchdev.h>
+#include <net/dsa.h>
+
+#include "dsa_loop.h"
+
+struct dsa_loop_vlan {
+ u16 members;
+ u16 untagged;
+};
+
+#define DSA_LOOP_VLANS 5
+
+struct dsa_loop_priv {
+ struct mii_bus *bus;
+ unsigned int port_base;
+ struct dsa_loop_vlan vlans[DSA_LOOP_VLANS];
+ struct net_device *netdev;
+ u16 pvid;
+};
+
+static struct phy_device *phydevs[PHY_MAX_ADDR];
+
+static enum dsa_tag_protocol dsa_loop_get_protocol(struct dsa_switch *ds)
+{
+ dev_dbg(ds->dev, "%s\n", __func__);
+
+ return DSA_TAG_PROTO_NONE;
+}
+
+static int dsa_loop_setup(struct dsa_switch *ds)
+{
+ dev_dbg(ds->dev, "%s\n", __func__);
+
+ return 0;
+}
+
+static int dsa_loop_set_addr(struct dsa_switch *ds, u8 *addr)
+{
+ dev_dbg(ds->dev, "%s\n", __func__);
+
+ return 0;
+}
+
+static int dsa_loop_phy_read(struct dsa_switch *ds, int port, int regnum)
+{
+ struct dsa_loop_priv *ps = ds->priv;
+ struct mii_bus *bus = ps->bus;
+
+ dev_dbg(ds->dev, "%s\n", __func__);
+
+ return mdiobus_read_nested(bus, ps->port_base + port, regnum);
+}
+
+static int dsa_loop_phy_write(struct dsa_switch *ds, int port,
+ int regnum, u16 value)
+{
+ struct dsa_loop_priv *ps = ds->priv;
+ struct mii_bus *bus = ps->bus;
+
+ dev_dbg(ds->dev, "%s\n", __func__);
+
+ return mdiobus_write_nested(bus, ps->port_base + port, regnum, value);
+}
+
+static int dsa_loop_port_bridge_join(struct dsa_switch *ds, int port,
+ struct net_device *bridge)
+{
+ dev_dbg(ds->dev, "%s\n", __func__);
+
+ return 0;
+}
+
+static void dsa_loop_port_bridge_leave(struct dsa_switch *ds, int port,
+ struct net_device *bridge)
+{
+ dev_dbg(ds->dev, "%s\n", __func__);
+}
+
+static void dsa_loop_port_stp_state_set(struct dsa_switch *ds, int port,
+ u8 state)
+{
+ dev_dbg(ds->dev, "%s\n", __func__);
+}
+
+static int dsa_loop_port_vlan_filtering(struct dsa_switch *ds, int port,
+ bool vlan_filtering)
+{
+ dev_dbg(ds->dev, "%s\n", __func__);
+
+ return 0;
+}
+
+static int dsa_loop_port_vlan_prepare(struct dsa_switch *ds, int port,
+ const struct switchdev_obj_port_vlan *vlan,
+ struct switchdev_trans *trans)
+{
+ struct dsa_loop_priv *ps = ds->priv;
+ struct mii_bus *bus = ps->bus;
+
+ dev_dbg(ds->dev, "%s\n", __func__);
+
+ /* Just do a sleeping operation to make lockdep checks effective */
+ mdiobus_read(bus, ps->port_base + port, MII_BMSR);
+
+ if (vlan->vid_end > DSA_LOOP_VLANS)
+ return -ERANGE;
+
+ return 0;
+}
+
+static void dsa_loop_port_vlan_add(struct dsa_switch *ds, int port,
+ const struct switchdev_obj_port_vlan *vlan,
+ struct switchdev_trans *trans)
+{
+ bool untagged = vlan->flags & BRIDGE_VLAN_INFO_UNTAGGED;
+ bool pvid = vlan->flags & BRIDGE_VLAN_INFO_PVID;
+ struct dsa_loop_priv *ps = ds->priv;
+ struct mii_bus *bus = ps->bus;
+ struct dsa_loop_vlan *vl;
+ u16 vid;
+
+ dev_dbg(ds->dev, "%s\n", __func__);
+
+ /* Just do a sleeping operation to make lockdep checks effective */
+ mdiobus_read(bus, ps->port_base + port, MII_BMSR);
+
+ for (vid = vlan->vid_begin; vid <= vlan->vid_end; ++vid) {
+ vl = &ps->vlans[vid];
+
+ vl->members |= BIT(port);
+ if (untagged)
+ vl->untagged |= BIT(port);
+ else
+ vl->untagged &= ~BIT(port);
+ }
+
+ if (pvid)
+ ps->pvid = vid;
+}
+
+static int dsa_loop_port_vlan_del(struct dsa_switch *ds, int port,
+ const struct switchdev_obj_port_vlan *vlan)
+{
+ bool untagged = vlan->flags & BRIDGE_VLAN_INFO_UNTAGGED;
+ struct dsa_loop_priv *ps = ds->priv;
+ struct mii_bus *bus = ps->bus;
+ struct dsa_loop_vlan *vl;
+ u16 vid, pvid = ps->pvid;
+
+ dev_dbg(ds->dev, "%s\n", __func__);
+
+ /* Just do a sleeping operation to make lockdep checks effective */
+ mdiobus_read(bus, ps->port_base + port, MII_BMSR);
+
+ for (vid = vlan->vid_begin; vid <= vlan->vid_end; ++vid) {
+ vl = &ps->vlans[vid];
+
+ vl->members &= ~BIT(port);
+ if (untagged)
+ vl->untagged &= ~BIT(port);
+
+ if (pvid == vid)
+ pvid = 1;
+ }
+ ps->pvid = pvid;
+
+ return 0;
+}
+
+static int dsa_loop_port_vlan_dump(struct dsa_switch *ds, int port,
+ struct switchdev_obj_port_vlan *vlan,
+ int (*cb)(struct switchdev_obj *obj))
+{
+ struct dsa_loop_priv *ps = ds->priv;
+ struct mii_bus *bus = ps->bus;
+ struct dsa_loop_vlan *vl;
+ u16 vid, vid_start = 0;
+ int err = 0;
+
+ dev_dbg(ds->dev, "%s\n", __func__);
+
+ /* Just do a sleeping operation to make lockdep checks effective */
+ mdiobus_read(bus, ps->port_base + port, MII_BMSR);
+
+ for (vid = vid_start; vid < DSA_LOOP_VLANS; vid++) {
+ vl = &ps->vlans[vid];
+
+ if (!(vl->members & BIT(port)))
+ continue;
+
+ vlan->vid_begin = vlan->vid_end = vid;
+ vlan->flags = 0;
+
+ if (vl->untagged & BIT(port))
+ vlan->flags |= BRIDGE_VLAN_INFO_UNTAGGED;
+ if (ps->pvid == vid)
+ vlan->flags |= BRIDGE_VLAN_INFO_PVID;
+
+ err = cb(&vlan->obj);
+ if (err)
+ break;
+ }
+
+ return err;
+}
+
+static struct dsa_switch_ops dsa_loop_driver = {
+ .get_tag_protocol = dsa_loop_get_protocol,
+ .setup = dsa_loop_setup,
+ .set_addr = dsa_loop_set_addr,
+ .phy_read = dsa_loop_phy_read,
+ .phy_write = dsa_loop_phy_write,
+ .port_bridge_join = dsa_loop_port_bridge_join,
+ .port_bridge_leave = dsa_loop_port_bridge_leave,
+ .port_stp_state_set = dsa_loop_port_stp_state_set,
+ .port_vlan_filtering = dsa_loop_port_vlan_filtering,
+ .port_vlan_prepare = dsa_loop_port_vlan_prepare,
+ .port_vlan_add = dsa_loop_port_vlan_add,
+ .port_vlan_del = dsa_loop_port_vlan_del,
+ .port_vlan_dump = dsa_loop_port_vlan_dump,
+};
+
+static int dsa_loop_drv_probe(struct mdio_device *mdiodev)
+{
+ struct dsa_loop_pdata *pdata = mdiodev->dev.platform_data;
+ struct dsa_loop_priv *ps;
+ struct dsa_switch *ds;
+
+ if (!pdata)
+ return -ENODEV;
+
+ dev_info(&mdiodev->dev, "%s: 0x%0x\n",
+ pdata->name, pdata->enabled_ports);
+
+ ds = dsa_switch_alloc(&mdiodev->dev, DSA_MAX_PORTS);
+ if (!ds)
+ return -ENOMEM;
+
+ ps = devm_kzalloc(&mdiodev->dev, sizeof(*ps), GFP_KERNEL);
+ ps->netdev = dev_get_by_name(&init_net, pdata->netdev);
+ if (!ps->netdev)
+ return -EPROBE_DEFER;
+
+ pdata->cd.netdev[DSA_LOOP_CPU_PORT] = &ps->netdev->dev;
+
+ ds->dev = &mdiodev->dev;
+ ds->ops = &dsa_loop_driver;
+ ds->priv = ps;
+ ps->bus = mdiodev->bus;
+
+ dev_set_drvdata(&mdiodev->dev, ds);
+
+ return dsa_register_switch(ds, ds->dev);
+}
+
+static void dsa_loop_drv_remove(struct mdio_device *mdiodev)
+{
+ struct dsa_switch *ds = dev_get_drvdata(&mdiodev->dev);
+ struct dsa_loop_priv *ps = ds->priv;
+
+ dsa_unregister_switch(ds);
+ dev_put(ps->netdev);
+}
+
+static struct mdio_driver dsa_loop_drv = {
+ .mdiodrv.driver = {
+ .name = "dsa-loop",
+ },
+ .probe = dsa_loop_drv_probe,
+ .remove = dsa_loop_drv_remove,
+};
+
+#define NUM_FIXED_PHYS (DSA_LOOP_NUM_PORTS - 2)
+
+static void unregister_fixed_phys(void)
+{
+ unsigned int i;
+
+ for (i = 0; i < NUM_FIXED_PHYS; i++)
+ if (phydevs[i])
+ fixed_phy_unregister(phydevs[i]);
+}
+
+static int __init dsa_loop_init(void)
+{
+ struct fixed_phy_status status = {
+ .link = 1,
+ .speed = SPEED_100,
+ .duplex = DUPLEX_FULL,
+ };
+ unsigned int i;
+
+ for (i = 0; i < NUM_FIXED_PHYS; i++)
+ phydevs[i] = fixed_phy_register(PHY_POLL, &status, -1, NULL);
+
+ return mdio_driver_register(&dsa_loop_drv);
+}
+module_init(dsa_loop_init);
+
+static void __exit dsa_loop_exit(void)
+{
+ mdio_driver_unregister(&dsa_loop_drv);
+ unregister_fixed_phys();
+}
+module_exit(dsa_loop_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Florian Fainelli");
+MODULE_DESCRIPTION("DSA loopback driver");
diff --git a/drivers/net/dsa/dsa_loop.h b/drivers/net/dsa/dsa_loop.h
new file mode 100644
index 0000000..dc39687
--- /dev/null
+++ b/drivers/net/dsa/dsa_loop.h
@@ -0,0 +1,19 @@
+#ifndef __DSA_LOOP_H
+#define __DSA_LOOP_H
+
+struct dsa_chip_data;
+
+struct dsa_loop_pdata {
+ /* Must be first, such that dsa_register_switch() can access this
+ * without gory pointer manipulations
+ */
+ struct dsa_chip_data cd;
+ const char *name;
+ unsigned int enabled_ports;
+ const char *netdev;
+};
+
+#define DSA_LOOP_NUM_PORTS 6
+#define DSA_LOOP_CPU_PORT (DSA_LOOP_NUM_PORTS - 1)
+
+#endif /* __DSA_LOOP_H */
diff --git a/drivers/net/dsa/dsa_loop_bdinfo.c b/drivers/net/dsa/dsa_loop_bdinfo.c
new file mode 100644
index 0000000..fb8d5dc
--- /dev/null
+++ b/drivers/net/dsa/dsa_loop_bdinfo.c
@@ -0,0 +1,34 @@
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/phy.h>
+#include <net/dsa.h>
+
+#include "dsa_loop.h"
+
+static struct dsa_loop_pdata dsa_loop_pdata = {
+ .cd = {
+ .port_names[0] = "lan1",
+ .port_names[1] = "lan2",
+ .port_names[2] = "lan3",
+ .port_names[3] = "lan4",
+ .port_names[DSA_LOOP_CPU_PORT] = "cpu",
+ },
+ .name = "DSA mockup driver",
+ .enabled_ports = 0x1f,
+ .netdev = "eth0",
+};
+
+static const struct mdio_board_info bdinfo = {
+ .bus_id = "fixed-0",
+ .modalias = "dsa-loop",
+ .mdio_addr = 31,
+ .platform_data = &dsa_loop_pdata,
+};
+
+static int __init dsa_loop_bdinfo_init(void)
+{
+ return mdiobus_register_board_info(&bdinfo, 1);
+}
+arch_initcall(dsa_loop_bdinfo_init)
+
+MODULE_LICENSE("GPL");
diff --git a/drivers/net/dsa/lan9303-core.c b/drivers/net/dsa/lan9303-core.c
new file mode 100644
index 0000000..c8b2423
--- /dev/null
+++ b/drivers/net/dsa/lan9303-core.c
@@ -0,0 +1,879 @@
+/*
+ * Copyright (C) 2017 Pengutronix, Juergen Borleis <kernel@pengutronix.de>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/gpio/consumer.h>
+#include <linux/regmap.h>
+#include <linux/mutex.h>
+#include <linux/mii.h>
+
+#include "lan9303.h"
+
+#define LAN9303_CHIP_REV 0x14
+# define LAN9303_CHIP_ID 0x9303
+#define LAN9303_IRQ_CFG 0x15
+# define LAN9303_IRQ_CFG_IRQ_ENABLE BIT(8)
+# define LAN9303_IRQ_CFG_IRQ_POL BIT(4)
+# define LAN9303_IRQ_CFG_IRQ_TYPE BIT(0)
+#define LAN9303_INT_STS 0x16
+# define LAN9303_INT_STS_PHY_INT2 BIT(27)
+# define LAN9303_INT_STS_PHY_INT1 BIT(26)
+#define LAN9303_INT_EN 0x17
+# define LAN9303_INT_EN_PHY_INT2_EN BIT(27)
+# define LAN9303_INT_EN_PHY_INT1_EN BIT(26)
+#define LAN9303_HW_CFG 0x1D
+# define LAN9303_HW_CFG_READY BIT(27)
+# define LAN9303_HW_CFG_AMDX_EN_PORT2 BIT(26)
+# define LAN9303_HW_CFG_AMDX_EN_PORT1 BIT(25)
+#define LAN9303_PMI_DATA 0x29
+#define LAN9303_PMI_ACCESS 0x2A
+# define LAN9303_PMI_ACCESS_PHY_ADDR(x) (((x) & 0x1f) << 11)
+# define LAN9303_PMI_ACCESS_MIIRINDA(x) (((x) & 0x1f) << 6)
+# define LAN9303_PMI_ACCESS_MII_BUSY BIT(0)
+# define LAN9303_PMI_ACCESS_MII_WRITE BIT(1)
+#define LAN9303_MANUAL_FC_1 0x68
+#define LAN9303_MANUAL_FC_2 0x69
+#define LAN9303_MANUAL_FC_0 0x6a
+#define LAN9303_SWITCH_CSR_DATA 0x6b
+#define LAN9303_SWITCH_CSR_CMD 0x6c
+#define LAN9303_SWITCH_CSR_CMD_BUSY BIT(31)
+#define LAN9303_SWITCH_CSR_CMD_RW BIT(30)
+#define LAN9303_SWITCH_CSR_CMD_LANES (BIT(19) | BIT(18) | BIT(17) | BIT(16))
+#define LAN9303_VIRT_PHY_BASE 0x70
+#define LAN9303_VIRT_SPECIAL_CTRL 0x77
+
+#define LAN9303_SW_DEV_ID 0x0000
+#define LAN9303_SW_RESET 0x0001
+#define LAN9303_SW_RESET_RESET BIT(0)
+#define LAN9303_SW_IMR 0x0004
+#define LAN9303_SW_IPR 0x0005
+#define LAN9303_MAC_VER_ID_0 0x0400
+#define LAN9303_MAC_RX_CFG_0 0x0401
+# define LAN9303_MAC_RX_CFG_X_REJECT_MAC_TYPES BIT(1)
+# define LAN9303_MAC_RX_CFG_X_RX_ENABLE BIT(0)
+#define LAN9303_MAC_RX_UNDSZE_CNT_0 0x0410
+#define LAN9303_MAC_RX_64_CNT_0 0x0411
+#define LAN9303_MAC_RX_127_CNT_0 0x0412
+#define LAN9303_MAC_RX_255_CNT_0 0x413
+#define LAN9303_MAC_RX_511_CNT_0 0x0414
+#define LAN9303_MAC_RX_1023_CNT_0 0x0415
+#define LAN9303_MAC_RX_MAX_CNT_0 0x0416
+#define LAN9303_MAC_RX_OVRSZE_CNT_0 0x0417
+#define LAN9303_MAC_RX_PKTOK_CNT_0 0x0418
+#define LAN9303_MAC_RX_CRCERR_CNT_0 0x0419
+#define LAN9303_MAC_RX_MULCST_CNT_0 0x041a
+#define LAN9303_MAC_RX_BRDCST_CNT_0 0x041b
+#define LAN9303_MAC_RX_PAUSE_CNT_0 0x041c
+#define LAN9303_MAC_RX_FRAG_CNT_0 0x041d
+#define LAN9303_MAC_RX_JABB_CNT_0 0x041e
+#define LAN9303_MAC_RX_ALIGN_CNT_0 0x041f
+#define LAN9303_MAC_RX_PKTLEN_CNT_0 0x0420
+#define LAN9303_MAC_RX_GOODPKTLEN_CNT_0 0x0421
+#define LAN9303_MAC_RX_SYMBL_CNT_0 0x0422
+#define LAN9303_MAC_RX_CTLFRM_CNT_0 0x0423
+
+#define LAN9303_MAC_TX_CFG_0 0x0440
+# define LAN9303_MAC_TX_CFG_X_TX_IFG_CONFIG_DEFAULT (21 << 2)
+# define LAN9303_MAC_TX_CFG_X_TX_PAD_ENABLE BIT(1)
+# define LAN9303_MAC_TX_CFG_X_TX_ENABLE BIT(0)
+#define LAN9303_MAC_TX_DEFER_CNT_0 0x0451
+#define LAN9303_MAC_TX_PAUSE_CNT_0 0x0452
+#define LAN9303_MAC_TX_PKTOK_CNT_0 0x0453
+#define LAN9303_MAC_TX_64_CNT_0 0x0454
+#define LAN9303_MAC_TX_127_CNT_0 0x0455
+#define LAN9303_MAC_TX_255_CNT_0 0x0456
+#define LAN9303_MAC_TX_511_CNT_0 0x0457
+#define LAN9303_MAC_TX_1023_CNT_0 0x0458
+#define LAN9303_MAC_TX_MAX_CNT_0 0x0459
+#define LAN9303_MAC_TX_UNDSZE_CNT_0 0x045a
+#define LAN9303_MAC_TX_PKTLEN_CNT_0 0x045c
+#define LAN9303_MAC_TX_BRDCST_CNT_0 0x045d
+#define LAN9303_MAC_TX_MULCST_CNT_0 0x045e
+#define LAN9303_MAC_TX_LATECOL_0 0x045f
+#define LAN9303_MAC_TX_EXCOL_CNT_0 0x0460
+#define LAN9303_MAC_TX_SNGLECOL_CNT_0 0x0461
+#define LAN9303_MAC_TX_MULTICOL_CNT_0 0x0462
+#define LAN9303_MAC_TX_TOTALCOL_CNT_0 0x0463
+
+#define LAN9303_MAC_VER_ID_1 0x0800
+#define LAN9303_MAC_RX_CFG_1 0x0801
+#define LAN9303_MAC_TX_CFG_1 0x0840
+#define LAN9303_MAC_VER_ID_2 0x0c00
+#define LAN9303_MAC_RX_CFG_2 0x0c01
+#define LAN9303_MAC_TX_CFG_2 0x0c40
+#define LAN9303_SWE_ALR_CMD 0x1800
+#define LAN9303_SWE_VLAN_CMD 0x180b
+# define LAN9303_SWE_VLAN_CMD_RNW BIT(5)
+# define LAN9303_SWE_VLAN_CMD_PVIDNVLAN BIT(4)
+#define LAN9303_SWE_VLAN_WR_DATA 0x180c
+#define LAN9303_SWE_VLAN_RD_DATA 0x180e
+# define LAN9303_SWE_VLAN_MEMBER_PORT2 BIT(17)
+# define LAN9303_SWE_VLAN_UNTAG_PORT2 BIT(16)
+# define LAN9303_SWE_VLAN_MEMBER_PORT1 BIT(15)
+# define LAN9303_SWE_VLAN_UNTAG_PORT1 BIT(14)
+# define LAN9303_SWE_VLAN_MEMBER_PORT0 BIT(13)
+# define LAN9303_SWE_VLAN_UNTAG_PORT0 BIT(12)
+#define LAN9303_SWE_VLAN_CMD_STS 0x1810
+#define LAN9303_SWE_GLB_INGRESS_CFG 0x1840
+#define LAN9303_SWE_PORT_STATE 0x1843
+# define LAN9303_SWE_PORT_STATE_FORWARDING_PORT2 (0)
+# define LAN9303_SWE_PORT_STATE_LEARNING_PORT2 BIT(5)
+# define LAN9303_SWE_PORT_STATE_BLOCKING_PORT2 BIT(4)
+# define LAN9303_SWE_PORT_STATE_FORWARDING_PORT1 (0)
+# define LAN9303_SWE_PORT_STATE_LEARNING_PORT1 BIT(3)
+# define LAN9303_SWE_PORT_STATE_BLOCKING_PORT1 BIT(2)
+# define LAN9303_SWE_PORT_STATE_FORWARDING_PORT0 (0)
+# define LAN9303_SWE_PORT_STATE_LEARNING_PORT0 BIT(1)
+# define LAN9303_SWE_PORT_STATE_BLOCKING_PORT0 BIT(0)
+#define LAN9303_SWE_PORT_MIRROR 0x1846
+# define LAN9303_SWE_PORT_MIRROR_SNIFF_ALL BIT(8)
+# define LAN9303_SWE_PORT_MIRROR_SNIFFER_PORT2 BIT(7)
+# define LAN9303_SWE_PORT_MIRROR_SNIFFER_PORT1 BIT(6)
+# define LAN9303_SWE_PORT_MIRROR_SNIFFER_PORT0 BIT(5)
+# define LAN9303_SWE_PORT_MIRROR_MIRRORED_PORT2 BIT(4)
+# define LAN9303_SWE_PORT_MIRROR_MIRRORED_PORT1 BIT(3)
+# define LAN9303_SWE_PORT_MIRROR_MIRRORED_PORT0 BIT(2)
+# define LAN9303_SWE_PORT_MIRROR_ENABLE_RX_MIRRORING BIT(1)
+# define LAN9303_SWE_PORT_MIRROR_ENABLE_TX_MIRRORING BIT(0)
+#define LAN9303_SWE_INGRESS_PORT_TYPE 0x1847
+#define LAN9303_BM_CFG 0x1c00
+#define LAN9303_BM_EGRSS_PORT_TYPE 0x1c0c
+# define LAN9303_BM_EGRSS_PORT_TYPE_SPECIAL_TAG_PORT2 (BIT(17) | BIT(16))
+# define LAN9303_BM_EGRSS_PORT_TYPE_SPECIAL_TAG_PORT1 (BIT(9) | BIT(8))
+# define LAN9303_BM_EGRSS_PORT_TYPE_SPECIAL_TAG_PORT0 (BIT(1) | BIT(0))
+
+#define LAN9303_PORT_0_OFFSET 0x400
+#define LAN9303_PORT_1_OFFSET 0x800
+#define LAN9303_PORT_2_OFFSET 0xc00
+
+/* the built-in PHYs are of type LAN911X */
+#define MII_LAN911X_SPECIAL_MODES 0x12
+#define MII_LAN911X_SPECIAL_CONTROL_STATUS 0x1f
+
+static const struct regmap_range lan9303_valid_regs[] = {
+ regmap_reg_range(0x14, 0x17), /* misc, interrupt */
+ regmap_reg_range(0x19, 0x19), /* endian test */
+ regmap_reg_range(0x1d, 0x1d), /* hardware config */
+ regmap_reg_range(0x23, 0x24), /* general purpose timer */
+ regmap_reg_range(0x27, 0x27), /* counter */
+ regmap_reg_range(0x29, 0x2a), /* PMI index regs */
+ regmap_reg_range(0x68, 0x6a), /* flow control */
+ regmap_reg_range(0x6b, 0x6c), /* switch fabric indirect regs */
+ regmap_reg_range(0x6d, 0x6f), /* misc */
+ regmap_reg_range(0x70, 0x77), /* virtual phy */
+ regmap_reg_range(0x78, 0x7a), /* GPIO */
+ regmap_reg_range(0x7c, 0x7e), /* MAC & reset */
+ regmap_reg_range(0x80, 0xb7), /* switch fabric direct regs (wr only) */
+};
+
+static const struct regmap_range lan9303_reserved_ranges[] = {
+ regmap_reg_range(0x00, 0x13),
+ regmap_reg_range(0x18, 0x18),
+ regmap_reg_range(0x1a, 0x1c),
+ regmap_reg_range(0x1e, 0x22),
+ regmap_reg_range(0x25, 0x26),
+ regmap_reg_range(0x28, 0x28),
+ regmap_reg_range(0x2b, 0x67),
+ regmap_reg_range(0x7b, 0x7b),
+ regmap_reg_range(0x7f, 0x7f),
+ regmap_reg_range(0xb8, 0xff),
+};
+
+const struct regmap_access_table lan9303_register_set = {
+ .yes_ranges = lan9303_valid_regs,
+ .n_yes_ranges = ARRAY_SIZE(lan9303_valid_regs),
+ .no_ranges = lan9303_reserved_ranges,
+ .n_no_ranges = ARRAY_SIZE(lan9303_reserved_ranges),
+};
+EXPORT_SYMBOL(lan9303_register_set);
+
+static int lan9303_read(struct regmap *regmap, unsigned int offset, u32 *reg)
+{
+ int ret, i;
+
+ /* we can lose arbitration for the I2C case, because the device
+ * tries to detect and read an external EEPROM after reset and acts as
+ * a master on the shared I2C bus itself. This conflicts with our
+ * attempts to access the device as a slave at the same moment.
+ */
+ for (i = 0; i < 5; i++) {
+ ret = regmap_read(regmap, offset, reg);
+ if (!ret)
+ return 0;
+ if (ret != -EAGAIN)
+ break;
+ msleep(500);
+ }
+
+ return -EIO;
+}
+
+static int lan9303_virt_phy_reg_read(struct lan9303 *chip, int regnum)
+{
+ int ret;
+ u32 val;
+
+ if (regnum > MII_EXPANSION)
+ return -EINVAL;
+
+ ret = lan9303_read(chip->regmap, LAN9303_VIRT_PHY_BASE + regnum, &val);
+ if (ret)
+ return ret;
+
+ return val & 0xffff;
+}
+
+static int lan9303_virt_phy_reg_write(struct lan9303 *chip, int regnum, u16 val)
+{
+ if (regnum > MII_EXPANSION)
+ return -EINVAL;
+
+ return regmap_write(chip->regmap, LAN9303_VIRT_PHY_BASE + regnum, val);
+}
+
+static int lan9303_port_phy_reg_wait_for_completion(struct lan9303 *chip)
+{
+ int ret, i;
+ u32 reg;
+
+ for (i = 0; i < 25; i++) {
+ ret = lan9303_read(chip->regmap, LAN9303_PMI_ACCESS, ®);
+ if (ret) {
+ dev_err(chip->dev,
+ "Failed to read pmi access status: %d\n", ret);
+ return ret;
+ }
+ if (!(reg & LAN9303_PMI_ACCESS_MII_BUSY))
+ return 0;
+ msleep(1);
+ }
+
+ return -EIO;
+}
+
+static int lan9303_port_phy_reg_read(struct lan9303 *chip, int addr, int regnum)
+{
+ int ret;
+ u32 val;
+
+ val = LAN9303_PMI_ACCESS_PHY_ADDR(addr);
+ val |= LAN9303_PMI_ACCESS_MIIRINDA(regnum);
+
+ mutex_lock(&chip->indirect_mutex);
+
+ ret = lan9303_port_phy_reg_wait_for_completion(chip);
+ if (ret)
+ goto on_error;
+
+ /* start the MII read cycle */
+ ret = regmap_write(chip->regmap, LAN9303_PMI_ACCESS, val);
+ if (ret)
+ goto on_error;
+
+ ret = lan9303_port_phy_reg_wait_for_completion(chip);
+ if (ret)
+ goto on_error;
+
+ /* read the result of this operation */
+ ret = lan9303_read(chip->regmap, LAN9303_PMI_DATA, &val);
+ if (ret)
+ goto on_error;
+
+ mutex_unlock(&chip->indirect_mutex);
+
+ return val & 0xffff;
+
+on_error:
+ mutex_unlock(&chip->indirect_mutex);
+ return ret;
+}
+
+static int lan9303_phy_reg_write(struct lan9303 *chip, int addr, int regnum,
+ unsigned int val)
+{
+ int ret;
+ u32 reg;
+
+ reg = LAN9303_PMI_ACCESS_PHY_ADDR(addr);
+ reg |= LAN9303_PMI_ACCESS_MIIRINDA(regnum);
+ reg |= LAN9303_PMI_ACCESS_MII_WRITE;
+
+ mutex_lock(&chip->indirect_mutex);
+
+ ret = lan9303_port_phy_reg_wait_for_completion(chip);
+ if (ret)
+ goto on_error;
+
+ /* write the data first... */
+ ret = regmap_write(chip->regmap, LAN9303_PMI_DATA, val);
+ if (ret)
+ goto on_error;
+
+ /* ...then start the MII write cycle */
+ ret = regmap_write(chip->regmap, LAN9303_PMI_ACCESS, reg);
+
+on_error:
+ mutex_unlock(&chip->indirect_mutex);
+ return ret;
+}
+
+static int lan9303_switch_wait_for_completion(struct lan9303 *chip)
+{
+ int ret, i;
+ u32 reg;
+
+ for (i = 0; i < 25; i++) {
+ ret = lan9303_read(chip->regmap, LAN9303_SWITCH_CSR_CMD, ®);
+ if (ret) {
+ dev_err(chip->dev,
+ "Failed to read csr command status: %d\n", ret);
+ return ret;
+ }
+ if (!(reg & LAN9303_SWITCH_CSR_CMD_BUSY))
+ return 0;
+ msleep(1);
+ }
+
+ return -EIO;
+}
+
+static int lan9303_write_switch_reg(struct lan9303 *chip, u16 regnum, u32 val)
+{
+ u32 reg;
+ int ret;
+
+ reg = regnum;
+ reg |= LAN9303_SWITCH_CSR_CMD_LANES;
+ reg |= LAN9303_SWITCH_CSR_CMD_BUSY;
+
+ mutex_lock(&chip->indirect_mutex);
+
+ ret = lan9303_switch_wait_for_completion(chip);
+ if (ret)
+ goto on_error;
+
+ ret = regmap_write(chip->regmap, LAN9303_SWITCH_CSR_DATA, val);
+ if (ret) {
+ dev_err(chip->dev, "Failed to write csr data reg: %d\n", ret);
+ goto on_error;
+ }
+
+ /* trigger write */
+ ret = regmap_write(chip->regmap, LAN9303_SWITCH_CSR_CMD, reg);
+ if (ret)
+ dev_err(chip->dev, "Failed to write csr command reg: %d\n",
+ ret);
+
+on_error:
+ mutex_unlock(&chip->indirect_mutex);
+ return ret;
+}
+
+static int lan9303_read_switch_reg(struct lan9303 *chip, u16 regnum, u32 *val)
+{
+ u32 reg;
+ int ret;
+
+ reg = regnum;
+ reg |= LAN9303_SWITCH_CSR_CMD_LANES;
+ reg |= LAN9303_SWITCH_CSR_CMD_RW;
+ reg |= LAN9303_SWITCH_CSR_CMD_BUSY;
+
+ mutex_lock(&chip->indirect_mutex);
+
+ ret = lan9303_switch_wait_for_completion(chip);
+ if (ret)
+ goto on_error;
+
+ /* trigger read */
+ ret = regmap_write(chip->regmap, LAN9303_SWITCH_CSR_CMD, reg);
+ if (ret) {
+ dev_err(chip->dev, "Failed to write csr command reg: %d\n",
+ ret);
+ goto on_error;
+ }
+
+ ret = lan9303_switch_wait_for_completion(chip);
+ if (ret)
+ goto on_error;
+
+ ret = lan9303_read(chip->regmap, LAN9303_SWITCH_CSR_DATA, val);
+ if (ret)
+ dev_err(chip->dev, "Failed to read csr data reg: %d\n", ret);
+on_error:
+ mutex_unlock(&chip->indirect_mutex);
+ return ret;
+}
+
+static int lan9303_detect_phy_setup(struct lan9303 *chip)
+{
+ int reg;
+
+ /* depending on the 'phy_addr_sel_strap' setting, the three phys are
+ * using IDs 0-1-2 or IDs 1-2-3. We cannot read back the
+ * 'phy_addr_sel_strap' setting directly, so we need a test, which
+ * configuration is active:
+ * Special reg 18 of phy 3 reads as 0x0000, if 'phy_addr_sel_strap' is 0
+ * and the IDs are 0-1-2, else it contains something different from
+ * 0x0000, which means 'phy_addr_sel_strap' is 1 and the IDs are 1-2-3.
+ */
+ reg = lan9303_port_phy_reg_read(chip, 3, MII_LAN911X_SPECIAL_MODES);
+ if (reg < 0) {
+ dev_err(chip->dev, "Failed to detect phy config: %d\n", reg);
+ return reg;
+ }
+
+ if (reg != 0)
+ chip->phy_addr_sel_strap = 1;
+ else
+ chip->phy_addr_sel_strap = 0;
+
+ dev_dbg(chip->dev, "Phy setup '%s' detected\n",
+ chip->phy_addr_sel_strap ? "1-2-3" : "0-1-2");
+
+ return 0;
+}
+
+#define LAN9303_MAC_RX_CFG_OFFS (LAN9303_MAC_RX_CFG_0 - LAN9303_PORT_0_OFFSET)
+#define LAN9303_MAC_TX_CFG_OFFS (LAN9303_MAC_TX_CFG_0 - LAN9303_PORT_0_OFFSET)
+
+static int lan9303_disable_packet_processing(struct lan9303 *chip,
+ unsigned int port)
+{
+ int ret;
+
+ /* disable RX, but keep register reset default values else */
+ ret = lan9303_write_switch_reg(chip, LAN9303_MAC_RX_CFG_OFFS + port,
+ LAN9303_MAC_RX_CFG_X_REJECT_MAC_TYPES);
+ if (ret)
+ return ret;
+
+ /* disable TX, but keep register reset default values else */
+ return lan9303_write_switch_reg(chip, LAN9303_MAC_TX_CFG_OFFS + port,
+ LAN9303_MAC_TX_CFG_X_TX_IFG_CONFIG_DEFAULT |
+ LAN9303_MAC_TX_CFG_X_TX_PAD_ENABLE);
+}
+
+static int lan9303_enable_packet_processing(struct lan9303 *chip,
+ unsigned int port)
+{
+ int ret;
+
+ /* enable RX and keep register reset default values else */
+ ret = lan9303_write_switch_reg(chip, LAN9303_MAC_RX_CFG_OFFS + port,
+ LAN9303_MAC_RX_CFG_X_REJECT_MAC_TYPES |
+ LAN9303_MAC_RX_CFG_X_RX_ENABLE);
+ if (ret)
+ return ret;
+
+ /* enable TX and keep register reset default values else */
+ return lan9303_write_switch_reg(chip, LAN9303_MAC_TX_CFG_OFFS + port,
+ LAN9303_MAC_TX_CFG_X_TX_IFG_CONFIG_DEFAULT |
+ LAN9303_MAC_TX_CFG_X_TX_PAD_ENABLE |
+ LAN9303_MAC_TX_CFG_X_TX_ENABLE);
+}
+
+/* We want a special working switch:
+ * - do not forward packets between port 1 and 2
+ * - forward everything from port 1 to port 0
+ * - forward everything from port 2 to port 0
+ * - forward special tagged packets from port 0 to port 1 *or* port 2
+ */
+static int lan9303_separate_ports(struct lan9303 *chip)
+{
+ int ret;
+
+ ret = lan9303_write_switch_reg(chip, LAN9303_SWE_PORT_MIRROR,
+ LAN9303_SWE_PORT_MIRROR_SNIFFER_PORT0 |
+ LAN9303_SWE_PORT_MIRROR_MIRRORED_PORT1 |
+ LAN9303_SWE_PORT_MIRROR_MIRRORED_PORT2 |
+ LAN9303_SWE_PORT_MIRROR_ENABLE_RX_MIRRORING |
+ LAN9303_SWE_PORT_MIRROR_SNIFF_ALL);
+ if (ret)
+ return ret;
+
+ /* enable defining the destination port via special VLAN tagging
+ * for port 0
+ */
+ ret = lan9303_write_switch_reg(chip, LAN9303_SWE_INGRESS_PORT_TYPE,
+ 0x03);
+ if (ret)
+ return ret;
+
+ /* tag incoming packets at port 1 and 2 on their way to port 0 to be
+ * able to discover their source port
+ */
+ ret = lan9303_write_switch_reg(chip, LAN9303_BM_EGRSS_PORT_TYPE,
+ LAN9303_BM_EGRSS_PORT_TYPE_SPECIAL_TAG_PORT0);
+ if (ret)
+ return ret;
+
+ /* prevent port 1 and 2 from forwarding packets by their own */
+ return lan9303_write_switch_reg(chip, LAN9303_SWE_PORT_STATE,
+ LAN9303_SWE_PORT_STATE_FORWARDING_PORT0 |
+ LAN9303_SWE_PORT_STATE_BLOCKING_PORT1 |
+ LAN9303_SWE_PORT_STATE_BLOCKING_PORT2);
+}
+
+static int lan9303_handle_reset(struct lan9303 *chip)
+{
+ if (!chip->reset_gpio)
+ return 0;
+
+ if (chip->reset_duration != 0)
+ msleep(chip->reset_duration);
+
+ /* release (deassert) reset and activate the device */
+ gpiod_set_value_cansleep(chip->reset_gpio, 0);
+
+ return 0;
+}
+
+/* stop processing packets for all ports */
+static int lan9303_disable_processing(struct lan9303 *chip)
+{
+ int ret;
+
+ ret = lan9303_disable_packet_processing(chip, LAN9303_PORT_0_OFFSET);
+ if (ret)
+ return ret;
+ ret = lan9303_disable_packet_processing(chip, LAN9303_PORT_1_OFFSET);
+ if (ret)
+ return ret;
+ return lan9303_disable_packet_processing(chip, LAN9303_PORT_2_OFFSET);
+}
+
+static int lan9303_check_device(struct lan9303 *chip)
+{
+ int ret;
+ u32 reg;
+
+ ret = lan9303_read(chip->regmap, LAN9303_CHIP_REV, ®);
+ if (ret) {
+ dev_err(chip->dev, "failed to read chip revision register: %d\n",
+ ret);
+ if (!chip->reset_gpio) {
+ dev_dbg(chip->dev,
+ "hint: maybe failed due to missing reset GPIO\n");
+ }
+ return ret;
+ }
+
+ if ((reg >> 16) != LAN9303_CHIP_ID) {
+ dev_err(chip->dev, "expecting LAN9303 chip, but found: %X\n",
+ reg >> 16);
+ return ret;
+ }
+
+ /* The default state of the LAN9303 device is to forward packets between
+ * all ports (if not configured differently by an external EEPROM).
+ * The initial state of a DSA device must be forwarding packets only
+ * between the external and the internal ports and no forwarding
+ * between the external ports. In preparation we stop packet handling
+ * at all for now until the LAN9303 device is re-programmed accordingly.
+ */
+ ret = lan9303_disable_processing(chip);
+ if (ret)
+ dev_warn(chip->dev, "failed to disable switching %d\n", ret);
+
+ dev_info(chip->dev, "Found LAN9303 rev. %u\n", reg & 0xffff);
+
+ ret = lan9303_detect_phy_setup(chip);
+ if (ret) {
+ dev_err(chip->dev,
+ "failed to discover phy bootstrap setup: %d\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+/* ---------------------------- DSA -----------------------------------*/
+
+static enum dsa_tag_protocol lan9303_get_tag_protocol(struct dsa_switch *ds)
+{
+ return DSA_TAG_PROTO_LAN9303;
+}
+
+static int lan9303_setup(struct dsa_switch *ds)
+{
+ struct lan9303 *chip = ds->priv;
+ int ret;
+
+ /* Make sure that port 0 is the cpu port */
+ if (!dsa_is_cpu_port(ds, 0)) {
+ dev_err(chip->dev, "port 0 is not the CPU port\n");
+ return -EINVAL;
+ }
+
+ ret = lan9303_separate_ports(chip);
+ if (ret)
+ dev_err(chip->dev, "failed to separate ports %d\n", ret);
+
+ ret = lan9303_enable_packet_processing(chip, LAN9303_PORT_0_OFFSET);
+ if (ret)
+ dev_err(chip->dev, "failed to re-enable switching %d\n", ret);
+
+ return 0;
+}
+
+struct lan9303_mib_desc {
+ unsigned int offset; /* offset of first MAC */
+ const char *name;
+};
+
+static const struct lan9303_mib_desc lan9303_mib[] = {
+ { .offset = LAN9303_MAC_RX_BRDCST_CNT_0, .name = "RxBroad", },
+ { .offset = LAN9303_MAC_RX_PAUSE_CNT_0, .name = "RxPause", },
+ { .offset = LAN9303_MAC_RX_MULCST_CNT_0, .name = "RxMulti", },
+ { .offset = LAN9303_MAC_RX_PKTOK_CNT_0, .name = "RxOk", },
+ { .offset = LAN9303_MAC_RX_CRCERR_CNT_0, .name = "RxCrcErr", },
+ { .offset = LAN9303_MAC_RX_ALIGN_CNT_0, .name = "RxAlignErr", },
+ { .offset = LAN9303_MAC_RX_JABB_CNT_0, .name = "RxJabber", },
+ { .offset = LAN9303_MAC_RX_FRAG_CNT_0, .name = "RxFragment", },
+ { .offset = LAN9303_MAC_RX_64_CNT_0, .name = "Rx64Byte", },
+ { .offset = LAN9303_MAC_RX_127_CNT_0, .name = "Rx128Byte", },
+ { .offset = LAN9303_MAC_RX_255_CNT_0, .name = "Rx256Byte", },
+ { .offset = LAN9303_MAC_RX_511_CNT_0, .name = "Rx512Byte", },
+ { .offset = LAN9303_MAC_RX_1023_CNT_0, .name = "Rx1024Byte", },
+ { .offset = LAN9303_MAC_RX_MAX_CNT_0, .name = "RxMaxByte", },
+ { .offset = LAN9303_MAC_RX_PKTLEN_CNT_0, .name = "RxByteCnt", },
+ { .offset = LAN9303_MAC_RX_SYMBL_CNT_0, .name = "RxSymbolCnt", },
+ { .offset = LAN9303_MAC_RX_CTLFRM_CNT_0, .name = "RxCfs", },
+ { .offset = LAN9303_MAC_RX_OVRSZE_CNT_0, .name = "RxOverFlow", },
+ { .offset = LAN9303_MAC_TX_UNDSZE_CNT_0, .name = "TxShort", },
+ { .offset = LAN9303_MAC_TX_BRDCST_CNT_0, .name = "TxBroad", },
+ { .offset = LAN9303_MAC_TX_PAUSE_CNT_0, .name = "TxPause", },
+ { .offset = LAN9303_MAC_TX_MULCST_CNT_0, .name = "TxMulti", },
+ { .offset = LAN9303_MAC_RX_UNDSZE_CNT_0, .name = "TxUnderRun", },
+ { .offset = LAN9303_MAC_TX_64_CNT_0, .name = "Tx64Byte", },
+ { .offset = LAN9303_MAC_TX_127_CNT_0, .name = "Tx128Byte", },
+ { .offset = LAN9303_MAC_TX_255_CNT_0, .name = "Tx256Byte", },
+ { .offset = LAN9303_MAC_TX_511_CNT_0, .name = "Tx512Byte", },
+ { .offset = LAN9303_MAC_TX_1023_CNT_0, .name = "Tx1024Byte", },
+ { .offset = LAN9303_MAC_TX_MAX_CNT_0, .name = "TxMaxByte", },
+ { .offset = LAN9303_MAC_TX_PKTLEN_CNT_0, .name = "TxByteCnt", },
+ { .offset = LAN9303_MAC_TX_PKTOK_CNT_0, .name = "TxOk", },
+ { .offset = LAN9303_MAC_TX_TOTALCOL_CNT_0, .name = "TxCollision", },
+ { .offset = LAN9303_MAC_TX_MULTICOL_CNT_0, .name = "TxMultiCol", },
+ { .offset = LAN9303_MAC_TX_SNGLECOL_CNT_0, .name = "TxSingleCol", },
+ { .offset = LAN9303_MAC_TX_EXCOL_CNT_0, .name = "TxExcCol", },
+ { .offset = LAN9303_MAC_TX_DEFER_CNT_0, .name = "TxDefer", },
+ { .offset = LAN9303_MAC_TX_LATECOL_0, .name = "TxLateCol", },
+};
+
+static void lan9303_get_strings(struct dsa_switch *ds, int port, uint8_t *data)
+{
+ unsigned int u;
+
+ for (u = 0; u < ARRAY_SIZE(lan9303_mib); u++) {
+ strncpy(data + u * ETH_GSTRING_LEN, lan9303_mib[u].name,
+ ETH_GSTRING_LEN);
+ }
+}
+
+static void lan9303_get_ethtool_stats(struct dsa_switch *ds, int port,
+ uint64_t *data)
+{
+ struct lan9303 *chip = ds->priv;
+ u32 reg;
+ unsigned int u, poff;
+ int ret;
+
+ poff = port * 0x400;
+
+ for (u = 0; u < ARRAY_SIZE(lan9303_mib); u++) {
+ ret = lan9303_read_switch_reg(chip,
+ lan9303_mib[u].offset + poff,
+ ®);
+ if (ret)
+ dev_warn(chip->dev, "Reading status reg %u failed\n",
+ lan9303_mib[u].offset + poff);
+ data[u] = reg;
+ }
+}
+
+static int lan9303_get_sset_count(struct dsa_switch *ds)
+{
+ return ARRAY_SIZE(lan9303_mib);
+}
+
+static int lan9303_phy_read(struct dsa_switch *ds, int phy, int regnum)
+{
+ struct lan9303 *chip = ds->priv;
+ int phy_base = chip->phy_addr_sel_strap;
+
+ if (phy == phy_base)
+ return lan9303_virt_phy_reg_read(chip, regnum);
+ if (phy > phy_base + 2)
+ return -ENODEV;
+
+ return lan9303_port_phy_reg_read(chip, phy, regnum);
+}
+
+static int lan9303_phy_write(struct dsa_switch *ds, int phy, int regnum,
+ u16 val)
+{
+ struct lan9303 *chip = ds->priv;
+ int phy_base = chip->phy_addr_sel_strap;
+
+ if (phy == phy_base)
+ return lan9303_virt_phy_reg_write(chip, regnum, val);
+ if (phy > phy_base + 2)
+ return -ENODEV;
+
+ return lan9303_phy_reg_write(chip, phy, regnum, val);
+}
+
+static int lan9303_port_enable(struct dsa_switch *ds, int port,
+ struct phy_device *phy)
+{
+ struct lan9303 *chip = ds->priv;
+
+ /* enable internal packet processing */
+ switch (port) {
+ case 1:
+ return lan9303_enable_packet_processing(chip,
+ LAN9303_PORT_1_OFFSET);
+ case 2:
+ return lan9303_enable_packet_processing(chip,
+ LAN9303_PORT_2_OFFSET);
+ default:
+ dev_dbg(chip->dev,
+ "Error: request to power up invalid port %d\n", port);
+ }
+
+ return -ENODEV;
+}
+
+static void lan9303_port_disable(struct dsa_switch *ds, int port,
+ struct phy_device *phy)
+{
+ struct lan9303 *chip = ds->priv;
+
+ /* disable internal packet processing */
+ switch (port) {
+ case 1:
+ lan9303_disable_packet_processing(chip, LAN9303_PORT_1_OFFSET);
+ lan9303_phy_reg_write(chip, chip->phy_addr_sel_strap + 1,
+ MII_BMCR, BMCR_PDOWN);
+ break;
+ case 2:
+ lan9303_disable_packet_processing(chip, LAN9303_PORT_2_OFFSET);
+ lan9303_phy_reg_write(chip, chip->phy_addr_sel_strap + 2,
+ MII_BMCR, BMCR_PDOWN);
+ break;
+ default:
+ dev_dbg(chip->dev,
+ "Error: request to power down invalid port %d\n", port);
+ }
+}
+
+static struct dsa_switch_ops lan9303_switch_ops = {
+ .get_tag_protocol = lan9303_get_tag_protocol,
+ .setup = lan9303_setup,
+ .get_strings = lan9303_get_strings,
+ .phy_read = lan9303_phy_read,
+ .phy_write = lan9303_phy_write,
+ .get_ethtool_stats = lan9303_get_ethtool_stats,
+ .get_sset_count = lan9303_get_sset_count,
+ .port_enable = lan9303_port_enable,
+ .port_disable = lan9303_port_disable,
+};
+
+static int lan9303_register_switch(struct lan9303 *chip)
+{
+ chip->ds = dsa_switch_alloc(chip->dev, DSA_MAX_PORTS);
+ if (!chip->ds)
+ return -ENOMEM;
+
+ chip->ds->priv = chip;
+ chip->ds->ops = &lan9303_switch_ops;
+ chip->ds->phys_mii_mask = chip->phy_addr_sel_strap ? 0xe : 0x7;
+
+ return dsa_register_switch(chip->ds, chip->dev);
+}
+
+static void lan9303_probe_reset_gpio(struct lan9303 *chip,
+ struct device_node *np)
+{
+ chip->reset_gpio = devm_gpiod_get_optional(chip->dev, "reset",
+ GPIOD_OUT_LOW);
+
+ if (!chip->reset_gpio) {
+ dev_dbg(chip->dev, "No reset GPIO defined\n");
+ return;
+ }
+
+ chip->reset_duration = 200;
+
+ if (np) {
+ of_property_read_u32(np, "reset-duration",
+ &chip->reset_duration);
+ } else {
+ dev_dbg(chip->dev, "reset duration defaults to 200 ms\n");
+ }
+
+ /* A sane reset duration should not be longer than 1s */
+ if (chip->reset_duration > 1000)
+ chip->reset_duration = 1000;
+}
+
+int lan9303_probe(struct lan9303 *chip, struct device_node *np)
+{
+ int ret;
+
+ mutex_init(&chip->indirect_mutex);
+
+ lan9303_probe_reset_gpio(chip, np);
+
+ ret = lan9303_handle_reset(chip);
+ if (ret)
+ return ret;
+
+ ret = lan9303_check_device(chip);
+ if (ret)
+ return ret;
+
+ ret = lan9303_register_switch(chip);
+ if (ret) {
+ dev_dbg(chip->dev, "Failed to register switch: %d\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(lan9303_probe);
+
+int lan9303_remove(struct lan9303 *chip)
+{
+ int rc;
+
+ rc = lan9303_disable_processing(chip);
+ if (rc != 0)
+ dev_warn(chip->dev, "shutting down failed\n");
+
+ dsa_unregister_switch(chip->ds);
+
+ /* assert reset to the whole device to prevent it from doing anything */
+ gpiod_set_value_cansleep(chip->reset_gpio, 1);
+ gpiod_unexport(chip->reset_gpio);
+
+ return 0;
+}
+EXPORT_SYMBOL(lan9303_remove);
+
+MODULE_AUTHOR("Juergen Borleis <kernel@pengutronix.de>");
+MODULE_DESCRIPTION("Core driver for SMSC/Microchip LAN9303 three port ethernet switch");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/net/dsa/lan9303.h b/drivers/net/dsa/lan9303.h
new file mode 100644
index 0000000..d1512da
--- /dev/null
+++ b/drivers/net/dsa/lan9303.h
@@ -0,0 +1,19 @@
+#include <linux/regmap.h>
+#include <linux/device.h>
+#include <net/dsa.h>
+
+struct lan9303 {
+ struct device *dev;
+ struct regmap *regmap;
+ struct regmap_irq_chip_data *irq_data;
+ struct gpio_desc *reset_gpio;
+ u32 reset_duration; /* in [ms] */
+ bool phy_addr_sel_strap;
+ struct dsa_switch *ds;
+ struct mutex indirect_mutex; /* protect indexed register access */
+};
+
+extern const struct regmap_access_table lan9303_register_set;
+
+int lan9303_probe(struct lan9303 *chip, struct device_node *np);
+int lan9303_remove(struct lan9303 *chip);
diff --git a/drivers/net/dsa/lan9303_i2c.c b/drivers/net/dsa/lan9303_i2c.c
new file mode 100644
index 0000000..ab3ce0d
--- /dev/null
+++ b/drivers/net/dsa/lan9303_i2c.c
@@ -0,0 +1,113 @@
+/*
+ * Copyright (C) 2017 Pengutronix, Juergen Borleis <kernel@pengutronix.de>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/i2c.h>
+#include <linux/of.h>
+
+#include "lan9303.h"
+
+struct lan9303_i2c {
+ struct i2c_client *device;
+ struct lan9303 chip;
+};
+
+static const struct regmap_config lan9303_i2c_regmap_config = {
+ .reg_bits = 8,
+ .val_bits = 32,
+ .reg_stride = 1,
+ .can_multi_write = true,
+ .max_register = 0x0ff, /* address bits 0..1 are not used */
+ .reg_format_endian = REGMAP_ENDIAN_LITTLE,
+
+ .volatile_table = &lan9303_register_set,
+ .wr_table = &lan9303_register_set,
+ .rd_table = &lan9303_register_set,
+
+ .cache_type = REGCACHE_NONE,
+};
+
+static int lan9303_i2c_probe(struct i2c_client *client,
+ const struct i2c_device_id *id)
+{
+ struct lan9303_i2c *sw_dev;
+ int ret;
+
+ sw_dev = devm_kzalloc(&client->dev, sizeof(struct lan9303_i2c),
+ GFP_KERNEL);
+ if (!sw_dev)
+ return -ENOMEM;
+
+ sw_dev->chip.regmap = devm_regmap_init_i2c(client,
+ &lan9303_i2c_regmap_config);
+ if (IS_ERR(sw_dev->chip.regmap)) {
+ ret = PTR_ERR(sw_dev->chip.regmap);
+ dev_err(&client->dev, "Failed to allocate register map: %d\n",
+ ret);
+ return ret;
+ }
+
+ /* link forward and backward */
+ sw_dev->device = client;
+ i2c_set_clientdata(client, sw_dev);
+ sw_dev->chip.dev = &client->dev;
+
+ ret = lan9303_probe(&sw_dev->chip, client->dev.of_node);
+ if (ret != 0)
+ return ret;
+
+ dev_info(&client->dev, "LAN9303 I2C driver loaded successfully\n");
+
+ return 0;
+}
+
+static int lan9303_i2c_remove(struct i2c_client *client)
+{
+ struct lan9303_i2c *sw_dev;
+
+ sw_dev = i2c_get_clientdata(client);
+ if (!sw_dev)
+ return -ENODEV;
+
+ return lan9303_remove(&sw_dev->chip);
+}
+
+/*-------------------------------------------------------------------------*/
+
+static const struct i2c_device_id lan9303_i2c_id[] = {
+ { "lan9303", 0 },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(i2c, lan9303_i2c_id);
+
+static const struct of_device_id lan9303_i2c_of_match[] = {
+ { .compatible = "smsc,lan9303-i2c", },
+ { /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, lan9303_i2c_of_match);
+
+static struct i2c_driver lan9303_i2c_driver = {
+ .driver = {
+ .name = "LAN9303_I2C",
+ .of_match_table = of_match_ptr(lan9303_i2c_of_match),
+ },
+ .probe = lan9303_i2c_probe,
+ .remove = lan9303_i2c_remove,
+ .id_table = lan9303_i2c_id,
+};
+module_i2c_driver(lan9303_i2c_driver);
+
+MODULE_AUTHOR("Juergen Borleis <kernel@pengutronix.de>");
+MODULE_DESCRIPTION("Driver for SMSC/Microchip LAN9303 three port ethernet switch in I2C managed mode");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/net/dsa/lan9303_mdio.c b/drivers/net/dsa/lan9303_mdio.c
new file mode 100644
index 0000000..93c36c0
--- /dev/null
+++ b/drivers/net/dsa/lan9303_mdio.c
@@ -0,0 +1,148 @@
+/*
+ * Copyright (C) 2017 Pengutronix, Juergen Borleis <kernel@pengutronix.de>
+ *
+ * Partially based on a patch from
+ * Copyright (c) 2014 Stefan Roese <sr@denx.de>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/mdio.h>
+#include <linux/phy.h>
+#include <linux/of.h>
+
+#include "lan9303.h"
+
+/* Generate phy-addr and -reg from the input address */
+#define PHY_ADDR(x) ((((x) >> 6) + 0x10) & 0x1f)
+#define PHY_REG(x) (((x) >> 1) & 0x1f)
+
+struct lan9303_mdio {
+ struct mdio_device *device;
+ struct lan9303 chip;
+};
+
+static void lan9303_mdio_real_write(struct mdio_device *mdio, int reg, u16 val)
+{
+ mdio->bus->write(mdio->bus, PHY_ADDR(reg), PHY_REG(reg), val);
+}
+
+static int lan9303_mdio_write(void *ctx, uint32_t reg, uint32_t val)
+{
+ struct lan9303_mdio *sw_dev = (struct lan9303_mdio *)ctx;
+
+ mutex_lock(&sw_dev->device->bus->mdio_lock);
+ lan9303_mdio_real_write(sw_dev->device, reg, val & 0xffff);
+ lan9303_mdio_real_write(sw_dev->device, reg + 2, (val >> 16) & 0xffff);
+ mutex_unlock(&sw_dev->device->bus->mdio_lock);
+
+ return 0;
+}
+
+static u16 lan9303_mdio_real_read(struct mdio_device *mdio, int reg)
+{
+ return mdio->bus->read(mdio->bus, PHY_ADDR(reg), PHY_REG(reg));
+}
+
+static int lan9303_mdio_read(void *ctx, uint32_t reg, uint32_t *val)
+{
+ struct lan9303_mdio *sw_dev = (struct lan9303_mdio *)ctx;
+
+ mutex_lock(&sw_dev->device->bus->mdio_lock);
+ *val = lan9303_mdio_real_read(sw_dev->device, reg);
+ *val |= (lan9303_mdio_real_read(sw_dev->device, reg + 2) << 16);
+ mutex_unlock(&sw_dev->device->bus->mdio_lock);
+
+ return 0;
+}
+
+static const struct regmap_config lan9303_mdio_regmap_config = {
+ .reg_bits = 8,
+ .val_bits = 32,
+ .reg_stride = 1,
+ .can_multi_write = true,
+ .max_register = 0x0ff, /* address bits 0..1 are not used */
+ .reg_format_endian = REGMAP_ENDIAN_LITTLE,
+
+ .volatile_table = &lan9303_register_set,
+ .wr_table = &lan9303_register_set,
+ .rd_table = &lan9303_register_set,
+
+ .reg_read = lan9303_mdio_read,
+ .reg_write = lan9303_mdio_write,
+
+ .cache_type = REGCACHE_NONE,
+};
+
+static int lan9303_mdio_probe(struct mdio_device *mdiodev)
+{
+ struct lan9303_mdio *sw_dev;
+ int ret;
+
+ sw_dev = devm_kzalloc(&mdiodev->dev, sizeof(struct lan9303_mdio),
+ GFP_KERNEL);
+ if (!sw_dev)
+ return -ENOMEM;
+
+ sw_dev->chip.regmap = devm_regmap_init(&mdiodev->dev, NULL, sw_dev,
+ &lan9303_mdio_regmap_config);
+ if (IS_ERR(sw_dev->chip.regmap)) {
+ ret = PTR_ERR(sw_dev->chip.regmap);
+ dev_err(&mdiodev->dev, "regmap init failed: %d\n", ret);
+ return ret;
+ }
+
+ /* link forward and backward */
+ sw_dev->device = mdiodev;
+ dev_set_drvdata(&mdiodev->dev, sw_dev);
+ sw_dev->chip.dev = &mdiodev->dev;
+
+ ret = lan9303_probe(&sw_dev->chip, mdiodev->dev.of_node);
+ if (ret != 0)
+ return ret;
+
+ dev_info(&mdiodev->dev, "LAN9303 MDIO driver loaded successfully\n");
+
+ return 0;
+}
+
+static void lan9303_mdio_remove(struct mdio_device *mdiodev)
+{
+ struct lan9303_mdio *sw_dev = dev_get_drvdata(&mdiodev->dev);
+
+ if (!sw_dev)
+ return;
+
+ lan9303_remove(&sw_dev->chip);
+}
+
+/*-------------------------------------------------------------------------*/
+
+static const struct of_device_id lan9303_mdio_of_match[] = {
+ { .compatible = "smsc,lan9303-mdio" },
+ { /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, lan9303_mdio_of_match);
+
+static struct mdio_driver lan9303_mdio_driver = {
+ .mdiodrv.driver = {
+ .name = "LAN9303_MDIO",
+ .of_match_table = of_match_ptr(lan9303_mdio_of_match),
+ },
+ .probe = lan9303_mdio_probe,
+ .remove = lan9303_mdio_remove,
+};
+mdio_module_driver(lan9303_mdio_driver);
+
+MODULE_AUTHOR("Stefan Roese <sr@denx.de>, Juergen Borleis <kernel@pengutronix.de>");
+MODULE_DESCRIPTION("Driver for SMSC/Microchip LAN9303 three port ethernet switch in MDIO managed mode");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/net/dsa/mt7530.c b/drivers/net/dsa/mt7530.c
new file mode 100644
index 0000000..b070c16
--- /dev/null
+++ b/drivers/net/dsa/mt7530.c
@@ -0,0 +1,1126 @@
+/*
+ * Mediatek MT7530 DSA Switch driver
+ * Copyright (C) 2017 Sean Wang <sean.wang@mediatek.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+#include <linux/etherdevice.h>
+#include <linux/if_bridge.h>
+#include <linux/iopoll.h>
+#include <linux/mdio.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/netdevice.h>
+#include <linux/of_gpio.h>
+#include <linux/of_mdio.h>
+#include <linux/of_net.h>
+#include <linux/of_platform.h>
+#include <linux/phy.h>
+#include <linux/regmap.h>
+#include <linux/regulator/consumer.h>
+#include <linux/reset.h>
+#include <linux/gpio/consumer.h>
+#include <net/dsa.h>
+#include <net/switchdev.h>
+
+#include "mt7530.h"
+
+/* String, offset, and register size in bytes if different from 4 bytes */
+static const struct mt7530_mib_desc mt7530_mib[] = {
+ MIB_DESC(1, 0x00, "TxDrop"),
+ MIB_DESC(1, 0x04, "TxCrcErr"),
+ MIB_DESC(1, 0x08, "TxUnicast"),
+ MIB_DESC(1, 0x0c, "TxMulticast"),
+ MIB_DESC(1, 0x10, "TxBroadcast"),
+ MIB_DESC(1, 0x14, "TxCollision"),
+ MIB_DESC(1, 0x18, "TxSingleCollision"),
+ MIB_DESC(1, 0x1c, "TxMultipleCollision"),
+ MIB_DESC(1, 0x20, "TxDeferred"),
+ MIB_DESC(1, 0x24, "TxLateCollision"),
+ MIB_DESC(1, 0x28, "TxExcessiveCollistion"),
+ MIB_DESC(1, 0x2c, "TxPause"),
+ MIB_DESC(1, 0x30, "TxPktSz64"),
+ MIB_DESC(1, 0x34, "TxPktSz65To127"),
+ MIB_DESC(1, 0x38, "TxPktSz128To255"),
+ MIB_DESC(1, 0x3c, "TxPktSz256To511"),
+ MIB_DESC(1, 0x40, "TxPktSz512To1023"),
+ MIB_DESC(1, 0x44, "Tx1024ToMax"),
+ MIB_DESC(2, 0x48, "TxBytes"),
+ MIB_DESC(1, 0x60, "RxDrop"),
+ MIB_DESC(1, 0x64, "RxFiltering"),
+ MIB_DESC(1, 0x6c, "RxMulticast"),
+ MIB_DESC(1, 0x70, "RxBroadcast"),
+ MIB_DESC(1, 0x74, "RxAlignErr"),
+ MIB_DESC(1, 0x78, "RxCrcErr"),
+ MIB_DESC(1, 0x7c, "RxUnderSizeErr"),
+ MIB_DESC(1, 0x80, "RxFragErr"),
+ MIB_DESC(1, 0x84, "RxOverSzErr"),
+ MIB_DESC(1, 0x88, "RxJabberErr"),
+ MIB_DESC(1, 0x8c, "RxPause"),
+ MIB_DESC(1, 0x90, "RxPktSz64"),
+ MIB_DESC(1, 0x94, "RxPktSz65To127"),
+ MIB_DESC(1, 0x98, "RxPktSz128To255"),
+ MIB_DESC(1, 0x9c, "RxPktSz256To511"),
+ MIB_DESC(1, 0xa0, "RxPktSz512To1023"),
+ MIB_DESC(1, 0xa4, "RxPktSz1024ToMax"),
+ MIB_DESC(2, 0xa8, "RxBytes"),
+ MIB_DESC(1, 0xb0, "RxCtrlDrop"),
+ MIB_DESC(1, 0xb4, "RxIngressDrop"),
+ MIB_DESC(1, 0xb8, "RxArlDrop"),
+};
+
+static int
+mt7623_trgmii_write(struct mt7530_priv *priv, u32 reg, u32 val)
+{
+ int ret;
+
+ ret = regmap_write(priv->ethernet, TRGMII_BASE(reg), val);
+ if (ret < 0)
+ dev_err(priv->dev,
+ "failed to priv write register\n");
+ return ret;
+}
+
+static u32
+mt7623_trgmii_read(struct mt7530_priv *priv, u32 reg)
+{
+ int ret;
+ u32 val;
+
+ ret = regmap_read(priv->ethernet, TRGMII_BASE(reg), &val);
+ if (ret < 0) {
+ dev_err(priv->dev,
+ "failed to priv read register\n");
+ return ret;
+ }
+
+ return val;
+}
+
+static void
+mt7623_trgmii_rmw(struct mt7530_priv *priv, u32 reg,
+ u32 mask, u32 set)
+{
+ u32 val;
+
+ val = mt7623_trgmii_read(priv, reg);
+ val &= ~mask;
+ val |= set;
+ mt7623_trgmii_write(priv, reg, val);
+}
+
+static void
+mt7623_trgmii_set(struct mt7530_priv *priv, u32 reg, u32 val)
+{
+ mt7623_trgmii_rmw(priv, reg, 0, val);
+}
+
+static void
+mt7623_trgmii_clear(struct mt7530_priv *priv, u32 reg, u32 val)
+{
+ mt7623_trgmii_rmw(priv, reg, val, 0);
+}
+
+static int
+core_read_mmd_indirect(struct mt7530_priv *priv, int prtad, int devad)
+{
+ struct mii_bus *bus = priv->bus;
+ int value, ret;
+
+ /* Write the desired MMD Devad */
+ ret = bus->write(bus, 0, MII_MMD_CTRL, devad);
+ if (ret < 0)
+ goto err;
+
+ /* Write the desired MMD register address */
+ ret = bus->write(bus, 0, MII_MMD_DATA, prtad);
+ if (ret < 0)
+ goto err;
+
+ /* Select the Function : DATA with no post increment */
+ ret = bus->write(bus, 0, MII_MMD_CTRL, (devad | MII_MMD_CTRL_NOINCR));
+ if (ret < 0)
+ goto err;
+
+ /* Read the content of the MMD's selected register */
+ value = bus->read(bus, 0, MII_MMD_DATA);
+
+ return value;
+err:
+ dev_err(&bus->dev, "failed to read mmd register\n");
+
+ return ret;
+}
+
+static int
+core_write_mmd_indirect(struct mt7530_priv *priv, int prtad,
+ int devad, u32 data)
+{
+ struct mii_bus *bus = priv->bus;
+ int ret;
+
+ /* Write the desired MMD Devad */
+ ret = bus->write(bus, 0, MII_MMD_CTRL, devad);
+ if (ret < 0)
+ goto err;
+
+ /* Write the desired MMD register address */
+ ret = bus->write(bus, 0, MII_MMD_DATA, prtad);
+ if (ret < 0)
+ goto err;
+
+ /* Select the Function : DATA with no post increment */
+ ret = bus->write(bus, 0, MII_MMD_CTRL, (devad | MII_MMD_CTRL_NOINCR));
+ if (ret < 0)
+ goto err;
+
+ /* Write the data into MMD's selected register */
+ ret = bus->write(bus, 0, MII_MMD_DATA, data);
+err:
+ if (ret < 0)
+ dev_err(&bus->dev,
+ "failed to write mmd register\n");
+ return ret;
+}
+
+static void
+core_write(struct mt7530_priv *priv, u32 reg, u32 val)
+{
+ struct mii_bus *bus = priv->bus;
+
+ mutex_lock_nested(&bus->mdio_lock, MDIO_MUTEX_NESTED);
+
+ core_write_mmd_indirect(priv, reg, MDIO_MMD_VEND2, val);
+
+ mutex_unlock(&bus->mdio_lock);
+}
+
+static void
+core_rmw(struct mt7530_priv *priv, u32 reg, u32 mask, u32 set)
+{
+ struct mii_bus *bus = priv->bus;
+ u32 val;
+
+ mutex_lock_nested(&bus->mdio_lock, MDIO_MUTEX_NESTED);
+
+ val = core_read_mmd_indirect(priv, reg, MDIO_MMD_VEND2);
+ val &= ~mask;
+ val |= set;
+ core_write_mmd_indirect(priv, reg, MDIO_MMD_VEND2, val);
+
+ mutex_unlock(&bus->mdio_lock);
+}
+
+static void
+core_set(struct mt7530_priv *priv, u32 reg, u32 val)
+{
+ core_rmw(priv, reg, 0, val);
+}
+
+static void
+core_clear(struct mt7530_priv *priv, u32 reg, u32 val)
+{
+ core_rmw(priv, reg, val, 0);
+}
+
+static int
+mt7530_mii_write(struct mt7530_priv *priv, u32 reg, u32 val)
+{
+ struct mii_bus *bus = priv->bus;
+ u16 page, r, lo, hi;
+ int ret;
+
+ page = (reg >> 6) & 0x3ff;
+ r = (reg >> 2) & 0xf;
+ lo = val & 0xffff;
+ hi = val >> 16;
+
+ /* MT7530 uses 31 as the pseudo port */
+ ret = bus->write(bus, 0x1f, 0x1f, page);
+ if (ret < 0)
+ goto err;
+
+ ret = bus->write(bus, 0x1f, r, lo);
+ if (ret < 0)
+ goto err;
+
+ ret = bus->write(bus, 0x1f, 0x10, hi);
+err:
+ if (ret < 0)
+ dev_err(&bus->dev,
+ "failed to write mt7530 register\n");
+ return ret;
+}
+
+static u32
+mt7530_mii_read(struct mt7530_priv *priv, u32 reg)
+{
+ struct mii_bus *bus = priv->bus;
+ u16 page, r, lo, hi;
+ int ret;
+
+ page = (reg >> 6) & 0x3ff;
+ r = (reg >> 2) & 0xf;
+
+ /* MT7530 uses 31 as the pseudo port */
+ ret = bus->write(bus, 0x1f, 0x1f, page);
+ if (ret < 0) {
+ dev_err(&bus->dev,
+ "failed to read mt7530 register\n");
+ return ret;
+ }
+
+ lo = bus->read(bus, 0x1f, r);
+ hi = bus->read(bus, 0x1f, 0x10);
+
+ return (hi << 16) | (lo & 0xffff);
+}
+
+static void
+mt7530_write(struct mt7530_priv *priv, u32 reg, u32 val)
+{
+ struct mii_bus *bus = priv->bus;
+
+ mutex_lock_nested(&bus->mdio_lock, MDIO_MUTEX_NESTED);
+
+ mt7530_mii_write(priv, reg, val);
+
+ mutex_unlock(&bus->mdio_lock);
+}
+
+static u32
+_mt7530_read(struct mt7530_dummy_poll *p)
+{
+ struct mii_bus *bus = p->priv->bus;
+ u32 val;
+
+ mutex_lock_nested(&bus->mdio_lock, MDIO_MUTEX_NESTED);
+
+ val = mt7530_mii_read(p->priv, p->reg);
+
+ mutex_unlock(&bus->mdio_lock);
+
+ return val;
+}
+
+static u32
+mt7530_read(struct mt7530_priv *priv, u32 reg)
+{
+ struct mt7530_dummy_poll p;
+
+ INIT_MT7530_DUMMY_POLL(&p, priv, reg);
+ return _mt7530_read(&p);
+}
+
+static void
+mt7530_rmw(struct mt7530_priv *priv, u32 reg,
+ u32 mask, u32 set)
+{
+ struct mii_bus *bus = priv->bus;
+ u32 val;
+
+ mutex_lock_nested(&bus->mdio_lock, MDIO_MUTEX_NESTED);
+
+ val = mt7530_mii_read(priv, reg);
+ val &= ~mask;
+ val |= set;
+ mt7530_mii_write(priv, reg, val);
+
+ mutex_unlock(&bus->mdio_lock);
+}
+
+static void
+mt7530_set(struct mt7530_priv *priv, u32 reg, u32 val)
+{
+ mt7530_rmw(priv, reg, 0, val);
+}
+
+static void
+mt7530_clear(struct mt7530_priv *priv, u32 reg, u32 val)
+{
+ mt7530_rmw(priv, reg, val, 0);
+}
+
+static int
+mt7530_fdb_cmd(struct mt7530_priv *priv, enum mt7530_fdb_cmd cmd, u32 *rsp)
+{
+ u32 val;
+ int ret;
+ struct mt7530_dummy_poll p;
+
+ /* Set the command operating upon the MAC address entries */
+ val = ATC_BUSY | ATC_MAT(0) | cmd;
+ mt7530_write(priv, MT7530_ATC, val);
+
+ INIT_MT7530_DUMMY_POLL(&p, priv, MT7530_ATC);
+ ret = readx_poll_timeout(_mt7530_read, &p, val,
+ !(val & ATC_BUSY), 20, 20000);
+ if (ret < 0) {
+ dev_err(priv->dev, "reset timeout\n");
+ return ret;
+ }
+
+ /* Additional sanity for read command if the specified
+ * entry is invalid
+ */
+ val = mt7530_read(priv, MT7530_ATC);
+ if ((cmd == MT7530_FDB_READ) && (val & ATC_INVALID))
+ return -EINVAL;
+
+ if (rsp)
+ *rsp = val;
+
+ return 0;
+}
+
+static void
+mt7530_fdb_read(struct mt7530_priv *priv, struct mt7530_fdb *fdb)
+{
+ u32 reg[3];
+ int i;
+
+ /* Read from ARL table into an array */
+ for (i = 0; i < 3; i++) {
+ reg[i] = mt7530_read(priv, MT7530_TSRA1 + (i * 4));
+
+ dev_dbg(priv->dev, "%s(%d) reg[%d]=0x%x\n",
+ __func__, __LINE__, i, reg[i]);
+ }
+
+ fdb->vid = (reg[1] >> CVID) & CVID_MASK;
+ fdb->aging = (reg[2] >> AGE_TIMER) & AGE_TIMER_MASK;
+ fdb->port_mask = (reg[2] >> PORT_MAP) & PORT_MAP_MASK;
+ fdb->mac[0] = (reg[0] >> MAC_BYTE_0) & MAC_BYTE_MASK;
+ fdb->mac[1] = (reg[0] >> MAC_BYTE_1) & MAC_BYTE_MASK;
+ fdb->mac[2] = (reg[0] >> MAC_BYTE_2) & MAC_BYTE_MASK;
+ fdb->mac[3] = (reg[0] >> MAC_BYTE_3) & MAC_BYTE_MASK;
+ fdb->mac[4] = (reg[1] >> MAC_BYTE_4) & MAC_BYTE_MASK;
+ fdb->mac[5] = (reg[1] >> MAC_BYTE_5) & MAC_BYTE_MASK;
+ fdb->noarp = ((reg[2] >> ENT_STATUS) & ENT_STATUS_MASK) == STATIC_ENT;
+}
+
+static void
+mt7530_fdb_write(struct mt7530_priv *priv, u16 vid,
+ u8 port_mask, const u8 *mac,
+ u8 aging, u8 type)
+{
+ u32 reg[3] = { 0 };
+ int i;
+
+ reg[1] |= vid & CVID_MASK;
+ reg[2] |= (aging & AGE_TIMER_MASK) << AGE_TIMER;
+ reg[2] |= (port_mask & PORT_MAP_MASK) << PORT_MAP;
+ /* STATIC_ENT indicate that entry is static wouldn't
+ * be aged out and STATIC_EMP specified as erasing an
+ * entry
+ */
+ reg[2] |= (type & ENT_STATUS_MASK) << ENT_STATUS;
+ reg[1] |= mac[5] << MAC_BYTE_5;
+ reg[1] |= mac[4] << MAC_BYTE_4;
+ reg[0] |= mac[3] << MAC_BYTE_3;
+ reg[0] |= mac[2] << MAC_BYTE_2;
+ reg[0] |= mac[1] << MAC_BYTE_1;
+ reg[0] |= mac[0] << MAC_BYTE_0;
+
+ /* Write array into the ARL table */
+ for (i = 0; i < 3; i++)
+ mt7530_write(priv, MT7530_ATA1 + (i * 4), reg[i]);
+}
+
+static int
+mt7530_pad_clk_setup(struct dsa_switch *ds, int mode)
+{
+ struct mt7530_priv *priv = ds->priv;
+ u32 ncpo1, ssc_delta, trgint, i;
+
+ switch (mode) {
+ case PHY_INTERFACE_MODE_RGMII:
+ trgint = 0;
+ ncpo1 = 0x0c80;
+ ssc_delta = 0x87;
+ break;
+ case PHY_INTERFACE_MODE_TRGMII:
+ trgint = 1;
+ ncpo1 = 0x1400;
+ ssc_delta = 0x57;
+ break;
+ default:
+ dev_err(priv->dev, "xMII mode %d not supported\n", mode);
+ return -EINVAL;
+ }
+
+ mt7530_rmw(priv, MT7530_P6ECR, P6_INTF_MODE_MASK,
+ P6_INTF_MODE(trgint));
+
+ /* Lower Tx Driving for TRGMII path */
+ for (i = 0 ; i < NUM_TRGMII_CTRL ; i++)
+ mt7530_write(priv, MT7530_TRGMII_TD_ODT(i),
+ TD_DM_DRVP(8) | TD_DM_DRVN(8));
+
+ /* Setup core clock for MT7530 */
+ if (!trgint) {
+ /* Disable MT7530 core clock */
+ core_clear(priv, CORE_TRGMII_GSW_CLK_CG, REG_GSWCK_EN);
+
+ /* Disable PLL, since phy_device has not yet been created
+ * provided for phy_[read,write]_mmd_indirect is called, we
+ * provide our own core_write_mmd_indirect to complete this
+ * function.
+ */
+ core_write_mmd_indirect(priv,
+ CORE_GSWPLL_GRP1,
+ MDIO_MMD_VEND2,
+ 0);
+
+ /* Set core clock into 500Mhz */
+ core_write(priv, CORE_GSWPLL_GRP2,
+ RG_GSWPLL_POSDIV_500M(1) |
+ RG_GSWPLL_FBKDIV_500M(25));
+
+ /* Enable PLL */
+ core_write(priv, CORE_GSWPLL_GRP1,
+ RG_GSWPLL_EN_PRE |
+ RG_GSWPLL_POSDIV_200M(2) |
+ RG_GSWPLL_FBKDIV_200M(32));
+
+ /* Enable MT7530 core clock */
+ core_set(priv, CORE_TRGMII_GSW_CLK_CG, REG_GSWCK_EN);
+ }
+
+ /* Setup the MT7530 TRGMII Tx Clock */
+ core_set(priv, CORE_TRGMII_GSW_CLK_CG, REG_GSWCK_EN);
+ core_write(priv, CORE_PLL_GROUP5, RG_LCDDS_PCW_NCPO1(ncpo1));
+ core_write(priv, CORE_PLL_GROUP6, RG_LCDDS_PCW_NCPO0(0));
+ core_write(priv, CORE_PLL_GROUP10, RG_LCDDS_SSC_DELTA(ssc_delta));
+ core_write(priv, CORE_PLL_GROUP11, RG_LCDDS_SSC_DELTA1(ssc_delta));
+ core_write(priv, CORE_PLL_GROUP4,
+ RG_SYSPLL_DDSFBK_EN | RG_SYSPLL_BIAS_EN |
+ RG_SYSPLL_BIAS_LPF_EN);
+ core_write(priv, CORE_PLL_GROUP2,
+ RG_SYSPLL_EN_NORMAL | RG_SYSPLL_VODEN |
+ RG_SYSPLL_POSDIV(1));
+ core_write(priv, CORE_PLL_GROUP7,
+ RG_LCDDS_PCW_NCPO_CHG | RG_LCCDS_C(3) |
+ RG_LCDDS_PWDB | RG_LCDDS_ISO_EN);
+ core_set(priv, CORE_TRGMII_GSW_CLK_CG,
+ REG_GSWCK_EN | REG_TRGMIICK_EN);
+
+ if (!trgint)
+ for (i = 0 ; i < NUM_TRGMII_CTRL; i++)
+ mt7530_rmw(priv, MT7530_TRGMII_RD(i),
+ RD_TAP_MASK, RD_TAP(16));
+ else
+ mt7623_trgmii_set(priv, GSW_INTF_MODE, INTF_MODE_TRGMII);
+
+ return 0;
+}
+
+static int
+mt7623_pad_clk_setup(struct dsa_switch *ds)
+{
+ struct mt7530_priv *priv = ds->priv;
+ int i;
+
+ for (i = 0 ; i < NUM_TRGMII_CTRL; i++)
+ mt7623_trgmii_write(priv, GSW_TRGMII_TD_ODT(i),
+ TD_DM_DRVP(8) | TD_DM_DRVN(8));
+
+ mt7623_trgmii_set(priv, GSW_TRGMII_RCK_CTRL, RX_RST | RXC_DQSISEL);
+ mt7623_trgmii_clear(priv, GSW_TRGMII_RCK_CTRL, RX_RST);
+
+ return 0;
+}
+
+static void
+mt7530_mib_reset(struct dsa_switch *ds)
+{
+ struct mt7530_priv *priv = ds->priv;
+
+ mt7530_write(priv, MT7530_MIB_CCR, CCR_MIB_FLUSH);
+ mt7530_write(priv, MT7530_MIB_CCR, CCR_MIB_ACTIVATE);
+}
+
+static void
+mt7530_port_set_status(struct mt7530_priv *priv, int port, int enable)
+{
+ u32 mask = PMCR_TX_EN | PMCR_RX_EN;
+
+ if (enable)
+ mt7530_set(priv, MT7530_PMCR_P(port), mask);
+ else
+ mt7530_clear(priv, MT7530_PMCR_P(port), mask);
+}
+
+static int mt7530_phy_read(struct dsa_switch *ds, int port, int regnum)
+{
+ struct mt7530_priv *priv = ds->priv;
+
+ return mdiobus_read_nested(priv->bus, port, regnum);
+}
+
+int mt7530_phy_write(struct dsa_switch *ds, int port, int regnum, u16 val)
+{
+ struct mt7530_priv *priv = ds->priv;
+
+ return mdiobus_write_nested(priv->bus, port, regnum, val);
+}
+
+static void
+mt7530_get_strings(struct dsa_switch *ds, int port, uint8_t *data)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(mt7530_mib); i++)
+ strncpy(data + i * ETH_GSTRING_LEN, mt7530_mib[i].name,
+ ETH_GSTRING_LEN);
+}
+
+static void
+mt7530_get_ethtool_stats(struct dsa_switch *ds, int port,
+ uint64_t *data)
+{
+ struct mt7530_priv *priv = ds->priv;
+ const struct mt7530_mib_desc *mib;
+ u32 reg, i;
+ u64 hi;
+
+ for (i = 0; i < ARRAY_SIZE(mt7530_mib); i++) {
+ mib = &mt7530_mib[i];
+ reg = MT7530_PORT_MIB_COUNTER(port) + mib->offset;
+
+ data[i] = mt7530_read(priv, reg);
+ if (mib->size == 2) {
+ hi = mt7530_read(priv, reg + 4);
+ data[i] |= hi << 32;
+ }
+ }
+}
+
+static int
+mt7530_get_sset_count(struct dsa_switch *ds)
+{
+ return ARRAY_SIZE(mt7530_mib);
+}
+
+static void mt7530_adjust_link(struct dsa_switch *ds, int port,
+ struct phy_device *phydev)
+{
+ struct mt7530_priv *priv = ds->priv;
+
+ if (phy_is_pseudo_fixed_link(phydev)) {
+ dev_dbg(priv->dev, "phy-mode for master device = %x\n",
+ phydev->interface);
+
+ /* Setup TX circuit incluing relevant PAD and driving */
+ mt7530_pad_clk_setup(ds, phydev->interface);
+
+ /* Setup RX circuit, relevant PAD and driving on the host
+ * which must be placed after the setup on the device side is
+ * all finished.
+ */
+ mt7623_pad_clk_setup(ds);
+ }
+}
+
+static int
+mt7530_cpu_port_enable(struct mt7530_priv *priv,
+ int port)
+{
+ /* Enable Mediatek header mode on the cpu port */
+ mt7530_write(priv, MT7530_PVC_P(port),
+ PORT_SPEC_TAG);
+
+ /* Setup the MAC by default for the cpu port */
+ mt7530_write(priv, MT7530_PMCR_P(port), PMCR_CPUP_LINK);
+
+ /* Disable auto learning on the cpu port */
+ mt7530_set(priv, MT7530_PSC_P(port), SA_DIS);
+
+ /* Unknown unicast frame fordwarding to the cpu port */
+ mt7530_set(priv, MT7530_MFC, UNU_FFP(BIT(port)));
+
+ /* CPU port gets connected to all user ports of
+ * the switch
+ */
+ mt7530_write(priv, MT7530_PCR_P(port),
+ PCR_MATRIX(priv->ds->enabled_port_mask));
+
+ return 0;
+}
+
+static int
+mt7530_port_enable(struct dsa_switch *ds, int port,
+ struct phy_device *phy)
+{
+ struct mt7530_priv *priv = ds->priv;
+
+ mutex_lock(&priv->reg_mutex);
+
+ /* Setup the MAC for the user port */
+ mt7530_write(priv, MT7530_PMCR_P(port), PMCR_USERP_LINK);
+
+ /* Allow the user port gets connected to the cpu port and also
+ * restore the port matrix if the port is the member of a certain
+ * bridge.
+ */
+ priv->ports[port].pm |= PCR_MATRIX(BIT(MT7530_CPU_PORT));
+ priv->ports[port].enable = true;
+ mt7530_rmw(priv, MT7530_PCR_P(port), PCR_MATRIX_MASK,
+ priv->ports[port].pm);
+ mt7530_port_set_status(priv, port, 1);
+
+ mutex_unlock(&priv->reg_mutex);
+
+ return 0;
+}
+
+static void
+mt7530_port_disable(struct dsa_switch *ds, int port,
+ struct phy_device *phy)
+{
+ struct mt7530_priv *priv = ds->priv;
+
+ mutex_lock(&priv->reg_mutex);
+
+ /* Clear up all port matrix which could be restored in the next
+ * enablement for the port.
+ */
+ priv->ports[port].enable = false;
+ mt7530_rmw(priv, MT7530_PCR_P(port), PCR_MATRIX_MASK,
+ PCR_MATRIX_CLR);
+ mt7530_port_set_status(priv, port, 0);
+
+ mutex_unlock(&priv->reg_mutex);
+}
+
+static void
+mt7530_stp_state_set(struct dsa_switch *ds, int port, u8 state)
+{
+ struct mt7530_priv *priv = ds->priv;
+ u32 stp_state;
+
+ switch (state) {
+ case BR_STATE_DISABLED:
+ stp_state = MT7530_STP_DISABLED;
+ break;
+ case BR_STATE_BLOCKING:
+ stp_state = MT7530_STP_BLOCKING;
+ break;
+ case BR_STATE_LISTENING:
+ stp_state = MT7530_STP_LISTENING;
+ break;
+ case BR_STATE_LEARNING:
+ stp_state = MT7530_STP_LEARNING;
+ break;
+ case BR_STATE_FORWARDING:
+ default:
+ stp_state = MT7530_STP_FORWARDING;
+ break;
+ }
+
+ mt7530_rmw(priv, MT7530_SSP_P(port), FID_PST_MASK, stp_state);
+}
+
+static int
+mt7530_port_bridge_join(struct dsa_switch *ds, int port,
+ struct net_device *bridge)
+{
+ struct mt7530_priv *priv = ds->priv;
+ u32 port_bitmap = BIT(MT7530_CPU_PORT);
+ int i;
+
+ mutex_lock(&priv->reg_mutex);
+
+ for (i = 0; i < MT7530_NUM_PORTS; i++) {
+ /* Add this port to the port matrix of the other ports in the
+ * same bridge. If the port is disabled, port matrix is kept
+ * and not being setup until the port becomes enabled.
+ */
+ if (ds->enabled_port_mask & BIT(i) && i != port) {
+ if (ds->ports[i].bridge_dev != bridge)
+ continue;
+ if (priv->ports[i].enable)
+ mt7530_set(priv, MT7530_PCR_P(i),
+ PCR_MATRIX(BIT(port)));
+ priv->ports[i].pm |= PCR_MATRIX(BIT(port));
+
+ port_bitmap |= BIT(i);
+ }
+ }
+
+ /* Add the all other ports to this port matrix. */
+ if (priv->ports[port].enable)
+ mt7530_rmw(priv, MT7530_PCR_P(port),
+ PCR_MATRIX_MASK, PCR_MATRIX(port_bitmap));
+ priv->ports[port].pm |= PCR_MATRIX(port_bitmap);
+
+ mutex_unlock(&priv->reg_mutex);
+
+ return 0;
+}
+
+static void
+mt7530_port_bridge_leave(struct dsa_switch *ds, int port,
+ struct net_device *bridge)
+{
+ struct mt7530_priv *priv = ds->priv;
+ int i;
+
+ mutex_lock(&priv->reg_mutex);
+
+ for (i = 0; i < MT7530_NUM_PORTS; i++) {
+ /* Remove this port from the port matrix of the other ports
+ * in the same bridge. If the port is disabled, port matrix
+ * is kept and not being setup until the port becomes enabled.
+ */
+ if (ds->enabled_port_mask & BIT(i) && i != port) {
+ if (ds->ports[i].bridge_dev != bridge)
+ continue;
+ if (priv->ports[i].enable)
+ mt7530_clear(priv, MT7530_PCR_P(i),
+ PCR_MATRIX(BIT(port)));
+ priv->ports[i].pm &= ~PCR_MATRIX(BIT(port));
+ }
+ }
+
+ /* Set the cpu port to be the only one in the port matrix of
+ * this port.
+ */
+ if (priv->ports[port].enable)
+ mt7530_rmw(priv, MT7530_PCR_P(port), PCR_MATRIX_MASK,
+ PCR_MATRIX(BIT(MT7530_CPU_PORT)));
+ priv->ports[port].pm = PCR_MATRIX(BIT(MT7530_CPU_PORT));
+
+ mutex_unlock(&priv->reg_mutex);
+}
+
+static int
+mt7530_port_fdb_prepare(struct dsa_switch *ds, int port,
+ const struct switchdev_obj_port_fdb *fdb,
+ struct switchdev_trans *trans)
+{
+ struct mt7530_priv *priv = ds->priv;
+ int ret;
+
+ /* Because auto-learned entrie shares the same FDB table.
+ * an entry is reserved with no port_mask to make sure fdb_add
+ * is called while the entry is still available.
+ */
+ mutex_lock(&priv->reg_mutex);
+ mt7530_fdb_write(priv, fdb->vid, 0, fdb->addr, -1, STATIC_ENT);
+ ret = mt7530_fdb_cmd(priv, MT7530_FDB_WRITE, 0);
+ mutex_unlock(&priv->reg_mutex);
+
+ return ret;
+}
+
+static void
+mt7530_port_fdb_add(struct dsa_switch *ds, int port,
+ const struct switchdev_obj_port_fdb *fdb,
+ struct switchdev_trans *trans)
+{
+ struct mt7530_priv *priv = ds->priv;
+ u8 port_mask = BIT(port);
+
+ mutex_lock(&priv->reg_mutex);
+ mt7530_fdb_write(priv, fdb->vid, port_mask, fdb->addr, -1, STATIC_ENT);
+ mt7530_fdb_cmd(priv, MT7530_FDB_WRITE, 0);
+ mutex_unlock(&priv->reg_mutex);
+}
+
+static int
+mt7530_port_fdb_del(struct dsa_switch *ds, int port,
+ const struct switchdev_obj_port_fdb *fdb)
+{
+ struct mt7530_priv *priv = ds->priv;
+ int ret;
+ u8 port_mask = BIT(port);
+
+ mutex_lock(&priv->reg_mutex);
+ mt7530_fdb_write(priv, fdb->vid, port_mask, fdb->addr, -1, STATIC_EMP);
+ ret = mt7530_fdb_cmd(priv, MT7530_FDB_WRITE, 0);
+ mutex_unlock(&priv->reg_mutex);
+
+ return ret;
+}
+
+static int
+mt7530_port_fdb_dump(struct dsa_switch *ds, int port,
+ struct switchdev_obj_port_fdb *fdb,
+ int (*cb)(struct switchdev_obj *obj))
+{
+ struct mt7530_priv *priv = ds->priv;
+ struct mt7530_fdb _fdb = { 0 };
+ int cnt = MT7530_NUM_FDB_RECORDS;
+ int ret = 0;
+ u32 rsp = 0;
+
+ mutex_lock(&priv->reg_mutex);
+
+ ret = mt7530_fdb_cmd(priv, MT7530_FDB_START, &rsp);
+ if (ret < 0)
+ goto err;
+
+ do {
+ if (rsp & ATC_SRCH_HIT) {
+ mt7530_fdb_read(priv, &_fdb);
+ if (_fdb.port_mask & BIT(port)) {
+ ether_addr_copy(fdb->addr, _fdb.mac);
+ fdb->vid = _fdb.vid;
+ fdb->ndm_state = _fdb.noarp ?
+ NUD_NOARP : NUD_REACHABLE;
+ ret = cb(&fdb->obj);
+ if (ret < 0)
+ break;
+ }
+ }
+ } while (--cnt &&
+ !(rsp & ATC_SRCH_END) &&
+ !mt7530_fdb_cmd(priv, MT7530_FDB_NEXT, &rsp));
+err:
+ mutex_unlock(&priv->reg_mutex);
+
+ return 0;
+}
+
+static enum dsa_tag_protocol
+mtk_get_tag_protocol(struct dsa_switch *ds)
+{
+ struct mt7530_priv *priv = ds->priv;
+
+ if (!dsa_is_cpu_port(ds, MT7530_CPU_PORT)) {
+ dev_warn(priv->dev,
+ "port not matched with tagging CPU port\n");
+ return DSA_TAG_PROTO_NONE;
+ } else {
+ return DSA_TAG_PROTO_MTK;
+ }
+}
+
+static int
+mt7530_setup(struct dsa_switch *ds)
+{
+ struct mt7530_priv *priv = ds->priv;
+ int ret, i;
+ u32 id, val;
+ struct device_node *dn;
+ struct mt7530_dummy_poll p;
+
+ /* The parent node of master_netdev which holds the common system
+ * controller also is the container for two GMACs nodes representing
+ * as two netdev instances.
+ */
+ dn = ds->master_netdev->dev.of_node->parent;
+ priv->ethernet = syscon_node_to_regmap(dn);
+ if (IS_ERR(priv->ethernet))
+ return PTR_ERR(priv->ethernet);
+
+ regulator_set_voltage(priv->core_pwr, 1000000, 1000000);
+ ret = regulator_enable(priv->core_pwr);
+ if (ret < 0) {
+ dev_err(priv->dev,
+ "Failed to enable core power: %d\n", ret);
+ return ret;
+ }
+
+ regulator_set_voltage(priv->io_pwr, 3300000, 3300000);
+ ret = regulator_enable(priv->io_pwr);
+ if (ret < 0) {
+ dev_err(priv->dev, "Failed to enable io pwr: %d\n",
+ ret);
+ return ret;
+ }
+
+ /* Reset whole chip through gpio pin or memory-mapped registers for
+ * different type of hardware
+ */
+ if (priv->mcm) {
+ reset_control_assert(priv->rstc);
+ usleep_range(1000, 1100);
+ reset_control_deassert(priv->rstc);
+ } else {
+ gpiod_set_value_cansleep(priv->reset, 0);
+ usleep_range(1000, 1100);
+ gpiod_set_value_cansleep(priv->reset, 1);
+ }
+
+ /* Waiting for MT7530 got to stable */
+ INIT_MT7530_DUMMY_POLL(&p, priv, MT7530_HWTRAP);
+ ret = readx_poll_timeout(_mt7530_read, &p, val, val != 0,
+ 20, 1000000);
+ if (ret < 0) {
+ dev_err(priv->dev, "reset timeout\n");
+ return ret;
+ }
+
+ id = mt7530_read(priv, MT7530_CREV);
+ id >>= CHIP_NAME_SHIFT;
+ if (id != MT7530_ID) {
+ dev_err(priv->dev, "chip %x can't be supported\n", id);
+ return -ENODEV;
+ }
+
+ /* Reset the switch through internal reset */
+ mt7530_write(priv, MT7530_SYS_CTRL,
+ SYS_CTRL_PHY_RST | SYS_CTRL_SW_RST |
+ SYS_CTRL_REG_RST);
+
+ /* Enable Port 6 only; P5 as GMAC5 which currently is not supported */
+ val = mt7530_read(priv, MT7530_MHWTRAP);
+ val &= ~MHWTRAP_P6_DIS & ~MHWTRAP_PHY_ACCESS;
+ val |= MHWTRAP_MANUAL;
+ mt7530_write(priv, MT7530_MHWTRAP, val);
+
+ /* Enable and reset MIB counters */
+ mt7530_mib_reset(ds);
+
+ mt7530_clear(priv, MT7530_MFC, UNU_FFP_MASK);
+
+ for (i = 0; i < MT7530_NUM_PORTS; i++) {
+ /* Disable forwarding by default on all ports */
+ mt7530_rmw(priv, MT7530_PCR_P(i), PCR_MATRIX_MASK,
+ PCR_MATRIX_CLR);
+
+ if (dsa_is_cpu_port(ds, i))
+ mt7530_cpu_port_enable(priv, i);
+ else
+ mt7530_port_disable(ds, i, NULL);
+ }
+
+ /* Flush the FDB table */
+ ret = mt7530_fdb_cmd(priv, MT7530_FDB_FLUSH, 0);
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+static struct dsa_switch_ops mt7530_switch_ops = {
+ .get_tag_protocol = mtk_get_tag_protocol,
+ .setup = mt7530_setup,
+ .get_strings = mt7530_get_strings,
+ .phy_read = mt7530_phy_read,
+ .phy_write = mt7530_phy_write,
+ .get_ethtool_stats = mt7530_get_ethtool_stats,
+ .get_sset_count = mt7530_get_sset_count,
+ .adjust_link = mt7530_adjust_link,
+ .port_enable = mt7530_port_enable,
+ .port_disable = mt7530_port_disable,
+ .port_stp_state_set = mt7530_stp_state_set,
+ .port_bridge_join = mt7530_port_bridge_join,
+ .port_bridge_leave = mt7530_port_bridge_leave,
+ .port_fdb_prepare = mt7530_port_fdb_prepare,
+ .port_fdb_add = mt7530_port_fdb_add,
+ .port_fdb_del = mt7530_port_fdb_del,
+ .port_fdb_dump = mt7530_port_fdb_dump,
+};
+
+static int
+mt7530_probe(struct mdio_device *mdiodev)
+{
+ struct mt7530_priv *priv;
+ struct device_node *dn;
+
+ dn = mdiodev->dev.of_node;
+
+ priv = devm_kzalloc(&mdiodev->dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ priv->ds = dsa_switch_alloc(&mdiodev->dev, DSA_MAX_PORTS);
+ if (!priv->ds)
+ return -ENOMEM;
+
+ /* Use medatek,mcm property to distinguish hardware type that would
+ * casues a little bit differences on power-on sequence.
+ */
+ priv->mcm = of_property_read_bool(dn, "mediatek,mcm");
+ if (priv->mcm) {
+ dev_info(&mdiodev->dev, "MT7530 adapts as multi-chip module\n");
+
+ priv->rstc = devm_reset_control_get(&mdiodev->dev, "mcm");
+ if (IS_ERR(priv->rstc)) {
+ dev_err(&mdiodev->dev, "Couldn't get our reset line\n");
+ return PTR_ERR(priv->rstc);
+ }
+ }
+
+ priv->core_pwr = devm_regulator_get(&mdiodev->dev, "core");
+ if (IS_ERR(priv->core_pwr))
+ return PTR_ERR(priv->core_pwr);
+
+ priv->io_pwr = devm_regulator_get(&mdiodev->dev, "io");
+ if (IS_ERR(priv->io_pwr))
+ return PTR_ERR(priv->io_pwr);
+
+ /* Not MCM that indicates switch works as the remote standalone
+ * integrated circuit so the GPIO pin would be used to complete
+ * the reset, otherwise memory-mapped register accessing used
+ * through syscon provides in the case of MCM.
+ */
+ if (!priv->mcm) {
+ priv->reset = devm_gpiod_get_optional(&mdiodev->dev, "reset",
+ GPIOD_OUT_LOW);
+ if (IS_ERR(priv->reset)) {
+ dev_err(&mdiodev->dev, "Couldn't get our reset line\n");
+ return PTR_ERR(priv->reset);
+ }
+ }
+
+ priv->bus = mdiodev->bus;
+ priv->dev = &mdiodev->dev;
+ priv->ds->priv = priv;
+ priv->ds->ops = &mt7530_switch_ops;
+ mutex_init(&priv->reg_mutex);
+ dev_set_drvdata(&mdiodev->dev, priv);
+
+ return dsa_register_switch(priv->ds, &mdiodev->dev);
+}
+
+static void
+mt7530_remove(struct mdio_device *mdiodev)
+{
+ struct mt7530_priv *priv = dev_get_drvdata(&mdiodev->dev);
+ int ret = 0;
+
+ ret = regulator_disable(priv->core_pwr);
+ if (ret < 0)
+ dev_err(priv->dev,
+ "Failed to disable core power: %d\n", ret);
+
+ ret = regulator_disable(priv->io_pwr);
+ if (ret < 0)
+ dev_err(priv->dev, "Failed to disable io pwr: %d\n",
+ ret);
+
+ dsa_unregister_switch(priv->ds);
+ mutex_destroy(&priv->reg_mutex);
+}
+
+static const struct of_device_id mt7530_of_match[] = {
+ { .compatible = "mediatek,mt7530" },
+ { /* sentinel */ },
+};
+
+static struct mdio_driver mt7530_mdio_driver = {
+ .probe = mt7530_probe,
+ .remove = mt7530_remove,
+ .mdiodrv.driver = {
+ .name = "mt7530",
+ .of_match_table = mt7530_of_match,
+ },
+};
+
+mdio_module_driver(mt7530_mdio_driver);
+
+MODULE_AUTHOR("Sean Wang <sean.wang@mediatek.com>");
+MODULE_DESCRIPTION("Driver for Mediatek MT7530 Switch");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:mediatek-mt7530");
diff --git a/drivers/net/dsa/mt7530.h b/drivers/net/dsa/mt7530.h
new file mode 100644
index 0000000..b83d76b
--- /dev/null
+++ b/drivers/net/dsa/mt7530.h
@@ -0,0 +1,402 @@
+/*
+ * Copyright (C) 2017 Sean Wang <sean.wang@mediatek.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __MT7530_H
+#define __MT7530_H
+
+#define MT7530_NUM_PORTS 7
+#define MT7530_CPU_PORT 6
+#define MT7530_NUM_FDB_RECORDS 2048
+
+#define NUM_TRGMII_CTRL 5
+
+#define TRGMII_BASE(x) (0x10000 + (x))
+
+/* Registers to ethsys access */
+#define ETHSYS_CLKCFG0 0x2c
+#define ETHSYS_TRGMII_CLK_SEL362_5 BIT(11)
+
+#define SYSC_REG_RSTCTRL 0x34
+#define RESET_MCM BIT(2)
+
+/* Registers to mac forward control for unknown frames */
+#define MT7530_MFC 0x10
+#define BC_FFP(x) (((x) & 0xff) << 24)
+#define UNM_FFP(x) (((x) & 0xff) << 16)
+#define UNU_FFP(x) (((x) & 0xff) << 8)
+#define UNU_FFP_MASK UNU_FFP(~0)
+
+/* Registers for address table access */
+#define MT7530_ATA1 0x74
+#define STATIC_EMP 0
+#define STATIC_ENT 3
+#define MT7530_ATA2 0x78
+
+/* Register for address table write data */
+#define MT7530_ATWD 0x7c
+
+/* Register for address table control */
+#define MT7530_ATC 0x80
+#define ATC_HASH (((x) & 0xfff) << 16)
+#define ATC_BUSY BIT(15)
+#define ATC_SRCH_END BIT(14)
+#define ATC_SRCH_HIT BIT(13)
+#define ATC_INVALID BIT(12)
+#define ATC_MAT(x) (((x) & 0xf) << 8)
+#define ATC_MAT_MACTAB ATC_MAT(0)
+
+enum mt7530_fdb_cmd {
+ MT7530_FDB_READ = 0,
+ MT7530_FDB_WRITE = 1,
+ MT7530_FDB_FLUSH = 2,
+ MT7530_FDB_START = 4,
+ MT7530_FDB_NEXT = 5,
+};
+
+/* Registers for table search read address */
+#define MT7530_TSRA1 0x84
+#define MAC_BYTE_0 24
+#define MAC_BYTE_1 16
+#define MAC_BYTE_2 8
+#define MAC_BYTE_3 0
+#define MAC_BYTE_MASK 0xff
+
+#define MT7530_TSRA2 0x88
+#define MAC_BYTE_4 24
+#define MAC_BYTE_5 16
+#define CVID 0
+#define CVID_MASK 0xfff
+
+#define MT7530_ATRD 0x8C
+#define AGE_TIMER 24
+#define AGE_TIMER_MASK 0xff
+#define PORT_MAP 4
+#define PORT_MAP_MASK 0xff
+#define ENT_STATUS 2
+#define ENT_STATUS_MASK 0x3
+
+/* Register for vlan table control */
+#define MT7530_VTCR 0x90
+#define VTCR_BUSY BIT(31)
+#define VTCR_FUNC (((x) & 0xf) << 12)
+#define VTCR_FUNC_RD_VID 0x1
+#define VTCR_FUNC_WR_VID 0x2
+#define VTCR_FUNC_INV_VID 0x3
+#define VTCR_FUNC_VAL_VID 0x4
+#define VTCR_VID ((x) & 0xfff)
+
+/* Register for setup vlan and acl write data */
+#define MT7530_VAWD1 0x94
+#define PORT_STAG BIT(31)
+#define IVL_MAC BIT(30)
+#define PORT_MEM(x) (((x) & 0xff) << 16)
+#define VALID BIT(1)
+
+#define MT7530_VAWD2 0x98
+
+/* Register for port STP state control */
+#define MT7530_SSP_P(x) (0x2000 + ((x) * 0x100))
+#define FID_PST(x) ((x) & 0x3)
+#define FID_PST_MASK FID_PST(0x3)
+
+enum mt7530_stp_state {
+ MT7530_STP_DISABLED = 0,
+ MT7530_STP_BLOCKING = 1,
+ MT7530_STP_LISTENING = 1,
+ MT7530_STP_LEARNING = 2,
+ MT7530_STP_FORWARDING = 3
+};
+
+/* Register for port control */
+#define MT7530_PCR_P(x) (0x2004 + ((x) * 0x100))
+#define PORT_VLAN(x) ((x) & 0x3)
+#define PCR_MATRIX(x) (((x) & 0xff) << 16)
+#define PORT_PRI(x) (((x) & 0x7) << 24)
+#define EG_TAG(x) (((x) & 0x3) << 28)
+#define PCR_MATRIX_MASK PCR_MATRIX(0xff)
+#define PCR_MATRIX_CLR PCR_MATRIX(0)
+
+/* Register for port security control */
+#define MT7530_PSC_P(x) (0x200c + ((x) * 0x100))
+#define SA_DIS BIT(4)
+
+/* Register for port vlan control */
+#define MT7530_PVC_P(x) (0x2010 + ((x) * 0x100))
+#define PORT_SPEC_TAG BIT(5)
+#define VLAN_ATTR(x) (((x) & 0x3) << 6)
+#define STAG_VPID (((x) & 0xffff) << 16)
+
+/* Register for port port-and-protocol based vlan 1 control */
+#define MT7530_PPBV1_P(x) (0x2014 + ((x) * 0x100))
+
+/* Register for port MAC control register */
+#define MT7530_PMCR_P(x) (0x3000 + ((x) * 0x100))
+#define PMCR_IFG_XMIT(x) (((x) & 0x3) << 18)
+#define PMCR_MAC_MODE BIT(16)
+#define PMCR_FORCE_MODE BIT(15)
+#define PMCR_TX_EN BIT(14)
+#define PMCR_RX_EN BIT(13)
+#define PMCR_BACKOFF_EN BIT(9)
+#define PMCR_BACKPR_EN BIT(8)
+#define PMCR_TX_FC_EN BIT(5)
+#define PMCR_RX_FC_EN BIT(4)
+#define PMCR_FORCE_SPEED_1000 BIT(3)
+#define PMCR_FORCE_FDX BIT(1)
+#define PMCR_FORCE_LNK BIT(0)
+#define PMCR_COMMON_LINK (PMCR_IFG_XMIT(1) | PMCR_MAC_MODE | \
+ PMCR_BACKOFF_EN | PMCR_BACKPR_EN | \
+ PMCR_TX_EN | PMCR_RX_EN | \
+ PMCR_TX_FC_EN | PMCR_RX_FC_EN)
+#define PMCR_CPUP_LINK (PMCR_COMMON_LINK | PMCR_FORCE_MODE | \
+ PMCR_FORCE_SPEED_1000 | \
+ PMCR_FORCE_FDX | \
+ PMCR_FORCE_LNK)
+#define PMCR_USERP_LINK PMCR_COMMON_LINK
+#define PMCR_FIXED_LINK (PMCR_IFG_XMIT(1) | PMCR_MAC_MODE | \
+ PMCR_FORCE_MODE | PMCR_TX_EN | \
+ PMCR_RX_EN | PMCR_BACKPR_EN | \
+ PMCR_BACKOFF_EN | \
+ PMCR_FORCE_SPEED_1000 | \
+ PMCR_FORCE_FDX | \
+ PMCR_FORCE_LNK)
+#define PMCR_FIXED_LINK_FC (PMCR_FIXED_LINK | \
+ PMCR_TX_FC_EN | PMCR_RX_FC_EN)
+
+#define MT7530_PMSR_P(x) (0x3008 + (x) * 0x100)
+
+/* Register for MIB */
+#define MT7530_PORT_MIB_COUNTER(x) (0x4000 + (x) * 0x100)
+#define MT7530_MIB_CCR 0x4fe0
+#define CCR_MIB_ENABLE BIT(31)
+#define CCR_RX_OCT_CNT_GOOD BIT(7)
+#define CCR_RX_OCT_CNT_BAD BIT(6)
+#define CCR_TX_OCT_CNT_GOOD BIT(5)
+#define CCR_TX_OCT_CNT_BAD BIT(4)
+#define CCR_MIB_FLUSH (CCR_RX_OCT_CNT_GOOD | \
+ CCR_RX_OCT_CNT_BAD | \
+ CCR_TX_OCT_CNT_GOOD | \
+ CCR_TX_OCT_CNT_BAD)
+#define CCR_MIB_ACTIVATE (CCR_MIB_ENABLE | \
+ CCR_RX_OCT_CNT_GOOD | \
+ CCR_RX_OCT_CNT_BAD | \
+ CCR_TX_OCT_CNT_GOOD | \
+ CCR_TX_OCT_CNT_BAD)
+/* Register for system reset */
+#define MT7530_SYS_CTRL 0x7000
+#define SYS_CTRL_PHY_RST BIT(2)
+#define SYS_CTRL_SW_RST BIT(1)
+#define SYS_CTRL_REG_RST BIT(0)
+
+/* Register for hw trap status */
+#define MT7530_HWTRAP 0x7800
+
+/* Register for hw trap modification */
+#define MT7530_MHWTRAP 0x7804
+#define MHWTRAP_MANUAL BIT(16)
+#define MHWTRAP_P5_MAC_SEL BIT(13)
+#define MHWTRAP_P6_DIS BIT(8)
+#define MHWTRAP_P5_RGMII_MODE BIT(7)
+#define MHWTRAP_P5_DIS BIT(6)
+#define MHWTRAP_PHY_ACCESS BIT(5)
+
+/* Register for TOP signal control */
+#define MT7530_TOP_SIG_CTRL 0x7808
+#define TOP_SIG_CTRL_NORMAL (BIT(17) | BIT(16))
+
+#define MT7530_IO_DRV_CR 0x7810
+#define P5_IO_CLK_DRV(x) ((x) & 0x3)
+#define P5_IO_DATA_DRV(x) (((x) & 0x3) << 4)
+
+#define MT7530_P6ECR 0x7830
+#define P6_INTF_MODE_MASK 0x3
+#define P6_INTF_MODE(x) ((x) & 0x3)
+
+/* Registers for TRGMII on the both side */
+#define MT7530_TRGMII_RCK_CTRL 0x7a00
+#define GSW_TRGMII_RCK_CTRL 0x300
+#define RX_RST BIT(31)
+#define RXC_DQSISEL BIT(30)
+#define DQSI1_TAP_MASK (0x7f << 8)
+#define DQSI0_TAP_MASK 0x7f
+#define DQSI1_TAP(x) (((x) & 0x7f) << 8)
+#define DQSI0_TAP(x) ((x) & 0x7f)
+
+#define MT7530_TRGMII_RCK_RTT 0x7a04
+#define GSW_TRGMII_RCK_RTT 0x304
+#define DQS1_GATE BIT(31)
+#define DQS0_GATE BIT(30)
+
+#define MT7530_TRGMII_RD(x) (0x7a10 + (x) * 8)
+#define GSW_TRGMII_RD(x) (0x310 + (x) * 8)
+#define BSLIP_EN BIT(31)
+#define EDGE_CHK BIT(30)
+#define RD_TAP_MASK 0x7f
+#define RD_TAP(x) ((x) & 0x7f)
+
+#define GSW_TRGMII_TXCTRL 0x340
+#define MT7530_TRGMII_TXCTRL 0x7a40
+#define TRAIN_TXEN BIT(31)
+#define TXC_INV BIT(30)
+#define TX_RST BIT(28)
+
+#define MT7530_TRGMII_TD_ODT(i) (0x7a54 + 8 * (i))
+#define GSW_TRGMII_TD_ODT(i) (0x354 + 8 * (i))
+#define TD_DM_DRVP(x) ((x) & 0xf)
+#define TD_DM_DRVN(x) (((x) & 0xf) << 4)
+
+#define GSW_INTF_MODE 0x390
+#define INTF_MODE_TRGMII BIT(1)
+
+#define MT7530_TRGMII_TCK_CTRL 0x7a78
+#define TCK_TAP(x) (((x) & 0xf) << 8)
+
+#define MT7530_P5RGMIIRXCR 0x7b00
+#define CSR_RGMII_EDGE_ALIGN BIT(8)
+#define CSR_RGMII_RXC_0DEG_CFG(x) ((x) & 0xf)
+
+#define MT7530_P5RGMIITXCR 0x7b04
+#define CSR_RGMII_TXC_CFG(x) ((x) & 0x1f)
+
+#define MT7530_CREV 0x7ffc
+#define CHIP_NAME_SHIFT 16
+#define MT7530_ID 0x7530
+
+/* Registers for core PLL access through mmd indirect */
+#define CORE_PLL_GROUP2 0x401
+#define RG_SYSPLL_EN_NORMAL BIT(15)
+#define RG_SYSPLL_VODEN BIT(14)
+#define RG_SYSPLL_LF BIT(13)
+#define RG_SYSPLL_RST_DLY(x) (((x) & 0x3) << 12)
+#define RG_SYSPLL_LVROD_EN BIT(10)
+#define RG_SYSPLL_PREDIV(x) (((x) & 0x3) << 8)
+#define RG_SYSPLL_POSDIV(x) (((x) & 0x3) << 5)
+#define RG_SYSPLL_FBKSEL BIT(4)
+#define RT_SYSPLL_EN_AFE_OLT BIT(0)
+
+#define CORE_PLL_GROUP4 0x403
+#define RG_SYSPLL_DDSFBK_EN BIT(12)
+#define RG_SYSPLL_BIAS_EN BIT(11)
+#define RG_SYSPLL_BIAS_LPF_EN BIT(10)
+
+#define CORE_PLL_GROUP5 0x404
+#define RG_LCDDS_PCW_NCPO1(x) ((x) & 0xffff)
+
+#define CORE_PLL_GROUP6 0x405
+#define RG_LCDDS_PCW_NCPO0(x) ((x) & 0xffff)
+
+#define CORE_PLL_GROUP7 0x406
+#define RG_LCDDS_PWDB BIT(15)
+#define RG_LCDDS_ISO_EN BIT(13)
+#define RG_LCCDS_C(x) (((x) & 0x7) << 4)
+#define RG_LCDDS_PCW_NCPO_CHG BIT(3)
+
+#define CORE_PLL_GROUP10 0x409
+#define RG_LCDDS_SSC_DELTA(x) ((x) & 0xfff)
+
+#define CORE_PLL_GROUP11 0x40a
+#define RG_LCDDS_SSC_DELTA1(x) ((x) & 0xfff)
+
+#define CORE_GSWPLL_GRP1 0x40d
+#define RG_GSWPLL_PREDIV(x) (((x) & 0x3) << 14)
+#define RG_GSWPLL_POSDIV_200M(x) (((x) & 0x3) << 12)
+#define RG_GSWPLL_EN_PRE BIT(11)
+#define RG_GSWPLL_FBKSEL BIT(10)
+#define RG_GSWPLL_BP BIT(9)
+#define RG_GSWPLL_BR BIT(8)
+#define RG_GSWPLL_FBKDIV_200M(x) ((x) & 0xff)
+
+#define CORE_GSWPLL_GRP2 0x40e
+#define RG_GSWPLL_POSDIV_500M(x) (((x) & 0x3) << 8)
+#define RG_GSWPLL_FBKDIV_500M(x) ((x) & 0xff)
+
+#define CORE_TRGMII_GSW_CLK_CG 0x410
+#define REG_GSWCK_EN BIT(0)
+#define REG_TRGMIICK_EN BIT(1)
+
+#define MIB_DESC(_s, _o, _n) \
+ { \
+ .size = (_s), \
+ .offset = (_o), \
+ .name = (_n), \
+ }
+
+struct mt7530_mib_desc {
+ unsigned int size;
+ unsigned int offset;
+ const char *name;
+};
+
+struct mt7530_fdb {
+ u16 vid;
+ u8 port_mask;
+ u8 aging;
+ u8 mac[6];
+ bool noarp;
+};
+
+struct mt7530_port {
+ bool enable;
+ u32 pm;
+};
+
+/* struct mt7530_priv - This is the main data structure for holding the state
+ * of the driver
+ * @dev: The device pointer
+ * @ds: The pointer to the dsa core structure
+ * @bus: The bus used for the device and built-in PHY
+ * @rstc: The pointer to reset control used by MCM
+ * @ethernet: The regmap used for access TRGMII-based registers
+ * @core_pwr: The power supplied into the core
+ * @io_pwr: The power supplied into the I/O
+ * @reset: The descriptor for GPIO line tied to its reset pin
+ * @mcm: Flag for distinguishing if standalone IC or module
+ * coupling
+ * @ports: Holding the state among ports
+ * @reg_mutex: The lock for protecting among process accessing
+ * registers
+ */
+struct mt7530_priv {
+ struct device *dev;
+ struct dsa_switch *ds;
+ struct mii_bus *bus;
+ struct reset_control *rstc;
+ struct regmap *ethernet;
+ struct regulator *core_pwr;
+ struct regulator *io_pwr;
+ struct gpio_desc *reset;
+ bool mcm;
+
+ struct mt7530_port ports[MT7530_NUM_PORTS];
+ /* protect among processes for registers access*/
+ struct mutex reg_mutex;
+};
+
+struct mt7530_hw_stats {
+ const char *string;
+ u16 reg;
+ u8 sizeof_stat;
+};
+
+struct mt7530_dummy_poll {
+ struct mt7530_priv *priv;
+ u32 reg;
+};
+
+static inline void INIT_MT7530_DUMMY_POLL(struct mt7530_dummy_poll *p,
+ struct mt7530_priv *priv, u32 reg)
+{
+ p->priv = priv;
+ p->reg = reg;
+}
+
+#endif /* __MT7530_H */
diff --git a/drivers/net/dsa/mv88e6xxx/Makefile b/drivers/net/dsa/mv88e6xxx/Makefile
index c36be31..6edd869 100644
--- a/drivers/net/dsa/mv88e6xxx/Makefile
+++ b/drivers/net/dsa/mv88e6xxx/Makefile
@@ -1,5 +1,7 @@
obj-$(CONFIG_NET_DSA_MV88E6XXX) += mv88e6xxx.o
mv88e6xxx-objs := chip.o
mv88e6xxx-objs += global1.o
+mv88e6xxx-objs += global1_atu.o
+mv88e6xxx-objs += global1_vtu.o
mv88e6xxx-$(CONFIG_NET_DSA_MV88E6XXX_GLOBAL2) += global2.o
mv88e6xxx-objs += port.o
diff --git a/drivers/net/dsa/mv88e6xxx/chip.c b/drivers/net/dsa/mv88e6xxx/chip.c
index 03dc886..19581d7 100644
--- a/drivers/net/dsa/mv88e6xxx/chip.c
+++ b/drivers/net/dsa/mv88e6xxx/chip.c
@@ -3,11 +3,11 @@
*
* Copyright (c) 2008 Marvell Semiconductor
*
- * Copyright (c) 2015 CMC Electronics, Inc.
- * Added support for VLAN Table Unit operations
- *
* Copyright (c) 2016 Andrew Lunn <andrew@lunn.ch>
*
+ * Copyright (c) 2016-2017 Savoir-faire Linux Inc.
+ * Vivien Didelot <vivien.didelot@savoirfairelinux.com>
+ *
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
@@ -677,36 +677,6 @@
return err;
}
-static bool mv88e6xxx_6097_family(struct mv88e6xxx_chip *chip)
-{
- return chip->info->family == MV88E6XXX_FAMILY_6097;
-}
-
-static bool mv88e6xxx_6165_family(struct mv88e6xxx_chip *chip)
-{
- return chip->info->family == MV88E6XXX_FAMILY_6165;
-}
-
-static bool mv88e6xxx_6320_family(struct mv88e6xxx_chip *chip)
-{
- return chip->info->family == MV88E6XXX_FAMILY_6320;
-}
-
-static bool mv88e6xxx_6341_family(struct mv88e6xxx_chip *chip)
-{
- return chip->info->family == MV88E6XXX_FAMILY_6341;
-}
-
-static bool mv88e6xxx_6351_family(struct mv88e6xxx_chip *chip)
-{
- return chip->info->family == MV88E6XXX_FAMILY_6351;
-}
-
-static bool mv88e6xxx_6352_family(struct mv88e6xxx_chip *chip)
-{
- return chip->info->family == MV88E6XXX_FAMILY_6352;
-}
-
static int mv88e6xxx_port_setup_mac(struct mv88e6xxx_chip *chip, int port,
int link, int speed, int duplex,
phy_interface_t mode)
@@ -1066,11 +1036,6 @@
mutex_unlock(&chip->reg_lock);
}
-static int _mv88e6xxx_atu_wait(struct mv88e6xxx_chip *chip)
-{
- return mv88e6xxx_g1_wait(chip, GLOBAL_ATU_OP, GLOBAL_ATU_OP_BUSY);
-}
-
static int mv88e6xxx_get_eee(struct dsa_switch *ds, int port,
struct ethtool_eee *e)
{
@@ -1130,143 +1095,42 @@
return err;
}
-static int _mv88e6xxx_atu_cmd(struct mv88e6xxx_chip *chip, u16 fid, u16 cmd)
+static u16 mv88e6xxx_port_vlan(struct mv88e6xxx_chip *chip, int dev, int port)
{
- u16 val;
- int err;
-
- if (mv88e6xxx_has(chip, MV88E6XXX_FLAG_G1_ATU_FID)) {
- err = mv88e6xxx_g1_write(chip, GLOBAL_ATU_FID, fid);
- if (err)
- return err;
- } else if (mv88e6xxx_num_databases(chip) == 256) {
- /* ATU DBNum[7:4] are located in ATU Control 15:12 */
- err = mv88e6xxx_g1_read(chip, GLOBAL_ATU_CONTROL, &val);
- if (err)
- return err;
-
- err = mv88e6xxx_g1_write(chip, GLOBAL_ATU_CONTROL,
- (val & 0xfff) | ((fid << 8) & 0xf000));
- if (err)
- return err;
-
- /* ATU DBNum[3:0] are located in ATU Operation 3:0 */
- cmd |= fid & 0xf;
- }
-
- err = mv88e6xxx_g1_write(chip, GLOBAL_ATU_OP, cmd);
- if (err)
- return err;
-
- return _mv88e6xxx_atu_wait(chip);
-}
-
-static int _mv88e6xxx_atu_data_write(struct mv88e6xxx_chip *chip,
- struct mv88e6xxx_atu_entry *entry)
-{
- u16 data = entry->state & GLOBAL_ATU_DATA_STATE_MASK;
-
- if (entry->state != GLOBAL_ATU_DATA_STATE_UNUSED) {
- unsigned int mask, shift;
-
- if (entry->trunk) {
- data |= GLOBAL_ATU_DATA_TRUNK;
- mask = GLOBAL_ATU_DATA_TRUNK_ID_MASK;
- shift = GLOBAL_ATU_DATA_TRUNK_ID_SHIFT;
- } else {
- mask = GLOBAL_ATU_DATA_PORT_VECTOR_MASK;
- shift = GLOBAL_ATU_DATA_PORT_VECTOR_SHIFT;
- }
-
- data |= (entry->portv_trunkid << shift) & mask;
- }
-
- return mv88e6xxx_g1_write(chip, GLOBAL_ATU_DATA, data);
-}
-
-static int _mv88e6xxx_atu_flush_move(struct mv88e6xxx_chip *chip,
- struct mv88e6xxx_atu_entry *entry,
- bool static_too)
-{
- int op;
- int err;
-
- err = _mv88e6xxx_atu_wait(chip);
- if (err)
- return err;
-
- err = _mv88e6xxx_atu_data_write(chip, entry);
- if (err)
- return err;
-
- if (entry->fid) {
- op = static_too ? GLOBAL_ATU_OP_FLUSH_MOVE_ALL_DB :
- GLOBAL_ATU_OP_FLUSH_MOVE_NON_STATIC_DB;
- } else {
- op = static_too ? GLOBAL_ATU_OP_FLUSH_MOVE_ALL :
- GLOBAL_ATU_OP_FLUSH_MOVE_NON_STATIC;
- }
-
- return _mv88e6xxx_atu_cmd(chip, entry->fid, op);
-}
-
-static int _mv88e6xxx_atu_flush(struct mv88e6xxx_chip *chip,
- u16 fid, bool static_too)
-{
- struct mv88e6xxx_atu_entry entry = {
- .fid = fid,
- .state = 0, /* EntryState bits must be 0 */
- };
-
- return _mv88e6xxx_atu_flush_move(chip, &entry, static_too);
-}
-
-static int _mv88e6xxx_atu_move(struct mv88e6xxx_chip *chip, u16 fid,
- int from_port, int to_port, bool static_too)
-{
- struct mv88e6xxx_atu_entry entry = {
- .trunk = false,
- .fid = fid,
- };
-
- /* EntryState bits must be 0xF */
- entry.state = GLOBAL_ATU_DATA_STATE_MASK;
-
- /* ToPort and FromPort are respectively in PortVec bits 7:4 and 3:0 */
- entry.portv_trunkid = (to_port & 0x0f) << 4;
- entry.portv_trunkid |= from_port & 0x0f;
-
- return _mv88e6xxx_atu_flush_move(chip, &entry, static_too);
-}
-
-static int _mv88e6xxx_atu_remove(struct mv88e6xxx_chip *chip, u16 fid,
- int port, bool static_too)
-{
- /* Destination port 0xF means remove the entries */
- return _mv88e6xxx_atu_move(chip, fid, port, 0x0f, static_too);
-}
-
-static int _mv88e6xxx_port_based_vlan_map(struct mv88e6xxx_chip *chip, int port)
-{
- struct dsa_switch *ds = chip->ds;
- struct net_device *bridge = ds->ports[port].bridge_dev;
- u16 output_ports = 0;
+ struct dsa_switch *ds = NULL;
+ struct net_device *br;
+ u16 pvlan;
int i;
- /* allow CPU port or DSA link(s) to send frames to every port */
- if (dsa_is_cpu_port(ds, port) || dsa_is_dsa_port(ds, port)) {
- output_ports = ~0;
- } else {
- for (i = 0; i < mv88e6xxx_num_ports(chip); ++i) {
- /* allow sending frames to every group member */
- if (bridge && ds->ports[i].bridge_dev == bridge)
- output_ports |= BIT(i);
+ if (dev < DSA_MAX_SWITCHES)
+ ds = chip->ds->dst->ds[dev];
- /* allow sending frames to CPU port and DSA link(s) */
- if (dsa_is_cpu_port(ds, i) || dsa_is_dsa_port(ds, i))
- output_ports |= BIT(i);
- }
- }
+ /* Prevent frames from unknown switch or port */
+ if (!ds || port >= ds->num_ports)
+ return 0;
+
+ /* Frames from DSA links and CPU ports can egress any local port */
+ if (dsa_is_cpu_port(ds, port) || dsa_is_dsa_port(ds, port))
+ return mv88e6xxx_port_mask(chip);
+
+ br = ds->ports[port].bridge_dev;
+ pvlan = 0;
+
+ /* Frames from user ports can egress any local DSA links and CPU ports,
+ * as well as any local member of their bridge group.
+ */
+ for (i = 0; i < mv88e6xxx_num_ports(chip); ++i)
+ if (dsa_is_cpu_port(chip->ds, i) ||
+ dsa_is_dsa_port(chip->ds, i) ||
+ (br && chip->ds->ports[i].bridge_dev == br))
+ pvlan |= BIT(i);
+
+ return pvlan;
+}
+
+static int mv88e6xxx_port_vlan_map(struct mv88e6xxx_chip *chip, int port)
+{
+ u16 output_ports = mv88e6xxx_port_vlan(chip, chip->ds->index, port);
/* prevent frames from going back out of the port they came in on */
output_ports &= ~BIT(port);
@@ -1306,182 +1170,98 @@
netdev_err(ds->ports[port].netdev, "failed to update state\n");
}
+static int mv88e6xxx_atu_setup(struct mv88e6xxx_chip *chip)
+{
+ int err;
+
+ err = mv88e6xxx_g1_atu_flush(chip, 0, true);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_atu_set_learn2all(chip, true);
+ if (err)
+ return err;
+
+ return mv88e6xxx_g1_atu_set_age_time(chip, 300000);
+}
+
+static int mv88e6xxx_pvt_map(struct mv88e6xxx_chip *chip, int dev, int port)
+{
+ u16 pvlan = 0;
+
+ if (!mv88e6xxx_has_pvt(chip))
+ return -EOPNOTSUPP;
+
+ /* Skip the local source device, which uses in-chip port VLAN */
+ if (dev != chip->ds->index)
+ pvlan = mv88e6xxx_port_vlan(chip, dev, port);
+
+ return mv88e6xxx_g2_pvt_write(chip, dev, port, pvlan);
+}
+
+static int mv88e6xxx_pvt_setup(struct mv88e6xxx_chip *chip)
+{
+ int dev, port;
+ int err;
+
+ if (!mv88e6xxx_has_pvt(chip))
+ return 0;
+
+ /* Clear 5 Bit Port for usage with Marvell Link Street devices:
+ * use 4 bits for the Src_Port/Src_Trunk and 5 bits for the Src_Dev.
+ */
+ err = mv88e6xxx_g2_misc_4_bit_port(chip);
+ if (err)
+ return err;
+
+ for (dev = 0; dev < MV88E6XXX_MAX_PVT_SWITCHES; ++dev) {
+ for (port = 0; port < MV88E6XXX_MAX_PVT_PORTS; ++port) {
+ err = mv88e6xxx_pvt_map(chip, dev, port);
+ if (err)
+ return err;
+ }
+ }
+
+ return 0;
+}
+
static void mv88e6xxx_port_fast_age(struct dsa_switch *ds, int port)
{
struct mv88e6xxx_chip *chip = ds->priv;
int err;
mutex_lock(&chip->reg_lock);
- err = _mv88e6xxx_atu_remove(chip, 0, port, false);
+ err = mv88e6xxx_g1_atu_remove(chip, 0, port, false);
mutex_unlock(&chip->reg_lock);
if (err)
netdev_err(ds->ports[port].netdev, "failed to flush ATU\n");
}
-static int _mv88e6xxx_vtu_wait(struct mv88e6xxx_chip *chip)
+static int mv88e6xxx_vtu_setup(struct mv88e6xxx_chip *chip)
{
- return mv88e6xxx_g1_wait(chip, GLOBAL_VTU_OP, GLOBAL_VTU_OP_BUSY);
+ if (!chip->info->max_vid)
+ return 0;
+
+ return mv88e6xxx_g1_vtu_flush(chip);
}
-static int _mv88e6xxx_vtu_cmd(struct mv88e6xxx_chip *chip, u16 op)
+static int mv88e6xxx_vtu_getnext(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
{
- int err;
+ if (!chip->info->ops->vtu_getnext)
+ return -EOPNOTSUPP;
- err = mv88e6xxx_g1_write(chip, GLOBAL_VTU_OP, op);
- if (err)
- return err;
-
- return _mv88e6xxx_vtu_wait(chip);
+ return chip->info->ops->vtu_getnext(chip, entry);
}
-static int _mv88e6xxx_vtu_stu_flush(struct mv88e6xxx_chip *chip)
-{
- int ret;
-
- ret = _mv88e6xxx_vtu_wait(chip);
- if (ret < 0)
- return ret;
-
- return _mv88e6xxx_vtu_cmd(chip, GLOBAL_VTU_OP_FLUSH_ALL);
-}
-
-static int _mv88e6xxx_vtu_stu_data_read(struct mv88e6xxx_chip *chip,
- struct mv88e6xxx_vtu_entry *entry,
- unsigned int nibble_offset)
-{
- u16 regs[3];
- int i, err;
-
- for (i = 0; i < 3; ++i) {
- u16 *reg = ®s[i];
-
- err = mv88e6xxx_g1_read(chip, GLOBAL_VTU_DATA_0_3 + i, reg);
- if (err)
- return err;
- }
-
- for (i = 0; i < mv88e6xxx_num_ports(chip); ++i) {
- unsigned int shift = (i % 4) * 4 + nibble_offset;
- u16 reg = regs[i / 4];
-
- entry->data[i] = (reg >> shift) & GLOBAL_VTU_STU_DATA_MASK;
- }
-
- return 0;
-}
-
-static int mv88e6xxx_vtu_data_read(struct mv88e6xxx_chip *chip,
+static int mv88e6xxx_vtu_loadpurge(struct mv88e6xxx_chip *chip,
struct mv88e6xxx_vtu_entry *entry)
{
- return _mv88e6xxx_vtu_stu_data_read(chip, entry, 0);
-}
+ if (!chip->info->ops->vtu_loadpurge)
+ return -EOPNOTSUPP;
-static int mv88e6xxx_stu_data_read(struct mv88e6xxx_chip *chip,
- struct mv88e6xxx_vtu_entry *entry)
-{
- return _mv88e6xxx_vtu_stu_data_read(chip, entry, 2);
-}
-
-static int _mv88e6xxx_vtu_stu_data_write(struct mv88e6xxx_chip *chip,
- struct mv88e6xxx_vtu_entry *entry,
- unsigned int nibble_offset)
-{
- u16 regs[3] = { 0 };
- int i, err;
-
- for (i = 0; i < mv88e6xxx_num_ports(chip); ++i) {
- unsigned int shift = (i % 4) * 4 + nibble_offset;
- u8 data = entry->data[i];
-
- regs[i / 4] |= (data & GLOBAL_VTU_STU_DATA_MASK) << shift;
- }
-
- for (i = 0; i < 3; ++i) {
- u16 reg = regs[i];
-
- err = mv88e6xxx_g1_write(chip, GLOBAL_VTU_DATA_0_3 + i, reg);
- if (err)
- return err;
- }
-
- return 0;
-}
-
-static int mv88e6xxx_vtu_data_write(struct mv88e6xxx_chip *chip,
- struct mv88e6xxx_vtu_entry *entry)
-{
- return _mv88e6xxx_vtu_stu_data_write(chip, entry, 0);
-}
-
-static int mv88e6xxx_stu_data_write(struct mv88e6xxx_chip *chip,
- struct mv88e6xxx_vtu_entry *entry)
-{
- return _mv88e6xxx_vtu_stu_data_write(chip, entry, 2);
-}
-
-static int _mv88e6xxx_vtu_vid_write(struct mv88e6xxx_chip *chip, u16 vid)
-{
- return mv88e6xxx_g1_write(chip, GLOBAL_VTU_VID,
- vid & GLOBAL_VTU_VID_MASK);
-}
-
-static int _mv88e6xxx_vtu_getnext(struct mv88e6xxx_chip *chip,
- struct mv88e6xxx_vtu_entry *entry)
-{
- struct mv88e6xxx_vtu_entry next = { 0 };
- u16 val;
- int err;
-
- err = _mv88e6xxx_vtu_wait(chip);
- if (err)
- return err;
-
- err = _mv88e6xxx_vtu_cmd(chip, GLOBAL_VTU_OP_VTU_GET_NEXT);
- if (err)
- return err;
-
- err = mv88e6xxx_g1_read(chip, GLOBAL_VTU_VID, &val);
- if (err)
- return err;
-
- next.vid = val & GLOBAL_VTU_VID_MASK;
- next.valid = !!(val & GLOBAL_VTU_VID_VALID);
-
- if (next.valid) {
- err = mv88e6xxx_vtu_data_read(chip, &next);
- if (err)
- return err;
-
- if (mv88e6xxx_has(chip, MV88E6XXX_FLAG_G1_VTU_FID)) {
- err = mv88e6xxx_g1_read(chip, GLOBAL_VTU_FID, &val);
- if (err)
- return err;
-
- next.fid = val & GLOBAL_VTU_FID_MASK;
- } else if (mv88e6xxx_num_databases(chip) == 256) {
- /* VTU DBNum[7:4] are located in VTU Operation 11:8, and
- * VTU DBNum[3:0] are located in VTU Operation 3:0
- */
- err = mv88e6xxx_g1_read(chip, GLOBAL_VTU_OP, &val);
- if (err)
- return err;
-
- next.fid = (val & 0xf00) >> 4;
- next.fid |= val & 0xf;
- }
-
- if (mv88e6xxx_has(chip, MV88E6XXX_FLAG_STU)) {
- err = mv88e6xxx_g1_read(chip, GLOBAL_VTU_SID, &val);
- if (err)
- return err;
-
- next.sid = val & GLOBAL_VTU_SID_MASK;
- }
- }
-
- *entry = next;
- return 0;
+ return chip->info->ops->vtu_loadpurge(chip, entry);
}
static int mv88e6xxx_port_vlan_dump(struct dsa_switch *ds, int port,
@@ -1489,11 +1269,13 @@
int (*cb)(struct switchdev_obj *obj))
{
struct mv88e6xxx_chip *chip = ds->priv;
- struct mv88e6xxx_vtu_entry next;
+ struct mv88e6xxx_vtu_entry next = {
+ .vid = chip->info->max_vid,
+ };
u16 pvid;
int err;
- if (!mv88e6xxx_has(chip, MV88E6XXX_FLAG_VTU))
+ if (!chip->info->max_vid)
return -EOPNOTSUPP;
mutex_lock(&chip->reg_lock);
@@ -1502,19 +1284,15 @@
if (err)
goto unlock;
- err = _mv88e6xxx_vtu_vid_write(chip, GLOBAL_VTU_VID_MASK);
- if (err)
- goto unlock;
-
do {
- err = _mv88e6xxx_vtu_getnext(chip, &next);
+ err = mv88e6xxx_vtu_getnext(chip, &next);
if (err)
break;
if (!next.valid)
break;
- if (next.data[port] == GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER)
+ if (next.member[port] == GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER)
continue;
/* reinit and dump this VLAN obj */
@@ -1522,7 +1300,7 @@
vlan->vid_end = next.vid;
vlan->flags = 0;
- if (next.data[port] == GLOBAL_VTU_DATA_MEMBER_TAG_UNTAGGED)
+ if (next.member[port] == GLOBAL_VTU_DATA_MEMBER_TAG_UNTAGGED)
vlan->flags |= BRIDGE_VLAN_INFO_UNTAGGED;
if (next.vid == pvid)
@@ -1531,7 +1309,7 @@
err = cb(&vlan->obj);
if (err)
break;
- } while (next.vid < GLOBAL_VTU_VID_MASK);
+ } while (next.vid < chip->info->max_vid);
unlock:
mutex_unlock(&chip->reg_lock);
@@ -1539,133 +1317,12 @@
return err;
}
-static int _mv88e6xxx_vtu_loadpurge(struct mv88e6xxx_chip *chip,
- struct mv88e6xxx_vtu_entry *entry)
-{
- u16 op = GLOBAL_VTU_OP_VTU_LOAD_PURGE;
- u16 reg = 0;
- int err;
-
- err = _mv88e6xxx_vtu_wait(chip);
- if (err)
- return err;
-
- if (!entry->valid)
- goto loadpurge;
-
- /* Write port member tags */
- err = mv88e6xxx_vtu_data_write(chip, entry);
- if (err)
- return err;
-
- if (mv88e6xxx_has(chip, MV88E6XXX_FLAG_STU)) {
- reg = entry->sid & GLOBAL_VTU_SID_MASK;
- err = mv88e6xxx_g1_write(chip, GLOBAL_VTU_SID, reg);
- if (err)
- return err;
- }
-
- if (mv88e6xxx_has(chip, MV88E6XXX_FLAG_G1_VTU_FID)) {
- reg = entry->fid & GLOBAL_VTU_FID_MASK;
- err = mv88e6xxx_g1_write(chip, GLOBAL_VTU_FID, reg);
- if (err)
- return err;
- } else if (mv88e6xxx_num_databases(chip) == 256) {
- /* VTU DBNum[7:4] are located in VTU Operation 11:8, and
- * VTU DBNum[3:0] are located in VTU Operation 3:0
- */
- op |= (entry->fid & 0xf0) << 8;
- op |= entry->fid & 0xf;
- }
-
- reg = GLOBAL_VTU_VID_VALID;
-loadpurge:
- reg |= entry->vid & GLOBAL_VTU_VID_MASK;
- err = mv88e6xxx_g1_write(chip, GLOBAL_VTU_VID, reg);
- if (err)
- return err;
-
- return _mv88e6xxx_vtu_cmd(chip, op);
-}
-
-static int _mv88e6xxx_stu_getnext(struct mv88e6xxx_chip *chip, u8 sid,
- struct mv88e6xxx_vtu_entry *entry)
-{
- struct mv88e6xxx_vtu_entry next = { 0 };
- u16 val;
- int err;
-
- err = _mv88e6xxx_vtu_wait(chip);
- if (err)
- return err;
-
- err = mv88e6xxx_g1_write(chip, GLOBAL_VTU_SID,
- sid & GLOBAL_VTU_SID_MASK);
- if (err)
- return err;
-
- err = _mv88e6xxx_vtu_cmd(chip, GLOBAL_VTU_OP_STU_GET_NEXT);
- if (err)
- return err;
-
- err = mv88e6xxx_g1_read(chip, GLOBAL_VTU_SID, &val);
- if (err)
- return err;
-
- next.sid = val & GLOBAL_VTU_SID_MASK;
-
- err = mv88e6xxx_g1_read(chip, GLOBAL_VTU_VID, &val);
- if (err)
- return err;
-
- next.valid = !!(val & GLOBAL_VTU_VID_VALID);
-
- if (next.valid) {
- err = mv88e6xxx_stu_data_read(chip, &next);
- if (err)
- return err;
- }
-
- *entry = next;
- return 0;
-}
-
-static int _mv88e6xxx_stu_loadpurge(struct mv88e6xxx_chip *chip,
- struct mv88e6xxx_vtu_entry *entry)
-{
- u16 reg = 0;
- int err;
-
- err = _mv88e6xxx_vtu_wait(chip);
- if (err)
- return err;
-
- if (!entry->valid)
- goto loadpurge;
-
- /* Write port states */
- err = mv88e6xxx_stu_data_write(chip, entry);
- if (err)
- return err;
-
- reg = GLOBAL_VTU_VID_VALID;
-loadpurge:
- err = mv88e6xxx_g1_write(chip, GLOBAL_VTU_VID, reg);
- if (err)
- return err;
-
- reg = entry->sid & GLOBAL_VTU_SID_MASK;
- err = mv88e6xxx_g1_write(chip, GLOBAL_VTU_SID, reg);
- if (err)
- return err;
-
- return _mv88e6xxx_vtu_cmd(chip, GLOBAL_VTU_OP_STU_LOAD_PURGE);
-}
-
-static int _mv88e6xxx_fid_new(struct mv88e6xxx_chip *chip, u16 *fid)
+static int mv88e6xxx_atu_new(struct mv88e6xxx_chip *chip, u16 *fid)
{
DECLARE_BITMAP(fid_bitmap, MV88E6XXX_N_FID);
- struct mv88e6xxx_vtu_entry vlan;
+ struct mv88e6xxx_vtu_entry vlan = {
+ .vid = chip->info->max_vid,
+ };
int i, err;
bitmap_zero(fid_bitmap, MV88E6XXX_N_FID);
@@ -1680,12 +1337,8 @@
}
/* Set every FID bit used by the VLAN entries */
- err = _mv88e6xxx_vtu_vid_write(chip, GLOBAL_VTU_VID_MASK);
- if (err)
- return err;
-
do {
- err = _mv88e6xxx_vtu_getnext(chip, &vlan);
+ err = mv88e6xxx_vtu_getnext(chip, &vlan);
if (err)
return err;
@@ -1693,7 +1346,7 @@
break;
set_bit(vlan.fid, fid_bitmap);
- } while (vlan.vid < GLOBAL_VTU_VID_MASK);
+ } while (vlan.vid < chip->info->max_vid);
/* The reset value 0x000 is used to indicate that multiple address
* databases are not needed. Return the next positive available.
@@ -1703,92 +1356,55 @@
return -ENOSPC;
/* Clear the database */
- return _mv88e6xxx_atu_flush(chip, *fid, true);
+ return mv88e6xxx_g1_atu_flush(chip, *fid, true);
}
-static int _mv88e6xxx_vtu_new(struct mv88e6xxx_chip *chip, u16 vid,
- struct mv88e6xxx_vtu_entry *entry)
-{
- struct dsa_switch *ds = chip->ds;
- struct mv88e6xxx_vtu_entry vlan = {
- .valid = true,
- .vid = vid,
- };
- int i, err;
-
- err = _mv88e6xxx_fid_new(chip, &vlan.fid);
- if (err)
- return err;
-
- /* exclude all ports except the CPU and DSA ports */
- for (i = 0; i < mv88e6xxx_num_ports(chip); ++i)
- vlan.data[i] = dsa_is_cpu_port(ds, i) || dsa_is_dsa_port(ds, i)
- ? GLOBAL_VTU_DATA_MEMBER_TAG_UNMODIFIED
- : GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER;
-
- if (mv88e6xxx_6097_family(chip) || mv88e6xxx_6165_family(chip) ||
- mv88e6xxx_6351_family(chip) || mv88e6xxx_6352_family(chip) ||
- mv88e6xxx_6341_family(chip)) {
- struct mv88e6xxx_vtu_entry vstp;
-
- /* Adding a VTU entry requires a valid STU entry. As VSTP is not
- * implemented, only one STU entry is needed to cover all VTU
- * entries. Thus, validate the SID 0.
- */
- vlan.sid = 0;
- err = _mv88e6xxx_stu_getnext(chip, GLOBAL_VTU_SID_MASK, &vstp);
- if (err)
- return err;
-
- if (vstp.sid != vlan.sid || !vstp.valid) {
- memset(&vstp, 0, sizeof(vstp));
- vstp.valid = true;
- vstp.sid = vlan.sid;
-
- err = _mv88e6xxx_stu_loadpurge(chip, &vstp);
- if (err)
- return err;
- }
- }
-
- *entry = vlan;
- return 0;
-}
-
-static int _mv88e6xxx_vtu_get(struct mv88e6xxx_chip *chip, u16 vid,
- struct mv88e6xxx_vtu_entry *entry, bool creat)
+static int mv88e6xxx_vtu_get(struct mv88e6xxx_chip *chip, u16 vid,
+ struct mv88e6xxx_vtu_entry *entry, bool new)
{
int err;
if (!vid)
return -EINVAL;
- err = _mv88e6xxx_vtu_vid_write(chip, vid - 1);
+ entry->vid = vid - 1;
+ entry->valid = false;
+
+ err = mv88e6xxx_vtu_getnext(chip, entry);
if (err)
return err;
- err = _mv88e6xxx_vtu_getnext(chip, entry);
- if (err)
- return err;
+ if (entry->vid == vid && entry->valid)
+ return 0;
- if (entry->vid != vid || !entry->valid) {
- if (!creat)
- return -EOPNOTSUPP;
- /* -ENOENT would've been more appropriate, but switchdev expects
- * -EOPNOTSUPP to inform bridge about an eventual software VLAN.
- */
+ if (new) {
+ int i;
- err = _mv88e6xxx_vtu_new(chip, vid, entry);
+ /* Initialize a fresh VLAN entry */
+ memset(entry, 0, sizeof(*entry));
+ entry->valid = true;
+ entry->vid = vid;
+
+ /* Include only CPU and DSA ports */
+ for (i = 0; i < mv88e6xxx_num_ports(chip); ++i)
+ entry->member[i] = dsa_is_normal_port(chip->ds, i) ?
+ GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER :
+ GLOBAL_VTU_DATA_MEMBER_TAG_UNMODIFIED;
+
+ return mv88e6xxx_atu_new(chip, &entry->fid);
}
- return err;
+ /* switchdev expects -EOPNOTSUPP to honor software VLANs */
+ return -EOPNOTSUPP;
}
static int mv88e6xxx_port_check_hw_vlan(struct dsa_switch *ds, int port,
u16 vid_begin, u16 vid_end)
{
struct mv88e6xxx_chip *chip = ds->priv;
- struct mv88e6xxx_vtu_entry vlan;
+ struct mv88e6xxx_vtu_entry vlan = {
+ .vid = vid_begin - 1,
+ };
int i, err;
if (!vid_begin)
@@ -1796,12 +1412,8 @@
mutex_lock(&chip->reg_lock);
- err = _mv88e6xxx_vtu_vid_write(chip, vid_begin - 1);
- if (err)
- goto unlock;
-
do {
- err = _mv88e6xxx_vtu_getnext(chip, &vlan);
+ err = mv88e6xxx_vtu_getnext(chip, &vlan);
if (err)
goto unlock;
@@ -1818,7 +1430,7 @@
if (!ds->ports[port].netdev)
continue;
- if (vlan.data[i] ==
+ if (vlan.member[i] ==
GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER)
continue;
@@ -1852,7 +1464,7 @@
PORT_CONTROL_2_8021Q_DISABLED;
int err;
- if (!mv88e6xxx_has(chip, MV88E6XXX_FLAG_VTU))
+ if (!chip->info->max_vid)
return -EOPNOTSUPP;
mutex_lock(&chip->reg_lock);
@@ -1870,7 +1482,7 @@
struct mv88e6xxx_chip *chip = ds->priv;
int err;
- if (!mv88e6xxx_has(chip, MV88E6XXX_FLAG_VTU))
+ if (!chip->info->max_vid)
return -EOPNOTSUPP;
/* If the requested port doesn't belong to the same bridge as the VLAN
@@ -1893,15 +1505,15 @@
struct mv88e6xxx_vtu_entry vlan;
int err;
- err = _mv88e6xxx_vtu_get(chip, vid, &vlan, true);
+ err = mv88e6xxx_vtu_get(chip, vid, &vlan, true);
if (err)
return err;
- vlan.data[port] = untagged ?
+ vlan.member[port] = untagged ?
GLOBAL_VTU_DATA_MEMBER_TAG_UNTAGGED :
GLOBAL_VTU_DATA_MEMBER_TAG_TAGGED;
- return _mv88e6xxx_vtu_loadpurge(chip, &vlan);
+ return mv88e6xxx_vtu_loadpurge(chip, &vlan);
}
static void mv88e6xxx_port_vlan_add(struct dsa_switch *ds, int port,
@@ -1913,7 +1525,7 @@
bool pvid = vlan->flags & BRIDGE_VLAN_INFO_PVID;
u16 vid;
- if (!mv88e6xxx_has(chip, MV88E6XXX_FLAG_VTU))
+ if (!chip->info->max_vid)
return;
mutex_lock(&chip->reg_lock);
@@ -1938,15 +1550,15 @@
struct mv88e6xxx_vtu_entry vlan;
int i, err;
- err = _mv88e6xxx_vtu_get(chip, vid, &vlan, false);
+ err = mv88e6xxx_vtu_get(chip, vid, &vlan, false);
if (err)
return err;
/* Tell switchdev if this VLAN is handled in software */
- if (vlan.data[port] == GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER)
+ if (vlan.member[port] == GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER)
return -EOPNOTSUPP;
- vlan.data[port] = GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER;
+ vlan.member[port] = GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER;
/* keep the VLAN unless all ports are excluded */
vlan.valid = false;
@@ -1954,17 +1566,17 @@
if (dsa_is_cpu_port(ds, i) || dsa_is_dsa_port(ds, i))
continue;
- if (vlan.data[i] != GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER) {
+ if (vlan.member[i] != GLOBAL_VTU_DATA_MEMBER_TAG_NON_MEMBER) {
vlan.valid = true;
break;
}
}
- err = _mv88e6xxx_vtu_loadpurge(chip, &vlan);
+ err = mv88e6xxx_vtu_loadpurge(chip, &vlan);
if (err)
return err;
- return _mv88e6xxx_atu_remove(chip, vlan.fid, port, false);
+ return mv88e6xxx_g1_atu_remove(chip, vlan.fid, port, false);
}
static int mv88e6xxx_port_vlan_del(struct dsa_switch *ds, int port,
@@ -1974,7 +1586,7 @@
u16 pvid, vid;
int err = 0;
- if (!mv88e6xxx_has(chip, MV88E6XXX_FLAG_VTU))
+ if (!chip->info->max_vid)
return -EOPNOTSUPP;
mutex_lock(&chip->reg_lock);
@@ -2001,96 +1613,6 @@
return err;
}
-static int _mv88e6xxx_atu_mac_write(struct mv88e6xxx_chip *chip,
- const unsigned char *addr)
-{
- int i, err;
-
- for (i = 0; i < 3; i++) {
- err = mv88e6xxx_g1_write(chip, GLOBAL_ATU_MAC_01 + i,
- (addr[i * 2] << 8) | addr[i * 2 + 1]);
- if (err)
- return err;
- }
-
- return 0;
-}
-
-static int _mv88e6xxx_atu_mac_read(struct mv88e6xxx_chip *chip,
- unsigned char *addr)
-{
- u16 val;
- int i, err;
-
- for (i = 0; i < 3; i++) {
- err = mv88e6xxx_g1_read(chip, GLOBAL_ATU_MAC_01 + i, &val);
- if (err)
- return err;
-
- addr[i * 2] = val >> 8;
- addr[i * 2 + 1] = val & 0xff;
- }
-
- return 0;
-}
-
-static int _mv88e6xxx_atu_load(struct mv88e6xxx_chip *chip,
- struct mv88e6xxx_atu_entry *entry)
-{
- int ret;
-
- ret = _mv88e6xxx_atu_wait(chip);
- if (ret < 0)
- return ret;
-
- ret = _mv88e6xxx_atu_mac_write(chip, entry->mac);
- if (ret < 0)
- return ret;
-
- ret = _mv88e6xxx_atu_data_write(chip, entry);
- if (ret < 0)
- return ret;
-
- return _mv88e6xxx_atu_cmd(chip, entry->fid, GLOBAL_ATU_OP_LOAD_DB);
-}
-
-static int _mv88e6xxx_atu_getnext(struct mv88e6xxx_chip *chip, u16 fid,
- struct mv88e6xxx_atu_entry *entry);
-
-static int mv88e6xxx_atu_get(struct mv88e6xxx_chip *chip, int fid,
- const u8 *addr, struct mv88e6xxx_atu_entry *entry)
-{
- struct mv88e6xxx_atu_entry next;
- int err;
-
- memcpy(next.mac, addr, ETH_ALEN);
- eth_addr_dec(next.mac);
-
- err = _mv88e6xxx_atu_mac_write(chip, next.mac);
- if (err)
- return err;
-
- do {
- err = _mv88e6xxx_atu_getnext(chip, fid, &next);
- if (err)
- return err;
-
- if (next.state == GLOBAL_ATU_DATA_STATE_UNUSED)
- break;
-
- if (ether_addr_equal(next.mac, addr)) {
- *entry = next;
- return 0;
- }
- } while (ether_addr_greater(addr, next.mac));
-
- memset(entry, 0, sizeof(*entry));
- entry->fid = fid;
- ether_addr_copy(entry->mac, addr);
-
- return 0;
-}
-
static int mv88e6xxx_port_db_load_purge(struct mv88e6xxx_chip *chip, int port,
const unsigned char *addr, u16 vid,
u8 state)
@@ -2103,25 +1625,36 @@
if (vid == 0)
err = mv88e6xxx_port_get_fid(chip, port, &vlan.fid);
else
- err = _mv88e6xxx_vtu_get(chip, vid, &vlan, false);
+ err = mv88e6xxx_vtu_get(chip, vid, &vlan, false);
if (err)
return err;
- err = mv88e6xxx_atu_get(chip, vlan.fid, addr, &entry);
+ entry.state = GLOBAL_ATU_DATA_STATE_UNUSED;
+ ether_addr_copy(entry.mac, addr);
+ eth_addr_dec(entry.mac);
+
+ err = mv88e6xxx_g1_atu_getnext(chip, vlan.fid, &entry);
if (err)
return err;
+ /* Initialize a fresh ATU entry if it isn't found */
+ if (entry.state == GLOBAL_ATU_DATA_STATE_UNUSED ||
+ !ether_addr_equal(entry.mac, addr)) {
+ memset(&entry, 0, sizeof(entry));
+ ether_addr_copy(entry.mac, addr);
+ }
+
/* Purge the ATU entry only if no port is using it anymore */
if (state == GLOBAL_ATU_DATA_STATE_UNUSED) {
- entry.portv_trunkid &= ~BIT(port);
- if (!entry.portv_trunkid)
+ entry.portvec &= ~BIT(port);
+ if (!entry.portvec)
entry.state = GLOBAL_ATU_DATA_STATE_UNUSED;
} else {
- entry.portv_trunkid |= BIT(port);
+ entry.portvec |= BIT(port);
entry.state = state;
}
- return _mv88e6xxx_atu_load(chip, &entry);
+ return mv88e6xxx_g1_atu_loadpurge(chip, vlan.fid, &entry);
}
static int mv88e6xxx_port_fdb_prepare(struct dsa_switch *ds, int port,
@@ -2161,75 +1694,26 @@
return err;
}
-static int _mv88e6xxx_atu_getnext(struct mv88e6xxx_chip *chip, u16 fid,
- struct mv88e6xxx_atu_entry *entry)
-{
- struct mv88e6xxx_atu_entry next = { 0 };
- u16 val;
- int err;
-
- next.fid = fid;
-
- err = _mv88e6xxx_atu_wait(chip);
- if (err)
- return err;
-
- err = _mv88e6xxx_atu_cmd(chip, fid, GLOBAL_ATU_OP_GET_NEXT_DB);
- if (err)
- return err;
-
- err = _mv88e6xxx_atu_mac_read(chip, next.mac);
- if (err)
- return err;
-
- err = mv88e6xxx_g1_read(chip, GLOBAL_ATU_DATA, &val);
- if (err)
- return err;
-
- next.state = val & GLOBAL_ATU_DATA_STATE_MASK;
- if (next.state != GLOBAL_ATU_DATA_STATE_UNUSED) {
- unsigned int mask, shift;
-
- if (val & GLOBAL_ATU_DATA_TRUNK) {
- next.trunk = true;
- mask = GLOBAL_ATU_DATA_TRUNK_ID_MASK;
- shift = GLOBAL_ATU_DATA_TRUNK_ID_SHIFT;
- } else {
- next.trunk = false;
- mask = GLOBAL_ATU_DATA_PORT_VECTOR_MASK;
- shift = GLOBAL_ATU_DATA_PORT_VECTOR_SHIFT;
- }
-
- next.portv_trunkid = (val & mask) >> shift;
- }
-
- *entry = next;
- return 0;
-}
-
static int mv88e6xxx_port_db_dump_fid(struct mv88e6xxx_chip *chip,
u16 fid, u16 vid, int port,
struct switchdev_obj *obj,
int (*cb)(struct switchdev_obj *obj))
{
- struct mv88e6xxx_atu_entry addr = {
- .mac = { 0xff, 0xff, 0xff, 0xff, 0xff, 0xff },
- };
+ struct mv88e6xxx_atu_entry addr;
int err;
- err = _mv88e6xxx_atu_mac_write(chip, addr.mac);
- if (err)
- return err;
+ addr.state = GLOBAL_ATU_DATA_STATE_UNUSED;
+ eth_broadcast_addr(addr.mac);
do {
- err = _mv88e6xxx_atu_getnext(chip, fid, &addr);
+ err = mv88e6xxx_g1_atu_getnext(chip, fid, &addr);
if (err)
return err;
if (addr.state == GLOBAL_ATU_DATA_STATE_UNUSED)
break;
- if (addr.trunk || (addr.portv_trunkid & BIT(port)) == 0)
+ if (addr.trunk || (addr.portvec & BIT(port)) == 0)
continue;
if (obj->id == SWITCHDEV_OBJ_ID_PORT_FDB) {
@@ -2271,7 +1755,7 @@
int (*cb)(struct switchdev_obj *obj))
{
struct mv88e6xxx_vtu_entry vlan = {
- .vid = GLOBAL_VTU_VID_MASK, /* all ones */
+ .vid = chip->info->max_vid,
};
u16 fid;
int err;
@@ -2286,12 +1770,8 @@
return err;
/* Dump VLANs' Filtering Information Databases */
- err = _mv88e6xxx_vtu_vid_write(chip, vlan.vid);
- if (err)
- return err;
-
do {
- err = _mv88e6xxx_vtu_getnext(chip, &vlan);
+ err = mv88e6xxx_vtu_getnext(chip, &vlan);
if (err)
return err;
@@ -2302,7 +1782,7 @@
obj, cb);
if (err)
return err;
- } while (vlan.vid < GLOBAL_VTU_VID_MASK);
+ } while (vlan.vid < chip->info->max_vid);
return err;
}
@@ -2321,23 +1801,52 @@
return err;
}
+static int mv88e6xxx_bridge_map(struct mv88e6xxx_chip *chip,
+ struct net_device *br)
+{
+ struct dsa_switch *ds;
+ int port;
+ int dev;
+ int err;
+
+ /* Remap the Port VLAN of each local bridge group member */
+ for (port = 0; port < mv88e6xxx_num_ports(chip); ++port) {
+ if (chip->ds->ports[port].bridge_dev == br) {
+ err = mv88e6xxx_port_vlan_map(chip, port);
+ if (err)
+ return err;
+ }
+ }
+
+ if (!mv88e6xxx_has_pvt(chip))
+ return 0;
+
+ /* Remap the Port VLAN of each cross-chip bridge group member */
+ for (dev = 0; dev < DSA_MAX_SWITCHES; ++dev) {
+ ds = chip->ds->dst->ds[dev];
+ if (!ds)
+ break;
+
+ for (port = 0; port < ds->num_ports; ++port) {
+ if (ds->ports[port].bridge_dev == br) {
+ err = mv88e6xxx_pvt_map(chip, dev, port);
+ if (err)
+ return err;
+ }
+ }
+ }
+
+ return 0;
+}
+
static int mv88e6xxx_port_bridge_join(struct dsa_switch *ds, int port,
struct net_device *br)
{
struct mv88e6xxx_chip *chip = ds->priv;
- int i, err = 0;
+ int err;
mutex_lock(&chip->reg_lock);
-
- /* Remap each port's VLANTable */
- for (i = 0; i < mv88e6xxx_num_ports(chip); ++i) {
- if (ds->ports[i].bridge_dev == br) {
- err = _mv88e6xxx_port_based_vlan_map(chip, i);
- if (err)
- break;
- }
- }
-
+ err = mv88e6xxx_bridge_map(chip, br);
mutex_unlock(&chip->reg_lock);
return err;
@@ -2347,17 +1856,41 @@
struct net_device *br)
{
struct mv88e6xxx_chip *chip = ds->priv;
- int i;
mutex_lock(&chip->reg_lock);
+ if (mv88e6xxx_bridge_map(chip, br) ||
+ mv88e6xxx_port_vlan_map(chip, port))
+ dev_err(ds->dev, "failed to remap in-chip Port VLAN\n");
+ mutex_unlock(&chip->reg_lock);
+}
- /* Remap each port's VLANTable */
- for (i = 0; i < mv88e6xxx_num_ports(chip); ++i)
- if (i == port || ds->ports[i].bridge_dev == br)
- if (_mv88e6xxx_port_based_vlan_map(chip, i))
- netdev_warn(ds->ports[i].netdev,
- "failed to remap\n");
+static int mv88e6xxx_crosschip_bridge_join(struct dsa_switch *ds, int dev,
+ int port, struct net_device *br)
+{
+ struct mv88e6xxx_chip *chip = ds->priv;
+ int err;
+ if (!mv88e6xxx_has_pvt(chip))
+ return 0;
+
+ mutex_lock(&chip->reg_lock);
+ err = mv88e6xxx_pvt_map(chip, dev, port);
+ mutex_unlock(&chip->reg_lock);
+
+ return err;
+}
+
+static void mv88e6xxx_crosschip_bridge_leave(struct dsa_switch *ds, int dev,
+ int port, struct net_device *br)
+{
+ struct mv88e6xxx_chip *chip = ds->priv;
+
+ if (!mv88e6xxx_has_pvt(chip))
+ return;
+
+ mutex_lock(&chip->reg_lock);
+ if (mv88e6xxx_pvt_map(chip, dev, port))
+ dev_err(ds->dev, "failed to remap cross-chip Port VLAN\n");
mutex_unlock(&chip->reg_lock);
}
@@ -2433,70 +1966,85 @@
return err;
}
-static int mv88e6xxx_setup_port_dsa(struct mv88e6xxx_chip *chip, int port,
- int upstream_port)
+static int mv88e6xxx_set_port_mode(struct mv88e6xxx_chip *chip, int port,
+ enum mv88e6xxx_frame_mode frame, u16 egress,
+ u16 etype)
{
int err;
- err = chip->info->ops->port_set_frame_mode(
- chip, port, MV88E6XXX_FRAME_MODE_DSA);
+ if (!chip->info->ops->port_set_frame_mode)
+ return -EOPNOTSUPP;
+
+ err = mv88e6xxx_port_set_egress_mode(chip, port, egress);
if (err)
return err;
- return chip->info->ops->port_set_egress_unknowns(
- chip, port, port == upstream_port);
+ err = chip->info->ops->port_set_frame_mode(chip, port, frame);
+ if (err)
+ return err;
+
+ if (chip->info->ops->port_set_ether_type)
+ return chip->info->ops->port_set_ether_type(chip, port, etype);
+
+ return 0;
}
-static int mv88e6xxx_setup_port_cpu(struct mv88e6xxx_chip *chip, int port)
+static int mv88e6xxx_set_port_mode_normal(struct mv88e6xxx_chip *chip, int port)
{
- int err;
-
- switch (chip->info->tag_protocol) {
- case DSA_TAG_PROTO_EDSA:
- err = chip->info->ops->port_set_frame_mode(
- chip, port, MV88E6XXX_FRAME_MODE_ETHERTYPE);
- if (err)
- return err;
-
- err = mv88e6xxx_port_set_egress_mode(
- chip, port, PORT_CONTROL_EGRESS_ADD_TAG);
- if (err)
- return err;
-
- if (chip->info->ops->port_set_ether_type)
- err = chip->info->ops->port_set_ether_type(
- chip, port, ETH_P_EDSA);
- break;
-
- case DSA_TAG_PROTO_DSA:
- err = chip->info->ops->port_set_frame_mode(
- chip, port, MV88E6XXX_FRAME_MODE_DSA);
- if (err)
- return err;
-
- err = mv88e6xxx_port_set_egress_mode(
- chip, port, PORT_CONTROL_EGRESS_UNMODIFIED);
- break;
- default:
- err = -EINVAL;
- }
-
- if (err)
- return err;
-
- return chip->info->ops->port_set_egress_unknowns(chip, port, true);
+ return mv88e6xxx_set_port_mode(chip, port, MV88E6XXX_FRAME_MODE_NORMAL,
+ PORT_CONTROL_EGRESS_UNMODIFIED,
+ PORT_ETH_TYPE_DEFAULT);
}
-static int mv88e6xxx_setup_port_normal(struct mv88e6xxx_chip *chip, int port)
+static int mv88e6xxx_set_port_mode_dsa(struct mv88e6xxx_chip *chip, int port)
{
- int err;
+ return mv88e6xxx_set_port_mode(chip, port, MV88E6XXX_FRAME_MODE_DSA,
+ PORT_CONTROL_EGRESS_UNMODIFIED,
+ PORT_ETH_TYPE_DEFAULT);
+}
- err = chip->info->ops->port_set_frame_mode(
- chip, port, MV88E6XXX_FRAME_MODE_NORMAL);
- if (err)
- return err;
+static int mv88e6xxx_set_port_mode_edsa(struct mv88e6xxx_chip *chip, int port)
+{
+ return mv88e6xxx_set_port_mode(chip, port,
+ MV88E6XXX_FRAME_MODE_ETHERTYPE,
+ PORT_CONTROL_EGRESS_ADD_TAG, ETH_P_EDSA);
+}
- return chip->info->ops->port_set_egress_unknowns(chip, port, false);
+static int mv88e6xxx_setup_port_mode(struct mv88e6xxx_chip *chip, int port)
+{
+ if (dsa_is_dsa_port(chip->ds, port))
+ return mv88e6xxx_set_port_mode_dsa(chip, port);
+
+ if (dsa_is_normal_port(chip->ds, port))
+ return mv88e6xxx_set_port_mode_normal(chip, port);
+
+ /* Setup CPU port mode depending on its supported tag format */
+ if (chip->info->tag_protocol == DSA_TAG_PROTO_DSA)
+ return mv88e6xxx_set_port_mode_dsa(chip, port);
+
+ if (chip->info->tag_protocol == DSA_TAG_PROTO_EDSA)
+ return mv88e6xxx_set_port_mode_edsa(chip, port);
+
+ return -EINVAL;
+}
+
+static int mv88e6xxx_setup_message_port(struct mv88e6xxx_chip *chip, int port)
+{
+ bool message = dsa_is_dsa_port(chip->ds, port);
+
+ return mv88e6xxx_port_set_message_port(chip, port, message);
+}
+
+static int mv88e6xxx_setup_egress_floods(struct mv88e6xxx_chip *chip, int port)
+{
+ bool flood = port == dsa_upstream_port(chip->ds);
+
+ /* Upstream ports flood frames with unknown unicast or multicast DA */
+ if (chip->info->ops->port_set_egress_floods)
+ return chip->info->ops->port_set_egress_floods(chip, port,
+ flood, flood);
+
+ return 0;
}
static int mv88e6xxx_setup_port(struct mv88e6xxx_chip *chip, int port)
@@ -2541,14 +2089,11 @@
if (err)
return err;
- if (dsa_is_cpu_port(ds, port)) {
- err = mv88e6xxx_setup_port_cpu(chip, port);
- } else if (dsa_is_dsa_port(ds, port)) {
- err = mv88e6xxx_setup_port_dsa(chip, port,
- dsa_upstream_port(ds));
- } else {
- err = mv88e6xxx_setup_port_normal(chip, port);
- }
+ err = mv88e6xxx_setup_port_mode(chip, port);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_setup_egress_floods(chip, port);
if (err)
return err;
@@ -2623,20 +2168,14 @@
return err;
}
- if (mv88e6xxx_6352_family(chip) || mv88e6xxx_6351_family(chip) ||
- mv88e6xxx_6165_family(chip) || mv88e6xxx_6097_family(chip) ||
- mv88e6xxx_6320_family(chip) || mv88e6xxx_6341_family(chip)) {
- /* Port ATU control: disable limiting the number of
- * address database entries that this port is allowed
- * to use.
- */
- err = mv88e6xxx_port_write(chip, port, PORT_ATU_CONTROL,
- 0x0000);
- /* Priority Override: disable DA, SA and VTU priority
- * override.
- */
- err = mv88e6xxx_port_write(chip, port, PORT_PRI_OVERRIDE,
- 0x0000);
+ if (chip->info->ops->port_disable_learn_limit) {
+ err = chip->info->ops->port_disable_learn_limit(chip, port);
+ if (err)
+ return err;
+ }
+
+ if (chip->info->ops->port_disable_pri_override) {
+ err = chip->info->ops->port_disable_pri_override(chip, port);
if (err)
return err;
}
@@ -2653,10 +2192,7 @@
return err;
}
- /* Port Control 1: disable trunking, disable sending
- * learning messages to this port.
- */
- err = mv88e6xxx_port_write(chip, port, PORT_CONTROL_1, 0x0000);
+ err = mv88e6xxx_setup_message_port(chip, port);
if (err)
return err;
@@ -2668,7 +2204,7 @@
if (err)
return err;
- err = _mv88e6xxx_port_based_vlan_map(chip, port);
+ err = mv88e6xxx_port_vlan_map(chip, port);
if (err)
return err;
@@ -2697,33 +2233,6 @@
return 0;
}
-static int mv88e6xxx_g1_set_age_time(struct mv88e6xxx_chip *chip,
- unsigned int msecs)
-{
- const unsigned int coeff = chip->info->age_time_coeff;
- const unsigned int min = 0x01 * coeff;
- const unsigned int max = 0xff * coeff;
- u8 age_time;
- u16 val;
- int err;
-
- if (msecs < min || msecs > max)
- return -ERANGE;
-
- /* Round to nearest multiple of coeff */
- age_time = (msecs + coeff / 2) / coeff;
-
- err = mv88e6xxx_g1_read(chip, GLOBAL_ATU_CONTROL, &val);
- if (err)
- return err;
-
- /* AgeTime is 11:4 bits */
- val &= ~0xff0;
- val |= age_time << 4;
-
- return mv88e6xxx_g1_write(chip, GLOBAL_ATU_CONTROL, val);
-}
-
static int mv88e6xxx_set_ageing_time(struct dsa_switch *ds,
unsigned int ageing_time)
{
@@ -2731,7 +2240,7 @@
int err;
mutex_lock(&chip->reg_lock);
- err = mv88e6xxx_g1_set_age_time(chip, ageing_time);
+ err = mv88e6xxx_g1_atu_set_age_time(chip, ageing_time);
mutex_unlock(&chip->reg_lock);
return err;
@@ -2769,29 +2278,6 @@
if (err)
return err;
- /* Clear all the VTU and STU entries */
- err = _mv88e6xxx_vtu_stu_flush(chip);
- if (err < 0)
- return err;
-
- /* Set the default address aging time to 5 minutes, and
- * enable address learn messages to be sent to all message
- * ports.
- */
- err = mv88e6xxx_g1_write(chip, GLOBAL_ATU_CONTROL,
- GLOBAL_ATU_CONTROL_LEARN2ALL);
- if (err)
- return err;
-
- err = mv88e6xxx_g1_set_age_time(chip, 300000);
- if (err)
- return err;
-
- /* Clear all ATU entries */
- err = _mv88e6xxx_atu_flush(chip, 0, true);
- if (err)
- return err;
-
/* Configure the IP ToS mapping registers. */
err = mv88e6xxx_g1_write(chip, GLOBAL_IP_PRI_0, 0x0000);
if (err)
@@ -2872,6 +2358,18 @@
goto unlock;
}
+ err = mv88e6xxx_vtu_setup(chip);
+ if (err)
+ goto unlock;
+
+ err = mv88e6xxx_pvt_setup(chip);
+ if (err)
+ goto unlock;
+
+ err = mv88e6xxx_atu_setup(chip);
+ if (err)
+ goto unlock;
+
/* Some generations have the configuration of sending reserved
* management frames to the CPU in global2, others in
* global1. Hence it does not fit the two setup functions
@@ -3101,10 +2599,12 @@
.port_set_speed = mv88e6185_port_set_speed,
.port_tag_remap = mv88e6095_port_tag_remap,
.port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
.port_set_ether_type = mv88e6351_port_set_ether_type,
.port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
.port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
.stats_snapshot = mv88e6xxx_g1_stats_snapshot,
.stats_get_sset_count = mv88e6095_stats_get_sset_count,
.stats_get_strings = mv88e6095_stats_get_strings,
@@ -3116,6 +2616,8 @@
.ppu_enable = mv88e6185_g1_ppu_enable,
.ppu_disable = mv88e6185_g1_ppu_disable,
.reset = mv88e6185_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
};
static const struct mv88e6xxx_ops mv88e6095_ops = {
@@ -3127,7 +2629,7 @@
.port_set_duplex = mv88e6xxx_port_set_duplex,
.port_set_speed = mv88e6185_port_set_speed,
.port_set_frame_mode = mv88e6085_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6095_port_set_egress_unknowns,
+ .port_set_egress_floods = mv88e6185_port_set_egress_floods,
.port_set_upstream_port = mv88e6095_port_set_upstream_port,
.stats_snapshot = mv88e6xxx_g1_stats_snapshot,
.stats_get_sset_count = mv88e6095_stats_get_sset_count,
@@ -3137,6 +2639,8 @@
.ppu_enable = mv88e6185_g1_ppu_enable,
.ppu_disable = mv88e6185_g1_ppu_disable,
.reset = mv88e6185_g1_reset,
+ .vtu_getnext = mv88e6185_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6185_g1_vtu_loadpurge,
};
static const struct mv88e6xxx_ops mv88e6097_ops = {
@@ -3149,11 +2653,13 @@
.port_set_speed = mv88e6185_port_set_speed,
.port_tag_remap = mv88e6095_port_tag_remap,
.port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
.port_set_ether_type = mv88e6351_port_set_ether_type,
.port_jumbo_config = mv88e6165_port_jumbo_config,
.port_egress_rate_limiting = mv88e6095_port_egress_rate_limiting,
.port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
.stats_snapshot = mv88e6xxx_g1_stats_snapshot,
.stats_get_sset_count = mv88e6095_stats_get_sset_count,
.stats_get_strings = mv88e6095_stats_get_strings,
@@ -3163,6 +2669,8 @@
.watchdog_ops = &mv88e6097_watchdog_ops,
.mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
.reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
};
static const struct mv88e6xxx_ops mv88e6123_ops = {
@@ -3174,7 +2682,9 @@
.port_set_duplex = mv88e6xxx_port_set_duplex,
.port_set_speed = mv88e6185_port_set_speed,
.port_set_frame_mode = mv88e6085_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6085_port_set_egress_unknowns,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
.stats_snapshot = mv88e6xxx_g1_stats_snapshot,
.stats_get_sset_count = mv88e6095_stats_get_sset_count,
.stats_get_strings = mv88e6095_stats_get_strings,
@@ -3184,6 +2694,8 @@
.watchdog_ops = &mv88e6097_watchdog_ops,
.mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
.reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
};
static const struct mv88e6xxx_ops mv88e6131_ops = {
@@ -3196,7 +2708,7 @@
.port_set_speed = mv88e6185_port_set_speed,
.port_tag_remap = mv88e6095_port_tag_remap,
.port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6095_port_set_egress_unknowns,
+ .port_set_egress_floods = mv88e6185_port_set_egress_floods,
.port_set_ether_type = mv88e6351_port_set_ether_type,
.port_set_upstream_port = mv88e6095_port_set_upstream_port,
.port_jumbo_config = mv88e6165_port_jumbo_config,
@@ -3213,466 +2725,8 @@
.ppu_enable = mv88e6185_g1_ppu_enable,
.ppu_disable = mv88e6185_g1_ppu_disable,
.reset = mv88e6185_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6161_ops = {
- /* MV88E6XXX_FAMILY_6165 */
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6165_phy_read,
- .phy_write = mv88e6165_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_speed = mv88e6185_port_set_speed,
- .port_tag_remap = mv88e6095_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_jumbo_config = mv88e6165_port_jumbo_config,
- .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
- .port_pause_config = mv88e6097_port_pause_config,
- .stats_snapshot = mv88e6xxx_g1_stats_snapshot,
- .stats_get_sset_count = mv88e6095_stats_get_sset_count,
- .stats_get_strings = mv88e6095_stats_get_strings,
- .stats_get_stats = mv88e6095_stats_get_stats,
- .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6095_g1_set_egress_port,
- .watchdog_ops = &mv88e6097_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6165_ops = {
- /* MV88E6XXX_FAMILY_6165 */
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6165_phy_read,
- .phy_write = mv88e6165_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_speed = mv88e6185_port_set_speed,
- .stats_snapshot = mv88e6xxx_g1_stats_snapshot,
- .stats_get_sset_count = mv88e6095_stats_get_sset_count,
- .stats_get_strings = mv88e6095_stats_get_strings,
- .stats_get_stats = mv88e6095_stats_get_stats,
- .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6095_g1_set_egress_port,
- .watchdog_ops = &mv88e6097_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6171_ops = {
- /* MV88E6XXX_FAMILY_6351 */
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
- .port_set_speed = mv88e6185_port_set_speed,
- .port_tag_remap = mv88e6095_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_jumbo_config = mv88e6165_port_jumbo_config,
- .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
- .port_pause_config = mv88e6097_port_pause_config,
- .stats_snapshot = mv88e6320_g1_stats_snapshot,
- .stats_get_sset_count = mv88e6095_stats_get_sset_count,
- .stats_get_strings = mv88e6095_stats_get_strings,
- .stats_get_stats = mv88e6095_stats_get_stats,
- .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6095_g1_set_egress_port,
- .watchdog_ops = &mv88e6097_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6172_ops = {
- /* MV88E6XXX_FAMILY_6352 */
- .get_eeprom = mv88e6xxx_g2_get_eeprom16,
- .set_eeprom = mv88e6xxx_g2_set_eeprom16,
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
- .port_set_speed = mv88e6352_port_set_speed,
- .port_tag_remap = mv88e6095_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_jumbo_config = mv88e6165_port_jumbo_config,
- .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
- .port_pause_config = mv88e6097_port_pause_config,
- .stats_snapshot = mv88e6320_g1_stats_snapshot,
- .stats_get_sset_count = mv88e6095_stats_get_sset_count,
- .stats_get_strings = mv88e6095_stats_get_strings,
- .stats_get_stats = mv88e6095_stats_get_stats,
- .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6095_g1_set_egress_port,
- .watchdog_ops = &mv88e6097_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6175_ops = {
- /* MV88E6XXX_FAMILY_6351 */
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
- .port_set_speed = mv88e6185_port_set_speed,
- .port_tag_remap = mv88e6095_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_jumbo_config = mv88e6165_port_jumbo_config,
- .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
- .port_pause_config = mv88e6097_port_pause_config,
- .stats_snapshot = mv88e6320_g1_stats_snapshot,
- .stats_get_sset_count = mv88e6095_stats_get_sset_count,
- .stats_get_strings = mv88e6095_stats_get_strings,
- .stats_get_stats = mv88e6095_stats_get_stats,
- .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6095_g1_set_egress_port,
- .watchdog_ops = &mv88e6097_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6176_ops = {
- /* MV88E6XXX_FAMILY_6352 */
- .get_eeprom = mv88e6xxx_g2_get_eeprom16,
- .set_eeprom = mv88e6xxx_g2_set_eeprom16,
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
- .port_set_speed = mv88e6352_port_set_speed,
- .port_tag_remap = mv88e6095_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_jumbo_config = mv88e6165_port_jumbo_config,
- .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
- .port_pause_config = mv88e6097_port_pause_config,
- .stats_snapshot = mv88e6320_g1_stats_snapshot,
- .stats_get_sset_count = mv88e6095_stats_get_sset_count,
- .stats_get_strings = mv88e6095_stats_get_strings,
- .stats_get_stats = mv88e6095_stats_get_stats,
- .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6095_g1_set_egress_port,
- .watchdog_ops = &mv88e6097_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6185_ops = {
- /* MV88E6XXX_FAMILY_6185 */
- .set_switch_mac = mv88e6xxx_g1_set_switch_mac,
- .phy_read = mv88e6xxx_phy_ppu_read,
- .phy_write = mv88e6xxx_phy_ppu_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_speed = mv88e6185_port_set_speed,
- .port_set_frame_mode = mv88e6085_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6095_port_set_egress_unknowns,
- .port_egress_rate_limiting = mv88e6095_port_egress_rate_limiting,
- .port_set_upstream_port = mv88e6095_port_set_upstream_port,
- .stats_snapshot = mv88e6xxx_g1_stats_snapshot,
- .stats_get_sset_count = mv88e6095_stats_get_sset_count,
- .stats_get_strings = mv88e6095_stats_get_strings,
- .stats_get_stats = mv88e6095_stats_get_stats,
- .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6095_g1_set_egress_port,
- .watchdog_ops = &mv88e6097_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
- .ppu_enable = mv88e6185_g1_ppu_enable,
- .ppu_disable = mv88e6185_g1_ppu_disable,
- .reset = mv88e6185_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6190_ops = {
- /* MV88E6XXX_FAMILY_6390 */
- .get_eeprom = mv88e6xxx_g2_get_eeprom8,
- .set_eeprom = mv88e6xxx_g2_set_eeprom8,
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_rgmii_delay = mv88e6390_port_set_rgmii_delay,
- .port_set_speed = mv88e6390_port_set_speed,
- .port_tag_remap = mv88e6390_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_pause_config = mv88e6390_port_pause_config,
- .stats_snapshot = mv88e6390_g1_stats_snapshot,
- .stats_set_histogram = mv88e6390_g1_stats_set_histogram,
- .stats_get_sset_count = mv88e6320_stats_get_sset_count,
- .stats_get_strings = mv88e6320_stats_get_strings,
- .stats_get_stats = mv88e6390_stats_get_stats,
- .g1_set_cpu_port = mv88e6390_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6390_g1_set_egress_port,
- .watchdog_ops = &mv88e6390_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6390_g1_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6190x_ops = {
- /* MV88E6XXX_FAMILY_6390 */
- .get_eeprom = mv88e6xxx_g2_get_eeprom8,
- .set_eeprom = mv88e6xxx_g2_set_eeprom8,
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_rgmii_delay = mv88e6390_port_set_rgmii_delay,
- .port_set_speed = mv88e6390x_port_set_speed,
- .port_tag_remap = mv88e6390_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_pause_config = mv88e6390_port_pause_config,
- .stats_snapshot = mv88e6390_g1_stats_snapshot,
- .stats_set_histogram = mv88e6390_g1_stats_set_histogram,
- .stats_get_sset_count = mv88e6320_stats_get_sset_count,
- .stats_get_strings = mv88e6320_stats_get_strings,
- .stats_get_stats = mv88e6390_stats_get_stats,
- .g1_set_cpu_port = mv88e6390_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6390_g1_set_egress_port,
- .watchdog_ops = &mv88e6390_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6390_g1_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6191_ops = {
- /* MV88E6XXX_FAMILY_6390 */
- .get_eeprom = mv88e6xxx_g2_get_eeprom8,
- .set_eeprom = mv88e6xxx_g2_set_eeprom8,
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_rgmii_delay = mv88e6390_port_set_rgmii_delay,
- .port_set_speed = mv88e6390_port_set_speed,
- .port_tag_remap = mv88e6390_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_pause_config = mv88e6390_port_pause_config,
- .stats_snapshot = mv88e6390_g1_stats_snapshot,
- .stats_set_histogram = mv88e6390_g1_stats_set_histogram,
- .stats_get_sset_count = mv88e6320_stats_get_sset_count,
- .stats_get_strings = mv88e6320_stats_get_strings,
- .stats_get_stats = mv88e6390_stats_get_stats,
- .g1_set_cpu_port = mv88e6390_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6390_g1_set_egress_port,
- .watchdog_ops = &mv88e6390_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6390_g1_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6240_ops = {
- /* MV88E6XXX_FAMILY_6352 */
- .get_eeprom = mv88e6xxx_g2_get_eeprom16,
- .set_eeprom = mv88e6xxx_g2_set_eeprom16,
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
- .port_set_speed = mv88e6352_port_set_speed,
- .port_tag_remap = mv88e6095_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_jumbo_config = mv88e6165_port_jumbo_config,
- .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
- .port_pause_config = mv88e6097_port_pause_config,
- .stats_snapshot = mv88e6320_g1_stats_snapshot,
- .stats_get_sset_count = mv88e6095_stats_get_sset_count,
- .stats_get_strings = mv88e6095_stats_get_strings,
- .stats_get_stats = mv88e6095_stats_get_stats,
- .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6095_g1_set_egress_port,
- .watchdog_ops = &mv88e6097_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6290_ops = {
- /* MV88E6XXX_FAMILY_6390 */
- .get_eeprom = mv88e6xxx_g2_get_eeprom8,
- .set_eeprom = mv88e6xxx_g2_set_eeprom8,
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_rgmii_delay = mv88e6390_port_set_rgmii_delay,
- .port_set_speed = mv88e6390_port_set_speed,
- .port_tag_remap = mv88e6390_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_pause_config = mv88e6390_port_pause_config,
- .port_set_cmode = mv88e6390x_port_set_cmode,
- .stats_snapshot = mv88e6390_g1_stats_snapshot,
- .stats_set_histogram = mv88e6390_g1_stats_set_histogram,
- .stats_get_sset_count = mv88e6320_stats_get_sset_count,
- .stats_get_strings = mv88e6320_stats_get_strings,
- .stats_get_stats = mv88e6390_stats_get_stats,
- .g1_set_cpu_port = mv88e6390_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6390_g1_set_egress_port,
- .watchdog_ops = &mv88e6390_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6390_g1_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6320_ops = {
- /* MV88E6XXX_FAMILY_6320 */
- .get_eeprom = mv88e6xxx_g2_get_eeprom16,
- .set_eeprom = mv88e6xxx_g2_set_eeprom16,
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_speed = mv88e6185_port_set_speed,
- .port_tag_remap = mv88e6095_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_jumbo_config = mv88e6165_port_jumbo_config,
- .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
- .port_pause_config = mv88e6097_port_pause_config,
- .stats_snapshot = mv88e6320_g1_stats_snapshot,
- .stats_get_sset_count = mv88e6320_stats_get_sset_count,
- .stats_get_strings = mv88e6320_stats_get_strings,
- .stats_get_stats = mv88e6320_stats_get_stats,
- .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6095_g1_set_egress_port,
- .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6321_ops = {
- /* MV88E6XXX_FAMILY_6321 */
- .get_eeprom = mv88e6xxx_g2_get_eeprom16,
- .set_eeprom = mv88e6xxx_g2_set_eeprom16,
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_speed = mv88e6185_port_set_speed,
- .port_tag_remap = mv88e6095_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_jumbo_config = mv88e6165_port_jumbo_config,
- .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
- .port_pause_config = mv88e6097_port_pause_config,
- .stats_snapshot = mv88e6320_g1_stats_snapshot,
- .stats_get_sset_count = mv88e6320_stats_get_sset_count,
- .stats_get_strings = mv88e6320_stats_get_strings,
- .stats_get_stats = mv88e6320_stats_get_stats,
- .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6095_g1_set_egress_port,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6350_ops = {
- /* MV88E6XXX_FAMILY_6351 */
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
- .port_set_speed = mv88e6185_port_set_speed,
- .port_tag_remap = mv88e6095_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_jumbo_config = mv88e6165_port_jumbo_config,
- .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
- .port_pause_config = mv88e6097_port_pause_config,
- .stats_snapshot = mv88e6320_g1_stats_snapshot,
- .stats_get_sset_count = mv88e6095_stats_get_sset_count,
- .stats_get_strings = mv88e6095_stats_get_strings,
- .stats_get_stats = mv88e6095_stats_get_stats,
- .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6095_g1_set_egress_port,
- .watchdog_ops = &mv88e6097_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6351_ops = {
- /* MV88E6XXX_FAMILY_6351 */
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
- .port_set_speed = mv88e6185_port_set_speed,
- .port_tag_remap = mv88e6095_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_jumbo_config = mv88e6165_port_jumbo_config,
- .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
- .port_pause_config = mv88e6097_port_pause_config,
- .stats_snapshot = mv88e6320_g1_stats_snapshot,
- .stats_get_sset_count = mv88e6095_stats_get_sset_count,
- .stats_get_strings = mv88e6095_stats_get_strings,
- .stats_get_stats = mv88e6095_stats_get_stats,
- .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6095_g1_set_egress_port,
- .watchdog_ops = &mv88e6097_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static const struct mv88e6xxx_ops mv88e6352_ops = {
- /* MV88E6XXX_FAMILY_6352 */
- .get_eeprom = mv88e6xxx_g2_get_eeprom16,
- .set_eeprom = mv88e6xxx_g2_set_eeprom16,
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
- .port_set_speed = mv88e6352_port_set_speed,
- .port_tag_remap = mv88e6095_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_jumbo_config = mv88e6165_port_jumbo_config,
- .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
- .port_pause_config = mv88e6097_port_pause_config,
- .stats_snapshot = mv88e6320_g1_stats_snapshot,
- .stats_get_sset_count = mv88e6095_stats_get_sset_count,
- .stats_get_strings = mv88e6095_stats_get_strings,
- .stats_get_stats = mv88e6095_stats_get_stats,
- .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6095_g1_set_egress_port,
- .watchdog_ops = &mv88e6097_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6185_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6185_g1_vtu_loadpurge,
};
static const struct mv88e6xxx_ops mv88e6141_ops = {
@@ -3688,11 +2742,13 @@
.port_set_speed = mv88e6390_port_set_speed,
.port_tag_remap = mv88e6095_port_tag_remap,
.port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
.port_set_ether_type = mv88e6351_port_set_ether_type,
.port_jumbo_config = mv88e6165_port_jumbo_config,
.port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
.port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
.stats_snapshot = mv88e6390_g1_stats_snapshot,
.stats_get_sset_count = mv88e6320_stats_get_sset_count,
.stats_get_strings = mv88e6320_stats_get_strings,
@@ -3702,6 +2758,439 @@
.watchdog_ops = &mv88e6390_watchdog_ops,
.mgmt_rsvd2cpu = mv88e6390_g1_mgmt_rsvd2cpu,
.reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6161_ops = {
+ /* MV88E6XXX_FAMILY_6165 */
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6165_phy_read,
+ .phy_write = mv88e6165_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_speed = mv88e6185_port_set_speed,
+ .port_tag_remap = mv88e6095_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_jumbo_config = mv88e6165_port_jumbo_config,
+ .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
+ .port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6xxx_g1_stats_snapshot,
+ .stats_get_sset_count = mv88e6095_stats_get_sset_count,
+ .stats_get_strings = mv88e6095_stats_get_strings,
+ .stats_get_stats = mv88e6095_stats_get_stats,
+ .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6095_g1_set_egress_port,
+ .watchdog_ops = &mv88e6097_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6165_ops = {
+ /* MV88E6XXX_FAMILY_6165 */
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6165_phy_read,
+ .phy_write = mv88e6165_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_speed = mv88e6185_port_set_speed,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6xxx_g1_stats_snapshot,
+ .stats_get_sset_count = mv88e6095_stats_get_sset_count,
+ .stats_get_strings = mv88e6095_stats_get_strings,
+ .stats_get_stats = mv88e6095_stats_get_stats,
+ .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6095_g1_set_egress_port,
+ .watchdog_ops = &mv88e6097_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6171_ops = {
+ /* MV88E6XXX_FAMILY_6351 */
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
+ .port_set_speed = mv88e6185_port_set_speed,
+ .port_tag_remap = mv88e6095_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_jumbo_config = mv88e6165_port_jumbo_config,
+ .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
+ .port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6320_g1_stats_snapshot,
+ .stats_get_sset_count = mv88e6095_stats_get_sset_count,
+ .stats_get_strings = mv88e6095_stats_get_strings,
+ .stats_get_stats = mv88e6095_stats_get_stats,
+ .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6095_g1_set_egress_port,
+ .watchdog_ops = &mv88e6097_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6172_ops = {
+ /* MV88E6XXX_FAMILY_6352 */
+ .get_eeprom = mv88e6xxx_g2_get_eeprom16,
+ .set_eeprom = mv88e6xxx_g2_set_eeprom16,
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
+ .port_set_speed = mv88e6352_port_set_speed,
+ .port_tag_remap = mv88e6095_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_jumbo_config = mv88e6165_port_jumbo_config,
+ .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
+ .port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6320_g1_stats_snapshot,
+ .stats_get_sset_count = mv88e6095_stats_get_sset_count,
+ .stats_get_strings = mv88e6095_stats_get_strings,
+ .stats_get_stats = mv88e6095_stats_get_stats,
+ .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6095_g1_set_egress_port,
+ .watchdog_ops = &mv88e6097_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6175_ops = {
+ /* MV88E6XXX_FAMILY_6351 */
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
+ .port_set_speed = mv88e6185_port_set_speed,
+ .port_tag_remap = mv88e6095_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_jumbo_config = mv88e6165_port_jumbo_config,
+ .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
+ .port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6320_g1_stats_snapshot,
+ .stats_get_sset_count = mv88e6095_stats_get_sset_count,
+ .stats_get_strings = mv88e6095_stats_get_strings,
+ .stats_get_stats = mv88e6095_stats_get_stats,
+ .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6095_g1_set_egress_port,
+ .watchdog_ops = &mv88e6097_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6176_ops = {
+ /* MV88E6XXX_FAMILY_6352 */
+ .get_eeprom = mv88e6xxx_g2_get_eeprom16,
+ .set_eeprom = mv88e6xxx_g2_set_eeprom16,
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
+ .port_set_speed = mv88e6352_port_set_speed,
+ .port_tag_remap = mv88e6095_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_jumbo_config = mv88e6165_port_jumbo_config,
+ .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
+ .port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6320_g1_stats_snapshot,
+ .stats_get_sset_count = mv88e6095_stats_get_sset_count,
+ .stats_get_strings = mv88e6095_stats_get_strings,
+ .stats_get_stats = mv88e6095_stats_get_stats,
+ .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6095_g1_set_egress_port,
+ .watchdog_ops = &mv88e6097_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6185_ops = {
+ /* MV88E6XXX_FAMILY_6185 */
+ .set_switch_mac = mv88e6xxx_g1_set_switch_mac,
+ .phy_read = mv88e6xxx_phy_ppu_read,
+ .phy_write = mv88e6xxx_phy_ppu_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_speed = mv88e6185_port_set_speed,
+ .port_set_frame_mode = mv88e6085_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6185_port_set_egress_floods,
+ .port_egress_rate_limiting = mv88e6095_port_egress_rate_limiting,
+ .port_set_upstream_port = mv88e6095_port_set_upstream_port,
+ .stats_snapshot = mv88e6xxx_g1_stats_snapshot,
+ .stats_get_sset_count = mv88e6095_stats_get_sset_count,
+ .stats_get_strings = mv88e6095_stats_get_strings,
+ .stats_get_stats = mv88e6095_stats_get_stats,
+ .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6095_g1_set_egress_port,
+ .watchdog_ops = &mv88e6097_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
+ .ppu_enable = mv88e6185_g1_ppu_enable,
+ .ppu_disable = mv88e6185_g1_ppu_disable,
+ .reset = mv88e6185_g1_reset,
+ .vtu_getnext = mv88e6185_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6185_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6190_ops = {
+ /* MV88E6XXX_FAMILY_6390 */
+ .get_eeprom = mv88e6xxx_g2_get_eeprom8,
+ .set_eeprom = mv88e6xxx_g2_set_eeprom8,
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_rgmii_delay = mv88e6390_port_set_rgmii_delay,
+ .port_set_speed = mv88e6390_port_set_speed,
+ .port_tag_remap = mv88e6390_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_pause_config = mv88e6390_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6390_g1_stats_snapshot,
+ .stats_set_histogram = mv88e6390_g1_stats_set_histogram,
+ .stats_get_sset_count = mv88e6320_stats_get_sset_count,
+ .stats_get_strings = mv88e6320_stats_get_strings,
+ .stats_get_stats = mv88e6390_stats_get_stats,
+ .g1_set_cpu_port = mv88e6390_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6390_g1_set_egress_port,
+ .watchdog_ops = &mv88e6390_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6390_g1_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6390_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6390_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6190x_ops = {
+ /* MV88E6XXX_FAMILY_6390 */
+ .get_eeprom = mv88e6xxx_g2_get_eeprom8,
+ .set_eeprom = mv88e6xxx_g2_set_eeprom8,
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_rgmii_delay = mv88e6390_port_set_rgmii_delay,
+ .port_set_speed = mv88e6390x_port_set_speed,
+ .port_tag_remap = mv88e6390_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_pause_config = mv88e6390_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6390_g1_stats_snapshot,
+ .stats_set_histogram = mv88e6390_g1_stats_set_histogram,
+ .stats_get_sset_count = mv88e6320_stats_get_sset_count,
+ .stats_get_strings = mv88e6320_stats_get_strings,
+ .stats_get_stats = mv88e6390_stats_get_stats,
+ .g1_set_cpu_port = mv88e6390_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6390_g1_set_egress_port,
+ .watchdog_ops = &mv88e6390_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6390_g1_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6390_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6390_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6191_ops = {
+ /* MV88E6XXX_FAMILY_6390 */
+ .get_eeprom = mv88e6xxx_g2_get_eeprom8,
+ .set_eeprom = mv88e6xxx_g2_set_eeprom8,
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_rgmii_delay = mv88e6390_port_set_rgmii_delay,
+ .port_set_speed = mv88e6390_port_set_speed,
+ .port_tag_remap = mv88e6390_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_pause_config = mv88e6390_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6390_g1_stats_snapshot,
+ .stats_set_histogram = mv88e6390_g1_stats_set_histogram,
+ .stats_get_sset_count = mv88e6320_stats_get_sset_count,
+ .stats_get_strings = mv88e6320_stats_get_strings,
+ .stats_get_stats = mv88e6390_stats_get_stats,
+ .g1_set_cpu_port = mv88e6390_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6390_g1_set_egress_port,
+ .watchdog_ops = &mv88e6390_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6390_g1_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6390_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6390_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6240_ops = {
+ /* MV88E6XXX_FAMILY_6352 */
+ .get_eeprom = mv88e6xxx_g2_get_eeprom16,
+ .set_eeprom = mv88e6xxx_g2_set_eeprom16,
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
+ .port_set_speed = mv88e6352_port_set_speed,
+ .port_tag_remap = mv88e6095_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_jumbo_config = mv88e6165_port_jumbo_config,
+ .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
+ .port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6320_g1_stats_snapshot,
+ .stats_get_sset_count = mv88e6095_stats_get_sset_count,
+ .stats_get_strings = mv88e6095_stats_get_strings,
+ .stats_get_stats = mv88e6095_stats_get_stats,
+ .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6095_g1_set_egress_port,
+ .watchdog_ops = &mv88e6097_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6290_ops = {
+ /* MV88E6XXX_FAMILY_6390 */
+ .get_eeprom = mv88e6xxx_g2_get_eeprom8,
+ .set_eeprom = mv88e6xxx_g2_set_eeprom8,
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_rgmii_delay = mv88e6390_port_set_rgmii_delay,
+ .port_set_speed = mv88e6390_port_set_speed,
+ .port_tag_remap = mv88e6390_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_pause_config = mv88e6390_port_pause_config,
+ .port_set_cmode = mv88e6390x_port_set_cmode,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6390_g1_stats_snapshot,
+ .stats_set_histogram = mv88e6390_g1_stats_set_histogram,
+ .stats_get_sset_count = mv88e6320_stats_get_sset_count,
+ .stats_get_strings = mv88e6320_stats_get_strings,
+ .stats_get_stats = mv88e6390_stats_get_stats,
+ .g1_set_cpu_port = mv88e6390_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6390_g1_set_egress_port,
+ .watchdog_ops = &mv88e6390_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6390_g1_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6390_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6390_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6320_ops = {
+ /* MV88E6XXX_FAMILY_6320 */
+ .get_eeprom = mv88e6xxx_g2_get_eeprom16,
+ .set_eeprom = mv88e6xxx_g2_set_eeprom16,
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_speed = mv88e6185_port_set_speed,
+ .port_tag_remap = mv88e6095_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_jumbo_config = mv88e6165_port_jumbo_config,
+ .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
+ .port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6320_g1_stats_snapshot,
+ .stats_get_sset_count = mv88e6320_stats_get_sset_count,
+ .stats_get_strings = mv88e6320_stats_get_strings,
+ .stats_get_stats = mv88e6320_stats_get_stats,
+ .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6095_g1_set_egress_port,
+ .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6185_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6185_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6321_ops = {
+ /* MV88E6XXX_FAMILY_6321 */
+ .get_eeprom = mv88e6xxx_g2_get_eeprom16,
+ .set_eeprom = mv88e6xxx_g2_set_eeprom16,
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_speed = mv88e6185_port_set_speed,
+ .port_tag_remap = mv88e6095_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_jumbo_config = mv88e6165_port_jumbo_config,
+ .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
+ .port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6320_g1_stats_snapshot,
+ .stats_get_sset_count = mv88e6320_stats_get_sset_count,
+ .stats_get_strings = mv88e6320_stats_get_strings,
+ .stats_get_stats = mv88e6320_stats_get_stats,
+ .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6095_g1_set_egress_port,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6185_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6185_g1_vtu_loadpurge,
};
static const struct mv88e6xxx_ops mv88e6341_ops = {
@@ -3717,11 +3206,13 @@
.port_set_speed = mv88e6390_port_set_speed,
.port_tag_remap = mv88e6095_port_tag_remap,
.port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
.port_set_ether_type = mv88e6351_port_set_ether_type,
.port_jumbo_config = mv88e6165_port_jumbo_config,
.port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
.port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
.stats_snapshot = mv88e6390_g1_stats_snapshot,
.stats_get_sset_count = mv88e6320_stats_get_sset_count,
.stats_get_strings = mv88e6320_stats_get_strings,
@@ -3731,6 +3222,103 @@
.watchdog_ops = &mv88e6390_watchdog_ops,
.mgmt_rsvd2cpu = mv88e6390_g1_mgmt_rsvd2cpu,
.reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6350_ops = {
+ /* MV88E6XXX_FAMILY_6351 */
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
+ .port_set_speed = mv88e6185_port_set_speed,
+ .port_tag_remap = mv88e6095_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_jumbo_config = mv88e6165_port_jumbo_config,
+ .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
+ .port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6320_g1_stats_snapshot,
+ .stats_get_sset_count = mv88e6095_stats_get_sset_count,
+ .stats_get_strings = mv88e6095_stats_get_strings,
+ .stats_get_stats = mv88e6095_stats_get_stats,
+ .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6095_g1_set_egress_port,
+ .watchdog_ops = &mv88e6097_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6351_ops = {
+ /* MV88E6XXX_FAMILY_6351 */
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
+ .port_set_speed = mv88e6185_port_set_speed,
+ .port_tag_remap = mv88e6095_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_jumbo_config = mv88e6165_port_jumbo_config,
+ .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
+ .port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6320_g1_stats_snapshot,
+ .stats_get_sset_count = mv88e6095_stats_get_sset_count,
+ .stats_get_strings = mv88e6095_stats_get_strings,
+ .stats_get_stats = mv88e6095_stats_get_stats,
+ .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6095_g1_set_egress_port,
+ .watchdog_ops = &mv88e6097_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
+};
+
+static const struct mv88e6xxx_ops mv88e6352_ops = {
+ /* MV88E6XXX_FAMILY_6352 */
+ .get_eeprom = mv88e6xxx_g2_get_eeprom16,
+ .set_eeprom = mv88e6xxx_g2_set_eeprom16,
+ .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
+ .phy_read = mv88e6xxx_g2_smi_phy_read,
+ .phy_write = mv88e6xxx_g2_smi_phy_write,
+ .port_set_link = mv88e6xxx_port_set_link,
+ .port_set_duplex = mv88e6xxx_port_set_duplex,
+ .port_set_rgmii_delay = mv88e6352_port_set_rgmii_delay,
+ .port_set_speed = mv88e6352_port_set_speed,
+ .port_tag_remap = mv88e6095_port_tag_remap,
+ .port_set_frame_mode = mv88e6351_port_set_frame_mode,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
+ .port_set_ether_type = mv88e6351_port_set_ether_type,
+ .port_jumbo_config = mv88e6165_port_jumbo_config,
+ .port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
+ .port_pause_config = mv88e6097_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
+ .stats_snapshot = mv88e6320_g1_stats_snapshot,
+ .stats_get_sset_count = mv88e6095_stats_get_sset_count,
+ .stats_get_strings = mv88e6095_stats_get_strings,
+ .stats_get_stats = mv88e6095_stats_get_stats,
+ .g1_set_cpu_port = mv88e6095_g1_set_cpu_port,
+ .g1_set_egress_port = mv88e6095_g1_set_egress_port,
+ .watchdog_ops = &mv88e6097_watchdog_ops,
+ .mgmt_rsvd2cpu = mv88e6095_g2_mgmt_rsvd2cpu,
+ .reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6352_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
};
static const struct mv88e6xxx_ops mv88e6390_ops = {
@@ -3746,12 +3334,14 @@
.port_set_speed = mv88e6390_port_set_speed,
.port_tag_remap = mv88e6390_port_tag_remap,
.port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
.port_set_ether_type = mv88e6351_port_set_ether_type,
.port_jumbo_config = mv88e6165_port_jumbo_config,
.port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
.port_pause_config = mv88e6390_port_pause_config,
.port_set_cmode = mv88e6390x_port_set_cmode,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
.stats_snapshot = mv88e6390_g1_stats_snapshot,
.stats_set_histogram = mv88e6390_g1_stats_set_histogram,
.stats_get_sset_count = mv88e6320_stats_get_sset_count,
@@ -3762,6 +3352,8 @@
.watchdog_ops = &mv88e6390_watchdog_ops,
.mgmt_rsvd2cpu = mv88e6390_g1_mgmt_rsvd2cpu,
.reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6390_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6390_g1_vtu_loadpurge,
};
static const struct mv88e6xxx_ops mv88e6390x_ops = {
@@ -3777,11 +3369,13 @@
.port_set_speed = mv88e6390x_port_set_speed,
.port_tag_remap = mv88e6390_port_tag_remap,
.port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
+ .port_set_egress_floods = mv88e6352_port_set_egress_floods,
.port_set_ether_type = mv88e6351_port_set_ether_type,
.port_jumbo_config = mv88e6165_port_jumbo_config,
.port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
.port_pause_config = mv88e6390_port_pause_config,
+ .port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
+ .port_disable_pri_override = mv88e6xxx_port_disable_pri_override,
.stats_snapshot = mv88e6390_g1_stats_snapshot,
.stats_set_histogram = mv88e6390_g1_stats_set_histogram,
.stats_get_sset_count = mv88e6320_stats_get_sset_count,
@@ -3792,52 +3386,10 @@
.watchdog_ops = &mv88e6390_watchdog_ops,
.mgmt_rsvd2cpu = mv88e6390_g1_mgmt_rsvd2cpu,
.reset = mv88e6352_g1_reset,
+ .vtu_getnext = mv88e6390_g1_vtu_getnext,
+ .vtu_loadpurge = mv88e6390_g1_vtu_loadpurge,
};
-static const struct mv88e6xxx_ops mv88e6391_ops = {
- /* MV88E6XXX_FAMILY_6390 */
- .get_eeprom = mv88e6xxx_g2_get_eeprom8,
- .set_eeprom = mv88e6xxx_g2_set_eeprom8,
- .set_switch_mac = mv88e6xxx_g2_set_switch_mac,
- .phy_read = mv88e6xxx_g2_smi_phy_read,
- .phy_write = mv88e6xxx_g2_smi_phy_write,
- .port_set_link = mv88e6xxx_port_set_link,
- .port_set_duplex = mv88e6xxx_port_set_duplex,
- .port_set_rgmii_delay = mv88e6390_port_set_rgmii_delay,
- .port_set_speed = mv88e6390_port_set_speed,
- .port_tag_remap = mv88e6390_port_tag_remap,
- .port_set_frame_mode = mv88e6351_port_set_frame_mode,
- .port_set_egress_unknowns = mv88e6351_port_set_egress_unknowns,
- .port_set_ether_type = mv88e6351_port_set_ether_type,
- .port_pause_config = mv88e6390_port_pause_config,
- .stats_snapshot = mv88e6390_g1_stats_snapshot,
- .stats_set_histogram = mv88e6390_g1_stats_set_histogram,
- .stats_get_sset_count = mv88e6320_stats_get_sset_count,
- .stats_get_strings = mv88e6320_stats_get_strings,
- .stats_get_stats = mv88e6390_stats_get_stats,
- .g1_set_cpu_port = mv88e6390_g1_set_cpu_port,
- .g1_set_egress_port = mv88e6390_g1_set_egress_port,
- .watchdog_ops = &mv88e6390_watchdog_ops,
- .mgmt_rsvd2cpu = mv88e6390_g1_mgmt_rsvd2cpu,
- .reset = mv88e6352_g1_reset,
-};
-
-static int mv88e6xxx_verify_madatory_ops(struct mv88e6xxx_chip *chip,
- const struct mv88e6xxx_ops *ops)
-{
- if (!ops->port_set_frame_mode) {
- dev_err(chip->dev, "Missing port_set_frame_mode");
- return -EINVAL;
- }
-
- if (!ops->port_set_egress_unknowns) {
- dev_err(chip->dev, "Missing port_set_egress_mode");
- return -EINVAL;
- }
-
- return 0;
-}
-
static const struct mv88e6xxx_info mv88e6xxx_table[] = {
[MV88E6085] = {
.prod_num = PORT_SWITCH_ID_PROD_NUM_6085,
@@ -3845,10 +3397,13 @@
.name = "Marvell 88E6085",
.num_databases = 4096,
.num_ports = 10,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 8,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_DSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6097,
.ops = &mv88e6085_ops,
@@ -3860,10 +3415,12 @@
.name = "Marvell 88E6095/88E6095F",
.num_databases = 256,
.num_ports = 11,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 8,
+ .atu_move_port_mask = 0xf,
.tag_protocol = DSA_TAG_PROTO_DSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6095,
.ops = &mv88e6095_ops,
@@ -3875,10 +3432,13 @@
.name = "Marvell 88E6097/88E6097F",
.num_databases = 4096,
.num_ports = 11,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 8,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_EDSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6097,
.ops = &mv88e6097_ops,
@@ -3890,10 +3450,13 @@
.name = "Marvell 88E6123",
.num_databases = 4096,
.num_ports = 3,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 9,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_DSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6165,
.ops = &mv88e6123_ops,
@@ -3905,25 +3468,47 @@
.name = "Marvell 88E6131",
.num_databases = 256,
.num_ports = 8,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 9,
+ .atu_move_port_mask = 0xf,
.tag_protocol = DSA_TAG_PROTO_DSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6185,
.ops = &mv88e6131_ops,
},
+ [MV88E6141] = {
+ .prod_num = PORT_SWITCH_ID_PROD_NUM_6141,
+ .family = MV88E6XXX_FAMILY_6341,
+ .name = "Marvell 88E6341",
+ .num_databases = 4096,
+ .num_ports = 6,
+ .max_vid = 4095,
+ .port_base_addr = 0x10,
+ .global1_addr = 0x1b,
+ .age_time_coeff = 3750,
+ .atu_move_port_mask = 0x1f,
+ .pvt = true,
+ .tag_protocol = DSA_TAG_PROTO_EDSA,
+ .flags = MV88E6XXX_FLAGS_FAMILY_6341,
+ .ops = &mv88e6141_ops,
+ },
+
[MV88E6161] = {
.prod_num = PORT_SWITCH_ID_PROD_NUM_6161,
.family = MV88E6XXX_FAMILY_6165,
.name = "Marvell 88E6161",
.num_databases = 4096,
.num_ports = 6,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 9,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_DSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6165,
.ops = &mv88e6161_ops,
@@ -3935,10 +3520,13 @@
.name = "Marvell 88E6165",
.num_databases = 4096,
.num_ports = 6,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 9,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_DSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6165,
.ops = &mv88e6165_ops,
@@ -3950,10 +3538,13 @@
.name = "Marvell 88E6171",
.num_databases = 4096,
.num_ports = 7,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 9,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_EDSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6351,
.ops = &mv88e6171_ops,
@@ -3965,10 +3556,13 @@
.name = "Marvell 88E6172",
.num_databases = 4096,
.num_ports = 7,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 9,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_EDSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6352,
.ops = &mv88e6172_ops,
@@ -3980,10 +3574,13 @@
.name = "Marvell 88E6175",
.num_databases = 4096,
.num_ports = 7,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 9,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_EDSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6351,
.ops = &mv88e6175_ops,
@@ -3995,10 +3592,13 @@
.name = "Marvell 88E6176",
.num_databases = 4096,
.num_ports = 7,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 9,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_EDSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6352,
.ops = &mv88e6176_ops,
@@ -4010,10 +3610,12 @@
.name = "Marvell 88E6185",
.num_databases = 256,
.num_ports = 10,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 8,
+ .atu_move_port_mask = 0xf,
.tag_protocol = DSA_TAG_PROTO_EDSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6185,
.ops = &mv88e6185_ops,
@@ -4025,11 +3627,14 @@
.name = "Marvell 88E6190",
.num_databases = 4096,
.num_ports = 11, /* 10 + Z80 */
+ .max_vid = 8191,
.port_base_addr = 0x0,
.global1_addr = 0x1b,
.tag_protocol = DSA_TAG_PROTO_DSA,
.age_time_coeff = 3750,
.g1_irqs = 9,
+ .pvt = true,
+ .atu_move_port_mask = 0x1f,
.flags = MV88E6XXX_FLAGS_FAMILY_6390,
.ops = &mv88e6190_ops,
},
@@ -4040,10 +3645,13 @@
.name = "Marvell 88E6190X",
.num_databases = 4096,
.num_ports = 11, /* 10 + Z80 */
+ .max_vid = 8191,
.port_base_addr = 0x0,
.global1_addr = 0x1b,
.age_time_coeff = 3750,
.g1_irqs = 9,
+ .atu_move_port_mask = 0x1f,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_DSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6390,
.ops = &mv88e6190x_ops,
@@ -4055,13 +3663,16 @@
.name = "Marvell 88E6191",
.num_databases = 4096,
.num_ports = 11, /* 10 + Z80 */
+ .max_vid = 8191,
.port_base_addr = 0x0,
.global1_addr = 0x1b,
.age_time_coeff = 3750,
.g1_irqs = 9,
+ .atu_move_port_mask = 0x1f,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_DSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6390,
- .ops = &mv88e6391_ops,
+ .ops = &mv88e6191_ops,
},
[MV88E6240] = {
@@ -4070,10 +3681,13 @@
.name = "Marvell 88E6240",
.num_databases = 4096,
.num_ports = 7,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 9,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_EDSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6352,
.ops = &mv88e6240_ops,
@@ -4085,10 +3699,13 @@
.name = "Marvell 88E6290",
.num_databases = 4096,
.num_ports = 11, /* 10 + Z80 */
+ .max_vid = 8191,
.port_base_addr = 0x0,
.global1_addr = 0x1b,
.age_time_coeff = 3750,
.g1_irqs = 9,
+ .atu_move_port_mask = 0x1f,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_DSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6390,
.ops = &mv88e6290_ops,
@@ -4100,10 +3717,13 @@
.name = "Marvell 88E6320",
.num_databases = 4096,
.num_ports = 7,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 8,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_EDSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6320,
.ops = &mv88e6320_ops,
@@ -4115,38 +3735,29 @@
.name = "Marvell 88E6321",
.num_databases = 4096,
.num_ports = 7,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 8,
+ .atu_move_port_mask = 0xf,
.tag_protocol = DSA_TAG_PROTO_EDSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6320,
.ops = &mv88e6321_ops,
},
- [MV88E6141] = {
- .prod_num = PORT_SWITCH_ID_PROD_NUM_6141,
- .family = MV88E6XXX_FAMILY_6341,
- .name = "Marvell 88E6341",
- .num_databases = 4096,
- .num_ports = 6,
- .port_base_addr = 0x10,
- .global1_addr = 0x1b,
- .age_time_coeff = 3750,
- .tag_protocol = DSA_TAG_PROTO_EDSA,
- .flags = MV88E6XXX_FLAGS_FAMILY_6341,
- .ops = &mv88e6141_ops,
- },
-
[MV88E6341] = {
.prod_num = PORT_SWITCH_ID_PROD_NUM_6341,
.family = MV88E6XXX_FAMILY_6341,
.name = "Marvell 88E6341",
.num_databases = 4096,
.num_ports = 6,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 3750,
+ .atu_move_port_mask = 0x1f,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_EDSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6341,
.ops = &mv88e6341_ops,
@@ -4158,10 +3769,13 @@
.name = "Marvell 88E6350",
.num_databases = 4096,
.num_ports = 7,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 9,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_EDSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6351,
.ops = &mv88e6350_ops,
@@ -4173,10 +3787,13 @@
.name = "Marvell 88E6351",
.num_databases = 4096,
.num_ports = 7,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 9,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_EDSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6351,
.ops = &mv88e6351_ops,
@@ -4188,10 +3805,13 @@
.name = "Marvell 88E6352",
.num_databases = 4096,
.num_ports = 7,
+ .max_vid = 4095,
.port_base_addr = 0x10,
.global1_addr = 0x1b,
.age_time_coeff = 15000,
.g1_irqs = 9,
+ .atu_move_port_mask = 0xf,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_EDSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6352,
.ops = &mv88e6352_ops,
@@ -4202,10 +3822,13 @@
.name = "Marvell 88E6390",
.num_databases = 4096,
.num_ports = 11, /* 10 + Z80 */
+ .max_vid = 8191,
.port_base_addr = 0x0,
.global1_addr = 0x1b,
.age_time_coeff = 3750,
.g1_irqs = 9,
+ .atu_move_port_mask = 0x1f,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_DSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6390,
.ops = &mv88e6390_ops,
@@ -4216,10 +3839,13 @@
.name = "Marvell 88E6390X",
.num_databases = 4096,
.num_ports = 11, /* 10 + Z80 */
+ .max_vid = 8191,
.port_base_addr = 0x0,
.global1_addr = 0x1b,
.age_time_coeff = 3750,
.g1_irqs = 9,
+ .atu_move_port_mask = 0x1f,
+ .pvt = true,
.tag_protocol = DSA_TAG_PROTO_DSA,
.flags = MV88E6XXX_FLAGS_FAMILY_6390,
.ops = &mv88e6390x_ops,
@@ -4455,6 +4081,8 @@
.port_mdb_add = mv88e6xxx_port_mdb_add,
.port_mdb_del = mv88e6xxx_port_mdb_del,
.port_mdb_dump = mv88e6xxx_port_mdb_dump,
+ .crosschip_bridge_join = mv88e6xxx_crosschip_bridge_join,
+ .crosschip_bridge_leave = mv88e6xxx_crosschip_bridge_leave,
};
static struct dsa_switch_driver mv88e6xxx_switch_drv = {
@@ -4466,12 +4094,14 @@
struct device *dev = chip->dev;
struct dsa_switch *ds;
- ds = dsa_switch_alloc(dev, DSA_MAX_PORTS);
+ ds = dsa_switch_alloc(dev, mv88e6xxx_num_ports(chip));
if (!ds)
return -ENOMEM;
ds->priv = chip;
ds->ops = &mv88e6xxx_switch_ops;
+ ds->ageing_time_min = chip->info->age_time_coeff;
+ ds->ageing_time_max = chip->info->age_time_coeff * U8_MAX;
dev_set_drvdata(dev, ds);
@@ -4502,10 +4132,6 @@
chip->info = compat_info;
- err = mv88e6xxx_verify_madatory_ops(chip, chip->info->ops);
- if (err)
- return err;
-
err = mv88e6xxx_smi_init(chip, mdiodev->bus, mdiodev->addr);
if (err)
return err;
diff --git a/drivers/net/dsa/mv88e6xxx/global1.c b/drivers/net/dsa/mv88e6xxx/global1.c
index 75af86a..3982583 100644
--- a/drivers/net/dsa/mv88e6xxx/global1.c
+++ b/drivers/net/dsa/mv88e6xxx/global1.c
@@ -3,7 +3,8 @@
*
* Copyright (c) 2008 Marvell Semiconductor
*
- * Copyright (c) 2016 Vivien Didelot <vivien.didelot@savoirfairelinux.com>
+ * Copyright (c) 2016-2017 Savoir-faire Linux Inc.
+ * Vivien Didelot <vivien.didelot@savoirfairelinux.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
diff --git a/drivers/net/dsa/mv88e6xxx/global1.h b/drivers/net/dsa/mv88e6xxx/global1.h
index 1aec738..46a4ea0 100644
--- a/drivers/net/dsa/mv88e6xxx/global1.h
+++ b/drivers/net/dsa/mv88e6xxx/global1.h
@@ -3,7 +3,8 @@
*
* Copyright (c) 2008 Marvell Semiconductor
*
- * Copyright (c) 2016 Vivien Didelot <vivien.didelot@savoirfairelinux.com>
+ * Copyright (c) 2016-2017 Savoir-faire Linux Inc.
+ * Vivien Didelot <vivien.didelot@savoirfairelinux.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -38,4 +39,29 @@
int mv88e6390_g1_set_cpu_port(struct mv88e6xxx_chip *chip, int port);
int mv88e6390_g1_mgmt_rsvd2cpu(struct mv88e6xxx_chip *chip);
+int mv88e6xxx_g1_atu_set_learn2all(struct mv88e6xxx_chip *chip, bool learn2all);
+int mv88e6xxx_g1_atu_set_age_time(struct mv88e6xxx_chip *chip,
+ unsigned int msecs);
+int mv88e6xxx_g1_atu_getnext(struct mv88e6xxx_chip *chip, u16 fid,
+ struct mv88e6xxx_atu_entry *entry);
+int mv88e6xxx_g1_atu_loadpurge(struct mv88e6xxx_chip *chip, u16 fid,
+ struct mv88e6xxx_atu_entry *entry);
+int mv88e6xxx_g1_atu_flush(struct mv88e6xxx_chip *chip, u16 fid, bool all);
+int mv88e6xxx_g1_atu_remove(struct mv88e6xxx_chip *chip, u16 fid, int port,
+ bool all);
+
+int mv88e6185_g1_vtu_getnext(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry);
+int mv88e6185_g1_vtu_loadpurge(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry);
+int mv88e6352_g1_vtu_getnext(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry);
+int mv88e6352_g1_vtu_loadpurge(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry);
+int mv88e6390_g1_vtu_getnext(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry);
+int mv88e6390_g1_vtu_loadpurge(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry);
+int mv88e6xxx_g1_vtu_flush(struct mv88e6xxx_chip *chip);
+
#endif /* _MV88E6XXX_GLOBAL1_H */
diff --git a/drivers/net/dsa/mv88e6xxx/global1_atu.c b/drivers/net/dsa/mv88e6xxx/global1_atu.c
new file mode 100644
index 0000000..fa7e7db
--- /dev/null
+++ b/drivers/net/dsa/mv88e6xxx/global1_atu.c
@@ -0,0 +1,305 @@
+/*
+ * Marvell 88E6xxx Address Translation Unit (ATU) support
+ *
+ * Copyright (c) 2008 Marvell Semiconductor
+ * Copyright (c) 2017 Savoir-faire Linux, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include "mv88e6xxx.h"
+#include "global1.h"
+
+/* Offset 0x01: ATU FID Register */
+
+static int mv88e6xxx_g1_atu_fid_write(struct mv88e6xxx_chip *chip, u16 fid)
+{
+ return mv88e6xxx_g1_write(chip, GLOBAL_ATU_FID, fid & 0xfff);
+}
+
+/* Offset 0x0A: ATU Control Register */
+
+int mv88e6xxx_g1_atu_set_learn2all(struct mv88e6xxx_chip *chip, bool learn2all)
+{
+ u16 val;
+ int err;
+
+ err = mv88e6xxx_g1_read(chip, GLOBAL_ATU_CONTROL, &val);
+ if (err)
+ return err;
+
+ if (learn2all)
+ val |= GLOBAL_ATU_CONTROL_LEARN2ALL;
+ else
+ val &= ~GLOBAL_ATU_CONTROL_LEARN2ALL;
+
+ return mv88e6xxx_g1_write(chip, GLOBAL_ATU_CONTROL, val);
+}
+
+int mv88e6xxx_g1_atu_set_age_time(struct mv88e6xxx_chip *chip,
+ unsigned int msecs)
+{
+ const unsigned int coeff = chip->info->age_time_coeff;
+ const unsigned int min = 0x01 * coeff;
+ const unsigned int max = 0xff * coeff;
+ u8 age_time;
+ u16 val;
+ int err;
+
+ if (msecs < min || msecs > max)
+ return -ERANGE;
+
+ /* Round to nearest multiple of coeff */
+ age_time = (msecs + coeff / 2) / coeff;
+
+ err = mv88e6xxx_g1_read(chip, GLOBAL_ATU_CONTROL, &val);
+ if (err)
+ return err;
+
+ /* AgeTime is 11:4 bits */
+ val &= ~0xff0;
+ val |= age_time << 4;
+
+ err = mv88e6xxx_g1_write(chip, GLOBAL_ATU_CONTROL, val);
+ if (err)
+ return err;
+
+ dev_dbg(chip->dev, "AgeTime set to 0x%02x (%d ms)\n", age_time,
+ age_time * coeff);
+
+ return 0;
+}
+
+/* Offset 0x0B: ATU Operation Register */
+
+static int mv88e6xxx_g1_atu_op_wait(struct mv88e6xxx_chip *chip)
+{
+ return mv88e6xxx_g1_wait(chip, GLOBAL_ATU_OP, GLOBAL_ATU_OP_BUSY);
+}
+
+static int mv88e6xxx_g1_atu_op(struct mv88e6xxx_chip *chip, u16 fid, u16 op)
+{
+ u16 val;
+ int err;
+
+ /* FID bits are dispatched all around gradually as more are supported */
+ if (mv88e6xxx_num_databases(chip) > 256) {
+ err = mv88e6xxx_g1_atu_fid_write(chip, fid);
+ if (err)
+ return err;
+ } else {
+ if (mv88e6xxx_num_databases(chip) > 16) {
+ /* ATU DBNum[7:4] are located in ATU Control 15:12 */
+ err = mv88e6xxx_g1_read(chip, GLOBAL_ATU_CONTROL, &val);
+ if (err)
+ return err;
+
+ val = (val & 0x0fff) | ((fid << 8) & 0xf000);
+ err = mv88e6xxx_g1_write(chip, GLOBAL_ATU_CONTROL, val);
+ if (err)
+ return err;
+ }
+
+ /* ATU DBNum[3:0] are located in ATU Operation 3:0 */
+ op |= fid & 0xf;
+ }
+
+ err = mv88e6xxx_g1_write(chip, GLOBAL_ATU_OP, op);
+ if (err)
+ return err;
+
+ return mv88e6xxx_g1_atu_op_wait(chip);
+}
+
+/* Offset 0x0C: ATU Data Register */
+
+static int mv88e6xxx_g1_atu_data_read(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_atu_entry *entry)
+{
+ u16 val;
+ int err;
+
+ err = mv88e6xxx_g1_read(chip, GLOBAL_ATU_DATA, &val);
+ if (err)
+ return err;
+
+ entry->state = val & 0xf;
+ if (entry->state != GLOBAL_ATU_DATA_STATE_UNUSED) {
+ entry->trunk = !!(val & GLOBAL_ATU_DATA_TRUNK);
+ entry->portvec = (val >> 4) & mv88e6xxx_port_mask(chip);
+ }
+
+ return 0;
+}
+
+static int mv88e6xxx_g1_atu_data_write(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_atu_entry *entry)
+{
+ u16 data = entry->state & 0xf;
+
+ if (entry->state != GLOBAL_ATU_DATA_STATE_UNUSED) {
+ if (entry->trunk)
+ data |= GLOBAL_ATU_DATA_TRUNK;
+
+ data |= (entry->portvec & mv88e6xxx_port_mask(chip)) << 4;
+ }
+
+ return mv88e6xxx_g1_write(chip, GLOBAL_ATU_DATA, data);
+}
+
+/* Offset 0x0D: ATU MAC Address Register Bytes 0 & 1
+ * Offset 0x0E: ATU MAC Address Register Bytes 2 & 3
+ * Offset 0x0F: ATU MAC Address Register Bytes 4 & 5
+ */
+
+static int mv88e6xxx_g1_atu_mac_read(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_atu_entry *entry)
+{
+ u16 val;
+ int i, err;
+
+ for (i = 0; i < 3; i++) {
+ err = mv88e6xxx_g1_read(chip, GLOBAL_ATU_MAC_01 + i, &val);
+ if (err)
+ return err;
+
+ entry->mac[i * 2] = val >> 8;
+ entry->mac[i * 2 + 1] = val & 0xff;
+ }
+
+ return 0;
+}
+
+static int mv88e6xxx_g1_atu_mac_write(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_atu_entry *entry)
+{
+ u16 val;
+ int i, err;
+
+ for (i = 0; i < 3; i++) {
+ val = (entry->mac[i * 2] << 8) | entry->mac[i * 2 + 1];
+ err = mv88e6xxx_g1_write(chip, GLOBAL_ATU_MAC_01 + i, val);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+/* Address Translation Unit operations */
+
+int mv88e6xxx_g1_atu_getnext(struct mv88e6xxx_chip *chip, u16 fid,
+ struct mv88e6xxx_atu_entry *entry)
+{
+ int err;
+
+ err = mv88e6xxx_g1_atu_op_wait(chip);
+ if (err)
+ return err;
+
+ /* Write the MAC address to iterate from only once */
+ if (entry->state == GLOBAL_ATU_DATA_STATE_UNUSED) {
+ err = mv88e6xxx_g1_atu_mac_write(chip, entry);
+ if (err)
+ return err;
+ }
+
+ err = mv88e6xxx_g1_atu_op(chip, fid, GLOBAL_ATU_OP_GET_NEXT_DB);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_atu_data_read(chip, entry);
+ if (err)
+ return err;
+
+ return mv88e6xxx_g1_atu_mac_read(chip, entry);
+}
+
+int mv88e6xxx_g1_atu_loadpurge(struct mv88e6xxx_chip *chip, u16 fid,
+ struct mv88e6xxx_atu_entry *entry)
+{
+ int err;
+
+ err = mv88e6xxx_g1_atu_op_wait(chip);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_atu_mac_write(chip, entry);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_atu_data_write(chip, entry);
+ if (err)
+ return err;
+
+ return mv88e6xxx_g1_atu_op(chip, fid, GLOBAL_ATU_OP_LOAD_DB);
+}
+
+static int mv88e6xxx_g1_atu_flushmove(struct mv88e6xxx_chip *chip, u16 fid,
+ struct mv88e6xxx_atu_entry *entry,
+ bool all)
+{
+ u16 op;
+ int err;
+
+ err = mv88e6xxx_g1_atu_op_wait(chip);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_atu_data_write(chip, entry);
+ if (err)
+ return err;
+
+ /* Flush/Move all or non-static entries from all or a given database */
+ if (all && fid)
+ op = GLOBAL_ATU_OP_FLUSH_MOVE_ALL_DB;
+ else if (fid)
+ op = GLOBAL_ATU_OP_FLUSH_MOVE_NON_STATIC_DB;
+ else if (all)
+ op = GLOBAL_ATU_OP_FLUSH_MOVE_ALL;
+ else
+ op = GLOBAL_ATU_OP_FLUSH_MOVE_NON_STATIC;
+
+ return mv88e6xxx_g1_atu_op(chip, fid, op);
+}
+
+int mv88e6xxx_g1_atu_flush(struct mv88e6xxx_chip *chip, u16 fid, bool all)
+{
+ struct mv88e6xxx_atu_entry entry = {
+ .state = 0, /* Null EntryState means Flush */
+ };
+
+ return mv88e6xxx_g1_atu_flushmove(chip, fid, &entry, all);
+}
+
+static int mv88e6xxx_g1_atu_move(struct mv88e6xxx_chip *chip, u16 fid,
+ int from_port, int to_port, bool all)
+{
+ struct mv88e6xxx_atu_entry entry = { 0 };
+ unsigned long mask;
+ int shift;
+
+ if (!chip->info->atu_move_port_mask)
+ return -EOPNOTSUPP;
+
+ mask = chip->info->atu_move_port_mask;
+ shift = bitmap_weight(&mask, 16);
+
+ entry.state = 0xf, /* Full EntryState means Move */
+ entry.portvec = from_port & mask;
+ entry.portvec |= (to_port & mask) << shift;
+
+ return mv88e6xxx_g1_atu_flushmove(chip, fid, &entry, all);
+}
+
+int mv88e6xxx_g1_atu_remove(struct mv88e6xxx_chip *chip, u16 fid, int port,
+ bool all)
+{
+ int from_port = port;
+ int to_port = chip->info->atu_move_port_mask;
+
+ return mv88e6xxx_g1_atu_move(chip, fid, from_port, to_port, all);
+}
diff --git a/drivers/net/dsa/mv88e6xxx/global1_vtu.c b/drivers/net/dsa/mv88e6xxx/global1_vtu.c
new file mode 100644
index 0000000..9aea22d
--- /dev/null
+++ b/drivers/net/dsa/mv88e6xxx/global1_vtu.c
@@ -0,0 +1,511 @@
+/*
+ * Marvell 88E6xxx VLAN [Spanning Tree] Translation Unit (VTU [STU]) support
+ *
+ * Copyright (c) 2008 Marvell Semiconductor
+ * Copyright (c) 2015 CMC Electronics, Inc.
+ * Copyright (c) 2017 Savoir-faire Linux, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include "mv88e6xxx.h"
+#include "global1.h"
+
+/* Offset 0x02: VTU FID Register */
+
+static int mv88e6xxx_g1_vtu_fid_read(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ u16 val;
+ int err;
+
+ err = mv88e6xxx_g1_read(chip, GLOBAL_VTU_FID, &val);
+ if (err)
+ return err;
+
+ entry->fid = val & GLOBAL_VTU_FID_MASK;
+
+ return 0;
+}
+
+static int mv88e6xxx_g1_vtu_fid_write(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ u16 val = entry->fid & GLOBAL_VTU_FID_MASK;
+
+ return mv88e6xxx_g1_write(chip, GLOBAL_VTU_FID, val);
+}
+
+/* Offset 0x03: VTU SID Register */
+
+static int mv88e6xxx_g1_vtu_sid_read(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ u16 val;
+ int err;
+
+ err = mv88e6xxx_g1_read(chip, GLOBAL_VTU_SID, &val);
+ if (err)
+ return err;
+
+ entry->sid = val & GLOBAL_VTU_SID_MASK;
+
+ return 0;
+}
+
+static int mv88e6xxx_g1_vtu_sid_write(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ u16 val = entry->sid & GLOBAL_VTU_SID_MASK;
+
+ return mv88e6xxx_g1_write(chip, GLOBAL_VTU_SID, val);
+}
+
+/* Offset 0x05: VTU Operation Register */
+
+static int mv88e6xxx_g1_vtu_op_wait(struct mv88e6xxx_chip *chip)
+{
+ return mv88e6xxx_g1_wait(chip, GLOBAL_VTU_OP, GLOBAL_VTU_OP_BUSY);
+}
+
+static int mv88e6xxx_g1_vtu_op(struct mv88e6xxx_chip *chip, u16 op)
+{
+ int err;
+
+ err = mv88e6xxx_g1_write(chip, GLOBAL_VTU_OP, op);
+ if (err)
+ return err;
+
+ return mv88e6xxx_g1_vtu_op_wait(chip);
+}
+
+/* Offset 0x06: VTU VID Register */
+
+static int mv88e6xxx_g1_vtu_vid_read(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ u16 val;
+ int err;
+
+ err = mv88e6xxx_g1_read(chip, GLOBAL_VTU_VID, &val);
+ if (err)
+ return err;
+
+ entry->vid = val & 0xfff;
+
+ if (val & GLOBAL_VTU_VID_PAGE)
+ entry->vid |= 0x1000;
+
+ entry->valid = !!(val & GLOBAL_VTU_VID_VALID);
+
+ return 0;
+}
+
+static int mv88e6xxx_g1_vtu_vid_write(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ u16 val = entry->vid & 0xfff;
+
+ if (entry->vid & 0x1000)
+ val |= GLOBAL_VTU_VID_PAGE;
+
+ if (entry->valid)
+ val |= GLOBAL_VTU_VID_VALID;
+
+ return mv88e6xxx_g1_write(chip, GLOBAL_VTU_VID, val);
+}
+
+/* Offset 0x07: VTU/STU Data Register 1
+ * Offset 0x08: VTU/STU Data Register 2
+ * Offset 0x09: VTU/STU Data Register 3
+ */
+
+static int mv88e6185_g1_vtu_data_read(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ u16 regs[3];
+ int i;
+
+ /* Read all 3 VTU/STU Data registers */
+ for (i = 0; i < 3; ++i) {
+ u16 *reg = ®s[i];
+ int err;
+
+ err = mv88e6xxx_g1_read(chip, GLOBAL_VTU_DATA_0_3 + i, reg);
+ if (err)
+ return err;
+ }
+
+ /* Extract MemberTag and PortState data */
+ for (i = 0; i < mv88e6xxx_num_ports(chip); ++i) {
+ unsigned int member_offset = (i % 4) * 4;
+ unsigned int state_offset = member_offset + 2;
+
+ entry->member[i] = (regs[i / 4] >> member_offset) & 0x3;
+ entry->state[i] = (regs[i / 4] >> state_offset) & 0x3;
+ }
+
+ return 0;
+}
+
+static int mv88e6185_g1_vtu_data_write(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ u16 regs[3] = { 0 };
+ int i;
+
+ /* Insert MemberTag and PortState data */
+ for (i = 0; i < mv88e6xxx_num_ports(chip); ++i) {
+ unsigned int member_offset = (i % 4) * 4;
+ unsigned int state_offset = member_offset + 2;
+
+ regs[i / 4] |= (entry->member[i] & 0x3) << member_offset;
+ regs[i / 4] |= (entry->state[i] & 0x3) << state_offset;
+ }
+
+ /* Write all 3 VTU/STU Data registers */
+ for (i = 0; i < 3; ++i) {
+ u16 reg = regs[i];
+ int err;
+
+ err = mv88e6xxx_g1_write(chip, GLOBAL_VTU_DATA_0_3 + i, reg);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+static int mv88e6390_g1_vtu_data_read(struct mv88e6xxx_chip *chip, u8 *data)
+{
+ u16 regs[2];
+ int i;
+
+ /* Read the 2 VTU/STU Data registers */
+ for (i = 0; i < 2; ++i) {
+ u16 *reg = ®s[i];
+ int err;
+
+ err = mv88e6xxx_g1_read(chip, GLOBAL_VTU_DATA_0_3 + i, reg);
+ if (err)
+ return err;
+ }
+
+ /* Extract data */
+ for (i = 0; i < mv88e6xxx_num_ports(chip); ++i) {
+ unsigned int offset = (i % 8) * 2;
+
+ data[i] = (regs[i / 8] >> offset) & 0x3;
+ }
+
+ return 0;
+}
+
+static int mv88e6390_g1_vtu_data_write(struct mv88e6xxx_chip *chip, u8 *data)
+{
+ u16 regs[2] = { 0 };
+ int i;
+
+ /* Insert data */
+ for (i = 0; i < mv88e6xxx_num_ports(chip); ++i) {
+ unsigned int offset = (i % 8) * 2;
+
+ regs[i / 8] |= (data[i] & 0x3) << offset;
+ }
+
+ /* Write the 2 VTU/STU Data registers */
+ for (i = 0; i < 2; ++i) {
+ u16 reg = regs[i];
+ int err;
+
+ err = mv88e6xxx_g1_write(chip, GLOBAL_VTU_DATA_0_3 + i, reg);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+/* VLAN Translation Unit Operations */
+
+static int mv88e6xxx_g1_vtu_stu_getnext(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ int err;
+
+ err = mv88e6xxx_g1_vtu_sid_write(chip, entry);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_vtu_op(chip, GLOBAL_VTU_OP_STU_GET_NEXT);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_vtu_sid_read(chip, entry);
+ if (err)
+ return err;
+
+ return mv88e6xxx_g1_vtu_vid_read(chip, entry);
+}
+
+static int mv88e6xxx_g1_vtu_stu_get(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *vtu)
+{
+ struct mv88e6xxx_vtu_entry stu;
+ int err;
+
+ err = mv88e6xxx_g1_vtu_sid_read(chip, vtu);
+ if (err)
+ return err;
+
+ stu.sid = vtu->sid - 1;
+
+ err = mv88e6xxx_g1_vtu_stu_getnext(chip, &stu);
+ if (err)
+ return err;
+
+ if (stu.sid != vtu->sid || !stu.valid)
+ return -EINVAL;
+
+ return 0;
+}
+
+static int mv88e6xxx_g1_vtu_getnext(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ int err;
+
+ err = mv88e6xxx_g1_vtu_op_wait(chip);
+ if (err)
+ return err;
+
+ /* To get the next higher active VID, the VTU GetNext operation can be
+ * started again without setting the VID registers since it already
+ * contains the last VID.
+ *
+ * To save a few hardware accesses and abstract this to the caller,
+ * write the VID only once, when the entry is given as invalid.
+ */
+ if (!entry->valid) {
+ err = mv88e6xxx_g1_vtu_vid_write(chip, entry);
+ if (err)
+ return err;
+ }
+
+ err = mv88e6xxx_g1_vtu_op(chip, GLOBAL_VTU_OP_VTU_GET_NEXT);
+ if (err)
+ return err;
+
+ return mv88e6xxx_g1_vtu_vid_read(chip, entry);
+}
+
+int mv88e6185_g1_vtu_getnext(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ u16 val;
+ int err;
+
+ err = mv88e6xxx_g1_vtu_getnext(chip, entry);
+ if (err)
+ return err;
+
+ if (entry->valid) {
+ err = mv88e6185_g1_vtu_data_read(chip, entry);
+ if (err)
+ return err;
+
+ /* VTU DBNum[3:0] are located in VTU Operation 3:0
+ * VTU DBNum[7:4] are located in VTU Operation 11:8
+ */
+ err = mv88e6xxx_g1_read(chip, GLOBAL_VTU_OP, &val);
+ if (err)
+ return err;
+
+ entry->fid = val & 0x000f;
+ entry->fid |= (val & 0x0f00) >> 4;
+ }
+
+ return 0;
+}
+
+int mv88e6352_g1_vtu_getnext(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ int err;
+
+ /* Fetch VLAN MemberTag data from the VTU */
+ err = mv88e6xxx_g1_vtu_getnext(chip, entry);
+ if (err)
+ return err;
+
+ if (entry->valid) {
+ /* Fetch (and mask) VLAN PortState data from the STU */
+ err = mv88e6xxx_g1_vtu_stu_get(chip, entry);
+ if (err)
+ return err;
+
+ err = mv88e6185_g1_vtu_data_read(chip, entry);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_vtu_fid_read(chip, entry);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+int mv88e6390_g1_vtu_getnext(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ int err;
+
+ /* Fetch VLAN MemberTag data from the VTU */
+ err = mv88e6xxx_g1_vtu_getnext(chip, entry);
+ if (err)
+ return err;
+
+ if (entry->valid) {
+ err = mv88e6390_g1_vtu_data_read(chip, entry->member);
+ if (err)
+ return err;
+
+ /* Fetch VLAN PortState data from the STU */
+ err = mv88e6xxx_g1_vtu_stu_get(chip, entry);
+ if (err)
+ return err;
+
+ err = mv88e6390_g1_vtu_data_read(chip, entry->state);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_vtu_fid_read(chip, entry);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+int mv88e6185_g1_vtu_loadpurge(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ u16 op = GLOBAL_VTU_OP_VTU_LOAD_PURGE;
+ int err;
+
+ err = mv88e6xxx_g1_vtu_op_wait(chip);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_vtu_vid_write(chip, entry);
+ if (err)
+ return err;
+
+ if (entry->valid) {
+ err = mv88e6185_g1_vtu_data_write(chip, entry);
+ if (err)
+ return err;
+
+ /* VTU DBNum[3:0] are located in VTU Operation 3:0
+ * VTU DBNum[7:4] are located in VTU Operation 11:8
+ */
+ op |= entry->fid & 0x000f;
+ op |= (entry->fid & 0x00f0) << 8;
+ }
+
+ return mv88e6xxx_g1_vtu_op(chip, op);
+}
+
+int mv88e6352_g1_vtu_loadpurge(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ int err;
+
+ err = mv88e6xxx_g1_vtu_op_wait(chip);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_vtu_vid_write(chip, entry);
+ if (err)
+ return err;
+
+ if (entry->valid) {
+ /* Write MemberTag and PortState data */
+ err = mv88e6185_g1_vtu_data_write(chip, entry);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_vtu_sid_write(chip, entry);
+ if (err)
+ return err;
+
+ /* Load STU entry */
+ err = mv88e6xxx_g1_vtu_op(chip, GLOBAL_VTU_OP_STU_LOAD_PURGE);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_vtu_fid_write(chip, entry);
+ if (err)
+ return err;
+ }
+
+ /* Load/Purge VTU entry */
+ return mv88e6xxx_g1_vtu_op(chip, GLOBAL_VTU_OP_VTU_LOAD_PURGE);
+}
+
+int mv88e6390_g1_vtu_loadpurge(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry)
+{
+ int err;
+
+ err = mv88e6xxx_g1_vtu_op_wait(chip);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_vtu_vid_write(chip, entry);
+ if (err)
+ return err;
+
+ if (entry->valid) {
+ /* Write PortState data */
+ err = mv88e6390_g1_vtu_data_write(chip, entry->state);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_vtu_sid_write(chip, entry);
+ if (err)
+ return err;
+
+ /* Load STU entry */
+ err = mv88e6xxx_g1_vtu_op(chip, GLOBAL_VTU_OP_STU_LOAD_PURGE);
+ if (err)
+ return err;
+
+ /* Write MemberTag data */
+ err = mv88e6390_g1_vtu_data_write(chip, entry->member);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g1_vtu_fid_write(chip, entry);
+ if (err)
+ return err;
+ }
+
+ /* Load/Purge VTU entry */
+ return mv88e6xxx_g1_vtu_op(chip, GLOBAL_VTU_OP_VTU_LOAD_PURGE);
+}
+
+int mv88e6xxx_g1_vtu_flush(struct mv88e6xxx_chip *chip)
+{
+ int err;
+
+ err = mv88e6xxx_g1_vtu_op_wait(chip);
+ if (err)
+ return err;
+
+ return mv88e6xxx_g1_vtu_op(chip, GLOBAL_VTU_OP_FLUSH_ALL);
+}
diff --git a/drivers/net/dsa/mv88e6xxx/global2.c b/drivers/net/dsa/mv88e6xxx/global2.c
index 8f15bc7..b3fea55 100644
--- a/drivers/net/dsa/mv88e6xxx/global2.c
+++ b/drivers/net/dsa/mv88e6xxx/global2.c
@@ -4,7 +4,8 @@
*
* Copyright (c) 2008 Marvell Semiconductor
*
- * Copyright (c) 2016 Vivien Didelot <vivien.didelot@savoirfairelinux.com>
+ * Copyright (c) 2016-2017 Savoir-faire Linux Inc.
+ * Vivien Didelot <vivien.didelot@savoirfairelinux.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -12,6 +13,7 @@
* (at your option) any later version.
*/
+#include <linux/interrupt.h>
#include <linux/irqdomain.h>
#include "mv88e6xxx.h"
#include "global2.h"
@@ -170,6 +172,50 @@
return err;
}
+/* Offset 0x0B: Cross-chip Port VLAN (Addr) Register
+ * Offset 0x0C: Cross-chip Port VLAN Data Register
+ */
+
+static int mv88e6xxx_g2_pvt_op_wait(struct mv88e6xxx_chip *chip)
+{
+ return mv88e6xxx_g2_wait(chip, GLOBAL2_PVT_ADDR, GLOBAL2_PVT_ADDR_BUSY);
+}
+
+static int mv88e6xxx_g2_pvt_op(struct mv88e6xxx_chip *chip, int src_dev,
+ int src_port, u16 op)
+{
+ int err;
+
+ /* 9-bit Cross-chip PVT pointer: with GLOBAL2_MISC_5_BIT_PORT cleared,
+ * source device is 5-bit, source port is 4-bit.
+ */
+ op |= (src_dev & 0x1f) << 4;
+ op |= (src_port & 0xf);
+
+ err = mv88e6xxx_g2_write(chip, GLOBAL2_PVT_ADDR, op);
+ if (err)
+ return err;
+
+ return mv88e6xxx_g2_pvt_op_wait(chip);
+}
+
+int mv88e6xxx_g2_pvt_write(struct mv88e6xxx_chip *chip, int src_dev,
+ int src_port, u16 data)
+{
+ int err;
+
+ err = mv88e6xxx_g2_pvt_op_wait(chip);
+ if (err)
+ return err;
+
+ err = mv88e6xxx_g2_write(chip, GLOBAL2_PVT_DATA, data);
+ if (err)
+ return err;
+
+ return mv88e6xxx_g2_pvt_op(chip, src_dev, src_port,
+ GLOBAL2_PVT_ADDR_OP_WRITE_PVLAN);
+}
+
/* Offset 0x0D: Switch MAC/WoL/WoF register */
static int mv88e6xxx_g2_switch_mac_write(struct mv88e6xxx_chip *chip,
@@ -522,8 +568,9 @@
return mv88e6xxx_g2_smi_phy_cmd(chip, cmd);
}
-int mv88e6xxx_g2_smi_phy_read_c45(struct mv88e6xxx_chip *chip, int addr,
- int reg_c45, u16 *val, bool external)
+static int mv88e6xxx_g2_smi_phy_read_c45(struct mv88e6xxx_chip *chip,
+ int addr, int reg_c45, u16 *val,
+ bool external)
{
int device = (reg_c45 >> 16) & 0x1f;
int reg = reg_c45 & 0xffff;
@@ -553,8 +600,9 @@
return 0;
}
-int mv88e6xxx_g2_smi_phy_read_c22(struct mv88e6xxx_chip *chip, int addr,
- int reg, u16 *val, bool external)
+static int mv88e6xxx_g2_smi_phy_read_c22(struct mv88e6xxx_chip *chip,
+ int addr, int reg, u16 *val,
+ bool external)
{
u16 cmd = GLOBAL2_SMI_PHY_CMD_OP_22_READ_DATA | (addr << 5) | reg;
int err;
@@ -586,8 +634,9 @@
return mv88e6xxx_g2_smi_phy_read_c22(chip, addr, reg, val, external);
}
-int mv88e6xxx_g2_smi_phy_write_c45(struct mv88e6xxx_chip *chip, int addr,
- int reg_c45, u16 val, bool external)
+static int mv88e6xxx_g2_smi_phy_write_c45(struct mv88e6xxx_chip *chip,
+ int addr, int reg_c45, u16 val,
+ bool external)
{
int device = (reg_c45 >> 16) & 0x1f;
int reg = reg_c45 & 0xffff;
@@ -615,8 +664,9 @@
return 0;
}
-int mv88e6xxx_g2_smi_phy_write_c22(struct mv88e6xxx_chip *chip, int addr,
- int reg, u16 val, bool external)
+static int mv88e6xxx_g2_smi_phy_write_c22(struct mv88e6xxx_chip *chip,
+ int addr, int reg, u16 val,
+ bool external)
{
u16 cmd = GLOBAL2_SMI_PHY_CMD_OP_22_WRITE_DATA | (addr << 5) | reg;
int err;
@@ -782,6 +832,31 @@
return err;
}
+/* Offset 0x1D: Misc Register */
+
+static int mv88e6xxx_g2_misc_5_bit_port(struct mv88e6xxx_chip *chip,
+ bool port_5_bit)
+{
+ u16 val;
+ int err;
+
+ err = mv88e6xxx_g2_read(chip, GLOBAL2_MISC, &val);
+ if (err)
+ return err;
+
+ if (port_5_bit)
+ val |= GLOBAL2_MISC_5_BIT_PORT;
+ else
+ val &= ~GLOBAL2_MISC_5_BIT_PORT;
+
+ return mv88e6xxx_g2_write(chip, GLOBAL2_MISC, val);
+}
+
+int mv88e6xxx_g2_misc_4_bit_port(struct mv88e6xxx_chip *chip)
+{
+ return mv88e6xxx_g2_misc_5_bit_port(chip, false);
+}
+
static void mv88e6xxx_g2_irq_mask(struct irq_data *d)
{
struct mv88e6xxx_chip *chip = irq_data_get_irq_chip_data(d);
@@ -964,14 +1039,6 @@
return err;
}
- if (mv88e6xxx_has(chip, MV88E6XXX_FLAGS_PVT)) {
- /* Initialize Cross-chip Port VLAN Table to reset defaults */
- err = mv88e6xxx_g2_write(chip, GLOBAL2_PVT_ADDR,
- GLOBAL2_PVT_ADDR_OP_INIT_ONES);
- if (err)
- return err;
- }
-
if (mv88e6xxx_has(chip, MV88E6XXX_FLAG_G2_POT)) {
/* Clear the priority override table. */
err = mv88e6xxx_g2_clear_pot(chip);
diff --git a/drivers/net/dsa/mv88e6xxx/global2.h b/drivers/net/dsa/mv88e6xxx/global2.h
index a8b2f94..96046bb 100644
--- a/drivers/net/dsa/mv88e6xxx/global2.h
+++ b/drivers/net/dsa/mv88e6xxx/global2.h
@@ -3,7 +3,8 @@
*
* Copyright (c) 2008 Marvell Semiconductor
*
- * Copyright (c) 2016 Vivien Didelot <vivien.didelot@savoirfairelinux.com>
+ * Copyright (c) 2016-2017 Savoir-faire Linux Inc.
+ * Vivien Didelot <vivien.didelot@savoirfairelinux.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -41,6 +42,10 @@
int mv88e6xxx_g2_set_eeprom16(struct mv88e6xxx_chip *chip,
struct ethtool_eeprom *eeprom, u8 *data);
+int mv88e6xxx_g2_pvt_write(struct mv88e6xxx_chip *chip, int src_dev,
+ int src_port, u16 data);
+int mv88e6xxx_g2_misc_4_bit_port(struct mv88e6xxx_chip *chip);
+
int mv88e6xxx_g2_setup(struct mv88e6xxx_chip *chip);
int mv88e6xxx_g2_irq_setup(struct mv88e6xxx_chip *chip);
void mv88e6xxx_g2_irq_free(struct mv88e6xxx_chip *chip);
@@ -109,6 +114,17 @@
return -EOPNOTSUPP;
}
+int mv88e6xxx_g2_pvt_write(struct mv88e6xxx_chip *chip, int src_dev,
+ int src_port, u16 data)
+{
+ return -EOPNOTSUPP;
+}
+
+int mv88e6xxx_g2_misc_4_bit_port(struct mv88e6xxx_chip *chip)
+{
+ return -EOPNOTSUPP;
+}
+
static inline int mv88e6xxx_g2_setup(struct mv88e6xxx_chip *chip)
{
return -EOPNOTSUPP;
diff --git a/drivers/net/dsa/mv88e6xxx/mv88e6xxx.h b/drivers/net/dsa/mv88e6xxx/mv88e6xxx.h
index 6033f2f..77236cd 100644
--- a/drivers/net/dsa/mv88e6xxx/mv88e6xxx.h
+++ b/drivers/net/dsa/mv88e6xxx/mv88e6xxx.h
@@ -16,6 +16,7 @@
#include <linux/irq.h>
#include <linux/gpio/consumer.h>
#include <linux/phy.h>
+#include <net/dsa.h>
#ifndef UINT64_MAX
#define UINT64_MAX (u64)(~((u64)0))
@@ -132,18 +133,19 @@
#define PORT_CONTROL_TAG_IF_BOTH BIT(6)
#define PORT_CONTROL_USE_IP BIT(5)
#define PORT_CONTROL_USE_TAG BIT(4)
-#define PORT_CONTROL_FORWARD_UNKNOWN_MC BIT(3)
#define PORT_CONTROL_FORWARD_UNKNOWN BIT(2)
-#define PORT_CONTROL_NOT_EGRESS_UNKNOWN_DA (0x0 << 2)
-#define PORT_CONTROL_NOT_EGRESS_UNKNOWN_MULTICAST_DA (0x1 << 2)
-#define PORT_CONTROL_NOT_EGRESS_UNKNOWN_UNITCAST_DA (0x2 << 2)
-#define PORT_CONTROL_EGRESS_ALL_UNKNOWN_DA (0x3 << 2)
+#define PORT_CONTROL_EGRESS_FLOODS_MASK (0x3 << 2)
+#define PORT_CONTROL_EGRESS_FLOODS_NO_UNKNOWN_DA (0x0 << 2)
+#define PORT_CONTROL_EGRESS_FLOODS_NO_UNKNOWN_MC_DA (0x1 << 2)
+#define PORT_CONTROL_EGRESS_FLOODS_NO_UNKNOWN_UC_DA (0x2 << 2)
+#define PORT_CONTROL_EGRESS_FLOODS_ALL_UNKNOWN_DA (0x3 << 2)
#define PORT_CONTROL_STATE_MASK 0x03
#define PORT_CONTROL_STATE_DISABLED 0x00
#define PORT_CONTROL_STATE_BLOCKING 0x01
#define PORT_CONTROL_STATE_LEARNING 0x02
#define PORT_CONTROL_STATE_FORWARDING 0x03
#define PORT_CONTROL_1 0x05
+#define PORT_CONTROL_1_MESSAGE_PORT BIT(15)
#define PORT_CONTROL_1_FID_11_4_MASK (0xff << 0)
#define PORT_BASE_VLAN 0x06
#define PORT_BASE_VLAN_FID_3_0_MASK (0xf << 12)
@@ -166,7 +168,6 @@
#define PORT_CONTROL_2_DISCARD_UNTAGGED BIT(8)
#define PORT_CONTROL_2_MAP_DA BIT(7)
#define PORT_CONTROL_2_DEFAULT_FORWARD BIT(6)
-#define PORT_CONTROL_2_FORWARD_UNKNOWN BIT(6)
#define PORT_CONTROL_2_EGRESS_MONITOR BIT(5)
#define PORT_CONTROL_2_INGRESS_MONITOR BIT(4)
#define PORT_CONTROL_2_UPSTREAM_MASK 0x0f
@@ -181,6 +182,7 @@
#define PORT_ATU_CONTROL 0x0c
#define PORT_PRI_OVERRIDE 0x0d
#define PORT_ETH_TYPE 0x0f
+#define PORT_ETH_TYPE_DEFAULT 0x9100
#define PORT_IN_DISCARD_LO 0x10
#define PORT_IN_DISCARD_HI 0x11
#define PORT_IN_FILTERED 0x12
@@ -247,6 +249,7 @@
#define GLOBAL_VTU_OP_STU_GET_NEXT ((0x06 << 12) | GLOBAL_VTU_OP_BUSY)
#define GLOBAL_VTU_VID 0x06
#define GLOBAL_VTU_VID_MASK 0xfff
+#define GLOBAL_VTU_VID_PAGE BIT(13)
#define GLOBAL_VTU_VID_VALID BIT(12)
#define GLOBAL_VTU_DATA_0_3 0x07
#define GLOBAL_VTU_DATA_4_7 0x08
@@ -437,9 +440,14 @@
#define GLOBAL2_WDOG_FORCE_IRQ BIT(0)
#define GLOBAL2_QOS_WEIGHT 0x1c
#define GLOBAL2_MISC 0x1d
+#define GLOBAL2_MISC_5_BIT_PORT BIT(14)
#define MV88E6XXX_N_FID 4096
+/* PVT limits for 4-bit port and 5-bit switch */
+#define MV88E6XXX_MAX_PVT_SWITCHES 32
+#define MV88E6XXX_MAX_PVT_PORTS 16
+
enum mv88e6xxx_frame_mode {
MV88E6XXX_FRAME_MODE_NORMAL,
MV88E6XXX_FRAME_MODE_DSA,
@@ -525,8 +533,6 @@
MV88E6XXX_CAP_G2_MGMT_EN_0X, /* (0x03) MGMT Enable Register 0x */
MV88E6XXX_CAP_G2_IRL_CMD, /* (0x09) Ingress Rate Command */
MV88E6XXX_CAP_G2_IRL_DATA, /* (0x0a) Ingress Rate Data */
- MV88E6XXX_CAP_G2_PVT_ADDR, /* (0x0b) Cross Chip Port VLAN Addr */
- MV88E6XXX_CAP_G2_PVT_DATA, /* (0x0c) Cross Chip Port VLAN Data */
MV88E6XXX_CAP_G2_POT, /* (0x0f) Priority Override Table */
/* Per VLAN Spanning Tree Unit (STU).
@@ -551,7 +557,6 @@
#define MV88E6XXX_FLAG_SERDES BIT_ULL(MV88E6XXX_CAP_SERDES)
-#define MV88E6XXX_FLAG_G1_ATU_FID BIT_ULL(MV88E6XXX_CAP_G1_ATU_FID)
#define MV88E6XXX_FLAG_G1_VTU_FID BIT_ULL(MV88E6XXX_CAP_G1_VTU_FID)
#define MV88E6XXX_FLAG_GLOBAL2 BIT_ULL(MV88E6XXX_CAP_GLOBAL2)
@@ -560,13 +565,8 @@
#define MV88E6XXX_FLAG_G2_MGMT_EN_0X BIT_ULL(MV88E6XXX_CAP_G2_MGMT_EN_0X)
#define MV88E6XXX_FLAG_G2_IRL_CMD BIT_ULL(MV88E6XXX_CAP_G2_IRL_CMD)
#define MV88E6XXX_FLAG_G2_IRL_DATA BIT_ULL(MV88E6XXX_CAP_G2_IRL_DATA)
-#define MV88E6XXX_FLAG_G2_PVT_ADDR BIT_ULL(MV88E6XXX_CAP_G2_PVT_ADDR)
-#define MV88E6XXX_FLAG_G2_PVT_DATA BIT_ULL(MV88E6XXX_CAP_G2_PVT_DATA)
#define MV88E6XXX_FLAG_G2_POT BIT_ULL(MV88E6XXX_CAP_G2_POT)
-#define MV88E6XXX_FLAG_STU BIT_ULL(MV88E6XXX_CAP_STU)
-#define MV88E6XXX_FLAG_VTU BIT_ULL(MV88E6XXX_CAP_VTU)
-
/* Ingress Rate Limit unit */
#define MV88E6XXX_FLAGS_IRL \
(MV88E6XXX_FLAG_G2_IRL_CMD | \
@@ -577,11 +577,6 @@
(MV88E6XXX_FLAG_SMI_CMD | \
MV88E6XXX_FLAG_SMI_DATA)
-/* Cross-chip Port VLAN Table */
-#define MV88E6XXX_FLAGS_PVT \
- (MV88E6XXX_FLAG_G2_PVT_ADDR | \
- MV88E6XXX_FLAG_G2_PVT_DATA)
-
/* Fiber/SERDES Registers at SMI address F, page 1 */
#define MV88E6XXX_FLAGS_SERDES \
(MV88E6XXX_FLAG_PHY_PAGE | \
@@ -590,43 +585,33 @@
#define MV88E6XXX_FLAGS_FAMILY_6095 \
(MV88E6XXX_FLAG_GLOBAL2 | \
MV88E6XXX_FLAG_G2_MGMT_EN_0X | \
- MV88E6XXX_FLAG_VTU | \
MV88E6XXX_FLAGS_MULTI_CHIP)
#define MV88E6XXX_FLAGS_FAMILY_6097 \
- (MV88E6XXX_FLAG_G1_ATU_FID | \
- MV88E6XXX_FLAG_G1_VTU_FID | \
+ (MV88E6XXX_FLAG_G1_VTU_FID | \
MV88E6XXX_FLAG_GLOBAL2 | \
MV88E6XXX_FLAG_G2_INT | \
MV88E6XXX_FLAG_G2_MGMT_EN_2X | \
MV88E6XXX_FLAG_G2_MGMT_EN_0X | \
MV88E6XXX_FLAG_G2_POT | \
- MV88E6XXX_FLAG_STU | \
- MV88E6XXX_FLAG_VTU | \
MV88E6XXX_FLAGS_IRL | \
- MV88E6XXX_FLAGS_MULTI_CHIP | \
- MV88E6XXX_FLAGS_PVT)
+ MV88E6XXX_FLAGS_MULTI_CHIP)
#define MV88E6XXX_FLAGS_FAMILY_6165 \
- (MV88E6XXX_FLAG_G1_ATU_FID | \
- MV88E6XXX_FLAG_G1_VTU_FID | \
+ (MV88E6XXX_FLAG_G1_VTU_FID | \
MV88E6XXX_FLAG_GLOBAL2 | \
MV88E6XXX_FLAG_G2_INT | \
MV88E6XXX_FLAG_G2_MGMT_EN_2X | \
MV88E6XXX_FLAG_G2_MGMT_EN_0X | \
MV88E6XXX_FLAG_G2_POT | \
- MV88E6XXX_FLAG_STU | \
- MV88E6XXX_FLAG_VTU | \
MV88E6XXX_FLAGS_IRL | \
- MV88E6XXX_FLAGS_MULTI_CHIP | \
- MV88E6XXX_FLAGS_PVT)
+ MV88E6XXX_FLAGS_MULTI_CHIP)
#define MV88E6XXX_FLAGS_FAMILY_6185 \
(MV88E6XXX_FLAG_GLOBAL2 | \
MV88E6XXX_FLAG_G2_INT | \
MV88E6XXX_FLAG_G2_MGMT_EN_0X | \
- MV88E6XXX_FLAGS_MULTI_CHIP | \
- MV88E6XXX_FLAG_VTU)
+ MV88E6XXX_FLAGS_MULTI_CHIP)
#define MV88E6XXX_FLAGS_FAMILY_6320 \
(MV88E6XXX_FLAG_EEE | \
@@ -634,64 +619,47 @@
MV88E6XXX_FLAG_G2_MGMT_EN_2X | \
MV88E6XXX_FLAG_G2_MGMT_EN_0X | \
MV88E6XXX_FLAG_G2_POT | \
- MV88E6XXX_FLAG_VTU | \
MV88E6XXX_FLAGS_IRL | \
- MV88E6XXX_FLAGS_MULTI_CHIP | \
- MV88E6XXX_FLAGS_PVT)
+ MV88E6XXX_FLAGS_MULTI_CHIP)
#define MV88E6XXX_FLAGS_FAMILY_6341 \
(MV88E6XXX_FLAG_EEE | \
- MV88E6XXX_FLAG_G1_ATU_FID | \
MV88E6XXX_FLAG_G1_VTU_FID | \
MV88E6XXX_FLAG_GLOBAL2 | \
MV88E6XXX_FLAG_G2_INT | \
MV88E6XXX_FLAG_G2_POT | \
- MV88E6XXX_FLAG_STU | \
- MV88E6XXX_FLAG_VTU | \
MV88E6XXX_FLAGS_IRL | \
MV88E6XXX_FLAGS_MULTI_CHIP | \
- MV88E6XXX_FLAGS_PVT | \
MV88E6XXX_FLAGS_SERDES)
#define MV88E6XXX_FLAGS_FAMILY_6351 \
- (MV88E6XXX_FLAG_G1_ATU_FID | \
- MV88E6XXX_FLAG_G1_VTU_FID | \
+ (MV88E6XXX_FLAG_G1_VTU_FID | \
MV88E6XXX_FLAG_GLOBAL2 | \
MV88E6XXX_FLAG_G2_INT | \
MV88E6XXX_FLAG_G2_MGMT_EN_2X | \
MV88E6XXX_FLAG_G2_MGMT_EN_0X | \
MV88E6XXX_FLAG_G2_POT | \
- MV88E6XXX_FLAG_STU | \
- MV88E6XXX_FLAG_VTU | \
MV88E6XXX_FLAGS_IRL | \
- MV88E6XXX_FLAGS_MULTI_CHIP | \
- MV88E6XXX_FLAGS_PVT)
+ MV88E6XXX_FLAGS_MULTI_CHIP)
#define MV88E6XXX_FLAGS_FAMILY_6352 \
(MV88E6XXX_FLAG_EEE | \
- MV88E6XXX_FLAG_G1_ATU_FID | \
MV88E6XXX_FLAG_G1_VTU_FID | \
MV88E6XXX_FLAG_GLOBAL2 | \
MV88E6XXX_FLAG_G2_INT | \
MV88E6XXX_FLAG_G2_MGMT_EN_2X | \
MV88E6XXX_FLAG_G2_MGMT_EN_0X | \
MV88E6XXX_FLAG_G2_POT | \
- MV88E6XXX_FLAG_STU | \
- MV88E6XXX_FLAG_VTU | \
MV88E6XXX_FLAGS_IRL | \
MV88E6XXX_FLAGS_MULTI_CHIP | \
- MV88E6XXX_FLAGS_PVT | \
MV88E6XXX_FLAGS_SERDES)
#define MV88E6XXX_FLAGS_FAMILY_6390 \
(MV88E6XXX_FLAG_EEE | \
MV88E6XXX_FLAG_GLOBAL2 | \
MV88E6XXX_FLAG_G2_INT | \
- MV88E6XXX_FLAG_STU | \
- MV88E6XXX_FLAG_VTU | \
MV88E6XXX_FLAGS_IRL | \
- MV88E6XXX_FLAGS_MULTI_CHIP | \
- MV88E6XXX_FLAGS_PVT)
+ MV88E6XXX_FLAGS_MULTI_CHIP)
struct mv88e6xxx_ops;
@@ -701,20 +669,26 @@
const char *name;
unsigned int num_databases;
unsigned int num_ports;
+ unsigned int max_vid;
unsigned int port_base_addr;
unsigned int global1_addr;
unsigned int age_time_coeff;
unsigned int g1_irqs;
+ bool pvt;
enum dsa_tag_protocol tag_protocol;
unsigned long long flags;
+
+ /* Mask for FromPort and ToPort value of PortVec used in ATU Move
+ * operation. 0 means that the ATU Move operation is not supported.
+ */
+ u8 atu_move_port_mask;
const struct mv88e6xxx_ops *ops;
};
struct mv88e6xxx_atu_entry {
- u16 fid;
u8 state;
bool trunk;
- u16 portv_trunkid;
+ u16 portvec;
u8 mac[ETH_ALEN];
};
@@ -723,7 +697,8 @@
u16 fid;
u8 sid;
bool valid;
- u8 data[DSA_MAX_PORTS];
+ u8 member[DSA_MAX_PORTS];
+ u8 state[DSA_MAX_PORTS];
};
struct mv88e6xxx_bus_ops;
@@ -864,14 +839,16 @@
int (*port_set_frame_mode)(struct mv88e6xxx_chip *chip, int port,
enum mv88e6xxx_frame_mode mode);
- int (*port_set_egress_unknowns)(struct mv88e6xxx_chip *chip, int port,
- bool on);
+ int (*port_set_egress_floods)(struct mv88e6xxx_chip *chip, int port,
+ bool unicast, bool multicast);
int (*port_set_ether_type)(struct mv88e6xxx_chip *chip, int port,
u16 etype);
int (*port_jumbo_config)(struct mv88e6xxx_chip *chip, int port);
int (*port_egress_rate_limiting)(struct mv88e6xxx_chip *chip, int port);
int (*port_pause_config)(struct mv88e6xxx_chip *chip, int port);
+ int (*port_disable_learn_limit)(struct mv88e6xxx_chip *chip, int port);
+ int (*port_disable_pri_override)(struct mv88e6xxx_chip *chip, int port);
/* CMODE control what PHY mode the MAC will use, eg. SGMII, RGMII, etc.
* Some chips allow this to be configured on specific ports.
@@ -906,6 +883,12 @@
/* Can be either in g1 or g2, so don't use a prefix */
int (*mgmt_rsvd2cpu)(struct mv88e6xxx_chip *chip);
+
+ /* VLAN Translation Unit operations */
+ int (*vtu_getnext)(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry);
+ int (*vtu_loadpurge)(struct mv88e6xxx_chip *chip,
+ struct mv88e6xxx_vtu_entry *entry);
};
struct mv88e6xxx_irq_ops {
@@ -934,6 +917,11 @@
return (chip->info->flags & flags) == flags;
}
+static inline bool mv88e6xxx_has_pvt(struct mv88e6xxx_chip *chip)
+{
+ return chip->info->pvt;
+}
+
static inline unsigned int mv88e6xxx_num_databases(struct mv88e6xxx_chip *chip)
{
return chip->info->num_databases;
@@ -944,6 +932,11 @@
return chip->info->num_ports;
}
+static inline u16 mv88e6xxx_port_mask(struct mv88e6xxx_chip *chip)
+{
+ return GENMASK(mv88e6xxx_num_ports(chip) - 1, 0);
+}
+
int mv88e6xxx_read(struct mv88e6xxx_chip *chip, int addr, int reg, u16 *val);
int mv88e6xxx_write(struct mv88e6xxx_chip *chip, int addr, int reg, u16 val);
int mv88e6xxx_update(struct mv88e6xxx_chip *chip, int addr, int reg,
diff --git a/drivers/net/dsa/mv88e6xxx/port.c b/drivers/net/dsa/mv88e6xxx/port.c
index 8875784..548a956 100644
--- a/drivers/net/dsa/mv88e6xxx/port.c
+++ b/drivers/net/dsa/mv88e6xxx/port.c
@@ -3,7 +3,8 @@
*
* Copyright (c) 2008 Marvell Semiconductor
*
- * Copyright (c) 2016 Vivien Didelot <vivien.didelot@savoirfairelinux.com>
+ * Copyright (c) 2016-2017 Savoir-faire Linux Inc.
+ * Vivien Didelot <vivien.didelot@savoirfairelinux.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -497,8 +498,8 @@
return mv88e6xxx_port_write(chip, port, PORT_CONTROL, reg);
}
-int mv88e6085_port_set_egress_unknowns(struct mv88e6xxx_chip *chip, int port,
- bool on)
+static int mv88e6185_port_set_forward_unknown(struct mv88e6xxx_chip *chip,
+ int port, bool unicast)
{
int err;
u16 reg;
@@ -507,7 +508,7 @@
if (err)
return err;
- if (on)
+ if (unicast)
reg |= PORT_CONTROL_FORWARD_UNKNOWN;
else
reg &= ~PORT_CONTROL_FORWARD_UNKNOWN;
@@ -515,8 +516,8 @@
return mv88e6xxx_port_write(chip, port, PORT_CONTROL, reg);
}
-int mv88e6351_port_set_egress_unknowns(struct mv88e6xxx_chip *chip, int port,
- bool on)
+int mv88e6352_port_set_egress_floods(struct mv88e6xxx_chip *chip, int port,
+ bool unicast, bool multicast)
{
int err;
u16 reg;
@@ -525,21 +526,45 @@
if (err)
return err;
- if (on)
- reg |= PORT_CONTROL_EGRESS_ALL_UNKNOWN_DA;
+ reg &= ~PORT_CONTROL_EGRESS_FLOODS_MASK;
+
+ if (unicast && multicast)
+ reg |= PORT_CONTROL_EGRESS_FLOODS_ALL_UNKNOWN_DA;
+ else if (unicast)
+ reg |= PORT_CONTROL_EGRESS_FLOODS_NO_UNKNOWN_MC_DA;
+ else if (multicast)
+ reg |= PORT_CONTROL_EGRESS_FLOODS_NO_UNKNOWN_UC_DA;
else
- reg &= ~PORT_CONTROL_EGRESS_ALL_UNKNOWN_DA;
+ reg |= PORT_CONTROL_EGRESS_FLOODS_NO_UNKNOWN_DA;
return mv88e6xxx_port_write(chip, port, PORT_CONTROL, reg);
}
/* Offset 0x05: Port Control 1 */
+int mv88e6xxx_port_set_message_port(struct mv88e6xxx_chip *chip, int port,
+ bool message_port)
+{
+ u16 val;
+ int err;
+
+ err = mv88e6xxx_port_read(chip, port, PORT_CONTROL_1, &val);
+ if (err)
+ return err;
+
+ if (message_port)
+ val |= PORT_CONTROL_1_MESSAGE_PORT;
+ else
+ val &= ~PORT_CONTROL_1_MESSAGE_PORT;
+
+ return mv88e6xxx_port_write(chip, port, PORT_CONTROL_1, val);
+}
+
/* Offset 0x06: Port Based VLAN Map */
int mv88e6xxx_port_set_vlan_map(struct mv88e6xxx_chip *chip, int port, u16 map)
{
- const u16 mask = GENMASK(mv88e6xxx_num_ports(chip) - 1, 0);
+ const u16 mask = mv88e6xxx_port_mask(chip);
u16 reg;
int err;
@@ -672,8 +697,8 @@
[PORT_CONTROL_2_8021Q_SECURE] = "Secure",
};
-int mv88e6095_port_set_egress_unknowns(struct mv88e6xxx_chip *chip, int port,
- bool on)
+static int mv88e6185_port_set_default_forward(struct mv88e6xxx_chip *chip,
+ int port, bool multicast)
{
int err;
u16 reg;
@@ -682,14 +707,26 @@
if (err)
return err;
- if (on)
- reg |= PORT_CONTROL_2_FORWARD_UNKNOWN;
+ if (multicast)
+ reg |= PORT_CONTROL_2_DEFAULT_FORWARD;
else
- reg &= ~PORT_CONTROL_2_FORWARD_UNKNOWN;
+ reg &= ~PORT_CONTROL_2_DEFAULT_FORWARD;
return mv88e6xxx_port_write(chip, port, PORT_CONTROL_2, reg);
}
+int mv88e6185_port_set_egress_floods(struct mv88e6xxx_chip *chip, int port,
+ bool unicast, bool multicast)
+{
+ int err;
+
+ err = mv88e6185_port_set_forward_unknown(chip, port, unicast);
+ if (err)
+ return err;
+
+ return mv88e6185_port_set_default_forward(chip, port, multicast);
+}
+
int mv88e6095_port_set_upstream_port(struct mv88e6xxx_chip *chip, int port,
int upstream_port)
{
@@ -769,6 +806,20 @@
return mv88e6xxx_port_write(chip, port, PORT_RATE_CONTROL, 0x0001);
}
+/* Offset 0x0C: Port ATU Control */
+
+int mv88e6xxx_port_disable_learn_limit(struct mv88e6xxx_chip *chip, int port)
+{
+ return mv88e6xxx_port_write(chip, port, PORT_ATU_CONTROL, 0);
+}
+
+/* Offset 0x0D: (Priority) Override Register */
+
+int mv88e6xxx_port_disable_pri_override(struct mv88e6xxx_chip *chip, int port)
+{
+ return mv88e6xxx_port_write(chip, port, PORT_PRI_OVERRIDE, 0);
+}
+
/* Offset 0x0f: Port Ether type */
int mv88e6351_port_set_ether_type(struct mv88e6xxx_chip *chip, int port,
diff --git a/drivers/net/dsa/mv88e6xxx/port.h b/drivers/net/dsa/mv88e6xxx/port.h
index c83cbb3..86f4088 100644
--- a/drivers/net/dsa/mv88e6xxx/port.h
+++ b/drivers/net/dsa/mv88e6xxx/port.h
@@ -3,7 +3,8 @@
*
* Copyright (c) 2008 Marvell Semiconductor
*
- * Copyright (c) 2016 Vivien Didelot <vivien.didelot@savoirfairelinux.com>
+ * Copyright (c) 2016-2017 Savoir-faire Linux Inc.
+ * Vivien Didelot <vivien.didelot@savoirfairelinux.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -56,14 +57,14 @@
enum mv88e6xxx_frame_mode mode);
int mv88e6351_port_set_frame_mode(struct mv88e6xxx_chip *chip, int port,
enum mv88e6xxx_frame_mode mode);
-int mv88e6085_port_set_egress_unknowns(struct mv88e6xxx_chip *chip, int port,
- bool on);
-int mv88e6095_port_set_egress_unknowns(struct mv88e6xxx_chip *chip, int port,
- bool on);
-int mv88e6351_port_set_egress_unknowns(struct mv88e6xxx_chip *chip, int port,
- bool on);
+int mv88e6185_port_set_egress_floods(struct mv88e6xxx_chip *chip, int port,
+ bool unicast, bool multicast);
+int mv88e6352_port_set_egress_floods(struct mv88e6xxx_chip *chip, int port,
+ bool unicast, bool multicast);
int mv88e6351_port_set_ether_type(struct mv88e6xxx_chip *chip, int port,
u16 etype);
+int mv88e6xxx_port_set_message_port(struct mv88e6xxx_chip *chip, int port,
+ bool message_port);
int mv88e6165_port_jumbo_config(struct mv88e6xxx_chip *chip, int port);
int mv88e6095_port_egress_rate_limiting(struct mv88e6xxx_chip *chip, int port);
int mv88e6097_port_egress_rate_limiting(struct mv88e6xxx_chip *chip, int port);
@@ -75,4 +76,8 @@
int mv88e6xxx_port_set_map_da(struct mv88e6xxx_chip *chip, int port);
int mv88e6095_port_set_upstream_port(struct mv88e6xxx_chip *chip, int port,
int upstream_port);
+
+int mv88e6xxx_port_disable_learn_limit(struct mv88e6xxx_chip *chip, int port);
+int mv88e6xxx_port_disable_pri_override(struct mv88e6xxx_chip *chip, int port);
+
#endif /* _MV88E6XXX_PORT_H */
diff --git a/drivers/net/dummy.c b/drivers/net/dummy.c
index 2c80611..149244a 100644
--- a/drivers/net/dummy.c
+++ b/drivers/net/dummy.c
@@ -35,6 +35,7 @@
#include <linux/init.h>
#include <linux/moduleparam.h>
#include <linux/rtnetlink.h>
+#include <linux/net_tstamp.h>
#include <net/rtnetlink.h>
#include <linux/u64_stats_sync.h>
@@ -125,6 +126,7 @@
dstats->tx_bytes += skb->len;
u64_stats_update_end(&dstats->syncp);
+ skb_tx_timestamp(skb);
dev_kfree_skb(skb);
return NETDEV_TX_OK;
}
@@ -304,8 +306,21 @@
strlcpy(info->version, DRV_VERSION, sizeof(info->version));
}
+static int dummy_get_ts_info(struct net_device *dev,
+ struct ethtool_ts_info *ts_info)
+{
+ ts_info->so_timestamping = SOF_TIMESTAMPING_TX_SOFTWARE |
+ SOF_TIMESTAMPING_RX_SOFTWARE |
+ SOF_TIMESTAMPING_SOFTWARE;
+
+ ts_info->phc_index = -1;
+
+ return 0;
+};
+
static const struct ethtool_ops dummy_ethtool_ops = {
.get_drvinfo = dummy_get_drvinfo,
+ .get_ts_info = dummy_get_ts_info,
};
static void dummy_free_netdev(struct net_device *dev)
diff --git a/drivers/net/ethernet/3com/typhoon.c b/drivers/net/ethernet/3com/typhoon.c
index 084a6d5..be823c18 100644
--- a/drivers/net/ethernet/3com/typhoon.c
+++ b/drivers/net/ethernet/3com/typhoon.c
@@ -283,7 +283,6 @@
spinlock_t command_lock ____cacheline_aligned;
struct basic_ring cmdRing;
struct basic_ring respRing;
- struct net_device_stats stats;
struct net_device_stats stats_saved;
struct typhoon_shared * shared;
dma_addr_t shared_dma;
@@ -898,7 +897,7 @@
static int
typhoon_do_get_stats(struct typhoon *tp)
{
- struct net_device_stats *stats = &tp->stats;
+ struct net_device_stats *stats = &tp->dev->stats;
struct net_device_stats *saved = &tp->stats_saved;
struct cmd_desc xp_cmd;
struct resp_desc xp_resp[7];
@@ -951,7 +950,7 @@
typhoon_get_stats(struct net_device *dev)
{
struct typhoon *tp = netdev_priv(dev);
- struct net_device_stats *stats = &tp->stats;
+ struct net_device_stats *stats = &tp->dev->stats;
struct net_device_stats *saved = &tp->stats_saved;
smp_rmb();
@@ -1991,7 +1990,7 @@
tp->card_state = Sleeping;
smp_wmb();
typhoon_do_get_stats(tp);
- memcpy(&tp->stats_saved, &tp->stats, sizeof(struct net_device_stats));
+ memcpy(&tp->stats_saved, &tp->dev->stats, sizeof(struct net_device_stats));
INIT_COMMAND_NO_RESPONSE(&xp_cmd, TYPHOON_CMD_HALT);
typhoon_issue_command(tp, 1, &xp_cmd, 0, NULL);
diff --git a/drivers/net/ethernet/Kconfig b/drivers/net/ethernet/Kconfig
index 8c08f9d..edae15ac 100644
--- a/drivers/net/ethernet/Kconfig
+++ b/drivers/net/ethernet/Kconfig
@@ -180,5 +180,6 @@
source "drivers/net/ethernet/wiznet/Kconfig"
source "drivers/net/ethernet/xilinx/Kconfig"
source "drivers/net/ethernet/xircom/Kconfig"
+source "drivers/net/ethernet/synopsys/Kconfig"
endif # ETHERNET
diff --git a/drivers/net/ethernet/Makefile b/drivers/net/ethernet/Makefile
index 26dce5b..bf7f450 100644
--- a/drivers/net/ethernet/Makefile
+++ b/drivers/net/ethernet/Makefile
@@ -91,3 +91,4 @@
obj-$(CONFIG_NET_VENDOR_WIZNET) += wiznet/
obj-$(CONFIG_NET_VENDOR_XILINX) += xilinx/
obj-$(CONFIG_NET_VENDOR_XIRCOM) += xircom/
+obj-$(CONFIG_NET_VENDOR_SYNOPSYS) += synopsys/
diff --git a/drivers/net/ethernet/adi/bfin_mac.h b/drivers/net/ethernet/adi/bfin_mac.h
index 8c3b561..4ad5b9b 100644
--- a/drivers/net/ethernet/adi/bfin_mac.h
+++ b/drivers/net/ethernet/adi/bfin_mac.h
@@ -68,13 +68,6 @@
};
struct bfin_mac_local {
- /*
- * these are things that the kernel wants me to keep, so users
- * can find out semi-useless statistics of how well the card is
- * performing
- */
- struct net_device_stats stats;
-
spinlock_t lock;
int wol; /* Wake On Lan */
diff --git a/drivers/net/ethernet/aeroflex/greth.c b/drivers/net/ethernet/aeroflex/greth.c
index 9f7422a..d8e133c 100644
--- a/drivers/net/ethernet/aeroflex/greth.c
+++ b/drivers/net/ethernet/aeroflex/greth.c
@@ -34,6 +34,7 @@
#include <linux/crc32.h>
#include <linux/mii.h>
#include <linux/of_device.h>
+#include <linux/of_net.h>
#include <linux/of_platform.h>
#include <linux/slab.h>
#include <asm/cacheflush.h>
@@ -1454,11 +1455,10 @@
break;
}
if (i == 6) {
- const unsigned char *addr;
- int len;
- addr = of_get_property(ofdev->dev.of_node, "local-mac-address",
- &len);
- if (addr != NULL && len == 6) {
+ const u8 *addr;
+
+ addr = of_get_mac_address(ofdev->dev.of_node);
+ if (addr) {
for (i = 0; i < 6; i++)
macaddr[i] = (unsigned int) addr[i];
} else {
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index 35f1943..7c1214d 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -133,7 +133,7 @@
int irq_idx = ENA_IO_IRQ_IDX(i);
rc = irq_cpu_rmap_add(adapter->netdev->rx_cpu_rmap,
- adapter->msix_entries[irq_idx].vector);
+ pci_irq_vector(adapter->pdev, irq_idx));
if (rc) {
free_irq_cpu_rmap(adapter->netdev->rx_cpu_rmap);
adapter->netdev->rx_cpu_rmap = NULL;
@@ -1208,13 +1208,7 @@
static int ena_enable_msix(struct ena_adapter *adapter, int num_queues)
{
- int i, msix_vecs, rc;
-
- if (test_bit(ENA_FLAG_MSIX_ENABLED, &adapter->flags)) {
- netif_err(adapter, probe, adapter->netdev,
- "Error, MSI-X is already enabled\n");
- return -EPERM;
- }
+ int msix_vecs, rc;
/* Reserved the max msix vectors we might need */
msix_vecs = ENA_MAX_MSIX_VEC(num_queues);
@@ -1222,16 +1216,9 @@
netif_dbg(adapter, probe, adapter->netdev,
"trying to enable MSI-X, vectors %d\n", msix_vecs);
- adapter->msix_entries = vzalloc(msix_vecs * sizeof(struct msix_entry));
-
- if (!adapter->msix_entries)
- return -ENOMEM;
-
- for (i = 0; i < msix_vecs; i++)
- adapter->msix_entries[i].entry = i;
-
- rc = pci_enable_msix(adapter->pdev, adapter->msix_entries, msix_vecs);
- if (rc != 0) {
+ rc = pci_alloc_irq_vectors(adapter->pdev, msix_vecs, msix_vecs,
+ PCI_IRQ_MSIX);
+ if (rc < 0) {
netif_err(adapter, probe, adapter->netdev,
"Failed to enable MSI-X, vectors %d rc %d\n",
msix_vecs, rc);
@@ -1248,7 +1235,6 @@
}
adapter->msix_vecs = msix_vecs;
- set_bit(ENA_FLAG_MSIX_ENABLED, &adapter->flags);
return 0;
}
@@ -1264,7 +1250,7 @@
ena_intr_msix_mgmnt;
adapter->irq_tbl[ENA_MGMNT_IRQ_IDX].data = adapter;
adapter->irq_tbl[ENA_MGMNT_IRQ_IDX].vector =
- adapter->msix_entries[ENA_MGMNT_IRQ_IDX].vector;
+ pci_irq_vector(adapter->pdev, ENA_MGMNT_IRQ_IDX);
cpu = cpumask_first(cpu_online_mask);
adapter->irq_tbl[ENA_MGMNT_IRQ_IDX].cpu = cpu;
cpumask_set_cpu(cpu,
@@ -1287,7 +1273,7 @@
adapter->irq_tbl[irq_idx].handler = ena_intr_msix_io;
adapter->irq_tbl[irq_idx].data = &adapter->ena_napi[i];
adapter->irq_tbl[irq_idx].vector =
- adapter->msix_entries[irq_idx].vector;
+ pci_irq_vector(adapter->pdev, irq_idx);
adapter->irq_tbl[irq_idx].cpu = cpu;
cpumask_set_cpu(cpu,
@@ -1325,12 +1311,6 @@
struct ena_irq *irq;
int rc = 0, i, k;
- if (!test_bit(ENA_FLAG_MSIX_ENABLED, &adapter->flags)) {
- netif_err(adapter, ifup, adapter->netdev,
- "Failed to request I/O IRQ: MSI-X is not enabled\n");
- return -EINVAL;
- }
-
for (i = ENA_IO_IRQ_FIRST_IDX; i < adapter->msix_vecs; i++) {
irq = &adapter->irq_tbl[i];
rc = request_irq(irq->vector, irq->handler, flags, irq->name,
@@ -1389,16 +1369,6 @@
}
}
-static void ena_disable_msix(struct ena_adapter *adapter)
-{
- if (test_and_clear_bit(ENA_FLAG_MSIX_ENABLED, &adapter->flags))
- pci_disable_msix(adapter->pdev);
-
- if (adapter->msix_entries)
- vfree(adapter->msix_entries);
- adapter->msix_entries = NULL;
-}
-
static void ena_disable_io_intr_sync(struct ena_adapter *adapter)
{
int i;
@@ -2479,8 +2449,7 @@
return 0;
err_disable_msix:
- ena_disable_msix(adapter);
-
+ pci_free_irq_vectors(adapter->pdev);
return rc;
}
@@ -2518,7 +2487,7 @@
ena_free_mgmnt_irq(adapter);
- ena_disable_msix(adapter);
+ pci_free_irq_vectors(adapter->pdev);
ena_com_abort_admin_commands(ena_dev);
@@ -2569,7 +2538,7 @@
return;
err_disable_msix:
ena_free_mgmnt_irq(adapter);
- ena_disable_msix(adapter);
+ pci_free_irq_vectors(adapter->pdev);
err_device_destroy:
ena_com_admin_destroy(ena_dev);
err:
@@ -3103,7 +3072,7 @@
err_free_msix:
ena_com_dev_reset(ena_dev);
ena_free_mgmnt_irq(adapter);
- ena_disable_msix(adapter);
+ pci_free_irq_vectors(adapter->pdev);
err_worker_destroy:
ena_com_destroy_interrupt_moderation(ena_dev);
del_timer(&adapter->timer_service);
@@ -3188,7 +3157,7 @@
ena_free_mgmnt_irq(adapter);
- ena_disable_msix(adapter);
+ pci_free_irq_vectors(adapter->pdev);
free_netdev(netdev);
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h b/drivers/net/ethernet/amazon/ena/ena_netdev.h
index ed62d8e..0e22bce 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.h
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h
@@ -248,7 +248,6 @@
ENA_FLAG_DEVICE_RUNNING,
ENA_FLAG_DEV_UP,
ENA_FLAG_LINK_UP,
- ENA_FLAG_MSIX_ENABLED,
ENA_FLAG_TRIGGER_RESET
};
@@ -267,7 +266,6 @@
int num_queues;
- struct msix_entry *msix_entries;
int msix_vecs;
u32 tx_usecs, rx_usecs; /* interrupt moderation */
diff --git a/drivers/net/ethernet/amd/nmclan_cs.c b/drivers/net/ethernet/amd/nmclan_cs.c
index b556c92..9c152d8 100644
--- a/drivers/net/ethernet/amd/nmclan_cs.c
+++ b/drivers/net/ethernet/amd/nmclan_cs.c
@@ -359,7 +359,6 @@
typedef struct _mace_private {
struct pcmcia_device *p_dev;
- struct net_device_stats linux_stats; /* Linux statistics counters */
mace_statistics mace_stats; /* MACE chip statistics counters */
/* restore_multicast_list() state variables */
@@ -879,7 +878,7 @@
service a transmit interrupt while we are in here.
*/
- lp->linux_stats.tx_bytes += skb->len;
+ dev->stats.tx_bytes += skb->len;
lp->tx_free_frames--;
/* WARNING: Write the _exact_ number of bytes written in the header! */
@@ -967,7 +966,7 @@
fifofc = inb(ioaddr + AM2150_MACE_BASE + MACE_FIFOFC);
if ((fifofc & MACE_FIFOFC_XMTFC)==0) {
- lp->linux_stats.tx_errors++;
+ dev->stats.tx_errors++;
outb(0xFF, ioaddr + AM2150_XMT_SKIP);
}
@@ -1016,7 +1015,7 @@
} /* if (xmtfs & MACE_XMTFS_XMTSV) */
- lp->linux_stats.tx_packets++;
+ dev->stats.tx_packets++;
lp->tx_free_frames++;
netif_wake_queue(dev);
} /* if (status & MACE_IR_XMTINT) */
@@ -1077,7 +1076,7 @@
" 0x%X.\n", dev->name, rx_framecnt, rx_status);
if (rx_status & MACE_RCVFS_RCVSTS) { /* Error, update stats. */
- lp->linux_stats.rx_errors++;
+ dev->stats.rx_errors++;
if (rx_status & MACE_RCVFS_OFLO) {
lp->mace_stats.oflo++;
}
@@ -1114,14 +1113,14 @@
netif_rx(skb); /* Send the packet to the upper (protocol) layers. */
- lp->linux_stats.rx_packets++;
- lp->linux_stats.rx_bytes += pkt_len;
+ dev->stats.rx_packets++;
+ dev->stats.rx_bytes += pkt_len;
outb(0xFF, ioaddr + AM2150_RCV_NEXT); /* skip to next frame */
continue;
} else {
pr_debug("%s: couldn't allocate a sk_buff of size"
" %d.\n", dev->name, pkt_len);
- lp->linux_stats.rx_dropped++;
+ dev->stats.rx_dropped++;
}
}
outb(0xFF, ioaddr + AM2150_RCV_NEXT); /* skip to next frame */
@@ -1231,13 +1230,13 @@
lp->mace_stats.rntpc += mace_read(lp, ioaddr, MACE_RNTPC);
lp->mace_stats.mpc += mace_read(lp, ioaddr, MACE_MPC);
/* At this point, mace_stats is fully updated for this call.
- We may now update the linux_stats. */
+ We may now update the netdev stats. */
- /* The MACE has no equivalent for linux_stats field which are commented
+ /* The MACE has no equivalent for netdev stats field which are commented
out. */
- /* lp->linux_stats.multicast; */
- lp->linux_stats.collisions =
+ /* dev->stats.multicast; */
+ dev->stats.collisions =
lp->mace_stats.rcvcco * 256 + lp->mace_stats.rcvcc;
/* Collision: The MACE may retry sending a packet 15 times
before giving up. The retry count is in XMTRC.
@@ -1245,22 +1244,22 @@
If so, why doesn't the RCVCC record these collisions? */
/* detailed rx_errors: */
- lp->linux_stats.rx_length_errors =
+ dev->stats.rx_length_errors =
lp->mace_stats.rntpco * 256 + lp->mace_stats.rntpc;
- /* lp->linux_stats.rx_over_errors */
- lp->linux_stats.rx_crc_errors = lp->mace_stats.fcs;
- lp->linux_stats.rx_frame_errors = lp->mace_stats.fram;
- lp->linux_stats.rx_fifo_errors = lp->mace_stats.oflo;
- lp->linux_stats.rx_missed_errors =
+ /* dev->stats.rx_over_errors */
+ dev->stats.rx_crc_errors = lp->mace_stats.fcs;
+ dev->stats.rx_frame_errors = lp->mace_stats.fram;
+ dev->stats.rx_fifo_errors = lp->mace_stats.oflo;
+ dev->stats.rx_missed_errors =
lp->mace_stats.mpco * 256 + lp->mace_stats.mpc;
/* detailed tx_errors */
- lp->linux_stats.tx_aborted_errors = lp->mace_stats.rtry;
- lp->linux_stats.tx_carrier_errors = lp->mace_stats.lcar;
+ dev->stats.tx_aborted_errors = lp->mace_stats.rtry;
+ dev->stats.tx_carrier_errors = lp->mace_stats.lcar;
/* LCAR usually results from bad cabling. */
- lp->linux_stats.tx_fifo_errors = lp->mace_stats.uflo;
- lp->linux_stats.tx_heartbeat_errors = lp->mace_stats.cerr;
- /* lp->linux_stats.tx_window_errors; */
+ dev->stats.tx_fifo_errors = lp->mace_stats.uflo;
+ dev->stats.tx_heartbeat_errors = lp->mace_stats.cerr;
+ /* dev->stats.tx_window_errors; */
} /* update_stats */
/* ----------------------------------------------------------------------------
@@ -1274,10 +1273,10 @@
update_stats(dev->base_addr, dev);
pr_debug("%s: updating the statistics.\n", dev->name);
- pr_linux_stats(&lp->linux_stats);
+ pr_linux_stats(&dev->stats);
pr_mace_stats(&lp->mace_stats);
- return &lp->linux_stats;
+ return &dev->stats;
} /* net_device_stats */
/* ----------------------------------------------------------------------------
diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-drv.c b/drivers/net/ethernet/amd/xgbe/xgbe-drv.c
index a713abd..c772420 100644
--- a/drivers/net/ethernet/amd/xgbe/xgbe-drv.c
+++ b/drivers/net/ethernet/amd/xgbe/xgbe-drv.c
@@ -118,6 +118,7 @@
#include <linux/spinlock.h>
#include <linux/tcp.h>
#include <linux/if_vlan.h>
+#include <linux/interrupt.h>
#include <net/busy_poll.h>
#include <linux/clk.h>
#include <linux/if_ether.h>
@@ -1854,7 +1855,8 @@
if (tc_to_netdev->type != TC_SETUP_MQPRIO)
return -EINVAL;
- tc = tc_to_netdev->tc;
+ tc_to_netdev->mqprio->hw = TC_MQPRIO_HW_OFFLOAD_TCS;
+ tc = tc_to_netdev->mqprio->num_tc;
if (tc > pdata->hw_feat.tc_cnt)
return -EINVAL;
diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-i2c.c b/drivers/net/ethernet/amd/xgbe/xgbe-i2c.c
index 0c7088a..417bdb5 100644
--- a/drivers/net/ethernet/amd/xgbe/xgbe-i2c.c
+++ b/drivers/net/ethernet/amd/xgbe/xgbe-i2c.c
@@ -115,6 +115,7 @@
*/
#include <linux/module.h>
+#include <linux/interrupt.h>
#include <linux/kmod.h>
#include <linux/delay.h>
#include <linux/completion.h>
diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-mdio.c b/drivers/net/ethernet/amd/xgbe/xgbe-mdio.c
index 4c5b90e..b672d92 100644
--- a/drivers/net/ethernet/amd/xgbe/xgbe-mdio.c
+++ b/drivers/net/ethernet/amd/xgbe/xgbe-mdio.c
@@ -114,6 +114,7 @@
* THE POSSIBILITY OF SUCH DAMAGE.
*/
+#include <linux/interrupt.h>
#include <linux/module.h>
#include <linux/kmod.h>
#include <linux/mdio.h>
diff --git a/drivers/net/ethernet/apm/Kconfig b/drivers/net/ethernet/apm/Kconfig
index ec63d70..59efe5b 100644
--- a/drivers/net/ethernet/apm/Kconfig
+++ b/drivers/net/ethernet/apm/Kconfig
@@ -1 +1,2 @@
source "drivers/net/ethernet/apm/xgene/Kconfig"
+source "drivers/net/ethernet/apm/xgene-v2/Kconfig"
diff --git a/drivers/net/ethernet/apm/Makefile b/drivers/net/ethernet/apm/Makefile
index 65ce32a..946b2a4 100644
--- a/drivers/net/ethernet/apm/Makefile
+++ b/drivers/net/ethernet/apm/Makefile
@@ -3,3 +3,4 @@
#
obj-$(CONFIG_NET_XGENE) += xgene/
+obj-$(CONFIG_NET_XGENE_V2) += xgene-v2/
diff --git a/drivers/net/ethernet/apm/xgene-v2/Kconfig b/drivers/net/ethernet/apm/xgene-v2/Kconfig
new file mode 100644
index 0000000..1205861
--- /dev/null
+++ b/drivers/net/ethernet/apm/xgene-v2/Kconfig
@@ -0,0 +1,11 @@
+config NET_XGENE_V2
+ tristate "APM X-Gene SoC Ethernet-v2 Driver"
+ depends on HAS_DMA
+ depends on ARCH_XGENE || COMPILE_TEST
+ help
+ This is the Ethernet driver for the on-chip ethernet interface
+ which uses a linked list of DMA descriptor architecture (v2) for
+ APM X-Gene SoCs.
+
+ To compile this driver as a module, choose M here. This module will
+ be called xgene-enet-v2.
diff --git a/drivers/net/ethernet/apm/xgene-v2/Makefile b/drivers/net/ethernet/apm/xgene-v2/Makefile
new file mode 100644
index 0000000..f16a2b3
--- /dev/null
+++ b/drivers/net/ethernet/apm/xgene-v2/Makefile
@@ -0,0 +1,6 @@
+#
+# Makefile for APM X-Gene Ethernet v2 driver
+#
+
+xgene-enet-v2-objs := main.o mac.o enet.o ring.o mdio.o ethtool.o
+obj-$(CONFIG_NET_XGENE_V2) += xgene-enet-v2.o
diff --git a/drivers/net/ethernet/apm/xgene-v2/enet.c b/drivers/net/ethernet/apm/xgene-v2/enet.c
new file mode 100644
index 0000000..5998da0
--- /dev/null
+++ b/drivers/net/ethernet/apm/xgene-v2/enet.c
@@ -0,0 +1,83 @@
+/*
+ * Applied Micro X-Gene SoC Ethernet v2 Driver
+ *
+ * Copyright (c) 2017, Applied Micro Circuits Corporation
+ * Author(s): Iyappan Subramanian <isubramanian@apm.com>
+ * Keyur Chudgar <kchudgar@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "main.h"
+
+void xge_wr_csr(struct xge_pdata *pdata, u32 offset, u32 val)
+{
+ void __iomem *addr = pdata->resources.base_addr + offset;
+
+ iowrite32(val, addr);
+}
+
+u32 xge_rd_csr(struct xge_pdata *pdata, u32 offset)
+{
+ void __iomem *addr = pdata->resources.base_addr + offset;
+
+ return ioread32(addr);
+}
+
+int xge_port_reset(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct device *dev = &pdata->pdev->dev;
+ u32 data, wait = 10;
+
+ xge_wr_csr(pdata, ENET_CLKEN, 0x3);
+ xge_wr_csr(pdata, ENET_SRST, 0xf);
+ xge_wr_csr(pdata, ENET_SRST, 0);
+ xge_wr_csr(pdata, CFG_MEM_RAM_SHUTDOWN, 1);
+ xge_wr_csr(pdata, CFG_MEM_RAM_SHUTDOWN, 0);
+
+ do {
+ usleep_range(100, 110);
+ data = xge_rd_csr(pdata, BLOCK_MEM_RDY);
+ } while (data != MEM_RDY && wait--);
+
+ if (data != MEM_RDY) {
+ dev_err(dev, "ECC init failed: %x\n", data);
+ return -ETIMEDOUT;
+ }
+
+ xge_wr_csr(pdata, ENET_SHIM, DEVM_ARAUX_COH | DEVM_AWAUX_COH);
+
+ return 0;
+}
+
+static void xge_traffic_resume(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+
+ xge_wr_csr(pdata, CFG_FORCE_LINK_STATUS_EN, 1);
+ xge_wr_csr(pdata, FORCE_LINK_STATUS, 1);
+
+ xge_wr_csr(pdata, CFG_LINK_AGGR_RESUME, 1);
+ xge_wr_csr(pdata, RX_DV_GATE_REG, 1);
+}
+
+void xge_port_init(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+
+ pdata->phy_speed = SPEED_1000;
+ xge_mac_init(pdata);
+ xge_traffic_resume(ndev);
+}
diff --git a/drivers/net/ethernet/apm/xgene-v2/enet.h b/drivers/net/ethernet/apm/xgene-v2/enet.h
new file mode 100644
index 0000000..3fd36dc6
--- /dev/null
+++ b/drivers/net/ethernet/apm/xgene-v2/enet.h
@@ -0,0 +1,45 @@
+/*
+ * Applied Micro X-Gene SoC Ethernet v2 Driver
+ *
+ * Copyright (c) 2017, Applied Micro Circuits Corporation
+ * Author(s): Iyappan Subramanian <isubramanian@apm.com>
+ * Keyur Chudgar <kchudgar@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __XGENE_ENET_V2_ENET_H__
+#define __XGENE_ENET_V2_ENET_H__
+
+#define ENET_CLKEN 0xc008
+#define ENET_SRST 0xc000
+#define ENET_SHIM 0xc010
+#define CFG_MEM_RAM_SHUTDOWN 0xd070
+#define BLOCK_MEM_RDY 0xd074
+
+#define MEM_RDY 0xffffffff
+#define DEVM_ARAUX_COH BIT(19)
+#define DEVM_AWAUX_COH BIT(3)
+
+#define CFG_FORCE_LINK_STATUS_EN 0x229c
+#define FORCE_LINK_STATUS 0x22a0
+#define CFG_LINK_AGGR_RESUME 0x27c8
+#define RX_DV_GATE_REG 0x2dfc
+
+void xge_wr_csr(struct xge_pdata *pdata, u32 offset, u32 val);
+u32 xge_rd_csr(struct xge_pdata *pdata, u32 offset);
+int xge_port_reset(struct net_device *ndev);
+void xge_port_init(struct net_device *ndev);
+
+#endif /* __XGENE_ENET_V2_ENET__H__ */
diff --git a/drivers/net/ethernet/apm/xgene-v2/ethtool.c b/drivers/net/ethernet/apm/xgene-v2/ethtool.c
new file mode 100644
index 0000000..b6666e4
--- /dev/null
+++ b/drivers/net/ethernet/apm/xgene-v2/ethtool.c
@@ -0,0 +1,187 @@
+/*
+ * Applied Micro X-Gene SoC Ethernet v2 Driver
+ *
+ * Copyright (c) 2017, Applied Micro Circuits Corporation
+ * Author(s): Iyappan Subramanian <isubramanian@apm.com>
+ * Keyur Chudgar <kchudgar@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "main.h"
+
+#define XGE_STAT(m) { #m, offsetof(struct xge_pdata, stats.m) }
+#define XGE_EXTD_STAT(m, n) \
+ { \
+ #m, \
+ n, \
+ 0 \
+ }
+
+static const struct xge_gstrings_stats gstrings_stats[] = {
+ XGE_STAT(rx_packets),
+ XGE_STAT(tx_packets),
+ XGE_STAT(rx_bytes),
+ XGE_STAT(tx_bytes),
+ XGE_STAT(rx_errors)
+};
+
+static struct xge_gstrings_extd_stats gstrings_extd_stats[] = {
+ XGE_EXTD_STAT(tx_rx_64b_frame_cntr, TR64),
+ XGE_EXTD_STAT(tx_rx_127b_frame_cntr, TR127),
+ XGE_EXTD_STAT(tx_rx_255b_frame_cntr, TR255),
+ XGE_EXTD_STAT(tx_rx_511b_frame_cntr, TR511),
+ XGE_EXTD_STAT(tx_rx_1023b_frame_cntr, TR1K),
+ XGE_EXTD_STAT(tx_rx_1518b_frame_cntr, TRMAX),
+ XGE_EXTD_STAT(tx_rx_1522b_frame_cntr, TRMGV),
+ XGE_EXTD_STAT(rx_fcs_error_cntr, RFCS),
+ XGE_EXTD_STAT(rx_multicast_pkt_cntr, RMCA),
+ XGE_EXTD_STAT(rx_broadcast_pkt_cntr, RBCA),
+ XGE_EXTD_STAT(rx_ctrl_frame_pkt_cntr, RXCF),
+ XGE_EXTD_STAT(rx_pause_frame_pkt_cntr, RXPF),
+ XGE_EXTD_STAT(rx_unk_opcode_cntr, RXUO),
+ XGE_EXTD_STAT(rx_align_err_cntr, RALN),
+ XGE_EXTD_STAT(rx_frame_len_err_cntr, RFLR),
+ XGE_EXTD_STAT(rx_code_err_cntr, RCDE),
+ XGE_EXTD_STAT(rx_carrier_sense_err_cntr, RCSE),
+ XGE_EXTD_STAT(rx_undersize_pkt_cntr, RUND),
+ XGE_EXTD_STAT(rx_oversize_pkt_cntr, ROVR),
+ XGE_EXTD_STAT(rx_fragments_cntr, RFRG),
+ XGE_EXTD_STAT(rx_jabber_cntr, RJBR),
+ XGE_EXTD_STAT(rx_dropped_pkt_cntr, RDRP),
+ XGE_EXTD_STAT(tx_multicast_pkt_cntr, TMCA),
+ XGE_EXTD_STAT(tx_broadcast_pkt_cntr, TBCA),
+ XGE_EXTD_STAT(tx_pause_ctrl_frame_cntr, TXPF),
+ XGE_EXTD_STAT(tx_defer_pkt_cntr, TDFR),
+ XGE_EXTD_STAT(tx_excv_defer_pkt_cntr, TEDF),
+ XGE_EXTD_STAT(tx_single_col_pkt_cntr, TSCL),
+ XGE_EXTD_STAT(tx_multi_col_pkt_cntr, TMCL),
+ XGE_EXTD_STAT(tx_late_col_pkt_cntr, TLCL),
+ XGE_EXTD_STAT(tx_excv_col_pkt_cntr, TXCL),
+ XGE_EXTD_STAT(tx_total_col_cntr, TNCL),
+ XGE_EXTD_STAT(tx_pause_frames_hnrd_cntr, TPFH),
+ XGE_EXTD_STAT(tx_drop_frame_cntr, TDRP),
+ XGE_EXTD_STAT(tx_jabber_frame_cntr, TJBR),
+ XGE_EXTD_STAT(tx_fcs_error_cntr, TFCS),
+ XGE_EXTD_STAT(tx_ctrl_frame_cntr, TXCF),
+ XGE_EXTD_STAT(tx_oversize_frame_cntr, TOVR),
+ XGE_EXTD_STAT(tx_undersize_frame_cntr, TUND),
+ XGE_EXTD_STAT(tx_fragments_cntr, TFRG)
+};
+
+#define XGE_STATS_LEN ARRAY_SIZE(gstrings_stats)
+#define XGE_EXTD_STATS_LEN ARRAY_SIZE(gstrings_extd_stats)
+
+static void xge_mac_get_extd_stats(struct xge_pdata *pdata)
+{
+ u32 data;
+ int i;
+
+ for (i = 0; i < XGE_EXTD_STATS_LEN; i++) {
+ data = xge_rd_csr(pdata, gstrings_extd_stats[i].addr);
+ gstrings_extd_stats[i].value += data;
+ }
+}
+
+static void xge_get_drvinfo(struct net_device *ndev,
+ struct ethtool_drvinfo *info)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct platform_device *pdev = pdata->pdev;
+
+ strcpy(info->driver, "xgene-enet-v2");
+ strcpy(info->version, XGENE_ENET_V2_VERSION);
+ snprintf(info->fw_version, ETHTOOL_FWVERS_LEN, "N/A");
+ sprintf(info->bus_info, "%s", pdev->name);
+}
+
+static void xge_get_strings(struct net_device *ndev, u32 stringset, u8 *data)
+{
+ u8 *p = data;
+ int i;
+
+ if (stringset != ETH_SS_STATS)
+ return;
+
+ for (i = 0; i < XGE_STATS_LEN; i++) {
+ memcpy(p, gstrings_stats[i].name, ETH_GSTRING_LEN);
+ p += ETH_GSTRING_LEN;
+ }
+
+ for (i = 0; i < XGE_EXTD_STATS_LEN; i++) {
+ memcpy(p, gstrings_extd_stats[i].name, ETH_GSTRING_LEN);
+ p += ETH_GSTRING_LEN;
+ }
+}
+
+static int xge_get_sset_count(struct net_device *ndev, int sset)
+{
+ if (sset != ETH_SS_STATS)
+ return -EINVAL;
+
+ return XGE_STATS_LEN + XGE_EXTD_STATS_LEN;
+}
+
+static void xge_get_ethtool_stats(struct net_device *ndev,
+ struct ethtool_stats *dummy,
+ u64 *data)
+{
+ void *pdata = netdev_priv(ndev);
+ int i;
+
+ for (i = 0; i < XGE_STATS_LEN; i++)
+ *data++ = *(u64 *)(pdata + gstrings_stats[i].offset);
+
+ xge_mac_get_extd_stats(pdata);
+
+ for (i = 0; i < XGE_EXTD_STATS_LEN; i++)
+ *data++ = gstrings_extd_stats[i].value;
+}
+
+static int xge_get_link_ksettings(struct net_device *ndev,
+ struct ethtool_link_ksettings *cmd)
+{
+ struct phy_device *phydev = ndev->phydev;
+
+ if (!phydev)
+ return -ENODEV;
+
+ return phy_ethtool_ksettings_get(phydev, cmd);
+}
+
+static int xge_set_link_ksettings(struct net_device *ndev,
+ const struct ethtool_link_ksettings *cmd)
+{
+ struct phy_device *phydev = ndev->phydev;
+
+ if (!phydev)
+ return -ENODEV;
+
+ return phy_ethtool_ksettings_set(phydev, cmd);
+}
+
+static const struct ethtool_ops xge_ethtool_ops = {
+ .get_drvinfo = xge_get_drvinfo,
+ .get_link = ethtool_op_get_link,
+ .get_strings = xge_get_strings,
+ .get_sset_count = xge_get_sset_count,
+ .get_ethtool_stats = xge_get_ethtool_stats,
+ .get_link_ksettings = xge_get_link_ksettings,
+ .set_link_ksettings = xge_set_link_ksettings,
+};
+
+void xge_set_ethtool_ops(struct net_device *ndev)
+{
+ ndev->ethtool_ops = &xge_ethtool_ops;
+}
diff --git a/drivers/net/ethernet/apm/xgene-v2/ethtool.h b/drivers/net/ethernet/apm/xgene-v2/ethtool.h
new file mode 100644
index 0000000..54b48d5
--- /dev/null
+++ b/drivers/net/ethernet/apm/xgene-v2/ethtool.h
@@ -0,0 +1,78 @@
+/*
+ * Applied Micro X-Gene SoC Ethernet v2 Driver
+ *
+ * Copyright (c) 2017, Applied Micro Circuits Corporation
+ * Author(s): Iyappan Subramanian <isubramanian@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __XGENE_ENET_V2_ETHTOOL_H__
+#define __XGENE_ENET_V2_ETHTOOL_H__
+
+struct xge_gstrings_stats {
+ char name[ETH_GSTRING_LEN];
+ int offset;
+};
+
+struct xge_gstrings_extd_stats {
+ char name[ETH_GSTRING_LEN];
+ u32 addr;
+ u32 value;
+};
+
+#define TR64 0xa080
+#define TR127 0xa084
+#define TR255 0xa088
+#define TR511 0xa08c
+#define TR1K 0xa090
+#define TRMAX 0xa094
+#define TRMGV 0xa098
+#define RFCS 0xa0a4
+#define RMCA 0xa0a8
+#define RBCA 0xa0ac
+#define RXCF 0xa0b0
+#define RXPF 0xa0b4
+#define RXUO 0xa0b8
+#define RALN 0xa0bc
+#define RFLR 0xa0c0
+#define RCDE 0xa0c4
+#define RCSE 0xa0c8
+#define RUND 0xa0cc
+#define ROVR 0xa0d0
+#define RFRG 0xa0d4
+#define RJBR 0xa0d8
+#define RDRP 0xa0dc
+#define TMCA 0xa0e8
+#define TBCA 0xa0ec
+#define TXPF 0xa0f0
+#define TDFR 0xa0f4
+#define TEDF 0xa0f8
+#define TSCL 0xa0fc
+#define TMCL 0xa100
+#define TLCL 0xa104
+#define TXCL 0xa108
+#define TNCL 0xa10c
+#define TPFH 0xa110
+#define TDRP 0xa114
+#define TJBR 0xa118
+#define TFCS 0xa11c
+#define TXCF 0xa120
+#define TOVR 0xa124
+#define TUND 0xa128
+#define TFRG 0xa12c
+
+void xge_set_ethtool_ops(struct net_device *ndev);
+
+#endif /* __XGENE_ENET_V2_ETHTOOL_H__ */
diff --git a/drivers/net/ethernet/apm/xgene-v2/mac.c b/drivers/net/ethernet/apm/xgene-v2/mac.c
new file mode 100644
index 0000000..ee431e3
--- /dev/null
+++ b/drivers/net/ethernet/apm/xgene-v2/mac.c
@@ -0,0 +1,116 @@
+/*
+ * Applied Micro X-Gene SoC Ethernet v2 Driver
+ *
+ * Copyright (c) 2017, Applied Micro Circuits Corporation
+ * Author(s): Iyappan Subramanian <isubramanian@apm.com>
+ * Keyur Chudgar <kchudgar@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "main.h"
+
+void xge_mac_reset(struct xge_pdata *pdata)
+{
+ xge_wr_csr(pdata, MAC_CONFIG_1, SOFT_RESET);
+ xge_wr_csr(pdata, MAC_CONFIG_1, 0);
+}
+
+void xge_mac_set_speed(struct xge_pdata *pdata)
+{
+ u32 icm0, icm2, ecm0, mc2;
+ u32 intf_ctrl, rgmii;
+
+ icm0 = xge_rd_csr(pdata, ICM_CONFIG0_REG_0);
+ icm2 = xge_rd_csr(pdata, ICM_CONFIG2_REG_0);
+ ecm0 = xge_rd_csr(pdata, ECM_CONFIG0_REG_0);
+ rgmii = xge_rd_csr(pdata, RGMII_REG_0);
+ mc2 = xge_rd_csr(pdata, MAC_CONFIG_2);
+ intf_ctrl = xge_rd_csr(pdata, INTERFACE_CONTROL);
+ icm2 |= CFG_WAITASYNCRD_EN;
+
+ switch (pdata->phy_speed) {
+ case SPEED_10:
+ SET_REG_BITS(&mc2, INTF_MODE, 1);
+ SET_REG_BITS(&intf_ctrl, HD_MODE, 0);
+ SET_REG_BITS(&icm0, CFG_MACMODE, 0);
+ SET_REG_BITS(&icm2, CFG_WAITASYNCRD, 500);
+ SET_REG_BIT(&rgmii, CFG_SPEED_125, 0);
+ break;
+ case SPEED_100:
+ SET_REG_BITS(&mc2, INTF_MODE, 1);
+ SET_REG_BITS(&intf_ctrl, HD_MODE, 1);
+ SET_REG_BITS(&icm0, CFG_MACMODE, 1);
+ SET_REG_BITS(&icm2, CFG_WAITASYNCRD, 80);
+ SET_REG_BIT(&rgmii, CFG_SPEED_125, 0);
+ break;
+ default:
+ SET_REG_BITS(&mc2, INTF_MODE, 2);
+ SET_REG_BITS(&intf_ctrl, HD_MODE, 2);
+ SET_REG_BITS(&icm0, CFG_MACMODE, 2);
+ SET_REG_BITS(&icm2, CFG_WAITASYNCRD, 16);
+ SET_REG_BIT(&rgmii, CFG_SPEED_125, 1);
+ break;
+ }
+
+ mc2 |= FULL_DUPLEX | CRC_EN | PAD_CRC;
+ SET_REG_BITS(&ecm0, CFG_WFIFOFULLTHR, 0x32);
+
+ xge_wr_csr(pdata, MAC_CONFIG_2, mc2);
+ xge_wr_csr(pdata, INTERFACE_CONTROL, intf_ctrl);
+ xge_wr_csr(pdata, RGMII_REG_0, rgmii);
+ xge_wr_csr(pdata, ICM_CONFIG0_REG_0, icm0);
+ xge_wr_csr(pdata, ICM_CONFIG2_REG_0, icm2);
+ xge_wr_csr(pdata, ECM_CONFIG0_REG_0, ecm0);
+}
+
+void xge_mac_set_station_addr(struct xge_pdata *pdata)
+{
+ u8 *dev_addr = pdata->ndev->dev_addr;
+ u32 addr0, addr1;
+
+ addr0 = (dev_addr[3] << 24) | (dev_addr[2] << 16) |
+ (dev_addr[1] << 8) | dev_addr[0];
+ addr1 = (dev_addr[5] << 24) | (dev_addr[4] << 16);
+
+ xge_wr_csr(pdata, STATION_ADDR0, addr0);
+ xge_wr_csr(pdata, STATION_ADDR1, addr1);
+}
+
+void xge_mac_init(struct xge_pdata *pdata)
+{
+ xge_mac_reset(pdata);
+ xge_mac_set_speed(pdata);
+ xge_mac_set_station_addr(pdata);
+}
+
+void xge_mac_enable(struct xge_pdata *pdata)
+{
+ u32 data;
+
+ data = xge_rd_csr(pdata, MAC_CONFIG_1);
+ data |= TX_EN | RX_EN;
+ xge_wr_csr(pdata, MAC_CONFIG_1, data);
+
+ data = xge_rd_csr(pdata, MAC_CONFIG_1);
+}
+
+void xge_mac_disable(struct xge_pdata *pdata)
+{
+ u32 data;
+
+ data = xge_rd_csr(pdata, MAC_CONFIG_1);
+ data &= ~(TX_EN | RX_EN);
+ xge_wr_csr(pdata, MAC_CONFIG_1, data);
+}
diff --git a/drivers/net/ethernet/apm/xgene-v2/mac.h b/drivers/net/ethernet/apm/xgene-v2/mac.h
new file mode 100644
index 0000000..3c83fa6
--- /dev/null
+++ b/drivers/net/ethernet/apm/xgene-v2/mac.h
@@ -0,0 +1,107 @@
+/*
+ * Applied Micro X-Gene SoC Ethernet v2 Driver
+ *
+ * Copyright (c) 2017, Applied Micro Circuits Corporation
+ * Author(s): Iyappan Subramanian <isubramanian@apm.com>
+ * Keyur Chudgar <kchudgar@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __XGENE_ENET_V2_MAC_H__
+#define __XGENE_ENET_V2_MAC_H__
+
+/* Register offsets */
+#define MAC_CONFIG_1 0xa000
+#define MAC_CONFIG_2 0xa004
+#define MII_MGMT_CONFIG 0xa020
+#define MII_MGMT_COMMAND 0xa024
+#define MII_MGMT_ADDRESS 0xa028
+#define MII_MGMT_CONTROL 0xa02c
+#define MII_MGMT_STATUS 0xa030
+#define MII_MGMT_INDICATORS 0xa034
+#define INTERFACE_CONTROL 0xa038
+#define STATION_ADDR0 0xa040
+#define STATION_ADDR1 0xa044
+
+#define RGMII_REG_0 0x27e0
+#define ICM_CONFIG0_REG_0 0x2c00
+#define ICM_CONFIG2_REG_0 0x2c08
+#define ECM_CONFIG0_REG_0 0x2d00
+
+/* Register fields */
+#define SOFT_RESET BIT(31)
+#define TX_EN BIT(0)
+#define RX_EN BIT(2)
+#define PAD_CRC BIT(2)
+#define CRC_EN BIT(1)
+#define FULL_DUPLEX BIT(0)
+
+#define INTF_MODE_POS 8
+#define INTF_MODE_LEN 2
+#define HD_MODE_POS 25
+#define HD_MODE_LEN 2
+#define CFG_MACMODE_POS 18
+#define CFG_MACMODE_LEN 2
+#define CFG_WAITASYNCRD_POS 0
+#define CFG_WAITASYNCRD_LEN 16
+#define CFG_SPEED_125_POS 24
+#define CFG_WFIFOFULLTHR_POS 0
+#define CFG_WFIFOFULLTHR_LEN 7
+#define MGMT_CLOCK_SEL_POS 0
+#define MGMT_CLOCK_SEL_LEN 3
+#define PHY_ADDR_POS 8
+#define PHY_ADDR_LEN 5
+#define REG_ADDR_POS 0
+#define REG_ADDR_LEN 5
+#define MII_MGMT_BUSY BIT(0)
+#define MII_READ_CYCLE BIT(0)
+#define CFG_WAITASYNCRD_EN BIT(16)
+
+static inline void xgene_set_reg_bits(u32 *var, int pos, int len, u32 val)
+{
+ u32 mask = GENMASK(pos + len, pos);
+
+ *var &= ~mask;
+ *var |= ((val << pos) & mask);
+}
+
+static inline u32 xgene_get_reg_bits(u32 var, int pos, int len)
+{
+ u32 mask = GENMASK(pos + len, pos);
+
+ return (var & mask) >> pos;
+}
+
+#define SET_REG_BITS(var, field, val) \
+ xgene_set_reg_bits(var, field ## _POS, field ## _LEN, val)
+
+#define SET_REG_BIT(var, field, val) \
+ xgene_set_reg_bits(var, field ## _POS, 1, val)
+
+#define GET_REG_BITS(var, field) \
+ xgene_get_reg_bits(var, field ## _POS, field ## _LEN)
+
+#define GET_REG_BIT(var, field) ((var) & (field))
+
+struct xge_pdata;
+
+void xge_mac_reset(struct xge_pdata *pdata);
+void xge_mac_set_speed(struct xge_pdata *pdata);
+void xge_mac_enable(struct xge_pdata *pdata);
+void xge_mac_disable(struct xge_pdata *pdata);
+void xge_mac_init(struct xge_pdata *pdata);
+void xge_mac_set_station_addr(struct xge_pdata *pdata);
+
+#endif /* __XGENE_ENET_V2_MAC_H__ */
diff --git a/drivers/net/ethernet/apm/xgene-v2/main.c b/drivers/net/ethernet/apm/xgene-v2/main.c
new file mode 100644
index 0000000..0f2ad50
--- /dev/null
+++ b/drivers/net/ethernet/apm/xgene-v2/main.c
@@ -0,0 +1,759 @@
+/*
+ * Applied Micro X-Gene SoC Ethernet v2 Driver
+ *
+ * Copyright (c) 2017, Applied Micro Circuits Corporation
+ * Author(s): Iyappan Subramanian <isubramanian@apm.com>
+ * Keyur Chudgar <kchudgar@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "main.h"
+
+static const struct acpi_device_id xge_acpi_match[];
+
+static int xge_get_resources(struct xge_pdata *pdata)
+{
+ struct platform_device *pdev;
+ struct net_device *ndev;
+ int phy_mode, ret = 0;
+ struct resource *res;
+ struct device *dev;
+
+ pdev = pdata->pdev;
+ dev = &pdev->dev;
+ ndev = pdata->ndev;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!res) {
+ dev_err(dev, "Resource enet_csr not defined\n");
+ return -ENODEV;
+ }
+
+ pdata->resources.base_addr = devm_ioremap(dev, res->start,
+ resource_size(res));
+ if (!pdata->resources.base_addr) {
+ dev_err(dev, "Unable to retrieve ENET Port CSR region\n");
+ return -ENOMEM;
+ }
+
+ if (!device_get_mac_address(dev, ndev->dev_addr, ETH_ALEN))
+ eth_hw_addr_random(ndev);
+
+ memcpy(ndev->perm_addr, ndev->dev_addr, ndev->addr_len);
+
+ phy_mode = device_get_phy_mode(dev);
+ if (phy_mode < 0) {
+ dev_err(dev, "Unable to get phy-connection-type\n");
+ return phy_mode;
+ }
+ pdata->resources.phy_mode = phy_mode;
+
+ if (pdata->resources.phy_mode != PHY_INTERFACE_MODE_RGMII) {
+ dev_err(dev, "Incorrect phy-connection-type specified\n");
+ return -ENODEV;
+ }
+
+ ret = platform_get_irq(pdev, 0);
+ if (ret < 0) {
+ dev_err(dev, "Unable to get irq\n");
+ return ret;
+ }
+ pdata->resources.irq = ret;
+
+ return 0;
+}
+
+static int xge_refill_buffers(struct net_device *ndev, u32 nbuf)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct xge_desc_ring *ring = pdata->rx_ring;
+ const u8 slots = XGENE_ENET_NUM_DESC - 1;
+ struct device *dev = &pdata->pdev->dev;
+ struct xge_raw_desc *raw_desc;
+ u64 addr_lo, addr_hi;
+ u8 tail = ring->tail;
+ struct sk_buff *skb;
+ dma_addr_t dma_addr;
+ u16 len;
+ int i;
+
+ for (i = 0; i < nbuf; i++) {
+ raw_desc = &ring->raw_desc[tail];
+
+ len = XGENE_ENET_STD_MTU;
+ skb = netdev_alloc_skb(ndev, len);
+ if (unlikely(!skb))
+ return -ENOMEM;
+
+ dma_addr = dma_map_single(dev, skb->data, len, DMA_FROM_DEVICE);
+ if (dma_mapping_error(dev, dma_addr)) {
+ netdev_err(ndev, "DMA mapping error\n");
+ dev_kfree_skb_any(skb);
+ return -EINVAL;
+ }
+
+ ring->pkt_info[tail].skb = skb;
+ ring->pkt_info[tail].dma_addr = dma_addr;
+
+ addr_hi = GET_BITS(NEXT_DESC_ADDRH, le64_to_cpu(raw_desc->m1));
+ addr_lo = GET_BITS(NEXT_DESC_ADDRL, le64_to_cpu(raw_desc->m1));
+ raw_desc->m1 = cpu_to_le64(SET_BITS(NEXT_DESC_ADDRL, addr_lo) |
+ SET_BITS(NEXT_DESC_ADDRH, addr_hi) |
+ SET_BITS(PKT_ADDRH,
+ upper_32_bits(dma_addr)));
+
+ dma_wmb();
+ raw_desc->m0 = cpu_to_le64(SET_BITS(PKT_ADDRL, dma_addr) |
+ SET_BITS(E, 1));
+ tail = (tail + 1) & slots;
+ }
+
+ ring->tail = tail;
+
+ return 0;
+}
+
+static int xge_init_hw(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ int ret;
+
+ ret = xge_port_reset(ndev);
+ if (ret)
+ return ret;
+
+ xge_port_init(ndev);
+ pdata->nbufs = NUM_BUFS;
+
+ return 0;
+}
+
+static irqreturn_t xge_irq(const int irq, void *data)
+{
+ struct xge_pdata *pdata = data;
+
+ if (napi_schedule_prep(&pdata->napi)) {
+ xge_intr_disable(pdata);
+ __napi_schedule(&pdata->napi);
+ }
+
+ return IRQ_HANDLED;
+}
+
+static int xge_request_irq(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ int ret;
+
+ snprintf(pdata->irq_name, IRQ_ID_SIZE, "%s", ndev->name);
+
+ ret = request_irq(pdata->resources.irq, xge_irq, 0, pdata->irq_name,
+ pdata);
+ if (ret)
+ netdev_err(ndev, "Failed to request irq %s\n", pdata->irq_name);
+
+ return ret;
+}
+
+static void xge_free_irq(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+
+ free_irq(pdata->resources.irq, pdata);
+}
+
+static bool is_tx_slot_available(struct xge_raw_desc *raw_desc)
+{
+ if (GET_BITS(E, le64_to_cpu(raw_desc->m0)) &&
+ (GET_BITS(PKT_SIZE, le64_to_cpu(raw_desc->m0)) == SLOT_EMPTY))
+ return true;
+
+ return false;
+}
+
+static netdev_tx_t xge_start_xmit(struct sk_buff *skb, struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct device *dev = &pdata->pdev->dev;
+ struct xge_desc_ring *tx_ring;
+ struct xge_raw_desc *raw_desc;
+ static dma_addr_t dma_addr;
+ u64 addr_lo, addr_hi;
+ void *pkt_buf;
+ u8 tail;
+ u16 len;
+
+ tx_ring = pdata->tx_ring;
+ tail = tx_ring->tail;
+ len = skb_headlen(skb);
+ raw_desc = &tx_ring->raw_desc[tail];
+
+ if (!is_tx_slot_available(raw_desc)) {
+ netif_stop_queue(ndev);
+ return NETDEV_TX_BUSY;
+ }
+
+ /* Packet buffers should be 64B aligned */
+ pkt_buf = dma_zalloc_coherent(dev, XGENE_ENET_STD_MTU, &dma_addr,
+ GFP_ATOMIC);
+ if (unlikely(!pkt_buf)) {
+ dev_kfree_skb_any(skb);
+ return NETDEV_TX_OK;
+ }
+ memcpy(pkt_buf, skb->data, len);
+
+ addr_hi = GET_BITS(NEXT_DESC_ADDRH, le64_to_cpu(raw_desc->m1));
+ addr_lo = GET_BITS(NEXT_DESC_ADDRL, le64_to_cpu(raw_desc->m1));
+ raw_desc->m1 = cpu_to_le64(SET_BITS(NEXT_DESC_ADDRL, addr_lo) |
+ SET_BITS(NEXT_DESC_ADDRH, addr_hi) |
+ SET_BITS(PKT_ADDRH,
+ upper_32_bits(dma_addr)));
+
+ tx_ring->pkt_info[tail].skb = skb;
+ tx_ring->pkt_info[tail].dma_addr = dma_addr;
+ tx_ring->pkt_info[tail].pkt_buf = pkt_buf;
+
+ dma_wmb();
+
+ raw_desc->m0 = cpu_to_le64(SET_BITS(PKT_ADDRL, dma_addr) |
+ SET_BITS(PKT_SIZE, len) |
+ SET_BITS(E, 0));
+ skb_tx_timestamp(skb);
+ xge_wr_csr(pdata, DMATXCTRL, 1);
+
+ tx_ring->tail = (tail + 1) & (XGENE_ENET_NUM_DESC - 1);
+
+ return NETDEV_TX_OK;
+}
+
+static bool is_tx_hw_done(struct xge_raw_desc *raw_desc)
+{
+ if (GET_BITS(E, le64_to_cpu(raw_desc->m0)) &&
+ !GET_BITS(PKT_SIZE, le64_to_cpu(raw_desc->m0)))
+ return true;
+
+ return false;
+}
+
+static void xge_txc_poll(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct device *dev = &pdata->pdev->dev;
+ struct xge_desc_ring *tx_ring;
+ struct xge_raw_desc *raw_desc;
+ dma_addr_t dma_addr;
+ struct sk_buff *skb;
+ void *pkt_buf;
+ u32 data;
+ u8 head;
+
+ tx_ring = pdata->tx_ring;
+ head = tx_ring->head;
+
+ data = xge_rd_csr(pdata, DMATXSTATUS);
+ if (!GET_BITS(TXPKTCOUNT, data))
+ return;
+
+ while (1) {
+ raw_desc = &tx_ring->raw_desc[head];
+
+ if (!is_tx_hw_done(raw_desc))
+ break;
+
+ dma_rmb();
+
+ skb = tx_ring->pkt_info[head].skb;
+ dma_addr = tx_ring->pkt_info[head].dma_addr;
+ pkt_buf = tx_ring->pkt_info[head].pkt_buf;
+ pdata->stats.tx_packets++;
+ pdata->stats.tx_bytes += skb->len;
+ dma_free_coherent(dev, XGENE_ENET_STD_MTU, pkt_buf, dma_addr);
+ dev_kfree_skb_any(skb);
+
+ /* clear pktstart address and pktsize */
+ raw_desc->m0 = cpu_to_le64(SET_BITS(E, 1) |
+ SET_BITS(PKT_SIZE, SLOT_EMPTY));
+ xge_wr_csr(pdata, DMATXSTATUS, 1);
+
+ head = (head + 1) & (XGENE_ENET_NUM_DESC - 1);
+ }
+
+ if (netif_queue_stopped(ndev))
+ netif_wake_queue(ndev);
+
+ tx_ring->head = head;
+}
+
+static int xge_rx_poll(struct net_device *ndev, unsigned int budget)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct device *dev = &pdata->pdev->dev;
+ struct xge_desc_ring *rx_ring;
+ struct xge_raw_desc *raw_desc;
+ struct sk_buff *skb;
+ dma_addr_t dma_addr;
+ int processed = 0;
+ u8 head, rx_error;
+ int i, ret;
+ u32 data;
+ u16 len;
+
+ rx_ring = pdata->rx_ring;
+ head = rx_ring->head;
+
+ data = xge_rd_csr(pdata, DMARXSTATUS);
+ if (!GET_BITS(RXPKTCOUNT, data))
+ return 0;
+
+ for (i = 0; i < budget; i++) {
+ raw_desc = &rx_ring->raw_desc[head];
+
+ if (GET_BITS(E, le64_to_cpu(raw_desc->m0)))
+ break;
+
+ dma_rmb();
+
+ skb = rx_ring->pkt_info[head].skb;
+ rx_ring->pkt_info[head].skb = NULL;
+ dma_addr = rx_ring->pkt_info[head].dma_addr;
+ len = GET_BITS(PKT_SIZE, le64_to_cpu(raw_desc->m0));
+ dma_unmap_single(dev, dma_addr, XGENE_ENET_STD_MTU,
+ DMA_FROM_DEVICE);
+
+ rx_error = GET_BITS(D, le64_to_cpu(raw_desc->m2));
+ if (unlikely(rx_error)) {
+ pdata->stats.rx_errors++;
+ dev_kfree_skb_any(skb);
+ goto out;
+ }
+
+ skb_put(skb, len);
+ skb->protocol = eth_type_trans(skb, ndev);
+
+ pdata->stats.rx_packets++;
+ pdata->stats.rx_bytes += len;
+ napi_gro_receive(&pdata->napi, skb);
+out:
+ ret = xge_refill_buffers(ndev, 1);
+ xge_wr_csr(pdata, DMARXSTATUS, 1);
+ xge_wr_csr(pdata, DMARXCTRL, 1);
+
+ if (ret)
+ break;
+
+ head = (head + 1) & (XGENE_ENET_NUM_DESC - 1);
+ processed++;
+ }
+
+ rx_ring->head = head;
+
+ return processed;
+}
+
+static void xge_delete_desc_ring(struct net_device *ndev,
+ struct xge_desc_ring *ring)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct device *dev = &pdata->pdev->dev;
+ u16 size;
+
+ if (!ring)
+ return;
+
+ size = XGENE_ENET_DESC_SIZE * XGENE_ENET_NUM_DESC;
+ if (ring->desc_addr)
+ dma_free_coherent(dev, size, ring->desc_addr, ring->dma_addr);
+
+ kfree(ring->pkt_info);
+ kfree(ring);
+}
+
+static void xge_free_buffers(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct xge_desc_ring *ring = pdata->rx_ring;
+ struct device *dev = &pdata->pdev->dev;
+ struct sk_buff *skb;
+ dma_addr_t dma_addr;
+ int i;
+
+ for (i = 0; i < XGENE_ENET_NUM_DESC; i++) {
+ skb = ring->pkt_info[i].skb;
+ dma_addr = ring->pkt_info[i].dma_addr;
+
+ if (!skb)
+ continue;
+
+ dma_unmap_single(dev, dma_addr, XGENE_ENET_STD_MTU,
+ DMA_FROM_DEVICE);
+ dev_kfree_skb_any(skb);
+ }
+}
+
+static void xge_delete_desc_rings(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+
+ xge_txc_poll(ndev);
+ xge_delete_desc_ring(ndev, pdata->tx_ring);
+
+ xge_rx_poll(ndev, 64);
+ xge_free_buffers(ndev);
+ xge_delete_desc_ring(ndev, pdata->rx_ring);
+}
+
+static struct xge_desc_ring *xge_create_desc_ring(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct device *dev = &pdata->pdev->dev;
+ struct xge_desc_ring *ring;
+ u16 size;
+
+ ring = kzalloc(sizeof(*ring), GFP_KERNEL);
+ if (!ring)
+ return NULL;
+
+ ring->ndev = ndev;
+
+ size = XGENE_ENET_DESC_SIZE * XGENE_ENET_NUM_DESC;
+ ring->desc_addr = dma_zalloc_coherent(dev, size, &ring->dma_addr,
+ GFP_KERNEL);
+ if (!ring->desc_addr)
+ goto err;
+
+ ring->pkt_info = kcalloc(XGENE_ENET_NUM_DESC, sizeof(*ring->pkt_info),
+ GFP_KERNEL);
+ if (!ring->pkt_info)
+ goto err;
+
+ xge_setup_desc(ring);
+
+ return ring;
+
+err:
+ xge_delete_desc_ring(ndev, ring);
+
+ return NULL;
+}
+
+static int xge_create_desc_rings(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct xge_desc_ring *ring;
+ int ret;
+
+ /* create tx ring */
+ ring = xge_create_desc_ring(ndev);
+ if (!ring)
+ goto err;
+
+ pdata->tx_ring = ring;
+ xge_update_tx_desc_addr(pdata);
+
+ /* create rx ring */
+ ring = xge_create_desc_ring(ndev);
+ if (!ring)
+ goto err;
+
+ pdata->rx_ring = ring;
+ xge_update_rx_desc_addr(pdata);
+
+ ret = xge_refill_buffers(ndev, XGENE_ENET_NUM_DESC);
+ if (ret)
+ goto err;
+
+ return 0;
+err:
+ xge_delete_desc_rings(ndev);
+
+ return -ENOMEM;
+}
+
+static int xge_open(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ int ret;
+
+ ret = xge_create_desc_rings(ndev);
+ if (ret)
+ return ret;
+
+ napi_enable(&pdata->napi);
+ ret = xge_request_irq(ndev);
+ if (ret)
+ return ret;
+
+ xge_intr_enable(pdata);
+ xge_wr_csr(pdata, DMARXCTRL, 1);
+
+ phy_start(ndev->phydev);
+ xge_mac_enable(pdata);
+ netif_start_queue(ndev);
+
+ return 0;
+}
+
+static int xge_close(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+
+ netif_stop_queue(ndev);
+ xge_mac_disable(pdata);
+ phy_stop(ndev->phydev);
+
+ xge_intr_disable(pdata);
+ xge_free_irq(ndev);
+ napi_disable(&pdata->napi);
+ xge_delete_desc_rings(ndev);
+
+ return 0;
+}
+
+static int xge_napi(struct napi_struct *napi, const int budget)
+{
+ struct net_device *ndev = napi->dev;
+ struct xge_pdata *pdata;
+ int processed;
+
+ pdata = netdev_priv(ndev);
+
+ xge_txc_poll(ndev);
+ processed = xge_rx_poll(ndev, budget);
+
+ if (processed < budget) {
+ napi_complete_done(napi, processed);
+ xge_intr_enable(pdata);
+ }
+
+ return processed;
+}
+
+static int xge_set_mac_addr(struct net_device *ndev, void *addr)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ int ret;
+
+ ret = eth_mac_addr(ndev, addr);
+ if (ret)
+ return ret;
+
+ xge_mac_set_station_addr(pdata);
+
+ return 0;
+}
+
+static bool is_tx_pending(struct xge_raw_desc *raw_desc)
+{
+ if (!GET_BITS(E, le64_to_cpu(raw_desc->m0)))
+ return true;
+
+ return false;
+}
+
+static void xge_free_pending_skb(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct device *dev = &pdata->pdev->dev;
+ struct xge_desc_ring *tx_ring;
+ struct xge_raw_desc *raw_desc;
+ dma_addr_t dma_addr;
+ struct sk_buff *skb;
+ void *pkt_buf;
+ int i;
+
+ tx_ring = pdata->tx_ring;
+
+ for (i = 0; i < XGENE_ENET_NUM_DESC; i++) {
+ raw_desc = &tx_ring->raw_desc[i];
+
+ if (!is_tx_pending(raw_desc))
+ continue;
+
+ skb = tx_ring->pkt_info[i].skb;
+ dma_addr = tx_ring->pkt_info[i].dma_addr;
+ pkt_buf = tx_ring->pkt_info[i].pkt_buf;
+ dma_free_coherent(dev, XGENE_ENET_STD_MTU, pkt_buf, dma_addr);
+ dev_kfree_skb_any(skb);
+ }
+}
+
+static void xge_timeout(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+
+ rtnl_lock();
+
+ if (!netif_running(ndev))
+ goto out;
+
+ netif_stop_queue(ndev);
+ xge_intr_disable(pdata);
+ napi_disable(&pdata->napi);
+
+ xge_wr_csr(pdata, DMATXCTRL, 0);
+ xge_txc_poll(ndev);
+ xge_free_pending_skb(ndev);
+ xge_wr_csr(pdata, DMATXSTATUS, ~0U);
+
+ xge_setup_desc(pdata->tx_ring);
+ xge_update_tx_desc_addr(pdata);
+ xge_mac_init(pdata);
+
+ napi_enable(&pdata->napi);
+ xge_intr_enable(pdata);
+ xge_mac_enable(pdata);
+ netif_start_queue(ndev);
+
+out:
+ rtnl_unlock();
+}
+
+static void xge_get_stats64(struct net_device *ndev,
+ struct rtnl_link_stats64 *storage)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct xge_stats *stats = &pdata->stats;
+
+ storage->tx_packets += stats->tx_packets;
+ storage->tx_bytes += stats->tx_bytes;
+
+ storage->rx_packets += stats->rx_packets;
+ storage->rx_bytes += stats->rx_bytes;
+ storage->rx_errors += stats->rx_errors;
+}
+
+static const struct net_device_ops xgene_ndev_ops = {
+ .ndo_open = xge_open,
+ .ndo_stop = xge_close,
+ .ndo_start_xmit = xge_start_xmit,
+ .ndo_set_mac_address = xge_set_mac_addr,
+ .ndo_tx_timeout = xge_timeout,
+ .ndo_get_stats64 = xge_get_stats64,
+};
+
+static int xge_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct net_device *ndev;
+ struct xge_pdata *pdata;
+ int ret;
+
+ ndev = alloc_etherdev(sizeof(*pdata));
+ if (!ndev)
+ return -ENOMEM;
+
+ pdata = netdev_priv(ndev);
+
+ pdata->pdev = pdev;
+ pdata->ndev = ndev;
+ SET_NETDEV_DEV(ndev, dev);
+ platform_set_drvdata(pdev, pdata);
+ ndev->netdev_ops = &xgene_ndev_ops;
+
+ ndev->features |= NETIF_F_GSO |
+ NETIF_F_GRO;
+
+ ret = xge_get_resources(pdata);
+ if (ret)
+ goto err;
+
+ ndev->hw_features = ndev->features;
+ xge_set_ethtool_ops(ndev);
+
+ ret = dma_coerce_mask_and_coherent(dev, DMA_BIT_MASK(64));
+ if (ret) {
+ netdev_err(ndev, "No usable DMA configuration\n");
+ goto err;
+ }
+
+ ret = xge_init_hw(ndev);
+ if (ret)
+ goto err;
+
+ ret = xge_mdio_config(ndev);
+ if (ret)
+ goto err;
+
+ netif_napi_add(ndev, &pdata->napi, xge_napi, NAPI_POLL_WEIGHT);
+
+ ret = register_netdev(ndev);
+ if (ret) {
+ netdev_err(ndev, "Failed to register netdev\n");
+ goto err;
+ }
+
+ return 0;
+
+err:
+ free_netdev(ndev);
+
+ return ret;
+}
+
+static int xge_remove(struct platform_device *pdev)
+{
+ struct xge_pdata *pdata;
+ struct net_device *ndev;
+
+ pdata = platform_get_drvdata(pdev);
+ ndev = pdata->ndev;
+
+ rtnl_lock();
+ if (netif_running(ndev))
+ dev_close(ndev);
+ rtnl_unlock();
+
+ xge_mdio_remove(ndev);
+ unregister_netdev(ndev);
+ free_netdev(ndev);
+
+ return 0;
+}
+
+static void xge_shutdown(struct platform_device *pdev)
+{
+ struct xge_pdata *pdata;
+
+ pdata = platform_get_drvdata(pdev);
+ if (!pdata)
+ return;
+
+ if (!pdata->ndev)
+ return;
+
+ xge_remove(pdev);
+}
+
+static const struct acpi_device_id xge_acpi_match[] = {
+ { "APMC0D80" },
+ { }
+};
+MODULE_DEVICE_TABLE(acpi, xge_acpi_match);
+
+static struct platform_driver xge_driver = {
+ .driver = {
+ .name = "xgene-enet-v2",
+ .acpi_match_table = ACPI_PTR(xge_acpi_match),
+ },
+ .probe = xge_probe,
+ .remove = xge_remove,
+ .shutdown = xge_shutdown,
+};
+module_platform_driver(xge_driver);
+
+MODULE_DESCRIPTION("APM X-Gene SoC Ethernet v2 driver");
+MODULE_AUTHOR("Iyappan Subramanian <isubramanian@apm.com>");
+MODULE_VERSION(XGENE_ENET_V2_VERSION);
+MODULE_LICENSE("GPL");
diff --git a/drivers/net/ethernet/apm/xgene-v2/main.h b/drivers/net/ethernet/apm/xgene-v2/main.h
new file mode 100644
index 0000000..969b258
--- /dev/null
+++ b/drivers/net/ethernet/apm/xgene-v2/main.h
@@ -0,0 +1,80 @@
+/*
+ * Applied Micro X-Gene SoC Ethernet v2 Driver
+ *
+ * Copyright (c) 2017, Applied Micro Circuits Corporation
+ * Author(s): Iyappan Subramanian <isubramanian@apm.com>
+ * Keyur Chudgar <kchudgar@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __XGENE_ENET_V2_MAIN_H__
+#define __XGENE_ENET_V2_MAIN_H__
+
+#include <linux/acpi.h>
+#include <linux/clk.h>
+#include <linux/efi.h>
+#include <linux/if_vlan.h>
+#include <linux/irq.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/of_platform.h>
+#include <linux/of_net.h>
+#include <linux/of_mdio.h>
+#include <linux/prefetch.h>
+#include <linux/phy.h>
+#include <net/ip.h>
+#include "mac.h"
+#include "enet.h"
+#include "ring.h"
+#include "ethtool.h"
+
+#define XGENE_ENET_V2_VERSION "v1.0"
+#define XGENE_ENET_STD_MTU 1536
+#define XGENE_ENET_MIN_FRAME 60
+#define IRQ_ID_SIZE 16
+
+struct xge_resource {
+ void __iomem *base_addr;
+ int phy_mode;
+ u32 irq;
+};
+
+struct xge_stats {
+ u64 tx_packets;
+ u64 tx_bytes;
+ u64 rx_packets;
+ u64 rx_bytes;
+ u64 rx_errors;
+};
+
+/* ethernet private data */
+struct xge_pdata {
+ struct xge_resource resources;
+ struct xge_desc_ring *tx_ring;
+ struct xge_desc_ring *rx_ring;
+ struct platform_device *pdev;
+ char irq_name[IRQ_ID_SIZE];
+ struct mii_bus *mdio_bus;
+ struct net_device *ndev;
+ struct napi_struct napi;
+ struct xge_stats stats;
+ int phy_speed;
+ u8 nbufs;
+};
+
+int xge_mdio_config(struct net_device *ndev);
+void xge_mdio_remove(struct net_device *ndev);
+
+#endif /* __XGENE_ENET_V2_MAIN_H__ */
diff --git a/drivers/net/ethernet/apm/xgene-v2/mdio.c b/drivers/net/ethernet/apm/xgene-v2/mdio.c
new file mode 100644
index 0000000..f5fe3bb
--- /dev/null
+++ b/drivers/net/ethernet/apm/xgene-v2/mdio.c
@@ -0,0 +1,168 @@
+/*
+ * Applied Micro X-Gene SoC Ethernet v2 Driver
+ *
+ * Copyright (c) 2017, Applied Micro Circuits Corporation
+ * Author(s): Iyappan Subramanian <isubramanian@apm.com>
+ * Keyur Chudgar <kchudgar@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "main.h"
+
+static int xge_mdio_write(struct mii_bus *bus, int phy_id, int reg, u16 data)
+{
+ struct xge_pdata *pdata = bus->priv;
+ u32 done, val = 0;
+ u8 wait = 10;
+
+ SET_REG_BITS(&val, PHY_ADDR, phy_id);
+ SET_REG_BITS(&val, REG_ADDR, reg);
+ xge_wr_csr(pdata, MII_MGMT_ADDRESS, val);
+
+ xge_wr_csr(pdata, MII_MGMT_CONTROL, data);
+ do {
+ usleep_range(5, 10);
+ done = xge_rd_csr(pdata, MII_MGMT_INDICATORS);
+ } while ((done & MII_MGMT_BUSY) && wait--);
+
+ if (done & MII_MGMT_BUSY) {
+ dev_err(&bus->dev, "MII_MGMT write failed\n");
+ return -ETIMEDOUT;
+ }
+
+ return 0;
+}
+
+static int xge_mdio_read(struct mii_bus *bus, int phy_id, int reg)
+{
+ struct xge_pdata *pdata = bus->priv;
+ u32 data, done, val = 0;
+ u8 wait = 10;
+
+ SET_REG_BITS(&val, PHY_ADDR, phy_id);
+ SET_REG_BITS(&val, REG_ADDR, reg);
+ xge_wr_csr(pdata, MII_MGMT_ADDRESS, val);
+
+ xge_wr_csr(pdata, MII_MGMT_COMMAND, MII_READ_CYCLE);
+ do {
+ usleep_range(5, 10);
+ done = xge_rd_csr(pdata, MII_MGMT_INDICATORS);
+ } while ((done & MII_MGMT_BUSY) && wait--);
+
+ if (done & MII_MGMT_BUSY) {
+ dev_err(&bus->dev, "MII_MGMT read failed\n");
+ return -ETIMEDOUT;
+ }
+
+ data = xge_rd_csr(pdata, MII_MGMT_STATUS);
+ xge_wr_csr(pdata, MII_MGMT_COMMAND, 0);
+
+ return data;
+}
+
+static void xge_adjust_link(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct phy_device *phydev = ndev->phydev;
+
+ if (phydev->link) {
+ if (pdata->phy_speed != phydev->speed) {
+ pdata->phy_speed = phydev->speed;
+ xge_mac_set_speed(pdata);
+ xge_mac_enable(pdata);
+ phy_print_status(phydev);
+ }
+ } else {
+ if (pdata->phy_speed != SPEED_UNKNOWN) {
+ pdata->phy_speed = SPEED_UNKNOWN;
+ xge_mac_disable(pdata);
+ phy_print_status(phydev);
+ }
+ }
+}
+
+void xge_mdio_remove(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct mii_bus *mdio_bus = pdata->mdio_bus;
+
+ if (ndev->phydev)
+ phy_disconnect(ndev->phydev);
+
+ if (mdio_bus->state == MDIOBUS_REGISTERED)
+ mdiobus_unregister(mdio_bus);
+
+ mdiobus_free(mdio_bus);
+}
+
+int xge_mdio_config(struct net_device *ndev)
+{
+ struct xge_pdata *pdata = netdev_priv(ndev);
+ struct device *dev = &pdata->pdev->dev;
+ struct mii_bus *mdio_bus;
+ struct phy_device *phydev;
+ int ret;
+
+ mdio_bus = mdiobus_alloc();
+ if (!mdio_bus)
+ return -ENOMEM;
+
+ mdio_bus->name = "APM X-Gene Ethernet (v2) MDIO Bus";
+ mdio_bus->read = xge_mdio_read;
+ mdio_bus->write = xge_mdio_write;
+ mdio_bus->priv = pdata;
+ mdio_bus->parent = dev;
+ snprintf(mdio_bus->id, MII_BUS_ID_SIZE, "%s-mii", dev_name(dev));
+ pdata->mdio_bus = mdio_bus;
+
+ mdio_bus->phy_mask = 0x1;
+ ret = mdiobus_register(mdio_bus);
+ if (ret)
+ goto err;
+
+ phydev = phy_find_first(mdio_bus);
+ if (!phydev) {
+ dev_err(dev, "no PHY found\n");
+ ret = -ENODEV;
+ goto err;
+ }
+ phydev = phy_connect(ndev, phydev_name(phydev),
+ &xge_adjust_link,
+ pdata->resources.phy_mode);
+
+ if (IS_ERR(phydev)) {
+ netdev_err(ndev, "Could not attach to PHY\n");
+ ret = PTR_ERR(phydev);
+ goto err;
+ }
+
+ phydev->supported &= ~(SUPPORTED_10baseT_Half |
+ SUPPORTED_10baseT_Full |
+ SUPPORTED_100baseT_Half |
+ SUPPORTED_100baseT_Full |
+ SUPPORTED_1000baseT_Half |
+ SUPPORTED_AUI |
+ SUPPORTED_MII |
+ SUPPORTED_FIBRE |
+ SUPPORTED_BNC);
+ phydev->advertising = phydev->supported;
+ pdata->phy_speed = SPEED_UNKNOWN;
+
+ return 0;
+err:
+ xge_mdio_remove(ndev);
+
+ return ret;
+}
diff --git a/drivers/net/ethernet/apm/xgene-v2/ring.c b/drivers/net/ethernet/apm/xgene-v2/ring.c
new file mode 100644
index 0000000..3881082
--- /dev/null
+++ b/drivers/net/ethernet/apm/xgene-v2/ring.c
@@ -0,0 +1,81 @@
+/*
+ * Applied Micro X-Gene SoC Ethernet v2 Driver
+ *
+ * Copyright (c) 2017, Applied Micro Circuits Corporation
+ * Author(s): Iyappan Subramanian <isubramanian@apm.com>
+ * Keyur Chudgar <kchudgar@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "main.h"
+
+/* create circular linked list of descriptors */
+void xge_setup_desc(struct xge_desc_ring *ring)
+{
+ struct xge_raw_desc *raw_desc;
+ dma_addr_t dma_h, next_dma;
+ u16 offset;
+ int i;
+
+ for (i = 0; i < XGENE_ENET_NUM_DESC; i++) {
+ raw_desc = &ring->raw_desc[i];
+
+ offset = (i + 1) & (XGENE_ENET_NUM_DESC - 1);
+ next_dma = ring->dma_addr + (offset * XGENE_ENET_DESC_SIZE);
+
+ raw_desc->m0 = cpu_to_le64(SET_BITS(E, 1) |
+ SET_BITS(PKT_SIZE, SLOT_EMPTY));
+ dma_h = upper_32_bits(next_dma);
+ raw_desc->m1 = cpu_to_le64(SET_BITS(NEXT_DESC_ADDRL, next_dma) |
+ SET_BITS(NEXT_DESC_ADDRH, dma_h));
+ }
+}
+
+void xge_update_tx_desc_addr(struct xge_pdata *pdata)
+{
+ struct xge_desc_ring *ring = pdata->tx_ring;
+ dma_addr_t dma_addr = ring->dma_addr;
+
+ xge_wr_csr(pdata, DMATXDESCL, dma_addr);
+ xge_wr_csr(pdata, DMATXDESCH, upper_32_bits(dma_addr));
+
+ ring->head = 0;
+ ring->tail = 0;
+}
+
+void xge_update_rx_desc_addr(struct xge_pdata *pdata)
+{
+ struct xge_desc_ring *ring = pdata->rx_ring;
+ dma_addr_t dma_addr = ring->dma_addr;
+
+ xge_wr_csr(pdata, DMARXDESCL, dma_addr);
+ xge_wr_csr(pdata, DMARXDESCH, upper_32_bits(dma_addr));
+
+ ring->head = 0;
+ ring->tail = 0;
+}
+
+void xge_intr_enable(struct xge_pdata *pdata)
+{
+ u32 data;
+
+ data = RX_PKT_RCVD | TX_PKT_SENT;
+ xge_wr_csr(pdata, DMAINTRMASK, data);
+}
+
+void xge_intr_disable(struct xge_pdata *pdata)
+{
+ xge_wr_csr(pdata, DMAINTRMASK, 0);
+}
diff --git a/drivers/net/ethernet/apm/xgene-v2/ring.h b/drivers/net/ethernet/apm/xgene-v2/ring.h
new file mode 100644
index 0000000..abc8c9a
--- /dev/null
+++ b/drivers/net/ethernet/apm/xgene-v2/ring.h
@@ -0,0 +1,119 @@
+/*
+ * Applied Micro X-Gene SoC Ethernet v2 Driver
+ *
+ * Copyright (c) 2017, Applied Micro Circuits Corporation
+ * Author(s): Iyappan Subramanian <isubramanian@apm.com>
+ * Keyur Chudgar <kchudgar@apm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __XGENE_ENET_V2_RING_H__
+#define __XGENE_ENET_V2_RING_H__
+
+#define XGENE_ENET_DESC_SIZE 64
+#define XGENE_ENET_NUM_DESC 256
+#define NUM_BUFS 8
+#define SLOT_EMPTY 0xfff
+
+#define DMATXCTRL 0xa180
+#define DMATXDESCL 0xa184
+#define DMATXDESCH 0xa1a0
+#define DMATXSTATUS 0xa188
+#define DMARXCTRL 0xa18c
+#define DMARXDESCL 0xa190
+#define DMARXDESCH 0xa1a4
+#define DMARXSTATUS 0xa194
+#define DMAINTRMASK 0xa198
+#define DMAINTERRUPT 0xa19c
+
+#define D_POS 62
+#define D_LEN 2
+#define E_POS 63
+#define E_LEN 1
+#define PKT_ADDRL_POS 0
+#define PKT_ADDRL_LEN 32
+#define PKT_ADDRH_POS 32
+#define PKT_ADDRH_LEN 10
+#define PKT_SIZE_POS 32
+#define PKT_SIZE_LEN 12
+#define NEXT_DESC_ADDRL_POS 0
+#define NEXT_DESC_ADDRL_LEN 32
+#define NEXT_DESC_ADDRH_POS 48
+#define NEXT_DESC_ADDRH_LEN 10
+
+#define TXPKTCOUNT_POS 16
+#define TXPKTCOUNT_LEN 8
+#define RXPKTCOUNT_POS 16
+#define RXPKTCOUNT_LEN 8
+
+#define TX_PKT_SENT BIT(0)
+#define TX_BUS_ERROR BIT(3)
+#define RX_PKT_RCVD BIT(4)
+#define RX_BUS_ERROR BIT(7)
+#define RXSTATUS_RXPKTRCVD BIT(0)
+
+struct xge_raw_desc {
+ __le64 m0;
+ __le64 m1;
+ __le64 m2;
+ __le64 m3;
+ __le64 m4;
+ __le64 m5;
+ __le64 m6;
+ __le64 m7;
+};
+
+struct pkt_info {
+ struct sk_buff *skb;
+ dma_addr_t dma_addr;
+ void *pkt_buf;
+};
+
+/* software context of a descriptor ring */
+struct xge_desc_ring {
+ struct net_device *ndev;
+ dma_addr_t dma_addr;
+ u8 head;
+ u8 tail;
+ union {
+ void *desc_addr;
+ struct xge_raw_desc *raw_desc;
+ };
+ struct pkt_info (*pkt_info);
+};
+
+static inline u64 xge_set_desc_bits(int pos, int len, u64 val)
+{
+ return (val & ((1ULL << len) - 1)) << pos;
+}
+
+static inline u64 xge_get_desc_bits(int pos, int len, u64 src)
+{
+ return (src >> pos) & ((1ULL << len) - 1);
+}
+
+#define SET_BITS(field, val) \
+ xge_set_desc_bits(field ## _POS, field ## _LEN, val)
+
+#define GET_BITS(field, src) \
+ xge_get_desc_bits(field ## _POS, field ## _LEN, src)
+
+void xge_setup_desc(struct xge_desc_ring *ring);
+void xge_update_tx_desc_addr(struct xge_pdata *pdata);
+void xge_update_rx_desc_addr(struct xge_pdata *pdata);
+void xge_intr_enable(struct xge_pdata *pdata);
+void xge_intr_disable(struct xge_pdata *pdata);
+
+#endif /* __XGENE_ENET_V2_RING_H__ */
diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_hw.c b/drivers/net/ethernet/apm/xgene/xgene_enet_hw.c
index 06e6816..2a835e0 100644
--- a/drivers/net/ethernet/apm/xgene/xgene_enet_hw.c
+++ b/drivers/net/ethernet/apm/xgene/xgene_enet_hw.c
@@ -494,7 +494,7 @@
break;
}
- mc2 |= FULL_DUPLEX2 | PAD_CRC;
+ mc2 |= FULL_DUPLEX2 | PAD_CRC | LENGTH_CHK;
xgene_enet_wr_mcx_mac(pdata, MAC_CONFIG_2_ADDR, mc2);
xgene_enet_wr_mcx_mac(pdata, INTERFACE_CONTROL_ADDR, intf_ctl);
xgene_enet_wr_csr(pdata, RGMII_REG_0_ADDR, rgmii);
@@ -623,6 +623,7 @@
xgene_enet_rd_csr(pdata, CLE_BYPASS_REG0_0_ADDR, &cb);
cb |= CFG_CLE_BYPASS_EN0;
CFG_CLE_IP_PROTOCOL0_SET(&cb, 3);
+ CFG_CLE_IP_HDR_LEN_SET(&cb, 0);
xgene_enet_wr_csr(pdata, CLE_BYPASS_REG0_0_ADDR, cb);
xgene_enet_rd_csr(pdata, CLE_BYPASS_REG1_0_ADDR, &cb);
diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_hw.h b/drivers/net/ethernet/apm/xgene/xgene_enet_hw.h
index 5f83037..d250bfe 100644
--- a/drivers/net/ethernet/apm/xgene/xgene_enet_hw.h
+++ b/drivers/net/ethernet/apm/xgene/xgene_enet_hw.h
@@ -163,6 +163,7 @@
#define CFG_RXCLK_MUXSEL0_SET(dst, val) xgene_set_bits(dst, val, 26, 3)
#define CFG_CLE_IP_PROTOCOL0_SET(dst, val) xgene_set_bits(dst, val, 16, 2)
+#define CFG_CLE_IP_HDR_LEN_SET(dst, val) xgene_set_bits(dst, val, 8, 5)
#define CFG_CLE_DSTQID0_SET(dst, val) xgene_set_bits(dst, val, 0, 12)
#define CFG_CLE_FPSEL0_SET(dst, val) xgene_set_bits(dst, val, 16, 4)
#define CFG_CLE_NXTFPSEL0_SET(dst, val) xgene_set_bits(dst, val, 20, 4)
@@ -215,6 +216,7 @@
#define ENET_GHD_MODE BIT(26)
#define FULL_DUPLEX2 BIT(0)
#define PAD_CRC BIT(2)
+#define LENGTH_CHK BIT(4)
#define SCAN_AUTO_INCR BIT(5)
#define TBYT_ADDR 0x38
#define TPKT_ADDR 0x39
diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
index b3568c4..5f37ed3 100644
--- a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
+++ b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
@@ -601,14 +601,24 @@
return NETDEV_TX_OK;
}
-static void xgene_enet_skip_csum(struct sk_buff *skb)
+static void xgene_enet_rx_csum(struct sk_buff *skb)
{
+ struct net_device *ndev = skb->dev;
struct iphdr *iph = ip_hdr(skb);
- if (!ip_is_fragment(iph) ||
- (iph->protocol != IPPROTO_TCP && iph->protocol != IPPROTO_UDP)) {
- skb->ip_summed = CHECKSUM_UNNECESSARY;
- }
+ if (!(ndev->features & NETIF_F_RXCSUM))
+ return;
+
+ if (skb->protocol != htons(ETH_P_IP))
+ return;
+
+ if (ip_is_fragment(iph))
+ return;
+
+ if (iph->protocol != IPPROTO_TCP && iph->protocol != IPPROTO_UDP)
+ return;
+
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
}
static void xgene_enet_free_pagepool(struct xgene_enet_desc_ring *buf_pool,
@@ -648,12 +658,24 @@
buf_pool->head = head;
}
+/* Errata 10GE_8 and ENET_11 - allow packet with length <=64B */
+static bool xgene_enet_errata_10GE_8(struct sk_buff *skb, u32 len, u8 status)
+{
+ if (status == INGRESS_PKT_LEN && len == ETHER_MIN_PACKET) {
+ if (ntohs(eth_hdr(skb)->h_proto) < 46)
+ return true;
+ }
+
+ return false;
+}
+
static int xgene_enet_rx_frame(struct xgene_enet_desc_ring *rx_ring,
struct xgene_enet_raw_desc *raw_desc,
struct xgene_enet_raw_desc *exp_desc)
{
struct xgene_enet_desc_ring *buf_pool, *page_pool;
u32 datalen, frag_size, skb_index;
+ struct xgene_enet_pdata *pdata;
struct net_device *ndev;
dma_addr_t dma_addr;
struct sk_buff *skb;
@@ -666,6 +688,7 @@
bool nv;
ndev = rx_ring->ndev;
+ pdata = netdev_priv(ndev);
dev = ndev_to_dev(rx_ring->ndev);
buf_pool = rx_ring->buf_pool;
page_pool = rx_ring->page_pool;
@@ -676,30 +699,29 @@
skb = buf_pool->rx_skb[skb_index];
buf_pool->rx_skb[skb_index] = NULL;
- /* checking for error */
- status = (GET_VAL(ELERR, le64_to_cpu(raw_desc->m0)) << LERR_LEN) ||
- GET_VAL(LERR, le64_to_cpu(raw_desc->m0));
- if (unlikely(status > 2)) {
- dev_kfree_skb_any(skb);
- xgene_enet_free_pagepool(page_pool, raw_desc, exp_desc);
- xgene_enet_parse_error(rx_ring, netdev_priv(rx_ring->ndev),
- status);
- ret = -EIO;
- goto out;
- }
-
- /* strip off CRC as HW isn't doing this */
datalen = xgene_enet_get_data_len(le64_to_cpu(raw_desc->m1));
-
- nv = GET_VAL(NV, le64_to_cpu(raw_desc->m0));
- if (!nv)
- datalen -= 4;
-
skb_put(skb, datalen);
prefetch(skb->data - NET_IP_ALIGN);
+ skb->protocol = eth_type_trans(skb, ndev);
- if (!nv)
+ /* checking for error */
+ status = (GET_VAL(ELERR, le64_to_cpu(raw_desc->m0)) << LERR_LEN) |
+ GET_VAL(LERR, le64_to_cpu(raw_desc->m0));
+ if (unlikely(status)) {
+ if (!xgene_enet_errata_10GE_8(skb, datalen, status)) {
+ dev_kfree_skb_any(skb);
+ xgene_enet_free_pagepool(page_pool, raw_desc, exp_desc);
+ xgene_enet_parse_error(rx_ring, pdata, status);
+ goto out;
+ }
+ }
+
+ nv = GET_VAL(NV, le64_to_cpu(raw_desc->m0));
+ if (!nv) {
+ /* strip off CRC as HW isn't doing this */
+ datalen -= 4;
goto skip_jumbo;
+ }
slots = page_pool->slots - 1;
head = page_pool->head;
@@ -728,11 +750,7 @@
skip_jumbo:
skb_checksum_none_assert(skb);
- skb->protocol = eth_type_trans(skb, ndev);
- if (likely((ndev->features & NETIF_F_IP_CSUM) &&
- skb->protocol == htons(ETH_P_IP))) {
- xgene_enet_skip_csum(skb);
- }
+ xgene_enet_rx_csum(skb);
rx_ring->rx_packets++;
rx_ring->rx_bytes += datalen;
@@ -2039,7 +2057,7 @@
xgene_enet_setup_ops(pdata);
if (pdata->phy_mode == PHY_INTERFACE_MODE_XGMII) {
- ndev->features |= NETIF_F_TSO;
+ ndev->features |= NETIF_F_TSO | NETIF_F_RXCSUM;
spin_lock_init(&pdata->mss_lock);
}
ndev->hw_features = ndev->features;
diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_main.h b/drivers/net/ethernet/apm/xgene/xgene_enet_main.h
index 5257174..0d4be24 100644
--- a/drivers/net/ethernet/apm/xgene/xgene_enet_main.h
+++ b/drivers/net/ethernet/apm/xgene/xgene_enet_main.h
@@ -41,6 +41,7 @@
#include "../../../phy/mdio-xgene.h"
#define XGENE_DRV_VERSION "v1.0"
+#define ETHER_MIN_PACKET 64
#define XGENE_ENET_STD_MTU 1536
#define XGENE_ENET_MAX_MTU 9600
#define SKB_BUFFER_SIZE (XGENE_ENET_STD_MTU - NET_IP_ALIGN)
diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_xgmac.c b/drivers/net/ethernet/apm/xgene/xgene_enet_xgmac.c
index ece19e6..423240c 100644
--- a/drivers/net/ethernet/apm/xgene/xgene_enet_xgmac.c
+++ b/drivers/net/ethernet/apm/xgene/xgene_enet_xgmac.c
@@ -341,8 +341,15 @@
xgene_enet_rd_csr(pdata, XG_RSIF_CONFIG_REG_ADDR, &data);
data |= CFG_RSIF_FPBUFF_TIMEOUT_EN;
+ /* Errata 10GE_1 - FIFO threshold default value incorrect */
+ RSIF_CLE_BUFF_THRESH_SET(&data, XG_RSIF_CLE_BUFF_THRESH);
xgene_enet_wr_csr(pdata, XG_RSIF_CONFIG_REG_ADDR, data);
+ /* Errata 10GE_1 - FIFO threshold default value incorrect */
+ xgene_enet_rd_csr(pdata, XG_RSIF_CONFIG1_REG_ADDR, &data);
+ RSIF_PLC_CLE_BUFF_THRESH_SET(&data, XG_RSIF_PLC_CLE_BUFF_THRESH);
+ xgene_enet_wr_csr(pdata, XG_RSIF_CONFIG1_REG_ADDR, data);
+
xgene_enet_rd_csr(pdata, XG_ENET_SPARE_CFG_REG_ADDR, &data);
data |= BIT(12);
xgene_enet_wr_csr(pdata, XG_ENET_SPARE_CFG_REG_ADDR, data);
diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_xgmac.h b/drivers/net/ethernet/apm/xgene/xgene_enet_xgmac.h
index 03b847a..e644a42 100644
--- a/drivers/net/ethernet/apm/xgene/xgene_enet_xgmac.h
+++ b/drivers/net/ethernet/apm/xgene/xgene_enet_xgmac.h
@@ -65,6 +65,11 @@
#define XG_DEF_PAUSE_THRES 0x390
#define XG_DEF_PAUSE_OFF_THRES 0x2c0
#define XG_RSIF_CONFIG_REG_ADDR 0x00a0
+#define XG_RSIF_CLE_BUFF_THRESH 0x3
+#define RSIF_CLE_BUFF_THRESH_SET(dst, val) xgene_set_bits(dst, val, 0, 3)
+#define XG_RSIF_CONFIG1_REG_ADDR 0x00b8
+#define XG_RSIF_PLC_CLE_BUFF_THRESH 0x1
+#define RSIF_PLC_CLE_BUFF_THRESH_SET(dst, val) xgene_set_bits(dst, val, 0, 2)
#define XCLE_BYPASS_REG0_ADDR 0x0160
#define XCLE_BYPASS_REG1_ADDR 0x0164
#define XG_CFG_BYPASS_ADDR 0x0204
diff --git a/drivers/net/ethernet/arc/emac_main.c b/drivers/net/ethernet/arc/emac_main.c
index 2387339..68de2f2 100644
--- a/drivers/net/ethernet/arc/emac_main.c
+++ b/drivers/net/ethernet/arc/emac_main.c
@@ -434,7 +434,7 @@
/* Enable EMAC */
arc_reg_or(priv, R_CTRL, EN_MASK);
- phy_start_aneg(ndev->phydev);
+ phy_start(ndev->phydev);
netif_start_queue(ndev);
@@ -556,6 +556,8 @@
napi_disable(&priv->napi);
netif_stop_queue(ndev);
+ phy_stop(ndev->phydev);
+
/* Disable interrupts */
arc_reg_clr(priv, R_ENABLE, RXINT_MASK | TXINT_MASK | ERR_MASK);
diff --git a/drivers/net/ethernet/atheros/alx/alx.h b/drivers/net/ethernet/atheros/alx/alx.h
index d4a4091..78c5de4 100644
--- a/drivers/net/ethernet/atheros/alx/alx.h
+++ b/drivers/net/ethernet/atheros/alx/alx.h
@@ -102,9 +102,6 @@
#define ALX_MAX_NAPIS 8
-#define ALX_FLAG_USING_MSIX BIT(0)
-#define ALX_FLAG_USING_MSI BIT(1)
-
struct alx_priv {
struct net_device *dev;
@@ -112,7 +109,6 @@
/* msi-x vectors */
int num_vec;
- struct msix_entry *msix_entries;
/* all descriptor memory */
struct {
@@ -139,8 +135,6 @@
u16 msg_enable;
- int flags;
-
/* protects hw.stats */
spinlock_t stats_lock;
};
diff --git a/drivers/net/ethernet/atheros/alx/main.c b/drivers/net/ethernet/atheros/alx/main.c
index 6a27c26..a8c2db8 100644
--- a/drivers/net/ethernet/atheros/alx/main.c
+++ b/drivers/net/ethernet/atheros/alx/main.c
@@ -314,7 +314,7 @@
napi_complete_done(&np->napi, work);
/* enable interrupt */
- if (alx->flags & ALX_FLAG_USING_MSIX) {
+ if (alx->hw.pdev->msix_enabled) {
alx_mask_msix(hw, np->vec_idx, false);
} else {
spin_lock_irqsave(&alx->irq_lock, flags);
@@ -811,7 +811,7 @@
u32 tbl[2] = {0, 0};
int i, vector, idx, shift;
- if (alx->flags & ALX_FLAG_USING_MSIX) {
+ if (alx->hw.pdev->msix_enabled) {
/* tx mappings */
for (i = 0, vector = 1; i < alx->num_txq; i++, vector++) {
idx = txq_vec_mapping_shift[i * 2];
@@ -828,29 +828,19 @@
alx_write_mem32(hw, ALX_MSI_ID_MAP, 0);
}
-static bool alx_enable_msix(struct alx_priv *alx)
+static int alx_enable_msix(struct alx_priv *alx)
{
- int i, err, num_vec, num_txq, num_rxq;
+ int err, num_vec, num_txq, num_rxq;
num_txq = min_t(int, num_online_cpus(), ALX_MAX_TX_QUEUES);
num_rxq = 1;
num_vec = max_t(int, num_txq, num_rxq) + 1;
- alx->msix_entries = kcalloc(num_vec, sizeof(struct msix_entry),
- GFP_KERNEL);
- if (!alx->msix_entries) {
- netdev_warn(alx->dev, "Allocation of msix entries failed!\n");
- return false;
- }
-
- for (i = 0; i < num_vec; i++)
- alx->msix_entries[i].entry = i;
-
- err = pci_enable_msix(alx->hw.pdev, alx->msix_entries, num_vec);
+ err = pci_alloc_irq_vectors(alx->hw.pdev, num_vec, num_vec,
+ PCI_IRQ_MSIX);
if (err) {
- kfree(alx->msix_entries);
netdev_warn(alx->dev, "Enabling MSI-X interrupts failed!\n");
- return false;
+ return err;
}
alx->num_vec = num_vec;
@@ -858,7 +848,7 @@
alx->num_txq = num_txq;
alx->num_rxq = num_rxq;
- return true;
+ return err;
}
static int alx_request_msix(struct alx_priv *alx)
@@ -866,7 +856,7 @@
struct net_device *netdev = alx->dev;
int i, err, vector = 0, free_vector = 0;
- err = request_irq(alx->msix_entries[0].vector, alx_intr_msix_misc,
+ err = request_irq(pci_irq_vector(alx->hw.pdev, 0), alx_intr_msix_misc,
0, netdev->name, alx);
if (err)
goto out_err;
@@ -889,7 +879,7 @@
sprintf(np->irq_lbl, "%s-unused", netdev->name);
np->vec_idx = vector;
- err = request_irq(alx->msix_entries[vector].vector,
+ err = request_irq(pci_irq_vector(alx->hw.pdev, vector),
alx_intr_msix_ring, 0, np->irq_lbl, np);
if (err)
goto out_free;
@@ -897,47 +887,31 @@
return 0;
out_free:
- free_irq(alx->msix_entries[free_vector++].vector, alx);
+ free_irq(pci_irq_vector(alx->hw.pdev, free_vector++), alx);
vector--;
for (i = 0; i < vector; i++)
- free_irq(alx->msix_entries[free_vector++].vector,
+ free_irq(pci_irq_vector(alx->hw.pdev,free_vector++),
alx->qnapi[i]);
out_err:
return err;
}
-static void alx_init_intr(struct alx_priv *alx, bool msix)
+static int alx_init_intr(struct alx_priv *alx)
{
- if (msix) {
- if (alx_enable_msix(alx))
- alx->flags |= ALX_FLAG_USING_MSIX;
- }
+ int ret;
- if (!(alx->flags & ALX_FLAG_USING_MSIX)) {
- alx->num_vec = 1;
- alx->num_napi = 1;
- alx->num_txq = 1;
- alx->num_rxq = 1;
+ ret = pci_alloc_irq_vectors(alx->hw.pdev, 1, 1,
+ PCI_IRQ_MSI | PCI_IRQ_LEGACY);
+ if (ret)
+ return ret;
- if (!pci_enable_msi(alx->hw.pdev))
- alx->flags |= ALX_FLAG_USING_MSI;
- }
-}
-
-static void alx_disable_advanced_intr(struct alx_priv *alx)
-{
- if (alx->flags & ALX_FLAG_USING_MSIX) {
- kfree(alx->msix_entries);
- pci_disable_msix(alx->hw.pdev);
- alx->flags &= ~ALX_FLAG_USING_MSIX;
- }
-
- if (alx->flags & ALX_FLAG_USING_MSI) {
- pci_disable_msi(alx->hw.pdev);
- alx->flags &= ~ALX_FLAG_USING_MSI;
- }
+ alx->num_vec = 1;
+ alx->num_napi = 1;
+ alx->num_txq = 1;
+ alx->num_rxq = 1;
+ return 0;
}
static void alx_irq_enable(struct alx_priv *alx)
@@ -950,10 +924,11 @@
alx_write_mem32(hw, ALX_IMR, alx->int_mask);
alx_post_write(hw);
- if (alx->flags & ALX_FLAG_USING_MSIX)
+ if (alx->hw.pdev->msix_enabled) {
/* enable all msix irqs */
for (i = 0; i < alx->num_vec; i++)
alx_mask_msix(hw, i, false);
+ }
}
static void alx_irq_disable(struct alx_priv *alx)
@@ -965,13 +940,13 @@
alx_write_mem32(hw, ALX_IMR, 0);
alx_post_write(hw);
- if (alx->flags & ALX_FLAG_USING_MSIX) {
+ if (alx->hw.pdev->msix_enabled) {
for (i = 0; i < alx->num_vec; i++) {
alx_mask_msix(hw, i, true);
- synchronize_irq(alx->msix_entries[i].vector);
+ synchronize_irq(pci_irq_vector(alx->hw.pdev, i));
}
} else {
- synchronize_irq(alx->hw.pdev->irq);
+ synchronize_irq(pci_irq_vector(alx->hw.pdev, 0));
}
}
@@ -981,8 +956,11 @@
alx_free_rings(alx);
alx_free_napis(alx);
- alx_disable_advanced_intr(alx);
- alx_init_intr(alx, false);
+ pci_free_irq_vectors(alx->hw.pdev);
+
+ err = alx_init_intr(alx);
+ if (err)
+ return err;
err = alx_alloc_napis(alx);
if (err)
@@ -1004,7 +982,7 @@
msi_ctrl = (hw->imt >> 1) << ALX_MSI_RETRANS_TM_SHIFT;
- if (alx->flags & ALX_FLAG_USING_MSIX) {
+ if (alx->hw.pdev->msix_enabled) {
alx_write_mem32(hw, ALX_MSI_RETRANS_TIMER, msi_ctrl);
err = alx_request_msix(alx);
if (!err)
@@ -1016,20 +994,20 @@
goto out;
}
- if (alx->flags & ALX_FLAG_USING_MSI) {
+ if (alx->hw.pdev->msi_enabled) {
alx_write_mem32(hw, ALX_MSI_RETRANS_TIMER,
msi_ctrl | ALX_MSI_MASK_SEL_LINE);
- err = request_irq(pdev->irq, alx_intr_msi, 0,
+ err = request_irq(pci_irq_vector(pdev, 0), alx_intr_msi, 0,
alx->dev->name, alx);
if (!err)
goto out;
+
/* fall back to legacy interrupt */
- alx->flags &= ~ALX_FLAG_USING_MSI;
- pci_disable_msi(alx->hw.pdev);
+ pci_free_irq_vectors(alx->hw.pdev);
}
alx_write_mem32(hw, ALX_MSI_RETRANS_TIMER, 0);
- err = request_irq(pdev->irq, alx_intr_legacy, IRQF_SHARED,
+ err = request_irq(pci_irq_vector(pdev, 0), alx_intr_legacy, IRQF_SHARED,
alx->dev->name, alx);
out:
if (!err)
@@ -1042,18 +1020,15 @@
static void alx_free_irq(struct alx_priv *alx)
{
struct pci_dev *pdev = alx->hw.pdev;
- int i, vector = 0;
+ int i;
- if (alx->flags & ALX_FLAG_USING_MSIX) {
- free_irq(alx->msix_entries[vector++].vector, alx);
+ free_irq(pci_irq_vector(pdev, 0), alx);
+ if (alx->hw.pdev->msix_enabled) {
for (i = 0; i < alx->num_napi; i++)
- free_irq(alx->msix_entries[vector++].vector,
- alx->qnapi[i]);
- } else {
- free_irq(pdev->irq, alx);
+ free_irq(pci_irq_vector(pdev, i + 1), alx->qnapi[i]);
}
- alx_disable_advanced_intr(alx);
+ pci_free_irq_vectors(pdev);
}
static int alx_identify_hw(struct alx_priv *alx)
@@ -1221,7 +1196,12 @@
{
int err;
- alx_init_intr(alx, true);
+ err = alx_enable_msix(alx);
+ if (err < 0) {
+ err = alx_init_intr(alx);
+ if (err)
+ return err;
+ }
if (!resume)
netif_carrier_off(alx->dev);
@@ -1264,7 +1244,7 @@
alx_free_rings(alx);
alx_free_napis(alx);
out_disable_adv_intr:
- alx_disable_advanced_intr(alx);
+ pci_free_irq_vectors(alx->hw.pdev);
return err;
}
@@ -1637,11 +1617,11 @@
struct alx_priv *alx = netdev_priv(netdev);
int i;
- if (alx->flags & ALX_FLAG_USING_MSIX) {
+ if (alx->hw.pdev->msix_enabled) {
alx_intr_msix_misc(0, alx);
for (i = 0; i < alx->num_txq; i++)
alx_intr_msix_ring(0, alx->qnapi[i]);
- } else if (alx->flags & ALX_FLAG_USING_MSI)
+ } else if (alx->hw.pdev->msi_enabled)
alx_intr_msi(0, alx);
else
alx_intr_legacy(0, alx);
@@ -1783,7 +1763,7 @@
netdev->netdev_ops = &alx_netdev_ops;
netdev->ethtool_ops = &alx_ethtool_ops;
- netdev->irq = pdev->irq;
+ netdev->irq = pci_irq_vector(pdev, 0);
netdev->watchdog_timeo = ALX_WATCHDOG_TIME;
if (ent->driver_data & ALX_DEV_QUIRK_MSI_INTX_DISABLE_BUG)
diff --git a/drivers/net/ethernet/atheros/atlx/atl1.c b/drivers/net/ethernet/atheros/atlx/atl1.c
index 022772e..83d2db2 100644
--- a/drivers/net/ethernet/atheros/atlx/atl1.c
+++ b/drivers/net/ethernet/atheros/atlx/atl1.c
@@ -1886,7 +1886,7 @@
buffer_info->skb = skb;
buffer_info->length = (u16) adapter->rx_buffer_len;
page = virt_to_page(skb->data);
- offset = (unsigned long)skb->data & ~PAGE_MASK;
+ offset = offset_in_page(skb->data);
buffer_info->dma = pci_map_page(pdev, page, offset,
adapter->rx_buffer_len,
PCI_DMA_FROMDEVICE);
@@ -2230,7 +2230,7 @@
hdr_len = skb_transport_offset(skb) + tcp_hdrlen(skb);
buffer_info->length = hdr_len;
page = virt_to_page(skb->data);
- offset = (unsigned long)skb->data & ~PAGE_MASK;
+ offset = offset_in_page(skb->data);
buffer_info->dma = pci_map_page(adapter->pdev, page,
offset, hdr_len,
PCI_DMA_TODEVICE);
@@ -2254,9 +2254,8 @@
data_len -= buffer_info->length;
page = virt_to_page(skb->data +
(hdr_len + i * ATL1_MAX_TX_BUF_LEN));
- offset = (unsigned long)(skb->data +
- (hdr_len + i * ATL1_MAX_TX_BUF_LEN)) &
- ~PAGE_MASK;
+ offset = offset_in_page(skb->data +
+ (hdr_len + i * ATL1_MAX_TX_BUF_LEN));
buffer_info->dma = pci_map_page(adapter->pdev,
page, offset, buffer_info->length,
PCI_DMA_TODEVICE);
@@ -2268,7 +2267,7 @@
/* not TSO */
buffer_info->length = buf_len;
page = virt_to_page(skb->data);
- offset = (unsigned long)skb->data & ~PAGE_MASK;
+ offset = offset_in_page(skb->data);
buffer_info->dma = pci_map_page(adapter->pdev, page,
offset, buf_len, PCI_DMA_TODEVICE);
if (++next_to_use == tpd_ring->count)
diff --git a/drivers/net/ethernet/broadcom/Kconfig b/drivers/net/ethernet/broadcom/Kconfig
index 940fb24..9641380 100644
--- a/drivers/net/ethernet/broadcom/Kconfig
+++ b/drivers/net/ethernet/broadcom/Kconfig
@@ -109,7 +109,6 @@
tristate "Broadcom Tigon3 support"
depends on PCI
select PHYLIB
- select HWMON
imply PTP_1588_CLOCK
---help---
This driver supports Broadcom Tigon3 based gigabit Ethernet cards.
@@ -117,6 +116,13 @@
To compile this driver as a module, choose M here: the module
will be called tg3. This is recommended.
+config TIGON3_HWMON
+ bool "Broadcom Tigon3 HWMON support"
+ default y
+ depends on TIGON3 && HWMON && !(TIGON3=y && HWMON=m)
+ ---help---
+ Say Y if you want to expose the thermal sensor on Tigon3 devices.
+
config BNX2X
tristate "Broadcom NetXtremeII 10Gb support"
depends on PCI
diff --git a/drivers/net/ethernet/broadcom/bcmsysport.c b/drivers/net/ethernet/broadcom/bcmsysport.c
index a68d4889..099b374 100644
--- a/drivers/net/ethernet/broadcom/bcmsysport.c
+++ b/drivers/net/ethernet/broadcom/bcmsysport.c
@@ -22,6 +22,7 @@
#include <linux/of_mdio.h>
#include <linux/phy.h>
#include <linux/phy_fixed.h>
+#include <net/dsa.h>
#include <net/ip.h>
#include <net/ipv6.h>
@@ -284,6 +285,7 @@
STAT_MIB_SOFT("alloc_rx_buff_failed", mib.alloc_rx_buff_failed),
STAT_MIB_SOFT("rx_dma_failed", mib.rx_dma_failed),
STAT_MIB_SOFT("tx_dma_failed", mib.tx_dma_failed),
+ /* Per TX-queue statistics are dynamically appended */
};
#define BCM_SYSPORT_STATS_LEN ARRAY_SIZE(bcm_sysport_gstrings_stats)
@@ -338,7 +340,8 @@
continue;
j++;
}
- return j;
+ /* Include per-queue statistics */
+ return j + dev->num_tx_queues * NUM_SYSPORT_TXQ_STAT;
default:
return -EOPNOTSUPP;
}
@@ -349,6 +352,7 @@
{
struct bcm_sysport_priv *priv = netdev_priv(dev);
const struct bcm_sysport_stats *s;
+ char buf[128];
int i, j;
switch (stringset) {
@@ -363,6 +367,18 @@
ETH_GSTRING_LEN);
j++;
}
+
+ for (i = 0; i < dev->num_tx_queues; i++) {
+ snprintf(buf, sizeof(buf), "txq%d_packets", i);
+ memcpy(data + j * ETH_GSTRING_LEN, buf,
+ ETH_GSTRING_LEN);
+ j++;
+
+ snprintf(buf, sizeof(buf), "txq%d_bytes", i);
+ memcpy(data + j * ETH_GSTRING_LEN, buf,
+ ETH_GSTRING_LEN);
+ j++;
+ }
break;
default:
break;
@@ -418,6 +434,7 @@
struct ethtool_stats *stats, u64 *data)
{
struct bcm_sysport_priv *priv = netdev_priv(dev);
+ struct bcm_sysport_tx_ring *ring;
int i, j;
if (netif_running(dev))
@@ -436,6 +453,22 @@
data[j] = *(unsigned long *)p;
j++;
}
+
+ /* For SYSTEMPORT Lite since we have holes in our statistics, j would
+ * be equal to BCM_SYSPORT_STATS_LEN at the end of the loop, but it
+ * needs to point to how many total statistics we have minus the
+ * number of per TX queue statistics
+ */
+ j = bcm_sysport_get_sset_count(dev, ETH_SS_STATS) -
+ dev->num_tx_queues * NUM_SYSPORT_TXQ_STAT;
+
+ for (i = 0; i < dev->num_tx_queues; i++) {
+ ring = &priv->tx_rings[i];
+ data[j] = ring->packets;
+ j++;
+ data[j] = ring->bytes;
+ j++;
+ }
}
static void bcm_sysport_get_wol(struct net_device *dev,
@@ -637,6 +670,9 @@
u16 len, status;
struct bcm_rsb *rsb;
+ /* Clear status before servicing to reduce spurious interrupts */
+ intrl2_0_writel(priv, INTRL2_0_RDMA_MBDONE, INTRL2_CPU_CLEAR);
+
/* Determine how much we should process since last call, SYSTEMPORT Lite
* groups the producer and consumer indexes into the same 32-bit
* which we access using RDMA_CONS_INDEX
@@ -647,11 +683,7 @@
p_index = rdma_readl(priv, RDMA_CONS_INDEX);
p_index &= RDMA_PROD_INDEX_MASK;
- if (p_index < priv->rx_c_index)
- to_process = (RDMA_CONS_INDEX_MASK + 1) -
- priv->rx_c_index + p_index;
- else
- to_process = p_index - priv->rx_c_index;
+ to_process = (p_index - priv->rx_c_index) & RDMA_CONS_INDEX_MASK;
netif_dbg(priv, rx_status, ndev,
"p_index=%d rx_c_index=%d to_process=%d\n",
@@ -746,26 +778,26 @@
return processed;
}
-static void bcm_sysport_tx_reclaim_one(struct bcm_sysport_priv *priv,
+static void bcm_sysport_tx_reclaim_one(struct bcm_sysport_tx_ring *ring,
struct bcm_sysport_cb *cb,
unsigned int *bytes_compl,
unsigned int *pkts_compl)
{
+ struct bcm_sysport_priv *priv = ring->priv;
struct device *kdev = &priv->pdev->dev;
- struct net_device *ndev = priv->netdev;
if (cb->skb) {
- ndev->stats.tx_bytes += cb->skb->len;
+ ring->bytes += cb->skb->len;
*bytes_compl += cb->skb->len;
dma_unmap_single(kdev, dma_unmap_addr(cb, dma_addr),
dma_unmap_len(cb, dma_len),
DMA_TO_DEVICE);
- ndev->stats.tx_packets++;
+ ring->packets++;
(*pkts_compl)++;
bcm_sysport_free_cb(cb);
/* SKB fragment */
} else if (dma_unmap_addr(cb, dma_addr)) {
- ndev->stats.tx_bytes += dma_unmap_len(cb, dma_len);
+ ring->bytes += dma_unmap_len(cb, dma_len);
dma_unmap_page(kdev, dma_unmap_addr(cb, dma_addr),
dma_unmap_len(cb, dma_len), DMA_TO_DEVICE);
dma_unmap_addr_set(cb, dma_addr, 0);
@@ -782,6 +814,13 @@
struct bcm_sysport_cb *cb;
u32 hw_ind;
+ /* Clear status before servicing to reduce spurious interrupts */
+ if (!ring->priv->is_lite)
+ intrl2_1_writel(ring->priv, BIT(ring->index), INTRL2_CPU_CLEAR);
+ else
+ intrl2_0_writel(ring->priv, BIT(ring->index +
+ INTRL2_0_TDMA_MBDONE_SHIFT), INTRL2_CPU_CLEAR);
+
/* Compute how many descriptors have been processed since last call */
hw_ind = tdma_readl(priv, TDMA_DESC_RING_PROD_CONS_INDEX(ring->index));
c_index = (hw_ind >> RING_CONS_INDEX_SHIFT) & RING_CONS_INDEX_MASK;
@@ -803,7 +842,7 @@
while (last_tx_cn-- > 0) {
cb = ring->cbs + last_c_index;
- bcm_sysport_tx_reclaim_one(priv, cb, &bytes_compl, &pkts_compl);
+ bcm_sysport_tx_reclaim_one(ring, cb, &bytes_compl, &pkts_compl);
ring->desc_count++;
last_c_index++;
@@ -1632,6 +1671,24 @@
return 0;
}
+static struct net_device_stats *bcm_sysport_get_nstats(struct net_device *dev)
+{
+ struct bcm_sysport_priv *priv = netdev_priv(dev);
+ unsigned long tx_bytes = 0, tx_packets = 0;
+ struct bcm_sysport_tx_ring *ring;
+ unsigned int q;
+
+ for (q = 0; q < dev->num_tx_queues; q++) {
+ ring = &priv->tx_rings[q];
+ tx_bytes += ring->bytes;
+ tx_packets += ring->packets;
+ }
+
+ dev->stats.tx_bytes = tx_bytes;
+ dev->stats.tx_packets = tx_packets;
+ return &dev->stats;
+}
+
static void bcm_sysport_netif_start(struct net_device *dev)
{
struct bcm_sysport_priv *priv = netdev_priv(dev);
@@ -1893,6 +1950,7 @@
#ifdef CONFIG_NET_POLL_CONTROLLER
.ndo_poll_controller = bcm_sysport_poll_controller,
#endif
+ .ndo_get_stats = bcm_sysport_get_nstats,
};
#define REV_FMT "v%2x.%02x"
diff --git a/drivers/net/ethernet/broadcom/bcmsysport.h b/drivers/net/ethernet/broadcom/bcmsysport.h
index 863ddd7..77a51c1 100644
--- a/drivers/net/ethernet/broadcom/bcmsysport.h
+++ b/drivers/net/ethernet/broadcom/bcmsysport.h
@@ -647,6 +647,9 @@
.reg_offset = ofs, \
}
+/* TX bytes and packets */
+#define NUM_SYSPORT_TXQ_STAT 2
+
struct bcm_sysport_stats {
char stat_string[ETH_GSTRING_LEN];
int stat_sizeof;
@@ -690,6 +693,8 @@
struct bcm_sysport_cb *cbs; /* Transmit control blocks */
struct dma_desc *desc_cpu; /* CPU view of the descriptor */
struct bcm_sysport_priv *priv; /* private context backpointer */
+ unsigned long packets; /* packets statistics */
+ unsigned long bytes; /* bytes statistics */
};
/* Driver private structure */
diff --git a/drivers/net/ethernet/broadcom/bgmac-bcma.c b/drivers/net/ethernet/broadcom/bgmac-bcma.c
index d59cfcc4..6322594 100644
--- a/drivers/net/ethernet/broadcom/bgmac-bcma.c
+++ b/drivers/net/ethernet/broadcom/bgmac-bcma.c
@@ -11,6 +11,7 @@
#include <linux/bcma/bcma.h>
#include <linux/brcmphy.h>
#include <linux/etherdevice.h>
+#include <linux/of_net.h>
#include "bgmac.h"
static inline bool bgmac_is_bcm4707_family(struct bcma_device *core)
@@ -114,7 +115,7 @@
struct ssb_sprom *sprom = &core->bus->sprom;
struct mii_bus *mii_bus;
struct bgmac *bgmac;
- u8 *mac;
+ const u8 *mac = NULL;
int err;
bgmac = bgmac_alloc(&core->dev);
@@ -127,21 +128,27 @@
bcma_set_drvdata(core, bgmac);
- switch (core->core_unit) {
- case 0:
- mac = sprom->et0mac;
- break;
- case 1:
- mac = sprom->et1mac;
- break;
- case 2:
- mac = sprom->et2mac;
- break;
- default:
- dev_err(bgmac->dev, "Unsupported core_unit %d\n",
- core->core_unit);
- err = -ENOTSUPP;
- goto err;
+ if (bgmac->dev->of_node)
+ mac = of_get_mac_address(bgmac->dev->of_node);
+
+ /* If no MAC address assigned via device tree, check SPROM */
+ if (!mac) {
+ switch (core->core_unit) {
+ case 0:
+ mac = sprom->et0mac;
+ break;
+ case 1:
+ mac = sprom->et1mac;
+ break;
+ case 2:
+ mac = sprom->et2mac;
+ break;
+ default:
+ dev_err(bgmac->dev, "Unsupported core_unit %d\n",
+ core->core_unit);
+ err = -ENOTSUPP;
+ goto err;
+ }
}
ether_addr_copy(bgmac->net_dev->dev_addr, mac);
@@ -192,48 +199,16 @@
goto err1;
}
- bgmac->has_robosw = !!(core->bus->sprom.boardflags_lo &
- BGMAC_BFL_ENETROBO);
+ bgmac->has_robosw = !!(sprom->boardflags_lo & BGMAC_BFL_ENETROBO);
if (bgmac->has_robosw)
dev_warn(bgmac->dev, "Support for Roboswitch not implemented\n");
- if (core->bus->sprom.boardflags_lo & BGMAC_BFL_ENETADM)
+ if (sprom->boardflags_lo & BGMAC_BFL_ENETADM)
dev_warn(bgmac->dev, "Support for ADMtek ethernet switch not implemented\n");
/* Feature Flags */
- switch (core->bus->chipinfo.id) {
- case BCMA_CHIP_ID_BCM5357:
- bgmac->feature_flags |= BGMAC_FEAT_SET_RXQ_CLK;
- bgmac->feature_flags |= BGMAC_FEAT_CLKCTLST;
- bgmac->feature_flags |= BGMAC_FEAT_FLW_CTRL1;
- bgmac->feature_flags |= BGMAC_FEAT_SW_TYPE_PHY;
- if (core->bus->chipinfo.pkg == BCMA_PKG_ID_BCM47186) {
- bgmac->feature_flags |= BGMAC_FEAT_IOST_ATTACHED;
- bgmac->feature_flags |= BGMAC_FEAT_SW_TYPE_RGMII;
- }
- if (core->bus->chipinfo.pkg == BCMA_PKG_ID_BCM5358)
- bgmac->feature_flags |= BGMAC_FEAT_SW_TYPE_EPHYRMII;
- break;
- case BCMA_CHIP_ID_BCM53572:
- bgmac->feature_flags |= BGMAC_FEAT_SET_RXQ_CLK;
- bgmac->feature_flags |= BGMAC_FEAT_CLKCTLST;
- bgmac->feature_flags |= BGMAC_FEAT_FLW_CTRL1;
- bgmac->feature_flags |= BGMAC_FEAT_SW_TYPE_PHY;
- if (core->bus->chipinfo.pkg == BCMA_PKG_ID_BCM47188) {
- bgmac->feature_flags |= BGMAC_FEAT_SW_TYPE_RGMII;
- bgmac->feature_flags |= BGMAC_FEAT_IOST_ATTACHED;
- }
- break;
- case BCMA_CHIP_ID_BCM4749:
- bgmac->feature_flags |= BGMAC_FEAT_SET_RXQ_CLK;
- bgmac->feature_flags |= BGMAC_FEAT_CLKCTLST;
- bgmac->feature_flags |= BGMAC_FEAT_FLW_CTRL1;
- bgmac->feature_flags |= BGMAC_FEAT_SW_TYPE_PHY;
- if (core->bus->chipinfo.pkg == 10) {
- bgmac->feature_flags |= BGMAC_FEAT_SW_TYPE_RGMII;
- bgmac->feature_flags |= BGMAC_FEAT_IOST_ATTACHED;
- }
- break;
+ switch (ci->id) {
+ /* BCM 471X/535X family */
case BCMA_CHIP_ID_BCM4716:
bgmac->feature_flags |= BGMAC_FEAT_CLKCTLST;
/* fallthrough */
@@ -241,13 +216,19 @@
bgmac->feature_flags |= BGMAC_FEAT_FLW_CTRL2;
bgmac->feature_flags |= BGMAC_FEAT_SET_RXQ_CLK;
break;
- /* bcm4707_family */
- case BCMA_CHIP_ID_BCM4707:
- case BCMA_CHIP_ID_BCM47094:
- case BCMA_CHIP_ID_BCM53018:
+ case BCMA_CHIP_ID_BCM5357:
+ case BCMA_CHIP_ID_BCM53572:
+ bgmac->feature_flags |= BGMAC_FEAT_SET_RXQ_CLK;
bgmac->feature_flags |= BGMAC_FEAT_CLKCTLST;
- bgmac->feature_flags |= BGMAC_FEAT_NO_RESET;
- bgmac->feature_flags |= BGMAC_FEAT_FORCE_SPEED_2500;
+ bgmac->feature_flags |= BGMAC_FEAT_FLW_CTRL1;
+ bgmac->feature_flags |= BGMAC_FEAT_SW_TYPE_PHY;
+ if (ci->pkg == BCMA_PKG_ID_BCM47188 ||
+ ci->pkg == BCMA_PKG_ID_BCM47186) {
+ bgmac->feature_flags |= BGMAC_FEAT_SW_TYPE_RGMII;
+ bgmac->feature_flags |= BGMAC_FEAT_IOST_ATTACHED;
+ }
+ if (ci->pkg == BCMA_PKG_ID_BCM5358)
+ bgmac->feature_flags |= BGMAC_FEAT_SW_TYPE_EPHYRMII;
break;
case BCMA_CHIP_ID_BCM53573:
bgmac->feature_flags |= BGMAC_FEAT_CLKCTLST;
@@ -264,6 +245,24 @@
bgmac->feature_flags |= BGMAC_FEAT_CC7_IF_TYPE_RGMII;
}
break;
+ case BCMA_CHIP_ID_BCM4749:
+ bgmac->feature_flags |= BGMAC_FEAT_SET_RXQ_CLK;
+ bgmac->feature_flags |= BGMAC_FEAT_CLKCTLST;
+ bgmac->feature_flags |= BGMAC_FEAT_FLW_CTRL1;
+ bgmac->feature_flags |= BGMAC_FEAT_SW_TYPE_PHY;
+ if (ci->pkg == 10) {
+ bgmac->feature_flags |= BGMAC_FEAT_SW_TYPE_RGMII;
+ bgmac->feature_flags |= BGMAC_FEAT_IOST_ATTACHED;
+ }
+ break;
+ /* bcm4707_family */
+ case BCMA_CHIP_ID_BCM4707:
+ case BCMA_CHIP_ID_BCM47094:
+ case BCMA_CHIP_ID_BCM53018:
+ bgmac->feature_flags |= BGMAC_FEAT_CLKCTLST;
+ bgmac->feature_flags |= BGMAC_FEAT_NO_RESET;
+ bgmac->feature_flags |= BGMAC_FEAT_FORCE_SPEED_2500;
+ break;
default:
bgmac->feature_flags |= BGMAC_FEAT_CLKCTLST;
bgmac->feature_flags |= BGMAC_FEAT_SET_RXQ_CLK;
diff --git a/drivers/net/ethernet/broadcom/bgmac-platform.c b/drivers/net/ethernet/broadcom/bgmac-platform.c
index da1b8b2..73aca97 100644
--- a/drivers/net/ethernet/broadcom/bgmac-platform.c
+++ b/drivers/net/ethernet/broadcom/bgmac-platform.c
@@ -21,8 +21,12 @@
#include <linux/of_net.h>
#include "bgmac.h"
+#define NICPM_PADRING_CFG 0x00000004
#define NICPM_IOMUX_CTRL 0x00000008
+#define NICPM_PADRING_CFG_INIT_VAL 0x74000000
+#define NICPM_IOMUX_CTRL_INIT_VAL_AX 0x21880000
+
#define NICPM_IOMUX_CTRL_INIT_VAL 0x3196e000
#define NICPM_IOMUX_CTRL_SPD_SHIFT 10
#define NICPM_IOMUX_CTRL_SPD_10M 0
@@ -113,6 +117,10 @@
if (!bgmac->plat.nicpm_base)
return;
+ /* SET RGMII IO CONFIG */
+ writel(NICPM_PADRING_CFG_INIT_VAL,
+ bgmac->plat.nicpm_base + NICPM_PADRING_CFG);
+
val = NICPM_IOMUX_CTRL_INIT_VAL;
switch (bgmac->net_dev->phydev->speed) {
default:
@@ -244,6 +252,31 @@
return 0;
}
+#ifdef CONFIG_PM
+static int bgmac_suspend(struct device *dev)
+{
+ struct bgmac *bgmac = dev_get_drvdata(dev);
+
+ return bgmac_enet_suspend(bgmac);
+}
+
+static int bgmac_resume(struct device *dev)
+{
+ struct bgmac *bgmac = dev_get_drvdata(dev);
+
+ return bgmac_enet_resume(bgmac);
+}
+
+static const struct dev_pm_ops bgmac_pm_ops = {
+ .suspend = bgmac_suspend,
+ .resume = bgmac_resume
+};
+
+#define BGMAC_PM_OPS (&bgmac_pm_ops)
+#else
+#define BGMAC_PM_OPS NULL
+#endif /* CONFIG_PM */
+
static const struct of_device_id bgmac_of_enet_match[] = {
{.compatible = "brcm,amac",},
{.compatible = "brcm,nsp-amac",},
@@ -257,6 +290,7 @@
.driver = {
.name = "bgmac-enet",
.of_match_table = bgmac_of_enet_match,
+ .pm = BGMAC_PM_OPS
},
.probe = bgmac_probe,
.remove = bgmac_remove,
diff --git a/drivers/net/ethernet/broadcom/bgmac.c b/drivers/net/ethernet/broadcom/bgmac.c
index fd66fca..ba4d2e1 100644
--- a/drivers/net/ethernet/broadcom/bgmac.c
+++ b/drivers/net/ethernet/broadcom/bgmac.c
@@ -11,6 +11,7 @@
#include <linux/bcma/bcma.h>
#include <linux/etherdevice.h>
+#include <linux/interrupt.h>
#include <linux/bcm47xx_nvram.h>
#include <linux/phy.h>
#include <linux/phy_fixed.h>
@@ -1480,6 +1481,7 @@
net_dev->irq = bgmac->irq;
SET_NETDEV_DEV(net_dev, bgmac->dev);
+ dev_set_drvdata(bgmac->dev, bgmac);
if (!is_valid_ether_addr(net_dev->dev_addr)) {
dev_err(bgmac->dev, "Invalid MAC addr: %pM\n",
@@ -1552,5 +1554,55 @@
}
EXPORT_SYMBOL_GPL(bgmac_enet_remove);
+int bgmac_enet_suspend(struct bgmac *bgmac)
+{
+ if (!netif_running(bgmac->net_dev))
+ return 0;
+
+ phy_stop(bgmac->net_dev->phydev);
+
+ netif_stop_queue(bgmac->net_dev);
+
+ napi_disable(&bgmac->napi);
+
+ netif_tx_lock(bgmac->net_dev);
+ netif_device_detach(bgmac->net_dev);
+ netif_tx_unlock(bgmac->net_dev);
+
+ bgmac_chip_intrs_off(bgmac);
+ bgmac_chip_reset(bgmac);
+ bgmac_dma_cleanup(bgmac);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(bgmac_enet_suspend);
+
+int bgmac_enet_resume(struct bgmac *bgmac)
+{
+ int rc;
+
+ if (!netif_running(bgmac->net_dev))
+ return 0;
+
+ rc = bgmac_dma_init(bgmac);
+ if (rc)
+ return rc;
+
+ bgmac_chip_init(bgmac);
+
+ napi_enable(&bgmac->napi);
+
+ netif_tx_lock(bgmac->net_dev);
+ netif_device_attach(bgmac->net_dev);
+ netif_tx_unlock(bgmac->net_dev);
+
+ netif_start_queue(bgmac->net_dev);
+
+ phy_start(bgmac->net_dev->phydev);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(bgmac_enet_resume);
+
MODULE_AUTHOR("Rafał Miłecki");
MODULE_LICENSE("GPL");
diff --git a/drivers/net/ethernet/broadcom/bgmac.h b/drivers/net/ethernet/broadcom/bgmac.h
index 6d1c6ff..c181876 100644
--- a/drivers/net/ethernet/broadcom/bgmac.h
+++ b/drivers/net/ethernet/broadcom/bgmac.h
@@ -402,7 +402,7 @@
#define BGMAC_WEIGHT 64
-#define ETHER_MAX_LEN 1518
+#define ETHER_MAX_LEN (ETH_FRAME_LEN + ETH_FCS_LEN)
/* Feature Flags */
#define BGMAC_FEAT_TX_MASK_SETUP BIT(0)
@@ -537,6 +537,8 @@
void bgmac_enet_remove(struct bgmac *bgmac);
void bgmac_adjust_link(struct net_device *net_dev);
int bgmac_phy_connect_direct(struct bgmac *bgmac);
+int bgmac_enet_suspend(struct bgmac *bgmac);
+int bgmac_enet_resume(struct bgmac *bgmac);
struct mii_bus *bcma_mdio_mii_register(struct bgmac *bgmac);
void bcma_mdio_mii_unregister(struct mii_bus *mii_bus);
diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c
index 9e8c061..eccb3d1 100644
--- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c
+++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c
@@ -2021,6 +2021,7 @@
ETH_OVERHEAD +
mtu +
BNX2X_FW_RX_ALIGN_END;
+ fp->rx_buf_size = SKB_DATA_ALIGN(fp->rx_buf_size);
/* Note : rx_buf_size doesn't take into account NET_SKB_PAD */
if (fp->rx_buf_size + NET_SKB_PAD <= PAGE_SIZE)
fp->rx_frag_size = fp->rx_buf_size + NET_SKB_PAD;
@@ -4277,7 +4278,10 @@
{
if (tc->type != TC_SETUP_MQPRIO)
return -EINVAL;
- return bnx2x_setup_tc(dev, tc->tc);
+
+ tc->mqprio->hw = TC_MQPRIO_HW_OFFLOAD_TCS;
+
+ return bnx2x_setup_tc(dev, tc->mqprio->num_tc);
}
/* called with rtnl_lock */
diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_link.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_link.c
index b209b7f..7dd83d0 100644
--- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_link.c
+++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_link.c
@@ -6161,103 +6161,40 @@
}
}
+static int bnx2x_null_format_ver(u32 spirom_ver, u8 *str, u16 *len)
+{
+ str[0] = '\0';
+ (*len)--;
+ return 0;
+}
+
static int bnx2x_format_ver(u32 num, u8 *str, u16 *len)
{
- u8 *str_ptr = str;
- u32 mask = 0xf0000000;
- u8 shift = 8*4;
- u8 digit;
- u8 remove_leading_zeros = 1;
+ u16 ret;
+
if (*len < 10) {
/* Need more than 10chars for this format */
- *str_ptr = '\0';
- (*len)--;
+ bnx2x_null_format_ver(num, str, len);
return -EINVAL;
}
- while (shift > 0) {
- shift -= 4;
- digit = ((num & mask) >> shift);
- if (digit == 0 && remove_leading_zeros) {
- *str_ptr = '0';
- } else {
- if (digit < 0xa)
- *str_ptr = digit + '0';
- else
- *str_ptr = digit - 0xa + 'a';
-
- remove_leading_zeros = 0;
- str_ptr++;
- (*len)--;
- }
- mask = mask >> 4;
- if (shift == 4*4) {
- if (remove_leading_zeros) {
- str_ptr++;
- (*len)--;
- }
- *str_ptr = '.';
- str_ptr++;
- (*len)--;
- remove_leading_zeros = 1;
- }
- }
- if (remove_leading_zeros)
- (*len)--;
+ ret = scnprintf(str, *len, "%hx.%hx", num >> 16, num);
+ *len -= ret;
return 0;
}
static int bnx2x_3_seq_format_ver(u32 num, u8 *str, u16 *len)
{
- u8 *str_ptr = str;
- u32 mask = 0x00f00000;
- u8 shift = 8*3;
- u8 digit;
- u8 remove_leading_zeros = 1;
+ u16 ret;
if (*len < 10) {
/* Need more than 10chars for this format */
- *str_ptr = '\0';
- (*len)--;
+ bnx2x_null_format_ver(num, str, len);
return -EINVAL;
}
- while (shift > 0) {
- shift -= 4;
- digit = ((num & mask) >> shift);
- if (digit == 0 && remove_leading_zeros) {
- *str_ptr = '0';
- } else {
- if (digit < 0xa)
- *str_ptr = digit + '0';
- else
- *str_ptr = digit - 0xa + 'a';
-
- remove_leading_zeros = 0;
- str_ptr++;
- (*len)--;
- }
- mask = mask >> 4;
- if ((shift == 4*4) || (shift == 4*2)) {
- if (remove_leading_zeros) {
- str_ptr++;
- (*len)--;
- }
- *str_ptr = '.';
- str_ptr++;
- (*len)--;
- remove_leading_zeros = 1;
- }
- }
- if (remove_leading_zeros)
- (*len)--;
- return 0;
-}
-
-static int bnx2x_null_format_ver(u32 spirom_ver, u8 *str, u16 *len)
-{
- str[0] = '\0';
- (*len)--;
+ ret = scnprintf(str, *len, "%hhx.%hhx.%hhx", num >> 16, num >> 8, num);
+ *len -= ret;
return 0;
}
@@ -10681,22 +10618,19 @@
static int bnx2x_8485x_format_ver(u32 raw_ver, u8 *str, u16 *len)
{
- int status = 0;
u32 num;
num = ((raw_ver & 0xF80) >> 7) << 16 | ((raw_ver & 0x7F) << 8) |
((raw_ver & 0xF000) >> 12);
- status = bnx2x_3_seq_format_ver(num, str, len);
- return status;
+ return bnx2x_3_seq_format_ver(num, str, len);
}
static int bnx2x_848xx_format_ver(u32 raw_ver, u8 *str, u16 *len)
{
- int status = 0;
u32 spirom_ver;
+
spirom_ver = ((raw_ver & 0xF80) >> 7) << 16 | (raw_ver & 0x7F);
- status = bnx2x_format_ver(spirom_ver, str, len);
- return status;
+ return bnx2x_format_ver(spirom_ver, str, len);
}
static void bnx2x_8481_hw_reset(struct bnx2x_phy *phy,
@@ -12547,13 +12481,12 @@
static int bnx2x_populate_phy(struct bnx2x *bp, u8 phy_index, u32 shmem_base,
u32 shmem2_base, u8 port, struct bnx2x_phy *phy)
{
- int status = 0;
phy->type = PORT_HW_CFG_XGXS_EXT_PHY_TYPE_NOT_CONN;
if (phy_index == INT_PHY)
return bnx2x_populate_int_phy(bp, shmem_base, port, phy);
- status = bnx2x_populate_ext_phy(bp, phy_index, shmem_base, shmem2_base,
+
+ return bnx2x_populate_ext_phy(bp, phy_index, shmem_base, shmem2_base,
port, phy);
- return status;
}
static void bnx2x_phy_def_cfg(struct link_params *params,
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 1f1e54b..b3ba660 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -4482,9 +4482,15 @@
vf->vlan = le16_to_cpu(resp->vlan) & VLAN_VID_MASK;
}
#endif
- if (BNXT_PF(bp) && (le16_to_cpu(resp->flags) &
- FUNC_QCFG_RESP_FLAGS_FW_DCBX_AGENT_ENABLED))
- bp->flags |= BNXT_FLAG_FW_LLDP_AGENT;
+ if (BNXT_PF(bp)) {
+ u16 flags = le16_to_cpu(resp->flags);
+
+ if (flags & (FUNC_QCFG_RESP_FLAGS_FW_DCBX_AGENT_ENABLED |
+ FUNC_QCFG_RESP_FLAGS_FW_LLDP_AGENT_ENABLED))
+ bp->flags |= BNXT_FLAG_FW_LLDP_AGENT;
+ if (flags & FUNC_QCFG_RESP_FLAGS_MULTI_HOST)
+ bp->flags |= BNXT_FLAG_MULTI_HOST;
+ }
switch (resp->port_partition_type) {
case FUNC_QCFG_RESP_PORT_PARTITION_TYPE_NPAR1_0:
@@ -4549,6 +4555,9 @@
pf->max_tx_wm_flows = le32_to_cpu(resp->max_tx_wm_flows);
pf->max_rx_em_flows = le32_to_cpu(resp->max_rx_em_flows);
pf->max_rx_wm_flows = le32_to_cpu(resp->max_rx_wm_flows);
+ if (resp->flags &
+ cpu_to_le32(FUNC_QCAPS_RESP_FLAGS_WOL_MAGICPKT_SUPPORTED))
+ bp->flags |= BNXT_FLAG_WOL_CAP;
} else {
#ifdef CONFIG_BNXT_SRIOV
struct bnxt_vf_info *vf = &bp->vf;
@@ -5198,9 +5207,10 @@
{
#if defined(CONFIG_BNXT_SRIOV)
if (BNXT_VF(bp))
- return bp->vf.max_irqs;
+ return min_t(unsigned int, bp->vf.max_irqs,
+ bp->vf.max_cp_rings);
#endif
- return bp->pf.max_irqs;
+ return min_t(unsigned int, bp->pf.max_irqs, bp->pf.max_cp_rings);
}
void bnxt_set_max_func_irqs(struct bnxt *bp, unsigned int max_irqs)
@@ -5467,7 +5477,8 @@
if (bp->link_info.link_up) {
const char *duplex;
const char *flow_ctrl;
- u16 speed, fec;
+ u32 speed;
+ u16 fec;
netif_carrier_on(bp->dev);
if (bp->link_info.duplex == BNXT_LINK_DUPLEX_FULL)
@@ -5483,7 +5494,7 @@
else
flow_ctrl = "none";
speed = bnxt_fw_to_ethtool_speed(bp->link_info.link_speed);
- netdev_info(bp->dev, "NIC Link is Up, %d Mbps %s duplex, Flow control: %s\n",
+ netdev_info(bp->dev, "NIC Link is Up, %u Mbps %s duplex, Flow control: %s\n",
speed, duplex, flow_ctrl);
if (bp->flags & BNXT_FLAG_EEE_CAP)
netdev_info(bp->dev, "EEE is %s\n",
@@ -5857,6 +5868,76 @@
return 0;
}
+int bnxt_hwrm_alloc_wol_fltr(struct bnxt *bp)
+{
+ struct hwrm_wol_filter_alloc_input req = {0};
+ struct hwrm_wol_filter_alloc_output *resp = bp->hwrm_cmd_resp_addr;
+ int rc;
+
+ bnxt_hwrm_cmd_hdr_init(bp, &req, HWRM_WOL_FILTER_ALLOC, -1, -1);
+ req.port_id = cpu_to_le16(bp->pf.port_id);
+ req.wol_type = WOL_FILTER_ALLOC_REQ_WOL_TYPE_MAGICPKT;
+ req.enables = cpu_to_le32(WOL_FILTER_ALLOC_REQ_ENABLES_MAC_ADDRESS);
+ memcpy(req.mac_address, bp->dev->dev_addr, ETH_ALEN);
+ mutex_lock(&bp->hwrm_cmd_lock);
+ rc = _hwrm_send_message(bp, &req, sizeof(req), HWRM_CMD_TIMEOUT);
+ if (!rc)
+ bp->wol_filter_id = resp->wol_filter_id;
+ mutex_unlock(&bp->hwrm_cmd_lock);
+ return rc;
+}
+
+int bnxt_hwrm_free_wol_fltr(struct bnxt *bp)
+{
+ struct hwrm_wol_filter_free_input req = {0};
+ int rc;
+
+ bnxt_hwrm_cmd_hdr_init(bp, &req, HWRM_WOL_FILTER_FREE, -1, -1);
+ req.port_id = cpu_to_le16(bp->pf.port_id);
+ req.enables = cpu_to_le32(WOL_FILTER_FREE_REQ_ENABLES_WOL_FILTER_ID);
+ req.wol_filter_id = bp->wol_filter_id;
+ rc = hwrm_send_message(bp, &req, sizeof(req), HWRM_CMD_TIMEOUT);
+ return rc;
+}
+
+static u16 bnxt_hwrm_get_wol_fltrs(struct bnxt *bp, u16 handle)
+{
+ struct hwrm_wol_filter_qcfg_input req = {0};
+ struct hwrm_wol_filter_qcfg_output *resp = bp->hwrm_cmd_resp_addr;
+ u16 next_handle = 0;
+ int rc;
+
+ bnxt_hwrm_cmd_hdr_init(bp, &req, HWRM_WOL_FILTER_QCFG, -1, -1);
+ req.port_id = cpu_to_le16(bp->pf.port_id);
+ req.handle = cpu_to_le16(handle);
+ mutex_lock(&bp->hwrm_cmd_lock);
+ rc = _hwrm_send_message(bp, &req, sizeof(req), HWRM_CMD_TIMEOUT);
+ if (!rc) {
+ next_handle = le16_to_cpu(resp->next_handle);
+ if (next_handle != 0) {
+ if (resp->wol_type ==
+ WOL_FILTER_ALLOC_REQ_WOL_TYPE_MAGICPKT) {
+ bp->wol = 1;
+ bp->wol_filter_id = resp->wol_filter_id;
+ }
+ }
+ }
+ mutex_unlock(&bp->hwrm_cmd_lock);
+ return next_handle;
+}
+
+static void bnxt_get_wol_settings(struct bnxt *bp)
+{
+ u16 handle = 0;
+
+ if (!BNXT_PF(bp) || !(bp->flags & BNXT_FLAG_WOL_CAP))
+ return;
+
+ do {
+ handle = bnxt_hwrm_get_wol_fltrs(bp, handle);
+ } while (handle && handle != 0xffff);
+}
+
static bool bnxt_eee_config_ok(struct bnxt *bp)
{
struct ethtool_eee *eee = &bp->eee;
@@ -6042,6 +6123,43 @@
return rc;
}
+/* rtnl_lock held, open the NIC half way by allocating all resources, but
+ * NAPI, IRQ, and TX are not enabled. This is mainly used for offline
+ * self tests.
+ */
+int bnxt_half_open_nic(struct bnxt *bp)
+{
+ int rc = 0;
+
+ rc = bnxt_alloc_mem(bp, false);
+ if (rc) {
+ netdev_err(bp->dev, "bnxt_alloc_mem err: %x\n", rc);
+ goto half_open_err;
+ }
+ rc = bnxt_init_nic(bp, false);
+ if (rc) {
+ netdev_err(bp->dev, "bnxt_init_nic err: %x\n", rc);
+ goto half_open_err;
+ }
+ return 0;
+
+half_open_err:
+ bnxt_free_skbs(bp);
+ bnxt_free_mem(bp, false);
+ dev_close(bp->dev);
+ return rc;
+}
+
+/* rtnl_lock held, this call can only be made after a previous successful
+ * call to bnxt_half_open_nic().
+ */
+void bnxt_half_close_nic(struct bnxt *bp)
+{
+ bnxt_hwrm_resource_free(bp, false, false);
+ bnxt_free_skbs(bp);
+ bnxt_free_mem(bp, false);
+}
+
static int bnxt_open(struct net_device *dev)
{
struct bnxt *bp = netdev_priv(dev);
@@ -6923,7 +7041,9 @@
if (ntc->type != TC_SETUP_MQPRIO)
return -EINVAL;
- return bnxt_setup_mq_tc(dev, ntc->tc);
+ ntc->mqprio->hw = TC_MQPRIO_HW_OFFLOAD_TCS;
+
+ return bnxt_setup_mq_tc(dev, ntc->mqprio->num_tc);
}
#ifdef CONFIG_RFS_ACCEL
@@ -7224,6 +7344,7 @@
bnxt_clear_int_mode(bp);
bnxt_hwrm_func_drv_unrgtr(bp);
bnxt_free_hwrm_resources(bp);
+ bnxt_ethtool_free(bp);
bnxt_dcb_free(bp);
kfree(bp->edev);
bp->edev = NULL;
@@ -7546,6 +7667,7 @@
bnxt_hwrm_func_qcfg(bp);
bnxt_hwrm_port_led_qcaps(bp);
+ bnxt_ethtool_init(bp);
bnxt_set_rx_skb_mode(bp, false);
bnxt_set_tpa_flags(bp);
@@ -7591,6 +7713,12 @@
if (rc)
goto init_err_pci_clean;
+ bnxt_get_wol_settings(bp);
+ if (bp->flags & BNXT_FLAG_WOL_CAP)
+ device_set_wakeup_enable(&pdev->dev, bp->wol);
+ else
+ device_set_wakeup_capable(&pdev->dev, false);
+
rc = register_netdev(dev);
if (rc)
goto init_err_clr_int;
@@ -7614,6 +7742,88 @@
return rc;
}
+static void bnxt_shutdown(struct pci_dev *pdev)
+{
+ struct net_device *dev = pci_get_drvdata(pdev);
+ struct bnxt *bp;
+
+ if (!dev)
+ return;
+
+ rtnl_lock();
+ bp = netdev_priv(dev);
+ if (!bp)
+ goto shutdown_exit;
+
+ if (netif_running(dev))
+ dev_close(dev);
+
+ if (system_state == SYSTEM_POWER_OFF) {
+ bnxt_clear_int_mode(bp);
+ pci_wake_from_d3(pdev, bp->wol);
+ pci_set_power_state(pdev, PCI_D3hot);
+ }
+
+shutdown_exit:
+ rtnl_unlock();
+}
+
+#ifdef CONFIG_PM_SLEEP
+static int bnxt_suspend(struct device *device)
+{
+ struct pci_dev *pdev = to_pci_dev(device);
+ struct net_device *dev = pci_get_drvdata(pdev);
+ struct bnxt *bp = netdev_priv(dev);
+ int rc = 0;
+
+ rtnl_lock();
+ if (netif_running(dev)) {
+ netif_device_detach(dev);
+ rc = bnxt_close(dev);
+ }
+ bnxt_hwrm_func_drv_unrgtr(bp);
+ rtnl_unlock();
+ return rc;
+}
+
+static int bnxt_resume(struct device *device)
+{
+ struct pci_dev *pdev = to_pci_dev(device);
+ struct net_device *dev = pci_get_drvdata(pdev);
+ struct bnxt *bp = netdev_priv(dev);
+ int rc = 0;
+
+ rtnl_lock();
+ if (bnxt_hwrm_ver_get(bp) || bnxt_hwrm_func_drv_rgtr(bp)) {
+ rc = -ENODEV;
+ goto resume_exit;
+ }
+ rc = bnxt_hwrm_func_reset(bp);
+ if (rc) {
+ rc = -EBUSY;
+ goto resume_exit;
+ }
+ bnxt_get_wol_settings(bp);
+ if (netif_running(dev)) {
+ rc = bnxt_open(dev);
+ if (!rc)
+ netif_device_attach(dev);
+ }
+
+resume_exit:
+ rtnl_unlock();
+ return rc;
+}
+
+static SIMPLE_DEV_PM_OPS(bnxt_pm_ops, bnxt_suspend, bnxt_resume);
+#define BNXT_PM_OPS (&bnxt_pm_ops)
+
+#else
+
+#define BNXT_PM_OPS NULL
+
+#endif /* CONFIG_PM_SLEEP */
+
/**
* bnxt_io_error_detected - called when PCI error is detected
* @pdev: Pointer to PCI device
@@ -7730,6 +7940,8 @@
.id_table = bnxt_pci_tbl,
.probe = bnxt_init_one,
.remove = bnxt_remove_one,
+ .shutdown = bnxt_shutdown,
+ .driver.pm = BNXT_PM_OPS,
.err_handler = &bnxt_err_handler,
#if defined(CONFIG_BNXT_SRIOV)
.sriov_configure = bnxt_sriov_configure,
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
index c7a5b84..3ef42db 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
@@ -18,6 +18,8 @@
#define DRV_VER_MIN 7
#define DRV_VER_UPD 0
+#include <linux/interrupt.h>
+
struct tx_bd {
__le32 tx_bd_len_flags_type;
#define TX_BD_TYPE (0x3f << 0)
@@ -424,8 +426,6 @@
#define BNXT_MIN_PKT_SIZE 52
-#define BNXT_NUM_TESTS(bp) 0
-
#define BNXT_DEFAULT_RX_RING_SIZE 511
#define BNXT_DEFAULT_TX_RING_SIZE 511
@@ -851,6 +851,7 @@
#define BNXT_LINK_SPEED_25GB PORT_PHY_QCFG_RESP_LINK_SPEED_25GB
#define BNXT_LINK_SPEED_40GB PORT_PHY_QCFG_RESP_LINK_SPEED_40GB
#define BNXT_LINK_SPEED_50GB PORT_PHY_QCFG_RESP_LINK_SPEED_50GB
+#define BNXT_LINK_SPEED_100GB PORT_PHY_QCFG_RESP_LINK_SPEED_100GB
u16 support_speeds;
u16 auto_link_speeds; /* fw adv setting */
#define BNXT_LINK_SPEED_MSK_100MB PORT_PHY_QCFG_RESP_SUPPORT_SPEEDS_100MB
@@ -862,6 +863,7 @@
#define BNXT_LINK_SPEED_MSK_25GB PORT_PHY_QCFG_RESP_SUPPORT_SPEEDS_25GB
#define BNXT_LINK_SPEED_MSK_40GB PORT_PHY_QCFG_RESP_SUPPORT_SPEEDS_40GB
#define BNXT_LINK_SPEED_MSK_50GB PORT_PHY_QCFG_RESP_SUPPORT_SPEEDS_50GB
+#define BNXT_LINK_SPEED_MSK_100GB PORT_PHY_QCFG_RESP_SUPPORT_SPEEDS_100GB
u16 support_auto_speeds;
u16 lp_auto_link_speeds;
u16 force_link_speed;
@@ -909,6 +911,14 @@
__le16 led_color_caps;
};
+#define BNXT_MAX_TEST 8
+
+struct bnxt_test_info {
+ u8 offline_mask;
+ u16 timeout;
+ char string[BNXT_MAX_TEST][ETH_GSTRING_LEN];
+};
+
#define BNXT_GRCPF_REG_WINDOW_BASE_OUT 0x400
#define BNXT_CAG_REG_LEGACY_INT_STATUS 0x4014
#define BNXT_CAG_REG_BASE 0x300000
@@ -987,6 +997,7 @@
#define BNXT_FLAG_UDP_RSS_CAP 0x800
#define BNXT_FLAG_EEE_CAP 0x1000
#define BNXT_FLAG_NEW_RSS_CAP 0x2000
+ #define BNXT_FLAG_WOL_CAP 0x4000
#define BNXT_FLAG_ROCEV1_CAP 0x8000
#define BNXT_FLAG_ROCEV2_CAP 0x10000
#define BNXT_FLAG_ROCE_CAP (BNXT_FLAG_ROCEV1_CAP | \
@@ -994,6 +1005,7 @@
#define BNXT_FLAG_NO_AGG_RINGS 0x20000
#define BNXT_FLAG_RX_PAGE_MODE 0x40000
#define BNXT_FLAG_FW_LLDP_AGENT 0x80000
+ #define BNXT_FLAG_MULTI_HOST 0x100000
#define BNXT_FLAG_CHIP_NITRO_A0 0x1000000
#define BNXT_FLAG_ALL_CONFIG_FEATS (BNXT_FLAG_TPA | \
@@ -1003,7 +1015,8 @@
#define BNXT_PF(bp) (!((bp)->flags & BNXT_FLAG_VF))
#define BNXT_VF(bp) ((bp)->flags & BNXT_FLAG_VF)
#define BNXT_NPAR(bp) ((bp)->port_partition_type)
-#define BNXT_SINGLE_PF(bp) (BNXT_PF(bp) && !BNXT_NPAR(bp))
+#define BNXT_MH(bp) ((bp)->flags & BNXT_FLAG_MULTI_HOST)
+#define BNXT_SINGLE_PF(bp) (BNXT_PF(bp) && !BNXT_NPAR(bp) && !BNXT_MH(bp))
#define BNXT_CHIP_TYPE_NITRO_A0(bp) ((bp)->flags & BNXT_FLAG_CHIP_NITRO_A0)
#define BNXT_RX_PAGE_MODE(bp) ((bp)->flags & BNXT_FLAG_RX_PAGE_MODE)
@@ -1178,6 +1191,12 @@
u32 lpi_tmr_lo;
u32 lpi_tmr_hi;
+ u8 num_tests;
+ struct bnxt_test_info *test_info;
+
+ u8 wol_filter_id;
+ u8 wol;
+
u8 num_leds;
struct bnxt_led_info leds[BNXT_MAX_LED];
@@ -1236,8 +1255,12 @@
void bnxt_tx_enable(struct bnxt *bp);
int bnxt_hwrm_set_pause(struct bnxt *);
int bnxt_hwrm_set_link_setting(struct bnxt *, bool, bool);
+int bnxt_hwrm_alloc_wol_fltr(struct bnxt *bp);
+int bnxt_hwrm_free_wol_fltr(struct bnxt *bp);
int bnxt_hwrm_fw_set_time(struct bnxt *);
int bnxt_open_nic(struct bnxt *, bool, bool);
+int bnxt_half_open_nic(struct bnxt *bp);
+void bnxt_half_close_nic(struct bnxt *bp);
int bnxt_close_nic(struct bnxt *, bool, bool);
int bnxt_reserve_rings(struct bnxt *bp, int tx, int rx, int tcs, int tx_xdp);
int bnxt_setup_mq_tc(struct net_device *dev, u8 tc);
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.c
index 03532061..46de2f8 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.c
@@ -1,6 +1,7 @@
/* Broadcom NetXtreme-C/E network driver.
*
* Copyright (c) 2014-2016 Broadcom Corporation
+ * Copyright (c) 2016-2017 Broadcom Limited
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -14,6 +15,7 @@
#include <linux/interrupt.h>
#include <linux/pci.h>
#include <linux/etherdevice.h>
+#include <rdma/ib_verbs.h>
#include "bnxt_hsi.h"
#include "bnxt.h"
#include "bnxt_dcb.h"
@@ -241,6 +243,92 @@
return 0;
}
+static int bnxt_hwrm_set_dcbx_app(struct bnxt *bp, struct dcb_app *app,
+ bool add)
+{
+ struct hwrm_fw_set_structured_data_input set = {0};
+ struct hwrm_fw_get_structured_data_input get = {0};
+ struct hwrm_struct_data_dcbx_app *fw_app;
+ struct hwrm_struct_hdr *data;
+ dma_addr_t mapping;
+ size_t data_len;
+ int rc, n, i;
+
+ if (bp->hwrm_spec_code < 0x10601)
+ return 0;
+
+ n = IEEE_8021QAZ_MAX_TCS;
+ data_len = sizeof(*data) + sizeof(*fw_app) * n;
+ data = dma_alloc_coherent(&bp->pdev->dev, data_len, &mapping,
+ GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+
+ memset(data, 0, data_len);
+ bnxt_hwrm_cmd_hdr_init(bp, &get, HWRM_FW_GET_STRUCTURED_DATA, -1, -1);
+ get.dest_data_addr = cpu_to_le64(mapping);
+ get.structure_id = cpu_to_le16(STRUCT_HDR_STRUCT_ID_DCBX_APP);
+ get.subtype = cpu_to_le16(HWRM_STRUCT_DATA_SUBTYPE_HOST_OPERATIONAL);
+ get.count = 0;
+ rc = hwrm_send_message(bp, &get, sizeof(get), HWRM_CMD_TIMEOUT);
+ if (rc)
+ goto set_app_exit;
+
+ fw_app = (struct hwrm_struct_data_dcbx_app *)(data + 1);
+
+ if (data->struct_id != cpu_to_le16(STRUCT_HDR_STRUCT_ID_DCBX_APP)) {
+ rc = -ENODEV;
+ goto set_app_exit;
+ }
+
+ n = data->count;
+ for (i = 0; i < n; i++, fw_app++) {
+ if (fw_app->protocol_id == cpu_to_be16(app->protocol) &&
+ fw_app->protocol_selector == app->selector &&
+ fw_app->priority == app->priority) {
+ if (add)
+ goto set_app_exit;
+ else
+ break;
+ }
+ }
+ if (add) {
+ /* append */
+ n++;
+ fw_app->protocol_id = cpu_to_be16(app->protocol);
+ fw_app->protocol_selector = app->selector;
+ fw_app->priority = app->priority;
+ fw_app->valid = 1;
+ } else {
+ size_t len = 0;
+
+ /* not found, nothing to delete */
+ if (n == i)
+ goto set_app_exit;
+
+ len = (n - 1 - i) * sizeof(*fw_app);
+ if (len)
+ memmove(fw_app, fw_app + 1, len);
+ n--;
+ memset(fw_app + n, 0, sizeof(*fw_app));
+ }
+ data->count = n;
+ data->len = cpu_to_le16(sizeof(*fw_app) * n);
+ data->subtype = cpu_to_le16(HWRM_STRUCT_DATA_SUBTYPE_HOST_OPERATIONAL);
+
+ bnxt_hwrm_cmd_hdr_init(bp, &set, HWRM_FW_SET_STRUCTURED_DATA, -1, -1);
+ set.src_data_addr = cpu_to_le64(mapping);
+ set.data_len = cpu_to_le16(sizeof(*data) + sizeof(*fw_app) * n);
+ set.hdr_cnt = 1;
+ rc = hwrm_send_message(bp, &set, sizeof(set), HWRM_CMD_TIMEOUT);
+ if (rc)
+ rc = -EIO;
+
+set_app_exit:
+ dma_free_coherent(&bp->pdev->dev, data_len, data, mapping);
+ return rc;
+}
+
static int bnxt_ets_validate(struct bnxt *bp, struct ieee_ets *ets, u8 *tc)
{
int total_ets_bw = 0;
@@ -417,6 +505,15 @@
return -EINVAL;
rc = dcb_ieee_setapp(dev, app);
+ if (rc)
+ return rc;
+
+ if ((app->selector == IEEE_8021QAZ_APP_SEL_ETHERTYPE &&
+ app->protocol == ETH_P_IBOE) ||
+ (app->selector == IEEE_8021QAZ_APP_SEL_DGRAM &&
+ app->protocol == ROCE_V2_UDP_DPORT))
+ rc = bnxt_hwrm_set_dcbx_app(bp, app, true);
+
return rc;
}
@@ -425,10 +522,19 @@
struct bnxt *bp = netdev_priv(dev);
int rc;
- if (!(bp->dcbx_cap & DCB_CAP_DCBX_VER_IEEE))
+ if (!(bp->dcbx_cap & DCB_CAP_DCBX_VER_IEEE) ||
+ !(bp->dcbx_cap & DCB_CAP_DCBX_HOST))
return -EINVAL;
rc = dcb_ieee_delapp(dev, app);
+ if (rc)
+ return rc;
+ if ((app->selector == IEEE_8021QAZ_APP_SEL_ETHERTYPE &&
+ app->protocol == ETH_P_IBOE) ||
+ (app->selector == IEEE_8021QAZ_APP_SEL_DGRAM &&
+ app->protocol == ROCE_V2_UDP_DPORT))
+ rc = bnxt_hwrm_set_dcbx_app(bp, app, false);
+
return rc;
}
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.h
index 35a0d28..ecd0a5e 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_dcb.h
@@ -1,6 +1,7 @@
/* Broadcom NetXtreme-C/E network driver.
*
* Copyright (c) 2014-2016 Broadcom Corporation
+ * Copyright (c) 2016-2017 Broadcom Limited
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
index 6903a87..11ddf0a 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
@@ -1,6 +1,7 @@
/* Broadcom NetXtreme-C/E network driver.
*
* Copyright (c) 2014-2016 Broadcom Corporation
+ * Copyright (c) 2016-2017 Broadcom Limited
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -17,6 +18,7 @@
#include <linux/firmware.h>
#include "bnxt_hsi.h"
#include "bnxt.h"
+#include "bnxt_xdp.h"
#include "bnxt_ethtool.h"
#include "bnxt_nvm_defs.h" /* NVRAM content constant and structure defs */
#include "bnxt_fw_hdr.h" /* Firmware hdr constant and structure defs */
@@ -209,6 +211,10 @@
return num_stats;
}
+ case ETH_SS_TEST:
+ if (!bp->num_tests)
+ return -EOPNOTSUPP;
+ return bp->num_tests;
default:
return -EOPNOTSUPP;
}
@@ -306,6 +312,11 @@
}
}
break;
+ case ETH_SS_TEST:
+ if (bp->num_tests)
+ memcpy(buf, bp->test_info->string,
+ bp->num_tests * ETH_GSTRING_LEN);
+ break;
default:
netdev_err(bp->dev, "bnxt_get_strings invalid request %x\n",
stringset);
@@ -824,7 +835,7 @@
sizeof(info->fw_version));
strlcpy(info->bus_info, pci_name(bp->pdev), sizeof(info->bus_info));
info->n_stats = BNXT_NUM_STATS * bp->cp_nr_rings;
- info->testinfo_len = BNXT_NUM_TESTS(bp);
+ info->testinfo_len = bp->num_tests;
/* TODO CHIMP_FW: eeprom dump details */
info->eedump_len = 0;
/* TODO CHIMP FW: reg dump details */
@@ -832,6 +843,45 @@
kfree(pkglog);
}
+static void bnxt_get_wol(struct net_device *dev, struct ethtool_wolinfo *wol)
+{
+ struct bnxt *bp = netdev_priv(dev);
+
+ wol->supported = 0;
+ wol->wolopts = 0;
+ memset(&wol->sopass, 0, sizeof(wol->sopass));
+ if (bp->flags & BNXT_FLAG_WOL_CAP) {
+ wol->supported = WAKE_MAGIC;
+ if (bp->wol)
+ wol->wolopts = WAKE_MAGIC;
+ }
+}
+
+static int bnxt_set_wol(struct net_device *dev, struct ethtool_wolinfo *wol)
+{
+ struct bnxt *bp = netdev_priv(dev);
+
+ if (wol->wolopts & ~WAKE_MAGIC)
+ return -EINVAL;
+
+ if (wol->wolopts & WAKE_MAGIC) {
+ if (!(bp->flags & BNXT_FLAG_WOL_CAP))
+ return -EINVAL;
+ if (!bp->wol) {
+ if (bnxt_hwrm_alloc_wol_fltr(bp))
+ return -EBUSY;
+ bp->wol = 1;
+ }
+ } else {
+ if (bp->wol) {
+ if (bnxt_hwrm_free_wol_fltr(bp))
+ return -EBUSY;
+ bp->wol = 0;
+ }
+ }
+ return 0;
+}
+
u32 _bnxt_fw_to_ethtool_adv_spds(u16 fw_speeds, u8 fw_pause)
{
u32 speed_mask = 0;
@@ -879,6 +929,9 @@
if ((fw_speeds) & BNXT_LINK_SPEED_MSK_50GB) \
ethtool_link_ksettings_add_link_mode(lk_ksettings, name,\
50000baseCR2_Full);\
+ if ((fw_speeds) & BNXT_LINK_SPEED_MSK_100GB) \
+ ethtool_link_ksettings_add_link_mode(lk_ksettings, name,\
+ 100000baseCR4_Full);\
if ((fw_pause) & BNXT_LINK_PAUSE_RX) { \
ethtool_link_ksettings_add_link_mode(lk_ksettings, name,\
Pause); \
@@ -915,6 +968,9 @@
if (ethtool_link_ksettings_test_link_mode(lk_ksettings, name, \
50000baseCR2_Full)) \
(fw_speeds) |= BNXT_LINK_SPEED_MSK_50GB; \
+ if (ethtool_link_ksettings_test_link_mode(lk_ksettings, name, \
+ 100000baseCR4_Full)) \
+ (fw_speeds) |= BNXT_LINK_SPEED_MSK_100GB; \
}
static void bnxt_fw_to_ethtool_advertised_spds(struct bnxt_link_info *link_info,
@@ -977,6 +1033,8 @@
return SPEED_40000;
case BNXT_LINK_SPEED_50GB:
return SPEED_50000;
+ case BNXT_LINK_SPEED_100GB:
+ return SPEED_100000;
default:
return SPEED_UNKNOWN;
}
@@ -1042,7 +1100,7 @@
return 0;
}
-static u32 bnxt_get_fw_speed(struct net_device *dev, u16 ethtool_speed)
+static u32 bnxt_get_fw_speed(struct net_device *dev, u32 ethtool_speed)
{
struct bnxt *bp = netdev_priv(dev);
struct bnxt_link_info *link_info = &bp->link_info;
@@ -1082,6 +1140,10 @@
if (support_spds & BNXT_LINK_SPEED_MSK_50GB)
fw_speed = PORT_PHY_CFG_REQ_AUTO_LINK_SPEED_50GB;
break;
+ case SPEED_100000:
+ if (support_spds & BNXT_LINK_SPEED_MSK_100GB)
+ fw_speed = PORT_PHY_CFG_REQ_AUTO_LINK_SPEED_100GB;
+ break;
default:
netdev_err(dev, "unsupported speed!\n");
break;
@@ -2128,12 +2190,372 @@
return rc;
}
+static int bnxt_hwrm_selftest_irq(struct bnxt *bp, u16 cmpl_ring)
+{
+ struct hwrm_selftest_irq_input req = {0};
+
+ bnxt_hwrm_cmd_hdr_init(bp, &req, HWRM_SELFTEST_IRQ, cmpl_ring, -1);
+ return hwrm_send_message(bp, &req, sizeof(req), HWRM_CMD_TIMEOUT);
+}
+
+static int bnxt_test_irq(struct bnxt *bp)
+{
+ int i;
+
+ for (i = 0; i < bp->cp_nr_rings; i++) {
+ u16 cmpl_ring = bp->grp_info[i].cp_fw_ring_id;
+ int rc;
+
+ rc = bnxt_hwrm_selftest_irq(bp, cmpl_ring);
+ if (rc)
+ return rc;
+ }
+ return 0;
+}
+
+static int bnxt_hwrm_mac_loopback(struct bnxt *bp, bool enable)
+{
+ struct hwrm_port_mac_cfg_input req = {0};
+
+ bnxt_hwrm_cmd_hdr_init(bp, &req, HWRM_PORT_MAC_CFG, -1, -1);
+
+ req.enables = cpu_to_le32(PORT_MAC_CFG_REQ_ENABLES_LPBK);
+ if (enable)
+ req.lpbk = PORT_MAC_CFG_REQ_LPBK_LOCAL;
+ else
+ req.lpbk = PORT_MAC_CFG_REQ_LPBK_NONE;
+ return hwrm_send_message(bp, &req, sizeof(req), HWRM_CMD_TIMEOUT);
+}
+
+static int bnxt_disable_an_for_lpbk(struct bnxt *bp,
+ struct hwrm_port_phy_cfg_input *req)
+{
+ struct bnxt_link_info *link_info = &bp->link_info;
+ u16 fw_advertising = link_info->advertising;
+ u16 fw_speed;
+ int rc;
+
+ if (!link_info->autoneg)
+ return 0;
+
+ fw_speed = PORT_PHY_CFG_REQ_FORCE_LINK_SPEED_1GB;
+ if (netif_carrier_ok(bp->dev))
+ fw_speed = bp->link_info.link_speed;
+ else if (fw_advertising & BNXT_LINK_SPEED_MSK_10GB)
+ fw_speed = PORT_PHY_CFG_REQ_FORCE_LINK_SPEED_10GB;
+ else if (fw_advertising & BNXT_LINK_SPEED_MSK_25GB)
+ fw_speed = PORT_PHY_CFG_REQ_FORCE_LINK_SPEED_25GB;
+ else if (fw_advertising & BNXT_LINK_SPEED_MSK_40GB)
+ fw_speed = PORT_PHY_CFG_REQ_FORCE_LINK_SPEED_40GB;
+ else if (fw_advertising & BNXT_LINK_SPEED_MSK_50GB)
+ fw_speed = PORT_PHY_CFG_REQ_FORCE_LINK_SPEED_50GB;
+
+ req->force_link_speed = cpu_to_le16(fw_speed);
+ req->flags |= cpu_to_le32(PORT_PHY_CFG_REQ_FLAGS_FORCE |
+ PORT_PHY_CFG_REQ_FLAGS_RESET_PHY);
+ rc = hwrm_send_message(bp, req, sizeof(*req), HWRM_CMD_TIMEOUT);
+ req->flags = 0;
+ req->force_link_speed = cpu_to_le16(0);
+ return rc;
+}
+
+static int bnxt_hwrm_phy_loopback(struct bnxt *bp, bool enable)
+{
+ struct hwrm_port_phy_cfg_input req = {0};
+
+ bnxt_hwrm_cmd_hdr_init(bp, &req, HWRM_PORT_PHY_CFG, -1, -1);
+
+ if (enable) {
+ bnxt_disable_an_for_lpbk(bp, &req);
+ req.lpbk = PORT_PHY_CFG_REQ_LPBK_LOCAL;
+ } else {
+ req.lpbk = PORT_PHY_CFG_REQ_LPBK_NONE;
+ }
+ req.enables = cpu_to_le32(PORT_PHY_CFG_REQ_ENABLES_LPBK);
+ return hwrm_send_message(bp, &req, sizeof(req), HWRM_CMD_TIMEOUT);
+}
+
+static int bnxt_rx_loopback(struct bnxt *bp, struct bnxt_napi *bnapi,
+ u32 raw_cons, int pkt_size)
+{
+ struct bnxt_cp_ring_info *cpr = &bnapi->cp_ring;
+ struct bnxt_rx_ring_info *rxr = bnapi->rx_ring;
+ struct bnxt_sw_rx_bd *rx_buf;
+ struct rx_cmp *rxcmp;
+ u16 cp_cons, cons;
+ u8 *data;
+ u32 len;
+ int i;
+
+ cp_cons = RING_CMP(raw_cons);
+ rxcmp = (struct rx_cmp *)
+ &cpr->cp_desc_ring[CP_RING(cp_cons)][CP_IDX(cp_cons)];
+ cons = rxcmp->rx_cmp_opaque;
+ rx_buf = &rxr->rx_buf_ring[cons];
+ data = rx_buf->data_ptr;
+ len = le32_to_cpu(rxcmp->rx_cmp_len_flags_type) >> RX_CMP_LEN_SHIFT;
+ if (len != pkt_size)
+ return -EIO;
+ i = ETH_ALEN;
+ if (!ether_addr_equal(data + i, bnapi->bp->dev->dev_addr))
+ return -EIO;
+ i += ETH_ALEN;
+ for ( ; i < pkt_size; i++) {
+ if (data[i] != (u8)(i & 0xff))
+ return -EIO;
+ }
+ return 0;
+}
+
+static int bnxt_poll_loopback(struct bnxt *bp, int pkt_size)
+{
+ struct bnxt_napi *bnapi = bp->bnapi[0];
+ struct bnxt_cp_ring_info *cpr;
+ struct tx_cmp *txcmp;
+ int rc = -EIO;
+ u32 raw_cons;
+ u32 cons;
+ int i;
+
+ cpr = &bnapi->cp_ring;
+ raw_cons = cpr->cp_raw_cons;
+ for (i = 0; i < 200; i++) {
+ cons = RING_CMP(raw_cons);
+ txcmp = &cpr->cp_desc_ring[CP_RING(cons)][CP_IDX(cons)];
+
+ if (!TX_CMP_VALID(txcmp, raw_cons)) {
+ udelay(5);
+ continue;
+ }
+
+ /* The valid test of the entry must be done first before
+ * reading any further.
+ */
+ dma_rmb();
+ if (TX_CMP_TYPE(txcmp) == CMP_TYPE_RX_L2_CMP) {
+ rc = bnxt_rx_loopback(bp, bnapi, raw_cons, pkt_size);
+ raw_cons = NEXT_RAW_CMP(raw_cons);
+ raw_cons = NEXT_RAW_CMP(raw_cons);
+ break;
+ }
+ raw_cons = NEXT_RAW_CMP(raw_cons);
+ }
+ cpr->cp_raw_cons = raw_cons;
+ return rc;
+}
+
+static int bnxt_run_loopback(struct bnxt *bp)
+{
+ struct bnxt_tx_ring_info *txr = &bp->tx_ring[0];
+ int pkt_size, i = 0;
+ struct sk_buff *skb;
+ dma_addr_t map;
+ u8 *data;
+ int rc;
+
+ pkt_size = min(bp->dev->mtu + ETH_HLEN, bp->rx_copy_thresh);
+ skb = netdev_alloc_skb(bp->dev, pkt_size);
+ if (!skb)
+ return -ENOMEM;
+ data = skb_put(skb, pkt_size);
+ eth_broadcast_addr(data);
+ i += ETH_ALEN;
+ ether_addr_copy(&data[i], bp->dev->dev_addr);
+ i += ETH_ALEN;
+ for ( ; i < pkt_size; i++)
+ data[i] = (u8)(i & 0xff);
+
+ map = dma_map_single(&bp->pdev->dev, skb->data, pkt_size,
+ PCI_DMA_TODEVICE);
+ if (dma_mapping_error(&bp->pdev->dev, map)) {
+ dev_kfree_skb(skb);
+ return -EIO;
+ }
+ bnxt_xmit_xdp(bp, txr, map, pkt_size, 0);
+
+ /* Sync BD data before updating doorbell */
+ wmb();
+
+ writel(DB_KEY_TX | txr->tx_prod, txr->tx_doorbell);
+ writel(DB_KEY_TX | txr->tx_prod, txr->tx_doorbell);
+ rc = bnxt_poll_loopback(bp, pkt_size);
+
+ dma_unmap_single(&bp->pdev->dev, map, pkt_size, PCI_DMA_TODEVICE);
+ dev_kfree_skb(skb);
+ return rc;
+}
+
+static int bnxt_run_fw_tests(struct bnxt *bp, u8 test_mask, u8 *test_results)
+{
+ struct hwrm_selftest_exec_output *resp = bp->hwrm_cmd_resp_addr;
+ struct hwrm_selftest_exec_input req = {0};
+ int rc;
+
+ bnxt_hwrm_cmd_hdr_init(bp, &req, HWRM_SELFTEST_EXEC, -1, -1);
+ mutex_lock(&bp->hwrm_cmd_lock);
+ resp->test_success = 0;
+ req.flags = test_mask;
+ rc = _hwrm_send_message(bp, &req, sizeof(req), bp->test_info->timeout);
+ *test_results = resp->test_success;
+ mutex_unlock(&bp->hwrm_cmd_lock);
+ return rc;
+}
+
+#define BNXT_DRV_TESTS 3
+#define BNXT_MACLPBK_TEST_IDX (bp->num_tests - BNXT_DRV_TESTS)
+#define BNXT_PHYLPBK_TEST_IDX (BNXT_MACLPBK_TEST_IDX + 1)
+#define BNXT_IRQ_TEST_IDX (BNXT_MACLPBK_TEST_IDX + 2)
+
+static void bnxt_self_test(struct net_device *dev, struct ethtool_test *etest,
+ u64 *buf)
+{
+ struct bnxt *bp = netdev_priv(dev);
+ bool offline = false;
+ u8 test_results = 0;
+ u8 test_mask = 0;
+ int rc, i;
+
+ if (!bp->num_tests || !BNXT_SINGLE_PF(bp))
+ return;
+ memset(buf, 0, sizeof(u64) * bp->num_tests);
+ if (!netif_running(dev)) {
+ etest->flags |= ETH_TEST_FL_FAILED;
+ return;
+ }
+
+ if (etest->flags & ETH_TEST_FL_OFFLINE) {
+ if (bp->pf.active_vfs) {
+ etest->flags |= ETH_TEST_FL_FAILED;
+ netdev_warn(dev, "Offline tests cannot be run with active VFs\n");
+ return;
+ }
+ offline = true;
+ }
+
+ for (i = 0; i < bp->num_tests - BNXT_DRV_TESTS; i++) {
+ u8 bit_val = 1 << i;
+
+ if (!(bp->test_info->offline_mask & bit_val))
+ test_mask |= bit_val;
+ else if (offline)
+ test_mask |= bit_val;
+ }
+ if (!offline) {
+ bnxt_run_fw_tests(bp, test_mask, &test_results);
+ } else {
+ rc = bnxt_close_nic(bp, false, false);
+ if (rc)
+ return;
+ bnxt_run_fw_tests(bp, test_mask, &test_results);
+
+ buf[BNXT_MACLPBK_TEST_IDX] = 1;
+ bnxt_hwrm_mac_loopback(bp, true);
+ msleep(250);
+ rc = bnxt_half_open_nic(bp);
+ if (rc) {
+ bnxt_hwrm_mac_loopback(bp, false);
+ etest->flags |= ETH_TEST_FL_FAILED;
+ return;
+ }
+ if (bnxt_run_loopback(bp))
+ etest->flags |= ETH_TEST_FL_FAILED;
+ else
+ buf[BNXT_MACLPBK_TEST_IDX] = 0;
+
+ bnxt_hwrm_mac_loopback(bp, false);
+ bnxt_hwrm_phy_loopback(bp, true);
+ msleep(1000);
+ if (bnxt_run_loopback(bp)) {
+ buf[BNXT_PHYLPBK_TEST_IDX] = 1;
+ etest->flags |= ETH_TEST_FL_FAILED;
+ }
+ bnxt_hwrm_phy_loopback(bp, false);
+ bnxt_half_close_nic(bp);
+ bnxt_open_nic(bp, false, true);
+ }
+ if (bnxt_test_irq(bp)) {
+ buf[BNXT_IRQ_TEST_IDX] = 1;
+ etest->flags |= ETH_TEST_FL_FAILED;
+ }
+ for (i = 0; i < bp->num_tests - BNXT_DRV_TESTS; i++) {
+ u8 bit_val = 1 << i;
+
+ if ((test_mask & bit_val) && !(test_results & bit_val)) {
+ buf[i] = 1;
+ etest->flags |= ETH_TEST_FL_FAILED;
+ }
+ }
+}
+
+void bnxt_ethtool_init(struct bnxt *bp)
+{
+ struct hwrm_selftest_qlist_output *resp = bp->hwrm_cmd_resp_addr;
+ struct hwrm_selftest_qlist_input req = {0};
+ struct bnxt_test_info *test_info;
+ int i, rc;
+
+ if (bp->hwrm_spec_code < 0x10704 || !BNXT_SINGLE_PF(bp))
+ return;
+
+ bnxt_hwrm_cmd_hdr_init(bp, &req, HWRM_SELFTEST_QLIST, -1, -1);
+ mutex_lock(&bp->hwrm_cmd_lock);
+ rc = _hwrm_send_message(bp, &req, sizeof(req), HWRM_CMD_TIMEOUT);
+ if (rc)
+ goto ethtool_init_exit;
+
+ test_info = kzalloc(sizeof(*bp->test_info), GFP_KERNEL);
+ if (!test_info)
+ goto ethtool_init_exit;
+
+ bp->test_info = test_info;
+ bp->num_tests = resp->num_tests + BNXT_DRV_TESTS;
+ if (bp->num_tests > BNXT_MAX_TEST)
+ bp->num_tests = BNXT_MAX_TEST;
+
+ test_info->offline_mask = resp->offline_tests;
+ test_info->timeout = le16_to_cpu(resp->test_timeout);
+ if (!test_info->timeout)
+ test_info->timeout = HWRM_CMD_TIMEOUT;
+ for (i = 0; i < bp->num_tests; i++) {
+ char *str = test_info->string[i];
+ char *fw_str = resp->test0_name + i * 32;
+
+ if (i == BNXT_MACLPBK_TEST_IDX) {
+ strcpy(str, "Mac loopback test (offline)");
+ } else if (i == BNXT_PHYLPBK_TEST_IDX) {
+ strcpy(str, "Phy loopback test (offline)");
+ } else if (i == BNXT_IRQ_TEST_IDX) {
+ strcpy(str, "Interrupt_test (offline)");
+ } else {
+ strlcpy(str, fw_str, ETH_GSTRING_LEN);
+ strncat(str, " test", ETH_GSTRING_LEN - strlen(str));
+ if (test_info->offline_mask & (1 << i))
+ strncat(str, " (offline)",
+ ETH_GSTRING_LEN - strlen(str));
+ else
+ strncat(str, " (online)",
+ ETH_GSTRING_LEN - strlen(str));
+ }
+ }
+
+ethtool_init_exit:
+ mutex_unlock(&bp->hwrm_cmd_lock);
+}
+
+void bnxt_ethtool_free(struct bnxt *bp)
+{
+ kfree(bp->test_info);
+ bp->test_info = NULL;
+}
+
const struct ethtool_ops bnxt_ethtool_ops = {
.get_link_ksettings = bnxt_get_link_ksettings,
.set_link_ksettings = bnxt_set_link_ksettings,
.get_pauseparam = bnxt_get_pauseparam,
.set_pauseparam = bnxt_set_pauseparam,
.get_drvinfo = bnxt_get_drvinfo,
+ .get_wol = bnxt_get_wol,
+ .set_wol = bnxt_set_wol,
.get_coalesce = bnxt_get_coalesce,
.set_coalesce = bnxt_set_coalesce,
.get_msglevel = bnxt_get_msglevel,
@@ -2161,4 +2583,5 @@
.get_module_eeprom = bnxt_get_module_eeprom,
.nway_reset = bnxt_nway_reset,
.set_phys_id = bnxt_set_phys_id,
+ .self_test = bnxt_self_test,
};
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.h
index ed1e555..f1bc90b 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.h
@@ -1,6 +1,7 @@
/* Broadcom NetXtreme-C/E network driver.
*
* Copyright (c) 2014-2016 Broadcom Corporation
+ * Copyright (c) 2016-2017 Broadcom Limited
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -38,5 +39,7 @@
u32 _bnxt_fw_to_ethtool_adv_spds(u16, u8);
u32 bnxt_fw_to_ethtool_speed(u16);
u16 bnxt_get_fw_auto_link_speeds(u32);
+void bnxt_ethtool_init(struct bnxt *bp);
+void bnxt_ethtool_free(struct bnxt *bp);
#endif
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h
index 6e275c2..7dc71bb 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h
@@ -11,19 +11,21 @@
#ifndef BNXT_HSI_H
#define BNXT_HSI_H
-/* HSI and HWRM Specification 1.7.0 */
+/* HSI and HWRM Specification 1.7.6 */
#define HWRM_VERSION_MAJOR 1
#define HWRM_VERSION_MINOR 7
-#define HWRM_VERSION_UPDATE 0
+#define HWRM_VERSION_UPDATE 6
-#define HWRM_VERSION_STR "1.7.0"
+#define HWRM_VERSION_RSVD 2 /* non-zero means beta version */
+
+#define HWRM_VERSION_STR "1.7.6.2"
/*
* Following is the signature for HWRM message field that indicates not
* applicable (All F's). Need to cast it the size of the field if needed.
*/
#define HWRM_NA_SIGNATURE ((__le32)(-1))
#define HWRM_MAX_REQ_LEN (128) /* hwrm_func_buf_rgtr */
-#define HWRM_MAX_RESP_LEN (176) /* hwrm_func_qstats */
+#define HWRM_MAX_RESP_LEN (248) /* hwrm_selftest_qlist */
#define HW_HASH_INDEX_SIZE 0x80 /* 7 bit indirection table index. */
#define HW_HASH_KEY_SIZE 40
#define HWRM_RESP_VALID_KEY 1 /* valid key for HWRM response */
@@ -571,9 +573,10 @@
__le16 max_req_win_len;
__le16 max_resp_len;
__le16 def_req_timeout;
+ u8 init_pending;
+ #define VER_GET_RESP_INIT_PENDING_DEV_NOT_RDY 0x1UL
u8 unused_0;
u8 unused_1;
- u8 unused_2;
u8 valid;
};
@@ -809,6 +812,8 @@
#define FUNC_QCFG_RESP_FLAGS_OOB_WOL_BMP_ENABLED 0x2UL
#define FUNC_QCFG_RESP_FLAGS_FW_DCBX_AGENT_ENABLED 0x4UL
#define FUNC_QCFG_RESP_FLAGS_STD_TX_RING_MODE_ENABLED 0x8UL
+ #define FUNC_QCFG_RESP_FLAGS_FW_LLDP_AGENT_ENABLED 0x10UL
+ #define FUNC_QCFG_RESP_FLAGS_MULTI_HOST 0x20UL
u8 mac_address[6];
__le16 pci_id;
__le16 alloc_rsscos_ctx;
@@ -827,10 +832,12 @@
#define FUNC_QCFG_RESP_PORT_PARTITION_TYPE_NPAR1_5 0x3UL
#define FUNC_QCFG_RESP_PORT_PARTITION_TYPE_NPAR2_0 0x4UL
#define FUNC_QCFG_RESP_PORT_PARTITION_TYPE_UNKNOWN 0xffUL
- u8 unused_0;
+ u8 port_pf_cnt;
+ #define FUNC_QCFG_RESP_PORT_PF_CNT_UNAVAIL 0x0UL
__le16 dflt_vnic_id;
- u8 unused_1;
- u8 unused_2;
+ u8 host_cnt;
+ #define FUNC_QCFG_RESP_HOST_CNT_UNAVAIL 0x0UL
+ u8 unused_0;
__le32 min_bw;
#define FUNC_QCFG_RESP_MIN_BW_BW_VALUE_MASK 0xfffffffUL
#define FUNC_QCFG_RESP_MIN_BW_BW_VALUE_SFT 0
@@ -867,12 +874,12 @@
#define FUNC_QCFG_RESP_EVB_MODE_NO_EVB 0x0UL
#define FUNC_QCFG_RESP_EVB_MODE_VEB 0x1UL
#define FUNC_QCFG_RESP_EVB_MODE_VEPA 0x2UL
- u8 unused_3;
+ u8 unused_1;
__le16 alloc_vfs;
__le32 alloc_mcast_filters;
__le32 alloc_hw_ring_grps;
__le16 alloc_sp_tx_rings;
- u8 unused_4;
+ u8 unused_2;
u8 valid;
};
@@ -888,16 +895,13 @@
u8 unused_0;
u8 unused_1;
__le32 flags;
- #define FUNC_CFG_REQ_FLAGS_PROM_MODE 0x1UL
- #define FUNC_CFG_REQ_FLAGS_SRC_MAC_ADDR_CHECK 0x2UL
- #define FUNC_CFG_REQ_FLAGS_SRC_IP_ADDR_CHECK 0x4UL
- #define FUNC_CFG_REQ_FLAGS_VLAN_PRI_MATCH 0x8UL
- #define FUNC_CFG_REQ_FLAGS_DFLT_PRI_NOMATCH 0x10UL
- #define FUNC_CFG_REQ_FLAGS_DISABLE_PAUSE 0x20UL
- #define FUNC_CFG_REQ_FLAGS_DISABLE_STP 0x40UL
- #define FUNC_CFG_REQ_FLAGS_DISABLE_LLDP 0x80UL
- #define FUNC_CFG_REQ_FLAGS_DISABLE_PTPV2 0x100UL
- #define FUNC_CFG_REQ_FLAGS_STD_TX_RING_MODE 0x200UL
+ #define FUNC_CFG_REQ_FLAGS_SRC_MAC_ADDR_CHECK_DISABLE 0x1UL
+ #define FUNC_CFG_REQ_FLAGS_SRC_MAC_ADDR_CHECK_ENABLE 0x2UL
+ #define FUNC_CFG_REQ_FLAGS_RSVD_MASK 0x1fcUL
+ #define FUNC_CFG_REQ_FLAGS_RSVD_SFT 2
+ #define FUNC_CFG_REQ_FLAGS_STD_TX_RING_MODE_ENABLE 0x200UL
+ #define FUNC_CFG_REQ_FLAGS_STD_TX_RING_MODE_DISABLE 0x400UL
+ #define FUNC_CFG_REQ_FLAGS_VIRT_MAC_PERSIST 0x800UL
__le32 enables;
#define FUNC_CFG_REQ_ENABLES_MTU 0x1UL
#define FUNC_CFG_REQ_ENABLES_MRU 0x2UL
@@ -1013,7 +1017,7 @@
__le64 tx_ucast_pkts;
__le64 tx_mcast_pkts;
__le64 tx_bcast_pkts;
- __le64 tx_err_pkts;
+ __le64 tx_discard_pkts;
__le64 tx_drop_pkts;
__le64 tx_ucast_bytes;
__le64 tx_mcast_bytes;
@@ -1021,7 +1025,7 @@
__le64 rx_ucast_pkts;
__le64 rx_mcast_pkts;
__le64 rx_bcast_pkts;
- __le64 rx_err_pkts;
+ __le64 rx_discard_pkts;
__le64 rx_drop_pkts;
__le64 rx_ucast_bytes;
__le64 rx_mcast_bytes;
@@ -4743,6 +4747,165 @@
u8 valid;
};
+/* hwrm_wol_filter_alloc */
+/* Input (64 bytes) */
+struct hwrm_wol_filter_alloc_input {
+ __le16 req_type;
+ __le16 cmpl_ring;
+ __le16 seq_id;
+ __le16 target_id;
+ __le64 resp_addr;
+ __le32 flags;
+ __le32 enables;
+ #define WOL_FILTER_ALLOC_REQ_ENABLES_MAC_ADDRESS 0x1UL
+ #define WOL_FILTER_ALLOC_REQ_ENABLES_PATTERN_OFFSET 0x2UL
+ #define WOL_FILTER_ALLOC_REQ_ENABLES_PATTERN_BUF_SIZE 0x4UL
+ #define WOL_FILTER_ALLOC_REQ_ENABLES_PATTERN_BUF_ADDR 0x8UL
+ #define WOL_FILTER_ALLOC_REQ_ENABLES_PATTERN_MASK_ADDR 0x10UL
+ #define WOL_FILTER_ALLOC_REQ_ENABLES_PATTERN_MASK_SIZE 0x20UL
+ __le16 port_id;
+ u8 wol_type;
+ #define WOL_FILTER_ALLOC_REQ_WOL_TYPE_MAGICPKT 0x0UL
+ #define WOL_FILTER_ALLOC_REQ_WOL_TYPE_BMP 0x1UL
+ #define WOL_FILTER_ALLOC_REQ_WOL_TYPE_INVALID 0xffUL
+ u8 unused_0;
+ __le32 unused_1;
+ u8 mac_address[6];
+ __le16 pattern_offset;
+ __le16 pattern_buf_size;
+ __le16 pattern_mask_size;
+ __le32 unused_2;
+ __le64 pattern_buf_addr;
+ __le64 pattern_mask_addr;
+};
+
+/* Output (16 bytes) */
+struct hwrm_wol_filter_alloc_output {
+ __le16 error_code;
+ __le16 req_type;
+ __le16 seq_id;
+ __le16 resp_len;
+ u8 wol_filter_id;
+ u8 unused_0;
+ __le16 unused_1;
+ u8 unused_2;
+ u8 unused_3;
+ u8 unused_4;
+ u8 valid;
+};
+
+/* hwrm_wol_filter_free */
+/* Input (32 bytes) */
+struct hwrm_wol_filter_free_input {
+ __le16 req_type;
+ __le16 cmpl_ring;
+ __le16 seq_id;
+ __le16 target_id;
+ __le64 resp_addr;
+ __le32 flags;
+ #define WOL_FILTER_FREE_REQ_FLAGS_FREE_ALL_WOL_FILTERS 0x1UL
+ __le32 enables;
+ #define WOL_FILTER_FREE_REQ_ENABLES_WOL_FILTER_ID 0x1UL
+ __le16 port_id;
+ u8 wol_filter_id;
+ u8 unused_0[5];
+};
+
+/* Output (16 bytes) */
+struct hwrm_wol_filter_free_output {
+ __le16 error_code;
+ __le16 req_type;
+ __le16 seq_id;
+ __le16 resp_len;
+ __le32 unused_0;
+ u8 unused_1;
+ u8 unused_2;
+ u8 unused_3;
+ u8 valid;
+};
+
+/* hwrm_wol_filter_qcfg */
+/* Input (56 bytes) */
+struct hwrm_wol_filter_qcfg_input {
+ __le16 req_type;
+ __le16 cmpl_ring;
+ __le16 seq_id;
+ __le16 target_id;
+ __le64 resp_addr;
+ __le16 port_id;
+ __le16 handle;
+ __le32 unused_0;
+ __le64 pattern_buf_addr;
+ __le16 pattern_buf_size;
+ u8 unused_1;
+ u8 unused_2;
+ u8 unused_3[3];
+ u8 unused_4;
+ __le64 pattern_mask_addr;
+ __le16 pattern_mask_size;
+ __le16 unused_5[3];
+};
+
+/* Output (32 bytes) */
+struct hwrm_wol_filter_qcfg_output {
+ __le16 error_code;
+ __le16 req_type;
+ __le16 seq_id;
+ __le16 resp_len;
+ __le16 next_handle;
+ u8 wol_filter_id;
+ u8 wol_type;
+ #define WOL_FILTER_QCFG_RESP_WOL_TYPE_MAGICPKT 0x0UL
+ #define WOL_FILTER_QCFG_RESP_WOL_TYPE_BMP 0x1UL
+ #define WOL_FILTER_QCFG_RESP_WOL_TYPE_INVALID 0xffUL
+ __le32 unused_0;
+ u8 mac_address[6];
+ __le16 pattern_offset;
+ __le16 pattern_size;
+ __le16 pattern_mask_size;
+ u8 unused_1;
+ u8 unused_2;
+ u8 unused_3;
+ u8 valid;
+};
+
+/* hwrm_wol_reason_qcfg */
+/* Input (40 bytes) */
+struct hwrm_wol_reason_qcfg_input {
+ __le16 req_type;
+ __le16 cmpl_ring;
+ __le16 seq_id;
+ __le16 target_id;
+ __le64 resp_addr;
+ __le16 port_id;
+ u8 unused_0;
+ u8 unused_1;
+ u8 unused_2[3];
+ u8 unused_3;
+ __le64 wol_pkt_buf_addr;
+ __le16 wol_pkt_buf_size;
+ __le16 unused_4[3];
+};
+
+/* Output (16 bytes) */
+struct hwrm_wol_reason_qcfg_output {
+ __le16 error_code;
+ __le16 req_type;
+ __le16 seq_id;
+ __le16 resp_len;
+ u8 wol_filter_id;
+ u8 wol_reason;
+ #define WOL_REASON_QCFG_RESP_WOL_REASON_MAGICPKT 0x0UL
+ #define WOL_REASON_QCFG_RESP_WOL_REASON_BMP 0x1UL
+ #define WOL_REASON_QCFG_RESP_WOL_REASON_INVALID 0xffUL
+ u8 wol_pkt_len;
+ u8 unused_0;
+ u8 unused_1;
+ u8 unused_2;
+ u8 unused_3;
+ u8 valid;
+};
+
/* hwrm_nvm_read */
/* Input (40 bytes) */
struct hwrm_nvm_read_input {
@@ -4773,32 +4936,6 @@
u8 valid;
};
-/* hwrm_nvm_raw_dump */
-/* Input (32 bytes) */
-struct hwrm_nvm_raw_dump_input {
- __le16 req_type;
- __le16 cmpl_ring;
- __le16 seq_id;
- __le16 target_id;
- __le64 resp_addr;
- __le64 host_dest_addr;
- __le32 offset;
- __le32 len;
-};
-
-/* Output (16 bytes) */
-struct hwrm_nvm_raw_dump_output {
- __le16 error_code;
- __le16 req_type;
- __le16 seq_id;
- __le16 resp_len;
- __le32 unused_0;
- u8 unused_1;
- u8 unused_2;
- u8 unused_3;
- u8 valid;
-};
-
/* hwrm_nvm_get_dir_entries */
/* Input (24 bytes) */
struct hwrm_nvm_get_dir_entries_input {
@@ -4881,6 +5018,15 @@
u8 valid;
};
+/* Command specific Error Codes (8 bytes) */
+struct hwrm_nvm_write_cmd_err {
+ u8 code;
+ #define NVM_WRITE_CMD_ERR_CODE_UNKNOWN 0x0UL
+ #define NVM_WRITE_CMD_ERR_CODE_FRAG_ERR 0x1UL
+ #define NVM_WRITE_CMD_ERR_CODE_NO_SPACE 0x2UL
+ u8 unused_0[7];
+};
+
/* hwrm_nvm_modify */
/* Input (40 bytes) */
struct hwrm_nvm_modify_input {
@@ -5112,6 +5258,100 @@
u8 unused_0[7];
};
+/* hwrm_selftest_qlist */
+/* Input (16 bytes) */
+struct hwrm_selftest_qlist_input {
+ __le16 req_type;
+ __le16 cmpl_ring;
+ __le16 seq_id;
+ __le16 target_id;
+ __le64 resp_addr;
+};
+
+/* Output (248 bytes) */
+struct hwrm_selftest_qlist_output {
+ __le16 error_code;
+ __le16 req_type;
+ __le16 seq_id;
+ __le16 resp_len;
+ u8 num_tests;
+ u8 available_tests;
+ #define SELFTEST_QLIST_RESP_AVAILABLE_TESTS_NVM_TEST 0x1UL
+ #define SELFTEST_QLIST_RESP_AVAILABLE_TESTS_LINK_TEST 0x2UL
+ #define SELFTEST_QLIST_RESP_AVAILABLE_TESTS_REGISTER_TEST 0x4UL
+ #define SELFTEST_QLIST_RESP_AVAILABLE_TESTS_MEMORY_TEST 0x8UL
+ u8 offline_tests;
+ #define SELFTEST_QLIST_RESP_OFFLINE_TESTS_NVM_TEST 0x1UL
+ #define SELFTEST_QLIST_RESP_OFFLINE_TESTS_LINK_TEST 0x2UL
+ #define SELFTEST_QLIST_RESP_OFFLINE_TESTS_REGISTER_TEST 0x4UL
+ #define SELFTEST_QLIST_RESP_OFFLINE_TESTS_MEMORY_TEST 0x8UL
+ u8 unused_0;
+ __le16 test_timeout;
+ u8 unused_1;
+ u8 unused_2;
+ char test0_name[32];
+ char test1_name[32];
+ char test2_name[32];
+ char test3_name[32];
+ char test4_name[32];
+ char test5_name[32];
+ char test6_name[32];
+ char test7_name[32];
+};
+
+/* hwrm_selftest_exec */
+/* Input (24 bytes) */
+struct hwrm_selftest_exec_input {
+ __le16 req_type;
+ __le16 cmpl_ring;
+ __le16 seq_id;
+ __le16 target_id;
+ __le64 resp_addr;
+ u8 flags;
+ #define SELFTEST_EXEC_REQ_FLAGS_NVM_TEST 0x1UL
+ #define SELFTEST_EXEC_REQ_FLAGS_LINK_TEST 0x2UL
+ #define SELFTEST_EXEC_REQ_FLAGS_REGISTER_TEST 0x4UL
+ #define SELFTEST_EXEC_REQ_FLAGS_MEMORY_TEST 0x8UL
+ u8 unused_0[7];
+};
+
+/* Output (16 bytes) */
+struct hwrm_selftest_exec_output {
+ __le16 error_code;
+ __le16 req_type;
+ __le16 seq_id;
+ __le16 resp_len;
+ u8 requested_tests;
+ #define SELFTEST_EXEC_RESP_REQUESTED_TESTS_NVM_TEST 0x1UL
+ #define SELFTEST_EXEC_RESP_REQUESTED_TESTS_LINK_TEST 0x2UL
+ #define SELFTEST_EXEC_RESP_REQUESTED_TESTS_REGISTER_TEST 0x4UL
+ #define SELFTEST_EXEC_RESP_REQUESTED_TESTS_MEMORY_TEST 0x8UL
+ u8 test_success;
+ #define SELFTEST_EXEC_RESP_TEST_SUCCESS_NVM_TEST 0x1UL
+ #define SELFTEST_EXEC_RESP_TEST_SUCCESS_LINK_TEST 0x2UL
+ #define SELFTEST_EXEC_RESP_TEST_SUCCESS_REGISTER_TEST 0x4UL
+ #define SELFTEST_EXEC_RESP_TEST_SUCCESS_MEMORY_TEST 0x8UL
+ __le16 unused_0[3];
+};
+
+/* hwrm_selftest_irq */
+/* Input (16 bytes) */
+struct hwrm_selftest_irq_input {
+ __le16 req_type;
+ __le16 cmpl_ring;
+ __le16 seq_id;
+ __le16 target_id;
+ __le64 resp_addr;
+};
+
+/* Output (8 bytes) */
+struct hwrm_selftest_irq_output {
+ __le16 error_code;
+ __le16 req_type;
+ __le16 seq_id;
+ __le16 resp_len;
+};
+
/* Hardware Resource Manager Specification */
/* Input (16 bytes) */
struct input {
@@ -5130,6 +5370,16 @@
__le16 resp_len;
};
+/* Short Command Structure (16 bytes) */
+struct hwrm_short_input {
+ __le16 req_type;
+ __le16 signature;
+ #define SHORT_REQ_SIGNATURE_SHORT_CMD 0x4321UL
+ __le16 unused_0;
+ __le16 size;
+ __le64 req_addr;
+};
+
/* Command numbering (8 bytes) */
struct cmd_nums {
__le16 req_type;
@@ -5252,11 +5502,15 @@
#define HWRM_CFA_FLOW_FLUSH (0x105UL)
#define HWRM_CFA_FLOW_STATS (0x106UL)
#define HWRM_CFA_FLOW_INFO (0x107UL)
+ #define HWRM_SELFTEST_QLIST (0x200UL)
+ #define HWRM_SELFTEST_EXEC (0x201UL)
+ #define HWRM_SELFTEST_IRQ (0x202UL)
#define HWRM_DBG_READ_DIRECT (0xff10UL)
#define HWRM_DBG_READ_INDIRECT (0xff11UL)
#define HWRM_DBG_WRITE_DIRECT (0xff12UL)
#define HWRM_DBG_WRITE_INDIRECT (0xff13UL)
#define HWRM_DBG_DUMP (0xff14UL)
+ #define HWRM_NVM_FACTORY_DEFAULTS (0xffeeUL)
#define HWRM_NVM_VALIDATE_OPTION (0xffefUL)
#define HWRM_NVM_FLUSH (0xfff0UL)
#define HWRM_NVM_GET_VARIABLE (0xfff1UL)
@@ -5464,6 +5718,7 @@
#define STRUCT_HDR_STRUCT_ID_DCBX_FEATURE_STATE 0x422UL
#define STRUCT_HDR_STRUCT_ID_LLDP_GENERIC 0x424UL
#define STRUCT_HDR_STRUCT_ID_LLDP_DEVICE 0x426UL
+ #define STRUCT_HDR_STRUCT_ID_AFM_OPAQUE 0x1UL
#define STRUCT_HDR_STRUCT_ID_PORT_DESCRIPTION 0xaUL
__le16 len;
u8 version;
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c
index 0b8cd74..b8e7248 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c
@@ -1,6 +1,7 @@
/* Broadcom NetXtreme-C/E network driver.
*
* Copyright (c) 2014-2016 Broadcom Corporation
+ * Copyright (c) 2016-2017 Broadcom Limited
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -84,6 +85,9 @@
u32 func_flags;
int rc;
+ if (bp->hwrm_spec_code < 0x10701)
+ return -ENOTSUPP;
+
rc = bnxt_vf_ndo_prep(bp, vf_id);
if (rc)
return rc;
@@ -96,9 +100,9 @@
func_flags = vf->func_flags;
if (setting)
- func_flags |= FUNC_CFG_REQ_FLAGS_SRC_MAC_ADDR_CHECK;
+ func_flags |= FUNC_CFG_REQ_FLAGS_SRC_MAC_ADDR_CHECK_ENABLE;
else
- func_flags &= ~FUNC_CFG_REQ_FLAGS_SRC_MAC_ADDR_CHECK;
+ func_flags |= FUNC_CFG_REQ_FLAGS_SRC_MAC_ADDR_CHECK_DISABLE;
/*TODO: if the driver supports VLAN filter on guest VLAN,
* the spoof check should also include vlan anti-spoofing
*/
@@ -134,8 +138,11 @@
ivi->max_tx_rate = vf->max_tx_rate;
ivi->min_tx_rate = vf->min_tx_rate;
ivi->vlan = vf->vlan;
- ivi->qos = vf->flags & BNXT_VF_QOS;
- ivi->spoofchk = vf->flags & BNXT_VF_SPOOFCHK;
+ if (vf->flags & BNXT_VF_QOS)
+ ivi->qos = vf->vlan >> VLAN_PRIO_SHIFT;
+ else
+ ivi->qos = 0;
+ ivi->spoofchk = !!(vf->flags & BNXT_VF_SPOOFCHK);
if (!(vf->flags & BNXT_VF_LINK_FORCED))
ivi->linkstate = IFLA_VF_LINK_STATE_AUTO;
else if (vf->flags & BNXT_VF_LINK_UP)
@@ -300,7 +307,6 @@
for (i = 0; i < num_vfs; i++) {
vf = &bp->pf.vf[i];
memset(vf, 0, sizeof(*vf));
- vf->flags = BNXT_VF_QOS | BNXT_VF_LINK_UP;
}
return 0;
}
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.h
index 1ab72e4..dbc8d97 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.h
@@ -1,6 +1,7 @@
/* Broadcom NetXtreme-C/E network driver.
*
* Copyright (c) 2014-2016 Broadcom Corporation
+ * Copyright (c) 2016-2017 Broadcom Limited
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c
index 899c30f..9dae327 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c
@@ -19,11 +19,10 @@
#include "bnxt.h"
#include "bnxt_xdp.h"
-static void bnxt_xmit_xdp(struct bnxt *bp, struct bnxt_tx_ring_info *txr,
- dma_addr_t mapping, u32 len, u16 rx_prod)
+void bnxt_xmit_xdp(struct bnxt *bp, struct bnxt_tx_ring_info *txr,
+ dma_addr_t mapping, u32 len, u16 rx_prod)
{
struct bnxt_sw_tx_bd *tx_buf;
- struct tx_bd_ext *txbd1;
struct tx_bd *txbd;
u32 flags;
u16 prod;
@@ -33,23 +32,13 @@
tx_buf->rx_prod = rx_prod;
txbd = &txr->tx_desc_ring[TX_RING(prod)][TX_IDX(prod)];
- flags = (len << TX_BD_LEN_SHIFT) | TX_BD_TYPE_LONG_TX_BD |
- (2 << TX_BD_FLAGS_BD_CNT_SHIFT) | TX_BD_FLAGS_COAL_NOW |
+ flags = (len << TX_BD_LEN_SHIFT) | (1 << TX_BD_FLAGS_BD_CNT_SHIFT) |
TX_BD_FLAGS_PACKET_END | bnxt_lhint_arr[len >> 9];
txbd->tx_bd_len_flags_type = cpu_to_le32(flags);
txbd->tx_bd_opaque = prod;
txbd->tx_bd_haddr = cpu_to_le64(mapping);
prod = NEXT_TX(prod);
- txbd1 = (struct tx_bd_ext *)
- &txr->tx_desc_ring[TX_RING(prod)][TX_IDX(prod)];
-
- txbd1->tx_bd_hsize_lflags = cpu_to_le32(0);
- txbd1->tx_bd_mss = cpu_to_le32(0);
- txbd1->tx_bd_cfa_action = cpu_to_le32(0);
- txbd1->tx_bd_cfa_meta = cpu_to_le32(0);
-
- prod = NEXT_TX(prod);
txr->tx_prod = prod;
}
@@ -66,7 +55,6 @@
for (i = 0; i < nr_pkts; i++) {
last_tx_cons = tx_cons;
tx_cons = NEXT_TX(tx_cons);
- tx_cons = NEXT_TX(tx_cons);
}
txr->tx_cons = tx_cons;
if (bnxt_tx_avail(bp, txr) == bp->tx_ring_size) {
@@ -133,7 +121,7 @@
return false;
case XDP_TX:
- if (tx_avail < 2) {
+ if (tx_avail < 1) {
trace_xdp_exception(bp->dev, xdp_prog, act);
bnxt_reuse_rx_data(rxr, cons, page);
return true;
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h
index b529f2c..12a5ad6 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.h
@@ -10,6 +10,8 @@
#ifndef BNXT_XDP_H
#define BNXT_XDP_H
+void bnxt_xmit_xdp(struct bnxt *bp, struct bnxt_tx_ring_info *txr,
+ dma_addr_t mapping, u32 len, u16 rx_prod);
void bnxt_tx_int_xdp(struct bnxt *bp, struct bnxt_napi *bnapi, int nr_pkts);
bool bnxt_rx_xdp(struct bnxt *bp, struct bnxt_rx_ring_info *rxr, u16 cons,
struct page *page, u8 **data_ptr, unsigned int *len,
diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.c b/drivers/net/ethernet/broadcom/genet/bcmgenet.c
index 365895e..a205a9f 100644
--- a/drivers/net/ethernet/broadcom/genet/bcmgenet.c
+++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.c
@@ -621,7 +621,7 @@
/* GENET TDMA hardware does not support a configurable timeout, but will
* always generate an interrupt either after MBDONE packets have been
- * transmitted, or when the ring is emtpy.
+ * transmitted, or when the ring is empty.
*/
if (ec->tx_coalesce_usecs || ec->tx_coalesce_usecs_high ||
ec->tx_coalesce_usecs_irq || ec->tx_coalesce_usecs_low)
@@ -707,6 +707,19 @@
.reg_offset = offset, \
}
+#define STAT_GENET_Q(num) \
+ STAT_GENET_SOFT_MIB("txq" __stringify(num) "_packets", \
+ tx_rings[num].packets), \
+ STAT_GENET_SOFT_MIB("txq" __stringify(num) "_bytes", \
+ tx_rings[num].bytes), \
+ STAT_GENET_SOFT_MIB("rxq" __stringify(num) "_bytes", \
+ rx_rings[num].bytes), \
+ STAT_GENET_SOFT_MIB("rxq" __stringify(num) "_packets", \
+ rx_rings[num].packets), \
+ STAT_GENET_SOFT_MIB("rxq" __stringify(num) "_errors", \
+ rx_rings[num].errors), \
+ STAT_GENET_SOFT_MIB("rxq" __stringify(num) "_dropped", \
+ rx_rings[num].dropped)
/* There is a 0xC gap between the end of RX and beginning of TX stats and then
* between the end of TX stats and the beginning of the RX RUNT
@@ -801,6 +814,12 @@
STAT_GENET_SOFT_MIB("alloc_rx_buff_failed", mib.alloc_rx_buff_failed),
STAT_GENET_SOFT_MIB("rx_dma_failed", mib.rx_dma_failed),
STAT_GENET_SOFT_MIB("tx_dma_failed", mib.tx_dma_failed),
+ /* Per TX queues */
+ STAT_GENET_Q(0),
+ STAT_GENET_Q(1),
+ STAT_GENET_Q(2),
+ STAT_GENET_Q(3),
+ STAT_GENET_Q(16),
};
#define BCMGENET_STATS_LEN ARRAY_SIZE(bcmgenet_gstrings_stats)
@@ -1078,8 +1097,17 @@
/* Power down LED */
if (priv->hw_params->flags & GENET_HAS_EXT) {
reg = bcmgenet_ext_readl(priv, EXT_EXT_PWR_MGMT);
- reg |= (EXT_PWR_DOWN_PHY |
- EXT_PWR_DOWN_DLL | EXT_PWR_DOWN_BIAS);
+ if (GENET_IS_V5(priv))
+ reg |= EXT_PWR_DOWN_PHY_EN |
+ EXT_PWR_DOWN_PHY_RD |
+ EXT_PWR_DOWN_PHY_SD |
+ EXT_PWR_DOWN_PHY_RX |
+ EXT_PWR_DOWN_PHY_TX |
+ EXT_IDDQ_GLBL_PWR;
+ else
+ reg |= EXT_PWR_DOWN_PHY;
+
+ reg |= (EXT_PWR_DOWN_DLL | EXT_PWR_DOWN_BIAS);
bcmgenet_ext_writel(priv, reg, EXT_EXT_PWR_MGMT);
bcmgenet_phy_power_set(priv->dev, false);
@@ -1104,12 +1132,34 @@
switch (mode) {
case GENET_POWER_PASSIVE:
- reg &= ~(EXT_PWR_DOWN_DLL | EXT_PWR_DOWN_PHY |
- EXT_PWR_DOWN_BIAS);
- /* fallthrough */
+ reg &= ~(EXT_PWR_DOWN_DLL | EXT_PWR_DOWN_BIAS);
+ if (GENET_IS_V5(priv)) {
+ reg &= ~(EXT_PWR_DOWN_PHY_EN |
+ EXT_PWR_DOWN_PHY_RD |
+ EXT_PWR_DOWN_PHY_SD |
+ EXT_PWR_DOWN_PHY_RX |
+ EXT_PWR_DOWN_PHY_TX |
+ EXT_IDDQ_GLBL_PWR);
+ reg |= EXT_PHY_RESET;
+ bcmgenet_ext_writel(priv, reg, EXT_EXT_PWR_MGMT);
+ mdelay(1);
+
+ reg &= ~EXT_PHY_RESET;
+ } else {
+ reg &= ~EXT_PWR_DOWN_PHY;
+ reg |= EXT_PWR_DN_EN_LD;
+ }
+ bcmgenet_ext_writel(priv, reg, EXT_EXT_PWR_MGMT);
+ bcmgenet_phy_power_set(priv->dev, true);
+ bcmgenet_mii_reset(priv->dev);
+ break;
+
case GENET_POWER_CABLE_SENSE:
/* enable APD */
- reg |= EXT_PWR_DN_EN_LD;
+ if (!GENET_IS_V5(priv)) {
+ reg |= EXT_PWR_DN_EN_LD;
+ bcmgenet_ext_writel(priv, reg, EXT_EXT_PWR_MGMT);
+ }
break;
case GENET_POWER_WOL_MAGIC:
bcmgenet_wol_power_up_cfg(priv, mode);
@@ -1117,39 +1167,20 @@
default:
break;
}
-
- bcmgenet_ext_writel(priv, reg, EXT_EXT_PWR_MGMT);
- if (mode == GENET_POWER_PASSIVE) {
- bcmgenet_phy_power_set(priv->dev, true);
- bcmgenet_mii_reset(priv->dev);
- }
}
/* ioctl handle special commands that are not present in ethtool. */
static int bcmgenet_ioctl(struct net_device *dev, struct ifreq *rq, int cmd)
{
struct bcmgenet_priv *priv = netdev_priv(dev);
- int val = 0;
if (!netif_running(dev))
return -EINVAL;
- switch (cmd) {
- case SIOCGMIIPHY:
- case SIOCGMIIREG:
- case SIOCSMIIREG:
- if (!priv->phydev)
- val = -ENODEV;
- else
- val = phy_mii_ioctl(priv->phydev, rq, cmd);
- break;
+ if (!priv->phydev)
+ return -ENODEV;
- default:
- val = -EINVAL;
- break;
- }
-
- return val;
+ return phy_mii_ioctl(priv->phydev, rq, cmd);
}
static struct enet_cb *bcmgenet_get_txcb(struct bcmgenet_priv *priv,
@@ -1240,14 +1271,18 @@
unsigned int txbds_ready;
unsigned int txbds_processed = 0;
- /* Compute how many buffers are transmitted since last xmit call */
- c_index = bcmgenet_tdma_ring_readl(priv, ring->index, TDMA_CONS_INDEX);
- c_index &= DMA_C_INDEX_MASK;
-
- if (likely(c_index >= ring->c_index))
- txbds_ready = c_index - ring->c_index;
+ /* Clear status before servicing to reduce spurious interrupts */
+ if (ring->index == DESC_INDEX)
+ bcmgenet_intrl2_0_writel(priv, UMAC_IRQ_TXDMA_DONE,
+ INTRL2_CPU_CLEAR);
else
- txbds_ready = (DMA_C_INDEX_MASK + 1) - ring->c_index + c_index;
+ bcmgenet_intrl2_1_writel(priv, (1 << ring->index),
+ INTRL2_CPU_CLEAR);
+
+ /* Compute how many buffers are transmitted since last xmit call */
+ c_index = bcmgenet_tdma_ring_readl(priv, ring->index, TDMA_CONS_INDEX)
+ & DMA_C_INDEX_MASK;
+ txbds_ready = (c_index - ring->c_index) & DMA_C_INDEX_MASK;
netif_dbg(priv, tx_done, dev,
"%s ring=%d old_c_index=%u c_index=%u txbds_ready=%u\n",
@@ -1280,15 +1315,15 @@
}
ring->free_bds += txbds_processed;
- ring->c_index = (ring->c_index + txbds_processed) & DMA_C_INDEX_MASK;
+ ring->c_index = c_index;
- dev->stats.tx_packets += pkts_compl;
- dev->stats.tx_bytes += bytes_compl;
+ ring->packets += pkts_compl;
+ ring->bytes += bytes_compl;
netdev_tx_completed_queue(netdev_get_tx_queue(dev, ring->queue),
pkts_compl, bytes_compl);
- return pkts_compl;
+ return txbds_processed;
}
static unsigned int bcmgenet_tx_reclaim(struct net_device *dev,
@@ -1657,18 +1692,28 @@
unsigned long dma_flag;
int len;
unsigned int rxpktprocessed = 0, rxpkttoprocess;
- unsigned int p_index;
+ unsigned int p_index, mask;
unsigned int discards;
unsigned int chksum_ok = 0;
+ /* Clear status before servicing to reduce spurious interrupts */
+ if (ring->index == DESC_INDEX) {
+ bcmgenet_intrl2_0_writel(priv, UMAC_IRQ_RXDMA_DONE,
+ INTRL2_CPU_CLEAR);
+ } else {
+ mask = 1 << (UMAC_IRQ1_RX_INTR_SHIFT + ring->index);
+ bcmgenet_intrl2_1_writel(priv,
+ mask,
+ INTRL2_CPU_CLEAR);
+ }
+
p_index = bcmgenet_rdma_ring_readl(priv, ring->index, RDMA_PROD_INDEX);
discards = (p_index >> DMA_P_INDEX_DISCARD_CNT_SHIFT) &
DMA_P_INDEX_DISCARD_CNT_MASK;
if (discards > ring->old_discards) {
discards = discards - ring->old_discards;
- dev->stats.rx_missed_errors += discards;
- dev->stats.rx_errors += discards;
+ ring->errors += discards;
ring->old_discards += discards;
/* Clear HW register when we reach 75% of maximum 0xFFFF */
@@ -1680,12 +1725,7 @@
}
p_index &= DMA_P_INDEX_MASK;
-
- if (likely(p_index >= ring->c_index))
- rxpkttoprocess = p_index - ring->c_index;
- else
- rxpkttoprocess = (DMA_C_INDEX_MASK + 1) - ring->c_index +
- p_index;
+ rxpkttoprocess = (p_index - ring->c_index) & DMA_C_INDEX_MASK;
netif_dbg(priv, rx_status, dev,
"RDMA: rxpkttoprocess=%d\n", rxpkttoprocess);
@@ -1696,7 +1736,7 @@
skb = bcmgenet_rx_refill(priv, cb);
if (unlikely(!skb)) {
- dev->stats.rx_dropped++;
+ ring->dropped++;
goto next;
}
@@ -1724,7 +1764,7 @@
if (unlikely(!(dma_flag & DMA_EOP) || !(dma_flag & DMA_SOP))) {
netif_err(priv, rx_status, dev,
"dropping fragmented packet!\n");
- dev->stats.rx_errors++;
+ ring->errors++;
dev_kfree_skb_any(skb);
goto next;
}
@@ -1773,8 +1813,8 @@
/*Finish setting up the received SKB and send it to the kernel*/
skb->protocol = eth_type_trans(skb, priv->dev);
- dev->stats.rx_packets++;
- dev->stats.rx_bytes += len;
+ ring->packets++;
+ ring->bytes += len;
if (dma_flag & DMA_RX_MULT)
dev->stats.multicast++;
@@ -1912,10 +1952,8 @@
/* Mask all interrupts.*/
bcmgenet_intrl2_0_writel(priv, 0xFFFFFFFF, INTRL2_CPU_MASK_SET);
bcmgenet_intrl2_0_writel(priv, 0xFFFFFFFF, INTRL2_CPU_CLEAR);
- bcmgenet_intrl2_0_writel(priv, 0, INTRL2_CPU_MASK_CLEAR);
bcmgenet_intrl2_1_writel(priv, 0xFFFFFFFF, INTRL2_CPU_MASK_SET);
bcmgenet_intrl2_1_writel(priv, 0xFFFFFFFF, INTRL2_CPU_CLEAR);
- bcmgenet_intrl2_1_writel(priv, 0, INTRL2_CPU_MASK_CLEAR);
}
static void bcmgenet_link_intr_enable(struct bcmgenet_priv *priv)
@@ -1942,8 +1980,6 @@
int ret;
u32 reg;
u32 int0_enable = 0;
- u32 int1_enable = 0;
- int i;
dev_dbg(&priv->pdev->dev, "bcmgenet: init_umac\n");
@@ -1970,12 +2006,6 @@
bcmgenet_intr_disable(priv);
- /* Enable Rx default queue 16 interrupts */
- int0_enable |= UMAC_IRQ_RXDMA_DONE;
-
- /* Enable Tx default queue 16 interrupts */
- int0_enable |= UMAC_IRQ_TXDMA_DONE;
-
/* Configure backpressure vectors for MoCA */
if (priv->phy_interface == PHY_INTERFACE_MODE_MOCA) {
reg = bcmgenet_bp_mc_get(priv);
@@ -1993,18 +2023,8 @@
if (priv->hw_params->flags & GENET_HAS_MDIO_INTR)
int0_enable |= (UMAC_IRQ_MDIO_DONE | UMAC_IRQ_MDIO_ERROR);
- /* Enable Rx priority queue interrupts */
- for (i = 0; i < priv->hw_params->rx_queues; ++i)
- int1_enable |= (1 << (UMAC_IRQ1_RX_INTR_SHIFT + i));
-
- /* Enable Tx priority queue interrupts */
- for (i = 0; i < priv->hw_params->tx_queues; ++i)
- int1_enable |= (1 << i);
-
bcmgenet_intrl2_0_writel(priv, int0_enable, INTRL2_CPU_MASK_CLEAR);
- bcmgenet_intrl2_1_writel(priv, int1_enable, INTRL2_CPU_MASK_CLEAR);
- /* Enable rx/tx engine.*/
dev_dbg(kdev, "done init umac\n");
return 0;
@@ -2136,22 +2156,33 @@
static void bcmgenet_enable_tx_napi(struct bcmgenet_priv *priv)
{
unsigned int i;
+ u32 int0_enable = UMAC_IRQ_TXDMA_DONE;
+ u32 int1_enable = 0;
struct bcmgenet_tx_ring *ring;
for (i = 0; i < priv->hw_params->tx_queues; ++i) {
ring = &priv->tx_rings[i];
napi_enable(&ring->napi);
+ int1_enable |= (1 << i);
}
ring = &priv->tx_rings[DESC_INDEX];
napi_enable(&ring->napi);
+
+ bcmgenet_intrl2_0_writel(priv, int0_enable, INTRL2_CPU_MASK_CLEAR);
+ bcmgenet_intrl2_1_writel(priv, int1_enable, INTRL2_CPU_MASK_CLEAR);
}
static void bcmgenet_disable_tx_napi(struct bcmgenet_priv *priv)
{
unsigned int i;
+ u32 int0_disable = UMAC_IRQ_TXDMA_DONE;
+ u32 int1_disable = 0xffff;
struct bcmgenet_tx_ring *ring;
+ bcmgenet_intrl2_0_writel(priv, int0_disable, INTRL2_CPU_MASK_SET);
+ bcmgenet_intrl2_1_writel(priv, int1_disable, INTRL2_CPU_MASK_SET);
+
for (i = 0; i < priv->hw_params->tx_queues; ++i) {
ring = &priv->tx_rings[i];
napi_disable(&ring->napi);
@@ -2264,22 +2295,33 @@
static void bcmgenet_enable_rx_napi(struct bcmgenet_priv *priv)
{
unsigned int i;
+ u32 int0_enable = UMAC_IRQ_RXDMA_DONE;
+ u32 int1_enable = 0;
struct bcmgenet_rx_ring *ring;
for (i = 0; i < priv->hw_params->rx_queues; ++i) {
ring = &priv->rx_rings[i];
napi_enable(&ring->napi);
+ int1_enable |= (1 << (UMAC_IRQ1_RX_INTR_SHIFT + i));
}
ring = &priv->rx_rings[DESC_INDEX];
napi_enable(&ring->napi);
+
+ bcmgenet_intrl2_0_writel(priv, int0_enable, INTRL2_CPU_MASK_CLEAR);
+ bcmgenet_intrl2_1_writel(priv, int1_enable, INTRL2_CPU_MASK_CLEAR);
}
static void bcmgenet_disable_rx_napi(struct bcmgenet_priv *priv)
{
unsigned int i;
+ u32 int0_disable = UMAC_IRQ_RXDMA_DONE;
+ u32 int1_disable = 0xffff << UMAC_IRQ1_RX_INTR_SHIFT;
struct bcmgenet_rx_ring *ring;
+ bcmgenet_intrl2_0_writel(priv, int0_disable, INTRL2_CPU_MASK_SET);
+ bcmgenet_intrl2_1_writel(priv, int1_disable, INTRL2_CPU_MASK_SET);
+
for (i = 0; i < priv->hw_params->rx_queues; ++i) {
ring = &priv->rx_rings[i];
napi_disable(&ring->napi);
@@ -2634,6 +2676,15 @@
}
}
+ if (priv->irq0_stat & (UMAC_IRQ_PHY_DET_R |
+ UMAC_IRQ_PHY_DET_F |
+ UMAC_IRQ_LINK_EVENT |
+ UMAC_IRQ_HFB_SM |
+ UMAC_IRQ_HFB_MM)) {
+ /* all other interested interrupts handled in bottom half */
+ schedule_work(&priv->bcmgenet_irq_work);
+ }
+
if ((priv->hw_params->flags & GENET_HAS_MDIO_INTR) &&
status & (UMAC_IRQ_MDIO_DONE | UMAC_IRQ_MDIO_ERROR)) {
wake_up(&priv->wq);
@@ -2921,7 +2972,7 @@
if (ret)
return ret;
- /* Disable MAC transmit. TX DMA disabled have to done before this */
+ /* Disable MAC transmit. TX DMA disabled must be done before this */
umac_enable_set(priv, CMD_TX_EN, false);
/* tx reclaim */
@@ -3101,6 +3152,48 @@
return 0;
}
+static struct net_device_stats *bcmgenet_get_stats(struct net_device *dev)
+{
+ struct bcmgenet_priv *priv = netdev_priv(dev);
+ unsigned long tx_bytes = 0, tx_packets = 0;
+ unsigned long rx_bytes = 0, rx_packets = 0;
+ unsigned long rx_errors = 0, rx_dropped = 0;
+ struct bcmgenet_tx_ring *tx_ring;
+ struct bcmgenet_rx_ring *rx_ring;
+ unsigned int q;
+
+ for (q = 0; q < priv->hw_params->tx_queues; q++) {
+ tx_ring = &priv->tx_rings[q];
+ tx_bytes += tx_ring->bytes;
+ tx_packets += tx_ring->packets;
+ }
+ tx_ring = &priv->tx_rings[DESC_INDEX];
+ tx_bytes += tx_ring->bytes;
+ tx_packets += tx_ring->packets;
+
+ for (q = 0; q < priv->hw_params->rx_queues; q++) {
+ rx_ring = &priv->rx_rings[q];
+
+ rx_bytes += rx_ring->bytes;
+ rx_packets += rx_ring->packets;
+ rx_errors += rx_ring->errors;
+ rx_dropped += rx_ring->dropped;
+ }
+ rx_ring = &priv->rx_rings[DESC_INDEX];
+ rx_bytes += rx_ring->bytes;
+ rx_packets += rx_ring->packets;
+ rx_errors += rx_ring->errors;
+ rx_dropped += rx_ring->dropped;
+
+ dev->stats.tx_bytes = tx_bytes;
+ dev->stats.tx_packets = tx_packets;
+ dev->stats.rx_bytes = rx_bytes;
+ dev->stats.rx_packets = rx_packets;
+ dev->stats.rx_errors = rx_errors;
+ dev->stats.rx_missed_errors = rx_errors;
+ return &dev->stats;
+}
+
static const struct net_device_ops bcmgenet_netdev_ops = {
.ndo_open = bcmgenet_open,
.ndo_stop = bcmgenet_close,
@@ -3113,6 +3206,7 @@
#ifdef CONFIG_NET_POLL_CONTROLLER
.ndo_poll_controller = bcmgenet_poll_controller,
#endif
+ .ndo_get_stats = bcmgenet_get_stats,
};
/* Array of GENET hardware parameters/characteristics */
@@ -3186,6 +3280,25 @@
.flags = GENET_HAS_40BITS | GENET_HAS_EXT |
GENET_HAS_MDIO_INTR | GENET_HAS_MOCA_LINK_DET,
},
+ [GENET_V5] = {
+ .tx_queues = 4,
+ .tx_bds_per_q = 32,
+ .rx_queues = 0,
+ .rx_bds_per_q = 0,
+ .bp_in_en_shift = 17,
+ .bp_in_mask = 0x1ffff,
+ .hfb_filter_cnt = 48,
+ .hfb_filter_size = 128,
+ .qtag_mask = 0x3F,
+ .tbuf_offset = 0x0600,
+ .hfb_offset = 0x8000,
+ .hfb_reg_offset = 0xfc00,
+ .rdma_offset = 0x2000,
+ .tdma_offset = 0x4000,
+ .words_per_bd = 3,
+ .flags = GENET_HAS_40BITS | GENET_HAS_EXT |
+ GENET_HAS_MDIO_INTR | GENET_HAS_MOCA_LINK_DET,
+ },
};
/* Infer hardware parameters from the detected GENET version */
@@ -3196,26 +3309,22 @@
u8 major;
u16 gphy_rev;
- if (GENET_IS_V4(priv)) {
+ if (GENET_IS_V5(priv) || GENET_IS_V4(priv)) {
bcmgenet_dma_regs = bcmgenet_dma_regs_v3plus;
genet_dma_ring_regs = genet_dma_ring_regs_v4;
priv->dma_rx_chk_bit = DMA_RX_CHK_V3PLUS;
- priv->version = GENET_V4;
} else if (GENET_IS_V3(priv)) {
bcmgenet_dma_regs = bcmgenet_dma_regs_v3plus;
genet_dma_ring_regs = genet_dma_ring_regs_v123;
priv->dma_rx_chk_bit = DMA_RX_CHK_V3PLUS;
- priv->version = GENET_V3;
} else if (GENET_IS_V2(priv)) {
bcmgenet_dma_regs = bcmgenet_dma_regs_v2;
genet_dma_ring_regs = genet_dma_ring_regs_v123;
priv->dma_rx_chk_bit = DMA_RX_CHK_V12;
- priv->version = GENET_V2;
} else if (GENET_IS_V1(priv)) {
bcmgenet_dma_regs = bcmgenet_dma_regs_v1;
genet_dma_ring_regs = genet_dma_ring_regs_v123;
priv->dma_rx_chk_bit = DMA_RX_CHK_V12;
- priv->version = GENET_V1;
}
/* enum genet_version starts at 1 */
@@ -3225,7 +3334,9 @@
/* Read GENET HW version */
reg = bcmgenet_sys_readl(priv, SYS_REV_CTRL);
major = (reg >> 24 & 0x0f);
- if (major == 5)
+ if (major == 6)
+ major = 5;
+ else if (major == 5)
major = 4;
else if (major == 0)
major = 1;
@@ -3253,19 +3364,25 @@
*/
gphy_rev = reg & 0xffff;
+ if (GENET_IS_V5(priv)) {
+ /* The EPHY revision should come from the MDIO registers of
+ * the PHY not from GENET.
+ */
+ if (gphy_rev != 0) {
+ pr_warn("GENET is reporting EPHY revision: 0x%04x\n",
+ gphy_rev);
+ }
/* This is reserved so should require special treatment */
- if (gphy_rev == 0 || gphy_rev == 0x01ff) {
+ } else if (gphy_rev == 0 || gphy_rev == 0x01ff) {
pr_warn("Invalid GPHY revision detected: 0x%04x\n", gphy_rev);
return;
- }
-
/* This is the good old scheme, just GPHY major, no minor nor patch */
- if ((gphy_rev & 0xf0) != 0)
+ } else if ((gphy_rev & 0xf0) != 0) {
priv->gphy_rev = gphy_rev << 8;
-
/* This is the new scheme, GPHY major rolls over with 0x10 = rev G0 */
- else if ((gphy_rev & 0xff00) != 0)
+ } else if ((gphy_rev & 0xff00) != 0) {
priv->gphy_rev = gphy_rev;
+ }
#ifdef CONFIG_PHYS_ADDR_T_64BIT
if (!(params->flags & GENET_HAS_40BITS))
@@ -3295,6 +3412,7 @@
{ .compatible = "brcm,genet-v2", .data = (void *)GENET_V2 },
{ .compatible = "brcm,genet-v3", .data = (void *)GENET_V3 },
{ .compatible = "brcm,genet-v4", .data = (void *)GENET_V4 },
+ { .compatible = "brcm,genet-v5", .data = (void *)GENET_V5 },
{ },
};
MODULE_DEVICE_TABLE(of, bcmgenet_match);
@@ -3493,7 +3611,7 @@
if (ret)
return ret;
- /* Disable MAC transmit. TX DMA disabled have to done before this */
+ /* Disable MAC transmit. TX DMA disabled must be done before this */
umac_enable_set(priv, CMD_TX_EN, false);
/* tx reclaim */
diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.h b/drivers/net/ethernet/broadcom/genet/bcmgenet.h
index db7f289..efd0702 100644
--- a/drivers/net/ethernet/broadcom/genet/bcmgenet.h
+++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.h
@@ -355,8 +355,14 @@
#define EXT_PWR_DN_EN_LD (1 << 3)
#define EXT_ENERGY_DET (1 << 4)
#define EXT_IDDQ_FROM_PHY (1 << 5)
+#define EXT_IDDQ_GLBL_PWR (1 << 7)
#define EXT_PHY_RESET (1 << 8)
#define EXT_ENERGY_DET_MASK (1 << 12)
+#define EXT_PWR_DOWN_PHY_TX (1 << 16)
+#define EXT_PWR_DOWN_PHY_RX (1 << 17)
+#define EXT_PWR_DOWN_PHY_SD (1 << 18)
+#define EXT_PWR_DOWN_PHY_RD (1 << 19)
+#define EXT_PWR_DOWN_PHY_EN (1 << 20)
#define EXT_RGMII_OOB_CTRL 0x0C
#define RGMII_LINK (1 << 4)
@@ -499,13 +505,15 @@
GENET_V1 = 1,
GENET_V2,
GENET_V3,
- GENET_V4
+ GENET_V4,
+ GENET_V5
};
#define GENET_IS_V1(p) ((p)->version == GENET_V1)
#define GENET_IS_V2(p) ((p)->version == GENET_V2)
#define GENET_IS_V3(p) ((p)->version == GENET_V3)
#define GENET_IS_V4(p) ((p)->version == GENET_V4)
+#define GENET_IS_V5(p) ((p)->version == GENET_V5)
/* Hardware flags */
#define GENET_HAS_40BITS (1 << 0)
@@ -544,6 +552,8 @@
struct bcmgenet_tx_ring {
spinlock_t lock; /* ring lock */
struct napi_struct napi; /* NAPI per tx queue */
+ unsigned long packets;
+ unsigned long bytes;
unsigned int index; /* ring index */
unsigned int queue; /* queue index */
struct enet_cb *cbs; /* tx ring buffer control block*/
@@ -562,6 +572,10 @@
struct bcmgenet_rx_ring {
struct napi_struct napi; /* Rx NAPI struct */
+ unsigned long bytes;
+ unsigned long packets;
+ unsigned long errors;
+ unsigned long dropped;
unsigned int index; /* Rx ring index */
struct enet_cb *cbs; /* Rx ring buffer control block */
unsigned int size; /* Rx ring size */
diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c b/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c
index b971229..2fbd027 100644
--- a/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c
+++ b/drivers/net/ethernet/broadcom/genet/bcmgenet_wol.c
@@ -1,7 +1,7 @@
/*
* Broadcom GENET (Gigabit Ethernet) Wake-on-LAN support
*
- * Copyright (c) 2014 Broadcom Corporation
+ * Copyright (c) 2014-2017 Broadcom
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -127,7 +127,6 @@
enum bcmgenet_power_mode mode)
{
struct net_device *dev = priv->dev;
- u32 cpu_mask_clear;
int retries = 0;
u32 reg;
@@ -173,18 +172,12 @@
bcmgenet_ext_writel(priv, reg, EXT_EXT_PWR_MGMT);
}
- /* Enable the MPD interrupt */
- cpu_mask_clear = UMAC_IRQ_MPD_R;
-
- bcmgenet_intrl2_0_writel(priv, cpu_mask_clear, INTRL2_CPU_MASK_CLEAR);
-
return 0;
}
void bcmgenet_wol_power_up_cfg(struct bcmgenet_priv *priv,
enum bcmgenet_power_mode mode)
{
- u32 cpu_mask_set;
u32 reg;
if (mode != GENET_POWER_WOL_MAGIC) {
@@ -201,10 +194,4 @@
reg &= ~CMD_CRC_FWD;
bcmgenet_umac_writel(priv, reg, UMAC_CMD);
priv->crc_fwd_en = 0;
-
- /* Stop monitoring magic packet IRQ */
- cpu_mask_set = UMAC_IRQ_MPD_R;
-
- /* Stop monitoring magic packet IRQ */
- bcmgenet_intrl2_0_writel(priv, cpu_mask_set, INTRL2_CPU_MASK_SET);
}
diff --git a/drivers/net/ethernet/broadcom/genet/bcmmii.c b/drivers/net/ethernet/broadcom/genet/bcmmii.c
index 2f92819..285676f 100644
--- a/drivers/net/ethernet/broadcom/genet/bcmmii.c
+++ b/drivers/net/ethernet/broadcom/genet/bcmmii.c
@@ -1,7 +1,7 @@
/*
* Broadcom GENET MDIO routines
*
- * Copyright (c) 2014 Broadcom Corporation
+ * Copyright (c) 2014-2017 Broadcom
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -195,39 +195,43 @@
u32 reg = 0;
/* EXT_GPHY_CTRL is only valid for GENETv4 and onward */
- if (!GENET_IS_V4(priv))
- return;
+ if (GENET_IS_V4(priv)) {
+ reg = bcmgenet_ext_readl(priv, EXT_GPHY_CTRL);
+ if (enable) {
+ reg &= ~EXT_CK25_DIS;
+ bcmgenet_ext_writel(priv, reg, EXT_GPHY_CTRL);
+ mdelay(1);
- reg = bcmgenet_ext_readl(priv, EXT_GPHY_CTRL);
- if (enable) {
- reg &= ~EXT_CK25_DIS;
+ reg &= ~(EXT_CFG_IDDQ_BIAS | EXT_CFG_PWR_DOWN);
+ reg |= EXT_GPHY_RESET;
+ bcmgenet_ext_writel(priv, reg, EXT_GPHY_CTRL);
+ mdelay(1);
+
+ reg &= ~EXT_GPHY_RESET;
+ } else {
+ reg |= EXT_CFG_IDDQ_BIAS | EXT_CFG_PWR_DOWN |
+ EXT_GPHY_RESET;
+ bcmgenet_ext_writel(priv, reg, EXT_GPHY_CTRL);
+ mdelay(1);
+ reg |= EXT_CK25_DIS;
+ }
bcmgenet_ext_writel(priv, reg, EXT_GPHY_CTRL);
- mdelay(1);
-
- reg &= ~(EXT_CFG_IDDQ_BIAS | EXT_CFG_PWR_DOWN);
- reg |= EXT_GPHY_RESET;
- bcmgenet_ext_writel(priv, reg, EXT_GPHY_CTRL);
- mdelay(1);
-
- reg &= ~EXT_GPHY_RESET;
+ udelay(60);
} else {
- reg |= EXT_CFG_IDDQ_BIAS | EXT_CFG_PWR_DOWN | EXT_GPHY_RESET;
- bcmgenet_ext_writel(priv, reg, EXT_GPHY_CTRL);
mdelay(1);
- reg |= EXT_CK25_DIS;
}
- bcmgenet_ext_writel(priv, reg, EXT_GPHY_CTRL);
- udelay(60);
}
static void bcmgenet_moca_phy_setup(struct bcmgenet_priv *priv)
{
u32 reg;
- /* Speed settings are set in bcmgenet_mii_setup() */
- reg = bcmgenet_sys_readl(priv, SYS_PORT_CTRL);
- reg |= LED_ACT_SOURCE_MAC;
- bcmgenet_sys_writel(priv, reg, SYS_PORT_CTRL);
+ if (!GENET_IS_V5(priv)) {
+ /* Speed settings are set in bcmgenet_mii_setup() */
+ reg = bcmgenet_sys_readl(priv, SYS_PORT_CTRL);
+ reg |= LED_ACT_SOURCE_MAC;
+ bcmgenet_sys_writel(priv, reg, SYS_PORT_CTRL);
+ }
if (priv->hw_params->flags & GENET_HAS_MOCA_LINK_DET)
fixed_phy_set_link_update(priv->phydev,
diff --git a/drivers/net/ethernet/broadcom/tg3.c b/drivers/net/ethernet/broadcom/tg3.c
index 30d1eb9..537d571 100644
--- a/drivers/net/ethernet/broadcom/tg3.c
+++ b/drivers/net/ethernet/broadcom/tg3.c
@@ -825,6 +825,7 @@
return timeout_us ? 0 : -EBUSY;
}
+#ifdef CONFIG_TIGON3_HWMON
static int tg3_ape_wait_for_event(struct tg3 *tp, u32 timeout_us)
{
u32 i, apedata;
@@ -904,6 +905,7 @@
return 0;
}
+#endif
static int tg3_ape_send_event(struct tg3 *tp, u32 event)
{
@@ -10744,6 +10746,7 @@
return tg3_reset_hw(tp, reset_phy);
}
+#ifdef CONFIG_TIGON3_HWMON
static void tg3_sd_scan_scratchpad(struct tg3 *tp, struct tg3_ocir *ocir)
{
int i;
@@ -10826,6 +10829,10 @@
dev_err(&pdev->dev, "Cannot register hwmon device, aborting\n");
}
}
+#else
+static inline void tg3_hwmon_close(struct tg3 *tp) { }
+static inline void tg3_hwmon_open(struct tg3 *tp) { }
+#endif /* CONFIG_TIGON3_HWMON */
#define TG3_STAT_ADD32(PSTAT, REG) \
@@ -11722,10 +11729,6 @@
tg3_stop(tp);
- /* Clear stats across close / open calls */
- memset(&tp->net_stats_prev, 0, sizeof(tp->net_stats_prev));
- memset(&tp->estats_prev, 0, sizeof(tp->estats_prev));
-
if (pci_device_is_present(tp->pdev)) {
tg3_power_down_prepare(tp);
diff --git a/drivers/net/ethernet/cadence/macb.c b/drivers/net/ethernet/cadence/macb.c
index 30606b1..91f7492 100644
--- a/drivers/net/ethernet/cadence/macb.c
+++ b/drivers/net/ethernet/cadence/macb.c
@@ -432,15 +432,17 @@
}
pdata = dev_get_platdata(&bp->pdev->dev);
- if (pdata && gpio_is_valid(pdata->phy_irq_pin)) {
- ret = devm_gpio_request(&bp->pdev->dev, pdata->phy_irq_pin,
- "phy int");
- if (!ret) {
- phy_irq = gpio_to_irq(pdata->phy_irq_pin);
- phydev->irq = (phy_irq < 0) ? PHY_POLL : phy_irq;
+ if (pdata) {
+ if (gpio_is_valid(pdata->phy_irq_pin)) {
+ ret = devm_gpio_request(&bp->pdev->dev,
+ pdata->phy_irq_pin, "phy int");
+ if (!ret) {
+ phy_irq = gpio_to_irq(pdata->phy_irq_pin);
+ phydev->irq = (phy_irq < 0) ? PHY_POLL : phy_irq;
+ }
+ } else {
+ phydev->irq = PHY_POLL;
}
- } else {
- phydev->irq = PHY_POLL;
}
/* attach the mac to the phy */
@@ -684,8 +686,8 @@
netdev_vdbg(bp->dev, "txerr skb %u (data %p) TX complete\n",
macb_tx_ring_wrap(bp, tail),
skb->data);
- bp->stats.tx_packets++;
- bp->stats.tx_bytes += skb->len;
+ bp->dev->stats.tx_packets++;
+ bp->dev->stats.tx_bytes += skb->len;
}
} else {
/* "Buffers exhausted mid-frame" errors may only happen
@@ -778,8 +780,8 @@
netdev_vdbg(bp->dev, "skb %u (data %p) TX complete\n",
macb_tx_ring_wrap(bp, tail),
skb->data);
- bp->stats.tx_packets++;
- bp->stats.tx_bytes += skb->len;
+ bp->dev->stats.tx_packets++;
+ bp->dev->stats.tx_bytes += skb->len;
}
/* Now we can safely release resources */
@@ -911,14 +913,14 @@
if (!(ctrl & MACB_BIT(RX_SOF) && ctrl & MACB_BIT(RX_EOF))) {
netdev_err(bp->dev,
"not whole frame pointed by descriptor\n");
- bp->stats.rx_dropped++;
+ bp->dev->stats.rx_dropped++;
break;
}
skb = bp->rx_skbuff[entry];
if (unlikely(!skb)) {
netdev_err(bp->dev,
"inconsistent Rx descriptor chain\n");
- bp->stats.rx_dropped++;
+ bp->dev->stats.rx_dropped++;
break;
}
/* now everything is ready for receiving packet */
@@ -938,8 +940,8 @@
GEM_BFEXT(RX_CSUM, ctrl) & GEM_RX_CSUM_CHECKED_MASK)
skb->ip_summed = CHECKSUM_UNNECESSARY;
- bp->stats.rx_packets++;
- bp->stats.rx_bytes += skb->len;
+ bp->dev->stats.rx_packets++;
+ bp->dev->stats.rx_bytes += skb->len;
#if defined(DEBUG) && defined(VERBOSE_DEBUG)
netdev_vdbg(bp->dev, "received skb of length %u, csum: %08x\n",
@@ -984,7 +986,7 @@
*/
skb = netdev_alloc_skb(bp->dev, len + NET_IP_ALIGN);
if (!skb) {
- bp->stats.rx_dropped++;
+ bp->dev->stats.rx_dropped++;
for (frag = first_frag; ; frag++) {
desc = macb_rx_desc(bp, frag);
desc->addr &= ~MACB_BIT(RX_USED);
@@ -1030,8 +1032,8 @@
__skb_pull(skb, NET_IP_ALIGN);
skb->protocol = eth_type_trans(skb, bp->dev);
- bp->stats.rx_packets++;
- bp->stats.rx_bytes += skb->len;
+ bp->dev->stats.rx_packets++;
+ bp->dev->stats.rx_bytes += skb->len;
netdev_vdbg(bp->dev, "received skb of length %u, csum: %08x\n",
skb->len, skb->csum);
netif_receive_skb(skb);
@@ -2210,7 +2212,7 @@
static struct net_device_stats *gem_get_stats(struct macb *bp)
{
struct gem_stats *hwstat = &bp->hw_stats.gem;
- struct net_device_stats *nstat = &bp->stats;
+ struct net_device_stats *nstat = &bp->dev->stats;
gem_update_stats(bp);
@@ -2281,7 +2283,7 @@
static struct net_device_stats *macb_get_stats(struct net_device *dev)
{
struct macb *bp = netdev_priv(dev);
- struct net_device_stats *nstat = &bp->stats;
+ struct net_device_stats *nstat = &bp->dev->stats;
struct macb_stats *hwstat = &bp->hw_stats.macb;
if (macb_is_gem(bp))
@@ -2993,15 +2995,15 @@
memcpy(skb_put(skb, pktlen), p_recv, pktlen);
skb->protocol = eth_type_trans(skb, dev);
- lp->stats.rx_packets++;
- lp->stats.rx_bytes += pktlen;
+ dev->stats.rx_packets++;
+ dev->stats.rx_bytes += pktlen;
netif_rx(skb);
} else {
- lp->stats.rx_dropped++;
+ dev->stats.rx_dropped++;
}
if (desc->ctrl & MACB_BIT(RX_MHASH_MATCH))
- lp->stats.multicast++;
+ dev->stats.multicast++;
/* reset ownership bit */
desc->addr &= ~MACB_BIT(RX_USED);
@@ -3036,15 +3038,15 @@
if (intstatus & MACB_BIT(TCOMP)) {
/* The TCOM bit is set even if the transmission failed */
if (intstatus & (MACB_BIT(ISR_TUND) | MACB_BIT(ISR_RLE)))
- lp->stats.tx_errors++;
+ dev->stats.tx_errors++;
if (lp->skb) {
dev_kfree_skb_irq(lp->skb);
lp->skb = NULL;
dma_unmap_single(NULL, lp->skb_physaddr,
lp->skb_length, DMA_TO_DEVICE);
- lp->stats.tx_packets++;
- lp->stats.tx_bytes += lp->skb_length;
+ dev->stats.tx_packets++;
+ dev->stats.tx_bytes += lp->skb_length;
}
netif_wake_queue(dev);
}
diff --git a/drivers/net/ethernet/cadence/macb.h b/drivers/net/ethernet/cadence/macb.h
index 234a49e..ec037b0 100644
--- a/drivers/net/ethernet/cadence/macb.h
+++ b/drivers/net/ethernet/cadence/macb.h
@@ -919,7 +919,6 @@
struct clk *rx_clk;
struct net_device *dev;
struct napi_struct napi;
- struct net_device_stats stats;
union {
struct macb_stats macb;
struct gem_stats gem;
diff --git a/drivers/net/ethernet/cavium/liquidio/cn23xx_pf_device.h b/drivers/net/ethernet/cavium/liquidio/cn23xx_pf_device.h
index 2fedd91..dee6046 100644
--- a/drivers/net/ethernet/cavium/liquidio/cn23xx_pf_device.h
+++ b/drivers/net/ethernet/cavium/liquidio/cn23xx_pf_device.h
@@ -43,6 +43,8 @@
struct octeon_config *conf;
};
+#define CN23XX_SLI_DEF_BP 0x40
+
int setup_cn23xx_octeon_pf_device(struct octeon_device *oct);
int validate_cn23xx_pf_config_info(struct octeon_device *oct,
diff --git a/drivers/net/ethernet/cavium/liquidio/lio_core.c b/drivers/net/ethernet/cavium/liquidio/lio_core.c
index f629c2f..796c2cb 100644
--- a/drivers/net/ethernet/cavium/liquidio/lio_core.c
+++ b/drivers/net/ethernet/cavium/liquidio/lio_core.c
@@ -26,6 +26,9 @@
#include "octeon_main.h"
#include "octeon_network.h"
+/* OOM task polling interval */
+#define LIO_OOM_POLL_INTERVAL_MS 250
+
int liquidio_set_feature(struct net_device *netdev, int cmd, u16 param1)
{
struct lio *lio = GET_LIO(netdev);
@@ -124,6 +127,17 @@
struct octeon_device *oct = lio->oct_dev;
u8 *mac;
+ if (nctrl->completion && nctrl->response_code) {
+ /* Signal whoever is interested that the response code from the
+ * firmware has arrived.
+ */
+ WRITE_ONCE(*nctrl->response_code, nctrl->status);
+ complete(nctrl->completion);
+ }
+
+ if (nctrl->status)
+ return;
+
switch (nctrl->ncmd.s.cmd) {
case OCTNET_CMD_CHANGE_DEVFLAGS:
case OCTNET_CMD_SET_MULTI_LIST:
@@ -131,11 +145,20 @@
case OCTNET_CMD_CHANGE_MACADDR:
mac = ((u8 *)&nctrl->udd[0]) + 2;
- netif_info(lio, probe, lio->netdev,
- "MACAddr changed to %2.2x:%2.2x:%2.2x:%2.2x:%2.2x:%2.2x\n",
- mac[0], mac[1],
- mac[2], mac[3],
- mac[4], mac[5]);
+ if (nctrl->ncmd.s.param1) {
+ /* vfidx is 0 based, but vf_num (param1) is 1 based */
+ int vfidx = nctrl->ncmd.s.param1 - 1;
+ bool mac_is_admin_assigned = nctrl->ncmd.s.param2;
+
+ if (mac_is_admin_assigned)
+ netif_info(lio, probe, lio->netdev,
+ "MAC Address %pM is configured for VF %d\n",
+ mac, vfidx);
+ } else {
+ netif_info(lio, probe, lio->netdev,
+ " MACAddr changed to %pM\n",
+ mac);
+ }
break;
case OCTNET_CMD_CHANGE_MTU:
@@ -284,3 +307,56 @@
* the PF did that already
*/
}
+
+static void octnet_poll_check_rxq_oom_status(struct work_struct *work)
+{
+ struct cavium_wk *wk = (struct cavium_wk *)work;
+ struct lio *lio = (struct lio *)wk->ctxptr;
+ struct octeon_device *oct = lio->oct_dev;
+ struct octeon_droq *droq;
+ int q, q_no = 0;
+
+ if (ifstate_check(lio, LIO_IFSTATE_RUNNING)) {
+ for (q = 0; q < lio->linfo.num_rxpciq; q++) {
+ q_no = lio->linfo.rxpciq[q].s.q_no;
+ droq = oct->droq[q_no];
+ if (!droq)
+ continue;
+ octeon_droq_check_oom(droq);
+ }
+ }
+ queue_delayed_work(lio->rxq_status_wq.wq,
+ &lio->rxq_status_wq.wk.work,
+ msecs_to_jiffies(LIO_OOM_POLL_INTERVAL_MS));
+}
+
+int setup_rx_oom_poll_fn(struct net_device *netdev)
+{
+ struct lio *lio = GET_LIO(netdev);
+ struct octeon_device *oct = lio->oct_dev;
+
+ lio->rxq_status_wq.wq = alloc_workqueue("rxq-oom-status",
+ WQ_MEM_RECLAIM, 0);
+ if (!lio->rxq_status_wq.wq) {
+ dev_err(&oct->pci_dev->dev, "unable to create cavium rxq oom status wq\n");
+ return -ENOMEM;
+ }
+ INIT_DELAYED_WORK(&lio->rxq_status_wq.wk.work,
+ octnet_poll_check_rxq_oom_status);
+ lio->rxq_status_wq.wk.ctxptr = lio;
+ queue_delayed_work(lio->rxq_status_wq.wq,
+ &lio->rxq_status_wq.wk.work,
+ msecs_to_jiffies(LIO_OOM_POLL_INTERVAL_MS));
+ return 0;
+}
+
+void cleanup_rx_oom_poll_fn(struct net_device *netdev)
+{
+ struct lio *lio = GET_LIO(netdev);
+
+ if (lio->rxq_status_wq.wq) {
+ cancel_delayed_work_sync(&lio->rxq_status_wq.wk.work);
+ flush_workqueue(lio->rxq_status_wq.wq);
+ destroy_workqueue(lio->rxq_status_wq.wq);
+ }
+}
diff --git a/drivers/net/ethernet/cavium/liquidio/lio_ethtool.c b/drivers/net/ethernet/cavium/liquidio/lio_ethtool.c
index 50384ce..579dc73 100644
--- a/drivers/net/ethernet/cavium/liquidio/lio_ethtool.c
+++ b/drivers/net/ethernet/cavium/liquidio/lio_ethtool.c
@@ -33,6 +33,19 @@
static int octnet_get_link_stats(struct net_device *netdev);
+struct oct_intrmod_context {
+ int octeon_id;
+ wait_queue_head_t wc;
+ int cond;
+ int status;
+};
+
+struct oct_intrmod_resp {
+ u64 rh;
+ struct oct_intrmod_cfg intrmod;
+ u64 status;
+};
+
struct oct_mdio_cmd_context {
int octeon_id;
wait_queue_head_t wc;
@@ -213,17 +226,23 @@
struct lio *lio = GET_LIO(netdev);
struct octeon_device *oct = lio->oct_dev;
struct oct_link_info *linfo;
- u32 supported, advertising;
+ u32 supported = 0, advertising = 0;
linfo = &lio->linfo;
if (linfo->link.s.if_mode == INTERFACE_MODE_XAUI ||
linfo->link.s.if_mode == INTERFACE_MODE_RXAUI ||
+ linfo->link.s.if_mode == INTERFACE_MODE_XLAUI ||
linfo->link.s.if_mode == INTERFACE_MODE_XFI) {
ecmd->base.port = PORT_FIBRE;
- supported = (SUPPORTED_10000baseT_Full | SUPPORTED_FIBRE |
- SUPPORTED_Pause);
- advertising = (ADVERTISED_10000baseT_Full | ADVERTISED_Pause);
+
+ if (linfo->link.s.speed == SPEED_10000) {
+ supported = SUPPORTED_10000baseT_Full;
+ advertising = ADVERTISED_10000baseT_Full;
+ }
+
+ supported |= SUPPORTED_FIBRE | SUPPORTED_Pause;
+ advertising |= ADVERTISED_Pause;
ethtool_convert_legacy_u32_to_link_mode(
ecmd->link_modes.supported, supported);
ethtool_convert_legacy_u32_to_link_mode(
@@ -1292,95 +1311,101 @@
}
}
-static int lio_get_intr_coalesce(struct net_device *netdev,
- struct ethtool_coalesce *intr_coal)
-{
- struct lio *lio = GET_LIO(netdev);
- struct octeon_device *oct = lio->oct_dev;
- struct octeon_instr_queue *iq;
- struct oct_intrmod_cfg *intrmod_cfg;
-
- intrmod_cfg = &oct->intrmod;
-
- switch (oct->chip_id) {
- case OCTEON_CN23XX_PF_VID:
- case OCTEON_CN23XX_VF_VID:
- if (!intrmod_cfg->rx_enable) {
- intr_coal->rx_coalesce_usecs = intrmod_cfg->rx_usecs;
- intr_coal->rx_max_coalesced_frames =
- intrmod_cfg->rx_frames;
- }
- if (!intrmod_cfg->tx_enable)
- intr_coal->tx_max_coalesced_frames =
- intrmod_cfg->tx_frames;
- break;
- case OCTEON_CN68XX:
- case OCTEON_CN66XX: {
- struct octeon_cn6xxx *cn6xxx =
- (struct octeon_cn6xxx *)oct->chip;
-
- if (!intrmod_cfg->rx_enable) {
- intr_coal->rx_coalesce_usecs =
- CFG_GET_OQ_INTR_TIME(cn6xxx->conf);
- intr_coal->rx_max_coalesced_frames =
- CFG_GET_OQ_INTR_PKT(cn6xxx->conf);
- }
- iq = oct->instr_queue[lio->linfo.txpciq[0].s.q_no];
- intr_coal->tx_max_coalesced_frames = iq->fill_threshold;
- break;
- }
- default:
- netif_info(lio, drv, lio->netdev, "Unknown Chip !!\n");
- return -EINVAL;
- }
- if (intrmod_cfg->rx_enable) {
- intr_coal->use_adaptive_rx_coalesce =
- intrmod_cfg->rx_enable;
- intr_coal->rate_sample_interval =
- intrmod_cfg->check_intrvl;
- intr_coal->pkt_rate_high =
- intrmod_cfg->maxpkt_ratethr;
- intr_coal->pkt_rate_low =
- intrmod_cfg->minpkt_ratethr;
- intr_coal->rx_max_coalesced_frames_high =
- intrmod_cfg->rx_maxcnt_trigger;
- intr_coal->rx_coalesce_usecs_high =
- intrmod_cfg->rx_maxtmr_trigger;
- intr_coal->rx_coalesce_usecs_low =
- intrmod_cfg->rx_mintmr_trigger;
- intr_coal->rx_max_coalesced_frames_low =
- intrmod_cfg->rx_mincnt_trigger;
- }
- if ((OCTEON_CN23XX_PF(oct) || OCTEON_CN23XX_VF(oct)) &&
- (intrmod_cfg->tx_enable)) {
- intr_coal->use_adaptive_tx_coalesce = intrmod_cfg->tx_enable;
- intr_coal->tx_max_coalesced_frames_high =
- intrmod_cfg->tx_maxcnt_trigger;
- intr_coal->tx_max_coalesced_frames_low =
- intrmod_cfg->tx_mincnt_trigger;
- }
- return 0;
-}
-
/* Callback function for intrmod */
static void octnet_intrmod_callback(struct octeon_device *oct_dev,
u32 status,
void *ptr)
{
- struct oct_intrmod_cmd *cmd = ptr;
- struct octeon_soft_command *sc = cmd->sc;
+ struct octeon_soft_command *sc = (struct octeon_soft_command *)ptr;
+ struct oct_intrmod_context *ctx;
- oct_dev = cmd->oct_dev;
+ ctx = (struct oct_intrmod_context *)sc->ctxptr;
- if (status)
- dev_err(&oct_dev->pci_dev->dev, "intrmod config failed. Status: %llx\n",
- CVM_CAST64(status));
- else
- dev_info(&oct_dev->pci_dev->dev,
- "Rx-Adaptive Interrupt moderation enabled:%llx\n",
- oct_dev->intrmod.rx_enable);
+ ctx->status = status;
+
+ WRITE_ONCE(ctx->cond, 1);
+
+ /* This barrier is required to be sure that the response has been
+ * written fully before waking up the handler
+ */
+ wmb();
+
+ wake_up_interruptible(&ctx->wc);
+}
+
+/* get interrupt moderation parameters */
+static int octnet_get_intrmod_cfg(struct lio *lio,
+ struct oct_intrmod_cfg *intr_cfg)
+{
+ struct octeon_soft_command *sc;
+ struct oct_intrmod_context *ctx;
+ struct oct_intrmod_resp *resp;
+ int retval;
+ struct octeon_device *oct_dev = lio->oct_dev;
+
+ /* Alloc soft command */
+ sc = (struct octeon_soft_command *)
+ octeon_alloc_soft_command(oct_dev,
+ 0,
+ sizeof(struct oct_intrmod_resp),
+ sizeof(struct oct_intrmod_context));
+
+ if (!sc)
+ return -ENOMEM;
+
+ resp = (struct oct_intrmod_resp *)sc->virtrptr;
+ memset(resp, 0, sizeof(struct oct_intrmod_resp));
+
+ ctx = (struct oct_intrmod_context *)sc->ctxptr;
+ memset(ctx, 0, sizeof(struct oct_intrmod_context));
+ WRITE_ONCE(ctx->cond, 0);
+ ctx->octeon_id = lio_get_device_id(oct_dev);
+ init_waitqueue_head(&ctx->wc);
+
+ sc->iq_no = lio->linfo.txpciq[0].s.q_no;
+
+ octeon_prepare_soft_command(oct_dev, sc, OPCODE_NIC,
+ OPCODE_NIC_INTRMOD_PARAMS, 0, 0, 0);
+
+ sc->callback = octnet_intrmod_callback;
+ sc->callback_arg = sc;
+ sc->wait_time = 1000;
+
+ retval = octeon_send_soft_command(oct_dev, sc);
+ if (retval == IQ_SEND_FAILED) {
+ octeon_free_soft_command(oct_dev, sc);
+ return -EINVAL;
+ }
+
+ /* Sleep on a wait queue till the cond flag indicates that the
+ * response arrived or timed-out.
+ */
+ if (sleep_cond(&ctx->wc, &ctx->cond) == -EINTR) {
+ dev_err(&oct_dev->pci_dev->dev, "Wait interrupted\n");
+ goto intrmod_info_wait_intr;
+ }
+
+ retval = ctx->status || resp->status;
+ if (retval) {
+ dev_err(&oct_dev->pci_dev->dev,
+ "Get interrupt moderation parameters failed\n");
+ goto intrmod_info_wait_fail;
+ }
+
+ octeon_swap_8B_data((u64 *)&resp->intrmod,
+ (sizeof(struct oct_intrmod_cfg)) / 8);
+ memcpy(intr_cfg, &resp->intrmod, sizeof(struct oct_intrmod_cfg));
+ octeon_free_soft_command(oct_dev, sc);
+
+ return 0;
+
+intrmod_info_wait_fail:
octeon_free_soft_command(oct_dev, sc);
+
+intrmod_info_wait_intr:
+
+ return -ENODEV;
}
/* Configure interrupt moderation parameters */
@@ -1388,7 +1413,7 @@
struct oct_intrmod_cfg *intr_cfg)
{
struct octeon_soft_command *sc;
- struct oct_intrmod_cmd *cmd;
+ struct oct_intrmod_context *ctx;
struct oct_intrmod_cfg *cfg;
int retval;
struct octeon_device *oct_dev = lio->oct_dev;
@@ -1398,19 +1423,21 @@
octeon_alloc_soft_command(oct_dev,
sizeof(struct oct_intrmod_cfg),
0,
- sizeof(struct oct_intrmod_cmd));
+ sizeof(struct oct_intrmod_context));
if (!sc)
return -ENOMEM;
- cmd = (struct oct_intrmod_cmd *)sc->ctxptr;
+ ctx = (struct oct_intrmod_context *)sc->ctxptr;
+
+ WRITE_ONCE(ctx->cond, 0);
+ ctx->octeon_id = lio_get_device_id(oct_dev);
+ init_waitqueue_head(&ctx->wc);
+
cfg = (struct oct_intrmod_cfg *)sc->virtdptr;
memcpy(cfg, intr_cfg, sizeof(struct oct_intrmod_cfg));
octeon_swap_8B_data((u64 *)cfg, (sizeof(struct oct_intrmod_cfg)) / 8);
- cmd->sc = sc;
- cmd->cfg = cfg;
- cmd->oct_dev = oct_dev;
sc->iq_no = lio->linfo.txpciq[0].s.q_no;
@@ -1418,7 +1445,7 @@
OPCODE_NIC_INTRMOD_CFG, 0, 0, 0);
sc->callback = octnet_intrmod_callback;
- sc->callback_arg = cmd;
+ sc->callback_arg = sc;
sc->wait_time = 1000;
retval = octeon_send_soft_command(oct_dev, sc);
@@ -1427,7 +1454,29 @@
return -EINVAL;
}
- return 0;
+ /* Sleep on a wait queue till the cond flag indicates that the
+ * response arrived or timed-out.
+ */
+ if (sleep_cond(&ctx->wc, &ctx->cond) != -EINTR) {
+ retval = ctx->status;
+ if (retval)
+ dev_err(&oct_dev->pci_dev->dev,
+ "intrmod config failed. Status: %llx\n",
+ CVM_CAST64(retval));
+ else
+ dev_info(&oct_dev->pci_dev->dev,
+ "Rx-Adaptive Interrupt moderation %s\n",
+ (intr_cfg->rx_enable) ?
+ "enabled" : "disabled");
+
+ octeon_free_soft_command(oct_dev, sc);
+
+ return ((retval) ? -ENODEV : 0);
+ }
+
+ dev_err(&oct_dev->pci_dev->dev, "iq/oq config failed\n");
+
+ return -EINTR;
}
static void
@@ -1584,80 +1633,106 @@
return 0;
}
+static int lio_get_intr_coalesce(struct net_device *netdev,
+ struct ethtool_coalesce *intr_coal)
+{
+ struct lio *lio = GET_LIO(netdev);
+ struct octeon_device *oct = lio->oct_dev;
+ struct octeon_instr_queue *iq;
+ struct oct_intrmod_cfg intrmod_cfg;
+
+ if (octnet_get_intrmod_cfg(lio, &intrmod_cfg))
+ return -ENODEV;
+
+ switch (oct->chip_id) {
+ case OCTEON_CN23XX_PF_VID:
+ case OCTEON_CN23XX_VF_VID: {
+ if (!intrmod_cfg.rx_enable) {
+ intr_coal->rx_coalesce_usecs = oct->rx_coalesce_usecs;
+ intr_coal->rx_max_coalesced_frames =
+ oct->rx_max_coalesced_frames;
+ }
+ if (!intrmod_cfg.tx_enable)
+ intr_coal->tx_max_coalesced_frames =
+ oct->tx_max_coalesced_frames;
+ break;
+ }
+ case OCTEON_CN68XX:
+ case OCTEON_CN66XX: {
+ struct octeon_cn6xxx *cn6xxx =
+ (struct octeon_cn6xxx *)oct->chip;
+
+ if (!intrmod_cfg.rx_enable) {
+ intr_coal->rx_coalesce_usecs =
+ CFG_GET_OQ_INTR_TIME(cn6xxx->conf);
+ intr_coal->rx_max_coalesced_frames =
+ CFG_GET_OQ_INTR_PKT(cn6xxx->conf);
+ }
+ iq = oct->instr_queue[lio->linfo.txpciq[0].s.q_no];
+ intr_coal->tx_max_coalesced_frames = iq->fill_threshold;
+ break;
+ }
+ default:
+ netif_info(lio, drv, lio->netdev, "Unknown Chip !!\n");
+ return -EINVAL;
+ }
+ if (intrmod_cfg.rx_enable) {
+ intr_coal->use_adaptive_rx_coalesce =
+ intrmod_cfg.rx_enable;
+ intr_coal->rate_sample_interval =
+ intrmod_cfg.check_intrvl;
+ intr_coal->pkt_rate_high =
+ intrmod_cfg.maxpkt_ratethr;
+ intr_coal->pkt_rate_low =
+ intrmod_cfg.minpkt_ratethr;
+ intr_coal->rx_max_coalesced_frames_high =
+ intrmod_cfg.rx_maxcnt_trigger;
+ intr_coal->rx_coalesce_usecs_high =
+ intrmod_cfg.rx_maxtmr_trigger;
+ intr_coal->rx_coalesce_usecs_low =
+ intrmod_cfg.rx_mintmr_trigger;
+ intr_coal->rx_max_coalesced_frames_low =
+ intrmod_cfg.rx_mincnt_trigger;
+ }
+ if ((OCTEON_CN23XX_PF(oct) || OCTEON_CN23XX_VF(oct)) &&
+ (intrmod_cfg.tx_enable)) {
+ intr_coal->use_adaptive_tx_coalesce =
+ intrmod_cfg.tx_enable;
+ intr_coal->tx_max_coalesced_frames_high =
+ intrmod_cfg.tx_maxcnt_trigger;
+ intr_coal->tx_max_coalesced_frames_low =
+ intrmod_cfg.tx_mincnt_trigger;
+ }
+ return 0;
+}
+
/* Enable/Disable auto interrupt Moderation */
-static int oct_cfg_adaptive_intr(struct lio *lio, struct ethtool_coalesce
- *intr_coal)
+static int oct_cfg_adaptive_intr(struct lio *lio,
+ struct oct_intrmod_cfg *intrmod_cfg,
+ struct ethtool_coalesce *intr_coal)
{
int ret = 0;
- struct octeon_device *oct = lio->oct_dev;
- struct oct_intrmod_cfg *intrmod_cfg;
- intrmod_cfg = &oct->intrmod;
-
- if (oct->intrmod.rx_enable || oct->intrmod.tx_enable) {
- if (intr_coal->rate_sample_interval)
- intrmod_cfg->check_intrvl =
- intr_coal->rate_sample_interval;
- else
- intrmod_cfg->check_intrvl =
- LIO_INTRMOD_CHECK_INTERVAL;
-
- if (intr_coal->pkt_rate_high)
- intrmod_cfg->maxpkt_ratethr =
- intr_coal->pkt_rate_high;
- else
- intrmod_cfg->maxpkt_ratethr =
- LIO_INTRMOD_MAXPKT_RATETHR;
-
- if (intr_coal->pkt_rate_low)
- intrmod_cfg->minpkt_ratethr =
- intr_coal->pkt_rate_low;
- else
- intrmod_cfg->minpkt_ratethr =
- LIO_INTRMOD_MINPKT_RATETHR;
+ if (intrmod_cfg->rx_enable || intrmod_cfg->tx_enable) {
+ intrmod_cfg->check_intrvl = intr_coal->rate_sample_interval;
+ intrmod_cfg->maxpkt_ratethr = intr_coal->pkt_rate_high;
+ intrmod_cfg->minpkt_ratethr = intr_coal->pkt_rate_low;
}
- if (oct->intrmod.rx_enable) {
- if (intr_coal->rx_max_coalesced_frames_high)
- intrmod_cfg->rx_maxcnt_trigger =
- intr_coal->rx_max_coalesced_frames_high;
- else
- intrmod_cfg->rx_maxcnt_trigger =
- LIO_INTRMOD_RXMAXCNT_TRIGGER;
-
- if (intr_coal->rx_coalesce_usecs_high)
- intrmod_cfg->rx_maxtmr_trigger =
- intr_coal->rx_coalesce_usecs_high;
- else
- intrmod_cfg->rx_maxtmr_trigger =
- LIO_INTRMOD_RXMAXTMR_TRIGGER;
-
- if (intr_coal->rx_coalesce_usecs_low)
- intrmod_cfg->rx_mintmr_trigger =
- intr_coal->rx_coalesce_usecs_low;
- else
- intrmod_cfg->rx_mintmr_trigger =
- LIO_INTRMOD_RXMINTMR_TRIGGER;
-
- if (intr_coal->rx_max_coalesced_frames_low)
- intrmod_cfg->rx_mincnt_trigger =
- intr_coal->rx_max_coalesced_frames_low;
- else
- intrmod_cfg->rx_mincnt_trigger =
- LIO_INTRMOD_RXMINCNT_TRIGGER;
+ if (intrmod_cfg->rx_enable) {
+ intrmod_cfg->rx_maxcnt_trigger =
+ intr_coal->rx_max_coalesced_frames_high;
+ intrmod_cfg->rx_maxtmr_trigger =
+ intr_coal->rx_coalesce_usecs_high;
+ intrmod_cfg->rx_mintmr_trigger =
+ intr_coal->rx_coalesce_usecs_low;
+ intrmod_cfg->rx_mincnt_trigger =
+ intr_coal->rx_max_coalesced_frames_low;
}
- if (oct->intrmod.tx_enable) {
- if (intr_coal->tx_max_coalesced_frames_high)
- intrmod_cfg->tx_maxcnt_trigger =
- intr_coal->tx_max_coalesced_frames_high;
- else
- intrmod_cfg->tx_maxcnt_trigger =
- LIO_INTRMOD_TXMAXCNT_TRIGGER;
- if (intr_coal->tx_max_coalesced_frames_low)
- intrmod_cfg->tx_mincnt_trigger =
- intr_coal->tx_max_coalesced_frames_low;
- else
- intrmod_cfg->tx_mincnt_trigger =
- LIO_INTRMOD_TXMINCNT_TRIGGER;
+ if (intrmod_cfg->tx_enable) {
+ intrmod_cfg->tx_maxcnt_trigger =
+ intr_coal->tx_max_coalesced_frames_high;
+ intrmod_cfg->tx_mincnt_trigger =
+ intr_coal->tx_max_coalesced_frames_low;
}
ret = octnet_set_intrmod_cfg(lio, intrmod_cfg);
@@ -1666,7 +1741,9 @@
}
static int
-oct_cfg_rx_intrcnt(struct lio *lio, struct ethtool_coalesce *intr_coal)
+oct_cfg_rx_intrcnt(struct lio *lio,
+ struct oct_intrmod_cfg *intrmod,
+ struct ethtool_coalesce *intr_coal)
{
struct octeon_device *oct = lio->oct_dev;
u32 rx_max_coalesced_frames;
@@ -1692,7 +1769,7 @@
int q_no;
if (!intr_coal->rx_max_coalesced_frames)
- rx_max_coalesced_frames = oct->intrmod.rx_frames;
+ rx_max_coalesced_frames = intrmod->rx_frames;
else
rx_max_coalesced_frames =
intr_coal->rx_max_coalesced_frames;
@@ -1703,17 +1780,18 @@
(octeon_read_csr64(
oct, CN23XX_SLI_OQ_PKT_INT_LEVELS(q_no)) &
(0x3fffff00000000UL)) |
- rx_max_coalesced_frames);
+ (rx_max_coalesced_frames - 1));
/*consider setting resend bit*/
}
- oct->intrmod.rx_frames = rx_max_coalesced_frames;
+ intrmod->rx_frames = rx_max_coalesced_frames;
+ oct->rx_max_coalesced_frames = rx_max_coalesced_frames;
break;
}
case OCTEON_CN23XX_VF_VID: {
int q_no;
if (!intr_coal->rx_max_coalesced_frames)
- rx_max_coalesced_frames = oct->intrmod.rx_frames;
+ rx_max_coalesced_frames = intrmod->rx_frames;
else
rx_max_coalesced_frames =
intr_coal->rx_max_coalesced_frames;
@@ -1724,9 +1802,10 @@
oct, CN23XX_VF_SLI_OQ_PKT_INT_LEVELS(q_no)) &
(0x3fffff00000000UL)) |
rx_max_coalesced_frames);
- /* consider writing to resend bit here */
+ /*consider writing to resend bit here*/
}
- oct->intrmod.rx_frames = rx_max_coalesced_frames;
+ intrmod->rx_frames = rx_max_coalesced_frames;
+ oct->rx_max_coalesced_frames = rx_max_coalesced_frames;
break;
}
default:
@@ -1736,6 +1815,7 @@
}
static int oct_cfg_rx_intrtime(struct lio *lio,
+ struct oct_intrmod_cfg *intrmod,
struct ethtool_coalesce *intr_coal)
{
struct octeon_device *oct = lio->oct_dev;
@@ -1766,7 +1846,7 @@
int q_no;
if (!intr_coal->rx_coalesce_usecs)
- rx_coalesce_usecs = oct->intrmod.rx_usecs;
+ rx_coalesce_usecs = intrmod->rx_usecs;
else
rx_coalesce_usecs = intr_coal->rx_coalesce_usecs;
time_threshold =
@@ -1775,11 +1855,12 @@
q_no += oct->sriov_info.pf_srn;
octeon_write_csr64(oct,
CN23XX_SLI_OQ_PKT_INT_LEVELS(q_no),
- (oct->intrmod.rx_frames |
- (time_threshold << 32)));
+ (intrmod->rx_frames |
+ ((u64)time_threshold << 32)));
/*consider writing to resend bit here*/
}
- oct->intrmod.rx_usecs = rx_coalesce_usecs;
+ intrmod->rx_usecs = rx_coalesce_usecs;
+ oct->rx_coalesce_usecs = rx_coalesce_usecs;
break;
}
case OCTEON_CN23XX_VF_VID: {
@@ -1787,7 +1868,7 @@
int q_no;
if (!intr_coal->rx_coalesce_usecs)
- rx_coalesce_usecs = oct->intrmod.rx_usecs;
+ rx_coalesce_usecs = intrmod->rx_usecs;
else
rx_coalesce_usecs = intr_coal->rx_coalesce_usecs;
@@ -1796,11 +1877,12 @@
for (q_no = 0; q_no < oct->num_oqs; q_no++) {
octeon_write_csr64(
oct, CN23XX_VF_SLI_OQ_PKT_INT_LEVELS(q_no),
- (oct->intrmod.rx_frames |
- (time_threshold << 32)));
- /* consider setting resend bit */
+ (intrmod->rx_frames |
+ ((u64)time_threshold << 32)));
+ /*consider setting resend bit*/
}
- oct->intrmod.rx_usecs = rx_coalesce_usecs;
+ intrmod->rx_usecs = rx_coalesce_usecs;
+ oct->rx_coalesce_usecs = rx_coalesce_usecs;
break;
}
default:
@@ -1811,8 +1893,9 @@
}
static int
-oct_cfg_tx_intrcnt(struct lio *lio, struct ethtool_coalesce *intr_coal
- __attribute__((unused)))
+oct_cfg_tx_intrcnt(struct lio *lio,
+ struct oct_intrmod_cfg *intrmod,
+ struct ethtool_coalesce *intr_coal)
{
struct octeon_device *oct = lio->oct_dev;
u32 iq_intr_pkt;
@@ -1839,12 +1922,13 @@
val = readq(inst_cnt_reg);
/*clear wmark and count.dont want to write count back*/
val = (val & 0xFFFF000000000000ULL) |
- ((u64)iq_intr_pkt
+ ((u64)(iq_intr_pkt - 1)
<< CN23XX_PKT_IN_DONE_WMARK_BIT_POS);
writeq(val, inst_cnt_reg);
/*consider setting resend bit*/
}
- oct->intrmod.tx_frames = iq_intr_pkt;
+ intrmod->tx_frames = iq_intr_pkt;
+ oct->tx_max_coalesced_frames = iq_intr_pkt;
break;
}
default:
@@ -1859,6 +1943,7 @@
struct lio *lio = GET_LIO(netdev);
int ret;
struct octeon_device *oct = lio->oct_dev;
+ struct oct_intrmod_cfg intrmod = {0};
u32 j, q_no;
int db_max, db_min;
@@ -1877,8 +1962,8 @@
} else {
dev_err(&oct->pci_dev->dev,
"LIQUIDIO: Invalid tx-frames:%d. Range is min:%d max:%d\n",
- intr_coal->tx_max_coalesced_frames, db_min,
- db_max);
+ intr_coal->tx_max_coalesced_frames,
+ db_min, db_max);
return -EINVAL;
}
break;
@@ -1889,24 +1974,36 @@
return -EINVAL;
}
- oct->intrmod.rx_enable = intr_coal->use_adaptive_rx_coalesce ? 1 : 0;
- oct->intrmod.tx_enable = intr_coal->use_adaptive_tx_coalesce ? 1 : 0;
+ intrmod.rx_enable = intr_coal->use_adaptive_rx_coalesce ? 1 : 0;
+ intrmod.tx_enable = intr_coal->use_adaptive_tx_coalesce ? 1 : 0;
+ intrmod.rx_frames = CFG_GET_OQ_INTR_PKT(octeon_get_conf(oct));
+ intrmod.rx_usecs = CFG_GET_OQ_INTR_TIME(octeon_get_conf(oct));
+ intrmod.tx_frames = CFG_GET_IQ_INTR_PKT(octeon_get_conf(oct));
- ret = oct_cfg_adaptive_intr(lio, intr_coal);
+ ret = oct_cfg_adaptive_intr(lio, &intrmod, intr_coal);
if (!intr_coal->use_adaptive_rx_coalesce) {
- ret = oct_cfg_rx_intrtime(lio, intr_coal);
+ ret = oct_cfg_rx_intrtime(lio, &intrmod, intr_coal);
if (ret)
goto ret_intrmod;
- ret = oct_cfg_rx_intrcnt(lio, intr_coal);
+ ret = oct_cfg_rx_intrcnt(lio, &intrmod, intr_coal);
if (ret)
goto ret_intrmod;
+ } else {
+ oct->rx_coalesce_usecs =
+ CFG_GET_OQ_INTR_TIME(octeon_get_conf(oct));
+ oct->rx_max_coalesced_frames =
+ CFG_GET_OQ_INTR_PKT(octeon_get_conf(oct));
}
+
if (!intr_coal->use_adaptive_tx_coalesce) {
- ret = oct_cfg_tx_intrcnt(lio, intr_coal);
+ ret = oct_cfg_tx_intrcnt(lio, &intrmod, intr_coal);
if (ret)
goto ret_intrmod;
+ } else {
+ oct->tx_max_coalesced_frames =
+ CFG_GET_IQ_INTR_PKT(octeon_get_conf(oct));
}
return 0;
diff --git a/drivers/net/ethernet/cavium/liquidio/lio_main.c b/drivers/net/ethernet/cavium/liquidio/lio_main.c
index 92f46b1..927617c 100644
--- a/drivers/net/ethernet/cavium/liquidio/lio_main.c
+++ b/drivers/net/ethernet/cavium/liquidio/lio_main.c
@@ -16,6 +16,7 @@
* NONINFRINGEMENT. See the GNU General Public License for more details.
***********************************************************************/
#include <linux/module.h>
+#include <linux/interrupt.h>
#include <linux/pci.h>
#include <linux/firmware.h>
#include <net/vxlan.h>
@@ -60,12 +61,6 @@
static int ptp_enable = 1;
-/* Bit mask values for lio->ifstate */
-#define LIO_IFSTATE_DROQ_OPS 0x01
-#define LIO_IFSTATE_REGISTERED 0x02
-#define LIO_IFSTATE_RUNNING 0x04
-#define LIO_IFSTATE_RX_TIMESTAMP_ENABLED 0x08
-
/* Polling interval for determining when NIC application is alive */
#define LIQUIDIO_STARTER_POLL_INTERVAL_MS 100
@@ -178,6 +173,8 @@
static void liquidio_remove(struct pci_dev *pdev);
static int liquidio_probe(struct pci_dev *pdev,
const struct pci_device_id *ent);
+static int liquidio_set_vf_link_state(struct net_device *netdev, int vfidx,
+ int linkstate);
static struct handshake handshake[MAX_OCTEON_DEVICES];
static struct completion first_stage;
@@ -531,36 +528,6 @@
}
/**
- * \brief check interface state
- * @param lio per-network private data
- * @param state_flag flag state to check
- */
-static inline int ifstate_check(struct lio *lio, int state_flag)
-{
- return atomic_read(&lio->ifstate) & state_flag;
-}
-
-/**
- * \brief set interface state
- * @param lio per-network private data
- * @param state_flag flag state to set
- */
-static inline void ifstate_set(struct lio *lio, int state_flag)
-{
- atomic_set(&lio->ifstate, (atomic_read(&lio->ifstate) | state_flag));
-}
-
-/**
- * \brief clear interface state
- * @param lio per-network private data
- * @param state_flag flag state to clear
- */
-static inline void ifstate_reset(struct lio *lio, int state_flag)
-{
- atomic_set(&lio->ifstate, (atomic_read(&lio->ifstate) & ~(state_flag)));
-}
-
-/**
* \brief Stop Tx queues
* @param netdev network device
*/
@@ -748,7 +715,8 @@
kfree(g);
} while (g);
- if (lio->glists_virt_base && lio->glists_virt_base[i]) {
+ if (lio->glists_virt_base && lio->glists_virt_base[i] &&
+ lio->glists_dma_base && lio->glists_dma_base[i]) {
lio_dma_free(lio->oct_dev,
lio->glist_entry_size * lio->tx_qsize,
lio->glists_virt_base[i],
@@ -805,7 +773,7 @@
}
for (i = 0; i < num_iqs; i++) {
- int numa_node = cpu_to_node(i % num_online_cpus());
+ int numa_node = dev_to_node(&oct->pci_dev->dev);
spin_lock_init(&lio->glist_lock[i]);
@@ -967,14 +935,13 @@
INCR_INSTRQUEUE_PKT_COUNT(lio->oct_dev, iq_num,
tx_restart, 1);
netif_wake_subqueue(netdev, iq->q_index);
- } else {
- if (!octnet_iq_is_full(oct, lio->txq)) {
- INCR_INSTRQUEUE_PKT_COUNT(lio->oct_dev,
- lio->txq,
- tx_restart, 1);
- wake_q(netdev, lio->txq);
- }
}
+ } else if (netif_queue_stopped(netdev) &&
+ lio->linfo.link.s.link_up &&
+ (!octnet_iq_is_full(oct, lio->txq))) {
+ INCR_INSTRQUEUE_PKT_COUNT(lio->oct_dev,
+ lio->txq, tx_restart, 1);
+ netif_wake_queue(netdev);
}
}
@@ -1084,16 +1051,35 @@
int i;
int num_ioq_vectors;
int num_alloc_ioq_vectors;
+ char *queue_irq_names = NULL;
+ char *aux_irq_name = NULL;
if (OCTEON_CN23XX_PF(oct) && oct->msix_on) {
oct->num_msix_irqs = oct->sriov_info.num_pf_rings;
/* one non ioq interrupt for handling sli_mac_pf_int_sum */
oct->num_msix_irqs += 1;
+ /* allocate storage for the names assigned to each irq */
+ oct->irq_name_storage =
+ kcalloc((MAX_IOQ_INTERRUPTS_PER_PF + 1), INTRNAMSIZ,
+ GFP_KERNEL);
+ if (!oct->irq_name_storage) {
+ dev_err(&oct->pci_dev->dev, "Irq name storage alloc failed...\n");
+ return -ENOMEM;
+ }
+
+ queue_irq_names = oct->irq_name_storage;
+ aux_irq_name = &queue_irq_names
+ [IRQ_NAME_OFF(MAX_IOQ_INTERRUPTS_PER_PF)];
+
oct->msix_entries = kcalloc(
oct->num_msix_irqs, sizeof(struct msix_entry), GFP_KERNEL);
- if (!oct->msix_entries)
- return 1;
+ if (!oct->msix_entries) {
+ dev_err(&oct->pci_dev->dev, "Memory Alloc failed...\n");
+ kfree(oct->irq_name_storage);
+ oct->irq_name_storage = NULL;
+ return -ENOMEM;
+ }
msix_entries = (struct msix_entry *)oct->msix_entries;
/*Assumption is that pf msix vectors start from pf srn to pf to
@@ -1111,7 +1097,9 @@
dev_err(&oct->pci_dev->dev, "unable to Allocate MSI-X interrupts\n");
kfree(oct->msix_entries);
oct->msix_entries = NULL;
- return 1;
+ kfree(oct->irq_name_storage);
+ oct->irq_name_storage = NULL;
+ return num_alloc_ioq_vectors;
}
dev_dbg(&oct->pci_dev->dev, "OCTEON: Enough MSI-X interrupts are allocated...\n");
@@ -1119,9 +1107,12 @@
/** For PF, there is one non-ioq interrupt handler */
num_ioq_vectors -= 1;
+
+ snprintf(aux_irq_name, INTRNAMSIZ,
+ "LiquidIO%u-pf%u-aux", oct->octeon_id, oct->pf_num);
irqret = request_irq(msix_entries[num_ioq_vectors].vector,
- liquidio_legacy_intr_handler, 0, "octeon",
- oct);
+ liquidio_legacy_intr_handler, 0,
+ aux_irq_name, oct);
if (irqret) {
dev_err(&oct->pci_dev->dev,
"OCTEON: Request_irq failed for MSIX interrupt Error: %d\n",
@@ -1129,13 +1120,20 @@
pci_disable_msix(oct->pci_dev);
kfree(oct->msix_entries);
oct->msix_entries = NULL;
- return 1;
+ kfree(oct->irq_name_storage);
+ oct->irq_name_storage = NULL;
+ return irqret;
}
for (i = 0; i < num_ioq_vectors; i++) {
+ snprintf(&queue_irq_names[IRQ_NAME_OFF(i)], INTRNAMSIZ,
+ "LiquidIO%u-pf%u-rxtx-%u",
+ oct->octeon_id, oct->pf_num, i);
+
irqret = request_irq(msix_entries[i].vector,
liquidio_msix_intr_handler, 0,
- "octeon", &oct->ioq_vector[i]);
+ &queue_irq_names[IRQ_NAME_OFF(i)],
+ &oct->ioq_vector[i]);
if (irqret) {
dev_err(&oct->pci_dev->dev,
"OCTEON: Request_irq failed for MSIX interrupt Error: %d\n",
@@ -1155,7 +1153,9 @@
pci_disable_msix(oct->pci_dev);
kfree(oct->msix_entries);
oct->msix_entries = NULL;
- return 1;
+ kfree(oct->irq_name_storage);
+ oct->irq_name_storage = NULL;
+ return irqret;
}
oct->ioq_vector[i].vector = msix_entries[i].vector;
/* assign the cpu mask for this msix interrupt vector */
@@ -1173,111 +1173,150 @@
else
oct->flags |= LIO_FLAG_MSI_ENABLED;
+ /* allocate storage for the names assigned to the irq */
+ oct->irq_name_storage = kcalloc(1, INTRNAMSIZ, GFP_KERNEL);
+ if (!oct->irq_name_storage)
+ return -ENOMEM;
+
+ queue_irq_names = oct->irq_name_storage;
+
+ snprintf(&queue_irq_names[IRQ_NAME_OFF(0)], INTRNAMSIZ,
+ "LiquidIO%u-pf%u-rxtx-%u",
+ oct->octeon_id, oct->pf_num, 0);
+
irqret = request_irq(oct->pci_dev->irq,
- liquidio_legacy_intr_handler, IRQF_SHARED,
- "octeon", oct);
+ liquidio_legacy_intr_handler,
+ IRQF_SHARED,
+ &queue_irq_names[IRQ_NAME_OFF(0)], oct);
if (irqret) {
if (oct->flags & LIO_FLAG_MSI_ENABLED)
pci_disable_msi(oct->pci_dev);
dev_err(&oct->pci_dev->dev, "Request IRQ failed with code: %d\n",
irqret);
- return 1;
+ kfree(oct->irq_name_storage);
+ oct->irq_name_storage = NULL;
+ return irqret;
}
}
return 0;
}
+static struct octeon_device *get_other_octeon_device(struct octeon_device *oct)
+{
+ struct octeon_device *other_oct;
+
+ other_oct = lio_get_device(oct->octeon_id + 1);
+
+ if (other_oct && other_oct->pci_dev) {
+ int oct_busnum, other_oct_busnum;
+
+ oct_busnum = oct->pci_dev->bus->number;
+ other_oct_busnum = other_oct->pci_dev->bus->number;
+
+ if (oct_busnum == other_oct_busnum) {
+ int oct_slot, other_oct_slot;
+
+ oct_slot = PCI_SLOT(oct->pci_dev->devfn);
+ other_oct_slot = PCI_SLOT(other_oct->pci_dev->devfn);
+
+ if (oct_slot == other_oct_slot)
+ return other_oct;
+ }
+ }
+
+ return NULL;
+}
+
+static void disable_all_vf_links(struct octeon_device *oct)
+{
+ struct net_device *netdev;
+ int max_vfs, vf, i;
+
+ if (!oct)
+ return;
+
+ max_vfs = oct->sriov_info.max_vfs;
+
+ for (i = 0; i < oct->ifcount; i++) {
+ netdev = oct->props[i].netdev;
+ if (!netdev)
+ continue;
+
+ for (vf = 0; vf < max_vfs; vf++)
+ liquidio_set_vf_link_state(netdev, vf,
+ IFLA_VF_LINK_STATE_DISABLE);
+ }
+}
+
static int liquidio_watchdog(void *param)
{
- u64 wdog;
- u16 mask_of_stuck_cores = 0;
- u16 mask_of_crashed_cores = 0;
- int core_num;
- u8 core_is_stuck[LIO_MAX_CORES];
- u8 core_crashed[LIO_MAX_CORES];
+ bool err_msg_was_printed[LIO_MAX_CORES];
+ u16 mask_of_crashed_or_stuck_cores = 0;
+ bool all_vf_links_are_disabled = false;
struct octeon_device *oct = param;
+ struct octeon_device *other_oct;
+#ifdef CONFIG_MODULE_UNLOAD
+ long refcount, vfs_referencing_pf;
+ u64 vfs_mask1, vfs_mask2;
+#endif
+ int core;
- memset(core_is_stuck, 0, sizeof(core_is_stuck));
- memset(core_crashed, 0, sizeof(core_crashed));
+ memset(err_msg_was_printed, 0, sizeof(err_msg_was_printed));
while (!kthread_should_stop()) {
- mask_of_crashed_cores =
+ /* sleep for a couple of seconds so that we don't hog the CPU */
+ set_current_state(TASK_INTERRUPTIBLE);
+ schedule_timeout(msecs_to_jiffies(2000));
+
+ mask_of_crashed_or_stuck_cores =
(u16)octeon_read_csr64(oct, CN23XX_SLI_SCRATCH2);
- for (core_num = 0; core_num < LIO_MAX_CORES; core_num++) {
- if (!core_is_stuck[core_num]) {
- wdog = lio_pci_readq(oct, CIU3_WDOG(core_num));
+ if (!mask_of_crashed_or_stuck_cores)
+ continue;
- /* look at watchdog state field */
- wdog &= CIU3_WDOG_MASK;
- if (wdog) {
- /* this watchdog timer has expired */
- core_is_stuck[core_num] =
- LIO_MONITOR_WDOG_EXPIRE;
- mask_of_stuck_cores |= (1 << core_num);
- }
- }
+ WRITE_ONCE(oct->cores_crashed, true);
+ other_oct = get_other_octeon_device(oct);
+ if (other_oct)
+ WRITE_ONCE(other_oct->cores_crashed, true);
- if (!core_crashed[core_num])
- core_crashed[core_num] =
- (mask_of_crashed_cores >> core_num) & 1;
- }
+ for (core = 0; core < LIO_MAX_CORES; core++) {
+ bool core_crashed_or_got_stuck;
- if (mask_of_stuck_cores) {
- for (core_num = 0; core_num < LIO_MAX_CORES;
- core_num++) {
- if (core_is_stuck[core_num] == 1) {
- dev_err(&oct->pci_dev->dev,
- "ERROR: Octeon core %d is stuck!\n",
- core_num);
- /* 2 means we have printk'd an error
- * so no need to repeat the same printk
- */
- core_is_stuck[core_num] =
- LIO_MONITOR_CORE_STUCK_MSGD;
- }
+ core_crashed_or_got_stuck =
+ (mask_of_crashed_or_stuck_cores
+ >> core) & 1;
+
+ if (core_crashed_or_got_stuck &&
+ !err_msg_was_printed[core]) {
+ dev_err(&oct->pci_dev->dev,
+ "ERROR: Octeon core %d crashed or got stuck! See oct-fwdump for details.\n",
+ core);
+ err_msg_was_printed[core] = true;
}
}
- if (mask_of_crashed_cores) {
- for (core_num = 0; core_num < LIO_MAX_CORES;
- core_num++) {
- if (core_crashed[core_num] == 1) {
- dev_err(&oct->pci_dev->dev,
- "ERROR: Octeon core %d crashed! See oct-fwdump for details.\n",
- core_num);
- /* 2 means we have printk'd an error
- * so no need to repeat the same printk
- */
- core_crashed[core_num] =
- LIO_MONITOR_CORE_STUCK_MSGD;
- }
- }
- }
+ if (all_vf_links_are_disabled)
+ continue;
+
+ disable_all_vf_links(oct);
+ disable_all_vf_links(other_oct);
+ all_vf_links_are_disabled = true;
+
#ifdef CONFIG_MODULE_UNLOAD
- if (mask_of_stuck_cores || mask_of_crashed_cores) {
- /* make module refcount=0 so that rmmod will work */
- long refcount;
+ vfs_mask1 = READ_ONCE(oct->sriov_info.vf_drv_loaded_mask);
+ vfs_mask2 = READ_ONCE(other_oct->sriov_info.vf_drv_loaded_mask);
- refcount = module_refcount(THIS_MODULE);
+ vfs_referencing_pf = hweight64(vfs_mask1);
+ vfs_referencing_pf += hweight64(vfs_mask2);
- while (refcount > 0) {
+ refcount = module_refcount(THIS_MODULE);
+ if (refcount >= vfs_referencing_pf) {
+ while (vfs_referencing_pf) {
module_put(THIS_MODULE);
- refcount = module_refcount(THIS_MODULE);
- }
-
- /* compensate for and withstand an unlikely (but still
- * possible) race condition
- */
- while (refcount < 0) {
- try_module_get(THIS_MODULE);
- refcount = module_refcount(THIS_MODULE);
+ vfs_referencing_pf--;
}
}
#endif
- /* sleep for two seconds */
- set_current_state(TASK_INTERRUPTIBLE);
- schedule_timeout(2 * HZ);
}
return 0;
@@ -1369,6 +1408,12 @@
return 0;
}
+static bool fw_type_is_none(void)
+{
+ return strncmp(fw_type, LIO_FW_NAME_TYPE_NONE,
+ sizeof(LIO_FW_NAME_TYPE_NONE)) == 0;
+}
+
/**
*\brief Destroy resources associated with octeon device
* @param pdev PCI device structure
@@ -1449,6 +1494,9 @@
pci_disable_msi(oct->pci_dev);
}
+ kfree(oct->irq_name_storage);
+ oct->irq_name_storage = NULL;
+
/* fallthrough */
case OCT_DEV_MSIX_ALLOC_VECTOR_DONE:
if (OCTEON_CN23XX_PF(oct))
@@ -1508,9 +1556,12 @@
/* fallthrough */
case OCT_DEV_PCI_MAP_DONE:
- /* Soft reset the octeon device before exiting */
- if ((!OCTEON_CN23XX_PF(oct)) || !oct->octeon_id)
- oct->fn_list.soft_reset(oct);
+ if (!fw_type_is_none()) {
+ /* Soft reset the octeon device before exiting */
+ if (!OCTEON_CN23XX_PF(oct) ||
+ (OCTEON_CN23XX_PF(oct) && !oct->octeon_id))
+ oct->fn_list.soft_reset(oct);
+ }
octeon_unmap_pci_barx(oct, 0);
octeon_unmap_pci_barx(oct, 1);
@@ -1643,6 +1694,15 @@
if (atomic_read(&lio->ifstate) & LIO_IFSTATE_RUNNING)
liquidio_stop(netdev);
+ if (fw_type_is_none()) {
+ struct octnic_ctrl_pkt nctrl;
+
+ memset(&nctrl, 0, sizeof(struct octnic_ctrl_pkt));
+ nctrl.ncmd.s.cmd = OCTNET_CMD_RESET_PF;
+ nctrl.iq_no = lio->linfo.txpciq[0].s.q_no;
+ octnet_send_nic_ctrl_pkt(oct, &nctrl);
+ }
+
if (oct->props[lio->ifidx].napi_enabled == 1) {
list_for_each_entry_safe(napi, n, &netdev->napi_list, dev_list)
napi_disable(napi);
@@ -1658,6 +1718,8 @@
cleanup_link_status_change_wq(netdev);
+ cleanup_rx_oom_poll_fn(netdev);
+
delete_glists(lio);
free_netdev(netdev);
@@ -2126,8 +2188,7 @@
char fw_name[LIO_MAX_FW_FILENAME_LEN];
char *tmp_fw_type;
- if (strncmp(fw_type, LIO_FW_NAME_TYPE_NONE,
- sizeof(LIO_FW_NAME_TYPE_NONE)) == 0) {
+ if (fw_type_is_none()) {
dev_info(&oct->pci_dev->dev, "Skipping firmware load\n");
return ret;
}
@@ -2211,8 +2272,8 @@
oct = lio_get_device(ctx->octeon_id);
if (resp->status)
- dev_err(&oct->pci_dev->dev, "nic if cfg instruction failed. Status: %llx\n",
- CVM_CAST64(resp->status));
+ dev_err(&oct->pci_dev->dev, "nic if cfg instruction failed. Status: 0x%llx (0x%08x)\n",
+ CVM_CAST64(resp->status), status);
WRITE_ONCE(ctx->cond, 1);
snprintf(oct->fw_info.liquidio_firmware_version, 32, "%s",
@@ -2437,8 +2498,11 @@
/* Flush the instruction queue */
iq = oct->instr_queue[iq_no];
if (iq) {
- /* Process iq buffers with in the budget limits */
- tx_done = octeon_flush_iq(oct, iq, budget);
+ if (atomic_read(&iq->instr_pending))
+ /* Process iq buffers with in the budget limits */
+ tx_done = octeon_flush_iq(oct, iq, budget);
+ else
+ tx_done = 1;
/* Update iq read-index rather than waiting for next interrupt.
* Return back if tx_done is false.
*/
@@ -2555,6 +2619,15 @@
__func__);
return 1;
}
+
+ if (octeon_dev->ioq_vector) {
+ struct octeon_ioq_vector *ioq_vector;
+
+ ioq_vector = &octeon_dev->ioq_vector[q];
+ netif_set_xps_queue(netdev,
+ &ioq_vector->affinity_mask,
+ ioq_vector->iq_index);
+ }
}
return 0;
@@ -3426,6 +3499,8 @@
struct octnic_ctrl_pkt nctrl;
int ret = 0;
+ memset(&nctrl, 0, sizeof(struct octnic_ctrl_pkt));
+
nctrl.ncmd.u64 = 0;
nctrl.ncmd.s.cmd = command;
nctrl.ncmd.s.param1 = rx_cmd;
@@ -3459,6 +3534,8 @@
struct octnic_ctrl_pkt nctrl;
int ret = 0;
+ memset(&nctrl, 0, sizeof(struct octnic_ctrl_pkt));
+
nctrl.ncmd.u64 = 0;
nctrl.ncmd.s.cmd = command;
nctrl.ncmd.s.more = vxlan_cmd_bit;
@@ -3596,7 +3673,8 @@
nctrl.ncmd.s.param2 = (is_admin_assigned ? 1 : 0);
nctrl.ncmd.s.more = 1;
nctrl.iq_no = lio->linfo.txpciq[0].s.q_no;
- nctrl.cb_fn = 0;
+ nctrl.netpndev = (u64)netdev;
+ nctrl.cb_fn = liquidio_link_ctrl_cmd_completion;
nctrl.wait_time = LIO_CMD_WAIT_TM;
nctrl.udd[0] = 0;
@@ -4122,6 +4200,9 @@
if (setup_link_status_change_wq(netdev))
goto setup_nic_dev_fail;
+ if (setup_rx_oom_poll_fn(netdev))
+ goto setup_nic_dev_fail;
+
/* Register the network device with the OS */
if (register_netdev(netdev)) {
dev_err(&octeon_dev->pci_dev->dev, "Device registration failed\n");
@@ -4271,7 +4352,6 @@
*/
static int liquidio_init_nic_module(struct octeon_device *oct)
{
- struct oct_intrmod_cfg *intrmod_cfg;
int i, retval = 0;
int num_nic_ports = CFG_GET_NUM_NIC_PORTS(octeon_get_conf(oct));
@@ -4296,22 +4376,6 @@
liquidio_ptp_init(oct);
- /* Initialize interrupt moderation params */
- intrmod_cfg = &((struct octeon_device *)oct)->intrmod;
- intrmod_cfg->rx_enable = 1;
- intrmod_cfg->check_intrvl = LIO_INTRMOD_CHECK_INTERVAL;
- intrmod_cfg->maxpkt_ratethr = LIO_INTRMOD_MAXPKT_RATETHR;
- intrmod_cfg->minpkt_ratethr = LIO_INTRMOD_MINPKT_RATETHR;
- intrmod_cfg->rx_maxcnt_trigger = LIO_INTRMOD_RXMAXCNT_TRIGGER;
- intrmod_cfg->rx_maxtmr_trigger = LIO_INTRMOD_RXMAXTMR_TRIGGER;
- intrmod_cfg->rx_mintmr_trigger = LIO_INTRMOD_RXMINTMR_TRIGGER;
- intrmod_cfg->rx_mincnt_trigger = LIO_INTRMOD_RXMINCNT_TRIGGER;
- intrmod_cfg->tx_enable = 1;
- intrmod_cfg->tx_maxcnt_trigger = LIO_INTRMOD_TXMAXCNT_TRIGGER;
- intrmod_cfg->tx_mincnt_trigger = LIO_INTRMOD_TXMINCNT_TRIGGER;
- intrmod_cfg->rx_frames = CFG_GET_OQ_INTR_PKT(octeon_get_conf(oct));
- intrmod_cfg->rx_usecs = CFG_GET_OQ_INTR_TIME(octeon_get_conf(oct));
- intrmod_cfg->tx_frames = CFG_GET_IQ_INTR_PKT(octeon_get_conf(oct));
dev_dbg(&oct->pci_dev->dev, "Network interfaces ready\n");
return retval;
@@ -4373,6 +4437,7 @@
struct octeon_device *oct = (struct octeon_device *)buf;
struct octeon_recv_pkt *recv_pkt = recv_info->recv_pkt;
int i, notice, vf_idx;
+ bool cores_crashed;
u64 *data, vf_num;
notice = recv_pkt->rh.r.ossp;
@@ -4383,19 +4448,23 @@
octeon_swap_8B_data(&vf_num, 1);
vf_idx = (int)vf_num - 1;
+ cores_crashed = READ_ONCE(oct->cores_crashed);
+
if (notice == VF_DRV_LOADED) {
if (!(oct->sriov_info.vf_drv_loaded_mask & BIT_ULL(vf_idx))) {
oct->sriov_info.vf_drv_loaded_mask |= BIT_ULL(vf_idx);
dev_info(&oct->pci_dev->dev,
"driver for VF%d was loaded\n", vf_idx);
- try_module_get(THIS_MODULE);
+ if (!cores_crashed)
+ try_module_get(THIS_MODULE);
}
} else if (notice == VF_DRV_REMOVED) {
if (oct->sriov_info.vf_drv_loaded_mask & BIT_ULL(vf_idx)) {
oct->sriov_info.vf_drv_loaded_mask &= ~BIT_ULL(vf_idx);
dev_info(&oct->pci_dev->dev,
"driver for VF%d was removed\n", vf_idx);
- module_put(THIS_MODULE);
+ if (!cores_crashed)
+ module_put(THIS_MODULE);
}
} else if (notice == VF_DRV_MACADDR_CHANGED) {
u8 *b = (u8 *)&data[1];
@@ -4447,14 +4516,16 @@
if (OCTEON_CN23XX_PF(octeon_dev)) {
if (!cn23xx_fw_loaded(octeon_dev)) {
fw_loaded = 0;
- /* Do a soft reset of the Octeon device. */
- if (octeon_dev->fn_list.soft_reset(octeon_dev))
- return 1;
- /* things might have changed */
- if (!cn23xx_fw_loaded(octeon_dev))
- fw_loaded = 0;
- else
- fw_loaded = 1;
+ if (!fw_type_is_none()) {
+ /* Do a soft reset of the Octeon device. */
+ if (octeon_dev->fn_list.soft_reset(octeon_dev))
+ return 1;
+ /* things might have changed */
+ if (!cn23xx_fw_loaded(octeon_dev))
+ fw_loaded = 0;
+ else
+ fw_loaded = 1;
+ }
} else {
fw_loaded = 1;
}
diff --git a/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c b/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c
index 7b83be4..34c7782 100644
--- a/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c
+++ b/drivers/net/ethernet/cavium/liquidio/lio_vf_main.c
@@ -16,6 +16,7 @@
* NONINFRINGEMENT. See the GNU General Public License for more details.
***********************************************************************/
#include <linux/module.h>
+#include <linux/interrupt.h>
#include <linux/pci.h>
#include <net/vxlan.h>
#include "liquidio_common.h"
@@ -39,12 +40,6 @@
#define DEFAULT_MSG_ENABLE (NETIF_MSG_DRV | NETIF_MSG_PROBE | NETIF_MSG_LINK)
-/* Bit mask values for lio->ifstate */
-#define LIO_IFSTATE_DROQ_OPS 0x01
-#define LIO_IFSTATE_REGISTERED 0x02
-#define LIO_IFSTATE_RUNNING 0x04
-#define LIO_IFSTATE_RX_TIMESTAMP_ENABLED 0x08
-
struct liquidio_if_cfg_context {
int octeon_id;
@@ -336,36 +331,6 @@
};
/**
- * \brief check interface state
- * @param lio per-network private data
- * @param state_flag flag state to check
- */
-static int ifstate_check(struct lio *lio, int state_flag)
-{
- return atomic_read(&lio->ifstate) & state_flag;
-}
-
-/**
- * \brief set interface state
- * @param lio per-network private data
- * @param state_flag flag state to set
- */
-static void ifstate_set(struct lio *lio, int state_flag)
-{
- atomic_set(&lio->ifstate, (atomic_read(&lio->ifstate) | state_flag));
-}
-
-/**
- * \brief clear interface state
- * @param lio per-network private data
- * @param state_flag flag state to clear
- */
-static void ifstate_reset(struct lio *lio, int state_flag)
-{
- atomic_set(&lio->ifstate, (atomic_read(&lio->ifstate) & ~(state_flag)));
-}
-
-/**
* \brief Stop Tx queues
* @param netdev network device
*/
@@ -506,7 +471,8 @@
kfree(g);
} while (g);
- if (lio->glists_virt_base && lio->glists_virt_base[i]) {
+ if (lio->glists_virt_base && lio->glists_virt_base[i] &&
+ lio->glists_dma_base && lio->glists_dma_base[i]) {
lio_dma_free(lio->oct_dev,
lio->glist_entry_size * lio->tx_qsize,
lio->glists_virt_base[i],
@@ -722,13 +688,12 @@
netif_wake_subqueue(netdev, iq->q_index);
INCR_INSTRQUEUE_PKT_COUNT(lio->oct_dev, iq_num,
tx_restart, 1);
- } else {
- if (!octnet_iq_is_full(oct, lio->txq)) {
- INCR_INSTRQUEUE_PKT_COUNT(
- lio->oct_dev, lio->txq, tx_restart, 1);
- wake_q(netdev, lio->txq);
- }
}
+ } else if (netif_queue_stopped(netdev) && lio->linfo.link.s.link_up &&
+ (!octnet_iq_is_full(oct, lio->txq))) {
+ INCR_INSTRQUEUE_PKT_COUNT(lio->oct_dev,
+ lio->txq, tx_restart, 1);
+ netif_wake_queue(netdev);
}
}
@@ -780,6 +745,7 @@
static int octeon_setup_interrupt(struct octeon_device *oct)
{
struct msix_entry *msix_entries;
+ char *queue_irq_names = NULL;
int num_alloc_ioq_vectors;
int num_ioq_vectors;
int irqret;
@@ -788,10 +754,25 @@
if (oct->msix_on) {
oct->num_msix_irqs = oct->sriov_info.rings_per_vf;
+ /* allocate storage for the names assigned to each irq */
+ oct->irq_name_storage =
+ kcalloc(MAX_IOQ_INTERRUPTS_PER_VF, INTRNAMSIZ,
+ GFP_KERNEL);
+ if (!oct->irq_name_storage) {
+ dev_err(&oct->pci_dev->dev, "Irq name storage alloc failed...\n");
+ return -ENOMEM;
+ }
+
+ queue_irq_names = oct->irq_name_storage;
+
oct->msix_entries = kcalloc(
oct->num_msix_irqs, sizeof(struct msix_entry), GFP_KERNEL);
- if (!oct->msix_entries)
- return 1;
+ if (!oct->msix_entries) {
+ dev_err(&oct->pci_dev->dev, "Memory Alloc failed...\n");
+ kfree(oct->irq_name_storage);
+ oct->irq_name_storage = NULL;
+ return -ENOMEM;
+ }
msix_entries = (struct msix_entry *)oct->msix_entries;
@@ -805,16 +786,23 @@
dev_err(&oct->pci_dev->dev, "unable to Allocate MSI-X interrupts\n");
kfree(oct->msix_entries);
oct->msix_entries = NULL;
- return 1;
+ kfree(oct->irq_name_storage);
+ oct->irq_name_storage = NULL;
+ return num_alloc_ioq_vectors;
}
dev_dbg(&oct->pci_dev->dev, "OCTEON: Enough MSI-X interrupts are allocated...\n");
num_ioq_vectors = oct->num_msix_irqs;
for (i = 0; i < num_ioq_vectors; i++) {
+ snprintf(&queue_irq_names[IRQ_NAME_OFF(i)], INTRNAMSIZ,
+ "LiquidIO%u-vf%u-rxtx-%u",
+ oct->octeon_id, oct->vf_num, i);
+
irqret = request_irq(msix_entries[i].vector,
liquidio_msix_intr_handler, 0,
- "octeon", &oct->ioq_vector[i]);
+ &queue_irq_names[IRQ_NAME_OFF(i)],
+ &oct->ioq_vector[i]);
if (irqret) {
dev_err(&oct->pci_dev->dev,
"OCTEON: Request_irq failed for MSIX interrupt Error: %d\n",
@@ -830,7 +818,9 @@
pci_disable_msix(oct->pci_dev);
kfree(oct->msix_entries);
oct->msix_entries = NULL;
- return 1;
+ kfree(oct->irq_name_storage);
+ oct->irq_name_storage = NULL;
+ return irqret;
}
oct->ioq_vector[i].vector = msix_entries[i].vector;
/* assign the cpu mask for this msix interrupt vector */
@@ -975,6 +965,8 @@
pci_disable_msix(oct->pci_dev);
kfree(oct->msix_entries);
oct->msix_entries = NULL;
+ kfree(oct->irq_name_storage);
+ oct->irq_name_storage = NULL;
}
/* Soft reset the octeon device before exiting */
if (oct->pci_dev->reset_fn)
@@ -1163,6 +1155,8 @@
if (atomic_read(&lio->ifstate) & LIO_IFSTATE_REGISTERED)
unregister_netdev(netdev);
+ cleanup_rx_oom_poll_fn(netdev);
+
cleanup_link_status_change_wq(netdev);
delete_glists(lio);
@@ -1642,8 +1636,12 @@
/* Flush the instruction queue */
iq = oct->instr_queue[iq_no];
if (iq) {
- /* Process iq buffers with in the budget limits */
- tx_done = octeon_flush_iq(oct, iq, budget);
+ if (atomic_read(&iq->instr_pending))
+ /* Process iq buffers with in the budget limits */
+ tx_done = octeon_flush_iq(oct, iq, budget);
+ else
+ tx_done = 1;
+
/* Update iq read-index rather than waiting for next interrupt.
* Return back if tx_done is false.
*/
@@ -2486,6 +2484,8 @@
struct lio *lio = GET_LIO(netdev);
struct octeon_device *oct = lio->oct_dev;
struct octnic_ctrl_pkt nctrl;
+ struct completion compl;
+ u16 response_code;
int ret = 0;
memset(&nctrl, 0, sizeof(struct octnic_ctrl_pkt));
@@ -2497,14 +2497,25 @@
nctrl.wait_time = 100;
nctrl.netpndev = (u64)netdev;
nctrl.cb_fn = liquidio_link_ctrl_cmd_completion;
+ init_completion(&compl);
+ nctrl.completion = &compl;
+ nctrl.response_code = &response_code;
ret = octnet_send_nic_ctrl_pkt(lio->oct_dev, &nctrl);
if (ret < 0) {
dev_err(&oct->pci_dev->dev, "Add VLAN filter failed in core (ret: 0x%x)\n",
ret);
+ return -EIO;
}
- return ret;
+ if (!wait_for_completion_timeout(&compl,
+ msecs_to_jiffies(nctrl.wait_time)))
+ return -EPERM;
+
+ if (READ_ONCE(response_code))
+ return -EPERM;
+
+ return 0;
}
static int
@@ -2549,6 +2560,8 @@
struct octnic_ctrl_pkt nctrl;
int ret = 0;
+ memset(&nctrl, 0, sizeof(struct octnic_ctrl_pkt));
+
nctrl.ncmd.u64 = 0;
nctrl.ncmd.s.cmd = command;
nctrl.ncmd.s.param1 = rx_cmd;
@@ -2581,6 +2594,8 @@
struct octnic_ctrl_pkt nctrl;
int ret = 0;
+ memset(&nctrl, 0, sizeof(struct octnic_ctrl_pkt));
+
nctrl.ncmd.u64 = 0;
nctrl.ncmd.s.cmd = command;
nctrl.ncmd.s.more = vxlan_cmd_bit;
@@ -3003,6 +3018,9 @@
if (setup_link_status_change_wq(netdev))
goto setup_nic_dev_fail;
+ if (setup_rx_oom_poll_fn(netdev))
+ goto setup_nic_dev_fail;
+
/* Register the network device with the OS */
if (register_netdev(netdev)) {
dev_err(&octeon_dev->pci_dev->dev, "Device registration failed\n");
@@ -3057,7 +3075,6 @@
*/
static int liquidio_init_nic_module(struct octeon_device *oct)
{
- struct oct_intrmod_cfg *intrmod_cfg;
int num_nic_ports = 1;
int i, retval = 0;
@@ -3079,22 +3096,6 @@
goto octnet_init_failure;
}
- /* Initialize interrupt moderation params */
- intrmod_cfg = &((struct octeon_device *)oct)->intrmod;
- intrmod_cfg->rx_enable = 1;
- intrmod_cfg->check_intrvl = LIO_INTRMOD_CHECK_INTERVAL;
- intrmod_cfg->maxpkt_ratethr = LIO_INTRMOD_MAXPKT_RATETHR;
- intrmod_cfg->minpkt_ratethr = LIO_INTRMOD_MINPKT_RATETHR;
- intrmod_cfg->rx_maxcnt_trigger = LIO_INTRMOD_RXMAXCNT_TRIGGER;
- intrmod_cfg->rx_maxtmr_trigger = LIO_INTRMOD_RXMAXTMR_TRIGGER;
- intrmod_cfg->rx_mintmr_trigger = LIO_INTRMOD_RXMINTMR_TRIGGER;
- intrmod_cfg->rx_mincnt_trigger = LIO_INTRMOD_RXMINCNT_TRIGGER;
- intrmod_cfg->tx_enable = 1;
- intrmod_cfg->tx_maxcnt_trigger = LIO_INTRMOD_TXMAXCNT_TRIGGER;
- intrmod_cfg->tx_mincnt_trigger = LIO_INTRMOD_TXMINCNT_TRIGGER;
- intrmod_cfg->rx_frames = CFG_GET_OQ_INTR_PKT(octeon_get_conf(oct));
- intrmod_cfg->rx_usecs = CFG_GET_OQ_INTR_TIME(octeon_get_conf(oct));
- intrmod_cfg->tx_frames = CFG_GET_IQ_INTR_PKT(octeon_get_conf(oct));
dev_dbg(&oct->pci_dev->dev, "Network interfaces ready\n");
return retval;
diff --git a/drivers/net/ethernet/cavium/liquidio/liquidio_common.h b/drivers/net/ethernet/cavium/liquidio/liquidio_common.h
index 294c6f3..8ea2323 100644
--- a/drivers/net/ethernet/cavium/liquidio/liquidio_common.h
+++ b/drivers/net/ethernet/cavium/liquidio/liquidio_common.h
@@ -27,7 +27,7 @@
#define LIQUIDIO_PACKAGE ""
#define LIQUIDIO_BASE_MAJOR_VERSION 1
-#define LIQUIDIO_BASE_MINOR_VERSION 4
+#define LIQUIDIO_BASE_MINOR_VERSION 5
#define LIQUIDIO_BASE_MICRO_VERSION 1
#define LIQUIDIO_BASE_VERSION __stringify(LIQUIDIO_BASE_MAJOR_VERSION) "." \
__stringify(LIQUIDIO_BASE_MINOR_VERSION)
@@ -83,6 +83,7 @@
#define OPCODE_NIC_INTRMOD_CFG 0x08
#define OPCODE_NIC_IF_CFG 0x09
#define OPCODE_NIC_VF_DRV_NOTICE 0x0A
+#define OPCODE_NIC_INTRMOD_PARAMS 0x0B
#define VF_DRV_LOADED 1
#define VF_DRV_REMOVED -1
#define VF_DRV_MACADDR_CHANGED 2
@@ -100,6 +101,11 @@
#define BYTES_PER_DHLEN_UNIT 8
#define MAX_REG_CNT 2000000U
+#define INTRNAMSIZ 32
+#define IRQ_NAME_OFF(i) ((i) * INTRNAMSIZ)
+#define MAX_IOQ_INTERRUPTS_PER_PF (64 * 2)
+#define MAX_IOQ_INTERRUPTS_PER_VF (8 * 2)
+
static inline u32 incr_index(u32 index, u32 count, u32 max)
{
@@ -181,6 +187,7 @@
#define OCTNET_CMD_Q 0
/* NIC Command types */
+#define OCTNET_CMD_RESET_PF 0x0
#define OCTNET_CMD_CHANGE_MTU 0x1
#define OCTNET_CMD_CHANGE_MACADDR 0x2
#define OCTNET_CMD_CHANGE_DEVFLAGS 0x3
@@ -845,29 +852,6 @@
#define OCT_LINK_STATS_SIZE (sizeof(struct oct_link_stats))
-/* intrmod: max. packet rate threshold */
-#define LIO_INTRMOD_MAXPKT_RATETHR 196608
-/* intrmod: min. packet rate threshold */
-#define LIO_INTRMOD_MINPKT_RATETHR 9216
-/* intrmod: max. packets to trigger interrupt */
-#define LIO_INTRMOD_RXMAXCNT_TRIGGER 384
-/* intrmod: min. packets to trigger interrupt */
-#define LIO_INTRMOD_RXMINCNT_TRIGGER 0
-/* intrmod: max. time to trigger interrupt */
-#define LIO_INTRMOD_RXMAXTMR_TRIGGER 128
-/* 66xx:intrmod: min. time to trigger interrupt
- * (value of 1 is optimum for TCP_RR)
- */
-#define LIO_INTRMOD_RXMINTMR_TRIGGER 1
-
-/* intrmod: max. packets to trigger interrupt */
-#define LIO_INTRMOD_TXMAXCNT_TRIGGER 64
-/* intrmod: min. packets to trigger interrupt */
-#define LIO_INTRMOD_TXMINCNT_TRIGGER 0
-
-/* intrmod: poll interval in seconds */
-#define LIO_INTRMOD_CHECK_INTERVAL 1
-
struct oct_intrmod_cfg {
u64 rx_enable;
u64 tx_enable;
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_device.c b/drivers/net/ethernet/cavium/liquidio/octeon_device.c
index 9675ffb..e21b477 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_device.c
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_device.c
@@ -793,7 +793,7 @@
u32 num_descs = 0;
u32 iq_no = 0;
union oct_txpciq txpciq;
- int numa_node = cpu_to_node(iq_no % num_online_cpus());
+ int numa_node = dev_to_node(&oct->pci_dev->dev);
if (OCTEON_CN6XXX(oct))
num_descs =
@@ -837,7 +837,7 @@
u32 num_descs = 0;
u32 desc_size = 0;
u32 oq_no = 0;
- int numa_node = cpu_to_node(oq_no % num_online_cpus());
+ int numa_node = dev_to_node(&oct->pci_dev->dev);
if (OCTEON_CN6XXX(oct)) {
num_descs =
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_device.h b/drivers/net/ethernet/cavium/liquidio/octeon_device.h
index c301a38..92f67de 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_device.h
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_device.h
@@ -453,9 +453,6 @@
/** List of dispatch functions */
struct octeon_dispatch_list dispatch;
- /* Interrupt Moderation */
- struct oct_intrmod_cfg intrmod;
-
u32 int_status;
u64 droq_intr;
@@ -517,6 +514,9 @@
void *msix_entries;
+ /* when requesting IRQs, the names are stored here */
+ void *irq_name_storage;
+
struct octeon_sriov_info sriov_info;
struct octeon_pf_vf_hs_word pfvf_hsword;
@@ -538,6 +538,12 @@
u32 priv_flags;
void *watchdog_task;
+
+ u32 rx_coalesce_usecs;
+ u32 rx_max_coalesced_frames;
+ u32 tx_max_coalesced_frames;
+
+ bool cores_crashed;
};
#define OCT_DRV_ONLINE 1
@@ -551,12 +557,6 @@
#define CHIP_CONF(oct, TYPE) \
(((struct octeon_ ## TYPE *)((oct)->chip))->conf)
-struct oct_intrmod_cmd {
- struct octeon_device *oct_dev;
- struct octeon_soft_command *sc;
- struct oct_intrmod_cfg *cfg;
-};
-
/*------------------ Function Prototypes ----------------------*/
/** Initialize device list memory */
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_droq.c b/drivers/net/ethernet/cavium/liquidio/octeon_droq.c
index 79f8094..286be55 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_droq.c
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_droq.c
@@ -226,8 +226,7 @@
struct octeon_droq *droq;
u32 desc_ring_size = 0, c_num_descs = 0, c_buf_size = 0;
u32 c_pkts_per_intr = 0, c_refill_threshold = 0;
- int orig_node = dev_to_node(&oct->pci_dev->dev);
- int numa_node = cpu_to_node(q_no % num_online_cpus());
+ int numa_node = dev_to_node(&oct->pci_dev->dev);
dev_dbg(&oct->pci_dev->dev, "%s[%d]\n", __func__, q_no);
@@ -267,13 +266,8 @@
droq->buffer_size = c_buf_size;
desc_ring_size = droq->max_count * OCT_DROQ_DESC_SIZE;
- set_dev_node(&oct->pci_dev->dev, numa_node);
droq->desc_ring = lio_dma_alloc(oct, desc_ring_size,
(dma_addr_t *)&droq->desc_ring_dma);
- set_dev_node(&oct->pci_dev->dev, orig_node);
- if (!droq->desc_ring)
- droq->desc_ring = lio_dma_alloc(oct, desc_ring_size,
- (dma_addr_t *)&droq->desc_ring_dma);
if (!droq->desc_ring) {
dev_err(&oct->pci_dev->dev,
@@ -519,6 +513,32 @@
return desc_refilled;
}
+/** check if we can allocate packets to get out of oom.
+ * @param droq - Droq being checked.
+ * @return does not return anything
+ */
+void octeon_droq_check_oom(struct octeon_droq *droq)
+{
+ int desc_refilled;
+ struct octeon_device *oct = droq->oct_dev;
+
+ if (readl(droq->pkts_credit_reg) <= CN23XX_SLI_DEF_BP) {
+ spin_lock_bh(&droq->lock);
+ desc_refilled = octeon_droq_refill(oct, droq);
+ if (desc_refilled) {
+ /* Flush the droq descriptor data to memory to be sure
+ * that when we update the credits the data in memory
+ * is accurate.
+ */
+ wmb();
+ writel(desc_refilled, droq->pkts_credit_reg);
+ /* make sure mmio write completes */
+ mmiowb();
+ }
+ spin_unlock_bh(&droq->lock);
+ }
+}
+
static inline u32
octeon_droq_get_bufcount(u32 buf_size, u32 total_len)
{
@@ -970,7 +990,7 @@
u32 desc_size, void *app_ctx)
{
struct octeon_droq *droq;
- int numa_node = cpu_to_node(q_no % num_online_cpus());
+ int numa_node = dev_to_node(&oct->pci_dev->dev);
if (oct->droq[q_no]) {
dev_dbg(&oct->pci_dev->dev, "Droq already in use. Cannot create droq %d again\n",
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_droq.h b/drivers/net/ethernet/cavium/liquidio/octeon_droq.h
index 6982c0a..9781577 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_droq.h
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_droq.h
@@ -426,4 +426,6 @@
int octeon_process_droq_poll_cmd(struct octeon_device *oct, u32 q_no,
int cmd, u32 arg);
+void octeon_droq_check_oom(struct octeon_droq *droq);
+
#endif /*__OCTEON_DROQ_H__ */
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_iq.h b/drivers/net/ethernet/cavium/liquidio/octeon_iq.h
index 4608a5a..5063a12 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_iq.h
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_iq.h
@@ -152,7 +152,7 @@
struct oct_iq_stats stats;
/** DMA mapped base address of the input descriptor ring. */
- u64 base_addr_dma;
+ dma_addr_t base_addr_dma;
/** Application context */
void *app_ctx;
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c b/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
index 201b987..5cca73b 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
@@ -313,6 +313,7 @@
return 0;
}
+ spin_unlock_irqrestore(&mbox->lock, flags);
WARN_ON(1);
return 0;
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_network.h b/drivers/net/ethernet/cavium/liquidio/octeon_network.h
index eef2a1e..bf48393 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_network.h
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_network.h
@@ -28,6 +28,12 @@
#define LIO_MAX_MTU_SIZE (OCTNET_MAX_FRM_SIZE - OCTNET_FRM_HEADER_SIZE)
#define LIO_MIN_MTU_SIZE ETH_MIN_MTU
+/* Bit mask values for lio->ifstate */
+#define LIO_IFSTATE_DROQ_OPS 0x01
+#define LIO_IFSTATE_REGISTERED 0x02
+#define LIO_IFSTATE_RUNNING 0x04
+#define LIO_IFSTATE_RX_TIMESTAMP_ENABLED 0x08
+
struct oct_nic_stats_resp {
u64 rh;
struct oct_link_stats stats;
@@ -123,6 +129,9 @@
/* work queue for txq status */
struct cavium_wq txq_status_wq;
+ /* work queue for rxq oom status */
+ struct cavium_wq rxq_status_wq;
+
/* work queue for link status */
struct cavium_wq link_status_wq;
@@ -132,10 +141,6 @@
#define LIO_SIZE (sizeof(struct lio))
#define GET_LIO(netdev) ((struct lio *)netdev_priv(netdev))
-#define CIU3_WDOG(c) (0x1010000020000ULL + ((c) << 3))
-#define CIU3_WDOG_MASK 12ULL
-#define LIO_MONITOR_WDOG_EXPIRE 1
-#define LIO_MONITOR_CORE_STUCK_MSGD 2
#define LIO_MAX_CORES 12
/**
@@ -146,6 +151,10 @@
*/
int liquidio_set_feature(struct net_device *netdev, int cmd, u16 param1);
+int setup_rx_oom_poll_fn(struct net_device *netdev);
+
+void cleanup_rx_oom_poll_fn(struct net_device *netdev);
+
/**
* \brief Link control command completion callback
* @param nctrl_ptr pointer to control packet structure
@@ -438,4 +447,34 @@
get_rbd(droq->recv_buf_list[idx].buffer), copy_len);
}
+/**
+ * \brief check interface state
+ * @param lio per-network private data
+ * @param state_flag flag state to check
+ */
+static inline int ifstate_check(struct lio *lio, int state_flag)
+{
+ return atomic_read(&lio->ifstate) & state_flag;
+}
+
+/**
+ * \brief set interface state
+ * @param lio per-network private data
+ * @param state_flag flag state to set
+ */
+static inline void ifstate_set(struct lio *lio, int state_flag)
+{
+ atomic_set(&lio->ifstate, (atomic_read(&lio->ifstate) | state_flag));
+}
+
+/**
+ * \brief clear interface state
+ * @param lio per-network private data
+ * @param state_flag flag state to clear
+ */
+static inline void ifstate_reset(struct lio *lio, int state_flag)
+{
+ atomic_set(&lio->ifstate, (atomic_read(&lio->ifstate) & ~(state_flag)));
+}
+
#endif
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_nic.c b/drivers/net/ethernet/cavium/liquidio/octeon_nic.c
index 0243be8..b457cf2 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_nic.c
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_nic.c
@@ -100,14 +100,16 @@
nctrl = (struct octnic_ctrl_pkt *)sc->ctxptr;
- /* Call the callback function if status is OK.
- * Status is OK only if a response was expected and core returned
- * success.
+ /* Call the callback function if status is zero (meaning OK) or status
+ * contains a firmware status code bigger than zero (meaning the
+ * firmware is reporting an error).
* If no response was expected, status is OK if the command was posted
* successfully.
*/
- if (!status && nctrl->cb_fn)
+ if ((!status || status > FIRMWARE_STATUS_CODE(0)) && nctrl->cb_fn) {
+ nctrl->status = status;
nctrl->cb_fn(nctrl);
+ }
octeon_free_soft_command(oct, sc);
}
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_nic.h b/drivers/net/ethernet/cavium/liquidio/octeon_nic.h
index 0c7a5c9..6480ef8 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_nic.h
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_nic.h
@@ -62,6 +62,10 @@
/** Callback function called when the command has been fetched */
octnic_ctrl_pkt_cb_fn_t cb_fn;
+
+ u32 status;
+ u16 *response_code;
+ struct completion *completion;
};
#define MAX_UDD_SIZE(nctrl) (sizeof((nctrl)->udd))
diff --git a/drivers/net/ethernet/cavium/liquidio/request_manager.c b/drivers/net/ethernet/cavium/liquidio/request_manager.c
index 707bc15..261f448 100644
--- a/drivers/net/ethernet/cavium/liquidio/request_manager.c
+++ b/drivers/net/ethernet/cavium/liquidio/request_manager.c
@@ -62,8 +62,7 @@
u32 iq_no = (u32)txpciq.s.q_no;
u32 q_size;
struct cavium_wq *db_wq;
- int orig_node = dev_to_node(&oct->pci_dev->dev);
- int numa_node = cpu_to_node(iq_no % num_online_cpus());
+ int numa_node = dev_to_node(&oct->pci_dev->dev);
if (OCTEON_CN6XXX(oct))
conf = &(CFG_GET_IQ_CFG(CHIP_CONF(oct, cn6xxx)));
@@ -91,13 +90,7 @@
iq->oct_dev = oct;
- set_dev_node(&oct->pci_dev->dev, numa_node);
- iq->base_addr = lio_dma_alloc(oct, q_size,
- (dma_addr_t *)&iq->base_addr_dma);
- set_dev_node(&oct->pci_dev->dev, orig_node);
- if (!iq->base_addr)
- iq->base_addr = lio_dma_alloc(oct, q_size,
- (dma_addr_t *)&iq->base_addr_dma);
+ iq->base_addr = lio_dma_alloc(oct, q_size, &iq->base_addr_dma);
if (!iq->base_addr) {
dev_err(&oct->pci_dev->dev, "Cannot allocate memory for instr queue %d\n",
iq_no);
@@ -211,7 +204,7 @@
void *app_ctx)
{
u32 iq_no = (u32)txpciq.s.q_no;
- int numa_node = cpu_to_node(iq_no % num_online_cpus());
+ int numa_node = dev_to_node(&oct->pci_dev->dev);
if (oct->instr_queue[iq_no]) {
dev_dbg(&oct->pci_dev->dev, "IQ is in use. Cannot create the IQ: %d again\n",
diff --git a/drivers/net/ethernet/cavium/liquidio/response_manager.c b/drivers/net/ethernet/cavium/liquidio/response_manager.c
index 2fbaae9..3d691c6 100644
--- a/drivers/net/ethernet/cavium/liquidio/response_manager.c
+++ b/drivers/net/ethernet/cavium/liquidio/response_manager.c
@@ -69,50 +69,53 @@
int resp_to_process = MAX_ORD_REQS_TO_PROCESS;
u32 status;
u64 status64;
- struct octeon_instr_rdp *rdp;
- u64 rptr;
ordered_sc_list = &octeon_dev->response_list[OCTEON_ORDERED_SC_LIST];
do {
spin_lock_bh(&ordered_sc_list->lock);
- if (ordered_sc_list->head.next == &ordered_sc_list->head) {
+ if (list_empty(&ordered_sc_list->head)) {
spin_unlock_bh(&ordered_sc_list->lock);
return 1;
}
- sc = (struct octeon_soft_command *)ordered_sc_list->
- head.next;
- if (OCTEON_CN23XX_PF(octeon_dev) ||
- OCTEON_CN23XX_VF(octeon_dev)) {
- rdp = (struct octeon_instr_rdp *)&sc->cmd.cmd3.rdp;
- rptr = sc->cmd.cmd3.rptr;
- } else {
- rdp = (struct octeon_instr_rdp *)&sc->cmd.cmd2.rdp;
- rptr = sc->cmd.cmd2.rptr;
- }
+ sc = list_first_entry(&ordered_sc_list->head,
+ struct octeon_soft_command, node);
status = OCTEON_REQUEST_PENDING;
/* check if octeon has finished DMA'ing a response
* to where rptr is pointing to
*/
- dma_sync_single_for_cpu(&octeon_dev->pci_dev->dev,
- rptr, rdp->rlen,
- DMA_FROM_DEVICE);
status64 = *sc->status_word;
if (status64 != COMPLETION_WORD_INIT) {
+ /* This logic ensures that all 64b have been written.
+ * 1. check byte 0 for non-FF
+ * 2. if non-FF, then swap result from BE to host order
+ * 3. check byte 7 (swapped to 0) for non-FF
+ * 4. if non-FF, use the low 32-bit status code
+ * 5. if either byte 0 or byte 7 is FF, don't use status
+ */
if ((status64 & 0xff) != 0xff) {
octeon_swap_8B_data(&status64, 1);
if (((status64 & 0xff) != 0xff)) {
- status = (u32)(status64 &
- 0xffffffffULL);
+ /* retrieve 16-bit firmware status */
+ status = (u32)(status64 & 0xffffULL);
+ if (status) {
+ status =
+ FIRMWARE_STATUS_CODE(status);
+ } else {
+ /* i.e. no error */
+ status = OCTEON_REQUEST_DONE;
+ }
}
}
} else if (force_quit || (sc->timeout &&
time_after(jiffies, (unsigned long)sc->timeout))) {
+ dev_err(&octeon_dev->pci_dev->dev, "%s: cmd failed, timeout (%ld, %ld)\n",
+ __func__, (long)jiffies, (long)sc->timeout);
status = OCTEON_REQUEST_TIMEOUT;
}
diff --git a/drivers/net/ethernet/cavium/liquidio/response_manager.h b/drivers/net/ethernet/cavium/liquidio/response_manager.h
index cbb2d84..9169c28 100644
--- a/drivers/net/ethernet/cavium/liquidio/response_manager.h
+++ b/drivers/net/ethernet/cavium/liquidio/response_manager.h
@@ -78,6 +78,8 @@
/*------------ Error codes used by host driver -----------------*/
#define DRIVER_MAJOR_ERROR_CODE 0x0000
+/*------ Error codes used by firmware (bits 15..0 set by firmware */
+#define FIRMWARE_MAJOR_ERROR_CODE 0x0001
/** A value of 0x00000000 indicates no error i.e. success */
#define DRIVER_ERROR_NONE 0x00000000
@@ -116,6 +118,9 @@
};
+#define FIRMWARE_STATUS_CODE(status) \
+ ((FIRMWARE_MAJOR_ERROR_CODE << 16) | (status))
+
/** Initialize the response lists. The number of response lists to create is
* given by count.
* @param octeon_dev - the octeon device structure.
diff --git a/drivers/net/ethernet/cavium/thunder/nic.h b/drivers/net/ethernet/cavium/thunder/nic.h
index 2269ff5..4a02e61 100644
--- a/drivers/net/ethernet/cavium/thunder/nic.h
+++ b/drivers/net/ethernet/cavium/thunder/nic.h
@@ -252,12 +252,14 @@
u64 tx_csum_overflow;
/* driver debug stats */
- u64 rcv_buffer_alloc_failures;
u64 tx_tso;
u64 tx_timeout;
u64 txq_stop;
u64 txq_wake;
+ u64 rcv_buffer_alloc_failures;
+ u64 page_alloc;
+
struct u64_stats_sync syncp;
};
@@ -266,9 +268,9 @@
struct net_device *netdev;
struct pci_dev *pdev;
void __iomem *reg_base;
+ struct bpf_prog *xdp_prog;
#define MAX_QUEUES_PER_QSET 8
struct queue_set *qs;
- struct nicvf_cq_poll *napi[8];
void *iommu_domain;
u8 vf_id;
u8 sqs_id;
@@ -294,6 +296,7 @@
/* Queue count */
u8 rx_queues;
u8 tx_queues;
+ u8 xdp_tx_queues;
u8 max_queues;
u8 node;
@@ -318,10 +321,11 @@
struct nicvf_drv_stats __percpu *drv_stats;
struct bgx_stats bgx_stats;
+ /* Napi */
+ struct nicvf_cq_poll *napi[8];
+
/* MSI-X */
- bool msix_enabled;
u8 num_vec;
- struct msix_entry msix_entries[NIC_VF_MSIX_VECTORS];
char irq_name[NIC_VF_MSIX_VECTORS][IFNAMSIZ + 15];
bool irq_allocated[NIC_VF_MSIX_VECTORS];
cpumask_var_t affinity_mask[NIC_VF_MSIX_VECTORS];
diff --git a/drivers/net/ethernet/cavium/thunder/nic_main.c b/drivers/net/ethernet/cavium/thunder/nic_main.c
index 767234e..fb770b0 100644
--- a/drivers/net/ethernet/cavium/thunder/nic_main.c
+++ b/drivers/net/ethernet/cavium/thunder/nic_main.c
@@ -65,9 +65,7 @@
bool mbx_lock[MAX_NUM_VFS_SUPPORTED];
/* MSI-X */
- bool msix_enabled;
u8 num_vec;
- struct msix_entry *msix_entries;
bool irq_allocated[NIC_PF_MSIX_VECTORS];
char irq_name[NIC_PF_MSIX_VECTORS][20];
};
@@ -1088,7 +1086,7 @@
u64 intr;
u8 vf, vf_per_mbx_reg = 64;
- if (irq == nic->msix_entries[NIC_PF_INTR_ID_MBOX0].vector)
+ if (irq == pci_irq_vector(nic->pdev, NIC_PF_INTR_ID_MBOX0))
mbx = 0;
else
mbx = 1;
@@ -1107,51 +1105,13 @@
return IRQ_HANDLED;
}
-static int nic_enable_msix(struct nicpf *nic)
-{
- int i, ret;
-
- nic->num_vec = pci_msix_vec_count(nic->pdev);
-
- nic->msix_entries = kmalloc_array(nic->num_vec,
- sizeof(struct msix_entry),
- GFP_KERNEL);
- if (!nic->msix_entries)
- return -ENOMEM;
-
- for (i = 0; i < nic->num_vec; i++)
- nic->msix_entries[i].entry = i;
-
- ret = pci_enable_msix(nic->pdev, nic->msix_entries, nic->num_vec);
- if (ret) {
- dev_err(&nic->pdev->dev,
- "Request for #%d msix vectors failed, returned %d\n",
- nic->num_vec, ret);
- kfree(nic->msix_entries);
- return ret;
- }
-
- nic->msix_enabled = 1;
- return 0;
-}
-
-static void nic_disable_msix(struct nicpf *nic)
-{
- if (nic->msix_enabled) {
- pci_disable_msix(nic->pdev);
- kfree(nic->msix_entries);
- nic->msix_enabled = 0;
- nic->num_vec = 0;
- }
-}
-
static void nic_free_all_interrupts(struct nicpf *nic)
{
int irq;
for (irq = 0; irq < nic->num_vec; irq++) {
if (nic->irq_allocated[irq])
- free_irq(nic->msix_entries[irq].vector, nic);
+ free_irq(pci_irq_vector(nic->pdev, irq), nic);
nic->irq_allocated[irq] = false;
}
}
@@ -1159,18 +1119,24 @@
static int nic_register_interrupts(struct nicpf *nic)
{
int i, ret;
+ nic->num_vec = pci_msix_vec_count(nic->pdev);
/* Enable MSI-X */
- ret = nic_enable_msix(nic);
- if (ret)
- return ret;
+ ret = pci_alloc_irq_vectors(nic->pdev, nic->num_vec, nic->num_vec,
+ PCI_IRQ_MSIX);
+ if (ret < 0) {
+ dev_err(&nic->pdev->dev,
+ "Request for #%d msix vectors failed, returned %d\n",
+ nic->num_vec, ret);
+ return 1;
+ }
/* Register mailbox interrupt handler */
for (i = NIC_PF_INTR_ID_MBOX0; i < nic->num_vec; i++) {
sprintf(nic->irq_name[i],
"NICPF Mbox%d", (i - NIC_PF_INTR_ID_MBOX0));
- ret = request_irq(nic->msix_entries[i].vector,
+ ret = request_irq(pci_irq_vector(nic->pdev, i),
nic_mbx_intr_handler, 0,
nic->irq_name[i], nic);
if (ret)
@@ -1186,14 +1152,16 @@
fail:
dev_err(&nic->pdev->dev, "Request irq failed\n");
nic_free_all_interrupts(nic);
- nic_disable_msix(nic);
+ pci_free_irq_vectors(nic->pdev);
+ nic->num_vec = 0;
return ret;
}
static void nic_unregister_interrupts(struct nicpf *nic)
{
nic_free_all_interrupts(nic);
- nic_disable_msix(nic);
+ pci_free_irq_vectors(nic->pdev);
+ nic->num_vec = 0;
}
static int nic_num_sqs_en(struct nicpf *nic, int vf_en)
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c b/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c
index 02a986c..b9ece9c 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c
@@ -100,11 +100,12 @@
NICVF_DRV_STAT(tx_csum_overlap),
NICVF_DRV_STAT(tx_csum_overflow),
- NICVF_DRV_STAT(rcv_buffer_alloc_failures),
NICVF_DRV_STAT(tx_tso),
NICVF_DRV_STAT(tx_timeout),
NICVF_DRV_STAT(txq_stop),
NICVF_DRV_STAT(txq_wake),
+ NICVF_DRV_STAT(rcv_buffer_alloc_failures),
+ NICVF_DRV_STAT(page_alloc),
};
static const struct nicvf_stat nicvf_queue_stats[] = {
@@ -720,7 +721,7 @@
struct nicvf *nic = netdev_priv(dev);
int err = 0;
bool if_up = netif_running(dev);
- int cqcount;
+ u8 cqcount, txq_count;
if (!channel->rx_count || !channel->tx_count)
return -EINVAL;
@@ -729,10 +730,26 @@
if (channel->tx_count > nic->max_queues)
return -EINVAL;
+ if (nic->xdp_prog &&
+ ((channel->tx_count + channel->rx_count) > nic->max_queues)) {
+ netdev_err(nic->netdev,
+ "XDP mode, RXQs + TXQs > Max %d\n",
+ nic->max_queues);
+ return -EINVAL;
+ }
+
if (if_up)
nicvf_stop(dev);
- cqcount = max(channel->rx_count, channel->tx_count);
+ nic->rx_queues = channel->rx_count;
+ nic->tx_queues = channel->tx_count;
+ if (!nic->xdp_prog)
+ nic->xdp_tx_queues = 0;
+ else
+ nic->xdp_tx_queues = channel->rx_count;
+
+ txq_count = nic->xdp_tx_queues + nic->tx_queues;
+ cqcount = max(nic->rx_queues, txq_count);
if (cqcount > MAX_CMP_QUEUES_PER_QS) {
nic->sqs_count = roundup(cqcount, MAX_CMP_QUEUES_PER_QS);
@@ -741,12 +758,10 @@
nic->sqs_count = 0;
}
- nic->qs->rq_cnt = min_t(u32, channel->rx_count, MAX_RCV_QUEUES_PER_QS);
- nic->qs->sq_cnt = min_t(u32, channel->tx_count, MAX_SND_QUEUES_PER_QS);
+ nic->qs->rq_cnt = min_t(u8, nic->rx_queues, MAX_RCV_QUEUES_PER_QS);
+ nic->qs->sq_cnt = min_t(u8, txq_count, MAX_SND_QUEUES_PER_QS);
nic->qs->cq_cnt = max(nic->qs->rq_cnt, nic->qs->sq_cnt);
- nic->rx_queues = channel->rx_count;
- nic->tx_queues = channel->tx_count;
err = nicvf_set_real_num_queues(dev, nic->tx_queues, nic->rx_queues);
if (err)
return err;
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_main.c b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
index 2401758..d6477af 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_main.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_main.c
@@ -17,6 +17,9 @@
#include <linux/prefetch.h>
#include <linux/irq.h>
#include <linux/iommu.h>
+#include <linux/bpf.h>
+#include <linux/bpf_trace.h>
+#include <linux/filter.h>
#include "nic_reg.h"
#include "nic.h"
@@ -397,8 +400,10 @@
if (nic->rx_queues > MAX_RCV_QUEUES_PER_QS)
rx_queues = nic->rx_queues - MAX_RCV_QUEUES_PER_QS;
- if (nic->tx_queues > MAX_SND_QUEUES_PER_QS)
- tx_queues = nic->tx_queues - MAX_SND_QUEUES_PER_QS;
+
+ tx_queues = nic->tx_queues + nic->xdp_tx_queues;
+ if (tx_queues > MAX_SND_QUEUES_PER_QS)
+ tx_queues = tx_queues - MAX_SND_QUEUES_PER_QS;
/* Set no of Rx/Tx queues in each of the SQsets */
for (sqs = 0; sqs < nic->sqs_count; sqs++) {
@@ -496,12 +501,99 @@
return 0;
}
+static inline bool nicvf_xdp_rx(struct nicvf *nic, struct bpf_prog *prog,
+ struct cqe_rx_t *cqe_rx, struct snd_queue *sq,
+ struct sk_buff **skb)
+{
+ struct xdp_buff xdp;
+ struct page *page;
+ u32 action;
+ u16 len, offset = 0;
+ u64 dma_addr, cpu_addr;
+ void *orig_data;
+
+ /* Retrieve packet buffer's DMA address and length */
+ len = *((u16 *)((void *)cqe_rx + (3 * sizeof(u64))));
+ dma_addr = *((u64 *)((void *)cqe_rx + (7 * sizeof(u64))));
+
+ cpu_addr = nicvf_iova_to_phys(nic, dma_addr);
+ if (!cpu_addr)
+ return false;
+ cpu_addr = (u64)phys_to_virt(cpu_addr);
+ page = virt_to_page((void *)cpu_addr);
+
+ xdp.data_hard_start = page_address(page);
+ xdp.data = (void *)cpu_addr;
+ xdp.data_end = xdp.data + len;
+ orig_data = xdp.data;
+
+ rcu_read_lock();
+ action = bpf_prog_run_xdp(prog, &xdp);
+ rcu_read_unlock();
+
+ /* Check if XDP program has changed headers */
+ if (orig_data != xdp.data) {
+ len = xdp.data_end - xdp.data;
+ offset = orig_data - xdp.data;
+ dma_addr -= offset;
+ }
+
+ switch (action) {
+ case XDP_PASS:
+ /* Check if it's a recycled page, if not
+ * unmap the DMA mapping.
+ *
+ * Recycled page holds an extra reference.
+ */
+ if (page_ref_count(page) == 1) {
+ dma_addr &= PAGE_MASK;
+ dma_unmap_page_attrs(&nic->pdev->dev, dma_addr,
+ RCV_FRAG_LEN + XDP_PACKET_HEADROOM,
+ DMA_FROM_DEVICE,
+ DMA_ATTR_SKIP_CPU_SYNC);
+ }
+
+ /* Build SKB and pass on packet to network stack */
+ *skb = build_skb(xdp.data,
+ RCV_FRAG_LEN - cqe_rx->align_pad + offset);
+ if (!*skb)
+ put_page(page);
+ else
+ skb_put(*skb, len);
+ return false;
+ case XDP_TX:
+ nicvf_xdp_sq_append_pkt(nic, sq, (u64)xdp.data, dma_addr, len);
+ return true;
+ default:
+ bpf_warn_invalid_xdp_action(action);
+ case XDP_ABORTED:
+ trace_xdp_exception(nic->netdev, prog, action);
+ case XDP_DROP:
+ /* Check if it's a recycled page, if not
+ * unmap the DMA mapping.
+ *
+ * Recycled page holds an extra reference.
+ */
+ if (page_ref_count(page) == 1) {
+ dma_addr &= PAGE_MASK;
+ dma_unmap_page_attrs(&nic->pdev->dev, dma_addr,
+ RCV_FRAG_LEN + XDP_PACKET_HEADROOM,
+ DMA_FROM_DEVICE,
+ DMA_ATTR_SKIP_CPU_SYNC);
+ }
+ put_page(page);
+ return true;
+ }
+ return false;
+}
+
static void nicvf_snd_pkt_handler(struct net_device *netdev,
struct cqe_send_t *cqe_tx,
- int cqe_type, int budget,
+ int budget, int *subdesc_cnt,
unsigned int *tx_pkts, unsigned int *tx_bytes)
{
struct sk_buff *skb = NULL;
+ struct page *page;
struct nicvf *nic = netdev_priv(netdev);
struct snd_queue *sq;
struct sq_hdr_subdesc *hdr;
@@ -513,12 +605,26 @@
if (hdr->subdesc_type != SQ_DESC_TYPE_HEADER)
return;
- netdev_dbg(nic->netdev,
- "%s Qset #%d SQ #%d SQ ptr #%d subdesc count %d\n",
- __func__, cqe_tx->sq_qs, cqe_tx->sq_idx,
- cqe_tx->sqe_ptr, hdr->subdesc_cnt);
+ /* Check for errors */
+ if (cqe_tx->send_status)
+ nicvf_check_cqe_tx_errs(nic->pnicvf, cqe_tx);
- nicvf_check_cqe_tx_errs(nic, cqe_tx);
+ /* Is this a XDP designated Tx queue */
+ if (sq->is_xdp) {
+ page = (struct page *)sq->xdp_page[cqe_tx->sqe_ptr];
+ /* Check if it's recycled page or else unmap DMA mapping */
+ if (page && (page_ref_count(page) == 1))
+ nicvf_unmap_sndq_buffers(nic, sq, cqe_tx->sqe_ptr,
+ hdr->subdesc_cnt);
+
+ /* Release page reference for recycling */
+ if (page)
+ put_page(page);
+ sq->xdp_page[cqe_tx->sqe_ptr] = (u64)NULL;
+ *subdesc_cnt += hdr->subdesc_cnt + 1;
+ return;
+ }
+
skb = (struct sk_buff *)sq->skbuff[cqe_tx->sqe_ptr];
if (skb) {
/* Check for dummy descriptor used for HW TSO offload on 88xx */
@@ -528,12 +634,12 @@
(struct sq_hdr_subdesc *)GET_SQ_DESC(sq, hdr->rsvd2);
nicvf_unmap_sndq_buffers(nic, sq, hdr->rsvd2,
tso_sqe->subdesc_cnt);
- nicvf_put_sq_desc(sq, tso_sqe->subdesc_cnt + 1);
+ *subdesc_cnt += tso_sqe->subdesc_cnt + 1;
} else {
nicvf_unmap_sndq_buffers(nic, sq, cqe_tx->sqe_ptr,
hdr->subdesc_cnt);
}
- nicvf_put_sq_desc(sq, hdr->subdesc_cnt + 1);
+ *subdesc_cnt += hdr->subdesc_cnt + 1;
prefetch(skb);
(*tx_pkts)++;
*tx_bytes += skb->len;
@@ -544,7 +650,7 @@
* a SKB attached, so just free SQEs here.
*/
if (!nic->hw_tso)
- nicvf_put_sq_desc(sq, hdr->subdesc_cnt + 1);
+ *subdesc_cnt += hdr->subdesc_cnt + 1;
}
}
@@ -578,9 +684,9 @@
static void nicvf_rcv_pkt_handler(struct net_device *netdev,
struct napi_struct *napi,
- struct cqe_rx_t *cqe_rx)
+ struct cqe_rx_t *cqe_rx, struct snd_queue *sq)
{
- struct sk_buff *skb;
+ struct sk_buff *skb = NULL;
struct nicvf *nic = netdev_priv(netdev);
struct nicvf *snic = nic;
int err = 0;
@@ -595,16 +701,25 @@
}
/* Check for errors */
- err = nicvf_check_cqe_rx_errs(nic, cqe_rx);
- if (err && !cqe_rx->rb_cnt)
- return;
-
- skb = nicvf_get_rcv_skb(snic, cqe_rx);
- if (!skb) {
- netdev_dbg(nic->netdev, "Packet not received\n");
- return;
+ if (cqe_rx->err_level || cqe_rx->err_opcode) {
+ err = nicvf_check_cqe_rx_errs(nic, cqe_rx);
+ if (err && !cqe_rx->rb_cnt)
+ return;
}
+ /* For XDP, ignore pkts spanning multiple pages */
+ if (nic->xdp_prog && (cqe_rx->rb_cnt == 1)) {
+ /* Packet consumed by XDP */
+ if (nicvf_xdp_rx(snic, nic->xdp_prog, cqe_rx, sq, &skb))
+ return;
+ } else {
+ skb = nicvf_get_rcv_skb(snic, cqe_rx,
+ nic->xdp_prog ? true : false);
+ }
+
+ if (!skb)
+ return;
+
if (netif_msg_pktdata(nic)) {
netdev_info(nic->netdev, "%s: skb 0x%p, len=%d\n", netdev->name,
skb, skb->len);
@@ -646,13 +761,14 @@
{
int processed_cqe, work_done = 0, tx_done = 0;
int cqe_count, cqe_head;
+ int subdesc_cnt = 0;
struct nicvf *nic = netdev_priv(netdev);
struct queue_set *qs = nic->qs;
struct cmp_queue *cq = &qs->cq[cq_idx];
struct cqe_rx_t *cq_desc;
struct netdev_queue *txq;
- struct snd_queue *sq;
- unsigned int tx_pkts = 0, tx_bytes = 0;
+ struct snd_queue *sq = &qs->sq[cq_idx];
+ unsigned int tx_pkts = 0, tx_bytes = 0, txq_idx;
spin_lock_bh(&cq->lock);
loop:
@@ -667,8 +783,6 @@
cqe_head = nicvf_queue_reg_read(nic, NIC_QSET_CQ_0_7_HEAD, cq_idx) >> 9;
cqe_head &= 0xFFFF;
- netdev_dbg(nic->netdev, "%s CQ%d cqe_count %d cqe_head %d\n",
- __func__, cq_idx, cqe_count, cqe_head);
while (processed_cqe < cqe_count) {
/* Get the CQ descriptor */
cq_desc = (struct cqe_rx_t *)GET_CQ_DESC(cq, cqe_head);
@@ -682,17 +796,15 @@
break;
}
- netdev_dbg(nic->netdev, "CQ%d cq_desc->cqe_type %d\n",
- cq_idx, cq_desc->cqe_type);
switch (cq_desc->cqe_type) {
case CQE_TYPE_RX:
- nicvf_rcv_pkt_handler(netdev, napi, cq_desc);
+ nicvf_rcv_pkt_handler(netdev, napi, cq_desc, sq);
work_done++;
break;
case CQE_TYPE_SEND:
- nicvf_snd_pkt_handler(netdev,
- (void *)cq_desc, CQE_TYPE_SEND,
- budget, &tx_pkts, &tx_bytes);
+ nicvf_snd_pkt_handler(netdev, (void *)cq_desc,
+ budget, &subdesc_cnt,
+ &tx_pkts, &tx_bytes);
tx_done++;
break;
case CQE_TYPE_INVALID:
@@ -704,9 +816,6 @@
}
processed_cqe++;
}
- netdev_dbg(nic->netdev,
- "%s CQ%d processed_cqe %d work_done %d budget %d\n",
- __func__, cq_idx, processed_cqe, work_done, budget);
/* Ring doorbell to inform H/W to reuse processed CQEs */
nicvf_queue_reg_write(nic, NIC_QSET_CQ_0_7_DOOR,
@@ -716,13 +825,26 @@
goto loop;
done:
+ /* Update SQ's descriptor free count */
+ if (subdesc_cnt)
+ nicvf_put_sq_desc(sq, subdesc_cnt);
+
+ txq_idx = nicvf_netdev_qidx(nic, cq_idx);
+ /* Handle XDP TX queues */
+ if (nic->pnicvf->xdp_prog) {
+ if (txq_idx < nic->pnicvf->xdp_tx_queues) {
+ nicvf_xdp_sq_doorbell(nic, sq, cq_idx);
+ goto out;
+ }
+ nic = nic->pnicvf;
+ txq_idx -= nic->pnicvf->xdp_tx_queues;
+ }
+
/* Wakeup TXQ if its stopped earlier due to SQ full */
- sq = &nic->qs->sq[cq_idx];
if (tx_done ||
(atomic_read(&sq->free_cnt) >= MIN_SQ_DESC_PER_PKT_XMIT)) {
netdev = nic->pnicvf->netdev;
- txq = netdev_get_tx_queue(netdev,
- nicvf_netdev_qidx(nic, cq_idx));
+ txq = netdev_get_tx_queue(netdev, txq_idx);
if (tx_pkts)
netdev_tx_completed_queue(txq, tx_pkts, tx_bytes);
@@ -735,10 +857,11 @@
if (netif_msg_tx_err(nic))
netdev_warn(netdev,
"%s: Transmit queue wakeup SQ%d\n",
- netdev->name, cq_idx);
+ netdev->name, txq_idx);
}
}
+out:
spin_unlock_bh(&cq->lock);
return work_done;
}
@@ -882,38 +1005,9 @@
return IRQ_HANDLED;
}
-static int nicvf_enable_msix(struct nicvf *nic)
-{
- int ret, vec;
-
- nic->num_vec = NIC_VF_MSIX_VECTORS;
-
- for (vec = 0; vec < nic->num_vec; vec++)
- nic->msix_entries[vec].entry = vec;
-
- ret = pci_enable_msix(nic->pdev, nic->msix_entries, nic->num_vec);
- if (ret) {
- netdev_err(nic->netdev,
- "Req for #%d msix vectors failed\n", nic->num_vec);
- return 0;
- }
- nic->msix_enabled = 1;
- return 1;
-}
-
-static void nicvf_disable_msix(struct nicvf *nic)
-{
- if (nic->msix_enabled) {
- pci_disable_msix(nic->pdev);
- nic->msix_enabled = 0;
- nic->num_vec = 0;
- }
-}
-
static void nicvf_set_irq_affinity(struct nicvf *nic)
{
int vec, cpu;
- int irqnum;
for (vec = 0; vec < nic->num_vec; vec++) {
if (!nic->irq_allocated[vec])
@@ -930,15 +1024,14 @@
cpumask_set_cpu(cpumask_local_spread(cpu, nic->node),
nic->affinity_mask[vec]);
- irqnum = nic->msix_entries[vec].vector;
- irq_set_affinity_hint(irqnum, nic->affinity_mask[vec]);
+ irq_set_affinity_hint(pci_irq_vector(nic->pdev, vec),
+ nic->affinity_mask[vec]);
}
}
static int nicvf_register_interrupts(struct nicvf *nic)
{
int irq, ret = 0;
- int vector;
for_each_cq_irq(irq)
sprintf(nic->irq_name[irq], "%s-rxtx-%d",
@@ -957,8 +1050,8 @@
/* Register CQ interrupts */
for (irq = 0; irq < nic->qs->cq_cnt; irq++) {
- vector = nic->msix_entries[irq].vector;
- ret = request_irq(vector, nicvf_intr_handler,
+ ret = request_irq(pci_irq_vector(nic->pdev, irq),
+ nicvf_intr_handler,
0, nic->irq_name[irq], nic->napi[irq]);
if (ret)
goto err;
@@ -968,8 +1061,8 @@
/* Register RBDR interrupt */
for (irq = NICVF_INTR_ID_RBDR;
irq < (NICVF_INTR_ID_RBDR + nic->qs->rbdr_cnt); irq++) {
- vector = nic->msix_entries[irq].vector;
- ret = request_irq(vector, nicvf_rbdr_intr_handler,
+ ret = request_irq(pci_irq_vector(nic->pdev, irq),
+ nicvf_rbdr_intr_handler,
0, nic->irq_name[irq], nic);
if (ret)
goto err;
@@ -981,7 +1074,7 @@
nic->pnicvf->netdev->name,
nic->sqs_mode ? (nic->sqs_id + 1) : 0);
irq = NICVF_INTR_ID_QS_ERR;
- ret = request_irq(nic->msix_entries[irq].vector,
+ ret = request_irq(pci_irq_vector(nic->pdev, irq),
nicvf_qs_err_intr_handler,
0, nic->irq_name[irq], nic);
if (ret)
@@ -1001,6 +1094,7 @@
static void nicvf_unregister_interrupts(struct nicvf *nic)
{
+ struct pci_dev *pdev = nic->pdev;
int irq;
/* Free registered interrupts */
@@ -1008,19 +1102,20 @@
if (!nic->irq_allocated[irq])
continue;
- irq_set_affinity_hint(nic->msix_entries[irq].vector, NULL);
+ irq_set_affinity_hint(pci_irq_vector(pdev, irq), NULL);
free_cpumask_var(nic->affinity_mask[irq]);
if (irq < NICVF_INTR_ID_SQ)
- free_irq(nic->msix_entries[irq].vector, nic->napi[irq]);
+ free_irq(pci_irq_vector(pdev, irq), nic->napi[irq]);
else
- free_irq(nic->msix_entries[irq].vector, nic);
+ free_irq(pci_irq_vector(pdev, irq), nic);
nic->irq_allocated[irq] = false;
}
/* Disable MSI-X */
- nicvf_disable_msix(nic);
+ pci_free_irq_vectors(pdev);
+ nic->num_vec = 0;
}
/* Initialize MSIX vectors and register MISC interrupt.
@@ -1032,16 +1127,22 @@
int irq = NICVF_INTR_ID_MISC;
/* Return if mailbox interrupt is already registered */
- if (nic->msix_enabled)
+ if (nic->pdev->msix_enabled)
return 0;
/* Enable MSI-X */
- if (!nicvf_enable_msix(nic))
+ nic->num_vec = pci_msix_vec_count(nic->pdev);
+ ret = pci_alloc_irq_vectors(nic->pdev, nic->num_vec, nic->num_vec,
+ PCI_IRQ_MSIX);
+ if (ret < 0) {
+ netdev_err(nic->netdev,
+ "Req for #%d msix vectors failed\n", nic->num_vec);
return 1;
+ }
sprintf(nic->irq_name[irq], "%s Mbox", "NICVF");
/* Register Misc interrupt */
- ret = request_irq(nic->msix_entries[irq].vector,
+ ret = request_irq(pci_irq_vector(nic->pdev, irq),
nicvf_misc_intr_handler, 0, nic->irq_name[irq], nic);
if (ret)
@@ -1076,6 +1177,13 @@
return NETDEV_TX_OK;
}
+ /* In XDP case, initial HW tx queues are used for XDP,
+ * but stack's queue mapping starts at '0', so skip the
+ * Tx queues attached to Rx queues for XDP.
+ */
+ if (nic->xdp_prog)
+ qid += nic->xdp_tx_queues;
+
snic = nic;
/* Get secondary Qset's SQ structure */
if (qid >= MAX_SND_QUEUES_PER_QS) {
@@ -1164,7 +1272,7 @@
/* Wait for pending IRQ handlers to finish */
for (irq = 0; irq < nic->num_vec; irq++)
- synchronize_irq(nic->msix_entries[irq].vector);
+ synchronize_irq(pci_irq_vector(nic->pdev, irq));
tasklet_kill(&nic->rbdr_task);
tasklet_kill(&nic->qs_err_task);
@@ -1365,7 +1473,7 @@
memcpy(netdev->dev_addr, addr->sa_data, netdev->addr_len);
- if (nic->msix_enabled) {
+ if (nic->pdev->msix_enabled) {
if (nicvf_hw_set_mac_addr(nic, netdev))
return -EBUSY;
} else {
@@ -1553,6 +1661,114 @@
return 0;
}
+static void nicvf_set_xdp_queues(struct nicvf *nic, bool bpf_attached)
+{
+ u8 cq_count, txq_count;
+
+ /* Set XDP Tx queue count same as Rx queue count */
+ if (!bpf_attached)
+ nic->xdp_tx_queues = 0;
+ else
+ nic->xdp_tx_queues = nic->rx_queues;
+
+ /* If queue count > MAX_CMP_QUEUES_PER_QS, then additional qsets
+ * needs to be allocated, check how many.
+ */
+ txq_count = nic->xdp_tx_queues + nic->tx_queues;
+ cq_count = max(nic->rx_queues, txq_count);
+ if (cq_count > MAX_CMP_QUEUES_PER_QS) {
+ nic->sqs_count = roundup(cq_count, MAX_CMP_QUEUES_PER_QS);
+ nic->sqs_count = (nic->sqs_count / MAX_CMP_QUEUES_PER_QS) - 1;
+ } else {
+ nic->sqs_count = 0;
+ }
+
+ /* Set primary Qset's resources */
+ nic->qs->rq_cnt = min_t(u8, nic->rx_queues, MAX_RCV_QUEUES_PER_QS);
+ nic->qs->sq_cnt = min_t(u8, txq_count, MAX_SND_QUEUES_PER_QS);
+ nic->qs->cq_cnt = max_t(u8, nic->qs->rq_cnt, nic->qs->sq_cnt);
+
+ /* Update stack */
+ nicvf_set_real_num_queues(nic->netdev, nic->tx_queues, nic->rx_queues);
+}
+
+static int nicvf_xdp_setup(struct nicvf *nic, struct bpf_prog *prog)
+{
+ struct net_device *dev = nic->netdev;
+ bool if_up = netif_running(nic->netdev);
+ struct bpf_prog *old_prog;
+ bool bpf_attached = false;
+
+ /* For now just support only the usual MTU sized frames */
+ if (prog && (dev->mtu > 1500)) {
+ netdev_warn(dev, "Jumbo frames not yet supported with XDP, current MTU %d.\n",
+ dev->mtu);
+ return -EOPNOTSUPP;
+ }
+
+ /* ALL SQs attached to CQs i.e same as RQs, are treated as
+ * XDP Tx queues and more Tx queues are allocated for
+ * network stack to send pkts out.
+ *
+ * No of Tx queues are either same as Rx queues or whatever
+ * is left in max no of queues possible.
+ */
+ if ((nic->rx_queues + nic->tx_queues) > nic->max_queues) {
+ netdev_warn(dev,
+ "Failed to attach BPF prog, RXQs + TXQs > Max %d\n",
+ nic->max_queues);
+ return -ENOMEM;
+ }
+
+ if (if_up)
+ nicvf_stop(nic->netdev);
+
+ old_prog = xchg(&nic->xdp_prog, prog);
+ /* Detach old prog, if any */
+ if (old_prog)
+ bpf_prog_put(old_prog);
+
+ if (nic->xdp_prog) {
+ /* Attach BPF program */
+ nic->xdp_prog = bpf_prog_add(nic->xdp_prog, nic->rx_queues - 1);
+ if (!IS_ERR(nic->xdp_prog))
+ bpf_attached = true;
+ }
+
+ /* Calculate Tx queues needed for XDP and network stack */
+ nicvf_set_xdp_queues(nic, bpf_attached);
+
+ if (if_up) {
+ /* Reinitialize interface, clean slate */
+ nicvf_open(nic->netdev);
+ netif_trans_update(nic->netdev);
+ }
+
+ return 0;
+}
+
+static int nicvf_xdp(struct net_device *netdev, struct netdev_xdp *xdp)
+{
+ struct nicvf *nic = netdev_priv(netdev);
+
+ /* To avoid checks while retrieving buffer address from CQE_RX,
+ * do not support XDP for T88 pass1.x silicons which are anyway
+ * not in use widely.
+ */
+ if (pass1_silicon(nic->pdev))
+ return -EOPNOTSUPP;
+
+ switch (xdp->command) {
+ case XDP_SETUP_PROG:
+ return nicvf_xdp_setup(nic, xdp->prog);
+ case XDP_QUERY_PROG:
+ xdp->prog_attached = !!nic->xdp_prog;
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
static const struct net_device_ops nicvf_netdev_ops = {
.ndo_open = nicvf_open,
.ndo_stop = nicvf_stop,
@@ -1563,6 +1779,7 @@
.ndo_tx_timeout = nicvf_tx_timeout,
.ndo_fix_features = nicvf_fix_features,
.ndo_set_features = nicvf_set_features,
+ .ndo_xdp = nicvf_xdp,
};
static int nicvf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
@@ -1665,8 +1882,9 @@
if (err)
goto err_unregister_interrupts;
- netdev->hw_features = (NETIF_F_RXCSUM | NETIF_F_IP_CSUM | NETIF_F_SG |
- NETIF_F_TSO | NETIF_F_GRO |
+ netdev->hw_features = (NETIF_F_RXCSUM | NETIF_F_SG |
+ NETIF_F_TSO | NETIF_F_GRO | NETIF_F_TSO6 |
+ NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |
NETIF_F_HW_VLAN_CTAG_RX);
netdev->hw_features |= NETIF_F_RXHASH;
@@ -1674,7 +1892,8 @@
netdev->features |= netdev->hw_features;
netdev->hw_features |= NETIF_F_LOOPBACK;
- netdev->vlan_features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_TSO;
+ netdev->vlan_features = NETIF_F_SG | NETIF_F_IP_CSUM |
+ NETIF_F_IPV6_CSUM | NETIF_F_TSO | NETIF_F_TSO6;
netdev->netdev_ops = &nicvf_netdev_ops;
netdev->watchdog_timeo = NICVF_TX_TIMEOUT;
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
index f13289f..2b18176 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c
@@ -19,16 +19,8 @@
#include "q_struct.h"
#include "nicvf_queues.h"
-#define NICVF_PAGE_ORDER ((PAGE_SIZE <= 4096) ? PAGE_ALLOC_COSTLY_ORDER : 0)
-
-static inline u64 nicvf_iova_to_phys(struct nicvf *nic, dma_addr_t dma_addr)
-{
- /* Translation is installed only when IOMMU is present */
- if (nic->iommu_domain)
- return iommu_iova_to_phys(nic->iommu_domain, dma_addr);
- return dma_addr;
-}
-
+static inline void nicvf_sq_add_gather_subdesc(struct snd_queue *sq, int qentry,
+ int size, u64 data);
static void nicvf_get_page(struct nicvf *nic)
{
if (!nic->rb_pageref || !nic->rb_page)
@@ -90,46 +82,152 @@
dmem->base = NULL;
}
-/* Allocate buffer for packet reception
- * HW returns memory address where packet is DMA'ed but not a pointer
- * into RBDR ring, so save buffer address at the start of fragment and
- * align the start address to a cache aligned address
- */
-static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, gfp_t gfp,
- u32 buf_len, u64 **rbuf)
-{
- int order = NICVF_PAGE_ORDER;
+#define XDP_PAGE_REFCNT_REFILL 256
- /* Check if request can be accomodated in previous allocated page */
- if (nic->rb_page &&
- ((nic->rb_page_offset + buf_len) < (PAGE_SIZE << order))) {
+/* Allocate a new page or recycle one if possible
+ *
+ * We cannot optimize dma mapping here, since
+ * 1. It's only one RBDR ring for 8 Rx queues.
+ * 2. CQE_RX gives address of the buffer where pkt has been DMA'ed
+ * and not idx into RBDR ring, so can't refer to saved info.
+ * 3. There are multiple receive buffers per page
+ */
+static inline struct pgcache *nicvf_alloc_page(struct nicvf *nic,
+ struct rbdr *rbdr, gfp_t gfp)
+{
+ int ref_count;
+ struct page *page = NULL;
+ struct pgcache *pgcache, *next;
+
+ /* Check if page is already allocated */
+ pgcache = &rbdr->pgcache[rbdr->pgidx];
+ page = pgcache->page;
+ /* Check if page can be recycled */
+ if (page) {
+ ref_count = page_ref_count(page);
+ /* Check if this page has been used once i.e 'put_page'
+ * called after packet transmission i.e internal ref_count
+ * and page's ref_count are equal i.e page can be recycled.
+ */
+ if (rbdr->is_xdp && (ref_count == pgcache->ref_count))
+ pgcache->ref_count--;
+ else
+ page = NULL;
+
+ /* In non-XDP mode, page's ref_count needs to be '1' for it
+ * to be recycled.
+ */
+ if (!rbdr->is_xdp && (ref_count != 1))
+ page = NULL;
+ }
+
+ if (!page) {
+ page = alloc_pages(gfp | __GFP_COMP | __GFP_NOWARN, 0);
+ if (!page)
+ return NULL;
+
+ this_cpu_inc(nic->pnicvf->drv_stats->page_alloc);
+
+ /* Check for space */
+ if (rbdr->pgalloc >= rbdr->pgcnt) {
+ /* Page can still be used */
+ nic->rb_page = page;
+ return NULL;
+ }
+
+ /* Save the page in page cache */
+ pgcache->page = page;
+ pgcache->dma_addr = 0;
+ pgcache->ref_count = 0;
+ rbdr->pgalloc++;
+ }
+
+ /* Take additional page references for recycling */
+ if (rbdr->is_xdp) {
+ /* Since there is single RBDR (i.e single core doing
+ * page recycling) per 8 Rx queues, in XDP mode adjusting
+ * page references atomically is the biggest bottleneck, so
+ * take bunch of references at a time.
+ *
+ * So here, below reference counts defer by '1'.
+ */
+ if (!pgcache->ref_count) {
+ pgcache->ref_count = XDP_PAGE_REFCNT_REFILL;
+ page_ref_add(page, XDP_PAGE_REFCNT_REFILL);
+ }
+ } else {
+ /* In non-XDP case, single 64K page is divided across multiple
+ * receive buffers, so cost of recycling is less anyway.
+ * So we can do with just one extra reference.
+ */
+ page_ref_add(page, 1);
+ }
+
+ rbdr->pgidx++;
+ rbdr->pgidx &= (rbdr->pgcnt - 1);
+
+ /* Prefetch refcount of next page in page cache */
+ next = &rbdr->pgcache[rbdr->pgidx];
+ page = next->page;
+ if (page)
+ prefetch(&page->_refcount);
+
+ return pgcache;
+}
+
+/* Allocate buffer for packet reception */
+static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, struct rbdr *rbdr,
+ gfp_t gfp, u32 buf_len, u64 *rbuf)
+{
+ struct pgcache *pgcache = NULL;
+
+ /* Check if request can be accomodated in previous allocated page.
+ * But in XDP mode only one buffer per page is permitted.
+ */
+ if (!rbdr->is_xdp && nic->rb_page &&
+ ((nic->rb_page_offset + buf_len) <= PAGE_SIZE)) {
nic->rb_pageref++;
goto ret;
}
nicvf_get_page(nic);
+ nic->rb_page = NULL;
- /* Allocate a new page */
- nic->rb_page = alloc_pages(gfp | __GFP_COMP | __GFP_NOWARN,
- order);
- if (!nic->rb_page) {
+ /* Get new page, either recycled or new one */
+ pgcache = nicvf_alloc_page(nic, rbdr, gfp);
+ if (!pgcache && !nic->rb_page) {
this_cpu_inc(nic->pnicvf->drv_stats->rcv_buffer_alloc_failures);
return -ENOMEM;
}
+
nic->rb_page_offset = 0;
+
+ /* Reserve space for header modifications by BPF program */
+ if (rbdr->is_xdp)
+ buf_len += XDP_PACKET_HEADROOM;
+
+ /* Check if it's recycled */
+ if (pgcache)
+ nic->rb_page = pgcache->page;
ret:
- /* HW will ensure data coherency, CPU sync not required */
- *rbuf = (u64 *)((u64)dma_map_page_attrs(&nic->pdev->dev, nic->rb_page,
+ if (rbdr->is_xdp && pgcache && pgcache->dma_addr) {
+ *rbuf = pgcache->dma_addr;
+ } else {
+ /* HW will ensure data coherency, CPU sync not required */
+ *rbuf = (u64)dma_map_page_attrs(&nic->pdev->dev, nic->rb_page,
nic->rb_page_offset, buf_len,
DMA_FROM_DEVICE,
- DMA_ATTR_SKIP_CPU_SYNC));
- if (dma_mapping_error(&nic->pdev->dev, (dma_addr_t)*rbuf)) {
- if (!nic->rb_page_offset)
- __free_pages(nic->rb_page, order);
- nic->rb_page = NULL;
- return -ENOMEM;
+ DMA_ATTR_SKIP_CPU_SYNC);
+ if (dma_mapping_error(&nic->pdev->dev, (dma_addr_t)*rbuf)) {
+ if (!nic->rb_page_offset)
+ __free_pages(nic->rb_page, 0);
+ nic->rb_page = NULL;
+ return -ENOMEM;
+ }
+ if (pgcache)
+ pgcache->dma_addr = *rbuf + XDP_PACKET_HEADROOM;
+ nic->rb_page_offset += buf_len;
}
- nic->rb_page_offset += buf_len;
return 0;
}
@@ -159,7 +257,7 @@
int ring_len, int buf_size)
{
int idx;
- u64 *rbuf;
+ u64 rbuf;
struct rbdr_entry_t *desc;
int err;
@@ -177,10 +275,34 @@
rbdr->head = 0;
rbdr->tail = 0;
+ /* Initialize page recycling stuff.
+ *
+ * Can't use single buffer per page especially with 64K pages.
+ * On embedded platforms i.e 81xx/83xx available memory itself
+ * is low and minimum ring size of RBDR is 8K, that takes away
+ * lots of memory.
+ *
+ * But for XDP it has to be a single buffer per page.
+ */
+ if (!nic->pnicvf->xdp_prog) {
+ rbdr->pgcnt = ring_len / (PAGE_SIZE / buf_size);
+ rbdr->is_xdp = false;
+ } else {
+ rbdr->pgcnt = ring_len;
+ rbdr->is_xdp = true;
+ }
+ rbdr->pgcnt = roundup_pow_of_two(rbdr->pgcnt);
+ rbdr->pgcache = kzalloc(sizeof(*rbdr->pgcache) *
+ rbdr->pgcnt, GFP_KERNEL);
+ if (!rbdr->pgcache)
+ return -ENOMEM;
+ rbdr->pgidx = 0;
+ rbdr->pgalloc = 0;
+
nic->rb_page = NULL;
for (idx = 0; idx < ring_len; idx++) {
- err = nicvf_alloc_rcv_buffer(nic, GFP_KERNEL, RCV_FRAG_LEN,
- &rbuf);
+ err = nicvf_alloc_rcv_buffer(nic, rbdr, GFP_KERNEL,
+ RCV_FRAG_LEN, &rbuf);
if (err) {
/* To free already allocated and mapped ones */
rbdr->tail = idx - 1;
@@ -188,7 +310,7 @@
}
desc = GET_RBDR_DESC(rbdr, idx);
- desc->buf_addr = (u64)rbuf >> NICVF_RCV_BUF_ALIGN;
+ desc->buf_addr = rbuf & ~(NICVF_RCV_BUF_ALIGN_BYTES - 1);
}
nicvf_get_page(nic);
@@ -201,6 +323,7 @@
{
int head, tail;
u64 buf_addr, phys_addr;
+ struct pgcache *pgcache;
struct rbdr_entry_t *desc;
if (!rbdr)
@@ -216,7 +339,7 @@
/* Release page references */
while (head != tail) {
desc = GET_RBDR_DESC(rbdr, head);
- buf_addr = ((u64)desc->buf_addr) << NICVF_RCV_BUF_ALIGN;
+ buf_addr = desc->buf_addr;
phys_addr = nicvf_iova_to_phys(nic, buf_addr);
dma_unmap_page_attrs(&nic->pdev->dev, buf_addr, RCV_FRAG_LEN,
DMA_FROM_DEVICE, DMA_ATTR_SKIP_CPU_SYNC);
@@ -227,13 +350,31 @@
}
/* Release buffer of tail desc */
desc = GET_RBDR_DESC(rbdr, tail);
- buf_addr = ((u64)desc->buf_addr) << NICVF_RCV_BUF_ALIGN;
+ buf_addr = desc->buf_addr;
phys_addr = nicvf_iova_to_phys(nic, buf_addr);
dma_unmap_page_attrs(&nic->pdev->dev, buf_addr, RCV_FRAG_LEN,
DMA_FROM_DEVICE, DMA_ATTR_SKIP_CPU_SYNC);
if (phys_addr)
put_page(virt_to_page(phys_to_virt(phys_addr)));
+ /* Sync page cache info */
+ smp_rmb();
+
+ /* Release additional page references held for recycling */
+ head = 0;
+ while (head < rbdr->pgcnt) {
+ pgcache = &rbdr->pgcache[head];
+ if (pgcache->page && page_ref_count(pgcache->page) != 0) {
+ if (!rbdr->is_xdp) {
+ put_page(pgcache->page);
+ continue;
+ }
+ page_ref_sub(pgcache->page, pgcache->ref_count - 1);
+ put_page(pgcache->page);
+ }
+ head++;
+ }
+
/* Free RBDR ring */
nicvf_free_q_desc_mem(nic, &rbdr->dmem);
}
@@ -248,7 +389,7 @@
int refill_rb_cnt;
struct rbdr *rbdr;
struct rbdr_entry_t *desc;
- u64 *rbuf;
+ u64 rbuf;
int new_rb = 0;
refill:
@@ -269,17 +410,20 @@
else
refill_rb_cnt = qs->rbdr_len - qcount - 1;
+ /* Sync page cache info */
+ smp_rmb();
+
/* Start filling descs from tail */
tail = nicvf_queue_reg_read(nic, NIC_QSET_RBDR_0_1_TAIL, rbdr_idx) >> 3;
while (refill_rb_cnt) {
tail++;
tail &= (rbdr->dmem.q_len - 1);
- if (nicvf_alloc_rcv_buffer(nic, gfp, RCV_FRAG_LEN, &rbuf))
+ if (nicvf_alloc_rcv_buffer(nic, rbdr, gfp, RCV_FRAG_LEN, &rbuf))
break;
desc = GET_RBDR_DESC(rbdr, tail);
- desc->buf_addr = (u64)rbuf >> NICVF_RCV_BUF_ALIGN;
+ desc->buf_addr = rbuf & ~(NICVF_RCV_BUF_ALIGN_BYTES - 1);
refill_rb_cnt--;
new_rb++;
}
@@ -362,7 +506,7 @@
/* Initialize transmit queue */
static int nicvf_init_snd_queue(struct nicvf *nic,
- struct snd_queue *sq, int q_len)
+ struct snd_queue *sq, int q_len, int qidx)
{
int err;
@@ -375,17 +519,38 @@
sq->skbuff = kcalloc(q_len, sizeof(u64), GFP_KERNEL);
if (!sq->skbuff)
return -ENOMEM;
+
sq->head = 0;
sq->tail = 0;
- atomic_set(&sq->free_cnt, q_len - 1);
sq->thresh = SND_QUEUE_THRESH;
- /* Preallocate memory for TSO segment's header */
- sq->tso_hdrs = dma_alloc_coherent(&nic->pdev->dev,
- q_len * TSO_HEADER_SIZE,
- &sq->tso_hdrs_phys, GFP_KERNEL);
- if (!sq->tso_hdrs)
- return -ENOMEM;
+ /* Check if this SQ is a XDP TX queue */
+ if (nic->sqs_mode)
+ qidx += ((nic->sqs_id + 1) * MAX_SND_QUEUES_PER_QS);
+ if (qidx < nic->pnicvf->xdp_tx_queues) {
+ /* Alloc memory to save page pointers for XDP_TX */
+ sq->xdp_page = kcalloc(q_len, sizeof(u64), GFP_KERNEL);
+ if (!sq->xdp_page)
+ return -ENOMEM;
+ sq->xdp_desc_cnt = 0;
+ sq->xdp_free_cnt = q_len - 1;
+ sq->is_xdp = true;
+ } else {
+ sq->xdp_page = NULL;
+ sq->xdp_desc_cnt = 0;
+ sq->xdp_free_cnt = 0;
+ sq->is_xdp = false;
+
+ atomic_set(&sq->free_cnt, q_len - 1);
+
+ /* Preallocate memory for TSO segment's header */
+ sq->tso_hdrs = dma_alloc_coherent(&nic->pdev->dev,
+ q_len * TSO_HEADER_SIZE,
+ &sq->tso_hdrs_phys,
+ GFP_KERNEL);
+ if (!sq->tso_hdrs)
+ return -ENOMEM;
+ }
return 0;
}
@@ -411,6 +576,7 @@
static void nicvf_free_snd_queue(struct nicvf *nic, struct snd_queue *sq)
{
struct sk_buff *skb;
+ struct page *page;
struct sq_hdr_subdesc *hdr;
struct sq_hdr_subdesc *tso_sqe;
@@ -428,8 +594,15 @@
smp_rmb();
while (sq->head != sq->tail) {
skb = (struct sk_buff *)sq->skbuff[sq->head];
- if (!skb)
+ if (!skb || !sq->xdp_page)
goto next;
+
+ page = (struct page *)sq->xdp_page[sq->head];
+ if (!page)
+ goto next;
+ else
+ put_page(page);
+
hdr = (struct sq_hdr_subdesc *)GET_SQ_DESC(sq, sq->head);
/* Check for dummy descriptor used for HW TSO offload on 88xx */
if (hdr->dont_send) {
@@ -442,12 +615,14 @@
nicvf_unmap_sndq_buffers(nic, sq, sq->head,
hdr->subdesc_cnt);
}
- dev_kfree_skb_any(skb);
+ if (skb)
+ dev_kfree_skb_any(skb);
next:
sq->head++;
sq->head &= (sq->dmem.q_len - 1);
}
kfree(sq->skbuff);
+ kfree(sq->xdp_page);
nicvf_free_q_desc_mem(nic, &sq->dmem);
}
@@ -838,7 +1013,7 @@
/* Alloc send queue */
for (qidx = 0; qidx < qs->sq_cnt; qidx++) {
- if (nicvf_init_snd_queue(nic, &qs->sq[qidx], qs->sq_len))
+ if (nicvf_init_snd_queue(nic, &qs->sq[qidx], qs->sq_len, qidx))
goto alloc_fail;
}
@@ -876,6 +1051,7 @@
nic->rx_queues = qs->rq_cnt;
nic->tx_queues = qs->sq_cnt;
+ nic->xdp_tx_queues = 0;
return 0;
}
@@ -940,7 +1116,10 @@
int qentry;
qentry = sq->tail;
- atomic_sub(desc_cnt, &sq->free_cnt);
+ if (!sq->is_xdp)
+ atomic_sub(desc_cnt, &sq->free_cnt);
+ else
+ sq->xdp_free_cnt -= desc_cnt;
sq->tail += desc_cnt;
sq->tail &= (sq->dmem.q_len - 1);
@@ -958,7 +1137,10 @@
/* Free descriptor back to SQ for future use */
void nicvf_put_sq_desc(struct snd_queue *sq, int desc_cnt)
{
- atomic_add(desc_cnt, &sq->free_cnt);
+ if (!sq->is_xdp)
+ atomic_add(desc_cnt, &sq->free_cnt);
+ else
+ sq->xdp_free_cnt += desc_cnt;
sq->head += desc_cnt;
sq->head &= (sq->dmem.q_len - 1);
}
@@ -1016,6 +1198,58 @@
}
}
+/* XDP Transmit APIs */
+void nicvf_xdp_sq_doorbell(struct nicvf *nic,
+ struct snd_queue *sq, int sq_num)
+{
+ if (!sq->xdp_desc_cnt)
+ return;
+
+ /* make sure all memory stores are done before ringing doorbell */
+ wmb();
+
+ /* Inform HW to xmit all TSO segments */
+ nicvf_queue_reg_write(nic, NIC_QSET_SQ_0_7_DOOR,
+ sq_num, sq->xdp_desc_cnt);
+ sq->xdp_desc_cnt = 0;
+}
+
+static inline void
+nicvf_xdp_sq_add_hdr_subdesc(struct snd_queue *sq, int qentry,
+ int subdesc_cnt, u64 data, int len)
+{
+ struct sq_hdr_subdesc *hdr;
+
+ hdr = (struct sq_hdr_subdesc *)GET_SQ_DESC(sq, qentry);
+ memset(hdr, 0, SND_QUEUE_DESC_SIZE);
+ hdr->subdesc_type = SQ_DESC_TYPE_HEADER;
+ hdr->subdesc_cnt = subdesc_cnt;
+ hdr->tot_len = len;
+ hdr->post_cqe = 1;
+ sq->xdp_page[qentry] = (u64)virt_to_page((void *)data);
+}
+
+int nicvf_xdp_sq_append_pkt(struct nicvf *nic, struct snd_queue *sq,
+ u64 bufaddr, u64 dma_addr, u16 len)
+{
+ int subdesc_cnt = MIN_SQ_DESC_PER_PKT_XMIT;
+ int qentry;
+
+ if (subdesc_cnt > sq->xdp_free_cnt)
+ return 0;
+
+ qentry = nicvf_get_sq_desc(sq, subdesc_cnt);
+
+ nicvf_xdp_sq_add_hdr_subdesc(sq, qentry, subdesc_cnt - 1, bufaddr, len);
+
+ qentry = nicvf_get_nxt_sqentry(sq, qentry);
+ nicvf_sq_add_gather_subdesc(sq, qentry, len, dma_addr);
+
+ sq->xdp_desc_cnt += subdesc_cnt;
+
+ return 1;
+}
+
/* Calculate no of SQ subdescriptors needed to transmit all
* segments of this TSO packet.
* Taken from 'Tilera network driver' with a minor modification.
@@ -1094,7 +1328,13 @@
{
int proto;
struct sq_hdr_subdesc *hdr;
+ union {
+ struct iphdr *v4;
+ struct ipv6hdr *v6;
+ unsigned char *hdr;
+ } ip;
+ ip.hdr = skb_network_header(skb);
hdr = (struct sq_hdr_subdesc *)GET_SQ_DESC(sq, qentry);
memset(hdr, 0, SND_QUEUE_DESC_SIZE);
hdr->subdesc_type = SQ_DESC_TYPE_HEADER;
@@ -1119,7 +1359,9 @@
hdr->l3_offset = skb_network_offset(skb);
hdr->l4_offset = skb_transport_offset(skb);
- proto = ip_hdr(skb)->protocol;
+ proto = (ip.v4->version == 4) ? ip.v4->protocol :
+ ip.v6->nexthdr;
+
switch (proto) {
case IPPROTO_TCP:
hdr->csum_l4 = SEND_L4_CSUM_TCP;
@@ -1366,8 +1608,33 @@
#endif
}
+static void nicvf_unmap_rcv_buffer(struct nicvf *nic, u64 dma_addr,
+ u64 buf_addr, bool xdp)
+{
+ struct page *page = NULL;
+ int len = RCV_FRAG_LEN;
+
+ if (xdp) {
+ page = virt_to_page(phys_to_virt(buf_addr));
+ /* Check if it's a recycled page, if not
+ * unmap the DMA mapping.
+ *
+ * Recycled page holds an extra reference.
+ */
+ if (page_ref_count(page) != 1)
+ return;
+
+ len += XDP_PACKET_HEADROOM;
+ /* Receive buffers in XDP mode are mapped from page start */
+ dma_addr &= PAGE_MASK;
+ }
+ dma_unmap_page_attrs(&nic->pdev->dev, dma_addr, len,
+ DMA_FROM_DEVICE, DMA_ATTR_SKIP_CPU_SYNC);
+}
+
/* Returns SKB for a received packet */
-struct sk_buff *nicvf_get_rcv_skb(struct nicvf *nic, struct cqe_rx_t *cqe_rx)
+struct sk_buff *nicvf_get_rcv_skb(struct nicvf *nic,
+ struct cqe_rx_t *cqe_rx, bool xdp)
{
int frag;
int payload_len = 0;
@@ -1402,10 +1669,9 @@
if (!frag) {
/* First fragment */
- dma_unmap_page_attrs(&nic->pdev->dev,
- *rb_ptrs - cqe_rx->align_pad,
- RCV_FRAG_LEN, DMA_FROM_DEVICE,
- DMA_ATTR_SKIP_CPU_SYNC);
+ nicvf_unmap_rcv_buffer(nic,
+ *rb_ptrs - cqe_rx->align_pad,
+ phys_addr, xdp);
skb = nicvf_rb_ptr_to_skb(nic,
phys_addr - cqe_rx->align_pad,
payload_len);
@@ -1415,9 +1681,7 @@
skb_put(skb, payload_len);
} else {
/* Add fragments */
- dma_unmap_page_attrs(&nic->pdev->dev, *rb_ptrs,
- RCV_FRAG_LEN, DMA_FROM_DEVICE,
- DMA_ATTR_SKIP_CPU_SYNC);
+ nicvf_unmap_rcv_buffer(nic, *rb_ptrs, phys_addr, xdp);
page = virt_to_page(phys_to_virt(phys_addr));
offset = phys_to_virt(phys_addr) - page_address(page);
skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, page,
@@ -1547,9 +1811,6 @@
/* Check for errors in the receive cmp.queue entry */
int nicvf_check_cqe_rx_errs(struct nicvf *nic, struct cqe_rx_t *cqe_rx)
{
- if (!cqe_rx->err_level && !cqe_rx->err_opcode)
- return 0;
-
if (netif_msg_rx_err(nic))
netdev_err(nic->netdev,
"%s: RX error CQE err_level 0x%x err_opcode 0x%x\n",
@@ -1638,8 +1899,6 @@
int nicvf_check_cqe_tx_errs(struct nicvf *nic, struct cqe_send_t *cqe_tx)
{
switch (cqe_tx->send_status) {
- case CQ_TX_ERROP_GOOD:
- return 0;
case CQ_TX_ERROP_DESC_FAULT:
this_cpu_inc(nic->drv_stats->tx_desc_fault);
break;
diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
index 10cb4b8..5785852 100644
--- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
+++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h
@@ -10,6 +10,7 @@
#define NICVF_QUEUES_H
#include <linux/netdevice.h>
+#include <linux/iommu.h>
#include "q_struct.h"
#define MAX_QUEUE_SET 128
@@ -213,6 +214,12 @@
void *unalign_base;
};
+struct pgcache {
+ struct page *page;
+ int ref_count;
+ u64 dma_addr;
+};
+
struct rbdr {
bool enable;
u32 dma_size;
@@ -222,6 +229,13 @@
u32 head;
u32 tail;
struct q_desc_mem dmem;
+ bool is_xdp;
+
+ /* For page recycling */
+ int pgidx;
+ int pgcnt;
+ int pgalloc;
+ struct pgcache *pgcache;
} ____cacheline_aligned_in_smp;
struct rcv_queue {
@@ -258,6 +272,10 @@
u32 tail;
u64 *skbuff;
void *desc;
+ u64 *xdp_page;
+ u16 xdp_desc_cnt;
+ u16 xdp_free_cnt;
+ bool is_xdp;
#define TSO_HEADER_SIZE 128
/* For TSO segment's header */
@@ -301,6 +319,14 @@
#define CQ_ERR_MASK (CQ_WR_FULL | CQ_WR_DISABLE | CQ_WR_FAULT)
+static inline u64 nicvf_iova_to_phys(struct nicvf *nic, dma_addr_t dma_addr)
+{
+ /* Translation is installed only when IOMMU is present */
+ if (nic->iommu_domain)
+ return iommu_iova_to_phys(nic->iommu_domain, dma_addr);
+ return dma_addr;
+}
+
void nicvf_unmap_sndq_buffers(struct nicvf *nic, struct snd_queue *sq,
int hdr_sqe, u8 subdesc_cnt);
void nicvf_config_vlan_stripping(struct nicvf *nic,
@@ -318,8 +344,12 @@
struct snd_queue *sq, int qidx);
int nicvf_sq_append_skb(struct nicvf *nic, struct snd_queue *sq,
struct sk_buff *skb, u8 sq_num);
+int nicvf_xdp_sq_append_pkt(struct nicvf *nic, struct snd_queue *sq,
+ u64 bufaddr, u64 dma_addr, u16 len);
+void nicvf_xdp_sq_doorbell(struct nicvf *nic, struct snd_queue *sq, int sq_num);
-struct sk_buff *nicvf_get_rcv_skb(struct nicvf *nic, struct cqe_rx_t *cqe_rx);
+struct sk_buff *nicvf_get_rcv_skb(struct nicvf *nic,
+ struct cqe_rx_t *cqe_rx, bool xdp);
void nicvf_rbdr_task(unsigned long data);
void nicvf_rbdr_work(struct work_struct *work);
diff --git a/drivers/net/ethernet/cavium/thunder/q_struct.h b/drivers/net/ethernet/cavium/thunder/q_struct.h
index f363472..e47205a 100644
--- a/drivers/net/ethernet/cavium/thunder/q_struct.h
+++ b/drivers/net/ethernet/cavium/thunder/q_struct.h
@@ -359,15 +359,7 @@
};
struct rbdr_entry_t {
-#if defined(__BIG_ENDIAN_BITFIELD)
- u64 rsvd0:15;
- u64 buf_addr:42;
- u64 cache_align:7;
-#elif defined(__LITTLE_ENDIAN_BITFIELD)
- u64 cache_align:7;
- u64 buf_addr:42;
- u64 rsvd0:15;
-#endif
+ u64 buf_addr;
};
/* TCP reassembly context */
diff --git a/drivers/net/ethernet/chelsio/cxgb/common.h b/drivers/net/ethernet/chelsio/cxgb/common.h
index 6916c62..94b9482f 100644
--- a/drivers/net/ethernet/chelsio/cxgb/common.h
+++ b/drivers/net/ethernet/chelsio/cxgb/common.h
@@ -223,7 +223,6 @@
struct cmac *mac;
struct cphy *phy;
struct link_config link_config;
- struct net_device_stats netstats;
};
struct sge;
diff --git a/drivers/net/ethernet/chelsio/cxgb/cxgb2.c b/drivers/net/ethernet/chelsio/cxgb/cxgb2.c
index d8aff7a..8623be1 100644
--- a/drivers/net/ethernet/chelsio/cxgb/cxgb2.c
+++ b/drivers/net/ethernet/chelsio/cxgb/cxgb2.c
@@ -296,7 +296,7 @@
{
struct adapter *adapter = dev->ml_priv;
struct port_info *p = &adapter->port[dev->if_port];
- struct net_device_stats *ns = &p->netstats;
+ struct net_device_stats *ns = &dev->stats;
const struct cmac_statistics *pstats;
/* Do a full update of the MAC stats */
diff --git a/drivers/net/ethernet/chelsio/cxgb3/adapter.h b/drivers/net/ethernet/chelsio/cxgb3/adapter.h
index 8b395b5..087ff0f 100644
--- a/drivers/net/ethernet/chelsio/cxgb3/adapter.h
+++ b/drivers/net/ethernet/chelsio/cxgb3/adapter.h
@@ -72,7 +72,6 @@
struct cphy phy;
struct cmac mac;
struct link_config link_config;
- struct net_device_stats netstats;
int activity;
__be32 iscsi_ipv4addr;
struct iscsi_config iscsic;
diff --git a/drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c b/drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c
index d764916..2ff6bd1 100644
--- a/drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c
+++ b/drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c
@@ -1489,7 +1489,7 @@
{
struct port_info *pi = netdev_priv(dev);
struct adapter *adapter = pi->adapter;
- struct net_device_stats *ns = &pi->netstats;
+ struct net_device_stats *ns = &dev->stats;
const struct mac_stats *pstats;
spin_lock(&adapter->stats_lock);
diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
index afb0967..c12c4a3 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
+++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
@@ -2338,6 +2338,10 @@
f->locked = 1;
f->fs.rpttid = 1;
+ /* Save the actual tid. We need this to get the corresponding
+ * filter entry structure in filter_rpl.
+ */
+ f->tid = stid + adap->tids.ftid_base;
ret = set_filter_wr(adap, stid);
if (ret) {
clear_filter(adap, f);
@@ -3809,6 +3813,15 @@
}
if (caps_cmd.cryptocaps) {
/* Should query params here...TODO */
+ params[0] = FW_PARAM_PFVF(NCRYPTO_LOOKASIDE);
+ ret = t4_query_params(adap, adap->mbox, adap->pf, 0, 2,
+ params, val);
+ if (ret < 0) {
+ if (ret != -EINVAL)
+ goto bye;
+ } else {
+ adap->vres.ncrypto_fc = val[0];
+ }
adap->params.crypto |= ULP_CRYPTO_LOOKASIDE;
adap->num_uld += 1;
}
diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h
index 4c85660..6e74040 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h
+++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h
@@ -272,6 +272,7 @@
struct cxgb4_range qp;
struct cxgb4_range cq;
struct cxgb4_range ocq;
+ unsigned int ncrypto_fc;
};
#define OCQ_WIN_OFFSET(pdev, vres) \
diff --git a/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c b/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c
index 87000cd..0de8eb7 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c
+++ b/drivers/net/ethernet/chelsio/cxgb4/t4_hw.c
@@ -6369,7 +6369,6 @@
unsigned int stat_len = cache_line_size > 64 ? 128 : 64;
unsigned int fl_align = cache_line_size < 32 ? 32 : cache_line_size;
unsigned int fl_align_log = fls(fl_align) - 1;
- unsigned int ingpad;
t4_write_reg(adap, SGE_HOST_PAGE_SIZE_A,
HOSTPAGESIZEPF0_V(sge_hps) |
@@ -6389,6 +6388,10 @@
INGPADBOUNDARY_SHIFT_X) |
EGRSTATUSPAGESIZE_V(stat_len != 64));
} else {
+ unsigned int pack_align;
+ unsigned int ingpad, ingpack;
+ unsigned int pcie_cap;
+
/* T5 introduced the separation of the Free List Padding and
* Packing Boundaries. Thus, we can select a smaller Padding
* Boundary to avoid uselessly chewing up PCIe Link and Memory
@@ -6401,27 +6404,62 @@
* Size (the minimum unit of transfer to/from Memory). If we
* have a Padding Boundary which is smaller than the Memory
* Line Size, that'll involve a Read-Modify-Write cycle on the
- * Memory Controller which is never good. For T5 the smallest
- * Padding Boundary which we can select is 32 bytes which is
- * larger than any known Memory Controller Line Size so we'll
- * use that.
- *
- * T5 has a different interpretation of the "0" value for the
- * Packing Boundary. This corresponds to 16 bytes instead of
- * the expected 32 bytes. We never have a Packing Boundary
- * less than 32 bytes so we can't use that special value but
- * on the other hand, if we wanted 32 bytes, the best we can
- * really do is 64 bytes.
- */
- if (fl_align <= 32) {
- fl_align = 64;
- fl_align_log = 6;
+ * Memory Controller which is never good.
+ */
+
+ /* We want the Packing Boundary to be based on the Cache Line
+ * Size in order to help avoid False Sharing performance
+ * issues between CPUs, etc. We also want the Packing
+ * Boundary to incorporate the PCI-E Maximum Payload Size. We
+ * get best performance when the Packing Boundary is a
+ * multiple of the Maximum Payload Size.
+ */
+ pack_align = fl_align;
+ pcie_cap = pci_find_capability(adap->pdev, PCI_CAP_ID_EXP);
+ if (pcie_cap) {
+ unsigned int mps, mps_log;
+ u16 devctl;
+
+ /* The PCIe Device Control Maximum Payload Size field
+ * [bits 7:5] encodes sizes as powers of 2 starting at
+ * 128 bytes.
+ */
+ pci_read_config_word(adap->pdev,
+ pcie_cap + PCI_EXP_DEVCTL,
+ &devctl);
+ mps_log = ((devctl & PCI_EXP_DEVCTL_PAYLOAD) >> 5) + 7;
+ mps = 1 << mps_log;
+ if (mps > pack_align)
+ pack_align = mps;
}
+ /* N.B. T5/T6 have a crazy special interpretation of the "0"
+ * value for the Packing Boundary. This corresponds to 16
+ * bytes instead of the expected 32 bytes. So if we want 32
+ * bytes, the best we can really do is 64 bytes ...
+ */
+ if (pack_align <= 16) {
+ ingpack = INGPACKBOUNDARY_16B_X;
+ fl_align = 16;
+ } else if (pack_align == 32) {
+ ingpack = INGPACKBOUNDARY_64B_X;
+ fl_align = 64;
+ } else {
+ unsigned int pack_align_log = fls(pack_align) - 1;
+
+ ingpack = pack_align_log - INGPACKBOUNDARY_SHIFT_X;
+ fl_align = pack_align;
+ }
+
+ /* Use the smallest Ingress Padding which isn't smaller than
+ * the Memory Controller Read/Write Size. We'll take that as
+ * being 8 bytes since we don't know of any system with a
+ * wider Memory Controller Bus Width.
+ */
if (is_t5(adap->params.chip))
- ingpad = INGPCIEBOUNDARY_32B_X;
+ ingpad = INGPADBOUNDARY_32B_X;
else
- ingpad = T6_INGPADBOUNDARY_32B_X;
+ ingpad = T6_INGPADBOUNDARY_8B_X;
t4_set_reg_field(adap, SGE_CONTROL_A,
INGPADBOUNDARY_V(INGPADBOUNDARY_M) |
@@ -6430,8 +6468,7 @@
EGRSTATUSPAGESIZE_V(stat_len != 64));
t4_set_reg_field(adap, SGE_CONTROL2_A,
INGPACKBOUNDARY_V(INGPACKBOUNDARY_M),
- INGPACKBOUNDARY_V(fl_align_log -
- INGPACKBOUNDARY_SHIFT_X));
+ INGPACKBOUNDARY_V(ingpack));
}
/*
* Adjust various SGE Free List Host Buffer Sizes.
diff --git a/drivers/net/ethernet/chelsio/cxgb4/t4_values.h b/drivers/net/ethernet/chelsio/cxgb4/t4_values.h
index 36cf307..f6558cb 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/t4_values.h
+++ b/drivers/net/ethernet/chelsio/cxgb4/t4_values.h
@@ -54,11 +54,15 @@
#define INGPADBOUNDARY_SHIFT_X 5
#define T6_INGPADBOUNDARY_SHIFT_X 3
+#define T6_INGPADBOUNDARY_8B_X 0
#define T6_INGPADBOUNDARY_32B_X 2
+#define INGPADBOUNDARY_32B_X 0
+
/* CONTROL2 register */
#define INGPACKBOUNDARY_SHIFT_X 5
#define INGPACKBOUNDARY_16B_X 0
+#define INGPACKBOUNDARY_64B_X 1
/* GTS register */
#define SGE_TIMERREGS 6
diff --git a/drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h b/drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h
index ccc05f8..8f8c079 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h
+++ b/drivers/net/ethernet/chelsio/cxgb4/t4fw_api.h
@@ -1167,7 +1167,8 @@
FW_PARAMS_PARAM_PFVF_ACTIVE_FILTER_START = 0x2D,
FW_PARAMS_PARAM_PFVF_ACTIVE_FILTER_END = 0x2E,
FW_PARAMS_PARAM_PFVF_ETHOFLD_END = 0x30,
- FW_PARAMS_PARAM_PFVF_CPLFW4MSG_ENCAP = 0x31
+ FW_PARAMS_PARAM_PFVF_CPLFW4MSG_ENCAP = 0x31,
+ FW_PARAMS_PARAM_PFVF_NCRYPTO_LOOKASIDE = 0x32
};
/*
diff --git a/drivers/net/ethernet/cirrus/cs89x0.c b/drivers/net/ethernet/cirrus/cs89x0.c
index 3647b28..47384f7 100644
--- a/drivers/net/ethernet/cirrus/cs89x0.c
+++ b/drivers/net/ethernet/cirrus/cs89x0.c
@@ -1896,7 +1896,7 @@
return 0;
}
-static const struct __maybe_unused of_device_id cs89x0_match[] = {
+static const struct of_device_id __maybe_unused cs89x0_match[] = {
{ .compatible = "cirrus,cs8900", },
{ .compatible = "cirrus,cs8920", },
{ },
diff --git a/drivers/net/ethernet/dec/tulip/de2104x.c b/drivers/net/ethernet/dec/tulip/de2104x.c
index 127ce970..91b8f6f 100644
--- a/drivers/net/ethernet/dec/tulip/de2104x.c
+++ b/drivers/net/ethernet/dec/tulip/de2104x.c
@@ -312,8 +312,6 @@
u32 msg_enable;
- struct net_device_stats net_stats;
-
struct pci_dev *pdev;
u16 setup_frame[DE_SETUP_FRAME_WORDS];
@@ -388,14 +386,14 @@
netif_warn(de, rx_err, de->dev,
"Oversized Ethernet frame spanned multiple buffers, status %08x!\n",
status);
- de->net_stats.rx_length_errors++;
+ de->dev->stats.rx_length_errors++;
}
} else if (status & RxError) {
/* There was a fatal error. */
- de->net_stats.rx_errors++; /* end of a packet.*/
- if (status & 0x0890) de->net_stats.rx_length_errors++;
- if (status & RxErrCRC) de->net_stats.rx_crc_errors++;
- if (status & RxErrFIFO) de->net_stats.rx_fifo_errors++;
+ de->dev->stats.rx_errors++; /* end of a packet.*/
+ if (status & 0x0890) de->dev->stats.rx_length_errors++;
+ if (status & RxErrCRC) de->dev->stats.rx_crc_errors++;
+ if (status & RxErrFIFO) de->dev->stats.rx_fifo_errors++;
}
}
@@ -423,7 +421,7 @@
mapping = de->rx_skb[rx_tail].mapping;
if (unlikely(drop)) {
- de->net_stats.rx_dropped++;
+ de->dev->stats.rx_dropped++;
goto rx_next;
}
@@ -441,7 +439,7 @@
buflen = copying_skb ? (len + RX_OFFSET) : de->rx_buf_sz;
copy_skb = netdev_alloc_skb(de->dev, buflen);
if (unlikely(!copy_skb)) {
- de->net_stats.rx_dropped++;
+ de->dev->stats.rx_dropped++;
drop = 1;
rx_work = 100;
goto rx_next;
@@ -470,8 +468,8 @@
skb->protocol = eth_type_trans (skb, de->dev);
- de->net_stats.rx_packets++;
- de->net_stats.rx_bytes += skb->len;
+ de->dev->stats.rx_packets++;
+ de->dev->stats.rx_bytes += skb->len;
rc = netif_rx (skb);
if (rc == NET_RX_DROP)
drop = 1;
@@ -572,18 +570,18 @@
netif_dbg(de, tx_err, de->dev,
"tx err, status 0x%x\n",
status);
- de->net_stats.tx_errors++;
+ de->dev->stats.tx_errors++;
if (status & TxOWC)
- de->net_stats.tx_window_errors++;
+ de->dev->stats.tx_window_errors++;
if (status & TxMaxCol)
- de->net_stats.tx_aborted_errors++;
+ de->dev->stats.tx_aborted_errors++;
if (status & TxLinkFail)
- de->net_stats.tx_carrier_errors++;
+ de->dev->stats.tx_carrier_errors++;
if (status & TxFIFOUnder)
- de->net_stats.tx_fifo_errors++;
+ de->dev->stats.tx_fifo_errors++;
} else {
- de->net_stats.tx_packets++;
- de->net_stats.tx_bytes += skb->len;
+ de->dev->stats.tx_packets++;
+ de->dev->stats.tx_bytes += skb->len;
netif_dbg(de, tx_done, de->dev,
"tx done, slot %d\n", tx_tail);
}
@@ -814,9 +812,9 @@
static inline void de_rx_missed(struct de_private *de, u32 rx_missed)
{
if (unlikely(rx_missed & RxMissedOver))
- de->net_stats.rx_missed_errors += RxMissedMask;
+ de->dev->stats.rx_missed_errors += RxMissedMask;
else
- de->net_stats.rx_missed_errors += (rx_missed & RxMissedMask);
+ de->dev->stats.rx_missed_errors += (rx_missed & RxMissedMask);
}
static void __de_get_stats(struct de_private *de)
@@ -836,7 +834,7 @@
__de_get_stats(de);
spin_unlock_irq(&de->lock);
- return &de->net_stats;
+ return &dev->stats;
}
static inline int de_is_running (struct de_private *de)
@@ -1348,7 +1346,7 @@
struct sk_buff *skb = de->tx_skb[i].skb;
if ((skb) && (skb != DE_DUMMY_SKB)) {
if (skb != DE_SETUP_SKB) {
- de->net_stats.tx_dropped++;
+ de->dev->stats.tx_dropped++;
pci_unmap_single(de->pdev,
de->tx_skb[i].mapping,
skb->len, PCI_DMA_TODEVICE);
diff --git a/drivers/net/ethernet/dlink/dl2k.c b/drivers/net/ethernet/dlink/dl2k.c
index 1e35013..778f974 100644
--- a/drivers/net/ethernet/dlink/dl2k.c
+++ b/drivers/net/ethernet/dlink/dl2k.c
@@ -878,10 +878,10 @@
frame_id = (tx_status & 0xffff0000);
printk (KERN_ERR "%s: Transmit error, TxStatus %4.4x, FrameId %d.\n",
dev->name, tx_status, frame_id);
- np->stats.tx_errors++;
+ dev->stats.tx_errors++;
/* Ttransmit Underrun */
if (tx_status & 0x10) {
- np->stats.tx_fifo_errors++;
+ dev->stats.tx_fifo_errors++;
dw16(TxStartThresh, dr16(TxStartThresh) + 0x10);
/* Transmit Underrun need to set TxReset, DMARest, FIFOReset */
dw16(ASICCtrl + 2,
@@ -903,7 +903,7 @@
}
/* Late Collision */
if (tx_status & 0x04) {
- np->stats.tx_fifo_errors++;
+ dev->stats.tx_fifo_errors++;
/* TxReset and clear FIFO */
dw16(ASICCtrl + 2, TxReset | FIFOReset);
/* Wait reset done */
@@ -916,13 +916,8 @@
/* Let TxStartThresh stay default value */
}
/* Maximum Collisions */
-#ifdef ETHER_STATS
if (tx_status & 0x08)
- np->stats.collisions16++;
-#else
- if (tx_status & 0x08)
- np->stats.collisions++;
-#endif
+ dev->stats.collisions++;
/* Restart the Tx */
dw32(MACCtrl, dr16(MACCtrl) | TxEnable);
}
@@ -952,15 +947,15 @@
break;
/* Update rx error statistics, drop packet. */
if (frame_status & RFS_Errors) {
- np->stats.rx_errors++;
+ dev->stats.rx_errors++;
if (frame_status & (RxRuntFrame | RxLengthError))
- np->stats.rx_length_errors++;
+ dev->stats.rx_length_errors++;
if (frame_status & RxFCSError)
- np->stats.rx_crc_errors++;
+ dev->stats.rx_crc_errors++;
if (frame_status & RxAlignmentError && np->speed != 1000)
- np->stats.rx_frame_errors++;
+ dev->stats.rx_frame_errors++;
if (frame_status & RxFIFOOverrun)
- np->stats.rx_fifo_errors++;
+ dev->stats.rx_fifo_errors++;
} else {
struct sk_buff *skb;
@@ -1096,23 +1091,23 @@
/* All statistics registers need to be acknowledged,
else statistic overflow could cause problems */
- np->stats.rx_packets += dr32(FramesRcvOk);
- np->stats.tx_packets += dr32(FramesXmtOk);
- np->stats.rx_bytes += dr32(OctetRcvOk);
- np->stats.tx_bytes += dr32(OctetXmtOk);
+ dev->stats.rx_packets += dr32(FramesRcvOk);
+ dev->stats.tx_packets += dr32(FramesXmtOk);
+ dev->stats.rx_bytes += dr32(OctetRcvOk);
+ dev->stats.tx_bytes += dr32(OctetXmtOk);
- np->stats.multicast = dr32(McstFramesRcvdOk);
- np->stats.collisions += dr32(SingleColFrames)
+ dev->stats.multicast = dr32(McstFramesRcvdOk);
+ dev->stats.collisions += dr32(SingleColFrames)
+ dr32(MultiColFrames);
/* detailed tx errors */
stat_reg = dr16(FramesAbortXSColls);
- np->stats.tx_aborted_errors += stat_reg;
- np->stats.tx_errors += stat_reg;
+ dev->stats.tx_aborted_errors += stat_reg;
+ dev->stats.tx_errors += stat_reg;
stat_reg = dr16(CarrierSenseErrors);
- np->stats.tx_carrier_errors += stat_reg;
- np->stats.tx_errors += stat_reg;
+ dev->stats.tx_carrier_errors += stat_reg;
+ dev->stats.tx_errors += stat_reg;
/* Clear all other statistic register. */
dr32(McstOctetXmtOk);
@@ -1142,7 +1137,7 @@
dr16(TCPCheckSumErrors);
dr16(UDPCheckSumErrors);
dr16(IPCheckSumErrors);
- return &np->stats;
+ return &dev->stats;
}
static int
diff --git a/drivers/net/ethernet/dlink/dl2k.h b/drivers/net/ethernet/dlink/dl2k.h
index 5d8ae53..10e98ba 100644
--- a/drivers/net/ethernet/dlink/dl2k.h
+++ b/drivers/net/ethernet/dlink/dl2k.h
@@ -377,7 +377,6 @@
void __iomem *eeprom_addr;
spinlock_t tx_lock;
spinlock_t rx_lock;
- struct net_device_stats stats;
unsigned int rx_buf_sz; /* Based on MTU+slack. */
unsigned int speed; /* Operating speed */
unsigned int vlan; /* VLAN Id */
diff --git a/drivers/net/ethernet/emulex/benet/be.h b/drivers/net/ethernet/emulex/benet/be.h
index d49528a..5056624 100644
--- a/drivers/net/ethernet/emulex/benet/be.h
+++ b/drivers/net/ethernet/emulex/benet/be.h
@@ -567,6 +567,12 @@
/* Ethtool priv_flags */
#define BE_DISABLE_TPE_RECOVERY 0x1
+struct be_vxlan_port {
+ struct list_head list;
+ __be16 port; /* VxLAN UDP dst port */
+ int port_aliases; /* alias count */
+};
+
struct be_adapter {
struct pci_dev *pdev;
struct net_device *netdev;
@@ -671,9 +677,9 @@
u32 sli_family;
u8 hba_port_num;
u16 pvid;
- __be16 vxlan_port;
- int vxlan_port_count;
- int vxlan_port_aliases;
+ __be16 vxlan_port; /* offloaded vxlan port num */
+ int vxlan_port_count; /* active vxlan port count */
+ struct list_head vxlan_port_list; /* vxlan port list */
struct phy_info phy;
u8 wol_cap;
bool wol_en;
diff --git a/drivers/net/ethernet/emulex/benet/be_main.c b/drivers/net/ethernet/emulex/benet/be_main.c
index 6be3b9a..f3a09ab 100644
--- a/drivers/net/ethernet/emulex/benet/be_main.c
+++ b/drivers/net/ethernet/emulex/benet/be_main.c
@@ -3857,6 +3857,44 @@
}
}
+static int be_enable_vxlan_offloads(struct be_adapter *adapter)
+{
+ struct net_device *netdev = adapter->netdev;
+ struct device *dev = &adapter->pdev->dev;
+ struct be_vxlan_port *vxlan_port;
+ __be16 port;
+ int status;
+
+ vxlan_port = list_first_entry(&adapter->vxlan_port_list,
+ struct be_vxlan_port, list);
+ port = vxlan_port->port;
+
+ status = be_cmd_manage_iface(adapter, adapter->if_handle,
+ OP_CONVERT_NORMAL_TO_TUNNEL);
+ if (status) {
+ dev_warn(dev, "Failed to convert normal interface to tunnel\n");
+ return status;
+ }
+ adapter->flags |= BE_FLAGS_VXLAN_OFFLOADS;
+
+ status = be_cmd_set_vxlan_port(adapter, port);
+ if (status) {
+ dev_warn(dev, "Failed to add VxLAN port\n");
+ return status;
+ }
+ adapter->vxlan_port = port;
+
+ netdev->hw_enc_features |= NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |
+ NETIF_F_TSO | NETIF_F_TSO6 |
+ NETIF_F_GSO_UDP_TUNNEL;
+ netdev->hw_features |= NETIF_F_GSO_UDP_TUNNEL;
+ netdev->features |= NETIF_F_GSO_UDP_TUNNEL;
+
+ dev_info(dev, "Enabled VxLAN offloads for UDP port %d\n",
+ be16_to_cpu(port));
+ return 0;
+}
+
static void be_disable_vxlan_offloads(struct be_adapter *adapter)
{
struct net_device *netdev = adapter->netdev;
@@ -4903,63 +4941,59 @@
* those other tunnels are unexported on the fly through ndo_features_check().
*
* Skyhawk supports VxLAN offloads only for one UDP dport. So, if the stack
- * adds more than one port, disable offloads and don't re-enable them again
- * until after all the tunnels are removed.
+ * adds more than one port, disable offloads and re-enable them again when
+ * there's only one port left. We maintain a list of ports for this purpose.
*/
static void be_work_add_vxlan_port(struct work_struct *work)
{
struct be_cmd_work *cmd_work =
container_of(work, struct be_cmd_work, work);
struct be_adapter *adapter = cmd_work->adapter;
- struct net_device *netdev = adapter->netdev;
struct device *dev = &adapter->pdev->dev;
__be16 port = cmd_work->info.vxlan_port;
+ struct be_vxlan_port *vxlan_port;
int status;
- if (adapter->vxlan_port == port && adapter->vxlan_port_count) {
- adapter->vxlan_port_aliases++;
- goto done;
+ /* Bump up the alias count if it is an existing port */
+ list_for_each_entry(vxlan_port, &adapter->vxlan_port_list, list) {
+ if (vxlan_port->port == port) {
+ vxlan_port->port_aliases++;
+ goto done;
+ }
}
+ /* Add a new port to our list. We don't need a lock here since port
+ * add/delete are done only in the context of a single-threaded work
+ * queue (be_wq).
+ */
+ vxlan_port = kzalloc(sizeof(*vxlan_port), GFP_KERNEL);
+ if (!vxlan_port)
+ goto done;
+
+ vxlan_port->port = port;
+ INIT_LIST_HEAD(&vxlan_port->list);
+ list_add_tail(&vxlan_port->list, &adapter->vxlan_port_list);
+ adapter->vxlan_port_count++;
+
if (adapter->flags & BE_FLAGS_VXLAN_OFFLOADS) {
dev_info(dev,
"Only one UDP port supported for VxLAN offloads\n");
dev_info(dev, "Disabling VxLAN offloads\n");
- adapter->vxlan_port_count++;
goto err;
}
- if (adapter->vxlan_port_count++ >= 1)
+ if (adapter->vxlan_port_count > 1)
goto done;
- status = be_cmd_manage_iface(adapter, adapter->if_handle,
- OP_CONVERT_NORMAL_TO_TUNNEL);
- if (status) {
- dev_warn(dev, "Failed to convert normal interface to tunnel\n");
- goto err;
- }
+ status = be_enable_vxlan_offloads(adapter);
+ if (!status)
+ goto done;
- status = be_cmd_set_vxlan_port(adapter, port);
- if (status) {
- dev_warn(dev, "Failed to add VxLAN port\n");
- goto err;
- }
- adapter->flags |= BE_FLAGS_VXLAN_OFFLOADS;
- adapter->vxlan_port = port;
-
- netdev->hw_enc_features |= NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |
- NETIF_F_TSO | NETIF_F_TSO6 |
- NETIF_F_GSO_UDP_TUNNEL;
- netdev->hw_features |= NETIF_F_GSO_UDP_TUNNEL;
- netdev->features |= NETIF_F_GSO_UDP_TUNNEL;
-
- dev_info(dev, "Enabled VxLAN offloads for UDP port %d\n",
- be16_to_cpu(port));
- goto done;
err:
be_disable_vxlan_offloads(adapter);
done:
kfree(cmd_work);
+ return;
}
static void be_work_del_vxlan_port(struct work_struct *work)
@@ -4968,23 +5002,40 @@
container_of(work, struct be_cmd_work, work);
struct be_adapter *adapter = cmd_work->adapter;
__be16 port = cmd_work->info.vxlan_port;
+ struct be_vxlan_port *vxlan_port;
- if (adapter->vxlan_port != port)
- goto done;
+ /* Nothing to be done if a port alias is being deleted */
+ list_for_each_entry(vxlan_port, &adapter->vxlan_port_list, list) {
+ if (vxlan_port->port == port) {
+ if (vxlan_port->port_aliases) {
+ vxlan_port->port_aliases--;
+ goto done;
+ }
+ break;
+ }
+ }
- if (adapter->vxlan_port_aliases) {
- adapter->vxlan_port_aliases--;
+ /* No port aliases left; delete the port from the list */
+ list_del(&vxlan_port->list);
+ adapter->vxlan_port_count--;
+
+ /* Disable VxLAN offload if this is the offloaded port */
+ if (adapter->vxlan_port == vxlan_port->port) {
+ WARN_ON(adapter->vxlan_port_count);
+ be_disable_vxlan_offloads(adapter);
+ dev_info(&adapter->pdev->dev,
+ "Disabled VxLAN offloads for UDP port %d\n",
+ be16_to_cpu(port));
goto out;
}
- be_disable_vxlan_offloads(adapter);
+ /* If only 1 port is left, re-enable VxLAN offload */
+ if (adapter->vxlan_port_count == 1)
+ be_enable_vxlan_offloads(adapter);
- dev_info(&adapter->pdev->dev,
- "Disabled VxLAN offloads for UDP port %d\n",
- be16_to_cpu(port));
-done:
- adapter->vxlan_port_count--;
out:
+ kfree(vxlan_port);
+done:
kfree(cmd_work);
}
@@ -5217,15 +5268,15 @@
dev_err(&adapter->pdev->dev, "Recoverable HW error code: 0x%x\n",
ue_err_code);
- if (jiffies - err_rec->probe_time <= initial_idle_time) {
+ if (time_before_eq(jiffies - err_rec->probe_time, initial_idle_time)) {
dev_err(&adapter->pdev->dev,
"Cannot recover within %lu sec from driver load\n",
jiffies_to_msecs(initial_idle_time) / MSEC_PER_SEC);
return false;
}
- if (err_rec->last_recovery_time &&
- (jiffies - err_rec->last_recovery_time <= recovery_interval)) {
+ if (err_rec->last_recovery_time && time_before_eq(
+ jiffies - err_rec->last_recovery_time, recovery_interval)) {
dev_err(&adapter->pdev->dev,
"Cannot recover within %lu sec from last recovery\n",
jiffies_to_msecs(recovery_interval) / MSEC_PER_SEC);
@@ -5626,6 +5677,7 @@
/* Must be a power of 2 or else MODULO will BUG_ON */
adapter->be_get_temp_freq = 64;
+ INIT_LIST_HEAD(&adapter->vxlan_port_list);
return 0;
free_rx_filter:
diff --git a/drivers/net/ethernet/ethoc.c b/drivers/net/ethernet/ethoc.c
index 23d8274..e863ba7 100644
--- a/drivers/net/ethernet/ethoc.c
+++ b/drivers/net/ethernet/ethoc.c
@@ -1148,14 +1148,14 @@
/* Allow the platform setup code to pass in a MAC address. */
if (pdata) {
- memcpy(netdev->dev_addr, pdata->hwaddr, IFHWADDRLEN);
+ ether_addr_copy(netdev->dev_addr, pdata->hwaddr);
priv->phy_id = pdata->phy_id;
} else {
const void *mac;
mac = of_get_mac_address(pdev->dev.of_node);
if (mac)
- memcpy(netdev->dev_addr, mac, IFHWADDRLEN);
+ ether_addr_copy(netdev->dev_addr, mac);
priv->phy_id = -1;
}
diff --git a/drivers/net/ethernet/ezchip/nps_enet.c b/drivers/net/ethernet/ezchip/nps_enet.c
index f819843..659f1ad 100644
--- a/drivers/net/ethernet/ezchip/nps_enet.c
+++ b/drivers/net/ethernet/ezchip/nps_enet.c
@@ -16,6 +16,7 @@
#include <linux/module.h>
#include <linux/etherdevice.h>
+#include <linux/interrupt.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
#include <linux/of_net.h>
diff --git a/drivers/net/ethernet/faraday/ftgmac100.c b/drivers/net/ethernet/faraday/ftgmac100.c
index ade6b3e..95bf5e8 100644
--- a/drivers/net/ethernet/faraday/ftgmac100.c
+++ b/drivers/net/ethernet/faraday/ftgmac100.c
@@ -32,6 +32,9 @@
#include <linux/phy.h>
#include <linux/platform_device.h>
#include <linux/property.h>
+#include <linux/crc32.h>
+#include <linux/if_vlan.h>
+#include <linux/of_net.h>
#include <net/ip.h>
#include <net/ncsi.h>
@@ -40,103 +43,137 @@
#define DRV_NAME "ftgmac100"
#define DRV_VERSION "0.7"
-#define RX_QUEUE_ENTRIES 256 /* must be power of 2 */
-#define TX_QUEUE_ENTRIES 512 /* must be power of 2 */
+/* Arbitrary values, I am not sure the HW has limits */
+#define MAX_RX_QUEUE_ENTRIES 1024
+#define MAX_TX_QUEUE_ENTRIES 1024
+#define MIN_RX_QUEUE_ENTRIES 32
+#define MIN_TX_QUEUE_ENTRIES 32
-#define MAX_PKT_SIZE 1518
-#define RX_BUF_SIZE PAGE_SIZE /* must be smaller than 0x3fff */
+/* Defaults */
+#define DEF_RX_QUEUE_ENTRIES 128
+#define DEF_TX_QUEUE_ENTRIES 128
-/******************************************************************************
- * private data
- *****************************************************************************/
-struct ftgmac100_descs {
- struct ftgmac100_rxdes rxdes[RX_QUEUE_ENTRIES];
- struct ftgmac100_txdes txdes[TX_QUEUE_ENTRIES];
-};
+#define MAX_PKT_SIZE 1536
+#define RX_BUF_SIZE MAX_PKT_SIZE /* must be smaller than 0x3fff */
+
+/* Min number of tx ring entries before stopping queue */
+#define TX_THRESHOLD (MAX_SKB_FRAGS + 1)
struct ftgmac100 {
+ /* Registers */
struct resource *res;
void __iomem *base;
- int irq;
- struct ftgmac100_descs *descs;
- dma_addr_t descs_dma_addr;
-
- struct page *rx_pages[RX_QUEUE_ENTRIES];
-
+ /* Rx ring */
+ unsigned int rx_q_entries;
+ struct ftgmac100_rxdes *rxdes;
+ dma_addr_t rxdes_dma;
+ struct sk_buff **rx_skbs;
unsigned int rx_pointer;
+ u32 rxdes0_edorr_mask;
+
+ /* Tx ring */
+ unsigned int tx_q_entries;
+ struct ftgmac100_txdes *txdes;
+ dma_addr_t txdes_dma;
+ struct sk_buff **tx_skbs;
unsigned int tx_clean_pointer;
unsigned int tx_pointer;
- unsigned int tx_pending;
+ u32 txdes0_edotr_mask;
- spinlock_t tx_lock;
+ /* Used to signal the reset task of ring change request */
+ unsigned int new_rx_q_entries;
+ unsigned int new_tx_q_entries;
+ /* Scratch page to use when rx skb alloc fails */
+ void *rx_scratch;
+ dma_addr_t rx_scratch_dma;
+
+ /* Component structures */
struct net_device *netdev;
struct device *dev;
struct ncsi_dev *ndev;
struct napi_struct napi;
-
+ struct work_struct reset_task;
struct mii_bus *mii_bus;
- int old_speed;
- int int_mask_all;
- bool use_ncsi;
- bool enabled;
- u32 rxdes0_edorr_mask;
- u32 txdes0_edotr_mask;
+ /* Link management */
+ int cur_speed;
+ int cur_duplex;
+ bool use_ncsi;
+
+ /* Multicast filter settings */
+ u32 maht0;
+ u32 maht1;
+
+ /* Flow control settings */
+ bool tx_pause;
+ bool rx_pause;
+ bool aneg_pause;
+
+ /* Misc */
+ bool need_mac_restart;
+ bool is_aspeed;
};
-static int ftgmac100_alloc_rx_page(struct ftgmac100 *priv,
- struct ftgmac100_rxdes *rxdes, gfp_t gfp);
-
-/******************************************************************************
- * internal functions (hardware register access)
- *****************************************************************************/
-static void ftgmac100_set_rx_ring_base(struct ftgmac100 *priv, dma_addr_t addr)
-{
- iowrite32(addr, priv->base + FTGMAC100_OFFSET_RXR_BADR);
-}
-
-static void ftgmac100_set_rx_buffer_size(struct ftgmac100 *priv,
- unsigned int size)
-{
- size = FTGMAC100_RBSR_SIZE(size);
- iowrite32(size, priv->base + FTGMAC100_OFFSET_RBSR);
-}
-
-static void ftgmac100_set_normal_prio_tx_ring_base(struct ftgmac100 *priv,
- dma_addr_t addr)
-{
- iowrite32(addr, priv->base + FTGMAC100_OFFSET_NPTXR_BADR);
-}
-
-static void ftgmac100_txdma_normal_prio_start_polling(struct ftgmac100 *priv)
-{
- iowrite32(1, priv->base + FTGMAC100_OFFSET_NPTXPD);
-}
-
-static int ftgmac100_reset_hw(struct ftgmac100 *priv)
+static int ftgmac100_reset_mac(struct ftgmac100 *priv, u32 maccr)
{
struct net_device *netdev = priv->netdev;
int i;
/* NOTE: reset clears all registers */
- iowrite32(FTGMAC100_MACCR_SW_RST, priv->base + FTGMAC100_OFFSET_MACCR);
- for (i = 0; i < 5; i++) {
+ iowrite32(maccr, priv->base + FTGMAC100_OFFSET_MACCR);
+ iowrite32(maccr | FTGMAC100_MACCR_SW_RST,
+ priv->base + FTGMAC100_OFFSET_MACCR);
+ for (i = 0; i < 50; i++) {
unsigned int maccr;
maccr = ioread32(priv->base + FTGMAC100_OFFSET_MACCR);
if (!(maccr & FTGMAC100_MACCR_SW_RST))
return 0;
- udelay(1000);
+ udelay(1);
}
- netdev_err(netdev, "software reset failed\n");
+ netdev_err(netdev, "Hardware reset failed\n");
return -EIO;
}
-static void ftgmac100_set_mac(struct ftgmac100 *priv, const unsigned char *mac)
+static int ftgmac100_reset_and_config_mac(struct ftgmac100 *priv)
+{
+ u32 maccr = 0;
+
+ switch (priv->cur_speed) {
+ case SPEED_10:
+ case 0: /* no link */
+ break;
+
+ case SPEED_100:
+ maccr |= FTGMAC100_MACCR_FAST_MODE;
+ break;
+
+ case SPEED_1000:
+ maccr |= FTGMAC100_MACCR_GIGA_MODE;
+ break;
+ default:
+ netdev_err(priv->netdev, "Unknown speed %d !\n",
+ priv->cur_speed);
+ break;
+ }
+
+ /* (Re)initialize the queue pointers */
+ priv->rx_pointer = 0;
+ priv->tx_clean_pointer = 0;
+ priv->tx_pointer = 0;
+
+ /* The doc says reset twice with 10us interval */
+ if (ftgmac100_reset_mac(priv, maccr))
+ return -EIO;
+ usleep_range(10, 1000);
+ return ftgmac100_reset_mac(priv, maccr);
+}
+
+static void ftgmac100_write_mac_addr(struct ftgmac100 *priv, const u8 *mac)
{
unsigned int maddr = mac[0] << 8 | mac[1];
unsigned int laddr = mac[2] << 24 | mac[3] << 16 | mac[4] << 8 | mac[5];
@@ -145,7 +182,7 @@
iowrite32(laddr, priv->base + FTGMAC100_OFFSET_MAC_LADR);
}
-static void ftgmac100_setup_mac(struct ftgmac100 *priv)
+static void ftgmac100_initial_mac(struct ftgmac100 *priv)
{
u8 mac[ETH_ALEN];
unsigned int m;
@@ -189,55 +226,122 @@
return ret;
eth_commit_mac_addr_change(dev, p);
- ftgmac100_set_mac(netdev_priv(dev), dev->dev_addr);
+ ftgmac100_write_mac_addr(netdev_priv(dev), dev->dev_addr);
return 0;
}
-static void ftgmac100_init_hw(struct ftgmac100 *priv)
+static void ftgmac100_config_pause(struct ftgmac100 *priv)
{
- /* setup ring buffer base registers */
- ftgmac100_set_rx_ring_base(priv,
- priv->descs_dma_addr +
- offsetof(struct ftgmac100_descs, rxdes));
- ftgmac100_set_normal_prio_tx_ring_base(priv,
- priv->descs_dma_addr +
- offsetof(struct ftgmac100_descs, txdes));
+ u32 fcr = FTGMAC100_FCR_PAUSE_TIME(16);
- ftgmac100_set_rx_buffer_size(priv, RX_BUF_SIZE);
+ /* Throttle tx queue when receiving pause frames */
+ if (priv->rx_pause)
+ fcr |= FTGMAC100_FCR_FC_EN;
- iowrite32(FTGMAC100_APTC_RXPOLL_CNT(1), priv->base + FTGMAC100_OFFSET_APTC);
+ /* Enables sending pause frames when the RX queue is past a
+ * certain threshold.
+ */
+ if (priv->tx_pause)
+ fcr |= FTGMAC100_FCR_FCTHR_EN;
- ftgmac100_set_mac(priv, priv->netdev->dev_addr);
+ iowrite32(fcr, priv->base + FTGMAC100_OFFSET_FCR);
}
-#define MACCR_ENABLE_ALL (FTGMAC100_MACCR_TXDMA_EN | \
- FTGMAC100_MACCR_RXDMA_EN | \
- FTGMAC100_MACCR_TXMAC_EN | \
- FTGMAC100_MACCR_RXMAC_EN | \
- FTGMAC100_MACCR_FULLDUP | \
- FTGMAC100_MACCR_CRC_APD | \
- FTGMAC100_MACCR_RX_RUNT | \
- FTGMAC100_MACCR_RX_BROADPKT)
-
-static void ftgmac100_start_hw(struct ftgmac100 *priv, int speed)
+static void ftgmac100_init_hw(struct ftgmac100 *priv)
{
- int maccr = MACCR_ENABLE_ALL;
+ u32 reg, rfifo_sz, tfifo_sz;
- switch (speed) {
- default:
- case 10:
- break;
+ /* Clear stale interrupts */
+ reg = ioread32(priv->base + FTGMAC100_OFFSET_ISR);
+ iowrite32(reg, priv->base + FTGMAC100_OFFSET_ISR);
- case 100:
- maccr |= FTGMAC100_MACCR_FAST_MODE;
- break;
+ /* Setup RX ring buffer base */
+ iowrite32(priv->rxdes_dma, priv->base + FTGMAC100_OFFSET_RXR_BADR);
- case 1000:
- maccr |= FTGMAC100_MACCR_GIGA_MODE;
- break;
- }
+ /* Setup TX ring buffer base */
+ iowrite32(priv->txdes_dma, priv->base + FTGMAC100_OFFSET_NPTXR_BADR);
+ /* Configure RX buffer size */
+ iowrite32(FTGMAC100_RBSR_SIZE(RX_BUF_SIZE),
+ priv->base + FTGMAC100_OFFSET_RBSR);
+
+ /* Set RX descriptor autopoll */
+ iowrite32(FTGMAC100_APTC_RXPOLL_CNT(1),
+ priv->base + FTGMAC100_OFFSET_APTC);
+
+ /* Write MAC address */
+ ftgmac100_write_mac_addr(priv, priv->netdev->dev_addr);
+
+ /* Write multicast filter */
+ iowrite32(priv->maht0, priv->base + FTGMAC100_OFFSET_MAHT0);
+ iowrite32(priv->maht1, priv->base + FTGMAC100_OFFSET_MAHT1);
+
+ /* Configure descriptor sizes and increase burst sizes according
+ * to values in Aspeed SDK. The FIFO arbitration is enabled and
+ * the thresholds set based on the recommended values in the
+ * AST2400 specification.
+ */
+ iowrite32(FTGMAC100_DBLAC_RXDES_SIZE(2) | /* 2*8 bytes RX descs */
+ FTGMAC100_DBLAC_TXDES_SIZE(2) | /* 2*8 bytes TX descs */
+ FTGMAC100_DBLAC_RXBURST_SIZE(3) | /* 512 bytes max RX bursts */
+ FTGMAC100_DBLAC_TXBURST_SIZE(3) | /* 512 bytes max TX bursts */
+ FTGMAC100_DBLAC_RX_THR_EN | /* Enable fifo threshold arb */
+ FTGMAC100_DBLAC_RXFIFO_HTHR(6) | /* 6/8 of FIFO high threshold */
+ FTGMAC100_DBLAC_RXFIFO_LTHR(2), /* 2/8 of FIFO low threshold */
+ priv->base + FTGMAC100_OFFSET_DBLAC);
+
+ /* Interrupt mitigation configured for 1 interrupt/packet. HW interrupt
+ * mitigation doesn't seem to provide any benefit with NAPI so leave
+ * it at that.
+ */
+ iowrite32(FTGMAC100_ITC_RXINT_THR(1) |
+ FTGMAC100_ITC_TXINT_THR(1),
+ priv->base + FTGMAC100_OFFSET_ITC);
+
+ /* Configure FIFO sizes in the TPAFCR register */
+ reg = ioread32(priv->base + FTGMAC100_OFFSET_FEAR);
+ rfifo_sz = reg & 0x00000007;
+ tfifo_sz = (reg >> 3) & 0x00000007;
+ reg = ioread32(priv->base + FTGMAC100_OFFSET_TPAFCR);
+ reg &= ~0x3f000000;
+ reg |= (tfifo_sz << 27);
+ reg |= (rfifo_sz << 24);
+ iowrite32(reg, priv->base + FTGMAC100_OFFSET_TPAFCR);
+}
+
+static void ftgmac100_start_hw(struct ftgmac100 *priv)
+{
+ u32 maccr = ioread32(priv->base + FTGMAC100_OFFSET_MACCR);
+
+ /* Keep the original GMAC and FAST bits */
+ maccr &= (FTGMAC100_MACCR_FAST_MODE | FTGMAC100_MACCR_GIGA_MODE);
+
+ /* Add all the main enable bits */
+ maccr |= FTGMAC100_MACCR_TXDMA_EN |
+ FTGMAC100_MACCR_RXDMA_EN |
+ FTGMAC100_MACCR_TXMAC_EN |
+ FTGMAC100_MACCR_RXMAC_EN |
+ FTGMAC100_MACCR_CRC_APD |
+ FTGMAC100_MACCR_PHY_LINK_LEVEL |
+ FTGMAC100_MACCR_RX_RUNT |
+ FTGMAC100_MACCR_RX_BROADPKT;
+
+ /* Add other bits as needed */
+ if (priv->cur_duplex == DUPLEX_FULL)
+ maccr |= FTGMAC100_MACCR_FULLDUP;
+ if (priv->netdev->flags & IFF_PROMISC)
+ maccr |= FTGMAC100_MACCR_RX_ALL;
+ if (priv->netdev->flags & IFF_ALLMULTI)
+ maccr |= FTGMAC100_MACCR_RX_MULTIPKT;
+ else if (netdev_mc_count(priv->netdev))
+ maccr |= FTGMAC100_MACCR_HT_MULTI_EN;
+
+ /* Vlan filtering enabled */
+ if (priv->netdev->features & NETIF_F_HW_VLAN_CTAG_RX)
+ maccr |= FTGMAC100_MACCR_RM_VLAN;
+
+ /* Hit the HW */
iowrite32(maccr, priv->base + FTGMAC100_OFFSET_MACCR);
}
@@ -246,237 +350,107 @@
iowrite32(0, priv->base + FTGMAC100_OFFSET_MACCR);
}
-/******************************************************************************
- * internal functions (receive descriptor)
- *****************************************************************************/
-static bool ftgmac100_rxdes_first_segment(struct ftgmac100_rxdes *rxdes)
+static void ftgmac100_calc_mc_hash(struct ftgmac100 *priv)
{
- return rxdes->rxdes0 & cpu_to_le32(FTGMAC100_RXDES0_FRS);
-}
+ struct netdev_hw_addr *ha;
-static bool ftgmac100_rxdes_last_segment(struct ftgmac100_rxdes *rxdes)
-{
- return rxdes->rxdes0 & cpu_to_le32(FTGMAC100_RXDES0_LRS);
-}
+ priv->maht1 = 0;
+ priv->maht0 = 0;
+ netdev_for_each_mc_addr(ha, priv->netdev) {
+ u32 crc_val = ether_crc_le(ETH_ALEN, ha->addr);
-static bool ftgmac100_rxdes_packet_ready(struct ftgmac100_rxdes *rxdes)
-{
- return rxdes->rxdes0 & cpu_to_le32(FTGMAC100_RXDES0_RXPKT_RDY);
-}
-
-static void ftgmac100_rxdes_set_dma_own(const struct ftgmac100 *priv,
- struct ftgmac100_rxdes *rxdes)
-{
- /* clear status bits */
- rxdes->rxdes0 &= cpu_to_le32(priv->rxdes0_edorr_mask);
-}
-
-static bool ftgmac100_rxdes_rx_error(struct ftgmac100_rxdes *rxdes)
-{
- return rxdes->rxdes0 & cpu_to_le32(FTGMAC100_RXDES0_RX_ERR);
-}
-
-static bool ftgmac100_rxdes_crc_error(struct ftgmac100_rxdes *rxdes)
-{
- return rxdes->rxdes0 & cpu_to_le32(FTGMAC100_RXDES0_CRC_ERR);
-}
-
-static bool ftgmac100_rxdes_frame_too_long(struct ftgmac100_rxdes *rxdes)
-{
- return rxdes->rxdes0 & cpu_to_le32(FTGMAC100_RXDES0_FTL);
-}
-
-static bool ftgmac100_rxdes_runt(struct ftgmac100_rxdes *rxdes)
-{
- return rxdes->rxdes0 & cpu_to_le32(FTGMAC100_RXDES0_RUNT);
-}
-
-static bool ftgmac100_rxdes_odd_nibble(struct ftgmac100_rxdes *rxdes)
-{
- return rxdes->rxdes0 & cpu_to_le32(FTGMAC100_RXDES0_RX_ODD_NB);
-}
-
-static unsigned int ftgmac100_rxdes_data_length(struct ftgmac100_rxdes *rxdes)
-{
- return le32_to_cpu(rxdes->rxdes0) & FTGMAC100_RXDES0_VDBC;
-}
-
-static bool ftgmac100_rxdes_multicast(struct ftgmac100_rxdes *rxdes)
-{
- return rxdes->rxdes0 & cpu_to_le32(FTGMAC100_RXDES0_MULTICAST);
-}
-
-static void ftgmac100_rxdes_set_end_of_ring(const struct ftgmac100 *priv,
- struct ftgmac100_rxdes *rxdes)
-{
- rxdes->rxdes0 |= cpu_to_le32(priv->rxdes0_edorr_mask);
-}
-
-static void ftgmac100_rxdes_set_dma_addr(struct ftgmac100_rxdes *rxdes,
- dma_addr_t addr)
-{
- rxdes->rxdes3 = cpu_to_le32(addr);
-}
-
-static dma_addr_t ftgmac100_rxdes_get_dma_addr(struct ftgmac100_rxdes *rxdes)
-{
- return le32_to_cpu(rxdes->rxdes3);
-}
-
-static bool ftgmac100_rxdes_is_tcp(struct ftgmac100_rxdes *rxdes)
-{
- return (rxdes->rxdes1 & cpu_to_le32(FTGMAC100_RXDES1_PROT_MASK)) ==
- cpu_to_le32(FTGMAC100_RXDES1_PROT_TCPIP);
-}
-
-static bool ftgmac100_rxdes_is_udp(struct ftgmac100_rxdes *rxdes)
-{
- return (rxdes->rxdes1 & cpu_to_le32(FTGMAC100_RXDES1_PROT_MASK)) ==
- cpu_to_le32(FTGMAC100_RXDES1_PROT_UDPIP);
-}
-
-static bool ftgmac100_rxdes_tcpcs_err(struct ftgmac100_rxdes *rxdes)
-{
- return rxdes->rxdes1 & cpu_to_le32(FTGMAC100_RXDES1_TCP_CHKSUM_ERR);
-}
-
-static bool ftgmac100_rxdes_udpcs_err(struct ftgmac100_rxdes *rxdes)
-{
- return rxdes->rxdes1 & cpu_to_le32(FTGMAC100_RXDES1_UDP_CHKSUM_ERR);
-}
-
-static bool ftgmac100_rxdes_ipcs_err(struct ftgmac100_rxdes *rxdes)
-{
- return rxdes->rxdes1 & cpu_to_le32(FTGMAC100_RXDES1_IP_CHKSUM_ERR);
-}
-
-static inline struct page **ftgmac100_rxdes_page_slot(struct ftgmac100 *priv,
- struct ftgmac100_rxdes *rxdes)
-{
- return &priv->rx_pages[rxdes - priv->descs->rxdes];
-}
-
-/*
- * rxdes2 is not used by hardware. We use it to keep track of page.
- * Since hardware does not touch it, we can skip cpu_to_le32()/le32_to_cpu().
- */
-static void ftgmac100_rxdes_set_page(struct ftgmac100 *priv,
- struct ftgmac100_rxdes *rxdes,
- struct page *page)
-{
- *ftgmac100_rxdes_page_slot(priv, rxdes) = page;
-}
-
-static struct page *ftgmac100_rxdes_get_page(struct ftgmac100 *priv,
- struct ftgmac100_rxdes *rxdes)
-{
- return *ftgmac100_rxdes_page_slot(priv, rxdes);
-}
-
-/******************************************************************************
- * internal functions (receive)
- *****************************************************************************/
-static int ftgmac100_next_rx_pointer(int pointer)
-{
- return (pointer + 1) & (RX_QUEUE_ENTRIES - 1);
-}
-
-static void ftgmac100_rx_pointer_advance(struct ftgmac100 *priv)
-{
- priv->rx_pointer = ftgmac100_next_rx_pointer(priv->rx_pointer);
-}
-
-static struct ftgmac100_rxdes *ftgmac100_current_rxdes(struct ftgmac100 *priv)
-{
- return &priv->descs->rxdes[priv->rx_pointer];
-}
-
-static struct ftgmac100_rxdes *
-ftgmac100_rx_locate_first_segment(struct ftgmac100 *priv)
-{
- struct ftgmac100_rxdes *rxdes = ftgmac100_current_rxdes(priv);
-
- while (ftgmac100_rxdes_packet_ready(rxdes)) {
- if (ftgmac100_rxdes_first_segment(rxdes))
- return rxdes;
-
- ftgmac100_rxdes_set_dma_own(priv, rxdes);
- ftgmac100_rx_pointer_advance(priv);
- rxdes = ftgmac100_current_rxdes(priv);
+ crc_val = (~(crc_val >> 2)) & 0x3f;
+ if (crc_val >= 32)
+ priv->maht1 |= 1ul << (crc_val - 32);
+ else
+ priv->maht0 |= 1ul << (crc_val);
}
-
- return NULL;
}
-static bool ftgmac100_rx_packet_error(struct ftgmac100 *priv,
- struct ftgmac100_rxdes *rxdes)
+static void ftgmac100_set_rx_mode(struct net_device *netdev)
+{
+ struct ftgmac100 *priv = netdev_priv(netdev);
+
+ /* Setup the hash filter */
+ ftgmac100_calc_mc_hash(priv);
+
+ /* Interface down ? that's all there is to do */
+ if (!netif_running(netdev))
+ return;
+
+ /* Update the HW */
+ iowrite32(priv->maht0, priv->base + FTGMAC100_OFFSET_MAHT0);
+ iowrite32(priv->maht1, priv->base + FTGMAC100_OFFSET_MAHT1);
+
+ /* Reconfigure MACCR */
+ ftgmac100_start_hw(priv);
+}
+
+static int ftgmac100_alloc_rx_buf(struct ftgmac100 *priv, unsigned int entry,
+ struct ftgmac100_rxdes *rxdes, gfp_t gfp)
{
struct net_device *netdev = priv->netdev;
- bool error = false;
+ struct sk_buff *skb;
+ dma_addr_t map;
+ int err;
- if (unlikely(ftgmac100_rxdes_rx_error(rxdes))) {
+ skb = netdev_alloc_skb_ip_align(netdev, RX_BUF_SIZE);
+ if (unlikely(!skb)) {
if (net_ratelimit())
- netdev_info(netdev, "rx err\n");
+ netdev_warn(netdev, "failed to allocate rx skb\n");
+ err = -ENOMEM;
+ map = priv->rx_scratch_dma;
+ } else {
+ map = dma_map_single(priv->dev, skb->data, RX_BUF_SIZE,
+ DMA_FROM_DEVICE);
+ if (unlikely(dma_mapping_error(priv->dev, map))) {
+ if (net_ratelimit())
+ netdev_err(netdev, "failed to map rx page\n");
+ dev_kfree_skb_any(skb);
+ map = priv->rx_scratch_dma;
+ skb = NULL;
+ err = -ENOMEM;
+ }
+ }
+ /* Store skb */
+ priv->rx_skbs[entry] = skb;
+
+ /* Store DMA address into RX desc */
+ rxdes->rxdes3 = cpu_to_le32(map);
+
+ /* Ensure the above is ordered vs clearing the OWN bit */
+ dma_wmb();
+
+ /* Clean status (which resets own bit) */
+ if (entry == (priv->rx_q_entries - 1))
+ rxdes->rxdes0 = cpu_to_le32(priv->rxdes0_edorr_mask);
+ else
+ rxdes->rxdes0 = 0;
+
+ return 0;
+}
+
+static unsigned int ftgmac100_next_rx_pointer(struct ftgmac100 *priv,
+ unsigned int pointer)
+{
+ return (pointer + 1) & (priv->rx_q_entries - 1);
+}
+
+static void ftgmac100_rx_packet_error(struct ftgmac100 *priv, u32 status)
+{
+ struct net_device *netdev = priv->netdev;
+
+ if (status & FTGMAC100_RXDES0_RX_ERR)
netdev->stats.rx_errors++;
- error = true;
- }
- if (unlikely(ftgmac100_rxdes_crc_error(rxdes))) {
- if (net_ratelimit())
- netdev_info(netdev, "rx crc err\n");
-
+ if (status & FTGMAC100_RXDES0_CRC_ERR)
netdev->stats.rx_crc_errors++;
- error = true;
- } else if (unlikely(ftgmac100_rxdes_ipcs_err(rxdes))) {
- if (net_ratelimit())
- netdev_info(netdev, "rx IP checksum err\n");
- error = true;
- }
-
- if (unlikely(ftgmac100_rxdes_frame_too_long(rxdes))) {
- if (net_ratelimit())
- netdev_info(netdev, "rx frame too long\n");
-
+ if (status & (FTGMAC100_RXDES0_FTL |
+ FTGMAC100_RXDES0_RUNT |
+ FTGMAC100_RXDES0_RX_ODD_NB))
netdev->stats.rx_length_errors++;
- error = true;
- } else if (unlikely(ftgmac100_rxdes_runt(rxdes))) {
- if (net_ratelimit())
- netdev_info(netdev, "rx runt\n");
-
- netdev->stats.rx_length_errors++;
- error = true;
- } else if (unlikely(ftgmac100_rxdes_odd_nibble(rxdes))) {
- if (net_ratelimit())
- netdev_info(netdev, "rx odd nibble\n");
-
- netdev->stats.rx_length_errors++;
- error = true;
- }
-
- return error;
-}
-
-static void ftgmac100_rx_drop_packet(struct ftgmac100 *priv)
-{
- struct net_device *netdev = priv->netdev;
- struct ftgmac100_rxdes *rxdes = ftgmac100_current_rxdes(priv);
- bool done = false;
-
- if (net_ratelimit())
- netdev_dbg(netdev, "drop packet %p\n", rxdes);
-
- do {
- if (ftgmac100_rxdes_last_segment(rxdes))
- done = true;
-
- ftgmac100_rxdes_set_dma_own(priv, rxdes);
- ftgmac100_rx_pointer_advance(priv);
- rxdes = ftgmac100_current_rxdes(priv);
- } while (!done && ftgmac100_rxdes_packet_ready(rxdes));
-
- netdev->stats.rx_dropped++;
}
static bool ftgmac100_rx_packet(struct ftgmac100 *priv, int *processed)
@@ -484,205 +458,183 @@
struct net_device *netdev = priv->netdev;
struct ftgmac100_rxdes *rxdes;
struct sk_buff *skb;
- bool done = false;
+ unsigned int pointer, size;
+ u32 status, csum_vlan;
+ dma_addr_t map;
- rxdes = ftgmac100_rx_locate_first_segment(priv);
- if (!rxdes)
+ /* Grab next RX descriptor */
+ pointer = priv->rx_pointer;
+ rxdes = &priv->rxdes[pointer];
+
+ /* Grab descriptor status */
+ status = le32_to_cpu(rxdes->rxdes0);
+
+ /* Do we have a packet ? */
+ if (!(status & FTGMAC100_RXDES0_RXPKT_RDY))
return false;
- if (unlikely(ftgmac100_rx_packet_error(priv, rxdes))) {
- ftgmac100_rx_drop_packet(priv);
- return true;
+ /* Order subsequent reads with the test for the ready bit */
+ dma_rmb();
+
+ /* We don't cope with fragmented RX packets */
+ if (unlikely(!(status & FTGMAC100_RXDES0_FRS) ||
+ !(status & FTGMAC100_RXDES0_LRS)))
+ goto drop;
+
+ /* Grab received size and csum vlan field in the descriptor */
+ size = status & FTGMAC100_RXDES0_VDBC;
+ csum_vlan = le32_to_cpu(rxdes->rxdes1);
+
+ /* Any error (other than csum offload) flagged ? */
+ if (unlikely(status & RXDES0_ANY_ERROR)) {
+ /* Correct for incorrect flagging of runt packets
+ * with vlan tags... Just accept a runt packet that
+ * has been flagged as vlan and whose size is at
+ * least 60 bytes.
+ */
+ if ((status & FTGMAC100_RXDES0_RUNT) &&
+ (csum_vlan & FTGMAC100_RXDES1_VLANTAG_AVAIL) &&
+ (size >= 60))
+ status &= ~FTGMAC100_RXDES0_RUNT;
+
+ /* Any error still in there ? */
+ if (status & RXDES0_ANY_ERROR) {
+ ftgmac100_rx_packet_error(priv, status);
+ goto drop;
+ }
}
- /* start processing */
- skb = netdev_alloc_skb_ip_align(netdev, 128);
- if (unlikely(!skb)) {
- if (net_ratelimit())
- netdev_err(netdev, "rx skb alloc failed\n");
-
- ftgmac100_rx_drop_packet(priv);
- return true;
+ /* If the packet had no skb (failed to allocate earlier)
+ * then try to allocate one and skip
+ */
+ skb = priv->rx_skbs[pointer];
+ if (!unlikely(skb)) {
+ ftgmac100_alloc_rx_buf(priv, pointer, rxdes, GFP_ATOMIC);
+ goto drop;
}
- if (unlikely(ftgmac100_rxdes_multicast(rxdes)))
+ if (unlikely(status & FTGMAC100_RXDES0_MULTICAST))
netdev->stats.multicast++;
- /*
- * It seems that HW does checksum incorrectly with fragmented packets,
- * so we are conservative here - if HW checksum error, let software do
- * the checksum again.
+ /* If the HW found checksum errors, bounce it to software.
+ *
+ * If we didn't, we need to see if the packet was recognized
+ * by HW as one of the supported checksummed protocols before
+ * we accept the HW test results.
*/
- if ((ftgmac100_rxdes_is_tcp(rxdes) && !ftgmac100_rxdes_tcpcs_err(rxdes)) ||
- (ftgmac100_rxdes_is_udp(rxdes) && !ftgmac100_rxdes_udpcs_err(rxdes)))
- skb->ip_summed = CHECKSUM_UNNECESSARY;
-
- do {
- dma_addr_t map = ftgmac100_rxdes_get_dma_addr(rxdes);
- struct page *page = ftgmac100_rxdes_get_page(priv, rxdes);
- unsigned int size;
-
- dma_unmap_page(priv->dev, map, RX_BUF_SIZE, DMA_FROM_DEVICE);
-
- size = ftgmac100_rxdes_data_length(rxdes);
- skb_fill_page_desc(skb, skb_shinfo(skb)->nr_frags, page, 0, size);
-
- skb->len += size;
- skb->data_len += size;
- skb->truesize += PAGE_SIZE;
-
- if (ftgmac100_rxdes_last_segment(rxdes))
- done = true;
-
- ftgmac100_alloc_rx_page(priv, rxdes, GFP_ATOMIC);
-
- ftgmac100_rx_pointer_advance(priv);
- rxdes = ftgmac100_current_rxdes(priv);
- } while (!done);
-
- /* Small frames are copied into linear part of skb to free one page */
- if (skb->len <= 128) {
- skb->truesize -= PAGE_SIZE;
- __pskb_pull_tail(skb, skb->len);
- } else {
- /* We pull the minimum amount into linear part */
- __pskb_pull_tail(skb, ETH_HLEN);
+ if (netdev->features & NETIF_F_RXCSUM) {
+ u32 err_bits = FTGMAC100_RXDES1_TCP_CHKSUM_ERR |
+ FTGMAC100_RXDES1_UDP_CHKSUM_ERR |
+ FTGMAC100_RXDES1_IP_CHKSUM_ERR;
+ if ((csum_vlan & err_bits) ||
+ !(csum_vlan & FTGMAC100_RXDES1_PROT_MASK))
+ skb->ip_summed = CHECKSUM_NONE;
+ else
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
}
+
+ /* Transfer received size to skb */
+ skb_put(skb, size);
+
+ /* Extract vlan tag */
+ if ((netdev->features & NETIF_F_HW_VLAN_CTAG_RX) &&
+ (csum_vlan & FTGMAC100_RXDES1_VLANTAG_AVAIL))
+ __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q),
+ csum_vlan & 0xffff);
+
+ /* Tear down DMA mapping, do necessary cache management */
+ map = le32_to_cpu(rxdes->rxdes3);
+
+#if defined(CONFIG_ARM) && !defined(CONFIG_ARM_DMA_USE_IOMMU)
+ /* When we don't have an iommu, we can save cycles by not
+ * invalidating the cache for the part of the packet that
+ * wasn't received.
+ */
+ dma_unmap_single(priv->dev, map, size, DMA_FROM_DEVICE);
+#else
+ dma_unmap_single(priv->dev, map, RX_BUF_SIZE, DMA_FROM_DEVICE);
+#endif
+
+
+ /* Resplenish rx ring */
+ ftgmac100_alloc_rx_buf(priv, pointer, rxdes, GFP_ATOMIC);
+ priv->rx_pointer = ftgmac100_next_rx_pointer(priv, pointer);
+
skb->protocol = eth_type_trans(skb, netdev);
netdev->stats.rx_packets++;
- netdev->stats.rx_bytes += skb->len;
+ netdev->stats.rx_bytes += size;
/* push packet to protocol stack */
- napi_gro_receive(&priv->napi, skb);
+ if (skb->ip_summed == CHECKSUM_NONE)
+ netif_receive_skb(skb);
+ else
+ napi_gro_receive(&priv->napi, skb);
(*processed)++;
return true;
+
+ drop:
+ /* Clean rxdes0 (which resets own bit) */
+ rxdes->rxdes0 = cpu_to_le32(status & priv->rxdes0_edorr_mask);
+ priv->rx_pointer = ftgmac100_next_rx_pointer(priv, pointer);
+ netdev->stats.rx_dropped++;
+ return true;
}
-/******************************************************************************
- * internal functions (transmit descriptor)
- *****************************************************************************/
-static void ftgmac100_txdes_reset(const struct ftgmac100 *priv,
- struct ftgmac100_txdes *txdes)
+static u32 ftgmac100_base_tx_ctlstat(struct ftgmac100 *priv,
+ unsigned int index)
{
- /* clear all except end of ring bit */
- txdes->txdes0 &= cpu_to_le32(priv->txdes0_edotr_mask);
- txdes->txdes1 = 0;
- txdes->txdes2 = 0;
- txdes->txdes3 = 0;
+ if (index == (priv->tx_q_entries - 1))
+ return priv->txdes0_edotr_mask;
+ else
+ return 0;
}
-static bool ftgmac100_txdes_owned_by_dma(struct ftgmac100_txdes *txdes)
+static unsigned int ftgmac100_next_tx_pointer(struct ftgmac100 *priv,
+ unsigned int pointer)
{
- return txdes->txdes0 & cpu_to_le32(FTGMAC100_TXDES0_TXDMA_OWN);
+ return (pointer + 1) & (priv->tx_q_entries - 1);
}
-static void ftgmac100_txdes_set_dma_own(struct ftgmac100_txdes *txdes)
+static u32 ftgmac100_tx_buf_avail(struct ftgmac100 *priv)
{
- /*
- * Make sure dma own bit will not be set before any other
- * descriptor fields.
+ /* Returns the number of available slots in the TX queue
+ *
+ * This always leaves one free slot so we don't have to
+ * worry about empty vs. full, and this simplifies the
+ * test for ftgmac100_tx_buf_cleanable() below
*/
- wmb();
- txdes->txdes0 |= cpu_to_le32(FTGMAC100_TXDES0_TXDMA_OWN);
+ return (priv->tx_clean_pointer - priv->tx_pointer - 1) &
+ (priv->tx_q_entries - 1);
}
-static void ftgmac100_txdes_set_end_of_ring(const struct ftgmac100 *priv,
- struct ftgmac100_txdes *txdes)
+static bool ftgmac100_tx_buf_cleanable(struct ftgmac100 *priv)
{
- txdes->txdes0 |= cpu_to_le32(priv->txdes0_edotr_mask);
+ return priv->tx_pointer != priv->tx_clean_pointer;
}
-static void ftgmac100_txdes_set_first_segment(struct ftgmac100_txdes *txdes)
+static void ftgmac100_free_tx_packet(struct ftgmac100 *priv,
+ unsigned int pointer,
+ struct sk_buff *skb,
+ struct ftgmac100_txdes *txdes,
+ u32 ctl_stat)
{
- txdes->txdes0 |= cpu_to_le32(FTGMAC100_TXDES0_FTS);
-}
+ dma_addr_t map = le32_to_cpu(txdes->txdes3);
+ size_t len;
-static void ftgmac100_txdes_set_last_segment(struct ftgmac100_txdes *txdes)
-{
- txdes->txdes0 |= cpu_to_le32(FTGMAC100_TXDES0_LTS);
-}
+ if (ctl_stat & FTGMAC100_TXDES0_FTS) {
+ len = skb_headlen(skb);
+ dma_unmap_single(priv->dev, map, len, DMA_TO_DEVICE);
+ } else {
+ len = FTGMAC100_TXDES0_TXBUF_SIZE(ctl_stat);
+ dma_unmap_page(priv->dev, map, len, DMA_TO_DEVICE);
+ }
-static void ftgmac100_txdes_set_buffer_size(struct ftgmac100_txdes *txdes,
- unsigned int len)
-{
- txdes->txdes0 |= cpu_to_le32(FTGMAC100_TXDES0_TXBUF_SIZE(len));
-}
-
-static void ftgmac100_txdes_set_txint(struct ftgmac100_txdes *txdes)
-{
- txdes->txdes1 |= cpu_to_le32(FTGMAC100_TXDES1_TXIC);
-}
-
-static void ftgmac100_txdes_set_tcpcs(struct ftgmac100_txdes *txdes)
-{
- txdes->txdes1 |= cpu_to_le32(FTGMAC100_TXDES1_TCP_CHKSUM);
-}
-
-static void ftgmac100_txdes_set_udpcs(struct ftgmac100_txdes *txdes)
-{
- txdes->txdes1 |= cpu_to_le32(FTGMAC100_TXDES1_UDP_CHKSUM);
-}
-
-static void ftgmac100_txdes_set_ipcs(struct ftgmac100_txdes *txdes)
-{
- txdes->txdes1 |= cpu_to_le32(FTGMAC100_TXDES1_IP_CHKSUM);
-}
-
-static void ftgmac100_txdes_set_dma_addr(struct ftgmac100_txdes *txdes,
- dma_addr_t addr)
-{
- txdes->txdes3 = cpu_to_le32(addr);
-}
-
-static dma_addr_t ftgmac100_txdes_get_dma_addr(struct ftgmac100_txdes *txdes)
-{
- return le32_to_cpu(txdes->txdes3);
-}
-
-/*
- * txdes2 is not used by hardware. We use it to keep track of socket buffer.
- * Since hardware does not touch it, we can skip cpu_to_le32()/le32_to_cpu().
- */
-static void ftgmac100_txdes_set_skb(struct ftgmac100_txdes *txdes,
- struct sk_buff *skb)
-{
- txdes->txdes2 = (unsigned int)skb;
-}
-
-static struct sk_buff *ftgmac100_txdes_get_skb(struct ftgmac100_txdes *txdes)
-{
- return (struct sk_buff *)txdes->txdes2;
-}
-
-/******************************************************************************
- * internal functions (transmit)
- *****************************************************************************/
-static int ftgmac100_next_tx_pointer(int pointer)
-{
- return (pointer + 1) & (TX_QUEUE_ENTRIES - 1);
-}
-
-static void ftgmac100_tx_pointer_advance(struct ftgmac100 *priv)
-{
- priv->tx_pointer = ftgmac100_next_tx_pointer(priv->tx_pointer);
-}
-
-static void ftgmac100_tx_clean_pointer_advance(struct ftgmac100 *priv)
-{
- priv->tx_clean_pointer = ftgmac100_next_tx_pointer(priv->tx_clean_pointer);
-}
-
-static struct ftgmac100_txdes *ftgmac100_current_txdes(struct ftgmac100 *priv)
-{
- return &priv->descs->txdes[priv->tx_pointer];
-}
-
-static struct ftgmac100_txdes *
-ftgmac100_current_clean_txdes(struct ftgmac100 *priv)
-{
- return &priv->descs->txdes[priv->tx_clean_pointer];
+ /* Free SKB on last segment */
+ if (ctl_stat & FTGMAC100_TXDES0_LTS)
+ dev_kfree_skb(skb);
+ priv->tx_skbs[pointer] = NULL;
}
static bool ftgmac100_tx_complete_packet(struct ftgmac100 *priv)
@@ -690,215 +642,417 @@
struct net_device *netdev = priv->netdev;
struct ftgmac100_txdes *txdes;
struct sk_buff *skb;
- dma_addr_t map;
+ unsigned int pointer;
+ u32 ctl_stat;
- if (priv->tx_pending == 0)
+ pointer = priv->tx_clean_pointer;
+ txdes = &priv->txdes[pointer];
+
+ ctl_stat = le32_to_cpu(txdes->txdes0);
+ if (ctl_stat & FTGMAC100_TXDES0_TXDMA_OWN)
return false;
- txdes = ftgmac100_current_clean_txdes(priv);
-
- if (ftgmac100_txdes_owned_by_dma(txdes))
- return false;
-
- skb = ftgmac100_txdes_get_skb(txdes);
- map = ftgmac100_txdes_get_dma_addr(txdes);
-
+ skb = priv->tx_skbs[pointer];
netdev->stats.tx_packets++;
netdev->stats.tx_bytes += skb->len;
+ ftgmac100_free_tx_packet(priv, pointer, skb, txdes, ctl_stat);
+ txdes->txdes0 = cpu_to_le32(ctl_stat & priv->txdes0_edotr_mask);
- dma_unmap_single(priv->dev, map, skb_headlen(skb), DMA_TO_DEVICE);
-
- dev_kfree_skb(skb);
-
- ftgmac100_txdes_reset(priv, txdes);
-
- ftgmac100_tx_clean_pointer_advance(priv);
-
- spin_lock(&priv->tx_lock);
- priv->tx_pending--;
- spin_unlock(&priv->tx_lock);
- netif_wake_queue(netdev);
+ priv->tx_clean_pointer = ftgmac100_next_tx_pointer(priv, pointer);
return true;
}
static void ftgmac100_tx_complete(struct ftgmac100 *priv)
{
- while (ftgmac100_tx_complete_packet(priv))
+ struct net_device *netdev = priv->netdev;
+
+ /* Process all completed packets */
+ while (ftgmac100_tx_buf_cleanable(priv) &&
+ ftgmac100_tx_complete_packet(priv))
;
+
+ /* Restart queue if needed */
+ smp_mb();
+ if (unlikely(netif_queue_stopped(netdev) &&
+ ftgmac100_tx_buf_avail(priv) >= TX_THRESHOLD)) {
+ struct netdev_queue *txq;
+
+ txq = netdev_get_tx_queue(netdev, 0);
+ __netif_tx_lock(txq, smp_processor_id());
+ if (netif_queue_stopped(netdev) &&
+ ftgmac100_tx_buf_avail(priv) >= TX_THRESHOLD)
+ netif_wake_queue(netdev);
+ __netif_tx_unlock(txq);
+ }
}
-static int ftgmac100_xmit(struct ftgmac100 *priv, struct sk_buff *skb,
- dma_addr_t map)
+static bool ftgmac100_prep_tx_csum(struct sk_buff *skb, u32 *csum_vlan)
{
- struct net_device *netdev = priv->netdev;
- struct ftgmac100_txdes *txdes;
- unsigned int len = (skb->len < ETH_ZLEN) ? ETH_ZLEN : skb->len;
+ if (skb->protocol == cpu_to_be16(ETH_P_IP)) {
+ u8 ip_proto = ip_hdr(skb)->protocol;
- txdes = ftgmac100_current_txdes(priv);
- ftgmac100_tx_pointer_advance(priv);
-
- /* setup TX descriptor */
- ftgmac100_txdes_set_skb(txdes, skb);
- ftgmac100_txdes_set_dma_addr(txdes, map);
- ftgmac100_txdes_set_buffer_size(txdes, len);
-
- ftgmac100_txdes_set_first_segment(txdes);
- ftgmac100_txdes_set_last_segment(txdes);
- ftgmac100_txdes_set_txint(txdes);
- if (skb->ip_summed == CHECKSUM_PARTIAL) {
- __be16 protocol = skb->protocol;
-
- if (protocol == cpu_to_be16(ETH_P_IP)) {
- u8 ip_proto = ip_hdr(skb)->protocol;
-
- ftgmac100_txdes_set_ipcs(txdes);
- if (ip_proto == IPPROTO_TCP)
- ftgmac100_txdes_set_tcpcs(txdes);
- else if (ip_proto == IPPROTO_UDP)
- ftgmac100_txdes_set_udpcs(txdes);
+ *csum_vlan |= FTGMAC100_TXDES1_IP_CHKSUM;
+ switch(ip_proto) {
+ case IPPROTO_TCP:
+ *csum_vlan |= FTGMAC100_TXDES1_TCP_CHKSUM;
+ return true;
+ case IPPROTO_UDP:
+ *csum_vlan |= FTGMAC100_TXDES1_UDP_CHKSUM;
+ return true;
+ case IPPROTO_IP:
+ return true;
}
}
-
- spin_lock(&priv->tx_lock);
- priv->tx_pending++;
- if (priv->tx_pending == TX_QUEUE_ENTRIES)
- netif_stop_queue(netdev);
-
- /* start transmit */
- ftgmac100_txdes_set_dma_own(txdes);
- spin_unlock(&priv->tx_lock);
-
- ftgmac100_txdma_normal_prio_start_polling(priv);
-
- return NETDEV_TX_OK;
+ return skb_checksum_help(skb) == 0;
}
-/******************************************************************************
- * internal functions (buffer)
- *****************************************************************************/
-static int ftgmac100_alloc_rx_page(struct ftgmac100 *priv,
- struct ftgmac100_rxdes *rxdes, gfp_t gfp)
+static int ftgmac100_hard_start_xmit(struct sk_buff *skb,
+ struct net_device *netdev)
{
- struct net_device *netdev = priv->netdev;
- struct page *page;
+ struct ftgmac100 *priv = netdev_priv(netdev);
+ struct ftgmac100_txdes *txdes, *first;
+ unsigned int pointer, nfrags, len, i, j;
+ u32 f_ctl_stat, ctl_stat, csum_vlan;
dma_addr_t map;
- page = alloc_page(gfp);
- if (!page) {
- if (net_ratelimit())
- netdev_err(netdev, "failed to allocate rx page\n");
- return -ENOMEM;
+ /* The HW doesn't pad small frames */
+ if (eth_skb_pad(skb)) {
+ netdev->stats.tx_dropped++;
+ return NETDEV_TX_OK;
}
- map = dma_map_page(priv->dev, page, 0, RX_BUF_SIZE, DMA_FROM_DEVICE);
- if (unlikely(dma_mapping_error(priv->dev, map))) {
+ /* Reject oversize packets */
+ if (unlikely(skb->len > MAX_PKT_SIZE)) {
if (net_ratelimit())
- netdev_err(netdev, "failed to map rx page\n");
- __free_page(page);
- return -ENOMEM;
+ netdev_dbg(netdev, "tx packet too big\n");
+ goto drop;
}
- ftgmac100_rxdes_set_page(priv, rxdes, page);
- ftgmac100_rxdes_set_dma_addr(rxdes, map);
- ftgmac100_rxdes_set_dma_own(priv, rxdes);
- return 0;
+ /* Do we have a limit on #fragments ? I yet have to get a reply
+ * from Aspeed. If there's one I haven't hit it.
+ */
+ nfrags = skb_shinfo(skb)->nr_frags;
+
+ /* Get header len */
+ len = skb_headlen(skb);
+
+ /* Map the packet head */
+ map = dma_map_single(priv->dev, skb->data, len, DMA_TO_DEVICE);
+ if (dma_mapping_error(priv->dev, map)) {
+ if (net_ratelimit())
+ netdev_err(netdev, "map tx packet head failed\n");
+ goto drop;
+ }
+
+ /* Grab the next free tx descriptor */
+ pointer = priv->tx_pointer;
+ txdes = first = &priv->txdes[pointer];
+
+ /* Setup it up with the packet head. Don't write the head to the
+ * ring just yet
+ */
+ priv->tx_skbs[pointer] = skb;
+ f_ctl_stat = ftgmac100_base_tx_ctlstat(priv, pointer);
+ f_ctl_stat |= FTGMAC100_TXDES0_TXDMA_OWN;
+ f_ctl_stat |= FTGMAC100_TXDES0_TXBUF_SIZE(len);
+ f_ctl_stat |= FTGMAC100_TXDES0_FTS;
+ if (nfrags == 0)
+ f_ctl_stat |= FTGMAC100_TXDES0_LTS;
+ txdes->txdes3 = cpu_to_le32(map);
+
+ /* Setup HW checksumming */
+ csum_vlan = 0;
+ if (skb->ip_summed == CHECKSUM_PARTIAL &&
+ !ftgmac100_prep_tx_csum(skb, &csum_vlan))
+ goto drop;
+
+ /* Add VLAN tag */
+ if (skb_vlan_tag_present(skb)) {
+ csum_vlan |= FTGMAC100_TXDES1_INS_VLANTAG;
+ csum_vlan |= skb_vlan_tag_get(skb) & 0xffff;
+ }
+
+ txdes->txdes1 = cpu_to_le32(csum_vlan);
+
+ /* Next descriptor */
+ pointer = ftgmac100_next_tx_pointer(priv, pointer);
+
+ /* Add the fragments */
+ for (i = 0; i < nfrags; i++) {
+ skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
+
+ len = frag->size;
+
+ /* Map it */
+ map = skb_frag_dma_map(priv->dev, frag, 0, len,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(priv->dev, map))
+ goto dma_err;
+
+ /* Setup descriptor */
+ priv->tx_skbs[pointer] = skb;
+ txdes = &priv->txdes[pointer];
+ ctl_stat = ftgmac100_base_tx_ctlstat(priv, pointer);
+ ctl_stat |= FTGMAC100_TXDES0_TXDMA_OWN;
+ ctl_stat |= FTGMAC100_TXDES0_TXBUF_SIZE(len);
+ if (i == (nfrags - 1))
+ ctl_stat |= FTGMAC100_TXDES0_LTS;
+ txdes->txdes0 = cpu_to_le32(ctl_stat);
+ txdes->txdes1 = 0;
+ txdes->txdes3 = cpu_to_le32(map);
+
+ /* Next one */
+ pointer = ftgmac100_next_tx_pointer(priv, pointer);
+ }
+
+ /* Order the previous packet and descriptor udpates
+ * before setting the OWN bit on the first descriptor.
+ */
+ dma_wmb();
+ first->txdes0 = cpu_to_le32(f_ctl_stat);
+
+ /* Update next TX pointer */
+ priv->tx_pointer = pointer;
+
+ /* If there isn't enough room for all the fragments of a new packet
+ * in the TX ring, stop the queue. The sequence below is race free
+ * vs. a concurrent restart in ftgmac100_poll()
+ */
+ if (unlikely(ftgmac100_tx_buf_avail(priv) < TX_THRESHOLD)) {
+ netif_stop_queue(netdev);
+ /* Order the queue stop with the test below */
+ smp_mb();
+ if (ftgmac100_tx_buf_avail(priv) >= TX_THRESHOLD)
+ netif_wake_queue(netdev);
+ }
+
+ /* Poke transmitter to read the updated TX descriptors */
+ iowrite32(1, priv->base + FTGMAC100_OFFSET_NPTXPD);
+
+ return NETDEV_TX_OK;
+
+ dma_err:
+ if (net_ratelimit())
+ netdev_err(netdev, "map tx fragment failed\n");
+
+ /* Free head */
+ pointer = priv->tx_pointer;
+ ftgmac100_free_tx_packet(priv, pointer, skb, first, f_ctl_stat);
+ first->txdes0 = cpu_to_le32(f_ctl_stat & priv->txdes0_edotr_mask);
+
+ /* Then all fragments */
+ for (j = 0; j < i; j++) {
+ pointer = ftgmac100_next_tx_pointer(priv, pointer);
+ txdes = &priv->txdes[pointer];
+ ctl_stat = le32_to_cpu(txdes->txdes0);
+ ftgmac100_free_tx_packet(priv, pointer, skb, txdes, ctl_stat);
+ txdes->txdes0 = cpu_to_le32(ctl_stat & priv->txdes0_edotr_mask);
+ }
+
+ /* This cannot be reached if we successfully mapped the
+ * last fragment, so we know ftgmac100_free_tx_packet()
+ * hasn't freed the skb yet.
+ */
+ drop:
+ /* Drop the packet */
+ dev_kfree_skb_any(skb);
+ netdev->stats.tx_dropped++;
+
+ return NETDEV_TX_OK;
}
static void ftgmac100_free_buffers(struct ftgmac100 *priv)
{
int i;
- for (i = 0; i < RX_QUEUE_ENTRIES; i++) {
- struct ftgmac100_rxdes *rxdes = &priv->descs->rxdes[i];
- struct page *page = ftgmac100_rxdes_get_page(priv, rxdes);
- dma_addr_t map = ftgmac100_rxdes_get_dma_addr(rxdes);
-
- if (!page)
- continue;
-
- dma_unmap_page(priv->dev, map, RX_BUF_SIZE, DMA_FROM_DEVICE);
- __free_page(page);
- }
-
- for (i = 0; i < TX_QUEUE_ENTRIES; i++) {
- struct ftgmac100_txdes *txdes = &priv->descs->txdes[i];
- struct sk_buff *skb = ftgmac100_txdes_get_skb(txdes);
- dma_addr_t map = ftgmac100_txdes_get_dma_addr(txdes);
+ /* Free all RX buffers */
+ for (i = 0; i < priv->rx_q_entries; i++) {
+ struct ftgmac100_rxdes *rxdes = &priv->rxdes[i];
+ struct sk_buff *skb = priv->rx_skbs[i];
+ dma_addr_t map = le32_to_cpu(rxdes->rxdes3);
if (!skb)
continue;
- dma_unmap_single(priv->dev, map, skb_headlen(skb), DMA_TO_DEVICE);
- kfree_skb(skb);
+ priv->rx_skbs[i] = NULL;
+ dma_unmap_single(priv->dev, map, RX_BUF_SIZE, DMA_FROM_DEVICE);
+ dev_kfree_skb_any(skb);
}
- dma_free_coherent(priv->dev, sizeof(struct ftgmac100_descs),
- priv->descs, priv->descs_dma_addr);
+ /* Free all TX buffers */
+ for (i = 0; i < priv->tx_q_entries; i++) {
+ struct ftgmac100_txdes *txdes = &priv->txdes[i];
+ struct sk_buff *skb = priv->tx_skbs[i];
+
+ if (!skb)
+ continue;
+ ftgmac100_free_tx_packet(priv, i, skb, txdes,
+ le32_to_cpu(txdes->txdes0));
+ }
}
-static int ftgmac100_alloc_buffers(struct ftgmac100 *priv)
+static void ftgmac100_free_rings(struct ftgmac100 *priv)
+{
+ /* Free skb arrays */
+ kfree(priv->rx_skbs);
+ kfree(priv->tx_skbs);
+
+ /* Free descriptors */
+ if (priv->rxdes)
+ dma_free_coherent(priv->dev, MAX_RX_QUEUE_ENTRIES *
+ sizeof(struct ftgmac100_rxdes),
+ priv->rxdes, priv->rxdes_dma);
+ priv->rxdes = NULL;
+
+ if (priv->txdes)
+ dma_free_coherent(priv->dev, MAX_TX_QUEUE_ENTRIES *
+ sizeof(struct ftgmac100_txdes),
+ priv->txdes, priv->txdes_dma);
+ priv->txdes = NULL;
+
+ /* Free scratch packet buffer */
+ if (priv->rx_scratch)
+ dma_free_coherent(priv->dev, RX_BUF_SIZE,
+ priv->rx_scratch, priv->rx_scratch_dma);
+}
+
+static int ftgmac100_alloc_rings(struct ftgmac100 *priv)
+{
+ /* Allocate skb arrays */
+ priv->rx_skbs = kcalloc(MAX_RX_QUEUE_ENTRIES, sizeof(void *),
+ GFP_KERNEL);
+ if (!priv->rx_skbs)
+ return -ENOMEM;
+ priv->tx_skbs = kcalloc(MAX_TX_QUEUE_ENTRIES, sizeof(void *),
+ GFP_KERNEL);
+ if (!priv->tx_skbs)
+ return -ENOMEM;
+
+ /* Allocate descriptors */
+ priv->rxdes = dma_zalloc_coherent(priv->dev,
+ MAX_RX_QUEUE_ENTRIES *
+ sizeof(struct ftgmac100_rxdes),
+ &priv->rxdes_dma, GFP_KERNEL);
+ if (!priv->rxdes)
+ return -ENOMEM;
+ priv->txdes = dma_zalloc_coherent(priv->dev,
+ MAX_TX_QUEUE_ENTRIES *
+ sizeof(struct ftgmac100_txdes),
+ &priv->txdes_dma, GFP_KERNEL);
+ if (!priv->txdes)
+ return -ENOMEM;
+
+ /* Allocate scratch packet buffer */
+ priv->rx_scratch = dma_alloc_coherent(priv->dev,
+ RX_BUF_SIZE,
+ &priv->rx_scratch_dma,
+ GFP_KERNEL);
+ if (!priv->rx_scratch)
+ return -ENOMEM;
+
+ return 0;
+}
+
+static void ftgmac100_init_rings(struct ftgmac100 *priv)
+{
+ struct ftgmac100_rxdes *rxdes = NULL;
+ struct ftgmac100_txdes *txdes = NULL;
+ int i;
+
+ /* Update entries counts */
+ priv->rx_q_entries = priv->new_rx_q_entries;
+ priv->tx_q_entries = priv->new_tx_q_entries;
+
+ if (WARN_ON(priv->rx_q_entries < MIN_RX_QUEUE_ENTRIES))
+ return;
+
+ /* Initialize RX ring */
+ for (i = 0; i < priv->rx_q_entries; i++) {
+ rxdes = &priv->rxdes[i];
+ rxdes->rxdes0 = 0;
+ rxdes->rxdes3 = cpu_to_le32(priv->rx_scratch_dma);
+ }
+ /* Mark the end of the ring */
+ rxdes->rxdes0 |= cpu_to_le32(priv->rxdes0_edorr_mask);
+
+ if (WARN_ON(priv->tx_q_entries < MIN_RX_QUEUE_ENTRIES))
+ return;
+
+ /* Initialize TX ring */
+ for (i = 0; i < priv->tx_q_entries; i++) {
+ txdes = &priv->txdes[i];
+ txdes->txdes0 = 0;
+ }
+ txdes->txdes0 |= cpu_to_le32(priv->txdes0_edotr_mask);
+}
+
+static int ftgmac100_alloc_rx_buffers(struct ftgmac100 *priv)
{
int i;
- priv->descs = dma_zalloc_coherent(priv->dev,
- sizeof(struct ftgmac100_descs),
- &priv->descs_dma_addr, GFP_KERNEL);
- if (!priv->descs)
- return -ENOMEM;
+ for (i = 0; i < priv->rx_q_entries; i++) {
+ struct ftgmac100_rxdes *rxdes = &priv->rxdes[i];
- /* initialize RX ring */
- ftgmac100_rxdes_set_end_of_ring(priv,
- &priv->descs->rxdes[RX_QUEUE_ENTRIES - 1]);
-
- for (i = 0; i < RX_QUEUE_ENTRIES; i++) {
- struct ftgmac100_rxdes *rxdes = &priv->descs->rxdes[i];
-
- if (ftgmac100_alloc_rx_page(priv, rxdes, GFP_KERNEL))
- goto err;
+ if (ftgmac100_alloc_rx_buf(priv, i, rxdes, GFP_KERNEL))
+ return -ENOMEM;
}
-
- /* initialize TX ring */
- ftgmac100_txdes_set_end_of_ring(priv,
- &priv->descs->txdes[TX_QUEUE_ENTRIES - 1]);
return 0;
-
-err:
- ftgmac100_free_buffers(priv);
- return -ENOMEM;
}
-/******************************************************************************
- * internal functions (mdio)
- *****************************************************************************/
static void ftgmac100_adjust_link(struct net_device *netdev)
{
struct ftgmac100 *priv = netdev_priv(netdev);
struct phy_device *phydev = netdev->phydev;
- int ier;
+ bool tx_pause, rx_pause;
+ int new_speed;
- if (phydev->speed == priv->old_speed)
+ /* We store "no link" as speed 0 */
+ if (!phydev->link)
+ new_speed = 0;
+ else
+ new_speed = phydev->speed;
+
+ /* Grab pause settings from PHY if configured to do so */
+ if (priv->aneg_pause) {
+ rx_pause = tx_pause = phydev->pause;
+ if (phydev->asym_pause)
+ tx_pause = !rx_pause;
+ } else {
+ rx_pause = priv->rx_pause;
+ tx_pause = priv->tx_pause;
+ }
+
+ /* Link hasn't changed, do nothing */
+ if (phydev->speed == priv->cur_speed &&
+ phydev->duplex == priv->cur_duplex &&
+ rx_pause == priv->rx_pause &&
+ tx_pause == priv->tx_pause)
return;
- priv->old_speed = phydev->speed;
+ /* Print status if we have a link or we had one and just lost it,
+ * don't print otherwise.
+ */
+ if (new_speed || priv->cur_speed)
+ phy_print_status(phydev);
- ier = ioread32(priv->base + FTGMAC100_OFFSET_IER);
+ priv->cur_speed = new_speed;
+ priv->cur_duplex = phydev->duplex;
+ priv->rx_pause = rx_pause;
+ priv->tx_pause = tx_pause;
- /* disable all interrupts */
+ /* Link is down, do nothing else */
+ if (!new_speed)
+ return;
+
+ /* Disable all interrupts */
iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
- netif_stop_queue(netdev);
- ftgmac100_stop_hw(priv);
-
- netif_start_queue(netdev);
- ftgmac100_init_hw(priv);
- ftgmac100_start_hw(priv, phydev->speed);
-
- /* re-enable interrupts */
- iowrite32(ier, priv->base + FTGMAC100_OFFSET_IER);
+ /* Reset the adapter asynchronously */
+ schedule_work(&priv->reset_task);
}
-static int ftgmac100_mii_probe(struct ftgmac100 *priv)
+static int ftgmac100_mii_probe(struct ftgmac100 *priv, phy_interface_t intf)
{
struct net_device *netdev = priv->netdev;
struct phy_device *phydev;
@@ -910,19 +1064,25 @@
}
phydev = phy_connect(netdev, phydev_name(phydev),
- &ftgmac100_adjust_link, PHY_INTERFACE_MODE_GMII);
+ &ftgmac100_adjust_link, intf);
if (IS_ERR(phydev)) {
netdev_err(netdev, "%s: Could not attach to PHY\n", netdev->name);
return PTR_ERR(phydev);
}
+ /* Indicate that we support PAUSE frames (see comment in
+ * Documentation/networking/phy.txt)
+ */
+ phydev->supported |= SUPPORTED_Pause | SUPPORTED_Asym_Pause;
+ phydev->advertising = phydev->supported;
+
+ /* Display what we found */
+ phy_attached_info(phydev);
+
return 0;
}
-/******************************************************************************
- * struct mii_bus functions
- *****************************************************************************/
static int ftgmac100_mdiobus_read(struct mii_bus *bus, int phy_addr, int regnum)
{
struct net_device *netdev = bus->priv;
@@ -994,9 +1154,6 @@
return -EIO;
}
-/******************************************************************************
- * struct ethtool_ops functions
- *****************************************************************************/
static void ftgmac100_get_drvinfo(struct net_device *netdev,
struct ethtool_drvinfo *info)
{
@@ -1005,175 +1162,365 @@
strlcpy(info->bus_info, dev_name(&netdev->dev), sizeof(info->bus_info));
}
+static void ftgmac100_get_ringparam(struct net_device *netdev,
+ struct ethtool_ringparam *ering)
+{
+ struct ftgmac100 *priv = netdev_priv(netdev);
+
+ memset(ering, 0, sizeof(*ering));
+ ering->rx_max_pending = MAX_RX_QUEUE_ENTRIES;
+ ering->tx_max_pending = MAX_TX_QUEUE_ENTRIES;
+ ering->rx_pending = priv->rx_q_entries;
+ ering->tx_pending = priv->tx_q_entries;
+}
+
+static int ftgmac100_set_ringparam(struct net_device *netdev,
+ struct ethtool_ringparam *ering)
+{
+ struct ftgmac100 *priv = netdev_priv(netdev);
+
+ if (ering->rx_pending > MAX_RX_QUEUE_ENTRIES ||
+ ering->tx_pending > MAX_TX_QUEUE_ENTRIES ||
+ ering->rx_pending < MIN_RX_QUEUE_ENTRIES ||
+ ering->tx_pending < MIN_TX_QUEUE_ENTRIES ||
+ !is_power_of_2(ering->rx_pending) ||
+ !is_power_of_2(ering->tx_pending))
+ return -EINVAL;
+
+ priv->new_rx_q_entries = ering->rx_pending;
+ priv->new_tx_q_entries = ering->tx_pending;
+ if (netif_running(netdev))
+ schedule_work(&priv->reset_task);
+
+ return 0;
+}
+
+static void ftgmac100_get_pauseparam(struct net_device *netdev,
+ struct ethtool_pauseparam *pause)
+{
+ struct ftgmac100 *priv = netdev_priv(netdev);
+
+ pause->autoneg = priv->aneg_pause;
+ pause->tx_pause = priv->tx_pause;
+ pause->rx_pause = priv->rx_pause;
+}
+
+static int ftgmac100_set_pauseparam(struct net_device *netdev,
+ struct ethtool_pauseparam *pause)
+{
+ struct ftgmac100 *priv = netdev_priv(netdev);
+ struct phy_device *phydev = netdev->phydev;
+
+ priv->aneg_pause = pause->autoneg;
+ priv->tx_pause = pause->tx_pause;
+ priv->rx_pause = pause->rx_pause;
+
+ if (phydev) {
+ phydev->advertising &= ~ADVERTISED_Pause;
+ phydev->advertising &= ~ADVERTISED_Asym_Pause;
+
+ if (pause->rx_pause) {
+ phydev->advertising |= ADVERTISED_Pause;
+ phydev->advertising |= ADVERTISED_Asym_Pause;
+ }
+
+ if (pause->tx_pause)
+ phydev->advertising ^= ADVERTISED_Asym_Pause;
+ }
+ if (netif_running(netdev)) {
+ if (phydev && priv->aneg_pause)
+ phy_start_aneg(phydev);
+ else
+ ftgmac100_config_pause(priv);
+ }
+
+ return 0;
+}
+
static const struct ethtool_ops ftgmac100_ethtool_ops = {
.get_drvinfo = ftgmac100_get_drvinfo,
.get_link = ethtool_op_get_link,
.get_link_ksettings = phy_ethtool_get_link_ksettings,
.set_link_ksettings = phy_ethtool_set_link_ksettings,
+ .nway_reset = phy_ethtool_nway_reset,
+ .get_ringparam = ftgmac100_get_ringparam,
+ .set_ringparam = ftgmac100_set_ringparam,
+ .get_pauseparam = ftgmac100_get_pauseparam,
+ .set_pauseparam = ftgmac100_set_pauseparam,
};
-/******************************************************************************
- * interrupt handler
- *****************************************************************************/
static irqreturn_t ftgmac100_interrupt(int irq, void *dev_id)
{
struct net_device *netdev = dev_id;
struct ftgmac100 *priv = netdev_priv(netdev);
+ unsigned int status, new_mask = FTGMAC100_INT_BAD;
- /* When running in NCSI mode, the interface should be ready for
- * receiving or transmitting NCSI packets before it's opened.
- */
- if (likely(priv->use_ncsi || netif_running(netdev))) {
- /* Disable interrupts for polling */
- iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
- napi_schedule(&priv->napi);
+ /* Fetch and clear interrupt bits, process abnormal ones */
+ status = ioread32(priv->base + FTGMAC100_OFFSET_ISR);
+ iowrite32(status, priv->base + FTGMAC100_OFFSET_ISR);
+ if (unlikely(status & FTGMAC100_INT_BAD)) {
+
+ /* RX buffer unavailable */
+ if (status & FTGMAC100_INT_NO_RXBUF)
+ netdev->stats.rx_over_errors++;
+
+ /* received packet lost due to RX FIFO full */
+ if (status & FTGMAC100_INT_RPKT_LOST)
+ netdev->stats.rx_fifo_errors++;
+
+ /* sent packet lost due to excessive TX collision */
+ if (status & FTGMAC100_INT_XPKT_LOST)
+ netdev->stats.tx_fifo_errors++;
+
+ /* AHB error -> Reset the chip */
+ if (status & FTGMAC100_INT_AHB_ERR) {
+ if (net_ratelimit())
+ netdev_warn(netdev,
+ "AHB bus error ! Resetting chip.\n");
+ iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
+ schedule_work(&priv->reset_task);
+ return IRQ_HANDLED;
+ }
+
+ /* We may need to restart the MAC after such errors, delay
+ * this until after we have freed some Rx buffers though
+ */
+ priv->need_mac_restart = true;
+
+ /* Disable those errors until we restart */
+ new_mask &= ~status;
}
+ /* Only enable "bad" interrupts while NAPI is on */
+ iowrite32(new_mask, priv->base + FTGMAC100_OFFSET_IER);
+
+ /* Schedule NAPI bh */
+ napi_schedule_irqoff(&priv->napi);
+
return IRQ_HANDLED;
}
-/******************************************************************************
- * struct napi_struct functions
- *****************************************************************************/
+static bool ftgmac100_check_rx(struct ftgmac100 *priv)
+{
+ struct ftgmac100_rxdes *rxdes = &priv->rxdes[priv->rx_pointer];
+
+ /* Do we have a packet ? */
+ return !!(rxdes->rxdes0 & cpu_to_le32(FTGMAC100_RXDES0_RXPKT_RDY));
+}
+
static int ftgmac100_poll(struct napi_struct *napi, int budget)
{
struct ftgmac100 *priv = container_of(napi, struct ftgmac100, napi);
- struct net_device *netdev = priv->netdev;
- unsigned int status;
- bool completed = true;
- int rx = 0;
+ int work_done = 0;
+ bool more;
- status = ioread32(priv->base + FTGMAC100_OFFSET_ISR);
- iowrite32(status, priv->base + FTGMAC100_OFFSET_ISR);
-
- if (status & (FTGMAC100_INT_RPKT_BUF | FTGMAC100_INT_NO_RXBUF)) {
- /*
- * FTGMAC100_INT_RPKT_BUF:
- * RX DMA has received packets into RX buffer successfully
- *
- * FTGMAC100_INT_NO_RXBUF:
- * RX buffer unavailable
- */
- bool retry;
-
- do {
- retry = ftgmac100_rx_packet(priv, &rx);
- } while (retry && rx < budget);
-
- if (retry && rx == budget)
- completed = false;
- }
-
- if (status & (FTGMAC100_INT_XPKT_ETH | FTGMAC100_INT_XPKT_LOST)) {
- /*
- * FTGMAC100_INT_XPKT_ETH:
- * packet transmitted to ethernet successfully
- *
- * FTGMAC100_INT_XPKT_LOST:
- * packet transmitted to ethernet lost due to late
- * collision or excessive collision
- */
+ /* Handle TX completions */
+ if (ftgmac100_tx_buf_cleanable(priv))
ftgmac100_tx_complete(priv);
- }
- if (status & priv->int_mask_all & (FTGMAC100_INT_NO_RXBUF |
- FTGMAC100_INT_RPKT_LOST | FTGMAC100_INT_AHB_ERR)) {
- if (net_ratelimit())
- netdev_info(netdev, "[ISR] = 0x%x: %s%s%s\n", status,
- status & FTGMAC100_INT_NO_RXBUF ? "NO_RXBUF " : "",
- status & FTGMAC100_INT_RPKT_LOST ? "RPKT_LOST " : "",
- status & FTGMAC100_INT_AHB_ERR ? "AHB_ERR " : "");
+ /* Handle RX packets */
+ do {
+ more = ftgmac100_rx_packet(priv, &work_done);
+ } while (more && work_done < budget);
- if (status & FTGMAC100_INT_NO_RXBUF) {
- /* RX buffer unavailable */
- netdev->stats.rx_over_errors++;
- }
- if (status & FTGMAC100_INT_RPKT_LOST) {
- /* received packet lost due to RX FIFO full */
- netdev->stats.rx_fifo_errors++;
- }
- }
+ /* The interrupt is telling us to kick the MAC back to life
+ * after an RX overflow
+ */
+ if (unlikely(priv->need_mac_restart)) {
+ ftgmac100_start_hw(priv);
- if (completed) {
- napi_complete(napi);
-
- /* enable all interrupts */
- iowrite32(priv->int_mask_all,
+ /* Re-enable "bad" interrupts */
+ iowrite32(FTGMAC100_INT_BAD,
priv->base + FTGMAC100_OFFSET_IER);
}
- return rx;
+ /* As long as we are waiting for transmit packets to be
+ * completed we keep NAPI going
+ */
+ if (ftgmac100_tx_buf_cleanable(priv))
+ work_done = budget;
+
+ if (work_done < budget) {
+ /* We are about to re-enable all interrupts. However
+ * the HW has been latching RX/TX packet interrupts while
+ * they were masked. So we clear them first, then we need
+ * to re-check if there's something to process
+ */
+ iowrite32(FTGMAC100_INT_RXTX,
+ priv->base + FTGMAC100_OFFSET_ISR);
+
+ /* Push the above (and provides a barrier vs. subsequent
+ * reads of the descriptor).
+ */
+ ioread32(priv->base + FTGMAC100_OFFSET_ISR);
+
+ /* Check RX and TX descriptors for more work to do */
+ if (ftgmac100_check_rx(priv) ||
+ ftgmac100_tx_buf_cleanable(priv))
+ return budget;
+
+ /* deschedule NAPI */
+ napi_complete(napi);
+
+ /* enable all interrupts */
+ iowrite32(FTGMAC100_INT_ALL,
+ priv->base + FTGMAC100_OFFSET_IER);
+ }
+
+ return work_done;
}
-/******************************************************************************
- * struct net_device_ops functions
- *****************************************************************************/
+static int ftgmac100_init_all(struct ftgmac100 *priv, bool ignore_alloc_err)
+{
+ int err = 0;
+
+ /* Re-init descriptors (adjust queue sizes) */
+ ftgmac100_init_rings(priv);
+
+ /* Realloc rx descriptors */
+ err = ftgmac100_alloc_rx_buffers(priv);
+ if (err && !ignore_alloc_err)
+ return err;
+
+ /* Reinit and restart HW */
+ ftgmac100_init_hw(priv);
+ ftgmac100_config_pause(priv);
+ ftgmac100_start_hw(priv);
+
+ /* Re-enable the device */
+ napi_enable(&priv->napi);
+ netif_start_queue(priv->netdev);
+
+ /* Enable all interrupts */
+ iowrite32(FTGMAC100_INT_ALL, priv->base + FTGMAC100_OFFSET_IER);
+
+ return err;
+}
+
+static void ftgmac100_reset_task(struct work_struct *work)
+{
+ struct ftgmac100 *priv = container_of(work, struct ftgmac100,
+ reset_task);
+ struct net_device *netdev = priv->netdev;
+ int err;
+
+ netdev_dbg(netdev, "Resetting NIC...\n");
+
+ /* Lock the world */
+ rtnl_lock();
+ if (netdev->phydev)
+ mutex_lock(&netdev->phydev->lock);
+ if (priv->mii_bus)
+ mutex_lock(&priv->mii_bus->mdio_lock);
+
+
+ /* Check if the interface is still up */
+ if (!netif_running(netdev))
+ goto bail;
+
+ /* Stop the network stack */
+ netif_trans_update(netdev);
+ napi_disable(&priv->napi);
+ netif_tx_disable(netdev);
+
+ /* Stop and reset the MAC */
+ ftgmac100_stop_hw(priv);
+ err = ftgmac100_reset_and_config_mac(priv);
+ if (err) {
+ /* Not much we can do ... it might come back... */
+ netdev_err(netdev, "attempting to continue...\n");
+ }
+
+ /* Free all rx and tx buffers */
+ ftgmac100_free_buffers(priv);
+
+ /* Setup everything again and restart chip */
+ ftgmac100_init_all(priv, true);
+
+ netdev_dbg(netdev, "Reset done !\n");
+ bail:
+ if (priv->mii_bus)
+ mutex_unlock(&priv->mii_bus->mdio_lock);
+ if (netdev->phydev)
+ mutex_unlock(&netdev->phydev->lock);
+ rtnl_unlock();
+}
+
static int ftgmac100_open(struct net_device *netdev)
{
struct ftgmac100 *priv = netdev_priv(netdev);
- unsigned int status;
int err;
- err = ftgmac100_alloc_buffers(priv);
+ /* Allocate ring buffers */
+ err = ftgmac100_alloc_rings(priv);
if (err) {
- netdev_err(netdev, "failed to allocate buffers\n");
- goto err_alloc;
+ netdev_err(netdev, "Failed to allocate descriptors\n");
+ return err;
}
- err = request_irq(priv->irq, ftgmac100_interrupt, 0, netdev->name, netdev);
- if (err) {
- netdev_err(netdev, "failed to request irq %d\n", priv->irq);
- goto err_irq;
+ /* When using NC-SI we force the speed to 100Mbit/s full duplex,
+ *
+ * Otherwise we leave it set to 0 (no link), the link
+ * message from the PHY layer will handle setting it up to
+ * something else if needed.
+ */
+ if (priv->use_ncsi) {
+ priv->cur_duplex = DUPLEX_FULL;
+ priv->cur_speed = SPEED_100;
+ } else {
+ priv->cur_duplex = 0;
+ priv->cur_speed = 0;
}
- priv->rx_pointer = 0;
- priv->tx_clean_pointer = 0;
- priv->tx_pointer = 0;
- priv->tx_pending = 0;
-
- err = ftgmac100_reset_hw(priv);
+ /* Reset the hardware */
+ err = ftgmac100_reset_and_config_mac(priv);
if (err)
goto err_hw;
- ftgmac100_init_hw(priv);
- ftgmac100_start_hw(priv, priv->use_ncsi ? 100 : 10);
+ /* Initialize NAPI */
+ netif_napi_add(netdev, &priv->napi, ftgmac100_poll, 64);
- /* Clear stale interrupts */
- status = ioread32(priv->base + FTGMAC100_OFFSET_ISR);
- iowrite32(status, priv->base + FTGMAC100_OFFSET_ISR);
+ /* Grab our interrupt */
+ err = request_irq(netdev->irq, ftgmac100_interrupt, 0, netdev->name, netdev);
+ if (err) {
+ netdev_err(netdev, "failed to request irq %d\n", netdev->irq);
+ goto err_irq;
+ }
- if (netdev->phydev)
+ /* Start things up */
+ err = ftgmac100_init_all(priv, false);
+ if (err) {
+ netdev_err(netdev, "Failed to allocate packet buffers\n");
+ goto err_alloc;
+ }
+
+ if (netdev->phydev) {
+ /* If we have a PHY, start polling */
phy_start(netdev->phydev);
- else if (priv->use_ncsi)
+ } else if (priv->use_ncsi) {
+ /* If using NC-SI, set our carrier on and start the stack */
netif_carrier_on(netdev);
- napi_enable(&priv->napi);
- netif_start_queue(netdev);
-
- /* enable all interrupts */
- iowrite32(priv->int_mask_all, priv->base + FTGMAC100_OFFSET_IER);
-
- /* Start the NCSI device */
- if (priv->use_ncsi) {
+ /* Start the NCSI device */
err = ncsi_start_dev(priv->ndev);
if (err)
goto err_ncsi;
}
- priv->enabled = true;
-
return 0;
-err_ncsi:
+ err_ncsi:
napi_disable(&priv->napi);
netif_stop_queue(netdev);
- iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
-err_hw:
- free_irq(priv->irq, netdev);
-err_irq:
+ err_alloc:
ftgmac100_free_buffers(priv);
-err_alloc:
+ free_irq(netdev->irq, netdev);
+ err_irq:
+ netif_napi_del(&priv->napi);
+ err_hw:
+ iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
+ ftgmac100_free_rings(priv);
return err;
}
@@ -1181,56 +1528,33 @@
{
struct ftgmac100 *priv = netdev_priv(netdev);
- if (!priv->enabled)
- return 0;
+ /* Note about the reset task: We are called with the rtnl lock
+ * held, so we are synchronized against the core of the reset
+ * task. We must not try to synchronously cancel it otherwise
+ * we can deadlock. But since it will test for netif_running()
+ * which has already been cleared by the net core, we don't
+ * anything special to do.
+ */
/* disable all interrupts */
- priv->enabled = false;
iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
netif_stop_queue(netdev);
napi_disable(&priv->napi);
+ netif_napi_del(&priv->napi);
if (netdev->phydev)
phy_stop(netdev->phydev);
else if (priv->use_ncsi)
ncsi_stop_dev(priv->ndev);
ftgmac100_stop_hw(priv);
- free_irq(priv->irq, netdev);
+ free_irq(netdev->irq, netdev);
ftgmac100_free_buffers(priv);
+ ftgmac100_free_rings(priv);
return 0;
}
-static int ftgmac100_hard_start_xmit(struct sk_buff *skb,
- struct net_device *netdev)
-{
- struct ftgmac100 *priv = netdev_priv(netdev);
- dma_addr_t map;
-
- if (unlikely(skb->len > MAX_PKT_SIZE)) {
- if (net_ratelimit())
- netdev_dbg(netdev, "tx packet too big\n");
-
- netdev->stats.tx_dropped++;
- kfree_skb(skb);
- return NETDEV_TX_OK;
- }
-
- map = dma_map_single(priv->dev, skb->data, skb_headlen(skb), DMA_TO_DEVICE);
- if (unlikely(dma_mapping_error(priv->dev, map))) {
- /* drop packet */
- if (net_ratelimit())
- netdev_err(netdev, "map socket buffer failed\n");
-
- netdev->stats.tx_dropped++;
- kfree_skb(skb);
- return NETDEV_TX_OK;
- }
-
- return ftgmac100_xmit(priv, skb, map);
-}
-
/* optional */
static int ftgmac100_do_ioctl(struct net_device *netdev, struct ifreq *ifr, int cmd)
{
@@ -1240,6 +1564,52 @@
return phy_mii_ioctl(netdev->phydev, ifr, cmd);
}
+static void ftgmac100_tx_timeout(struct net_device *netdev)
+{
+ struct ftgmac100 *priv = netdev_priv(netdev);
+
+ /* Disable all interrupts */
+ iowrite32(0, priv->base + FTGMAC100_OFFSET_IER);
+
+ /* Do the reset outside of interrupt context */
+ schedule_work(&priv->reset_task);
+}
+
+static int ftgmac100_set_features(struct net_device *netdev,
+ netdev_features_t features)
+{
+ struct ftgmac100 *priv = netdev_priv(netdev);
+ netdev_features_t changed = netdev->features ^ features;
+
+ if (!netif_running(netdev))
+ return 0;
+
+ /* Update the vlan filtering bit */
+ if (changed & NETIF_F_HW_VLAN_CTAG_RX) {
+ u32 maccr;
+
+ maccr = ioread32(priv->base + FTGMAC100_OFFSET_MACCR);
+ if (priv->netdev->features & NETIF_F_HW_VLAN_CTAG_RX)
+ maccr |= FTGMAC100_MACCR_RM_VLAN;
+ else
+ maccr &= ~FTGMAC100_MACCR_RM_VLAN;
+ iowrite32(maccr, priv->base + FTGMAC100_OFFSET_MACCR);
+ }
+
+ return 0;
+}
+
+#ifdef CONFIG_NET_POLL_CONTROLLER
+static void ftgmac100_poll_controller(struct net_device *netdev)
+{
+ unsigned long flags;
+
+ local_irq_save(flags);
+ ftgmac100_interrupt(netdev->irq, netdev);
+ local_irq_restore(flags);
+}
+#endif
+
static const struct net_device_ops ftgmac100_netdev_ops = {
.ndo_open = ftgmac100_open,
.ndo_stop = ftgmac100_stop,
@@ -1247,12 +1617,20 @@
.ndo_set_mac_address = ftgmac100_set_mac_addr,
.ndo_validate_addr = eth_validate_addr,
.ndo_do_ioctl = ftgmac100_do_ioctl,
+ .ndo_tx_timeout = ftgmac100_tx_timeout,
+ .ndo_set_rx_mode = ftgmac100_set_rx_mode,
+ .ndo_set_features = ftgmac100_set_features,
+#ifdef CONFIG_NET_POLL_CONTROLLER
+ .ndo_poll_controller = ftgmac100_poll_controller,
+#endif
};
static int ftgmac100_setup_mdio(struct net_device *netdev)
{
struct ftgmac100 *priv = netdev_priv(netdev);
struct platform_device *pdev = to_platform_device(priv->dev);
+ int phy_intf = PHY_INTERFACE_MODE_RGMII;
+ struct device_node *np = pdev->dev.of_node;
int i, err = 0;
u32 reg;
@@ -1261,14 +1639,46 @@
if (!priv->mii_bus)
return -EIO;
- if (of_machine_is_compatible("aspeed,ast2400") ||
- of_machine_is_compatible("aspeed,ast2500")) {
+ if (priv->is_aspeed) {
/* This driver supports the old MDIO interface */
reg = ioread32(priv->base + FTGMAC100_OFFSET_REVR);
reg &= ~FTGMAC100_REVR_NEW_MDIO_INTERFACE;
iowrite32(reg, priv->base + FTGMAC100_OFFSET_REVR);
};
+ /* Get PHY mode from device-tree */
+ if (np) {
+ /* Default to RGMII. It's a gigabit part after all */
+ phy_intf = of_get_phy_mode(np);
+ if (phy_intf < 0)
+ phy_intf = PHY_INTERFACE_MODE_RGMII;
+
+ /* Aspeed only supports these. I don't know about other IP
+ * block vendors so I'm going to just let them through for
+ * now. Note that this is only a warning if for some obscure
+ * reason the DT really means to lie about it or it's a newer
+ * part we don't know about.
+ *
+ * On the Aspeed SoC there are additionally straps and SCU
+ * control bits that could tell us what the interface is
+ * (or allow us to configure it while the IP block is held
+ * in reset). For now I chose to keep this driver away from
+ * those SoC specific bits and assume the device-tree is
+ * right and the SCU has been configured properly by pinmux
+ * or the firmware.
+ */
+ if (priv->is_aspeed &&
+ phy_intf != PHY_INTERFACE_MODE_RMII &&
+ phy_intf != PHY_INTERFACE_MODE_RGMII &&
+ phy_intf != PHY_INTERFACE_MODE_RGMII_ID &&
+ phy_intf != PHY_INTERFACE_MODE_RGMII_RXID &&
+ phy_intf != PHY_INTERFACE_MODE_RGMII_TXID) {
+ netdev_warn(netdev,
+ "Unsupported PHY mode %s !\n",
+ phy_modes(phy_intf));
+ }
+ }
+
priv->mii_bus->name = "ftgmac100_mdio";
snprintf(priv->mii_bus->id, MII_BUS_ID_SIZE, "%s-%d",
pdev->name, pdev->id);
@@ -1285,7 +1695,7 @@
goto err_register_mdiobus;
}
- err = ftgmac100_mii_probe(priv);
+ err = ftgmac100_mii_probe(priv, phy_intf);
if (err) {
dev_err(priv->dev, "MII Probe failed!\n");
goto err_mii_probe;
@@ -1321,15 +1731,13 @@
nd->link_up ? "up" : "down");
}
-/******************************************************************************
- * struct platform_driver functions
- *****************************************************************************/
static int ftgmac100_probe(struct platform_device *pdev)
{
struct resource *res;
int irq;
struct net_device *netdev;
struct ftgmac100 *priv;
+ struct device_node *np;
int err = 0;
if (!pdev)
@@ -1354,6 +1762,7 @@
netdev->ethtool_ops = &ftgmac100_ethtool_ops;
netdev->netdev_ops = &ftgmac100_netdev_ops;
+ netdev->watchdog_timeo = 5 * HZ;
platform_set_drvdata(pdev, netdev);
@@ -1361,11 +1770,7 @@
priv = netdev_priv(netdev);
priv->netdev = netdev;
priv->dev = &pdev->dev;
-
- spin_lock_init(&priv->tx_lock);
-
- /* initialize NAPI */
- netif_napi_add(netdev, &priv->napi, ftgmac100_poll, 64);
+ INIT_WORK(&priv->reset_task, ftgmac100_reset_task);
/* map io memory */
priv->res = request_mem_region(res->start, resource_size(res),
@@ -1383,29 +1788,28 @@
goto err_ioremap;
}
- priv->irq = irq;
+ netdev->irq = irq;
+
+ /* Enable pause */
+ priv->tx_pause = true;
+ priv->rx_pause = true;
+ priv->aneg_pause = true;
/* MAC address from chip or random one */
- ftgmac100_setup_mac(priv);
+ ftgmac100_initial_mac(priv);
- priv->int_mask_all = (FTGMAC100_INT_RPKT_LOST |
- FTGMAC100_INT_XPKT_ETH |
- FTGMAC100_INT_XPKT_LOST |
- FTGMAC100_INT_AHB_ERR |
- FTGMAC100_INT_RPKT_BUF |
- FTGMAC100_INT_NO_RXBUF);
-
- if (of_machine_is_compatible("aspeed,ast2400") ||
- of_machine_is_compatible("aspeed,ast2500")) {
+ np = pdev->dev.of_node;
+ if (np && (of_device_is_compatible(np, "aspeed,ast2400-mac") ||
+ of_device_is_compatible(np, "aspeed,ast2500-mac"))) {
priv->rxdes0_edorr_mask = BIT(30);
priv->txdes0_edotr_mask = BIT(30);
+ priv->is_aspeed = true;
} else {
priv->rxdes0_edorr_mask = BIT(15);
priv->txdes0_edotr_mask = BIT(15);
}
- if (pdev->dev.of_node &&
- of_get_property(pdev->dev.of_node, "use-ncsi", NULL)) {
+ if (np && of_get_property(np, "use-ncsi", NULL)) {
if (!IS_ENABLED(CONFIG_NET_NCSI)) {
dev_err(&pdev->dev, "NCSI stack not enabled\n");
goto err_ncsi_dev;
@@ -1423,15 +1827,21 @@
goto err_setup_mdio;
}
- /* We have to disable on-chip IP checksum functionality
- * when NCSI is enabled on the interface. It doesn't work
- * in that case.
- */
- netdev->features = NETIF_F_IP_CSUM | NETIF_F_GRO;
- if (priv->use_ncsi &&
- of_get_property(pdev->dev.of_node, "no-hw-checksum", NULL))
- netdev->features &= ~NETIF_F_IP_CSUM;
+ /* Default ring sizes */
+ priv->rx_q_entries = priv->new_rx_q_entries = DEF_RX_QUEUE_ENTRIES;
+ priv->tx_q_entries = priv->new_tx_q_entries = DEF_TX_QUEUE_ENTRIES;
+ /* Base feature set */
+ netdev->hw_features = NETIF_F_RXCSUM | NETIF_F_HW_CSUM |
+ NETIF_F_GRO | NETIF_F_SG | NETIF_F_HW_VLAN_CTAG_RX |
+ NETIF_F_HW_VLAN_CTAG_TX;
+
+ /* AST2400 doesn't have working HW checksum generation */
+ if (np && (of_device_is_compatible(np, "aspeed,ast2400-mac")))
+ netdev->hw_features &= ~NETIF_F_HW_CSUM;
+ if (np && of_get_property(np, "no-hw-checksum", NULL))
+ netdev->hw_features &= ~(NETIF_F_HW_CSUM | NETIF_F_RXCSUM);
+ netdev->features |= netdev->hw_features;
/* register network device */
err = register_netdev(netdev);
@@ -1440,7 +1850,7 @@
goto err_register_netdev;
}
- netdev_info(netdev, "irq %d, mapped at %p\n", priv->irq, priv->base);
+ netdev_info(netdev, "irq %d, mapped at %p\n", netdev->irq, priv->base);
return 0;
@@ -1467,6 +1877,12 @@
priv = netdev_priv(netdev);
unregister_netdev(netdev);
+
+ /* There's a small chance the reset task will have been re-queued,
+ * during stop, make sure it's gone before we free the structure.
+ */
+ cancel_work_sync(&priv->reset_task);
+
ftgmac100_destroy_mdio(netdev);
iounmap(priv->base);
diff --git a/drivers/net/ethernet/faraday/ftgmac100.h b/drivers/net/ethernet/faraday/ftgmac100.h
index a7ce0ac..0653d81 100644
--- a/drivers/net/ethernet/faraday/ftgmac100.h
+++ b/drivers/net/ethernet/faraday/ftgmac100.h
@@ -86,6 +86,20 @@
#define FTGMAC100_INT_PHYSTS_CHG (1 << 9)
#define FTGMAC100_INT_NO_HPTXBUF (1 << 10)
+/* Interrupts we care about in NAPI mode */
+#define FTGMAC100_INT_BAD (FTGMAC100_INT_RPKT_LOST | \
+ FTGMAC100_INT_XPKT_LOST | \
+ FTGMAC100_INT_AHB_ERR | \
+ FTGMAC100_INT_NO_RXBUF)
+
+/* Normal RX/TX interrupts, enabled when NAPI off */
+#define FTGMAC100_INT_RXTX (FTGMAC100_INT_XPKT_ETH | \
+ FTGMAC100_INT_RPKT_BUF)
+
+/* All the interrupts we care about */
+#define FTGMAC100_INT_ALL (FTGMAC100_INT_RPKT_BUF | \
+ FTGMAC100_INT_BAD)
+
/*
* Interrupt timer control register
*/
@@ -185,13 +199,20 @@
#define FTGMAC100_PHYDATA_MIIRDATA(phydata) (((phydata) >> 16) & 0xffff)
/*
+ * Flow control register
+ */
+#define FTGMAC100_FCR_FC_EN (1 << 0)
+#define FTGMAC100_FCR_FCTHR_EN (1 << 2)
+#define FTGMAC100_FCR_PAUSE_TIME(x) (((x) & 0xffff) << 16)
+
+/*
* Transmit descriptor, aligned to 16 bytes
*/
struct ftgmac100_txdes {
- unsigned int txdes0;
- unsigned int txdes1;
- unsigned int txdes2; /* not used by HW */
- unsigned int txdes3; /* TXBUF_BADR */
+ __le32 txdes0; /* Control & status bits */
+ __le32 txdes1; /* Irq, checksum and vlan control */
+ __le32 txdes2; /* Reserved */
+ __le32 txdes3; /* DMA buffer address */
} __attribute__ ((aligned(16)));
#define FTGMAC100_TXDES0_TXBUF_SIZE(x) ((x) & 0x3fff)
@@ -213,10 +234,10 @@
* Receive descriptor, aligned to 16 bytes
*/
struct ftgmac100_rxdes {
- unsigned int rxdes0;
- unsigned int rxdes1;
- unsigned int rxdes2; /* not used by HW */
- unsigned int rxdes3; /* RXBUF_BADR */
+ __le32 rxdes0; /* Control & status bits */
+ __le32 rxdes1; /* Checksum and vlan status */
+ __le32 rxdes2; /* length/type on AST2500 */
+ __le32 rxdes3; /* DMA buffer address */
} __attribute__ ((aligned(16)));
#define FTGMAC100_RXDES0_VDBC 0x3fff
@@ -234,6 +255,14 @@
#define FTGMAC100_RXDES0_FRS (1 << 29)
#define FTGMAC100_RXDES0_RXPKT_RDY (1 << 31)
+/* Errors we care about for dropping packets */
+#define RXDES0_ANY_ERROR ( \
+ FTGMAC100_RXDES0_RX_ERR | \
+ FTGMAC100_RXDES0_CRC_ERR | \
+ FTGMAC100_RXDES0_FTL | \
+ FTGMAC100_RXDES0_RUNT | \
+ FTGMAC100_RXDES0_RX_ODD_NB)
+
#define FTGMAC100_RXDES1_VLANTAG_CI 0xffff
#define FTGMAC100_RXDES1_PROT_MASK (0x3 << 20)
#define FTGMAC100_RXDES1_PROT_NONIP (0x0 << 20)
diff --git a/drivers/net/ethernet/freescale/dpaa/dpaa_eth.c b/drivers/net/ethernet/freescale/dpaa/dpaa_eth.c
index e2ca107..9a520e4 100644
--- a/drivers/net/ethernet/freescale/dpaa/dpaa_eth.c
+++ b/drivers/net/ethernet/freescale/dpaa/dpaa_eth.c
@@ -137,6 +137,13 @@
/* L4 Type field: TCP */
#define FM_L4_PARSE_RESULT_TCP 0x20
+/* FD status field indicating whether the FM Parser has attempted to validate
+ * the L4 csum of the frame.
+ * Note that having this bit set doesn't necessarily imply that the checksum
+ * is valid. One would have to check the parse results to find that out.
+ */
+#define FM_FD_STAT_L4CV 0x00000004
+
#define DPAA_SGT_MAX_ENTRIES 16 /* maximum number of entries in SG Table */
#define DPAA_BUFF_RELEASE_MAX 8 /* maximum number of buffers released at once */
@@ -235,6 +242,7 @@
* For conformity, we'll still declare GSO explicitly.
*/
net_dev->features |= NETIF_F_GSO;
+ net_dev->features |= NETIF_F_RXCSUM;
net_dev->priv_flags |= IFF_LIVE_ADDR_CHANGE;
/* we do not want shared skbs on TX */
@@ -334,6 +342,45 @@
}
}
+static int dpaa_setup_tc(struct net_device *net_dev, u32 handle, __be16 proto,
+ struct tc_to_netdev *tc)
+{
+ struct dpaa_priv *priv = netdev_priv(net_dev);
+ u8 num_tc;
+ int i;
+
+ if (tc->type != TC_SETUP_MQPRIO)
+ return -EINVAL;
+
+ tc->mqprio->hw = TC_MQPRIO_HW_OFFLOAD_TCS;
+ num_tc = tc->mqprio->num_tc;
+
+ if (num_tc == priv->num_tc)
+ return 0;
+
+ if (!num_tc) {
+ netdev_reset_tc(net_dev);
+ goto out;
+ }
+
+ if (num_tc > DPAA_TC_NUM) {
+ netdev_err(net_dev, "Too many traffic classes: max %d supported.\n",
+ DPAA_TC_NUM);
+ return -EINVAL;
+ }
+
+ netdev_set_num_tc(net_dev, num_tc);
+
+ for (i = 0; i < num_tc; i++)
+ netdev_set_tc_queue(net_dev, i, DPAA_TC_TXQ_NUM,
+ i * DPAA_TC_TXQ_NUM);
+
+out:
+ priv->num_tc = num_tc ? : 1;
+ netif_set_real_num_tx_queues(net_dev, priv->num_tc * DPAA_TC_TXQ_NUM);
+ return 0;
+}
+
static struct mac_device *dpaa_mac_dev_get(struct platform_device *pdev)
{
struct platform_device *of_dev;
@@ -557,16 +604,18 @@
/* Use multiple WQs for FQ assignment:
* - Tx Confirmation queues go to WQ1.
- * - Rx Error and Tx Error queues go to WQ2 (giving them a better chance
- * to be scheduled, in case there are many more FQs in WQ3).
- * - Rx Default and Tx queues go to WQ3 (no differentiation between
- * Rx and Tx traffic).
+ * - Rx Error and Tx Error queues go to WQ5 (giving them a better chance
+ * to be scheduled, in case there are many more FQs in WQ6).
+ * - Rx Default goes to WQ6.
+ * - Tx queues go to different WQs depending on their priority. Equal
+ * chunks of NR_CPUS queues go to WQ6 (lowest priority), WQ2, WQ1 and
+ * WQ0 (highest priority).
* This ensures that Tx-confirmed buffers are timely released. In particular,
* it avoids congestion on the Tx Confirm FQs, which can pile up PFDRs if they
* are greatly outnumbered by other FQs in the system, while
* dequeue scheduling is round-robin.
*/
-static inline void dpaa_assign_wq(struct dpaa_fq *fq)
+static inline void dpaa_assign_wq(struct dpaa_fq *fq, int idx)
{
switch (fq->fq_type) {
case FQ_TYPE_TX_CONFIRM:
@@ -575,11 +624,33 @@
break;
case FQ_TYPE_RX_ERROR:
case FQ_TYPE_TX_ERROR:
- fq->wq = 2;
+ fq->wq = 5;
break;
case FQ_TYPE_RX_DEFAULT:
+ fq->wq = 6;
+ break;
case FQ_TYPE_TX:
- fq->wq = 3;
+ switch (idx / DPAA_TC_TXQ_NUM) {
+ case 0:
+ /* Low priority (best effort) */
+ fq->wq = 6;
+ break;
+ case 1:
+ /* Medium priority */
+ fq->wq = 2;
+ break;
+ case 2:
+ /* High priority */
+ fq->wq = 1;
+ break;
+ case 3:
+ /* Very high priority */
+ fq->wq = 0;
+ break;
+ default:
+ WARN(1, "Too many TX FQs: more than %d!\n",
+ DPAA_ETH_TXQ_NUM);
+ }
break;
default:
WARN(1, "Invalid FQ type %d for FQID %d!\n",
@@ -607,7 +678,7 @@
}
for (i = 0; i < count; i++)
- dpaa_assign_wq(dpaa_fq + i);
+ dpaa_assign_wq(dpaa_fq + i, i);
return dpaa_fq;
}
@@ -903,7 +974,7 @@
* Tx Confirmation FQs.
*/
if (dpaa_fq->fq_type == FQ_TYPE_TX_CONFIRM)
- initfq.fqd.fq_ctrl |= cpu_to_be16(QM_FQCTRL_HOLDACTIVE);
+ initfq.fqd.fq_ctrl |= cpu_to_be16(QM_FQCTRL_AVOIDBLOCK);
/* FQ placement */
initfq.we_mask |= cpu_to_be16(QM_INITFQ_WE_DESTWQ);
@@ -985,7 +1056,8 @@
/* Initialization common to all ingress queues */
if (dpaa_fq->flags & QMAN_FQ_FLAG_NO_ENQUEUE) {
initfq.we_mask |= cpu_to_be16(QM_INITFQ_WE_CONTEXTA);
- initfq.fqd.fq_ctrl |= cpu_to_be16(QM_FQCTRL_HOLDACTIVE);
+ initfq.fqd.fq_ctrl |= cpu_to_be16(QM_FQCTRL_HOLDACTIVE |
+ QM_FQCTRL_CTXASTASHING);
initfq.fqd.context_a.stashing.exclusive =
QM_STASHING_EXCL_DATA | QM_STASHING_EXCL_CTX |
QM_STASHING_EXCL_ANNOTATION;
@@ -1055,9 +1127,9 @@
return err;
}
-static void dpaa_eth_init_tx_port(struct fman_port *port, struct dpaa_fq *errq,
- struct dpaa_fq *defq,
- struct dpaa_buffer_layout *buf_layout)
+static int dpaa_eth_init_tx_port(struct fman_port *port, struct dpaa_fq *errq,
+ struct dpaa_fq *defq,
+ struct dpaa_buffer_layout *buf_layout)
{
struct fman_buffer_prefix_content buf_prefix_content;
struct fman_port_params params;
@@ -1076,23 +1148,29 @@
params.specific_params.non_rx_params.dflt_fqid = defq->fqid;
err = fman_port_config(port, ¶ms);
- if (err)
+ if (err) {
pr_err("%s: fman_port_config failed\n", __func__);
+ return err;
+ }
err = fman_port_cfg_buf_prefix_content(port, &buf_prefix_content);
- if (err)
+ if (err) {
pr_err("%s: fman_port_cfg_buf_prefix_content failed\n",
__func__);
+ return err;
+ }
err = fman_port_init(port);
if (err)
pr_err("%s: fm_port_init failed\n", __func__);
+
+ return err;
}
-static void dpaa_eth_init_rx_port(struct fman_port *port, struct dpaa_bp **bps,
- size_t count, struct dpaa_fq *errq,
- struct dpaa_fq *defq,
- struct dpaa_buffer_layout *buf_layout)
+static int dpaa_eth_init_rx_port(struct fman_port *port, struct dpaa_bp **bps,
+ size_t count, struct dpaa_fq *errq,
+ struct dpaa_fq *defq,
+ struct dpaa_buffer_layout *buf_layout)
{
struct fman_buffer_prefix_content buf_prefix_content;
struct fman_port_rx_params *rx_p;
@@ -1120,32 +1198,44 @@
}
err = fman_port_config(port, ¶ms);
- if (err)
+ if (err) {
pr_err("%s: fman_port_config failed\n", __func__);
+ return err;
+ }
err = fman_port_cfg_buf_prefix_content(port, &buf_prefix_content);
- if (err)
+ if (err) {
pr_err("%s: fman_port_cfg_buf_prefix_content failed\n",
__func__);
+ return err;
+ }
err = fman_port_init(port);
if (err)
pr_err("%s: fm_port_init failed\n", __func__);
+
+ return err;
}
-static void dpaa_eth_init_ports(struct mac_device *mac_dev,
- struct dpaa_bp **bps, size_t count,
- struct fm_port_fqs *port_fqs,
- struct dpaa_buffer_layout *buf_layout,
- struct device *dev)
+static int dpaa_eth_init_ports(struct mac_device *mac_dev,
+ struct dpaa_bp **bps, size_t count,
+ struct fm_port_fqs *port_fqs,
+ struct dpaa_buffer_layout *buf_layout,
+ struct device *dev)
{
struct fman_port *rxport = mac_dev->port[RX];
struct fman_port *txport = mac_dev->port[TX];
+ int err;
- dpaa_eth_init_tx_port(txport, port_fqs->tx_errq,
- port_fqs->tx_defq, &buf_layout[TX]);
- dpaa_eth_init_rx_port(rxport, bps, count, port_fqs->rx_errq,
- port_fqs->rx_defq, &buf_layout[RX]);
+ err = dpaa_eth_init_tx_port(txport, port_fqs->tx_errq,
+ port_fqs->tx_defq, &buf_layout[TX]);
+ if (err)
+ return err;
+
+ err = dpaa_eth_init_rx_port(rxport, bps, count, port_fqs->rx_errq,
+ port_fqs->rx_defq, &buf_layout[RX]);
+
+ return err;
}
static int dpaa_bman_release(const struct dpaa_bp *dpaa_bp,
@@ -1526,6 +1616,23 @@
return skb;
}
+static u8 rx_csum_offload(const struct dpaa_priv *priv, const struct qm_fd *fd)
+{
+ /* The parser has run and performed L4 checksum validation.
+ * We know there were no parser errors (and implicitly no
+ * L4 csum error), otherwise we wouldn't be here.
+ */
+ if ((priv->net_dev->features & NETIF_F_RXCSUM) &&
+ (be32_to_cpu(fd->status) & FM_FD_STAT_L4CV))
+ return CHECKSUM_UNNECESSARY;
+
+ /* We're here because either the parser didn't run or the L4 checksum
+ * was not verified. This may include the case of a UDP frame with
+ * checksum zero or an L4 proto other than TCP/UDP
+ */
+ return CHECKSUM_NONE;
+}
+
/* Build a linear skb around the received buffer.
* We are guaranteed there is enough room at the end of the data buffer to
* accommodate the shared info area of the skb.
@@ -1556,7 +1663,7 @@
skb_reserve(skb, fd_off);
skb_put(skb, qm_fd_get_length(fd));
- skb->ip_summed = CHECKSUM_NONE;
+ skb->ip_summed = rx_csum_offload(priv, fd);
return skb;
@@ -1616,7 +1723,7 @@
if (WARN_ON(unlikely(!skb)))
goto free_buffers;
- skb->ip_summed = CHECKSUM_NONE;
+ skb->ip_summed = rx_csum_offload(priv, fd);
/* Make sure forwarded skbs will have enough space
* on Tx, if extra headers are added.
@@ -2093,7 +2200,7 @@
dma_addr_t addr = qm_fd_addr(fd);
enum qm_fd_format fd_format;
struct net_device *net_dev;
- u32 fd_status = fd->status;
+ u32 fd_status;
struct dpaa_bp *dpaa_bp;
struct dpaa_priv *priv;
unsigned int skb_len;
@@ -2350,6 +2457,7 @@
.ndo_validate_addr = eth_validate_addr,
.ndo_set_rx_mode = dpaa_set_rx_mode,
.ndo_do_ioctl = dpaa_ioctl,
+ .ndo_setup_tc = dpaa_setup_tc,
};
static int dpaa_napi_add(struct net_device *net_dev)
@@ -2624,8 +2732,10 @@
priv->rx_headroom = dpaa_get_headroom(&priv->buf_layout[RX]);
/* All real interfaces need their ports initialized */
- dpaa_eth_init_ports(mac_dev, dpaa_bps, DPAA_BPS_NUM, &port_fqs,
- &priv->buf_layout[0], dev);
+ err = dpaa_eth_init_ports(mac_dev, dpaa_bps, DPAA_BPS_NUM, &port_fqs,
+ &priv->buf_layout[0], dev);
+ if (err)
+ goto init_ports_failed;
priv->percpu_priv = devm_alloc_percpu(dev, *priv->percpu_priv);
if (!priv->percpu_priv) {
@@ -2638,6 +2748,9 @@
memset(percpu_priv, 0, sizeof(*percpu_priv));
}
+ priv->num_tc = 1;
+ netif_set_real_num_tx_queues(net_dev, priv->num_tc * DPAA_TC_TXQ_NUM);
+
/* Initialize NAPI */
err = dpaa_napi_add(net_dev);
if (err < 0)
@@ -2658,6 +2771,7 @@
napi_add_failed:
dpaa_napi_del(net_dev);
alloc_percpu_failed:
+init_ports_failed:
dpaa_fq_free(dev, &priv->dpaa_fq_list);
fq_alloc_failed:
qman_delete_cgr_safe(&priv->ingress_cgr);
diff --git a/drivers/net/ethernet/freescale/dpaa/dpaa_eth.h b/drivers/net/ethernet/freescale/dpaa/dpaa_eth.h
index 1f9aebf..9941a78 100644
--- a/drivers/net/ethernet/freescale/dpaa/dpaa_eth.h
+++ b/drivers/net/ethernet/freescale/dpaa/dpaa_eth.h
@@ -39,7 +39,12 @@
#include "mac.h"
#include "dpaa_eth_trace.h"
-#define DPAA_ETH_TXQ_NUM NR_CPUS
+/* Number of prioritised traffic classes */
+#define DPAA_TC_NUM 4
+/* Number of Tx queues per traffic class */
+#define DPAA_TC_TXQ_NUM NR_CPUS
+/* Total number of Tx queues */
+#define DPAA_ETH_TXQ_NUM (DPAA_TC_NUM * DPAA_TC_TXQ_NUM)
#define DPAA_BPS_NUM 3 /* number of bpools per interface */
@@ -152,6 +157,7 @@
u16 channel;
struct list_head dpaa_fq_list;
+ u8 num_tc;
u32 msg_enable; /* net_device message level */
struct {
diff --git a/drivers/net/ethernet/freescale/fec_main.c b/drivers/net/ethernet/freescale/fec_main.c
index 91a1664..56a563f 100644
--- a/drivers/net/ethernet/freescale/fec_main.c
+++ b/drivers/net/ethernet/freescale/fec_main.c
@@ -117,8 +117,9 @@
.name = "imx6ul-fec",
.driver_data = FEC_QUIRK_ENET_MAC | FEC_QUIRK_HAS_GBIT |
FEC_QUIRK_HAS_BUFDESC_EX | FEC_QUIRK_HAS_CSUM |
- FEC_QUIRK_HAS_VLAN | FEC_QUIRK_BUG_CAPTURE |
- FEC_QUIRK_HAS_RACC | FEC_QUIRK_HAS_COALESCE,
+ FEC_QUIRK_HAS_VLAN | FEC_QUIRK_ERR007885 |
+ FEC_QUIRK_BUG_CAPTURE | FEC_QUIRK_HAS_RACC |
+ FEC_QUIRK_HAS_COALESCE,
}, {
/* sentinel */
}
@@ -235,14 +236,14 @@
struct bufdesc_prop *bd)
{
return (bdp >= bd->last) ? bd->base
- : (struct bufdesc *)(((unsigned)bdp) + bd->dsize);
+ : (struct bufdesc *)(((void *)bdp) + bd->dsize);
}
static struct bufdesc *fec_enet_get_prevdesc(struct bufdesc *bdp,
struct bufdesc_prop *bd)
{
return (bdp <= bd->base) ? bd->last
- : (struct bufdesc *)(((unsigned)bdp) - bd->dsize);
+ : (struct bufdesc *)(((void *)bdp) - bd->dsize);
}
static int fec_enet_get_bd_index(struct bufdesc *bdp,
@@ -1266,7 +1267,7 @@
}
}
- /* ERR006538: Keep the transmitter going */
+ /* ERR006358: Keep the transmitter going */
if (bdp != txq->bd.cur &&
readl(txq->bd.reg_desc_active) == 0)
writel(0, txq->bd.reg_desc_active);
@@ -2001,7 +2002,7 @@
mii_speed--;
if (mii_speed > 63) {
dev_err(&pdev->dev,
- "fec clock (%lu) to fast to get right mii speed\n",
+ "fec clock (%lu) too fast to get right mii speed\n",
clk_get_rate(fep->clk_ipg));
err = -EINVAL;
goto err_out;
@@ -2651,7 +2652,7 @@
for (i = 0; i < fep->num_tx_queues; i++)
if (fep->tx_queue[i] && fep->tx_queue[i]->tso_hdrs) {
txq = fep->tx_queue[i];
- dma_free_coherent(NULL,
+ dma_free_coherent(&fep->pdev->dev,
txq->bd.ring_size * TSO_HEADER_SIZE,
txq->tso_hdrs,
txq->tso_hdrs_dma);
@@ -2685,7 +2686,7 @@
txq->tx_wake_threshold =
(txq->bd.ring_size - txq->tx_stop_threshold) / 2;
- txq->tso_hdrs = dma_alloc_coherent(NULL,
+ txq->tso_hdrs = dma_alloc_coherent(&fep->pdev->dev,
txq->bd.ring_size * TSO_HEADER_SIZE,
&txq->tso_hdrs_dma,
GFP_KERNEL);
@@ -2947,7 +2948,7 @@
}
/* only upper 6 bits (FEC_HASH_BITS) are used
- * which point to specific bit in he hash registers
+ * which point to specific bit in the hash registers
*/
hash = (crc >> (32 - FEC_HASH_BITS)) & 0x3f;
@@ -3187,7 +3188,7 @@
}
#ifdef CONFIG_OF
-static void fec_reset_phy(struct platform_device *pdev)
+static int fec_reset_phy(struct platform_device *pdev)
{
int err, phy_reset;
bool active_high = false;
@@ -3195,16 +3196,18 @@
struct device_node *np = pdev->dev.of_node;
if (!np)
- return;
+ return 0;
- of_property_read_u32(np, "phy-reset-duration", &msec);
+ err = of_property_read_u32(np, "phy-reset-duration", &msec);
/* A sane reset duration should not be longer than 1s */
- if (msec > 1000)
+ if (!err && msec > 1000)
msec = 1;
phy_reset = of_get_named_gpio(np, "phy-reset-gpios", 0);
- if (!gpio_is_valid(phy_reset))
- return;
+ if (phy_reset == -EPROBE_DEFER)
+ return phy_reset;
+ else if (!gpio_is_valid(phy_reset))
+ return 0;
active_high = of_property_read_bool(np, "phy-reset-active-high");
@@ -3213,7 +3216,7 @@
"phy-reset");
if (err) {
dev_err(&pdev->dev, "failed to get phy-reset-gpios: %d\n", err);
- return;
+ return err;
}
if (msec > 20)
@@ -3222,14 +3225,17 @@
usleep_range(msec * 1000, msec * 1000 + 1000);
gpio_set_value_cansleep(phy_reset, !active_high);
+
+ return 0;
}
#else /* CONFIG_OF */
-static void fec_reset_phy(struct platform_device *pdev)
+static int fec_reset_phy(struct platform_device *pdev)
{
/*
* In case of platform probe, the reset has been done
* by machine code.
*/
+ return 0;
}
#endif /* CONFIG_OF */
@@ -3400,6 +3406,7 @@
if (ret) {
dev_err(&pdev->dev,
"Failed to enable phy regulator: %d\n", ret);
+ clk_disable_unprepare(fep->clk_ipg);
goto failed_regulator;
}
} else {
@@ -3412,7 +3419,9 @@
pm_runtime_set_active(&pdev->dev);
pm_runtime_enable(&pdev->dev);
- fec_reset_phy(pdev);
+ ret = fec_reset_phy(pdev);
+ if (ret)
+ goto failed_reset;
if (fep->bufdesc_ex)
fec_ptp_init(pdev);
@@ -3473,8 +3482,10 @@
fec_ptp_stop(pdev);
if (fep->reg_phy)
regulator_disable(fep->reg_phy);
+failed_reset:
+ pm_runtime_put(&pdev->dev);
+ pm_runtime_disable(&pdev->dev);
failed_regulator:
- clk_disable_unprepare(fep->clk_ipg);
failed_clk_ipg:
fec_enet_clk_enable(ndev, false);
failed_clk:
diff --git a/drivers/net/ethernet/freescale/fman/fman.c b/drivers/net/ethernet/freescale/fman/fman.c
index f60845f..4aefe24 100644
--- a/drivers/net/ethernet/freescale/fman/fman.c
+++ b/drivers/net/ethernet/freescale/fman/fman.c
@@ -59,6 +59,7 @@
#define DMA_OFFSET 0x000C2000
#define FPM_OFFSET 0x000C3000
#define IMEM_OFFSET 0x000C4000
+#define HWP_OFFSET 0x000C7000
#define CGP_OFFSET 0x000DB000
/* Exceptions bit map */
@@ -218,6 +219,9 @@
#define QMI_GS_HALT_NOT_BUSY 0x00000002
+/* HWP defines */
+#define HWP_RPIMAC_PEN 0x00000001
+
/* IRAM defines */
#define IRAM_IADD_AIE 0x80000000
#define IRAM_READY 0x80000000
@@ -475,6 +479,12 @@
u32 res00e0[0x400 - 56];
};
+struct fman_hwp_regs {
+ u32 res0000[0x844 / 4]; /* 0x000..0x843 */
+ u32 fmprrpimac; /* FM Parser Internal memory access control */
+ u32 res[(0x1000 - 0x848) / 4]; /* 0x848..0xFFF */
+};
+
/* Structure that holds current FMan state.
* Used for saving run time information.
*/
@@ -606,6 +616,7 @@
struct fman_bmi_regs __iomem *bmi_regs;
struct fman_qmi_regs __iomem *qmi_regs;
struct fman_dma_regs __iomem *dma_regs;
+ struct fman_hwp_regs __iomem *hwp_regs;
fman_exceptions_cb *exception_cb;
fman_bus_error_cb *bus_error_cb;
/* Spinlock for FMan use */
@@ -999,6 +1010,12 @@
iowrite32be(tmp_reg, &qmi_rg->fmqm_ien);
}
+static void hwp_init(struct fman_hwp_regs __iomem *hwp_rg)
+{
+ /* enable HW Parser */
+ iowrite32be(HWP_RPIMAC_PEN, &hwp_rg->fmprrpimac);
+}
+
static int enable(struct fman *fman, struct fman_cfg *cfg)
{
u32 cfg_reg = 0;
@@ -1195,7 +1212,7 @@
state->max_num_of_open_dmas = 32;
state->fm_port_num_of_cg = 256;
state->num_of_rx_ports = 6;
- state->total_fifo_size = 122 * 1024;
+ state->total_fifo_size = 136 * 1024;
break;
case 2:
@@ -1793,6 +1810,7 @@
fman->bmi_regs = base_addr + BMI_OFFSET;
fman->qmi_regs = base_addr + QMI_OFFSET;
fman->dma_regs = base_addr + DMA_OFFSET;
+ fman->hwp_regs = base_addr + HWP_OFFSET;
fman->base_addr = base_addr;
spin_lock_init(&fman->spinlock);
@@ -2062,6 +2080,9 @@
/* Init QMI Registers */
qmi_init(fman->qmi_regs, fman->cfg);
+ /* Init HW Parser */
+ hwp_init(fman->hwp_regs);
+
err = enable(fman, cfg);
if (err != 0)
return err;
diff --git a/drivers/net/ethernet/freescale/fman/fman.h b/drivers/net/ethernet/freescale/fman/fman.h
index 57aae8d..f53e147 100644
--- a/drivers/net/ethernet/freescale/fman/fman.h
+++ b/drivers/net/ethernet/freescale/fman/fman.h
@@ -134,14 +134,14 @@
struct fman_prs_result {
u8 lpid; /* Logical port id */
u8 shimr; /* Shim header result */
- u16 l2r; /* Layer 2 result */
- u16 l3r; /* Layer 3 result */
+ __be16 l2r; /* Layer 2 result */
+ __be16 l3r; /* Layer 3 result */
u8 l4r; /* Layer 4 result */
u8 cplan; /* Classification plan id */
- u16 nxthdr; /* Next Header */
- u16 cksum; /* Running-sum */
+ __be16 nxthdr; /* Next Header */
+ __be16 cksum; /* Running-sum */
/* Flags&fragment-offset field of the last IP-header */
- u16 flags_frag_off;
+ __be16 flags_frag_off;
/* Routing type field of a IPV6 routing extension header */
u8 route_type;
/* Routing Extension Header Present; last bit is IP valid */
diff --git a/drivers/net/ethernet/freescale/fman/fman_dtsec.c b/drivers/net/ethernet/freescale/fman/fman_dtsec.c
index 84ea130..98bba10 100644
--- a/drivers/net/ethernet/freescale/fman/fman_dtsec.c
+++ b/drivers/net/ethernet/freescale/fman/fman_dtsec.c
@@ -381,6 +381,9 @@
/* check RGMII support */
if (iface == PHY_INTERFACE_MODE_RGMII ||
+ iface == PHY_INTERFACE_MODE_RGMII_ID ||
+ iface == PHY_INTERFACE_MODE_RGMII_RXID ||
+ iface == PHY_INTERFACE_MODE_RGMII_TXID ||
iface == PHY_INTERFACE_MODE_RMII)
if (tmp & DTSEC_ID2_INT_REDUCED_OFF)
return -EINVAL;
@@ -390,7 +393,10 @@
if (tmp & DTSEC_ID2_INT_REDUCED_OFF)
return -EINVAL;
- is_rgmii = iface == PHY_INTERFACE_MODE_RGMII;
+ is_rgmii = iface == PHY_INTERFACE_MODE_RGMII ||
+ iface == PHY_INTERFACE_MODE_RGMII_ID ||
+ iface == PHY_INTERFACE_MODE_RGMII_RXID ||
+ iface == PHY_INTERFACE_MODE_RGMII_TXID;
is_sgmii = iface == PHY_INTERFACE_MODE_SGMII;
is_qsgmii = iface == PHY_INTERFACE_MODE_QSGMII;
diff --git a/drivers/net/ethernet/freescale/fman/fman_memac.c b/drivers/net/ethernet/freescale/fman/fman_memac.c
index cd6a53e..c029688 100644
--- a/drivers/net/ethernet/freescale/fman/fman_memac.c
+++ b/drivers/net/ethernet/freescale/fman/fman_memac.c
@@ -443,7 +443,10 @@
break;
default:
tmp |= IF_MODE_GMII;
- if (phy_if == PHY_INTERFACE_MODE_RGMII)
+ if (phy_if == PHY_INTERFACE_MODE_RGMII ||
+ phy_if == PHY_INTERFACE_MODE_RGMII_ID ||
+ phy_if == PHY_INTERFACE_MODE_RGMII_RXID ||
+ phy_if == PHY_INTERFACE_MODE_RGMII_TXID)
tmp |= IF_MODE_RGMII | IF_MODE_RGMII_AUTO;
}
iowrite32be(tmp, ®s->if_mode);
diff --git a/drivers/net/ethernet/freescale/fman/fman_memac.h b/drivers/net/ethernet/freescale/fman/fman_memac.h
index 173d8e0..c4a6646 100644
--- a/drivers/net/ethernet/freescale/fman/fman_memac.h
+++ b/drivers/net/ethernet/freescale/fman/fman_memac.h
@@ -36,6 +36,7 @@
#include "fman_mac.h"
#include <linux/netdevice.h>
+#include <linux/phy_fixed.h>
struct fman_mac *memac_config(struct fman_mac_params *params);
int memac_set_promiscuous(struct fman_mac *memac, bool new_val);
diff --git a/drivers/net/ethernet/freescale/fman/fman_port.c b/drivers/net/ethernet/freescale/fman/fman_port.c
index 9f3bb50..57bf44f 100644
--- a/drivers/net/ethernet/freescale/fman/fman_port.c
+++ b/drivers/net/ethernet/freescale/fman/fman_port.c
@@ -62,6 +62,7 @@
#define BMI_PORT_REGS_OFFSET 0
#define QMI_PORT_REGS_OFFSET 0x400
+#define HWP_PORT_REGS_OFFSET 0x800
/* Default values */
#define DFLT_PORT_BUFFER_PREFIX_CONTEXT_DATA_ALIGN \
@@ -182,7 +183,7 @@
#define NIA_ENG_BMI 0x00500000
#define NIA_ENG_QMI_ENQ 0x00540000
#define NIA_ENG_QMI_DEQ 0x00580000
-
+#define NIA_ENG_HWP 0x00440000
#define NIA_BMI_AC_ENQ_FRAME 0x00000002
#define NIA_BMI_AC_TX_RELEASE 0x000002C0
#define NIA_BMI_AC_RELEASE 0x000000C0
@@ -317,6 +318,19 @@
u32 fmqm_pndcc; /* PortID n Dequeue Confirm Counter */
};
+#define HWP_HXS_COUNT 16
+#define HWP_HXS_PHE_REPORT 0x00000800
+#define HWP_HXS_PCAC_PSTAT 0x00000100
+#define HWP_HXS_PCAC_PSTOP 0x00000001
+struct fman_port_hwp_regs {
+ struct {
+ u32 ssa; /* Soft Sequence Attachment */
+ u32 lcv; /* Line-up Enable Confirmation Mask */
+ } pmda[HWP_HXS_COUNT]; /* Parse Memory Direct Access Registers */
+ u32 reserved080[(0x3f8 - 0x080) / 4]; /* (0x080-0x3f7) */
+ u32 fmpr_pcac; /* Configuration Access Control */
+};
+
/* QMI dequeue prefetch modes */
enum fman_port_deq_prefetch {
FMAN_PORT_DEQ_NO_PREFETCH, /* No prefetch mode */
@@ -436,6 +450,7 @@
union fman_port_bmi_regs __iomem *bmi_regs;
struct fman_port_qmi_regs __iomem *qmi_regs;
+ struct fman_port_hwp_regs __iomem *hwp_regs;
struct fman_sp_buffer_offsets buffer_offsets;
@@ -521,9 +536,12 @@
/* NIA */
tmp = (u32)cfg->rx_fd_bits << BMI_NEXT_ENG_FD_BITS_SHIFT;
- tmp |= NIA_ENG_BMI | NIA_BMI_AC_ENQ_FRAME;
+ tmp |= NIA_ENG_HWP;
iowrite32be(tmp, ®s->fmbm_rfne);
+ /* Parser Next Engine NIA */
+ iowrite32be(NIA_ENG_BMI | NIA_BMI_AC_ENQ_FRAME, ®s->fmbm_rfpne);
+
/* Enqueue NIA */
iowrite32be(NIA_ENG_QMI_ENQ | NIA_ORDER_RESTOR, ®s->fmbm_rfene);
@@ -665,6 +683,50 @@
return 0;
}
+static void stop_port_hwp(struct fman_port *port)
+{
+ struct fman_port_hwp_regs __iomem *regs = port->hwp_regs;
+ int cnt = 100;
+
+ iowrite32be(HWP_HXS_PCAC_PSTOP, ®s->fmpr_pcac);
+
+ while (cnt-- > 0 &&
+ (ioread32be(®s->fmpr_pcac) & HWP_HXS_PCAC_PSTAT))
+ udelay(10);
+ if (!cnt)
+ pr_err("Timeout stopping HW Parser\n");
+}
+
+static void start_port_hwp(struct fman_port *port)
+{
+ struct fman_port_hwp_regs __iomem *regs = port->hwp_regs;
+ int cnt = 100;
+
+ iowrite32be(0, ®s->fmpr_pcac);
+
+ while (cnt-- > 0 &&
+ !(ioread32be(®s->fmpr_pcac) & HWP_HXS_PCAC_PSTAT))
+ udelay(10);
+ if (!cnt)
+ pr_err("Timeout starting HW Parser\n");
+}
+
+static void init_hwp(struct fman_port *port)
+{
+ struct fman_port_hwp_regs __iomem *regs = port->hwp_regs;
+ int i;
+
+ stop_port_hwp(port);
+
+ for (i = 0; i < HWP_HXS_COUNT; i++) {
+ /* enable HXS error reporting into FD[STATUS] PHE */
+ iowrite32be(0x00000000, ®s->pmda[i].ssa);
+ iowrite32be(0xffffffff, ®s->pmda[i].lcv);
+ }
+
+ start_port_hwp(port);
+}
+
static int init(struct fman_port *port)
{
int err;
@@ -673,6 +735,8 @@
switch (port->port_type) {
case FMAN_PORT_TYPE_RX:
err = init_bmi_rx(port);
+ if (!err)
+ init_hwp(port);
break;
case FMAN_PORT_TYPE_TX:
err = init_bmi_tx(port);
@@ -686,7 +750,8 @@
/* Init QMI registers */
err = init_qmi(port);
- return err;
+ if (err)
+ return err;
return 0;
}
@@ -1247,7 +1312,7 @@
/* Allocate the FM driver's parameters structure */
port->cfg = kzalloc(sizeof(*port->cfg), GFP_KERNEL);
if (!port->cfg)
- goto err_params;
+ return -EINVAL;
/* Initialize FM port parameters which will be kept by the driver */
port->port_type = port->dts_params.type;
@@ -1276,6 +1341,7 @@
/* set memory map pointers */
port->bmi_regs = base_addr + BMI_PORT_REGS_OFFSET;
port->qmi_regs = base_addr + QMI_PORT_REGS_OFFSET;
+ port->hwp_regs = base_addr + HWP_PORT_REGS_OFFSET;
port->max_frame_length = DFLT_PORT_MAX_FRAME_LENGTH;
/* resource distribution. */
@@ -1327,8 +1393,6 @@
err_port_cfg:
kfree(port->cfg);
-err_params:
- kfree(port);
return -EINVAL;
}
EXPORT_SYMBOL(fman_port_config);
diff --git a/drivers/net/ethernet/freescale/fs_enet/mac-fec.c b/drivers/net/ethernet/freescale/fs_enet/mac-fec.c
index db9c0bc..1fc27c9 100644
--- a/drivers/net/ethernet/freescale/fs_enet/mac-fec.c
+++ b/drivers/net/ethernet/freescale/fs_enet/mac-fec.c
@@ -38,12 +38,6 @@
#include <asm/irq.h>
#include <linux/uaccess.h>
-#ifdef CONFIG_8xx
-#include <asm/8xx_immap.h>
-#include <asm/pgtable.h>
-#include <asm/cpm1.h>
-#endif
-
#include "fs_enet.h"
#include "fec.h"
diff --git a/drivers/net/ethernet/freescale/fs_enet/mac-scc.c b/drivers/net/ethernet/freescale/fs_enet/mac-scc.c
index 96d44cf..64300ac 100644
--- a/drivers/net/ethernet/freescale/fs_enet/mac-scc.c
+++ b/drivers/net/ethernet/freescale/fs_enet/mac-scc.c
@@ -37,12 +37,6 @@
#include <asm/irq.h>
#include <linux/uaccess.h>
-#ifdef CONFIG_8xx
-#include <asm/8xx_immap.h>
-#include <asm/pgtable.h>
-#include <asm/cpm1.h>
-#endif
-
#include "fs_enet.h"
/*************************************************/
diff --git a/drivers/net/ethernet/hisilicon/hns/hnae.c b/drivers/net/ethernet/hisilicon/hns/hnae.c
index b6ed818..9d9b6e6 100644
--- a/drivers/net/ethernet/hisilicon/hns/hnae.c
+++ b/drivers/net/ethernet/hisilicon/hns/hnae.c
@@ -9,9 +9,9 @@
#include <linux/dma-mapping.h>
#include <linux/interrupt.h>
+#include <linux/of.h>
#include <linux/skbuff.h>
#include <linux/slab.h>
-
#include "hnae.h"
#define cls_to_ae_dev(dev) container_of(dev, struct hnae_ae_dev, cls_dev)
@@ -57,11 +57,15 @@
static void hnae_free_buffer(struct hnae_ring *ring, struct hnae_desc_cb *cb)
{
+ if (unlikely(!cb->priv))
+ return;
+
if (cb->type == DESC_TYPE_SKB)
dev_kfree_skb_any((struct sk_buff *)cb->priv);
else if (unlikely(is_rx_ring(ring)))
put_page((struct page *)cb->priv);
- memset(cb, 0, sizeof(*cb));
+
+ cb->priv = NULL;
}
static int hnae_map_buffer(struct hnae_ring *ring, struct hnae_desc_cb *cb)
@@ -197,6 +201,7 @@
ring->q = q;
ring->flags = flags;
+ spin_lock_init(&ring->lock);
assert(!ring->desc && !ring->desc_cb && !ring->desc_dma_addr);
/* not matter for tx or rx ring, the ntc and ntc start from 0 */
diff --git a/drivers/net/ethernet/hisilicon/hns/hnae.h b/drivers/net/ethernet/hisilicon/hns/hnae.h
index 8016854..04211ac 100644
--- a/drivers/net/ethernet/hisilicon/hns/hnae.h
+++ b/drivers/net/ethernet/hisilicon/hns/hnae.h
@@ -67,6 +67,8 @@
#define AE_IS_VER1(ver) ((ver) == AE_VERSION_1)
#define AE_NAME_SIZE 16
+#define BD_SIZE_2048_MAX_MTU 6000
+
/* some said the RX and TX RCB format should not be the same in the future. But
* it is the same now...
*/
@@ -101,7 +103,6 @@
#define HNS_RX_FLAG_L4ID_TCP 0x1
#define HNS_RX_FLAG_L4ID_SCTP 0x3
-
#define HNS_TXD_ASID_S 0
#define HNS_TXD_ASID_M (0xff << HNS_TXD_ASID_S)
#define HNS_TXD_BUFNUM_S 8
@@ -273,6 +274,9 @@
/* statistic */
struct ring_stats stats;
+ /* ring lock for poll one */
+ spinlock_t lock;
+
dma_addr_t desc_dma_addr;
u32 buf_size; /* size for hnae_desc->addr, preset by AE */
u16 desc_num; /* total number of desc */
@@ -483,11 +487,11 @@
u32 auto_neg, u32 rx_en, u32 tx_en);
void (*get_coalesce_usecs)(struct hnae_handle *handle,
u32 *tx_usecs, u32 *rx_usecs);
- void (*get_rx_max_coalesced_frames)(struct hnae_handle *handle,
- u32 *tx_frames, u32 *rx_frames);
+ void (*get_max_coalesced_frames)(struct hnae_handle *handle,
+ u32 *tx_frames, u32 *rx_frames);
int (*set_coalesce_usecs)(struct hnae_handle *handle, u32 timeout);
int (*set_coalesce_frames)(struct hnae_handle *handle,
- u32 coalesce_frames);
+ u32 tx_frames, u32 rx_frames);
void (*get_coalesce_range)(struct hnae_handle *handle,
u32 *tx_frames_low, u32 *rx_frames_low,
u32 *tx_frames_high, u32 *rx_frames_high,
@@ -646,6 +650,41 @@
ring->desc[i].rx.ipoff_bnum_pid_flag = 0;
}
+/* when reinit buffer size, we should reinit buffer description */
+static inline void hnae_reinit_all_ring_desc(struct hnae_handle *h)
+{
+ int i, j;
+ struct hnae_ring *ring;
+
+ for (i = 0; i < h->q_num; i++) {
+ ring = &h->qs[i]->rx_ring;
+ for (j = 0; j < ring->desc_num; j++)
+ ring->desc[j].addr = cpu_to_le64(ring->desc_cb[j].dma);
+ }
+
+ wmb(); /* commit all data before submit */
+}
+
+/* when reinit buffer size, we should reinit page offset */
+static inline void hnae_reinit_all_ring_page_off(struct hnae_handle *h)
+{
+ int i, j;
+ struct hnae_ring *ring;
+
+ for (i = 0; i < h->q_num; i++) {
+ ring = &h->qs[i]->rx_ring;
+ for (j = 0; j < ring->desc_num; j++) {
+ ring->desc_cb[j].page_offset = 0;
+ if (ring->desc[j].addr !=
+ cpu_to_le64(ring->desc_cb[j].dma))
+ ring->desc[j].addr =
+ cpu_to_le64(ring->desc_cb[j].dma);
+ }
+ }
+
+ wmb(); /* commit all data before submit */
+}
+
#define hnae_set_field(origin, mask, shift, val) \
do { \
(origin) &= (~(mask)); \
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_ae_adapt.c b/drivers/net/ethernet/hisilicon/hns/hns_ae_adapt.c
index 0a9cdf0..ff864a1 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_ae_adapt.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_ae_adapt.c
@@ -267,8 +267,32 @@
static int hns_ae_set_mtu(struct hnae_handle *handle, int new_mtu)
{
struct hns_mac_cb *mac_cb = hns_get_mac_cb(handle);
+ struct hnae_queue *q;
+ u32 rx_buf_size;
+ int i, ret;
- return hns_mac_set_mtu(mac_cb, new_mtu);
+ /* when buf_size is 2048, max mtu is 6K for rx ring max bd num is 3. */
+ if (!AE_IS_VER1(mac_cb->dsaf_dev->dsaf_ver)) {
+ if (new_mtu <= BD_SIZE_2048_MAX_MTU)
+ rx_buf_size = 2048;
+ else
+ rx_buf_size = 4096;
+ } else {
+ rx_buf_size = mac_cb->dsaf_dev->buf_size;
+ }
+
+ ret = hns_mac_set_mtu(mac_cb, new_mtu, rx_buf_size);
+
+ if (!ret) {
+ /* reinit ring buf_size */
+ for (i = 0; i < handle->q_num; i++) {
+ q = handle->qs[i];
+ q->rx_ring.buf_size = rx_buf_size;
+ hns_rcb_set_rx_ring_bs(q, rx_buf_size);
+ }
+ }
+
+ return ret;
}
static void hns_ae_set_tso_stats(struct hnae_handle *handle, int enable)
@@ -463,15 +487,21 @@
ring_pair->port_id_in_comm);
}
-static void hns_ae_get_rx_max_coalesced_frames(struct hnae_handle *handle,
- u32 *tx_frames, u32 *rx_frames)
+static void hns_ae_get_max_coalesced_frames(struct hnae_handle *handle,
+ u32 *tx_frames, u32 *rx_frames)
{
struct ring_pair_cb *ring_pair =
container_of(handle->qs[0], struct ring_pair_cb, q);
+ struct dsaf_device *dsaf_dev = hns_ae_get_dsaf_dev(handle->dev);
- *tx_frames = hns_rcb_get_coalesced_frames(ring_pair->rcb_common,
- ring_pair->port_id_in_comm);
- *rx_frames = hns_rcb_get_coalesced_frames(ring_pair->rcb_common,
+ if (AE_IS_VER1(dsaf_dev->dsaf_ver) ||
+ handle->port_type == HNAE_PORT_DEBUG)
+ *tx_frames = hns_rcb_get_rx_coalesced_frames(
+ ring_pair->rcb_common, ring_pair->port_id_in_comm);
+ else
+ *tx_frames = hns_rcb_get_tx_coalesced_frames(
+ ring_pair->rcb_common, ring_pair->port_id_in_comm);
+ *rx_frames = hns_rcb_get_rx_coalesced_frames(ring_pair->rcb_common,
ring_pair->port_id_in_comm);
}
@@ -485,15 +515,34 @@
ring_pair->rcb_common, ring_pair->port_id_in_comm, timeout);
}
-static int hns_ae_set_coalesce_frames(struct hnae_handle *handle,
- u32 coalesce_frames)
+static int hns_ae_set_coalesce_frames(struct hnae_handle *handle,
+ u32 tx_frames, u32 rx_frames)
{
+ int ret;
struct ring_pair_cb *ring_pair =
container_of(handle->qs[0], struct ring_pair_cb, q);
+ struct dsaf_device *dsaf_dev = hns_ae_get_dsaf_dev(handle->dev);
- return hns_rcb_set_coalesced_frames(
- ring_pair->rcb_common,
- ring_pair->port_id_in_comm, coalesce_frames);
+ if (AE_IS_VER1(dsaf_dev->dsaf_ver) ||
+ handle->port_type == HNAE_PORT_DEBUG) {
+ if (tx_frames != rx_frames)
+ return -EINVAL;
+ return hns_rcb_set_rx_coalesced_frames(
+ ring_pair->rcb_common,
+ ring_pair->port_id_in_comm, rx_frames);
+ } else {
+ if (tx_frames != 1)
+ return -EINVAL;
+ ret = hns_rcb_set_tx_coalesced_frames(
+ ring_pair->rcb_common,
+ ring_pair->port_id_in_comm, tx_frames);
+ if (ret)
+ return ret;
+
+ return hns_rcb_set_rx_coalesced_frames(
+ ring_pair->rcb_common,
+ ring_pair->port_id_in_comm, rx_frames);
+ }
}
static void hns_ae_get_coalesce_range(struct hnae_handle *handle,
@@ -504,20 +553,27 @@
{
struct dsaf_device *dsaf_dev;
+ assert(handle);
+
dsaf_dev = hns_ae_get_dsaf_dev(handle->dev);
- *tx_frames_low = HNS_RCB_MIN_COALESCED_FRAMES;
- *rx_frames_low = HNS_RCB_MIN_COALESCED_FRAMES;
- *tx_frames_high =
- (dsaf_dev->desc_num - 1 > HNS_RCB_MAX_COALESCED_FRAMES) ?
- HNS_RCB_MAX_COALESCED_FRAMES : dsaf_dev->desc_num - 1;
- *rx_frames_high =
- (dsaf_dev->desc_num - 1 > HNS_RCB_MAX_COALESCED_FRAMES) ?
- HNS_RCB_MAX_COALESCED_FRAMES : dsaf_dev->desc_num - 1;
- *tx_usecs_low = 0;
- *rx_usecs_low = 0;
- *tx_usecs_high = HNS_RCB_MAX_COALESCED_USECS;
- *rx_usecs_high = HNS_RCB_MAX_COALESCED_USECS;
+ *tx_frames_low = HNS_RCB_TX_FRAMES_LOW;
+ *rx_frames_low = HNS_RCB_RX_FRAMES_LOW;
+
+ if (AE_IS_VER1(dsaf_dev->dsaf_ver) ||
+ handle->port_type == HNAE_PORT_DEBUG)
+ *tx_frames_high =
+ (dsaf_dev->desc_num - 1 > HNS_RCB_TX_FRAMES_HIGH) ?
+ HNS_RCB_TX_FRAMES_HIGH : dsaf_dev->desc_num - 1;
+ else
+ *tx_frames_high = 1;
+
+ *rx_frames_high = (dsaf_dev->desc_num - 1 > HNS_RCB_RX_FRAMES_HIGH) ?
+ HNS_RCB_RX_FRAMES_HIGH : dsaf_dev->desc_num - 1;
+ *tx_usecs_low = HNS_RCB_TX_USECS_LOW;
+ *rx_usecs_low = HNS_RCB_RX_USECS_LOW;
+ *tx_usecs_high = HNS_RCB_TX_USECS_HIGH;
+ *rx_usecs_high = HNS_RCB_RX_USECS_HIGH;
}
void hns_ae_update_stats(struct hnae_handle *handle,
@@ -802,8 +858,9 @@
memcpy(key, ppe_cb->rss_key, HNS_PPEV2_RSS_KEY_SIZE);
/* update the current hash->queue mappings from the shadow RSS table */
- memcpy(indir, ppe_cb->rss_indir_table,
- HNS_PPEV2_RSS_IND_TBL_SIZE * sizeof(*indir));
+ if (indir)
+ memcpy(indir, ppe_cb->rss_indir_table,
+ HNS_PPEV2_RSS_IND_TBL_SIZE * sizeof(*indir));
return 0;
}
@@ -814,15 +871,19 @@
struct hns_ppe_cb *ppe_cb = hns_get_ppe_cb(handle);
/* set the RSS Hash Key if specififed by the user */
- if (key)
- hns_ppe_set_rss_key(ppe_cb, (u32 *)key);
+ if (key) {
+ memcpy(ppe_cb->rss_key, key, HNS_PPEV2_RSS_KEY_SIZE);
+ hns_ppe_set_rss_key(ppe_cb, ppe_cb->rss_key);
+ }
- /* update the shadow RSS table with user specified qids */
- memcpy(ppe_cb->rss_indir_table, indir,
- HNS_PPEV2_RSS_IND_TBL_SIZE * sizeof(*indir));
+ if (indir) {
+ /* update the shadow RSS table with user specified qids */
+ memcpy(ppe_cb->rss_indir_table, indir,
+ HNS_PPEV2_RSS_IND_TBL_SIZE * sizeof(*indir));
- /* now update the hardware */
- hns_ppe_set_indir_table(ppe_cb, ppe_cb->rss_indir_table);
+ /* now update the hardware */
+ hns_ppe_set_indir_table(ppe_cb, ppe_cb->rss_indir_table);
+ }
return 0;
}
@@ -846,7 +907,7 @@
.get_autoneg = hns_ae_get_autoneg,
.set_pauseparam = hns_ae_set_pauseparam,
.get_coalesce_usecs = hns_ae_get_coalesce_usecs,
- .get_rx_max_coalesced_frames = hns_ae_get_rx_max_coalesced_frames,
+ .get_max_coalesced_frames = hns_ae_get_max_coalesced_frames,
.set_coalesce_usecs = hns_ae_set_coalesce_usecs,
.set_coalesce_frames = hns_ae_set_coalesce_frames,
.get_coalesce_range = hns_ae_get_coalesce_range,
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_gmac.c b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_gmac.c
index 3382441..86944bc 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_gmac.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_gmac.c
@@ -86,12 +86,11 @@
dsaf_set_dev_bit(drv, GMAC_PORT_EN_REG, GMAC_PORT_RX_EN_B, 0);
}
-/**
-*hns_gmac_get_en - get port enable
-*@mac_drv:mac device
-*@rx:rx enable
-*@tx:tx enable
-*/
+/* hns_gmac_get_en - get port enable
+ * @mac_drv:mac device
+ * @rx:rx enable
+ * @tx:tx enable
+ */
static void hns_gmac_get_en(void *mac_drv, u32 *rx, u32 *tx)
{
struct mac_driver *drv = (struct mac_driver *)mac_drv;
@@ -148,6 +147,17 @@
GMAC_MAX_FRM_SIZE_S, newval);
}
+static void hns_gmac_config_pad_and_crc(void *mac_drv, u8 newval)
+{
+ u32 tx_ctrl;
+ struct mac_driver *drv = (struct mac_driver *)mac_drv;
+
+ tx_ctrl = dsaf_read_dev(drv, GMAC_TRANSMIT_CONTROL_REG);
+ dsaf_set_bit(tx_ctrl, GMAC_TX_PAD_EN_B, !!newval);
+ dsaf_set_bit(tx_ctrl, GMAC_TX_CRC_ADD_B, !!newval);
+ dsaf_write_dev(drv, GMAC_TRANSMIT_CONTROL_REG, tx_ctrl);
+}
+
static void hns_gmac_config_an_mode(void *mac_drv, u8 newval)
{
struct mac_driver *drv = (struct mac_driver *)mac_drv;
@@ -250,7 +260,6 @@
static int hns_gmac_adjust_link(void *mac_drv, enum mac_speed speed,
u32 full_duplex)
{
- u32 tx_ctrl;
struct mac_driver *drv = (struct mac_driver *)mac_drv;
dsaf_set_dev_bit(drv, GMAC_DUPLEX_TYPE_REG,
@@ -279,14 +288,6 @@
return -EINVAL;
}
- tx_ctrl = dsaf_read_dev(drv, GMAC_TRANSMIT_CONTROL_REG);
- dsaf_set_bit(tx_ctrl, GMAC_TX_PAD_EN_B, 1);
- dsaf_set_bit(tx_ctrl, GMAC_TX_CRC_ADD_B, 1);
- dsaf_write_dev(drv, GMAC_TRANSMIT_CONTROL_REG, tx_ctrl);
-
- dsaf_set_dev_bit(drv, GMAC_MODE_CHANGE_EN_REG,
- GMAC_MODE_CHANGE_EB_B, 1);
-
return 0;
}
@@ -325,6 +326,17 @@
hns_gmac_tx_loop_pkt_dis(mac_drv);
if (drv->mac_cb->mac_type == HNAE_PORT_DEBUG)
hns_gmac_set_uc_match(mac_drv, 0);
+
+ hns_gmac_config_pad_and_crc(mac_drv, 1);
+
+ dsaf_set_dev_bit(drv, GMAC_MODE_CHANGE_EN_REG,
+ GMAC_MODE_CHANGE_EB_B, 1);
+
+ /* reduce gmac tx water line to avoid gmac hang-up
+ * in speed 100M and duplex half.
+ */
+ dsaf_set_dev_field(drv, GMAC_TX_WATER_LINE_REG, GMAC_TX_WATER_LINE_MASK,
+ GMAC_TX_WATER_LINE_SHIFT, 8);
}
void hns_gmac_update_stats(void *mac_drv)
@@ -453,24 +465,6 @@
return 0;
}
-static void hns_gmac_config_pad_and_crc(void *mac_drv, u8 newval)
-{
- u32 tx_ctrl;
- struct mac_driver *drv = (struct mac_driver *)mac_drv;
-
- tx_ctrl = dsaf_read_dev(drv, GMAC_TRANSMIT_CONTROL_REG);
- dsaf_set_bit(tx_ctrl, GMAC_TX_PAD_EN_B, !!newval);
- dsaf_set_bit(tx_ctrl, GMAC_TX_CRC_ADD_B, !!newval);
- dsaf_write_dev(drv, GMAC_TRANSMIT_CONTROL_REG, tx_ctrl);
-}
-
-static void hns_gmac_get_id(void *mac_drv, u8 *mac_id)
-{
- struct mac_driver *drv = (struct mac_driver *)mac_drv;
-
- *mac_id = drv->mac_id;
-}
-
static void hns_gmac_get_info(void *mac_drv, struct mac_info *mac_info)
{
enum hns_gmac_duplex_mdoe duplex;
@@ -672,7 +666,7 @@
static int hns_gmac_get_sset_count(int stringset)
{
- if (stringset == ETH_SS_STATS)
+ if (stringset == ETH_SS_STATS || stringset == ETH_SS_PRIV_FLAGS)
return ARRAY_SIZE(g_gmac_stats_string);
return 0;
@@ -712,7 +706,6 @@
mac_drv->config_pad_and_crc = hns_gmac_config_pad_and_crc;
mac_drv->config_half_duplex = hns_gmac_set_duplex_type;
mac_drv->set_rx_ignore_pause_frames = hns_gmac_set_rx_auto_pause_frames;
- mac_drv->mac_get_id = hns_gmac_get_id;
mac_drv->get_info = hns_gmac_get_info;
mac_drv->autoneg_stat = hns_gmac_autoneg_stat;
mac_drv->get_pause_enable = hns_gmac_get_pausefrm_cfg;
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.c b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.c
index bdd8cdd..8b5cdf4 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.c
@@ -335,44 +335,6 @@
return 0;
}
-/**
- *hns_mac_del_mac - delete mac address into dsaf table,can't delete the same
- * address twice
- *@net_dev: net device
- *@vfn : vf lan
- *@mac : mac address
- *return status
- */
-int hns_mac_del_mac(struct hns_mac_cb *mac_cb, u32 vfn, char *mac)
-{
- struct mac_entry_idx *old_mac;
- struct dsaf_device *dsaf_dev;
- u32 ret;
-
- dsaf_dev = mac_cb->dsaf_dev;
-
- if (vfn < DSAF_MAX_VM_NUM) {
- old_mac = &mac_cb->addr_entry_idx[vfn];
- } else {
- dev_err(mac_cb->dev,
- "vf queue is too large, %s mac%d queue = %#x!\n",
- mac_cb->dsaf_dev->ae_dev.name, mac_cb->mac_id, vfn);
- return -EINVAL;
- }
-
- if (dsaf_dev) {
- ret = hns_dsaf_del_mac_entry(dsaf_dev, old_mac->vlan_id,
- mac_cb->mac_id, old_mac->addr);
- if (ret)
- return ret;
-
- if (memcmp(old_mac->addr, mac, sizeof(old_mac->addr)) == 0)
- old_mac->valid = 0;
- }
-
- return 0;
-}
-
int hns_mac_clr_multicast(struct hns_mac_cb *mac_cb, int vfn)
{
struct dsaf_device *dsaf_dev = mac_cb->dsaf_dev;
@@ -494,10 +456,9 @@
}
}
-int hns_mac_set_mtu(struct hns_mac_cb *mac_cb, u32 new_mtu)
+int hns_mac_set_mtu(struct hns_mac_cb *mac_cb, u32 new_mtu, u32 buf_size)
{
struct mac_driver *drv = hns_mac_get_drv(mac_cb);
- u32 buf_size = mac_cb->dsaf_dev->buf_size;
u32 new_frm = new_mtu + ETH_HLEN + ETH_FCS_LEN + VLAN_HLEN;
u32 max_frm = AE_IS_VER1(mac_cb->dsaf_dev->dsaf_ver) ?
MAC_MAX_MTU : MAC_MAX_MTU_V2;
@@ -735,6 +696,8 @@
rc = phy_device_register(phy);
if (rc) {
phy_device_free(phy);
+ dev_err(&mdio->dev, "registered phy fail at address %i\n",
+ addr);
return -ENODEV;
}
@@ -745,7 +708,7 @@
return 0;
}
-static void hns_mac_register_phy(struct hns_mac_cb *mac_cb)
+static int hns_mac_register_phy(struct hns_mac_cb *mac_cb)
{
struct acpi_reference_args args;
struct platform_device *pdev;
@@ -755,24 +718,39 @@
/* Loop over the child nodes and register a phy_device for each one */
if (!to_acpi_device_node(mac_cb->fw_port))
- return;
+ return -ENODEV;
rc = acpi_node_get_property_reference(
mac_cb->fw_port, "mdio-node", 0, &args);
if (rc)
- return;
+ return rc;
addr = hns_mac_phy_parse_addr(mac_cb->dev, mac_cb->fw_port);
if (addr < 0)
- return;
+ return addr;
/* dev address in adev */
pdev = hns_dsaf_find_platform_device(acpi_fwnode_handle(args.adev));
+ if (!pdev) {
+ dev_err(mac_cb->dev, "mac%d mdio pdev is NULL\n",
+ mac_cb->mac_id);
+ return -EINVAL;
+ }
+
mii_bus = platform_get_drvdata(pdev);
+ if (!mii_bus) {
+ dev_err(mac_cb->dev,
+ "mac%d mdio is NULL, dsaf will probe again later\n",
+ mac_cb->mac_id);
+ return -EPROBE_DEFER;
+ }
+
rc = hns_mac_register_phydev(mii_bus, mac_cb, addr);
if (!rc)
dev_dbg(mac_cb->dev, "mac%d register phy addr:%d\n",
mac_cb->mac_id, addr);
+
+ return rc;
}
#define MAC_MEDIA_TYPE_MAX_LEN 16
@@ -793,7 +771,7 @@
*@np:device node
* return: 0 --success, negative --fail
*/
-static int hns_mac_get_info(struct hns_mac_cb *mac_cb)
+static int hns_mac_get_info(struct hns_mac_cb *mac_cb)
{
struct device_node *np;
struct regmap *syscon;
@@ -903,7 +881,15 @@
}
}
} else if (is_acpi_node(mac_cb->fw_port)) {
- hns_mac_register_phy(mac_cb);
+ ret = hns_mac_register_phy(mac_cb);
+ /*
+ * Mac can work well if there is phy or not.If the port don't
+ * connect with phy, the return value will be ignored. Only
+ * when there is phy but can't find mdio bus, the return value
+ * will be handled.
+ */
+ if (ret == -EPROBE_DEFER)
+ return ret;
} else {
dev_err(mac_cb->dev, "mac%d cannot find phy node\n",
mac_cb->mac_id);
@@ -1065,6 +1051,7 @@
dsaf_dev->mac_cb[port_id] = mac_cb;
}
}
+
/* init mac_cb for all port */
for (port_id = 0; port_id < max_port_num; port_id++) {
mac_cb = dsaf_dev->mac_cb[port_id];
@@ -1074,6 +1061,7 @@
ret = hns_mac_get_cfg(dsaf_dev, mac_cb);
if (ret)
return ret;
+
ret = hns_mac_init_ex(mac_cb);
if (ret)
return ret;
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.h b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.h
index 2bb3d1e..24dfba5 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.h
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.h
@@ -373,8 +373,6 @@
void (*set_rx_ignore_pause_frames)(void *mac_drv, u32 enable);
/* config rx mode for promiscuous*/
void (*set_promiscuous)(void *mac_drv, u8 enable);
- /* get mac id */
- void (*mac_get_id)(void *mac_drv, u8 *mac_id);
void (*mac_pausefrm_cfg)(void *mac_drv, u32 rx_en, u32 tx_en);
void (*autoneg_stat)(void *mac_drv, u32 *enable);
@@ -436,7 +434,6 @@
int hns_mac_vm_config_bc_en(struct hns_mac_cb *mac_cb, u32 vm, bool enable);
void hns_mac_start(struct hns_mac_cb *mac_cb);
void hns_mac_stop(struct hns_mac_cb *mac_cb);
-int hns_mac_del_mac(struct hns_mac_cb *mac_cb, u32 vfn, char *mac);
void hns_mac_uninit(struct dsaf_device *dsaf_dev);
void hns_mac_adjust_link(struct hns_mac_cb *mac_cb, int speed, int duplex);
void hns_mac_reset(struct hns_mac_cb *mac_cb);
@@ -444,7 +441,7 @@
void hns_mac_get_pauseparam(struct hns_mac_cb *mac_cb, u32 *rx_en, u32 *tx_en);
int hns_mac_set_autoneg(struct hns_mac_cb *mac_cb, u8 enable);
int hns_mac_set_pauseparam(struct hns_mac_cb *mac_cb, u32 rx_en, u32 tx_en);
-int hns_mac_set_mtu(struct hns_mac_cb *mac_cb, u32 new_mtu);
+int hns_mac_set_mtu(struct hns_mac_cb *mac_cb, u32 new_mtu, u32 buf_size);
int hns_mac_get_port_info(struct hns_mac_cb *mac_cb,
u8 *auto_neg, u16 *speed, u8 *duplex);
int hns_mac_config_mac_loopback(struct hns_mac_cb *mac_cb,
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.c b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.c
index 403ea9d..e0bc79e 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.c
@@ -510,10 +510,10 @@
o_sbm_bp_cfg = dsaf_read_dev(dsaf_dev, reg);
dsaf_set_field(o_sbm_bp_cfg,
DSAFV2_SBM_CFG3_SET_BUF_NUM_NO_PFC_M,
- DSAFV2_SBM_CFG3_SET_BUF_NUM_NO_PFC_S, 48);
+ DSAFV2_SBM_CFG3_SET_BUF_NUM_NO_PFC_S, 55);
dsaf_set_field(o_sbm_bp_cfg,
DSAFV2_SBM_CFG3_RESET_BUF_NUM_NO_PFC_M,
- DSAFV2_SBM_CFG3_RESET_BUF_NUM_NO_PFC_S, 80);
+ DSAFV2_SBM_CFG3_RESET_BUF_NUM_NO_PFC_S, 110);
dsaf_write_dev(dsaf_dev, reg, o_sbm_bp_cfg);
/* for no enable pfc mode */
@@ -521,10 +521,10 @@
o_sbm_bp_cfg = dsaf_read_dev(dsaf_dev, reg);
dsaf_set_field(o_sbm_bp_cfg,
DSAFV2_SBM_CFG4_SET_BUF_NUM_NO_PFC_M,
- DSAFV2_SBM_CFG4_SET_BUF_NUM_NO_PFC_S, 192);
+ DSAFV2_SBM_CFG4_SET_BUF_NUM_NO_PFC_S, 128);
dsaf_set_field(o_sbm_bp_cfg,
DSAFV2_SBM_CFG4_RESET_BUF_NUM_NO_PFC_M,
- DSAFV2_SBM_CFG4_RESET_BUF_NUM_NO_PFC_S, 240);
+ DSAFV2_SBM_CFG4_RESET_BUF_NUM_NO_PFC_S, 192);
dsaf_write_dev(dsaf_dev, reg, o_sbm_bp_cfg);
}
@@ -1648,87 +1648,6 @@
mac_entry->addr);
}
-/**
- * hns_dsaf_set_mac_mc_entry - set mac mc-entry
- * @dsaf_dev: dsa fabric device struct pointer
- * @mac_entry: mc-mac entry
- */
-int hns_dsaf_set_mac_mc_entry(
- struct dsaf_device *dsaf_dev,
- struct dsaf_drv_mac_multi_dest_entry *mac_entry)
-{
- u16 entry_index = DSAF_INVALID_ENTRY_IDX;
- struct dsaf_drv_tbl_tcam_key mac_key;
- struct dsaf_tbl_tcam_mcast_cfg mac_data;
- struct dsaf_drv_priv *priv =
- (struct dsaf_drv_priv *)hns_dsaf_dev_priv(dsaf_dev);
- struct dsaf_drv_soft_mac_tbl *soft_mac_entry = priv->soft_mac_tbl;
- struct dsaf_drv_tbl_tcam_key tmp_mac_key;
- struct dsaf_tbl_tcam_data tcam_data;
-
- /* mac addr check */
- if (MAC_IS_ALL_ZEROS(mac_entry->addr)) {
- dev_err(dsaf_dev->dev, "set uc %s Mac %pM err!\n",
- dsaf_dev->ae_dev.name, mac_entry->addr);
- return -EINVAL;
- }
-
- /*config key */
- hns_dsaf_set_mac_key(dsaf_dev, &mac_key,
- mac_entry->in_vlan_id,
- mac_entry->in_port_num, mac_entry->addr);
-
- /* entry ie exist? */
- entry_index = hns_dsaf_find_soft_mac_entry(dsaf_dev, &mac_key);
- if (entry_index == DSAF_INVALID_ENTRY_IDX) {
- /*if hasnot, find enpty entry*/
- entry_index = hns_dsaf_find_empty_mac_entry(dsaf_dev);
- if (entry_index == DSAF_INVALID_ENTRY_IDX) {
- /*if hasnot empty, error*/
- dev_err(dsaf_dev->dev,
- "set_uc_entry failed, %s Mac key(%#x:%#x)\n",
- dsaf_dev->ae_dev.name,
- mac_key.high.val, mac_key.low.val);
- return -EINVAL;
- }
-
- /* config hardware entry */
- memset(mac_data.tbl_mcast_port_msk,
- 0, sizeof(mac_data.tbl_mcast_port_msk));
- } else {
- /* config hardware entry */
- hns_dsaf_tcam_mc_get(dsaf_dev, entry_index, &tcam_data,
- &mac_data);
-
- tmp_mac_key.high.val =
- le32_to_cpu(tcam_data.tbl_tcam_data_high);
- tmp_mac_key.low.val = le32_to_cpu(tcam_data.tbl_tcam_data_low);
- }
- mac_data.tbl_mcast_old_en = 0;
- mac_data.tbl_mcast_item_vld = 1;
- dsaf_set_field(mac_data.tbl_mcast_port_msk[0],
- 0x3F, 0, mac_entry->port_mask[0]);
-
- dev_dbg(dsaf_dev->dev,
- "set_uc_entry, %s key(%#x:%#x) entry_index%d\n",
- dsaf_dev->ae_dev.name, mac_key.high.val,
- mac_key.low.val, entry_index);
-
- tcam_data.tbl_tcam_data_high = cpu_to_le32(mac_key.high.val);
- tcam_data.tbl_tcam_data_low = cpu_to_le32(mac_key.low.val);
-
- hns_dsaf_tcam_mc_cfg(dsaf_dev, entry_index, &tcam_data, NULL,
- &mac_data);
-
- /* config software entry */
- soft_mac_entry += entry_index;
- soft_mac_entry->index = entry_index;
- soft_mac_entry->tcam_key.high.val = mac_key.high.val;
- soft_mac_entry->tcam_key.low.val = mac_key.low.val;
-
- return 0;
-}
-
static void hns_dsaf_mc_mask_bit_clear(char *dst, const char *src)
{
u16 *a = (u16 *)dst;
@@ -2090,166 +2009,6 @@
return ret;
}
-/**
- * hns_dsaf_get_mac_uc_entry - get mac uc entry
- * @dsaf_dev: dsa fabric device struct pointer
- * @mac_entry: mac entry
- */
-int hns_dsaf_get_mac_uc_entry(struct dsaf_device *dsaf_dev,
- struct dsaf_drv_mac_single_dest_entry *mac_entry)
-{
- u16 entry_index = DSAF_INVALID_ENTRY_IDX;
- struct dsaf_drv_tbl_tcam_key mac_key;
-
- struct dsaf_tbl_tcam_ucast_cfg mac_data;
- struct dsaf_tbl_tcam_data tcam_data;
-
- /* check macaddr */
- if (MAC_IS_ALL_ZEROS(mac_entry->addr) ||
- MAC_IS_BROADCAST(mac_entry->addr)) {
- dev_err(dsaf_dev->dev, "get_entry failed,addr %pM\n",
- mac_entry->addr);
- return -EINVAL;
- }
-
- /*config key */
- hns_dsaf_set_mac_key(dsaf_dev, &mac_key, mac_entry->in_vlan_id,
- mac_entry->in_port_num, mac_entry->addr);
-
- /*check exist? */
- entry_index = hns_dsaf_find_soft_mac_entry(dsaf_dev, &mac_key);
- if (entry_index == DSAF_INVALID_ENTRY_IDX) {
- /*find none, error */
- dev_err(dsaf_dev->dev,
- "get_uc_entry failed, %s Mac key(%#x:%#x)\n",
- dsaf_dev->ae_dev.name,
- mac_key.high.val, mac_key.low.val);
- return -EINVAL;
- }
- dev_dbg(dsaf_dev->dev,
- "get_uc_entry, %s Mac key(%#x:%#x) entry_index%d\n",
- dsaf_dev->ae_dev.name, mac_key.high.val,
- mac_key.low.val, entry_index);
-
- /* read entry */
- hns_dsaf_tcam_uc_get(dsaf_dev, entry_index, &tcam_data, &mac_data);
-
- mac_key.high.val = le32_to_cpu(tcam_data.tbl_tcam_data_high);
- mac_key.low.val = le32_to_cpu(tcam_data.tbl_tcam_data_low);
-
- mac_entry->port_num = mac_data.tbl_ucast_out_port;
-
- return 0;
-}
-
-/**
- * hns_dsaf_get_mac_mc_entry - get mac mc entry
- * @dsaf_dev: dsa fabric device struct pointer
- * @mac_entry: mac entry
- */
-int hns_dsaf_get_mac_mc_entry(struct dsaf_device *dsaf_dev,
- struct dsaf_drv_mac_multi_dest_entry *mac_entry)
-{
- u16 entry_index = DSAF_INVALID_ENTRY_IDX;
- struct dsaf_drv_tbl_tcam_key mac_key;
-
- struct dsaf_tbl_tcam_mcast_cfg mac_data;
- struct dsaf_tbl_tcam_data tcam_data;
-
- /*check mac addr */
- if (MAC_IS_ALL_ZEROS(mac_entry->addr) ||
- MAC_IS_BROADCAST(mac_entry->addr)) {
- dev_err(dsaf_dev->dev, "get_entry failed,addr %pM\n",
- mac_entry->addr);
- return -EINVAL;
- }
-
- /*config key */
- hns_dsaf_set_mac_key(dsaf_dev, &mac_key, mac_entry->in_vlan_id,
- mac_entry->in_port_num, mac_entry->addr);
-
- /*check exist? */
- entry_index = hns_dsaf_find_soft_mac_entry(dsaf_dev, &mac_key);
- if (entry_index == DSAF_INVALID_ENTRY_IDX) {
- /* find none, error */
- dev_err(dsaf_dev->dev,
- "get_mac_uc_entry failed, %s Mac key(%#x:%#x)\n",
- dsaf_dev->ae_dev.name, mac_key.high.val,
- mac_key.low.val);
- return -EINVAL;
- }
- dev_dbg(dsaf_dev->dev,
- "get_mac_uc_entry, %s Mac key(%#x:%#x) entry_index%d\n",
- dsaf_dev->ae_dev.name, mac_key.high.val,
- mac_key.low.val, entry_index);
-
- /*read entry */
- hns_dsaf_tcam_mc_get(dsaf_dev, entry_index, &tcam_data, &mac_data);
-
- mac_key.high.val = le32_to_cpu(tcam_data.tbl_tcam_data_high);
- mac_key.low.val = le32_to_cpu(tcam_data.tbl_tcam_data_low);
-
- mac_entry->port_mask[0] = mac_data.tbl_mcast_port_msk[0] & 0x3F;
- return 0;
-}
-
-/**
- * hns_dsaf_get_mac_entry_by_index - get mac entry by tab index
- * @dsaf_dev: dsa fabric device struct pointer
- * @entry_index: tab entry index
- * @mac_entry: mac entry
- */
-int hns_dsaf_get_mac_entry_by_index(
- struct dsaf_device *dsaf_dev,
- u16 entry_index, struct dsaf_drv_mac_multi_dest_entry *mac_entry)
-{
- struct dsaf_drv_tbl_tcam_key mac_key;
-
- struct dsaf_tbl_tcam_mcast_cfg mac_data;
- struct dsaf_tbl_tcam_ucast_cfg mac_uc_data;
- struct dsaf_tbl_tcam_data tcam_data;
- char mac_addr[ETH_ALEN] = {0};
-
- if (entry_index >= dsaf_dev->tcam_max_num) {
- /* find none, del error */
- dev_err(dsaf_dev->dev, "get_uc_entry failed, %s\n",
- dsaf_dev->ae_dev.name);
- return -EINVAL;
- }
-
- /* mc entry, do read opt */
- hns_dsaf_tcam_mc_get(dsaf_dev, entry_index, &tcam_data, &mac_data);
-
- mac_key.high.val = le32_to_cpu(tcam_data.tbl_tcam_data_high);
- mac_key.low.val = le32_to_cpu(tcam_data.tbl_tcam_data_low);
-
- mac_entry->port_mask[0] = mac_data.tbl_mcast_port_msk[0] & 0x3F;
-
- /***get mac addr*/
- mac_addr[0] = mac_key.high.bits.mac_0;
- mac_addr[1] = mac_key.high.bits.mac_1;
- mac_addr[2] = mac_key.high.bits.mac_2;
- mac_addr[3] = mac_key.high.bits.mac_3;
- mac_addr[4] = mac_key.low.bits.mac_4;
- mac_addr[5] = mac_key.low.bits.mac_5;
- /**is mc or uc*/
- if (MAC_IS_MULTICAST((u8 *)mac_addr) ||
- MAC_IS_L3_MULTICAST((u8 *)mac_addr)) {
- /**mc donot do*/
- } else {
- /*is not mc, just uc... */
- hns_dsaf_tcam_uc_get(dsaf_dev, entry_index, &tcam_data,
- &mac_uc_data);
-
- mac_key.high.val = le32_to_cpu(tcam_data.tbl_tcam_data_high);
- mac_key.low.val = le32_to_cpu(tcam_data.tbl_tcam_data_low);
-
- mac_entry->port_mask[0] = (1 << mac_uc_data.tbl_ucast_out_port);
- }
-
- return 0;
-}
-
static struct dsaf_device *hns_dsaf_alloc_dev(struct device *dev,
size_t sizeof_priv)
{
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.h b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.h
index cef6bf4..4507e82 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.h
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.h
@@ -68,7 +68,7 @@
};
#define DSAF_STATS_READ(p, offset) (*((u64 *)((u8 *)(p) + (offset))))
-#define HNS_DSAF_IS_DEBUG(dev) (dev->dsaf_mode == DSAF_MODE_DISABLE_SP)
+#define HNS_DSAF_IS_DEBUG(dev) ((dev)->dsaf_mode == DSAF_MODE_DISABLE_SP)
enum hal_dsaf_mode {
HRD_DSAF_NO_DSAF_MODE = 0x0,
@@ -429,23 +429,12 @@
int hns_dsaf_set_mac_uc_entry(struct dsaf_device *dsaf_dev,
struct dsaf_drv_mac_single_dest_entry *mac_entry);
-int hns_dsaf_set_mac_mc_entry(struct dsaf_device *dsaf_dev,
- struct dsaf_drv_mac_multi_dest_entry *mac_entry);
int hns_dsaf_add_mac_mc_port(struct dsaf_device *dsaf_dev,
struct dsaf_drv_mac_single_dest_entry *mac_entry);
int hns_dsaf_del_mac_entry(struct dsaf_device *dsaf_dev, u16 vlan_id,
u8 in_port_num, u8 *addr);
int hns_dsaf_del_mac_mc_port(struct dsaf_device *dsaf_dev,
struct dsaf_drv_mac_single_dest_entry *mac_entry);
-int hns_dsaf_get_mac_uc_entry(struct dsaf_device *dsaf_dev,
- struct dsaf_drv_mac_single_dest_entry *mac_entry);
-int hns_dsaf_get_mac_mc_entry(struct dsaf_device *dsaf_dev,
- struct dsaf_drv_mac_multi_dest_entry *mac_entry);
-int hns_dsaf_get_mac_entry_by_index(
- struct dsaf_device *dsaf_dev,
- u16 entry_index,
- struct dsaf_drv_mac_multi_dest_entry *mac_entry);
-
void hns_dsaf_fix_mac_mode(struct hns_mac_cb *mac_cb);
int hns_dsaf_ae_init(struct dsaf_device *dsaf_dev);
@@ -475,5 +464,4 @@
int hns_dsaf_clr_mac_mc_port(struct dsaf_device *dsaf_dev,
u8 mac_id, u8 port_num);
-
#endif /* __HNS_DSAF_MAIN_H__ */
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_ppe.c b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_ppe.c
index 6ea8722..b62816c 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_ppe.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_ppe.c
@@ -422,7 +422,7 @@
int hns_ppe_get_sset_count(int stringset)
{
- if (stringset == ETH_SS_STATS)
+ if (stringset == ETH_SS_STATS || stringset == ETH_SS_PRIV_FLAGS)
return ETH_PPE_STATIC_NUM;
return 0;
}
@@ -496,21 +496,23 @@
*/
int hns_ppe_init(struct dsaf_device *dsaf_dev)
{
- int i, k;
int ret;
+ int i;
for (i = 0; i < HNS_PPE_COM_NUM; i++) {
ret = hns_ppe_common_get_cfg(dsaf_dev, i);
if (ret)
- goto get_ppe_cfg_fail;
+ goto get_cfg_fail;
ret = hns_rcb_common_get_cfg(dsaf_dev, i);
if (ret)
- goto get_rcb_cfg_fail;
+ goto get_cfg_fail;
hns_ppe_get_cfg(dsaf_dev->ppe_common[i]);
- hns_rcb_get_cfg(dsaf_dev->rcb_common[i]);
+ ret = hns_rcb_get_cfg(dsaf_dev->rcb_common[i]);
+ if (ret)
+ goto get_cfg_fail;
}
for (i = 0; i < HNS_PPE_COM_NUM; i++)
@@ -518,13 +520,12 @@
return 0;
-get_rcb_cfg_fail:
- hns_ppe_common_free_cfg(dsaf_dev, i);
-get_ppe_cfg_fail:
- for (k = i - 1; k >= 0; k--) {
- hns_rcb_common_free_cfg(dsaf_dev, k);
- hns_ppe_common_free_cfg(dsaf_dev, k);
+get_cfg_fail:
+ for (i = 0; i < HNS_PPE_COM_NUM; i++) {
+ hns_rcb_common_free_cfg(dsaf_dev, i);
+ hns_ppe_common_free_cfg(dsaf_dev, i);
}
+
return ret;
}
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_rcb.c b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_rcb.c
index f0ed80d6..6f3570c 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_rcb.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_rcb.c
@@ -32,6 +32,9 @@
#define RCB_RESET_WAIT_TIMES 30
#define RCB_RESET_TRY_TIMES 10
+/* Because default mtu is 1500, rcb buffer size is set to 2048 enough */
+#define RCB_DEFAULT_BUFFER_SIZE 2048
+
/**
*hns_rcb_wait_fbd_clean - clean fbd
*@qs: ring struct pointer array
@@ -192,6 +195,30 @@
wmb(); /* Sync point after breakpoint */
}
+/* hns_rcb_set_tx_ring_bs - init rcb ring buf size regester
+ *@q: hnae_queue
+ *@buf_size: buffer size set to hw
+ */
+void hns_rcb_set_tx_ring_bs(struct hnae_queue *q, u32 buf_size)
+{
+ u32 bd_size_type = hns_rcb_buf_size2type(buf_size);
+
+ dsaf_write_dev(q, RCB_RING_TX_RING_BD_LEN_REG,
+ bd_size_type);
+}
+
+/* hns_rcb_set_rx_ring_bs - init rcb ring buf size regester
+ *@q: hnae_queue
+ *@buf_size: buffer size set to hw
+ */
+void hns_rcb_set_rx_ring_bs(struct hnae_queue *q, u32 buf_size)
+{
+ u32 bd_size_type = hns_rcb_buf_size2type(buf_size);
+
+ dsaf_write_dev(q, RCB_RING_RX_RING_BD_LEN_REG,
+ bd_size_type);
+}
+
/**
*hns_rcb_ring_init - init rcb ring
*@ring_pair: ring pair control block
@@ -200,8 +227,6 @@
static void hns_rcb_ring_init(struct ring_pair_cb *ring_pair, int ring_type)
{
struct hnae_queue *q = &ring_pair->q;
- struct rcb_common_cb *rcb_common = ring_pair->rcb_common;
- u32 bd_size_type = rcb_common->dsaf_dev->buf_size_type;
struct hnae_ring *ring =
(ring_type == RX_RING) ? &q->rx_ring : &q->tx_ring;
dma_addr_t dma = ring->desc_dma_addr;
@@ -212,8 +237,8 @@
dsaf_write_dev(q, RCB_RING_RX_RING_BASEADDR_H_REG,
(u32)((dma >> 31) >> 1));
- dsaf_write_dev(q, RCB_RING_RX_RING_BD_LEN_REG,
- bd_size_type);
+ hns_rcb_set_rx_ring_bs(q, ring->buf_size);
+
dsaf_write_dev(q, RCB_RING_RX_RING_BD_NUM_REG,
ring_pair->port_id_in_comm);
dsaf_write_dev(q, RCB_RING_RX_RING_PKTLINE_REG,
@@ -224,12 +249,12 @@
dsaf_write_dev(q, RCB_RING_TX_RING_BASEADDR_H_REG,
(u32)((dma >> 31) >> 1));
- dsaf_write_dev(q, RCB_RING_TX_RING_BD_LEN_REG,
- bd_size_type);
+ hns_rcb_set_tx_ring_bs(q, ring->buf_size);
+
dsaf_write_dev(q, RCB_RING_TX_RING_BD_NUM_REG,
ring_pair->port_id_in_comm);
dsaf_write_dev(q, RCB_RING_TX_RING_PKTLINE_REG,
- ring_pair->port_id_in_comm);
+ ring_pair->port_id_in_comm + HNS_RCB_TX_PKTLINE_OFFSET);
}
}
@@ -259,13 +284,27 @@
static void hns_rcb_set_port_timeout(
struct rcb_common_cb *rcb_common, u32 port_idx, u32 timeout)
{
- if (AE_IS_VER1(rcb_common->dsaf_dev->dsaf_ver))
+ if (AE_IS_VER1(rcb_common->dsaf_dev->dsaf_ver)) {
dsaf_write_dev(rcb_common, RCB_CFG_OVERTIME_REG,
timeout * HNS_RCB_CLK_FREQ_MHZ);
- else
+ } else if (!HNS_DSAF_IS_DEBUG(rcb_common->dsaf_dev)) {
+ if (timeout > HNS_RCB_DEF_GAP_TIME_USECS)
+ dsaf_write_dev(rcb_common,
+ RCB_PORT_INT_GAPTIME_REG + port_idx * 4,
+ HNS_RCB_DEF_GAP_TIME_USECS);
+ else
+ dsaf_write_dev(rcb_common,
+ RCB_PORT_INT_GAPTIME_REG + port_idx * 4,
+ timeout);
+
dsaf_write_dev(rcb_common,
RCB_PORT_CFG_OVERTIME_REG + port_idx * 4,
timeout);
+ } else {
+ dsaf_write_dev(rcb_common,
+ RCB_PORT_CFG_OVERTIME_REG + port_idx * 4,
+ timeout);
+ }
}
static int hns_rcb_common_get_port_num(struct rcb_common_cb *rcb_common)
@@ -327,8 +366,12 @@
for (i = 0; i < port_num; i++) {
hns_rcb_set_port_desc_cnt(rcb_common, i, rcb_common->desc_num);
- (void)hns_rcb_set_coalesced_frames(
- rcb_common, i, HNS_RCB_DEF_COALESCED_FRAMES);
+ hns_rcb_set_rx_coalesced_frames(
+ rcb_common, i, HNS_RCB_DEF_RX_COALESCED_FRAMES);
+ if (!AE_IS_VER1(rcb_common->dsaf_dev->dsaf_ver) &&
+ !HNS_DSAF_IS_DEBUG(rcb_common->dsaf_dev))
+ hns_rcb_set_tx_coalesced_frames(
+ rcb_common, i, HNS_RCB_DEF_TX_COALESCED_FRAMES);
hns_rcb_set_port_timeout(
rcb_common, i, HNS_RCB_DEF_COALESCED_USECS);
}
@@ -380,7 +423,6 @@
struct hnae_ring *ring;
struct rcb_common_cb *rcb_common;
struct ring_pair_cb *ring_pair_cb;
- u32 buf_size;
u16 desc_num, mdnum_ppkt;
bool irq_idx, is_ver1;
@@ -401,7 +443,6 @@
}
rcb_common = ring_pair_cb->rcb_common;
- buf_size = rcb_common->dsaf_dev->buf_size;
desc_num = rcb_common->dsaf_dev->desc_num;
ring->desc = NULL;
@@ -410,7 +451,7 @@
ring->irq = ring_pair_cb->virq[irq_idx];
ring->desc_dma_addr = 0;
- ring->buf_size = buf_size;
+ ring->buf_size = RCB_DEFAULT_BUFFER_SIZE;
ring->desc_num = desc_num;
ring->max_desc_num_per_pkt = mdnum_ppkt;
ring->max_raw_data_sz_per_desc = HNS_RCB_MAX_PKT_SIZE;
@@ -430,7 +471,6 @@
static int hns_rcb_get_port_in_comm(
struct rcb_common_cb *rcb_common, int ring_idx)
{
-
return ring_idx / (rcb_common->max_q_per_vf * rcb_common->max_vfn);
}
@@ -452,7 +492,7 @@
*hns_rcb_get_cfg - get rcb config
*@rcb_common: rcb common device
*/
-void hns_rcb_get_cfg(struct rcb_common_cb *rcb_common)
+int hns_rcb_get_cfg(struct rcb_common_cb *rcb_common)
{
struct ring_pair_cb *ring_pair_cb;
u32 i;
@@ -477,26 +517,48 @@
ring_pair_cb->virq[HNS_RCB_IRQ_IDX_RX] =
is_ver1 ? platform_get_irq(pdev, base_irq_idx + i * 2 + 1) :
platform_get_irq(pdev, base_irq_idx + i * 3);
+ if ((ring_pair_cb->virq[HNS_RCB_IRQ_IDX_TX] == -EPROBE_DEFER) ||
+ (ring_pair_cb->virq[HNS_RCB_IRQ_IDX_RX] == -EPROBE_DEFER))
+ return -EPROBE_DEFER;
+
ring_pair_cb->q.phy_base =
RCB_COMM_BASE_TO_RING_BASE(rcb_common->phy_base, i);
hns_rcb_ring_pair_get_cfg(ring_pair_cb);
}
+
+ return 0;
}
/**
- *hns_rcb_get_coalesced_frames - get rcb port coalesced frames
+ *hns_rcb_get_rx_coalesced_frames - get rcb port rx coalesced frames
*@rcb_common: rcb_common device
*@port_idx:port id in comm
*
*Returns: coalesced_frames
*/
-u32 hns_rcb_get_coalesced_frames(
+u32 hns_rcb_get_rx_coalesced_frames(
struct rcb_common_cb *rcb_common, u32 port_idx)
{
return dsaf_read_dev(rcb_common, RCB_CFG_PKTLINE_REG + port_idx * 4);
}
/**
+ *hns_rcb_get_tx_coalesced_frames - get rcb port tx coalesced frames
+ *@rcb_common: rcb_common device
+ *@port_idx:port id in comm
+ *
+ *Returns: coalesced_frames
+ */
+u32 hns_rcb_get_tx_coalesced_frames(
+ struct rcb_common_cb *rcb_common, u32 port_idx)
+{
+ u64 reg;
+
+ reg = RCB_CFG_PKTLINE_REG + (port_idx + HNS_RCB_TX_PKTLINE_OFFSET) * 4;
+ return dsaf_read_dev(rcb_common, reg);
+}
+
+/**
*hns_rcb_get_coalesce_usecs - get rcb port coalesced time_out
*@rcb_common: rcb_common device
*@port_idx:port id in comm
@@ -538,33 +600,17 @@
return -EINVAL;
}
}
- if (timeout > HNS_RCB_MAX_COALESCED_USECS) {
+ if (timeout > HNS_RCB_MAX_COALESCED_USECS || timeout == 0) {
dev_err(rcb_common->dsaf_dev->dev,
- "error: coalesce_usecs setting supports 0~1023us\n");
+ "error: coalesce_usecs setting supports 1~1023us\n");
return -EINVAL;
}
-
- if (!AE_IS_VER1(rcb_common->dsaf_dev->dsaf_ver)) {
- if (timeout == 0)
- /* set timeout to 0, Disable gap time */
- dsaf_set_reg_field(rcb_common->io_base,
- RCB_INT_GAP_TIME_REG + port_idx * 4,
- PPE_INT_GAPTIME_M, PPE_INT_GAPTIME_B,
- 0);
- else
- /* set timeout non 0, restore gap time to 1 */
- dsaf_set_reg_field(rcb_common->io_base,
- RCB_INT_GAP_TIME_REG + port_idx * 4,
- PPE_INT_GAPTIME_M, PPE_INT_GAPTIME_B,
- 1);
- }
-
hns_rcb_set_port_timeout(rcb_common, port_idx, timeout);
return 0;
}
/**
- *hns_rcb_set_coalesced_frames - set rcb coalesced frames
+ *hns_rcb_set_tx_coalesced_frames - set rcb coalesced frames
*@rcb_common: rcb_common device
*@port_idx:port id in comm
*@coalesced_frames:tx/rx BD num for coalesced frames
@@ -572,10 +618,41 @@
* Returns:
* Zero for success, or an error code in case of failure
*/
-int hns_rcb_set_coalesced_frames(
+int hns_rcb_set_tx_coalesced_frames(
struct rcb_common_cb *rcb_common, u32 port_idx, u32 coalesced_frames)
{
- u32 old_waterline = hns_rcb_get_coalesced_frames(rcb_common, port_idx);
+ u32 old_waterline =
+ hns_rcb_get_tx_coalesced_frames(rcb_common, port_idx);
+ u64 reg;
+
+ if (coalesced_frames == old_waterline)
+ return 0;
+
+ if (coalesced_frames != 1) {
+ dev_err(rcb_common->dsaf_dev->dev,
+ "error: not support tx coalesce_frames setting!\n");
+ return -EINVAL;
+ }
+
+ reg = RCB_CFG_PKTLINE_REG + (port_idx + HNS_RCB_TX_PKTLINE_OFFSET) * 4;
+ dsaf_write_dev(rcb_common, reg, coalesced_frames);
+ return 0;
+}
+
+/**
+ *hns_rcb_set_rx_coalesced_frames - set rcb rx coalesced frames
+ *@rcb_common: rcb_common device
+ *@port_idx:port id in comm
+ *@coalesced_frames:tx/rx BD num for coalesced frames
+ *
+ * Returns:
+ * Zero for success, or an error code in case of failure
+ */
+int hns_rcb_set_rx_coalesced_frames(
+ struct rcb_common_cb *rcb_common, u32 port_idx, u32 coalesced_frames)
+{
+ u32 old_waterline =
+ hns_rcb_get_rx_coalesced_frames(rcb_common, port_idx);
if (coalesced_frames == old_waterline)
return 0;
@@ -799,7 +876,7 @@
*/
int hns_rcb_get_ring_sset_count(int stringset)
{
- if (stringset == ETH_SS_STATS)
+ if (stringset == ETH_SS_STATS || stringset == ETH_SS_PRIV_FLAGS)
return HNS_RING_STATIC_REG_NUM;
return 0;
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_rcb.h b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_rcb.h
index 99b4e1b..6028164 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_rcb.h
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_rcb.h
@@ -35,12 +35,23 @@
#define HNS_RCB_REG_OFFSET 0x10000
+#define HNS_RCB_TX_FRAMES_LOW 1
+#define HNS_RCB_RX_FRAMES_LOW 1
+#define HNS_RCB_TX_FRAMES_HIGH 1023
+#define HNS_RCB_RX_FRAMES_HIGH 1023
+#define HNS_RCB_TX_USECS_LOW 1
+#define HNS_RCB_RX_USECS_LOW 1
+#define HNS_RCB_TX_USECS_HIGH 1023
+#define HNS_RCB_RX_USECS_HIGH 1023
#define HNS_RCB_MAX_COALESCED_FRAMES 1023
#define HNS_RCB_MIN_COALESCED_FRAMES 1
-#define HNS_RCB_DEF_COALESCED_FRAMES 50
+#define HNS_RCB_DEF_RX_COALESCED_FRAMES 50
+#define HNS_RCB_DEF_TX_COALESCED_FRAMES 1
#define HNS_RCB_CLK_FREQ_MHZ 350
#define HNS_RCB_MAX_COALESCED_USECS 0x3ff
-#define HNS_RCB_DEF_COALESCED_USECS 50
+#define HNS_RCB_DEF_COALESCED_USECS 30
+#define HNS_RCB_DEF_GAP_TIME_USECS 20
+#define HNS_RCB_TX_PKTLINE_OFFSET 8
#define HNS_RCB_COMMON_ENDIAN 1
@@ -110,7 +121,7 @@
void hns_rcb_common_free_cfg(struct dsaf_device *dsaf_dev, u32 comm_index);
int hns_rcb_common_init_hw(struct rcb_common_cb *rcb_common);
void hns_rcb_start(struct hnae_queue *q, u32 val);
-void hns_rcb_get_cfg(struct rcb_common_cb *rcb_common);
+int hns_rcb_get_cfg(struct rcb_common_cb *rcb_common);
void hns_rcb_get_queue_mode(enum dsaf_mode dsaf_mode,
u16 *max_vfn, u16 *max_q_per_vf);
@@ -125,13 +136,17 @@
void hns_rcb_init_hw(struct ring_pair_cb *ring);
void hns_rcb_reset_ring_hw(struct hnae_queue *q);
void hns_rcb_wait_fbd_clean(struct hnae_queue **qs, int q_num, u32 flag);
-u32 hns_rcb_get_coalesced_frames(
+u32 hns_rcb_get_rx_coalesced_frames(
+ struct rcb_common_cb *rcb_common, u32 port_idx);
+u32 hns_rcb_get_tx_coalesced_frames(
struct rcb_common_cb *rcb_common, u32 port_idx);
u32 hns_rcb_get_coalesce_usecs(
struct rcb_common_cb *rcb_common, u32 port_idx);
int hns_rcb_set_coalesce_usecs(
struct rcb_common_cb *rcb_common, u32 port_idx, u32 timeout);
-int hns_rcb_set_coalesced_frames(
+int hns_rcb_set_rx_coalesced_frames(
+ struct rcb_common_cb *rcb_common, u32 port_idx, u32 coalesced_frames);
+int hns_rcb_set_tx_coalesced_frames(
struct rcb_common_cb *rcb_common, u32 port_idx, u32 coalesced_frames);
void hns_rcb_update_stats(struct hnae_queue *queue);
@@ -146,4 +161,7 @@
void hns_rcb_get_ring_regs(struct hnae_queue *queue, void *data);
void hns_rcb_get_strings(int stringset, u8 *data, int index);
+void hns_rcb_set_rx_ring_bs(struct hnae_queue *q, u32 buf_size);
+void hns_rcb_set_tx_ring_bs(struct hnae_queue *q, u32 buf_size);
+
#endif /* _HNS_DSAF_RCB_H */
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_reg.h b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_reg.h
index 8fa18fc..46a52d9 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_reg.h
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_reg.h
@@ -421,7 +421,7 @@
#define RCB_CFG_OVERTIME_REG 0x9300
#define RCB_CFG_PKTLINE_INT_NUM_REG 0x9304
#define RCB_CFG_OVERTIME_INT_NUM_REG 0x9308
-#define RCB_INT_GAP_TIME_REG 0x9400
+#define RCB_PORT_INT_GAPTIME_REG 0x9400
#define RCB_PORT_CFG_OVERTIME_REG 0x9430
#define RCB_RING_RX_RING_BASEADDR_L_REG 0x00000
@@ -466,6 +466,7 @@
#define GMAC_DUPLEX_TYPE_REG 0x0008UL
#define GMAC_FD_FC_TYPE_REG 0x000CUL
+#define GMAC_TX_WATER_LINE_REG 0x0010UL
#define GMAC_FC_TX_TIMER_REG 0x001CUL
#define GMAC_FD_FC_ADDR_LOW_REG 0x0020UL
#define GMAC_FD_FC_ADDR_HIGH_REG 0x0024UL
@@ -912,6 +913,9 @@
#define GMAC_DUPLEX_TYPE_B 0
+#define GMAC_TX_WATER_LINE_MASK ((1UL << 8) - 1)
+#define GMAC_TX_WATER_LINE_SHIFT 0
+
#define GMAC_FC_TX_TIMER_S 0
#define GMAC_FC_TX_TIMER_M 0xffff
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_xgmac.c b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_xgmac.c
index aae830a..51e7e9f 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_xgmac.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_xgmac.c
@@ -300,18 +300,6 @@
}
/**
- *hns_xgmac_get_id - get xgmac port id
- *@mac_drv: mac driver
- *@newval:xgmac max frame length
- */
-static void hns_xgmac_get_id(void *mac_drv, u8 *mac_id)
-{
- struct mac_driver *drv = (struct mac_driver *)mac_drv;
-
- *mac_id = drv->mac_id;
-}
-
-/**
*hns_xgmac_config_max_frame_length - set xgmac max frame length
*@mac_drv: mac driver
*@newval:xgmac max frame length
@@ -793,7 +781,7 @@
*/
static int hns_xgmac_get_sset_count(int stringset)
{
- if (stringset == ETH_SS_STATS)
+ if (stringset == ETH_SS_STATS || stringset == ETH_SS_PRIV_FLAGS)
return ARRAY_SIZE(g_xgmac_stats_string);
return 0;
@@ -833,7 +821,6 @@
mac_drv->config_half_duplex = NULL;
mac_drv->set_rx_ignore_pause_frames =
hns_xgmac_set_rx_ignore_pause_frames;
- mac_drv->mac_get_id = hns_xgmac_get_id;
mac_drv->mac_free = hns_xgmac_free;
mac_drv->adjust_link = NULL;
mac_drv->set_tx_auto_pause_frames = hns_xgmac_set_tx_auto_pause_frames;
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_enet.c b/drivers/net/ethernet/hisilicon/hns/hns_enet.c
index fca37e2..c6700b9 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_enet.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_enet.c
@@ -512,7 +512,8 @@
int last_offset;
bool twobufs;
- twobufs = ((PAGE_SIZE < 8192) && hnae_buf_size(ring) == HNS_BUFFER_SIZE_2048);
+ twobufs = ((PAGE_SIZE < 8192) &&
+ hnae_buf_size(ring) == HNS_BUFFER_SIZE_2048);
desc = &ring->desc[ring->next_to_clean];
size = le16_to_cpu(desc->rx.size);
@@ -859,7 +860,7 @@
return recv_pkts;
}
-static void hns_nic_rx_fini_pro(struct hns_nic_ring_data *ring_data)
+static bool hns_nic_rx_fini_pro(struct hns_nic_ring_data *ring_data)
{
struct hnae_ring *ring = ring_data->ring;
int num = 0;
@@ -873,22 +874,23 @@
ring_data->ring->q->handle->dev->ops->toggle_ring_irq(
ring_data->ring, 1);
- napi_schedule(&ring_data->napi);
+ return false;
+ } else {
+ return true;
}
}
-static void hns_nic_rx_fini_pro_v2(struct hns_nic_ring_data *ring_data)
+static bool hns_nic_rx_fini_pro_v2(struct hns_nic_ring_data *ring_data)
{
struct hnae_ring *ring = ring_data->ring;
- int num = 0;
+ int num;
num = readl_relaxed(ring->io_base + RCB_REG_FBDNUM);
- if (num == 0)
- ring_data->ring->q->handle->dev->ops->toggle_ring_irq(
- ring, 0);
+ if (!num)
+ return true;
else
- napi_schedule(&ring_data->napi);
+ return false;
}
static inline void hns_nic_reclaim_one_desc(struct hnae_ring *ring,
@@ -921,12 +923,13 @@
/* netif_tx_lock will turn down the performance, set only when necessary */
#ifdef CONFIG_NET_POLL_CONTROLLER
-#define NETIF_TX_LOCK(ndev) netif_tx_lock(ndev)
-#define NETIF_TX_UNLOCK(ndev) netif_tx_unlock(ndev)
+#define NETIF_TX_LOCK(ring) spin_lock(&(ring)->lock)
+#define NETIF_TX_UNLOCK(ring) spin_unlock(&(ring)->lock)
#else
-#define NETIF_TX_LOCK(ndev)
-#define NETIF_TX_UNLOCK(ndev)
+#define NETIF_TX_LOCK(ring)
+#define NETIF_TX_UNLOCK(ring)
#endif
+
/* reclaim all desc in one budget
* return error or number of desc left
*/
@@ -940,13 +943,13 @@
int head;
int bytes, pkts;
- NETIF_TX_LOCK(ndev);
+ NETIF_TX_LOCK(ring);
head = readl_relaxed(ring->io_base + RCB_REG_HEAD);
rmb(); /* make sure head is ready before touch any data */
if (is_ring_empty(ring) || head == ring->next_to_clean) {
- NETIF_TX_UNLOCK(ndev);
+ NETIF_TX_UNLOCK(ring);
return 0; /* no data to poll */
}
@@ -954,7 +957,7 @@
netdev_err(ndev, "wrong head (%d, %d-%d)\n", head,
ring->next_to_use, ring->next_to_clean);
ring->stats.io_err_cnt++;
- NETIF_TX_UNLOCK(ndev);
+ NETIF_TX_UNLOCK(ring);
return -EIO;
}
@@ -966,7 +969,7 @@
prefetch(&ring->desc_cb[ring->next_to_clean]);
}
- NETIF_TX_UNLOCK(ndev);
+ NETIF_TX_UNLOCK(ring);
dev_queue = netdev_get_tx_queue(ndev, ring_data->queue_index);
netdev_tx_completed_queue(dev_queue, pkts, bytes);
@@ -989,7 +992,7 @@
return 0;
}
-static void hns_nic_tx_fini_pro(struct hns_nic_ring_data *ring_data)
+static bool hns_nic_tx_fini_pro(struct hns_nic_ring_data *ring_data)
{
struct hnae_ring *ring = ring_data->ring;
int head;
@@ -1002,20 +1005,21 @@
ring_data->ring->q->handle->dev->ops->toggle_ring_irq(
ring_data->ring, 1);
- napi_schedule(&ring_data->napi);
+ return false;
+ } else {
+ return true;
}
}
-static void hns_nic_tx_fini_pro_v2(struct hns_nic_ring_data *ring_data)
+static bool hns_nic_tx_fini_pro_v2(struct hns_nic_ring_data *ring_data)
{
struct hnae_ring *ring = ring_data->ring;
int head = readl_relaxed(ring->io_base + RCB_REG_HEAD);
if (head == ring->next_to_clean)
- ring_data->ring->q->handle->dev->ops->toggle_ring_irq(
- ring, 0);
+ return true;
else
- napi_schedule(&ring_data->napi);
+ return false;
}
static void hns_nic_tx_clr_all_bufs(struct hns_nic_ring_data *ring_data)
@@ -1026,7 +1030,7 @@
int head;
int bytes, pkts;
- NETIF_TX_LOCK(ndev);
+ NETIF_TX_LOCK(ring);
head = ring->next_to_use; /* ntu :soft setted ring position*/
bytes = 0;
@@ -1034,7 +1038,7 @@
while (head != ring->next_to_clean)
hns_nic_reclaim_one_desc(ring, &bytes, &pkts);
- NETIF_TX_UNLOCK(ndev);
+ NETIF_TX_UNLOCK(ring);
dev_queue = netdev_get_tx_queue(ndev, ring_data->queue_index);
netdev_tx_reset_queue(dev_queue);
@@ -1042,15 +1046,23 @@
static int hns_nic_common_poll(struct napi_struct *napi, int budget)
{
+ int clean_complete = 0;
struct hns_nic_ring_data *ring_data =
container_of(napi, struct hns_nic_ring_data, napi);
- int clean_complete = ring_data->poll_one(
- ring_data, budget, ring_data->ex_process);
+ struct hnae_ring *ring = ring_data->ring;
- if (clean_complete >= 0 && clean_complete < budget) {
- napi_complete(napi);
- ring_data->fini_process(ring_data);
- return 0;
+try_again:
+ clean_complete += ring_data->poll_one(
+ ring_data, budget - clean_complete,
+ ring_data->ex_process);
+
+ if (clean_complete < budget) {
+ if (ring_data->fini_process(ring_data)) {
+ napi_complete(napi);
+ ring->q->handle->dev->ops->toggle_ring_irq(ring, 0);
+ } else {
+ goto try_again;
+ }
}
return clean_complete;
@@ -1196,54 +1208,31 @@
napi_disable(&priv->ring_data[idx].napi);
}
-static void hns_set_irq_affinity(struct hns_nic_priv *priv)
+static int hns_nic_init_affinity_mask(int q_num, int ring_idx,
+ struct hnae_ring *ring, cpumask_t *mask)
{
- struct hnae_handle *h = priv->ae_handle;
- struct hns_nic_ring_data *rd;
- int i;
int cpu;
- cpumask_var_t mask;
- if (!alloc_cpumask_var(&mask, GFP_KERNEL))
- return;
-
- /*diffrent irq banlance for 16core and 32core*/
- if (h->q_num == num_possible_cpus()) {
- for (i = 0; i < h->q_num * 2; i++) {
- rd = &priv->ring_data[i];
- if (cpu_online(rd->queue_index)) {
- cpumask_clear(mask);
- cpu = rd->queue_index;
- cpumask_set_cpu(cpu, mask);
- (void)irq_set_affinity_hint(rd->ring->irq,
- mask);
- }
- }
+ /* Diffrent irq banlance between 16core and 32core.
+ * The cpu mask set by ring index according to the ring flag
+ * which indicate the ring is tx or rx.
+ */
+ if (q_num == num_possible_cpus()) {
+ if (is_tx_ring(ring))
+ cpu = ring_idx;
+ else
+ cpu = ring_idx - q_num;
} else {
- for (i = 0; i < h->q_num; i++) {
- rd = &priv->ring_data[i];
- if (cpu_online(rd->queue_index * 2)) {
- cpumask_clear(mask);
- cpu = rd->queue_index * 2;
- cpumask_set_cpu(cpu, mask);
- (void)irq_set_affinity_hint(rd->ring->irq,
- mask);
- }
- }
-
- for (i = h->q_num; i < h->q_num * 2; i++) {
- rd = &priv->ring_data[i];
- if (cpu_online(rd->queue_index * 2 + 1)) {
- cpumask_clear(mask);
- cpu = rd->queue_index * 2 + 1;
- cpumask_set_cpu(cpu, mask);
- (void)irq_set_affinity_hint(rd->ring->irq,
- mask);
- }
- }
+ if (is_tx_ring(ring))
+ cpu = ring_idx * 2;
+ else
+ cpu = (ring_idx - q_num) * 2 + 1;
}
- free_cpumask_var(mask);
+ cpumask_clear(mask);
+ cpumask_set_cpu(cpu, mask);
+
+ return cpu;
}
static int hns_nic_init_irq(struct hns_nic_priv *priv)
@@ -1252,6 +1241,7 @@
struct hns_nic_ring_data *rd;
int i;
int ret;
+ int cpu;
for (i = 0; i < h->q_num * 2; i++) {
rd = &priv->ring_data[i];
@@ -1261,7 +1251,7 @@
snprintf(rd->ring->ring_name, RCB_RING_NAME_LEN,
"%s-%s%d", priv->netdev->name,
- (i < h->q_num ? "tx" : "rx"), rd->queue_index);
+ (is_tx_ring(rd->ring) ? "tx" : "rx"), rd->queue_index);
rd->ring->ring_name[RCB_RING_NAME_LEN - 1] = '\0';
@@ -1273,12 +1263,17 @@
return ret;
}
disable_irq(rd->ring->irq);
+
+ cpu = hns_nic_init_affinity_mask(h->q_num, i,
+ rd->ring, &rd->mask);
+
+ if (cpu_online(cpu))
+ irq_set_affinity_hint(rd->ring->irq,
+ &rd->mask);
+
rd->ring->irq_init_flag = RCB_IRQ_INITED;
}
- /*set cpu affinity*/
- hns_set_irq_affinity(priv);
-
return 0;
}
@@ -1487,32 +1482,259 @@
return (netdev_tx_t)ret;
}
+static void hns_nic_drop_rx_fetch(struct hns_nic_ring_data *ring_data,
+ struct sk_buff *skb)
+{
+ dev_kfree_skb_any(skb);
+}
+
+#define HNS_LB_TX_RING 0
+static struct sk_buff *hns_assemble_skb(struct net_device *ndev)
+{
+ struct sk_buff *skb;
+ struct ethhdr *ethhdr;
+ int frame_len;
+
+ /* allocate test skb */
+ skb = alloc_skb(64, GFP_KERNEL);
+ if (!skb)
+ return NULL;
+
+ skb_put(skb, 64);
+ skb->dev = ndev;
+ memset(skb->data, 0xFF, skb->len);
+
+ /* must be tcp/ip package */
+ ethhdr = (struct ethhdr *)skb->data;
+ ethhdr->h_proto = htons(ETH_P_IP);
+
+ frame_len = skb->len & (~1ul);
+ memset(&skb->data[frame_len / 2], 0xAA,
+ frame_len / 2 - 1);
+
+ skb->queue_mapping = HNS_LB_TX_RING;
+
+ return skb;
+}
+
+static int hns_enable_serdes_lb(struct net_device *ndev)
+{
+ struct hns_nic_priv *priv = netdev_priv(ndev);
+ struct hnae_handle *h = priv->ae_handle;
+ struct hnae_ae_ops *ops = h->dev->ops;
+ int speed, duplex;
+ int ret;
+
+ ret = ops->set_loopback(h, MAC_INTERNALLOOP_SERDES, 1);
+ if (ret)
+ return ret;
+
+ ret = ops->start ? ops->start(h) : 0;
+ if (ret)
+ return ret;
+
+ /* link adjust duplex*/
+ if (h->phy_if != PHY_INTERFACE_MODE_XGMII)
+ speed = 1000;
+ else
+ speed = 10000;
+ duplex = 1;
+
+ ops->adjust_link(h, speed, duplex);
+
+ /* wait h/w ready */
+ mdelay(300);
+
+ return 0;
+}
+
+static void hns_disable_serdes_lb(struct net_device *ndev)
+{
+ struct hns_nic_priv *priv = netdev_priv(ndev);
+ struct hnae_handle *h = priv->ae_handle;
+ struct hnae_ae_ops *ops = h->dev->ops;
+
+ ops->stop(h);
+ ops->set_loopback(h, MAC_INTERNALLOOP_SERDES, 0);
+}
+
+/**
+ *hns_nic_clear_all_rx_fetch - clear the chip fetched descriptions. The
+ *function as follows:
+ * 1. if one rx ring has found the page_offset is not equal 0 between head
+ * and tail, it means that the chip fetched the wrong descs for the ring
+ * which buffer size is 4096.
+ * 2. we set the chip serdes loopback and set rss indirection to the ring.
+ * 3. construct 64-bytes ip broadcast packages, wait the associated rx ring
+ * recieving all packages and it will fetch new descriptions.
+ * 4. recover to the original state.
+ *
+ *@ndev: net device
+ */
+static int hns_nic_clear_all_rx_fetch(struct net_device *ndev)
+{
+ struct hns_nic_priv *priv = netdev_priv(ndev);
+ struct hnae_handle *h = priv->ae_handle;
+ struct hnae_ae_ops *ops = h->dev->ops;
+ struct hns_nic_ring_data *rd;
+ struct hnae_ring *ring;
+ struct sk_buff *skb;
+ u32 *org_indir;
+ u32 *cur_indir;
+ int indir_size;
+ int head, tail;
+ int fetch_num;
+ int i, j;
+ bool found;
+ int retry_times;
+ int ret = 0;
+
+ /* alloc indir memory */
+ indir_size = ops->get_rss_indir_size(h) * sizeof(*org_indir);
+ org_indir = kzalloc(indir_size, GFP_KERNEL);
+ if (!org_indir)
+ return -ENOMEM;
+
+ /* store the orginal indirection */
+ ops->get_rss(h, org_indir, NULL, NULL);
+
+ cur_indir = kzalloc(indir_size, GFP_KERNEL);
+ if (!cur_indir) {
+ ret = -ENOMEM;
+ goto cur_indir_alloc_err;
+ }
+
+ /* set loopback */
+ if (hns_enable_serdes_lb(ndev)) {
+ ret = -EINVAL;
+ goto enable_serdes_lb_err;
+ }
+
+ /* foreach every rx ring to clear fetch desc */
+ for (i = 0; i < h->q_num; i++) {
+ ring = &h->qs[i]->rx_ring;
+ head = readl_relaxed(ring->io_base + RCB_REG_HEAD);
+ tail = readl_relaxed(ring->io_base + RCB_REG_TAIL);
+ found = false;
+ fetch_num = ring_dist(ring, head, tail);
+
+ while (head != tail) {
+ if (ring->desc_cb[head].page_offset != 0) {
+ found = true;
+ break;
+ }
+
+ head++;
+ if (head == ring->desc_num)
+ head = 0;
+ }
+
+ if (found) {
+ for (j = 0; j < indir_size / sizeof(*org_indir); j++)
+ cur_indir[j] = i;
+ ops->set_rss(h, cur_indir, NULL, 0);
+
+ for (j = 0; j < fetch_num; j++) {
+ /* alloc one skb and init */
+ skb = hns_assemble_skb(ndev);
+ if (!skb)
+ goto out;
+ rd = &tx_ring_data(priv, skb->queue_mapping);
+ hns_nic_net_xmit_hw(ndev, skb, rd);
+
+ retry_times = 0;
+ while (retry_times++ < 10) {
+ mdelay(10);
+ /* clean rx */
+ rd = &rx_ring_data(priv, i);
+ if (rd->poll_one(rd, fetch_num,
+ hns_nic_drop_rx_fetch))
+ break;
+ }
+
+ retry_times = 0;
+ while (retry_times++ < 10) {
+ mdelay(10);
+ /* clean tx ring 0 send package */
+ rd = &tx_ring_data(priv,
+ HNS_LB_TX_RING);
+ if (rd->poll_one(rd, fetch_num, NULL))
+ break;
+ }
+ }
+ }
+ }
+
+out:
+ /* restore everything */
+ ops->set_rss(h, org_indir, NULL, 0);
+ hns_disable_serdes_lb(ndev);
+enable_serdes_lb_err:
+ kfree(cur_indir);
+cur_indir_alloc_err:
+ kfree(org_indir);
+
+ return ret;
+}
+
static int hns_nic_change_mtu(struct net_device *ndev, int new_mtu)
{
struct hns_nic_priv *priv = netdev_priv(ndev);
struct hnae_handle *h = priv->ae_handle;
+ bool if_running = netif_running(ndev);
int ret;
+ /* MTU < 68 is an error and causes problems on some kernels */
+ if (new_mtu < 68)
+ return -EINVAL;
+
+ /* MTU no change */
+ if (new_mtu == ndev->mtu)
+ return 0;
+
if (!h->dev->ops->set_mtu)
return -ENOTSUPP;
- if (netif_running(ndev)) {
+ if (if_running) {
(void)hns_nic_net_stop(ndev);
msleep(100);
-
- ret = h->dev->ops->set_mtu(h, new_mtu);
- if (ret)
- netdev_err(ndev, "set mtu fail, return value %d\n",
- ret);
-
- if (hns_nic_net_open(ndev))
- netdev_err(ndev, "hns net open fail\n");
- } else {
- ret = h->dev->ops->set_mtu(h, new_mtu);
}
- if (!ret)
- ndev->mtu = new_mtu;
+ if (priv->enet_ver != AE_VERSION_1 &&
+ ndev->mtu <= BD_SIZE_2048_MAX_MTU &&
+ new_mtu > BD_SIZE_2048_MAX_MTU) {
+ /* update desc */
+ hnae_reinit_all_ring_desc(h);
+
+ /* clear the package which the chip has fetched */
+ ret = hns_nic_clear_all_rx_fetch(ndev);
+
+ /* the page offset must be consist with desc */
+ hnae_reinit_all_ring_page_off(h);
+
+ if (ret) {
+ netdev_err(ndev, "clear the fetched desc fail\n");
+ goto out;
+ }
+ }
+
+ ret = h->dev->ops->set_mtu(h, new_mtu);
+ if (ret) {
+ netdev_err(ndev, "set mtu fail, return value %d\n",
+ ret);
+ goto out;
+ }
+
+ /* finally, set new mtu to netdevice */
+ ndev->mtu = new_mtu;
+
+out:
+ if (if_running) {
+ if (hns_nic_net_open(ndev)) {
+ netdev_err(ndev, "hns net open fail\n");
+ ret = -EINVAL;
+ }
+ }
return ret;
}
@@ -1791,7 +2013,7 @@
static void hns_nic_service_event_complete(struct hns_nic_priv *priv)
{
WARN_ON(!test_bit(NIC_STATE_SERVICE_SCHED, &priv->state));
-
+ /* make sure to commit the things */
smp_mb__before_atomic();
clear_bit(NIC_STATE_SERVICE_SCHED, &priv->state);
}
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_enet.h b/drivers/net/ethernet/hisilicon/hns/hns_enet.h
index 5b412de..1b83232 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_enet.h
+++ b/drivers/net/ethernet/hisilicon/hns/hns_enet.h
@@ -37,10 +37,11 @@
struct hns_nic_ring_data {
struct hnae_ring *ring;
struct napi_struct napi;
+ cpumask_t mask; /* affinity mask */
int queue_index;
int (*poll_one)(struct hns_nic_ring_data *, int, void *);
void (*ex_process)(struct hns_nic_ring_data *, struct sk_buff *);
- void (*fini_process)(struct hns_nic_ring_data *);
+ bool (*fini_process)(struct hns_nic_ring_data *);
};
/* compatible the difference between two versions */
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_ethtool.c b/drivers/net/ethernet/hisilicon/hns/hns_ethtool.c
index 3ac2183..b8fab14 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_ethtool.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_ethtool.c
@@ -146,7 +146,7 @@
/* When there is no phy, autoneg is off. */
cmd->base.autoneg = false;
- cmd->base.cmd = speed;
+ cmd->base.speed = speed;
cmd->base.duplex = duplex;
if (net_dev->phydev)
@@ -764,14 +764,14 @@
ec->use_adaptive_tx_coalesce = 1;
if ((!ops->get_coalesce_usecs) ||
- (!ops->get_rx_max_coalesced_frames))
+ (!ops->get_max_coalesced_frames))
return -ESRCH;
ops->get_coalesce_usecs(priv->ae_handle,
&ec->tx_coalesce_usecs,
&ec->rx_coalesce_usecs);
- ops->get_rx_max_coalesced_frames(
+ ops->get_max_coalesced_frames(
priv->ae_handle,
&ec->tx_max_coalesced_frames,
&ec->rx_max_coalesced_frames);
@@ -801,30 +801,28 @@
{
struct hns_nic_priv *priv = netdev_priv(net_dev);
struct hnae_ae_ops *ops;
- int ret;
+ int rc1, rc2;
ops = priv->ae_handle->dev->ops;
if (ec->tx_coalesce_usecs != ec->rx_coalesce_usecs)
return -EINVAL;
- if (ec->rx_max_coalesced_frames != ec->tx_max_coalesced_frames)
- return -EINVAL;
-
if ((!ops->set_coalesce_usecs) ||
(!ops->set_coalesce_frames))
return -ESRCH;
- ret = ops->set_coalesce_usecs(priv->ae_handle,
+ rc1 = ops->set_coalesce_usecs(priv->ae_handle,
ec->rx_coalesce_usecs);
- if (ret)
- return ret;
- ret = ops->set_coalesce_frames(
- priv->ae_handle,
- ec->rx_max_coalesced_frames);
+ rc2 = ops->set_coalesce_frames(priv->ae_handle,
+ ec->tx_max_coalesced_frames,
+ ec->rx_max_coalesced_frames);
- return ret;
+ if (rc1 || rc2)
+ return -EINVAL;
+
+ return 0;
}
/**
@@ -1253,12 +1251,10 @@
ops = priv->ae_handle->dev->ops;
- /* currently hfunc can only be Toeplitz hash */
- if (key ||
- (hfunc != ETH_RSS_HASH_NO_CHANGE && hfunc != ETH_RSS_HASH_TOP))
+ if (hfunc != ETH_RSS_HASH_NO_CHANGE && hfunc != ETH_RSS_HASH_TOP) {
+ netdev_err(netdev, "Invalid hfunc!\n");
return -EOPNOTSUPP;
- if (!indir)
- return 0;
+ }
return ops->set_rss(priv->ae_handle, indir, key, hfunc);
}
diff --git a/drivers/net/ethernet/hisilicon/hns_mdio.c b/drivers/net/ethernet/hisilicon/hns_mdio.c
index 501eb20..e5221d9 100644
--- a/drivers/net/ethernet/hisilicon/hns_mdio.c
+++ b/drivers/net/ethernet/hisilicon/hns_mdio.c
@@ -23,17 +23,9 @@
#include <linux/phy.h>
#include <linux/platform_device.h>
#include <linux/regmap.h>
-#include <linux/spinlock_types.h>
#define MDIO_DRV_NAME "Hi-HNS_MDIO"
#define MDIO_BUS_NAME "Hisilicon MII Bus"
-#define MDIO_DRV_VERSION "1.3.0"
-#define MDIO_COPYRIGHT "Copyright(c) 2015 Huawei Corporation."
-#define MDIO_DRV_STRING MDIO_BUS_NAME
-#define MDIO_DEFAULT_DEVICE_DESCR MDIO_BUS_NAME
-
-#define MDIO_CTL_DEV_ADDR(x) (x & 0x1f)
-#define MDIO_CTL_PORT_ADDR(x) ((x & 0x1f) << 5)
#define MDIO_TIMEOUT 1000000
@@ -64,9 +56,7 @@
#define MDIO_CMD_DEVAD_S 0
#define MDIO_CMD_PRTAD_M 0x1f
#define MDIO_CMD_PRTAD_S 5
-#define MDIO_CMD_OP_M 0x3
#define MDIO_CMD_OP_S 10
-#define MDIO_CMD_ST_M 0x3
#define MDIO_CMD_ST_S 12
#define MDIO_CMD_START_B 14
@@ -185,18 +175,20 @@
static int hns_mdio_wait_ready(struct mii_bus *bus)
{
struct hns_mdio_device *mdio_dev = bus->priv;
+ u32 cmd_reg_value;
int i;
- u32 cmd_reg_value = 1;
/* waitting for MDIO_COMMAND_REG 's mdio_start==0 */
/* after that can do read or write*/
- for (i = 0; cmd_reg_value; i++) {
+ for (i = 0; i < MDIO_TIMEOUT; i++) {
cmd_reg_value = MDIO_GET_REG_BIT(mdio_dev,
MDIO_COMMAND_REG,
MDIO_CMD_START_B);
- if (i == MDIO_TIMEOUT)
- return -ETIMEDOUT;
+ if (!cmd_reg_value)
+ break;
}
+ if ((i == MDIO_TIMEOUT) && cmd_reg_value)
+ return -ETIMEDOUT;
return 0;
}
diff --git a/drivers/net/ethernet/ibm/emac/Makefile b/drivers/net/ethernet/ibm/emac/Makefile
index eba2183..98768ba 100644
--- a/drivers/net/ethernet/ibm/emac/Makefile
+++ b/drivers/net/ethernet/ibm/emac/Makefile
@@ -8,4 +8,3 @@
ibm_emac-$(CONFIG_IBM_EMAC_ZMII) += zmii.o
ibm_emac-$(CONFIG_IBM_EMAC_RGMII) += rgmii.o
ibm_emac-$(CONFIG_IBM_EMAC_TAH) += tah.o
-ibm_emac-$(CONFIG_IBM_EMAC_DEBUG) += debug.o
diff --git a/drivers/net/ethernet/ibm/emac/core.c b/drivers/net/ethernet/ibm/emac/core.c
index c44036d..508923f 100644
--- a/drivers/net/ethernet/ibm/emac/core.c
+++ b/drivers/net/ethernet/ibm/emac/core.c
@@ -1929,7 +1929,7 @@
struct emac_instance *dev = netdev_priv(ndev);
struct emac_stats *st = &dev->stats;
struct emac_error_stats *est = &dev->estats;
- struct net_device_stats *nst = &dev->nstats;
+ struct net_device_stats *nst = &ndev->stats;
unsigned long flags;
DBG2(dev, "stats" NL);
@@ -3173,8 +3173,6 @@
printk("%s: found %s PHY (0x%02x)\n", ndev->name,
dev->phy.def->name, dev->phy.address);
- emac_dbg_register(dev);
-
/* Life is good */
return 0;
@@ -3243,7 +3241,6 @@
mal_unregister_commac(dev->mal, &dev->commac);
emac_put_deps(dev);
- emac_dbg_unregister(dev);
iounmap(dev->emacp);
if (dev->wol_irq)
@@ -3326,9 +3323,6 @@
printk(KERN_INFO DRV_DESC ", version " DRV_VERSION "\n");
- /* Init debug stuff */
- emac_init_debug();
-
/* Build EMAC boot list */
emac_make_bootlist();
@@ -3373,7 +3367,6 @@
rgmii_exit();
zmii_exit();
mal_exit();
- emac_fini_debug();
/* Destroy EMAC boot list */
for (i = 0; i < EMAC_BOOT_LIST_SIZE; i++)
diff --git a/drivers/net/ethernet/ibm/emac/core.h b/drivers/net/ethernet/ibm/emac/core.h
index 0710a66..f10e156 100644
--- a/drivers/net/ethernet/ibm/emac/core.h
+++ b/drivers/net/ethernet/ibm/emac/core.h
@@ -265,7 +265,6 @@
/* Stats
*/
struct emac_error_stats estats;
- struct net_device_stats nstats;
struct emac_stats stats;
/* Misc
diff --git a/drivers/net/ethernet/ibm/emac/debug.c b/drivers/net/ethernet/ibm/emac/debug.c
deleted file mode 100644
index a559f32..0000000
--- a/drivers/net/ethernet/ibm/emac/debug.c
+++ /dev/null
@@ -1,270 +0,0 @@
-/*
- * drivers/net/ethernet/ibm/emac/debug.c
- *
- * Driver for PowerPC 4xx on-chip ethernet controller, debug print routines.
- *
- * Copyright 2007 Benjamin Herrenschmidt, IBM Corp.
- * <benh@kernel.crashing.org>
- *
- * Based on the arch/ppc version of the driver:
- *
- * Copyright (c) 2004, 2005 Zultys Technologies
- * Eugene Surovegin <eugene.surovegin@zultys.com> or <ebs@ebshome.net>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the
- * Free Software Foundation; either version 2 of the License, or (at your
- * option) any later version.
- *
- */
-#include <linux/init.h>
-#include <linux/module.h>
-#include <linux/kernel.h>
-#include <linux/netdevice.h>
-#include <linux/sysrq.h>
-#include <asm/io.h>
-
-#include "core.h"
-
-static DEFINE_SPINLOCK(emac_dbg_lock);
-
-static void emac_desc_dump(struct emac_instance *p)
-{
- int i;
- printk("** EMAC %s TX BDs **\n"
- " tx_cnt = %d tx_slot = %d ack_slot = %d\n",
- p->ofdev->dev.of_node->full_name,
- p->tx_cnt, p->tx_slot, p->ack_slot);
- for (i = 0; i < NUM_TX_BUFF / 2; ++i)
- printk
- ("bd[%2d] 0x%08x %c 0x%04x %4u - bd[%2d] 0x%08x %c 0x%04x %4u\n",
- i, p->tx_desc[i].data_ptr, p->tx_skb[i] ? 'V' : ' ',
- p->tx_desc[i].ctrl, p->tx_desc[i].data_len,
- NUM_TX_BUFF / 2 + i,
- p->tx_desc[NUM_TX_BUFF / 2 + i].data_ptr,
- p->tx_skb[NUM_TX_BUFF / 2 + i] ? 'V' : ' ',
- p->tx_desc[NUM_TX_BUFF / 2 + i].ctrl,
- p->tx_desc[NUM_TX_BUFF / 2 + i].data_len);
-
- printk("** EMAC %s RX BDs **\n"
- " rx_slot = %d flags = 0x%lx rx_skb_size = %d rx_sync_size = %d\n"
- " rx_sg_skb = 0x%p\n",
- p->ofdev->dev.of_node->full_name,
- p->rx_slot, p->commac.flags, p->rx_skb_size,
- p->rx_sync_size, p->rx_sg_skb);
- for (i = 0; i < NUM_RX_BUFF / 2; ++i)
- printk
- ("bd[%2d] 0x%08x %c 0x%04x %4u - bd[%2d] 0x%08x %c 0x%04x %4u\n",
- i, p->rx_desc[i].data_ptr, p->rx_skb[i] ? 'V' : ' ',
- p->rx_desc[i].ctrl, p->rx_desc[i].data_len,
- NUM_RX_BUFF / 2 + i,
- p->rx_desc[NUM_RX_BUFF / 2 + i].data_ptr,
- p->rx_skb[NUM_RX_BUFF / 2 + i] ? 'V' : ' ',
- p->rx_desc[NUM_RX_BUFF / 2 + i].ctrl,
- p->rx_desc[NUM_RX_BUFF / 2 + i].data_len);
-}
-
-static void emac_mac_dump(struct emac_instance *dev)
-{
- struct emac_regs __iomem *p = dev->emacp;
- const int xaht_regs = EMAC_XAHT_REGS(dev);
- u32 *gaht_base = emac_gaht_base(dev);
- u32 *iaht_base = emac_iaht_base(dev);
- int emac4sync = emac_has_feature(dev, EMAC_FTR_EMAC4SYNC);
- int n;
-
- printk("** EMAC %s registers **\n"
- "MR0 = 0x%08x MR1 = 0x%08x TMR0 = 0x%08x TMR1 = 0x%08x\n"
- "RMR = 0x%08x ISR = 0x%08x ISER = 0x%08x\n"
- "IAR = %04x%08x VTPID = 0x%04x VTCI = 0x%04x\n",
- dev->ofdev->dev.of_node->full_name,
- in_be32(&p->mr0), in_be32(&p->mr1),
- in_be32(&p->tmr0), in_be32(&p->tmr1),
- in_be32(&p->rmr), in_be32(&p->isr), in_be32(&p->iser),
- in_be32(&p->iahr), in_be32(&p->ialr), in_be32(&p->vtpid),
- in_be32(&p->vtci)
- );
-
- if (emac4sync)
- printk("MAR = %04x%08x MMAR = %04x%08x\n",
- in_be32(&p->u0.emac4sync.mahr),
- in_be32(&p->u0.emac4sync.malr),
- in_be32(&p->u0.emac4sync.mmahr),
- in_be32(&p->u0.emac4sync.mmalr)
- );
-
- for (n = 0; n < xaht_regs; n++)
- printk("IAHT%02d = 0x%08x\n", n + 1, in_be32(iaht_base + n));
-
- for (n = 0; n < xaht_regs; n++)
- printk("GAHT%02d = 0x%08x\n", n + 1, in_be32(gaht_base + n));
-
- printk("LSA = %04x%08x IPGVR = 0x%04x\n"
- "STACR = 0x%08x TRTR = 0x%08x RWMR = 0x%08x\n"
- "OCTX = 0x%08x OCRX = 0x%08x\n",
- in_be32(&p->lsah), in_be32(&p->lsal), in_be32(&p->ipgvr),
- in_be32(&p->stacr), in_be32(&p->trtr), in_be32(&p->rwmr),
- in_be32(&p->octx), in_be32(&p->ocrx)
- );
-
- if (!emac4sync) {
- printk("IPCR = 0x%08x\n",
- in_be32(&p->u1.emac4.ipcr)
- );
- } else {
- printk("REVID = 0x%08x TPC = 0x%08x\n",
- in_be32(&p->u1.emac4sync.revid),
- in_be32(&p->u1.emac4sync.tpc)
- );
- }
-
- emac_desc_dump(dev);
-}
-
-static void emac_mal_dump(struct mal_instance *mal)
-{
- int i;
-
- printk("** MAL %s Registers **\n"
- "CFG = 0x%08x ESR = 0x%08x IER = 0x%08x\n"
- "TX|CASR = 0x%08x CARR = 0x%08x EOBISR = 0x%08x DEIR = 0x%08x\n"
- "RX|CASR = 0x%08x CARR = 0x%08x EOBISR = 0x%08x DEIR = 0x%08x\n",
- mal->ofdev->dev.of_node->full_name,
- get_mal_dcrn(mal, MAL_CFG), get_mal_dcrn(mal, MAL_ESR),
- get_mal_dcrn(mal, MAL_IER),
- get_mal_dcrn(mal, MAL_TXCASR), get_mal_dcrn(mal, MAL_TXCARR),
- get_mal_dcrn(mal, MAL_TXEOBISR), get_mal_dcrn(mal, MAL_TXDEIR),
- get_mal_dcrn(mal, MAL_RXCASR), get_mal_dcrn(mal, MAL_RXCARR),
- get_mal_dcrn(mal, MAL_RXEOBISR), get_mal_dcrn(mal, MAL_RXDEIR)
- );
-
- printk("TX|");
- for (i = 0; i < mal->num_tx_chans; ++i) {
- if (i && !(i % 4))
- printk("\n ");
- printk("CTP%d = 0x%08x ", i, get_mal_dcrn(mal, MAL_TXCTPR(i)));
- }
- printk("\nRX|");
- for (i = 0; i < mal->num_rx_chans; ++i) {
- if (i && !(i % 4))
- printk("\n ");
- printk("CTP%d = 0x%08x ", i, get_mal_dcrn(mal, MAL_RXCTPR(i)));
- }
- printk("\n ");
- for (i = 0; i < mal->num_rx_chans; ++i) {
- u32 r = get_mal_dcrn(mal, MAL_RCBS(i));
- if (i && !(i % 3))
- printk("\n ");
- printk("RCBS%d = 0x%08x (%d) ", i, r, r * 16);
- }
- printk("\n");
-}
-
-static struct emac_instance *__emacs[4];
-static struct mal_instance *__mals[1];
-
-void emac_dbg_register(struct emac_instance *dev)
-{
- unsigned long flags;
- int i;
-
- spin_lock_irqsave(&emac_dbg_lock, flags);
- for (i = 0; i < ARRAY_SIZE(__emacs); i++)
- if (__emacs[i] == NULL) {
- __emacs[i] = dev;
- break;
- }
- spin_unlock_irqrestore(&emac_dbg_lock, flags);
-}
-
-void emac_dbg_unregister(struct emac_instance *dev)
-{
- unsigned long flags;
- int i;
-
- spin_lock_irqsave(&emac_dbg_lock, flags);
- for (i = 0; i < ARRAY_SIZE(__emacs); i++)
- if (__emacs[i] == dev) {
- __emacs[i] = NULL;
- break;
- }
- spin_unlock_irqrestore(&emac_dbg_lock, flags);
-}
-
-void mal_dbg_register(struct mal_instance *mal)
-{
- unsigned long flags;
- int i;
-
- spin_lock_irqsave(&emac_dbg_lock, flags);
- for (i = 0; i < ARRAY_SIZE(__mals); i++)
- if (__mals[i] == NULL) {
- __mals[i] = mal;
- break;
- }
- spin_unlock_irqrestore(&emac_dbg_lock, flags);
-}
-
-void mal_dbg_unregister(struct mal_instance *mal)
-{
- unsigned long flags;
- int i;
-
- spin_lock_irqsave(&emac_dbg_lock, flags);
- for (i = 0; i < ARRAY_SIZE(__mals); i++)
- if (__mals[i] == mal) {
- __mals[i] = NULL;
- break;
- }
- spin_unlock_irqrestore(&emac_dbg_lock, flags);
-}
-
-void emac_dbg_dump_all(void)
-{
- unsigned int i;
- unsigned long flags;
-
- spin_lock_irqsave(&emac_dbg_lock, flags);
-
- for (i = 0; i < ARRAY_SIZE(__mals); ++i)
- if (__mals[i])
- emac_mal_dump(__mals[i]);
-
- for (i = 0; i < ARRAY_SIZE(__emacs); ++i)
- if (__emacs[i])
- emac_mac_dump(__emacs[i]);
-
- spin_unlock_irqrestore(&emac_dbg_lock, flags);
-}
-
-#if defined(CONFIG_MAGIC_SYSRQ)
-static void emac_sysrq_handler(int key)
-{
- emac_dbg_dump_all();
-}
-
-static struct sysrq_key_op emac_sysrq_op = {
- .handler = emac_sysrq_handler,
- .help_msg = "emac(c)",
- .action_msg = "Show EMAC(s) status",
-};
-
-int __init emac_init_debug(void)
-{
- return register_sysrq_key('c', &emac_sysrq_op);
-}
-
-void __exit emac_fini_debug(void)
-{
- unregister_sysrq_key('c', &emac_sysrq_op);
-}
-
-#else
-int __init emac_init_debug(void)
-{
- return 0;
-}
-void __exit emac_fini_debug(void)
-{
-}
-#endif /* CONFIG_MAGIC_SYSRQ */
diff --git a/drivers/net/ethernet/ibm/emac/debug.h b/drivers/net/ethernet/ibm/emac/debug.h
index 9c45efe..5bdfc17 100644
--- a/drivers/net/ethernet/ibm/emac/debug.h
+++ b/drivers/net/ethernet/ibm/emac/debug.h
@@ -25,32 +25,9 @@
#include "core.h"
#if defined(CONFIG_IBM_EMAC_DEBUG)
-
-struct emac_instance;
-struct mal_instance;
-
-void emac_dbg_register(struct emac_instance *dev);
-void emac_dbg_unregister(struct emac_instance *dev);
-void mal_dbg_register(struct mal_instance *mal);
-void mal_dbg_unregister(struct mal_instance *mal);
-int emac_init_debug(void) __init;
-void emac_fini_debug(void) __exit;
-void emac_dbg_dump_all(void);
-
# define DBG_LEVEL 1
-
#else
-
-# define emac_dbg_register(x) do { } while(0)
-# define emac_dbg_unregister(x) do { } while(0)
-# define mal_dbg_register(x) do { } while(0)
-# define mal_dbg_unregister(x) do { } while(0)
-# define emac_init_debug() do { } while(0)
-# define emac_fini_debug() do { } while(0)
-# define emac_dbg_dump_all() do { } while(0)
-
# define DBG_LEVEL 0
-
#endif
#define EMAC_DBG(d, name, fmt, arg...) \
diff --git a/drivers/net/ethernet/ibm/emac/mal.c b/drivers/net/ethernet/ibm/emac/mal.c
index cd3227b..91b1a55 100644
--- a/drivers/net/ethernet/ibm/emac/mal.c
+++ b/drivers/net/ethernet/ibm/emac/mal.c
@@ -695,8 +695,6 @@
wmb();
platform_set_drvdata(ofdev, mal);
- mal_dbg_register(mal);
-
return 0;
fail6:
@@ -740,8 +738,6 @@
mal_reset(mal);
- mal_dbg_unregister(mal);
-
dma_free_coherent(&ofdev->dev,
sizeof(struct mal_descriptor) *
(NUM_TX_BUFF * mal->num_tx_chans +
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index 7acda04..ed8780c 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -146,7 +146,6 @@
struct vio_dev *vdev;
struct net_device *netdev;
struct napi_struct napi;
- struct net_device_stats stats;
unsigned int mcastFilterSize;
void * buffer_list_addr;
void * filter_list_addr;
diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index b23d654..4fcd2f0 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -65,7 +65,6 @@
#include <linux/irq.h>
#include <linux/kthread.h>
#include <linux/seq_file.h>
-#include <linux/debugfs.h>
#include <linux/interrupt.h>
#include <net/net_namespace.h>
#include <asm/hvcall.h>
@@ -75,6 +74,7 @@
#include <linux/uaccess.h>
#include <asm/firmware.h>
#include <linux/workqueue.h>
+#include <linux/if_vlan.h>
#include "ibmvnic.h"
@@ -89,7 +89,6 @@
static int ibmvnic_version = IBMVNIC_INITIAL_VERSION;
static int ibmvnic_remove(struct vio_dev *);
static void release_sub_crqs(struct ibmvnic_adapter *);
-static void release_sub_crqs_no_irqs(struct ibmvnic_adapter *);
static int ibmvnic_reset_crq(struct ibmvnic_adapter *);
static int ibmvnic_send_crq_init(struct ibmvnic_adapter *);
static int ibmvnic_reenable_crq_queue(struct ibmvnic_adapter *);
@@ -110,6 +109,11 @@
static void send_map_query(struct ibmvnic_adapter *adapter);
static void send_request_map(struct ibmvnic_adapter *, dma_addr_t, __be32, u8);
static void send_request_unmap(struct ibmvnic_adapter *, u8);
+static void send_login(struct ibmvnic_adapter *adapter);
+static void send_cap_queries(struct ibmvnic_adapter *adapter);
+static int init_sub_crq_irqs(struct ibmvnic_adapter *adapter);
+static int ibmvnic_init(struct ibmvnic_adapter *);
+static void release_crq_queue(struct ibmvnic_adapter *);
struct ibmvnic_stat {
char name[ETH_GSTRING_LEN];
@@ -159,21 +163,6 @@
return rc;
}
-/* net_device_ops functions */
-
-static void init_rx_pool(struct ibmvnic_adapter *adapter,
- struct ibmvnic_rx_pool *rx_pool, int num, int index,
- int buff_size, int active)
-{
- netdev_dbg(adapter->netdev,
- "Initializing rx_pool %d, %d buffs, %d bytes each\n",
- index, num, buff_size);
- rx_pool->size = num;
- rx_pool->index = index;
- rx_pool->buff_size = buff_size;
- rx_pool->active = active;
-}
-
static int alloc_long_term_buff(struct ibmvnic_adapter *adapter,
struct ibmvnic_long_term_buff *ltb, int size)
{
@@ -202,45 +191,12 @@
{
struct device *dev = &adapter->vdev->dev;
- dma_free_coherent(dev, ltb->size, ltb->buff, ltb->addr);
+ if (!ltb->buff)
+ return;
+
if (!adapter->failover)
send_request_unmap(adapter, ltb->map_id);
-}
-
-static int alloc_rx_pool(struct ibmvnic_adapter *adapter,
- struct ibmvnic_rx_pool *pool)
-{
- struct device *dev = &adapter->vdev->dev;
- int i;
-
- pool->free_map = kcalloc(pool->size, sizeof(int), GFP_KERNEL);
- if (!pool->free_map)
- return -ENOMEM;
-
- pool->rx_buff = kcalloc(pool->size, sizeof(struct ibmvnic_rx_buff),
- GFP_KERNEL);
-
- if (!pool->rx_buff) {
- dev_err(dev, "Couldn't alloc rx buffers\n");
- kfree(pool->free_map);
- return -ENOMEM;
- }
-
- if (alloc_long_term_buff(adapter, &pool->long_term_buff,
- pool->size * pool->buff_size)) {
- kfree(pool->free_map);
- kfree(pool->rx_buff);
- return -ENOMEM;
- }
-
- for (i = 0; i < pool->size; ++i)
- pool->free_map[i] = i;
-
- atomic_set(&pool->available, 0);
- pool->next_alloc = 0;
- pool->next_free = 0;
-
- return 0;
+ dma_free_coherent(dev, ltb->size, ltb->buff, ltb->addr);
}
static void replenish_rx_pool(struct ibmvnic_adapter *adapter,
@@ -347,93 +303,195 @@
}
}
-static void free_rx_pool(struct ibmvnic_adapter *adapter,
- struct ibmvnic_rx_pool *pool)
+static void release_stats_token(struct ibmvnic_adapter *adapter)
{
- int i;
+ struct device *dev = &adapter->vdev->dev;
- kfree(pool->free_map);
- pool->free_map = NULL;
-
- if (!pool->rx_buff)
+ if (!adapter->stats_token)
return;
- for (i = 0; i < pool->size; i++) {
- if (pool->rx_buff[i].skb) {
- dev_kfree_skb_any(pool->rx_buff[i].skb);
- pool->rx_buff[i].skb = NULL;
- }
- }
- kfree(pool->rx_buff);
- pool->rx_buff = NULL;
+ dma_unmap_single(dev, adapter->stats_token,
+ sizeof(struct ibmvnic_statistics),
+ DMA_FROM_DEVICE);
+ adapter->stats_token = 0;
}
-static int ibmvnic_open(struct net_device *netdev)
+static int init_stats_token(struct ibmvnic_adapter *adapter)
+{
+ struct device *dev = &adapter->vdev->dev;
+ dma_addr_t stok;
+
+ stok = dma_map_single(dev, &adapter->stats,
+ sizeof(struct ibmvnic_statistics),
+ DMA_FROM_DEVICE);
+ if (dma_mapping_error(dev, stok)) {
+ dev_err(dev, "Couldn't map stats buffer\n");
+ return -1;
+ }
+
+ adapter->stats_token = stok;
+ return 0;
+}
+
+static void release_rx_pools(struct ibmvnic_adapter *adapter)
+{
+ struct ibmvnic_rx_pool *rx_pool;
+ int rx_scrqs;
+ int i, j;
+
+ if (!adapter->rx_pool)
+ return;
+
+ rx_scrqs = be32_to_cpu(adapter->login_rsp_buf->num_rxadd_subcrqs);
+ for (i = 0; i < rx_scrqs; i++) {
+ rx_pool = &adapter->rx_pool[i];
+
+ kfree(rx_pool->free_map);
+ free_long_term_buff(adapter, &rx_pool->long_term_buff);
+
+ if (!rx_pool->rx_buff)
+ continue;
+
+ for (j = 0; j < rx_pool->size; j++) {
+ if (rx_pool->rx_buff[j].skb) {
+ dev_kfree_skb_any(rx_pool->rx_buff[i].skb);
+ rx_pool->rx_buff[i].skb = NULL;
+ }
+ }
+
+ kfree(rx_pool->rx_buff);
+ }
+
+ kfree(adapter->rx_pool);
+ adapter->rx_pool = NULL;
+}
+
+static int init_rx_pools(struct net_device *netdev)
+{
+ struct ibmvnic_adapter *adapter = netdev_priv(netdev);
+ struct device *dev = &adapter->vdev->dev;
+ struct ibmvnic_rx_pool *rx_pool;
+ int rxadd_subcrqs;
+ u64 *size_array;
+ int i, j;
+
+ rxadd_subcrqs =
+ be32_to_cpu(adapter->login_rsp_buf->num_rxadd_subcrqs);
+ size_array = (u64 *)((u8 *)(adapter->login_rsp_buf) +
+ be32_to_cpu(adapter->login_rsp_buf->off_rxadd_buff_size));
+
+ adapter->rx_pool = kcalloc(rxadd_subcrqs,
+ sizeof(struct ibmvnic_rx_pool),
+ GFP_KERNEL);
+ if (!adapter->rx_pool) {
+ dev_err(dev, "Failed to allocate rx pools\n");
+ return -1;
+ }
+
+ for (i = 0; i < rxadd_subcrqs; i++) {
+ rx_pool = &adapter->rx_pool[i];
+
+ netdev_dbg(adapter->netdev,
+ "Initializing rx_pool %d, %lld buffs, %lld bytes each\n",
+ i, adapter->req_rx_add_entries_per_subcrq,
+ be64_to_cpu(size_array[i]));
+
+ rx_pool->size = adapter->req_rx_add_entries_per_subcrq;
+ rx_pool->index = i;
+ rx_pool->buff_size = be64_to_cpu(size_array[i]);
+ rx_pool->active = 1;
+
+ rx_pool->free_map = kcalloc(rx_pool->size, sizeof(int),
+ GFP_KERNEL);
+ if (!rx_pool->free_map) {
+ release_rx_pools(adapter);
+ return -1;
+ }
+
+ rx_pool->rx_buff = kcalloc(rx_pool->size,
+ sizeof(struct ibmvnic_rx_buff),
+ GFP_KERNEL);
+ if (!rx_pool->rx_buff) {
+ dev_err(dev, "Couldn't alloc rx buffers\n");
+ release_rx_pools(adapter);
+ return -1;
+ }
+
+ if (alloc_long_term_buff(adapter, &rx_pool->long_term_buff,
+ rx_pool->size * rx_pool->buff_size)) {
+ release_rx_pools(adapter);
+ return -1;
+ }
+
+ for (j = 0; j < rx_pool->size; ++j)
+ rx_pool->free_map[j] = j;
+
+ atomic_set(&rx_pool->available, 0);
+ rx_pool->next_alloc = 0;
+ rx_pool->next_free = 0;
+ }
+
+ return 0;
+}
+
+static void release_tx_pools(struct ibmvnic_adapter *adapter)
+{
+ struct ibmvnic_tx_pool *tx_pool;
+ int i, tx_scrqs;
+
+ if (!adapter->tx_pool)
+ return;
+
+ tx_scrqs = be32_to_cpu(adapter->login_rsp_buf->num_txsubm_subcrqs);
+ for (i = 0; i < tx_scrqs; i++) {
+ tx_pool = &adapter->tx_pool[i];
+ kfree(tx_pool->tx_buff);
+ free_long_term_buff(adapter, &tx_pool->long_term_buff);
+ kfree(tx_pool->free_map);
+ }
+
+ kfree(adapter->tx_pool);
+ adapter->tx_pool = NULL;
+}
+
+static int init_tx_pools(struct net_device *netdev)
{
struct ibmvnic_adapter *adapter = netdev_priv(netdev);
struct device *dev = &adapter->vdev->dev;
struct ibmvnic_tx_pool *tx_pool;
- union ibmvnic_crq crq;
- int rxadd_subcrqs;
- u64 *size_array;
int tx_subcrqs;
int i, j;
- rxadd_subcrqs =
- be32_to_cpu(adapter->login_rsp_buf->num_rxadd_subcrqs);
- tx_subcrqs =
- be32_to_cpu(adapter->login_rsp_buf->num_txsubm_subcrqs);
- size_array = (u64 *)((u8 *)(adapter->login_rsp_buf) +
- be32_to_cpu(adapter->login_rsp_buf->
- off_rxadd_buff_size));
- adapter->map_id = 1;
- adapter->napi = kcalloc(adapter->req_rx_queues,
- sizeof(struct napi_struct), GFP_KERNEL);
- if (!adapter->napi)
- goto alloc_napi_failed;
- for (i = 0; i < adapter->req_rx_queues; i++) {
- netif_napi_add(netdev, &adapter->napi[i], ibmvnic_poll,
- NAPI_POLL_WEIGHT);
- napi_enable(&adapter->napi[i]);
- }
- adapter->rx_pool =
- kcalloc(rxadd_subcrqs, sizeof(struct ibmvnic_rx_pool), GFP_KERNEL);
-
- if (!adapter->rx_pool)
- goto rx_pool_arr_alloc_failed;
- send_map_query(adapter);
- for (i = 0; i < rxadd_subcrqs; i++) {
- init_rx_pool(adapter, &adapter->rx_pool[i],
- adapter->req_rx_add_entries_per_subcrq, i,
- be64_to_cpu(size_array[i]), 1);
- if (alloc_rx_pool(adapter, &adapter->rx_pool[i])) {
- dev_err(dev, "Couldn't alloc rx pool\n");
- goto rx_pool_alloc_failed;
- }
- }
- adapter->tx_pool =
- kcalloc(tx_subcrqs, sizeof(struct ibmvnic_tx_pool), GFP_KERNEL);
-
+ tx_subcrqs = be32_to_cpu(adapter->login_rsp_buf->num_txsubm_subcrqs);
+ adapter->tx_pool = kcalloc(tx_subcrqs,
+ sizeof(struct ibmvnic_tx_pool), GFP_KERNEL);
if (!adapter->tx_pool)
- goto tx_pool_arr_alloc_failed;
+ return -1;
+
for (i = 0; i < tx_subcrqs; i++) {
tx_pool = &adapter->tx_pool[i];
- tx_pool->tx_buff =
- kcalloc(adapter->req_tx_entries_per_subcrq,
- sizeof(struct ibmvnic_tx_buff), GFP_KERNEL);
- if (!tx_pool->tx_buff)
- goto tx_pool_alloc_failed;
+ tx_pool->tx_buff = kcalloc(adapter->req_tx_entries_per_subcrq,
+ sizeof(struct ibmvnic_tx_buff),
+ GFP_KERNEL);
+ if (!tx_pool->tx_buff) {
+ dev_err(dev, "tx pool buffer allocation failed\n");
+ release_tx_pools(adapter);
+ return -1;
+ }
if (alloc_long_term_buff(adapter, &tx_pool->long_term_buff,
adapter->req_tx_entries_per_subcrq *
- adapter->req_mtu))
- goto tx_ltb_alloc_failed;
+ adapter->req_mtu)) {
+ release_tx_pools(adapter);
+ return -1;
+ }
- tx_pool->free_map =
- kcalloc(adapter->req_tx_entries_per_subcrq,
- sizeof(int), GFP_KERNEL);
- if (!tx_pool->free_map)
- goto tx_fm_alloc_failed;
+ tx_pool->free_map = kcalloc(adapter->req_tx_entries_per_subcrq,
+ sizeof(int), GFP_KERNEL);
+ if (!tx_pool->free_map) {
+ release_tx_pools(adapter);
+ return -1;
+ }
for (j = 0; j < adapter->req_tx_entries_per_subcrq; j++)
tx_pool->free_map[j] = j;
@@ -441,20 +499,183 @@
tx_pool->consumer_index = 0;
tx_pool->producer_index = 0;
}
- adapter->bounce_buffer_size =
- (netdev->mtu + ETH_HLEN - 1) / PAGE_SIZE + 1;
- adapter->bounce_buffer = kmalloc(adapter->bounce_buffer_size,
- GFP_KERNEL);
- if (!adapter->bounce_buffer)
- goto bounce_alloc_failed;
- adapter->bounce_buffer_dma = dma_map_single(dev, adapter->bounce_buffer,
- adapter->bounce_buffer_size,
- DMA_TO_DEVICE);
- if (dma_mapping_error(dev, adapter->bounce_buffer_dma)) {
- dev_err(dev, "Couldn't map tx bounce buffer\n");
- goto bounce_map_failed;
+ return 0;
+}
+
+static void release_error_buffers(struct ibmvnic_adapter *adapter)
+{
+ struct device *dev = &adapter->vdev->dev;
+ struct ibmvnic_error_buff *error_buff, *tmp;
+ unsigned long flags;
+
+ spin_lock_irqsave(&adapter->error_list_lock, flags);
+ list_for_each_entry_safe(error_buff, tmp, &adapter->errors, list) {
+ list_del(&error_buff->list);
+ dma_unmap_single(dev, error_buff->dma, error_buff->len,
+ DMA_FROM_DEVICE);
+ kfree(error_buff->buff);
+ kfree(error_buff);
}
+ spin_unlock_irqrestore(&adapter->error_list_lock, flags);
+}
+
+static int ibmvnic_login(struct net_device *netdev)
+{
+ struct ibmvnic_adapter *adapter = netdev_priv(netdev);
+ unsigned long timeout = msecs_to_jiffies(30000);
+ struct device *dev = &adapter->vdev->dev;
+
+ do {
+ if (adapter->renegotiate) {
+ adapter->renegotiate = false;
+ release_sub_crqs(adapter);
+
+ reinit_completion(&adapter->init_done);
+ send_cap_queries(adapter);
+ if (!wait_for_completion_timeout(&adapter->init_done,
+ timeout)) {
+ dev_err(dev, "Capabilities query timeout\n");
+ return -1;
+ }
+ }
+
+ reinit_completion(&adapter->init_done);
+ send_login(adapter);
+ if (!wait_for_completion_timeout(&adapter->init_done,
+ timeout)) {
+ dev_err(dev, "Login timeout\n");
+ return -1;
+ }
+ } while (adapter->renegotiate);
+
+ return 0;
+}
+
+static void release_resources(struct ibmvnic_adapter *adapter)
+{
+ release_tx_pools(adapter);
+ release_rx_pools(adapter);
+
+ release_stats_token(adapter);
+ release_error_buffers(adapter);
+}
+
+static int set_link_state(struct ibmvnic_adapter *adapter, u8 link_state)
+{
+ struct net_device *netdev = adapter->netdev;
+ unsigned long timeout = msecs_to_jiffies(30000);
+ union ibmvnic_crq crq;
+ bool resend;
+ int rc;
+
+ if (adapter->logical_link_state == link_state) {
+ netdev_dbg(netdev, "Link state already %d\n", link_state);
+ return 0;
+ }
+
+ netdev_err(netdev, "setting link state %d\n", link_state);
+ memset(&crq, 0, sizeof(crq));
+ crq.logical_link_state.first = IBMVNIC_CRQ_CMD;
+ crq.logical_link_state.cmd = LOGICAL_LINK_STATE;
+ crq.logical_link_state.link_state = link_state;
+
+ do {
+ resend = false;
+
+ reinit_completion(&adapter->init_done);
+ rc = ibmvnic_send_crq(adapter, &crq);
+ if (rc) {
+ netdev_err(netdev, "Failed to set link state\n");
+ return rc;
+ }
+
+ if (!wait_for_completion_timeout(&adapter->init_done,
+ timeout)) {
+ netdev_err(netdev, "timeout setting link state\n");
+ return -1;
+ }
+
+ if (adapter->init_done_rc == 1) {
+ /* Partuial success, delay and re-send */
+ mdelay(1000);
+ resend = true;
+ }
+ } while (resend);
+
+ return 0;
+}
+
+static int set_real_num_queues(struct net_device *netdev)
+{
+ struct ibmvnic_adapter *adapter = netdev_priv(netdev);
+ int rc;
+
+ rc = netif_set_real_num_tx_queues(netdev, adapter->req_tx_queues);
+ if (rc) {
+ netdev_err(netdev, "failed to set the number of tx queues\n");
+ return rc;
+ }
+
+ rc = netif_set_real_num_rx_queues(netdev, adapter->req_rx_queues);
+ if (rc)
+ netdev_err(netdev, "failed to set the number of rx queues\n");
+
+ return rc;
+}
+
+static int ibmvnic_open(struct net_device *netdev)
+{
+ struct ibmvnic_adapter *adapter = netdev_priv(netdev);
+ struct device *dev = &adapter->vdev->dev;
+ int rc = 0;
+ int i;
+
+ if (adapter->is_closed) {
+ rc = ibmvnic_init(adapter);
+ if (rc)
+ return rc;
+ }
+
+ rc = ibmvnic_login(netdev);
+ if (rc)
+ return rc;
+
+ rc = set_real_num_queues(netdev);
+ if (rc)
+ return rc;
+
+ rc = init_sub_crq_irqs(adapter);
+ if (rc) {
+ dev_err(dev, "failed to initialize sub crq irqs\n");
+ return -1;
+ }
+
+ rc = init_stats_token(adapter);
+ if (rc)
+ return rc;
+
+ adapter->map_id = 1;
+ adapter->napi = kcalloc(adapter->req_rx_queues,
+ sizeof(struct napi_struct), GFP_KERNEL);
+ if (!adapter->napi)
+ goto ibmvnic_open_fail;
+ for (i = 0; i < adapter->req_rx_queues; i++) {
+ netif_napi_add(netdev, &adapter->napi[i], ibmvnic_poll,
+ NAPI_POLL_WEIGHT);
+ napi_enable(&adapter->napi[i]);
+ }
+
+ send_map_query(adapter);
+
+ rc = init_rx_pools(netdev);
+ if (rc)
+ goto ibmvnic_open_fail;
+
+ rc = init_tx_pools(netdev);
+ if (rc)
+ goto ibmvnic_open_fail;
+
replenish_pools(adapter);
/* We're ready to receive frames, enable the sub-crq interrupts and
@@ -466,106 +687,63 @@
for (i = 0; i < adapter->req_tx_queues; i++)
enable_scrq_irq(adapter, adapter->tx_scrq[i]);
- memset(&crq, 0, sizeof(crq));
- crq.logical_link_state.first = IBMVNIC_CRQ_CMD;
- crq.logical_link_state.cmd = LOGICAL_LINK_STATE;
- crq.logical_link_state.link_state = IBMVNIC_LOGICAL_LNK_UP;
- ibmvnic_send_crq(adapter, &crq);
+ rc = set_link_state(adapter, IBMVNIC_LOGICAL_LNK_UP);
+ if (rc)
+ goto ibmvnic_open_fail;
netif_tx_start_all_queues(netdev);
+ adapter->is_closed = false;
return 0;
-bounce_map_failed:
- kfree(adapter->bounce_buffer);
-bounce_alloc_failed:
- i = tx_subcrqs - 1;
- kfree(adapter->tx_pool[i].free_map);
-tx_fm_alloc_failed:
- free_long_term_buff(adapter, &adapter->tx_pool[i].long_term_buff);
-tx_ltb_alloc_failed:
- kfree(adapter->tx_pool[i].tx_buff);
-tx_pool_alloc_failed:
- for (j = 0; j < i; j++) {
- kfree(adapter->tx_pool[j].tx_buff);
- free_long_term_buff(adapter,
- &adapter->tx_pool[j].long_term_buff);
- kfree(adapter->tx_pool[j].free_map);
- }
- kfree(adapter->tx_pool);
- adapter->tx_pool = NULL;
-tx_pool_arr_alloc_failed:
- i = rxadd_subcrqs;
-rx_pool_alloc_failed:
- for (j = 0; j < i; j++) {
- free_rx_pool(adapter, &adapter->rx_pool[j]);
- free_long_term_buff(adapter,
- &adapter->rx_pool[j].long_term_buff);
- }
- kfree(adapter->rx_pool);
- adapter->rx_pool = NULL;
-rx_pool_arr_alloc_failed:
+ibmvnic_open_fail:
for (i = 0; i < adapter->req_rx_queues; i++)
napi_disable(&adapter->napi[i]);
-alloc_napi_failed:
+ release_resources(adapter);
return -ENOMEM;
}
+static void disable_sub_crqs(struct ibmvnic_adapter *adapter)
+{
+ int i;
+
+ if (adapter->tx_scrq) {
+ for (i = 0; i < adapter->req_tx_queues; i++)
+ if (adapter->tx_scrq[i])
+ disable_irq(adapter->tx_scrq[i]->irq);
+ }
+
+ if (adapter->rx_scrq) {
+ for (i = 0; i < adapter->req_rx_queues; i++)
+ if (adapter->rx_scrq[i])
+ disable_irq(adapter->rx_scrq[i]->irq);
+ }
+}
+
static int ibmvnic_close(struct net_device *netdev)
{
struct ibmvnic_adapter *adapter = netdev_priv(netdev);
- struct device *dev = &adapter->vdev->dev;
- union ibmvnic_crq crq;
+ int rc = 0;
int i;
adapter->closing = true;
+ disable_sub_crqs(adapter);
- for (i = 0; i < adapter->req_rx_queues; i++)
- napi_disable(&adapter->napi[i]);
+ if (adapter->napi) {
+ for (i = 0; i < adapter->req_rx_queues; i++)
+ napi_disable(&adapter->napi[i]);
+ }
if (!adapter->failover)
netif_tx_stop_all_queues(netdev);
- if (adapter->bounce_buffer) {
- if (!dma_mapping_error(dev, adapter->bounce_buffer_dma)) {
- dma_unmap_single(&adapter->vdev->dev,
- adapter->bounce_buffer_dma,
- adapter->bounce_buffer_size,
- DMA_BIDIRECTIONAL);
- adapter->bounce_buffer_dma = DMA_ERROR_CODE;
- }
- kfree(adapter->bounce_buffer);
- adapter->bounce_buffer = NULL;
- }
+ rc = set_link_state(adapter, IBMVNIC_LOGICAL_LNK_DN);
- memset(&crq, 0, sizeof(crq));
- crq.logical_link_state.first = IBMVNIC_CRQ_CMD;
- crq.logical_link_state.cmd = LOGICAL_LINK_STATE;
- crq.logical_link_state.link_state = IBMVNIC_LOGICAL_LNK_DN;
- ibmvnic_send_crq(adapter, &crq);
+ release_resources(adapter);
- for (i = 0; i < be32_to_cpu(adapter->login_rsp_buf->num_txsubm_subcrqs);
- i++) {
- kfree(adapter->tx_pool[i].tx_buff);
- free_long_term_buff(adapter,
- &adapter->tx_pool[i].long_term_buff);
- kfree(adapter->tx_pool[i].free_map);
- }
- kfree(adapter->tx_pool);
- adapter->tx_pool = NULL;
-
- for (i = 0; i < be32_to_cpu(adapter->login_rsp_buf->num_rxadd_subcrqs);
- i++) {
- free_rx_pool(adapter, &adapter->rx_pool[i]);
- free_long_term_buff(adapter,
- &adapter->rx_pool[i].long_term_buff);
- }
- kfree(adapter->rx_pool);
- adapter->rx_pool = NULL;
-
+ adapter->is_closed = true;
adapter->closing = false;
-
- return 0;
+ return rc;
}
/**
@@ -714,7 +892,6 @@
unsigned int tx_bytes = 0;
dma_addr_t data_dma_addr;
struct netdev_queue *txq;
- bool used_bounce = false;
unsigned long lpar_rc;
union sub_crq tx_crq;
unsigned int offset;
@@ -731,9 +908,13 @@
be32_to_cpu(adapter->login_rsp_buf->
off_txsubm_subcrqs));
if (adapter->migrated) {
+ if (!netif_subqueue_stopped(netdev, skb))
+ netif_stop_subqueue(netdev, queue_num);
+ dev_kfree_skb_any(skb);
+
tx_send_failed++;
tx_dropped++;
- ret = NETDEV_TX_BUSY;
+ ret = NETDEV_TX_OK;
goto out;
}
@@ -755,7 +936,6 @@
tx_buff->index = index;
tx_buff->pool_index = queue_num;
tx_buff->last_frag = true;
- tx_buff->used_bounce = used_bounce;
memset(&tx_crq, 0, sizeof(tx_crq));
tx_crq.v1.first = IBMVNIC_CRQ_CMD;
@@ -800,11 +980,13 @@
sizeof(tx_buff->indir_arr),
DMA_TO_DEVICE);
if (dma_mapping_error(dev, tx_buff->indir_dma)) {
+ dev_kfree_skb_any(skb);
+ tx_buff->skb = NULL;
if (!firmware_has_feature(FW_FEATURE_CMO))
dev_err(dev, "tx: unable to map descriptor array\n");
tx_map_failed++;
tx_dropped++;
- ret = NETDEV_TX_BUSY;
+ ret = NETDEV_TX_OK;
goto out;
}
lpar_rc = send_subcrq_indirect(adapter, handle_array[queue_num],
@@ -823,15 +1005,20 @@
else
tx_pool->consumer_index--;
+ dev_kfree_skb_any(skb);
+ tx_buff->skb = NULL;
+
+ if (lpar_rc == H_CLOSED)
+ netif_stop_subqueue(netdev, queue_num);
+
tx_send_failed++;
tx_dropped++;
- ret = NETDEV_TX_BUSY;
+ ret = NETDEV_TX_OK;
goto out;
}
- atomic_inc(&tx_scrq->used);
-
- if (atomic_read(&tx_scrq->used) >= adapter->req_tx_entries_per_subcrq) {
+ if (atomic_inc_return(&tx_scrq->used)
+ >= adapter->req_tx_entries_per_subcrq) {
netdev_info(netdev, "Stopping queue %d\n", queue_num);
netif_stop_subqueue(netdev, queue_num);
}
@@ -975,7 +1162,15 @@
skb = rx_buff->skb;
skb_copy_to_linear_data(skb, rx_buff->data + offset,
length);
- skb->vlan_tci = be16_to_cpu(next->rx_comp.vlan_tci);
+
+ /* VLAN Header has been stripped by the system firmware and
+ * needs to be inserted by the driver
+ */
+ if (adapter->rx_vlan_header_insertion &&
+ (flags & IBMVNIC_VLAN_STRIPPED))
+ __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q),
+ ntohs(next->rx_comp.vlan_tci));
+
/* free the entry */
next->rx_comp.first = 0;
remove_buff_from_pool(adapter, rx_buff);
@@ -1176,6 +1371,12 @@
scrq->crq_num);
} while (rc == H_BUSY || H_IS_LONG_BUSY(rc));
+ if (rc) {
+ netdev_err(adapter->netdev,
+ "Failed to release sub-CRQ %16lx, rc = %ld\n",
+ scrq->crq_num, rc);
+ }
+
dma_unmap_single(dev, scrq->msg_token, 4 * PAGE_SIZE,
DMA_BIDIRECTIONAL);
free_pages((unsigned long)scrq->msgs, 2);
@@ -1189,12 +1390,12 @@
struct ibmvnic_sub_crq_queue *scrq;
int rc;
- scrq = kmalloc(sizeof(*scrq), GFP_ATOMIC);
+ scrq = kzalloc(sizeof(*scrq), GFP_KERNEL);
if (!scrq)
return NULL;
- scrq->msgs = (union sub_crq *)__get_free_pages(GFP_ATOMIC, 2);
- memset(scrq->msgs, 0, 4 * PAGE_SIZE);
+ scrq->msgs =
+ (union sub_crq *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, 2);
if (!scrq->msgs) {
dev_warn(dev, "Couldn't allocate crq queue messages page\n");
goto zero_page_failed;
@@ -1222,9 +1423,6 @@
scrq->adapter = adapter;
scrq->size = 4 * PAGE_SIZE / sizeof(*scrq->msgs);
- scrq->cur = 0;
- atomic_set(&scrq->used, 0);
- scrq->rx_skb_top = NULL;
spin_lock_init(&scrq->lock);
netdev_dbg(adapter->netdev,
@@ -1249,53 +1447,44 @@
int i;
if (adapter->tx_scrq) {
- for (i = 0; i < adapter->req_tx_queues; i++)
- if (adapter->tx_scrq[i]) {
+ for (i = 0; i < adapter->req_tx_queues; i++) {
+ if (!adapter->tx_scrq[i])
+ continue;
+
+ if (adapter->tx_scrq[i]->irq) {
free_irq(adapter->tx_scrq[i]->irq,
adapter->tx_scrq[i]);
irq_dispose_mapping(adapter->tx_scrq[i]->irq);
- release_sub_crq_queue(adapter,
- adapter->tx_scrq[i]);
+ adapter->tx_scrq[i]->irq = 0;
}
+
+ release_sub_crq_queue(adapter, adapter->tx_scrq[i]);
+ }
+
kfree(adapter->tx_scrq);
adapter->tx_scrq = NULL;
}
if (adapter->rx_scrq) {
- for (i = 0; i < adapter->req_rx_queues; i++)
- if (adapter->rx_scrq[i]) {
+ for (i = 0; i < adapter->req_rx_queues; i++) {
+ if (!adapter->rx_scrq[i])
+ continue;
+
+ if (adapter->rx_scrq[i]->irq) {
free_irq(adapter->rx_scrq[i]->irq,
adapter->rx_scrq[i]);
irq_dispose_mapping(adapter->rx_scrq[i]->irq);
- release_sub_crq_queue(adapter,
- adapter->rx_scrq[i]);
+ adapter->rx_scrq[i]->irq = 0;
}
+
+ release_sub_crq_queue(adapter, adapter->rx_scrq[i]);
+ }
+
kfree(adapter->rx_scrq);
adapter->rx_scrq = NULL;
}
}
-static void release_sub_crqs_no_irqs(struct ibmvnic_adapter *adapter)
-{
- int i;
-
- if (adapter->tx_scrq) {
- for (i = 0; i < adapter->req_tx_queues; i++)
- if (adapter->tx_scrq[i])
- release_sub_crq_queue(adapter,
- adapter->tx_scrq[i]);
- adapter->tx_scrq = NULL;
- }
-
- if (adapter->rx_scrq) {
- for (i = 0; i < adapter->req_rx_queues; i++)
- if (adapter->rx_scrq[i])
- release_sub_crq_queue(adapter,
- adapter->rx_scrq[i]);
- adapter->rx_scrq = NULL;
- }
-}
-
static int disable_scrq_irq(struct ibmvnic_adapter *adapter,
struct ibmvnic_sub_crq_queue *scrq)
{
@@ -1358,7 +1547,6 @@
continue;
txbuff->data_dma[j] = 0;
- txbuff->used_bounce = false;
}
/* if sub_crq was sent indirectly */
first = txbuff->indir_arr[0].generic.first;
@@ -1369,9 +1557,8 @@
}
if (txbuff->last_frag) {
- atomic_dec(&scrq->used);
-
- if (atomic_read(&scrq->used) <=
+ if (atomic_sub_return(next->tx_comp.num_comps,
+ &scrq->used) <=
(adapter->req_tx_entries_per_subcrq / 2) &&
netif_subqueue_stopped(adapter->netdev,
txbuff->skb)) {
@@ -1487,52 +1674,24 @@
free_irq(adapter->tx_scrq[j]->irq, adapter->tx_scrq[j]);
irq_dispose_mapping(adapter->rx_scrq[j]->irq);
}
- release_sub_crqs_no_irqs(adapter);
+ release_sub_crqs(adapter);
return rc;
}
-static void init_sub_crqs(struct ibmvnic_adapter *adapter, int retry)
+static int init_sub_crqs(struct ibmvnic_adapter *adapter)
{
struct device *dev = &adapter->vdev->dev;
struct ibmvnic_sub_crq_queue **allqueues;
int registered_queues = 0;
- union ibmvnic_crq crq;
int total_queues;
int more = 0;
int i;
- if (!retry) {
- /* Sub-CRQ entries are 32 byte long */
- int entries_page = 4 * PAGE_SIZE / (sizeof(u64) * 4);
-
- if (adapter->min_tx_entries_per_subcrq > entries_page ||
- adapter->min_rx_add_entries_per_subcrq > entries_page) {
- dev_err(dev, "Fatal, invalid entries per sub-crq\n");
- goto allqueues_failed;
- }
-
- /* Get the minimum between the queried max and the entries
- * that fit in our PAGE_SIZE
- */
- adapter->req_tx_entries_per_subcrq =
- adapter->max_tx_entries_per_subcrq > entries_page ?
- entries_page : adapter->max_tx_entries_per_subcrq;
- adapter->req_rx_add_entries_per_subcrq =
- adapter->max_rx_add_entries_per_subcrq > entries_page ?
- entries_page : adapter->max_rx_add_entries_per_subcrq;
-
- adapter->req_tx_queues = adapter->opt_tx_comp_sub_queues;
- adapter->req_rx_queues = adapter->opt_rx_comp_queues;
- adapter->req_rx_add_queues = adapter->max_rx_add_queues;
-
- adapter->req_mtu = adapter->netdev->mtu + ETH_HLEN;
- }
-
total_queues = adapter->req_tx_queues + adapter->req_rx_queues;
- allqueues = kcalloc(total_queues, sizeof(*allqueues), GFP_ATOMIC);
+ allqueues = kcalloc(total_queues, sizeof(*allqueues), GFP_KERNEL);
if (!allqueues)
- goto allqueues_failed;
+ return -1;
for (i = 0; i < total_queues; i++) {
allqueues[i] = init_sub_crq_queue(adapter);
@@ -1570,7 +1729,7 @@
}
adapter->tx_scrq = kcalloc(adapter->req_tx_queues,
- sizeof(*adapter->tx_scrq), GFP_ATOMIC);
+ sizeof(*adapter->tx_scrq), GFP_KERNEL);
if (!adapter->tx_scrq)
goto tx_failed;
@@ -1580,7 +1739,7 @@
}
adapter->rx_scrq = kcalloc(adapter->req_rx_queues,
- sizeof(*adapter->rx_scrq), GFP_ATOMIC);
+ sizeof(*adapter->rx_scrq), GFP_KERNEL);
if (!adapter->rx_scrq)
goto rx_failed;
@@ -1589,6 +1748,51 @@
adapter->rx_scrq[i]->scrq_num = i;
}
+ kfree(allqueues);
+ return 0;
+
+rx_failed:
+ kfree(adapter->tx_scrq);
+ adapter->tx_scrq = NULL;
+tx_failed:
+ for (i = 0; i < registered_queues; i++)
+ release_sub_crq_queue(adapter, allqueues[i]);
+ kfree(allqueues);
+ return -1;
+}
+
+static void ibmvnic_send_req_caps(struct ibmvnic_adapter *adapter, int retry)
+{
+ struct device *dev = &adapter->vdev->dev;
+ union ibmvnic_crq crq;
+
+ if (!retry) {
+ /* Sub-CRQ entries are 32 byte long */
+ int entries_page = 4 * PAGE_SIZE / (sizeof(u64) * 4);
+
+ if (adapter->min_tx_entries_per_subcrq > entries_page ||
+ adapter->min_rx_add_entries_per_subcrq > entries_page) {
+ dev_err(dev, "Fatal, invalid entries per sub-crq\n");
+ return;
+ }
+
+ /* Get the minimum between the queried max and the entries
+ * that fit in our PAGE_SIZE
+ */
+ adapter->req_tx_entries_per_subcrq =
+ adapter->max_tx_entries_per_subcrq > entries_page ?
+ entries_page : adapter->max_tx_entries_per_subcrq;
+ adapter->req_rx_add_entries_per_subcrq =
+ adapter->max_rx_add_entries_per_subcrq > entries_page ?
+ entries_page : adapter->max_rx_add_entries_per_subcrq;
+
+ adapter->req_tx_queues = adapter->opt_tx_comp_sub_queues;
+ adapter->req_rx_queues = adapter->opt_rx_comp_queues;
+ adapter->req_rx_add_queues = adapter->max_rx_add_queues;
+
+ adapter->req_mtu = adapter->netdev->mtu + ETH_HLEN;
+ }
+
memset(&crq, 0, sizeof(crq));
crq.request_capability.first = IBMVNIC_CRQ_CMD;
crq.request_capability.cmd = REQUEST_CAPABILITY;
@@ -1642,20 +1846,6 @@
atomic_inc(&adapter->running_cap_crqs);
ibmvnic_send_crq(adapter, &crq);
}
-
- kfree(allqueues);
-
- return;
-
-rx_failed:
- kfree(adapter->tx_scrq);
- adapter->tx_scrq = NULL;
-tx_failed:
- for (i = 0; i < registered_queues; i++)
- release_sub_crq_queue(adapter, allqueues[i]);
- kfree(allqueues);
-allqueues_failed:
- ibmvnic_remove(adapter->vdev);
}
static int pending_scrq(struct ibmvnic_adapter *adapter,
@@ -1829,13 +2019,11 @@
{
struct ibmvnic_login_rsp_buffer *login_rsp_buffer;
struct ibmvnic_login_buffer *login_buffer;
- struct ibmvnic_inflight_cmd *inflight_cmd;
struct device *dev = &adapter->vdev->dev;
dma_addr_t rsp_buffer_token;
dma_addr_t buffer_token;
size_t rsp_buffer_size;
union ibmvnic_crq crq;
- unsigned long flags;
size_t buffer_size;
__be64 *tx_list_p;
__be64 *rx_list_p;
@@ -1872,11 +2060,7 @@
dev_err(dev, "Couldn't map login rsp buffer\n");
goto buf_rsp_map_failed;
}
- inflight_cmd = kmalloc(sizeof(*inflight_cmd), GFP_ATOMIC);
- if (!inflight_cmd) {
- dev_err(dev, "Couldn't allocate inflight_cmd\n");
- goto inflight_alloc_failed;
- }
+
adapter->login_buf = login_buffer;
adapter->login_buf_token = buffer_token;
adapter->login_buf_sz = buffer_size;
@@ -1927,20 +2111,10 @@
crq.login.cmd = LOGIN;
crq.login.ioba = cpu_to_be32(buffer_token);
crq.login.len = cpu_to_be32(buffer_size);
-
- memcpy(&inflight_cmd->crq, &crq, sizeof(crq));
-
- spin_lock_irqsave(&adapter->inflight_lock, flags);
- list_add_tail(&inflight_cmd->list, &adapter->inflight);
- spin_unlock_irqrestore(&adapter->inflight_lock, flags);
-
ibmvnic_send_crq(adapter, &crq);
return;
-inflight_alloc_failed:
- dma_unmap_single(dev, rsp_buffer_token, rsp_buffer_size,
- DMA_FROM_DEVICE);
buf_rsp_map_failed:
kfree(login_rsp_buffer);
buf_rsp_alloc_failed:
@@ -2064,6 +2238,10 @@
atomic_inc(&adapter->running_cap_crqs);
ibmvnic_send_crq(adapter, &crq);
+ crq.query_capability.capability = cpu_to_be16(RX_VLAN_HEADER_INSERTION);
+ atomic_inc(&adapter->running_cap_crqs);
+ ibmvnic_send_crq(adapter, &crq);
+
crq.query_capability.capability = cpu_to_be16(MAX_TX_SG_ENTRIES);
atomic_inc(&adapter->running_cap_crqs);
ibmvnic_send_crq(adapter, &crq);
@@ -2242,77 +2420,22 @@
kfree(error_buff);
}
-static void handle_dump_size_rsp(union ibmvnic_crq *crq,
- struct ibmvnic_adapter *adapter)
+static void request_error_information(struct ibmvnic_adapter *adapter,
+ union ibmvnic_crq *err_crq)
{
- int len = be32_to_cpu(crq->request_dump_size_rsp.len);
- struct ibmvnic_inflight_cmd *inflight_cmd;
struct device *dev = &adapter->vdev->dev;
- union ibmvnic_crq newcrq;
- unsigned long flags;
-
- /* allocate and map buffer */
- adapter->dump_data = kmalloc(len, GFP_KERNEL);
- if (!adapter->dump_data) {
- complete(&adapter->fw_done);
- return;
- }
-
- adapter->dump_data_token = dma_map_single(dev, adapter->dump_data, len,
- DMA_FROM_DEVICE);
-
- if (dma_mapping_error(dev, adapter->dump_data_token)) {
- if (!firmware_has_feature(FW_FEATURE_CMO))
- dev_err(dev, "Couldn't map dump data\n");
- kfree(adapter->dump_data);
- complete(&adapter->fw_done);
- return;
- }
-
- inflight_cmd = kmalloc(sizeof(*inflight_cmd), GFP_ATOMIC);
- if (!inflight_cmd) {
- dma_unmap_single(dev, adapter->dump_data_token, len,
- DMA_FROM_DEVICE);
- kfree(adapter->dump_data);
- complete(&adapter->fw_done);
- return;
- }
-
- memset(&newcrq, 0, sizeof(newcrq));
- newcrq.request_dump.first = IBMVNIC_CRQ_CMD;
- newcrq.request_dump.cmd = REQUEST_DUMP;
- newcrq.request_dump.ioba = cpu_to_be32(adapter->dump_data_token);
- newcrq.request_dump.len = cpu_to_be32(adapter->dump_data_size);
-
- memcpy(&inflight_cmd->crq, &newcrq, sizeof(newcrq));
-
- spin_lock_irqsave(&adapter->inflight_lock, flags);
- list_add_tail(&inflight_cmd->list, &adapter->inflight);
- spin_unlock_irqrestore(&adapter->inflight_lock, flags);
-
- ibmvnic_send_crq(adapter, &newcrq);
-}
-
-static void handle_error_indication(union ibmvnic_crq *crq,
- struct ibmvnic_adapter *adapter)
-{
- int detail_len = be32_to_cpu(crq->error_indication.detail_error_sz);
- struct ibmvnic_inflight_cmd *inflight_cmd;
- struct device *dev = &adapter->vdev->dev;
+ struct net_device *netdev = adapter->netdev;
struct ibmvnic_error_buff *error_buff;
- union ibmvnic_crq new_crq;
+ unsigned long timeout = msecs_to_jiffies(30000);
+ union ibmvnic_crq crq;
unsigned long flags;
-
- dev_err(dev, "Firmware reports %serror id %x, cause %d\n",
- crq->error_indication.
- flags & IBMVNIC_FATAL_ERROR ? "FATAL " : "",
- be32_to_cpu(crq->error_indication.error_id),
- be16_to_cpu(crq->error_indication.error_cause));
+ int rc, detail_len;
error_buff = kmalloc(sizeof(*error_buff), GFP_ATOMIC);
if (!error_buff)
return;
+ detail_len = be32_to_cpu(err_crq->error_indication.detail_error_sz);
error_buff->buff = kmalloc(detail_len, GFP_ATOMIC);
if (!error_buff->buff) {
kfree(error_buff);
@@ -2322,43 +2445,61 @@
error_buff->dma = dma_map_single(dev, error_buff->buff, detail_len,
DMA_FROM_DEVICE);
if (dma_mapping_error(dev, error_buff->dma)) {
- if (!firmware_has_feature(FW_FEATURE_CMO))
- dev_err(dev, "Couldn't map error buffer\n");
- kfree(error_buff->buff);
- kfree(error_buff);
- return;
- }
-
- inflight_cmd = kmalloc(sizeof(*inflight_cmd), GFP_ATOMIC);
- if (!inflight_cmd) {
- dma_unmap_single(dev, error_buff->dma, detail_len,
- DMA_FROM_DEVICE);
+ netdev_err(netdev, "Couldn't map error buffer\n");
kfree(error_buff->buff);
kfree(error_buff);
return;
}
error_buff->len = detail_len;
- error_buff->error_id = crq->error_indication.error_id;
+ error_buff->error_id = err_crq->error_indication.error_id;
spin_lock_irqsave(&adapter->error_list_lock, flags);
list_add_tail(&error_buff->list, &adapter->errors);
spin_unlock_irqrestore(&adapter->error_list_lock, flags);
- memset(&new_crq, 0, sizeof(new_crq));
- new_crq.request_error_info.first = IBMVNIC_CRQ_CMD;
- new_crq.request_error_info.cmd = REQUEST_ERROR_INFO;
- new_crq.request_error_info.ioba = cpu_to_be32(error_buff->dma);
- new_crq.request_error_info.len = cpu_to_be32(detail_len);
- new_crq.request_error_info.error_id = crq->error_indication.error_id;
+ memset(&crq, 0, sizeof(crq));
+ crq.request_error_info.first = IBMVNIC_CRQ_CMD;
+ crq.request_error_info.cmd = REQUEST_ERROR_INFO;
+ crq.request_error_info.ioba = cpu_to_be32(error_buff->dma);
+ crq.request_error_info.len = cpu_to_be32(detail_len);
+ crq.request_error_info.error_id = err_crq->error_indication.error_id;
- memcpy(&inflight_cmd->crq, &crq, sizeof(crq));
+ rc = ibmvnic_send_crq(adapter, &crq);
+ if (rc) {
+ netdev_err(netdev, "failed to request error information\n");
+ goto err_info_fail;
+ }
- spin_lock_irqsave(&adapter->inflight_lock, flags);
- list_add_tail(&inflight_cmd->list, &adapter->inflight);
- spin_unlock_irqrestore(&adapter->inflight_lock, flags);
+ if (!wait_for_completion_timeout(&adapter->init_done, timeout)) {
+ netdev_err(netdev, "timeout waiting for error information\n");
+ goto err_info_fail;
+ }
- ibmvnic_send_crq(adapter, &new_crq);
+ return;
+
+err_info_fail:
+ spin_lock_irqsave(&adapter->error_list_lock, flags);
+ list_del(&error_buff->list);
+ spin_unlock_irqrestore(&adapter->error_list_lock, flags);
+
+ kfree(error_buff->buff);
+ kfree(error_buff);
+}
+
+static void handle_error_indication(union ibmvnic_crq *crq,
+ struct ibmvnic_adapter *adapter)
+{
+ struct device *dev = &adapter->vdev->dev;
+
+ dev_err(dev, "Firmware reports %serror id %x, cause %d\n",
+ crq->error_indication.flags
+ & IBMVNIC_FATAL_ERROR ? "FATAL " : "",
+ be32_to_cpu(crq->error_indication.error_id),
+ be16_to_cpu(crq->error_indication.error_cause));
+
+ if (be32_to_cpu(crq->error_indication.error_id))
+ request_error_information(adapter, crq);
}
static void handle_change_mac_rsp(union ibmvnic_crq *crq,
@@ -2428,9 +2569,9 @@
*req_value,
(long int)be64_to_cpu(crq->request_capability_rsp.
number), name);
- release_sub_crqs_no_irqs(adapter);
+ release_sub_crqs(adapter);
*req_value = be64_to_cpu(crq->request_capability_rsp.number);
- init_sub_crqs(adapter, 1);
+ ibmvnic_send_req_caps(adapter, 1);
return;
default:
dev_err(dev, "Error %d in request cap rsp\n",
@@ -2473,7 +2614,6 @@
struct device *dev = &adapter->vdev->dev;
struct ibmvnic_login_rsp_buffer *login_rsp = adapter->login_rsp_buf;
struct ibmvnic_login_buffer *login = adapter->login_buf;
- union ibmvnic_crq crq;
int i;
dma_unmap_single(dev, adapter->login_buf_token, adapter->login_buf_sz,
@@ -2508,11 +2648,6 @@
}
complete(&adapter->init_done);
- memset(&crq, 0, sizeof(crq));
- crq.request_ras_comp_num.first = IBMVNIC_CRQ_CMD;
- crq.request_ras_comp_num.cmd = REQUEST_RAS_COMP_NUM;
- ibmvnic_send_crq(adapter, &crq);
-
return 0;
}
@@ -2687,6 +2822,12 @@
netdev_dbg(netdev, "vlan_header_insertion = %lld\n",
adapter->vlan_header_insertion);
break;
+ case RX_VLAN_HEADER_INSERTION:
+ adapter->rx_vlan_header_insertion =
+ be64_to_cpu(crq->query_capability.number);
+ netdev_dbg(netdev, "rx_vlan_header_insertion = %lld\n",
+ adapter->rx_vlan_header_insertion);
+ break;
case MAX_TX_SG_ENTRIES:
adapter->max_tx_sg_entries =
be64_to_cpu(crq->query_capability.number);
@@ -2743,526 +2884,10 @@
out:
if (atomic_read(&adapter->running_cap_crqs) == 0) {
adapter->wait_capability = false;
- init_sub_crqs(adapter, 0);
- /* We're done querying the capabilities, initialize sub-crqs */
+ ibmvnic_send_req_caps(adapter, 0);
}
}
-static void handle_control_ras_rsp(union ibmvnic_crq *crq,
- struct ibmvnic_adapter *adapter)
-{
- u8 correlator = crq->control_ras_rsp.correlator;
- struct device *dev = &adapter->vdev->dev;
- bool found = false;
- int i;
-
- if (crq->control_ras_rsp.rc.code) {
- dev_warn(dev, "Control ras failed rc=%d\n",
- crq->control_ras_rsp.rc.code);
- return;
- }
-
- for (i = 0; i < adapter->ras_comp_num; i++) {
- if (adapter->ras_comps[i].correlator == correlator) {
- found = true;
- break;
- }
- }
-
- if (!found) {
- dev_warn(dev, "Correlator not found on control_ras_rsp\n");
- return;
- }
-
- switch (crq->control_ras_rsp.op) {
- case IBMVNIC_TRACE_LEVEL:
- adapter->ras_comps[i].trace_level = crq->control_ras.level;
- break;
- case IBMVNIC_ERROR_LEVEL:
- adapter->ras_comps[i].error_check_level =
- crq->control_ras.level;
- break;
- case IBMVNIC_TRACE_PAUSE:
- adapter->ras_comp_int[i].paused = 1;
- break;
- case IBMVNIC_TRACE_RESUME:
- adapter->ras_comp_int[i].paused = 0;
- break;
- case IBMVNIC_TRACE_ON:
- adapter->ras_comps[i].trace_on = 1;
- break;
- case IBMVNIC_TRACE_OFF:
- adapter->ras_comps[i].trace_on = 0;
- break;
- case IBMVNIC_CHG_TRACE_BUFF_SZ:
- /* trace_buff_sz is 3 bytes, stuff it into an int */
- ((u8 *)(&adapter->ras_comps[i].trace_buff_size))[0] = 0;
- ((u8 *)(&adapter->ras_comps[i].trace_buff_size))[1] =
- crq->control_ras_rsp.trace_buff_sz[0];
- ((u8 *)(&adapter->ras_comps[i].trace_buff_size))[2] =
- crq->control_ras_rsp.trace_buff_sz[1];
- ((u8 *)(&adapter->ras_comps[i].trace_buff_size))[3] =
- crq->control_ras_rsp.trace_buff_sz[2];
- break;
- default:
- dev_err(dev, "invalid op %d on control_ras_rsp",
- crq->control_ras_rsp.op);
- }
-}
-
-static ssize_t trace_read(struct file *file, char __user *user_buf, size_t len,
- loff_t *ppos)
-{
- struct ibmvnic_fw_comp_internal *ras_comp_int = file->private_data;
- struct ibmvnic_adapter *adapter = ras_comp_int->adapter;
- struct device *dev = &adapter->vdev->dev;
- struct ibmvnic_fw_trace_entry *trace;
- int num = ras_comp_int->num;
- union ibmvnic_crq crq;
- dma_addr_t trace_tok;
-
- if (*ppos >= be32_to_cpu(adapter->ras_comps[num].trace_buff_size))
- return 0;
-
- trace =
- dma_alloc_coherent(dev,
- be32_to_cpu(adapter->ras_comps[num].
- trace_buff_size), &trace_tok,
- GFP_KERNEL);
- if (!trace) {
- dev_err(dev, "Couldn't alloc trace buffer\n");
- return 0;
- }
-
- memset(&crq, 0, sizeof(crq));
- crq.collect_fw_trace.first = IBMVNIC_CRQ_CMD;
- crq.collect_fw_trace.cmd = COLLECT_FW_TRACE;
- crq.collect_fw_trace.correlator = adapter->ras_comps[num].correlator;
- crq.collect_fw_trace.ioba = cpu_to_be32(trace_tok);
- crq.collect_fw_trace.len = adapter->ras_comps[num].trace_buff_size;
-
- init_completion(&adapter->fw_done);
- ibmvnic_send_crq(adapter, &crq);
- wait_for_completion(&adapter->fw_done);
-
- if (*ppos + len > be32_to_cpu(adapter->ras_comps[num].trace_buff_size))
- len =
- be32_to_cpu(adapter->ras_comps[num].trace_buff_size) -
- *ppos;
-
- copy_to_user(user_buf, &((u8 *)trace)[*ppos], len);
-
- dma_free_coherent(dev,
- be32_to_cpu(adapter->ras_comps[num].trace_buff_size),
- trace, trace_tok);
- *ppos += len;
- return len;
-}
-
-static const struct file_operations trace_ops = {
- .owner = THIS_MODULE,
- .open = simple_open,
- .read = trace_read,
-};
-
-static ssize_t paused_read(struct file *file, char __user *user_buf, size_t len,
- loff_t *ppos)
-{
- struct ibmvnic_fw_comp_internal *ras_comp_int = file->private_data;
- struct ibmvnic_adapter *adapter = ras_comp_int->adapter;
- int num = ras_comp_int->num;
- char buff[5]; /* 1 or 0 plus \n and \0 */
- int size;
-
- size = sprintf(buff, "%d\n", adapter->ras_comp_int[num].paused);
-
- if (*ppos >= size)
- return 0;
-
- copy_to_user(user_buf, buff, size);
- *ppos += size;
- return size;
-}
-
-static ssize_t paused_write(struct file *file, const char __user *user_buf,
- size_t len, loff_t *ppos)
-{
- struct ibmvnic_fw_comp_internal *ras_comp_int = file->private_data;
- struct ibmvnic_adapter *adapter = ras_comp_int->adapter;
- int num = ras_comp_int->num;
- union ibmvnic_crq crq;
- unsigned long val;
- char buff[9]; /* decimal max int plus \n and \0 */
-
- copy_from_user(buff, user_buf, sizeof(buff));
- val = kstrtoul(buff, 10, NULL);
-
- adapter->ras_comp_int[num].paused = val ? 1 : 0;
-
- memset(&crq, 0, sizeof(crq));
- crq.control_ras.first = IBMVNIC_CRQ_CMD;
- crq.control_ras.cmd = CONTROL_RAS;
- crq.control_ras.correlator = adapter->ras_comps[num].correlator;
- crq.control_ras.op = val ? IBMVNIC_TRACE_PAUSE : IBMVNIC_TRACE_RESUME;
- ibmvnic_send_crq(adapter, &crq);
-
- return len;
-}
-
-static const struct file_operations paused_ops = {
- .owner = THIS_MODULE,
- .open = simple_open,
- .read = paused_read,
- .write = paused_write,
-};
-
-static ssize_t tracing_read(struct file *file, char __user *user_buf,
- size_t len, loff_t *ppos)
-{
- struct ibmvnic_fw_comp_internal *ras_comp_int = file->private_data;
- struct ibmvnic_adapter *adapter = ras_comp_int->adapter;
- int num = ras_comp_int->num;
- char buff[5]; /* 1 or 0 plus \n and \0 */
- int size;
-
- size = sprintf(buff, "%d\n", adapter->ras_comps[num].trace_on);
-
- if (*ppos >= size)
- return 0;
-
- copy_to_user(user_buf, buff, size);
- *ppos += size;
- return size;
-}
-
-static ssize_t tracing_write(struct file *file, const char __user *user_buf,
- size_t len, loff_t *ppos)
-{
- struct ibmvnic_fw_comp_internal *ras_comp_int = file->private_data;
- struct ibmvnic_adapter *adapter = ras_comp_int->adapter;
- int num = ras_comp_int->num;
- union ibmvnic_crq crq;
- unsigned long val;
- char buff[9]; /* decimal max int plus \n and \0 */
-
- copy_from_user(buff, user_buf, sizeof(buff));
- val = kstrtoul(buff, 10, NULL);
-
- memset(&crq, 0, sizeof(crq));
- crq.control_ras.first = IBMVNIC_CRQ_CMD;
- crq.control_ras.cmd = CONTROL_RAS;
- crq.control_ras.correlator = adapter->ras_comps[num].correlator;
- crq.control_ras.op = val ? IBMVNIC_TRACE_ON : IBMVNIC_TRACE_OFF;
-
- return len;
-}
-
-static const struct file_operations tracing_ops = {
- .owner = THIS_MODULE,
- .open = simple_open,
- .read = tracing_read,
- .write = tracing_write,
-};
-
-static ssize_t error_level_read(struct file *file, char __user *user_buf,
- size_t len, loff_t *ppos)
-{
- struct ibmvnic_fw_comp_internal *ras_comp_int = file->private_data;
- struct ibmvnic_adapter *adapter = ras_comp_int->adapter;
- int num = ras_comp_int->num;
- char buff[5]; /* decimal max char plus \n and \0 */
- int size;
-
- size = sprintf(buff, "%d\n", adapter->ras_comps[num].error_check_level);
-
- if (*ppos >= size)
- return 0;
-
- copy_to_user(user_buf, buff, size);
- *ppos += size;
- return size;
-}
-
-static ssize_t error_level_write(struct file *file, const char __user *user_buf,
- size_t len, loff_t *ppos)
-{
- struct ibmvnic_fw_comp_internal *ras_comp_int = file->private_data;
- struct ibmvnic_adapter *adapter = ras_comp_int->adapter;
- int num = ras_comp_int->num;
- union ibmvnic_crq crq;
- unsigned long val;
- char buff[9]; /* decimal max int plus \n and \0 */
-
- copy_from_user(buff, user_buf, sizeof(buff));
- val = kstrtoul(buff, 10, NULL);
-
- if (val > 9)
- val = 9;
-
- memset(&crq, 0, sizeof(crq));
- crq.control_ras.first = IBMVNIC_CRQ_CMD;
- crq.control_ras.cmd = CONTROL_RAS;
- crq.control_ras.correlator = adapter->ras_comps[num].correlator;
- crq.control_ras.op = IBMVNIC_ERROR_LEVEL;
- crq.control_ras.level = val;
- ibmvnic_send_crq(adapter, &crq);
-
- return len;
-}
-
-static const struct file_operations error_level_ops = {
- .owner = THIS_MODULE,
- .open = simple_open,
- .read = error_level_read,
- .write = error_level_write,
-};
-
-static ssize_t trace_level_read(struct file *file, char __user *user_buf,
- size_t len, loff_t *ppos)
-{
- struct ibmvnic_fw_comp_internal *ras_comp_int = file->private_data;
- struct ibmvnic_adapter *adapter = ras_comp_int->adapter;
- int num = ras_comp_int->num;
- char buff[5]; /* decimal max char plus \n and \0 */
- int size;
-
- size = sprintf(buff, "%d\n", adapter->ras_comps[num].trace_level);
- if (*ppos >= size)
- return 0;
-
- copy_to_user(user_buf, buff, size);
- *ppos += size;
- return size;
-}
-
-static ssize_t trace_level_write(struct file *file, const char __user *user_buf,
- size_t len, loff_t *ppos)
-{
- struct ibmvnic_fw_comp_internal *ras_comp_int = file->private_data;
- struct ibmvnic_adapter *adapter = ras_comp_int->adapter;
- union ibmvnic_crq crq;
- unsigned long val;
- char buff[9]; /* decimal max int plus \n and \0 */
-
- copy_from_user(buff, user_buf, sizeof(buff));
- val = kstrtoul(buff, 10, NULL);
- if (val > 9)
- val = 9;
-
- memset(&crq, 0, sizeof(crq));
- crq.control_ras.first = IBMVNIC_CRQ_CMD;
- crq.control_ras.cmd = CONTROL_RAS;
- crq.control_ras.correlator =
- adapter->ras_comps[ras_comp_int->num].correlator;
- crq.control_ras.op = IBMVNIC_TRACE_LEVEL;
- crq.control_ras.level = val;
- ibmvnic_send_crq(adapter, &crq);
-
- return len;
-}
-
-static const struct file_operations trace_level_ops = {
- .owner = THIS_MODULE,
- .open = simple_open,
- .read = trace_level_read,
- .write = trace_level_write,
-};
-
-static ssize_t trace_buff_size_read(struct file *file, char __user *user_buf,
- size_t len, loff_t *ppos)
-{
- struct ibmvnic_fw_comp_internal *ras_comp_int = file->private_data;
- struct ibmvnic_adapter *adapter = ras_comp_int->adapter;
- int num = ras_comp_int->num;
- char buff[9]; /* decimal max int plus \n and \0 */
- int size;
-
- size = sprintf(buff, "%d\n", adapter->ras_comps[num].trace_buff_size);
- if (*ppos >= size)
- return 0;
-
- copy_to_user(user_buf, buff, size);
- *ppos += size;
- return size;
-}
-
-static ssize_t trace_buff_size_write(struct file *file,
- const char __user *user_buf, size_t len,
- loff_t *ppos)
-{
- struct ibmvnic_fw_comp_internal *ras_comp_int = file->private_data;
- struct ibmvnic_adapter *adapter = ras_comp_int->adapter;
- union ibmvnic_crq crq;
- unsigned long val;
- char buff[9]; /* decimal max int plus \n and \0 */
-
- copy_from_user(buff, user_buf, sizeof(buff));
- val = kstrtoul(buff, 10, NULL);
-
- memset(&crq, 0, sizeof(crq));
- crq.control_ras.first = IBMVNIC_CRQ_CMD;
- crq.control_ras.cmd = CONTROL_RAS;
- crq.control_ras.correlator =
- adapter->ras_comps[ras_comp_int->num].correlator;
- crq.control_ras.op = IBMVNIC_CHG_TRACE_BUFF_SZ;
- /* trace_buff_sz is 3 bytes, stuff an int into it */
- crq.control_ras.trace_buff_sz[0] = ((u8 *)(&val))[5];
- crq.control_ras.trace_buff_sz[1] = ((u8 *)(&val))[6];
- crq.control_ras.trace_buff_sz[2] = ((u8 *)(&val))[7];
- ibmvnic_send_crq(adapter, &crq);
-
- return len;
-}
-
-static const struct file_operations trace_size_ops = {
- .owner = THIS_MODULE,
- .open = simple_open,
- .read = trace_buff_size_read,
- .write = trace_buff_size_write,
-};
-
-static void handle_request_ras_comps_rsp(union ibmvnic_crq *crq,
- struct ibmvnic_adapter *adapter)
-{
- struct device *dev = &adapter->vdev->dev;
- struct dentry *dir_ent;
- struct dentry *ent;
- int i;
-
- debugfs_remove_recursive(adapter->ras_comps_ent);
-
- adapter->ras_comps_ent = debugfs_create_dir("ras_comps",
- adapter->debugfs_dir);
- if (!adapter->ras_comps_ent || IS_ERR(adapter->ras_comps_ent)) {
- dev_info(dev, "debugfs create ras_comps dir failed\n");
- return;
- }
-
- for (i = 0; i < adapter->ras_comp_num; i++) {
- dir_ent = debugfs_create_dir(adapter->ras_comps[i].name,
- adapter->ras_comps_ent);
- if (!dir_ent || IS_ERR(dir_ent)) {
- dev_info(dev, "debugfs create %s dir failed\n",
- adapter->ras_comps[i].name);
- continue;
- }
-
- adapter->ras_comp_int[i].adapter = adapter;
- adapter->ras_comp_int[i].num = i;
- adapter->ras_comp_int[i].desc_blob.data =
- &adapter->ras_comps[i].description;
- adapter->ras_comp_int[i].desc_blob.size =
- sizeof(adapter->ras_comps[i].description);
-
- /* Don't need to remember the dentry's because the debugfs dir
- * gets removed recursively
- */
- ent = debugfs_create_blob("description", S_IRUGO, dir_ent,
- &adapter->ras_comp_int[i].desc_blob);
- ent = debugfs_create_file("trace_buf_size", S_IRUGO | S_IWUSR,
- dir_ent, &adapter->ras_comp_int[i],
- &trace_size_ops);
- ent = debugfs_create_file("trace_level",
- S_IRUGO |
- (adapter->ras_comps[i].trace_level !=
- 0xFF ? S_IWUSR : 0),
- dir_ent, &adapter->ras_comp_int[i],
- &trace_level_ops);
- ent = debugfs_create_file("error_level",
- S_IRUGO |
- (adapter->
- ras_comps[i].error_check_level !=
- 0xFF ? S_IWUSR : 0),
- dir_ent, &adapter->ras_comp_int[i],
- &trace_level_ops);
- ent = debugfs_create_file("tracing", S_IRUGO | S_IWUSR,
- dir_ent, &adapter->ras_comp_int[i],
- &tracing_ops);
- ent = debugfs_create_file("paused", S_IRUGO | S_IWUSR,
- dir_ent, &adapter->ras_comp_int[i],
- &paused_ops);
- ent = debugfs_create_file("trace", S_IRUGO, dir_ent,
- &adapter->ras_comp_int[i],
- &trace_ops);
- }
-}
-
-static void handle_request_ras_comp_num_rsp(union ibmvnic_crq *crq,
- struct ibmvnic_adapter *adapter)
-{
- int len = adapter->ras_comp_num * sizeof(struct ibmvnic_fw_component);
- struct device *dev = &adapter->vdev->dev;
- union ibmvnic_crq newcrq;
-
- adapter->ras_comps = dma_alloc_coherent(dev, len,
- &adapter->ras_comps_tok,
- GFP_KERNEL);
- if (!adapter->ras_comps) {
- if (!firmware_has_feature(FW_FEATURE_CMO))
- dev_err(dev, "Couldn't alloc fw comps buffer\n");
- return;
- }
-
- adapter->ras_comp_int = kmalloc(adapter->ras_comp_num *
- sizeof(struct ibmvnic_fw_comp_internal),
- GFP_KERNEL);
- if (!adapter->ras_comp_int)
- dma_free_coherent(dev, len, adapter->ras_comps,
- adapter->ras_comps_tok);
-
- memset(&newcrq, 0, sizeof(newcrq));
- newcrq.request_ras_comps.first = IBMVNIC_CRQ_CMD;
- newcrq.request_ras_comps.cmd = REQUEST_RAS_COMPS;
- newcrq.request_ras_comps.ioba = cpu_to_be32(adapter->ras_comps_tok);
- newcrq.request_ras_comps.len = cpu_to_be32(len);
- ibmvnic_send_crq(adapter, &newcrq);
-}
-
-static void ibmvnic_free_inflight(struct ibmvnic_adapter *adapter)
-{
- struct ibmvnic_inflight_cmd *inflight_cmd, *tmp1;
- struct device *dev = &adapter->vdev->dev;
- struct ibmvnic_error_buff *error_buff, *tmp2;
- unsigned long flags;
- unsigned long flags2;
-
- spin_lock_irqsave(&adapter->inflight_lock, flags);
- list_for_each_entry_safe(inflight_cmd, tmp1, &adapter->inflight, list) {
- switch (inflight_cmd->crq.generic.cmd) {
- case LOGIN:
- dma_unmap_single(dev, adapter->login_buf_token,
- adapter->login_buf_sz,
- DMA_BIDIRECTIONAL);
- dma_unmap_single(dev, adapter->login_rsp_buf_token,
- adapter->login_rsp_buf_sz,
- DMA_BIDIRECTIONAL);
- kfree(adapter->login_rsp_buf);
- kfree(adapter->login_buf);
- break;
- case REQUEST_DUMP:
- complete(&adapter->fw_done);
- break;
- case REQUEST_ERROR_INFO:
- spin_lock_irqsave(&adapter->error_list_lock, flags2);
- list_for_each_entry_safe(error_buff, tmp2,
- &adapter->errors, list) {
- dma_unmap_single(dev, error_buff->dma,
- error_buff->len,
- DMA_FROM_DEVICE);
- kfree(error_buff->buff);
- list_del(&error_buff->list);
- kfree(error_buff);
- }
- spin_unlock_irqrestore(&adapter->error_list_lock,
- flags2);
- break;
- }
- list_del(&inflight_cmd->list);
- kfree(inflight_cmd);
- }
- spin_unlock_irqrestore(&adapter->inflight_lock, flags);
-}
-
static void ibmvnic_xport_event(struct work_struct *work)
{
struct ibmvnic_adapter *adapter = container_of(work,
@@ -3271,7 +2896,6 @@
struct device *dev = &adapter->vdev->dev;
long rc;
- ibmvnic_free_inflight(adapter);
release_sub_crqs(adapter);
if (adapter->migrated) {
rc = ibmvnic_reenable_crq_queue(adapter);
@@ -3290,11 +2914,12 @@
struct ibmvnic_generic_crq *gen_crq = &crq->generic;
struct net_device *netdev = adapter->netdev;
struct device *dev = &adapter->vdev->dev;
+ u64 *u64_crq = (u64 *)crq;
long rc;
netdev_dbg(netdev, "Handling CRQ: %016lx %016lx\n",
- ((unsigned long int *)crq)[0],
- ((unsigned long int *)crq)[1]);
+ (unsigned long int)cpu_to_be64(u64_crq[0]),
+ (unsigned long int)cpu_to_be64(u64_crq[1]));
switch (gen_crq->first) {
case IBMVNIC_CRQ_INIT_RSP:
switch (gen_crq->cmd) {
@@ -3374,9 +2999,14 @@
handle_login_rsp(crq, adapter);
break;
case LOGICAL_LINK_STATE_RSP:
- netdev_dbg(netdev, "Got Logical Link State Response\n");
+ netdev_dbg(netdev,
+ "Got Logical Link State Response, state: %d rc: %d\n",
+ crq->logical_link_state_rsp.link_state,
+ crq->logical_link_state_rsp.rc.code);
adapter->logical_link_state =
crq->logical_link_state_rsp.link_state;
+ adapter->init_done_rc = crq->logical_link_state_rsp.rc.code;
+ complete(&adapter->init_done);
break;
case LINK_STATE_INDICATION:
netdev_dbg(netdev, "Got Logical Link State Indication\n");
@@ -3401,14 +3031,6 @@
netdev_dbg(netdev, "Got Statistics Response\n");
complete(&adapter->stats_done);
break;
- case REQUEST_DUMP_SIZE_RSP:
- netdev_dbg(netdev, "Got Request Dump Size Response\n");
- handle_dump_size_rsp(crq, adapter);
- break;
- case REQUEST_DUMP_RSP:
- netdev_dbg(netdev, "Got Request Dump Response\n");
- complete(&adapter->fw_done);
- break;
case QUERY_IP_OFFLOAD_RSP:
netdev_dbg(netdev, "Got Query IP offload Response\n");
handle_query_ip_offload_rsp(adapter);
@@ -3421,26 +3043,7 @@
dma_unmap_single(dev, adapter->ip_offload_ctrl_tok,
sizeof(adapter->ip_offload_ctrl),
DMA_TO_DEVICE);
- /* We're done with the queries, perform the login */
- send_login(adapter);
- break;
- case REQUEST_RAS_COMP_NUM_RSP:
- netdev_dbg(netdev, "Got Request RAS Comp Num Response\n");
- if (crq->request_ras_comp_num_rsp.rc.code == 10) {
- netdev_dbg(netdev, "Request RAS Comp Num not supported\n");
- break;
- }
- adapter->ras_comp_num =
- be32_to_cpu(crq->request_ras_comp_num_rsp.num_components);
- handle_request_ras_comp_num_rsp(crq, adapter);
- break;
- case REQUEST_RAS_COMPS_RSP:
- netdev_dbg(netdev, "Got Request RAS Comps Response\n");
- handle_request_ras_comps_rsp(crq, adapter);
- break;
- case CONTROL_RAS_RSP:
- netdev_dbg(netdev, "Got Control RAS Response\n");
- handle_control_ras_rsp(crq, adapter);
+ complete(&adapter->init_done);
break;
case COLLECT_FW_TRACE_RSP:
netdev_dbg(netdev, "Got Collect firmware trace Response\n");
@@ -3455,12 +3058,8 @@
static irqreturn_t ibmvnic_interrupt(int irq, void *instance)
{
struct ibmvnic_adapter *adapter = instance;
- unsigned long flags;
- spin_lock_irqsave(&adapter->crq.lock, flags);
- vio_disable_interrupts(adapter->vdev);
tasklet_schedule(&adapter->tasklet);
- spin_unlock_irqrestore(&adapter->crq.lock, flags);
return IRQ_HANDLED;
}
@@ -3468,32 +3067,23 @@
{
struct ibmvnic_adapter *adapter = data;
struct ibmvnic_crq_queue *queue = &adapter->crq;
- struct vio_dev *vdev = adapter->vdev;
union ibmvnic_crq *crq;
unsigned long flags;
bool done = false;
spin_lock_irqsave(&queue->lock, flags);
- vio_disable_interrupts(vdev);
while (!done) {
/* Pull all the valid messages off the CRQ */
while ((crq = ibmvnic_next_crq(adapter)) != NULL) {
ibmvnic_handle_crq(crq, adapter);
crq->generic.first = 0;
}
- vio_enable_interrupts(vdev);
- crq = ibmvnic_next_crq(adapter);
- if (crq) {
- vio_disable_interrupts(vdev);
- ibmvnic_handle_crq(crq, adapter);
- crq->generic.first = 0;
- } else {
- /* remain in tasklet until all
- * capabilities responses are received
- */
- if (!adapter->wait_capability)
- done = true;
- }
+
+ /* remain in tasklet until all
+ * capabilities responses are received
+ */
+ if (!adapter->wait_capability)
+ done = true;
}
/* if capabilities CRQ's were sent in this tasklet, the following
* tasklet must wait until all responses are received
@@ -3547,12 +3137,15 @@
return rc;
}
-static void ibmvnic_release_crq_queue(struct ibmvnic_adapter *adapter)
+static void release_crq_queue(struct ibmvnic_adapter *adapter)
{
struct ibmvnic_crq_queue *crq = &adapter->crq;
struct vio_dev *vdev = adapter->vdev;
long rc;
+ if (!crq->msgs)
+ return;
+
netdev_dbg(adapter->netdev, "Releasing CRQ\n");
free_irq(vdev->irq, adapter);
tasklet_kill(&adapter->tasklet);
@@ -3563,15 +3156,19 @@
dma_unmap_single(&vdev->dev, crq->msg_token, PAGE_SIZE,
DMA_BIDIRECTIONAL);
free_page((unsigned long)crq->msgs);
+ crq->msgs = NULL;
}
-static int ibmvnic_init_crq_queue(struct ibmvnic_adapter *adapter)
+static int init_crq_queue(struct ibmvnic_adapter *adapter)
{
struct ibmvnic_crq_queue *crq = &adapter->crq;
struct device *dev = &adapter->vdev->dev;
struct vio_dev *vdev = adapter->vdev;
int rc, retrc = -ENOMEM;
+ if (crq->msgs)
+ return 0;
+
crq->msgs = (union ibmvnic_crq *)get_zeroed_page(GFP_KERNEL);
/* Should we allocate more than one page? */
@@ -3633,48 +3230,10 @@
dma_unmap_single(dev, crq->msg_token, PAGE_SIZE, DMA_BIDIRECTIONAL);
map_failed:
free_page((unsigned long)crq->msgs);
+ crq->msgs = NULL;
return retrc;
}
-/* debugfs for dump */
-static int ibmvnic_dump_show(struct seq_file *seq, void *v)
-{
- struct net_device *netdev = seq->private;
- struct ibmvnic_adapter *adapter = netdev_priv(netdev);
- struct device *dev = &adapter->vdev->dev;
- union ibmvnic_crq crq;
-
- memset(&crq, 0, sizeof(crq));
- crq.request_dump_size.first = IBMVNIC_CRQ_CMD;
- crq.request_dump_size.cmd = REQUEST_DUMP_SIZE;
-
- init_completion(&adapter->fw_done);
- ibmvnic_send_crq(adapter, &crq);
- wait_for_completion(&adapter->fw_done);
-
- seq_write(seq, adapter->dump_data, adapter->dump_data_size);
-
- dma_unmap_single(dev, adapter->dump_data_token, adapter->dump_data_size,
- DMA_BIDIRECTIONAL);
-
- kfree(adapter->dump_data);
-
- return 0;
-}
-
-static int ibmvnic_dump_open(struct inode *inode, struct file *file)
-{
- return single_open(file, ibmvnic_dump_show, inode->i_private);
-}
-
-static const struct file_operations ibmvnic_dump_ops = {
- .owner = THIS_MODULE,
- .open = ibmvnic_dump_open,
- .read = seq_read,
- .llseek = seq_lseek,
- .release = single_release,
-};
-
static void handle_crq_init_rsp(struct work_struct *work)
{
struct ibmvnic_adapter *adapter = container_of(work,
@@ -3702,26 +3261,6 @@
goto task_failed;
}
- do {
- if (adapter->renegotiate) {
- adapter->renegotiate = false;
- release_sub_crqs_no_irqs(adapter);
-
- reinit_completion(&adapter->init_done);
- send_cap_queries(adapter);
- if (!wait_for_completion_timeout(&adapter->init_done,
- timeout)) {
- dev_err(dev, "Passive init timeout\n");
- goto task_failed;
- }
- }
- } while (adapter->renegotiate);
- rc = init_sub_crq_irqs(adapter);
-
- if (rc)
- goto task_failed;
-
- netdev->real_num_tx_queues = adapter->req_tx_queues;
netdev->mtu = adapter->req_mtu - ETH_HLEN;
if (adapter->failover) {
@@ -3753,14 +3292,40 @@
dev_err(dev, "Passive initialization was not successful\n");
}
+static int ibmvnic_init(struct ibmvnic_adapter *adapter)
+{
+ struct device *dev = &adapter->vdev->dev;
+ unsigned long timeout = msecs_to_jiffies(30000);
+ int rc;
+
+ rc = init_crq_queue(adapter);
+ if (rc) {
+ dev_err(dev, "Couldn't initialize crq. rc=%d\n", rc);
+ return rc;
+ }
+
+ init_completion(&adapter->init_done);
+ ibmvnic_send_crq_init(adapter);
+ if (!wait_for_completion_timeout(&adapter->init_done, timeout)) {
+ dev_err(dev, "Initialization sequence timed out\n");
+ release_crq_queue(adapter);
+ return -1;
+ }
+
+ rc = init_sub_crqs(adapter);
+ if (rc) {
+ dev_err(dev, "Initialization of sub crqs failed\n");
+ release_crq_queue(adapter);
+ }
+
+ return rc;
+}
+
static int ibmvnic_probe(struct vio_dev *dev, const struct vio_device_id *id)
{
- unsigned long timeout = msecs_to_jiffies(30000);
struct ibmvnic_adapter *adapter;
struct net_device *netdev;
unsigned char *mac_addr_p;
- struct dentry *ent;
- char buf[17]; /* debugfs name buf */
int rc;
dev_dbg(&dev->dev, "entering ibmvnic_probe for UA 0x%x\n",
@@ -3798,91 +3363,27 @@
spin_lock_init(&adapter->stats_lock);
- rc = ibmvnic_init_crq_queue(adapter);
- if (rc) {
- dev_err(&dev->dev, "Couldn't initialize crq. rc=%d\n", rc);
- goto free_netdev;
- }
-
INIT_LIST_HEAD(&adapter->errors);
- INIT_LIST_HEAD(&adapter->inflight);
spin_lock_init(&adapter->error_list_lock);
- spin_lock_init(&adapter->inflight_lock);
- adapter->stats_token = dma_map_single(&dev->dev, &adapter->stats,
- sizeof(struct ibmvnic_statistics),
- DMA_FROM_DEVICE);
- if (dma_mapping_error(&dev->dev, adapter->stats_token)) {
- if (!firmware_has_feature(FW_FEATURE_CMO))
- dev_err(&dev->dev, "Couldn't map stats buffer\n");
- rc = -ENOMEM;
- goto free_crq;
- }
-
- snprintf(buf, sizeof(buf), "ibmvnic_%x", dev->unit_address);
- ent = debugfs_create_dir(buf, NULL);
- if (!ent || IS_ERR(ent)) {
- dev_info(&dev->dev, "debugfs create directory failed\n");
- adapter->debugfs_dir = NULL;
- } else {
- adapter->debugfs_dir = ent;
- ent = debugfs_create_file("dump", S_IRUGO, adapter->debugfs_dir,
- netdev, &ibmvnic_dump_ops);
- if (!ent || IS_ERR(ent)) {
- dev_info(&dev->dev,
- "debugfs create dump file failed\n");
- adapter->debugfs_dump = NULL;
- } else {
- adapter->debugfs_dump = ent;
- }
- }
-
- init_completion(&adapter->init_done);
- ibmvnic_send_crq_init(adapter);
- if (!wait_for_completion_timeout(&adapter->init_done, timeout))
- return 0;
-
- do {
- if (adapter->renegotiate) {
- adapter->renegotiate = false;
- release_sub_crqs_no_irqs(adapter);
-
- reinit_completion(&adapter->init_done);
- send_cap_queries(adapter);
- if (!wait_for_completion_timeout(&adapter->init_done,
- timeout))
- return 0;
- }
- } while (adapter->renegotiate);
-
- rc = init_sub_crq_irqs(adapter);
+ rc = ibmvnic_init(adapter);
if (rc) {
- dev_err(&dev->dev, "failed to initialize sub crq irqs\n");
- goto free_debugfs;
+ free_netdev(netdev);
+ return rc;
}
- netdev->real_num_tx_queues = adapter->req_tx_queues;
netdev->mtu = adapter->req_mtu - ETH_HLEN;
+ adapter->is_closed = false;
rc = register_netdev(netdev);
if (rc) {
dev_err(&dev->dev, "failed to register netdev rc=%d\n", rc);
- goto free_sub_crqs;
+ free_netdev(netdev);
+ return rc;
}
dev_info(&dev->dev, "ibmvnic registered\n");
return 0;
-
-free_sub_crqs:
- release_sub_crqs(adapter);
-free_debugfs:
- if (adapter->debugfs_dir && !IS_ERR(adapter->debugfs_dir))
- debugfs_remove_recursive(adapter->debugfs_dir);
-free_crq:
- ibmvnic_release_crq_queue(adapter);
-free_netdev:
- free_netdev(netdev);
- return rc;
}
static int ibmvnic_remove(struct vio_dev *dev)
@@ -3892,23 +3393,9 @@
unregister_netdev(netdev);
+ release_resources(adapter);
release_sub_crqs(adapter);
-
- ibmvnic_release_crq_queue(adapter);
-
- if (adapter->debugfs_dir && !IS_ERR(adapter->debugfs_dir))
- debugfs_remove_recursive(adapter->debugfs_dir);
-
- dma_unmap_single(&dev->dev, adapter->stats_token,
- sizeof(struct ibmvnic_statistics), DMA_FROM_DEVICE);
-
- if (adapter->ras_comps)
- dma_free_coherent(&dev->dev,
- adapter->ras_comp_num *
- sizeof(struct ibmvnic_fw_component),
- adapter->ras_comps, adapter->ras_comps_tok);
-
- kfree(adapter->ras_comp_int);
+ release_crq_queue(adapter);
free_netdev(netdev);
dev_set_drvdata(&dev->dev, NULL);
@@ -3933,7 +3420,6 @@
adapter = netdev_priv(netdev);
ret += PAGE_SIZE; /* the crq message queue */
- ret += adapter->bounce_buffer_size;
ret += IOMMU_PAGE_ALIGN(sizeof(struct ibmvnic_statistics), tbl);
for (i = 0; i < adapter->req_tx_queues + adapter->req_rx_queues; i++)
diff --git a/drivers/net/ethernet/ibm/ibmvnic.h b/drivers/net/ethernet/ibm/ibmvnic.h
index 1993b42..a69979f 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.h
+++ b/drivers/net/ethernet/ibm/ibmvnic.h
@@ -518,8 +518,8 @@
u8 first;
u8 cmd;
u8 mac_addr[6];
- struct ibmvnic_rc rc;
u8 reserved[4];
+ struct ibmvnic_rc rc;
} __packed __aligned(8);
struct ibmvnic_multicast_ctrl {
@@ -733,6 +733,7 @@
REQ_MTU = 21,
MAX_MULTICAST_FILTERS = 22,
VLAN_HEADER_INSERTION = 23,
+ RX_VLAN_HEADER_INSERTION = 24,
MAX_TX_SG_ENTRIES = 25,
RX_SG_SUPPORTED = 26,
RX_SG_REQUESTED = 27,
@@ -772,20 +773,10 @@
ERROR_INDICATION = 0x08,
REQUEST_ERROR_INFO = 0x09,
REQUEST_ERROR_RSP = 0x89,
- REQUEST_DUMP_SIZE = 0x0A,
- REQUEST_DUMP_SIZE_RSP = 0x8A,
- REQUEST_DUMP = 0x0B,
- REQUEST_DUMP_RSP = 0x8B,
LOGICAL_LINK_STATE = 0x0C,
LOGICAL_LINK_STATE_RSP = 0x8C,
REQUEST_STATISTICS = 0x0D,
REQUEST_STATISTICS_RSP = 0x8D,
- REQUEST_RAS_COMP_NUM = 0x0E,
- REQUEST_RAS_COMP_NUM_RSP = 0x8E,
- REQUEST_RAS_COMPS = 0x0F,
- REQUEST_RAS_COMPS_RSP = 0x8F,
- CONTROL_RAS = 0x10,
- CONTROL_RAS_RSP = 0x90,
COLLECT_FW_TRACE = 0x11,
COLLECT_FW_TRACE_RSP = 0x91,
LINK_STATE_INDICATION = 0x12,
@@ -806,8 +797,6 @@
ACL_CHANGE_INDICATION = 0x1A,
ACL_QUERY = 0x1B,
ACL_QUERY_RSP = 0x9B,
- REQUEST_DEBUG_STATS = 0x1C,
- REQUEST_DEBUG_STATS_RSP = 0x9C,
QUERY_MAP = 0x1D,
QUERY_MAP_RSP = 0x9D,
REQUEST_MAP = 0x1E,
@@ -880,7 +869,6 @@
int index;
int pool_index;
bool last_frag;
- bool used_bounce;
union sub_crq indir_arr[6];
u8 hdr_data[140];
dma_addr_t indir_dma;
@@ -925,18 +913,6 @@
__be32 error_id;
};
-struct ibmvnic_fw_comp_internal {
- struct ibmvnic_adapter *adapter;
- int num;
- struct debugfs_blob_wrapper desc_blob;
- int paused;
-};
-
-struct ibmvnic_inflight_cmd {
- union ibmvnic_crq crq;
- struct list_head list;
-};
-
struct ibmvnic_adapter {
struct vio_dev *vdev;
struct net_device *netdev;
@@ -948,12 +924,8 @@
dma_addr_t ip_offload_ctrl_tok;
bool migrated;
u32 msg_enable;
- void *bounce_buffer;
- int bounce_buffer_size;
- dma_addr_t bounce_buffer_dma;
/* Statistics */
- struct net_device_stats net_stats;
struct ibmvnic_statistics stats;
dma_addr_t stats_token;
struct completion stats_done;
@@ -992,26 +964,12 @@
struct ibmvnic_tx_pool *tx_pool;
bool closing;
struct completion init_done;
+ int init_done_rc;
struct list_head errors;
spinlock_t error_list_lock;
- /* debugfs */
- struct dentry *debugfs_dir;
- struct dentry *debugfs_dump;
struct completion fw_done;
- char *dump_data;
- dma_addr_t dump_data_token;
- int dump_data_size;
- int ras_comp_num;
- struct ibmvnic_fw_component *ras_comps;
- struct ibmvnic_fw_comp_internal *ras_comp_int;
- dma_addr_t ras_comps_tok;
- struct dentry *ras_comps_ent;
-
- /* in-flight commands that allocate and/or map memory*/
- struct list_head inflight;
- spinlock_t inflight_lock;
/* partner capabilities */
u64 min_tx_queues;
@@ -1037,6 +995,7 @@
u64 req_mtu;
u64 max_multicast_filters;
u64 vlan_header_insertion;
+ u64 rx_vlan_header_insertion;
u64 max_tx_sg_entries;
u64 rx_sg_supported;
u64 rx_sg_requested;
@@ -1052,4 +1011,5 @@
struct work_struct ibmvnic_xport;
struct tasklet_struct tasklet;
bool failover;
+ bool is_closed;
};
diff --git a/drivers/net/ethernet/intel/Kconfig b/drivers/net/ethernet/intel/Kconfig
index 1349b45..1542a21 100644
--- a/drivers/net/ethernet/intel/Kconfig
+++ b/drivers/net/ethernet/intel/Kconfig
@@ -235,17 +235,6 @@
If unsure, say N.
-config I40E_FCOE
- bool "Fibre Channel over Ethernet (FCoE)"
- default n
- depends on I40E && DCB && FCOE
- ---help---
- Say Y here if you want to use Fibre Channel over Ethernet (FCoE)
- in the driver. This will create new netdev for exclusive FCoE
- use with XL710 FCoE offloads enabled.
-
- If unsure, say N.
-
config I40EVF
tristate "Intel(R) XL710 X710 Virtual Function Ethernet support"
depends on PCI_MSI
diff --git a/drivers/net/ethernet/intel/e1000/e1000_ethtool.c b/drivers/net/ethernet/intel/e1000/e1000_ethtool.c
index 975eeb8..ec8aa45 100644
--- a/drivers/net/ethernet/intel/e1000/e1000_ethtool.c
+++ b/drivers/net/ethernet/intel/e1000/e1000_ethtool.c
@@ -103,104 +103,104 @@
#define E1000_TEST_LEN ARRAY_SIZE(e1000_gstrings_test)
-static int e1000_get_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+static int e1000_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
{
struct e1000_adapter *adapter = netdev_priv(netdev);
struct e1000_hw *hw = &adapter->hw;
+ u32 supported, advertising;
if (hw->media_type == e1000_media_type_copper) {
- ecmd->supported = (SUPPORTED_10baseT_Half |
- SUPPORTED_10baseT_Full |
- SUPPORTED_100baseT_Half |
- SUPPORTED_100baseT_Full |
- SUPPORTED_1000baseT_Full|
- SUPPORTED_Autoneg |
- SUPPORTED_TP);
- ecmd->advertising = ADVERTISED_TP;
+ supported = (SUPPORTED_10baseT_Half |
+ SUPPORTED_10baseT_Full |
+ SUPPORTED_100baseT_Half |
+ SUPPORTED_100baseT_Full |
+ SUPPORTED_1000baseT_Full|
+ SUPPORTED_Autoneg |
+ SUPPORTED_TP);
+ advertising = ADVERTISED_TP;
if (hw->autoneg == 1) {
- ecmd->advertising |= ADVERTISED_Autoneg;
+ advertising |= ADVERTISED_Autoneg;
/* the e1000 autoneg seems to match ethtool nicely */
- ecmd->advertising |= hw->autoneg_advertised;
+ advertising |= hw->autoneg_advertised;
}
- ecmd->port = PORT_TP;
- ecmd->phy_address = hw->phy_addr;
-
- if (hw->mac_type == e1000_82543)
- ecmd->transceiver = XCVR_EXTERNAL;
- else
- ecmd->transceiver = XCVR_INTERNAL;
-
+ cmd->base.port = PORT_TP;
+ cmd->base.phy_address = hw->phy_addr;
} else {
- ecmd->supported = (SUPPORTED_1000baseT_Full |
- SUPPORTED_FIBRE |
- SUPPORTED_Autoneg);
+ supported = (SUPPORTED_1000baseT_Full |
+ SUPPORTED_FIBRE |
+ SUPPORTED_Autoneg);
- ecmd->advertising = (ADVERTISED_1000baseT_Full |
- ADVERTISED_FIBRE |
- ADVERTISED_Autoneg);
+ advertising = (ADVERTISED_1000baseT_Full |
+ ADVERTISED_FIBRE |
+ ADVERTISED_Autoneg);
- ecmd->port = PORT_FIBRE;
-
- if (hw->mac_type >= e1000_82545)
- ecmd->transceiver = XCVR_INTERNAL;
- else
- ecmd->transceiver = XCVR_EXTERNAL;
+ cmd->base.port = PORT_FIBRE;
}
if (er32(STATUS) & E1000_STATUS_LU) {
e1000_get_speed_and_duplex(hw, &adapter->link_speed,
&adapter->link_duplex);
- ethtool_cmd_speed_set(ecmd, adapter->link_speed);
+ cmd->base.speed = adapter->link_speed;
/* unfortunately FULL_DUPLEX != DUPLEX_FULL
* and HALF_DUPLEX != DUPLEX_HALF
*/
if (adapter->link_duplex == FULL_DUPLEX)
- ecmd->duplex = DUPLEX_FULL;
+ cmd->base.duplex = DUPLEX_FULL;
else
- ecmd->duplex = DUPLEX_HALF;
+ cmd->base.duplex = DUPLEX_HALF;
} else {
- ethtool_cmd_speed_set(ecmd, SPEED_UNKNOWN);
- ecmd->duplex = DUPLEX_UNKNOWN;
+ cmd->base.speed = SPEED_UNKNOWN;
+ cmd->base.duplex = DUPLEX_UNKNOWN;
}
- ecmd->autoneg = ((hw->media_type == e1000_media_type_fiber) ||
+ cmd->base.autoneg = ((hw->media_type == e1000_media_type_fiber) ||
hw->autoneg) ? AUTONEG_ENABLE : AUTONEG_DISABLE;
/* MDI-X => 1; MDI => 0 */
if ((hw->media_type == e1000_media_type_copper) &&
netif_carrier_ok(netdev))
- ecmd->eth_tp_mdix = (!!adapter->phy_info.mdix_mode ?
+ cmd->base.eth_tp_mdix = (!!adapter->phy_info.mdix_mode ?
ETH_TP_MDI_X : ETH_TP_MDI);
else
- ecmd->eth_tp_mdix = ETH_TP_MDI_INVALID;
+ cmd->base.eth_tp_mdix = ETH_TP_MDI_INVALID;
if (hw->mdix == AUTO_ALL_MODES)
- ecmd->eth_tp_mdix_ctrl = ETH_TP_MDI_AUTO;
+ cmd->base.eth_tp_mdix_ctrl = ETH_TP_MDI_AUTO;
else
- ecmd->eth_tp_mdix_ctrl = hw->mdix;
+ cmd->base.eth_tp_mdix_ctrl = hw->mdix;
+
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
+ advertising);
+
return 0;
}
-static int e1000_set_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+static int e1000_set_link_ksettings(struct net_device *netdev,
+ const struct ethtool_link_ksettings *cmd)
{
struct e1000_adapter *adapter = netdev_priv(netdev);
struct e1000_hw *hw = &adapter->hw;
+ u32 advertising;
+
+ ethtool_convert_link_mode_to_legacy_u32(&advertising,
+ cmd->link_modes.advertising);
/* MDI setting is only allowed when autoneg enabled because
* some hardware doesn't allow MDI setting when speed or
* duplex is forced.
*/
- if (ecmd->eth_tp_mdix_ctrl) {
+ if (cmd->base.eth_tp_mdix_ctrl) {
if (hw->media_type != e1000_media_type_copper)
return -EOPNOTSUPP;
- if ((ecmd->eth_tp_mdix_ctrl != ETH_TP_MDI_AUTO) &&
- (ecmd->autoneg != AUTONEG_ENABLE)) {
+ if ((cmd->base.eth_tp_mdix_ctrl != ETH_TP_MDI_AUTO) &&
+ (cmd->base.autoneg != AUTONEG_ENABLE)) {
e_err(drv, "forcing MDI/MDI-X state is not supported when link speed and/or duplex are forced\n");
return -EINVAL;
}
@@ -209,32 +209,31 @@
while (test_and_set_bit(__E1000_RESETTING, &adapter->flags))
msleep(1);
- if (ecmd->autoneg == AUTONEG_ENABLE) {
+ if (cmd->base.autoneg == AUTONEG_ENABLE) {
hw->autoneg = 1;
if (hw->media_type == e1000_media_type_fiber)
hw->autoneg_advertised = ADVERTISED_1000baseT_Full |
- ADVERTISED_FIBRE |
- ADVERTISED_Autoneg;
+ ADVERTISED_FIBRE |
+ ADVERTISED_Autoneg;
else
- hw->autoneg_advertised = ecmd->advertising |
+ hw->autoneg_advertised = advertising |
ADVERTISED_TP |
ADVERTISED_Autoneg;
- ecmd->advertising = hw->autoneg_advertised;
} else {
- u32 speed = ethtool_cmd_speed(ecmd);
+ u32 speed = cmd->base.speed;
/* calling this overrides forced MDI setting */
- if (e1000_set_spd_dplx(adapter, speed, ecmd->duplex)) {
+ if (e1000_set_spd_dplx(adapter, speed, cmd->base.duplex)) {
clear_bit(__E1000_RESETTING, &adapter->flags);
return -EINVAL;
}
}
/* MDI-X => 2; MDI => 1; Auto => 3 */
- if (ecmd->eth_tp_mdix_ctrl) {
- if (ecmd->eth_tp_mdix_ctrl == ETH_TP_MDI_AUTO)
+ if (cmd->base.eth_tp_mdix_ctrl) {
+ if (cmd->base.eth_tp_mdix_ctrl == ETH_TP_MDI_AUTO)
hw->mdix = AUTO_ALL_MODES;
else
- hw->mdix = ecmd->eth_tp_mdix_ctrl;
+ hw->mdix = cmd->base.eth_tp_mdix_ctrl;
}
/* reset the link */
@@ -1875,8 +1874,6 @@
}
static const struct ethtool_ops e1000_ethtool_ops = {
- .get_settings = e1000_get_settings,
- .set_settings = e1000_set_settings,
.get_drvinfo = e1000_get_drvinfo,
.get_regs_len = e1000_get_regs_len,
.get_regs = e1000_get_regs,
@@ -1901,6 +1898,8 @@
.get_coalesce = e1000_get_coalesce,
.set_coalesce = e1000_set_coalesce,
.get_ts_info = ethtool_op_get_ts_info,
+ .get_link_ksettings = e1000_get_link_ksettings,
+ .set_link_ksettings = e1000_set_link_ksettings,
};
void e1000_set_ethtool_ops(struct net_device *netdev)
diff --git a/drivers/net/ethernet/intel/e1000/e1000_main.c b/drivers/net/ethernet/intel/e1000/e1000_main.c
index 93fc6c6..bd8b05f 100644
--- a/drivers/net/ethernet/intel/e1000/e1000_main.c
+++ b/drivers/net/ethernet/intel/e1000/e1000_main.c
@@ -131,7 +131,6 @@
static void e1000_82547_tx_fifo_stall_task(struct work_struct *work);
static netdev_tx_t e1000_xmit_frame(struct sk_buff *skb,
struct net_device *netdev);
-static struct net_device_stats *e1000_get_stats(struct net_device *netdev);
static int e1000_change_mtu(struct net_device *netdev, int new_mtu);
static int e1000_set_mac(struct net_device *netdev, void *p);
static irqreturn_t e1000_intr(int irq, void *data);
@@ -846,7 +845,6 @@
.ndo_open = e1000_open,
.ndo_stop = e1000_close,
.ndo_start_xmit = e1000_xmit_frame,
- .ndo_get_stats = e1000_get_stats,
.ndo_set_rx_mode = e1000_set_rx_mode,
.ndo_set_mac_address = e1000_set_mac,
.ndo_tx_timeout = e1000_tx_timeout,
@@ -3530,19 +3528,6 @@
}
/**
- * e1000_get_stats - Get System Network Statistics
- * @netdev: network interface device structure
- *
- * Returns the address of the device statistics structure.
- * The statistics are actually updated from the watchdog.
- **/
-static struct net_device_stats *e1000_get_stats(struct net_device *netdev)
-{
- /* only return the current stats */
- return &netdev->stats;
-}
-
-/**
* e1000_change_mtu - Change the Maximum Transfer Unit
* @netdev: network interface device structure
* @new_mtu: new value for maximum frame size
diff --git a/drivers/net/ethernet/intel/e1000e/e1000.h b/drivers/net/ethernet/intel/e1000e/e1000.h
index a29b12e8..c7c994e 100644
--- a/drivers/net/ethernet/intel/e1000e/e1000.h
+++ b/drivers/net/ethernet/intel/e1000e/e1000.h
@@ -135,7 +135,8 @@
board_pchlan,
board_pch2lan,
board_pch_lpt,
- board_pch_spt
+ board_pch_spt,
+ board_pch_cnp
};
struct e1000_ps_page {
@@ -378,18 +379,22 @@
* INCVALUE_n into the TIMINCA register allowing 32+8+(24-INCVALUE_SHIFT_n)
* bits to count nanoseconds leaving the rest for fractional nonseconds.
*/
-#define INCVALUE_96MHz 125
-#define INCVALUE_SHIFT_96MHz 17
-#define INCPERIOD_SHIFT_96MHz 2
-#define INCPERIOD_96MHz (12 >> INCPERIOD_SHIFT_96MHz)
+#define INCVALUE_96MHZ 125
+#define INCVALUE_SHIFT_96MHZ 17
+#define INCPERIOD_SHIFT_96MHZ 2
+#define INCPERIOD_96MHZ (12 >> INCPERIOD_SHIFT_96MHZ)
-#define INCVALUE_25MHz 40
-#define INCVALUE_SHIFT_25MHz 18
-#define INCPERIOD_25MHz 1
+#define INCVALUE_25MHZ 40
+#define INCVALUE_SHIFT_25MHZ 18
+#define INCPERIOD_25MHZ 1
-#define INCVALUE_24MHz 125
-#define INCVALUE_SHIFT_24MHz 14
-#define INCPERIOD_24MHz 3
+#define INCVALUE_24MHZ 125
+#define INCVALUE_SHIFT_24MHZ 14
+#define INCPERIOD_24MHZ 3
+
+#define INCVALUE_38400KHZ 26
+#define INCVALUE_SHIFT_38400KHZ 19
+#define INCPERIOD_38400KHZ 1
/* Another drawback of scaling the incvalue by a large factor is the
* 64-bit SYSTIM register overflows more quickly. This is dealt with
@@ -515,6 +520,7 @@
extern const struct e1000_info e1000_pch2_info;
extern const struct e1000_info e1000_pch_lpt_info;
extern const struct e1000_info e1000_pch_spt_info;
+extern const struct e1000_info e1000_pch_cnp_info;
extern const struct e1000_info e1000_es2_info;
void e1000e_ptp_init(struct e1000_adapter *adapter);
diff --git a/drivers/net/ethernet/intel/e1000e/ethtool.c b/drivers/net/ethernet/intel/e1000e/ethtool.c
index 7aff68a..e23dbd9 100644
--- a/drivers/net/ethernet/intel/e1000e/ethtool.c
+++ b/drivers/net/ethernet/intel/e1000e/ethtool.c
@@ -117,55 +117,52 @@
#define E1000_TEST_LEN ARRAY_SIZE(e1000_gstrings_test)
-static int e1000_get_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+static int e1000_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
{
struct e1000_adapter *adapter = netdev_priv(netdev);
struct e1000_hw *hw = &adapter->hw;
- u32 speed;
+ u32 speed, supported, advertising;
if (hw->phy.media_type == e1000_media_type_copper) {
- ecmd->supported = (SUPPORTED_10baseT_Half |
- SUPPORTED_10baseT_Full |
- SUPPORTED_100baseT_Half |
- SUPPORTED_100baseT_Full |
- SUPPORTED_1000baseT_Full |
- SUPPORTED_Autoneg |
- SUPPORTED_TP);
+ supported = (SUPPORTED_10baseT_Half |
+ SUPPORTED_10baseT_Full |
+ SUPPORTED_100baseT_Half |
+ SUPPORTED_100baseT_Full |
+ SUPPORTED_1000baseT_Full |
+ SUPPORTED_Autoneg |
+ SUPPORTED_TP);
if (hw->phy.type == e1000_phy_ife)
- ecmd->supported &= ~SUPPORTED_1000baseT_Full;
- ecmd->advertising = ADVERTISED_TP;
+ supported &= ~SUPPORTED_1000baseT_Full;
+ advertising = ADVERTISED_TP;
if (hw->mac.autoneg == 1) {
- ecmd->advertising |= ADVERTISED_Autoneg;
+ advertising |= ADVERTISED_Autoneg;
/* the e1000 autoneg seems to match ethtool nicely */
- ecmd->advertising |= hw->phy.autoneg_advertised;
+ advertising |= hw->phy.autoneg_advertised;
}
- ecmd->port = PORT_TP;
- ecmd->phy_address = hw->phy.addr;
- ecmd->transceiver = XCVR_INTERNAL;
-
+ cmd->base.port = PORT_TP;
+ cmd->base.phy_address = hw->phy.addr;
} else {
- ecmd->supported = (SUPPORTED_1000baseT_Full |
- SUPPORTED_FIBRE |
- SUPPORTED_Autoneg);
+ supported = (SUPPORTED_1000baseT_Full |
+ SUPPORTED_FIBRE |
+ SUPPORTED_Autoneg);
- ecmd->advertising = (ADVERTISED_1000baseT_Full |
- ADVERTISED_FIBRE |
- ADVERTISED_Autoneg);
+ advertising = (ADVERTISED_1000baseT_Full |
+ ADVERTISED_FIBRE |
+ ADVERTISED_Autoneg);
- ecmd->port = PORT_FIBRE;
- ecmd->transceiver = XCVR_EXTERNAL;
+ cmd->base.port = PORT_FIBRE;
}
speed = SPEED_UNKNOWN;
- ecmd->duplex = DUPLEX_UNKNOWN;
+ cmd->base.duplex = DUPLEX_UNKNOWN;
if (netif_running(netdev)) {
if (netif_carrier_ok(netdev)) {
speed = adapter->link_speed;
- ecmd->duplex = adapter->link_duplex - 1;
+ cmd->base.duplex = adapter->link_duplex - 1;
}
} else if (!pm_runtime_suspended(netdev->dev.parent)) {
u32 status = er32(STATUS);
@@ -179,30 +176,36 @@
speed = SPEED_10;
if (status & E1000_STATUS_FD)
- ecmd->duplex = DUPLEX_FULL;
+ cmd->base.duplex = DUPLEX_FULL;
else
- ecmd->duplex = DUPLEX_HALF;
+ cmd->base.duplex = DUPLEX_HALF;
}
}
- ethtool_cmd_speed_set(ecmd, speed);
- ecmd->autoneg = ((hw->phy.media_type == e1000_media_type_fiber) ||
+ cmd->base.speed = speed;
+ cmd->base.autoneg = ((hw->phy.media_type == e1000_media_type_fiber) ||
hw->mac.autoneg) ? AUTONEG_ENABLE : AUTONEG_DISABLE;
/* MDI-X => 2; MDI =>1; Invalid =>0 */
if ((hw->phy.media_type == e1000_media_type_copper) &&
netif_carrier_ok(netdev))
- ecmd->eth_tp_mdix = hw->phy.is_mdix ? ETH_TP_MDI_X : ETH_TP_MDI;
+ cmd->base.eth_tp_mdix = hw->phy.is_mdix ?
+ ETH_TP_MDI_X : ETH_TP_MDI;
else
- ecmd->eth_tp_mdix = ETH_TP_MDI_INVALID;
+ cmd->base.eth_tp_mdix = ETH_TP_MDI_INVALID;
if (hw->phy.mdix == AUTO_ALL_MODES)
- ecmd->eth_tp_mdix_ctrl = ETH_TP_MDI_AUTO;
+ cmd->base.eth_tp_mdix_ctrl = ETH_TP_MDI_AUTO;
else
- ecmd->eth_tp_mdix_ctrl = hw->phy.mdix;
+ cmd->base.eth_tp_mdix_ctrl = hw->phy.mdix;
if (hw->phy.media_type != e1000_media_type_copper)
- ecmd->eth_tp_mdix_ctrl = ETH_TP_MDI_INVALID;
+ cmd->base.eth_tp_mdix_ctrl = ETH_TP_MDI_INVALID;
+
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
+ advertising);
return 0;
}
@@ -262,12 +265,16 @@
return -EINVAL;
}
-static int e1000_set_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+static int e1000_set_link_ksettings(struct net_device *netdev,
+ const struct ethtool_link_ksettings *cmd)
{
struct e1000_adapter *adapter = netdev_priv(netdev);
struct e1000_hw *hw = &adapter->hw;
int ret_val = 0;
+ u32 advertising;
+
+ ethtool_convert_link_mode_to_legacy_u32(&advertising,
+ cmd->link_modes.advertising);
pm_runtime_get_sync(netdev->dev.parent);
@@ -285,14 +292,14 @@
* some hardware doesn't allow MDI setting when speed or
* duplex is forced.
*/
- if (ecmd->eth_tp_mdix_ctrl) {
+ if (cmd->base.eth_tp_mdix_ctrl) {
if (hw->phy.media_type != e1000_media_type_copper) {
ret_val = -EOPNOTSUPP;
goto out;
}
- if ((ecmd->eth_tp_mdix_ctrl != ETH_TP_MDI_AUTO) &&
- (ecmd->autoneg != AUTONEG_ENABLE)) {
+ if ((cmd->base.eth_tp_mdix_ctrl != ETH_TP_MDI_AUTO) &&
+ (cmd->base.autoneg != AUTONEG_ENABLE)) {
e_err("forcing MDI/MDI-X state is not supported when link speed and/or duplex are forced\n");
ret_val = -EINVAL;
goto out;
@@ -302,35 +309,35 @@
while (test_and_set_bit(__E1000_RESETTING, &adapter->state))
usleep_range(1000, 2000);
- if (ecmd->autoneg == AUTONEG_ENABLE) {
+ if (cmd->base.autoneg == AUTONEG_ENABLE) {
hw->mac.autoneg = 1;
if (hw->phy.media_type == e1000_media_type_fiber)
hw->phy.autoneg_advertised = ADVERTISED_1000baseT_Full |
ADVERTISED_FIBRE | ADVERTISED_Autoneg;
else
- hw->phy.autoneg_advertised = ecmd->advertising |
+ hw->phy.autoneg_advertised = advertising |
ADVERTISED_TP | ADVERTISED_Autoneg;
- ecmd->advertising = hw->phy.autoneg_advertised;
+ advertising = hw->phy.autoneg_advertised;
if (adapter->fc_autoneg)
hw->fc.requested_mode = e1000_fc_default;
} else {
- u32 speed = ethtool_cmd_speed(ecmd);
+ u32 speed = cmd->base.speed;
/* calling this overrides forced MDI setting */
- if (e1000_set_spd_dplx(adapter, speed, ecmd->duplex)) {
+ if (e1000_set_spd_dplx(adapter, speed, cmd->base.duplex)) {
ret_val = -EINVAL;
goto out;
}
}
/* MDI-X => 2; MDI => 1; Auto => 3 */
- if (ecmd->eth_tp_mdix_ctrl) {
+ if (cmd->base.eth_tp_mdix_ctrl) {
/* fix up the value for auto (3 => 0) as zero is mapped
* internally to auto
*/
- if (ecmd->eth_tp_mdix_ctrl == ETH_TP_MDI_AUTO)
+ if (cmd->base.eth_tp_mdix_ctrl == ETH_TP_MDI_AUTO)
hw->phy.mdix = AUTO_ALL_MODES;
else
- hw->phy.mdix = ecmd->eth_tp_mdix_ctrl;
+ hw->phy.mdix = cmd->base.eth_tp_mdix_ctrl;
}
/* reset the link */
@@ -904,19 +911,20 @@
case e1000_pch2lan:
case e1000_pch_lpt:
case e1000_pch_spt:
+ /* fall through */
+ case e1000_pch_cnp:
mask |= BIT(18);
break;
default:
break;
}
- if ((mac->type == e1000_pch_lpt) || (mac->type == e1000_pch_spt))
+ if (mac->type >= e1000_pch_lpt)
wlock_mac = (er32(FWSM) & E1000_FWSM_WLOCK_MAC_MASK) >>
E1000_FWSM_WLOCK_MAC_SHIFT;
for (i = 0; i < mac->rar_entry_count; i++) {
- if ((mac->type == e1000_pch_lpt) ||
- (mac->type == e1000_pch_spt)) {
+ if (mac->type >= e1000_pch_lpt) {
/* Cannot test write-protected SHRAL[n] registers */
if ((wlock_mac == 1) || (wlock_mac && (i > wlock_mac)))
continue;
@@ -1525,7 +1533,7 @@
struct e1000_hw *hw = &adapter->hw;
u32 rctl, fext_nvm11, tarc0;
- if (hw->mac.type == e1000_pch_spt) {
+ if (hw->mac.type >= e1000_pch_spt) {
fext_nvm11 = er32(FEXTNVM11);
fext_nvm11 |= E1000_FEXTNVM11_DISABLE_MULR_FIX;
ew32(FEXTNVM11, fext_nvm11);
@@ -1569,6 +1577,7 @@
switch (hw->mac.type) {
case e1000_pch_spt:
+ case e1000_pch_cnp:
fext_nvm11 = er32(FEXTNVM11);
fext_nvm11 &= ~E1000_FEXTNVM11_DISABLE_MULR_FIX;
ew32(FEXTNVM11, fext_nvm11);
@@ -2313,8 +2322,6 @@
}
static const struct ethtool_ops e1000_ethtool_ops = {
- .get_settings = e1000_get_settings,
- .set_settings = e1000_set_settings,
.get_drvinfo = e1000_get_drvinfo,
.get_regs_len = e1000_get_regs_len,
.get_regs = e1000_get_regs,
@@ -2342,6 +2349,8 @@
.get_ts_info = e1000e_get_ts_info,
.get_eee = e1000e_get_eee,
.set_eee = e1000e_set_eee,
+ .get_link_ksettings = e1000_get_link_ksettings,
+ .set_link_ksettings = e1000_set_link_ksettings,
};
void e1000e_set_ethtool_ops(struct net_device *netdev)
diff --git a/drivers/net/ethernet/intel/e1000e/hw.h b/drivers/net/ethernet/intel/e1000e/hw.h
index 4e733bf..66bd506 100644
--- a/drivers/net/ethernet/intel/e1000e/hw.h
+++ b/drivers/net/ethernet/intel/e1000e/hw.h
@@ -96,6 +96,10 @@
#define E1000_DEV_ID_PCH_SPT_I219_V4 0x15D8
#define E1000_DEV_ID_PCH_SPT_I219_LM5 0x15E3
#define E1000_DEV_ID_PCH_SPT_I219_V5 0x15D6
+#define E1000_DEV_ID_PCH_CNP_I219_LM6 0x15BD
+#define E1000_DEV_ID_PCH_CNP_I219_V6 0x15BE
+#define E1000_DEV_ID_PCH_CNP_I219_LM7 0x15BB
+#define E1000_DEV_ID_PCH_CNP_I219_V7 0x15BC
#define E1000_REVISION_4 4
@@ -118,6 +122,7 @@
e1000_pch2lan,
e1000_pch_lpt,
e1000_pch_spt,
+ e1000_pch_cnp,
};
enum e1000_media_type {
diff --git a/drivers/net/ethernet/intel/e1000e/ich8lan.c b/drivers/net/ethernet/intel/e1000e/ich8lan.c
index f3aaca7..68ea8b4 100644
--- a/drivers/net/ethernet/intel/e1000e/ich8lan.c
+++ b/drivers/net/ethernet/intel/e1000e/ich8lan.c
@@ -237,7 +237,7 @@
if (ret_val)
return false;
out:
- if ((hw->mac.type == e1000_pch_lpt) || (hw->mac.type == e1000_pch_spt)) {
+ if (hw->mac.type >= e1000_pch_lpt) {
/* Only unforce SMBus if ME is not active */
if (!(er32(FWSM) & E1000_ICH_FWSM_FW_VALID)) {
/* Unforce SMBus mode in PHY */
@@ -333,6 +333,7 @@
switch (hw->mac.type) {
case e1000_pch_lpt:
case e1000_pch_spt:
+ case e1000_pch_cnp:
if (e1000_phy_is_accessible_pchlan(hw))
break;
@@ -474,6 +475,7 @@
case e1000_pch2lan:
case e1000_pch_lpt:
case e1000_pch_spt:
+ case e1000_pch_cnp:
/* In case the PHY needs to be in mdio slow mode,
* set slow mode and try to get the PHY id again.
*/
@@ -607,7 +609,7 @@
nvm->type = e1000_nvm_flash_sw;
- if (hw->mac.type == e1000_pch_spt) {
+ if (hw->mac.type >= e1000_pch_spt) {
/* in SPT, gfpreg doesn't exist. NVM size is taken from the
* STRAP register. This is because in SPT the GbE Flash region
* is no longer accessed through the flash registers. Instead,
@@ -715,6 +717,7 @@
/* fall-through */
case e1000_pch_lpt:
case e1000_pch_spt:
+ case e1000_pch_cnp:
case e1000_pchlan:
/* check management mode */
mac->ops.check_mng_mode = e1000_check_mng_mode_pchlan;
@@ -732,7 +735,7 @@
break;
}
- if ((mac->type == e1000_pch_lpt) || (mac->type == e1000_pch_spt)) {
+ if (mac->type >= e1000_pch_lpt) {
mac->rar_entry_count = E1000_PCH_LPT_RAR_ENTRIES;
mac->ops.rar_set = e1000_rar_set_pch_lpt;
mac->ops.setup_physical_interface =
@@ -1399,9 +1402,7 @@
* aggressive resulting in many collisions. To avoid this, increase
* the IPG and reduce Rx latency in the PHY.
*/
- if (((hw->mac.type == e1000_pch2lan) ||
- (hw->mac.type == e1000_pch_lpt) ||
- (hw->mac.type == e1000_pch_spt)) && link) {
+ if ((hw->mac.type >= e1000_pch2lan) && link) {
u16 speed, duplex;
e1000e_get_speed_and_duplex_copper(hw, &speed, &duplex);
@@ -1412,7 +1413,7 @@
tipg_reg |= 0xFF;
/* Reduce Rx latency in analog PHY */
emi_val = 0;
- } else if (hw->mac.type == e1000_pch_spt &&
+ } else if (hw->mac.type >= e1000_pch_spt &&
duplex == FULL_DUPLEX && speed != SPEED_1000) {
tipg_reg |= 0xC;
emi_val = 1;
@@ -1435,8 +1436,7 @@
emi_addr = I217_RX_CONFIG;
ret_val = e1000_write_emi_reg_locked(hw, emi_addr, emi_val);
- if (hw->mac.type == e1000_pch_lpt ||
- hw->mac.type == e1000_pch_spt) {
+ if (hw->mac.type >= e1000_pch_lpt) {
u16 phy_reg;
e1e_rphy_locked(hw, I217_PLL_CLOCK_GATE_REG, &phy_reg);
@@ -1452,7 +1452,7 @@
if (ret_val)
return ret_val;
- if (hw->mac.type == e1000_pch_spt) {
+ if (hw->mac.type >= e1000_pch_spt) {
u16 data;
u16 ptr_gap;
@@ -1502,7 +1502,7 @@
* on power up.
* Set the Beacon Duration for I217 to 8 usec
*/
- if ((hw->mac.type == e1000_pch_lpt) || (hw->mac.type == e1000_pch_spt)) {
+ if (hw->mac.type >= e1000_pch_lpt) {
u32 mac_reg;
mac_reg = er32(FEXTNVM4);
@@ -1520,8 +1520,7 @@
if (ret_val)
return ret_val;
}
- if ((hw->mac.type == e1000_pch_lpt) ||
- (hw->mac.type == e1000_pch_spt)) {
+ if (hw->mac.type >= e1000_pch_lpt) {
/* Set platform power management values for
* Latency Tolerance Reporting (LTR)
*/
@@ -1533,15 +1532,18 @@
/* Clear link partner's EEE ability */
hw->dev_spec.ich8lan.eee_lp_ability = 0;
- /* FEXTNVM6 K1-off workaround */
- if (hw->mac.type == e1000_pch_spt) {
- u32 pcieanacfg = er32(PCIEANACFG);
+ if (hw->mac.type >= e1000_pch_lpt) {
u32 fextnvm6 = er32(FEXTNVM6);
- if (pcieanacfg & E1000_FEXTNVM6_K1_OFF_ENABLE)
- fextnvm6 |= E1000_FEXTNVM6_K1_OFF_ENABLE;
- else
- fextnvm6 &= ~E1000_FEXTNVM6_K1_OFF_ENABLE;
+ if (hw->mac.type == e1000_pch_spt) {
+ /* FEXTNVM6 K1-off workaround - for SPT only */
+ u32 pcieanacfg = er32(PCIEANACFG);
+
+ if (pcieanacfg & E1000_FEXTNVM6_K1_OFF_ENABLE)
+ fextnvm6 |= E1000_FEXTNVM6_K1_OFF_ENABLE;
+ else
+ fextnvm6 &= ~E1000_FEXTNVM6_K1_OFF_ENABLE;
+ }
ew32(FEXTNVM6, fextnvm6);
}
@@ -1640,6 +1642,7 @@
case e1000_pch2lan:
case e1000_pch_lpt:
case e1000_pch_spt:
+ case e1000_pch_cnp:
rc = e1000_init_phy_params_pchlan(hw);
break;
default:
@@ -2091,6 +2094,7 @@
case e1000_pch2lan:
case e1000_pch_lpt:
case e1000_pch_spt:
+ case e1000_pch_cnp:
sw_cfg_mask = E1000_FEXTNVM_SW_CONFIG_ICH8M;
break;
default:
@@ -3125,6 +3129,7 @@
switch (hw->mac.type) {
case e1000_pch_spt:
+ case e1000_pch_cnp:
bank1_offset = nvm->flash_bank_size;
act_offset = E1000_ICH_NVM_SIG_WORD;
@@ -3380,7 +3385,7 @@
/* Clear FCERR and DAEL in hw status by writing 1 */
hsfsts.hsf_status.flcerr = 1;
hsfsts.hsf_status.dael = 1;
- if (hw->mac.type == e1000_pch_spt)
+ if (hw->mac.type >= e1000_pch_spt)
ew32flash(ICH_FLASH_HSFSTS, hsfsts.regval & 0xFFFF);
else
ew16flash(ICH_FLASH_HSFSTS, hsfsts.regval);
@@ -3399,7 +3404,7 @@
* Begin by setting Flash Cycle Done.
*/
hsfsts.hsf_status.flcdone = 1;
- if (hw->mac.type == e1000_pch_spt)
+ if (hw->mac.type >= e1000_pch_spt)
ew32flash(ICH_FLASH_HSFSTS, hsfsts.regval & 0xFFFF);
else
ew16flash(ICH_FLASH_HSFSTS, hsfsts.regval);
@@ -3423,7 +3428,7 @@
* now set the Flash Cycle Done.
*/
hsfsts.hsf_status.flcdone = 1;
- if (hw->mac.type == e1000_pch_spt)
+ if (hw->mac.type >= e1000_pch_spt)
ew32flash(ICH_FLASH_HSFSTS,
hsfsts.regval & 0xFFFF);
else
@@ -3450,13 +3455,13 @@
u32 i = 0;
/* Start a cycle by writing 1 in Flash Cycle Go in Hw Flash Control */
- if (hw->mac.type == e1000_pch_spt)
+ if (hw->mac.type >= e1000_pch_spt)
hsflctl.regval = er32flash(ICH_FLASH_HSFSTS) >> 16;
else
hsflctl.regval = er16flash(ICH_FLASH_HSFCTL);
hsflctl.hsf_ctrl.flcgo = 1;
- if (hw->mac.type == e1000_pch_spt)
+ if (hw->mac.type >= e1000_pch_spt)
ew32flash(ICH_FLASH_HSFSTS, hsflctl.regval << 16);
else
ew16flash(ICH_FLASH_HSFCTL, hsflctl.regval);
@@ -3527,7 +3532,7 @@
/* In SPT, only 32 bits access is supported,
* so this function should not be called.
*/
- if (hw->mac.type == e1000_pch_spt)
+ if (hw->mac.type >= e1000_pch_spt)
return -E1000_ERR_NVM;
else
ret_val = e1000_read_flash_data_ich8lan(hw, offset, 1, &word);
@@ -3634,8 +3639,7 @@
s32 ret_val = -E1000_ERR_NVM;
u8 count = 0;
- if (offset > ICH_FLASH_LINEAR_ADDR_MASK ||
- hw->mac.type != e1000_pch_spt)
+ if (offset > ICH_FLASH_LINEAR_ADDR_MASK || hw->mac.type < e1000_pch_spt)
return -E1000_ERR_NVM;
flash_linear_addr = ((ICH_FLASH_LINEAR_ADDR_MASK & offset) +
hw->nvm.flash_base_addr);
@@ -4068,6 +4072,7 @@
switch (hw->mac.type) {
case e1000_pch_lpt:
case e1000_pch_spt:
+ case e1000_pch_cnp:
word = NVM_COMPAT;
valid_csum_mask = NVM_COMPAT_VALID_CSUM;
break;
@@ -4153,7 +4158,7 @@
s32 ret_val;
u8 count = 0;
- if (hw->mac.type == e1000_pch_spt) {
+ if (hw->mac.type >= e1000_pch_spt) {
if (size != 4 || offset > ICH_FLASH_LINEAR_ADDR_MASK)
return -E1000_ERR_NVM;
} else {
@@ -4173,7 +4178,7 @@
/* In SPT, This register is in Lan memory space, not
* flash. Therefore, only 32 bit access is supported
*/
- if (hw->mac.type == e1000_pch_spt)
+ if (hw->mac.type >= e1000_pch_spt)
hsflctl.regval = er32flash(ICH_FLASH_HSFSTS) >> 16;
else
hsflctl.regval = er16flash(ICH_FLASH_HSFCTL);
@@ -4185,7 +4190,7 @@
* not flash. Therefore, only 32 bit access is
* supported
*/
- if (hw->mac.type == e1000_pch_spt)
+ if (hw->mac.type >= e1000_pch_spt)
ew32flash(ICH_FLASH_HSFSTS, hsflctl.regval << 16);
else
ew16flash(ICH_FLASH_HSFCTL, hsflctl.regval);
@@ -4243,7 +4248,7 @@
s32 ret_val;
u8 count = 0;
- if (hw->mac.type == e1000_pch_spt) {
+ if (hw->mac.type >= e1000_pch_spt) {
if (offset > ICH_FLASH_LINEAR_ADDR_MASK)
return -E1000_ERR_NVM;
}
@@ -4259,7 +4264,7 @@
/* In SPT, This register is in Lan memory space, not
* flash. Therefore, only 32 bit access is supported
*/
- if (hw->mac.type == e1000_pch_spt)
+ if (hw->mac.type >= e1000_pch_spt)
hsflctl.regval = er32flash(ICH_FLASH_HSFSTS)
>> 16;
else
@@ -4272,7 +4277,7 @@
* not flash. Therefore, only 32 bit access is
* supported
*/
- if (hw->mac.type == e1000_pch_spt)
+ if (hw->mac.type >= e1000_pch_spt)
ew32flash(ICH_FLASH_HSFSTS, hsflctl.regval << 16);
else
ew16flash(ICH_FLASH_HSFCTL, hsflctl.regval);
@@ -4464,14 +4469,14 @@
/* Write a value 11 (block Erase) in Flash
* Cycle field in hw flash control
*/
- if (hw->mac.type == e1000_pch_spt)
+ if (hw->mac.type >= e1000_pch_spt)
hsflctl.regval =
er32flash(ICH_FLASH_HSFSTS) >> 16;
else
hsflctl.regval = er16flash(ICH_FLASH_HSFCTL);
hsflctl.hsf_ctrl.flcycle = ICH_CYCLE_ERASE;
- if (hw->mac.type == e1000_pch_spt)
+ if (hw->mac.type >= e1000_pch_spt)
ew32flash(ICH_FLASH_HSFSTS,
hsflctl.regval << 16);
else
@@ -4894,8 +4899,7 @@
ew32(RFCTL, reg);
/* Enable ECC on Lynxpoint */
- if ((hw->mac.type == e1000_pch_lpt) ||
- (hw->mac.type == e1000_pch_spt)) {
+ if (hw->mac.type >= e1000_pch_lpt) {
reg = er32(PBECCSTS);
reg |= E1000_PBECCSTS_ECC_ENABLE;
ew32(PBECCSTS, reg);
@@ -5299,7 +5303,7 @@
(device_id == E1000_DEV_ID_PCH_LPTLP_I218_V) ||
(device_id == E1000_DEV_ID_PCH_I218_LM3) ||
(device_id == E1000_DEV_ID_PCH_I218_V3) ||
- (hw->mac.type == e1000_pch_spt)) {
+ (hw->mac.type >= e1000_pch_spt)) {
u32 fextnvm6 = er32(FEXTNVM6);
ew32(FEXTNVM6, fextnvm6 & ~E1000_FEXTNVM6_REQ_PLL_CLK);
@@ -5865,7 +5869,8 @@
| FLAG_HAS_JUMBO_FRAMES
| FLAG_APME_IN_WUC,
.flags2 = FLAG2_HAS_PHY_STATS
- | FLAG2_HAS_EEE,
+ | FLAG2_HAS_EEE
+ | FLAG2_CHECK_SYSTIM_OVERFLOW,
.pba = 26,
.max_hw_frame_size = 9022,
.get_variants = e1000_get_variants_ich8lan,
@@ -5914,3 +5919,23 @@
.phy_ops = &ich8_phy_ops,
.nvm_ops = &spt_nvm_ops,
};
+
+const struct e1000_info e1000_pch_cnp_info = {
+ .mac = e1000_pch_cnp,
+ .flags = FLAG_IS_ICH
+ | FLAG_HAS_WOL
+ | FLAG_HAS_HW_TIMESTAMP
+ | FLAG_HAS_CTRLEXT_ON_LOAD
+ | FLAG_HAS_AMT
+ | FLAG_HAS_FLASH
+ | FLAG_HAS_JUMBO_FRAMES
+ | FLAG_APME_IN_WUC,
+ .flags2 = FLAG2_HAS_PHY_STATS
+ | FLAG2_HAS_EEE,
+ .pba = 26,
+ .max_hw_frame_size = 9022,
+ .get_variants = e1000_get_variants_ich8lan,
+ .mac_ops = &ich8_mac_ops,
+ .phy_ops = &ich8_phy_ops,
+ .nvm_ops = &spt_nvm_ops,
+};
diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c b/drivers/net/ethernet/intel/e1000e/netdev.c
index e9af89a..b367972 100644
--- a/drivers/net/ethernet/intel/e1000e/netdev.c
+++ b/drivers/net/ethernet/intel/e1000e/netdev.c
@@ -71,6 +71,7 @@
[board_pch2lan] = &e1000_pch2_info,
[board_pch_lpt] = &e1000_pch_lpt_info,
[board_pch_spt] = &e1000_pch_spt_info,
+ [board_pch_cnp] = &e1000_pch_cnp_info,
};
struct e1000_reg_info {
@@ -1791,8 +1792,7 @@
}
/* Reset on uncorrectable ECC error */
- if ((icr & E1000_ICR_ECCER) && ((hw->mac.type == e1000_pch_lpt) ||
- (hw->mac.type == e1000_pch_spt))) {
+ if ((icr & E1000_ICR_ECCER) && (hw->mac.type >= e1000_pch_lpt)) {
u32 pbeccsts = er32(PBECCSTS);
adapter->corr_errors +=
@@ -1872,8 +1872,7 @@
}
/* Reset on uncorrectable ECC error */
- if ((icr & E1000_ICR_ECCER) && ((hw->mac.type == e1000_pch_lpt) ||
- (hw->mac.type == e1000_pch_spt))) {
+ if ((icr & E1000_ICR_ECCER) && (hw->mac.type >= e1000_pch_lpt)) {
u32 pbeccsts = er32(PBECCSTS);
adapter->corr_errors +=
@@ -2241,8 +2240,7 @@
if (adapter->msix_entries) {
ew32(EIAC_82574, adapter->eiac_mask & E1000_EIAC_MASK_82574);
ew32(IMS, adapter->eiac_mask | E1000_IMS_LSC);
- } else if ((hw->mac.type == e1000_pch_lpt) ||
- (hw->mac.type == e1000_pch_spt)) {
+ } else if (hw->mac.type >= e1000_pch_lpt) {
ew32(IMS, IMS_ENABLE_MASK | E1000_IMS_ECCER);
} else {
ew32(IMS, IMS_ENABLE_MASK);
@@ -3000,8 +2998,8 @@
hw->mac.ops.config_collision_dist(hw);
- /* SPT Si errata workaround to avoid data corruption */
- if (hw->mac.type == e1000_pch_spt) {
+ /* SPT and CNP Si errata workaround to avoid data corruption */
+ if (hw->mac.type >= e1000_pch_spt) {
u32 reg_val;
reg_val = er32(IOSFPC);
@@ -3497,8 +3495,7 @@
/* Make sure clock is enabled on I217/I218/I219 before checking
* the frequency
*/
- if (((hw->mac.type == e1000_pch_lpt) ||
- (hw->mac.type == e1000_pch_spt)) &&
+ if ((hw->mac.type >= e1000_pch_lpt) &&
!(er32(TSYNCTXCTL) & E1000_TSYNCTXCTL_ENABLED) &&
!(er32(TSYNCRXCTL) & E1000_TSYNCRXCTL_ENABLED)) {
u32 fextnvm7 = er32(FEXTNVM7);
@@ -3511,37 +3508,58 @@
switch (hw->mac.type) {
case e1000_pch2lan:
+ /* Stable 96MHz frequency */
+ incperiod = INCPERIOD_96MHZ;
+ incvalue = INCVALUE_96MHZ;
+ shift = INCVALUE_SHIFT_96MHZ;
+ adapter->cc.shift = shift + INCPERIOD_SHIFT_96MHZ;
+ break;
case e1000_pch_lpt:
if (er32(TSYNCRXCTL) & E1000_TSYNCRXCTL_SYSCFI) {
/* Stable 96MHz frequency */
- incperiod = INCPERIOD_96MHz;
- incvalue = INCVALUE_96MHz;
- shift = INCVALUE_SHIFT_96MHz;
- adapter->cc.shift = shift + INCPERIOD_SHIFT_96MHz;
+ incperiod = INCPERIOD_96MHZ;
+ incvalue = INCVALUE_96MHZ;
+ shift = INCVALUE_SHIFT_96MHZ;
+ adapter->cc.shift = shift + INCPERIOD_SHIFT_96MHZ;
} else {
/* Stable 25MHz frequency */
- incperiod = INCPERIOD_25MHz;
- incvalue = INCVALUE_25MHz;
- shift = INCVALUE_SHIFT_25MHz;
+ incperiod = INCPERIOD_25MHZ;
+ incvalue = INCVALUE_25MHZ;
+ shift = INCVALUE_SHIFT_25MHZ;
adapter->cc.shift = shift;
}
break;
case e1000_pch_spt:
if (er32(TSYNCRXCTL) & E1000_TSYNCRXCTL_SYSCFI) {
/* Stable 24MHz frequency */
- incperiod = INCPERIOD_24MHz;
- incvalue = INCVALUE_24MHz;
- shift = INCVALUE_SHIFT_24MHz;
+ incperiod = INCPERIOD_24MHZ;
+ incvalue = INCVALUE_24MHZ;
+ shift = INCVALUE_SHIFT_24MHZ;
adapter->cc.shift = shift;
break;
}
return -EINVAL;
+ case e1000_pch_cnp:
+ if (er32(TSYNCRXCTL) & E1000_TSYNCRXCTL_SYSCFI) {
+ /* Stable 24MHz frequency */
+ incperiod = INCPERIOD_24MHZ;
+ incvalue = INCVALUE_24MHZ;
+ shift = INCVALUE_SHIFT_24MHZ;
+ adapter->cc.shift = shift;
+ } else {
+ /* Stable 38400KHz frequency */
+ incperiod = INCPERIOD_38400KHZ;
+ incvalue = INCVALUE_38400KHZ;
+ shift = INCVALUE_SHIFT_38400KHZ;
+ adapter->cc.shift = shift;
+ }
+ break;
case e1000_82574:
case e1000_82583:
/* Stable 25MHz frequency */
- incperiod = INCPERIOD_25MHz;
- incvalue = INCVALUE_25MHz;
- shift = INCVALUE_SHIFT_25MHz;
+ incperiod = INCPERIOD_25MHZ;
+ incvalue = INCVALUE_25MHZ;
+ shift = INCVALUE_SHIFT_25MHZ;
adapter->cc.shift = shift;
break;
default:
@@ -4032,6 +4050,7 @@
case e1000_pch2lan:
case e1000_pch_lpt:
case e1000_pch_spt:
+ case e1000_pch_cnp:
fc->refresh_time = 0x0400;
if (adapter->netdev->mtu <= ETH_DATA_LEN) {
@@ -4076,7 +4095,7 @@
}
}
- if (hw->mac.type == e1000_pch_spt)
+ if (hw->mac.type >= e1000_pch_spt)
e1000_flush_desc_rings(adapter);
/* Allow time for pending master requests to run */
mac->ops.reset_hw(hw);
@@ -4151,7 +4170,7 @@
phy_data &= ~IGP02E1000_PM_SPD;
e1e_wphy(hw, IGP02E1000_PHY_POWER_MGMT, phy_data);
}
- if (hw->mac.type == e1000_pch_spt && adapter->int_mode == 0) {
+ if (hw->mac.type >= e1000_pch_spt && adapter->int_mode == 0) {
u32 reg;
/* Fextnvm7 @ 0xe4[2] = 1 */
@@ -4285,7 +4304,7 @@
if (!pci_channel_offline(adapter->pdev)) {
if (reset)
e1000e_reset(adapter);
- else if (hw->mac.type == e1000_pch_spt)
+ else if (hw->mac.type >= e1000_pch_spt)
e1000_flush_desc_rings(adapter);
}
e1000_clean_tx_ring(adapter->tx_ring);
@@ -4973,8 +4992,7 @@
adapter->stats.mgpdc += er32(MGTPDC);
/* Correctable ECC Errors */
- if ((hw->mac.type == e1000_pch_lpt) ||
- (hw->mac.type == e1000_pch_spt)) {
+ if (hw->mac.type >= e1000_pch_lpt) {
u32 pbeccsts = er32(PBECCSTS);
adapter->corr_errors +=
@@ -6348,8 +6366,7 @@
if (adapter->hw.phy.type == e1000_phy_igp_3) {
e1000e_igp3_phy_powerdown_workaround_ich8lan(&adapter->hw);
- } else if ((hw->mac.type == e1000_pch_lpt) ||
- (hw->mac.type == e1000_pch_spt)) {
+ } else if (hw->mac.type >= e1000_pch_lpt) {
if (!(wufc & (E1000_WUFC_EX | E1000_WUFC_MC | E1000_WUFC_BC)))
/* ULP does not support wake from unicast, multicast
* or broadcast.
@@ -7508,6 +7525,10 @@
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_SPT_I219_V4), board_pch_spt },
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_SPT_I219_LM5), board_pch_spt },
{ PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_SPT_I219_V5), board_pch_spt },
+ { PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_CNP_I219_LM6), board_pch_cnp },
+ { PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_CNP_I219_V6), board_pch_cnp },
+ { PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_CNP_I219_LM7), board_pch_cnp },
+ { PCI_VDEVICE(INTEL, E1000_DEV_ID_PCH_CNP_I219_V7), board_pch_cnp },
{ 0, 0, 0, 0, 0, 0, 0 } /* terminate list */
};
diff --git a/drivers/net/ethernet/intel/e1000e/ptp.c b/drivers/net/ethernet/intel/e1000e/ptp.c
index 34cc3be..b366885 100644
--- a/drivers/net/ethernet/intel/e1000e/ptp.c
+++ b/drivers/net/ethernet/intel/e1000e/ptp.c
@@ -301,8 +301,8 @@
case e1000_pch2lan:
case e1000_pch_lpt:
case e1000_pch_spt:
- if (((hw->mac.type != e1000_pch_lpt) &&
- (hw->mac.type != e1000_pch_spt)) ||
+ case e1000_pch_cnp:
+ if ((hw->mac.type < e1000_pch_lpt) ||
(er32(TSYNCRXCTL) & E1000_TSYNCRXCTL_SYSCFI)) {
adapter->ptp_clock_info.max_adj = 24000000 - 1;
break;
diff --git a/drivers/net/ethernet/intel/fm10k/fm10k.h b/drivers/net/ethernet/intel/fm10k/fm10k.h
index 52b9794..689c413 100644
--- a/drivers/net/ethernet/intel/fm10k/fm10k.h
+++ b/drivers/net/ethernet/intel/fm10k/fm10k.h
@@ -1,5 +1,5 @@
/* Intel(R) Ethernet Switch Host Interface Driver
- * Copyright(c) 2013 - 2016 Intel Corporation.
+ * Copyright(c) 2013 - 2017 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
@@ -65,14 +65,16 @@
__FM10K_TX_DETECT_HANG,
__FM10K_HANG_CHECK_ARMED,
__FM10K_TX_XPS_INIT_DONE,
+ /* This must be last and is used to calculate BITMAP size */
+ __FM10K_TX_STATE_SIZE__,
};
#define check_for_tx_hang(ring) \
- test_bit(__FM10K_TX_DETECT_HANG, &(ring)->state)
+ test_bit(__FM10K_TX_DETECT_HANG, (ring)->state)
#define set_check_for_tx_hang(ring) \
- set_bit(__FM10K_TX_DETECT_HANG, &(ring)->state)
+ set_bit(__FM10K_TX_DETECT_HANG, (ring)->state)
#define clear_check_for_tx_hang(ring) \
- clear_bit(__FM10K_TX_DETECT_HANG, &(ring)->state)
+ clear_bit(__FM10K_TX_DETECT_HANG, (ring)->state)
struct fm10k_tx_buffer {
struct fm10k_tx_desc *next_to_watch;
@@ -126,7 +128,7 @@
struct fm10k_rx_buffer *rx_buffer;
};
u32 __iomem *tail;
- unsigned long state;
+ DECLARE_BITMAP(state, __FM10K_TX_STATE_SIZE__);
dma_addr_t dma; /* phys. address of descriptor ring */
unsigned int size; /* length in bytes */
@@ -249,18 +251,46 @@
/* one work queue for entire driver */
extern struct workqueue_struct *fm10k_workqueue;
+/* The following enumeration contains flags which indicate or enable modified
+ * driver behaviors. To avoid race conditions, the flags are stored in
+ * a BITMAP in the fm10k_intfc structure. The BITMAP should be accessed using
+ * atomic *_bit() operations.
+ */
+enum fm10k_flags_t {
+ FM10K_FLAG_RESET_REQUESTED,
+ FM10K_FLAG_RSS_FIELD_IPV4_UDP,
+ FM10K_FLAG_RSS_FIELD_IPV6_UDP,
+ FM10K_FLAG_SWPRI_CONFIG,
+ /* __FM10K_FLAGS_SIZE__ is used to calculate the size of
+ * interface->flags and must be the last value in this
+ * enumeration.
+ */
+ __FM10K_FLAGS_SIZE__
+};
+
+enum fm10k_state_t {
+ __FM10K_RESETTING,
+ __FM10K_DOWN,
+ __FM10K_SERVICE_SCHED,
+ __FM10K_SERVICE_REQUEST,
+ __FM10K_SERVICE_DISABLE,
+ __FM10K_MBX_LOCK,
+ __FM10K_LINK_DOWN,
+ __FM10K_UPDATING_STATS,
+ /* This value must be last and determines the BITMAP size */
+ __FM10K_STATE_SIZE__,
+};
+
struct fm10k_intfc {
unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)];
struct net_device *netdev;
struct fm10k_l2_accel *l2_accel; /* pointer to L2 acceleration list */
struct pci_dev *pdev;
- unsigned long state;
+ DECLARE_BITMAP(state, __FM10K_STATE_SIZE__);
- u32 flags;
-#define FM10K_FLAG_RESET_REQUESTED (u32)(BIT(0))
-#define FM10K_FLAG_RSS_FIELD_IPV4_UDP (u32)(BIT(1))
-#define FM10K_FLAG_RSS_FIELD_IPV6_UDP (u32)(BIT(2))
-#define FM10K_FLAG_SWPRI_CONFIG (u32)(BIT(3))
+ /* Access flag values using atomic *_bit() operations */
+ DECLARE_BITMAP(flags, __FM10K_FLAGS_SIZE__);
+
int xcast_mode;
/* Tx fast path data */
@@ -352,22 +382,12 @@
u16 vid;
};
-enum fm10k_state_t {
- __FM10K_RESETTING,
- __FM10K_DOWN,
- __FM10K_SERVICE_SCHED,
- __FM10K_SERVICE_DISABLE,
- __FM10K_MBX_LOCK,
- __FM10K_LINK_DOWN,
- __FM10K_UPDATING_STATS,
-};
-
static inline void fm10k_mbx_lock(struct fm10k_intfc *interface)
{
/* busy loop if we cannot obtain the lock as some calls
* such as ndo_set_rx_mode may be made in atomic context
*/
- while (test_and_set_bit(__FM10K_MBX_LOCK, &interface->state))
+ while (test_and_set_bit(__FM10K_MBX_LOCK, interface->state))
udelay(20);
}
@@ -375,12 +395,12 @@
{
/* flush memory to make sure state is correct */
smp_mb__before_atomic();
- clear_bit(__FM10K_MBX_LOCK, &interface->state);
+ clear_bit(__FM10K_MBX_LOCK, interface->state);
}
static inline int fm10k_mbx_trylock(struct fm10k_intfc *interface)
{
- return !test_and_set_bit(__FM10K_MBX_LOCK, &interface->state);
+ return !test_and_set_bit(__FM10K_MBX_LOCK, interface->state);
}
/* fm10k_test_staterr - test bits in Rx descriptor status and error fields */
diff --git a/drivers/net/ethernet/intel/fm10k/fm10k_ethtool.c b/drivers/net/ethernet/intel/fm10k/fm10k_ethtool.c
index 0c84fef..c7234f3 100644
--- a/drivers/net/ethernet/intel/fm10k/fm10k_ethtool.c
+++ b/drivers/net/ethernet/intel/fm10k/fm10k_ethtool.c
@@ -1,5 +1,5 @@
/* Intel(R) Ethernet Switch Host Interface Driver
- * Copyright(c) 2013 - 2016 Intel Corporation.
+ * Copyright(c) 2013 - 2017 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
@@ -562,7 +562,7 @@
return 0;
}
- while (test_and_set_bit(__FM10K_RESETTING, &interface->state))
+ while (test_and_set_bit(__FM10K_RESETTING, interface->state))
usleep_range(1000, 2000);
if (!netif_running(interface->netdev)) {
@@ -648,7 +648,7 @@
fm10k_up(interface);
vfree(temp_ring);
clear_reset:
- clear_bit(__FM10K_RESETTING, &interface->state);
+ clear_bit(__FM10K_RESETTING, interface->state);
return err;
}
@@ -716,7 +716,8 @@
cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3;
/* fall through */
case UDP_V4_FLOW:
- if (interface->flags & FM10K_FLAG_RSS_FIELD_IPV4_UDP)
+ if (test_bit(FM10K_FLAG_RSS_FIELD_IPV4_UDP,
+ interface->flags))
cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3;
/* fall through */
case SCTP_V4_FLOW:
@@ -732,7 +733,8 @@
cmd->data |= RXH_IP_SRC | RXH_IP_DST;
break;
case UDP_V6_FLOW:
- if (interface->flags & FM10K_FLAG_RSS_FIELD_IPV6_UDP)
+ if (test_bit(FM10K_FLAG_RSS_FIELD_IPV6_UDP,
+ interface->flags))
cmd->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3;
cmd->data |= RXH_IP_SRC | RXH_IP_DST;
break;
@@ -764,12 +766,13 @@
return ret;
}
-#define UDP_RSS_FLAGS (FM10K_FLAG_RSS_FIELD_IPV4_UDP | \
- FM10K_FLAG_RSS_FIELD_IPV6_UDP)
static int fm10k_set_rss_hash_opt(struct fm10k_intfc *interface,
struct ethtool_rxnfc *nfc)
{
- u32 flags = interface->flags;
+ int rss_ipv4_udp = test_bit(FM10K_FLAG_RSS_FIELD_IPV4_UDP,
+ interface->flags);
+ int rss_ipv6_udp = test_bit(FM10K_FLAG_RSS_FIELD_IPV6_UDP,
+ interface->flags);
/* RSS does not support anything other than hashing
* to queues on src and dst IPs and ports
@@ -793,10 +796,12 @@
return -EINVAL;
switch (nfc->data & (RXH_L4_B_0_1 | RXH_L4_B_2_3)) {
case 0:
- flags &= ~FM10K_FLAG_RSS_FIELD_IPV4_UDP;
+ clear_bit(FM10K_FLAG_RSS_FIELD_IPV4_UDP,
+ interface->flags);
break;
case (RXH_L4_B_0_1 | RXH_L4_B_2_3):
- flags |= FM10K_FLAG_RSS_FIELD_IPV4_UDP;
+ set_bit(FM10K_FLAG_RSS_FIELD_IPV4_UDP,
+ interface->flags);
break;
default:
return -EINVAL;
@@ -808,10 +813,12 @@
return -EINVAL;
switch (nfc->data & (RXH_L4_B_0_1 | RXH_L4_B_2_3)) {
case 0:
- flags &= ~FM10K_FLAG_RSS_FIELD_IPV6_UDP;
+ clear_bit(FM10K_FLAG_RSS_FIELD_IPV6_UDP,
+ interface->flags);
break;
case (RXH_L4_B_0_1 | RXH_L4_B_2_3):
- flags |= FM10K_FLAG_RSS_FIELD_IPV6_UDP;
+ set_bit(FM10K_FLAG_RSS_FIELD_IPV6_UDP,
+ interface->flags);
break;
default:
return -EINVAL;
@@ -835,28 +842,41 @@
return -EINVAL;
}
- /* if we changed something we need to update flags */
- if (flags != interface->flags) {
+ /* If something changed we need to update the MRQC register. Note that
+ * test_bit() is guaranteed to return strictly 0 or 1, so testing for
+ * equality is safe.
+ */
+ if ((rss_ipv4_udp != test_bit(FM10K_FLAG_RSS_FIELD_IPV4_UDP,
+ interface->flags)) ||
+ (rss_ipv6_udp != test_bit(FM10K_FLAG_RSS_FIELD_IPV6_UDP,
+ interface->flags))) {
struct fm10k_hw *hw = &interface->hw;
+ bool warn = false;
u32 mrqc;
- if ((flags & UDP_RSS_FLAGS) &&
- !(interface->flags & UDP_RSS_FLAGS))
- netif_warn(interface, drv, interface->netdev,
- "enabling UDP RSS: fragmented packets may arrive out of order to the stack above\n");
-
- interface->flags = flags;
-
/* Perform hash on these packet types */
mrqc = FM10K_MRQC_IPV4 |
FM10K_MRQC_TCP_IPV4 |
FM10K_MRQC_IPV6 |
FM10K_MRQC_TCP_IPV6;
- if (flags & FM10K_FLAG_RSS_FIELD_IPV4_UDP)
+ if (test_bit(FM10K_FLAG_RSS_FIELD_IPV4_UDP,
+ interface->flags)) {
mrqc |= FM10K_MRQC_UDP_IPV4;
- if (flags & FM10K_FLAG_RSS_FIELD_IPV6_UDP)
+ warn = true;
+ }
+ if (test_bit(FM10K_FLAG_RSS_FIELD_IPV6_UDP,
+ interface->flags)) {
mrqc |= FM10K_MRQC_UDP_IPV6;
+ warn = true;
+ }
+
+ /* If we enable UDP RSS display a warning that this may cause
+ * fragmented UDP packets to arrive out of order.
+ */
+ if (warn)
+ netif_warn(interface, drv, interface->netdev,
+ "enabling UDP RSS: fragmented packets may arrive out of order to the stack above\n");
fm10k_write_reg(hw, FM10K_MRQC(0), mrqc);
}
@@ -939,7 +959,7 @@
memset(data, 0, sizeof(*data) * FM10K_TEST_LEN);
- if (FM10K_REMOVED(hw)) {
+ if (FM10K_REMOVED(hw->hw_addr)) {
netif_err(interface, drv, dev,
"Interface removed - test blocked\n");
eth_test->flags |= ETH_TEST_FL_FAILED;
diff --git a/drivers/net/ethernet/intel/fm10k/fm10k_main.c b/drivers/net/ethernet/intel/fm10k/fm10k_main.c
index 5bb233a..9dffaba 100644
--- a/drivers/net/ethernet/intel/fm10k/fm10k_main.c
+++ b/drivers/net/ethernet/intel/fm10k/fm10k_main.c
@@ -1,5 +1,5 @@
/* Intel(R) Ethernet Switch Host Interface Driver
- * Copyright(c) 2013 - 2016 Intel Corporation.
+ * Copyright(c) 2013 - 2017 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
@@ -34,7 +34,7 @@
char fm10k_driver_name[] = "fm10k";
static const char fm10k_driver_string[] = DRV_SUMMARY;
static const char fm10k_copyright[] =
- "Copyright (c) 2013 - 2016 Intel Corporation.";
+ "Copyright(c) 2013 - 2017 Intel Corporation.";
MODULE_AUTHOR("Intel Corporation, <linux.nics@intel.com>");
MODULE_DESCRIPTION(DRV_SUMMARY);
@@ -1175,13 +1175,13 @@
/* update completed stats and continue */
tx_ring->tx_stats.tx_done_old = tx_done;
/* reset the countdown */
- clear_bit(__FM10K_HANG_CHECK_ARMED, &tx_ring->state);
+ clear_bit(__FM10K_HANG_CHECK_ARMED, tx_ring->state);
return false;
}
/* make sure it is true for two checks in a row */
- return test_and_set_bit(__FM10K_HANG_CHECK_ARMED, &tx_ring->state);
+ return test_and_set_bit(__FM10K_HANG_CHECK_ARMED, tx_ring->state);
}
/**
@@ -1191,9 +1191,9 @@
void fm10k_tx_timeout_reset(struct fm10k_intfc *interface)
{
/* Do the reset outside of interrupt context */
- if (!test_bit(__FM10K_DOWN, &interface->state)) {
+ if (!test_bit(__FM10K_DOWN, interface->state)) {
interface->tx_timeout_count++;
- interface->flags |= FM10K_FLAG_RESET_REQUESTED;
+ set_bit(FM10K_FLAG_RESET_REQUESTED, interface->flags);
fm10k_service_event_schedule(interface);
}
}
@@ -1214,7 +1214,7 @@
unsigned int budget = q_vector->tx.work_limit;
unsigned int i = tx_ring->next_to_clean;
- if (test_bit(__FM10K_DOWN, &interface->state))
+ if (test_bit(__FM10K_DOWN, interface->state))
return true;
tx_buffer = &tx_ring->tx_buffer[i];
@@ -1344,7 +1344,7 @@
smp_mb();
if (__netif_subqueue_stopped(tx_ring->netdev,
tx_ring->queue_index) &&
- !test_bit(__FM10K_DOWN, &interface->state)) {
+ !test_bit(__FM10K_DOWN, interface->state)) {
netif_wake_subqueue(tx_ring->netdev,
tx_ring->queue_index);
++tx_ring->tx_stats.restart_queue;
diff --git a/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c b/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c
index 01db688..24f2f6f 100644
--- a/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c
+++ b/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c
@@ -1,5 +1,5 @@
/* Intel(R) Ethernet Switch Host Interface Driver
- * Copyright(c) 2013 - 2016 Intel Corporation.
+ * Copyright(c) 2013 - 2017 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
@@ -737,6 +737,23 @@
}
}
+/**
+ * fm10k_host_mbx_ready - Check PF interface's mailbox readiness
+ * @interface: board private structure
+ *
+ * This function checks if the PF interface's mailbox is ready before queueing
+ * mailbox messages for transmission. This will prevent filling the TX mailbox
+ * queue when the receiver is not ready. VF interfaces are exempt from this
+ * check since it will block all PF-VF mailbox messages from being sent from
+ * the VF to the PF at initialization.
+ **/
+static bool fm10k_host_mbx_ready(struct fm10k_intfc *interface)
+{
+ struct fm10k_hw *hw = &interface->hw;
+
+ return (hw->mac.type == fm10k_mac_vf || interface->host_ready);
+}
+
static int fm10k_uc_vlan_unsync(struct net_device *netdev,
const unsigned char *uc_addr)
{
@@ -745,12 +762,15 @@
u16 glort = interface->glort;
u16 vid = interface->vid;
bool set = !!(vid / VLAN_N_VID);
- int err;
+ int err = -EHOSTDOWN;
/* drop any leading bits on the VLAN ID */
vid &= VLAN_N_VID - 1;
- err = hw->mac.ops.update_uc_addr(hw, glort, uc_addr, vid, set, 0);
+ if (fm10k_host_mbx_ready(interface))
+ err = hw->mac.ops.update_uc_addr(hw, glort, uc_addr,
+ vid, set, 0);
+
if (err)
return err;
@@ -766,12 +786,14 @@
u16 glort = interface->glort;
u16 vid = interface->vid;
bool set = !!(vid / VLAN_N_VID);
- int err;
+ int err = -EHOSTDOWN;
/* drop any leading bits on the VLAN ID */
vid &= VLAN_N_VID - 1;
- err = hw->mac.ops.update_mc_addr(hw, glort, mc_addr, vid, set);
+ if (fm10k_host_mbx_ready(interface))
+ err = hw->mac.ops.update_mc_addr(hw, glort, mc_addr, vid, set);
+
if (err)
return err;
@@ -822,7 +844,7 @@
/* Do not throw an error if the interface is down. We will sync once
* we come up
*/
- if (test_bit(__FM10K_DOWN, &interface->state))
+ if (test_bit(__FM10K_DOWN, interface->state))
return 0;
fm10k_mbx_lock(interface);
@@ -834,9 +856,13 @@
goto err_out;
}
- /* update our base MAC address */
- err = hw->mac.ops.update_uc_addr(hw, interface->glort, hw->mac.addr,
- vid, set, 0);
+ /* update our base MAC address if host's mailbox is ready */
+ if (fm10k_host_mbx_ready(interface))
+ err = hw->mac.ops.update_uc_addr(hw, interface->glort,
+ hw->mac.addr, vid, set, 0);
+ else
+ err = -EHOSTDOWN;
+
if (err)
goto err_out;
@@ -907,12 +933,15 @@
if (!is_valid_ether_addr(addr))
return -EADDRNOTAVAIL;
- /* update table with current entries */
+ /* update table with current entries if host's mailbox is ready */
+ if (!fm10k_host_mbx_ready(interface))
+ return -EHOSTDOWN;
+
for (vid = hw->mac.default_vid ? fm10k_find_next_vlan(interface, 0) : 1;
vid < VLAN_N_VID;
vid = fm10k_find_next_vlan(interface, vid)) {
err = hw->mac.ops.update_uc_addr(hw, glort, addr,
- vid, sync, 0);
+ vid, sync, 0);
if (err)
return err;
}
@@ -970,7 +999,10 @@
struct fm10k_hw *hw = &interface->hw;
u16 vid, glort = interface->glort;
- /* update table with current entries */
+ /* update table with current entries if host's mailbox is ready */
+ if (!fm10k_host_mbx_ready(interface))
+ return 0;
+
for (vid = hw->mac.default_vid ? fm10k_find_next_vlan(interface, 0) : 1;
vid < VLAN_N_VID;
vid = fm10k_find_next_vlan(interface, vid)) {
@@ -1018,8 +1050,10 @@
if (interface->xcast_mode == FM10K_XCAST_MODE_PROMISC)
fm10k_clear_unused_vlans(interface);
- /* update xcast mode */
- hw->mac.ops.update_xcast_mode(hw, interface->glort, xcast_mode);
+ /* update xcast mode if host's mailbox is ready */
+ if (fm10k_host_mbx_ready(interface))
+ hw->mac.ops.update_xcast_mode(hw, interface->glort,
+ xcast_mode);
/* record updated xcast mode state */
interface->xcast_mode = xcast_mode;
@@ -1054,8 +1088,10 @@
fm10k_mbx_lock(interface);
- /* Enable logical port */
- hw->mac.ops.update_lport_state(hw, glort, interface->glort_count, true);
+ /* Enable logical port if host's mailbox is ready */
+ if (fm10k_host_mbx_ready(interface))
+ hw->mac.ops.update_lport_state(hw, glort,
+ interface->glort_count, true);
/* update VLAN table */
hw->mac.ops.update_vlan(hw, FM10K_VLAN_ALL, 0,
@@ -1069,12 +1105,18 @@
vid < VLAN_N_VID;
vid = fm10k_find_next_vlan(interface, vid)) {
hw->mac.ops.update_vlan(hw, vid, 0, true);
- hw->mac.ops.update_uc_addr(hw, glort, hw->mac.addr,
- vid, true, 0);
+
+ /* Update unicast entries if host's mailbox is ready */
+ if (fm10k_host_mbx_ready(interface))
+ hw->mac.ops.update_uc_addr(hw, glort, hw->mac.addr,
+ vid, true, 0);
}
- /* update xcast mode before synchronizing addresses */
- hw->mac.ops.update_xcast_mode(hw, glort, xcast_mode);
+ /* update xcast mode before synchronizing addresses if host's mailbox
+ * is ready
+ */
+ if (fm10k_host_mbx_ready(interface))
+ hw->mac.ops.update_xcast_mode(hw, glort, xcast_mode);
/* synchronize all of the addresses */
__dev_uc_sync(netdev, fm10k_uc_sync, fm10k_uc_unsync);
@@ -1096,9 +1138,12 @@
fm10k_mbx_lock(interface);
- /* clear the logical port state on lower device */
- hw->mac.ops.update_lport_state(hw, interface->glort,
- interface->glort_count, false);
+ /* clear the logical port state on lower device if host's mailbox is
+ * ready
+ */
+ if (fm10k_host_mbx_ready(interface))
+ hw->mac.ops.update_lport_state(hw, interface->glort,
+ interface->glort_count, false);
fm10k_mbx_unlock(interface);
@@ -1115,8 +1160,8 @@
* @netdev: network interface device structure
* @stats: storage space for 64bit statistics
*
- * Returns 64bit statistics, for use in the ndo_get_stats64 callback. This
- * function replaces fm10k_get_stats for kernels which support it.
+ * Obtain 64bit statistics in a way that is safe for both 32bit and 64bit
+ * architectures.
*/
static void fm10k_get_stats64(struct net_device *netdev,
struct rtnl_link_stats64 *stats)
@@ -1207,7 +1252,7 @@
goto err_open;
/* flag to indicate SWPRI has yet to be updated */
- interface->flags |= FM10K_FLAG_SWPRI_CONFIG;
+ set_bit(FM10K_FLAG_SWPRI_CONFIG, interface->flags);
return 0;
err_open:
@@ -1226,7 +1271,9 @@
if (tc->type != TC_SETUP_MQPRIO)
return -EINVAL;
- return fm10k_setup_tc(dev, tc->tc);
+ tc->mqprio->hw = TC_MQPRIO_HW_OFFLOAD_TCS;
+
+ return fm10k_setup_tc(dev, tc->mqprio->num_tc);
}
static void fm10k_assign_l2_accel(struct fm10k_intfc *interface,
@@ -1317,8 +1364,13 @@
fm10k_mbx_lock(interface);
glort = l2_accel->dglort + 1 + i;
- hw->mac.ops.update_xcast_mode(hw, glort, FM10K_XCAST_MODE_MULTI);
- hw->mac.ops.update_uc_addr(hw, glort, sdev->dev_addr, 0, true, 0);
+
+ if (fm10k_host_mbx_ready(interface)) {
+ hw->mac.ops.update_xcast_mode(hw, glort,
+ FM10K_XCAST_MODE_MULTI);
+ hw->mac.ops.update_uc_addr(hw, glort, sdev->dev_addr,
+ 0, true, 0);
+ }
fm10k_mbx_unlock(interface);
@@ -1352,8 +1404,13 @@
fm10k_mbx_lock(interface);
glort = l2_accel->dglort + 1 + i;
- hw->mac.ops.update_xcast_mode(hw, glort, FM10K_XCAST_MODE_NONE);
- hw->mac.ops.update_uc_addr(hw, glort, sdev->dev_addr, 0, false, 0);
+
+ if (fm10k_host_mbx_ready(interface)) {
+ hw->mac.ops.update_xcast_mode(hw, glort,
+ FM10K_XCAST_MODE_NONE);
+ hw->mac.ops.update_uc_addr(hw, glort, sdev->dev_addr,
+ 0, false, 0);
+ }
fm10k_mbx_unlock(interface);
diff --git a/drivers/net/ethernet/intel/fm10k/fm10k_pci.c b/drivers/net/ethernet/intel/fm10k/fm10k_pci.c
index e372a58..3e26d27 100644
--- a/drivers/net/ethernet/intel/fm10k/fm10k_pci.c
+++ b/drivers/net/ethernet/intel/fm10k/fm10k_pci.c
@@ -1,5 +1,5 @@
/* Intel(R) Ethernet Switch Host Interface Driver
- * Copyright(c) 2013 - 2016 Intel Corporation.
+ * Copyright(c) 2013 - 2017 Intel Corporation.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
@@ -19,6 +19,7 @@
*/
#include <linux/module.h>
+#include <linux/interrupt.h>
#include <linux/aer.h>
#include "fm10k.h"
@@ -92,18 +93,29 @@
void fm10k_service_event_schedule(struct fm10k_intfc *interface)
{
- if (!test_bit(__FM10K_SERVICE_DISABLE, &interface->state) &&
- !test_and_set_bit(__FM10K_SERVICE_SCHED, &interface->state))
+ if (!test_bit(__FM10K_SERVICE_DISABLE, interface->state) &&
+ !test_and_set_bit(__FM10K_SERVICE_SCHED, interface->state)) {
+ clear_bit(__FM10K_SERVICE_REQUEST, interface->state);
queue_work(fm10k_workqueue, &interface->service_task);
+ } else {
+ set_bit(__FM10K_SERVICE_REQUEST, interface->state);
+ }
}
static void fm10k_service_event_complete(struct fm10k_intfc *interface)
{
- WARN_ON(!test_bit(__FM10K_SERVICE_SCHED, &interface->state));
+ WARN_ON(!test_bit(__FM10K_SERVICE_SCHED, interface->state));
/* flush memory to make sure state is correct before next watchog */
smp_mb__before_atomic();
- clear_bit(__FM10K_SERVICE_SCHED, &interface->state);
+ clear_bit(__FM10K_SERVICE_SCHED, interface->state);
+
+ /* If a service event was requested since we started, immediately
+ * re-schedule now. This ensures we don't drop a request until the
+ * next timer event.
+ */
+ if (test_bit(__FM10K_SERVICE_REQUEST, interface->state))
+ fm10k_service_event_schedule(interface);
}
/**
@@ -136,7 +148,7 @@
if (~value) {
interface->hw.hw_addr = interface->uc_addr;
netif_device_attach(netdev);
- interface->flags |= FM10K_FLAG_RESET_REQUESTED;
+ set_bit(FM10K_FLAG_RESET_REQUESTED, interface->flags);
netdev_warn(netdev, "PCIe link restored, device now attached\n");
return;
}
@@ -158,7 +170,7 @@
/* put off any impending NetWatchDogTimeout */
netif_trans_update(netdev);
- while (test_and_set_bit(__FM10K_RESETTING, &interface->state))
+ while (test_and_set_bit(__FM10K_RESETTING, interface->state))
usleep_range(1000, 2000);
rtnl_lock();
@@ -241,7 +253,7 @@
rtnl_unlock();
- clear_bit(__FM10K_RESETTING, &interface->state);
+ clear_bit(__FM10K_RESETTING, interface->state);
return err;
err_open:
@@ -253,7 +265,7 @@
rtnl_unlock();
- clear_bit(__FM10K_RESETTING, &interface->state);
+ clear_bit(__FM10K_RESETTING, interface->state);
return err;
}
@@ -272,11 +284,10 @@
static void fm10k_reset_subtask(struct fm10k_intfc *interface)
{
- if (!(interface->flags & FM10K_FLAG_RESET_REQUESTED))
+ if (!test_and_clear_bit(FM10K_FLAG_RESET_REQUESTED,
+ interface->flags))
return;
- interface->flags &= ~FM10K_FLAG_RESET_REQUESTED;
-
netdev_err(interface->netdev, "Reset interface\n");
fm10k_reinit(interface);
@@ -295,7 +306,7 @@
int i;
/* clear flag indicating update is needed */
- interface->flags &= ~FM10K_FLAG_SWPRI_CONFIG;
+ clear_bit(FM10K_FLAG_SWPRI_CONFIG, interface->flags);
/* these registers are only available on the PF */
if (hw->mac.type != fm10k_mac_pf)
@@ -316,14 +327,14 @@
struct fm10k_hw *hw = &interface->hw;
s32 err;
- if (test_bit(__FM10K_LINK_DOWN, &interface->state)) {
+ if (test_bit(__FM10K_LINK_DOWN, interface->state)) {
interface->host_ready = false;
if (time_is_after_jiffies(interface->link_down_event))
return;
- clear_bit(__FM10K_LINK_DOWN, &interface->state);
+ clear_bit(__FM10K_LINK_DOWN, interface->state);
}
- if (interface->flags & FM10K_FLAG_SWPRI_CONFIG) {
+ if (test_bit(FM10K_FLAG_SWPRI_CONFIG, interface->flags)) {
if (rtnl_trylock()) {
fm10k_configure_swpri_map(interface);
rtnl_unlock();
@@ -335,7 +346,7 @@
err = hw->mac.ops.get_host_state(hw, &interface->host_ready);
if (err && time_is_before_jiffies(interface->last_reset))
- interface->flags |= FM10K_FLAG_RESET_REQUESTED;
+ set_bit(FM10K_FLAG_RESET_REQUESTED, interface->flags);
/* free the lock */
fm10k_mbx_unlock(interface);
@@ -411,7 +422,7 @@
int i;
/* ensure only one thread updates stats at a time */
- if (test_and_set_bit(__FM10K_UPDATING_STATS, &interface->state))
+ if (test_and_set_bit(__FM10K_UPDATING_STATS, interface->state))
return;
/* do not allow stats update via service task for next second */
@@ -492,7 +503,7 @@
net_stats->rx_errors = rx_errors;
net_stats->rx_dropped = interface->stats.nodesc_drop.count;
- clear_bit(__FM10K_UPDATING_STATS, &interface->state);
+ clear_bit(__FM10K_UPDATING_STATS, interface->state);
}
/**
@@ -522,7 +533,7 @@
* controller to flush Tx.
*/
if (some_tx_pending)
- interface->flags |= FM10K_FLAG_RESET_REQUESTED;
+ set_bit(FM10K_FLAG_RESET_REQUESTED, interface->flags);
}
/**
@@ -532,8 +543,8 @@
static void fm10k_watchdog_subtask(struct fm10k_intfc *interface)
{
/* if interface is down do nothing */
- if (test_bit(__FM10K_DOWN, &interface->state) ||
- test_bit(__FM10K_RESETTING, &interface->state))
+ if (test_bit(__FM10K_DOWN, interface->state) ||
+ test_bit(__FM10K_RESETTING, interface->state))
return;
if (interface->host_ready)
@@ -563,8 +574,8 @@
int i;
/* If we're down or resetting, just bail */
- if (test_bit(__FM10K_DOWN, &interface->state) ||
- test_bit(__FM10K_RESETTING, &interface->state))
+ if (test_bit(__FM10K_DOWN, interface->state) ||
+ test_bit(__FM10K_RESETTING, interface->state))
return;
/* rate limit tx hang checks to only once every 2 seconds */
@@ -663,7 +674,7 @@
FM10K_PFVTCTL_FTAG_DESC_ENABLE);
/* Initialize XPS */
- if (!test_and_set_bit(__FM10K_TX_XPS_INIT_DONE, &ring->state) &&
+ if (!test_and_set_bit(__FM10K_TX_XPS_INIT_DONE, ring->state) &&
ring->q_vector)
netif_set_xps_queue(ring->netdev,
&ring->q_vector->affinity_mask,
@@ -743,6 +754,7 @@
/* disable queue to avoid issues while updating state */
rxqctl = fm10k_read_reg(hw, FM10K_RXQCTL(reg_idx));
rxqctl &= ~FM10K_RXQCTL_ENABLE;
+ fm10k_write_reg(hw, FM10K_RXQCTL(reg_idx), rxqctl);
fm10k_write_flush(hw);
/* possible poll here to verify ring resources have been cleaned */
@@ -863,9 +875,9 @@
FM10K_MRQC_IPV6 |
FM10K_MRQC_TCP_IPV6;
- if (interface->flags & FM10K_FLAG_RSS_FIELD_IPV4_UDP)
+ if (test_bit(FM10K_FLAG_RSS_FIELD_IPV4_UDP, interface->flags))
mrqc |= FM10K_MRQC_UDP_IPV4;
- if (interface->flags & FM10K_FLAG_RSS_FIELD_IPV6_UDP)
+ if (test_bit(FM10K_FLAG_RSS_FIELD_IPV6_UDP, interface->flags))
mrqc |= FM10K_MRQC_UDP_IPV6;
fm10k_write_reg(hw, FM10K_MRQC(0), mrqc);
@@ -980,7 +992,7 @@
int i;
/* if interface is down do nothing */
- if (test_bit(__FM10K_DOWN, &interface->state))
+ if (test_bit(__FM10K_DOWN, interface->state))
return;
for (i = 0; i < interface->num_q_vectors; i++)
@@ -1167,13 +1179,13 @@
}
if (err == FM10K_ERR_RESET_REQUESTED)
- interface->flags |= FM10K_FLAG_RESET_REQUESTED;
+ set_bit(FM10K_FLAG_RESET_REQUESTED, interface->flags);
/* if switch toggled state we should reset GLORTs */
if (eicr & FM10K_EICR_SWITCHNOTREADY) {
/* force link down for at least 4 seconds */
interface->link_down_event = jiffies + (4 * HZ);
- set_bit(__FM10K_LINK_DOWN, &interface->state);
+ set_bit(__FM10K_LINK_DOWN, interface->state);
/* reset dglort_map back to no config */
hw->mac.dglort_map = FM10K_DGLORTMAP_NONE;
@@ -1246,12 +1258,12 @@
/* MAC was changed so we need reset */
if (is_valid_ether_addr(hw->mac.perm_addr) &&
!ether_addr_equal(hw->mac.perm_addr, hw->mac.addr))
- interface->flags |= FM10K_FLAG_RESET_REQUESTED;
+ set_bit(FM10K_FLAG_RESET_REQUESTED, interface->flags);
/* VLAN override was changed, or default VLAN changed */
if ((vlan_override != hw->mac.vlan_override) ||
(default_vid != hw->mac.default_vid))
- interface->flags |= FM10K_FLAG_RESET_REQUESTED;
+ set_bit(FM10K_FLAG_RESET_REQUESTED, interface->flags);
return 0;
}
@@ -1325,7 +1337,7 @@
if (!err && hw->swapi.status) {
/* force link down for a reasonable delay */
interface->link_down_event = jiffies + (2 * HZ);
- set_bit(__FM10K_LINK_DOWN, &interface->state);
+ set_bit(__FM10K_LINK_DOWN, interface->state);
/* reset dglort_map back to no config */
hw->mac.dglort_map = FM10K_DGLORTMAP_NONE;
@@ -1356,7 +1368,7 @@
/* we need to reset if port count was just updated */
if (dglort_map != hw->mac.dglort_map)
- interface->flags |= FM10K_FLAG_RESET_REQUESTED;
+ set_bit(FM10K_FLAG_RESET_REQUESTED, interface->flags);
return 0;
}
@@ -1395,7 +1407,7 @@
/* we need to reset if default VLAN was just updated */
if (pvid != hw->mac.default_vid)
- interface->flags |= FM10K_FLAG_RESET_REQUESTED;
+ set_bit(FM10K_FLAG_RESET_REQUESTED, interface->flags);
hw->mac.default_vid = pvid;
@@ -1623,10 +1635,10 @@
hw->mac.ops.update_int_moderator(hw);
/* enable statistics capture again */
- clear_bit(__FM10K_UPDATING_STATS, &interface->state);
+ clear_bit(__FM10K_UPDATING_STATS, interface->state);
/* clear down bit to indicate we are ready to go */
- clear_bit(__FM10K_DOWN, &interface->state);
+ clear_bit(__FM10K_DOWN, interface->state);
/* enable polling cleanups */
fm10k_napi_enable_all(interface);
@@ -1660,7 +1672,7 @@
int err, i = 0, count = 0;
/* signal that we are down to the interrupt handler and service task */
- if (test_and_set_bit(__FM10K_DOWN, &interface->state))
+ if (test_and_set_bit(__FM10K_DOWN, interface->state))
return;
/* call carrier off first to avoid false dev_watchdog timeouts */
@@ -1680,7 +1692,7 @@
fm10k_update_stats(interface);
/* prevent updating statistics while we're down */
- while (test_and_set_bit(__FM10K_UPDATING_STATS, &interface->state))
+ while (test_and_set_bit(__FM10K_UPDATING_STATS, interface->state))
usleep_range(1000, 2000);
/* skip waiting for TX DMA if we lost PCIe link */
@@ -1849,8 +1861,8 @@
memcpy(interface->rssrk, rss_key, sizeof(rss_key));
/* Start off interface as being down */
- set_bit(__FM10K_DOWN, &interface->state);
- set_bit(__FM10K_UPDATING_STATS, &interface->state);
+ set_bit(__FM10K_DOWN, interface->state);
+ set_bit(__FM10K_UPDATING_STATS, interface->state);
return 0;
}
@@ -2027,7 +2039,7 @@
* must ensure it is disabled since we haven't yet requested the timer
* or work item.
*/
- set_bit(__FM10K_SERVICE_DISABLE, &interface->state);
+ set_bit(__FM10K_SERVICE_DISABLE, interface->state);
err = fm10k_mbx_request_irq(interface);
if (err)
@@ -2068,7 +2080,7 @@
fm10k_iov_configure(pdev, 0);
/* clear the service task disable bit to allow service task to start */
- clear_bit(__FM10K_SERVICE_DISABLE, &interface->state);
+ clear_bit(__FM10K_SERVICE_DISABLE, interface->state);
return 0;
@@ -2106,7 +2118,7 @@
del_timer_sync(&interface->service_timer);
- set_bit(__FM10K_SERVICE_DISABLE, &interface->state);
+ set_bit(__FM10K_SERVICE_DISABLE, interface->state);
cancel_work_sync(&interface->service_task);
/* free netdev, this may bounce the interrupts due to setup_tc */
@@ -2145,7 +2157,7 @@
* stopped. We stop the watchdog task until after we resume software
* activity.
*/
- set_bit(__FM10K_SERVICE_DISABLE, &interface->state);
+ set_bit(__FM10K_SERVICE_DISABLE, interface->state);
cancel_work_sync(&interface->service_task);
fm10k_prepare_for_reset(interface);
@@ -2171,10 +2183,10 @@
/* force link to stay down for a second to prevent link flutter */
interface->link_down_event = jiffies + (HZ);
- set_bit(__FM10K_LINK_DOWN, &interface->state);
+ set_bit(__FM10K_LINK_DOWN, interface->state);
/* clear the service task disable bit to allow service task to start */
- clear_bit(__FM10K_SERVICE_DISABLE, &interface->state);
+ clear_bit(__FM10K_SERVICE_DISABLE, interface->state);
fm10k_service_event_schedule(interface);
return err;
diff --git a/drivers/net/ethernet/intel/i40e/Makefile b/drivers/net/ethernet/intel/i40e/Makefile
index 3b3c63e..3da482c 100644
--- a/drivers/net/ethernet/intel/i40e/Makefile
+++ b/drivers/net/ethernet/intel/i40e/Makefile
@@ -28,6 +28,9 @@
# Makefile for the Intel(R) Ethernet Connection XL710 (i40e.ko) driver
#
+ccflags-y += -I$(src)
+subdir-ccflags-y += -I$(src)
+
obj-$(CONFIG_I40E) += i40e.o
i40e-objs := i40e_main.o \
@@ -45,4 +48,3 @@
i40e_virtchnl_pf.o
i40e-$(CONFIG_I40E_DCB) += i40e_dcb.o i40e_dcb_nl.o
-i40e-$(CONFIG_I40E_FCOE) += i40e_fcoe.o
diff --git a/drivers/net/ethernet/intel/i40e/i40e.h b/drivers/net/ethernet/intel/i40e/i40e.h
index 82d8040..cdde3cc2 100644
--- a/drivers/net/ethernet/intel/i40e/i40e.h
+++ b/drivers/net/ethernet/intel/i40e/i40e.h
@@ -56,9 +56,6 @@
#include <linux/ptp_clock_kernel.h>
#include "i40e_type.h"
#include "i40e_prototype.h"
-#ifdef I40E_FCOE
-#include "i40e_fcoe.h"
-#endif
#include "i40e_client.h"
#include "i40e_virtchnl.h"
#include "i40e_virtchnl_pf.h"
@@ -85,10 +82,6 @@
(((pf)->flags & I40E_FLAG_128_QP_RSS_CAPABLE) ? 128 : 64)
#define I40E_FDIR_RING 0
#define I40E_FDIR_RING_COUNT 32
-#ifdef I40E_FCOE
-#define I40E_DEFAULT_FCOE 8 /* default number of QPs for FCoE */
-#define I40E_MINIMUM_FCOE 1 /* minimum number of QPs for FCoE */
-#endif /* I40E_FCOE */
#define I40E_MAX_AQ_BUF_SIZE 4096
#define I40E_AQ_LEN 256
#define I40E_AQ_WORK_LIMIT 66 /* max number of VFs + a little */
@@ -98,14 +91,6 @@
#define I40E_QUEUE_WAIT_RETRY_LIMIT 10
#define I40E_INT_NAME_STR_LEN (IFNAMSIZ + 16)
-/* Ethtool Private Flags */
-#define I40E_PRIV_FLAGS_MFP_FLAG BIT(0)
-#define I40E_PRIV_FLAGS_LINKPOLL_FLAG BIT(1)
-#define I40E_PRIV_FLAGS_FD_ATR BIT(2)
-#define I40E_PRIV_FLAGS_VEB_STATS BIT(3)
-#define I40E_PRIV_FLAGS_HW_ATR_EVICT BIT(4)
-#define I40E_PRIV_FLAGS_TRUE_PROMISC_SUPPORT BIT(5)
-
#define I40E_NVM_VERSION_LO_SHIFT 0
#define I40E_NVM_VERSION_LO_MASK (0xff << I40E_NVM_VERSION_LO_SHIFT)
#define I40E_NVM_VERSION_HI_SHIFT 12
@@ -140,7 +125,6 @@
__I40E_CONFIG_BUSY,
__I40E_CONFIG_DONE,
__I40E_DOWN,
- __I40E_NEEDS_RESTART,
__I40E_SERVICE_SCHED,
__I40E_ADMINQ_EVENT_PENDING,
__I40E_MDD_EVENT_PENDING,
@@ -153,15 +137,28 @@
__I40E_GLOBAL_RESET_REQUESTED,
__I40E_EMP_RESET_REQUESTED,
__I40E_EMP_RESET_INTR_RECEIVED,
- __I40E_FILTER_OVERFLOW_PROMISC,
__I40E_SUSPENDED,
__I40E_PTP_TX_IN_PROGRESS,
__I40E_BAD_EEPROM,
__I40E_DOWN_REQUESTED,
__I40E_FD_FLUSH_REQUESTED,
__I40E_RESET_FAILED,
- __I40E_PORT_TX_SUSPENDED,
+ __I40E_PORT_SUSPENDED,
__I40E_VF_DISABLE,
+ /* This must be last as it determines the size of the BITMAP */
+ __I40E_STATE_SIZE__,
+};
+
+/* VSI state flags */
+enum i40e_vsi_state_t {
+ __I40E_VSI_DOWN,
+ __I40E_VSI_NEEDS_RESTART,
+ __I40E_VSI_SYNCING_FILTERS,
+ __I40E_VSI_OVERFLOW_PROMISC,
+ __I40E_VSI_REINIT_REQUESTED,
+ __I40E_VSI_DOWN_REQUESTED,
+ /* This must be last as it determines the size of the BITMAP */
+ __I40E_VSI_STATE_SIZE__,
};
enum i40e_interrupt_policy {
@@ -202,17 +199,32 @@
#define I40E_FD_ATR_TUNNEL_STAT_IDX(pf_id) \
(I40E_FD_STAT_PF_IDX(pf_id) + I40E_FD_STAT_ATR_TUNNEL)
+/* The following structure contains the data parsed from the user-defined
+ * field of the ethtool_rx_flow_spec structure.
+ */
+struct i40e_rx_flow_userdef {
+ bool flex_filter;
+ u16 flex_word;
+ u16 flex_offset;
+};
+
struct i40e_fdir_filter {
struct hlist_node fdir_node;
/* filter ipnut set */
u8 flow_type;
u8 ip4_proto;
/* TX packet view of src and dst */
- __be32 dst_ip[4];
- __be32 src_ip[4];
+ __be32 dst_ip;
+ __be32 src_ip;
__be16 src_port;
__be16 dst_port;
__be32 sctp_v_tag;
+
+ /* Flexible data to match within the packet payload */
+ __be16 flex_word;
+ u16 flex_offset;
+ bool flex_filter;
+
/* filter control */
u16 q_index;
u8 flex_off;
@@ -244,15 +256,85 @@
};
struct i40e_udp_port_config {
- __be16 index;
+ /* AdminQ command interface expects port number in Host byte order */
+ u16 port;
u8 type;
};
+/* macros related to FLX_PIT */
+#define I40E_FLEX_SET_FSIZE(fsize) (((fsize) << \
+ I40E_PRTQF_FLX_PIT_FSIZE_SHIFT) & \
+ I40E_PRTQF_FLX_PIT_FSIZE_MASK)
+#define I40E_FLEX_SET_DST_WORD(dst) (((dst) << \
+ I40E_PRTQF_FLX_PIT_DEST_OFF_SHIFT) & \
+ I40E_PRTQF_FLX_PIT_DEST_OFF_MASK)
+#define I40E_FLEX_SET_SRC_WORD(src) (((src) << \
+ I40E_PRTQF_FLX_PIT_SOURCE_OFF_SHIFT) & \
+ I40E_PRTQF_FLX_PIT_SOURCE_OFF_MASK)
+#define I40E_FLEX_PREP_VAL(dst, fsize, src) (I40E_FLEX_SET_DST_WORD(dst) | \
+ I40E_FLEX_SET_FSIZE(fsize) | \
+ I40E_FLEX_SET_SRC_WORD(src))
+
+#define I40E_FLEX_PIT_GET_SRC(flex) (((flex) & \
+ I40E_PRTQF_FLX_PIT_SOURCE_OFF_MASK) >> \
+ I40E_PRTQF_FLX_PIT_SOURCE_OFF_SHIFT)
+#define I40E_FLEX_PIT_GET_DST(flex) (((flex) & \
+ I40E_PRTQF_FLX_PIT_DEST_OFF_MASK) >> \
+ I40E_PRTQF_FLX_PIT_DEST_OFF_SHIFT)
+#define I40E_FLEX_PIT_GET_FSIZE(flex) (((flex) & \
+ I40E_PRTQF_FLX_PIT_FSIZE_MASK) >> \
+ I40E_PRTQF_FLX_PIT_FSIZE_SHIFT)
+
+#define I40E_MAX_FLEX_SRC_OFFSET 0x1F
+
+/* macros related to GLQF_ORT */
+#define I40E_ORT_SET_IDX(idx) (((idx) << \
+ I40E_GLQF_ORT_PIT_INDX_SHIFT) & \
+ I40E_GLQF_ORT_PIT_INDX_MASK)
+
+#define I40E_ORT_SET_COUNT(count) (((count) << \
+ I40E_GLQF_ORT_FIELD_CNT_SHIFT) & \
+ I40E_GLQF_ORT_FIELD_CNT_MASK)
+
+#define I40E_ORT_SET_PAYLOAD(payload) (((payload) << \
+ I40E_GLQF_ORT_FLX_PAYLOAD_SHIFT) & \
+ I40E_GLQF_ORT_FLX_PAYLOAD_MASK)
+
+#define I40E_ORT_PREP_VAL(idx, count, payload) (I40E_ORT_SET_IDX(idx) | \
+ I40E_ORT_SET_COUNT(count) | \
+ I40E_ORT_SET_PAYLOAD(payload))
+
+#define I40E_L3_GLQF_ORT_IDX 34
+#define I40E_L4_GLQF_ORT_IDX 35
+
+/* Flex PIT register index */
+#define I40E_FLEX_PIT_IDX_START_L2 0
+#define I40E_FLEX_PIT_IDX_START_L3 3
+#define I40E_FLEX_PIT_IDX_START_L4 6
+
+#define I40E_FLEX_PIT_TABLE_SIZE 3
+
+#define I40E_FLEX_DEST_UNUSED 63
+
+#define I40E_FLEX_INDEX_ENTRIES 8
+
+/* Flex MASK to disable all flexible entries */
+#define I40E_FLEX_INPUT_MASK (I40E_FLEX_50_MASK | I40E_FLEX_51_MASK | \
+ I40E_FLEX_52_MASK | I40E_FLEX_53_MASK | \
+ I40E_FLEX_54_MASK | I40E_FLEX_55_MASK | \
+ I40E_FLEX_56_MASK | I40E_FLEX_57_MASK)
+
+struct i40e_flex_pit {
+ struct list_head list;
+ u16 src_offset;
+ u8 pit_index;
+};
+
/* struct that defines the Ethernet device */
struct i40e_pf {
struct pci_dev *pdev;
struct i40e_hw hw;
- unsigned long state;
+ DECLARE_BITMAP(state, __I40E_STATE_SIZE__);
struct msix_entry *msix_entries;
bool fc_autoneg_status;
@@ -262,10 +344,6 @@
u16 num_vmdq_msix; /* num queue vectors per vmdq pool */
u16 num_req_vfs; /* num VFs requested for this VF */
u16 num_vf_qps; /* num queue pairs per VF */
-#ifdef I40E_FCOE
- u16 num_fcoe_qps; /* num fcoe queues this PF has set up */
- u16 num_fcoe_msix; /* num queue vectors per fcoe pool */
-#endif /* I40E_FCOE */
u16 num_lan_qps; /* num lan queues this PF has set up */
u16 num_lan_msix; /* num queue vectors for the base PF vsi */
u16 num_fdsb_msix; /* num queue vectors for sideband Fdir */
@@ -285,7 +363,23 @@
u32 fd_flush_cnt;
u32 fd_add_err;
u32 fd_atr_cnt;
- u32 fd_tcp_rule;
+
+ /* Book-keeping of side-band filter count per flow-type.
+ * This is used to detect and handle input set changes for
+ * respective flow-type.
+ */
+ u16 fd_tcp4_filter_cnt;
+ u16 fd_udp4_filter_cnt;
+ u16 fd_sctp4_filter_cnt;
+ u16 fd_ip4_filter_cnt;
+
+ /* Flexible filter table values that need to be programmed into
+ * hardware, which expects L3 and L4 to be programmed separately. We
+ * need to ensure that the values are in ascended order and don't have
+ * duplicates, so we track each L3 and L4 values in separate lists.
+ */
+ struct list_head l3_flex_pit_list;
+ struct list_head l4_flex_pit_list;
struct i40e_udp_port_config udp_ports[I40E_MAX_PF_UDP_OFFLOAD_PORTS];
u16 pending_udp_bitmap;
@@ -307,21 +401,15 @@
#define I40E_FLAG_MSIX_ENABLED BIT_ULL(3)
#define I40E_FLAG_RSS_ENABLED BIT_ULL(6)
#define I40E_FLAG_VMDQ_ENABLED BIT_ULL(7)
-#define I40E_FLAG_FDIR_REQUIRES_REINIT BIT_ULL(8)
-#define I40E_FLAG_NEED_LINK_UPDATE BIT_ULL(9)
#define I40E_FLAG_IWARP_ENABLED BIT_ULL(10)
-#ifdef I40E_FCOE
-#define I40E_FLAG_FCOE_ENABLED BIT_ULL(11)
-#endif /* I40E_FCOE */
-#define I40E_FLAG_CLEAN_ADMINQ BIT_ULL(14)
#define I40E_FLAG_FILTER_SYNC BIT_ULL(15)
#define I40E_FLAG_SERVICE_CLIENT_REQUESTED BIT_ULL(16)
-#define I40E_FLAG_PROCESS_MDD_EVENT BIT_ULL(17)
-#define I40E_FLAG_PROCESS_VFLR_EVENT BIT_ULL(18)
#define I40E_FLAG_SRIOV_ENABLED BIT_ULL(19)
#define I40E_FLAG_DCB_ENABLED BIT_ULL(20)
#define I40E_FLAG_FD_SB_ENABLED BIT_ULL(21)
#define I40E_FLAG_FD_ATR_ENABLED BIT_ULL(22)
+#define I40E_FLAG_FD_SB_AUTO_DISABLED BIT_ULL(23)
+#define I40E_FLAG_FD_ATR_AUTO_DISABLED BIT_ULL(24)
#define I40E_FLAG_PTP BIT_ULL(25)
#define I40E_FLAG_MFP_ENABLED BIT_ULL(26)
#define I40E_FLAG_UDP_FILTER_SYNC BIT_ULL(27)
@@ -348,16 +436,13 @@
#define I40E_FLAG_TRUE_PROMISC_SUPPORT BIT_ULL(51)
#define I40E_FLAG_HAVE_CRT_RETIMER BIT_ULL(52)
#define I40E_FLAG_PTP_L4_CAPABLE BIT_ULL(53)
-#define I40E_FLAG_WOL_MC_MAGIC_PKT_WAKE BIT_ULL(54)
+#define I40E_FLAG_CLIENT_RESET BIT_ULL(54)
#define I40E_FLAG_TEMP_LINK_POLLING BIT_ULL(55)
+#define I40E_FLAG_CLIENT_L2_CHANGE BIT_ULL(56)
+#define I40E_FLAG_WOL_MC_MAGIC_PKT_WAKE BIT_ULL(57)
+#define I40E_FLAG_LEGACY_RX BIT_ULL(58)
- /* tracks features that get auto disabled by errors */
- u64 auto_disable_flags;
-
-#ifdef I40E_FCOE
- struct i40e_fcoe fcoe;
-
-#endif /* I40E_FCOE */
+ struct i40e_client_instance *cinst;
bool stat_offsets_loaded;
struct i40e_hw_port_stats stats;
struct i40e_hw_port_stats stats_offsets;
@@ -412,8 +497,6 @@
*/
u16 dcbx_cap;
- u32 fcoe_hmc_filt_num;
- u32 fcoe_hmc_cntx_num;
struct i40e_filter_control_settings filter_settings;
struct ptp_clock *ptp_clock;
@@ -517,7 +600,7 @@
bool stat_offsets_loaded;
u32 current_netdev_flags;
- unsigned long state;
+ DECLARE_BITMAP(state, __I40E_VSI_STATE_SIZE__);
#define I40E_VSI_FLAG_FILTER_CHANGED BIT(0)
#define I40E_VSI_FLAG_VEB_OWNER BIT(1)
unsigned long flags;
@@ -533,16 +616,10 @@
struct rtnl_link_stats64 net_stats_offsets;
struct i40e_eth_stats eth_stats;
struct i40e_eth_stats eth_stats_offsets;
-#ifdef I40E_FCOE
- struct i40e_fcoe_stats fcoe_stats;
- struct i40e_fcoe_stats fcoe_stats_offsets;
- bool fcoe_stat_offsets_loaded;
-#endif
u32 tx_restart;
u32 tx_busy;
u64 tx_linearize;
u64 tx_force_wb;
- u64 tx_lost_interrupt;
u32 rx_buf_failed;
u32 rx_page_failed;
@@ -628,9 +705,6 @@
u8 num_ringpairs; /* total number of ring pairs in vector */
-#define I40E_Q_VECTOR_HUNG_DETECT 0 /* Bit Index for hung detection logic */
- unsigned long hung_detected; /* Set/Reset for hung_detection logic */
-
cpumask_t affinity_mask;
struct irq_affinity_notify affinity_notify;
@@ -696,21 +770,6 @@
}
/**
- * i40e_rx_is_programming_status - check for programming status descriptor
- * @qw: the first quad word of the program status descriptor
- *
- * The value of in the descriptor length field indicate if this
- * is a programming status descriptor for flow director or FCoE
- * by the value of I40E_RX_PROG_STATUS_DESC_LENGTH, otherwise
- * it is a packet descriptor.
- **/
-static inline bool i40e_rx_is_programming_status(u64 qw)
-{
- return I40E_RX_PROG_STATUS_DESC_LENGTH ==
- (qw >> I40E_RX_PROG_STATUS_DESC_LENGTH_SHIFT);
-}
-
-/**
* i40e_get_fd_cnt_all - get the total FD filter space available
* @pf: pointer to the PF struct
**/
@@ -719,13 +778,50 @@
return pf->hw.fdir_shared_filter_count + pf->fdir_pf_filter_count;
}
+/**
+ * i40e_read_fd_input_set - reads value of flow director input set register
+ * @pf: pointer to the PF struct
+ * @addr: register addr
+ *
+ * This function reads value of flow director input set register
+ * specified by 'addr' (which is specific to flow-type)
+ **/
+static inline u64 i40e_read_fd_input_set(struct i40e_pf *pf, u16 addr)
+{
+ u64 val;
+
+ val = i40e_read_rx_ctl(&pf->hw, I40E_PRTQF_FD_INSET(addr, 1));
+ val <<= 32;
+ val += i40e_read_rx_ctl(&pf->hw, I40E_PRTQF_FD_INSET(addr, 0));
+
+ return val;
+}
+
+/**
+ * i40e_write_fd_input_set - writes value into flow director input set register
+ * @pf: pointer to the PF struct
+ * @addr: register addr
+ * @val: value to be written
+ *
+ * This function writes specified value to the register specified by 'addr'.
+ * This register is input set register based on flow-type.
+ **/
+static inline void i40e_write_fd_input_set(struct i40e_pf *pf,
+ u16 addr, u64 val)
+{
+ i40e_write_rx_ctl(&pf->hw, I40E_PRTQF_FD_INSET(addr, 1),
+ (u32)(val >> 32));
+ i40e_write_rx_ctl(&pf->hw, I40E_PRTQF_FD_INSET(addr, 0),
+ (u32)(val & 0xFFFFFFFFULL));
+}
+
/* needed by i40e_ethtool.c */
int i40e_up(struct i40e_vsi *vsi);
void i40e_down(struct i40e_vsi *vsi);
extern const char i40e_driver_name[];
extern const char i40e_driver_version_str[];
void i40e_do_reset_safe(struct i40e_pf *pf, u32 reset_flags);
-void i40e_do_reset(struct i40e_pf *pf, u32 reset_flags);
+void i40e_do_reset(struct i40e_pf *pf, u32 reset_flags, bool lock_acquired);
int i40e_config_rss(struct i40e_vsi *vsi, u8 *seed, u8 *lut, u16 lut_size);
int i40e_get_rss(struct i40e_vsi *vsi, u8 *seed, u8 *lut, u16 lut_size);
void i40e_fill_rss_lut(struct i40e_pf *pf, u8 *lut,
@@ -773,17 +869,14 @@
struct i40e_vsi *i40e_vsi_setup(struct i40e_pf *pf, u8 type,
u16 uplink, u32 param1);
int i40e_vsi_release(struct i40e_vsi *vsi);
-#ifdef I40E_FCOE
-void i40e_vsi_setup_queue_map(struct i40e_vsi *vsi,
- struct i40e_vsi_context *ctxt,
- u8 enabled_tc, bool is_add);
-#endif
void i40e_service_event_schedule(struct i40e_pf *pf);
void i40e_notify_client_of_vf_msg(struct i40e_vsi *vsi, u32 vf_id,
u8 *msg, u16 len);
int i40e_vsi_start_rings(struct i40e_vsi *vsi);
void i40e_vsi_stop_rings(struct i40e_vsi *vsi);
+void i40e_vsi_stop_rings_no_wait(struct i40e_vsi *vsi);
+int i40e_vsi_wait_queues_disabled(struct i40e_vsi *vsi);
int i40e_reconfig_rss_queues(struct i40e_pf *pf, int queue_count);
struct i40e_veb *i40e_veb_setup(struct i40e_pf *pf, u16 flags, u16 uplink_seid,
u16 downlink_seid, u8 enabled_tc);
@@ -813,8 +906,7 @@
void i40e_notify_client_of_netdev_close(struct i40e_vsi *vsi, bool reset);
void i40e_notify_client_of_vf_enable(struct i40e_pf *pf, u32 num_vfs);
void i40e_notify_client_of_vf_reset(struct i40e_pf *pf, u32 vf_id);
-int i40e_vf_client_capable(struct i40e_pf *pf, u32 vf_id,
- enum i40e_client_type type);
+int i40e_vf_client_capable(struct i40e_pf *pf, u32 vf_id);
/**
* i40e_irq_dynamic_enable - Enable default interrupt generation settings
* @vsi: pointer to a vsi
@@ -838,20 +930,7 @@
void i40e_irq_dynamic_disable_icr0(struct i40e_pf *pf);
void i40e_irq_dynamic_enable_icr0(struct i40e_pf *pf, bool clearpba);
-#ifdef I40E_FCOE
-void i40e_get_netdev_stats_struct(struct net_device *netdev,
- struct rtnl_link_stats64 *storage);
-int i40e_set_mac(struct net_device *netdev, void *p);
-void i40e_set_rx_mode(struct net_device *netdev);
-#endif
int i40e_ioctl(struct net_device *netdev, struct ifreq *ifr, int cmd);
-#ifdef I40E_FCOE
-void i40e_tx_timeout(struct net_device *netdev);
-int i40e_vlan_rx_add_vid(struct net_device *netdev,
- __always_unused __be16 proto, u16 vid);
-int i40e_vlan_rx_kill_vid(struct net_device *netdev,
- __always_unused __be16 proto, u16 vid);
-#endif
int i40e_open(struct net_device *netdev);
int i40e_close(struct net_device *netdev);
int i40e_vsi_open(struct i40e_vsi *vsi);
@@ -865,25 +944,6 @@
int i40e_del_mac_filter(struct i40e_vsi *vsi, const u8 *macaddr);
bool i40e_is_vsi_in_vlan(struct i40e_vsi *vsi);
struct i40e_mac_filter *i40e_find_mac(struct i40e_vsi *vsi, const u8 *macaddr);
-#ifdef I40E_FCOE
-int __i40e_setup_tc(struct net_device *netdev, u32 handle, __be16 proto,
- struct tc_to_netdev *tc);
-void i40e_netpoll(struct net_device *netdev);
-int i40e_fcoe_enable(struct net_device *netdev);
-int i40e_fcoe_disable(struct net_device *netdev);
-int i40e_fcoe_vsi_init(struct i40e_vsi *vsi, struct i40e_vsi_context *ctxt);
-u8 i40e_get_fcoe_tc_map(struct i40e_pf *pf);
-void i40e_fcoe_config_netdev(struct net_device *netdev, struct i40e_vsi *vsi);
-void i40e_fcoe_vsi_setup(struct i40e_pf *pf);
-void i40e_init_pf_fcoe(struct i40e_pf *pf);
-int i40e_fcoe_setup_ddp_resources(struct i40e_vsi *vsi);
-void i40e_fcoe_free_ddp_resources(struct i40e_vsi *vsi);
-int i40e_fcoe_handle_offload(struct i40e_ring *rx_ring,
- union i40e_rx_desc *rx_desc,
- struct sk_buff *skb);
-void i40e_fcoe_handle_status(struct i40e_ring *rx_ring,
- union i40e_rx_desc *rx_desc, u8 prog_id);
-#endif /* I40E_FCOE */
void i40e_vlan_stripping_enable(struct i40e_vsi *vsi);
#ifdef CONFIG_I40E_DCB
void i40e_dcbnl_flush_apps(struct i40e_pf *pf,
diff --git a/drivers/net/ethernet/intel/i40e/i40e_adminq.c b/drivers/net/ethernet/intel/i40e/i40e_adminq.c
index 56fb272..ba04988 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_adminq.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_adminq.c
@@ -850,8 +850,8 @@
*/
if (i40e_asq_done(hw))
break;
- usleep_range(1000, 2000);
- total_delay++;
+ udelay(50);
+ total_delay += 50;
} while (total_delay < hw->aq.asq_cmd_timeout);
}
diff --git a/drivers/net/ethernet/intel/i40e/i40e_adminq.h b/drivers/net/ethernet/intel/i40e/i40e_adminq.h
index d92aad3..2349fbe 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_adminq.h
+++ b/drivers/net/ethernet/intel/i40e/i40e_adminq.h
@@ -151,7 +151,7 @@
/* general information */
#define I40E_AQ_LARGE_BUF 512
-#define I40E_ASQ_CMD_TIMEOUT 250 /* msecs */
+#define I40E_ASQ_CMD_TIMEOUT 250000 /* usecs */
void i40e_fill_default_direct_cmd_desc(struct i40e_aq_desc *desc,
u16 opcode);
diff --git a/drivers/net/ethernet/intel/i40e/i40e_adminq_cmd.h b/drivers/net/ethernet/intel/i40e/i40e_adminq_cmd.h
index 451f48b..5eb0411 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_adminq_cmd.h
+++ b/drivers/net/ethernet/intel/i40e/i40e_adminq_cmd.h
@@ -132,6 +132,10 @@
i40e_aqc_opc_list_func_capabilities = 0x000A,
i40e_aqc_opc_list_dev_capabilities = 0x000B,
+ /* Proxy commands */
+ i40e_aqc_opc_set_proxy_config = 0x0104,
+ i40e_aqc_opc_set_ns_proxy_table_entry = 0x0105,
+
/* LAA */
i40e_aqc_opc_mac_address_read = 0x0107,
i40e_aqc_opc_mac_address_write = 0x0108,
@@ -139,6 +143,10 @@
/* PXE */
i40e_aqc_opc_clear_pxe_mode = 0x0110,
+ /* WoL commands */
+ i40e_aqc_opc_set_wol_filter = 0x0120,
+ i40e_aqc_opc_get_wake_reason = 0x0121,
+
/* internal switch commands */
i40e_aqc_opc_get_switch_config = 0x0200,
i40e_aqc_opc_add_statistics = 0x0201,
@@ -177,10 +185,15 @@
i40e_aqc_opc_remove_control_packet_filter = 0x025B,
i40e_aqc_opc_add_cloud_filters = 0x025C,
i40e_aqc_opc_remove_cloud_filters = 0x025D,
+ i40e_aqc_opc_clear_wol_switch_filters = 0x025E,
i40e_aqc_opc_add_mirror_rule = 0x0260,
i40e_aqc_opc_delete_mirror_rule = 0x0261,
+ /* Pipeline Personalization Profile */
+ i40e_aqc_opc_write_personalization_profile = 0x0270,
+ i40e_aqc_opc_get_personalization_profile_list = 0x0271,
+
/* DCB commands */
i40e_aqc_opc_dcb_ignore_pfc = 0x0301,
i40e_aqc_opc_dcb_updated = 0x0302,
@@ -563,6 +576,56 @@
I40E_CHECK_CMD_LENGTH(i40e_aqc_clear_pxe);
+/* Set WoL Filter (0x0120) */
+
+struct i40e_aqc_set_wol_filter {
+ __le16 filter_index;
+#define I40E_AQC_MAX_NUM_WOL_FILTERS 8
+#define I40E_AQC_SET_WOL_FILTER_TYPE_MAGIC_SHIFT 15
+#define I40E_AQC_SET_WOL_FILTER_TYPE_MAGIC_MASK (0x1 << \
+ I40E_AQC_SET_WOL_FILTER_TYPE_MAGIC_SHIFT)
+
+#define I40E_AQC_SET_WOL_FILTER_INDEX_SHIFT 0
+#define I40E_AQC_SET_WOL_FILTER_INDEX_MASK (0x7 << \
+ I40E_AQC_SET_WOL_FILTER_INDEX_SHIFT)
+ __le16 cmd_flags;
+#define I40E_AQC_SET_WOL_FILTER 0x8000
+#define I40E_AQC_SET_WOL_FILTER_NO_TCO_WOL 0x4000
+#define I40E_AQC_SET_WOL_FILTER_ACTION_CLEAR 0
+#define I40E_AQC_SET_WOL_FILTER_ACTION_SET 1
+ __le16 valid_flags;
+#define I40E_AQC_SET_WOL_FILTER_ACTION_VALID 0x8000
+#define I40E_AQC_SET_WOL_FILTER_NO_TCO_ACTION_VALID 0x4000
+ u8 reserved[2];
+ __le32 address_high;
+ __le32 address_low;
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_set_wol_filter);
+
+struct i40e_aqc_set_wol_filter_data {
+ u8 filter[128];
+ u8 mask[16];
+};
+
+I40E_CHECK_STRUCT_LEN(0x90, i40e_aqc_set_wol_filter_data);
+
+/* Get Wake Reason (0x0121) */
+
+struct i40e_aqc_get_wake_reason_completion {
+ u8 reserved_1[2];
+ __le16 wake_reason;
+#define I40E_AQC_GET_WAKE_UP_REASON_WOL_REASON_MATCHED_INDEX_SHIFT 0
+#define I40E_AQC_GET_WAKE_UP_REASON_WOL_REASON_MATCHED_INDEX_MASK (0xFF << \
+ I40E_AQC_GET_WAKE_UP_REASON_WOL_REASON_MATCHED_INDEX_SHIFT)
+#define I40E_AQC_GET_WAKE_UP_REASON_WOL_REASON_RESERVED_SHIFT 8
+#define I40E_AQC_GET_WAKE_UP_REASON_WOL_REASON_RESERVED_MASK (0xFF << \
+ I40E_AQC_GET_WAKE_UP_REASON_WOL_REASON_RESERVED_SHIFT)
+ u8 reserved_2[12];
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_get_wake_reason_completion);
+
/* Switch configuration commands (0x02xx) */
/* Used by many indirect commands that only pass an seid and a buffer in the
@@ -645,6 +708,8 @@
#define I40E_AQ_SET_P_PARAMS_PAD_SHORT_PACKETS 2 /* must set! */
#define I40E_AQ_SET_P_PARAMS_DOUBLE_VLAN_ENA 4
__le16 bad_frame_vsi;
+#define I40E_AQ_SET_P_PARAMS_BFRAME_SEID_SHIFT 0x0
+#define I40E_AQ_SET_P_PARAMS_BFRAME_SEID_MASK 0x3FF
__le16 default_seid; /* reserved for command */
u8 reserved[10];
};
@@ -696,6 +761,7 @@
/* Set Switch Configuration (direct 0x0205) */
struct i40e_aqc_set_switch_config {
__le16 flags;
+/* flags used for both fields below */
#define I40E_AQ_SET_SWITCH_CFG_PROMISC 0x0001
#define I40E_AQ_SET_SWITCH_CFG_L2_FILTER 0x0002
__le16 valid_flags;
@@ -1369,6 +1435,36 @@
I40E_CHECK_CMD_LENGTH(i40e_aqc_add_delete_mirror_rule_completion);
+/* Pipeline Personalization Profile */
+struct i40e_aqc_write_personalization_profile {
+ u8 flags;
+ u8 reserved[3];
+ __le32 profile_track_id;
+ __le32 addr_high;
+ __le32 addr_low;
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_write_personalization_profile);
+
+struct i40e_aqc_write_ppp_resp {
+ __le32 error_offset;
+ __le32 error_info;
+ __le32 addr_high;
+ __le32 addr_low;
+};
+
+struct i40e_aqc_get_applied_profiles {
+ u8 flags;
+#define I40E_AQC_GET_PPP_GET_CONF 0x1
+#define I40E_AQC_GET_PPP_GET_RDPU_CONF 0x2
+ u8 rsv[3];
+ __le32 reserved;
+ __le32 addr_high;
+ __le32 addr_low;
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_get_applied_profiles);
+
/* DCB 0x03xx*/
/* PFC Ignore (direct 0x0301)
@@ -1844,11 +1940,12 @@
#define I40E_AQ_CONFIG_FEC_RS_ENA 0x02
#define I40E_AQ_CONFIG_CRC_ENA 0x04
#define I40E_AQ_CONFIG_PACING_MASK 0x78
- u8 external_power_ability;
+ u8 power_desc;
#define I40E_AQ_LINK_POWER_CLASS_1 0x00
#define I40E_AQ_LINK_POWER_CLASS_2 0x01
#define I40E_AQ_LINK_POWER_CLASS_3 0x02
#define I40E_AQ_LINK_POWER_CLASS_4 0x03
+#define I40E_AQ_PWR_CLASS_MASK 0x03
u8 reserved[4];
};
diff --git a/drivers/net/ethernet/intel/i40e/i40e_client.c b/drivers/net/ethernet/intel/i40e/i40e_client.c
index d570219..c3b81a9 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_client.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_client.c
@@ -32,16 +32,10 @@
#include "i40e_client.h"
static const char i40e_client_interface_version_str[] = I40E_CLIENT_VERSION_STR;
-
+static struct i40e_client *registered_client;
static LIST_HEAD(i40e_devices);
static DEFINE_MUTEX(i40e_device_mutex);
-static LIST_HEAD(i40e_clients);
-static DEFINE_MUTEX(i40e_client_mutex);
-
-static LIST_HEAD(i40e_client_instances);
-static DEFINE_MUTEX(i40e_client_instance_mutex);
-
static int i40e_client_virtchnl_send(struct i40e_info *ldev,
struct i40e_client *client,
u32 vf_id, u8 *msg, u16 len);
@@ -67,28 +61,6 @@
};
/**
- * i40e_client_type_to_vsi_type - convert client type to vsi type
- * @client_type: the i40e_client type
- *
- * returns the related vsi type value
- **/
-static
-enum i40e_vsi_type i40e_client_type_to_vsi_type(enum i40e_client_type type)
-{
- switch (type) {
- case I40E_CLIENT_IWARP:
- return I40E_VSI_IWARP;
-
- case I40E_CLIENT_VMDQ2:
- return I40E_VSI_VMDQ2;
-
- default:
- pr_err("i40e: Client type unknown\n");
- return I40E_VSI_TYPE_UNKNOWN;
- }
-}
-
-/**
* i40e_client_get_params - Get the params that can change at runtime
* @vsi: the VSI with the message
* @param: clinet param struct
@@ -134,31 +106,22 @@
void
i40e_notify_client_of_vf_msg(struct i40e_vsi *vsi, u32 vf_id, u8 *msg, u16 len)
{
- struct i40e_client_instance *cdev;
+ struct i40e_pf *pf = vsi->back;
+ struct i40e_client_instance *cdev = pf->cinst;
- if (!vsi)
+ if (!cdev || !cdev->client)
return;
- mutex_lock(&i40e_client_instance_mutex);
- list_for_each_entry(cdev, &i40e_client_instances, list) {
- if (cdev->lan_info.pf == vsi->back) {
- if (!cdev->client ||
- !cdev->client->ops ||
- !cdev->client->ops->virtchnl_receive) {
- dev_dbg(&vsi->back->pdev->dev,
- "Cannot locate client instance virtual channel receive routine\n");
- continue;
- }
- if (!test_bit(__I40E_CLIENT_INSTANCE_OPENED,
- &cdev->state)) {
- dev_dbg(&vsi->back->pdev->dev, "Client is not open, abort virtchnl_receive\n");
- continue;
- }
- cdev->client->ops->virtchnl_receive(&cdev->lan_info,
- cdev->client,
- vf_id, msg, len);
- }
+ if (!cdev->client->ops || !cdev->client->ops->virtchnl_receive) {
+ dev_dbg(&pf->pdev->dev,
+ "Cannot locate client instance virtual channel receive routine\n");
+ return;
}
- mutex_unlock(&i40e_client_instance_mutex);
+ if (!test_bit(__I40E_CLIENT_INSTANCE_OPENED, &cdev->state)) {
+ dev_dbg(&pf->pdev->dev, "Client is not open, abort virtchnl_receive\n");
+ return;
+ }
+ cdev->client->ops->virtchnl_receive(&cdev->lan_info, cdev->client,
+ vf_id, msg, len);
}
/**
@@ -169,39 +132,30 @@
**/
void i40e_notify_client_of_l2_param_changes(struct i40e_vsi *vsi)
{
- struct i40e_client_instance *cdev;
+ struct i40e_pf *pf = vsi->back;
+ struct i40e_client_instance *cdev = pf->cinst;
struct i40e_params params;
- if (!vsi)
+ if (!cdev || !cdev->client)
return;
- mutex_lock(&i40e_client_instance_mutex);
- list_for_each_entry(cdev, &i40e_client_instances, list) {
- if (cdev->lan_info.pf == vsi->back) {
- if (!cdev->client ||
- !cdev->client->ops ||
- !cdev->client->ops->l2_param_change) {
- dev_dbg(&vsi->back->pdev->dev,
- "Cannot locate client instance l2_param_change routine\n");
- continue;
- }
+ if (!cdev->client->ops || !cdev->client->ops->l2_param_change) {
+ dev_dbg(&vsi->back->pdev->dev,
+ "Cannot locate client instance l2_param_change routine\n");
+ return;
+ }
+ if (!test_bit(__I40E_CLIENT_INSTANCE_OPENED, &cdev->state)) {
+ dev_dbg(&vsi->back->pdev->dev, "Client is not open, abort l2 param change\n");
+ return;
+ }
memset(¶ms, 0, sizeof(params));
i40e_client_get_params(vsi, ¶ms);
- if (!test_bit(__I40E_CLIENT_INSTANCE_OPENED,
- &cdev->state)) {
- dev_dbg(&vsi->back->pdev->dev, "Client is not open, abort l2 param change\n");
- continue;
- }
- cdev->lan_info.params = params;
- cdev->client->ops->l2_param_change(&cdev->lan_info,
- cdev->client,
- ¶ms);
- }
- }
- mutex_unlock(&i40e_client_instance_mutex);
+ memcpy(&cdev->lan_info.params, ¶ms, sizeof(struct i40e_params));
+ cdev->client->ops->l2_param_change(&cdev->lan_info, cdev->client,
+ ¶ms);
}
/**
- * i40e_client_release_qvlist
+ * i40e_client_release_qvlist - release MSI-X vector mapping for client
* @ldev: pointer to L2 context.
*
**/
@@ -237,26 +191,19 @@
**/
void i40e_notify_client_of_netdev_close(struct i40e_vsi *vsi, bool reset)
{
- struct i40e_client_instance *cdev;
+ struct i40e_pf *pf = vsi->back;
+ struct i40e_client_instance *cdev = pf->cinst;
- if (!vsi)
+ if (!cdev || !cdev->client)
return;
- mutex_lock(&i40e_client_instance_mutex);
- list_for_each_entry(cdev, &i40e_client_instances, list) {
- if (cdev->lan_info.netdev == vsi->netdev) {
- if (!cdev->client ||
- !cdev->client->ops || !cdev->client->ops->close) {
- dev_dbg(&vsi->back->pdev->dev,
- "Cannot locate client instance close routine\n");
- continue;
- }
- cdev->client->ops->close(&cdev->lan_info, cdev->client,
- reset);
- clear_bit(__I40E_CLIENT_INSTANCE_OPENED, &cdev->state);
- i40e_client_release_qvlist(&cdev->lan_info);
- }
+ if (!cdev->client->ops || !cdev->client->ops->close) {
+ dev_dbg(&vsi->back->pdev->dev,
+ "Cannot locate client instance close routine\n");
+ return;
}
- mutex_unlock(&i40e_client_instance_mutex);
+ cdev->client->ops->close(&cdev->lan_info, cdev->client, reset);
+ clear_bit(__I40E_CLIENT_INSTANCE_OPENED, &cdev->state);
+ i40e_client_release_qvlist(&cdev->lan_info);
}
/**
@@ -268,30 +215,20 @@
**/
void i40e_notify_client_of_vf_reset(struct i40e_pf *pf, u32 vf_id)
{
- struct i40e_client_instance *cdev;
+ struct i40e_client_instance *cdev = pf->cinst;
- if (!pf)
+ if (!cdev || !cdev->client)
return;
- mutex_lock(&i40e_client_instance_mutex);
- list_for_each_entry(cdev, &i40e_client_instances, list) {
- if (cdev->lan_info.pf == pf) {
- if (!cdev->client ||
- !cdev->client->ops ||
- !cdev->client->ops->vf_reset) {
- dev_dbg(&pf->pdev->dev,
- "Cannot locate client instance VF reset routine\n");
- continue;
- }
- if (!test_bit(__I40E_CLIENT_INSTANCE_OPENED,
- &cdev->state)) {
- dev_dbg(&pf->pdev->dev, "Client is not open, abort vf-reset\n");
- continue;
- }
- cdev->client->ops->vf_reset(&cdev->lan_info,
- cdev->client, vf_id);
- }
+ if (!cdev->client->ops || !cdev->client->ops->vf_reset) {
+ dev_dbg(&pf->pdev->dev,
+ "Cannot locate client instance VF reset routine\n");
+ return;
}
- mutex_unlock(&i40e_client_instance_mutex);
+ if (!test_bit(__I40E_CLIENT_INSTANCE_OPENED, &cdev->state)) {
+ dev_dbg(&pf->pdev->dev, "Client is not open, abort vf-reset\n");
+ return;
+ }
+ cdev->client->ops->vf_reset(&cdev->lan_info, cdev->client, vf_id);
}
/**
@@ -303,30 +240,21 @@
**/
void i40e_notify_client_of_vf_enable(struct i40e_pf *pf, u32 num_vfs)
{
- struct i40e_client_instance *cdev;
+ struct i40e_client_instance *cdev = pf->cinst;
- if (!pf)
+ if (!cdev || !cdev->client)
return;
- mutex_lock(&i40e_client_instance_mutex);
- list_for_each_entry(cdev, &i40e_client_instances, list) {
- if (cdev->lan_info.pf == pf) {
- if (!cdev->client ||
- !cdev->client->ops ||
- !cdev->client->ops->vf_enable) {
- dev_dbg(&pf->pdev->dev,
- "Cannot locate client instance VF enable routine\n");
- continue;
- }
- if (!test_bit(__I40E_CLIENT_INSTANCE_OPENED,
- &cdev->state)) {
- dev_dbg(&pf->pdev->dev, "Client is not open, abort vf-enable\n");
- continue;
- }
- cdev->client->ops->vf_enable(&cdev->lan_info,
- cdev->client, num_vfs);
- }
+ if (!cdev->client->ops || !cdev->client->ops->vf_enable) {
+ dev_dbg(&pf->pdev->dev,
+ "Cannot locate client instance VF enable routine\n");
+ return;
}
- mutex_unlock(&i40e_client_instance_mutex);
+ if (!test_bit(__I40E_CLIENT_INSTANCE_OPENED,
+ &cdev->state)) {
+ dev_dbg(&pf->pdev->dev, "Client is not open, abort vf-enable\n");
+ return;
+ }
+ cdev->client->ops->vf_enable(&cdev->lan_info, cdev->client, num_vfs);
}
/**
@@ -337,37 +265,25 @@
* If there is a client of the specified type attached to this PF, call
* its vf_capable routine
**/
-int i40e_vf_client_capable(struct i40e_pf *pf, u32 vf_id,
- enum i40e_client_type type)
+int i40e_vf_client_capable(struct i40e_pf *pf, u32 vf_id)
{
- struct i40e_client_instance *cdev;
+ struct i40e_client_instance *cdev = pf->cinst;
int capable = false;
- if (!pf)
- return false;
- mutex_lock(&i40e_client_instance_mutex);
- list_for_each_entry(cdev, &i40e_client_instances, list) {
- if (cdev->lan_info.pf == pf) {
- if (!cdev->client ||
- !cdev->client->ops ||
- !cdev->client->ops->vf_capable ||
- !(cdev->client->type == type)) {
- dev_dbg(&pf->pdev->dev,
- "Cannot locate client instance VF capability routine\n");
- continue;
- }
- if (!test_bit(__I40E_CLIENT_INSTANCE_OPENED,
- &cdev->state)) {
- dev_dbg(&pf->pdev->dev, "Client is not open, abort vf-capable\n");
- continue;
- }
- capable = cdev->client->ops->vf_capable(&cdev->lan_info,
- cdev->client,
- vf_id);
- break;
- }
+ if (!cdev || !cdev->client)
+ goto out;
+ if (!cdev->client->ops || !cdev->client->ops->vf_capable) {
+ dev_info(&pf->pdev->dev,
+ "Cannot locate client instance VF capability routine\n");
+ goto out;
}
- mutex_unlock(&i40e_client_instance_mutex);
+ if (!test_bit(__I40E_CLIENT_INSTANCE_OPENED, &cdev->state))
+ goto out;
+
+ capable = cdev->client->ops->vf_capable(&cdev->lan_info,
+ cdev->client,
+ vf_id);
+out:
return capable;
}
@@ -377,27 +293,19 @@
* @client: pointer to a client struct in the client list.
* @existing: if there was already an existing instance
*
- * Returns cdev ptr on success or if already exists, NULL on failure
**/
-static
-struct i40e_client_instance *i40e_client_add_instance(struct i40e_pf *pf,
- struct i40e_client *client,
- bool *existing)
+static void i40e_client_add_instance(struct i40e_pf *pf)
{
- struct i40e_client_instance *cdev;
+ struct i40e_client_instance *cdev = NULL;
struct netdev_hw_addr *mac = NULL;
struct i40e_vsi *vsi = pf->vsi[pf->lan_vsi];
- mutex_lock(&i40e_client_instance_mutex);
- list_for_each_entry(cdev, &i40e_client_instances, list) {
- if ((cdev->lan_info.pf == pf) && (cdev->client == client)) {
- *existing = true;
- goto out;
- }
- }
+ if (!registered_client || pf->cinst)
+ return;
+
cdev = kzalloc(sizeof(*cdev), GFP_KERNEL);
if (!cdev)
- goto out;
+ return;
cdev->lan_info.pf = (void *)pf;
cdev->lan_info.netdev = vsi->netdev;
@@ -417,7 +325,7 @@
if (i40e_client_get_params(vsi, &cdev->lan_info.params)) {
kfree(cdev);
cdev = NULL;
- goto out;
+ return;
}
cdev->lan_info.msix_count = pf->num_iwarp_msix;
@@ -430,41 +338,20 @@
else
dev_err(&pf->pdev->dev, "MAC address list is empty!\n");
- cdev->client = client;
- INIT_LIST_HEAD(&cdev->list);
- list_add(&cdev->list, &i40e_client_instances);
-out:
- mutex_unlock(&i40e_client_instance_mutex);
- return cdev;
+ cdev->client = registered_client;
+ pf->cinst = cdev;
}
/**
* i40e_client_del_instance - removes a client instance from the list
* @pf: pointer to the board struct
*
- * Returns 0 on success or non-0 on error
**/
static
-int i40e_client_del_instance(struct i40e_pf *pf, struct i40e_client *client)
+void i40e_client_del_instance(struct i40e_pf *pf)
{
- struct i40e_client_instance *cdev, *tmp;
- int ret = -ENODEV;
-
- mutex_lock(&i40e_client_instance_mutex);
- list_for_each_entry_safe(cdev, tmp, &i40e_client_instances, list) {
- if ((cdev->lan_info.pf != pf) || (cdev->client != client))
- continue;
-
- dev_info(&pf->pdev->dev, "Deleted instance of Client %s, of dev %d bus=0x%02x func=0x%02x)\n",
- client->name, pf->hw.pf_id,
- pf->hw.bus.device, pf->hw.bus.func);
- list_del(&cdev->list);
- kfree(cdev);
- ret = 0;
- break;
- }
- mutex_unlock(&i40e_client_instance_mutex);
- return ret;
+ kfree(pf->cinst);
+ pf->cinst = NULL;
}
/**
@@ -473,67 +360,50 @@
**/
void i40e_client_subtask(struct i40e_pf *pf)
{
+ struct i40e_client *client = registered_client;
struct i40e_client_instance *cdev;
- struct i40e_client *client;
- bool existing = false;
+ struct i40e_vsi *vsi = pf->vsi[pf->lan_vsi];
int ret = 0;
if (!(pf->flags & I40E_FLAG_SERVICE_CLIENT_REQUESTED))
return;
pf->flags &= ~I40E_FLAG_SERVICE_CLIENT_REQUESTED;
+ cdev = pf->cinst;
/* If we're down or resetting, just bail */
- if (test_bit(__I40E_DOWN, &pf->state) ||
- test_bit(__I40E_CONFIG_BUSY, &pf->state))
+ if (test_bit(__I40E_DOWN, pf->state) ||
+ test_bit(__I40E_CONFIG_BUSY, pf->state))
return;
- /* Check client state and instantiate client if client registered */
- mutex_lock(&i40e_client_mutex);
- list_for_each_entry(client, &i40e_clients, list) {
- /* first check client is registered */
- if (!test_bit(__I40E_CLIENT_REGISTERED, &client->state))
- continue;
+ if (!client || !cdev)
+ return;
- /* Do we also need the LAN VSI to be up, to create instance */
- if (!(client->flags & I40E_CLIENT_FLAGS_LAUNCH_ON_PROBE)) {
- /* check if L2 VSI is up, if not we are not ready */
- if (test_bit(__I40E_DOWN, &pf->vsi[pf->lan_vsi]->state))
- continue;
- } else {
- dev_warn(&pf->pdev->dev, "This client %s is being instantiated at probe\n",
- client->name);
- }
-
- /* Add the client instance to the instance list */
- cdev = i40e_client_add_instance(pf, client, &existing);
- if (!cdev)
- continue;
-
- if (!existing) {
- dev_info(&pf->pdev->dev, "Added instance of Client %s to PF%d bus=0x%02x dev=0x%02x func=0x%02x\n",
- client->name, pf->hw.pf_id,
- pf->hw.bus.bus_id, pf->hw.bus.device,
- pf->hw.bus.func);
- }
-
- mutex_lock(&i40e_client_instance_mutex);
- if (!test_bit(__I40E_CLIENT_INSTANCE_OPENED,
- &cdev->state)) {
- /* Send an Open request to the client */
- if (client->ops && client->ops->open)
- ret = client->ops->open(&cdev->lan_info,
- client);
- if (!ret) {
- set_bit(__I40E_CLIENT_INSTANCE_OPENED,
- &cdev->state);
- } else {
- /* remove client instance */
- i40e_client_del_instance(pf, client);
+ /* Here we handle client opens. If the client is down, but
+ * the netdev is up, then open the client.
+ */
+ if (!test_bit(__I40E_CLIENT_INSTANCE_OPENED, &cdev->state)) {
+ if (!test_bit(__I40E_VSI_DOWN, vsi->state) &&
+ client->ops && client->ops->open) {
+ set_bit(__I40E_CLIENT_INSTANCE_OPENED, &cdev->state);
+ ret = client->ops->open(&cdev->lan_info, client);
+ if (ret) {
+ /* Remove failed client instance */
+ clear_bit(__I40E_CLIENT_INSTANCE_OPENED,
+ &cdev->state);
+ i40e_client_del_instance(pf);
}
}
- mutex_unlock(&i40e_client_instance_mutex);
+ } else {
+ /* Likewise for client close. If the client is up, but the netdev
+ * is down, then close the client.
+ */
+ if (test_bit(__I40E_VSI_DOWN, vsi->state) &&
+ client->ops && client->ops->close) {
+ clear_bit(__I40E_CLIENT_INSTANCE_OPENED, &cdev->state);
+ client->ops->close(&cdev->lan_info, client, false);
+ i40e_client_release_qvlist(&cdev->lan_info);
+ }
}
- mutex_unlock(&i40e_client_mutex);
}
/**
@@ -566,6 +436,12 @@
pf->hw.pf_id, pf->hw.bus.bus_id,
pf->hw.bus.device, pf->hw.bus.func);
+ /* If a client has already been registered, we need to add an instance
+ * of it to our new LAN device.
+ */
+ if (registered_client)
+ i40e_client_add_instance(pf);
+
/* Since in some cases register may have happened before a device gets
* added, we can schedule a subtask to go initiate the clients if
* they can be launched at probe time.
@@ -589,6 +465,9 @@
struct i40e_device *ldev, *tmp;
int ret = -ENODEV;
+ /* First, remove any client instance. */
+ i40e_client_del_instance(pf);
+
mutex_lock(&i40e_device_mutex);
list_for_each_entry_safe(ldev, tmp, &i40e_devices, list) {
if (ldev->pf == pf) {
@@ -601,7 +480,6 @@
break;
}
}
-
mutex_unlock(&i40e_device_mutex);
return ret;
}
@@ -610,22 +488,24 @@
* i40e_client_release - release client specific resources
* @client: pointer to the registered client
*
- * Return 0 on success or < 0 on error
**/
-static int i40e_client_release(struct i40e_client *client)
+static void i40e_client_release(struct i40e_client *client)
{
- struct i40e_client_instance *cdev, *tmp;
+ struct i40e_client_instance *cdev;
+ struct i40e_device *ldev;
struct i40e_pf *pf;
- int ret = 0;
- LIST_HEAD(cdevs_tmp);
-
- mutex_lock(&i40e_client_instance_mutex);
- list_for_each_entry_safe(cdev, tmp, &i40e_client_instances, list) {
- if (strncmp(cdev->client->name, client->name,
- I40E_CLIENT_STR_LENGTH))
+ mutex_lock(&i40e_device_mutex);
+ list_for_each_entry(ldev, &i40e_devices, list) {
+ pf = ldev->pf;
+ cdev = pf->cinst;
+ if (!cdev)
continue;
- pf = (struct i40e_pf *)cdev->lan_info.pf;
+
+ while (test_and_set_bit(__I40E_SERVICE_SCHED,
+ pf->state))
+ usleep_range(500, 1000);
+
if (test_bit(__I40E_CLIENT_INSTANCE_OPENED, &cdev->state)) {
if (client->ops && client->ops->close)
client->ops->close(&cdev->lan_info, client,
@@ -637,18 +517,13 @@
"Client %s instance for PF id %d closed\n",
client->name, pf->hw.pf_id);
}
- /* delete the client instance from the list */
- list_move(&cdev->list, &cdevs_tmp);
+ /* delete the client instance */
+ i40e_client_del_instance(pf);
dev_info(&pf->pdev->dev, "Deleted client instance of Client %s\n",
client->name);
+ clear_bit(__I40E_SERVICE_SCHED, pf->state);
}
- mutex_unlock(&i40e_client_instance_mutex);
-
- /* free the client device and release its vsi */
- list_for_each_entry_safe(cdev, tmp, &cdevs_tmp, list) {
- kfree(cdev);
- }
- return ret;
+ mutex_unlock(&i40e_device_mutex);
}
/**
@@ -664,6 +539,7 @@
mutex_lock(&i40e_device_mutex);
list_for_each_entry(ldev, &i40e_devices, list) {
pf = ldev->pf;
+ i40e_client_add_instance(pf);
/* Start the client subtask */
pf->flags |= I40E_FLAG_SERVICE_CLIENT_REQUESTED;
i40e_service_event_schedule(pf);
@@ -785,15 +661,15 @@
switch (reset_level) {
case I40E_CLIENT_RESET_LEVEL_PF:
- set_bit(__I40E_PF_RESET_REQUESTED, &pf->state);
+ set_bit(__I40E_PF_RESET_REQUESTED, pf->state);
break;
case I40E_CLIENT_RESET_LEVEL_CORE:
- set_bit(__I40E_PF_RESET_REQUESTED, &pf->state);
+ set_bit(__I40E_PF_RESET_REQUESTED, pf->state);
break;
default:
dev_warn(&pf->pdev->dev,
- "Client %s instance for PF id %d request an unsupported reset: %d.\n",
- client->name, pf->hw.pf_id, reset_level);
+ "Client for PF id %d requested an unsupported reset: %d.\n",
+ pf->hw.pf_id, reset_level);
break;
}
@@ -852,8 +728,8 @@
} else {
update = false;
dev_warn(&pf->pdev->dev,
- "Client %s instance for PF id %d request an unsupported Config: %x.\n",
- client->name, pf->hw.pf_id, flag);
+ "Client for PF id %d request an unsupported Config: %x.\n",
+ pf->hw.pf_id, flag);
}
if (update) {
@@ -878,7 +754,6 @@
int i40e_register_client(struct i40e_client *client)
{
int ret = 0;
- enum i40e_vsi_type vsi_type;
if (!client) {
ret = -EIO;
@@ -891,11 +766,9 @@
goto out;
}
- mutex_lock(&i40e_client_mutex);
- if (i40e_client_is_registered(client)) {
+ if (registered_client) {
pr_info("i40e: Client %s has already been registered!\n",
client->name);
- mutex_unlock(&i40e_client_mutex);
ret = -EEXIST;
goto out;
}
@@ -908,22 +781,11 @@
client->version.major, client->version.minor,
client->version.build,
i40e_client_interface_version_str);
- mutex_unlock(&i40e_client_mutex);
ret = -EIO;
goto out;
}
- vsi_type = i40e_client_type_to_vsi_type(client->type);
- if (vsi_type == I40E_VSI_TYPE_UNKNOWN) {
- pr_info("i40e: Failed to register client %s due to unknown client type %d\n",
- client->name, client->type);
- mutex_unlock(&i40e_client_mutex);
- ret = -EIO;
- goto out;
- }
- list_add(&client->list, &i40e_clients);
- set_bit(__I40E_CLIENT_REGISTERED, &client->state);
- mutex_unlock(&i40e_client_mutex);
+ registered_client = client;
i40e_client_prepare(client);
@@ -943,29 +805,21 @@
{
int ret = 0;
- /* When a unregister request comes through we would have to send
- * a close for each of the client instances that were opened.
- * client_release function is called to handle this.
- */
- mutex_lock(&i40e_client_mutex);
- if (!client || i40e_client_release(client)) {
- ret = -EIO;
- goto out;
- }
-
- /* TODO: check if device is in reset, or if that matters? */
- if (!i40e_client_is_registered(client)) {
+ if (registered_client != client) {
pr_info("i40e: Client %s has not been registered\n",
client->name);
ret = -ENODEV;
goto out;
}
- clear_bit(__I40E_CLIENT_REGISTERED, &client->state);
- list_del(&client->list);
- pr_info("i40e: Unregistered client %s with return code %d\n",
- client->name, ret);
+ registered_client = NULL;
+ /* When a unregister request comes through we would have to send
+ * a close for each of the client instances that were opened.
+ * client_release function is called to handle this.
+ */
+ i40e_client_release(client);
+
+ pr_info("i40e: Unregistered client %s\n", client->name);
out:
- mutex_unlock(&i40e_client_mutex);
return ret;
}
EXPORT_SYMBOL(i40e_unregister_client);
diff --git a/drivers/net/ethernet/intel/i40e/i40e_client.h b/drivers/net/ethernet/intel/i40e/i40e_client.h
index 528bd79..15b21a5 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_client.h
+++ b/drivers/net/ethernet/intel/i40e/i40e_client.h
@@ -57,11 +57,6 @@
__I40E_CLIENT_INSTANCE_OPENED,
};
-enum i40e_client_type {
- I40E_CLIENT_IWARP,
- I40E_CLIENT_VMDQ2
-};
-
struct i40e_ops;
struct i40e_client;
@@ -214,7 +209,8 @@
u32 flags;
#define I40E_CLIENT_FLAGS_LAUNCH_ON_PROBE BIT(0)
#define I40E_TX_FLAGS_NOTIFY_OTHER_EVENTS BIT(2)
- enum i40e_client_type type;
+ u8 type;
+#define I40E_CLIENT_IWARP 0
const struct i40e_client_ops *ops; /* client ops provided by the client */
};
diff --git a/drivers/net/ethernet/intel/i40e/i40e_common.c b/drivers/net/ethernet/intel/i40e/i40e_common.c
index ece57d6..24f0206 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_common.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_common.c
@@ -1088,33 +1088,6 @@
wr32(hw, I40E_GLLAN_TXPRE_QDIS(reg_block), reg_val);
}
-#ifdef I40E_FCOE
-
-/**
- * i40e_get_san_mac_addr - get SAN MAC address
- * @hw: pointer to the HW structure
- * @mac_addr: pointer to SAN MAC address
- *
- * Reads the adapter's SAN MAC address from NVM
- **/
-i40e_status i40e_get_san_mac_addr(struct i40e_hw *hw, u8 *mac_addr)
-{
- struct i40e_aqc_mac_address_read_data addrs;
- i40e_status status;
- u16 flags = 0;
-
- status = i40e_aq_mac_address_read(hw, &flags, &addrs, NULL);
- if (status)
- return status;
-
- if (flags & I40E_AQC_SAN_ADDR_VALID)
- ether_addr_copy(mac_addr, addrs.pf_san_mac);
- else
- status = I40E_ERR_INVALID_MAC_ADDR;
-
- return status;
-}
-#endif
/**
* i40e_read_pba_string - Reads part number string from EEPROM
@@ -4990,7 +4963,9 @@
int retry = 5;
u32 val = 0;
- use_register = (hw->aq.api_maj_ver == 1) && (hw->aq.api_min_ver < 5);
+ use_register = (((hw->aq.api_maj_ver == 1) &&
+ (hw->aq.api_min_ver < 5)) ||
+ (hw->mac.type == I40E_MAC_X722));
if (!use_register) {
do_retry:
status = i40e_aq_rx_ctl_read_register(hw, reg_addr, &val, NULL);
@@ -5049,7 +5024,9 @@
bool use_register;
int retry = 5;
- use_register = (hw->aq.api_maj_ver == 1) && (hw->aq.api_min_ver < 5);
+ use_register = (((hw->aq.api_maj_ver == 1) &&
+ (hw->aq.api_min_ver < 5)) ||
+ (hw->mac.type == I40E_MAC_X722));
if (!use_register) {
do_retry:
status = i40e_aq_rx_ctl_write_register(hw, reg_addr,
@@ -5065,3 +5042,215 @@
if (status || use_register)
wr32(hw, reg_addr, reg_val);
}
+
+/**
+ * i40e_aq_write_ppp - Write pipeline personalization profile (ppp)
+ * @hw: pointer to the hw struct
+ * @buff: command buffer (size in bytes = buff_size)
+ * @buff_size: buffer size in bytes
+ * @track_id: package tracking id
+ * @error_offset: returns error offset
+ * @error_info: returns error information
+ * @cmd_details: pointer to command details structure or NULL
+ **/
+enum
+i40e_status_code i40e_aq_write_ppp(struct i40e_hw *hw, void *buff,
+ u16 buff_size, u32 track_id,
+ u32 *error_offset, u32 *error_info,
+ struct i40e_asq_cmd_details *cmd_details)
+{
+ struct i40e_aq_desc desc;
+ struct i40e_aqc_write_personalization_profile *cmd =
+ (struct i40e_aqc_write_personalization_profile *)
+ &desc.params.raw;
+ struct i40e_aqc_write_ppp_resp *resp;
+ i40e_status status;
+
+ i40e_fill_default_direct_cmd_desc(&desc,
+ i40e_aqc_opc_write_personalization_profile);
+
+ desc.flags |= cpu_to_le16(I40E_AQ_FLAG_BUF | I40E_AQ_FLAG_RD);
+ if (buff_size > I40E_AQ_LARGE_BUF)
+ desc.flags |= cpu_to_le16((u16)I40E_AQ_FLAG_LB);
+
+ desc.datalen = cpu_to_le16(buff_size);
+
+ cmd->profile_track_id = cpu_to_le32(track_id);
+
+ status = i40e_asq_send_command(hw, &desc, buff, buff_size, cmd_details);
+ if (!status) {
+ resp = (struct i40e_aqc_write_ppp_resp *)&desc.params.raw;
+ if (error_offset)
+ *error_offset = le32_to_cpu(resp->error_offset);
+ if (error_info)
+ *error_info = le32_to_cpu(resp->error_info);
+ }
+
+ return status;
+}
+
+/**
+ * i40e_aq_get_ppp_list - Read pipeline personalization profile (ppp)
+ * @hw: pointer to the hw struct
+ * @buff: command buffer (size in bytes = buff_size)
+ * @buff_size: buffer size in bytes
+ * @cmd_details: pointer to command details structure or NULL
+ **/
+enum
+i40e_status_code i40e_aq_get_ppp_list(struct i40e_hw *hw, void *buff,
+ u16 buff_size, u8 flags,
+ struct i40e_asq_cmd_details *cmd_details)
+{
+ struct i40e_aq_desc desc;
+ struct i40e_aqc_get_applied_profiles *cmd =
+ (struct i40e_aqc_get_applied_profiles *)&desc.params.raw;
+ i40e_status status;
+
+ i40e_fill_default_direct_cmd_desc(&desc,
+ i40e_aqc_opc_get_personalization_profile_list);
+
+ desc.flags |= cpu_to_le16((u16)I40E_AQ_FLAG_BUF);
+ if (buff_size > I40E_AQ_LARGE_BUF)
+ desc.flags |= cpu_to_le16((u16)I40E_AQ_FLAG_LB);
+ desc.datalen = cpu_to_le16(buff_size);
+
+ cmd->flags = flags;
+
+ status = i40e_asq_send_command(hw, &desc, buff, buff_size, cmd_details);
+
+ return status;
+}
+
+/**
+ * i40e_find_segment_in_package
+ * @segment_type: the segment type to search for (i.e., SEGMENT_TYPE_I40E)
+ * @pkg_hdr: pointer to the package header to be searched
+ *
+ * This function searches a package file for a particular segment type. On
+ * success it returns a pointer to the segment header, otherwise it will
+ * return NULL.
+ **/
+struct i40e_generic_seg_header *
+i40e_find_segment_in_package(u32 segment_type,
+ struct i40e_package_header *pkg_hdr)
+{
+ struct i40e_generic_seg_header *segment;
+ u32 i;
+
+ /* Search all package segments for the requested segment type */
+ for (i = 0; i < pkg_hdr->segment_count; i++) {
+ segment =
+ (struct i40e_generic_seg_header *)((u8 *)pkg_hdr +
+ pkg_hdr->segment_offset[i]);
+
+ if (segment->type == segment_type)
+ return segment;
+ }
+
+ return NULL;
+}
+
+/**
+ * i40e_write_profile
+ * @hw: pointer to the hardware structure
+ * @profile: pointer to the profile segment of the package to be downloaded
+ * @track_id: package tracking id
+ *
+ * Handles the download of a complete package.
+ */
+enum i40e_status_code
+i40e_write_profile(struct i40e_hw *hw, struct i40e_profile_segment *profile,
+ u32 track_id)
+{
+ i40e_status status = 0;
+ struct i40e_section_table *sec_tbl;
+ struct i40e_profile_section_header *sec = NULL;
+ u32 dev_cnt;
+ u32 vendor_dev_id;
+ u32 *nvm;
+ u32 section_size = 0;
+ u32 offset = 0, info = 0;
+ u32 i;
+
+ if (!track_id) {
+ i40e_debug(hw, I40E_DEBUG_PACKAGE, "Track_id can't be 0.");
+ return I40E_NOT_SUPPORTED;
+ }
+
+ dev_cnt = profile->device_table_count;
+
+ for (i = 0; i < dev_cnt; i++) {
+ vendor_dev_id = profile->device_table[i].vendor_dev_id;
+ if ((vendor_dev_id >> 16) == PCI_VENDOR_ID_INTEL)
+ if (hw->device_id == (vendor_dev_id & 0xFFFF))
+ break;
+ }
+ if (i == dev_cnt) {
+ i40e_debug(hw, I40E_DEBUG_PACKAGE, "Device doesn't support PPP");
+ return I40E_ERR_DEVICE_NOT_SUPPORTED;
+ }
+
+ nvm = (u32 *)&profile->device_table[dev_cnt];
+ sec_tbl = (struct i40e_section_table *)&nvm[nvm[0] + 1];
+
+ for (i = 0; i < sec_tbl->section_count; i++) {
+ sec = (struct i40e_profile_section_header *)((u8 *)profile +
+ sec_tbl->section_offset[i]);
+
+ /* Skip 'AQ', 'note' and 'name' sections */
+ if (sec->section.type != SECTION_TYPE_MMIO)
+ continue;
+
+ section_size = sec->section.size +
+ sizeof(struct i40e_profile_section_header);
+
+ /* Write profile */
+ status = i40e_aq_write_ppp(hw, (void *)sec, (u16)section_size,
+ track_id, &offset, &info, NULL);
+ if (status) {
+ i40e_debug(hw, I40E_DEBUG_PACKAGE,
+ "Failed to write profile: offset %d, info %d",
+ offset, info);
+ break;
+ }
+ }
+ return status;
+}
+
+/**
+ * i40e_add_pinfo_to_list
+ * @hw: pointer to the hardware structure
+ * @profile: pointer to the profile segment of the package
+ * @profile_info_sec: buffer for information section
+ * @track_id: package tracking id
+ *
+ * Register a profile to the list of loaded profiles.
+ */
+enum i40e_status_code
+i40e_add_pinfo_to_list(struct i40e_hw *hw,
+ struct i40e_profile_segment *profile,
+ u8 *profile_info_sec, u32 track_id)
+{
+ i40e_status status = 0;
+ struct i40e_profile_section_header *sec = NULL;
+ struct i40e_profile_info *pinfo;
+ u32 offset = 0, info = 0;
+
+ sec = (struct i40e_profile_section_header *)profile_info_sec;
+ sec->tbl_size = 1;
+ sec->data_end = sizeof(struct i40e_profile_section_header) +
+ sizeof(struct i40e_profile_info);
+ sec->section.type = SECTION_TYPE_INFO;
+ sec->section.offset = sizeof(struct i40e_profile_section_header);
+ sec->section.size = sizeof(struct i40e_profile_info);
+ pinfo = (struct i40e_profile_info *)(profile_info_sec +
+ sec->section.offset);
+ pinfo->track_id = track_id;
+ pinfo->version = profile->version;
+ pinfo->op = I40E_PPP_ADD_TRACKID;
+ memcpy(pinfo->name, profile->name, I40E_PPP_NAME_SIZE);
+
+ status = i40e_aq_write_ppp(hw, (void *)sec, sec->data_end,
+ track_id, &offset, &info, NULL);
+ return status;
+}
diff --git a/drivers/net/ethernet/intel/i40e/i40e_debugfs.c b/drivers/net/ethernet/intel/i40e/i40e_debugfs.c
index 267ad25..8f326f8 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_debugfs.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_debugfs.c
@@ -158,9 +158,12 @@
dev_info(&pf->pdev->dev,
" vlgrp: & = %p\n", vsi->active_vlans);
dev_info(&pf->pdev->dev,
- " state = %li flags = 0x%08lx, netdev_registered = %i, current_netdev_flags = 0x%04x\n",
- vsi->state, vsi->flags,
- vsi->netdev_registered, vsi->current_netdev_flags);
+ " flags = 0x%08lx, netdev_registered = %i, current_netdev_flags = 0x%04x\n",
+ vsi->flags, vsi->netdev_registered, vsi->current_netdev_flags);
+ for (i = 0; i < BITS_TO_LONGS(__I40E_VSI_STATE_SIZE__); i++)
+ dev_info(&pf->pdev->dev,
+ " state[%d] = %08lx\n",
+ i, vsi->state[i]);
if (vsi == pf->vsi[pf->lan_vsi])
dev_info(&pf->pdev->dev, " MAC address: %pM SAN MAC: %pM Port MAC: %pM\n",
pf->hw.mac.addr,
@@ -174,7 +177,7 @@
}
dev_info(&pf->pdev->dev, " active_filters %u, promisc_threshold %u, overflow promisc %s\n",
vsi->active_filters, vsi->promisc_threshold,
- (test_bit(__I40E_FILTER_OVERFLOW_PROMISC, &vsi->state) ?
+ (test_bit(__I40E_VSI_OVERFLOW_PROMISC, vsi->state) ?
"ON" : "OFF"));
nstat = i40e_get_vsi_stats_struct(vsi);
dev_info(&pf->pdev->dev,
@@ -384,6 +387,8 @@
" base_queue = %d, num_queue_pairs = %d, num_desc = %d\n",
vsi->base_queue, vsi->num_queue_pairs, vsi->num_desc);
dev_info(&pf->pdev->dev, " type = %i\n", vsi->type);
+ if (vsi->type == I40E_VSI_SRIOV)
+ dev_info(&pf->pdev->dev, " VF ID = %i\n", vsi->vf_id);
dev_info(&pf->pdev->dev,
" info: valid_sections = 0x%04x, switch_id = 0x%04x\n",
vsi->info.valid_sections, vsi->info.switch_id);
@@ -484,25 +489,6 @@
vsi->bw_ets_limit_credits[i],
vsi->bw_ets_max_quanta[i]);
}
-#ifdef I40E_FCOE
- if (vsi->type == I40E_VSI_FCOE) {
- dev_info(&pf->pdev->dev,
- " fcoe_stats: rx_packets = %llu, rx_dwords = %llu, rx_dropped = %llu\n",
- vsi->fcoe_stats.rx_fcoe_packets,
- vsi->fcoe_stats.rx_fcoe_dwords,
- vsi->fcoe_stats.rx_fcoe_dropped);
- dev_info(&pf->pdev->dev,
- " fcoe_stats: tx_packets = %llu, tx_dwords = %llu\n",
- vsi->fcoe_stats.tx_fcoe_packets,
- vsi->fcoe_stats.tx_fcoe_dwords);
- dev_info(&pf->pdev->dev,
- " fcoe_stats: bad_crc = %llu, last_error = %llu\n",
- vsi->fcoe_stats.fcoe_bad_fccrc,
- vsi->fcoe_stats.fcoe_last_error);
- dev_info(&pf->pdev->dev, " fcoe_stats: ddp_count = %llu\n",
- vsi->fcoe_stats.fcoe_ddp_count);
- }
-#endif
}
/**
@@ -713,6 +699,47 @@
}
}
+/**
+ * i40e_dbg_dump_vf - dump VF info
+ * @pf: the i40e_pf created in command write
+ * @vf_id: the vf_id from the user
+ **/
+static void i40e_dbg_dump_vf(struct i40e_pf *pf, int vf_id)
+{
+ struct i40e_vf *vf;
+ struct i40e_vsi *vsi;
+
+ if (!pf->num_alloc_vfs) {
+ dev_info(&pf->pdev->dev, "no VFs allocated\n");
+ } else if ((vf_id >= 0) && (vf_id < pf->num_alloc_vfs)) {
+ vf = &pf->vf[vf_id];
+ vsi = pf->vsi[vf->lan_vsi_idx];
+ dev_info(&pf->pdev->dev, "vf %2d: VSI id=%d, seid=%d, qps=%d\n",
+ vf_id, vf->lan_vsi_id, vsi->seid, vf->num_queue_pairs);
+ dev_info(&pf->pdev->dev, " num MDD=%lld, invalid msg=%lld, valid msg=%lld\n",
+ vf->num_mdd_events,
+ vf->num_invalid_msgs,
+ vf->num_valid_msgs);
+ } else {
+ dev_info(&pf->pdev->dev, "invalid VF id %d\n", vf_id);
+ }
+}
+
+/**
+ * i40e_dbg_dump_vf_all - dump VF info for all VFs
+ * @pf: the i40e_pf created in command write
+ **/
+static void i40e_dbg_dump_vf_all(struct i40e_pf *pf)
+{
+ int i;
+
+ if (!pf->num_alloc_vfs)
+ dev_info(&pf->pdev->dev, "no VFs enabled!\n");
+ else
+ for (i = 0; i < pf->num_alloc_vfs; i++)
+ i40e_dbg_dump_vf(pf, i);
+}
+
#define I40E_MAX_DEBUG_OUT_BUFFER (4096*4)
/**
* i40e_dbg_command_write - write into command datum
@@ -731,6 +758,7 @@
struct i40e_vsi *vsi;
int vsi_seid;
int veb_seid;
+ int vf_id;
int cnt;
/* don't allow partial writes */
@@ -933,6 +961,12 @@
i40e_dbg_dump_veb_seid(pf, vsi_seid);
else
i40e_dbg_dump_veb_all(pf);
+ } else if (strncmp(&cmd_buf[5], "vf", 2) == 0) {
+ cnt = sscanf(&cmd_buf[7], "%i", &vf_id);
+ if (cnt > 0)
+ i40e_dbg_dump_vf(pf, vf_id);
+ else
+ i40e_dbg_dump_vf_all(pf);
} else if (strncmp(&cmd_buf[5], "desc", 4) == 0) {
int ring_id, desc_n;
if (strncmp(&cmd_buf[10], "rx", 2) == 0) {
@@ -1128,6 +1162,7 @@
dev_info(&pf->pdev->dev, "dump vsi [seid]\n");
dev_info(&pf->pdev->dev, "dump reset stats\n");
dev_info(&pf->pdev->dev, "dump port\n");
+ dev_info(&pf->pdev->dev, "dump vf [vf_id]\n");
dev_info(&pf->pdev->dev,
"dump debug fwdata <cluster_id> <table_id> <index>\n");
}
@@ -1674,7 +1709,7 @@
} else if (!vsi->netdev) {
dev_info(&pf->pdev->dev, "tx_timeout: no netdev for VSI %d\n",
vsi_seid);
- } else if (test_bit(__I40E_DOWN, &vsi->state)) {
+ } else if (test_bit(__I40E_VSI_DOWN, vsi->state)) {
dev_info(&pf->pdev->dev, "tx_timeout: VSI %d not UP\n",
vsi_seid);
} else if (rtnl_trylock()) {
diff --git a/drivers/net/ethernet/intel/i40e/i40e_ethtool.c b/drivers/net/ethernet/intel/i40e/i40e_ethtool.c
index a22e262..7a8eb48 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_ethtool.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_ethtool.c
@@ -89,7 +89,6 @@
I40E_VSI_STAT("rx_unknown_protocol", eth_stats.rx_unknown_protocol),
I40E_VSI_STAT("tx_linearize", tx_linearize),
I40E_VSI_STAT("tx_force_wb", tx_force_wb),
- I40E_VSI_STAT("tx_lost_interrupt", tx_lost_interrupt),
I40E_VSI_STAT("rx_alloc_fail", rx_buf_failed),
I40E_VSI_STAT("rx_pg_alloc_fail", rx_page_failed),
};
@@ -162,19 +161,6 @@
I40E_PF_STAT("rx_lpi_count", stats.rx_lpi_count),
};
-#ifdef I40E_FCOE
-static const struct i40e_stats i40e_gstrings_fcoe_stats[] = {
- I40E_VSI_STAT("fcoe_bad_fccrc", fcoe_stats.fcoe_bad_fccrc),
- I40E_VSI_STAT("rx_fcoe_dropped", fcoe_stats.rx_fcoe_dropped),
- I40E_VSI_STAT("rx_fcoe_packets", fcoe_stats.rx_fcoe_packets),
- I40E_VSI_STAT("rx_fcoe_dwords", fcoe_stats.rx_fcoe_dwords),
- I40E_VSI_STAT("fcoe_ddp_count", fcoe_stats.fcoe_ddp_count),
- I40E_VSI_STAT("fcoe_last_error", fcoe_stats.fcoe_last_error),
- I40E_VSI_STAT("tx_fcoe_packets", fcoe_stats.tx_fcoe_packets),
- I40E_VSI_STAT("tx_fcoe_dwords", fcoe_stats.tx_fcoe_dwords),
-};
-
-#endif /* I40E_FCOE */
#define I40E_QUEUE_STATS_LEN(n) \
(((struct i40e_netdev_priv *)netdev_priv((n)))->vsi->num_queue_pairs \
* 2 /* Tx and Rx together */ \
@@ -182,17 +168,9 @@
#define I40E_GLOBAL_STATS_LEN ARRAY_SIZE(i40e_gstrings_stats)
#define I40E_NETDEV_STATS_LEN ARRAY_SIZE(i40e_gstrings_net_stats)
#define I40E_MISC_STATS_LEN ARRAY_SIZE(i40e_gstrings_misc_stats)
-#ifdef I40E_FCOE
-#define I40E_FCOE_STATS_LEN ARRAY_SIZE(i40e_gstrings_fcoe_stats)
-#define I40E_VSI_STATS_LEN(n) (I40E_NETDEV_STATS_LEN + \
- I40E_FCOE_STATS_LEN + \
- I40E_MISC_STATS_LEN + \
- I40E_QUEUE_STATS_LEN((n)))
-#else
#define I40E_VSI_STATS_LEN(n) (I40E_NETDEV_STATS_LEN + \
I40E_MISC_STATS_LEN + \
I40E_QUEUE_STATS_LEN((n)))
-#endif /* I40E_FCOE */
#define I40E_PFC_STATS_LEN ( \
(FIELD_SIZEOF(struct i40e_pf, stats.priority_xoff_rx) + \
FIELD_SIZEOF(struct i40e_pf, stats.priority_xon_rx) + \
@@ -228,22 +206,37 @@
#define I40E_TEST_LEN (sizeof(i40e_gstrings_test) / ETH_GSTRING_LEN)
-static const char i40e_priv_flags_strings[][ETH_GSTRING_LEN] = {
- "MFP",
- "LinkPolling",
- "flow-director-atr",
- "veb-stats",
- "hw-atr-eviction",
+struct i40e_priv_flags {
+ char flag_string[ETH_GSTRING_LEN];
+ u64 flag;
+ bool read_only;
};
-#define I40E_PRIV_FLAGS_STR_LEN ARRAY_SIZE(i40e_priv_flags_strings)
+#define I40E_PRIV_FLAG(_name, _flag, _read_only) { \
+ .flag_string = _name, \
+ .flag = _flag, \
+ .read_only = _read_only, \
+}
+
+static const struct i40e_priv_flags i40e_gstrings_priv_flags[] = {
+ /* NOTE: MFP setting cannot be changed */
+ I40E_PRIV_FLAG("MFP", I40E_FLAG_MFP_ENABLED, 1),
+ I40E_PRIV_FLAG("LinkPolling", I40E_FLAG_LINK_POLLING_ENABLED, 0),
+ I40E_PRIV_FLAG("flow-director-atr", I40E_FLAG_FD_ATR_ENABLED, 0),
+ I40E_PRIV_FLAG("veb-stats", I40E_FLAG_VEB_STATS_ENABLED, 0),
+ I40E_PRIV_FLAG("hw-atr-eviction", I40E_FLAG_HW_ATR_EVICT_CAPABLE, 0),
+ I40E_PRIV_FLAG("legacy-rx", I40E_FLAG_LEGACY_RX, 0),
+};
+
+#define I40E_PRIV_FLAGS_STR_LEN ARRAY_SIZE(i40e_gstrings_priv_flags)
/* Private flags with a global effect, restricted to PF 0 */
-static const char i40e_gl_priv_flags_strings[][ETH_GSTRING_LEN] = {
- "vf-true-promisc-support",
+static const struct i40e_priv_flags i40e_gl_gstrings_priv_flags[] = {
+ I40E_PRIV_FLAG("vf-true-promisc-support",
+ I40E_FLAG_TRUE_PROMISC_SUPPORT, 0),
};
-#define I40E_GL_PRIV_FLAGS_STR_LEN ARRAY_SIZE(i40e_gl_priv_flags_strings)
+#define I40E_GL_PRIV_FLAGS_STR_LEN ARRAY_SIZE(i40e_gl_gstrings_priv_flags)
/**
* i40e_partition_setting_complaint - generic complaint for MFP restriction
@@ -387,7 +380,7 @@
*
**/
static void i40e_get_settings_link_up(struct i40e_hw *hw,
- struct ethtool_cmd *ecmd,
+ struct ethtool_link_ksettings *cmd,
struct net_device *netdev,
struct i40e_pf *pf)
{
@@ -395,90 +388,96 @@
u32 link_speed = hw_link_info->link_speed;
u32 e_advertising = 0x0;
u32 e_supported = 0x0;
+ u32 supported, advertising;
+
+ ethtool_convert_link_mode_to_legacy_u32(&supported,
+ cmd->link_modes.supported);
+ ethtool_convert_link_mode_to_legacy_u32(&advertising,
+ cmd->link_modes.advertising);
/* Initialize supported and advertised settings based on phy settings */
switch (hw_link_info->phy_type) {
case I40E_PHY_TYPE_40GBASE_CR4:
case I40E_PHY_TYPE_40GBASE_CR4_CU:
- ecmd->supported = SUPPORTED_Autoneg |
- SUPPORTED_40000baseCR4_Full;
- ecmd->advertising = ADVERTISED_Autoneg |
- ADVERTISED_40000baseCR4_Full;
+ supported = SUPPORTED_Autoneg |
+ SUPPORTED_40000baseCR4_Full;
+ advertising = ADVERTISED_Autoneg |
+ ADVERTISED_40000baseCR4_Full;
break;
case I40E_PHY_TYPE_XLAUI:
case I40E_PHY_TYPE_XLPPI:
case I40E_PHY_TYPE_40GBASE_AOC:
- ecmd->supported = SUPPORTED_40000baseCR4_Full;
+ supported = SUPPORTED_40000baseCR4_Full;
break;
case I40E_PHY_TYPE_40GBASE_SR4:
- ecmd->supported = SUPPORTED_40000baseSR4_Full;
+ supported = SUPPORTED_40000baseSR4_Full;
break;
case I40E_PHY_TYPE_40GBASE_LR4:
- ecmd->supported = SUPPORTED_40000baseLR4_Full;
+ supported = SUPPORTED_40000baseLR4_Full;
break;
case I40E_PHY_TYPE_10GBASE_SR:
case I40E_PHY_TYPE_10GBASE_LR:
case I40E_PHY_TYPE_1000BASE_SX:
case I40E_PHY_TYPE_1000BASE_LX:
- ecmd->supported = SUPPORTED_10000baseT_Full;
+ supported = SUPPORTED_10000baseT_Full;
if (hw_link_info->module_type[2] &
I40E_MODULE_TYPE_1000BASE_SX ||
hw_link_info->module_type[2] &
I40E_MODULE_TYPE_1000BASE_LX) {
- ecmd->supported |= SUPPORTED_1000baseT_Full;
+ supported |= SUPPORTED_1000baseT_Full;
if (hw_link_info->requested_speeds &
I40E_LINK_SPEED_1GB)
- ecmd->advertising |= ADVERTISED_1000baseT_Full;
+ advertising |= ADVERTISED_1000baseT_Full;
}
if (hw_link_info->requested_speeds & I40E_LINK_SPEED_10GB)
- ecmd->advertising |= ADVERTISED_10000baseT_Full;
+ advertising |= ADVERTISED_10000baseT_Full;
break;
case I40E_PHY_TYPE_10GBASE_T:
case I40E_PHY_TYPE_1000BASE_T:
case I40E_PHY_TYPE_100BASE_TX:
- ecmd->supported = SUPPORTED_Autoneg |
- SUPPORTED_10000baseT_Full |
- SUPPORTED_1000baseT_Full |
- SUPPORTED_100baseT_Full;
- ecmd->advertising = ADVERTISED_Autoneg;
+ supported = SUPPORTED_Autoneg |
+ SUPPORTED_10000baseT_Full |
+ SUPPORTED_1000baseT_Full |
+ SUPPORTED_100baseT_Full;
+ advertising = ADVERTISED_Autoneg;
if (hw_link_info->requested_speeds & I40E_LINK_SPEED_10GB)
- ecmd->advertising |= ADVERTISED_10000baseT_Full;
+ advertising |= ADVERTISED_10000baseT_Full;
if (hw_link_info->requested_speeds & I40E_LINK_SPEED_1GB)
- ecmd->advertising |= ADVERTISED_1000baseT_Full;
+ advertising |= ADVERTISED_1000baseT_Full;
if (hw_link_info->requested_speeds & I40E_LINK_SPEED_100MB)
- ecmd->advertising |= ADVERTISED_100baseT_Full;
+ advertising |= ADVERTISED_100baseT_Full;
break;
case I40E_PHY_TYPE_1000BASE_T_OPTICAL:
- ecmd->supported = SUPPORTED_Autoneg |
- SUPPORTED_1000baseT_Full;
- ecmd->advertising = ADVERTISED_Autoneg |
- ADVERTISED_1000baseT_Full;
+ supported = SUPPORTED_Autoneg |
+ SUPPORTED_1000baseT_Full;
+ advertising = ADVERTISED_Autoneg |
+ ADVERTISED_1000baseT_Full;
break;
case I40E_PHY_TYPE_10GBASE_CR1_CU:
case I40E_PHY_TYPE_10GBASE_CR1:
- ecmd->supported = SUPPORTED_Autoneg |
- SUPPORTED_10000baseT_Full;
- ecmd->advertising = ADVERTISED_Autoneg |
- ADVERTISED_10000baseT_Full;
+ supported = SUPPORTED_Autoneg |
+ SUPPORTED_10000baseT_Full;
+ advertising = ADVERTISED_Autoneg |
+ ADVERTISED_10000baseT_Full;
break;
case I40E_PHY_TYPE_XAUI:
case I40E_PHY_TYPE_XFI:
case I40E_PHY_TYPE_SFI:
case I40E_PHY_TYPE_10GBASE_SFPP_CU:
case I40E_PHY_TYPE_10GBASE_AOC:
- ecmd->supported = SUPPORTED_10000baseT_Full;
- ecmd->advertising = SUPPORTED_10000baseT_Full;
+ supported = SUPPORTED_10000baseT_Full;
+ advertising = SUPPORTED_10000baseT_Full;
break;
case I40E_PHY_TYPE_SGMII:
- ecmd->supported = SUPPORTED_Autoneg |
- SUPPORTED_1000baseT_Full;
+ supported = SUPPORTED_Autoneg |
+ SUPPORTED_1000baseT_Full;
if (hw_link_info->requested_speeds & I40E_LINK_SPEED_1GB)
- ecmd->advertising |= ADVERTISED_1000baseT_Full;
+ advertising |= ADVERTISED_1000baseT_Full;
if (pf->flags & I40E_FLAG_100M_SGMII_CAPABLE) {
- ecmd->supported |= SUPPORTED_100baseT_Full;
+ supported |= SUPPORTED_100baseT_Full;
if (hw_link_info->requested_speeds &
I40E_LINK_SPEED_100MB)
- ecmd->advertising |= ADVERTISED_100baseT_Full;
+ advertising |= ADVERTISED_100baseT_Full;
}
break;
case I40E_PHY_TYPE_40GBASE_KR4:
@@ -486,25 +485,25 @@
case I40E_PHY_TYPE_10GBASE_KR:
case I40E_PHY_TYPE_10GBASE_KX4:
case I40E_PHY_TYPE_1000BASE_KX:
- ecmd->supported |= SUPPORTED_40000baseKR4_Full |
- SUPPORTED_20000baseKR2_Full |
- SUPPORTED_10000baseKR_Full |
- SUPPORTED_10000baseKX4_Full |
- SUPPORTED_1000baseKX_Full |
- SUPPORTED_Autoneg;
- ecmd->advertising |= ADVERTISED_40000baseKR4_Full |
- ADVERTISED_20000baseKR2_Full |
- ADVERTISED_10000baseKR_Full |
- ADVERTISED_10000baseKX4_Full |
- ADVERTISED_1000baseKX_Full |
- ADVERTISED_Autoneg;
+ supported |= SUPPORTED_40000baseKR4_Full |
+ SUPPORTED_20000baseKR2_Full |
+ SUPPORTED_10000baseKR_Full |
+ SUPPORTED_10000baseKX4_Full |
+ SUPPORTED_1000baseKX_Full |
+ SUPPORTED_Autoneg;
+ advertising |= ADVERTISED_40000baseKR4_Full |
+ ADVERTISED_20000baseKR2_Full |
+ ADVERTISED_10000baseKR_Full |
+ ADVERTISED_10000baseKX4_Full |
+ ADVERTISED_1000baseKX_Full |
+ ADVERTISED_Autoneg;
break;
case I40E_PHY_TYPE_25GBASE_KR:
case I40E_PHY_TYPE_25GBASE_CR:
case I40E_PHY_TYPE_25GBASE_SR:
case I40E_PHY_TYPE_25GBASE_LR:
- ecmd->supported = SUPPORTED_Autoneg;
- ecmd->advertising = ADVERTISED_Autoneg;
+ supported = SUPPORTED_Autoneg;
+ advertising = ADVERTISED_Autoneg;
/* TODO: add speeds when ethtool is ready to support*/
break;
default:
@@ -520,38 +519,43 @@
i40e_phy_type_to_ethtool(pf, &e_supported,
&e_advertising);
- ecmd->supported = ecmd->supported & e_supported;
- ecmd->advertising = ecmd->advertising & e_advertising;
+ supported = supported & e_supported;
+ advertising = advertising & e_advertising;
/* Set speed and duplex */
switch (link_speed) {
case I40E_LINK_SPEED_40GB:
- ethtool_cmd_speed_set(ecmd, SPEED_40000);
+ cmd->base.speed = SPEED_40000;
break;
case I40E_LINK_SPEED_25GB:
#ifdef SPEED_25000
- ethtool_cmd_speed_set(ecmd, SPEED_25000);
+ cmd->base.speed = SPEED_25000;
#else
netdev_info(netdev,
"Speed is 25G, display not supported by this version of ethtool.\n");
#endif
break;
case I40E_LINK_SPEED_20GB:
- ethtool_cmd_speed_set(ecmd, SPEED_20000);
+ cmd->base.speed = SPEED_20000;
break;
case I40E_LINK_SPEED_10GB:
- ethtool_cmd_speed_set(ecmd, SPEED_10000);
+ cmd->base.speed = SPEED_10000;
break;
case I40E_LINK_SPEED_1GB:
- ethtool_cmd_speed_set(ecmd, SPEED_1000);
+ cmd->base.speed = SPEED_1000;
break;
case I40E_LINK_SPEED_100MB:
- ethtool_cmd_speed_set(ecmd, SPEED_100);
+ cmd->base.speed = SPEED_100;
break;
default:
break;
}
- ecmd->duplex = DUPLEX_FULL;
+ cmd->base.duplex = DUPLEX_FULL;
+
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
+ advertising);
}
/**
@@ -562,18 +566,24 @@
* Reports link settings that can be determined when link is down
**/
static void i40e_get_settings_link_down(struct i40e_hw *hw,
- struct ethtool_cmd *ecmd,
+ struct ethtool_link_ksettings *cmd,
struct i40e_pf *pf)
{
+ u32 supported, advertising;
+
/* link is down and the driver needs to fall back on
* supported phy types to figure out what info to display
*/
- i40e_phy_type_to_ethtool(pf, &ecmd->supported,
- &ecmd->advertising);
+ i40e_phy_type_to_ethtool(pf, &supported, &advertising);
+
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
+ advertising);
/* With no link speed and duplex are unknown */
- ethtool_cmd_speed_set(ecmd, SPEED_UNKNOWN);
- ecmd->duplex = DUPLEX_UNKNOWN;
+ cmd->base.speed = SPEED_UNKNOWN;
+ cmd->base.duplex = DUPLEX_UNKNOWN;
}
/**
@@ -583,74 +593,85 @@
*
* Reports speed/duplex settings based on media_type
**/
-static int i40e_get_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+static int i40e_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
{
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_pf *pf = np->vsi->back;
struct i40e_hw *hw = &pf->hw;
struct i40e_link_status *hw_link_info = &hw->phy.link_info;
bool link_up = hw_link_info->link_info & I40E_AQ_LINK_UP;
+ u32 advertising;
if (link_up)
- i40e_get_settings_link_up(hw, ecmd, netdev, pf);
+ i40e_get_settings_link_up(hw, cmd, netdev, pf);
else
- i40e_get_settings_link_down(hw, ecmd, pf);
+ i40e_get_settings_link_down(hw, cmd, pf);
/* Now set the settings that don't rely on link being up/down */
/* Set autoneg settings */
- ecmd->autoneg = ((hw_link_info->an_info & I40E_AQ_AN_COMPLETED) ?
+ cmd->base.autoneg = ((hw_link_info->an_info & I40E_AQ_AN_COMPLETED) ?
AUTONEG_ENABLE : AUTONEG_DISABLE);
switch (hw->phy.media_type) {
case I40E_MEDIA_TYPE_BACKPLANE:
- ecmd->supported |= SUPPORTED_Autoneg |
- SUPPORTED_Backplane;
- ecmd->advertising |= ADVERTISED_Autoneg |
- ADVERTISED_Backplane;
- ecmd->port = PORT_NONE;
+ ethtool_link_ksettings_add_link_mode(cmd, supported,
+ Autoneg);
+ ethtool_link_ksettings_add_link_mode(cmd, supported,
+ Backplane);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ Autoneg);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ Backplane);
+ cmd->base.port = PORT_NONE;
break;
case I40E_MEDIA_TYPE_BASET:
- ecmd->supported |= SUPPORTED_TP;
- ecmd->advertising |= ADVERTISED_TP;
- ecmd->port = PORT_TP;
+ ethtool_link_ksettings_add_link_mode(cmd, supported, TP);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, TP);
+ cmd->base.port = PORT_TP;
break;
case I40E_MEDIA_TYPE_DA:
case I40E_MEDIA_TYPE_CX4:
- ecmd->supported |= SUPPORTED_FIBRE;
- ecmd->advertising |= ADVERTISED_FIBRE;
- ecmd->port = PORT_DA;
+ ethtool_link_ksettings_add_link_mode(cmd, supported, FIBRE);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, FIBRE);
+ cmd->base.port = PORT_DA;
break;
case I40E_MEDIA_TYPE_FIBER:
- ecmd->supported |= SUPPORTED_FIBRE;
- ecmd->port = PORT_FIBRE;
+ ethtool_link_ksettings_add_link_mode(cmd, supported, FIBRE);
+ cmd->base.port = PORT_FIBRE;
break;
case I40E_MEDIA_TYPE_UNKNOWN:
default:
- ecmd->port = PORT_OTHER;
+ cmd->base.port = PORT_OTHER;
break;
}
- /* Set transceiver */
- ecmd->transceiver = XCVR_EXTERNAL;
-
/* Set flow control settings */
- ecmd->supported |= SUPPORTED_Pause;
+ ethtool_link_ksettings_add_link_mode(cmd, supported, Pause);
switch (hw->fc.requested_mode) {
case I40E_FC_FULL:
- ecmd->advertising |= ADVERTISED_Pause;
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ Pause);
break;
case I40E_FC_TX_PAUSE:
- ecmd->advertising |= ADVERTISED_Asym_Pause;
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ Asym_Pause);
break;
case I40E_FC_RX_PAUSE:
- ecmd->advertising |= (ADVERTISED_Pause |
- ADVERTISED_Asym_Pause);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ Pause);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ Asym_Pause);
break;
default:
- ecmd->advertising &= ~(ADVERTISED_Pause |
- ADVERTISED_Asym_Pause);
+ ethtool_convert_link_mode_to_legacy_u32(
+ &advertising, cmd->link_modes.advertising);
+
+ advertising &= ~(ADVERTISED_Pause | ADVERTISED_Asym_Pause);
+
+ ethtool_convert_legacy_u32_to_link_mode(
+ cmd->link_modes.advertising, advertising);
break;
}
@@ -664,8 +685,8 @@
*
* Set speed/duplex per media_types advertised/forced
**/
-static int i40e_set_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+static int i40e_set_link_ksettings(struct net_device *netdev,
+ const struct ethtool_link_ksettings *cmd)
{
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_aq_get_phy_abilities_resp abilities;
@@ -673,12 +694,15 @@
struct i40e_pf *pf = np->vsi->back;
struct i40e_vsi *vsi = np->vsi;
struct i40e_hw *hw = &pf->hw;
- struct ethtool_cmd safe_ecmd;
+ struct ethtool_link_ksettings safe_cmd;
+ struct ethtool_link_ksettings copy_cmd;
i40e_status status = 0;
bool change = false;
+ int timeout = 50;
int err = 0;
- u8 autoneg;
+ u32 autoneg;
u32 advertise;
+ u32 tmp;
/* Changing port settings is not supported if this isn't the
* port's controlling PF
@@ -706,33 +730,47 @@
return -EOPNOTSUPP;
}
- /* get our own copy of the bits to check against */
- memset(&safe_ecmd, 0, sizeof(struct ethtool_cmd));
- i40e_get_settings(netdev, &safe_ecmd);
+ /* copy the cmd to copy_cmd to avoid modifying the origin */
+ memcpy(©_cmd, cmd, sizeof(struct ethtool_link_ksettings));
- /* save autoneg and speed out of ecmd */
- autoneg = ecmd->autoneg;
- advertise = ecmd->advertising;
+ /* get our own copy of the bits to check against */
+ memset(&safe_cmd, 0, sizeof(struct ethtool_link_ksettings));
+ i40e_get_link_ksettings(netdev, &safe_cmd);
+
+ /* save autoneg and speed out of cmd */
+ autoneg = cmd->base.autoneg;
+ ethtool_convert_link_mode_to_legacy_u32(&advertise,
+ cmd->link_modes.advertising);
/* set autoneg and speed back to what they currently are */
- ecmd->autoneg = safe_ecmd.autoneg;
- ecmd->advertising = safe_ecmd.advertising;
+ copy_cmd.base.autoneg = safe_cmd.base.autoneg;
+ ethtool_convert_link_mode_to_legacy_u32(
+ &tmp, safe_cmd.link_modes.advertising);
+ ethtool_convert_legacy_u32_to_link_mode(
+ copy_cmd.link_modes.advertising, tmp);
- ecmd->cmd = safe_ecmd.cmd;
- /* If ecmd and safe_ecmd are not the same now, then they are
+ copy_cmd.base.cmd = safe_cmd.base.cmd;
+
+ /* If copy_cmd and safe_cmd are not the same now, then they are
* trying to set something that we do not support
*/
- if (memcmp(ecmd, &safe_ecmd, sizeof(struct ethtool_cmd)))
+ if (memcmp(©_cmd, &safe_cmd, sizeof(struct ethtool_link_ksettings)))
return -EOPNOTSUPP;
- while (test_bit(__I40E_CONFIG_BUSY, &vsi->state))
+ while (test_and_set_bit(__I40E_CONFIG_BUSY, pf->state)) {
+ timeout--;
+ if (!timeout)
+ return -EBUSY;
usleep_range(1000, 2000);
+ }
/* Get the current phy config */
status = i40e_aq_get_phy_capabilities(hw, false, false, &abilities,
NULL);
- if (status)
- return -EAGAIN;
+ if (status) {
+ err = -EAGAIN;
+ goto done;
+ }
/* Copy abilities to config in case autoneg is not
* set below
@@ -745,9 +783,11 @@
/* If autoneg was not already enabled */
if (!(hw->phy.link_info.an_info & I40E_AQ_AN_COMPLETED)) {
/* If autoneg is not supported, return error */
- if (!(safe_ecmd.supported & SUPPORTED_Autoneg)) {
+ if (!ethtool_link_ksettings_test_link_mode(
+ &safe_cmd, supported, Autoneg)) {
netdev_info(netdev, "Autoneg not supported on this phy\n");
- return -EINVAL;
+ err = -EINVAL;
+ goto done;
}
/* Autoneg is allowed to change */
config.abilities = abilities.abilities |
@@ -760,11 +800,13 @@
/* If autoneg is supported 10GBASE_T is the only PHY
* that can disable it, so otherwise return error
*/
- if (safe_ecmd.supported & SUPPORTED_Autoneg &&
+ if (ethtool_link_ksettings_test_link_mode(
+ &safe_cmd, supported, Autoneg) &&
hw->phy.link_info.phy_type !=
I40E_PHY_TYPE_10GBASE_T) {
netdev_info(netdev, "Autoneg cannot be disabled on this phy\n");
- return -EINVAL;
+ err = -EINVAL;
+ goto done;
}
/* Autoneg is allowed to change */
config.abilities = abilities.abilities &
@@ -773,8 +815,12 @@
}
}
- if (advertise & ~safe_ecmd.supported)
- return -EINVAL;
+ ethtool_convert_link_mode_to_legacy_u32(&tmp,
+ safe_cmd.link_modes.supported);
+ if (advertise & ~tmp) {
+ err = -EINVAL;
+ goto done;
+ }
if (advertise & ADVERTISED_100baseT_Full)
config.link_speed |= I40E_LINK_SPEED_100MB;
@@ -830,7 +876,8 @@
netdev_info(netdev, "Set phy config failed, err %s aq_err %s\n",
i40e_stat_str(hw, status),
i40e_aq_str(hw, hw->aq.asq_last_status));
- return -EAGAIN;
+ err = -EAGAIN;
+ goto done;
}
status = i40e_update_link_info(hw);
@@ -843,6 +890,9 @@
netdev_info(netdev, "Nothing changed, exiting without setting anything.\n");
}
+done:
+ clear_bit(__I40E_CONFIG_BUSY, pf->state);
+
return err;
}
@@ -937,7 +987,7 @@
}
/* If we have link and don't have autoneg */
- if (!test_bit(__I40E_DOWN, &pf->state) &&
+ if (!test_bit(__I40E_DOWN, pf->state) &&
!(hw_link_info->an_info & I40E_AQ_AN_COMPLETED)) {
/* Send message that it might not necessarily work*/
netdev_info(netdev, "Autoneg did not complete so changing settings may not result in an actual change.\n");
@@ -989,10 +1039,10 @@
err = -EAGAIN;
}
- if (!test_bit(__I40E_DOWN, &pf->state)) {
+ if (!test_bit(__I40E_DOWN, pf->state)) {
/* Give it a little more time to try to come back */
msleep(75);
- if (!test_bit(__I40E_DOWN, &pf->state))
+ if (!test_bit(__I40E_DOWN, pf->state))
return i40e_nway_reset(netdev);
}
@@ -1089,8 +1139,8 @@
/* make sure it is the right magic for NVMUpdate */
if ((eeprom->magic >> 16) != hw->device_id)
errno = -EINVAL;
- else if (test_bit(__I40E_RESET_RECOVERY_PENDING, &pf->state) ||
- test_bit(__I40E_RESET_INTR_RECEIVED, &pf->state))
+ else if (test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state) ||
+ test_bit(__I40E_RESET_INTR_RECEIVED, pf->state))
errno = -EBUSY;
else
ret_val = i40e_nvmupd_command(hw, cmd, bytes, &errno);
@@ -1165,6 +1215,11 @@
struct i40e_hw *hw = &np->vsi->back->hw;
u32 val;
+#define X722_EEPROM_SCOPE_LIMIT 0x5B9FFF
+ if (hw->mac.type == I40E_MAC_X722) {
+ val = X722_EEPROM_SCOPE_LIMIT + 1;
+ return val;
+ }
val = (rd32(hw, I40E_GLPCI_LBARCTRL)
& I40E_GLPCI_LBARCTRL_FL_SIZE_MASK)
>> I40E_GLPCI_LBARCTRL_FL_SIZE_SHIFT;
@@ -1191,8 +1246,8 @@
/* check for NVMUpdate access method */
else if (!eeprom->magic || (eeprom->magic >> 16) != hw->device_id)
errno = -EINVAL;
- else if (test_bit(__I40E_RESET_RECOVERY_PENDING, &pf->state) ||
- test_bit(__I40E_RESET_INTR_RECEIVED, &pf->state))
+ else if (test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state) ||
+ test_bit(__I40E_RESET_INTR_RECEIVED, pf->state))
errno = -EBUSY;
else
ret_val = i40e_nvmupd_command(hw, cmd, bytes, &errno);
@@ -1252,6 +1307,7 @@
struct i40e_vsi *vsi = np->vsi;
struct i40e_pf *pf = vsi->back;
u32 new_rx_count, new_tx_count;
+ int timeout = 50;
int i, err = 0;
if ((ring->rx_mini_pending) || (ring->rx_jumbo_pending))
@@ -1276,8 +1332,12 @@
(new_rx_count == vsi->rx_rings[0]->count))
return 0;
- while (test_and_set_bit(__I40E_CONFIG_BUSY, &pf->state))
+ while (test_and_set_bit(__I40E_CONFIG_BUSY, pf->state)) {
+ timeout--;
+ if (!timeout)
+ return -EBUSY;
usleep_range(1000, 2000);
+ }
if (!netif_running(vsi->netdev)) {
/* simple case - set for the next time the netdev is started */
@@ -1425,7 +1485,7 @@
}
done:
- clear_bit(__I40E_CONFIG_BUSY, &pf->state);
+ clear_bit(__I40E_CONFIG_BUSY, pf->state);
return err;
}
@@ -1483,13 +1543,6 @@
data[i++] = (i40e_gstrings_misc_stats[j].sizeof_stat ==
sizeof(u64)) ? *(u64 *)p : *(u32 *)p;
}
-#ifdef I40E_FCOE
- for (j = 0; j < I40E_FCOE_STATS_LEN; j++) {
- p = (char *)vsi + i40e_gstrings_fcoe_stats[j].stat_offset;
- data[i++] = (i40e_gstrings_fcoe_stats[j].sizeof_stat ==
- sizeof(u64)) ? *(u64 *)p : *(u32 *)p;
- }
-#endif
rcu_read_lock();
for (j = 0; j < vsi->num_queue_pairs; j++) {
tx_ring = ACCESS_ONCE(vsi->tx_rings[j]);
@@ -1577,13 +1630,6 @@
i40e_gstrings_misc_stats[i].stat_string);
p += ETH_GSTRING_LEN;
}
-#ifdef I40E_FCOE
- for (i = 0; i < I40E_FCOE_STATS_LEN; i++) {
- snprintf(p, ETH_GSTRING_LEN, "%s",
- i40e_gstrings_fcoe_stats[i].stat_string);
- p += ETH_GSTRING_LEN;
- }
-#endif
for (i = 0; i < vsi->num_queue_pairs; i++) {
snprintf(p, ETH_GSTRING_LEN, "tx-%d.tx_packets", i);
p += ETH_GSTRING_LEN;
@@ -1648,12 +1694,18 @@
/* BUG_ON(p - data != I40E_STATS_LEN * ETH_GSTRING_LEN); */
break;
case ETH_SS_PRIV_FLAGS:
- memcpy(data, i40e_priv_flags_strings,
- I40E_PRIV_FLAGS_STR_LEN * ETH_GSTRING_LEN);
- data += I40E_PRIV_FLAGS_STR_LEN * ETH_GSTRING_LEN;
- if (pf->hw.pf_id == 0)
- memcpy(data, i40e_gl_priv_flags_strings,
- I40E_GL_PRIV_FLAGS_STR_LEN * ETH_GSTRING_LEN);
+ for (i = 0; i < I40E_PRIV_FLAGS_STR_LEN; i++) {
+ snprintf(p, ETH_GSTRING_LEN, "%s",
+ i40e_gstrings_priv_flags[i].flag_string);
+ p += ETH_GSTRING_LEN;
+ }
+ if (pf->hw.pf_id != 0)
+ break;
+ for (i = 0; i < I40E_GL_PRIV_FLAGS_STR_LEN; i++) {
+ snprintf(p, ETH_GSTRING_LEN, "%s",
+ i40e_gl_gstrings_priv_flags[i].flag_string);
+ p += ETH_GSTRING_LEN;
+ }
break;
default:
break;
@@ -1774,7 +1826,7 @@
int i;
for (i = 0; i < pf->num_alloc_vfs; i++)
- if (test_bit(I40E_VF_STAT_ACTIVE, &vfs[i].vf_states))
+ if (test_bit(I40E_VF_STATE_ACTIVE, &vfs[i].vf_states))
return true;
return false;
}
@@ -1795,7 +1847,7 @@
/* Offline tests */
netif_info(pf, drv, netdev, "offline testing starting\n");
- set_bit(__I40E_TESTING, &pf->state);
+ set_bit(__I40E_TESTING, pf->state);
if (i40e_active_vfs(pf) || i40e_active_vmdqs(pf)) {
dev_warn(&pf->pdev->dev,
@@ -1805,7 +1857,7 @@
data[I40E_ETH_TEST_INTR] = 1;
data[I40E_ETH_TEST_LINK] = 1;
eth_test->flags |= ETH_TEST_FL_FAILED;
- clear_bit(__I40E_TESTING, &pf->state);
+ clear_bit(__I40E_TESTING, pf->state);
goto skip_ol_tests;
}
@@ -1819,7 +1871,7 @@
* link then the following link test would have
* to be moved to before the reset
*/
- i40e_do_reset(pf, BIT(__I40E_PF_RESET_REQUESTED));
+ i40e_do_reset(pf, BIT(__I40E_PF_RESET_REQUESTED), true);
if (i40e_link_test(netdev, &data[I40E_ETH_TEST_LINK]))
eth_test->flags |= ETH_TEST_FL_FAILED;
@@ -1834,8 +1886,8 @@
if (i40e_reg_test(netdev, &data[I40E_ETH_TEST_REG]))
eth_test->flags |= ETH_TEST_FL_FAILED;
- clear_bit(__I40E_TESTING, &pf->state);
- i40e_do_reset(pf, BIT(__I40E_PF_RESET_REQUESTED));
+ clear_bit(__I40E_TESTING, pf->state);
+ i40e_do_reset(pf, BIT(__I40E_PF_RESET_REQUESTED), true);
if (if_running)
i40e_open(netdev);
@@ -2285,6 +2337,102 @@
}
/**
+ * i40e_check_mask - Check whether a mask field is set
+ * @mask: the full mask value
+ * @field; mask of the field to check
+ *
+ * If the given mask is fully set, return positive value. If the mask for the
+ * field is fully unset, return zero. Otherwise return a negative error code.
+ **/
+static int i40e_check_mask(u64 mask, u64 field)
+{
+ u64 value = mask & field;
+
+ if (value == field)
+ return 1;
+ else if (!value)
+ return 0;
+ else
+ return -1;
+}
+
+/**
+ * i40e_parse_rx_flow_user_data - Deconstruct user-defined data
+ * @fsp: pointer to rx flow specification
+ * @data: pointer to userdef data structure for storage
+ *
+ * Read the user-defined data and deconstruct the value into a structure. No
+ * other code should read the user-defined data, so as to ensure that every
+ * place consistently reads the value correctly.
+ *
+ * The user-defined field is a 64bit Big Endian format value, which we
+ * deconstruct by reading bits or bit fields from it. Single bit flags shall
+ * be defined starting from the highest bits, while small bit field values
+ * shall be defined starting from the lowest bits.
+ *
+ * Returns 0 if the data is valid, and non-zero if the userdef data is invalid
+ * and the filter should be rejected. The data structure will always be
+ * modified even if FLOW_EXT is not set.
+ *
+ **/
+static int i40e_parse_rx_flow_user_data(struct ethtool_rx_flow_spec *fsp,
+ struct i40e_rx_flow_userdef *data)
+{
+ u64 value, mask;
+ int valid;
+
+ /* Zero memory first so it's always consistent. */
+ memset(data, 0, sizeof(*data));
+
+ if (!(fsp->flow_type & FLOW_EXT))
+ return 0;
+
+ value = be64_to_cpu(*((__be64 *)fsp->h_ext.data));
+ mask = be64_to_cpu(*((__be64 *)fsp->m_ext.data));
+
+#define I40E_USERDEF_FLEX_WORD GENMASK_ULL(15, 0)
+#define I40E_USERDEF_FLEX_OFFSET GENMASK_ULL(31, 16)
+#define I40E_USERDEF_FLEX_FILTER GENMASK_ULL(31, 0)
+
+ valid = i40e_check_mask(mask, I40E_USERDEF_FLEX_FILTER);
+ if (valid < 0) {
+ return -EINVAL;
+ } else if (valid) {
+ data->flex_word = value & I40E_USERDEF_FLEX_WORD;
+ data->flex_offset =
+ (value & I40E_USERDEF_FLEX_OFFSET) >> 16;
+ data->flex_filter = true;
+ }
+
+ return 0;
+}
+
+/**
+ * i40e_fill_rx_flow_user_data - Fill in user-defined data field
+ * @fsp: pointer to rx_flow specification
+ *
+ * Reads the userdef data structure and properly fills in the user defined
+ * fields of the rx_flow_spec.
+ **/
+static void i40e_fill_rx_flow_user_data(struct ethtool_rx_flow_spec *fsp,
+ struct i40e_rx_flow_userdef *data)
+{
+ u64 value = 0, mask = 0;
+
+ if (data->flex_filter) {
+ value |= data->flex_word;
+ value |= (u64)data->flex_offset << 16;
+ mask |= I40E_USERDEF_FLEX_FILTER;
+ }
+
+ if (value || mask)
+ fsp->flow_type |= FLOW_EXT;
+
+ *((__be64 *)fsp->h_ext.data) = cpu_to_be64(value);
+ *((__be64 *)fsp->m_ext.data) = cpu_to_be64(mask);
+}
+
+/**
* i40e_get_ethtool_fdir_all - Populates the rule count of a command
* @pf: Pointer to the physical function struct
* @cmd: The command to get or set Rx flow classification rules
@@ -2335,8 +2483,11 @@
{
struct ethtool_rx_flow_spec *fsp =
(struct ethtool_rx_flow_spec *)&cmd->fs;
+ struct i40e_rx_flow_userdef userdef = {0};
struct i40e_fdir_filter *rule = NULL;
struct hlist_node *node2;
+ u64 input_set;
+ u16 index;
hlist_for_each_entry_safe(rule, node2,
&pf->fdir_filter_list, fdir_node) {
@@ -2359,8 +2510,48 @@
*/
fsp->h_u.tcp_ip4_spec.psrc = rule->dst_port;
fsp->h_u.tcp_ip4_spec.pdst = rule->src_port;
- fsp->h_u.tcp_ip4_spec.ip4src = rule->dst_ip[0];
- fsp->h_u.tcp_ip4_spec.ip4dst = rule->src_ip[0];
+ fsp->h_u.tcp_ip4_spec.ip4src = rule->dst_ip;
+ fsp->h_u.tcp_ip4_spec.ip4dst = rule->src_ip;
+
+ switch (rule->flow_type) {
+ case SCTP_V4_FLOW:
+ index = I40E_FILTER_PCTYPE_NONF_IPV4_SCTP;
+ break;
+ case TCP_V4_FLOW:
+ index = I40E_FILTER_PCTYPE_NONF_IPV4_TCP;
+ break;
+ case UDP_V4_FLOW:
+ index = I40E_FILTER_PCTYPE_NONF_IPV4_UDP;
+ break;
+ case IP_USER_FLOW:
+ index = I40E_FILTER_PCTYPE_NONF_IPV4_OTHER;
+ break;
+ default:
+ /* If we have stored a filter with a flow type not listed here
+ * it is almost certainly a driver bug. WARN(), and then
+ * assign the input_set as if all fields are enabled to avoid
+ * reading unassigned memory.
+ */
+ WARN(1, "Missing input set index for flow_type %d\n",
+ rule->flow_type);
+ input_set = 0xFFFFFFFFFFFFFFFFULL;
+ goto no_input_set;
+ }
+
+ input_set = i40e_read_fd_input_set(pf, index);
+
+no_input_set:
+ if (input_set & I40E_L3_SRC_MASK)
+ fsp->m_u.tcp_ip4_spec.ip4src = htonl(0xFFFF);
+
+ if (input_set & I40E_L3_DST_MASK)
+ fsp->m_u.tcp_ip4_spec.ip4dst = htonl(0xFFFF);
+
+ if (input_set & I40E_L4_SRC_MASK)
+ fsp->m_u.tcp_ip4_spec.psrc = htons(0xFFFFFFFF);
+
+ if (input_set & I40E_L4_DST_MASK)
+ fsp->m_u.tcp_ip4_spec.pdst = htons(0xFFFFFFFF);
if (rule->dest_ctl == I40E_FILTER_PROGRAM_DESC_DEST_DROP_PACKET)
fsp->ring_cookie = RX_CLS_FLOW_DISC;
@@ -2372,11 +2563,24 @@
vsi = i40e_find_vsi_from_id(pf, rule->dest_vsi);
if (vsi && vsi->type == I40E_VSI_SRIOV) {
- fsp->h_ext.data[1] = htonl(vsi->vf_id);
- fsp->m_ext.data[1] = htonl(0x1);
+ /* VFs are zero-indexed by the driver, but ethtool
+ * expects them to be one-indexed, so add one here
+ */
+ u64 ring_vf = vsi->vf_id + 1;
+
+ ring_vf <<= ETHTOOL_RX_FLOW_SPEC_RING_VF_OFF;
+ fsp->ring_cookie |= ring_vf;
}
}
+ if (rule->flex_filter) {
+ userdef.flex_filter = true;
+ userdef.flex_word = be16_to_cpu(rule->flex_word);
+ userdef.flex_offset = rule->flex_offset;
+ }
+
+ i40e_fill_rx_flow_user_data(fsp, &userdef);
+
return 0;
}
@@ -2574,24 +2778,6 @@
}
/**
- * i40e_match_fdir_input_set - Match a new filter against an existing one
- * @rule: The filter already added
- * @input: The new filter to comapre against
- *
- * Returns true if the two input set match
- **/
-static bool i40e_match_fdir_input_set(struct i40e_fdir_filter *rule,
- struct i40e_fdir_filter *input)
-{
- if ((rule->dst_ip[0] != input->dst_ip[0]) ||
- (rule->src_ip[0] != input->src_ip[0]) ||
- (rule->dst_port != input->dst_port) ||
- (rule->src_port != input->src_port))
- return false;
- return true;
-}
-
-/**
* i40e_update_ethtool_fdir_entry - Updates the fdir filter entry
* @vsi: Pointer to the targeted VSI
* @input: The filter to update or NULL to indicate deletion
@@ -2626,22 +2812,22 @@
/* if there is an old rule occupying our place remove it */
if (rule && (rule->fd_id == sw_idx)) {
- if (input && !i40e_match_fdir_input_set(rule, input))
- err = i40e_add_del_fdir(vsi, rule, false);
- else if (!input)
- err = i40e_add_del_fdir(vsi, rule, false);
+ /* Remove this rule, since we're either deleting it, or
+ * replacing it.
+ */
+ err = i40e_add_del_fdir(vsi, rule, false);
hlist_del(&rule->fdir_node);
kfree(rule);
pf->fdir_pf_active_filters--;
}
- /* If no input this was a delete, err should be 0 if a rule was
- * successfully found and removed from the list else -EINVAL
+ /* If we weren't given an input, this is a delete, so just return the
+ * error code indicating if there was an entry at the requested slot
*/
if (!input)
return err;
- /* initialize node and set software index */
+ /* Otherwise, install the new rule as requested */
INIT_HLIST_NODE(&input->fdir_node);
/* add filter to the list */
@@ -2658,6 +2844,69 @@
}
/**
+ * i40e_prune_flex_pit_list - Cleanup unused entries in FLX_PIT table
+ * @pf: pointer to PF structure
+ *
+ * This function searches the list of filters and determines which FLX_PIT
+ * entries are still required. It will prune any entries which are no longer
+ * in use after the deletion.
+ **/
+static void i40e_prune_flex_pit_list(struct i40e_pf *pf)
+{
+ struct i40e_flex_pit *entry, *tmp;
+ struct i40e_fdir_filter *rule;
+
+ /* First, we'll check the l3 table */
+ list_for_each_entry_safe(entry, tmp, &pf->l3_flex_pit_list, list) {
+ bool found = false;
+
+ hlist_for_each_entry(rule, &pf->fdir_filter_list, fdir_node) {
+ if (rule->flow_type != IP_USER_FLOW)
+ continue;
+ if (rule->flex_filter &&
+ rule->flex_offset == entry->src_offset) {
+ found = true;
+ break;
+ }
+ }
+
+ /* If we didn't find the filter, then we can prune this entry
+ * from the list.
+ */
+ if (!found) {
+ list_del(&entry->list);
+ kfree(entry);
+ }
+ }
+
+ /* Followed by the L4 table */
+ list_for_each_entry_safe(entry, tmp, &pf->l4_flex_pit_list, list) {
+ bool found = false;
+
+ hlist_for_each_entry(rule, &pf->fdir_filter_list, fdir_node) {
+ /* Skip this filter if it's L3, since we already
+ * checked those in the above loop
+ */
+ if (rule->flow_type == IP_USER_FLOW)
+ continue;
+ if (rule->flex_filter &&
+ rule->flex_offset == entry->src_offset) {
+ found = true;
+ break;
+ }
+ }
+
+ /* If we didn't find the filter, then we can prune this entry
+ * from the list.
+ */
+ if (!found) {
+ list_del(&entry->list);
+ kfree(entry);
+ }
+ }
+}
+
+/**
* i40e_del_fdir_entry - Deletes a Flow Director filter entry
* @vsi: Pointer to the targeted VSI
* @cmd: The command to get or set Rx flow classification rules
@@ -2675,20 +2924,700 @@
struct i40e_pf *pf = vsi->back;
int ret = 0;
- if (test_bit(__I40E_RESET_RECOVERY_PENDING, &pf->state) ||
- test_bit(__I40E_RESET_INTR_RECEIVED, &pf->state))
+ if (test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state) ||
+ test_bit(__I40E_RESET_INTR_RECEIVED, pf->state))
return -EBUSY;
- if (test_bit(__I40E_FD_FLUSH_REQUESTED, &pf->state))
+ if (test_bit(__I40E_FD_FLUSH_REQUESTED, pf->state))
return -EBUSY;
ret = i40e_update_ethtool_fdir_entry(vsi, NULL, fsp->location, cmd);
+ i40e_prune_flex_pit_list(pf);
+
i40e_fdir_check_and_reenable(pf);
return ret;
}
/**
+ * i40e_unused_pit_index - Find an unused PIT index for given list
+ * @pf: the PF data structure
+ *
+ * Find the first unused flexible PIT index entry. We search both the L3 and
+ * L4 flexible PIT lists so that the returned index is unique and unused by
+ * either currently programmed L3 or L4 filters. We use a bit field as storage
+ * to track which indexes are already used.
+ **/
+static u8 i40e_unused_pit_index(struct i40e_pf *pf)
+{
+ unsigned long available_index = 0xFF;
+ struct i40e_flex_pit *entry;
+
+ /* We need to make sure that the new index isn't in use by either L3
+ * or L4 filters so that IP_USER_FLOW filters can program both L3 and
+ * L4 to use the same index.
+ */
+
+ list_for_each_entry(entry, &pf->l4_flex_pit_list, list)
+ clear_bit(entry->pit_index, &available_index);
+
+ list_for_each_entry(entry, &pf->l3_flex_pit_list, list)
+ clear_bit(entry->pit_index, &available_index);
+
+ return find_first_bit(&available_index, 8);
+}
+
+/**
+ * i40e_find_flex_offset - Find an existing flex src_offset
+ * @flex_pit_list: L3 or L4 flex PIT list
+ * @src_offset: new src_offset to find
+ *
+ * Searches the flex_pit_list for an existing offset. If no offset is
+ * currently programmed, then this will return an ERR_PTR if there is no space
+ * to add a new offset, otherwise it returns NULL.
+ **/
+static
+struct i40e_flex_pit *i40e_find_flex_offset(struct list_head *flex_pit_list,
+ u16 src_offset)
+{
+ struct i40e_flex_pit *entry;
+ int size = 0;
+
+ /* Search for the src_offset first. If we find a matching entry
+ * already programmed, we can simply re-use it.
+ */
+ list_for_each_entry(entry, flex_pit_list, list) {
+ size++;
+ if (entry->src_offset == src_offset)
+ return entry;
+ }
+
+ /* If we haven't found an entry yet, then the provided src offset has
+ * not yet been programmed. We will program the src offset later on,
+ * but we need to indicate whether there is enough space to do so
+ * here. We'll make use of ERR_PTR for this purpose.
+ */
+ if (size >= I40E_FLEX_PIT_TABLE_SIZE)
+ return ERR_PTR(-ENOSPC);
+
+ return NULL;
+}
+
+/**
+ * i40e_add_flex_offset - Add src_offset to flex PIT table list
+ * @flex_pit_list: L3 or L4 flex PIT list
+ * @src_offset: new src_offset to add
+ * @pit_index: the PIT index to program
+ *
+ * This function programs the new src_offset to the list. It is expected that
+ * i40e_find_flex_offset has already been tried and returned NULL, indicating
+ * that this offset is not programmed, and that the list has enough space to
+ * store another offset.
+ *
+ * Returns 0 on success, and negative value on error.
+ **/
+static int i40e_add_flex_offset(struct list_head *flex_pit_list,
+ u16 src_offset,
+ u8 pit_index)
+{
+ struct i40e_flex_pit *new_pit, *entry;
+
+ new_pit = kzalloc(sizeof(*entry), GFP_KERNEL);
+ if (!new_pit)
+ return -ENOMEM;
+
+ new_pit->src_offset = src_offset;
+ new_pit->pit_index = pit_index;
+
+ /* We need to insert this item such that the list is sorted by
+ * src_offset in ascending order.
+ */
+ list_for_each_entry(entry, flex_pit_list, list) {
+ if (new_pit->src_offset < entry->src_offset) {
+ list_add_tail(&new_pit->list, &entry->list);
+ return 0;
+ }
+
+ /* If we found an entry with our offset already programmed we
+ * can simply return here, after freeing the memory. However,
+ * if the pit_index does not match we need to report an error.
+ */
+ if (new_pit->src_offset == entry->src_offset) {
+ int err = 0;
+
+ /* If the PIT index is not the same we can't re-use
+ * the entry, so we must report an error.
+ */
+ if (new_pit->pit_index != entry->pit_index)
+ err = -EINVAL;
+
+ kfree(new_pit);
+ return err;
+ }
+ }
+
+ /* If we reached here, then we haven't yet added the item. This means
+ * that we should add the item at the end of the list.
+ */
+ list_add_tail(&new_pit->list, flex_pit_list);
+ return 0;
+}
+
+/**
+ * __i40e_reprogram_flex_pit - Re-program specific FLX_PIT table
+ * @pf: Pointer to the PF structure
+ * @flex_pit_list: list of flexible src offsets in use
+ * #flex_pit_start: index to first entry for this section of the table
+ *
+ * In order to handle flexible data, the hardware uses a table of values
+ * called the FLX_PIT table. This table is used to indicate which sections of
+ * the input correspond to what PIT index values. Unfortunately, hardware is
+ * very restrictive about programming this table. Entries must be ordered by
+ * src_offset in ascending order, without duplicates. Additionally, unused
+ * entries must be set to the unused index value, and must have valid size and
+ * length according to the src_offset ordering.
+ *
+ * This function will reprogram the FLX_PIT register from a book-keeping
+ * structure that we guarantee is already ordered correctly, and has no more
+ * than 3 entries.
+ *
+ * To make things easier, we only support flexible values of one word length,
+ * rather than allowing variable length flexible values.
+ **/
+static void __i40e_reprogram_flex_pit(struct i40e_pf *pf,
+ struct list_head *flex_pit_list,
+ int flex_pit_start)
+{
+ struct i40e_flex_pit *entry = NULL;
+ u16 last_offset = 0;
+ int i = 0, j = 0;
+
+ /* First, loop over the list of flex PIT entries, and reprogram the
+ * registers.
+ */
+ list_for_each_entry(entry, flex_pit_list, list) {
+ /* We have to be careful when programming values for the
+ * largest SRC_OFFSET value. It is possible that adding
+ * additional empty values at the end would overflow the space
+ * for the SRC_OFFSET in the FLX_PIT register. To avoid this,
+ * we check here and add the empty values prior to adding the
+ * largest value.
+ *
+ * To determine this, we will use a loop from i+1 to 3, which
+ * will determine whether the unused entries would have valid
+ * SRC_OFFSET. Note that there cannot be extra entries past
+ * this value, because the only valid values would have been
+ * larger than I40E_MAX_FLEX_SRC_OFFSET, and thus would not
+ * have been added to the list in the first place.
+ */
+ for (j = i + 1; j < 3; j++) {
+ u16 offset = entry->src_offset + j;
+ int index = flex_pit_start + i;
+ u32 value = I40E_FLEX_PREP_VAL(I40E_FLEX_DEST_UNUSED,
+ 1,
+ offset - 3);
+
+ if (offset > I40E_MAX_FLEX_SRC_OFFSET) {
+ i40e_write_rx_ctl(&pf->hw,
+ I40E_PRTQF_FLX_PIT(index),
+ value);
+ i++;
+ }
+ }
+
+ /* Now, we can program the actual value into the table */
+ i40e_write_rx_ctl(&pf->hw,
+ I40E_PRTQF_FLX_PIT(flex_pit_start + i),
+ I40E_FLEX_PREP_VAL(entry->pit_index + 50,
+ 1,
+ entry->src_offset));
+ i++;
+ }
+
+ /* In order to program the last entries in the table, we need to
+ * determine the valid offset. If the list is empty, we'll just start
+ * with 0. Otherwise, we'll start with the last item offset and add 1.
+ * This ensures that all entries have valid sizes. If we don't do this
+ * correctly, the hardware will disable flexible field parsing.
+ */
+ if (!list_empty(flex_pit_list))
+ last_offset = list_prev_entry(entry, list)->src_offset + 1;
+
+ for (; i < 3; i++, last_offset++) {
+ i40e_write_rx_ctl(&pf->hw,
+ I40E_PRTQF_FLX_PIT(flex_pit_start + i),
+ I40E_FLEX_PREP_VAL(I40E_FLEX_DEST_UNUSED,
+ 1,
+ last_offset));
+ }
+}
+
+/**
+ * i40e_reprogram_flex_pit - Reprogram all FLX_PIT tables after input set change
+ * @pf: pointer to the PF structure
+ *
+ * This function reprograms both the L3 and L4 FLX_PIT tables. See the
+ * internal helper function for implementation details.
+ **/
+static void i40e_reprogram_flex_pit(struct i40e_pf *pf)
+{
+ __i40e_reprogram_flex_pit(pf, &pf->l3_flex_pit_list,
+ I40E_FLEX_PIT_IDX_START_L3);
+
+ __i40e_reprogram_flex_pit(pf, &pf->l4_flex_pit_list,
+ I40E_FLEX_PIT_IDX_START_L4);
+
+ /* We also need to program the L3 and L4 GLQF ORT register */
+ i40e_write_rx_ctl(&pf->hw,
+ I40E_GLQF_ORT(I40E_L3_GLQF_ORT_IDX),
+ I40E_ORT_PREP_VAL(I40E_FLEX_PIT_IDX_START_L3,
+ 3, 1));
+
+ i40e_write_rx_ctl(&pf->hw,
+ I40E_GLQF_ORT(I40E_L4_GLQF_ORT_IDX),
+ I40E_ORT_PREP_VAL(I40E_FLEX_PIT_IDX_START_L4,
+ 3, 1));
+}
+
+/**
+ * i40e_flow_str - Converts a flow_type into a human readable string
+ * @flow_type: the flow type from a flow specification
+ *
+ * Currently only flow types we support are included here, and the string
+ * value attempts to match what ethtool would use to configure this flow type.
+ **/
+static const char *i40e_flow_str(struct ethtool_rx_flow_spec *fsp)
+{
+ switch (fsp->flow_type & ~FLOW_EXT) {
+ case TCP_V4_FLOW:
+ return "tcp4";
+ case UDP_V4_FLOW:
+ return "udp4";
+ case SCTP_V4_FLOW:
+ return "sctp4";
+ case IP_USER_FLOW:
+ return "ip4";
+ default:
+ return "unknown";
+ }
+}
+
+/**
+ * i40e_pit_index_to_mask - Return the FLEX mask for a given PIT index
+ * @pit_index: PIT index to convert
+ *
+ * Returns the mask for a given PIT index. Will return 0 if the pit_index is
+ * of range.
+ **/
+static u64 i40e_pit_index_to_mask(int pit_index)
+{
+ switch (pit_index) {
+ case 0:
+ return I40E_FLEX_50_MASK;
+ case 1:
+ return I40E_FLEX_51_MASK;
+ case 2:
+ return I40E_FLEX_52_MASK;
+ case 3:
+ return I40E_FLEX_53_MASK;
+ case 4:
+ return I40E_FLEX_54_MASK;
+ case 5:
+ return I40E_FLEX_55_MASK;
+ case 6:
+ return I40E_FLEX_56_MASK;
+ case 7:
+ return I40E_FLEX_57_MASK;
+ default:
+ return 0;
+ }
+}
+
+/**
+ * i40e_print_input_set - Show changes between two input sets
+ * @vsi: the vsi being configured
+ * @old: the old input set
+ * @new: the new input set
+ *
+ * Print the difference between old and new input sets by showing which series
+ * of words are toggled on or off. Only displays the bits we actually support
+ * changing.
+ **/
+static void i40e_print_input_set(struct i40e_vsi *vsi, u64 old, u64 new)
+{
+ struct i40e_pf *pf = vsi->back;
+ bool old_value, new_value;
+ int i;
+
+ old_value = !!(old & I40E_L3_SRC_MASK);
+ new_value = !!(new & I40E_L3_SRC_MASK);
+ if (old_value != new_value)
+ netif_info(pf, drv, vsi->netdev, "L3 source address: %s -> %s\n",
+ old_value ? "ON" : "OFF",
+ new_value ? "ON" : "OFF");
+
+ old_value = !!(old & I40E_L3_DST_MASK);
+ new_value = !!(new & I40E_L3_DST_MASK);
+ if (old_value != new_value)
+ netif_info(pf, drv, vsi->netdev, "L3 destination address: %s -> %s\n",
+ old_value ? "ON" : "OFF",
+ new_value ? "ON" : "OFF");
+
+ old_value = !!(old & I40E_L4_SRC_MASK);
+ new_value = !!(new & I40E_L4_SRC_MASK);
+ if (old_value != new_value)
+ netif_info(pf, drv, vsi->netdev, "L4 source port: %s -> %s\n",
+ old_value ? "ON" : "OFF",
+ new_value ? "ON" : "OFF");
+
+ old_value = !!(old & I40E_L4_DST_MASK);
+ new_value = !!(new & I40E_L4_DST_MASK);
+ if (old_value != new_value)
+ netif_info(pf, drv, vsi->netdev, "L4 destination port: %s -> %s\n",
+ old_value ? "ON" : "OFF",
+ new_value ? "ON" : "OFF");
+
+ old_value = !!(old & I40E_VERIFY_TAG_MASK);
+ new_value = !!(new & I40E_VERIFY_TAG_MASK);
+ if (old_value != new_value)
+ netif_info(pf, drv, vsi->netdev, "SCTP verification tag: %s -> %s\n",
+ old_value ? "ON" : "OFF",
+ new_value ? "ON" : "OFF");
+
+ /* Show change of flexible filter entries */
+ for (i = 0; i < I40E_FLEX_INDEX_ENTRIES; i++) {
+ u64 flex_mask = i40e_pit_index_to_mask(i);
+
+ old_value = !!(old & flex_mask);
+ new_value = !!(new & flex_mask);
+ if (old_value != new_value)
+ netif_info(pf, drv, vsi->netdev, "FLEX index %d: %s -> %s\n",
+ i,
+ old_value ? "ON" : "OFF",
+ new_value ? "ON" : "OFF");
+ }
+
+ netif_info(pf, drv, vsi->netdev, " Current input set: %0llx\n",
+ old);
+ netif_info(pf, drv, vsi->netdev, "Requested input set: %0llx\n",
+ new);
+}
+
+/**
+ * i40e_check_fdir_input_set - Check that a given rx_flow_spec mask is valid
+ * @vsi: pointer to the targeted VSI
+ * @fsp: pointer to Rx flow specification
+ * @userdef: userdefined data from flow specification
+ *
+ * Ensures that a given ethtool_rx_flow_spec has a valid mask. Some support
+ * for partial matches exists with a few limitations. First, hardware only
+ * supports masking by word boundary (2 bytes) and not per individual bit.
+ * Second, hardware is limited to using one mask for a flow type and cannot
+ * use a separate mask for each filter.
+ *
+ * To support these limitations, if we already have a configured filter for
+ * the specified type, this function enforces that new filters of the type
+ * match the configured input set. Otherwise, if we do not have a filter of
+ * the specified type, we allow the input set to be updated to match the
+ * desired filter.
+ *
+ * To help ensure that administrators understand why filters weren't displayed
+ * as supported, we print a diagnostic message displaying how the input set
+ * would change and warning to delete the preexisting filters if required.
+ *
+ * Returns 0 on successful input set match, and a negative return code on
+ * failure.
+ **/
+static int i40e_check_fdir_input_set(struct i40e_vsi *vsi,
+ struct ethtool_rx_flow_spec *fsp,
+ struct i40e_rx_flow_userdef *userdef)
+{
+ struct i40e_pf *pf = vsi->back;
+ struct ethtool_tcpip4_spec *tcp_ip4_spec;
+ struct ethtool_usrip4_spec *usr_ip4_spec;
+ u64 current_mask, new_mask;
+ bool new_flex_offset = false;
+ bool flex_l3 = false;
+ u16 *fdir_filter_count;
+ u16 index, src_offset = 0;
+ u8 pit_index = 0;
+ int err;
+
+ switch (fsp->flow_type & ~FLOW_EXT) {
+ case SCTP_V4_FLOW:
+ index = I40E_FILTER_PCTYPE_NONF_IPV4_SCTP;
+ fdir_filter_count = &pf->fd_sctp4_filter_cnt;
+ break;
+ case TCP_V4_FLOW:
+ index = I40E_FILTER_PCTYPE_NONF_IPV4_TCP;
+ fdir_filter_count = &pf->fd_tcp4_filter_cnt;
+ break;
+ case UDP_V4_FLOW:
+ index = I40E_FILTER_PCTYPE_NONF_IPV4_UDP;
+ fdir_filter_count = &pf->fd_udp4_filter_cnt;
+ break;
+ case IP_USER_FLOW:
+ index = I40E_FILTER_PCTYPE_NONF_IPV4_OTHER;
+ fdir_filter_count = &pf->fd_ip4_filter_cnt;
+ flex_l3 = true;
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ /* Read the current input set from register memory. */
+ current_mask = i40e_read_fd_input_set(pf, index);
+ new_mask = current_mask;
+
+ /* Determine, if any, the required changes to the input set in order
+ * to support the provided mask.
+ *
+ * Hardware only supports masking at word (2 byte) granularity and does
+ * not support full bitwise masking. This implementation simplifies
+ * even further and only supports fully enabled or fully disabled
+ * masks for each field, even though we could split the ip4src and
+ * ip4dst fields.
+ */
+ switch (fsp->flow_type & ~FLOW_EXT) {
+ case SCTP_V4_FLOW:
+ new_mask &= ~I40E_VERIFY_TAG_MASK;
+ /* Fall through */
+ case TCP_V4_FLOW:
+ case UDP_V4_FLOW:
+ tcp_ip4_spec = &fsp->m_u.tcp_ip4_spec;
+
+ /* IPv4 source address */
+ if (tcp_ip4_spec->ip4src == htonl(0xFFFFFFFF))
+ new_mask |= I40E_L3_SRC_MASK;
+ else if (!tcp_ip4_spec->ip4src)
+ new_mask &= ~I40E_L3_SRC_MASK;
+ else
+ return -EOPNOTSUPP;
+
+ /* IPv4 destination address */
+ if (tcp_ip4_spec->ip4dst == htonl(0xFFFFFFFF))
+ new_mask |= I40E_L3_DST_MASK;
+ else if (!tcp_ip4_spec->ip4dst)
+ new_mask &= ~I40E_L3_DST_MASK;
+ else
+ return -EOPNOTSUPP;
+
+ /* L4 source port */
+ if (tcp_ip4_spec->psrc == htons(0xFFFF))
+ new_mask |= I40E_L4_SRC_MASK;
+ else if (!tcp_ip4_spec->psrc)
+ new_mask &= ~I40E_L4_SRC_MASK;
+ else
+ return -EOPNOTSUPP;
+
+ /* L4 destination port */
+ if (tcp_ip4_spec->pdst == htons(0xFFFF))
+ new_mask |= I40E_L4_DST_MASK;
+ else if (!tcp_ip4_spec->pdst)
+ new_mask &= ~I40E_L4_DST_MASK;
+ else
+ return -EOPNOTSUPP;
+
+ /* Filtering on Type of Service is not supported. */
+ if (tcp_ip4_spec->tos)
+ return -EOPNOTSUPP;
+
+ break;
+ case IP_USER_FLOW:
+ usr_ip4_spec = &fsp->m_u.usr_ip4_spec;
+
+ /* IPv4 source address */
+ if (usr_ip4_spec->ip4src == htonl(0xFFFFFFFF))
+ new_mask |= I40E_L3_SRC_MASK;
+ else if (!usr_ip4_spec->ip4src)
+ new_mask &= ~I40E_L3_SRC_MASK;
+ else
+ return -EOPNOTSUPP;
+
+ /* IPv4 destination address */
+ if (usr_ip4_spec->ip4dst == htonl(0xFFFFFFFF))
+ new_mask |= I40E_L3_DST_MASK;
+ else if (!usr_ip4_spec->ip4dst)
+ new_mask &= ~I40E_L3_DST_MASK;
+ else
+ return -EOPNOTSUPP;
+
+ /* First 4 bytes of L4 header */
+ if (usr_ip4_spec->l4_4_bytes == htonl(0xFFFFFFFF))
+ new_mask |= I40E_L4_SRC_MASK | I40E_L4_DST_MASK;
+ else if (!usr_ip4_spec->l4_4_bytes)
+ new_mask &= ~(I40E_L4_SRC_MASK | I40E_L4_DST_MASK);
+ else
+ return -EOPNOTSUPP;
+
+ /* Filtering on Type of Service is not supported. */
+ if (usr_ip4_spec->tos)
+ return -EOPNOTSUPP;
+
+ /* Filtering on IP version is not supported */
+ if (usr_ip4_spec->ip_ver)
+ return -EINVAL;
+
+ /* Filtering on L4 protocol is not supported */
+ if (usr_ip4_spec->proto)
+ return -EINVAL;
+
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ /* First, clear all flexible filter entries */
+ new_mask &= ~I40E_FLEX_INPUT_MASK;
+
+ /* If we have a flexible filter, try to add this offset to the correct
+ * flexible filter PIT list. Once finished, we can update the mask.
+ * If the src_offset changed, we will get a new mask value which will
+ * trigger an input set change.
+ */
+ if (userdef->flex_filter) {
+ struct i40e_flex_pit *l3_flex_pit = NULL, *flex_pit = NULL;
+
+ /* Flexible offset must be even, since the flexible payload
+ * must be aligned on 2-byte boundary.
+ */
+ if (userdef->flex_offset & 0x1) {
+ dev_warn(&pf->pdev->dev,
+ "Flexible data offset must be 2-byte aligned\n");
+ return -EINVAL;
+ }
+
+ src_offset = userdef->flex_offset >> 1;
+
+ /* FLX_PIT source offset value is only so large */
+ if (src_offset > I40E_MAX_FLEX_SRC_OFFSET) {
+ dev_warn(&pf->pdev->dev,
+ "Flexible data must reside within first 64 bytes of the packet payload\n");
+ return -EINVAL;
+ }
+
+ /* See if this offset has already been programmed. If we get
+ * an ERR_PTR, then the filter is not safe to add. Otherwise,
+ * if we get a NULL pointer, this means we will need to add
+ * the offset.
+ */
+ flex_pit = i40e_find_flex_offset(&pf->l4_flex_pit_list,
+ src_offset);
+ if (IS_ERR(flex_pit))
+ return PTR_ERR(flex_pit);
+
+ /* IP_USER_FLOW filters match both L4 (ICMP) and L3 (unknown)
+ * packet types, and thus we need to program both L3 and L4
+ * flexible values. These must have identical flexible index,
+ * as otherwise we can't correctly program the input set. So
+ * we'll find both an L3 and L4 index and make sure they are
+ * the same.
+ */
+ if (flex_l3) {
+ l3_flex_pit =
+ i40e_find_flex_offset(&pf->l3_flex_pit_list,
+ src_offset);
+ if (IS_ERR(l3_flex_pit))
+ return PTR_ERR(l3_flex_pit);
+
+ if (flex_pit) {
+ /* If we already had a matching L4 entry, we
+ * need to make sure that the L3 entry we
+ * obtained uses the same index.
+ */
+ if (l3_flex_pit) {
+ if (l3_flex_pit->pit_index !=
+ flex_pit->pit_index) {
+ return -EINVAL;
+ }
+ } else {
+ new_flex_offset = true;
+ }
+ } else {
+ flex_pit = l3_flex_pit;
+ }
+ }
+
+ /* If we didn't find an existing flex offset, we need to
+ * program a new one. However, we don't immediately program it
+ * here because we will wait to program until after we check
+ * that it is safe to change the input set.
+ */
+ if (!flex_pit) {
+ new_flex_offset = true;
+ pit_index = i40e_unused_pit_index(pf);
+ } else {
+ pit_index = flex_pit->pit_index;
+ }
+
+ /* Update the mask with the new offset */
+ new_mask |= i40e_pit_index_to_mask(pit_index);
+ }
+
+ /* If the mask and flexible filter offsets for this filter match the
+ * currently programmed values we don't need any input set change, so
+ * this filter is safe to install.
+ */
+ if (new_mask == current_mask && !new_flex_offset)
+ return 0;
+
+ netif_info(pf, drv, vsi->netdev, "Input set change requested for %s flows:\n",
+ i40e_flow_str(fsp));
+ i40e_print_input_set(vsi, current_mask, new_mask);
+ if (new_flex_offset) {
+ netif_info(pf, drv, vsi->netdev, "FLEX index %d: Offset -> %d",
+ pit_index, src_offset);
+ }
+
+ /* Hardware input sets are global across multiple ports, so even the
+ * main port cannot change them when in MFP mode as this would impact
+ * any filters on the other ports.
+ */
+ if (pf->flags & I40E_FLAG_MFP_ENABLED) {
+ netif_err(pf, drv, vsi->netdev, "Cannot change Flow Director input sets while MFP is enabled\n");
+ return -EOPNOTSUPP;
+ }
+
+ /* This filter requires us to update the input set. However, hardware
+ * only supports one input set per flow type, and does not support
+ * separate masks for each filter. This means that we can only support
+ * a single mask for all filters of a specific type.
+ *
+ * If we have preexisting filters, they obviously depend on the
+ * current programmed input set. Display a diagnostic message in this
+ * case explaining why the filter could not be accepted.
+ */
+ if (*fdir_filter_count) {
+ netif_err(pf, drv, vsi->netdev, "Cannot change input set for %s flows until %d preexisting filters are removed\n",
+ i40e_flow_str(fsp),
+ *fdir_filter_count);
+ return -EOPNOTSUPP;
+ }
+
+ i40e_write_fd_input_set(pf, index, new_mask);
+
+ /* Add the new offset and update table, if necessary */
+ if (new_flex_offset) {
+ err = i40e_add_flex_offset(&pf->l4_flex_pit_list, src_offset,
+ pit_index);
+ if (err)
+ return err;
+
+ if (flex_l3) {
+ err = i40e_add_flex_offset(&pf->l3_flex_pit_list,
+ src_offset,
+ pit_index);
+ if (err)
+ return err;
+ }
+
+ i40e_reprogram_flex_pit(pf);
+ }
+
+ return 0;
+}
+
+/**
* i40e_add_fdir_ethtool - Add/Remove Flow Director filters
* @vsi: pointer to the targeted VSI
* @cmd: command to get or set RX flow classification rules
@@ -2699,11 +3628,13 @@
static int i40e_add_fdir_ethtool(struct i40e_vsi *vsi,
struct ethtool_rxnfc *cmd)
{
+ struct i40e_rx_flow_userdef userdef;
struct ethtool_rx_flow_spec *fsp;
struct i40e_fdir_filter *input;
+ u16 dest_vsi = 0, q_index = 0;
struct i40e_pf *pf;
int ret = -EINVAL;
- u16 vf_id;
+ u8 dest_ctl;
if (!vsi)
return -EINVAL;
@@ -2712,26 +3643,61 @@
if (!(pf->flags & I40E_FLAG_FD_SB_ENABLED))
return -EOPNOTSUPP;
- if (pf->auto_disable_flags & I40E_FLAG_FD_SB_ENABLED)
+ if (pf->flags & I40E_FLAG_FD_SB_AUTO_DISABLED)
return -ENOSPC;
- if (test_bit(__I40E_RESET_RECOVERY_PENDING, &pf->state) ||
- test_bit(__I40E_RESET_INTR_RECEIVED, &pf->state))
+ if (test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state) ||
+ test_bit(__I40E_RESET_INTR_RECEIVED, pf->state))
return -EBUSY;
- if (test_bit(__I40E_FD_FLUSH_REQUESTED, &pf->state))
+ if (test_bit(__I40E_FD_FLUSH_REQUESTED, pf->state))
return -EBUSY;
fsp = (struct ethtool_rx_flow_spec *)&cmd->fs;
+ /* Parse the user-defined field */
+ if (i40e_parse_rx_flow_user_data(fsp, &userdef))
+ return -EINVAL;
+
+ /* Extended MAC field is not supported */
+ if (fsp->flow_type & FLOW_MAC_EXT)
+ return -EINVAL;
+
+ ret = i40e_check_fdir_input_set(vsi, fsp, &userdef);
+ if (ret)
+ return ret;
+
if (fsp->location >= (pf->hw.func_caps.fd_filters_best_effort +
pf->hw.func_caps.fd_filters_guaranteed)) {
return -EINVAL;
}
- if ((fsp->ring_cookie != RX_CLS_FLOW_DISC) &&
- (fsp->ring_cookie >= vsi->num_queue_pairs))
- return -EINVAL;
+ /* ring_cookie is either the drop index, or is a mask of the queue
+ * index and VF id we wish to target.
+ */
+ if (fsp->ring_cookie == RX_CLS_FLOW_DISC) {
+ dest_ctl = I40E_FILTER_PROGRAM_DESC_DEST_DROP_PACKET;
+ } else {
+ u32 ring = ethtool_get_flow_spec_ring(fsp->ring_cookie);
+ u8 vf = ethtool_get_flow_spec_ring_vf(fsp->ring_cookie);
+
+ if (!vf) {
+ if (ring >= vsi->num_queue_pairs)
+ return -EINVAL;
+ dest_vsi = vsi->id;
+ } else {
+ /* VFs are zero-indexed, so we subtract one here */
+ vf--;
+
+ if (vf >= pf->num_alloc_vfs)
+ return -EINVAL;
+ if (ring >= pf->vf[vf].num_queue_pairs)
+ return -EINVAL;
+ dest_vsi = pf->vf[vf].lan_vsi_id;
+ }
+ dest_ctl = I40E_FILTER_PROGRAM_DESC_DEST_DIRECT_PACKET_QINDEX;
+ q_index = ring;
+ }
input = kzalloc(sizeof(*input), GFP_KERNEL);
@@ -2739,20 +3705,14 @@
return -ENOMEM;
input->fd_id = fsp->location;
-
- if (fsp->ring_cookie == RX_CLS_FLOW_DISC)
- input->dest_ctl = I40E_FILTER_PROGRAM_DESC_DEST_DROP_PACKET;
- else
- input->dest_ctl =
- I40E_FILTER_PROGRAM_DESC_DEST_DIRECT_PACKET_QINDEX;
-
- input->q_index = fsp->ring_cookie;
- input->flex_off = 0;
- input->pctype = 0;
- input->dest_vsi = vsi->id;
+ input->q_index = q_index;
+ input->dest_vsi = dest_vsi;
+ input->dest_ctl = dest_ctl;
input->fd_status = I40E_FILTER_PROGRAM_DESC_FD_STATUS_FD_ID;
input->cnt_index = I40E_FD_SB_STAT_IDX(pf->hw.pf_id);
- input->flow_type = fsp->flow_type;
+ input->dst_ip = fsp->h_u.tcp_ip4_spec.ip4src;
+ input->src_ip = fsp->h_u.tcp_ip4_spec.ip4dst;
+ input->flow_type = fsp->flow_type & ~FLOW_EXT;
input->ip4_proto = fsp->h_u.usr_ip4_spec.proto;
/* Reverse the src and dest notion, since the HW expects them to be from
@@ -2760,33 +3720,29 @@
*/
input->dst_port = fsp->h_u.tcp_ip4_spec.psrc;
input->src_port = fsp->h_u.tcp_ip4_spec.pdst;
- input->dst_ip[0] = fsp->h_u.tcp_ip4_spec.ip4src;
- input->src_ip[0] = fsp->h_u.tcp_ip4_spec.ip4dst;
+ input->dst_ip = fsp->h_u.tcp_ip4_spec.ip4src;
+ input->src_ip = fsp->h_u.tcp_ip4_spec.ip4dst;
- if (ntohl(fsp->m_ext.data[1])) {
- vf_id = ntohl(fsp->h_ext.data[1]);
- if (vf_id >= pf->num_alloc_vfs) {
- netif_info(pf, drv, vsi->netdev,
- "Invalid VF id %d\n", vf_id);
- goto free_input;
- }
- /* Find vsi id from vf id and override dest vsi */
- input->dest_vsi = pf->vf[vf_id].lan_vsi_id;
- if (input->q_index >= pf->vf[vf_id].num_queue_pairs) {
- netif_info(pf, drv, vsi->netdev,
- "Invalid queue id %d for VF %d\n",
- input->q_index, vf_id);
- goto free_input;
- }
+ if (userdef.flex_filter) {
+ input->flex_filter = true;
+ input->flex_word = cpu_to_be16(userdef.flex_word);
+ input->flex_offset = userdef.flex_offset;
}
ret = i40e_add_del_fdir(vsi, input, true);
-free_input:
if (ret)
- kfree(input);
- else
- i40e_update_ethtool_fdir_entry(vsi, input, fsp->location, NULL);
+ goto free_input;
+ /* Add the input filter to the fdir_input_list, possibly replacing
+ * a previous filter. Do not free the input structure after adding it
+ * to the list as this would cause a use-after-free bug.
+ */
+ i40e_update_ethtool_fdir_entry(vsi, input, fsp->location, NULL);
+
+ return 0;
+
+free_input:
+ kfree(input);
return ret;
}
@@ -3036,7 +3992,7 @@
* @dev: network interface device structure
*
* The get string set count and the string set should be matched for each
- * flag returned. Add new strings for each flag to the i40e_priv_flags_strings
+ * flag returned. Add new strings for each flag to the i40e_gstrings_priv_flags
* array.
*
* Returns a u32 bitmap of flags.
@@ -3046,19 +4002,27 @@
struct i40e_netdev_priv *np = netdev_priv(dev);
struct i40e_vsi *vsi = np->vsi;
struct i40e_pf *pf = vsi->back;
- u32 ret_flags = 0;
+ u32 i, j, ret_flags = 0;
- ret_flags |= pf->flags & I40E_FLAG_LINK_POLLING_ENABLED ?
- I40E_PRIV_FLAGS_LINKPOLL_FLAG : 0;
- ret_flags |= pf->flags & I40E_FLAG_FD_ATR_ENABLED ?
- I40E_PRIV_FLAGS_FD_ATR : 0;
- ret_flags |= pf->flags & I40E_FLAG_VEB_STATS_ENABLED ?
- I40E_PRIV_FLAGS_VEB_STATS : 0;
- ret_flags |= pf->auto_disable_flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE ?
- 0 : I40E_PRIV_FLAGS_HW_ATR_EVICT;
- if (pf->hw.pf_id == 0) {
- ret_flags |= pf->flags & I40E_FLAG_TRUE_PROMISC_SUPPORT ?
- I40E_PRIV_FLAGS_TRUE_PROMISC_SUPPORT : 0;
+ for (i = 0; i < I40E_PRIV_FLAGS_STR_LEN; i++) {
+ const struct i40e_priv_flags *priv_flags;
+
+ priv_flags = &i40e_gstrings_priv_flags[i];
+
+ if (priv_flags->flag & pf->flags)
+ ret_flags |= BIT(i);
+ }
+
+ if (pf->hw.pf_id != 0)
+ return ret_flags;
+
+ for (j = 0; j < I40E_GL_PRIV_FLAGS_STR_LEN; j++) {
+ const struct i40e_priv_flags *priv_flags;
+
+ priv_flags = &i40e_gl_gstrings_priv_flags[j];
+
+ if (priv_flags->flag & pf->flags)
+ ret_flags |= BIT(i + j);
}
return ret_flags;
@@ -3074,54 +4038,66 @@
struct i40e_netdev_priv *np = netdev_priv(dev);
struct i40e_vsi *vsi = np->vsi;
struct i40e_pf *pf = vsi->back;
- u16 sw_flags = 0, valid_flags = 0;
- bool reset_required = false;
- bool promisc_change = false;
- int ret;
+ u64 changed_flags;
+ u32 i, j;
- /* NOTE: MFP is not settable */
+ changed_flags = pf->flags;
- if (flags & I40E_PRIV_FLAGS_LINKPOLL_FLAG)
- pf->flags |= I40E_FLAG_LINK_POLLING_ENABLED;
- else
- pf->flags &= ~I40E_FLAG_LINK_POLLING_ENABLED;
+ for (i = 0; i < I40E_PRIV_FLAGS_STR_LEN; i++) {
+ const struct i40e_priv_flags *priv_flags;
- /* allow the user to control the state of the Flow
- * Director ATR (Application Targeted Routing) feature
- * of the driver
+ priv_flags = &i40e_gstrings_priv_flags[i];
+
+ if (priv_flags->read_only)
+ continue;
+
+ if (flags & BIT(i))
+ pf->flags |= priv_flags->flag;
+ else
+ pf->flags &= ~(priv_flags->flag);
+ }
+
+ if (pf->hw.pf_id != 0)
+ goto flags_complete;
+
+ for (j = 0; j < I40E_GL_PRIV_FLAGS_STR_LEN; j++) {
+ const struct i40e_priv_flags *priv_flags;
+
+ priv_flags = &i40e_gl_gstrings_priv_flags[j];
+
+ if (priv_flags->read_only)
+ continue;
+
+ if (flags & BIT(i + j))
+ pf->flags |= priv_flags->flag;
+ else
+ pf->flags &= ~(priv_flags->flag);
+ }
+
+flags_complete:
+ /* check for flags that changed */
+ changed_flags ^= pf->flags;
+
+ /* Process any additional changes needed as a result of flag changes.
+ * The changed_flags value reflects the list of bits that were
+ * changed in the code above.
*/
- if (flags & I40E_PRIV_FLAGS_FD_ATR) {
- pf->flags |= I40E_FLAG_FD_ATR_ENABLED;
- } else {
- pf->flags &= ~I40E_FLAG_FD_ATR_ENABLED;
- pf->auto_disable_flags |= I40E_FLAG_FD_ATR_ENABLED;
- /* flush current ATR settings */
- set_bit(__I40E_FD_FLUSH_REQUESTED, &pf->state);
+ /* Flush current ATR settings if ATR was disabled */
+ if ((changed_flags & I40E_FLAG_FD_ATR_ENABLED) &&
+ !(pf->flags & I40E_FLAG_FD_ATR_ENABLED)) {
+ pf->flags |= I40E_FLAG_FD_ATR_AUTO_DISABLED;
+ set_bit(__I40E_FD_FLUSH_REQUESTED, pf->state);
}
- if ((flags & I40E_PRIV_FLAGS_VEB_STATS) &&
- !(pf->flags & I40E_FLAG_VEB_STATS_ENABLED)) {
- pf->flags |= I40E_FLAG_VEB_STATS_ENABLED;
- reset_required = true;
- } else if (!(flags & I40E_PRIV_FLAGS_VEB_STATS) &&
- (pf->flags & I40E_FLAG_VEB_STATS_ENABLED)) {
- pf->flags &= ~I40E_FLAG_VEB_STATS_ENABLED;
- reset_required = true;
- }
+ /* Only allow ATR evict on hardware that is capable of handling it */
+ if (pf->flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE)
+ pf->flags &= ~I40E_FLAG_HW_ATR_EVICT_CAPABLE;
- if (pf->hw.pf_id == 0) {
- if ((flags & I40E_PRIV_FLAGS_TRUE_PROMISC_SUPPORT) &&
- !(pf->flags & I40E_FLAG_TRUE_PROMISC_SUPPORT)) {
- pf->flags |= I40E_FLAG_TRUE_PROMISC_SUPPORT;
- promisc_change = true;
- } else if (!(flags & I40E_PRIV_FLAGS_TRUE_PROMISC_SUPPORT) &&
- (pf->flags & I40E_FLAG_TRUE_PROMISC_SUPPORT)) {
- pf->flags &= ~I40E_FLAG_TRUE_PROMISC_SUPPORT;
- promisc_change = true;
- }
- }
- if (promisc_change) {
+ if (changed_flags & I40E_FLAG_TRUE_PROMISC_SUPPORT) {
+ u16 sw_flags = 0, valid_flags = 0;
+ int ret;
+
if (!(pf->flags & I40E_FLAG_TRUE_PROMISC_SUPPORT))
sw_flags = I40E_AQ_SET_SWITCH_CFG_PROMISC;
valid_flags = I40E_AQ_SET_SWITCH_CFG_PROMISC;
@@ -3137,22 +4113,17 @@
}
}
- if ((flags & I40E_PRIV_FLAGS_HW_ATR_EVICT) &&
- (pf->flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE))
- pf->auto_disable_flags &= ~I40E_FLAG_HW_ATR_EVICT_CAPABLE;
- else
- pf->auto_disable_flags |= I40E_FLAG_HW_ATR_EVICT_CAPABLE;
-
- /* if needed, issue reset to cause things to take effect */
- if (reset_required)
- i40e_do_reset(pf, BIT(__I40E_PF_RESET_REQUESTED));
+ /* Issue reset to cause things to take effect, as additional bits
+ * are added we will need to create a mask of bits requiring reset
+ */
+ if ((changed_flags & I40E_FLAG_VEB_STATS_ENABLED) ||
+ ((changed_flags & I40E_FLAG_LEGACY_RX) && netif_running(dev)))
+ i40e_do_reset(pf, BIT(__I40E_PF_RESET_REQUESTED), true);
return 0;
}
static const struct ethtool_ops i40e_ethtool_ops = {
- .get_settings = i40e_get_settings,
- .set_settings = i40e_set_settings,
.get_drvinfo = i40e_get_drvinfo,
.get_regs_len = i40e_get_regs_len,
.get_regs = i40e_get_regs,
@@ -3189,6 +4160,8 @@
.set_priv_flags = i40e_set_priv_flags,
.get_per_queue_coalesce = i40e_get_per_queue_coalesce,
.set_per_queue_coalesce = i40e_set_per_queue_coalesce,
+ .get_link_ksettings = i40e_get_link_ksettings,
+ .set_link_ksettings = i40e_set_link_ksettings,
};
void i40e_set_ethtool_ops(struct net_device *netdev)
diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c
index 82a95cc..d5c9c9e 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_main.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_main.c
@@ -32,6 +32,12 @@
#include "i40e.h"
#include "i40e_diag.h"
#include <net/udp_tunnel.h>
+/* All i40e tracepoints are defined by the include below, which
+ * must be included exactly once across the whole kernel with
+ * CREATE_TRACE_POINTS defined
+ */
+#define CREATE_TRACE_POINTS
+#include "i40e_trace.h"
const char i40e_driver_name[] = "i40e";
static const char i40e_driver_string[] =
@@ -39,9 +45,9 @@
#define DRV_KERN "-k"
-#define DRV_VERSION_MAJOR 1
-#define DRV_VERSION_MINOR 6
-#define DRV_VERSION_BUILD 27
+#define DRV_VERSION_MAJOR 2
+#define DRV_VERSION_MINOR 1
+#define DRV_VERSION_BUILD 14
#define DRV_VERSION __stringify(DRV_VERSION_MAJOR) "." \
__stringify(DRV_VERSION_MINOR) "." \
__stringify(DRV_VERSION_BUILD) DRV_KERN
@@ -50,13 +56,16 @@
/* a bit of forward declarations */
static void i40e_vsi_reinit_locked(struct i40e_vsi *vsi);
-static void i40e_handle_reset_warning(struct i40e_pf *pf);
+static void i40e_handle_reset_warning(struct i40e_pf *pf, bool lock_acquired);
static int i40e_add_vsi(struct i40e_vsi *vsi);
static int i40e_add_veb(struct i40e_veb *veb, struct i40e_vsi *vsi);
static int i40e_setup_pf_switch(struct i40e_pf *pf, bool reinit);
static int i40e_setup_misc_vector(struct i40e_pf *pf);
static void i40e_determine_queue_usage(struct i40e_pf *pf);
static int i40e_setup_pf_filter_control(struct i40e_pf *pf);
+static void i40e_prep_for_reset(struct i40e_pf *pf, bool lock_acquired);
+static int i40e_reset(struct i40e_pf *pf);
+static void i40e_rebuild(struct i40e_pf *pf, bool reinit, bool lock_acquired);
static void i40e_fdir_sb_setup(struct i40e_pf *pf);
static int i40e_veb_get_bw_info(struct i40e_veb *veb);
@@ -286,8 +295,8 @@
**/
void i40e_service_event_schedule(struct i40e_pf *pf)
{
- if (!test_bit(__I40E_DOWN, &pf->state) &&
- !test_bit(__I40E_RESET_RECOVERY_PENDING, &pf->state))
+ if (!test_bit(__I40E_VSI_DOWN, pf->state) &&
+ !test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state))
queue_work(i40e_wq, &pf->service_task);
}
@@ -299,11 +308,7 @@
* device is munged, not just the one netdev port, so go for the full
* reset.
**/
-#ifdef I40E_FCOE
-void i40e_tx_timeout(struct net_device *netdev)
-#else
static void i40e_tx_timeout(struct net_device *netdev)
-#endif
{
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_vsi *vsi = np->vsi;
@@ -372,13 +377,13 @@
switch (pf->tx_timeout_recovery_level) {
case 1:
- set_bit(__I40E_PF_RESET_REQUESTED, &pf->state);
+ set_bit(__I40E_PF_RESET_REQUESTED, pf->state);
break;
case 2:
- set_bit(__I40E_CORE_RESET_REQUESTED, &pf->state);
+ set_bit(__I40E_CORE_RESET_REQUESTED, pf->state);
break;
case 3:
- set_bit(__I40E_GLOBAL_RESET_REQUESTED, &pf->state);
+ set_bit(__I40E_GLOBAL_RESET_REQUESTED, pf->state);
break;
default:
netdev_err(netdev, "tx_timeout recovery unsuccessful\n");
@@ -408,10 +413,7 @@
* Returns the address of the device statistics structure.
* The statistics are actually updated from the service task.
**/
-#ifndef I40E_FCOE
-static
-#endif
-void i40e_get_netdev_stats_struct(struct net_device *netdev,
+static void i40e_get_netdev_stats_struct(struct net_device *netdev,
struct rtnl_link_stats64 *stats)
{
struct i40e_netdev_priv *np = netdev_priv(netdev);
@@ -420,7 +422,7 @@
struct rtnl_link_stats64 *vsi_stats = i40e_get_vsi_stats_struct(vsi);
int i;
- if (test_bit(__I40E_DOWN, &vsi->state))
+ if (test_bit(__I40E_VSI_DOWN, vsi->state))
return;
if (!vsi->tx_rings)
@@ -723,55 +725,6 @@
veb->stat_offsets_loaded = true;
}
-#ifdef I40E_FCOE
-/**
- * i40e_update_fcoe_stats - Update FCoE-specific ethernet statistics counters.
- * @vsi: the VSI that is capable of doing FCoE
- **/
-static void i40e_update_fcoe_stats(struct i40e_vsi *vsi)
-{
- struct i40e_pf *pf = vsi->back;
- struct i40e_hw *hw = &pf->hw;
- struct i40e_fcoe_stats *ofs;
- struct i40e_fcoe_stats *fs; /* device's eth stats */
- int idx;
-
- if (vsi->type != I40E_VSI_FCOE)
- return;
-
- idx = hw->pf_id + I40E_FCOE_PF_STAT_OFFSET;
- fs = &vsi->fcoe_stats;
- ofs = &vsi->fcoe_stats_offsets;
-
- i40e_stat_update32(hw, I40E_GL_FCOEPRC(idx),
- vsi->fcoe_stat_offsets_loaded,
- &ofs->rx_fcoe_packets, &fs->rx_fcoe_packets);
- i40e_stat_update48(hw, I40E_GL_FCOEDWRCH(idx), I40E_GL_FCOEDWRCL(idx),
- vsi->fcoe_stat_offsets_loaded,
- &ofs->rx_fcoe_dwords, &fs->rx_fcoe_dwords);
- i40e_stat_update32(hw, I40E_GL_FCOERPDC(idx),
- vsi->fcoe_stat_offsets_loaded,
- &ofs->rx_fcoe_dropped, &fs->rx_fcoe_dropped);
- i40e_stat_update32(hw, I40E_GL_FCOEPTC(idx),
- vsi->fcoe_stat_offsets_loaded,
- &ofs->tx_fcoe_packets, &fs->tx_fcoe_packets);
- i40e_stat_update48(hw, I40E_GL_FCOEDWTCH(idx), I40E_GL_FCOEDWTCL(idx),
- vsi->fcoe_stat_offsets_loaded,
- &ofs->tx_fcoe_dwords, &fs->tx_fcoe_dwords);
- i40e_stat_update32(hw, I40E_GL_FCOECRC(idx),
- vsi->fcoe_stat_offsets_loaded,
- &ofs->fcoe_bad_fccrc, &fs->fcoe_bad_fccrc);
- i40e_stat_update32(hw, I40E_GL_FCOELAST(idx),
- vsi->fcoe_stat_offsets_loaded,
- &ofs->fcoe_last_error, &fs->fcoe_last_error);
- i40e_stat_update32(hw, I40E_GL_FCOEDDPC(idx),
- vsi->fcoe_stat_offsets_loaded,
- &ofs->fcoe_ddp_count, &fs->fcoe_ddp_count);
-
- vsi->fcoe_stat_offsets_loaded = true;
-}
-
-#endif
/**
* i40e_update_vsi_stats - Update the vsi statistics counters.
* @vsi: the VSI to be updated
@@ -790,7 +743,6 @@
struct i40e_eth_stats *oes;
struct i40e_eth_stats *es; /* device's eth stats */
u32 tx_restart, tx_busy;
- u64 tx_lost_interrupt;
struct i40e_ring *p;
u32 rx_page, rx_buf;
u64 bytes, packets;
@@ -801,8 +753,8 @@
u64 tx_p, tx_b;
u16 q;
- if (test_bit(__I40E_DOWN, &vsi->state) ||
- test_bit(__I40E_CONFIG_BUSY, &pf->state))
+ if (test_bit(__I40E_VSI_DOWN, vsi->state) ||
+ test_bit(__I40E_CONFIG_BUSY, pf->state))
return;
ns = i40e_get_vsi_stats_struct(vsi);
@@ -816,7 +768,6 @@
rx_b = rx_p = 0;
tx_b = tx_p = 0;
tx_restart = tx_busy = tx_linearize = tx_force_wb = 0;
- tx_lost_interrupt = 0;
rx_page = 0;
rx_buf = 0;
rcu_read_lock();
@@ -835,7 +786,6 @@
tx_busy += p->tx_stats.tx_busy;
tx_linearize += p->tx_stats.tx_linearize;
tx_force_wb += p->tx_stats.tx_force_wb;
- tx_lost_interrupt += p->tx_stats.tx_lost_interrupt;
/* Rx queue is part of the same block as Tx queue */
p = &p[1];
@@ -854,7 +804,6 @@
vsi->tx_busy = tx_busy;
vsi->tx_linearize = tx_linearize;
vsi->tx_force_wb = tx_force_wb;
- vsi->tx_lost_interrupt = tx_lost_interrupt;
vsi->rx_page_failed = rx_page;
vsi->rx_buf_failed = rx_buf;
@@ -1101,13 +1050,13 @@
&osd->rx_lpi_count, &nsd->rx_lpi_count);
if (pf->flags & I40E_FLAG_FD_SB_ENABLED &&
- !(pf->auto_disable_flags & I40E_FLAG_FD_SB_ENABLED))
+ !(pf->flags & I40E_FLAG_FD_SB_AUTO_DISABLED))
nsd->fd_sb_status = true;
else
nsd->fd_sb_status = false;
if (pf->flags & I40E_FLAG_FD_ATR_ENABLED &&
- !(pf->auto_disable_flags & I40E_FLAG_FD_ATR_ENABLED))
+ !(pf->flags & I40E_FLAG_FD_ATR_AUTO_DISABLED))
nsd->fd_atr_status = true;
else
nsd->fd_atr_status = false;
@@ -1129,9 +1078,6 @@
i40e_update_pf_stats(pf);
i40e_update_vsi_stats(vsi);
-#ifdef I40E_FCOE
- i40e_update_fcoe_stats(vsi);
-#endif
}
/**
@@ -1400,7 +1346,7 @@
* to failed, so we don't bother to try sending the filter
* to the hardware.
*/
- if (test_bit(__I40E_FILTER_OVERFLOW_PROMISC, &vsi->state))
+ if (test_bit(__I40E_VSI_OVERFLOW_PROMISC, vsi->state))
f->state = I40E_FILTER_FAILED;
else
f->state = I40E_FILTER_NEW;
@@ -1562,11 +1508,7 @@
*
* Returns 0 on success, negative on failure
**/
-#ifdef I40E_FCOE
-int i40e_set_mac(struct net_device *netdev, void *p)
-#else
static int i40e_set_mac(struct net_device *netdev, void *p)
-#endif
{
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_vsi *vsi = np->vsi;
@@ -1583,8 +1525,8 @@
return 0;
}
- if (test_bit(__I40E_DOWN, &vsi->back->state) ||
- test_bit(__I40E_RESET_RECOVERY_PENDING, &vsi->back->state))
+ if (test_bit(__I40E_VSI_DOWN, vsi->back->state) ||
+ test_bit(__I40E_RESET_RECOVERY_PENDING, vsi->back->state))
return -EADDRNOTAVAIL;
if (ether_addr_equal(hw->mac.addr, addr->sa_data))
@@ -1626,17 +1568,10 @@
*
* Setup VSI queue mapping for enabled traffic classes.
**/
-#ifdef I40E_FCOE
-void i40e_vsi_setup_queue_map(struct i40e_vsi *vsi,
- struct i40e_vsi_context *ctxt,
- u8 enabled_tc,
- bool is_add)
-#else
static void i40e_vsi_setup_queue_map(struct i40e_vsi *vsi,
struct i40e_vsi_context *ctxt,
u8 enabled_tc,
bool is_add)
-#endif
{
struct i40e_pf *pf = vsi->back;
u16 sections = 0;
@@ -1686,11 +1621,6 @@
qcount = min_t(int, pf->alloc_rss_size,
num_tc_qps);
break;
-#ifdef I40E_FCOE
- case I40E_VSI_FCOE:
- qcount = num_tc_qps;
- break;
-#endif
case I40E_VSI_FDIR:
case I40E_VSI_SRIOV:
case I40E_VSI_VMDQ2:
@@ -1800,11 +1730,7 @@
* i40e_set_rx_mode - NDO callback to set the netdev filters
* @netdev: network interface device structure
**/
-#ifdef I40E_FCOE
-void i40e_set_rx_mode(struct net_device *netdev)
-#else
static void i40e_set_rx_mode(struct net_device *netdev)
-#endif
{
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_vsi *vsi = np->vsi;
@@ -1883,19 +1809,12 @@
static
struct i40e_new_mac_filter *i40e_next_filter(struct i40e_new_mac_filter *next)
{
- while (next) {
- next = hlist_entry(next->hlist.next,
- typeof(struct i40e_new_mac_filter),
- hlist);
-
- /* keep going if we found a broadcast filter */
- if (next && is_broadcast_ether_addr(next->f->macaddr))
- continue;
-
- break;
+ hlist_for_each_entry_continue(next, hlist) {
+ if (!is_broadcast_ether_addr(next->f->macaddr))
+ return next;
}
- return next;
+ return NULL;
}
/**
@@ -2001,7 +1920,7 @@
if (fcnt != num_add) {
*promisc_changed = true;
- set_bit(__I40E_FILTER_OVERFLOW_PROMISC, &vsi->state);
+ set_bit(__I40E_VSI_OVERFLOW_PROMISC, vsi->state);
dev_warn(&vsi->back->pdev->dev,
"Error %s adding RX filters on %s, promiscuous mode forced on\n",
i40e_aq_str(hw, aq_err),
@@ -2084,7 +2003,7 @@
struct i40e_aqc_add_macvlan_element_data *add_list;
struct i40e_aqc_remove_macvlan_element_data *del_list;
- while (test_and_set_bit(__I40E_CONFIG_BUSY, &vsi->state))
+ while (test_and_set_bit(__I40E_VSI_SYNCING_FILTERS, vsi->state))
usleep_range(1000, 2000);
pf = vsi->back;
@@ -2220,8 +2139,8 @@
num_add = 0;
hlist_for_each_entry_safe(new, h, &tmp_add_list, hlist) {
- if (test_bit(__I40E_FILTER_OVERFLOW_PROMISC,
- &vsi->state)) {
+ if (test_bit(__I40E_VSI_OVERFLOW_PROMISC,
+ vsi->state)) {
new->state = I40E_FILTER_FAILED;
continue;
}
@@ -2308,20 +2227,20 @@
* safely exit if we didn't just enter, we no longer have any failed
* filters, and we have reduced filters below the threshold value.
*/
- if (test_bit(__I40E_FILTER_OVERFLOW_PROMISC, &vsi->state) &&
+ if (test_bit(__I40E_VSI_OVERFLOW_PROMISC, vsi->state) &&
!promisc_changed && !failed_filters &&
(vsi->active_filters < vsi->promisc_threshold)) {
dev_info(&pf->pdev->dev,
"filter logjam cleared on %s, leaving overflow promiscuous mode\n",
vsi_name);
- clear_bit(__I40E_FILTER_OVERFLOW_PROMISC, &vsi->state);
+ clear_bit(__I40E_VSI_OVERFLOW_PROMISC, vsi->state);
promisc_changed = true;
vsi->promisc_threshold = 0;
}
/* if the VF is not trusted do not do promisc */
if ((vsi->type == I40E_VSI_SRIOV) && !pf->vf[vsi->vf_id].trusted) {
- clear_bit(__I40E_FILTER_OVERFLOW_PROMISC, &vsi->state);
+ clear_bit(__I40E_VSI_OVERFLOW_PROMISC, vsi->state);
goto out;
}
@@ -2346,12 +2265,12 @@
}
if ((changed_flags & IFF_PROMISC) ||
(promisc_changed &&
- test_bit(__I40E_FILTER_OVERFLOW_PROMISC, &vsi->state))) {
+ test_bit(__I40E_VSI_OVERFLOW_PROMISC, vsi->state))) {
bool cur_promisc;
cur_promisc = (!!(vsi->current_netdev_flags & IFF_PROMISC) ||
- test_bit(__I40E_FILTER_OVERFLOW_PROMISC,
- &vsi->state));
+ test_bit(__I40E_VSI_OVERFLOW_PROMISC,
+ vsi->state));
if ((vsi->type == I40E_VSI_MAIN) &&
(pf->lan_veb != I40E_NO_VEB) &&
!(pf->flags & I40E_FLAG_MFP_ENABLED)) {
@@ -2434,7 +2353,7 @@
if (retval)
vsi->flags |= I40E_VSI_FLAG_FILTER_CHANGED;
- clear_bit(__I40E_CONFIG_BUSY, &vsi->state);
+ clear_bit(__I40E_VSI_SYNCING_FILTERS, vsi->state);
return retval;
err_no_memory:
@@ -2446,7 +2365,7 @@
spin_unlock_bh(&vsi->mac_filter_hash_lock);
vsi->flags |= I40E_VSI_FLAG_FILTER_CHANGED;
- clear_bit(__I40E_CONFIG_BUSY, &vsi->state);
+ clear_bit(__I40E_VSI_SYNCING_FILTERS, vsi->state);
return -ENOMEM;
}
@@ -2487,13 +2406,15 @@
{
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_vsi *vsi = np->vsi;
+ struct i40e_pf *pf = vsi->back;
netdev_info(netdev, "changing MTU from %d to %d\n",
netdev->mtu, new_mtu);
netdev->mtu = new_mtu;
if (netif_running(netdev))
i40e_vsi_reinit_locked(vsi);
- i40e_notify_client_of_l2_param_changes(vsi);
+ pf->flags |= (I40E_FLAG_SERVICE_CLIENT_REQUESTED |
+ I40E_FLAG_CLIENT_L2_CHANGE);
return 0;
}
@@ -2707,13 +2628,8 @@
*
* net_device_ops implementation for adding vlan ids
**/
-#ifdef I40E_FCOE
-int i40e_vlan_rx_add_vid(struct net_device *netdev,
- __always_unused __be16 proto, u16 vid)
-#else
static int i40e_vlan_rx_add_vid(struct net_device *netdev,
__always_unused __be16 proto, u16 vid)
-#endif
{
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_vsi *vsi = np->vsi;
@@ -2744,13 +2660,8 @@
*
* net_device_ops implementation for removing vlan ids
**/
-#ifdef I40E_FCOE
-int i40e_vlan_rx_kill_vid(struct net_device *netdev,
- __always_unused __be16 proto, u16 vid)
-#else
static int i40e_vlan_rx_kill_vid(struct net_device *netdev,
__always_unused __be16 proto, u16 vid)
-#endif
{
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_vsi *vsi = np->vsi;
@@ -2920,9 +2831,6 @@
for (i = 0; i < vsi->num_queue_pairs && !err; i++)
err = i40e_setup_rx_descriptors(vsi->rx_rings[i]);
-#ifdef I40E_FCOE
- i40e_fcoe_setup_ddp_resources(vsi);
-#endif
return err;
}
@@ -2942,9 +2850,6 @@
for (i = 0; i < vsi->num_queue_pairs; i++)
if (vsi->rx_rings[i] && vsi->rx_rings[i]->desc)
i40e_free_rx_resources(vsi->rx_rings[i]);
-#ifdef I40E_FCOE
- i40e_fcoe_free_ddp_resources(vsi);
-#endif
}
/**
@@ -3015,9 +2920,6 @@
tx_ctx.qlen = ring->count;
tx_ctx.fd_ena = !!(vsi->back->flags & (I40E_FLAG_FD_SB_ENABLED |
I40E_FLAG_FD_ATR_ENABLED));
-#ifdef I40E_FCOE
- tx_ctx.fc_ena = (vsi->type == I40E_VSI_FCOE);
-#endif
tx_ctx.timesync_ena = !!(vsi->back->flags & I40E_FLAG_PTP);
/* FDIR VSI tx ring can still use RS bit and writebacks */
if (vsi->type != I40E_VSI_FDIR)
@@ -3098,7 +3000,8 @@
ring->rx_buf_len = vsi->rx_buf_len;
- rx_ctx.dbuff = ring->rx_buf_len >> I40E_RXQ_CTX_DBUFF_SHIFT;
+ rx_ctx.dbuff = DIV_ROUND_UP(ring->rx_buf_len,
+ BIT_ULL(I40E_RXQ_CTX_DBUFF_SHIFT));
rx_ctx.base = (ring->dma / 128);
rx_ctx.qlen = ring->count;
@@ -3120,9 +3023,6 @@
rx_ctx.l2tsel = 1;
/* this controls whether VLAN is stripped from inner headers */
rx_ctx.showiv = 0;
-#ifdef I40E_FCOE
- rx_ctx.fc_ena = (vsi->type == I40E_VSI_FCOE);
-#endif
/* set the prefena field to 1 because the manual says to */
rx_ctx.prefena = 1;
@@ -3144,6 +3044,12 @@
return -ENOMEM;
}
+ /* configure Rx buffer alignment */
+ if (!vsi->netdev || (vsi->back->flags & I40E_FLAG_LEGACY_RX))
+ clear_ring_build_skb_enabled(ring);
+ else
+ set_ring_build_skb_enabled(ring);
+
/* cache tail for quicker writes, and clear the reg before use */
ring->tail = hw->hw_addr + I40E_QRX_TAIL(pf_q);
writel(0, ring->tail);
@@ -3181,27 +3087,21 @@
int err = 0;
u16 i;
- if (vsi->netdev && (vsi->netdev->mtu > ETH_DATA_LEN))
- vsi->max_frame = vsi->netdev->mtu + ETH_HLEN
- + ETH_FCS_LEN + VLAN_HLEN;
- else
- vsi->max_frame = I40E_RXBUFFER_2048;
-
- vsi->rx_buf_len = I40E_RXBUFFER_2048;
-
-#ifdef I40E_FCOE
- /* setup rx buffer for FCoE */
- if ((vsi->type == I40E_VSI_FCOE) &&
- (vsi->back->flags & I40E_FLAG_FCOE_ENABLED)) {
- vsi->rx_buf_len = I40E_RXBUFFER_3072;
- vsi->max_frame = I40E_RXBUFFER_3072;
+ if (!vsi->netdev || (vsi->back->flags & I40E_FLAG_LEGACY_RX)) {
+ vsi->max_frame = I40E_MAX_RXBUFFER;
+ vsi->rx_buf_len = I40E_RXBUFFER_2048;
+#if (PAGE_SIZE < 8192)
+ } else if (!I40E_2K_TOO_SMALL_WITH_PADDING &&
+ (vsi->netdev->mtu <= ETH_DATA_LEN)) {
+ vsi->max_frame = I40E_RXBUFFER_1536 - NET_IP_ALIGN;
+ vsi->rx_buf_len = I40E_RXBUFFER_1536 - NET_IP_ALIGN;
+#endif
+ } else {
+ vsi->max_frame = I40E_MAX_RXBUFFER;
+ vsi->rx_buf_len = (PAGE_SIZE < 8192) ? I40E_RXBUFFER_3072 :
+ I40E_RXBUFFER_2048;
}
-#endif /* I40E_FCOE */
- /* round up for the chip's needs */
- vsi->rx_buf_len = ALIGN(vsi->rx_buf_len,
- BIT_ULL(I40E_RXQ_CTX_DBUFF_SHIFT));
-
/* set up individual rings */
for (i = 0; i < vsi->num_queue_pairs && !err; i++)
err = i40e_configure_rx_ring(vsi->rx_rings[i]);
@@ -3281,6 +3181,12 @@
if (!(pf->flags & I40E_FLAG_FD_SB_ENABLED))
return;
+ /* Reset FDir counters as we're replaying all existing filters */
+ pf->fd_tcp4_filter_cnt = 0;
+ pf->fd_udp4_filter_cnt = 0;
+ pf->fd_sctp4_filter_cnt = 0;
+ pf->fd_ip4_filter_cnt = 0;
+
hlist_for_each_entry_safe(filter, node,
&pf->fdir_filter_list, fdir_node) {
i40e_add_del_fdir(vsi, filter, true);
@@ -3705,29 +3611,29 @@
* this is not a performance path and napi_schedule()
* can deal with rescheduling.
*/
- if (!test_bit(__I40E_DOWN, &pf->state))
+ if (!test_bit(__I40E_VSI_DOWN, pf->state))
napi_schedule_irqoff(&q_vector->napi);
}
if (icr0 & I40E_PFINT_ICR0_ADMINQ_MASK) {
ena_mask &= ~I40E_PFINT_ICR0_ENA_ADMINQ_MASK;
- set_bit(__I40E_ADMINQ_EVENT_PENDING, &pf->state);
+ set_bit(__I40E_ADMINQ_EVENT_PENDING, pf->state);
i40e_debug(&pf->hw, I40E_DEBUG_NVM, "AdminQ event\n");
}
if (icr0 & I40E_PFINT_ICR0_MAL_DETECT_MASK) {
ena_mask &= ~I40E_PFINT_ICR0_ENA_MAL_DETECT_MASK;
- set_bit(__I40E_MDD_EVENT_PENDING, &pf->state);
+ set_bit(__I40E_MDD_EVENT_PENDING, pf->state);
}
if (icr0 & I40E_PFINT_ICR0_VFLR_MASK) {
ena_mask &= ~I40E_PFINT_ICR0_ENA_VFLR_MASK;
- set_bit(__I40E_VFLR_EVENT_PENDING, &pf->state);
+ set_bit(__I40E_VFLR_EVENT_PENDING, pf->state);
}
if (icr0 & I40E_PFINT_ICR0_GRST_MASK) {
- if (!test_bit(__I40E_RESET_RECOVERY_PENDING, &pf->state))
- set_bit(__I40E_RESET_INTR_RECEIVED, &pf->state);
+ if (!test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state))
+ set_bit(__I40E_RESET_INTR_RECEIVED, pf->state);
ena_mask &= ~I40E_PFINT_ICR0_ENA_GRST_MASK;
val = rd32(hw, I40E_GLGEN_RSTAT);
val = (val & I40E_GLGEN_RSTAT_RESET_TYPE_MASK)
@@ -3738,7 +3644,7 @@
pf->globr_count++;
} else if (val == I40E_RESET_EMPR) {
pf->empr_count++;
- set_bit(__I40E_EMP_RESET_INTR_RECEIVED, &pf->state);
+ set_bit(__I40E_EMP_RESET_INTR_RECEIVED, pf->state);
}
}
@@ -3771,7 +3677,7 @@
(icr0_remaining & I40E_PFINT_ICR0_PCI_EXCEPTION_MASK) ||
(icr0_remaining & I40E_PFINT_ICR0_ECC_ERR_MASK)) {
dev_info(&pf->pdev->dev, "device will be reset\n");
- set_bit(__I40E_PF_RESET_REQUESTED, &pf->state);
+ set_bit(__I40E_PF_RESET_REQUESTED, pf->state);
i40e_service_event_schedule(pf);
}
ena_mask &= ~icr0_remaining;
@@ -3781,7 +3687,7 @@
enable_intr:
/* re-enable interrupt causes */
wr32(hw, I40E_PFINT_ICR0_ENA, ena_mask);
- if (!test_bit(__I40E_DOWN, &pf->state)) {
+ if (!test_bit(__I40E_VSI_DOWN, pf->state)) {
i40e_service_event_schedule(pf);
i40e_irq_dynamic_enable_icr0(pf, false);
}
@@ -3993,11 +3899,7 @@
* This is used by netconsole to send skbs without having to re-enable
* interrupts. It's not called while the normal interrupt routine is executing.
**/
-#ifdef I40E_FCOE
-void i40e_netpoll(struct net_device *netdev)
-#else
static void i40e_netpoll(struct net_device *netdev)
-#endif
{
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_vsi *vsi = np->vsi;
@@ -4005,7 +3907,7 @@
int i;
/* if interface is down do nothing */
- if (test_bit(__I40E_DOWN, &vsi->state))
+ if (test_bit(__I40E_VSI_DOWN, vsi->state))
return;
if (pf->flags & I40E_FLAG_MSIX_ENABLED) {
@@ -4017,6 +3919,8 @@
}
#endif
+#define I40E_QTX_ENA_WAIT_COUNT 50
+
/**
* i40e_pf_txq_wait - Wait for a PF's Tx queue to be enabled or disabled
* @pf: the PF being configured
@@ -4047,6 +3951,50 @@
}
/**
+ * i40e_control_tx_q - Start or stop a particular Tx queue
+ * @pf: the PF structure
+ * @pf_q: the PF queue to configure
+ * @enable: start or stop the queue
+ *
+ * This function enables or disables a single queue. Note that any delay
+ * required after the operation is expected to be handled by the caller of
+ * this function.
+ **/
+static void i40e_control_tx_q(struct i40e_pf *pf, int pf_q, bool enable)
+{
+ struct i40e_hw *hw = &pf->hw;
+ u32 tx_reg;
+ int i;
+
+ /* warn the TX unit of coming changes */
+ i40e_pre_tx_queue_cfg(&pf->hw, pf_q, enable);
+ if (!enable)
+ usleep_range(10, 20);
+
+ for (i = 0; i < I40E_QTX_ENA_WAIT_COUNT; i++) {
+ tx_reg = rd32(hw, I40E_QTX_ENA(pf_q));
+ if (((tx_reg >> I40E_QTX_ENA_QENA_REQ_SHIFT) & 1) ==
+ ((tx_reg >> I40E_QTX_ENA_QENA_STAT_SHIFT) & 1))
+ break;
+ usleep_range(1000, 2000);
+ }
+
+ /* Skip if the queue is already in the requested state */
+ if (enable == !!(tx_reg & I40E_QTX_ENA_QENA_STAT_MASK))
+ return;
+
+ /* turn on/off the queue */
+ if (enable) {
+ wr32(hw, I40E_QTX_HEAD(pf_q), 0);
+ tx_reg |= I40E_QTX_ENA_QENA_REQ_MASK;
+ } else {
+ tx_reg &= ~I40E_QTX_ENA_QENA_REQ_MASK;
+ }
+
+ wr32(hw, I40E_QTX_ENA(pf_q), tx_reg);
+}
+
+/**
* i40e_vsi_control_tx - Start or stop a VSI's rings
* @vsi: the VSI being configured
* @enable: start or stop the rings
@@ -4054,41 +4002,11 @@
static int i40e_vsi_control_tx(struct i40e_vsi *vsi, bool enable)
{
struct i40e_pf *pf = vsi->back;
- struct i40e_hw *hw = &pf->hw;
- int i, j, pf_q, ret = 0;
- u32 tx_reg;
+ int i, pf_q, ret = 0;
pf_q = vsi->base_queue;
for (i = 0; i < vsi->num_queue_pairs; i++, pf_q++) {
-
- /* warn the TX unit of coming changes */
- i40e_pre_tx_queue_cfg(&pf->hw, pf_q, enable);
- if (!enable)
- usleep_range(10, 20);
-
- for (j = 0; j < 50; j++) {
- tx_reg = rd32(hw, I40E_QTX_ENA(pf_q));
- if (((tx_reg >> I40E_QTX_ENA_QENA_REQ_SHIFT) & 1) ==
- ((tx_reg >> I40E_QTX_ENA_QENA_STAT_SHIFT) & 1))
- break;
- usleep_range(1000, 2000);
- }
- /* Skip if the queue is already in the requested state */
- if (enable == !!(tx_reg & I40E_QTX_ENA_QENA_STAT_MASK))
- continue;
-
- /* turn on/off the queue */
- if (enable) {
- wr32(hw, I40E_QTX_HEAD(pf_q), 0);
- tx_reg |= I40E_QTX_ENA_QENA_REQ_MASK;
- } else {
- tx_reg &= ~I40E_QTX_ENA_QENA_REQ_MASK;
- }
-
- wr32(hw, I40E_QTX_ENA(pf_q), tx_reg);
- /* No waiting for the Tx queue to disable */
- if (!enable && test_bit(__I40E_PORT_TX_SUSPENDED, &pf->state))
- continue;
+ i40e_control_tx_q(pf, pf_q, enable);
/* wait for the change to finish */
ret = i40e_pf_txq_wait(pf, pf_q, enable);
@@ -4100,8 +4018,6 @@
}
}
- if (hw->revision_id == 0)
- mdelay(50);
return ret;
}
@@ -4135,6 +4051,43 @@
}
/**
+ * i40e_control_rx_q - Start or stop a particular Rx queue
+ * @pf: the PF structure
+ * @pf_q: the PF queue to configure
+ * @enable: start or stop the queue
+ *
+ * This function enables or disables a single queue. Note that any delay
+ * required after the operation is expected to be handled by the caller of
+ * this function.
+ **/
+static void i40e_control_rx_q(struct i40e_pf *pf, int pf_q, bool enable)
+{
+ struct i40e_hw *hw = &pf->hw;
+ u32 rx_reg;
+ int i;
+
+ for (i = 0; i < I40E_QTX_ENA_WAIT_COUNT; i++) {
+ rx_reg = rd32(hw, I40E_QRX_ENA(pf_q));
+ if (((rx_reg >> I40E_QRX_ENA_QENA_REQ_SHIFT) & 1) ==
+ ((rx_reg >> I40E_QRX_ENA_QENA_STAT_SHIFT) & 1))
+ break;
+ usleep_range(1000, 2000);
+ }
+
+ /* Skip if the queue is already in the requested state */
+ if (enable == !!(rx_reg & I40E_QRX_ENA_QENA_STAT_MASK))
+ return;
+
+ /* turn on/off the queue */
+ if (enable)
+ rx_reg |= I40E_QRX_ENA_QENA_REQ_MASK;
+ else
+ rx_reg &= ~I40E_QRX_ENA_QENA_REQ_MASK;
+
+ wr32(hw, I40E_QRX_ENA(pf_q), rx_reg);
+}
+
+/**
* i40e_vsi_control_rx - Start or stop a VSI's rings
* @vsi: the VSI being configured
* @enable: start or stop the rings
@@ -4142,33 +4095,11 @@
static int i40e_vsi_control_rx(struct i40e_vsi *vsi, bool enable)
{
struct i40e_pf *pf = vsi->back;
- struct i40e_hw *hw = &pf->hw;
- int i, j, pf_q, ret = 0;
- u32 rx_reg;
+ int i, pf_q, ret = 0;
pf_q = vsi->base_queue;
for (i = 0; i < vsi->num_queue_pairs; i++, pf_q++) {
- for (j = 0; j < 50; j++) {
- rx_reg = rd32(hw, I40E_QRX_ENA(pf_q));
- if (((rx_reg >> I40E_QRX_ENA_QENA_REQ_SHIFT) & 1) ==
- ((rx_reg >> I40E_QRX_ENA_QENA_STAT_SHIFT) & 1))
- break;
- usleep_range(1000, 2000);
- }
-
- /* Skip if the queue is already in the requested state */
- if (enable == !!(rx_reg & I40E_QRX_ENA_QENA_STAT_MASK))
- continue;
-
- /* turn on/off the queue */
- if (enable)
- rx_reg |= I40E_QRX_ENA_QENA_REQ_MASK;
- else
- rx_reg &= ~I40E_QRX_ENA_QENA_REQ_MASK;
- wr32(hw, I40E_QRX_ENA(pf_q), rx_reg);
- /* No waiting for the Tx queue to disable */
- if (!enable && test_bit(__I40E_PORT_TX_SUSPENDED, &pf->state))
- continue;
+ i40e_control_rx_q(pf, pf_q, enable);
/* wait for the change to finish */
ret = i40e_pf_rxq_wait(pf, pf_q, enable);
@@ -4180,6 +4111,12 @@
}
}
+ /* Due to HW errata, on Rx disable only, the register can indicate done
+ * before it really is. Needs 50ms to be sure
+ */
+ if (!enable)
+ mdelay(50);
+
return ret;
}
@@ -4206,6 +4143,10 @@
**/
void i40e_vsi_stop_rings(struct i40e_vsi *vsi)
{
+ /* When port TX is suspended, don't wait */
+ if (test_bit(__I40E_PORT_SUSPENDED, vsi->back->state))
+ return i40e_vsi_stop_rings_no_wait(vsi);
+
/* do rx first for enable and last for disable
* Ignore return value, we need to shutdown whatever we can
*/
@@ -4214,6 +4155,29 @@
}
/**
+ * i40e_vsi_stop_rings_no_wait - Stop a VSI's rings and do not delay
+ * @vsi: the VSI being shutdown
+ *
+ * This function stops all the rings for a VSI but does not delay to verify
+ * that rings have been disabled. It is expected that the caller is shutting
+ * down multiple VSIs at once and will delay together for all the VSIs after
+ * initiating the shutdown. This is particularly useful for shutting down lots
+ * of VFs together. Otherwise, a large delay can be incurred while configuring
+ * each VSI in serial.
+ **/
+void i40e_vsi_stop_rings_no_wait(struct i40e_vsi *vsi)
+{
+ struct i40e_pf *pf = vsi->back;
+ int i, pf_q;
+
+ pf_q = vsi->base_queue;
+ for (i = 0; i < vsi->num_queue_pairs; i++, pf_q++) {
+ i40e_control_tx_q(pf, pf_q, false);
+ i40e_control_rx_q(pf, pf_q, false);
+ }
+}
+
+/**
* i40e_vsi_free_irq - Free the irq association with the OS
* @vsi: the VSI being configured
**/
@@ -4471,17 +4435,16 @@
**/
static void i40e_vsi_close(struct i40e_vsi *vsi)
{
- bool reset = false;
-
- if (!test_and_set_bit(__I40E_DOWN, &vsi->state))
+ struct i40e_pf *pf = vsi->back;
+ if (!test_and_set_bit(__I40E_VSI_DOWN, vsi->state))
i40e_down(vsi);
i40e_vsi_free_irq(vsi);
i40e_vsi_free_tx_resources(vsi);
i40e_vsi_free_rx_resources(vsi);
vsi->current_netdev_flags = 0;
- if (test_bit(__I40E_RESET_RECOVERY_PENDING, &vsi->back->state))
- reset = true;
- i40e_notify_client_of_netdev_close(vsi, reset);
+ pf->flags |= I40E_FLAG_SERVICE_CLIENT_REQUESTED;
+ if (test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state))
+ pf->flags |= I40E_FLAG_CLIENT_RESET;
}
/**
@@ -4490,18 +4453,10 @@
**/
static void i40e_quiesce_vsi(struct i40e_vsi *vsi)
{
- if (test_bit(__I40E_DOWN, &vsi->state))
+ if (test_bit(__I40E_VSI_DOWN, vsi->state))
return;
- /* No need to disable FCoE VSI when Tx suspended */
- if ((test_bit(__I40E_PORT_TX_SUSPENDED, &vsi->back->state)) &&
- vsi->type == I40E_VSI_FCOE) {
- dev_dbg(&vsi->back->pdev->dev,
- "VSI seid %d skipping FCoE VSI disable\n", vsi->seid);
- return;
- }
-
- set_bit(__I40E_NEEDS_RESTART, &vsi->state);
+ set_bit(__I40E_VSI_NEEDS_RESTART, vsi->state);
if (vsi->netdev && netif_running(vsi->netdev))
vsi->netdev->netdev_ops->ndo_stop(vsi->netdev);
else
@@ -4514,10 +4469,9 @@
**/
static void i40e_unquiesce_vsi(struct i40e_vsi *vsi)
{
- if (!test_bit(__I40E_NEEDS_RESTART, &vsi->state))
+ if (!test_and_clear_bit(__I40E_VSI_NEEDS_RESTART, vsi->state))
return;
- clear_bit(__I40E_NEEDS_RESTART, &vsi->state);
if (vsi->netdev && netif_running(vsi->netdev))
vsi->netdev->netdev_ops->ndo_open(vsi->netdev);
else
@@ -4552,21 +4506,20 @@
}
}
-#ifdef CONFIG_I40E_DCB
/**
* i40e_vsi_wait_queues_disabled - Wait for VSI's queues to be disabled
* @vsi: the VSI being configured
*
- * This function waits for the given VSI's queues to be disabled.
+ * Wait until all queues on a given VSI have been disabled.
**/
-static int i40e_vsi_wait_queues_disabled(struct i40e_vsi *vsi)
+int i40e_vsi_wait_queues_disabled(struct i40e_vsi *vsi)
{
struct i40e_pf *pf = vsi->back;
int i, pf_q, ret;
pf_q = vsi->base_queue;
for (i = 0; i < vsi->num_queue_pairs; i++, pf_q++) {
- /* Check and wait for the disable status of the queue */
+ /* Check and wait for the Tx queue */
ret = i40e_pf_txq_wait(pf, pf_q, false);
if (ret) {
dev_info(&pf->pdev->dev,
@@ -4574,11 +4527,7 @@
vsi->seid, pf_q);
return ret;
}
- }
-
- pf_q = vsi->base_queue;
- for (i = 0; i < vsi->num_queue_pairs; i++, pf_q++) {
- /* Check and wait for the disable status of the queue */
+ /* Check and wait for the Tx queue */
ret = i40e_pf_rxq_wait(pf, pf_q, false);
if (ret) {
dev_info(&pf->pdev->dev,
@@ -4591,6 +4540,7 @@
return 0;
}
+#ifdef CONFIG_I40E_DCB
/**
* i40e_pf_wait_queues_disabled - Wait for all queues of PF VSIs to be disabled
* @pf: the PF
@@ -4603,8 +4553,7 @@
int v, ret = 0;
for (v = 0; v < pf->hw.func_caps.num_vsis; v++) {
- /* No need to wait for FCoE VSI queues */
- if (pf->vsi[v] && pf->vsi[v]->type != I40E_VSI_FCOE) {
+ if (pf->vsi[v]) {
ret = i40e_vsi_wait_queues_disabled(pf->vsi[v]);
if (ret)
break;
@@ -4622,16 +4571,15 @@
* @vsi: Pointer to VSI struct
*
* This function checks specified queue for given VSI. Detects hung condition.
- * Sets hung bit since it is two step process. Before next run of service task
- * if napi_poll runs, it reset 'hung' bit for respective q_vector. If not,
- * hung condition remain unchanged and during subsequent run, this function
- * issues SW interrupt to recover from hung condition.
+ * We proactively detect hung TX queues by checking if interrupts are disabled
+ * but there are pending descriptors. If it appears hung, attempt to recover
+ * by triggering a SW interrupt.
**/
static void i40e_detect_recover_hung_queue(int q_idx, struct i40e_vsi *vsi)
{
struct i40e_ring *tx_ring = NULL;
struct i40e_pf *pf;
- u32 head, val, tx_pending_hw;
+ u32 val, tx_pending;
int i;
pf = vsi->back;
@@ -4657,47 +4605,15 @@
else
val = rd32(&pf->hw, I40E_PFINT_DYN_CTL0);
- head = i40e_get_head(tx_ring);
+ tx_pending = i40e_get_tx_pending(tx_ring);
- tx_pending_hw = i40e_get_tx_pending(tx_ring, false);
-
- /* HW is done executing descriptors, updated HEAD write back,
- * but SW hasn't processed those descriptors. If interrupt is
- * not generated from this point ON, it could result into
- * dev_watchdog detecting timeout on those netdev_queue,
- * hence proactively trigger SW interrupt.
+ /* Interrupts are disabled and TX pending is non-zero,
+ * trigger the SW interrupt (don't wait). Worst case
+ * there will be one extra interrupt which may result
+ * into not cleaning any queues because queues are cleaned.
*/
- if (tx_pending_hw && (!(val & I40E_PFINT_DYN_CTLN_INTENA_MASK))) {
- /* NAPI Poll didn't run and clear since it was set */
- if (test_and_clear_bit(I40E_Q_VECTOR_HUNG_DETECT,
- &tx_ring->q_vector->hung_detected)) {
- netdev_info(vsi->netdev, "VSI_seid %d, Hung TX queue %d, tx_pending_hw: %d, NTC:0x%x, HWB: 0x%x, NTU: 0x%x, TAIL: 0x%x\n",
- vsi->seid, q_idx, tx_pending_hw,
- tx_ring->next_to_clean, head,
- tx_ring->next_to_use,
- readl(tx_ring->tail));
- netdev_info(vsi->netdev, "VSI_seid %d, Issuing force_wb for TX queue %d, Interrupt Reg: 0x%x\n",
- vsi->seid, q_idx, val);
- i40e_force_wb(vsi, tx_ring->q_vector);
- } else {
- /* First Chance - detected possible hung */
- set_bit(I40E_Q_VECTOR_HUNG_DETECT,
- &tx_ring->q_vector->hung_detected);
- }
- }
-
- /* This is the case where we have interrupts missing,
- * so the tx_pending in HW will most likely be 0, but we
- * will have tx_pending in SW since the WB happened but the
- * interrupt got lost.
- */
- if ((!tx_pending_hw) && i40e_get_tx_pending(tx_ring, true) &&
- (!(val & I40E_PFINT_DYN_CTLN_INTENA_MASK))) {
- local_bh_disable();
- if (napi_reschedule(&tx_ring->q_vector->napi))
- tx_ring->tx_stats.tx_lost_interrupt++;
- local_bh_enable();
- }
+ if (tx_pending && (!(val & I40E_PFINT_DYN_CTLN_INTENA_MASK)))
+ i40e_force_wb(vsi, tx_ring->q_vector);
}
/**
@@ -4721,8 +4637,8 @@
return;
/* Make sure, VSI state is not DOWN/RECOVERY_PENDING */
- if (test_bit(__I40E_DOWN, &vsi->back->state) ||
- test_bit(__I40E_RESET_RECOVERY_PENDING, &vsi->back->state))
+ if (test_bit(__I40E_VSI_DOWN, vsi->back->state) ||
+ test_bit(__I40E_RESET_RECOVERY_PENDING, vsi->back->state))
return;
/* Make sure type is MAIN VSI */
@@ -5228,20 +5144,12 @@
continue;
/* - Enable all TCs for the LAN VSI
-#ifdef I40E_FCOE
- * - For FCoE VSI only enable the TC configured
- * as per the APP TLV
-#endif
* - For all others keep them at TC0 for now
*/
if (v == pf->lan_vsi)
tc_map = i40e_pf_get_tc_map(pf);
else
tc_map = I40E_DEFAULT_TRAFFIC_CLASS;
-#ifdef I40E_FCOE
- if (pf->vsi[v]->type == I40E_VSI_FCOE)
- tc_map = i40e_get_fcoe_tc_map(pf);
-#endif /* #ifdef I40E_FCOE */
ret = i40e_vsi_config_tc(pf->vsi[v], tc_map);
if (ret) {
@@ -5277,7 +5185,7 @@
i40e_stat_str(&pf->hw, ret),
i40e_aq_str(&pf->hw, pf->hw.aq.asq_last_status));
/* Schedule PF reset to recover */
- set_bit(__I40E_PF_RESET_REQUESTED, &pf->state);
+ set_bit(__I40E_PF_RESET_REQUESTED, pf->state);
i40e_service_event_schedule(pf);
}
@@ -5308,10 +5216,6 @@
(hw->dcbx_status == I40E_DCBX_STATUS_DISABLED)) {
dev_info(&pf->pdev->dev,
"DCBX offload is not supported or is disabled for this PF.\n");
-
- if (pf->flags & I40E_FLAG_MFP_ENABLED)
- goto out;
-
} else {
/* When status is not DISABLED then DCBX in FW */
pf->dcbx_cap = DCB_CAP_DCBX_LLD_MANAGED |
@@ -5449,7 +5353,7 @@
if (err)
return err;
- clear_bit(__I40E_DOWN, &vsi->state);
+ clear_bit(__I40E_VSI_DOWN, vsi->state);
i40e_napi_enable_all(vsi);
i40e_vsi_enable_irq(vsi);
@@ -5472,13 +5376,8 @@
/* replay FDIR SB filters */
if (vsi->type == I40E_VSI_FDIR) {
/* reset fd counters */
- pf->fd_add_err = pf->fd_atr_cnt = 0;
- if (pf->fd_tcp_rule > 0) {
- pf->auto_disable_flags |= I40E_FLAG_FD_ATR_ENABLED;
- if (I40E_DEBUG_FD & pf->hw.debug_mask)
- dev_info(&pf->pdev->dev, "Forcing ATR off, sideband rules for TCP/IPv4 exist\n");
- pf->fd_tcp_rule = 0;
- }
+ pf->fd_add_err = 0;
+ pf->fd_atr_cnt = 0;
i40e_fdir_filter_restore(vsi);
}
@@ -5503,12 +5402,12 @@
struct i40e_pf *pf = vsi->back;
WARN_ON(in_interrupt());
- while (test_and_set_bit(__I40E_CONFIG_BUSY, &pf->state))
+ while (test_and_set_bit(__I40E_CONFIG_BUSY, pf->state))
usleep_range(1000, 2000);
i40e_down(vsi);
i40e_up(vsi);
- clear_bit(__I40E_CONFIG_BUSY, &pf->state);
+ clear_bit(__I40E_CONFIG_BUSY, pf->state);
}
/**
@@ -5535,7 +5434,7 @@
int i;
/* It is assumed that the caller of this function
- * sets the vsi->state __I40E_DOWN bit.
+ * sets the vsi->state __I40E_VSI_DOWN bit.
*/
if (vsi->netdev) {
netif_carrier_off(vsi->netdev);
@@ -5550,8 +5449,6 @@
i40e_clean_rx_ring(vsi->rx_rings[i]);
}
- i40e_notify_client_of_netdev_close(vsi, false);
-
}
/**
@@ -5612,17 +5509,15 @@
return ret;
}
-#ifdef I40E_FCOE
-int __i40e_setup_tc(struct net_device *netdev, u32 handle, __be16 proto,
- struct tc_to_netdev *tc)
-#else
static int __i40e_setup_tc(struct net_device *netdev, u32 handle, __be16 proto,
struct tc_to_netdev *tc)
-#endif
{
- if (handle != TC_H_ROOT || tc->type != TC_SETUP_MQPRIO)
+ if (tc->type != TC_SETUP_MQPRIO)
return -EINVAL;
- return i40e_setup_tc(netdev, tc->tc);
+
+ tc->mqprio->hw = TC_MQPRIO_HW_OFFLOAD_TCS;
+
+ return i40e_setup_tc(netdev, tc->mqprio->num_tc);
}
/**
@@ -5645,8 +5540,8 @@
int err;
/* disallow open during test or if eeprom is broken */
- if (test_bit(__I40E_TESTING, &pf->state) ||
- test_bit(__I40E_BAD_EEPROM, &pf->state))
+ if (test_bit(__I40E_TESTING, pf->state) ||
+ test_bit(__I40E_BAD_EEPROM, pf->state))
return -EBUSY;
netif_carrier_off(netdev);
@@ -5675,6 +5570,8 @@
* Finish initialization of the VSI.
*
* Returns 0 on success, negative value on failure
+ *
+ * Note: expects to be called while under rtnl_lock()
**/
int i40e_vsi_open(struct i40e_vsi *vsi)
{
@@ -5738,7 +5635,7 @@
err_setup_tx:
i40e_vsi_free_tx_resources(vsi);
if (vsi == pf->vsi[pf->lan_vsi])
- i40e_do_reset(pf, BIT_ULL(__I40E_PF_RESET_REQUESTED));
+ i40e_do_reset(pf, BIT_ULL(__I40E_PF_RESET_REQUESTED), true);
return err;
}
@@ -5753,6 +5650,7 @@
static void i40e_fdir_filter_exit(struct i40e_pf *pf)
{
struct i40e_fdir_filter *filter;
+ struct i40e_flex_pit *pit_entry, *tmp;
struct hlist_node *node2;
hlist_for_each_entry_safe(filter, node2,
@@ -5760,7 +5658,43 @@
hlist_del(&filter->fdir_node);
kfree(filter);
}
+
+ list_for_each_entry_safe(pit_entry, tmp, &pf->l3_flex_pit_list, list) {
+ list_del(&pit_entry->list);
+ kfree(pit_entry);
+ }
+ INIT_LIST_HEAD(&pf->l3_flex_pit_list);
+
+ list_for_each_entry_safe(pit_entry, tmp, &pf->l4_flex_pit_list, list) {
+ list_del(&pit_entry->list);
+ kfree(pit_entry);
+ }
+ INIT_LIST_HEAD(&pf->l4_flex_pit_list);
+
pf->fdir_pf_active_filters = 0;
+ pf->fd_tcp4_filter_cnt = 0;
+ pf->fd_udp4_filter_cnt = 0;
+ pf->fd_sctp4_filter_cnt = 0;
+ pf->fd_ip4_filter_cnt = 0;
+
+ /* Reprogram the default input set for TCP/IPv4 */
+ i40e_write_fd_input_set(pf, I40E_FILTER_PCTYPE_NONF_IPV4_TCP,
+ I40E_L3_SRC_MASK | I40E_L3_DST_MASK |
+ I40E_L4_SRC_MASK | I40E_L4_DST_MASK);
+
+ /* Reprogram the default input set for UDP/IPv4 */
+ i40e_write_fd_input_set(pf, I40E_FILTER_PCTYPE_NONF_IPV4_UDP,
+ I40E_L3_SRC_MASK | I40E_L3_DST_MASK |
+ I40E_L4_SRC_MASK | I40E_L4_DST_MASK);
+
+ /* Reprogram the default input set for SCTP/IPv4 */
+ i40e_write_fd_input_set(pf, I40E_FILTER_PCTYPE_NONF_IPV4_SCTP,
+ I40E_L3_SRC_MASK | I40E_L3_DST_MASK |
+ I40E_L4_SRC_MASK | I40E_L4_DST_MASK);
+
+ /* Reprogram the default input set for Other/IPv4 */
+ i40e_write_fd_input_set(pf, I40E_FILTER_PCTYPE_NONF_IPV4_OTHER,
+ I40E_L3_SRC_MASK | I40E_L3_DST_MASK);
}
/**
@@ -5787,12 +5721,14 @@
* i40e_do_reset - Start a PF or Core Reset sequence
* @pf: board private structure
* @reset_flags: which reset is requested
+ * @lock_acquired: indicates whether or not the lock has been acquired
+ * before this function was called.
*
* The essential difference in resets is that the PF Reset
* doesn't clear the packet buffers, doesn't reset the PE
* firmware, and doesn't bother the other PFs on the chip.
**/
-void i40e_do_reset(struct i40e_pf *pf, u32 reset_flags)
+void i40e_do_reset(struct i40e_pf *pf, u32 reset_flags, bool lock_acquired)
{
u32 val;
@@ -5838,7 +5774,7 @@
* for the Core Reset.
*/
dev_dbg(&pf->pdev->dev, "PFR requested\n");
- i40e_handle_reset_warning(pf);
+ i40e_handle_reset_warning(pf, lock_acquired);
} else if (reset_flags & BIT_ULL(__I40E_REINIT_REQUESTED)) {
int v;
@@ -5850,10 +5786,9 @@
struct i40e_vsi *vsi = pf->vsi[v];
if (vsi != NULL &&
- test_bit(__I40E_REINIT_REQUESTED, &vsi->state)) {
+ test_and_clear_bit(__I40E_VSI_REINIT_REQUESTED,
+ vsi->state))
i40e_vsi_reinit_locked(pf->vsi[v]);
- clear_bit(__I40E_REINIT_REQUESTED, &vsi->state);
- }
}
} else if (reset_flags & BIT_ULL(__I40E_DOWN_REQUESTED)) {
int v;
@@ -5864,10 +5799,10 @@
struct i40e_vsi *vsi = pf->vsi[v];
if (vsi != NULL &&
- test_bit(__I40E_DOWN_REQUESTED, &vsi->state)) {
- set_bit(__I40E_DOWN, &vsi->state);
+ test_and_clear_bit(__I40E_VSI_DOWN_REQUESTED,
+ vsi->state)) {
+ set_bit(__I40E_VSI_DOWN, vsi->state);
i40e_down(vsi);
- clear_bit(__I40E_DOWN_REQUESTED, &vsi->state);
}
}
} else {
@@ -6007,7 +5942,7 @@
else
pf->flags &= ~I40E_FLAG_DCB_ENABLED;
- set_bit(__I40E_PORT_TX_SUSPENDED, &pf->state);
+ set_bit(__I40E_PORT_SUSPENDED, pf->state);
/* Reconfiguration needed quiesce all VSIs */
i40e_pf_quiesce_all_vsi(pf);
@@ -6016,7 +5951,7 @@
ret = i40e_resume_port_tx(pf);
- clear_bit(__I40E_PORT_TX_SUSPENDED, &pf->state);
+ clear_bit(__I40E_PORT_SUSPENDED, pf->state);
/* In case of error no point in resuming VSIs */
if (ret)
goto exit;
@@ -6025,12 +5960,12 @@
ret = i40e_pf_wait_queues_disabled(pf);
if (ret) {
/* Schedule PF reset to recover */
- set_bit(__I40E_PF_RESET_REQUESTED, &pf->state);
+ set_bit(__I40E_PF_RESET_REQUESTED, pf->state);
i40e_service_event_schedule(pf);
} else {
i40e_pf_unquiesce_all_vsi(pf);
- /* Notify the client for the DCB changes */
- i40e_notify_client_of_l2_param_changes(pf->vsi[pf->lan_vsi]);
+ pf->flags |= (I40E_FLAG_SERVICE_CLIENT_REQUESTED |
+ I40E_FLAG_CLIENT_L2_CHANGE);
}
exit:
@@ -6047,7 +5982,7 @@
void i40e_do_reset_safe(struct i40e_pf *pf, u32 reset_flags)
{
rtnl_lock();
- i40e_do_reset(pf, reset_flags);
+ i40e_do_reset(pf, reset_flags, true);
rtnl_unlock();
}
@@ -6140,34 +6075,33 @@
u32 fcnt_prog, fcnt_avail;
struct hlist_node *node;
- if (test_bit(__I40E_FD_FLUSH_REQUESTED, &pf->state))
+ if (test_bit(__I40E_FD_FLUSH_REQUESTED, pf->state))
return;
- /* Check if, FD SB or ATR was auto disabled and if there is enough room
- * to re-enable
- */
+ /* Check if we have enough room to re-enable FDir SB capability. */
fcnt_prog = i40e_get_global_fd_count(pf);
fcnt_avail = pf->fdir_pf_filter_count;
if ((fcnt_prog < (fcnt_avail - I40E_FDIR_BUFFER_HEAD_ROOM)) ||
(pf->fd_add_err == 0) ||
(i40e_get_current_atr_cnt(pf) < pf->fd_atr_cnt)) {
- if ((pf->flags & I40E_FLAG_FD_SB_ENABLED) &&
- (pf->auto_disable_flags & I40E_FLAG_FD_SB_ENABLED)) {
- pf->auto_disable_flags &= ~I40E_FLAG_FD_SB_ENABLED;
- if (I40E_DEBUG_FD & pf->hw.debug_mask)
+ if (pf->flags & I40E_FLAG_FD_SB_AUTO_DISABLED) {
+ pf->flags &= ~I40E_FLAG_FD_SB_AUTO_DISABLED;
+ if ((pf->flags & I40E_FLAG_FD_SB_ENABLED) &&
+ (I40E_DEBUG_FD & pf->hw.debug_mask))
dev_info(&pf->pdev->dev, "FD Sideband/ntuple is being enabled since we have space in the table now\n");
}
}
- /* Wait for some more space to be available to turn on ATR. We also
- * must check that no existing ntuple rules for TCP are in effect
+ /* We should wait for even more space before re-enabling ATR.
+ * Additionally, we cannot enable ATR as long as we still have TCP SB
+ * rules active.
*/
- if (fcnt_prog < (fcnt_avail - I40E_FDIR_BUFFER_HEAD_ROOM * 2)) {
- if ((pf->flags & I40E_FLAG_FD_ATR_ENABLED) &&
- (pf->auto_disable_flags & I40E_FLAG_FD_ATR_ENABLED) &&
- (pf->fd_tcp_rule == 0)) {
- pf->auto_disable_flags &= ~I40E_FLAG_FD_ATR_ENABLED;
- if (I40E_DEBUG_FD & pf->hw.debug_mask)
+ if ((fcnt_prog < (fcnt_avail - I40E_FDIR_BUFFER_HEAD_ROOM_FOR_ATR)) &&
+ (pf->fd_tcp4_filter_cnt == 0)) {
+ if (pf->flags & I40E_FLAG_FD_ATR_AUTO_DISABLED) {
+ pf->flags &= ~I40E_FLAG_FD_ATR_AUTO_DISABLED;
+ if ((pf->flags & I40E_FLAG_FD_ATR_ENABLED) &&
+ (I40E_DEBUG_FD & pf->hw.debug_mask))
dev_info(&pf->pdev->dev, "ATR is being enabled since we have space in the table and there are no conflicting ntuple rules\n");
}
}
@@ -6218,7 +6152,7 @@
}
pf->fd_flush_timestamp = jiffies;
- pf->auto_disable_flags |= I40E_FLAG_FD_ATR_ENABLED;
+ pf->flags |= I40E_FLAG_FD_ATR_AUTO_DISABLED;
/* flush all filters */
wr32(&pf->hw, I40E_PFQF_CTL_1,
I40E_PFQF_CTL_1_CLEARFDTABLE_MASK);
@@ -6237,9 +6171,9 @@
} else {
/* replay sideband filters */
i40e_fdir_filter_restore(pf->vsi[pf->lan_vsi]);
- if (!disable_atr)
- pf->auto_disable_flags &= ~I40E_FLAG_FD_ATR_ENABLED;
- clear_bit(__I40E_FD_FLUSH_REQUESTED, &pf->state);
+ if (!disable_atr && !pf->fd_tcp4_filter_cnt)
+ pf->flags &= ~I40E_FLAG_FD_ATR_AUTO_DISABLED;
+ clear_bit(__I40E_FD_FLUSH_REQUESTED, pf->state);
if (I40E_DEBUG_FD & pf->hw.debug_mask)
dev_info(&pf->pdev->dev, "FD Filter table flushed and FD-SB replayed.\n");
}
@@ -6269,10 +6203,10 @@
{
/* if interface is down do nothing */
- if (test_bit(__I40E_DOWN, &pf->state))
+ if (test_bit(__I40E_VSI_DOWN, pf->state))
return;
- if (test_bit(__I40E_FD_FLUSH_REQUESTED, &pf->state))
+ if (test_bit(__I40E_FD_FLUSH_REQUESTED, pf->state))
i40e_fdir_flush_and_replay(pf);
i40e_fdir_check_and_reenable(pf);
@@ -6286,14 +6220,11 @@
**/
static void i40e_vsi_link_event(struct i40e_vsi *vsi, bool link_up)
{
- if (!vsi || test_bit(__I40E_DOWN, &vsi->state))
+ if (!vsi || test_bit(__I40E_VSI_DOWN, vsi->state))
return;
switch (vsi->type) {
case I40E_VSI_MAIN:
-#ifdef I40E_FCOE
- case I40E_VSI_FCOE:
-#endif
if (!vsi->netdev || !vsi->netdev_registered)
break;
@@ -6382,11 +6313,11 @@
if (new_link == old_link &&
new_link_speed == old_link_speed &&
- (test_bit(__I40E_DOWN, &vsi->state) ||
+ (test_bit(__I40E_VSI_DOWN, vsi->state) ||
new_link == netif_carrier_ok(vsi->netdev)))
return;
- if (!test_bit(__I40E_DOWN, &vsi->state))
+ if (!test_bit(__I40E_VSI_DOWN, vsi->state))
i40e_print_link_message(vsi, new_link);
/* Notify the base of the switch tree connected to
@@ -6413,8 +6344,8 @@
int i;
/* if interface is down do nothing */
- if (test_bit(__I40E_DOWN, &pf->state) ||
- test_bit(__I40E_CONFIG_BUSY, &pf->state))
+ if (test_bit(__I40E_VSI_DOWN, pf->state) ||
+ test_bit(__I40E_CONFIG_BUSY, pf->state))
return;
/* make sure we don't do these things too often */
@@ -6452,44 +6383,44 @@
{
u32 reset_flags = 0;
- rtnl_lock();
- if (test_bit(__I40E_REINIT_REQUESTED, &pf->state)) {
+ if (test_bit(__I40E_REINIT_REQUESTED, pf->state)) {
reset_flags |= BIT(__I40E_REINIT_REQUESTED);
- clear_bit(__I40E_REINIT_REQUESTED, &pf->state);
+ clear_bit(__I40E_REINIT_REQUESTED, pf->state);
}
- if (test_bit(__I40E_PF_RESET_REQUESTED, &pf->state)) {
+ if (test_bit(__I40E_PF_RESET_REQUESTED, pf->state)) {
reset_flags |= BIT(__I40E_PF_RESET_REQUESTED);
- clear_bit(__I40E_PF_RESET_REQUESTED, &pf->state);
+ clear_bit(__I40E_PF_RESET_REQUESTED, pf->state);
}
- if (test_bit(__I40E_CORE_RESET_REQUESTED, &pf->state)) {
+ if (test_bit(__I40E_CORE_RESET_REQUESTED, pf->state)) {
reset_flags |= BIT(__I40E_CORE_RESET_REQUESTED);
- clear_bit(__I40E_CORE_RESET_REQUESTED, &pf->state);
+ clear_bit(__I40E_CORE_RESET_REQUESTED, pf->state);
}
- if (test_bit(__I40E_GLOBAL_RESET_REQUESTED, &pf->state)) {
+ if (test_bit(__I40E_GLOBAL_RESET_REQUESTED, pf->state)) {
reset_flags |= BIT(__I40E_GLOBAL_RESET_REQUESTED);
- clear_bit(__I40E_GLOBAL_RESET_REQUESTED, &pf->state);
+ clear_bit(__I40E_GLOBAL_RESET_REQUESTED, pf->state);
}
- if (test_bit(__I40E_DOWN_REQUESTED, &pf->state)) {
- reset_flags |= BIT(__I40E_DOWN_REQUESTED);
- clear_bit(__I40E_DOWN_REQUESTED, &pf->state);
+ if (test_bit(__I40E_VSI_DOWN_REQUESTED, pf->state)) {
+ reset_flags |= BIT(__I40E_VSI_DOWN_REQUESTED);
+ clear_bit(__I40E_VSI_DOWN_REQUESTED, pf->state);
}
/* If there's a recovery already waiting, it takes
* precedence before starting a new reset sequence.
*/
- if (test_bit(__I40E_RESET_INTR_RECEIVED, &pf->state)) {
- i40e_handle_reset_warning(pf);
- goto unlock;
+ if (test_bit(__I40E_RESET_INTR_RECEIVED, pf->state)) {
+ i40e_prep_for_reset(pf, false);
+ i40e_reset(pf);
+ i40e_rebuild(pf, false, false);
}
/* If we're already down or resetting, just bail */
if (reset_flags &&
- !test_bit(__I40E_DOWN, &pf->state) &&
- !test_bit(__I40E_CONFIG_BUSY, &pf->state))
- i40e_do_reset(pf, reset_flags);
-
-unlock:
- rtnl_unlock();
+ !test_bit(__I40E_VSI_DOWN, pf->state) &&
+ !test_bit(__I40E_CONFIG_BUSY, pf->state)) {
+ rtnl_lock();
+ i40e_do_reset(pf, reset_flags, true);
+ rtnl_unlock();
+ }
}
/**
@@ -6534,7 +6465,7 @@
u32 val;
/* Do not run clean AQ when PF reset fails */
- if (test_bit(__I40E_RESET_FAILED, &pf->state))
+ if (test_bit(__I40E_RESET_FAILED, pf->state))
return;
/* check for error indications */
@@ -6635,9 +6566,11 @@
opcode);
break;
}
- } while (pending && (i++ < pf->adminq_work_limit));
+ } while (i++ < pf->adminq_work_limit);
- clear_bit(__I40E_ADMINQ_EVENT_PENDING, &pf->state);
+ if (i < pf->adminq_work_limit)
+ clear_bit(__I40E_ADMINQ_EVENT_PENDING, pf->state);
+
/* re-enable Admin queue interrupt cause */
val = rd32(hw, I40E_PFINT_ICR0_ENA);
val |= I40E_PFINT_ICR0_ENA_ADMINQ_MASK;
@@ -6662,13 +6595,13 @@
if (err) {
dev_info(&pf->pdev->dev, "eeprom check failed (%d), Tx/Rx traffic disabled\n",
err);
- set_bit(__I40E_BAD_EEPROM, &pf->state);
+ set_bit(__I40E_BAD_EEPROM, pf->state);
}
}
- if (!err && test_bit(__I40E_BAD_EEPROM, &pf->state)) {
+ if (!err && test_bit(__I40E_BAD_EEPROM, pf->state)) {
dev_info(&pf->pdev->dev, "eeprom check passed, Tx/Rx traffic enabled\n");
- clear_bit(__I40E_BAD_EEPROM, &pf->state);
+ clear_bit(__I40E_BAD_EEPROM, pf->state);
}
}
@@ -6975,17 +6908,19 @@
/**
* i40e_prep_for_reset - prep for the core to reset
* @pf: board private structure
+ * @lock_acquired: indicates whether or not the lock has been acquired
+ * before this function was called.
*
* Close up the VFs and other things in prep for PF Reset.
**/
-static void i40e_prep_for_reset(struct i40e_pf *pf)
+static void i40e_prep_for_reset(struct i40e_pf *pf, bool lock_acquired)
{
struct i40e_hw *hw = &pf->hw;
i40e_status ret = 0;
u32 v;
- clear_bit(__I40E_RESET_INTR_RECEIVED, &pf->state);
- if (test_and_set_bit(__I40E_RESET_RECOVERY_PENDING, &pf->state))
+ clear_bit(__I40E_RESET_INTR_RECEIVED, pf->state);
+ if (test_and_set_bit(__I40E_RESET_RECOVERY_PENDING, pf->state))
return;
if (i40e_check_asq_alive(&pf->hw))
i40e_vc_notify_reset(pf);
@@ -6993,7 +6928,12 @@
dev_dbg(&pf->pdev->dev, "Tearing down internal switch for reset\n");
/* quiesce the VSIs and their queues that are not already DOWN */
+ /* pf_quiesce_all_vsi modifies netdev structures -rtnl_lock needed */
+ if (!lock_acquired)
+ rtnl_lock();
i40e_pf_quiesce_all_vsi(pf);
+ if (!lock_acquired)
+ rtnl_unlock();
for (v = 0; v < pf->num_alloc_vsi; v++) {
if (pf->vsi[v])
@@ -7028,31 +6968,41 @@
}
/**
- * i40e_reset_and_rebuild - reset and rebuild using a saved config
+ * i40e_reset - wait for core reset to finish reset, reset pf if corer not seen
+ * @pf: board private structure
+ **/
+static int i40e_reset(struct i40e_pf *pf)
+{
+ struct i40e_hw *hw = &pf->hw;
+ i40e_status ret;
+
+ ret = i40e_pf_reset(hw);
+ if (ret) {
+ dev_info(&pf->pdev->dev, "PF reset failed, %d\n", ret);
+ set_bit(__I40E_RESET_FAILED, pf->state);
+ clear_bit(__I40E_RESET_RECOVERY_PENDING, pf->state);
+ } else {
+ pf->pfr_count++;
+ }
+ return ret;
+}
+
+/**
+ * i40e_rebuild - rebuild using a saved config
* @pf: board private structure
* @reinit: if the Main VSI needs to re-initialized.
+ * @lock_acquired: indicates whether or not the lock has been acquired
+ * before this function was called.
**/
-static void i40e_reset_and_rebuild(struct i40e_pf *pf, bool reinit)
+static void i40e_rebuild(struct i40e_pf *pf, bool reinit, bool lock_acquired)
{
struct i40e_hw *hw = &pf->hw;
u8 set_fc_aq_fail = 0;
i40e_status ret;
u32 val;
- u32 v;
+ int v;
- /* Now we wait for GRST to settle out.
- * We don't have to delete the VEBs or VSIs from the hw switch
- * because the reset will make them disappear.
- */
- ret = i40e_pf_reset(hw);
- if (ret) {
- dev_info(&pf->pdev->dev, "PF reset failed, %d\n", ret);
- set_bit(__I40E_RESET_FAILED, &pf->state);
- goto clear_recovery;
- }
- pf->pfr_count++;
-
- if (test_bit(__I40E_DOWN, &pf->state))
+ if (test_bit(__I40E_VSI_DOWN, pf->state))
goto clear_recovery;
dev_dbg(&pf->pdev->dev, "Rebuilding internal switch\n");
@@ -7066,7 +7016,7 @@
}
/* re-verify the eeprom if we just had an EMP reset */
- if (test_and_clear_bit(__I40E_EMP_RESET_INTR_RECEIVED, &pf->state))
+ if (test_and_clear_bit(__I40E_EMP_RESET_INTR_RECEIVED, pf->state))
i40e_verify_eeprom(pf);
i40e_clear_pxe_mode(hw);
@@ -7075,8 +7025,7 @@
goto end_core_reset;
ret = i40e_init_lan_hmc(hw, hw->func_caps.num_tx_qp,
- hw->func_caps.num_rx_qp,
- pf->fcoe_hmc_cntx_num, pf->fcoe_hmc_filt_num);
+ hw->func_caps.num_rx_qp, 0, 0);
if (ret) {
dev_info(&pf->pdev->dev, "init_lan_hmc failed: %d\n", ret);
goto end_core_reset;
@@ -7095,14 +7044,12 @@
/* Continue without DCB enabled */
}
#endif /* CONFIG_I40E_DCB */
-#ifdef I40E_FCOE
- i40e_init_pf_fcoe(pf);
-
-#endif
/* do basic switch setup */
+ if (!lock_acquired)
+ rtnl_lock();
ret = i40e_setup_pf_switch(pf, reinit);
if (ret)
- goto end_core_reset;
+ goto end_unlock;
/* The driver only wants link up/down and module qualification
* reports from firmware. Note the negative logic.
@@ -7173,7 +7120,7 @@
if (ret) {
dev_info(&pf->pdev->dev,
"rebuild of Main VSI failed: %d\n", ret);
- goto end_core_reset;
+ goto end_unlock;
}
}
@@ -7216,18 +7163,45 @@
/* restart the VSIs that were rebuilt and running before the reset */
i40e_pf_unquiesce_all_vsi(pf);
- if (pf->num_alloc_vfs) {
- for (v = 0; v < pf->num_alloc_vfs; v++)
- i40e_reset_vf(&pf->vf[v], true);
- }
+ /* Release the RTNL lock before we start resetting VFs */
+ if (!lock_acquired)
+ rtnl_unlock();
+
+ i40e_reset_all_vfs(pf, true);
/* tell the firmware that we're starting */
i40e_send_version(pf);
+ /* We've already released the lock, so don't do it again */
+ goto end_core_reset;
+
+end_unlock:
+ if (!lock_acquired)
+ rtnl_unlock();
end_core_reset:
- clear_bit(__I40E_RESET_FAILED, &pf->state);
+ clear_bit(__I40E_RESET_FAILED, pf->state);
clear_recovery:
- clear_bit(__I40E_RESET_RECOVERY_PENDING, &pf->state);
+ clear_bit(__I40E_RESET_RECOVERY_PENDING, pf->state);
+}
+
+/**
+ * i40e_reset_and_rebuild - reset and rebuild using a saved config
+ * @pf: board private structure
+ * @reinit: if the Main VSI needs to re-initialized.
+ * @lock_acquired: indicates whether or not the lock has been acquired
+ * before this function was called.
+ **/
+static void i40e_reset_and_rebuild(struct i40e_pf *pf, bool reinit,
+ bool lock_acquired)
+{
+ int ret;
+ /* Now we wait for GRST to settle out.
+ * We don't have to delete the VEBs or VSIs from the hw switch
+ * because the reset will make them disappear.
+ */
+ ret = i40e_reset(pf);
+ if (!ret)
+ i40e_rebuild(pf, reinit, lock_acquired);
}
/**
@@ -7236,11 +7210,13 @@
*
* Close up the VFs and other things in prep for a Core Reset,
* then get ready to rebuild the world.
+ * @lock_acquired: indicates whether or not the lock has been acquired
+ * before this function was called.
**/
-static void i40e_handle_reset_warning(struct i40e_pf *pf)
+static void i40e_handle_reset_warning(struct i40e_pf *pf, bool lock_acquired)
{
- i40e_prep_for_reset(pf);
- i40e_reset_and_rebuild(pf, false);
+ i40e_prep_for_reset(pf, lock_acquired);
+ i40e_reset_and_rebuild(pf, false, lock_acquired);
}
/**
@@ -7258,7 +7234,7 @@
u32 reg;
int i;
- if (!test_bit(__I40E_MDD_EVENT_PENDING, &pf->state))
+ if (!test_bit(__I40E_MDD_EVENT_PENDING, pf->state))
return;
/* find what triggered the MDD event */
@@ -7310,7 +7286,7 @@
}
/* Queue belongs to the PF, initiate a reset */
if (pf_mdd_detected) {
- set_bit(__I40E_PF_RESET_REQUESTED, &pf->state);
+ set_bit(__I40E_PF_RESET_REQUESTED, pf->state);
i40e_service_event_schedule(pf);
}
}
@@ -7339,12 +7315,12 @@
"Too many MDD events on VF %d, disabled\n", i);
dev_info(&pf->pdev->dev,
"Use PF Control I/F to re-enable the VF\n");
- set_bit(I40E_VF_STAT_DISABLED, &vf->vf_states);
+ set_bit(I40E_VF_STATE_DISABLED, &vf->vf_states);
}
}
/* re-enable mdd interrupt cause */
- clear_bit(__I40E_MDD_EVENT_PENDING, &pf->state);
+ clear_bit(__I40E_MDD_EVENT_PENDING, pf->state);
reg = rd32(hw, I40E_PFINT_ICR0_ENA);
reg |= I40E_PFINT_ICR0_ENA_MAL_DETECT_MASK;
wr32(hw, I40E_PFINT_ICR0_ENA, reg);
@@ -7352,6 +7328,23 @@
}
/**
+ * i40e_sync_udp_filters - Trigger a sync event for existing UDP filters
+ * @pf: board private structure
+ **/
+static void i40e_sync_udp_filters(struct i40e_pf *pf)
+{
+ int i;
+
+ /* loop through and set pending bit for all active UDP filters */
+ for (i = 0; i < I40E_MAX_PF_UDP_OFFLOAD_PORTS; i++) {
+ if (pf->udp_ports[i].port)
+ pf->pending_udp_bitmap |= BIT_ULL(i);
+ }
+
+ pf->flags |= I40E_FLAG_UDP_FILTER_SYNC;
+}
+
+/**
* i40e_sync_udp_filters_subtask - Sync the VSI filter list with HW
* @pf: board private structure
**/
@@ -7359,7 +7352,7 @@
{
struct i40e_hw *hw = &pf->hw;
i40e_status ret;
- __be16 port;
+ u16 port;
int i;
if (!(pf->flags & I40E_FLAG_UDP_FILTER_SYNC))
@@ -7370,7 +7363,7 @@
for (i = 0; i < I40E_MAX_PF_UDP_OFFLOAD_PORTS; i++) {
if (pf->pending_udp_bitmap & BIT_ULL(i)) {
pf->pending_udp_bitmap &= ~BIT_ULL(i);
- port = pf->udp_ports[i].index;
+ port = pf->udp_ports[i].port;
if (port)
ret = i40e_aq_add_udp_tunnel(hw, port,
pf->udp_ports[i].type,
@@ -7383,11 +7376,11 @@
"%s %s port %d, index %d failed, err %s aq_err %s\n",
pf->udp_ports[i].type ? "vxlan" : "geneve",
port ? "add" : "delete",
- ntohs(port), i,
+ port, i,
i40e_stat_str(&pf->hw, ret),
i40e_aq_str(&pf->hw,
pf->hw.aq.asq_last_status));
- pf->udp_ports[i].index = 0;
+ pf->udp_ports[i].port = 0;
}
}
}
@@ -7405,11 +7398,10 @@
unsigned long start_time = jiffies;
/* don't bother with service tasks if a reset is in progress */
- if (test_bit(__I40E_RESET_RECOVERY_PENDING, &pf->state)) {
+ if (test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state))
return;
- }
- if (test_and_set_bit(__I40E_SERVICE_SCHED, &pf->state))
+ if (test_and_set_bit(__I40E_SERVICE_SCHED, pf->state))
return;
i40e_detect_recover_hung(pf);
@@ -7419,23 +7411,34 @@
i40e_vc_process_vflr_event(pf);
i40e_watchdog_subtask(pf);
i40e_fdir_reinit_subtask(pf);
- i40e_client_subtask(pf);
+ if (pf->flags & I40E_FLAG_CLIENT_RESET) {
+ /* Client subtask will reopen next time through. */
+ i40e_notify_client_of_netdev_close(pf->vsi[pf->lan_vsi], true);
+ pf->flags &= ~I40E_FLAG_CLIENT_RESET;
+ } else {
+ i40e_client_subtask(pf);
+ if (pf->flags & I40E_FLAG_CLIENT_L2_CHANGE) {
+ i40e_notify_client_of_l2_param_changes(
+ pf->vsi[pf->lan_vsi]);
+ pf->flags &= ~I40E_FLAG_CLIENT_L2_CHANGE;
+ }
+ }
i40e_sync_filters_subtask(pf);
i40e_sync_udp_filters_subtask(pf);
i40e_clean_adminq_subtask(pf);
/* flush memory to make sure state is correct before next watchdog */
smp_mb__before_atomic();
- clear_bit(__I40E_SERVICE_SCHED, &pf->state);
+ clear_bit(__I40E_SERVICE_SCHED, pf->state);
/* If the tasks have taken longer than one timer cycle or there
* is more work to be done, reschedule the service task now
* rather than wait for the timer to tick again.
*/
if (time_after(jiffies, (start_time + pf->service_timer_period)) ||
- test_bit(__I40E_ADMINQ_EVENT_PENDING, &pf->state) ||
- test_bit(__I40E_MDD_EVENT_PENDING, &pf->state) ||
- test_bit(__I40E_VFLR_EVENT_PENDING, &pf->state))
+ test_bit(__I40E_ADMINQ_EVENT_PENDING, pf->state) ||
+ test_bit(__I40E_MDD_EVENT_PENDING, pf->state) ||
+ test_bit(__I40E_VFLR_EVENT_PENDING, pf->state))
i40e_service_event_schedule(pf);
}
@@ -7492,15 +7495,6 @@
I40E_REQ_DESCRIPTOR_MULTIPLE);
break;
-#ifdef I40E_FCOE
- case I40E_VSI_FCOE:
- vsi->alloc_queue_pairs = pf->num_fcoe_qps;
- vsi->num_desc = ALIGN(I40E_DEFAULT_NUM_DESCRIPTORS,
- I40E_REQ_DESCRIPTOR_MULTIPLE);
- vsi->num_q_vectors = pf->num_fcoe_msix;
- break;
-
-#endif /* I40E_FCOE */
default:
WARN_ON(1);
return -ENODATA;
@@ -7593,7 +7587,7 @@
}
vsi->type = type;
vsi->back = pf;
- set_bit(__I40E_DOWN, &vsi->state);
+ set_bit(__I40E_VSI_DOWN, vsi->state);
vsi->flags = 0;
vsi->idx = vsi_idx;
vsi->int_rate_limit = 0;
@@ -7817,6 +7811,7 @@
static int i40e_init_msix(struct i40e_pf *pf)
{
struct i40e_hw *hw = &pf->hw;
+ int cpus, extra_vectors;
int vectors_left;
int v_budget, i;
int v_actual;
@@ -7835,9 +7830,6 @@
* - assumes symmetric Tx/Rx pairing
* - The number of VMDq pairs
* - The CPU count within the NUMA node if iWARP is enabled
-#ifdef I40E_FCOE
- * - The number of FCOE qps.
-#endif
* Once we count this up, try the request.
*
* If we can't get what we want, we'll simplify to nearly nothing
@@ -7852,10 +7844,16 @@
vectors_left--;
}
- /* reserve vectors for the main PF traffic queues */
- pf->num_lan_msix = min_t(int, num_online_cpus(), vectors_left);
+ /* reserve some vectors for the main PF traffic queues. Initially we
+ * only reserve at most 50% of the available vectors, in the case that
+ * the number of online CPUs is large. This ensures that we can enable
+ * extra features as well. Once we've enabled the other features, we
+ * will use any remaining vectors to reach as close as we can to the
+ * number of online CPUs.
+ */
+ cpus = num_online_cpus();
+ pf->num_lan_msix = min_t(int, cpus, vectors_left / 2);
vectors_left -= pf->num_lan_msix;
- v_budget += pf->num_lan_msix;
/* reserve one vector for sideband flow director */
if (pf->flags & I40E_FLAG_FD_SB_ENABLED) {
@@ -7868,20 +7866,6 @@
}
}
-#ifdef I40E_FCOE
- /* can we reserve enough for FCoE? */
- if (pf->flags & I40E_FLAG_FCOE_ENABLED) {
- if (!vectors_left)
- pf->num_fcoe_msix = 0;
- else if (vectors_left >= pf->num_fcoe_qps)
- pf->num_fcoe_msix = pf->num_fcoe_qps;
- else
- pf->num_fcoe_msix = 1;
- v_budget += pf->num_fcoe_msix;
- vectors_left -= pf->num_fcoe_msix;
- }
-
-#endif
/* can we reserve enough for iWARP? */
if (pf->flags & I40E_FLAG_IWARP_ENABLED) {
iwarp_requested = pf->num_iwarp_msix;
@@ -7918,6 +7902,23 @@
}
}
+ /* On systems with a large number of SMP cores, we previously limited
+ * the number of vectors for num_lan_msix to be at most 50% of the
+ * available vectors, to allow for other features. Now, we add back
+ * the remaining vectors. However, we ensure that the total
+ * num_lan_msix will not exceed num_online_cpus(). To do this, we
+ * calculate the number of vectors we can add without going over the
+ * cap of CPUs. For systems with a small number of CPUs this will be
+ * zero.
+ */
+ extra_vectors = min_t(int, cpus - pf->num_lan_msix, vectors_left);
+ pf->num_lan_msix += extra_vectors;
+ vectors_left -= extra_vectors;
+
+ WARN(vectors_left < 0,
+ "Calculation of remaining vectors underflowed. This is an accounting bug when determining total MSI-X vectors.\n");
+
+ v_budget += pf->num_lan_msix;
pf->msix_entries = kcalloc(v_budget, sizeof(struct msix_entry),
GFP_KERNEL);
if (!pf->msix_entries)
@@ -7958,10 +7959,6 @@
pf->num_vmdq_msix = 1; /* force VMDqs to only one vector */
pf->num_vmdq_vsis = 1;
pf->num_vmdq_qps = 1;
-#ifdef I40E_FCOE
- pf->num_fcoe_qps = 0;
- pf->num_fcoe_msix = 0;
-#endif
/* partition out the remaining vectors */
switch (vec) {
@@ -7975,13 +7972,6 @@
} else {
pf->num_lan_msix = 2;
}
-#ifdef I40E_FCOE
- /* give one vector to FCoE */
- if (pf->flags & I40E_FLAG_FCOE_ENABLED) {
- pf->num_lan_msix = 1;
- pf->num_fcoe_msix = 1;
- }
-#endif
break;
default:
if (pf->flags & I40E_FLAG_IWARP_ENABLED) {
@@ -8001,13 +7991,6 @@
(vec - (pf->num_iwarp_msix + pf->num_vmdq_vsis)),
pf->num_lan_msix);
pf->num_lan_qps = pf->num_lan_msix;
-#ifdef I40E_FCOE
- /* give one vector to FCoE */
- if (pf->flags & I40E_FLAG_FCOE_ENABLED) {
- pf->num_fcoe_msix = 1;
- vec--;
- }
-#endif
break;
}
}
@@ -8028,13 +8011,6 @@
dev_info(&pf->pdev->dev, "IWARP disabled, not enough MSI-X vectors\n");
pf->flags &= ~I40E_FLAG_IWARP_ENABLED;
}
-#ifdef I40E_FCOE
-
- if ((pf->flags & I40E_FLAG_FCOE_ENABLED) && (pf->num_fcoe_msix == 0)) {
- dev_info(&pf->pdev->dev, "FCOE disabled, not enough MSI-X vectors\n");
- pf->flags &= ~I40E_FLAG_FCOE_ENABLED;
- }
-#endif
i40e_debug(&pf->hw, I40E_DEBUG_INIT,
"MSI-X vector distribution: PF %d, VMDq %d, FDSB %d, iWARP %d\n",
pf->num_lan_msix,
@@ -8133,9 +8109,6 @@
if (vectors < 0) {
pf->flags &= ~(I40E_FLAG_MSIX_ENABLED |
I40E_FLAG_IWARP_ENABLED |
-#ifdef I40E_FCOE
- I40E_FLAG_FCOE_ENABLED |
-#endif
I40E_FLAG_RSS_ENABLED |
I40E_FLAG_DCB_CAPABLE |
I40E_FLAG_DCB_ENABLED |
@@ -8196,7 +8169,7 @@
/* Only request the irq if this is the first time through, and
* not when we're rebuilding after a Reset
*/
- if (!test_bit(__I40E_RESET_RECOVERY_PENDING, &pf->state)) {
+ if (!test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state)) {
err = request_irq(pf->msix_entries[0].vector,
i40e_intr, 0, pf->int_name, pf);
if (err) {
@@ -8368,13 +8341,10 @@
if (vsi->type == I40E_VSI_MAIN) {
for (i = 0; i <= I40E_PFQF_HKEY_MAX_INDEX; i++)
- i40e_write_rx_ctl(hw, I40E_PFQF_HKEY(i),
- seed_dw[i]);
+ wr32(hw, I40E_PFQF_HKEY(i), seed_dw[i]);
} else if (vsi->type == I40E_VSI_SRIOV) {
for (i = 0; i <= I40E_VFQF_HKEY1_MAX_INDEX; i++)
- i40e_write_rx_ctl(hw,
- I40E_VFQF_HKEY1(i, vf_id),
- seed_dw[i]);
+ wr32(hw, I40E_VFQF_HKEY1(i, vf_id), seed_dw[i]);
} else {
dev_err(&pf->pdev->dev, "Cannot set RSS seed - invalid VSI type\n");
}
@@ -8392,9 +8362,7 @@
if (lut_size != I40E_VF_HLUT_ARRAY_SIZE)
return -EINVAL;
for (i = 0; i <= I40E_VFQF_HLUT_MAX_INDEX; i++)
- i40e_write_rx_ctl(hw,
- I40E_VFQF_HLUT1(i, vf_id),
- lut_dw[i]);
+ wr32(hw, I40E_VFQF_HLUT1(i, vf_id), lut_dw[i]);
} else {
dev_err(&pf->pdev->dev, "Cannot set RSS LUT - invalid VSI type\n");
}
@@ -8522,9 +8490,12 @@
i40e_write_rx_ctl(hw, I40E_PFQF_CTL_0, reg_val);
/* Determine the RSS size of the VSI */
- if (!vsi->rss_size)
- vsi->rss_size = min_t(int, pf->alloc_rss_size,
- vsi->num_queue_pairs);
+ if (!vsi->rss_size) {
+ u16 qcount;
+
+ qcount = vsi->num_queue_pairs / vsi->tc_config.numtc;
+ vsi->rss_size = min_t(int, pf->alloc_rss_size, qcount);
+ }
if (!vsi->rss_size)
return -EINVAL;
@@ -8558,6 +8529,7 @@
*
* returns 0 if rss is not enabled, if enabled returns the final rss queue
* count which may be different from the requested queue count.
+ * Note: expects to be called while under rtnl_lock()
**/
int i40e_reconfig_rss_queues(struct i40e_pf *pf, int queue_count)
{
@@ -8570,12 +8542,14 @@
new_rss_size = min_t(int, queue_count, pf->rss_size_max);
if (queue_count != vsi->num_queue_pairs) {
+ u16 qcount;
+
vsi->req_queue_pairs = queue_count;
- i40e_prep_for_reset(pf);
+ i40e_prep_for_reset(pf, true);
pf->alloc_rss_size = new_rss_size;
- i40e_reset_and_rebuild(pf, true);
+ i40e_reset_and_rebuild(pf, true, true);
/* Discard the user configured hash keys and lut, if less
* queues are enabled.
@@ -8587,8 +8561,8 @@
}
/* Reset vsi->rss_size, as number of enabled queues changed */
- vsi->rss_size = min_t(int, pf->alloc_rss_size,
- vsi->num_queue_pairs);
+ qcount = vsi->num_queue_pairs / vsi->tc_config.numtc;
+ vsi->rss_size = min_t(int, pf->alloc_rss_size, qcount);
i40e_pf_config_rss(pf);
}
@@ -8821,10 +8795,6 @@
pf->num_iwarp_msix = (int)num_online_cpus() + 1;
}
-#ifdef I40E_FCOE
- i40e_init_pf_fcoe(pf);
-
-#endif /* I40E_FCOE */
#ifdef CONFIG_PCI_IOV
if (pf->hw.func_caps.num_vfs && pf->hw.partition_id == 1) {
pf->num_vf_qps = I40E_DEFAULT_QUEUES_PER_VF;
@@ -8851,9 +8821,9 @@
(pf->hw.aq.api_min_ver > 4))) {
/* Supported in FW API version higher than 1.4 */
pf->flags |= I40E_FLAG_GENEVE_OFFLOAD_CAPABLE;
- pf->auto_disable_flags = I40E_FLAG_HW_ATR_EVICT_CAPABLE;
+ pf->flags = I40E_FLAG_HW_ATR_EVICT_CAPABLE;
} else {
- pf->auto_disable_flags = I40E_FLAG_HW_ATR_EVICT_CAPABLE;
+ pf->flags = I40E_FLAG_HW_ATR_EVICT_CAPABLE;
}
pf->eeprom_version = 0xDEAD;
@@ -8913,16 +8883,16 @@
need_reset = true;
i40e_fdir_filter_exit(pf);
}
- pf->flags &= ~I40E_FLAG_FD_SB_ENABLED;
- pf->auto_disable_flags &= ~I40E_FLAG_FD_SB_ENABLED;
+ pf->flags &= ~(I40E_FLAG_FD_SB_ENABLED |
+ I40E_FLAG_FD_SB_AUTO_DISABLED);
/* reset fd counters */
- pf->fd_add_err = pf->fd_atr_cnt = pf->fd_tcp_rule = 0;
- pf->fdir_pf_active_filters = 0;
+ pf->fd_add_err = 0;
+ pf->fd_atr_cnt = 0;
/* if ATR was auto disabled it can be re-enabled. */
- if ((pf->flags & I40E_FLAG_FD_ATR_ENABLED) &&
- (pf->auto_disable_flags & I40E_FLAG_FD_ATR_ENABLED)) {
- pf->auto_disable_flags &= ~I40E_FLAG_FD_ATR_ENABLED;
- if (I40E_DEBUG_FD & pf->hw.debug_mask)
+ if (pf->flags & I40E_FLAG_FD_ATR_AUTO_DISABLED) {
+ pf->flags &= ~I40E_FLAG_FD_ATR_AUTO_DISABLED;
+ if ((pf->flags & I40E_FLAG_FD_ATR_ENABLED) &&
+ (I40E_DEBUG_FD & pf->hw.debug_mask))
dev_info(&pf->pdev->dev, "ATR re-enabled.\n");
}
}
@@ -8955,6 +8925,7 @@
* i40e_set_features - set the netdev feature flags
* @netdev: ptr to the netdev being adjusted
* @features: the feature set that the stack is suggesting
+ * Note: expects to be called while under rtnl_lock()
**/
static int i40e_set_features(struct net_device *netdev,
netdev_features_t features)
@@ -8978,7 +8949,7 @@
need_reset = i40e_set_ntuple(pf, features);
if (need_reset)
- i40e_do_reset(pf, BIT_ULL(__I40E_PF_RESET_REQUESTED));
+ i40e_do_reset(pf, BIT_ULL(__I40E_PF_RESET_REQUESTED), true);
return 0;
}
@@ -8990,12 +8961,12 @@
*
* Returns the index number or I40E_MAX_PF_UDP_OFFLOAD_PORTS if port not found
**/
-static u8 i40e_get_udp_port_idx(struct i40e_pf *pf, __be16 port)
+static u8 i40e_get_udp_port_idx(struct i40e_pf *pf, u16 port)
{
u8 i;
for (i = 0; i < I40E_MAX_PF_UDP_OFFLOAD_PORTS; i++) {
- if (pf->udp_ports[i].index == port)
+ if (pf->udp_ports[i].port == port)
return i;
}
@@ -9013,7 +8984,7 @@
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_vsi *vsi = np->vsi;
struct i40e_pf *pf = vsi->back;
- __be16 port = ti->port;
+ u16 port = ntohs(ti->port);
u8 next_idx;
u8 idx;
@@ -9021,8 +8992,7 @@
/* Check if port already exists */
if (idx < I40E_MAX_PF_UDP_OFFLOAD_PORTS) {
- netdev_info(netdev, "port %d already offloaded\n",
- ntohs(port));
+ netdev_info(netdev, "port %d already offloaded\n", port);
return;
}
@@ -9031,7 +9001,7 @@
if (next_idx == I40E_MAX_PF_UDP_OFFLOAD_PORTS) {
netdev_info(netdev, "maximum number of offloaded UDP ports reached, not adding port %d\n",
- ntohs(port));
+ port);
return;
}
@@ -9049,7 +9019,7 @@
}
/* New port: add it and mark its index in the bitmap */
- pf->udp_ports[next_idx].index = port;
+ pf->udp_ports[next_idx].port = port;
pf->pending_udp_bitmap |= BIT_ULL(next_idx);
pf->flags |= I40E_FLAG_UDP_FILTER_SYNC;
}
@@ -9065,7 +9035,7 @@
struct i40e_netdev_priv *np = netdev_priv(netdev);
struct i40e_vsi *vsi = np->vsi;
struct i40e_pf *pf = vsi->back;
- __be16 port = ti->port;
+ u16 port = ntohs(ti->port);
u8 idx;
idx = i40e_get_udp_port_idx(pf, port);
@@ -9090,14 +9060,14 @@
/* if port exists, set it to 0 (mark for deletion)
* and make it pending
*/
- pf->udp_ports[idx].index = 0;
+ pf->udp_ports[idx].port = 0;
pf->pending_udp_bitmap |= BIT_ULL(idx);
pf->flags |= I40E_FLAG_UDP_FILTER_SYNC;
return;
not_found:
netdev_warn(netdev, "UDP port %d was not found, not deleting\n",
- ntohs(port));
+ port);
}
static int i40e_get_phys_port_id(struct net_device *netdev,
@@ -9174,6 +9144,8 @@
* is to change the mode then that requires a PF reset to
* allow rebuild of the components with required hardware
* bridge mode enabled.
+ *
+ * Note: expects to be called while under rtnl_lock()
**/
static int i40e_ndo_bridge_setlink(struct net_device *dev,
struct nlmsghdr *nlh,
@@ -9229,7 +9201,8 @@
pf->flags |= I40E_FLAG_VEB_MODE_ENABLED;
else
pf->flags &= ~I40E_FLAG_VEB_MODE_ENABLED;
- i40e_do_reset(pf, BIT_ULL(__I40E_PF_RESET_REQUESTED));
+ i40e_do_reset(pf, BIT_ULL(__I40E_PF_RESET_REQUESTED),
+ true);
break;
}
}
@@ -9352,10 +9325,6 @@
.ndo_poll_controller = i40e_netpoll,
#endif
.ndo_setup_tc = __i40e_setup_tc,
-#ifdef I40E_FCOE
- .ndo_fcoe_enable = i40e_fcoe_enable,
- .ndo_fcoe_disable = i40e_fcoe_disable,
-#endif
.ndo_set_features = i40e_set_features,
.ndo_set_vf_mac = i40e_ndo_set_vf_mac,
.ndo_set_vf_vlan = i40e_ndo_set_vf_port_vlan,
@@ -9388,6 +9357,8 @@
u8 broadcast[ETH_ALEN];
u8 mac_addr[ETH_ALEN];
int etherdev_size;
+ netdev_features_t hw_enc_features;
+ netdev_features_t hw_features;
etherdev_size = sizeof(struct i40e_netdev_priv);
netdev = alloc_etherdev_mq(etherdev_size, vsi->alloc_queue_pairs);
@@ -9398,52 +9369,57 @@
np = netdev_priv(netdev);
np->vsi = vsi;
- netdev->hw_enc_features |= NETIF_F_SG |
- NETIF_F_IP_CSUM |
- NETIF_F_IPV6_CSUM |
- NETIF_F_HIGHDMA |
- NETIF_F_SOFT_FEATURES |
- NETIF_F_TSO |
- NETIF_F_TSO_ECN |
- NETIF_F_TSO6 |
- NETIF_F_GSO_GRE |
- NETIF_F_GSO_GRE_CSUM |
- NETIF_F_GSO_IPXIP4 |
- NETIF_F_GSO_IPXIP6 |
- NETIF_F_GSO_UDP_TUNNEL |
- NETIF_F_GSO_UDP_TUNNEL_CSUM |
- NETIF_F_GSO_PARTIAL |
- NETIF_F_SCTP_CRC |
- NETIF_F_RXHASH |
- NETIF_F_RXCSUM |
- 0;
+ hw_enc_features = NETIF_F_SG |
+ NETIF_F_IP_CSUM |
+ NETIF_F_IPV6_CSUM |
+ NETIF_F_HIGHDMA |
+ NETIF_F_SOFT_FEATURES |
+ NETIF_F_TSO |
+ NETIF_F_TSO_ECN |
+ NETIF_F_TSO6 |
+ NETIF_F_GSO_GRE |
+ NETIF_F_GSO_GRE_CSUM |
+ NETIF_F_GSO_PARTIAL |
+ NETIF_F_GSO_UDP_TUNNEL |
+ NETIF_F_GSO_UDP_TUNNEL_CSUM |
+ NETIF_F_SCTP_CRC |
+ NETIF_F_RXHASH |
+ NETIF_F_RXCSUM |
+ 0;
if (!(pf->flags & I40E_FLAG_OUTER_UDP_CSUM_CAPABLE))
netdev->gso_partial_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
netdev->gso_partial_features |= NETIF_F_GSO_GRE_CSUM;
+ netdev->hw_enc_features |= hw_enc_features;
+
/* record features VLANs can make use of */
- netdev->vlan_features |= netdev->hw_enc_features |
- NETIF_F_TSO_MANGLEID;
+ netdev->vlan_features |= hw_enc_features | NETIF_F_TSO_MANGLEID;
if (!(pf->flags & I40E_FLAG_MFP_ENABLED))
netdev->hw_features |= NETIF_F_NTUPLE;
+ hw_features = hw_enc_features |
+ NETIF_F_HW_VLAN_CTAG_TX |
+ NETIF_F_HW_VLAN_CTAG_RX;
- netdev->hw_features |= netdev->hw_enc_features |
- NETIF_F_HW_VLAN_CTAG_TX |
- NETIF_F_HW_VLAN_CTAG_RX;
+ netdev->hw_features |= hw_features;
- netdev->features |= netdev->hw_features | NETIF_F_HW_VLAN_CTAG_FILTER;
+ netdev->features |= hw_features | NETIF_F_HW_VLAN_CTAG_FILTER;
netdev->hw_enc_features |= NETIF_F_TSO_MANGLEID;
if (vsi->type == I40E_VSI_MAIN) {
SET_NETDEV_DEV(netdev, &pf->pdev->dev);
ether_addr_copy(mac_addr, hw->mac.perm_addr);
- /* The following steps are necessary to prevent reception
- * of tagged packets - some older NVM configurations load a
- * default a MAC-VLAN filter that accepts any tagged packet
- * which must be replaced by a normal filter.
+ /* The following steps are necessary for two reasons. First,
+ * some older NVM configurations load a default MAC-VLAN
+ * filter that will accept any tagged packet, and we want to
+ * replace this with a normal filter. Additionally, it is
+ * possible our MAC address was provided by the platform using
+ * Open Firmware or similar.
+ *
+ * Thus, we need to remove the default filter and install one
+ * specific to the MAC address.
*/
i40e_rm_default_mac_filter(vsi, mac_addr);
spin_lock_bh(&vsi->mac_filter_hash_lock);
@@ -9489,9 +9465,6 @@
netdev->netdev_ops = &i40e_netdev_ops;
netdev->watchdog_timeo = 5 * HZ;
i40e_set_ethtool_ops(netdev);
-#ifdef I40E_FCOE
- i40e_fcoe_config_netdev(netdev, vsi);
-#endif
/* MTU range: 68 - 9706 */
netdev->min_mtu = ETH_MIN_MTU;
@@ -9715,16 +9688,6 @@
i40e_vsi_setup_queue_map(vsi, &ctxt, enabled_tc, true);
break;
-#ifdef I40E_FCOE
- case I40E_VSI_FCOE:
- ret = i40e_fcoe_vsi_init(vsi, &ctxt);
- if (ret) {
- dev_info(&pf->pdev->dev, "failed to initialize FCoE VSI\n");
- return ret;
- }
- break;
-
-#endif /* I40E_FCOE */
case I40E_VSI_IWARP:
/* send down message to iWARP */
break;
@@ -9751,7 +9714,7 @@
}
vsi->active_filters = 0;
- clear_bit(__I40E_FILTER_OVERFLOW_PROMISC, &vsi->state);
+ clear_bit(__I40E_VSI_OVERFLOW_PROMISC, vsi->state);
spin_lock_bh(&vsi->mac_filter_hash_lock);
/* If macvlan filters already exist, force them to get loaded */
hash_for_each_safe(vsi->mac_filter_hash, bkt, h, f, hlist) {
@@ -9804,7 +9767,7 @@
return -ENODEV;
}
if (vsi == pf->vsi[pf->lan_vsi] &&
- !test_bit(__I40E_DOWN, &pf->state)) {
+ !test_bit(__I40E_VSI_DOWN, pf->state)) {
dev_info(&pf->pdev->dev, "Can't remove PF VSI\n");
return -ENODEV;
}
@@ -10141,7 +10104,6 @@
}
}
case I40E_VSI_VMDQ2:
- case I40E_VSI_FCOE:
ret = i40e_config_netdev(vsi);
if (ret)
goto err_netdev;
@@ -10789,6 +10751,9 @@
i40e_ptp_init(pf);
+ /* repopulate tunnel port filters */
+ i40e_sync_udp_filters(pf);
+
return ret;
}
@@ -10801,9 +10766,6 @@
int queues_left;
pf->num_lan_qps = 0;
-#ifdef I40E_FCOE
- pf->num_fcoe_qps = 0;
-#endif
/* Find the max queues to be put into basic use. We'll always be
* using TC0, whether or not DCB is running, and TC0 will get the
@@ -10820,9 +10782,6 @@
/* make sure all the fancies are disabled */
pf->flags &= ~(I40E_FLAG_RSS_ENABLED |
I40E_FLAG_IWARP_ENABLED |
-#ifdef I40E_FCOE
- I40E_FLAG_FCOE_ENABLED |
-#endif
I40E_FLAG_FD_SB_ENABLED |
I40E_FLAG_FD_ATR_ENABLED |
I40E_FLAG_DCB_CAPABLE |
@@ -10839,9 +10798,6 @@
pf->flags &= ~(I40E_FLAG_RSS_ENABLED |
I40E_FLAG_IWARP_ENABLED |
-#ifdef I40E_FCOE
- I40E_FLAG_FCOE_ENABLED |
-#endif
I40E_FLAG_FD_SB_ENABLED |
I40E_FLAG_FD_ATR_ENABLED |
I40E_FLAG_DCB_ENABLED |
@@ -10862,22 +10818,6 @@
queues_left -= pf->num_lan_qps;
}
-#ifdef I40E_FCOE
- if (pf->flags & I40E_FLAG_FCOE_ENABLED) {
- if (I40E_DEFAULT_FCOE <= queues_left) {
- pf->num_fcoe_qps = I40E_DEFAULT_FCOE;
- } else if (I40E_MINIMUM_FCOE <= queues_left) {
- pf->num_fcoe_qps = I40E_MINIMUM_FCOE;
- } else {
- pf->num_fcoe_qps = 0;
- pf->flags &= ~I40E_FLAG_FCOE_ENABLED;
- dev_info(&pf->pdev->dev, "not enough queues for FCoE. FCoE feature will be disabled\n");
- }
-
- queues_left -= pf->num_fcoe_qps;
- }
-
-#endif
if (pf->flags & I40E_FLAG_FD_SB_ENABLED) {
if (queues_left > 1) {
queues_left -= 1; /* save 1 queue for FD */
@@ -10909,9 +10849,6 @@
pf->num_lan_qps, pf->alloc_rss_size, pf->num_req_vfs,
pf->num_vf_qps, pf->num_vmdq_vsis, pf->num_vmdq_qps,
queues_left);
-#ifdef I40E_FCOE
- dev_dbg(&pf->pdev->dev, "fcoe queues = %d\n", pf->num_fcoe_qps);
-#endif
}
/**
@@ -10978,10 +10915,6 @@
i += snprintf(&buf[i], REMAIN(i), " Geneve");
if (pf->flags & I40E_FLAG_PTP)
i += snprintf(&buf[i], REMAIN(i), " PTP");
-#ifdef I40E_FCOE
- if (pf->flags & I40E_FLAG_FCOE_ENABLED)
- i += snprintf(&buf[i], REMAIN(i), " FCOE");
-#endif
if (pf->flags & I40E_FLAG_VEB_MODE_ENABLED)
i += snprintf(&buf[i], REMAIN(i), " VEB");
else
@@ -10994,20 +10927,18 @@
/**
* i40e_get_platform_mac_addr - get platform-specific MAC address
- *
* @pdev: PCI device information struct
* @pf: board private structure
*
- * Look up the MAC address in Open Firmware on systems that support it,
- * and use IDPROM on SPARC if no OF address is found. On return, the
- * I40E_FLAG_PF_MAC will be wset in pf->flags if a platform-specific value
- * has been selected.
+ * Look up the MAC address for the device. First we'll try
+ * eth_platform_get_mac_address, which will check Open Firmware, or arch
+ * specific fallback. Otherwise, we'll default to the stored value in
+ * firmware.
**/
static void i40e_get_platform_mac_addr(struct pci_dev *pdev, struct i40e_pf *pf)
{
- pf->flags &= ~I40E_FLAG_PF_MAC;
- if (!eth_platform_get_mac_address(&pdev->dev, pf->hw.mac.addr))
- pf->flags |= I40E_FLAG_PF_MAC;
+ if (eth_platform_get_mac_address(&pdev->dev, pf->hw.mac.addr))
+ i40e_get_mac_addr(&pf->hw, pf->hw.mac.addr);
}
/**
@@ -11072,7 +11003,7 @@
}
pf->next_vsi = 0;
pf->pdev = pdev;
- set_bit(__I40E_DOWN, &pf->state);
+ set_bit(__I40E_VSI_DOWN, pf->state);
hw = &pf->hw;
hw->back = pf;
@@ -11098,6 +11029,9 @@
hw->bus.bus_id = pdev->bus->number;
pf->instance = pfs_found;
+ INIT_LIST_HEAD(&pf->l3_flex_pit_list);
+ INIT_LIST_HEAD(&pf->l4_flex_pit_list);
+
/* set up the locks for the AQ, do this only once in probe
* and destroy them only once in remove
*/
@@ -11196,8 +11130,7 @@
}
err = i40e_init_lan_hmc(hw, hw->func_caps.num_tx_qp,
- hw->func_caps.num_rx_qp,
- pf->fcoe_hmc_cntx_num, pf->fcoe_hmc_filt_num);
+ hw->func_caps.num_rx_qp, 0, 0);
if (err) {
dev_info(&pdev->dev, "init_lan_hmc failed: %d\n", err);
goto err_init_lan_hmc;
@@ -11219,9 +11152,9 @@
i40e_aq_stop_lldp(hw, true, NULL);
}
- i40e_get_mac_addr(hw, hw->mac.addr);
/* allow a platform config to override the HW addr */
i40e_get_platform_mac_addr(pdev, pf);
+
if (!is_valid_ether_addr(hw->mac.addr)) {
dev_info(&pdev->dev, "invalid MAC address %pM\n", hw->mac.addr);
err = -EIO;
@@ -11232,18 +11165,6 @@
i40e_get_port_mac_addr(hw, hw->mac.port_addr);
if (is_valid_ether_addr(hw->mac.port_addr))
pf->flags |= I40E_FLAG_PORT_ID_VALID;
-#ifdef I40E_FCOE
- err = i40e_get_san_mac_addr(hw, hw->mac.san_addr);
- if (err)
- dev_info(&pdev->dev,
- "(non-fatal) SAN MAC retrieval failed: %d\n", err);
- if (!is_valid_ether_addr(hw->mac.san_addr)) {
- dev_warn(&pdev->dev, "invalid SAN MAC address %pM, falling back to LAN MAC\n",
- hw->mac.san_addr);
- ether_addr_copy(hw->mac.san_addr, hw->mac.addr);
- }
- dev_info(&pf->pdev->dev, "SAN MAC: %pM\n", hw->mac.san_addr);
-#endif /* I40E_FCOE */
pci_set_drvdata(pdev, pf);
pci_save_state(pdev);
@@ -11261,8 +11182,7 @@
pf->service_timer_period = HZ;
INIT_WORK(&pf->service_task, i40e_service_task);
- clear_bit(__I40E_SERVICE_SCHED, &pf->state);
- pf->flags |= I40E_FLAG_NEED_LINK_UPDATE;
+ clear_bit(__I40E_SERVICE_SCHED, pf->state);
/* NVM bit on means WoL disabled for the port */
i40e_read_nvm_word(hw, I40E_SR_NVM_WAKE_ON_LAN, &wol_nvm_bits);
@@ -11300,7 +11220,7 @@
/* prep for VF support */
if ((pf->flags & I40E_FLAG_SRIOV_ENABLED) &&
(pf->flags & I40E_FLAG_MSIX_ENABLED) &&
- !test_bit(__I40E_BAD_EEPROM, &pf->state)) {
+ !test_bit(__I40E_BAD_EEPROM, pf->state)) {
if (pci_num_vf(pdev))
pf->flags |= I40E_FLAG_VEB_MODE_ENABLED;
}
@@ -11373,7 +11293,7 @@
* before setting up the misc vector or we get a race and the vector
* ends up disabled forever.
*/
- clear_bit(__I40E_DOWN, &pf->state);
+ clear_bit(__I40E_VSI_DOWN, pf->state);
/* In case of MSIX we are going to setup the misc vector right here
* to handle admin queue events etc. In case of legacy and MSI
@@ -11393,7 +11313,7 @@
/* prep for VF support */
if ((pf->flags & I40E_FLAG_SRIOV_ENABLED) &&
(pf->flags & I40E_FLAG_MSIX_ENABLED) &&
- !test_bit(__I40E_BAD_EEPROM, &pf->state)) {
+ !test_bit(__I40E_BAD_EEPROM, pf->state)) {
/* disable link interrupts for VFs */
val = rd32(hw, I40E_PFGEN_PORTMDIO_NUM);
val &= ~I40E_PFGEN_PORTMDIO_NUM_VFLINK_STAT_ENA_MASK;
@@ -11434,16 +11354,13 @@
round_jiffies(jiffies + pf->service_timer_period));
/* add this PF to client device list and launch a client service task */
- err = i40e_lan_add_device(pf);
- if (err)
- dev_info(&pdev->dev, "Failed to add PF to client API service list: %d\n",
- err);
+ if (pf->flags & I40E_FLAG_IWARP_ENABLED) {
+ err = i40e_lan_add_device(pf);
+ if (err)
+ dev_info(&pdev->dev, "Failed to add PF to client API service list: %d\n",
+ err);
+ }
-#ifdef I40E_FCOE
- /* create FCoE interface */
- i40e_fcoe_vsi_setup(pf);
-
-#endif
#define PCI_SPEED_SIZE 8
#define PCI_WIDTH_SIZE 8
/* Devices on the IOSF bus do not have this information
@@ -11531,7 +11448,7 @@
/* Unwind what we've done if something failed in the setup */
err_vsis:
- set_bit(__I40E_DOWN, &pf->state);
+ set_bit(__I40E_VSI_DOWN, pf->state);
i40e_clear_interrupt_scheme(pf);
kfree(pf->vsi);
err_switch_setup:
@@ -11582,13 +11499,18 @@
i40e_write_rx_ctl(hw, I40E_PFQF_HENA(1), 0);
/* no more scheduling of any task */
- set_bit(__I40E_SUSPENDED, &pf->state);
- set_bit(__I40E_DOWN, &pf->state);
+ set_bit(__I40E_SUSPENDED, pf->state);
+ set_bit(__I40E_VSI_DOWN, pf->state);
if (pf->service_timer.data)
del_timer_sync(&pf->service_timer);
if (pf->service_task.func)
cancel_work_sync(&pf->service_task);
+ /* Client close must be called explicitly here because the timer
+ * has been stopped.
+ */
+ i40e_notify_client_of_netdev_close(pf->vsi[pf->lan_vsi], false);
+
if (pf->flags & I40E_FLAG_SRIOV_ENABLED) {
i40e_free_vfs(pf);
pf->flags &= ~I40E_FLAG_SRIOV_ENABLED;
@@ -11615,10 +11537,11 @@
i40e_vsi_release(pf->vsi[pf->lan_vsi]);
/* remove attached clients */
- ret_code = i40e_lan_del_device(pf);
- if (ret_code) {
- dev_warn(&pdev->dev, "Failed to delete client device: %d\n",
- ret_code);
+ if (pf->flags & I40E_FLAG_IWARP_ENABLED) {
+ ret_code = i40e_lan_del_device(pf);
+ if (ret_code)
+ dev_warn(&pdev->dev, "Failed to delete client device: %d\n",
+ ret_code);
}
/* shutdown and destroy the HMC */
@@ -11685,9 +11608,9 @@
}
/* shutdown all operations */
- if (!test_bit(__I40E_SUSPENDED, &pf->state)) {
+ if (!test_bit(__I40E_SUSPENDED, pf->state)) {
rtnl_lock();
- i40e_prep_for_reset(pf);
+ i40e_prep_for_reset(pf, true);
rtnl_unlock();
}
@@ -11752,11 +11675,11 @@
struct i40e_pf *pf = pci_get_drvdata(pdev);
dev_dbg(&pdev->dev, "%s\n", __func__);
- if (test_bit(__I40E_SUSPENDED, &pf->state))
+ if (test_bit(__I40E_SUSPENDED, pf->state))
return;
rtnl_lock();
- i40e_handle_reset_warning(pf);
+ i40e_handle_reset_warning(pf, true);
rtnl_unlock();
}
@@ -11816,10 +11739,10 @@
struct i40e_pf *pf = pci_get_drvdata(pdev);
struct i40e_hw *hw = &pf->hw;
- set_bit(__I40E_SUSPENDED, &pf->state);
- set_bit(__I40E_DOWN, &pf->state);
+ set_bit(__I40E_SUSPENDED, pf->state);
+ set_bit(__I40E_VSI_DOWN, pf->state);
rtnl_lock();
- i40e_prep_for_reset(pf);
+ i40e_prep_for_reset(pf, true);
rtnl_unlock();
wr32(hw, I40E_PFPM_APM, (pf->wol_en ? I40E_PFPM_APM_APME_MASK : 0));
@@ -11829,11 +11752,16 @@
cancel_work_sync(&pf->service_task);
i40e_fdir_teardown(pf);
+ /* Client close must be called explicitly here because the timer
+ * has been stopped.
+ */
+ i40e_notify_client_of_netdev_close(pf->vsi[pf->lan_vsi], false);
+
if (pf->wol_en && (pf->flags & I40E_FLAG_WOL_MC_MAGIC_PKT_WAKE))
i40e_enable_mc_magic_wake(pf);
rtnl_lock();
- i40e_prep_for_reset(pf);
+ i40e_prep_for_reset(pf, true);
rtnl_unlock();
wr32(hw, I40E_PFPM_APM,
@@ -11860,14 +11788,14 @@
struct i40e_hw *hw = &pf->hw;
int retval = 0;
- set_bit(__I40E_SUSPENDED, &pf->state);
- set_bit(__I40E_DOWN, &pf->state);
+ set_bit(__I40E_SUSPENDED, pf->state);
+ set_bit(__I40E_VSI_DOWN, pf->state);
if (pf->wol_en && (pf->flags & I40E_FLAG_WOL_MC_MAGIC_PKT_WAKE))
i40e_enable_mc_magic_wake(pf);
rtnl_lock();
- i40e_prep_for_reset(pf);
+ i40e_prep_for_reset(pf, true);
rtnl_unlock();
wr32(hw, I40E_PFPM_APM, (pf->wol_en ? I40E_PFPM_APM_APME_MASK : 0));
@@ -11912,10 +11840,10 @@
pci_wake_from_d3(pdev, false);
/* handling the reset will rebuild the device state */
- if (test_and_clear_bit(__I40E_SUSPENDED, &pf->state)) {
- clear_bit(__I40E_DOWN, &pf->state);
+ if (test_and_clear_bit(__I40E_SUSPENDED, pf->state)) {
+ clear_bit(__I40E_VSI_DOWN, pf->state);
rtnl_lock();
- i40e_reset_and_rebuild(pf, false);
+ i40e_reset_and_rebuild(pf, false, true);
rtnl_unlock();
}
diff --git a/drivers/net/ethernet/intel/i40e/i40e_nvm.c b/drivers/net/ethernet/intel/i40e/i40e_nvm.c
index 38ee18f..800bd55 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_nvm.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_nvm.c
@@ -292,14 +292,14 @@
{
enum i40e_status_code ret_code = 0;
- if (hw->flags & I40E_HW_FLAG_AQ_SRCTL_ACCESS_ENABLE) {
- ret_code = i40e_acquire_nvm(hw, I40E_RESOURCE_READ);
- if (!ret_code) {
+ ret_code = i40e_acquire_nvm(hw, I40E_RESOURCE_READ);
+ if (!ret_code) {
+ if (hw->flags & I40E_HW_FLAG_AQ_SRCTL_ACCESS_ENABLE) {
ret_code = i40e_read_nvm_word_aq(hw, offset, data);
- i40e_release_nvm(hw);
+ } else {
+ ret_code = i40e_read_nvm_word_srctl(hw, offset, data);
}
- } else {
- ret_code = i40e_read_nvm_word_srctl(hw, offset, data);
+ i40e_release_nvm(hw);
}
return ret_code;
}
diff --git a/drivers/net/ethernet/intel/i40e/i40e_osdep.h b/drivers/net/ethernet/intel/i40e/i40e_osdep.h
index fea81ed..80e66da 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_osdep.h
+++ b/drivers/net/ethernet/intel/i40e/i40e_osdep.h
@@ -78,7 +78,4 @@
} while (0)
typedef enum i40e_status_code i40e_status;
-#ifdef CONFIG_I40E_FCOE
-#define I40E_FCOE
-#endif
#endif /* _I40E_OSDEP_H_ */
diff --git a/drivers/net/ethernet/intel/i40e/i40e_prototype.h b/drivers/net/ethernet/intel/i40e/i40e_prototype.h
index 2551fc8..c56d976 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_prototype.h
+++ b/drivers/net/ethernet/intel/i40e/i40e_prototype.h
@@ -304,9 +304,6 @@
u32 pba_num_size);
i40e_status i40e_validate_mac_addr(u8 *mac_addr);
void i40e_pre_tx_queue_cfg(struct i40e_hw *hw, u32 queue, bool enable);
-#ifdef I40E_FCOE
-i40e_status i40e_get_san_mac_addr(struct i40e_hw *hw, u8 *mac_addr);
-#endif
/* prototype for functions used for NVM access */
i40e_status i40e_init_nvm(struct i40e_hw *hw);
i40e_status i40e_acquire_nvm(struct i40e_hw *hw,
@@ -380,4 +377,21 @@
u8 i40e_get_phy_address(struct i40e_hw *hw, u8 dev_num);
i40e_status i40e_blink_phy_link_led(struct i40e_hw *hw,
u32 time, u32 interval);
+i40e_status i40e_aq_write_ppp(struct i40e_hw *hw, void *buff,
+ u16 buff_size, u32 track_id,
+ u32 *error_offset, u32 *error_info,
+ struct i40e_asq_cmd_details *cmd_details);
+i40e_status i40e_aq_get_ppp_list(struct i40e_hw *hw, void *buff,
+ u16 buff_size, u8 flags,
+ struct i40e_asq_cmd_details *cmd_details);
+struct i40e_generic_seg_header *
+i40e_find_segment_in_package(u32 segment_type,
+ struct i40e_package_header *pkg_header);
+enum i40e_status_code
+i40e_write_profile(struct i40e_hw *hw, struct i40e_profile_segment *i40e_seg,
+ u32 track_id);
+enum i40e_status_code
+i40e_add_pinfo_to_list(struct i40e_hw *hw,
+ struct i40e_profile_segment *profile,
+ u8 *profile_info_sec, u32 track_id);
#endif /* _I40E_PROTOTYPE_H_ */
diff --git a/drivers/net/ethernet/intel/i40e/i40e_ptp.c b/drivers/net/ethernet/intel/i40e/i40e_ptp.c
index 2caee35..18c1cc0 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_ptp.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_ptp.c
@@ -358,7 +358,7 @@
skb_tstamp_tx(pf->ptp_tx_skb, &shhwtstamps);
dev_kfree_skb_any(pf->ptp_tx_skb);
pf->ptp_tx_skb = NULL;
- clear_bit_unlock(__I40E_PTP_TX_IN_PROGRESS, &pf->state);
+ clear_bit_unlock(__I40E_PTP_TX_IN_PROGRESS, pf->state);
}
/**
@@ -768,7 +768,7 @@
if (pf->ptp_tx_skb) {
dev_kfree_skb_any(pf->ptp_tx_skb);
pf->ptp_tx_skb = NULL;
- clear_bit_unlock(__I40E_PTP_TX_IN_PROGRESS, &pf->state);
+ clear_bit_unlock(__I40E_PTP_TX_IN_PROGRESS, pf->state);
}
if (pf->ptp_clock) {
diff --git a/drivers/net/ethernet/intel/i40e/i40e_trace.h b/drivers/net/ethernet/intel/i40e/i40e_trace.h
new file mode 100644
index 0000000..d3e55f5
--- /dev/null
+++ b/drivers/net/ethernet/intel/i40e/i40e_trace.h
@@ -0,0 +1,229 @@
+/*******************************************************************************
+ *
+ * Intel(R) 40-10 Gigabit Ethernet Connection Network Driver
+ * Copyright(c) 2013 - 2017 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Contact Information:
+ * e1000-devel Mailing List <e1000-devel@lists.sourceforge.net>
+ * Intel Corporation, 5200 N.E. Elam Young Parkway, Hillsboro, OR 97124-6497
+ *
+ ******************************************************************************/
+
+/* Modeled on trace-events-sample.h */
+
+/* The trace subsystem name for i40e will be "i40e".
+ *
+ * This file is named i40e_trace.h.
+ *
+ * Since this include file's name is different from the trace
+ * subsystem name, we'll have to define TRACE_INCLUDE_FILE at the end
+ * of this file.
+ */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM i40e
+
+/* See trace-events-sample.h for a detailed description of why this
+ * guard clause is different from most normal include files.
+ */
+#if !defined(_I40E_TRACE_H_) || defined(TRACE_HEADER_MULTI_READ)
+#define _I40E_TRACE_H_
+
+#include <linux/tracepoint.h>
+
+/**
+ * i40e_trace() macro enables shared code to refer to trace points
+ * like:
+ *
+ * trace_i40e{,vf}_example(args...)
+ *
+ * ... as:
+ *
+ * i40e_trace(example, args...)
+ *
+ * ... to resolve to the PF or VF version of the tracepoint without
+ * ifdefs, and to allow tracepoints to be disabled entirely at build
+ * time.
+ *
+ * Trace point should always be referred to in the driver via this
+ * macro.
+ *
+ * Similarly, i40e_trace_enabled(trace_name) wraps references to
+ * trace_i40e{,vf}_<trace_name>_enabled() functions.
+ */
+#define _I40E_TRACE_NAME(trace_name) (trace_ ## i40e ## _ ## trace_name)
+#define I40E_TRACE_NAME(trace_name) _I40E_TRACE_NAME(trace_name)
+
+#define i40e_trace(trace_name, args...) I40E_TRACE_NAME(trace_name)(args)
+
+#define i40e_trace_enabled(trace_name) I40E_TRACE_NAME(trace_name##_enabled)()
+
+/* Events common to PF and VF. Corresponding versions will be defined
+ * for both, named trace_i40e_* and trace_i40evf_*. The i40e_trace()
+ * macro above will select the right trace point name for the driver
+ * being built from shared code.
+ */
+
+/* Events related to a vsi & ring */
+DECLARE_EVENT_CLASS(
+ i40e_tx_template,
+
+ TP_PROTO(struct i40e_ring *ring,
+ struct i40e_tx_desc *desc,
+ struct i40e_tx_buffer *buf),
+
+ TP_ARGS(ring, desc, buf),
+
+ /* The convention here is to make the first fields in the
+ * TP_STRUCT match the TP_PROTO exactly. This enables the use
+ * of the args struct generated by the tplist tool (from the
+ * bcc-tools package) to be used for those fields. To access
+ * fields other than the tracepoint args will require the
+ * tplist output to be adjusted.
+ */
+ TP_STRUCT__entry(
+ __field(void*, ring)
+ __field(void*, desc)
+ __field(void*, buf)
+ __string(devname, ring->netdev->name)
+ ),
+
+ TP_fast_assign(
+ __entry->ring = ring;
+ __entry->desc = desc;
+ __entry->buf = buf;
+ __assign_str(devname, ring->netdev->name);
+ ),
+
+ TP_printk(
+ "netdev: %s ring: %p desc: %p buf %p",
+ __get_str(devname), __entry->ring,
+ __entry->desc, __entry->buf)
+);
+
+DEFINE_EVENT(
+ i40e_tx_template, i40e_clean_tx_irq,
+ TP_PROTO(struct i40e_ring *ring,
+ struct i40e_tx_desc *desc,
+ struct i40e_tx_buffer *buf),
+
+ TP_ARGS(ring, desc, buf));
+
+DEFINE_EVENT(
+ i40e_tx_template, i40e_clean_tx_irq_unmap,
+ TP_PROTO(struct i40e_ring *ring,
+ struct i40e_tx_desc *desc,
+ struct i40e_tx_buffer *buf),
+
+ TP_ARGS(ring, desc, buf));
+
+DECLARE_EVENT_CLASS(
+ i40e_rx_template,
+
+ TP_PROTO(struct i40e_ring *ring,
+ union i40e_32byte_rx_desc *desc,
+ struct sk_buff *skb),
+
+ TP_ARGS(ring, desc, skb),
+
+ TP_STRUCT__entry(
+ __field(void*, ring)
+ __field(void*, desc)
+ __field(void*, skb)
+ __string(devname, ring->netdev->name)
+ ),
+
+ TP_fast_assign(
+ __entry->ring = ring;
+ __entry->desc = desc;
+ __entry->skb = skb;
+ __assign_str(devname, ring->netdev->name);
+ ),
+
+ TP_printk(
+ "netdev: %s ring: %p desc: %p skb %p",
+ __get_str(devname), __entry->ring,
+ __entry->desc, __entry->skb)
+);
+
+DEFINE_EVENT(
+ i40e_rx_template, i40e_clean_rx_irq,
+ TP_PROTO(struct i40e_ring *ring,
+ union i40e_32byte_rx_desc *desc,
+ struct sk_buff *skb),
+
+ TP_ARGS(ring, desc, skb));
+
+DEFINE_EVENT(
+ i40e_rx_template, i40e_clean_rx_irq_rx,
+ TP_PROTO(struct i40e_ring *ring,
+ union i40e_32byte_rx_desc *desc,
+ struct sk_buff *skb),
+
+ TP_ARGS(ring, desc, skb));
+
+DECLARE_EVENT_CLASS(
+ i40e_xmit_template,
+
+ TP_PROTO(struct sk_buff *skb,
+ struct i40e_ring *ring),
+
+ TP_ARGS(skb, ring),
+
+ TP_STRUCT__entry(
+ __field(void*, skb)
+ __field(void*, ring)
+ __string(devname, ring->netdev->name)
+ ),
+
+ TP_fast_assign(
+ __entry->skb = skb;
+ __entry->ring = ring;
+ __assign_str(devname, ring->netdev->name);
+ ),
+
+ TP_printk(
+ "netdev: %s skb: %p ring: %p",
+ __get_str(devname), __entry->skb,
+ __entry->ring)
+);
+
+DEFINE_EVENT(
+ i40e_xmit_template, i40e_xmit_frame_ring,
+ TP_PROTO(struct sk_buff *skb,
+ struct i40e_ring *ring),
+
+ TP_ARGS(skb, ring));
+
+DEFINE_EVENT(
+ i40e_xmit_template, i40e_xmit_frame_ring_drop,
+ TP_PROTO(struct sk_buff *skb,
+ struct i40e_ring *ring),
+
+ TP_ARGS(skb, ring));
+
+/* Events unique to the PF. */
+
+#endif /* _I40E_TRACE_H_ */
+/* This must be outside ifdef _I40E_TRACE_H */
+
+/* This trace include file is not located in the .../include/trace
+ * with the kernel tracepoint definitions, because we're a loadable
+ * module.
+ */
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE i40e_trace
+#include <trace/define_trace.h>
diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c
index 97d4605..29321a6 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c
@@ -27,6 +27,7 @@
#include <linux/prefetch.h>
#include <net/busy_poll.h>
#include "i40e.h"
+#include "i40e_trace.h"
#include "i40e_prototype.h"
static inline __le64 build_ctob(u32 td_cmd, u32 td_offset, unsigned int size,
@@ -71,6 +72,9 @@
flex_ptype |= I40E_TXD_FLTR_QW0_PCTYPE_MASK &
(fdata->pctype << I40E_TXD_FLTR_QW0_PCTYPE_SHIFT);
+ flex_ptype |= I40E_TXD_FLTR_QW0_PCTYPE_MASK &
+ (fdata->flex_offset << I40E_TXD_FLTR_QW0_FLEXOFF_SHIFT);
+
/* Use LAN VSI Id if not programmed by user */
flex_ptype |= I40E_TXD_FLTR_QW0_DEST_VSI_MASK &
((u32)(fdata->dest_vsi ? : pf->vsi[pf->lan_vsi]->id) <<
@@ -203,7 +207,6 @@
struct i40e_pf *pf = vsi->back;
struct udphdr *udp;
struct iphdr *ip;
- bool err = false;
u8 *raw_packet;
int ret;
static char packet[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x08, 0,
@@ -219,18 +222,28 @@
udp = (struct udphdr *)(raw_packet + IP_HEADER_OFFSET
+ sizeof(struct iphdr));
- ip->daddr = fd_data->dst_ip[0];
+ ip->daddr = fd_data->dst_ip;
udp->dest = fd_data->dst_port;
- ip->saddr = fd_data->src_ip[0];
+ ip->saddr = fd_data->src_ip;
udp->source = fd_data->src_port;
+ if (fd_data->flex_filter) {
+ u8 *payload = raw_packet + I40E_UDPIP_DUMMY_PACKET_LEN;
+ __be16 pattern = fd_data->flex_word;
+ u16 off = fd_data->flex_offset;
+
+ *((__force __be16 *)(payload + off)) = pattern;
+ }
+
fd_data->pctype = I40E_FILTER_PCTYPE_NONF_IPV4_UDP;
ret = i40e_program_fdir_filter(fd_data, raw_packet, pf, add);
if (ret) {
dev_info(&pf->pdev->dev,
"PCTYPE:%d, Filter command send failed for fd_id:%d (ret = %d)\n",
fd_data->pctype, fd_data->fd_id, ret);
- err = true;
+ /* Free the packet buffer since it wasn't added to the ring */
+ kfree(raw_packet);
+ return -EOPNOTSUPP;
} else if (I40E_DEBUG_FD & pf->hw.debug_mask) {
if (add)
dev_info(&pf->pdev->dev,
@@ -241,10 +254,13 @@
"Filter deleted for PCTYPE %d loc = %d\n",
fd_data->pctype, fd_data->fd_id);
}
- if (err)
- kfree(raw_packet);
- return err ? -EOPNOTSUPP : 0;
+ if (add)
+ pf->fd_udp4_filter_cnt++;
+ else
+ pf->fd_udp4_filter_cnt--;
+
+ return 0;
}
#define I40E_TCPIP_DUMMY_PACKET_LEN 54
@@ -263,7 +279,6 @@
struct i40e_pf *pf = vsi->back;
struct tcphdr *tcp;
struct iphdr *ip;
- bool err = false;
u8 *raw_packet;
int ret;
/* Dummy packet */
@@ -281,36 +296,28 @@
tcp = (struct tcphdr *)(raw_packet + IP_HEADER_OFFSET
+ sizeof(struct iphdr));
- ip->daddr = fd_data->dst_ip[0];
+ ip->daddr = fd_data->dst_ip;
tcp->dest = fd_data->dst_port;
- ip->saddr = fd_data->src_ip[0];
+ ip->saddr = fd_data->src_ip;
tcp->source = fd_data->src_port;
- if (add) {
- pf->fd_tcp_rule++;
- if ((pf->flags & I40E_FLAG_FD_ATR_ENABLED) &&
- I40E_DEBUG_FD & pf->hw.debug_mask)
- dev_info(&pf->pdev->dev, "Forcing ATR off, sideband rules for TCP/IPv4 flow being applied\n");
- pf->auto_disable_flags |= I40E_FLAG_FD_ATR_ENABLED;
- } else {
- pf->fd_tcp_rule = (pf->fd_tcp_rule > 0) ?
- (pf->fd_tcp_rule - 1) : 0;
- if (pf->fd_tcp_rule == 0) {
- if ((pf->flags & I40E_FLAG_FD_ATR_ENABLED) &&
- I40E_DEBUG_FD & pf->hw.debug_mask)
- dev_info(&pf->pdev->dev, "ATR re-enabled due to no sideband TCP/IPv4 rules\n");
- pf->auto_disable_flags &= ~I40E_FLAG_FD_ATR_ENABLED;
- }
+ if (fd_data->flex_filter) {
+ u8 *payload = raw_packet + I40E_TCPIP_DUMMY_PACKET_LEN;
+ __be16 pattern = fd_data->flex_word;
+ u16 off = fd_data->flex_offset;
+
+ *((__force __be16 *)(payload + off)) = pattern;
}
fd_data->pctype = I40E_FILTER_PCTYPE_NONF_IPV4_TCP;
ret = i40e_program_fdir_filter(fd_data, raw_packet, pf, add);
-
if (ret) {
dev_info(&pf->pdev->dev,
"PCTYPE:%d, Filter command send failed for fd_id:%d (ret = %d)\n",
fd_data->pctype, fd_data->fd_id, ret);
- err = true;
+ /* Free the packet buffer since it wasn't added to the ring */
+ kfree(raw_packet);
+ return -EOPNOTSUPP;
} else if (I40E_DEBUG_FD & pf->hw.debug_mask) {
if (add)
dev_info(&pf->pdev->dev, "Filter OK for PCTYPE %d loc = %d)\n",
@@ -321,10 +328,91 @@
fd_data->pctype, fd_data->fd_id);
}
- if (err)
- kfree(raw_packet);
+ if (add) {
+ pf->fd_tcp4_filter_cnt++;
+ if ((pf->flags & I40E_FLAG_FD_ATR_ENABLED) &&
+ I40E_DEBUG_FD & pf->hw.debug_mask)
+ dev_info(&pf->pdev->dev, "Forcing ATR off, sideband rules for TCP/IPv4 flow being applied\n");
+ pf->flags |= I40E_FLAG_FD_ATR_AUTO_DISABLED;
+ } else {
+ pf->fd_tcp4_filter_cnt--;
+ }
- return err ? -EOPNOTSUPP : 0;
+ return 0;
+}
+
+#define I40E_SCTPIP_DUMMY_PACKET_LEN 46
+/**
+ * i40e_add_del_fdir_sctpv4 - Add/Remove SCTPv4 Flow Director filters for
+ * a specific flow spec
+ * @vsi: pointer to the targeted VSI
+ * @fd_data: the flow director data required for the FDir descriptor
+ * @add: true adds a filter, false removes it
+ *
+ * Returns 0 if the filters were successfully added or removed
+ **/
+static int i40e_add_del_fdir_sctpv4(struct i40e_vsi *vsi,
+ struct i40e_fdir_filter *fd_data,
+ bool add)
+{
+ struct i40e_pf *pf = vsi->back;
+ struct sctphdr *sctp;
+ struct iphdr *ip;
+ u8 *raw_packet;
+ int ret;
+ /* Dummy packet */
+ static char packet[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x08, 0,
+ 0x45, 0, 0, 0x20, 0, 0, 0x40, 0, 0x40, 0x84, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
+
+ raw_packet = kzalloc(I40E_FDIR_MAX_RAW_PACKET_SIZE, GFP_KERNEL);
+ if (!raw_packet)
+ return -ENOMEM;
+ memcpy(raw_packet, packet, I40E_SCTPIP_DUMMY_PACKET_LEN);
+
+ ip = (struct iphdr *)(raw_packet + IP_HEADER_OFFSET);
+ sctp = (struct sctphdr *)(raw_packet + IP_HEADER_OFFSET
+ + sizeof(struct iphdr));
+
+ ip->daddr = fd_data->dst_ip;
+ sctp->dest = fd_data->dst_port;
+ ip->saddr = fd_data->src_ip;
+ sctp->source = fd_data->src_port;
+
+ if (fd_data->flex_filter) {
+ u8 *payload = raw_packet + I40E_SCTPIP_DUMMY_PACKET_LEN;
+ __be16 pattern = fd_data->flex_word;
+ u16 off = fd_data->flex_offset;
+
+ *((__force __be16 *)(payload + off)) = pattern;
+ }
+
+ fd_data->pctype = I40E_FILTER_PCTYPE_NONF_IPV4_SCTP;
+ ret = i40e_program_fdir_filter(fd_data, raw_packet, pf, add);
+ if (ret) {
+ dev_info(&pf->pdev->dev,
+ "PCTYPE:%d, Filter command send failed for fd_id:%d (ret = %d)\n",
+ fd_data->pctype, fd_data->fd_id, ret);
+ /* Free the packet buffer since it wasn't added to the ring */
+ kfree(raw_packet);
+ return -EOPNOTSUPP;
+ } else if (I40E_DEBUG_FD & pf->hw.debug_mask) {
+ if (add)
+ dev_info(&pf->pdev->dev,
+ "Filter OK for PCTYPE %d loc = %d\n",
+ fd_data->pctype, fd_data->fd_id);
+ else
+ dev_info(&pf->pdev->dev,
+ "Filter deleted for PCTYPE %d loc = %d\n",
+ fd_data->pctype, fd_data->fd_id);
+ }
+
+ if (add)
+ pf->fd_sctp4_filter_cnt++;
+ else
+ pf->fd_sctp4_filter_cnt--;
+
+ return 0;
}
#define I40E_IP_DUMMY_PACKET_LEN 34
@@ -343,7 +431,6 @@
{
struct i40e_pf *pf = vsi->back;
struct iphdr *ip;
- bool err = false;
u8 *raw_packet;
int ret;
int i;
@@ -359,18 +446,29 @@
memcpy(raw_packet, packet, I40E_IP_DUMMY_PACKET_LEN);
ip = (struct iphdr *)(raw_packet + IP_HEADER_OFFSET);
- ip->saddr = fd_data->src_ip[0];
- ip->daddr = fd_data->dst_ip[0];
+ ip->saddr = fd_data->src_ip;
+ ip->daddr = fd_data->dst_ip;
ip->protocol = 0;
+ if (fd_data->flex_filter) {
+ u8 *payload = raw_packet + I40E_IP_DUMMY_PACKET_LEN;
+ __be16 pattern = fd_data->flex_word;
+ u16 off = fd_data->flex_offset;
+
+ *((__force __be16 *)(payload + off)) = pattern;
+ }
+
fd_data->pctype = i;
ret = i40e_program_fdir_filter(fd_data, raw_packet, pf, add);
-
if (ret) {
dev_info(&pf->pdev->dev,
"PCTYPE:%d, Filter command send failed for fd_id:%d (ret = %d)\n",
fd_data->pctype, fd_data->fd_id, ret);
- err = true;
+ /* The packet buffer wasn't added to the ring so we
+ * need to free it now.
+ */
+ kfree(raw_packet);
+ return -EOPNOTSUPP;
} else if (I40E_DEBUG_FD & pf->hw.debug_mask) {
if (add)
dev_info(&pf->pdev->dev,
@@ -383,10 +481,12 @@
}
}
- if (err)
- kfree(raw_packet);
+ if (add)
+ pf->fd_ip4_filter_cnt++;
+ else
+ pf->fd_ip4_filter_cnt--;
- return err ? -EOPNOTSUPP : 0;
+ return 0;
}
/**
@@ -409,6 +509,9 @@
case UDP_V4_FLOW:
ret = i40e_add_del_fdir_udpv4(vsi, input, add);
break;
+ case SCTP_V4_FLOW:
+ ret = i40e_add_del_fdir_sctpv4(vsi, input, add);
+ break;
case IP_USER_FLOW:
switch (input->ip4_proto) {
case IPPROTO_TCP:
@@ -417,19 +520,23 @@
case IPPROTO_UDP:
ret = i40e_add_del_fdir_udpv4(vsi, input, add);
break;
+ case IPPROTO_SCTP:
+ ret = i40e_add_del_fdir_sctpv4(vsi, input, add);
+ break;
case IPPROTO_IP:
ret = i40e_add_del_fdir_ipv4(vsi, input, add);
break;
default:
/* We cannot support masking based on protocol */
- goto unsupported_flow;
+ dev_info(&pf->pdev->dev, "Unsupported IPv4 protocol 0x%02x\n",
+ input->ip4_proto);
+ return -EINVAL;
}
break;
default:
-unsupported_flow:
- dev_info(&pf->pdev->dev, "Could not specify spec type %d\n",
+ dev_info(&pf->pdev->dev, "Unsupported flow type 0x%02x\n",
input->flow_type);
- ret = -EINVAL;
+ return -EINVAL;
}
/* The buffer allocated here will be normally be freed by
@@ -476,7 +583,7 @@
* progress do nothing, once flush is complete the state will
* be cleared.
*/
- if (test_bit(__I40E_FD_FLUSH_REQUESTED, &pf->state))
+ if (test_bit(__I40E_FD_FLUSH_REQUESTED, pf->state))
return;
pf->fd_add_err++;
@@ -484,9 +591,9 @@
pf->fd_atr_cnt = i40e_get_current_atr_cnt(pf);
if ((rx_desc->wb.qword0.hi_dword.fd_id == 0) &&
- (pf->auto_disable_flags & I40E_FLAG_FD_SB_ENABLED)) {
- pf->auto_disable_flags |= I40E_FLAG_FD_ATR_ENABLED;
- set_bit(__I40E_FD_FLUSH_REQUESTED, &pf->state);
+ pf->flags & I40E_FLAG_FD_SB_AUTO_DISABLED) {
+ pf->flags |= I40E_FLAG_FD_ATR_AUTO_DISABLED;
+ set_bit(__I40E_FD_FLUSH_REQUESTED, pf->state);
}
/* filter programming failed most likely due to table full */
@@ -498,12 +605,10 @@
*/
if (fcnt_prog >= (fcnt_avail - I40E_FDIR_BUFFER_FULL_MARGIN)) {
if ((pf->flags & I40E_FLAG_FD_SB_ENABLED) &&
- !(pf->auto_disable_flags &
- I40E_FLAG_FD_SB_ENABLED)) {
+ !(pf->flags & I40E_FLAG_FD_SB_AUTO_DISABLED)) {
+ pf->flags |= I40E_FLAG_FD_SB_AUTO_DISABLED;
if (I40E_DEBUG_FD & pf->hw.debug_mask)
dev_warn(&pdev->dev, "FD filter space full, new ntuple rules will not be added\n");
- pf->auto_disable_flags |=
- I40E_FLAG_FD_SB_ENABLED;
}
}
} else if (error == BIT(I40E_RX_PROG_STATUS_DESC_NO_FD_ENTRY_SHIFT)) {
@@ -599,19 +704,15 @@
/**
* i40e_get_tx_pending - how many tx descriptors not processed
* @tx_ring: the ring of descriptors
- * @in_sw: is tx_pending being checked in SW or HW
*
* Since there is no access to the ring head register
* in XL710, we need to use our local copies
**/
-u32 i40e_get_tx_pending(struct i40e_ring *ring, bool in_sw)
+u32 i40e_get_tx_pending(struct i40e_ring *ring)
{
u32 head, tail;
- if (!in_sw)
- head = i40e_get_head(ring);
- else
- head = ring->next_to_clean;
+ head = i40e_get_head(ring);
tail = readl(ring->tail);
if (head != tail)
@@ -657,6 +758,7 @@
/* prevent any other reads prior to eop_desc */
read_barrier_depends();
+ i40e_trace(clean_tx_irq, tx_ring, tx_desc, tx_buf);
/* we have caught up to head, no work left to do */
if (tx_head == tx_desc)
break;
@@ -683,6 +785,8 @@
/* unmap remaining buffers */
while (tx_desc != eop_desc) {
+ i40e_trace(clean_tx_irq_unmap,
+ tx_ring, tx_desc, tx_buf);
tx_buf++;
tx_desc++;
@@ -734,11 +838,11 @@
* them to be written back in case we stay in NAPI.
* In this mode on X722 we do not enable Interrupt.
*/
- unsigned int j = i40e_get_tx_pending(tx_ring, false);
+ unsigned int j = i40e_get_tx_pending(tx_ring);
if (budget &&
((j / WB_STRIDE) == 0) && (j > 0) &&
- !test_bit(__I40E_DOWN, &vsi->state) &&
+ !test_bit(__I40E_VSI_DOWN, vsi->state) &&
(I40E_DESC_UNUSED(tx_ring) != tx_ring->count))
tx_ring->arm_wb = true;
}
@@ -756,7 +860,7 @@
smp_mb();
if (__netif_subqueue_stopped(tx_ring->netdev,
tx_ring->queue_index) &&
- !test_bit(__I40E_DOWN, &vsi->state)) {
+ !test_bit(__I40E_VSI_DOWN, vsi->state)) {
netif_wake_subqueue(tx_ring->netdev,
tx_ring->queue_index);
++tx_ring->tx_stats.restart_queue;
@@ -930,9 +1034,29 @@
}
/**
+ * i40e_rx_is_programming_status - check for programming status descriptor
+ * @qw: qword representing status_error_len in CPU ordering
+ *
+ * The value of in the descriptor length field indicate if this
+ * is a programming status descriptor for flow director or FCoE
+ * by the value of I40E_RX_PROG_STATUS_DESC_LENGTH, otherwise
+ * it is a packet descriptor.
+ **/
+static inline bool i40e_rx_is_programming_status(u64 qw)
+{
+ /* The Rx filter programming status and SPH bit occupy the same
+ * spot in the descriptor. Since we don't support packet split we
+ * can just reuse the bit as an indication that this is a
+ * programming status descriptor.
+ */
+ return qw & I40E_RXD_QW1_LENGTH_SPH_MASK;
+}
+
+/**
* i40e_clean_programming_status - clean the programming status descriptor
* @rx_ring: the rx ring that has this descriptor
* @rx_desc: the rx descriptor written back by HW
+ * @qw: qword representing status_error_len in CPU ordering
*
* Flow director should handle FD_FILTER_STATUS to check its filter programming
* status being successful or not and take actions accordingly. FCoE should
@@ -940,22 +1064,23 @@
*
**/
static void i40e_clean_programming_status(struct i40e_ring *rx_ring,
- union i40e_rx_desc *rx_desc)
+ union i40e_rx_desc *rx_desc,
+ u64 qw)
{
- u64 qw;
+ u32 ntc = rx_ring->next_to_clean + 1;
u8 id;
- qw = le64_to_cpu(rx_desc->wb.qword1.status_error_len);
+ /* fetch, update, and store next to clean */
+ ntc = (ntc < rx_ring->count) ? ntc : 0;
+ rx_ring->next_to_clean = ntc;
+
+ prefetch(I40E_RX_DESC(rx_ring, ntc));
+
id = (qw & I40E_RX_PROG_STATUS_DESC_QW1_PROGID_MASK) >>
I40E_RX_PROG_STATUS_DESC_QW1_PROGID_SHIFT;
if (id == I40E_RX_PROG_STATUS_DESC_FD_FILTER_STATUS)
i40e_fd_handle_status(rx_ring, rx_desc, id);
-#ifdef I40E_FCOE
- else if ((id == I40E_RX_PROG_STATUS_DESC_FCOE_CTXT_PROG_STATUS) ||
- (id == I40E_RX_PROG_STATUS_DESC_FCOE_CTXT_INVL_STATUS))
- i40e_fcoe_handle_status(rx_ring, rx_desc, id);
-#endif
}
/**
@@ -1010,7 +1135,6 @@
**/
void i40e_clean_rx_ring(struct i40e_ring *rx_ring)
{
- struct device *dev = rx_ring->dev;
unsigned long bi_size;
u16 i;
@@ -1030,8 +1154,22 @@
if (!rx_bi->page)
continue;
- dma_unmap_page(dev, rx_bi->dma, PAGE_SIZE, DMA_FROM_DEVICE);
- __free_pages(rx_bi->page, 0);
+ /* Invalidate cache lines that may have been written to by
+ * device so that we avoid corrupting memory.
+ */
+ dma_sync_single_range_for_cpu(rx_ring->dev,
+ rx_bi->dma,
+ rx_bi->page_offset,
+ rx_ring->rx_buf_len,
+ DMA_FROM_DEVICE);
+
+ /* free resources associated with mapping */
+ dma_unmap_page_attrs(rx_ring->dev, rx_bi->dma,
+ i40e_rx_pg_size(rx_ring),
+ DMA_FROM_DEVICE,
+ I40E_RX_DMA_ATTR);
+
+ __page_frag_cache_drain(rx_bi->page, rx_bi->pagecnt_bias);
rx_bi->page = NULL;
rx_bi->page_offset = 0;
@@ -1132,6 +1270,17 @@
}
/**
+ * i40e_rx_offset - Return expected offset into page to access data
+ * @rx_ring: Ring we are requesting offset of
+ *
+ * Returns the offset value for ring into the data buffer.
+ */
+static inline unsigned int i40e_rx_offset(struct i40e_ring *rx_ring)
+{
+ return ring_uses_build_skb(rx_ring) ? I40E_SKB_PAD : 0;
+}
+
+/**
* i40e_alloc_mapped_page - recycle or make a new page
* @rx_ring: ring to use
* @bi: rx_buffer struct to modify
@@ -1152,27 +1301,33 @@
}
/* alloc new page for storage */
- page = dev_alloc_page();
+ page = dev_alloc_pages(i40e_rx_pg_order(rx_ring));
if (unlikely(!page)) {
rx_ring->rx_stats.alloc_page_failed++;
return false;
}
/* map page for use */
- dma = dma_map_page(rx_ring->dev, page, 0, PAGE_SIZE, DMA_FROM_DEVICE);
+ dma = dma_map_page_attrs(rx_ring->dev, page, 0,
+ i40e_rx_pg_size(rx_ring),
+ DMA_FROM_DEVICE,
+ I40E_RX_DMA_ATTR);
/* if mapping failed free memory back to system since
* there isn't much point in holding memory we can't use
*/
if (dma_mapping_error(rx_ring->dev, dma)) {
- __free_pages(page, 0);
+ __free_pages(page, i40e_rx_pg_order(rx_ring));
rx_ring->rx_stats.alloc_page_failed++;
return false;
}
bi->dma = dma;
bi->page = page;
- bi->page_offset = 0;
+ bi->page_offset = i40e_rx_offset(rx_ring);
+
+ /* initialize pagecnt_bias to 1 representing we fully own page */
+ bi->pagecnt_bias = 1;
return true;
}
@@ -1219,6 +1374,12 @@
if (!i40e_alloc_mapped_page(rx_ring, bi))
goto no_buffers;
+ /* sync the buffer for use by the device */
+ dma_sync_single_range_for_device(rx_ring->dev, bi->dma,
+ bi->page_offset,
+ rx_ring->rx_buf_len,
+ DMA_FROM_DEVICE);
+
/* Refresh the desc even if buffer_addrs didn't change
* because each write-back erases this info.
*/
@@ -1259,8 +1420,6 @@
* @vsi: the VSI we care about
* @skb: skb currently being received and modified
* @rx_desc: the receive descriptor
- *
- * skb->protocol must be set before this function is called
**/
static inline void i40e_rx_checksum(struct i40e_vsi *vsi,
struct sk_buff *skb,
@@ -1422,12 +1581,12 @@
i40e_rx_hash(rx_ring, rx_desc, skb, rx_ptype);
- /* modifies the skb - consumes the enet header */
- skb->protocol = eth_type_trans(skb, rx_ring->netdev);
-
i40e_rx_checksum(rx_ring->vsi, skb, rx_desc);
skb_record_rx_queue(skb, rx_ring->queue_index);
+
+ /* modifies the skb - consumes the enet header */
+ skb->protocol = eth_type_trans(skb, rx_ring->netdev);
}
/**
@@ -1472,7 +1631,10 @@
rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;
/* transfer page from old buffer to new buffer */
- *new_buff = *old_buff;
+ new_buff->dma = old_buff->dma;
+ new_buff->page = old_buff->page;
+ new_buff->page_offset = old_buff->page_offset;
+ new_buff->pagecnt_bias = old_buff->pagecnt_bias;
}
/**
@@ -1493,8 +1655,6 @@
* the adapter for another receive
*
* @rx_buffer: buffer containing the page
- * @page: page address from rx_buffer
- * @truesize: actual size of the buffer in this page
*
* If page is reusable, rx_buffer->page_offset is adjusted to point to
* an unused region in the page.
@@ -1517,13 +1677,10 @@
*
* In either case, if the page is reusable its refcount is increased.
**/
-static bool i40e_can_reuse_rx_page(struct i40e_rx_buffer *rx_buffer,
- struct page *page,
- const unsigned int truesize)
+static bool i40e_can_reuse_rx_page(struct i40e_rx_buffer *rx_buffer)
{
-#if (PAGE_SIZE >= 8192)
- unsigned int last_offset = PAGE_SIZE - I40E_RXBUFFER_2048;
-#endif
+ unsigned int pagecnt_bias = rx_buffer->pagecnt_bias;
+ struct page *page = rx_buffer->page;
/* Is any reuse possible? */
if (unlikely(!i40e_page_is_reusable(page)))
@@ -1531,21 +1688,23 @@
#if (PAGE_SIZE < 8192)
/* if we are only owner of page we can reuse it */
- if (unlikely(page_count(page) != 1))
+ if (unlikely((page_count(page) - pagecnt_bias) > 1))
return false;
-
- /* flip page offset to other buffer */
- rx_buffer->page_offset ^= truesize;
#else
- /* move offset up to the next cache line */
- rx_buffer->page_offset += truesize;
-
- if (rx_buffer->page_offset > last_offset)
+#define I40E_LAST_OFFSET \
+ (SKB_WITH_OVERHEAD(PAGE_SIZE) - I40E_RXBUFFER_2048)
+ if (rx_buffer->page_offset > I40E_LAST_OFFSET)
return false;
#endif
- /* Inc ref count on page before passing it up to the stack */
- get_page(page);
+ /* If we have drained the page fragment pool we need to update
+ * the pagecnt_bias and page count so that we fully restock the
+ * number of references the driver holds.
+ */
+ if (unlikely(!pagecnt_bias)) {
+ page_ref_add(page, USHRT_MAX);
+ rx_buffer->pagecnt_bias = USHRT_MAX;
+ }
return true;
}
@@ -1554,122 +1713,51 @@
* i40e_add_rx_frag - Add contents of Rx buffer to sk_buff
* @rx_ring: rx descriptor ring to transact packets on
* @rx_buffer: buffer containing page to add
- * @size: packet length from rx_desc
* @skb: sk_buff to place the data into
+ * @size: packet length from rx_desc
*
* This function will add the data contained in rx_buffer->page to the skb.
- * This is done either through a direct copy if the data in the buffer is
- * less than the skb header size, otherwise it will just attach the page as
- * a frag to the skb.
+ * It will just attach the page as a frag to the skb.
*
- * The function will then update the page offset if necessary and return
- * true if the buffer can be reused by the adapter.
+ * The function will then update the page offset.
**/
-static bool i40e_add_rx_frag(struct i40e_ring *rx_ring,
+static void i40e_add_rx_frag(struct i40e_ring *rx_ring,
struct i40e_rx_buffer *rx_buffer,
- unsigned int size,
- struct sk_buff *skb)
+ struct sk_buff *skb,
+ unsigned int size)
{
- struct page *page = rx_buffer->page;
- unsigned char *va = page_address(page) + rx_buffer->page_offset;
#if (PAGE_SIZE < 8192)
- unsigned int truesize = I40E_RXBUFFER_2048;
+ unsigned int truesize = i40e_rx_pg_size(rx_ring) / 2;
#else
- unsigned int truesize = ALIGN(size, L1_CACHE_BYTES);
+ unsigned int truesize = SKB_DATA_ALIGN(size + i40e_rx_offset(rx_ring));
#endif
- unsigned int pull_len;
- if (unlikely(skb_is_nonlinear(skb)))
- goto add_tail_frag;
+ skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buffer->page,
+ rx_buffer->page_offset, size, truesize);
- /* will the data fit in the skb we allocated? if so, just
- * copy it as it is pretty small anyway
- */
- if (size <= I40E_RX_HDR_SIZE) {
- memcpy(__skb_put(skb, size), va, ALIGN(size, sizeof(long)));
-
- /* page is reusable, we can reuse buffer as-is */
- if (likely(i40e_page_is_reusable(page)))
- return true;
-
- /* this page cannot be reused so discard it */
- __free_pages(page, 0);
- return false;
- }
-
- /* we need the header to contain the greater of either
- * ETH_HLEN or 60 bytes if the skb->len is less than
- * 60 for skb_pad.
- */
- pull_len = eth_get_headlen(va, I40E_RX_HDR_SIZE);
-
- /* align pull length to size of long to optimize
- * memcpy performance
- */
- memcpy(__skb_put(skb, pull_len), va, ALIGN(pull_len, sizeof(long)));
-
- /* update all of the pointers */
- va += pull_len;
- size -= pull_len;
-
-add_tail_frag:
- skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, page,
- (unsigned long)va & ~PAGE_MASK, size, truesize);
-
- return i40e_can_reuse_rx_page(rx_buffer, page, truesize);
+ /* page is being used so we must update the page offset */
+#if (PAGE_SIZE < 8192)
+ rx_buffer->page_offset ^= truesize;
+#else
+ rx_buffer->page_offset += truesize;
+#endif
}
/**
- * i40e_fetch_rx_buffer - Allocate skb and populate it
+ * i40e_get_rx_buffer - Fetch Rx buffer and synchronize data for use
* @rx_ring: rx descriptor ring to transact packets on
- * @rx_desc: descriptor containing info written by hardware
+ * @size: size of buffer to add to skb
*
- * This function allocates an skb on the fly, and populates it with the page
- * data from the current receive descriptor, taking care to set up the skb
- * correctly, as well as handling calling the page recycle function if
- * necessary.
+ * This function will pull an Rx buffer from the ring and synchronize it
+ * for use by the CPU.
*/
-static inline
-struct sk_buff *i40e_fetch_rx_buffer(struct i40e_ring *rx_ring,
- union i40e_rx_desc *rx_desc,
- struct sk_buff *skb)
+static struct i40e_rx_buffer *i40e_get_rx_buffer(struct i40e_ring *rx_ring,
+ const unsigned int size)
{
- u64 local_status_error_len =
- le64_to_cpu(rx_desc->wb.qword1.status_error_len);
- unsigned int size =
- (local_status_error_len & I40E_RXD_QW1_LENGTH_PBUF_MASK) >>
- I40E_RXD_QW1_LENGTH_PBUF_SHIFT;
struct i40e_rx_buffer *rx_buffer;
- struct page *page;
rx_buffer = &rx_ring->rx_bi[rx_ring->next_to_clean];
- page = rx_buffer->page;
- prefetchw(page);
-
- if (likely(!skb)) {
- void *page_addr = page_address(page) + rx_buffer->page_offset;
-
- /* prefetch first cache line of first page */
- prefetch(page_addr);
-#if L1_CACHE_BYTES < 128
- prefetch(page_addr + L1_CACHE_BYTES);
-#endif
-
- /* allocate a skb to store the frags */
- skb = __napi_alloc_skb(&rx_ring->q_vector->napi,
- I40E_RX_HDR_SIZE,
- GFP_ATOMIC | __GFP_NOWARN);
- if (unlikely(!skb)) {
- rx_ring->rx_stats.alloc_buff_failed++;
- return NULL;
- }
-
- /* we will be copying header into skb->data in
- * pskb_may_pull so it is in our interest to prefetch
- * it now to avoid a possible cache miss
- */
- prefetchw(skb->data);
- }
+ prefetchw(rx_buffer->page);
/* we are reusing so sync this buffer for CPU use */
dma_sync_single_range_for_cpu(rx_ring->dev,
@@ -1678,21 +1766,148 @@
size,
DMA_FROM_DEVICE);
- /* pull page into skb */
- if (i40e_add_rx_frag(rx_ring, rx_buffer, size, skb)) {
+ /* We have pulled a buffer for use, so decrement pagecnt_bias */
+ rx_buffer->pagecnt_bias--;
+
+ return rx_buffer;
+}
+
+/**
+ * i40e_construct_skb - Allocate skb and populate it
+ * @rx_ring: rx descriptor ring to transact packets on
+ * @rx_buffer: rx buffer to pull data from
+ * @size: size of buffer to add to skb
+ *
+ * This function allocates an skb. It then populates it with the page
+ * data from the current receive descriptor, taking care to set up the
+ * skb correctly.
+ */
+static struct sk_buff *i40e_construct_skb(struct i40e_ring *rx_ring,
+ struct i40e_rx_buffer *rx_buffer,
+ unsigned int size)
+{
+ void *va = page_address(rx_buffer->page) + rx_buffer->page_offset;
+#if (PAGE_SIZE < 8192)
+ unsigned int truesize = i40e_rx_pg_size(rx_ring) / 2;
+#else
+ unsigned int truesize = SKB_DATA_ALIGN(size);
+#endif
+ unsigned int headlen;
+ struct sk_buff *skb;
+
+ /* prefetch first cache line of first page */
+ prefetch(va);
+#if L1_CACHE_BYTES < 128
+ prefetch(va + L1_CACHE_BYTES);
+#endif
+
+ /* allocate a skb to store the frags */
+ skb = __napi_alloc_skb(&rx_ring->q_vector->napi,
+ I40E_RX_HDR_SIZE,
+ GFP_ATOMIC | __GFP_NOWARN);
+ if (unlikely(!skb))
+ return NULL;
+
+ /* Determine available headroom for copy */
+ headlen = size;
+ if (headlen > I40E_RX_HDR_SIZE)
+ headlen = eth_get_headlen(va, I40E_RX_HDR_SIZE);
+
+ /* align pull length to size of long to optimize memcpy performance */
+ memcpy(__skb_put(skb, headlen), va, ALIGN(headlen, sizeof(long)));
+
+ /* update all of the pointers */
+ size -= headlen;
+ if (size) {
+ skb_add_rx_frag(skb, 0, rx_buffer->page,
+ rx_buffer->page_offset + headlen,
+ size, truesize);
+
+ /* buffer is used by skb, update page_offset */
+#if (PAGE_SIZE < 8192)
+ rx_buffer->page_offset ^= truesize;
+#else
+ rx_buffer->page_offset += truesize;
+#endif
+ } else {
+ /* buffer is unused, reset bias back to rx_buffer */
+ rx_buffer->pagecnt_bias++;
+ }
+
+ return skb;
+}
+
+/**
+ * i40e_build_skb - Build skb around an existing buffer
+ * @rx_ring: Rx descriptor ring to transact packets on
+ * @rx_buffer: Rx buffer to pull data from
+ * @size: size of buffer to add to skb
+ *
+ * This function builds an skb around an existing Rx buffer, taking care
+ * to set up the skb correctly and avoid any memcpy overhead.
+ */
+static struct sk_buff *i40e_build_skb(struct i40e_ring *rx_ring,
+ struct i40e_rx_buffer *rx_buffer,
+ unsigned int size)
+{
+ void *va = page_address(rx_buffer->page) + rx_buffer->page_offset;
+#if (PAGE_SIZE < 8192)
+ unsigned int truesize = i40e_rx_pg_size(rx_ring) / 2;
+#else
+ unsigned int truesize = SKB_DATA_ALIGN(size);
+#endif
+ struct sk_buff *skb;
+
+ /* prefetch first cache line of first page */
+ prefetch(va);
+#if L1_CACHE_BYTES < 128
+ prefetch(va + L1_CACHE_BYTES);
+#endif
+ /* build an skb around the page buffer */
+ skb = build_skb(va - I40E_SKB_PAD, truesize);
+ if (unlikely(!skb))
+ return NULL;
+
+ /* update pointers within the skb to store the data */
+ skb_reserve(skb, I40E_SKB_PAD);
+ __skb_put(skb, size);
+
+ /* buffer is used by skb, update page_offset */
+#if (PAGE_SIZE < 8192)
+ rx_buffer->page_offset ^= truesize;
+#else
+ rx_buffer->page_offset += truesize;
+#endif
+
+ return skb;
+}
+
+/**
+ * i40e_put_rx_buffer - Clean up used buffer and either recycle or free
+ * @rx_ring: rx descriptor ring to transact packets on
+ * @rx_buffer: rx buffer to pull data from
+ *
+ * This function will clean up the contents of the rx_buffer. It will
+ * either recycle the bufer or unmap it and free the associated resources.
+ */
+static void i40e_put_rx_buffer(struct i40e_ring *rx_ring,
+ struct i40e_rx_buffer *rx_buffer)
+{
+ if (i40e_can_reuse_rx_page(rx_buffer)) {
/* hand second half of page back to the ring */
i40e_reuse_rx_page(rx_ring, rx_buffer);
rx_ring->rx_stats.page_reuse_count++;
} else {
/* we are not reusing the buffer so unmap it */
- dma_unmap_page(rx_ring->dev, rx_buffer->dma, PAGE_SIZE,
- DMA_FROM_DEVICE);
+ dma_unmap_page_attrs(rx_ring->dev, rx_buffer->dma,
+ i40e_rx_pg_size(rx_ring),
+ DMA_FROM_DEVICE, I40E_RX_DMA_ATTR);
+ __page_frag_cache_drain(rx_buffer->page,
+ rx_buffer->pagecnt_bias);
}
/* clear contents of buffer_info */
rx_buffer->page = NULL;
-
- return skb;
}
/**
@@ -1718,11 +1933,6 @@
prefetch(I40E_RX_DESC(rx_ring, ntc));
-#define staterrlen rx_desc->wb.qword1.status_error_len
- if (unlikely(i40e_rx_is_programming_status(le64_to_cpu(staterrlen)))) {
- i40e_clean_programming_status(rx_ring, rx_desc);
- return true;
- }
/* if we are the last buffer then there is nothing else to do */
#define I40E_RXD_EOF BIT(I40E_RX_DESC_STATUS_EOF_SHIFT)
if (likely(i40e_test_staterr(rx_desc, I40E_RXD_EOF)))
@@ -1753,7 +1963,9 @@
bool failure = false;
while (likely(total_rx_packets < budget)) {
+ struct i40e_rx_buffer *rx_buffer;
union i40e_rx_desc *rx_desc;
+ unsigned int size;
u16 vlan_tag;
u8 rx_ptype;
u64 qword;
@@ -1770,22 +1982,44 @@
/* status_error_len will always be zero for unused descriptors
* because it's cleared in cleanup, and overlaps with hdr_addr
* which is always zero because packet split isn't used, if the
- * hardware wrote DD then it will be non-zero
+ * hardware wrote DD then the length will be non-zero
*/
- if (!i40e_test_staterr(rx_desc,
- BIT(I40E_RX_DESC_STATUS_DD_SHIFT)))
- break;
+ qword = le64_to_cpu(rx_desc->wb.qword1.status_error_len);
/* This memory barrier is needed to keep us from reading
- * any other fields out of the rx_desc until we know the
- * DD bit is set.
+ * any other fields out of the rx_desc until we have
+ * verified the descriptor has been written back.
*/
dma_rmb();
- skb = i40e_fetch_rx_buffer(rx_ring, rx_desc, skb);
- if (!skb)
+ if (unlikely(i40e_rx_is_programming_status(qword))) {
+ i40e_clean_programming_status(rx_ring, rx_desc, qword);
+ continue;
+ }
+ size = (qword & I40E_RXD_QW1_LENGTH_PBUF_MASK) >>
+ I40E_RXD_QW1_LENGTH_PBUF_SHIFT;
+ if (!size)
break;
+ i40e_trace(clean_rx_irq, rx_ring, rx_desc, skb);
+ rx_buffer = i40e_get_rx_buffer(rx_ring, size);
+
+ /* retrieve a buffer from the ring */
+ if (skb)
+ i40e_add_rx_frag(rx_ring, rx_buffer, skb, size);
+ else if (ring_uses_build_skb(rx_ring))
+ skb = i40e_build_skb(rx_ring, rx_buffer, size);
+ else
+ skb = i40e_construct_skb(rx_ring, rx_buffer, size);
+
+ /* exit if we failed to retrieve a buffer */
+ if (!skb) {
+ rx_ring->rx_stats.alloc_buff_failed++;
+ rx_buffer->pagecnt_bias++;
+ break;
+ }
+
+ i40e_put_rx_buffer(rx_ring, rx_buffer);
cleaned_count++;
if (i40e_is_non_eop(rx_ring, rx_desc, skb))
@@ -1798,6 +2032,7 @@
*/
if (unlikely(i40e_test_staterr(rx_desc, BIT(I40E_RXD_QW1_ERROR_SHIFT)))) {
dev_kfree_skb_any(skb);
+ skb = NULL;
continue;
}
@@ -1816,18 +2051,10 @@
/* populate checksum, VLAN, and protocol */
i40e_process_skb_fields(rx_ring, rx_desc, skb, rx_ptype);
-#ifdef I40E_FCOE
- if (unlikely(
- i40e_rx_is_fcoe(rx_ptype) &&
- !i40e_fcoe_handle_offload(rx_ring, rx_desc, skb))) {
- dev_kfree_skb_any(skb);
- continue;
- }
-#endif
-
vlan_tag = (qword & BIT(I40E_RX_DESC_STATUS_L2TAG1P_SHIFT)) ?
le16_to_cpu(rx_desc->wb.qword0.lo_dword.l2tag1) : 0;
+ i40e_trace(clean_rx_irq_rx, rx_ring, rx_desc, skb);
i40e_receive_skb(rx_ring, skb, vlan_tag);
skb = NULL;
@@ -1944,7 +2171,7 @@
}
enable_int:
- if (!test_bit(__I40E_DOWN, &vsi->state))
+ if (!test_bit(__I40E_VSI_DOWN, vsi->state))
wr32(hw, INTREG(vector - 1), txval);
if (q_vector->itr_countdown)
@@ -1973,13 +2200,11 @@
int budget_per_ring;
int work_done = 0;
- if (test_bit(__I40E_DOWN, &vsi->state)) {
+ if (test_bit(__I40E_VSI_DOWN, vsi->state)) {
napi_complete(napi);
return 0;
}
- /* Clear hung_detected bit */
- clear_bit(I40E_Q_VECTOR_HUNG_DETECT, &q_vector->hung_detected);
/* Since the actual Tx work is minimal, we can give the Tx a larger
* budget and be more aggressive about cleaning up the Tx descriptors.
*/
@@ -2079,7 +2304,7 @@
if (!(pf->flags & I40E_FLAG_FD_ATR_ENABLED))
return;
- if ((pf->auto_disable_flags & I40E_FLAG_FD_ATR_ENABLED))
+ if (pf->flags & I40E_FLAG_FD_ATR_AUTO_DISABLED)
return;
/* if sampling is disabled do nothing */
@@ -2113,10 +2338,9 @@
th = (struct tcphdr *)(hdr.network + hlen);
/* Due to lack of space, no more new filters can be programmed */
- if (th->syn && (pf->auto_disable_flags & I40E_FLAG_FD_ATR_ENABLED))
+ if (th->syn && (pf->flags & I40E_FLAG_FD_ATR_AUTO_DISABLED))
return;
- if ((pf->flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE) &&
- (!(pf->auto_disable_flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE))) {
+ if (pf->flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE) {
/* HW ATR eviction will take care of removing filters on FIN
* and RST packets.
*/
@@ -2178,8 +2402,7 @@
I40E_TXD_FLTR_QW1_CNTINDEX_SHIFT) &
I40E_TXD_FLTR_QW1_CNTINDEX_MASK;
- if ((pf->flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE) &&
- (!(pf->auto_disable_flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE)))
+ if (pf->flags & I40E_FLAG_HW_ATR_EVICT_CAPABLE)
dtype_cmd |= I40E_TXD_FLTR_QW1_ATR_MASK;
fdir_desc->qindex_flex_ptype_vsi = cpu_to_le32(flex_ptype);
@@ -2200,15 +2423,9 @@
* Returns error code indicate the frame should be dropped upon error and the
* otherwise returns 0 to indicate the flags has been set properly.
**/
-#ifdef I40E_FCOE
-inline int i40e_tx_prepare_vlan_flags(struct sk_buff *skb,
- struct i40e_ring *tx_ring,
- u32 *flags)
-#else
static inline int i40e_tx_prepare_vlan_flags(struct sk_buff *skb,
struct i40e_ring *tx_ring,
u32 *flags)
-#endif
{
__be16 protocol = skb->protocol;
u32 tx_flags = 0;
@@ -2409,7 +2626,7 @@
return 0;
if (pf->ptp_tx &&
- !test_and_set_bit_lock(__I40E_PTP_TX_IN_PROGRESS, &pf->state)) {
+ !test_and_set_bit_lock(__I40E_PTP_TX_IN_PROGRESS, pf->state)) {
skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
pf->ptp_tx_skb = skb_get(skb);
} else {
@@ -2716,15 +2933,9 @@
* @td_cmd: the command field in the descriptor
* @td_offset: offset for checksum or crc
**/
-#ifdef I40E_FCOE
-inline void i40e_tx_map(struct i40e_ring *tx_ring, struct sk_buff *skb,
- struct i40e_tx_buffer *first, u32 tx_flags,
- const u8 hdr_len, u32 td_cmd, u32 td_offset)
-#else
static inline void i40e_tx_map(struct i40e_ring *tx_ring, struct sk_buff *skb,
struct i40e_tx_buffer *first, u32 tx_flags,
const u8 hdr_len, u32 td_cmd, u32 td_offset)
-#endif
{
unsigned int data_len = skb->data_len;
unsigned int size = skb_headlen(skb);
@@ -2925,6 +3136,8 @@
/* prefetch the data, we'll need it later */
prefetch(skb->data);
+ i40e_trace(xmit_frame_ring, skb, tx_ring);
+
count = i40e_xmit_descriptor_count(skb);
if (i40e_chk_linearize(skb, count)) {
if (__skb_linearize(skb)) {
@@ -3003,6 +3216,7 @@
return NETDEV_TX_OK;
out_drop:
+ i40e_trace(xmit_frame_ring_drop, first->skb, tx_ring);
dev_kfree_skb_any(first->skb);
first->skb = NULL;
return NETDEV_TX_OK;
diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.h b/drivers/net/ethernet/intel/i40e/i40e_txrx.h
index f809790..f5de511 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_txrx.h
+++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.h
@@ -117,10 +117,9 @@
/* Supported Rx Buffer Sizes (a multiple of 128) */
#define I40E_RXBUFFER_256 256
+#define I40E_RXBUFFER_1536 1536 /* 128B aligned standard Ethernet frame */
#define I40E_RXBUFFER_2048 2048
-#define I40E_RXBUFFER_3072 3072 /* For FCoE MTU of 2158 */
-#define I40E_RXBUFFER_4096 4096
-#define I40E_RXBUFFER_8192 8192
+#define I40E_RXBUFFER_3072 3072 /* Used for large frames w/ padding */
#define I40E_MAX_RXBUFFER 9728 /* largest size for single descriptor */
/* NOTE: netdev_alloc_skb reserves up to 64 bytes, NET_IP_ALIGN means we
@@ -133,6 +132,61 @@
#define I40E_RX_HDR_SIZE I40E_RXBUFFER_256
#define i40e_rx_desc i40e_32byte_rx_desc
+#define I40E_RX_DMA_ATTR \
+ (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING)
+
+/* Attempt to maximize the headroom available for incoming frames. We
+ * use a 2K buffer for receives and need 1536/1534 to store the data for
+ * the frame. This leaves us with 512 bytes of room. From that we need
+ * to deduct the space needed for the shared info and the padding needed
+ * to IP align the frame.
+ *
+ * Note: For cache line sizes 256 or larger this value is going to end
+ * up negative. In these cases we should fall back to the legacy
+ * receive path.
+ */
+#if (PAGE_SIZE < 8192)
+#define I40E_2K_TOO_SMALL_WITH_PADDING \
+((NET_SKB_PAD + I40E_RXBUFFER_1536) > SKB_WITH_OVERHEAD(I40E_RXBUFFER_2048))
+
+static inline int i40e_compute_pad(int rx_buf_len)
+{
+ int page_size, pad_size;
+
+ page_size = ALIGN(rx_buf_len, PAGE_SIZE / 2);
+ pad_size = SKB_WITH_OVERHEAD(page_size) - rx_buf_len;
+
+ return pad_size;
+}
+
+static inline int i40e_skb_pad(void)
+{
+ int rx_buf_len;
+
+ /* If a 2K buffer cannot handle a standard Ethernet frame then
+ * optimize padding for a 3K buffer instead of a 1.5K buffer.
+ *
+ * For a 3K buffer we need to add enough padding to allow for
+ * tailroom due to NET_IP_ALIGN possibly shifting us out of
+ * cache-line alignment.
+ */
+ if (I40E_2K_TOO_SMALL_WITH_PADDING)
+ rx_buf_len = I40E_RXBUFFER_3072 + SKB_DATA_ALIGN(NET_IP_ALIGN);
+ else
+ rx_buf_len = I40E_RXBUFFER_1536;
+
+ /* if needed make room for NET_IP_ALIGN */
+ rx_buf_len -= NET_IP_ALIGN;
+
+ return i40e_compute_pad(rx_buf_len);
+}
+
+#define I40E_SKB_PAD i40e_skb_pad()
+#else
+#define I40E_2K_TOO_SMALL_WITH_PADDING false
+#define I40E_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN)
+#endif
+
/**
* i40e_test_staterr - tests bits in Rx descriptor status and error fields
* @rx_desc: pointer to receive descriptor (in le64 format)
@@ -255,7 +309,12 @@
struct i40e_rx_buffer {
dma_addr_t dma;
struct page *page;
- unsigned int page_offset;
+#if (BITS_PER_LONG > 32) || (PAGE_SIZE >= 65536)
+ __u32 page_offset;
+#else
+ __u16 page_offset;
+#endif
+ __u16 pagecnt_bias;
};
struct i40e_queue_stats {
@@ -269,7 +328,6 @@
u64 tx_done_old;
u64 tx_linearize;
u64 tx_force_wb;
- u64 tx_lost_interrupt;
};
struct i40e_rx_queue_stats {
@@ -335,7 +393,8 @@
u8 packet_stride;
u16 flags;
-#define I40E_TXR_FLAGS_WB_ON_ITR BIT(0)
+#define I40E_TXR_FLAGS_WB_ON_ITR BIT(0)
+#define I40E_RXR_FLAGS_BUILD_SKB_ENABLED BIT(1)
/* stats structs */
struct i40e_queue_stats stats;
@@ -363,6 +422,21 @@
*/
} ____cacheline_internodealigned_in_smp;
+static inline bool ring_uses_build_skb(struct i40e_ring *ring)
+{
+ return !!(ring->flags & I40E_RXR_FLAGS_BUILD_SKB_ENABLED);
+}
+
+static inline void set_ring_build_skb_enabled(struct i40e_ring *ring)
+{
+ ring->flags |= I40E_RXR_FLAGS_BUILD_SKB_ENABLED;
+}
+
+static inline void clear_ring_build_skb_enabled(struct i40e_ring *ring)
+{
+ ring->flags &= ~I40E_RXR_FLAGS_BUILD_SKB_ENABLED;
+}
+
enum i40e_latency_range {
I40E_LOWEST_LATENCY = 0,
I40E_LOW_LATENCY = 1,
@@ -384,6 +458,17 @@
#define i40e_for_each_ring(pos, head) \
for (pos = (head).ring; pos != NULL; pos = pos->next)
+static inline unsigned int i40e_rx_pg_order(struct i40e_ring *ring)
+{
+#if (PAGE_SIZE < 8192)
+ if (ring->rx_buf_len > (PAGE_SIZE / 2))
+ return 1;
+#endif
+ return 0;
+}
+
+#define i40e_rx_pg_size(_ring) (PAGE_SIZE << i40e_rx_pg_order(_ring))
+
bool i40e_alloc_rx_buffers(struct i40e_ring *rxr, u16 cleaned_count);
netdev_tx_t i40e_lan_xmit_frame(struct sk_buff *skb, struct net_device *netdev);
void i40e_clean_tx_ring(struct i40e_ring *tx_ring);
@@ -393,15 +478,8 @@
void i40e_free_tx_resources(struct i40e_ring *tx_ring);
void i40e_free_rx_resources(struct i40e_ring *rx_ring);
int i40e_napi_poll(struct napi_struct *napi, int budget);
-#ifdef I40E_FCOE
-void i40e_tx_map(struct i40e_ring *tx_ring, struct sk_buff *skb,
- struct i40e_tx_buffer *first, u32 tx_flags,
- const u8 hdr_len, u32 td_cmd, u32 td_offset);
-int i40e_tx_prepare_vlan_flags(struct sk_buff *skb,
- struct i40e_ring *tx_ring, u32 *flags);
-#endif
void i40e_force_wb(struct i40e_vsi *vsi, struct i40e_q_vector *q_vector);
-u32 i40e_get_tx_pending(struct i40e_ring *ring, bool in_sw);
+u32 i40e_get_tx_pending(struct i40e_ring *ring);
int __i40e_maybe_stop_tx(struct i40e_ring *tx_ring, int size);
bool __i40e_chk_linearize(struct sk_buff *skb);
@@ -483,16 +561,6 @@
}
/**
- * i40e_rx_is_fcoe - returns true if the Rx packet type is FCoE
- * @ptype: the packet type field from Rx descriptor write-back
- **/
-static inline bool i40e_rx_is_fcoe(u16 ptype)
-{
- return (ptype >= I40E_RX_PTYPE_L2_FCOE_PAY3) &&
- (ptype <= I40E_RX_PTYPE_L2_FCOE_VFT_FCOTHER);
-}
-
-/**
* txring_txq - Find the netdev Tx ring based on the i40e Tx ring
* @ring: Tx ring to find the netdev equivalent of
**/
diff --git a/drivers/net/ethernet/intel/i40e/i40e_type.h b/drivers/net/ethernet/intel/i40e/i40e_type.h
index 939f9fd..3a18ed1 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_type.h
+++ b/drivers/net/ethernet/intel/i40e/i40e_type.h
@@ -78,6 +78,7 @@
I40E_DEBUG_DCB = 0x00000400,
I40E_DEBUG_DIAG = 0x00000800,
I40E_DEBUG_FD = 0x00001000,
+ I40E_DEBUG_PACKAGE = 0x00002000,
I40E_DEBUG_IWARP = 0x00F00000,
I40E_DEBUG_AQ_MESSAGE = 0x01000000,
I40E_DEBUG_AQ_DESCRIPTOR = 0x02000000,
@@ -1213,25 +1214,6 @@
u64 tc_tx_bytes[I40E_MAX_TRAFFIC_CLASS];
};
-#ifdef I40E_FCOE
-/* Statistics collected per function for FCoE */
-struct i40e_fcoe_stats {
- u64 rx_fcoe_packets; /* fcoeprc */
- u64 rx_fcoe_dwords; /* focedwrc */
- u64 rx_fcoe_dropped; /* fcoerpdc */
- u64 tx_fcoe_packets; /* fcoeptc */
- u64 tx_fcoe_dwords; /* focedwtc */
- u64 fcoe_bad_fccrc; /* fcoecrc */
- u64 fcoe_last_error; /* fcoelast */
- u64 fcoe_ddp_count; /* fcoeddpc */
-};
-
-/* offset to per function FCoE statistics block */
-#define I40E_FCOE_VF_STAT_OFFSET 0
-#define I40E_FCOE_PF_STAT_OFFSET 128
-#define I40E_FCOE_STAT_MAX (I40E_FCOE_PF_STAT_OFFSET + I40E_MAX_PF)
-
-#endif
/* Statistics collected by the MAC */
struct i40e_hw_port_stats {
/* eth stats collected by the port */
@@ -1319,125 +1301,6 @@
#define I40E_SRRD_SRCTL_ATTEMPTS 100000
-#ifdef I40E_FCOE
-/* FCoE Tx context descriptor - Use the i40e_tx_context_desc struct */
-
-enum i40E_fcoe_tx_ctx_desc_cmd_bits {
- I40E_FCOE_TX_CTX_DESC_OPCODE_SINGLE_SEND = 0x00, /* 4 BITS */
- I40E_FCOE_TX_CTX_DESC_OPCODE_TSO_FC_CLASS2 = 0x01, /* 4 BITS */
- I40E_FCOE_TX_CTX_DESC_OPCODE_TSO_FC_CLASS3 = 0x05, /* 4 BITS */
- I40E_FCOE_TX_CTX_DESC_OPCODE_ETSO_FC_CLASS2 = 0x02, /* 4 BITS */
- I40E_FCOE_TX_CTX_DESC_OPCODE_ETSO_FC_CLASS3 = 0x06, /* 4 BITS */
- I40E_FCOE_TX_CTX_DESC_OPCODE_DWO_FC_CLASS2 = 0x03, /* 4 BITS */
- I40E_FCOE_TX_CTX_DESC_OPCODE_DWO_FC_CLASS3 = 0x07, /* 4 BITS */
- I40E_FCOE_TX_CTX_DESC_OPCODE_DDP_CTX_INVL = 0x08, /* 4 BITS */
- I40E_FCOE_TX_CTX_DESC_OPCODE_DWO_CTX_INVL = 0x09, /* 4 BITS */
- I40E_FCOE_TX_CTX_DESC_RELOFF = 0x10,
- I40E_FCOE_TX_CTX_DESC_CLRSEQ = 0x20,
- I40E_FCOE_TX_CTX_DESC_DIFENA = 0x40,
- I40E_FCOE_TX_CTX_DESC_IL2TAG2 = 0x80
-};
-
-/* FCoE DDP Context descriptor */
-struct i40e_fcoe_ddp_context_desc {
- __le64 rsvd;
- __le64 type_cmd_foff_lsize;
-};
-
-#define I40E_FCOE_DDP_CTX_QW1_DTYPE_SHIFT 0
-#define I40E_FCOE_DDP_CTX_QW1_DTYPE_MASK (0xFULL << \
- I40E_FCOE_DDP_CTX_QW1_DTYPE_SHIFT)
-
-#define I40E_FCOE_DDP_CTX_QW1_CMD_SHIFT 4
-#define I40E_FCOE_DDP_CTX_QW1_CMD_MASK (0xFULL << \
- I40E_FCOE_DDP_CTX_QW1_CMD_SHIFT)
-
-enum i40e_fcoe_ddp_ctx_desc_cmd_bits {
- I40E_FCOE_DDP_CTX_DESC_BSIZE_512B = 0x00, /* 2 BITS */
- I40E_FCOE_DDP_CTX_DESC_BSIZE_4K = 0x01, /* 2 BITS */
- I40E_FCOE_DDP_CTX_DESC_BSIZE_8K = 0x02, /* 2 BITS */
- I40E_FCOE_DDP_CTX_DESC_BSIZE_16K = 0x03, /* 2 BITS */
- I40E_FCOE_DDP_CTX_DESC_DIFENA = 0x04, /* 1 BIT */
- I40E_FCOE_DDP_CTX_DESC_LASTSEQH = 0x08, /* 1 BIT */
-};
-
-#define I40E_FCOE_DDP_CTX_QW1_FOFF_SHIFT 16
-#define I40E_FCOE_DDP_CTX_QW1_FOFF_MASK (0x3FFFULL << \
- I40E_FCOE_DDP_CTX_QW1_FOFF_SHIFT)
-
-#define I40E_FCOE_DDP_CTX_QW1_LSIZE_SHIFT 32
-#define I40E_FCOE_DDP_CTX_QW1_LSIZE_MASK (0x3FFFULL << \
- I40E_FCOE_DDP_CTX_QW1_LSIZE_SHIFT)
-
-/* FCoE DDP/DWO Queue Context descriptor */
-struct i40e_fcoe_queue_context_desc {
- __le64 dmaindx_fbase; /* 0:11 DMAINDX, 12:63 FBASE */
- __le64 flen_tph; /* 0:12 FLEN, 13:15 TPH */
-};
-
-#define I40E_FCOE_QUEUE_CTX_QW0_DMAINDX_SHIFT 0
-#define I40E_FCOE_QUEUE_CTX_QW0_DMAINDX_MASK (0xFFFULL << \
- I40E_FCOE_QUEUE_CTX_QW0_DMAINDX_SHIFT)
-
-#define I40E_FCOE_QUEUE_CTX_QW0_FBASE_SHIFT 12
-#define I40E_FCOE_QUEUE_CTX_QW0_FBASE_MASK (0xFFFFFFFFFFFFFULL << \
- I40E_FCOE_QUEUE_CTX_QW0_FBASE_SHIFT)
-
-#define I40E_FCOE_QUEUE_CTX_QW1_FLEN_SHIFT 0
-#define I40E_FCOE_QUEUE_CTX_QW1_FLEN_MASK (0x1FFFULL << \
- I40E_FCOE_QUEUE_CTX_QW1_FLEN_SHIFT)
-
-#define I40E_FCOE_QUEUE_CTX_QW1_TPH_SHIFT 13
-#define I40E_FCOE_QUEUE_CTX_QW1_TPH_MASK (0x7ULL << \
- I40E_FCOE_QUEUE_CTX_QW1_FLEN_SHIFT)
-
-enum i40e_fcoe_queue_ctx_desc_tph_bits {
- I40E_FCOE_QUEUE_CTX_DESC_TPHRDESC = 0x1,
- I40E_FCOE_QUEUE_CTX_DESC_TPHDATA = 0x2
-};
-
-#define I40E_FCOE_QUEUE_CTX_QW1_RECIPE_SHIFT 30
-#define I40E_FCOE_QUEUE_CTX_QW1_RECIPE_MASK (0x3ULL << \
- I40E_FCOE_QUEUE_CTX_QW1_RECIPE_SHIFT)
-
-/* FCoE DDP/DWO Filter Context descriptor */
-struct i40e_fcoe_filter_context_desc {
- __le32 param;
- __le16 seqn;
-
- /* 48:51(0:3) RSVD, 52:63(4:15) DMAINDX */
- __le16 rsvd_dmaindx;
-
- /* 0:7 FLAGS, 8:52 RSVD, 53:63 LANQ */
- __le64 flags_rsvd_lanq;
-};
-
-#define I40E_FCOE_FILTER_CTX_QW0_DMAINDX_SHIFT 4
-#define I40E_FCOE_FILTER_CTX_QW0_DMAINDX_MASK (0xFFF << \
- I40E_FCOE_FILTER_CTX_QW0_DMAINDX_SHIFT)
-
-enum i40e_fcoe_filter_ctx_desc_flags_bits {
- I40E_FCOE_FILTER_CTX_DESC_CTYP_DDP = 0x00,
- I40E_FCOE_FILTER_CTX_DESC_CTYP_DWO = 0x01,
- I40E_FCOE_FILTER_CTX_DESC_ENODE_INIT = 0x00,
- I40E_FCOE_FILTER_CTX_DESC_ENODE_RSP = 0x02,
- I40E_FCOE_FILTER_CTX_DESC_FC_CLASS2 = 0x00,
- I40E_FCOE_FILTER_CTX_DESC_FC_CLASS3 = 0x04
-};
-
-#define I40E_FCOE_FILTER_CTX_QW1_FLAGS_SHIFT 0
-#define I40E_FCOE_FILTER_CTX_QW1_FLAGS_MASK (0xFFULL << \
- I40E_FCOE_FILTER_CTX_QW1_FLAGS_SHIFT)
-
-#define I40E_FCOE_FILTER_CTX_QW1_PCTYPE_SHIFT 8
-#define I40E_FCOE_FILTER_CTX_QW1_PCTYPE_MASK (0x3FULL << \
- I40E_FCOE_FILTER_CTX_QW1_PCTYPE_SHIFT)
-
-#define I40E_FCOE_FILTER_CTX_QW1_LANQINDX_SHIFT 53
-#define I40E_FCOE_FILTER_CTX_QW1_LANQINDX_MASK (0x7FFULL << \
- I40E_FCOE_FILTER_CTX_QW1_LANQINDX_SHIFT)
-
-#endif /* I40E_FCOE */
enum i40e_switch_element_types {
I40E_SWITCH_ELEMENT_TYPE_MAC = 1,
I40E_SWITCH_ELEMENT_TYPE_PF = 2,
@@ -1600,4 +1463,83 @@
#define I40E_FLEX_56_MASK (0x1ULL << I40E_FLEX_56_SHIFT)
#define I40E_FLEX_57_SHIFT 6
#define I40E_FLEX_57_MASK (0x1ULL << I40E_FLEX_57_SHIFT)
+
+/* Version format for PPP */
+struct i40e_ppp_version {
+ u8 major;
+ u8 minor;
+ u8 update;
+ u8 draft;
+};
+
+#define I40E_PPP_NAME_SIZE 32
+
+/* Package header */
+struct i40e_package_header {
+ struct i40e_ppp_version version;
+ u32 segment_count;
+ u32 segment_offset[1];
+};
+
+/* Generic segment header */
+struct i40e_generic_seg_header {
+#define SEGMENT_TYPE_METADATA 0x00000001
+#define SEGMENT_TYPE_NOTES 0x00000002
+#define SEGMENT_TYPE_I40E 0x00000011
+#define SEGMENT_TYPE_X722 0x00000012
+ u32 type;
+ struct i40e_ppp_version version;
+ u32 size;
+ char name[I40E_PPP_NAME_SIZE];
+};
+
+struct i40e_metadata_segment {
+ struct i40e_generic_seg_header header;
+ struct i40e_ppp_version version;
+ u32 track_id;
+ char name[I40E_PPP_NAME_SIZE];
+};
+
+struct i40e_device_id_entry {
+ u32 vendor_dev_id;
+ u32 sub_vendor_dev_id;
+};
+
+struct i40e_profile_segment {
+ struct i40e_generic_seg_header header;
+ struct i40e_ppp_version version;
+ char name[I40E_PPP_NAME_SIZE];
+ u32 device_table_count;
+ struct i40e_device_id_entry device_table[1];
+};
+
+struct i40e_section_table {
+ u32 section_count;
+ u32 section_offset[1];
+};
+
+struct i40e_profile_section_header {
+ u16 tbl_size;
+ u16 data_end;
+ struct {
+#define SECTION_TYPE_INFO 0x00000010
+#define SECTION_TYPE_MMIO 0x00000800
+#define SECTION_TYPE_AQ 0x00000801
+#define SECTION_TYPE_NOTE 0x80000000
+#define SECTION_TYPE_NAME 0x80000001
+ u32 type;
+ u32 offset;
+ u32 size;
+ } section;
+};
+
+struct i40e_profile_info {
+ u32 track_id;
+ struct i40e_ppp_version version;
+ u8 op;
+#define I40E_PPP_ADD_TRACKID 0x01
+#define I40E_PPP_REMOVE_TRACKID 0x02
+ u8 reserved[7];
+ u8 name[I40E_PPP_NAME_SIZE];
+};
#endif /* _I40E_TYPE_H_ */
diff --git a/drivers/net/ethernet/intel/i40e/i40e_virtchnl.h b/drivers/net/ethernet/intel/i40e/i40e_virtchnl.h
index 974ba2b..8552192 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_virtchnl.h
+++ b/drivers/net/ethernet/intel/i40e/i40e_virtchnl.h
@@ -163,7 +163,8 @@
#define I40E_VIRTCHNL_VF_OFFLOAD_RX_POLLING 0x00020000
#define I40E_VIRTCHNL_VF_OFFLOAD_RSS_PCTYPE_V2 0x00040000
#define I40E_VIRTCHNL_VF_OFFLOAD_RSS_PF 0X00080000
-#define I40E_VIRTCHNL_VF_OFFLOAD_ENCAP_CSUM 0X00100000
+#define I40E_VIRTCHNL_VF_OFFLOAD_ENCAP 0X00100000
+#define I40E_VIRTCHNL_VF_OFFLOAD_ENCAP_CSUM 0X00200000
#define I40E_VF_BASE_MODE_OFFLOADS (I40E_VIRTCHNL_VF_OFFLOAD_L2 | \
I40E_VIRTCHNL_VF_OFFLOAD_VLAN | \
diff --git a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c
index 78460c5..95c23fb 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c
@@ -50,8 +50,8 @@
for (i = 0; i < pf->num_alloc_vfs; i++, vf++) {
int abs_vf_id = vf->vf_id + (int)hw->func_caps.vf_base_id;
/* Not all vfs are enabled so skip the ones that are not */
- if (!test_bit(I40E_VF_STAT_INIT, &vf->vf_states) &&
- !test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states))
+ if (!test_bit(I40E_VF_STATE_INIT, &vf->vf_states) &&
+ !test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states))
continue;
/* Ignore return value on purpose - a given VF may fail, but
@@ -137,8 +137,8 @@
return;
/* verify if the VF is in either init or active before proceeding */
- if (!test_bit(I40E_VF_STAT_INIT, &vf->vf_states) &&
- !test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states))
+ if (!test_bit(I40E_VF_STATE_INIT, &vf->vf_states) &&
+ !test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states))
return;
abs_vf_id = vf->vf_id + (int)vf->pf->hw.func_caps.vf_base_id;
@@ -702,10 +702,8 @@
dev_info(&pf->pdev->dev,
"Could not allocate VF broadcast filter\n");
spin_unlock_bh(&vsi->mac_filter_hash_lock);
- i40e_write_rx_ctl(&pf->hw, I40E_VFQF_HENA1(0, vf->vf_id),
- (u32)hena);
- i40e_write_rx_ctl(&pf->hw, I40E_VFQF_HENA1(1, vf->vf_id),
- (u32)(hena >> 32));
+ wr32(&pf->hw, I40E_VFQF_HENA1(0, vf->vf_id), (u32)hena);
+ wr32(&pf->hw, I40E_VFQF_HENA1(1, vf->vf_id), (u32)(hena >> 32));
}
/* program mac filter */
@@ -811,6 +809,11 @@
u32 reg_idx, reg;
int i, msix_vf;
+ /* Start by disabling VF's configuration API to prevent the OS from
+ * accessing the VF's VSI after it's freed / invalidated.
+ */
+ clear_bit(I40E_VF_STATE_INIT, &vf->vf_states);
+
/* free vsi & disconnect it from the parent uplink */
if (vf->lan_vsi_idx) {
i40e_vsi_release(pf->vsi[vf->lan_vsi_idx]);
@@ -850,7 +853,6 @@
/* reset some of the state variables keeping track of the resources */
vf->num_queue_pairs = 0;
vf->vf_states = 0;
- clear_bit(I40E_VF_STAT_INIT, &vf->vf_states);
}
/**
@@ -882,7 +884,7 @@
vf->num_queue_pairs = total_queue_pairs;
/* VF is now completely initialized */
- set_bit(I40E_VF_STAT_INIT, &vf->vf_states);
+ set_bit(I40E_VF_STATE_INIT, &vf->vf_states);
error_alloc:
if (ret)
@@ -921,25 +923,30 @@
}
/**
- * i40e_reset_vf
+ * i40e_trigger_vf_reset
* @vf: pointer to the VF structure
* @flr: VFLR was issued or not
*
- * reset the VF
+ * Trigger hardware to start a reset for a particular VF. Expects the caller
+ * to wait the proper amount of time to allow hardware to reset the VF before
+ * it cleans up and restores VF functionality.
**/
-void i40e_reset_vf(struct i40e_vf *vf, bool flr)
+static void i40e_trigger_vf_reset(struct i40e_vf *vf, bool flr)
{
struct i40e_pf *pf = vf->pf;
struct i40e_hw *hw = &pf->hw;
u32 reg, reg_idx, bit_idx;
- bool rsd = false;
- int i;
-
- if (test_and_set_bit(__I40E_VF_DISABLE, &pf->state))
- return;
/* warn the VF */
- clear_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states);
+ clear_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states);
+
+ /* Disable VF's configuration API during reset. The flag is re-enabled
+ * in i40e_alloc_vf_res(), when it's safe again to access VF's VSI.
+ * It's normally disabled in i40e_free_vf_res(), but it's safer
+ * to do it earlier to give some time to finish to any VF config
+ * functions that may still be running at this point.
+ */
+ clear_bit(I40E_VF_STATE_INIT, &vf->vf_states);
/* In the case of a VFLR, the HW has already reset the VF and we
* just need to clean up, so don't hit the VFRTRIG register.
@@ -960,6 +967,79 @@
if (i40e_quiesce_vf_pci(vf))
dev_err(&pf->pdev->dev, "VF %d PCI transactions stuck\n",
vf->vf_id);
+}
+
+/**
+ * i40e_cleanup_reset_vf
+ * @vf: pointer to the VF structure
+ *
+ * Cleanup a VF after the hardware reset is finished. Expects the caller to
+ * have verified whether the reset is finished properly, and ensure the
+ * minimum amount of wait time has passed.
+ **/
+static void i40e_cleanup_reset_vf(struct i40e_vf *vf)
+{
+ struct i40e_pf *pf = vf->pf;
+ struct i40e_hw *hw = &pf->hw;
+ u32 reg;
+
+ /* free VF resources to begin resetting the VSI state */
+ i40e_free_vf_res(vf);
+
+ /* Enable hardware by clearing the reset bit in the VPGEN_VFRTRIG reg.
+ * By doing this we allow HW to access VF memory at any point. If we
+ * did it any sooner, HW could access memory while it was being freed
+ * in i40e_free_vf_res(), causing an IOMMU fault.
+ *
+ * On the other hand, this needs to be done ASAP, because the VF driver
+ * is waiting for this to happen and may report a timeout. It's
+ * harmless, but it gets logged into Guest OS kernel log, so best avoid
+ * it.
+ */
+ reg = rd32(hw, I40E_VPGEN_VFRTRIG(vf->vf_id));
+ reg &= ~I40E_VPGEN_VFRTRIG_VFSWR_MASK;
+ wr32(hw, I40E_VPGEN_VFRTRIG(vf->vf_id), reg);
+
+ /* reallocate VF resources to finish resetting the VSI state */
+ if (!i40e_alloc_vf_res(vf)) {
+ int abs_vf_id = vf->vf_id + hw->func_caps.vf_base_id;
+ i40e_enable_vf_mappings(vf);
+ set_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states);
+ clear_bit(I40E_VF_STATE_DISABLED, &vf->vf_states);
+ /* Do not notify the client during VF init */
+ if (test_and_clear_bit(I40E_VF_STATE_PRE_ENABLE,
+ &vf->vf_states))
+ i40e_notify_client_of_vf_reset(pf, abs_vf_id);
+ vf->num_vlan = 0;
+ }
+
+ /* Tell the VF driver the reset is done. This needs to be done only
+ * after VF has been fully initialized, because the VF driver may
+ * request resources immediately after setting this flag.
+ */
+ wr32(hw, I40E_VFGEN_RSTAT1(vf->vf_id), I40E_VFR_VFACTIVE);
+}
+
+/**
+ * i40e_reset_vf
+ * @vf: pointer to the VF structure
+ * @flr: VFLR was issued or not
+ *
+ * reset the VF
+ **/
+void i40e_reset_vf(struct i40e_vf *vf, bool flr)
+{
+ struct i40e_pf *pf = vf->pf;
+ struct i40e_hw *hw = &pf->hw;
+ bool rsd = false;
+ u32 reg;
+ int i;
+
+ /* If VFs have been disabled, there is no need to reset */
+ if (test_and_set_bit(__I40E_VF_DISABLE, pf->state))
+ return;
+
+ i40e_trigger_vf_reset(vf, flr);
/* poll VPGEN_VFRSTAT reg to make sure
* that reset is complete
@@ -984,35 +1064,116 @@
if (!rsd)
dev_err(&pf->pdev->dev, "VF reset check timeout on VF %d\n",
vf->vf_id);
- wr32(hw, I40E_VFGEN_RSTAT1(vf->vf_id), I40E_VFR_COMPLETED);
- /* clear the reset bit in the VPGEN_VFRTRIG reg */
- reg = rd32(hw, I40E_VPGEN_VFRTRIG(vf->vf_id));
- reg &= ~I40E_VPGEN_VFRTRIG_VFSWR_MASK;
- wr32(hw, I40E_VPGEN_VFRTRIG(vf->vf_id), reg);
+ usleep_range(10000, 20000);
- /* On initial reset, we won't have any queues */
- if (vf->lan_vsi_idx == 0)
- goto complete_reset;
+ /* On initial reset, we don't have any queues to disable */
+ if (vf->lan_vsi_idx != 0)
+ i40e_vsi_stop_rings(pf->vsi[vf->lan_vsi_idx]);
- i40e_vsi_stop_rings(pf->vsi[vf->lan_vsi_idx]);
-complete_reset:
- /* reallocate VF resources to reset the VSI state */
- i40e_free_vf_res(vf);
- if (!i40e_alloc_vf_res(vf)) {
- int abs_vf_id = vf->vf_id + hw->func_caps.vf_base_id;
- i40e_enable_vf_mappings(vf);
- set_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states);
- clear_bit(I40E_VF_STAT_DISABLED, &vf->vf_states);
- /* Do not notify the client during VF init */
- if (vf->pf->num_alloc_vfs)
- i40e_notify_client_of_vf_reset(pf, abs_vf_id);
- vf->num_vlan = 0;
- }
- /* tell the VF the reset is done */
- wr32(hw, I40E_VFGEN_RSTAT1(vf->vf_id), I40E_VFR_VFACTIVE);
+ i40e_cleanup_reset_vf(vf);
i40e_flush(hw);
- clear_bit(__I40E_VF_DISABLE, &pf->state);
+ clear_bit(__I40E_VF_DISABLE, pf->state);
+}
+
+/**
+ * i40e_reset_all_vfs
+ * @pf: pointer to the PF structure
+ * @flr: VFLR was issued or not
+ *
+ * Reset all allocated VFs in one go. First, tell the hardware to reset each
+ * VF, then do all the waiting in one chunk, and finally finish restoring each
+ * VF after the wait. This is useful during PF routines which need to reset
+ * all VFs, as otherwise it must perform these resets in a serialized fashion.
+ **/
+void i40e_reset_all_vfs(struct i40e_pf *pf, bool flr)
+{
+ struct i40e_hw *hw = &pf->hw;
+ struct i40e_vf *vf;
+ int i, v;
+ u32 reg;
+
+ /* If we don't have any VFs, then there is nothing to reset */
+ if (!pf->num_alloc_vfs)
+ return;
+
+ /* If VFs have been disabled, there is no need to reset */
+ if (test_and_set_bit(__I40E_VF_DISABLE, pf->state))
+ return;
+
+ /* Begin reset on all VFs at once */
+ for (v = 0; v < pf->num_alloc_vfs; v++)
+ i40e_trigger_vf_reset(&pf->vf[v], flr);
+
+ /* HW requires some time to make sure it can flush the FIFO for a VF
+ * when it resets it. Poll the VPGEN_VFRSTAT register for each VF in
+ * sequence to make sure that it has completed. We'll keep track of
+ * the VFs using a simple iterator that increments once that VF has
+ * finished resetting.
+ */
+ for (i = 0, v = 0; i < 10 && v < pf->num_alloc_vfs; i++) {
+ usleep_range(10000, 20000);
+
+ /* Check each VF in sequence, beginning with the VF to fail
+ * the previous check.
+ */
+ while (v < pf->num_alloc_vfs) {
+ vf = &pf->vf[v];
+ reg = rd32(hw, I40E_VPGEN_VFRSTAT(vf->vf_id));
+ if (!(reg & I40E_VPGEN_VFRSTAT_VFRD_MASK))
+ break;
+
+ /* If the current VF has finished resetting, move on
+ * to the next VF in sequence.
+ */
+ v++;
+ }
+ }
+
+ if (flr)
+ usleep_range(10000, 20000);
+
+ /* Display a warning if at least one VF didn't manage to reset in
+ * time, but continue on with the operation.
+ */
+ if (v < pf->num_alloc_vfs)
+ dev_err(&pf->pdev->dev, "VF reset check timeout on VF %d\n",
+ pf->vf[v].vf_id);
+ usleep_range(10000, 20000);
+
+ /* Begin disabling all the rings associated with VFs, but do not wait
+ * between each VF.
+ */
+ for (v = 0; v < pf->num_alloc_vfs; v++) {
+ /* On initial reset, we don't have any queues to disable */
+ if (pf->vf[v].lan_vsi_idx == 0)
+ continue;
+
+ i40e_vsi_stop_rings_no_wait(pf->vsi[pf->vf[v].lan_vsi_idx]);
+ }
+
+ /* Now that we've notified HW to disable all of the VF rings, wait
+ * until they finish.
+ */
+ for (v = 0; v < pf->num_alloc_vfs; v++) {
+ /* On initial reset, we don't have any queues to disable */
+ if (pf->vf[v].lan_vsi_idx == 0)
+ continue;
+
+ i40e_vsi_wait_queues_disabled(pf->vsi[pf->vf[v].lan_vsi_idx]);
+ }
+
+ /* Hw may need up to 50ms to finish disabling the RX queues. We
+ * minimize the wait by delaying only once for all VFs.
+ */
+ mdelay(50);
+
+ /* Finish the reset on each VF */
+ for (v = 0; v < pf->num_alloc_vfs; v++)
+ i40e_cleanup_reset_vf(&pf->vf[v]);
+
+ i40e_flush(hw);
+ clear_bit(__I40E_VF_DISABLE, pf->state);
}
/**
@@ -1029,13 +1190,25 @@
if (!pf->vf)
return;
- while (test_and_set_bit(__I40E_VF_DISABLE, &pf->state))
+ while (test_and_set_bit(__I40E_VF_DISABLE, pf->state))
usleep_range(1000, 2000);
i40e_notify_client_of_vf_enable(pf, 0);
- for (i = 0; i < pf->num_alloc_vfs; i++)
- if (test_bit(I40E_VF_STAT_INIT, &pf->vf[i].vf_states))
- i40e_vsi_stop_rings(pf->vsi[pf->vf[i].lan_vsi_idx]);
+
+ /* Amortize wait time by stopping all VFs at the same time */
+ for (i = 0; i < pf->num_alloc_vfs; i++) {
+ if (test_bit(I40E_VF_STATE_INIT, &pf->vf[i].vf_states))
+ continue;
+
+ i40e_vsi_stop_rings_no_wait(pf->vsi[pf->vf[i].lan_vsi_idx]);
+ }
+
+ for (i = 0; i < pf->num_alloc_vfs; i++) {
+ if (test_bit(I40E_VF_STATE_INIT, &pf->vf[i].vf_states))
+ continue;
+
+ i40e_vsi_wait_queues_disabled(pf->vsi[pf->vf[i].lan_vsi_idx]);
+ }
/* Disable IOV before freeing resources. This lets any VF drivers
* running in the host get themselves cleaned up before we yank
@@ -1046,13 +1219,11 @@
else
dev_warn(&pf->pdev->dev, "VFs are assigned - not disabling SR-IOV\n");
- msleep(20); /* let any messages in transit get finished up */
-
/* free up VF resources */
tmp = pf->num_alloc_vfs;
pf->num_alloc_vfs = 0;
for (i = 0; i < tmp; i++) {
- if (test_bit(I40E_VF_STAT_INIT, &pf->vf[i].vf_states))
+ if (test_bit(I40E_VF_STATE_INIT, &pf->vf[i].vf_states))
i40e_free_vf_res(&pf->vf[i]);
/* disable qp mappings */
i40e_disable_vf_mappings(&pf->vf[i]);
@@ -1075,7 +1246,7 @@
wr32(hw, I40E_GLGEN_VFLRSTAT(reg_idx), BIT(bit_idx));
}
}
- clear_bit(__I40E_VF_DISABLE, &pf->state);
+ clear_bit(__I40E_VF_DISABLE, pf->state);
}
#ifdef CONFIG_PCI_IOV
@@ -1120,12 +1291,15 @@
/* assign default capabilities */
set_bit(I40E_VIRTCHNL_VF_CAP_L2, &vfs[i].vf_caps);
vfs[i].spoofchk = true;
- /* VF resources get allocated during reset */
- i40e_reset_vf(&vfs[i], false);
+
+ set_bit(I40E_VF_STATE_PRE_ENABLE, &vfs[i].vf_states);
}
pf->num_alloc_vfs = num_alloc_vfs;
+ /* VF resources get allocated during reset */
+ i40e_reset_all_vfs(pf, false);
+
i40e_notify_client_of_vf_enable(pf, num_alloc_vfs);
err_alloc:
@@ -1152,7 +1326,7 @@
int pre_existing_vfs = pci_num_vf(pdev);
int err = 0;
- if (test_bit(__I40E_TESTING, &pf->state)) {
+ if (test_bit(__I40E_TESTING, pf->state)) {
dev_warn(&pdev->dev,
"Cannot enable SR-IOV virtual functions while the device is undergoing diagnostic testing\n");
err = -EPERM;
@@ -1258,7 +1432,7 @@
"Number of invalid messages exceeded for VF %d\n",
vf->vf_id);
dev_err(&pf->pdev->dev, "Use PF Control I/F to enable the VF\n");
- set_bit(I40E_VF_STAT_DISABLED, &vf->vf_states);
+ set_bit(I40E_VF_STATE_DISABLED, &vf->vf_states);
}
} else {
vf->num_valid_msgs++;
@@ -1333,7 +1507,7 @@
int len = 0;
int ret;
- if (!test_bit(I40E_VF_STAT_INIT, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_INIT, &vf->vf_states)) {
aq_ret = I40E_ERR_PARAM;
goto err;
}
@@ -1359,10 +1533,10 @@
if (!vsi->info.pvid)
vfres->vf_offload_flags |= I40E_VIRTCHNL_VF_OFFLOAD_VLAN;
- if (i40e_vf_client_capable(pf, vf->vf_id, I40E_CLIENT_IWARP) &&
+ if (i40e_vf_client_capable(pf, vf->vf_id) &&
(vf->driver_caps & I40E_VIRTCHNL_VF_OFFLOAD_IWARP)) {
vfres->vf_offload_flags |= I40E_VIRTCHNL_VF_OFFLOAD_IWARP;
- set_bit(I40E_VF_STAT_IWARPENA, &vf->vf_states);
+ set_bit(I40E_VF_STATE_IWARPENA, &vf->vf_states);
}
if (vf->driver_caps & I40E_VIRTCHNL_VF_OFFLOAD_RSS_PF) {
@@ -1383,6 +1557,13 @@
I40E_VIRTCHNL_VF_OFFLOAD_RSS_PCTYPE_V2;
}
+ if (vf->driver_caps & I40E_VIRTCHNL_VF_OFFLOAD_ENCAP)
+ vfres->vf_offload_flags |= I40E_VIRTCHNL_VF_OFFLOAD_ENCAP;
+
+ if ((pf->flags & I40E_FLAG_OUTER_UDP_CSUM_CAPABLE) &&
+ (vf->driver_caps & I40E_VIRTCHNL_VF_OFFLOAD_ENCAP_CSUM))
+ vfres->vf_offload_flags |= I40E_VIRTCHNL_VF_OFFLOAD_ENCAP_CSUM;
+
if (vf->driver_caps & I40E_VIRTCHNL_VF_OFFLOAD_RX_POLLING) {
if (pf->flags & I40E_FLAG_MFP_ENABLED) {
dev_err(&pf->pdev->dev,
@@ -1416,7 +1597,7 @@
ether_addr_copy(vfres->vsi_res[0].default_mac_addr,
vf->default_lan_addr.addr);
}
- set_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states);
+ set_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states);
err:
/* send the response back to the VF */
@@ -1439,7 +1620,7 @@
**/
static void i40e_vc_reset_vf_msg(struct i40e_vf *vf)
{
- if (test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states))
+ if (test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states))
i40e_reset_vf(vf, false);
}
@@ -1487,7 +1668,7 @@
int bkt;
vsi = i40e_find_vsi_from_id(pf, info->vsi_id);
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states) ||
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states) ||
!i40e_vc_isvalid_vsi_id(vf, info->vsi_id) ||
!vsi) {
aq_ret = I40E_ERR_PARAM;
@@ -1548,9 +1729,9 @@
"VF %d successfully set multicast promiscuous mode\n",
vf->vf_id);
if (allmulti)
- set_bit(I40E_VF_STAT_MC_PROMISC, &vf->vf_states);
+ set_bit(I40E_VF_STATE_MC_PROMISC, &vf->vf_states);
else
- clear_bit(I40E_VF_STAT_MC_PROMISC, &vf->vf_states);
+ clear_bit(I40E_VF_STATE_MC_PROMISC, &vf->vf_states);
}
if (info->flags & I40E_FLAG_VF_UNICAST_PROMISC)
@@ -1599,9 +1780,9 @@
"VF %d successfully set unicast promiscuous mode\n",
vf->vf_id);
if (alluni)
- set_bit(I40E_VF_STAT_UC_PROMISC, &vf->vf_states);
+ set_bit(I40E_VF_STATE_UC_PROMISC, &vf->vf_states);
else
- clear_bit(I40E_VF_STAT_UC_PROMISC, &vf->vf_states);
+ clear_bit(I40E_VF_STATE_UC_PROMISC, &vf->vf_states);
}
error_param:
@@ -1630,7 +1811,7 @@
i40e_status aq_ret = 0;
int i;
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states)) {
aq_ret = I40E_ERR_PARAM;
goto error_param;
}
@@ -1687,7 +1868,7 @@
unsigned long tempmap;
int i;
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states)) {
aq_ret = I40E_ERR_PARAM;
goto error_param;
}
@@ -1747,7 +1928,7 @@
u16 vsi_id = vqs->vsi_id;
i40e_status aq_ret = 0;
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states)) {
aq_ret = I40E_ERR_PARAM;
goto error_param;
}
@@ -1786,7 +1967,7 @@
struct i40e_pf *pf = vf->pf;
i40e_status aq_ret = 0;
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states)) {
aq_ret = I40E_ERR_PARAM;
goto error_param;
}
@@ -1828,7 +2009,7 @@
memset(&stats, 0, sizeof(struct i40e_eth_stats));
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states)) {
aq_ret = I40E_ERR_PARAM;
goto error_param;
}
@@ -1853,7 +2034,7 @@
}
/* If the VF is not trusted restrict the number of MAC/VLAN it can program */
-#define I40E_VC_MAX_MAC_ADDR_PER_VF 8
+#define I40E_VC_MAX_MAC_ADDR_PER_VF 12
#define I40E_VC_MAX_VLAN_PER_VF 8
/**
@@ -1915,7 +2096,7 @@
i40e_status ret = 0;
int i;
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states) ||
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states) ||
!i40e_vc_isvalid_vsi_id(vf, vsi_id)) {
ret = I40E_ERR_PARAM;
goto error_param;
@@ -1984,7 +2165,7 @@
i40e_status ret = 0;
int i;
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states) ||
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states) ||
!i40e_vc_isvalid_vsi_id(vf, vsi_id)) {
ret = I40E_ERR_PARAM;
goto error_param;
@@ -2050,7 +2231,7 @@
"VF is not trusted, switch the VF to trusted to add more VLAN addresses\n");
goto error_param;
}
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states) ||
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states) ||
!i40e_vc_isvalid_vsi_id(vf, vsi_id)) {
aq_ret = I40E_ERR_PARAM;
goto error_param;
@@ -2077,12 +2258,12 @@
if (!ret)
vf->num_vlan++;
- if (test_bit(I40E_VF_STAT_UC_PROMISC, &vf->vf_states))
+ if (test_bit(I40E_VF_STATE_UC_PROMISC, &vf->vf_states))
i40e_aq_set_vsi_uc_promisc_on_vlan(&pf->hw, vsi->seid,
true,
vfl->vlan_id[i],
NULL);
- if (test_bit(I40E_VF_STAT_MC_PROMISC, &vf->vf_states))
+ if (test_bit(I40E_VF_STATE_MC_PROMISC, &vf->vf_states))
i40e_aq_set_vsi_mc_promisc_on_vlan(&pf->hw, vsi->seid,
true,
vfl->vlan_id[i],
@@ -2117,7 +2298,7 @@
i40e_status aq_ret = 0;
int i;
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states) ||
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states) ||
!i40e_vc_isvalid_vsi_id(vf, vsi_id)) {
aq_ret = I40E_ERR_PARAM;
goto error_param;
@@ -2140,12 +2321,12 @@
i40e_vsi_kill_vlan(vsi, vfl->vlan_id[i]);
vf->num_vlan--;
- if (test_bit(I40E_VF_STAT_UC_PROMISC, &vf->vf_states))
+ if (test_bit(I40E_VF_STATE_UC_PROMISC, &vf->vf_states))
i40e_aq_set_vsi_uc_promisc_on_vlan(&pf->hw, vsi->seid,
false,
vfl->vlan_id[i],
NULL);
- if (test_bit(I40E_VF_STAT_MC_PROMISC, &vf->vf_states))
+ if (test_bit(I40E_VF_STATE_MC_PROMISC, &vf->vf_states))
i40e_aq_set_vsi_mc_promisc_on_vlan(&pf->hw, vsi->seid,
false,
vfl->vlan_id[i],
@@ -2171,8 +2352,8 @@
int abs_vf_id = vf->vf_id + pf->hw.func_caps.vf_base_id;
i40e_status aq_ret = 0;
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states) ||
- !test_bit(I40E_VF_STAT_IWARPENA, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states) ||
+ !test_bit(I40E_VF_STATE_IWARPENA, &vf->vf_states)) {
aq_ret = I40E_ERR_PARAM;
goto error_param;
}
@@ -2202,8 +2383,8 @@
(struct i40e_virtchnl_iwarp_qvlist_info *)msg;
i40e_status aq_ret = 0;
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states) ||
- !test_bit(I40E_VF_STAT_IWARPENA, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states) ||
+ !test_bit(I40E_VF_STATE_IWARPENA, &vf->vf_states)) {
aq_ret = I40E_ERR_PARAM;
goto error_param;
}
@@ -2240,7 +2421,7 @@
u16 vsi_id = vrk->vsi_id;
i40e_status aq_ret = 0;
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states) ||
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states) ||
!i40e_vc_isvalid_vsi_id(vf, vsi_id) ||
(vrk->key_len != I40E_HKEY_ARRAY_SIZE)) {
aq_ret = I40E_ERR_PARAM;
@@ -2272,7 +2453,7 @@
u16 vsi_id = vrl->vsi_id;
i40e_status aq_ret = 0;
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states) ||
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states) ||
!i40e_vc_isvalid_vsi_id(vf, vsi_id) ||
(vrl->lut_entries != I40E_VF_HLUT_ARRAY_SIZE)) {
aq_ret = I40E_ERR_PARAM;
@@ -2302,7 +2483,7 @@
i40e_status aq_ret = 0;
int len = 0;
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states)) {
aq_ret = I40E_ERR_PARAM;
goto err;
}
@@ -2339,7 +2520,7 @@
struct i40e_hw *hw = &pf->hw;
i40e_status aq_ret = 0;
- if (!test_bit(I40E_VF_STAT_ACTIVE, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states)) {
aq_ret = I40E_ERR_PARAM;
goto err;
}
@@ -2369,7 +2550,7 @@
int valid_len = 0;
/* Check if VF is disabled. */
- if (test_bit(I40E_VF_STAT_DISABLED, &vf->vf_states))
+ if (test_bit(I40E_VF_STATE_DISABLED, &vf->vf_states))
return I40E_ERR_PARAM;
/* Validate message length. */
@@ -2637,7 +2818,7 @@
struct i40e_vf *vf;
int vf_id;
- if (!test_bit(__I40E_VFLR_EVENT_PENDING, &pf->state))
+ if (!test_bit(__I40E_VFLR_EVENT_PENDING, pf->state))
return 0;
/* Re-enable the VFLR interrupt cause here, before looking for which
@@ -2650,7 +2831,7 @@
wr32(hw, I40E_PFINT_ICR0_ENA, reg);
i40e_flush(hw);
- clear_bit(__I40E_VFLR_EVENT_PENDING, &pf->state);
+ clear_bit(__I40E_VFLR_EVENT_PENDING, pf->state);
for (vf_id = 0; vf_id < pf->num_alloc_vfs; vf_id++) {
reg_idx = (hw->func_caps.vf_base_id + vf_id) / 32;
bit_idx = (hw->func_caps.vf_base_id + vf_id) % 32;
@@ -2693,7 +2874,7 @@
vf = &(pf->vf[vf_id]);
vsi = pf->vsi[vf->lan_vsi_idx];
- if (!test_bit(I40E_VF_STAT_INIT, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_INIT, &vf->vf_states)) {
dev_err(&pf->pdev->dev, "VF %d still in reset. Try again.\n",
vf_id);
ret = -EAGAIN;
@@ -2782,7 +2963,7 @@
vf = &(pf->vf[vf_id]);
vsi = pf->vsi[vf->lan_vsi_idx];
- if (!test_bit(I40E_VF_STAT_INIT, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_INIT, &vf->vf_states)) {
dev_err(&pf->pdev->dev, "VF %d still in reset. Try again.\n",
vf_id);
ret = -EAGAIN;
@@ -2914,7 +3095,7 @@
vf = &(pf->vf[vf_id]);
vsi = pf->vsi[vf->lan_vsi_idx];
- if (!test_bit(I40E_VF_STAT_INIT, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_INIT, &vf->vf_states)) {
dev_err(&pf->pdev->dev, "VF %d still in reset. Try again.\n",
vf_id);
ret = -EAGAIN;
@@ -2995,7 +3176,7 @@
vf = &(pf->vf[vf_id]);
/* first vsi is always the LAN vsi */
vsi = pf->vsi[vf->lan_vsi_idx];
- if (!test_bit(I40E_VF_STAT_INIT, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_INIT, &vf->vf_states)) {
dev_err(&pf->pdev->dev, "VF %d still in reset. Try again.\n",
vf_id);
ret = -EAGAIN;
@@ -3114,7 +3295,7 @@
}
vf = &(pf->vf[vf_id]);
- if (!test_bit(I40E_VF_STAT_INIT, &vf->vf_states)) {
+ if (!test_bit(I40E_VF_STATE_INIT, &vf->vf_states)) {
dev_err(&pf->pdev->dev, "VF %d still in reset. Try again.\n",
vf_id);
ret = -EAGAIN;
diff --git a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.h b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.h
index 4012d06..20d7c81 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.h
+++ b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.h
@@ -56,13 +56,14 @@
/* VF states */
enum i40e_vf_states {
- I40E_VF_STAT_INIT = 0,
- I40E_VF_STAT_ACTIVE,
- I40E_VF_STAT_IWARPENA,
- I40E_VF_STAT_FCOEENA,
- I40E_VF_STAT_DISABLED,
- I40E_VF_STAT_MC_PROMISC,
- I40E_VF_STAT_UC_PROMISC,
+ I40E_VF_STATE_INIT = 0,
+ I40E_VF_STATE_ACTIVE,
+ I40E_VF_STATE_IWARPENA,
+ I40E_VF_STATE_FCOEENA,
+ I40E_VF_STATE_DISABLED,
+ I40E_VF_STATE_MC_PROMISC,
+ I40E_VF_STATE_UC_PROMISC,
+ I40E_VF_STATE_PRE_ENABLE,
};
/* VF capabilities */
@@ -87,7 +88,6 @@
u16 stag;
struct i40e_virtchnl_ether_addr default_lan_addr;
- struct i40e_virtchnl_ether_addr default_fcoe_addr;
u16 port_vlan_id;
bool pf_set_mac; /* The VMM admin set the VF MAC address */
bool trusted;
@@ -125,6 +125,7 @@
u32 v_retval, u8 *msg, u16 msglen);
int i40e_vc_process_vflr_event(struct i40e_pf *pf);
void i40e_reset_vf(struct i40e_vf *vf, bool flr);
+void i40e_reset_all_vfs(struct i40e_pf *pf, bool flr);
void i40e_vc_notify_vf_reset(struct i40e_vf *vf);
/* VF configuration related iplink handlers */
diff --git a/drivers/net/ethernet/intel/i40evf/Makefile b/drivers/net/ethernet/intel/i40evf/Makefile
index 3a42383..a393f4a 100644
--- a/drivers/net/ethernet/intel/i40evf/Makefile
+++ b/drivers/net/ethernet/intel/i40evf/Makefile
@@ -29,8 +29,11 @@
#
#
+ccflags-y += -I$(src)
+subdir-ccflags-y += -I$(src)
+
obj-$(CONFIG_I40EVF) += i40evf.o
i40evf-objs := i40evf_main.o i40evf_ethtool.o i40evf_virtchnl.o \
- i40e_txrx.o i40e_common.o i40e_adminq.o
+ i40e_txrx.o i40e_common.o i40e_adminq.o i40evf_client.o
diff --git a/drivers/net/ethernet/intel/i40evf/i40e_adminq.c b/drivers/net/ethernet/intel/i40evf/i40e_adminq.c
index 9638515..8b0d4b2 100644
--- a/drivers/net/ethernet/intel/i40evf/i40e_adminq.c
+++ b/drivers/net/ethernet/intel/i40evf/i40e_adminq.c
@@ -797,8 +797,8 @@
*/
if (i40evf_asq_done(hw))
break;
- usleep_range(1000, 2000);
- total_delay++;
+ udelay(50);
+ total_delay += 50;
} while (total_delay < hw->aq.asq_cmd_timeout);
}
diff --git a/drivers/net/ethernet/intel/i40evf/i40e_adminq.h b/drivers/net/ethernet/intel/i40evf/i40e_adminq.h
index 1f9b3b5..e0bfaa3 100644
--- a/drivers/net/ethernet/intel/i40evf/i40e_adminq.h
+++ b/drivers/net/ethernet/intel/i40evf/i40e_adminq.h
@@ -151,7 +151,7 @@
/* general information */
#define I40E_AQ_LARGE_BUF 512
-#define I40E_ASQ_CMD_TIMEOUT 250 /* msecs */
+#define I40E_ASQ_CMD_TIMEOUT 250000 /* usecs */
void i40evf_fill_default_direct_cmd_desc(struct i40e_aq_desc *desc,
u16 opcode);
diff --git a/drivers/net/ethernet/intel/i40evf/i40e_adminq_cmd.h b/drivers/net/ethernet/intel/i40evf/i40e_adminq_cmd.h
index eeb9864..91d8786 100644
--- a/drivers/net/ethernet/intel/i40evf/i40e_adminq_cmd.h
+++ b/drivers/net/ethernet/intel/i40evf/i40e_adminq_cmd.h
@@ -132,6 +132,10 @@
i40e_aqc_opc_list_func_capabilities = 0x000A,
i40e_aqc_opc_list_dev_capabilities = 0x000B,
+ /* Proxy commands */
+ i40e_aqc_opc_set_proxy_config = 0x0104,
+ i40e_aqc_opc_set_ns_proxy_table_entry = 0x0105,
+
/* LAA */
i40e_aqc_opc_mac_address_read = 0x0107,
i40e_aqc_opc_mac_address_write = 0x0108,
@@ -139,6 +143,10 @@
/* PXE */
i40e_aqc_opc_clear_pxe_mode = 0x0110,
+ /* WoL commands */
+ i40e_aqc_opc_set_wol_filter = 0x0120,
+ i40e_aqc_opc_get_wake_reason = 0x0121,
+
/* internal switch commands */
i40e_aqc_opc_get_switch_config = 0x0200,
i40e_aqc_opc_add_statistics = 0x0201,
@@ -177,10 +185,15 @@
i40e_aqc_opc_remove_control_packet_filter = 0x025B,
i40e_aqc_opc_add_cloud_filters = 0x025C,
i40e_aqc_opc_remove_cloud_filters = 0x025D,
+ i40e_aqc_opc_clear_wol_switch_filters = 0x025E,
i40e_aqc_opc_add_mirror_rule = 0x0260,
i40e_aqc_opc_delete_mirror_rule = 0x0261,
+ /* Pipeline Personalization Profile */
+ i40e_aqc_opc_write_personalization_profile = 0x0270,
+ i40e_aqc_opc_get_personalization_profile_list = 0x0271,
+
/* DCB commands */
i40e_aqc_opc_dcb_ignore_pfc = 0x0301,
i40e_aqc_opc_dcb_updated = 0x0302,
@@ -558,6 +571,56 @@
I40E_CHECK_CMD_LENGTH(i40e_aqc_clear_pxe);
+/* Set WoL Filter (0x0120) */
+
+struct i40e_aqc_set_wol_filter {
+ __le16 filter_index;
+#define I40E_AQC_MAX_NUM_WOL_FILTERS 8
+#define I40E_AQC_SET_WOL_FILTER_TYPE_MAGIC_SHIFT 15
+#define I40E_AQC_SET_WOL_FILTER_TYPE_MAGIC_MASK (0x1 << \
+ I40E_AQC_SET_WOL_FILTER_TYPE_MAGIC_SHIFT)
+
+#define I40E_AQC_SET_WOL_FILTER_INDEX_SHIFT 0
+#define I40E_AQC_SET_WOL_FILTER_INDEX_MASK (0x7 << \
+ I40E_AQC_SET_WOL_FILTER_INDEX_SHIFT)
+ __le16 cmd_flags;
+#define I40E_AQC_SET_WOL_FILTER 0x8000
+#define I40E_AQC_SET_WOL_FILTER_NO_TCO_WOL 0x4000
+#define I40E_AQC_SET_WOL_FILTER_ACTION_CLEAR 0
+#define I40E_AQC_SET_WOL_FILTER_ACTION_SET 1
+ __le16 valid_flags;
+#define I40E_AQC_SET_WOL_FILTER_ACTION_VALID 0x8000
+#define I40E_AQC_SET_WOL_FILTER_NO_TCO_ACTION_VALID 0x4000
+ u8 reserved[2];
+ __le32 address_high;
+ __le32 address_low;
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_set_wol_filter);
+
+struct i40e_aqc_set_wol_filter_data {
+ u8 filter[128];
+ u8 mask[16];
+};
+
+I40E_CHECK_STRUCT_LEN(0x90, i40e_aqc_set_wol_filter_data);
+
+/* Get Wake Reason (0x0121) */
+
+struct i40e_aqc_get_wake_reason_completion {
+ u8 reserved_1[2];
+ __le16 wake_reason;
+#define I40E_AQC_GET_WAKE_UP_REASON_WOL_REASON_MATCHED_INDEX_SHIFT 0
+#define I40E_AQC_GET_WAKE_UP_REASON_WOL_REASON_MATCHED_INDEX_MASK (0xFF << \
+ I40E_AQC_GET_WAKE_UP_REASON_WOL_REASON_MATCHED_INDEX_SHIFT)
+#define I40E_AQC_GET_WAKE_UP_REASON_WOL_REASON_RESERVED_SHIFT 8
+#define I40E_AQC_GET_WAKE_UP_REASON_WOL_REASON_RESERVED_MASK (0xFF << \
+ I40E_AQC_GET_WAKE_UP_REASON_WOL_REASON_RESERVED_SHIFT)
+ u8 reserved_2[12];
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_get_wake_reason_completion);
+
/* Switch configuration commands (0x02xx) */
/* Used by many indirect commands that only pass an seid and a buffer in the
@@ -640,6 +703,8 @@
#define I40E_AQ_SET_P_PARAMS_PAD_SHORT_PACKETS 2 /* must set! */
#define I40E_AQ_SET_P_PARAMS_DOUBLE_VLAN_ENA 4
__le16 bad_frame_vsi;
+#define I40E_AQ_SET_P_PARAMS_BFRAME_SEID_SHIFT 0x0
+#define I40E_AQ_SET_P_PARAMS_BFRAME_SEID_MASK 0x3FF
__le16 default_seid; /* reserved for command */
u8 reserved[10];
};
@@ -691,6 +756,7 @@
/* Set Switch Configuration (direct 0x0205) */
struct i40e_aqc_set_switch_config {
__le16 flags;
+/* flags used for both fields below */
#define I40E_AQ_SET_SWITCH_CFG_PROMISC 0x0001
#define I40E_AQ_SET_SWITCH_CFG_L2_FILTER 0x0002
__le16 valid_flags;
@@ -1364,6 +1430,36 @@
I40E_CHECK_CMD_LENGTH(i40e_aqc_add_delete_mirror_rule_completion);
+/* Pipeline Personalization Profile */
+struct i40e_aqc_write_personalization_profile {
+ u8 flags;
+ u8 reserved[3];
+ __le32 profile_track_id;
+ __le32 addr_high;
+ __le32 addr_low;
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_write_personalization_profile);
+
+struct i40e_aqc_write_ppp_resp {
+ __le32 error_offset;
+ __le32 error_info;
+ __le32 addr_high;
+ __le32 addr_low;
+};
+
+struct i40e_aqc_get_applied_profiles {
+ u8 flags;
+#define I40E_AQC_GET_PPP_GET_CONF 0x1
+#define I40E_AQC_GET_PPP_GET_RDPU_CONF 0x2
+ u8 rsv[3];
+ __le32 reserved;
+ __le32 addr_high;
+ __le32 addr_low;
+};
+
+I40E_CHECK_CMD_LENGTH(i40e_aqc_get_applied_profiles);
+
/* DCB 0x03xx*/
/* PFC Ignore (direct 0x0301)
@@ -1839,11 +1935,12 @@
#define I40E_AQ_CONFIG_FEC_RS_ENA 0x02
#define I40E_AQ_CONFIG_CRC_ENA 0x04
#define I40E_AQ_CONFIG_PACING_MASK 0x78
- u8 external_power_ability;
+ u8 power_desc;
#define I40E_AQ_LINK_POWER_CLASS_1 0x00
#define I40E_AQ_LINK_POWER_CLASS_2 0x01
#define I40E_AQ_LINK_POWER_CLASS_3 0x02
#define I40E_AQ_LINK_POWER_CLASS_4 0x03
+#define I40E_AQ_PWR_CLASS_MASK 0x03
u8 reserved[4];
};
diff --git a/drivers/net/ethernet/intel/i40evf/i40e_common.c b/drivers/net/ethernet/intel/i40evf/i40e_common.c
index 89dfdbc..43f1076 100644
--- a/drivers/net/ethernet/intel/i40evf/i40e_common.c
+++ b/drivers/net/ethernet/intel/i40evf/i40e_common.c
@@ -958,7 +958,9 @@
int retry = 5;
u32 val = 0;
- use_register = (hw->aq.api_maj_ver == 1) && (hw->aq.api_min_ver < 5);
+ use_register = (((hw->aq.api_maj_ver == 1) &&
+ (hw->aq.api_min_ver < 5)) ||
+ (hw->mac.type == I40E_MAC_X722));
if (!use_register) {
do_retry:
status = i40evf_aq_rx_ctl_read_register(hw, reg_addr,
@@ -1019,7 +1021,9 @@
bool use_register;
int retry = 5;
- use_register = (hw->aq.api_maj_ver == 1) && (hw->aq.api_min_ver < 5);
+ use_register = (((hw->aq.api_maj_ver == 1) &&
+ (hw->aq.api_min_ver < 5)) ||
+ (hw->mac.type == I40E_MAC_X722));
if (!use_register) {
do_retry:
status = i40evf_aq_rx_ctl_write_register(hw, reg_addr,
@@ -1127,3 +1131,215 @@
return i40e_aq_send_msg_to_pf(hw, I40E_VIRTCHNL_OP_RESET_VF,
0, NULL, 0, NULL);
}
+
+/**
+ * i40evf_aq_write_ppp - Write pipeline personalization profile (ppp)
+ * @hw: pointer to the hw struct
+ * @buff: command buffer (size in bytes = buff_size)
+ * @buff_size: buffer size in bytes
+ * @track_id: package tracking id
+ * @error_offset: returns error offset
+ * @error_info: returns error information
+ * @cmd_details: pointer to command details structure or NULL
+ **/
+enum
+i40e_status_code i40evf_aq_write_ppp(struct i40e_hw *hw, void *buff,
+ u16 buff_size, u32 track_id,
+ u32 *error_offset, u32 *error_info,
+ struct i40e_asq_cmd_details *cmd_details)
+{
+ struct i40e_aq_desc desc;
+ struct i40e_aqc_write_personalization_profile *cmd =
+ (struct i40e_aqc_write_personalization_profile *)
+ &desc.params.raw;
+ struct i40e_aqc_write_ppp_resp *resp;
+ i40e_status status;
+
+ i40evf_fill_default_direct_cmd_desc(&desc,
+ i40e_aqc_opc_write_personalization_profile);
+
+ desc.flags |= cpu_to_le16(I40E_AQ_FLAG_BUF | I40E_AQ_FLAG_RD);
+ if (buff_size > I40E_AQ_LARGE_BUF)
+ desc.flags |= cpu_to_le16((u16)I40E_AQ_FLAG_LB);
+
+ desc.datalen = cpu_to_le16(buff_size);
+
+ cmd->profile_track_id = cpu_to_le32(track_id);
+
+ status = i40evf_asq_send_command(hw, &desc, buff, buff_size, cmd_details);
+ if (!status) {
+ resp = (struct i40e_aqc_write_ppp_resp *)&desc.params.raw;
+ if (error_offset)
+ *error_offset = le32_to_cpu(resp->error_offset);
+ if (error_info)
+ *error_info = le32_to_cpu(resp->error_info);
+ }
+
+ return status;
+}
+
+/**
+ * i40evf_aq_get_ppp_list - Read pipeline personalization profile (ppp)
+ * @hw: pointer to the hw struct
+ * @buff: command buffer (size in bytes = buff_size)
+ * @buff_size: buffer size in bytes
+ * @cmd_details: pointer to command details structure or NULL
+ **/
+enum
+i40e_status_code i40evf_aq_get_ppp_list(struct i40e_hw *hw, void *buff,
+ u16 buff_size, u8 flags,
+ struct i40e_asq_cmd_details *cmd_details)
+{
+ struct i40e_aq_desc desc;
+ struct i40e_aqc_get_applied_profiles *cmd =
+ (struct i40e_aqc_get_applied_profiles *)&desc.params.raw;
+ i40e_status status;
+
+ i40evf_fill_default_direct_cmd_desc(&desc,
+ i40e_aqc_opc_get_personalization_profile_list);
+
+ desc.flags |= cpu_to_le16((u16)I40E_AQ_FLAG_BUF);
+ if (buff_size > I40E_AQ_LARGE_BUF)
+ desc.flags |= cpu_to_le16((u16)I40E_AQ_FLAG_LB);
+ desc.datalen = cpu_to_le16(buff_size);
+
+ cmd->flags = flags;
+
+ status = i40evf_asq_send_command(hw, &desc, buff, buff_size, cmd_details);
+
+ return status;
+}
+
+/**
+ * i40evf_find_segment_in_package
+ * @segment_type: the segment type to search for (i.e., SEGMENT_TYPE_I40E)
+ * @pkg_hdr: pointer to the package header to be searched
+ *
+ * This function searches a package file for a particular segment type. On
+ * success it returns a pointer to the segment header, otherwise it will
+ * return NULL.
+ **/
+struct i40e_generic_seg_header *
+i40evf_find_segment_in_package(u32 segment_type,
+ struct i40e_package_header *pkg_hdr)
+{
+ struct i40e_generic_seg_header *segment;
+ u32 i;
+
+ /* Search all package segments for the requested segment type */
+ for (i = 0; i < pkg_hdr->segment_count; i++) {
+ segment =
+ (struct i40e_generic_seg_header *)((u8 *)pkg_hdr +
+ pkg_hdr->segment_offset[i]);
+
+ if (segment->type == segment_type)
+ return segment;
+ }
+
+ return NULL;
+}
+
+/**
+ * i40evf_write_profile
+ * @hw: pointer to the hardware structure
+ * @profile: pointer to the profile segment of the package to be downloaded
+ * @track_id: package tracking id
+ *
+ * Handles the download of a complete package.
+ */
+enum i40e_status_code
+i40evf_write_profile(struct i40e_hw *hw, struct i40e_profile_segment *profile,
+ u32 track_id)
+{
+ i40e_status status = 0;
+ struct i40e_section_table *sec_tbl;
+ struct i40e_profile_section_header *sec = NULL;
+ u32 dev_cnt;
+ u32 vendor_dev_id;
+ u32 *nvm;
+ u32 section_size = 0;
+ u32 offset = 0, info = 0;
+ u32 i;
+
+ if (!track_id) {
+ i40e_debug(hw, I40E_DEBUG_PACKAGE, "Track_id can't be 0.");
+ return I40E_NOT_SUPPORTED;
+ }
+
+ dev_cnt = profile->device_table_count;
+
+ for (i = 0; i < dev_cnt; i++) {
+ vendor_dev_id = profile->device_table[i].vendor_dev_id;
+ if ((vendor_dev_id >> 16) == PCI_VENDOR_ID_INTEL)
+ if (hw->device_id == (vendor_dev_id & 0xFFFF))
+ break;
+ }
+ if (i == dev_cnt) {
+ i40e_debug(hw, I40E_DEBUG_PACKAGE, "Device doesn't support PPP");
+ return I40E_ERR_DEVICE_NOT_SUPPORTED;
+ }
+
+ nvm = (u32 *)&profile->device_table[dev_cnt];
+ sec_tbl = (struct i40e_section_table *)&nvm[nvm[0] + 1];
+
+ for (i = 0; i < sec_tbl->section_count; i++) {
+ sec = (struct i40e_profile_section_header *)((u8 *)profile +
+ sec_tbl->section_offset[i]);
+
+ /* Skip 'AQ', 'note' and 'name' sections */
+ if (sec->section.type != SECTION_TYPE_MMIO)
+ continue;
+
+ section_size = sec->section.size +
+ sizeof(struct i40e_profile_section_header);
+
+ /* Write profile */
+ status = i40evf_aq_write_ppp(hw, (void *)sec, (u16)section_size,
+ track_id, &offset, &info, NULL);
+ if (status) {
+ i40e_debug(hw, I40E_DEBUG_PACKAGE,
+ "Failed to write profile: offset %d, info %d",
+ offset, info);
+ break;
+ }
+ }
+ return status;
+}
+
+/**
+ * i40evf_add_pinfo_to_list
+ * @hw: pointer to the hardware structure
+ * @profile: pointer to the profile segment of the package
+ * @profile_info_sec: buffer for information section
+ * @track_id: package tracking id
+ *
+ * Register a profile to the list of loaded profiles.
+ */
+enum i40e_status_code
+i40evf_add_pinfo_to_list(struct i40e_hw *hw,
+ struct i40e_profile_segment *profile,
+ u8 *profile_info_sec, u32 track_id)
+{
+ i40e_status status = 0;
+ struct i40e_profile_section_header *sec = NULL;
+ struct i40e_profile_info *pinfo;
+ u32 offset = 0, info = 0;
+
+ sec = (struct i40e_profile_section_header *)profile_info_sec;
+ sec->tbl_size = 1;
+ sec->data_end = sizeof(struct i40e_profile_section_header) +
+ sizeof(struct i40e_profile_info);
+ sec->section.type = SECTION_TYPE_INFO;
+ sec->section.offset = sizeof(struct i40e_profile_section_header);
+ sec->section.size = sizeof(struct i40e_profile_info);
+ pinfo = (struct i40e_profile_info *)(profile_info_sec +
+ sec->section.offset);
+ pinfo->track_id = track_id;
+ pinfo->version = profile->version;
+ pinfo->op = I40E_PPP_ADD_TRACKID;
+ memcpy(pinfo->name, profile->name, I40E_PPP_NAME_SIZE);
+
+ status = i40evf_aq_write_ppp(hw, (void *)sec, sec->data_end,
+ track_id, &offset, &info, NULL);
+ return status;
+}
diff --git a/drivers/net/ethernet/intel/i40evf/i40e_prototype.h b/drivers/net/ethernet/intel/i40evf/i40e_prototype.h
index ba6c6bd..741223d 100644
--- a/drivers/net/ethernet/intel/i40evf/i40e_prototype.h
+++ b/drivers/net/ethernet/intel/i40evf/i40e_prototype.h
@@ -122,4 +122,21 @@
u8 i40e_get_phy_address(struct i40e_hw *hw, u8 dev_num);
i40e_status i40e_blink_phy_link_led(struct i40e_hw *hw,
u32 time, u32 interval);
+i40e_status i40evf_aq_write_ppp(struct i40e_hw *hw, void *buff,
+ u16 buff_size, u32 track_id,
+ u32 *error_offset, u32 *error_info,
+ struct i40e_asq_cmd_details *cmd_details);
+i40e_status i40evf_aq_get_ppp_list(struct i40e_hw *hw, void *buff,
+ u16 buff_size, u8 flags,
+ struct i40e_asq_cmd_details *cmd_details);
+struct i40e_generic_seg_header *
+i40evf_find_segment_in_package(u32 segment_type,
+ struct i40e_package_header *pkg_header);
+enum i40e_status_code
+i40evf_write_profile(struct i40e_hw *hw, struct i40e_profile_segment *i40e_seg,
+ u32 track_id);
+enum i40e_status_code
+i40evf_add_pinfo_to_list(struct i40e_hw *hw,
+ struct i40e_profile_segment *profile,
+ u8 *profile_info_sec, u32 track_id);
#endif /* _I40E_PROTOTYPE_H_ */
diff --git a/drivers/net/ethernet/intel/i40evf/i40e_trace.h b/drivers/net/ethernet/intel/i40evf/i40e_trace.h
new file mode 100644
index 0000000..9a5100b
--- /dev/null
+++ b/drivers/net/ethernet/intel/i40evf/i40e_trace.h
@@ -0,0 +1,229 @@
+/*******************************************************************************
+ *
+ * Intel(R) 40-10 Gigabit Ethernet Virtual Function Driver
+ * Copyright(c) 2013 - 2017 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Contact Information:
+ * e1000-devel Mailing List <e1000-devel@lists.sourceforge.net>
+ * Intel Corporation, 5200 N.E. Elam Young Parkway, Hillsboro, OR 97124-6497
+ *
+ ******************************************************************************/
+
+/* Modeled on trace-events-sample.h */
+
+/* The trace subsystem name for i40evf will be "i40evf".
+ *
+ * This file is named i40e_trace.h.
+ *
+ * Since this include file's name is different from the trace
+ * subsystem name, we'll have to define TRACE_INCLUDE_FILE at the end
+ * of this file.
+ */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM i40evf
+
+/* See trace-events-sample.h for a detailed description of why this
+ * guard clause is different from most normal include files.
+ */
+#if !defined(_I40E_TRACE_H_) || defined(TRACE_HEADER_MULTI_READ)
+#define _I40E_TRACE_H_
+
+#include <linux/tracepoint.h>
+
+/**
+ * i40e_trace() macro enables shared code to refer to trace points
+ * like:
+ *
+ * trace_i40e{,vf}_example(args...)
+ *
+ * ... as:
+ *
+ * i40e_trace(example, args...)
+ *
+ * ... to resolve to the PF or VF version of the tracepoint without
+ * ifdefs, and to allow tracepoints to be disabled entirely at build
+ * time.
+ *
+ * Trace point should always be referred to in the driver via this
+ * macro.
+ *
+ * Similarly, i40e_trace_enabled(trace_name) wraps references to
+ * trace_i40e{,vf}_<trace_name>_enabled() functions.
+ */
+#define _I40E_TRACE_NAME(trace_name) (trace_ ## i40evf ## _ ## trace_name)
+#define I40E_TRACE_NAME(trace_name) _I40E_TRACE_NAME(trace_name)
+
+#define i40e_trace(trace_name, args...) I40E_TRACE_NAME(trace_name)(args)
+
+#define i40e_trace_enabled(trace_name) I40E_TRACE_NAME(trace_name##_enabled)()
+
+/* Events common to PF and VF. Corresponding versions will be defined
+ * for both, named trace_i40e_* and trace_i40evf_*. The i40e_trace()
+ * macro above will select the right trace point name for the driver
+ * being built from shared code.
+ */
+
+/* Events related to a vsi & ring */
+DECLARE_EVENT_CLASS(
+ i40evf_tx_template,
+
+ TP_PROTO(struct i40e_ring *ring,
+ struct i40e_tx_desc *desc,
+ struct i40e_tx_buffer *buf),
+
+ TP_ARGS(ring, desc, buf),
+
+ /* The convention here is to make the first fields in the
+ * TP_STRUCT match the TP_PROTO exactly. This enables the use
+ * of the args struct generated by the tplist tool (from the
+ * bcc-tools package) to be used for those fields. To access
+ * fields other than the tracepoint args will require the
+ * tplist output to be adjusted.
+ */
+ TP_STRUCT__entry(
+ __field(void*, ring)
+ __field(void*, desc)
+ __field(void*, buf)
+ __string(devname, ring->netdev->name)
+ ),
+
+ TP_fast_assign(
+ __entry->ring = ring;
+ __entry->desc = desc;
+ __entry->buf = buf;
+ __assign_str(devname, ring->netdev->name);
+ ),
+
+ TP_printk(
+ "netdev: %s ring: %p desc: %p buf %p",
+ __get_str(devname), __entry->ring,
+ __entry->desc, __entry->buf)
+);
+
+DEFINE_EVENT(
+ i40evf_tx_template, i40evf_clean_tx_irq,
+ TP_PROTO(struct i40e_ring *ring,
+ struct i40e_tx_desc *desc,
+ struct i40e_tx_buffer *buf),
+
+ TP_ARGS(ring, desc, buf));
+
+DEFINE_EVENT(
+ i40evf_tx_template, i40evf_clean_tx_irq_unmap,
+ TP_PROTO(struct i40e_ring *ring,
+ struct i40e_tx_desc *desc,
+ struct i40e_tx_buffer *buf),
+
+ TP_ARGS(ring, desc, buf));
+
+DECLARE_EVENT_CLASS(
+ i40evf_rx_template,
+
+ TP_PROTO(struct i40e_ring *ring,
+ union i40e_32byte_rx_desc *desc,
+ struct sk_buff *skb),
+
+ TP_ARGS(ring, desc, skb),
+
+ TP_STRUCT__entry(
+ __field(void*, ring)
+ __field(void*, desc)
+ __field(void*, skb)
+ __string(devname, ring->netdev->name)
+ ),
+
+ TP_fast_assign(
+ __entry->ring = ring;
+ __entry->desc = desc;
+ __entry->skb = skb;
+ __assign_str(devname, ring->netdev->name);
+ ),
+
+ TP_printk(
+ "netdev: %s ring: %p desc: %p skb %p",
+ __get_str(devname), __entry->ring,
+ __entry->desc, __entry->skb)
+);
+
+DEFINE_EVENT(
+ i40evf_rx_template, i40evf_clean_rx_irq,
+ TP_PROTO(struct i40e_ring *ring,
+ union i40e_32byte_rx_desc *desc,
+ struct sk_buff *skb),
+
+ TP_ARGS(ring, desc, skb));
+
+DEFINE_EVENT(
+ i40evf_rx_template, i40evf_clean_rx_irq_rx,
+ TP_PROTO(struct i40e_ring *ring,
+ union i40e_32byte_rx_desc *desc,
+ struct sk_buff *skb),
+
+ TP_ARGS(ring, desc, skb));
+
+DECLARE_EVENT_CLASS(
+ i40evf_xmit_template,
+
+ TP_PROTO(struct sk_buff *skb,
+ struct i40e_ring *ring),
+
+ TP_ARGS(skb, ring),
+
+ TP_STRUCT__entry(
+ __field(void*, skb)
+ __field(void*, ring)
+ __string(devname, ring->netdev->name)
+ ),
+
+ TP_fast_assign(
+ __entry->skb = skb;
+ __entry->ring = ring;
+ __assign_str(devname, ring->netdev->name);
+ ),
+
+ TP_printk(
+ "netdev: %s skb: %p ring: %p",
+ __get_str(devname), __entry->skb,
+ __entry->ring)
+);
+
+DEFINE_EVENT(
+ i40evf_xmit_template, i40evf_xmit_frame_ring,
+ TP_PROTO(struct sk_buff *skb,
+ struct i40e_ring *ring),
+
+ TP_ARGS(skb, ring));
+
+DEFINE_EVENT(
+ i40evf_xmit_template, i40evf_xmit_frame_ring_drop,
+ TP_PROTO(struct sk_buff *skb,
+ struct i40e_ring *ring),
+
+ TP_ARGS(skb, ring));
+
+/* Events unique to the VF. */
+
+#endif /* _I40E_TRACE_H_ */
+/* This must be outside ifdef _I40E_TRACE_H */
+
+/* This trace include file is not located in the .../include/trace
+ * with the kernel tracepoint definitions, because we're a loadable
+ * module.
+ */
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE i40e_trace
+#include <trace/define_trace.h>
diff --git a/drivers/net/ethernet/intel/i40evf/i40e_txrx.c b/drivers/net/ethernet/intel/i40evf/i40e_txrx.c
index c91fcf4..dfe241a 100644
--- a/drivers/net/ethernet/intel/i40evf/i40e_txrx.c
+++ b/drivers/net/ethernet/intel/i40evf/i40e_txrx.c
@@ -28,6 +28,7 @@
#include <net/busy_poll.h>
#include "i40evf.h"
+#include "i40e_trace.h"
#include "i40e_prototype.h"
static inline __le64 build_ctob(u32 td_cmd, u32 td_offset, unsigned int size,
@@ -137,10 +138,7 @@
{
u32 head, tail;
- if (!in_sw)
- head = i40e_get_head(ring);
- else
- head = ring->next_to_clean;
+ head = ring->next_to_clean;
tail = readl(ring->tail);
if (head != tail)
@@ -165,7 +163,6 @@
{
u16 i = tx_ring->next_to_clean;
struct i40e_tx_buffer *tx_buf;
- struct i40e_tx_desc *tx_head;
struct i40e_tx_desc *tx_desc;
unsigned int total_bytes = 0, total_packets = 0;
unsigned int budget = vsi->work_limit;
@@ -174,8 +171,6 @@
tx_desc = I40E_TX_DESC(tx_ring, i);
i -= tx_ring->count;
- tx_head = I40E_TX_DESC(tx_ring, i40e_get_head(tx_ring));
-
do {
struct i40e_tx_desc *eop_desc = tx_buf->next_to_watch;
@@ -186,8 +181,10 @@
/* prevent any other reads prior to eop_desc */
read_barrier_depends();
- /* we have caught up to head, no work left to do */
- if (tx_head == tx_desc)
+ i40e_trace(clean_tx_irq, tx_ring, tx_desc, tx_buf);
+ /* if the descriptor isn't done, no work yet to do */
+ if (!(eop_desc->cmd_type_offset_bsz &
+ cpu_to_le64(I40E_TX_DESC_DTYPE_DESC_DONE)))
break;
/* clear next_to_watch to prevent false hangs */
@@ -212,6 +209,8 @@
/* unmap remaining buffers */
while (tx_desc != eop_desc) {
+ i40e_trace(clean_tx_irq_unmap,
+ tx_ring, tx_desc, tx_buf);
tx_buf++;
tx_desc++;
@@ -267,7 +266,7 @@
if (budget &&
((j / WB_STRIDE) == 0) && (j > 0) &&
- !test_bit(__I40E_DOWN, &vsi->state) &&
+ !test_bit(__I40E_VSI_DOWN, vsi->state) &&
(I40E_DESC_UNUSED(tx_ring) != tx_ring->count))
tx_ring->arm_wb = true;
}
@@ -285,7 +284,7 @@
smp_mb();
if (__netif_subqueue_stopped(tx_ring->netdev,
tx_ring->queue_index) &&
- !test_bit(__I40E_DOWN, &vsi->state)) {
+ !test_bit(__I40E_VSI_DOWN, vsi->state)) {
netif_wake_subqueue(tx_ring->netdev,
tx_ring->queue_index);
++tx_ring->tx_stats.restart_queue;
@@ -464,10 +463,6 @@
/* round up to nearest 4K */
tx_ring->size = tx_ring->count * sizeof(struct i40e_tx_desc);
- /* add u32 for head writeback, align after this takes care of
- * guaranteeing this is at least one cache line in size
- */
- tx_ring->size += sizeof(u32);
tx_ring->size = ALIGN(tx_ring->size, 4096);
tx_ring->desc = dma_alloc_coherent(dev, tx_ring->size,
&tx_ring->dma, GFP_KERNEL);
@@ -493,7 +488,6 @@
**/
void i40evf_clean_rx_ring(struct i40e_ring *rx_ring)
{
- struct device *dev = rx_ring->dev;
unsigned long bi_size;
u16 i;
@@ -513,8 +507,22 @@
if (!rx_bi->page)
continue;
- dma_unmap_page(dev, rx_bi->dma, PAGE_SIZE, DMA_FROM_DEVICE);
- __free_pages(rx_bi->page, 0);
+ /* Invalidate cache lines that may have been written to by
+ * device so that we avoid corrupting memory.
+ */
+ dma_sync_single_range_for_cpu(rx_ring->dev,
+ rx_bi->dma,
+ rx_bi->page_offset,
+ rx_ring->rx_buf_len,
+ DMA_FROM_DEVICE);
+
+ /* free resources associated with mapping */
+ dma_unmap_page_attrs(rx_ring->dev, rx_bi->dma,
+ i40e_rx_pg_size(rx_ring),
+ DMA_FROM_DEVICE,
+ I40E_RX_DMA_ATTR);
+
+ __page_frag_cache_drain(rx_bi->page, rx_bi->pagecnt_bias);
rx_bi->page = NULL;
rx_bi->page_offset = 0;
@@ -615,6 +623,17 @@
}
/**
+ * i40e_rx_offset - Return expected offset into page to access data
+ * @rx_ring: Ring we are requesting offset of
+ *
+ * Returns the offset value for ring into the data buffer.
+ */
+static inline unsigned int i40e_rx_offset(struct i40e_ring *rx_ring)
+{
+ return ring_uses_build_skb(rx_ring) ? I40E_SKB_PAD : 0;
+}
+
+/**
* i40e_alloc_mapped_page - recycle or make a new page
* @rx_ring: ring to use
* @bi: rx_buffer struct to modify
@@ -635,27 +654,33 @@
}
/* alloc new page for storage */
- page = dev_alloc_page();
+ page = dev_alloc_pages(i40e_rx_pg_order(rx_ring));
if (unlikely(!page)) {
rx_ring->rx_stats.alloc_page_failed++;
return false;
}
/* map page for use */
- dma = dma_map_page(rx_ring->dev, page, 0, PAGE_SIZE, DMA_FROM_DEVICE);
+ dma = dma_map_page_attrs(rx_ring->dev, page, 0,
+ i40e_rx_pg_size(rx_ring),
+ DMA_FROM_DEVICE,
+ I40E_RX_DMA_ATTR);
/* if mapping failed free memory back to system since
* there isn't much point in holding memory we can't use
*/
if (dma_mapping_error(rx_ring->dev, dma)) {
- __free_pages(page, 0);
+ __free_pages(page, i40e_rx_pg_order(rx_ring));
rx_ring->rx_stats.alloc_page_failed++;
return false;
}
bi->dma = dma;
bi->page = page;
- bi->page_offset = 0;
+ bi->page_offset = i40e_rx_offset(rx_ring);
+
+ /* initialize pagecnt_bias to 1 representing we fully own page */
+ bi->pagecnt_bias = 1;
return true;
}
@@ -702,6 +727,12 @@
if (!i40e_alloc_mapped_page(rx_ring, bi))
goto no_buffers;
+ /* sync the buffer for use by the device */
+ dma_sync_single_range_for_device(rx_ring->dev, bi->dma,
+ bi->page_offset,
+ rx_ring->rx_buf_len,
+ DMA_FROM_DEVICE);
+
/* Refresh the desc even if buffer_addrs didn't change
* because each write-back erases this info.
*/
@@ -742,8 +773,6 @@
* @vsi: the VSI we care about
* @skb: skb currently being received and modified
* @rx_desc: the receive descriptor
- *
- * skb->protocol must be set before this function is called
**/
static inline void i40e_rx_checksum(struct i40e_vsi *vsi,
struct sk_buff *skb,
@@ -806,13 +835,6 @@
if (rx_error & BIT(I40E_RX_DESC_ERROR_PPRS_SHIFT))
return;
- /* If there is an outer header present that might contain a checksum
- * we need to bump the checksum level by 1 to reflect the fact that
- * we are indicating we validated the inner checksum.
- */
- if (decoded.tunnel_type >= I40E_RX_PTYPE_TUNNEL_IP_GRENAT)
- skb->csum_level = 1;
-
/* Only report checksum unnecessary for TCP, UDP, or SCTP */
switch (decoded.inner_prot) {
case I40E_RX_PTYPE_INNER_PROT_TCP:
@@ -895,12 +917,12 @@
{
i40e_rx_hash(rx_ring, rx_desc, skb, rx_ptype);
- /* modifies the skb - consumes the enet header */
- skb->protocol = eth_type_trans(skb, rx_ring->netdev);
-
i40e_rx_checksum(rx_ring->vsi, skb, rx_desc);
skb_record_rx_queue(skb, rx_ring->queue_index);
+
+ /* modifies the skb - consumes the enet header */
+ skb->protocol = eth_type_trans(skb, rx_ring->netdev);
}
/**
@@ -945,7 +967,10 @@
rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;
/* transfer page from old buffer to new buffer */
- *new_buff = *old_buff;
+ new_buff->dma = old_buff->dma;
+ new_buff->page = old_buff->page;
+ new_buff->page_offset = old_buff->page_offset;
+ new_buff->pagecnt_bias = old_buff->pagecnt_bias;
}
/**
@@ -966,8 +991,6 @@
* the adapter for another receive
*
* @rx_buffer: buffer containing the page
- * @page: page address from rx_buffer
- * @truesize: actual size of the buffer in this page
*
* If page is reusable, rx_buffer->page_offset is adjusted to point to
* an unused region in the page.
@@ -990,13 +1013,10 @@
*
* In either case, if the page is reusable its refcount is increased.
**/
-static bool i40e_can_reuse_rx_page(struct i40e_rx_buffer *rx_buffer,
- struct page *page,
- const unsigned int truesize)
+static bool i40e_can_reuse_rx_page(struct i40e_rx_buffer *rx_buffer)
{
-#if (PAGE_SIZE >= 8192)
- unsigned int last_offset = PAGE_SIZE - I40E_RXBUFFER_2048;
-#endif
+ unsigned int pagecnt_bias = rx_buffer->pagecnt_bias;
+ struct page *page = rx_buffer->page;
/* Is any reuse possible? */
if (unlikely(!i40e_page_is_reusable(page)))
@@ -1004,21 +1024,23 @@
#if (PAGE_SIZE < 8192)
/* if we are only owner of page we can reuse it */
- if (unlikely(page_count(page) != 1))
+ if (unlikely((page_count(page) - pagecnt_bias) > 1))
return false;
-
- /* flip page offset to other buffer */
- rx_buffer->page_offset ^= truesize;
#else
- /* move offset up to the next cache line */
- rx_buffer->page_offset += truesize;
-
- if (rx_buffer->page_offset > last_offset)
+#define I40E_LAST_OFFSET \
+ (SKB_WITH_OVERHEAD(PAGE_SIZE) - I40E_RXBUFFER_2048)
+ if (rx_buffer->page_offset > I40E_LAST_OFFSET)
return false;
#endif
- /* Inc ref count on page before passing it up to the stack */
- get_page(page);
+ /* If we have drained the page fragment pool we need to update
+ * the pagecnt_bias and page count so that we fully restock the
+ * number of references the driver holds.
+ */
+ if (unlikely(!pagecnt_bias)) {
+ page_ref_add(page, USHRT_MAX);
+ rx_buffer->pagecnt_bias = USHRT_MAX;
+ }
return true;
}
@@ -1027,122 +1049,51 @@
* i40e_add_rx_frag - Add contents of Rx buffer to sk_buff
* @rx_ring: rx descriptor ring to transact packets on
* @rx_buffer: buffer containing page to add
- * @size: packet length from rx_desc
* @skb: sk_buff to place the data into
+ * @size: packet length from rx_desc
*
* This function will add the data contained in rx_buffer->page to the skb.
- * This is done either through a direct copy if the data in the buffer is
- * less than the skb header size, otherwise it will just attach the page as
- * a frag to the skb.
+ * It will just attach the page as a frag to the skb.
*
- * The function will then update the page offset if necessary and return
- * true if the buffer can be reused by the adapter.
+ * The function will then update the page offset.
**/
-static bool i40e_add_rx_frag(struct i40e_ring *rx_ring,
+static void i40e_add_rx_frag(struct i40e_ring *rx_ring,
struct i40e_rx_buffer *rx_buffer,
- unsigned int size,
- struct sk_buff *skb)
+ struct sk_buff *skb,
+ unsigned int size)
{
- struct page *page = rx_buffer->page;
- unsigned char *va = page_address(page) + rx_buffer->page_offset;
#if (PAGE_SIZE < 8192)
- unsigned int truesize = I40E_RXBUFFER_2048;
+ unsigned int truesize = i40e_rx_pg_size(rx_ring) / 2;
#else
- unsigned int truesize = ALIGN(size, L1_CACHE_BYTES);
+ unsigned int truesize = SKB_DATA_ALIGN(size + i40e_rx_offset(rx_ring));
#endif
- unsigned int pull_len;
- if (unlikely(skb_is_nonlinear(skb)))
- goto add_tail_frag;
+ skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buffer->page,
+ rx_buffer->page_offset, size, truesize);
- /* will the data fit in the skb we allocated? if so, just
- * copy it as it is pretty small anyway
- */
- if (size <= I40E_RX_HDR_SIZE) {
- memcpy(__skb_put(skb, size), va, ALIGN(size, sizeof(long)));
-
- /* page is reusable, we can reuse buffer as-is */
- if (likely(i40e_page_is_reusable(page)))
- return true;
-
- /* this page cannot be reused so discard it */
- __free_pages(page, 0);
- return false;
- }
-
- /* we need the header to contain the greater of either
- * ETH_HLEN or 60 bytes if the skb->len is less than
- * 60 for skb_pad.
- */
- pull_len = eth_get_headlen(va, I40E_RX_HDR_SIZE);
-
- /* align pull length to size of long to optimize
- * memcpy performance
- */
- memcpy(__skb_put(skb, pull_len), va, ALIGN(pull_len, sizeof(long)));
-
- /* update all of the pointers */
- va += pull_len;
- size -= pull_len;
-
-add_tail_frag:
- skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, page,
- (unsigned long)va & ~PAGE_MASK, size, truesize);
-
- return i40e_can_reuse_rx_page(rx_buffer, page, truesize);
+ /* page is being used so we must update the page offset */
+#if (PAGE_SIZE < 8192)
+ rx_buffer->page_offset ^= truesize;
+#else
+ rx_buffer->page_offset += truesize;
+#endif
}
/**
- * i40evf_fetch_rx_buffer - Allocate skb and populate it
+ * i40e_get_rx_buffer - Fetch Rx buffer and synchronize data for use
* @rx_ring: rx descriptor ring to transact packets on
- * @rx_desc: descriptor containing info written by hardware
+ * @size: size of buffer to add to skb
*
- * This function allocates an skb on the fly, and populates it with the page
- * data from the current receive descriptor, taking care to set up the skb
- * correctly, as well as handling calling the page recycle function if
- * necessary.
+ * This function will pull an Rx buffer from the ring and synchronize it
+ * for use by the CPU.
*/
-static inline
-struct sk_buff *i40evf_fetch_rx_buffer(struct i40e_ring *rx_ring,
- union i40e_rx_desc *rx_desc,
- struct sk_buff *skb)
+static struct i40e_rx_buffer *i40e_get_rx_buffer(struct i40e_ring *rx_ring,
+ const unsigned int size)
{
- u64 local_status_error_len =
- le64_to_cpu(rx_desc->wb.qword1.status_error_len);
- unsigned int size =
- (local_status_error_len & I40E_RXD_QW1_LENGTH_PBUF_MASK) >>
- I40E_RXD_QW1_LENGTH_PBUF_SHIFT;
struct i40e_rx_buffer *rx_buffer;
- struct page *page;
rx_buffer = &rx_ring->rx_bi[rx_ring->next_to_clean];
- page = rx_buffer->page;
- prefetchw(page);
-
- if (likely(!skb)) {
- void *page_addr = page_address(page) + rx_buffer->page_offset;
-
- /* prefetch first cache line of first page */
- prefetch(page_addr);
-#if L1_CACHE_BYTES < 128
- prefetch(page_addr + L1_CACHE_BYTES);
-#endif
-
- /* allocate a skb to store the frags */
- skb = __napi_alloc_skb(&rx_ring->q_vector->napi,
- I40E_RX_HDR_SIZE,
- GFP_ATOMIC | __GFP_NOWARN);
- if (unlikely(!skb)) {
- rx_ring->rx_stats.alloc_buff_failed++;
- return NULL;
- }
-
- /* we will be copying header into skb->data in
- * pskb_may_pull so it is in our interest to prefetch
- * it now to avoid a possible cache miss
- */
- prefetchw(skb->data);
- }
+ prefetchw(rx_buffer->page);
/* we are reusing so sync this buffer for CPU use */
dma_sync_single_range_for_cpu(rx_ring->dev,
@@ -1151,21 +1102,148 @@
size,
DMA_FROM_DEVICE);
- /* pull page into skb */
- if (i40e_add_rx_frag(rx_ring, rx_buffer, size, skb)) {
+ /* We have pulled a buffer for use, so decrement pagecnt_bias */
+ rx_buffer->pagecnt_bias--;
+
+ return rx_buffer;
+}
+
+/**
+ * i40e_construct_skb - Allocate skb and populate it
+ * @rx_ring: rx descriptor ring to transact packets on
+ * @rx_buffer: rx buffer to pull data from
+ * @size: size of buffer to add to skb
+ *
+ * This function allocates an skb. It then populates it with the page
+ * data from the current receive descriptor, taking care to set up the
+ * skb correctly.
+ */
+static struct sk_buff *i40e_construct_skb(struct i40e_ring *rx_ring,
+ struct i40e_rx_buffer *rx_buffer,
+ unsigned int size)
+{
+ void *va = page_address(rx_buffer->page) + rx_buffer->page_offset;
+#if (PAGE_SIZE < 8192)
+ unsigned int truesize = i40e_rx_pg_size(rx_ring) / 2;
+#else
+ unsigned int truesize = SKB_DATA_ALIGN(size);
+#endif
+ unsigned int headlen;
+ struct sk_buff *skb;
+
+ /* prefetch first cache line of first page */
+ prefetch(va);
+#if L1_CACHE_BYTES < 128
+ prefetch(va + L1_CACHE_BYTES);
+#endif
+
+ /* allocate a skb to store the frags */
+ skb = __napi_alloc_skb(&rx_ring->q_vector->napi,
+ I40E_RX_HDR_SIZE,
+ GFP_ATOMIC | __GFP_NOWARN);
+ if (unlikely(!skb))
+ return NULL;
+
+ /* Determine available headroom for copy */
+ headlen = size;
+ if (headlen > I40E_RX_HDR_SIZE)
+ headlen = eth_get_headlen(va, I40E_RX_HDR_SIZE);
+
+ /* align pull length to size of long to optimize memcpy performance */
+ memcpy(__skb_put(skb, headlen), va, ALIGN(headlen, sizeof(long)));
+
+ /* update all of the pointers */
+ size -= headlen;
+ if (size) {
+ skb_add_rx_frag(skb, 0, rx_buffer->page,
+ rx_buffer->page_offset + headlen,
+ size, truesize);
+
+ /* buffer is used by skb, update page_offset */
+#if (PAGE_SIZE < 8192)
+ rx_buffer->page_offset ^= truesize;
+#else
+ rx_buffer->page_offset += truesize;
+#endif
+ } else {
+ /* buffer is unused, reset bias back to rx_buffer */
+ rx_buffer->pagecnt_bias++;
+ }
+
+ return skb;
+}
+
+/**
+ * i40e_build_skb - Build skb around an existing buffer
+ * @rx_ring: Rx descriptor ring to transact packets on
+ * @rx_buffer: Rx buffer to pull data from
+ * @size: size of buffer to add to skb
+ *
+ * This function builds an skb around an existing Rx buffer, taking care
+ * to set up the skb correctly and avoid any memcpy overhead.
+ */
+static struct sk_buff *i40e_build_skb(struct i40e_ring *rx_ring,
+ struct i40e_rx_buffer *rx_buffer,
+ unsigned int size)
+{
+ void *va = page_address(rx_buffer->page) + rx_buffer->page_offset;
+#if (PAGE_SIZE < 8192)
+ unsigned int truesize = i40e_rx_pg_size(rx_ring) / 2;
+#else
+ unsigned int truesize = SKB_DATA_ALIGN(size);
+#endif
+ struct sk_buff *skb;
+
+ /* prefetch first cache line of first page */
+ prefetch(va);
+#if L1_CACHE_BYTES < 128
+ prefetch(va + L1_CACHE_BYTES);
+#endif
+ /* build an skb around the page buffer */
+ skb = build_skb(va - I40E_SKB_PAD, truesize);
+ if (unlikely(!skb))
+ return NULL;
+
+ /* update pointers within the skb to store the data */
+ skb_reserve(skb, I40E_SKB_PAD);
+ __skb_put(skb, size);
+
+ /* buffer is used by skb, update page_offset */
+#if (PAGE_SIZE < 8192)
+ rx_buffer->page_offset ^= truesize;
+#else
+ rx_buffer->page_offset += truesize;
+#endif
+
+ return skb;
+}
+
+/**
+ * i40e_put_rx_buffer - Clean up used buffer and either recycle or free
+ * @rx_ring: rx descriptor ring to transact packets on
+ * @rx_buffer: rx buffer to pull data from
+ *
+ * This function will clean up the contents of the rx_buffer. It will
+ * either recycle the bufer or unmap it and free the associated resources.
+ */
+static void i40e_put_rx_buffer(struct i40e_ring *rx_ring,
+ struct i40e_rx_buffer *rx_buffer)
+{
+ if (i40e_can_reuse_rx_page(rx_buffer)) {
/* hand second half of page back to the ring */
i40e_reuse_rx_page(rx_ring, rx_buffer);
rx_ring->rx_stats.page_reuse_count++;
} else {
/* we are not reusing the buffer so unmap it */
- dma_unmap_page(rx_ring->dev, rx_buffer->dma, PAGE_SIZE,
- DMA_FROM_DEVICE);
+ dma_unmap_page_attrs(rx_ring->dev, rx_buffer->dma,
+ i40e_rx_pg_size(rx_ring),
+ DMA_FROM_DEVICE, I40E_RX_DMA_ATTR);
+ __page_frag_cache_drain(rx_buffer->page,
+ rx_buffer->pagecnt_bias);
}
/* clear contents of buffer_info */
rx_buffer->page = NULL;
-
- return skb;
}
/**
@@ -1221,7 +1299,9 @@
bool failure = false;
while (likely(total_rx_packets < budget)) {
+ struct i40e_rx_buffer *rx_buffer;
union i40e_rx_desc *rx_desc;
+ unsigned int size;
u16 vlan_tag;
u8 rx_ptype;
u64 qword;
@@ -1238,22 +1318,40 @@
/* status_error_len will always be zero for unused descriptors
* because it's cleared in cleanup, and overlaps with hdr_addr
* which is always zero because packet split isn't used, if the
- * hardware wrote DD then it will be non-zero
+ * hardware wrote DD then the length will be non-zero
*/
- if (!i40e_test_staterr(rx_desc,
- BIT(I40E_RX_DESC_STATUS_DD_SHIFT)))
- break;
+ qword = le64_to_cpu(rx_desc->wb.qword1.status_error_len);
/* This memory barrier is needed to keep us from reading
- * any other fields out of the rx_desc until we know the
- * DD bit is set.
+ * any other fields out of the rx_desc until we have
+ * verified the descriptor has been written back.
*/
dma_rmb();
- skb = i40evf_fetch_rx_buffer(rx_ring, rx_desc, skb);
- if (!skb)
+ size = (qword & I40E_RXD_QW1_LENGTH_PBUF_MASK) >>
+ I40E_RXD_QW1_LENGTH_PBUF_SHIFT;
+ if (!size)
break;
+ i40e_trace(clean_rx_irq, rx_ring, rx_desc, skb);
+ rx_buffer = i40e_get_rx_buffer(rx_ring, size);
+
+ /* retrieve a buffer from the ring */
+ if (skb)
+ i40e_add_rx_frag(rx_ring, rx_buffer, skb, size);
+ else if (ring_uses_build_skb(rx_ring))
+ skb = i40e_build_skb(rx_ring, rx_buffer, size);
+ else
+ skb = i40e_construct_skb(rx_ring, rx_buffer, size);
+
+ /* exit if we failed to retrieve a buffer */
+ if (!skb) {
+ rx_ring->rx_stats.alloc_buff_failed++;
+ rx_buffer->pagecnt_bias++;
+ break;
+ }
+
+ i40e_put_rx_buffer(rx_ring, rx_buffer);
cleaned_count++;
if (i40e_is_non_eop(rx_ring, rx_desc, skb))
@@ -1266,6 +1364,7 @@
*/
if (unlikely(i40e_test_staterr(rx_desc, BIT(I40E_RXD_QW1_ERROR_SHIFT)))) {
dev_kfree_skb_any(skb);
+ skb = NULL;
continue;
}
@@ -1288,6 +1387,7 @@
vlan_tag = (qword & BIT(I40E_RX_DESC_STATUS_L2TAG1P_SHIFT)) ?
le16_to_cpu(rx_desc->wb.qword0.lo_dword.l2tag1) : 0;
+ i40e_trace(clean_rx_irq_rx, rx_ring, rx_desc, skb);
i40e_receive_skb(rx_ring, skb, vlan_tag);
skb = NULL;
@@ -1408,7 +1508,7 @@
}
enable_int:
- if (!test_bit(__I40E_DOWN, &vsi->state))
+ if (!test_bit(__I40E_VSI_DOWN, vsi->state))
wr32(hw, INTREG(vector - 1), txval);
if (q_vector->itr_countdown)
@@ -1437,7 +1537,7 @@
int budget_per_ring;
int work_done = 0;
- if (test_bit(__I40E_DOWN, &vsi->state)) {
+ if (test_bit(__I40E_VSI_DOWN, vsi->state)) {
napi_complete(napi);
return 0;
}
@@ -1980,7 +2080,6 @@
u16 i = tx_ring->next_to_use;
u32 td_tag = 0;
dma_addr_t dma;
- u16 desc_count = 1;
if (tx_flags & I40E_TX_FLAGS_HW_VLAN) {
td_cmd |= I40E_TX_DESC_CMD_IL2TAG1;
@@ -2016,7 +2115,6 @@
tx_desc++;
i++;
- desc_count++;
if (i == tx_ring->count) {
tx_desc = I40E_TX_DESC(tx_ring, 0);
@@ -2038,7 +2136,6 @@
tx_desc++;
i++;
- desc_count++;
if (i == tx_ring->count) {
tx_desc = I40E_TX_DESC(tx_ring, 0);
@@ -2064,46 +2161,8 @@
i40e_maybe_stop_tx(tx_ring, DESC_NEEDED);
- /* write last descriptor with EOP bit */
- td_cmd |= I40E_TX_DESC_CMD_EOP;
-
- /* We can OR these values together as they both are checked against
- * 4 below and at this point desc_count will be used as a boolean value
- * after this if/else block.
- */
- desc_count |= ++tx_ring->packet_stride;
-
- /* Algorithm to optimize tail and RS bit setting:
- * if queue is stopped
- * mark RS bit
- * reset packet counter
- * else if xmit_more is supported and is true
- * advance packet counter to 4
- * reset desc_count to 0
- *
- * if desc_count >= 4
- * mark RS bit
- * reset packet counter
- * if desc_count > 0
- * update tail
- *
- * Note: If there are less than 4 descriptors
- * pending and interrupts were disabled the service task will
- * trigger a force WB.
- */
- if (netif_xmit_stopped(txring_txq(tx_ring))) {
- goto do_rs;
- } else if (skb->xmit_more) {
- /* set stride to arm on next packet and reset desc_count */
- tx_ring->packet_stride = WB_STRIDE;
- desc_count = 0;
- } else if (desc_count >= WB_STRIDE) {
-do_rs:
- /* write last descriptor with RS bit set */
- td_cmd |= I40E_TX_DESC_CMD_RS;
- tx_ring->packet_stride = 0;
- }
-
+ /* write last descriptor with RS and EOP bits */
+ td_cmd |= I40E_TXD_CMD;
tx_desc->cmd_type_offset_bsz =
build_ctob(td_cmd, td_offset, size, td_tag);
@@ -2119,7 +2178,7 @@
first->next_to_watch = tx_desc;
/* notify HW of packet */
- if (desc_count) {
+ if (netif_xmit_stopped(txring_txq(tx_ring)) || !skb->xmit_more) {
writel(i, tx_ring->tail);
/* we need this if more than one processor can write to our tail
@@ -2170,6 +2229,8 @@
/* prefetch the data, we'll need it later */
prefetch(skb->data);
+ i40e_trace(xmit_frame_ring, skb, tx_ring);
+
count = i40e_xmit_descriptor_count(skb);
if (i40e_chk_linearize(skb, count)) {
if (__skb_linearize(skb)) {
@@ -2237,6 +2298,7 @@
return NETDEV_TX_OK;
out_drop:
+ i40e_trace(xmit_frame_ring_drop, first->skb, tx_ring);
dev_kfree_skb_any(first->skb);
first->skb = NULL;
return NETDEV_TX_OK;
diff --git a/drivers/net/ethernet/intel/i40evf/i40e_txrx.h b/drivers/net/ethernet/intel/i40evf/i40e_txrx.h
index 8274ba6..901282c 100644
--- a/drivers/net/ethernet/intel/i40evf/i40e_txrx.h
+++ b/drivers/net/ethernet/intel/i40evf/i40e_txrx.h
@@ -104,10 +104,9 @@
/* Supported Rx Buffer Sizes (a multiple of 128) */
#define I40E_RXBUFFER_256 256
+#define I40E_RXBUFFER_1536 1536 /* 128B aligned standard Ethernet frame */
#define I40E_RXBUFFER_2048 2048
-#define I40E_RXBUFFER_3072 3072 /* For FCoE MTU of 2158 */
-#define I40E_RXBUFFER_4096 4096
-#define I40E_RXBUFFER_8192 8192
+#define I40E_RXBUFFER_3072 3072 /* Used for large frames w/ padding */
#define I40E_MAX_RXBUFFER 9728 /* largest size for single descriptor */
/* NOTE: netdev_alloc_skb reserves up to 64 bytes, NET_IP_ALIGN means we
@@ -120,6 +119,61 @@
#define I40E_RX_HDR_SIZE I40E_RXBUFFER_256
#define i40e_rx_desc i40e_32byte_rx_desc
+#define I40E_RX_DMA_ATTR \
+ (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING)
+
+/* Attempt to maximize the headroom available for incoming frames. We
+ * use a 2K buffer for receives and need 1536/1534 to store the data for
+ * the frame. This leaves us with 512 bytes of room. From that we need
+ * to deduct the space needed for the shared info and the padding needed
+ * to IP align the frame.
+ *
+ * Note: For cache line sizes 256 or larger this value is going to end
+ * up negative. In these cases we should fall back to the legacy
+ * receive path.
+ */
+#if (PAGE_SIZE < 8192)
+#define I40E_2K_TOO_SMALL_WITH_PADDING \
+((NET_SKB_PAD + I40E_RXBUFFER_1536) > SKB_WITH_OVERHEAD(I40E_RXBUFFER_2048))
+
+static inline int i40e_compute_pad(int rx_buf_len)
+{
+ int page_size, pad_size;
+
+ page_size = ALIGN(rx_buf_len, PAGE_SIZE / 2);
+ pad_size = SKB_WITH_OVERHEAD(page_size) - rx_buf_len;
+
+ return pad_size;
+}
+
+static inline int i40e_skb_pad(void)
+{
+ int rx_buf_len;
+
+ /* If a 2K buffer cannot handle a standard Ethernet frame then
+ * optimize padding for a 3K buffer instead of a 1.5K buffer.
+ *
+ * For a 3K buffer we need to add enough padding to allow for
+ * tailroom due to NET_IP_ALIGN possibly shifting us out of
+ * cache-line alignment.
+ */
+ if (I40E_2K_TOO_SMALL_WITH_PADDING)
+ rx_buf_len = I40E_RXBUFFER_3072 + SKB_DATA_ALIGN(NET_IP_ALIGN);
+ else
+ rx_buf_len = I40E_RXBUFFER_1536;
+
+ /* if needed make room for NET_IP_ALIGN */
+ rx_buf_len -= NET_IP_ALIGN;
+
+ return i40e_compute_pad(rx_buf_len);
+}
+
+#define I40E_SKB_PAD i40e_skb_pad()
+#else
+#define I40E_2K_TOO_SMALL_WITH_PADDING false
+#define I40E_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN)
+#endif
+
/**
* i40e_test_staterr - tests bits in Rx descriptor status and error fields
* @rx_desc: pointer to receive descriptor (in le64 format)
@@ -241,7 +295,12 @@
struct i40e_rx_buffer {
dma_addr_t dma;
struct page *page;
- unsigned int page_offset;
+#if (BITS_PER_LONG > 32) || (PAGE_SIZE >= 65536)
+ __u32 page_offset;
+#else
+ __u16 page_offset;
+#endif
+ __u16 pagecnt_bias;
};
struct i40e_queue_stats {
@@ -321,7 +380,8 @@
u8 packet_stride;
u16 flags;
-#define I40E_TXR_FLAGS_WB_ON_ITR BIT(0)
+#define I40E_TXR_FLAGS_WB_ON_ITR BIT(0)
+#define I40E_RXR_FLAGS_BUILD_SKB_ENABLED BIT(1)
/* stats structs */
struct i40e_queue_stats stats;
@@ -349,6 +409,21 @@
*/
} ____cacheline_internodealigned_in_smp;
+static inline bool ring_uses_build_skb(struct i40e_ring *ring)
+{
+ return !!(ring->flags & I40E_RXR_FLAGS_BUILD_SKB_ENABLED);
+}
+
+static inline void set_ring_build_skb_enabled(struct i40e_ring *ring)
+{
+ ring->flags |= I40E_RXR_FLAGS_BUILD_SKB_ENABLED;
+}
+
+static inline void clear_ring_build_skb_enabled(struct i40e_ring *ring)
+{
+ ring->flags &= ~I40E_RXR_FLAGS_BUILD_SKB_ENABLED;
+}
+
enum i40e_latency_range {
I40E_LOWEST_LATENCY = 0,
I40E_LOW_LATENCY = 1,
@@ -370,6 +445,17 @@
#define i40e_for_each_ring(pos, head) \
for (pos = (head).ring; pos != NULL; pos = pos->next)
+static inline unsigned int i40e_rx_pg_order(struct i40e_ring *ring)
+{
+#if (PAGE_SIZE < 8192)
+ if (ring->rx_buf_len > (PAGE_SIZE / 2))
+ return 1;
+#endif
+ return 0;
+}
+
+#define i40e_rx_pg_size(_ring) (PAGE_SIZE << i40e_rx_pg_order(_ring))
+
bool i40evf_alloc_rx_buffers(struct i40e_ring *rxr, u16 cleaned_count);
netdev_tx_t i40evf_xmit_frame(struct sk_buff *skb, struct net_device *netdev);
void i40evf_clean_tx_ring(struct i40e_ring *tx_ring);
@@ -385,20 +471,6 @@
bool __i40evf_chk_linearize(struct sk_buff *skb);
/**
- * i40e_get_head - Retrieve head from head writeback
- * @tx_ring: Tx ring to fetch head of
- *
- * Returns value of Tx ring head based on value stored
- * in head write-back location
- **/
-static inline u32 i40e_get_head(struct i40e_ring *tx_ring)
-{
- void *head = (struct i40e_tx_desc *)tx_ring->desc + tx_ring->count;
-
- return le32_to_cpu(*(volatile __le32 *)head);
-}
-
-/**
* i40e_xmit_descriptor_count - calculate number of Tx descriptors needed
* @skb: send buffer
* @tx_ring: ring to send buffer on
@@ -460,19 +532,7 @@
/* we can support up to 8 data buffers for a single send */
return count != I40E_MAX_BUFFER_TXD;
}
-
/**
- * i40e_rx_is_fcoe - returns true if the Rx packet type is FCoE
- * @ptype: the packet type field from Rx descriptor write-back
- **/
-static inline bool i40e_rx_is_fcoe(u16 ptype)
-{
- return (ptype >= I40E_RX_PTYPE_L2_FCOE_PAY3) &&
- (ptype <= I40E_RX_PTYPE_L2_FCOE_VFT_FCOTHER);
-}
-
-/**
- * txring_txq - Find the netdev Tx ring based on the i40e Tx ring
* @ring: Tx ring to find the netdev equivalent of
**/
static inline struct netdev_queue *txring_txq(const struct i40e_ring *ring)
diff --git a/drivers/net/ethernet/intel/i40evf/i40e_type.h b/drivers/net/ethernet/intel/i40evf/i40e_type.h
index 16bb880..bde7f24 100644
--- a/drivers/net/ethernet/intel/i40evf/i40e_type.h
+++ b/drivers/net/ethernet/intel/i40evf/i40e_type.h
@@ -78,6 +78,7 @@
I40E_DEBUG_DCB = 0x00000400,
I40E_DEBUG_DIAG = 0x00000800,
I40E_DEBUG_FD = 0x00001000,
+ I40E_DEBUG_PACKAGE = 0x00002000,
I40E_DEBUG_AQ_MESSAGE = 0x01000000,
I40E_DEBUG_AQ_DESCRIPTOR = 0x02000000,
@@ -1396,4 +1397,83 @@
#define I40E_FD_INSET_FLEX_WORD57_SHIFT 10
#define I40E_FD_INSET_FLEX_WORD57_MASK (0x1ULL << \
I40E_FD_INSET_FLEX_WORD57_SHIFT)
+
+/* Version format for PPP */
+struct i40e_ppp_version {
+ u8 major;
+ u8 minor;
+ u8 update;
+ u8 draft;
+};
+
+#define I40E_PPP_NAME_SIZE 32
+
+/* Package header */
+struct i40e_package_header {
+ struct i40e_ppp_version version;
+ u32 segment_count;
+ u32 segment_offset[1];
+};
+
+/* Generic segment header */
+struct i40e_generic_seg_header {
+#define SEGMENT_TYPE_METADATA 0x00000001
+#define SEGMENT_TYPE_NOTES 0x00000002
+#define SEGMENT_TYPE_I40E 0x00000011
+#define SEGMENT_TYPE_X722 0x00000012
+ u32 type;
+ struct i40e_ppp_version version;
+ u32 size;
+ char name[I40E_PPP_NAME_SIZE];
+};
+
+struct i40e_metadata_segment {
+ struct i40e_generic_seg_header header;
+ struct i40e_ppp_version version;
+ u32 track_id;
+ char name[I40E_PPP_NAME_SIZE];
+};
+
+struct i40e_device_id_entry {
+ u32 vendor_dev_id;
+ u32 sub_vendor_dev_id;
+};
+
+struct i40e_profile_segment {
+ struct i40e_generic_seg_header header;
+ struct i40e_ppp_version version;
+ char name[I40E_PPP_NAME_SIZE];
+ u32 device_table_count;
+ struct i40e_device_id_entry device_table[1];
+};
+
+struct i40e_section_table {
+ u32 section_count;
+ u32 section_offset[1];
+};
+
+struct i40e_profile_section_header {
+ u16 tbl_size;
+ u16 data_end;
+ struct {
+#define SECTION_TYPE_INFO 0x00000010
+#define SECTION_TYPE_MMIO 0x00000800
+#define SECTION_TYPE_AQ 0x00000801
+#define SECTION_TYPE_NOTE 0x80000000
+#define SECTION_TYPE_NAME 0x80000001
+ u32 type;
+ u32 offset;
+ u32 size;
+ } section;
+};
+
+struct i40e_profile_info {
+ u32 track_id;
+ struct i40e_ppp_version version;
+ u8 op;
+#define I40E_PPP_ADD_TRACKID 0x01
+#define I40E_PPP_REMOVE_TRACKID 0x02
+ u8 reserved[7];
+ u8 name[I40E_PPP_NAME_SIZE];
+};
#endif /* _I40E_TYPE_H_ */
diff --git a/drivers/net/ethernet/intel/i40evf/i40e_virtchnl.h b/drivers/net/ethernet/intel/i40evf/i40e_virtchnl.h
index d38a2b2..c5ad038 100644
--- a/drivers/net/ethernet/intel/i40evf/i40e_virtchnl.h
+++ b/drivers/net/ethernet/intel/i40evf/i40e_virtchnl.h
@@ -81,7 +81,9 @@
I40E_VIRTCHNL_OP_GET_STATS = 15,
I40E_VIRTCHNL_OP_FCOE = 16,
I40E_VIRTCHNL_OP_EVENT = 17, /* must ALWAYS be 17 */
+ I40E_VIRTCHNL_OP_IWARP = 20,
I40E_VIRTCHNL_OP_CONFIG_IWARP_IRQ_MAP = 21,
+ I40E_VIRTCHNL_OP_RELEASE_IWARP_IRQ_MAP = 22,
I40E_VIRTCHNL_OP_CONFIG_RSS_KEY = 23,
I40E_VIRTCHNL_OP_CONFIG_RSS_LUT = 24,
I40E_VIRTCHNL_OP_GET_RSS_HENA_CAPS = 25,
@@ -161,7 +163,8 @@
#define I40E_VIRTCHNL_VF_OFFLOAD_RX_POLLING 0x00020000
#define I40E_VIRTCHNL_VF_OFFLOAD_RSS_PCTYPE_V2 0x00040000
#define I40E_VIRTCHNL_VF_OFFLOAD_RSS_PF 0X00080000
-#define I40E_VIRTCHNL_VF_OFFLOAD_ENCAP_CSUM 0X00100000
+#define I40E_VIRTCHNL_VF_OFFLOAD_ENCAP 0X00100000
+#define I40E_VIRTCHNL_VF_OFFLOAD_ENCAP_CSUM 0X00200000
#define I40E_VF_BASE_MODE_OFFLOADS (I40E_VIRTCHNL_VF_OFFLOAD_L2 | \
I40E_VIRTCHNL_VF_OFFLOAD_VLAN | \
@@ -393,6 +396,37 @@
int severity;
};
+/* I40E_VIRTCHNL_OP_CONFIG_IWARP_IRQ_MAP
+ * VF uses this message to request PF to map IWARP vectors to IWARP queues.
+ * The request for this originates from the VF IWARP driver through
+ * a client interface between VF LAN and VF IWARP driver.
+ * A vector could have an AEQ and CEQ attached to it although
+ * there is a single AEQ per VF IWARP instance in which case
+ * most vectors will have an INVALID_IDX for aeq and valid idx for ceq.
+ * There will never be a case where there will be multiple CEQs attached
+ * to a single vector.
+ * PF configures interrupt mapping and returns status.
+ */
+
+/* HW does not define a type value for AEQ; only for RX/TX and CEQ.
+ * In order for us to keep the interface simple, SW will define a
+ * unique type value for AEQ.
+ */
+#define I40E_QUEUE_TYPE_PE_AEQ 0x80
+#define I40E_QUEUE_INVALID_IDX 0xFFFF
+
+struct i40e_virtchnl_iwarp_qv_info {
+ u32 v_idx; /* msix_vector */
+ u16 ceq_idx;
+ u16 aeq_idx;
+ u8 itr_idx;
+};
+
+struct i40e_virtchnl_iwarp_qvlist_info {
+ u32 num_vectors;
+ struct i40e_virtchnl_iwarp_qv_info qv_info[1];
+};
+
/* VF reset states - these are written into the RSTAT register:
* I40E_VFGEN_RSTAT1 on the PF
* I40E_VFGEN_RSTAT on the VF
diff --git a/drivers/net/ethernet/intel/i40evf/i40evf.h b/drivers/net/ethernet/intel/i40evf/i40evf.h
index 00c42d8..b8ada6d 100644
--- a/drivers/net/ethernet/intel/i40evf/i40evf.h
+++ b/drivers/net/ethernet/intel/i40evf/i40evf.h
@@ -49,6 +49,13 @@
#define DEFAULT_DEBUG_LEVEL_SHIFT 3
#define PFX "i40evf: "
+/* VSI state flags shared with common code */
+enum i40evf_vsi_state_t {
+ __I40E_VSI_DOWN,
+ /* This must be last as it determines the size of the BITMAP */
+ __I40E_VSI_STATE_SIZE__,
+};
+
/* dummy struct to make common code less painful */
struct i40e_vsi {
struct i40evf_adapter *back;
@@ -56,10 +63,11 @@
unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)];
u16 seid;
u16 id;
- unsigned long state;
+ DECLARE_BITMAP(state, __I40E_VSI_STATE_SIZE__);
int base_vector;
u16 work_limit;
u16 qs_handle;
+ void *priv; /* client driver data reference. */
};
/* How many Rx Buffers do we bundle into one write to the hardware ? */
@@ -71,10 +79,6 @@
#define I40EVF_MAX_RXD 4096
#define I40EVF_MIN_RXD 64
#define I40EVF_REQ_DESCRIPTOR_MULTIPLE 32
-
-/* Supported Rx Buffer Sizes */
-#define I40EVF_RXBUFFER_2048 2048
-#define I40EVF_MAX_RXBUFFER 16384 /* largest size for single descriptor */
#define I40EVF_MAX_AQ_BUF_SIZE 4096
#define I40EVF_AQ_LEN 32
#define I40EVF_AQ_MAX_ERR 20 /* times to try before resetting AQ */
@@ -169,15 +173,15 @@
enum i40evf_critical_section_t {
__I40EVF_IN_CRITICAL_TASK, /* cannot be interrupted */
+ __I40EVF_IN_CLIENT_TASK,
};
-/* make common code happy */
-#define __I40E_DOWN __I40EVF_DOWN
/* board specific private data structure */
struct i40evf_adapter {
struct timer_list watchdog_timer;
struct work_struct reset_task;
struct work_struct adminq_task;
+ struct delayed_work client_task;
struct delayed_work init_task;
struct i40e_q_vector *q_vectors;
struct list_head vlan_filter_list;
@@ -195,15 +199,16 @@
u64 hw_csum_rx_error;
u32 rx_desc_count;
int num_msix_vectors;
+ int num_iwarp_msix;
+ int iwarp_base_vector;
u32 client_pending;
+ struct i40e_client_instance *cinst;
struct msix_entry *msix_entries;
u32 flags;
#define I40EVF_FLAG_RX_CSUM_ENABLED BIT(0)
-#define I40EVF_FLAG_IN_NETPOLL BIT(4)
#define I40EVF_FLAG_IMIR_ENABLED BIT(5)
#define I40EVF_FLAG_MQ_CAPABLE BIT(6)
-#define I40EVF_FLAG_NEED_LINK_UPDATE BIT(7)
#define I40EVF_FLAG_PF_COMMS_FAILED BIT(8)
#define I40EVF_FLAG_RESET_PENDING BIT(9)
#define I40EVF_FLAG_RESET_NEEDED BIT(10)
@@ -211,15 +216,18 @@
#define I40EVF_FLAG_OUTER_UDP_CSUM_CAPABLE BIT(12)
#define I40EVF_FLAG_ADDR_SET_BY_PF BIT(13)
#define I40EVF_FLAG_SERVICE_CLIENT_REQUESTED BIT(14)
-#define I40EVF_FLAG_PROMISC_ON BIT(15)
-#define I40EVF_FLAG_ALLMULTI_ON BIT(16)
+#define I40EVF_FLAG_CLIENT_NEEDS_OPEN BIT(15)
+#define I40EVF_FLAG_CLIENT_NEEDS_CLOSE BIT(16)
+#define I40EVF_FLAG_CLIENT_NEEDS_L2_PARAMS BIT(17)
+#define I40EVF_FLAG_PROMISC_ON BIT(18)
+#define I40EVF_FLAG_ALLMULTI_ON BIT(19)
+#define I40EVF_FLAG_LEGACY_RX BIT(20)
/* duplicates for common code */
-#define I40E_FLAG_FDIR_ATR_ENABLED 0
#define I40E_FLAG_DCB_ENABLED 0
-#define I40E_FLAG_IN_NETPOLL I40EVF_FLAG_IN_NETPOLL
#define I40E_FLAG_RX_CSUM_ENABLED I40EVF_FLAG_RX_CSUM_ENABLED
#define I40E_FLAG_WB_ON_ITR_CAPABLE I40EVF_FLAG_WB_ON_ITR_CAPABLE
#define I40E_FLAG_OUTER_UDP_CSUM_CAPABLE I40EVF_FLAG_OUTER_UDP_CSUM_CAPABLE
+#define I40E_FLAG_LEGACY_RX I40EVF_FLAG_LEGACY_RX
/* flags for admin queue service task */
u32 aq_required;
#define I40EVF_FLAG_AQ_ENABLE_QUEUES BIT(0)
@@ -246,7 +254,6 @@
/* OS defined structs */
struct net_device *netdev;
struct pci_dev *pdev;
- struct net_device_stats net_stats;
struct i40e_hw hw; /* defined in i40e_type.h */
@@ -258,10 +265,11 @@
bool link_up;
enum i40e_aq_link_speed link_speed;
enum i40e_virtchnl_ops current_op;
-#define CLIENT_ENABLED(_a) ((_a)->vf_res ? \
+#define CLIENT_ALLOWED(_a) ((_a)->vf_res ? \
(_a)->vf_res->vf_offload_flags & \
I40E_VIRTCHNL_VF_OFFLOAD_IWARP : \
0)
+#define CLIENT_ENABLED(_a) ((_a)->cinst)
/* RSS by the PF should be preferred over RSS via other methods. */
#define RSS_PF(_a) ((_a)->vf_res->vf_offload_flags & \
I40E_VIRTCHNL_VF_OFFLOAD_RSS_PF)
@@ -292,6 +300,12 @@
/* Ethtool Private Flags */
+/* lan device */
+struct i40e_device {
+ struct list_head list;
+ struct i40evf_adapter *vf;
+};
+
/* needed by i40evf_ethtool.c */
extern char i40evf_driver_name[];
extern const char i40evf_driver_version[];
@@ -337,4 +351,11 @@
enum i40e_virtchnl_ops v_opcode,
i40e_status v_retval, u8 *msg, u16 msglen);
int i40evf_config_rss(struct i40evf_adapter *adapter);
+int i40evf_lan_add_device(struct i40evf_adapter *adapter);
+int i40evf_lan_del_device(struct i40evf_adapter *adapter);
+void i40evf_client_subtask(struct i40evf_adapter *adapter);
+void i40evf_notify_client_message(struct i40e_vsi *vsi, u8 *msg, u16 len);
+void i40evf_notify_client_l2_params(struct i40e_vsi *vsi);
+void i40evf_notify_client_open(struct i40e_vsi *vsi);
+void i40evf_notify_client_close(struct i40e_vsi *vsi, bool reset);
#endif /* _I40EVF_H_ */
diff --git a/drivers/net/ethernet/intel/i40evf/i40evf_client.c b/drivers/net/ethernet/intel/i40evf/i40evf_client.c
new file mode 100644
index 0000000..ee73768
--- /dev/null
+++ b/drivers/net/ethernet/intel/i40evf/i40evf_client.c
@@ -0,0 +1,564 @@
+#include <linux/list.h>
+#include <linux/errno.h>
+
+#include "i40evf.h"
+#include "i40e_prototype.h"
+#include "i40evf_client.h"
+
+static
+const char i40evf_client_interface_version_str[] = I40EVF_CLIENT_VERSION_STR;
+static struct i40e_client *vf_registered_client;
+static LIST_HEAD(i40evf_devices);
+static DEFINE_MUTEX(i40evf_device_mutex);
+
+static u32 i40evf_client_virtchnl_send(struct i40e_info *ldev,
+ struct i40e_client *client,
+ u8 *msg, u16 len);
+
+static int i40evf_client_setup_qvlist(struct i40e_info *ldev,
+ struct i40e_client *client,
+ struct i40e_qvlist_info *qvlist_info);
+
+static struct i40e_ops i40evf_lan_ops = {
+ .virtchnl_send = i40evf_client_virtchnl_send,
+ .setup_qvlist = i40evf_client_setup_qvlist,
+};
+
+/**
+ * i40evf_notify_client_message - call the client message receive callback
+ * @vsi: the VSI associated with this client
+ * @msg: message buffer
+ * @len: length of message
+ *
+ * If there is a client to this VSI, call the client
+ **/
+void i40evf_notify_client_message(struct i40e_vsi *vsi, u8 *msg, u16 len)
+{
+ struct i40e_client_instance *cinst;
+
+ if (!vsi)
+ return;
+
+ cinst = vsi->back->cinst;
+ if (!cinst || !cinst->client || !cinst->client->ops ||
+ !cinst->client->ops->virtchnl_receive) {
+ dev_dbg(&vsi->back->pdev->dev,
+ "Cannot locate client instance virtchnl_receive function\n");
+ return;
+ }
+ cinst->client->ops->virtchnl_receive(&cinst->lan_info, cinst->client,
+ msg, len);
+}
+
+/**
+ * i40evf_notify_client_l2_params - call the client notify callback
+ * @vsi: the VSI with l2 param changes
+ *
+ * If there is a client to this VSI, call the client
+ **/
+void i40evf_notify_client_l2_params(struct i40e_vsi *vsi)
+{
+ struct i40e_client_instance *cinst;
+ struct i40e_params params;
+
+ if (!vsi)
+ return;
+
+ cinst = vsi->back->cinst;
+ memset(¶ms, 0, sizeof(params));
+ params.mtu = vsi->netdev->mtu;
+ params.link_up = vsi->back->link_up;
+ params.qos.prio_qos[0].qs_handle = vsi->qs_handle;
+
+ if (!cinst || !cinst->client || !cinst->client->ops ||
+ !cinst->client->ops->l2_param_change) {
+ dev_dbg(&vsi->back->pdev->dev,
+ "Cannot locate client instance l2_param_change function\n");
+ return;
+ }
+ cinst->client->ops->l2_param_change(&cinst->lan_info, cinst->client,
+ ¶ms);
+}
+
+/**
+ * i40evf_notify_client_open - call the client open callback
+ * @vsi: the VSI with netdev opened
+ *
+ * If there is a client to this netdev, call the client with open
+ **/
+void i40evf_notify_client_open(struct i40e_vsi *vsi)
+{
+ struct i40evf_adapter *adapter = vsi->back;
+ struct i40e_client_instance *cinst = adapter->cinst;
+ int ret;
+
+ if (!cinst || !cinst->client || !cinst->client->ops ||
+ !cinst->client->ops->open) {
+ dev_dbg(&vsi->back->pdev->dev,
+ "Cannot locate client instance open function\n");
+ return;
+ }
+ if (!(test_bit(__I40E_CLIENT_INSTANCE_OPENED, &cinst->state))) {
+ ret = cinst->client->ops->open(&cinst->lan_info, cinst->client);
+ if (!ret)
+ set_bit(__I40E_CLIENT_INSTANCE_OPENED, &cinst->state);
+ }
+}
+
+/**
+ * i40evf_client_release_qvlist - send a message to the PF to release iwarp qv map
+ * @ldev: pointer to L2 context.
+ *
+ * Return 0 on success or < 0 on error
+ **/
+static int i40evf_client_release_qvlist(struct i40e_info *ldev)
+{
+ struct i40evf_adapter *adapter = ldev->vf;
+ i40e_status err;
+
+ if (adapter->aq_required)
+ return -EAGAIN;
+
+ err = i40e_aq_send_msg_to_pf(&adapter->hw,
+ I40E_VIRTCHNL_OP_RELEASE_IWARP_IRQ_MAP,
+ I40E_SUCCESS, NULL, 0, NULL);
+
+ if (err)
+ dev_err(&adapter->pdev->dev,
+ "Unable to send iWarp vector release message to PF, error %d, aq status %d\n",
+ err, adapter->hw.aq.asq_last_status);
+
+ return err;
+}
+
+/**
+ * i40evf_notify_client_close - call the client close callback
+ * @vsi: the VSI with netdev closed
+ * @reset: true when close called due to reset pending
+ *
+ * If there is a client to this netdev, call the client with close
+ **/
+void i40evf_notify_client_close(struct i40e_vsi *vsi, bool reset)
+{
+ struct i40evf_adapter *adapter = vsi->back;
+ struct i40e_client_instance *cinst = adapter->cinst;
+
+ if (!cinst || !cinst->client || !cinst->client->ops ||
+ !cinst->client->ops->close) {
+ dev_dbg(&vsi->back->pdev->dev,
+ "Cannot locate client instance close function\n");
+ return;
+ }
+ cinst->client->ops->close(&cinst->lan_info, cinst->client, reset);
+ i40evf_client_release_qvlist(&cinst->lan_info);
+ clear_bit(__I40E_CLIENT_INSTANCE_OPENED, &cinst->state);
+}
+
+/**
+ * i40evf_client_add_instance - add a client instance to the instance list
+ * @adapter: pointer to the board struct
+ * @client: pointer to a client struct in the client list.
+ *
+ * Returns cinst ptr on success, NULL on failure
+ **/
+static struct i40e_client_instance *
+i40evf_client_add_instance(struct i40evf_adapter *adapter)
+{
+ struct i40e_client_instance *cinst = NULL;
+ struct netdev_hw_addr *mac = NULL;
+ struct i40e_vsi *vsi = &adapter->vsi;
+ int i;
+
+ if (!vf_registered_client)
+ goto out;
+
+ if (adapter->cinst) {
+ cinst = adapter->cinst;
+ goto out;
+ }
+
+ cinst = kzalloc(sizeof(*cinst), GFP_KERNEL);
+ if (!cinst)
+ goto out;
+
+ cinst->lan_info.vf = (void *)adapter;
+ cinst->lan_info.netdev = vsi->netdev;
+ cinst->lan_info.pcidev = adapter->pdev;
+ cinst->lan_info.fid = 0;
+ cinst->lan_info.ftype = I40E_CLIENT_FTYPE_VF;
+ cinst->lan_info.hw_addr = adapter->hw.hw_addr;
+ cinst->lan_info.ops = &i40evf_lan_ops;
+ cinst->lan_info.version.major = I40EVF_CLIENT_VERSION_MAJOR;
+ cinst->lan_info.version.minor = I40EVF_CLIENT_VERSION_MINOR;
+ cinst->lan_info.version.build = I40EVF_CLIENT_VERSION_BUILD;
+ set_bit(__I40E_CLIENT_INSTANCE_NONE, &cinst->state);
+
+ cinst->lan_info.msix_count = adapter->num_iwarp_msix;
+ cinst->lan_info.msix_entries =
+ &adapter->msix_entries[adapter->iwarp_base_vector];
+
+ for (i = 0; i < I40E_MAX_USER_PRIORITY; i++) {
+ cinst->lan_info.params.qos.prio_qos[i].tc = 0;
+ cinst->lan_info.params.qos.prio_qos[i].qs_handle =
+ vsi->qs_handle;
+ }
+
+ mac = list_first_entry(&cinst->lan_info.netdev->dev_addrs.list,
+ struct netdev_hw_addr, list);
+ if (mac)
+ ether_addr_copy(cinst->lan_info.lanmac, mac->addr);
+ else
+ dev_err(&adapter->pdev->dev, "MAC address list is empty!\n");
+
+ cinst->client = vf_registered_client;
+ adapter->cinst = cinst;
+out:
+ return cinst;
+}
+
+/**
+ * i40evf_client_del_instance - removes a client instance from the list
+ * @adapter: pointer to the board struct
+ * @client: pointer to the client struct
+ *
+ **/
+static
+void i40evf_client_del_instance(struct i40evf_adapter *adapter)
+{
+ kfree(adapter->cinst);
+ adapter->cinst = NULL;
+}
+
+/**
+ * i40evf_client_subtask - client maintenance work
+ * @adapter: board private structure
+ **/
+void i40evf_client_subtask(struct i40evf_adapter *adapter)
+{
+ struct i40e_client *client = vf_registered_client;
+ struct i40e_client_instance *cinst;
+ int ret = 0;
+
+ if (adapter->state < __I40EVF_DOWN)
+ return;
+
+ /* first check client is registered */
+ if (!client)
+ return;
+
+ /* Add the client instance to the instance list */
+ cinst = i40evf_client_add_instance(adapter);
+ if (!cinst)
+ return;
+
+ dev_info(&adapter->pdev->dev, "Added instance of Client %s\n",
+ client->name);
+
+ if (!test_bit(__I40E_CLIENT_INSTANCE_OPENED, &cinst->state)) {
+ /* Send an Open request to the client */
+
+ if (client->ops && client->ops->open)
+ ret = client->ops->open(&cinst->lan_info, client);
+ if (!ret)
+ set_bit(__I40E_CLIENT_INSTANCE_OPENED,
+ &cinst->state);
+ else
+ /* remove client instance */
+ i40evf_client_del_instance(adapter);
+ }
+}
+
+/**
+ * i40evf_lan_add_device - add a lan device struct to the list of lan devices
+ * @adapter: pointer to the board struct
+ *
+ * Returns 0 on success or none 0 on error
+ **/
+int i40evf_lan_add_device(struct i40evf_adapter *adapter)
+{
+ struct i40e_device *ldev;
+ int ret = 0;
+
+ mutex_lock(&i40evf_device_mutex);
+ list_for_each_entry(ldev, &i40evf_devices, list) {
+ if (ldev->vf == adapter) {
+ ret = -EEXIST;
+ goto out;
+ }
+ }
+ ldev = kzalloc(sizeof(*ldev), GFP_KERNEL);
+ if (!ldev) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ ldev->vf = adapter;
+ INIT_LIST_HEAD(&ldev->list);
+ list_add(&ldev->list, &i40evf_devices);
+ dev_info(&adapter->pdev->dev, "Added LAN device bus=0x%02x dev=0x%02x func=0x%02x\n",
+ adapter->hw.bus.bus_id, adapter->hw.bus.device,
+ adapter->hw.bus.func);
+
+ /* Since in some cases register may have happened before a device gets
+ * added, we can schedule a subtask to go initiate the clients.
+ */
+ adapter->flags |= I40EVF_FLAG_SERVICE_CLIENT_REQUESTED;
+
+out:
+ mutex_unlock(&i40evf_device_mutex);
+ return ret;
+}
+
+/**
+ * i40evf_lan_del_device - removes a lan device from the device list
+ * @adapter: pointer to the board struct
+ *
+ * Returns 0 on success or non-0 on error
+ **/
+int i40evf_lan_del_device(struct i40evf_adapter *adapter)
+{
+ struct i40e_device *ldev, *tmp;
+ int ret = -ENODEV;
+
+ mutex_lock(&i40evf_device_mutex);
+ list_for_each_entry_safe(ldev, tmp, &i40evf_devices, list) {
+ if (ldev->vf == adapter) {
+ dev_info(&adapter->pdev->dev,
+ "Deleted LAN device bus=0x%02x dev=0x%02x func=0x%02x\n",
+ adapter->hw.bus.bus_id, adapter->hw.bus.device,
+ adapter->hw.bus.func);
+ list_del(&ldev->list);
+ kfree(ldev);
+ ret = 0;
+ break;
+ }
+ }
+
+ mutex_unlock(&i40evf_device_mutex);
+ return ret;
+}
+
+/**
+ * i40evf_client_release - release client specific resources
+ * @client: pointer to the registered client
+ *
+ **/
+static void i40evf_client_release(struct i40e_client *client)
+{
+ struct i40e_client_instance *cinst;
+ struct i40e_device *ldev;
+ struct i40evf_adapter *adapter;
+
+ mutex_lock(&i40evf_device_mutex);
+ list_for_each_entry(ldev, &i40evf_devices, list) {
+ adapter = ldev->vf;
+ cinst = adapter->cinst;
+ if (!cinst)
+ continue;
+ if (test_bit(__I40E_CLIENT_INSTANCE_OPENED, &cinst->state)) {
+ if (client->ops && client->ops->close)
+ client->ops->close(&cinst->lan_info, client,
+ false);
+ i40evf_client_release_qvlist(&cinst->lan_info);
+ clear_bit(__I40E_CLIENT_INSTANCE_OPENED, &cinst->state);
+
+ dev_warn(&adapter->pdev->dev,
+ "Client %s instance closed\n", client->name);
+ }
+ /* delete the client instance */
+ i40evf_client_del_instance(adapter);
+ dev_info(&adapter->pdev->dev, "Deleted client instance of Client %s\n",
+ client->name);
+ }
+ mutex_unlock(&i40evf_device_mutex);
+}
+
+/**
+ * i40evf_client_prepare - prepare client specific resources
+ * @client: pointer to the registered client
+ *
+ **/
+static void i40evf_client_prepare(struct i40e_client *client)
+{
+ struct i40e_device *ldev;
+ struct i40evf_adapter *adapter;
+
+ mutex_lock(&i40evf_device_mutex);
+ list_for_each_entry(ldev, &i40evf_devices, list) {
+ adapter = ldev->vf;
+ /* Signal the watchdog to service the client */
+ adapter->flags |= I40EVF_FLAG_SERVICE_CLIENT_REQUESTED;
+ }
+ mutex_unlock(&i40evf_device_mutex);
+}
+
+/**
+ * i40evf_client_virtchnl_send - send a message to the PF instance
+ * @ldev: pointer to L2 context.
+ * @client: Client pointer.
+ * @msg: pointer to message buffer
+ * @len: message length
+ *
+ * Return 0 on success or < 0 on error
+ **/
+static u32 i40evf_client_virtchnl_send(struct i40e_info *ldev,
+ struct i40e_client *client,
+ u8 *msg, u16 len)
+{
+ struct i40evf_adapter *adapter = ldev->vf;
+ i40e_status err;
+
+ if (adapter->aq_required)
+ return -EAGAIN;
+
+ err = i40e_aq_send_msg_to_pf(&adapter->hw, I40E_VIRTCHNL_OP_IWARP,
+ I40E_SUCCESS, msg, len, NULL);
+ if (err)
+ dev_err(&adapter->pdev->dev, "Unable to send iWarp message to PF, error %d, aq status %d\n",
+ err, adapter->hw.aq.asq_last_status);
+
+ return err;
+}
+
+/**
+ * i40evf_client_setup_qvlist - send a message to the PF to setup iwarp qv map
+ * @ldev: pointer to L2 context.
+ * @client: Client pointer.
+ * @qv_info: queue and vector list
+ *
+ * Return 0 on success or < 0 on error
+ **/
+static int i40evf_client_setup_qvlist(struct i40e_info *ldev,
+ struct i40e_client *client,
+ struct i40e_qvlist_info *qvlist_info)
+{
+ struct i40e_virtchnl_iwarp_qvlist_info *v_qvlist_info;
+ struct i40evf_adapter *adapter = ldev->vf;
+ struct i40e_qv_info *qv_info;
+ i40e_status err;
+ u32 v_idx, i;
+ u32 msg_size;
+
+ if (adapter->aq_required)
+ return -EAGAIN;
+
+ /* A quick check on whether the vectors belong to the client */
+ for (i = 0; i < qvlist_info->num_vectors; i++) {
+ qv_info = &qvlist_info->qv_info[i];
+ if (!qv_info)
+ continue;
+ v_idx = qv_info->v_idx;
+ if ((v_idx >=
+ (adapter->iwarp_base_vector + adapter->num_iwarp_msix)) ||
+ (v_idx < adapter->iwarp_base_vector))
+ return -EINVAL;
+ }
+
+ v_qvlist_info = (struct i40e_virtchnl_iwarp_qvlist_info *)qvlist_info;
+ msg_size = sizeof(struct i40e_virtchnl_iwarp_qvlist_info) +
+ (sizeof(struct i40e_virtchnl_iwarp_qv_info) *
+ (v_qvlist_info->num_vectors - 1));
+
+ adapter->client_pending |= BIT(I40E_VIRTCHNL_OP_CONFIG_IWARP_IRQ_MAP);
+ err = i40e_aq_send_msg_to_pf(&adapter->hw,
+ I40E_VIRTCHNL_OP_CONFIG_IWARP_IRQ_MAP,
+ I40E_SUCCESS, (u8 *)v_qvlist_info, msg_size, NULL);
+
+ if (err) {
+ dev_err(&adapter->pdev->dev,
+ "Unable to send iWarp vector config message to PF, error %d, aq status %d\n",
+ err, adapter->hw.aq.asq_last_status);
+ goto out;
+ }
+
+ err = -EBUSY;
+ for (i = 0; i < 5; i++) {
+ msleep(100);
+ if (!(adapter->client_pending &
+ BIT(I40E_VIRTCHNL_OP_CONFIG_IWARP_IRQ_MAP))) {
+ err = 0;
+ break;
+ }
+ }
+out:
+ return err;
+}
+
+/**
+ * i40evf_register_client - Register a i40e client driver with the L2 driver
+ * @client: pointer to the i40e_client struct
+ *
+ * Returns 0 on success or non-0 on error
+ **/
+int i40evf_register_client(struct i40e_client *client)
+{
+ int ret = 0;
+
+ if (!client) {
+ ret = -EIO;
+ goto out;
+ }
+
+ if (strlen(client->name) == 0) {
+ pr_info("i40evf: Failed to register client with no name\n");
+ ret = -EIO;
+ goto out;
+ }
+
+ if (vf_registered_client) {
+ pr_info("i40evf: Client %s has already been registered!\n",
+ client->name);
+ ret = -EEXIST;
+ goto out;
+ }
+
+ if ((client->version.major != I40EVF_CLIENT_VERSION_MAJOR) ||
+ (client->version.minor != I40EVF_CLIENT_VERSION_MINOR)) {
+ pr_info("i40evf: Failed to register client %s due to mismatched client interface version\n",
+ client->name);
+ pr_info("Client is using version: %02d.%02d.%02d while LAN driver supports %s\n",
+ client->version.major, client->version.minor,
+ client->version.build,
+ i40evf_client_interface_version_str);
+ ret = -EIO;
+ goto out;
+ }
+
+ vf_registered_client = client;
+
+ i40evf_client_prepare(client);
+
+ pr_info("i40evf: Registered client %s with return code %d\n",
+ client->name, ret);
+out:
+ return ret;
+}
+EXPORT_SYMBOL(i40evf_register_client);
+
+/**
+ * i40evf_unregister_client - Unregister a i40e client driver with the L2 driver
+ * @client: pointer to the i40e_client struct
+ *
+ * Returns 0 on success or non-0 on error
+ **/
+int i40evf_unregister_client(struct i40e_client *client)
+{
+ int ret = 0;
+
+ /* When a unregister request comes through we would have to send
+ * a close for each of the client instances that were opened.
+ * client_release function is called to handle this.
+ */
+ i40evf_client_release(client);
+
+ if (vf_registered_client != client) {
+ pr_info("i40evf: Client %s has not been registered\n",
+ client->name);
+ ret = -ENODEV;
+ goto out;
+ }
+ vf_registered_client = NULL;
+ pr_info("i40evf: Unregistered client %s\n", client->name);
+out:
+ return ret;
+}
+EXPORT_SYMBOL(i40evf_unregister_client);
diff --git a/drivers/net/ethernet/intel/i40evf/i40evf_client.h b/drivers/net/ethernet/intel/i40evf/i40evf_client.h
new file mode 100644
index 0000000..7d283c7
--- /dev/null
+++ b/drivers/net/ethernet/intel/i40evf/i40evf_client.h
@@ -0,0 +1,166 @@
+#ifndef _I40E_CLIENT_H_
+#define _I40E_CLIENT_H_
+
+#define I40EVF_CLIENT_STR_LENGTH 10
+
+/* Client interface version should be updated anytime there is a change in the
+ * existing APIs or data structures.
+ */
+#define I40EVF_CLIENT_VERSION_MAJOR 0
+#define I40EVF_CLIENT_VERSION_MINOR 01
+#define I40EVF_CLIENT_VERSION_BUILD 00
+#define I40EVF_CLIENT_VERSION_STR \
+ __stringify(I40EVF_CLIENT_VERSION_MAJOR) "." \
+ __stringify(I40EVF_CLIENT_VERSION_MINOR) "." \
+ __stringify(I40EVF_CLIENT_VERSION_BUILD)
+
+struct i40e_client_version {
+ u8 major;
+ u8 minor;
+ u8 build;
+ u8 rsvd;
+};
+
+enum i40e_client_state {
+ __I40E_CLIENT_NULL,
+ __I40E_CLIENT_REGISTERED
+};
+
+enum i40e_client_instance_state {
+ __I40E_CLIENT_INSTANCE_NONE,
+ __I40E_CLIENT_INSTANCE_OPENED,
+};
+
+struct i40e_ops;
+struct i40e_client;
+
+/* HW does not define a type value for AEQ; only for RX/TX and CEQ.
+ * In order for us to keep the interface simple, SW will define a
+ * unique type value for AEQ.
+ */
+#define I40E_QUEUE_TYPE_PE_AEQ 0x80
+#define I40E_QUEUE_INVALID_IDX 0xFFFF
+
+struct i40e_qv_info {
+ u32 v_idx; /* msix_vector */
+ u16 ceq_idx;
+ u16 aeq_idx;
+ u8 itr_idx;
+};
+
+struct i40e_qvlist_info {
+ u32 num_vectors;
+ struct i40e_qv_info qv_info[1];
+};
+
+#define I40E_CLIENT_MSIX_ALL 0xFFFFFFFF
+
+/* set of LAN parameters useful for clients managed by LAN */
+
+/* Struct to hold per priority info */
+struct i40e_prio_qos_params {
+ u16 qs_handle; /* qs handle for prio */
+ u8 tc; /* TC mapped to prio */
+ u8 reserved;
+};
+
+#define I40E_CLIENT_MAX_USER_PRIORITY 8
+/* Struct to hold Client QoS */
+struct i40e_qos_params {
+ struct i40e_prio_qos_params prio_qos[I40E_CLIENT_MAX_USER_PRIORITY];
+};
+
+struct i40e_params {
+ struct i40e_qos_params qos;
+ u16 mtu;
+ u16 link_up; /* boolean */
+};
+
+/* Structure to hold LAN device info for a client device */
+struct i40e_info {
+ struct i40e_client_version version;
+ u8 lanmac[6];
+ struct net_device *netdev;
+ struct pci_dev *pcidev;
+ u8 __iomem *hw_addr;
+ u8 fid; /* function id, PF id or VF id */
+#define I40E_CLIENT_FTYPE_PF 0
+#define I40E_CLIENT_FTYPE_VF 1
+ u8 ftype; /* function type, PF or VF */
+ void *vf; /* cast to i40evf_adapter */
+
+ /* All L2 params that could change during the life span of the device
+ * and needs to be communicated to the client when they change
+ */
+ struct i40e_params params;
+ struct i40e_ops *ops;
+
+ u16 msix_count; /* number of msix vectors*/
+ /* Array down below will be dynamically allocated based on msix_count */
+ struct msix_entry *msix_entries;
+ u16 itr_index; /* Which ITR index the PE driver is suppose to use */
+};
+
+struct i40e_ops {
+ /* setup_q_vector_list enables queues with a particular vector */
+ int (*setup_qvlist)(struct i40e_info *ldev, struct i40e_client *client,
+ struct i40e_qvlist_info *qv_info);
+
+ u32 (*virtchnl_send)(struct i40e_info *ldev, struct i40e_client *client,
+ u8 *msg, u16 len);
+
+ /* If the PE Engine is unresponsive, RDMA driver can request a reset.*/
+ void (*request_reset)(struct i40e_info *ldev,
+ struct i40e_client *client);
+};
+
+struct i40e_client_ops {
+ /* Should be called from register_client() or whenever the driver is
+ * ready to create a specific client instance.
+ */
+ int (*open)(struct i40e_info *ldev, struct i40e_client *client);
+
+ /* Should be closed when netdev is unavailable or when unregister
+ * call comes in. If the close happens due to a reset, set the reset
+ * bit to true.
+ */
+ void (*close)(struct i40e_info *ldev, struct i40e_client *client,
+ bool reset);
+
+ /* called when some l2 managed parameters changes - mss */
+ void (*l2_param_change)(struct i40e_info *ldev,
+ struct i40e_client *client,
+ struct i40e_params *params);
+
+ /* called when a message is received from the PF */
+ int (*virtchnl_receive)(struct i40e_info *ldev,
+ struct i40e_client *client,
+ u8 *msg, u16 len);
+};
+
+/* Client device */
+struct i40e_client_instance {
+ struct list_head list;
+ struct i40e_info lan_info;
+ struct i40e_client *client;
+ unsigned long state;
+};
+
+struct i40e_client {
+ struct list_head list; /* list of registered clients */
+ char name[I40EVF_CLIENT_STR_LENGTH];
+ struct i40e_client_version version;
+ unsigned long state; /* client state */
+ atomic_t ref_cnt; /* Count of all the client devices of this kind */
+ u32 flags;
+#define I40E_CLIENT_FLAGS_LAUNCH_ON_PROBE BIT(0)
+#define I40E_TX_FLAGS_NOTIFY_OTHER_EVENTS BIT(2)
+ u8 type;
+#define I40E_CLIENT_IWARP 0
+ struct i40e_client_ops *ops; /* client ops provided by the client */
+};
+
+/* used by clients */
+int i40evf_register_client(struct i40e_client *client);
+int i40evf_unregister_client(struct i40e_client *client);
+#endif /* _I40E_CLIENT_H_ */
diff --git a/drivers/net/ethernet/intel/i40evf/i40evf_ethtool.c b/drivers/net/ethernet/intel/i40evf/i40evf_ethtool.c
index 272d600..9bb2cc7 100644
--- a/drivers/net/ethernet/intel/i40evf/i40evf_ethtool.c
+++ b/drivers/net/ethernet/intel/i40evf/i40evf_ethtool.c
@@ -63,52 +63,74 @@
#define I40EVF_STATS_LEN(_dev) \
(I40EVF_GLOBAL_STATS_LEN + I40EVF_QUEUE_STATS_LEN(_dev))
+/* For now we have one and only one private flag and it is only defined
+ * when we have support for the SKIP_CPU_SYNC DMA attribute. Instead
+ * of leaving all this code sitting around empty we will strip it unless
+ * our one private flag is actually available.
+ */
+struct i40evf_priv_flags {
+ char flag_string[ETH_GSTRING_LEN];
+ u32 flag;
+ bool read_only;
+};
+
+#define I40EVF_PRIV_FLAG(_name, _flag, _read_only) { \
+ .flag_string = _name, \
+ .flag = _flag, \
+ .read_only = _read_only, \
+}
+
+static const struct i40evf_priv_flags i40evf_gstrings_priv_flags[] = {
+ I40EVF_PRIV_FLAG("legacy-rx", I40EVF_FLAG_LEGACY_RX, 0),
+};
+
+#define I40EVF_PRIV_FLAGS_STR_LEN ARRAY_SIZE(i40evf_gstrings_priv_flags)
+
/**
- * i40evf_get_settings - Get Link Speed and Duplex settings
+ * i40evf_get_link_ksettings - Get Link Speed and Duplex settings
* @netdev: network interface device structure
- * @ecmd: ethtool command
+ * @cmd: ethtool command
*
* Reports speed/duplex settings. Because this is a VF, we don't know what
* kind of link we really have, so we fake it.
**/
-static int i40evf_get_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+static int i40evf_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
{
struct i40evf_adapter *adapter = netdev_priv(netdev);
- ecmd->supported = 0;
- ecmd->autoneg = AUTONEG_DISABLE;
- ecmd->transceiver = XCVR_DUMMY1;
- ecmd->port = PORT_NONE;
+ ethtool_link_ksettings_zero_link_mode(cmd, supported);
+ cmd->base.autoneg = AUTONEG_DISABLE;
+ cmd->base.port = PORT_NONE;
/* Set speed and duplex */
switch (adapter->link_speed) {
case I40E_LINK_SPEED_40GB:
- ethtool_cmd_speed_set(ecmd, SPEED_40000);
+ cmd->base.speed = SPEED_40000;
break;
case I40E_LINK_SPEED_25GB:
#ifdef SPEED_25000
- ethtool_cmd_speed_set(ecmd, SPEED_25000);
+ cmd->base.speed = SPEED_25000;
#else
netdev_info(netdev,
"Speed is 25G, display not supported by this version of ethtool.\n");
#endif
break;
case I40E_LINK_SPEED_20GB:
- ethtool_cmd_speed_set(ecmd, SPEED_20000);
+ cmd->base.speed = SPEED_20000;
break;
case I40E_LINK_SPEED_10GB:
- ethtool_cmd_speed_set(ecmd, SPEED_10000);
+ cmd->base.speed = SPEED_10000;
break;
case I40E_LINK_SPEED_1GB:
- ethtool_cmd_speed_set(ecmd, SPEED_1000);
+ cmd->base.speed = SPEED_1000;
break;
case I40E_LINK_SPEED_100MB:
- ethtool_cmd_speed_set(ecmd, SPEED_100);
+ cmd->base.speed = SPEED_100;
break;
default:
break;
}
- ecmd->duplex = DUPLEX_FULL;
+ cmd->base.duplex = DUPLEX_FULL;
return 0;
}
@@ -125,6 +147,8 @@
{
if (sset == ETH_SS_STATS)
return I40EVF_STATS_LEN(netdev);
+ else if (sset == ETH_SS_PRIV_FLAGS)
+ return I40EVF_PRIV_FLAGS_STR_LEN;
else
return -EINVAL;
}
@@ -190,10 +214,86 @@
snprintf(p, ETH_GSTRING_LEN, "rx-%u.bytes", i);
p += ETH_GSTRING_LEN;
}
+ } else if (sset == ETH_SS_PRIV_FLAGS) {
+ for (i = 0; i < I40EVF_PRIV_FLAGS_STR_LEN; i++) {
+ snprintf(p, ETH_GSTRING_LEN, "%s",
+ i40evf_gstrings_priv_flags[i].flag_string);
+ p += ETH_GSTRING_LEN;
+ }
}
}
/**
+ * i40evf_get_priv_flags - report device private flags
+ * @dev: network interface device structure
+ *
+ * The get string set count and the string set should be matched for each
+ * flag returned. Add new strings for each flag to the i40e_gstrings_priv_flags
+ * array.
+ *
+ * Returns a u32 bitmap of flags.
+ **/
+static u32 i40evf_get_priv_flags(struct net_device *netdev)
+{
+ struct i40evf_adapter *adapter = netdev_priv(netdev);
+ u32 i, ret_flags = 0;
+
+ for (i = 0; i < I40EVF_PRIV_FLAGS_STR_LEN; i++) {
+ const struct i40evf_priv_flags *priv_flags;
+
+ priv_flags = &i40evf_gstrings_priv_flags[i];
+
+ if (priv_flags->flag & adapter->flags)
+ ret_flags |= BIT(i);
+ }
+
+ return ret_flags;
+}
+
+/**
+ * i40evf_set_priv_flags - set private flags
+ * @dev: network interface device structure
+ * @flags: bit flags to be set
+ **/
+static int i40evf_set_priv_flags(struct net_device *netdev, u32 flags)
+{
+ struct i40evf_adapter *adapter = netdev_priv(netdev);
+ u64 changed_flags;
+ u32 i;
+
+ changed_flags = adapter->flags;
+
+ for (i = 0; i < I40EVF_PRIV_FLAGS_STR_LEN; i++) {
+ const struct i40evf_priv_flags *priv_flags;
+
+ priv_flags = &i40evf_gstrings_priv_flags[i];
+
+ if (priv_flags->read_only)
+ continue;
+
+ if (flags & BIT(i))
+ adapter->flags |= priv_flags->flag;
+ else
+ adapter->flags &= ~(priv_flags->flag);
+ }
+
+ /* check for flags that changed */
+ changed_flags ^= adapter->flags;
+
+ /* Process any additional changes needed as a result of flag changes. */
+
+ /* issue a reset to force legacy-rx change to take effect */
+ if (changed_flags & I40EVF_FLAG_LEGACY_RX) {
+ if (netif_running(netdev)) {
+ adapter->flags |= I40EVF_FLAG_RESET_NEEDED;
+ schedule_work(&adapter->reset_task);
+ }
+ }
+
+ return 0;
+}
+
+/**
* i40evf_get_msglevel - Get debug message level
* @netdev: network interface device structure
*
@@ -239,6 +339,7 @@
strlcpy(drvinfo->version, i40evf_driver_version, 32);
strlcpy(drvinfo->fw_version, "N/A", 4);
strlcpy(drvinfo->bus_info, pci_name(adapter->pdev), 32);
+ drvinfo->n_priv_flags = I40EVF_PRIV_FLAGS_STR_LEN;
}
/**
@@ -643,7 +744,6 @@
}
static const struct ethtool_ops i40evf_ethtool_ops = {
- .get_settings = i40evf_get_settings,
.get_drvinfo = i40evf_get_drvinfo,
.get_link = ethtool_op_get_link,
.get_ringparam = i40evf_get_ringparam,
@@ -651,6 +751,8 @@
.get_strings = i40evf_get_strings,
.get_ethtool_stats = i40evf_get_ethtool_stats,
.get_sset_count = i40evf_get_sset_count,
+ .get_priv_flags = i40evf_get_priv_flags,
+ .set_priv_flags = i40evf_set_priv_flags,
.get_msglevel = i40evf_get_msglevel,
.set_msglevel = i40evf_set_msglevel,
.get_coalesce = i40evf_get_coalesce,
@@ -663,6 +765,7 @@
.set_rxfh = i40evf_set_rxfh,
.get_channels = i40evf_get_channels,
.get_rxfh_key_size = i40evf_get_rxfh_key_size,
+ .get_link_ksettings = i40evf_get_link_ksettings,
};
/**
diff --git a/drivers/net/ethernet/intel/i40evf/i40evf_main.c b/drivers/net/ethernet/intel/i40evf/i40evf_main.c
index f35dcaa..ea110a7 100644
--- a/drivers/net/ethernet/intel/i40evf/i40evf_main.c
+++ b/drivers/net/ethernet/intel/i40evf/i40evf_main.c
@@ -26,6 +26,14 @@
#include "i40evf.h"
#include "i40e_prototype.h"
+#include "i40evf_client.h"
+/* All i40evf tracepoints are defined by the include below, which must
+ * be included exactly once across the whole kernel with
+ * CREATE_TRACE_POINTS defined
+ */
+#define CREATE_TRACE_POINTS
+#include "i40e_trace.h"
+
static int i40evf_setup_all_tx_resources(struct i40evf_adapter *adapter);
static int i40evf_setup_all_rx_resources(struct i40evf_adapter *adapter);
static int i40evf_close(struct net_device *netdev);
@@ -36,9 +44,9 @@
#define DRV_KERN "-k"
-#define DRV_VERSION_MAJOR 1
-#define DRV_VERSION_MINOR 6
-#define DRV_VERSION_BUILD 27
+#define DRV_VERSION_MAJOR 2
+#define DRV_VERSION_MINOR 1
+#define DRV_VERSION_BUILD 14
#define DRV_VERSION __stringify(DRV_VERSION_MAJOR) "." \
__stringify(DRV_VERSION_MINOR) "." \
__stringify(DRV_VERSION_BUILD) \
@@ -489,7 +497,7 @@
int i;
/* if interface is down do nothing */
- if (test_bit(__I40E_DOWN, &adapter->vsi.state))
+ if (test_bit(__I40E_VSI_DOWN, adapter->vsi.state))
return;
for (i = 0; i < q_vectors; i++)
@@ -685,12 +693,39 @@
**/
static void i40evf_configure_rx(struct i40evf_adapter *adapter)
{
+ unsigned int rx_buf_len = I40E_RXBUFFER_2048;
struct i40e_hw *hw = &adapter->hw;
int i;
+ /* Legacy Rx will always default to a 2048 buffer size. */
+#if (PAGE_SIZE < 8192)
+ if (!(adapter->flags & I40EVF_FLAG_LEGACY_RX)) {
+ struct net_device *netdev = adapter->netdev;
+
+ /* For jumbo frames on systems with 4K pages we have to use
+ * an order 1 page, so we might as well increase the size
+ * of our Rx buffer to make better use of the available space
+ */
+ rx_buf_len = I40E_RXBUFFER_3072;
+
+ /* We use a 1536 buffer size for configurations with
+ * standard Ethernet mtu. On x86 this gives us enough room
+ * for shared info and 192 bytes of padding.
+ */
+ if (!I40E_2K_TOO_SMALL_WITH_PADDING &&
+ (netdev->mtu <= ETH_DATA_LEN))
+ rx_buf_len = I40E_RXBUFFER_1536 - NET_IP_ALIGN;
+ }
+#endif
+
for (i = 0; i < adapter->num_active_queues; i++) {
adapter->rx_rings[i].tail = hw->hw_addr + I40E_QRX_TAIL1(i);
- adapter->rx_rings[i].rx_buf_len = I40EVF_RXBUFFER_2048;
+ adapter->rx_rings[i].rx_buf_len = rx_buf_len;
+
+ if (adapter->flags & I40EVF_FLAG_LEGACY_RX)
+ clear_ring_build_skb_enabled(&adapter->rx_rings[i]);
+ else
+ set_ring_build_skb_enabled(&adapter->rx_rings[i]);
}
}
@@ -1053,11 +1088,13 @@
static void i40evf_up_complete(struct i40evf_adapter *adapter)
{
adapter->state = __I40EVF_RUNNING;
- clear_bit(__I40E_DOWN, &adapter->vsi.state);
+ clear_bit(__I40E_VSI_DOWN, adapter->vsi.state);
i40evf_napi_enable_all(adapter);
adapter->aq_required |= I40EVF_FLAG_AQ_ENABLE_QUEUES;
+ if (CLIENT_ENABLED(adapter))
+ adapter->flags |= I40EVF_FLAG_CLIENT_NEEDS_OPEN;
mod_timer_pending(&adapter->watchdog_timer, jiffies + 1);
}
@@ -1235,13 +1272,13 @@
}
pairs = adapter->num_active_queues;
- /* It's easy to be greedy for MSI-X vectors, but it really
- * doesn't do us much good if we have a lot more vectors
- * than CPU's. So let's be conservative and only ask for
- * (roughly) twice the number of vectors as there are CPU's.
+ /* It's easy to be greedy for MSI-X vectors, but it really doesn't do
+ * us much good if we have more vectors than CPUs. However, we already
+ * limit the total number of queues by the number of CPUs so we do not
+ * need any further limiting here.
*/
- v_budget = min_t(int, pairs, (int)(num_online_cpus() * 2)) + NONQ_VECS;
- v_budget = min_t(int, v_budget, (int)adapter->vf_res->max_vectors);
+ v_budget = min_t(int, pairs + NONQ_VECS,
+ (int)adapter->vf_res->max_vectors);
adapter->msix_entries = kcalloc(v_budget,
sizeof(struct msix_entry), GFP_KERNEL);
@@ -1472,6 +1509,13 @@
{
int err;
+ err = i40evf_alloc_queues(adapter);
+ if (err) {
+ dev_err(&adapter->pdev->dev,
+ "Unable to allocate memory for queues\n");
+ goto err_alloc_queues;
+ }
+
rtnl_lock();
err = i40evf_set_interrupt_capability(adapter);
rtnl_unlock();
@@ -1488,23 +1532,16 @@
goto err_alloc_q_vectors;
}
- err = i40evf_alloc_queues(adapter);
- if (err) {
- dev_err(&adapter->pdev->dev,
- "Unable to allocate memory for queues\n");
- goto err_alloc_queues;
- }
-
dev_info(&adapter->pdev->dev, "Multiqueue %s: Queue pair count = %u",
(adapter->num_active_queues > 1) ? "Enabled" : "Disabled",
adapter->num_active_queues);
return 0;
-err_alloc_queues:
- i40evf_free_q_vectors(adapter);
err_alloc_q_vectors:
i40evf_reset_interrupt_capability(adapter);
err_set_interrupt:
+ i40evf_free_queues(adapter);
+err_alloc_queues:
return err;
}
@@ -1685,6 +1722,7 @@
i40evf_set_promiscuous(adapter, 0);
goto watchdog_done;
}
+ schedule_delayed_work(&adapter->client_task, msecs_to_jiffies(5));
if (adapter->state == __I40EVF_RUNNING)
i40evf_request_stats(adapter);
@@ -1716,7 +1754,7 @@
adapter->flags |= I40EVF_FLAG_PF_COMMS_FAILED;
if (netif_running(adapter->netdev)) {
- set_bit(__I40E_DOWN, &adapter->vsi.state);
+ set_bit(__I40E_VSI_DOWN, adapter->vsi.state);
netif_carrier_off(adapter->netdev);
netif_tx_disable(adapter->netdev);
adapter->link_up = false;
@@ -1773,10 +1811,17 @@
u32 reg_val;
int i = 0, err;
- while (test_and_set_bit(__I40EVF_IN_CRITICAL_TASK,
+ while (test_and_set_bit(__I40EVF_IN_CLIENT_TASK,
&adapter->crit_section))
usleep_range(500, 1000);
-
+ if (CLIENT_ENABLED(adapter)) {
+ adapter->flags &= ~(I40EVF_FLAG_CLIENT_NEEDS_OPEN |
+ I40EVF_FLAG_CLIENT_NEEDS_CLOSE |
+ I40EVF_FLAG_CLIENT_NEEDS_L2_PARAMS |
+ I40EVF_FLAG_SERVICE_CLIENT_REQUESTED);
+ cancel_delayed_work_sync(&adapter->client_task);
+ i40evf_notify_client_close(&adapter->vsi, true);
+ }
i40evf_misc_irq_disable(adapter);
if (adapter->flags & I40EVF_FLAG_RESET_NEEDED) {
adapter->flags &= ~I40EVF_FLAG_RESET_NEEDED;
@@ -1819,6 +1864,7 @@
dev_err(&adapter->pdev->dev, "Reset never finished (%x)\n",
reg_val);
i40evf_disable_vf(adapter);
+ clear_bit(__I40EVF_IN_CLIENT_TASK, &adapter->crit_section);
return; /* Do not attempt to reinit. It's dead, Jim. */
}
@@ -1861,9 +1907,8 @@
}
adapter->aq_required |= I40EVF_FLAG_AQ_ADD_MAC_FILTER;
adapter->aq_required |= I40EVF_FLAG_AQ_ADD_VLAN_FILTER;
- /* Open RDMA Client again */
- adapter->aq_required |= I40EVF_FLAG_SERVICE_CLIENT_REQUESTED;
clear_bit(__I40EVF_IN_CRITICAL_TASK, &adapter->crit_section);
+ clear_bit(__I40EVF_IN_CLIENT_TASK, &adapter->crit_section);
i40evf_misc_irq_enable(adapter);
mod_timer(&adapter->watchdog_timer, jiffies + 2);
@@ -1980,6 +2025,48 @@
}
/**
+ * i40evf_client_task - worker thread to perform client work
+ * @work: pointer to work_struct containing our data
+ *
+ * This task handles client interactions. Because client calls can be
+ * reentrant, we can't handle them in the watchdog.
+ **/
+static void i40evf_client_task(struct work_struct *work)
+{
+ struct i40evf_adapter *adapter =
+ container_of(work, struct i40evf_adapter, client_task.work);
+
+ /* If we can't get the client bit, just give up. We'll be rescheduled
+ * later.
+ */
+
+ if (test_and_set_bit(__I40EVF_IN_CLIENT_TASK, &adapter->crit_section))
+ return;
+
+ if (adapter->flags & I40EVF_FLAG_SERVICE_CLIENT_REQUESTED) {
+ i40evf_client_subtask(adapter);
+ adapter->flags &= ~I40EVF_FLAG_SERVICE_CLIENT_REQUESTED;
+ goto out;
+ }
+ if (adapter->flags & I40EVF_FLAG_CLIENT_NEEDS_CLOSE) {
+ i40evf_notify_client_close(&adapter->vsi, false);
+ adapter->flags &= ~I40EVF_FLAG_CLIENT_NEEDS_CLOSE;
+ goto out;
+ }
+ if (adapter->flags & I40EVF_FLAG_CLIENT_NEEDS_OPEN) {
+ i40evf_notify_client_open(&adapter->vsi);
+ adapter->flags &= ~I40EVF_FLAG_CLIENT_NEEDS_OPEN;
+ goto out;
+ }
+ if (adapter->flags & I40EVF_FLAG_CLIENT_NEEDS_L2_PARAMS) {
+ i40evf_notify_client_l2_params(&adapter->vsi);
+ adapter->flags &= ~I40EVF_FLAG_CLIENT_NEEDS_L2_PARAMS;
+ }
+out:
+ clear_bit(__I40EVF_IN_CLIENT_TASK, &adapter->crit_section);
+}
+
+/**
* i40evf_free_all_tx_resources - Free Tx Resources for All Queues
* @adapter: board private structure
*
@@ -2147,7 +2234,9 @@
return 0;
- set_bit(__I40E_DOWN, &adapter->vsi.state);
+ set_bit(__I40E_VSI_DOWN, adapter->vsi.state);
+ if (CLIENT_ENABLED(adapter))
+ adapter->flags |= I40EVF_FLAG_CLIENT_NEEDS_CLOSE;
i40evf_down(adapter);
adapter->state = __I40EVF_DOWN_PENDING;
@@ -2162,21 +2251,6 @@
}
/**
- * i40evf_get_stats - Get System Network Statistics
- * @netdev: network interface device structure
- *
- * Returns the address of the device statistics structure.
- * The statistics are actually updated from the timer callback.
- **/
-static struct net_device_stats *i40evf_get_stats(struct net_device *netdev)
-{
- struct i40evf_adapter *adapter = netdev_priv(netdev);
-
- /* only return the current stats */
- return &adapter->net_stats;
-}
-
-/**
* i40evf_change_mtu - Change the Maximum Transfer Unit
* @netdev: network interface device structure
* @new_mtu: new value for maximum frame size
@@ -2188,6 +2262,10 @@
struct i40evf_adapter *adapter = netdev_priv(netdev);
netdev->mtu = new_mtu;
+ if (CLIENT_ENABLED(adapter)) {
+ i40evf_notify_client_l2_params(&adapter->vsi);
+ adapter->flags |= I40EVF_FLAG_SERVICE_CLIENT_REQUESTED;
+ }
adapter->flags |= I40EVF_FLAG_RESET_NEEDED;
schedule_work(&adapter->reset_task);
@@ -2278,7 +2356,6 @@
.ndo_open = i40evf_open,
.ndo_stop = i40evf_close,
.ndo_start_xmit = i40evf_xmit_frame,
- .ndo_get_stats = i40evf_get_stats,
.ndo_set_rx_mode = i40evf_set_rx_mode,
.ndo_validate_addr = eth_validate_addr,
.ndo_set_mac_address = i40evf_set_mac,
@@ -2328,6 +2405,8 @@
struct net_device *netdev = adapter->netdev;
struct i40e_vsi *vsi = &adapter->vsi;
int i;
+ netdev_features_t hw_enc_features;
+ netdev_features_t hw_features;
/* got VF config message back from PF, now we can parse it */
for (i = 0; i < vfres->num_vsis; i++) {
@@ -2339,46 +2418,52 @@
return -ENODEV;
}
- netdev->hw_enc_features |= NETIF_F_SG |
- NETIF_F_IP_CSUM |
- NETIF_F_IPV6_CSUM |
- NETIF_F_HIGHDMA |
- NETIF_F_SOFT_FEATURES |
- NETIF_F_TSO |
- NETIF_F_TSO_ECN |
- NETIF_F_TSO6 |
+ hw_enc_features = NETIF_F_SG |
+ NETIF_F_IP_CSUM |
+ NETIF_F_IPV6_CSUM |
+ NETIF_F_HIGHDMA |
+ NETIF_F_SOFT_FEATURES |
+ NETIF_F_TSO |
+ NETIF_F_TSO_ECN |
+ NETIF_F_TSO6 |
+ NETIF_F_SCTP_CRC |
+ NETIF_F_RXHASH |
+ NETIF_F_RXCSUM |
+ 0;
+
+ /* advertise to stack only if offloads for encapsulated packets is
+ * supported
+ */
+ if (vfres->vf_offload_flags & I40E_VIRTCHNL_VF_OFFLOAD_ENCAP) {
+ hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL |
NETIF_F_GSO_GRE |
NETIF_F_GSO_GRE_CSUM |
NETIF_F_GSO_IPXIP4 |
NETIF_F_GSO_IPXIP6 |
- NETIF_F_GSO_UDP_TUNNEL |
NETIF_F_GSO_UDP_TUNNEL_CSUM |
NETIF_F_GSO_PARTIAL |
- NETIF_F_SCTP_CRC |
- NETIF_F_RXHASH |
- NETIF_F_RXCSUM |
0;
- if (!(adapter->flags & I40EVF_FLAG_OUTER_UDP_CSUM_CAPABLE))
- netdev->gso_partial_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
+ if (!(vfres->vf_offload_flags &
+ I40E_VIRTCHNL_VF_OFFLOAD_ENCAP_CSUM))
+ netdev->gso_partial_features |=
+ NETIF_F_GSO_UDP_TUNNEL_CSUM;
- netdev->gso_partial_features |= NETIF_F_GSO_GRE_CSUM;
-
+ netdev->gso_partial_features |= NETIF_F_GSO_GRE_CSUM;
+ netdev->hw_enc_features |= NETIF_F_TSO_MANGLEID;
+ netdev->hw_enc_features |= hw_enc_features;
+ }
/* record features VLANs can make use of */
- netdev->vlan_features |= netdev->hw_enc_features |
- NETIF_F_TSO_MANGLEID;
+ netdev->vlan_features |= hw_enc_features | NETIF_F_TSO_MANGLEID;
/* Write features and hw_features separately to avoid polluting
- * with, or dropping, features that are set when we registgered.
+ * with, or dropping, features that are set when we registered.
*/
- netdev->hw_features |= netdev->hw_enc_features;
+ hw_features = hw_enc_features;
- netdev->features |= netdev->hw_enc_features | I40EVF_VLAN_FEATURES;
- netdev->hw_enc_features |= NETIF_F_TSO_MANGLEID;
+ netdev->hw_features |= hw_features;
- /* disable VLAN features if not supported */
- if (!(vfres->vf_offload_flags & I40E_VIRTCHNL_VF_OFFLOAD_VLAN))
- netdev->features ^= I40EVF_VLAN_FEATURES;
+ netdev->features |= hw_features | I40EVF_VLAN_FEATURES;
adapter->vsi.id = adapter->vsi_res->vsi_id;
@@ -2519,9 +2604,6 @@
goto err_alloc;
}
- if (hw->mac.type == I40E_MAC_X722_VF)
- adapter->flags |= I40EVF_FLAG_OUTER_UDP_CSUM_CAPABLE;
-
if (i40evf_process_config(adapter))
goto err_alloc;
adapter->current_op = I40E_VIRTCHNL_OP_UNKNOWN;
@@ -2581,13 +2663,19 @@
adapter->netdev_registered = true;
netif_tx_stop_all_queues(netdev);
+ if (CLIENT_ALLOWED(adapter)) {
+ err = i40evf_lan_add_device(adapter);
+ if (err)
+ dev_info(&pdev->dev, "Failed to add VF to client API service list: %d\n",
+ err);
+ }
dev_info(&pdev->dev, "MAC address: %pM\n", adapter->hw.mac.addr);
if (netdev->features & NETIF_F_GRO)
dev_info(&pdev->dev, "GRO is enabled\n");
adapter->state = __I40EVF_DOWN;
- set_bit(__I40E_DOWN, &adapter->vsi.state);
+ set_bit(__I40E_VSI_DOWN, adapter->vsi.state);
i40evf_misc_irq_enable(adapter);
adapter->rss_key = kzalloc(adapter->rss_key_size, GFP_KERNEL);
@@ -2745,6 +2833,7 @@
INIT_WORK(&adapter->reset_task, i40evf_reset_task);
INIT_WORK(&adapter->adminq_task, i40evf_adminq_task);
INIT_WORK(&adapter->watchdog_task, i40evf_watchdog_task);
+ INIT_DELAYED_WORK(&adapter->client_task, i40evf_client_task);
INIT_DELAYED_WORK(&adapter->init_task, i40evf_init_task);
schedule_delayed_work(&adapter->init_task,
msecs_to_jiffies(5 * (pdev->devfn & 0x07)));
@@ -2857,14 +2946,21 @@
struct i40evf_adapter *adapter = netdev_priv(netdev);
struct i40evf_mac_filter *f, *ftmp;
struct i40e_hw *hw = &adapter->hw;
+ int err;
cancel_delayed_work_sync(&adapter->init_task);
cancel_work_sync(&adapter->reset_task);
-
+ cancel_delayed_work_sync(&adapter->client_task);
if (adapter->netdev_registered) {
unregister_netdev(netdev);
adapter->netdev_registered = false;
}
+ if (CLIENT_ALLOWED(adapter)) {
+ err = i40evf_lan_del_device(adapter);
+ if (err)
+ dev_warn(&pdev->dev, "Failed to delete client device: %d\n",
+ err);
+ }
/* Shut down all the garbage mashers on the detention level */
adapter->state = __I40EVF_REMOVE;
diff --git a/drivers/net/ethernet/intel/i40evf/i40evf_virtchnl.c b/drivers/net/ethernet/intel/i40evf/i40evf_virtchnl.c
index bee58af..deb2cb8 100644
--- a/drivers/net/ethernet/intel/i40evf/i40evf_virtchnl.c
+++ b/drivers/net/ethernet/intel/i40evf/i40evf_virtchnl.c
@@ -26,6 +26,7 @@
#include "i40evf.h"
#include "i40e_prototype.h"
+#include "i40evf_client.h"
/* busy wait delay in msec */
#define I40EVF_BUSY_WAIT_DELAY 10
@@ -158,7 +159,9 @@
I40E_VIRTCHNL_VF_OFFLOAD_RSS_REG |
I40E_VIRTCHNL_VF_OFFLOAD_VLAN |
I40E_VIRTCHNL_VF_OFFLOAD_WB_ON_ITR |
- I40E_VIRTCHNL_VF_OFFLOAD_RSS_PCTYPE_V2;
+ I40E_VIRTCHNL_VF_OFFLOAD_RSS_PCTYPE_V2 |
+ I40E_VIRTCHNL_VF_OFFLOAD_ENCAP |
+ I40E_VIRTCHNL_VF_OFFLOAD_ENCAP_CSUM;
adapter->current_op = I40E_VIRTCHNL_OP_GET_VF_RESOURCES;
adapter->aq_required &= ~I40EVF_FLAG_AQ_GET_CONFIG;
@@ -233,7 +236,7 @@
struct i40e_virtchnl_vsi_queue_config_info *vqci;
struct i40e_virtchnl_queue_pair_info *vqpi;
int pairs = adapter->num_active_queues;
- int i, len;
+ int i, len, max_frame = I40E_MAX_RXBUFFER;
if (adapter->current_op != I40E_VIRTCHNL_OP_UNKNOWN) {
/* bail because we already have a command pending */
@@ -248,6 +251,11 @@
if (!vqci)
return;
+ /* Limit maximum frame size when jumbo frames is not enabled */
+ if (!(adapter->flags & I40EVF_FLAG_LEGACY_RX) &&
+ (adapter->netdev->mtu <= ETH_DATA_LEN))
+ max_frame = I40E_RXBUFFER_1536 - NET_IP_ALIGN;
+
vqci->vsi_id = adapter->vsi_res->vsi_id;
vqci->num_queue_pairs = pairs;
vqpi = vqci->qpair;
@@ -259,17 +267,14 @@
vqpi->txq.queue_id = i;
vqpi->txq.ring_len = adapter->tx_rings[i].count;
vqpi->txq.dma_ring_addr = adapter->tx_rings[i].dma;
- vqpi->txq.headwb_enabled = 1;
- vqpi->txq.dma_headwb_addr = vqpi->txq.dma_ring_addr +
- (vqpi->txq.ring_len * sizeof(struct i40e_tx_desc));
-
vqpi->rxq.vsi_id = vqci->vsi_id;
vqpi->rxq.queue_id = i;
vqpi->rxq.ring_len = adapter->rx_rings[i].count;
vqpi->rxq.dma_ring_addr = adapter->rx_rings[i].dma;
- vqpi->rxq.max_pkt_size = adapter->netdev->mtu
- + ETH_HLEN + VLAN_HLEN + ETH_FCS_LEN;
- vqpi->rxq.databuffer_size = adapter->rx_rings[i].rx_buf_len;
+ vqpi->rxq.max_pkt_size = max_frame;
+ vqpi->rxq.databuffer_size =
+ ALIGN(adapter->rx_rings[i].rx_buf_len,
+ BIT_ULL(I40E_RXQ_CTX_DBUFF_SHIFT));
vqpi++;
}
@@ -955,17 +960,17 @@
case I40E_VIRTCHNL_OP_GET_STATS: {
struct i40e_eth_stats *stats =
(struct i40e_eth_stats *)msg;
- adapter->net_stats.rx_packets = stats->rx_unicast +
- stats->rx_multicast +
- stats->rx_broadcast;
- adapter->net_stats.tx_packets = stats->tx_unicast +
- stats->tx_multicast +
- stats->tx_broadcast;
- adapter->net_stats.rx_bytes = stats->rx_bytes;
- adapter->net_stats.tx_bytes = stats->tx_bytes;
- adapter->net_stats.tx_errors = stats->tx_errors;
- adapter->net_stats.rx_dropped = stats->rx_discards;
- adapter->net_stats.tx_dropped = stats->tx_discards;
+ netdev->stats.rx_packets = stats->rx_unicast +
+ stats->rx_multicast +
+ stats->rx_broadcast;
+ netdev->stats.tx_packets = stats->tx_unicast +
+ stats->tx_multicast +
+ stats->tx_broadcast;
+ netdev->stats.rx_bytes = stats->rx_bytes;
+ netdev->stats.tx_bytes = stats->tx_bytes;
+ netdev->stats.tx_errors = stats->tx_errors;
+ netdev->stats.rx_dropped = stats->rx_discards;
+ netdev->stats.tx_dropped = stats->tx_discards;
adapter->current_stats = *stats;
}
break;
@@ -999,6 +1004,16 @@
if (v_opcode != adapter->current_op)
return;
break;
+ case I40E_VIRTCHNL_OP_IWARP:
+ /* Gobble zero-length replies from the PF. They indicate that
+ * a previous message was received OK, and the client doesn't
+ * care about that.
+ */
+ if (msglen && CLIENT_ENABLED(adapter))
+ i40evf_notify_client_message(&adapter->vsi,
+ msg, msglen);
+ break;
+
case I40E_VIRTCHNL_OP_CONFIG_IWARP_IRQ_MAP:
adapter->client_pending &=
~(BIT(I40E_VIRTCHNL_OP_CONFIG_IWARP_IRQ_MAP));
@@ -1014,7 +1029,7 @@
}
break;
default:
- if (v_opcode != adapter->current_op)
+ if (adapter->current_op && (v_opcode != adapter->current_op))
dev_warn(&adapter->pdev->dev, "Expected response %d from PF, received %d\n",
adapter->current_op, v_opcode);
break;
diff --git a/drivers/net/ethernet/intel/igb/e1000_defines.h b/drivers/net/ethernet/intel/igb/e1000_defines.h
index 8aee314..d851777 100644
--- a/drivers/net/ethernet/intel/igb/e1000_defines.h
+++ b/drivers/net/ethernet/intel/igb/e1000_defines.h
@@ -39,6 +39,27 @@
#define E1000_WUFC_MC 0x00000008 /* Directed Multicast Wakeup Enable */
#define E1000_WUFC_BC 0x00000010 /* Broadcast Wakeup Enable */
+/* Wake Up Status */
+#define E1000_WUS_EX 0x00000004 /* Directed Exact */
+#define E1000_WUS_ARPD 0x00000020 /* Directed ARP Request */
+#define E1000_WUS_IPV4 0x00000040 /* Directed IPv4 */
+#define E1000_WUS_IPV6 0x00000080 /* Directed IPv6 */
+#define E1000_WUS_NSD 0x00000400 /* Directed IPv6 Neighbor Solicitation */
+
+/* Packet types that are enabled for wake packet delivery */
+#define WAKE_PKT_WUS ( \
+ E1000_WUS_EX | \
+ E1000_WUS_ARPD | \
+ E1000_WUS_IPV4 | \
+ E1000_WUS_IPV6 | \
+ E1000_WUS_NSD)
+
+/* Wake Up Packet Length */
+#define E1000_WUPL_MASK 0x00000FFF
+
+/* Wake Up Packet Memory stores the first 128 bytes of the wake up packet */
+#define E1000_WUPM_BYTES 128
+
/* Extended Device Control */
#define E1000_CTRL_EXT_SDP2_DATA 0x00000040 /* Value of SW Defineable Pin 2 */
#define E1000_CTRL_EXT_SDP3_DATA 0x00000080 /* Value of SW Defineable Pin 3 */
diff --git a/drivers/net/ethernet/intel/igb/e1000_mbx.h b/drivers/net/ethernet/intel/igb/e1000_mbx.h
index d20af6b..3e7fed7 100644
--- a/drivers/net/ethernet/intel/igb/e1000_mbx.h
+++ b/drivers/net/ethernet/intel/igb/e1000_mbx.h
@@ -55,6 +55,10 @@
#define E1000_VF_RESET 0x01 /* VF requests reset */
#define E1000_VF_SET_MAC_ADDR 0x02 /* VF requests to set MAC addr */
+/* VF requests to clear all unicast MAC filters */
+#define E1000_VF_MAC_FILTER_CLR (0x01 << E1000_VT_MSGINFO_SHIFT)
+/* VF requests to add unicast MAC filter */
+#define E1000_VF_MAC_FILTER_ADD (0x02 << E1000_VT_MSGINFO_SHIFT)
#define E1000_VF_SET_MULTICAST 0x03 /* VF requests to set MC addr */
#define E1000_VF_SET_VLAN 0x04 /* VF requests to set VLAN */
#define E1000_VF_SET_LPE 0x05 /* VF requests to set VMOLR.LPE */
diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h
index acbc3ab..bf9bf90 100644
--- a/drivers/net/ethernet/intel/igb/igb.h
+++ b/drivers/net/ethernet/intel/igb/igb.h
@@ -111,6 +111,16 @@
bool spoofchk_enabled;
};
+/* Number of unicast MAC filters reserved for the PF in the RAR registers */
+#define IGB_PF_MAC_FILTERS_RESERVED 3
+
+struct vf_mac_filter {
+ struct list_head l;
+ int vf;
+ bool free;
+ u8 vf_mac[ETH_ALEN];
+};
+
#define IGB_VF_FLAG_CTS 0x00000001 /* VF is clear to send data */
#define IGB_VF_FLAG_UNI_PROMISC 0x00000002 /* VF has unicast promisc */
#define IGB_VF_FLAG_MULTI_PROMISC 0x00000004 /* VF has multicast promisc */
@@ -142,12 +152,24 @@
/* Supported Rx Buffer Sizes */
#define IGB_RXBUFFER_256 256
#define IGB_RXBUFFER_2048 2048
+#define IGB_RXBUFFER_3072 3072
#define IGB_RX_HDR_LEN IGB_RXBUFFER_256
-#define IGB_RX_BUFSZ IGB_RXBUFFER_2048
+#define IGB_TS_HDR_LEN 16
+
+#define IGB_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN)
+#if (PAGE_SIZE < 8192)
+#define IGB_MAX_FRAME_BUILD_SKB \
+ (SKB_WITH_OVERHEAD(IGB_RXBUFFER_2048) - IGB_SKB_PAD - IGB_TS_HDR_LEN)
+#else
+#define IGB_MAX_FRAME_BUILD_SKB (IGB_RXBUFFER_2048 - IGB_TS_HDR_LEN)
+#endif
/* How many Rx Buffers do we bundle into one write to the hardware ? */
#define IGB_RX_BUFFER_WRITE 16 /* Must be power of 2 */
+#define IGB_RX_DMA_ATTR \
+ (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_WEAK_ORDERING)
+
#define AUTO_ALL_MODES 0
#define IGB_EEPROM_APME 0x0400
@@ -301,12 +323,51 @@
};
enum e1000_ring_flags_t {
+ IGB_RING_FLAG_RX_3K_BUFFER,
+ IGB_RING_FLAG_RX_BUILD_SKB_ENABLED,
IGB_RING_FLAG_RX_SCTP_CSUM,
IGB_RING_FLAG_RX_LB_VLAN_BSWAP,
IGB_RING_FLAG_TX_CTX_IDX,
IGB_RING_FLAG_TX_DETECT_HANG
};
+#define ring_uses_large_buffer(ring) \
+ test_bit(IGB_RING_FLAG_RX_3K_BUFFER, &(ring)->flags)
+#define set_ring_uses_large_buffer(ring) \
+ set_bit(IGB_RING_FLAG_RX_3K_BUFFER, &(ring)->flags)
+#define clear_ring_uses_large_buffer(ring) \
+ clear_bit(IGB_RING_FLAG_RX_3K_BUFFER, &(ring)->flags)
+
+#define ring_uses_build_skb(ring) \
+ test_bit(IGB_RING_FLAG_RX_BUILD_SKB_ENABLED, &(ring)->flags)
+#define set_ring_build_skb_enabled(ring) \
+ set_bit(IGB_RING_FLAG_RX_BUILD_SKB_ENABLED, &(ring)->flags)
+#define clear_ring_build_skb_enabled(ring) \
+ clear_bit(IGB_RING_FLAG_RX_BUILD_SKB_ENABLED, &(ring)->flags)
+
+static inline unsigned int igb_rx_bufsz(struct igb_ring *ring)
+{
+#if (PAGE_SIZE < 8192)
+ if (ring_uses_large_buffer(ring))
+ return IGB_RXBUFFER_3072;
+
+ if (ring_uses_build_skb(ring))
+ return IGB_MAX_FRAME_BUILD_SKB + IGB_TS_HDR_LEN;
+#endif
+ return IGB_RXBUFFER_2048;
+}
+
+static inline unsigned int igb_rx_pg_order(struct igb_ring *ring)
+{
+#if (PAGE_SIZE < 8192)
+ if (ring_uses_large_buffer(ring))
+ return 1;
+#endif
+ return 0;
+}
+
+#define igb_rx_pg_size(_ring) (PAGE_SIZE << igb_rx_pg_order(_ring))
+
#define IGB_TXD_DCMD (E1000_ADVTXD_DCMD_EOP | E1000_ADVTXD_DCMD_RS)
#define IGB_RX_DESC(R, i) \
@@ -398,6 +459,15 @@
u16 action;
};
+struct igb_mac_addr {
+ u8 addr[ETH_ALEN];
+ u8 queue;
+ u8 state; /* bitmask */
+};
+
+#define IGB_MAC_STATE_DEFAULT 0x1
+#define IGB_MAC_STATE_IN_USE 0x2
+
/* board specific private data structure */
struct igb_adapter {
unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)];
@@ -524,6 +594,10 @@
/* lock for RX network flow classification filter */
spinlock_t nfc_lock;
bool etype_bitmap[MAX_ETYPE_FILTER];
+
+ struct igb_mac_addr *mac_table;
+ struct vf_mac_filter vf_macs;
+ struct vf_mac_filter *vf_mac_list;
};
/* flags controlling PTP/1588 function */
@@ -545,6 +619,7 @@
#define IGB_FLAG_HAS_MSIX BIT(13)
#define IGB_FLAG_EEE BIT(14)
#define IGB_FLAG_VLAN_PROMISC BIT(15)
+#define IGB_FLAG_RX_LEGACY BIT(16)
/* Media Auto Sense */
#define IGB_MAS_ENABLE_0 0X0001
@@ -558,7 +633,6 @@
#define IGB_DMCTLX_DCFLUSH_DIS 0x80000000 /* Disable DMA Coal Flush */
#define IGB_82576_TSYNC_SHIFT 19
-#define IGB_TS_HDR_LEN 16
enum e1000_state_t {
__IGB_TESTING,
__IGB_RESETTING,
@@ -591,7 +665,6 @@
void igb_setup_tctl(struct igb_adapter *);
void igb_setup_rctl(struct igb_adapter *);
netdev_tx_t igb_xmit_frame_ring(struct sk_buff *, struct igb_ring *);
-void igb_unmap_and_free_tx_resource(struct igb_ring *, struct igb_tx_buffer *);
void igb_alloc_rx_buffers(struct igb_ring *, u16);
void igb_update_stats(struct igb_adapter *, struct rtnl_link_stats64 *);
bool igb_has_link(struct igb_adapter *adapter);
@@ -604,7 +677,7 @@
void igb_ptp_suspend(struct igb_adapter *adapter);
void igb_ptp_rx_hang(struct igb_adapter *adapter);
void igb_ptp_rx_rgtstamp(struct igb_q_vector *q_vector, struct sk_buff *skb);
-void igb_ptp_rx_pktstamp(struct igb_q_vector *q_vector, unsigned char *va,
+void igb_ptp_rx_pktstamp(struct igb_q_vector *q_vector, void *va,
struct sk_buff *skb);
int igb_ptp_set_ts_config(struct net_device *netdev, struct ifreq *ifr);
int igb_ptp_get_ts_config(struct net_device *netdev, struct ifreq *ifr);
diff --git a/drivers/net/ethernet/intel/igb/igb_ethtool.c b/drivers/net/ethernet/intel/igb/igb_ethtool.c
index 737b664..0efb62d 100644
--- a/drivers/net/ethernet/intel/igb/igb_ethtool.c
+++ b/drivers/net/ethernet/intel/igb/igb_ethtool.c
@@ -144,7 +144,15 @@
};
#define IGB_TEST_LEN (sizeof(igb_gstrings_test) / ETH_GSTRING_LEN)
-static int igb_get_settings(struct net_device *netdev, struct ethtool_cmd *ecmd)
+static const char igb_priv_flags_strings[][ETH_GSTRING_LEN] = {
+#define IGB_PRIV_FLAGS_LEGACY_RX BIT(0)
+ "legacy-rx",
+};
+
+#define IGB_PRIV_FLAGS_STR_LEN ARRAY_SIZE(igb_priv_flags_strings)
+
+static int igb_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
{
struct igb_adapter *adapter = netdev_priv(netdev);
struct e1000_hw *hw = &adapter->hw;
@@ -152,76 +160,73 @@
struct e1000_sfp_flags *eth_flags = &dev_spec->eth_flags;
u32 status;
u32 speed;
+ u32 supported, advertising;
status = rd32(E1000_STATUS);
if (hw->phy.media_type == e1000_media_type_copper) {
- ecmd->supported = (SUPPORTED_10baseT_Half |
- SUPPORTED_10baseT_Full |
- SUPPORTED_100baseT_Half |
- SUPPORTED_100baseT_Full |
- SUPPORTED_1000baseT_Full|
- SUPPORTED_Autoneg |
- SUPPORTED_TP |
- SUPPORTED_Pause);
- ecmd->advertising = ADVERTISED_TP;
+ supported = (SUPPORTED_10baseT_Half |
+ SUPPORTED_10baseT_Full |
+ SUPPORTED_100baseT_Half |
+ SUPPORTED_100baseT_Full |
+ SUPPORTED_1000baseT_Full|
+ SUPPORTED_Autoneg |
+ SUPPORTED_TP |
+ SUPPORTED_Pause);
+ advertising = ADVERTISED_TP;
if (hw->mac.autoneg == 1) {
- ecmd->advertising |= ADVERTISED_Autoneg;
+ advertising |= ADVERTISED_Autoneg;
/* the e1000 autoneg seems to match ethtool nicely */
- ecmd->advertising |= hw->phy.autoneg_advertised;
+ advertising |= hw->phy.autoneg_advertised;
}
- ecmd->port = PORT_TP;
- ecmd->phy_address = hw->phy.addr;
- ecmd->transceiver = XCVR_INTERNAL;
+ cmd->base.port = PORT_TP;
+ cmd->base.phy_address = hw->phy.addr;
} else {
- ecmd->supported = (SUPPORTED_FIBRE |
- SUPPORTED_1000baseKX_Full |
- SUPPORTED_Autoneg |
- SUPPORTED_Pause);
- ecmd->advertising = (ADVERTISED_FIBRE |
- ADVERTISED_1000baseKX_Full);
+ supported = (SUPPORTED_FIBRE |
+ SUPPORTED_1000baseKX_Full |
+ SUPPORTED_Autoneg |
+ SUPPORTED_Pause);
+ advertising = (ADVERTISED_FIBRE |
+ ADVERTISED_1000baseKX_Full);
if (hw->mac.type == e1000_i354) {
if ((hw->device_id ==
E1000_DEV_ID_I354_BACKPLANE_2_5GBPS) &&
!(status & E1000_STATUS_2P5_SKU_OVER)) {
- ecmd->supported |= SUPPORTED_2500baseX_Full;
- ecmd->supported &=
- ~SUPPORTED_1000baseKX_Full;
- ecmd->advertising |= ADVERTISED_2500baseX_Full;
- ecmd->advertising &=
- ~ADVERTISED_1000baseKX_Full;
+ supported |= SUPPORTED_2500baseX_Full;
+ supported &= ~SUPPORTED_1000baseKX_Full;
+ advertising |= ADVERTISED_2500baseX_Full;
+ advertising &= ~ADVERTISED_1000baseKX_Full;
}
}
if (eth_flags->e100_base_fx) {
- ecmd->supported |= SUPPORTED_100baseT_Full;
- ecmd->advertising |= ADVERTISED_100baseT_Full;
+ supported |= SUPPORTED_100baseT_Full;
+ advertising |= ADVERTISED_100baseT_Full;
}
if (hw->mac.autoneg == 1)
- ecmd->advertising |= ADVERTISED_Autoneg;
+ advertising |= ADVERTISED_Autoneg;
- ecmd->port = PORT_FIBRE;
- ecmd->transceiver = XCVR_EXTERNAL;
+ cmd->base.port = PORT_FIBRE;
}
if (hw->mac.autoneg != 1)
- ecmd->advertising &= ~(ADVERTISED_Pause |
- ADVERTISED_Asym_Pause);
+ advertising &= ~(ADVERTISED_Pause |
+ ADVERTISED_Asym_Pause);
switch (hw->fc.requested_mode) {
case e1000_fc_full:
- ecmd->advertising |= ADVERTISED_Pause;
+ advertising |= ADVERTISED_Pause;
break;
case e1000_fc_rx_pause:
- ecmd->advertising |= (ADVERTISED_Pause |
- ADVERTISED_Asym_Pause);
+ advertising |= (ADVERTISED_Pause |
+ ADVERTISED_Asym_Pause);
break;
case e1000_fc_tx_pause:
- ecmd->advertising |= ADVERTISED_Asym_Pause;
+ advertising |= ADVERTISED_Asym_Pause;
break;
default:
- ecmd->advertising &= ~(ADVERTISED_Pause |
- ADVERTISED_Asym_Pause);
+ advertising &= ~(ADVERTISED_Pause |
+ ADVERTISED_Asym_Pause);
}
if (status & E1000_STATUS_LU) {
if ((status & E1000_STATUS_2P5_SKU) &&
@@ -236,39 +241,46 @@
}
if ((status & E1000_STATUS_FD) ||
hw->phy.media_type != e1000_media_type_copper)
- ecmd->duplex = DUPLEX_FULL;
+ cmd->base.duplex = DUPLEX_FULL;
else
- ecmd->duplex = DUPLEX_HALF;
+ cmd->base.duplex = DUPLEX_HALF;
} else {
speed = SPEED_UNKNOWN;
- ecmd->duplex = DUPLEX_UNKNOWN;
+ cmd->base.duplex = DUPLEX_UNKNOWN;
}
- ethtool_cmd_speed_set(ecmd, speed);
+ cmd->base.speed = speed;
if ((hw->phy.media_type == e1000_media_type_fiber) ||
hw->mac.autoneg)
- ecmd->autoneg = AUTONEG_ENABLE;
+ cmd->base.autoneg = AUTONEG_ENABLE;
else
- ecmd->autoneg = AUTONEG_DISABLE;
+ cmd->base.autoneg = AUTONEG_DISABLE;
/* MDI-X => 2; MDI =>1; Invalid =>0 */
if (hw->phy.media_type == e1000_media_type_copper)
- ecmd->eth_tp_mdix = hw->phy.is_mdix ? ETH_TP_MDI_X :
+ cmd->base.eth_tp_mdix = hw->phy.is_mdix ? ETH_TP_MDI_X :
ETH_TP_MDI;
else
- ecmd->eth_tp_mdix = ETH_TP_MDI_INVALID;
+ cmd->base.eth_tp_mdix = ETH_TP_MDI_INVALID;
if (hw->phy.mdix == AUTO_ALL_MODES)
- ecmd->eth_tp_mdix_ctrl = ETH_TP_MDI_AUTO;
+ cmd->base.eth_tp_mdix_ctrl = ETH_TP_MDI_AUTO;
else
- ecmd->eth_tp_mdix_ctrl = hw->phy.mdix;
+ cmd->base.eth_tp_mdix_ctrl = hw->phy.mdix;
+
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
+ advertising);
return 0;
}
-static int igb_set_settings(struct net_device *netdev, struct ethtool_cmd *ecmd)
+static int igb_set_link_ksettings(struct net_device *netdev,
+ const struct ethtool_link_ksettings *cmd)
{
struct igb_adapter *adapter = netdev_priv(netdev);
struct e1000_hw *hw = &adapter->hw;
+ u32 advertising;
/* When SoL/IDER sessions are active, autoneg/speed/duplex
* cannot be changed
@@ -283,12 +295,12 @@
* some hardware doesn't allow MDI setting when speed or
* duplex is forced.
*/
- if (ecmd->eth_tp_mdix_ctrl) {
+ if (cmd->base.eth_tp_mdix_ctrl) {
if (hw->phy.media_type != e1000_media_type_copper)
return -EOPNOTSUPP;
- if ((ecmd->eth_tp_mdix_ctrl != ETH_TP_MDI_AUTO) &&
- (ecmd->autoneg != AUTONEG_ENABLE)) {
+ if ((cmd->base.eth_tp_mdix_ctrl != ETH_TP_MDI_AUTO) &&
+ (cmd->base.autoneg != AUTONEG_ENABLE)) {
dev_err(&adapter->pdev->dev, "forcing MDI/MDI-X state is not supported when link speed and/or duplex are forced\n");
return -EINVAL;
}
@@ -297,10 +309,13 @@
while (test_and_set_bit(__IGB_RESETTING, &adapter->state))
usleep_range(1000, 2000);
- if (ecmd->autoneg == AUTONEG_ENABLE) {
+ ethtool_convert_link_mode_to_legacy_u32(&advertising,
+ cmd->link_modes.advertising);
+
+ if (cmd->base.autoneg == AUTONEG_ENABLE) {
hw->mac.autoneg = 1;
if (hw->phy.media_type == e1000_media_type_fiber) {
- hw->phy.autoneg_advertised = ecmd->advertising |
+ hw->phy.autoneg_advertised = advertising |
ADVERTISED_FIBRE |
ADVERTISED_Autoneg;
switch (adapter->link_speed) {
@@ -320,31 +335,31 @@
break;
}
} else {
- hw->phy.autoneg_advertised = ecmd->advertising |
+ hw->phy.autoneg_advertised = advertising |
ADVERTISED_TP |
ADVERTISED_Autoneg;
}
- ecmd->advertising = hw->phy.autoneg_advertised;
+ advertising = hw->phy.autoneg_advertised;
if (adapter->fc_autoneg)
hw->fc.requested_mode = e1000_fc_default;
} else {
- u32 speed = ethtool_cmd_speed(ecmd);
+ u32 speed = cmd->base.speed;
/* calling this overrides forced MDI setting */
- if (igb_set_spd_dplx(adapter, speed, ecmd->duplex)) {
+ if (igb_set_spd_dplx(adapter, speed, cmd->base.duplex)) {
clear_bit(__IGB_RESETTING, &adapter->state);
return -EINVAL;
}
}
/* MDI-X => 2; MDI => 1; Auto => 3 */
- if (ecmd->eth_tp_mdix_ctrl) {
+ if (cmd->base.eth_tp_mdix_ctrl) {
/* fix up the value for auto (3 => 0) as zero is mapped
* internally to auto
*/
- if (ecmd->eth_tp_mdix_ctrl == ETH_TP_MDI_AUTO)
+ if (cmd->base.eth_tp_mdix_ctrl == ETH_TP_MDI_AUTO)
hw->phy.mdix = AUTO_ALL_MODES;
else
- hw->phy.mdix = ecmd->eth_tp_mdix_ctrl;
+ hw->phy.mdix = cmd->base.eth_tp_mdix_ctrl;
}
/* reset the link */
@@ -852,6 +867,8 @@
sizeof(drvinfo->fw_version));
strlcpy(drvinfo->bus_info, pci_name(adapter->pdev),
sizeof(drvinfo->bus_info));
+
+ drvinfo->n_priv_flags = IGB_PRIV_FLAGS_STR_LEN;
}
static void igb_get_ringparam(struct net_device *netdev,
@@ -1811,14 +1828,14 @@
tx_ntc = tx_ring->next_to_clean;
rx_desc = IGB_RX_DESC(rx_ring, rx_ntc);
- while (igb_test_staterr(rx_desc, E1000_RXD_STAT_DD)) {
+ while (rx_desc->wb.upper.length) {
/* check Rx buffer */
rx_buffer_info = &rx_ring->rx_buffer_info[rx_ntc];
/* sync Rx buffer for CPU read */
dma_sync_single_for_cpu(rx_ring->dev,
rx_buffer_info->dma,
- IGB_RX_BUFSZ,
+ size,
DMA_FROM_DEVICE);
/* verify contents of skb */
@@ -1828,12 +1845,21 @@
/* sync Rx buffer for device write */
dma_sync_single_for_device(rx_ring->dev,
rx_buffer_info->dma,
- IGB_RX_BUFSZ,
+ size,
DMA_FROM_DEVICE);
/* unmap buffer on Tx side */
tx_buffer_info = &tx_ring->tx_buffer_info[tx_ntc];
- igb_unmap_and_free_tx_resource(tx_ring, tx_buffer_info);
+
+ /* Free all the Tx ring sk_buffs */
+ dev_kfree_skb_any(tx_buffer_info->skb);
+
+ /* unmap skb header data */
+ dma_unmap_single(tx_ring->dev,
+ dma_unmap_addr(tx_buffer_info, dma),
+ dma_unmap_len(tx_buffer_info, len),
+ DMA_TO_DEVICE);
+ dma_unmap_len_set(tx_buffer_info, len, 0);
/* increment Rx/Tx next to clean counters */
rx_ntc++;
@@ -2271,6 +2297,8 @@
return IGB_STATS_LEN;
case ETH_SS_TEST:
return IGB_TEST_LEN;
+ case ETH_SS_PRIV_FLAGS:
+ return IGB_PRIV_FLAGS_STR_LEN;
default:
return -ENOTSUPP;
}
@@ -2376,6 +2404,10 @@
}
/* BUG_ON(p - data != IGB_STATS_LEN * ETH_GSTRING_LEN); */
break;
+ case ETH_SS_PRIV_FLAGS:
+ memcpy(data, igb_priv_flags_strings,
+ IGB_PRIV_FLAGS_STR_LEN * ETH_GSTRING_LEN);
+ break;
}
}
@@ -3388,9 +3420,38 @@
return 0;
}
+static u32 igb_get_priv_flags(struct net_device *netdev)
+{
+ struct igb_adapter *adapter = netdev_priv(netdev);
+ u32 priv_flags = 0;
+
+ if (adapter->flags & IGB_FLAG_RX_LEGACY)
+ priv_flags |= IGB_PRIV_FLAGS_LEGACY_RX;
+
+ return priv_flags;
+}
+
+static int igb_set_priv_flags(struct net_device *netdev, u32 priv_flags)
+{
+ struct igb_adapter *adapter = netdev_priv(netdev);
+ unsigned int flags = adapter->flags;
+
+ flags &= ~IGB_FLAG_RX_LEGACY;
+ if (priv_flags & IGB_PRIV_FLAGS_LEGACY_RX)
+ flags |= IGB_FLAG_RX_LEGACY;
+
+ if (flags != adapter->flags) {
+ adapter->flags = flags;
+
+ /* reset interface to repopulate queues */
+ if (netif_running(netdev))
+ igb_reinit_locked(adapter);
+ }
+
+ return 0;
+}
+
static const struct ethtool_ops igb_ethtool_ops = {
- .get_settings = igb_get_settings,
- .set_settings = igb_set_settings,
.get_drvinfo = igb_get_drvinfo,
.get_regs_len = igb_get_regs_len,
.get_regs = igb_get_regs,
@@ -3426,8 +3487,12 @@
.set_rxfh = igb_set_rxfh,
.get_channels = igb_get_channels,
.set_channels = igb_set_channels,
+ .get_priv_flags = igb_get_priv_flags,
+ .set_priv_flags = igb_set_priv_flags,
.begin = igb_ethtool_begin,
.complete = igb_ethtool_complete,
+ .get_link_ksettings = igb_get_link_ksettings,
+ .set_link_ksettings = igb_set_link_ksettings,
};
void igb_set_ethtool_ops(struct net_device *netdev)
diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c
index be456ba..1cf74aa 100644
--- a/drivers/net/ethernet/intel/igb/igb_main.c
+++ b/drivers/net/ethernet/intel/igb/igb_main.c
@@ -161,11 +161,16 @@
static int igb_vlan_rx_add_vid(struct net_device *, __be16, u16);
static int igb_vlan_rx_kill_vid(struct net_device *, __be16, u16);
static void igb_restore_vlan(struct igb_adapter *);
-static void igb_rar_set_qsel(struct igb_adapter *, u8 *, u32 , u8);
+static void igb_rar_set_index(struct igb_adapter *, u32);
static void igb_ping_all_vfs(struct igb_adapter *);
static void igb_msg_task(struct igb_adapter *);
static void igb_vmm_control(struct igb_adapter *);
static int igb_set_vf_mac(struct igb_adapter *, int, unsigned char *);
+static void igb_flush_mac_table(struct igb_adapter *);
+static int igb_available_rars(struct igb_adapter *, u8);
+static void igb_set_default_mac_filter(struct igb_adapter *);
+static int igb_uc_sync(struct net_device *, const unsigned char *);
+static int igb_uc_unsync(struct net_device *, const unsigned char *);
static void igb_restore_vf_multicasts(struct igb_adapter *adapter);
static int igb_ndo_set_vf_mac(struct net_device *netdev, int vf, u8 *mac);
static int igb_ndo_set_vf_vlan(struct net_device *netdev,
@@ -554,7 +559,7 @@
16, 1,
page_address(buffer_info->page) +
buffer_info->page_offset,
- IGB_RX_BUFSZ, true);
+ igb_rx_bufsz(rx_ring), true);
}
}
}
@@ -1153,6 +1158,8 @@
pci_disable_sriov(adapter->pdev);
msleep(500);
+ kfree(adapter->vf_mac_list);
+ adapter->vf_mac_list = NULL;
kfree(adapter->vf_data);
adapter->vf_data = NULL;
wr32(E1000_IOVCTL, E1000_IOVCTL_REUSE_VFQ);
@@ -1987,6 +1994,13 @@
if (hw->mac.ops.init_hw(hw))
dev_err(&pdev->dev, "Hardware Error\n");
+ /* RAR registers were cleared during init_hw, clear mac table */
+ igb_flush_mac_table(adapter);
+ __dev_uc_unsync(adapter->netdev, NULL);
+
+ /* Recover default RAR entry */
+ igb_set_default_mac_filter(adapter);
+
/* Flow control settings reset on hardware reset, so guarantee flow
* control is off when forcing speed.
*/
@@ -2095,11 +2109,9 @@
/* guarantee we can provide a unique filter for the unicast address */
if (is_unicast_ether_addr(addr) || is_link_local_ether_addr(addr)) {
struct igb_adapter *adapter = netdev_priv(dev);
- struct e1000_hw *hw = &adapter->hw;
int vfn = adapter->vfs_allocated_count;
- int rar_entries = hw->mac.rar_entry_count - (vfn + 1);
- if (netdev_uc_count(dev) >= rar_entries)
+ if (netdev_uc_count(dev) >= igb_available_rars(adapter, vfn))
return -ENOMEM;
}
@@ -2517,6 +2529,8 @@
goto err_eeprom;
}
+ igb_set_default_mac_filter(adapter);
+
/* get firmware version for ethtool -i */
igb_set_fw_version(adapter);
@@ -2761,6 +2775,7 @@
if (hw->flash_address)
iounmap(hw->flash_address);
err_sw_init:
+ kfree(adapter->mac_table);
kfree(adapter->shadow_vfta);
igb_clear_interrupt_scheme(adapter);
#ifdef CONFIG_PCI_IOV
@@ -2796,6 +2811,8 @@
msleep(500);
}
+ kfree(adapter->vf_mac_list);
+ adapter->vf_mac_list = NULL;
kfree(adapter->vf_data);
adapter->vf_data = NULL;
adapter->vfs_allocated_count = 0;
@@ -2816,8 +2833,9 @@
struct net_device *netdev = pci_get_drvdata(pdev);
struct igb_adapter *adapter = netdev_priv(netdev);
int old_vfs = pci_num_vf(pdev);
+ struct vf_mac_filter *mac_list;
int err = 0;
- int i;
+ int num_vf_mac_filters, i;
if (!(adapter->flags & IGB_FLAG_HAS_MSIX) || num_vfs > 7) {
err = -EPERM;
@@ -2845,6 +2863,38 @@
goto out;
}
+ /* Due to the limited number of RAR entries calculate potential
+ * number of MAC filters available for the VFs. Reserve entries
+ * for PF default MAC, PF MAC filters and at least one RAR entry
+ * for each VF for VF MAC.
+ */
+ num_vf_mac_filters = adapter->hw.mac.rar_entry_count -
+ (1 + IGB_PF_MAC_FILTERS_RESERVED +
+ adapter->vfs_allocated_count);
+
+ adapter->vf_mac_list = kcalloc(num_vf_mac_filters,
+ sizeof(struct vf_mac_filter),
+ GFP_KERNEL);
+
+ mac_list = adapter->vf_mac_list;
+ INIT_LIST_HEAD(&adapter->vf_macs.l);
+
+ if (adapter->vf_mac_list) {
+ /* Initialize list of VF MAC filters */
+ for (i = 0; i < num_vf_mac_filters; i++) {
+ mac_list->vf = -1;
+ mac_list->free = true;
+ list_add(&mac_list->l, &adapter->vf_macs.l);
+ mac_list++;
+ }
+ } else {
+ /* If we could not allocate memory for the VF MAC filters
+ * we can continue without this feature but warn user.
+ */
+ dev_err(&pdev->dev,
+ "Unable to allocate memory for VF MAC filter list\n");
+ }
+
/* only call pci_enable_sriov() if no VFs are allocated already */
if (!old_vfs) {
err = pci_enable_sriov(pdev, adapter->vfs_allocated_count);
@@ -2861,6 +2911,8 @@
goto out;
err_out:
+ kfree(adapter->vf_mac_list);
+ adapter->vf_mac_list = NULL;
kfree(adapter->vf_data);
adapter->vf_data = NULL;
adapter->vfs_allocated_count = 0;
@@ -2937,6 +2989,7 @@
iounmap(hw->flash_address);
pci_release_mem_regions(pdev);
+ kfree(adapter->mac_table);
kfree(adapter->shadow_vfta);
free_netdev(netdev);
@@ -3099,6 +3152,11 @@
/* Assume MSI-X interrupts, will be checked during IRQ allocation */
adapter->flags |= IGB_FLAG_HAS_MSIX;
+ adapter->mac_table = kzalloc(sizeof(struct igb_mac_addr) *
+ hw->mac.rar_entry_count, GFP_ATOMIC);
+ if (!adapter->mac_table)
+ return -ENOMEM;
+
igb_probe_vfs(adapter);
igb_init_queue_configuration(adapter);
@@ -3293,7 +3351,7 @@
size = sizeof(struct igb_tx_buffer) * tx_ring->count;
- tx_ring->tx_buffer_info = vzalloc(size);
+ tx_ring->tx_buffer_info = vmalloc(size);
if (!tx_ring->tx_buffer_info)
goto err;
@@ -3404,6 +3462,10 @@
txdctl |= IGB_TX_HTHRESH << 8;
txdctl |= IGB_TX_WTHRESH << 16;
+ /* reinitialize tx_buffer_info */
+ memset(ring->tx_buffer_info, 0,
+ sizeof(struct igb_tx_buffer) * ring->count);
+
txdctl |= E1000_TXDCTL_QUEUE_ENABLE;
wr32(E1000_TXDCTL(reg_idx), txdctl);
}
@@ -3435,7 +3497,7 @@
size = sizeof(struct igb_rx_buffer) * rx_ring->count;
- rx_ring->rx_buffer_info = vzalloc(size);
+ rx_ring->rx_buffer_info = vmalloc(size);
if (!rx_ring->rx_buffer_info)
goto err;
@@ -3720,6 +3782,7 @@
struct igb_ring *ring)
{
struct e1000_hw *hw = &adapter->hw;
+ union e1000_adv_rx_desc *rx_desc;
u64 rdba = ring->dma;
int reg_idx = ring->reg_idx;
u32 srrctl = 0, rxdctl = 0;
@@ -3741,7 +3804,10 @@
/* set descriptor configuration */
srrctl = IGB_RX_HDR_LEN << E1000_SRRCTL_BSIZEHDRSIZE_SHIFT;
- srrctl |= IGB_RX_BUFSZ >> E1000_SRRCTL_BSIZEPKT_SHIFT;
+ if (ring_uses_large_buffer(ring))
+ srrctl |= IGB_RXBUFFER_3072 >> E1000_SRRCTL_BSIZEPKT_SHIFT;
+ else
+ srrctl |= IGB_RXBUFFER_2048 >> E1000_SRRCTL_BSIZEPKT_SHIFT;
srrctl |= E1000_SRRCTL_DESCTYPE_ADV_ONEBUF;
if (hw->mac.type >= e1000_82580)
srrctl |= E1000_SRRCTL_TIMESTAMP;
@@ -3758,11 +3824,39 @@
rxdctl |= IGB_RX_HTHRESH << 8;
rxdctl |= IGB_RX_WTHRESH << 16;
+ /* initialize rx_buffer_info */
+ memset(ring->rx_buffer_info, 0,
+ sizeof(struct igb_rx_buffer) * ring->count);
+
+ /* initialize Rx descriptor 0 */
+ rx_desc = IGB_RX_DESC(ring, 0);
+ rx_desc->wb.upper.length = 0;
+
/* enable receive descriptor fetching */
rxdctl |= E1000_RXDCTL_QUEUE_ENABLE;
wr32(E1000_RXDCTL(reg_idx), rxdctl);
}
+static void igb_set_rx_buffer_len(struct igb_adapter *adapter,
+ struct igb_ring *rx_ring)
+{
+ /* set build_skb and buffer size flags */
+ clear_ring_build_skb_enabled(rx_ring);
+ clear_ring_uses_large_buffer(rx_ring);
+
+ if (adapter->flags & IGB_FLAG_RX_LEGACY)
+ return;
+
+ set_ring_build_skb_enabled(rx_ring);
+
+#if (PAGE_SIZE < 8192)
+ if (adapter->max_frame_size <= IGB_MAX_FRAME_BUILD_SKB)
+ return;
+
+ set_ring_uses_large_buffer(rx_ring);
+#endif
+}
+
/**
* igb_configure_rx - Configure receive Unit after Reset
* @adapter: board private structure
@@ -3774,14 +3868,17 @@
int i;
/* set the correct pool for the PF default MAC address in entry 0 */
- igb_rar_set_qsel(adapter, adapter->hw.mac.addr, 0,
- adapter->vfs_allocated_count);
+ igb_set_default_mac_filter(adapter);
/* Setup the HW Rx Head and Tail Descriptor Pointers and
* the Base and Length of the Rx Descriptor Ring
*/
- for (i = 0; i < adapter->num_rx_queues; i++)
- igb_configure_rx_ring(adapter, adapter->rx_ring[i]);
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ struct igb_ring *rx_ring = adapter->rx_ring[i];
+
+ igb_set_rx_buffer_len(adapter, rx_ring);
+ igb_configure_rx_ring(adapter, rx_ring);
+ }
}
/**
@@ -3822,55 +3919,63 @@
igb_free_tx_resources(adapter->tx_ring[i]);
}
-void igb_unmap_and_free_tx_resource(struct igb_ring *ring,
- struct igb_tx_buffer *tx_buffer)
-{
- if (tx_buffer->skb) {
- dev_kfree_skb_any(tx_buffer->skb);
- if (dma_unmap_len(tx_buffer, len))
- dma_unmap_single(ring->dev,
- dma_unmap_addr(tx_buffer, dma),
- dma_unmap_len(tx_buffer, len),
- DMA_TO_DEVICE);
- } else if (dma_unmap_len(tx_buffer, len)) {
- dma_unmap_page(ring->dev,
- dma_unmap_addr(tx_buffer, dma),
- dma_unmap_len(tx_buffer, len),
- DMA_TO_DEVICE);
- }
- tx_buffer->next_to_watch = NULL;
- tx_buffer->skb = NULL;
- dma_unmap_len_set(tx_buffer, len, 0);
- /* buffer_info must be completely set up in the transmit path */
-}
-
/**
* igb_clean_tx_ring - Free Tx Buffers
* @tx_ring: ring to be cleaned
**/
static void igb_clean_tx_ring(struct igb_ring *tx_ring)
{
- struct igb_tx_buffer *buffer_info;
- unsigned long size;
- u16 i;
+ u16 i = tx_ring->next_to_clean;
+ struct igb_tx_buffer *tx_buffer = &tx_ring->tx_buffer_info[i];
- if (!tx_ring->tx_buffer_info)
- return;
- /* Free all the Tx ring sk_buffs */
+ while (i != tx_ring->next_to_use) {
+ union e1000_adv_tx_desc *eop_desc, *tx_desc;
- for (i = 0; i < tx_ring->count; i++) {
- buffer_info = &tx_ring->tx_buffer_info[i];
- igb_unmap_and_free_tx_resource(tx_ring, buffer_info);
+ /* Free all the Tx ring sk_buffs */
+ dev_kfree_skb_any(tx_buffer->skb);
+
+ /* unmap skb header data */
+ dma_unmap_single(tx_ring->dev,
+ dma_unmap_addr(tx_buffer, dma),
+ dma_unmap_len(tx_buffer, len),
+ DMA_TO_DEVICE);
+
+ /* check for eop_desc to determine the end of the packet */
+ eop_desc = tx_buffer->next_to_watch;
+ tx_desc = IGB_TX_DESC(tx_ring, i);
+
+ /* unmap remaining buffers */
+ while (tx_desc != eop_desc) {
+ tx_buffer++;
+ tx_desc++;
+ i++;
+ if (unlikely(i == tx_ring->count)) {
+ i = 0;
+ tx_buffer = tx_ring->tx_buffer_info;
+ tx_desc = IGB_TX_DESC(tx_ring, 0);
+ }
+
+ /* unmap any remaining paged data */
+ if (dma_unmap_len(tx_buffer, len))
+ dma_unmap_page(tx_ring->dev,
+ dma_unmap_addr(tx_buffer, dma),
+ dma_unmap_len(tx_buffer, len),
+ DMA_TO_DEVICE);
+ }
+
+ /* move us one more past the eop_desc for start of next pkt */
+ tx_buffer++;
+ i++;
+ if (unlikely(i == tx_ring->count)) {
+ i = 0;
+ tx_buffer = tx_ring->tx_buffer_info;
+ }
}
+ /* reset BQL for queue */
netdev_tx_reset_queue(txring_txq(tx_ring));
- size = sizeof(struct igb_tx_buffer) * tx_ring->count;
- memset(tx_ring->tx_buffer_info, 0, size);
-
- /* Zero out the descriptor ring */
- memset(tx_ring->desc, 0, tx_ring->size);
-
+ /* reset next_to_use and next_to_clean */
tx_ring->next_to_use = 0;
tx_ring->next_to_clean = 0;
}
@@ -3932,50 +4037,39 @@
**/
static void igb_clean_rx_ring(struct igb_ring *rx_ring)
{
- unsigned long size;
- u16 i;
+ u16 i = rx_ring->next_to_clean;
if (rx_ring->skb)
dev_kfree_skb(rx_ring->skb);
rx_ring->skb = NULL;
- if (!rx_ring->rx_buffer_info)
- return;
-
/* Free all the Rx ring sk_buffs */
- for (i = 0; i < rx_ring->count; i++) {
+ while (i != rx_ring->next_to_alloc) {
struct igb_rx_buffer *buffer_info = &rx_ring->rx_buffer_info[i];
- if (!buffer_info->page)
- continue;
-
/* Invalidate cache lines that may have been written to by
* device so that we avoid corrupting memory.
*/
dma_sync_single_range_for_cpu(rx_ring->dev,
buffer_info->dma,
buffer_info->page_offset,
- IGB_RX_BUFSZ,
+ igb_rx_bufsz(rx_ring),
DMA_FROM_DEVICE);
/* free resources associated with mapping */
dma_unmap_page_attrs(rx_ring->dev,
buffer_info->dma,
- PAGE_SIZE,
+ igb_rx_pg_size(rx_ring),
DMA_FROM_DEVICE,
- DMA_ATTR_SKIP_CPU_SYNC);
+ IGB_RX_DMA_ATTR);
__page_frag_cache_drain(buffer_info->page,
buffer_info->pagecnt_bias);
- buffer_info->page = NULL;
+ i++;
+ if (i == rx_ring->count)
+ i = 0;
}
- size = sizeof(struct igb_rx_buffer) * rx_ring->count;
- memset(rx_ring->rx_buffer_info, 0, size);
-
- /* Zero out the descriptor ring */
- memset(rx_ring->desc, 0, rx_ring->size);
-
rx_ring->next_to_alloc = 0;
rx_ring->next_to_clean = 0;
rx_ring->next_to_use = 0;
@@ -4014,8 +4108,7 @@
memcpy(hw->mac.addr, addr->sa_data, netdev->addr_len);
/* set the correct pool for the new PF MAC address in entry 0 */
- igb_rar_set_qsel(adapter, hw->mac.addr, 0,
- adapter->vfs_allocated_count);
+ igb_set_default_mac_filter(adapter);
return 0;
}
@@ -4059,49 +4152,6 @@
return netdev_mc_count(netdev);
}
-/**
- * igb_write_uc_addr_list - write unicast addresses to RAR table
- * @netdev: network interface device structure
- *
- * Writes unicast address list to the RAR table.
- * Returns: -ENOMEM on failure/insufficient address space
- * 0 on no addresses written
- * X on writing X addresses to the RAR table
- **/
-static int igb_write_uc_addr_list(struct net_device *netdev)
-{
- struct igb_adapter *adapter = netdev_priv(netdev);
- struct e1000_hw *hw = &adapter->hw;
- unsigned int vfn = adapter->vfs_allocated_count;
- unsigned int rar_entries = hw->mac.rar_entry_count - (vfn + 1);
- int count = 0;
-
- /* return ENOMEM indicating insufficient memory for addresses */
- if (netdev_uc_count(netdev) > rar_entries)
- return -ENOMEM;
-
- if (!netdev_uc_empty(netdev) && rar_entries) {
- struct netdev_hw_addr *ha;
-
- netdev_for_each_uc_addr(ha, netdev) {
- if (!rar_entries)
- break;
- igb_rar_set_qsel(adapter, ha->addr,
- rar_entries--,
- vfn);
- count++;
- }
- }
- /* write the addresses in reverse order to avoid write combining */
- for (; rar_entries > 0 ; rar_entries--) {
- wr32(E1000_RAH(rar_entries), 0);
- wr32(E1000_RAL(rar_entries), 0);
- }
- wrfl();
-
- return count;
-}
-
static int igb_vlan_promisc_enable(struct igb_adapter *adapter)
{
struct e1000_hw *hw = &adapter->hw;
@@ -4240,7 +4290,7 @@
struct igb_adapter *adapter = netdev_priv(netdev);
struct e1000_hw *hw = &adapter->hw;
unsigned int vfn = adapter->vfs_allocated_count;
- u32 rctl = 0, vmolr = 0;
+ u32 rctl = 0, vmolr = 0, rlpml = MAX_JUMBO_FRAME_SIZE;
int count;
/* Check for Promiscuous and All Multicast modes */
@@ -4274,8 +4324,7 @@
* sufficient space to store all the addresses then enable
* unicast promiscuous mode
*/
- count = igb_write_uc_addr_list(netdev);
- if (count < 0) {
+ if (__dev_uc_sync(netdev, igb_uc_sync, igb_uc_unsync)) {
rctl |= E1000_RCTL_UPE;
vmolr |= E1000_VMOLR_ROPE;
}
@@ -4298,6 +4347,14 @@
E1000_RCTL_VFE);
wr32(E1000_RCTL, rctl);
+#if (PAGE_SIZE < 8192)
+ if (!adapter->vfs_allocated_count) {
+ if (adapter->max_frame_size <= IGB_MAX_FRAME_BUILD_SKB)
+ rlpml = IGB_MAX_FRAME_BUILD_SKB;
+ }
+#endif
+ wr32(E1000_RLPML, rlpml);
+
/* In order to support SR-IOV and eventually VMDq it is necessary to set
* the VMOLR to enable the appropriate modes. Without this workaround
* we will have issues with VLAN tag stripping not being done for frames
@@ -4312,12 +4369,17 @@
vmolr |= rd32(E1000_VMOLR(vfn)) &
~(E1000_VMOLR_ROPE | E1000_VMOLR_MPME | E1000_VMOLR_ROMPE);
- /* enable Rx jumbo frames, no need for restriction */
+ /* enable Rx jumbo frames, restrict as needed to support build_skb */
vmolr &= ~E1000_VMOLR_RLPML_MASK;
- vmolr |= MAX_JUMBO_FRAME_SIZE | E1000_VMOLR_LPE;
+#if (PAGE_SIZE < 8192)
+ if (adapter->max_frame_size <= IGB_MAX_FRAME_BUILD_SKB)
+ vmolr |= IGB_MAX_FRAME_BUILD_SKB;
+ else
+#endif
+ vmolr |= MAX_JUMBO_FRAME_SIZE;
+ vmolr |= E1000_VMOLR_LPE;
wr32(E1000_VMOLR(vfn), vmolr);
- wr32(E1000_RLPML, MAX_JUMBO_FRAME_SIZE);
igb_restore_vf_multicasts(adapter);
}
@@ -5256,18 +5318,32 @@
dma_error:
dev_err(tx_ring->dev, "TX DMA map failed\n");
+ tx_buffer = &tx_ring->tx_buffer_info[i];
/* clear dma mappings for failed tx_buffer_info map */
- for (;;) {
+ while (tx_buffer != first) {
+ if (dma_unmap_len(tx_buffer, len))
+ dma_unmap_page(tx_ring->dev,
+ dma_unmap_addr(tx_buffer, dma),
+ dma_unmap_len(tx_buffer, len),
+ DMA_TO_DEVICE);
+ dma_unmap_len_set(tx_buffer, len, 0);
+
+ if (i--)
+ i += tx_ring->count;
tx_buffer = &tx_ring->tx_buffer_info[i];
- igb_unmap_and_free_tx_resource(tx_ring, tx_buffer);
- if (tx_buffer == first)
- break;
- if (i == 0)
- i = tx_ring->count;
- i--;
}
+ if (dma_unmap_len(tx_buffer, len))
+ dma_unmap_single(tx_ring->dev,
+ dma_unmap_addr(tx_buffer, dma),
+ dma_unmap_len(tx_buffer, len),
+ DMA_TO_DEVICE);
+ dma_unmap_len_set(tx_buffer, len, 0);
+
+ dev_kfree_skb_any(tx_buffer->skb);
+ tx_buffer->skb = NULL;
+
tx_ring->next_to_use = i;
}
@@ -5339,7 +5415,8 @@
return NETDEV_TX_OK;
out_drop:
- igb_unmap_and_free_tx_resource(tx_ring, first);
+ dev_kfree_skb_any(first->skb);
+ first->skb = NULL;
return NETDEV_TX_OK;
}
@@ -6304,7 +6381,6 @@
{
struct e1000_hw *hw = &adapter->hw;
unsigned char *vf_mac = adapter->vf_data[vf].vf_mac_addresses;
- int rar_entry = hw->mac.rar_entry_count - (vf + 1);
u32 reg, msgbuf[3];
u8 *addr = (u8 *)(&msgbuf[1]);
@@ -6312,7 +6388,7 @@
igb_vf_reset(adapter, vf);
/* set vf mac address */
- igb_rar_set_qsel(adapter, vf_mac, rar_entry, vf);
+ igb_set_vf_mac(adapter, vf, vf_mac);
/* enable transmit and receive for vf */
reg = rd32(E1000_VFTE);
@@ -6332,18 +6408,238 @@
igb_write_mbx(hw, msgbuf, 3, vf);
}
+static void igb_flush_mac_table(struct igb_adapter *adapter)
+{
+ struct e1000_hw *hw = &adapter->hw;
+ int i;
+
+ for (i = 0; i < hw->mac.rar_entry_count; i++) {
+ adapter->mac_table[i].state &= ~IGB_MAC_STATE_IN_USE;
+ memset(adapter->mac_table[i].addr, 0, ETH_ALEN);
+ adapter->mac_table[i].queue = 0;
+ igb_rar_set_index(adapter, i);
+ }
+}
+
+static int igb_available_rars(struct igb_adapter *adapter, u8 queue)
+{
+ struct e1000_hw *hw = &adapter->hw;
+ /* do not count rar entries reserved for VFs MAC addresses */
+ int rar_entries = hw->mac.rar_entry_count -
+ adapter->vfs_allocated_count;
+ int i, count = 0;
+
+ for (i = 0; i < rar_entries; i++) {
+ /* do not count default entries */
+ if (adapter->mac_table[i].state & IGB_MAC_STATE_DEFAULT)
+ continue;
+
+ /* do not count "in use" entries for different queues */
+ if ((adapter->mac_table[i].state & IGB_MAC_STATE_IN_USE) &&
+ (adapter->mac_table[i].queue != queue))
+ continue;
+
+ count++;
+ }
+
+ return count;
+}
+
+/* Set default MAC address for the PF in the first RAR entry */
+static void igb_set_default_mac_filter(struct igb_adapter *adapter)
+{
+ struct igb_mac_addr *mac_table = &adapter->mac_table[0];
+
+ ether_addr_copy(mac_table->addr, adapter->hw.mac.addr);
+ mac_table->queue = adapter->vfs_allocated_count;
+ mac_table->state = IGB_MAC_STATE_DEFAULT | IGB_MAC_STATE_IN_USE;
+
+ igb_rar_set_index(adapter, 0);
+}
+
+int igb_add_mac_filter(struct igb_adapter *adapter, const u8 *addr,
+ const u8 queue)
+{
+ struct e1000_hw *hw = &adapter->hw;
+ int rar_entries = hw->mac.rar_entry_count -
+ adapter->vfs_allocated_count;
+ int i;
+
+ if (is_zero_ether_addr(addr))
+ return -EINVAL;
+
+ /* Search for the first empty entry in the MAC table.
+ * Do not touch entries at the end of the table reserved for the VF MAC
+ * addresses.
+ */
+ for (i = 0; i < rar_entries; i++) {
+ if (adapter->mac_table[i].state & IGB_MAC_STATE_IN_USE)
+ continue;
+
+ ether_addr_copy(adapter->mac_table[i].addr, addr);
+ adapter->mac_table[i].queue = queue;
+ adapter->mac_table[i].state |= IGB_MAC_STATE_IN_USE;
+
+ igb_rar_set_index(adapter, i);
+ return i;
+ }
+
+ return -ENOSPC;
+}
+
+int igb_del_mac_filter(struct igb_adapter *adapter, const u8 *addr,
+ const u8 queue)
+{
+ struct e1000_hw *hw = &adapter->hw;
+ int rar_entries = hw->mac.rar_entry_count -
+ adapter->vfs_allocated_count;
+ int i;
+
+ if (is_zero_ether_addr(addr))
+ return -EINVAL;
+
+ /* Search for matching entry in the MAC table based on given address
+ * and queue. Do not touch entries at the end of the table reserved
+ * for the VF MAC addresses.
+ */
+ for (i = 0; i < rar_entries; i++) {
+ if (!(adapter->mac_table[i].state & IGB_MAC_STATE_IN_USE))
+ continue;
+ if (adapter->mac_table[i].queue != queue)
+ continue;
+ if (!ether_addr_equal(adapter->mac_table[i].addr, addr))
+ continue;
+
+ adapter->mac_table[i].state &= ~IGB_MAC_STATE_IN_USE;
+ memset(adapter->mac_table[i].addr, 0, ETH_ALEN);
+ adapter->mac_table[i].queue = 0;
+
+ igb_rar_set_index(adapter, i);
+ return 0;
+ }
+
+ return -ENOENT;
+}
+
+static int igb_uc_sync(struct net_device *netdev, const unsigned char *addr)
+{
+ struct igb_adapter *adapter = netdev_priv(netdev);
+ int ret;
+
+ ret = igb_add_mac_filter(adapter, addr, adapter->vfs_allocated_count);
+
+ return min_t(int, ret, 0);
+}
+
+static int igb_uc_unsync(struct net_device *netdev, const unsigned char *addr)
+{
+ struct igb_adapter *adapter = netdev_priv(netdev);
+
+ igb_del_mac_filter(adapter, addr, adapter->vfs_allocated_count);
+
+ return 0;
+}
+
+int igb_set_vf_mac_filter(struct igb_adapter *adapter, const int vf,
+ const u32 info, const u8 *addr)
+{
+ struct pci_dev *pdev = adapter->pdev;
+ struct vf_data_storage *vf_data = &adapter->vf_data[vf];
+ struct list_head *pos;
+ struct vf_mac_filter *entry = NULL;
+ int ret = 0;
+
+ switch (info) {
+ case E1000_VF_MAC_FILTER_CLR:
+ /* remove all unicast MAC filters related to the current VF */
+ list_for_each(pos, &adapter->vf_macs.l) {
+ entry = list_entry(pos, struct vf_mac_filter, l);
+ if (entry->vf == vf) {
+ entry->vf = -1;
+ entry->free = true;
+ igb_del_mac_filter(adapter, entry->vf_mac, vf);
+ }
+ }
+ break;
+ case E1000_VF_MAC_FILTER_ADD:
+ if (vf_data->flags & IGB_VF_FLAG_PF_SET_MAC) {
+ dev_warn(&pdev->dev,
+ "VF %d requested MAC filter but is administratively denied\n",
+ vf);
+ return -EINVAL;
+ }
+
+ if (!is_valid_ether_addr(addr)) {
+ dev_warn(&pdev->dev,
+ "VF %d attempted to set invalid MAC filter\n",
+ vf);
+ return -EINVAL;
+ }
+
+ /* try to find empty slot in the list */
+ list_for_each(pos, &adapter->vf_macs.l) {
+ entry = list_entry(pos, struct vf_mac_filter, l);
+ if (entry->free)
+ break;
+ }
+
+ if (entry && entry->free) {
+ entry->free = false;
+ entry->vf = vf;
+ ether_addr_copy(entry->vf_mac, addr);
+
+ ret = igb_add_mac_filter(adapter, addr, vf);
+ ret = min_t(int, ret, 0);
+ } else {
+ ret = -ENOSPC;
+ }
+
+ if (ret == -ENOSPC)
+ dev_warn(&pdev->dev,
+ "VF %d has requested MAC filter but there is no space for it\n",
+ vf);
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
static int igb_set_vf_mac_addr(struct igb_adapter *adapter, u32 *msg, int vf)
{
+ struct pci_dev *pdev = adapter->pdev;
+ struct vf_data_storage *vf_data = &adapter->vf_data[vf];
+ u32 info = msg[0] & E1000_VT_MSGINFO_MASK;
+
/* The VF MAC Address is stored in a packed array of bytes
* starting at the second 32 bit word of the msg array
*/
- unsigned char *addr = (char *)&msg[1];
- int err = -1;
+ unsigned char *addr = (unsigned char *)&msg[1];
+ int ret = 0;
- if (is_valid_ether_addr(addr))
- err = igb_set_vf_mac(adapter, vf, addr);
+ if (!info) {
+ if (vf_data->flags & IGB_VF_FLAG_PF_SET_MAC) {
+ dev_warn(&pdev->dev,
+ "VF %d attempted to override administratively set MAC address\nReload the VF driver to resume operations\n",
+ vf);
+ return -EINVAL;
+ }
- return err;
+ if (!is_valid_ether_addr(addr)) {
+ dev_warn(&pdev->dev,
+ "VF %d attempted to set invalid MAC\n",
+ vf);
+ return -EINVAL;
+ }
+
+ ret = igb_set_vf_mac(adapter, vf, addr);
+ } else {
+ ret = igb_set_vf_mac_filter(adapter, vf, info, addr);
+ }
+
+ return ret;
}
static void igb_rcv_ack_from_vf(struct igb_adapter *adapter, u32 vf)
@@ -6400,13 +6696,7 @@
switch ((msgbuf[0] & 0xFFFF)) {
case E1000_VF_SET_MAC_ADDR:
- retval = -EINVAL;
- if (!(vf_data->flags & IGB_VF_FLAG_PF_SET_MAC))
- retval = igb_set_vf_mac_addr(adapter, msgbuf, vf);
- else
- dev_warn(&pdev->dev,
- "VF %d attempted to override administratively set MAC address\nReload the VF driver to resume operations\n",
- vf);
+ retval = igb_set_vf_mac_addr(adapter, msgbuf, vf);
break;
case E1000_VF_SET_PROMISC:
retval = igb_set_vf_promisc(adapter, msgbuf, vf);
@@ -6686,7 +6976,6 @@
DMA_TO_DEVICE);
/* clear tx_buffer data */
- tx_buffer->skb = NULL;
dma_unmap_len_set(tx_buffer, len, 0);
/* clear last DMA location and unmap remaining buffers */
@@ -6822,8 +7111,14 @@
nta++;
rx_ring->next_to_alloc = (nta < rx_ring->count) ? nta : 0;
- /* transfer page from old buffer to new buffer */
- *new_buff = *old_buff;
+ /* Transfer page from old buffer to new buffer.
+ * Move each member individually to avoid possible store
+ * forwarding stalls.
+ */
+ new_buff->dma = old_buff->dma;
+ new_buff->page = old_buff->page;
+ new_buff->page_offset = old_buff->page_offset;
+ new_buff->pagecnt_bias = old_buff->pagecnt_bias;
}
static inline bool igb_page_is_reserved(struct page *page)
@@ -6831,11 +7126,10 @@
return (page_to_nid(page) != numa_mem_id()) || page_is_pfmemalloc(page);
}
-static bool igb_can_reuse_rx_page(struct igb_rx_buffer *rx_buffer,
- struct page *page,
- unsigned int truesize)
+static bool igb_can_reuse_rx_page(struct igb_rx_buffer *rx_buffer)
{
- unsigned int pagecnt_bias = rx_buffer->pagecnt_bias--;
+ unsigned int pagecnt_bias = rx_buffer->pagecnt_bias;
+ struct page *page = rx_buffer->page;
/* avoid re-using remote pages */
if (unlikely(igb_page_is_reserved(page)))
@@ -6843,16 +7137,13 @@
#if (PAGE_SIZE < 8192)
/* if we are only owner of page we can reuse it */
- if (unlikely(page_ref_count(page) != pagecnt_bias))
+ if (unlikely((page_ref_count(page) - pagecnt_bias) > 1))
return false;
-
- /* flip page offset to other buffer */
- rx_buffer->page_offset ^= IGB_RX_BUFSZ;
#else
- /* move offset up to the next cache line */
- rx_buffer->page_offset += truesize;
+#define IGB_LAST_OFFSET \
+ (SKB_WITH_OVERHEAD(PAGE_SIZE) - IGB_RXBUFFER_2048)
- if (rx_buffer->page_offset > (PAGE_SIZE - IGB_RX_BUFSZ))
+ if (rx_buffer->page_offset > IGB_LAST_OFFSET)
return false;
#endif
@@ -6860,7 +7151,7 @@
* the pagecnt_bias and page count so that we fully restock the
* number of references the driver holds.
*/
- if (unlikely(pagecnt_bias == 1)) {
+ if (unlikely(!pagecnt_bias)) {
page_ref_add(page, USHRT_MAX);
rx_buffer->pagecnt_bias = USHRT_MAX;
}
@@ -6872,34 +7163,56 @@
* igb_add_rx_frag - Add contents of Rx buffer to sk_buff
* @rx_ring: rx descriptor ring to transact packets on
* @rx_buffer: buffer containing page to add
- * @rx_desc: descriptor containing length of buffer written by hardware
* @skb: sk_buff to place the data into
+ * @size: size of buffer to be added
*
* This function will add the data contained in rx_buffer->page to the skb.
- * This is done either through a direct copy if the data in the buffer is
- * less than the skb header size, otherwise it will just attach the page as
- * a frag to the skb.
- *
- * The function will then update the page offset if necessary and return
- * true if the buffer can be reused by the adapter.
**/
-static bool igb_add_rx_frag(struct igb_ring *rx_ring,
+static void igb_add_rx_frag(struct igb_ring *rx_ring,
struct igb_rx_buffer *rx_buffer,
- unsigned int size,
- union e1000_adv_rx_desc *rx_desc,
- struct sk_buff *skb)
+ struct sk_buff *skb,
+ unsigned int size)
{
- struct page *page = rx_buffer->page;
- unsigned char *va = page_address(page) + rx_buffer->page_offset;
#if (PAGE_SIZE < 8192)
- unsigned int truesize = IGB_RX_BUFSZ;
+ unsigned int truesize = igb_rx_pg_size(rx_ring) / 2;
+#else
+ unsigned int truesize = ring_uses_build_skb(rx_ring) ?
+ SKB_DATA_ALIGN(IGB_SKB_PAD + size) :
+ SKB_DATA_ALIGN(size);
+#endif
+ skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_buffer->page,
+ rx_buffer->page_offset, size, truesize);
+#if (PAGE_SIZE < 8192)
+ rx_buffer->page_offset ^= truesize;
+#else
+ rx_buffer->page_offset += truesize;
+#endif
+}
+
+static struct sk_buff *igb_construct_skb(struct igb_ring *rx_ring,
+ struct igb_rx_buffer *rx_buffer,
+ union e1000_adv_rx_desc *rx_desc,
+ unsigned int size)
+{
+ void *va = page_address(rx_buffer->page) + rx_buffer->page_offset;
+#if (PAGE_SIZE < 8192)
+ unsigned int truesize = igb_rx_pg_size(rx_ring) / 2;
#else
unsigned int truesize = SKB_DATA_ALIGN(size);
#endif
- unsigned int pull_len;
+ unsigned int headlen;
+ struct sk_buff *skb;
- if (unlikely(skb_is_nonlinear(skb)))
- goto add_tail_frag;
+ /* prefetch first cache line of first page */
+ prefetch(va);
+#if L1_CACHE_BYTES < 128
+ prefetch(va + L1_CACHE_BYTES);
+#endif
+
+ /* allocate a skb to store the frags */
+ skb = napi_alloc_skb(&rx_ring->q_vector->napi, IGB_RX_HDR_LEN);
+ if (unlikely(!skb))
+ return NULL;
if (unlikely(igb_test_staterr(rx_desc, E1000_RXDADV_STAT_TSIP))) {
igb_ptp_rx_pktstamp(rx_ring->q_vector, va, skb);
@@ -6907,95 +7220,73 @@
size -= IGB_TS_HDR_LEN;
}
- if (likely(size <= IGB_RX_HDR_LEN)) {
- memcpy(__skb_put(skb, size), va, ALIGN(size, sizeof(long)));
-
- /* page is not reserved, we can reuse buffer as-is */
- if (likely(!igb_page_is_reserved(page)))
- return true;
-
- /* this page cannot be reused so discard it */
- return false;
- }
-
- /* we need the header to contain the greater of either ETH_HLEN or
- * 60 bytes if the skb->len is less than 60 for skb_pad.
- */
- pull_len = eth_get_headlen(va, IGB_RX_HDR_LEN);
+ /* Determine available headroom for copy */
+ headlen = size;
+ if (headlen > IGB_RX_HDR_LEN)
+ headlen = eth_get_headlen(va, IGB_RX_HDR_LEN);
/* align pull length to size of long to optimize memcpy performance */
- memcpy(__skb_put(skb, pull_len), va, ALIGN(pull_len, sizeof(long)));
+ memcpy(__skb_put(skb, headlen), va, ALIGN(headlen, sizeof(long)));
/* update all of the pointers */
- va += pull_len;
- size -= pull_len;
+ size -= headlen;
+ if (size) {
+ skb_add_rx_frag(skb, 0, rx_buffer->page,
+ (va + headlen) - page_address(rx_buffer->page),
+ size, truesize);
+#if (PAGE_SIZE < 8192)
+ rx_buffer->page_offset ^= truesize;
+#else
+ rx_buffer->page_offset += truesize;
+#endif
+ } else {
+ rx_buffer->pagecnt_bias++;
+ }
-add_tail_frag:
- skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, page,
- (unsigned long)va & ~PAGE_MASK, size, truesize);
-
- return igb_can_reuse_rx_page(rx_buffer, page, truesize);
+ return skb;
}
-static struct sk_buff *igb_fetch_rx_buffer(struct igb_ring *rx_ring,
- union e1000_adv_rx_desc *rx_desc,
- struct sk_buff *skb)
+static struct sk_buff *igb_build_skb(struct igb_ring *rx_ring,
+ struct igb_rx_buffer *rx_buffer,
+ union e1000_adv_rx_desc *rx_desc,
+ unsigned int size)
{
- unsigned int size = le16_to_cpu(rx_desc->wb.upper.length);
- struct igb_rx_buffer *rx_buffer;
- struct page *page;
+ void *va = page_address(rx_buffer->page) + rx_buffer->page_offset;
+#if (PAGE_SIZE < 8192)
+ unsigned int truesize = igb_rx_pg_size(rx_ring) / 2;
+#else
+ unsigned int truesize = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) +
+ SKB_DATA_ALIGN(IGB_SKB_PAD + size);
+#endif
+ struct sk_buff *skb;
- rx_buffer = &rx_ring->rx_buffer_info[rx_ring->next_to_clean];
- page = rx_buffer->page;
- prefetchw(page);
-
- /* we are reusing so sync this buffer for CPU use */
- dma_sync_single_range_for_cpu(rx_ring->dev,
- rx_buffer->dma,
- rx_buffer->page_offset,
- size,
- DMA_FROM_DEVICE);
-
- if (likely(!skb)) {
- void *page_addr = page_address(page) +
- rx_buffer->page_offset;
-
- /* prefetch first cache line of first page */
- prefetch(page_addr);
+ /* prefetch first cache line of first page */
+ prefetch(va);
#if L1_CACHE_BYTES < 128
- prefetch(page_addr + L1_CACHE_BYTES);
+ prefetch(va + L1_CACHE_BYTES);
#endif
- /* allocate a skb to store the frags */
- skb = napi_alloc_skb(&rx_ring->q_vector->napi, IGB_RX_HDR_LEN);
- if (unlikely(!skb)) {
- rx_ring->rx_stats.alloc_failed++;
- return NULL;
- }
+ /* build an skb around the page buffer */
+ skb = build_skb(va - IGB_SKB_PAD, truesize);
+ if (unlikely(!skb))
+ return NULL;
- /* we will be copying header into skb->data in
- * pskb_may_pull so it is in our interest to prefetch
- * it now to avoid a possible cache miss
- */
- prefetchw(skb->data);
+ /* update pointers within the skb to store the data */
+ skb_reserve(skb, IGB_SKB_PAD);
+ __skb_put(skb, size);
+
+ /* pull timestamp out of packet data */
+ if (igb_test_staterr(rx_desc, E1000_RXDADV_STAT_TSIP)) {
+ igb_ptp_rx_pktstamp(rx_ring->q_vector, skb->data, skb);
+ __skb_pull(skb, IGB_TS_HDR_LEN);
}
- /* pull page into skb */
- if (igb_add_rx_frag(rx_ring, rx_buffer, size, rx_desc, skb)) {
- /* hand second half of page back to the ring */
- igb_reuse_rx_page(rx_ring, rx_buffer);
- } else {
- /* We are not reusing the buffer so unmap it and free
- * any references we are holding to it
- */
- dma_unmap_page_attrs(rx_ring->dev, rx_buffer->dma,
- PAGE_SIZE, DMA_FROM_DEVICE,
- DMA_ATTR_SKIP_CPU_SYNC);
- __page_frag_cache_drain(page, rx_buffer->pagecnt_bias);
- }
-
- /* clear contents of rx_buffer */
- rx_buffer->page = NULL;
+ /* update buffer offset */
+#if (PAGE_SIZE < 8192)
+ rx_buffer->page_offset ^= truesize;
+#else
+ rx_buffer->page_offset += truesize;
+#endif
return skb;
}
@@ -7154,6 +7445,47 @@
skb->protocol = eth_type_trans(skb, rx_ring->netdev);
}
+static struct igb_rx_buffer *igb_get_rx_buffer(struct igb_ring *rx_ring,
+ const unsigned int size)
+{
+ struct igb_rx_buffer *rx_buffer;
+
+ rx_buffer = &rx_ring->rx_buffer_info[rx_ring->next_to_clean];
+ prefetchw(rx_buffer->page);
+
+ /* we are reusing so sync this buffer for CPU use */
+ dma_sync_single_range_for_cpu(rx_ring->dev,
+ rx_buffer->dma,
+ rx_buffer->page_offset,
+ size,
+ DMA_FROM_DEVICE);
+
+ rx_buffer->pagecnt_bias--;
+
+ return rx_buffer;
+}
+
+static void igb_put_rx_buffer(struct igb_ring *rx_ring,
+ struct igb_rx_buffer *rx_buffer)
+{
+ if (igb_can_reuse_rx_page(rx_buffer)) {
+ /* hand second half of page back to the ring */
+ igb_reuse_rx_page(rx_ring, rx_buffer);
+ } else {
+ /* We are not reusing the buffer so unmap it and free
+ * any references we are holding to it
+ */
+ dma_unmap_page_attrs(rx_ring->dev, rx_buffer->dma,
+ igb_rx_pg_size(rx_ring), DMA_FROM_DEVICE,
+ IGB_RX_DMA_ATTR);
+ __page_frag_cache_drain(rx_buffer->page,
+ rx_buffer->pagecnt_bias);
+ }
+
+ /* clear contents of rx_buffer */
+ rx_buffer->page = NULL;
+}
+
static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget)
{
struct igb_ring *rx_ring = q_vector->rx.ring;
@@ -7163,6 +7495,8 @@
while (likely(total_packets < budget)) {
union e1000_adv_rx_desc *rx_desc;
+ struct igb_rx_buffer *rx_buffer;
+ unsigned int size;
/* return some buffers to hardware, one at a time is too slow */
if (cleaned_count >= IGB_RX_BUFFER_WRITE) {
@@ -7171,8 +7505,8 @@
}
rx_desc = IGB_RX_DESC(rx_ring, rx_ring->next_to_clean);
-
- if (!rx_desc->wb.upper.status_error)
+ size = le16_to_cpu(rx_desc->wb.upper.length);
+ if (!size)
break;
/* This memory barrier is needed to keep us from reading
@@ -7181,13 +7515,25 @@
*/
dma_rmb();
+ rx_buffer = igb_get_rx_buffer(rx_ring, size);
+
/* retrieve a buffer from the ring */
- skb = igb_fetch_rx_buffer(rx_ring, rx_desc, skb);
+ if (skb)
+ igb_add_rx_frag(rx_ring, rx_buffer, skb, size);
+ else if (ring_uses_build_skb(rx_ring))
+ skb = igb_build_skb(rx_ring, rx_buffer, rx_desc, size);
+ else
+ skb = igb_construct_skb(rx_ring, rx_buffer,
+ rx_desc, size);
/* exit if we failed to retrieve a buffer */
- if (!skb)
+ if (!skb) {
+ rx_ring->rx_stats.alloc_failed++;
+ rx_buffer->pagecnt_bias++;
break;
+ }
+ igb_put_rx_buffer(rx_ring, rx_buffer);
cleaned_count++;
/* fetch next buffer in frame if non-eop */
@@ -7231,6 +7577,11 @@
return total_packets;
}
+static inline unsigned int igb_rx_offset(struct igb_ring *rx_ring)
+{
+ return ring_uses_build_skb(rx_ring) ? IGB_SKB_PAD : 0;
+}
+
static bool igb_alloc_mapped_page(struct igb_ring *rx_ring,
struct igb_rx_buffer *bi)
{
@@ -7242,21 +7593,23 @@
return true;
/* alloc new page for storage */
- page = dev_alloc_page();
+ page = dev_alloc_pages(igb_rx_pg_order(rx_ring));
if (unlikely(!page)) {
rx_ring->rx_stats.alloc_failed++;
return false;
}
/* map page for use */
- dma = dma_map_page_attrs(rx_ring->dev, page, 0, PAGE_SIZE,
- DMA_FROM_DEVICE, DMA_ATTR_SKIP_CPU_SYNC);
+ dma = dma_map_page_attrs(rx_ring->dev, page, 0,
+ igb_rx_pg_size(rx_ring),
+ DMA_FROM_DEVICE,
+ IGB_RX_DMA_ATTR);
/* if mapping failed free memory back to system since
* there isn't much point in holding memory we can't use
*/
if (dma_mapping_error(rx_ring->dev, dma)) {
- __free_page(page);
+ __free_pages(page, igb_rx_pg_order(rx_ring));
rx_ring->rx_stats.alloc_failed++;
return false;
@@ -7264,7 +7617,7 @@
bi->dma = dma;
bi->page = page;
- bi->page_offset = 0;
+ bi->page_offset = igb_rx_offset(rx_ring);
bi->pagecnt_bias = 1;
return true;
@@ -7279,6 +7632,7 @@
union e1000_adv_rx_desc *rx_desc;
struct igb_rx_buffer *bi;
u16 i = rx_ring->next_to_use;
+ u16 bufsz;
/* nothing to do */
if (!cleaned_count)
@@ -7288,14 +7642,15 @@
bi = &rx_ring->rx_buffer_info[i];
i -= rx_ring->count;
+ bufsz = igb_rx_bufsz(rx_ring);
+
do {
if (!igb_alloc_mapped_page(rx_ring, bi))
break;
/* sync the buffer for use by the device */
dma_sync_single_range_for_device(rx_ring->dev, bi->dma,
- bi->page_offset,
- IGB_RX_BUFSZ,
+ bi->page_offset, bufsz,
DMA_FROM_DEVICE);
/* Refresh the desc even if buffer_addrs didn't change
@@ -7312,8 +7667,8 @@
i -= rx_ring->count;
}
- /* clear the status bits for the next_to_use descriptor */
- rx_desc->wb.upper.status_error = 0;
+ /* clear the length for the next_to_use descriptor */
+ rx_desc->wb.upper.length = 0;
cleaned_count--;
} while (cleaned_count);
@@ -7630,6 +7985,36 @@
return 0;
}
+static void igb_deliver_wake_packet(struct net_device *netdev)
+{
+ struct igb_adapter *adapter = netdev_priv(netdev);
+ struct e1000_hw *hw = &adapter->hw;
+ struct sk_buff *skb;
+ u32 wupl;
+
+ wupl = rd32(E1000_WUPL) & E1000_WUPL_MASK;
+
+ /* WUPM stores only the first 128 bytes of the wake packet.
+ * Read the packet only if we have the whole thing.
+ */
+ if ((wupl == 0) || (wupl > E1000_WUPM_BYTES))
+ return;
+
+ skb = netdev_alloc_skb_ip_align(netdev, E1000_WUPM_BYTES);
+ if (!skb)
+ return;
+
+ skb_put(skb, wupl);
+
+ /* Ensure reads are 32-bit aligned */
+ wupl = roundup(wupl, 4);
+
+ memcpy_fromio(skb->data, hw->hw_addr + E1000_WUPM_REG(0), wupl);
+
+ skb->protocol = eth_type_trans(skb, netdev);
+ netif_rx(skb);
+}
+
#ifdef CONFIG_PM
#ifdef CONFIG_PM_SLEEP
static int igb_suspend(struct device *dev)
@@ -7659,7 +8044,7 @@
struct net_device *netdev = pci_get_drvdata(pdev);
struct igb_adapter *adapter = netdev_priv(netdev);
struct e1000_hw *hw = &adapter->hw;
- u32 err;
+ u32 err, val;
pci_set_power_state(pdev, PCI_D0);
pci_restore_state(pdev);
@@ -7690,6 +8075,10 @@
*/
igb_get_hw_control(adapter);
+ val = rd32(E1000_WUS);
+ if (val & WAKE_PKT_WUS)
+ igb_deliver_wake_packet(netdev);
+
wr32(E1000_WUS, ~0);
rtnl_lock();
@@ -7948,11 +8337,16 @@
igb_get_hw_control(adapter);
}
-static void igb_rar_set_qsel(struct igb_adapter *adapter, u8 *addr, u32 index,
- u8 qsel)
+/**
+ * igb_rar_set_index - Sync RAL[index] and RAH[index] registers with MAC table
+ * @adapter: Pointer to adapter structure
+ * @index: Index of the RAR entry which need to be synced with MAC table
+ **/
+static void igb_rar_set_index(struct igb_adapter *adapter, u32 index)
{
struct e1000_hw *hw = &adapter->hw;
u32 rar_low, rar_high;
+ u8 *addr = adapter->mac_table[index].addr;
/* HW expects these to be in network order when they are plugged
* into the registers which are little endian. In order to guarantee
@@ -7963,12 +8357,16 @@
rar_high = le16_to_cpup((__le16 *)(addr + 4));
/* Indicate to hardware the Address is Valid. */
- rar_high |= E1000_RAH_AV;
+ if (adapter->mac_table[index].state & IGB_MAC_STATE_IN_USE) {
+ rar_high |= E1000_RAH_AV;
- if (hw->mac.type == e1000_82575)
- rar_high |= E1000_RAH_POOL_1 * qsel;
- else
- rar_high |= E1000_RAH_POOL_1 << qsel;
+ if (hw->mac.type == e1000_82575)
+ rar_high |= E1000_RAH_POOL_1 *
+ adapter->mac_table[index].queue;
+ else
+ rar_high |= E1000_RAH_POOL_1 <<
+ adapter->mac_table[index].queue;
+ }
wr32(E1000_RAL(index), rar_low);
wrfl();
@@ -7984,10 +8382,13 @@
* towards the first, as a result a collision should not be possible
*/
int rar_entry = hw->mac.rar_entry_count - (vf + 1);
+ unsigned char *vf_mac_addr = adapter->vf_data[vf].vf_mac_addresses;
- memcpy(adapter->vf_data[vf].vf_mac_addresses, mac_addr, ETH_ALEN);
-
- igb_rar_set_qsel(adapter, mac_addr, rar_entry, vf);
+ ether_addr_copy(vf_mac_addr, mac_addr);
+ ether_addr_copy(adapter->mac_table[rar_entry].addr, mac_addr);
+ adapter->mac_table[rar_entry].queue = vf;
+ adapter->mac_table[rar_entry].state |= IGB_MAC_STATE_IN_USE;
+ igb_rar_set_index(adapter, rar_entry);
return 0;
}
diff --git a/drivers/net/ethernet/intel/igb/igb_ptp.c b/drivers/net/ethernet/intel/igb/igb_ptp.c
index c447755..7a3fd4d 100644
--- a/drivers/net/ethernet/intel/igb/igb_ptp.c
+++ b/drivers/net/ethernet/intel/igb/igb_ptp.c
@@ -764,8 +764,7 @@
* incoming frame. The value is stored in little endian format starting on
* byte 8.
**/
-void igb_ptp_rx_pktstamp(struct igb_q_vector *q_vector,
- unsigned char *va,
+void igb_ptp_rx_pktstamp(struct igb_q_vector *q_vector, void *va,
struct sk_buff *skb)
{
__le64 *regval = (__le64 *)va;
diff --git a/drivers/net/ethernet/intel/igbvf/ethtool.c b/drivers/net/ethernet/intel/igbvf/ethtool.c
index 8dea1b1..34faa11 100644
--- a/drivers/net/ethernet/intel/igbvf/ethtool.c
+++ b/drivers/net/ethernet/intel/igbvf/ethtool.c
@@ -71,45 +71,45 @@
#define IGBVF_TEST_LEN ARRAY_SIZE(igbvf_gstrings_test)
-static int igbvf_get_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+static int igbvf_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
{
struct igbvf_adapter *adapter = netdev_priv(netdev);
struct e1000_hw *hw = &adapter->hw;
u32 status;
- ecmd->supported = SUPPORTED_1000baseT_Full;
+ ethtool_link_ksettings_zero_link_mode(cmd, supported);
+ ethtool_link_ksettings_add_link_mode(cmd, supported, 1000baseT_Full);
+ ethtool_link_ksettings_zero_link_mode(cmd, advertising);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, 1000baseT_Full);
- ecmd->advertising = ADVERTISED_1000baseT_Full;
-
- ecmd->port = -1;
- ecmd->transceiver = XCVR_DUMMY1;
+ cmd->base.port = -1;
status = er32(STATUS);
if (status & E1000_STATUS_LU) {
if (status & E1000_STATUS_SPEED_1000)
- ethtool_cmd_speed_set(ecmd, SPEED_1000);
+ cmd->base.speed = SPEED_1000;
else if (status & E1000_STATUS_SPEED_100)
- ethtool_cmd_speed_set(ecmd, SPEED_100);
+ cmd->base.speed = SPEED_100;
else
- ethtool_cmd_speed_set(ecmd, SPEED_10);
+ cmd->base.speed = SPEED_10;
if (status & E1000_STATUS_FD)
- ecmd->duplex = DUPLEX_FULL;
+ cmd->base.duplex = DUPLEX_FULL;
else
- ecmd->duplex = DUPLEX_HALF;
+ cmd->base.duplex = DUPLEX_HALF;
} else {
- ethtool_cmd_speed_set(ecmd, SPEED_UNKNOWN);
- ecmd->duplex = DUPLEX_UNKNOWN;
+ cmd->base.speed = SPEED_UNKNOWN;
+ cmd->base.duplex = DUPLEX_UNKNOWN;
}
- ecmd->autoneg = AUTONEG_DISABLE;
+ cmd->base.autoneg = AUTONEG_DISABLE;
return 0;
}
-static int igbvf_set_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+static int igbvf_set_link_ksettings(struct net_device *netdev,
+ const struct ethtool_link_ksettings *cmd)
{
return -EOPNOTSUPP;
}
@@ -443,8 +443,6 @@
}
static const struct ethtool_ops igbvf_ethtool_ops = {
- .get_settings = igbvf_get_settings,
- .set_settings = igbvf_set_settings,
.get_drvinfo = igbvf_get_drvinfo,
.get_regs_len = igbvf_get_regs_len,
.get_regs = igbvf_get_regs,
@@ -467,6 +465,8 @@
.get_ethtool_stats = igbvf_get_ethtool_stats,
.get_coalesce = igbvf_get_coalesce,
.set_coalesce = igbvf_set_coalesce,
+ .get_link_ksettings = igbvf_get_link_ksettings,
+ .set_link_ksettings = igbvf_set_link_ksettings,
};
void igbvf_set_ethtool_ops(struct net_device *netdev)
diff --git a/drivers/net/ethernet/intel/igbvf/igbvf.h b/drivers/net/ethernet/intel/igbvf/igbvf.h
index 6f4290d..bf69f01 100644
--- a/drivers/net/ethernet/intel/igbvf/igbvf.h
+++ b/drivers/net/ethernet/intel/igbvf/igbvf.h
@@ -101,6 +101,8 @@
#define IGBVF_MNG_VLAN_NONE (-1)
+#define IGBVF_MAX_MAC_FILTERS 3
+
/* Number of packet split data buffers (not including the header buffer) */
#define PS_PAGE_BUFFERS (MAX_PS_BUFFERS - 1)
@@ -241,7 +243,6 @@
/* OS defined structs */
struct net_device *netdev;
struct pci_dev *pdev;
- struct net_device_stats net_stats;
spinlock_t stats_lock; /* prevent concurrent stats updates */
/* structs defined in e1000_hw.h */
diff --git a/drivers/net/ethernet/intel/igbvf/mbx.h b/drivers/net/ethernet/intel/igbvf/mbx.h
index f800bf8..30d58c4 100644
--- a/drivers/net/ethernet/intel/igbvf/mbx.h
+++ b/drivers/net/ethernet/intel/igbvf/mbx.h
@@ -62,6 +62,10 @@
#define E1000_VF_RESET 0x01 /* VF requests reset */
#define E1000_VF_SET_MAC_ADDR 0x02 /* VF requests PF to set MAC addr */
+/* VF requests PF to clear all unicast MAC filters */
+#define E1000_VF_MAC_FILTER_CLR (0x01 << E1000_VT_MSGINFO_SHIFT)
+/* VF requests PF to add unicast MAC filter */
+#define E1000_VF_MAC_FILTER_ADD (0x02 << E1000_VT_MSGINFO_SHIFT)
#define E1000_VF_SET_MULTICAST 0x03 /* VF requests PF to set MC addr */
#define E1000_VF_SET_VLAN 0x04 /* VF requests PF to set VLAN */
#define E1000_VF_SET_LPE 0x05 /* VF requests PF to set VMOLR.LPE */
diff --git a/drivers/net/ethernet/intel/igbvf/netdev.c b/drivers/net/ethernet/intel/igbvf/netdev.c
index 839ba11..1b9cbbe 100644
--- a/drivers/net/ethernet/intel/igbvf/netdev.c
+++ b/drivers/net/ethernet/intel/igbvf/netdev.c
@@ -400,8 +400,8 @@
adapter->total_rx_packets += total_packets;
adapter->total_rx_bytes += total_bytes;
- adapter->net_stats.rx_bytes += total_bytes;
- adapter->net_stats.rx_packets += total_packets;
+ netdev->stats.rx_bytes += total_bytes;
+ netdev->stats.rx_packets += total_packets;
return cleaned;
}
@@ -864,8 +864,8 @@
}
}
- adapter->net_stats.tx_bytes += total_bytes;
- adapter->net_stats.tx_packets += total_packets;
+ netdev->stats.tx_bytes += total_bytes;
+ netdev->stats.tx_packets += total_packets;
return count < tx_ring->count;
}
@@ -1433,12 +1433,52 @@
}
/**
+ * igbvf_set_uni - Configure unicast MAC filters
+ * @netdev: network interface device structure
+ *
+ * This routine is responsible for configuring the hardware for proper
+ * unicast filters.
+ **/
+static int igbvf_set_uni(struct net_device *netdev)
+{
+ struct igbvf_adapter *adapter = netdev_priv(netdev);
+ struct e1000_hw *hw = &adapter->hw;
+
+ if (netdev_uc_count(netdev) > IGBVF_MAX_MAC_FILTERS) {
+ pr_err("Too many unicast filters - No Space\n");
+ return -ENOSPC;
+ }
+
+ /* Clear all unicast MAC filters */
+ hw->mac.ops.set_uc_addr(hw, E1000_VF_MAC_FILTER_CLR, NULL);
+
+ if (!netdev_uc_empty(netdev)) {
+ struct netdev_hw_addr *ha;
+
+ /* Add MAC filters one by one */
+ netdev_for_each_uc_addr(ha, netdev) {
+ hw->mac.ops.set_uc_addr(hw, E1000_VF_MAC_FILTER_ADD,
+ ha->addr);
+ udelay(200);
+ }
+ }
+
+ return 0;
+}
+
+static void igbvf_set_rx_mode(struct net_device *netdev)
+{
+ igbvf_set_multi(netdev);
+ igbvf_set_uni(netdev);
+}
+
+/**
* igbvf_configure - configure the hardware for Rx and Tx
* @adapter: private board structure
**/
static void igbvf_configure(struct igbvf_adapter *adapter)
{
- igbvf_set_multi(adapter->netdev);
+ igbvf_set_rx_mode(adapter->netdev);
igbvf_restore_vlan(adapter);
@@ -1798,7 +1838,7 @@
UPDATE_VF_COUNTER(VFGPRLBC, gprlbc);
/* Fill out the OS statistics structure */
- adapter->net_stats.multicast = adapter->stats.mprc;
+ adapter->netdev->stats.multicast = adapter->stats.mprc;
}
static void igbvf_print_link_info(struct igbvf_adapter *adapter)
@@ -2334,21 +2374,6 @@
}
/**
- * igbvf_get_stats - Get System Network Statistics
- * @netdev: network interface device structure
- *
- * Returns the address of the device statistics structure.
- * The statistics are actually updated from the timer callback.
- **/
-static struct net_device_stats *igbvf_get_stats(struct net_device *netdev)
-{
- struct igbvf_adapter *adapter = netdev_priv(netdev);
-
- /* only return the current stats */
- return &adapter->net_stats;
-}
-
-/**
* igbvf_change_mtu - Change the Maximum Transfer Unit
* @netdev: network interface device structure
* @new_mtu: new value for maximum frame size
@@ -2635,8 +2660,7 @@
.ndo_open = igbvf_open,
.ndo_stop = igbvf_close,
.ndo_start_xmit = igbvf_xmit_frame,
- .ndo_get_stats = igbvf_get_stats,
- .ndo_set_rx_mode = igbvf_set_multi,
+ .ndo_set_rx_mode = igbvf_set_rx_mode,
.ndo_set_mac_address = igbvf_set_mac,
.ndo_change_mtu = igbvf_change_mtu,
.ndo_do_ioctl = igbvf_ioctl,
diff --git a/drivers/net/ethernet/intel/igbvf/vf.c b/drivers/net/ethernet/intel/igbvf/vf.c
index 335ba66..528be11 100644
--- a/drivers/net/ethernet/intel/igbvf/vf.c
+++ b/drivers/net/ethernet/intel/igbvf/vf.c
@@ -36,6 +36,7 @@
u32, u32, u32);
static void e1000_rar_set_vf(struct e1000_hw *, u8 *, u32);
static s32 e1000_read_mac_addr_vf(struct e1000_hw *);
+static s32 e1000_set_uc_addr_vf(struct e1000_hw *hw, u32 subcmd, u8 *addr);
static s32 e1000_set_vfta_vf(struct e1000_hw *, u16, bool);
/**
@@ -66,6 +67,8 @@
mac->ops.rar_set = e1000_rar_set_vf;
/* read mac address */
mac->ops.read_mac_addr = e1000_read_mac_addr_vf;
+ /* set mac filter */
+ mac->ops.set_uc_addr = e1000_set_uc_addr_vf;
/* set vlan filter table array */
mac->ops.set_vfta = e1000_set_vfta_vf;
@@ -338,6 +341,44 @@
}
/**
+ * e1000_set_uc_addr_vf - Set or clear unicast filters
+ * @hw: pointer to the HW structure
+ * @sub_cmd: add or clear filters
+ * @addr: pointer to the filter MAC address
+ **/
+static s32 e1000_set_uc_addr_vf(struct e1000_hw *hw, u32 sub_cmd, u8 *addr)
+{
+ struct e1000_mbx_info *mbx = &hw->mbx;
+ u32 msgbuf[3], msgbuf_chk;
+ u8 *msg_addr = (u8 *)(&msgbuf[1]);
+ s32 ret_val;
+
+ memset(msgbuf, 0, sizeof(msgbuf));
+ msgbuf[0] |= sub_cmd;
+ msgbuf[0] |= E1000_VF_SET_MAC_ADDR;
+ msgbuf_chk = msgbuf[0];
+
+ if (addr)
+ memcpy(msg_addr, addr, ETH_ALEN);
+
+ ret_val = mbx->ops.write_posted(hw, msgbuf, 3);
+
+ if (!ret_val)
+ ret_val = mbx->ops.read_posted(hw, msgbuf, 3);
+
+ msgbuf[0] &= ~E1000_VT_MSGTYPE_CTS;
+
+ if (!ret_val) {
+ msgbuf[0] &= ~E1000_VT_MSGTYPE_CTS;
+
+ if (msgbuf[0] == (msgbuf_chk | E1000_VT_MSGTYPE_NACK))
+ return -ENOSPC;
+ }
+
+ return ret_val;
+}
+
+/**
* e1000_check_for_link_vf - Check for link for a virtual interface
* @hw: pointer to the HW structure
*
diff --git a/drivers/net/ethernet/intel/igbvf/vf.h b/drivers/net/ethernet/intel/igbvf/vf.h
index f00a41d..4cf78b0 100644
--- a/drivers/net/ethernet/intel/igbvf/vf.h
+++ b/drivers/net/ethernet/intel/igbvf/vf.h
@@ -179,6 +179,7 @@
s32 (*get_bus_info)(struct e1000_hw *);
s32 (*get_link_up_info)(struct e1000_hw *, u16 *, u16 *);
void (*update_mc_addr_list)(struct e1000_hw *, u8 *, u32, u32, u32);
+ s32 (*set_uc_addr)(struct e1000_hw *, u32, u8 *);
s32 (*reset_hw)(struct e1000_hw *);
s32 (*init_hw)(struct e1000_hw *);
s32 (*setup_link)(struct e1000_hw *);
diff --git a/drivers/net/ethernet/intel/ixgb/ixgb_ethtool.c b/drivers/net/ethernet/intel/ixgb/ixgb_ethtool.c
index e5d7255..d10a0d2 100644
--- a/drivers/net/ethernet/intel/ixgb/ixgb_ethtool.c
+++ b/drivers/net/ethernet/intel/ixgb/ixgb_ethtool.c
@@ -94,24 +94,30 @@
#define IXGB_STATS_LEN ARRAY_SIZE(ixgb_gstrings_stats)
static int
-ixgb_get_settings(struct net_device *netdev, struct ethtool_cmd *ecmd)
+ixgb_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
{
struct ixgb_adapter *adapter = netdev_priv(netdev);
- ecmd->supported = (SUPPORTED_10000baseT_Full | SUPPORTED_FIBRE);
- ecmd->advertising = (ADVERTISED_10000baseT_Full | ADVERTISED_FIBRE);
- ecmd->port = PORT_FIBRE;
- ecmd->transceiver = XCVR_EXTERNAL;
+ ethtool_link_ksettings_zero_link_mode(cmd, supported);
+ ethtool_link_ksettings_add_link_mode(cmd, supported, 10000baseT_Full);
+ ethtool_link_ksettings_add_link_mode(cmd, supported, FIBRE);
+
+ ethtool_link_ksettings_zero_link_mode(cmd, advertising);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, 10000baseT_Full);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, FIBRE);
+
+ cmd->base.port = PORT_FIBRE;
if (netif_carrier_ok(adapter->netdev)) {
- ethtool_cmd_speed_set(ecmd, SPEED_10000);
- ecmd->duplex = DUPLEX_FULL;
+ cmd->base.speed = SPEED_10000;
+ cmd->base.duplex = DUPLEX_FULL;
} else {
- ethtool_cmd_speed_set(ecmd, SPEED_UNKNOWN);
- ecmd->duplex = DUPLEX_UNKNOWN;
+ cmd->base.speed = SPEED_UNKNOWN;
+ cmd->base.duplex = DUPLEX_UNKNOWN;
}
- ecmd->autoneg = AUTONEG_DISABLE;
+ cmd->base.autoneg = AUTONEG_DISABLE;
return 0;
}
@@ -126,13 +132,14 @@
}
static int
-ixgb_set_settings(struct net_device *netdev, struct ethtool_cmd *ecmd)
+ixgb_set_link_ksettings(struct net_device *netdev,
+ const struct ethtool_link_ksettings *cmd)
{
struct ixgb_adapter *adapter = netdev_priv(netdev);
- u32 speed = ethtool_cmd_speed(ecmd);
+ u32 speed = cmd->base.speed;
- if (ecmd->autoneg == AUTONEG_ENABLE ||
- (speed + ecmd->duplex != SPEED_10000 + DUPLEX_FULL))
+ if (cmd->base.autoneg == AUTONEG_ENABLE ||
+ (speed + cmd->base.duplex != SPEED_10000 + DUPLEX_FULL))
return -EINVAL;
if (netif_running(adapter->netdev)) {
@@ -630,8 +637,6 @@
}
static const struct ethtool_ops ixgb_ethtool_ops = {
- .get_settings = ixgb_get_settings,
- .set_settings = ixgb_set_settings,
.get_drvinfo = ixgb_get_drvinfo,
.get_regs_len = ixgb_get_regs_len,
.get_regs = ixgb_get_regs,
@@ -649,6 +654,8 @@
.set_phys_id = ixgb_set_phys_id,
.get_sset_count = ixgb_get_sset_count,
.get_ethtool_stats = ixgb_get_ethtool_stats,
+ .get_link_ksettings = ixgb_get_link_ksettings,
+ .set_link_ksettings = ixgb_set_link_ksettings,
};
void ixgb_set_ethtool_ops(struct net_device *netdev)
diff --git a/drivers/net/ethernet/intel/ixgb/ixgb_main.c b/drivers/net/ethernet/intel/ixgb/ixgb_main.c
index fbd220d..5a71319 100644
--- a/drivers/net/ethernet/intel/ixgb/ixgb_main.c
+++ b/drivers/net/ethernet/intel/ixgb/ixgb_main.c
@@ -86,7 +86,6 @@
static void ixgb_watchdog(unsigned long data);
static netdev_tx_t ixgb_xmit_frame(struct sk_buff *skb,
struct net_device *netdev);
-static struct net_device_stats *ixgb_get_stats(struct net_device *netdev);
static int ixgb_change_mtu(struct net_device *netdev, int new_mtu);
static int ixgb_set_mac(struct net_device *netdev, void *p);
static irqreturn_t ixgb_intr(int irq, void *data);
@@ -367,7 +366,6 @@
.ndo_open = ixgb_open,
.ndo_stop = ixgb_close,
.ndo_start_xmit = ixgb_xmit_frame,
- .ndo_get_stats = ixgb_get_stats,
.ndo_set_rx_mode = ixgb_set_multi,
.ndo_validate_addr = eth_validate_addr,
.ndo_set_mac_address = ixgb_set_mac,
@@ -1597,20 +1595,6 @@
}
/**
- * ixgb_get_stats - Get System Network Statistics
- * @netdev: network interface device structure
- *
- * Returns the address of the device statistics structure.
- * The statistics are actually updated from the timer callback.
- **/
-
-static struct net_device_stats *
-ixgb_get_stats(struct net_device *netdev)
-{
- return &netdev->stats;
-}
-
-/**
* ixgb_change_mtu - Change the Maximum Transfer Unit
* @netdev: network interface device structure
* @new_mtu: new value for maximum frame size
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe.h b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
index b1ecc26..7626376 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
@@ -86,17 +86,62 @@
/* Supported Rx Buffer Sizes */
#define IXGBE_RXBUFFER_256 256 /* Used for skb receive header */
+#define IXGBE_RXBUFFER_1536 1536
#define IXGBE_RXBUFFER_2K 2048
#define IXGBE_RXBUFFER_3K 3072
#define IXGBE_RXBUFFER_4K 4096
#define IXGBE_MAX_RXBUFFER 16384 /* largest size for a single descriptor */
-#define IXGBE_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN)
+/* Attempt to maximize the headroom available for incoming frames. We
+ * use a 2K buffer for receives and need 1536/1534 to store the data for
+ * the frame. This leaves us with 512 bytes of room. From that we need
+ * to deduct the space needed for the shared info and the padding needed
+ * to IP align the frame.
+ *
+ * Note: For cache line sizes 256 or larger this value is going to end
+ * up negative. In these cases we should fall back to the 3K
+ * buffers.
+ */
#if (PAGE_SIZE < 8192)
-#define IXGBE_MAX_FRAME_BUILD_SKB \
- (SKB_WITH_OVERHEAD(IXGBE_RXBUFFER_2K) - IXGBE_SKB_PAD)
+#define IXGBE_MAX_2K_FRAME_BUILD_SKB (IXGBE_RXBUFFER_1536 - NET_IP_ALIGN)
+#define IXGBE_2K_TOO_SMALL_WITH_PADDING \
+((NET_SKB_PAD + IXGBE_RXBUFFER_1536) > SKB_WITH_OVERHEAD(IXGBE_RXBUFFER_2K))
+
+static inline int ixgbe_compute_pad(int rx_buf_len)
+{
+ int page_size, pad_size;
+
+ page_size = ALIGN(rx_buf_len, PAGE_SIZE / 2);
+ pad_size = SKB_WITH_OVERHEAD(page_size) - rx_buf_len;
+
+ return pad_size;
+}
+
+static inline int ixgbe_skb_pad(void)
+{
+ int rx_buf_len;
+
+ /* If a 2K buffer cannot handle a standard Ethernet frame then
+ * optimize padding for a 3K buffer instead of a 1.5K buffer.
+ *
+ * For a 3K buffer we need to add enough padding to allow for
+ * tailroom due to NET_IP_ALIGN possibly shifting us out of
+ * cache-line alignment.
+ */
+ if (IXGBE_2K_TOO_SMALL_WITH_PADDING)
+ rx_buf_len = IXGBE_RXBUFFER_3K + SKB_DATA_ALIGN(NET_IP_ALIGN);
+ else
+ rx_buf_len = IXGBE_RXBUFFER_1536;
+
+ /* if needed make room for NET_IP_ALIGN */
+ rx_buf_len -= NET_IP_ALIGN;
+
+ return ixgbe_compute_pad(rx_buf_len);
+}
+
+#define IXGBE_SKB_PAD ixgbe_skb_pad()
#else
-#define IXGBE_MAX_FRAME_BUILD_SKB IXGBE_RXBUFFER_2K
+#define IXGBE_SKB_PAD (NET_SKB_PAD + NET_IP_ALIGN)
#endif
/*
@@ -190,7 +235,11 @@
struct ixgbe_tx_buffer {
union ixgbe_adv_tx_desc *next_to_watch;
unsigned long time_stamp;
- struct sk_buff *skb;
+ union {
+ struct sk_buff *skb;
+ /* XDP uses address ptr on irq_clean */
+ void *data;
+ };
unsigned int bytecount;
unsigned short gso_segs;
__be16 protocol;
@@ -243,6 +292,7 @@
__IXGBE_TX_XPS_INIT_DONE,
__IXGBE_TX_DETECT_HANG,
__IXGBE_HANG_CHECK_ARMED,
+ __IXGBE_TX_XDP_RING,
};
#define ring_uses_build_skb(ring) \
@@ -269,10 +319,17 @@
set_bit(__IXGBE_RX_RSC_ENABLED, &(ring)->state)
#define clear_ring_rsc_enabled(ring) \
clear_bit(__IXGBE_RX_RSC_ENABLED, &(ring)->state)
+#define ring_is_xdp(ring) \
+ test_bit(__IXGBE_TX_XDP_RING, &(ring)->state)
+#define set_ring_xdp(ring) \
+ set_bit(__IXGBE_TX_XDP_RING, &(ring)->state)
+#define clear_ring_xdp(ring) \
+ clear_bit(__IXGBE_TX_XDP_RING, &(ring)->state)
struct ixgbe_ring {
struct ixgbe_ring *next; /* pointer to next ring in q_vector */
struct ixgbe_q_vector *q_vector; /* backpointer to host q_vector */
struct net_device *netdev; /* netdev ring belongs to */
+ struct bpf_prog *xdp_prog;
struct device *dev; /* device for DMA mapping */
struct ixgbe_fwd_adapter *l2_accel_priv;
void *desc; /* descriptor ring memory */
@@ -334,6 +391,7 @@
#define IXGBE_MAX_FCOE_INDICES 8
#define MAX_RX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)
#define MAX_TX_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)
+#define MAX_XDP_QUEUES (IXGBE_MAX_FDIR_INDICES + 1)
#define IXGBE_MAX_L2A_QUEUES 4
#define IXGBE_BAD_L2A_QUEUE 3
#define IXGBE_MAX_MACVLANS 31
@@ -361,7 +419,7 @@
return IXGBE_RXBUFFER_3K;
#if (PAGE_SIZE < 8192)
if (ring_uses_build_skb(ring))
- return IXGBE_MAX_FRAME_BUILD_SKB;
+ return IXGBE_MAX_2K_FRAME_BUILD_SKB;
#endif
return IXGBE_RXBUFFER_2K;
}
@@ -510,6 +568,7 @@
unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)];
/* OS defined structs */
struct net_device *netdev;
+ struct bpf_prog *xdp_prog;
struct pci_dev *pdev;
unsigned long state;
@@ -576,6 +635,10 @@
__be16 vxlan_port;
__be16 geneve_port;
+ /* XDP */
+ int num_xdp_queues;
+ struct ixgbe_ring *xdp_ring[MAX_XDP_QUEUES];
+
/* TX */
struct ixgbe_ring *tx_ring[MAX_TX_QUEUES] ____cacheline_aligned_in_smp;
@@ -622,6 +685,7 @@
u64 tx_busy;
unsigned int tx_ring_count;
+ unsigned int xdp_ring_count;
unsigned int rx_ring_count;
u32 link_speed;
@@ -705,7 +769,7 @@
u8 rss_indir_tbl[IXGBE_MAX_RETA_ENTRIES];
#define IXGBE_RSS_KEY_SIZE 40 /* size of RSS Hash Key in bytes */
- u32 rss_key[IXGBE_RSS_KEY_SIZE / sizeof(u32)];
+ u32 *rss_key;
};
static inline u8 ixgbe_max_rss_indices(struct ixgbe_adapter *adapter)
@@ -762,6 +826,7 @@
board_X540,
board_X550,
board_X550EM_x,
+ board_x550em_x_fw,
board_x550em_a,
board_x550em_a_fw,
};
@@ -771,6 +836,7 @@
extern const struct ixgbe_info ixgbe_X540_info;
extern const struct ixgbe_info ixgbe_X550_info;
extern const struct ixgbe_info ixgbe_X550EM_x_info;
+extern const struct ixgbe_info ixgbe_x550em_x_fw_info;
extern const struct ixgbe_info ixgbe_x550em_a_info;
extern const struct ixgbe_info ixgbe_x550em_a_fw_info;
#ifdef CONFIG_IXGBE_DCB
@@ -790,7 +856,7 @@
void ixgbe_reinit_locked(struct ixgbe_adapter *adapter);
void ixgbe_reset(struct ixgbe_adapter *adapter);
void ixgbe_set_ethtool_ops(struct net_device *netdev);
-int ixgbe_setup_rx_resources(struct ixgbe_ring *);
+int ixgbe_setup_rx_resources(struct ixgbe_adapter *, struct ixgbe_ring *);
int ixgbe_setup_tx_resources(struct ixgbe_ring *);
void ixgbe_free_rx_resources(struct ixgbe_ring *);
void ixgbe_free_tx_resources(struct ixgbe_ring *);
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
index 90fa5bf..7e5e336 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c
@@ -179,6 +179,7 @@
case IXGBE_DEV_ID_82598_BX:
case IXGBE_DEV_ID_82599_KR:
case IXGBE_DEV_ID_X550EM_X_KR:
+ case IXGBE_DEV_ID_X550EM_X_XFI:
return SUPPORTED_10000baseKR_Full;
default:
return SUPPORTED_10000baseKX4_Full |
@@ -186,60 +187,62 @@
}
}
-static int ixgbe_get_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+static int ixgbe_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
{
struct ixgbe_adapter *adapter = netdev_priv(netdev);
struct ixgbe_hw *hw = &adapter->hw;
ixgbe_link_speed supported_link;
bool autoneg = false;
+ u32 supported, advertising;
+
+ ethtool_convert_link_mode_to_legacy_u32(&supported,
+ cmd->link_modes.supported);
hw->mac.ops.get_link_capabilities(hw, &supported_link, &autoneg);
/* set the supported link speeds */
if (supported_link & IXGBE_LINK_SPEED_10GB_FULL)
- ecmd->supported |= ixgbe_get_supported_10gtypes(hw);
+ supported |= ixgbe_get_supported_10gtypes(hw);
if (supported_link & IXGBE_LINK_SPEED_1GB_FULL)
- ecmd->supported |= (ixgbe_isbackplane(hw->phy.media_type)) ?
+ supported |= (ixgbe_isbackplane(hw->phy.media_type)) ?
SUPPORTED_1000baseKX_Full :
SUPPORTED_1000baseT_Full;
if (supported_link & IXGBE_LINK_SPEED_100_FULL)
- ecmd->supported |= SUPPORTED_100baseT_Full;
+ supported |= SUPPORTED_100baseT_Full;
if (supported_link & IXGBE_LINK_SPEED_10_FULL)
- ecmd->supported |= SUPPORTED_10baseT_Full;
+ supported |= SUPPORTED_10baseT_Full;
/* default advertised speed if phy.autoneg_advertised isn't set */
- ecmd->advertising = ecmd->supported;
+ advertising = supported;
/* set the advertised speeds */
if (hw->phy.autoneg_advertised) {
- ecmd->advertising = 0;
+ advertising = 0;
if (hw->phy.autoneg_advertised & IXGBE_LINK_SPEED_10_FULL)
- ecmd->advertising |= ADVERTISED_10baseT_Full;
+ advertising |= ADVERTISED_10baseT_Full;
if (hw->phy.autoneg_advertised & IXGBE_LINK_SPEED_100_FULL)
- ecmd->advertising |= ADVERTISED_100baseT_Full;
+ advertising |= ADVERTISED_100baseT_Full;
if (hw->phy.autoneg_advertised & IXGBE_LINK_SPEED_10GB_FULL)
- ecmd->advertising |= ecmd->supported & ADVRTSD_MSK_10G;
+ advertising |= supported & ADVRTSD_MSK_10G;
if (hw->phy.autoneg_advertised & IXGBE_LINK_SPEED_1GB_FULL) {
- if (ecmd->supported & SUPPORTED_1000baseKX_Full)
- ecmd->advertising |= ADVERTISED_1000baseKX_Full;
+ if (supported & SUPPORTED_1000baseKX_Full)
+ advertising |= ADVERTISED_1000baseKX_Full;
else
- ecmd->advertising |= ADVERTISED_1000baseT_Full;
+ advertising |= ADVERTISED_1000baseT_Full;
}
} else {
if (hw->phy.multispeed_fiber && !autoneg) {
if (supported_link & IXGBE_LINK_SPEED_10GB_FULL)
- ecmd->advertising = ADVERTISED_10000baseT_Full;
+ advertising = ADVERTISED_10000baseT_Full;
}
}
if (autoneg) {
- ecmd->supported |= SUPPORTED_Autoneg;
- ecmd->advertising |= ADVERTISED_Autoneg;
- ecmd->autoneg = AUTONEG_ENABLE;
+ supported |= SUPPORTED_Autoneg;
+ advertising |= ADVERTISED_Autoneg;
+ cmd->base.autoneg = AUTONEG_ENABLE;
} else
- ecmd->autoneg = AUTONEG_DISABLE;
-
- ecmd->transceiver = XCVR_EXTERNAL;
+ cmd->base.autoneg = AUTONEG_DISABLE;
/* Determine the remaining settings based on the PHY type. */
switch (adapter->hw.phy.type) {
@@ -248,14 +251,14 @@
case ixgbe_phy_x550em_ext_t:
case ixgbe_phy_fw:
case ixgbe_phy_cu_unknown:
- ecmd->supported |= SUPPORTED_TP;
- ecmd->advertising |= ADVERTISED_TP;
- ecmd->port = PORT_TP;
+ supported |= SUPPORTED_TP;
+ advertising |= ADVERTISED_TP;
+ cmd->base.port = PORT_TP;
break;
case ixgbe_phy_qt:
- ecmd->supported |= SUPPORTED_FIBRE;
- ecmd->advertising |= ADVERTISED_FIBRE;
- ecmd->port = PORT_FIBRE;
+ supported |= SUPPORTED_FIBRE;
+ advertising |= ADVERTISED_FIBRE;
+ cmd->base.port = PORT_FIBRE;
break;
case ixgbe_phy_nl:
case ixgbe_phy_sfp_passive_tyco:
@@ -273,9 +276,9 @@
case ixgbe_sfp_type_da_cu:
case ixgbe_sfp_type_da_cu_core0:
case ixgbe_sfp_type_da_cu_core1:
- ecmd->supported |= SUPPORTED_FIBRE;
- ecmd->advertising |= ADVERTISED_FIBRE;
- ecmd->port = PORT_DA;
+ supported |= SUPPORTED_FIBRE;
+ advertising |= ADVERTISED_FIBRE;
+ cmd->base.port = PORT_DA;
break;
case ixgbe_sfp_type_sr:
case ixgbe_sfp_type_lr:
@@ -285,102 +288,113 @@
case ixgbe_sfp_type_1g_sx_core1:
case ixgbe_sfp_type_1g_lx_core0:
case ixgbe_sfp_type_1g_lx_core1:
- ecmd->supported |= SUPPORTED_FIBRE;
- ecmd->advertising |= ADVERTISED_FIBRE;
- ecmd->port = PORT_FIBRE;
+ supported |= SUPPORTED_FIBRE;
+ advertising |= ADVERTISED_FIBRE;
+ cmd->base.port = PORT_FIBRE;
break;
case ixgbe_sfp_type_not_present:
- ecmd->supported |= SUPPORTED_FIBRE;
- ecmd->advertising |= ADVERTISED_FIBRE;
- ecmd->port = PORT_NONE;
+ supported |= SUPPORTED_FIBRE;
+ advertising |= ADVERTISED_FIBRE;
+ cmd->base.port = PORT_NONE;
break;
case ixgbe_sfp_type_1g_cu_core0:
case ixgbe_sfp_type_1g_cu_core1:
- ecmd->supported |= SUPPORTED_TP;
- ecmd->advertising |= ADVERTISED_TP;
- ecmd->port = PORT_TP;
+ supported |= SUPPORTED_TP;
+ advertising |= ADVERTISED_TP;
+ cmd->base.port = PORT_TP;
break;
case ixgbe_sfp_type_unknown:
default:
- ecmd->supported |= SUPPORTED_FIBRE;
- ecmd->advertising |= ADVERTISED_FIBRE;
- ecmd->port = PORT_OTHER;
+ supported |= SUPPORTED_FIBRE;
+ advertising |= ADVERTISED_FIBRE;
+ cmd->base.port = PORT_OTHER;
break;
}
break;
case ixgbe_phy_xaui:
- ecmd->supported |= SUPPORTED_FIBRE;
- ecmd->advertising |= ADVERTISED_FIBRE;
- ecmd->port = PORT_NONE;
+ supported |= SUPPORTED_FIBRE;
+ advertising |= ADVERTISED_FIBRE;
+ cmd->base.port = PORT_NONE;
break;
case ixgbe_phy_unknown:
case ixgbe_phy_generic:
case ixgbe_phy_sfp_unsupported:
default:
- ecmd->supported |= SUPPORTED_FIBRE;
- ecmd->advertising |= ADVERTISED_FIBRE;
- ecmd->port = PORT_OTHER;
+ supported |= SUPPORTED_FIBRE;
+ advertising |= ADVERTISED_FIBRE;
+ cmd->base.port = PORT_OTHER;
break;
}
/* Indicate pause support */
- ecmd->supported |= SUPPORTED_Pause;
+ supported |= SUPPORTED_Pause;
switch (hw->fc.requested_mode) {
case ixgbe_fc_full:
- ecmd->advertising |= ADVERTISED_Pause;
+ advertising |= ADVERTISED_Pause;
break;
case ixgbe_fc_rx_pause:
- ecmd->advertising |= ADVERTISED_Pause |
+ advertising |= ADVERTISED_Pause |
ADVERTISED_Asym_Pause;
break;
case ixgbe_fc_tx_pause:
- ecmd->advertising |= ADVERTISED_Asym_Pause;
+ advertising |= ADVERTISED_Asym_Pause;
break;
default:
- ecmd->advertising &= ~(ADVERTISED_Pause |
+ advertising &= ~(ADVERTISED_Pause |
ADVERTISED_Asym_Pause);
}
if (netif_carrier_ok(netdev)) {
switch (adapter->link_speed) {
case IXGBE_LINK_SPEED_10GB_FULL:
- ethtool_cmd_speed_set(ecmd, SPEED_10000);
+ cmd->base.speed = SPEED_10000;
break;
case IXGBE_LINK_SPEED_5GB_FULL:
- ethtool_cmd_speed_set(ecmd, SPEED_5000);
+ cmd->base.speed = SPEED_5000;
break;
case IXGBE_LINK_SPEED_2_5GB_FULL:
- ethtool_cmd_speed_set(ecmd, SPEED_2500);
+ cmd->base.speed = SPEED_2500;
break;
case IXGBE_LINK_SPEED_1GB_FULL:
- ethtool_cmd_speed_set(ecmd, SPEED_1000);
+ cmd->base.speed = SPEED_1000;
break;
case IXGBE_LINK_SPEED_100_FULL:
- ethtool_cmd_speed_set(ecmd, SPEED_100);
+ cmd->base.speed = SPEED_100;
break;
case IXGBE_LINK_SPEED_10_FULL:
- ethtool_cmd_speed_set(ecmd, SPEED_10);
+ cmd->base.speed = SPEED_10;
break;
default:
break;
}
- ecmd->duplex = DUPLEX_FULL;
+ cmd->base.duplex = DUPLEX_FULL;
} else {
- ethtool_cmd_speed_set(ecmd, SPEED_UNKNOWN);
- ecmd->duplex = DUPLEX_UNKNOWN;
+ cmd->base.speed = SPEED_UNKNOWN;
+ cmd->base.duplex = DUPLEX_UNKNOWN;
}
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
+ advertising);
+
return 0;
}
-static int ixgbe_set_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+static int ixgbe_set_link_ksettings(struct net_device *netdev,
+ const struct ethtool_link_ksettings *cmd)
{
struct ixgbe_adapter *adapter = netdev_priv(netdev);
struct ixgbe_hw *hw = &adapter->hw;
u32 advertised, old;
s32 err = 0;
+ u32 supported, advertising;
+
+ ethtool_convert_link_mode_to_legacy_u32(&supported,
+ cmd->link_modes.supported);
+ ethtool_convert_link_mode_to_legacy_u32(&advertising,
+ cmd->link_modes.advertising);
if ((hw->phy.media_type == ixgbe_media_type_copper) ||
(hw->phy.multispeed_fiber)) {
@@ -388,12 +402,12 @@
* this function does not support duplex forcing, but can
* limit the advertising of the adapter to the specified speed
*/
- if (ecmd->advertising & ~ecmd->supported)
+ if (advertising & ~supported)
return -EINVAL;
/* only allow one speed at a time if no autoneg */
- if (!ecmd->autoneg && hw->phy.multispeed_fiber) {
- if (ecmd->advertising ==
+ if (!cmd->base.autoneg && hw->phy.multispeed_fiber) {
+ if (advertising ==
(ADVERTISED_10000baseT_Full |
ADVERTISED_1000baseT_Full))
return -EINVAL;
@@ -401,16 +415,16 @@
old = hw->phy.autoneg_advertised;
advertised = 0;
- if (ecmd->advertising & ADVERTISED_10000baseT_Full)
+ if (advertising & ADVERTISED_10000baseT_Full)
advertised |= IXGBE_LINK_SPEED_10GB_FULL;
- if (ecmd->advertising & ADVERTISED_1000baseT_Full)
+ if (advertising & ADVERTISED_1000baseT_Full)
advertised |= IXGBE_LINK_SPEED_1GB_FULL;
- if (ecmd->advertising & ADVERTISED_100baseT_Full)
+ if (advertising & ADVERTISED_100baseT_Full)
advertised |= IXGBE_LINK_SPEED_100_FULL;
- if (ecmd->advertising & ADVERTISED_10baseT_Full)
+ if (advertising & ADVERTISED_10baseT_Full)
advertised |= IXGBE_LINK_SPEED_10_FULL;
if (old == advertised)
@@ -428,10 +442,11 @@
clear_bit(__IXGBE_IN_SFP_INIT, &adapter->state);
} else {
/* in this case we currently only support 10Gb/FULL */
- u32 speed = ethtool_cmd_speed(ecmd);
- if ((ecmd->autoneg == AUTONEG_ENABLE) ||
- (ecmd->advertising != ADVERTISED_10000baseT_Full) ||
- (speed + ecmd->duplex != SPEED_10000 + DUPLEX_FULL))
+ u32 speed = cmd->base.speed;
+
+ if ((cmd->base.autoneg == AUTONEG_ENABLE) ||
+ (advertising != ADVERTISED_10000baseT_Full) ||
+ (speed + cmd->base.duplex != SPEED_10000 + DUPLEX_FULL))
return -EINVAL;
}
@@ -1056,15 +1071,19 @@
if (!netif_running(adapter->netdev)) {
for (i = 0; i < adapter->num_tx_queues; i++)
adapter->tx_ring[i]->count = new_tx_count;
+ for (i = 0; i < adapter->num_xdp_queues; i++)
+ adapter->xdp_ring[i]->count = new_tx_count;
for (i = 0; i < adapter->num_rx_queues; i++)
adapter->rx_ring[i]->count = new_rx_count;
adapter->tx_ring_count = new_tx_count;
+ adapter->xdp_ring_count = new_tx_count;
adapter->rx_ring_count = new_rx_count;
goto clear_reset;
}
/* allocate temporary buffer to store rings in */
i = max_t(int, adapter->num_tx_queues, adapter->num_rx_queues);
+ i = max_t(int, i, adapter->num_xdp_queues);
temp_ring = vmalloc(i * sizeof(struct ixgbe_ring));
if (!temp_ring) {
@@ -1096,12 +1115,33 @@
}
}
+ for (i = 0; i < adapter->num_xdp_queues; i++) {
+ memcpy(&temp_ring[i], adapter->xdp_ring[i],
+ sizeof(struct ixgbe_ring));
+
+ temp_ring[i].count = new_tx_count;
+ err = ixgbe_setup_tx_resources(&temp_ring[i]);
+ if (err) {
+ while (i) {
+ i--;
+ ixgbe_free_tx_resources(&temp_ring[i]);
+ }
+ goto err_setup;
+ }
+ }
+
for (i = 0; i < adapter->num_tx_queues; i++) {
ixgbe_free_tx_resources(adapter->tx_ring[i]);
memcpy(adapter->tx_ring[i], &temp_ring[i],
sizeof(struct ixgbe_ring));
}
+ for (i = 0; i < adapter->num_xdp_queues; i++) {
+ ixgbe_free_tx_resources(adapter->xdp_ring[i]);
+
+ memcpy(adapter->xdp_ring[i], &temp_ring[i],
+ sizeof(struct ixgbe_ring));
+ }
adapter->tx_ring_count = new_tx_count;
}
@@ -1113,7 +1153,7 @@
sizeof(struct ixgbe_ring));
temp_ring[i].count = new_rx_count;
- err = ixgbe_setup_rx_resources(&temp_ring[i]);
+ err = ixgbe_setup_rx_resources(adapter, &temp_ring[i]);
if (err) {
while (i) {
i--;
@@ -1746,7 +1786,7 @@
rx_ring->netdev = adapter->netdev;
rx_ring->reg_idx = adapter->rx_ring[0]->reg_idx;
- err = ixgbe_setup_rx_resources(rx_ring);
+ err = ixgbe_setup_rx_resources(adapter, rx_ring);
if (err) {
ret_val = 4;
goto err_nomem;
@@ -2927,9 +2967,7 @@
static u32 ixgbe_get_rxfh_key_size(struct net_device *netdev)
{
- struct ixgbe_adapter *adapter = netdev_priv(netdev);
-
- return sizeof(adapter->rss_key);
+ return IXGBE_RSS_KEY_SIZE;
}
static u32 ixgbe_rss_indir_size(struct net_device *netdev)
@@ -3402,8 +3440,6 @@
}
static const struct ethtool_ops ixgbe_ethtool_ops = {
- .get_settings = ixgbe_get_settings,
- .set_settings = ixgbe_set_settings,
.get_drvinfo = ixgbe_get_drvinfo,
.get_regs_len = ixgbe_get_regs_len,
.get_regs = ixgbe_get_regs,
@@ -3442,6 +3478,8 @@
.get_ts_info = ixgbe_get_ts_info,
.get_module_info = ixgbe_get_module_info,
.get_module_eeprom = ixgbe_get_module_eeprom,
+ .get_link_ksettings = ixgbe_get_link_ksettings,
+ .set_link_ksettings = ixgbe_set_link_ksettings,
};
void ixgbe_set_ethtool_ops(struct net_device *netdev)
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c
index 1b8be7d..b45fdc9 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c
@@ -267,12 +267,14 @@
**/
static bool ixgbe_cache_ring_rss(struct ixgbe_adapter *adapter)
{
- int i;
+ int i, reg_idx;
for (i = 0; i < adapter->num_rx_queues; i++)
adapter->rx_ring[i]->reg_idx = i;
- for (i = 0; i < adapter->num_tx_queues; i++)
- adapter->tx_ring[i]->reg_idx = i;
+ for (i = 0, reg_idx = 0; i < adapter->num_tx_queues; i++, reg_idx++)
+ adapter->tx_ring[i]->reg_idx = reg_idx;
+ for (i = 0; i < adapter->num_xdp_queues; i++, reg_idx++)
+ adapter->xdp_ring[i]->reg_idx = reg_idx;
return true;
}
@@ -308,6 +310,11 @@
ixgbe_cache_ring_rss(adapter);
}
+static int ixgbe_xdp_queues(struct ixgbe_adapter *adapter)
+{
+ return adapter->xdp_prog ? nr_cpu_ids : 0;
+}
+
#define IXGBE_RSS_64Q_MASK 0x3F
#define IXGBE_RSS_16Q_MASK 0xF
#define IXGBE_RSS_8Q_MASK 0x7
@@ -382,6 +389,7 @@
adapter->num_rx_queues_per_pool = tcs;
adapter->num_tx_queues = vmdq_i * tcs;
+ adapter->num_xdp_queues = 0;
adapter->num_rx_queues = vmdq_i * tcs;
#ifdef IXGBE_FCOE
@@ -479,6 +487,7 @@
netdev_set_tc_queue(dev, i, rss_i, rss_i * i);
adapter->num_tx_queues = rss_i * tcs;
+ adapter->num_xdp_queues = 0;
adapter->num_rx_queues = rss_i * tcs;
return true;
@@ -549,6 +558,7 @@
adapter->num_rx_queues = vmdq_i * rss_i;
adapter->num_tx_queues = vmdq_i * rss_i;
+ adapter->num_xdp_queues = 0;
/* disable ATR as it is not supported when VMDq is enabled */
adapter->flags &= ~IXGBE_FLAG_FDIR_HASH_CAPABLE;
@@ -669,6 +679,7 @@
#endif /* IXGBE_FCOE */
adapter->num_rx_queues = rss_i;
adapter->num_tx_queues = rss_i;
+ adapter->num_xdp_queues = ixgbe_xdp_queues(adapter);
return true;
}
@@ -689,6 +700,7 @@
/* Start with base case */
adapter->num_rx_queues = 1;
adapter->num_tx_queues = 1;
+ adapter->num_xdp_queues = 0;
adapter->num_rx_pools = adapter->num_rx_queues;
adapter->num_rx_queues_per_pool = 1;
@@ -719,8 +731,11 @@
struct ixgbe_hw *hw = &adapter->hw;
int i, vectors, vector_threshold;
- /* We start by asking for one vector per queue pair */
+ /* We start by asking for one vector per queue pair with XDP queues
+ * being stacked with TX queues.
+ */
vectors = max(adapter->num_rx_queues, adapter->num_tx_queues);
+ vectors = max(vectors, adapter->num_xdp_queues);
/* It is easy to be greedy for MSI-X vectors. However, it really
* doesn't do much good if we have a lot more vectors than CPUs. We'll
@@ -800,6 +815,8 @@
* @v_idx: index of vector in adapter struct
* @txr_count: total number of Tx rings to allocate
* @txr_idx: index of first Tx ring to allocate
+ * @xdp_count: total number of XDP rings to allocate
+ * @xdp_idx: index of first XDP ring to allocate
* @rxr_count: total number of Rx rings to allocate
* @rxr_idx: index of first Rx ring to allocate
*
@@ -808,6 +825,7 @@
static int ixgbe_alloc_q_vector(struct ixgbe_adapter *adapter,
int v_count, int v_idx,
int txr_count, int txr_idx,
+ int xdp_count, int xdp_idx,
int rxr_count, int rxr_idx)
{
struct ixgbe_q_vector *q_vector;
@@ -817,7 +835,7 @@
int ring_count, size;
u8 tcs = netdev_get_num_tc(adapter->netdev);
- ring_count = txr_count + rxr_count;
+ ring_count = txr_count + rxr_count + xdp_count;
size = sizeof(struct ixgbe_q_vector) +
(sizeof(struct ixgbe_ring) * ring_count);
@@ -909,6 +927,33 @@
ring++;
}
+ while (xdp_count) {
+ /* assign generic ring traits */
+ ring->dev = &adapter->pdev->dev;
+ ring->netdev = adapter->netdev;
+
+ /* configure backlink on ring */
+ ring->q_vector = q_vector;
+
+ /* update q_vector Tx values */
+ ixgbe_add_ring(ring, &q_vector->tx);
+
+ /* apply Tx specific ring traits */
+ ring->count = adapter->tx_ring_count;
+ ring->queue_index = xdp_idx;
+ set_ring_xdp(ring);
+
+ /* assign ring to adapter */
+ adapter->xdp_ring[xdp_idx] = ring;
+
+ /* update count and index */
+ xdp_count--;
+ xdp_idx++;
+
+ /* push pointer to next ring */
+ ring++;
+ }
+
while (rxr_count) {
/* assign generic ring traits */
ring->dev = &adapter->pdev->dev;
@@ -1002,17 +1047,18 @@
int q_vectors = adapter->num_q_vectors;
int rxr_remaining = adapter->num_rx_queues;
int txr_remaining = adapter->num_tx_queues;
- int rxr_idx = 0, txr_idx = 0, v_idx = 0;
+ int xdp_remaining = adapter->num_xdp_queues;
+ int rxr_idx = 0, txr_idx = 0, xdp_idx = 0, v_idx = 0;
int err;
/* only one q_vector if MSI-X is disabled. */
if (!(adapter->flags & IXGBE_FLAG_MSIX_ENABLED))
q_vectors = 1;
- if (q_vectors >= (rxr_remaining + txr_remaining)) {
+ if (q_vectors >= (rxr_remaining + txr_remaining + xdp_remaining)) {
for (; rxr_remaining; v_idx++) {
err = ixgbe_alloc_q_vector(adapter, q_vectors, v_idx,
- 0, 0, 1, rxr_idx);
+ 0, 0, 0, 0, 1, rxr_idx);
if (err)
goto err_out;
@@ -1026,8 +1072,11 @@
for (; v_idx < q_vectors; v_idx++) {
int rqpv = DIV_ROUND_UP(rxr_remaining, q_vectors - v_idx);
int tqpv = DIV_ROUND_UP(txr_remaining, q_vectors - v_idx);
+ int xqpv = DIV_ROUND_UP(xdp_remaining, q_vectors - v_idx);
+
err = ixgbe_alloc_q_vector(adapter, q_vectors, v_idx,
tqpv, txr_idx,
+ xqpv, xdp_idx,
rqpv, rxr_idx);
if (err)
@@ -1036,14 +1085,17 @@
/* update counts and index */
rxr_remaining -= rqpv;
txr_remaining -= tqpv;
+ xdp_remaining -= xqpv;
rxr_idx++;
txr_idx++;
+ xdp_idx += xqpv;
}
return 0;
err_out:
adapter->num_tx_queues = 0;
+ adapter->num_xdp_queues = 0;
adapter->num_rx_queues = 0;
adapter->num_q_vectors = 0;
@@ -1066,6 +1118,7 @@
int v_idx = adapter->num_q_vectors;
adapter->num_tx_queues = 0;
+ adapter->num_xdp_queues = 0;
adapter->num_rx_queues = 0;
adapter->num_q_vectors = 0;
@@ -1172,9 +1225,10 @@
ixgbe_cache_ring_register(adapter);
- e_dev_info("Multiqueue %s: Rx Queue count = %u, Tx Queue count = %u\n",
+ e_dev_info("Multiqueue %s: Rx Queue count = %u, Tx Queue count = %u XDP Queue count = %u\n",
(adapter->num_rx_queues > 1) ? "Enabled" : "Disabled",
- adapter->num_rx_queues, adapter->num_tx_queues);
+ adapter->num_rx_queues, adapter->num_tx_queues,
+ adapter->num_xdp_queues);
set_bit(__IXGBE_DOWN, &adapter->state);
@@ -1195,6 +1249,7 @@
void ixgbe_clear_interrupt_scheme(struct ixgbe_adapter *adapter)
{
adapter->num_tx_queues = 0;
+ adapter->num_xdp_queues = 0;
adapter->num_rx_queues = 0;
ixgbe_free_q_vectors(adapter);
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index a7a430a..22a29df 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -49,6 +49,9 @@
#include <linux/if_macvlan.h>
#include <linux/if_bridge.h>
#include <linux/prefetch.h>
+#include <linux/bpf.h>
+#include <linux/bpf_trace.h>
+#include <linux/atomic.h>
#include <scsi/fc/fc_fcoe.h>
#include <net/udp_tunnel.h>
#include <net/pkt_cls.h>
@@ -85,6 +88,7 @@
[board_X540] = &ixgbe_X540_info,
[board_X550] = &ixgbe_X550_info,
[board_X550EM_x] = &ixgbe_X550EM_x_info,
+ [board_x550em_x_fw] = &ixgbe_x550em_x_fw_info,
[board_x550em_a] = &ixgbe_x550em_a_info,
[board_x550em_a_fw] = &ixgbe_x550em_a_fw_info,
};
@@ -131,9 +135,11 @@
{PCI_VDEVICE(INTEL, IXGBE_DEV_ID_X550T), board_X550},
{PCI_VDEVICE(INTEL, IXGBE_DEV_ID_X550T1), board_X550},
{PCI_VDEVICE(INTEL, IXGBE_DEV_ID_X550EM_X_KX4), board_X550EM_x},
+ {PCI_VDEVICE(INTEL, IXGBE_DEV_ID_X550EM_X_XFI), board_X550EM_x},
{PCI_VDEVICE(INTEL, IXGBE_DEV_ID_X550EM_X_KR), board_X550EM_x},
{PCI_VDEVICE(INTEL, IXGBE_DEV_ID_X550EM_X_10G_T), board_X550EM_x},
{PCI_VDEVICE(INTEL, IXGBE_DEV_ID_X550EM_X_SFP), board_X550EM_x},
+ {PCI_VDEVICE(INTEL, IXGBE_DEV_ID_X550EM_X_1G_T), board_x550em_x_fw},
{PCI_VDEVICE(INTEL, IXGBE_DEV_ID_X550EM_A_KR), board_x550em_a },
{PCI_VDEVICE(INTEL, IXGBE_DEV_ID_X550EM_A_KR_L), board_x550em_a },
{PCI_VDEVICE(INTEL, IXGBE_DEV_ID_X550EM_A_SFP_N), board_x550em_a },
@@ -508,7 +514,7 @@
*/
static void ixgbe_regdump(struct ixgbe_hw *hw, struct ixgbe_reg_info *reginfo)
{
- int i = 0, j = 0;
+ int i;
char rname[16];
u32 regs[64];
@@ -570,21 +576,38 @@
regs[i] = IXGBE_READ_REG(hw, IXGBE_TXDCTL(i));
break;
default:
- pr_info("%-15s %08x\n", reginfo->name,
- IXGBE_READ_REG(hw, reginfo->ofs));
+ pr_info("%-15s %08x\n",
+ reginfo->name, IXGBE_READ_REG(hw, reginfo->ofs));
return;
}
- for (i = 0; i < 8; i++) {
- snprintf(rname, 16, "%s[%d-%d]", reginfo->name, i*8, i*8+7);
- pr_err("%-15s", rname);
+ i = 0;
+ while (i < 64) {
+ int j;
+ char buf[9 * 8 + 1];
+ char *p = buf;
+
+ snprintf(rname, 16, "%s[%d-%d]", reginfo->name, i, i + 7);
for (j = 0; j < 8; j++)
- pr_cont(" %08x", regs[i*8+j]);
- pr_cont("\n");
+ p += sprintf(p, " %08x", regs[i++]);
+ pr_err("%-15s%s\n", rname, buf);
}
}
+static void ixgbe_print_buffer(struct ixgbe_ring *ring, int n)
+{
+ struct ixgbe_tx_buffer *tx_buffer;
+
+ tx_buffer = &ring->tx_buffer_info[ring->next_to_clean];
+ pr_info(" %5d %5X %5X %016llX %08X %p %016llX\n",
+ n, ring->next_to_use, ring->next_to_clean,
+ (u64)dma_unmap_addr(tx_buffer, dma),
+ dma_unmap_len(tx_buffer, len),
+ tx_buffer->next_to_watch,
+ (u64)tx_buffer->time_stamp);
+}
+
/*
* ixgbe_dump - Print registers, tx-rings and rx-rings
*/
@@ -594,14 +617,13 @@
struct ixgbe_hw *hw = &adapter->hw;
struct ixgbe_reg_info *reginfo;
int n = 0;
- struct ixgbe_ring *tx_ring;
+ struct ixgbe_ring *ring;
struct ixgbe_tx_buffer *tx_buffer;
union ixgbe_adv_tx_desc *tx_desc;
struct my_u0 { u64 a; u64 b; } *u0;
struct ixgbe_ring *rx_ring;
union ixgbe_adv_rx_desc *rx_desc;
struct ixgbe_rx_buffer *rx_buffer_info;
- u32 staterr;
int i = 0;
if (!netif_msg_hw(adapter))
@@ -635,14 +657,13 @@
"Queue [NTU] [NTC] [bi(ntc)->dma ]",
"leng", "ntw", "timestamp");
for (n = 0; n < adapter->num_tx_queues; n++) {
- tx_ring = adapter->tx_ring[n];
- tx_buffer = &tx_ring->tx_buffer_info[tx_ring->next_to_clean];
- pr_info(" %5d %5X %5X %016llX %08X %p %016llX\n",
- n, tx_ring->next_to_use, tx_ring->next_to_clean,
- (u64)dma_unmap_addr(tx_buffer, dma),
- dma_unmap_len(tx_buffer, len),
- tx_buffer->next_to_watch,
- (u64)tx_buffer->time_stamp);
+ ring = adapter->tx_ring[n];
+ ixgbe_print_buffer(ring, n);
+ }
+
+ for (n = 0; n < adapter->num_xdp_queues; n++) {
+ ring = adapter->xdp_ring[n];
+ ixgbe_print_buffer(ring, n);
}
/* Print TX Rings */
@@ -687,21 +708,32 @@
*/
for (n = 0; n < adapter->num_tx_queues; n++) {
- tx_ring = adapter->tx_ring[n];
+ ring = adapter->tx_ring[n];
pr_info("------------------------------------\n");
- pr_info("TX QUEUE INDEX = %d\n", tx_ring->queue_index);
+ pr_info("TX QUEUE INDEX = %d\n", ring->queue_index);
pr_info("------------------------------------\n");
pr_info("%s%s %s %s %s %s\n",
"T [desc] [address 63:0 ] ",
"[PlPOIdStDDt Ln] [bi->dma ] ",
"leng", "ntw", "timestamp", "bi->skb");
- for (i = 0; tx_ring->desc && (i < tx_ring->count); i++) {
- tx_desc = IXGBE_TX_DESC(tx_ring, i);
- tx_buffer = &tx_ring->tx_buffer_info[i];
+ for (i = 0; ring->desc && (i < ring->count); i++) {
+ tx_desc = IXGBE_TX_DESC(ring, i);
+ tx_buffer = &ring->tx_buffer_info[i];
u0 = (struct my_u0 *)tx_desc;
if (dma_unmap_len(tx_buffer, len) > 0) {
- pr_info("T [0x%03X] %016llX %016llX %016llX %08X %p %016llX %p",
+ const char *ring_desc;
+
+ if (i == ring->next_to_use &&
+ i == ring->next_to_clean)
+ ring_desc = " NTC/U";
+ else if (i == ring->next_to_use)
+ ring_desc = " NTU";
+ else if (i == ring->next_to_clean)
+ ring_desc = " NTC";
+ else
+ ring_desc = "";
+ pr_info("T [0x%03X] %016llX %016llX %016llX %08X %p %016llX %p%s",
i,
le64_to_cpu(u0->a),
le64_to_cpu(u0->b),
@@ -709,16 +741,8 @@
dma_unmap_len(tx_buffer, len),
tx_buffer->next_to_watch,
(u64)tx_buffer->time_stamp,
- tx_buffer->skb);
- if (i == tx_ring->next_to_use &&
- i == tx_ring->next_to_clean)
- pr_cont(" NTC/U\n");
- else if (i == tx_ring->next_to_use)
- pr_cont(" NTU\n");
- else if (i == tx_ring->next_to_clean)
- pr_cont(" NTC\n");
- else
- pr_cont("\n");
+ tx_buffer->skb,
+ ring_desc);
if (netif_msg_pktdata(adapter) &&
tx_buffer->skb)
@@ -797,34 +821,44 @@
pr_info("------------------------------------\n");
pr_info("RX QUEUE INDEX = %d\n", rx_ring->queue_index);
pr_info("------------------------------------\n");
- pr_info("%s%s%s",
+ pr_info("%s%s%s\n",
"R [desc] [ PktBuf A0] ",
"[ HeadBuf DD] [bi->dma ] [bi->skb ] ",
- "<-- Adv Rx Read format\n");
- pr_info("%s%s%s",
+ "<-- Adv Rx Read format");
+ pr_info("%s%s%s\n",
"RWB[desc] [PcsmIpSHl PtRs] ",
"[vl er S cks ln] ---------------- [bi->skb ] ",
- "<-- Adv Rx Write-Back format\n");
+ "<-- Adv Rx Write-Back format");
for (i = 0; i < rx_ring->count; i++) {
+ const char *ring_desc;
+
+ if (i == rx_ring->next_to_use)
+ ring_desc = " NTU";
+ else if (i == rx_ring->next_to_clean)
+ ring_desc = " NTC";
+ else
+ ring_desc = "";
+
rx_buffer_info = &rx_ring->rx_buffer_info[i];
rx_desc = IXGBE_RX_DESC(rx_ring, i);
u0 = (struct my_u0 *)rx_desc;
- staterr = le32_to_cpu(rx_desc->wb.upper.status_error);
- if (staterr & IXGBE_RXD_STAT_DD) {
+ if (rx_desc->wb.upper.length) {
/* Descriptor Done */
- pr_info("RWB[0x%03X] %016llX "
- "%016llX ---------------- %p", i,
+ pr_info("RWB[0x%03X] %016llX %016llX ---------------- %p%s\n",
+ i,
le64_to_cpu(u0->a),
le64_to_cpu(u0->b),
- rx_buffer_info->skb);
+ rx_buffer_info->skb,
+ ring_desc);
} else {
- pr_info("R [0x%03X] %016llX "
- "%016llX %016llX %p", i,
+ pr_info("R [0x%03X] %016llX %016llX %016llX %p%s\n",
+ i,
le64_to_cpu(u0->a),
le64_to_cpu(u0->b),
(u64)rx_buffer_info->dma,
- rx_buffer_info->skb);
+ rx_buffer_info->skb,
+ ring_desc);
if (netif_msg_pktdata(adapter) &&
rx_buffer_info->dma) {
@@ -835,14 +869,6 @@
ixgbe_rx_bufsz(rx_ring), true);
}
}
-
- if (i == rx_ring->next_to_use)
- pr_cont(" NTU\n");
- else if (i == rx_ring->next_to_clean)
- pr_cont(" NTC\n");
- else
- pr_cont("\n");
-
}
}
}
@@ -972,6 +998,10 @@
for (i = 0; i < adapter->num_tx_queues; i++)
clear_bit(__IXGBE_HANG_CHECK_ARMED,
&adapter->tx_ring[i]->state);
+
+ for (i = 0; i < adapter->num_xdp_queues; i++)
+ clear_bit(__IXGBE_HANG_CHECK_ARMED,
+ &adapter->xdp_ring[i]->state);
}
static void ixgbe_update_xoff_received(struct ixgbe_adapter *adapter)
@@ -1016,6 +1046,14 @@
if (xoff[tc])
clear_bit(__IXGBE_HANG_CHECK_ARMED, &tx_ring->state);
}
+
+ for (i = 0; i < adapter->num_xdp_queues; i++) {
+ struct ixgbe_ring *xdp_ring = adapter->xdp_ring[i];
+
+ tc = xdp_ring->dcb_tc;
+ if (xoff[tc])
+ clear_bit(__IXGBE_HANG_CHECK_ARMED, &xdp_ring->state);
+ }
}
static u64 ixgbe_get_tx_completed(struct ixgbe_ring *ring)
@@ -1167,7 +1205,10 @@
total_packets += tx_buffer->gso_segs;
/* free the skb */
- napi_consume_skb(tx_buffer->skb, napi_budget);
+ if (ring_is_xdp(tx_ring))
+ page_frag_free(tx_buffer->data);
+ else
+ napi_consume_skb(tx_buffer->skb, napi_budget);
/* unmap skb header data */
dma_unmap_single(tx_ring->dev,
@@ -1228,7 +1269,7 @@
if (check_for_tx_hang(tx_ring) && ixgbe_check_tx_hang(tx_ring)) {
/* schedule immediate reset if we believe we hung */
struct ixgbe_hw *hw = &adapter->hw;
- e_err(drv, "Detected Tx Unit Hang\n"
+ e_err(drv, "Detected Tx Unit Hang %s\n"
" Tx Queue <%d>\n"
" TDH, TDT <%x>, <%x>\n"
" next_to_use <%x>\n"
@@ -1236,13 +1277,16 @@
"tx_buffer_info[next_to_clean]\n"
" time_stamp <%lx>\n"
" jiffies <%lx>\n",
+ ring_is_xdp(tx_ring) ? "(XDP)" : "",
tx_ring->queue_index,
IXGBE_READ_REG(hw, IXGBE_TDH(tx_ring->reg_idx)),
IXGBE_READ_REG(hw, IXGBE_TDT(tx_ring->reg_idx)),
tx_ring->next_to_use, i,
tx_ring->tx_buffer_info[i].time_stamp, jiffies);
- netif_stop_subqueue(tx_ring->netdev, tx_ring->queue_index);
+ if (!ring_is_xdp(tx_ring))
+ netif_stop_subqueue(tx_ring->netdev,
+ tx_ring->queue_index);
e_info(probe,
"tx hang %d detected on queue %d, resetting adapter\n",
@@ -1255,6 +1299,9 @@
return true;
}
+ if (ring_is_xdp(tx_ring))
+ return !!budget;
+
netdev_tx_completed_queue(txring_txq(tx_ring),
total_packets, total_bytes);
@@ -1846,6 +1893,10 @@
* @rx_desc: pointer to the EOP Rx descriptor
* @skb: pointer to current skb being fixed
*
+ * Check if the skb is valid in the XDP case it will be an error pointer.
+ * Return true in this case to abort processing and advance to next
+ * descriptor.
+ *
* Check for corrupted packet headers caused by senders on the local L2
* embedded NIC switch not setting up their Tx Descriptors right. These
* should be very rare.
@@ -1864,6 +1915,10 @@
{
struct net_device *netdev = rx_ring->netdev;
+ /* XDP packets use error pointer so abort at this point */
+ if (IS_ERR(skb))
+ return true;
+
/* verify that the packet does not have any known errors */
if (unlikely(ixgbe_test_staterr(rx_desc,
IXGBE_RXDADV_ERR_FRAME_ERR_MASK) &&
@@ -2039,7 +2094,7 @@
/* hand second half of page back to the ring */
ixgbe_reuse_rx_page(rx_ring, rx_buffer);
} else {
- if (IXGBE_CB(skb)->dma == rx_buffer->dma) {
+ if (!IS_ERR(skb) && IXGBE_CB(skb)->dma == rx_buffer->dma) {
/* the page has been released from the ring */
IXGBE_CB(skb)->page_released = true;
} else {
@@ -2060,21 +2115,22 @@
static struct sk_buff *ixgbe_construct_skb(struct ixgbe_ring *rx_ring,
struct ixgbe_rx_buffer *rx_buffer,
- union ixgbe_adv_rx_desc *rx_desc,
- unsigned int size)
+ struct xdp_buff *xdp,
+ union ixgbe_adv_rx_desc *rx_desc)
{
- void *va = page_address(rx_buffer->page) + rx_buffer->page_offset;
+ unsigned int size = xdp->data_end - xdp->data;
#if (PAGE_SIZE < 8192)
unsigned int truesize = ixgbe_rx_pg_size(rx_ring) / 2;
#else
- unsigned int truesize = SKB_DATA_ALIGN(size);
+ unsigned int truesize = SKB_DATA_ALIGN(xdp->data_end -
+ xdp->data_hard_start);
#endif
struct sk_buff *skb;
/* prefetch first cache line of first page */
- prefetch(va);
+ prefetch(xdp->data);
#if L1_CACHE_BYTES < 128
- prefetch(va + L1_CACHE_BYTES);
+ prefetch(xdp->data + L1_CACHE_BYTES);
#endif
/* allocate a skb to store the frags */
@@ -2087,7 +2143,7 @@
IXGBE_CB(skb)->dma = rx_buffer->dma;
skb_add_rx_frag(skb, 0, rx_buffer->page,
- rx_buffer->page_offset,
+ xdp->data - page_address(rx_buffer->page),
size, truesize);
#if (PAGE_SIZE < 8192)
rx_buffer->page_offset ^= truesize;
@@ -2095,7 +2151,8 @@
rx_buffer->page_offset += truesize;
#endif
} else {
- memcpy(__skb_put(skb, size), va, ALIGN(size, sizeof(long)));
+ memcpy(__skb_put(skb, size),
+ xdp->data, ALIGN(size, sizeof(long)));
rx_buffer->pagecnt_bias++;
}
@@ -2104,32 +2161,32 @@
static struct sk_buff *ixgbe_build_skb(struct ixgbe_ring *rx_ring,
struct ixgbe_rx_buffer *rx_buffer,
- union ixgbe_adv_rx_desc *rx_desc,
- unsigned int size)
+ struct xdp_buff *xdp,
+ union ixgbe_adv_rx_desc *rx_desc)
{
- void *va = page_address(rx_buffer->page) + rx_buffer->page_offset;
#if (PAGE_SIZE < 8192)
unsigned int truesize = ixgbe_rx_pg_size(rx_ring) / 2;
#else
unsigned int truesize = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) +
- SKB_DATA_ALIGN(IXGBE_SKB_PAD + size);
+ SKB_DATA_ALIGN(xdp->data_end -
+ xdp->data_hard_start);
#endif
struct sk_buff *skb;
/* prefetch first cache line of first page */
- prefetch(va);
+ prefetch(xdp->data);
#if L1_CACHE_BYTES < 128
- prefetch(va + L1_CACHE_BYTES);
+ prefetch(xdp->data + L1_CACHE_BYTES);
#endif
/* build an skb to around the page buffer */
- skb = build_skb(va - IXGBE_SKB_PAD, truesize);
+ skb = build_skb(xdp->data_hard_start, truesize);
if (unlikely(!skb))
return NULL;
/* update pointers within the skb to store the data */
- skb_reserve(skb, IXGBE_SKB_PAD);
- __skb_put(skb, size);
+ skb_reserve(skb, xdp->data - xdp->data_hard_start);
+ __skb_put(skb, xdp->data_end - xdp->data);
/* record DMA address if this is the start of a chain of buffers */
if (!ixgbe_test_staterr(rx_desc, IXGBE_RXD_STAT_EOP))
@@ -2145,6 +2202,65 @@
return skb;
}
+#define IXGBE_XDP_PASS 0
+#define IXGBE_XDP_CONSUMED 1
+#define IXGBE_XDP_TX 2
+
+static int ixgbe_xmit_xdp_ring(struct ixgbe_adapter *adapter,
+ struct xdp_buff *xdp);
+
+static struct sk_buff *ixgbe_run_xdp(struct ixgbe_adapter *adapter,
+ struct ixgbe_ring *rx_ring,
+ struct xdp_buff *xdp)
+{
+ int result = IXGBE_XDP_PASS;
+ struct bpf_prog *xdp_prog;
+ u32 act;
+
+ rcu_read_lock();
+ xdp_prog = READ_ONCE(rx_ring->xdp_prog);
+
+ if (!xdp_prog)
+ goto xdp_out;
+
+ act = bpf_prog_run_xdp(xdp_prog, xdp);
+ switch (act) {
+ case XDP_PASS:
+ break;
+ case XDP_TX:
+ result = ixgbe_xmit_xdp_ring(adapter, xdp);
+ break;
+ default:
+ bpf_warn_invalid_xdp_action(act);
+ case XDP_ABORTED:
+ trace_xdp_exception(rx_ring->netdev, xdp_prog, act);
+ /* fallthrough -- handle aborts by dropping packet */
+ case XDP_DROP:
+ result = IXGBE_XDP_CONSUMED;
+ break;
+ }
+xdp_out:
+ rcu_read_unlock();
+ return ERR_PTR(-result);
+}
+
+static void ixgbe_rx_buffer_flip(struct ixgbe_ring *rx_ring,
+ struct ixgbe_rx_buffer *rx_buffer,
+ unsigned int size)
+{
+#if (PAGE_SIZE < 8192)
+ unsigned int truesize = ixgbe_rx_pg_size(rx_ring) / 2;
+
+ rx_buffer->page_offset ^= truesize;
+#else
+ unsigned int truesize = ring_uses_build_skb(rx_ring) ?
+ SKB_DATA_ALIGN(IXGBE_SKB_PAD + size) :
+ SKB_DATA_ALIGN(size);
+
+ rx_buffer->page_offset += truesize;
+#endif
+}
+
/**
* ixgbe_clean_rx_irq - Clean completed descriptors from Rx ring - bounce buf
* @q_vector: structure containing interrupt and ring information
@@ -2163,17 +2279,19 @@
const int budget)
{
unsigned int total_rx_bytes = 0, total_rx_packets = 0;
-#ifdef IXGBE_FCOE
struct ixgbe_adapter *adapter = q_vector->adapter;
+#ifdef IXGBE_FCOE
int ddp_bytes;
unsigned int mss = 0;
#endif /* IXGBE_FCOE */
u16 cleaned_count = ixgbe_desc_unused(rx_ring);
+ bool xdp_xmit = false;
while (likely(total_rx_packets < budget)) {
union ixgbe_adv_rx_desc *rx_desc;
struct ixgbe_rx_buffer *rx_buffer;
struct sk_buff *skb;
+ struct xdp_buff xdp;
unsigned int size;
/* return some buffers to hardware, one at a time is too slow */
@@ -2196,14 +2314,34 @@
rx_buffer = ixgbe_get_rx_buffer(rx_ring, rx_desc, &skb, size);
/* retrieve a buffer from the ring */
- if (skb)
+ if (!skb) {
+ xdp.data = page_address(rx_buffer->page) +
+ rx_buffer->page_offset;
+ xdp.data_hard_start = xdp.data -
+ ixgbe_rx_offset(rx_ring);
+ xdp.data_end = xdp.data + size;
+
+ skb = ixgbe_run_xdp(adapter, rx_ring, &xdp);
+ }
+
+ if (IS_ERR(skb)) {
+ if (PTR_ERR(skb) == -IXGBE_XDP_TX) {
+ xdp_xmit = true;
+ ixgbe_rx_buffer_flip(rx_ring, rx_buffer, size);
+ } else {
+ rx_buffer->pagecnt_bias++;
+ }
+ total_rx_packets++;
+ total_rx_bytes += size;
+ } else if (skb) {
ixgbe_add_rx_frag(rx_ring, rx_buffer, skb, size);
- else if (ring_uses_build_skb(rx_ring))
+ } else if (ring_uses_build_skb(rx_ring)) {
skb = ixgbe_build_skb(rx_ring, rx_buffer,
- rx_desc, size);
- else
+ &xdp, rx_desc);
+ } else {
skb = ixgbe_construct_skb(rx_ring, rx_buffer,
- rx_desc, size);
+ &xdp, rx_desc);
+ }
/* exit if we failed to retrieve a buffer */
if (!skb) {
@@ -2260,6 +2398,16 @@
total_rx_packets++;
}
+ if (xdp_xmit) {
+ struct ixgbe_ring *ring = adapter->xdp_ring[smp_processor_id()];
+
+ /* Force memory writes to complete before letting h/w
+ * know there are new descriptors to fetch.
+ */
+ wmb();
+ writel(ring->next_to_use, ring->tail);
+ }
+
u64_stats_update_begin(&rx_ring->syncp);
rx_ring->stats.packets += total_rx_packets;
rx_ring->stats.bytes += total_rx_bytes;
@@ -3364,6 +3512,8 @@
/* Setup the HW Tx Head and Tail descriptor pointers */
for (i = 0; i < adapter->num_tx_queues; i++)
ixgbe_configure_tx_ring(adapter, adapter->tx_ring[i]);
+ for (i = 0; i < adapter->num_xdp_queues; i++)
+ ixgbe_configure_tx_ring(adapter, adapter->xdp_ring[i]);
}
static void ixgbe_enable_rx_drop(struct ixgbe_adapter *adapter,
@@ -3489,6 +3639,28 @@
}
/**
+ * ixgbe_init_rss_key - Initialize adapter RSS key
+ * @adapter: device handle
+ *
+ * Allocates and initializes the RSS key if it is not allocated.
+ **/
+static inline int ixgbe_init_rss_key(struct ixgbe_adapter *adapter)
+{
+ u32 *rss_key;
+
+ if (!adapter->rss_key) {
+ rss_key = kzalloc(IXGBE_RSS_KEY_SIZE, GFP_KERNEL);
+ if (unlikely(!rss_key))
+ return -ENOMEM;
+
+ netdev_rss_key_fill(rss_key, IXGBE_RSS_KEY_SIZE);
+ adapter->rss_key = rss_key;
+ }
+
+ return 0;
+}
+
+/**
* ixgbe_store_reta - Write the RETA table to HW
* @adapter: device handle
*
@@ -3590,7 +3762,7 @@
/* Fill out hash function seeds */
for (i = 0; i < 10; i++)
IXGBE_WRITE_REG(hw, IXGBE_PFVFRSSRK(i, pf_pool),
- adapter->rss_key[i]);
+ *(adapter->rss_key + i));
/* Fill out the redirection table */
for (i = 0, j = 0; i < 64; i++, j++) {
@@ -3651,7 +3823,6 @@
if (adapter->flags2 & IXGBE_FLAG2_RSS_FIELD_IPV6_UDP)
rss_field |= IXGBE_MRQC_RSS_FIELD_IPV6_UDP;
- netdev_rss_key_fill(adapter->rss_key, sizeof(adapter->rss_key));
if ((hw->mac.type >= ixgbe_mac_X550) &&
(adapter->flags & IXGBE_FLAG_SRIOV_ENABLED)) {
unsigned int pf_pool = adapter->num_vfs;
@@ -3802,7 +3973,7 @@
/* Limit the maximum frame size so we don't overrun the skb */
if (ring_uses_build_skb(ring) &&
!test_bit(__IXGBE_RX_3K_BUFFER, &ring->state))
- rxdctl |= IXGBE_MAX_FRAME_BUILD_SKB |
+ rxdctl |= IXGBE_MAX_2K_FRAME_BUILD_SKB |
IXGBE_RXDCTL_RLPML_EN;
#endif
}
@@ -3972,8 +4143,8 @@
if (adapter->flags2 & IXGBE_FLAG2_RSC_ENABLED)
set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state);
- if ((max_frame > (ETH_FRAME_LEN + ETH_FCS_LEN)) ||
- (max_frame > IXGBE_MAX_FRAME_BUILD_SKB))
+ if (IXGBE_2K_TOO_SMALL_WITH_PADDING ||
+ (max_frame > (ETH_FRAME_LEN + ETH_FCS_LEN)))
set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state);
#endif
}
@@ -5505,7 +5676,10 @@
union ixgbe_adv_tx_desc *eop_desc, *tx_desc;
/* Free all the Tx ring sk_buffs */
- dev_kfree_skb_any(tx_buffer->skb);
+ if (ring_is_xdp(tx_ring))
+ page_frag_free(tx_buffer->data);
+ else
+ dev_kfree_skb_any(tx_buffer->skb);
/* unmap skb header data */
dma_unmap_single(tx_ring->dev,
@@ -5546,7 +5720,8 @@
}
/* reset BQL for queue */
- netdev_tx_reset_queue(txring_txq(tx_ring));
+ if (!ring_is_xdp(tx_ring))
+ netdev_tx_reset_queue(txring_txq(tx_ring));
/* reset next_to_use and next_to_clean */
tx_ring->next_to_use = 0;
@@ -5575,6 +5750,8 @@
for (i = 0; i < adapter->num_tx_queues; i++)
ixgbe_clean_tx_ring(adapter->tx_ring[i]);
+ for (i = 0; i < adapter->num_xdp_queues; i++)
+ ixgbe_clean_tx_ring(adapter->xdp_ring[i]);
}
static void ixgbe_fdir_filter_exit(struct ixgbe_adapter *adapter)
@@ -5669,6 +5846,11 @@
u8 reg_idx = adapter->tx_ring[i]->reg_idx;
IXGBE_WRITE_REG(hw, IXGBE_TXDCTL(reg_idx), IXGBE_TXDCTL_SWFLSH);
}
+ for (i = 0; i < adapter->num_xdp_queues; i++) {
+ u8 reg_idx = adapter->xdp_ring[i]->reg_idx;
+
+ IXGBE_WRITE_REG(hw, IXGBE_TXDCTL(reg_idx), IXGBE_TXDCTL_SWFLSH);
+ }
/* Disable the Tx DMA engine on 82599 and later MAC */
switch (hw->mac.type) {
@@ -5854,6 +6036,9 @@
if (!adapter->mac_table)
return -ENOMEM;
+ if (ixgbe_init_rss_key(adapter))
+ return -ENOMEM;
+
/* Set MAC specific capability flags and exceptions */
switch (hw->mac.type) {
case ixgbe_mac_82598EB:
@@ -5944,10 +6129,8 @@
/* assign number of SR-IOV VFs */
if (hw->mac.type != ixgbe_mac_82598EB) {
if (max_vfs > IXGBE_MAX_VFS_DRV_LIMIT) {
- adapter->num_vfs = 0;
+ max_vfs = 0;
e_dev_warn("max_vfs parameter out of range. Not assigning any SR-IOV VFs\n");
- } else {
- adapter->num_vfs = max_vfs;
}
}
#endif /* CONFIG_PCI_IOV */
@@ -6041,7 +6224,7 @@
**/
static int ixgbe_setup_all_tx_resources(struct ixgbe_adapter *adapter)
{
- int i, err = 0;
+ int i, j = 0, err = 0;
for (i = 0; i < adapter->num_tx_queues; i++) {
err = ixgbe_setup_tx_resources(adapter->tx_ring[i]);
@@ -6051,10 +6234,20 @@
e_err(probe, "Allocation for Tx Queue %u failed\n", i);
goto err_setup_tx;
}
+ for (j = 0; j < adapter->num_xdp_queues; j++) {
+ err = ixgbe_setup_tx_resources(adapter->xdp_ring[j]);
+ if (!err)
+ continue;
+
+ e_err(probe, "Allocation for Tx Queue %u failed\n", j);
+ goto err_setup_tx;
+ }
return 0;
err_setup_tx:
/* rewind the index freeing the rings as we go */
+ while (j--)
+ ixgbe_free_tx_resources(adapter->xdp_ring[j]);
while (i--)
ixgbe_free_tx_resources(adapter->tx_ring[i]);
return err;
@@ -6066,7 +6259,8 @@
*
* Returns 0 on success, negative on failure
**/
-int ixgbe_setup_rx_resources(struct ixgbe_ring *rx_ring)
+int ixgbe_setup_rx_resources(struct ixgbe_adapter *adapter,
+ struct ixgbe_ring *rx_ring)
{
struct device *dev = rx_ring->dev;
int orig_node = dev_to_node(dev);
@@ -6105,6 +6299,8 @@
rx_ring->next_to_clean = 0;
rx_ring->next_to_use = 0;
+ rx_ring->xdp_prog = adapter->xdp_prog;
+
return 0;
err:
vfree(rx_ring->rx_buffer_info);
@@ -6128,7 +6324,7 @@
int i, err = 0;
for (i = 0; i < adapter->num_rx_queues; i++) {
- err = ixgbe_setup_rx_resources(adapter->rx_ring[i]);
+ err = ixgbe_setup_rx_resources(adapter, adapter->rx_ring[i]);
if (!err)
continue;
@@ -6184,6 +6380,9 @@
for (i = 0; i < adapter->num_tx_queues; i++)
if (adapter->tx_ring[i]->desc)
ixgbe_free_tx_resources(adapter->tx_ring[i]);
+ for (i = 0; i < adapter->num_xdp_queues; i++)
+ if (adapter->xdp_ring[i]->desc)
+ ixgbe_free_tx_resources(adapter->xdp_ring[i]);
}
/**
@@ -6196,6 +6395,7 @@
{
ixgbe_clean_rx_ring(rx_ring);
+ rx_ring->xdp_prog = NULL;
vfree(rx_ring->rx_buffer_info);
rx_ring->rx_buffer_info = NULL;
@@ -6602,6 +6802,14 @@
bytes += tx_ring->stats.bytes;
packets += tx_ring->stats.packets;
}
+ for (i = 0; i < adapter->num_xdp_queues; i++) {
+ struct ixgbe_ring *xdp_ring = adapter->xdp_ring[i];
+
+ restart_queue += xdp_ring->tx_stats.restart_queue;
+ tx_busy += xdp_ring->tx_stats.tx_busy;
+ bytes += xdp_ring->stats.bytes;
+ packets += xdp_ring->stats.packets;
+ }
adapter->restart_queue = restart_queue;
adapter->tx_busy = tx_busy;
netdev->stats.tx_bytes = bytes;
@@ -6795,6 +7003,9 @@
for (i = 0; i < adapter->num_tx_queues; i++)
set_bit(__IXGBE_TX_FDIR_INIT_DONE,
&(adapter->tx_ring[i]->state));
+ for (i = 0; i < adapter->num_xdp_queues; i++)
+ set_bit(__IXGBE_TX_FDIR_INIT_DONE,
+ &adapter->xdp_ring[i]->state);
/* re-enable flow director interrupts */
IXGBE_WRITE_REG(hw, IXGBE_EIMS, IXGBE_EIMS_FLOW_DIR);
} else {
@@ -6828,6 +7039,8 @@
if (netif_carrier_ok(adapter->netdev)) {
for (i = 0; i < adapter->num_tx_queues; i++)
set_check_for_tx_hang(adapter->tx_ring[i]);
+ for (i = 0; i < adapter->num_xdp_queues; i++)
+ set_check_for_tx_hang(adapter->xdp_ring[i]);
}
if (!(adapter->flags & IXGBE_FLAG_MSIX_ENABLED)) {
@@ -7058,6 +7271,13 @@
return true;
}
+ for (i = 0; i < adapter->num_xdp_queues; i++) {
+ struct ixgbe_ring *ring = adapter->xdp_ring[i];
+
+ if (ring->next_to_use != ring->next_to_clean)
+ return true;
+ }
+
return false;
}
@@ -8015,6 +8235,62 @@
#endif
}
+static int ixgbe_xmit_xdp_ring(struct ixgbe_adapter *adapter,
+ struct xdp_buff *xdp)
+{
+ struct ixgbe_ring *ring = adapter->xdp_ring[smp_processor_id()];
+ struct ixgbe_tx_buffer *tx_buffer;
+ union ixgbe_adv_tx_desc *tx_desc;
+ u32 len, cmd_type;
+ dma_addr_t dma;
+ u16 i;
+
+ len = xdp->data_end - xdp->data;
+
+ if (unlikely(!ixgbe_desc_unused(ring)))
+ return IXGBE_XDP_CONSUMED;
+
+ dma = dma_map_single(ring->dev, xdp->data, len, DMA_TO_DEVICE);
+ if (dma_mapping_error(ring->dev, dma))
+ return IXGBE_XDP_CONSUMED;
+
+ /* record the location of the first descriptor for this packet */
+ tx_buffer = &ring->tx_buffer_info[ring->next_to_use];
+ tx_buffer->bytecount = len;
+ tx_buffer->gso_segs = 1;
+ tx_buffer->protocol = 0;
+
+ i = ring->next_to_use;
+ tx_desc = IXGBE_TX_DESC(ring, i);
+
+ dma_unmap_len_set(tx_buffer, len, len);
+ dma_unmap_addr_set(tx_buffer, dma, dma);
+ tx_buffer->data = xdp->data;
+ tx_desc->read.buffer_addr = cpu_to_le64(dma);
+
+ /* put descriptor type bits */
+ cmd_type = IXGBE_ADVTXD_DTYP_DATA |
+ IXGBE_ADVTXD_DCMD_DEXT |
+ IXGBE_ADVTXD_DCMD_IFCS;
+ cmd_type |= len | IXGBE_TXD_CMD;
+ tx_desc->read.cmd_type_len = cpu_to_le32(cmd_type);
+ tx_desc->read.olinfo_status =
+ cpu_to_le32(len << IXGBE_ADVTXD_PAYLEN_SHIFT);
+
+ /* Avoid any potential race with xdp_xmit and cleanup */
+ smp_wmb();
+
+ /* set next_to_watch value indicating a packet is present */
+ i++;
+ if (i == ring->count)
+ i = 0;
+
+ tx_buffer->next_to_watch = tx_desc;
+ ring->next_to_use = i;
+
+ return IXGBE_XDP_TX;
+}
+
netdev_tx_t ixgbe_xmit_frame_ring(struct sk_buff *skb,
struct ixgbe_adapter *adapter,
struct ixgbe_ring *tx_ring)
@@ -8306,6 +8582,23 @@
#endif
+static void ixgbe_get_ring_stats64(struct rtnl_link_stats64 *stats,
+ struct ixgbe_ring *ring)
+{
+ u64 bytes, packets;
+ unsigned int start;
+
+ if (ring) {
+ do {
+ start = u64_stats_fetch_begin_irq(&ring->syncp);
+ packets = ring->stats.packets;
+ bytes = ring->stats.bytes;
+ } while (u64_stats_fetch_retry_irq(&ring->syncp, start));
+ stats->tx_packets += packets;
+ stats->tx_bytes += bytes;
+ }
+}
+
static void ixgbe_get_stats64(struct net_device *netdev,
struct rtnl_link_stats64 *stats)
{
@@ -8331,18 +8624,13 @@
for (i = 0; i < adapter->num_tx_queues; i++) {
struct ixgbe_ring *ring = ACCESS_ONCE(adapter->tx_ring[i]);
- u64 bytes, packets;
- unsigned int start;
- if (ring) {
- do {
- start = u64_stats_fetch_begin_irq(&ring->syncp);
- packets = ring->stats.packets;
- bytes = ring->stats.bytes;
- } while (u64_stats_fetch_retry_irq(&ring->syncp, start));
- stats->tx_packets += packets;
- stats->tx_bytes += bytes;
- }
+ ixgbe_get_ring_stats64(stats, ring);
+ }
+ for (i = 0; i < adapter->num_xdp_queues; i++) {
+ struct ixgbe_ring *ring = ACCESS_ONCE(adapter->xdp_ring[i]);
+
+ ixgbe_get_ring_stats64(stats, ring);
}
rcu_read_unlock();
@@ -8948,7 +9236,9 @@
if (tc->type != TC_SETUP_MQPRIO)
return -EINVAL;
- return ixgbe_setup_tc(dev, tc->tc);
+ tc->mqprio->hw = TC_MQPRIO_HW_OFFLOAD_TCS;
+
+ return ixgbe_setup_tc(dev, tc->mqprio->num_tc);
}
#ifdef CONFIG_PCI_IOV
@@ -9459,6 +9749,68 @@
return features;
}
+static int ixgbe_xdp_setup(struct net_device *dev, struct bpf_prog *prog)
+{
+ int i, frame_size = dev->mtu + ETH_HLEN + ETH_FCS_LEN + VLAN_HLEN;
+ struct ixgbe_adapter *adapter = netdev_priv(dev);
+ struct bpf_prog *old_prog;
+
+ if (adapter->flags & IXGBE_FLAG_SRIOV_ENABLED)
+ return -EINVAL;
+
+ if (adapter->flags & IXGBE_FLAG_DCB_ENABLED)
+ return -EINVAL;
+
+ /* verify ixgbe ring attributes are sufficient for XDP */
+ for (i = 0; i < adapter->num_rx_queues; i++) {
+ struct ixgbe_ring *ring = adapter->rx_ring[i];
+
+ if (ring_is_rsc_enabled(ring))
+ return -EINVAL;
+
+ if (frame_size > ixgbe_rx_bufsz(ring))
+ return -EINVAL;
+ }
+
+ if (nr_cpu_ids > MAX_XDP_QUEUES)
+ return -ENOMEM;
+
+ old_prog = xchg(&adapter->xdp_prog, prog);
+
+ /* If transitioning XDP modes reconfigure rings */
+ if (!!prog != !!old_prog) {
+ int err = ixgbe_setup_tc(dev, netdev_get_num_tc(dev));
+
+ if (err) {
+ rcu_assign_pointer(adapter->xdp_prog, old_prog);
+ return -EINVAL;
+ }
+ } else {
+ for (i = 0; i < adapter->num_rx_queues; i++)
+ xchg(&adapter->rx_ring[i]->xdp_prog, adapter->xdp_prog);
+ }
+
+ if (old_prog)
+ bpf_prog_put(old_prog);
+
+ return 0;
+}
+
+static int ixgbe_xdp(struct net_device *dev, struct netdev_xdp *xdp)
+{
+ struct ixgbe_adapter *adapter = netdev_priv(dev);
+
+ switch (xdp->command) {
+ case XDP_SETUP_PROG:
+ return ixgbe_xdp_setup(dev, xdp->prog);
+ case XDP_QUERY_PROG:
+ xdp->prog_attached = !!(adapter->xdp_prog);
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
static const struct net_device_ops ixgbe_netdev_ops = {
.ndo_open = ixgbe_open,
.ndo_stop = ixgbe_close,
@@ -9504,6 +9856,7 @@
.ndo_udp_tunnel_add = ixgbe_add_udp_tunnel_port,
.ndo_udp_tunnel_del = ixgbe_del_udp_tunnel_port,
.ndo_features_check = ixgbe_features_check,
+ .ndo_xdp = ixgbe_xdp,
};
/**
@@ -9808,7 +10161,7 @@
ixgbe_init_mbx_params_pf(hw);
hw->mbx.ops = ii->mbx_ops;
pci_sriov_set_totalvfs(pdev, IXGBE_MAX_VFS_DRV_LIMIT);
- ixgbe_enable_sriov(adapter);
+ ixgbe_enable_sriov(adapter, max_vfs);
skip_sriov:
#endif
@@ -9934,6 +10287,9 @@
if (err)
goto err_sw_init;
+ for (i = 0; i < adapter->num_xdp_queues; i++)
+ u64_stats_init(&adapter->xdp_ring[i]->syncp);
+
/* WOL not supported for all devices */
adapter->wol = 0;
hw->eeprom.ops.read(hw, 0x2c, &adapter->eeprom_cap);
@@ -10059,6 +10415,7 @@
iounmap(adapter->io_addr);
kfree(adapter->jump_tables[0]);
kfree(adapter->mac_table);
+ kfree(adapter->rss_key);
err_ioremap:
disable_dev = !test_and_set_bit(__IXGBE_DISABLED, &adapter->state);
free_netdev(netdev);
@@ -10143,6 +10500,7 @@
}
kfree(adapter->mac_table);
+ kfree(adapter->rss_key);
disable_dev = !test_and_set_bit(__IXGBE_DISABLED, &adapter->state);
free_netdev(netdev);
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.c
index e55b260..654a402 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_phy.c
@@ -792,7 +792,8 @@
hw->phy.autoneg_advertised |= IXGBE_LINK_SPEED_10_FULL;
/* Setup link based on the new speed settings */
- hw->phy.ops.setup_link(hw);
+ if (hw->phy.ops.setup_link)
+ hw->phy.ops.setup_link(hw);
return 0;
}
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c
index 044cb44..8baf298 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c
@@ -46,88 +46,96 @@
#include "ixgbe_sriov.h"
#ifdef CONFIG_PCI_IOV
-static int __ixgbe_enable_sriov(struct ixgbe_adapter *adapter)
+static inline void ixgbe_alloc_vf_macvlans(struct ixgbe_adapter *adapter,
+ unsigned int num_vfs)
{
struct ixgbe_hw *hw = &adapter->hw;
- int num_vf_macvlans, i;
struct vf_macvlans *mv_list;
+ int num_vf_macvlans, i;
+
+ num_vf_macvlans = hw->mac.num_rar_entries -
+ (IXGBE_MAX_PF_MACVLANS + 1 + num_vfs);
+ if (!num_vf_macvlans)
+ return;
+
+ mv_list = kcalloc(num_vf_macvlans, sizeof(struct vf_macvlans),
+ GFP_KERNEL);
+ if (mv_list) {
+ /* Initialize list of VF macvlans */
+ INIT_LIST_HEAD(&adapter->vf_mvs.l);
+ for (i = 0; i < num_vf_macvlans; i++) {
+ mv_list[i].vf = -1;
+ mv_list[i].free = true;
+ list_add(&mv_list[i].l, &adapter->vf_mvs.l);
+ }
+ adapter->mv_list = mv_list;
+ }
+}
+
+static int __ixgbe_enable_sriov(struct ixgbe_adapter *adapter,
+ unsigned int num_vfs)
+{
+ struct ixgbe_hw *hw = &adapter->hw;
+ int i;
adapter->flags |= IXGBE_FLAG_SRIOV_ENABLED;
- e_info(probe, "SR-IOV enabled with %d VFs\n", adapter->num_vfs);
/* Enable VMDq flag so device will be set in VM mode */
adapter->flags |= IXGBE_FLAG_VMDQ_ENABLED;
if (!adapter->ring_feature[RING_F_VMDQ].limit)
adapter->ring_feature[RING_F_VMDQ].limit = 1;
- adapter->ring_feature[RING_F_VMDQ].offset = adapter->num_vfs;
- num_vf_macvlans = hw->mac.num_rar_entries -
- (IXGBE_MAX_PF_MACVLANS + 1 + adapter->num_vfs);
+ /* Allocate memory for per VF control structures */
+ adapter->vfinfo = kcalloc(num_vfs, sizeof(struct vf_data_storage),
+ GFP_KERNEL);
+ if (!adapter->vfinfo)
+ return -ENOMEM;
- adapter->mv_list = mv_list = kcalloc(num_vf_macvlans,
- sizeof(struct vf_macvlans),
- GFP_KERNEL);
- if (mv_list) {
- /* Initialize list of VF macvlans */
- INIT_LIST_HEAD(&adapter->vf_mvs.l);
- for (i = 0; i < num_vf_macvlans; i++) {
- mv_list->vf = -1;
- mv_list->free = true;
- list_add(&mv_list->l, &adapter->vf_mvs.l);
- mv_list++;
- }
- }
+ adapter->num_vfs = num_vfs;
+
+ ixgbe_alloc_vf_macvlans(adapter, num_vfs);
+ adapter->ring_feature[RING_F_VMDQ].offset = num_vfs;
/* Initialize default switching mode VEB */
IXGBE_WRITE_REG(hw, IXGBE_PFDTXGSWC, IXGBE_PFDTXGSWC_VT_LBEN);
adapter->bridge_mode = BRIDGE_MODE_VEB;
- /* If call to enable VFs succeeded then allocate memory
- * for per VF control structures.
- */
- adapter->vfinfo =
- kcalloc(adapter->num_vfs,
- sizeof(struct vf_data_storage), GFP_KERNEL);
- if (adapter->vfinfo) {
- /* limit trafffic classes based on VFs enabled */
- if ((adapter->hw.mac.type == ixgbe_mac_82599EB) &&
- (adapter->num_vfs < 16)) {
- adapter->dcb_cfg.num_tcs.pg_tcs = MAX_TRAFFIC_CLASS;
- adapter->dcb_cfg.num_tcs.pfc_tcs = MAX_TRAFFIC_CLASS;
- } else if (adapter->num_vfs < 32) {
- adapter->dcb_cfg.num_tcs.pg_tcs = 4;
- adapter->dcb_cfg.num_tcs.pfc_tcs = 4;
- } else {
- adapter->dcb_cfg.num_tcs.pg_tcs = 1;
- adapter->dcb_cfg.num_tcs.pfc_tcs = 1;
- }
-
- /* Disable RSC when in SR-IOV mode */
- adapter->flags2 &= ~(IXGBE_FLAG2_RSC_CAPABLE |
- IXGBE_FLAG2_RSC_ENABLED);
-
- for (i = 0; i < adapter->num_vfs; i++) {
- /* enable spoof checking for all VFs */
- adapter->vfinfo[i].spoofchk_enabled = true;
-
- /* We support VF RSS querying only for 82599 and x540
- * devices at the moment. These devices share RSS
- * indirection table and RSS hash key with PF therefore
- * we want to disable the querying by default.
- */
- adapter->vfinfo[i].rss_query_enabled = 0;
-
- /* Untrust all VFs */
- adapter->vfinfo[i].trusted = false;
-
- /* set the default xcast mode */
- adapter->vfinfo[i].xcast_mode = IXGBEVF_XCAST_MODE_NONE;
- }
-
- return 0;
+ /* limit trafffic classes based on VFs enabled */
+ if ((adapter->hw.mac.type == ixgbe_mac_82599EB) && (num_vfs < 16)) {
+ adapter->dcb_cfg.num_tcs.pg_tcs = MAX_TRAFFIC_CLASS;
+ adapter->dcb_cfg.num_tcs.pfc_tcs = MAX_TRAFFIC_CLASS;
+ } else if (num_vfs < 32) {
+ adapter->dcb_cfg.num_tcs.pg_tcs = 4;
+ adapter->dcb_cfg.num_tcs.pfc_tcs = 4;
+ } else {
+ adapter->dcb_cfg.num_tcs.pg_tcs = 1;
+ adapter->dcb_cfg.num_tcs.pfc_tcs = 1;
}
- return -ENOMEM;
+ /* Disable RSC when in SR-IOV mode */
+ adapter->flags2 &= ~(IXGBE_FLAG2_RSC_CAPABLE |
+ IXGBE_FLAG2_RSC_ENABLED);
+
+ for (i = 0; i < num_vfs; i++) {
+ /* enable spoof checking for all VFs */
+ adapter->vfinfo[i].spoofchk_enabled = true;
+
+ /* We support VF RSS querying only for 82599 and x540
+ * devices at the moment. These devices share RSS
+ * indirection table and RSS hash key with PF therefore
+ * we want to disable the querying by default.
+ */
+ adapter->vfinfo[i].rss_query_enabled = 0;
+
+ /* Untrust all VFs */
+ adapter->vfinfo[i].trusted = false;
+
+ /* set the default xcast mode */
+ adapter->vfinfo[i].xcast_mode = IXGBEVF_XCAST_MODE_NONE;
+ }
+
+ e_info(probe, "SR-IOV enabled with %d VFs\n", num_vfs);
+ return 0;
}
/**
@@ -165,12 +173,13 @@
/* Note this function is called when the user wants to enable SR-IOV
* VFs using the now deprecated module parameter
*/
-void ixgbe_enable_sriov(struct ixgbe_adapter *adapter)
+void ixgbe_enable_sriov(struct ixgbe_adapter *adapter, unsigned int max_vfs)
{
int pre_existing_vfs = 0;
+ unsigned int num_vfs;
pre_existing_vfs = pci_num_vf(adapter->pdev);
- if (!pre_existing_vfs && !adapter->num_vfs)
+ if (!pre_existing_vfs && !max_vfs)
return;
/* If there are pre-existing VFs then we have to force
@@ -180,7 +189,7 @@
* have been created via the new PCI SR-IOV sysfs interface.
*/
if (pre_existing_vfs) {
- adapter->num_vfs = pre_existing_vfs;
+ num_vfs = pre_existing_vfs;
dev_warn(&adapter->pdev->dev,
"Virtual Functions already enabled for this device - Please reload all VF drivers to avoid spoofed packet errors\n");
} else {
@@ -192,17 +201,16 @@
* physical function. If the user requests greater than
* 63 VFs then it is an error - reset to default of zero.
*/
- adapter->num_vfs = min_t(unsigned int, adapter->num_vfs, IXGBE_MAX_VFS_DRV_LIMIT);
+ num_vfs = min_t(unsigned int, max_vfs, IXGBE_MAX_VFS_DRV_LIMIT);
- err = pci_enable_sriov(adapter->pdev, adapter->num_vfs);
+ err = pci_enable_sriov(adapter->pdev, num_vfs);
if (err) {
e_err(probe, "Failed to enable PCI sriov: %d\n", err);
- adapter->num_vfs = 0;
return;
}
}
- if (!__ixgbe_enable_sriov(adapter)) {
+ if (!__ixgbe_enable_sriov(adapter, num_vfs)) {
ixgbe_get_vfs(adapter);
return;
}
@@ -298,6 +306,7 @@
#ifdef CONFIG_PCI_IOV
struct ixgbe_adapter *adapter = pci_get_drvdata(dev);
int err = 0;
+ u8 num_tc;
int i;
int pre_existing_vfs = pci_num_vf(dev);
@@ -310,23 +319,41 @@
return err;
/* While the SR-IOV capability structure reports total VFs to be 64,
- * we have to limit the actual number allocated based on two factors.
+ * we limit the actual number allocated as below based on two factors.
+ * Num_TCs MAX_VFs
+ * 1 63
+ * <=4 31
+ * >4 15
* First, we reserve some transmit/receive resources for the PF.
* Second, VMDQ also uses the same pools that SR-IOV does. We need to
* account for this, so that we don't accidentally allocate more VFs
* than we have available pools. The PCI bus driver already checks for
* other values out of range.
*/
- if ((num_vfs + adapter->num_rx_pools) > IXGBE_MAX_VF_FUNCTIONS)
- return -EPERM;
+ num_tc = netdev_get_num_tc(adapter->netdev);
- adapter->num_vfs = num_vfs;
+ if (num_tc > 4) {
+ if ((num_vfs + adapter->num_rx_pools) > IXGBE_MAX_VFS_8TC) {
+ e_dev_err("Currently the device is configured with %d TCs, Creating more than %d VFs is not allowed\n", num_tc, IXGBE_MAX_VFS_8TC);
+ return -EPERM;
+ }
+ } else if ((num_tc > 1) && (num_tc <= 4)) {
+ if ((num_vfs + adapter->num_rx_pools) > IXGBE_MAX_VFS_4TC) {
+ e_dev_err("Currently the device is configured with %d TCs, Creating more than %d VFs is not allowed\n", num_tc, IXGBE_MAX_VFS_4TC);
+ return -EPERM;
+ }
+ } else {
+ if ((num_vfs + adapter->num_rx_pools) > IXGBE_MAX_VFS_1TC) {
+ e_dev_err("Currently the device is configured with %d TCs, Creating more than %d VFs is not allowed\n", num_tc, IXGBE_MAX_VFS_1TC);
+ return -EPERM;
+ }
+ }
- err = __ixgbe_enable_sriov(adapter);
+ err = __ixgbe_enable_sriov(adapter, num_vfs);
if (err)
return err;
- for (i = 0; i < adapter->num_vfs; i++)
+ for (i = 0; i < num_vfs; i++)
ixgbe_vf_configuration(dev, (i | 0x10000000));
/* reset before enabling SRIOV to avoid mailbox issues */
@@ -650,58 +677,6 @@
}
}
-static inline void ixgbe_vf_reset_event(struct ixgbe_adapter *adapter, u32 vf)
-{
- struct ixgbe_hw *hw = &adapter->hw;
- struct vf_data_storage *vfinfo = &adapter->vfinfo[vf];
- u8 num_tcs = netdev_get_num_tc(adapter->netdev);
-
- /* remove VLAN filters beloning to this VF */
- ixgbe_clear_vf_vlans(adapter, vf);
-
- /* add back PF assigned VLAN or VLAN 0 */
- ixgbe_set_vf_vlan(adapter, true, vfinfo->pf_vlan, vf);
-
- /* reset offloads to defaults */
- ixgbe_set_vmolr(hw, vf, !vfinfo->pf_vlan);
-
- /* set outgoing tags for VFs */
- if (!vfinfo->pf_vlan && !vfinfo->pf_qos && !num_tcs) {
- ixgbe_clear_vmvir(adapter, vf);
- } else {
- if (vfinfo->pf_qos || !num_tcs)
- ixgbe_set_vmvir(adapter, vfinfo->pf_vlan,
- vfinfo->pf_qos, vf);
- else
- ixgbe_set_vmvir(adapter, vfinfo->pf_vlan,
- adapter->default_up, vf);
-
- if (vfinfo->spoofchk_enabled)
- hw->mac.ops.set_vlan_anti_spoofing(hw, true, vf);
- }
-
- /* reset multicast table array for vf */
- adapter->vfinfo[vf].num_vf_mc_hashes = 0;
-
- /* Flush and reset the mta with the new values */
- ixgbe_set_rx_mode(adapter->netdev);
-
- ixgbe_del_mac_filter(adapter, adapter->vfinfo[vf].vf_mac_addresses, vf);
-
- /* reset VF api back to unknown */
- adapter->vfinfo[vf].vf_api = ixgbe_mbox_api_10;
-}
-
-static int ixgbe_set_vf_mac(struct ixgbe_adapter *adapter,
- int vf, unsigned char *mac_addr)
-{
- ixgbe_del_mac_filter(adapter, adapter->vfinfo[vf].vf_mac_addresses, vf);
- memcpy(adapter->vfinfo[vf].vf_mac_addresses, mac_addr, ETH_ALEN);
- ixgbe_add_mac_filter(adapter, adapter->vfinfo[vf].vf_mac_addresses, vf);
-
- return 0;
-}
-
static int ixgbe_set_vf_macvlan(struct ixgbe_adapter *adapter,
int vf, int index, unsigned char *mac_addr)
{
@@ -757,6 +732,59 @@
return 0;
}
+static inline void ixgbe_vf_reset_event(struct ixgbe_adapter *adapter, u32 vf)
+{
+ struct ixgbe_hw *hw = &adapter->hw;
+ struct vf_data_storage *vfinfo = &adapter->vfinfo[vf];
+ u8 num_tcs = netdev_get_num_tc(adapter->netdev);
+
+ /* remove VLAN filters beloning to this VF */
+ ixgbe_clear_vf_vlans(adapter, vf);
+
+ /* add back PF assigned VLAN or VLAN 0 */
+ ixgbe_set_vf_vlan(adapter, true, vfinfo->pf_vlan, vf);
+
+ /* reset offloads to defaults */
+ ixgbe_set_vmolr(hw, vf, !vfinfo->pf_vlan);
+
+ /* set outgoing tags for VFs */
+ if (!vfinfo->pf_vlan && !vfinfo->pf_qos && !num_tcs) {
+ ixgbe_clear_vmvir(adapter, vf);
+ } else {
+ if (vfinfo->pf_qos || !num_tcs)
+ ixgbe_set_vmvir(adapter, vfinfo->pf_vlan,
+ vfinfo->pf_qos, vf);
+ else
+ ixgbe_set_vmvir(adapter, vfinfo->pf_vlan,
+ adapter->default_up, vf);
+
+ if (vfinfo->spoofchk_enabled)
+ hw->mac.ops.set_vlan_anti_spoofing(hw, true, vf);
+ }
+
+ /* reset multicast table array for vf */
+ adapter->vfinfo[vf].num_vf_mc_hashes = 0;
+
+ /* Flush and reset the mta with the new values */
+ ixgbe_set_rx_mode(adapter->netdev);
+
+ ixgbe_del_mac_filter(adapter, adapter->vfinfo[vf].vf_mac_addresses, vf);
+ ixgbe_set_vf_macvlan(adapter, vf, 0, NULL);
+
+ /* reset VF api back to unknown */
+ adapter->vfinfo[vf].vf_api = ixgbe_mbox_api_10;
+}
+
+static int ixgbe_set_vf_mac(struct ixgbe_adapter *adapter,
+ int vf, unsigned char *mac_addr)
+{
+ ixgbe_del_mac_filter(adapter, adapter->vfinfo[vf].vf_mac_addresses, vf);
+ memcpy(adapter->vfinfo[vf].vf_mac_addresses, mac_addr, ETH_ALEN);
+ ixgbe_add_mac_filter(adapter, adapter->vfinfo[vf].vf_mac_addresses, vf);
+
+ return 0;
+}
+
int ixgbe_vf_configuration(struct pci_dev *pdev, unsigned int event_mask)
{
struct ixgbe_adapter *adapter = pci_get_drvdata(pdev);
@@ -1085,7 +1113,7 @@
return -EOPNOTSUPP;
}
- memcpy(rss_key, adapter->rss_key, sizeof(adapter->rss_key));
+ memcpy(rss_key, adapter->rss_key, IXGBE_RSS_KEY_SIZE);
return 0;
}
@@ -1319,18 +1347,26 @@
int ixgbe_ndo_set_vf_mac(struct net_device *netdev, int vf, u8 *mac)
{
struct ixgbe_adapter *adapter = netdev_priv(netdev);
- if (!is_valid_ether_addr(mac) || (vf >= adapter->num_vfs))
+
+ if (vf >= adapter->num_vfs)
return -EINVAL;
- adapter->vfinfo[vf].pf_set_mac = true;
- dev_info(&adapter->pdev->dev, "setting MAC %pM on VF %d\n", mac, vf);
- dev_info(&adapter->pdev->dev, "Reload the VF driver to make this"
- " change effective.");
- if (test_bit(__IXGBE_DOWN, &adapter->state)) {
- dev_warn(&adapter->pdev->dev, "The VF MAC address has been set,"
- " but the PF device is not up.\n");
- dev_warn(&adapter->pdev->dev, "Bring the PF device up before"
- " attempting to use the VF device.\n");
+
+ if (is_zero_ether_addr(mac)) {
+ adapter->vfinfo[vf].pf_set_mac = false;
+ dev_info(&adapter->pdev->dev, "removing MAC on VF %d\n", vf);
+ } else if (is_valid_ether_addr(mac)) {
+ adapter->vfinfo[vf].pf_set_mac = true;
+ dev_info(&adapter->pdev->dev, "setting MAC %pM on VF %d\n",
+ mac, vf);
+ dev_info(&adapter->pdev->dev, "Reload the VF driver to make this change effective.");
+ if (test_bit(__IXGBE_DOWN, &adapter->state)) {
+ dev_warn(&adapter->pdev->dev, "The VF MAC address has been set, but the PF device is not up.\n");
+ dev_warn(&adapter->pdev->dev, "Bring the PF device up before attempting to use the VF device.\n");
+ }
+ } else {
+ return -EINVAL;
}
+
return ixgbe_set_vf_mac(adapter, vf, mac);
}
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.h
index 0c7977d..cf67b9b 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.h
@@ -33,6 +33,9 @@
* 63 (IXGBE_MAX_VF_FUNCTIONS - 1)
*/
#define IXGBE_MAX_VFS_DRV_LIMIT (IXGBE_MAX_VF_FUNCTIONS - 1)
+#define IXGBE_MAX_VFS_1TC IXGBE_MAX_VF_FUNCTIONS
+#define IXGBE_MAX_VFS_4TC 32
+#define IXGBE_MAX_VFS_8TC 16
#ifdef CONFIG_PCI_IOV
void ixgbe_restore_vf_multicasts(struct ixgbe_adapter *adapter);
@@ -56,7 +59,7 @@
void ixgbe_check_vf_rate_limit(struct ixgbe_adapter *adapter);
int ixgbe_disable_sriov(struct ixgbe_adapter *adapter);
#ifdef CONFIG_PCI_IOV
-void ixgbe_enable_sriov(struct ixgbe_adapter *adapter);
+void ixgbe_enable_sriov(struct ixgbe_adapter *adapter, unsigned int max_vfs);
#endif
int ixgbe_pci_sriov_configure(struct pci_dev *dev, int num_vfs);
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h b/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h
index 1d07f2e..9c2460c 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_type.h
@@ -85,6 +85,7 @@
#define IXGBE_DEV_ID_X550EM_X_SFP 0x15AC
#define IXGBE_DEV_ID_X550EM_X_10G_T 0x15AD
#define IXGBE_DEV_ID_X550EM_X_1G_T 0x15AE
+#define IXGBE_DEV_ID_X550EM_X_XFI 0x15B0
#define IXGBE_DEV_ID_X550EM_A_KR 0x15C2
#define IXGBE_DEV_ID_X550EM_A_KR_L 0x15C3
#define IXGBE_DEV_ID_X550EM_A_SFP_N 0x15C4
@@ -1387,9 +1388,6 @@
#define ATH_PHY_ID 0x03429050
#define AQ_FW_REV 0x20
-/* PHY Types */
-#define IXGBE_M88E1145_E_PHY_ID 0x01410CD0
-
/* Special PHY Init Routine */
#define IXGBE_PHY_INIT_OFFSET_NL 0x002B
#define IXGBE_PHY_INIT_END_NL 0xFFFF
@@ -3128,7 +3126,9 @@
ixgbe_phy_aq,
ixgbe_phy_x550em_kr,
ixgbe_phy_x550em_kx4,
+ ixgbe_phy_x550em_xfi,
ixgbe_phy_x550em_ext_t,
+ ixgbe_phy_ext_1g_t,
ixgbe_phy_cu_unknown,
ixgbe_phy_qt,
ixgbe_phy_xaui,
@@ -3754,15 +3754,6 @@
#define IXGBE_KRM_TX_COEFF_CTRL_1_CZERO_EN BIT(3)
#define IXGBE_KRM_TX_COEFF_CTRL_1_OVRRD_EN BIT(31)
-#define IXGBE_KX4_LINK_CNTL_1 0x4C
-#define IXGBE_KX4_LINK_CNTL_1_TETH_AN_CAP_KX BIT(16)
-#define IXGBE_KX4_LINK_CNTL_1_TETH_AN_CAP_KX4 BIT(17)
-#define IXGBE_KX4_LINK_CNTL_1_TETH_EEE_CAP_KX BIT(24)
-#define IXGBE_KX4_LINK_CNTL_1_TETH_EEE_CAP_KX4 BIT(25)
-#define IXGBE_KX4_LINK_CNTL_1_TETH_AN_ENABLE BIT(29)
-#define IXGBE_KX4_LINK_CNTL_1_TETH_FORCE_LINK_UP BIT(30)
-#define IXGBE_KX4_LINK_CNTL_1_TETH_AN_RESTART BIT(31)
-
#define IXGBE_SB_IOSF_INDIRECT_CTRL 0x00011144
#define IXGBE_SB_IOSF_INDIRECT_DATA 0x00011148
@@ -3779,12 +3770,14 @@
#define IXGBE_SB_IOSF_CTRL_BUSY_SHIFT 31
#define IXGBE_SB_IOSF_CTRL_BUSY BIT(IXGBE_SB_IOSF_CTRL_BUSY_SHIFT)
#define IXGBE_SB_IOSF_TARGET_KR_PHY 0
-#define IXGBE_SB_IOSF_TARGET_KX4_UNIPHY 1
-#define IXGBE_SB_IOSF_TARGET_KX4_PCS0 2
-#define IXGBE_SB_IOSF_TARGET_KX4_PCS1 3
#define IXGBE_NW_MNG_IF_SEL 0x00011178
#define IXGBE_NW_MNG_IF_SEL_MDIO_ACT BIT(1)
+#define IXGBE_NW_MNG_IF_SEL_PHY_SPEED_10M BIT(17)
+#define IXGBE_NW_MNG_IF_SEL_PHY_SPEED_100M BIT(18)
+#define IXGBE_NW_MNG_IF_SEL_PHY_SPEED_1G BIT(19)
+#define IXGBE_NW_MNG_IF_SEL_PHY_SPEED_2_5G BIT(20)
+#define IXGBE_NW_MNG_IF_SEL_PHY_SPEED_10G BIT(21)
#define IXGBE_NW_MNG_IF_SEL_ENABLE_10_100M BIT(23)
#define IXGBE_NW_MNG_IF_SEL_INT_PHY_MODE BIT(24)
#define IXGBE_NW_MNG_IF_SEL_MDIO_PHY_ADD_SHIFT 3
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c
index 84a467a..6ea0d6a 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_x540.c
@@ -95,6 +95,7 @@
{
s32 status;
u32 ctrl, i;
+ u32 swfw_mask = hw->phy.phy_semaphore_mask;
/* Call adapter stop to disable tx/rx and clear interrupts */
status = hw->mac.ops.stop_adapter(hw);
@@ -105,10 +106,17 @@
ixgbe_clear_tx_pending(hw);
mac_reset_top:
+ status = hw->mac.ops.acquire_swfw_sync(hw, swfw_mask);
+ if (status) {
+ hw_dbg(hw, "semaphore failed with %d", status);
+ return IXGBE_ERR_SWFW_SYNC;
+ }
+
ctrl = IXGBE_CTRL_RST;
ctrl |= IXGBE_READ_REG(hw, IXGBE_CTRL);
IXGBE_WRITE_REG(hw, IXGBE_CTRL, ctrl);
IXGBE_WRITE_FLUSH(hw);
+ hw->mac.ops.release_swfw_sync(hw, swfw_mask);
usleep_range(1000, 1200);
/* Poll for reset bit to self-clear indicating reset is complete */
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_x550.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_x550.c
index 200f847..2ba024b 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_x550.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_x550.c
@@ -49,6 +49,18 @@
return 0;
}
+static s32 ixgbe_get_invariants_X550_x_fw(struct ixgbe_hw *hw)
+{
+ struct ixgbe_phy_info *phy = &hw->phy;
+
+ /* Start with X540 invariants, since so similar */
+ ixgbe_get_invariants_X540(hw);
+
+ phy->ops.set_phy_power = NULL;
+
+ return 0;
+}
+
static s32 ixgbe_get_invariants_X550_a(struct ixgbe_hw *hw)
{
struct ixgbe_mac_info *mac = &hw->mac;
@@ -320,6 +332,9 @@
case IXGBE_DEV_ID_X550EM_X_KX4:
hw->phy.type = ixgbe_phy_x550em_kx4;
break;
+ case IXGBE_DEV_ID_X550EM_X_XFI:
+ hw->phy.type = ixgbe_phy_x550em_xfi;
+ break;
case IXGBE_DEV_ID_X550EM_X_KR:
case IXGBE_DEV_ID_X550EM_A_KR:
case IXGBE_DEV_ID_X550EM_A_KR_L:
@@ -331,9 +346,21 @@
else
hw->phy.phy_semaphore_mask = IXGBE_GSSR_PHY0_SM;
/* Fallthrough */
- case IXGBE_DEV_ID_X550EM_X_1G_T:
case IXGBE_DEV_ID_X550EM_X_10G_T:
return ixgbe_identify_phy_generic(hw);
+ case IXGBE_DEV_ID_X550EM_X_1G_T:
+ hw->phy.type = ixgbe_phy_ext_1g_t;
+ break;
+ case IXGBE_DEV_ID_X550EM_A_1G_T:
+ case IXGBE_DEV_ID_X550EM_A_1G_T_L:
+ hw->phy.type = ixgbe_phy_fw;
+ hw->phy.ops.read_reg = NULL;
+ hw->phy.ops.write_reg = NULL;
+ if (hw->bus.lan_id)
+ hw->phy.phy_semaphore_mask |= IXGBE_GSSR_PHY1_SM;
+ else
+ hw->phy.phy_semaphore_mask |= IXGBE_GSSR_PHY0_SM;
+ break;
default:
break;
}
@@ -2145,6 +2172,8 @@
ixgbe_set_soft_rate_select_speed;
break;
case ixgbe_media_type_copper:
+ if (hw->device_id == IXGBE_DEV_ID_X550EM_X_1G_T)
+ break;
mac->ops.setup_link = ixgbe_setup_mac_link_t_X550em;
mac->ops.setup_fc = ixgbe_setup_fc_generic;
mac->ops.check_link = ixgbe_check_link_t_X550em;
@@ -2215,8 +2244,39 @@
else
*speed = IXGBE_LINK_SPEED_10GB_FULL;
} else {
- *speed = IXGBE_LINK_SPEED_10GB_FULL |
- IXGBE_LINK_SPEED_1GB_FULL;
+ switch (hw->phy.type) {
+ case ixgbe_phy_x550em_kx4:
+ *speed = IXGBE_LINK_SPEED_1GB_FULL |
+ IXGBE_LINK_SPEED_2_5GB_FULL |
+ IXGBE_LINK_SPEED_10GB_FULL;
+ break;
+ case ixgbe_phy_x550em_xfi:
+ *speed = IXGBE_LINK_SPEED_1GB_FULL |
+ IXGBE_LINK_SPEED_10GB_FULL;
+ break;
+ case ixgbe_phy_ext_1g_t:
+ case ixgbe_phy_sgmii:
+ *speed = IXGBE_LINK_SPEED_1GB_FULL;
+ break;
+ case ixgbe_phy_x550em_kr:
+ if (hw->mac.type == ixgbe_mac_x550em_a) {
+ /* check different backplane modes */
+ if (hw->phy.nw_mng_if_sel &
+ IXGBE_NW_MNG_IF_SEL_PHY_SPEED_2_5G) {
+ *speed = IXGBE_LINK_SPEED_2_5GB_FULL;
+ break;
+ } else if (hw->device_id ==
+ IXGBE_DEV_ID_X550EM_A_KR_L) {
+ *speed = IXGBE_LINK_SPEED_1GB_FULL;
+ break;
+ }
+ }
+ /* fall through */
+ default:
+ *speed = IXGBE_LINK_SPEED_10GB_FULL |
+ IXGBE_LINK_SPEED_1GB_FULL;
+ break;
+ }
*autoneg = true;
}
return 0;
@@ -2473,44 +2533,6 @@
return ixgbe_restart_an_internal_phy_x550em(hw);
}
-/** ixgbe_setup_kx4_x550em - Configure the KX4 PHY.
- * @hw: pointer to hardware structure
- *
- * Configures the integrated KX4 PHY.
- **/
-static s32 ixgbe_setup_kx4_x550em(struct ixgbe_hw *hw)
-{
- s32 status;
- u32 reg_val;
-
- status = hw->mac.ops.read_iosf_sb_reg(hw, IXGBE_KX4_LINK_CNTL_1,
- IXGBE_SB_IOSF_TARGET_KX4_PCS0 +
- hw->bus.lan_id, ®_val);
- if (status)
- return status;
-
- reg_val &= ~(IXGBE_KX4_LINK_CNTL_1_TETH_AN_CAP_KX4 |
- IXGBE_KX4_LINK_CNTL_1_TETH_AN_CAP_KX);
-
- reg_val |= IXGBE_KX4_LINK_CNTL_1_TETH_AN_ENABLE;
-
- /* Advertise 10G support. */
- if (hw->phy.autoneg_advertised & IXGBE_LINK_SPEED_10GB_FULL)
- reg_val |= IXGBE_KX4_LINK_CNTL_1_TETH_AN_CAP_KX4;
-
- /* Advertise 1G support. */
- if (hw->phy.autoneg_advertised & IXGBE_LINK_SPEED_1GB_FULL)
- reg_val |= IXGBE_KX4_LINK_CNTL_1_TETH_AN_CAP_KX;
-
- /* Restart auto-negotiation. */
- reg_val |= IXGBE_KX4_LINK_CNTL_1_TETH_AN_RESTART;
- status = hw->mac.ops.write_iosf_sb_reg(hw, IXGBE_KX4_LINK_CNTL_1,
- IXGBE_SB_IOSF_TARGET_KX4_PCS0 +
- hw->bus.lan_id, reg_val);
-
- return status;
-}
-
/**
* ixgbe_setup_kr_x550em - Configure the KR PHY
* @hw: pointer to hardware structure
@@ -2521,6 +2543,9 @@
if (hw->phy.autoneg_advertised & IXGBE_LINK_SPEED_2_5GB_FULL)
return 0;
+ if (ixgbe_check_reset_blocked(hw))
+ return 0;
+
return ixgbe_setup_kr_speed_x550em(hw, hw->phy.autoneg_advertised);
}
@@ -3134,7 +3159,7 @@
/* Set functions pointers based on phy type */
switch (hw->phy.type) {
case ixgbe_phy_x550em_kx4:
- phy->ops.setup_link = ixgbe_setup_kx4_x550em;
+ phy->ops.setup_link = NULL;
phy->ops.read_reg = ixgbe_read_phy_reg_x550em;
phy->ops.write_reg = ixgbe_write_phy_reg_x550em;
break;
@@ -3143,6 +3168,12 @@
phy->ops.read_reg = ixgbe_read_phy_reg_x550em;
phy->ops.write_reg = ixgbe_write_phy_reg_x550em;
break;
+ case ixgbe_phy_x550em_xfi:
+ /* link is managed by HW */
+ phy->ops.setup_link = NULL;
+ phy->ops.read_reg = ixgbe_read_phy_reg_x550em;
+ phy->ops.write_reg = ixgbe_write_phy_reg_x550em;
+ break;
case ixgbe_phy_x550em_ext_t:
/* Save NW management interface connected on board. This is used
* to determine internal PHY mode
@@ -3164,10 +3195,18 @@
phy->ops.handle_lasi = ixgbe_handle_lasi_ext_t_x550em;
phy->ops.reset = ixgbe_reset_phy_t_X550em;
break;
+ case ixgbe_phy_sgmii:
+ phy->ops.setup_link = NULL;
+ break;
case ixgbe_phy_fw:
phy->ops.setup_link = ixgbe_setup_fw_link;
phy->ops.reset = ixgbe_reset_phy_fw;
break;
+ case ixgbe_phy_ext_1g_t:
+ phy->ops.setup_link = NULL;
+ phy->ops.read_reg = NULL;
+ phy->ops.write_reg = NULL;
+ break;
default:
break;
}
@@ -3193,6 +3232,7 @@
/* Fallthrough */
case IXGBE_DEV_ID_X550EM_X_KR:
case IXGBE_DEV_ID_X550EM_X_KX4:
+ case IXGBE_DEV_ID_X550EM_X_XFI:
case IXGBE_DEV_ID_X550EM_A_KR:
case IXGBE_DEV_ID_X550EM_A_KR_L:
media_type = ixgbe_media_type_backplane;
@@ -3300,6 +3340,7 @@
u32 ctrl = 0;
u32 i;
bool link_up = false;
+ u32 swfw_mask = hw->phy.phy_semaphore_mask;
/* Call adapter stop to disable Tx/Rx and clear interrupts */
status = hw->mac.ops.stop_adapter(hw);
@@ -3345,9 +3386,16 @@
ctrl = IXGBE_CTRL_RST;
}
+ status = hw->mac.ops.acquire_swfw_sync(hw, swfw_mask);
+ if (status) {
+ hw_dbg(hw, "semaphore failed with %d", status);
+ return IXGBE_ERR_SWFW_SYNC;
+ }
+
ctrl |= IXGBE_READ_REG(hw, IXGBE_CTRL);
IXGBE_WRITE_REG(hw, IXGBE_CTRL, ctrl);
IXGBE_WRITE_FLUSH(hw);
+ hw->mac.ops.release_swfw_sync(hw, swfw_mask);
usleep_range(1000, 1200);
/* Poll for reset bit to self-clear meaning reset is complete */
@@ -3780,7 +3828,7 @@
.get_media_type = ixgbe_get_media_type_X550em,
.get_san_mac_addr = NULL,
.get_wwn_prefix = NULL,
- .setup_link = NULL, /* defined later */
+ .setup_link = &ixgbe_setup_mac_link_X540,
.get_link_capabilities = ixgbe_get_link_capabilities_X550em,
.get_bus_info = ixgbe_get_bus_info_X550em,
.setup_sfp = ixgbe_setup_sfp_modules_X550em,
@@ -3862,6 +3910,17 @@
.write_reg = &ixgbe_write_phy_reg_generic,
};
+static const struct ixgbe_phy_operations phy_ops_x550em_x_fw = {
+ X550_COMMON_PHY
+ .check_overtemp = NULL,
+ .init = ixgbe_init_phy_ops_X550em,
+ .identify = ixgbe_identify_phy_x550em,
+ .read_reg = NULL,
+ .write_reg = NULL,
+ .read_reg_mdi = NULL,
+ .write_reg_mdi = NULL,
+};
+
static const struct ixgbe_phy_operations phy_ops_x550em_a = {
X550_COMMON_PHY
.check_overtemp = &ixgbe_tn_check_overtemp,
@@ -3924,6 +3983,16 @@
.link_ops = &link_ops_x550em_x,
};
+const struct ixgbe_info ixgbe_x550em_x_fw_info = {
+ .mac = ixgbe_mac_X550EM_x,
+ .get_invariants = ixgbe_get_invariants_X550_x_fw,
+ .mac_ops = &mac_ops_X550EM_x,
+ .eeprom_ops = &eeprom_ops_X550EM_x,
+ .phy_ops = &phy_ops_x550em_x_fw,
+ .mbx_ops = &mbx_ops_generic,
+ .mvals = ixgbe_mvals_X550EM_x,
+};
+
const struct ixgbe_info ixgbe_x550em_a_info = {
.mac = ixgbe_mac_x550em_a,
.get_invariants = &ixgbe_get_invariants_X550_a,
diff --git a/drivers/net/ethernet/intel/ixgbevf/ethtool.c b/drivers/net/ethernet/intel/ixgbevf/ethtool.c
index 1f6c0ec..ff9d05f 100644
--- a/drivers/net/ethernet/intel/ixgbevf/ethtool.c
+++ b/drivers/net/ethernet/intel/ixgbevf/ethtool.c
@@ -80,7 +80,7 @@
#define IXGBEVF_QUEUE_STATS_LEN ( \
(((struct ixgbevf_adapter *)netdev_priv(netdev))->num_tx_queues + \
((struct ixgbevf_adapter *)netdev_priv(netdev))->num_rx_queues) * \
- (sizeof(struct ixgbe_stats) / sizeof(u64)))
+ (sizeof(struct ixgbevf_stats) / sizeof(u64)))
#define IXGBEVF_GLOBAL_STATS_LEN ARRAY_SIZE(ixgbevf_gstrings_stats)
#define IXGBEVF_STATS_LEN (IXGBEVF_GLOBAL_STATS_LEN + IXGBEVF_QUEUE_STATS_LEN)
@@ -91,18 +91,18 @@
#define IXGBEVF_TEST_LEN (sizeof(ixgbe_gstrings_test) / ETH_GSTRING_LEN)
-static int ixgbevf_get_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+static int ixgbevf_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
{
struct ixgbevf_adapter *adapter = netdev_priv(netdev);
struct ixgbe_hw *hw = &adapter->hw;
u32 link_speed = 0;
bool link_up;
- ecmd->supported = SUPPORTED_10000baseT_Full;
- ecmd->autoneg = AUTONEG_DISABLE;
- ecmd->transceiver = XCVR_DUMMY1;
- ecmd->port = -1;
+ ethtool_link_ksettings_zero_link_mode(cmd, supported);
+ ethtool_link_ksettings_add_link_mode(cmd, supported, 10000baseT_Full);
+ cmd->base.autoneg = AUTONEG_DISABLE;
+ cmd->base.port = -1;
hw->mac.get_link_status = 1;
hw->mac.ops.check_link(hw, &link_speed, &link_up, false);
@@ -122,11 +122,11 @@
break;
}
- ethtool_cmd_speed_set(ecmd, speed);
- ecmd->duplex = DUPLEX_FULL;
+ cmd->base.speed = speed;
+ cmd->base.duplex = DUPLEX_FULL;
} else {
- ethtool_cmd_speed_set(ecmd, SPEED_UNKNOWN);
- ecmd->duplex = DUPLEX_UNKNOWN;
+ cmd->base.speed = SPEED_UNKNOWN;
+ cmd->base.duplex = DUPLEX_UNKNOWN;
}
return 0;
@@ -855,7 +855,8 @@
if (adapter->hw.mac.type >= ixgbe_mac_X550_vf) {
if (key)
- memcpy(key, adapter->rss_key, sizeof(adapter->rss_key));
+ memcpy(key, adapter->rss_key,
+ ixgbevf_get_rxfh_key_size(netdev));
if (indir) {
int i;
@@ -885,7 +886,6 @@
}
static const struct ethtool_ops ixgbevf_ethtool_ops = {
- .get_settings = ixgbevf_get_settings,
.get_drvinfo = ixgbevf_get_drvinfo,
.get_regs_len = ixgbevf_get_regs_len,
.get_regs = ixgbevf_get_regs,
@@ -905,6 +905,7 @@
.get_rxfh_indir_size = ixgbevf_get_rxfh_indir_size,
.get_rxfh_key_size = ixgbevf_get_rxfh_key_size,
.get_rxfh = ixgbevf_get_rxfh,
+ .get_link_ksettings = ixgbevf_get_link_ksettings,
};
void ixgbevf_set_ethtool_ops(struct net_device *netdev)
diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h b/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h
index a8cbc2d..581f44b 100644
--- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h
+++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf.h
@@ -319,7 +319,7 @@
spinlock_t mbx_lock;
unsigned long last_reset;
- u32 rss_key[IXGBEVF_VFRSSRK_REGS];
+ u32 *rss_key;
u8 rss_indir_tbl[IXGBEVF_X550_VFRETA_SIZE];
};
diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c
index 80bab26..eee29bd 100644
--- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c
+++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c
@@ -1660,6 +1660,28 @@
reg_idx);
}
+/**
+ * ixgbevf_init_rss_key - Initialize adapter RSS key
+ * @adapter: device handle
+ *
+ * Allocates and initializes the RSS key if it is not allocated.
+ **/
+static inline int ixgbevf_init_rss_key(struct ixgbevf_adapter *adapter)
+{
+ u32 *rss_key;
+
+ if (!adapter->rss_key) {
+ rss_key = kzalloc(IXGBEVF_RSS_HASH_KEY_SIZE, GFP_KERNEL);
+ if (unlikely(!rss_key))
+ return -ENOMEM;
+
+ netdev_rss_key_fill(rss_key, IXGBEVF_RSS_HASH_KEY_SIZE);
+ adapter->rss_key = rss_key;
+ }
+
+ return 0;
+}
+
static void ixgbevf_setup_vfmrqc(struct ixgbevf_adapter *adapter)
{
struct ixgbe_hw *hw = &adapter->hw;
@@ -1668,9 +1690,8 @@
u8 i, j;
/* Fill out hash function seeds */
- netdev_rss_key_fill(adapter->rss_key, sizeof(adapter->rss_key));
for (i = 0; i < IXGBEVF_VFRSSRK_REGS; i++)
- IXGBE_WRITE_REG(hw, IXGBE_VFRSSRK(i), adapter->rss_key[i]);
+ IXGBE_WRITE_REG(hw, IXGBE_VFRSSRK(i), *(adapter->rss_key + i));
for (i = 0, j = 0; i < IXGBEVF_X550_VFRETA_SIZE; i++, j++) {
if (j == rss_i)
@@ -2611,6 +2632,12 @@
hw->mbx.ops.init_params(hw);
+ if (hw->mac.type >= ixgbe_mac_X550_vf) {
+ err = ixgbevf_init_rss_key(adapter);
+ if (err)
+ goto out;
+ }
+
/* assume legacy case in which PF would only give VF 2 queues */
hw->mac.max_tx_queues = 2;
hw->mac.max_rx_queues = 2;
@@ -4127,6 +4154,7 @@
err_sw_init:
ixgbevf_reset_interrupt_capability(adapter);
iounmap(adapter->io_addr);
+ kfree(adapter->rss_key);
err_ioremap:
disable_dev = !test_and_set_bit(__IXGBEVF_DISABLED, &adapter->state);
free_netdev(netdev);
@@ -4173,6 +4201,7 @@
hw_dbg(&adapter->hw, "Remove complete\n");
+ kfree(adapter->rss_key);
disable_dev = !test_and_set_bit(__IXGBEVF_DISABLED, &adapter->state);
free_netdev(netdev);
diff --git a/drivers/net/ethernet/intel/ixgbevf/vf.c b/drivers/net/ethernet/intel/ixgbevf/vf.c
index 8a5db9d..b6d0c01 100644
--- a/drivers/net/ethernet/intel/ixgbevf/vf.c
+++ b/drivers/net/ethernet/intel/ixgbevf/vf.c
@@ -333,7 +333,7 @@
switch (hw->api_version) {
case ixgbe_mbox_api_13:
case ixgbe_mbox_api_12:
- if (hw->mac.type >= ixgbe_mac_X550_vf)
+ if (hw->mac.type < ixgbe_mac_X550_vf)
break;
default:
return -EOPNOTSUPP;
@@ -399,7 +399,7 @@
switch (hw->api_version) {
case ixgbe_mbox_api_13:
case ixgbe_mbox_api_12:
- if (hw->mac.type >= ixgbe_mac_X550_vf)
+ if (hw->mac.type < ixgbe_mac_X550_vf)
break;
default:
return -EOPNOTSUPP;
@@ -419,7 +419,7 @@
msgbuf[0] &= ~IXGBE_VT_MSGTYPE_CTS;
/* If the operation has been refused by a PF return -EPERM */
- if (msgbuf[0] == (IXGBE_VF_GET_RETA | IXGBE_VT_MSGTYPE_NACK))
+ if (msgbuf[0] == (IXGBE_VF_GET_RSS_KEY | IXGBE_VT_MSGTYPE_NACK))
return -EPERM;
/* If we didn't get an ACK there must have been
diff --git a/drivers/net/ethernet/marvell/Kconfig b/drivers/net/ethernet/marvell/Kconfig
index d2555e8b..da6fb82 100644
--- a/drivers/net/ethernet/marvell/Kconfig
+++ b/drivers/net/ethernet/marvell/Kconfig
@@ -82,13 +82,13 @@
that all dependencies are met.
config MVPP2
- tristate "Marvell Armada 375 network interface support"
+ tristate "Marvell Armada 375/7K/8K network interface support"
depends on ARCH_MVEBU || COMPILE_TEST
depends on HAS_DMA
select MVMDIO
---help---
This driver supports the network interface units in the
- Marvell ARMADA 375 SoC.
+ Marvell ARMADA 375, 7K and 8K SoCs.
config PXA168_ETH
tristate "Marvell pxa168 ethernet support"
diff --git a/drivers/net/ethernet/marvell/mvmdio.c b/drivers/net/ethernet/marvell/mvmdio.c
index a0d1b08..90a60b9 100644
--- a/drivers/net/ethernet/marvell/mvmdio.c
+++ b/drivers/net/ethernet/marvell/mvmdio.c
@@ -53,7 +53,7 @@
struct orion_mdio_dev {
struct mutex lock;
void __iomem *regs;
- struct clk *clk;
+ struct clk *clk[3];
/*
* If we have access to the error interrupt pin (which is
* somewhat misnamed as it not only reflects internal errors
@@ -187,7 +187,7 @@
struct resource *r;
struct mii_bus *bus;
struct orion_mdio_dev *dev;
- int ret;
+ int i, ret;
r = platform_get_resource(pdev, IORESOURCE_MEM, 0);
if (!r) {
@@ -216,11 +216,20 @@
init_waitqueue_head(&dev->smi_busy_wait);
- dev->clk = devm_clk_get(&pdev->dev, NULL);
- if (!IS_ERR(dev->clk))
- clk_prepare_enable(dev->clk);
+ for (i = 0; i < ARRAY_SIZE(dev->clk); i++) {
+ dev->clk[i] = of_clk_get(pdev->dev.of_node, i);
+ if (IS_ERR(dev->clk[i]))
+ break;
+ clk_prepare_enable(dev->clk[i]);
+ }
dev->err_interrupt = platform_get_irq(pdev, 0);
+ if (dev->err_interrupt > 0 &&
+ resource_size(r) < MVMDIO_ERR_INT_MASK + 4) {
+ dev_err(&pdev->dev,
+ "disabling interrupt, resource size is too small\n");
+ dev->err_interrupt = 0;
+ }
if (dev->err_interrupt > 0) {
ret = devm_request_irq(&pdev->dev, dev->err_interrupt,
orion_mdio_err_irq,
@@ -251,8 +260,16 @@
return 0;
out_mdio:
- if (!IS_ERR(dev->clk))
- clk_disable_unprepare(dev->clk);
+ if (dev->err_interrupt > 0)
+ writel(0, dev->regs + MVMDIO_ERR_INT_MASK);
+
+ for (i = 0; i < ARRAY_SIZE(dev->clk); i++) {
+ if (IS_ERR(dev->clk[i]))
+ break;
+ clk_disable_unprepare(dev->clk[i]);
+ clk_put(dev->clk[i]);
+ }
+
return ret;
}
@@ -260,11 +277,18 @@
{
struct mii_bus *bus = platform_get_drvdata(pdev);
struct orion_mdio_dev *dev = bus->priv;
+ int i;
- writel(0, dev->regs + MVMDIO_ERR_INT_MASK);
+ if (dev->err_interrupt > 0)
+ writel(0, dev->regs + MVMDIO_ERR_INT_MASK);
mdiobus_unregister(bus);
- if (!IS_ERR(dev->clk))
- clk_disable_unprepare(dev->clk);
+
+ for (i = 0; i < ARRAY_SIZE(dev->clk); i++) {
+ if (IS_ERR(dev->clk[i]))
+ break;
+ clk_disable_unprepare(dev->clk[i]);
+ clk_put(dev->clk[i]);
+ }
return 0;
}
diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index 61dd446..d297011 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -431,6 +431,7 @@
/* Flags for special SoC configurations */
bool neta_armada3700;
u16 rx_offset_correction;
+ const struct mbus_dram_target_info *dram_target_info;
};
/* The mvneta_tx_desc and mvneta_rx_desc structures describe the
@@ -1106,7 +1107,7 @@
q_map = 0;
for (queue = 0; queue < txq_number; queue++) {
struct mvneta_tx_queue *txq = &pp->txqs[queue];
- if (txq->descs != NULL)
+ if (txq->descs)
q_map |= (1 << queue);
}
mvreg_write(pp, MVNETA_TXQ_CMD, q_map);
@@ -1115,7 +1116,7 @@
for (queue = 0; queue < rxq_number; queue++) {
struct mvneta_rx_queue *rxq = &pp->rxqs[queue];
- if (rxq->descs != NULL)
+ if (rxq->descs)
q_map |= (1 << queue);
}
mvreg_write(pp, MVNETA_RXQ_CMD, q_map);
@@ -2849,7 +2850,7 @@
rxq->descs = dma_alloc_coherent(pp->dev->dev.parent,
rxq->size * MVNETA_DESC_ALIGNED_SIZE,
&rxq->descs_phys, GFP_KERNEL);
- if (rxq->descs == NULL)
+ if (!rxq->descs)
return -ENOMEM;
rxq->last_desc = rxq->size - 1;
@@ -2919,7 +2920,7 @@
txq->descs = dma_alloc_coherent(pp->dev->dev.parent,
txq->size * MVNETA_DESC_ALIGNED_SIZE,
&txq->descs_phys, GFP_KERNEL);
- if (txq->descs == NULL)
+ if (!txq->descs)
return -ENOMEM;
txq->last_desc = txq->size - 1;
@@ -2932,8 +2933,9 @@
mvreg_write(pp, MVNETA_TXQ_BASE_ADDR_REG(txq->id), txq->descs_phys);
mvreg_write(pp, MVNETA_TXQ_SIZE_REG(txq->id), txq->size);
- txq->tx_skb = kmalloc(txq->size * sizeof(*txq->tx_skb), GFP_KERNEL);
- if (txq->tx_skb == NULL) {
+ txq->tx_skb = kmalloc_array(txq->size, sizeof(*txq->tx_skb),
+ GFP_KERNEL);
+ if (!txq->tx_skb) {
dma_free_coherent(pp->dev->dev.parent,
txq->size * MVNETA_DESC_ALIGNED_SIZE,
txq->descs, txq->descs_phys);
@@ -2944,7 +2946,7 @@
txq->tso_hdrs = dma_alloc_coherent(pp->dev->dev.parent,
txq->size * TSO_HEADER_SIZE,
&txq->tso_hdrs_phys, GFP_KERNEL);
- if (txq->tso_hdrs == NULL) {
+ if (!txq->tso_hdrs) {
kfree(txq->tx_skb);
dma_free_coherent(pp->dev->dev.parent,
txq->size * MVNETA_DESC_ALIGNED_SIZE,
@@ -3317,6 +3319,7 @@
static int mvneta_mdio_probe(struct mvneta_port *pp)
{
struct phy_device *phy_dev;
+ struct ethtool_wolinfo wol = { .cmd = ETHTOOL_GWOL };
phy_dev = of_phy_connect(pp->dev, pp->phy_node, mvneta_adjust_link, 0,
pp->phy_interface);
@@ -3325,6 +3328,9 @@
return -ENODEV;
}
+ phy_ethtool_get_wol(phy_dev, &wol);
+ device_set_wakeup_capable(&pp->dev->dev, !!wol.supported);
+
phy_dev->supported &= PHY_GBIT_FEATURES;
phy_dev->advertising = phy_dev->supported;
@@ -3941,10 +3947,16 @@
static int mvneta_ethtool_set_wol(struct net_device *dev,
struct ethtool_wolinfo *wol)
{
+ int ret;
+
if (!dev->phydev)
return -EOPNOTSUPP;
- return phy_ethtool_set_wol(dev->phydev, wol);
+ ret = phy_ethtool_set_wol(dev->phydev, wol);
+ if (!ret)
+ device_set_wakeup_enable(&dev->dev, !!wol->wolopts);
+
+ return ret;
}
static const struct net_device_ops mvneta_netdev_ops = {
@@ -3991,8 +4003,7 @@
/* Set port default values */
mvneta_defaults_set(pp);
- pp->txqs = devm_kcalloc(dev, txq_number, sizeof(struct mvneta_tx_queue),
- GFP_KERNEL);
+ pp->txqs = devm_kcalloc(dev, txq_number, sizeof(*pp->txqs), GFP_KERNEL);
if (!pp->txqs)
return -ENOMEM;
@@ -4004,8 +4015,7 @@
txq->done_pkts_coal = MVNETA_TXDONE_COAL_PKTS;
}
- pp->rxqs = devm_kcalloc(dev, rxq_number, sizeof(struct mvneta_rx_queue),
- GFP_KERNEL);
+ pp->rxqs = devm_kcalloc(dev, rxq_number, sizeof(*pp->rxqs), GFP_KERNEL);
if (!pp->rxqs)
return -ENOMEM;
@@ -4016,9 +4026,11 @@
rxq->size = pp->rx_ring_size;
rxq->pkts_coal = MVNETA_RX_COAL_PKTS;
rxq->time_coal = MVNETA_RX_COAL_USEC;
- rxq->buf_virt_addr = devm_kmalloc(pp->dev->dev.parent,
- rxq->size * sizeof(void *),
- GFP_KERNEL);
+ rxq->buf_virt_addr
+ = devm_kmalloc_array(pp->dev->dev.parent,
+ rxq->size,
+ sizeof(*rxq->buf_virt_addr),
+ GFP_KERNEL);
if (!rxq->buf_virt_addr)
return -ENOMEM;
}
@@ -4098,6 +4110,8 @@
break;
case PHY_INTERFACE_MODE_RGMII:
case PHY_INTERFACE_MODE_RGMII_ID:
+ case PHY_INTERFACE_MODE_RGMII_RXID:
+ case PHY_INTERFACE_MODE_RGMII_TXID:
ctrl |= MVNETA_GMAC2_PORT_RGMII;
break;
default:
@@ -4118,7 +4132,6 @@
/* Device initialization routine */
static int mvneta_probe(struct platform_device *pdev)
{
- const struct mbus_dram_target_info *dram_target_info;
struct resource *res;
struct device_node *dn = pdev->dev.of_node;
struct device_node *phy_node;
@@ -4267,13 +4280,13 @@
pp->tx_csum_limit = tx_csum_limit;
- dram_target_info = mv_mbus_dram_info();
+ pp->dram_target_info = mv_mbus_dram_info();
/* Armada3700 requires setting default configuration of Mbus
* windows, however without using filled mbus_dram_target_info
* structure.
*/
- if (dram_target_info || pp->neta_armada3700)
- mvneta_conf_mbus_windows(pp, dram_target_info);
+ if (pp->dram_target_info || pp->neta_armada3700)
+ mvneta_conf_mbus_windows(pp, pp->dram_target_info);
pp->tx_ring_size = MVNETA_MAX_TXD;
pp->rx_ring_size = MVNETA_MAX_RXD;
@@ -4405,6 +4418,61 @@
return 0;
}
+#ifdef CONFIG_PM_SLEEP
+static int mvneta_suspend(struct device *device)
+{
+ struct net_device *dev = dev_get_drvdata(device);
+ struct mvneta_port *pp = netdev_priv(dev);
+
+ if (netif_running(dev))
+ mvneta_stop(dev);
+ netif_device_detach(dev);
+ clk_disable_unprepare(pp->clk_bus);
+ clk_disable_unprepare(pp->clk);
+ return 0;
+}
+
+static int mvneta_resume(struct device *device)
+{
+ struct platform_device *pdev = to_platform_device(device);
+ struct net_device *dev = dev_get_drvdata(device);
+ struct mvneta_port *pp = netdev_priv(dev);
+ int err;
+
+ clk_prepare_enable(pp->clk);
+ if (!IS_ERR(pp->clk_bus))
+ clk_prepare_enable(pp->clk_bus);
+ if (pp->dram_target_info || pp->neta_armada3700)
+ mvneta_conf_mbus_windows(pp, pp->dram_target_info);
+ if (pp->bm_priv) {
+ err = mvneta_bm_port_init(pdev, pp);
+ if (err < 0) {
+ dev_info(&pdev->dev, "use SW buffer management\n");
+ pp->bm_priv = NULL;
+ }
+ }
+ mvneta_defaults_set(pp);
+ err = mvneta_port_power_up(pp, pp->phy_interface);
+ if (err < 0) {
+ dev_err(device, "can't power up port\n");
+ return err;
+ }
+
+ if (pp->use_inband_status)
+ mvneta_fixed_link_update(pp, dev->phydev);
+
+ netif_device_attach(dev);
+ if (netif_running(dev)) {
+ mvneta_open(dev);
+ mvneta_set_rx_mode(dev);
+ }
+
+ return 0;
+}
+#endif
+
+static SIMPLE_DEV_PM_OPS(mvneta_pm_ops, mvneta_suspend, mvneta_resume);
+
static const struct of_device_id mvneta_match[] = {
{ .compatible = "marvell,armada-370-neta" },
{ .compatible = "marvell,armada-xp-neta" },
@@ -4419,6 +4487,7 @@
.driver = {
.name = MVNETA_DRIVER_NAME,
.of_match_table = mvneta_match,
+ .pm = &mvneta_pm_ops,
},
};
diff --git a/drivers/net/ethernet/marvell/mvpp2.c b/drivers/net/ethernet/marvell/mvpp2.c
index d00421b..9b875d7 100644
--- a/drivers/net/ethernet/marvell/mvpp2.c
+++ b/drivers/net/ethernet/marvell/mvpp2.c
@@ -25,6 +25,7 @@
#include <linux/of_mdio.h>
#include <linux/of_net.h>
#include <linux/of_address.h>
+#include <linux/of_device.h>
#include <linux/phy.h>
#include <linux/clk.h>
#include <linux/hrtimer.h>
@@ -49,9 +50,11 @@
#define MVPP2_SNOOP_PKT_SIZE_MASK 0x1ff
#define MVPP2_SNOOP_BUF_HDR_MASK BIT(9)
#define MVPP2_RXQ_POOL_SHORT_OFFS 20
-#define MVPP2_RXQ_POOL_SHORT_MASK 0x700000
+#define MVPP21_RXQ_POOL_SHORT_MASK 0x700000
+#define MVPP22_RXQ_POOL_SHORT_MASK 0xf00000
#define MVPP2_RXQ_POOL_LONG_OFFS 24
-#define MVPP2_RXQ_POOL_LONG_MASK 0x7000000
+#define MVPP21_RXQ_POOL_LONG_MASK 0x7000000
+#define MVPP22_RXQ_POOL_LONG_MASK 0xf000000
#define MVPP2_RXQ_PACKET_OFFSET_OFFS 28
#define MVPP2_RXQ_PACKET_OFFSET_MASK 0x70000000
#define MVPP2_RXQ_DISABLE_MASK BIT(31)
@@ -99,6 +102,7 @@
/* Descriptor Manager Top Registers */
#define MVPP2_RXQ_NUM_REG 0x2040
#define MVPP2_RXQ_DESC_ADDR_REG 0x2044
+#define MVPP22_DESC_ADDR_OFFS 8
#define MVPP2_RXQ_DESC_SIZE_REG 0x2048
#define MVPP2_RXQ_DESC_SIZE_MASK 0x3ff0
#define MVPP2_RXQ_STATUS_UPDATE_REG(rxq) (0x3000 + 4 * (rxq))
@@ -117,9 +121,6 @@
#define MVPP2_TXQ_DESC_SIZE_REG 0x2088
#define MVPP2_TXQ_DESC_SIZE_MASK 0x3ff0
#define MVPP2_AGGR_TXQ_UPDATE_REG 0x2090
-#define MVPP2_TXQ_THRESH_REG 0x2094
-#define MVPP2_TRANSMITTED_THRESH_OFFSET 16
-#define MVPP2_TRANSMITTED_THRESH_MASK 0x3fff0000
#define MVPP2_TXQ_INDEX_REG 0x2098
#define MVPP2_TXQ_PREF_BUF_REG 0x209c
#define MVPP2_PREF_BUF_PTR(desc) ((desc) & 0xfff)
@@ -140,6 +141,7 @@
#define MVPP2_TXQ_RSVD_CLR_REG 0x20b8
#define MVPP2_TXQ_RSVD_CLR_OFFSET 16
#define MVPP2_AGGR_TXQ_DESC_ADDR_REG(cpu) (0x2100 + 4 * (cpu))
+#define MVPP22_AGGR_TXQ_DESC_ADDR_OFFS 8
#define MVPP2_AGGR_TXQ_DESC_SIZE_REG(cpu) (0x2140 + 4 * (cpu))
#define MVPP2_AGGR_TXQ_DESC_SIZE_MASK 0x3ff0
#define MVPP2_AGGR_TXQ_STATUS_REG(cpu) (0x2180 + 4 * (cpu))
@@ -152,10 +154,52 @@
#define MVPP2_WIN_REMAP(w) (0x4040 + ((w) << 2))
#define MVPP2_BASE_ADDR_ENABLE 0x4060
+/* AXI Bridge Registers */
+#define MVPP22_AXI_BM_WR_ATTR_REG 0x4100
+#define MVPP22_AXI_BM_RD_ATTR_REG 0x4104
+#define MVPP22_AXI_AGGRQ_DESCR_RD_ATTR_REG 0x4110
+#define MVPP22_AXI_TXQ_DESCR_WR_ATTR_REG 0x4114
+#define MVPP22_AXI_TXQ_DESCR_RD_ATTR_REG 0x4118
+#define MVPP22_AXI_RXQ_DESCR_WR_ATTR_REG 0x411c
+#define MVPP22_AXI_RX_DATA_WR_ATTR_REG 0x4120
+#define MVPP22_AXI_TX_DATA_RD_ATTR_REG 0x4130
+#define MVPP22_AXI_RD_NORMAL_CODE_REG 0x4150
+#define MVPP22_AXI_RD_SNOOP_CODE_REG 0x4154
+#define MVPP22_AXI_WR_NORMAL_CODE_REG 0x4160
+#define MVPP22_AXI_WR_SNOOP_CODE_REG 0x4164
+
+/* Values for AXI Bridge registers */
+#define MVPP22_AXI_ATTR_CACHE_OFFS 0
+#define MVPP22_AXI_ATTR_DOMAIN_OFFS 12
+
+#define MVPP22_AXI_CODE_CACHE_OFFS 0
+#define MVPP22_AXI_CODE_DOMAIN_OFFS 4
+
+#define MVPP22_AXI_CODE_CACHE_NON_CACHE 0x3
+#define MVPP22_AXI_CODE_CACHE_WR_CACHE 0x7
+#define MVPP22_AXI_CODE_CACHE_RD_CACHE 0xb
+
+#define MVPP22_AXI_CODE_DOMAIN_OUTER_DOM 2
+#define MVPP22_AXI_CODE_DOMAIN_SYSTEM 3
+
/* Interrupt Cause and Mask registers */
#define MVPP2_ISR_RX_THRESHOLD_REG(rxq) (0x5200 + 4 * (rxq))
#define MVPP2_MAX_ISR_RX_THRESHOLD 0xfffff0
-#define MVPP2_ISR_RXQ_GROUP_REG(rxq) (0x5400 + 4 * (rxq))
+#define MVPP21_ISR_RXQ_GROUP_REG(rxq) (0x5400 + 4 * (rxq))
+
+#define MVPP22_ISR_RXQ_GROUP_INDEX_REG 0x5400
+#define MVPP22_ISR_RXQ_GROUP_INDEX_SUBGROUP_MASK 0xf
+#define MVPP22_ISR_RXQ_GROUP_INDEX_GROUP_MASK 0x380
+#define MVPP22_ISR_RXQ_GROUP_INDEX_GROUP_OFFSET 7
+
+#define MVPP22_ISR_RXQ_GROUP_INDEX_SUBGROUP_MASK 0xf
+#define MVPP22_ISR_RXQ_GROUP_INDEX_GROUP_MASK 0x380
+
+#define MVPP22_ISR_RXQ_SUB_GROUP_CONFIG_REG 0x5404
+#define MVPP22_ISR_RXQ_SUB_GROUP_STARTQ_MASK 0x1f
+#define MVPP22_ISR_RXQ_SUB_GROUP_SIZE_MASK 0xf00
+#define MVPP22_ISR_RXQ_SUB_GROUP_SIZE_OFFSET 8
+
#define MVPP2_ISR_ENABLE_REG(port) (0x5420 + 4 * (port))
#define MVPP2_ISR_ENABLE_INTERRUPT(mask) ((mask) & 0xffff)
#define MVPP2_ISR_DISABLE_INTERRUPT(mask) (((mask) << 16) & 0xffff0000)
@@ -210,14 +254,19 @@
#define MVPP2_BM_PHY_ALLOC_REG(pool) (0x6400 + ((pool) * 4))
#define MVPP2_BM_PHY_ALLOC_GRNTD_MASK BIT(0)
#define MVPP2_BM_VIRT_ALLOC_REG 0x6440
+#define MVPP22_BM_ADDR_HIGH_ALLOC 0x6444
+#define MVPP22_BM_ADDR_HIGH_PHYS_MASK 0xff
+#define MVPP22_BM_ADDR_HIGH_VIRT_MASK 0xff00
+#define MVPP22_BM_ADDR_HIGH_VIRT_SHIFT 8
#define MVPP2_BM_PHY_RLS_REG(pool) (0x6480 + ((pool) * 4))
#define MVPP2_BM_PHY_RLS_MC_BUFF_MASK BIT(0)
#define MVPP2_BM_PHY_RLS_PRIO_EN_MASK BIT(1)
#define MVPP2_BM_PHY_RLS_GRNTD_MASK BIT(2)
#define MVPP2_BM_VIRT_RLS_REG 0x64c0
-#define MVPP2_BM_MC_RLS_REG 0x64c4
-#define MVPP2_BM_MC_ID_MASK 0xfff
-#define MVPP2_BM_FORCE_RELEASE_MASK BIT(12)
+#define MVPP22_BM_ADDR_HIGH_RLS_REG 0x64c4
+#define MVPP22_BM_ADDR_HIGH_PHYS_RLS_MASK 0xff
+#define MVPP22_BM_ADDR_HIGH_VIRT_RLS_MASK 0xff00
+#define MVPP22_BM_ADDR_HIGH_VIRT_RLS_SHIFT 8
/* TX Scheduler registers */
#define MVPP2_TXP_SCHED_PORT_INDEX_REG 0x8000
@@ -287,6 +336,24 @@
#define MVPP2_GMAC_TX_FIFO_MIN_TH_ALL_MASK 0x1fc0
#define MVPP2_GMAC_TX_FIFO_MIN_TH_MASK(v) (((v) << 6) & \
MVPP2_GMAC_TX_FIFO_MIN_TH_ALL_MASK)
+#define MVPP22_GMAC_CTRL_4_REG 0x90
+#define MVPP22_CTRL4_EXT_PIN_GMII_SEL BIT(0)
+#define MVPP22_CTRL4_DP_CLK_SEL BIT(5)
+#define MVPP22_CTRL4_SYNC_BYPASS BIT(6)
+#define MVPP22_CTRL4_QSGMII_BYPASS_ACTIVE BIT(7)
+
+/* Per-port XGMAC registers. PPv2.2 only, only for GOP port 0,
+ * relative to port->base.
+ */
+#define MVPP22_XLG_CTRL3_REG 0x11c
+#define MVPP22_XLG_CTRL3_MACMODESELECT_MASK (7 << 13)
+#define MVPP22_XLG_CTRL3_MACMODESELECT_GMAC (0 << 13)
+
+/* SMI registers. PPv2.2 only, relative to priv->iface_base. */
+#define MVPP22_SMI_MISC_CFG_REG 0x1204
+#define MVPP22_SMI_POLLING_EN BIT(10)
+
+#define MVPP22_GMAC_BASE(port) (0x7000 + (port) * 0x1000 + 0xe00)
#define MVPP2_CAUSE_TXQ_SENT_DESC_ALL_MASK 0xff
@@ -335,15 +402,9 @@
/* Maximum number of TXQs used by single port */
#define MVPP2_MAX_TXQ 8
-/* Maximum number of RXQs used by single port */
-#define MVPP2_MAX_RXQ 8
-
/* Dfault number of RXQs in use */
#define MVPP2_DEFAULT_RXQ 4
-/* Total number of RXQs available to all ports */
-#define MVPP2_RXQ_TOTAL_NUM (MVPP2_MAX_PORTS * MVPP2_MAX_RXQ)
-
/* Max number of Rx descriptors */
#define MVPP2_MAX_RXD 128
@@ -615,6 +676,11 @@
*/
#define MVPP2_BM_SHORT_PKT_SIZE MVPP2_RX_MAX_PKT_SIZE(512)
+#define MVPP21_ADDR_SPACE_SZ 0
+#define MVPP22_ADDR_SPACE_SZ SZ_64K
+
+#define MVPP2_MAX_CPUS 4
+
enum mvpp2_bm_type {
MVPP2_BM_FREE,
MVPP2_BM_SWF_LONG,
@@ -626,12 +692,19 @@
/* Shared Packet Processor resources */
struct mvpp2 {
/* Shared registers' base addresses */
- void __iomem *base;
void __iomem *lms_base;
+ void __iomem *iface_base;
+
+ /* On PPv2.2, each CPU can access the base register through a
+ * separate address space, each 64 KB apart from each
+ * other.
+ */
+ void __iomem *cpu_base[MVPP2_MAX_CPUS];
/* Common clocks */
struct clk *pp_clk;
struct clk *gop_clk;
+ struct clk *mg_clk;
/* List of pointers to port structures */
struct mvpp2_port **port_list;
@@ -649,6 +722,12 @@
/* Tclk value */
u32 tclk;
+
+ /* HW version */
+ enum { MVPP21, MVPP22 } hw_version;
+
+ /* Maximum number of RXQs per port */
+ unsigned int max_port_rxqs;
};
struct mvpp2_pcpu_stats {
@@ -670,6 +749,11 @@
struct mvpp2_port {
u8 id;
+ /* Index of the port from the "group of ports" complex point
+ * of view
+ */
+ int gop_id;
+
int irq;
struct mvpp2 *priv;
@@ -741,22 +825,24 @@
#define MVPP2_RXD_L3_IP6 BIT(30)
#define MVPP2_RXD_BUF_HDR BIT(31)
-struct mvpp2_tx_desc {
+/* HW TX descriptor for PPv2.1 */
+struct mvpp21_tx_desc {
u32 command; /* Options used by HW for packet transmitting.*/
u8 packet_offset; /* the offset from the buffer beginning */
u8 phys_txq; /* destination queue ID */
u16 data_size; /* data size of transmitted packet in bytes */
- u32 buf_phys_addr; /* physical addr of transmitted buffer */
+ u32 buf_dma_addr; /* physical addr of transmitted buffer */
u32 buf_cookie; /* cookie for access to TX buffer in tx path */
u32 reserved1[3]; /* hw_cmd (for future use, BM, PON, PNC) */
u32 reserved2; /* reserved (for future use) */
};
-struct mvpp2_rx_desc {
+/* HW RX descriptor for PPv2.1 */
+struct mvpp21_rx_desc {
u32 status; /* info about received packet */
u16 reserved1; /* parser_info (for future use, PnC) */
u16 data_size; /* size of received packet in bytes */
- u32 buf_phys_addr; /* physical address of the buffer */
+ u32 buf_dma_addr; /* physical address of the buffer */
u32 buf_cookie; /* cookie for access to RX buffer in rx path */
u16 reserved2; /* gem_port_id (for future use, PON) */
u16 reserved3; /* csum_l4 (for future use, PnC) */
@@ -767,12 +853,51 @@
u32 reserved8;
};
+/* HW TX descriptor for PPv2.2 */
+struct mvpp22_tx_desc {
+ u32 command;
+ u8 packet_offset;
+ u8 phys_txq;
+ u16 data_size;
+ u64 reserved1;
+ u64 buf_dma_addr_ptp;
+ u64 buf_cookie_misc;
+};
+
+/* HW RX descriptor for PPv2.2 */
+struct mvpp22_rx_desc {
+ u32 status;
+ u16 reserved1;
+ u16 data_size;
+ u32 reserved2;
+ u32 reserved3;
+ u64 buf_dma_addr_key_hash;
+ u64 buf_cookie_misc;
+};
+
+/* Opaque type used by the driver to manipulate the HW TX and RX
+ * descriptors
+ */
+struct mvpp2_tx_desc {
+ union {
+ struct mvpp21_tx_desc pp21;
+ struct mvpp22_tx_desc pp22;
+ };
+};
+
+struct mvpp2_rx_desc {
+ union {
+ struct mvpp21_rx_desc pp21;
+ struct mvpp22_rx_desc pp22;
+ };
+};
+
struct mvpp2_txq_pcpu_buf {
/* Transmitted SKB */
struct sk_buff *skb;
/* Physical address of transmitted buffer */
- dma_addr_t phys;
+ dma_addr_t dma;
/* Size transmitted */
size_t size;
@@ -825,7 +950,7 @@
struct mvpp2_tx_desc *descs;
/* DMA address of the Tx DMA descriptors array */
- dma_addr_t descs_phys;
+ dma_addr_t descs_dma;
/* Index of the last Tx DMA descriptor */
int last_desc;
@@ -848,7 +973,7 @@
struct mvpp2_rx_desc *descs;
/* DMA address of the RX DMA descriptors array */
- dma_addr_t descs_phys;
+ dma_addr_t descs_dma;
/* Index of the last RX DMA descriptor */
int last_desc;
@@ -912,6 +1037,8 @@
/* Buffer Pointers Pool External (BPPE) size */
int size;
+ /* BPPE size in bytes */
+ int size_bytes;
/* Number of buffers for this pool */
int buf_num;
/* Pool buffer size */
@@ -922,29 +1049,13 @@
/* BPPE virtual base address */
u32 *virt_addr;
- /* BPPE physical base address */
- dma_addr_t phys_addr;
+ /* BPPE DMA base address */
+ dma_addr_t dma_addr;
/* Ports using BM pool */
u32 port_map;
};
-struct mvpp2_buff_hdr {
- u32 next_buff_phys_addr;
- u32 next_buff_virt_addr;
- u16 byte_count;
- u16 info;
- u8 reserved1; /* bm_qset (for future use, BM) */
-};
-
-/* Buffer header info bits */
-#define MVPP2_B_HDR_INFO_MC_ID_MASK 0xfff
-#define MVPP2_B_HDR_INFO_MC_ID(info) ((info) & MVPP2_B_HDR_INFO_MC_ID_MASK)
-#define MVPP2_B_HDR_INFO_LAST_OFFS 12
-#define MVPP2_B_HDR_INFO_LAST_MASK BIT(12)
-#define MVPP2_B_HDR_INFO_IS_LAST(info) \
- ((info & MVPP2_B_HDR_INFO_LAST_MASK) >> MVPP2_B_HDR_INFO_LAST_OFFS)
-
/* Static declaractions */
/* Number of RXQs used by single port */
@@ -959,12 +1070,177 @@
static void mvpp2_write(struct mvpp2 *priv, u32 offset, u32 data)
{
- writel(data, priv->base + offset);
+ writel(data, priv->cpu_base[0] + offset);
}
static u32 mvpp2_read(struct mvpp2 *priv, u32 offset)
{
- return readl(priv->base + offset);
+ return readl(priv->cpu_base[0] + offset);
+}
+
+/* These accessors should be used to access:
+ *
+ * - per-CPU registers, where each CPU has its own copy of the
+ * register.
+ *
+ * MVPP2_BM_VIRT_ALLOC_REG
+ * MVPP2_BM_ADDR_HIGH_ALLOC
+ * MVPP22_BM_ADDR_HIGH_RLS_REG
+ * MVPP2_BM_VIRT_RLS_REG
+ * MVPP2_ISR_RX_TX_CAUSE_REG
+ * MVPP2_ISR_RX_TX_MASK_REG
+ * MVPP2_TXQ_NUM_REG
+ * MVPP2_AGGR_TXQ_UPDATE_REG
+ * MVPP2_TXQ_RSVD_REQ_REG
+ * MVPP2_TXQ_RSVD_RSLT_REG
+ * MVPP2_TXQ_SENT_REG
+ * MVPP2_RXQ_NUM_REG
+ *
+ * - global registers that must be accessed through a specific CPU
+ * window, because they are related to an access to a per-CPU
+ * register
+ *
+ * MVPP2_BM_PHY_ALLOC_REG (related to MVPP2_BM_VIRT_ALLOC_REG)
+ * MVPP2_BM_PHY_RLS_REG (related to MVPP2_BM_VIRT_RLS_REG)
+ * MVPP2_RXQ_THRESH_REG (related to MVPP2_RXQ_NUM_REG)
+ * MVPP2_RXQ_DESC_ADDR_REG (related to MVPP2_RXQ_NUM_REG)
+ * MVPP2_RXQ_DESC_SIZE_REG (related to MVPP2_RXQ_NUM_REG)
+ * MVPP2_RXQ_INDEX_REG (related to MVPP2_RXQ_NUM_REG)
+ * MVPP2_TXQ_PENDING_REG (related to MVPP2_TXQ_NUM_REG)
+ * MVPP2_TXQ_DESC_ADDR_REG (related to MVPP2_TXQ_NUM_REG)
+ * MVPP2_TXQ_DESC_SIZE_REG (related to MVPP2_TXQ_NUM_REG)
+ * MVPP2_TXQ_INDEX_REG (related to MVPP2_TXQ_NUM_REG)
+ * MVPP2_TXQ_PENDING_REG (related to MVPP2_TXQ_NUM_REG)
+ * MVPP2_TXQ_PREF_BUF_REG (related to MVPP2_TXQ_NUM_REG)
+ * MVPP2_TXQ_PREF_BUF_REG (related to MVPP2_TXQ_NUM_REG)
+ */
+static void mvpp2_percpu_write(struct mvpp2 *priv, int cpu,
+ u32 offset, u32 data)
+{
+ writel(data, priv->cpu_base[cpu] + offset);
+}
+
+static u32 mvpp2_percpu_read(struct mvpp2 *priv, int cpu,
+ u32 offset)
+{
+ return readl(priv->cpu_base[cpu] + offset);
+}
+
+static dma_addr_t mvpp2_txdesc_dma_addr_get(struct mvpp2_port *port,
+ struct mvpp2_tx_desc *tx_desc)
+{
+ if (port->priv->hw_version == MVPP21)
+ return tx_desc->pp21.buf_dma_addr;
+ else
+ return tx_desc->pp22.buf_dma_addr_ptp & GENMASK_ULL(40, 0);
+}
+
+static void mvpp2_txdesc_dma_addr_set(struct mvpp2_port *port,
+ struct mvpp2_tx_desc *tx_desc,
+ dma_addr_t dma_addr)
+{
+ if (port->priv->hw_version == MVPP21) {
+ tx_desc->pp21.buf_dma_addr = dma_addr;
+ } else {
+ u64 val = (u64)dma_addr;
+
+ tx_desc->pp22.buf_dma_addr_ptp &= ~GENMASK_ULL(40, 0);
+ tx_desc->pp22.buf_dma_addr_ptp |= val;
+ }
+}
+
+static size_t mvpp2_txdesc_size_get(struct mvpp2_port *port,
+ struct mvpp2_tx_desc *tx_desc)
+{
+ if (port->priv->hw_version == MVPP21)
+ return tx_desc->pp21.data_size;
+ else
+ return tx_desc->pp22.data_size;
+}
+
+static void mvpp2_txdesc_size_set(struct mvpp2_port *port,
+ struct mvpp2_tx_desc *tx_desc,
+ size_t size)
+{
+ if (port->priv->hw_version == MVPP21)
+ tx_desc->pp21.data_size = size;
+ else
+ tx_desc->pp22.data_size = size;
+}
+
+static void mvpp2_txdesc_txq_set(struct mvpp2_port *port,
+ struct mvpp2_tx_desc *tx_desc,
+ unsigned int txq)
+{
+ if (port->priv->hw_version == MVPP21)
+ tx_desc->pp21.phys_txq = txq;
+ else
+ tx_desc->pp22.phys_txq = txq;
+}
+
+static void mvpp2_txdesc_cmd_set(struct mvpp2_port *port,
+ struct mvpp2_tx_desc *tx_desc,
+ unsigned int command)
+{
+ if (port->priv->hw_version == MVPP21)
+ tx_desc->pp21.command = command;
+ else
+ tx_desc->pp22.command = command;
+}
+
+static void mvpp2_txdesc_offset_set(struct mvpp2_port *port,
+ struct mvpp2_tx_desc *tx_desc,
+ unsigned int offset)
+{
+ if (port->priv->hw_version == MVPP21)
+ tx_desc->pp21.packet_offset = offset;
+ else
+ tx_desc->pp22.packet_offset = offset;
+}
+
+static unsigned int mvpp2_txdesc_offset_get(struct mvpp2_port *port,
+ struct mvpp2_tx_desc *tx_desc)
+{
+ if (port->priv->hw_version == MVPP21)
+ return tx_desc->pp21.packet_offset;
+ else
+ return tx_desc->pp22.packet_offset;
+}
+
+static dma_addr_t mvpp2_rxdesc_dma_addr_get(struct mvpp2_port *port,
+ struct mvpp2_rx_desc *rx_desc)
+{
+ if (port->priv->hw_version == MVPP21)
+ return rx_desc->pp21.buf_dma_addr;
+ else
+ return rx_desc->pp22.buf_dma_addr_key_hash & GENMASK_ULL(40, 0);
+}
+
+static unsigned long mvpp2_rxdesc_cookie_get(struct mvpp2_port *port,
+ struct mvpp2_rx_desc *rx_desc)
+{
+ if (port->priv->hw_version == MVPP21)
+ return rx_desc->pp21.buf_cookie;
+ else
+ return rx_desc->pp22.buf_cookie_misc & GENMASK_ULL(40, 0);
+}
+
+static size_t mvpp2_rxdesc_size_get(struct mvpp2_port *port,
+ struct mvpp2_rx_desc *rx_desc)
+{
+ if (port->priv->hw_version == MVPP21)
+ return rx_desc->pp21.data_size;
+ else
+ return rx_desc->pp22.data_size;
+}
+
+static u32 mvpp2_rxdesc_status_get(struct mvpp2_port *port,
+ struct mvpp2_rx_desc *rx_desc)
+{
+ if (port->priv->hw_version == MVPP21)
+ return rx_desc->pp21.status;
+ else
+ return rx_desc->pp22.status;
}
static void mvpp2_txq_inc_get(struct mvpp2_txq_pcpu *txq_pcpu)
@@ -974,15 +1250,17 @@
txq_pcpu->txq_get_index = 0;
}
-static void mvpp2_txq_inc_put(struct mvpp2_txq_pcpu *txq_pcpu,
+static void mvpp2_txq_inc_put(struct mvpp2_port *port,
+ struct mvpp2_txq_pcpu *txq_pcpu,
struct sk_buff *skb,
struct mvpp2_tx_desc *tx_desc)
{
struct mvpp2_txq_pcpu_buf *tx_buf =
txq_pcpu->buffs + txq_pcpu->txq_put_index;
tx_buf->skb = skb;
- tx_buf->size = tx_desc->data_size;
- tx_buf->phys = tx_desc->buf_phys_addr + tx_desc->packet_offset;
+ tx_buf->size = mvpp2_txdesc_size_get(port, tx_desc);
+ tx_buf->dma = mvpp2_txdesc_dma_addr_get(port, tx_desc) +
+ mvpp2_txdesc_offset_get(port, tx_desc);
txq_pcpu->txq_put_index++;
if (txq_pcpu->txq_put_index == txq_pcpu->size)
txq_pcpu->txq_put_index = 0;
@@ -1411,7 +1689,7 @@
mvpp2_prs_hw_read(priv, &pe);
} else {
/* Entry doesn't exist - create new */
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_MAC);
pe.index = MVPP2_PE_DROP_ALL;
@@ -1448,7 +1726,7 @@
mvpp2_prs_hw_read(priv, &pe);
} else {
/* Entry doesn't exist - create new */
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_MAC);
pe.index = MVPP2_PE_MAC_PROMISCUOUS;
@@ -1494,7 +1772,7 @@
mvpp2_prs_hw_read(priv, &pe);
} else {
/* Entry doesn't exist - create new */
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_MAC);
pe.index = index;
@@ -1546,7 +1824,7 @@
mvpp2_prs_hw_read(priv, &pe);
} else {
/* Entry doesn't exist - create new */
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_DSA);
pe.index = tid;
@@ -1609,7 +1887,7 @@
mvpp2_prs_hw_read(priv, &pe);
} else {
/* Entry doesn't exist - create new */
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_DSA);
pe.index = tid;
@@ -1743,10 +2021,10 @@
if (tid <= tid_aux) {
ret = -EINVAL;
- goto error;
+ goto free_pe;
}
- memset(pe, 0 , sizeof(struct mvpp2_prs_entry));
+ memset(pe, 0, sizeof(*pe));
mvpp2_prs_tcam_lu_set(pe, MVPP2_PRS_LU_VLAN);
pe->index = tid;
@@ -1775,8 +2053,7 @@
mvpp2_prs_tcam_port_map_set(pe, port_map);
mvpp2_prs_hw_write(priv, pe);
-
-error:
+free_pe:
kfree(pe);
return ret;
@@ -1861,7 +2138,7 @@
ai = mvpp2_prs_double_vlan_ai_free_get(priv);
if (ai < 0) {
ret = ai;
- goto error;
+ goto free_pe;
}
/* Get first single/triple vlan tid */
@@ -1884,10 +2161,10 @@
if (tid >= tid_aux) {
ret = -ERANGE;
- goto error;
+ goto free_pe;
}
- memset(pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(pe, 0, sizeof(*pe));
mvpp2_prs_tcam_lu_set(pe, MVPP2_PRS_LU_VLAN);
pe->index = tid;
@@ -1911,8 +2188,7 @@
/* Update ports' mask */
mvpp2_prs_tcam_port_map_set(pe, port_map);
mvpp2_prs_hw_write(priv, pe);
-
-error:
+free_pe:
kfree(pe);
return ret;
}
@@ -1934,7 +2210,7 @@
if (tid < 0)
return tid;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_IP4);
pe.index = tid;
@@ -1992,7 +2268,7 @@
if (tid < 0)
return tid;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_IP4);
pe.index = tid;
@@ -2048,7 +2324,7 @@
if (tid < 0)
return tid;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_IP6);
pe.index = tid;
@@ -2087,7 +2363,7 @@
if (tid < 0)
return tid;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_IP6);
pe.index = tid;
@@ -2147,7 +2423,7 @@
int port;
for (port = 0; port < MVPP2_MAX_PORTS; port++) {
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_FLOWS);
pe.index = MVPP2_PE_FIRST_DEFAULT_FLOW - port;
@@ -2169,7 +2445,7 @@
{
struct mvpp2_prs_entry pe;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
pe.index = MVPP2_PE_MH_DEFAULT;
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_MH);
@@ -2192,7 +2468,7 @@
{
struct mvpp2_prs_entry pe;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
/* Non-promiscuous mode for all ports - DROP unknown packets */
pe.index = MVPP2_PE_MAC_NON_PROMISCUOUS;
@@ -2253,7 +2529,7 @@
MVPP2_PRS_TAGGED, MVPP2_PRS_DSA);
/* Set default entry, in case DSA or EDSA tag not found */
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_DSA);
pe.index = MVPP2_PE_DSA_DEFAULT;
mvpp2_prs_sram_next_lu_set(&pe, MVPP2_PRS_LU_VLAN);
@@ -2283,7 +2559,7 @@
if (tid < 0)
return tid;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_L2);
pe.index = tid;
@@ -2309,7 +2585,7 @@
if (tid < 0)
return tid;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_L2);
pe.index = tid;
@@ -2339,7 +2615,7 @@
if (tid < 0)
return tid;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_L2);
pe.index = tid;
@@ -2373,7 +2649,7 @@
if (tid < 0)
return tid;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_L2);
pe.index = tid;
@@ -2438,7 +2714,7 @@
if (tid < 0)
return tid;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_L2);
pe.index = tid;
@@ -2535,7 +2811,7 @@
return err;
/* Set default double vlan entry */
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_VLAN);
pe.index = MVPP2_PE_VLAN_DBL;
@@ -2555,7 +2831,7 @@
mvpp2_prs_hw_write(priv, &pe);
/* Set default vlan none entry */
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_VLAN);
pe.index = MVPP2_PE_VLAN_NONE;
@@ -2585,7 +2861,7 @@
if (tid < 0)
return tid;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_PPPOE);
pe.index = tid;
@@ -2635,7 +2911,7 @@
if (tid < 0)
return tid;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_PPPOE);
pe.index = tid;
@@ -2662,7 +2938,7 @@
if (tid < 0)
return tid;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_PPPOE);
pe.index = tid;
@@ -2720,7 +2996,7 @@
return err;
/* Default IPv4 entry for unknown protocols */
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_IP4);
pe.index = MVPP2_PE_IP4_PROTO_UN;
@@ -2745,7 +3021,7 @@
mvpp2_prs_hw_write(priv, &pe);
/* Default IPv4 entry for unicast address */
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_IP4);
pe.index = MVPP2_PE_IP4_ADDR_UN;
@@ -2813,7 +3089,7 @@
if (tid < 0)
return tid;
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_IP6);
pe.index = tid;
@@ -2834,7 +3110,7 @@
mvpp2_prs_hw_write(priv, &pe);
/* Default IPv6 entry for unknown protocols */
- memset(&pe, 0, sizeof(struct mvpp2_prs_entry));
+ memset(&pe, 0, sizeof(pe));
mvpp2_prs_tcam_lu_set(&pe, MVPP2_PRS_LU_IP6);
pe.index = MVPP2_PE_IP6_PROTO_UN;
@@ -2927,7 +3203,7 @@
mvpp2_prs_hw_inv(priv, index);
priv->prs_shadow = devm_kcalloc(&pdev->dev, MVPP2_PRS_TCAM_SRAM_SIZE,
- sizeof(struct mvpp2_prs_shadow),
+ sizeof(*priv->prs_shadow),
GFP_KERNEL);
if (!priv->prs_shadow)
return -ENOMEM;
@@ -3378,27 +3654,39 @@
struct mvpp2 *priv,
struct mvpp2_bm_pool *bm_pool, int size)
{
- int size_bytes;
u32 val;
- size_bytes = sizeof(u32) * size;
- bm_pool->virt_addr = dma_alloc_coherent(&pdev->dev, size_bytes,
- &bm_pool->phys_addr,
+ /* Number of buffer pointers must be a multiple of 16, as per
+ * hardware constraints
+ */
+ if (!IS_ALIGNED(size, 16))
+ return -EINVAL;
+
+ /* PPv2.1 needs 8 bytes per buffer pointer, PPv2.2 needs 16
+ * bytes per buffer pointer
+ */
+ if (priv->hw_version == MVPP21)
+ bm_pool->size_bytes = 2 * sizeof(u32) * size;
+ else
+ bm_pool->size_bytes = 2 * sizeof(u64) * size;
+
+ bm_pool->virt_addr = dma_alloc_coherent(&pdev->dev, bm_pool->size_bytes,
+ &bm_pool->dma_addr,
GFP_KERNEL);
if (!bm_pool->virt_addr)
return -ENOMEM;
if (!IS_ALIGNED((unsigned long)bm_pool->virt_addr,
MVPP2_BM_POOL_PTR_ALIGN)) {
- dma_free_coherent(&pdev->dev, size_bytes, bm_pool->virt_addr,
- bm_pool->phys_addr);
+ dma_free_coherent(&pdev->dev, bm_pool->size_bytes,
+ bm_pool->virt_addr, bm_pool->dma_addr);
dev_err(&pdev->dev, "BM pool %d is not %d bytes aligned\n",
bm_pool->id, MVPP2_BM_POOL_PTR_ALIGN);
return -ENOMEM;
}
mvpp2_write(priv, MVPP2_BM_POOL_BASE_REG(bm_pool->id),
- bm_pool->phys_addr);
+ lower_32_bits(bm_pool->dma_addr));
mvpp2_write(priv, MVPP2_BM_POOL_SIZE_REG(bm_pool->id), size);
val = mvpp2_read(priv, MVPP2_BM_POOL_CTRL_REG(bm_pool->id));
@@ -3426,6 +3714,34 @@
mvpp2_write(priv, MVPP2_POOL_BUF_SIZE_REG(bm_pool->id), val);
}
+static void mvpp2_bm_bufs_get_addrs(struct device *dev, struct mvpp2 *priv,
+ struct mvpp2_bm_pool *bm_pool,
+ dma_addr_t *dma_addr,
+ phys_addr_t *phys_addr)
+{
+ int cpu = smp_processor_id();
+
+ *dma_addr = mvpp2_percpu_read(priv, cpu,
+ MVPP2_BM_PHY_ALLOC_REG(bm_pool->id));
+ *phys_addr = mvpp2_percpu_read(priv, cpu, MVPP2_BM_VIRT_ALLOC_REG);
+
+ if (priv->hw_version == MVPP22) {
+ u32 val;
+ u32 dma_addr_highbits, phys_addr_highbits;
+
+ val = mvpp2_percpu_read(priv, cpu, MVPP22_BM_ADDR_HIGH_ALLOC);
+ dma_addr_highbits = (val & MVPP22_BM_ADDR_HIGH_PHYS_MASK);
+ phys_addr_highbits = (val & MVPP22_BM_ADDR_HIGH_VIRT_MASK) >>
+ MVPP22_BM_ADDR_HIGH_VIRT_SHIFT;
+
+ if (sizeof(dma_addr_t) == 8)
+ *dma_addr |= (u64)dma_addr_highbits << 32;
+
+ if (sizeof(phys_addr_t) == 8)
+ *phys_addr |= (u64)phys_addr_highbits << 32;
+ }
+}
+
/* Free all buffers from the pool */
static void mvpp2_bm_bufs_free(struct device *dev, struct mvpp2 *priv,
struct mvpp2_bm_pool *bm_pool)
@@ -3433,21 +3749,21 @@
int i;
for (i = 0; i < bm_pool->buf_num; i++) {
- dma_addr_t buf_phys_addr;
- unsigned long vaddr;
+ dma_addr_t buf_dma_addr;
+ phys_addr_t buf_phys_addr;
+ void *data;
- /* Get buffer virtual address (indirect access) */
- buf_phys_addr = mvpp2_read(priv,
- MVPP2_BM_PHY_ALLOC_REG(bm_pool->id));
- vaddr = mvpp2_read(priv, MVPP2_BM_VIRT_ALLOC_REG);
+ mvpp2_bm_bufs_get_addrs(dev, priv, bm_pool,
+ &buf_dma_addr, &buf_phys_addr);
- dma_unmap_single(dev, buf_phys_addr,
+ dma_unmap_single(dev, buf_dma_addr,
bm_pool->buf_size, DMA_FROM_DEVICE);
- if (!vaddr)
+ data = (void *)phys_to_virt(buf_phys_addr);
+ if (!data)
break;
- mvpp2_frag_free(bm_pool, (void *)vaddr);
+ mvpp2_frag_free(bm_pool, data);
}
/* Update BM driver with number of buffers removed from pool */
@@ -3471,9 +3787,9 @@
val |= MVPP2_BM_STOP_MASK;
mvpp2_write(priv, MVPP2_BM_POOL_CTRL_REG(bm_pool->id), val);
- dma_free_coherent(&pdev->dev, sizeof(u32) * bm_pool->size,
+ dma_free_coherent(&pdev->dev, bm_pool->size_bytes,
bm_pool->virt_addr,
- bm_pool->phys_addr);
+ bm_pool->dma_addr);
return 0;
}
@@ -3515,7 +3831,7 @@
/* Allocate and initialize BM pools */
priv->bm_pools = devm_kcalloc(&pdev->dev, MVPP2_BM_POOLS_NUM,
- sizeof(struct mvpp2_bm_pool), GFP_KERNEL);
+ sizeof(*priv->bm_pools), GFP_KERNEL);
if (!priv->bm_pools)
return -ENOMEM;
@@ -3529,17 +3845,20 @@
static void mvpp2_rxq_long_pool_set(struct mvpp2_port *port,
int lrxq, int long_pool)
{
- u32 val;
+ u32 val, mask;
int prxq;
/* Get queue physical ID */
prxq = port->rxqs[lrxq]->id;
- val = mvpp2_read(port->priv, MVPP2_RXQ_CONFIG_REG(prxq));
- val &= ~MVPP2_RXQ_POOL_LONG_MASK;
- val |= ((long_pool << MVPP2_RXQ_POOL_LONG_OFFS) &
- MVPP2_RXQ_POOL_LONG_MASK);
+ if (port->priv->hw_version == MVPP21)
+ mask = MVPP21_RXQ_POOL_LONG_MASK;
+ else
+ mask = MVPP22_RXQ_POOL_LONG_MASK;
+ val = mvpp2_read(port->priv, MVPP2_RXQ_CONFIG_REG(prxq));
+ val &= ~mask;
+ val |= (long_pool << MVPP2_RXQ_POOL_LONG_OFFS) & mask;
mvpp2_write(port->priv, MVPP2_RXQ_CONFIG_REG(prxq), val);
}
@@ -3547,40 +3866,45 @@
static void mvpp2_rxq_short_pool_set(struct mvpp2_port *port,
int lrxq, int short_pool)
{
- u32 val;
+ u32 val, mask;
int prxq;
/* Get queue physical ID */
prxq = port->rxqs[lrxq]->id;
- val = mvpp2_read(port->priv, MVPP2_RXQ_CONFIG_REG(prxq));
- val &= ~MVPP2_RXQ_POOL_SHORT_MASK;
- val |= ((short_pool << MVPP2_RXQ_POOL_SHORT_OFFS) &
- MVPP2_RXQ_POOL_SHORT_MASK);
+ if (port->priv->hw_version == MVPP21)
+ mask = MVPP21_RXQ_POOL_SHORT_MASK;
+ else
+ mask = MVPP22_RXQ_POOL_SHORT_MASK;
+ val = mvpp2_read(port->priv, MVPP2_RXQ_CONFIG_REG(prxq));
+ val &= ~mask;
+ val |= (short_pool << MVPP2_RXQ_POOL_SHORT_OFFS) & mask;
mvpp2_write(port->priv, MVPP2_RXQ_CONFIG_REG(prxq), val);
}
static void *mvpp2_buf_alloc(struct mvpp2_port *port,
struct mvpp2_bm_pool *bm_pool,
- dma_addr_t *buf_phys_addr,
+ dma_addr_t *buf_dma_addr,
+ phys_addr_t *buf_phys_addr,
gfp_t gfp_mask)
{
- dma_addr_t phys_addr;
+ dma_addr_t dma_addr;
void *data;
data = mvpp2_frag_alloc(bm_pool);
if (!data)
return NULL;
- phys_addr = dma_map_single(port->dev->dev.parent, data,
- MVPP2_RX_BUF_SIZE(bm_pool->pkt_size),
- DMA_FROM_DEVICE);
- if (unlikely(dma_mapping_error(port->dev->dev.parent, phys_addr))) {
+ dma_addr = dma_map_single(port->dev->dev.parent, data,
+ MVPP2_RX_BUF_SIZE(bm_pool->pkt_size),
+ DMA_FROM_DEVICE);
+ if (unlikely(dma_mapping_error(port->dev->dev.parent, dma_addr))) {
mvpp2_frag_free(bm_pool, data);
return NULL;
}
- *buf_phys_addr = phys_addr;
+ *buf_dma_addr = dma_addr;
+ *buf_phys_addr = virt_to_phys(data);
return data;
}
@@ -3604,37 +3928,46 @@
/* Release buffer to BM */
static inline void mvpp2_bm_pool_put(struct mvpp2_port *port, int pool,
- dma_addr_t buf_phys_addr,
- unsigned long buf_virt_addr)
+ dma_addr_t buf_dma_addr,
+ phys_addr_t buf_phys_addr)
{
- mvpp2_write(port->priv, MVPP2_BM_VIRT_RLS_REG, buf_virt_addr);
- mvpp2_write(port->priv, MVPP2_BM_PHY_RLS_REG(pool), buf_phys_addr);
-}
+ int cpu = smp_processor_id();
-/* Release multicast buffer */
-static void mvpp2_bm_pool_mc_put(struct mvpp2_port *port, int pool,
- dma_addr_t buf_phys_addr,
- unsigned long buf_virt_addr,
- int mc_id)
-{
- u32 val = 0;
+ if (port->priv->hw_version == MVPP22) {
+ u32 val = 0;
- val |= (mc_id & MVPP2_BM_MC_ID_MASK);
- mvpp2_write(port->priv, MVPP2_BM_MC_RLS_REG, val);
+ if (sizeof(dma_addr_t) == 8)
+ val |= upper_32_bits(buf_dma_addr) &
+ MVPP22_BM_ADDR_HIGH_PHYS_RLS_MASK;
- mvpp2_bm_pool_put(port, pool,
- buf_phys_addr | MVPP2_BM_PHY_RLS_MC_BUFF_MASK,
- buf_virt_addr);
+ if (sizeof(phys_addr_t) == 8)
+ val |= (upper_32_bits(buf_phys_addr)
+ << MVPP22_BM_ADDR_HIGH_VIRT_RLS_SHIFT) &
+ MVPP22_BM_ADDR_HIGH_VIRT_RLS_MASK;
+
+ mvpp2_percpu_write(port->priv, cpu,
+ MVPP22_BM_ADDR_HIGH_RLS_REG, val);
+ }
+
+ /* MVPP2_BM_VIRT_RLS_REG is not interpreted by HW, and simply
+ * returned in the "cookie" field of the RX
+ * descriptor. Instead of storing the virtual address, we
+ * store the physical address
+ */
+ mvpp2_percpu_write(port->priv, cpu,
+ MVPP2_BM_VIRT_RLS_REG, buf_phys_addr);
+ mvpp2_percpu_write(port->priv, cpu,
+ MVPP2_BM_PHY_RLS_REG(pool), buf_dma_addr);
}
/* Refill BM pool */
static void mvpp2_pool_refill(struct mvpp2_port *port, u32 bm,
- dma_addr_t phys_addr,
- unsigned long cookie)
+ dma_addr_t dma_addr,
+ phys_addr_t phys_addr)
{
int pool = mvpp2_bm_cookie_pool_get(bm);
- mvpp2_bm_pool_put(port, pool, phys_addr, cookie);
+ mvpp2_bm_pool_put(port, pool, dma_addr, phys_addr);
}
/* Allocate buffers for the pool */
@@ -3642,7 +3975,8 @@
struct mvpp2_bm_pool *bm_pool, int buf_num)
{
int i, buf_size, total_size;
- dma_addr_t phys_addr;
+ dma_addr_t dma_addr;
+ phys_addr_t phys_addr;
void *buf;
buf_size = MVPP2_RX_BUF_SIZE(bm_pool->pkt_size);
@@ -3657,12 +3991,13 @@
}
for (i = 0; i < buf_num; i++) {
- buf = mvpp2_buf_alloc(port, bm_pool, &phys_addr, GFP_KERNEL);
+ buf = mvpp2_buf_alloc(port, bm_pool, &dma_addr,
+ &phys_addr, GFP_KERNEL);
if (!buf)
break;
- mvpp2_bm_pool_put(port, bm_pool->id, phys_addr,
- (unsigned long)buf);
+ mvpp2_bm_pool_put(port, bm_pool->id, dma_addr,
+ phys_addr);
}
/* Update BM driver with number of buffers added to pool */
@@ -3830,7 +4165,8 @@
{
struct mvpp2_port *port = arg;
- mvpp2_write(port->priv, MVPP2_ISR_RX_TX_MASK_REG(port->id), 0);
+ mvpp2_percpu_write(port->priv, smp_processor_id(),
+ MVPP2_ISR_RX_TX_MASK_REG(port->id), 0);
}
/* Unmask the current CPU's Rx/Tx interrupts */
@@ -3838,17 +4174,46 @@
{
struct mvpp2_port *port = arg;
- mvpp2_write(port->priv, MVPP2_ISR_RX_TX_MASK_REG(port->id),
- (MVPP2_CAUSE_MISC_SUM_MASK |
- MVPP2_CAUSE_RXQ_OCCUP_DESC_ALL_MASK));
+ mvpp2_percpu_write(port->priv, smp_processor_id(),
+ MVPP2_ISR_RX_TX_MASK_REG(port->id),
+ (MVPP2_CAUSE_MISC_SUM_MASK |
+ MVPP2_CAUSE_RXQ_OCCUP_DESC_ALL_MASK));
}
/* Port configuration routines */
+static void mvpp22_port_mii_set(struct mvpp2_port *port)
+{
+ u32 val;
+
+ return;
+
+ /* Only GOP port 0 has an XLG MAC */
+ if (port->gop_id == 0) {
+ val = readl(port->base + MVPP22_XLG_CTRL3_REG);
+ val &= ~MVPP22_XLG_CTRL3_MACMODESELECT_MASK;
+ val |= MVPP22_XLG_CTRL3_MACMODESELECT_GMAC;
+ writel(val, port->base + MVPP22_XLG_CTRL3_REG);
+ }
+
+ val = readl(port->base + MVPP22_GMAC_CTRL_4_REG);
+ if (port->phy_interface == PHY_INTERFACE_MODE_RGMII)
+ val |= MVPP22_CTRL4_EXT_PIN_GMII_SEL;
+ else
+ val &= ~MVPP22_CTRL4_EXT_PIN_GMII_SEL;
+ val &= ~MVPP22_CTRL4_DP_CLK_SEL;
+ val |= MVPP22_CTRL4_SYNC_BYPASS;
+ val |= MVPP22_CTRL4_QSGMII_BYPASS_ACTIVE;
+ writel(val, port->base + MVPP22_GMAC_CTRL_4_REG);
+}
+
static void mvpp2_port_mii_set(struct mvpp2_port *port)
{
u32 val;
+ if (port->priv->hw_version == MVPP22)
+ mvpp22_port_mii_set(port);
+
val = readl(port->base + MVPP2_GMAC_CTRL_2_REG);
switch (port->phy_interface) {
@@ -3952,16 +4317,18 @@
{
int tx_port_num, val, queue, ptxq, lrxq;
- /* Configure port to loopback if needed */
- if (port->flags & MVPP2_F_LOOPBACK)
- mvpp2_port_loopback_set(port);
+ if (port->priv->hw_version == MVPP21) {
+ /* Configure port to loopback if needed */
+ if (port->flags & MVPP2_F_LOOPBACK)
+ mvpp2_port_loopback_set(port);
- /* Update TX FIFO MIN Threshold */
- val = readl(port->base + MVPP2_GMAC_PORT_FIFO_CFG_1_REG);
- val &= ~MVPP2_GMAC_TX_FIFO_MIN_TH_ALL_MASK;
- /* Min. TX threshold must be less than minimal packet length */
- val |= MVPP2_GMAC_TX_FIFO_MIN_TH_MASK(64 - 4 - 2);
- writel(val, port->base + MVPP2_GMAC_PORT_FIFO_CFG_1_REG);
+ /* Update TX FIFO MIN Threshold */
+ val = readl(port->base + MVPP2_GMAC_PORT_FIFO_CFG_1_REG);
+ val &= ~MVPP2_GMAC_TX_FIFO_MIN_TH_ALL_MASK;
+ /* Min. TX threshold must be less than minimal packet length */
+ val |= MVPP2_GMAC_TX_FIFO_MIN_TH_MASK(64 - 4 - 2);
+ writel(val, port->base + MVPP2_GMAC_PORT_FIFO_CFG_1_REG);
+ }
/* Disable Legacy WRR, Disable EJP, Release from reset */
tx_port_num = mvpp2_egress_port(port);
@@ -4048,7 +4415,7 @@
for (queue = 0; queue < txq_number; queue++) {
struct mvpp2_tx_queue *txq = port->txqs[queue];
- if (txq->descs != NULL)
+ if (txq->descs)
qmap |= (1 << queue);
}
@@ -4149,11 +4516,15 @@
}
/* Obtain BM cookie information from descriptor */
-static u32 mvpp2_bm_cookie_build(struct mvpp2_rx_desc *rx_desc)
+static u32 mvpp2_bm_cookie_build(struct mvpp2_port *port,
+ struct mvpp2_rx_desc *rx_desc)
{
- int pool = (rx_desc->status & MVPP2_RXD_BM_POOL_ID_MASK) >>
- MVPP2_RXD_BM_POOL_ID_OFFS;
int cpu = smp_processor_id();
+ int pool;
+
+ pool = (mvpp2_rxdesc_status_get(port, rx_desc) &
+ MVPP2_RXD_BM_POOL_ID_MASK) >>
+ MVPP2_RXD_BM_POOL_ID_OFFS;
return ((pool & 0xFF) << MVPP2_BM_COOKIE_POOL_OFFS) |
((cpu & 0xFF) << MVPP2_BM_COOKIE_CPU_OFFS);
@@ -4161,18 +4532,6 @@
/* Tx descriptors helper methods */
-/* Get number of Tx descriptors waiting to be transmitted by HW */
-static int mvpp2_txq_pend_desc_num_get(struct mvpp2_port *port,
- struct mvpp2_tx_queue *txq)
-{
- u32 val;
-
- mvpp2_write(port->priv, MVPP2_TXQ_NUM_REG, txq->id);
- val = mvpp2_read(port->priv, MVPP2_TXQ_PENDING_REG);
-
- return val & MVPP2_TXQ_PENDING_MASK;
-}
-
/* Get pointer to next Tx descriptor to be processed (send) by HW */
static struct mvpp2_tx_desc *
mvpp2_txq_next_desc_get(struct mvpp2_tx_queue *txq)
@@ -4187,7 +4546,8 @@
static void mvpp2_aggr_txq_pend_desc_add(struct mvpp2_port *port, int pending)
{
/* aggregated access - relevant TXQ number is written in TX desc */
- mvpp2_write(port->priv, MVPP2_AGGR_TXQ_UPDATE_REG, pending);
+ mvpp2_percpu_write(port->priv, smp_processor_id(),
+ MVPP2_AGGR_TXQ_UPDATE_REG, pending);
}
@@ -4216,11 +4576,12 @@
struct mvpp2_tx_queue *txq, int num)
{
u32 val;
+ int cpu = smp_processor_id();
val = (txq->id << MVPP2_TXQ_RSVD_REQ_Q_OFFSET) | num;
- mvpp2_write(priv, MVPP2_TXQ_RSVD_REQ_REG, val);
+ mvpp2_percpu_write(priv, cpu, MVPP2_TXQ_RSVD_REQ_REG, val);
- val = mvpp2_read(priv, MVPP2_TXQ_RSVD_RSLT_REG);
+ val = mvpp2_percpu_read(priv, cpu, MVPP2_TXQ_RSVD_RSLT_REG);
return val & MVPP2_TXQ_RSVD_RSLT_MASK;
}
@@ -4321,7 +4682,8 @@
u32 val;
/* Reading status reg resets transmitted descriptor counter */
- val = mvpp2_read(port->priv, MVPP2_TXQ_SENT_REG(txq->id));
+ val = mvpp2_percpu_read(port->priv, smp_processor_id(),
+ MVPP2_TXQ_SENT_REG(txq->id));
return (val & MVPP2_TRANSMITTED_COUNT_MASK) >>
MVPP2_TRANSMITTED_COUNT_OFFSET;
@@ -4335,7 +4697,8 @@
for (queue = 0; queue < txq_number; queue++) {
int id = port->txqs[queue]->id;
- mvpp2_read(port->priv, MVPP2_TXQ_SENT_REG(id));
+ mvpp2_percpu_read(port->priv, smp_processor_id(),
+ MVPP2_TXQ_SENT_REG(id));
}
}
@@ -4394,12 +4757,14 @@
static void mvpp2_rx_pkts_coal_set(struct mvpp2_port *port,
struct mvpp2_rx_queue *rxq)
{
+ int cpu = smp_processor_id();
+
if (rxq->pkts_coal > MVPP2_OCCUPIED_THRESH_MASK)
rxq->pkts_coal = MVPP2_OCCUPIED_THRESH_MASK;
- mvpp2_write(port->priv, MVPP2_RXQ_NUM_REG, rxq->id);
- mvpp2_write(port->priv, MVPP2_RXQ_THRESH_REG,
- rxq->pkts_coal);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_RXQ_NUM_REG, rxq->id);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_RXQ_THRESH_REG,
+ rxq->pkts_coal);
}
static u32 mvpp2_usec_to_cycles(u32 usec, unsigned long clk_hz)
@@ -4449,7 +4814,7 @@
struct mvpp2_txq_pcpu_buf *tx_buf =
txq_pcpu->buffs + txq_pcpu->txq_get_index;
- dma_unmap_single(port->dev->dev.parent, tx_buf->phys,
+ dma_unmap_single(port->dev->dev.parent, tx_buf->dma,
tx_buf->size, DMA_TO_DEVICE);
if (tx_buf->skb)
dev_kfree_skb_any(tx_buf->skb);
@@ -4527,10 +4892,12 @@
int desc_num, int cpu,
struct mvpp2 *priv)
{
+ u32 txq_dma;
+
/* Allocate memory for TX descriptors */
aggr_txq->descs = dma_alloc_coherent(&pdev->dev,
desc_num * MVPP2_DESC_ALIGNED_SIZE,
- &aggr_txq->descs_phys, GFP_KERNEL);
+ &aggr_txq->descs_dma, GFP_KERNEL);
if (!aggr_txq->descs)
return -ENOMEM;
@@ -4540,10 +4907,16 @@
aggr_txq->next_desc_to_proc = mvpp2_read(priv,
MVPP2_AGGR_TXQ_INDEX_REG(cpu));
- /* Set Tx descriptors queue starting address */
- /* indirect access */
- mvpp2_write(priv, MVPP2_AGGR_TXQ_DESC_ADDR_REG(cpu),
- aggr_txq->descs_phys);
+ /* Set Tx descriptors queue starting address indirect
+ * access
+ */
+ if (priv->hw_version == MVPP21)
+ txq_dma = aggr_txq->descs_dma;
+ else
+ txq_dma = aggr_txq->descs_dma >>
+ MVPP22_AGGR_TXQ_DESC_ADDR_OFFS;
+
+ mvpp2_write(priv, MVPP2_AGGR_TXQ_DESC_ADDR_REG(cpu), txq_dma);
mvpp2_write(priv, MVPP2_AGGR_TXQ_DESC_SIZE_REG(cpu), desc_num);
return 0;
@@ -4554,12 +4927,15 @@
struct mvpp2_rx_queue *rxq)
{
+ u32 rxq_dma;
+ int cpu;
+
rxq->size = port->rx_ring_size;
/* Allocate memory for RX descriptors */
rxq->descs = dma_alloc_coherent(port->dev->dev.parent,
rxq->size * MVPP2_DESC_ALIGNED_SIZE,
- &rxq->descs_phys, GFP_KERNEL);
+ &rxq->descs_dma, GFP_KERNEL);
if (!rxq->descs)
return -ENOMEM;
@@ -4569,10 +4945,15 @@
mvpp2_write(port->priv, MVPP2_RXQ_STATUS_REG(rxq->id), 0);
/* Set Rx descriptors queue starting address - indirect access */
- mvpp2_write(port->priv, MVPP2_RXQ_NUM_REG, rxq->id);
- mvpp2_write(port->priv, MVPP2_RXQ_DESC_ADDR_REG, rxq->descs_phys);
- mvpp2_write(port->priv, MVPP2_RXQ_DESC_SIZE_REG, rxq->size);
- mvpp2_write(port->priv, MVPP2_RXQ_INDEX_REG, 0);
+ cpu = smp_processor_id();
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_RXQ_NUM_REG, rxq->id);
+ if (port->priv->hw_version == MVPP21)
+ rxq_dma = rxq->descs_dma;
+ else
+ rxq_dma = rxq->descs_dma >> MVPP22_DESC_ADDR_OFFS;
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_RXQ_DESC_ADDR_REG, rxq_dma);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_RXQ_DESC_SIZE_REG, rxq->size);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_RXQ_INDEX_REG, 0);
/* Set Offset */
mvpp2_rxq_offset_set(port, rxq->id, NET_SKB_PAD);
@@ -4599,10 +4980,11 @@
for (i = 0; i < rx_received; i++) {
struct mvpp2_rx_desc *rx_desc = mvpp2_rxq_next_desc_get(rxq);
- u32 bm = mvpp2_bm_cookie_build(rx_desc);
+ u32 bm = mvpp2_bm_cookie_build(port, rx_desc);
- mvpp2_pool_refill(port, bm, rx_desc->buf_phys_addr,
- rx_desc->buf_cookie);
+ mvpp2_pool_refill(port, bm,
+ mvpp2_rxdesc_dma_addr_get(port, rx_desc),
+ mvpp2_rxdesc_cookie_get(port, rx_desc));
}
mvpp2_rxq_status_update(port, rxq->id, rx_received, rx_received);
}
@@ -4611,26 +4993,29 @@
static void mvpp2_rxq_deinit(struct mvpp2_port *port,
struct mvpp2_rx_queue *rxq)
{
+ int cpu;
+
mvpp2_rxq_drop_pkts(port, rxq);
if (rxq->descs)
dma_free_coherent(port->dev->dev.parent,
rxq->size * MVPP2_DESC_ALIGNED_SIZE,
rxq->descs,
- rxq->descs_phys);
+ rxq->descs_dma);
rxq->descs = NULL;
rxq->last_desc = 0;
rxq->next_desc_to_proc = 0;
- rxq->descs_phys = 0;
+ rxq->descs_dma = 0;
/* Clear Rx descriptors queue starting address and size;
* free descriptor number
*/
mvpp2_write(port->priv, MVPP2_RXQ_STATUS_REG(rxq->id), 0);
- mvpp2_write(port->priv, MVPP2_RXQ_NUM_REG, rxq->id);
- mvpp2_write(port->priv, MVPP2_RXQ_DESC_ADDR_REG, 0);
- mvpp2_write(port->priv, MVPP2_RXQ_DESC_SIZE_REG, 0);
+ cpu = smp_processor_id();
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_RXQ_NUM_REG, rxq->id);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_RXQ_DESC_ADDR_REG, 0);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_RXQ_DESC_SIZE_REG, 0);
}
/* Create and initialize a Tx queue */
@@ -4646,23 +5031,25 @@
/* Allocate memory for Tx descriptors */
txq->descs = dma_alloc_coherent(port->dev->dev.parent,
txq->size * MVPP2_DESC_ALIGNED_SIZE,
- &txq->descs_phys, GFP_KERNEL);
+ &txq->descs_dma, GFP_KERNEL);
if (!txq->descs)
return -ENOMEM;
txq->last_desc = txq->size - 1;
/* Set Tx descriptors queue starting address - indirect access */
- mvpp2_write(port->priv, MVPP2_TXQ_NUM_REG, txq->id);
- mvpp2_write(port->priv, MVPP2_TXQ_DESC_ADDR_REG, txq->descs_phys);
- mvpp2_write(port->priv, MVPP2_TXQ_DESC_SIZE_REG, txq->size &
- MVPP2_TXQ_DESC_SIZE_MASK);
- mvpp2_write(port->priv, MVPP2_TXQ_INDEX_REG, 0);
- mvpp2_write(port->priv, MVPP2_TXQ_RSVD_CLR_REG,
- txq->id << MVPP2_TXQ_RSVD_CLR_OFFSET);
- val = mvpp2_read(port->priv, MVPP2_TXQ_PENDING_REG);
+ cpu = smp_processor_id();
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_TXQ_NUM_REG, txq->id);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_TXQ_DESC_ADDR_REG,
+ txq->descs_dma);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_TXQ_DESC_SIZE_REG,
+ txq->size & MVPP2_TXQ_DESC_SIZE_MASK);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_TXQ_INDEX_REG, 0);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_TXQ_RSVD_CLR_REG,
+ txq->id << MVPP2_TXQ_RSVD_CLR_OFFSET);
+ val = mvpp2_percpu_read(port->priv, cpu, MVPP2_TXQ_PENDING_REG);
val &= ~MVPP2_TXQ_PENDING_MASK;
- mvpp2_write(port->priv, MVPP2_TXQ_PENDING_REG, val);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_TXQ_PENDING_REG, val);
/* Calculate base address in prefetch buffer. We reserve 16 descriptors
* for each existing TXQ.
@@ -4673,9 +5060,9 @@
desc = (port->id * MVPP2_MAX_TXQ * desc_per_txq) +
(txq->log_id * desc_per_txq);
- mvpp2_write(port->priv, MVPP2_TXQ_PREF_BUF_REG,
- MVPP2_PREF_BUF_PTR(desc) | MVPP2_PREF_BUF_SIZE_16 |
- MVPP2_PREF_BUF_THRESH(desc_per_txq/2));
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_TXQ_PREF_BUF_REG,
+ MVPP2_PREF_BUF_PTR(desc) | MVPP2_PREF_BUF_SIZE_16 |
+ MVPP2_PREF_BUF_THRESH(desc_per_txq / 2));
/* WRR / EJP configuration - indirect access */
tx_port_num = mvpp2_egress_port(port);
@@ -4694,11 +5081,11 @@
for_each_present_cpu(cpu) {
txq_pcpu = per_cpu_ptr(txq->pcpu, cpu);
txq_pcpu->size = txq->size;
- txq_pcpu->buffs = kmalloc(txq_pcpu->size *
- sizeof(struct mvpp2_txq_pcpu_buf),
- GFP_KERNEL);
+ txq_pcpu->buffs = kmalloc_array(txq_pcpu->size,
+ sizeof(*txq_pcpu->buffs),
+ GFP_KERNEL);
if (!txq_pcpu->buffs)
- goto error;
+ goto cleanup;
txq_pcpu->count = 0;
txq_pcpu->reserved_num = 0;
@@ -4707,8 +5094,7 @@
}
return 0;
-
-error:
+cleanup:
for_each_present_cpu(cpu) {
txq_pcpu = per_cpu_ptr(txq->pcpu, cpu);
kfree(txq_pcpu->buffs);
@@ -4716,7 +5102,7 @@
dma_free_coherent(port->dev->dev.parent,
txq->size * MVPP2_DESC_ALIGNED_SIZE,
- txq->descs, txq->descs_phys);
+ txq->descs, txq->descs_dma);
return -ENOMEM;
}
@@ -4736,20 +5122,21 @@
if (txq->descs)
dma_free_coherent(port->dev->dev.parent,
txq->size * MVPP2_DESC_ALIGNED_SIZE,
- txq->descs, txq->descs_phys);
+ txq->descs, txq->descs_dma);
txq->descs = NULL;
txq->last_desc = 0;
txq->next_desc_to_proc = 0;
- txq->descs_phys = 0;
+ txq->descs_dma = 0;
/* Set minimum bandwidth for disabled TXQs */
mvpp2_write(port->priv, MVPP2_TXQ_SCHED_TOKEN_CNTR_REG(txq->id), 0);
/* Set Tx descriptors queue starting address and size */
- mvpp2_write(port->priv, MVPP2_TXQ_NUM_REG, txq->id);
- mvpp2_write(port->priv, MVPP2_TXQ_DESC_ADDR_REG, 0);
- mvpp2_write(port->priv, MVPP2_TXQ_DESC_SIZE_REG, 0);
+ cpu = smp_processor_id();
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_TXQ_NUM_REG, txq->id);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_TXQ_DESC_ADDR_REG, 0);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_TXQ_DESC_SIZE_REG, 0);
}
/* Cleanup Tx ports */
@@ -4759,10 +5146,11 @@
int delay, pending, cpu;
u32 val;
- mvpp2_write(port->priv, MVPP2_TXQ_NUM_REG, txq->id);
- val = mvpp2_read(port->priv, MVPP2_TXQ_PREF_BUF_REG);
+ cpu = smp_processor_id();
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_TXQ_NUM_REG, txq->id);
+ val = mvpp2_percpu_read(port->priv, cpu, MVPP2_TXQ_PREF_BUF_REG);
val |= MVPP2_TXQ_DRAIN_EN_MASK;
- mvpp2_write(port->priv, MVPP2_TXQ_PREF_BUF_REG, val);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_TXQ_PREF_BUF_REG, val);
/* The napi queue has been stopped so wait for all packets
* to be transmitted.
@@ -4778,11 +5166,13 @@
mdelay(1);
delay++;
- pending = mvpp2_txq_pend_desc_num_get(port, txq);
+ pending = mvpp2_percpu_read(port->priv, cpu,
+ MVPP2_TXQ_PENDING_REG);
+ pending &= MVPP2_TXQ_PENDING_MASK;
} while (pending);
val &= ~MVPP2_TXQ_DRAIN_EN_MASK;
- mvpp2_write(port->priv, MVPP2_TXQ_PREF_BUF_REG, val);
+ mvpp2_percpu_write(port->priv, cpu, MVPP2_TXQ_PREF_BUF_REG, val);
for_each_present_cpu(cpu) {
txq_pcpu = per_cpu_ptr(txq->pcpu, cpu);
@@ -4991,20 +5381,21 @@
static void mvpp2_rx_error(struct mvpp2_port *port,
struct mvpp2_rx_desc *rx_desc)
{
- u32 status = rx_desc->status;
+ u32 status = mvpp2_rxdesc_status_get(port, rx_desc);
+ size_t sz = mvpp2_rxdesc_size_get(port, rx_desc);
switch (status & MVPP2_RXD_ERR_CODE_MASK) {
case MVPP2_RXD_ERR_CRC:
- netdev_err(port->dev, "bad rx status %08x (crc error), size=%d\n",
- status, rx_desc->data_size);
+ netdev_err(port->dev, "bad rx status %08x (crc error), size=%zu\n",
+ status, sz);
break;
case MVPP2_RXD_ERR_OVERRUN:
- netdev_err(port->dev, "bad rx status %08x (overrun error), size=%d\n",
- status, rx_desc->data_size);
+ netdev_err(port->dev, "bad rx status %08x (overrun error), size=%zu\n",
+ status, sz);
break;
case MVPP2_RXD_ERR_RESOURCE:
- netdev_err(port->dev, "bad rx status %08x (resource error), size=%d\n",
- status, rx_desc->data_size);
+ netdev_err(port->dev, "bad rx status %08x (resource error), size=%zu\n",
+ status, sz);
break;
}
}
@@ -5031,15 +5422,17 @@
static int mvpp2_rx_refill(struct mvpp2_port *port,
struct mvpp2_bm_pool *bm_pool, u32 bm)
{
- dma_addr_t phys_addr;
+ dma_addr_t dma_addr;
+ phys_addr_t phys_addr;
void *buf;
/* No recycle or too many buffers are in use, so allocate a new skb */
- buf = mvpp2_buf_alloc(port, bm_pool, &phys_addr, GFP_ATOMIC);
+ buf = mvpp2_buf_alloc(port, bm_pool, &dma_addr, &phys_addr,
+ GFP_ATOMIC);
if (!buf)
return -ENOMEM;
- mvpp2_pool_refill(port, bm, phys_addr, (unsigned long)buf);
+ mvpp2_pool_refill(port, bm, dma_addr, phys_addr);
return 0;
}
@@ -5075,43 +5468,6 @@
return MVPP2_TXD_L4_CSUM_NOT | MVPP2_TXD_IP_CSUM_DISABLE;
}
-static void mvpp2_buff_hdr_rx(struct mvpp2_port *port,
- struct mvpp2_rx_desc *rx_desc)
-{
- struct mvpp2_buff_hdr *buff_hdr;
- struct sk_buff *skb;
- u32 rx_status = rx_desc->status;
- dma_addr_t buff_phys_addr;
- unsigned long buff_virt_addr;
- dma_addr_t buff_phys_addr_next;
- unsigned long buff_virt_addr_next;
- int mc_id;
- int pool_id;
-
- pool_id = (rx_status & MVPP2_RXD_BM_POOL_ID_MASK) >>
- MVPP2_RXD_BM_POOL_ID_OFFS;
- buff_phys_addr = rx_desc->buf_phys_addr;
- buff_virt_addr = rx_desc->buf_cookie;
-
- do {
- skb = (struct sk_buff *)buff_virt_addr;
- buff_hdr = (struct mvpp2_buff_hdr *)skb->head;
-
- mc_id = MVPP2_B_HDR_INFO_MC_ID(buff_hdr->info);
-
- buff_phys_addr_next = buff_hdr->next_buff_phys_addr;
- buff_virt_addr_next = buff_hdr->next_buff_virt_addr;
-
- /* Release buffer */
- mvpp2_bm_pool_mc_put(port, pool_id, buff_phys_addr,
- buff_virt_addr, mc_id);
-
- buff_phys_addr = buff_phys_addr_next;
- buff_virt_addr = buff_virt_addr_next;
-
- } while (!MVPP2_B_HDR_INFO_IS_LAST(buff_hdr->info));
-}
-
/* Main rx processing */
static int mvpp2_rx(struct mvpp2_port *port, int rx_todo,
struct mvpp2_rx_queue *rxq)
@@ -5132,25 +5488,23 @@
struct mvpp2_bm_pool *bm_pool;
struct sk_buff *skb;
unsigned int frag_size;
- dma_addr_t phys_addr;
+ dma_addr_t dma_addr;
+ phys_addr_t phys_addr;
u32 bm, rx_status;
int pool, rx_bytes, err;
void *data;
rx_done++;
- rx_status = rx_desc->status;
- rx_bytes = rx_desc->data_size - MVPP2_MH_SIZE;
- phys_addr = rx_desc->buf_phys_addr;
- data = (void *)(uintptr_t)rx_desc->buf_cookie;
+ rx_status = mvpp2_rxdesc_status_get(port, rx_desc);
+ rx_bytes = mvpp2_rxdesc_size_get(port, rx_desc);
+ rx_bytes -= MVPP2_MH_SIZE;
+ dma_addr = mvpp2_rxdesc_dma_addr_get(port, rx_desc);
+ phys_addr = mvpp2_rxdesc_cookie_get(port, rx_desc);
+ data = (void *)phys_to_virt(phys_addr);
- bm = mvpp2_bm_cookie_build(rx_desc);
+ bm = mvpp2_bm_cookie_build(port, rx_desc);
pool = mvpp2_bm_cookie_pool_get(bm);
bm_pool = &port->priv->bm_pools[pool];
- /* Check if buffer header is used */
- if (rx_status & MVPP2_RXD_BUF_HDR) {
- mvpp2_buff_hdr_rx(port, rx_desc);
- continue;
- }
/* In case of an error, release the requested buffer pointer
* to the Buffer Manager. This request process is controlled
@@ -5158,13 +5512,11 @@
* comprised by the RX descriptor.
*/
if (rx_status & MVPP2_RXD_ERR_SUMMARY) {
- err_drop_frame:
+err_drop_frame:
dev->stats.rx_errors++;
mvpp2_rx_error(port, rx_desc);
/* Return the buffer to the pool */
-
- mvpp2_pool_refill(port, bm, rx_desc->buf_phys_addr,
- rx_desc->buf_cookie);
+ mvpp2_pool_refill(port, bm, dma_addr, phys_addr);
continue;
}
@@ -5185,7 +5537,7 @@
goto err_drop_frame;
}
- dma_unmap_single(dev->dev.parent, phys_addr,
+ dma_unmap_single(dev->dev.parent, dma_addr,
bm_pool->buf_size, DMA_FROM_DEVICE);
rcvd_pkts++;
@@ -5216,11 +5568,15 @@
}
static inline void
-tx_desc_unmap_put(struct device *dev, struct mvpp2_tx_queue *txq,
+tx_desc_unmap_put(struct mvpp2_port *port, struct mvpp2_tx_queue *txq,
struct mvpp2_tx_desc *desc)
{
- dma_unmap_single(dev, desc->buf_phys_addr,
- desc->data_size, DMA_TO_DEVICE);
+ dma_addr_t buf_dma_addr =
+ mvpp2_txdesc_dma_addr_get(port, desc);
+ size_t buf_sz =
+ mvpp2_txdesc_size_get(port, desc);
+ dma_unmap_single(port->dev->dev.parent, buf_dma_addr,
+ buf_sz, DMA_TO_DEVICE);
mvpp2_txq_desc_put(txq);
}
@@ -5232,47 +5588,49 @@
struct mvpp2_txq_pcpu *txq_pcpu = this_cpu_ptr(txq->pcpu);
struct mvpp2_tx_desc *tx_desc;
int i;
- dma_addr_t buf_phys_addr;
+ dma_addr_t buf_dma_addr;
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
void *addr = page_address(frag->page.p) + frag->page_offset;
tx_desc = mvpp2_txq_next_desc_get(aggr_txq);
- tx_desc->phys_txq = txq->id;
- tx_desc->data_size = frag->size;
+ mvpp2_txdesc_txq_set(port, tx_desc, txq->id);
+ mvpp2_txdesc_size_set(port, tx_desc, frag->size);
- buf_phys_addr = dma_map_single(port->dev->dev.parent, addr,
- tx_desc->data_size,
+ buf_dma_addr = dma_map_single(port->dev->dev.parent, addr,
+ frag->size,
DMA_TO_DEVICE);
- if (dma_mapping_error(port->dev->dev.parent, buf_phys_addr)) {
+ if (dma_mapping_error(port->dev->dev.parent, buf_dma_addr)) {
mvpp2_txq_desc_put(txq);
- goto error;
+ goto cleanup;
}
- tx_desc->packet_offset = buf_phys_addr & MVPP2_TX_DESC_ALIGN;
- tx_desc->buf_phys_addr = buf_phys_addr & (~MVPP2_TX_DESC_ALIGN);
+ mvpp2_txdesc_offset_set(port, tx_desc,
+ buf_dma_addr & MVPP2_TX_DESC_ALIGN);
+ mvpp2_txdesc_dma_addr_set(port, tx_desc,
+ buf_dma_addr & ~MVPP2_TX_DESC_ALIGN);
if (i == (skb_shinfo(skb)->nr_frags - 1)) {
/* Last descriptor */
- tx_desc->command = MVPP2_TXD_L_DESC;
- mvpp2_txq_inc_put(txq_pcpu, skb, tx_desc);
+ mvpp2_txdesc_cmd_set(port, tx_desc,
+ MVPP2_TXD_L_DESC);
+ mvpp2_txq_inc_put(port, txq_pcpu, skb, tx_desc);
} else {
/* Descriptor in the middle: Not First, Not Last */
- tx_desc->command = 0;
- mvpp2_txq_inc_put(txq_pcpu, NULL, tx_desc);
+ mvpp2_txdesc_cmd_set(port, tx_desc, 0);
+ mvpp2_txq_inc_put(port, txq_pcpu, NULL, tx_desc);
}
}
return 0;
-
-error:
+cleanup:
/* Release all descriptors that were used to map fragments of
* this packet, as well as the corresponding DMA mappings
*/
for (i = i - 1; i >= 0; i--) {
tx_desc = txq->descs + i;
- tx_desc_unmap_put(port->dev->dev.parent, txq, tx_desc);
+ tx_desc_unmap_put(port, txq, tx_desc);
}
return -ENOMEM;
@@ -5285,7 +5643,7 @@
struct mvpp2_tx_queue *txq, *aggr_txq;
struct mvpp2_txq_pcpu *txq_pcpu;
struct mvpp2_tx_desc *tx_desc;
- dma_addr_t buf_phys_addr;
+ dma_addr_t buf_dma_addr;
int frags = 0;
u16 txq_id;
u32 tx_cmd;
@@ -5307,35 +5665,38 @@
/* Get a descriptor for the first part of the packet */
tx_desc = mvpp2_txq_next_desc_get(aggr_txq);
- tx_desc->phys_txq = txq->id;
- tx_desc->data_size = skb_headlen(skb);
+ mvpp2_txdesc_txq_set(port, tx_desc, txq->id);
+ mvpp2_txdesc_size_set(port, tx_desc, skb_headlen(skb));
- buf_phys_addr = dma_map_single(dev->dev.parent, skb->data,
- tx_desc->data_size, DMA_TO_DEVICE);
- if (unlikely(dma_mapping_error(dev->dev.parent, buf_phys_addr))) {
+ buf_dma_addr = dma_map_single(dev->dev.parent, skb->data,
+ skb_headlen(skb), DMA_TO_DEVICE);
+ if (unlikely(dma_mapping_error(dev->dev.parent, buf_dma_addr))) {
mvpp2_txq_desc_put(txq);
frags = 0;
goto out;
}
- tx_desc->packet_offset = buf_phys_addr & MVPP2_TX_DESC_ALIGN;
- tx_desc->buf_phys_addr = buf_phys_addr & ~MVPP2_TX_DESC_ALIGN;
+
+ mvpp2_txdesc_offset_set(port, tx_desc,
+ buf_dma_addr & MVPP2_TX_DESC_ALIGN);
+ mvpp2_txdesc_dma_addr_set(port, tx_desc,
+ buf_dma_addr & ~MVPP2_TX_DESC_ALIGN);
tx_cmd = mvpp2_skb_tx_csum(port, skb);
if (frags == 1) {
/* First and Last descriptor */
tx_cmd |= MVPP2_TXD_F_DESC | MVPP2_TXD_L_DESC;
- tx_desc->command = tx_cmd;
- mvpp2_txq_inc_put(txq_pcpu, skb, tx_desc);
+ mvpp2_txdesc_cmd_set(port, tx_desc, tx_cmd);
+ mvpp2_txq_inc_put(port, txq_pcpu, skb, tx_desc);
} else {
/* First but not Last */
tx_cmd |= MVPP2_TXD_F_DESC | MVPP2_TXD_PADDING_DISABLE;
- tx_desc->command = tx_cmd;
- mvpp2_txq_inc_put(txq_pcpu, NULL, tx_desc);
+ mvpp2_txdesc_cmd_set(port, tx_desc, tx_cmd);
+ mvpp2_txq_inc_put(port, txq_pcpu, NULL, tx_desc);
/* Continue with other skb fragments */
if (mvpp2_tx_frag_process(port, skb, aggr_txq, txq)) {
- tx_desc_unmap_put(port->dev->dev.parent, txq, tx_desc);
+ tx_desc_unmap_put(port, txq, tx_desc);
frags = 0;
goto out;
}
@@ -5396,6 +5757,7 @@
u32 cause_rx_tx, cause_rx, cause_misc;
int rx_done = 0;
struct mvpp2_port *port = netdev_priv(napi->dev);
+ int cpu = smp_processor_id();
/* Rx/Tx cause register
*
@@ -5407,8 +5769,8 @@
*
* Each CPU has its own Rx/Tx cause register
*/
- cause_rx_tx = mvpp2_read(port->priv,
- MVPP2_ISR_RX_TX_CAUSE_REG(port->id));
+ cause_rx_tx = mvpp2_percpu_read(port->priv, cpu,
+ MVPP2_ISR_RX_TX_CAUSE_REG(port->id));
cause_rx_tx &= ~MVPP2_CAUSE_TXQ_OCCUP_DESC_ALL_MASK;
cause_misc = cause_rx_tx & MVPP2_CAUSE_MISC_SUM_MASK;
@@ -5417,8 +5779,9 @@
/* Clear the cause register */
mvpp2_write(port->priv, MVPP2_ISR_MISC_CAUSE_REG, 0);
- mvpp2_write(port->priv, MVPP2_ISR_RX_TX_CAUSE_REG(port->id),
- cause_rx_tx & ~MVPP2_CAUSE_MISC_SUM_MASK);
+ mvpp2_percpu_write(port->priv, cpu,
+ MVPP2_ISR_RX_TX_CAUSE_REG(port->id),
+ cause_rx_tx & ~MVPP2_CAUSE_MISC_SUM_MASK);
}
cause_rx = cause_rx_tx & MVPP2_CAUSE_RXQ_OCCUP_DESC_ALL_MASK;
@@ -5530,7 +5893,7 @@
return 0;
}
-static void mvpp2_get_mac_address(struct mvpp2_port *port, unsigned char *addr)
+static void mvpp21_get_mac_address(struct mvpp2_port *port, unsigned char *addr)
{
u32 mac_addr_l, mac_addr_m, mac_addr_h;
@@ -5698,7 +6061,7 @@
if (!is_valid_ether_addr(addr->sa_data)) {
err = -EADDRNOTAVAIL;
- goto error;
+ goto log_error;
}
if (!netif_running(dev)) {
@@ -5708,7 +6071,7 @@
/* Reconfigure parser to accept the original MAC address */
err = mvpp2_prs_update_mac_da(dev, dev->dev_addr);
if (err)
- goto error;
+ goto log_error;
}
mvpp2_stop_dev(port);
@@ -5720,15 +6083,14 @@
/* Reconfigure parser accept the original MAC address */
err = mvpp2_prs_update_mac_da(dev, dev->dev_addr);
if (err)
- goto error;
+ goto log_error;
out_start:
mvpp2_start_dev(port);
mvpp2_egress_enable(port);
mvpp2_ingress_enable(port);
return 0;
-
-error:
- netdev_err(dev, "fail to change MAC address\n");
+log_error:
+ netdev_err(dev, "failed to change MAC address\n");
return err;
}
@@ -5753,7 +6115,7 @@
/* Reconfigure BM to the original MTU */
err = mvpp2_bm_update_mtu(dev, dev->mtu);
if (err)
- goto error;
+ goto log_error;
}
mvpp2_stop_dev(port);
@@ -5767,7 +6129,7 @@
/* Reconfigure BM to the original MTU */
err = mvpp2_bm_update_mtu(dev, dev->mtu);
if (err)
- goto error;
+ goto log_error;
out_start:
mvpp2_start_dev(port);
@@ -5775,9 +6137,8 @@
mvpp2_ingress_enable(port);
return 0;
-
-error:
- netdev_err(dev, "fail to change MTU\n");
+log_error:
+ netdev_err(dev, "failed to change MTU\n");
return err;
}
@@ -5946,7 +6307,7 @@
err_clean_rxqs:
mvpp2_cleanup_rxqs(port);
err_out:
- netdev_err(dev, "fail to change ring parameters");
+ netdev_err(dev, "failed to change ring parameters");
return err;
}
@@ -5975,16 +6336,6 @@
.set_link_ksettings = phy_ethtool_set_link_ksettings,
};
-/* Driver initialization */
-
-static void mvpp2_port_power_up(struct mvpp2_port *port)
-{
- mvpp2_port_mii_set(port);
- mvpp2_port_periodic_xon_disable(port);
- mvpp2_port_fc_adv_enable(port);
- mvpp2_port_reset(port);
-}
-
/* Initialize port HW */
static int mvpp2_port_init(struct mvpp2_port *port)
{
@@ -5993,7 +6344,8 @@
struct mvpp2_txq_pcpu *txq_pcpu;
int queue, cpu, err;
- if (port->first_rxq + rxq_number > MVPP2_RXQ_TOTAL_NUM)
+ if (port->first_rxq + rxq_number >
+ MVPP2_MAX_PORTS * priv->max_port_rxqs)
return -EINVAL;
/* Disable port */
@@ -6061,7 +6413,18 @@
}
/* Configure Rx queue group interrupt for this port */
- mvpp2_write(priv, MVPP2_ISR_RXQ_GROUP_REG(port->id), rxq_number);
+ if (priv->hw_version == MVPP21) {
+ mvpp2_write(priv, MVPP21_ISR_RXQ_GROUP_REG(port->id),
+ rxq_number);
+ } else {
+ u32 val;
+
+ val = (port->id << MVPP22_ISR_RXQ_GROUP_INDEX_GROUP_OFFSET);
+ mvpp2_write(priv, MVPP22_ISR_RXQ_GROUP_INDEX_REG, val);
+
+ val = (rxq_number << MVPP22_ISR_RXQ_SUB_GROUP_SIZE_OFFSET);
+ mvpp2_write(priv, MVPP22_ISR_RXQ_SUB_GROUP_CONFIG_REG, val);
+ }
/* Create Rx descriptor rings */
for (queue = 0; queue < rxq_number; queue++) {
@@ -6103,8 +6466,7 @@
/* Ports initialization */
static int mvpp2_port_probe(struct platform_device *pdev,
struct device_node *port_node,
- struct mvpp2 *priv,
- int *next_first_rxq)
+ struct mvpp2 *priv)
{
struct device_node *phy_node;
struct mvpp2_port *port;
@@ -6117,11 +6479,9 @@
u32 id;
int features;
int phy_mode;
- int priv_common_regs_num = 2;
int err, i, cpu;
- dev = alloc_etherdev_mqs(sizeof(struct mvpp2_port), txq_number,
- rxq_number);
+ dev = alloc_etherdev_mqs(sizeof(*port), txq_number, rxq_number);
if (!dev)
return -ENOMEM;
@@ -6163,16 +6523,30 @@
port->priv = priv;
port->id = id;
- port->first_rxq = *next_first_rxq;
+ if (priv->hw_version == MVPP21)
+ port->first_rxq = port->id * rxq_number;
+ else
+ port->first_rxq = port->id * priv->max_port_rxqs;
+
port->phy_node = phy_node;
port->phy_interface = phy_mode;
- res = platform_get_resource(pdev, IORESOURCE_MEM,
- priv_common_regs_num + id);
- port->base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(port->base)) {
- err = PTR_ERR(port->base);
- goto err_free_irq;
+ if (priv->hw_version == MVPP21) {
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 2 + id);
+ port->base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(port->base)) {
+ err = PTR_ERR(port->base);
+ goto err_free_irq;
+ }
+ } else {
+ if (of_property_read_u32(port_node, "gop-port-id",
+ &port->gop_id)) {
+ err = -EINVAL;
+ dev_err(&pdev->dev, "missing gop-port-id value\n");
+ goto err_free_irq;
+ }
+
+ port->base = priv->iface_base + MVPP22_GMAC_BASE(port->gop_id);
}
/* Alloc per-cpu stats */
@@ -6187,7 +6561,8 @@
mac_from = "device tree";
ether_addr_copy(dev->dev_addr, dt_mac_addr);
} else {
- mvpp2_get_mac_address(port, hw_mac_addr);
+ if (priv->hw_version == MVPP21)
+ mvpp21_get_mac_address(port, hw_mac_addr);
if (is_valid_ether_addr(hw_mac_addr)) {
mac_from = "hardware";
ether_addr_copy(dev->dev_addr, hw_mac_addr);
@@ -6207,7 +6582,14 @@
dev_err(&pdev->dev, "failed to init port %d\n", id);
goto err_free_stats;
}
- mvpp2_port_power_up(port);
+
+ mvpp2_port_mii_set(port);
+ mvpp2_port_periodic_xon_disable(port);
+
+ if (priv->hw_version == MVPP21)
+ mvpp2_port_fc_adv_enable(port);
+
+ mvpp2_port_reset(port);
port->pcpu = alloc_percpu(struct mvpp2_port_pcpu);
if (!port->pcpu) {
@@ -6245,8 +6627,6 @@
}
netdev_info(dev, "Using %s mac address %pM\n", mac_from, dev->dev_addr);
- /* Increment the first Rx queue number to be used by the next port */
- *next_first_rxq += rxq_number;
priv->port_list[id] = port;
return 0;
@@ -6330,6 +6710,60 @@
mvpp2_write(priv, MVPP2_RX_FIFO_INIT_REG, 0x1);
}
+static void mvpp2_axi_init(struct mvpp2 *priv)
+{
+ u32 val, rdval, wrval;
+
+ mvpp2_write(priv, MVPP22_BM_ADDR_HIGH_RLS_REG, 0x0);
+
+ /* AXI Bridge Configuration */
+
+ rdval = MVPP22_AXI_CODE_CACHE_RD_CACHE
+ << MVPP22_AXI_ATTR_CACHE_OFFS;
+ rdval |= MVPP22_AXI_CODE_DOMAIN_OUTER_DOM
+ << MVPP22_AXI_ATTR_DOMAIN_OFFS;
+
+ wrval = MVPP22_AXI_CODE_CACHE_WR_CACHE
+ << MVPP22_AXI_ATTR_CACHE_OFFS;
+ wrval |= MVPP22_AXI_CODE_DOMAIN_OUTER_DOM
+ << MVPP22_AXI_ATTR_DOMAIN_OFFS;
+
+ /* BM */
+ mvpp2_write(priv, MVPP22_AXI_BM_WR_ATTR_REG, wrval);
+ mvpp2_write(priv, MVPP22_AXI_BM_RD_ATTR_REG, rdval);
+
+ /* Descriptors */
+ mvpp2_write(priv, MVPP22_AXI_AGGRQ_DESCR_RD_ATTR_REG, rdval);
+ mvpp2_write(priv, MVPP22_AXI_TXQ_DESCR_WR_ATTR_REG, wrval);
+ mvpp2_write(priv, MVPP22_AXI_TXQ_DESCR_RD_ATTR_REG, rdval);
+ mvpp2_write(priv, MVPP22_AXI_RXQ_DESCR_WR_ATTR_REG, wrval);
+
+ /* Buffer Data */
+ mvpp2_write(priv, MVPP22_AXI_TX_DATA_RD_ATTR_REG, rdval);
+ mvpp2_write(priv, MVPP22_AXI_RX_DATA_WR_ATTR_REG, wrval);
+
+ val = MVPP22_AXI_CODE_CACHE_NON_CACHE
+ << MVPP22_AXI_CODE_CACHE_OFFS;
+ val |= MVPP22_AXI_CODE_DOMAIN_SYSTEM
+ << MVPP22_AXI_CODE_DOMAIN_OFFS;
+ mvpp2_write(priv, MVPP22_AXI_RD_NORMAL_CODE_REG, val);
+ mvpp2_write(priv, MVPP22_AXI_WR_NORMAL_CODE_REG, val);
+
+ val = MVPP22_AXI_CODE_CACHE_RD_CACHE
+ << MVPP22_AXI_CODE_CACHE_OFFS;
+ val |= MVPP22_AXI_CODE_DOMAIN_OUTER_DOM
+ << MVPP22_AXI_CODE_DOMAIN_OFFS;
+
+ mvpp2_write(priv, MVPP22_AXI_RD_SNOOP_CODE_REG, val);
+
+ val = MVPP22_AXI_CODE_CACHE_WR_CACHE
+ << MVPP22_AXI_CODE_CACHE_OFFS;
+ val |= MVPP22_AXI_CODE_DOMAIN_OUTER_DOM
+ << MVPP22_AXI_CODE_DOMAIN_OFFS;
+
+ mvpp2_write(priv, MVPP22_AXI_WR_SNOOP_CODE_REG, val);
+}
+
/* Initialize network controller common part HW */
static int mvpp2_init(struct platform_device *pdev, struct mvpp2 *priv)
{
@@ -6338,7 +6772,7 @@
u32 val;
/* Checks for hardware constraints */
- if (rxq_number % 4 || (rxq_number > MVPP2_MAX_RXQ) ||
+ if (rxq_number % 4 || (rxq_number > priv->max_port_rxqs) ||
(txq_number > MVPP2_MAX_TXQ)) {
dev_err(&pdev->dev, "invalid queue size parameter\n");
return -EINVAL;
@@ -6349,14 +6783,23 @@
if (dram_target_info)
mvpp2_conf_mbus_windows(dram_target_info, priv);
+ if (priv->hw_version == MVPP22)
+ mvpp2_axi_init(priv);
+
/* Disable HW PHY polling */
- val = readl(priv->lms_base + MVPP2_PHY_AN_CFG0_REG);
- val |= MVPP2_PHY_AN_STOP_SMI0_MASK;
- writel(val, priv->lms_base + MVPP2_PHY_AN_CFG0_REG);
+ if (priv->hw_version == MVPP21) {
+ val = readl(priv->lms_base + MVPP2_PHY_AN_CFG0_REG);
+ val |= MVPP2_PHY_AN_STOP_SMI0_MASK;
+ writel(val, priv->lms_base + MVPP2_PHY_AN_CFG0_REG);
+ } else {
+ val = readl(priv->iface_base + MVPP22_SMI_MISC_CFG_REG);
+ val &= ~MVPP22_SMI_POLLING_EN;
+ writel(val, priv->iface_base + MVPP22_SMI_MISC_CFG_REG);
+ }
/* Allocate and initialize aggregated TXQs */
priv->aggr_txqs = devm_kcalloc(&pdev->dev, num_present_cpus(),
- sizeof(struct mvpp2_tx_queue),
+ sizeof(*priv->aggr_txqs),
GFP_KERNEL);
if (!priv->aggr_txqs)
return -ENOMEM;
@@ -6374,11 +6817,25 @@
mvpp2_rx_fifo_init(priv);
/* Reset Rx queue group interrupt configuration */
- for (i = 0; i < MVPP2_MAX_PORTS; i++)
- mvpp2_write(priv, MVPP2_ISR_RXQ_GROUP_REG(i), rxq_number);
+ for (i = 0; i < MVPP2_MAX_PORTS; i++) {
+ if (priv->hw_version == MVPP21) {
+ mvpp2_write(priv, MVPP21_ISR_RXQ_GROUP_REG(i),
+ rxq_number);
+ continue;
+ } else {
+ u32 val;
- writel(MVPP2_EXT_GLOBAL_CTRL_DEFAULT,
- priv->lms_base + MVPP2_MNG_EXTENDED_GLOBAL_CTRL_REG);
+ val = (i << MVPP22_ISR_RXQ_GROUP_INDEX_GROUP_OFFSET);
+ mvpp2_write(priv, MVPP22_ISR_RXQ_GROUP_INDEX_REG, val);
+
+ val = (rxq_number << MVPP22_ISR_RXQ_SUB_GROUP_SIZE_OFFSET);
+ mvpp2_write(priv, MVPP22_ISR_RXQ_SUB_GROUP_CONFIG_REG, val);
+ }
+ }
+
+ if (priv->hw_version == MVPP21)
+ writel(MVPP2_EXT_GLOBAL_CTRL_DEFAULT,
+ priv->lms_base + MVPP2_MNG_EXTENDED_GLOBAL_CTRL_REG);
/* Allow cache snoop when transmiting packets */
mvpp2_write(priv, MVPP2_TX_SNOOP_REG, 0x1);
@@ -6405,22 +6862,46 @@
struct device_node *port_node;
struct mvpp2 *priv;
struct resource *res;
- int port_count, first_rxq;
+ void __iomem *base;
+ int port_count, cpu;
int err;
- priv = devm_kzalloc(&pdev->dev, sizeof(struct mvpp2), GFP_KERNEL);
+ priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
if (!priv)
return -ENOMEM;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- priv->base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(priv->base))
- return PTR_ERR(priv->base);
+ priv->hw_version =
+ (unsigned long)of_device_get_match_data(&pdev->dev);
- res = platform_get_resource(pdev, IORESOURCE_MEM, 1);
- priv->lms_base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(priv->lms_base))
- return PTR_ERR(priv->lms_base);
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+
+ if (priv->hw_version == MVPP21) {
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 1);
+ priv->lms_base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(priv->lms_base))
+ return PTR_ERR(priv->lms_base);
+ } else {
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 1);
+ priv->iface_base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(priv->iface_base))
+ return PTR_ERR(priv->iface_base);
+ }
+
+ for_each_present_cpu(cpu) {
+ u32 addr_space_sz;
+
+ addr_space_sz = (priv->hw_version == MVPP21 ?
+ MVPP21_ADDR_SPACE_SZ : MVPP22_ADDR_SPACE_SZ);
+ priv->cpu_base[cpu] = base + cpu * addr_space_sz;
+ }
+
+ if (priv->hw_version == MVPP21)
+ priv->max_port_rxqs = 8;
+ else
+ priv->max_port_rxqs = 32;
priv->pp_clk = devm_clk_get(&pdev->dev, "pp_clk");
if (IS_ERR(priv->pp_clk))
@@ -6438,42 +6919,70 @@
if (err < 0)
goto err_pp_clk;
+ if (priv->hw_version == MVPP22) {
+ priv->mg_clk = devm_clk_get(&pdev->dev, "mg_clk");
+ if (IS_ERR(priv->mg_clk)) {
+ err = PTR_ERR(priv->mg_clk);
+ goto err_gop_clk;
+ }
+
+ err = clk_prepare_enable(priv->mg_clk);
+ if (err < 0)
+ goto err_gop_clk;
+ }
+
/* Get system's tclk rate */
priv->tclk = clk_get_rate(priv->pp_clk);
+ if (priv->hw_version == MVPP22) {
+ err = dma_set_mask(&pdev->dev, DMA_BIT_MASK(40));
+ if (err)
+ goto err_mg_clk;
+ /* Sadly, the BM pools all share the same register to
+ * store the high 32 bits of their address. So they
+ * must all have the same high 32 bits, which forces
+ * us to restrict coherent memory to DMA_BIT_MASK(32).
+ */
+ err = dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(32));
+ if (err)
+ goto err_mg_clk;
+ }
+
/* Initialize network controller */
err = mvpp2_init(pdev, priv);
if (err < 0) {
dev_err(&pdev->dev, "failed to initialize controller\n");
- goto err_gop_clk;
+ goto err_mg_clk;
}
port_count = of_get_available_child_count(dn);
if (port_count == 0) {
dev_err(&pdev->dev, "no ports enabled\n");
err = -ENODEV;
- goto err_gop_clk;
+ goto err_mg_clk;
}
priv->port_list = devm_kcalloc(&pdev->dev, port_count,
- sizeof(struct mvpp2_port *),
- GFP_KERNEL);
+ sizeof(*priv->port_list),
+ GFP_KERNEL);
if (!priv->port_list) {
err = -ENOMEM;
- goto err_gop_clk;
+ goto err_mg_clk;
}
/* Initialize ports */
- first_rxq = 0;
for_each_available_child_of_node(dn, port_node) {
- err = mvpp2_port_probe(pdev, port_node, priv, &first_rxq);
+ err = mvpp2_port_probe(pdev, port_node, priv);
if (err < 0)
- goto err_gop_clk;
+ goto err_mg_clk;
}
platform_set_drvdata(pdev, priv);
return 0;
+err_mg_clk:
+ if (priv->hw_version == MVPP22)
+ clk_disable_unprepare(priv->mg_clk);
err_gop_clk:
clk_disable_unprepare(priv->gop_clk);
err_pp_clk:
@@ -6506,9 +7015,10 @@
dma_free_coherent(&pdev->dev,
MVPP2_AGGR_TXQ_SIZE * MVPP2_DESC_ALIGNED_SIZE,
aggr_txq->descs,
- aggr_txq->descs_phys);
+ aggr_txq->descs_dma);
}
+ clk_disable_unprepare(priv->mg_clk);
clk_disable_unprepare(priv->pp_clk);
clk_disable_unprepare(priv->gop_clk);
@@ -6516,7 +7026,14 @@
}
static const struct of_device_id mvpp2_match[] = {
- { .compatible = "marvell,armada-375-pp2" },
+ {
+ .compatible = "marvell,armada-375-pp2",
+ .data = (void *)MVPP21,
+ },
+ {
+ .compatible = "marvell,armada-7k-pp22",
+ .data = (void *)MVPP22,
+ },
{ }
};
MODULE_DEVICE_TABLE(of, mvpp2_match);
diff --git a/drivers/net/ethernet/marvell/pxa168_eth.c b/drivers/net/ethernet/marvell/pxa168_eth.c
index 28cb36d..9937249 100644
--- a/drivers/net/ethernet/marvell/pxa168_eth.c
+++ b/drivers/net/ethernet/marvell/pxa168_eth.c
@@ -556,11 +556,11 @@
* function.Driver can dynamically switch to them if the 1/2kB hash
* table is full.
*/
- if (pep->htpr == NULL) {
+ if (!pep->htpr) {
pep->htpr = dma_zalloc_coherent(pep->dev->dev.parent,
HASH_ADDR_TABLE_SIZE,
&pep->htpr_dma, GFP_KERNEL);
- if (pep->htpr == NULL)
+ if (!pep->htpr)
return -ENOMEM;
} else {
memset(pep->htpr, 0, HASH_ADDR_TABLE_SIZE);
@@ -1036,8 +1036,7 @@
int rx_desc_num = pep->rx_ring_size;
/* Allocate RX skb rings */
- pep->rx_skb = kzalloc(sizeof(*pep->rx_skb) * pep->rx_ring_size,
- GFP_KERNEL);
+ pep->rx_skb = kcalloc(rx_desc_num, sizeof(*pep->rx_skb), GFP_KERNEL);
if (!pep->rx_skb)
return -ENOMEM;
@@ -1096,8 +1095,7 @@
int size = 0, i = 0;
int tx_desc_num = pep->tx_ring_size;
- pep->tx_skb = kzalloc(sizeof(*pep->tx_skb) * pep->tx_ring_size,
- GFP_KERNEL);
+ pep->tx_skb = kcalloc(tx_desc_num, sizeof(*pep->tx_skb), GFP_KERNEL);
if (!pep->tx_skb)
return -ENOMEM;
@@ -1358,7 +1356,7 @@
static int pxa168_eth_do_ioctl(struct net_device *dev, struct ifreq *ifr,
int cmd)
{
- if (dev->phydev != NULL)
+ if (dev->phydev)
return phy_mii_ioctl(dev->phydev, ifr, cmd);
return -EOPNOTSUPP;
@@ -1503,7 +1501,7 @@
pep->timeout.data = (unsigned long)pep;
pep->smi_bus = mdiobus_alloc();
- if (pep->smi_bus == NULL) {
+ if (!pep->smi_bus) {
err = -ENOMEM;
goto err_netdev;
}
diff --git a/drivers/net/ethernet/marvell/skge.c b/drivers/net/ethernet/marvell/skge.c
index edb9527..5d7d94d 100644
--- a/drivers/net/ethernet/marvell/skge.c
+++ b/drivers/net/ethernet/marvell/skge.c
@@ -2657,7 +2657,7 @@
struct skge_hw *hw = skge->hw;
int port = skge->port;
- if (skge->mem == NULL)
+ if (!skge->mem)
return 0;
netif_info(skge, ifdown, skge->netdev, "disabling interface\n");
@@ -3718,7 +3718,7 @@
t->csum_offs, t->csum_write, t->csum_start);
}
- seq_printf(seq, "\nRx Ring:\n");
+ seq_puts(seq, "\nRx Ring:\n");
for (e = skge->rx_ring.to_clean; ; e = e->next) {
const struct skge_rx_desc *r = e->desc;
diff --git a/drivers/net/ethernet/marvell/sky2.c b/drivers/net/ethernet/marvell/sky2.c
index 2b2cc3f..1145cde 100644
--- a/drivers/net/ethernet/marvell/sky2.c
+++ b/drivers/net/ethernet/marvell/sky2.c
@@ -4544,7 +4544,7 @@
sky2_read32(hw, B0_Y2_SP_ICR));
if (!netif_running(dev)) {
- seq_printf(seq, "network not running\n");
+ seq_puts(seq, "network not running\n");
return 0;
}
diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
index 9394913..16f9755 100644
--- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
+++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
@@ -1849,6 +1849,12 @@
/* GE2, Force 1000M/FD, FC ON */
mtk_w32(eth, MAC_MCR_FIXED_LINK, MTK_MAC_MCR(1));
+ /* Indicates CDM to parse the MTK special tag from CPU
+ * which also is working out for untag packets.
+ */
+ val = mtk_r32(eth, MTK_CDMQ_IG_CTRL);
+ mtk_w32(eth, val | MTK_CDMQ_STAG_EN, MTK_CDMQ_IG_CTRL);
+
/* Enable RX VLan Offloading */
mtk_w32(eth, 1, MTK_CDMP_EG_CTRL);
@@ -1911,10 +1917,9 @@
/* If the mac address is invalid, use random mac address */
if (!is_valid_ether_addr(dev->dev_addr)) {
- random_ether_addr(dev->dev_addr);
+ eth_hw_addr_random(dev);
dev_err(eth->dev, "generated random MAC address %pM\n",
dev->dev_addr);
- dev->addr_assign_type = NET_ADDR_RANDOM;
}
return mtk_phy_connect(dev);
@@ -2320,6 +2325,8 @@
eth->netdev[id]->ethtool_ops = &mtk_ethtool_ops;
eth->netdev[id]->irq = eth->irq[0];
+ eth->netdev[id]->dev.of_node = np;
+
return 0;
free_netdev:
diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.h b/drivers/net/ethernet/mediatek/mtk_eth_soc.h
index 08285a9..3c46a3b 100644
--- a/drivers/net/ethernet/mediatek/mtk_eth_soc.h
+++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.h
@@ -70,6 +70,10 @@
/* Frame Engine Interrupt Grouping Register */
#define MTK_FE_INT_GRP 0x20
+/* CDMP Ingress Control Register */
+#define MTK_CDMQ_IG_CTRL 0x1400
+#define MTK_CDMQ_STAG_EN BIT(0)
+
/* CDMP Exgress Control Register */
#define MTK_CDMP_EG_CTRL 0x404
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx4/en_ethtool.c
index c4d714f..ffbcb27 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_ethtool.c
@@ -117,7 +117,7 @@
/* port statistics */
"tso_packets",
"xmit_more",
- "queue_stopped", "wake_queue", "tx_timeout", "rx_alloc_failed",
+ "queue_stopped", "wake_queue", "tx_timeout", "rx_alloc_pages",
"rx_csum_good", "rx_csum_none", "rx_csum_complete", "tx_chksum_offload",
/* pf statistics */
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
index 6142047..94fab20 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
@@ -92,7 +92,9 @@
if (tc->type != TC_SETUP_MQPRIO)
return -EINVAL;
- return mlx4_en_setup_tc(dev, tc->tc);
+ tc->mqprio->hw = TC_MQPRIO_HW_OFFLOAD_TCS;
+
+ return mlx4_en_setup_tc(dev, tc->mqprio->num_tc);
}
#ifdef CONFIG_RFS_ACCEL
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_port.c b/drivers/net/ethernet/mellanox/mlx4/en_port.c
index 9166d90..e0eb695 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_port.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_port.c
@@ -213,6 +213,7 @@
priv->port_stats.rx_chksum_good = 0;
priv->port_stats.rx_chksum_none = 0;
priv->port_stats.rx_chksum_complete = 0;
+ priv->port_stats.rx_alloc_pages = 0;
priv->xdp_stats.rx_xdp_drop = 0;
priv->xdp_stats.rx_xdp_tx = 0;
priv->xdp_stats.rx_xdp_tx_full = 0;
@@ -223,6 +224,7 @@
priv->port_stats.rx_chksum_good += READ_ONCE(ring->csum_ok);
priv->port_stats.rx_chksum_none += READ_ONCE(ring->csum_none);
priv->port_stats.rx_chksum_complete += READ_ONCE(ring->csum_complete);
+ priv->port_stats.rx_alloc_pages += READ_ONCE(ring->rx_alloc_pages);
priv->xdp_stats.rx_xdp_drop += READ_ONCE(ring->xdp_drop);
priv->xdp_stats.rx_xdp_tx += READ_ONCE(ring->xdp_tx);
priv->xdp_stats.rx_xdp_tx_full += READ_ONCE(ring->xdp_tx_full);
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_rx.c b/drivers/net/ethernet/mellanox/mlx4/en_rx.c
index 8672928..aa074e5 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_rx.c
@@ -50,173 +50,62 @@
#include "mlx4_en.h"
-static int mlx4_alloc_pages(struct mlx4_en_priv *priv,
- struct mlx4_en_rx_alloc *page_alloc,
- const struct mlx4_en_frag_info *frag_info,
- gfp_t _gfp)
+static int mlx4_alloc_page(struct mlx4_en_priv *priv,
+ struct mlx4_en_rx_alloc *frag,
+ gfp_t gfp)
{
- int order;
struct page *page;
dma_addr_t dma;
- for (order = frag_info->order; ;) {
- gfp_t gfp = _gfp;
-
- if (order)
- gfp |= __GFP_COMP | __GFP_NOWARN | __GFP_NOMEMALLOC;
- page = alloc_pages(gfp, order);
- if (likely(page))
- break;
- if (--order < 0 ||
- ((PAGE_SIZE << order) < frag_info->frag_size))
- return -ENOMEM;
- }
- dma = dma_map_page(priv->ddev, page, 0, PAGE_SIZE << order,
- frag_info->dma_dir);
+ page = alloc_page(gfp);
+ if (unlikely(!page))
+ return -ENOMEM;
+ dma = dma_map_page(priv->ddev, page, 0, PAGE_SIZE, priv->dma_dir);
if (unlikely(dma_mapping_error(priv->ddev, dma))) {
- put_page(page);
+ __free_page(page);
return -ENOMEM;
}
- page_alloc->page_size = PAGE_SIZE << order;
- page_alloc->page = page;
- page_alloc->dma = dma;
- page_alloc->page_offset = 0;
- /* Not doing get_page() for each frag is a big win
- * on asymetric workloads. Note we can not use atomic_set().
- */
- page_ref_add(page, page_alloc->page_size / frag_info->frag_stride - 1);
+ frag->page = page;
+ frag->dma = dma;
+ frag->page_offset = priv->rx_headroom;
return 0;
}
static int mlx4_en_alloc_frags(struct mlx4_en_priv *priv,
+ struct mlx4_en_rx_ring *ring,
struct mlx4_en_rx_desc *rx_desc,
struct mlx4_en_rx_alloc *frags,
- struct mlx4_en_rx_alloc *ring_alloc,
gfp_t gfp)
{
- struct mlx4_en_rx_alloc page_alloc[MLX4_EN_MAX_RX_FRAGS];
- const struct mlx4_en_frag_info *frag_info;
- struct page *page;
int i;
- for (i = 0; i < priv->num_frags; i++) {
- frag_info = &priv->frag_info[i];
- page_alloc[i] = ring_alloc[i];
- page_alloc[i].page_offset += frag_info->frag_stride;
-
- if (page_alloc[i].page_offset + frag_info->frag_stride <=
- ring_alloc[i].page_size)
- continue;
-
- if (unlikely(mlx4_alloc_pages(priv, &page_alloc[i],
- frag_info, gfp)))
- goto out;
- }
-
- for (i = 0; i < priv->num_frags; i++) {
- frags[i] = ring_alloc[i];
- frags[i].page_offset += priv->frag_info[i].rx_headroom;
- rx_desc->data[i].addr = cpu_to_be64(frags[i].dma +
- frags[i].page_offset);
- ring_alloc[i] = page_alloc[i];
- }
-
- return 0;
-
-out:
- while (i--) {
- if (page_alloc[i].page != ring_alloc[i].page) {
- dma_unmap_page(priv->ddev, page_alloc[i].dma,
- page_alloc[i].page_size,
- priv->frag_info[i].dma_dir);
- page = page_alloc[i].page;
- /* Revert changes done by mlx4_alloc_pages */
- page_ref_sub(page, page_alloc[i].page_size /
- priv->frag_info[i].frag_stride - 1);
- put_page(page);
+ for (i = 0; i < priv->num_frags; i++, frags++) {
+ if (!frags->page) {
+ if (mlx4_alloc_page(priv, frags, gfp))
+ return -ENOMEM;
+ ring->rx_alloc_pages++;
}
- }
- return -ENOMEM;
-}
-
-static void mlx4_en_free_frag(struct mlx4_en_priv *priv,
- struct mlx4_en_rx_alloc *frags,
- int i)
-{
- const struct mlx4_en_frag_info *frag_info = &priv->frag_info[i];
- u32 next_frag_end = frags[i].page_offset + 2 * frag_info->frag_stride;
-
-
- if (next_frag_end > frags[i].page_size)
- dma_unmap_page(priv->ddev, frags[i].dma, frags[i].page_size,
- frag_info->dma_dir);
-
- if (frags[i].page)
- put_page(frags[i].page);
-}
-
-static int mlx4_en_init_allocator(struct mlx4_en_priv *priv,
- struct mlx4_en_rx_ring *ring)
-{
- int i;
- struct mlx4_en_rx_alloc *page_alloc;
-
- for (i = 0; i < priv->num_frags; i++) {
- const struct mlx4_en_frag_info *frag_info = &priv->frag_info[i];
-
- if (mlx4_alloc_pages(priv, &ring->page_alloc[i],
- frag_info, GFP_KERNEL | __GFP_COLD))
- goto out;
-
- en_dbg(DRV, priv, " frag %d allocator: - size:%d frags:%d\n",
- i, ring->page_alloc[i].page_size,
- page_ref_count(ring->page_alloc[i].page));
+ rx_desc->data[i].addr = cpu_to_be64(frags->dma +
+ frags->page_offset);
}
return 0;
-
-out:
- while (i--) {
- struct page *page;
-
- page_alloc = &ring->page_alloc[i];
- dma_unmap_page(priv->ddev, page_alloc->dma,
- page_alloc->page_size,
- priv->frag_info[i].dma_dir);
- page = page_alloc->page;
- /* Revert changes done by mlx4_alloc_pages */
- page_ref_sub(page, page_alloc->page_size /
- priv->frag_info[i].frag_stride - 1);
- put_page(page);
- page_alloc->page = NULL;
- }
- return -ENOMEM;
}
-static void mlx4_en_destroy_allocator(struct mlx4_en_priv *priv,
- struct mlx4_en_rx_ring *ring)
+static void mlx4_en_free_frag(const struct mlx4_en_priv *priv,
+ struct mlx4_en_rx_alloc *frag)
{
- struct mlx4_en_rx_alloc *page_alloc;
- int i;
-
- for (i = 0; i < priv->num_frags; i++) {
- const struct mlx4_en_frag_info *frag_info = &priv->frag_info[i];
-
- page_alloc = &ring->page_alloc[i];
- en_dbg(DRV, priv, "Freeing allocator:%d count:%d\n",
- i, page_count(page_alloc->page));
-
- dma_unmap_page(priv->ddev, page_alloc->dma,
- page_alloc->page_size, frag_info->dma_dir);
- while (page_alloc->page_offset + frag_info->frag_stride <
- page_alloc->page_size) {
- put_page(page_alloc->page);
- page_alloc->page_offset += frag_info->frag_stride;
- }
- page_alloc->page = NULL;
+ if (frag->page) {
+ dma_unmap_page(priv->ddev, frag->dma,
+ PAGE_SIZE, priv->dma_dir);
+ __free_page(frag->page);
}
+ /* We need to clear all fields, otherwise a change of priv->log_rx_info
+ * could lead to see garbage later in frag->page.
+ */
+ memset(frag, 0, sizeof(*frag));
}
-static void mlx4_en_init_rx_desc(struct mlx4_en_priv *priv,
+static void mlx4_en_init_rx_desc(const struct mlx4_en_priv *priv,
struct mlx4_en_rx_ring *ring, int index)
{
struct mlx4_en_rx_desc *rx_desc = ring->buf + ring->stride * index;
@@ -248,18 +137,23 @@
struct mlx4_en_rx_desc *rx_desc = ring->buf + (index * ring->stride);
struct mlx4_en_rx_alloc *frags = ring->rx_info +
(index << priv->log_rx_info);
-
if (ring->page_cache.index > 0) {
- frags[0] = ring->page_cache.buf[--ring->page_cache.index];
- rx_desc->data[0].addr = cpu_to_be64(frags[0].dma +
- frags[0].page_offset);
+ /* XDP uses a single page per frame */
+ if (!frags->page) {
+ ring->page_cache.index--;
+ frags->page = ring->page_cache.buf[ring->page_cache.index].page;
+ frags->dma = ring->page_cache.buf[ring->page_cache.index].dma;
+ }
+ frags->page_offset = XDP_PACKET_HEADROOM;
+ rx_desc->data[0].addr = cpu_to_be64(frags->dma +
+ XDP_PACKET_HEADROOM);
return 0;
}
- return mlx4_en_alloc_frags(priv, rx_desc, frags, ring->page_alloc, gfp);
+ return mlx4_en_alloc_frags(priv, ring, rx_desc, frags, gfp);
}
-static inline bool mlx4_en_is_ring_empty(struct mlx4_en_rx_ring *ring)
+static bool mlx4_en_is_ring_empty(const struct mlx4_en_rx_ring *ring)
{
return ring->prod == ring->cons;
}
@@ -269,7 +163,8 @@
*ring->wqres.db.db = cpu_to_be32(ring->prod & 0xffff);
}
-static void mlx4_en_free_rx_desc(struct mlx4_en_priv *priv,
+/* slow path */
+static void mlx4_en_free_rx_desc(const struct mlx4_en_priv *priv,
struct mlx4_en_rx_ring *ring,
int index)
{
@@ -279,7 +174,7 @@
frags = ring->rx_info + (index << priv->log_rx_info);
for (nr = 0; nr < priv->num_frags; nr++) {
en_dbg(DRV, priv, "Freeing fragment:%d\n", nr);
- mlx4_en_free_frag(priv, frags, nr);
+ mlx4_en_free_frag(priv, frags + nr);
}
}
@@ -335,12 +230,12 @@
ring->cons, ring->prod);
/* Unmap and free Rx buffers */
- while (!mlx4_en_is_ring_empty(ring)) {
- index = ring->cons & ring->size_mask;
+ for (index = 0; index < ring->size; index++) {
en_dbg(DRV, priv, "Processing descriptor:%d\n", index);
mlx4_en_free_rx_desc(priv, ring, index);
- ++ring->cons;
}
+ ring->cons = 0;
+ ring->prod = 0;
}
void mlx4_en_set_num_rx_rings(struct mlx4_en_dev *mdev)
@@ -392,9 +287,9 @@
tmp = size * roundup_pow_of_two(MLX4_EN_MAX_RX_FRAGS *
sizeof(struct mlx4_en_rx_alloc));
- ring->rx_info = vmalloc_node(tmp, node);
+ ring->rx_info = vzalloc_node(tmp, node);
if (!ring->rx_info) {
- ring->rx_info = vmalloc(tmp);
+ ring->rx_info = vzalloc(tmp);
if (!ring->rx_info) {
err = -ENOMEM;
goto err_ring;
@@ -464,16 +359,6 @@
/* Initialize all descriptors */
for (i = 0; i < ring->size; i++)
mlx4_en_init_rx_desc(priv, ring, i);
-
- /* Initialize page allocators */
- err = mlx4_en_init_allocator(priv, ring);
- if (err) {
- en_err(priv, "Failed initializing ring allocator\n");
- if (ring->stride <= TXBB_SIZE)
- ring->buf -= TXBB_SIZE;
- ring_ind--;
- goto err_allocator;
- }
}
err = mlx4_en_fill_rx_buffers(priv);
if (err)
@@ -493,11 +378,9 @@
mlx4_en_free_rx_buf(priv, priv->rx_ring[ring_ind]);
ring_ind = priv->rx_ring_num - 1;
-err_allocator:
while (ring_ind >= 0) {
if (priv->rx_ring[ring_ind]->stride <= TXBB_SIZE)
priv->rx_ring[ring_ind]->buf -= TXBB_SIZE;
- mlx4_en_destroy_allocator(priv, priv->rx_ring[ring_ind]);
ring_ind--;
}
return err;
@@ -537,7 +420,9 @@
if (cache->index >= MLX4_EN_CACHE_SIZE)
return false;
- cache->buf[cache->index++] = *frame;
+ cache->buf[cache->index].page = frame->page;
+ cache->buf[cache->index].dma = frame->dma;
+ cache->index++;
return true;
}
@@ -567,136 +452,91 @@
int i;
for (i = 0; i < ring->page_cache.index; i++) {
- struct mlx4_en_rx_alloc *frame = &ring->page_cache.buf[i];
-
- dma_unmap_page(priv->ddev, frame->dma, frame->page_size,
- priv->frag_info[0].dma_dir);
- put_page(frame->page);
+ dma_unmap_page(priv->ddev, ring->page_cache.buf[i].dma,
+ PAGE_SIZE, priv->dma_dir);
+ put_page(ring->page_cache.buf[i].page);
}
ring->page_cache.index = 0;
mlx4_en_free_rx_buf(priv, ring);
if (ring->stride <= TXBB_SIZE)
ring->buf -= TXBB_SIZE;
- mlx4_en_destroy_allocator(priv, ring);
}
static int mlx4_en_complete_rx_desc(struct mlx4_en_priv *priv,
- struct mlx4_en_rx_desc *rx_desc,
struct mlx4_en_rx_alloc *frags,
struct sk_buff *skb,
int length)
{
- struct skb_frag_struct *skb_frags_rx = skb_shinfo(skb)->frags;
- struct mlx4_en_frag_info *frag_info;
- int nr;
+ const struct mlx4_en_frag_info *frag_info = priv->frag_info;
+ unsigned int truesize = 0;
+ int nr, frag_size;
+ struct page *page;
dma_addr_t dma;
+ bool release;
/* Collect used fragments while replacing them in the HW descriptors */
- for (nr = 0; nr < priv->num_frags; nr++) {
- frag_info = &priv->frag_info[nr];
- if (length <= frag_info->frag_prefix_size)
- break;
- if (unlikely(!frags[nr].page))
+ for (nr = 0;; frags++) {
+ frag_size = min_t(int, length, frag_info->frag_size);
+
+ page = frags->page;
+ if (unlikely(!page))
goto fail;
- dma = be64_to_cpu(rx_desc->data[nr].addr);
- dma_sync_single_for_cpu(priv->ddev, dma, frag_info->frag_size,
- DMA_FROM_DEVICE);
+ dma = frags->dma;
+ dma_sync_single_range_for_cpu(priv->ddev, dma, frags->page_offset,
+ frag_size, priv->dma_dir);
- __skb_fill_page_desc(skb, nr, frags[nr].page,
- frags[nr].page_offset,
- frag_info->frag_size);
+ __skb_fill_page_desc(skb, nr, page, frags->page_offset,
+ frag_size);
- skb->truesize += frag_info->frag_stride;
- frags[nr].page = NULL;
+ truesize += frag_info->frag_stride;
+ if (frag_info->frag_stride == PAGE_SIZE / 2) {
+ frags->page_offset ^= PAGE_SIZE / 2;
+ release = page_count(page) != 1 ||
+ page_is_pfmemalloc(page) ||
+ page_to_nid(page) != numa_mem_id();
+ } else {
+ u32 sz_align = ALIGN(frag_size, SMP_CACHE_BYTES);
+
+ frags->page_offset += sz_align;
+ release = frags->page_offset + frag_info->frag_size > PAGE_SIZE;
+ }
+ if (release) {
+ dma_unmap_page(priv->ddev, dma, PAGE_SIZE, priv->dma_dir);
+ frags->page = NULL;
+ } else {
+ page_ref_inc(page);
+ }
+
+ nr++;
+ length -= frag_size;
+ if (!length)
+ break;
+ frag_info++;
}
- /* Adjust size of last fragment to match actual length */
- if (nr > 0)
- skb_frag_size_set(&skb_frags_rx[nr - 1],
- length - priv->frag_info[nr - 1].frag_prefix_size);
+ skb->truesize += truesize;
return nr;
fail:
while (nr > 0) {
nr--;
- __skb_frag_unref(&skb_frags_rx[nr]);
+ __skb_frag_unref(skb_shinfo(skb)->frags + nr);
}
return 0;
}
-
-static struct sk_buff *mlx4_en_rx_skb(struct mlx4_en_priv *priv,
- struct mlx4_en_rx_desc *rx_desc,
- struct mlx4_en_rx_alloc *frags,
- unsigned int length)
+static void validate_loopback(struct mlx4_en_priv *priv, void *va)
{
- struct sk_buff *skb;
- void *va;
- int used_frags;
- dma_addr_t dma;
-
- skb = netdev_alloc_skb(priv->dev, SMALL_PACKET_SIZE + NET_IP_ALIGN);
- if (unlikely(!skb)) {
- en_dbg(RX_ERR, priv, "Failed allocating skb\n");
- return NULL;
- }
- skb_reserve(skb, NET_IP_ALIGN);
- skb->len = length;
-
- /* Get pointer to first fragment so we could copy the headers into the
- * (linear part of the) skb */
- va = page_address(frags[0].page) + frags[0].page_offset;
-
- if (length <= SMALL_PACKET_SIZE) {
- /* We are copying all relevant data to the skb - temporarily
- * sync buffers for the copy */
- dma = be64_to_cpu(rx_desc->data[0].addr);
- dma_sync_single_for_cpu(priv->ddev, dma, length,
- DMA_FROM_DEVICE);
- skb_copy_to_linear_data(skb, va, length);
- skb->tail += length;
- } else {
- unsigned int pull_len;
-
- /* Move relevant fragments to skb */
- used_frags = mlx4_en_complete_rx_desc(priv, rx_desc, frags,
- skb, length);
- if (unlikely(!used_frags)) {
- kfree_skb(skb);
- return NULL;
- }
- skb_shinfo(skb)->nr_frags = used_frags;
-
- pull_len = eth_get_headlen(va, SMALL_PACKET_SIZE);
- /* Copy headers into the skb linear buffer */
- memcpy(skb->data, va, pull_len);
- skb->tail += pull_len;
-
- /* Skip headers in first fragment */
- skb_shinfo(skb)->frags[0].page_offset += pull_len;
-
- /* Adjust size of first fragment */
- skb_frag_size_sub(&skb_shinfo(skb)->frags[0], pull_len);
- skb->data_len = length - pull_len;
- }
- return skb;
-}
-
-static void validate_loopback(struct mlx4_en_priv *priv, struct sk_buff *skb)
-{
+ const unsigned char *data = va + ETH_HLEN;
int i;
- int offset = ETH_HLEN;
- for (i = 0; i < MLX4_LOOPBACK_TEST_PAYLOAD; i++, offset++) {
- if (*(skb->data + offset) != (unsigned char) (i & 0xff))
- goto out_loopback;
+ for (i = 0; i < MLX4_LOOPBACK_TEST_PAYLOAD; i++) {
+ if (data[i] != (unsigned char)i)
+ return;
}
/* Loopback found */
priv->loopback_ok = 1;
-
-out_loopback:
- dev_kfree_skb_any(skb);
}
static bool mlx4_en_refill_rx_buffers(struct mlx4_en_priv *priv,
@@ -801,7 +641,6 @@
struct mlx4_cqe *cqe;
struct mlx4_en_rx_ring *ring = priv->rx_ring[cq->ring];
struct mlx4_en_rx_alloc *frags;
- struct mlx4_en_rx_desc *rx_desc;
struct bpf_prog *xdp_prog;
int doorbell_pending;
struct sk_buff *skb;
@@ -834,10 +673,10 @@
/* Process all completed CQEs */
while (XNOR(cqe->owner_sr_opcode & MLX4_CQE_OWNER_MASK,
cq->mcq.cons_index & cq->size)) {
+ void *va;
frags = ring->rx_info + (index << priv->log_rx_info);
- rx_desc = ring->buf + (index << ring->log_stride);
-
+ va = page_address(frags[0].page) + frags[0].page_offset;
/*
* make sure we read the CQE after we read the ownership bit
*/
@@ -860,16 +699,14 @@
* and not performing the selftest or flb disabled
*/
if (priv->flags & MLX4_EN_FLAG_RX_FILTER_NEEDED) {
- struct ethhdr *ethh;
+ const struct ethhdr *ethh = va;
dma_addr_t dma;
/* Get pointer to first fragment since we haven't
* skb yet and cast it to ethhdr struct
*/
- dma = be64_to_cpu(rx_desc->data[0].addr);
+ dma = frags[0].dma + frags[0].page_offset;
dma_sync_single_for_cpu(priv->ddev, dma, sizeof(*ethh),
DMA_FROM_DEVICE);
- ethh = (struct ethhdr *)(page_address(frags[0].page) +
- frags[0].page_offset);
if (is_multicast_ether_addr(ethh->h_dest)) {
struct mlx4_mac_entry *entry;
@@ -887,13 +724,16 @@
}
}
+ if (unlikely(priv->validate_loopback)) {
+ validate_loopback(priv, va);
+ goto next;
+ }
+
/*
* Packet is OK - process it.
*/
length = be32_to_cpu(cqe->byte_cnt);
length -= ring->fcs_del;
- l2_tunnel = (dev->hw_enc_features & NETIF_F_RXCSUM) &&
- (cqe->vlan_my_qpn & cpu_to_be32(MLX4_CQE_L2_TUNNEL));
/* A bpf program gets first chance to drop the packet. It may
* read bytes but not past the end of the frag.
@@ -904,13 +744,13 @@
void *orig_data;
u32 act;
- dma = be64_to_cpu(rx_desc->data[0].addr);
+ dma = frags[0].dma + frags[0].page_offset;
dma_sync_single_for_cpu(priv->ddev, dma,
priv->frag_info[0].frag_size,
DMA_FROM_DEVICE);
- xdp.data_hard_start = page_address(frags[0].page);
- xdp.data = xdp.data_hard_start + frags[0].page_offset;
+ xdp.data_hard_start = va - frags[0].page_offset;
+ xdp.data = va;
xdp.data_end = xdp.data + length;
orig_data = xdp.data;
@@ -920,6 +760,7 @@
length = xdp.data_end - xdp.data;
frags[0].page_offset = xdp.data -
xdp.data_hard_start;
+ va = xdp.data;
}
switch (act) {
@@ -928,8 +769,10 @@
case XDP_TX:
if (likely(!mlx4_en_xmit_frame(ring, frags, dev,
length, cq->ring,
- &doorbell_pending)))
- goto consumed;
+ &doorbell_pending))) {
+ frags[0].page = NULL;
+ goto next;
+ }
trace_xdp_exception(dev, xdp_prog, act);
goto xdp_drop_no_cnt; /* Drop on xmit failure */
default:
@@ -939,8 +782,6 @@
case XDP_DROP:
ring->xdp_drop++;
xdp_drop_no_cnt:
- if (likely(mlx4_en_rx_recycle(ring, frags)))
- goto consumed;
goto next;
}
}
@@ -948,129 +789,51 @@
ring->bytes += length;
ring->packets++;
+ skb = napi_get_frags(&cq->napi);
+ if (!skb)
+ goto next;
+
+ if (unlikely(ring->hwtstamp_rx_filter == HWTSTAMP_FILTER_ALL)) {
+ timestamp = mlx4_en_get_cqe_ts(cqe);
+ mlx4_en_fill_hwtstamps(mdev, skb_hwtstamps(skb),
+ timestamp);
+ }
+ skb_record_rx_queue(skb, cq->ring);
+
if (likely(dev->features & NETIF_F_RXCSUM)) {
if (cqe->status & cpu_to_be16(MLX4_CQE_STATUS_TCP |
MLX4_CQE_STATUS_UDP)) {
if ((cqe->status & cpu_to_be16(MLX4_CQE_STATUS_IPOK)) &&
cqe->checksum == cpu_to_be16(0xffff)) {
ip_summed = CHECKSUM_UNNECESSARY;
+ l2_tunnel = (dev->hw_enc_features & NETIF_F_RXCSUM) &&
+ (cqe->vlan_my_qpn & cpu_to_be32(MLX4_CQE_L2_TUNNEL));
+ if (l2_tunnel)
+ skb->csum_level = 1;
ring->csum_ok++;
} else {
- ip_summed = CHECKSUM_NONE;
- ring->csum_none++;
+ goto csum_none;
}
} else {
if (priv->flags & MLX4_EN_FLAG_RX_CSUM_NON_TCP_UDP &&
(cqe->status & cpu_to_be16(MLX4_CQE_STATUS_IPV4 |
MLX4_CQE_STATUS_IPV6))) {
- ip_summed = CHECKSUM_COMPLETE;
- ring->csum_complete++;
+ if (check_csum(cqe, skb, va, dev->features)) {
+ goto csum_none;
+ } else {
+ ip_summed = CHECKSUM_COMPLETE;
+ ring->csum_complete++;
+ }
} else {
- ip_summed = CHECKSUM_NONE;
- ring->csum_none++;
+ goto csum_none;
}
}
} else {
+csum_none:
ip_summed = CHECKSUM_NONE;
ring->csum_none++;
}
-
- /* This packet is eligible for GRO if it is:
- * - DIX Ethernet (type interpretation)
- * - TCP/IP (v4)
- * - without IP options
- * - not an IP fragment
- */
- if (dev->features & NETIF_F_GRO) {
- struct sk_buff *gro_skb = napi_get_frags(&cq->napi);
- if (!gro_skb)
- goto next;
-
- nr = mlx4_en_complete_rx_desc(priv,
- rx_desc, frags, gro_skb,
- length);
- if (!nr)
- goto next;
-
- if (ip_summed == CHECKSUM_COMPLETE) {
- void *va = skb_frag_address(skb_shinfo(gro_skb)->frags);
- if (check_csum(cqe, gro_skb, va,
- dev->features)) {
- ip_summed = CHECKSUM_NONE;
- ring->csum_none++;
- ring->csum_complete--;
- }
- }
-
- skb_shinfo(gro_skb)->nr_frags = nr;
- gro_skb->len = length;
- gro_skb->data_len = length;
- gro_skb->ip_summed = ip_summed;
-
- if (l2_tunnel && ip_summed == CHECKSUM_UNNECESSARY)
- gro_skb->csum_level = 1;
-
- if ((cqe->vlan_my_qpn &
- cpu_to_be32(MLX4_CQE_CVLAN_PRESENT_MASK)) &&
- (dev->features & NETIF_F_HW_VLAN_CTAG_RX)) {
- u16 vid = be16_to_cpu(cqe->sl_vid);
-
- __vlan_hwaccel_put_tag(gro_skb, htons(ETH_P_8021Q), vid);
- } else if ((be32_to_cpu(cqe->vlan_my_qpn) &
- MLX4_CQE_SVLAN_PRESENT_MASK) &&
- (dev->features & NETIF_F_HW_VLAN_STAG_RX)) {
- __vlan_hwaccel_put_tag(gro_skb,
- htons(ETH_P_8021AD),
- be16_to_cpu(cqe->sl_vid));
- }
-
- if (dev->features & NETIF_F_RXHASH)
- skb_set_hash(gro_skb,
- be32_to_cpu(cqe->immed_rss_invalid),
- (ip_summed == CHECKSUM_UNNECESSARY) ?
- PKT_HASH_TYPE_L4 :
- PKT_HASH_TYPE_L3);
-
- skb_record_rx_queue(gro_skb, cq->ring);
-
- if (ring->hwtstamp_rx_filter == HWTSTAMP_FILTER_ALL) {
- timestamp = mlx4_en_get_cqe_ts(cqe);
- mlx4_en_fill_hwtstamps(mdev,
- skb_hwtstamps(gro_skb),
- timestamp);
- }
-
- napi_gro_frags(&cq->napi);
- goto next;
- }
-
- /* GRO not possible, complete processing here */
- skb = mlx4_en_rx_skb(priv, rx_desc, frags, length);
- if (unlikely(!skb)) {
- ring->dropped++;
- goto next;
- }
-
- if (unlikely(priv->validate_loopback)) {
- validate_loopback(priv, skb);
- goto next;
- }
-
- if (ip_summed == CHECKSUM_COMPLETE) {
- if (check_csum(cqe, skb, skb->data, dev->features)) {
- ip_summed = CHECKSUM_NONE;
- ring->csum_complete--;
- ring->csum_none++;
- }
- }
-
skb->ip_summed = ip_summed;
- skb->protocol = eth_type_trans(skb, dev);
- skb_record_rx_queue(skb, cq->ring);
-
- if (l2_tunnel && ip_summed == CHECKSUM_UNNECESSARY)
- skb->csum_level = 1;
-
if (dev->features & NETIF_F_RXHASH)
skb_set_hash(skb,
be32_to_cpu(cqe->immed_rss_invalid),
@@ -1078,36 +841,36 @@
PKT_HASH_TYPE_L4 :
PKT_HASH_TYPE_L3);
- if ((be32_to_cpu(cqe->vlan_my_qpn) &
- MLX4_CQE_CVLAN_PRESENT_MASK) &&
+
+ if ((cqe->vlan_my_qpn &
+ cpu_to_be32(MLX4_CQE_CVLAN_PRESENT_MASK)) &&
(dev->features & NETIF_F_HW_VLAN_CTAG_RX))
- __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), be16_to_cpu(cqe->sl_vid));
- else if ((be32_to_cpu(cqe->vlan_my_qpn) &
- MLX4_CQE_SVLAN_PRESENT_MASK) &&
+ __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q),
+ be16_to_cpu(cqe->sl_vid));
+ else if ((cqe->vlan_my_qpn &
+ cpu_to_be32(MLX4_CQE_SVLAN_PRESENT_MASK)) &&
(dev->features & NETIF_F_HW_VLAN_STAG_RX))
__vlan_hwaccel_put_tag(skb, htons(ETH_P_8021AD),
be16_to_cpu(cqe->sl_vid));
- if (ring->hwtstamp_rx_filter == HWTSTAMP_FILTER_ALL) {
- timestamp = mlx4_en_get_cqe_ts(cqe);
- mlx4_en_fill_hwtstamps(mdev, skb_hwtstamps(skb),
- timestamp);
+ nr = mlx4_en_complete_rx_desc(priv, frags, skb, length);
+ if (likely(nr)) {
+ skb_shinfo(skb)->nr_frags = nr;
+ skb->len = length;
+ skb->data_len = length;
+ napi_gro_frags(&cq->napi);
+ } else {
+ skb->vlan_tci = 0;
+ skb_clear_hash(skb);
}
-
- napi_gro_receive(&cq->napi, skb);
next:
- for (nr = 0; nr < priv->num_frags; nr++)
- mlx4_en_free_frag(priv, frags, nr);
-
-consumed:
++cq->mcq.cons_index;
index = (cq->mcq.cons_index) & ring->size_mask;
cqe = mlx4_en_get_cqe(cq->buf, index, priv->cqe_size) + factor;
if (++polled == budget)
- goto out;
+ break;
}
-out:
rcu_read_unlock();
if (polled) {
@@ -1178,13 +941,6 @@
return done;
}
-static const int frag_sizes[] = {
- FRAG_SZ0,
- FRAG_SZ1,
- FRAG_SZ2,
- FRAG_SZ3
-};
-
void mlx4_en_calc_rx_buf(struct net_device *dev)
{
struct mlx4_en_priv *priv = netdev_priv(dev);
@@ -1195,33 +951,43 @@
* This only works when num_frags == 1.
*/
if (priv->tx_ring_num[TX_XDP]) {
- priv->frag_info[0].order = 0;
priv->frag_info[0].frag_size = eff_mtu;
- priv->frag_info[0].frag_prefix_size = 0;
/* This will gain efficient xdp frame recycling at the
* expense of more costly truesize accounting
*/
priv->frag_info[0].frag_stride = PAGE_SIZE;
- priv->frag_info[0].dma_dir = PCI_DMA_BIDIRECTIONAL;
- priv->frag_info[0].rx_headroom = XDP_PACKET_HEADROOM;
+ priv->dma_dir = PCI_DMA_BIDIRECTIONAL;
+ priv->rx_headroom = XDP_PACKET_HEADROOM;
i = 1;
} else {
- int buf_size = 0;
+ int frag_size_max = 2048, buf_size = 0;
+
+ /* should not happen, right ? */
+ if (eff_mtu > PAGE_SIZE + (MLX4_EN_MAX_RX_FRAGS - 1) * 2048)
+ frag_size_max = PAGE_SIZE;
while (buf_size < eff_mtu) {
- priv->frag_info[i].order = MLX4_EN_ALLOC_PREFER_ORDER;
- priv->frag_info[i].frag_size =
- (eff_mtu > buf_size + frag_sizes[i]) ?
- frag_sizes[i] : eff_mtu - buf_size;
- priv->frag_info[i].frag_prefix_size = buf_size;
- priv->frag_info[i].frag_stride =
- ALIGN(priv->frag_info[i].frag_size,
- SMP_CACHE_BYTES);
- priv->frag_info[i].dma_dir = PCI_DMA_FROMDEVICE;
- priv->frag_info[i].rx_headroom = 0;
- buf_size += priv->frag_info[i].frag_size;
+ int frag_stride, frag_size = eff_mtu - buf_size;
+ int pad, nb;
+
+ if (i < MLX4_EN_MAX_RX_FRAGS - 1)
+ frag_size = min(frag_size, frag_size_max);
+
+ priv->frag_info[i].frag_size = frag_size;
+ frag_stride = ALIGN(frag_size, SMP_CACHE_BYTES);
+ /* We can only pack 2 1536-bytes frames in on 4K page
+ * Therefore, each frame would consume more bytes (truesize)
+ */
+ nb = PAGE_SIZE / frag_stride;
+ pad = (PAGE_SIZE - nb * frag_stride) / nb;
+ pad &= ~(SMP_CACHE_BYTES - 1);
+ priv->frag_info[i].frag_stride = frag_stride + pad;
+
+ buf_size += frag_size;
i++;
}
+ priv->dma_dir = PCI_DMA_FROMDEVICE;
+ priv->rx_headroom = 0;
}
priv->num_frags = i;
@@ -1232,10 +998,9 @@
eff_mtu, priv->num_frags);
for (i = 0; i < priv->num_frags; i++) {
en_err(priv,
- " frag:%d - size:%d prefix:%d stride:%d\n",
+ " frag:%d - size:%d stride:%d\n",
i,
priv->frag_info[i].frag_size,
- priv->frag_info[i].frag_prefix_size,
priv->frag_info[i].frag_stride);
}
}
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_selftest.c b/drivers/net/ethernet/mellanox/mlx4/en_selftest.c
index 95290e1..17112fa 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_selftest.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_selftest.c
@@ -81,14 +81,11 @@
{
u32 loopback_ok = 0;
int i;
- bool gro_enabled;
priv->loopback_ok = 0;
priv->validate_loopback = 1;
- gro_enabled = priv->dev->features & NETIF_F_GRO;
mlx4_en_update_loopback_state(priv->dev, priv->dev->features);
- priv->dev->features &= ~NETIF_F_GRO;
/* xmit */
if (mlx4_en_test_loopback_xmit(priv)) {
@@ -111,9 +108,6 @@
priv->validate_loopback = 0;
- if (gro_enabled)
- priv->dev->features |= NETIF_F_GRO;
-
mlx4_en_update_loopback_state(priv->dev, priv->dev->features);
return !loopback_ok;
}
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
index 3ed4219..3ba89bc 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
@@ -354,13 +354,11 @@
struct mlx4_en_rx_alloc frame = {
.page = tx_info->page,
.dma = tx_info->map0_dma,
- .page_offset = XDP_PACKET_HEADROOM,
- .page_size = PAGE_SIZE,
};
if (!mlx4_en_rx_recycle(ring->recycle_ring, &frame)) {
dma_unmap_page(priv->ddev, tx_info->map0_dma,
- PAGE_SIZE, priv->frag_info[0].dma_dir);
+ PAGE_SIZE, priv->dma_dir);
put_page(tx_info->page);
}
@@ -980,8 +978,7 @@
ring->tso_packets++;
- i = ((skb->len - lso_header_size) / shinfo->gso_size) +
- !!((skb->len - lso_header_size) % shinfo->gso_size);
+ i = shinfo->gso_segs;
tx_info->nr_bytes = skb->len + (i - 1) * lso_header_size;
ring->packets += i;
} else {
diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
index 3629ce1..39f401a 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
+++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
@@ -102,17 +102,6 @@
/* Use the maximum between 16384 and a single page */
#define MLX4_EN_ALLOC_SIZE PAGE_ALIGN(16384)
-#define MLX4_EN_ALLOC_PREFER_ORDER min_t(int, get_order(32768), \
- PAGE_ALLOC_COSTLY_ORDER)
-
-/* Receive fragment sizes; we use at most 3 fragments (for 9600 byte MTU
- * and 4K allocations) */
-enum {
- FRAG_SZ0 = 1536 - NET_IP_ALIGN,
- FRAG_SZ1 = 4096,
- FRAG_SZ2 = 4096,
- FRAG_SZ3 = MLX4_EN_ALLOC_SIZE
-};
#define MLX4_EN_MAX_RX_FRAGS 4
/* Maximum ring sizes */
@@ -264,13 +253,16 @@
struct page *page;
dma_addr_t dma;
u32 page_offset;
- u32 page_size;
};
#define MLX4_EN_CACHE_SIZE (2 * NAPI_POLL_WEIGHT)
+
struct mlx4_en_page_cache {
u32 index;
- struct mlx4_en_rx_alloc buf[MLX4_EN_CACHE_SIZE];
+ struct {
+ struct page *page;
+ dma_addr_t dma;
+ } buf[MLX4_EN_CACHE_SIZE];
};
struct mlx4_en_priv;
@@ -335,7 +327,6 @@
struct mlx4_en_rx_ring {
struct mlx4_hwq_resources wqres;
- struct mlx4_en_rx_alloc page_alloc[MLX4_EN_MAX_RX_FRAGS];
u32 size ; /* number of Rx descs*/
u32 actual_size;
u32 size_mask;
@@ -355,6 +346,7 @@
unsigned long csum_ok;
unsigned long csum_none;
unsigned long csum_complete;
+ unsigned long rx_alloc_pages;
unsigned long xdp_drop;
unsigned long xdp_tx;
unsigned long xdp_tx_full;
@@ -472,11 +464,7 @@
struct mlx4_en_frag_info {
u16 frag_size;
- u16 frag_prefix_size;
u32 frag_stride;
- enum dma_data_direction dma_dir;
- u16 order;
- u16 rx_headroom;
};
#ifdef CONFIG_MLX4_EN_DCB
@@ -584,8 +572,10 @@
u32 rx_ring_num;
u32 rx_skb_size;
struct mlx4_en_frag_info frag_info[MLX4_EN_MAX_RX_FRAGS];
- u16 num_frags;
- u16 log_rx_info;
+ u8 num_frags;
+ u8 log_rx_info;
+ u8 dma_dir;
+ u16 rx_headroom;
struct mlx4_en_tx_ring **tx_ring[MLX4_EN_NUM_TX_TYPES];
struct mlx4_en_rx_ring *rx_ring[MAX_RX_RINGS];
diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_stats.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_stats.h
index 48641cb..926f3c3 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mlx4_stats.h
+++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_stats.h
@@ -37,7 +37,7 @@
unsigned long queue_stopped;
unsigned long wake_queue;
unsigned long tx_timeout;
- unsigned long rx_alloc_failed;
+ unsigned long rx_alloc_pages;
unsigned long rx_chksum_good;
unsigned long rx_chksum_none;
unsigned long rx_chksum_complete;
diff --git a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
index d8d5d16..4aa29ee 100644
--- a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
+++ b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
@@ -2749,7 +2749,7 @@
int err;
int index = vhcr->in_modifier;
struct res_mtt *mtt;
- struct res_mpt *mpt;
+ struct res_mpt *mpt = NULL;
int mtt_base = mr_get_mtt_addr(inbox->buf) / dev->caps.mtt_entry_sz;
int phys;
int id;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig
index 1171700..a84b652 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig
@@ -31,3 +31,10 @@
This flag is depended on the kernel's DCB support.
If unsure, set to Y
+
+config MLX5_CORE_IPOIB
+ bool "Mellanox Technologies ConnectX-4 IPoIB offloads support"
+ depends on MLX5_CORE_EN
+ default y
+ ---help---
+ MLX5 IPoIB offloads & acceleration support.
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index 9f43beb..9e64461 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -11,3 +11,5 @@
en_tc.o en_arfs.o en_rep.o en_fs_ethtool.o en_selftest.o
mlx5_core-$(CONFIG_MLX5_CORE_EN_DCB) += en_dcbnl.o
+
+mlx5_core-$(CONFIG_MLX5_CORE_IPOIB) += ipoib.o
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
index a380353..5bdaf3d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
@@ -279,6 +279,8 @@
case MLX5_CMD_OP_DESTROY_XRC_SRQ:
case MLX5_CMD_OP_DESTROY_DCT:
case MLX5_CMD_OP_DEALLOC_Q_COUNTER:
+ case MLX5_CMD_OP_DESTROY_SCHEDULING_ELEMENT:
+ case MLX5_CMD_OP_DESTROY_QOS_PARA_VPORT:
case MLX5_CMD_OP_DEALLOC_PD:
case MLX5_CMD_OP_DEALLOC_UAR:
case MLX5_CMD_OP_DETACH_FROM_MCG:
@@ -305,8 +307,7 @@
case MLX5_CMD_OP_SET_FLOW_TABLE_ENTRY:
case MLX5_CMD_OP_SET_FLOW_TABLE_ROOT:
case MLX5_CMD_OP_DEALLOC_ENCAP_HEADER:
- case MLX5_CMD_OP_DESTROY_SCHEDULING_ELEMENT:
- case MLX5_CMD_OP_DESTROY_QOS_PARA_VPORT:
+ case MLX5_CMD_OP_DEALLOC_MODIFY_HEADER_CONTEXT:
return MLX5_CMD_STAT_OK;
case MLX5_CMD_OP_QUERY_HCA_CAP:
@@ -363,6 +364,10 @@
case MLX5_CMD_OP_QUERY_Q_COUNTER:
case MLX5_CMD_OP_SET_RATE_LIMIT:
case MLX5_CMD_OP_QUERY_RATE_LIMIT:
+ case MLX5_CMD_OP_CREATE_SCHEDULING_ELEMENT:
+ case MLX5_CMD_OP_QUERY_SCHEDULING_ELEMENT:
+ case MLX5_CMD_OP_MODIFY_SCHEDULING_ELEMENT:
+ case MLX5_CMD_OP_CREATE_QOS_PARA_VPORT:
case MLX5_CMD_OP_ALLOC_PD:
case MLX5_CMD_OP_ALLOC_UAR:
case MLX5_CMD_OP_CONFIG_INT_MODERATION:
@@ -414,10 +419,7 @@
case MLX5_CMD_OP_ALLOC_FLOW_COUNTER:
case MLX5_CMD_OP_QUERY_FLOW_COUNTER:
case MLX5_CMD_OP_ALLOC_ENCAP_HEADER:
- case MLX5_CMD_OP_CREATE_SCHEDULING_ELEMENT:
- case MLX5_CMD_OP_QUERY_SCHEDULING_ELEMENT:
- case MLX5_CMD_OP_MODIFY_SCHEDULING_ELEMENT:
- case MLX5_CMD_OP_CREATE_QOS_PARA_VPORT:
+ case MLX5_CMD_OP_ALLOC_MODIFY_HEADER_CONTEXT:
*status = MLX5_DRIVER_STATUS_ABORTED;
*synd = MLX5_DRIVER_SYND;
return -EIO;
@@ -501,6 +503,12 @@
MLX5_COMMAND_STR_CASE(QUERY_Q_COUNTER);
MLX5_COMMAND_STR_CASE(SET_RATE_LIMIT);
MLX5_COMMAND_STR_CASE(QUERY_RATE_LIMIT);
+ MLX5_COMMAND_STR_CASE(CREATE_SCHEDULING_ELEMENT);
+ MLX5_COMMAND_STR_CASE(DESTROY_SCHEDULING_ELEMENT);
+ MLX5_COMMAND_STR_CASE(QUERY_SCHEDULING_ELEMENT);
+ MLX5_COMMAND_STR_CASE(MODIFY_SCHEDULING_ELEMENT);
+ MLX5_COMMAND_STR_CASE(CREATE_QOS_PARA_VPORT);
+ MLX5_COMMAND_STR_CASE(DESTROY_QOS_PARA_VPORT);
MLX5_COMMAND_STR_CASE(ALLOC_PD);
MLX5_COMMAND_STR_CASE(DEALLOC_PD);
MLX5_COMMAND_STR_CASE(ALLOC_UAR);
@@ -576,12 +584,8 @@
MLX5_COMMAND_STR_CASE(MODIFY_FLOW_TABLE);
MLX5_COMMAND_STR_CASE(ALLOC_ENCAP_HEADER);
MLX5_COMMAND_STR_CASE(DEALLOC_ENCAP_HEADER);
- MLX5_COMMAND_STR_CASE(CREATE_SCHEDULING_ELEMENT);
- MLX5_COMMAND_STR_CASE(DESTROY_SCHEDULING_ELEMENT);
- MLX5_COMMAND_STR_CASE(QUERY_SCHEDULING_ELEMENT);
- MLX5_COMMAND_STR_CASE(MODIFY_SCHEDULING_ELEMENT);
- MLX5_COMMAND_STR_CASE(CREATE_QOS_PARA_VPORT);
- MLX5_COMMAND_STR_CASE(DESTROY_QOS_PARA_VPORT);
+ MLX5_COMMAND_STR_CASE(ALLOC_MODIFY_HEADER_CONTEXT);
+ MLX5_COMMAND_STR_CASE(DEALLOC_MODIFY_HEADER_CONTEXT);
default: return "unknown command opcode";
}
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index dc52053..0099a3e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -37,6 +37,7 @@
#include <linux/timecounter.h>
#include <linux/net_tstamp.h>
#include <linux/ptp_clock_kernel.h>
+#include <linux/crash_dump.h>
#include <linux/mlx5/driver.h>
#include <linux/mlx5/qp.h>
#include <linux/mlx5/cq.h>
@@ -90,7 +91,7 @@
#define MLX5E_VALID_NUM_MTTS(num_mtts) (MLX5_MTT_OCTW(num_mtts) - 1 <= U16_MAX)
#define MLX5_UMR_ALIGN (2048)
-#define MLX5_MPWRQ_SMALL_PACKET_THRESHOLD (128)
+#define MLX5_MPWRQ_SMALL_PACKET_THRESHOLD (256)
#define MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ (64 * 1024)
#define MLX5E_DEFAULT_LRO_TIMEOUT 32
@@ -111,18 +112,13 @@
#define MLX5E_MAX_NUM_SQS (MLX5E_MAX_NUM_CHANNELS * MLX5E_MAX_NUM_TC)
#define MLX5E_TX_CQ_POLL_BUDGET 128
#define MLX5E_UPDATE_STATS_INTERVAL 200 /* msecs */
-#define MLX5E_SQ_BF_BUDGET 16
#define MLX5E_ICOSQ_MAX_WQEBBS \
(DIV_ROUND_UP(sizeof(struct mlx5e_umr_wqe), MLX5_SEND_WQE_BB))
#define MLX5E_XDP_MIN_INLINE (ETH_HLEN + VLAN_HLEN)
-#define MLX5E_XDP_IHS_DS_COUNT \
- DIV_ROUND_UP(MLX5E_XDP_MIN_INLINE - 2, MLX5_SEND_WQE_DS)
#define MLX5E_XDP_TX_DS_COUNT \
((sizeof(struct mlx5e_tx_wqe) / MLX5_SEND_WQE_DS) + 1 /* SG DS */)
-#define MLX5E_XDP_TX_WQEBBS \
- DIV_ROUND_UP(MLX5E_XDP_TX_DS_COUNT, MLX5_SEND_WQEBB_NUM_DS)
#define MLX5E_NUM_MAIN_GROUPS 9
@@ -158,6 +154,14 @@
}
}
+static inline int mlx5e_get_max_num_channels(struct mlx5_core_dev *mdev)
+{
+ return is_kdump_kernel() ?
+ MLX5E_MIN_NUM_CHANNELS :
+ min_t(int, mdev->priv.eq_table.num_comp_vectors,
+ MLX5E_MAX_NUM_CHANNELS);
+}
+
struct mlx5e_tx_wqe {
struct mlx5_wqe_ctrl_seg ctrl;
struct mlx5_wqe_eth_seg eth;
@@ -187,15 +191,15 @@
MLX5E_PFLAG_RX_CQE_COMPRESS = (1 << 1),
};
-#define MLX5E_SET_PFLAG(priv, pflag, enable) \
+#define MLX5E_SET_PFLAG(params, pflag, enable) \
do { \
if (enable) \
- (priv)->params.pflags |= (pflag); \
+ (params)->pflags |= (pflag); \
else \
- (priv)->params.pflags &= ~(pflag); \
+ (params)->pflags &= ~(pflag); \
} while (0)
-#define MLX5E_GET_PFLAG(priv, pflag) (!!((priv)->params.pflags & (pflag)))
+#define MLX5E_GET_PFLAG(params, pflag) (!!((params)->pflags & (pflag)))
#ifdef CONFIG_MLX5_CORE_EN_DCB
#define MLX5E_MAX_BW_ALLOC 100 /* Max percentage of BW allocation */
@@ -218,7 +222,6 @@
bool rx_cqe_compress_def;
struct mlx5e_cq_moder rx_cq_moderation;
struct mlx5e_cq_moder tx_cq_moderation;
- u16 min_rx_wqes;
bool lro_en;
u32 lro_wqe_sz;
u16 tx_max_inline;
@@ -227,9 +230,11 @@
u8 toeplitz_hash_key[40];
u32 indirection_rqt[MLX5E_INDIR_RQT_SIZE];
bool vlan_strip_disable;
+ bool scatter_fcs_en;
bool rx_am_enabled;
u32 lro_timeout;
u32 pflags;
+ struct bpf_prog *xdp_prog;
};
#ifdef CONFIG_MLX5_CORE_EN_DCB
@@ -285,7 +290,6 @@
struct napi_struct *napi;
struct mlx5_core_cq mcq;
struct mlx5e_channel *channel;
- struct mlx5e_priv *priv;
/* cqe decompression */
struct mlx5_cqe64 title;
@@ -295,22 +299,163 @@
u16 decmprs_wqe_counter;
/* control */
+ struct mlx5_core_dev *mdev;
struct mlx5_frag_wq_ctrl wq_ctrl;
} ____cacheline_aligned_in_smp;
-struct mlx5e_rq;
-typedef void (*mlx5e_fp_handle_rx_cqe)(struct mlx5e_rq *rq,
- struct mlx5_cqe64 *cqe);
-typedef int (*mlx5e_fp_alloc_wqe)(struct mlx5e_rq *rq, struct mlx5e_rx_wqe *wqe,
- u16 ix);
+struct mlx5e_tx_wqe_info {
+ struct sk_buff *skb;
+ u32 num_bytes;
+ u8 num_wqebbs;
+ u8 num_dma;
+};
-typedef void (*mlx5e_fp_dealloc_wqe)(struct mlx5e_rq *rq, u16 ix);
+enum mlx5e_dma_map_type {
+ MLX5E_DMA_MAP_SINGLE,
+ MLX5E_DMA_MAP_PAGE
+};
+
+struct mlx5e_sq_dma {
+ dma_addr_t addr;
+ u32 size;
+ enum mlx5e_dma_map_type type;
+};
+
+enum {
+ MLX5E_SQ_STATE_ENABLED,
+};
+
+struct mlx5e_sq_wqe_info {
+ u8 opcode;
+ u8 num_wqebbs;
+};
+
+struct mlx5e_txqsq {
+ /* data path */
+
+ /* dirtied @completion */
+ u16 cc;
+ u32 dma_fifo_cc;
+
+ /* dirtied @xmit */
+ u16 pc ____cacheline_aligned_in_smp;
+ u32 dma_fifo_pc;
+ struct mlx5e_sq_stats stats;
+
+ struct mlx5e_cq cq;
+
+ /* write@xmit, read@completion */
+ struct {
+ struct mlx5e_sq_dma *dma_fifo;
+ struct mlx5e_tx_wqe_info *wqe_info;
+ } db;
+
+ /* read only */
+ struct mlx5_wq_cyc wq;
+ u32 dma_fifo_mask;
+ void __iomem *uar_map;
+ struct netdev_queue *txq;
+ u32 sqn;
+ u16 max_inline;
+ u8 min_inline_mode;
+ u16 edge;
+ struct device *pdev;
+ struct mlx5e_tstamp *tstamp;
+ __be32 mkey_be;
+ unsigned long state;
+
+ /* control path */
+ struct mlx5_wq_ctrl wq_ctrl;
+ struct mlx5e_channel *channel;
+ int txq_ix;
+ u32 rate_limit;
+} ____cacheline_aligned_in_smp;
+
+struct mlx5e_xdpsq {
+ /* data path */
+
+ /* dirtied @rx completion */
+ u16 cc;
+ u16 pc;
+
+ struct mlx5e_cq cq;
+
+ /* write@xmit, read@completion */
+ struct {
+ struct mlx5e_dma_info *di;
+ bool doorbell;
+ } db;
+
+ /* read only */
+ struct mlx5_wq_cyc wq;
+ void __iomem *uar_map;
+ u32 sqn;
+ struct device *pdev;
+ __be32 mkey_be;
+ u8 min_inline_mode;
+ unsigned long state;
+
+ /* control path */
+ struct mlx5_wq_ctrl wq_ctrl;
+ struct mlx5e_channel *channel;
+} ____cacheline_aligned_in_smp;
+
+struct mlx5e_icosq {
+ /* data path */
+
+ /* dirtied @completion */
+ u16 cc;
+
+ /* dirtied @xmit */
+ u16 pc ____cacheline_aligned_in_smp;
+ u32 dma_fifo_pc;
+ u16 prev_cc;
+
+ struct mlx5e_cq cq;
+
+ /* write@xmit, read@completion */
+ struct {
+ struct mlx5e_sq_wqe_info *ico_wqe;
+ } db;
+
+ /* read only */
+ struct mlx5_wq_cyc wq;
+ void __iomem *uar_map;
+ u32 sqn;
+ u16 edge;
+ struct device *pdev;
+ __be32 mkey_be;
+ unsigned long state;
+
+ /* control path */
+ struct mlx5_wq_ctrl wq_ctrl;
+ struct mlx5e_channel *channel;
+} ____cacheline_aligned_in_smp;
+
+static inline bool
+mlx5e_wqc_has_room_for(struct mlx5_wq_cyc *wq, u16 cc, u16 pc, u16 n)
+{
+ return (((wq->sz_m1 & (cc - pc)) >= n) || (cc == pc));
+}
struct mlx5e_dma_info {
struct page *page;
dma_addr_t addr;
};
+struct mlx5e_umr_dma_info {
+ __be64 *mtt;
+ dma_addr_t mtt_addr;
+ struct mlx5e_dma_info dma_info[MLX5_MPWRQ_PAGES_PER_WQE];
+ struct mlx5e_umr_wqe wqe;
+};
+
+struct mlx5e_mpw_info {
+ struct mlx5e_umr_dma_info umr;
+ u16 consumed_strides;
+ u16 skbs_frags[MLX5_MPWRQ_PAGES_PER_WQE];
+};
+
struct mlx5e_rx_am_stats {
int ppms; /* packets per msec */
int epms; /* events per msec */
@@ -347,6 +492,11 @@
struct mlx5e_dma_info page_cache[MLX5E_CACHE_SIZE];
};
+struct mlx5e_rq;
+typedef void (*mlx5e_fp_handle_rx_cqe)(struct mlx5e_rq*, struct mlx5_cqe64*);
+typedef int (*mlx5e_fp_alloc_wqe)(struct mlx5e_rq*, struct mlx5e_rx_wqe*, u16);
+typedef void (*mlx5e_fp_dealloc_wqe)(struct mlx5e_rq*, u16);
+
struct mlx5e_rq {
/* data path */
struct mlx5_wq_ll wq;
@@ -381,7 +531,10 @@
u16 rx_headroom;
struct mlx5e_rx_am am; /* Adaptive Moderation */
+
+ /* XDP */
struct bpf_prog *xdp_prog;
+ struct mlx5e_xdpsq xdpsq;
/* control */
struct mlx5_wq_ctrl wq_ctrl;
@@ -390,118 +543,10 @@
u32 mpwqe_num_strides;
u32 rqn;
struct mlx5e_channel *channel;
- struct mlx5e_priv *priv;
+ struct mlx5_core_dev *mdev;
struct mlx5_core_mkey umr_mkey;
} ____cacheline_aligned_in_smp;
-struct mlx5e_umr_dma_info {
- __be64 *mtt;
- dma_addr_t mtt_addr;
- struct mlx5e_dma_info dma_info[MLX5_MPWRQ_PAGES_PER_WQE];
- struct mlx5e_umr_wqe wqe;
-};
-
-struct mlx5e_mpw_info {
- struct mlx5e_umr_dma_info umr;
- u16 consumed_strides;
- u16 skbs_frags[MLX5_MPWRQ_PAGES_PER_WQE];
-};
-
-struct mlx5e_tx_wqe_info {
- u32 num_bytes;
- u8 num_wqebbs;
- u8 num_dma;
-};
-
-enum mlx5e_dma_map_type {
- MLX5E_DMA_MAP_SINGLE,
- MLX5E_DMA_MAP_PAGE
-};
-
-struct mlx5e_sq_dma {
- dma_addr_t addr;
- u32 size;
- enum mlx5e_dma_map_type type;
-};
-
-enum {
- MLX5E_SQ_STATE_ENABLED,
- MLX5E_SQ_STATE_BF_ENABLE,
-};
-
-struct mlx5e_sq_wqe_info {
- u8 opcode;
- u8 num_wqebbs;
-};
-
-enum mlx5e_sq_type {
- MLX5E_SQ_TXQ,
- MLX5E_SQ_ICO,
- MLX5E_SQ_XDP
-};
-
-struct mlx5e_sq {
- /* data path */
-
- /* dirtied @completion */
- u16 cc;
- u32 dma_fifo_cc;
-
- /* dirtied @xmit */
- u16 pc ____cacheline_aligned_in_smp;
- u32 dma_fifo_pc;
- u16 bf_offset;
- u16 prev_cc;
- u8 bf_budget;
- struct mlx5e_sq_stats stats;
-
- struct mlx5e_cq cq;
-
- /* pointers to per tx element info: write@xmit, read@completion */
- union {
- struct {
- struct sk_buff **skb;
- struct mlx5e_sq_dma *dma_fifo;
- struct mlx5e_tx_wqe_info *wqe_info;
- } txq;
- struct mlx5e_sq_wqe_info *ico_wqe;
- struct {
- struct mlx5e_sq_wqe_info *wqe_info;
- struct mlx5e_dma_info *di;
- bool doorbell;
- } xdp;
- } db;
-
- /* read only */
- struct mlx5_wq_cyc wq;
- u32 dma_fifo_mask;
- void __iomem *uar_map;
- struct netdev_queue *txq;
- u32 sqn;
- u16 bf_buf_size;
- u16 max_inline;
- u8 min_inline_mode;
- u16 edge;
- struct device *pdev;
- struct mlx5e_tstamp *tstamp;
- __be32 mkey_be;
- unsigned long state;
-
- /* control path */
- struct mlx5_wq_ctrl wq_ctrl;
- struct mlx5_sq_bfreg bfreg;
- struct mlx5e_channel *channel;
- int tc;
- u32 rate_limit;
- u8 type;
-} ____cacheline_aligned_in_smp;
-
-static inline bool mlx5e_sq_has_room_for(struct mlx5e_sq *sq, u16 n)
-{
- return (((sq->wq.sz_m1 & (sq->cc - sq->pc)) >= n) ||
- (sq->cc == sq->pc));
-}
-
enum channel_flags {
MLX5E_CHANNEL_NAPI_SCHED = 1,
};
@@ -509,9 +554,8 @@
struct mlx5e_channel {
/* data path */
struct mlx5e_rq rq;
- struct mlx5e_sq xdp_sq;
- struct mlx5e_sq sq[MLX5E_MAX_NUM_TC];
- struct mlx5e_sq icosq; /* internal control operations */
+ struct mlx5e_txqsq sq[MLX5E_MAX_NUM_TC];
+ struct mlx5e_icosq icosq; /* internal control operations */
bool xdp;
struct napi_struct napi;
struct device *pdev;
@@ -522,10 +566,18 @@
/* control */
struct mlx5e_priv *priv;
+ struct mlx5_core_dev *mdev;
+ struct mlx5e_tstamp *tstamp;
int ix;
int cpu;
};
+struct mlx5e_channels {
+ struct mlx5e_channel **c;
+ unsigned int num;
+ struct mlx5e_params params;
+};
+
enum mlx5e_traffic_types {
MLX5E_TT_IPV4_TCP,
MLX5E_TT_IPV6_TCP,
@@ -675,34 +727,17 @@
MLX5E_NIC_PRIO
};
-struct mlx5e_profile {
- void (*init)(struct mlx5_core_dev *mdev,
- struct net_device *netdev,
- const struct mlx5e_profile *profile, void *ppriv);
- void (*cleanup)(struct mlx5e_priv *priv);
- int (*init_rx)(struct mlx5e_priv *priv);
- void (*cleanup_rx)(struct mlx5e_priv *priv);
- int (*init_tx)(struct mlx5e_priv *priv);
- void (*cleanup_tx)(struct mlx5e_priv *priv);
- void (*enable)(struct mlx5e_priv *priv);
- void (*disable)(struct mlx5e_priv *priv);
- void (*update_stats)(struct mlx5e_priv *priv);
- int (*max_nch)(struct mlx5_core_dev *mdev);
- int max_tc;
-};
-
struct mlx5e_priv {
/* priv data path fields - start */
- struct mlx5e_sq **txq_to_sq_map;
- int channeltc_to_txq_map[MLX5E_MAX_NUM_CHANNELS][MLX5E_MAX_NUM_TC];
- struct bpf_prog *xdp_prog;
+ struct mlx5e_txqsq *txq2sq[MLX5E_MAX_NUM_CHANNELS * MLX5E_MAX_NUM_TC];
+ int channel_tc2txq[MLX5E_MAX_NUM_CHANNELS][MLX5E_MAX_NUM_TC];
/* priv data path fields - end */
unsigned long state;
struct mutex state_lock; /* Protects Interface state */
struct mlx5e_rq drop_rq;
- struct mlx5e_channel **channel;
+ struct mlx5e_channels channels;
u32 tisn[MLX5E_MAX_NUM_TC];
struct mlx5e_rqt indir_rqt;
struct mlx5e_tir indir_tir[MLX5E_NUM_INDIR_TIRS];
@@ -712,7 +747,6 @@
struct mlx5e_flow_steering fs;
struct mlx5e_vxlan_db vxlan;
- struct mlx5e_params params;
struct workqueue_struct *wq;
struct work_struct update_carrier_work;
struct work_struct set_rx_mode_work;
@@ -732,9 +766,28 @@
void *ppriv;
};
+struct mlx5e_profile {
+ void (*init)(struct mlx5_core_dev *mdev,
+ struct net_device *netdev,
+ const struct mlx5e_profile *profile, void *ppriv);
+ void (*cleanup)(struct mlx5e_priv *priv);
+ int (*init_rx)(struct mlx5e_priv *priv);
+ void (*cleanup_rx)(struct mlx5e_priv *priv);
+ int (*init_tx)(struct mlx5e_priv *priv);
+ void (*cleanup_tx)(struct mlx5e_priv *priv);
+ void (*enable)(struct mlx5e_priv *priv);
+ void (*disable)(struct mlx5e_priv *priv);
+ void (*update_stats)(struct mlx5e_priv *priv);
+ int (*max_nch)(struct mlx5_core_dev *mdev);
+ struct {
+ mlx5e_fp_handle_rx_cqe handle_rx_cqe;
+ mlx5e_fp_handle_rx_cqe handle_rx_cqe_mpwqe;
+ } rx_handlers;
+ int max_tc;
+};
+
void mlx5e_build_ptys2ethtool_map(void);
-void mlx5e_send_nop(struct mlx5e_sq *sq, bool notify_hw);
u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb,
void *accel_priv, select_queue_fallback_t fallback);
netdev_tx_t mlx5e_xmit(struct sk_buff *skb, struct net_device *dev);
@@ -744,7 +797,9 @@
int mlx5e_napi_poll(struct napi_struct *napi, int budget);
bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget);
int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget);
-void mlx5e_free_sq_descs(struct mlx5e_sq *sq);
+bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq);
+void mlx5e_free_txqsq_descs(struct mlx5e_txqsq *sq);
+void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq);
void mlx5e_page_release(struct mlx5e_rq *rq, struct mlx5e_dma_info *dma_info,
bool recycle);
@@ -792,7 +847,7 @@
struct ptp_clock_event *event);
int mlx5e_hwstamp_set(struct net_device *dev, struct ifreq *ifr);
int mlx5e_hwstamp_get(struct net_device *dev, struct ifreq *ifr);
-void mlx5e_modify_rx_cqe_compression_locked(struct mlx5e_priv *priv, bool val);
+int mlx5e_modify_rx_cqe_compression_locked(struct mlx5e_priv *priv, bool val);
int mlx5e_vlan_rx_add_vid(struct net_device *dev, __always_unused __be16 proto,
u16 vid);
@@ -801,14 +856,40 @@
void mlx5e_enable_vlan_filter(struct mlx5e_priv *priv);
void mlx5e_disable_vlan_filter(struct mlx5e_priv *priv);
-int mlx5e_modify_rqs_vsd(struct mlx5e_priv *priv, bool vsd);
+struct mlx5e_redirect_rqt_param {
+ bool is_rss;
+ union {
+ u32 rqn; /* Direct RQN (Non-RSS) */
+ struct {
+ u8 hfunc;
+ struct mlx5e_channels *channels;
+ } rss; /* RSS data */
+ };
+};
-int mlx5e_redirect_rqt(struct mlx5e_priv *priv, u32 rqtn, int sz, int ix);
-void mlx5e_build_indir_tir_ctx_hash(struct mlx5e_priv *priv, void *tirc,
- enum mlx5e_traffic_types tt);
+int mlx5e_redirect_rqt(struct mlx5e_priv *priv, u32 rqtn, int sz,
+ struct mlx5e_redirect_rqt_param rrp);
+void mlx5e_build_indir_tir_ctx_hash(struct mlx5e_params *params,
+ enum mlx5e_traffic_types tt,
+ void *tirc);
int mlx5e_open_locked(struct net_device *netdev);
int mlx5e_close_locked(struct net_device *netdev);
+
+int mlx5e_open_channels(struct mlx5e_priv *priv,
+ struct mlx5e_channels *chs);
+void mlx5e_close_channels(struct mlx5e_channels *chs);
+
+/* Function pointer to be used to modify WH settings while
+ * switching channels
+ */
+typedef int (*mlx5e_fp_hw_modify)(struct mlx5e_priv *priv);
+void mlx5e_switch_priv_channels(struct mlx5e_priv *priv,
+ struct mlx5e_channels *new_chs,
+ mlx5e_fp_hw_modify hw_modify);
+void mlx5e_activate_priv_channels(struct mlx5e_priv *priv);
+void mlx5e_deactivate_priv_channels(struct mlx5e_priv *priv);
+
void mlx5e_build_default_indir_rqt(struct mlx5_core_dev *mdev,
u32 *indirection_rqt, int len,
int num_channels);
@@ -816,30 +897,43 @@
void mlx5e_set_rx_cq_mode_params(struct mlx5e_params *params,
u8 cq_period_mode);
-void mlx5e_set_rq_type_params(struct mlx5e_priv *priv, u8 rq_type);
+void mlx5e_set_rq_type_params(struct mlx5_core_dev *mdev,
+ struct mlx5e_params *params, u8 rq_type);
-static inline void mlx5e_tx_notify_hw(struct mlx5e_sq *sq,
- struct mlx5_wqe_ctrl_seg *ctrl, int bf_sz)
+static inline
+struct mlx5e_tx_wqe *mlx5e_post_nop(struct mlx5_wq_cyc *wq, u32 sqn, u16 *pc)
{
- u16 ofst = sq->bf_offset;
+ u16 pi = *pc & wq->sz_m1;
+ struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(wq, pi);
+ struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+ memset(cseg, 0, sizeof(*cseg));
+
+ cseg->opmod_idx_opcode = cpu_to_be32((*pc << 8) | MLX5_OPCODE_NOP);
+ cseg->qpn_ds = cpu_to_be32((sqn << 8) | 0x01);
+
+ (*pc)++;
+
+ return wqe;
+}
+
+static inline
+void mlx5e_notify_hw(struct mlx5_wq_cyc *wq, u16 pc,
+ void __iomem *uar_map,
+ struct mlx5_wqe_ctrl_seg *ctrl)
+{
+ ctrl->fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE;
/* ensure wqe is visible to device before updating doorbell record */
dma_wmb();
- *sq->wq.db = cpu_to_be32(sq->pc);
+ *wq->db = cpu_to_be32(pc);
/* ensure doorbell record is visible to device before ringing the
* doorbell
*/
wmb();
- if (bf_sz)
- __iowrite64_copy(sq->uar_map + ofst, ctrl, bf_sz);
- else
- mlx5_write64((__be32 *)ctrl, sq->uar_map + ofst, NULL);
- /* flush the write-combining mapped buffer */
- wmb();
- sq->bf_offset ^= sq->bf_buf_size;
+ mlx5_write64((__be32 *)ctrl, uar_map, NULL);
}
static inline void mlx5e_cq_arm(struct mlx5e_cq *cq)
@@ -895,44 +989,43 @@
struct mlx5e_tir *tir);
int mlx5e_create_mdev_resources(struct mlx5_core_dev *mdev);
void mlx5e_destroy_mdev_resources(struct mlx5_core_dev *mdev);
-int mlx5e_refresh_tirs_self_loopback(struct mlx5_core_dev *mdev,
- bool enable_uc_lb);
+int mlx5e_refresh_tirs(struct mlx5e_priv *priv, bool enable_uc_lb);
-struct mlx5_eswitch_rep;
-int mlx5e_vport_rep_load(struct mlx5_eswitch *esw,
- struct mlx5_eswitch_rep *rep);
-void mlx5e_vport_rep_unload(struct mlx5_eswitch *esw,
- struct mlx5_eswitch_rep *rep);
-int mlx5e_nic_rep_load(struct mlx5_eswitch *esw, struct mlx5_eswitch_rep *rep);
-void mlx5e_nic_rep_unload(struct mlx5_eswitch *esw,
- struct mlx5_eswitch_rep *rep);
-int mlx5e_add_sqs_fwd_rules(struct mlx5e_priv *priv);
-void mlx5e_remove_sqs_fwd_rules(struct mlx5e_priv *priv);
-int mlx5e_attr_get(struct net_device *dev, struct switchdev_attr *attr);
-void mlx5e_handle_rx_cqe_rep(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe);
-void mlx5e_update_hw_rep_counters(struct mlx5e_priv *priv);
+/* common netdev helpers */
+int mlx5e_create_indirect_rqt(struct mlx5e_priv *priv);
+
+int mlx5e_create_indirect_tirs(struct mlx5e_priv *priv);
+void mlx5e_destroy_indirect_tirs(struct mlx5e_priv *priv);
int mlx5e_create_direct_rqts(struct mlx5e_priv *priv);
-void mlx5e_destroy_rqt(struct mlx5e_priv *priv, struct mlx5e_rqt *rqt);
+void mlx5e_destroy_direct_rqts(struct mlx5e_priv *priv);
int mlx5e_create_direct_tirs(struct mlx5e_priv *priv);
void mlx5e_destroy_direct_tirs(struct mlx5e_priv *priv);
+void mlx5e_destroy_rqt(struct mlx5e_priv *priv, struct mlx5e_rqt *rqt);
+
+int mlx5e_create_ttc_table(struct mlx5e_priv *priv, u32 underlay_qpn);
+void mlx5e_destroy_ttc_table(struct mlx5e_priv *priv);
+
+int mlx5e_create_tis(struct mlx5_core_dev *mdev, int tc,
+ u32 underlay_qpn, u32 *tisn);
+void mlx5e_destroy_tis(struct mlx5_core_dev *mdev, u32 tisn);
+
int mlx5e_create_tises(struct mlx5e_priv *priv);
void mlx5e_cleanup_nic_tx(struct mlx5e_priv *priv);
int mlx5e_close(struct net_device *netdev);
int mlx5e_open(struct net_device *netdev);
void mlx5e_update_stats_work(struct work_struct *work);
-struct net_device *mlx5e_create_netdev(struct mlx5_core_dev *mdev,
- const struct mlx5e_profile *profile,
- void *ppriv);
-void mlx5e_destroy_netdev(struct mlx5_core_dev *mdev, struct mlx5e_priv *priv);
-int mlx5e_attach_netdev(struct mlx5_core_dev *mdev, struct net_device *netdev);
-void mlx5e_detach_netdev(struct mlx5_core_dev *mdev, struct net_device *netdev);
u32 mlx5e_choose_lro_timeout(struct mlx5_core_dev *mdev, u32 wanted_timeout);
-int mlx5e_get_offload_stats(int attr_id, const struct net_device *dev,
- void *sp);
-bool mlx5e_has_offload_stats(const struct net_device *dev, int attr_id);
+/* mlx5e generic netdev management API */
+struct net_device*
+mlx5e_create_netdev(struct mlx5_core_dev *mdev, const struct mlx5e_profile *profile,
+ void *ppriv);
+int mlx5e_attach_netdev(struct mlx5e_priv *priv);
+void mlx5e_detach_netdev(struct mlx5e_priv *priv);
+void mlx5e_destroy_netdev(struct mlx5e_priv *priv);
+void mlx5e_build_nic_params(struct mlx5_core_dev *mdev,
+ struct mlx5e_params *params,
+ u16 max_channels);
-bool mlx5e_is_uplink_rep(struct mlx5e_priv *priv);
-bool mlx5e_is_vf_vport_rep(struct mlx5e_priv *priv);
#endif /* __MLX5_EN_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
index 68419a0..c8a0053 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
@@ -174,13 +174,9 @@
enum arfs_type type)
{
struct arfs_table *arfs_t = &priv->fs.arfs.arfs_tables[type];
- struct mlx5_flow_act flow_act = {
- .action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- .flow_tag = MLX5_FS_DEFAULT_FLOW_TAG,
- .encap_id = 0,
- };
- struct mlx5_flow_destination dest;
struct mlx5e_tir *tir = priv->indir_tir;
+ struct mlx5_flow_destination dest;
+ MLX5_DECLARE_FLOW_ACT(flow_act);
struct mlx5_flow_spec *spec;
int err = 0;
@@ -325,10 +321,16 @@
{
struct mlx5e_arfs_tables *arfs = &priv->fs.arfs;
struct mlx5e_flow_table *ft = &arfs->arfs_tables[type].ft;
+ struct mlx5_flow_table_attr ft_attr = {};
int err;
- ft->t = mlx5_create_flow_table(priv->fs.ns, MLX5E_NIC_PRIO,
- MLX5E_ARFS_TABLE_SIZE, MLX5E_ARFS_FT_LEVEL, 0);
+ ft->num_groups = 0;
+
+ ft_attr.max_fte = MLX5E_ARFS_TABLE_SIZE;
+ ft_attr.level = MLX5E_ARFS_FT_LEVEL;
+ ft_attr.prio = MLX5E_NIC_PRIO;
+
+ ft->t = mlx5_create_flow_table(priv->fs.ns, &ft_attr);
if (IS_ERR(ft->t)) {
err = PTR_ERR(ft->t);
ft->t = NULL;
@@ -469,15 +471,11 @@
static struct mlx5_flow_handle *arfs_add_rule(struct mlx5e_priv *priv,
struct arfs_rule *arfs_rule)
{
- struct mlx5_flow_act flow_act = {
- .action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- .flow_tag = MLX5_FS_DEFAULT_FLOW_TAG,
- .encap_id = 0,
- };
struct mlx5e_arfs_tables *arfs = &priv->fs.arfs;
struct arfs_tuple *tuple = &arfs_rule->tuple;
struct mlx5_flow_handle *rule = NULL;
struct mlx5_flow_destination dest;
+ MLX5_DECLARE_FLOW_ACT(flow_act);
struct arfs_table *arfs_table;
struct mlx5_flow_spec *spec;
struct mlx5_flow_table *ft;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_clock.c b/drivers/net/ethernet/mellanox/mlx5/core/en_clock.c
index 37e66ee..e706a87 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_clock.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_clock.c
@@ -90,6 +90,7 @@
{
struct mlx5e_priv *priv = netdev_priv(dev);
struct hwtstamp_config config;
+ int err;
if (!MLX5_CAP_GEN(priv->mdev, device_frequency_khz))
return -EOPNOTSUPP;
@@ -111,7 +112,7 @@
switch (config.rx_filter) {
case HWTSTAMP_FILTER_NONE:
/* Reset CQE compression to Admin default */
- mlx5e_modify_rx_cqe_compression_locked(priv, priv->params.rx_cqe_compress_def);
+ mlx5e_modify_rx_cqe_compression_locked(priv, priv->channels.params.rx_cqe_compress_def);
break;
case HWTSTAMP_FILTER_ALL:
case HWTSTAMP_FILTER_SOME:
@@ -129,7 +130,12 @@
case HWTSTAMP_FILTER_PTP_V2_DELAY_REQ:
/* Disable CQE compression */
netdev_warn(dev, "Disabling cqe compression");
- mlx5e_modify_rx_cqe_compression_locked(priv, false);
+ err = mlx5e_modify_rx_cqe_compression_locked(priv, false);
+ if (err) {
+ netdev_err(dev, "Failed disabling cqe compression err=%d\n", err);
+ mutex_unlock(&priv->state_lock);
+ return err;
+ }
config.rx_filter = HWTSTAMP_FILTER_ALL;
break;
default:
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
index bd898d8..f1f17f7 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
@@ -107,10 +107,18 @@
goto err_dealloc_transport_domain;
}
+ err = mlx5_alloc_bfreg(mdev, &res->bfreg, false, false);
+ if (err) {
+ mlx5_core_err(mdev, "alloc bfreg failed, %d\n", err);
+ goto err_destroy_mkey;
+ }
+
INIT_LIST_HEAD(&mdev->mlx5e_res.td.tirs_list);
return 0;
+err_destroy_mkey:
+ mlx5_core_destroy_mkey(mdev, &res->mkey);
err_dealloc_transport_domain:
mlx5_core_dealloc_transport_domain(mdev, res->td.tdn);
err_dealloc_pd:
@@ -122,23 +130,26 @@
{
struct mlx5e_resources *res = &mdev->mlx5e_res;
+ mlx5_free_bfreg(mdev, &res->bfreg);
mlx5_core_destroy_mkey(mdev, &res->mkey);
mlx5_core_dealloc_transport_domain(mdev, res->td.tdn);
mlx5_core_dealloc_pd(mdev, res->pdn);
}
-int mlx5e_refresh_tirs_self_loopback(struct mlx5_core_dev *mdev,
- bool enable_uc_lb)
+int mlx5e_refresh_tirs(struct mlx5e_priv *priv, bool enable_uc_lb)
{
+ struct mlx5_core_dev *mdev = priv->mdev;
struct mlx5e_tir *tir;
- void *in;
+ int err = -ENOMEM;
+ u32 tirn = 0;
int inlen;
- int err = 0;
+ void *in;
+
inlen = MLX5_ST_SZ_BYTES(modify_tir_in);
in = mlx5_vzalloc(inlen);
if (!in)
- return -ENOMEM;
+ goto out;
if (enable_uc_lb)
MLX5_SET(modify_tir_in, in, ctx.self_lb_block,
@@ -147,13 +158,16 @@
MLX5_SET(modify_tir_in, in, bitmask.self_lb_en, 1);
list_for_each_entry(tir, &mdev->mlx5e_res.td.tirs_list, list) {
- err = mlx5_core_modify_tir(mdev, tir->tirn, in, inlen);
+ tirn = tir->tirn;
+ err = mlx5_core_modify_tir(mdev, tirn, in, inlen);
if (err)
goto out;
}
out:
kvfree(in);
+ if (err)
+ netdev_err(priv->netdev, "refresh tir(0x%x) failed, %d\n", tirn, err);
return err;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
index a004a5a..ce7b09d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
@@ -42,8 +42,9 @@
strlcpy(drvinfo->version, DRIVER_VERSION " (" DRIVER_RELDATE ")",
sizeof(drvinfo->version));
snprintf(drvinfo->fw_version, sizeof(drvinfo->fw_version),
- "%d.%d.%d",
- fw_rev_maj(mdev), fw_rev_min(mdev), fw_rev_sub(mdev));
+ "%d.%d.%04d (%.16s)",
+ fw_rev_maj(mdev), fw_rev_min(mdev), fw_rev_sub(mdev),
+ mdev->board_id);
strlcpy(drvinfo->bus_info, pci_name(mdev->pdev),
sizeof(drvinfo->bus_info));
}
@@ -152,12 +153,9 @@
}
#define MLX5E_NUM_Q_CNTRS(priv) (NUM_Q_COUNTERS * (!!priv->q_counter))
-#define MLX5E_NUM_RQ_STATS(priv) \
- (NUM_RQ_STATS * priv->params.num_channels * \
- test_bit(MLX5E_STATE_OPENED, &priv->state))
+#define MLX5E_NUM_RQ_STATS(priv) (NUM_RQ_STATS * (priv)->channels.num)
#define MLX5E_NUM_SQ_STATS(priv) \
- (NUM_SQ_STATS * priv->params.num_channels * priv->params.num_tc * \
- test_bit(MLX5E_STATE_OPENED, &priv->state))
+ (NUM_SQ_STATS * (priv)->channels.num * (priv)->channels.params.num_tc)
#define MLX5E_NUM_PFC_COUNTERS(priv) \
((mlx5e_query_global_pause_combined(priv) + hweight8(mlx5e_query_pfc_combined(priv))) * \
NUM_PPORT_PER_PRIO_PFC_COUNTERS)
@@ -262,17 +260,17 @@
return;
/* per channel counters */
- for (i = 0; i < priv->params.num_channels; i++)
+ for (i = 0; i < priv->channels.num; i++)
for (j = 0; j < NUM_RQ_STATS; j++)
sprintf(data + (idx++) * ETH_GSTRING_LEN,
rq_stats_desc[j].format, i);
- for (tc = 0; tc < priv->params.num_tc; tc++)
- for (i = 0; i < priv->params.num_channels; i++)
+ for (tc = 0; tc < priv->channels.params.num_tc; tc++)
+ for (i = 0; i < priv->channels.num; i++)
for (j = 0; j < NUM_SQ_STATS; j++)
sprintf(data + (idx++) * ETH_GSTRING_LEN,
sq_stats_desc[j].format,
- priv->channeltc_to_txq_map[i][tc]);
+ priv->channel_tc2txq[i][tc]);
}
static void mlx5e_get_strings(struct net_device *dev,
@@ -303,6 +301,7 @@
struct ethtool_stats *stats, u64 *data)
{
struct mlx5e_priv *priv = netdev_priv(dev);
+ struct mlx5e_channels *channels;
struct mlx5_priv *mlx5_priv;
int i, j, tc, prio, idx = 0;
unsigned long pfc_combined;
@@ -313,6 +312,7 @@
mutex_lock(&priv->state_lock);
if (test_bit(MLX5E_STATE_OPENED, &priv->state))
mlx5e_update_stats(priv);
+ channels = &priv->channels;
mutex_unlock(&priv->state_lock);
for (i = 0; i < NUM_SW_COUNTERS; i++)
@@ -382,16 +382,16 @@
return;
/* per channel counters */
- for (i = 0; i < priv->params.num_channels; i++)
+ for (i = 0; i < channels->num; i++)
for (j = 0; j < NUM_RQ_STATS; j++)
data[idx++] =
- MLX5E_READ_CTR64_CPU(&priv->channel[i]->rq.stats,
+ MLX5E_READ_CTR64_CPU(&channels->c[i]->rq.stats,
rq_stats_desc, j);
- for (tc = 0; tc < priv->params.num_tc; tc++)
- for (i = 0; i < priv->params.num_channels; i++)
+ for (tc = 0; tc < priv->channels.params.num_tc; tc++)
+ for (i = 0; i < channels->num; i++)
for (j = 0; j < NUM_SQ_STATS; j++)
- data[idx++] = MLX5E_READ_CTR64_CPU(&priv->channel[i]->sq[tc].stats,
+ data[idx++] = MLX5E_READ_CTR64_CPU(&channels->c[i]->sq[tc].stats,
sq_stats_desc, j);
}
@@ -406,8 +406,8 @@
if (rq_wq_type != MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ)
return num_wqe;
- stride_size = 1 << priv->params.mpwqe_log_stride_sz;
- num_strides = 1 << priv->params.mpwqe_log_num_strides;
+ stride_size = 1 << priv->channels.params.mpwqe_log_stride_sz;
+ num_strides = 1 << priv->channels.params.mpwqe_log_num_strides;
wqe_size = stride_size * num_strides;
packets_per_wqe = wqe_size /
@@ -427,8 +427,8 @@
if (rq_wq_type != MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ)
return num_packets;
- stride_size = 1 << priv->params.mpwqe_log_stride_sz;
- num_strides = 1 << priv->params.mpwqe_log_num_strides;
+ stride_size = 1 << priv->channels.params.mpwqe_log_stride_sz;
+ num_strides = 1 << priv->channels.params.mpwqe_log_num_strides;
wqe_size = stride_size * num_strides;
num_packets = (1 << order_base_2(num_packets));
@@ -443,26 +443,25 @@
struct ethtool_ringparam *param)
{
struct mlx5e_priv *priv = netdev_priv(dev);
- int rq_wq_type = priv->params.rq_wq_type;
+ int rq_wq_type = priv->channels.params.rq_wq_type;
param->rx_max_pending = mlx5e_rx_wqes_to_packets(priv, rq_wq_type,
1 << mlx5_max_log_rq_size(rq_wq_type));
param->tx_max_pending = 1 << MLX5E_PARAMS_MAXIMUM_LOG_SQ_SIZE;
param->rx_pending = mlx5e_rx_wqes_to_packets(priv, rq_wq_type,
- 1 << priv->params.log_rq_size);
- param->tx_pending = 1 << priv->params.log_sq_size;
+ 1 << priv->channels.params.log_rq_size);
+ param->tx_pending = 1 << priv->channels.params.log_sq_size;
}
static int mlx5e_set_ringparam(struct net_device *dev,
struct ethtool_ringparam *param)
{
struct mlx5e_priv *priv = netdev_priv(dev);
- bool was_opened;
- int rq_wq_type = priv->params.rq_wq_type;
+ int rq_wq_type = priv->channels.params.rq_wq_type;
+ struct mlx5e_channels new_channels = {};
u32 rx_pending_wqes;
u32 min_rq_size;
u32 max_rq_size;
- u16 min_rx_wqes;
u8 log_rq_size;
u8 log_sq_size;
u32 num_mtts;
@@ -500,7 +499,7 @@
}
num_mtts = MLX5E_REQUIRED_MTTS(rx_pending_wqes);
- if (priv->params.rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ &&
+ if (priv->channels.params.rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ &&
!MLX5E_VALID_NUM_MTTS(num_mtts)) {
netdev_info(dev, "%s: rx_pending (%d) request can't be satisfied, try to reduce.\n",
__func__, param->rx_pending);
@@ -522,26 +521,29 @@
log_rq_size = order_base_2(rx_pending_wqes);
log_sq_size = order_base_2(param->tx_pending);
- min_rx_wqes = mlx5_min_rx_wqes(rq_wq_type, rx_pending_wqes);
- if (log_rq_size == priv->params.log_rq_size &&
- log_sq_size == priv->params.log_sq_size &&
- min_rx_wqes == priv->params.min_rx_wqes)
+ if (log_rq_size == priv->channels.params.log_rq_size &&
+ log_sq_size == priv->channels.params.log_sq_size)
return 0;
mutex_lock(&priv->state_lock);
- was_opened = test_bit(MLX5E_STATE_OPENED, &priv->state);
- if (was_opened)
- mlx5e_close_locked(dev);
+ new_channels.params = priv->channels.params;
+ new_channels.params.log_rq_size = log_rq_size;
+ new_channels.params.log_sq_size = log_sq_size;
- priv->params.log_rq_size = log_rq_size;
- priv->params.log_sq_size = log_sq_size;
- priv->params.min_rx_wqes = min_rx_wqes;
+ if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {
+ priv->channels.params = new_channels.params;
+ goto unlock;
+ }
- if (was_opened)
- err = mlx5e_open_locked(dev);
+ err = mlx5e_open_channels(priv, &new_channels);
+ if (err)
+ goto unlock;
+ mlx5e_switch_priv_channels(priv, &new_channels, NULL);
+
+unlock:
mutex_unlock(&priv->state_lock);
return err;
@@ -553,7 +555,7 @@
struct mlx5e_priv *priv = netdev_priv(dev);
ch->max_combined = priv->profile->max_nch(priv->mdev);
- ch->combined_count = priv->params.num_channels;
+ ch->combined_count = priv->channels.params.num_channels;
}
static int mlx5e_set_channels(struct net_device *dev,
@@ -561,8 +563,8 @@
{
struct mlx5e_priv *priv = netdev_priv(dev);
unsigned int count = ch->combined_count;
+ struct mlx5e_channels new_channels = {};
bool arfs_enabled;
- bool was_opened;
int err = 0;
if (!count) {
@@ -571,27 +573,32 @@
return -EINVAL;
}
- if (priv->params.num_channels == count)
+ if (priv->channels.params.num_channels == count)
return 0;
mutex_lock(&priv->state_lock);
- was_opened = test_bit(MLX5E_STATE_OPENED, &priv->state);
- if (was_opened)
- mlx5e_close_locked(dev);
+ new_channels.params = priv->channels.params;
+ new_channels.params.num_channels = count;
+ mlx5e_build_default_indir_rqt(priv->mdev, new_channels.params.indirection_rqt,
+ MLX5E_INDIR_RQT_SIZE, count);
+
+ if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {
+ priv->channels.params = new_channels.params;
+ goto out;
+ }
+
+ /* Create fresh channels with new parameters */
+ err = mlx5e_open_channels(priv, &new_channels);
+ if (err)
+ goto out;
arfs_enabled = dev->features & NETIF_F_NTUPLE;
if (arfs_enabled)
mlx5e_arfs_disable(priv);
- priv->params.num_channels = count;
- mlx5e_build_default_indir_rqt(priv->mdev, priv->params.indirection_rqt,
- MLX5E_INDIR_RQT_SIZE, count);
-
- if (was_opened)
- err = mlx5e_open_locked(dev);
- if (err)
- goto out;
+ /* Switch to new channels, set new parameters and close old ones */
+ mlx5e_switch_priv_channels(priv, &new_channels, NULL);
if (arfs_enabled) {
err = mlx5e_arfs_enable(priv);
@@ -614,49 +621,24 @@
if (!MLX5_CAP_GEN(priv->mdev, cq_moderation))
return -EOPNOTSUPP;
- coal->rx_coalesce_usecs = priv->params.rx_cq_moderation.usec;
- coal->rx_max_coalesced_frames = priv->params.rx_cq_moderation.pkts;
- coal->tx_coalesce_usecs = priv->params.tx_cq_moderation.usec;
- coal->tx_max_coalesced_frames = priv->params.tx_cq_moderation.pkts;
- coal->use_adaptive_rx_coalesce = priv->params.rx_am_enabled;
+ coal->rx_coalesce_usecs = priv->channels.params.rx_cq_moderation.usec;
+ coal->rx_max_coalesced_frames = priv->channels.params.rx_cq_moderation.pkts;
+ coal->tx_coalesce_usecs = priv->channels.params.tx_cq_moderation.usec;
+ coal->tx_max_coalesced_frames = priv->channels.params.tx_cq_moderation.pkts;
+ coal->use_adaptive_rx_coalesce = priv->channels.params.rx_am_enabled;
return 0;
}
-static int mlx5e_set_coalesce(struct net_device *netdev,
- struct ethtool_coalesce *coal)
+static void
+mlx5e_set_priv_channels_coalesce(struct mlx5e_priv *priv, struct ethtool_coalesce *coal)
{
- struct mlx5e_priv *priv = netdev_priv(netdev);
struct mlx5_core_dev *mdev = priv->mdev;
- struct mlx5e_channel *c;
- bool restart =
- !!coal->use_adaptive_rx_coalesce != priv->params.rx_am_enabled;
- bool was_opened;
- int err = 0;
int tc;
int i;
- if (!MLX5_CAP_GEN(mdev, cq_moderation))
- return -EOPNOTSUPP;
-
- mutex_lock(&priv->state_lock);
-
- was_opened = test_bit(MLX5E_STATE_OPENED, &priv->state);
- if (was_opened && restart) {
- mlx5e_close_locked(netdev);
- priv->params.rx_am_enabled = !!coal->use_adaptive_rx_coalesce;
- }
-
- priv->params.tx_cq_moderation.usec = coal->tx_coalesce_usecs;
- priv->params.tx_cq_moderation.pkts = coal->tx_max_coalesced_frames;
- priv->params.rx_cq_moderation.usec = coal->rx_coalesce_usecs;
- priv->params.rx_cq_moderation.pkts = coal->rx_max_coalesced_frames;
-
- if (!was_opened || restart)
- goto out;
-
- for (i = 0; i < priv->params.num_channels; ++i) {
- c = priv->channel[i];
+ for (i = 0; i < priv->channels.num; ++i) {
+ struct mlx5e_channel *c = priv->channels.c[i];
for (tc = 0; tc < c->num_tc; tc++) {
mlx5_core_modify_cq_moderation(mdev,
@@ -669,11 +651,50 @@
coal->rx_coalesce_usecs,
coal->rx_max_coalesced_frames);
}
+}
+
+static int mlx5e_set_coalesce(struct net_device *netdev,
+ struct ethtool_coalesce *coal)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5e_channels new_channels = {};
+ int err = 0;
+ bool reset;
+
+ if (!MLX5_CAP_GEN(mdev, cq_moderation))
+ return -EOPNOTSUPP;
+
+ mutex_lock(&priv->state_lock);
+ new_channels.params = priv->channels.params;
+
+ new_channels.params.tx_cq_moderation.usec = coal->tx_coalesce_usecs;
+ new_channels.params.tx_cq_moderation.pkts = coal->tx_max_coalesced_frames;
+ new_channels.params.rx_cq_moderation.usec = coal->rx_coalesce_usecs;
+ new_channels.params.rx_cq_moderation.pkts = coal->rx_max_coalesced_frames;
+ new_channels.params.rx_am_enabled = !!coal->use_adaptive_rx_coalesce;
+
+ if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {
+ priv->channels.params = new_channels.params;
+ goto out;
+ }
+ /* we are opened */
+
+ reset = !!coal->use_adaptive_rx_coalesce != priv->channels.params.rx_am_enabled;
+ if (!reset) {
+ mlx5e_set_priv_channels_coalesce(priv, coal);
+ priv->channels.params = new_channels.params;
+ goto out;
+ }
+
+ /* open fresh channels with new coal parameters */
+ err = mlx5e_open_channels(priv, &new_channels);
+ if (err)
+ goto out;
+
+ mlx5e_switch_priv_channels(priv, &new_channels, NULL);
out:
- if (was_opened && restart)
- err = mlx5e_open_locked(netdev);
-
mutex_unlock(&priv->state_lock);
return err;
}
@@ -968,7 +989,7 @@
{
struct mlx5e_priv *priv = netdev_priv(netdev);
- return sizeof(priv->params.toeplitz_hash_key);
+ return sizeof(priv->channels.params.toeplitz_hash_key);
}
static u32 mlx5e_get_rxfh_indir_size(struct net_device *netdev)
@@ -982,15 +1003,15 @@
struct mlx5e_priv *priv = netdev_priv(netdev);
if (indir)
- memcpy(indir, priv->params.indirection_rqt,
- sizeof(priv->params.indirection_rqt));
+ memcpy(indir, priv->channels.params.indirection_rqt,
+ sizeof(priv->channels.params.indirection_rqt));
if (key)
- memcpy(key, priv->params.toeplitz_hash_key,
- sizeof(priv->params.toeplitz_hash_key));
+ memcpy(key, priv->channels.params.toeplitz_hash_key,
+ sizeof(priv->channels.params.toeplitz_hash_key));
if (hfunc)
- *hfunc = priv->params.rss_hfunc;
+ *hfunc = priv->channels.params.rss_hfunc;
return 0;
}
@@ -1006,7 +1027,7 @@
for (tt = 0; tt < MLX5E_NUM_INDIR_TIRS; tt++) {
memset(tirc, 0, ctxlen);
- mlx5e_build_indir_tir_ctx_hash(priv, tirc, tt);
+ mlx5e_build_indir_tir_ctx_hash(&priv->channels.params, tt, tirc);
mlx5_core_modify_tir(mdev, priv->indir_tir[tt].tirn, in, inlen);
}
}
@@ -1030,25 +1051,37 @@
mutex_lock(&priv->state_lock);
- if (indir) {
- u32 rqtn = priv->indir_rqt.rqtn;
-
- memcpy(priv->params.indirection_rqt, indir,
- sizeof(priv->params.indirection_rqt));
- mlx5e_redirect_rqt(priv, rqtn, MLX5E_INDIR_RQT_SIZE, 0);
- }
-
if (hfunc != ETH_RSS_HASH_NO_CHANGE &&
- hfunc != priv->params.rss_hfunc) {
- priv->params.rss_hfunc = hfunc;
+ hfunc != priv->channels.params.rss_hfunc) {
+ priv->channels.params.rss_hfunc = hfunc;
hash_changed = true;
}
+ if (indir) {
+ memcpy(priv->channels.params.indirection_rqt, indir,
+ sizeof(priv->channels.params.indirection_rqt));
+
+ if (test_bit(MLX5E_STATE_OPENED, &priv->state)) {
+ u32 rqtn = priv->indir_rqt.rqtn;
+ struct mlx5e_redirect_rqt_param rrp = {
+ .is_rss = true,
+ {
+ .rss = {
+ .hfunc = priv->channels.params.rss_hfunc,
+ .channels = &priv->channels,
+ },
+ },
+ };
+
+ mlx5e_redirect_rqt(priv, rqtn, MLX5E_INDIR_RQT_SIZE, rrp);
+ }
+ }
+
if (key) {
- memcpy(priv->params.toeplitz_hash_key, key,
- sizeof(priv->params.toeplitz_hash_key));
+ memcpy(priv->channels.params.toeplitz_hash_key, key,
+ sizeof(priv->channels.params.toeplitz_hash_key));
hash_changed = hash_changed ||
- priv->params.rss_hfunc == ETH_RSS_HASH_TOP;
+ priv->channels.params.rss_hfunc == ETH_RSS_HASH_TOP;
}
if (hash_changed)
@@ -1069,7 +1102,7 @@
switch (info->cmd) {
case ETHTOOL_GRXRINGS:
- info->data = priv->params.num_channels;
+ info->data = priv->channels.params.num_channels;
break;
case ETHTOOL_GRXCLSRLCNT:
info->rule_cnt = priv->fs.ethtool.tot_num_rules;
@@ -1097,7 +1130,7 @@
switch (tuna->id) {
case ETHTOOL_TX_COPYBREAK:
- *(u32 *)data = priv->params.tx_max_inline;
+ *(u32 *)data = priv->channels.params.tx_max_inline;
break;
default:
err = -EINVAL;
@@ -1113,9 +1146,11 @@
{
struct mlx5e_priv *priv = netdev_priv(dev);
struct mlx5_core_dev *mdev = priv->mdev;
- bool was_opened;
- u32 val;
+ struct mlx5e_channels new_channels = {};
int err = 0;
+ u32 val;
+
+ mutex_lock(&priv->state_lock);
switch (tuna->id) {
case ETHTOOL_TX_COPYBREAK:
@@ -1125,24 +1160,26 @@
break;
}
- mutex_lock(&priv->state_lock);
+ new_channels.params = priv->channels.params;
+ new_channels.params.tx_max_inline = val;
- was_opened = test_bit(MLX5E_STATE_OPENED, &priv->state);
- if (was_opened)
- mlx5e_close_locked(dev);
+ if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {
+ priv->channels.params = new_channels.params;
+ break;
+ }
- priv->params.tx_max_inline = val;
+ err = mlx5e_open_channels(priv, &new_channels);
+ if (err)
+ break;
+ mlx5e_switch_priv_channels(priv, &new_channels, NULL);
- if (was_opened)
- err = mlx5e_open_locked(dev);
-
- mutex_unlock(&priv->state_lock);
break;
default:
err = -EINVAL;
break;
}
+ mutex_unlock(&priv->state_lock);
return err;
}
@@ -1442,15 +1479,15 @@
{
struct mlx5e_priv *priv = netdev_priv(netdev);
struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5e_channels new_channels = {};
bool rx_mode_changed;
u8 rx_cq_period_mode;
int err = 0;
- bool reset;
rx_cq_period_mode = enable ?
MLX5_CQ_PERIOD_MODE_START_FROM_CQE :
MLX5_CQ_PERIOD_MODE_START_FROM_EQE;
- rx_mode_changed = rx_cq_period_mode != priv->params.rx_cq_period_mode;
+ rx_mode_changed = rx_cq_period_mode != priv->channels.params.rx_cq_period_mode;
if (rx_cq_period_mode == MLX5_CQ_PERIOD_MODE_START_FROM_CQE &&
!MLX5_CAP_GEN(mdev, cq_period_start_from_cqe))
@@ -1459,16 +1496,51 @@
if (!rx_mode_changed)
return 0;
- reset = test_bit(MLX5E_STATE_OPENED, &priv->state);
- if (reset)
- mlx5e_close_locked(netdev);
+ new_channels.params = priv->channels.params;
+ mlx5e_set_rx_cq_mode_params(&new_channels.params, rx_cq_period_mode);
- mlx5e_set_rx_cq_mode_params(&priv->params, rx_cq_period_mode);
+ if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {
+ priv->channels.params = new_channels.params;
+ return 0;
+ }
- if (reset)
- err = mlx5e_open_locked(netdev);
+ err = mlx5e_open_channels(priv, &new_channels);
+ if (err)
+ return err;
- return err;
+ mlx5e_switch_priv_channels(priv, &new_channels, NULL);
+ return 0;
+}
+
+int mlx5e_modify_rx_cqe_compression_locked(struct mlx5e_priv *priv, bool new_val)
+{
+ bool curr_val = MLX5E_GET_PFLAG(&priv->channels.params, MLX5E_PFLAG_RX_CQE_COMPRESS);
+ struct mlx5e_channels new_channels = {};
+ int err = 0;
+
+ if (!MLX5_CAP_GEN(priv->mdev, cqe_compression))
+ return new_val ? -EOPNOTSUPP : 0;
+
+ if (curr_val == new_val)
+ return 0;
+
+ new_channels.params = priv->channels.params;
+ MLX5E_SET_PFLAG(&new_channels.params, MLX5E_PFLAG_RX_CQE_COMPRESS, new_val);
+
+ mlx5e_set_rq_type_params(priv->mdev, &new_channels.params,
+ new_channels.params.rq_wq_type);
+
+ if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {
+ priv->channels.params = new_channels.params;
+ return 0;
+ }
+
+ err = mlx5e_open_channels(priv, &new_channels);
+ if (err)
+ return err;
+
+ mlx5e_switch_priv_channels(priv, &new_channels, NULL);
+ return 0;
}
static int set_pflag_rx_cqe_compress(struct net_device *netdev,
@@ -1486,8 +1558,7 @@
}
mlx5e_modify_rx_cqe_compression_locked(priv, enable);
- priv->params.rx_cqe_compress_def = enable;
- mlx5e_set_rq_type_params(priv, priv->params.rq_wq_type);
+ priv->channels.params.rx_cqe_compress_def = enable;
return 0;
}
@@ -1499,7 +1570,7 @@
{
struct mlx5e_priv *priv = netdev_priv(netdev);
bool enable = !!(wanted_flags & flag);
- u32 changes = wanted_flags ^ priv->params.pflags;
+ u32 changes = wanted_flags ^ priv->channels.params.pflags;
int err;
if (!(changes & flag))
@@ -1512,7 +1583,7 @@
return err;
}
- MLX5E_SET_PFLAG(priv, flag, enable);
+ MLX5E_SET_PFLAG(&priv->channels.params, flag, enable);
return 0;
}
@@ -1541,7 +1612,7 @@
{
struct mlx5e_priv *priv = netdev_priv(netdev);
- return priv->params.pflags;
+ return priv->channels.params.pflags;
}
static int mlx5e_set_rxnfc(struct net_device *dev, struct ethtool_rxnfc *cmd)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
index f2762e4..576d678 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
@@ -159,14 +159,10 @@
enum mlx5e_vlan_rule_type rule_type,
u16 vid, struct mlx5_flow_spec *spec)
{
- struct mlx5_flow_act flow_act = {
- .action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- .flow_tag = MLX5_FS_DEFAULT_FLOW_TAG,
- .encap_id = 0,
- };
struct mlx5_flow_table *ft = priv->fs.vlan.ft.t;
struct mlx5_flow_destination dest;
struct mlx5_flow_handle **rule_p;
+ MLX5_DECLARE_FLOW_ACT(flow_act);
int err = 0;
dest.type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
@@ -659,11 +655,7 @@
u16 etype,
u8 proto)
{
- struct mlx5_flow_act flow_act = {
- .action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- .flow_tag = MLX5_FS_DEFAULT_FLOW_TAG,
- .encap_id = 0,
- };
+ MLX5_DECLARE_FLOW_ACT(flow_act);
struct mlx5_flow_handle *rule;
struct mlx5_flow_spec *spec;
int err = 0;
@@ -800,7 +792,7 @@
return err;
}
-static void mlx5e_destroy_ttc_table(struct mlx5e_priv *priv)
+void mlx5e_destroy_ttc_table(struct mlx5e_priv *priv)
{
struct mlx5e_ttc_table *ttc = &priv->fs.ttc;
@@ -808,14 +800,19 @@
mlx5e_destroy_flow_table(&ttc->ft);
}
-static int mlx5e_create_ttc_table(struct mlx5e_priv *priv)
+int mlx5e_create_ttc_table(struct mlx5e_priv *priv, u32 underlay_qpn)
{
struct mlx5e_ttc_table *ttc = &priv->fs.ttc;
+ struct mlx5_flow_table_attr ft_attr = {};
struct mlx5e_flow_table *ft = &ttc->ft;
int err;
- ft->t = mlx5_create_flow_table(priv->fs.ns, MLX5E_NIC_PRIO,
- MLX5E_TTC_TABLE_SIZE, MLX5E_TTC_FT_LEVEL, 0);
+ ft_attr.max_fte = MLX5E_TTC_TABLE_SIZE;
+ ft_attr.level = MLX5E_TTC_FT_LEVEL;
+ ft_attr.prio = MLX5E_NIC_PRIO;
+ ft_attr.underlay_qpn = underlay_qpn;
+
+ ft->t = mlx5_create_flow_table(priv->fs.ns, &ft_attr);
if (IS_ERR(ft->t)) {
err = PTR_ERR(ft->t);
ft->t = NULL;
@@ -848,13 +845,9 @@
static int mlx5e_add_l2_flow_rule(struct mlx5e_priv *priv,
struct mlx5e_l2_rule *ai, int type)
{
- struct mlx5_flow_act flow_act = {
- .action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- .flow_tag = MLX5_FS_DEFAULT_FLOW_TAG,
- .encap_id = 0,
- };
struct mlx5_flow_table *ft = priv->fs.l2.ft.t;
struct mlx5_flow_destination dest;
+ MLX5_DECLARE_FLOW_ACT(flow_act);
struct mlx5_flow_spec *spec;
int err = 0;
u8 *mc_dmac;
@@ -985,12 +978,16 @@
{
struct mlx5e_l2_table *l2_table = &priv->fs.l2;
struct mlx5e_flow_table *ft = &l2_table->ft;
+ struct mlx5_flow_table_attr ft_attr = {};
int err;
ft->num_groups = 0;
- ft->t = mlx5_create_flow_table(priv->fs.ns, MLX5E_NIC_PRIO,
- MLX5E_L2_TABLE_SIZE, MLX5E_L2_FT_LEVEL, 0);
+ ft_attr.max_fte = MLX5E_L2_TABLE_SIZE;
+ ft_attr.level = MLX5E_L2_FT_LEVEL;
+ ft_attr.prio = MLX5E_NIC_PRIO;
+
+ ft->t = mlx5_create_flow_table(priv->fs.ns, &ft_attr);
if (IS_ERR(ft->t)) {
err = PTR_ERR(ft->t);
ft->t = NULL;
@@ -1088,11 +1085,16 @@
static int mlx5e_create_vlan_table(struct mlx5e_priv *priv)
{
struct mlx5e_flow_table *ft = &priv->fs.vlan.ft;
+ struct mlx5_flow_table_attr ft_attr = {};
int err;
ft->num_groups = 0;
- ft->t = mlx5_create_flow_table(priv->fs.ns, MLX5E_NIC_PRIO,
- MLX5E_VLAN_TABLE_SIZE, MLX5E_VLAN_FT_LEVEL, 0);
+
+ ft_attr.max_fte = MLX5E_VLAN_TABLE_SIZE;
+ ft_attr.level = MLX5E_VLAN_FT_LEVEL;
+ ft_attr.prio = MLX5E_NIC_PRIO;
+
+ ft->t = mlx5_create_flow_table(priv->fs.ns, &ft_attr);
if (IS_ERR(ft->t)) {
err = PTR_ERR(ft->t);
@@ -1145,7 +1147,7 @@
priv->netdev->hw_features &= ~NETIF_F_NTUPLE;
}
- err = mlx5e_create_ttc_table(priv);
+ err = mlx5e_create_ttc_table(priv, 0);
if (err) {
netdev_err(priv->netdev, "Failed to create ttc table, err=%d\n",
err);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_fs_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_fs_ethtool.c
index d55fff0..85bf4a3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_fs_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_fs_ethtool.c
@@ -390,7 +390,7 @@
if (fs->location >= MAX_NUM_OF_ETHTOOL_RULES)
return -EINVAL;
- if (fs->ring_cookie >= priv->params.num_channels &&
+ if (fs->ring_cookie >= priv->channels.params.num_channels &&
fs->ring_cookie != RX_CLS_FLOW_DISC)
return -EINVAL;
@@ -564,6 +564,7 @@
int idx = 0;
int err = 0;
+ info->data = MAX_NUM_OF_ETHTOOL_RULES;
while ((!err || err == -ENOENT) && idx < info->rule_cnt) {
err = mlx5e_ethtool_get_flow(priv, info, location);
if (!err)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 66c1337..a61b71b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -31,28 +31,24 @@
*/
#include <net/tc_act/tc_gact.h>
-#include <linux/crash_dump.h>
#include <net/pkt_cls.h>
#include <linux/mlx5/fs.h>
#include <net/vxlan.h>
#include <linux/bpf.h>
+#include "eswitch.h"
#include "en.h"
#include "en_tc.h"
-#include "eswitch.h"
+#include "en_rep.h"
#include "vxlan.h"
struct mlx5e_rq_param {
u32 rqc[MLX5_ST_SZ_DW(rqc)];
struct mlx5_wq_param wq;
- bool am_enabled;
};
struct mlx5e_sq_param {
u32 sqc[MLX5_ST_SZ_DW(sqc)];
struct mlx5_wq_param wq;
- u16 max_inline;
- u8 min_inline_mode;
- enum mlx5e_sq_type type;
};
struct mlx5e_cq_param {
@@ -79,49 +75,47 @@
MLX5_CAP_ETH(mdev, reg_umr_sq);
}
-void mlx5e_set_rq_type_params(struct mlx5e_priv *priv, u8 rq_type)
+void mlx5e_set_rq_type_params(struct mlx5_core_dev *mdev,
+ struct mlx5e_params *params, u8 rq_type)
{
- priv->params.rq_wq_type = rq_type;
- priv->params.lro_wqe_sz = MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ;
- switch (priv->params.rq_wq_type) {
+ params->rq_wq_type = rq_type;
+ params->lro_wqe_sz = MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ;
+ switch (params->rq_wq_type) {
case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ:
- priv->params.log_rq_size = is_kdump_kernel() ?
+ params->log_rq_size = is_kdump_kernel() ?
MLX5E_PARAMS_MINIMUM_LOG_RQ_SIZE_MPW :
MLX5E_PARAMS_DEFAULT_LOG_RQ_SIZE_MPW;
- priv->params.mpwqe_log_stride_sz =
- MLX5E_GET_PFLAG(priv, MLX5E_PFLAG_RX_CQE_COMPRESS) ?
- MLX5_MPWRQ_CQE_CMPRS_LOG_STRIDE_SZ(priv->mdev) :
- MLX5_MPWRQ_DEF_LOG_STRIDE_SZ(priv->mdev);
- priv->params.mpwqe_log_num_strides = MLX5_MPWRQ_LOG_WQE_SZ -
- priv->params.mpwqe_log_stride_sz;
+ params->mpwqe_log_stride_sz =
+ MLX5E_GET_PFLAG(params, MLX5E_PFLAG_RX_CQE_COMPRESS) ?
+ MLX5_MPWRQ_CQE_CMPRS_LOG_STRIDE_SZ(mdev) :
+ MLX5_MPWRQ_DEF_LOG_STRIDE_SZ(mdev);
+ params->mpwqe_log_num_strides = MLX5_MPWRQ_LOG_WQE_SZ -
+ params->mpwqe_log_stride_sz;
break;
default: /* MLX5_WQ_TYPE_LINKED_LIST */
- priv->params.log_rq_size = is_kdump_kernel() ?
+ params->log_rq_size = is_kdump_kernel() ?
MLX5E_PARAMS_MINIMUM_LOG_RQ_SIZE :
MLX5E_PARAMS_DEFAULT_LOG_RQ_SIZE;
/* Extra room needed for build_skb */
- priv->params.lro_wqe_sz -= MLX5_RX_HEADROOM +
+ params->lro_wqe_sz -= MLX5_RX_HEADROOM +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
}
- priv->params.min_rx_wqes = mlx5_min_rx_wqes(priv->params.rq_wq_type,
- BIT(priv->params.log_rq_size));
- mlx5_core_info(priv->mdev,
- "MLX5E: StrdRq(%d) RqSz(%ld) StrdSz(%ld) RxCqeCmprss(%d)\n",
- priv->params.rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ,
- BIT(priv->params.log_rq_size),
- BIT(priv->params.mpwqe_log_stride_sz),
- MLX5E_GET_PFLAG(priv, MLX5E_PFLAG_RX_CQE_COMPRESS));
+ mlx5_core_info(mdev, "MLX5E: StrdRq(%d) RqSz(%ld) StrdSz(%ld) RxCqeCmprss(%d)\n",
+ params->rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ,
+ BIT(params->log_rq_size),
+ BIT(params->mpwqe_log_stride_sz),
+ MLX5E_GET_PFLAG(params, MLX5E_PFLAG_RX_CQE_COMPRESS));
}
-static void mlx5e_set_rq_priv_params(struct mlx5e_priv *priv)
+static void mlx5e_set_rq_params(struct mlx5_core_dev *mdev, struct mlx5e_params *params)
{
- u8 rq_type = mlx5e_check_fragmented_striding_rq_cap(priv->mdev) &&
- !priv->xdp_prog ?
+ u8 rq_type = mlx5e_check_fragmented_striding_rq_cap(mdev) &&
+ !params->xdp_prog ?
MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ :
MLX5_WQ_TYPE_LINKED_LIST;
- mlx5e_set_rq_type_params(priv, rq_type);
+ mlx5e_set_rq_type_params(mdev, params, rq_type);
}
static void mlx5e_update_carrier(struct mlx5e_priv *priv)
@@ -174,15 +168,17 @@
static void mlx5e_update_sw_counters(struct mlx5e_priv *priv)
{
- struct mlx5e_sw_stats *s = &priv->stats.sw;
+ struct mlx5e_sw_stats temp, *s = &temp;
struct mlx5e_rq_stats *rq_stats;
struct mlx5e_sq_stats *sq_stats;
u64 tx_offload_none = 0;
int i, j;
memset(s, 0, sizeof(*s));
- for (i = 0; i < priv->params.num_channels; i++) {
- rq_stats = &priv->channel[i]->rq.stats;
+ for (i = 0; i < priv->channels.num; i++) {
+ struct mlx5e_channel *c = priv->channels.c[i];
+
+ rq_stats = &c->rq.stats;
s->rx_packets += rq_stats->packets;
s->rx_bytes += rq_stats->bytes;
@@ -204,8 +200,8 @@
s->rx_cache_empty += rq_stats->cache_empty;
s->rx_cache_busy += rq_stats->cache_busy;
- for (j = 0; j < priv->params.num_tc; j++) {
- sq_stats = &priv->channel[i]->sq[j].stats;
+ for (j = 0; j < priv->channels.params.num_tc; j++) {
+ sq_stats = &c->sq[j].stats;
s->tx_packets += sq_stats->packets;
s->tx_bytes += sq_stats->bytes;
@@ -229,6 +225,7 @@
s->link_down_events_phy = MLX5_GET(ppcnt_reg,
priv->stats.pport.phy_counters,
counter_set.phys_layer_cntrs.link_down_events);
+ memcpy(&priv->stats.sw, s, sizeof(*s));
}
static void mlx5e_update_vport_counters(struct mlx5e_priv *priv)
@@ -243,7 +240,6 @@
MLX5_SET(query_vport_counter_in, in, op_mod, 0);
MLX5_SET(query_vport_counter_in, in, other_vport, 0);
- memset(out, 0, outlen);
mlx5_cmd_exec(mdev, in, sizeof(in), out, outlen);
}
@@ -402,8 +398,10 @@
MLX5_UMR_MTT_ALIGNMENT);
}
-static inline void mlx5e_build_umr_wqe(struct mlx5e_rq *rq, struct mlx5e_sq *sq,
- struct mlx5e_umr_wqe *wqe, u16 ix)
+static inline void mlx5e_build_umr_wqe(struct mlx5e_rq *rq,
+ struct mlx5e_icosq *sq,
+ struct mlx5e_umr_wqe *wqe,
+ u16 ix)
{
struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;
@@ -493,11 +491,10 @@
kfree(rq->mpwqe.info);
}
-static int mlx5e_create_umr_mkey(struct mlx5e_priv *priv,
+static int mlx5e_create_umr_mkey(struct mlx5_core_dev *mdev,
u64 npages, u8 page_shift,
struct mlx5_core_mkey *umr_mkey)
{
- struct mlx5_core_dev *mdev = priv->mdev;
int inlen = MLX5_ST_SZ_BYTES(create_mkey_in);
void *mkc;
u32 *in;
@@ -531,21 +528,20 @@
return err;
}
-static int mlx5e_create_rq_umr_mkey(struct mlx5e_rq *rq)
+static int mlx5e_create_rq_umr_mkey(struct mlx5_core_dev *mdev, struct mlx5e_rq *rq)
{
- struct mlx5e_priv *priv = rq->priv;
- u64 num_mtts = MLX5E_REQUIRED_MTTS(BIT(priv->params.log_rq_size));
+ u64 num_mtts = MLX5E_REQUIRED_MTTS(mlx5_wq_ll_get_size(&rq->wq));
- return mlx5e_create_umr_mkey(priv, num_mtts, PAGE_SHIFT, &rq->umr_mkey);
+ return mlx5e_create_umr_mkey(mdev, num_mtts, PAGE_SHIFT, &rq->umr_mkey);
}
-static int mlx5e_create_rq(struct mlx5e_channel *c,
- struct mlx5e_rq_param *param,
- struct mlx5e_rq *rq)
+static int mlx5e_alloc_rq(struct mlx5e_channel *c,
+ struct mlx5e_params *params,
+ struct mlx5e_rq_param *rqp,
+ struct mlx5e_rq *rq)
{
- struct mlx5e_priv *priv = c->priv;
- struct mlx5_core_dev *mdev = priv->mdev;
- void *rqc = param->rqc;
+ struct mlx5_core_dev *mdev = c->mdev;
+ void *rqc = rqp->rqc;
void *rqc_wq = MLX5_ADDR_OF(rqc, rqc, wq);
u32 byte_count;
u32 frag_sz;
@@ -554,9 +550,9 @@
int err;
int i;
- param->wq.db_numa_node = cpu_to_node(c->cpu);
+ rqp->wq.db_numa_node = cpu_to_node(c->cpu);
- err = mlx5_wq_ll_create(mdev, ¶m->wq, rqc_wq, &rq->wq,
+ err = mlx5_wq_ll_create(mdev, &rqp->wq, rqc_wq, &rq->wq,
&rq->wq_ctrl);
if (err)
return err;
@@ -565,15 +561,15 @@
wq_sz = mlx5_wq_ll_get_size(&rq->wq);
- rq->wq_type = priv->params.rq_wq_type;
+ rq->wq_type = params->rq_wq_type;
rq->pdev = c->pdev;
rq->netdev = c->netdev;
- rq->tstamp = &priv->tstamp;
+ rq->tstamp = c->tstamp;
rq->channel = c;
rq->ix = c->ix;
- rq->priv = c->priv;
+ rq->mdev = mdev;
- rq->xdp_prog = priv->xdp_prog ? bpf_prog_inc(priv->xdp_prog) : NULL;
+ rq->xdp_prog = params->xdp_prog ? bpf_prog_inc(params->xdp_prog) : NULL;
if (IS_ERR(rq->xdp_prog)) {
err = PTR_ERR(rq->xdp_prog);
rq->xdp_prog = NULL;
@@ -588,24 +584,26 @@
rq->rx_headroom = MLX5_RX_HEADROOM;
}
- switch (priv->params.rq_wq_type) {
+ switch (rq->wq_type) {
case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ:
- if (mlx5e_is_vf_vport_rep(priv)) {
- err = -EINVAL;
- goto err_rq_wq_destroy;
- }
- rq->handle_rx_cqe = mlx5e_handle_rx_cqe_mpwrq;
rq->alloc_wqe = mlx5e_alloc_rx_mpwqe;
rq->dealloc_wqe = mlx5e_dealloc_rx_mpwqe;
- rq->mpwqe_stride_sz = BIT(priv->params.mpwqe_log_stride_sz);
- rq->mpwqe_num_strides = BIT(priv->params.mpwqe_log_num_strides);
+ rq->handle_rx_cqe = c->priv->profile->rx_handlers.handle_rx_cqe_mpwqe;
+ if (!rq->handle_rx_cqe) {
+ err = -EINVAL;
+ netdev_err(c->netdev, "RX handler of MPWQE RQ is not set, err %d\n", err);
+ goto err_rq_wq_destroy;
+ }
+
+ rq->mpwqe_stride_sz = BIT(params->mpwqe_log_stride_sz);
+ rq->mpwqe_num_strides = BIT(params->mpwqe_log_num_strides);
rq->buff.wqe_sz = rq->mpwqe_stride_sz * rq->mpwqe_num_strides;
byte_count = rq->buff.wqe_sz;
- err = mlx5e_create_rq_umr_mkey(rq);
+ err = mlx5e_create_rq_umr_mkey(mdev, rq);
if (err)
goto err_rq_wq_destroy;
rq->mkey_be = cpu_to_be32(rq->umr_mkey.key);
@@ -621,18 +619,20 @@
err = -ENOMEM;
goto err_rq_wq_destroy;
}
-
- if (mlx5e_is_vf_vport_rep(priv))
- rq->handle_rx_cqe = mlx5e_handle_rx_cqe_rep;
- else
- rq->handle_rx_cqe = mlx5e_handle_rx_cqe;
-
rq->alloc_wqe = mlx5e_alloc_rx_wqe;
rq->dealloc_wqe = mlx5e_dealloc_rx_wqe;
- rq->buff.wqe_sz = (priv->params.lro_en) ?
- priv->params.lro_wqe_sz :
- MLX5E_SW2HW_MTU(priv->netdev->mtu);
+ rq->handle_rx_cqe = c->priv->profile->rx_handlers.handle_rx_cqe;
+ if (!rq->handle_rx_cqe) {
+ kfree(rq->dma_info);
+ err = -EINVAL;
+ netdev_err(c->netdev, "RX handler of RQ is not set, err %d\n", err);
+ goto err_rq_wq_destroy;
+ }
+
+ rq->buff.wqe_sz = params->lro_en ?
+ params->lro_wqe_sz :
+ MLX5E_SW2HW_MTU(c->netdev->mtu);
byte_count = rq->buff.wqe_sz;
/* calc the required page order */
@@ -656,8 +656,7 @@
}
INIT_WORK(&rq->am.work, mlx5e_rx_am_work);
- rq->am.mode = priv->params.rx_cq_period_mode;
-
+ rq->am.mode = params->rx_cq_period_mode;
rq->page_cache.head = 0;
rq->page_cache.tail = 0;
@@ -674,7 +673,7 @@
return err;
}
-static void mlx5e_destroy_rq(struct mlx5e_rq *rq)
+static void mlx5e_free_rq(struct mlx5e_rq *rq)
{
int i;
@@ -684,7 +683,7 @@
switch (rq->wq_type) {
case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ:
mlx5e_rq_free_mpwqe_info(rq);
- mlx5_core_destroy_mkey(rq->priv->mdev, &rq->umr_mkey);
+ mlx5_core_destroy_mkey(rq->mdev, &rq->umr_mkey);
break;
default: /* MLX5_WQ_TYPE_LINKED_LIST */
kfree(rq->dma_info);
@@ -699,10 +698,10 @@
mlx5_wq_destroy(&rq->wq_ctrl);
}
-static int mlx5e_enable_rq(struct mlx5e_rq *rq, struct mlx5e_rq_param *param)
+static int mlx5e_create_rq(struct mlx5e_rq *rq,
+ struct mlx5e_rq_param *param)
{
- struct mlx5e_priv *priv = rq->priv;
- struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5_core_dev *mdev = rq->mdev;
void *in;
void *rqc;
@@ -723,7 +722,6 @@
MLX5_SET(rqc, rqc, cqn, rq->cq.mcq.cqn);
MLX5_SET(rqc, rqc, state, MLX5_RQC_STATE_RST);
- MLX5_SET(rqc, rqc, vsd, priv->params.vlan_strip_disable);
MLX5_SET(wq, wq, log_wq_pg_sz, rq->wq_ctrl.buf.page_shift -
MLX5_ADAPTER_PAGE_SHIFT);
MLX5_SET64(wq, wq, dbr_addr, rq->wq_ctrl.db.dma);
@@ -742,8 +740,7 @@
int next_state)
{
struct mlx5e_channel *c = rq->channel;
- struct mlx5e_priv *priv = c->priv;
- struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5_core_dev *mdev = c->mdev;
void *in;
void *rqc;
@@ -767,7 +764,7 @@
return err;
}
-static int mlx5e_modify_rq_vsd(struct mlx5e_rq *rq, bool vsd)
+static int mlx5e_modify_rq_scatter_fcs(struct mlx5e_rq *rq, bool enable)
{
struct mlx5e_channel *c = rq->channel;
struct mlx5e_priv *priv = c->priv;
@@ -787,6 +784,35 @@
MLX5_SET(modify_rq_in, in, rq_state, MLX5_RQC_STATE_RDY);
MLX5_SET64(modify_rq_in, in, modify_bitmask,
+ MLX5_MODIFY_RQ_IN_MODIFY_BITMASK_SCATTER_FCS);
+ MLX5_SET(rqc, rqc, scatter_fcs, enable);
+ MLX5_SET(rqc, rqc, state, MLX5_RQC_STATE_RDY);
+
+ err = mlx5_core_modify_rq(mdev, rq->rqn, in, inlen);
+
+ kvfree(in);
+
+ return err;
+}
+
+static int mlx5e_modify_rq_vsd(struct mlx5e_rq *rq, bool vsd)
+{
+ struct mlx5e_channel *c = rq->channel;
+ struct mlx5_core_dev *mdev = c->mdev;
+ void *in;
+ void *rqc;
+ int inlen;
+ int err;
+
+ inlen = MLX5_ST_SZ_BYTES(modify_rq_in);
+ in = mlx5_vzalloc(inlen);
+ if (!in)
+ return -ENOMEM;
+
+ rqc = MLX5_ADDR_OF(modify_rq_in, in, ctx);
+
+ MLX5_SET(modify_rq_in, in, rq_state, MLX5_RQC_STATE_RDY);
+ MLX5_SET64(modify_rq_in, in, modify_bitmask,
MLX5_MODIFY_RQ_IN_MODIFY_BITMASK_VSD);
MLX5_SET(rqc, rqc, vsd, vsd);
MLX5_SET(rqc, rqc, state, MLX5_RQC_STATE_RDY);
@@ -798,25 +824,28 @@
return err;
}
-static void mlx5e_disable_rq(struct mlx5e_rq *rq)
+static void mlx5e_destroy_rq(struct mlx5e_rq *rq)
{
- mlx5_core_destroy_rq(rq->priv->mdev, rq->rqn);
+ mlx5_core_destroy_rq(rq->mdev, rq->rqn);
}
static int mlx5e_wait_for_min_rx_wqes(struct mlx5e_rq *rq)
{
unsigned long exp_time = jiffies + msecs_to_jiffies(20000);
struct mlx5e_channel *c = rq->channel;
- struct mlx5e_priv *priv = c->priv;
+
struct mlx5_wq_ll *wq = &rq->wq;
+ u16 min_wqes = mlx5_min_rx_wqes(rq->wq_type, mlx5_wq_ll_get_size(wq));
while (time_before(jiffies, exp_time)) {
- if (wq->cur_sz >= priv->params.min_rx_wqes)
+ if (wq->cur_sz >= min_wqes)
return 0;
msleep(20);
}
+ netdev_warn(c->netdev, "Failed to get min RX wqes on RQN[0x%x] wq cur_sz(%d) min_rx_wqes(%d)\n",
+ rq->rqn, wq->cur_sz, min_wqes);
return -ETIMEDOUT;
}
@@ -842,83 +871,128 @@
}
static int mlx5e_open_rq(struct mlx5e_channel *c,
+ struct mlx5e_params *params,
struct mlx5e_rq_param *param,
struct mlx5e_rq *rq)
{
- struct mlx5e_sq *sq = &c->icosq;
- u16 pi = sq->pc & sq->wq.sz_m1;
int err;
- err = mlx5e_create_rq(c, param, rq);
+ err = mlx5e_alloc_rq(c, params, param, rq);
if (err)
return err;
- err = mlx5e_enable_rq(rq, param);
+ err = mlx5e_create_rq(rq, param);
+ if (err)
+ goto err_free_rq;
+
+ err = mlx5e_modify_rq_state(rq, MLX5_RQC_STATE_RST, MLX5_RQC_STATE_RDY);
if (err)
goto err_destroy_rq;
- set_bit(MLX5E_RQ_STATE_ENABLED, &rq->state);
- err = mlx5e_modify_rq_state(rq, MLX5_RQC_STATE_RST, MLX5_RQC_STATE_RDY);
- if (err)
- goto err_disable_rq;
-
- if (param->am_enabled)
+ if (params->rx_am_enabled)
set_bit(MLX5E_RQ_STATE_AM, &c->rq.state);
- sq->db.ico_wqe[pi].opcode = MLX5_OPCODE_NOP;
- sq->db.ico_wqe[pi].num_wqebbs = 1;
- mlx5e_send_nop(sq, true); /* trigger mlx5e_post_rx_wqes() */
-
return 0;
-err_disable_rq:
- clear_bit(MLX5E_RQ_STATE_ENABLED, &rq->state);
- mlx5e_disable_rq(rq);
err_destroy_rq:
mlx5e_destroy_rq(rq);
+err_free_rq:
+ mlx5e_free_rq(rq);
return err;
}
-static void mlx5e_close_rq(struct mlx5e_rq *rq)
+static void mlx5e_activate_rq(struct mlx5e_rq *rq)
+{
+ struct mlx5e_icosq *sq = &rq->channel->icosq;
+ u16 pi = sq->pc & sq->wq.sz_m1;
+ struct mlx5e_tx_wqe *nopwqe;
+
+ set_bit(MLX5E_RQ_STATE_ENABLED, &rq->state);
+ sq->db.ico_wqe[pi].opcode = MLX5_OPCODE_NOP;
+ sq->db.ico_wqe[pi].num_wqebbs = 1;
+ nopwqe = mlx5e_post_nop(&sq->wq, sq->sqn, &sq->pc);
+ mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &nopwqe->ctrl);
+}
+
+static void mlx5e_deactivate_rq(struct mlx5e_rq *rq)
{
clear_bit(MLX5E_RQ_STATE_ENABLED, &rq->state);
napi_synchronize(&rq->channel->napi); /* prevent mlx5e_post_rx_wqes */
- cancel_work_sync(&rq->am.work);
-
- mlx5e_disable_rq(rq);
- mlx5e_free_rx_descs(rq);
- mlx5e_destroy_rq(rq);
}
-static void mlx5e_free_sq_xdp_db(struct mlx5e_sq *sq)
+static void mlx5e_close_rq(struct mlx5e_rq *rq)
{
- kfree(sq->db.xdp.di);
- kfree(sq->db.xdp.wqe_info);
+ cancel_work_sync(&rq->am.work);
+ mlx5e_destroy_rq(rq);
+ mlx5e_free_rx_descs(rq);
+ mlx5e_free_rq(rq);
}
-static int mlx5e_alloc_sq_xdp_db(struct mlx5e_sq *sq, int numa)
+static void mlx5e_free_xdpsq_db(struct mlx5e_xdpsq *sq)
+{
+ kfree(sq->db.di);
+}
+
+static int mlx5e_alloc_xdpsq_db(struct mlx5e_xdpsq *sq, int numa)
{
int wq_sz = mlx5_wq_cyc_get_size(&sq->wq);
- sq->db.xdp.di = kzalloc_node(sizeof(*sq->db.xdp.di) * wq_sz,
+ sq->db.di = kzalloc_node(sizeof(*sq->db.di) * wq_sz,
GFP_KERNEL, numa);
- sq->db.xdp.wqe_info = kzalloc_node(sizeof(*sq->db.xdp.wqe_info) * wq_sz,
- GFP_KERNEL, numa);
- if (!sq->db.xdp.di || !sq->db.xdp.wqe_info) {
- mlx5e_free_sq_xdp_db(sq);
+ if (!sq->db.di) {
+ mlx5e_free_xdpsq_db(sq);
return -ENOMEM;
}
return 0;
}
-static void mlx5e_free_sq_ico_db(struct mlx5e_sq *sq)
+static int mlx5e_alloc_xdpsq(struct mlx5e_channel *c,
+ struct mlx5e_params *params,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_xdpsq *sq)
+{
+ void *sqc_wq = MLX5_ADDR_OF(sqc, param->sqc, wq);
+ struct mlx5_core_dev *mdev = c->mdev;
+ int err;
+
+ sq->pdev = c->pdev;
+ sq->mkey_be = c->mkey_be;
+ sq->channel = c;
+ sq->uar_map = mdev->mlx5e_res.bfreg.map;
+ sq->min_inline_mode = params->tx_min_inline_mode;
+
+ param->wq.db_numa_node = cpu_to_node(c->cpu);
+ err = mlx5_wq_cyc_create(mdev, ¶m->wq, sqc_wq, &sq->wq, &sq->wq_ctrl);
+ if (err)
+ return err;
+ sq->wq.db = &sq->wq.db[MLX5_SND_DBR];
+
+ err = mlx5e_alloc_xdpsq_db(sq, cpu_to_node(c->cpu));
+ if (err)
+ goto err_sq_wq_destroy;
+
+ return 0;
+
+err_sq_wq_destroy:
+ mlx5_wq_destroy(&sq->wq_ctrl);
+
+ return err;
+}
+
+static void mlx5e_free_xdpsq(struct mlx5e_xdpsq *sq)
+{
+ mlx5e_free_xdpsq_db(sq);
+ mlx5_wq_destroy(&sq->wq_ctrl);
+}
+
+static void mlx5e_free_icosq_db(struct mlx5e_icosq *sq)
{
kfree(sq->db.ico_wqe);
}
-static int mlx5e_alloc_sq_ico_db(struct mlx5e_sq *sq, int numa)
+static int mlx5e_alloc_icosq_db(struct mlx5e_icosq *sq, int numa)
{
u8 wq_sz = mlx5_wq_cyc_get_size(&sq->wq);
@@ -930,26 +1004,62 @@
return 0;
}
-static void mlx5e_free_sq_txq_db(struct mlx5e_sq *sq)
+static int mlx5e_alloc_icosq(struct mlx5e_channel *c,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_icosq *sq)
{
- kfree(sq->db.txq.wqe_info);
- kfree(sq->db.txq.dma_fifo);
- kfree(sq->db.txq.skb);
+ void *sqc_wq = MLX5_ADDR_OF(sqc, param->sqc, wq);
+ struct mlx5_core_dev *mdev = c->mdev;
+ int err;
+
+ sq->pdev = c->pdev;
+ sq->mkey_be = c->mkey_be;
+ sq->channel = c;
+ sq->uar_map = mdev->mlx5e_res.bfreg.map;
+
+ param->wq.db_numa_node = cpu_to_node(c->cpu);
+ err = mlx5_wq_cyc_create(mdev, ¶m->wq, sqc_wq, &sq->wq, &sq->wq_ctrl);
+ if (err)
+ return err;
+ sq->wq.db = &sq->wq.db[MLX5_SND_DBR];
+
+ err = mlx5e_alloc_icosq_db(sq, cpu_to_node(c->cpu));
+ if (err)
+ goto err_sq_wq_destroy;
+
+ sq->edge = (sq->wq.sz_m1 + 1) - MLX5E_ICOSQ_MAX_WQEBBS;
+
+ return 0;
+
+err_sq_wq_destroy:
+ mlx5_wq_destroy(&sq->wq_ctrl);
+
+ return err;
}
-static int mlx5e_alloc_sq_txq_db(struct mlx5e_sq *sq, int numa)
+static void mlx5e_free_icosq(struct mlx5e_icosq *sq)
+{
+ mlx5e_free_icosq_db(sq);
+ mlx5_wq_destroy(&sq->wq_ctrl);
+}
+
+static void mlx5e_free_txqsq_db(struct mlx5e_txqsq *sq)
+{
+ kfree(sq->db.wqe_info);
+ kfree(sq->db.dma_fifo);
+}
+
+static int mlx5e_alloc_txqsq_db(struct mlx5e_txqsq *sq, int numa)
{
int wq_sz = mlx5_wq_cyc_get_size(&sq->wq);
int df_sz = wq_sz * MLX5_SEND_WQEBB_NUM_DS;
- sq->db.txq.skb = kzalloc_node(wq_sz * sizeof(*sq->db.txq.skb),
- GFP_KERNEL, numa);
- sq->db.txq.dma_fifo = kzalloc_node(df_sz * sizeof(*sq->db.txq.dma_fifo),
+ sq->db.dma_fifo = kzalloc_node(df_sz * sizeof(*sq->db.dma_fifo),
GFP_KERNEL, numa);
- sq->db.txq.wqe_info = kzalloc_node(wq_sz * sizeof(*sq->db.txq.wqe_info),
+ sq->db.wqe_info = kzalloc_node(wq_sz * sizeof(*sq->db.wqe_info),
GFP_KERNEL, numa);
- if (!sq->db.txq.skb || !sq->db.txq.dma_fifo || !sq->db.txq.wqe_info) {
- mlx5e_free_sq_txq_db(sq);
+ if (!sq->db.dma_fifo || !sq->db.wqe_info) {
+ mlx5e_free_txqsq_db(sq);
return -ENOMEM;
}
@@ -958,127 +1068,64 @@
return 0;
}
-static void mlx5e_free_sq_db(struct mlx5e_sq *sq)
+static int mlx5e_alloc_txqsq(struct mlx5e_channel *c,
+ int txq_ix,
+ struct mlx5e_params *params,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_txqsq *sq)
{
- switch (sq->type) {
- case MLX5E_SQ_TXQ:
- mlx5e_free_sq_txq_db(sq);
- break;
- case MLX5E_SQ_ICO:
- mlx5e_free_sq_ico_db(sq);
- break;
- case MLX5E_SQ_XDP:
- mlx5e_free_sq_xdp_db(sq);
- break;
- }
-}
-
-static int mlx5e_alloc_sq_db(struct mlx5e_sq *sq, int numa)
-{
- switch (sq->type) {
- case MLX5E_SQ_TXQ:
- return mlx5e_alloc_sq_txq_db(sq, numa);
- case MLX5E_SQ_ICO:
- return mlx5e_alloc_sq_ico_db(sq, numa);
- case MLX5E_SQ_XDP:
- return mlx5e_alloc_sq_xdp_db(sq, numa);
- }
-
- return 0;
-}
-
-static int mlx5e_sq_get_max_wqebbs(u8 sq_type)
-{
- switch (sq_type) {
- case MLX5E_SQ_ICO:
- return MLX5E_ICOSQ_MAX_WQEBBS;
- case MLX5E_SQ_XDP:
- return MLX5E_XDP_TX_WQEBBS;
- }
- return MLX5_SEND_WQE_MAX_WQEBBS;
-}
-
-static int mlx5e_create_sq(struct mlx5e_channel *c,
- int tc,
- struct mlx5e_sq_param *param,
- struct mlx5e_sq *sq)
-{
- struct mlx5e_priv *priv = c->priv;
- struct mlx5_core_dev *mdev = priv->mdev;
-
- void *sqc = param->sqc;
- void *sqc_wq = MLX5_ADDR_OF(sqc, sqc, wq);
+ void *sqc_wq = MLX5_ADDR_OF(sqc, param->sqc, wq);
+ struct mlx5_core_dev *mdev = c->mdev;
int err;
- sq->type = param->type;
sq->pdev = c->pdev;
- sq->tstamp = &priv->tstamp;
+ sq->tstamp = c->tstamp;
sq->mkey_be = c->mkey_be;
sq->channel = c;
- sq->tc = tc;
+ sq->txq_ix = txq_ix;
+ sq->uar_map = mdev->mlx5e_res.bfreg.map;
+ sq->max_inline = params->tx_max_inline;
+ sq->min_inline_mode = params->tx_min_inline_mode;
- err = mlx5_alloc_bfreg(mdev, &sq->bfreg, MLX5_CAP_GEN(mdev, bf), false);
+ param->wq.db_numa_node = cpu_to_node(c->cpu);
+ err = mlx5_wq_cyc_create(mdev, ¶m->wq, sqc_wq, &sq->wq, &sq->wq_ctrl);
if (err)
return err;
+ sq->wq.db = &sq->wq.db[MLX5_SND_DBR];
- sq->uar_map = sq->bfreg.map;
- param->wq.db_numa_node = cpu_to_node(c->cpu);
-
- err = mlx5_wq_cyc_create(mdev, ¶m->wq, sqc_wq, &sq->wq,
- &sq->wq_ctrl);
- if (err)
- goto err_unmap_free_uar;
-
- sq->wq.db = &sq->wq.db[MLX5_SND_DBR];
- if (sq->bfreg.wc)
- set_bit(MLX5E_SQ_STATE_BF_ENABLE, &sq->state);
-
- sq->bf_buf_size = (1 << MLX5_CAP_GEN(mdev, log_bf_reg_size)) / 2;
- sq->max_inline = param->max_inline;
- sq->min_inline_mode = param->min_inline_mode;
-
- err = mlx5e_alloc_sq_db(sq, cpu_to_node(c->cpu));
+ err = mlx5e_alloc_txqsq_db(sq, cpu_to_node(c->cpu));
if (err)
goto err_sq_wq_destroy;
- if (sq->type == MLX5E_SQ_TXQ) {
- int txq_ix;
-
- txq_ix = c->ix + tc * priv->params.num_channels;
- sq->txq = netdev_get_tx_queue(priv->netdev, txq_ix);
- priv->txq_to_sq_map[txq_ix] = sq;
- }
-
- sq->edge = (sq->wq.sz_m1 + 1) - mlx5e_sq_get_max_wqebbs(sq->type);
- sq->bf_budget = MLX5E_SQ_BF_BUDGET;
+ sq->edge = (sq->wq.sz_m1 + 1) - MLX5_SEND_WQE_MAX_WQEBBS;
return 0;
err_sq_wq_destroy:
mlx5_wq_destroy(&sq->wq_ctrl);
-err_unmap_free_uar:
- mlx5_free_bfreg(mdev, &sq->bfreg);
-
return err;
}
-static void mlx5e_destroy_sq(struct mlx5e_sq *sq)
+static void mlx5e_free_txqsq(struct mlx5e_txqsq *sq)
{
- struct mlx5e_channel *c = sq->channel;
- struct mlx5e_priv *priv = c->priv;
-
- mlx5e_free_sq_db(sq);
+ mlx5e_free_txqsq_db(sq);
mlx5_wq_destroy(&sq->wq_ctrl);
- mlx5_free_bfreg(priv->mdev, &sq->bfreg);
}
-static int mlx5e_enable_sq(struct mlx5e_sq *sq, struct mlx5e_sq_param *param)
-{
- struct mlx5e_channel *c = sq->channel;
- struct mlx5e_priv *priv = c->priv;
- struct mlx5_core_dev *mdev = priv->mdev;
+struct mlx5e_create_sq_param {
+ struct mlx5_wq_ctrl *wq_ctrl;
+ u32 cqn;
+ u32 tisn;
+ u8 tis_lst_sz;
+ u8 min_inline_mode;
+};
+static int mlx5e_create_sq(struct mlx5_core_dev *mdev,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_create_sq_param *csp,
+ u32 *sqn)
+{
void *in;
void *sqc;
void *wq;
@@ -1086,7 +1133,7 @@
int err;
inlen = MLX5_ST_SZ_BYTES(create_sq_in) +
- sizeof(u64) * sq->wq_ctrl.buf.npages;
+ sizeof(u64) * csp->wq_ctrl->buf.npages;
in = mlx5_vzalloc(inlen);
if (!in)
return -ENOMEM;
@@ -1095,40 +1142,40 @@
wq = MLX5_ADDR_OF(sqc, sqc, wq);
memcpy(sqc, param->sqc, sizeof(param->sqc));
-
- MLX5_SET(sqc, sqc, tis_num_0, param->type == MLX5E_SQ_ICO ?
- 0 : priv->tisn[sq->tc]);
- MLX5_SET(sqc, sqc, cqn, sq->cq.mcq.cqn);
+ MLX5_SET(sqc, sqc, tis_lst_sz, csp->tis_lst_sz);
+ MLX5_SET(sqc, sqc, tis_num_0, csp->tisn);
+ MLX5_SET(sqc, sqc, cqn, csp->cqn);
if (MLX5_CAP_ETH(mdev, wqe_inline_mode) == MLX5_CAP_INLINE_MODE_VPORT_CONTEXT)
- MLX5_SET(sqc, sqc, min_wqe_inline_mode, sq->min_inline_mode);
+ MLX5_SET(sqc, sqc, min_wqe_inline_mode, csp->min_inline_mode);
- MLX5_SET(sqc, sqc, state, MLX5_SQC_STATE_RST);
- MLX5_SET(sqc, sqc, tis_lst_sz, param->type == MLX5E_SQ_ICO ? 0 : 1);
+ MLX5_SET(sqc, sqc, state, MLX5_SQC_STATE_RST);
MLX5_SET(wq, wq, wq_type, MLX5_WQ_TYPE_CYCLIC);
- MLX5_SET(wq, wq, uar_page, sq->bfreg.index);
- MLX5_SET(wq, wq, log_wq_pg_sz, sq->wq_ctrl.buf.page_shift -
+ MLX5_SET(wq, wq, uar_page, mdev->mlx5e_res.bfreg.index);
+ MLX5_SET(wq, wq, log_wq_pg_sz, csp->wq_ctrl->buf.page_shift -
MLX5_ADAPTER_PAGE_SHIFT);
- MLX5_SET64(wq, wq, dbr_addr, sq->wq_ctrl.db.dma);
+ MLX5_SET64(wq, wq, dbr_addr, csp->wq_ctrl->db.dma);
- mlx5_fill_page_array(&sq->wq_ctrl.buf,
- (__be64 *)MLX5_ADDR_OF(wq, wq, pas));
+ mlx5_fill_page_array(&csp->wq_ctrl->buf, (__be64 *)MLX5_ADDR_OF(wq, wq, pas));
- err = mlx5_core_create_sq(mdev, in, inlen, &sq->sqn);
+ err = mlx5_core_create_sq(mdev, in, inlen, sqn);
kvfree(in);
return err;
}
-static int mlx5e_modify_sq(struct mlx5e_sq *sq, int curr_state,
- int next_state, bool update_rl, int rl_index)
-{
- struct mlx5e_channel *c = sq->channel;
- struct mlx5e_priv *priv = c->priv;
- struct mlx5_core_dev *mdev = priv->mdev;
+struct mlx5e_modify_sq_param {
+ int curr_state;
+ int next_state;
+ bool rl_update;
+ int rl_index;
+};
+static int mlx5e_modify_sq(struct mlx5_core_dev *mdev, u32 sqn,
+ struct mlx5e_modify_sq_param *p)
+{
void *in;
void *sqc;
int inlen;
@@ -1141,68 +1188,94 @@
sqc = MLX5_ADDR_OF(modify_sq_in, in, ctx);
- MLX5_SET(modify_sq_in, in, sq_state, curr_state);
- MLX5_SET(sqc, sqc, state, next_state);
- if (update_rl && next_state == MLX5_SQC_STATE_RDY) {
+ MLX5_SET(modify_sq_in, in, sq_state, p->curr_state);
+ MLX5_SET(sqc, sqc, state, p->next_state);
+ if (p->rl_update && p->next_state == MLX5_SQC_STATE_RDY) {
MLX5_SET64(modify_sq_in, in, modify_bitmask, 1);
- MLX5_SET(sqc, sqc, packet_pacing_rate_limit_index, rl_index);
+ MLX5_SET(sqc, sqc, packet_pacing_rate_limit_index, p->rl_index);
}
- err = mlx5_core_modify_sq(mdev, sq->sqn, in, inlen);
+ err = mlx5_core_modify_sq(mdev, sqn, in, inlen);
kvfree(in);
return err;
}
-static void mlx5e_disable_sq(struct mlx5e_sq *sq)
+static void mlx5e_destroy_sq(struct mlx5_core_dev *mdev, u32 sqn)
{
- struct mlx5e_channel *c = sq->channel;
- struct mlx5e_priv *priv = c->priv;
- struct mlx5_core_dev *mdev = priv->mdev;
-
- mlx5_core_destroy_sq(mdev, sq->sqn);
- if (sq->rate_limit)
- mlx5_rl_remove_rate(mdev, sq->rate_limit);
+ mlx5_core_destroy_sq(mdev, sqn);
}
-static int mlx5e_open_sq(struct mlx5e_channel *c,
- int tc,
- struct mlx5e_sq_param *param,
- struct mlx5e_sq *sq)
+static int mlx5e_create_sq_rdy(struct mlx5_core_dev *mdev,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_create_sq_param *csp,
+ u32 *sqn)
{
+ struct mlx5e_modify_sq_param msp = {0};
int err;
- err = mlx5e_create_sq(c, tc, param, sq);
+ err = mlx5e_create_sq(mdev, param, csp, sqn);
if (err)
return err;
- err = mlx5e_enable_sq(sq, param);
+ msp.curr_state = MLX5_SQC_STATE_RST;
+ msp.next_state = MLX5_SQC_STATE_RDY;
+ err = mlx5e_modify_sq(mdev, *sqn, &msp);
if (err)
- goto err_destroy_sq;
+ mlx5e_destroy_sq(mdev, *sqn);
- set_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
- err = mlx5e_modify_sq(sq, MLX5_SQC_STATE_RST, MLX5_SQC_STATE_RDY,
- false, 0);
+ return err;
+}
+
+static int mlx5e_set_sq_maxrate(struct net_device *dev,
+ struct mlx5e_txqsq *sq, u32 rate);
+
+static int mlx5e_open_txqsq(struct mlx5e_channel *c,
+ u32 tisn,
+ int txq_ix,
+ struct mlx5e_params *params,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_txqsq *sq)
+{
+ struct mlx5e_create_sq_param csp = {};
+ u32 tx_rate;
+ int err;
+
+ err = mlx5e_alloc_txqsq(c, txq_ix, params, param, sq);
if (err)
- goto err_disable_sq;
+ return err;
- if (sq->txq) {
- netdev_tx_reset_queue(sq->txq);
- netif_tx_start_queue(sq->txq);
- }
+ csp.tisn = tisn;
+ csp.tis_lst_sz = 1;
+ csp.cqn = sq->cq.mcq.cqn;
+ csp.wq_ctrl = &sq->wq_ctrl;
+ csp.min_inline_mode = sq->min_inline_mode;
+ err = mlx5e_create_sq_rdy(c->mdev, param, &csp, &sq->sqn);
+ if (err)
+ goto err_free_txqsq;
+
+ tx_rate = c->priv->tx_rates[sq->txq_ix];
+ if (tx_rate)
+ mlx5e_set_sq_maxrate(c->netdev, sq, tx_rate);
return 0;
-err_disable_sq:
+err_free_txqsq:
clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
- mlx5e_disable_sq(sq);
-err_destroy_sq:
- mlx5e_destroy_sq(sq);
+ mlx5e_free_txqsq(sq);
return err;
}
+static void mlx5e_activate_txqsq(struct mlx5e_txqsq *sq)
+{
+ sq->txq = netdev_get_tx_queue(sq->channel->netdev, sq->txq_ix);
+ set_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
+ netdev_tx_reset_queue(sq->txq);
+ netif_tx_start_queue(sq->txq);
+}
+
static inline void netif_tx_disable_queue(struct netdev_queue *txq)
{
__netif_tx_lock_bh(txq);
@@ -1210,43 +1283,153 @@
__netif_tx_unlock_bh(txq);
}
-static void mlx5e_close_sq(struct mlx5e_sq *sq)
+static void mlx5e_deactivate_txqsq(struct mlx5e_txqsq *sq)
{
+ struct mlx5e_channel *c = sq->channel;
+
clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
/* prevent netif_tx_wake_queue */
- napi_synchronize(&sq->channel->napi);
+ napi_synchronize(&c->napi);
- if (sq->txq) {
- netif_tx_disable_queue(sq->txq);
+ netif_tx_disable_queue(sq->txq);
- /* last doorbell out, godspeed .. */
- if (mlx5e_sq_has_room_for(sq, 1)) {
- sq->db.txq.skb[(sq->pc & sq->wq.sz_m1)] = NULL;
- mlx5e_send_nop(sq, true);
- }
+ /* last doorbell out, godspeed .. */
+ if (mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, 1)) {
+ struct mlx5e_tx_wqe *nop;
+
+ sq->db.wqe_info[(sq->pc & sq->wq.sz_m1)].skb = NULL;
+ nop = mlx5e_post_nop(&sq->wq, sq->sqn, &sq->pc);
+ mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &nop->ctrl);
}
-
- mlx5e_disable_sq(sq);
- mlx5e_free_sq_descs(sq);
- mlx5e_destroy_sq(sq);
}
-static int mlx5e_create_cq(struct mlx5e_channel *c,
- struct mlx5e_cq_param *param,
- struct mlx5e_cq *cq)
+static void mlx5e_close_txqsq(struct mlx5e_txqsq *sq)
{
- struct mlx5e_priv *priv = c->priv;
- struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5e_channel *c = sq->channel;
+ struct mlx5_core_dev *mdev = c->mdev;
+
+ mlx5e_destroy_sq(mdev, sq->sqn);
+ if (sq->rate_limit)
+ mlx5_rl_remove_rate(mdev, sq->rate_limit);
+ mlx5e_free_txqsq_descs(sq);
+ mlx5e_free_txqsq(sq);
+}
+
+static int mlx5e_open_icosq(struct mlx5e_channel *c,
+ struct mlx5e_params *params,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_icosq *sq)
+{
+ struct mlx5e_create_sq_param csp = {};
+ int err;
+
+ err = mlx5e_alloc_icosq(c, param, sq);
+ if (err)
+ return err;
+
+ csp.cqn = sq->cq.mcq.cqn;
+ csp.wq_ctrl = &sq->wq_ctrl;
+ csp.min_inline_mode = params->tx_min_inline_mode;
+ set_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
+ err = mlx5e_create_sq_rdy(c->mdev, param, &csp, &sq->sqn);
+ if (err)
+ goto err_free_icosq;
+
+ return 0;
+
+err_free_icosq:
+ clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
+ mlx5e_free_icosq(sq);
+
+ return err;
+}
+
+static void mlx5e_close_icosq(struct mlx5e_icosq *sq)
+{
+ struct mlx5e_channel *c = sq->channel;
+
+ clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
+ napi_synchronize(&c->napi);
+
+ mlx5e_destroy_sq(c->mdev, sq->sqn);
+ mlx5e_free_icosq(sq);
+}
+
+static int mlx5e_open_xdpsq(struct mlx5e_channel *c,
+ struct mlx5e_params *params,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_xdpsq *sq)
+{
+ unsigned int ds_cnt = MLX5E_XDP_TX_DS_COUNT;
+ struct mlx5e_create_sq_param csp = {};
+ unsigned int inline_hdr_sz = 0;
+ int err;
+ int i;
+
+ err = mlx5e_alloc_xdpsq(c, params, param, sq);
+ if (err)
+ return err;
+
+ csp.tis_lst_sz = 1;
+ csp.tisn = c->priv->tisn[0]; /* tc = 0 */
+ csp.cqn = sq->cq.mcq.cqn;
+ csp.wq_ctrl = &sq->wq_ctrl;
+ csp.min_inline_mode = sq->min_inline_mode;
+ set_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
+ err = mlx5e_create_sq_rdy(c->mdev, param, &csp, &sq->sqn);
+ if (err)
+ goto err_free_xdpsq;
+
+ if (sq->min_inline_mode != MLX5_INLINE_MODE_NONE) {
+ inline_hdr_sz = MLX5E_XDP_MIN_INLINE;
+ ds_cnt++;
+ }
+
+ /* Pre initialize fixed WQE fields */
+ for (i = 0; i < mlx5_wq_cyc_get_size(&sq->wq); i++) {
+ struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(&sq->wq, i);
+ struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+ struct mlx5_wqe_eth_seg *eseg = &wqe->eth;
+ struct mlx5_wqe_data_seg *dseg;
+
+ cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt);
+ eseg->inline_hdr.sz = cpu_to_be16(inline_hdr_sz);
+
+ dseg = (struct mlx5_wqe_data_seg *)cseg + (ds_cnt - 1);
+ dseg->lkey = sq->mkey_be;
+ }
+
+ return 0;
+
+err_free_xdpsq:
+ clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
+ mlx5e_free_xdpsq(sq);
+
+ return err;
+}
+
+static void mlx5e_close_xdpsq(struct mlx5e_xdpsq *sq)
+{
+ struct mlx5e_channel *c = sq->channel;
+
+ clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
+ napi_synchronize(&c->napi);
+
+ mlx5e_destroy_sq(c->mdev, sq->sqn);
+ mlx5e_free_xdpsq_descs(sq);
+ mlx5e_free_xdpsq(sq);
+}
+
+static int mlx5e_alloc_cq_common(struct mlx5_core_dev *mdev,
+ struct mlx5e_cq_param *param,
+ struct mlx5e_cq *cq)
+{
struct mlx5_core_cq *mcq = &cq->mcq;
int eqn_not_used;
unsigned int irqn;
int err;
u32 i;
- param->wq.buf_numa_node = cpu_to_node(c->cpu);
- param->wq.db_numa_node = cpu_to_node(c->cpu);
- param->eq_ix = c->ix;
-
err = mlx5_cqwq_create(mdev, ¶m->wq, param->cqc, &cq->wq,
&cq->wq_ctrl);
if (err)
@@ -1254,8 +1437,6 @@
mlx5_vector2eqn(mdev, param->eq_ix, &eqn_not_used, &irqn);
- cq->napi = &c->napi;
-
mcq->cqe_sz = 64;
mcq->set_ci_db = cq->wq_ctrl.db.db;
mcq->arm_db = cq->wq_ctrl.db.db + 1;
@@ -1272,21 +1453,38 @@
cqe->op_own = 0xf1;
}
- cq->channel = c;
- cq->priv = priv;
+ cq->mdev = mdev;
return 0;
}
-static void mlx5e_destroy_cq(struct mlx5e_cq *cq)
+static int mlx5e_alloc_cq(struct mlx5e_channel *c,
+ struct mlx5e_cq_param *param,
+ struct mlx5e_cq *cq)
+{
+ struct mlx5_core_dev *mdev = c->priv->mdev;
+ int err;
+
+ param->wq.buf_numa_node = cpu_to_node(c->cpu);
+ param->wq.db_numa_node = cpu_to_node(c->cpu);
+ param->eq_ix = c->ix;
+
+ err = mlx5e_alloc_cq_common(mdev, param, cq);
+
+ cq->napi = &c->napi;
+ cq->channel = c;
+
+ return err;
+}
+
+static void mlx5e_free_cq(struct mlx5e_cq *cq)
{
mlx5_cqwq_destroy(&cq->wq_ctrl);
}
-static int mlx5e_enable_cq(struct mlx5e_cq *cq, struct mlx5e_cq_param *param)
+static int mlx5e_create_cq(struct mlx5e_cq *cq, struct mlx5e_cq_param *param)
{
- struct mlx5e_priv *priv = cq->priv;
- struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5_core_dev *mdev = cq->mdev;
struct mlx5_core_cq *mcq = &cq->mcq;
void *in;
@@ -1330,47 +1528,41 @@
return 0;
}
-static void mlx5e_disable_cq(struct mlx5e_cq *cq)
+static void mlx5e_destroy_cq(struct mlx5e_cq *cq)
{
- struct mlx5e_priv *priv = cq->priv;
- struct mlx5_core_dev *mdev = priv->mdev;
-
- mlx5_core_destroy_cq(mdev, &cq->mcq);
+ mlx5_core_destroy_cq(cq->mdev, &cq->mcq);
}
static int mlx5e_open_cq(struct mlx5e_channel *c,
+ struct mlx5e_cq_moder moder,
struct mlx5e_cq_param *param,
- struct mlx5e_cq *cq,
- struct mlx5e_cq_moder moderation)
+ struct mlx5e_cq *cq)
{
+ struct mlx5_core_dev *mdev = c->mdev;
int err;
- struct mlx5e_priv *priv = c->priv;
- struct mlx5_core_dev *mdev = priv->mdev;
- err = mlx5e_create_cq(c, param, cq);
+ err = mlx5e_alloc_cq(c, param, cq);
if (err)
return err;
- err = mlx5e_enable_cq(cq, param);
+ err = mlx5e_create_cq(cq, param);
if (err)
- goto err_destroy_cq;
+ goto err_free_cq;
if (MLX5_CAP_GEN(mdev, cq_moderation))
- mlx5_core_modify_cq_moderation(mdev, &cq->mcq,
- moderation.usec,
- moderation.pkts);
+ mlx5_core_modify_cq_moderation(mdev, &cq->mcq, moder.usec, moder.pkts);
return 0;
-err_destroy_cq:
- mlx5e_destroy_cq(cq);
+err_free_cq:
+ mlx5e_free_cq(cq);
return err;
}
static void mlx5e_close_cq(struct mlx5e_cq *cq)
{
- mlx5e_disable_cq(cq);
mlx5e_destroy_cq(cq);
+ mlx5e_free_cq(cq);
}
static int mlx5e_get_cpu(struct mlx5e_priv *priv, int ix)
@@ -1379,15 +1571,15 @@
}
static int mlx5e_open_tx_cqs(struct mlx5e_channel *c,
+ struct mlx5e_params *params,
struct mlx5e_channel_param *cparam)
{
- struct mlx5e_priv *priv = c->priv;
int err;
int tc;
for (tc = 0; tc < c->num_tc; tc++) {
- err = mlx5e_open_cq(c, &cparam->tx_cq, &c->sq[tc].cq,
- priv->params.tx_cq_moderation);
+ err = mlx5e_open_cq(c, params->tx_cq_moderation,
+ &cparam->tx_cq, &c->sq[tc].cq);
if (err)
goto err_close_tx_cqs;
}
@@ -1410,13 +1602,17 @@
}
static int mlx5e_open_sqs(struct mlx5e_channel *c,
+ struct mlx5e_params *params,
struct mlx5e_channel_param *cparam)
{
int err;
int tc;
- for (tc = 0; tc < c->num_tc; tc++) {
- err = mlx5e_open_sq(c, tc, &cparam->sq, &c->sq[tc]);
+ for (tc = 0; tc < params->num_tc; tc++) {
+ int txq_ix = c->ix + tc * params->num_channels;
+
+ err = mlx5e_open_txqsq(c, c->priv->tisn[tc], txq_ix,
+ params, &cparam->sq, &c->sq[tc]);
if (err)
goto err_close_sqs;
}
@@ -1425,7 +1621,7 @@
err_close_sqs:
for (tc--; tc >= 0; tc--)
- mlx5e_close_sq(&c->sq[tc]);
+ mlx5e_close_txqsq(&c->sq[tc]);
return err;
}
@@ -1435,23 +1631,15 @@
int tc;
for (tc = 0; tc < c->num_tc; tc++)
- mlx5e_close_sq(&c->sq[tc]);
-}
-
-static void mlx5e_build_channeltc_to_txq_map(struct mlx5e_priv *priv, int ix)
-{
- int i;
-
- for (i = 0; i < priv->profile->max_tc; i++)
- priv->channeltc_to_txq_map[ix][i] =
- ix + i * priv->params.num_channels;
+ mlx5e_close_txqsq(&c->sq[tc]);
}
static int mlx5e_set_sq_maxrate(struct net_device *dev,
- struct mlx5e_sq *sq, u32 rate)
+ struct mlx5e_txqsq *sq, u32 rate)
{
struct mlx5e_priv *priv = netdev_priv(dev);
struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5e_modify_sq_param msp = {0};
u16 rl_index = 0;
int err;
@@ -1474,8 +1662,11 @@
}
}
- err = mlx5e_modify_sq(sq, MLX5_SQC_STATE_RDY,
- MLX5_SQC_STATE_RDY, true, rl_index);
+ msp.curr_state = MLX5_SQC_STATE_RDY;
+ msp.next_state = MLX5_SQC_STATE_RDY;
+ msp.rl_index = rl_index;
+ msp.rl_update = true;
+ err = mlx5e_modify_sq(mdev, sq->sqn, &msp);
if (err) {
netdev_err(dev, "Failed configuring rate %u: %d\n",
rate, err);
@@ -1493,7 +1684,7 @@
{
struct mlx5e_priv *priv = netdev_priv(dev);
struct mlx5_core_dev *mdev = priv->mdev;
- struct mlx5e_sq *sq = priv->txq_to_sq_map[index];
+ struct mlx5e_txqsq *sq = priv->txq2sq[index];
int err = 0;
if (!mlx5_rl_is_supported(mdev)) {
@@ -1520,114 +1711,87 @@
return err;
}
-static inline int mlx5e_get_max_num_channels(struct mlx5_core_dev *mdev)
-{
- return is_kdump_kernel() ?
- MLX5E_MIN_NUM_CHANNELS :
- min_t(int, mdev->priv.eq_table.num_comp_vectors,
- MLX5E_MAX_NUM_CHANNELS);
-}
-
static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
+ struct mlx5e_params *params,
struct mlx5e_channel_param *cparam,
struct mlx5e_channel **cp)
{
- struct mlx5e_cq_moder icosq_cq_moder = {0, 0};
+ struct mlx5e_cq_moder icocq_moder = {0, 0};
struct net_device *netdev = priv->netdev;
- struct mlx5e_cq_moder rx_cq_profile;
int cpu = mlx5e_get_cpu(priv, ix);
struct mlx5e_channel *c;
- struct mlx5e_sq *sq;
int err;
- int i;
c = kzalloc_node(sizeof(*c), GFP_KERNEL, cpu_to_node(cpu));
if (!c)
return -ENOMEM;
c->priv = priv;
+ c->mdev = priv->mdev;
+ c->tstamp = &priv->tstamp;
c->ix = ix;
c->cpu = cpu;
c->pdev = &priv->mdev->pdev->dev;
c->netdev = priv->netdev;
c->mkey_be = cpu_to_be32(priv->mdev->mlx5e_res.mkey.key);
- c->num_tc = priv->params.num_tc;
- c->xdp = !!priv->xdp_prog;
-
- if (priv->params.rx_am_enabled)
- rx_cq_profile = mlx5e_am_get_def_profile(priv->params.rx_cq_period_mode);
- else
- rx_cq_profile = priv->params.rx_cq_moderation;
-
- mlx5e_build_channeltc_to_txq_map(priv, ix);
+ c->num_tc = params->num_tc;
+ c->xdp = !!params->xdp_prog;
netif_napi_add(netdev, &c->napi, mlx5e_napi_poll, 64);
- err = mlx5e_open_cq(c, &cparam->icosq_cq, &c->icosq.cq, icosq_cq_moder);
+ err = mlx5e_open_cq(c, icocq_moder, &cparam->icosq_cq, &c->icosq.cq);
if (err)
goto err_napi_del;
- err = mlx5e_open_tx_cqs(c, cparam);
+ err = mlx5e_open_tx_cqs(c, params, cparam);
if (err)
goto err_close_icosq_cq;
- err = mlx5e_open_cq(c, &cparam->rx_cq, &c->rq.cq,
- rx_cq_profile);
+ err = mlx5e_open_cq(c, params->rx_cq_moderation, &cparam->rx_cq, &c->rq.cq);
if (err)
goto err_close_tx_cqs;
/* XDP SQ CQ params are same as normal TXQ sq CQ params */
- err = c->xdp ? mlx5e_open_cq(c, &cparam->tx_cq, &c->xdp_sq.cq,
- priv->params.tx_cq_moderation) : 0;
+ err = c->xdp ? mlx5e_open_cq(c, params->tx_cq_moderation,
+ &cparam->tx_cq, &c->rq.xdpsq.cq) : 0;
if (err)
goto err_close_rx_cq;
napi_enable(&c->napi);
- err = mlx5e_open_sq(c, 0, &cparam->icosq, &c->icosq);
+ err = mlx5e_open_icosq(c, params, &cparam->icosq, &c->icosq);
if (err)
goto err_disable_napi;
- err = mlx5e_open_sqs(c, cparam);
+ err = mlx5e_open_sqs(c, params, cparam);
if (err)
goto err_close_icosq;
- for (i = 0; i < priv->params.num_tc; i++) {
- u32 txq_ix = priv->channeltc_to_txq_map[ix][i];
-
- if (priv->tx_rates[txq_ix]) {
- sq = priv->txq_to_sq_map[txq_ix];
- mlx5e_set_sq_maxrate(priv->netdev, sq,
- priv->tx_rates[txq_ix]);
- }
- }
-
- err = c->xdp ? mlx5e_open_sq(c, 0, &cparam->xdp_sq, &c->xdp_sq) : 0;
+ err = c->xdp ? mlx5e_open_xdpsq(c, params, &cparam->xdp_sq, &c->rq.xdpsq) : 0;
if (err)
goto err_close_sqs;
- err = mlx5e_open_rq(c, &cparam->rq, &c->rq);
+ err = mlx5e_open_rq(c, params, &cparam->rq, &c->rq);
if (err)
goto err_close_xdp_sq;
- netif_set_xps_queue(netdev, get_cpu_mask(c->cpu), ix);
*cp = c;
return 0;
err_close_xdp_sq:
if (c->xdp)
- mlx5e_close_sq(&c->xdp_sq);
+ mlx5e_close_xdpsq(&c->rq.xdpsq);
err_close_sqs:
mlx5e_close_sqs(c);
err_close_icosq:
- mlx5e_close_sq(&c->icosq);
+ mlx5e_close_icosq(&c->icosq);
err_disable_napi:
napi_disable(&c->napi);
if (c->xdp)
- mlx5e_close_cq(&c->xdp_sq.cq);
+ mlx5e_close_cq(&c->rq.xdpsq.cq);
err_close_rx_cq:
mlx5e_close_cq(&c->rq.cq);
@@ -1645,16 +1809,35 @@
return err;
}
+static void mlx5e_activate_channel(struct mlx5e_channel *c)
+{
+ int tc;
+
+ for (tc = 0; tc < c->num_tc; tc++)
+ mlx5e_activate_txqsq(&c->sq[tc]);
+ mlx5e_activate_rq(&c->rq);
+ netif_set_xps_queue(c->netdev, get_cpu_mask(c->cpu), c->ix);
+}
+
+static void mlx5e_deactivate_channel(struct mlx5e_channel *c)
+{
+ int tc;
+
+ mlx5e_deactivate_rq(&c->rq);
+ for (tc = 0; tc < c->num_tc; tc++)
+ mlx5e_deactivate_txqsq(&c->sq[tc]);
+}
+
static void mlx5e_close_channel(struct mlx5e_channel *c)
{
mlx5e_close_rq(&c->rq);
if (c->xdp)
- mlx5e_close_sq(&c->xdp_sq);
+ mlx5e_close_xdpsq(&c->rq.xdpsq);
mlx5e_close_sqs(c);
- mlx5e_close_sq(&c->icosq);
+ mlx5e_close_icosq(&c->icosq);
napi_disable(&c->napi);
if (c->xdp)
- mlx5e_close_cq(&c->xdp_sq.cq);
+ mlx5e_close_cq(&c->rq.xdpsq.cq);
mlx5e_close_cq(&c->rq.cq);
mlx5e_close_tx_cqs(c);
mlx5e_close_cq(&c->icosq.cq);
@@ -1664,17 +1847,16 @@
}
static void mlx5e_build_rq_param(struct mlx5e_priv *priv,
+ struct mlx5e_params *params,
struct mlx5e_rq_param *param)
{
void *rqc = param->rqc;
void *wq = MLX5_ADDR_OF(rqc, rqc, wq);
- switch (priv->params.rq_wq_type) {
+ switch (params->rq_wq_type) {
case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ:
- MLX5_SET(wq, wq, log_wqe_num_of_strides,
- priv->params.mpwqe_log_num_strides - 9);
- MLX5_SET(wq, wq, log_wqe_stride_size,
- priv->params.mpwqe_log_stride_sz - 6);
+ MLX5_SET(wq, wq, log_wqe_num_of_strides, params->mpwqe_log_num_strides - 9);
+ MLX5_SET(wq, wq, log_wqe_stride_size, params->mpwqe_log_stride_sz - 6);
MLX5_SET(wq, wq, wq_type, MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ);
break;
default: /* MLX5_WQ_TYPE_LINKED_LIST */
@@ -1683,14 +1865,14 @@
MLX5_SET(wq, wq, end_padding_mode, MLX5_WQ_END_PAD_MODE_ALIGN);
MLX5_SET(wq, wq, log_wq_stride, ilog2(sizeof(struct mlx5e_rx_wqe)));
- MLX5_SET(wq, wq, log_wq_sz, priv->params.log_rq_size);
+ MLX5_SET(wq, wq, log_wq_sz, params->log_rq_size);
MLX5_SET(wq, wq, pd, priv->mdev->mlx5e_res.pdn);
MLX5_SET(rqc, rqc, counter_set_id, priv->q_counter);
+ MLX5_SET(rqc, rqc, vsd, params->vlan_strip_disable);
+ MLX5_SET(rqc, rqc, scatter_fcs, params->scatter_fcs_en);
param->wq.buf_numa_node = dev_to_node(&priv->mdev->pdev->dev);
param->wq.linear = 1;
-
- param->am_enabled = priv->params.rx_am_enabled;
}
static void mlx5e_build_drop_rq_param(struct mlx5e_rq_param *param)
@@ -1715,17 +1897,14 @@
}
static void mlx5e_build_sq_param(struct mlx5e_priv *priv,
+ struct mlx5e_params *params,
struct mlx5e_sq_param *param)
{
void *sqc = param->sqc;
void *wq = MLX5_ADDR_OF(sqc, sqc, wq);
mlx5e_build_sq_param_common(priv, param);
- MLX5_SET(wq, wq, log_wq_sz, priv->params.log_sq_size);
-
- param->max_inline = priv->params.tx_max_inline;
- param->min_inline_mode = priv->params.tx_min_inline_mode;
- param->type = MLX5E_SQ_TXQ;
+ MLX5_SET(wq, wq, log_wq_sz, params->log_sq_size);
}
static void mlx5e_build_common_cq_param(struct mlx5e_priv *priv,
@@ -1737,37 +1916,36 @@
}
static void mlx5e_build_rx_cq_param(struct mlx5e_priv *priv,
+ struct mlx5e_params *params,
struct mlx5e_cq_param *param)
{
void *cqc = param->cqc;
u8 log_cq_size;
- switch (priv->params.rq_wq_type) {
+ switch (params->rq_wq_type) {
case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ:
- log_cq_size = priv->params.log_rq_size +
- priv->params.mpwqe_log_num_strides;
+ log_cq_size = params->log_rq_size + params->mpwqe_log_num_strides;
break;
default: /* MLX5_WQ_TYPE_LINKED_LIST */
- log_cq_size = priv->params.log_rq_size;
+ log_cq_size = params->log_rq_size;
}
MLX5_SET(cqc, cqc, log_cq_size, log_cq_size);
- if (MLX5E_GET_PFLAG(priv, MLX5E_PFLAG_RX_CQE_COMPRESS)) {
+ if (MLX5E_GET_PFLAG(params, MLX5E_PFLAG_RX_CQE_COMPRESS)) {
MLX5_SET(cqc, cqc, mini_cqe_res_format, MLX5_CQE_FORMAT_CSUM);
MLX5_SET(cqc, cqc, cqe_comp_en, 1);
}
mlx5e_build_common_cq_param(priv, param);
-
- param->cq_period_mode = priv->params.rx_cq_period_mode;
}
static void mlx5e_build_tx_cq_param(struct mlx5e_priv *priv,
+ struct mlx5e_params *params,
struct mlx5e_cq_param *param)
{
void *cqc = param->cqc;
- MLX5_SET(cqc, cqc, log_cq_size, priv->params.log_sq_size);
+ MLX5_SET(cqc, cqc, log_cq_size, params->log_sq_size);
mlx5e_build_common_cq_param(priv, param);
@@ -1775,8 +1953,8 @@
}
static void mlx5e_build_ico_cq_param(struct mlx5e_priv *priv,
- struct mlx5e_cq_param *param,
- u8 log_wq_size)
+ u8 log_wq_size,
+ struct mlx5e_cq_param *param)
{
void *cqc = param->cqc;
@@ -1788,8 +1966,8 @@
}
static void mlx5e_build_icosq_param(struct mlx5e_priv *priv,
- struct mlx5e_sq_param *param,
- u8 log_wq_size)
+ u8 log_wq_size,
+ struct mlx5e_sq_param *param)
{
void *sqc = param->sqc;
void *wq = MLX5_ADDR_OF(sqc, sqc, wq);
@@ -1798,105 +1976,187 @@
MLX5_SET(wq, wq, log_wq_sz, log_wq_size);
MLX5_SET(sqc, sqc, reg_umr, MLX5_CAP_ETH(priv->mdev, reg_umr_sq));
-
- param->type = MLX5E_SQ_ICO;
}
static void mlx5e_build_xdpsq_param(struct mlx5e_priv *priv,
+ struct mlx5e_params *params,
struct mlx5e_sq_param *param)
{
void *sqc = param->sqc;
void *wq = MLX5_ADDR_OF(sqc, sqc, wq);
mlx5e_build_sq_param_common(priv, param);
- MLX5_SET(wq, wq, log_wq_sz, priv->params.log_sq_size);
-
- param->max_inline = priv->params.tx_max_inline;
- param->min_inline_mode = priv->params.tx_min_inline_mode;
- param->type = MLX5E_SQ_XDP;
+ MLX5_SET(wq, wq, log_wq_sz, params->log_sq_size);
}
-static void mlx5e_build_channel_param(struct mlx5e_priv *priv, struct mlx5e_channel_param *cparam)
+static void mlx5e_build_channel_param(struct mlx5e_priv *priv,
+ struct mlx5e_params *params,
+ struct mlx5e_channel_param *cparam)
{
u8 icosq_log_wq_sz = MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE;
- mlx5e_build_rq_param(priv, &cparam->rq);
- mlx5e_build_sq_param(priv, &cparam->sq);
- mlx5e_build_xdpsq_param(priv, &cparam->xdp_sq);
- mlx5e_build_icosq_param(priv, &cparam->icosq, icosq_log_wq_sz);
- mlx5e_build_rx_cq_param(priv, &cparam->rx_cq);
- mlx5e_build_tx_cq_param(priv, &cparam->tx_cq);
- mlx5e_build_ico_cq_param(priv, &cparam->icosq_cq, icosq_log_wq_sz);
+ mlx5e_build_rq_param(priv, params, &cparam->rq);
+ mlx5e_build_sq_param(priv, params, &cparam->sq);
+ mlx5e_build_xdpsq_param(priv, params, &cparam->xdp_sq);
+ mlx5e_build_icosq_param(priv, icosq_log_wq_sz, &cparam->icosq);
+ mlx5e_build_rx_cq_param(priv, params, &cparam->rx_cq);
+ mlx5e_build_tx_cq_param(priv, params, &cparam->tx_cq);
+ mlx5e_build_ico_cq_param(priv, icosq_log_wq_sz, &cparam->icosq_cq);
}
-static int mlx5e_open_channels(struct mlx5e_priv *priv)
+int mlx5e_open_channels(struct mlx5e_priv *priv,
+ struct mlx5e_channels *chs)
{
struct mlx5e_channel_param *cparam;
- int nch = priv->params.num_channels;
int err = -ENOMEM;
int i;
- int j;
- priv->channel = kcalloc(nch, sizeof(struct mlx5e_channel *),
- GFP_KERNEL);
+ chs->num = chs->params.num_channels;
- priv->txq_to_sq_map = kcalloc(nch * priv->params.num_tc,
- sizeof(struct mlx5e_sq *), GFP_KERNEL);
-
+ chs->c = kcalloc(chs->num, sizeof(struct mlx5e_channel *), GFP_KERNEL);
cparam = kzalloc(sizeof(struct mlx5e_channel_param), GFP_KERNEL);
+ if (!chs->c || !cparam)
+ goto err_free;
- if (!priv->channel || !priv->txq_to_sq_map || !cparam)
- goto err_free_txq_to_sq_map;
-
- mlx5e_build_channel_param(priv, cparam);
-
- for (i = 0; i < nch; i++) {
- err = mlx5e_open_channel(priv, i, cparam, &priv->channel[i]);
+ mlx5e_build_channel_param(priv, &chs->params, cparam);
+ for (i = 0; i < chs->num; i++) {
+ err = mlx5e_open_channel(priv, i, &chs->params, cparam, &chs->c[i]);
if (err)
goto err_close_channels;
}
- for (j = 0; j < nch; j++) {
- err = mlx5e_wait_for_min_rx_wqes(&priv->channel[j]->rq);
- if (err)
- goto err_close_channels;
- }
-
- /* FIXME: This is a W/A for tx timeout watch dog false alarm when
- * polling for inactive tx queues.
- */
- netif_tx_start_all_queues(priv->netdev);
-
kfree(cparam);
return 0;
err_close_channels:
for (i--; i >= 0; i--)
- mlx5e_close_channel(priv->channel[i]);
+ mlx5e_close_channel(chs->c[i]);
-err_free_txq_to_sq_map:
- kfree(priv->txq_to_sq_map);
- kfree(priv->channel);
+err_free:
+ kfree(chs->c);
kfree(cparam);
+ chs->num = 0;
+ return err;
+}
+
+static void mlx5e_activate_channels(struct mlx5e_channels *chs)
+{
+ int i;
+
+ for (i = 0; i < chs->num; i++)
+ mlx5e_activate_channel(chs->c[i]);
+}
+
+static int mlx5e_wait_channels_min_rx_wqes(struct mlx5e_channels *chs)
+{
+ int err = 0;
+ int i;
+
+ for (i = 0; i < chs->num; i++) {
+ err = mlx5e_wait_for_min_rx_wqes(&chs->c[i]->rq);
+ if (err)
+ break;
+ }
return err;
}
-static void mlx5e_close_channels(struct mlx5e_priv *priv)
+static void mlx5e_deactivate_channels(struct mlx5e_channels *chs)
{
int i;
- /* FIXME: This is a W/A only for tx timeout watch dog false alarm when
- * polling for inactive tx queues.
- */
- netif_tx_stop_all_queues(priv->netdev);
- netif_tx_disable(priv->netdev);
+ for (i = 0; i < chs->num; i++)
+ mlx5e_deactivate_channel(chs->c[i]);
+}
- for (i = 0; i < priv->params.num_channels; i++)
- mlx5e_close_channel(priv->channel[i]);
+void mlx5e_close_channels(struct mlx5e_channels *chs)
+{
+ int i;
- kfree(priv->txq_to_sq_map);
- kfree(priv->channel);
+ for (i = 0; i < chs->num; i++)
+ mlx5e_close_channel(chs->c[i]);
+
+ kfree(chs->c);
+ chs->num = 0;
+}
+
+static int
+mlx5e_create_rqt(struct mlx5e_priv *priv, int sz, struct mlx5e_rqt *rqt)
+{
+ struct mlx5_core_dev *mdev = priv->mdev;
+ void *rqtc;
+ int inlen;
+ int err;
+ u32 *in;
+ int i;
+
+ inlen = MLX5_ST_SZ_BYTES(create_rqt_in) + sizeof(u32) * sz;
+ in = mlx5_vzalloc(inlen);
+ if (!in)
+ return -ENOMEM;
+
+ rqtc = MLX5_ADDR_OF(create_rqt_in, in, rqt_context);
+
+ MLX5_SET(rqtc, rqtc, rqt_actual_size, sz);
+ MLX5_SET(rqtc, rqtc, rqt_max_size, sz);
+
+ for (i = 0; i < sz; i++)
+ MLX5_SET(rqtc, rqtc, rq_num[i], priv->drop_rq.rqn);
+
+ err = mlx5_core_create_rqt(mdev, in, inlen, &rqt->rqtn);
+ if (!err)
+ rqt->enabled = true;
+
+ kvfree(in);
+ return err;
+}
+
+void mlx5e_destroy_rqt(struct mlx5e_priv *priv, struct mlx5e_rqt *rqt)
+{
+ rqt->enabled = false;
+ mlx5_core_destroy_rqt(priv->mdev, rqt->rqtn);
+}
+
+int mlx5e_create_indirect_rqt(struct mlx5e_priv *priv)
+{
+ struct mlx5e_rqt *rqt = &priv->indir_rqt;
+ int err;
+
+ err = mlx5e_create_rqt(priv, MLX5E_INDIR_RQT_SIZE, rqt);
+ if (err)
+ mlx5_core_warn(priv->mdev, "create indirect rqts failed, %d\n", err);
+ return err;
+}
+
+int mlx5e_create_direct_rqts(struct mlx5e_priv *priv)
+{
+ struct mlx5e_rqt *rqt;
+ int err;
+ int ix;
+
+ for (ix = 0; ix < priv->profile->max_nch(priv->mdev); ix++) {
+ rqt = &priv->direct_tir[ix].rqt;
+ err = mlx5e_create_rqt(priv, 1 /*size */, rqt);
+ if (err)
+ goto err_destroy_rqts;
+ }
+
+ return 0;
+
+err_destroy_rqts:
+ mlx5_core_warn(priv->mdev, "create direct rqts failed, %d\n", err);
+ for (ix--; ix >= 0; ix--)
+ mlx5e_destroy_rqt(priv, &priv->direct_tir[ix].rqt);
+
+ return err;
+}
+
+void mlx5e_destroy_direct_rqts(struct mlx5e_priv *priv)
+{
+ int i;
+
+ for (i = 0; i < priv->profile->max_nch(priv->mdev); i++)
+ mlx5e_destroy_rqt(priv, &priv->direct_tir[i].rqt);
}
static int mlx5e_rx_hash_fn(int hfunc)
@@ -1917,103 +2177,31 @@
return inv;
}
-static void mlx5e_fill_indir_rqt_rqns(struct mlx5e_priv *priv, void *rqtc)
+static void mlx5e_fill_rqt_rqns(struct mlx5e_priv *priv, int sz,
+ struct mlx5e_redirect_rqt_param rrp, void *rqtc)
{
int i;
- for (i = 0; i < MLX5E_INDIR_RQT_SIZE; i++) {
- int ix = i;
+ for (i = 0; i < sz; i++) {
u32 rqn;
- if (priv->params.rss_hfunc == ETH_RSS_HASH_XOR)
- ix = mlx5e_bits_invert(i, MLX5E_LOG_INDIR_RQT_SIZE);
+ if (rrp.is_rss) {
+ int ix = i;
- ix = priv->params.indirection_rqt[ix];
- rqn = test_bit(MLX5E_STATE_OPENED, &priv->state) ?
- priv->channel[ix]->rq.rqn :
- priv->drop_rq.rqn;
+ if (rrp.rss.hfunc == ETH_RSS_HASH_XOR)
+ ix = mlx5e_bits_invert(i, ilog2(sz));
+
+ ix = priv->channels.params.indirection_rqt[ix];
+ rqn = rrp.rss.channels->c[ix]->rq.rqn;
+ } else {
+ rqn = rrp.rqn;
+ }
MLX5_SET(rqtc, rqtc, rq_num[i], rqn);
}
}
-static void mlx5e_fill_direct_rqt_rqn(struct mlx5e_priv *priv, void *rqtc,
- int ix)
-{
- u32 rqn = test_bit(MLX5E_STATE_OPENED, &priv->state) ?
- priv->channel[ix]->rq.rqn :
- priv->drop_rq.rqn;
-
- MLX5_SET(rqtc, rqtc, rq_num[0], rqn);
-}
-
-static int mlx5e_create_rqt(struct mlx5e_priv *priv, int sz,
- int ix, struct mlx5e_rqt *rqt)
-{
- struct mlx5_core_dev *mdev = priv->mdev;
- void *rqtc;
- int inlen;
- int err;
- u32 *in;
-
- inlen = MLX5_ST_SZ_BYTES(create_rqt_in) + sizeof(u32) * sz;
- in = mlx5_vzalloc(inlen);
- if (!in)
- return -ENOMEM;
-
- rqtc = MLX5_ADDR_OF(create_rqt_in, in, rqt_context);
-
- MLX5_SET(rqtc, rqtc, rqt_actual_size, sz);
- MLX5_SET(rqtc, rqtc, rqt_max_size, sz);
-
- if (sz > 1) /* RSS */
- mlx5e_fill_indir_rqt_rqns(priv, rqtc);
- else
- mlx5e_fill_direct_rqt_rqn(priv, rqtc, ix);
-
- err = mlx5_core_create_rqt(mdev, in, inlen, &rqt->rqtn);
- if (!err)
- rqt->enabled = true;
-
- kvfree(in);
- return err;
-}
-
-void mlx5e_destroy_rqt(struct mlx5e_priv *priv, struct mlx5e_rqt *rqt)
-{
- rqt->enabled = false;
- mlx5_core_destroy_rqt(priv->mdev, rqt->rqtn);
-}
-
-static int mlx5e_create_indirect_rqts(struct mlx5e_priv *priv)
-{
- struct mlx5e_rqt *rqt = &priv->indir_rqt;
-
- return mlx5e_create_rqt(priv, MLX5E_INDIR_RQT_SIZE, 0, rqt);
-}
-
-int mlx5e_create_direct_rqts(struct mlx5e_priv *priv)
-{
- struct mlx5e_rqt *rqt;
- int err;
- int ix;
-
- for (ix = 0; ix < priv->profile->max_nch(priv->mdev); ix++) {
- rqt = &priv->direct_tir[ix].rqt;
- err = mlx5e_create_rqt(priv, 1 /*size */, ix, rqt);
- if (err)
- goto err_destroy_rqts;
- }
-
- return 0;
-
-err_destroy_rqts:
- for (ix--; ix >= 0; ix--)
- mlx5e_destroy_rqt(priv, &priv->direct_tir[ix].rqt);
-
- return err;
-}
-
-int mlx5e_redirect_rqt(struct mlx5e_priv *priv, u32 rqtn, int sz, int ix)
+int mlx5e_redirect_rqt(struct mlx5e_priv *priv, u32 rqtn, int sz,
+ struct mlx5e_redirect_rqt_param rrp)
{
struct mlx5_core_dev *mdev = priv->mdev;
void *rqtc;
@@ -2029,41 +2217,86 @@
rqtc = MLX5_ADDR_OF(modify_rqt_in, in, ctx);
MLX5_SET(rqtc, rqtc, rqt_actual_size, sz);
- if (sz > 1) /* RSS */
- mlx5e_fill_indir_rqt_rqns(priv, rqtc);
- else
- mlx5e_fill_direct_rqt_rqn(priv, rqtc, ix);
-
MLX5_SET(modify_rqt_in, in, bitmask.rqn_list, 1);
-
+ mlx5e_fill_rqt_rqns(priv, sz, rrp, rqtc);
err = mlx5_core_modify_rqt(mdev, rqtn, in, inlen);
kvfree(in);
-
return err;
}
-static void mlx5e_redirect_rqts(struct mlx5e_priv *priv)
+static u32 mlx5e_get_direct_rqn(struct mlx5e_priv *priv, int ix,
+ struct mlx5e_redirect_rqt_param rrp)
+{
+ if (!rrp.is_rss)
+ return rrp.rqn;
+
+ if (ix >= rrp.rss.channels->num)
+ return priv->drop_rq.rqn;
+
+ return rrp.rss.channels->c[ix]->rq.rqn;
+}
+
+static void mlx5e_redirect_rqts(struct mlx5e_priv *priv,
+ struct mlx5e_redirect_rqt_param rrp)
{
u32 rqtn;
int ix;
if (priv->indir_rqt.enabled) {
+ /* RSS RQ table */
rqtn = priv->indir_rqt.rqtn;
- mlx5e_redirect_rqt(priv, rqtn, MLX5E_INDIR_RQT_SIZE, 0);
+ mlx5e_redirect_rqt(priv, rqtn, MLX5E_INDIR_RQT_SIZE, rrp);
}
- for (ix = 0; ix < priv->params.num_channels; ix++) {
+ for (ix = 0; ix < priv->profile->max_nch(priv->mdev); ix++) {
+ struct mlx5e_redirect_rqt_param direct_rrp = {
+ .is_rss = false,
+ {
+ .rqn = mlx5e_get_direct_rqn(priv, ix, rrp)
+ },
+ };
+
+ /* Direct RQ Tables */
if (!priv->direct_tir[ix].rqt.enabled)
continue;
+
rqtn = priv->direct_tir[ix].rqt.rqtn;
- mlx5e_redirect_rqt(priv, rqtn, 1, ix);
+ mlx5e_redirect_rqt(priv, rqtn, 1, direct_rrp);
}
}
-static void mlx5e_build_tir_ctx_lro(void *tirc, struct mlx5e_priv *priv)
+static void mlx5e_redirect_rqts_to_channels(struct mlx5e_priv *priv,
+ struct mlx5e_channels *chs)
{
- if (!priv->params.lro_en)
+ struct mlx5e_redirect_rqt_param rrp = {
+ .is_rss = true,
+ {
+ .rss = {
+ .channels = chs,
+ .hfunc = chs->params.rss_hfunc,
+ }
+ },
+ };
+
+ mlx5e_redirect_rqts(priv, rrp);
+}
+
+static void mlx5e_redirect_rqts_to_drop(struct mlx5e_priv *priv)
+{
+ struct mlx5e_redirect_rqt_param drop_rrp = {
+ .is_rss = false,
+ {
+ .rqn = priv->drop_rq.rqn,
+ },
+ };
+
+ mlx5e_redirect_rqts(priv, drop_rrp);
+}
+
+static void mlx5e_build_tir_ctx_lro(struct mlx5e_params *params, void *tirc)
+{
+ if (!params->lro_en)
return;
#define ROUGH_MAX_L2_L3_HDR_SZ 256
@@ -2072,13 +2305,13 @@
MLX5_TIRC_LRO_ENABLE_MASK_IPV4_LRO |
MLX5_TIRC_LRO_ENABLE_MASK_IPV6_LRO);
MLX5_SET(tirc, tirc, lro_max_ip_payload_size,
- (priv->params.lro_wqe_sz -
- ROUGH_MAX_L2_L3_HDR_SZ) >> 8);
- MLX5_SET(tirc, tirc, lro_timeout_period_usecs, priv->params.lro_timeout);
+ (params->lro_wqe_sz - ROUGH_MAX_L2_L3_HDR_SZ) >> 8);
+ MLX5_SET(tirc, tirc, lro_timeout_period_usecs, params->lro_timeout);
}
-void mlx5e_build_indir_tir_ctx_hash(struct mlx5e_priv *priv, void *tirc,
- enum mlx5e_traffic_types tt)
+void mlx5e_build_indir_tir_ctx_hash(struct mlx5e_params *params,
+ enum mlx5e_traffic_types tt,
+ void *tirc)
{
void *hfso = MLX5_ADDR_OF(tirc, tirc, rx_hash_field_selector_outer);
@@ -2094,16 +2327,15 @@
MLX5_HASH_FIELD_SEL_DST_IP |\
MLX5_HASH_FIELD_SEL_IPSEC_SPI)
- MLX5_SET(tirc, tirc, rx_hash_fn,
- mlx5e_rx_hash_fn(priv->params.rss_hfunc));
- if (priv->params.rss_hfunc == ETH_RSS_HASH_TOP) {
+ MLX5_SET(tirc, tirc, rx_hash_fn, mlx5e_rx_hash_fn(params->rss_hfunc));
+ if (params->rss_hfunc == ETH_RSS_HASH_TOP) {
void *rss_key = MLX5_ADDR_OF(tirc, tirc,
rx_hash_toeplitz_key);
size_t len = MLX5_FLD_SZ_BYTES(tirc,
rx_hash_toeplitz_key);
MLX5_SET(tirc, tirc, rx_hash_symmetric, 1);
- memcpy(rss_key, priv->params.toeplitz_hash_key, len);
+ memcpy(rss_key, params->toeplitz_hash_key, len);
}
switch (tt) {
@@ -2208,7 +2440,7 @@
MLX5_SET(modify_tir_in, in, bitmask.lro, 1);
tirc = MLX5_ADDR_OF(modify_tir_in, in, ctx);
- mlx5e_build_tir_ctx_lro(tirc, priv);
+ mlx5e_build_tir_ctx_lro(&priv->channels.params, tirc);
for (tt = 0; tt < MLX5E_NUM_INDIR_TIRS; tt++) {
err = mlx5_core_modify_tir(mdev, priv->indir_tir[tt].tirn, in,
@@ -2258,9 +2490,9 @@
*mtu = MLX5E_HW2SW_MTU(hw_mtu);
}
-static int mlx5e_set_dev_port_mtu(struct net_device *netdev)
+static int mlx5e_set_dev_port_mtu(struct mlx5e_priv *priv)
{
- struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct net_device *netdev = priv->netdev;
u16 mtu;
int err;
@@ -2280,8 +2512,8 @@
static void mlx5e_netdev_set_tcs(struct net_device *netdev)
{
struct mlx5e_priv *priv = netdev_priv(netdev);
- int nch = priv->params.num_channels;
- int ntc = priv->params.num_tc;
+ int nch = priv->channels.params.num_channels;
+ int ntc = priv->channels.params.num_tc;
int tc;
netdev_reset_tc(netdev);
@@ -2298,53 +2530,116 @@
netdev_set_tc_queue(netdev, tc, nch, 0);
}
+static void mlx5e_build_channels_tx_maps(struct mlx5e_priv *priv)
+{
+ struct mlx5e_channel *c;
+ struct mlx5e_txqsq *sq;
+ int i, tc;
+
+ for (i = 0; i < priv->channels.num; i++)
+ for (tc = 0; tc < priv->profile->max_tc; tc++)
+ priv->channel_tc2txq[i][tc] = i + tc * priv->channels.num;
+
+ for (i = 0; i < priv->channels.num; i++) {
+ c = priv->channels.c[i];
+ for (tc = 0; tc < c->num_tc; tc++) {
+ sq = &c->sq[tc];
+ priv->txq2sq[sq->txq_ix] = sq;
+ }
+ }
+}
+
+static bool mlx5e_is_eswitch_vport_mngr(struct mlx5_core_dev *mdev)
+{
+ return (MLX5_CAP_GEN(mdev, vport_group_manager) &&
+ MLX5_CAP_GEN(mdev, port_type) == MLX5_CAP_PORT_TYPE_ETH);
+}
+
+void mlx5e_activate_priv_channels(struct mlx5e_priv *priv)
+{
+ int num_txqs = priv->channels.num * priv->channels.params.num_tc;
+ struct net_device *netdev = priv->netdev;
+
+ mlx5e_netdev_set_tcs(netdev);
+ netif_set_real_num_tx_queues(netdev, num_txqs);
+ netif_set_real_num_rx_queues(netdev, priv->channels.num);
+
+ mlx5e_build_channels_tx_maps(priv);
+ mlx5e_activate_channels(&priv->channels);
+ netif_tx_start_all_queues(priv->netdev);
+
+ if (mlx5e_is_eswitch_vport_mngr(priv->mdev))
+ mlx5e_add_sqs_fwd_rules(priv);
+
+ mlx5e_wait_channels_min_rx_wqes(&priv->channels);
+ mlx5e_redirect_rqts_to_channels(priv, &priv->channels);
+}
+
+void mlx5e_deactivate_priv_channels(struct mlx5e_priv *priv)
+{
+ mlx5e_redirect_rqts_to_drop(priv);
+
+ if (mlx5e_is_eswitch_vport_mngr(priv->mdev))
+ mlx5e_remove_sqs_fwd_rules(priv);
+
+ /* FIXME: This is a W/A only for tx timeout watch dog false alarm when
+ * polling for inactive tx queues.
+ */
+ netif_tx_stop_all_queues(priv->netdev);
+ netif_tx_disable(priv->netdev);
+ mlx5e_deactivate_channels(&priv->channels);
+}
+
+void mlx5e_switch_priv_channels(struct mlx5e_priv *priv,
+ struct mlx5e_channels *new_chs,
+ mlx5e_fp_hw_modify hw_modify)
+{
+ struct net_device *netdev = priv->netdev;
+ int new_num_txqs;
+
+ new_num_txqs = new_chs->num * new_chs->params.num_tc;
+
+ netif_carrier_off(netdev);
+
+ if (new_num_txqs < netdev->real_num_tx_queues)
+ netif_set_real_num_tx_queues(netdev, new_num_txqs);
+
+ mlx5e_deactivate_priv_channels(priv);
+ mlx5e_close_channels(&priv->channels);
+
+ priv->channels = *new_chs;
+
+ /* New channels are ready to roll, modify HW settings if needed */
+ if (hw_modify)
+ hw_modify(priv);
+
+ mlx5e_refresh_tirs(priv, false);
+ mlx5e_activate_priv_channels(priv);
+
+ mlx5e_update_carrier(priv);
+}
+
int mlx5e_open_locked(struct net_device *netdev)
{
struct mlx5e_priv *priv = netdev_priv(netdev);
- struct mlx5_core_dev *mdev = priv->mdev;
- int num_txqs;
int err;
set_bit(MLX5E_STATE_OPENED, &priv->state);
- mlx5e_netdev_set_tcs(netdev);
-
- num_txqs = priv->params.num_channels * priv->params.num_tc;
- netif_set_real_num_tx_queues(netdev, num_txqs);
- netif_set_real_num_rx_queues(netdev, priv->params.num_channels);
-
- err = mlx5e_open_channels(priv);
- if (err) {
- netdev_err(netdev, "%s: mlx5e_open_channels failed, %d\n",
- __func__, err);
+ err = mlx5e_open_channels(priv, &priv->channels);
+ if (err)
goto err_clear_state_opened_flag;
- }
- err = mlx5e_refresh_tirs_self_loopback(priv->mdev, false);
- if (err) {
- netdev_err(netdev, "%s: mlx5e_refresh_tirs_self_loopback_enable failed, %d\n",
- __func__, err);
- goto err_close_channels;
- }
-
- mlx5e_redirect_rqts(priv);
+ mlx5e_refresh_tirs(priv, false);
+ mlx5e_activate_priv_channels(priv);
mlx5e_update_carrier(priv);
mlx5e_timestamp_init(priv);
-#ifdef CONFIG_RFS_ACCEL
- priv->netdev->rx_cpu_rmap = priv->mdev->rmap;
-#endif
+
if (priv->profile->update_stats)
queue_delayed_work(priv->wq, &priv->update_stats_work, 0);
- if (MLX5_CAP_GEN(mdev, vport_group_manager)) {
- err = mlx5e_add_sqs_fwd_rules(priv);
- if (err)
- goto err_close_channels;
- }
return 0;
-err_close_channels:
- mlx5e_close_channels(priv);
err_clear_state_opened_flag:
clear_bit(MLX5E_STATE_OPENED, &priv->state);
return err;
@@ -2365,7 +2660,6 @@
int mlx5e_close_locked(struct net_device *netdev)
{
struct mlx5e_priv *priv = netdev_priv(netdev);
- struct mlx5_core_dev *mdev = priv->mdev;
/* May already be CLOSED in case a previous configuration operation
* (e.g RX/TX queue size change) that involves close&open failed.
@@ -2375,13 +2669,10 @@
clear_bit(MLX5E_STATE_OPENED, &priv->state);
- if (MLX5_CAP_GEN(mdev, vport_group_manager))
- mlx5e_remove_sqs_fwd_rules(priv);
-
mlx5e_timestamp_cleanup(priv);
netif_carrier_off(priv->netdev);
- mlx5e_redirect_rqts(priv);
- mlx5e_close_channels(priv);
+ mlx5e_deactivate_priv_channels(priv);
+ mlx5e_close_channels(&priv->channels);
return 0;
}
@@ -2401,11 +2692,10 @@
return err;
}
-static int mlx5e_create_drop_rq(struct mlx5e_priv *priv,
- struct mlx5e_rq *rq,
- struct mlx5e_rq_param *param)
+static int mlx5e_alloc_drop_rq(struct mlx5_core_dev *mdev,
+ struct mlx5e_rq *rq,
+ struct mlx5e_rq_param *param)
{
- struct mlx5_core_dev *mdev = priv->mdev;
void *rqc = param->rqc;
void *rqc_wq = MLX5_ADDR_OF(rqc, rqc, wq);
int err;
@@ -2417,111 +2707,85 @@
if (err)
return err;
- rq->priv = priv;
+ rq->mdev = mdev;
return 0;
}
-static int mlx5e_create_drop_cq(struct mlx5e_priv *priv,
- struct mlx5e_cq *cq,
- struct mlx5e_cq_param *param)
+static int mlx5e_alloc_drop_cq(struct mlx5_core_dev *mdev,
+ struct mlx5e_cq *cq,
+ struct mlx5e_cq_param *param)
{
- struct mlx5_core_dev *mdev = priv->mdev;
- struct mlx5_core_cq *mcq = &cq->mcq;
- int eqn_not_used;
- unsigned int irqn;
- int err;
-
- err = mlx5_cqwq_create(mdev, ¶m->wq, param->cqc, &cq->wq,
- &cq->wq_ctrl);
- if (err)
- return err;
-
- mlx5_vector2eqn(mdev, param->eq_ix, &eqn_not_used, &irqn);
-
- mcq->cqe_sz = 64;
- mcq->set_ci_db = cq->wq_ctrl.db.db;
- mcq->arm_db = cq->wq_ctrl.db.db + 1;
- *mcq->set_ci_db = 0;
- *mcq->arm_db = 0;
- mcq->vector = param->eq_ix;
- mcq->comp = mlx5e_completion_event;
- mcq->event = mlx5e_cq_error_event;
- mcq->irqn = irqn;
-
- cq->priv = priv;
-
- return 0;
+ return mlx5e_alloc_cq_common(mdev, param, cq);
}
-static int mlx5e_open_drop_rq(struct mlx5e_priv *priv)
+static int mlx5e_open_drop_rq(struct mlx5_core_dev *mdev,
+ struct mlx5e_rq *drop_rq)
{
- struct mlx5e_cq_param cq_param;
- struct mlx5e_rq_param rq_param;
- struct mlx5e_rq *rq = &priv->drop_rq;
- struct mlx5e_cq *cq = &priv->drop_rq.cq;
+ struct mlx5e_cq_param cq_param = {};
+ struct mlx5e_rq_param rq_param = {};
+ struct mlx5e_cq *cq = &drop_rq->cq;
int err;
- memset(&cq_param, 0, sizeof(cq_param));
- memset(&rq_param, 0, sizeof(rq_param));
mlx5e_build_drop_rq_param(&rq_param);
- err = mlx5e_create_drop_cq(priv, cq, &cq_param);
+ err = mlx5e_alloc_drop_cq(mdev, cq, &cq_param);
if (err)
return err;
- err = mlx5e_enable_cq(cq, &cq_param);
+ err = mlx5e_create_cq(cq, &cq_param);
+ if (err)
+ goto err_free_cq;
+
+ err = mlx5e_alloc_drop_rq(mdev, drop_rq, &rq_param);
if (err)
goto err_destroy_cq;
- err = mlx5e_create_drop_rq(priv, rq, &rq_param);
+ err = mlx5e_create_rq(drop_rq, &rq_param);
if (err)
- goto err_disable_cq;
-
- err = mlx5e_enable_rq(rq, &rq_param);
- if (err)
- goto err_destroy_rq;
+ goto err_free_rq;
return 0;
-err_destroy_rq:
- mlx5e_destroy_rq(&priv->drop_rq);
-
-err_disable_cq:
- mlx5e_disable_cq(&priv->drop_rq.cq);
+err_free_rq:
+ mlx5e_free_rq(drop_rq);
err_destroy_cq:
- mlx5e_destroy_cq(&priv->drop_rq.cq);
+ mlx5e_destroy_cq(cq);
+
+err_free_cq:
+ mlx5e_free_cq(cq);
return err;
}
-static void mlx5e_close_drop_rq(struct mlx5e_priv *priv)
+static void mlx5e_close_drop_rq(struct mlx5e_rq *drop_rq)
{
- mlx5e_disable_rq(&priv->drop_rq);
- mlx5e_destroy_rq(&priv->drop_rq);
- mlx5e_disable_cq(&priv->drop_rq.cq);
- mlx5e_destroy_cq(&priv->drop_rq.cq);
+ mlx5e_destroy_rq(drop_rq);
+ mlx5e_free_rq(drop_rq);
+ mlx5e_destroy_cq(&drop_rq->cq);
+ mlx5e_free_cq(&drop_rq->cq);
}
-static int mlx5e_create_tis(struct mlx5e_priv *priv, int tc)
+int mlx5e_create_tis(struct mlx5_core_dev *mdev, int tc,
+ u32 underlay_qpn, u32 *tisn)
{
- struct mlx5_core_dev *mdev = priv->mdev;
u32 in[MLX5_ST_SZ_DW(create_tis_in)] = {0};
void *tisc = MLX5_ADDR_OF(create_tis_in, in, ctx);
MLX5_SET(tisc, tisc, prio, tc << 1);
+ MLX5_SET(tisc, tisc, underlay_qpn, underlay_qpn);
MLX5_SET(tisc, tisc, transport_domain, mdev->mlx5e_res.td.tdn);
if (mlx5_lag_is_lacp_owner(mdev))
MLX5_SET(tisc, tisc, strict_lag_tx_port_affinity, 1);
- return mlx5_core_create_tis(mdev, in, sizeof(in), &priv->tisn[tc]);
+ return mlx5_core_create_tis(mdev, in, sizeof(in), tisn);
}
-static void mlx5e_destroy_tis(struct mlx5e_priv *priv, int tc)
+void mlx5e_destroy_tis(struct mlx5_core_dev *mdev, u32 tisn)
{
- mlx5_core_destroy_tis(priv->mdev, priv->tisn[tc]);
+ mlx5_core_destroy_tis(mdev, tisn);
}
int mlx5e_create_tises(struct mlx5e_priv *priv)
@@ -2530,7 +2794,7 @@
int tc;
for (tc = 0; tc < priv->profile->max_tc; tc++) {
- err = mlx5e_create_tis(priv, tc);
+ err = mlx5e_create_tis(priv->mdev, tc, 0, &priv->tisn[tc]);
if (err)
goto err_close_tises;
}
@@ -2539,7 +2803,7 @@
err_close_tises:
for (tc--; tc >= 0; tc--)
- mlx5e_destroy_tis(priv, tc);
+ mlx5e_destroy_tis(priv->mdev, priv->tisn[tc]);
return err;
}
@@ -2549,34 +2813,34 @@
int tc;
for (tc = 0; tc < priv->profile->max_tc; tc++)
- mlx5e_destroy_tis(priv, tc);
+ mlx5e_destroy_tis(priv->mdev, priv->tisn[tc]);
}
-static void mlx5e_build_indir_tir_ctx(struct mlx5e_priv *priv, u32 *tirc,
- enum mlx5e_traffic_types tt)
+static void mlx5e_build_indir_tir_ctx(struct mlx5e_priv *priv,
+ enum mlx5e_traffic_types tt,
+ u32 *tirc)
{
MLX5_SET(tirc, tirc, transport_domain, priv->mdev->mlx5e_res.td.tdn);
- mlx5e_build_tir_ctx_lro(tirc, priv);
+ mlx5e_build_tir_ctx_lro(&priv->channels.params, tirc);
MLX5_SET(tirc, tirc, disp_type, MLX5_TIRC_DISP_TYPE_INDIRECT);
MLX5_SET(tirc, tirc, indirect_table, priv->indir_rqt.rqtn);
- mlx5e_build_indir_tir_ctx_hash(priv, tirc, tt);
+ mlx5e_build_indir_tir_ctx_hash(&priv->channels.params, tt, tirc);
}
-static void mlx5e_build_direct_tir_ctx(struct mlx5e_priv *priv, u32 *tirc,
- u32 rqtn)
+static void mlx5e_build_direct_tir_ctx(struct mlx5e_priv *priv, u32 rqtn, u32 *tirc)
{
MLX5_SET(tirc, tirc, transport_domain, priv->mdev->mlx5e_res.td.tdn);
- mlx5e_build_tir_ctx_lro(tirc, priv);
+ mlx5e_build_tir_ctx_lro(&priv->channels.params, tirc);
MLX5_SET(tirc, tirc, disp_type, MLX5_TIRC_DISP_TYPE_INDIRECT);
MLX5_SET(tirc, tirc, indirect_table, rqtn);
MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FN_INVERTED_XOR8);
}
-static int mlx5e_create_indirect_tirs(struct mlx5e_priv *priv)
+int mlx5e_create_indirect_tirs(struct mlx5e_priv *priv)
{
struct mlx5e_tir *tir;
void *tirc;
@@ -2594,7 +2858,7 @@
memset(in, 0, inlen);
tir = &priv->indir_tir[tt];
tirc = MLX5_ADDR_OF(create_tir_in, in, ctx);
- mlx5e_build_indir_tir_ctx(priv, tirc, tt);
+ mlx5e_build_indir_tir_ctx(priv, tt, tirc);
err = mlx5e_create_tir(priv->mdev, tir, in, inlen);
if (err)
goto err_destroy_tirs;
@@ -2605,6 +2869,7 @@
return 0;
err_destroy_tirs:
+ mlx5_core_warn(priv->mdev, "create indirect tirs failed, %d\n", err);
for (tt--; tt >= 0; tt--)
mlx5e_destroy_tir(priv->mdev, &priv->indir_tir[tt]);
@@ -2632,8 +2897,7 @@
memset(in, 0, inlen);
tir = &priv->direct_tir[ix];
tirc = MLX5_ADDR_OF(create_tir_in, in, ctx);
- mlx5e_build_direct_tir_ctx(priv, tirc,
- priv->direct_tir[ix].rqt.rqtn);
+ mlx5e_build_direct_tir_ctx(priv, priv->direct_tir[ix].rqt.rqtn, tirc);
err = mlx5e_create_tir(priv->mdev, tir, in, inlen);
if (err)
goto err_destroy_ch_tirs;
@@ -2644,6 +2908,7 @@
return 0;
err_destroy_ch_tirs:
+ mlx5_core_warn(priv->mdev, "create direct tirs failed, %d\n", err);
for (ix--; ix >= 0; ix--)
mlx5e_destroy_tir(priv->mdev, &priv->direct_tir[ix]);
@@ -2652,7 +2917,7 @@
return err;
}
-static void mlx5e_destroy_indirect_tirs(struct mlx5e_priv *priv)
+void mlx5e_destroy_indirect_tirs(struct mlx5e_priv *priv)
{
int i;
@@ -2669,16 +2934,27 @@
mlx5e_destroy_tir(priv->mdev, &priv->direct_tir[i]);
}
-int mlx5e_modify_rqs_vsd(struct mlx5e_priv *priv, bool vsd)
+static int mlx5e_modify_channels_scatter_fcs(struct mlx5e_channels *chs, bool enable)
{
int err = 0;
int i;
- if (!test_bit(MLX5E_STATE_OPENED, &priv->state))
- return 0;
+ for (i = 0; i < chs->num; i++) {
+ err = mlx5e_modify_rq_scatter_fcs(&chs->c[i]->rq, enable);
+ if (err)
+ return err;
+ }
- for (i = 0; i < priv->params.num_channels; i++) {
- err = mlx5e_modify_rq_vsd(&priv->channel[i]->rq, vsd);
+ return 0;
+}
+
+static int mlx5e_modify_channels_vsd(struct mlx5e_channels *chs, bool vsd)
+{
+ int err = 0;
+ int i;
+
+ for (i = 0; i < chs->num; i++) {
+ err = mlx5e_modify_rq_vsd(&chs->c[i]->rq, vsd);
if (err)
return err;
}
@@ -2689,7 +2965,7 @@
static int mlx5e_setup_tc(struct net_device *netdev, u8 tc)
{
struct mlx5e_priv *priv = netdev_priv(netdev);
- bool was_opened;
+ struct mlx5e_channels new_channels = {};
int err = 0;
if (tc && tc != MLX5E_MAX_NUM_TC)
@@ -2697,17 +2973,21 @@
mutex_lock(&priv->state_lock);
- was_opened = test_bit(MLX5E_STATE_OPENED, &priv->state);
- if (was_opened)
- mlx5e_close_locked(priv->netdev);
+ new_channels.params = priv->channels.params;
+ new_channels.params.num_tc = tc ? tc : 1;
- priv->params.num_tc = tc ? tc : 1;
+ if (test_bit(MLX5E_STATE_OPENED, &priv->state)) {
+ priv->channels.params = new_channels.params;
+ goto out;
+ }
- if (was_opened)
- err = mlx5e_open_locked(priv->netdev);
+ err = mlx5e_open_channels(priv, &new_channels);
+ if (err)
+ goto out;
+ mlx5e_switch_priv_channels(priv, &new_channels, NULL);
+out:
mutex_unlock(&priv->state_lock);
-
return err;
}
@@ -2737,7 +3017,9 @@
if (tc->type != TC_SETUP_MQPRIO)
return -EINVAL;
- return mlx5e_setup_tc(dev, tc->tc);
+ tc->mqprio->hw = TC_MQPRIO_HW_OFFLOAD_TCS;
+
+ return mlx5e_setup_tc(dev, tc->mqprio->num_tc);
}
static void
@@ -2822,26 +3104,31 @@
static int set_feature_lro(struct net_device *netdev, bool enable)
{
struct mlx5e_priv *priv = netdev_priv(netdev);
- bool was_opened = test_bit(MLX5E_STATE_OPENED, &priv->state);
- int err;
+ struct mlx5e_channels new_channels = {};
+ int err = 0;
+ bool reset;
mutex_lock(&priv->state_lock);
- if (was_opened && (priv->params.rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST))
- mlx5e_close_locked(priv->netdev);
+ reset = (priv->channels.params.rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST);
+ reset = reset && test_bit(MLX5E_STATE_OPENED, &priv->state);
- priv->params.lro_en = enable;
- err = mlx5e_modify_tirs_lro(priv);
- if (err) {
- netdev_err(netdev, "lro modify failed, %d\n", err);
- priv->params.lro_en = !enable;
+ new_channels.params = priv->channels.params;
+ new_channels.params.lro_en = enable;
+
+ if (!reset) {
+ priv->channels.params = new_channels.params;
+ err = mlx5e_modify_tirs_lro(priv);
+ goto out;
}
- if (was_opened && (priv->params.rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST))
- mlx5e_open_locked(priv->netdev);
+ err = mlx5e_open_channels(priv, &new_channels);
+ if (err)
+ goto out;
+ mlx5e_switch_priv_channels(priv, &new_channels, mlx5e_modify_tirs_lro);
+out:
mutex_unlock(&priv->state_lock);
-
return err;
}
@@ -2878,23 +3165,44 @@
return mlx5_set_port_fcs(mdev, !enable);
}
-static int set_feature_rx_vlan(struct net_device *netdev, bool enable)
+static int set_feature_rx_fcs(struct net_device *netdev, bool enable)
{
struct mlx5e_priv *priv = netdev_priv(netdev);
int err;
mutex_lock(&priv->state_lock);
- priv->params.vlan_strip_disable = !enable;
- err = mlx5e_modify_rqs_vsd(priv, !enable);
+ priv->channels.params.scatter_fcs_en = enable;
+ err = mlx5e_modify_channels_scatter_fcs(&priv->channels, enable);
if (err)
- priv->params.vlan_strip_disable = enable;
+ priv->channels.params.scatter_fcs_en = !enable;
mutex_unlock(&priv->state_lock);
return err;
}
+static int set_feature_rx_vlan(struct net_device *netdev, bool enable)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ int err = 0;
+
+ mutex_lock(&priv->state_lock);
+
+ priv->channels.params.vlan_strip_disable = !enable;
+ if (!test_bit(MLX5E_STATE_OPENED, &priv->state))
+ goto unlock;
+
+ err = mlx5e_modify_channels_vsd(&priv->channels, !enable);
+ if (err)
+ priv->channels.params.vlan_strip_disable = enable;
+
+unlock:
+ mutex_unlock(&priv->state_lock);
+
+ return err;
+}
+
#ifdef CONFIG_RFS_ACCEL
static int set_feature_arfs(struct net_device *netdev, bool enable)
{
@@ -2947,6 +3255,8 @@
set_feature_tc_num_filters);
err |= mlx5e_handle_feature(netdev, features, NETIF_F_RXALL,
set_feature_rx_all);
+ err |= mlx5e_handle_feature(netdev, features, NETIF_F_RXFCS,
+ set_feature_rx_fcs);
err |= mlx5e_handle_feature(netdev, features, NETIF_F_HW_VLAN_CTAG_RX,
set_feature_rx_vlan);
#ifdef CONFIG_RFS_ACCEL
@@ -2960,28 +3270,38 @@
static int mlx5e_change_mtu(struct net_device *netdev, int new_mtu)
{
struct mlx5e_priv *priv = netdev_priv(netdev);
- bool was_opened;
+ struct mlx5e_channels new_channels = {};
+ int curr_mtu;
int err = 0;
bool reset;
mutex_lock(&priv->state_lock);
- reset = !priv->params.lro_en &&
- (priv->params.rq_wq_type !=
+ reset = !priv->channels.params.lro_en &&
+ (priv->channels.params.rq_wq_type !=
MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ);
- was_opened = test_bit(MLX5E_STATE_OPENED, &priv->state);
- if (was_opened && reset)
- mlx5e_close_locked(netdev);
+ reset = reset && test_bit(MLX5E_STATE_OPENED, &priv->state);
+ curr_mtu = netdev->mtu;
netdev->mtu = new_mtu;
- mlx5e_set_dev_port_mtu(netdev);
- if (was_opened && reset)
- err = mlx5e_open_locked(netdev);
+ if (!reset) {
+ mlx5e_set_dev_port_mtu(priv);
+ goto out;
+ }
+ new_channels.params = priv->channels.params;
+ err = mlx5e_open_channels(priv, &new_channels);
+ if (err) {
+ netdev->mtu = curr_mtu;
+ goto out;
+ }
+
+ mlx5e_switch_priv_channels(priv, &new_channels, mlx5e_set_dev_port_mtu);
+
+out:
mutex_unlock(&priv->state_lock);
-
return err;
}
@@ -3186,8 +3506,8 @@
netdev_err(dev, "TX timeout detected\n");
- for (i = 0; i < priv->params.num_channels * priv->params.num_tc; i++) {
- struct mlx5e_sq *sq = priv->txq_to_sq_map[i];
+ for (i = 0; i < priv->channels.num * priv->channels.params.num_tc; i++) {
+ struct mlx5e_txqsq *sq = priv->txq2sq[i];
if (!netif_xmit_stopped(netdev_get_tx_queue(dev, i)))
continue;
@@ -3219,7 +3539,7 @@
was_opened = test_bit(MLX5E_STATE_OPENED, &priv->state);
/* no need for full reset when exchanging programs */
- reset = (!priv->xdp_prog || !prog);
+ reset = (!priv->channels.params.xdp_prog || !prog);
if (was_opened && reset)
mlx5e_close_locked(netdev);
@@ -3227,7 +3547,7 @@
/* num_channels is invariant here, so we can take the
* batched reference right upfront.
*/
- prog = bpf_prog_add(prog, priv->params.num_channels);
+ prog = bpf_prog_add(prog, priv->channels.num);
if (IS_ERR(prog)) {
err = PTR_ERR(prog);
goto unlock;
@@ -3237,12 +3557,12 @@
/* exchange programs, extra prog reference we got from caller
* as long as we don't fail from this point onwards.
*/
- old_prog = xchg(&priv->xdp_prog, prog);
+ old_prog = xchg(&priv->channels.params.xdp_prog, prog);
if (old_prog)
bpf_prog_put(old_prog);
if (reset) /* change RQ type according to priv->xdp_prog */
- mlx5e_set_rq_priv_params(priv);
+ mlx5e_set_rq_params(priv->mdev, &priv->channels.params);
if (was_opened && reset)
mlx5e_open_locked(netdev);
@@ -3253,8 +3573,8 @@
/* exchanging programs w/o reset, we update ref counts on behalf
* of the channels RQs here.
*/
- for (i = 0; i < priv->params.num_channels; i++) {
- struct mlx5e_channel *c = priv->channel[i];
+ for (i = 0; i < priv->channels.num; i++) {
+ struct mlx5e_channel *c = priv->channels.c[i];
clear_bit(MLX5E_RQ_STATE_ENABLED, &c->rq.state);
napi_synchronize(&c->napi);
@@ -3280,7 +3600,7 @@
{
struct mlx5e_priv *priv = netdev_priv(dev);
- return !!priv->xdp_prog;
+ return !!priv->channels.params.xdp_prog;
}
static int mlx5e_xdp(struct net_device *dev, struct netdev_xdp *xdp)
@@ -3303,10 +3623,12 @@
static void mlx5e_netpoll(struct net_device *dev)
{
struct mlx5e_priv *priv = netdev_priv(dev);
+ struct mlx5e_channels *chs = &priv->channels;
+
int i;
- for (i = 0; i < priv->params.num_channels; i++)
- napi_schedule(&priv->channel[i]->napi);
+ for (i = 0; i < chs->num; i++)
+ napi_schedule(&chs->c[i]->napi);
}
#endif
@@ -3463,6 +3785,12 @@
(pci_bw < 40000) && (pci_bw < link_speed));
}
+static bool hw_lro_heuristic(u32 link_speed, u32 pci_bw)
+{
+ return !(link_speed && pci_bw &&
+ (pci_bw <= 16000) && (pci_bw < link_speed));
+}
+
void mlx5e_set_rx_cq_mode_params(struct mlx5e_params *params, u8 cq_period_mode)
{
params->rx_cq_period_mode = cq_period_mode;
@@ -3475,6 +3803,13 @@
if (cq_period_mode == MLX5_CQ_PERIOD_MODE_START_FROM_CQE)
params->rx_cq_moderation.usec =
MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_USEC_FROM_CQE;
+
+ if (params->rx_am_enabled)
+ params->rx_cq_moderation =
+ mlx5e_am_get_def_profile(params->rx_cq_period_mode);
+
+ MLX5E_SET_PFLAG(params, MLX5E_PFLAG_RX_CQE_BASED_MODER,
+ params->rx_cq_period_mode == MLX5_CQ_PERIOD_MODE_START_FROM_CQE);
}
u32 mlx5e_choose_lro_timeout(struct mlx5_core_dev *mdev, u32 wanted_timeout)
@@ -3489,75 +3824,81 @@
return MLX5_CAP_ETH(mdev, lro_timer_supported_periods[i]);
}
+void mlx5e_build_nic_params(struct mlx5_core_dev *mdev,
+ struct mlx5e_params *params,
+ u16 max_channels)
+{
+ u8 cq_period_mode = 0;
+ u32 link_speed = 0;
+ u32 pci_bw = 0;
+
+ params->num_channels = max_channels;
+ params->num_tc = 1;
+
+ mlx5e_get_max_linkspeed(mdev, &link_speed);
+ mlx5e_get_pci_bw(mdev, &pci_bw);
+ mlx5_core_dbg(mdev, "Max link speed = %d, PCI BW = %d\n",
+ link_speed, pci_bw);
+
+ /* SQ */
+ params->log_sq_size = is_kdump_kernel() ?
+ MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE :
+ MLX5E_PARAMS_DEFAULT_LOG_SQ_SIZE;
+
+ /* set CQE compression */
+ params->rx_cqe_compress_def = false;
+ if (MLX5_CAP_GEN(mdev, cqe_compression) &&
+ MLX5_CAP_GEN(mdev, vport_group_manager))
+ params->rx_cqe_compress_def = cqe_compress_heuristic(link_speed, pci_bw);
+
+ MLX5E_SET_PFLAG(params, MLX5E_PFLAG_RX_CQE_COMPRESS, params->rx_cqe_compress_def);
+
+ /* RQ */
+ mlx5e_set_rq_params(mdev, params);
+
+ /* HW LRO */
+ /* TODO: && MLX5_CAP_ETH(mdev, lro_cap) */
+ if (params->rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ)
+ params->lro_en = hw_lro_heuristic(link_speed, pci_bw);
+ params->lro_timeout = mlx5e_choose_lro_timeout(mdev, MLX5E_DEFAULT_LRO_TIMEOUT);
+
+ /* CQ moderation params */
+ cq_period_mode = MLX5_CAP_GEN(mdev, cq_period_start_from_cqe) ?
+ MLX5_CQ_PERIOD_MODE_START_FROM_CQE :
+ MLX5_CQ_PERIOD_MODE_START_FROM_EQE;
+ params->rx_am_enabled = MLX5_CAP_GEN(mdev, cq_moderation);
+ mlx5e_set_rx_cq_mode_params(params, cq_period_mode);
+
+ params->tx_cq_moderation.usec = MLX5E_PARAMS_DEFAULT_TX_CQ_MODERATION_USEC;
+ params->tx_cq_moderation.pkts = MLX5E_PARAMS_DEFAULT_TX_CQ_MODERATION_PKTS;
+
+ /* TX inline */
+ params->tx_max_inline = mlx5e_get_max_inline_cap(mdev);
+ mlx5_query_min_inline(mdev, ¶ms->tx_min_inline_mode);
+ if (params->tx_min_inline_mode == MLX5_INLINE_MODE_NONE &&
+ !MLX5_CAP_ETH(mdev, wqe_vlan_insert))
+ params->tx_min_inline_mode = MLX5_INLINE_MODE_L2;
+
+ /* RSS */
+ params->rss_hfunc = ETH_RSS_HASH_XOR;
+ netdev_rss_key_fill(params->toeplitz_hash_key, sizeof(params->toeplitz_hash_key));
+ mlx5e_build_default_indir_rqt(mdev, params->indirection_rqt,
+ MLX5E_INDIR_RQT_SIZE, max_channels);
+}
+
static void mlx5e_build_nic_netdev_priv(struct mlx5_core_dev *mdev,
struct net_device *netdev,
const struct mlx5e_profile *profile,
void *ppriv)
{
struct mlx5e_priv *priv = netdev_priv(netdev);
- u32 link_speed = 0;
- u32 pci_bw = 0;
- u8 cq_period_mode = MLX5_CAP_GEN(mdev, cq_period_start_from_cqe) ?
- MLX5_CQ_PERIOD_MODE_START_FROM_CQE :
- MLX5_CQ_PERIOD_MODE_START_FROM_EQE;
- priv->mdev = mdev;
- priv->netdev = netdev;
- priv->params.num_channels = profile->max_nch(mdev);
- priv->profile = profile;
- priv->ppriv = ppriv;
+ priv->mdev = mdev;
+ priv->netdev = netdev;
+ priv->profile = profile;
+ priv->ppriv = ppriv;
- priv->params.lro_timeout =
- mlx5e_choose_lro_timeout(mdev, MLX5E_DEFAULT_LRO_TIMEOUT);
-
- priv->params.log_sq_size = is_kdump_kernel() ?
- MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE :
- MLX5E_PARAMS_DEFAULT_LOG_SQ_SIZE;
-
- /* set CQE compression */
- priv->params.rx_cqe_compress_def = false;
- if (MLX5_CAP_GEN(mdev, cqe_compression) &&
- MLX5_CAP_GEN(mdev, vport_group_manager)) {
- mlx5e_get_max_linkspeed(mdev, &link_speed);
- mlx5e_get_pci_bw(mdev, &pci_bw);
- mlx5_core_dbg(mdev, "Max link speed = %d, PCI BW = %d\n",
- link_speed, pci_bw);
- priv->params.rx_cqe_compress_def =
- cqe_compress_heuristic(link_speed, pci_bw);
- }
-
- MLX5E_SET_PFLAG(priv, MLX5E_PFLAG_RX_CQE_COMPRESS,
- priv->params.rx_cqe_compress_def);
-
- mlx5e_set_rq_priv_params(priv);
- if (priv->params.rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ)
- priv->params.lro_en = true;
-
- priv->params.rx_am_enabled = MLX5_CAP_GEN(mdev, cq_moderation);
- mlx5e_set_rx_cq_mode_params(&priv->params, cq_period_mode);
-
- priv->params.tx_cq_moderation.usec =
- MLX5E_PARAMS_DEFAULT_TX_CQ_MODERATION_USEC;
- priv->params.tx_cq_moderation.pkts =
- MLX5E_PARAMS_DEFAULT_TX_CQ_MODERATION_PKTS;
- priv->params.tx_max_inline = mlx5e_get_max_inline_cap(mdev);
- mlx5_query_min_inline(mdev, &priv->params.tx_min_inline_mode);
- if (priv->params.tx_min_inline_mode == MLX5_INLINE_MODE_NONE &&
- !MLX5_CAP_ETH(mdev, wqe_vlan_insert))
- priv->params.tx_min_inline_mode = MLX5_INLINE_MODE_L2;
-
- priv->params.num_tc = 1;
- priv->params.rss_hfunc = ETH_RSS_HASH_XOR;
-
- netdev_rss_key_fill(priv->params.toeplitz_hash_key,
- sizeof(priv->params.toeplitz_hash_key));
-
- mlx5e_build_default_indir_rqt(mdev, priv->params.indirection_rqt,
- MLX5E_INDIR_RQT_SIZE, profile->max_nch(mdev));
-
- /* Initialize pflags */
- MLX5E_SET_PFLAG(priv, MLX5E_PFLAG_RX_CQE_BASED_MODER,
- priv->params.rx_cq_period_mode == MLX5_CQ_PERIOD_MODE_START_FROM_CQE);
+ mlx5e_build_nic_params(mdev, &priv->channels.params, profile->max_nch(mdev));
mutex_init(&priv->state_lock);
@@ -3642,13 +3983,19 @@
if (fcs_supported)
netdev->hw_features |= NETIF_F_RXALL;
+ if (MLX5_CAP_ETH(mdev, scatter_fcs))
+ netdev->hw_features |= NETIF_F_RXFCS;
+
netdev->features = netdev->hw_features;
- if (!priv->params.lro_en)
+ if (!priv->channels.params.lro_en)
netdev->features &= ~NETIF_F_LRO;
if (fcs_enabled)
netdev->features &= ~NETIF_F_RXALL;
+ if (!priv->channels.params.scatter_fcs_en)
+ netdev->features &= ~NETIF_F_RXFCS;
+
#define FT_CAP(f) MLX5_CAP_FLOWTABLE(mdev, flow_table_properties_nic_receive.f)
if (FT_CAP(flow_modify_en) &&
FT_CAP(modify_root) &&
@@ -3708,39 +4055,30 @@
{
mlx5e_vxlan_cleanup(priv);
- if (priv->xdp_prog)
- bpf_prog_put(priv->xdp_prog);
+ if (priv->channels.params.xdp_prog)
+ bpf_prog_put(priv->channels.params.xdp_prog);
}
static int mlx5e_init_nic_rx(struct mlx5e_priv *priv)
{
struct mlx5_core_dev *mdev = priv->mdev;
int err;
- int i;
- err = mlx5e_create_indirect_rqts(priv);
- if (err) {
- mlx5_core_warn(mdev, "create indirect rqts failed, %d\n", err);
+ err = mlx5e_create_indirect_rqt(priv);
+ if (err)
return err;
- }
err = mlx5e_create_direct_rqts(priv);
- if (err) {
- mlx5_core_warn(mdev, "create direct rqts failed, %d\n", err);
+ if (err)
goto err_destroy_indirect_rqts;
- }
err = mlx5e_create_indirect_tirs(priv);
- if (err) {
- mlx5_core_warn(mdev, "create indirect tirs failed, %d\n", err);
+ if (err)
goto err_destroy_direct_rqts;
- }
err = mlx5e_create_direct_tirs(priv);
- if (err) {
- mlx5_core_warn(mdev, "create direct tirs failed, %d\n", err);
+ if (err)
goto err_destroy_indirect_tirs;
- }
err = mlx5e_create_flow_steering(priv);
if (err) {
@@ -3761,8 +4099,7 @@
err_destroy_indirect_tirs:
mlx5e_destroy_indirect_tirs(priv);
err_destroy_direct_rqts:
- for (i = 0; i < priv->profile->max_nch(mdev); i++)
- mlx5e_destroy_rqt(priv, &priv->direct_tir[i].rqt);
+ mlx5e_destroy_direct_rqts(priv);
err_destroy_indirect_rqts:
mlx5e_destroy_rqt(priv, &priv->indir_rqt);
return err;
@@ -3770,14 +4107,11 @@
static void mlx5e_cleanup_nic_rx(struct mlx5e_priv *priv)
{
- int i;
-
mlx5e_tc_cleanup(priv);
mlx5e_destroy_flow_steering(priv);
mlx5e_destroy_direct_tirs(priv);
mlx5e_destroy_indirect_tirs(priv);
- for (i = 0; i < priv->profile->max_nch(priv->mdev); i++)
- mlx5e_destroy_rqt(priv, &priv->direct_tir[i].rqt);
+ mlx5e_destroy_direct_rqts(priv);
mlx5e_destroy_rqt(priv, &priv->indir_rqt);
}
@@ -3801,21 +4135,22 @@
{
struct net_device *netdev = priv->netdev;
struct mlx5_core_dev *mdev = priv->mdev;
- struct mlx5_eswitch *esw = mdev->priv.eswitch;
- struct mlx5_eswitch_rep rep;
+ u16 max_mtu;
+
+ mlx5e_init_l2_addr(priv);
+
+ /* MTU range: 68 - hw-specific max */
+ netdev->min_mtu = ETH_MIN_MTU;
+ mlx5_query_port_max_mtu(priv->mdev, &max_mtu, 1);
+ netdev->max_mtu = MLX5E_HW2SW_MTU(max_mtu);
+ mlx5e_set_dev_port_mtu(priv);
mlx5_lag_add(mdev, netdev);
mlx5e_enable_async_events(priv);
- if (MLX5_CAP_GEN(mdev, vport_group_manager)) {
- mlx5_query_nic_vport_mac_address(mdev, 0, rep.hw_id);
- rep.load = mlx5e_nic_rep_load;
- rep.unload = mlx5e_nic_rep_unload;
- rep.vport = FDB_UPLINK_VPORT;
- rep.netdev = netdev;
- mlx5_eswitch_register_vport_rep(esw, 0, &rep);
- }
+ if (MLX5_CAP_GEN(mdev, vport_group_manager))
+ mlx5e_register_vport_reps(priv);
if (netdev->reg_state != NETREG_REGISTERED)
return;
@@ -3828,16 +4163,29 @@
}
queue_work(priv->wq, &priv->set_rx_mode_work);
+
+ rtnl_lock();
+ if (netif_running(netdev))
+ mlx5e_open(netdev);
+ netif_device_attach(netdev);
+ rtnl_unlock();
}
static void mlx5e_nic_disable(struct mlx5e_priv *priv)
{
struct mlx5_core_dev *mdev = priv->mdev;
- struct mlx5_eswitch *esw = mdev->priv.eswitch;
+
+ rtnl_lock();
+ if (netif_running(priv->netdev))
+ mlx5e_close(priv->netdev);
+ netif_device_detach(priv->netdev);
+ rtnl_unlock();
queue_work(priv->wq, &priv->set_rx_mode_work);
+
if (MLX5_CAP_GEN(mdev, vport_group_manager))
- mlx5_eswitch_unregister_vport_rep(esw, 0);
+ mlx5e_unregister_vport_reps(priv);
+
mlx5e_disable_async_events(priv);
mlx5_lag_remove(mdev);
}
@@ -3853,9 +4201,13 @@
.disable = mlx5e_nic_disable,
.update_stats = mlx5e_update_stats,
.max_nch = mlx5e_get_max_num_channels,
+ .rx_handlers.handle_rx_cqe = mlx5e_handle_rx_cqe,
+ .rx_handlers.handle_rx_cqe_mpwqe = mlx5e_handle_rx_cqe_mpwrq,
.max_tc = MLX5E_MAX_NUM_TC,
};
+/* mlx5e generic netdev management API (move to en_common.c) */
+
struct net_device *mlx5e_create_netdev(struct mlx5_core_dev *mdev,
const struct mlx5e_profile *profile,
void *ppriv)
@@ -3872,6 +4224,10 @@
return NULL;
}
+#ifdef CONFIG_RFS_ACCEL
+ netdev->rx_cpu_rmap = mdev->rmap;
+#endif
+
profile->init(mdev, netdev, profile, ppriv);
netif_carrier_off(netdev);
@@ -3891,14 +4247,12 @@
return NULL;
}
-int mlx5e_attach_netdev(struct mlx5_core_dev *mdev, struct net_device *netdev)
+int mlx5e_attach_netdev(struct mlx5e_priv *priv)
{
+ struct mlx5_core_dev *mdev = priv->mdev;
const struct mlx5e_profile *profile;
- struct mlx5e_priv *priv;
- u16 max_mtu;
int err;
- priv = netdev_priv(netdev);
profile = priv->profile;
clear_bit(MLX5E_STATE_DESTROYING, &priv->state);
@@ -3906,7 +4260,7 @@
if (err)
goto out;
- err = mlx5e_open_drop_rq(priv);
+ err = mlx5e_open_drop_rq(mdev, &priv->drop_rq);
if (err) {
mlx5_core_err(mdev, "open drop rq failed, %d\n", err);
goto err_cleanup_tx;
@@ -3918,28 +4272,13 @@
mlx5e_create_q_counter(priv);
- mlx5e_init_l2_addr(priv);
-
- /* MTU range: 68 - hw-specific max */
- netdev->min_mtu = ETH_MIN_MTU;
- mlx5_query_port_max_mtu(priv->mdev, &max_mtu, 1);
- netdev->max_mtu = MLX5E_HW2SW_MTU(max_mtu);
-
- mlx5e_set_dev_port_mtu(netdev);
-
if (profile->enable)
profile->enable(priv);
- rtnl_lock();
- if (netif_running(netdev))
- mlx5e_open(netdev);
- netif_device_attach(netdev);
- rtnl_unlock();
-
return 0;
err_close_drop_rq:
- mlx5e_close_drop_rq(priv);
+ mlx5e_close_drop_rq(&priv->drop_rq);
err_cleanup_tx:
profile->cleanup_tx(priv);
@@ -3948,66 +4287,34 @@
return err;
}
-static void mlx5e_register_vport_rep(struct mlx5_core_dev *mdev)
+void mlx5e_detach_netdev(struct mlx5e_priv *priv)
{
- struct mlx5_eswitch *esw = mdev->priv.eswitch;
- int total_vfs = MLX5_TOTAL_VPORTS(mdev);
- int vport;
- u8 mac[ETH_ALEN];
-
- if (!MLX5_CAP_GEN(mdev, vport_group_manager))
- return;
-
- mlx5_query_nic_vport_mac_address(mdev, 0, mac);
-
- for (vport = 1; vport < total_vfs; vport++) {
- struct mlx5_eswitch_rep rep;
-
- rep.load = mlx5e_vport_rep_load;
- rep.unload = mlx5e_vport_rep_unload;
- rep.vport = vport;
- ether_addr_copy(rep.hw_id, mac);
- mlx5_eswitch_register_vport_rep(esw, vport, &rep);
- }
-}
-
-static void mlx5e_unregister_vport_rep(struct mlx5_core_dev *mdev)
-{
- struct mlx5_eswitch *esw = mdev->priv.eswitch;
- int total_vfs = MLX5_TOTAL_VPORTS(mdev);
- int vport;
-
- if (!MLX5_CAP_GEN(mdev, vport_group_manager))
- return;
-
- for (vport = 1; vport < total_vfs; vport++)
- mlx5_eswitch_unregister_vport_rep(esw, vport);
-}
-
-void mlx5e_detach_netdev(struct mlx5_core_dev *mdev, struct net_device *netdev)
-{
- struct mlx5e_priv *priv = netdev_priv(netdev);
const struct mlx5e_profile *profile = priv->profile;
set_bit(MLX5E_STATE_DESTROYING, &priv->state);
- rtnl_lock();
- if (netif_running(netdev))
- mlx5e_close(netdev);
- netif_device_detach(netdev);
- rtnl_unlock();
-
if (profile->disable)
profile->disable(priv);
flush_workqueue(priv->wq);
mlx5e_destroy_q_counter(priv);
profile->cleanup_rx(priv);
- mlx5e_close_drop_rq(priv);
+ mlx5e_close_drop_rq(&priv->drop_rq);
profile->cleanup_tx(priv);
cancel_delayed_work_sync(&priv->update_stats_work);
}
+void mlx5e_destroy_netdev(struct mlx5e_priv *priv)
+{
+ const struct mlx5e_profile *profile = priv->profile;
+ struct net_device *netdev = priv->netdev;
+
+ destroy_workqueue(priv->wq);
+ if (profile->cleanup)
+ profile->cleanup(priv);
+ free_netdev(netdev);
+}
+
/* mlx5e_attach and mlx5e_detach scope should be only creating/destroying
* hardware contexts and to connect it to the current netdev.
*/
@@ -4024,13 +4331,12 @@
if (err)
return err;
- err = mlx5e_attach_netdev(mdev, netdev);
+ err = mlx5e_attach_netdev(priv);
if (err) {
mlx5e_destroy_mdev_resources(mdev);
return err;
}
- mlx5e_register_vport_rep(mdev);
return 0;
}
@@ -4042,8 +4348,7 @@
if (!netif_device_present(netdev))
return;
- mlx5e_unregister_vport_rep(mdev);
- mlx5e_detach_netdev(mdev, netdev);
+ mlx5e_detach_netdev(priv);
mlx5e_destroy_mdev_resources(mdev);
}
@@ -4051,7 +4356,7 @@
{
struct mlx5_eswitch *esw = mdev->priv.eswitch;
int total_vfs = MLX5_TOTAL_VPORTS(mdev);
- void *ppriv = NULL;
+ struct mlx5e_rep_priv *rpriv = NULL;
void *priv;
int vport;
int err;
@@ -4061,10 +4366,17 @@
if (err)
return NULL;
- if (MLX5_CAP_GEN(mdev, vport_group_manager))
- ppriv = &esw->offloads.vport_reps[0];
+ if (MLX5_CAP_GEN(mdev, vport_group_manager)) {
+ rpriv = kzalloc(sizeof(*rpriv), GFP_KERNEL);
+ if (!rpriv) {
+ mlx5_core_warn(mdev,
+ "Not creating net device, Failed to alloc rep priv data\n");
+ return NULL;
+ }
+ rpriv->rep = &esw->offloads.vport_reps[0];
+ }
- netdev = mlx5e_create_netdev(mdev, &mlx5e_nic_profile, ppriv);
+ netdev = mlx5e_create_netdev(mdev, &mlx5e_nic_profile, rpriv);
if (!netdev) {
mlx5_core_err(mdev, "mlx5e_create_netdev failed\n");
goto err_unregister_reps;
@@ -4090,33 +4402,25 @@
mlx5e_detach(mdev, priv);
err_destroy_netdev:
- mlx5e_destroy_netdev(mdev, priv);
+ mlx5e_destroy_netdev(priv);
err_unregister_reps:
for (vport = 1; vport < total_vfs; vport++)
mlx5_eswitch_unregister_vport_rep(esw, vport);
+ kfree(rpriv);
return NULL;
}
-void mlx5e_destroy_netdev(struct mlx5_core_dev *mdev, struct mlx5e_priv *priv)
-{
- const struct mlx5e_profile *profile = priv->profile;
- struct net_device *netdev = priv->netdev;
-
- destroy_workqueue(priv->wq);
- if (profile->cleanup)
- profile->cleanup(priv);
- free_netdev(netdev);
-}
-
static void mlx5e_remove(struct mlx5_core_dev *mdev, void *vpriv)
{
struct mlx5e_priv *priv = vpriv;
+ void *ppriv = priv->ppriv;
unregister_netdev(priv->netdev);
mlx5e_detach(mdev, vpriv);
- mlx5e_destroy_netdev(mdev, priv);
+ mlx5e_destroy_netdev(priv);
+ kfree(ppriv);
}
static void *mlx5e_get_netdev(void *vpriv)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
index f621373..79462c0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
@@ -34,10 +34,14 @@
#include <linux/mlx5/fs.h>
#include <net/switchdev.h>
#include <net/pkt_cls.h>
+#include <net/netevent.h>
+#include <net/arp.h>
#include "eswitch.h"
#include "en.h"
+#include "en_rep.h"
#include "en_tc.h"
+#include "fs_core.h"
static const char mlx5e_rep_driver_name[] = "mlx5e_rep";
@@ -75,7 +79,8 @@
static void mlx5e_rep_update_hw_counters(struct mlx5e_priv *priv)
{
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
- struct mlx5_eswitch_rep *rep = priv->ppriv;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5_eswitch_rep *rep = rpriv->rep;
struct rtnl_link_stats64 *vport_stats;
struct ifla_vf_stats vf_stats;
int err;
@@ -102,14 +107,16 @@
int i, j;
memset(s, 0, sizeof(*s));
- for (i = 0; i < priv->params.num_channels; i++) {
- rq_stats = &priv->channel[i]->rq.stats;
+ for (i = 0; i < priv->channels.num; i++) {
+ struct mlx5e_channel *c = priv->channels.c[i];
+
+ rq_stats = &c->rq.stats;
s->rx_packets += rq_stats->packets;
s->rx_bytes += rq_stats->bytes;
- for (j = 0; j < priv->params.num_tc; j++) {
- sq_stats = &priv->channel[i]->sq[j].stats;
+ for (j = 0; j < priv->channels.params.num_tc; j++) {
+ sq_stats = &c->sq[j].stats;
s->tx_packets += sq_stats->packets;
s->tx_bytes += sq_stats->bytes;
@@ -163,7 +170,8 @@
int mlx5e_attr_get(struct net_device *dev, struct switchdev_attr *attr)
{
struct mlx5e_priv *priv = netdev_priv(dev);
- struct mlx5_eswitch_rep *rep = priv->ppriv;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5_eswitch_rep *rep = rpriv->rep;
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
if (esw->mode == SRIOV_NONE)
@@ -182,66 +190,426 @@
}
int mlx5e_add_sqs_fwd_rules(struct mlx5e_priv *priv)
-
{
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
- struct mlx5_eswitch_rep *rep = priv->ppriv;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5_eswitch_rep *rep = rpriv->rep;
struct mlx5e_channel *c;
- int n, tc, err, num_sqs = 0;
+ int n, tc, num_sqs = 0;
+ int err = -ENOMEM;
u16 *sqs;
- sqs = kcalloc(priv->params.num_channels * priv->params.num_tc, sizeof(u16), GFP_KERNEL);
+ sqs = kcalloc(priv->channels.num * priv->channels.params.num_tc, sizeof(u16), GFP_KERNEL);
if (!sqs)
- return -ENOMEM;
+ goto out;
- for (n = 0; n < priv->params.num_channels; n++) {
- c = priv->channel[n];
+ for (n = 0; n < priv->channels.num; n++) {
+ c = priv->channels.c[n];
for (tc = 0; tc < c->num_tc; tc++)
sqs[num_sqs++] = c->sq[tc].sqn;
}
err = mlx5_eswitch_sqs2vport_start(esw, rep, sqs, num_sqs);
-
kfree(sqs);
+
+out:
+ if (err)
+ netdev_warn(priv->netdev, "Failed to add SQs FWD rules %d\n", err);
return err;
}
-int mlx5e_nic_rep_load(struct mlx5_eswitch *esw, struct mlx5_eswitch_rep *rep)
-{
- struct net_device *netdev = rep->netdev;
- struct mlx5e_priv *priv = netdev_priv(netdev);
-
- if (test_bit(MLX5E_STATE_OPENED, &priv->state))
- return mlx5e_add_sqs_fwd_rules(priv);
- return 0;
-}
-
void mlx5e_remove_sqs_fwd_rules(struct mlx5e_priv *priv)
{
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
- struct mlx5_eswitch_rep *rep = priv->ppriv;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5_eswitch_rep *rep = rpriv->rep;
mlx5_eswitch_sqs2vport_stop(esw, rep);
}
-void mlx5e_nic_rep_unload(struct mlx5_eswitch *esw,
- struct mlx5_eswitch_rep *rep)
+static void mlx5e_rep_neigh_update_init_interval(struct mlx5e_rep_priv *rpriv)
{
- struct net_device *netdev = rep->netdev;
+#if IS_ENABLED(CONFIG_IPV6)
+ unsigned long ipv6_interval = NEIGH_VAR(&ipv6_stub->nd_tbl->parms,
+ DELAY_PROBE_TIME);
+#else
+ unsigned long ipv6_interval = ~0UL;
+#endif
+ unsigned long ipv4_interval = NEIGH_VAR(&arp_tbl.parms,
+ DELAY_PROBE_TIME);
+ struct net_device *netdev = rpriv->rep->netdev;
struct mlx5e_priv *priv = netdev_priv(netdev);
- if (test_bit(MLX5E_STATE_OPENED, &priv->state))
- mlx5e_remove_sqs_fwd_rules(priv);
+ rpriv->neigh_update.min_interval = min_t(unsigned long, ipv6_interval, ipv4_interval);
+ mlx5_fc_update_sampling_interval(priv->mdev, rpriv->neigh_update.min_interval);
+}
- /* clean (and re-init) existing uplink offloaded TC rules */
- mlx5e_tc_cleanup(priv);
- mlx5e_tc_init(priv);
+void mlx5e_rep_queue_neigh_stats_work(struct mlx5e_priv *priv)
+{
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5e_neigh_update_table *neigh_update = &rpriv->neigh_update;
+
+ mlx5_fc_queue_stats_work(priv->mdev,
+ &neigh_update->neigh_stats_work,
+ neigh_update->min_interval);
+}
+
+static void mlx5e_rep_neigh_stats_work(struct work_struct *work)
+{
+ struct mlx5e_rep_priv *rpriv = container_of(work, struct mlx5e_rep_priv,
+ neigh_update.neigh_stats_work.work);
+ struct net_device *netdev = rpriv->rep->netdev;
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5e_neigh_hash_entry *nhe;
+
+ rtnl_lock();
+ if (!list_empty(&rpriv->neigh_update.neigh_list))
+ mlx5e_rep_queue_neigh_stats_work(priv);
+
+ list_for_each_entry(nhe, &rpriv->neigh_update.neigh_list, neigh_list)
+ mlx5e_tc_update_neigh_used_value(nhe);
+
+ rtnl_unlock();
+}
+
+static void mlx5e_rep_neigh_entry_hold(struct mlx5e_neigh_hash_entry *nhe)
+{
+ refcount_inc(&nhe->refcnt);
+}
+
+static void mlx5e_rep_neigh_entry_release(struct mlx5e_neigh_hash_entry *nhe)
+{
+ if (refcount_dec_and_test(&nhe->refcnt))
+ kfree(nhe);
+}
+
+static void mlx5e_rep_update_flows(struct mlx5e_priv *priv,
+ struct mlx5e_encap_entry *e,
+ bool neigh_connected,
+ unsigned char ha[ETH_ALEN])
+{
+ struct ethhdr *eth = (struct ethhdr *)e->encap_header;
+
+ ASSERT_RTNL();
+
+ if ((!neigh_connected && (e->flags & MLX5_ENCAP_ENTRY_VALID)) ||
+ !ether_addr_equal(e->h_dest, ha))
+ mlx5e_tc_encap_flows_del(priv, e);
+
+ if (neigh_connected && !(e->flags & MLX5_ENCAP_ENTRY_VALID)) {
+ ether_addr_copy(e->h_dest, ha);
+ ether_addr_copy(eth->h_dest, ha);
+
+ mlx5e_tc_encap_flows_add(priv, e);
+ }
+}
+
+static void mlx5e_rep_neigh_update(struct work_struct *work)
+{
+ struct mlx5e_neigh_hash_entry *nhe =
+ container_of(work, struct mlx5e_neigh_hash_entry, neigh_update_work);
+ struct neighbour *n = nhe->n;
+ struct mlx5e_encap_entry *e;
+ unsigned char ha[ETH_ALEN];
+ struct mlx5e_priv *priv;
+ bool neigh_connected;
+ bool encap_connected;
+ u8 nud_state, dead;
+
+ rtnl_lock();
+
+ /* If these parameters are changed after we release the lock,
+ * we'll receive another event letting us know about it.
+ * We use this lock to avoid inconsistency between the neigh validity
+ * and it's hw address.
+ */
+ read_lock_bh(&n->lock);
+ memcpy(ha, n->ha, ETH_ALEN);
+ nud_state = n->nud_state;
+ dead = n->dead;
+ read_unlock_bh(&n->lock);
+
+ neigh_connected = (nud_state & NUD_VALID) && !dead;
+
+ list_for_each_entry(e, &nhe->encap_list, encap_list) {
+ encap_connected = !!(e->flags & MLX5_ENCAP_ENTRY_VALID);
+ priv = netdev_priv(e->out_dev);
+
+ if (encap_connected != neigh_connected ||
+ !ether_addr_equal(e->h_dest, ha))
+ mlx5e_rep_update_flows(priv, e, neigh_connected, ha);
+ }
+ mlx5e_rep_neigh_entry_release(nhe);
+ rtnl_unlock();
+ neigh_release(n);
+}
+
+static struct mlx5e_neigh_hash_entry *
+mlx5e_rep_neigh_entry_lookup(struct mlx5e_priv *priv,
+ struct mlx5e_neigh *m_neigh);
+
+static int mlx5e_rep_netevent_event(struct notifier_block *nb,
+ unsigned long event, void *ptr)
+{
+ struct mlx5e_rep_priv *rpriv = container_of(nb, struct mlx5e_rep_priv,
+ neigh_update.netevent_nb);
+ struct mlx5e_neigh_update_table *neigh_update = &rpriv->neigh_update;
+ struct net_device *netdev = rpriv->rep->netdev;
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5e_neigh_hash_entry *nhe = NULL;
+ struct mlx5e_neigh m_neigh = {};
+ struct neigh_parms *p;
+ struct neighbour *n;
+ bool found = false;
+
+ switch (event) {
+ case NETEVENT_NEIGH_UPDATE:
+ n = ptr;
+#if IS_ENABLED(CONFIG_IPV6)
+ if (n->tbl != ipv6_stub->nd_tbl && n->tbl != &arp_tbl)
+#else
+ if (n->tbl != &arp_tbl)
+#endif
+ return NOTIFY_DONE;
+
+ m_neigh.dev = n->dev;
+ m_neigh.family = n->ops->family;
+ memcpy(&m_neigh.dst_ip, n->primary_key, n->tbl->key_len);
+
+ /* We are in atomic context and can't take RTNL mutex, so use
+ * spin_lock_bh to lookup the neigh table. bh is used since
+ * netevent can be called from a softirq context.
+ */
+ spin_lock_bh(&neigh_update->encap_lock);
+ nhe = mlx5e_rep_neigh_entry_lookup(priv, &m_neigh);
+ if (!nhe) {
+ spin_unlock_bh(&neigh_update->encap_lock);
+ return NOTIFY_DONE;
+ }
+
+ /* This assignment is valid as long as the the neigh reference
+ * is taken
+ */
+ nhe->n = n;
+
+ /* Take a reference to ensure the neighbour and mlx5 encap
+ * entry won't be destructed until we drop the reference in
+ * delayed work.
+ */
+ neigh_hold(n);
+ mlx5e_rep_neigh_entry_hold(nhe);
+
+ if (!queue_work(priv->wq, &nhe->neigh_update_work)) {
+ mlx5e_rep_neigh_entry_release(nhe);
+ neigh_release(n);
+ }
+ spin_unlock_bh(&neigh_update->encap_lock);
+ break;
+
+ case NETEVENT_DELAY_PROBE_TIME_UPDATE:
+ p = ptr;
+
+ /* We check the device is present since we don't care about
+ * changes in the default table, we only care about changes
+ * done per device delay prob time parameter.
+ */
+#if IS_ENABLED(CONFIG_IPV6)
+ if (!p->dev || (p->tbl != ipv6_stub->nd_tbl && p->tbl != &arp_tbl))
+#else
+ if (!p->dev || p->tbl != &arp_tbl)
+#endif
+ return NOTIFY_DONE;
+
+ /* We are in atomic context and can't take RTNL mutex,
+ * so use spin_lock_bh to walk the neigh list and look for
+ * the relevant device. bh is used since netevent can be
+ * called from a softirq context.
+ */
+ spin_lock_bh(&neigh_update->encap_lock);
+ list_for_each_entry(nhe, &neigh_update->neigh_list, neigh_list) {
+ if (p->dev == nhe->m_neigh.dev) {
+ found = true;
+ break;
+ }
+ }
+ spin_unlock_bh(&neigh_update->encap_lock);
+ if (!found)
+ return NOTIFY_DONE;
+
+ neigh_update->min_interval = min_t(unsigned long,
+ NEIGH_VAR(p, DELAY_PROBE_TIME),
+ neigh_update->min_interval);
+ mlx5_fc_update_sampling_interval(priv->mdev,
+ neigh_update->min_interval);
+ break;
+ }
+ return NOTIFY_DONE;
+}
+
+static const struct rhashtable_params mlx5e_neigh_ht_params = {
+ .head_offset = offsetof(struct mlx5e_neigh_hash_entry, rhash_node),
+ .key_offset = offsetof(struct mlx5e_neigh_hash_entry, m_neigh),
+ .key_len = sizeof(struct mlx5e_neigh),
+ .automatic_shrinking = true,
+};
+
+static int mlx5e_rep_neigh_init(struct mlx5e_rep_priv *rpriv)
+{
+ struct mlx5e_neigh_update_table *neigh_update = &rpriv->neigh_update;
+ int err;
+
+ err = rhashtable_init(&neigh_update->neigh_ht, &mlx5e_neigh_ht_params);
+ if (err)
+ return err;
+
+ INIT_LIST_HEAD(&neigh_update->neigh_list);
+ spin_lock_init(&neigh_update->encap_lock);
+ INIT_DELAYED_WORK(&neigh_update->neigh_stats_work,
+ mlx5e_rep_neigh_stats_work);
+ mlx5e_rep_neigh_update_init_interval(rpriv);
+
+ rpriv->neigh_update.netevent_nb.notifier_call = mlx5e_rep_netevent_event;
+ err = register_netevent_notifier(&rpriv->neigh_update.netevent_nb);
+ if (err)
+ goto out_err;
+ return 0;
+
+out_err:
+ rhashtable_destroy(&neigh_update->neigh_ht);
+ return err;
+}
+
+static void mlx5e_rep_neigh_cleanup(struct mlx5e_rep_priv *rpriv)
+{
+ struct mlx5e_neigh_update_table *neigh_update = &rpriv->neigh_update;
+ struct mlx5e_priv *priv = netdev_priv(rpriv->rep->netdev);
+
+ unregister_netevent_notifier(&neigh_update->netevent_nb);
+
+ flush_workqueue(priv->wq); /* flush neigh update works */
+
+ cancel_delayed_work_sync(&rpriv->neigh_update.neigh_stats_work);
+
+ rhashtable_destroy(&neigh_update->neigh_ht);
+}
+
+static int mlx5e_rep_neigh_entry_insert(struct mlx5e_priv *priv,
+ struct mlx5e_neigh_hash_entry *nhe)
+{
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ int err;
+
+ err = rhashtable_insert_fast(&rpriv->neigh_update.neigh_ht,
+ &nhe->rhash_node,
+ mlx5e_neigh_ht_params);
+ if (err)
+ return err;
+
+ list_add(&nhe->neigh_list, &rpriv->neigh_update.neigh_list);
+
+ return err;
+}
+
+static void mlx5e_rep_neigh_entry_remove(struct mlx5e_priv *priv,
+ struct mlx5e_neigh_hash_entry *nhe)
+{
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+
+ spin_lock_bh(&rpriv->neigh_update.encap_lock);
+
+ list_del(&nhe->neigh_list);
+
+ rhashtable_remove_fast(&rpriv->neigh_update.neigh_ht,
+ &nhe->rhash_node,
+ mlx5e_neigh_ht_params);
+ spin_unlock_bh(&rpriv->neigh_update.encap_lock);
+}
+
+/* This function must only be called under RTNL lock or under the
+ * representor's encap_lock in case RTNL mutex can't be held.
+ */
+static struct mlx5e_neigh_hash_entry *
+mlx5e_rep_neigh_entry_lookup(struct mlx5e_priv *priv,
+ struct mlx5e_neigh *m_neigh)
+{
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5e_neigh_update_table *neigh_update = &rpriv->neigh_update;
+
+ return rhashtable_lookup_fast(&neigh_update->neigh_ht, m_neigh,
+ mlx5e_neigh_ht_params);
+}
+
+static int mlx5e_rep_neigh_entry_create(struct mlx5e_priv *priv,
+ struct mlx5e_encap_entry *e,
+ struct mlx5e_neigh_hash_entry **nhe)
+{
+ int err;
+
+ *nhe = kzalloc(sizeof(**nhe), GFP_KERNEL);
+ if (!*nhe)
+ return -ENOMEM;
+
+ memcpy(&(*nhe)->m_neigh, &e->m_neigh, sizeof(e->m_neigh));
+ INIT_WORK(&(*nhe)->neigh_update_work, mlx5e_rep_neigh_update);
+ INIT_LIST_HEAD(&(*nhe)->encap_list);
+ refcount_set(&(*nhe)->refcnt, 1);
+
+ err = mlx5e_rep_neigh_entry_insert(priv, *nhe);
+ if (err)
+ goto out_free;
+ return 0;
+
+out_free:
+ kfree(*nhe);
+ return err;
+}
+
+static void mlx5e_rep_neigh_entry_destroy(struct mlx5e_priv *priv,
+ struct mlx5e_neigh_hash_entry *nhe)
+{
+ /* The neigh hash entry must be removed from the hash table regardless
+ * of the reference count value, so it won't be found by the next
+ * neigh notification call. The neigh hash entry reference count is
+ * incremented only during creation and neigh notification calls and
+ * protects from freeing the nhe struct.
+ */
+ mlx5e_rep_neigh_entry_remove(priv, nhe);
+ mlx5e_rep_neigh_entry_release(nhe);
+}
+
+int mlx5e_rep_encap_entry_attach(struct mlx5e_priv *priv,
+ struct mlx5e_encap_entry *e)
+{
+ struct mlx5e_neigh_hash_entry *nhe;
+ int err;
+
+ nhe = mlx5e_rep_neigh_entry_lookup(priv, &e->m_neigh);
+ if (!nhe) {
+ err = mlx5e_rep_neigh_entry_create(priv, e, &nhe);
+ if (err)
+ return err;
+ }
+ list_add(&e->encap_list, &nhe->encap_list);
+ return 0;
+}
+
+void mlx5e_rep_encap_entry_detach(struct mlx5e_priv *priv,
+ struct mlx5e_encap_entry *e)
+{
+ struct mlx5e_neigh_hash_entry *nhe;
+
+ list_del(&e->encap_list);
+ nhe = mlx5e_rep_neigh_entry_lookup(priv, &e->m_neigh);
+
+ if (list_empty(&nhe->encap_list))
+ mlx5e_rep_neigh_entry_destroy(priv, nhe);
}
static int mlx5e_rep_open(struct net_device *dev)
{
struct mlx5e_priv *priv = netdev_priv(dev);
- struct mlx5_eswitch_rep *rep = priv->ppriv;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5_eswitch_rep *rep = rpriv->rep;
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
int err;
@@ -259,7 +627,8 @@
static int mlx5e_rep_close(struct net_device *dev)
{
struct mlx5e_priv *priv = netdev_priv(dev);
- struct mlx5_eswitch_rep *rep = priv->ppriv;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5_eswitch_rep *rep = rpriv->rep;
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
(void)mlx5_eswitch_set_vport_state(esw, rep->vport, MLX5_ESW_VPORT_ADMIN_STATE_DOWN);
@@ -271,7 +640,8 @@
char *buf, size_t len)
{
struct mlx5e_priv *priv = netdev_priv(dev);
- struct mlx5_eswitch_rep *rep = priv->ppriv;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5_eswitch_rep *rep = rpriv->rep;
int ret;
ret = snprintf(buf, len, "%d", rep->vport - 1);
@@ -314,18 +684,25 @@
bool mlx5e_is_uplink_rep(struct mlx5e_priv *priv)
{
- struct mlx5_eswitch_rep *rep = (struct mlx5_eswitch_rep *)priv->ppriv;
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5_eswitch_rep *rep;
- if (rep && rep->vport == FDB_UPLINK_VPORT && esw->mode == SRIOV_OFFLOADS)
+ if (!MLX5_CAP_GEN(priv->mdev, vport_group_manager))
+ return false;
+
+ rep = rpriv->rep;
+ if (esw->mode == SRIOV_OFFLOADS &&
+ rep && rep->vport == FDB_UPLINK_VPORT)
return true;
return false;
}
-bool mlx5e_is_vf_vport_rep(struct mlx5e_priv *priv)
+static bool mlx5e_is_vf_vport_rep(struct mlx5e_priv *priv)
{
- struct mlx5_eswitch_rep *rep = (struct mlx5_eswitch_rep *)priv->ppriv;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5_eswitch_rep *rep = rpriv->rep;
if (rep && rep->vport != FDB_UPLINK_VPORT)
return true;
@@ -397,42 +774,23 @@
.ndo_get_offload_stats = mlx5e_get_offload_stats,
};
-static void mlx5e_build_rep_netdev_priv(struct mlx5_core_dev *mdev,
- struct net_device *netdev,
- const struct mlx5e_profile *profile,
- void *ppriv)
+static void mlx5e_build_rep_params(struct mlx5_core_dev *mdev,
+ struct mlx5e_params *params)
{
- struct mlx5e_priv *priv = netdev_priv(netdev);
u8 cq_period_mode = MLX5_CAP_GEN(mdev, cq_period_start_from_cqe) ?
MLX5_CQ_PERIOD_MODE_START_FROM_CQE :
MLX5_CQ_PERIOD_MODE_START_FROM_EQE;
- priv->params.log_sq_size =
- MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE;
- priv->params.rq_wq_type = MLX5_WQ_TYPE_LINKED_LIST;
- priv->params.log_rq_size = MLX5E_PARAMS_MINIMUM_LOG_RQ_SIZE;
+ params->log_sq_size = MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE;
+ params->rq_wq_type = MLX5_WQ_TYPE_LINKED_LIST;
+ params->log_rq_size = MLX5E_PARAMS_MINIMUM_LOG_RQ_SIZE;
- priv->params.min_rx_wqes = mlx5_min_rx_wqes(priv->params.rq_wq_type,
- BIT(priv->params.log_rq_size));
+ params->rx_am_enabled = MLX5_CAP_GEN(mdev, cq_moderation);
+ mlx5e_set_rx_cq_mode_params(params, cq_period_mode);
- priv->params.rx_am_enabled = MLX5_CAP_GEN(mdev, cq_moderation);
- mlx5e_set_rx_cq_mode_params(&priv->params, cq_period_mode);
-
- priv->params.tx_max_inline = mlx5e_get_max_inline_cap(mdev);
- priv->params.num_tc = 1;
-
- priv->params.lro_wqe_sz =
- MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ;
-
- priv->mdev = mdev;
- priv->netdev = netdev;
- priv->params.num_channels = profile->max_nch(mdev);
- priv->profile = profile;
- priv->ppriv = ppriv;
-
- mutex_init(&priv->state_lock);
-
- INIT_DELAYED_WORK(&priv->update_stats_work, mlx5e_update_stats_work);
+ params->tx_max_inline = mlx5e_get_max_inline_cap(mdev);
+ params->num_tc = 1;
+ params->lro_wqe_sz = MLX5E_PARAMS_DEFAULT_LRO_WQE_SZ;
}
static void mlx5e_build_rep_netdev(struct net_device *netdev)
@@ -458,30 +816,39 @@
const struct mlx5e_profile *profile,
void *ppriv)
{
- mlx5e_build_rep_netdev_priv(mdev, netdev, profile, ppriv);
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+
+ priv->mdev = mdev;
+ priv->netdev = netdev;
+ priv->profile = profile;
+ priv->ppriv = ppriv;
+
+ mutex_init(&priv->state_lock);
+
+ INIT_DELAYED_WORK(&priv->update_stats_work, mlx5e_update_stats_work);
+
+ priv->channels.params.num_channels = profile->max_nch(mdev);
+ mlx5e_build_rep_params(mdev, &priv->channels.params);
mlx5e_build_rep_netdev(netdev);
}
static int mlx5e_init_rep_rx(struct mlx5e_priv *priv)
{
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
- struct mlx5_eswitch_rep *rep = priv->ppriv;
- struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5_eswitch_rep *rep = rpriv->rep;
struct mlx5_flow_handle *flow_rule;
int err;
- int i;
+
+ mlx5e_init_l2_addr(priv);
err = mlx5e_create_direct_rqts(priv);
- if (err) {
- mlx5_core_warn(mdev, "create direct rqts failed, %d\n", err);
+ if (err)
return err;
- }
err = mlx5e_create_direct_tirs(priv);
- if (err) {
- mlx5_core_warn(mdev, "create direct tirs failed, %d\n", err);
+ if (err)
goto err_destroy_direct_rqts;
- }
flow_rule = mlx5_eswitch_create_vport_rx_rule(esw,
rep->vport,
@@ -503,21 +870,19 @@
err_destroy_direct_tirs:
mlx5e_destroy_direct_tirs(priv);
err_destroy_direct_rqts:
- for (i = 0; i < priv->params.num_channels; i++)
- mlx5e_destroy_rqt(priv, &priv->direct_tir[i].rqt);
+ mlx5e_destroy_direct_rqts(priv);
return err;
}
static void mlx5e_cleanup_rep_rx(struct mlx5e_priv *priv)
{
- struct mlx5_eswitch_rep *rep = priv->ppriv;
- int i;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5_eswitch_rep *rep = rpriv->rep;
mlx5e_tc_cleanup(priv);
mlx5_del_flow_rules(rep->vport_rx_rule);
mlx5e_destroy_direct_tirs(priv);
- for (i = 0; i < priv->params.num_channels; i++)
- mlx5e_destroy_rqt(priv, &priv->direct_tir[i].rqt);
+ mlx5e_destroy_direct_rqts(priv);
}
static int mlx5e_init_rep_tx(struct mlx5e_priv *priv)
@@ -546,56 +911,181 @@
.cleanup_tx = mlx5e_cleanup_nic_tx,
.update_stats = mlx5e_rep_update_stats,
.max_nch = mlx5e_get_rep_max_num_channels,
+ .rx_handlers.handle_rx_cqe = mlx5e_handle_rx_cqe_rep,
+ .rx_handlers.handle_rx_cqe_mpwqe = NULL /* Not supported */,
.max_tc = 1,
};
-int mlx5e_vport_rep_load(struct mlx5_eswitch *esw,
- struct mlx5_eswitch_rep *rep)
+/* e-Switch vport representors */
+
+static int
+mlx5e_nic_rep_load(struct mlx5_eswitch *esw, struct mlx5_eswitch_rep *rep)
{
+ struct mlx5e_priv *priv = netdev_priv(rep->netdev);
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+
+ int err;
+
+ if (test_bit(MLX5E_STATE_OPENED, &priv->state)) {
+ err = mlx5e_add_sqs_fwd_rules(priv);
+ if (err)
+ return err;
+ }
+
+ err = mlx5e_rep_neigh_init(rpriv);
+ if (err)
+ goto err_remove_sqs;
+
+ return 0;
+
+err_remove_sqs:
+ mlx5e_remove_sqs_fwd_rules(priv);
+ return err;
+}
+
+static void
+mlx5e_nic_rep_unload(struct mlx5_eswitch *esw, struct mlx5_eswitch_rep *rep)
+{
+ struct mlx5e_priv *priv = netdev_priv(rep->netdev);
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+
+ if (test_bit(MLX5E_STATE_OPENED, &priv->state))
+ mlx5e_remove_sqs_fwd_rules(priv);
+
+ /* clean (and re-init) existing uplink offloaded TC rules */
+ mlx5e_tc_cleanup(priv);
+ mlx5e_tc_init(priv);
+
+ mlx5e_rep_neigh_cleanup(rpriv);
+}
+
+static int
+mlx5e_vport_rep_load(struct mlx5_eswitch *esw, struct mlx5_eswitch_rep *rep)
+{
+ struct mlx5e_rep_priv *rpriv;
struct net_device *netdev;
int err;
- netdev = mlx5e_create_netdev(esw->dev, &mlx5e_rep_profile, rep);
+ rpriv = kzalloc(sizeof(*rpriv), GFP_KERNEL);
+ if (!rpriv)
+ return -ENOMEM;
+
+ netdev = mlx5e_create_netdev(esw->dev, &mlx5e_rep_profile, rpriv);
if (!netdev) {
pr_warn("Failed to create representor netdev for vport %d\n",
rep->vport);
+ kfree(rpriv);
return -EINVAL;
}
rep->netdev = netdev;
+ rpriv->rep = rep;
- err = mlx5e_attach_netdev(esw->dev, netdev);
+ err = mlx5e_attach_netdev(netdev_priv(netdev));
if (err) {
pr_warn("Failed to attach representor netdev for vport %d\n",
rep->vport);
goto err_destroy_netdev;
}
- err = register_netdev(netdev);
+ err = mlx5e_rep_neigh_init(rpriv);
if (err) {
- pr_warn("Failed to register representor netdev for vport %d\n",
+ pr_warn("Failed to initialized neighbours handling for vport %d\n",
rep->vport);
goto err_detach_netdev;
}
+ err = register_netdev(netdev);
+ if (err) {
+ pr_warn("Failed to register representor netdev for vport %d\n",
+ rep->vport);
+ goto err_neigh_cleanup;
+ }
+
return 0;
+err_neigh_cleanup:
+ mlx5e_rep_neigh_cleanup(rpriv);
+
err_detach_netdev:
- mlx5e_detach_netdev(esw->dev, netdev);
+ mlx5e_detach_netdev(netdev_priv(netdev));
err_destroy_netdev:
- mlx5e_destroy_netdev(esw->dev, netdev_priv(netdev));
-
+ mlx5e_destroy_netdev(netdev_priv(netdev));
+ kfree(rpriv);
return err;
}
-void mlx5e_vport_rep_unload(struct mlx5_eswitch *esw,
- struct mlx5_eswitch_rep *rep)
+static void
+mlx5e_vport_rep_unload(struct mlx5_eswitch *esw, struct mlx5_eswitch_rep *rep)
{
struct net_device *netdev = rep->netdev;
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ void *ppriv = priv->ppriv;
- unregister_netdev(netdev);
- mlx5e_detach_netdev(esw->dev, netdev);
- mlx5e_destroy_netdev(esw->dev, netdev_priv(netdev));
+ unregister_netdev(rep->netdev);
+
+ mlx5e_rep_neigh_cleanup(rpriv);
+ mlx5e_detach_netdev(priv);
+ mlx5e_destroy_netdev(priv);
+ kfree(ppriv); /* mlx5e_rep_priv */
+}
+
+static void mlx5e_rep_register_vf_vports(struct mlx5e_priv *priv)
+{
+ struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5_eswitch *esw = mdev->priv.eswitch;
+ int total_vfs = MLX5_TOTAL_VPORTS(mdev);
+ int vport;
+ u8 mac[ETH_ALEN];
+
+ mlx5_query_nic_vport_mac_address(mdev, 0, mac);
+
+ for (vport = 1; vport < total_vfs; vport++) {
+ struct mlx5_eswitch_rep rep;
+
+ rep.load = mlx5e_vport_rep_load;
+ rep.unload = mlx5e_vport_rep_unload;
+ rep.vport = vport;
+ ether_addr_copy(rep.hw_id, mac);
+ mlx5_eswitch_register_vport_rep(esw, vport, &rep);
+ }
+}
+
+static void mlx5e_rep_unregister_vf_vports(struct mlx5e_priv *priv)
+{
+ struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5_eswitch *esw = mdev->priv.eswitch;
+ int total_vfs = MLX5_TOTAL_VPORTS(mdev);
+ int vport;
+
+ for (vport = 1; vport < total_vfs; vport++)
+ mlx5_eswitch_unregister_vport_rep(esw, vport);
+}
+
+void mlx5e_register_vport_reps(struct mlx5e_priv *priv)
+{
+ struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5_eswitch *esw = mdev->priv.eswitch;
+ struct mlx5_eswitch_rep rep;
+
+ mlx5_query_nic_vport_mac_address(mdev, 0, rep.hw_id);
+ rep.load = mlx5e_nic_rep_load;
+ rep.unload = mlx5e_nic_rep_unload;
+ rep.vport = FDB_UPLINK_VPORT;
+ rep.netdev = priv->netdev;
+ mlx5_eswitch_register_vport_rep(esw, 0, &rep); /* UPLINK PF vport*/
+
+ mlx5e_rep_register_vf_vports(priv); /* VFs vports */
+}
+
+void mlx5e_unregister_vport_reps(struct mlx5e_priv *priv)
+{
+ struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5_eswitch *esw = mdev->priv.eswitch;
+
+ mlx5e_rep_unregister_vf_vports(priv); /* VFs vports */
+ mlx5_eswitch_unregister_vport_rep(esw, 0); /* UPLINK PF*/
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h
new file mode 100644
index 0000000..a0a1a7a
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h
@@ -0,0 +1,145 @@
+/*
+ * Copyright (c) 2017, Mellanox Technologies. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#ifndef __MLX5E_REP_H__
+#define __MLX5E_REP_H__
+
+#include <net/ip_tunnels.h>
+#include <linux/rhashtable.h>
+#include "eswitch.h"
+#include "en.h"
+
+struct mlx5e_neigh_update_table {
+ struct rhashtable neigh_ht;
+ /* Save the neigh hash entries in a list in addition to the hash table
+ * (neigh_ht). In order to iterate easily over the neigh entries.
+ * Used for stats query.
+ */
+ struct list_head neigh_list;
+ /* protect lookup/remove operations */
+ spinlock_t encap_lock;
+ struct notifier_block netevent_nb;
+ struct delayed_work neigh_stats_work;
+ unsigned long min_interval; /* jiffies */
+};
+
+struct mlx5e_rep_priv {
+ struct mlx5_eswitch_rep *rep;
+ struct mlx5e_neigh_update_table neigh_update;
+};
+
+struct mlx5e_neigh {
+ struct net_device *dev;
+ union {
+ __be32 v4;
+ struct in6_addr v6;
+ } dst_ip;
+ int family;
+};
+
+struct mlx5e_neigh_hash_entry {
+ struct rhash_head rhash_node;
+ struct mlx5e_neigh m_neigh;
+
+ /* Save the neigh hash entry in a list on the representor in
+ * addition to the hash table. In order to iterate easily over the
+ * neighbour entries. Used for stats query.
+ */
+ struct list_head neigh_list;
+
+ /* encap list sharing the same neigh */
+ struct list_head encap_list;
+
+ /* valid only when the neigh reference is taken during
+ * neigh_update_work workqueue callback.
+ */
+ struct neighbour *n;
+ struct work_struct neigh_update_work;
+
+ /* neigh hash entry can be deleted only when the refcount is zero.
+ * refcount is needed to avoid neigh hash entry removal by TC, while
+ * it's used by the neigh notification call.
+ */
+ refcount_t refcnt;
+
+ /* Save the last reported time offloaded trafic pass over one of the
+ * neigh hash entry flows. Use it to periodically update the neigh
+ * 'used' value and avoid neigh deleting by the kernel.
+ */
+ unsigned long reported_lastuse;
+};
+
+enum {
+ /* set when the encap entry is successfully offloaded into HW */
+ MLX5_ENCAP_ENTRY_VALID = BIT(0),
+};
+
+struct mlx5e_encap_entry {
+ /* neigh hash entry list of encaps sharing the same neigh */
+ struct list_head encap_list;
+ struct mlx5e_neigh m_neigh;
+ /* a node of the eswitch encap hash table which keeping all the encap
+ * entries
+ */
+ struct hlist_node encap_hlist;
+ struct list_head flows;
+ u32 encap_id;
+ struct ip_tunnel_info tun_info;
+ unsigned char h_dest[ETH_ALEN]; /* destination eth addr */
+
+ struct net_device *out_dev;
+ int tunnel_type;
+ u8 flags;
+ char *encap_header;
+ int encap_size;
+};
+
+void mlx5e_register_vport_reps(struct mlx5e_priv *priv);
+void mlx5e_unregister_vport_reps(struct mlx5e_priv *priv);
+bool mlx5e_is_uplink_rep(struct mlx5e_priv *priv);
+int mlx5e_add_sqs_fwd_rules(struct mlx5e_priv *priv);
+void mlx5e_remove_sqs_fwd_rules(struct mlx5e_priv *priv);
+
+int mlx5e_get_offload_stats(int attr_id, const struct net_device *dev, void *sp);
+bool mlx5e_has_offload_stats(const struct net_device *dev, int attr_id);
+
+int mlx5e_attr_get(struct net_device *dev, struct switchdev_attr *attr);
+void mlx5e_handle_rx_cqe_rep(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe);
+
+int mlx5e_rep_encap_entry_attach(struct mlx5e_priv *priv,
+ struct mlx5e_encap_entry *e);
+void mlx5e_rep_encap_entry_detach(struct mlx5e_priv *priv,
+ struct mlx5e_encap_entry *e);
+
+void mlx5e_rep_queue_neigh_stats_work(struct mlx5e_priv *priv);
+
+#endif /* __MLX5E_REP_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index bafcb34..7b1566f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -39,6 +39,8 @@
#include "en.h"
#include "en_tc.h"
#include "eswitch.h"
+#include "en_rep.h"
+#include "ipoib.h"
static inline bool mlx5e_rx_hw_stamp(struct mlx5e_tstamp *tstamp)
{
@@ -156,28 +158,6 @@
return mlx5e_decompress_cqes_cont(rq, cq, 1, budget_rem) - 1;
}
-void mlx5e_modify_rx_cqe_compression_locked(struct mlx5e_priv *priv, bool val)
-{
- bool was_opened;
-
- if (!MLX5_CAP_GEN(priv->mdev, cqe_compression))
- return;
-
- if (MLX5E_GET_PFLAG(priv, MLX5E_PFLAG_RX_CQE_COMPRESS) == val)
- return;
-
- was_opened = test_bit(MLX5E_STATE_OPENED, &priv->state);
- if (was_opened)
- mlx5e_close_locked(priv->netdev);
-
- MLX5E_SET_PFLAG(priv, MLX5E_PFLAG_RX_CQE_COMPRESS, val);
- mlx5e_set_rq_type_params(priv, priv->params.rq_wq_type);
-
- if (was_opened)
- mlx5e_open_locked(priv->netdev);
-
-}
-
#define RQ_PAGE_SIZE(rq) ((1 << rq->buff.page_order) << PAGE_SHIFT)
static inline bool mlx5e_rx_cache_put(struct mlx5e_rq *rq,
@@ -331,7 +311,7 @@
static inline void mlx5e_post_umr_wqe(struct mlx5e_rq *rq, u16 ix)
{
struct mlx5e_mpw_info *wi = &rq->mpwqe.info[ix];
- struct mlx5e_sq *sq = &rq->channel->icosq;
+ struct mlx5e_icosq *sq = &rq->channel->icosq;
struct mlx5_wq_cyc *wq = &sq->wq;
struct mlx5e_umr_wqe *wqe;
u8 num_wqebbs = DIV_ROUND_UP(sizeof(*wqe), MLX5_SEND_WQE_BB);
@@ -341,7 +321,7 @@
while ((pi = (sq->pc & wq->sz_m1)) > sq->edge) {
sq->db.ico_wqe[pi].opcode = MLX5_OPCODE_NOP;
sq->db.ico_wqe[pi].num_wqebbs = 1;
- mlx5e_send_nop(sq, false);
+ mlx5e_post_nop(wq, sq->sqn, &sq->pc);
}
wqe = mlx5_wq_cyc_get_wqe(wq, pi);
@@ -353,7 +333,7 @@
sq->db.ico_wqe[pi].opcode = MLX5_OPCODE_UMR;
sq->db.ico_wqe[pi].num_wqebbs = num_wqebbs;
sq->pc += num_wqebbs;
- mlx5e_tx_notify_hw(sq, &wqe->ctrl, 0);
+ mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl);
}
static int mlx5e_alloc_rx_umr_mpwqe(struct mlx5e_rq *rq,
@@ -637,37 +617,36 @@
mlx5e_build_rx_skb(cqe, cqe_bcnt, rq, skb);
}
-static inline void mlx5e_xmit_xdp_doorbell(struct mlx5e_sq *sq)
+static inline void mlx5e_xmit_xdp_doorbell(struct mlx5e_xdpsq *sq)
{
struct mlx5_wq_cyc *wq = &sq->wq;
struct mlx5e_tx_wqe *wqe;
- u16 pi = (sq->pc - MLX5E_XDP_TX_WQEBBS) & wq->sz_m1; /* last pi */
+ u16 pi = (sq->pc - 1) & wq->sz_m1; /* last pi */
wqe = mlx5_wq_cyc_get_wqe(wq, pi);
- wqe->ctrl.fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE;
- mlx5e_tx_notify_hw(sq, &wqe->ctrl, 0);
+ mlx5e_notify_hw(wq, sq->pc, sq->uar_map, &wqe->ctrl);
}
static inline bool mlx5e_xmit_xdp_frame(struct mlx5e_rq *rq,
struct mlx5e_dma_info *di,
const struct xdp_buff *xdp)
{
- struct mlx5e_sq *sq = &rq->channel->xdp_sq;
+ struct mlx5e_xdpsq *sq = &rq->xdpsq;
struct mlx5_wq_cyc *wq = &sq->wq;
- u16 pi = sq->pc & wq->sz_m1;
+ u16 pi = sq->pc & wq->sz_m1;
struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(wq, pi);
- struct mlx5e_sq_wqe_info *wi = &sq->db.xdp.wqe_info[pi];
struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
struct mlx5_wqe_eth_seg *eseg = &wqe->eth;
struct mlx5_wqe_data_seg *dseg;
- u8 ds_cnt = MLX5E_XDP_TX_DS_COUNT;
ptrdiff_t data_offset = xdp->data - xdp->data_hard_start;
dma_addr_t dma_addr = di->addr + data_offset;
unsigned int dma_len = xdp->data_end - xdp->data;
+ prefetchw(wqe);
+
if (unlikely(dma_len < MLX5E_XDP_MIN_INLINE ||
MLX5E_SW2HW_MTU(rq->netdev->mtu) < dma_len)) {
rq->stats.xdp_drop++;
@@ -675,48 +654,42 @@
return false;
}
- if (unlikely(!mlx5e_sq_has_room_for(sq, MLX5E_XDP_TX_WQEBBS))) {
- if (sq->db.xdp.doorbell) {
+ if (unlikely(!mlx5e_wqc_has_room_for(wq, sq->cc, sq->pc, 1))) {
+ if (sq->db.doorbell) {
/* SQ is full, ring doorbell */
mlx5e_xmit_xdp_doorbell(sq);
- sq->db.xdp.doorbell = false;
+ sq->db.doorbell = false;
}
rq->stats.xdp_tx_full++;
mlx5e_page_release(rq, di, true);
return false;
}
- dma_sync_single_for_device(sq->pdev, dma_addr, dma_len,
- PCI_DMA_TODEVICE);
+ dma_sync_single_for_device(sq->pdev, dma_addr, dma_len, PCI_DMA_TODEVICE);
- memset(wqe, 0, sizeof(*wqe));
+ cseg->fm_ce_se = 0;
dseg = (struct mlx5_wqe_data_seg *)eseg + 1;
+
/* copy the inline part if required */
if (sq->min_inline_mode != MLX5_INLINE_MODE_NONE) {
memcpy(eseg->inline_hdr.start, xdp->data, MLX5E_XDP_MIN_INLINE);
eseg->inline_hdr.sz = cpu_to_be16(MLX5E_XDP_MIN_INLINE);
dma_len -= MLX5E_XDP_MIN_INLINE;
dma_addr += MLX5E_XDP_MIN_INLINE;
-
- ds_cnt += MLX5E_XDP_IHS_DS_COUNT;
dseg++;
}
/* write the dma part */
dseg->addr = cpu_to_be64(dma_addr);
dseg->byte_count = cpu_to_be32(dma_len);
- dseg->lkey = sq->mkey_be;
cseg->opmod_idx_opcode = cpu_to_be32((sq->pc << 8) | MLX5_OPCODE_SEND);
- cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt);
- sq->db.xdp.di[pi] = *di;
- wi->opcode = MLX5_OPCODE_SEND;
- wi->num_wqebbs = MLX5E_XDP_TX_WQEBBS;
- sq->pc += MLX5E_XDP_TX_WQEBBS;
+ sq->db.di[pi] = *di;
+ sq->pc++;
- sq->db.xdp.doorbell = true;
+ sq->db.doorbell = true;
rq->stats.xdp_tx++;
return true;
}
@@ -837,7 +810,8 @@
{
struct net_device *netdev = rq->netdev;
struct mlx5e_priv *priv = netdev_priv(netdev);
- struct mlx5_eswitch_rep *rep = priv->ppriv;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5_eswitch_rep *rep = rpriv->rep;
struct mlx5e_rx_wqe *wqe;
struct sk_buff *skb;
__be16 wqe_counter_be;
@@ -932,7 +906,7 @@
goto mpwrq_cqe_out;
}
- prefetch(skb->data);
+ prefetchw(skb->data);
cqe_bcnt = mpwrq_get_cqe_byte_cnt(cqe);
mlx5e_mpwqe_fill_rx_skb(rq, cqe, wi, cqe_bcnt, skb);
@@ -950,7 +924,7 @@
int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget)
{
struct mlx5e_rq *rq = container_of(cq, struct mlx5e_rq, cq);
- struct mlx5e_sq *xdp_sq = &rq->channel->xdp_sq;
+ struct mlx5e_xdpsq *xdpsq = &rq->xdpsq;
int work_done = 0;
if (unlikely(!test_bit(MLX5E_RQ_STATE_ENABLED, &rq->state)))
@@ -977,9 +951,9 @@
rq->handle_rx_cqe(rq, cqe);
}
- if (xdp_sq->db.xdp.doorbell) {
- mlx5e_xmit_xdp_doorbell(xdp_sq);
- xdp_sq->db.xdp.doorbell = false;
+ if (xdpsq->db.doorbell) {
+ mlx5e_xmit_xdp_doorbell(xdpsq);
+ xdpsq->db.doorbell = false;
}
mlx5_cqwq_update_db_record(&cq->wq);
@@ -989,3 +963,152 @@
return work_done;
}
+
+bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
+{
+ struct mlx5e_xdpsq *sq;
+ struct mlx5e_rq *rq;
+ u16 sqcc;
+ int i;
+
+ sq = container_of(cq, struct mlx5e_xdpsq, cq);
+
+ if (unlikely(!test_bit(MLX5E_SQ_STATE_ENABLED, &sq->state)))
+ return false;
+
+ rq = container_of(sq, struct mlx5e_rq, xdpsq);
+
+ /* sq->cc must be updated only after mlx5_cqwq_update_db_record(),
+ * otherwise a cq overrun may occur
+ */
+ sqcc = sq->cc;
+
+ for (i = 0; i < MLX5E_TX_CQ_POLL_BUDGET; i++) {
+ struct mlx5_cqe64 *cqe;
+ u16 wqe_counter;
+ bool last_wqe;
+
+ cqe = mlx5e_get_cqe(cq);
+ if (!cqe)
+ break;
+
+ mlx5_cqwq_pop(&cq->wq);
+
+ wqe_counter = be16_to_cpu(cqe->wqe_counter);
+
+ do {
+ struct mlx5e_dma_info *di;
+ u16 ci;
+
+ last_wqe = (sqcc == wqe_counter);
+
+ ci = sqcc & sq->wq.sz_m1;
+ di = &sq->db.di[ci];
+
+ sqcc++;
+ /* Recycle RX page */
+ mlx5e_page_release(rq, di, true);
+ } while (!last_wqe);
+ }
+
+ mlx5_cqwq_update_db_record(&cq->wq);
+
+ /* ensure cq space is freed before enabling more cqes */
+ wmb();
+
+ sq->cc = sqcc;
+ return (i == MLX5E_TX_CQ_POLL_BUDGET);
+}
+
+void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq)
+{
+ struct mlx5e_rq *rq = container_of(sq, struct mlx5e_rq, xdpsq);
+ struct mlx5e_dma_info *di;
+ u16 ci;
+
+ while (sq->cc != sq->pc) {
+ ci = sq->cc & sq->wq.sz_m1;
+ di = &sq->db.di[ci];
+ sq->cc++;
+
+ mlx5e_page_release(rq, di, false);
+ }
+}
+
+#ifdef CONFIG_MLX5_CORE_IPOIB
+
+#define MLX5_IB_GRH_DGID_OFFSET 24
+#define MLX5_IB_GRH_BYTES 40
+#define MLX5_IPOIB_ENCAP_LEN 4
+#define MLX5_GID_SIZE 16
+
+static inline void mlx5i_complete_rx_cqe(struct mlx5e_rq *rq,
+ struct mlx5_cqe64 *cqe,
+ u32 cqe_bcnt,
+ struct sk_buff *skb)
+{
+ struct net_device *netdev = rq->netdev;
+ u8 *dgid;
+ u8 g;
+
+ g = (be32_to_cpu(cqe->flags_rqpn) >> 28) & 3;
+ dgid = skb->data + MLX5_IB_GRH_DGID_OFFSET;
+ if ((!g) || dgid[0] != 0xff)
+ skb->pkt_type = PACKET_HOST;
+ else if (memcmp(dgid, netdev->broadcast + 4, MLX5_GID_SIZE) == 0)
+ skb->pkt_type = PACKET_BROADCAST;
+ else
+ skb->pkt_type = PACKET_MULTICAST;
+
+ /* TODO: IB/ipoib: Allow mcast packets from other VFs
+ * 68996a6e760e5c74654723eeb57bf65628ae87f4
+ */
+
+ skb_pull(skb, MLX5_IB_GRH_BYTES);
+
+ skb->protocol = *((__be16 *)(skb->data));
+
+ skb->ip_summed = CHECKSUM_COMPLETE;
+ skb->csum = csum_unfold((__force __sum16)cqe->check_sum);
+
+ skb_record_rx_queue(skb, rq->ix);
+
+ if (likely(netdev->features & NETIF_F_RXHASH))
+ mlx5e_skb_set_hash(cqe, skb);
+
+ skb_reset_mac_header(skb);
+ skb_pull(skb, MLX5_IPOIB_ENCAP_LEN);
+
+ skb->dev = netdev;
+
+ rq->stats.csum_complete++;
+ rq->stats.packets++;
+ rq->stats.bytes += cqe_bcnt;
+}
+
+void mlx5i_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)
+{
+ struct mlx5e_rx_wqe *wqe;
+ __be16 wqe_counter_be;
+ struct sk_buff *skb;
+ u16 wqe_counter;
+ u32 cqe_bcnt;
+
+ wqe_counter_be = cqe->wqe_counter;
+ wqe_counter = be16_to_cpu(wqe_counter_be);
+ wqe = mlx5_wq_ll_get_wqe(&rq->wq, wqe_counter);
+ cqe_bcnt = be32_to_cpu(cqe->byte_cnt);
+
+ skb = skb_from_cqe(rq, cqe, wqe_counter, cqe_bcnt);
+ if (!skb)
+ goto wq_ll_pop;
+
+ mlx5i_complete_rx_cqe(rq, cqe, cqe_bcnt, skb);
+ napi_gro_receive(rq->cq.napi, skb);
+
+wq_ll_pop:
+ mlx5_wq_ll_pop(&rq->wq, wqe_counter_be,
+ &wqe->next.next_wqe_index);
+}
+
+#endif /* CONFIG_MLX5_CORE_IPOIB */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
index cbfac06..02dd3a9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
@@ -293,7 +293,7 @@
struct mlx5e_rq *rq = container_of(am, struct mlx5e_rq, am);
struct mlx5e_cq_moder cur_profile = profile[am->mode][am->profile_ix];
- mlx5_core_modify_cq_moderation(rq->priv->mdev, &rq->cq.mcq,
+ mlx5_core_modify_cq_moderation(rq->mdev, &rq->cq.mcq,
cur_profile.usec, cur_profile.pkts);
am->state = MLX5E_AM_START_MEASURE;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c b/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c
index 5621dcf..5225f22 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c
@@ -236,12 +236,9 @@
{
int err = 0;
- err = mlx5e_refresh_tirs_self_loopback(priv->mdev, true);
- if (err) {
- netdev_err(priv->netdev,
- "\tFailed to enable UC loopback err(%d)\n", err);
+ err = mlx5e_refresh_tirs(priv, true);
+ if (err)
return err;
- }
lbtp->loopback_ok = false;
init_completion(&lbtp->comp);
@@ -258,7 +255,7 @@
struct mlx5e_lbt_priv *lbtp)
{
dev_remove_pack(&lbtp->pt);
- mlx5e_refresh_tirs_self_loopback(priv->mdev, false);
+ mlx5e_refresh_tirs(priv, false);
}
#define MLX5E_LB_VERIFY_TIMEOUT (msecs_to_jiffies(200))
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
index fade723..11c27e4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
@@ -42,14 +42,25 @@
#include <net/tc_act/tc_mirred.h>
#include <net/tc_act/tc_vlan.h>
#include <net/tc_act/tc_tunnel_key.h>
+#include <net/tc_act/tc_pedit.h>
#include <net/vxlan.h>
+#include <net/arp.h>
#include "en.h"
+#include "en_rep.h"
#include "en_tc.h"
#include "eswitch.h"
#include "vxlan.h"
+struct mlx5_nic_flow_attr {
+ u32 action;
+ u32 flow_tag;
+ u32 mod_hdr_id;
+};
+
enum {
MLX5E_TC_FLOW_ESWITCH = BIT(0),
+ MLX5E_TC_FLOW_NIC = BIT(1),
+ MLX5E_TC_FLOW_OFFLOADED = BIT(2),
};
struct mlx5e_tc_flow {
@@ -58,7 +69,16 @@
u8 flags;
struct mlx5_flow_handle *rule;
struct list_head encap; /* flows sharing the same encap */
- struct mlx5_esw_flow_attr *attr;
+ union {
+ struct mlx5_esw_flow_attr esw_attr[0];
+ struct mlx5_nic_flow_attr nic_attr[0];
+ };
+};
+
+struct mlx5e_tc_flow_parse_attr {
+ struct mlx5_flow_spec spec;
+ int num_mod_hdr_actions;
+ void *mod_hdr_actions;
};
enum {
@@ -71,24 +91,26 @@
static struct mlx5_flow_handle *
mlx5e_tc_add_nic_flow(struct mlx5e_priv *priv,
- struct mlx5_flow_spec *spec,
- u32 action, u32 flow_tag)
+ struct mlx5e_tc_flow_parse_attr *parse_attr,
+ struct mlx5e_tc_flow *flow)
{
+ struct mlx5_nic_flow_attr *attr = flow->nic_attr;
struct mlx5_core_dev *dev = priv->mdev;
- struct mlx5_flow_destination dest = { 0 };
+ struct mlx5_flow_destination dest = {};
struct mlx5_flow_act flow_act = {
- .action = action,
- .flow_tag = flow_tag,
+ .action = attr->action,
+ .flow_tag = attr->flow_tag,
.encap_id = 0,
};
struct mlx5_fc *counter = NULL;
struct mlx5_flow_handle *rule;
bool table_created = false;
+ int err;
- if (action & MLX5_FLOW_CONTEXT_ACTION_FWD_DEST) {
+ if (attr->action & MLX5_FLOW_CONTEXT_ACTION_FWD_DEST) {
dest.type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
dest.ft = priv->fs.vlan.ft.t;
- } else if (action & MLX5_FLOW_CONTEXT_ACTION_COUNT) {
+ } else if (attr->action & MLX5_FLOW_CONTEXT_ACTION_COUNT) {
counter = mlx5_fc_create(dev, true);
if (IS_ERR(counter))
return ERR_CAST(counter);
@@ -97,6 +119,19 @@
dest.counter = counter;
}
+ if (attr->action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR) {
+ err = mlx5_modify_header_alloc(dev, MLX5_FLOW_NAMESPACE_KERNEL,
+ parse_attr->num_mod_hdr_actions,
+ parse_attr->mod_hdr_actions,
+ &attr->mod_hdr_id);
+ flow_act.modify_id = attr->mod_hdr_id;
+ kfree(parse_attr->mod_hdr_actions);
+ if (err) {
+ rule = ERR_PTR(err);
+ goto err_create_mod_hdr_id;
+ }
+ }
+
if (IS_ERR_OR_NULL(priv->fs.tc.t)) {
priv->fs.tc.t =
mlx5_create_auto_grouped_flow_table(priv->fs.ns,
@@ -114,8 +149,9 @@
table_created = true;
}
- spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
- rule = mlx5_add_flow_rules(priv->fs.tc.t, spec, &flow_act, &dest, 1);
+ parse_attr->spec.match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
+ rule = mlx5_add_flow_rules(priv->fs.tc.t, &parse_attr->spec,
+ &flow_act, &dest, 1);
if (IS_ERR(rule))
goto err_add_rule;
@@ -128,6 +164,10 @@
priv->fs.tc.t = NULL;
}
err_create_ft:
+ if (attr->action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)
+ mlx5_modify_header_dealloc(priv->mdev,
+ attr->mod_hdr_id);
+err_create_mod_hdr_id:
mlx5_fc_destroy(dev, counter);
return rule;
@@ -138,47 +178,195 @@
{
struct mlx5_fc *counter = NULL;
- if (!IS_ERR(flow->rule)) {
- counter = mlx5_flow_rule_counter(flow->rule);
- mlx5_del_flow_rules(flow->rule);
- mlx5_fc_destroy(priv->mdev, counter);
- }
+ counter = mlx5_flow_rule_counter(flow->rule);
+ mlx5_del_flow_rules(flow->rule);
+ mlx5_fc_destroy(priv->mdev, counter);
if (!mlx5e_tc_num_filters(priv) && (priv->fs.tc.t)) {
mlx5_destroy_flow_table(priv->fs.tc.t);
priv->fs.tc.t = NULL;
}
-}
-static struct mlx5_flow_handle *
-mlx5e_tc_add_fdb_flow(struct mlx5e_priv *priv,
- struct mlx5_flow_spec *spec,
- struct mlx5_esw_flow_attr *attr)
-{
- struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
- int err;
-
- err = mlx5_eswitch_add_vlan_action(esw, attr);
- if (err)
- return ERR_PTR(err);
-
- return mlx5_eswitch_add_offloaded_rule(esw, spec, attr);
+ if (flow->nic_attr->action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)
+ mlx5_modify_header_dealloc(priv->mdev,
+ flow->nic_attr->mod_hdr_id);
}
static void mlx5e_detach_encap(struct mlx5e_priv *priv,
struct mlx5e_tc_flow *flow);
+static struct mlx5_flow_handle *
+mlx5e_tc_add_fdb_flow(struct mlx5e_priv *priv,
+ struct mlx5e_tc_flow_parse_attr *parse_attr,
+ struct mlx5e_tc_flow *flow)
+{
+ struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
+ struct mlx5_esw_flow_attr *attr = flow->esw_attr;
+ struct mlx5_flow_handle *rule;
+ int err;
+
+ err = mlx5_eswitch_add_vlan_action(esw, attr);
+ if (err) {
+ rule = ERR_PTR(err);
+ goto err_add_vlan;
+ }
+
+ if (attr->action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR) {
+ err = mlx5_modify_header_alloc(priv->mdev, MLX5_FLOW_NAMESPACE_FDB,
+ parse_attr->num_mod_hdr_actions,
+ parse_attr->mod_hdr_actions,
+ &attr->mod_hdr_id);
+ kfree(parse_attr->mod_hdr_actions);
+ if (err) {
+ rule = ERR_PTR(err);
+ goto err_mod_hdr;
+ }
+ }
+
+ rule = mlx5_eswitch_add_offloaded_rule(esw, &parse_attr->spec, attr);
+ if (IS_ERR(rule))
+ goto err_add_rule;
+
+ return rule;
+
+err_add_rule:
+ if (flow->esw_attr->action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)
+ mlx5_modify_header_dealloc(priv->mdev,
+ attr->mod_hdr_id);
+err_mod_hdr:
+ mlx5_eswitch_del_vlan_action(esw, attr);
+err_add_vlan:
+ if (attr->action & MLX5_FLOW_CONTEXT_ACTION_ENCAP)
+ mlx5e_detach_encap(priv, flow);
+ return rule;
+}
+
static void mlx5e_tc_del_fdb_flow(struct mlx5e_priv *priv,
struct mlx5e_tc_flow *flow)
{
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
+ struct mlx5_esw_flow_attr *attr = flow->esw_attr;
- mlx5_eswitch_del_offloaded_rule(esw, flow->rule, flow->attr);
+ if (flow->flags & MLX5E_TC_FLOW_OFFLOADED) {
+ flow->flags &= ~MLX5E_TC_FLOW_OFFLOADED;
+ mlx5_eswitch_del_offloaded_rule(esw, flow->rule, flow->esw_attr);
+ }
- mlx5_eswitch_del_vlan_action(esw, flow->attr);
+ mlx5_eswitch_del_vlan_action(esw, flow->esw_attr);
- if (flow->attr->action & MLX5_FLOW_CONTEXT_ACTION_ENCAP)
+ if (flow->esw_attr->action & MLX5_FLOW_CONTEXT_ACTION_ENCAP) {
mlx5e_detach_encap(priv, flow);
+ kvfree(flow->esw_attr->parse_attr);
+ }
+
+ if (flow->esw_attr->action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)
+ mlx5_modify_header_dealloc(priv->mdev,
+ attr->mod_hdr_id);
+}
+
+void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv,
+ struct mlx5e_encap_entry *e)
+{
+ struct mlx5e_tc_flow *flow;
+ int err;
+
+ err = mlx5_encap_alloc(priv->mdev, e->tunnel_type,
+ e->encap_size, e->encap_header,
+ &e->encap_id);
+ if (err) {
+ mlx5_core_warn(priv->mdev, "Failed to offload cached encapsulation header, %d\n",
+ err);
+ return;
+ }
+ e->flags |= MLX5_ENCAP_ENTRY_VALID;
+ mlx5e_rep_queue_neigh_stats_work(priv);
+
+ list_for_each_entry(flow, &e->flows, encap) {
+ flow->esw_attr->encap_id = e->encap_id;
+ flow->rule = mlx5e_tc_add_fdb_flow(priv,
+ flow->esw_attr->parse_attr,
+ flow);
+ if (IS_ERR(flow->rule)) {
+ err = PTR_ERR(flow->rule);
+ mlx5_core_warn(priv->mdev, "Failed to update cached encapsulation flow, %d\n",
+ err);
+ continue;
+ }
+ flow->flags |= MLX5E_TC_FLOW_OFFLOADED;
+ }
+}
+
+void mlx5e_tc_encap_flows_del(struct mlx5e_priv *priv,
+ struct mlx5e_encap_entry *e)
+{
+ struct mlx5e_tc_flow *flow;
+ struct mlx5_fc *counter;
+
+ list_for_each_entry(flow, &e->flows, encap) {
+ if (flow->flags & MLX5E_TC_FLOW_OFFLOADED) {
+ flow->flags &= ~MLX5E_TC_FLOW_OFFLOADED;
+ counter = mlx5_flow_rule_counter(flow->rule);
+ mlx5_del_flow_rules(flow->rule);
+ mlx5_fc_destroy(priv->mdev, counter);
+ }
+ }
+
+ if (e->flags & MLX5_ENCAP_ENTRY_VALID) {
+ e->flags &= ~MLX5_ENCAP_ENTRY_VALID;
+ mlx5_encap_dealloc(priv->mdev, e->encap_id);
+ }
+}
+
+void mlx5e_tc_update_neigh_used_value(struct mlx5e_neigh_hash_entry *nhe)
+{
+ struct mlx5e_neigh *m_neigh = &nhe->m_neigh;
+ u64 bytes, packets, lastuse = 0;
+ struct mlx5e_tc_flow *flow;
+ struct mlx5e_encap_entry *e;
+ struct mlx5_fc *counter;
+ struct neigh_table *tbl;
+ bool neigh_used = false;
+ struct neighbour *n;
+
+ if (m_neigh->family == AF_INET)
+ tbl = &arp_tbl;
+#if IS_ENABLED(CONFIG_IPV6)
+ else if (m_neigh->family == AF_INET6)
+ tbl = ipv6_stub->nd_tbl;
+#endif
+ else
+ return;
+
+ list_for_each_entry(e, &nhe->encap_list, encap_list) {
+ if (!(e->flags & MLX5_ENCAP_ENTRY_VALID))
+ continue;
+ list_for_each_entry(flow, &e->flows, encap) {
+ if (flow->flags & MLX5E_TC_FLOW_OFFLOADED) {
+ counter = mlx5_flow_rule_counter(flow->rule);
+ mlx5_fc_query_cached(counter, &bytes, &packets, &lastuse);
+ if (time_after((unsigned long)lastuse, nhe->reported_lastuse)) {
+ neigh_used = true;
+ break;
+ }
+ }
+ }
+ }
+
+ if (neigh_used) {
+ nhe->reported_lastuse = jiffies;
+
+ /* find the relevant neigh according to the cached device and
+ * dst ip pair
+ */
+ n = neigh_lookup(tbl, &m_neigh->dst_ip, m_neigh->dev);
+ if (!n) {
+ WARN(1, "The neighbour already freed\n");
+ return;
+ }
+
+ neigh_event_send(n, NULL);
+ neigh_release(n);
+ }
}
static void mlx5e_detach_encap(struct mlx5e_priv *priv,
@@ -188,22 +376,20 @@
list_del(&flow->encap);
if (list_empty(next)) {
- struct mlx5_encap_entry *e;
+ struct mlx5e_encap_entry *e;
- e = list_entry(next, struct mlx5_encap_entry, flows);
- if (e->n) {
+ e = list_entry(next, struct mlx5e_encap_entry, flows);
+ mlx5e_rep_encap_entry_detach(netdev_priv(e->out_dev), e);
+
+ if (e->flags & MLX5_ENCAP_ENTRY_VALID)
mlx5_encap_dealloc(priv->mdev, e->encap_id);
- neigh_release(e->n);
- }
+
hlist_del_rcu(&e->encap_hlist);
+ kfree(e->encap_header);
kfree(e);
}
}
-/* we get here also when setting rule to the FW failed, etc. It means that the
- * flow rule itself might not exist, but some offloading related to the actions
- * should be cleaned.
- */
static void mlx5e_tc_del_flow(struct mlx5e_priv *priv,
struct mlx5e_tc_flow *flow)
{
@@ -631,15 +817,18 @@
{
struct mlx5_core_dev *dev = priv->mdev;
struct mlx5_eswitch *esw = dev->priv.eswitch;
- struct mlx5_eswitch_rep *rep = priv->ppriv;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
+ struct mlx5_eswitch_rep *rep;
u8 min_inline;
int err;
err = __parse_cls_flower(priv, spec, f, &min_inline);
- if (!err && (flow->flags & MLX5E_TC_FLOW_ESWITCH) &&
- rep->vport != FDB_UPLINK_VPORT) {
- if (min_inline > esw->offloads.inline_mode) {
+ if (!err && (flow->flags & MLX5E_TC_FLOW_ESWITCH)) {
+ rep = rpriv->rep;
+ if (rep->vport != FDB_UPLINK_VPORT &&
+ (esw->offloads.inline_mode != MLX5_INLINE_MODE_NONE &&
+ esw->offloads.inline_mode < min_inline)) {
netdev_warn(priv->netdev,
"Flow is not offloaded due to min inline setting, required %d actual %d\n",
min_inline, esw->offloads.inline_mode);
@@ -650,29 +839,313 @@
return err;
}
-static int parse_tc_nic_actions(struct mlx5e_priv *priv, struct tcf_exts *exts,
- u32 *action, u32 *flow_tag)
+struct pedit_headers {
+ struct ethhdr eth;
+ struct iphdr ip4;
+ struct ipv6hdr ip6;
+ struct tcphdr tcp;
+ struct udphdr udp;
+};
+
+static int pedit_header_offsets[] = {
+ [TCA_PEDIT_KEY_EX_HDR_TYPE_ETH] = offsetof(struct pedit_headers, eth),
+ [TCA_PEDIT_KEY_EX_HDR_TYPE_IP4] = offsetof(struct pedit_headers, ip4),
+ [TCA_PEDIT_KEY_EX_HDR_TYPE_IP6] = offsetof(struct pedit_headers, ip6),
+ [TCA_PEDIT_KEY_EX_HDR_TYPE_TCP] = offsetof(struct pedit_headers, tcp),
+ [TCA_PEDIT_KEY_EX_HDR_TYPE_UDP] = offsetof(struct pedit_headers, udp),
+};
+
+#define pedit_header(_ph, _htype) ((void *)(_ph) + pedit_header_offsets[_htype])
+
+static int set_pedit_val(u8 hdr_type, u32 mask, u32 val, u32 offset,
+ struct pedit_headers *masks,
+ struct pedit_headers *vals)
{
+ u32 *curr_pmask, *curr_pval;
+
+ if (hdr_type >= __PEDIT_HDR_TYPE_MAX)
+ goto out_err;
+
+ curr_pmask = (u32 *)(pedit_header(masks, hdr_type) + offset);
+ curr_pval = (u32 *)(pedit_header(vals, hdr_type) + offset);
+
+ if (*curr_pmask & mask) /* disallow acting twice on the same location */
+ goto out_err;
+
+ *curr_pmask |= mask;
+ *curr_pval |= (val & mask);
+
+ return 0;
+
+out_err:
+ return -EOPNOTSUPP;
+}
+
+struct mlx5_fields {
+ u8 field;
+ u8 size;
+ u32 offset;
+};
+
+static struct mlx5_fields fields[] = {
+ {MLX5_ACTION_IN_FIELD_OUT_DMAC_47_16, 4, offsetof(struct pedit_headers, eth.h_dest[0])},
+ {MLX5_ACTION_IN_FIELD_OUT_DMAC_15_0, 2, offsetof(struct pedit_headers, eth.h_dest[4])},
+ {MLX5_ACTION_IN_FIELD_OUT_SMAC_47_16, 4, offsetof(struct pedit_headers, eth.h_source[0])},
+ {MLX5_ACTION_IN_FIELD_OUT_SMAC_15_0, 2, offsetof(struct pedit_headers, eth.h_source[4])},
+ {MLX5_ACTION_IN_FIELD_OUT_ETHERTYPE, 2, offsetof(struct pedit_headers, eth.h_proto)},
+
+ {MLX5_ACTION_IN_FIELD_OUT_IP_DSCP, 1, offsetof(struct pedit_headers, ip4.tos)},
+ {MLX5_ACTION_IN_FIELD_OUT_IP_TTL, 1, offsetof(struct pedit_headers, ip4.ttl)},
+ {MLX5_ACTION_IN_FIELD_OUT_SIPV4, 4, offsetof(struct pedit_headers, ip4.saddr)},
+ {MLX5_ACTION_IN_FIELD_OUT_DIPV4, 4, offsetof(struct pedit_headers, ip4.daddr)},
+
+ {MLX5_ACTION_IN_FIELD_OUT_SIPV6_127_96, 4, offsetof(struct pedit_headers, ip6.saddr.s6_addr32[0])},
+ {MLX5_ACTION_IN_FIELD_OUT_SIPV6_95_64, 4, offsetof(struct pedit_headers, ip6.saddr.s6_addr32[1])},
+ {MLX5_ACTION_IN_FIELD_OUT_SIPV6_63_32, 4, offsetof(struct pedit_headers, ip6.saddr.s6_addr32[2])},
+ {MLX5_ACTION_IN_FIELD_OUT_SIPV6_31_0, 4, offsetof(struct pedit_headers, ip6.saddr.s6_addr32[3])},
+ {MLX5_ACTION_IN_FIELD_OUT_DIPV6_127_96, 4, offsetof(struct pedit_headers, ip6.daddr.s6_addr32[0])},
+ {MLX5_ACTION_IN_FIELD_OUT_DIPV6_95_64, 4, offsetof(struct pedit_headers, ip6.daddr.s6_addr32[1])},
+ {MLX5_ACTION_IN_FIELD_OUT_DIPV6_63_32, 4, offsetof(struct pedit_headers, ip6.daddr.s6_addr32[2])},
+ {MLX5_ACTION_IN_FIELD_OUT_DIPV6_31_0, 4, offsetof(struct pedit_headers, ip6.daddr.s6_addr32[3])},
+
+ {MLX5_ACTION_IN_FIELD_OUT_TCP_SPORT, 2, offsetof(struct pedit_headers, tcp.source)},
+ {MLX5_ACTION_IN_FIELD_OUT_TCP_DPORT, 2, offsetof(struct pedit_headers, tcp.dest)},
+ {MLX5_ACTION_IN_FIELD_OUT_TCP_FLAGS, 1, offsetof(struct pedit_headers, tcp.ack_seq) + 5},
+
+ {MLX5_ACTION_IN_FIELD_OUT_UDP_SPORT, 2, offsetof(struct pedit_headers, udp.source)},
+ {MLX5_ACTION_IN_FIELD_OUT_UDP_DPORT, 2, offsetof(struct pedit_headers, udp.dest)},
+};
+
+/* On input attr->num_mod_hdr_actions tells how many HW actions can be parsed at
+ * max from the SW pedit action. On success, it says how many HW actions were
+ * actually parsed.
+ */
+static int offload_pedit_fields(struct pedit_headers *masks,
+ struct pedit_headers *vals,
+ struct mlx5e_tc_flow_parse_attr *parse_attr)
+{
+ struct pedit_headers *set_masks, *add_masks, *set_vals, *add_vals;
+ int i, action_size, nactions, max_actions, first, last;
+ void *s_masks_p, *a_masks_p, *vals_p;
+ u32 s_mask, a_mask, val;
+ struct mlx5_fields *f;
+ u8 cmd, field_bsize;
+ unsigned long mask;
+ void *action;
+
+ set_masks = &masks[TCA_PEDIT_KEY_EX_CMD_SET];
+ add_masks = &masks[TCA_PEDIT_KEY_EX_CMD_ADD];
+ set_vals = &vals[TCA_PEDIT_KEY_EX_CMD_SET];
+ add_vals = &vals[TCA_PEDIT_KEY_EX_CMD_ADD];
+
+ action_size = MLX5_UN_SZ_BYTES(set_action_in_add_action_in_auto);
+ action = parse_attr->mod_hdr_actions;
+ max_actions = parse_attr->num_mod_hdr_actions;
+ nactions = 0;
+
+ for (i = 0; i < ARRAY_SIZE(fields); i++) {
+ f = &fields[i];
+ /* avoid seeing bits set from previous iterations */
+ s_mask = a_mask = mask = val = 0;
+
+ s_masks_p = (void *)set_masks + f->offset;
+ a_masks_p = (void *)add_masks + f->offset;
+
+ memcpy(&s_mask, s_masks_p, f->size);
+ memcpy(&a_mask, a_masks_p, f->size);
+
+ if (!s_mask && !a_mask) /* nothing to offload here */
+ continue;
+
+ if (s_mask && a_mask) {
+ printk(KERN_WARNING "mlx5: can't set and add to the same HW field (%x)\n", f->field);
+ return -EOPNOTSUPP;
+ }
+
+ if (nactions == max_actions) {
+ printk(KERN_WARNING "mlx5: parsed %d pedit actions, can't do more\n", nactions);
+ return -EOPNOTSUPP;
+ }
+
+ if (s_mask) {
+ cmd = MLX5_ACTION_TYPE_SET;
+ mask = s_mask;
+ vals_p = (void *)set_vals + f->offset;
+ /* clear to denote we consumed this field */
+ memset(s_masks_p, 0, f->size);
+ } else {
+ cmd = MLX5_ACTION_TYPE_ADD;
+ mask = a_mask;
+ vals_p = (void *)add_vals + f->offset;
+ /* clear to denote we consumed this field */
+ memset(a_masks_p, 0, f->size);
+ }
+
+ memcpy(&val, vals_p, f->size);
+
+ field_bsize = f->size * BITS_PER_BYTE;
+ first = find_first_bit(&mask, field_bsize);
+ last = find_last_bit(&mask, field_bsize);
+ if (first > 0 || last != (field_bsize - 1)) {
+ printk(KERN_WARNING "mlx5: partial rewrite (mask %lx) is currently not offloaded\n",
+ mask);
+ return -EOPNOTSUPP;
+ }
+
+ MLX5_SET(set_action_in, action, action_type, cmd);
+ MLX5_SET(set_action_in, action, field, f->field);
+
+ if (cmd == MLX5_ACTION_TYPE_SET) {
+ MLX5_SET(set_action_in, action, offset, 0);
+ /* length is num of bits to be written, zero means length of 32 */
+ MLX5_SET(set_action_in, action, length, field_bsize);
+ }
+
+ if (field_bsize == 32)
+ MLX5_SET(set_action_in, action, data, ntohl(val));
+ else if (field_bsize == 16)
+ MLX5_SET(set_action_in, action, data, ntohs(val));
+ else if (field_bsize == 8)
+ MLX5_SET(set_action_in, action, data, val);
+
+ action += action_size;
+ nactions++;
+ }
+
+ parse_attr->num_mod_hdr_actions = nactions;
+ return 0;
+}
+
+static int alloc_mod_hdr_actions(struct mlx5e_priv *priv,
+ const struct tc_action *a, int namespace,
+ struct mlx5e_tc_flow_parse_attr *parse_attr)
+{
+ int nkeys, action_size, max_actions;
+
+ nkeys = tcf_pedit_nkeys(a);
+ action_size = MLX5_UN_SZ_BYTES(set_action_in_add_action_in_auto);
+
+ if (namespace == MLX5_FLOW_NAMESPACE_FDB) /* FDB offloading */
+ max_actions = MLX5_CAP_ESW_FLOWTABLE_FDB(priv->mdev, max_modify_header_actions);
+ else /* namespace is MLX5_FLOW_NAMESPACE_KERNEL - NIC offloading */
+ max_actions = MLX5_CAP_FLOWTABLE_NIC_RX(priv->mdev, max_modify_header_actions);
+
+ /* can get up to crazingly 16 HW actions in 32 bits pedit SW key */
+ max_actions = min(max_actions, nkeys * 16);
+
+ parse_attr->mod_hdr_actions = kcalloc(max_actions, action_size, GFP_KERNEL);
+ if (!parse_attr->mod_hdr_actions)
+ return -ENOMEM;
+
+ parse_attr->num_mod_hdr_actions = max_actions;
+ return 0;
+}
+
+static const struct pedit_headers zero_masks = {};
+
+static int parse_tc_pedit_action(struct mlx5e_priv *priv,
+ const struct tc_action *a, int namespace,
+ struct mlx5e_tc_flow_parse_attr *parse_attr)
+{
+ struct pedit_headers masks[__PEDIT_CMD_MAX], vals[__PEDIT_CMD_MAX], *cmd_masks;
+ int nkeys, i, err = -EOPNOTSUPP;
+ u32 mask, val, offset;
+ u8 cmd, htype;
+
+ nkeys = tcf_pedit_nkeys(a);
+
+ memset(masks, 0, sizeof(struct pedit_headers) * __PEDIT_CMD_MAX);
+ memset(vals, 0, sizeof(struct pedit_headers) * __PEDIT_CMD_MAX);
+
+ for (i = 0; i < nkeys; i++) {
+ htype = tcf_pedit_htype(a, i);
+ cmd = tcf_pedit_cmd(a, i);
+ err = -EOPNOTSUPP; /* can't be all optimistic */
+
+ if (htype == TCA_PEDIT_KEY_EX_HDR_TYPE_NETWORK) {
+ printk(KERN_WARNING "mlx5: legacy pedit isn't offloaded\n");
+ goto out_err;
+ }
+
+ if (cmd != TCA_PEDIT_KEY_EX_CMD_SET && cmd != TCA_PEDIT_KEY_EX_CMD_ADD) {
+ printk(KERN_WARNING "mlx5: pedit cmd %d isn't offloaded\n", cmd);
+ goto out_err;
+ }
+
+ mask = tcf_pedit_mask(a, i);
+ val = tcf_pedit_val(a, i);
+ offset = tcf_pedit_offset(a, i);
+
+ err = set_pedit_val(htype, ~mask, val, offset, &masks[cmd], &vals[cmd]);
+ if (err)
+ goto out_err;
+ }
+
+ err = alloc_mod_hdr_actions(priv, a, namespace, parse_attr);
+ if (err)
+ goto out_err;
+
+ err = offload_pedit_fields(masks, vals, parse_attr);
+ if (err < 0)
+ goto out_dealloc_parsed_actions;
+
+ for (cmd = 0; cmd < __PEDIT_CMD_MAX; cmd++) {
+ cmd_masks = &masks[cmd];
+ if (memcmp(cmd_masks, &zero_masks, sizeof(zero_masks))) {
+ printk(KERN_WARNING "mlx5: attempt to offload an unsupported field (cmd %d)\n",
+ cmd);
+ print_hex_dump(KERN_WARNING, "mask: ", DUMP_PREFIX_ADDRESS,
+ 16, 1, cmd_masks, sizeof(zero_masks), true);
+ err = -EOPNOTSUPP;
+ goto out_dealloc_parsed_actions;
+ }
+ }
+
+ return 0;
+
+out_dealloc_parsed_actions:
+ kfree(parse_attr->mod_hdr_actions);
+out_err:
+ return err;
+}
+
+static int parse_tc_nic_actions(struct mlx5e_priv *priv, struct tcf_exts *exts,
+ struct mlx5e_tc_flow_parse_attr *parse_attr,
+ struct mlx5e_tc_flow *flow)
+{
+ struct mlx5_nic_flow_attr *attr = flow->nic_attr;
const struct tc_action *a;
LIST_HEAD(actions);
+ int err;
if (tc_no_actions(exts))
return -EINVAL;
- *flow_tag = MLX5_FS_DEFAULT_FLOW_TAG;
- *action = 0;
+ attr->flow_tag = MLX5_FS_DEFAULT_FLOW_TAG;
+ attr->action = 0;
tcf_exts_to_list(exts, &actions);
list_for_each_entry(a, &actions, list) {
/* Only support a single action per rule */
- if (*action)
+ if (attr->action)
return -EINVAL;
if (is_tcf_gact_shot(a)) {
- *action |= MLX5_FLOW_CONTEXT_ACTION_DROP;
+ attr->action |= MLX5_FLOW_CONTEXT_ACTION_DROP;
if (MLX5_CAP_FLOWTABLE(priv->mdev,
flow_table_properties_nic_receive.flow_counter))
- *action |= MLX5_FLOW_CONTEXT_ACTION_COUNT;
+ attr->action |= MLX5_FLOW_CONTEXT_ACTION_COUNT;
+ continue;
+ }
+
+ if (is_tcf_pedit(a)) {
+ err = parse_tc_pedit_action(priv, a, MLX5_FLOW_NAMESPACE_KERNEL,
+ parse_attr);
+ if (err)
+ return err;
+
+ attr->action |= MLX5_FLOW_CONTEXT_ACTION_MOD_HDR |
+ MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
continue;
}
@@ -685,8 +1158,8 @@
return -EINVAL;
}
- *flow_tag = mark;
- *action |= MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
+ attr->flow_tag = mark;
+ attr->action |= MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
continue;
}
@@ -785,16 +1258,15 @@
return 0;
}
-static int gen_vxlan_header_ipv4(struct net_device *out_dev,
- char buf[],
- unsigned char h_dest[ETH_ALEN],
- int ttl,
- __be32 daddr,
- __be32 saddr,
- __be16 udp_dst_port,
- __be32 vx_vni)
+static void gen_vxlan_header_ipv4(struct net_device *out_dev,
+ char buf[], int encap_size,
+ unsigned char h_dest[ETH_ALEN],
+ int ttl,
+ __be32 daddr,
+ __be32 saddr,
+ __be16 udp_dst_port,
+ __be32 vx_vni)
{
- int encap_size = VXLAN_HLEN + sizeof(struct iphdr) + ETH_HLEN;
struct ethhdr *eth = (struct ethhdr *)buf;
struct iphdr *ip = (struct iphdr *)((char *)eth + sizeof(struct ethhdr));
struct udphdr *udp = (struct udphdr *)((char *)ip + sizeof(struct iphdr));
@@ -817,20 +1289,17 @@
udp->dest = udp_dst_port;
vxh->vx_flags = VXLAN_HF_VNI;
vxh->vx_vni = vxlan_vni_field(vx_vni);
-
- return encap_size;
}
-static int gen_vxlan_header_ipv6(struct net_device *out_dev,
- char buf[],
- unsigned char h_dest[ETH_ALEN],
- int ttl,
- struct in6_addr *daddr,
- struct in6_addr *saddr,
- __be16 udp_dst_port,
- __be32 vx_vni)
+static void gen_vxlan_header_ipv6(struct net_device *out_dev,
+ char buf[], int encap_size,
+ unsigned char h_dest[ETH_ALEN],
+ int ttl,
+ struct in6_addr *daddr,
+ struct in6_addr *saddr,
+ __be16 udp_dst_port,
+ __be32 vx_vni)
{
- int encap_size = VXLAN_HLEN + sizeof(struct ipv6hdr) + ETH_HLEN;
struct ethhdr *eth = (struct ethhdr *)buf;
struct ipv6hdr *ip6h = (struct ipv6hdr *)((char *)eth + sizeof(struct ethhdr));
struct udphdr *udp = (struct udphdr *)((char *)ip6h + sizeof(struct ipv6hdr));
@@ -852,23 +1321,29 @@
udp->dest = udp_dst_port;
vxh->vx_flags = VXLAN_HF_VNI;
vxh->vx_vni = vxlan_vni_field(vx_vni);
-
- return encap_size;
}
static int mlx5e_create_encap_header_ipv4(struct mlx5e_priv *priv,
struct net_device *mirred_dev,
- struct mlx5_encap_entry *e,
- struct net_device **out_dev)
+ struct mlx5e_encap_entry *e)
{
int max_encap_size = MLX5_CAP_ESW(priv->mdev, max_encap_header_size);
+ int ipv4_encap_size = ETH_HLEN + sizeof(struct iphdr) + VXLAN_HLEN;
struct ip_tunnel_key *tun_key = &e->tun_info.key;
- int encap_size, ttl, err;
+ struct net_device *out_dev;
struct neighbour *n = NULL;
struct flowi4 fl4 = {};
char *encap_header;
+ int ttl, err;
+ u8 nud_state;
- encap_header = kzalloc(max_encap_size, GFP_KERNEL);
+ if (max_encap_size < ipv4_encap_size) {
+ mlx5_core_warn(priv->mdev, "encap size %d too big, max supported is %d\n",
+ ipv4_encap_size, max_encap_size);
+ return -EOPNOTSUPP;
+ }
+
+ encap_header = kzalloc(ipv4_encap_size, GFP_KERNEL);
if (!encap_header)
return -ENOMEM;
@@ -885,58 +1360,94 @@
fl4.daddr = tun_key->u.ipv4.dst;
fl4.saddr = tun_key->u.ipv4.src;
- err = mlx5e_route_lookup_ipv4(priv, mirred_dev, out_dev,
+ err = mlx5e_route_lookup_ipv4(priv, mirred_dev, &out_dev,
&fl4, &n, &ttl);
if (err)
goto out;
- if (!(n->nud_state & NUD_VALID)) {
- pr_warn("%s: can't offload, neighbour to %pI4 invalid\n", __func__, &fl4.daddr);
- err = -EOPNOTSUPP;
+ /* used by mlx5e_detach_encap to lookup a neigh hash table
+ * entry in the neigh hash table when a user deletes a rule
+ */
+ e->m_neigh.dev = n->dev;
+ e->m_neigh.family = n->ops->family;
+ memcpy(&e->m_neigh.dst_ip, n->primary_key, n->tbl->key_len);
+ e->out_dev = out_dev;
+
+ /* It's importent to add the neigh to the hash table before checking
+ * the neigh validity state. So if we'll get a notification, in case the
+ * neigh changes it's validity state, we would find the relevant neigh
+ * in the hash.
+ */
+ err = mlx5e_rep_encap_entry_attach(netdev_priv(out_dev), e);
+ if (err)
goto out;
- }
- e->n = n;
- e->out_dev = *out_dev;
-
- neigh_ha_snapshot(e->h_dest, n, *out_dev);
+ read_lock_bh(&n->lock);
+ nud_state = n->nud_state;
+ ether_addr_copy(e->h_dest, n->ha);
+ read_unlock_bh(&n->lock);
switch (e->tunnel_type) {
case MLX5_HEADER_TYPE_VXLAN:
- encap_size = gen_vxlan_header_ipv4(*out_dev, encap_header,
- e->h_dest, ttl,
- fl4.daddr,
- fl4.saddr, tun_key->tp_dst,
- tunnel_id_to_key32(tun_key->tun_id));
+ gen_vxlan_header_ipv4(out_dev, encap_header,
+ ipv4_encap_size, e->h_dest, ttl,
+ fl4.daddr,
+ fl4.saddr, tun_key->tp_dst,
+ tunnel_id_to_key32(tun_key->tun_id));
break;
default:
err = -EOPNOTSUPP;
- goto out;
+ goto destroy_neigh_entry;
+ }
+ e->encap_size = ipv4_encap_size;
+ e->encap_header = encap_header;
+
+ if (!(nud_state & NUD_VALID)) {
+ neigh_event_send(n, NULL);
+ neigh_release(n);
+ return -EAGAIN;
}
err = mlx5_encap_alloc(priv->mdev, e->tunnel_type,
- encap_size, encap_header, &e->encap_id);
+ ipv4_encap_size, encap_header, &e->encap_id);
+ if (err)
+ goto destroy_neigh_entry;
+
+ e->flags |= MLX5_ENCAP_ENTRY_VALID;
+ mlx5e_rep_queue_neigh_stats_work(netdev_priv(out_dev));
+ neigh_release(n);
+ return err;
+
+destroy_neigh_entry:
+ mlx5e_rep_encap_entry_detach(netdev_priv(e->out_dev), e);
out:
- if (err && n)
- neigh_release(n);
kfree(encap_header);
+ if (n)
+ neigh_release(n);
return err;
}
static int mlx5e_create_encap_header_ipv6(struct mlx5e_priv *priv,
struct net_device *mirred_dev,
- struct mlx5_encap_entry *e,
- struct net_device **out_dev)
-
+ struct mlx5e_encap_entry *e)
{
int max_encap_size = MLX5_CAP_ESW(priv->mdev, max_encap_header_size);
+ int ipv6_encap_size = ETH_HLEN + sizeof(struct ipv6hdr) + VXLAN_HLEN;
struct ip_tunnel_key *tun_key = &e->tun_info.key;
- int encap_size, err, ttl = 0;
+ struct net_device *out_dev;
struct neighbour *n = NULL;
struct flowi6 fl6 = {};
char *encap_header;
+ int err, ttl = 0;
+ u8 nud_state;
- encap_header = kzalloc(max_encap_size, GFP_KERNEL);
+ if (max_encap_size < ipv6_encap_size) {
+ mlx5_core_warn(priv->mdev, "encap size %d too big, max supported is %d\n",
+ ipv6_encap_size, max_encap_size);
+ return -EOPNOTSUPP;
+ }
+
+ encap_header = kzalloc(ipv6_encap_size, GFP_KERNEL);
if (!encap_header)
return -ENOMEM;
@@ -954,57 +1465,88 @@
fl6.daddr = tun_key->u.ipv6.dst;
fl6.saddr = tun_key->u.ipv6.src;
- err = mlx5e_route_lookup_ipv6(priv, mirred_dev, out_dev,
+ err = mlx5e_route_lookup_ipv6(priv, mirred_dev, &out_dev,
&fl6, &n, &ttl);
if (err)
goto out;
- if (!(n->nud_state & NUD_VALID)) {
- pr_warn("%s: can't offload, neighbour to %pI6 invalid\n", __func__, &fl6.daddr);
- err = -EOPNOTSUPP;
+ /* used by mlx5e_detach_encap to lookup a neigh hash table
+ * entry in the neigh hash table when a user deletes a rule
+ */
+ e->m_neigh.dev = n->dev;
+ e->m_neigh.family = n->ops->family;
+ memcpy(&e->m_neigh.dst_ip, n->primary_key, n->tbl->key_len);
+ e->out_dev = out_dev;
+
+ /* It's importent to add the neigh to the hash table before checking
+ * the neigh validity state. So if we'll get a notification, in case the
+ * neigh changes it's validity state, we would find the relevant neigh
+ * in the hash.
+ */
+ err = mlx5e_rep_encap_entry_attach(netdev_priv(out_dev), e);
+ if (err)
goto out;
- }
- e->n = n;
- e->out_dev = *out_dev;
-
- neigh_ha_snapshot(e->h_dest, n, *out_dev);
+ read_lock_bh(&n->lock);
+ nud_state = n->nud_state;
+ ether_addr_copy(e->h_dest, n->ha);
+ read_unlock_bh(&n->lock);
switch (e->tunnel_type) {
case MLX5_HEADER_TYPE_VXLAN:
- encap_size = gen_vxlan_header_ipv6(*out_dev, encap_header,
- e->h_dest, ttl,
- &fl6.daddr,
- &fl6.saddr, tun_key->tp_dst,
- tunnel_id_to_key32(tun_key->tun_id));
+ gen_vxlan_header_ipv6(out_dev, encap_header,
+ ipv6_encap_size, e->h_dest, ttl,
+ &fl6.daddr,
+ &fl6.saddr, tun_key->tp_dst,
+ tunnel_id_to_key32(tun_key->tun_id));
break;
default:
err = -EOPNOTSUPP;
- goto out;
+ goto destroy_neigh_entry;
+ }
+
+ e->encap_size = ipv6_encap_size;
+ e->encap_header = encap_header;
+
+ if (!(nud_state & NUD_VALID)) {
+ neigh_event_send(n, NULL);
+ neigh_release(n);
+ return -EAGAIN;
}
err = mlx5_encap_alloc(priv->mdev, e->tunnel_type,
- encap_size, encap_header, &e->encap_id);
+ ipv6_encap_size, encap_header, &e->encap_id);
+ if (err)
+ goto destroy_neigh_entry;
+
+ e->flags |= MLX5_ENCAP_ENTRY_VALID;
+ mlx5e_rep_queue_neigh_stats_work(netdev_priv(out_dev));
+ neigh_release(n);
+ return err;
+
+destroy_neigh_entry:
+ mlx5e_rep_encap_entry_detach(netdev_priv(e->out_dev), e);
out:
- if (err && n)
- neigh_release(n);
kfree(encap_header);
+ if (n)
+ neigh_release(n);
return err;
}
static int mlx5e_attach_encap(struct mlx5e_priv *priv,
struct ip_tunnel_info *tun_info,
struct net_device *mirred_dev,
- struct mlx5_esw_flow_attr *attr)
+ struct net_device **encap_dev,
+ struct mlx5e_tc_flow *flow)
{
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
struct net_device *up_dev = mlx5_eswitch_get_uplink_netdev(esw);
- struct mlx5e_priv *up_priv = netdev_priv(up_dev);
unsigned short family = ip_tunnel_info_af(tun_info);
+ struct mlx5e_priv *up_priv = netdev_priv(up_dev);
+ struct mlx5_esw_flow_attr *attr = flow->esw_attr;
struct ip_tunnel_key *key = &tun_info->key;
- struct mlx5_encap_entry *e;
- struct net_device *out_dev;
- int tunnel_type, err = -EOPNOTSUPP;
+ struct mlx5e_encap_entry *e;
+ int tunnel_type, err = 0;
uintptr_t hash_key;
bool found = false;
@@ -1039,10 +1581,8 @@
}
}
- if (found) {
- attr->encap = e;
- return 0;
- }
+ if (found)
+ goto attach_flow;
e = kzalloc(sizeof(*e), GFP_KERNEL);
if (!e)
@@ -1053,16 +1593,21 @@
INIT_LIST_HEAD(&e->flows);
if (family == AF_INET)
- err = mlx5e_create_encap_header_ipv4(priv, mirred_dev, e, &out_dev);
+ err = mlx5e_create_encap_header_ipv4(priv, mirred_dev, e);
else if (family == AF_INET6)
- err = mlx5e_create_encap_header_ipv6(priv, mirred_dev, e, &out_dev);
+ err = mlx5e_create_encap_header_ipv6(priv, mirred_dev, e);
- if (err)
+ if (err && err != -EAGAIN)
goto out_err;
- attr->encap = e;
hash_add_rcu(esw->offloads.encap_tbl, &e->encap_hlist, hash_key);
+attach_flow:
+ list_add(&flow->encap, &e->flows);
+ *encap_dev = e->out_dev;
+ if (e->flags & MLX5_ENCAP_ENTRY_VALID)
+ attr->encap_id = e->encap_id;
+
return err;
out_err:
@@ -1071,20 +1616,22 @@
}
static int parse_tc_fdb_actions(struct mlx5e_priv *priv, struct tcf_exts *exts,
+ struct mlx5e_tc_flow_parse_attr *parse_attr,
struct mlx5e_tc_flow *flow)
{
- struct mlx5_esw_flow_attr *attr = flow->attr;
+ struct mlx5_esw_flow_attr *attr = flow->esw_attr;
+ struct mlx5e_rep_priv *rpriv = priv->ppriv;
struct ip_tunnel_info *info = NULL;
const struct tc_action *a;
LIST_HEAD(actions);
bool encap = false;
- int err;
+ int err = 0;
if (tc_no_actions(exts))
return -EINVAL;
memset(attr, 0, sizeof(*attr));
- attr->in_rep = priv->ppriv;
+ attr->in_rep = rpriv->rep;
tcf_exts_to_list(exts, &actions);
list_for_each_entry(a, &actions, list) {
@@ -1094,9 +1641,19 @@
continue;
}
+ if (is_tcf_pedit(a)) {
+ err = parse_tc_pedit_action(priv, a, MLX5_FLOW_NAMESPACE_FDB,
+ parse_attr);
+ if (err)
+ return err;
+
+ attr->action |= MLX5_FLOW_CONTEXT_ACTION_MOD_HDR;
+ continue;
+ }
+
if (is_tcf_mirred_egress_redirect(a)) {
int ifindex = tcf_mirred_ifindex(a);
- struct net_device *out_dev;
+ struct net_device *out_dev, *encap_dev = NULL;
struct mlx5e_priv *out_priv;
out_dev = __dev_get_by_index(dev_net(priv->netdev), ifindex);
@@ -1106,18 +1663,20 @@
attr->action |= MLX5_FLOW_CONTEXT_ACTION_FWD_DEST |
MLX5_FLOW_CONTEXT_ACTION_COUNT;
out_priv = netdev_priv(out_dev);
- attr->out_rep = out_priv->ppriv;
+ rpriv = out_priv->ppriv;
+ attr->out_rep = rpriv->rep;
} else if (encap) {
err = mlx5e_attach_encap(priv, info,
- out_dev, attr);
- if (err)
+ out_dev, &encap_dev, flow);
+ if (err && err != -EAGAIN)
return err;
- list_add(&flow->encap, &attr->encap->flows);
attr->action |= MLX5_FLOW_CONTEXT_ACTION_ENCAP |
MLX5_FLOW_CONTEXT_ACTION_FWD_DEST |
MLX5_FLOW_CONTEXT_ACTION_COUNT;
- out_priv = netdev_priv(attr->encap->out_dev);
- attr->out_rep = out_priv->ppriv;
+ out_priv = netdev_priv(encap_dev);
+ rpriv = out_priv->ppriv;
+ attr->out_rep = rpriv->rep;
+ attr->parse_attr = parse_attr;
} else {
pr_err("devices %s %s not on same switch HW, can't offload forwarding\n",
priv->netdev->name, out_dev->name);
@@ -1157,28 +1716,30 @@
return -EINVAL;
}
- return 0;
+ return err;
}
int mlx5e_configure_flower(struct mlx5e_priv *priv, __be16 protocol,
struct tc_cls_flower_offload *f)
{
- struct mlx5e_tc_table *tc = &priv->fs.tc;
- int err, attr_size = 0;
- u32 flow_tag, action;
- struct mlx5e_tc_flow *flow;
- struct mlx5_flow_spec *spec;
struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
+ struct mlx5e_tc_flow_parse_attr *parse_attr;
+ struct mlx5e_tc_table *tc = &priv->fs.tc;
+ struct mlx5e_tc_flow *flow;
+ int attr_size, err = 0;
u8 flow_flags = 0;
if (esw && esw->mode == SRIOV_OFFLOADS) {
flow_flags = MLX5E_TC_FLOW_ESWITCH;
attr_size = sizeof(struct mlx5_esw_flow_attr);
+ } else {
+ flow_flags = MLX5E_TC_FLOW_NIC;
+ attr_size = sizeof(struct mlx5_nic_flow_attr);
}
flow = kzalloc(sizeof(*flow) + attr_size, GFP_KERNEL);
- spec = mlx5_vzalloc(sizeof(*spec));
- if (!spec || !flow) {
+ parse_attr = mlx5_vzalloc(sizeof(*parse_attr));
+ if (!parse_attr || !flow) {
err = -ENOMEM;
goto err_free;
}
@@ -1186,42 +1747,54 @@
flow->cookie = f->cookie;
flow->flags = flow_flags;
- err = parse_cls_flower(priv, flow, spec, f);
+ err = parse_cls_flower(priv, flow, &parse_attr->spec, f);
if (err < 0)
goto err_free;
if (flow->flags & MLX5E_TC_FLOW_ESWITCH) {
- flow->attr = (struct mlx5_esw_flow_attr *)(flow + 1);
- err = parse_tc_fdb_actions(priv, f->exts, flow);
+ err = parse_tc_fdb_actions(priv, f->exts, parse_attr, flow);
if (err < 0)
- goto err_free;
- flow->rule = mlx5e_tc_add_fdb_flow(priv, spec, flow->attr);
+ goto err_handle_encap_flow;
+ flow->rule = mlx5e_tc_add_fdb_flow(priv, parse_attr, flow);
} else {
- err = parse_tc_nic_actions(priv, f->exts, &action, &flow_tag);
+ err = parse_tc_nic_actions(priv, f->exts, parse_attr, flow);
if (err < 0)
goto err_free;
- flow->rule = mlx5e_tc_add_nic_flow(priv, spec, action, flow_tag);
+ flow->rule = mlx5e_tc_add_nic_flow(priv, parse_attr, flow);
}
if (IS_ERR(flow->rule)) {
err = PTR_ERR(flow->rule);
- goto err_del_rule;
+ goto err_free;
}
+ flow->flags |= MLX5E_TC_FLOW_OFFLOADED;
err = rhashtable_insert_fast(&tc->ht, &flow->node,
tc->ht_params);
if (err)
goto err_del_rule;
- goto out;
+ if (flow->flags & MLX5E_TC_FLOW_ESWITCH &&
+ !(flow->esw_attr->action & MLX5_FLOW_CONTEXT_ACTION_ENCAP))
+ kvfree(parse_attr);
+ return err;
err_del_rule:
mlx5e_tc_del_flow(priv, flow);
+err_handle_encap_flow:
+ if (err == -EAGAIN) {
+ err = rhashtable_insert_fast(&tc->ht, &flow->node,
+ tc->ht_params);
+ if (err)
+ mlx5e_tc_del_flow(priv, flow);
+ else
+ return 0;
+ }
+
err_free:
+ kvfree(parse_attr);
kfree(flow);
-out:
- kvfree(spec);
return err;
}
@@ -1240,7 +1813,6 @@
mlx5e_tc_del_flow(priv, flow);
-
kfree(flow);
return 0;
@@ -1263,6 +1835,9 @@
if (!flow)
return -EINVAL;
+ if (!(flow->flags & MLX5E_TC_FLOW_OFFLOADED))
+ return 0;
+
counter = mlx5_flow_rule_counter(flow->rule);
if (!counter)
return 0;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.h b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.h
index 34bf903..ecbe30d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.h
@@ -46,6 +46,15 @@
int mlx5e_stats_flower(struct mlx5e_priv *priv,
struct tc_cls_flower_offload *f);
+struct mlx5e_encap_entry;
+void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv,
+ struct mlx5e_encap_entry *e);
+void mlx5e_tc_encap_flows_del(struct mlx5e_priv *priv,
+ struct mlx5e_encap_entry *e);
+
+struct mlx5e_neigh_hash_entry;
+void mlx5e_tc_update_neigh_used_value(struct mlx5e_neigh_hash_entry *nhe);
+
static inline int mlx5e_tc_num_filters(struct mlx5e_priv *priv)
{
return atomic_read(&priv->fs.tc.ht.nelems);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index 57f5e2d..ab3bb02 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -33,34 +33,12 @@
#include <linux/tcp.h>
#include <linux/if_vlan.h>
#include "en.h"
+#include "ipoib.h"
#define MLX5E_SQ_NOPS_ROOM MLX5_SEND_WQE_MAX_WQEBBS
#define MLX5E_SQ_STOP_ROOM (MLX5_SEND_WQE_MAX_WQEBBS +\
MLX5E_SQ_NOPS_ROOM)
-void mlx5e_send_nop(struct mlx5e_sq *sq, bool notify_hw)
-{
- struct mlx5_wq_cyc *wq = &sq->wq;
-
- u16 pi = sq->pc & wq->sz_m1;
- struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(wq, pi);
-
- struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
-
- memset(cseg, 0, sizeof(*cseg));
-
- cseg->opmod_idx_opcode = cpu_to_be32((sq->pc << 8) | MLX5_OPCODE_NOP);
- cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | 0x01);
-
- sq->pc++;
- sq->stats.nop++;
-
- if (notify_hw) {
- cseg->fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE;
- mlx5e_tx_notify_hw(sq, &wqe->ctrl, 0);
- }
-}
-
static inline void mlx5e_tx_dma_unmap(struct device *pdev,
struct mlx5e_sq_dma *dma)
{
@@ -76,25 +54,25 @@
}
}
-static inline void mlx5e_dma_push(struct mlx5e_sq *sq,
+static inline void mlx5e_dma_push(struct mlx5e_txqsq *sq,
dma_addr_t addr,
u32 size,
enum mlx5e_dma_map_type map_type)
{
u32 i = sq->dma_fifo_pc & sq->dma_fifo_mask;
- sq->db.txq.dma_fifo[i].addr = addr;
- sq->db.txq.dma_fifo[i].size = size;
- sq->db.txq.dma_fifo[i].type = map_type;
+ sq->db.dma_fifo[i].addr = addr;
+ sq->db.dma_fifo[i].size = size;
+ sq->db.dma_fifo[i].type = map_type;
sq->dma_fifo_pc++;
}
-static inline struct mlx5e_sq_dma *mlx5e_dma_get(struct mlx5e_sq *sq, u32 i)
+static inline struct mlx5e_sq_dma *mlx5e_dma_get(struct mlx5e_txqsq *sq, u32 i)
{
- return &sq->db.txq.dma_fifo[i & sq->dma_fifo_mask];
+ return &sq->db.dma_fifo[i & sq->dma_fifo_mask];
}
-static void mlx5e_dma_unmap_wqe_err(struct mlx5e_sq *sq, u8 num_dma)
+static void mlx5e_dma_unmap_wqe_err(struct mlx5e_txqsq *sq, u8 num_dma)
{
int i;
@@ -111,6 +89,7 @@
{
struct mlx5e_priv *priv = netdev_priv(dev);
int channel_ix = fallback(dev, skb);
+ u16 num_channels;
int up = 0;
if (!netdev_get_num_tc(dev))
@@ -122,11 +101,11 @@
/* channel_ix can be larger than num_channels since
* dev->num_real_tx_queues = num_channels * num_tc
*/
- if (channel_ix >= priv->params.num_channels)
- channel_ix = reciprocal_scale(channel_ix,
- priv->params.num_channels);
+ num_channels = priv->channels.params.num_channels;
+ if (channel_ix >= num_channels)
+ channel_ix = reciprocal_scale(channel_ix, num_channels);
- return priv->channeltc_to_txq_map[channel_ix][up];
+ return priv->channel_tc2txq[channel_ix][up];
}
static inline int mlx5e_skb_l2_header_offset(struct sk_buff *skb)
@@ -175,25 +154,6 @@
}
}
-static inline u16 mlx5e_get_inline_hdr_size(struct mlx5e_sq *sq,
- struct sk_buff *skb, bool bf)
-{
- /* Some NIC TX decisions, e.g loopback, are based on the packet
- * headers and occur before the data gather.
- * Therefore these headers must be copied into the WQE
- */
- if (bf) {
- u16 ihs = skb_headlen(skb);
-
- if (skb_vlan_tag_present(skb))
- ihs += VLAN_HLEN;
-
- if (ihs <= sq->max_inline)
- return skb_headlen(skb);
- }
- return mlx5e_calc_min_inline(sq->min_inline_mode, skb);
-}
-
static inline void mlx5e_tx_skb_pull_inline(unsigned char **skb_data,
unsigned int *skb_len,
unsigned int len)
@@ -218,31 +178,9 @@
mlx5e_tx_skb_pull_inline(skb_data, skb_len, cpy2_sz);
}
-static netdev_tx_t mlx5e_sq_xmit(struct mlx5e_sq *sq, struct sk_buff *skb)
+static inline void
+mlx5e_txwqe_build_eseg_csum(struct mlx5e_txqsq *sq, struct sk_buff *skb, struct mlx5_wqe_eth_seg *eseg)
{
- struct mlx5_wq_cyc *wq = &sq->wq;
-
- u16 pi = sq->pc & wq->sz_m1;
- struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(wq, pi);
- struct mlx5e_tx_wqe_info *wi = &sq->db.txq.wqe_info[pi];
-
- struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
- struct mlx5_wqe_eth_seg *eseg = &wqe->eth;
- struct mlx5_wqe_data_seg *dseg;
-
- unsigned char *skb_data = skb->data;
- unsigned int skb_len = skb->len;
- u8 opcode = MLX5_OPCODE_SEND;
- dma_addr_t dma_addr = 0;
- unsigned int num_bytes;
- bool bf = false;
- u16 headlen;
- u16 ds_cnt;
- u16 ihs;
- int i;
-
- memset(wqe, 0, sizeof(*wqe));
-
if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM;
if (skb->encapsulation) {
@@ -254,39 +192,148 @@
}
} else
sq->stats.csum_none++;
+}
- if (sq->cc != sq->prev_cc) {
- sq->prev_cc = sq->cc;
- sq->bf_budget = (sq->cc == sq->pc) ? MLX5E_SQ_BF_BUDGET : 0;
+static inline u16
+mlx5e_txwqe_build_eseg_gso(struct mlx5e_txqsq *sq, struct sk_buff *skb,
+ struct mlx5_wqe_eth_seg *eseg, unsigned int *num_bytes)
+{
+ u16 ihs;
+
+ eseg->mss = cpu_to_be16(skb_shinfo(skb)->gso_size);
+
+ if (skb->encapsulation) {
+ ihs = skb_inner_transport_offset(skb) + inner_tcp_hdrlen(skb);
+ sq->stats.tso_inner_packets++;
+ sq->stats.tso_inner_bytes += skb->len - ihs;
+ } else {
+ ihs = skb_transport_offset(skb) + tcp_hdrlen(skb);
+ sq->stats.tso_packets++;
+ sq->stats.tso_bytes += skb->len - ihs;
}
+ *num_bytes = skb->len + (skb_shinfo(skb)->gso_segs - 1) * ihs;
+ return ihs;
+}
+
+static inline int
+mlx5e_txwqe_build_dsegs(struct mlx5e_txqsq *sq, struct sk_buff *skb,
+ unsigned char *skb_data, u16 headlen,
+ struct mlx5_wqe_data_seg *dseg)
+{
+ dma_addr_t dma_addr = 0;
+ u8 num_dma = 0;
+ int i;
+
+ if (headlen) {
+ dma_addr = dma_map_single(sq->pdev, skb_data, headlen,
+ DMA_TO_DEVICE);
+ if (unlikely(dma_mapping_error(sq->pdev, dma_addr)))
+ return -ENOMEM;
+
+ dseg->addr = cpu_to_be64(dma_addr);
+ dseg->lkey = sq->mkey_be;
+ dseg->byte_count = cpu_to_be32(headlen);
+
+ mlx5e_dma_push(sq, dma_addr, headlen, MLX5E_DMA_MAP_SINGLE);
+ num_dma++;
+ dseg++;
+ }
+
+ for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
+ struct skb_frag_struct *frag = &skb_shinfo(skb)->frags[i];
+ int fsz = skb_frag_size(frag);
+
+ dma_addr = skb_frag_dma_map(sq->pdev, frag, 0, fsz,
+ DMA_TO_DEVICE);
+ if (unlikely(dma_mapping_error(sq->pdev, dma_addr)))
+ return -ENOMEM;
+
+ dseg->addr = cpu_to_be64(dma_addr);
+ dseg->lkey = sq->mkey_be;
+ dseg->byte_count = cpu_to_be32(fsz);
+
+ mlx5e_dma_push(sq, dma_addr, fsz, MLX5E_DMA_MAP_PAGE);
+ num_dma++;
+ dseg++;
+ }
+
+ return num_dma;
+}
+
+static inline void
+mlx5e_txwqe_complete(struct mlx5e_txqsq *sq, struct sk_buff *skb,
+ u8 opcode, u16 ds_cnt, u32 num_bytes, u8 num_dma,
+ struct mlx5e_tx_wqe_info *wi, struct mlx5_wqe_ctrl_seg *cseg)
+{
+ struct mlx5_wq_cyc *wq = &sq->wq;
+ u16 pi;
+
+ wi->num_bytes = num_bytes;
+ wi->num_dma = num_dma;
+ wi->num_wqebbs = DIV_ROUND_UP(ds_cnt, MLX5_SEND_WQEBB_NUM_DS);
+ wi->skb = skb;
+
+ cseg->opmod_idx_opcode = cpu_to_be32((sq->pc << 8) | opcode);
+ cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt);
+
+ netdev_tx_sent_queue(sq->txq, num_bytes);
+
+ if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP))
+ skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
+
+ sq->pc += wi->num_wqebbs;
+ if (unlikely(!mlx5e_wqc_has_room_for(wq, sq->cc, sq->pc, MLX5E_SQ_STOP_ROOM))) {
+ netif_tx_stop_queue(sq->txq);
+ sq->stats.stopped++;
+ }
+
+ if (!skb->xmit_more || netif_xmit_stopped(sq->txq))
+ mlx5e_notify_hw(wq, sq->pc, sq->uar_map, cseg);
+
+ /* fill sq edge with nops to avoid wqe wrap around */
+ while ((pi = (sq->pc & wq->sz_m1)) > sq->edge) {
+ sq->db.wqe_info[pi].skb = NULL;
+ mlx5e_post_nop(wq, sq->sqn, &sq->pc);
+ sq->stats.nop++;
+ }
+}
+
+static netdev_tx_t mlx5e_sq_xmit(struct mlx5e_txqsq *sq, struct sk_buff *skb)
+{
+ struct mlx5_wq_cyc *wq = &sq->wq;
+
+ u16 pi = sq->pc & wq->sz_m1;
+ struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(wq, pi);
+ struct mlx5e_tx_wqe_info *wi = &sq->db.wqe_info[pi];
+
+ struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+ struct mlx5_wqe_eth_seg *eseg = &wqe->eth;
+
+ unsigned char *skb_data = skb->data;
+ unsigned int skb_len = skb->len;
+ u8 opcode = MLX5_OPCODE_SEND;
+ unsigned int num_bytes;
+ int num_dma;
+ u16 headlen;
+ u16 ds_cnt;
+ u16 ihs;
+
+ memset(wqe, 0, sizeof(*wqe));
+
+ mlx5e_txwqe_build_eseg_csum(sq, skb, eseg);
+
if (skb_is_gso(skb)) {
- eseg->mss = cpu_to_be16(skb_shinfo(skb)->gso_size);
- opcode = MLX5_OPCODE_LSO;
-
- if (skb->encapsulation) {
- ihs = skb_inner_transport_offset(skb) + inner_tcp_hdrlen(skb);
- sq->stats.tso_inner_packets++;
- sq->stats.tso_inner_bytes += skb->len - ihs;
- } else {
- ihs = skb_transport_offset(skb) + tcp_hdrlen(skb);
- sq->stats.tso_packets++;
- sq->stats.tso_bytes += skb->len - ihs;
- }
-
+ opcode = MLX5_OPCODE_LSO;
+ ihs = mlx5e_txwqe_build_eseg_gso(sq, skb, eseg, &num_bytes);
sq->stats.packets += skb_shinfo(skb)->gso_segs;
- num_bytes = skb->len + (skb_shinfo(skb)->gso_segs - 1) * ihs;
} else {
- bf = sq->bf_budget &&
- !skb->xmit_more &&
- !skb_shinfo(skb)->nr_frags;
- ihs = mlx5e_get_inline_hdr_size(sq, skb, bf);
- sq->stats.packets++;
+ ihs = mlx5e_calc_min_inline(sq->min_inline_mode, skb);
num_bytes = max_t(unsigned int, skb->len, ETH_ZLEN);
+ sq->stats.packets++;
}
-
sq->stats.bytes += num_bytes;
- wi->num_bytes = num_bytes;
+ sq->stats.xmit_more += skb->xmit_more;
ds_cnt = sizeof(*wqe) / MLX5_SEND_WQE_DS;
if (ihs) {
@@ -304,85 +351,14 @@
eseg->insert.vlan_tci = cpu_to_be16(skb_vlan_tag_get(skb));
}
- dseg = (struct mlx5_wqe_data_seg *)cseg + ds_cnt;
-
- wi->num_dma = 0;
-
headlen = skb_len - skb->data_len;
- if (headlen) {
- dma_addr = dma_map_single(sq->pdev, skb_data, headlen,
- DMA_TO_DEVICE);
- if (unlikely(dma_mapping_error(sq->pdev, dma_addr)))
- goto dma_unmap_wqe_err;
+ num_dma = mlx5e_txwqe_build_dsegs(sq, skb, skb_data, headlen,
+ (struct mlx5_wqe_data_seg *)cseg + ds_cnt);
+ if (unlikely(num_dma < 0))
+ goto dma_unmap_wqe_err;
- dseg->addr = cpu_to_be64(dma_addr);
- dseg->lkey = sq->mkey_be;
- dseg->byte_count = cpu_to_be32(headlen);
-
- mlx5e_dma_push(sq, dma_addr, headlen, MLX5E_DMA_MAP_SINGLE);
- wi->num_dma++;
-
- dseg++;
- }
-
- for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
- struct skb_frag_struct *frag = &skb_shinfo(skb)->frags[i];
- int fsz = skb_frag_size(frag);
-
- dma_addr = skb_frag_dma_map(sq->pdev, frag, 0, fsz,
- DMA_TO_DEVICE);
- if (unlikely(dma_mapping_error(sq->pdev, dma_addr)))
- goto dma_unmap_wqe_err;
-
- dseg->addr = cpu_to_be64(dma_addr);
- dseg->lkey = sq->mkey_be;
- dseg->byte_count = cpu_to_be32(fsz);
-
- mlx5e_dma_push(sq, dma_addr, fsz, MLX5E_DMA_MAP_PAGE);
- wi->num_dma++;
-
- dseg++;
- }
-
- ds_cnt += wi->num_dma;
-
- cseg->opmod_idx_opcode = cpu_to_be32((sq->pc << 8) | opcode);
- cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt);
-
- sq->db.txq.skb[pi] = skb;
-
- wi->num_wqebbs = DIV_ROUND_UP(ds_cnt, MLX5_SEND_WQEBB_NUM_DS);
- sq->pc += wi->num_wqebbs;
-
- netdev_tx_sent_queue(sq->txq, wi->num_bytes);
-
- if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP))
- skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
-
- if (unlikely(!mlx5e_sq_has_room_for(sq, MLX5E_SQ_STOP_ROOM))) {
- netif_tx_stop_queue(sq->txq);
- sq->stats.stopped++;
- }
-
- sq->stats.xmit_more += skb->xmit_more;
- if (!skb->xmit_more || netif_xmit_stopped(sq->txq)) {
- int bf_sz = 0;
-
- if (bf && test_bit(MLX5E_SQ_STATE_BF_ENABLE, &sq->state))
- bf_sz = wi->num_wqebbs << 3;
-
- cseg->fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE;
- mlx5e_tx_notify_hw(sq, &wqe->ctrl, bf_sz);
- }
-
- /* fill sq edge with nops to avoid wqe wrap around */
- while ((pi = (sq->pc & wq->sz_m1)) > sq->edge) {
- sq->db.txq.skb[pi] = NULL;
- mlx5e_send_nop(sq, false);
- }
-
- if (bf)
- sq->bf_budget--;
+ mlx5e_txwqe_complete(sq, skb, opcode, ds_cnt + num_dma,
+ num_bytes, num_dma, wi, cseg);
return NETDEV_TX_OK;
@@ -398,21 +374,21 @@
netdev_tx_t mlx5e_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct mlx5e_priv *priv = netdev_priv(dev);
- struct mlx5e_sq *sq = priv->txq_to_sq_map[skb_get_queue_mapping(skb)];
+ struct mlx5e_txqsq *sq = priv->txq2sq[skb_get_queue_mapping(skb)];
return mlx5e_sq_xmit(sq, skb);
}
bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget)
{
- struct mlx5e_sq *sq;
+ struct mlx5e_txqsq *sq;
u32 dma_fifo_cc;
u32 nbytes;
u16 npkts;
u16 sqcc;
int i;
- sq = container_of(cq, struct mlx5e_sq, cq);
+ sq = container_of(cq, struct mlx5e_txqsq, cq);
if (unlikely(!test_bit(MLX5E_SQ_STATE_ENABLED, &sq->state)))
return false;
@@ -450,8 +426,8 @@
last_wqe = (sqcc == wqe_counter);
ci = sqcc & sq->wq.sz_m1;
- skb = sq->db.txq.skb[ci];
- wi = &sq->db.txq.wqe_info[ci];
+ wi = &sq->db.wqe_info[ci];
+ skb = wi->skb;
if (unlikely(!skb)) { /* nop */
sqcc++;
@@ -492,7 +468,7 @@
netdev_tx_completed_queue(sq->txq, npkts, nbytes);
if (netif_tx_queue_stopped(sq->txq) &&
- mlx5e_sq_has_room_for(sq, MLX5E_SQ_STOP_ROOM)) {
+ mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, MLX5E_SQ_STOP_ROOM)) {
netif_tx_wake_queue(sq->txq);
sq->stats.wake++;
}
@@ -500,7 +476,7 @@
return (i == MLX5E_TX_CQ_POLL_BUDGET);
}
-static void mlx5e_free_txq_sq_descs(struct mlx5e_sq *sq)
+void mlx5e_free_txqsq_descs(struct mlx5e_txqsq *sq)
{
struct mlx5e_tx_wqe_info *wi;
struct sk_buff *skb;
@@ -509,8 +485,8 @@
while (sq->cc != sq->pc) {
ci = sq->cc & sq->wq.sz_m1;
- skb = sq->db.txq.skb[ci];
- wi = &sq->db.txq.wqe_info[ci];
+ wi = &sq->db.wqe_info[ci];
+ skb = wi->skb;
if (!skb) { /* nop */
sq->cc++;
@@ -529,36 +505,89 @@
}
}
-static void mlx5e_free_xdp_sq_descs(struct mlx5e_sq *sq)
+#ifdef CONFIG_MLX5_CORE_IPOIB
+
+struct mlx5_wqe_eth_pad {
+ u8 rsvd0[16];
+};
+
+struct mlx5i_tx_wqe {
+ struct mlx5_wqe_ctrl_seg ctrl;
+ struct mlx5_wqe_datagram_seg datagram;
+ struct mlx5_wqe_eth_pad pad;
+ struct mlx5_wqe_eth_seg eth;
+};
+
+static inline void
+mlx5i_txwqe_build_datagram(struct mlx5_av *av, u32 dqpn, u32 dqkey,
+ struct mlx5_wqe_datagram_seg *dseg)
{
- struct mlx5e_sq_wqe_info *wi;
- struct mlx5e_dma_info *di;
- u16 ci;
-
- while (sq->cc != sq->pc) {
- ci = sq->cc & sq->wq.sz_m1;
- di = &sq->db.xdp.di[ci];
- wi = &sq->db.xdp.wqe_info[ci];
-
- if (wi->opcode == MLX5_OPCODE_NOP) {
- sq->cc++;
- continue;
- }
-
- sq->cc += wi->num_wqebbs;
-
- mlx5e_page_release(&sq->channel->rq, di, false);
- }
+ memcpy(&dseg->av, av, sizeof(struct mlx5_av));
+ dseg->av.dqp_dct = cpu_to_be32(dqpn | MLX5_EXTENDED_UD_AV);
+ dseg->av.key.qkey.qkey = cpu_to_be32(dqkey);
}
-void mlx5e_free_sq_descs(struct mlx5e_sq *sq)
+netdev_tx_t mlx5i_sq_xmit(struct mlx5e_txqsq *sq, struct sk_buff *skb,
+ struct mlx5_av *av, u32 dqpn, u32 dqkey)
{
- switch (sq->type) {
- case MLX5E_SQ_TXQ:
- mlx5e_free_txq_sq_descs(sq);
- break;
- case MLX5E_SQ_XDP:
- mlx5e_free_xdp_sq_descs(sq);
- break;
+ struct mlx5_wq_cyc *wq = &sq->wq;
+ u16 pi = sq->pc & wq->sz_m1;
+ struct mlx5i_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(wq, pi);
+ struct mlx5e_tx_wqe_info *wi = &sq->db.wqe_info[pi];
+
+ struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+ struct mlx5_wqe_datagram_seg *datagram = &wqe->datagram;
+ struct mlx5_wqe_eth_seg *eseg = &wqe->eth;
+
+ unsigned char *skb_data = skb->data;
+ unsigned int skb_len = skb->len;
+ u8 opcode = MLX5_OPCODE_SEND;
+ unsigned int num_bytes;
+ int num_dma;
+ u16 headlen;
+ u16 ds_cnt;
+ u16 ihs;
+
+ memset(wqe, 0, sizeof(*wqe));
+
+ mlx5i_txwqe_build_datagram(av, dqpn, dqkey, datagram);
+
+ mlx5e_txwqe_build_eseg_csum(sq, skb, eseg);
+
+ if (skb_is_gso(skb)) {
+ opcode = MLX5_OPCODE_LSO;
+ ihs = mlx5e_txwqe_build_eseg_gso(sq, skb, eseg, &num_bytes);
+ } else {
+ ihs = mlx5e_calc_min_inline(sq->min_inline_mode, skb);
+ num_bytes = max_t(unsigned int, skb->len, ETH_ZLEN);
}
+
+ ds_cnt = sizeof(*wqe) / MLX5_SEND_WQE_DS;
+ if (ihs) {
+ memcpy(eseg->inline_hdr.start, skb_data, ihs);
+ mlx5e_tx_skb_pull_inline(&skb_data, &skb_len, ihs);
+ eseg->inline_hdr.sz = cpu_to_be16(ihs);
+ ds_cnt += DIV_ROUND_UP(ihs - sizeof(eseg->inline_hdr.start), MLX5_SEND_WQE_DS);
+ }
+
+ headlen = skb_len - skb->data_len;
+ num_dma = mlx5e_txwqe_build_dsegs(sq, skb, skb_data, headlen,
+ (struct mlx5_wqe_data_seg *)cseg + ds_cnt);
+ if (unlikely(num_dma < 0))
+ goto dma_unmap_wqe_err;
+
+ mlx5e_txwqe_complete(sq, skb, opcode, ds_cnt + num_dma,
+ num_bytes, num_dma, wi, cseg);
+
+ return NETDEV_TX_OK;
+
+dma_unmap_wqe_err:
+ sq->stats.dropped++;
+ mlx5e_dma_unmap_wqe_err(sq, wi->num_dma);
+
+ dev_kfree_skb_any(skb);
+
+ return NETDEV_TX_OK;
}
+
+#endif
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index e5c12a7..5ca6714 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -37,22 +37,52 @@
struct mlx5_cqwq *wq = &cq->wq;
u32 ci = mlx5_cqwq_get_ci(wq);
struct mlx5_cqe64 *cqe = mlx5_cqwq_get_wqe(wq, ci);
- int cqe_ownership_bit = cqe->op_own & MLX5_CQE_OWNER_MASK;
- int sw_ownership_val = mlx5_cqwq_get_wrap_cnt(wq) & 1;
+ u8 cqe_ownership_bit = cqe->op_own & MLX5_CQE_OWNER_MASK;
+ u8 sw_ownership_val = mlx5_cqwq_get_wrap_cnt(wq) & 1;
if (cqe_ownership_bit != sw_ownership_val)
return NULL;
/* ensure cqe content is read after cqe ownership bit */
- rmb();
+ dma_rmb();
return cqe;
}
+static inline void mlx5e_poll_ico_single_cqe(struct mlx5e_cq *cq,
+ struct mlx5e_icosq *sq,
+ struct mlx5_cqe64 *cqe,
+ u16 *sqcc)
+{
+ struct mlx5_wq_cyc *wq = &sq->wq;
+ u16 ci = be16_to_cpu(cqe->wqe_counter) & wq->sz_m1;
+ struct mlx5e_sq_wqe_info *icowi = &sq->db.ico_wqe[ci];
+ struct mlx5e_rq *rq = &sq->channel->rq;
+
+ prefetch(rq);
+ mlx5_cqwq_pop(&cq->wq);
+ *sqcc += icowi->num_wqebbs;
+
+ if (unlikely((cqe->op_own >> 4) != MLX5_CQE_REQ)) {
+ WARN_ONCE(true, "mlx5e: Bad OP in ICOSQ CQE: 0x%x\n",
+ cqe->op_own);
+ return;
+ }
+
+ if (likely(icowi->opcode == MLX5_OPCODE_UMR)) {
+ mlx5e_post_rx_mpwqe(rq);
+ return;
+ }
+
+ if (unlikely(icowi->opcode != MLX5_OPCODE_NOP))
+ WARN_ONCE(true,
+ "mlx5e: Bad OPCODE in ICOSQ WQE info: 0x%x\n",
+ icowi->opcode);
+}
+
static void mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
{
- struct mlx5e_sq *sq = container_of(cq, struct mlx5e_sq, cq);
- struct mlx5_wq_cyc *wq;
+ struct mlx5e_icosq *sq = container_of(cq, struct mlx5e_icosq, cq);
struct mlx5_cqe64 *cqe;
u16 sqcc;
@@ -63,39 +93,13 @@
if (likely(!cqe))
return;
- wq = &sq->wq;
-
/* sq->cc must be updated only after mlx5_cqwq_update_db_record(),
* otherwise a cq overrun may occur
*/
sqcc = sq->cc;
- do {
- u16 ci = be16_to_cpu(cqe->wqe_counter) & wq->sz_m1;
- struct mlx5e_sq_wqe_info *icowi = &sq->db.ico_wqe[ci];
-
- mlx5_cqwq_pop(&cq->wq);
- sqcc += icowi->num_wqebbs;
-
- if (unlikely((cqe->op_own >> 4) != MLX5_CQE_REQ)) {
- WARN_ONCE(true, "mlx5e: Bad OP in ICOSQ CQE: 0x%x\n",
- cqe->op_own);
- break;
- }
-
- switch (icowi->opcode) {
- case MLX5_OPCODE_NOP:
- break;
- case MLX5_OPCODE_UMR:
- mlx5e_post_rx_mpwqe(&sq->channel->rq);
- break;
- default:
- WARN_ONCE(true,
- "mlx5e: Bad OPCODE in ICOSQ WQE info: 0x%x\n",
- icowi->opcode);
- }
-
- } while ((cqe = mlx5e_get_cqe(cq)));
+ /* by design, there's only a single cqe */
+ mlx5e_poll_ico_single_cqe(cq, sq, cqe, &sqcc);
mlx5_cqwq_update_db_record(&cq->wq);
@@ -105,66 +109,6 @@
sq->cc = sqcc;
}
-static inline bool mlx5e_poll_xdp_tx_cq(struct mlx5e_cq *cq)
-{
- struct mlx5e_sq *sq;
- u16 sqcc;
- int i;
-
- sq = container_of(cq, struct mlx5e_sq, cq);
-
- if (unlikely(!test_bit(MLX5E_SQ_STATE_ENABLED, &sq->state)))
- return false;
-
- /* sq->cc must be updated only after mlx5_cqwq_update_db_record(),
- * otherwise a cq overrun may occur
- */
- sqcc = sq->cc;
-
- for (i = 0; i < MLX5E_TX_CQ_POLL_BUDGET; i++) {
- struct mlx5_cqe64 *cqe;
- u16 wqe_counter;
- bool last_wqe;
-
- cqe = mlx5e_get_cqe(cq);
- if (!cqe)
- break;
-
- mlx5_cqwq_pop(&cq->wq);
-
- wqe_counter = be16_to_cpu(cqe->wqe_counter);
-
- do {
- struct mlx5e_sq_wqe_info *wi;
- struct mlx5e_dma_info *di;
- u16 ci;
-
- last_wqe = (sqcc == wqe_counter);
-
- ci = sqcc & sq->wq.sz_m1;
- di = &sq->db.xdp.di[ci];
- wi = &sq->db.xdp.wqe_info[ci];
-
- if (unlikely(wi->opcode == MLX5_OPCODE_NOP)) {
- sqcc++;
- continue;
- }
-
- sqcc += wi->num_wqebbs;
- /* Recycle RX page */
- mlx5e_page_release(&sq->channel->rq, di, true);
- } while (!last_wqe);
- }
-
- mlx5_cqwq_update_db_record(&cq->wq);
-
- /* ensure cq space is freed before enabling more cqes */
- wmb();
-
- sq->cc = sqcc;
- return (i == MLX5E_TX_CQ_POLL_BUDGET);
-}
-
int mlx5e_napi_poll(struct napi_struct *napi, int budget)
{
struct mlx5e_channel *c = container_of(napi, struct mlx5e_channel,
@@ -178,12 +122,12 @@
for (i = 0; i < c->num_tc; i++)
busy |= mlx5e_poll_tx_cq(&c->sq[i].cq, budget);
+ if (c->xdp)
+ busy |= mlx5e_poll_xdpsq_cq(&c->rq.xdpsq.cq);
+
work_done = mlx5e_poll_rx_cq(&c->rq.cq, budget);
busy |= work_done == budget;
- if (c->xdp)
- busy |= mlx5e_poll_xdp_tx_cq(&c->xdp_sq.cq);
-
mlx5e_poll_ico_cq(&c->icosq.cq);
busy |= mlx5e_post_rx_wqes(&c->rq);
@@ -224,8 +168,7 @@
{
struct mlx5e_cq *cq = container_of(mcq, struct mlx5e_cq, mcq);
struct mlx5e_channel *c = cq->channel;
- struct mlx5e_priv *priv = c->priv;
- struct net_device *netdev = priv->netdev;
+ struct net_device *netdev = c->netdev;
netdev_err(netdev, "%s: cqn=0x%.6x event=0x%.2x\n",
__func__, mcq->cqn, event);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
index fcd5bc7e..2e34d95 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
@@ -53,13 +53,6 @@
u32 vport;
};
-/* E-Switch MC FDB table hash node */
-struct esw_mc_addr { /* SRIOV only */
- struct l2addr_node node;
- struct mlx5_flow_handle *uplink_rule; /* Forward to uplink rule */
- u32 refcnt;
-};
-
/* Vport UC/MC hash node */
struct vport_addr {
struct l2addr_node node;
@@ -337,6 +330,7 @@
static int esw_create_legacy_fdb_table(struct mlx5_eswitch *esw, int nvports)
{
int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
+ struct mlx5_flow_table_attr ft_attr = {};
struct mlx5_core_dev *dev = esw->dev;
struct mlx5_flow_namespace *root_ns;
struct mlx5_flow_table *fdb;
@@ -362,7 +356,9 @@
memset(flow_group_in, 0, inlen);
table_size = BIT(MLX5_CAP_ESW_FLOWTABLE_FDB(dev, log_max_ft_size));
- fdb = mlx5_create_flow_table(root_ns, 0, table_size, 0, 0);
+
+ ft_attr.max_fte = table_size;
+ fdb = mlx5_create_flow_table(root_ns, &ft_attr);
if (IS_ERR(fdb)) {
err = PTR_ERR(fdb);
esw_warn(dev, "Failed to create FDB Table err %d\n", err);
@@ -814,7 +810,7 @@
static void esw_apply_vport_rx_mode(struct mlx5_eswitch *esw, u32 vport_num,
bool promisc, bool mc_promisc)
{
- struct esw_mc_addr *allmulti_addr = esw->mc_promisc;
+ struct esw_mc_addr *allmulti_addr = &esw->mc_promisc;
struct mlx5_vport *vport = &esw->vports[vport_num];
if (IS_ERR_OR_NULL(vport->allmulti_rule) != mc_promisc)
@@ -1685,7 +1681,7 @@
esw_info(esw->dev, "disable SRIOV: active vports(%d) mode(%d)\n",
esw->enabled_vports, esw->mode);
- mc_promisc = esw->mc_promisc;
+ mc_promisc = &esw->mc_promisc;
nvports = esw->enabled_vports;
for (i = 0; i < esw->total_vports; i++)
@@ -1729,7 +1725,6 @@
{
int l2_table_size = 1 << MLX5_CAP_GEN(dev, log_max_l2_table);
int total_vports = MLX5_TOTAL_VPORTS(dev);
- struct esw_mc_addr *mc_promisc;
struct mlx5_eswitch *esw;
int vport_num;
int err;
@@ -1758,13 +1753,6 @@
}
esw->l2_table.size = l2_table_size;
- mc_promisc = kzalloc(sizeof(*mc_promisc), GFP_KERNEL);
- if (!mc_promisc) {
- err = -ENOMEM;
- goto abort;
- }
- esw->mc_promisc = mc_promisc;
-
esw->work_queue = create_singlethread_workqueue("mlx5_esw_wq");
if (!esw->work_queue) {
err = -ENOMEM;
@@ -1803,6 +1791,11 @@
esw->enabled_vports = 0;
esw->mode = SRIOV_NONE;
esw->offloads.inline_mode = MLX5_INLINE_MODE_NONE;
+ if (MLX5_CAP_ESW_FLOWTABLE_FDB(dev, encap) &&
+ MLX5_CAP_ESW_FLOWTABLE_FDB(dev, decap))
+ esw->offloads.encap = DEVLINK_ESWITCH_ENCAP_MODE_BASIC;
+ else
+ esw->offloads.encap = DEVLINK_ESWITCH_ENCAP_MODE_NONE;
dev->priv.eswitch = esw;
return 0;
@@ -1827,7 +1820,6 @@
esw->dev->priv.eswitch = NULL;
destroy_workqueue(esw->work_queue);
kfree(esw->l2_table.bitmap);
- kfree(esw->mc_promisc);
kfree(esw->offloads.vport_reps);
kfree(esw->vports);
kfree(esw);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
index ad329b16..b746f62 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
@@ -36,7 +36,6 @@
#include <linux/if_ether.h>
#include <linux/if_link.h>
#include <net/devlink.h>
-#include <net/ip_tunnels.h>
#include <linux/mlx5/device.h>
#define MLX5_MAX_UC_PER_VPORT(dev) \
@@ -210,6 +209,14 @@
DECLARE_HASHTABLE(encap_tbl, 8);
u8 inline_mode;
u64 num_flows;
+ u8 encap;
+};
+
+/* E-Switch MC FDB table hash node */
+struct esw_mc_addr { /* SRIOV only */
+ struct l2addr_node node;
+ struct mlx5_flow_handle *uplink_rule; /* Forward to uplink rule */
+ u32 refcnt;
};
struct mlx5_eswitch {
@@ -225,7 +232,7 @@
* and async SRIOV admin state changes
*/
struct mutex state_lock;
- struct esw_mc_addr *mc_promisc;
+ struct esw_mc_addr mc_promisc;
struct {
bool enabled;
@@ -285,20 +292,8 @@
SET_VLAN_INSERT = BIT(1)
};
-#define MLX5_FLOW_CONTEXT_ACTION_VLAN_POP 0x40
-#define MLX5_FLOW_CONTEXT_ACTION_VLAN_PUSH 0x80
-
-struct mlx5_encap_entry {
- struct hlist_node encap_hlist;
- struct list_head flows;
- u32 encap_id;
- struct neighbour *n;
- struct ip_tunnel_info tun_info;
- unsigned char h_dest[ETH_ALEN]; /* destination eth addr */
-
- struct net_device *out_dev;
- int tunnel_type;
-};
+#define MLX5_FLOW_CONTEXT_ACTION_VLAN_POP 0x4000
+#define MLX5_FLOW_CONTEXT_ACTION_VLAN_PUSH 0x8000
struct mlx5_esw_flow_attr {
struct mlx5_eswitch_rep *in_rep;
@@ -307,7 +302,9 @@
int action;
u16 vlan;
bool vlan_handled;
- struct mlx5_encap_entry *encap;
+ u32 encap_id;
+ u32 mod_hdr_id;
+ struct mlx5e_tc_flow_parse_attr *parse_attr;
};
int mlx5_eswitch_sqs2vport_start(struct mlx5_eswitch *esw,
@@ -321,6 +318,8 @@
int mlx5_devlink_eswitch_inline_mode_set(struct devlink *devlink, u8 mode);
int mlx5_devlink_eswitch_inline_mode_get(struct devlink *devlink, u8 *mode);
int mlx5_eswitch_inline_mode_get(struct mlx5_eswitch *esw, int nvfs, u8 *mode);
+int mlx5_devlink_eswitch_encap_mode_set(struct devlink *devlink, u8 encap);
+int mlx5_devlink_eswitch_encap_mode_get(struct devlink *devlink, u8 *encap);
void mlx5_eswitch_register_vport_rep(struct mlx5_eswitch *esw,
int vport_index,
struct mlx5_eswitch_rep *rep);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
index 307ec6c..f991f66 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
@@ -68,8 +68,10 @@
}
if (flow_act.action & MLX5_FLOW_CONTEXT_ACTION_COUNT) {
counter = mlx5_fc_create(esw->dev, true);
- if (IS_ERR(counter))
- return ERR_CAST(counter);
+ if (IS_ERR(counter)) {
+ rule = ERR_CAST(counter);
+ goto err_counter_alloc;
+ }
dest[i].type = MLX5_FLOW_DESTINATION_TYPE_COUNTER;
dest[i].counter = counter;
i++;
@@ -86,17 +88,25 @@
if (flow_act.action & MLX5_FLOW_CONTEXT_ACTION_DECAP)
spec->match_criteria_enable |= MLX5_MATCH_INNER_HEADERS;
- if (attr->encap)
- flow_act.encap_id = attr->encap->encap_id;
+ if (attr->action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)
+ flow_act.modify_id = attr->mod_hdr_id;
+
+ if (attr->action & MLX5_FLOW_CONTEXT_ACTION_ENCAP)
+ flow_act.encap_id = attr->encap_id;
rule = mlx5_add_flow_rules((struct mlx5_flow_table *)esw->fdb_table.fdb,
spec, &flow_act, dest, i);
if (IS_ERR(rule))
- mlx5_fc_destroy(esw->dev, counter);
+ goto err_add_rule;
else
esw->offloads.num_flows++;
return rule;
+
+err_add_rule:
+ mlx5_fc_destroy(esw->dev, counter);
+err_counter_alloc:
+ return rule;
}
void
@@ -106,12 +116,10 @@
{
struct mlx5_fc *counter = NULL;
- if (!IS_ERR(rule)) {
- counter = mlx5_flow_rule_counter(rule);
- mlx5_del_flow_rules(rule);
- mlx5_fc_destroy(esw->dev, counter);
- esw->offloads.num_flows--;
- }
+ counter = mlx5_flow_rule_counter(rule);
+ mlx5_del_flow_rules(rule);
+ mlx5_fc_destroy(esw->dev, counter);
+ esw->offloads.num_flows--;
}
static int esw_set_global_vlan_pop(struct mlx5_eswitch *esw, u8 val)
@@ -418,21 +426,68 @@
return err;
}
-#define MAX_PF_SQ 256
#define ESW_OFFLOADS_NUM_GROUPS 4
-static int esw_create_offloads_fdb_table(struct mlx5_eswitch *esw, int nvports)
+static int esw_create_offloads_fast_fdb_table(struct mlx5_eswitch *esw)
{
- int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
- int table_size, ix, esw_size, err = 0;
struct mlx5_core_dev *dev = esw->dev;
struct mlx5_flow_namespace *root_ns;
struct mlx5_flow_table *fdb = NULL;
- struct mlx5_flow_group *g;
- u32 *flow_group_in;
- void *match_criteria;
+ int esw_size, err = 0;
u32 flags = 0;
+ root_ns = mlx5_get_flow_namespace(dev, MLX5_FLOW_NAMESPACE_FDB);
+ if (!root_ns) {
+ esw_warn(dev, "Failed to get FDB flow namespace\n");
+ err = -EOPNOTSUPP;
+ goto out;
+ }
+
+ esw_debug(dev, "Create offloads FDB table, min (max esw size(2^%d), max counters(%d)*groups(%d))\n",
+ MLX5_CAP_ESW_FLOWTABLE_FDB(dev, log_max_ft_size),
+ MLX5_CAP_GEN(dev, max_flow_counter), ESW_OFFLOADS_NUM_GROUPS);
+
+ esw_size = min_t(int, MLX5_CAP_GEN(dev, max_flow_counter) * ESW_OFFLOADS_NUM_GROUPS,
+ 1 << MLX5_CAP_ESW_FLOWTABLE_FDB(dev, log_max_ft_size));
+
+ if (esw->offloads.encap != DEVLINK_ESWITCH_ENCAP_MODE_NONE)
+ flags |= MLX5_FLOW_TABLE_TUNNEL_EN;
+
+ fdb = mlx5_create_auto_grouped_flow_table(root_ns, FDB_FAST_PATH,
+ esw_size,
+ ESW_OFFLOADS_NUM_GROUPS, 0,
+ flags);
+ if (IS_ERR(fdb)) {
+ err = PTR_ERR(fdb);
+ esw_warn(dev, "Failed to create Fast path FDB Table err %d\n", err);
+ goto out;
+ }
+ esw->fdb_table.fdb = fdb;
+
+out:
+ return err;
+}
+
+static void esw_destroy_offloads_fast_fdb_table(struct mlx5_eswitch *esw)
+{
+ mlx5_destroy_flow_table(esw->fdb_table.fdb);
+}
+
+#define MAX_PF_SQ 256
+
+static int esw_create_offloads_fdb_tables(struct mlx5_eswitch *esw, int nvports)
+{
+ int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
+ struct mlx5_flow_table_attr ft_attr = {};
+ struct mlx5_core_dev *dev = esw->dev;
+ struct mlx5_flow_namespace *root_ns;
+ struct mlx5_flow_table *fdb = NULL;
+ int table_size, ix, err = 0;
+ struct mlx5_flow_group *g;
+ void *match_criteria;
+ u32 *flow_group_in;
+
+ esw_debug(esw->dev, "Create offloads FDB Tables\n");
flow_group_in = mlx5_vzalloc(inlen);
if (!flow_group_in)
return -ENOMEM;
@@ -444,30 +499,16 @@
goto ns_err;
}
- esw_debug(dev, "Create offloads FDB table, min (max esw size(2^%d), max counters(%d)*groups(%d))\n",
- MLX5_CAP_ESW_FLOWTABLE_FDB(dev, log_max_ft_size),
- MLX5_CAP_GEN(dev, max_flow_counter), ESW_OFFLOADS_NUM_GROUPS);
-
- esw_size = min_t(int, MLX5_CAP_GEN(dev, max_flow_counter) * ESW_OFFLOADS_NUM_GROUPS,
- 1 << MLX5_CAP_ESW_FLOWTABLE_FDB(dev, log_max_ft_size));
-
- if (MLX5_CAP_ESW_FLOWTABLE_FDB(dev, encap) &&
- MLX5_CAP_ESW_FLOWTABLE_FDB(dev, decap))
- flags |= MLX5_FLOW_TABLE_TUNNEL_EN;
-
- fdb = mlx5_create_auto_grouped_flow_table(root_ns, FDB_FAST_PATH,
- esw_size,
- ESW_OFFLOADS_NUM_GROUPS, 0,
- flags);
- if (IS_ERR(fdb)) {
- err = PTR_ERR(fdb);
- esw_warn(dev, "Failed to create Fast path FDB Table err %d\n", err);
+ err = esw_create_offloads_fast_fdb_table(esw);
+ if (err)
goto fast_fdb_err;
- }
- esw->fdb_table.fdb = fdb;
table_size = nvports + MAX_PF_SQ + 1;
- fdb = mlx5_create_flow_table(root_ns, FDB_SLOW_PATH, table_size, 0, 0);
+
+ ft_attr.max_fte = table_size;
+ ft_attr.prio = FDB_SLOW_PATH;
+
+ fdb = mlx5_create_flow_table(root_ns, &ft_attr);
if (IS_ERR(fdb)) {
err = PTR_ERR(fdb);
esw_warn(dev, "Failed to create slow path FDB Table err %d\n", err);
@@ -532,25 +573,26 @@
return err;
}
-static void esw_destroy_offloads_fdb_table(struct mlx5_eswitch *esw)
+static void esw_destroy_offloads_fdb_tables(struct mlx5_eswitch *esw)
{
if (!esw->fdb_table.fdb)
return;
- esw_debug(esw->dev, "Destroy offloads FDB Table\n");
+ esw_debug(esw->dev, "Destroy offloads FDB Tables\n");
mlx5_del_flow_rules(esw->fdb_table.offloads.miss_rule);
mlx5_destroy_flow_group(esw->fdb_table.offloads.send_to_vport_grp);
mlx5_destroy_flow_group(esw->fdb_table.offloads.miss_grp);
mlx5_destroy_flow_table(esw->fdb_table.offloads.fdb);
- mlx5_destroy_flow_table(esw->fdb_table.fdb);
+ esw_destroy_offloads_fast_fdb_table(esw);
}
static int esw_create_offloads_table(struct mlx5_eswitch *esw)
{
- struct mlx5_flow_namespace *ns;
- struct mlx5_flow_table *ft_offloads;
+ struct mlx5_flow_table_attr ft_attr = {};
struct mlx5_core_dev *dev = esw->dev;
+ struct mlx5_flow_table *ft_offloads;
+ struct mlx5_flow_namespace *ns;
int err = 0;
ns = mlx5_get_flow_namespace(dev, MLX5_FLOW_NAMESPACE_OFFLOADS);
@@ -559,7 +601,9 @@
return -EOPNOTSUPP;
}
- ft_offloads = mlx5_create_flow_table(ns, 0, dev->priv.sriov.num_vfs + 2, 0, 0);
+ ft_attr.max_fte = dev->priv.sriov.num_vfs + 2;
+
+ ft_offloads = mlx5_create_flow_table(ns, &ft_attr);
if (IS_ERR(ft_offloads)) {
err = PTR_ERR(ft_offloads);
esw_warn(esw->dev, "Failed to create offloads table, err %d\n", err);
@@ -700,7 +744,7 @@
mlx5_remove_dev_by_protocol(esw->dev, MLX5_INTERFACE_PROTOCOL_IB);
mlx5_dev_list_unlock();
- err = esw_create_offloads_fdb_table(esw, nvports);
+ err = esw_create_offloads_fdb_tables(esw, nvports);
if (err)
goto create_fdb_err;
@@ -737,7 +781,7 @@
esw_destroy_offloads_table(esw);
create_ft_err:
- esw_destroy_offloads_fdb_table(esw);
+ esw_destroy_offloads_fdb_tables(esw);
create_fdb_err:
/* enable back PF RoCE */
@@ -783,7 +827,7 @@
esw_destroy_vport_rx_group(esw);
esw_destroy_offloads_table(esw);
- esw_destroy_offloads_fdb_table(esw);
+ esw_destroy_offloads_fdb_tables(esw);
}
static int esw_mode_from_devlink(u16 mode, u16 *mlx5_mode)
@@ -911,8 +955,7 @@
struct mlx5_core_dev *dev = devlink_priv(devlink);
struct mlx5_eswitch *esw = dev->priv.eswitch;
int num_vports = esw->enabled_vports;
- int err;
- int vport;
+ int err, vport;
u8 mlx5_mode;
if (!MLX5_CAP_GEN(dev, vport_group_manager))
@@ -921,9 +964,17 @@
if (esw->mode == SRIOV_NONE)
return -EOPNOTSUPP;
- if (MLX5_CAP_ETH(dev, wqe_inline_mode) !=
- MLX5_CAP_INLINE_MODE_VPORT_CONTEXT)
+ switch (MLX5_CAP_ETH(dev, wqe_inline_mode)) {
+ case MLX5_CAP_INLINE_MODE_NOT_REQUIRED:
+ if (mode == DEVLINK_ESWITCH_INLINE_MODE_NONE)
+ return 0;
+ /* fall through */
+ case MLX5_CAP_INLINE_MODE_L2:
+ esw_warn(dev, "Inline mode can't be set\n");
return -EOPNOTSUPP;
+ case MLX5_CAP_INLINE_MODE_VPORT_CONTEXT:
+ break;
+ }
if (esw->offloads.num_flows > 0) {
esw_warn(dev, "Can't set inline mode when flows are configured\n");
@@ -966,18 +1017,14 @@
if (esw->mode == SRIOV_NONE)
return -EOPNOTSUPP;
- if (MLX5_CAP_ETH(dev, wqe_inline_mode) !=
- MLX5_CAP_INLINE_MODE_VPORT_CONTEXT)
- return -EOPNOTSUPP;
-
return esw_inline_mode_to_devlink(esw->offloads.inline_mode, mode);
}
int mlx5_eswitch_inline_mode_get(struct mlx5_eswitch *esw, int nvfs, u8 *mode)
{
+ u8 prev_mlx5_mode, mlx5_mode = MLX5_INLINE_MODE_L2;
struct mlx5_core_dev *dev = esw->dev;
int vport;
- u8 prev_mlx5_mode, mlx5_mode = MLX5_INLINE_MODE_L2;
if (!MLX5_CAP_GEN(dev, vport_group_manager))
return -EOPNOTSUPP;
@@ -985,10 +1032,18 @@
if (esw->mode == SRIOV_NONE)
return -EOPNOTSUPP;
- if (MLX5_CAP_ETH(dev, wqe_inline_mode) !=
- MLX5_CAP_INLINE_MODE_VPORT_CONTEXT)
- return -EOPNOTSUPP;
+ switch (MLX5_CAP_ETH(dev, wqe_inline_mode)) {
+ case MLX5_CAP_INLINE_MODE_NOT_REQUIRED:
+ mlx5_mode = MLX5_INLINE_MODE_NONE;
+ goto out;
+ case MLX5_CAP_INLINE_MODE_L2:
+ mlx5_mode = MLX5_INLINE_MODE_L2;
+ goto out;
+ case MLX5_CAP_INLINE_MODE_VPORT_CONTEXT:
+ goto query_vports;
+ }
+query_vports:
for (vport = 1; vport <= nvfs; vport++) {
mlx5_query_nic_vport_min_inline(dev, vport, &mlx5_mode);
if (vport > 1 && prev_mlx5_mode != mlx5_mode)
@@ -996,10 +1051,71 @@
prev_mlx5_mode = mlx5_mode;
}
+out:
*mode = mlx5_mode;
return 0;
}
+int mlx5_devlink_eswitch_encap_mode_set(struct devlink *devlink, u8 encap)
+{
+ struct mlx5_core_dev *dev = devlink_priv(devlink);
+ struct mlx5_eswitch *esw = dev->priv.eswitch;
+ int err;
+
+ if (!MLX5_CAP_GEN(dev, vport_group_manager))
+ return -EOPNOTSUPP;
+
+ if (esw->mode == SRIOV_NONE)
+ return -EOPNOTSUPP;
+
+ if (encap != DEVLINK_ESWITCH_ENCAP_MODE_NONE &&
+ (!MLX5_CAP_ESW_FLOWTABLE_FDB(dev, encap) ||
+ !MLX5_CAP_ESW_FLOWTABLE_FDB(dev, decap)))
+ return -EOPNOTSUPP;
+
+ if (encap && encap != DEVLINK_ESWITCH_ENCAP_MODE_BASIC)
+ return -EOPNOTSUPP;
+
+ if (esw->mode == SRIOV_LEGACY) {
+ esw->offloads.encap = encap;
+ return 0;
+ }
+
+ if (esw->offloads.encap == encap)
+ return 0;
+
+ if (esw->offloads.num_flows > 0) {
+ esw_warn(dev, "Can't set encapsulation when flows are configured\n");
+ return -EOPNOTSUPP;
+ }
+
+ esw_destroy_offloads_fast_fdb_table(esw);
+
+ esw->offloads.encap = encap;
+ err = esw_create_offloads_fast_fdb_table(esw);
+ if (err) {
+ esw_warn(esw->dev, "Failed re-creating fast FDB table, err %d\n", err);
+ esw->offloads.encap = !encap;
+ (void) esw_create_offloads_fast_fdb_table(esw);
+ }
+ return err;
+}
+
+int mlx5_devlink_eswitch_encap_mode_get(struct devlink *devlink, u8 *encap)
+{
+ struct mlx5_core_dev *dev = devlink_priv(devlink);
+ struct mlx5_eswitch *esw = dev->priv.eswitch;
+
+ if (!MLX5_CAP_GEN(dev, vport_group_manager))
+ return -EOPNOTSUPP;
+
+ if (esw->mode == SRIOV_NONE)
+ return -EOPNOTSUPP;
+
+ *encap = esw->offloads.encap;
+ return 0;
+}
+
void mlx5_eswitch_register_vport_rep(struct mlx5_eswitch *esw,
int vport_index,
struct mlx5_eswitch_rep *__rep)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c
index b64a781..19e3d2f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_cmd.c
@@ -45,6 +45,10 @@
u32 in[MLX5_ST_SZ_DW(set_flow_table_root_in)] = {0};
u32 out[MLX5_ST_SZ_DW(set_flow_table_root_out)] = {0};
+ if ((MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_IB) &&
+ ft->underlay_qpn == 0)
+ return 0;
+
MLX5_SET(set_flow_table_root_in, in, opcode,
MLX5_CMD_OP_SET_FLOW_TABLE_ROOT);
MLX5_SET(set_flow_table_root_in, in, table_type, ft->type);
@@ -54,6 +58,10 @@
MLX5_SET(set_flow_table_root_in, in, other_vport, 1);
}
+ if ((MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_IB) &&
+ ft->underlay_qpn != 0)
+ MLX5_SET(set_flow_table_root_in, in, underlay_qpn, ft->underlay_qpn);
+
return mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
}
@@ -249,6 +257,7 @@
MLX5_SET(flow_context, in_flow_context, flow_tag, fte->flow_tag);
MLX5_SET(flow_context, in_flow_context, action, fte->action);
MLX5_SET(flow_context, in_flow_context, encap_id, fte->encap_id);
+ MLX5_SET(flow_context, in_flow_context, modify_header_id, fte->modify_id);
in_match_value = MLX5_ADDR_OF(flow_context, in_flow_context,
match_value);
memcpy(in_match_value, &fte->val, MLX5_ST_SZ_BYTES(fte_match_param));
@@ -515,3 +524,69 @@
mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
}
+
+int mlx5_modify_header_alloc(struct mlx5_core_dev *dev,
+ u8 namespace, u8 num_actions,
+ void *modify_actions, u32 *modify_header_id)
+{
+ u32 out[MLX5_ST_SZ_DW(alloc_modify_header_context_out)];
+ int max_actions, actions_size, inlen, err;
+ void *actions_in;
+ u8 table_type;
+ u32 *in;
+
+ switch (namespace) {
+ case MLX5_FLOW_NAMESPACE_FDB:
+ max_actions = MLX5_CAP_ESW_FLOWTABLE_FDB(dev, max_modify_header_actions);
+ table_type = FS_FT_FDB;
+ break;
+ case MLX5_FLOW_NAMESPACE_KERNEL:
+ max_actions = MLX5_CAP_FLOWTABLE_NIC_RX(dev, max_modify_header_actions);
+ table_type = FS_FT_NIC_RX;
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ if (num_actions > max_actions) {
+ mlx5_core_warn(dev, "too many modify header actions %d, max supported %d\n",
+ num_actions, max_actions);
+ return -EOPNOTSUPP;
+ }
+
+ actions_size = MLX5_UN_SZ_BYTES(set_action_in_add_action_in_auto) * num_actions;
+ inlen = MLX5_ST_SZ_BYTES(alloc_modify_header_context_in) + actions_size;
+
+ in = kzalloc(inlen, GFP_KERNEL);
+ if (!in)
+ return -ENOMEM;
+
+ MLX5_SET(alloc_modify_header_context_in, in, opcode,
+ MLX5_CMD_OP_ALLOC_MODIFY_HEADER_CONTEXT);
+ MLX5_SET(alloc_modify_header_context_in, in, table_type, table_type);
+ MLX5_SET(alloc_modify_header_context_in, in, num_of_actions, num_actions);
+
+ actions_in = MLX5_ADDR_OF(alloc_modify_header_context_in, in, actions);
+ memcpy(actions_in, modify_actions, actions_size);
+
+ memset(out, 0, sizeof(out));
+ err = mlx5_cmd_exec(dev, in, inlen, out, sizeof(out));
+
+ *modify_header_id = MLX5_GET(alloc_modify_header_context_out, out, modify_header_id);
+ kfree(in);
+ return err;
+}
+
+void mlx5_modify_header_dealloc(struct mlx5_core_dev *dev, u32 modify_header_id)
+{
+ u32 in[MLX5_ST_SZ_DW(dealloc_modify_header_context_in)];
+ u32 out[MLX5_ST_SZ_DW(dealloc_modify_header_context_out)];
+
+ memset(in, 0, sizeof(in));
+ MLX5_SET(dealloc_modify_header_context_in, in, opcode,
+ MLX5_CMD_OP_DEALLOC_MODIFY_HEADER_CONTEXT);
+ MLX5_SET(dealloc_modify_header_context_in, in, modify_header_id,
+ modify_header_id);
+
+ mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
index ded27bb..b8a1765 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
@@ -476,6 +476,7 @@
fte->index = index;
fte->action = flow_act->action;
fte->encap_id = flow_act->encap_id;
+ fte->modify_id = flow_act->modify_id;
return fte;
}
@@ -777,18 +778,16 @@
}
static struct mlx5_flow_table *__mlx5_create_flow_table(struct mlx5_flow_namespace *ns,
+ struct mlx5_flow_table_attr *ft_attr,
enum fs_flow_table_op_mod op_mod,
- u16 vport, int prio,
- int max_fte, u32 level,
- u32 flags)
+ u16 vport)
{
+ struct mlx5_flow_root_namespace *root = find_root(&ns->node);
struct mlx5_flow_table *next_ft = NULL;
- struct mlx5_flow_table *ft;
- int err;
- int log_table_sz;
- struct mlx5_flow_root_namespace *root =
- find_root(&ns->node);
struct fs_prio *fs_prio = NULL;
+ struct mlx5_flow_table *ft;
+ int log_table_sz;
+ int err;
if (!root) {
pr_err("mlx5: flow steering failed to find root of namespace\n");
@@ -796,29 +795,31 @@
}
mutex_lock(&root->chain_lock);
- fs_prio = find_prio(ns, prio);
+ fs_prio = find_prio(ns, ft_attr->prio);
if (!fs_prio) {
err = -EINVAL;
goto unlock_root;
}
- if (level >= fs_prio->num_levels) {
+ if (ft_attr->level >= fs_prio->num_levels) {
err = -ENOSPC;
goto unlock_root;
}
/* The level is related to the
* priority level range.
*/
- level += fs_prio->start_level;
- ft = alloc_flow_table(level,
+ ft_attr->level += fs_prio->start_level;
+ ft = alloc_flow_table(ft_attr->level,
vport,
- max_fte ? roundup_pow_of_two(max_fte) : 0,
+ ft_attr->max_fte ? roundup_pow_of_two(ft_attr->max_fte) : 0,
root->table_type,
- op_mod, flags);
+ op_mod, ft_attr->flags);
if (!ft) {
err = -ENOMEM;
goto unlock_root;
}
+ ft->underlay_qpn = ft_attr->underlay_qpn;
+
tree_init_node(&ft->node, 1, del_flow_table);
log_table_sz = ft->max_fte ? ilog2(ft->max_fte) : 0;
next_ft = find_next_chained_ft(fs_prio);
@@ -848,44 +849,56 @@
}
struct mlx5_flow_table *mlx5_create_flow_table(struct mlx5_flow_namespace *ns,
- int prio, int max_fte,
- u32 level,
- u32 flags)
+ struct mlx5_flow_table_attr *ft_attr)
{
- return __mlx5_create_flow_table(ns, FS_FT_OP_MOD_NORMAL, 0, prio,
- max_fte, level, flags);
+ return __mlx5_create_flow_table(ns, ft_attr, FS_FT_OP_MOD_NORMAL, 0);
}
struct mlx5_flow_table *mlx5_create_vport_flow_table(struct mlx5_flow_namespace *ns,
int prio, int max_fte,
u32 level, u16 vport)
{
- return __mlx5_create_flow_table(ns, FS_FT_OP_MOD_NORMAL, vport, prio,
- max_fte, level, 0);
+ struct mlx5_flow_table_attr ft_attr = {};
+
+ ft_attr.max_fte = max_fte;
+ ft_attr.level = level;
+ ft_attr.prio = prio;
+
+ return __mlx5_create_flow_table(ns, &ft_attr, FS_FT_OP_MOD_NORMAL, 0);
}
-struct mlx5_flow_table *mlx5_create_lag_demux_flow_table(
- struct mlx5_flow_namespace *ns,
- int prio, u32 level)
+struct mlx5_flow_table*
+mlx5_create_lag_demux_flow_table(struct mlx5_flow_namespace *ns,
+ int prio, u32 level)
{
- return __mlx5_create_flow_table(ns, FS_FT_OP_MOD_LAG_DEMUX, 0, prio, 0,
- level, 0);
+ struct mlx5_flow_table_attr ft_attr = {};
+
+ ft_attr.level = level;
+ ft_attr.prio = prio;
+ return __mlx5_create_flow_table(ns, &ft_attr, FS_FT_OP_MOD_LAG_DEMUX, 0);
}
EXPORT_SYMBOL(mlx5_create_lag_demux_flow_table);
-struct mlx5_flow_table *mlx5_create_auto_grouped_flow_table(struct mlx5_flow_namespace *ns,
- int prio,
- int num_flow_table_entries,
- int max_num_groups,
- u32 level,
- u32 flags)
+struct mlx5_flow_table*
+mlx5_create_auto_grouped_flow_table(struct mlx5_flow_namespace *ns,
+ int prio,
+ int num_flow_table_entries,
+ int max_num_groups,
+ u32 level,
+ u32 flags)
{
+ struct mlx5_flow_table_attr ft_attr = {};
struct mlx5_flow_table *ft;
if (max_num_groups > num_flow_table_entries)
return ERR_PTR(-EINVAL);
- ft = mlx5_create_flow_table(ns, prio, num_flow_table_entries, level, flags);
+ ft_attr.max_fte = num_flow_table_entries;
+ ft_attr.prio = prio;
+ ft_attr.level = level;
+ ft_attr.flags = flags;
+
+ ft = mlx5_create_flow_table(ns, &ft_attr);
if (IS_ERR(ft))
return ft;
@@ -1827,12 +1840,18 @@
static int create_anchor_flow_table(struct mlx5_flow_steering *steering)
{
struct mlx5_flow_namespace *ns = NULL;
+ struct mlx5_flow_table_attr ft_attr = {};
struct mlx5_flow_table *ft;
ns = mlx5_get_flow_namespace(steering->dev, MLX5_FLOW_NAMESPACE_ANCHOR);
if (WARN_ON(!ns))
return -EINVAL;
- ft = mlx5_create_flow_table(ns, ANCHOR_PRIO, ANCHOR_SIZE, ANCHOR_LEVEL, 0);
+
+ ft_attr.max_fte = ANCHOR_SIZE;
+ ft_attr.level = ANCHOR_LEVEL;
+ ft_attr.prio = ANCHOR_PRIO;
+
+ ft = mlx5_create_flow_table(ns, &ft_attr);
if (IS_ERR(ft)) {
mlx5_core_err(steering->dev, "Failed to create last anchor flow table");
return PTR_ERR(ft);
@@ -1886,9 +1905,6 @@
{
struct mlx5_flow_steering *steering = dev->priv.steering;
- if (MLX5_CAP_GEN(dev, port_type) != MLX5_CAP_PORT_TYPE_ETH)
- return;
-
cleanup_root_ns(steering->root_ns);
cleanup_root_ns(steering->esw_egress_root_ns);
cleanup_root_ns(steering->esw_ingress_root_ns);
@@ -1991,9 +2007,6 @@
struct mlx5_flow_steering *steering;
int err = 0;
- if (MLX5_CAP_GEN(dev, port_type) != MLX5_CAP_PORT_TYPE_ETH)
- return 0;
-
err = mlx5_init_fc_stats(dev);
if (err)
return err;
@@ -2004,7 +2017,10 @@
steering->dev = dev;
dev->priv.steering = steering;
- if (MLX5_CAP_GEN(dev, nic_flow_table) &&
+ if ((((MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_ETH) &&
+ (MLX5_CAP_GEN(dev, nic_flow_table))) ||
+ ((MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_IB) &&
+ MLX5_CAP_GEN(dev, ipoib_enhanced_offloads))) &&
MLX5_CAP_FLOWTABLE_NIC_RX(dev, ft_support)) {
err = init_root_ns(steering);
if (err)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h
index 8e668c6..81eafc7 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h
@@ -118,6 +118,7 @@
/* FWD rules that point on this flow table */
struct list_head fwd_rules;
u32 flags;
+ u32 underlay_qpn;
};
struct mlx5_fc_cache {
@@ -152,6 +153,7 @@
u32 index;
u32 action;
u32 encap_id;
+ u32 modify_id;
enum fs_fte_status status;
struct mlx5_fc *counter;
};
@@ -197,6 +199,11 @@
int mlx5_init_fc_stats(struct mlx5_core_dev *dev);
void mlx5_cleanup_fc_stats(struct mlx5_core_dev *dev);
+void mlx5_fc_queue_stats_work(struct mlx5_core_dev *dev,
+ struct delayed_work *dwork,
+ unsigned long delay);
+void mlx5_fc_update_sampling_interval(struct mlx5_core_dev *dev,
+ unsigned long interval);
int mlx5_init_fs(struct mlx5_core_dev *dev);
void mlx5_cleanup_fs(struct mlx5_core_dev *dev);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_counters.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_counters.c
index 7431f63..6507d8a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_counters.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_counters.c
@@ -165,7 +165,8 @@
list_splice_tail_init(&fc_stats->addlist, &tmplist);
if (!list_empty(&tmplist) || !RB_EMPTY_ROOT(&fc_stats->counters))
- queue_delayed_work(fc_stats->wq, &fc_stats->work, MLX5_FC_STATS_PERIOD);
+ queue_delayed_work(fc_stats->wq, &fc_stats->work,
+ fc_stats->sampling_interval);
spin_unlock(&fc_stats->addlist_lock);
@@ -200,7 +201,7 @@
node = mlx5_fc_stats_query(dev, counter, last->id);
}
- fc_stats->next_query = now + MLX5_FC_STATS_PERIOD;
+ fc_stats->next_query = now + fc_stats->sampling_interval;
}
struct mlx5_fc *mlx5_fc_create(struct mlx5_core_dev *dev, bool aging)
@@ -265,6 +266,7 @@
if (!fc_stats->wq)
return -ENOMEM;
+ fc_stats->sampling_interval = MLX5_FC_STATS_PERIOD;
INIT_DELAYED_WORK(&fc_stats->work, mlx5_fc_stats_work);
return 0;
@@ -317,3 +319,21 @@
counter->lastbytes = c.bytes;
counter->lastpackets = c.packets;
}
+
+void mlx5_fc_queue_stats_work(struct mlx5_core_dev *dev,
+ struct delayed_work *dwork,
+ unsigned long delay)
+{
+ struct mlx5_fc_stats *fc_stats = &dev->priv.fc_stats;
+
+ queue_delayed_work(fc_stats->wq, dwork, delay);
+}
+
+void mlx5_fc_update_sampling_interval(struct mlx5_core_dev *dev,
+ unsigned long interval)
+{
+ struct mlx5_fc_stats *fc_stats = &dev->priv.fc_stats;
+
+ fc_stats->sampling_interval = min_t(unsigned long, interval,
+ fc_stats->sampling_interval);
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fw.c b/drivers/net/ethernet/mellanox/mlx5/core/fw.c
index d0bbefa..1bc14d0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fw.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fw.c
@@ -137,7 +137,8 @@
return err;
}
- if (MLX5_CAP_GEN(dev, nic_flow_table)) {
+ if (MLX5_CAP_GEN(dev, nic_flow_table) ||
+ MLX5_CAP_GEN(dev, ipoib_enhanced_offloads)) {
err = mlx5_core_get_caps(dev, MLX5_CAP_FLOW_TABLE);
if (err)
return err;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/ipoib.c b/drivers/net/ethernet/mellanox/mlx5/core/ipoib.c
new file mode 100644
index 0000000..3c84e36a
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/ipoib.c
@@ -0,0 +1,498 @@
+/*
+ * Copyright (c) 2017, Mellanox Technologies. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include <linux/mlx5/fs.h>
+#include "en.h"
+#include "ipoib.h"
+
+#define IB_DEFAULT_Q_KEY 0xb1b
+
+static int mlx5i_open(struct net_device *netdev);
+static int mlx5i_close(struct net_device *netdev);
+static int mlx5i_dev_init(struct net_device *dev);
+static void mlx5i_dev_cleanup(struct net_device *dev);
+
+static const struct net_device_ops mlx5i_netdev_ops = {
+ .ndo_open = mlx5i_open,
+ .ndo_stop = mlx5i_close,
+ .ndo_init = mlx5i_dev_init,
+ .ndo_uninit = mlx5i_dev_cleanup,
+};
+
+/* IPoIB mlx5 netdev profile */
+
+/* Called directly after IPoIB netdevice was created to initialize SW structs */
+static void mlx5i_init(struct mlx5_core_dev *mdev,
+ struct net_device *netdev,
+ const struct mlx5e_profile *profile,
+ void *ppriv)
+{
+ struct mlx5e_priv *priv = mlx5i_epriv(netdev);
+
+ priv->mdev = mdev;
+ priv->netdev = netdev;
+ priv->profile = profile;
+ priv->ppriv = ppriv;
+
+ mlx5e_build_nic_params(mdev, &priv->channels.params, profile->max_nch(mdev));
+
+ mutex_init(&priv->state_lock);
+
+ netdev->hw_features |= NETIF_F_SG;
+ netdev->hw_features |= NETIF_F_IP_CSUM;
+ netdev->hw_features |= NETIF_F_IPV6_CSUM;
+ netdev->hw_features |= NETIF_F_GRO;
+ netdev->hw_features |= NETIF_F_TSO;
+ netdev->hw_features |= NETIF_F_TSO6;
+ netdev->hw_features |= NETIF_F_RXCSUM;
+ netdev->hw_features |= NETIF_F_RXHASH;
+
+ netdev->netdev_ops = &mlx5i_netdev_ops;
+}
+
+/* Called directly before IPoIB netdevice is destroyed to cleanup SW structs */
+static void mlx5i_cleanup(struct mlx5e_priv *priv)
+{
+ /* Do nothing .. */
+}
+
+#define MLX5_QP_ENHANCED_ULP_STATELESS_MODE 2
+
+static int mlx5i_create_underlay_qp(struct mlx5_core_dev *mdev, struct mlx5_core_qp *qp)
+{
+ struct mlx5_qp_context *context = NULL;
+ u32 *in = NULL;
+ void *addr_path;
+ int ret = 0;
+ int inlen;
+ void *qpc;
+
+ inlen = MLX5_ST_SZ_BYTES(create_qp_in);
+ in = mlx5_vzalloc(inlen);
+ if (!in)
+ return -ENOMEM;
+
+ qpc = MLX5_ADDR_OF(create_qp_in, in, qpc);
+ MLX5_SET(qpc, qpc, st, MLX5_QP_ST_UD);
+ MLX5_SET(qpc, qpc, pm_state, MLX5_QP_PM_MIGRATED);
+ MLX5_SET(qpc, qpc, ulp_stateless_offload_mode,
+ MLX5_QP_ENHANCED_ULP_STATELESS_MODE);
+
+ addr_path = MLX5_ADDR_OF(qpc, qpc, primary_address_path);
+ MLX5_SET(ads, addr_path, port, 1);
+ MLX5_SET(ads, addr_path, grh, 1);
+
+ ret = mlx5_core_create_qp(mdev, qp, in, inlen);
+ if (ret) {
+ mlx5_core_err(mdev, "Failed creating IPoIB QP err : %d\n", ret);
+ goto out;
+ }
+
+ /* QP states */
+ context = kzalloc(sizeof(*context), GFP_KERNEL);
+ if (!context) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ context->flags = cpu_to_be32(MLX5_QP_PM_MIGRATED << 11);
+ context->pri_path.port = 1;
+ context->qkey = cpu_to_be32(IB_DEFAULT_Q_KEY);
+
+ ret = mlx5_core_qp_modify(mdev, MLX5_CMD_OP_RST2INIT_QP, 0, context, qp);
+ if (ret) {
+ mlx5_core_err(mdev, "Failed to modify qp RST2INIT, err: %d\n", ret);
+ goto out;
+ }
+ memset(context, 0, sizeof(*context));
+
+ ret = mlx5_core_qp_modify(mdev, MLX5_CMD_OP_INIT2RTR_QP, 0, context, qp);
+ if (ret) {
+ mlx5_core_err(mdev, "Failed to modify qp INIT2RTR, err: %d\n", ret);
+ goto out;
+ }
+
+ ret = mlx5_core_qp_modify(mdev, MLX5_CMD_OP_RTR2RTS_QP, 0, context, qp);
+ if (ret) {
+ mlx5_core_err(mdev, "Failed to modify qp RTR2RTS, err: %d\n", ret);
+ goto out;
+ }
+
+out:
+ kfree(context);
+ kvfree(in);
+ return ret;
+}
+
+static void mlx5i_destroy_underlay_qp(struct mlx5_core_dev *mdev, struct mlx5_core_qp *qp)
+{
+ mlx5_core_destroy_qp(mdev, qp);
+}
+
+static int mlx5i_init_tx(struct mlx5e_priv *priv)
+{
+ struct mlx5i_priv *ipriv = priv->ppriv;
+ int err;
+
+ err = mlx5i_create_underlay_qp(priv->mdev, &ipriv->qp);
+ if (err) {
+ mlx5_core_warn(priv->mdev, "create underlay QP failed, %d\n", err);
+ return err;
+ }
+
+ err = mlx5e_create_tis(priv->mdev, 0 /* tc */, ipriv->qp.qpn, &priv->tisn[0]);
+ if (err) {
+ mlx5_core_warn(priv->mdev, "create tis failed, %d\n", err);
+ return err;
+ }
+
+ return 0;
+}
+
+static void mlx5i_cleanup_tx(struct mlx5e_priv *priv)
+{
+ struct mlx5i_priv *ipriv = priv->ppriv;
+
+ mlx5e_destroy_tis(priv->mdev, priv->tisn[0]);
+ mlx5i_destroy_underlay_qp(priv->mdev, &ipriv->qp);
+}
+
+static int mlx5i_create_flow_steering(struct mlx5e_priv *priv)
+{
+ struct mlx5i_priv *ipriv = priv->ppriv;
+ int err;
+
+ priv->fs.ns = mlx5_get_flow_namespace(priv->mdev,
+ MLX5_FLOW_NAMESPACE_KERNEL);
+
+ if (!priv->fs.ns)
+ return -EINVAL;
+
+ err = mlx5e_arfs_create_tables(priv);
+ if (err) {
+ netdev_err(priv->netdev, "Failed to create arfs tables, err=%d\n",
+ err);
+ priv->netdev->hw_features &= ~NETIF_F_NTUPLE;
+ }
+
+ err = mlx5e_create_ttc_table(priv, ipriv->qp.qpn);
+ if (err) {
+ netdev_err(priv->netdev, "Failed to create ttc table, err=%d\n",
+ err);
+ goto err_destroy_arfs_tables;
+ }
+
+ return 0;
+
+err_destroy_arfs_tables:
+ mlx5e_arfs_destroy_tables(priv);
+
+ return err;
+}
+
+static void mlx5i_destroy_flow_steering(struct mlx5e_priv *priv)
+{
+ mlx5e_destroy_ttc_table(priv);
+ mlx5e_arfs_destroy_tables(priv);
+}
+
+static int mlx5i_init_rx(struct mlx5e_priv *priv)
+{
+ int err;
+
+ err = mlx5e_create_indirect_rqt(priv);
+ if (err)
+ return err;
+
+ err = mlx5e_create_direct_rqts(priv);
+ if (err)
+ goto err_destroy_indirect_rqts;
+
+ err = mlx5e_create_indirect_tirs(priv);
+ if (err)
+ goto err_destroy_direct_rqts;
+
+ err = mlx5e_create_direct_tirs(priv);
+ if (err)
+ goto err_destroy_indirect_tirs;
+
+ err = mlx5i_create_flow_steering(priv);
+ if (err)
+ goto err_destroy_direct_tirs;
+
+ return 0;
+
+err_destroy_direct_tirs:
+ mlx5e_destroy_direct_tirs(priv);
+err_destroy_indirect_tirs:
+ mlx5e_destroy_indirect_tirs(priv);
+err_destroy_direct_rqts:
+ mlx5e_destroy_direct_rqts(priv);
+err_destroy_indirect_rqts:
+ mlx5e_destroy_rqt(priv, &priv->indir_rqt);
+ return err;
+}
+
+static void mlx5i_cleanup_rx(struct mlx5e_priv *priv)
+{
+ mlx5i_destroy_flow_steering(priv);
+ mlx5e_destroy_direct_tirs(priv);
+ mlx5e_destroy_indirect_tirs(priv);
+ mlx5e_destroy_direct_rqts(priv);
+ mlx5e_destroy_rqt(priv, &priv->indir_rqt);
+}
+
+static const struct mlx5e_profile mlx5i_nic_profile = {
+ .init = mlx5i_init,
+ .cleanup = mlx5i_cleanup,
+ .init_tx = mlx5i_init_tx,
+ .cleanup_tx = mlx5i_cleanup_tx,
+ .init_rx = mlx5i_init_rx,
+ .cleanup_rx = mlx5i_cleanup_rx,
+ .enable = NULL, /* mlx5i_enable */
+ .disable = NULL, /* mlx5i_disable */
+ .update_stats = NULL, /* mlx5i_update_stats */
+ .max_nch = mlx5e_get_max_num_channels,
+ .rx_handlers.handle_rx_cqe = mlx5i_handle_rx_cqe,
+ .rx_handlers.handle_rx_cqe_mpwqe = NULL, /* Not supported */
+ .max_tc = MLX5I_MAX_NUM_TC,
+};
+
+/* mlx5i netdev NDos */
+
+static int mlx5i_dev_init(struct net_device *dev)
+{
+ struct mlx5e_priv *priv = mlx5i_epriv(dev);
+ struct mlx5i_priv *ipriv = priv->ppriv;
+
+ /* Set dev address using underlay QP */
+ dev->dev_addr[1] = (ipriv->qp.qpn >> 16) & 0xff;
+ dev->dev_addr[2] = (ipriv->qp.qpn >> 8) & 0xff;
+ dev->dev_addr[3] = (ipriv->qp.qpn) & 0xff;
+
+ return 0;
+}
+
+static void mlx5i_dev_cleanup(struct net_device *dev)
+{
+ struct mlx5e_priv *priv = mlx5i_epriv(dev);
+ struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5i_priv *ipriv = priv->ppriv;
+ struct mlx5_qp_context context;
+
+ /* detach qp from flow-steering by reset it */
+ mlx5_core_qp_modify(mdev, MLX5_CMD_OP_2RST_QP, 0, &context, &ipriv->qp);
+}
+
+static int mlx5i_open(struct net_device *netdev)
+{
+ struct mlx5e_priv *priv = mlx5i_epriv(netdev);
+ int err;
+
+ mutex_lock(&priv->state_lock);
+
+ set_bit(MLX5E_STATE_OPENED, &priv->state);
+
+ err = mlx5e_open_channels(priv, &priv->channels);
+ if (err)
+ goto err_clear_state_opened_flag;
+
+ mlx5e_refresh_tirs(priv, false);
+ mlx5e_activate_priv_channels(priv);
+ mutex_unlock(&priv->state_lock);
+ return 0;
+
+err_clear_state_opened_flag:
+ clear_bit(MLX5E_STATE_OPENED, &priv->state);
+ mutex_unlock(&priv->state_lock);
+ return err;
+}
+
+static int mlx5i_close(struct net_device *netdev)
+{
+ struct mlx5e_priv *priv = mlx5i_epriv(netdev);
+
+ /* May already be CLOSED in case a previous configuration operation
+ * (e.g RX/TX queue size change) that involves close&open failed.
+ */
+ mutex_lock(&priv->state_lock);
+
+ if (!test_bit(MLX5E_STATE_OPENED, &priv->state))
+ goto unlock;
+
+ clear_bit(MLX5E_STATE_OPENED, &priv->state);
+
+ netif_carrier_off(priv->netdev);
+ mlx5e_deactivate_priv_channels(priv);
+ mlx5e_close_channels(&priv->channels);
+unlock:
+ mutex_unlock(&priv->state_lock);
+ return 0;
+}
+
+#ifdef notusedyet
+/* IPoIB RDMA netdev callbacks */
+static int mlx5i_attach_mcast(struct net_device *netdev, struct ib_device *hca,
+ union ib_gid *gid, u16 lid, int set_qkey)
+{
+ struct mlx5e_priv *epriv = mlx5i_epriv(netdev);
+ struct mlx5_core_dev *mdev = epriv->mdev;
+ struct mlx5i_priv *ipriv = epriv->ppriv;
+ int err;
+
+ mlx5_core_dbg(mdev, "attaching QPN 0x%x, MGID %pI6\n", ipriv->qp.qpn, gid->raw);
+ err = mlx5_core_attach_mcg(mdev, gid, ipriv->qp.qpn);
+ if (err)
+ mlx5_core_warn(mdev, "failed attaching QPN 0x%x, MGID %pI6\n",
+ ipriv->qp.qpn, gid->raw);
+
+ return err;
+}
+
+static int mlx5i_detach_mcast(struct net_device *netdev, struct ib_device *hca,
+ union ib_gid *gid, u16 lid)
+{
+ struct mlx5e_priv *epriv = mlx5i_epriv(netdev);
+ struct mlx5_core_dev *mdev = epriv->mdev;
+ struct mlx5i_priv *ipriv = epriv->ppriv;
+ int err;
+
+ mlx5_core_dbg(mdev, "detaching QPN 0x%x, MGID %pI6\n", ipriv->qp.qpn, gid->raw);
+
+ err = mlx5_core_detach_mcg(mdev, gid, ipriv->qp.qpn);
+ if (err)
+ mlx5_core_dbg(mdev, "failed dettaching QPN 0x%x, MGID %pI6\n",
+ ipriv->qp.qpn, gid->raw);
+
+ return err;
+}
+
+static int mlx5i_xmit(struct net_device *dev, struct sk_buff *skb,
+ struct ib_ah *address, u32 dqpn, u32 dqkey)
+{
+ struct mlx5e_priv *epriv = mlx5i_epriv(dev);
+ struct mlx5e_txqsq *sq = epriv->txq2sq[skb_get_queue_mapping(skb)];
+ struct mlx5_ib_ah *mah = to_mah(address);
+
+ return mlx5i_sq_xmit(sq, skb, &mah->av, dqpn, dqkey);
+}
+#endif
+
+static int mlx5i_check_required_hca_cap(struct mlx5_core_dev *mdev)
+{
+ if (MLX5_CAP_GEN(mdev, port_type) != MLX5_CAP_PORT_TYPE_IB)
+ return -EOPNOTSUPP;
+
+ if (!MLX5_CAP_GEN(mdev, ipoib_enhanced_offloads)) {
+ mlx5_core_warn(mdev, "IPoIB enhanced offloads are not supported\n");
+ return -ENOTSUPP;
+ }
+
+ return 0;
+}
+
+static struct net_device *mlx5_rdma_netdev_alloc(struct mlx5_core_dev *mdev,
+ struct ib_device *ibdev,
+ const char *name,
+ void (*setup)(struct net_device *))
+{
+ const struct mlx5e_profile *profile = &mlx5i_nic_profile;
+ int nch = profile->max_nch(mdev);
+ struct net_device *netdev;
+ struct mlx5i_priv *ipriv;
+ struct mlx5e_priv *epriv;
+ int err;
+
+ if (mlx5i_check_required_hca_cap(mdev)) {
+ mlx5_core_warn(mdev, "Accelerated mode is not supported\n");
+ return ERR_PTR(-EOPNOTSUPP);
+ }
+
+ /* This function should only be called once per mdev */
+ err = mlx5e_create_mdev_resources(mdev);
+ if (err)
+ return NULL;
+
+ netdev = alloc_netdev_mqs(sizeof(struct mlx5i_priv) + sizeof(struct mlx5e_priv),
+ name, NET_NAME_UNKNOWN,
+ setup,
+ nch * MLX5E_MAX_NUM_TC,
+ nch);
+ if (!netdev) {
+ mlx5_core_warn(mdev, "alloc_netdev_mqs failed\n");
+ goto free_mdev_resources;
+ }
+
+ ipriv = netdev_priv(netdev);
+ epriv = mlx5i_epriv(netdev);
+
+ epriv->wq = create_singlethread_workqueue("mlx5i");
+ if (!epriv->wq)
+ goto err_free_netdev;
+
+ profile->init(mdev, netdev, profile, ipriv);
+
+ mlx5e_attach_netdev(epriv);
+ netif_carrier_off(netdev);
+
+ /* TODO: set rdma_netdev func pointers
+ * rn = &ipriv->rn;
+ * rn->hca = ibdev;
+ * rn->send = mlx5i_xmit;
+ * rn->attach_mcast = mlx5i_attach_mcast;
+ * rn->detach_mcast = mlx5i_detach_mcast;
+ */
+ return netdev;
+
+err_free_netdev:
+ free_netdev(netdev);
+free_mdev_resources:
+ mlx5e_destroy_mdev_resources(mdev);
+
+ return NULL;
+}
+EXPORT_SYMBOL(mlx5_rdma_netdev_alloc);
+
+static void mlx5_rdma_netdev_free(struct net_device *netdev)
+{
+ struct mlx5e_priv *priv = mlx5i_epriv(netdev);
+ const struct mlx5e_profile *profile = priv->profile;
+
+ mlx5e_detach_netdev(priv);
+ profile->cleanup(priv);
+ destroy_workqueue(priv->wq);
+ free_netdev(netdev);
+
+ mlx5e_destroy_mdev_resources(priv->mdev);
+}
+EXPORT_SYMBOL(mlx5_rdma_netdev_free);
+
diff --git a/drivers/infiniband/hw/qedr/qedr_hsi.h b/drivers/net/ethernet/mellanox/mlx5/core/ipoib.h
similarity index 60%
rename from drivers/infiniband/hw/qedr/qedr_hsi.h
rename to drivers/net/ethernet/mellanox/mlx5/core/ipoib.h
index 66d2752..bae0a5c 100644
--- a/drivers/infiniband/hw/qedr/qedr_hsi.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/ipoib.h
@@ -1,5 +1,5 @@
-/* QLogic qedr NIC Driver
- * Copyright (c) 2015-2016 QLogic Corporation
+/*
+ * Copyright (c) 2017, Mellanox Technologies. All rights reserved.
*
* This software is available to you under a choice of one of two
* licenses. You may choose to be licensed under the terms of the GNU
@@ -17,7 +17,7 @@
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
- * disclaimer in the documentation and /or other materials
+ * disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
@@ -29,28 +29,26 @@
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
-#ifndef __QED_HSI_ROCE__
-#define __QED_HSI_ROCE__
-#include <linux/qed/common_hsi.h>
-#include <linux/qed/roce_common.h>
-#include "qedr_hsi_rdma.h"
+#ifndef __MLX5E_IPOB_H__
+#define __MLX5E_IPOB_H__
-/* Affiliated asynchronous events / errors enumeration */
-enum roce_async_events_type {
- ROCE_ASYNC_EVENT_NONE = 0,
- ROCE_ASYNC_EVENT_COMM_EST = 1,
- ROCE_ASYNC_EVENT_SQ_DRAINED,
- ROCE_ASYNC_EVENT_SRQ_LIMIT,
- ROCE_ASYNC_EVENT_LAST_WQE_REACHED,
- ROCE_ASYNC_EVENT_CQ_ERR,
- ROCE_ASYNC_EVENT_LOCAL_INVALID_REQUEST_ERR,
- ROCE_ASYNC_EVENT_LOCAL_CATASTROPHIC_ERR,
- ROCE_ASYNC_EVENT_LOCAL_ACCESS_ERR,
- ROCE_ASYNC_EVENT_QP_CATASTROPHIC_ERR,
- ROCE_ASYNC_EVENT_CQ_OVERFLOW_ERR,
- ROCE_ASYNC_EVENT_SRQ_EMPTY,
- MAX_ROCE_ASYNC_EVENTS_TYPE
+#include <linux/mlx5/fs.h>
+#include "en.h"
+
+#define MLX5I_MAX_NUM_TC 1
+
+/* ipoib rdma netdev's private data structure */
+struct mlx5i_priv {
+ struct mlx5_core_qp qp;
+ char *mlx5e_priv[0];
};
-#endif /* __QED_HSI_ROCE__ */
+/* Extract mlx5e_priv from IPoIB netdev */
+#define mlx5i_epriv(netdev) ((void *)(((struct mlx5i_priv *)netdev_priv(netdev))->mlx5e_priv))
+
+netdev_tx_t mlx5i_sq_xmit(struct mlx5e_txqsq *sq, struct sk_buff *skb,
+ struct mlx5_av *av, u32 dqpn, u32 dqkey);
+void mlx5i_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe);
+
+#endif /* __MLX5E_IPOB_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index 60154a17..0c123d5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -1029,7 +1029,7 @@
if (err) {
dev_err(&dev->pdev->dev, "Firmware over %d MS in initializing state, aborting\n",
FW_INIT_TIMEOUT_MILI);
- goto out_err;
+ goto err_cmd_cleanup;
}
err = mlx5_core_enable_hca(dev, 0);
@@ -1280,6 +1280,8 @@
.eswitch_mode_get = mlx5_devlink_eswitch_mode_get,
.eswitch_inline_mode_set = mlx5_devlink_eswitch_inline_mode_set,
.eswitch_inline_mode_get = mlx5_devlink_eswitch_inline_mode_get,
+ .eswitch_encap_mode_set = mlx5_devlink_eswitch_encap_mode_set,
+ .eswitch_encap_mode_get = mlx5_devlink_eswitch_encap_mode_get,
#endif
};
@@ -1514,8 +1516,10 @@
{ PCI_VDEVICE(MELLANOX, 0x1016), MLX5_PCI_DEV_IS_VF}, /* ConnectX-4LX VF */
{ PCI_VDEVICE(MELLANOX, 0x1017) }, /* ConnectX-5, PCIe 3.0 */
{ PCI_VDEVICE(MELLANOX, 0x1018), MLX5_PCI_DEV_IS_VF}, /* ConnectX-5 VF */
- { PCI_VDEVICE(MELLANOX, 0x1019) }, /* ConnectX-5, PCIe 4.0 */
- { PCI_VDEVICE(MELLANOX, 0x101a), MLX5_PCI_DEV_IS_VF}, /* ConnectX-5, PCIe 4.0 VF */
+ { PCI_VDEVICE(MELLANOX, 0x1019) }, /* ConnectX-5 Ex */
+ { PCI_VDEVICE(MELLANOX, 0x101a), MLX5_PCI_DEV_IS_VF}, /* ConnectX-5 Ex VF */
+ { PCI_VDEVICE(MELLANOX, 0x101b) }, /* ConnectX-6 */
+ { PCI_VDEVICE(MELLANOX, 0x101c), MLX5_PCI_DEV_IS_VF}, /* ConnectX-6 VF */
{ 0, }
};
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
index b3dabe6..fbc6e9e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
@@ -141,6 +141,11 @@
u32 *encap_id);
void mlx5_encap_dealloc(struct mlx5_core_dev *dev, u32 encap_id);
+int mlx5_modify_header_alloc(struct mlx5_core_dev *dev,
+ u8 namespace, u8 num_actions,
+ void *modify_actions, u32 *modify_header_id);
+void mlx5_modify_header_dealloc(struct mlx5_core_dev *dev, u32 modify_header_id);
+
bool mlx5_lag_intf_add(struct mlx5_interface *intf, struct mlx5_priv *priv);
int mlx5_query_mtpps(struct mlx5_core_dev *dev, u32 *mtpps, u32 mtpps_size);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/uar.c b/drivers/net/ethernet/mellanox/mlx5/core/uar.c
index 2e6b0f2..222b259 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/uar.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/uar.c
@@ -87,6 +87,7 @@
struct mlx5_uars_page *up = container_of(kref, struct mlx5_uars_page, ref_count);
list_del(&up->list);
+ iounmap(up->map);
if (mlx5_cmd_free_uar(up->mdev, up->index))
mlx5_core_warn(up->mdev, "failed to free uar index %d\n", up->index);
kfree(up->reg_bitmap);
diff --git a/drivers/net/ethernet/mellanox/mlxsw/Makefile b/drivers/net/ethernet/mellanox/mlxsw/Makefile
index 6b6c30d..2fb8c65 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/Makefile
+++ b/drivers/net/ethernet/mellanox/mlxsw/Makefile
@@ -15,7 +15,8 @@
mlxsw_spectrum-objs := spectrum.o spectrum_buffers.o \
spectrum_switchdev.o spectrum_router.o \
spectrum_kvdl.o spectrum_acl_tcam.o \
- spectrum_acl.o spectrum_flower.o
+ spectrum_acl.o spectrum_flower.o \
+ spectrum_cnt.o spectrum_dpipe.o
mlxsw_spectrum-$(CONFIG_MLXSW_SPECTRUM_DCB) += spectrum_dcb.o
obj-$(CONFIG_MLXSW_MINIMAL) += mlxsw_minimal.o
mlxsw_minimal-objs := minimal.o
diff --git a/drivers/net/ethernet/mellanox/mlxsw/cmd.h b/drivers/net/ethernet/mellanox/mlxsw/cmd.h
index a1b4842..479511c 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/cmd.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/cmd.h
@@ -1043,13 +1043,6 @@
*/
MLXSW_ITEM32(cmd_mbox, sw2hw_cq, c_eqn, 0x00, 24, 1);
-/* cmd_mbox_sw2hw_cq_oi
- * When set, overrun ignore is enabled. When set, updates of
- * CQ consumer counter (poll for completion) or Request completion
- * notifications (Arm CQ) DoorBells should not be rung on that CQ.
- */
-MLXSW_ITEM32(cmd_mbox, sw2hw_cq, oi, 0x00, 12, 1);
-
/* cmd_mbox_sw2hw_cq_st
* Event delivery state machine
* 0x0 - FIRED
@@ -1132,11 +1125,6 @@
*/
MLXSW_ITEM32(cmd_mbox, sw2hw_eq, int_msix, 0x00, 24, 1);
-/* cmd_mbox_sw2hw_eq_oi
- * When set, overrun ignore is enabled.
- */
-MLXSW_ITEM32(cmd_mbox, sw2hw_eq, oi, 0x00, 12, 1);
-
/* cmd_mbox_sw2hw_eq_st
* Event delivery state machine
* 0x0 - FIRED
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.c b/drivers/net/ethernet/mellanox/mlxsw/core.c
index a4c0784..affe84e 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.c
@@ -40,9 +40,6 @@
#include <linux/export.h>
#include <linux/err.h>
#include <linux/if_link.h>
-#include <linux/debugfs.h>
-#include <linux/seq_file.h>
-#include <linux/u64_stats_sync.h>
#include <linux/netdevice.h>
#include <linux/completion.h>
#include <linux/skbuff.h>
@@ -74,23 +71,9 @@
static const char mlxsw_core_driver_name[] = "mlxsw_core";
-static struct dentry *mlxsw_core_dbg_root;
-
static struct workqueue_struct *mlxsw_wq;
static struct workqueue_struct *mlxsw_owq;
-struct mlxsw_core_pcpu_stats {
- u64 trap_rx_packets[MLXSW_TRAP_ID_MAX];
- u64 trap_rx_bytes[MLXSW_TRAP_ID_MAX];
- u64 port_rx_packets[MLXSW_PORT_MAX_PORTS];
- u64 port_rx_bytes[MLXSW_PORT_MAX_PORTS];
- struct u64_stats_sync syncp;
- u32 trap_rx_dropped[MLXSW_TRAP_ID_MAX];
- u32 port_rx_dropped[MLXSW_PORT_MAX_PORTS];
- u32 trap_rx_invalid;
- u32 port_rx_invalid;
-};
-
struct mlxsw_core_port {
struct devlink_port devlink_port;
void *port_driver_priv;
@@ -121,23 +104,48 @@
spinlock_t trans_list_lock; /* protects trans_list writes */
bool use_emad;
} emad;
- struct mlxsw_core_pcpu_stats __percpu *pcpu_stats;
- struct dentry *dbg_dir;
- struct {
- struct debugfs_blob_wrapper vsd_blob;
- struct debugfs_blob_wrapper psid_blob;
- } dbg;
struct {
u8 *mapping; /* lag_id+port_index to local_port mapping */
} lag;
struct mlxsw_res res;
struct mlxsw_hwmon *hwmon;
struct mlxsw_thermal *thermal;
- struct mlxsw_core_port ports[MLXSW_PORT_MAX_PORTS];
+ struct mlxsw_core_port *ports;
+ unsigned int max_ports;
unsigned long driver_priv[0];
/* driver_priv has to be always the last item */
};
+#define MLXSW_PORT_MAX_PORTS_DEFAULT 0x40
+
+static int mlxsw_ports_init(struct mlxsw_core *mlxsw_core)
+{
+ /* Switch ports are numbered from 1 to queried value */
+ if (MLXSW_CORE_RES_VALID(mlxsw_core, MAX_SYSTEM_PORT))
+ mlxsw_core->max_ports = MLXSW_CORE_RES_GET(mlxsw_core,
+ MAX_SYSTEM_PORT) + 1;
+ else
+ mlxsw_core->max_ports = MLXSW_PORT_MAX_PORTS_DEFAULT + 1;
+
+ mlxsw_core->ports = kcalloc(mlxsw_core->max_ports,
+ sizeof(struct mlxsw_core_port), GFP_KERNEL);
+ if (!mlxsw_core->ports)
+ return -ENOMEM;
+
+ return 0;
+}
+
+static void mlxsw_ports_fini(struct mlxsw_core *mlxsw_core)
+{
+ kfree(mlxsw_core->ports);
+}
+
+unsigned int mlxsw_core_max_ports(const struct mlxsw_core *mlxsw_core)
+{
+ return mlxsw_core->max_ports;
+}
+EXPORT_SYMBOL(mlxsw_core_max_ports);
+
void *mlxsw_core_driver_priv(struct mlxsw_core *mlxsw_core)
{
return mlxsw_core->driver_priv;
@@ -703,91 +711,6 @@
* Core functions
*****************/
-static int mlxsw_core_rx_stats_dbg_read(struct seq_file *file, void *data)
-{
- struct mlxsw_core *mlxsw_core = file->private;
- struct mlxsw_core_pcpu_stats *p;
- u64 rx_packets, rx_bytes;
- u64 tmp_rx_packets, tmp_rx_bytes;
- u32 rx_dropped, rx_invalid;
- unsigned int start;
- int i;
- int j;
- static const char hdr[] =
- " NUM RX_PACKETS RX_BYTES RX_DROPPED\n";
-
- seq_printf(file, hdr);
- for (i = 0; i < MLXSW_TRAP_ID_MAX; i++) {
- rx_packets = 0;
- rx_bytes = 0;
- rx_dropped = 0;
- for_each_possible_cpu(j) {
- p = per_cpu_ptr(mlxsw_core->pcpu_stats, j);
- do {
- start = u64_stats_fetch_begin(&p->syncp);
- tmp_rx_packets = p->trap_rx_packets[i];
- tmp_rx_bytes = p->trap_rx_bytes[i];
- } while (u64_stats_fetch_retry(&p->syncp, start));
-
- rx_packets += tmp_rx_packets;
- rx_bytes += tmp_rx_bytes;
- rx_dropped += p->trap_rx_dropped[i];
- }
- seq_printf(file, "trap %3d %12llu %12llu %10u\n",
- i, rx_packets, rx_bytes, rx_dropped);
- }
- rx_invalid = 0;
- for_each_possible_cpu(j) {
- p = per_cpu_ptr(mlxsw_core->pcpu_stats, j);
- rx_invalid += p->trap_rx_invalid;
- }
- seq_printf(file, "trap INV %10u\n",
- rx_invalid);
-
- for (i = 0; i < MLXSW_PORT_MAX_PORTS; i++) {
- rx_packets = 0;
- rx_bytes = 0;
- rx_dropped = 0;
- for_each_possible_cpu(j) {
- p = per_cpu_ptr(mlxsw_core->pcpu_stats, j);
- do {
- start = u64_stats_fetch_begin(&p->syncp);
- tmp_rx_packets = p->port_rx_packets[i];
- tmp_rx_bytes = p->port_rx_bytes[i];
- } while (u64_stats_fetch_retry(&p->syncp, start));
-
- rx_packets += tmp_rx_packets;
- rx_bytes += tmp_rx_bytes;
- rx_dropped += p->port_rx_dropped[i];
- }
- seq_printf(file, "port %3d %12llu %12llu %10u\n",
- i, rx_packets, rx_bytes, rx_dropped);
- }
- rx_invalid = 0;
- for_each_possible_cpu(j) {
- p = per_cpu_ptr(mlxsw_core->pcpu_stats, j);
- rx_invalid += p->port_rx_invalid;
- }
- seq_printf(file, "port INV %10u\n",
- rx_invalid);
- return 0;
-}
-
-static int mlxsw_core_rx_stats_dbg_open(struct inode *inode, struct file *f)
-{
- struct mlxsw_core *mlxsw_core = inode->i_private;
-
- return single_open(f, mlxsw_core_rx_stats_dbg_read, mlxsw_core);
-}
-
-static const struct file_operations mlxsw_core_rx_stats_dbg_ops = {
- .owner = THIS_MODULE,
- .open = mlxsw_core_rx_stats_dbg_open,
- .release = single_release,
- .read = seq_read,
- .llseek = seq_lseek
-};
-
int mlxsw_core_driver_register(struct mlxsw_driver *mlxsw_driver)
{
spin_lock(&mlxsw_core_driver_list_lock);
@@ -835,39 +758,13 @@
spin_unlock(&mlxsw_core_driver_list_lock);
}
-static int mlxsw_core_debugfs_init(struct mlxsw_core *mlxsw_core)
-{
- const struct mlxsw_bus_info *bus_info = mlxsw_core->bus_info;
-
- mlxsw_core->dbg_dir = debugfs_create_dir(bus_info->device_name,
- mlxsw_core_dbg_root);
- if (!mlxsw_core->dbg_dir)
- return -ENOMEM;
- debugfs_create_file("rx_stats", S_IRUGO, mlxsw_core->dbg_dir,
- mlxsw_core, &mlxsw_core_rx_stats_dbg_ops);
- mlxsw_core->dbg.vsd_blob.data = (void *) &bus_info->vsd;
- mlxsw_core->dbg.vsd_blob.size = sizeof(bus_info->vsd);
- debugfs_create_blob("vsd", S_IRUGO, mlxsw_core->dbg_dir,
- &mlxsw_core->dbg.vsd_blob);
- mlxsw_core->dbg.psid_blob.data = (void *) &bus_info->psid;
- mlxsw_core->dbg.psid_blob.size = sizeof(bus_info->psid);
- debugfs_create_blob("psid", S_IRUGO, mlxsw_core->dbg_dir,
- &mlxsw_core->dbg.psid_blob);
- return 0;
-}
-
-static void mlxsw_core_debugfs_fini(struct mlxsw_core *mlxsw_core)
-{
- debugfs_remove_recursive(mlxsw_core->dbg_dir);
-}
-
static int mlxsw_devlink_port_split(struct devlink *devlink,
unsigned int port_index,
unsigned int count)
{
struct mlxsw_core *mlxsw_core = devlink_priv(devlink);
- if (port_index >= MLXSW_PORT_MAX_PORTS)
+ if (port_index >= mlxsw_core->max_ports)
return -EINVAL;
if (!mlxsw_core->driver->port_split)
return -EOPNOTSUPP;
@@ -879,7 +776,7 @@
{
struct mlxsw_core *mlxsw_core = devlink_priv(devlink);
- if (port_index >= MLXSW_PORT_MAX_PORTS)
+ if (port_index >= mlxsw_core->max_ports)
return -EINVAL;
if (!mlxsw_core->driver->port_unsplit)
return -EOPNOTSUPP;
@@ -1101,18 +998,15 @@
mlxsw_core->bus_priv = bus_priv;
mlxsw_core->bus_info = mlxsw_bus_info;
- mlxsw_core->pcpu_stats =
- netdev_alloc_pcpu_stats(struct mlxsw_core_pcpu_stats);
- if (!mlxsw_core->pcpu_stats) {
- err = -ENOMEM;
- goto err_alloc_stats;
- }
-
err = mlxsw_bus->init(bus_priv, mlxsw_core, mlxsw_driver->profile,
&mlxsw_core->res);
if (err)
goto err_bus_init;
+ err = mlxsw_ports_init(mlxsw_core);
+ if (err)
+ goto err_ports_init;
+
if (MLXSW_CORE_RES_VALID(mlxsw_core, MAX_LAG) &&
MLXSW_CORE_RES_VALID(mlxsw_core, MAX_LAG_MEMBERS)) {
alloc_size = sizeof(u8) *
@@ -1148,15 +1042,8 @@
goto err_driver_init;
}
- err = mlxsw_core_debugfs_init(mlxsw_core);
- if (err)
- goto err_debugfs_init;
-
return 0;
-err_debugfs_init:
- if (mlxsw_core->driver->fini)
- mlxsw_core->driver->fini(mlxsw_core);
err_driver_init:
mlxsw_thermal_fini(mlxsw_core->thermal);
err_thermal_init:
@@ -1167,10 +1054,10 @@
err_emad_init:
kfree(mlxsw_core->lag.mapping);
err_alloc_lag_mapping:
+ mlxsw_ports_fini(mlxsw_core);
+err_ports_init:
mlxsw_bus->fini(bus_priv);
err_bus_init:
- free_percpu(mlxsw_core->pcpu_stats);
-err_alloc_stats:
devlink_free(devlink);
err_devlink_alloc:
mlxsw_core_driver_put(device_kind);
@@ -1183,15 +1070,14 @@
const char *device_kind = mlxsw_core->bus_info->device_kind;
struct devlink *devlink = priv_to_devlink(mlxsw_core);
- mlxsw_core_debugfs_fini(mlxsw_core);
if (mlxsw_core->driver->fini)
mlxsw_core->driver->fini(mlxsw_core);
mlxsw_thermal_fini(mlxsw_core->thermal);
devlink_unregister(devlink);
mlxsw_emad_fini(mlxsw_core);
kfree(mlxsw_core->lag.mapping);
+ mlxsw_ports_fini(mlxsw_core);
mlxsw_core->bus->fini(mlxsw_core->bus_priv);
- free_percpu(mlxsw_core->pcpu_stats);
devlink_free(devlink);
mlxsw_core_driver_put(device_kind);
}
@@ -1639,7 +1525,6 @@
{
struct mlxsw_rx_listener_item *rxl_item;
const struct mlxsw_rx_listener *rxl;
- struct mlxsw_core_pcpu_stats *pcpu_stats;
u8 local_port;
bool found = false;
@@ -1661,7 +1546,7 @@
__func__, local_port, rx_info->trap_id);
if ((rx_info->trap_id >= MLXSW_TRAP_ID_MAX) ||
- (local_port >= MLXSW_PORT_MAX_PORTS))
+ (local_port >= mlxsw_core->max_ports))
goto drop;
rcu_read_lock();
@@ -1678,26 +1563,10 @@
if (!found)
goto drop;
- pcpu_stats = this_cpu_ptr(mlxsw_core->pcpu_stats);
- u64_stats_update_begin(&pcpu_stats->syncp);
- pcpu_stats->port_rx_packets[local_port]++;
- pcpu_stats->port_rx_bytes[local_port] += skb->len;
- pcpu_stats->trap_rx_packets[rx_info->trap_id]++;
- pcpu_stats->trap_rx_bytes[rx_info->trap_id] += skb->len;
- u64_stats_update_end(&pcpu_stats->syncp);
-
rxl->func(skb, local_port, rxl_item->priv);
return;
drop:
- if (rx_info->trap_id >= MLXSW_TRAP_ID_MAX)
- this_cpu_inc(mlxsw_core->pcpu_stats->trap_rx_invalid);
- else
- this_cpu_inc(mlxsw_core->pcpu_stats->trap_rx_dropped[rx_info->trap_id]);
- if (local_port >= MLXSW_PORT_MAX_PORTS)
- this_cpu_inc(mlxsw_core->pcpu_stats->port_rx_invalid);
- else
- this_cpu_inc(mlxsw_core->pcpu_stats->port_rx_dropped[local_port]);
dev_kfree_skb(skb);
}
EXPORT_SYMBOL(mlxsw_core_skb_receive);
@@ -1926,15 +1795,8 @@
err = -ENOMEM;
goto err_alloc_ordered_workqueue;
}
- mlxsw_core_dbg_root = debugfs_create_dir(mlxsw_core_driver_name, NULL);
- if (!mlxsw_core_dbg_root) {
- err = -ENOMEM;
- goto err_debugfs_create_dir;
- }
return 0;
-err_debugfs_create_dir:
- destroy_workqueue(mlxsw_owq);
err_alloc_ordered_workqueue:
destroy_workqueue(mlxsw_wq);
return err;
@@ -1942,7 +1804,6 @@
static void __exit mlxsw_core_module_exit(void)
{
- debugfs_remove_recursive(mlxsw_core_dbg_root);
destroy_workqueue(mlxsw_owq);
destroy_workqueue(mlxsw_wq);
}
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core.h b/drivers/net/ethernet/mellanox/mlxsw/core.h
index cf38cf9..7fb3539 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/core.h
@@ -57,6 +57,8 @@
struct mlxsw_bus;
struct mlxsw_bus_info;
+unsigned int mlxsw_core_max_ports(const struct mlxsw_core *mlxsw_core);
+
void *mlxsw_core_driver_priv(struct mlxsw_core *mlxsw_core);
int mlxsw_core_driver_register(struct mlxsw_driver *mlxsw_driver);
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core_acl_flex_actions.c b/drivers/net/ethernet/mellanox/mlxsw/core_acl_flex_actions.c
index 5f337715..46304ff 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core_acl_flex_actions.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/core_acl_flex_actions.c
@@ -567,6 +567,89 @@
return oneact + MLXSW_AFA_PAYLOAD_OFFSET;
}
+/* VLAN Action
+ * -----------
+ * VLAN action is used for manipulating VLANs. It can be used to implement QinQ,
+ * VLAN translation, change of PCP bits of the VLAN tag, push, pop as swap VLANs
+ * and more.
+ */
+
+#define MLXSW_AFA_VLAN_CODE 0x02
+#define MLXSW_AFA_VLAN_SIZE 1
+
+enum mlxsw_afa_vlan_vlan_tag_cmd {
+ MLXSW_AFA_VLAN_VLAN_TAG_CMD_NOP,
+ MLXSW_AFA_VLAN_VLAN_TAG_CMD_PUSH_TAG,
+ MLXSW_AFA_VLAN_VLAN_TAG_CMD_POP_TAG,
+};
+
+enum mlxsw_afa_vlan_cmd {
+ MLXSW_AFA_VLAN_CMD_NOP,
+ MLXSW_AFA_VLAN_CMD_SET_OUTER,
+ MLXSW_AFA_VLAN_CMD_SET_INNER,
+ MLXSW_AFA_VLAN_CMD_COPY_OUTER_TO_INNER,
+ MLXSW_AFA_VLAN_CMD_COPY_INNER_TO_OUTER,
+ MLXSW_AFA_VLAN_CMD_SWAP,
+};
+
+/* afa_vlan_vlan_tag_cmd
+ * Tag command: push, pop, nop VLAN header.
+ */
+MLXSW_ITEM32(afa, vlan, vlan_tag_cmd, 0x00, 29, 3);
+
+/* afa_vlan_vid_cmd */
+MLXSW_ITEM32(afa, vlan, vid_cmd, 0x04, 29, 3);
+
+/* afa_vlan_vid */
+MLXSW_ITEM32(afa, vlan, vid, 0x04, 0, 12);
+
+/* afa_vlan_ethertype_cmd */
+MLXSW_ITEM32(afa, vlan, ethertype_cmd, 0x08, 29, 3);
+
+/* afa_vlan_ethertype
+ * Index to EtherTypes in Switch VLAN EtherType Register (SVER).
+ */
+MLXSW_ITEM32(afa, vlan, ethertype, 0x08, 24, 3);
+
+/* afa_vlan_pcp_cmd */
+MLXSW_ITEM32(afa, vlan, pcp_cmd, 0x08, 13, 3);
+
+/* afa_vlan_pcp */
+MLXSW_ITEM32(afa, vlan, pcp, 0x08, 8, 3);
+
+static inline void
+mlxsw_afa_vlan_pack(char *payload,
+ enum mlxsw_afa_vlan_vlan_tag_cmd vlan_tag_cmd,
+ enum mlxsw_afa_vlan_cmd vid_cmd, u16 vid,
+ enum mlxsw_afa_vlan_cmd pcp_cmd, u8 pcp,
+ enum mlxsw_afa_vlan_cmd ethertype_cmd, u8 ethertype)
+{
+ mlxsw_afa_vlan_vlan_tag_cmd_set(payload, vlan_tag_cmd);
+ mlxsw_afa_vlan_vid_cmd_set(payload, vid_cmd);
+ mlxsw_afa_vlan_vid_set(payload, vid);
+ mlxsw_afa_vlan_pcp_cmd_set(payload, pcp_cmd);
+ mlxsw_afa_vlan_pcp_set(payload, pcp);
+ mlxsw_afa_vlan_ethertype_cmd_set(payload, ethertype_cmd);
+ mlxsw_afa_vlan_ethertype_set(payload, ethertype);
+}
+
+int mlxsw_afa_block_append_vlan_modify(struct mlxsw_afa_block *block,
+ u16 vid, u8 pcp, u8 et)
+{
+ char *act = mlxsw_afa_block_append_action(block,
+ MLXSW_AFA_VLAN_CODE,
+ MLXSW_AFA_VLAN_SIZE);
+
+ if (!act)
+ return -ENOBUFS;
+ mlxsw_afa_vlan_pack(act, MLXSW_AFA_VLAN_VLAN_TAG_CMD_NOP,
+ MLXSW_AFA_VLAN_CMD_SET_OUTER, vid,
+ MLXSW_AFA_VLAN_CMD_SET_OUTER, pcp,
+ MLXSW_AFA_VLAN_CMD_SET_OUTER, et);
+ return 0;
+}
+EXPORT_SYMBOL(mlxsw_afa_block_append_vlan_modify);
+
/* Trap / Discard Action
* ---------------------
* The Trap / Discard action enables trapping / mirroring packets to the CPU
@@ -677,3 +760,98 @@
return err;
}
EXPORT_SYMBOL(mlxsw_afa_block_append_fwd);
+
+/* Policing and Counting Action
+ * ----------------------------
+ * Policing and Counting action is used for binding policer and counter
+ * to ACL rules.
+ */
+
+#define MLXSW_AFA_POLCNT_CODE 0x08
+#define MLXSW_AFA_POLCNT_SIZE 1
+
+enum mlxsw_afa_polcnt_counter_set_type {
+ /* No count */
+ MLXSW_AFA_POLCNT_COUNTER_SET_TYPE_NO_COUNT = 0x00,
+ /* Count packets and bytes */
+ MLXSW_AFA_POLCNT_COUNTER_SET_TYPE_PACKETS_BYTES = 0x03,
+ /* Count only packets */
+ MLXSW_AFA_POLCNT_COUNTER_SET_TYPE_PACKETS = 0x05,
+};
+
+/* afa_polcnt_counter_set_type
+ * Counter set type for flow counters.
+ */
+MLXSW_ITEM32(afa, polcnt, counter_set_type, 0x04, 24, 8);
+
+/* afa_polcnt_counter_index
+ * Counter index for flow counters.
+ */
+MLXSW_ITEM32(afa, polcnt, counter_index, 0x04, 0, 24);
+
+static inline void
+mlxsw_afa_polcnt_pack(char *payload,
+ enum mlxsw_afa_polcnt_counter_set_type set_type,
+ u32 counter_index)
+{
+ mlxsw_afa_polcnt_counter_set_type_set(payload, set_type);
+ mlxsw_afa_polcnt_counter_index_set(payload, counter_index);
+}
+
+int mlxsw_afa_block_append_counter(struct mlxsw_afa_block *block,
+ u32 counter_index)
+{
+ char *act = mlxsw_afa_block_append_action(block,
+ MLXSW_AFA_POLCNT_CODE,
+ MLXSW_AFA_POLCNT_SIZE);
+ if (!act)
+ return -ENOBUFS;
+ mlxsw_afa_polcnt_pack(act, MLXSW_AFA_POLCNT_COUNTER_SET_TYPE_PACKETS_BYTES,
+ counter_index);
+ return 0;
+}
+EXPORT_SYMBOL(mlxsw_afa_block_append_counter);
+
+/* Virtual Router and Forwarding Domain Action
+ * -------------------------------------------
+ * Virtual Switch action is used for manipulate the Virtual Router (VR),
+ * MPLS label space and the Forwarding Identifier (FID).
+ */
+
+#define MLXSW_AFA_VIRFWD_CODE 0x0E
+#define MLXSW_AFA_VIRFWD_SIZE 1
+
+enum mlxsw_afa_virfwd_fid_cmd {
+ /* Do nothing */
+ MLXSW_AFA_VIRFWD_FID_CMD_NOOP,
+ /* Set the Forwarding Identifier (FID) to fid */
+ MLXSW_AFA_VIRFWD_FID_CMD_SET,
+};
+
+/* afa_virfwd_fid_cmd */
+MLXSW_ITEM32(afa, virfwd, fid_cmd, 0x08, 29, 3);
+
+/* afa_virfwd_fid
+ * The FID value.
+ */
+MLXSW_ITEM32(afa, virfwd, fid, 0x08, 0, 16);
+
+static inline void mlxsw_afa_virfwd_pack(char *payload,
+ enum mlxsw_afa_virfwd_fid_cmd fid_cmd,
+ u16 fid)
+{
+ mlxsw_afa_virfwd_fid_cmd_set(payload, fid_cmd);
+ mlxsw_afa_virfwd_fid_set(payload, fid);
+}
+
+int mlxsw_afa_block_append_fid_set(struct mlxsw_afa_block *block, u16 fid)
+{
+ char *act = mlxsw_afa_block_append_action(block,
+ MLXSW_AFA_VIRFWD_CODE,
+ MLXSW_AFA_VIRFWD_SIZE);
+ if (!act)
+ return -ENOBUFS;
+ mlxsw_afa_virfwd_pack(act, MLXSW_AFA_VIRFWD_FID_CMD_SET, fid);
+ return 0;
+}
+EXPORT_SYMBOL(mlxsw_afa_block_append_fid_set);
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core_acl_flex_actions.h b/drivers/net/ethernet/mellanox/mlxsw/core_acl_flex_actions.h
index 43f78dc..bd8b91d 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core_acl_flex_actions.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/core_acl_flex_actions.h
@@ -62,5 +62,10 @@
int mlxsw_afa_block_append_drop(struct mlxsw_afa_block *block);
int mlxsw_afa_block_append_fwd(struct mlxsw_afa_block *block,
u8 local_port, bool in_port);
+int mlxsw_afa_block_append_vlan_modify(struct mlxsw_afa_block *block,
+ u16 vid, u8 pcp, u8 et);
+int mlxsw_afa_block_append_counter(struct mlxsw_afa_block *block,
+ u32 counter_index);
+int mlxsw_afa_block_append_fid_set(struct mlxsw_afa_block *block, u16 fid);
#endif
diff --git a/drivers/net/ethernet/mellanox/mlxsw/core_acl_flex_keys.h b/drivers/net/ethernet/mellanox/mlxsw/core_acl_flex_keys.h
index e4fcba7..c75e914 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/core_acl_flex_keys.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/core_acl_flex_keys.h
@@ -54,6 +54,8 @@
MLXSW_AFK_ELEMENT_DST_IP6_LO,
MLXSW_AFK_ELEMENT_DST_L4_PORT,
MLXSW_AFK_ELEMENT_SRC_L4_PORT,
+ MLXSW_AFK_ELEMENT_VID,
+ MLXSW_AFK_ELEMENT_PCP,
MLXSW_AFK_ELEMENT_MAX,
};
@@ -88,7 +90,7 @@
MLXSW_AFK_ELEMENT_INFO(MLXSW_AFK_ELEMENT_TYPE_BUF, \
_element, _offset, 0, _size)
-/* For the purpose of the driver, define a internal storage scratchpad
+/* For the purpose of the driver, define an internal storage scratchpad
* that will be used to store key/mask values. For each defined element type
* define an internal storage geometry.
*/
@@ -98,6 +100,8 @@
MLXSW_AFK_ELEMENT_INFO_BUF(SMAC, 0x0A, 6),
MLXSW_AFK_ELEMENT_INFO_U32(ETHERTYPE, 0x00, 0, 16),
MLXSW_AFK_ELEMENT_INFO_U32(IP_PROTO, 0x10, 0, 8),
+ MLXSW_AFK_ELEMENT_INFO_U32(VID, 0x10, 8, 12),
+ MLXSW_AFK_ELEMENT_INFO_U32(PCP, 0x10, 20, 3),
MLXSW_AFK_ELEMENT_INFO_U32(SRC_IP4, 0x18, 0, 32),
MLXSW_AFK_ELEMENT_INFO_U32(DST_IP4, 0x1C, 0, 32),
MLXSW_AFK_ELEMENT_INFO_BUF(SRC_IP6_HI, 0x18, 8),
diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c
index a223c85..23f7d82 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/pci.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c
@@ -44,8 +44,6 @@
#include <linux/skbuff.h>
#include <linux/if_vlan.h>
#include <linux/log2.h>
-#include <linux/debugfs.h>
-#include <linux/seq_file.h>
#include <linux/string.h>
#include "pci_hw.h"
@@ -57,8 +55,6 @@
static const char mlxsw_pci_driver_name[] = "mlxsw_pci";
-static struct dentry *mlxsw_pci_dbg_root;
-
#define mlxsw_pci_write32(mlxsw_pci, reg, val) \
iowrite32be(val, (mlxsw_pci)->hw_addr + (MLXSW_PCI_ ## reg))
#define mlxsw_pci_read32(mlxsw_pci, reg) \
@@ -71,21 +67,6 @@
MLXSW_PCI_QUEUE_TYPE_EQ,
};
-static const char *mlxsw_pci_queue_type_str(enum mlxsw_pci_queue_type q_type)
-{
- switch (q_type) {
- case MLXSW_PCI_QUEUE_TYPE_SDQ:
- return "sdq";
- case MLXSW_PCI_QUEUE_TYPE_RDQ:
- return "rdq";
- case MLXSW_PCI_QUEUE_TYPE_CQ:
- return "cq";
- case MLXSW_PCI_QUEUE_TYPE_EQ:
- return "eq";
- }
- BUG();
-}
-
#define MLXSW_PCI_QUEUE_TYPE_COUNT 4
static const u16 mlxsw_pci_doorbell_type_offset[] = {
@@ -155,7 +136,6 @@
u8 __iomem *hw_addr;
struct mlxsw_pci_queue_type_group queues[MLXSW_PCI_QUEUE_TYPE_COUNT];
u32 doorbell_offset;
- struct msix_entry msix_entry;
struct mlxsw_core *core;
struct {
struct mlxsw_pci_mem_item *items;
@@ -174,7 +154,6 @@
} comp;
} cmd;
struct mlxsw_bus_info bus_info;
- struct dentry *dbg_dir;
};
static void mlxsw_pci_queue_tasklet_schedule(struct mlxsw_pci_queue *q)
@@ -261,21 +240,11 @@
return __mlxsw_pci_queue_count(mlxsw_pci, MLXSW_PCI_QUEUE_TYPE_SDQ);
}
-static u8 mlxsw_pci_rdq_count(struct mlxsw_pci *mlxsw_pci)
-{
- return __mlxsw_pci_queue_count(mlxsw_pci, MLXSW_PCI_QUEUE_TYPE_RDQ);
-}
-
static u8 mlxsw_pci_cq_count(struct mlxsw_pci *mlxsw_pci)
{
return __mlxsw_pci_queue_count(mlxsw_pci, MLXSW_PCI_QUEUE_TYPE_CQ);
}
-static u8 mlxsw_pci_eq_count(struct mlxsw_pci *mlxsw_pci)
-{
- return __mlxsw_pci_queue_count(mlxsw_pci, MLXSW_PCI_QUEUE_TYPE_EQ);
-}
-
static struct mlxsw_pci_queue *
__mlxsw_pci_queue_get(struct mlxsw_pci *mlxsw_pci,
enum mlxsw_pci_queue_type q_type, u8 q_num)
@@ -390,26 +359,6 @@
mlxsw_cmd_hw2sw_sdq(mlxsw_pci->core, q->num);
}
-static int mlxsw_pci_sdq_dbg_read(struct seq_file *file, void *data)
-{
- struct mlxsw_pci *mlxsw_pci = dev_get_drvdata(file->private);
- struct mlxsw_pci_queue *q;
- int i;
- static const char hdr[] =
- "NUM PROD_COUNT CONS_COUNT COUNT\n";
-
- seq_printf(file, hdr);
- for (i = 0; i < mlxsw_pci_sdq_count(mlxsw_pci); i++) {
- q = mlxsw_pci_sdq_get(mlxsw_pci, i);
- spin_lock_bh(&q->lock);
- seq_printf(file, "%3d %10d %10d %5d\n",
- i, q->producer_counter, q->consumer_counter,
- q->count);
- spin_unlock_bh(&q->lock);
- }
- return 0;
-}
-
static int mlxsw_pci_wqe_frag_map(struct mlxsw_pci *mlxsw_pci, char *wqe,
int index, char *frag_data, size_t frag_len,
int direction)
@@ -544,26 +493,6 @@
}
}
-static int mlxsw_pci_rdq_dbg_read(struct seq_file *file, void *data)
-{
- struct mlxsw_pci *mlxsw_pci = dev_get_drvdata(file->private);
- struct mlxsw_pci_queue *q;
- int i;
- static const char hdr[] =
- "NUM PROD_COUNT CONS_COUNT COUNT\n";
-
- seq_printf(file, hdr);
- for (i = 0; i < mlxsw_pci_rdq_count(mlxsw_pci); i++) {
- q = mlxsw_pci_rdq_get(mlxsw_pci, i);
- spin_lock_bh(&q->lock);
- seq_printf(file, "%3d %10d %10d %5d\n",
- i, q->producer_counter, q->consumer_counter,
- q->count);
- spin_unlock_bh(&q->lock);
- }
- return 0;
-}
-
static int mlxsw_pci_cq_init(struct mlxsw_pci *mlxsw_pci, char *mbox,
struct mlxsw_pci_queue *q)
{
@@ -580,7 +509,6 @@
mlxsw_cmd_mbox_sw2hw_cq_cv_set(mbox, 0); /* CQE ver 0 */
mlxsw_cmd_mbox_sw2hw_cq_c_eqn_set(mbox, MLXSW_PCI_EQ_COMP_NUM);
- mlxsw_cmd_mbox_sw2hw_cq_oi_set(mbox, 0);
mlxsw_cmd_mbox_sw2hw_cq_st_set(mbox, 0);
mlxsw_cmd_mbox_sw2hw_cq_log_cq_size_set(mbox, ilog2(q->count));
for (i = 0; i < MLXSW_PCI_AQ_PAGES; i++) {
@@ -602,27 +530,6 @@
mlxsw_cmd_hw2sw_cq(mlxsw_pci->core, q->num);
}
-static int mlxsw_pci_cq_dbg_read(struct seq_file *file, void *data)
-{
- struct mlxsw_pci *mlxsw_pci = dev_get_drvdata(file->private);
-
- struct mlxsw_pci_queue *q;
- int i;
- static const char hdr[] =
- "NUM CONS_INDEX SDQ_COUNT RDQ_COUNT COUNT\n";
-
- seq_printf(file, hdr);
- for (i = 0; i < mlxsw_pci_cq_count(mlxsw_pci); i++) {
- q = mlxsw_pci_cq_get(mlxsw_pci, i);
- spin_lock_bh(&q->lock);
- seq_printf(file, "%3d %10d %10d %10d %5d\n",
- i, q->consumer_counter, q->u.cq.comp_sdq_count,
- q->u.cq.comp_rdq_count, q->count);
- spin_unlock_bh(&q->lock);
- }
- return 0;
-}
-
static void mlxsw_pci_cqe_sdq_handle(struct mlxsw_pci *mlxsw_pci,
struct mlxsw_pci_queue *q,
u16 consumer_counter_limit,
@@ -755,7 +662,6 @@
}
mlxsw_cmd_mbox_sw2hw_eq_int_msix_set(mbox, 1); /* MSI-X used */
- mlxsw_cmd_mbox_sw2hw_eq_oi_set(mbox, 0);
mlxsw_cmd_mbox_sw2hw_eq_st_set(mbox, 1); /* armed */
mlxsw_cmd_mbox_sw2hw_eq_log_eq_size_set(mbox, ilog2(q->count));
for (i = 0; i < MLXSW_PCI_AQ_PAGES; i++) {
@@ -777,27 +683,6 @@
mlxsw_cmd_hw2sw_eq(mlxsw_pci->core, q->num);
}
-static int mlxsw_pci_eq_dbg_read(struct seq_file *file, void *data)
-{
- struct mlxsw_pci *mlxsw_pci = dev_get_drvdata(file->private);
- struct mlxsw_pci_queue *q;
- int i;
- static const char hdr[] =
- "NUM CONS_COUNT EV_CMD EV_COMP EV_OTHER COUNT\n";
-
- seq_printf(file, hdr);
- for (i = 0; i < mlxsw_pci_eq_count(mlxsw_pci); i++) {
- q = mlxsw_pci_eq_get(mlxsw_pci, i);
- spin_lock_bh(&q->lock);
- seq_printf(file, "%3d %10d %10d %10d %10d %5d\n",
- i, q->consumer_counter, q->u.eq.ev_cmd_count,
- q->u.eq.ev_comp_count, q->u.eq.ev_other_count,
- q->count);
- spin_unlock_bh(&q->lock);
- }
- return 0;
-}
-
static void mlxsw_pci_eq_cmd_event(struct mlxsw_pci *mlxsw_pci, char *eqe)
{
mlxsw_pci->cmd.comp.status = mlxsw_pci_eqe_cmd_status_get(eqe);
@@ -868,7 +753,6 @@
void (*fini)(struct mlxsw_pci *mlxsw_pci,
struct mlxsw_pci_queue *q);
void (*tasklet)(unsigned long data);
- int (*dbg_read)(struct seq_file *s, void *data);
u16 elem_count;
u8 elem_size;
};
@@ -877,7 +761,6 @@
.type = MLXSW_PCI_QUEUE_TYPE_SDQ,
.init = mlxsw_pci_sdq_init,
.fini = mlxsw_pci_sdq_fini,
- .dbg_read = mlxsw_pci_sdq_dbg_read,
.elem_count = MLXSW_PCI_WQE_COUNT,
.elem_size = MLXSW_PCI_WQE_SIZE,
};
@@ -886,7 +769,6 @@
.type = MLXSW_PCI_QUEUE_TYPE_RDQ,
.init = mlxsw_pci_rdq_init,
.fini = mlxsw_pci_rdq_fini,
- .dbg_read = mlxsw_pci_rdq_dbg_read,
.elem_count = MLXSW_PCI_WQE_COUNT,
.elem_size = MLXSW_PCI_WQE_SIZE
};
@@ -896,7 +778,6 @@
.init = mlxsw_pci_cq_init,
.fini = mlxsw_pci_cq_fini,
.tasklet = mlxsw_pci_cq_tasklet,
- .dbg_read = mlxsw_pci_cq_dbg_read,
.elem_count = MLXSW_PCI_CQE_COUNT,
.elem_size = MLXSW_PCI_CQE_SIZE
};
@@ -906,7 +787,6 @@
.init = mlxsw_pci_eq_init,
.fini = mlxsw_pci_eq_fini,
.tasklet = mlxsw_pci_eq_tasklet,
- .dbg_read = mlxsw_pci_eq_dbg_read,
.elem_count = MLXSW_PCI_EQE_COUNT,
.elem_size = MLXSW_PCI_EQE_SIZE
};
@@ -984,9 +864,7 @@
const struct mlxsw_pci_queue_ops *q_ops,
u8 num_qs)
{
- struct pci_dev *pdev = mlxsw_pci->pdev;
struct mlxsw_pci_queue_type_group *queue_group;
- char tmp[16];
int i;
int err;
@@ -1003,10 +881,6 @@
}
queue_group->count = num_qs;
- sprintf(tmp, "%s_stats", mlxsw_pci_queue_type_str(q_ops->type));
- debugfs_create_devm_seqfile(&pdev->dev, tmp, mlxsw_pci->dbg_dir,
- q_ops->dbg_read);
-
return 0;
err_queue_init:
@@ -1534,7 +1408,7 @@
if (err)
goto err_aqs_init;
- err = request_irq(mlxsw_pci->msix_entry.vector,
+ err = request_irq(pci_irq_vector(pdev, 0),
mlxsw_pci_eq_irq_handler, 0,
mlxsw_pci->bus_info.device_kind, mlxsw_pci);
if (err) {
@@ -1567,7 +1441,7 @@
{
struct mlxsw_pci *mlxsw_pci = bus_priv;
- free_irq(mlxsw_pci->msix_entry.vector, mlxsw_pci);
+ free_irq(pci_irq_vector(mlxsw_pci->pdev, 0), mlxsw_pci);
mlxsw_pci_aqs_fini(mlxsw_pci);
mlxsw_pci_fw_area_fini(mlxsw_pci);
mlxsw_pci_mbox_free(mlxsw_pci, &mlxsw_pci->cmd.out_mbox);
@@ -1842,8 +1716,8 @@
goto err_sw_reset;
}
- err = pci_enable_msix_exact(pdev, &mlxsw_pci->msix_entry, 1);
- if (err) {
+ err = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSIX);
+ if (err < 0) {
dev_err(&pdev->dev, "MSI-X init failed\n");
goto err_msix_init;
}
@@ -1852,14 +1726,6 @@
mlxsw_pci->bus_info.device_name = pci_name(mlxsw_pci->pdev);
mlxsw_pci->bus_info.dev = &pdev->dev;
- mlxsw_pci->dbg_dir = debugfs_create_dir(mlxsw_pci->bus_info.device_name,
- mlxsw_pci_dbg_root);
- if (!mlxsw_pci->dbg_dir) {
- dev_err(&pdev->dev, "Failed to create debugfs dir\n");
- err = -ENOMEM;
- goto err_dbg_create_dir;
- }
-
err = mlxsw_core_bus_device_register(&mlxsw_pci->bus_info,
&mlxsw_pci_bus, mlxsw_pci);
if (err) {
@@ -1870,9 +1736,7 @@
return 0;
err_bus_device_register:
- debugfs_remove_recursive(mlxsw_pci->dbg_dir);
-err_dbg_create_dir:
- pci_disable_msix(mlxsw_pci->pdev);
+ pci_free_irq_vectors(mlxsw_pci->pdev);
err_msix_init:
err_sw_reset:
iounmap(mlxsw_pci->hw_addr);
@@ -1892,8 +1756,7 @@
struct mlxsw_pci *mlxsw_pci = pci_get_drvdata(pdev);
mlxsw_core_bus_device_unregister(mlxsw_pci->core);
- debugfs_remove_recursive(mlxsw_pci->dbg_dir);
- pci_disable_msix(mlxsw_pci->pdev);
+ pci_free_irq_vectors(mlxsw_pci->pdev);
iounmap(mlxsw_pci->hw_addr);
pci_release_regions(mlxsw_pci->pdev);
pci_disable_device(mlxsw_pci->pdev);
@@ -1916,15 +1779,11 @@
static int __init mlxsw_pci_module_init(void)
{
- mlxsw_pci_dbg_root = debugfs_create_dir(mlxsw_pci_driver_name, NULL);
- if (!mlxsw_pci_dbg_root)
- return -ENOMEM;
return 0;
}
static void __exit mlxsw_pci_module_exit(void)
{
- debugfs_remove_recursive(mlxsw_pci_dbg_root);
}
module_init(mlxsw_pci_module_init);
diff --git a/drivers/net/ethernet/mellanox/mlxsw/port.h b/drivers/net/ethernet/mellanox/mlxsw/port.h
index 3d42146..c580abb 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/port.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/port.h
@@ -49,20 +49,12 @@
#define MLXSW_PORT_MID 0xd000
-#define MLXSW_PORT_MAX_PHY_PORTS 0x40
-#define MLXSW_PORT_MAX_PORTS (MLXSW_PORT_MAX_PHY_PORTS + 1)
-
#define MLXSW_PORT_MAX_IB_PHY_PORTS 36
#define MLXSW_PORT_MAX_IB_PORTS (MLXSW_PORT_MAX_IB_PHY_PORTS + 1)
-#define MLXSW_PORT_DEVID_BITS_OFFSET 10
-#define MLXSW_PORT_PHY_BITS_OFFSET 4
-#define MLXSW_PORT_PHY_BITS_MASK (MLXSW_PORT_MAX_PHY_PORTS - 1)
-
#define MLXSW_PORT_CPU_PORT 0x0
-#define MLXSW_PORT_ROUTER_PORT (MLXSW_PORT_MAX_PHY_PORTS + 2)
-#define MLXSW_PORT_DONT_CARE (MLXSW_PORT_MAX_PORTS)
+#define MLXSW_PORT_DONT_CARE 0xFF
#define MLXSW_PORT_MODULE_MAX_WIDTH 4
diff --git a/drivers/net/ethernet/mellanox/mlxsw/reg.h b/drivers/net/ethernet/mellanox/mlxsw/reg.h
index d9616da..83b277c 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/reg.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/reg.h
@@ -4125,6 +4125,60 @@
*/
MLXSW_ITEM32(reg, ritr, sp_if_vid, 0x18, 0, 12);
+/* Shared between ingress/egress */
+enum mlxsw_reg_ritr_counter_set_type {
+ /* No Count. */
+ MLXSW_REG_RITR_COUNTER_SET_TYPE_NO_COUNT = 0x0,
+ /* Basic. Used for router interfaces, counting the following:
+ * - Error and Discard counters.
+ * - Unicast, Multicast and Broadcast counters. Sharing the
+ * same set of counters for the different type of traffic
+ * (IPv4, IPv6 and mpls).
+ */
+ MLXSW_REG_RITR_COUNTER_SET_TYPE_BASIC = 0x9,
+};
+
+/* reg_ritr_ingress_counter_index
+ * Counter Index for flow counter.
+ * Access: RW
+ */
+MLXSW_ITEM32(reg, ritr, ingress_counter_index, 0x38, 0, 24);
+
+/* reg_ritr_ingress_counter_set_type
+ * Igress Counter Set Type for router interface counter.
+ * Access: RW
+ */
+MLXSW_ITEM32(reg, ritr, ingress_counter_set_type, 0x38, 24, 8);
+
+/* reg_ritr_egress_counter_index
+ * Counter Index for flow counter.
+ * Access: RW
+ */
+MLXSW_ITEM32(reg, ritr, egress_counter_index, 0x3C, 0, 24);
+
+/* reg_ritr_egress_counter_set_type
+ * Egress Counter Set Type for router interface counter.
+ * Access: RW
+ */
+MLXSW_ITEM32(reg, ritr, egress_counter_set_type, 0x3C, 24, 8);
+
+static inline void mlxsw_reg_ritr_counter_pack(char *payload, u32 index,
+ bool enable, bool egress)
+{
+ enum mlxsw_reg_ritr_counter_set_type set_type;
+
+ if (enable)
+ set_type = MLXSW_REG_RITR_COUNTER_SET_TYPE_BASIC;
+ else
+ set_type = MLXSW_REG_RITR_COUNTER_SET_TYPE_NO_COUNT;
+ mlxsw_reg_ritr_egress_counter_set_type_set(payload, set_type);
+
+ if (egress)
+ mlxsw_reg_ritr_egress_counter_index_set(payload, index);
+ else
+ mlxsw_reg_ritr_ingress_counter_index_set(payload, index);
+}
+
static inline void mlxsw_reg_ritr_rif_pack(char *payload, u16 rif)
{
MLXSW_REG_ZERO(ritr, payload);
@@ -4141,7 +4195,8 @@
static inline void mlxsw_reg_ritr_pack(char *payload, bool enable,
enum mlxsw_reg_ritr_if_type type,
- u16 rif, u16 mtu, const char *mac)
+ u16 rif, u16 vr_id, u16 mtu,
+ const char *mac)
{
bool op = enable ? MLXSW_REG_RITR_RIF_CREATE : MLXSW_REG_RITR_RIF_DEL;
@@ -4153,6 +4208,7 @@
mlxsw_reg_ritr_rif_set(payload, rif);
mlxsw_reg_ritr_ipv4_fe_set(payload, 1);
mlxsw_reg_ritr_lb_en_set(payload, 1);
+ mlxsw_reg_ritr_virtual_router_set(payload, vr_id);
mlxsw_reg_ritr_mtu_set(payload, mtu);
mlxsw_reg_ritr_if_mac_memcpy_to(payload, mac);
}
@@ -4285,6 +4341,129 @@
mlxsw_reg_ratr_eth_destination_mac_memcpy_to(payload, dest_mac);
}
+/* RICNT - Router Interface Counter Register
+ * -----------------------------------------
+ * The RICNT register retrieves per port performance counters
+ */
+#define MLXSW_REG_RICNT_ID 0x800B
+#define MLXSW_REG_RICNT_LEN 0x100
+
+MLXSW_REG_DEFINE(ricnt, MLXSW_REG_RICNT_ID, MLXSW_REG_RICNT_LEN);
+
+/* reg_ricnt_counter_index
+ * Counter index
+ * Access: RW
+ */
+MLXSW_ITEM32(reg, ricnt, counter_index, 0x04, 0, 24);
+
+enum mlxsw_reg_ricnt_counter_set_type {
+ /* No Count. */
+ MLXSW_REG_RICNT_COUNTER_SET_TYPE_NO_COUNT = 0x00,
+ /* Basic. Used for router interfaces, counting the following:
+ * - Error and Discard counters.
+ * - Unicast, Multicast and Broadcast counters. Sharing the
+ * same set of counters for the different type of traffic
+ * (IPv4, IPv6 and mpls).
+ */
+ MLXSW_REG_RICNT_COUNTER_SET_TYPE_BASIC = 0x09,
+};
+
+/* reg_ricnt_counter_set_type
+ * Counter Set Type for router interface counter
+ * Access: RW
+ */
+MLXSW_ITEM32(reg, ricnt, counter_set_type, 0x04, 24, 8);
+
+enum mlxsw_reg_ricnt_opcode {
+ /* Nop. Supported only for read access*/
+ MLXSW_REG_RICNT_OPCODE_NOP = 0x00,
+ /* Clear. Setting the clr bit will reset the counter value for
+ * all counters of the specified Router Interface.
+ */
+ MLXSW_REG_RICNT_OPCODE_CLEAR = 0x08,
+};
+
+/* reg_ricnt_opcode
+ * Opcode
+ * Access: RW
+ */
+MLXSW_ITEM32(reg, ricnt, op, 0x00, 28, 4);
+
+/* reg_ricnt_good_unicast_packets
+ * good unicast packets.
+ * Access: RW
+ */
+MLXSW_ITEM64(reg, ricnt, good_unicast_packets, 0x08, 0, 64);
+
+/* reg_ricnt_good_multicast_packets
+ * good multicast packets.
+ * Access: RW
+ */
+MLXSW_ITEM64(reg, ricnt, good_multicast_packets, 0x10, 0, 64);
+
+/* reg_ricnt_good_broadcast_packets
+ * good broadcast packets
+ * Access: RW
+ */
+MLXSW_ITEM64(reg, ricnt, good_broadcast_packets, 0x18, 0, 64);
+
+/* reg_ricnt_good_unicast_bytes
+ * A count of L3 data and padding octets not including L2 headers
+ * for good unicast frames.
+ * Access: RW
+ */
+MLXSW_ITEM64(reg, ricnt, good_unicast_bytes, 0x20, 0, 64);
+
+/* reg_ricnt_good_multicast_bytes
+ * A count of L3 data and padding octets not including L2 headers
+ * for good multicast frames.
+ * Access: RW
+ */
+MLXSW_ITEM64(reg, ricnt, good_multicast_bytes, 0x28, 0, 64);
+
+/* reg_ritr_good_broadcast_bytes
+ * A count of L3 data and padding octets not including L2 headers
+ * for good broadcast frames.
+ * Access: RW
+ */
+MLXSW_ITEM64(reg, ricnt, good_broadcast_bytes, 0x30, 0, 64);
+
+/* reg_ricnt_error_packets
+ * A count of errored frames that do not pass the router checks.
+ * Access: RW
+ */
+MLXSW_ITEM64(reg, ricnt, error_packets, 0x38, 0, 64);
+
+/* reg_ricnt_discrad_packets
+ * A count of non-errored frames that do not pass the router checks.
+ * Access: RW
+ */
+MLXSW_ITEM64(reg, ricnt, discard_packets, 0x40, 0, 64);
+
+/* reg_ricnt_error_bytes
+ * A count of L3 data and padding octets not including L2 headers
+ * for errored frames.
+ * Access: RW
+ */
+MLXSW_ITEM64(reg, ricnt, error_bytes, 0x48, 0, 64);
+
+/* reg_ricnt_discard_bytes
+ * A count of L3 data and padding octets not including L2 headers
+ * for non-errored frames that do not pass the router checks.
+ * Access: RW
+ */
+MLXSW_ITEM64(reg, ricnt, discard_bytes, 0x50, 0, 64);
+
+static inline void mlxsw_reg_ricnt_pack(char *payload, u32 index,
+ enum mlxsw_reg_ricnt_opcode op)
+{
+ MLXSW_REG_ZERO(ricnt, payload);
+ mlxsw_reg_ricnt_op_set(payload, op);
+ mlxsw_reg_ricnt_counter_index_set(payload, index);
+ mlxsw_reg_ricnt_counter_set_type_set(payload,
+ MLXSW_REG_RICNT_COUNTER_SET_TYPE_BASIC);
+}
+
/* RALTA - Router Algorithmic LPM Tree Allocation Register
* -------------------------------------------------------
* RALTA is used to allocate the LPM trees of the SHSPM method.
@@ -5504,6 +5683,70 @@
mlxsw_reg_mpsc_rate_set(payload, rate);
}
+/* MGPC - Monitoring General Purpose Counter Set Register
+ * The MGPC register retrieves and sets the General Purpose Counter Set.
+ */
+#define MLXSW_REG_MGPC_ID 0x9081
+#define MLXSW_REG_MGPC_LEN 0x18
+
+MLXSW_REG_DEFINE(mgpc, MLXSW_REG_MGPC_ID, MLXSW_REG_MGPC_LEN);
+
+enum mlxsw_reg_mgpc_counter_set_type {
+ /* No count */
+ MLXSW_REG_MGPC_COUNTER_SET_TYPE_NO_COUT = 0x00,
+ /* Count packets and bytes */
+ MLXSW_REG_MGPC_COUNTER_SET_TYPE_PACKETS_BYTES = 0x03,
+ /* Count only packets */
+ MLXSW_REG_MGPC_COUNTER_SET_TYPE_PACKETS = 0x05,
+};
+
+/* reg_mgpc_counter_set_type
+ * Counter set type.
+ * Access: OP
+ */
+MLXSW_ITEM32(reg, mgpc, counter_set_type, 0x00, 24, 8);
+
+/* reg_mgpc_counter_index
+ * Counter index.
+ * Access: Index
+ */
+MLXSW_ITEM32(reg, mgpc, counter_index, 0x00, 0, 24);
+
+enum mlxsw_reg_mgpc_opcode {
+ /* Nop */
+ MLXSW_REG_MGPC_OPCODE_NOP = 0x00,
+ /* Clear counters */
+ MLXSW_REG_MGPC_OPCODE_CLEAR = 0x08,
+};
+
+/* reg_mgpc_opcode
+ * Opcode.
+ * Access: OP
+ */
+MLXSW_ITEM32(reg, mgpc, opcode, 0x04, 28, 4);
+
+/* reg_mgpc_byte_counter
+ * Byte counter value.
+ * Access: RW
+ */
+MLXSW_ITEM64(reg, mgpc, byte_counter, 0x08, 0, 64);
+
+/* reg_mgpc_packet_counter
+ * Packet counter value.
+ * Access: RW
+ */
+MLXSW_ITEM64(reg, mgpc, packet_counter, 0x10, 0, 64);
+
+static inline void mlxsw_reg_mgpc_pack(char *payload, u32 counter_index,
+ enum mlxsw_reg_mgpc_opcode opcode,
+ enum mlxsw_reg_mgpc_counter_set_type set_type)
+{
+ MLXSW_REG_ZERO(mgpc, payload);
+ mlxsw_reg_mgpc_counter_index_set(payload, counter_index);
+ mlxsw_reg_mgpc_counter_set_type_set(payload, set_type);
+ mlxsw_reg_mgpc_opcode_set(payload, opcode);
+}
+
/* SBPR - Shared Buffer Pools Register
* -----------------------------------
* The SBPR configures and retrieves the shared buffer pools and configuration.
@@ -5960,6 +6203,7 @@
MLXSW_REG(rgcr),
MLXSW_REG(ritr),
MLXSW_REG(ratr),
+ MLXSW_REG(ricnt),
MLXSW_REG(ralta),
MLXSW_REG(ralst),
MLXSW_REG(raltb),
@@ -5977,6 +6221,7 @@
MLXSW_REG(mpar),
MLXSW_REG(mlcr),
MLXSW_REG(mpsc),
+ MLXSW_REG(mgpc),
MLXSW_REG(sbpr),
MLXSW_REG(sbcm),
MLXSW_REG(sbpm),
diff --git a/drivers/net/ethernet/mellanox/mlxsw/resources.h b/drivers/net/ethernet/mellanox/mlxsw/resources.h
index bce8c2e..9556d93 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/resources.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/resources.h
@@ -43,11 +43,15 @@
MLXSW_RES_ID_KVD_SINGLE_MIN_SIZE,
MLXSW_RES_ID_KVD_DOUBLE_MIN_SIZE,
MLXSW_RES_ID_MAX_TRAP_GROUPS,
+ MLXSW_RES_ID_COUNTER_POOL_SIZE,
MLXSW_RES_ID_MAX_SPAN,
+ MLXSW_RES_ID_COUNTER_SIZE_PACKETS_BYTES,
+ MLXSW_RES_ID_COUNTER_SIZE_ROUTER_BASIC,
MLXSW_RES_ID_MAX_SYSTEM_PORT,
MLXSW_RES_ID_MAX_LAG,
MLXSW_RES_ID_MAX_LAG_MEMBERS,
MLXSW_RES_ID_MAX_BUFFER_SIZE,
+ MLXSW_RES_ID_CELL_SIZE,
MLXSW_RES_ID_ACL_MAX_TCAM_REGIONS,
MLXSW_RES_ID_ACL_MAX_TCAM_RULES,
MLXSW_RES_ID_ACL_MAX_REGIONS,
@@ -59,6 +63,7 @@
MLXSW_RES_ID_MAX_CPU_POLICERS,
MLXSW_RES_ID_MAX_VRS,
MLXSW_RES_ID_MAX_RIFS,
+ MLXSW_RES_ID_MAX_LPM_TREES,
/* Internal resources.
* Determined by the SW, not queried from the HW.
@@ -75,11 +80,15 @@
[MLXSW_RES_ID_KVD_SINGLE_MIN_SIZE] = 0x1002,
[MLXSW_RES_ID_KVD_DOUBLE_MIN_SIZE] = 0x1003,
[MLXSW_RES_ID_MAX_TRAP_GROUPS] = 0x2201,
+ [MLXSW_RES_ID_COUNTER_POOL_SIZE] = 0x2410,
[MLXSW_RES_ID_MAX_SPAN] = 0x2420,
+ [MLXSW_RES_ID_COUNTER_SIZE_PACKETS_BYTES] = 0x2443,
+ [MLXSW_RES_ID_COUNTER_SIZE_ROUTER_BASIC] = 0x2449,
[MLXSW_RES_ID_MAX_SYSTEM_PORT] = 0x2502,
[MLXSW_RES_ID_MAX_LAG] = 0x2520,
[MLXSW_RES_ID_MAX_LAG_MEMBERS] = 0x2521,
[MLXSW_RES_ID_MAX_BUFFER_SIZE] = 0x2802, /* Bytes */
+ [MLXSW_RES_ID_CELL_SIZE] = 0x2803, /* Bytes */
[MLXSW_RES_ID_ACL_MAX_TCAM_REGIONS] = 0x2901,
[MLXSW_RES_ID_ACL_MAX_TCAM_RULES] = 0x2902,
[MLXSW_RES_ID_ACL_MAX_REGIONS] = 0x2903,
@@ -91,6 +100,7 @@
[MLXSW_RES_ID_MAX_CPU_POLICERS] = 0x2A13,
[MLXSW_RES_ID_MAX_VRS] = 0x2C01,
[MLXSW_RES_ID_MAX_RIFS] = 0x2C02,
+ [MLXSW_RES_ID_MAX_LPM_TREES] = 0x2C30,
};
struct mlxsw_res {
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
index 16484f2..88357ce 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
@@ -66,6 +66,8 @@
#include "port.h"
#include "trap.h"
#include "txheader.h"
+#include "spectrum_cnt.h"
+#include "spectrum_dpipe.h"
static const char mlxsw_sp_driver_name[] = "mlxsw_spectrum";
static const char mlxsw_sp_driver_version[] = "1.0";
@@ -138,6 +140,60 @@
*/
MLXSW_ITEM32(tx, hdr, type, 0x0C, 0, 4);
+int mlxsw_sp_flow_counter_get(struct mlxsw_sp *mlxsw_sp,
+ unsigned int counter_index, u64 *packets,
+ u64 *bytes)
+{
+ char mgpc_pl[MLXSW_REG_MGPC_LEN];
+ int err;
+
+ mlxsw_reg_mgpc_pack(mgpc_pl, counter_index, MLXSW_REG_MGPC_OPCODE_NOP,
+ MLXSW_REG_MGPC_COUNTER_SET_TYPE_PACKETS_BYTES);
+ err = mlxsw_reg_query(mlxsw_sp->core, MLXSW_REG(mgpc), mgpc_pl);
+ if (err)
+ return err;
+ *packets = mlxsw_reg_mgpc_packet_counter_get(mgpc_pl);
+ *bytes = mlxsw_reg_mgpc_byte_counter_get(mgpc_pl);
+ return 0;
+}
+
+static int mlxsw_sp_flow_counter_clear(struct mlxsw_sp *mlxsw_sp,
+ unsigned int counter_index)
+{
+ char mgpc_pl[MLXSW_REG_MGPC_LEN];
+
+ mlxsw_reg_mgpc_pack(mgpc_pl, counter_index, MLXSW_REG_MGPC_OPCODE_CLEAR,
+ MLXSW_REG_MGPC_COUNTER_SET_TYPE_PACKETS_BYTES);
+ return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(mgpc), mgpc_pl);
+}
+
+int mlxsw_sp_flow_counter_alloc(struct mlxsw_sp *mlxsw_sp,
+ unsigned int *p_counter_index)
+{
+ int err;
+
+ err = mlxsw_sp_counter_alloc(mlxsw_sp, MLXSW_SP_COUNTER_SUB_POOL_FLOW,
+ p_counter_index);
+ if (err)
+ return err;
+ err = mlxsw_sp_flow_counter_clear(mlxsw_sp, *p_counter_index);
+ if (err)
+ goto err_counter_clear;
+ return 0;
+
+err_counter_clear:
+ mlxsw_sp_counter_free(mlxsw_sp, MLXSW_SP_COUNTER_SUB_POOL_FLOW,
+ *p_counter_index);
+ return err;
+}
+
+void mlxsw_sp_flow_counter_free(struct mlxsw_sp *mlxsw_sp,
+ unsigned int counter_index)
+{
+ mlxsw_sp_counter_free(mlxsw_sp, MLXSW_SP_COUNTER_SUB_POOL_FLOW,
+ counter_index);
+}
+
static void mlxsw_sp_txhdr_construct(struct sk_buff *skb,
const struct mlxsw_tx_info *tx_info)
{
@@ -304,9 +360,10 @@
return false;
}
-static int mlxsw_sp_span_mtu_to_buffsize(int mtu)
+static int mlxsw_sp_span_mtu_to_buffsize(const struct mlxsw_sp *mlxsw_sp,
+ int mtu)
{
- return MLXSW_SP_BYTES_TO_CELLS(mtu * 5 / 2) + 1;
+ return mlxsw_sp_bytes_cells(mlxsw_sp, mtu * 5 / 2) + 1;
}
static int mlxsw_sp_span_port_mtu_update(struct mlxsw_sp_port *port, u16 mtu)
@@ -319,8 +376,9 @@
* updated according to the mtu value
*/
if (mlxsw_sp_span_is_egress_mirror(port)) {
- mlxsw_reg_sbib_pack(sbib_pl, port->local_port,
- mlxsw_sp_span_mtu_to_buffsize(mtu));
+ u32 buffsize = mlxsw_sp_span_mtu_to_buffsize(mlxsw_sp, mtu);
+
+ mlxsw_reg_sbib_pack(sbib_pl, port->local_port, buffsize);
err = mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(sbib), sbib_pl);
if (err) {
netdev_err(port->dev, "Could not update shared buffer for mirroring\n");
@@ -357,8 +415,10 @@
/* if it is an egress SPAN, bind a shared buffer to it */
if (type == MLXSW_SP_SPAN_EGRESS) {
- mlxsw_reg_sbib_pack(sbib_pl, port->local_port,
- mlxsw_sp_span_mtu_to_buffsize(port->dev->mtu));
+ u32 buffsize = mlxsw_sp_span_mtu_to_buffsize(mlxsw_sp,
+ port->dev->mtu);
+
+ mlxsw_reg_sbib_pack(sbib_pl, port->local_port, buffsize);
err = mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(sbib), sbib_pl);
if (err) {
netdev_err(port->dev, "Could not create shared buffer for mirroring\n");
@@ -745,19 +805,47 @@
return 0;
}
-static void mlxsw_sp_pg_buf_pack(char *pbmc_pl, int pg_index, int mtu,
- bool pause_en, bool pfc_en, u16 delay)
+static u16 mlxsw_sp_pg_buf_threshold_get(const struct mlxsw_sp *mlxsw_sp,
+ int mtu)
{
- u16 pg_size = 2 * MLXSW_SP_BYTES_TO_CELLS(mtu);
+ return 2 * mlxsw_sp_bytes_cells(mlxsw_sp, mtu);
+}
- delay = pfc_en ? mlxsw_sp_pfc_delay_get(mtu, delay) :
- MLXSW_SP_PAUSE_DELAY;
+#define MLXSW_SP_CELL_FACTOR 2 /* 2 * cell_size / (IPG + cell_size + 1) */
- if (pause_en || pfc_en)
- mlxsw_reg_pbmc_lossless_buffer_pack(pbmc_pl, pg_index,
- pg_size + delay, pg_size);
+static u16 mlxsw_sp_pfc_delay_get(const struct mlxsw_sp *mlxsw_sp, int mtu,
+ u16 delay)
+{
+ delay = mlxsw_sp_bytes_cells(mlxsw_sp, DIV_ROUND_UP(delay,
+ BITS_PER_BYTE));
+ return MLXSW_SP_CELL_FACTOR * delay + mlxsw_sp_bytes_cells(mlxsw_sp,
+ mtu);
+}
+
+/* Maximum delay buffer needed in case of PAUSE frames, in bytes.
+ * Assumes 100m cable and maximum MTU.
+ */
+#define MLXSW_SP_PAUSE_DELAY 58752
+
+static u16 mlxsw_sp_pg_buf_delay_get(const struct mlxsw_sp *mlxsw_sp, int mtu,
+ u16 delay, bool pfc, bool pause)
+{
+ if (pfc)
+ return mlxsw_sp_pfc_delay_get(mlxsw_sp, mtu, delay);
+ else if (pause)
+ return mlxsw_sp_bytes_cells(mlxsw_sp, MLXSW_SP_PAUSE_DELAY);
else
- mlxsw_reg_pbmc_lossy_buffer_pack(pbmc_pl, pg_index, pg_size);
+ return 0;
+}
+
+static void mlxsw_sp_pg_buf_pack(char *pbmc_pl, int index, u16 size, u16 thres,
+ bool lossy)
+{
+ if (lossy)
+ mlxsw_reg_pbmc_lossy_buffer_pack(pbmc_pl, index, size);
+ else
+ mlxsw_reg_pbmc_lossless_buffer_pack(pbmc_pl, index, size,
+ thres);
}
int __mlxsw_sp_port_headroom_set(struct mlxsw_sp_port *mlxsw_sp_port, int mtu,
@@ -778,6 +866,8 @@
for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) {
bool configure = false;
bool pfc = false;
+ bool lossy;
+ u16 thres;
for (j = 0; j < IEEE_8021QAZ_MAX_TCS; j++) {
if (prio_tc[j] == i) {
@@ -789,7 +879,12 @@
if (!configure)
continue;
- mlxsw_sp_pg_buf_pack(pbmc_pl, i, mtu, pause_en, pfc, delay);
+
+ lossy = !(pfc || pause_en);
+ thres = mlxsw_sp_pg_buf_threshold_get(mlxsw_sp, mtu);
+ delay = mlxsw_sp_pg_buf_delay_get(mlxsw_sp, mtu, delay, pfc,
+ pause_en);
+ mlxsw_sp_pg_buf_pack(pbmc_pl, i, thres + delay, thres, lossy);
}
return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(pbmc), pbmc_pl);
@@ -966,8 +1061,9 @@
memcpy(stats, mlxsw_sp_port->hw_stats.cache, sizeof(*stats));
}
-int mlxsw_sp_port_vlan_set(struct mlxsw_sp_port *mlxsw_sp_port, u16 vid_begin,
- u16 vid_end, bool is_member, bool untagged)
+static int __mlxsw_sp_port_vlan_set(struct mlxsw_sp_port *mlxsw_sp_port,
+ u16 vid_begin, u16 vid_end,
+ bool is_member, bool untagged)
{
struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp;
char *spvm_pl;
@@ -984,6 +1080,26 @@
return err;
}
+int mlxsw_sp_port_vlan_set(struct mlxsw_sp_port *mlxsw_sp_port, u16 vid_begin,
+ u16 vid_end, bool is_member, bool untagged)
+{
+ u16 vid, vid_e;
+ int err;
+
+ for (vid = vid_begin; vid <= vid_end;
+ vid += MLXSW_REG_SPVM_REC_MAX_COUNT) {
+ vid_e = min((u16) (vid + MLXSW_REG_SPVM_REC_MAX_COUNT - 1),
+ vid_end);
+
+ err = __mlxsw_sp_port_vlan_set(mlxsw_sp_port, vid, vid_e,
+ is_member, untagged);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
static int mlxsw_sp_port_vp_mode_trans(struct mlxsw_sp_port *mlxsw_sp_port)
{
enum mlxsw_reg_svfa_mt mt = MLXSW_REG_SVFA_MT_PORT_VID_TO_FID;
@@ -1368,7 +1484,7 @@
tc->cls_mall);
return 0;
default:
- return -EINVAL;
+ return -EOPNOTSUPP;
}
case TC_SETUP_CLSFLOWER:
switch (tc->cls_flower->command) {
@@ -1379,6 +1495,9 @@
mlxsw_sp_flower_destroy(mlxsw_sp_port, ingress,
tc->cls_flower);
return 0;
+ case TC_CLSFLOWER_STATS:
+ return mlxsw_sp_flower_stats(mlxsw_sp_port, ingress,
+ tc->cls_flower);
default:
return -EOPNOTSUPP;
}
@@ -1492,6 +1611,7 @@
struct mlxsw_sp_port_hw_stats {
char str[ETH_GSTRING_LEN];
u64 (*getter)(const char *payload);
+ bool cells_bytes;
};
static struct mlxsw_sp_port_hw_stats mlxsw_sp_port_hw_stats[] = {
@@ -1612,17 +1732,11 @@
#define MLXSW_SP_PORT_HW_PRIO_STATS_LEN ARRAY_SIZE(mlxsw_sp_port_hw_prio_stats)
-static u64 mlxsw_reg_ppcnt_tc_transmit_queue_bytes_get(const char *ppcnt_pl)
-{
- u64 transmit_queue = mlxsw_reg_ppcnt_tc_transmit_queue_get(ppcnt_pl);
-
- return MLXSW_SP_CELLS_TO_BYTES(transmit_queue);
-}
-
static struct mlxsw_sp_port_hw_stats mlxsw_sp_port_hw_tc_stats[] = {
{
.str = "tc_transmit_queue_tc",
- .getter = mlxsw_reg_ppcnt_tc_transmit_queue_bytes_get,
+ .getter = mlxsw_reg_ppcnt_tc_transmit_queue_get,
+ .cells_bytes = true,
},
{
.str = "tc_no_buffer_discard_uc_tc",
@@ -1734,6 +1848,8 @@
enum mlxsw_reg_ppcnt_grp grp, int prio,
u64 *data, int data_index)
{
+ struct mlxsw_sp_port *mlxsw_sp_port = netdev_priv(dev);
+ struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp;
struct mlxsw_sp_port_hw_stats *hw_stats;
char ppcnt_pl[MLXSW_REG_PPCNT_LEN];
int i, len;
@@ -1743,8 +1859,13 @@
if (err)
return;
mlxsw_sp_port_get_stats_raw(dev, grp, prio, ppcnt_pl);
- for (i = 0; i < len; i++)
+ for (i = 0; i < len; i++) {
data[data_index + i] = hw_stats[i].getter(ppcnt_pl);
+ if (!hw_stats[i].cells_bytes)
+ continue;
+ data[data_index + i] = mlxsw_sp_cells_bytes(mlxsw_sp,
+ data[data_index + i]);
+ }
}
static void mlxsw_sp_port_get_stats(struct net_device *dev,
@@ -2537,25 +2658,33 @@
{
int i;
- for (i = 1; i < MLXSW_PORT_MAX_PORTS; i++)
+ for (i = 1; i < mlxsw_core_max_ports(mlxsw_sp->core); i++)
if (mlxsw_sp_port_created(mlxsw_sp, i))
mlxsw_sp_port_remove(mlxsw_sp, i);
+ kfree(mlxsw_sp->port_to_module);
kfree(mlxsw_sp->ports);
}
static int mlxsw_sp_ports_create(struct mlxsw_sp *mlxsw_sp)
{
+ unsigned int max_ports = mlxsw_core_max_ports(mlxsw_sp->core);
u8 module, width, lane;
size_t alloc_size;
int i;
int err;
- alloc_size = sizeof(struct mlxsw_sp_port *) * MLXSW_PORT_MAX_PORTS;
+ alloc_size = sizeof(struct mlxsw_sp_port *) * max_ports;
mlxsw_sp->ports = kzalloc(alloc_size, GFP_KERNEL);
if (!mlxsw_sp->ports)
return -ENOMEM;
- for (i = 1; i < MLXSW_PORT_MAX_PORTS; i++) {
+ mlxsw_sp->port_to_module = kcalloc(max_ports, sizeof(u8), GFP_KERNEL);
+ if (!mlxsw_sp->port_to_module) {
+ err = -ENOMEM;
+ goto err_port_to_module_alloc;
+ }
+
+ for (i = 1; i < max_ports; i++) {
err = mlxsw_sp_port_module_info_get(mlxsw_sp, i, &module,
&width, &lane);
if (err)
@@ -2575,6 +2704,8 @@
for (i--; i >= 1; i--)
if (mlxsw_sp_port_created(mlxsw_sp, i))
mlxsw_sp_port_remove(mlxsw_sp, i);
+ kfree(mlxsw_sp->port_to_module);
+err_port_to_module_alloc:
kfree(mlxsw_sp->ports);
return err;
}
@@ -2877,6 +3008,7 @@
MLXSW_SP_RXL_NO_MARK(IGMP_V3_REPORT, TRAP_TO_CPU, IGMP, false),
MLXSW_SP_RXL_MARK(ARPBC, MIRROR_TO_CPU, ARP, false),
MLXSW_SP_RXL_MARK(ARPUC, MIRROR_TO_CPU, ARP, false),
+ MLXSW_SP_RXL_NO_MARK(FID_MISS, TRAP_TO_CPU, IP2ME, false),
/* L3 traps */
MLXSW_SP_RXL_NO_MARK(MTUERROR, TRAP_TO_CPU, ROUTER_EXP, false),
MLXSW_SP_RXL_NO_MARK(TTLERROR, TRAP_TO_CPU, ROUTER_EXP, false),
@@ -3158,6 +3290,18 @@
return mlxsw_reg_write(mlxsw_core, MLXSW_REG(htgt), htgt_pl);
}
+static int mlxsw_sp_vfid_op(struct mlxsw_sp *mlxsw_sp, u16 fid, bool create);
+
+static int mlxsw_sp_dummy_fid_init(struct mlxsw_sp *mlxsw_sp)
+{
+ return mlxsw_sp_vfid_op(mlxsw_sp, MLXSW_SP_DUMMY_FID, true);
+}
+
+static void mlxsw_sp_dummy_fid_fini(struct mlxsw_sp *mlxsw_sp)
+{
+ mlxsw_sp_vfid_op(mlxsw_sp, MLXSW_SP_DUMMY_FID, false);
+}
+
static int mlxsw_sp_init(struct mlxsw_core *mlxsw_core,
const struct mlxsw_bus_info *mlxsw_bus_info)
{
@@ -3224,6 +3368,24 @@
goto err_acl_init;
}
+ err = mlxsw_sp_counter_pool_init(mlxsw_sp);
+ if (err) {
+ dev_err(mlxsw_sp->bus_info->dev, "Failed to init counter pool\n");
+ goto err_counter_pool_init;
+ }
+
+ err = mlxsw_sp_dpipe_init(mlxsw_sp);
+ if (err) {
+ dev_err(mlxsw_sp->bus_info->dev, "Failed to init pipeline debug\n");
+ goto err_dpipe_init;
+ }
+
+ err = mlxsw_sp_dummy_fid_init(mlxsw_sp);
+ if (err) {
+ dev_err(mlxsw_sp->bus_info->dev, "Failed to init dummy FID\n");
+ goto err_dummy_fid_init;
+ }
+
err = mlxsw_sp_ports_create(mlxsw_sp);
if (err) {
dev_err(mlxsw_sp->bus_info->dev, "Failed to create ports\n");
@@ -3233,6 +3395,12 @@
return 0;
err_ports_create:
+ mlxsw_sp_dummy_fid_fini(mlxsw_sp);
+err_dummy_fid_init:
+ mlxsw_sp_dpipe_fini(mlxsw_sp);
+err_dpipe_init:
+ mlxsw_sp_counter_pool_fini(mlxsw_sp);
+err_counter_pool_init:
mlxsw_sp_acl_fini(mlxsw_sp);
err_acl_init:
mlxsw_sp_span_fini(mlxsw_sp);
@@ -3255,6 +3423,9 @@
struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core);
mlxsw_sp_ports_remove(mlxsw_sp);
+ mlxsw_sp_dummy_fid_fini(mlxsw_sp);
+ mlxsw_sp_dpipe_fini(mlxsw_sp);
+ mlxsw_sp_counter_pool_fini(mlxsw_sp);
mlxsw_sp_acl_fini(mlxsw_sp);
mlxsw_sp_span_fini(mlxsw_sp);
mlxsw_sp_router_fini(mlxsw_sp);
@@ -3326,13 +3497,13 @@
return dev->netdev_ops == &mlxsw_sp_port_netdev_ops;
}
-static int mlxsw_lower_dev_walk(struct net_device *lower_dev, void *data)
+static int mlxsw_sp_lower_dev_walk(struct net_device *lower_dev, void *data)
{
- struct mlxsw_sp_port **port = data;
+ struct mlxsw_sp_port **p_mlxsw_sp_port = data;
int ret = 0;
if (mlxsw_sp_port_dev_check(lower_dev)) {
- *port = netdev_priv(lower_dev);
+ *p_mlxsw_sp_port = netdev_priv(lower_dev);
ret = 1;
}
@@ -3341,18 +3512,18 @@
static struct mlxsw_sp_port *mlxsw_sp_port_dev_lower_find(struct net_device *dev)
{
- struct mlxsw_sp_port *port;
+ struct mlxsw_sp_port *mlxsw_sp_port;
if (mlxsw_sp_port_dev_check(dev))
return netdev_priv(dev);
- port = NULL;
- netdev_walk_all_lower_dev(dev, mlxsw_lower_dev_walk, &port);
+ mlxsw_sp_port = NULL;
+ netdev_walk_all_lower_dev(dev, mlxsw_sp_lower_dev_walk, &mlxsw_sp_port);
- return port;
+ return mlxsw_sp_port;
}
-static struct mlxsw_sp *mlxsw_sp_lower_get(struct net_device *dev)
+struct mlxsw_sp *mlxsw_sp_lower_get(struct net_device *dev)
{
struct mlxsw_sp_port *mlxsw_sp_port;
@@ -3362,15 +3533,16 @@
static struct mlxsw_sp_port *mlxsw_sp_port_dev_lower_find_rcu(struct net_device *dev)
{
- struct mlxsw_sp_port *port;
+ struct mlxsw_sp_port *mlxsw_sp_port;
if (mlxsw_sp_port_dev_check(dev))
return netdev_priv(dev);
- port = NULL;
- netdev_walk_all_lower_dev_rcu(dev, mlxsw_lower_dev_walk, &port);
+ mlxsw_sp_port = NULL;
+ netdev_walk_all_lower_dev_rcu(dev, mlxsw_sp_lower_dev_walk,
+ &mlxsw_sp_port);
- return port;
+ return mlxsw_sp_port;
}
struct mlxsw_sp_port *mlxsw_sp_port_lower_dev_hold(struct net_device *dev)
@@ -3390,546 +3562,6 @@
dev_put(mlxsw_sp_port->dev);
}
-static bool mlxsw_sp_rif_should_config(struct mlxsw_sp_rif *r,
- unsigned long event)
-{
- switch (event) {
- case NETDEV_UP:
- if (!r)
- return true;
- r->ref_count++;
- return false;
- case NETDEV_DOWN:
- if (r && --r->ref_count == 0)
- return true;
- /* It is possible we already removed the RIF ourselves
- * if it was assigned to a netdev that is now a bridge
- * or LAG slave.
- */
- return false;
- }
-
- return false;
-}
-
-static int mlxsw_sp_avail_rif_get(struct mlxsw_sp *mlxsw_sp)
-{
- int i;
-
- for (i = 0; i < MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_RIFS); i++)
- if (!mlxsw_sp->rifs[i])
- return i;
-
- return MLXSW_SP_INVALID_RIF;
-}
-
-static void mlxsw_sp_vport_rif_sp_attr_get(struct mlxsw_sp_port *mlxsw_sp_vport,
- bool *p_lagged, u16 *p_system_port)
-{
- u8 local_port = mlxsw_sp_vport->local_port;
-
- *p_lagged = mlxsw_sp_vport->lagged;
- *p_system_port = *p_lagged ? mlxsw_sp_vport->lag_id : local_port;
-}
-
-static int mlxsw_sp_vport_rif_sp_op(struct mlxsw_sp_port *mlxsw_sp_vport,
- struct net_device *l3_dev, u16 rif,
- bool create)
-{
- struct mlxsw_sp *mlxsw_sp = mlxsw_sp_vport->mlxsw_sp;
- bool lagged = mlxsw_sp_vport->lagged;
- char ritr_pl[MLXSW_REG_RITR_LEN];
- u16 system_port;
-
- mlxsw_reg_ritr_pack(ritr_pl, create, MLXSW_REG_RITR_SP_IF, rif,
- l3_dev->mtu, l3_dev->dev_addr);
-
- mlxsw_sp_vport_rif_sp_attr_get(mlxsw_sp_vport, &lagged, &system_port);
- mlxsw_reg_ritr_sp_if_pack(ritr_pl, lagged, system_port,
- mlxsw_sp_vport_vid_get(mlxsw_sp_vport));
-
- return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ritr), ritr_pl);
-}
-
-static void mlxsw_sp_vport_rif_sp_leave(struct mlxsw_sp_port *mlxsw_sp_vport);
-
-static struct mlxsw_sp_fid *
-mlxsw_sp_rfid_alloc(u16 fid, struct net_device *l3_dev)
-{
- struct mlxsw_sp_fid *f;
-
- f = kzalloc(sizeof(*f), GFP_KERNEL);
- if (!f)
- return NULL;
-
- f->leave = mlxsw_sp_vport_rif_sp_leave;
- f->ref_count = 0;
- f->dev = l3_dev;
- f->fid = fid;
-
- return f;
-}
-
-static struct mlxsw_sp_rif *
-mlxsw_sp_rif_alloc(u16 rif, struct net_device *l3_dev, struct mlxsw_sp_fid *f)
-{
- struct mlxsw_sp_rif *r;
-
- r = kzalloc(sizeof(*r), GFP_KERNEL);
- if (!r)
- return NULL;
-
- INIT_LIST_HEAD(&r->nexthop_list);
- INIT_LIST_HEAD(&r->neigh_list);
- ether_addr_copy(r->addr, l3_dev->dev_addr);
- r->mtu = l3_dev->mtu;
- r->ref_count = 1;
- r->dev = l3_dev;
- r->rif = rif;
- r->f = f;
-
- return r;
-}
-
-static struct mlxsw_sp_rif *
-mlxsw_sp_vport_rif_sp_create(struct mlxsw_sp_port *mlxsw_sp_vport,
- struct net_device *l3_dev)
-{
- struct mlxsw_sp *mlxsw_sp = mlxsw_sp_vport->mlxsw_sp;
- struct mlxsw_sp_fid *f;
- struct mlxsw_sp_rif *r;
- u16 fid, rif;
- int err;
-
- rif = mlxsw_sp_avail_rif_get(mlxsw_sp);
- if (rif == MLXSW_SP_INVALID_RIF)
- return ERR_PTR(-ERANGE);
-
- err = mlxsw_sp_vport_rif_sp_op(mlxsw_sp_vport, l3_dev, rif, true);
- if (err)
- return ERR_PTR(err);
-
- fid = mlxsw_sp_rif_sp_to_fid(rif);
- err = mlxsw_sp_rif_fdb_op(mlxsw_sp, l3_dev->dev_addr, fid, true);
- if (err)
- goto err_rif_fdb_op;
-
- f = mlxsw_sp_rfid_alloc(fid, l3_dev);
- if (!f) {
- err = -ENOMEM;
- goto err_rfid_alloc;
- }
-
- r = mlxsw_sp_rif_alloc(rif, l3_dev, f);
- if (!r) {
- err = -ENOMEM;
- goto err_rif_alloc;
- }
-
- f->r = r;
- mlxsw_sp->rifs[rif] = r;
-
- return r;
-
-err_rif_alloc:
- kfree(f);
-err_rfid_alloc:
- mlxsw_sp_rif_fdb_op(mlxsw_sp, l3_dev->dev_addr, fid, false);
-err_rif_fdb_op:
- mlxsw_sp_vport_rif_sp_op(mlxsw_sp_vport, l3_dev, rif, false);
- return ERR_PTR(err);
-}
-
-static void mlxsw_sp_vport_rif_sp_destroy(struct mlxsw_sp_port *mlxsw_sp_vport,
- struct mlxsw_sp_rif *r)
-{
- struct mlxsw_sp *mlxsw_sp = mlxsw_sp_vport->mlxsw_sp;
- struct net_device *l3_dev = r->dev;
- struct mlxsw_sp_fid *f = r->f;
- u16 fid = f->fid;
- u16 rif = r->rif;
-
- mlxsw_sp_router_rif_gone_sync(mlxsw_sp, r);
-
- mlxsw_sp->rifs[rif] = NULL;
- f->r = NULL;
-
- kfree(r);
-
- kfree(f);
-
- mlxsw_sp_rif_fdb_op(mlxsw_sp, l3_dev->dev_addr, fid, false);
-
- mlxsw_sp_vport_rif_sp_op(mlxsw_sp_vport, l3_dev, rif, false);
-}
-
-static int mlxsw_sp_vport_rif_sp_join(struct mlxsw_sp_port *mlxsw_sp_vport,
- struct net_device *l3_dev)
-{
- struct mlxsw_sp *mlxsw_sp = mlxsw_sp_vport->mlxsw_sp;
- struct mlxsw_sp_rif *r;
-
- r = mlxsw_sp_rif_find_by_dev(mlxsw_sp, l3_dev);
- if (!r) {
- r = mlxsw_sp_vport_rif_sp_create(mlxsw_sp_vport, l3_dev);
- if (IS_ERR(r))
- return PTR_ERR(r);
- }
-
- mlxsw_sp_vport_fid_set(mlxsw_sp_vport, r->f);
- r->f->ref_count++;
-
- netdev_dbg(mlxsw_sp_vport->dev, "Joined FID=%d\n", r->f->fid);
-
- return 0;
-}
-
-static void mlxsw_sp_vport_rif_sp_leave(struct mlxsw_sp_port *mlxsw_sp_vport)
-{
- struct mlxsw_sp_fid *f = mlxsw_sp_vport_fid_get(mlxsw_sp_vport);
-
- netdev_dbg(mlxsw_sp_vport->dev, "Left FID=%d\n", f->fid);
-
- mlxsw_sp_vport_fid_set(mlxsw_sp_vport, NULL);
- if (--f->ref_count == 0)
- mlxsw_sp_vport_rif_sp_destroy(mlxsw_sp_vport, f->r);
-}
-
-static int mlxsw_sp_inetaddr_vport_event(struct net_device *l3_dev,
- struct net_device *port_dev,
- unsigned long event, u16 vid)
-{
- struct mlxsw_sp_port *mlxsw_sp_port = netdev_priv(port_dev);
- struct mlxsw_sp_port *mlxsw_sp_vport;
-
- mlxsw_sp_vport = mlxsw_sp_port_vport_find(mlxsw_sp_port, vid);
- if (WARN_ON(!mlxsw_sp_vport))
- return -EINVAL;
-
- switch (event) {
- case NETDEV_UP:
- return mlxsw_sp_vport_rif_sp_join(mlxsw_sp_vport, l3_dev);
- case NETDEV_DOWN:
- mlxsw_sp_vport_rif_sp_leave(mlxsw_sp_vport);
- break;
- }
-
- return 0;
-}
-
-static int mlxsw_sp_inetaddr_port_event(struct net_device *port_dev,
- unsigned long event)
-{
- if (netif_is_bridge_port(port_dev) || netif_is_lag_port(port_dev))
- return 0;
-
- return mlxsw_sp_inetaddr_vport_event(port_dev, port_dev, event, 1);
-}
-
-static int __mlxsw_sp_inetaddr_lag_event(struct net_device *l3_dev,
- struct net_device *lag_dev,
- unsigned long event, u16 vid)
-{
- struct net_device *port_dev;
- struct list_head *iter;
- int err;
-
- netdev_for_each_lower_dev(lag_dev, port_dev, iter) {
- if (mlxsw_sp_port_dev_check(port_dev)) {
- err = mlxsw_sp_inetaddr_vport_event(l3_dev, port_dev,
- event, vid);
- if (err)
- return err;
- }
- }
-
- return 0;
-}
-
-static int mlxsw_sp_inetaddr_lag_event(struct net_device *lag_dev,
- unsigned long event)
-{
- if (netif_is_bridge_port(lag_dev))
- return 0;
-
- return __mlxsw_sp_inetaddr_lag_event(lag_dev, lag_dev, event, 1);
-}
-
-static struct mlxsw_sp_fid *mlxsw_sp_bridge_fid_get(struct mlxsw_sp *mlxsw_sp,
- struct net_device *l3_dev)
-{
- u16 fid;
-
- if (is_vlan_dev(l3_dev))
- fid = vlan_dev_vlan_id(l3_dev);
- else if (mlxsw_sp->master_bridge.dev == l3_dev)
- fid = 1;
- else
- return mlxsw_sp_vfid_find(mlxsw_sp, l3_dev);
-
- return mlxsw_sp_fid_find(mlxsw_sp, fid);
-}
-
-static enum mlxsw_flood_table_type mlxsw_sp_flood_table_type_get(u16 fid)
-{
- return mlxsw_sp_fid_is_vfid(fid) ? MLXSW_REG_SFGC_TABLE_TYPE_FID :
- MLXSW_REG_SFGC_TABLE_TYPE_FID_OFFEST;
-}
-
-static u16 mlxsw_sp_flood_table_index_get(u16 fid)
-{
- return mlxsw_sp_fid_is_vfid(fid) ? mlxsw_sp_fid_to_vfid(fid) : fid;
-}
-
-static int mlxsw_sp_router_port_flood_set(struct mlxsw_sp *mlxsw_sp, u16 fid,
- bool set)
-{
- enum mlxsw_flood_table_type table_type;
- char *sftr_pl;
- u16 index;
- int err;
-
- sftr_pl = kmalloc(MLXSW_REG_SFTR_LEN, GFP_KERNEL);
- if (!sftr_pl)
- return -ENOMEM;
-
- table_type = mlxsw_sp_flood_table_type_get(fid);
- index = mlxsw_sp_flood_table_index_get(fid);
- mlxsw_reg_sftr_pack(sftr_pl, MLXSW_SP_FLOOD_TABLE_BC, index, table_type,
- 1, MLXSW_PORT_ROUTER_PORT, set);
- err = mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(sftr), sftr_pl);
-
- kfree(sftr_pl);
- return err;
-}
-
-static enum mlxsw_reg_ritr_if_type mlxsw_sp_rif_type_get(u16 fid)
-{
- if (mlxsw_sp_fid_is_vfid(fid))
- return MLXSW_REG_RITR_FID_IF;
- else
- return MLXSW_REG_RITR_VLAN_IF;
-}
-
-static int mlxsw_sp_rif_bridge_op(struct mlxsw_sp *mlxsw_sp,
- struct net_device *l3_dev,
- u16 fid, u16 rif,
- bool create)
-{
- enum mlxsw_reg_ritr_if_type rif_type;
- char ritr_pl[MLXSW_REG_RITR_LEN];
-
- rif_type = mlxsw_sp_rif_type_get(fid);
- mlxsw_reg_ritr_pack(ritr_pl, create, rif_type, rif, l3_dev->mtu,
- l3_dev->dev_addr);
- mlxsw_reg_ritr_fid_set(ritr_pl, rif_type, fid);
-
- return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ritr), ritr_pl);
-}
-
-static int mlxsw_sp_rif_bridge_create(struct mlxsw_sp *mlxsw_sp,
- struct net_device *l3_dev,
- struct mlxsw_sp_fid *f)
-{
- struct mlxsw_sp_rif *r;
- u16 rif;
- int err;
-
- rif = mlxsw_sp_avail_rif_get(mlxsw_sp);
- if (rif == MLXSW_SP_INVALID_RIF)
- return -ERANGE;
-
- err = mlxsw_sp_router_port_flood_set(mlxsw_sp, f->fid, true);
- if (err)
- return err;
-
- err = mlxsw_sp_rif_bridge_op(mlxsw_sp, l3_dev, f->fid, rif, true);
- if (err)
- goto err_rif_bridge_op;
-
- err = mlxsw_sp_rif_fdb_op(mlxsw_sp, l3_dev->dev_addr, f->fid, true);
- if (err)
- goto err_rif_fdb_op;
-
- r = mlxsw_sp_rif_alloc(rif, l3_dev, f);
- if (!r) {
- err = -ENOMEM;
- goto err_rif_alloc;
- }
-
- f->r = r;
- mlxsw_sp->rifs[rif] = r;
-
- netdev_dbg(l3_dev, "RIF=%d created\n", rif);
-
- return 0;
-
-err_rif_alloc:
- mlxsw_sp_rif_fdb_op(mlxsw_sp, l3_dev->dev_addr, f->fid, false);
-err_rif_fdb_op:
- mlxsw_sp_rif_bridge_op(mlxsw_sp, l3_dev, f->fid, rif, false);
-err_rif_bridge_op:
- mlxsw_sp_router_port_flood_set(mlxsw_sp, f->fid, false);
- return err;
-}
-
-void mlxsw_sp_rif_bridge_destroy(struct mlxsw_sp *mlxsw_sp,
- struct mlxsw_sp_rif *r)
-{
- struct net_device *l3_dev = r->dev;
- struct mlxsw_sp_fid *f = r->f;
- u16 rif = r->rif;
-
- mlxsw_sp_router_rif_gone_sync(mlxsw_sp, r);
-
- mlxsw_sp->rifs[rif] = NULL;
- f->r = NULL;
-
- kfree(r);
-
- mlxsw_sp_rif_fdb_op(mlxsw_sp, l3_dev->dev_addr, f->fid, false);
-
- mlxsw_sp_rif_bridge_op(mlxsw_sp, l3_dev, f->fid, rif, false);
-
- mlxsw_sp_router_port_flood_set(mlxsw_sp, f->fid, false);
-
- netdev_dbg(l3_dev, "RIF=%d destroyed\n", rif);
-}
-
-static int mlxsw_sp_inetaddr_bridge_event(struct net_device *l3_dev,
- struct net_device *br_dev,
- unsigned long event)
-{
- struct mlxsw_sp *mlxsw_sp = mlxsw_sp_lower_get(l3_dev);
- struct mlxsw_sp_fid *f;
-
- /* FID can either be an actual FID if the L3 device is the
- * VLAN-aware bridge or a VLAN device on top. Otherwise, the
- * L3 device is a VLAN-unaware bridge and we get a vFID.
- */
- f = mlxsw_sp_bridge_fid_get(mlxsw_sp, l3_dev);
- if (WARN_ON(!f))
- return -EINVAL;
-
- switch (event) {
- case NETDEV_UP:
- return mlxsw_sp_rif_bridge_create(mlxsw_sp, l3_dev, f);
- case NETDEV_DOWN:
- mlxsw_sp_rif_bridge_destroy(mlxsw_sp, f->r);
- break;
- }
-
- return 0;
-}
-
-static int mlxsw_sp_inetaddr_vlan_event(struct net_device *vlan_dev,
- unsigned long event)
-{
- struct net_device *real_dev = vlan_dev_real_dev(vlan_dev);
- struct mlxsw_sp *mlxsw_sp = mlxsw_sp_lower_get(vlan_dev);
- u16 vid = vlan_dev_vlan_id(vlan_dev);
-
- if (mlxsw_sp_port_dev_check(real_dev))
- return mlxsw_sp_inetaddr_vport_event(vlan_dev, real_dev, event,
- vid);
- else if (netif_is_lag_master(real_dev))
- return __mlxsw_sp_inetaddr_lag_event(vlan_dev, real_dev, event,
- vid);
- else if (netif_is_bridge_master(real_dev) &&
- mlxsw_sp->master_bridge.dev == real_dev)
- return mlxsw_sp_inetaddr_bridge_event(vlan_dev, real_dev,
- event);
-
- return 0;
-}
-
-static int mlxsw_sp_inetaddr_event(struct notifier_block *unused,
- unsigned long event, void *ptr)
-{
- struct in_ifaddr *ifa = (struct in_ifaddr *) ptr;
- struct net_device *dev = ifa->ifa_dev->dev;
- struct mlxsw_sp *mlxsw_sp;
- struct mlxsw_sp_rif *r;
- int err = 0;
-
- mlxsw_sp = mlxsw_sp_lower_get(dev);
- if (!mlxsw_sp)
- goto out;
-
- r = mlxsw_sp_rif_find_by_dev(mlxsw_sp, dev);
- if (!mlxsw_sp_rif_should_config(r, event))
- goto out;
-
- if (mlxsw_sp_port_dev_check(dev))
- err = mlxsw_sp_inetaddr_port_event(dev, event);
- else if (netif_is_lag_master(dev))
- err = mlxsw_sp_inetaddr_lag_event(dev, event);
- else if (netif_is_bridge_master(dev))
- err = mlxsw_sp_inetaddr_bridge_event(dev, dev, event);
- else if (is_vlan_dev(dev))
- err = mlxsw_sp_inetaddr_vlan_event(dev, event);
-
-out:
- return notifier_from_errno(err);
-}
-
-static int mlxsw_sp_rif_edit(struct mlxsw_sp *mlxsw_sp, u16 rif,
- const char *mac, int mtu)
-{
- char ritr_pl[MLXSW_REG_RITR_LEN];
- int err;
-
- mlxsw_reg_ritr_rif_pack(ritr_pl, rif);
- err = mlxsw_reg_query(mlxsw_sp->core, MLXSW_REG(ritr), ritr_pl);
- if (err)
- return err;
-
- mlxsw_reg_ritr_mtu_set(ritr_pl, mtu);
- mlxsw_reg_ritr_if_mac_memcpy_to(ritr_pl, mac);
- mlxsw_reg_ritr_op_set(ritr_pl, MLXSW_REG_RITR_RIF_CREATE);
- return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ritr), ritr_pl);
-}
-
-static int mlxsw_sp_netdevice_router_port_event(struct net_device *dev)
-{
- struct mlxsw_sp *mlxsw_sp;
- struct mlxsw_sp_rif *r;
- int err;
-
- mlxsw_sp = mlxsw_sp_lower_get(dev);
- if (!mlxsw_sp)
- return 0;
-
- r = mlxsw_sp_rif_find_by_dev(mlxsw_sp, dev);
- if (!r)
- return 0;
-
- err = mlxsw_sp_rif_fdb_op(mlxsw_sp, r->addr, r->f->fid, false);
- if (err)
- return err;
-
- err = mlxsw_sp_rif_edit(mlxsw_sp, r->rif, dev->dev_addr, dev->mtu);
- if (err)
- goto err_rif_edit;
-
- err = mlxsw_sp_rif_fdb_op(mlxsw_sp, dev->dev_addr, r->f->fid, true);
- if (err)
- goto err_rif_fdb_op;
-
- ether_addr_copy(r->addr, dev->dev_addr);
- r->mtu = dev->mtu;
-
- netdev_dbg(dev, "Updated RIF=%d\n", r->rif);
-
- return 0;
-
-err_rif_fdb_op:
- mlxsw_sp_rif_edit(mlxsw_sp, r->rif, r->addr, r->mtu);
-err_rif_edit:
- mlxsw_sp_rif_fdb_op(mlxsw_sp, r->addr, r->f->fid, true);
- return err;
-}
-
static bool mlxsw_sp_lag_port_fid_member(struct mlxsw_sp_port *lag_port,
u16 fid)
{
@@ -4220,7 +3852,7 @@
static void
mlxsw_sp_port_pvid_vport_lag_join(struct mlxsw_sp_port *mlxsw_sp_port,
- u16 lag_id)
+ struct net_device *lag_dev, u16 lag_id)
{
struct mlxsw_sp_port *mlxsw_sp_vport;
struct mlxsw_sp_fid *f;
@@ -4238,6 +3870,7 @@
mlxsw_sp_vport->lag_id = lag_id;
mlxsw_sp_vport->lagged = 1;
+ mlxsw_sp_vport->dev = lag_dev;
}
static void
@@ -4254,6 +3887,7 @@
if (f)
f->leave(mlxsw_sp_vport);
+ mlxsw_sp_vport->dev = mlxsw_sp_port->dev;
mlxsw_sp_vport->lagged = 0;
}
@@ -4293,7 +3927,7 @@
mlxsw_sp_port->lagged = 1;
lag->ref_count++;
- mlxsw_sp_port_pvid_vport_lag_join(mlxsw_sp_port, lag_id);
+ mlxsw_sp_port_pvid_vport_lag_join(mlxsw_sp_port, lag_dev, lag_id);
return 0;
@@ -4403,6 +4037,56 @@
mlxsw_sp_vport->dev = mlxsw_sp_port->dev;
}
+static int mlxsw_sp_port_stp_set(struct mlxsw_sp_port *mlxsw_sp_port,
+ bool enable)
+{
+ struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp;
+ enum mlxsw_reg_spms_state spms_state;
+ char *spms_pl;
+ u16 vid;
+ int err;
+
+ spms_state = enable ? MLXSW_REG_SPMS_STATE_FORWARDING :
+ MLXSW_REG_SPMS_STATE_DISCARDING;
+
+ spms_pl = kmalloc(MLXSW_REG_SPMS_LEN, GFP_KERNEL);
+ if (!spms_pl)
+ return -ENOMEM;
+ mlxsw_reg_spms_pack(spms_pl, mlxsw_sp_port->local_port);
+
+ for (vid = 0; vid < VLAN_N_VID; vid++)
+ mlxsw_reg_spms_vid_pack(spms_pl, vid, spms_state);
+
+ err = mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(spms), spms_pl);
+ kfree(spms_pl);
+ return err;
+}
+
+static int mlxsw_sp_port_ovs_join(struct mlxsw_sp_port *mlxsw_sp_port)
+{
+ int err;
+
+ err = mlxsw_sp_port_stp_set(mlxsw_sp_port, true);
+ if (err)
+ return err;
+ err = mlxsw_sp_port_vlan_set(mlxsw_sp_port, 2, VLAN_N_VID - 1,
+ true, false);
+ if (err)
+ goto err_port_vlan_set;
+ return 0;
+
+err_port_vlan_set:
+ mlxsw_sp_port_stp_set(mlxsw_sp_port, false);
+ return err;
+}
+
+static void mlxsw_sp_port_ovs_leave(struct mlxsw_sp_port *mlxsw_sp_port)
+{
+ mlxsw_sp_port_vlan_set(mlxsw_sp_port, 2, VLAN_N_VID - 1,
+ false, false);
+ mlxsw_sp_port_stp_set(mlxsw_sp_port, false);
+}
+
static int mlxsw_sp_netdevice_port_upper_event(struct net_device *dev,
unsigned long event, void *ptr)
{
@@ -4421,7 +4105,8 @@
upper_dev = info->upper_dev;
if (!is_vlan_dev(upper_dev) &&
!netif_is_lag_master(upper_dev) &&
- !netif_is_bridge_master(upper_dev))
+ !netif_is_bridge_master(upper_dev) &&
+ !netif_is_ovs_master(upper_dev))
return -EINVAL;
if (!info->linking)
break;
@@ -4438,6 +4123,10 @@
if (netif_is_lag_port(dev) && is_vlan_dev(upper_dev) &&
!netif_is_lag_master(vlan_dev_real_dev(upper_dev)))
return -EINVAL;
+ if (netif_is_ovs_master(upper_dev) && vlan_uses_dev(dev))
+ return -EINVAL;
+ if (netif_is_ovs_port(dev) && is_vlan_dev(upper_dev))
+ return -EINVAL;
break;
case NETDEV_CHANGEUPPER:
upper_dev = info->upper_dev;
@@ -4446,8 +4135,8 @@
err = mlxsw_sp_port_vlan_link(mlxsw_sp_port,
upper_dev);
else
- mlxsw_sp_port_vlan_unlink(mlxsw_sp_port,
- upper_dev);
+ mlxsw_sp_port_vlan_unlink(mlxsw_sp_port,
+ upper_dev);
} else if (netif_is_bridge_master(upper_dev)) {
if (info->linking)
err = mlxsw_sp_port_bridge_join(mlxsw_sp_port,
@@ -4461,6 +4150,11 @@
else
mlxsw_sp_port_lag_leave(mlxsw_sp_port,
upper_dev);
+ } else if (netif_is_ovs_master(upper_dev)) {
+ if (info->linking)
+ err = mlxsw_sp_port_ovs_join(mlxsw_sp_port);
+ else
+ mlxsw_sp_port_ovs_leave(mlxsw_sp_port);
} else {
err = -EINVAL;
WARN_ON(1);
@@ -4552,8 +4246,8 @@
struct mlxsw_sp_fid *f;
f = mlxsw_sp_fid_find(mlxsw_sp, fid);
- if (f && f->r)
- mlxsw_sp_rif_bridge_destroy(mlxsw_sp, f->r);
+ if (f && f->rif)
+ mlxsw_sp_rif_bridge_destroy(mlxsw_sp, f->rif);
if (f && --f->ref_count == 0)
mlxsw_sp_fid_destroy(mlxsw_sp, f);
}
@@ -4564,33 +4258,40 @@
struct netdev_notifier_changeupper_info *info;
struct net_device *upper_dev;
struct mlxsw_sp *mlxsw_sp;
- int err;
+ int err = 0;
mlxsw_sp = mlxsw_sp_lower_get(br_dev);
if (!mlxsw_sp)
return 0;
- if (br_dev != mlxsw_sp->master_bridge.dev)
- return 0;
info = ptr;
switch (event) {
- case NETDEV_CHANGEUPPER:
+ case NETDEV_PRECHANGEUPPER:
upper_dev = info->upper_dev;
if (!is_vlan_dev(upper_dev))
- break;
- if (info->linking) {
- err = mlxsw_sp_master_bridge_vlan_link(mlxsw_sp,
- upper_dev);
- if (err)
- return err;
+ return -EINVAL;
+ if (is_vlan_dev(upper_dev) &&
+ br_dev != mlxsw_sp->master_bridge.dev)
+ return -EINVAL;
+ break;
+ case NETDEV_CHANGEUPPER:
+ upper_dev = info->upper_dev;
+ if (is_vlan_dev(upper_dev)) {
+ if (info->linking)
+ err = mlxsw_sp_master_bridge_vlan_link(mlxsw_sp,
+ upper_dev);
+ else
+ mlxsw_sp_master_bridge_vlan_unlink(mlxsw_sp,
+ upper_dev);
} else {
- mlxsw_sp_master_bridge_vlan_unlink(mlxsw_sp, upper_dev);
+ err = -EINVAL;
+ WARN_ON(1);
}
break;
}
- return 0;
+ return err;
}
static u16 mlxsw_sp_avail_vfid_get(const struct mlxsw_sp *mlxsw_sp)
@@ -4657,8 +4358,8 @@
clear_bit(vfid, mlxsw_sp->vfids.mapped);
list_del(&f->list);
- if (f->r)
- mlxsw_sp_rif_bridge_destroy(mlxsw_sp, f->r);
+ if (f->rif)
+ mlxsw_sp_rif_bridge_destroy(mlxsw_sp, f->rif);
kfree(f);
@@ -4810,6 +4511,8 @@
int err = 0;
mlxsw_sp_vport = mlxsw_sp_port_vport_find(mlxsw_sp_port, vid);
+ if (!mlxsw_sp_vport)
+ return 0;
switch (event) {
case NETDEV_PRECHANGEUPPER:
@@ -4821,22 +4524,24 @@
/* We can't have multiple VLAN interfaces configured on
* the same port and being members in the same bridge.
*/
- if (!mlxsw_sp_port_master_bridge_check(mlxsw_sp_port,
+ if (netif_is_bridge_master(upper_dev) &&
+ !mlxsw_sp_port_master_bridge_check(mlxsw_sp_port,
upper_dev))
return -EINVAL;
break;
case NETDEV_CHANGEUPPER:
upper_dev = info->upper_dev;
- if (info->linking) {
- if (WARN_ON(!mlxsw_sp_vport))
- return -EINVAL;
- err = mlxsw_sp_vport_bridge_join(mlxsw_sp_vport,
- upper_dev);
+ if (netif_is_bridge_master(upper_dev)) {
+ if (info->linking)
+ err = mlxsw_sp_vport_bridge_join(mlxsw_sp_vport,
+ upper_dev);
+ else
+ mlxsw_sp_vport_bridge_leave(mlxsw_sp_vport);
} else {
- if (!mlxsw_sp_vport)
- return 0;
- mlxsw_sp_vport_bridge_leave(mlxsw_sp_vport);
+ err = -EINVAL;
+ WARN_ON(1);
}
+ break;
}
return err;
@@ -4878,6 +4583,15 @@
return 0;
}
+static bool mlxsw_sp_is_vrf_event(unsigned long event, void *ptr)
+{
+ struct netdev_notifier_changeupper_info *info = ptr;
+
+ if (event != NETDEV_PRECHANGEUPPER && event != NETDEV_CHANGEUPPER)
+ return false;
+ return netif_is_l3_master(info->upper_dev);
+}
+
static int mlxsw_sp_netdevice_event(struct notifier_block *unused,
unsigned long event, void *ptr)
{
@@ -4886,6 +4600,8 @@
if (event == NETDEV_CHANGEADDR || event == NETDEV_CHANGEMTU)
err = mlxsw_sp_netdevice_router_port_event(dev);
+ else if (mlxsw_sp_is_vrf_event(event, ptr))
+ err = mlxsw_sp_netdevice_vrf_event(dev, event, ptr);
else if (mlxsw_sp_port_dev_check(dev))
err = mlxsw_sp_netdevice_port_event(dev, event, ptr);
else if (netif_is_lag_master(dev))
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
index 13ec85e..0c23bc1 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h
@@ -57,41 +57,21 @@
#define MLXSW_SP_VFID_BASE VLAN_N_VID
#define MLXSW_SP_VFID_MAX 1024 /* Bridged VLAN interfaces */
+#define MLXSW_SP_DUMMY_FID 15359
+
#define MLXSW_SP_RFID_BASE 15360
-#define MLXSW_SP_INVALID_RIF 0xffff
#define MLXSW_SP_MID_MAX 7000
#define MLXSW_SP_PORTS_PER_CLUSTER_MAX 4
-#define MLXSW_SP_LPM_TREE_MIN 2 /* trees 0 and 1 are reserved */
-#define MLXSW_SP_LPM_TREE_MAX 22
-#define MLXSW_SP_LPM_TREE_COUNT (MLXSW_SP_LPM_TREE_MAX - MLXSW_SP_LPM_TREE_MIN)
-
#define MLXSW_SP_PORT_BASE_SPEED 25000 /* Mb/s */
-#define MLXSW_SP_BYTES_PER_CELL 96
-
-#define MLXSW_SP_BYTES_TO_CELLS(b) DIV_ROUND_UP(b, MLXSW_SP_BYTES_PER_CELL)
-#define MLXSW_SP_CELLS_TO_BYTES(c) (c * MLXSW_SP_BYTES_PER_CELL)
-
#define MLXSW_SP_KVD_LINEAR_SIZE 65536 /* entries */
#define MLXSW_SP_KVD_GRANULARITY 128
-/* Maximum delay buffer needed in case of PAUSE frames, in cells.
- * Assumes 100m cable and maximum MTU.
- */
-#define MLXSW_SP_PAUSE_DELAY 612
-
-#define MLXSW_SP_CELL_FACTOR 2 /* 2 * cell_size / (IPG + cell_size + 1) */
-
-static inline u16 mlxsw_sp_pfc_delay_get(int mtu, u16 delay)
-{
- delay = MLXSW_SP_BYTES_TO_CELLS(DIV_ROUND_UP(delay, BITS_PER_BYTE));
- return MLXSW_SP_CELL_FACTOR * delay + MLXSW_SP_BYTES_TO_CELLS(mtu);
-}
-
struct mlxsw_sp_port;
+struct mlxsw_sp_rif;
struct mlxsw_sp_upper {
struct net_device *dev;
@@ -103,21 +83,10 @@
struct list_head list;
unsigned int ref_count;
struct net_device *dev;
- struct mlxsw_sp_rif *r;
+ struct mlxsw_sp_rif *rif;
u16 fid;
};
-struct mlxsw_sp_rif {
- struct list_head nexthop_list;
- struct list_head neigh_list;
- struct net_device *dev;
- unsigned int ref_count;
- struct mlxsw_sp_fid *f;
- unsigned char addr[ETH_ALEN];
- int mtu;
- u16 rif;
-};
-
struct mlxsw_sp_mid {
struct list_head list;
unsigned char addr[ETH_ALEN];
@@ -138,17 +107,7 @@
static inline bool mlxsw_sp_fid_is_vfid(u16 fid)
{
- return fid >= MLXSW_SP_VFID_BASE && fid < MLXSW_SP_RFID_BASE;
-}
-
-static inline bool mlxsw_sp_fid_is_rfid(u16 fid)
-{
- return fid >= MLXSW_SP_RFID_BASE;
-}
-
-static inline u16 mlxsw_sp_rif_sp_to_fid(u16 rif)
-{
- return MLXSW_SP_RFID_BASE + rif;
+ return fid >= MLXSW_SP_VFID_BASE && fid < MLXSW_SP_DUMMY_FID;
}
struct mlxsw_sp_sb_pr {
@@ -177,12 +136,15 @@
#define MLXSW_SP_SB_POOL_COUNT 4
#define MLXSW_SP_SB_TC_COUNT 8
+struct mlxsw_sp_sb_port {
+ struct mlxsw_sp_sb_cm cms[2][MLXSW_SP_SB_TC_COUNT];
+ struct mlxsw_sp_sb_pm pms[2][MLXSW_SP_SB_POOL_COUNT];
+};
+
struct mlxsw_sp_sb {
struct mlxsw_sp_sb_pr prs[2][MLXSW_SP_SB_POOL_COUNT];
- struct {
- struct mlxsw_sp_sb_cm cms[2][MLXSW_SP_SB_TC_COUNT];
- struct mlxsw_sp_sb_pm pms[2][MLXSW_SP_SB_POOL_COUNT];
- } ports[MLXSW_PORT_MAX_PORTS];
+ struct mlxsw_sp_sb_port *ports;
+ u32 cell_size;
};
#define MLXSW_SP_PREFIX_COUNT (sizeof(struct in6_addr) * BITS_PER_BYTE)
@@ -207,11 +169,9 @@
struct mlxsw_sp_vr {
u16 id; /* virtual router ID */
- bool used;
- enum mlxsw_sp_l3proto proto;
u32 tb_id; /* kernel fib table id */
- struct mlxsw_sp_lpm_tree *lpm_tree;
- struct mlxsw_sp_fib *fib;
+ unsigned int rif_count;
+ struct mlxsw_sp_fib *fib4;
};
enum mlxsw_sp_span_type {
@@ -253,12 +213,15 @@
};
struct mlxsw_sp_router {
- struct mlxsw_sp_lpm_tree lpm_trees[MLXSW_SP_LPM_TREE_COUNT];
struct mlxsw_sp_vr *vrs;
struct rhashtable neigh_ht;
struct rhashtable nexthop_group_ht;
struct rhashtable nexthop_ht;
struct {
+ struct mlxsw_sp_lpm_tree *trees;
+ unsigned int tree_count;
+ } lpm;
+ struct {
struct delayed_work dw;
unsigned long interval; /* ms */
} neighs_update;
@@ -269,6 +232,7 @@
};
struct mlxsw_sp_acl;
+struct mlxsw_sp_counter_pool;
struct mlxsw_sp {
struct {
@@ -296,7 +260,7 @@
u32 ageing_time;
struct mlxsw_sp_upper master_bridge;
struct mlxsw_sp_upper *lags;
- u8 port_to_module[MLXSW_PORT_MAX_PORTS];
+ u8 *port_to_module;
struct mlxsw_sp_sb sb;
struct mlxsw_sp_router router;
struct mlxsw_sp_acl *acl;
@@ -304,6 +268,7 @@
DECLARE_BITMAP(usage, MLXSW_SP_KVD_LINEAR_SIZE);
} kvdl;
+ struct mlxsw_sp_counter_pool *counter_pool;
struct {
struct mlxsw_sp_span_entry *entries;
int entries_count;
@@ -317,6 +282,18 @@
return &mlxsw_sp->lags[lag_id];
}
+static inline u32 mlxsw_sp_cells_bytes(const struct mlxsw_sp *mlxsw_sp,
+ u32 cells)
+{
+ return mlxsw_sp->sb.cell_size * cells;
+}
+
+static inline u32 mlxsw_sp_bytes_cells(const struct mlxsw_sp *mlxsw_sp,
+ u32 bytes)
+{
+ return DIV_ROUND_UP(bytes, mlxsw_sp->sb.cell_size);
+}
+
struct mlxsw_sp_port_pcpu_stats {
u64 rx_packets;
u64 rx_bytes;
@@ -386,6 +363,7 @@
};
bool mlxsw_sp_port_dev_check(const struct net_device *dev);
+struct mlxsw_sp *mlxsw_sp_lower_get(struct net_device *dev);
struct mlxsw_sp_port *mlxsw_sp_port_lower_dev_hold(struct net_device *dev);
void mlxsw_sp_port_dev_put(struct mlxsw_sp_port *mlxsw_sp_port);
@@ -497,19 +475,6 @@
return NULL;
}
-static inline struct mlxsw_sp_rif *
-mlxsw_sp_rif_find_by_dev(const struct mlxsw_sp *mlxsw_sp,
- const struct net_device *dev)
-{
- int i;
-
- for (i = 0; i < MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_RIFS); i++)
- if (mlxsw_sp->rifs[i] && mlxsw_sp->rifs[i]->dev == dev)
- return mlxsw_sp->rifs[i];
-
- return NULL;
-}
-
enum mlxsw_sp_flood_table {
MLXSW_SP_FLOOD_TABLE_UC,
MLXSW_SP_FLOOD_TABLE_BC,
@@ -570,8 +535,6 @@
bool adding);
struct mlxsw_sp_fid *mlxsw_sp_fid_create(struct mlxsw_sp *mlxsw_sp, u16 fid);
void mlxsw_sp_fid_destroy(struct mlxsw_sp *mlxsw_sp, struct mlxsw_sp_fid *f);
-void mlxsw_sp_rif_bridge_destroy(struct mlxsw_sp *mlxsw_sp,
- struct mlxsw_sp_rif *r);
int mlxsw_sp_port_ets_set(struct mlxsw_sp_port *mlxsw_sp_port,
enum mlxsw_reg_qeec_hr hr, u8 index, u8 next_index,
bool dwrr, u8 dwrr_weight);
@@ -608,10 +571,16 @@
void mlxsw_sp_router_fini(struct mlxsw_sp *mlxsw_sp);
int mlxsw_sp_router_netevent_event(struct notifier_block *unused,
unsigned long event, void *ptr);
-void mlxsw_sp_router_rif_gone_sync(struct mlxsw_sp *mlxsw_sp,
- struct mlxsw_sp_rif *r);
+int mlxsw_sp_netdevice_router_port_event(struct net_device *dev);
+int mlxsw_sp_inetaddr_event(struct notifier_block *unused,
+ unsigned long event, void *ptr);
+void mlxsw_sp_rif_bridge_destroy(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_rif *rif);
+int mlxsw_sp_netdevice_vrf_event(struct net_device *l3_dev, unsigned long event,
+ struct netdev_notifier_changeupper_info *info);
-int mlxsw_sp_kvdl_alloc(struct mlxsw_sp *mlxsw_sp, unsigned int entry_count);
+int mlxsw_sp_kvdl_alloc(struct mlxsw_sp *mlxsw_sp, unsigned int entry_count,
+ u32 *p_entry_index);
void mlxsw_sp_kvdl_free(struct mlxsw_sp *mlxsw_sp, int entry_index);
struct mlxsw_afk *mlxsw_sp_acl_afk(struct mlxsw_sp_acl *acl);
@@ -620,6 +589,8 @@
unsigned int priority;
struct mlxsw_afk_element_values values;
struct mlxsw_afa_block *act_block;
+ unsigned int counter_index;
+ bool counter_valid;
};
enum mlxsw_sp_acl_profile {
@@ -639,6 +610,8 @@
void *ruleset_priv, void *rule_priv,
struct mlxsw_sp_acl_rule_info *rulei);
void (*rule_del)(struct mlxsw_sp *mlxsw_sp, void *rule_priv);
+ int (*rule_activity_get)(struct mlxsw_sp *mlxsw_sp, void *rule_priv,
+ bool *activity);
};
struct mlxsw_sp_acl_ops {
@@ -679,6 +652,14 @@
int mlxsw_sp_acl_rulei_act_fwd(struct mlxsw_sp *mlxsw_sp,
struct mlxsw_sp_acl_rule_info *rulei,
struct net_device *out_dev);
+int mlxsw_sp_acl_rulei_act_vlan(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_rule_info *rulei,
+ u32 action, u16 vid, u16 proto, u8 prio);
+int mlxsw_sp_acl_rulei_act_count(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_rule_info *rulei);
+int mlxsw_sp_acl_rulei_act_fid_set(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_rule_info *rulei,
+ u16 fid);
struct mlxsw_sp_acl_rule;
@@ -698,6 +679,9 @@
unsigned long cookie);
struct mlxsw_sp_acl_rule_info *
mlxsw_sp_acl_rule_rulei(struct mlxsw_sp_acl_rule *rule);
+int mlxsw_sp_acl_rule_get_stats(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_rule *rule,
+ u64 *packets, u64 *bytes, u64 *last_use);
int mlxsw_sp_acl_init(struct mlxsw_sp *mlxsw_sp);
void mlxsw_sp_acl_fini(struct mlxsw_sp *mlxsw_sp);
@@ -708,5 +692,14 @@
__be16 protocol, struct tc_cls_flower_offload *f);
void mlxsw_sp_flower_destroy(struct mlxsw_sp_port *mlxsw_sp_port, bool ingress,
struct tc_cls_flower_offload *f);
+int mlxsw_sp_flower_stats(struct mlxsw_sp_port *mlxsw_sp_port, bool ingress,
+ struct tc_cls_flower_offload *f);
+int mlxsw_sp_flow_counter_get(struct mlxsw_sp *mlxsw_sp,
+ unsigned int counter_index, u64 *packets,
+ u64 *bytes);
+int mlxsw_sp_flow_counter_alloc(struct mlxsw_sp *mlxsw_sp,
+ unsigned int *p_counter_index);
+void mlxsw_sp_flow_counter_free(struct mlxsw_sp *mlxsw_sp,
+ unsigned int counter_index);
#endif
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl.c
index 8a18b3a..317f7b1 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl.c
@@ -39,6 +39,7 @@
#include <linux/string.h>
#include <linux/rhashtable.h>
#include <linux/netdevice.h>
+#include <net/tc_act/tc_vlan.h>
#include "reg.h"
#include "core.h"
@@ -49,10 +50,17 @@
#include "spectrum_acl_flex_keys.h"
struct mlxsw_sp_acl {
+ struct mlxsw_sp *mlxsw_sp;
struct mlxsw_afk *afk;
struct mlxsw_afa *afa;
const struct mlxsw_sp_acl_ops *ops;
struct rhashtable ruleset_ht;
+ struct list_head rules;
+ struct {
+ struct delayed_work dw;
+ unsigned long interval; /* ms */
+#define MLXSW_SP_ACL_RULE_ACTIVITY_UPDATE_PERIOD_MS 1000
+ } rule_activity_update;
unsigned long priv[0];
/* priv has to be always the last item */
};
@@ -79,9 +87,13 @@
struct mlxsw_sp_acl_rule {
struct rhash_head ht_node; /* Member of rule HT */
+ struct list_head list;
unsigned long cookie; /* HT key */
struct mlxsw_sp_acl_ruleset *ruleset;
struct mlxsw_sp_acl_rule_info *rulei;
+ u64 last_used;
+ u64 last_packets;
+ u64 last_bytes;
unsigned long priv[0];
/* priv has to be always the last item */
};
@@ -237,6 +249,27 @@
mlxsw_sp_acl_ruleset_ref_dec(mlxsw_sp, ruleset);
}
+static int
+mlxsw_sp_acl_rulei_counter_alloc(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_rule_info *rulei)
+{
+ int err;
+
+ err = mlxsw_sp_flow_counter_alloc(mlxsw_sp, &rulei->counter_index);
+ if (err)
+ return err;
+ rulei->counter_valid = true;
+ return 0;
+}
+
+static void
+mlxsw_sp_acl_rulei_counter_free(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_rule_info *rulei)
+{
+ rulei->counter_valid = false;
+ mlxsw_sp_flow_counter_free(mlxsw_sp, rulei->counter_index);
+}
+
struct mlxsw_sp_acl_rule_info *
mlxsw_sp_acl_rulei_create(struct mlxsw_sp_acl *acl)
{
@@ -335,6 +368,48 @@
local_port, in_port);
}
+int mlxsw_sp_acl_rulei_act_vlan(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_rule_info *rulei,
+ u32 action, u16 vid, u16 proto, u8 prio)
+{
+ u8 ethertype;
+
+ if (action == TCA_VLAN_ACT_MODIFY) {
+ switch (proto) {
+ case ETH_P_8021Q:
+ ethertype = 0;
+ break;
+ case ETH_P_8021AD:
+ ethertype = 1;
+ break;
+ default:
+ dev_err(mlxsw_sp->bus_info->dev, "Unsupported VLAN protocol %#04x\n",
+ proto);
+ return -EINVAL;
+ }
+
+ return mlxsw_afa_block_append_vlan_modify(rulei->act_block,
+ vid, prio, ethertype);
+ } else {
+ dev_err(mlxsw_sp->bus_info->dev, "Unsupported VLAN action\n");
+ return -EINVAL;
+ }
+}
+
+int mlxsw_sp_acl_rulei_act_count(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_rule_info *rulei)
+{
+ return mlxsw_afa_block_append_counter(rulei->act_block,
+ rulei->counter_index);
+}
+
+int mlxsw_sp_acl_rulei_act_fid_set(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_rule_info *rulei,
+ u16 fid)
+{
+ return mlxsw_afa_block_append_fid_set(rulei->act_block, fid);
+}
+
struct mlxsw_sp_acl_rule *
mlxsw_sp_acl_rule_create(struct mlxsw_sp *mlxsw_sp,
struct mlxsw_sp_acl_ruleset *ruleset,
@@ -358,8 +433,14 @@
err = PTR_ERR(rule->rulei);
goto err_rulei_create;
}
+
+ err = mlxsw_sp_acl_rulei_counter_alloc(mlxsw_sp, rule->rulei);
+ if (err)
+ goto err_counter_alloc;
return rule;
+err_counter_alloc:
+ mlxsw_sp_acl_rulei_destroy(rule->rulei);
err_rulei_create:
kfree(rule);
err_alloc:
@@ -372,6 +453,7 @@
{
struct mlxsw_sp_acl_ruleset *ruleset = rule->ruleset;
+ mlxsw_sp_acl_rulei_counter_free(mlxsw_sp, rule->rulei);
mlxsw_sp_acl_rulei_destroy(rule->rulei);
kfree(rule);
mlxsw_sp_acl_ruleset_ref_dec(mlxsw_sp, ruleset);
@@ -393,6 +475,7 @@
if (err)
goto err_rhashtable_insert;
+ list_add_tail(&rule->list, &mlxsw_sp->acl->rules);
return 0;
err_rhashtable_insert:
@@ -406,6 +489,7 @@
struct mlxsw_sp_acl_ruleset *ruleset = rule->ruleset;
const struct mlxsw_sp_acl_profile_ops *ops = ruleset->ht_key.ops;
+ list_del(&rule->list);
rhashtable_remove_fast(&ruleset->rule_ht, &rule->ht_node,
mlxsw_sp_acl_rule_ht_params);
ops->rule_del(mlxsw_sp, rule->priv);
@@ -426,6 +510,90 @@
return rule->rulei;
}
+static int mlxsw_sp_acl_rule_activity_update(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_rule *rule)
+{
+ struct mlxsw_sp_acl_ruleset *ruleset = rule->ruleset;
+ const struct mlxsw_sp_acl_profile_ops *ops = ruleset->ht_key.ops;
+ bool active;
+ int err;
+
+ err = ops->rule_activity_get(mlxsw_sp, rule->priv, &active);
+ if (err)
+ return err;
+ if (active)
+ rule->last_used = jiffies;
+ return 0;
+}
+
+static int mlxsw_sp_acl_rules_activity_update(struct mlxsw_sp_acl *acl)
+{
+ struct mlxsw_sp_acl_rule *rule;
+ int err;
+
+ /* Protect internal structures from changes */
+ rtnl_lock();
+ list_for_each_entry(rule, &acl->rules, list) {
+ err = mlxsw_sp_acl_rule_activity_update(acl->mlxsw_sp,
+ rule);
+ if (err)
+ goto err_rule_update;
+ }
+ rtnl_unlock();
+ return 0;
+
+err_rule_update:
+ rtnl_unlock();
+ return err;
+}
+
+static void mlxsw_sp_acl_rule_activity_work_schedule(struct mlxsw_sp_acl *acl)
+{
+ unsigned long interval = acl->rule_activity_update.interval;
+
+ mlxsw_core_schedule_dw(&acl->rule_activity_update.dw,
+ msecs_to_jiffies(interval));
+}
+
+static void mlxsw_sp_acl_rul_activity_update_work(struct work_struct *work)
+{
+ struct mlxsw_sp_acl *acl = container_of(work, struct mlxsw_sp_acl,
+ rule_activity_update.dw.work);
+ int err;
+
+ err = mlxsw_sp_acl_rules_activity_update(acl);
+ if (err)
+ dev_err(acl->mlxsw_sp->bus_info->dev, "Could not update acl activity");
+
+ mlxsw_sp_acl_rule_activity_work_schedule(acl);
+}
+
+int mlxsw_sp_acl_rule_get_stats(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_rule *rule,
+ u64 *packets, u64 *bytes, u64 *last_use)
+
+{
+ struct mlxsw_sp_acl_rule_info *rulei;
+ u64 current_packets;
+ u64 current_bytes;
+ int err;
+
+ rulei = mlxsw_sp_acl_rule_rulei(rule);
+ err = mlxsw_sp_flow_counter_get(mlxsw_sp, rulei->counter_index,
+ ¤t_packets, ¤t_bytes);
+ if (err)
+ return err;
+
+ *packets = current_packets - rule->last_packets;
+ *bytes = current_bytes - rule->last_bytes;
+ *last_use = rule->last_used;
+
+ rule->last_bytes = current_bytes;
+ rule->last_packets = current_packets;
+
+ return 0;
+}
+
#define MLXSW_SP_KDVL_ACT_EXT_SIZE 1
static int mlxsw_sp_act_kvdl_set_add(void *priv, u32 *p_kvdl_index,
@@ -434,7 +602,6 @@
struct mlxsw_sp *mlxsw_sp = priv;
char pefa_pl[MLXSW_REG_PEFA_LEN];
u32 kvdl_index;
- int ret;
int err;
/* The first action set of a TCAM entry is stored directly in TCAM,
@@ -443,10 +610,10 @@
if (is_first)
return 0;
- ret = mlxsw_sp_kvdl_alloc(mlxsw_sp, MLXSW_SP_KDVL_ACT_EXT_SIZE);
- if (ret < 0)
- return ret;
- kvdl_index = ret;
+ err = mlxsw_sp_kvdl_alloc(mlxsw_sp, MLXSW_SP_KDVL_ACT_EXT_SIZE,
+ &kvdl_index);
+ if (err)
+ return err;
mlxsw_reg_pefa_pack(pefa_pl, kvdl_index, enc_actions);
err = mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(pefa), pefa_pl);
if (err)
@@ -475,13 +642,11 @@
struct mlxsw_sp *mlxsw_sp = priv;
char ppbs_pl[MLXSW_REG_PPBS_LEN];
u32 kvdl_index;
- int ret;
int err;
- ret = mlxsw_sp_kvdl_alloc(mlxsw_sp, 1);
- if (ret < 0)
- return ret;
- kvdl_index = ret;
+ err = mlxsw_sp_kvdl_alloc(mlxsw_sp, 1, &kvdl_index);
+ if (err)
+ return err;
mlxsw_reg_ppbs_pack(ppbs_pl, kvdl_index, local_port);
err = mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ppbs), ppbs_pl);
if (err)
@@ -518,7 +683,7 @@
if (!acl)
return -ENOMEM;
mlxsw_sp->acl = acl;
-
+ acl->mlxsw_sp = mlxsw_sp;
acl->afk = mlxsw_afk_create(MLXSW_CORE_RES_GET(mlxsw_sp->core,
ACL_FLEX_KEYS),
mlxsw_sp_afk_blocks,
@@ -541,11 +706,18 @@
if (err)
goto err_rhashtable_init;
+ INIT_LIST_HEAD(&acl->rules);
err = acl_ops->init(mlxsw_sp, acl->priv);
if (err)
goto err_acl_ops_init;
acl->ops = acl_ops;
+
+ /* Create the delayed work for the rule activity_update */
+ INIT_DELAYED_WORK(&acl->rule_activity_update.dw,
+ mlxsw_sp_acl_rul_activity_update_work);
+ acl->rule_activity_update.interval = MLXSW_SP_ACL_RULE_ACTIVITY_UPDATE_PERIOD_MS;
+ mlxsw_core_schedule_dw(&acl->rule_activity_update.dw, 0);
return 0;
err_acl_ops_init:
@@ -564,7 +736,9 @@
struct mlxsw_sp_acl *acl = mlxsw_sp->acl;
const struct mlxsw_sp_acl_ops *acl_ops = acl->ops;
+ cancel_delayed_work_sync(&mlxsw_sp->acl->rule_activity_update.dw);
acl_ops->fini(mlxsw_sp, acl->priv);
+ WARN_ON(!list_empty(&acl->rules));
rhashtable_destroy(&acl->ruleset_ht);
mlxsw_afa_destroy(acl->afa);
mlxsw_afk_destroy(acl->afk);
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_flex_keys.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_flex_keys.h
index 82b81cf..af7b7ba 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_flex_keys.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_flex_keys.h
@@ -39,11 +39,15 @@
static struct mlxsw_afk_element_inst mlxsw_sp_afk_element_info_l2_dmac[] = {
MLXSW_AFK_ELEMENT_INST_BUF(DMAC, 0x00, 6),
+ MLXSW_AFK_ELEMENT_INST_U32(PCP, 0x08, 13, 3),
+ MLXSW_AFK_ELEMENT_INST_U32(VID, 0x08, 0, 12),
MLXSW_AFK_ELEMENT_INST_U32(SRC_SYS_PORT, 0x0C, 0, 16),
};
static struct mlxsw_afk_element_inst mlxsw_sp_afk_element_info_l2_smac[] = {
MLXSW_AFK_ELEMENT_INST_BUF(SMAC, 0x00, 6),
+ MLXSW_AFK_ELEMENT_INST_U32(PCP, 0x08, 13, 3),
+ MLXSW_AFK_ELEMENT_INST_U32(VID, 0x08, 0, 12),
MLXSW_AFK_ELEMENT_INST_U32(SRC_SYS_PORT, 0x0C, 0, 16),
};
@@ -65,6 +69,8 @@
};
static struct mlxsw_afk_element_inst mlxsw_sp_afk_element_info_ipv4_ex[] = {
+ MLXSW_AFK_ELEMENT_INST_U32(VID, 0x00, 0, 12),
+ MLXSW_AFK_ELEMENT_INST_U32(PCP, 0x08, 29, 3),
MLXSW_AFK_ELEMENT_INST_U32(SRC_L4_PORT, 0x08, 0, 16),
MLXSW_AFK_ELEMENT_INST_U32(DST_L4_PORT, 0x0C, 0, 16),
};
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_tcam.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_tcam.c
index 7382832..3a24289 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_tcam.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl_tcam.c
@@ -561,6 +561,24 @@
mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ptce2), ptce2_pl);
}
+static int
+mlxsw_sp_acl_tcam_region_entry_activity_get(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_tcam_region *region,
+ unsigned int offset,
+ bool *activity)
+{
+ char ptce2_pl[MLXSW_REG_PTCE2_LEN];
+ int err;
+
+ mlxsw_reg_ptce2_pack(ptce2_pl, true, MLXSW_REG_PTCE2_OP_QUERY_CLEAR_ON_READ,
+ region->tcam_region_info, offset);
+ err = mlxsw_reg_query(mlxsw_sp->core, MLXSW_REG(ptce2), ptce2_pl);
+ if (err)
+ return err;
+ *activity = mlxsw_reg_ptce2_a_get(ptce2_pl);
+ return 0;
+}
+
#define MLXSW_SP_ACL_TCAM_CATCHALL_PRIO (~0U)
static int
@@ -940,6 +958,19 @@
mlxsw_sp_acl_tcam_chunk_put(mlxsw_sp, chunk);
}
+static int
+mlxsw_sp_acl_tcam_entry_activity_get(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_acl_tcam_entry *entry,
+ bool *activity)
+{
+ struct mlxsw_sp_acl_tcam_chunk *chunk = entry->chunk;
+ struct mlxsw_sp_acl_tcam_region *region = chunk->region;
+
+ return mlxsw_sp_acl_tcam_region_entry_activity_get(mlxsw_sp, region,
+ entry->parman_item.index,
+ activity);
+}
+
static const enum mlxsw_afk_element mlxsw_sp_acl_tcam_pattern_ipv4[] = {
MLXSW_AFK_ELEMENT_SRC_SYS_PORT,
MLXSW_AFK_ELEMENT_DMAC,
@@ -950,6 +981,8 @@
MLXSW_AFK_ELEMENT_DST_IP4,
MLXSW_AFK_ELEMENT_DST_L4_PORT,
MLXSW_AFK_ELEMENT_SRC_L4_PORT,
+ MLXSW_AFK_ELEMENT_VID,
+ MLXSW_AFK_ELEMENT_PCP,
};
static const enum mlxsw_afk_element mlxsw_sp_acl_tcam_pattern_ipv6[] = {
@@ -1046,6 +1079,16 @@
mlxsw_sp_acl_tcam_entry_del(mlxsw_sp, &rule->entry);
}
+static int
+mlxsw_sp_acl_tcam_flower_rule_activity_get(struct mlxsw_sp *mlxsw_sp,
+ void *rule_priv, bool *activity)
+{
+ struct mlxsw_sp_acl_tcam_flower_rule *rule = rule_priv;
+
+ return mlxsw_sp_acl_tcam_entry_activity_get(mlxsw_sp, &rule->entry,
+ activity);
+}
+
static const struct mlxsw_sp_acl_profile_ops mlxsw_sp_acl_tcam_flower_ops = {
.ruleset_priv_size = sizeof(struct mlxsw_sp_acl_tcam_flower_ruleset),
.ruleset_add = mlxsw_sp_acl_tcam_flower_ruleset_add,
@@ -1055,6 +1098,7 @@
.rule_priv_size = sizeof(struct mlxsw_sp_acl_tcam_flower_rule),
.rule_add = mlxsw_sp_acl_tcam_flower_rule_add,
.rule_del = mlxsw_sp_acl_tcam_flower_rule_del,
+ .rule_activity_get = mlxsw_sp_acl_tcam_flower_rule_activity_get,
};
static const struct mlxsw_sp_acl_profile_ops *
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_buffers.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_buffers.c
index a746826..997189c 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_buffers.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_buffers.c
@@ -162,8 +162,8 @@
}
static const u16 mlxsw_sp_pbs[] = {
- [0] = 2 * MLXSW_SP_BYTES_TO_CELLS(ETH_FRAME_LEN),
- [9] = 2 * MLXSW_SP_BYTES_TO_CELLS(MLXSW_PORT_MAX_MTU),
+ [0] = 2 * ETH_FRAME_LEN,
+ [9] = 2 * MLXSW_PORT_MAX_MTU,
};
#define MLXSW_SP_PBS_LEN ARRAY_SIZE(mlxsw_sp_pbs)
@@ -171,20 +171,22 @@
static int mlxsw_sp_port_pb_init(struct mlxsw_sp_port *mlxsw_sp_port)
{
+ struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp;
char pbmc_pl[MLXSW_REG_PBMC_LEN];
int i;
mlxsw_reg_pbmc_pack(pbmc_pl, mlxsw_sp_port->local_port,
0xffff, 0xffff / 2);
for (i = 0; i < MLXSW_SP_PBS_LEN; i++) {
+ u16 size = mlxsw_sp_bytes_cells(mlxsw_sp, mlxsw_sp_pbs[i]);
+
if (i == MLXSW_SP_PB_UNUSED)
continue;
- mlxsw_reg_pbmc_lossy_buffer_pack(pbmc_pl, i, mlxsw_sp_pbs[i]);
+ mlxsw_reg_pbmc_lossy_buffer_pack(pbmc_pl, i, size);
}
mlxsw_reg_pbmc_lossy_buffer_pack(pbmc_pl,
MLXSW_REG_PBMC_PORT_SHARED_BUF_IDX, 0);
- return mlxsw_reg_write(mlxsw_sp_port->mlxsw_sp->core,
- MLXSW_REG(pbmc), pbmc_pl);
+ return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(pbmc), pbmc_pl);
}
static int mlxsw_sp_port_pb_prio_init(struct mlxsw_sp_port *mlxsw_sp_port)
@@ -209,11 +211,25 @@
return mlxsw_sp_port_pb_prio_init(mlxsw_sp_port);
}
-#define MLXSW_SP_SB_PR_INGRESS_SIZE \
- (15000000 - (2 * 20000 * MLXSW_PORT_MAX_PORTS))
+static int mlxsw_sp_sb_ports_init(struct mlxsw_sp *mlxsw_sp)
+{
+ unsigned int max_ports = mlxsw_core_max_ports(mlxsw_sp->core);
+
+ mlxsw_sp->sb.ports = kcalloc(max_ports, sizeof(struct mlxsw_sp_sb_port),
+ GFP_KERNEL);
+ if (!mlxsw_sp->sb.ports)
+ return -ENOMEM;
+ return 0;
+}
+
+static void mlxsw_sp_sb_ports_fini(struct mlxsw_sp *mlxsw_sp)
+{
+ kfree(mlxsw_sp->sb.ports);
+}
+
+#define MLXSW_SP_SB_PR_INGRESS_SIZE 12440000
#define MLXSW_SP_SB_PR_INGRESS_MNG_SIZE (200 * 1000)
-#define MLXSW_SP_SB_PR_EGRESS_SIZE \
- (14000000 - (8 * 1500 * MLXSW_PORT_MAX_PORTS))
+#define MLXSW_SP_SB_PR_EGRESS_SIZE 13232000
#define MLXSW_SP_SB_PR(_mode, _size) \
{ \
@@ -223,18 +239,17 @@
static const struct mlxsw_sp_sb_pr mlxsw_sp_sb_prs_ingress[] = {
MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC,
- MLXSW_SP_BYTES_TO_CELLS(MLXSW_SP_SB_PR_INGRESS_SIZE)),
+ MLXSW_SP_SB_PR_INGRESS_SIZE),
MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, 0),
MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, 0),
MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC,
- MLXSW_SP_BYTES_TO_CELLS(MLXSW_SP_SB_PR_INGRESS_MNG_SIZE)),
+ MLXSW_SP_SB_PR_INGRESS_MNG_SIZE),
};
#define MLXSW_SP_SB_PRS_INGRESS_LEN ARRAY_SIZE(mlxsw_sp_sb_prs_ingress)
static const struct mlxsw_sp_sb_pr mlxsw_sp_sb_prs_egress[] = {
- MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC,
- MLXSW_SP_BYTES_TO_CELLS(MLXSW_SP_SB_PR_EGRESS_SIZE)),
+ MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, MLXSW_SP_SB_PR_EGRESS_SIZE),
MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, 0),
MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, 0),
MLXSW_SP_SB_PR(MLXSW_REG_SBPR_MODE_DYNAMIC, 0),
@@ -251,11 +266,9 @@
int err;
for (i = 0; i < prs_len; i++) {
- const struct mlxsw_sp_sb_pr *pr;
+ u32 size = mlxsw_sp_bytes_cells(mlxsw_sp, prs[i].size);
- pr = &prs[i];
- err = mlxsw_sp_sb_pr_write(mlxsw_sp, i, dir,
- pr->mode, pr->size);
+ err = mlxsw_sp_sb_pr_write(mlxsw_sp, i, dir, prs[i].mode, size);
if (err)
return err;
}
@@ -284,7 +297,7 @@
}
static const struct mlxsw_sp_sb_cm mlxsw_sp_sb_cms_ingress[] = {
- MLXSW_SP_SB_CM(MLXSW_SP_BYTES_TO_CELLS(10000), 8, 0),
+ MLXSW_SP_SB_CM(10000, 8, 0),
MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0),
MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0),
MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0),
@@ -293,20 +306,20 @@
MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0),
MLXSW_SP_SB_CM(0, MLXSW_REG_SBXX_DYN_MAX_BUFF_MIN, 0),
MLXSW_SP_SB_CM(0, 0, 0), /* dummy, this PG does not exist */
- MLXSW_SP_SB_CM(MLXSW_SP_BYTES_TO_CELLS(20000), 1, 3),
+ MLXSW_SP_SB_CM(20000, 1, 3),
};
#define MLXSW_SP_SB_CMS_INGRESS_LEN ARRAY_SIZE(mlxsw_sp_sb_cms_ingress)
static const struct mlxsw_sp_sb_cm mlxsw_sp_sb_cms_egress[] = {
- MLXSW_SP_SB_CM(MLXSW_SP_BYTES_TO_CELLS(1500), 9, 0),
- MLXSW_SP_SB_CM(MLXSW_SP_BYTES_TO_CELLS(1500), 9, 0),
- MLXSW_SP_SB_CM(MLXSW_SP_BYTES_TO_CELLS(1500), 9, 0),
- MLXSW_SP_SB_CM(MLXSW_SP_BYTES_TO_CELLS(1500), 9, 0),
- MLXSW_SP_SB_CM(MLXSW_SP_BYTES_TO_CELLS(1500), 9, 0),
- MLXSW_SP_SB_CM(MLXSW_SP_BYTES_TO_CELLS(1500), 9, 0),
- MLXSW_SP_SB_CM(MLXSW_SP_BYTES_TO_CELLS(1500), 9, 0),
- MLXSW_SP_SB_CM(MLXSW_SP_BYTES_TO_CELLS(1500), 9, 0),
+ MLXSW_SP_SB_CM(1500, 9, 0),
+ MLXSW_SP_SB_CM(1500, 9, 0),
+ MLXSW_SP_SB_CM(1500, 9, 0),
+ MLXSW_SP_SB_CM(1500, 9, 0),
+ MLXSW_SP_SB_CM(1500, 9, 0),
+ MLXSW_SP_SB_CM(1500, 9, 0),
+ MLXSW_SP_SB_CM(1500, 9, 0),
+ MLXSW_SP_SB_CM(1500, 9, 0),
MLXSW_SP_SB_CM(0, 0, 0),
MLXSW_SP_SB_CM(0, 0, 0),
MLXSW_SP_SB_CM(0, 0, 0),
@@ -330,7 +343,7 @@
MLXSW_SP_CPU_PORT_SB_CM,
MLXSW_SP_CPU_PORT_SB_CM,
MLXSW_SP_CPU_PORT_SB_CM,
- MLXSW_SP_SB_CM(MLXSW_SP_BYTES_TO_CELLS(10000), 0, 0),
+ MLXSW_SP_SB_CM(10000, 0, 0),
MLXSW_SP_CPU_PORT_SB_CM,
MLXSW_SP_CPU_PORT_SB_CM,
MLXSW_SP_CPU_PORT_SB_CM,
@@ -370,13 +383,17 @@
for (i = 0; i < cms_len; i++) {
const struct mlxsw_sp_sb_cm *cm;
+ u32 min_buff;
if (i == 8 && dir == MLXSW_REG_SBXX_DIR_INGRESS)
continue; /* PG number 8 does not exist, skip it */
cm = &cms[i];
+ /* All pools are initialized using dynamic thresholds,
+ * therefore 'max_buff' isn't specified in cells.
+ */
+ min_buff = mlxsw_sp_bytes_cells(mlxsw_sp, cm->min_buff);
err = mlxsw_sp_sb_cm_write(mlxsw_sp, local_port, i, dir,
- cm->min_buff, cm->max_buff,
- cm->pool);
+ min_buff, cm->max_buff, cm->pool);
if (err)
return err;
}
@@ -484,21 +501,21 @@
}
static const struct mlxsw_sp_sb_mm mlxsw_sp_sb_mms[] = {
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
- MLXSW_SP_SB_MM(MLXSW_SP_BYTES_TO_CELLS(20000), 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
+ MLXSW_SP_SB_MM(20000, 0xff, 0),
};
#define MLXSW_SP_SB_MMS_LEN ARRAY_SIZE(mlxsw_sp_sb_mms)
@@ -511,10 +528,15 @@
for (i = 0; i < MLXSW_SP_SB_MMS_LEN; i++) {
const struct mlxsw_sp_sb_mm *mc;
+ u32 min_buff;
mc = &mlxsw_sp_sb_mms[i];
- mlxsw_reg_sbmm_pack(sbmm_pl, i, mc->min_buff,
- mc->max_buff, mc->pool);
+ /* All pools are initialized using dynamic thresholds,
+ * therefore 'max_buff' isn't specified in cells.
+ */
+ min_buff = mlxsw_sp_bytes_cells(mlxsw_sp, mc->min_buff);
+ mlxsw_reg_sbmm_pack(sbmm_pl, i, min_buff, mc->max_buff,
+ mc->pool);
err = mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(sbmm), sbmm_pl);
if (err)
return err;
@@ -522,32 +544,53 @@
return 0;
}
-#define MLXSW_SP_SB_SIZE (16 * 1024 * 1024)
-
int mlxsw_sp_buffers_init(struct mlxsw_sp *mlxsw_sp)
{
+ u64 sb_size;
int err;
+ if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, CELL_SIZE))
+ return -EIO;
+ mlxsw_sp->sb.cell_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, CELL_SIZE);
+
+ if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, MAX_BUFFER_SIZE))
+ return -EIO;
+ sb_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_BUFFER_SIZE);
+
+ err = mlxsw_sp_sb_ports_init(mlxsw_sp);
+ if (err)
+ return err;
err = mlxsw_sp_sb_prs_init(mlxsw_sp);
if (err)
- return err;
+ goto err_sb_prs_init;
err = mlxsw_sp_cpu_port_sb_cms_init(mlxsw_sp);
if (err)
- return err;
+ goto err_sb_cpu_port_sb_cms_init;
err = mlxsw_sp_sb_mms_init(mlxsw_sp);
if (err)
- return err;
- return devlink_sb_register(priv_to_devlink(mlxsw_sp->core), 0,
- MLXSW_SP_SB_SIZE,
- MLXSW_SP_SB_POOL_COUNT,
- MLXSW_SP_SB_POOL_COUNT,
- MLXSW_SP_SB_TC_COUNT,
- MLXSW_SP_SB_TC_COUNT);
+ goto err_sb_mms_init;
+ err = devlink_sb_register(priv_to_devlink(mlxsw_sp->core), 0, sb_size,
+ MLXSW_SP_SB_POOL_COUNT,
+ MLXSW_SP_SB_POOL_COUNT,
+ MLXSW_SP_SB_TC_COUNT,
+ MLXSW_SP_SB_TC_COUNT);
+ if (err)
+ goto err_devlink_sb_register;
+
+ return 0;
+
+err_devlink_sb_register:
+err_sb_mms_init:
+err_sb_cpu_port_sb_cms_init:
+err_sb_prs_init:
+ mlxsw_sp_sb_ports_fini(mlxsw_sp);
+ return err;
}
void mlxsw_sp_buffers_fini(struct mlxsw_sp *mlxsw_sp)
{
devlink_sb_unregister(priv_to_devlink(mlxsw_sp->core), 0);
+ mlxsw_sp_sb_ports_fini(mlxsw_sp);
}
int mlxsw_sp_port_buffers_init(struct mlxsw_sp_port *mlxsw_sp_port)
@@ -596,7 +639,7 @@
struct mlxsw_sp_sb_pr *pr = mlxsw_sp_sb_pr_get(mlxsw_sp, pool, dir);
pool_info->pool_type = (enum devlink_sb_pool_type) dir;
- pool_info->size = MLXSW_SP_CELLS_TO_BYTES(pr->size);
+ pool_info->size = mlxsw_sp_cells_bytes(mlxsw_sp, pr->size);
pool_info->threshold_type = (enum devlink_sb_threshold_type) pr->mode;
return 0;
}
@@ -606,9 +649,9 @@
enum devlink_sb_threshold_type threshold_type)
{
struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core);
+ u32 pool_size = mlxsw_sp_bytes_cells(mlxsw_sp, size);
u8 pool = pool_get(pool_index);
enum mlxsw_reg_sbxx_dir dir = dir_get(pool_index);
- u32 pool_size = MLXSW_SP_BYTES_TO_CELLS(size);
enum mlxsw_reg_sbpr_mode mode;
if (size > MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_BUFFER_SIZE))
@@ -627,7 +670,7 @@
if (pr->mode == MLXSW_REG_SBPR_MODE_DYNAMIC)
return max_buff - MLXSW_SP_SB_THRESHOLD_TO_ALPHA_OFFSET;
- return MLXSW_SP_CELLS_TO_BYTES(max_buff);
+ return mlxsw_sp_cells_bytes(mlxsw_sp, max_buff);
}
static int mlxsw_sp_sb_threshold_in(struct mlxsw_sp *mlxsw_sp, u8 pool,
@@ -645,7 +688,7 @@
return -EINVAL;
*p_max_buff = val;
} else {
- *p_max_buff = MLXSW_SP_BYTES_TO_CELLS(threshold);
+ *p_max_buff = mlxsw_sp_bytes_cells(mlxsw_sp, threshold);
}
return 0;
}
@@ -761,7 +804,7 @@
masked_count = 0;
for (local_port = cb_ctx.local_port_1;
- local_port < MLXSW_PORT_MAX_PORTS; local_port++) {
+ local_port < mlxsw_core_max_ports(mlxsw_core); local_port++) {
if (!mlxsw_sp->ports[local_port])
continue;
for (i = 0; i < MLXSW_SP_SB_TC_COUNT; i++) {
@@ -775,7 +818,7 @@
}
masked_count = 0;
for (local_port = cb_ctx.local_port_1;
- local_port < MLXSW_PORT_MAX_PORTS; local_port++) {
+ local_port < mlxsw_core_max_ports(mlxsw_core); local_port++) {
if (!mlxsw_sp->ports[local_port])
continue;
for (i = 0; i < MLXSW_SP_SB_TC_COUNT; i++) {
@@ -817,7 +860,7 @@
mlxsw_reg_sbsr_pg_buff_mask_set(sbsr_pl, i, 1);
mlxsw_reg_sbsr_tclass_mask_set(sbsr_pl, i, 1);
}
- for (; local_port < MLXSW_PORT_MAX_PORTS; local_port++) {
+ for (; local_port < mlxsw_core_max_ports(mlxsw_core); local_port++) {
if (!mlxsw_sp->ports[local_port])
continue;
mlxsw_reg_sbsr_ingress_port_mask_set(sbsr_pl, local_port, 1);
@@ -847,7 +890,7 @@
cb_priv);
if (err)
goto out;
- if (local_port < MLXSW_PORT_MAX_PORTS)
+ if (local_port < mlxsw_core_max_ports(mlxsw_core))
goto next_batch;
out:
@@ -882,7 +925,7 @@
mlxsw_reg_sbsr_pg_buff_mask_set(sbsr_pl, i, 1);
mlxsw_reg_sbsr_tclass_mask_set(sbsr_pl, i, 1);
}
- for (; local_port < MLXSW_PORT_MAX_PORTS; local_port++) {
+ for (; local_port < mlxsw_core_max_ports(mlxsw_core); local_port++) {
if (!mlxsw_sp->ports[local_port])
continue;
mlxsw_reg_sbsr_ingress_port_mask_set(sbsr_pl, local_port, 1);
@@ -908,7 +951,7 @@
&bulk_list, NULL, 0);
if (err)
goto out;
- if (local_port < MLXSW_PORT_MAX_PORTS)
+ if (local_port < mlxsw_core_max_ports(mlxsw_core))
goto next_batch;
out:
@@ -932,8 +975,8 @@
struct mlxsw_sp_sb_pm *pm = mlxsw_sp_sb_pm_get(mlxsw_sp, local_port,
pool, dir);
- *p_cur = MLXSW_SP_CELLS_TO_BYTES(pm->occ.cur);
- *p_max = MLXSW_SP_CELLS_TO_BYTES(pm->occ.max);
+ *p_cur = mlxsw_sp_cells_bytes(mlxsw_sp, pm->occ.cur);
+ *p_max = mlxsw_sp_cells_bytes(mlxsw_sp, pm->occ.max);
return 0;
}
@@ -951,7 +994,7 @@
struct mlxsw_sp_sb_cm *cm = mlxsw_sp_sb_cm_get(mlxsw_sp, local_port,
pg_buff, dir);
- *p_cur = MLXSW_SP_CELLS_TO_BYTES(cm->occ.cur);
- *p_max = MLXSW_SP_CELLS_TO_BYTES(cm->occ.max);
+ *p_cur = mlxsw_sp_cells_bytes(mlxsw_sp, cm->occ.cur);
+ *p_max = mlxsw_sp_cells_bytes(mlxsw_sp, cm->occ.max);
return 0;
}
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
new file mode 100644
index 0000000..0f46775
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
@@ -0,0 +1,207 @@
+/*
+ * drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.c
+ * Copyright (c) 2017 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2017 Arkadi Sharshevsky <arkadis@mellanox.com>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ * 3. Neither the names of the copyright holders nor the names of its
+ * contributors may be used to endorse or promote products derived from
+ * this software without specific prior written permission.
+ *
+ * Alternatively, this software may be distributed under the terms of the
+ * GNU General Public License ("GPL") version 2 as published by the Free
+ * Software Foundation.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <linux/kernel.h>
+#include <linux/bitops.h>
+
+#include "spectrum_cnt.h"
+
+#define MLXSW_SP_COUNTER_POOL_BANK_SIZE 4096
+
+struct mlxsw_sp_counter_sub_pool {
+ unsigned int base_index;
+ unsigned int size;
+ unsigned int entry_size;
+ unsigned int bank_count;
+};
+
+struct mlxsw_sp_counter_pool {
+ unsigned int pool_size;
+ unsigned long *usage; /* Usage bitmap */
+ struct mlxsw_sp_counter_sub_pool *sub_pools;
+};
+
+static struct mlxsw_sp_counter_sub_pool mlxsw_sp_counter_sub_pools[] = {
+ [MLXSW_SP_COUNTER_SUB_POOL_FLOW] = {
+ .bank_count = 6,
+ },
+ [MLXSW_SP_COUNTER_SUB_POOL_RIF] = {
+ .bank_count = 2,
+ }
+};
+
+static int mlxsw_sp_counter_pool_validate(struct mlxsw_sp *mlxsw_sp)
+{
+ unsigned int total_bank_config = 0;
+ unsigned int pool_size;
+ int i;
+
+ pool_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, COUNTER_POOL_SIZE);
+ /* Check config is valid, no bank over subscription */
+ for (i = 0; i < ARRAY_SIZE(mlxsw_sp_counter_sub_pools); i++)
+ total_bank_config += mlxsw_sp_counter_sub_pools[i].bank_count;
+ if (total_bank_config > pool_size / MLXSW_SP_COUNTER_POOL_BANK_SIZE + 1)
+ return -EINVAL;
+ return 0;
+}
+
+static int mlxsw_sp_counter_sub_pools_prepare(struct mlxsw_sp *mlxsw_sp)
+{
+ struct mlxsw_sp_counter_sub_pool *sub_pool;
+
+ /* Prepare generic flow pool*/
+ sub_pool = &mlxsw_sp_counter_sub_pools[MLXSW_SP_COUNTER_SUB_POOL_FLOW];
+ if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, COUNTER_SIZE_PACKETS_BYTES))
+ return -EIO;
+ sub_pool->entry_size = MLXSW_CORE_RES_GET(mlxsw_sp->core,
+ COUNTER_SIZE_PACKETS_BYTES);
+ /* Prepare erif pool*/
+ sub_pool = &mlxsw_sp_counter_sub_pools[MLXSW_SP_COUNTER_SUB_POOL_RIF];
+ if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, COUNTER_SIZE_ROUTER_BASIC))
+ return -EIO;
+ sub_pool->entry_size = MLXSW_CORE_RES_GET(mlxsw_sp->core,
+ COUNTER_SIZE_ROUTER_BASIC);
+ return 0;
+}
+
+int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp)
+{
+ struct mlxsw_sp_counter_sub_pool *sub_pool;
+ struct mlxsw_sp_counter_pool *pool;
+ unsigned int base_index;
+ unsigned int map_size;
+ int i;
+ int err;
+
+ if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, COUNTER_POOL_SIZE))
+ return -EIO;
+
+ err = mlxsw_sp_counter_pool_validate(mlxsw_sp);
+ if (err)
+ return err;
+
+ err = mlxsw_sp_counter_sub_pools_prepare(mlxsw_sp);
+ if (err)
+ return err;
+
+ pool = kzalloc(sizeof(*pool), GFP_KERNEL);
+ if (!pool)
+ return -ENOMEM;
+
+ pool->pool_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, COUNTER_POOL_SIZE);
+ map_size = BITS_TO_LONGS(pool->pool_size) * sizeof(unsigned long);
+
+ pool->usage = kzalloc(map_size, GFP_KERNEL);
+ if (!pool->usage) {
+ err = -ENOMEM;
+ goto err_usage_alloc;
+ }
+
+ pool->sub_pools = mlxsw_sp_counter_sub_pools;
+ /* Allocation is based on bank count which should be
+ * specified for each sub pool statically.
+ */
+ base_index = 0;
+ for (i = 0; i < ARRAY_SIZE(mlxsw_sp_counter_sub_pools); i++) {
+ sub_pool = &pool->sub_pools[i];
+ sub_pool->size = sub_pool->bank_count *
+ MLXSW_SP_COUNTER_POOL_BANK_SIZE;
+ sub_pool->base_index = base_index;
+ base_index += sub_pool->size;
+ /* The last bank can't be fully used */
+ if (sub_pool->base_index + sub_pool->size > pool->pool_size)
+ sub_pool->size = pool->pool_size - sub_pool->base_index;
+ }
+
+ mlxsw_sp->counter_pool = pool;
+ return 0;
+
+err_usage_alloc:
+ kfree(pool);
+ return err;
+}
+
+void mlxsw_sp_counter_pool_fini(struct mlxsw_sp *mlxsw_sp)
+{
+ struct mlxsw_sp_counter_pool *pool = mlxsw_sp->counter_pool;
+
+ WARN_ON(find_first_bit(pool->usage, pool->pool_size) !=
+ pool->pool_size);
+ kfree(pool->usage);
+ kfree(pool);
+}
+
+int mlxsw_sp_counter_alloc(struct mlxsw_sp *mlxsw_sp,
+ enum mlxsw_sp_counter_sub_pool_id sub_pool_id,
+ unsigned int *p_counter_index)
+{
+ struct mlxsw_sp_counter_pool *pool = mlxsw_sp->counter_pool;
+ struct mlxsw_sp_counter_sub_pool *sub_pool;
+ unsigned int entry_index;
+ unsigned int stop_index;
+ int i;
+
+ sub_pool = &mlxsw_sp_counter_sub_pools[sub_pool_id];
+ stop_index = sub_pool->base_index + sub_pool->size;
+ entry_index = sub_pool->base_index;
+
+ entry_index = find_next_zero_bit(pool->usage, stop_index, entry_index);
+ if (entry_index == stop_index)
+ return -ENOBUFS;
+ /* The sub-pools can contain non-integer number of entries
+ * so we must check for overflow
+ */
+ if (entry_index + sub_pool->entry_size > stop_index)
+ return -ENOBUFS;
+ for (i = 0; i < sub_pool->entry_size; i++)
+ __set_bit(entry_index + i, pool->usage);
+
+ *p_counter_index = entry_index;
+ return 0;
+}
+
+void mlxsw_sp_counter_free(struct mlxsw_sp *mlxsw_sp,
+ enum mlxsw_sp_counter_sub_pool_id sub_pool_id,
+ unsigned int counter_index)
+{
+ struct mlxsw_sp_counter_pool *pool = mlxsw_sp->counter_pool;
+ struct mlxsw_sp_counter_sub_pool *sub_pool;
+ int i;
+
+ if (WARN_ON(counter_index >= pool->pool_size))
+ return;
+ sub_pool = &mlxsw_sp_counter_sub_pools[sub_pool_id];
+ for (i = 0; i < sub_pool->entry_size; i++)
+ __clear_bit(counter_index + i, pool->usage);
+}
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.h
new file mode 100644
index 0000000..fd34d0a
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.h
@@ -0,0 +1,54 @@
+/*
+ * drivers/net/ethernet/mellanox/mlxsw/spectrum_cnt.h
+ * Copyright (c) 2017 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2017 Arkadi Sharshevsky <arkdis@mellanox.com>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ * 3. Neither the names of the copyright holders nor the names of its
+ * contributors may be used to endorse or promote products derived from
+ * this software without specific prior written permission.
+ *
+ * Alternatively, this software may be distributed under the terms of the
+ * GNU General Public License ("GPL") version 2 as published by the Free
+ * Software Foundation.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _MLXSW_SPECTRUM_CNT_H
+#define _MLXSW_SPECTRUM_CNT_H
+
+#include "spectrum.h"
+
+enum mlxsw_sp_counter_sub_pool_id {
+ MLXSW_SP_COUNTER_SUB_POOL_FLOW,
+ MLXSW_SP_COUNTER_SUB_POOL_RIF,
+};
+
+int mlxsw_sp_counter_alloc(struct mlxsw_sp *mlxsw_sp,
+ enum mlxsw_sp_counter_sub_pool_id sub_pool_id,
+ unsigned int *p_counter_index);
+void mlxsw_sp_counter_free(struct mlxsw_sp *mlxsw_sp,
+ enum mlxsw_sp_counter_sub_pool_id sub_pool_id,
+ unsigned int counter_index);
+int mlxsw_sp_counter_pool_init(struct mlxsw_sp *mlxsw_sp);
+void mlxsw_sp_counter_pool_fini(struct mlxsw_sp *mlxsw_sp);
+
+#endif
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_dpipe.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_dpipe.c
new file mode 100644
index 0000000..ea56f6a
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_dpipe.c
@@ -0,0 +1,351 @@
+/*
+ * drivers/net/ethernet/mellanox/mlxsw/spectrum_dpipe.c
+ * Copyright (c) 2017 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2017 Arkadi Sharshevsky <arakdis@mellanox.com>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ * 3. Neither the names of the copyright holders nor the names of its
+ * contributors may be used to endorse or promote products derived from
+ * this software without specific prior written permission.
+ *
+ * Alternatively, this software may be distributed under the terms of the
+ * GNU General Public License ("GPL") version 2 as published by the Free
+ * Software Foundation.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <linux/kernel.h>
+#include <net/devlink.h>
+
+#include "spectrum.h"
+#include "spectrum_dpipe.h"
+#include "spectrum_router.h"
+
+enum mlxsw_sp_field_metadata_id {
+ MLXSW_SP_DPIPE_FIELD_METADATA_ERIF_PORT,
+ MLXSW_SP_DPIPE_FIELD_METADATA_L3_FORWARD,
+ MLXSW_SP_DPIPE_FIELD_METADATA_L3_DROP,
+};
+
+static struct devlink_dpipe_field mlxsw_sp_dpipe_fields_metadata[] = {
+ { .name = "erif_port",
+ .id = MLXSW_SP_DPIPE_FIELD_METADATA_ERIF_PORT,
+ .bitwidth = 32,
+ .mapping_type = DEVLINK_DPIPE_FIELD_MAPPING_TYPE_IFINDEX,
+ },
+ { .name = "l3_forward",
+ .id = MLXSW_SP_DPIPE_FIELD_METADATA_L3_FORWARD,
+ .bitwidth = 1,
+ },
+ { .name = "l3_drop",
+ .id = MLXSW_SP_DPIPE_FIELD_METADATA_L3_DROP,
+ .bitwidth = 1,
+ },
+};
+
+enum mlxsw_sp_dpipe_header_id {
+ MLXSW_SP_DPIPE_HEADER_METADATA,
+};
+
+static struct devlink_dpipe_header mlxsw_sp_dpipe_header_metadata = {
+ .name = "mlxsw_meta",
+ .id = MLXSW_SP_DPIPE_HEADER_METADATA,
+ .fields = mlxsw_sp_dpipe_fields_metadata,
+ .fields_count = ARRAY_SIZE(mlxsw_sp_dpipe_fields_metadata),
+};
+
+static struct devlink_dpipe_header *mlxsw_dpipe_headers[] = {
+ &mlxsw_sp_dpipe_header_metadata,
+};
+
+static struct devlink_dpipe_headers mlxsw_sp_dpipe_headers = {
+ .headers = mlxsw_dpipe_headers,
+ .headers_count = ARRAY_SIZE(mlxsw_dpipe_headers),
+};
+
+static int mlxsw_sp_dpipe_table_erif_actions_dump(void *priv,
+ struct sk_buff *skb)
+{
+ struct devlink_dpipe_action action = {0};
+ int err;
+
+ action.type = DEVLINK_DPIPE_ACTION_TYPE_FIELD_MODIFY;
+ action.header = &mlxsw_sp_dpipe_header_metadata;
+ action.field_id = MLXSW_SP_DPIPE_FIELD_METADATA_L3_FORWARD;
+
+ err = devlink_dpipe_action_put(skb, &action);
+ if (err)
+ return err;
+
+ action.type = DEVLINK_DPIPE_ACTION_TYPE_FIELD_MODIFY;
+ action.header = &mlxsw_sp_dpipe_header_metadata;
+ action.field_id = MLXSW_SP_DPIPE_FIELD_METADATA_L3_DROP;
+
+ return devlink_dpipe_action_put(skb, &action);
+}
+
+static int mlxsw_sp_dpipe_table_erif_matches_dump(void *priv,
+ struct sk_buff *skb)
+{
+ struct devlink_dpipe_match match = {0};
+
+ match.type = DEVLINK_DPIPE_MATCH_TYPE_FIELD_EXACT;
+ match.header = &mlxsw_sp_dpipe_header_metadata;
+ match.field_id = MLXSW_SP_DPIPE_FIELD_METADATA_ERIF_PORT;
+
+ return devlink_dpipe_match_put(skb, &match);
+}
+
+static void mlxsw_sp_erif_entry_clear(struct devlink_dpipe_entry *entry)
+{
+ unsigned int value_count, value_index;
+ struct devlink_dpipe_value *value;
+
+ value = entry->action_values;
+ value_count = entry->action_values_count;
+ for (value_index = 0; value_index < value_count; value_index++) {
+ kfree(value[value_index].value);
+ kfree(value[value_index].mask);
+ }
+
+ value = entry->match_values;
+ value_count = entry->match_values_count;
+ for (value_index = 0; value_index < value_count; value_index++) {
+ kfree(value[value_index].value);
+ kfree(value[value_index].mask);
+ }
+}
+
+static void
+mlxsw_sp_erif_match_action_prepare(struct devlink_dpipe_match *match,
+ struct devlink_dpipe_action *action)
+{
+ action->type = DEVLINK_DPIPE_ACTION_TYPE_FIELD_MODIFY;
+ action->header = &mlxsw_sp_dpipe_header_metadata;
+ action->field_id = MLXSW_SP_DPIPE_FIELD_METADATA_L3_FORWARD;
+
+ match->type = DEVLINK_DPIPE_MATCH_TYPE_FIELD_EXACT;
+ match->header = &mlxsw_sp_dpipe_header_metadata;
+ match->field_id = MLXSW_SP_DPIPE_FIELD_METADATA_ERIF_PORT;
+}
+
+static int mlxsw_sp_erif_entry_prepare(struct devlink_dpipe_entry *entry,
+ struct devlink_dpipe_value *match_value,
+ struct devlink_dpipe_match *match,
+ struct devlink_dpipe_value *action_value,
+ struct devlink_dpipe_action *action)
+{
+ entry->match_values = match_value;
+ entry->match_values_count = 1;
+
+ entry->action_values = action_value;
+ entry->action_values_count = 1;
+
+ match_value->match = match;
+ match_value->value_size = sizeof(u32);
+ match_value->value = kmalloc(match_value->value_size, GFP_KERNEL);
+ if (!match_value->value)
+ return -ENOMEM;
+
+ action_value->action = action;
+ action_value->value_size = sizeof(u32);
+ action_value->value = kmalloc(action_value->value_size, GFP_KERNEL);
+ if (!action_value->value)
+ goto err_action_alloc;
+ return 0;
+
+err_action_alloc:
+ kfree(match_value->value);
+ return -ENOMEM;
+}
+
+static int mlxsw_sp_erif_entry_get(struct mlxsw_sp *mlxsw_sp,
+ struct devlink_dpipe_entry *entry,
+ struct mlxsw_sp_rif *rif,
+ bool counters_enabled)
+{
+ u32 *action_value;
+ u32 *rif_value;
+ u64 cnt;
+ int err;
+
+ /* Set Match RIF index */
+ rif_value = entry->match_values->value;
+ *rif_value = mlxsw_sp_rif_index(rif);
+ entry->match_values->mapping_value = mlxsw_sp_rif_dev_ifindex(rif);
+ entry->match_values->mapping_valid = true;
+
+ /* Set Action Forwarding */
+ action_value = entry->action_values->value;
+ *action_value = 1;
+
+ entry->counter_valid = false;
+ entry->counter = 0;
+ if (!counters_enabled)
+ return 0;
+
+ entry->index = mlxsw_sp_rif_index(rif);
+ err = mlxsw_sp_rif_counter_value_get(mlxsw_sp, rif,
+ MLXSW_SP_RIF_COUNTER_EGRESS,
+ &cnt);
+ if (!err) {
+ entry->counter = cnt;
+ entry->counter_valid = true;
+ }
+ return 0;
+}
+
+static int
+mlxsw_sp_table_erif_entries_dump(void *priv, bool counters_enabled,
+ struct devlink_dpipe_dump_ctx *dump_ctx)
+{
+ struct devlink_dpipe_value match_value = {{0}}, action_value = {{0}};
+ struct devlink_dpipe_action action = {0};
+ struct devlink_dpipe_match match = {0};
+ struct devlink_dpipe_entry entry = {0};
+ struct mlxsw_sp *mlxsw_sp = priv;
+ unsigned int rif_count;
+ int i, j;
+ int err;
+
+ mlxsw_sp_erif_match_action_prepare(&match, &action);
+ err = mlxsw_sp_erif_entry_prepare(&entry, &match_value, &match,
+ &action_value, &action);
+ if (err)
+ return err;
+
+ rif_count = MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_RIFS);
+ rtnl_lock();
+ i = 0;
+start_again:
+ err = devlink_dpipe_entry_ctx_prepare(dump_ctx);
+ if (err)
+ return err;
+ j = 0;
+ for (; i < rif_count; i++) {
+ if (!mlxsw_sp->rifs[i])
+ continue;
+ err = mlxsw_sp_erif_entry_get(mlxsw_sp, &entry,
+ mlxsw_sp->rifs[i],
+ counters_enabled);
+ if (err)
+ goto err_entry_get;
+ err = devlink_dpipe_entry_ctx_append(dump_ctx, &entry);
+ if (err) {
+ if (err == -EMSGSIZE) {
+ if (!j)
+ goto err_entry_append;
+ break;
+ }
+ goto err_entry_append;
+ }
+ j++;
+ }
+
+ devlink_dpipe_entry_ctx_close(dump_ctx);
+ if (i != rif_count)
+ goto start_again;
+ rtnl_unlock();
+
+ mlxsw_sp_erif_entry_clear(&entry);
+ return 0;
+err_entry_append:
+err_entry_get:
+ rtnl_unlock();
+ mlxsw_sp_erif_entry_clear(&entry);
+ return err;
+}
+
+static int mlxsw_sp_table_erif_counters_update(void *priv, bool enable)
+{
+ struct mlxsw_sp *mlxsw_sp = priv;
+ int i;
+
+ rtnl_lock();
+ for (i = 0; i < MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_RIFS); i++) {
+ if (!mlxsw_sp->rifs[i])
+ continue;
+ if (enable)
+ mlxsw_sp_rif_counter_alloc(mlxsw_sp,
+ mlxsw_sp->rifs[i],
+ MLXSW_SP_RIF_COUNTER_EGRESS);
+ else
+ mlxsw_sp_rif_counter_free(mlxsw_sp,
+ mlxsw_sp->rifs[i],
+ MLXSW_SP_RIF_COUNTER_EGRESS);
+ }
+ rtnl_unlock();
+ return 0;
+}
+
+static struct devlink_dpipe_table_ops mlxsw_sp_erif_ops = {
+ .matches_dump = mlxsw_sp_dpipe_table_erif_matches_dump,
+ .actions_dump = mlxsw_sp_dpipe_table_erif_actions_dump,
+ .entries_dump = mlxsw_sp_table_erif_entries_dump,
+ .counters_set_update = mlxsw_sp_table_erif_counters_update,
+};
+
+static int mlxsw_sp_dpipe_erif_table_init(struct mlxsw_sp *mlxsw_sp)
+{
+ struct devlink *devlink = priv_to_devlink(mlxsw_sp->core);
+ u64 table_size;
+
+ table_size = MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_RIFS);
+ return devlink_dpipe_table_register(devlink,
+ MLXSW_SP_DPIPE_TABLE_NAME_ERIF,
+ &mlxsw_sp_erif_ops,
+ mlxsw_sp, table_size,
+ false);
+}
+
+static void mlxsw_sp_dpipe_erif_table_fini(struct mlxsw_sp *mlxsw_sp)
+{
+ struct devlink *devlink = priv_to_devlink(mlxsw_sp->core);
+
+ devlink_dpipe_table_unregister(devlink, MLXSW_SP_DPIPE_TABLE_NAME_ERIF);
+}
+
+int mlxsw_sp_dpipe_init(struct mlxsw_sp *mlxsw_sp)
+{
+ struct devlink *devlink = priv_to_devlink(mlxsw_sp->core);
+ int err;
+
+ err = devlink_dpipe_headers_register(devlink,
+ &mlxsw_sp_dpipe_headers);
+ if (err)
+ return err;
+ err = mlxsw_sp_dpipe_erif_table_init(mlxsw_sp);
+ if (err)
+ goto err_erif_register;
+ return 0;
+
+err_erif_register:
+ devlink_dpipe_headers_unregister(priv_to_devlink(mlxsw_sp->core));
+ return err;
+}
+
+void mlxsw_sp_dpipe_fini(struct mlxsw_sp *mlxsw_sp)
+{
+ struct devlink *devlink = priv_to_devlink(mlxsw_sp->core);
+
+ mlxsw_sp_dpipe_erif_table_fini(mlxsw_sp);
+ devlink_dpipe_headers_unregister(devlink);
+}
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_dpipe.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum_dpipe.h
new file mode 100644
index 0000000..d208929
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_dpipe.h
@@ -0,0 +1,43 @@
+/*
+ * drivers/net/ethernet/mellanox/mlxsw/spectrum_dpipe.h
+ * Copyright (c) 2017 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2017 Arkadi Sharshevsky <arkadis@mellanox.com>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ * 3. Neither the names of the copyright holders nor the names of its
+ * contributors may be used to endorse or promote products derived from
+ * this software without specific prior written permission.
+ *
+ * Alternatively, this software may be distributed under the terms of the
+ * GNU General Public License ("GPL") version 2 as published by the Free
+ * Software Foundation.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _MLXSW_PIPELINE_H_
+#define _MLXSW_PIPELINE_H_
+
+int mlxsw_sp_dpipe_init(struct mlxsw_sp *mlxsw_sp);
+void mlxsw_sp_dpipe_fini(struct mlxsw_sp *mlxsw_sp);
+
+#define MLXSW_SP_DPIPE_TABLE_NAME_ERIF "mlxsw_erif"
+
+#endif /* _MLXSW_PIPELINE_H_*/
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c
index ae6cccc..7d87e23 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c
@@ -39,6 +39,7 @@
#include <net/pkt_cls.h>
#include <net/tc_act/tc_gact.h>
#include <net/tc_act/tc_mirred.h>
+#include <net/tc_act/tc_vlan.h>
#include "spectrum.h"
#include "core_acl_flex_keys.h"
@@ -55,6 +56,11 @@
if (tc_no_actions(exts))
return 0;
+ /* Count action is inserted first */
+ err = mlxsw_sp_acl_rulei_act_count(mlxsw_sp, rulei);
+ if (err)
+ return err;
+
tcf_exts_to_list(exts, &actions);
list_for_each_entry(a, &actions, list) {
if (is_tcf_gact_shot(a)) {
@@ -65,6 +71,11 @@
int ifindex = tcf_mirred_ifindex(a);
struct net_device *out_dev;
+ err = mlxsw_sp_acl_rulei_act_fid_set(mlxsw_sp, rulei,
+ MLXSW_SP_DUMMY_FID);
+ if (err)
+ return err;
+
out_dev = __dev_get_by_index(dev_net(dev), ifindex);
if (out_dev == dev)
out_dev = NULL;
@@ -73,6 +84,15 @@
out_dev);
if (err)
return err;
+ } else if (is_tcf_vlan(a)) {
+ u16 proto = be16_to_cpu(tcf_vlan_push_proto(a));
+ u32 action = tcf_vlan_action(a);
+ u8 prio = tcf_vlan_push_prio(a);
+ u16 vid = tcf_vlan_push_vid(a);
+
+ return mlxsw_sp_acl_rulei_act_vlan(mlxsw_sp, rulei,
+ action, vid,
+ proto, prio);
} else {
dev_err(mlxsw_sp->bus_info->dev, "Unsupported action\n");
return -EOPNOTSUPP;
@@ -173,7 +193,8 @@
BIT(FLOW_DISSECTOR_KEY_ETH_ADDRS) |
BIT(FLOW_DISSECTOR_KEY_IPV4_ADDRS) |
BIT(FLOW_DISSECTOR_KEY_IPV6_ADDRS) |
- BIT(FLOW_DISSECTOR_KEY_PORTS))) {
+ BIT(FLOW_DISSECTOR_KEY_PORTS) |
+ BIT(FLOW_DISSECTOR_KEY_VLAN))) {
dev_err(mlxsw_sp->bus_info->dev, "Unsupported key\n");
return -EOPNOTSUPP;
}
@@ -234,6 +255,27 @@
sizeof(key->src));
}
+ if (dissector_uses_key(f->dissector, FLOW_DISSECTOR_KEY_VLAN)) {
+ struct flow_dissector_key_vlan *key =
+ skb_flow_dissector_target(f->dissector,
+ FLOW_DISSECTOR_KEY_VLAN,
+ f->key);
+ struct flow_dissector_key_vlan *mask =
+ skb_flow_dissector_target(f->dissector,
+ FLOW_DISSECTOR_KEY_VLAN,
+ f->mask);
+ if (mask->vlan_id != 0)
+ mlxsw_sp_acl_rulei_keymask_u32(rulei,
+ MLXSW_AFK_ELEMENT_VID,
+ key->vlan_id,
+ mask->vlan_id);
+ if (mask->vlan_priority != 0)
+ mlxsw_sp_acl_rulei_keymask_u32(rulei,
+ MLXSW_AFK_ELEMENT_PCP,
+ key->vlan_priority,
+ mask->vlan_priority);
+ }
+
if (addr_type == FLOW_DISSECTOR_KEY_IPV4_ADDRS)
mlxsw_sp_flower_parse_ipv4(rulei, f);
@@ -314,3 +356,47 @@
mlxsw_sp_acl_ruleset_put(mlxsw_sp, ruleset);
}
+
+int mlxsw_sp_flower_stats(struct mlxsw_sp_port *mlxsw_sp_port, bool ingress,
+ struct tc_cls_flower_offload *f)
+{
+ struct mlxsw_sp *mlxsw_sp = mlxsw_sp_port->mlxsw_sp;
+ struct mlxsw_sp_acl_ruleset *ruleset;
+ struct mlxsw_sp_acl_rule *rule;
+ struct tc_action *a;
+ LIST_HEAD(actions);
+ u64 packets;
+ u64 lastuse;
+ u64 bytes;
+ int err;
+
+ ruleset = mlxsw_sp_acl_ruleset_get(mlxsw_sp, mlxsw_sp_port->dev,
+ ingress,
+ MLXSW_SP_ACL_PROFILE_FLOWER);
+ if (WARN_ON(IS_ERR(ruleset)))
+ return -EINVAL;
+
+ rule = mlxsw_sp_acl_rule_lookup(mlxsw_sp, ruleset, f->cookie);
+ if (!rule)
+ return -EINVAL;
+
+ err = mlxsw_sp_acl_rule_get_stats(mlxsw_sp, rule, &packets, &bytes,
+ &lastuse);
+ if (err)
+ goto err_rule_get_stats;
+
+ preempt_disable();
+
+ tcf_exts_to_list(f->exts, &actions);
+ list_for_each_entry(a, &actions, list)
+ tcf_action_stats_update(a, bytes, packets, lastuse);
+
+ preempt_enable();
+
+ mlxsw_sp_acl_ruleset_put(mlxsw_sp, ruleset);
+ return 0;
+
+err_rule_get_stats:
+ mlxsw_sp_acl_ruleset_put(mlxsw_sp, ruleset);
+ return err;
+}
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
index ac321e8..26c26cd 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_kvdl.c
@@ -45,7 +45,8 @@
(MLXSW_SP_KVD_LINEAR_SIZE - MLXSW_SP_KVDL_CHUNKS_BASE)
#define MLXSW_SP_CHUNK_MAX 32
-int mlxsw_sp_kvdl_alloc(struct mlxsw_sp *mlxsw_sp, unsigned int entry_count)
+int mlxsw_sp_kvdl_alloc(struct mlxsw_sp *mlxsw_sp, unsigned int entry_count,
+ u32 *p_entry_index)
{
int entry_index;
int size;
@@ -72,7 +73,8 @@
for (i = 0; i < type_entries; i++)
set_bit(entry_index + i, mlxsw_sp->kvdl.usage);
- return entry_index;
+ *p_entry_index = entry_index;
+ return 0;
}
return -ENOBUFS;
}
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c
index bd8de6b..33cec1c 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.c
@@ -41,14 +41,184 @@
#include <linux/in6.h>
#include <linux/notifier.h>
#include <linux/inetdevice.h>
+#include <linux/netdevice.h>
#include <net/netevent.h>
#include <net/neighbour.h>
#include <net/arp.h>
#include <net/ip_fib.h>
+#include <net/fib_rules.h>
+#include <net/l3mdev.h>
#include "spectrum.h"
#include "core.h"
#include "reg.h"
+#include "spectrum_cnt.h"
+#include "spectrum_dpipe.h"
+#include "spectrum_router.h"
+
+struct mlxsw_sp_rif {
+ struct list_head nexthop_list;
+ struct list_head neigh_list;
+ struct net_device *dev;
+ struct mlxsw_sp_fid *f;
+ unsigned char addr[ETH_ALEN];
+ int mtu;
+ u16 rif_index;
+ u16 vr_id;
+ unsigned int counter_ingress;
+ bool counter_ingress_valid;
+ unsigned int counter_egress;
+ bool counter_egress_valid;
+};
+
+static unsigned int *
+mlxsw_sp_rif_p_counter_get(struct mlxsw_sp_rif *rif,
+ enum mlxsw_sp_rif_counter_dir dir)
+{
+ switch (dir) {
+ case MLXSW_SP_RIF_COUNTER_EGRESS:
+ return &rif->counter_egress;
+ case MLXSW_SP_RIF_COUNTER_INGRESS:
+ return &rif->counter_ingress;
+ }
+ return NULL;
+}
+
+static bool
+mlxsw_sp_rif_counter_valid_get(struct mlxsw_sp_rif *rif,
+ enum mlxsw_sp_rif_counter_dir dir)
+{
+ switch (dir) {
+ case MLXSW_SP_RIF_COUNTER_EGRESS:
+ return rif->counter_egress_valid;
+ case MLXSW_SP_RIF_COUNTER_INGRESS:
+ return rif->counter_ingress_valid;
+ }
+ return false;
+}
+
+static void
+mlxsw_sp_rif_counter_valid_set(struct mlxsw_sp_rif *rif,
+ enum mlxsw_sp_rif_counter_dir dir,
+ bool valid)
+{
+ switch (dir) {
+ case MLXSW_SP_RIF_COUNTER_EGRESS:
+ rif->counter_egress_valid = valid;
+ break;
+ case MLXSW_SP_RIF_COUNTER_INGRESS:
+ rif->counter_ingress_valid = valid;
+ break;
+ }
+}
+
+static int mlxsw_sp_rif_counter_edit(struct mlxsw_sp *mlxsw_sp, u16 rif_index,
+ unsigned int counter_index, bool enable,
+ enum mlxsw_sp_rif_counter_dir dir)
+{
+ char ritr_pl[MLXSW_REG_RITR_LEN];
+ bool is_egress = false;
+ int err;
+
+ if (dir == MLXSW_SP_RIF_COUNTER_EGRESS)
+ is_egress = true;
+ mlxsw_reg_ritr_rif_pack(ritr_pl, rif_index);
+ err = mlxsw_reg_query(mlxsw_sp->core, MLXSW_REG(ritr), ritr_pl);
+ if (err)
+ return err;
+
+ mlxsw_reg_ritr_counter_pack(ritr_pl, counter_index, enable,
+ is_egress);
+ return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ritr), ritr_pl);
+}
+
+int mlxsw_sp_rif_counter_value_get(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_rif *rif,
+ enum mlxsw_sp_rif_counter_dir dir, u64 *cnt)
+{
+ char ricnt_pl[MLXSW_REG_RICNT_LEN];
+ unsigned int *p_counter_index;
+ bool valid;
+ int err;
+
+ valid = mlxsw_sp_rif_counter_valid_get(rif, dir);
+ if (!valid)
+ return -EINVAL;
+
+ p_counter_index = mlxsw_sp_rif_p_counter_get(rif, dir);
+ if (!p_counter_index)
+ return -EINVAL;
+ mlxsw_reg_ricnt_pack(ricnt_pl, *p_counter_index,
+ MLXSW_REG_RICNT_OPCODE_NOP);
+ err = mlxsw_reg_query(mlxsw_sp->core, MLXSW_REG(ricnt), ricnt_pl);
+ if (err)
+ return err;
+ *cnt = mlxsw_reg_ricnt_good_unicast_packets_get(ricnt_pl);
+ return 0;
+}
+
+static int mlxsw_sp_rif_counter_clear(struct mlxsw_sp *mlxsw_sp,
+ unsigned int counter_index)
+{
+ char ricnt_pl[MLXSW_REG_RICNT_LEN];
+
+ mlxsw_reg_ricnt_pack(ricnt_pl, counter_index,
+ MLXSW_REG_RICNT_OPCODE_CLEAR);
+ return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ricnt), ricnt_pl);
+}
+
+int mlxsw_sp_rif_counter_alloc(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_rif *rif,
+ enum mlxsw_sp_rif_counter_dir dir)
+{
+ unsigned int *p_counter_index;
+ int err;
+
+ p_counter_index = mlxsw_sp_rif_p_counter_get(rif, dir);
+ if (!p_counter_index)
+ return -EINVAL;
+ err = mlxsw_sp_counter_alloc(mlxsw_sp, MLXSW_SP_COUNTER_SUB_POOL_RIF,
+ p_counter_index);
+ if (err)
+ return err;
+
+ err = mlxsw_sp_rif_counter_clear(mlxsw_sp, *p_counter_index);
+ if (err)
+ goto err_counter_clear;
+
+ err = mlxsw_sp_rif_counter_edit(mlxsw_sp, rif->rif_index,
+ *p_counter_index, true, dir);
+ if (err)
+ goto err_counter_edit;
+ mlxsw_sp_rif_counter_valid_set(rif, dir, true);
+ return 0;
+
+err_counter_edit:
+err_counter_clear:
+ mlxsw_sp_counter_free(mlxsw_sp, MLXSW_SP_COUNTER_SUB_POOL_RIF,
+ *p_counter_index);
+ return err;
+}
+
+void mlxsw_sp_rif_counter_free(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_rif *rif,
+ enum mlxsw_sp_rif_counter_dir dir)
+{
+ unsigned int *p_counter_index;
+
+ p_counter_index = mlxsw_sp_rif_p_counter_get(rif, dir);
+ if (WARN_ON(!p_counter_index))
+ return;
+ mlxsw_sp_rif_counter_edit(mlxsw_sp, rif->rif_index,
+ *p_counter_index, false, dir);
+ mlxsw_sp_counter_free(mlxsw_sp, MLXSW_SP_COUNTER_SUB_POOL_RIF,
+ *p_counter_index);
+ mlxsw_sp_rif_counter_valid_set(rif, dir, false);
+}
+
+static struct mlxsw_sp_rif *
+mlxsw_sp_rif_find_by_dev(const struct mlxsw_sp *mlxsw_sp,
+ const struct net_device *dev);
#define mlxsw_sp_prefix_usage_for_each(prefix, prefix_usage) \
for_each_set_bit(prefix, (prefix_usage)->b, MLXSW_SP_PREFIX_COUNT)
@@ -89,12 +259,6 @@
}
static void
-mlxsw_sp_prefix_usage_zero(struct mlxsw_sp_prefix_usage *prefix_usage)
-{
- memset(prefix_usage, 0, sizeof(*prefix_usage));
-}
-
-static void
mlxsw_sp_prefix_usage_set(struct mlxsw_sp_prefix_usage *prefix_usage,
unsigned char prefix_len)
{
@@ -125,7 +289,7 @@
struct list_head entry_list;
struct list_head list;
struct rhash_head ht_node;
- struct mlxsw_sp_vr *vr;
+ struct mlxsw_sp_fib *fib;
struct mlxsw_sp_fib_key key;
};
@@ -149,13 +313,17 @@
struct mlxsw_sp_fib {
struct rhashtable ht;
struct list_head node_list;
+ struct mlxsw_sp_vr *vr;
+ struct mlxsw_sp_lpm_tree *lpm_tree;
unsigned long prefix_ref_count[MLXSW_SP_PREFIX_COUNT];
struct mlxsw_sp_prefix_usage prefix_usage;
+ enum mlxsw_sp_l3proto proto;
};
static const struct rhashtable_params mlxsw_sp_fib_ht_params;
-static struct mlxsw_sp_fib *mlxsw_sp_fib_create(void)
+static struct mlxsw_sp_fib *mlxsw_sp_fib_create(struct mlxsw_sp_vr *vr,
+ enum mlxsw_sp_l3proto proto)
{
struct mlxsw_sp_fib *fib;
int err;
@@ -167,6 +335,8 @@
if (err)
goto err_rhashtable_init;
INIT_LIST_HEAD(&fib->node_list);
+ fib->proto = proto;
+ fib->vr = vr;
return fib;
err_rhashtable_init:
@@ -177,24 +347,21 @@
static void mlxsw_sp_fib_destroy(struct mlxsw_sp_fib *fib)
{
WARN_ON(!list_empty(&fib->node_list));
+ WARN_ON(fib->lpm_tree);
rhashtable_destroy(&fib->ht);
kfree(fib);
}
static struct mlxsw_sp_lpm_tree *
-mlxsw_sp_lpm_tree_find_unused(struct mlxsw_sp *mlxsw_sp, bool one_reserved)
+mlxsw_sp_lpm_tree_find_unused(struct mlxsw_sp *mlxsw_sp)
{
static struct mlxsw_sp_lpm_tree *lpm_tree;
int i;
- for (i = 0; i < MLXSW_SP_LPM_TREE_COUNT; i++) {
- lpm_tree = &mlxsw_sp->router.lpm_trees[i];
- if (lpm_tree->ref_count == 0) {
- if (one_reserved)
- one_reserved = false;
- else
- return lpm_tree;
- }
+ for (i = 0; i < mlxsw_sp->router.lpm.tree_count; i++) {
+ lpm_tree = &mlxsw_sp->router.lpm.trees[i];
+ if (lpm_tree->ref_count == 0)
+ return lpm_tree;
}
return NULL;
}
@@ -248,12 +415,12 @@
static struct mlxsw_sp_lpm_tree *
mlxsw_sp_lpm_tree_create(struct mlxsw_sp *mlxsw_sp,
struct mlxsw_sp_prefix_usage *prefix_usage,
- enum mlxsw_sp_l3proto proto, bool one_reserved)
+ enum mlxsw_sp_l3proto proto)
{
struct mlxsw_sp_lpm_tree *lpm_tree;
int err;
- lpm_tree = mlxsw_sp_lpm_tree_find_unused(mlxsw_sp, one_reserved);
+ lpm_tree = mlxsw_sp_lpm_tree_find_unused(mlxsw_sp);
if (!lpm_tree)
return ERR_PTR(-EBUSY);
lpm_tree->proto = proto;
@@ -283,13 +450,13 @@
static struct mlxsw_sp_lpm_tree *
mlxsw_sp_lpm_tree_get(struct mlxsw_sp *mlxsw_sp,
struct mlxsw_sp_prefix_usage *prefix_usage,
- enum mlxsw_sp_l3proto proto, bool one_reserved)
+ enum mlxsw_sp_l3proto proto)
{
struct mlxsw_sp_lpm_tree *lpm_tree;
int i;
- for (i = 0; i < MLXSW_SP_LPM_TREE_COUNT; i++) {
- lpm_tree = &mlxsw_sp->router.lpm_trees[i];
+ for (i = 0; i < mlxsw_sp->router.lpm.tree_count; i++) {
+ lpm_tree = &mlxsw_sp->router.lpm.trees[i];
if (lpm_tree->ref_count != 0 &&
lpm_tree->proto == proto &&
mlxsw_sp_prefix_usage_eq(&lpm_tree->prefix_usage,
@@ -297,7 +464,7 @@
goto inc_ref_count;
}
lpm_tree = mlxsw_sp_lpm_tree_create(mlxsw_sp, prefix_usage,
- proto, one_reserved);
+ proto);
if (IS_ERR(lpm_tree))
return lpm_tree;
@@ -314,15 +481,41 @@
return 0;
}
-static void mlxsw_sp_lpm_init(struct mlxsw_sp *mlxsw_sp)
+#define MLXSW_SP_LPM_TREE_MIN 2 /* trees 0 and 1 are reserved */
+
+static int mlxsw_sp_lpm_init(struct mlxsw_sp *mlxsw_sp)
{
struct mlxsw_sp_lpm_tree *lpm_tree;
+ u64 max_trees;
int i;
- for (i = 0; i < MLXSW_SP_LPM_TREE_COUNT; i++) {
- lpm_tree = &mlxsw_sp->router.lpm_trees[i];
+ if (!MLXSW_CORE_RES_VALID(mlxsw_sp->core, MAX_LPM_TREES))
+ return -EIO;
+
+ max_trees = MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_LPM_TREES);
+ mlxsw_sp->router.lpm.tree_count = max_trees - MLXSW_SP_LPM_TREE_MIN;
+ mlxsw_sp->router.lpm.trees = kcalloc(mlxsw_sp->router.lpm.tree_count,
+ sizeof(struct mlxsw_sp_lpm_tree),
+ GFP_KERNEL);
+ if (!mlxsw_sp->router.lpm.trees)
+ return -ENOMEM;
+
+ for (i = 0; i < mlxsw_sp->router.lpm.tree_count; i++) {
+ lpm_tree = &mlxsw_sp->router.lpm.trees[i];
lpm_tree->id = i + MLXSW_SP_LPM_TREE_MIN;
}
+
+ return 0;
+}
+
+static void mlxsw_sp_lpm_fini(struct mlxsw_sp *mlxsw_sp)
+{
+ kfree(mlxsw_sp->router.lpm.trees);
+}
+
+static bool mlxsw_sp_vr_is_used(const struct mlxsw_sp_vr *vr)
+{
+ return !!vr->fib4;
}
static struct mlxsw_sp_vr *mlxsw_sp_vr_find_unused(struct mlxsw_sp *mlxsw_sp)
@@ -332,31 +525,31 @@
for (i = 0; i < MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_VRS); i++) {
vr = &mlxsw_sp->router.vrs[i];
- if (!vr->used)
+ if (!mlxsw_sp_vr_is_used(vr))
return vr;
}
return NULL;
}
static int mlxsw_sp_vr_lpm_tree_bind(struct mlxsw_sp *mlxsw_sp,
- struct mlxsw_sp_vr *vr)
+ const struct mlxsw_sp_fib *fib)
{
char raltb_pl[MLXSW_REG_RALTB_LEN];
- mlxsw_reg_raltb_pack(raltb_pl, vr->id,
- (enum mlxsw_reg_ralxx_protocol) vr->proto,
- vr->lpm_tree->id);
+ mlxsw_reg_raltb_pack(raltb_pl, fib->vr->id,
+ (enum mlxsw_reg_ralxx_protocol) fib->proto,
+ fib->lpm_tree->id);
return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(raltb), raltb_pl);
}
static int mlxsw_sp_vr_lpm_tree_unbind(struct mlxsw_sp *mlxsw_sp,
- struct mlxsw_sp_vr *vr)
+ const struct mlxsw_sp_fib *fib)
{
char raltb_pl[MLXSW_REG_RALTB_LEN];
/* Bind to tree 0 which is default */
- mlxsw_reg_raltb_pack(raltb_pl, vr->id,
- (enum mlxsw_reg_ralxx_protocol) vr->proto, 0);
+ mlxsw_reg_raltb_pack(raltb_pl, fib->vr->id,
+ (enum mlxsw_reg_ralxx_protocol) fib->proto, 0);
return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(raltb), raltb_pl);
}
@@ -369,8 +562,7 @@
}
static struct mlxsw_sp_vr *mlxsw_sp_vr_find(struct mlxsw_sp *mlxsw_sp,
- u32 tb_id,
- enum mlxsw_sp_l3proto proto)
+ u32 tb_id)
{
struct mlxsw_sp_vr *vr;
int i;
@@ -379,69 +571,50 @@
for (i = 0; i < MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_VRS); i++) {
vr = &mlxsw_sp->router.vrs[i];
- if (vr->used && vr->proto == proto && vr->tb_id == tb_id)
+ if (mlxsw_sp_vr_is_used(vr) && vr->tb_id == tb_id)
return vr;
}
return NULL;
}
-static struct mlxsw_sp_vr *mlxsw_sp_vr_create(struct mlxsw_sp *mlxsw_sp,
- unsigned char prefix_len,
- u32 tb_id,
- enum mlxsw_sp_l3proto proto)
+static struct mlxsw_sp_fib *mlxsw_sp_vr_fib(const struct mlxsw_sp_vr *vr,
+ enum mlxsw_sp_l3proto proto)
{
- struct mlxsw_sp_prefix_usage req_prefix_usage;
- struct mlxsw_sp_lpm_tree *lpm_tree;
+ switch (proto) {
+ case MLXSW_SP_L3_PROTO_IPV4:
+ return vr->fib4;
+ case MLXSW_SP_L3_PROTO_IPV6:
+ BUG_ON(1);
+ }
+ return NULL;
+}
+
+static struct mlxsw_sp_vr *mlxsw_sp_vr_create(struct mlxsw_sp *mlxsw_sp,
+ u32 tb_id)
+{
struct mlxsw_sp_vr *vr;
- int err;
vr = mlxsw_sp_vr_find_unused(mlxsw_sp);
if (!vr)
return ERR_PTR(-EBUSY);
- vr->fib = mlxsw_sp_fib_create();
- if (IS_ERR(vr->fib))
- return ERR_CAST(vr->fib);
-
- vr->proto = proto;
+ vr->fib4 = mlxsw_sp_fib_create(vr, MLXSW_SP_L3_PROTO_IPV4);
+ if (IS_ERR(vr->fib4))
+ return ERR_CAST(vr->fib4);
vr->tb_id = tb_id;
- mlxsw_sp_prefix_usage_zero(&req_prefix_usage);
- mlxsw_sp_prefix_usage_set(&req_prefix_usage, prefix_len);
- lpm_tree = mlxsw_sp_lpm_tree_get(mlxsw_sp, &req_prefix_usage,
- proto, true);
- if (IS_ERR(lpm_tree)) {
- err = PTR_ERR(lpm_tree);
- goto err_tree_get;
- }
- vr->lpm_tree = lpm_tree;
- err = mlxsw_sp_vr_lpm_tree_bind(mlxsw_sp, vr);
- if (err)
- goto err_tree_bind;
-
- vr->used = true;
return vr;
-
-err_tree_bind:
- mlxsw_sp_lpm_tree_put(mlxsw_sp, vr->lpm_tree);
-err_tree_get:
- mlxsw_sp_fib_destroy(vr->fib);
-
- return ERR_PTR(err);
}
-static void mlxsw_sp_vr_destroy(struct mlxsw_sp *mlxsw_sp,
- struct mlxsw_sp_vr *vr)
+static void mlxsw_sp_vr_destroy(struct mlxsw_sp_vr *vr)
{
- mlxsw_sp_vr_lpm_tree_unbind(mlxsw_sp, vr);
- mlxsw_sp_lpm_tree_put(mlxsw_sp, vr->lpm_tree);
- mlxsw_sp_fib_destroy(vr->fib);
- vr->used = false;
+ mlxsw_sp_fib_destroy(vr->fib4);
+ vr->fib4 = NULL;
}
static int
-mlxsw_sp_vr_lpm_tree_check(struct mlxsw_sp *mlxsw_sp, struct mlxsw_sp_vr *vr,
+mlxsw_sp_vr_lpm_tree_check(struct mlxsw_sp *mlxsw_sp, struct mlxsw_sp_fib *fib,
struct mlxsw_sp_prefix_usage *req_prefix_usage)
{
- struct mlxsw_sp_lpm_tree *lpm_tree = vr->lpm_tree;
+ struct mlxsw_sp_lpm_tree *lpm_tree = fib->lpm_tree;
struct mlxsw_sp_lpm_tree *new_tree;
int err;
@@ -449,7 +622,7 @@
return 0;
new_tree = mlxsw_sp_lpm_tree_get(mlxsw_sp, req_prefix_usage,
- vr->proto, false);
+ fib->proto);
if (IS_ERR(new_tree)) {
/* We failed to get a tree according to the required
* prefix usage. However, the current tree might be still good
@@ -463,8 +636,8 @@
}
/* Prevent packet loss by overwriting existing binding */
- vr->lpm_tree = new_tree;
- err = mlxsw_sp_vr_lpm_tree_bind(mlxsw_sp, vr);
+ fib->lpm_tree = new_tree;
+ err = mlxsw_sp_vr_lpm_tree_bind(mlxsw_sp, fib);
if (err)
goto err_tree_bind;
mlxsw_sp_lpm_tree_put(mlxsw_sp, lpm_tree);
@@ -472,53 +645,26 @@
return 0;
err_tree_bind:
- vr->lpm_tree = lpm_tree;
+ fib->lpm_tree = lpm_tree;
mlxsw_sp_lpm_tree_put(mlxsw_sp, new_tree);
return err;
}
-static struct mlxsw_sp_vr *mlxsw_sp_vr_get(struct mlxsw_sp *mlxsw_sp,
- unsigned char prefix_len,
- u32 tb_id,
- enum mlxsw_sp_l3proto proto)
+static struct mlxsw_sp_vr *mlxsw_sp_vr_get(struct mlxsw_sp *mlxsw_sp, u32 tb_id)
{
struct mlxsw_sp_vr *vr;
- int err;
tb_id = mlxsw_sp_fix_tb_id(tb_id);
- vr = mlxsw_sp_vr_find(mlxsw_sp, tb_id, proto);
- if (!vr) {
- vr = mlxsw_sp_vr_create(mlxsw_sp, prefix_len, tb_id, proto);
- if (IS_ERR(vr))
- return vr;
- } else {
- struct mlxsw_sp_prefix_usage req_prefix_usage;
-
- mlxsw_sp_prefix_usage_cpy(&req_prefix_usage,
- &vr->fib->prefix_usage);
- mlxsw_sp_prefix_usage_set(&req_prefix_usage, prefix_len);
- /* Need to replace LPM tree in case new prefix is required. */
- err = mlxsw_sp_vr_lpm_tree_check(mlxsw_sp, vr,
- &req_prefix_usage);
- if (err)
- return ERR_PTR(err);
- }
+ vr = mlxsw_sp_vr_find(mlxsw_sp, tb_id);
+ if (!vr)
+ vr = mlxsw_sp_vr_create(mlxsw_sp, tb_id);
return vr;
}
-static void mlxsw_sp_vr_put(struct mlxsw_sp *mlxsw_sp, struct mlxsw_sp_vr *vr)
+static void mlxsw_sp_vr_put(struct mlxsw_sp_vr *vr)
{
- /* Destroy virtual router entity in case the associated FIB is empty
- * and allow it to be used for other tables in future. Otherwise,
- * check if some prefix usage did not disappear and change tree if
- * that is the case. Note that in case new, smaller tree cannot be
- * allocated, the original one will be kept being used.
- */
- if (mlxsw_sp_prefix_usage_none(&vr->fib->prefix_usage))
- mlxsw_sp_vr_destroy(mlxsw_sp, vr);
- else
- mlxsw_sp_vr_lpm_tree_check(mlxsw_sp, vr,
- &vr->fib->prefix_usage);
+ if (!vr->rif_count && list_empty(&vr->fib4->node_list))
+ mlxsw_sp_vr_destroy(vr);
}
static int mlxsw_sp_vrs_init(struct mlxsw_sp *mlxsw_sp)
@@ -627,14 +773,14 @@
mlxsw_sp_neigh_entry_create(struct mlxsw_sp *mlxsw_sp, struct neighbour *n)
{
struct mlxsw_sp_neigh_entry *neigh_entry;
- struct mlxsw_sp_rif *r;
+ struct mlxsw_sp_rif *rif;
int err;
- r = mlxsw_sp_rif_find_by_dev(mlxsw_sp, n->dev);
- if (!r)
+ rif = mlxsw_sp_rif_find_by_dev(mlxsw_sp, n->dev);
+ if (!rif)
return ERR_PTR(-EINVAL);
- neigh_entry = mlxsw_sp_neigh_entry_alloc(mlxsw_sp, n, r->rif);
+ neigh_entry = mlxsw_sp_neigh_entry_alloc(mlxsw_sp, n, rif->rif_index);
if (!neigh_entry)
return ERR_PTR(-ENOMEM);
@@ -642,7 +788,7 @@
if (err)
goto err_neigh_entry_insert;
- list_add(&neigh_entry->rif_list_node, &r->neigh_list);
+ list_add(&neigh_entry->rif_list_node, &rif->neigh_list);
return neigh_entry;
@@ -1050,22 +1196,22 @@
}
static int mlxsw_sp_neigh_rif_flush(struct mlxsw_sp *mlxsw_sp,
- const struct mlxsw_sp_rif *r)
+ const struct mlxsw_sp_rif *rif)
{
char rauht_pl[MLXSW_REG_RAUHT_LEN];
mlxsw_reg_rauht_pack(rauht_pl, MLXSW_REG_RAUHT_OP_WRITE_DELETE_ALL,
- r->rif, r->addr);
+ rif->rif_index, rif->addr);
return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(rauht), rauht_pl);
}
static void mlxsw_sp_neigh_rif_gone_sync(struct mlxsw_sp *mlxsw_sp,
- struct mlxsw_sp_rif *r)
+ struct mlxsw_sp_rif *rif)
{
struct mlxsw_sp_neigh_entry *neigh_entry, *tmp;
- mlxsw_sp_neigh_rif_flush(mlxsw_sp, r);
- list_for_each_entry_safe(neigh_entry, tmp, &r->neigh_list,
+ mlxsw_sp_neigh_rif_flush(mlxsw_sp, rif);
+ list_for_each_entry_safe(neigh_entry, tmp, &rif->neigh_list,
rif_list_node)
mlxsw_sp_neigh_entry_destroy(mlxsw_sp, neigh_entry);
}
@@ -1082,7 +1228,7 @@
*/
struct rhash_head ht_node;
struct mlxsw_sp_nexthop_key key;
- struct mlxsw_sp_rif *r;
+ struct mlxsw_sp_rif *rif;
u8 should_offload:1, /* set indicates this neigh is connected and
* should be put to KVD linear area of this group.
*/
@@ -1109,7 +1255,7 @@
u16 ecmp_size;
u16 count;
struct mlxsw_sp_nexthop nexthops[0];
-#define nh_rif nexthops[0].r
+#define nh_rif nexthops[0].rif
};
static const struct rhashtable_params mlxsw_sp_nexthop_group_ht_params = {
@@ -1171,7 +1317,7 @@
}
static int mlxsw_sp_adj_index_mass_update_vr(struct mlxsw_sp *mlxsw_sp,
- struct mlxsw_sp_vr *vr,
+ const struct mlxsw_sp_fib *fib,
u32 adj_index, u16 ecmp_size,
u32 new_adj_index,
u16 new_ecmp_size)
@@ -1179,8 +1325,8 @@
char raleu_pl[MLXSW_REG_RALEU_LEN];
mlxsw_reg_raleu_pack(raleu_pl,
- (enum mlxsw_reg_ralxx_protocol) vr->proto, vr->id,
- adj_index, ecmp_size, new_adj_index,
+ (enum mlxsw_reg_ralxx_protocol) fib->proto,
+ fib->vr->id, adj_index, ecmp_size, new_adj_index,
new_ecmp_size);
return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(raleu), raleu_pl);
}
@@ -1190,14 +1336,14 @@
u32 old_adj_index, u16 old_ecmp_size)
{
struct mlxsw_sp_fib_entry *fib_entry;
- struct mlxsw_sp_vr *vr = NULL;
+ struct mlxsw_sp_fib *fib = NULL;
int err;
list_for_each_entry(fib_entry, &nh_grp->fib_list, nexthop_group_node) {
- if (vr == fib_entry->fib_node->vr)
+ if (fib == fib_entry->fib_node->fib)
continue;
- vr = fib_entry->fib_node->vr;
- err = mlxsw_sp_adj_index_mass_update_vr(mlxsw_sp, vr,
+ fib = fib_entry->fib_node->fib;
+ err = mlxsw_sp_adj_index_mass_update_vr(mlxsw_sp, fib,
old_adj_index,
old_ecmp_size,
nh_grp->adj_index,
@@ -1280,7 +1426,6 @@
bool old_adj_index_valid;
u32 old_adj_index;
u16 old_ecmp_size;
- int ret;
int i;
int err;
@@ -1318,15 +1463,14 @@
*/
goto set_trap;
- ret = mlxsw_sp_kvdl_alloc(mlxsw_sp, ecmp_size);
- if (ret < 0) {
+ err = mlxsw_sp_kvdl_alloc(mlxsw_sp, ecmp_size, &adj_index);
+ if (err) {
/* We ran out of KVD linear space, just set the
* trap and let everything flow through kernel.
*/
dev_warn(mlxsw_sp->bus_info->dev, "Failed to allocate KVD linear area for nexthop group.\n");
goto set_trap;
}
- adj_index = ret;
old_adj_index_valid = nh_grp->adj_index_valid;
old_adj_index = nh_grp->adj_index;
old_ecmp_size = nh_grp->ecmp_size;
@@ -1399,22 +1543,22 @@
}
static void mlxsw_sp_nexthop_rif_init(struct mlxsw_sp_nexthop *nh,
- struct mlxsw_sp_rif *r)
+ struct mlxsw_sp_rif *rif)
{
- if (nh->r)
+ if (nh->rif)
return;
- nh->r = r;
- list_add(&nh->rif_list_node, &r->nexthop_list);
+ nh->rif = rif;
+ list_add(&nh->rif_list_node, &rif->nexthop_list);
}
static void mlxsw_sp_nexthop_rif_fini(struct mlxsw_sp_nexthop *nh)
{
- if (!nh->r)
+ if (!nh->rif)
return;
list_del(&nh->rif_list_node);
- nh->r = NULL;
+ nh->rif = NULL;
}
static int mlxsw_sp_nexthop_neigh_init(struct mlxsw_sp *mlxsw_sp,
@@ -1505,7 +1649,7 @@
{
struct net_device *dev = fib_nh->nh_dev;
struct in_device *in_dev;
- struct mlxsw_sp_rif *r;
+ struct mlxsw_sp_rif *rif;
int err;
nh->nh_grp = nh_grp;
@@ -1514,15 +1658,18 @@
if (err)
return err;
+ if (!dev)
+ return 0;
+
in_dev = __in_dev_get_rtnl(dev);
if (in_dev && IN_DEV_IGNORE_ROUTES_WITH_LINKDOWN(in_dev) &&
fib_nh->nh_flags & RTNH_F_LINKDOWN)
return 0;
- r = mlxsw_sp_rif_find_by_dev(mlxsw_sp, dev);
- if (!r)
+ rif = mlxsw_sp_rif_find_by_dev(mlxsw_sp, dev);
+ if (!rif)
return 0;
- mlxsw_sp_nexthop_rif_init(nh, r);
+ mlxsw_sp_nexthop_rif_init(nh, rif);
err = mlxsw_sp_nexthop_neigh_init(mlxsw_sp, nh);
if (err)
@@ -1548,7 +1695,7 @@
{
struct mlxsw_sp_nexthop_key key;
struct mlxsw_sp_nexthop *nh;
- struct mlxsw_sp_rif *r;
+ struct mlxsw_sp_rif *rif;
if (mlxsw_sp->router.aborted)
return;
@@ -1558,13 +1705,13 @@
if (WARN_ON_ONCE(!nh))
return;
- r = mlxsw_sp_rif_find_by_dev(mlxsw_sp, fib_nh->nh_dev);
- if (!r)
+ rif = mlxsw_sp_rif_find_by_dev(mlxsw_sp, fib_nh->nh_dev);
+ if (!rif)
return;
switch (event) {
case FIB_EVENT_NH_ADD:
- mlxsw_sp_nexthop_rif_init(nh, r);
+ mlxsw_sp_nexthop_rif_init(nh, rif);
mlxsw_sp_nexthop_neigh_init(mlxsw_sp, nh);
break;
case FIB_EVENT_NH_DEL:
@@ -1577,11 +1724,11 @@
}
static void mlxsw_sp_nexthop_rif_gone_sync(struct mlxsw_sp *mlxsw_sp,
- struct mlxsw_sp_rif *r)
+ struct mlxsw_sp_rif *rif)
{
struct mlxsw_sp_nexthop *nh, *tmp;
- list_for_each_entry_safe(nh, tmp, &r->nexthop_list, rif_list_node) {
+ list_for_each_entry_safe(nh, tmp, &rif->nexthop_list, rif_list_node) {
mlxsw_sp_nexthop_neigh_fini(mlxsw_sp, nh);
mlxsw_sp_nexthop_rif_fini(nh);
mlxsw_sp_nexthop_group_refresh(mlxsw_sp, nh->nh_grp);
@@ -1699,7 +1846,7 @@
{
fib_entry->offloaded = true;
- switch (fib_entry->fib_node->vr->proto) {
+ switch (fib_entry->fib_node->fib->proto) {
case MLXSW_SP_L3_PROTO_IPV4:
fib_info_offload_inc(fib_entry->nh_group->key.fi);
break;
@@ -1711,7 +1858,7 @@
static void
mlxsw_sp_fib_entry_offload_unset(struct mlxsw_sp_fib_entry *fib_entry)
{
- switch (fib_entry->fib_node->vr->proto) {
+ switch (fib_entry->fib_node->fib->proto) {
case MLXSW_SP_L3_PROTO_IPV4:
fib_info_offload_dec(fib_entry->nh_group->key.fi);
break;
@@ -1751,8 +1898,8 @@
enum mlxsw_reg_ralue_op op)
{
char ralue_pl[MLXSW_REG_RALUE_LEN];
+ struct mlxsw_sp_fib *fib = fib_entry->fib_node->fib;
u32 *p_dip = (u32 *) fib_entry->fib_node->key.addr;
- struct mlxsw_sp_vr *vr = fib_entry->fib_node->vr;
enum mlxsw_reg_ralue_trap_action trap_action;
u16 trap_id = 0;
u32 adjacency_index = 0;
@@ -1772,8 +1919,8 @@
}
mlxsw_reg_ralue_pack4(ralue_pl,
- (enum mlxsw_reg_ralxx_protocol) vr->proto, op,
- vr->id, fib_entry->fib_node->key.prefix_len,
+ (enum mlxsw_reg_ralxx_protocol) fib->proto, op,
+ fib->vr->id, fib_entry->fib_node->key.prefix_len,
*p_dip);
mlxsw_reg_ralue_act_remote_pack(ralue_pl, trap_action, trap_id,
adjacency_index, ecmp_size);
@@ -1784,27 +1931,28 @@
struct mlxsw_sp_fib_entry *fib_entry,
enum mlxsw_reg_ralue_op op)
{
- struct mlxsw_sp_rif *r = fib_entry->nh_group->nh_rif;
+ struct mlxsw_sp_rif *rif = fib_entry->nh_group->nh_rif;
+ struct mlxsw_sp_fib *fib = fib_entry->fib_node->fib;
enum mlxsw_reg_ralue_trap_action trap_action;
char ralue_pl[MLXSW_REG_RALUE_LEN];
u32 *p_dip = (u32 *) fib_entry->fib_node->key.addr;
- struct mlxsw_sp_vr *vr = fib_entry->fib_node->vr;
u16 trap_id = 0;
- u16 rif = 0;
+ u16 rif_index = 0;
if (mlxsw_sp_fib_entry_should_offload(fib_entry)) {
trap_action = MLXSW_REG_RALUE_TRAP_ACTION_NOP;
- rif = r->rif;
+ rif_index = rif->rif_index;
} else {
trap_action = MLXSW_REG_RALUE_TRAP_ACTION_TRAP;
trap_id = MLXSW_TRAP_ID_RTR_INGRESS0;
}
mlxsw_reg_ralue_pack4(ralue_pl,
- (enum mlxsw_reg_ralxx_protocol) vr->proto, op,
- vr->id, fib_entry->fib_node->key.prefix_len,
+ (enum mlxsw_reg_ralxx_protocol) fib->proto, op,
+ fib->vr->id, fib_entry->fib_node->key.prefix_len,
*p_dip);
- mlxsw_reg_ralue_act_local_pack(ralue_pl, trap_action, trap_id, rif);
+ mlxsw_reg_ralue_act_local_pack(ralue_pl, trap_action, trap_id,
+ rif_index);
return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ralue), ralue_pl);
}
@@ -1812,13 +1960,13 @@
struct mlxsw_sp_fib_entry *fib_entry,
enum mlxsw_reg_ralue_op op)
{
+ struct mlxsw_sp_fib *fib = fib_entry->fib_node->fib;
char ralue_pl[MLXSW_REG_RALUE_LEN];
u32 *p_dip = (u32 *) fib_entry->fib_node->key.addr;
- struct mlxsw_sp_vr *vr = fib_entry->fib_node->vr;
mlxsw_reg_ralue_pack4(ralue_pl,
- (enum mlxsw_reg_ralxx_protocol) vr->proto, op,
- vr->id, fib_entry->fib_node->key.prefix_len,
+ (enum mlxsw_reg_ralxx_protocol) fib->proto, op,
+ fib->vr->id, fib_entry->fib_node->key.prefix_len,
*p_dip);
mlxsw_reg_ralue_act_ip2me_pack(ralue_pl);
return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ralue), ralue_pl);
@@ -1845,7 +1993,7 @@
{
int err = -EINVAL;
- switch (fib_entry->fib_node->vr->proto) {
+ switch (fib_entry->fib_node->fib->proto) {
case MLXSW_SP_L3_PROTO_IPV4:
err = mlxsw_sp_fib_entry_op4(mlxsw_sp, fib_entry, op);
break;
@@ -1877,17 +2025,29 @@
{
struct fib_info *fi = fen_info->fi;
- if (fen_info->type == RTN_LOCAL || fen_info->type == RTN_BROADCAST) {
+ switch (fen_info->type) {
+ case RTN_BROADCAST: /* fall through */
+ case RTN_LOCAL:
fib_entry->type = MLXSW_SP_FIB_ENTRY_TYPE_TRAP;
return 0;
- }
- if (fen_info->type != RTN_UNICAST)
- return -EINVAL;
- if (fi->fib_nh->nh_scope != RT_SCOPE_LINK)
+ case RTN_UNREACHABLE: /* fall through */
+ case RTN_BLACKHOLE: /* fall through */
+ case RTN_PROHIBIT:
+ /* Packets hitting these routes need to be trapped, but
+ * can do so with a lower priority than packets directed
+ * at the host, so use action type local instead of trap.
+ */
fib_entry->type = MLXSW_SP_FIB_ENTRY_TYPE_LOCAL;
- else
- fib_entry->type = MLXSW_SP_FIB_ENTRY_TYPE_REMOTE;
- return 0;
+ return 0;
+ case RTN_UNICAST:
+ if (fi->fib_nh->nh_scope != RT_SCOPE_LINK)
+ fib_entry->type = MLXSW_SP_FIB_ENTRY_TYPE_LOCAL;
+ else
+ fib_entry->type = MLXSW_SP_FIB_ENTRY_TYPE_REMOTE;
+ return 0;
+ default:
+ return -EINVAL;
+ }
}
static struct mlxsw_sp_fib_entry *
@@ -1996,7 +2156,7 @@
}
static struct mlxsw_sp_fib_node *
-mlxsw_sp_fib_node_create(struct mlxsw_sp_vr *vr, const void *addr,
+mlxsw_sp_fib_node_create(struct mlxsw_sp_fib *fib, const void *addr,
size_t addr_len, unsigned char prefix_len)
{
struct mlxsw_sp_fib_node *fib_node;
@@ -2006,18 +2166,15 @@
return NULL;
INIT_LIST_HEAD(&fib_node->entry_list);
- list_add(&fib_node->list, &vr->fib->node_list);
+ list_add(&fib_node->list, &fib->node_list);
memcpy(fib_node->key.addr, addr, addr_len);
fib_node->key.prefix_len = prefix_len;
- mlxsw_sp_fib_node_insert(vr->fib, fib_node);
- fib_node->vr = vr;
return fib_node;
}
static void mlxsw_sp_fib_node_destroy(struct mlxsw_sp_fib_node *fib_node)
{
- mlxsw_sp_fib_node_remove(fib_node->vr->fib, fib_node);
list_del(&fib_node->list);
WARN_ON(!list_empty(&fib_node->entry_list));
kfree(fib_node);
@@ -2034,7 +2191,7 @@
static void mlxsw_sp_fib_node_prefix_inc(struct mlxsw_sp_fib_node *fib_node)
{
unsigned char prefix_len = fib_node->key.prefix_len;
- struct mlxsw_sp_fib *fib = fib_node->vr->fib;
+ struct mlxsw_sp_fib *fib = fib_node->fib;
if (fib->prefix_ref_count[prefix_len]++ == 0)
mlxsw_sp_prefix_usage_set(&fib->prefix_usage, prefix_len);
@@ -2043,32 +2200,98 @@
static void mlxsw_sp_fib_node_prefix_dec(struct mlxsw_sp_fib_node *fib_node)
{
unsigned char prefix_len = fib_node->key.prefix_len;
- struct mlxsw_sp_fib *fib = fib_node->vr->fib;
+ struct mlxsw_sp_fib *fib = fib_node->fib;
if (--fib->prefix_ref_count[prefix_len] == 0)
mlxsw_sp_prefix_usage_clear(&fib->prefix_usage, prefix_len);
}
+static int mlxsw_sp_fib_node_init(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_fib_node *fib_node,
+ struct mlxsw_sp_fib *fib)
+{
+ struct mlxsw_sp_prefix_usage req_prefix_usage;
+ struct mlxsw_sp_lpm_tree *lpm_tree;
+ int err;
+
+ err = mlxsw_sp_fib_node_insert(fib, fib_node);
+ if (err)
+ return err;
+ fib_node->fib = fib;
+
+ mlxsw_sp_prefix_usage_cpy(&req_prefix_usage, &fib->prefix_usage);
+ mlxsw_sp_prefix_usage_set(&req_prefix_usage, fib_node->key.prefix_len);
+
+ if (!mlxsw_sp_prefix_usage_none(&fib->prefix_usage)) {
+ err = mlxsw_sp_vr_lpm_tree_check(mlxsw_sp, fib,
+ &req_prefix_usage);
+ if (err)
+ goto err_tree_check;
+ } else {
+ lpm_tree = mlxsw_sp_lpm_tree_get(mlxsw_sp, &req_prefix_usage,
+ fib->proto);
+ if (IS_ERR(lpm_tree))
+ return PTR_ERR(lpm_tree);
+ fib->lpm_tree = lpm_tree;
+ err = mlxsw_sp_vr_lpm_tree_bind(mlxsw_sp, fib);
+ if (err)
+ goto err_tree_bind;
+ }
+
+ mlxsw_sp_fib_node_prefix_inc(fib_node);
+
+ return 0;
+
+err_tree_bind:
+ fib->lpm_tree = NULL;
+ mlxsw_sp_lpm_tree_put(mlxsw_sp, lpm_tree);
+err_tree_check:
+ fib_node->fib = NULL;
+ mlxsw_sp_fib_node_remove(fib, fib_node);
+ return err;
+}
+
+static void mlxsw_sp_fib_node_fini(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_fib_node *fib_node)
+{
+ struct mlxsw_sp_lpm_tree *lpm_tree = fib_node->fib->lpm_tree;
+ struct mlxsw_sp_fib *fib = fib_node->fib;
+
+ mlxsw_sp_fib_node_prefix_dec(fib_node);
+
+ if (mlxsw_sp_prefix_usage_none(&fib->prefix_usage)) {
+ mlxsw_sp_vr_lpm_tree_unbind(mlxsw_sp, fib);
+ fib->lpm_tree = NULL;
+ mlxsw_sp_lpm_tree_put(mlxsw_sp, lpm_tree);
+ } else {
+ mlxsw_sp_vr_lpm_tree_check(mlxsw_sp, fib, &fib->prefix_usage);
+ }
+
+ fib_node->fib = NULL;
+ mlxsw_sp_fib_node_remove(fib, fib_node);
+}
+
static struct mlxsw_sp_fib_node *
mlxsw_sp_fib4_node_get(struct mlxsw_sp *mlxsw_sp,
const struct fib_entry_notifier_info *fen_info)
{
struct mlxsw_sp_fib_node *fib_node;
+ struct mlxsw_sp_fib *fib;
struct mlxsw_sp_vr *vr;
int err;
- vr = mlxsw_sp_vr_get(mlxsw_sp, fen_info->dst_len, fen_info->tb_id,
- MLXSW_SP_L3_PROTO_IPV4);
+ vr = mlxsw_sp_vr_get(mlxsw_sp, fen_info->tb_id);
if (IS_ERR(vr))
return ERR_CAST(vr);
+ fib = mlxsw_sp_vr_fib(vr, MLXSW_SP_L3_PROTO_IPV4);
- fib_node = mlxsw_sp_fib_node_lookup(vr->fib, &fen_info->dst,
+ fib_node = mlxsw_sp_fib_node_lookup(fib, &fen_info->dst,
sizeof(fen_info->dst),
fen_info->dst_len);
if (fib_node)
return fib_node;
- fib_node = mlxsw_sp_fib_node_create(vr, &fen_info->dst,
+ fib_node = mlxsw_sp_fib_node_create(fib, &fen_info->dst,
sizeof(fen_info->dst),
fen_info->dst_len);
if (!fib_node) {
@@ -2076,22 +2299,29 @@
goto err_fib_node_create;
}
+ err = mlxsw_sp_fib_node_init(mlxsw_sp, fib_node, fib);
+ if (err)
+ goto err_fib_node_init;
+
return fib_node;
+err_fib_node_init:
+ mlxsw_sp_fib_node_destroy(fib_node);
err_fib_node_create:
- mlxsw_sp_vr_put(mlxsw_sp, vr);
+ mlxsw_sp_vr_put(vr);
return ERR_PTR(err);
}
static void mlxsw_sp_fib4_node_put(struct mlxsw_sp *mlxsw_sp,
struct mlxsw_sp_fib_node *fib_node)
{
- struct mlxsw_sp_vr *vr = fib_node->vr;
+ struct mlxsw_sp_vr *vr = fib_node->fib->vr;
if (!list_empty(&fib_node->entry_list))
return;
+ mlxsw_sp_fib_node_fini(mlxsw_sp, fib_node);
mlxsw_sp_fib_node_destroy(fib_node);
- mlxsw_sp_vr_put(mlxsw_sp, vr);
+ mlxsw_sp_vr_put(vr);
}
static struct mlxsw_sp_fib_entry *
@@ -2236,8 +2466,6 @@
if (err)
goto err_fib4_node_entry_add;
- mlxsw_sp_fib_node_prefix_inc(fib_node);
-
return 0;
err_fib4_node_entry_add:
@@ -2251,7 +2479,6 @@
{
struct mlxsw_sp_fib_node *fib_node = fib_entry->fib_node;
- mlxsw_sp_fib_node_prefix_dec(fib_node);
mlxsw_sp_fib4_node_entry_del(mlxsw_sp, fib_node, fib_entry);
mlxsw_sp_fib4_node_list_remove(fib_entry);
}
@@ -2340,9 +2567,7 @@
{
char ralta_pl[MLXSW_REG_RALTA_LEN];
char ralst_pl[MLXSW_REG_RALST_LEN];
- char raltb_pl[MLXSW_REG_RALTB_LEN];
- char ralue_pl[MLXSW_REG_RALUE_LEN];
- int err;
+ int i, err;
mlxsw_reg_ralta_pack(ralta_pl, true, MLXSW_REG_RALXX_PROTOCOL_IPV4,
MLXSW_SP_LPM_TREE_MIN);
@@ -2355,16 +2580,33 @@
if (err)
return err;
- mlxsw_reg_raltb_pack(raltb_pl, 0, MLXSW_REG_RALXX_PROTOCOL_IPV4,
- MLXSW_SP_LPM_TREE_MIN);
- err = mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(raltb), raltb_pl);
- if (err)
- return err;
+ for (i = 0; i < MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_VRS); i++) {
+ struct mlxsw_sp_vr *vr = &mlxsw_sp->router.vrs[i];
+ char raltb_pl[MLXSW_REG_RALTB_LEN];
+ char ralue_pl[MLXSW_REG_RALUE_LEN];
- mlxsw_reg_ralue_pack4(ralue_pl, MLXSW_SP_L3_PROTO_IPV4,
- MLXSW_REG_RALUE_OP_WRITE_WRITE, 0, 0, 0);
- mlxsw_reg_ralue_act_ip2me_pack(ralue_pl);
- return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ralue), ralue_pl);
+ if (!mlxsw_sp_vr_is_used(vr))
+ continue;
+
+ mlxsw_reg_raltb_pack(raltb_pl, vr->id,
+ MLXSW_REG_RALXX_PROTOCOL_IPV4,
+ MLXSW_SP_LPM_TREE_MIN);
+ err = mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(raltb),
+ raltb_pl);
+ if (err)
+ return err;
+
+ mlxsw_reg_ralue_pack4(ralue_pl, MLXSW_SP_L3_PROTO_IPV4,
+ MLXSW_REG_RALUE_OP_WRITE_WRITE, vr->id, 0,
+ 0);
+ mlxsw_reg_ralue_act_ip2me_pack(ralue_pl);
+ err = mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ralue),
+ ralue_pl);
+ if (err)
+ return err;
+ }
+
+ return 0;
}
static void mlxsw_sp_fib4_node_flush(struct mlxsw_sp *mlxsw_sp,
@@ -2390,7 +2632,7 @@
static void mlxsw_sp_fib_node_flush(struct mlxsw_sp *mlxsw_sp,
struct mlxsw_sp_fib_node *fib_node)
{
- switch (fib_node->vr->proto) {
+ switch (fib_node->fib->proto) {
case MLXSW_SP_L3_PROTO_IPV4:
mlxsw_sp_fib4_node_flush(mlxsw_sp, fib_node);
break;
@@ -2400,26 +2642,32 @@
}
}
+static void mlxsw_sp_vr_fib_flush(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_vr *vr,
+ enum mlxsw_sp_l3proto proto)
+{
+ struct mlxsw_sp_fib *fib = mlxsw_sp_vr_fib(vr, proto);
+ struct mlxsw_sp_fib_node *fib_node, *tmp;
+
+ list_for_each_entry_safe(fib_node, tmp, &fib->node_list, list) {
+ bool do_break = &tmp->list == &fib->node_list;
+
+ mlxsw_sp_fib_node_flush(mlxsw_sp, fib_node);
+ if (do_break)
+ break;
+ }
+}
+
static void mlxsw_sp_router_fib_flush(struct mlxsw_sp *mlxsw_sp)
{
- struct mlxsw_sp_fib_node *fib_node, *tmp;
- struct mlxsw_sp_vr *vr;
int i;
for (i = 0; i < MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_VRS); i++) {
- vr = &mlxsw_sp->router.vrs[i];
+ struct mlxsw_sp_vr *vr = &mlxsw_sp->router.vrs[i];
- if (!vr->used)
+ if (!mlxsw_sp_vr_is_used(vr))
continue;
-
- list_for_each_entry_safe(fib_node, tmp, &vr->fib->node_list,
- list) {
- bool do_break = &tmp->list == &vr->fib->node_list;
-
- mlxsw_sp_fib_node_flush(mlxsw_sp, fib_node);
- if (do_break)
- break;
- }
+ mlxsw_sp_vr_fib_flush(mlxsw_sp, vr, MLXSW_SP_L3_PROTO_IPV4);
}
}
@@ -2437,6 +2685,122 @@
dev_warn(mlxsw_sp->bus_info->dev, "Failed to set abort trap.\n");
}
+struct mlxsw_sp_fib_event_work {
+ struct work_struct work;
+ union {
+ struct fib_entry_notifier_info fen_info;
+ struct fib_rule_notifier_info fr_info;
+ struct fib_nh_notifier_info fnh_info;
+ };
+ struct mlxsw_sp *mlxsw_sp;
+ unsigned long event;
+};
+
+static void mlxsw_sp_router_fib_event_work(struct work_struct *work)
+{
+ struct mlxsw_sp_fib_event_work *fib_work =
+ container_of(work, struct mlxsw_sp_fib_event_work, work);
+ struct mlxsw_sp *mlxsw_sp = fib_work->mlxsw_sp;
+ struct fib_rule *rule;
+ bool replace, append;
+ int err;
+
+ /* Protect internal structures from changes */
+ rtnl_lock();
+ switch (fib_work->event) {
+ case FIB_EVENT_ENTRY_REPLACE: /* fall through */
+ case FIB_EVENT_ENTRY_APPEND: /* fall through */
+ case FIB_EVENT_ENTRY_ADD:
+ replace = fib_work->event == FIB_EVENT_ENTRY_REPLACE;
+ append = fib_work->event == FIB_EVENT_ENTRY_APPEND;
+ err = mlxsw_sp_router_fib4_add(mlxsw_sp, &fib_work->fen_info,
+ replace, append);
+ if (err)
+ mlxsw_sp_router_fib4_abort(mlxsw_sp);
+ fib_info_put(fib_work->fen_info.fi);
+ break;
+ case FIB_EVENT_ENTRY_DEL:
+ mlxsw_sp_router_fib4_del(mlxsw_sp, &fib_work->fen_info);
+ fib_info_put(fib_work->fen_info.fi);
+ break;
+ case FIB_EVENT_RULE_ADD: /* fall through */
+ case FIB_EVENT_RULE_DEL:
+ rule = fib_work->fr_info.rule;
+ if (!fib4_rule_default(rule) && !rule->l3mdev)
+ mlxsw_sp_router_fib4_abort(mlxsw_sp);
+ fib_rule_put(rule);
+ break;
+ case FIB_EVENT_NH_ADD: /* fall through */
+ case FIB_EVENT_NH_DEL:
+ mlxsw_sp_nexthop_event(mlxsw_sp, fib_work->event,
+ fib_work->fnh_info.fib_nh);
+ fib_info_put(fib_work->fnh_info.fib_nh->nh_parent);
+ break;
+ }
+ rtnl_unlock();
+ kfree(fib_work);
+}
+
+/* Called with rcu_read_lock() */
+static int mlxsw_sp_router_fib_event(struct notifier_block *nb,
+ unsigned long event, void *ptr)
+{
+ struct mlxsw_sp *mlxsw_sp = container_of(nb, struct mlxsw_sp, fib_nb);
+ struct mlxsw_sp_fib_event_work *fib_work;
+ struct fib_notifier_info *info = ptr;
+
+ if (!net_eq(info->net, &init_net))
+ return NOTIFY_DONE;
+
+ fib_work = kzalloc(sizeof(*fib_work), GFP_ATOMIC);
+ if (WARN_ON(!fib_work))
+ return NOTIFY_BAD;
+
+ INIT_WORK(&fib_work->work, mlxsw_sp_router_fib_event_work);
+ fib_work->mlxsw_sp = mlxsw_sp;
+ fib_work->event = event;
+
+ switch (event) {
+ case FIB_EVENT_ENTRY_REPLACE: /* fall through */
+ case FIB_EVENT_ENTRY_APPEND: /* fall through */
+ case FIB_EVENT_ENTRY_ADD: /* fall through */
+ case FIB_EVENT_ENTRY_DEL:
+ memcpy(&fib_work->fen_info, ptr, sizeof(fib_work->fen_info));
+ /* Take referece on fib_info to prevent it from being
+ * freed while work is queued. Release it afterwards.
+ */
+ fib_info_hold(fib_work->fen_info.fi);
+ break;
+ case FIB_EVENT_RULE_ADD: /* fall through */
+ case FIB_EVENT_RULE_DEL:
+ memcpy(&fib_work->fr_info, ptr, sizeof(fib_work->fr_info));
+ fib_rule_get(fib_work->fr_info.rule);
+ break;
+ case FIB_EVENT_NH_ADD: /* fall through */
+ case FIB_EVENT_NH_DEL:
+ memcpy(&fib_work->fnh_info, ptr, sizeof(fib_work->fnh_info));
+ fib_info_hold(fib_work->fnh_info.fib_nh->nh_parent);
+ break;
+ }
+
+ mlxsw_core_schedule_work(&fib_work->work);
+
+ return NOTIFY_DONE;
+}
+
+static struct mlxsw_sp_rif *
+mlxsw_sp_rif_find_by_dev(const struct mlxsw_sp *mlxsw_sp,
+ const struct net_device *dev)
+{
+ int i;
+
+ for (i = 0; i < MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_RIFS); i++)
+ if (mlxsw_sp->rifs[i] && mlxsw_sp->rifs[i]->dev == dev)
+ return mlxsw_sp->rifs[i];
+
+ return NULL;
+}
+
static int mlxsw_sp_router_rif_disable(struct mlxsw_sp *mlxsw_sp, u16 rif)
{
char ritr_pl[MLXSW_REG_RITR_LEN];
@@ -2451,12 +2815,690 @@
return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ritr), ritr_pl);
}
-void mlxsw_sp_router_rif_gone_sync(struct mlxsw_sp *mlxsw_sp,
- struct mlxsw_sp_rif *r)
+static void mlxsw_sp_router_rif_gone_sync(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_rif *rif)
{
- mlxsw_sp_router_rif_disable(mlxsw_sp, r->rif);
- mlxsw_sp_nexthop_rif_gone_sync(mlxsw_sp, r);
- mlxsw_sp_neigh_rif_gone_sync(mlxsw_sp, r);
+ mlxsw_sp_router_rif_disable(mlxsw_sp, rif->rif_index);
+ mlxsw_sp_nexthop_rif_gone_sync(mlxsw_sp, rif);
+ mlxsw_sp_neigh_rif_gone_sync(mlxsw_sp, rif);
+}
+
+static bool mlxsw_sp_rif_should_config(struct mlxsw_sp_rif *rif,
+ const struct in_device *in_dev,
+ unsigned long event)
+{
+ switch (event) {
+ case NETDEV_UP:
+ if (!rif)
+ return true;
+ return false;
+ case NETDEV_DOWN:
+ if (rif && !in_dev->ifa_list &&
+ !netif_is_l3_slave(rif->dev))
+ return true;
+ /* It is possible we already removed the RIF ourselves
+ * if it was assigned to a netdev that is now a bridge
+ * or LAG slave.
+ */
+ return false;
+ }
+
+ return false;
+}
+
+#define MLXSW_SP_INVALID_INDEX_RIF 0xffff
+static int mlxsw_sp_avail_rif_get(struct mlxsw_sp *mlxsw_sp)
+{
+ int i;
+
+ for (i = 0; i < MLXSW_CORE_RES_GET(mlxsw_sp->core, MAX_RIFS); i++)
+ if (!mlxsw_sp->rifs[i])
+ return i;
+
+ return MLXSW_SP_INVALID_INDEX_RIF;
+}
+
+static void mlxsw_sp_vport_rif_sp_attr_get(struct mlxsw_sp_port *mlxsw_sp_vport,
+ bool *p_lagged, u16 *p_system_port)
+{
+ u8 local_port = mlxsw_sp_vport->local_port;
+
+ *p_lagged = mlxsw_sp_vport->lagged;
+ *p_system_port = *p_lagged ? mlxsw_sp_vport->lag_id : local_port;
+}
+
+static int mlxsw_sp_vport_rif_sp_op(struct mlxsw_sp_port *mlxsw_sp_vport,
+ u16 vr_id, struct net_device *l3_dev,
+ u16 rif_index, bool create)
+{
+ struct mlxsw_sp *mlxsw_sp = mlxsw_sp_vport->mlxsw_sp;
+ bool lagged = mlxsw_sp_vport->lagged;
+ char ritr_pl[MLXSW_REG_RITR_LEN];
+ u16 system_port;
+
+ mlxsw_reg_ritr_pack(ritr_pl, create, MLXSW_REG_RITR_SP_IF, rif_index,
+ vr_id, l3_dev->mtu, l3_dev->dev_addr);
+
+ mlxsw_sp_vport_rif_sp_attr_get(mlxsw_sp_vport, &lagged, &system_port);
+ mlxsw_reg_ritr_sp_if_pack(ritr_pl, lagged, system_port,
+ mlxsw_sp_vport_vid_get(mlxsw_sp_vport));
+
+ return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ritr), ritr_pl);
+}
+
+static void mlxsw_sp_vport_rif_sp_leave(struct mlxsw_sp_port *mlxsw_sp_vport);
+
+static u16 mlxsw_sp_rif_sp_to_fid(u16 rif_index)
+{
+ return MLXSW_SP_RFID_BASE + rif_index;
+}
+
+static struct mlxsw_sp_fid *
+mlxsw_sp_rfid_alloc(u16 fid, struct net_device *l3_dev)
+{
+ struct mlxsw_sp_fid *f;
+
+ f = kzalloc(sizeof(*f), GFP_KERNEL);
+ if (!f)
+ return NULL;
+
+ f->leave = mlxsw_sp_vport_rif_sp_leave;
+ f->ref_count = 0;
+ f->dev = l3_dev;
+ f->fid = fid;
+
+ return f;
+}
+
+static struct mlxsw_sp_rif *
+mlxsw_sp_rif_alloc(u16 rif_index, u16 vr_id, struct net_device *l3_dev,
+ struct mlxsw_sp_fid *f)
+{
+ struct mlxsw_sp_rif *rif;
+
+ rif = kzalloc(sizeof(*rif), GFP_KERNEL);
+ if (!rif)
+ return NULL;
+
+ INIT_LIST_HEAD(&rif->nexthop_list);
+ INIT_LIST_HEAD(&rif->neigh_list);
+ ether_addr_copy(rif->addr, l3_dev->dev_addr);
+ rif->mtu = l3_dev->mtu;
+ rif->vr_id = vr_id;
+ rif->dev = l3_dev;
+ rif->rif_index = rif_index;
+ rif->f = f;
+
+ return rif;
+}
+
+u16 mlxsw_sp_rif_index(const struct mlxsw_sp_rif *rif)
+{
+ return rif->rif_index;
+}
+
+int mlxsw_sp_rif_dev_ifindex(const struct mlxsw_sp_rif *rif)
+{
+ return rif->dev->ifindex;
+}
+
+static struct mlxsw_sp_rif *
+mlxsw_sp_vport_rif_sp_create(struct mlxsw_sp_port *mlxsw_sp_vport,
+ struct net_device *l3_dev)
+{
+ struct mlxsw_sp *mlxsw_sp = mlxsw_sp_vport->mlxsw_sp;
+ u32 tb_id = l3mdev_fib_table(l3_dev);
+ struct mlxsw_sp_vr *vr;
+ struct mlxsw_sp_fid *f;
+ struct mlxsw_sp_rif *rif;
+ u16 fid, rif_index;
+ int err;
+
+ rif_index = mlxsw_sp_avail_rif_get(mlxsw_sp);
+ if (rif_index == MLXSW_SP_INVALID_INDEX_RIF)
+ return ERR_PTR(-ERANGE);
+
+ vr = mlxsw_sp_vr_get(mlxsw_sp, tb_id ? : RT_TABLE_MAIN);
+ if (IS_ERR(vr))
+ return ERR_CAST(vr);
+
+ err = mlxsw_sp_vport_rif_sp_op(mlxsw_sp_vport, vr->id, l3_dev,
+ rif_index, true);
+ if (err)
+ goto err_vport_rif_sp_op;
+
+ fid = mlxsw_sp_rif_sp_to_fid(rif_index);
+ err = mlxsw_sp_rif_fdb_op(mlxsw_sp, l3_dev->dev_addr, fid, true);
+ if (err)
+ goto err_rif_fdb_op;
+
+ f = mlxsw_sp_rfid_alloc(fid, l3_dev);
+ if (!f) {
+ err = -ENOMEM;
+ goto err_rfid_alloc;
+ }
+
+ rif = mlxsw_sp_rif_alloc(rif_index, vr->id, l3_dev, f);
+ if (!rif) {
+ err = -ENOMEM;
+ goto err_rif_alloc;
+ }
+
+ if (devlink_dpipe_table_counter_enabled(priv_to_devlink(mlxsw_sp->core),
+ MLXSW_SP_DPIPE_TABLE_NAME_ERIF)) {
+ err = mlxsw_sp_rif_counter_alloc(mlxsw_sp, rif,
+ MLXSW_SP_RIF_COUNTER_EGRESS);
+ if (err)
+ netdev_dbg(mlxsw_sp_vport->dev,
+ "Counter alloc Failed err=%d\n", err);
+ }
+
+ f->rif = rif;
+ mlxsw_sp->rifs[rif_index] = rif;
+ vr->rif_count++;
+
+ return rif;
+
+err_rif_alloc:
+ kfree(f);
+err_rfid_alloc:
+ mlxsw_sp_rif_fdb_op(mlxsw_sp, l3_dev->dev_addr, fid, false);
+err_rif_fdb_op:
+ mlxsw_sp_vport_rif_sp_op(mlxsw_sp_vport, vr->id, l3_dev, rif_index,
+ false);
+err_vport_rif_sp_op:
+ mlxsw_sp_vr_put(vr);
+ return ERR_PTR(err);
+}
+
+static void mlxsw_sp_vport_rif_sp_destroy(struct mlxsw_sp_port *mlxsw_sp_vport,
+ struct mlxsw_sp_rif *rif)
+{
+ struct mlxsw_sp *mlxsw_sp = mlxsw_sp_vport->mlxsw_sp;
+ struct mlxsw_sp_vr *vr = &mlxsw_sp->router.vrs[rif->vr_id];
+ struct net_device *l3_dev = rif->dev;
+ struct mlxsw_sp_fid *f = rif->f;
+ u16 rif_index = rif->rif_index;
+ u16 fid = f->fid;
+
+ mlxsw_sp_router_rif_gone_sync(mlxsw_sp, rif);
+
+ mlxsw_sp_rif_counter_free(mlxsw_sp, rif, MLXSW_SP_RIF_COUNTER_EGRESS);
+ mlxsw_sp_rif_counter_free(mlxsw_sp, rif, MLXSW_SP_RIF_COUNTER_INGRESS);
+
+ vr->rif_count--;
+ mlxsw_sp->rifs[rif_index] = NULL;
+ f->rif = NULL;
+
+ kfree(rif);
+
+ kfree(f);
+
+ mlxsw_sp_rif_fdb_op(mlxsw_sp, l3_dev->dev_addr, fid, false);
+
+ mlxsw_sp_vport_rif_sp_op(mlxsw_sp_vport, vr->id, l3_dev, rif_index,
+ false);
+ mlxsw_sp_vr_put(vr);
+}
+
+static int mlxsw_sp_vport_rif_sp_join(struct mlxsw_sp_port *mlxsw_sp_vport,
+ struct net_device *l3_dev)
+{
+ struct mlxsw_sp *mlxsw_sp = mlxsw_sp_vport->mlxsw_sp;
+ struct mlxsw_sp_rif *rif;
+
+ rif = mlxsw_sp_rif_find_by_dev(mlxsw_sp, l3_dev);
+ if (!rif) {
+ rif = mlxsw_sp_vport_rif_sp_create(mlxsw_sp_vport, l3_dev);
+ if (IS_ERR(rif))
+ return PTR_ERR(rif);
+ }
+
+ mlxsw_sp_vport_fid_set(mlxsw_sp_vport, rif->f);
+ rif->f->ref_count++;
+
+ netdev_dbg(mlxsw_sp_vport->dev, "Joined FID=%d\n", rif->f->fid);
+
+ return 0;
+}
+
+static void mlxsw_sp_vport_rif_sp_leave(struct mlxsw_sp_port *mlxsw_sp_vport)
+{
+ struct mlxsw_sp_fid *f = mlxsw_sp_vport_fid_get(mlxsw_sp_vport);
+
+ netdev_dbg(mlxsw_sp_vport->dev, "Left FID=%d\n", f->fid);
+
+ mlxsw_sp_vport_fid_set(mlxsw_sp_vport, NULL);
+ if (--f->ref_count == 0)
+ mlxsw_sp_vport_rif_sp_destroy(mlxsw_sp_vport, f->rif);
+}
+
+static int mlxsw_sp_inetaddr_vport_event(struct net_device *l3_dev,
+ struct net_device *port_dev,
+ unsigned long event, u16 vid)
+{
+ struct mlxsw_sp_port *mlxsw_sp_port = netdev_priv(port_dev);
+ struct mlxsw_sp_port *mlxsw_sp_vport;
+
+ mlxsw_sp_vport = mlxsw_sp_port_vport_find(mlxsw_sp_port, vid);
+ if (WARN_ON(!mlxsw_sp_vport))
+ return -EINVAL;
+
+ switch (event) {
+ case NETDEV_UP:
+ return mlxsw_sp_vport_rif_sp_join(mlxsw_sp_vport, l3_dev);
+ case NETDEV_DOWN:
+ mlxsw_sp_vport_rif_sp_leave(mlxsw_sp_vport);
+ break;
+ }
+
+ return 0;
+}
+
+static int mlxsw_sp_inetaddr_port_event(struct net_device *port_dev,
+ unsigned long event)
+{
+ if (netif_is_bridge_port(port_dev) ||
+ netif_is_lag_port(port_dev) ||
+ netif_is_ovs_port(port_dev))
+ return 0;
+
+ return mlxsw_sp_inetaddr_vport_event(port_dev, port_dev, event, 1);
+}
+
+static int __mlxsw_sp_inetaddr_lag_event(struct net_device *l3_dev,
+ struct net_device *lag_dev,
+ unsigned long event, u16 vid)
+{
+ struct net_device *port_dev;
+ struct list_head *iter;
+ int err;
+
+ netdev_for_each_lower_dev(lag_dev, port_dev, iter) {
+ if (mlxsw_sp_port_dev_check(port_dev)) {
+ err = mlxsw_sp_inetaddr_vport_event(l3_dev, port_dev,
+ event, vid);
+ if (err)
+ return err;
+ }
+ }
+
+ return 0;
+}
+
+static int mlxsw_sp_inetaddr_lag_event(struct net_device *lag_dev,
+ unsigned long event)
+{
+ if (netif_is_bridge_port(lag_dev))
+ return 0;
+
+ return __mlxsw_sp_inetaddr_lag_event(lag_dev, lag_dev, event, 1);
+}
+
+static struct mlxsw_sp_fid *mlxsw_sp_bridge_fid_get(struct mlxsw_sp *mlxsw_sp,
+ struct net_device *l3_dev)
+{
+ u16 fid;
+
+ if (is_vlan_dev(l3_dev))
+ fid = vlan_dev_vlan_id(l3_dev);
+ else if (mlxsw_sp->master_bridge.dev == l3_dev)
+ fid = 1;
+ else
+ return mlxsw_sp_vfid_find(mlxsw_sp, l3_dev);
+
+ return mlxsw_sp_fid_find(mlxsw_sp, fid);
+}
+
+static u8 mlxsw_sp_router_port(const struct mlxsw_sp *mlxsw_sp)
+{
+ return mlxsw_core_max_ports(mlxsw_sp->core) + 1;
+}
+
+static enum mlxsw_flood_table_type mlxsw_sp_flood_table_type_get(u16 fid)
+{
+ return mlxsw_sp_fid_is_vfid(fid) ? MLXSW_REG_SFGC_TABLE_TYPE_FID :
+ MLXSW_REG_SFGC_TABLE_TYPE_FID_OFFEST;
+}
+
+static u16 mlxsw_sp_flood_table_index_get(u16 fid)
+{
+ return mlxsw_sp_fid_is_vfid(fid) ? mlxsw_sp_fid_to_vfid(fid) : fid;
+}
+
+static int mlxsw_sp_router_port_flood_set(struct mlxsw_sp *mlxsw_sp, u16 fid,
+ bool set)
+{
+ u8 router_port = mlxsw_sp_router_port(mlxsw_sp);
+ enum mlxsw_flood_table_type table_type;
+ char *sftr_pl;
+ u16 index;
+ int err;
+
+ sftr_pl = kmalloc(MLXSW_REG_SFTR_LEN, GFP_KERNEL);
+ if (!sftr_pl)
+ return -ENOMEM;
+
+ table_type = mlxsw_sp_flood_table_type_get(fid);
+ index = mlxsw_sp_flood_table_index_get(fid);
+ mlxsw_reg_sftr_pack(sftr_pl, MLXSW_SP_FLOOD_TABLE_BC, index, table_type,
+ 1, router_port, set);
+ err = mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(sftr), sftr_pl);
+
+ kfree(sftr_pl);
+ return err;
+}
+
+static enum mlxsw_reg_ritr_if_type mlxsw_sp_rif_type_get(u16 fid)
+{
+ if (mlxsw_sp_fid_is_vfid(fid))
+ return MLXSW_REG_RITR_FID_IF;
+ else
+ return MLXSW_REG_RITR_VLAN_IF;
+}
+
+static int mlxsw_sp_rif_bridge_op(struct mlxsw_sp *mlxsw_sp, u16 vr_id,
+ struct net_device *l3_dev,
+ u16 fid, u16 rif,
+ bool create)
+{
+ enum mlxsw_reg_ritr_if_type rif_type;
+ char ritr_pl[MLXSW_REG_RITR_LEN];
+
+ rif_type = mlxsw_sp_rif_type_get(fid);
+ mlxsw_reg_ritr_pack(ritr_pl, create, rif_type, rif, vr_id, l3_dev->mtu,
+ l3_dev->dev_addr);
+ mlxsw_reg_ritr_fid_set(ritr_pl, rif_type, fid);
+
+ return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ritr), ritr_pl);
+}
+
+static int mlxsw_sp_rif_bridge_create(struct mlxsw_sp *mlxsw_sp,
+ struct net_device *l3_dev,
+ struct mlxsw_sp_fid *f)
+{
+ u32 tb_id = l3mdev_fib_table(l3_dev);
+ struct mlxsw_sp_rif *rif;
+ struct mlxsw_sp_vr *vr;
+ u16 rif_index;
+ int err;
+
+ rif_index = mlxsw_sp_avail_rif_get(mlxsw_sp);
+ if (rif_index == MLXSW_SP_INVALID_INDEX_RIF)
+ return -ERANGE;
+
+ vr = mlxsw_sp_vr_get(mlxsw_sp, tb_id ? : RT_TABLE_MAIN);
+ if (IS_ERR(vr))
+ return PTR_ERR(vr);
+
+ err = mlxsw_sp_router_port_flood_set(mlxsw_sp, f->fid, true);
+ if (err)
+ goto err_port_flood_set;
+
+ err = mlxsw_sp_rif_bridge_op(mlxsw_sp, vr->id, l3_dev, f->fid,
+ rif_index, true);
+ if (err)
+ goto err_rif_bridge_op;
+
+ err = mlxsw_sp_rif_fdb_op(mlxsw_sp, l3_dev->dev_addr, f->fid, true);
+ if (err)
+ goto err_rif_fdb_op;
+
+ rif = mlxsw_sp_rif_alloc(rif_index, vr->id, l3_dev, f);
+ if (!rif) {
+ err = -ENOMEM;
+ goto err_rif_alloc;
+ }
+
+ f->rif = rif;
+ mlxsw_sp->rifs[rif_index] = rif;
+ vr->rif_count++;
+
+ netdev_dbg(l3_dev, "RIF=%d created\n", rif_index);
+
+ return 0;
+
+err_rif_alloc:
+ mlxsw_sp_rif_fdb_op(mlxsw_sp, l3_dev->dev_addr, f->fid, false);
+err_rif_fdb_op:
+ mlxsw_sp_rif_bridge_op(mlxsw_sp, vr->id, l3_dev, f->fid, rif_index,
+ false);
+err_rif_bridge_op:
+ mlxsw_sp_router_port_flood_set(mlxsw_sp, f->fid, false);
+err_port_flood_set:
+ mlxsw_sp_vr_put(vr);
+ return err;
+}
+
+void mlxsw_sp_rif_bridge_destroy(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_rif *rif)
+{
+ struct mlxsw_sp_vr *vr = &mlxsw_sp->router.vrs[rif->vr_id];
+ struct net_device *l3_dev = rif->dev;
+ struct mlxsw_sp_fid *f = rif->f;
+ u16 rif_index = rif->rif_index;
+
+ mlxsw_sp_router_rif_gone_sync(mlxsw_sp, rif);
+
+ vr->rif_count--;
+ mlxsw_sp->rifs[rif_index] = NULL;
+ f->rif = NULL;
+
+ kfree(rif);
+
+ mlxsw_sp_rif_fdb_op(mlxsw_sp, l3_dev->dev_addr, f->fid, false);
+
+ mlxsw_sp_rif_bridge_op(mlxsw_sp, vr->id, l3_dev, f->fid, rif_index,
+ false);
+
+ mlxsw_sp_router_port_flood_set(mlxsw_sp, f->fid, false);
+
+ mlxsw_sp_vr_put(vr);
+
+ netdev_dbg(l3_dev, "RIF=%d destroyed\n", rif_index);
+}
+
+static int mlxsw_sp_inetaddr_bridge_event(struct net_device *l3_dev,
+ struct net_device *br_dev,
+ unsigned long event)
+{
+ struct mlxsw_sp *mlxsw_sp = mlxsw_sp_lower_get(l3_dev);
+ struct mlxsw_sp_fid *f;
+
+ /* FID can either be an actual FID if the L3 device is the
+ * VLAN-aware bridge or a VLAN device on top. Otherwise, the
+ * L3 device is a VLAN-unaware bridge and we get a vFID.
+ */
+ f = mlxsw_sp_bridge_fid_get(mlxsw_sp, l3_dev);
+ if (WARN_ON(!f))
+ return -EINVAL;
+
+ switch (event) {
+ case NETDEV_UP:
+ return mlxsw_sp_rif_bridge_create(mlxsw_sp, l3_dev, f);
+ case NETDEV_DOWN:
+ mlxsw_sp_rif_bridge_destroy(mlxsw_sp, f->rif);
+ break;
+ }
+
+ return 0;
+}
+
+static int mlxsw_sp_inetaddr_vlan_event(struct net_device *vlan_dev,
+ unsigned long event)
+{
+ struct net_device *real_dev = vlan_dev_real_dev(vlan_dev);
+ struct mlxsw_sp *mlxsw_sp = mlxsw_sp_lower_get(vlan_dev);
+ u16 vid = vlan_dev_vlan_id(vlan_dev);
+
+ if (mlxsw_sp_port_dev_check(real_dev))
+ return mlxsw_sp_inetaddr_vport_event(vlan_dev, real_dev, event,
+ vid);
+ else if (netif_is_lag_master(real_dev))
+ return __mlxsw_sp_inetaddr_lag_event(vlan_dev, real_dev, event,
+ vid);
+ else if (netif_is_bridge_master(real_dev) &&
+ mlxsw_sp->master_bridge.dev == real_dev)
+ return mlxsw_sp_inetaddr_bridge_event(vlan_dev, real_dev,
+ event);
+
+ return 0;
+}
+
+static int __mlxsw_sp_inetaddr_event(struct net_device *dev,
+ unsigned long event)
+{
+ if (mlxsw_sp_port_dev_check(dev))
+ return mlxsw_sp_inetaddr_port_event(dev, event);
+ else if (netif_is_lag_master(dev))
+ return mlxsw_sp_inetaddr_lag_event(dev, event);
+ else if (netif_is_bridge_master(dev))
+ return mlxsw_sp_inetaddr_bridge_event(dev, dev, event);
+ else if (is_vlan_dev(dev))
+ return mlxsw_sp_inetaddr_vlan_event(dev, event);
+ else
+ return 0;
+}
+
+int mlxsw_sp_inetaddr_event(struct notifier_block *unused,
+ unsigned long event, void *ptr)
+{
+ struct in_ifaddr *ifa = (struct in_ifaddr *) ptr;
+ struct net_device *dev = ifa->ifa_dev->dev;
+ struct mlxsw_sp *mlxsw_sp;
+ struct mlxsw_sp_rif *rif;
+ int err = 0;
+
+ mlxsw_sp = mlxsw_sp_lower_get(dev);
+ if (!mlxsw_sp)
+ goto out;
+
+ rif = mlxsw_sp_rif_find_by_dev(mlxsw_sp, dev);
+ if (!mlxsw_sp_rif_should_config(rif, ifa->ifa_dev, event))
+ goto out;
+
+ err = __mlxsw_sp_inetaddr_event(dev, event);
+out:
+ return notifier_from_errno(err);
+}
+
+static int mlxsw_sp_rif_edit(struct mlxsw_sp *mlxsw_sp, u16 rif_index,
+ const char *mac, int mtu)
+{
+ char ritr_pl[MLXSW_REG_RITR_LEN];
+ int err;
+
+ mlxsw_reg_ritr_rif_pack(ritr_pl, rif_index);
+ err = mlxsw_reg_query(mlxsw_sp->core, MLXSW_REG(ritr), ritr_pl);
+ if (err)
+ return err;
+
+ mlxsw_reg_ritr_mtu_set(ritr_pl, mtu);
+ mlxsw_reg_ritr_if_mac_memcpy_to(ritr_pl, mac);
+ mlxsw_reg_ritr_op_set(ritr_pl, MLXSW_REG_RITR_RIF_CREATE);
+ return mlxsw_reg_write(mlxsw_sp->core, MLXSW_REG(ritr), ritr_pl);
+}
+
+int mlxsw_sp_netdevice_router_port_event(struct net_device *dev)
+{
+ struct mlxsw_sp *mlxsw_sp;
+ struct mlxsw_sp_rif *rif;
+ int err;
+
+ mlxsw_sp = mlxsw_sp_lower_get(dev);
+ if (!mlxsw_sp)
+ return 0;
+
+ rif = mlxsw_sp_rif_find_by_dev(mlxsw_sp, dev);
+ if (!rif)
+ return 0;
+
+ err = mlxsw_sp_rif_fdb_op(mlxsw_sp, rif->addr, rif->f->fid, false);
+ if (err)
+ return err;
+
+ err = mlxsw_sp_rif_edit(mlxsw_sp, rif->rif_index, dev->dev_addr,
+ dev->mtu);
+ if (err)
+ goto err_rif_edit;
+
+ err = mlxsw_sp_rif_fdb_op(mlxsw_sp, dev->dev_addr, rif->f->fid, true);
+ if (err)
+ goto err_rif_fdb_op;
+
+ ether_addr_copy(rif->addr, dev->dev_addr);
+ rif->mtu = dev->mtu;
+
+ netdev_dbg(dev, "Updated RIF=%d\n", rif->rif_index);
+
+ return 0;
+
+err_rif_fdb_op:
+ mlxsw_sp_rif_edit(mlxsw_sp, rif->rif_index, rif->addr, rif->mtu);
+err_rif_edit:
+ mlxsw_sp_rif_fdb_op(mlxsw_sp, rif->addr, rif->f->fid, true);
+ return err;
+}
+
+static int mlxsw_sp_port_vrf_join(struct mlxsw_sp *mlxsw_sp,
+ struct net_device *l3_dev)
+{
+ struct mlxsw_sp_rif *rif;
+
+ /* If netdev is already associated with a RIF, then we need to
+ * destroy it and create a new one with the new virtual router ID.
+ */
+ rif = mlxsw_sp_rif_find_by_dev(mlxsw_sp, l3_dev);
+ if (rif)
+ __mlxsw_sp_inetaddr_event(l3_dev, NETDEV_DOWN);
+
+ return __mlxsw_sp_inetaddr_event(l3_dev, NETDEV_UP);
+}
+
+static void mlxsw_sp_port_vrf_leave(struct mlxsw_sp *mlxsw_sp,
+ struct net_device *l3_dev)
+{
+ struct mlxsw_sp_rif *rif;
+
+ rif = mlxsw_sp_rif_find_by_dev(mlxsw_sp, l3_dev);
+ if (!rif)
+ return;
+ __mlxsw_sp_inetaddr_event(l3_dev, NETDEV_DOWN);
+}
+
+int mlxsw_sp_netdevice_vrf_event(struct net_device *l3_dev, unsigned long event,
+ struct netdev_notifier_changeupper_info *info)
+{
+ struct mlxsw_sp *mlxsw_sp = mlxsw_sp_lower_get(l3_dev);
+ int err = 0;
+
+ if (!mlxsw_sp)
+ return 0;
+
+ switch (event) {
+ case NETDEV_PRECHANGEUPPER:
+ return 0;
+ case NETDEV_CHANGEUPPER:
+ if (info->linking)
+ err = mlxsw_sp_port_vrf_join(mlxsw_sp, l3_dev);
+ else
+ mlxsw_sp_port_vrf_leave(mlxsw_sp, l3_dev);
+ break;
+ }
+
+ return err;
+}
+
+static void mlxsw_sp_router_fib_dump_flush(struct notifier_block *nb)
+{
+ struct mlxsw_sp *mlxsw_sp = container_of(nb, struct mlxsw_sp, fib_nb);
+
+ /* Flush pending FIB notifications and then flush the device's
+ * table before requesting another dump. The FIB notification
+ * block is unregistered, so no need to take RTNL.
+ */
+ mlxsw_core_flush_owq();
+ mlxsw_sp_router_fib_flush(mlxsw_sp);
}
static int __mlxsw_sp_router_init(struct mlxsw_sp *mlxsw_sp)
@@ -2501,111 +3543,6 @@
kfree(mlxsw_sp->rifs);
}
-struct mlxsw_sp_fib_event_work {
- struct work_struct work;
- union {
- struct fib_entry_notifier_info fen_info;
- struct fib_nh_notifier_info fnh_info;
- };
- struct mlxsw_sp *mlxsw_sp;
- unsigned long event;
-};
-
-static void mlxsw_sp_router_fib_event_work(struct work_struct *work)
-{
- struct mlxsw_sp_fib_event_work *fib_work =
- container_of(work, struct mlxsw_sp_fib_event_work, work);
- struct mlxsw_sp *mlxsw_sp = fib_work->mlxsw_sp;
- bool replace, append;
- int err;
-
- /* Protect internal structures from changes */
- rtnl_lock();
- switch (fib_work->event) {
- case FIB_EVENT_ENTRY_REPLACE: /* fall through */
- case FIB_EVENT_ENTRY_APPEND: /* fall through */
- case FIB_EVENT_ENTRY_ADD:
- replace = fib_work->event == FIB_EVENT_ENTRY_REPLACE;
- append = fib_work->event == FIB_EVENT_ENTRY_APPEND;
- err = mlxsw_sp_router_fib4_add(mlxsw_sp, &fib_work->fen_info,
- replace, append);
- if (err)
- mlxsw_sp_router_fib4_abort(mlxsw_sp);
- fib_info_put(fib_work->fen_info.fi);
- break;
- case FIB_EVENT_ENTRY_DEL:
- mlxsw_sp_router_fib4_del(mlxsw_sp, &fib_work->fen_info);
- fib_info_put(fib_work->fen_info.fi);
- break;
- case FIB_EVENT_RULE_ADD: /* fall through */
- case FIB_EVENT_RULE_DEL:
- mlxsw_sp_router_fib4_abort(mlxsw_sp);
- break;
- case FIB_EVENT_NH_ADD: /* fall through */
- case FIB_EVENT_NH_DEL:
- mlxsw_sp_nexthop_event(mlxsw_sp, fib_work->event,
- fib_work->fnh_info.fib_nh);
- fib_info_put(fib_work->fnh_info.fib_nh->nh_parent);
- break;
- }
- rtnl_unlock();
- kfree(fib_work);
-}
-
-/* Called with rcu_read_lock() */
-static int mlxsw_sp_router_fib_event(struct notifier_block *nb,
- unsigned long event, void *ptr)
-{
- struct mlxsw_sp *mlxsw_sp = container_of(nb, struct mlxsw_sp, fib_nb);
- struct mlxsw_sp_fib_event_work *fib_work;
- struct fib_notifier_info *info = ptr;
-
- if (!net_eq(info->net, &init_net))
- return NOTIFY_DONE;
-
- fib_work = kzalloc(sizeof(*fib_work), GFP_ATOMIC);
- if (WARN_ON(!fib_work))
- return NOTIFY_BAD;
-
- INIT_WORK(&fib_work->work, mlxsw_sp_router_fib_event_work);
- fib_work->mlxsw_sp = mlxsw_sp;
- fib_work->event = event;
-
- switch (event) {
- case FIB_EVENT_ENTRY_REPLACE: /* fall through */
- case FIB_EVENT_ENTRY_APPEND: /* fall through */
- case FIB_EVENT_ENTRY_ADD: /* fall through */
- case FIB_EVENT_ENTRY_DEL:
- memcpy(&fib_work->fen_info, ptr, sizeof(fib_work->fen_info));
- /* Take referece on fib_info to prevent it from being
- * freed while work is queued. Release it afterwards.
- */
- fib_info_hold(fib_work->fen_info.fi);
- break;
- case FIB_EVENT_NH_ADD: /* fall through */
- case FIB_EVENT_NH_DEL:
- memcpy(&fib_work->fnh_info, ptr, sizeof(fib_work->fnh_info));
- fib_info_hold(fib_work->fnh_info.fib_nh->nh_parent);
- break;
- }
-
- mlxsw_core_schedule_work(&fib_work->work);
-
- return NOTIFY_DONE;
-}
-
-static void mlxsw_sp_router_fib_dump_flush(struct notifier_block *nb)
-{
- struct mlxsw_sp *mlxsw_sp = container_of(nb, struct mlxsw_sp, fib_nb);
-
- /* Flush pending FIB notifications and then flush the device's
- * table before requesting another dump. The FIB notification
- * block is unregistered, so no need to take RTNL.
- */
- mlxsw_core_flush_owq();
- mlxsw_sp_router_fib_flush(mlxsw_sp);
-}
-
int mlxsw_sp_router_init(struct mlxsw_sp *mlxsw_sp)
{
int err;
@@ -2625,7 +3562,10 @@
if (err)
goto err_nexthop_group_ht_init;
- mlxsw_sp_lpm_init(mlxsw_sp);
+ err = mlxsw_sp_lpm_init(mlxsw_sp);
+ if (err)
+ goto err_lpm_init;
+
err = mlxsw_sp_vrs_init(mlxsw_sp);
if (err)
goto err_vrs_init;
@@ -2647,6 +3587,8 @@
err_neigh_init:
mlxsw_sp_vrs_fini(mlxsw_sp);
err_vrs_init:
+ mlxsw_sp_lpm_fini(mlxsw_sp);
+err_lpm_init:
rhashtable_destroy(&mlxsw_sp->router.nexthop_group_ht);
err_nexthop_group_ht_init:
rhashtable_destroy(&mlxsw_sp->router.nexthop_ht);
@@ -2660,6 +3602,7 @@
unregister_fib_notifier(&mlxsw_sp->fib_nb);
mlxsw_sp_neigh_fini(mlxsw_sp);
mlxsw_sp_vrs_fini(mlxsw_sp);
+ mlxsw_sp_lpm_fini(mlxsw_sp);
rhashtable_destroy(&mlxsw_sp->router.nexthop_group_ht);
rhashtable_destroy(&mlxsw_sp->router.nexthop_ht);
__mlxsw_sp_router_fini(mlxsw_sp);
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.h
new file mode 100644
index 0000000..c3095fe
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_router.h
@@ -0,0 +1,58 @@
+/*
+ * drivers/net/ethernet/mellanox/mlxsw/spectrum_router.h
+ * Copyright (c) 2017 Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2017 Arkadi Sharshevsky <arkadis@mellanox.com>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ * 3. Neither the names of the copyright holders nor the names of its
+ * contributors may be used to endorse or promote products derived from
+ * this software without specific prior written permission.
+ *
+ * Alternatively, this software may be distributed under the terms of the
+ * GNU General Public License ("GPL") version 2 as published by the Free
+ * Software Foundation.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef _MLXSW_ROUTER_H_
+#define _MLXSW_ROUTER_H_
+
+#include "spectrum.h"
+
+enum mlxsw_sp_rif_counter_dir {
+ MLXSW_SP_RIF_COUNTER_INGRESS,
+ MLXSW_SP_RIF_COUNTER_EGRESS,
+};
+
+u16 mlxsw_sp_rif_index(const struct mlxsw_sp_rif *rif);
+int mlxsw_sp_rif_dev_ifindex(const struct mlxsw_sp_rif *rif);
+int mlxsw_sp_rif_counter_value_get(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_rif *rif,
+ enum mlxsw_sp_rif_counter_dir dir,
+ u64 *cnt);
+void mlxsw_sp_rif_counter_free(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_rif *rif,
+ enum mlxsw_sp_rif_counter_dir dir);
+int mlxsw_sp_rif_counter_alloc(struct mlxsw_sp *mlxsw_sp,
+ struct mlxsw_sp_rif *rif,
+ enum mlxsw_sp_rif_counter_dir dir);
+
+#endif /* _MLXSW_ROUTER_H_*/
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_switchdev.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_switchdev.c
index 598727d..0d8411f 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_switchdev.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_switchdev.c
@@ -568,8 +568,8 @@
list_del(&f->list);
- if (f->r)
- mlxsw_sp_rif_bridge_destroy(mlxsw_sp, f->r);
+ if (f->rif)
+ mlxsw_sp_rif_bridge_destroy(mlxsw_sp, f->rif);
kfree(f);
@@ -745,27 +745,6 @@
return err;
}
-static int __mlxsw_sp_port_vlans_set(struct mlxsw_sp_port *mlxsw_sp_port,
- u16 vid_begin, u16 vid_end, bool is_member,
- bool untagged)
-{
- u16 vid, vid_e;
- int err;
-
- for (vid = vid_begin; vid <= vid_end;
- vid += MLXSW_REG_SPVM_REC_MAX_COUNT) {
- vid_e = min((u16) (vid + MLXSW_REG_SPVM_REC_MAX_COUNT - 1),
- vid_end);
-
- err = mlxsw_sp_port_vlan_set(mlxsw_sp_port, vid, vid_e,
- is_member, untagged);
- if (err)
- return err;
- }
-
- return 0;
-}
-
static int mlxsw_sp_port_vid_learning_set(struct mlxsw_sp_port *mlxsw_sp_port,
u16 vid_begin, u16 vid_end,
bool learn_enable)
@@ -804,8 +783,8 @@
return err;
}
- err = __mlxsw_sp_port_vlans_set(mlxsw_sp_port, vid_begin, vid_end,
- true, flag_untagged);
+ err = mlxsw_sp_port_vlan_set(mlxsw_sp_port, vid_begin, vid_end,
+ true, flag_untagged);
if (err) {
netdev_err(dev, "Unable to add VIDs %d-%d\n", vid_begin,
vid_end);
@@ -863,8 +842,8 @@
if (old_pvid != mlxsw_sp_port->pvid)
mlxsw_sp_port_pvid_set(mlxsw_sp_port, old_pvid);
err_port_pvid_set:
- __mlxsw_sp_port_vlans_set(mlxsw_sp_port, vid_begin, vid_end, false,
- false);
+ mlxsw_sp_port_vlan_set(mlxsw_sp_port, vid_begin, vid_end,
+ false, false);
err_port_vlans_set:
mlxsw_sp_port_fid_leave(mlxsw_sp_port, vid_begin, vid_end);
return err;
@@ -1012,7 +991,7 @@
mlxsw_reg_smid_pack(smid_pl, mid, mlxsw_sp_port->local_port, add);
if (clear_all_ports) {
- for (i = 1; i < MLXSW_PORT_MAX_PORTS; i++)
+ for (i = 1; i < mlxsw_core_max_ports(mlxsw_sp->core); i++)
if (mlxsw_sp->ports[i])
mlxsw_reg_smid_port_mask_set(smid_pl, i, 1);
}
@@ -1171,8 +1150,8 @@
if (pvid >= vid_begin && pvid <= vid_end)
mlxsw_sp_port_pvid_set(mlxsw_sp_port, 0);
- __mlxsw_sp_port_vlans_set(mlxsw_sp_port, vid_begin, vid_end, false,
- false);
+ mlxsw_sp_port_vlan_set(mlxsw_sp_port, vid_begin, vid_end,
+ false, false);
mlxsw_sp_port_fid_leave(mlxsw_sp_port, vid_begin, vid_end);
diff --git a/drivers/net/ethernet/mellanox/mlxsw/switchx2.c b/drivers/net/ethernet/mellanox/mlxsw/switchx2.c
index ec1e886..3b0f724 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/switchx2.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/switchx2.c
@@ -1321,7 +1321,7 @@
{
int i;
- for (i = 1; i < MLXSW_PORT_MAX_PORTS; i++)
+ for (i = 1; i < mlxsw_core_max_ports(mlxsw_sx->core); i++)
if (mlxsw_sx_port_created(mlxsw_sx, i))
mlxsw_sx_port_remove(mlxsw_sx, i);
kfree(mlxsw_sx->ports);
@@ -1329,17 +1329,18 @@
static int mlxsw_sx_ports_create(struct mlxsw_sx *mlxsw_sx)
{
+ unsigned int max_ports = mlxsw_core_max_ports(mlxsw_sx->core);
size_t alloc_size;
u8 module, width;
int i;
int err;
- alloc_size = sizeof(struct mlxsw_sx_port *) * MLXSW_PORT_MAX_PORTS;
+ alloc_size = sizeof(struct mlxsw_sx_port *) * max_ports;
mlxsw_sx->ports = kzalloc(alloc_size, GFP_KERNEL);
if (!mlxsw_sx->ports)
return -ENOMEM;
- for (i = 1; i < MLXSW_PORT_MAX_PORTS; i++) {
+ for (i = 1; i < max_ports; i++) {
err = mlxsw_sx_port_module_info_get(mlxsw_sx, i, &module,
&width);
if (err)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/trap.h b/drivers/net/ethernet/mellanox/mlxsw/trap.h
index 02ea48b..e008fdb 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/trap.h
+++ b/drivers/net/ethernet/mellanox/mlxsw/trap.h
@@ -55,6 +55,7 @@
MLXSW_TRAP_ID_IGMP_V2_LEAVE = 0x33,
MLXSW_TRAP_ID_IGMP_V3_REPORT = 0x34,
MLXSW_TRAP_ID_PKT_SAMPLE = 0x38,
+ MLXSW_TRAP_ID_FID_MISS = 0x3D,
MLXSW_TRAP_ID_ARPBC = 0x50,
MLXSW_TRAP_ID_ARPUC = 0x51,
MLXSW_TRAP_ID_MTUERROR = 0x52,
diff --git a/drivers/net/ethernet/micrel/ks8851.c b/drivers/net/ethernet/micrel/ks8851.c
index 279ee46..20358f8 100644
--- a/drivers/net/ethernet/micrel/ks8851.c
+++ b/drivers/net/ethernet/micrel/ks8851.c
@@ -212,25 +212,6 @@
}
/**
- * ks8851_rx_1msg - select whether to use one or two messages for spi read
- * @ks: The device structure
- *
- * Return whether to generate a single message with a tx and rx buffer
- * supplied to spi_sync(), or alternatively send the tx and rx buffers
- * as separate messages.
- *
- * Depending on the hardware in use, a single message may be more efficient
- * on interrupts or work done by the driver.
- *
- * This currently always returns true until we add some per-device data passed
- * from the platform code to specify which mode is better.
- */
-static inline bool ks8851_rx_1msg(struct ks8851_net *ks)
-{
- return true;
-}
-
-/**
* ks8851_rdreg - issue read register command and return the data
* @ks: The device state
* @op: The register address and byte enables in message format.
@@ -251,14 +232,7 @@
txb[0] = cpu_to_le16(op | KS_SPIOP_RD);
- if (ks8851_rx_1msg(ks)) {
- msg = &ks->spi_msg1;
- xfer = &ks->spi_xfer1;
-
- xfer->tx_buf = txb;
- xfer->rx_buf = trx;
- xfer->len = rxl + 2;
- } else {
+ if (ks->spidev->master->flags & SPI_MASTER_HALF_DUPLEX) {
msg = &ks->spi_msg2;
xfer = ks->spi_xfer2;
@@ -270,15 +244,22 @@
xfer->tx_buf = NULL;
xfer->rx_buf = trx;
xfer->len = rxl;
+ } else {
+ msg = &ks->spi_msg1;
+ xfer = &ks->spi_xfer1;
+
+ xfer->tx_buf = txb;
+ xfer->rx_buf = trx;
+ xfer->len = rxl + 2;
}
ret = spi_sync(ks->spidev, msg);
if (ret < 0)
netdev_err(ks->netdev, "read: spi_sync() failed\n");
- else if (ks8851_rx_1msg(ks))
- memcpy(rxb, trx + 2, rxl);
- else
+ else if (ks->spidev->master->flags & SPI_MASTER_HALF_DUPLEX)
memcpy(rxb, trx, rxl);
+ else
+ memcpy(rxb, trx + 2, rxl);
}
/**
diff --git a/drivers/net/ethernet/moxa/moxart_ether.c b/drivers/net/ethernet/moxa/moxart_ether.c
index 6ad44be..c0d7d5e 100644
--- a/drivers/net/ethernet/moxa/moxart_ether.c
+++ b/drivers/net/ethernet/moxa/moxart_ether.c
@@ -228,8 +228,8 @@
if (desc0 & (RX_DESC0_ERR | RX_DESC0_CRC_ERR | RX_DESC0_FTL |
RX_DESC0_RUNT | RX_DESC0_ODD_NB)) {
net_dbg_ratelimited("packet error\n");
- priv->stats.rx_dropped++;
- priv->stats.rx_errors++;
+ ndev->stats.rx_dropped++;
+ ndev->stats.rx_errors++;
goto rx_next;
}
@@ -245,8 +245,8 @@
if (unlikely(!skb)) {
net_dbg_ratelimited("netdev_alloc_skb_ip_align failed\n");
- priv->stats.rx_dropped++;
- priv->stats.rx_errors++;
+ ndev->stats.rx_dropped++;
+ ndev->stats.rx_errors++;
goto rx_next;
}
@@ -256,10 +256,10 @@
napi_gro_receive(&priv->napi, skb);
rx++;
- priv->stats.rx_packets++;
- priv->stats.rx_bytes += len;
+ ndev->stats.rx_packets++;
+ ndev->stats.rx_bytes += len;
if (desc0 & RX_DESC0_MULTICAST)
- priv->stats.multicast++;
+ ndev->stats.multicast++;
rx_next:
wmb(); /* prevent setting ownership back too early */
@@ -296,8 +296,8 @@
dma_unmap_single(&ndev->dev, priv->tx_mapping[tx_tail],
priv->tx_len[tx_tail], DMA_TO_DEVICE);
- priv->stats.tx_packets++;
- priv->stats.tx_bytes += priv->tx_skb[tx_tail]->len;
+ ndev->stats.tx_packets++;
+ ndev->stats.tx_bytes += priv->tx_skb[tx_tail]->len;
dev_kfree_skb_irq(priv->tx_skb[tx_tail]);
priv->tx_skb[tx_tail] = NULL;
@@ -349,7 +349,7 @@
if (moxart_desc_read(desc + TX_REG_OFFSET_DESC0) & TX_DESC0_DMA_OWN) {
net_dbg_ratelimited("no TX space for packet\n");
- priv->stats.tx_dropped++;
+ ndev->stats.tx_dropped++;
goto out_unlock;
}
rmb(); /* ensure data is only read that had TX_DESC0_DMA_OWN cleared */
@@ -400,13 +400,6 @@
return ret;
}
-static struct net_device_stats *moxart_mac_get_stats(struct net_device *ndev)
-{
- struct moxart_mac_priv_t *priv = netdev_priv(ndev);
-
- return &priv->stats;
-}
-
static void moxart_mac_setmulticast(struct net_device *ndev)
{
struct moxart_mac_priv_t *priv = netdev_priv(ndev);
@@ -456,7 +449,6 @@
.ndo_open = moxart_mac_open,
.ndo_stop = moxart_mac_stop,
.ndo_start_xmit = moxart_mac_start_xmit,
- .ndo_get_stats = moxart_mac_get_stats,
.ndo_set_rx_mode = moxart_mac_set_rx_mode,
.ndo_set_mac_address = moxart_set_mac_address,
.ndo_validate_addr = eth_validate_addr,
diff --git a/drivers/net/ethernet/moxa/moxart_ether.h b/drivers/net/ethernet/moxa/moxart_ether.h
index afc32ec..686b895 100644
--- a/drivers/net/ethernet/moxa/moxart_ether.h
+++ b/drivers/net/ethernet/moxa/moxart_ether.h
@@ -293,7 +293,6 @@
struct moxart_mac_priv_t {
void __iomem *base;
- struct net_device_stats stats;
unsigned int reg_maccr;
unsigned int reg_imr;
struct napi_struct napi;
diff --git a/drivers/net/ethernet/netronome/nfp/Makefile b/drivers/net/ethernet/netronome/nfp/Makefile
index 6933afa..4b15f0f 100644
--- a/drivers/net/ethernet/netronome/nfp/Makefile
+++ b/drivers/net/ethernet/netronome/nfp/Makefile
@@ -6,8 +6,10 @@
nfpcore/nfp_cpplib.o \
nfpcore/nfp_hwinfo.o \
nfpcore/nfp_mip.o \
+ nfpcore/nfp_mutex.o \
nfpcore/nfp_nffw.o \
nfpcore/nfp_nsp.o \
+ nfpcore/nfp_nsp_cmds.o \
nfpcore/nfp_nsp_eth.o \
nfpcore/nfp_resource.o \
nfpcore/nfp_rtsym.o \
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_bpf_jit.c b/drivers/net/ethernet/netronome/nfp/nfp_bpf_jit.c
index 335beb8..97a8f00 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_bpf_jit.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_bpf_jit.c
@@ -798,7 +798,7 @@
const struct bpf_insn *insn = &meta->insn;
if (insn->off < 0) /* TODO */
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
wrp_test_reg_one(nfp_prog, insn->dst_reg * 2, alu_op,
insn->src_reg * 2, br_mask, insn->off);
@@ -818,7 +818,7 @@
u32 tmp_reg;
if (insn->off < 0) /* TODO */
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
tmp_reg = ur_load_imm_any(nfp_prog, imm & ~0U, imm_b(nfp_prog));
if (!swap)
@@ -847,7 +847,7 @@
u8 areg = insn->src_reg * 2, breg = insn->dst_reg * 2;
if (insn->off < 0) /* TODO */
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
if (swap) {
areg ^= breg;
@@ -1132,7 +1132,7 @@
emit_alu(nfp_prog, reg_both(meta->insn.dst_reg * 2),
reg_none(), ALU_OP_NONE, NFP_BPF_ABI_LEN);
else
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
return 0;
}
@@ -1143,7 +1143,7 @@
if (meta->insn.off != offsetof(struct xdp_md, data) &&
meta->insn.off != offsetof(struct xdp_md, data_end))
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
emit_alu(nfp_prog, dst, reg_none(), ALU_OP_NONE, NFP_BPF_ABI_PKT);
@@ -1174,12 +1174,12 @@
if (meta->insn.off == offsetof(struct sk_buff, mark))
return wrp_set_mark(nfp_prog, meta->insn.src_reg * 2);
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
}
static int mem_stx4_xdp(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
{
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
}
static int mem_stx4(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
@@ -1192,7 +1192,7 @@
static int jump(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta)
{
if (meta->insn.off < 0) /* TODO */
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
emit_br(nfp_prog, BR_UNC, meta->insn.off, 0);
return 0;
@@ -1206,7 +1206,7 @@
u32 tmp_reg;
if (insn->off < 0) /* TODO */
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
if (imm & ~0U) {
tmp_reg = ur_load_imm_any(nfp_prog, imm & ~0U, imm_b(nfp_prog));
@@ -1245,7 +1245,7 @@
u32 tmp_reg;
if (insn->off < 0) /* TODO */
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
if (!imm) {
meta->skip = true;
@@ -1276,7 +1276,7 @@
u32 tmp_reg;
if (insn->off < 0) /* TODO */
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
if (!imm) {
emit_alu(nfp_prog, reg_none(), reg_a(insn->dst_reg * 2),
@@ -1302,7 +1302,7 @@
const struct bpf_insn *insn = &meta->insn;
if (insn->off < 0) /* TODO */
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
emit_alu(nfp_prog, imm_a(nfp_prog), reg_a(insn->dst_reg * 2),
ALU_OP_XOR, reg_b(insn->src_reg * 2));
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.c b/drivers/net/ethernet/netronome/nfp/nfp_main.c
index dedac72..dde35da 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.c
@@ -48,7 +48,7 @@
#include "nfpcore/nfp.h"
#include "nfpcore/nfp_cpp.h"
#include "nfpcore/nfp_nffw.h"
-#include "nfpcore/nfp_nsp_eth.h"
+#include "nfpcore/nfp_nsp.h"
#include "nfpcore/nfp6000_pcie.h"
@@ -253,6 +253,7 @@
static int nfp_nsp_init(struct pci_dev *pdev, struct nfp_pf *pf)
{
+ struct nfp_nsp_identify *nspi;
struct nfp_nsp *nsp;
int err;
@@ -269,6 +270,12 @@
pf->eth_tbl = __nfp_eth_read_ports(pf->cpp, nsp);
+ nspi = __nfp_nsp_identify(nsp);
+ if (nspi) {
+ dev_info(&pdev->dev, "BSP: %s\n", nspi->version);
+ kfree(nspi);
+ }
+
err = nfp_fw_load(pdev, pf, nsp);
if (err < 0) {
kfree(pf->eth_tbl);
@@ -385,8 +392,7 @@
{
struct nfp_pf *pf = pci_get_drvdata(pdev);
- if (!list_empty(&pf->ports))
- nfp_net_pci_remove(pf);
+ nfp_net_pci_remove(pf);
nfp_pcie_sriov_disable(pdev);
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_main.h b/drivers/net/ethernet/netronome/nfp/nfp_main.h
index 39105d0..b57de04 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_main.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_main.h
@@ -42,7 +42,9 @@
#include <linux/list.h>
#include <linux/types.h>
#include <linux/msi.h>
+#include <linux/mutex.h>
#include <linux/pci.h>
+#include <linux/workqueue.h>
struct dentry;
struct pci_dev;
@@ -64,8 +66,11 @@
* @fw_loaded: Is the firmware loaded?
* @eth_tbl: NSP ETH table
* @ddir: Per-device debugfs directory
- * @num_ports: Number of adapter ports
+ * @num_ports: Number of adapter ports app firmware supports
+ * @num_netdevs: Number of netdevs spawned
* @ports: Linked list of port structures (struct nfp_net)
+ * @port_lock: Protects @ports, @num_ports, @num_netdevs
+ * @port_refresh_work: Work entry for taking netdevs out
*/
struct nfp_pf {
struct pci_dev *pdev;
@@ -88,7 +93,11 @@
struct dentry *ddir;
unsigned int num_ports;
+ unsigned int num_netdevs;
+
struct list_head ports;
+ struct work_struct port_refresh_work;
+ struct mutex port_lock;
};
extern struct pci_driver nfp_netvf_pci_driver;
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net.h b/drivers/net/ethernet/netronome/nfp/nfp_net.h
index e614a37..fcf81b3 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net.h
@@ -50,14 +50,14 @@
#include "nfp_net_ctrl.h"
-#define nn_err(nn, fmt, args...) netdev_err((nn)->netdev, fmt, ## args)
-#define nn_warn(nn, fmt, args...) netdev_warn((nn)->netdev, fmt, ## args)
-#define nn_info(nn, fmt, args...) netdev_info((nn)->netdev, fmt, ## args)
-#define nn_dbg(nn, fmt, args...) netdev_dbg((nn)->netdev, fmt, ## args)
-#define nn_warn_ratelimit(nn, fmt, args...) \
+#define nn_err(nn, fmt, args...) netdev_err((nn)->dp.netdev, fmt, ## args)
+#define nn_warn(nn, fmt, args...) netdev_warn((nn)->dp.netdev, fmt, ## args)
+#define nn_info(nn, fmt, args...) netdev_info((nn)->dp.netdev, fmt, ## args)
+#define nn_dbg(nn, fmt, args...) netdev_dbg((nn)->dp.netdev, fmt, ## args)
+#define nn_dp_warn(dp, fmt, args...) \
do { \
if (unlikely(net_ratelimit())) \
- netdev_warn((nn)->netdev, fmt, ## args); \
+ netdev_warn((dp)->netdev, fmt, ## args); \
} while (0)
/* Max time to wait for NFP to respond on updates (in seconds) */
@@ -112,6 +112,7 @@
/* Forward declarations */
struct nfp_cpp;
+struct nfp_eth_table_port;
struct nfp_net;
struct nfp_net_r_vector;
@@ -200,6 +201,7 @@
* @txds: Virtual address of TX ring in host memory
* @dma: DMA address of the TX ring
* @size: Size, in bytes, of the TX ring (needed to free)
+ * @is_xdp: Is this a XDP TX ring?
*/
struct nfp_net_tx_ring {
struct nfp_net_r_vector *r_vec;
@@ -220,6 +222,7 @@
dma_addr_t dma;
unsigned int size;
+ bool is_xdp;
} ____cacheline_aligned;
/* RX and freelist descriptor format */
@@ -283,6 +286,12 @@
#define NFP_NET_META_FIELD_MASK GENMASK(NFP_NET_META_FIELD_SIZE - 1, 0)
+struct nfp_meta_parsed {
+ u32 hash_type;
+ u32 hash;
+ u32 mark;
+};
+
struct nfp_net_rx_hash {
__be32 hash_type;
__be32 hash;
@@ -306,17 +315,13 @@
* @rd_p: FL/RX ring read pointer (free running)
* @idx: Ring index from Linux's perspective
* @fl_qcidx: Queue Controller Peripheral (QCP) queue index for the freelist
- * @rx_qcidx: Queue Controller Peripheral (QCP) queue index for the RX queue
* @qcp_fl: Pointer to base of the QCP freelist queue
- * @qcp_rx: Pointer to base of the QCP RX queue
* @wr_ptr_add: Accumulated number of buffers to add to QCP write pointer
* (used for free list batching)
* @rxbufs: Array of transmitted FL/RX buffers
* @rxds: Virtual address of FL/RX ring in host memory
* @dma: DMA address of the FL/RX ring
* @size: Size, in bytes, of the FL/RX ring (needed to free)
- * @bufsz: Buffer allocation size for convenience of management routines
- * (NOTE: this is in second cache line, do not use on fast path!)
*/
struct nfp_net_rx_ring {
struct nfp_net_r_vector *r_vec;
@@ -325,20 +330,17 @@
u32 wr_p;
u32 rd_p;
- u16 idx;
- u16 wr_ptr_add;
+ u32 idx;
+ u32 wr_ptr_add;
int fl_qcidx;
- int rx_qcidx;
u8 __iomem *qcp_fl;
- u8 __iomem *qcp_rx;
struct nfp_net_rx_buf *rxbufs;
struct nfp_net_rx_desc *rxds;
dma_addr_t dma;
unsigned int size;
- unsigned int bufsz;
} ____cacheline_aligned;
/**
@@ -433,19 +435,76 @@
};
/**
- * struct nfp_net - NFP network device structure
- * @pdev: Backpointer to PCI device
- * @netdev: Backpointer to net_device structure
- * @is_vf: Is the driver attached to a VF?
+ * struct nfp_net_dp - NFP network device datapath data structure
+ * @dev: Backpointer to struct device
+ * @netdev: Backpointer to net_device structure
+ * @is_vf: Is the driver attached to a VF?
* @bpf_offload_skip_sw: Offloaded BPF program will not be rerun by cls_bpf
* @bpf_offload_xdp: Offloaded BPF program is XDP
- * @ctrl: Local copy of the control register/word.
- * @fl_bufsz: Currently configured size of the freelist buffers
+ * @chained_metadata_format: Firemware will use new metadata format
+ * @rx_dma_dir: Mapping direction for RX buffers
+ * @rx_dma_off: Offset at which DMA packets (for XDP headroom)
* @rx_offset: Offset in the RX buffers where packet data starts
+ * @ctrl: Local copy of the control register/word.
+ * @fl_bufsz: Currently configured size of the freelist buffers
* @xdp_prog: Installed XDP program
- * @fw_ver: Firmware version
+ * @tx_rings: Array of pre-allocated TX ring structures
+ * @rx_rings: Array of pre-allocated RX ring structures
+ * @ctrl_bar: Pointer to mapped control BAR
+ *
+ * @txd_cnt: Size of the TX ring in number of descriptors
+ * @rxd_cnt: Size of the RX ring in number of descriptors
+ * @num_r_vecs: Number of used ring vectors
+ * @num_tx_rings: Currently configured number of TX rings
+ * @num_stack_tx_rings: Number of TX rings used by the stack (not XDP)
+ * @num_rx_rings: Currently configured number of RX rings
+ * @mtu: Device MTU
+ */
+struct nfp_net_dp {
+ struct device *dev;
+ struct net_device *netdev;
+
+ u8 is_vf:1;
+ u8 bpf_offload_skip_sw:1;
+ u8 bpf_offload_xdp:1;
+ u8 chained_metadata_format:1;
+
+ u8 rx_dma_dir;
+ u8 rx_offset;
+
+ u32 rx_dma_off;
+
+ u32 ctrl;
+ u32 fl_bufsz;
+
+ struct bpf_prog *xdp_prog;
+
+ struct nfp_net_tx_ring *tx_rings;
+ struct nfp_net_rx_ring *rx_rings;
+
+ u8 __iomem *ctrl_bar;
+
+ /* Cold data follows */
+
+ unsigned int txd_cnt;
+ unsigned int rxd_cnt;
+
+ unsigned int num_r_vecs;
+
+ unsigned int num_tx_rings;
+ unsigned int num_stack_tx_rings;
+ unsigned int num_rx_rings;
+
+ unsigned int mtu;
+};
+
+/**
+ * struct nfp_net - NFP network device structure
+ * @dp: Datapath structure
+ * @fw_ver: Firmware version
* @cap: Capabilities advertised by the Firmware
* @max_mtu: Maximum support MTU advertised by the Firmware
+ * @rss_hfunc: RSS selected hash function
* @rss_cfg: RSS configuration
* @rss_key: RSS secret key
* @rss_itbl: RSS indirection table
@@ -454,17 +513,9 @@
* @rx_filter_change: Jiffies when statistics last changed
* @rx_filter_stats_timer: Timer for polling filter offload statistics
* @rx_filter_lock: Lock protecting timer state changes (teardown)
+ * @max_r_vecs: Number of allocated interrupt vectors for RX/TX
* @max_tx_rings: Maximum number of TX rings supported by the Firmware
* @max_rx_rings: Maximum number of RX rings supported by the Firmware
- * @num_tx_rings: Currently configured number of TX rings
- * @num_stack_tx_rings: Number of TX rings used by the stack (not XDP)
- * @num_rx_rings: Currently configured number of RX rings
- * @txd_cnt: Size of the TX ring in number of descriptors
- * @rxd_cnt: Size of the RX ring in number of descriptors
- * @tx_rings: Array of pre-allocated TX ring structures
- * @rx_rings: Array of pre-allocated RX ring structures
- * @max_r_vecs: Number of allocated interrupt vectors for RX/TX
- * @num_r_vecs: Number of used ring vectors
* @r_vecs: Pre-allocated array of ring vectors
* @irq_entries: Pre-allocated array of MSI-X entries
* @lsc_handler: Handler for Link State Change interrupt
@@ -480,7 +531,8 @@
* @reconfig_sync_present: Some thread is performing synchronous reconfig
* @reconfig_timer: Timer for async reading of reconfig results
* @link_up: Is the link up?
- * @link_status_lock: Protects @link_up and ensures atomicity with BAR reading
+ * @link_changed: Has link state changes since last port refresh?
+ * @link_status_lock: Protects @link_* and ensures atomicity with BAR reading
* @rx_coalesce_usecs: RX interrupt moderation usecs delay parameter
* @rx_coalesce_max_frames: RX interrupt moderation frame count parameter
* @tx_coalesce_usecs: TX interrupt moderation usecs delay parameter
@@ -488,36 +540,24 @@
* @vxlan_ports: VXLAN ports for RX inner csum offload communicated to HW
* @vxlan_usecnt: IPv4/IPv6 VXLAN port use counts
* @qcp_cfg: Pointer to QCP queue used for configuration notification
- * @ctrl_bar: Pointer to mapped control BAR
* @tx_bar: Pointer to mapped TX queues
* @rx_bar: Pointer to mapped FL/RX queues
* @debugfs_dir: Device directory in debugfs
* @ethtool_dump_flag: Ethtool dump flag
* @port_list: Entry on device port list
+ * @pdev: Backpointer to PCI device
* @cpp: CPP device handle if available
+ * @eth_port: Translated ETH Table port entry
*/
struct nfp_net {
- struct pci_dev *pdev;
- struct net_device *netdev;
-
- unsigned is_vf:1;
- unsigned bpf_offload_skip_sw:1;
- unsigned bpf_offload_xdp:1;
-
- u32 ctrl;
- u32 fl_bufsz;
-
- u32 rx_offset;
-
- struct bpf_prog *xdp_prog;
-
- struct nfp_net_tx_ring *tx_rings;
- struct nfp_net_rx_ring *rx_rings;
+ struct nfp_net_dp dp;
struct nfp_net_fw_version fw_ver;
+
u32 cap;
u32 max_mtu;
+ u8 rss_hfunc;
u32 rss_cfg;
u8 rss_key[NFP_NET_CFG_RSS_KEY_SZ];
u8 rss_itbl[NFP_NET_CFG_RSS_ITBL_SZ];
@@ -530,18 +570,10 @@
unsigned int max_tx_rings;
unsigned int max_rx_rings;
- unsigned int num_tx_rings;
- unsigned int num_stack_tx_rings;
- unsigned int num_rx_rings;
-
int stride_tx;
int stride_rx;
- int txd_cnt;
- int rxd_cnt;
-
unsigned int max_r_vecs;
- unsigned int num_r_vecs;
struct nfp_net_r_vector r_vecs[NFP_NET_MAX_R_VECS];
struct msix_entry irq_entries[NFP_NET_MAX_IRQS];
@@ -557,6 +589,7 @@
u32 me_freq_mhz;
bool link_up;
+ bool link_changed;
spinlock_t link_status_lock;
spinlock_t reconfig_lock;
@@ -575,7 +608,6 @@
u8 __iomem *qcp_cfg;
- u8 __iomem *ctrl_bar;
u8 __iomem *tx_bar;
u8 __iomem *rx_bar;
@@ -584,14 +616,10 @@
struct list_head port_list;
+ struct pci_dev *pdev;
struct nfp_cpp *cpp;
-};
-struct nfp_net_ring_set {
- unsigned int n_rings;
- unsigned int mtu;
- unsigned int dcnt;
- void *rings;
+ struct nfp_eth_table_port *eth_port;
};
/* Functions to read/write from/to a BAR
@@ -599,42 +627,42 @@
*/
static inline u16 nn_readb(struct nfp_net *nn, int off)
{
- return readb(nn->ctrl_bar + off);
+ return readb(nn->dp.ctrl_bar + off);
}
static inline void nn_writeb(struct nfp_net *nn, int off, u8 val)
{
- writeb(val, nn->ctrl_bar + off);
+ writeb(val, nn->dp.ctrl_bar + off);
}
static inline u16 nn_readw(struct nfp_net *nn, int off)
{
- return readw(nn->ctrl_bar + off);
+ return readw(nn->dp.ctrl_bar + off);
}
static inline void nn_writew(struct nfp_net *nn, int off, u16 val)
{
- writew(val, nn->ctrl_bar + off);
+ writew(val, nn->dp.ctrl_bar + off);
}
static inline u32 nn_readl(struct nfp_net *nn, int off)
{
- return readl(nn->ctrl_bar + off);
+ return readl(nn->dp.ctrl_bar + off);
}
static inline void nn_writel(struct nfp_net *nn, int off, u32 val)
{
- writel(val, nn->ctrl_bar + off);
+ writel(val, nn->dp.ctrl_bar + off);
}
static inline u64 nn_readq(struct nfp_net *nn, int off)
{
- return readq(nn->ctrl_bar + off);
+ return readq(nn->dp.ctrl_bar + off);
}
static inline void nn_writeq(struct nfp_net *nn, int off, u64 val)
{
- writeq(val, nn->ctrl_bar + off);
+ writeq(val, nn->dp.ctrl_bar + off);
}
/* Flush posted PCI writes by reading something without side effects */
@@ -776,6 +804,7 @@
void nfp_net_set_ethtool_ops(struct net_device *netdev);
void nfp_net_info(struct nfp_net *nn);
int nfp_net_reconfig(struct nfp_net *nn, u32 update);
+unsigned int nfp_net_rss_key_sz(struct nfp_net *nn);
void nfp_net_rss_write_itbl(struct nfp_net *nn);
void nfp_net_rss_write_key(struct nfp_net *nn);
void nfp_net_coalesce_write_cfg(struct nfp_net *nn);
@@ -787,9 +816,14 @@
void
nfp_net_irqs_assign(struct nfp_net *nn, struct msix_entry *irq_entries,
unsigned int n);
-int
-nfp_net_ring_reconfig(struct nfp_net *nn, struct bpf_prog **xdp_prog,
- struct nfp_net_ring_set *rx, struct nfp_net_ring_set *tx);
+
+struct nfp_net_dp *nfp_net_clone_dp(struct nfp_net *nn);
+int nfp_net_ring_reconfig(struct nfp_net *nn, struct nfp_net_dp *new,
+ struct netlink_ext_ack *extack);
+
+bool nfp_net_link_changed_read_clear(struct nfp_net *nn);
+int nfp_net_refresh_eth_port(struct nfp_net *nn);
+void nfp_net_refresh_port_table(struct nfp_net *nn);
#ifdef CONFIG_NFP_DEBUG
void nfp_net_debugfs_create(void);
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
index a41377e..82bd6b0 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
@@ -41,6 +41,7 @@
* Chris Telfer <chris.telfer@netronome.com>
*/
+#include <linux/bitfield.h>
#include <linux/bpf.h>
#include <linux/bpf_trace.h>
#include <linux/module.h>
@@ -66,6 +67,7 @@
#include <net/pkt_cls.h>
#include <net/vxlan.h>
+#include "nfpcore/nfp_nsp.h"
#include "nfp_net_ctrl.h"
#include "nfp_net.h"
@@ -83,20 +85,33 @@
put_unaligned_le32(reg, fw_ver);
}
-static dma_addr_t
-nfp_net_dma_map_rx(struct nfp_net *nn, void *frag, unsigned int bufsz,
- int direction)
+static dma_addr_t nfp_net_dma_map_rx(struct nfp_net_dp *dp, void *frag)
{
- return dma_map_single(&nn->pdev->dev, frag + NFP_NET_RX_BUF_HEADROOM,
- bufsz - NFP_NET_RX_BUF_NON_DATA, direction);
+ return dma_map_single_attrs(dp->dev, frag + NFP_NET_RX_BUF_HEADROOM,
+ dp->fl_bufsz - NFP_NET_RX_BUF_NON_DATA,
+ dp->rx_dma_dir, DMA_ATTR_SKIP_CPU_SYNC);
}
static void
-nfp_net_dma_unmap_rx(struct nfp_net *nn, dma_addr_t dma_addr,
- unsigned int bufsz, int direction)
+nfp_net_dma_sync_dev_rx(const struct nfp_net_dp *dp, dma_addr_t dma_addr)
{
- dma_unmap_single(&nn->pdev->dev, dma_addr,
- bufsz - NFP_NET_RX_BUF_NON_DATA, direction);
+ dma_sync_single_for_device(dp->dev, dma_addr,
+ dp->fl_bufsz - NFP_NET_RX_BUF_NON_DATA,
+ dp->rx_dma_dir);
+}
+
+static void nfp_net_dma_unmap_rx(struct nfp_net_dp *dp, dma_addr_t dma_addr)
+{
+ dma_unmap_single_attrs(dp->dev, dma_addr,
+ dp->fl_bufsz - NFP_NET_RX_BUF_NON_DATA,
+ dp->rx_dma_dir, DMA_ATTR_SKIP_CPU_SYNC);
+}
+
+static void nfp_net_dma_sync_cpu_rx(struct nfp_net_dp *dp, dma_addr_t dma_addr,
+ unsigned int len)
+{
+ dma_sync_single_for_cpu(dp->dev, dma_addr - NFP_NET_RX_BUF_HEADROOM,
+ len, dp->rx_dma_dir);
}
/* Firmware reconfig
@@ -327,19 +342,22 @@
nfp_net_irqs_assign(struct nfp_net *nn, struct msix_entry *irq_entries,
unsigned int n)
{
+ struct nfp_net_dp *dp = &nn->dp;
+
nn->max_r_vecs = n - NFP_NET_NON_Q_VECTORS;
- nn->num_r_vecs = nn->max_r_vecs;
+ dp->num_r_vecs = nn->max_r_vecs;
memcpy(nn->irq_entries, irq_entries, sizeof(*irq_entries) * n);
- if (nn->num_rx_rings > nn->num_r_vecs ||
- nn->num_tx_rings > nn->num_r_vecs)
- nn_warn(nn, "More rings (%d,%d) than vectors (%d).\n",
- nn->num_rx_rings, nn->num_tx_rings, nn->num_r_vecs);
+ if (dp->num_rx_rings > dp->num_r_vecs ||
+ dp->num_tx_rings > dp->num_r_vecs)
+ dev_warn(nn->dp.dev, "More rings (%d,%d) than vectors (%d).\n",
+ dp->num_rx_rings, dp->num_tx_rings,
+ dp->num_r_vecs);
- nn->num_rx_rings = min(nn->num_r_vecs, nn->num_rx_rings);
- nn->num_tx_rings = min(nn->num_r_vecs, nn->num_tx_rings);
- nn->num_stack_tx_rings = nn->num_tx_rings;
+ dp->num_rx_rings = min(dp->num_r_vecs, dp->num_rx_rings);
+ dp->num_tx_rings = min(dp->num_r_vecs, dp->num_tx_rings);
+ dp->num_stack_tx_rings = dp->num_tx_rings;
}
/**
@@ -373,6 +391,19 @@
return IRQ_HANDLED;
}
+bool nfp_net_link_changed_read_clear(struct nfp_net *nn)
+{
+ unsigned long flags;
+ bool ret;
+
+ spin_lock_irqsave(&nn->link_status_lock, flags);
+ ret = nn->link_changed;
+ nn->link_changed = false;
+ spin_unlock_irqrestore(&nn->link_status_lock, flags);
+
+ return ret;
+}
+
/**
* nfp_net_read_link_status() - Reread link status from control BAR
* @nn: NFP Network structure
@@ -392,13 +423,14 @@
goto out;
nn->link_up = link_up;
+ nn->link_changed = true;
if (nn->link_up) {
- netif_carrier_on(nn->netdev);
- netdev_info(nn->netdev, "NIC Link is Up\n");
+ netif_carrier_on(nn->dp.netdev);
+ netdev_info(nn->dp.netdev, "NIC Link is Up\n");
} else {
- netif_carrier_off(nn->netdev);
- netdev_info(nn->netdev, "NIC Link is Down\n");
+ netif_carrier_off(nn->dp.netdev);
+ netdev_info(nn->dp.netdev, "NIC Link is Down\n");
}
out:
spin_unlock_irqrestore(&nn->link_status_lock, flags);
@@ -446,15 +478,18 @@
* @tx_ring: TX ring structure
* @r_vec: IRQ vector servicing this ring
* @idx: Ring index
+ * @is_xdp: Is this an XDP TX ring?
*/
static void
nfp_net_tx_ring_init(struct nfp_net_tx_ring *tx_ring,
- struct nfp_net_r_vector *r_vec, unsigned int idx)
+ struct nfp_net_r_vector *r_vec, unsigned int idx,
+ bool is_xdp)
{
struct nfp_net *nn = r_vec->nfp_net;
tx_ring->idx = idx;
tx_ring->r_vec = r_vec;
+ tx_ring->is_xdp = is_xdp;
tx_ring->qcidx = tx_ring->idx * nn->stride_tx;
tx_ring->qcp_q = nn->tx_bar + NFP_QCP_QUEUE_OFF(tx_ring->qcidx);
@@ -476,10 +511,7 @@
rx_ring->r_vec = r_vec;
rx_ring->fl_qcidx = rx_ring->idx * nn->stride_rx;
- rx_ring->rx_qcidx = rx_ring->fl_qcidx + (nn->stride_rx - 1);
-
rx_ring->qcp_fl = nn->rx_bar + NFP_QCP_QUEUE_OFF(rx_ring->fl_qcidx);
- rx_ring->qcp_rx = nn->rx_bar + NFP_QCP_QUEUE_OFF(rx_ring->rx_qcidx);
}
/**
@@ -530,7 +562,7 @@
entry = &nn->irq_entries[vector_idx];
- snprintf(name, name_sz, format, netdev_name(nn->netdev));
+ snprintf(name, name_sz, format, netdev_name(nn->dp.netdev));
err = request_irq(entry->vector, handler, 0, name, nn);
if (err) {
nn_err(nn, "Failed to request IRQ %d (err=%d).\n",
@@ -617,7 +649,6 @@
/**
* nfp_net_tx_tso() - Set up Tx descriptor for LSO
- * @nn: NFP Net device
* @r_vec: per-ring structure
* @txbuf: Pointer to driver soft TX descriptor
* @txd: Pointer to HW TX descriptor
@@ -626,7 +657,7 @@
* Set up Tx descriptor for LSO, do nothing for non-LSO skbs.
* Return error on packet header greater than maximum supported LSO header size.
*/
-static void nfp_net_tx_tso(struct nfp_net *nn, struct nfp_net_r_vector *r_vec,
+static void nfp_net_tx_tso(struct nfp_net_r_vector *r_vec,
struct nfp_net_tx_buf *txbuf,
struct nfp_net_tx_desc *txd, struct sk_buff *skb)
{
@@ -657,7 +688,7 @@
/**
* nfp_net_tx_csum() - Set TX CSUM offload flags in TX descriptor
- * @nn: NFP Net device
+ * @dp: NFP Net data path struct
* @r_vec: per-ring structure
* @txbuf: Pointer to driver soft TX descriptor
* @txd: Pointer to TX descriptor
@@ -666,7 +697,8 @@
* This function sets the TX checksum flags in the TX descriptor based
* on the configuration and the protocol of the packet to be transmitted.
*/
-static void nfp_net_tx_csum(struct nfp_net *nn, struct nfp_net_r_vector *r_vec,
+static void nfp_net_tx_csum(struct nfp_net_dp *dp,
+ struct nfp_net_r_vector *r_vec,
struct nfp_net_tx_buf *txbuf,
struct nfp_net_tx_desc *txd, struct sk_buff *skb)
{
@@ -674,7 +706,7 @@
struct iphdr *iph;
u8 l4_hdr;
- if (!(nn->ctrl & NFP_NET_CFG_CTRL_TXCSUM))
+ if (!(dp->ctrl & NFP_NET_CFG_CTRL_TXCSUM))
return;
if (skb->ip_summed != CHECKSUM_PARTIAL)
@@ -693,8 +725,7 @@
} else if (ipv6h->version == 6) {
l4_hdr = ipv6h->nexthdr;
} else {
- nn_warn_ratelimit(nn, "partial checksum but ipv=%x!\n",
- iph->version);
+ nn_dp_warn(dp, "partial checksum but ipv=%x!\n", iph->version);
return;
}
@@ -706,8 +737,7 @@
txd->flags |= PCIE_DESC_TX_UDP_CSUM;
break;
default:
- nn_warn_ratelimit(nn, "partial checksum but l4 proto=%x!\n",
- l4_hdr);
+ nn_dp_warn(dp, "partial checksum but l4 proto=%x!\n", l4_hdr);
return;
}
@@ -737,28 +767,31 @@
{
struct nfp_net *nn = netdev_priv(netdev);
const struct skb_frag_struct *frag;
- struct nfp_net_r_vector *r_vec;
struct nfp_net_tx_desc *txd, txdg;
- struct nfp_net_tx_buf *txbuf;
struct nfp_net_tx_ring *tx_ring;
+ struct nfp_net_r_vector *r_vec;
+ struct nfp_net_tx_buf *txbuf;
struct netdev_queue *nd_q;
+ struct nfp_net_dp *dp;
dma_addr_t dma_addr;
unsigned int fsize;
int f, nr_frags;
int wr_idx;
u16 qidx;
+ dp = &nn->dp;
qidx = skb_get_queue_mapping(skb);
- tx_ring = &nn->tx_rings[qidx];
+ tx_ring = &dp->tx_rings[qidx];
r_vec = tx_ring->r_vec;
- nd_q = netdev_get_tx_queue(nn->netdev, qidx);
+ nd_q = netdev_get_tx_queue(dp->netdev, qidx);
nr_frags = skb_shinfo(skb)->nr_frags;
if (unlikely(nfp_net_tx_full(tx_ring, nr_frags + 1))) {
- nn_warn_ratelimit(nn, "TX ring %d busy. wrp=%u rdp=%u\n",
- qidx, tx_ring->wr_p, tx_ring->rd_p);
+ nn_dp_warn(dp, "TX ring %d busy. wrp=%u rdp=%u\n",
+ qidx, tx_ring->wr_p, tx_ring->rd_p);
netif_tx_stop_queue(nd_q);
+ nfp_net_tx_xmit_more_flush(tx_ring);
u64_stats_update_begin(&r_vec->tx_sync);
r_vec->tx_busy++;
u64_stats_update_end(&r_vec->tx_sync);
@@ -766,9 +799,9 @@
}
/* Start with the head skbuf */
- dma_addr = dma_map_single(&nn->pdev->dev, skb->data, skb_headlen(skb),
+ dma_addr = dma_map_single(dp->dev, skb->data, skb_headlen(skb),
DMA_TO_DEVICE);
- if (dma_mapping_error(&nn->pdev->dev, dma_addr))
+ if (dma_mapping_error(dp->dev, dma_addr))
goto err_free;
wr_idx = tx_ring->wr_p & (tx_ring->cnt - 1);
@@ -792,11 +825,11 @@
txd->mss = 0;
txd->l4_offset = 0;
- nfp_net_tx_tso(nn, r_vec, txbuf, txd, skb);
+ nfp_net_tx_tso(r_vec, txbuf, txd, skb);
- nfp_net_tx_csum(nn, r_vec, txbuf, txd, skb);
+ nfp_net_tx_csum(dp, r_vec, txbuf, txd, skb);
- if (skb_vlan_tag_present(skb) && nn->ctrl & NFP_NET_CFG_CTRL_TXVLAN) {
+ if (skb_vlan_tag_present(skb) && dp->ctrl & NFP_NET_CFG_CTRL_TXVLAN) {
txd->flags |= PCIE_DESC_TX_VLAN;
txd->vlan = cpu_to_le16(skb_vlan_tag_get(skb));
}
@@ -810,9 +843,9 @@
frag = &skb_shinfo(skb)->frags[f];
fsize = skb_frag_size(frag);
- dma_addr = skb_frag_dma_map(&nn->pdev->dev, frag, 0,
+ dma_addr = skb_frag_dma_map(dp->dev, frag, 0,
fsize, DMA_TO_DEVICE);
- if (dma_mapping_error(&nn->pdev->dev, dma_addr))
+ if (dma_mapping_error(dp->dev, dma_addr))
goto err_unmap;
wr_idx = (wr_idx + 1) & (tx_ring->cnt - 1);
@@ -851,8 +884,7 @@
--f;
while (f >= 0) {
frag = &skb_shinfo(skb)->frags[f];
- dma_unmap_page(&nn->pdev->dev,
- tx_ring->txbufs[wr_idx].dma_addr,
+ dma_unmap_page(dp->dev, tx_ring->txbufs[wr_idx].dma_addr,
skb_frag_size(frag), DMA_TO_DEVICE);
tx_ring->txbufs[wr_idx].skb = NULL;
tx_ring->txbufs[wr_idx].dma_addr = 0;
@@ -861,13 +893,14 @@
if (wr_idx < 0)
wr_idx += tx_ring->cnt;
}
- dma_unmap_single(&nn->pdev->dev, tx_ring->txbufs[wr_idx].dma_addr,
+ dma_unmap_single(dp->dev, tx_ring->txbufs[wr_idx].dma_addr,
skb_headlen(skb), DMA_TO_DEVICE);
tx_ring->txbufs[wr_idx].skb = NULL;
tx_ring->txbufs[wr_idx].dma_addr = 0;
tx_ring->txbufs[wr_idx].fidx = -2;
err_free:
- nn_warn_ratelimit(nn, "Failed to map DMA TX buffer\n");
+ nn_dp_warn(dp, "Failed to map DMA TX buffer\n");
+ nfp_net_tx_xmit_more_flush(tx_ring);
u64_stats_update_begin(&r_vec->tx_sync);
r_vec->tx_errors++;
u64_stats_update_end(&r_vec->tx_sync);
@@ -884,7 +917,7 @@
static void nfp_net_tx_complete(struct nfp_net_tx_ring *tx_ring)
{
struct nfp_net_r_vector *r_vec = tx_ring->r_vec;
- struct nfp_net *nn = r_vec->nfp_net;
+ struct nfp_net_dp *dp = &r_vec->nfp_net->dp;
const struct skb_frag_struct *frag;
struct netdev_queue *nd_q;
u32 done_pkts = 0, done_bytes = 0;
@@ -894,6 +927,9 @@
int fidx;
int idx;
+ if (tx_ring->wr_p == tx_ring->rd_p)
+ return;
+
/* Work out how many descriptors have been transmitted */
qcp_rd_p = nfp_qcp_rd_ptr_read(tx_ring->qcp_q);
@@ -918,8 +954,7 @@
if (fidx == -1) {
/* unmap head */
- dma_unmap_single(&nn->pdev->dev,
- tx_ring->txbufs[idx].dma_addr,
+ dma_unmap_single(dp->dev, tx_ring->txbufs[idx].dma_addr,
skb_headlen(skb), DMA_TO_DEVICE);
done_pkts += tx_ring->txbufs[idx].pkt_cnt;
@@ -927,8 +962,7 @@
} else {
/* unmap fragment */
frag = &skb_shinfo(skb)->frags[fidx];
- dma_unmap_page(&nn->pdev->dev,
- tx_ring->txbufs[idx].dma_addr,
+ dma_unmap_page(dp->dev, tx_ring->txbufs[idx].dma_addr,
skb_frag_size(frag), DMA_TO_DEVICE);
}
@@ -948,7 +982,7 @@
r_vec->tx_pkts += done_pkts;
u64_stats_update_end(&r_vec->tx_sync);
- nd_q = netdev_get_tx_queue(nn->netdev, tx_ring->idx);
+ nd_q = netdev_get_tx_queue(dp->netdev, tx_ring->idx);
netdev_tx_completed_queue(nd_q, done_pkts, done_bytes);
if (nfp_net_tx_ring_should_wake(tx_ring)) {
/* Make sure TX thread will see updated tx_ring->rd_p */
@@ -966,11 +1000,13 @@
static void nfp_net_xdp_complete(struct nfp_net_tx_ring *tx_ring)
{
struct nfp_net_r_vector *r_vec = tx_ring->r_vec;
- struct nfp_net *nn = r_vec->nfp_net;
u32 done_pkts = 0, done_bytes = 0;
int idx, todo;
u32 qcp_rd_p;
+ if (tx_ring->wr_p == tx_ring->rd_p)
+ return;
+
/* Work out how many descriptors have been transmitted */
qcp_rd_p = nfp_qcp_rd_ptr_read(tx_ring->qcp_q);
@@ -982,23 +1018,12 @@
else
todo = qcp_rd_p + tx_ring->cnt - tx_ring->qcp_rd_p;
+ done_pkts = todo;
while (todo--) {
idx = tx_ring->rd_p & (tx_ring->cnt - 1);
tx_ring->rd_p++;
- if (!tx_ring->txbufs[idx].frag)
- continue;
-
- nfp_net_dma_unmap_rx(nn, tx_ring->txbufs[idx].dma_addr,
- nn->fl_bufsz, DMA_BIDIRECTIONAL);
- __free_page(virt_to_page(tx_ring->txbufs[idx].frag));
-
- done_pkts++;
done_bytes += tx_ring->txbufs[idx].real_len;
-
- tx_ring->txbufs[idx].dma_addr = 0;
- tx_ring->txbufs[idx].frag = NULL;
- tx_ring->txbufs[idx].fidx = -2;
}
tx_ring->qcp_rd_p = qcp_rd_p;
@@ -1015,52 +1040,43 @@
/**
* nfp_net_tx_ring_reset() - Free any untransmitted buffers and reset pointers
- * @nn: NFP Net device
+ * @dp: NFP Net data path struct
* @tx_ring: TX ring structure
*
* Assumes that the device is stopped
*/
static void
-nfp_net_tx_ring_reset(struct nfp_net *nn, struct nfp_net_tx_ring *tx_ring)
+nfp_net_tx_ring_reset(struct nfp_net_dp *dp, struct nfp_net_tx_ring *tx_ring)
{
- struct nfp_net_r_vector *r_vec = tx_ring->r_vec;
const struct skb_frag_struct *frag;
- struct pci_dev *pdev = nn->pdev;
struct netdev_queue *nd_q;
- while (tx_ring->rd_p != tx_ring->wr_p) {
+ while (!tx_ring->is_xdp && tx_ring->rd_p != tx_ring->wr_p) {
struct nfp_net_tx_buf *tx_buf;
- int idx;
+ struct sk_buff *skb;
+ int idx, nr_frags;
idx = tx_ring->rd_p & (tx_ring->cnt - 1);
tx_buf = &tx_ring->txbufs[idx];
- if (tx_ring == r_vec->xdp_ring) {
- nfp_net_dma_unmap_rx(nn, tx_buf->dma_addr,
- nn->fl_bufsz, DMA_BIDIRECTIONAL);
- __free_page(virt_to_page(tx_ring->txbufs[idx].frag));
+ skb = tx_ring->txbufs[idx].skb;
+ nr_frags = skb_shinfo(skb)->nr_frags;
+
+ if (tx_buf->fidx == -1) {
+ /* unmap head */
+ dma_unmap_single(dp->dev, tx_buf->dma_addr,
+ skb_headlen(skb), DMA_TO_DEVICE);
} else {
- struct sk_buff *skb = tx_ring->txbufs[idx].skb;
- int nr_frags = skb_shinfo(skb)->nr_frags;
-
- if (tx_buf->fidx == -1) {
- /* unmap head */
- dma_unmap_single(&pdev->dev, tx_buf->dma_addr,
- skb_headlen(skb),
- DMA_TO_DEVICE);
- } else {
- /* unmap fragment */
- frag = &skb_shinfo(skb)->frags[tx_buf->fidx];
- dma_unmap_page(&pdev->dev, tx_buf->dma_addr,
- skb_frag_size(frag),
- DMA_TO_DEVICE);
- }
-
- /* check for last gather fragment */
- if (tx_buf->fidx == nr_frags - 1)
- dev_kfree_skb_any(skb);
+ /* unmap fragment */
+ frag = &skb_shinfo(skb)->frags[tx_buf->fidx];
+ dma_unmap_page(dp->dev, tx_buf->dma_addr,
+ skb_frag_size(frag), DMA_TO_DEVICE);
}
+ /* check for last gather fragment */
+ if (tx_buf->fidx == nr_frags - 1)
+ dev_kfree_skb_any(skb);
+
tx_buf->dma_addr = 0;
tx_buf->skb = NULL;
tx_buf->fidx = -2;
@@ -1075,10 +1091,10 @@
tx_ring->qcp_rd_p = 0;
tx_ring->wr_ptr_add = 0;
- if (tx_ring == r_vec->xdp_ring)
+ if (tx_ring->is_xdp)
return;
- nd_q = netdev_get_tx_queue(nn->netdev, tx_ring->idx);
+ nd_q = netdev_get_tx_queue(dp->netdev, tx_ring->idx);
netdev_tx_reset_queue(nd_q);
}
@@ -1087,7 +1103,7 @@
struct nfp_net *nn = netdev_priv(netdev);
int i;
- for (i = 0; i < nn->netdev->real_num_tx_queues; i++) {
+ for (i = 0; i < nn->dp.netdev->real_num_tx_queues; i++) {
if (!netif_tx_queue_stopped(netdev_get_tx_queue(netdev, i)))
continue;
nn_warn(nn, "TX timeout on ring: %d\n", i);
@@ -1098,16 +1114,17 @@
/* Receive processing
*/
static unsigned int
-nfp_net_calc_fl_bufsz(struct nfp_net *nn, unsigned int mtu)
+nfp_net_calc_fl_bufsz(struct nfp_net_dp *dp)
{
unsigned int fl_bufsz;
fl_bufsz = NFP_NET_RX_BUF_HEADROOM;
- if (nn->rx_offset == NFP_NET_CFG_RX_OFFSET_DYNAMIC)
+ fl_bufsz += dp->rx_dma_off;
+ if (dp->rx_offset == NFP_NET_CFG_RX_OFFSET_DYNAMIC)
fl_bufsz += NFP_NET_MAX_PREPEND;
else
- fl_bufsz += nn->rx_offset;
- fl_bufsz += ETH_HLEN + VLAN_HLEN * 2 + mtu;
+ fl_bufsz += dp->rx_offset;
+ fl_bufsz += ETH_HLEN + VLAN_HLEN * 2 + dp->mtu;
fl_bufsz = SKB_DATA_ALIGN(fl_bufsz);
fl_bufsz += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
@@ -1126,62 +1143,53 @@
/**
* nfp_net_rx_alloc_one() - Allocate and map page frag for RX
- * @rx_ring: RX ring structure of the skb
+ * @dp: NFP Net data path struct
* @dma_addr: Pointer to storage for DMA address (output param)
- * @fl_bufsz: size of freelist buffers
- * @xdp: Whether XDP is enabled
*
* This function will allcate a new page frag, map it for DMA.
*
* Return: allocated page frag or NULL on failure.
*/
-static void *
-nfp_net_rx_alloc_one(struct nfp_net_rx_ring *rx_ring, dma_addr_t *dma_addr,
- unsigned int fl_bufsz, bool xdp)
+static void *nfp_net_rx_alloc_one(struct nfp_net_dp *dp, dma_addr_t *dma_addr)
{
- struct nfp_net *nn = rx_ring->r_vec->nfp_net;
- int direction;
void *frag;
- if (!xdp)
- frag = netdev_alloc_frag(fl_bufsz);
+ if (!dp->xdp_prog)
+ frag = netdev_alloc_frag(dp->fl_bufsz);
else
frag = page_address(alloc_page(GFP_KERNEL | __GFP_COLD));
if (!frag) {
- nn_warn_ratelimit(nn, "Failed to alloc receive page frag\n");
+ nn_dp_warn(dp, "Failed to alloc receive page frag\n");
return NULL;
}
- direction = xdp ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE;
-
- *dma_addr = nfp_net_dma_map_rx(nn, frag, fl_bufsz, direction);
- if (dma_mapping_error(&nn->pdev->dev, *dma_addr)) {
- nfp_net_free_frag(frag, xdp);
- nn_warn_ratelimit(nn, "Failed to map DMA RX buffer\n");
+ *dma_addr = nfp_net_dma_map_rx(dp, frag);
+ if (dma_mapping_error(dp->dev, *dma_addr)) {
+ nfp_net_free_frag(frag, dp->xdp_prog);
+ nn_dp_warn(dp, "Failed to map DMA RX buffer\n");
return NULL;
}
return frag;
}
-static void *
-nfp_net_napi_alloc_one(struct nfp_net *nn, int direction, dma_addr_t *dma_addr)
+static void *nfp_net_napi_alloc_one(struct nfp_net_dp *dp, dma_addr_t *dma_addr)
{
void *frag;
- if (!nn->xdp_prog)
- frag = napi_alloc_frag(nn->fl_bufsz);
+ if (!dp->xdp_prog)
+ frag = napi_alloc_frag(dp->fl_bufsz);
else
frag = page_address(alloc_page(GFP_ATOMIC | __GFP_COLD));
if (!frag) {
- nn_warn_ratelimit(nn, "Failed to alloc receive page frag\n");
+ nn_dp_warn(dp, "Failed to alloc receive page frag\n");
return NULL;
}
- *dma_addr = nfp_net_dma_map_rx(nn, frag, nn->fl_bufsz, direction);
- if (dma_mapping_error(&nn->pdev->dev, *dma_addr)) {
- nfp_net_free_frag(frag, nn->xdp_prog);
- nn_warn_ratelimit(nn, "Failed to map DMA RX buffer\n");
+ *dma_addr = nfp_net_dma_map_rx(dp, frag);
+ if (dma_mapping_error(dp->dev, *dma_addr)) {
+ nfp_net_free_frag(frag, dp->xdp_prog);
+ nn_dp_warn(dp, "Failed to map DMA RX buffer\n");
return NULL;
}
@@ -1190,17 +1198,21 @@
/**
* nfp_net_rx_give_one() - Put mapped skb on the software and hardware rings
+ * @dp: NFP Net data path struct
* @rx_ring: RX ring structure
* @frag: page fragment buffer
* @dma_addr: DMA address of skb mapping
*/
-static void nfp_net_rx_give_one(struct nfp_net_rx_ring *rx_ring,
+static void nfp_net_rx_give_one(const struct nfp_net_dp *dp,
+ struct nfp_net_rx_ring *rx_ring,
void *frag, dma_addr_t dma_addr)
{
unsigned int wr_idx;
wr_idx = rx_ring->wr_p & (rx_ring->cnt - 1);
+ nfp_net_dma_sync_dev_rx(dp, dma_addr);
+
/* Stash SKB and DMA address away */
rx_ring->rxbufs[wr_idx].frag = frag;
rx_ring->rxbufs[wr_idx].dma_addr = dma_addr;
@@ -1208,7 +1220,8 @@
/* Fill freelist descriptor */
rx_ring->rxds[wr_idx].fld.reserved = 0;
rx_ring->rxds[wr_idx].fld.meta_len_dd = 0;
- nfp_desc_set_dma_addr(&rx_ring->rxds[wr_idx].fld, dma_addr);
+ nfp_desc_set_dma_addr(&rx_ring->rxds[wr_idx].fld,
+ dma_addr + dp->rx_dma_off);
rx_ring->wr_p++;
rx_ring->wr_ptr_add++;
@@ -1249,19 +1262,17 @@
/**
* nfp_net_rx_ring_bufs_free() - Free any buffers currently on the RX ring
- * @nn: NFP Net device
+ * @dp: NFP Net data path struct
* @rx_ring: RX ring to remove buffers from
- * @xdp: Whether XDP is enabled
*
* Assumes that the device is stopped and buffers are in [0, ring->cnt - 1)
* entries. After device is disabled nfp_net_rx_ring_reset() must be called
* to restore required ring geometry.
*/
static void
-nfp_net_rx_ring_bufs_free(struct nfp_net *nn, struct nfp_net_rx_ring *rx_ring,
- bool xdp)
+nfp_net_rx_ring_bufs_free(struct nfp_net_dp *dp,
+ struct nfp_net_rx_ring *rx_ring)
{
- int direction = xdp ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE;
unsigned int i;
for (i = 0; i < rx_ring->cnt - 1; i++) {
@@ -1272,9 +1283,8 @@
if (!rx_ring->rxbufs[i].frag)
continue;
- nfp_net_dma_unmap_rx(nn, rx_ring->rxbufs[i].dma_addr,
- rx_ring->bufsz, direction);
- nfp_net_free_frag(rx_ring->rxbufs[i].frag, xdp);
+ nfp_net_dma_unmap_rx(dp, rx_ring->rxbufs[i].dma_addr);
+ nfp_net_free_frag(rx_ring->rxbufs[i].frag, dp->xdp_prog);
rx_ring->rxbufs[i].dma_addr = 0;
rx_ring->rxbufs[i].frag = NULL;
}
@@ -1282,13 +1292,12 @@
/**
* nfp_net_rx_ring_bufs_alloc() - Fill RX ring with buffers (don't give to FW)
- * @nn: NFP Net device
+ * @dp: NFP Net data path struct
* @rx_ring: RX ring to remove buffers from
- * @xdp: Whether XDP is enabled
*/
static int
-nfp_net_rx_ring_bufs_alloc(struct nfp_net *nn, struct nfp_net_rx_ring *rx_ring,
- bool xdp)
+nfp_net_rx_ring_bufs_alloc(struct nfp_net_dp *dp,
+ struct nfp_net_rx_ring *rx_ring)
{
struct nfp_net_rx_buf *rxbufs;
unsigned int i;
@@ -1296,11 +1305,9 @@
rxbufs = rx_ring->rxbufs;
for (i = 0; i < rx_ring->cnt - 1; i++) {
- rxbufs[i].frag =
- nfp_net_rx_alloc_one(rx_ring, &rxbufs[i].dma_addr,
- rx_ring->bufsz, xdp);
+ rxbufs[i].frag = nfp_net_rx_alloc_one(dp, &rxbufs[i].dma_addr);
if (!rxbufs[i].frag) {
- nfp_net_rx_ring_bufs_free(nn, rx_ring, xdp);
+ nfp_net_rx_ring_bufs_free(dp, rx_ring);
return -ENOMEM;
}
}
@@ -1310,14 +1317,17 @@
/**
* nfp_net_rx_ring_fill_freelist() - Give buffers from the ring to FW
+ * @dp: NFP Net data path struct
* @rx_ring: RX ring to fill
*/
-static void nfp_net_rx_ring_fill_freelist(struct nfp_net_rx_ring *rx_ring)
+static void
+nfp_net_rx_ring_fill_freelist(struct nfp_net_dp *dp,
+ struct nfp_net_rx_ring *rx_ring)
{
unsigned int i;
for (i = 0; i < rx_ring->cnt - 1; i++)
- nfp_net_rx_give_one(rx_ring, rx_ring->rxbufs[i].frag,
+ nfp_net_rx_give_one(dp, rx_ring, rx_ring->rxbufs[i].frag,
rx_ring->rxbufs[i].dma_addr);
}
@@ -1337,17 +1347,18 @@
/**
* nfp_net_rx_csum() - set SKB checksum field based on RX descriptor flags
- * @nn: NFP Net device
+ * @dp: NFP Net data path struct
* @r_vec: per-ring structure
* @rxd: Pointer to RX descriptor
* @skb: Pointer to SKB
*/
-static void nfp_net_rx_csum(struct nfp_net *nn, struct nfp_net_r_vector *r_vec,
+static void nfp_net_rx_csum(struct nfp_net_dp *dp,
+ struct nfp_net_r_vector *r_vec,
struct nfp_net_rx_desc *rxd, struct sk_buff *skb)
{
skb_checksum_none_assert(skb);
- if (!(nn->netdev->features & NETIF_F_RXCSUM))
+ if (!(dp->netdev->features & NETIF_F_RXCSUM))
return;
if (nfp_net_rx_csum_has_errors(le16_to_cpu(rxd->rxd.flags))) {
@@ -1378,8 +1389,9 @@
}
}
-static void nfp_net_set_hash(struct net_device *netdev, struct sk_buff *skb,
- unsigned int type, __be32 *hash)
+static void
+nfp_net_set_hash(struct net_device *netdev, struct nfp_meta_parsed *meta,
+ unsigned int type, __be32 *hash)
{
if (!(netdev->features & NETIF_F_RXHASH))
return;
@@ -1388,34 +1400,33 @@
case NFP_NET_RSS_IPV4:
case NFP_NET_RSS_IPV6:
case NFP_NET_RSS_IPV6_EX:
- skb_set_hash(skb, get_unaligned_be32(hash), PKT_HASH_TYPE_L3);
+ meta->hash_type = PKT_HASH_TYPE_L3;
break;
default:
- skb_set_hash(skb, get_unaligned_be32(hash), PKT_HASH_TYPE_L4);
+ meta->hash_type = PKT_HASH_TYPE_L4;
break;
}
+
+ meta->hash = get_unaligned_be32(hash);
}
static void
-nfp_net_set_hash_desc(struct net_device *netdev, struct sk_buff *skb,
- struct nfp_net_rx_desc *rxd)
+nfp_net_set_hash_desc(struct net_device *netdev, struct nfp_meta_parsed *meta,
+ void *data, struct nfp_net_rx_desc *rxd)
{
- struct nfp_net_rx_hash *rx_hash;
+ struct nfp_net_rx_hash *rx_hash = data;
if (!(rxd->rxd.flags & PCIE_DESC_RX_RSS))
return;
- rx_hash = (struct nfp_net_rx_hash *)(skb->data - sizeof(*rx_hash));
-
- nfp_net_set_hash(netdev, skb, get_unaligned_be32(&rx_hash->hash_type),
+ nfp_net_set_hash(netdev, meta, get_unaligned_be32(&rx_hash->hash_type),
&rx_hash->hash);
}
static void *
-nfp_net_parse_meta(struct net_device *netdev, struct sk_buff *skb,
- int meta_len)
+nfp_net_parse_meta(struct net_device *netdev, struct nfp_meta_parsed *meta,
+ void *data, int meta_len)
{
- u8 *data = skb->data - meta_len;
u32 meta_info;
meta_info = get_unaligned_be32(data);
@@ -1425,13 +1436,13 @@
switch (meta_info & NFP_NET_META_FIELD_MASK) {
case NFP_NET_META_HASH:
meta_info >>= NFP_NET_META_FIELD_SIZE;
- nfp_net_set_hash(netdev, skb,
+ nfp_net_set_hash(netdev, meta,
meta_info & NFP_NET_META_FIELD_MASK,
(__be32 *)data);
data += 4;
break;
case NFP_NET_META_MARK:
- skb->mark = get_unaligned_be32(data);
+ meta->mark = get_unaligned_be32(data);
data += 4;
break;
default:
@@ -1445,8 +1456,9 @@
}
static void
-nfp_net_rx_drop(struct nfp_net_r_vector *r_vec, struct nfp_net_rx_ring *rx_ring,
- struct nfp_net_rx_buf *rxbuf, struct sk_buff *skb)
+nfp_net_rx_drop(const struct nfp_net_dp *dp, struct nfp_net_r_vector *r_vec,
+ struct nfp_net_rx_ring *rx_ring, struct nfp_net_rx_buf *rxbuf,
+ struct sk_buff *skb)
{
u64_stats_update_begin(&r_vec->rx_sync);
r_vec->rx_drops++;
@@ -1458,53 +1470,47 @@
if (skb && rxbuf && skb->head == rxbuf->frag)
page_ref_inc(virt_to_head_page(rxbuf->frag));
if (rxbuf)
- nfp_net_rx_give_one(rx_ring, rxbuf->frag, rxbuf->dma_addr);
+ nfp_net_rx_give_one(dp, rx_ring, rxbuf->frag, rxbuf->dma_addr);
if (skb)
dev_kfree_skb_any(skb);
}
static bool
-nfp_net_tx_xdp_buf(struct nfp_net *nn, struct nfp_net_rx_ring *rx_ring,
+nfp_net_tx_xdp_buf(struct nfp_net_dp *dp, struct nfp_net_rx_ring *rx_ring,
struct nfp_net_tx_ring *tx_ring,
- struct nfp_net_rx_buf *rxbuf, unsigned int pkt_off,
+ struct nfp_net_rx_buf *rxbuf, unsigned int dma_off,
unsigned int pkt_len)
{
struct nfp_net_tx_buf *txbuf;
struct nfp_net_tx_desc *txd;
- dma_addr_t new_dma_addr;
- void *new_frag;
int wr_idx;
if (unlikely(nfp_net_tx_full(tx_ring, 1))) {
- nfp_net_rx_drop(rx_ring->r_vec, rx_ring, rxbuf, NULL);
+ nfp_net_rx_drop(dp, rx_ring->r_vec, rx_ring, rxbuf, NULL);
return false;
}
- new_frag = nfp_net_napi_alloc_one(nn, DMA_BIDIRECTIONAL, &new_dma_addr);
- if (unlikely(!new_frag)) {
- nfp_net_rx_drop(rx_ring->r_vec, rx_ring, rxbuf, NULL);
- return false;
- }
- nfp_net_rx_give_one(rx_ring, new_frag, new_dma_addr);
-
wr_idx = tx_ring->wr_p & (tx_ring->cnt - 1);
/* Stash the soft descriptor of the head then initialize it */
txbuf = &tx_ring->txbufs[wr_idx];
+
+ nfp_net_rx_give_one(dp, rx_ring, txbuf->frag, txbuf->dma_addr);
+
txbuf->frag = rxbuf->frag;
txbuf->dma_addr = rxbuf->dma_addr;
txbuf->fidx = -1;
txbuf->pkt_cnt = 1;
txbuf->real_len = pkt_len;
- dma_sync_single_for_device(&nn->pdev->dev, rxbuf->dma_addr + pkt_off,
+ dma_sync_single_for_device(dp->dev, rxbuf->dma_addr + dma_off,
pkt_len, DMA_BIDIRECTIONAL);
/* Build TX descriptor */
txd = &tx_ring->txds[wr_idx];
txd->offset_eop = PCIE_DESC_TX_EOP;
txd->dma_len = cpu_to_le16(pkt_len);
- nfp_desc_set_dma_addr(txd, rxbuf->dma_addr + pkt_off);
+ nfp_desc_set_dma_addr(txd, rxbuf->dma_addr + dma_off);
txd->data_len = cpu_to_le16(pkt_len);
txd->flags = 0;
@@ -1516,14 +1522,24 @@
return true;
}
-static int nfp_net_run_xdp(struct bpf_prog *prog, void *data, unsigned int len)
+static int nfp_net_run_xdp(struct bpf_prog *prog, void *data, void *hard_start,
+ unsigned int *off, unsigned int *len)
{
struct xdp_buff xdp;
+ void *orig_data;
+ int ret;
- xdp.data = data;
- xdp.data_end = data + len;
+ xdp.data_hard_start = hard_start;
+ xdp.data = data + *off;
+ xdp.data_end = data + *off + *len;
- return bpf_prog_run_xdp(prog, &xdp);
+ orig_data = xdp.data;
+ ret = bpf_prog_run_xdp(prog, &xdp);
+
+ *len -= xdp.data - orig_data;
+ *off += xdp.data - orig_data;
+
+ return ret;
}
/**
@@ -1540,25 +1556,24 @@
static int nfp_net_rx(struct nfp_net_rx_ring *rx_ring, int budget)
{
struct nfp_net_r_vector *r_vec = rx_ring->r_vec;
- struct nfp_net *nn = r_vec->nfp_net;
+ struct nfp_net_dp *dp = &r_vec->nfp_net->dp;
struct nfp_net_tx_ring *tx_ring;
struct bpf_prog *xdp_prog;
unsigned int true_bufsz;
struct sk_buff *skb;
int pkts_polled = 0;
- int rx_dma_map_dir;
int idx;
rcu_read_lock();
- xdp_prog = READ_ONCE(nn->xdp_prog);
- rx_dma_map_dir = xdp_prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE;
- true_bufsz = xdp_prog ? PAGE_SIZE : nn->fl_bufsz;
+ xdp_prog = READ_ONCE(dp->xdp_prog);
+ true_bufsz = xdp_prog ? PAGE_SIZE : dp->fl_bufsz;
tx_ring = r_vec->xdp_ring;
while (pkts_polled < budget) {
- unsigned int meta_len, data_len, data_off, pkt_len, pkt_off;
+ unsigned int meta_len, data_len, meta_off, pkt_len, pkt_off;
struct nfp_net_rx_buf *rxbuf;
struct nfp_net_rx_desc *rxd;
+ struct nfp_meta_parsed meta;
dma_addr_t new_dma_addr;
void *new_frag;
@@ -1573,6 +1588,8 @@
*/
dma_rmb();
+ memset(&meta, 0, sizeof(meta));
+
rx_ring->rd_p++;
pkts_polled++;
@@ -1593,11 +1610,12 @@
data_len = le16_to_cpu(rxd->rxd.data_len);
pkt_len = data_len - meta_len;
- if (nn->rx_offset == NFP_NET_CFG_RX_OFFSET_DYNAMIC)
- pkt_off = meta_len;
+ pkt_off = NFP_NET_RX_BUF_HEADROOM + dp->rx_dma_off;
+ if (dp->rx_offset == NFP_NET_CFG_RX_OFFSET_DYNAMIC)
+ pkt_off += meta_len;
else
- pkt_off = nn->rx_offset;
- data_off = NFP_NET_RX_BUF_HEADROOM + pkt_off;
+ pkt_off += dp->rx_offset;
+ meta_off = pkt_off - meta_len;
/* Stats update */
u64_stats_update_begin(&r_vec->rx_sync);
@@ -1605,30 +1623,62 @@
r_vec->rx_bytes += pkt_len;
u64_stats_update_end(&r_vec->rx_sync);
+ if (unlikely(meta_len > NFP_NET_MAX_PREPEND ||
+ (dp->rx_offset && meta_len > dp->rx_offset))) {
+ nn_dp_warn(dp, "oversized RX packet metadata %u\n",
+ meta_len);
+ nfp_net_rx_drop(dp, r_vec, rx_ring, rxbuf, NULL);
+ continue;
+ }
+
+ nfp_net_dma_sync_cpu_rx(dp, rxbuf->dma_addr + meta_off,
+ data_len);
+
+ if (!dp->chained_metadata_format) {
+ nfp_net_set_hash_desc(dp->netdev, &meta,
+ rxbuf->frag + meta_off, rxd);
+ } else if (meta_len) {
+ void *end;
+
+ end = nfp_net_parse_meta(dp->netdev, &meta,
+ rxbuf->frag + meta_off,
+ meta_len);
+ if (unlikely(end != rxbuf->frag + pkt_off)) {
+ nn_dp_warn(dp, "invalid RX packet metadata\n");
+ nfp_net_rx_drop(dp, r_vec, rx_ring, rxbuf,
+ NULL);
+ continue;
+ }
+ }
+
if (xdp_prog && !(rxd->rxd.flags & PCIE_DESC_RX_BPF &&
- nn->bpf_offload_xdp)) {
+ dp->bpf_offload_xdp)) {
+ unsigned int dma_off;
+ void *hard_start;
int act;
- dma_sync_single_for_cpu(&nn->pdev->dev,
- rxbuf->dma_addr + pkt_off,
- pkt_len, DMA_BIDIRECTIONAL);
- act = nfp_net_run_xdp(xdp_prog, rxbuf->frag + data_off,
- pkt_len);
+ hard_start = rxbuf->frag + NFP_NET_RX_BUF_HEADROOM;
+
+ act = nfp_net_run_xdp(xdp_prog, rxbuf->frag, hard_start,
+ &pkt_off, &pkt_len);
switch (act) {
case XDP_PASS:
break;
case XDP_TX:
- if (unlikely(!nfp_net_tx_xdp_buf(nn, rx_ring,
+ dma_off = pkt_off - NFP_NET_RX_BUF_HEADROOM;
+ if (unlikely(!nfp_net_tx_xdp_buf(dp, rx_ring,
tx_ring, rxbuf,
- pkt_off, pkt_len)))
- trace_xdp_exception(nn->netdev, xdp_prog, act);
+ dma_off,
+ pkt_len)))
+ trace_xdp_exception(dp->netdev,
+ xdp_prog, act);
continue;
default:
bpf_warn_invalid_xdp_action(act);
case XDP_ABORTED:
- trace_xdp_exception(nn->netdev, xdp_prog, act);
+ trace_xdp_exception(dp->netdev, xdp_prog, act);
case XDP_DROP:
- nfp_net_rx_give_one(rx_ring, rxbuf->frag,
+ nfp_net_rx_give_one(dp, rx_ring, rxbuf->frag,
rxbuf->dma_addr);
continue;
}
@@ -1636,41 +1686,29 @@
skb = build_skb(rxbuf->frag, true_bufsz);
if (unlikely(!skb)) {
- nfp_net_rx_drop(r_vec, rx_ring, rxbuf, NULL);
+ nfp_net_rx_drop(dp, r_vec, rx_ring, rxbuf, NULL);
continue;
}
- new_frag = nfp_net_napi_alloc_one(nn, rx_dma_map_dir,
- &new_dma_addr);
+ new_frag = nfp_net_napi_alloc_one(dp, &new_dma_addr);
if (unlikely(!new_frag)) {
- nfp_net_rx_drop(r_vec, rx_ring, rxbuf, skb);
+ nfp_net_rx_drop(dp, r_vec, rx_ring, rxbuf, skb);
continue;
}
- nfp_net_dma_unmap_rx(nn, rxbuf->dma_addr, nn->fl_bufsz,
- rx_dma_map_dir);
+ nfp_net_dma_unmap_rx(dp, rxbuf->dma_addr);
- nfp_net_rx_give_one(rx_ring, new_frag, new_dma_addr);
+ nfp_net_rx_give_one(dp, rx_ring, new_frag, new_dma_addr);
- skb_reserve(skb, data_off);
+ skb_reserve(skb, pkt_off);
skb_put(skb, pkt_len);
- if (nn->fw_ver.major <= 3) {
- nfp_net_set_hash_desc(nn->netdev, skb, rxd);
- } else if (meta_len) {
- void *end;
-
- end = nfp_net_parse_meta(nn->netdev, skb, meta_len);
- if (unlikely(end != skb->data)) {
- nn_warn_ratelimit(nn, "invalid RX packet metadata\n");
- nfp_net_rx_drop(r_vec, rx_ring, NULL, skb);
- continue;
- }
- }
+ skb->mark = meta.mark;
+ skb_set_hash(skb, meta.hash, meta.hash_type);
skb_record_rx_queue(skb, rx_ring->idx);
- skb->protocol = eth_type_trans(skb, nn->netdev);
+ skb->protocol = eth_type_trans(skb, dp->netdev);
- nfp_net_rx_csum(nn, r_vec, rxd, skb);
+ nfp_net_rx_csum(dp, r_vec, rxd, skb);
if (rxd->rxd.flags & PCIE_DESC_RX_VLAN)
__vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q),
@@ -1707,10 +1745,9 @@
nfp_net_xdp_complete(r_vec->xdp_ring);
}
- if (pkts_polled < budget) {
- napi_complete_done(napi, pkts_polled);
- nfp_net_irq_unmask(r_vec->nfp_net, r_vec->irq_entry);
- }
+ if (pkts_polled < budget)
+ if (napi_complete_done(napi, pkts_polled))
+ nfp_net_irq_unmask(r_vec->nfp_net, r_vec->irq_entry);
return pkts_polled;
}
@@ -1725,13 +1762,12 @@
static void nfp_net_tx_ring_free(struct nfp_net_tx_ring *tx_ring)
{
struct nfp_net_r_vector *r_vec = tx_ring->r_vec;
- struct nfp_net *nn = r_vec->nfp_net;
- struct pci_dev *pdev = nn->pdev;
+ struct nfp_net_dp *dp = &r_vec->nfp_net->dp;
kfree(tx_ring->txbufs);
if (tx_ring->txds)
- dma_free_coherent(&pdev->dev, tx_ring->size,
+ dma_free_coherent(dp->dev, tx_ring->size,
tx_ring->txds, tx_ring->dma);
tx_ring->cnt = 0;
@@ -1743,24 +1779,21 @@
/**
* nfp_net_tx_ring_alloc() - Allocate resource for a TX ring
+ * @dp: NFP Net data path struct
* @tx_ring: TX Ring structure to allocate
- * @cnt: Ring buffer count
- * @is_xdp: True if ring will be used for XDP
*
* Return: 0 on success, negative errno otherwise.
*/
static int
-nfp_net_tx_ring_alloc(struct nfp_net_tx_ring *tx_ring, u32 cnt, bool is_xdp)
+nfp_net_tx_ring_alloc(struct nfp_net_dp *dp, struct nfp_net_tx_ring *tx_ring)
{
struct nfp_net_r_vector *r_vec = tx_ring->r_vec;
- struct nfp_net *nn = r_vec->nfp_net;
- struct pci_dev *pdev = nn->pdev;
int sz;
- tx_ring->cnt = cnt;
+ tx_ring->cnt = dp->txd_cnt;
tx_ring->size = sizeof(*tx_ring->txds) * tx_ring->cnt;
- tx_ring->txds = dma_zalloc_coherent(&pdev->dev, tx_ring->size,
+ tx_ring->txds = dma_zalloc_coherent(dp->dev, tx_ring->size,
&tx_ring->dma, GFP_KERNEL);
if (!tx_ring->txds)
goto err_alloc;
@@ -1770,15 +1803,10 @@
if (!tx_ring->txbufs)
goto err_alloc;
- if (!is_xdp)
- netif_set_xps_queue(nn->netdev, &r_vec->affinity_mask,
+ if (!tx_ring->is_xdp)
+ netif_set_xps_queue(dp->netdev, &r_vec->affinity_mask,
tx_ring->idx);
- nn_dbg(nn, "TxQ%02d: QCidx=%02d cnt=%d dma=%#llx host=%p %s\n",
- tx_ring->idx, tx_ring->qcidx,
- tx_ring->cnt, (unsigned long long)tx_ring->dma, tx_ring->txds,
- is_xdp ? "XDP" : "");
-
return 0;
err_alloc:
@@ -1786,62 +1814,92 @@
return -ENOMEM;
}
-static struct nfp_net_tx_ring *
-nfp_net_tx_ring_set_prepare(struct nfp_net *nn, struct nfp_net_ring_set *s,
- unsigned int num_stack_tx_rings)
+static void
+nfp_net_tx_ring_bufs_free(struct nfp_net_dp *dp,
+ struct nfp_net_tx_ring *tx_ring)
{
- struct nfp_net_tx_ring *rings;
- unsigned int r;
+ unsigned int i;
- rings = kcalloc(s->n_rings, sizeof(*rings), GFP_KERNEL);
- if (!rings)
- return NULL;
+ if (!tx_ring->is_xdp)
+ return;
- for (r = 0; r < s->n_rings; r++) {
- int bias = 0;
+ for (i = 0; i < tx_ring->cnt; i++) {
+ if (!tx_ring->txbufs[i].frag)
+ return;
- if (r >= num_stack_tx_rings)
- bias = num_stack_tx_rings;
+ nfp_net_dma_unmap_rx(dp, tx_ring->txbufs[i].dma_addr);
+ __free_page(virt_to_page(tx_ring->txbufs[i].frag));
+ }
+}
- nfp_net_tx_ring_init(&rings[r], &nn->r_vecs[r - bias], r);
+static int
+nfp_net_tx_ring_bufs_alloc(struct nfp_net_dp *dp,
+ struct nfp_net_tx_ring *tx_ring)
+{
+ struct nfp_net_tx_buf *txbufs = tx_ring->txbufs;
+ unsigned int i;
- if (nfp_net_tx_ring_alloc(&rings[r], s->dcnt, bias))
- goto err_free_prev;
+ if (!tx_ring->is_xdp)
+ return 0;
+
+ for (i = 0; i < tx_ring->cnt; i++) {
+ txbufs[i].frag = nfp_net_rx_alloc_one(dp, &txbufs[i].dma_addr);
+ if (!txbufs[i].frag) {
+ nfp_net_tx_ring_bufs_free(dp, tx_ring);
+ return -ENOMEM;
+ }
}
- return s->rings = rings;
-
-err_free_prev:
- while (r--)
- nfp_net_tx_ring_free(&rings[r]);
- kfree(rings);
- return NULL;
+ return 0;
}
-static void
-nfp_net_tx_ring_set_swap(struct nfp_net *nn, struct nfp_net_ring_set *s)
+static int nfp_net_tx_rings_prepare(struct nfp_net *nn, struct nfp_net_dp *dp)
{
- struct nfp_net_ring_set new = *s;
-
- s->dcnt = nn->txd_cnt;
- s->rings = nn->tx_rings;
- s->n_rings = nn->num_tx_rings;
-
- nn->txd_cnt = new.dcnt;
- nn->tx_rings = new.rings;
- nn->num_tx_rings = new.n_rings;
-}
-
-static void
-nfp_net_tx_ring_set_free(struct nfp_net *nn, struct nfp_net_ring_set *s)
-{
- struct nfp_net_tx_ring *rings = s->rings;
unsigned int r;
- for (r = 0; r < s->n_rings; r++)
- nfp_net_tx_ring_free(&rings[r]);
+ dp->tx_rings = kcalloc(dp->num_tx_rings, sizeof(*dp->tx_rings),
+ GFP_KERNEL);
+ if (!dp->tx_rings)
+ return -ENOMEM;
- kfree(rings);
+ for (r = 0; r < dp->num_tx_rings; r++) {
+ int bias = 0;
+
+ if (r >= dp->num_stack_tx_rings)
+ bias = dp->num_stack_tx_rings;
+
+ nfp_net_tx_ring_init(&dp->tx_rings[r], &nn->r_vecs[r - bias],
+ r, bias);
+
+ if (nfp_net_tx_ring_alloc(dp, &dp->tx_rings[r]))
+ goto err_free_prev;
+
+ if (nfp_net_tx_ring_bufs_alloc(dp, &dp->tx_rings[r]))
+ goto err_free_ring;
+ }
+
+ return 0;
+
+err_free_prev:
+ while (r--) {
+ nfp_net_tx_ring_bufs_free(dp, &dp->tx_rings[r]);
+err_free_ring:
+ nfp_net_tx_ring_free(&dp->tx_rings[r]);
+ }
+ kfree(dp->tx_rings);
+ return -ENOMEM;
+}
+
+static void nfp_net_tx_rings_free(struct nfp_net_dp *dp)
+{
+ unsigned int r;
+
+ for (r = 0; r < dp->num_tx_rings; r++) {
+ nfp_net_tx_ring_bufs_free(dp, &dp->tx_rings[r]);
+ nfp_net_tx_ring_free(&dp->tx_rings[r]);
+ }
+
+ kfree(dp->tx_rings);
}
/**
@@ -1851,13 +1909,12 @@
static void nfp_net_rx_ring_free(struct nfp_net_rx_ring *rx_ring)
{
struct nfp_net_r_vector *r_vec = rx_ring->r_vec;
- struct nfp_net *nn = r_vec->nfp_net;
- struct pci_dev *pdev = nn->pdev;
+ struct nfp_net_dp *dp = &r_vec->nfp_net->dp;
kfree(rx_ring->rxbufs);
if (rx_ring->rxds)
- dma_free_coherent(&pdev->dev, rx_ring->size,
+ dma_free_coherent(dp->dev, rx_ring->size,
rx_ring->rxds, rx_ring->dma);
rx_ring->cnt = 0;
@@ -1869,26 +1926,19 @@
/**
* nfp_net_rx_ring_alloc() - Allocate resource for a RX ring
+ * @dp: NFP Net data path struct
* @rx_ring: RX ring to allocate
- * @fl_bufsz: Size of buffers to allocate
- * @cnt: Ring buffer count
*
* Return: 0 on success, negative errno otherwise.
*/
static int
-nfp_net_rx_ring_alloc(struct nfp_net_rx_ring *rx_ring, unsigned int fl_bufsz,
- u32 cnt)
+nfp_net_rx_ring_alloc(struct nfp_net_dp *dp, struct nfp_net_rx_ring *rx_ring)
{
- struct nfp_net_r_vector *r_vec = rx_ring->r_vec;
- struct nfp_net *nn = r_vec->nfp_net;
- struct pci_dev *pdev = nn->pdev;
int sz;
- rx_ring->cnt = cnt;
- rx_ring->bufsz = fl_bufsz;
-
+ rx_ring->cnt = dp->rxd_cnt;
rx_ring->size = sizeof(*rx_ring->rxds) * rx_ring->cnt;
- rx_ring->rxds = dma_zalloc_coherent(&pdev->dev, rx_ring->size,
+ rx_ring->rxds = dma_zalloc_coherent(dp->dev, rx_ring->size,
&rx_ring->dma, GFP_KERNEL);
if (!rx_ring->rxds)
goto err_alloc;
@@ -1898,10 +1948,6 @@
if (!rx_ring->rxbufs)
goto err_alloc;
- nn_dbg(nn, "RxQ%02d: FlQCidx=%02d RxQCidx=%02d cnt=%d dma=%#llx host=%p\n",
- rx_ring->idx, rx_ring->fl_qcidx, rx_ring->rx_qcidx,
- rx_ring->cnt, (unsigned long long)rx_ring->dma, rx_ring->rxds);
-
return 0;
err_alloc:
@@ -1909,82 +1955,59 @@
return -ENOMEM;
}
-static struct nfp_net_rx_ring *
-nfp_net_rx_ring_set_prepare(struct nfp_net *nn, struct nfp_net_ring_set *s,
- bool xdp)
+static int nfp_net_rx_rings_prepare(struct nfp_net *nn, struct nfp_net_dp *dp)
{
- unsigned int fl_bufsz = nfp_net_calc_fl_bufsz(nn, s->mtu);
- struct nfp_net_rx_ring *rings;
unsigned int r;
- rings = kcalloc(s->n_rings, sizeof(*rings), GFP_KERNEL);
- if (!rings)
- return NULL;
+ dp->rx_rings = kcalloc(dp->num_rx_rings, sizeof(*dp->rx_rings),
+ GFP_KERNEL);
+ if (!dp->rx_rings)
+ return -ENOMEM;
- for (r = 0; r < s->n_rings; r++) {
- nfp_net_rx_ring_init(&rings[r], &nn->r_vecs[r], r);
+ for (r = 0; r < dp->num_rx_rings; r++) {
+ nfp_net_rx_ring_init(&dp->rx_rings[r], &nn->r_vecs[r], r);
- if (nfp_net_rx_ring_alloc(&rings[r], fl_bufsz, s->dcnt))
+ if (nfp_net_rx_ring_alloc(dp, &dp->rx_rings[r]))
goto err_free_prev;
- if (nfp_net_rx_ring_bufs_alloc(nn, &rings[r], xdp))
+ if (nfp_net_rx_ring_bufs_alloc(dp, &dp->rx_rings[r]))
goto err_free_ring;
}
- return s->rings = rings;
+ return 0;
err_free_prev:
while (r--) {
- nfp_net_rx_ring_bufs_free(nn, &rings[r], xdp);
+ nfp_net_rx_ring_bufs_free(dp, &dp->rx_rings[r]);
err_free_ring:
- nfp_net_rx_ring_free(&rings[r]);
+ nfp_net_rx_ring_free(&dp->rx_rings[r]);
}
- kfree(rings);
- return NULL;
+ kfree(dp->rx_rings);
+ return -ENOMEM;
}
-static void
-nfp_net_rx_ring_set_swap(struct nfp_net *nn, struct nfp_net_ring_set *s)
+static void nfp_net_rx_rings_free(struct nfp_net_dp *dp)
{
- struct nfp_net_ring_set new = *s;
-
- s->mtu = nn->netdev->mtu;
- s->dcnt = nn->rxd_cnt;
- s->rings = nn->rx_rings;
- s->n_rings = nn->num_rx_rings;
-
- nn->netdev->mtu = new.mtu;
- nn->fl_bufsz = nfp_net_calc_fl_bufsz(nn, new.mtu);
- nn->rxd_cnt = new.dcnt;
- nn->rx_rings = new.rings;
- nn->num_rx_rings = new.n_rings;
-}
-
-static void
-nfp_net_rx_ring_set_free(struct nfp_net *nn, struct nfp_net_ring_set *s,
- bool xdp)
-{
- struct nfp_net_rx_ring *rings = s->rings;
unsigned int r;
- for (r = 0; r < s->n_rings; r++) {
- nfp_net_rx_ring_bufs_free(nn, &rings[r], xdp);
- nfp_net_rx_ring_free(&rings[r]);
+ for (r = 0; r < dp->num_rx_rings; r++) {
+ nfp_net_rx_ring_bufs_free(dp, &dp->rx_rings[r]);
+ nfp_net_rx_ring_free(&dp->rx_rings[r]);
}
- kfree(rings);
+ kfree(dp->rx_rings);
}
static void
-nfp_net_vector_assign_rings(struct nfp_net *nn, struct nfp_net_r_vector *r_vec,
- int idx)
+nfp_net_vector_assign_rings(struct nfp_net_dp *dp,
+ struct nfp_net_r_vector *r_vec, int idx)
{
- r_vec->rx_ring = idx < nn->num_rx_rings ? &nn->rx_rings[idx] : NULL;
+ r_vec->rx_ring = idx < dp->num_rx_rings ? &dp->rx_rings[idx] : NULL;
r_vec->tx_ring =
- idx < nn->num_stack_tx_rings ? &nn->tx_rings[idx] : NULL;
+ idx < dp->num_stack_tx_rings ? &dp->tx_rings[idx] : NULL;
- r_vec->xdp_ring = idx < nn->num_tx_rings - nn->num_stack_tx_rings ?
- &nn->tx_rings[nn->num_stack_tx_rings + idx] : NULL;
+ r_vec->xdp_ring = idx < dp->num_tx_rings - dp->num_stack_tx_rings ?
+ &dp->tx_rings[dp->num_stack_tx_rings + idx] : NULL;
}
static int
@@ -1994,11 +2017,11 @@
int err;
/* Setup NAPI */
- netif_napi_add(nn->netdev, &r_vec->napi,
+ netif_napi_add(nn->dp.netdev, &r_vec->napi,
nfp_net_poll, NAPI_POLL_WEIGHT);
snprintf(r_vec->name, sizeof(r_vec->name),
- "%s-rxtx-%d", nn->netdev->name, idx);
+ "%s-rxtx-%d", nn->dp.netdev->name, idx);
err = request_irq(r_vec->irq_vector, r_vec->handler, 0, r_vec->name,
r_vec);
if (err) {
@@ -2045,7 +2068,7 @@
{
int i;
- for (i = 0; i < NFP_NET_CFG_RSS_KEY_SZ; i += 4)
+ for (i = 0; i < nfp_net_rss_key_sz(nn); i += 4)
nn_writel(nn, NFP_NET_CFG_RSS_KEY + i,
get_unaligned_le32(nn->rss_key + i));
}
@@ -2069,13 +2092,13 @@
/* copy RX interrupt coalesce parameters */
value = (nn->rx_coalesce_max_frames << 16) |
(factor * nn->rx_coalesce_usecs);
- for (i = 0; i < nn->num_rx_rings; i++)
+ for (i = 0; i < nn->dp.num_rx_rings; i++)
nn_writel(nn, NFP_NET_CFG_RXR_IRQ_MOD(i), value);
/* copy TX interrupt coalesce parameters */
value = (nn->tx_coalesce_max_frames << 16) |
(factor * nn->tx_coalesce_usecs);
- for (i = 0; i < nn->num_tx_rings; i++)
+ for (i = 0; i < nn->dp.num_tx_rings; i++)
nn_writel(nn, NFP_NET_CFG_TXR_IRQ_MOD(i), value);
}
@@ -2090,9 +2113,9 @@
static void nfp_net_write_mac_addr(struct nfp_net *nn)
{
nn_writel(nn, NFP_NET_CFG_MACADDR + 0,
- get_unaligned_be32(nn->netdev->dev_addr));
+ get_unaligned_be32(nn->dp.netdev->dev_addr));
nn_writew(nn, NFP_NET_CFG_MACADDR + 6,
- get_unaligned_be16(nn->netdev->dev_addr + 4));
+ get_unaligned_be16(nn->dp.netdev->dev_addr + 4));
}
static void nfp_net_vec_clear_ring_data(struct nfp_net *nn, unsigned int idx)
@@ -2116,7 +2139,7 @@
unsigned int r;
int err;
- new_ctrl = nn->ctrl;
+ new_ctrl = nn->dp.ctrl;
new_ctrl &= ~NFP_NET_CFG_CTRL_ENABLE;
update = NFP_NET_CFG_UPDATE_GEN;
update |= NFP_NET_CFG_UPDATE_MSIX;
@@ -2133,14 +2156,14 @@
if (err)
nn_err(nn, "Could not disable device: %d\n", err);
- for (r = 0; r < nn->num_rx_rings; r++)
- nfp_net_rx_ring_reset(&nn->rx_rings[r]);
- for (r = 0; r < nn->num_tx_rings; r++)
- nfp_net_tx_ring_reset(nn, &nn->tx_rings[r]);
- for (r = 0; r < nn->num_r_vecs; r++)
+ for (r = 0; r < nn->dp.num_rx_rings; r++)
+ nfp_net_rx_ring_reset(&nn->dp.rx_rings[r]);
+ for (r = 0; r < nn->dp.num_tx_rings; r++)
+ nfp_net_tx_ring_reset(&nn->dp, &nn->dp.tx_rings[r]);
+ for (r = 0; r < nn->dp.num_r_vecs; r++)
nfp_net_vec_clear_ring_data(nn, r);
- nn->ctrl = new_ctrl;
+ nn->dp.ctrl = new_ctrl;
}
static void
@@ -2162,13 +2185,17 @@
nn_writeb(nn, NFP_NET_CFG_TXR_VEC(idx), tx_ring->r_vec->irq_entry);
}
-static int __nfp_net_set_config_and_enable(struct nfp_net *nn)
+/**
+ * nfp_net_set_config_and_enable() - Write control BAR and enable NFP
+ * @nn: NFP Net device to reconfigure
+ */
+static int nfp_net_set_config_and_enable(struct nfp_net *nn)
{
- u32 new_ctrl, update = 0;
+ u32 bufsz, new_ctrl, update = 0;
unsigned int r;
int err;
- new_ctrl = nn->ctrl;
+ new_ctrl = nn->dp.ctrl;
if (nn->cap & NFP_NET_CFG_CTRL_RSS) {
nfp_net_rss_write_key(nn);
@@ -2184,22 +2211,23 @@
update |= NFP_NET_CFG_UPDATE_IRQMOD;
}
- for (r = 0; r < nn->num_tx_rings; r++)
- nfp_net_tx_ring_hw_cfg_write(nn, &nn->tx_rings[r], r);
- for (r = 0; r < nn->num_rx_rings; r++)
- nfp_net_rx_ring_hw_cfg_write(nn, &nn->rx_rings[r], r);
+ for (r = 0; r < nn->dp.num_tx_rings; r++)
+ nfp_net_tx_ring_hw_cfg_write(nn, &nn->dp.tx_rings[r], r);
+ for (r = 0; r < nn->dp.num_rx_rings; r++)
+ nfp_net_rx_ring_hw_cfg_write(nn, &nn->dp.rx_rings[r], r);
- nn_writeq(nn, NFP_NET_CFG_TXRS_ENABLE, nn->num_tx_rings == 64 ?
- 0xffffffffffffffffULL : ((u64)1 << nn->num_tx_rings) - 1);
+ nn_writeq(nn, NFP_NET_CFG_TXRS_ENABLE, nn->dp.num_tx_rings == 64 ?
+ 0xffffffffffffffffULL : ((u64)1 << nn->dp.num_tx_rings) - 1);
- nn_writeq(nn, NFP_NET_CFG_RXRS_ENABLE, nn->num_rx_rings == 64 ?
- 0xffffffffffffffffULL : ((u64)1 << nn->num_rx_rings) - 1);
+ nn_writeq(nn, NFP_NET_CFG_RXRS_ENABLE, nn->dp.num_rx_rings == 64 ?
+ 0xffffffffffffffffULL : ((u64)1 << nn->dp.num_rx_rings) - 1);
nfp_net_write_mac_addr(nn);
- nn_writel(nn, NFP_NET_CFG_MTU, nn->netdev->mtu);
- nn_writel(nn, NFP_NET_CFG_FLBUFSZ,
- nn->fl_bufsz - NFP_NET_RX_BUF_NON_DATA);
+ nn_writel(nn, NFP_NET_CFG_MTU, nn->dp.netdev->mtu);
+
+ bufsz = nn->dp.fl_bufsz - nn->dp.rx_dma_off - NFP_NET_RX_BUF_NON_DATA;
+ nn_writel(nn, NFP_NET_CFG_FLBUFSZ, bufsz);
/* Enable device */
new_ctrl |= NFP_NET_CFG_CTRL_ENABLE;
@@ -2211,37 +2239,26 @@
nn_writel(nn, NFP_NET_CFG_CTRL, new_ctrl);
err = nfp_net_reconfig(nn, update);
+ if (err) {
+ nfp_net_clear_config_and_disable(nn);
+ return err;
+ }
- nn->ctrl = new_ctrl;
+ nn->dp.ctrl = new_ctrl;
- for (r = 0; r < nn->num_rx_rings; r++)
- nfp_net_rx_ring_fill_freelist(&nn->rx_rings[r]);
+ for (r = 0; r < nn->dp.num_rx_rings; r++)
+ nfp_net_rx_ring_fill_freelist(&nn->dp, &nn->dp.rx_rings[r]);
/* Since reconfiguration requests while NFP is down are ignored we
* have to wipe the entire VXLAN configuration and reinitialize it.
*/
- if (nn->ctrl & NFP_NET_CFG_CTRL_VXLAN) {
+ if (nn->dp.ctrl & NFP_NET_CFG_CTRL_VXLAN) {
memset(&nn->vxlan_ports, 0, sizeof(nn->vxlan_ports));
memset(&nn->vxlan_usecnt, 0, sizeof(nn->vxlan_usecnt));
- udp_tunnel_get_rx_info(nn->netdev);
+ udp_tunnel_get_rx_info(nn->dp.netdev);
}
- return err;
-}
-
-/**
- * nfp_net_set_config_and_enable() - Write control BAR and enable NFP
- * @nn: NFP Net device to reconfigure
- */
-static int nfp_net_set_config_and_enable(struct nfp_net *nn)
-{
- int err;
-
- err = __nfp_net_set_config_and_enable(nn);
- if (err)
- nfp_net_clear_config_and_disable(nn);
-
- return err;
+ return 0;
}
/**
@@ -2252,12 +2269,12 @@
{
unsigned int r;
- for (r = 0; r < nn->num_r_vecs; r++) {
+ for (r = 0; r < nn->dp.num_r_vecs; r++) {
napi_enable(&nn->r_vecs[r].napi);
enable_irq(nn->r_vecs[r].irq_vector);
}
- netif_tx_wake_all_queues(nn->netdev);
+ netif_tx_wake_all_queues(nn->dp.netdev);
enable_irq(nn->irq_entries[NFP_NET_IRQ_LSC_IDX].vector);
nfp_net_read_link_status(nn);
@@ -2266,22 +2283,8 @@
static int nfp_net_netdev_open(struct net_device *netdev)
{
struct nfp_net *nn = netdev_priv(netdev);
- struct nfp_net_ring_set rx = {
- .n_rings = nn->num_rx_rings,
- .mtu = nn->netdev->mtu,
- .dcnt = nn->rxd_cnt,
- };
- struct nfp_net_ring_set tx = {
- .n_rings = nn->num_tx_rings,
- .dcnt = nn->txd_cnt,
- };
int err, r;
- if (nn->ctrl & NFP_NET_CFG_CTRL_ENABLE) {
- nn_err(nn, "Dev is already enabled: 0x%08x\n", nn->ctrl);
- return -EBUSY;
- }
-
/* Step 1: Allocate resources for rings and the like
* - Request interrupts
* - Allocate RX and TX ring resources
@@ -2299,33 +2302,28 @@
goto err_free_exn;
disable_irq(nn->irq_entries[NFP_NET_IRQ_LSC_IDX].vector);
- for (r = 0; r < nn->num_r_vecs; r++) {
+ for (r = 0; r < nn->dp.num_r_vecs; r++) {
err = nfp_net_prepare_vector(nn, &nn->r_vecs[r], r);
if (err)
goto err_cleanup_vec_p;
}
- nn->rx_rings = nfp_net_rx_ring_set_prepare(nn, &rx, nn->xdp_prog);
- if (!nn->rx_rings) {
- err = -ENOMEM;
+ err = nfp_net_rx_rings_prepare(nn, &nn->dp);
+ if (err)
goto err_cleanup_vec;
- }
- nn->tx_rings = nfp_net_tx_ring_set_prepare(nn, &tx,
- nn->num_stack_tx_rings);
- if (!nn->tx_rings) {
- err = -ENOMEM;
+ err = nfp_net_tx_rings_prepare(nn, &nn->dp);
+ if (err)
goto err_free_rx_rings;
- }
for (r = 0; r < nn->max_r_vecs; r++)
- nfp_net_vector_assign_rings(nn, &nn->r_vecs[r], r);
+ nfp_net_vector_assign_rings(&nn->dp, &nn->r_vecs[r], r);
- err = netif_set_real_num_tx_queues(netdev, nn->num_stack_tx_rings);
+ err = netif_set_real_num_tx_queues(netdev, nn->dp.num_stack_tx_rings);
if (err)
goto err_free_rings;
- err = netif_set_real_num_rx_queues(netdev, nn->num_rx_rings);
+ err = netif_set_real_num_rx_queues(netdev, nn->dp.num_rx_rings);
if (err)
goto err_free_rings;
@@ -2351,11 +2349,11 @@
return 0;
err_free_rings:
- nfp_net_tx_ring_set_free(nn, &tx);
+ nfp_net_tx_rings_free(&nn->dp);
err_free_rx_rings:
- nfp_net_rx_ring_set_free(nn, &rx, nn->xdp_prog);
+ nfp_net_rx_rings_free(&nn->dp);
err_cleanup_vec:
- r = nn->num_r_vecs;
+ r = nn->dp.num_r_vecs;
err_cleanup_vec_p:
while (r--)
nfp_net_cleanup_vector(nn, &nn->r_vecs[r]);
@@ -2374,15 +2372,15 @@
unsigned int r;
disable_irq(nn->irq_entries[NFP_NET_IRQ_LSC_IDX].vector);
- netif_carrier_off(nn->netdev);
+ netif_carrier_off(nn->dp.netdev);
nn->link_up = false;
- for (r = 0; r < nn->num_r_vecs; r++) {
+ for (r = 0; r < nn->dp.num_r_vecs; r++) {
disable_irq(nn->r_vecs[r].irq_vector);
napi_disable(&nn->r_vecs[r].napi);
}
- netif_tx_disable(nn->netdev);
+ netif_tx_disable(nn->dp.netdev);
}
/**
@@ -2393,17 +2391,19 @@
{
unsigned int r;
- for (r = 0; r < nn->num_rx_rings; r++) {
- nfp_net_rx_ring_bufs_free(nn, &nn->rx_rings[r], nn->xdp_prog);
- nfp_net_rx_ring_free(&nn->rx_rings[r]);
+ for (r = 0; r < nn->dp.num_rx_rings; r++) {
+ nfp_net_rx_ring_bufs_free(&nn->dp, &nn->dp.rx_rings[r]);
+ nfp_net_rx_ring_free(&nn->dp.rx_rings[r]);
}
- for (r = 0; r < nn->num_tx_rings; r++)
- nfp_net_tx_ring_free(&nn->tx_rings[r]);
- for (r = 0; r < nn->num_r_vecs; r++)
+ for (r = 0; r < nn->dp.num_tx_rings; r++) {
+ nfp_net_tx_ring_bufs_free(&nn->dp, &nn->dp.tx_rings[r]);
+ nfp_net_tx_ring_free(&nn->dp.tx_rings[r]);
+ }
+ for (r = 0; r < nn->dp.num_r_vecs; r++)
nfp_net_cleanup_vector(nn, &nn->r_vecs[r]);
- kfree(nn->rx_rings);
- kfree(nn->tx_rings);
+ kfree(nn->dp.rx_rings);
+ kfree(nn->dp.tx_rings);
nfp_net_aux_irq_free(nn, NFP_NET_CFG_LSC, NFP_NET_IRQ_LSC_IDX);
nfp_net_aux_irq_free(nn, NFP_NET_CFG_EXN, NFP_NET_IRQ_EXN_IDX);
@@ -2417,11 +2417,6 @@
{
struct nfp_net *nn = netdev_priv(netdev);
- if (!(nn->ctrl & NFP_NET_CFG_CTRL_ENABLE)) {
- nn_err(nn, "Dev is not up: 0x%08x\n", nn->ctrl);
- return 0;
- }
-
/* Step 1: Disable RX and TX rings from the Linux kernel perspective
*/
nfp_net_close_stack(nn);
@@ -2443,7 +2438,7 @@
struct nfp_net *nn = netdev_priv(netdev);
u32 new_ctrl;
- new_ctrl = nn->ctrl;
+ new_ctrl = nn->dp.ctrl;
if (netdev->flags & IFF_PROMISC) {
if (nn->cap & NFP_NET_CFG_CTRL_PROMISC)
@@ -2454,13 +2449,13 @@
new_ctrl &= ~NFP_NET_CFG_CTRL_PROMISC;
}
- if (new_ctrl == nn->ctrl)
+ if (new_ctrl == nn->dp.ctrl)
return;
nn_writel(nn, NFP_NET_CFG_CTRL, new_ctrl);
nfp_net_reconfig_post(nn, NFP_NET_CFG_UPDATE_GEN);
- nn->ctrl = new_ctrl;
+ nn->dp.ctrl = new_ctrl;
}
static void nfp_net_rss_init_itbl(struct nfp_net *nn)
@@ -2469,181 +2464,174 @@
for (i = 0; i < sizeof(nn->rss_itbl); i++)
nn->rss_itbl[i] =
- ethtool_rxfh_indir_default(i, nn->num_rx_rings);
+ ethtool_rxfh_indir_default(i, nn->dp.num_rx_rings);
}
-static int
-nfp_net_ring_swap_enable(struct nfp_net *nn, unsigned int *num_vecs,
- unsigned int *stack_tx_rings,
- struct bpf_prog **xdp_prog,
- struct nfp_net_ring_set *rx,
- struct nfp_net_ring_set *tx)
+static void nfp_net_dp_swap(struct nfp_net *nn, struct nfp_net_dp *dp)
+{
+ struct nfp_net_dp new_dp = *dp;
+
+ *dp = nn->dp;
+ nn->dp = new_dp;
+
+ nn->dp.netdev->mtu = new_dp.mtu;
+
+ if (!netif_is_rxfh_configured(nn->dp.netdev))
+ nfp_net_rss_init_itbl(nn);
+}
+
+static int nfp_net_dp_swap_enable(struct nfp_net *nn, struct nfp_net_dp *dp)
{
unsigned int r;
int err;
- if (rx)
- nfp_net_rx_ring_set_swap(nn, rx);
- if (tx)
- nfp_net_tx_ring_set_swap(nn, tx);
-
- swap(*num_vecs, nn->num_r_vecs);
- swap(*stack_tx_rings, nn->num_stack_tx_rings);
- *xdp_prog = xchg(&nn->xdp_prog, *xdp_prog);
+ nfp_net_dp_swap(nn, dp);
for (r = 0; r < nn->max_r_vecs; r++)
- nfp_net_vector_assign_rings(nn, &nn->r_vecs[r], r);
+ nfp_net_vector_assign_rings(&nn->dp, &nn->r_vecs[r], r);
- if (!netif_is_rxfh_configured(nn->netdev))
- nfp_net_rss_init_itbl(nn);
-
- err = netif_set_real_num_rx_queues(nn->netdev,
- nn->num_rx_rings);
+ err = netif_set_real_num_rx_queues(nn->dp.netdev, nn->dp.num_rx_rings);
if (err)
return err;
- if (nn->netdev->real_num_tx_queues != nn->num_stack_tx_rings) {
- err = netif_set_real_num_tx_queues(nn->netdev,
- nn->num_stack_tx_rings);
+ if (nn->dp.netdev->real_num_tx_queues != nn->dp.num_stack_tx_rings) {
+ err = netif_set_real_num_tx_queues(nn->dp.netdev,
+ nn->dp.num_stack_tx_rings);
if (err)
return err;
}
- return __nfp_net_set_config_and_enable(nn);
+ return nfp_net_set_config_and_enable(nn);
+}
+
+struct nfp_net_dp *nfp_net_clone_dp(struct nfp_net *nn)
+{
+ struct nfp_net_dp *new;
+
+ new = kmalloc(sizeof(*new), GFP_KERNEL);
+ if (!new)
+ return NULL;
+
+ *new = nn->dp;
+
+ /* Clear things which need to be recomputed */
+ new->fl_bufsz = 0;
+ new->tx_rings = NULL;
+ new->rx_rings = NULL;
+ new->num_r_vecs = 0;
+ new->num_stack_tx_rings = 0;
+
+ return new;
}
static int
-nfp_net_check_config(struct nfp_net *nn, struct bpf_prog *xdp_prog,
- struct nfp_net_ring_set *rx, struct nfp_net_ring_set *tx)
+nfp_net_check_config(struct nfp_net *nn, struct nfp_net_dp *dp,
+ struct netlink_ext_ack *extack)
{
/* XDP-enabled tests */
- if (!xdp_prog)
+ if (!dp->xdp_prog)
return 0;
- if (rx && nfp_net_calc_fl_bufsz(nn, rx->mtu) > PAGE_SIZE) {
- nn_warn(nn, "MTU too large w/ XDP enabled\n");
+ if (dp->fl_bufsz > PAGE_SIZE) {
+ NL_SET_ERR_MSG_MOD(extack, "MTU too large w/ XDP enabled");
return -EINVAL;
}
- if (tx && tx->n_rings > nn->max_tx_rings) {
- nn_warn(nn, "Insufficient number of TX rings w/ XDP enabled\n");
+ if (dp->num_tx_rings > nn->max_tx_rings) {
+ NL_SET_ERR_MSG_MOD(extack, "Insufficient number of TX rings w/ XDP enabled");
return -EINVAL;
}
return 0;
}
-static void
-nfp_net_ring_reconfig_down(struct nfp_net *nn, struct bpf_prog **xdp_prog,
- struct nfp_net_ring_set *rx,
- struct nfp_net_ring_set *tx,
- unsigned int stack_tx_rings, unsigned int num_vecs)
+int nfp_net_ring_reconfig(struct nfp_net *nn, struct nfp_net_dp *dp,
+ struct netlink_ext_ack *extack)
{
- nn->netdev->mtu = rx ? rx->mtu : nn->netdev->mtu;
- nn->fl_bufsz = nfp_net_calc_fl_bufsz(nn, nn->netdev->mtu);
- nn->rxd_cnt = rx ? rx->dcnt : nn->rxd_cnt;
- nn->txd_cnt = tx ? tx->dcnt : nn->txd_cnt;
- nn->num_rx_rings = rx ? rx->n_rings : nn->num_rx_rings;
- nn->num_tx_rings = tx ? tx->n_rings : nn->num_tx_rings;
- nn->num_stack_tx_rings = stack_tx_rings;
- nn->num_r_vecs = num_vecs;
- *xdp_prog = xchg(&nn->xdp_prog, *xdp_prog);
+ int r, err;
- if (!netif_is_rxfh_configured(nn->netdev))
- nfp_net_rss_init_itbl(nn);
-}
+ dp->fl_bufsz = nfp_net_calc_fl_bufsz(dp);
-int
-nfp_net_ring_reconfig(struct nfp_net *nn, struct bpf_prog **xdp_prog,
- struct nfp_net_ring_set *rx, struct nfp_net_ring_set *tx)
-{
- unsigned int stack_tx_rings, num_vecs, r;
- int err;
+ dp->num_stack_tx_rings = dp->num_tx_rings;
+ if (dp->xdp_prog)
+ dp->num_stack_tx_rings -= dp->num_rx_rings;
- stack_tx_rings = tx ? tx->n_rings : nn->num_tx_rings;
- if (*xdp_prog)
- stack_tx_rings -= rx ? rx->n_rings : nn->num_rx_rings;
+ dp->num_r_vecs = max(dp->num_rx_rings, dp->num_stack_tx_rings);
- num_vecs = max(rx ? rx->n_rings : nn->num_rx_rings, stack_tx_rings);
-
- err = nfp_net_check_config(nn, *xdp_prog, rx, tx);
+ err = nfp_net_check_config(nn, dp, extack);
if (err)
- return err;
+ goto exit_free_dp;
- if (!netif_running(nn->netdev)) {
- nfp_net_ring_reconfig_down(nn, xdp_prog, rx, tx,
- stack_tx_rings, num_vecs);
- return 0;
+ if (!netif_running(dp->netdev)) {
+ nfp_net_dp_swap(nn, dp);
+ err = 0;
+ goto exit_free_dp;
}
/* Prepare new rings */
- for (r = nn->num_r_vecs; r < num_vecs; r++) {
+ for (r = nn->dp.num_r_vecs; r < dp->num_r_vecs; r++) {
err = nfp_net_prepare_vector(nn, &nn->r_vecs[r], r);
if (err) {
- num_vecs = r;
+ dp->num_r_vecs = r;
goto err_cleanup_vecs;
}
}
- if (rx) {
- if (!nfp_net_rx_ring_set_prepare(nn, rx, *xdp_prog)) {
- err = -ENOMEM;
- goto err_cleanup_vecs;
- }
- }
- if (tx) {
- if (!nfp_net_tx_ring_set_prepare(nn, tx, stack_tx_rings)) {
- err = -ENOMEM;
- goto err_free_rx;
- }
- }
+
+ err = nfp_net_rx_rings_prepare(nn, dp);
+ if (err)
+ goto err_cleanup_vecs;
+
+ err = nfp_net_tx_rings_prepare(nn, dp);
+ if (err)
+ goto err_free_rx;
/* Stop device, swap in new rings, try to start the firmware */
nfp_net_close_stack(nn);
nfp_net_clear_config_and_disable(nn);
- err = nfp_net_ring_swap_enable(nn, &num_vecs, &stack_tx_rings,
- xdp_prog, rx, tx);
+ err = nfp_net_dp_swap_enable(nn, dp);
if (err) {
int err2;
nfp_net_clear_config_and_disable(nn);
/* Try with old configuration and old rings */
- err2 = nfp_net_ring_swap_enable(nn, &num_vecs, &stack_tx_rings,
- xdp_prog, rx, tx);
+ err2 = nfp_net_dp_swap_enable(nn, dp);
if (err2)
nn_err(nn, "Can't restore ring config - FW communication failed (%d,%d)\n",
err, err2);
}
- for (r = num_vecs - 1; r >= nn->num_r_vecs; r--)
+ for (r = dp->num_r_vecs - 1; r >= nn->dp.num_r_vecs; r--)
nfp_net_cleanup_vector(nn, &nn->r_vecs[r]);
- if (rx)
- nfp_net_rx_ring_set_free(nn, rx, *xdp_prog);
- if (tx)
- nfp_net_tx_ring_set_free(nn, tx);
+ nfp_net_rx_rings_free(dp);
+ nfp_net_tx_rings_free(dp);
nfp_net_open_stack(nn);
+exit_free_dp:
+ kfree(dp);
return err;
err_free_rx:
- if (rx)
- nfp_net_rx_ring_set_free(nn, rx, *xdp_prog);
+ nfp_net_rx_rings_free(dp);
err_cleanup_vecs:
- for (r = num_vecs - 1; r >= nn->num_r_vecs; r--)
+ for (r = dp->num_r_vecs - 1; r >= nn->dp.num_r_vecs; r--)
nfp_net_cleanup_vector(nn, &nn->r_vecs[r]);
+ kfree(dp);
return err;
}
static int nfp_net_change_mtu(struct net_device *netdev, int new_mtu)
{
struct nfp_net *nn = netdev_priv(netdev);
- struct nfp_net_ring_set rx = {
- .n_rings = nn->num_rx_rings,
- .mtu = new_mtu,
- .dcnt = nn->rxd_cnt,
- };
+ struct nfp_net_dp *dp;
- return nfp_net_ring_reconfig(nn, &nn->xdp_prog, &rx, NULL);
+ dp = nfp_net_clone_dp(nn);
+ if (!dp)
+ return -ENOMEM;
+
+ dp->mtu = new_mtu;
+
+ return nfp_net_ring_reconfig(nn, dp, NULL);
}
static void nfp_net_stat64(struct net_device *netdev,
@@ -2652,7 +2640,7 @@
struct nfp_net *nn = netdev_priv(netdev);
int r;
- for (r = 0; r < nn->num_r_vecs; r++) {
+ for (r = 0; r < nn->dp.num_r_vecs; r++) {
struct nfp_net_r_vector *r_vec = &nn->r_vecs[r];
u64 data[3];
unsigned int start;
@@ -2694,12 +2682,12 @@
struct nfp_net *nn = netdev_priv(netdev);
if (TC_H_MAJ(handle) != TC_H_MAJ(TC_H_INGRESS))
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
if (proto != htons(ETH_P_ALL))
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
if (tc->type == TC_SETUP_CLSBPF && nfp_net_ebpf_capable(nn)) {
- if (!nn->bpf_offload_xdp)
+ if (!nn->dp.bpf_offload_xdp)
return nfp_net_bpf_offload(nn, tc->cls_bpf);
else
return -EBUSY;
@@ -2718,7 +2706,7 @@
/* Assume this is not called with features we have not advertised */
- new_ctrl = nn->ctrl;
+ new_ctrl = nn->dp.ctrl;
if (changed & NETIF_F_RXCSUM) {
if (features & NETIF_F_RXCSUM)
@@ -2762,7 +2750,7 @@
new_ctrl &= ~NFP_NET_CFG_CTRL_GATHER;
}
- if (changed & NETIF_F_HW_TC && nn->ctrl & NFP_NET_CFG_CTRL_BPF) {
+ if (changed & NETIF_F_HW_TC && nn->dp.ctrl & NFP_NET_CFG_CTRL_BPF) {
nn_err(nn, "Cannot disable HW TC offload while in use\n");
return -EBUSY;
}
@@ -2770,16 +2758,16 @@
nn_dbg(nn, "Feature change 0x%llx -> 0x%llx (changed=0x%llx)\n",
netdev->features, features, changed);
- if (new_ctrl == nn->ctrl)
+ if (new_ctrl == nn->dp.ctrl)
return 0;
- nn_dbg(nn, "NIC ctrl: 0x%x -> 0x%x\n", nn->ctrl, new_ctrl);
+ nn_dbg(nn, "NIC ctrl: 0x%x -> 0x%x\n", nn->dp.ctrl, new_ctrl);
nn_writel(nn, NFP_NET_CFG_CTRL, new_ctrl);
err = nfp_net_reconfig(nn, NFP_NET_CFG_UPDATE_GEN);
if (err)
return err;
- nn->ctrl = new_ctrl;
+ nn->dp.ctrl = new_ctrl;
return 0;
}
@@ -2830,6 +2818,26 @@
return features;
}
+static int
+nfp_net_get_phys_port_name(struct net_device *netdev, char *name, size_t len)
+{
+ struct nfp_net *nn = netdev_priv(netdev);
+ int err;
+
+ if (!nn->eth_port)
+ return -EOPNOTSUPP;
+
+ if (!nn->eth_port->is_split)
+ err = snprintf(name, len, "p%d", nn->eth_port->label_port);
+ else
+ err = snprintf(name, len, "p%ds%d", nn->eth_port->label_port,
+ nn->eth_port->label_subport);
+ if (err >= len)
+ return -EINVAL;
+
+ return 0;
+}
+
/**
* nfp_net_set_vxlan_port() - set vxlan port in SW and reconfigure HW
* @nn: NFP Net device to reconfigure
@@ -2842,7 +2850,7 @@
nn->vxlan_ports[idx] = port;
- if (!(nn->ctrl & NFP_NET_CFG_CTRL_VXLAN))
+ if (!(nn->dp.ctrl & NFP_NET_CFG_CTRL_VXLAN))
return;
BUILD_BUG_ON(NFP_NET_N_VXLAN_PORTS & 1);
@@ -2921,8 +2929,8 @@
if (!nfp_net_ebpf_capable(nn))
return -EINVAL;
- if (nn->ctrl & NFP_NET_CFG_CTRL_BPF) {
- if (!nn->bpf_offload_xdp)
+ if (nn->dp.ctrl & NFP_NET_CFG_CTRL_BPF) {
+ if (!nn->dp.bpf_offload_xdp)
return prog ? -EBUSY : 0;
cmd.command = prog ? TC_CLSBPF_REPLACE : TC_CLSBPF_DESTROY;
} else {
@@ -2935,48 +2943,44 @@
/* Stop offload if replace not possible */
if (ret && cmd.command == TC_CLSBPF_REPLACE)
nfp_net_xdp_offload(nn, NULL);
- nn->bpf_offload_xdp = prog && !ret;
+ nn->dp.bpf_offload_xdp = prog && !ret;
return ret;
}
-static int nfp_net_xdp_setup(struct nfp_net *nn, struct bpf_prog *prog)
+static int nfp_net_xdp_setup(struct nfp_net *nn, struct netdev_xdp *xdp)
{
- struct nfp_net_ring_set rx = {
- .n_rings = nn->num_rx_rings,
- .mtu = nn->netdev->mtu,
- .dcnt = nn->rxd_cnt,
- };
- struct nfp_net_ring_set tx = {
- .n_rings = nn->num_tx_rings,
- .dcnt = nn->txd_cnt,
- };
+ struct bpf_prog *old_prog = nn->dp.xdp_prog;
+ struct bpf_prog *prog = xdp->prog;
+ struct nfp_net_dp *dp;
int err;
- if (prog && prog->xdp_adjust_head) {
- nn_err(nn, "Does not support bpf_xdp_adjust_head()\n");
- return -EOPNOTSUPP;
- }
- if (!prog && !nn->xdp_prog)
+ if (!prog && !nn->dp.xdp_prog)
return 0;
- if (prog && nn->xdp_prog) {
- prog = xchg(&nn->xdp_prog, prog);
+ if (prog && nn->dp.xdp_prog) {
+ prog = xchg(&nn->dp.xdp_prog, prog);
bpf_prog_put(prog);
- nfp_net_xdp_offload(nn, nn->xdp_prog);
+ nfp_net_xdp_offload(nn, nn->dp.xdp_prog);
return 0;
}
- tx.n_rings += prog ? nn->num_rx_rings : -nn->num_rx_rings;
+ dp = nfp_net_clone_dp(nn);
+ if (!dp)
+ return -ENOMEM;
+
+ dp->xdp_prog = prog;
+ dp->num_tx_rings += prog ? nn->dp.num_rx_rings : -nn->dp.num_rx_rings;
+ dp->rx_dma_dir = prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE;
+ dp->rx_dma_off = prog ? XDP_PACKET_HEADROOM - nn->dp.rx_offset : 0;
/* We need RX reconfig to remap the buffers (BIDIR vs FROM_DEV) */
- err = nfp_net_ring_reconfig(nn, &prog, &rx, &tx);
+ err = nfp_net_ring_reconfig(nn, dp, xdp->extack);
if (err)
return err;
- /* @prog got swapped and is now the old one */
- if (prog)
- bpf_prog_put(prog);
+ if (old_prog)
+ bpf_prog_put(old_prog);
- nfp_net_xdp_offload(nn, nn->xdp_prog);
+ nfp_net_xdp_offload(nn, nn->dp.xdp_prog);
return 0;
}
@@ -2987,9 +2991,9 @@
switch (xdp->command) {
case XDP_SETUP_PROG:
- return nfp_net_xdp_setup(nn, xdp->prog);
+ return nfp_net_xdp_setup(nn, xdp);
case XDP_QUERY_PROG:
- xdp->prog_attached = !!nn->xdp_prog;
+ xdp->prog_attached = !!nn->dp.xdp_prog;
return 0;
default:
return -EINVAL;
@@ -3008,6 +3012,7 @@
.ndo_set_mac_address = eth_mac_addr,
.ndo_set_features = nfp_net_set_features,
.ndo_features_check = nfp_net_features_check,
+ .ndo_get_phys_port_name = nfp_net_get_phys_port_name,
.ndo_udp_tunnel_add = nfp_net_add_vxlan_port,
.ndo_udp_tunnel_del = nfp_net_del_vxlan_port,
.ndo_xdp = nfp_net_xdp,
@@ -3020,9 +3025,9 @@
void nfp_net_info(struct nfp_net *nn)
{
nn_info(nn, "Netronome NFP-6xxx %sNetdev: TxQs=%d/%d RxQs=%d/%d\n",
- nn->is_vf ? "VF " : "",
- nn->num_tx_rings, nn->max_tx_rings,
- nn->num_rx_rings, nn->max_rx_rings);
+ nn->dp.is_vf ? "VF " : "",
+ nn->dp.num_tx_rings, nn->max_tx_rings,
+ nn->dp.num_rx_rings, nn->max_rx_rings);
nn_info(nn, "VER: %d.%d.%d.%d, Maximum supported MTU: %d\n",
nn->fw_ver.resv, nn->fw_ver.class,
nn->fw_ver.major, nn->fw_ver.minor,
@@ -3074,21 +3079,24 @@
SET_NETDEV_DEV(netdev, &pdev->dev);
nn = netdev_priv(netdev);
- nn->netdev = netdev;
+ nn->dp.netdev = netdev;
+ nn->dp.dev = &pdev->dev;
nn->pdev = pdev;
nn->max_tx_rings = max_tx_rings;
nn->max_rx_rings = max_rx_rings;
- nn->num_tx_rings = min_t(unsigned int, max_tx_rings, num_online_cpus());
- nn->num_rx_rings = min_t(unsigned int, max_rx_rings,
+ nn->dp.num_tx_rings = min_t(unsigned int,
+ max_tx_rings, num_online_cpus());
+ nn->dp.num_rx_rings = min_t(unsigned int, max_rx_rings,
netif_get_num_default_rss_queues());
- nn->num_r_vecs = max(nn->num_tx_rings, nn->num_rx_rings);
- nn->num_r_vecs = min_t(unsigned int, nn->num_r_vecs, num_online_cpus());
+ nn->dp.num_r_vecs = max(nn->dp.num_tx_rings, nn->dp.num_rx_rings);
+ nn->dp.num_r_vecs = min_t(unsigned int,
+ nn->dp.num_r_vecs, num_online_cpus());
- nn->txd_cnt = NFP_NET_TX_DESCS_DEFAULT;
- nn->rxd_cnt = NFP_NET_RX_DESCS_DEFAULT;
+ nn->dp.txd_cnt = NFP_NET_TX_DESCS_DEFAULT;
+ nn->dp.rxd_cnt = NFP_NET_RX_DESCS_DEFAULT;
spin_lock_init(&nn->reconfig_lock);
spin_lock_init(&nn->rx_filter_lock);
@@ -3108,7 +3116,28 @@
*/
void nfp_net_netdev_free(struct nfp_net *nn)
{
- free_netdev(nn->netdev);
+ free_netdev(nn->dp.netdev);
+}
+
+/**
+ * nfp_net_rss_key_sz() - Get current size of the RSS key
+ * @nn: NFP Net device instance
+ *
+ * Return: size of the RSS key for currently selected hash function.
+ */
+unsigned int nfp_net_rss_key_sz(struct nfp_net *nn)
+{
+ switch (nn->rss_hfunc) {
+ case ETH_RSS_HASH_TOP:
+ return NFP_NET_CFG_RSS_KEY_SZ;
+ case ETH_RSS_HASH_XOR:
+ return 0;
+ case ETH_RSS_HASH_CRC32:
+ return 4;
+ }
+
+ nn_warn(nn, "Unknown hash function: %u\n", nn->rss_hfunc);
+ return 0;
}
/**
@@ -3117,14 +3146,32 @@
*/
static void nfp_net_rss_init(struct nfp_net *nn)
{
- netdev_rss_key_fill(nn->rss_key, NFP_NET_CFG_RSS_KEY_SZ);
+ unsigned long func_bit, rss_cap_hfunc;
+ u32 reg;
+
+ /* Read the RSS function capability and select first supported func */
+ reg = nn_readl(nn, NFP_NET_CFG_RSS_CAP);
+ rss_cap_hfunc = FIELD_GET(NFP_NET_CFG_RSS_CAP_HFUNC, reg);
+ if (!rss_cap_hfunc)
+ rss_cap_hfunc = FIELD_GET(NFP_NET_CFG_RSS_CAP_HFUNC,
+ NFP_NET_CFG_RSS_TOEPLITZ);
+
+ func_bit = find_first_bit(&rss_cap_hfunc, NFP_NET_CFG_RSS_HFUNCS);
+ if (func_bit == NFP_NET_CFG_RSS_HFUNCS) {
+ dev_warn(nn->dp.dev,
+ "Bad RSS config, defaulting to Toeplitz hash\n");
+ func_bit = ETH_RSS_HASH_TOP_BIT;
+ }
+ nn->rss_hfunc = 1 << func_bit;
+
+ netdev_rss_key_fill(nn->rss_key, nfp_net_rss_key_sz(nn));
nfp_net_rss_init_itbl(nn);
/* Enable IPv4/IPv6 TCP by default */
nn->rss_cfg = NFP_NET_CFG_RSS_IPV4_TCP |
NFP_NET_CFG_RSS_IPV6_TCP |
- NFP_NET_CFG_RSS_TOEPLITZ |
+ FIELD_PREP(NFP_NET_CFG_RSS_HFUNC, nn->rss_hfunc) |
NFP_NET_CFG_RSS_MASK;
}
@@ -3151,6 +3198,10 @@
struct nfp_net *nn = netdev_priv(netdev);
int err;
+ nn->dp.chained_metadata_format = nn->fw_ver.major > 3;
+
+ nn->dp.rx_dma_dir = DMA_FROM_DEVICE;
+
/* Get some of the read-only fields from the BAR */
nn->cap = nn_readl(nn, NFP_NET_CFG_CAP);
nn->max_mtu = nn_readl(nn, NFP_NET_CFG_MAX_MTU);
@@ -3158,17 +3209,26 @@
nfp_net_write_mac_addr(nn);
/* Determine RX packet/metadata boundary offset */
- if (nn->fw_ver.major >= 2)
- nn->rx_offset = nn_readl(nn, NFP_NET_CFG_RX_OFFSET);
- else
- nn->rx_offset = NFP_NET_RX_OFFSET;
+ if (nn->fw_ver.major >= 2) {
+ u32 reg;
+
+ reg = nn_readl(nn, NFP_NET_CFG_RX_OFFSET);
+ if (reg > NFP_NET_MAX_PREPEND) {
+ nn_err(nn, "Invalid rx offset: %d\n", reg);
+ return -EINVAL;
+ }
+ nn->dp.rx_offset = reg;
+ } else {
+ nn->dp.rx_offset = NFP_NET_RX_OFFSET;
+ }
/* Set default MTU and Freelist buffer size */
if (nn->max_mtu < NFP_NET_DEFAULT_MTU)
netdev->mtu = nn->max_mtu;
else
netdev->mtu = NFP_NET_DEFAULT_MTU;
- nn->fl_bufsz = nfp_net_calc_fl_bufsz(nn, netdev->mtu);
+ nn->dp.mtu = netdev->mtu;
+ nn->dp.fl_bufsz = nfp_net_calc_fl_bufsz(&nn->dp);
/* Advertise/enable offloads based on capabilities
*
@@ -3179,31 +3239,31 @@
netdev->hw_features = NETIF_F_HIGHDMA;
if (nn->cap & NFP_NET_CFG_CTRL_RXCSUM) {
netdev->hw_features |= NETIF_F_RXCSUM;
- nn->ctrl |= NFP_NET_CFG_CTRL_RXCSUM;
+ nn->dp.ctrl |= NFP_NET_CFG_CTRL_RXCSUM;
}
if (nn->cap & NFP_NET_CFG_CTRL_TXCSUM) {
netdev->hw_features |= NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM;
- nn->ctrl |= NFP_NET_CFG_CTRL_TXCSUM;
+ nn->dp.ctrl |= NFP_NET_CFG_CTRL_TXCSUM;
}
if (nn->cap & NFP_NET_CFG_CTRL_GATHER) {
netdev->hw_features |= NETIF_F_SG;
- nn->ctrl |= NFP_NET_CFG_CTRL_GATHER;
+ nn->dp.ctrl |= NFP_NET_CFG_CTRL_GATHER;
}
if ((nn->cap & NFP_NET_CFG_CTRL_LSO) && nn->fw_ver.major > 2) {
netdev->hw_features |= NETIF_F_TSO | NETIF_F_TSO6;
- nn->ctrl |= NFP_NET_CFG_CTRL_LSO;
+ nn->dp.ctrl |= NFP_NET_CFG_CTRL_LSO;
}
if (nn->cap & NFP_NET_CFG_CTRL_RSS) {
netdev->hw_features |= NETIF_F_RXHASH;
nfp_net_rss_init(nn);
- nn->ctrl |= NFP_NET_CFG_CTRL_RSS;
+ nn->dp.ctrl |= NFP_NET_CFG_CTRL_RSS;
}
if (nn->cap & NFP_NET_CFG_CTRL_VXLAN &&
nn->cap & NFP_NET_CFG_CTRL_NVGRE) {
if (nn->cap & NFP_NET_CFG_CTRL_LSO)
netdev->hw_features |= NETIF_F_GSO_GRE |
NETIF_F_GSO_UDP_TUNNEL;
- nn->ctrl |= NFP_NET_CFG_CTRL_VXLAN | NFP_NET_CFG_CTRL_NVGRE;
+ nn->dp.ctrl |= NFP_NET_CFG_CTRL_VXLAN | NFP_NET_CFG_CTRL_NVGRE;
netdev->hw_enc_features = netdev->hw_features;
}
@@ -3212,11 +3272,11 @@
if (nn->cap & NFP_NET_CFG_CTRL_RXVLAN) {
netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_RX;
- nn->ctrl |= NFP_NET_CFG_CTRL_RXVLAN;
+ nn->dp.ctrl |= NFP_NET_CFG_CTRL_RXVLAN;
}
if (nn->cap & NFP_NET_CFG_CTRL_TXVLAN) {
netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_TX;
- nn->ctrl |= NFP_NET_CFG_CTRL_TXVLAN;
+ nn->dp.ctrl |= NFP_NET_CFG_CTRL_TXVLAN;
}
netdev->features = netdev->hw_features;
@@ -3229,14 +3289,14 @@
/* Allow L2 Broadcast and Multicast through by default, if supported */
if (nn->cap & NFP_NET_CFG_CTRL_L2BC)
- nn->ctrl |= NFP_NET_CFG_CTRL_L2BC;
+ nn->dp.ctrl |= NFP_NET_CFG_CTRL_L2BC;
if (nn->cap & NFP_NET_CFG_CTRL_L2MC)
- nn->ctrl |= NFP_NET_CFG_CTRL_L2MC;
+ nn->dp.ctrl |= NFP_NET_CFG_CTRL_L2MC;
/* Allow IRQ moderation, if supported */
if (nn->cap & NFP_NET_CFG_CTRL_IRQMOD) {
nfp_net_irqmod_init(nn);
- nn->ctrl |= NFP_NET_CFG_CTRL_IRQMOD;
+ nn->dp.ctrl |= NFP_NET_CFG_CTRL_IRQMOD;
}
/* Stash the re-configuration queue away. First odd queue in TX Bar */
@@ -3275,10 +3335,10 @@
{
struct nfp_net *nn = netdev_priv(netdev);
- unregister_netdev(nn->netdev);
+ unregister_netdev(nn->dp.netdev);
- if (nn->xdp_prog)
- bpf_prog_put(nn->xdp_prog);
- if (nn->bpf_offload_xdp)
+ if (nn->dp.xdp_prog)
+ bpf_prog_put(nn->dp.xdp_prog);
+ if (nn->dp.bpf_offload_xdp)
nfp_net_xdp_offload(nn, NULL);
}
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h b/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h
index 385ba35..d04ccc9 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h
@@ -1,5 +1,5 @@
/*
- * Copyright (C) 2015 Netronome Systems, Inc.
+ * Copyright (C) 2015-2017 Netronome Systems, Inc.
*
* This software is dual licensed under the GNU General License Version 2,
* June 1991 as shown in the file COPYING in the top-level directory of this
@@ -177,6 +177,19 @@
#define NFP_NET_CFG_VERSION_MINOR(x) (((x) & 0xff) << 0)
#define NFP_NET_CFG_STS 0x0034
#define NFP_NET_CFG_STS_LINK (0x1 << 0) /* Link up or down */
+/* Link rate */
+#define NFP_NET_CFG_STS_LINK_RATE_SHIFT 1
+#define NFP_NET_CFG_STS_LINK_RATE_MASK 0xF
+#define NFP_NET_CFG_STS_LINK_RATE \
+ (NFP_NET_CFG_STS_LINK_RATE_MASK << NFP_NET_CFG_STS_LINK_RATE_SHIFT)
+#define NFP_NET_CFG_STS_LINK_RATE_UNSUPPORTED 0
+#define NFP_NET_CFG_STS_LINK_RATE_UNKNOWN 1
+#define NFP_NET_CFG_STS_LINK_RATE_1G 2
+#define NFP_NET_CFG_STS_LINK_RATE_10G 3
+#define NFP_NET_CFG_STS_LINK_RATE_25G 4
+#define NFP_NET_CFG_STS_LINK_RATE_40G 5
+#define NFP_NET_CFG_STS_LINK_RATE_50G 6
+#define NFP_NET_CFG_STS_LINK_RATE_100G 7
#define NFP_NET_CFG_CAP 0x0038
#define NFP_NET_CFG_MAX_TXRINGS 0x003c
#define NFP_NET_CFG_MAX_RXRINGS 0x0040
@@ -192,6 +205,14 @@
#define NFP_NET_CFG_RX_OFFSET_DYNAMIC 0 /* Prepend mode */
/**
+ * RSS capabilities
+ * @NFP_NET_CFG_RSS_CAP_HFUNC: supported hash functions (same bits as
+ * @NFP_NET_CFG_RSS_HFUNC)
+ */
+#define NFP_NET_CFG_RSS_CAP 0x0054
+#define NFP_NET_CFG_RSS_CAP_HFUNC 0xff000000
+
+/**
* VXLAN/UDP encap configuration
* @NFP_NET_CFG_VXLAN_PORT: Base address of table of tunnels' UDP dst ports
* @NFP_NET_CFG_VXLAN_SZ: Size of the UDP port table in bytes
@@ -249,7 +270,11 @@
#define NFP_NET_CFG_RSS_IPV4_UDP (1 << 11) /* RSS for IPv4/UDP */
#define NFP_NET_CFG_RSS_IPV6_TCP (1 << 12) /* RSS for IPv6/TCP */
#define NFP_NET_CFG_RSS_IPV6_UDP (1 << 13) /* RSS for IPv6/UDP */
+#define NFP_NET_CFG_RSS_HFUNC 0xff000000
#define NFP_NET_CFG_RSS_TOEPLITZ (1 << 24) /* Use Toeplitz hash */
+#define NFP_NET_CFG_RSS_XOR (1 << 25) /* Use XOR as hash */
+#define NFP_NET_CFG_RSS_CRC32 (1 << 26) /* Use CRC32 as hash */
+#define NFP_NET_CFG_RSS_HFUNCS 3
#define NFP_NET_CFG_RSS_KEY (NFP_NET_CFG_RSS_BASE + 0x4)
#define NFP_NET_CFG_RSS_KEY_SZ 0x28
#define NFP_NET_CFG_RSS_ITBL (NFP_NET_CFG_RSS_BASE + 0x4 + \
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_debugfs.c b/drivers/net/ethernet/netronome/nfp/nfp_net_debugfs.c
index 6e9372a..4077c59 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_debugfs.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_debugfs.c
@@ -40,9 +40,9 @@
static int nfp_net_debugfs_rx_q_read(struct seq_file *file, void *data)
{
- int fl_rd_p, fl_wr_p, rx_rd_p, rx_wr_p, rxd_cnt;
struct nfp_net_r_vector *r_vec = file->private;
struct nfp_net_rx_ring *rx_ring;
+ int fl_rd_p, fl_wr_p, rxd_cnt;
struct nfp_net_rx_desc *rxd;
struct nfp_net *nn;
void *frag;
@@ -54,19 +54,18 @@
goto out;
nn = r_vec->nfp_net;
rx_ring = r_vec->rx_ring;
- if (!netif_running(nn->netdev))
+ if (!netif_running(nn->dp.netdev))
goto out;
rxd_cnt = rx_ring->cnt;
fl_rd_p = nfp_qcp_rd_ptr_read(rx_ring->qcp_fl);
fl_wr_p = nfp_qcp_wr_ptr_read(rx_ring->qcp_fl);
- rx_rd_p = nfp_qcp_rd_ptr_read(rx_ring->qcp_rx);
- rx_wr_p = nfp_qcp_wr_ptr_read(rx_ring->qcp_rx);
- seq_printf(file, "RX[%02d]: H_RD=%d H_WR=%d FL_RD=%d FL_WR=%d RX_RD=%d RX_WR=%d\n",
- rx_ring->idx, rx_ring->rd_p, rx_ring->wr_p,
- fl_rd_p, fl_wr_p, rx_rd_p, rx_wr_p);
+ seq_printf(file, "RX[%02d,%02d]: cnt=%d dma=%pad host=%p H_RD=%d H_WR=%d FL_RD=%d FL_WR=%d\n",
+ rx_ring->idx, rx_ring->fl_qcidx,
+ rx_ring->cnt, &rx_ring->dma, rx_ring->rxds,
+ rx_ring->rd_p, rx_ring->wr_p, fl_rd_p, fl_wr_p);
for (i = 0; i < rxd_cnt; i++) {
rxd = &rx_ring->rxds[i];
@@ -89,10 +88,6 @@
seq_puts(file, " FL_RD");
if (i == fl_wr_p % rxd_cnt)
seq_puts(file, " FL_WR");
- if (i == rx_rd_p % rxd_cnt)
- seq_puts(file, " RX_RD");
- if (i == rx_wr_p % rxd_cnt)
- seq_puts(file, " RX_WR");
seq_putc(file, '\n');
}
@@ -143,7 +138,7 @@
if (!r_vec->nfp_net || !tx_ring)
goto out;
nn = r_vec->nfp_net;
- if (!netif_running(nn->netdev))
+ if (!netif_running(nn->dp.netdev))
goto out;
txd_cnt = tx_ring->cnt;
@@ -151,8 +146,11 @@
d_rd_p = nfp_qcp_rd_ptr_read(tx_ring->qcp_q);
d_wr_p = nfp_qcp_wr_ptr_read(tx_ring->qcp_q);
- seq_printf(file, "TX[%02d]: H_RD=%d H_WR=%d D_RD=%d D_WR=%d\n",
- tx_ring->idx, tx_ring->rd_p, tx_ring->wr_p, d_rd_p, d_wr_p);
+ seq_printf(file, "TX[%02d,%02d%s]: cnt=%d dma=%pad host=%p H_RD=%d H_WR=%d D_RD=%d D_WR=%d\n",
+ tx_ring->idx, tx_ring->qcidx,
+ tx_ring == r_vec->tx_ring ? "" : "xdp",
+ tx_ring->cnt, &tx_ring->dma, tx_ring->txds,
+ tx_ring->rd_p, tx_ring->wr_p, d_rd_p, d_wr_p);
for (i = 0; i < txd_cnt; i++) {
txd = &tx_ring->txds[i];
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_ethtool.c b/drivers/net/ethernet/netronome/nfp/nfp_net_ethtool.c
index 2649f75..abbb47e 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_ethtool.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_ethtool.c
@@ -40,6 +40,7 @@
* Brad Petrus <brad.petrus@netronome.com>
*/
+#include <linux/bitfield.h>
#include <linux/kernel.h>
#include <linux/netdevice.h>
#include <linux/etherdevice.h>
@@ -48,6 +49,7 @@
#include <linux/ethtool.h>
#include "nfpcore/nfp.h"
+#include "nfpcore/nfp_nsp.h"
#include "nfp_net_ctrl.h"
#include "nfp_net.h"
@@ -126,9 +128,9 @@
};
#define NN_ET_GLOBAL_STATS_LEN ARRAY_SIZE(nfp_net_et_stats)
-#define NN_ET_RVEC_STATS_LEN (nn->num_r_vecs * 3)
+#define NN_ET_RVEC_STATS_LEN (nn->dp.num_r_vecs * 3)
#define NN_ET_RVEC_GATHER_STATS 7
-#define NN_ET_QUEUE_STATS_LEN ((nn->num_tx_rings + nn->num_rx_rings) * 2)
+#define NN_ET_QUEUE_STATS_LEN ((nn->dp.num_tx_rings + nn->dp.num_rx_rings) * 2)
#define NN_ET_STATS_LEN (NN_ET_GLOBAL_STATS_LEN + NN_ET_RVEC_GATHER_STATS + \
NN_ET_RVEC_STATS_LEN + NN_ET_QUEUE_STATS_LEN)
@@ -172,6 +174,119 @@
drvinfo->regdump_len = NFP_NET_CFG_BAR_SZ;
}
+/**
+ * nfp_net_get_link_ksettings - Get Link Speed settings
+ * @netdev: network interface device structure
+ * @cmd: ethtool command
+ *
+ * Reports speed settings based on info in the BAR provided by the fw.
+ */
+static int
+nfp_net_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
+{
+ static const u32 ls_to_ethtool[] = {
+ [NFP_NET_CFG_STS_LINK_RATE_UNSUPPORTED] = 0,
+ [NFP_NET_CFG_STS_LINK_RATE_UNKNOWN] = SPEED_UNKNOWN,
+ [NFP_NET_CFG_STS_LINK_RATE_1G] = SPEED_1000,
+ [NFP_NET_CFG_STS_LINK_RATE_10G] = SPEED_10000,
+ [NFP_NET_CFG_STS_LINK_RATE_25G] = SPEED_25000,
+ [NFP_NET_CFG_STS_LINK_RATE_40G] = SPEED_40000,
+ [NFP_NET_CFG_STS_LINK_RATE_50G] = SPEED_50000,
+ [NFP_NET_CFG_STS_LINK_RATE_100G] = SPEED_100000,
+ };
+ struct nfp_net *nn = netdev_priv(netdev);
+ u32 sts, ls;
+
+ ethtool_link_ksettings_add_link_mode(cmd, supported, FIBRE);
+ cmd->base.port = PORT_OTHER;
+ cmd->base.speed = SPEED_UNKNOWN;
+ cmd->base.duplex = DUPLEX_UNKNOWN;
+
+ if (nn->eth_port)
+ cmd->base.autoneg = nn->eth_port->aneg != NFP_ANEG_DISABLED ?
+ AUTONEG_ENABLE : AUTONEG_DISABLE;
+
+ if (!netif_carrier_ok(netdev))
+ return 0;
+
+ /* Use link speed from ETH table if available, otherwise try the BAR */
+ if (nn->eth_port) {
+ int err;
+
+ if (nfp_net_link_changed_read_clear(nn)) {
+ err = nfp_net_refresh_eth_port(nn);
+ if (err)
+ return err;
+ }
+
+ cmd->base.port = nn->eth_port->port_type;
+ cmd->base.speed = nn->eth_port->speed;
+ cmd->base.duplex = DUPLEX_FULL;
+ return 0;
+ }
+
+ sts = nn_readl(nn, NFP_NET_CFG_STS);
+
+ ls = FIELD_GET(NFP_NET_CFG_STS_LINK_RATE, sts);
+ if (ls == NFP_NET_CFG_STS_LINK_RATE_UNSUPPORTED)
+ return -EOPNOTSUPP;
+
+ if (ls == NFP_NET_CFG_STS_LINK_RATE_UNKNOWN ||
+ ls >= ARRAY_SIZE(ls_to_ethtool))
+ return 0;
+
+ cmd->base.speed = ls_to_ethtool[sts];
+ cmd->base.duplex = DUPLEX_FULL;
+
+ return 0;
+}
+
+static int
+nfp_net_set_link_ksettings(struct net_device *netdev,
+ const struct ethtool_link_ksettings *cmd)
+{
+ struct nfp_net *nn = netdev_priv(netdev);
+ struct nfp_nsp *nsp;
+ int err;
+
+ if (!nn->eth_port)
+ return -EOPNOTSUPP;
+
+ if (netif_running(netdev)) {
+ nn_warn(nn, "Changing settings not allowed on an active interface. It may cause the port to be disabled until reboot.\n");
+ return -EBUSY;
+ }
+
+ nsp = nfp_eth_config_start(nn->cpp, nn->eth_port->index);
+ if (IS_ERR(nsp))
+ return PTR_ERR(nsp);
+
+ err = __nfp_eth_set_aneg(nsp, cmd->base.autoneg == AUTONEG_ENABLE ?
+ NFP_ANEG_AUTO : NFP_ANEG_DISABLED);
+ if (err)
+ goto err_bad_set;
+ if (cmd->base.speed != SPEED_UNKNOWN) {
+ u32 speed = cmd->base.speed / nn->eth_port->lanes;
+
+ err = __nfp_eth_set_speed(nsp, speed);
+ if (err)
+ goto err_bad_set;
+ }
+
+ err = nfp_eth_config_commit_end(nsp);
+ if (err > 0)
+ return 0; /* no change */
+
+ nfp_net_refresh_port_table(nn);
+
+ return err;
+
+err_bad_set:
+ nfp_eth_config_cleanup_end(nsp);
+ return err;
+}
+
static void nfp_net_get_ringparam(struct net_device *netdev,
struct ethtool_ringparam *ring)
{
@@ -179,30 +294,22 @@
ring->rx_max_pending = NFP_NET_MAX_RX_DESCS;
ring->tx_max_pending = NFP_NET_MAX_TX_DESCS;
- ring->rx_pending = nn->rxd_cnt;
- ring->tx_pending = nn->txd_cnt;
+ ring->rx_pending = nn->dp.rxd_cnt;
+ ring->tx_pending = nn->dp.txd_cnt;
}
static int nfp_net_set_ring_size(struct nfp_net *nn, u32 rxd_cnt, u32 txd_cnt)
{
- struct nfp_net_ring_set *reconfig_rx = NULL, *reconfig_tx = NULL;
- struct nfp_net_ring_set rx = {
- .n_rings = nn->num_rx_rings,
- .mtu = nn->netdev->mtu,
- .dcnt = rxd_cnt,
- };
- struct nfp_net_ring_set tx = {
- .n_rings = nn->num_tx_rings,
- .dcnt = txd_cnt,
- };
+ struct nfp_net_dp *dp;
- if (nn->rxd_cnt != rxd_cnt)
- reconfig_rx = ℞
- if (nn->txd_cnt != txd_cnt)
- reconfig_tx = &tx;
+ dp = nfp_net_clone_dp(nn);
+ if (!dp)
+ return -ENOMEM;
- return nfp_net_ring_reconfig(nn, &nn->xdp_prog,
- reconfig_rx, reconfig_tx);
+ dp->rxd_cnt = rxd_cnt;
+ dp->txd_cnt = txd_cnt;
+
+ return nfp_net_ring_reconfig(nn, dp, NULL);
}
static int nfp_net_set_ringparam(struct net_device *netdev,
@@ -223,11 +330,11 @@
txd_cnt < NFP_NET_MIN_TX_DESCS || txd_cnt > NFP_NET_MAX_TX_DESCS)
return -EINVAL;
- if (nn->rxd_cnt == rxd_cnt && nn->txd_cnt == txd_cnt)
+ if (nn->dp.rxd_cnt == rxd_cnt && nn->dp.txd_cnt == txd_cnt)
return 0;
nn_dbg(nn, "Change ring size: RxQ %u->%u, TxQ %u->%u\n",
- nn->rxd_cnt, rxd_cnt, nn->txd_cnt, txd_cnt);
+ nn->dp.rxd_cnt, rxd_cnt, nn->dp.txd_cnt, txd_cnt);
return nfp_net_set_ring_size(nn, rxd_cnt, txd_cnt);
}
@@ -245,7 +352,7 @@
memcpy(p, nfp_net_et_stats[i].name, ETH_GSTRING_LEN);
p += ETH_GSTRING_LEN;
}
- for (i = 0; i < nn->num_r_vecs; i++) {
+ for (i = 0; i < nn->dp.num_r_vecs; i++) {
sprintf(p, "rvec_%u_rx_pkts", i);
p += ETH_GSTRING_LEN;
sprintf(p, "rvec_%u_tx_pkts", i);
@@ -267,13 +374,13 @@
p += ETH_GSTRING_LEN;
strncpy(p, "tx_lso", ETH_GSTRING_LEN);
p += ETH_GSTRING_LEN;
- for (i = 0; i < nn->num_tx_rings; i++) {
+ for (i = 0; i < nn->dp.num_tx_rings; i++) {
sprintf(p, "txq_%u_pkts", i);
p += ETH_GSTRING_LEN;
sprintf(p, "txq_%u_bytes", i);
p += ETH_GSTRING_LEN;
}
- for (i = 0; i < nn->num_rx_rings; i++) {
+ for (i = 0; i < nn->dp.num_rx_rings; i++) {
sprintf(p, "rxq_%u_pkts", i);
p += ETH_GSTRING_LEN;
sprintf(p, "rxq_%u_bytes", i);
@@ -306,12 +413,12 @@
break;
case NFP_NET_DEV_ET_STATS:
- io_p = nn->ctrl_bar + nfp_net_et_stats[i].off;
+ io_p = nn->dp.ctrl_bar + nfp_net_et_stats[i].off;
data[i] = readq(io_p);
break;
}
}
- for (j = 0; j < nn->num_r_vecs; j++) {
+ for (j = 0; j < nn->dp.num_r_vecs; j++) {
unsigned int start;
do {
@@ -337,16 +444,16 @@
}
for (j = 0; j < NN_ET_RVEC_GATHER_STATS; j++)
data[i++] = gathered_stats[j];
- for (j = 0; j < nn->num_tx_rings; j++) {
- io_p = nn->ctrl_bar + NFP_NET_CFG_TXR_STATS(j);
+ for (j = 0; j < nn->dp.num_tx_rings; j++) {
+ io_p = nn->dp.ctrl_bar + NFP_NET_CFG_TXR_STATS(j);
data[i++] = readq(io_p);
- io_p = nn->ctrl_bar + NFP_NET_CFG_TXR_STATS(j) + 8;
+ io_p = nn->dp.ctrl_bar + NFP_NET_CFG_TXR_STATS(j) + 8;
data[i++] = readq(io_p);
}
- for (j = 0; j < nn->num_rx_rings; j++) {
- io_p = nn->ctrl_bar + NFP_NET_CFG_RXR_STATS(j);
+ for (j = 0; j < nn->dp.num_rx_rings; j++) {
+ io_p = nn->dp.ctrl_bar + NFP_NET_CFG_RXR_STATS(j);
data[i++] = readq(io_p);
- io_p = nn->ctrl_bar + NFP_NET_CFG_RXR_STATS(j) + 8;
+ io_p = nn->dp.ctrl_bar + NFP_NET_CFG_RXR_STATS(j) + 8;
data[i++] = readq(io_p);
}
}
@@ -410,7 +517,7 @@
switch (cmd->cmd) {
case ETHTOOL_GRXRINGS:
- cmd->data = nn->num_rx_rings;
+ cmd->data = nn->dp.num_rx_rings;
return 0;
case ETHTOOL_GRXFH:
return nfp_net_get_rss_hash_opts(nn, cmd);
@@ -454,13 +561,13 @@
return -EINVAL;
}
- new_rss_cfg |= NFP_NET_CFG_RSS_TOEPLITZ;
+ new_rss_cfg |= FIELD_PREP(NFP_NET_CFG_RSS_HFUNC, nn->rss_hfunc);
new_rss_cfg |= NFP_NET_CFG_RSS_MASK;
if (new_rss_cfg == nn->rss_cfg)
return 0;
- writel(new_rss_cfg, nn->ctrl_bar + NFP_NET_CFG_RSS_CTRL);
+ writel(new_rss_cfg, nn->dp.ctrl_bar + NFP_NET_CFG_RSS_CTRL);
err = nfp_net_reconfig(nn, NFP_NET_CFG_UPDATE_RSS);
if (err)
return err;
@@ -496,7 +603,12 @@
static u32 nfp_net_get_rxfh_key_size(struct net_device *netdev)
{
- return NFP_NET_CFG_RSS_KEY_SZ;
+ struct nfp_net *nn = netdev_priv(netdev);
+
+ if (!(nn->cap & NFP_NET_CFG_CTRL_RSS))
+ return -EOPNOTSUPP;
+
+ return nfp_net_rss_key_sz(nn);
}
static int nfp_net_get_rxfh(struct net_device *netdev, u32 *indir, u8 *key,
@@ -512,9 +624,12 @@
for (i = 0; i < ARRAY_SIZE(nn->rss_itbl); i++)
indir[i] = nn->rss_itbl[i];
if (key)
- memcpy(key, nn->rss_key, NFP_NET_CFG_RSS_KEY_SZ);
- if (hfunc)
- *hfunc = ETH_RSS_HASH_TOP;
+ memcpy(key, nn->rss_key, nfp_net_rss_key_sz(nn));
+ if (hfunc) {
+ *hfunc = nn->rss_hfunc;
+ if (*hfunc >= 1 << ETH_RSS_HASH_FUNCS_COUNT)
+ *hfunc = ETH_RSS_HASH_UNKNOWN;
+ }
return 0;
}
@@ -527,14 +642,14 @@
int i;
if (!(nn->cap & NFP_NET_CFG_CTRL_RSS) ||
- !(hfunc == ETH_RSS_HASH_NO_CHANGE || hfunc == ETH_RSS_HASH_TOP))
+ !(hfunc == ETH_RSS_HASH_NO_CHANGE || hfunc == nn->rss_hfunc))
return -EOPNOTSUPP;
if (!key && !indir)
return 0;
if (key) {
- memcpy(nn->rss_key, key, NFP_NET_CFG_RSS_KEY_SZ);
+ memcpy(nn->rss_key, key, nfp_net_rss_key_sz(nn));
nfp_net_rss_write_key(nn);
}
if (indir) {
@@ -564,7 +679,7 @@
regs->version = nn_readl(nn, NFP_NET_CFG_VERSION);
for (i = 0; i < NFP_NET_CFG_BAR_SZ / sizeof(u32); i++)
- regs_buf[i] = readl(nn->ctrl_bar + (i * sizeof(u32)));
+ regs_buf[i] = readl(nn->dp.ctrl_bar + (i * sizeof(u32)));
}
static int nfp_net_get_coalesce(struct net_device *netdev,
@@ -676,7 +791,7 @@
ec->tx_coalesce_usecs_high ||
ec->tx_max_coalesced_frames_high ||
ec->rate_sample_interval)
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
/* Compute factor used to convert coalesce '_usecs' parameters to
* ME timestamp ticks. There are 16 ME clock cycles for each timestamp
@@ -736,16 +851,16 @@
struct nfp_net *nn = netdev_priv(netdev);
unsigned int num_tx_rings;
- num_tx_rings = nn->num_tx_rings;
- if (nn->xdp_prog)
- num_tx_rings -= nn->num_rx_rings;
+ num_tx_rings = nn->dp.num_tx_rings;
+ if (nn->dp.xdp_prog)
+ num_tx_rings -= nn->dp.num_rx_rings;
channel->max_rx = min(nn->max_rx_rings, nn->max_r_vecs);
channel->max_tx = min(nn->max_tx_rings, nn->max_r_vecs);
channel->max_combined = min(channel->max_rx, channel->max_tx);
channel->max_other = NFP_NET_NON_Q_VECTORS;
- channel->combined_count = min(nn->num_rx_rings, num_tx_rings);
- channel->rx_count = nn->num_rx_rings - channel->combined_count;
+ channel->combined_count = min(nn->dp.num_rx_rings, num_tx_rings);
+ channel->rx_count = nn->dp.num_rx_rings - channel->combined_count;
channel->tx_count = num_tx_rings - channel->combined_count;
channel->other_count = NFP_NET_NON_Q_VECTORS;
}
@@ -753,29 +868,19 @@
static int nfp_net_set_num_rings(struct nfp_net *nn, unsigned int total_rx,
unsigned int total_tx)
{
- struct nfp_net_ring_set *reconfig_rx = NULL, *reconfig_tx = NULL;
- struct nfp_net_ring_set rx = {
- .n_rings = total_rx,
- .mtu = nn->netdev->mtu,
- .dcnt = nn->rxd_cnt,
- };
- struct nfp_net_ring_set tx = {
- .n_rings = total_tx,
- .dcnt = nn->txd_cnt,
- };
+ struct nfp_net_dp *dp;
- if (nn->num_rx_rings != total_rx)
- reconfig_rx = ℞
- if (nn->num_stack_tx_rings != total_tx ||
- (nn->xdp_prog && reconfig_rx))
- reconfig_tx = &tx;
+ dp = nfp_net_clone_dp(nn);
+ if (!dp)
+ return -ENOMEM;
- /* nfp_net_check_config() will catch tx.n_rings > nn->max_tx_rings */
- if (nn->xdp_prog)
- tx.n_rings += total_rx;
+ dp->num_rx_rings = total_rx;
+ dp->num_tx_rings = total_tx;
+ /* nfp_net_check_config() will catch num_tx_rings > nn->max_tx_rings */
+ if (dp->xdp_prog)
+ dp->num_tx_rings += total_rx;
- return nfp_net_ring_reconfig(nn, &nn->xdp_prog,
- reconfig_rx, reconfig_tx);
+ return nfp_net_ring_reconfig(nn, dp, NULL);
}
static int nfp_net_set_channels(struct net_device *netdev,
@@ -823,6 +928,8 @@
.set_coalesce = nfp_net_set_coalesce,
.get_channels = nfp_net_get_channels,
.set_channels = nfp_net_set_channels,
+ .get_link_ksettings = nfp_net_get_link_ksettings,
+ .set_link_ksettings = nfp_net_set_link_ksettings,
};
void nfp_net_set_ethtool_ops(struct net_device *netdev)
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
index 3afcdc1..8cb87cb 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_main.c
@@ -47,11 +47,12 @@
#include <linux/pci_regs.h>
#include <linux/msi.h>
#include <linux/random.h>
+#include <linux/rtnetlink.h>
#include "nfpcore/nfp.h"
#include "nfpcore/nfp_cpp.h"
#include "nfpcore/nfp_nffw.h"
-#include "nfpcore/nfp_nsp_eth.h"
+#include "nfpcore/nfp_nsp.h"
#include "nfpcore/nfp6000_pcie.h"
#include "nfp_net_ctrl.h"
@@ -129,61 +130,61 @@
return (u8 __iomem *)ERR_PTR(err);
}
-static void
-nfp_net_get_mac_addr_hwinfo(struct nfp_net *nn, struct nfp_cpp *cpp,
- unsigned int id)
-{
- u8 mac_addr[ETH_ALEN];
- const char *mac_str;
- char name[32];
-
- snprintf(name, sizeof(name), "eth%d.mac", id);
-
- mac_str = nfp_hwinfo_lookup(cpp, name);
- if (!mac_str) {
- dev_warn(&nn->pdev->dev,
- "Can't lookup MAC address. Generate\n");
- eth_hw_addr_random(nn->netdev);
- return;
- }
-
- if (sscanf(mac_str, "%02hhx:%02hhx:%02hhx:%02hhx:%02hhx:%02hhx",
- &mac_addr[0], &mac_addr[1], &mac_addr[2],
- &mac_addr[3], &mac_addr[4], &mac_addr[5]) != 6) {
- dev_warn(&nn->pdev->dev,
- "Can't parse MAC address (%s). Generate.\n", mac_str);
- eth_hw_addr_random(nn->netdev);
- return;
- }
-
- ether_addr_copy(nn->netdev->dev_addr, mac_addr);
- ether_addr_copy(nn->netdev->perm_addr, mac_addr);
-}
-
/**
* nfp_net_get_mac_addr() - Get the MAC address.
* @nn: NFP Network structure
- * @pf: NFP PF device structure
+ * @cpp: NFP CPP handle
* @id: NFP port id
*
* First try to get the MAC address from NSP ETH table. If that
* fails try HWInfo. As a last resort generate a random address.
*/
static void
-nfp_net_get_mac_addr(struct nfp_net *nn, struct nfp_pf *pf, unsigned int id)
+nfp_net_get_mac_addr(struct nfp_net *nn, struct nfp_cpp *cpp, unsigned int id)
+{
+ struct nfp_net_dp *dp = &nn->dp;
+ u8 mac_addr[ETH_ALEN];
+ const char *mac_str;
+ char name[32];
+
+ if (nn->eth_port) {
+ ether_addr_copy(dp->netdev->dev_addr, nn->eth_port->mac_addr);
+ ether_addr_copy(dp->netdev->perm_addr, nn->eth_port->mac_addr);
+ return;
+ }
+
+ snprintf(name, sizeof(name), "eth%d.mac", id);
+
+ mac_str = nfp_hwinfo_lookup(cpp, name);
+ if (!mac_str) {
+ dev_warn(dp->dev, "Can't lookup MAC address. Generate\n");
+ eth_hw_addr_random(dp->netdev);
+ return;
+ }
+
+ if (sscanf(mac_str, "%02hhx:%02hhx:%02hhx:%02hhx:%02hhx:%02hhx",
+ &mac_addr[0], &mac_addr[1], &mac_addr[2],
+ &mac_addr[3], &mac_addr[4], &mac_addr[5]) != 6) {
+ dev_warn(dp->dev,
+ "Can't parse MAC address (%s). Generate.\n", mac_str);
+ eth_hw_addr_random(dp->netdev);
+ return;
+ }
+
+ ether_addr_copy(dp->netdev->dev_addr, mac_addr);
+ ether_addr_copy(dp->netdev->perm_addr, mac_addr);
+}
+
+static struct nfp_eth_table_port *
+nfp_net_find_port(struct nfp_eth_table *eth_tbl, unsigned int id)
{
int i;
- for (i = 0; pf->eth_tbl && i < pf->eth_tbl->count; i++)
- if (pf->eth_tbl->ports[i].eth_index == id) {
- const u8 *mac_addr = pf->eth_tbl->ports[i].mac_addr;
+ for (i = 0; eth_tbl && i < eth_tbl->count; i++)
+ if (eth_tbl->ports[i].eth_index == id)
+ return ð_tbl->ports[i];
- ether_addr_copy(nn->netdev->dev_addr, mac_addr);
- ether_addr_copy(nn->netdev->perm_addr, mac_addr);
- return;
- }
-
- nfp_net_get_mac_addr_hwinfo(nn, pf->cpp, id);
+ return NULL;
}
static unsigned int nfp_net_pf_get_num_ports(struct nfp_pf *pf)
@@ -282,6 +283,7 @@
while (!list_empty(&pf->ports)) {
nn = list_first_entry(&pf->ports, struct nfp_net, port_list);
list_del(&nn->port_list);
+ pf->num_netdevs--;
nfp_net_netdev_free(nn);
}
@@ -290,7 +292,8 @@
static struct nfp_net *
nfp_net_pf_alloc_port_netdev(struct nfp_pf *pf, void __iomem *ctrl_bar,
void __iomem *tx_bar, void __iomem *rx_bar,
- int stride, struct nfp_net_fw_version *fw_ver)
+ int stride, struct nfp_net_fw_version *fw_ver,
+ struct nfp_eth_table_port *eth_port)
{
u32 n_tx_rings, n_rx_rings;
struct nfp_net *nn;
@@ -305,12 +308,13 @@
nn->cpp = pf->cpp;
nn->fw_ver = *fw_ver;
- nn->ctrl_bar = ctrl_bar;
+ nn->dp.ctrl_bar = ctrl_bar;
nn->tx_bar = tx_bar;
nn->rx_bar = rx_bar;
- nn->is_vf = 0;
+ nn->dp.is_vf = 0;
nn->stride_rx = stride;
nn->stride_tx = stride;
+ nn->eth_port = eth_port;
return nn;
}
@@ -322,7 +326,7 @@
int err;
/* Get MAC address */
- nfp_net_get_mac_addr(nn, pf, id);
+ nfp_net_get_mac_addr(nn, pf->cpp, id);
/* Get ME clock frequency from ctrl BAR
* XXX for now frequency is hardcoded until we figure out how
@@ -330,7 +334,7 @@
*/
nn->me_freq_mhz = 1200;
- err = nfp_net_netdev_init(nn->netdev);
+ err = nfp_net_netdev_init(nn->dp.netdev);
if (err)
return err;
@@ -347,6 +351,7 @@
int stride, struct nfp_net_fw_version *fw_ver)
{
u32 prev_tx_base, prev_rx_base, tgt_tx_base, tgt_rx_base;
+ struct nfp_eth_table_port *eth_port;
struct nfp_net *nn;
unsigned int i;
int err;
@@ -362,17 +367,27 @@
prev_tx_base = tgt_tx_base;
prev_rx_base = tgt_rx_base;
- nn = nfp_net_pf_alloc_port_netdev(pf, ctrl_bar, tx_bar, rx_bar,
- stride, fw_ver);
- if (IS_ERR(nn)) {
- err = PTR_ERR(nn);
- goto err_free_prev;
+ eth_port = nfp_net_find_port(pf->eth_tbl, i);
+ if (eth_port && eth_port->override_changed) {
+ nfp_warn(pf->cpp, "Config changed for port #%d, reboot required before port will be operational\n", i);
+ } else {
+ nn = nfp_net_pf_alloc_port_netdev(pf, ctrl_bar, tx_bar,
+ rx_bar, stride,
+ fw_ver, eth_port);
+ if (IS_ERR(nn)) {
+ err = PTR_ERR(nn);
+ goto err_free_prev;
+ }
+ list_add_tail(&nn->port_list, &pf->ports);
+ pf->num_netdevs++;
}
- list_add_tail(&nn->port_list, &pf->ports);
ctrl_bar += NFP_PF_CSR_SLICE_SIZE;
}
+ if (list_empty(&pf->ports))
+ return -ENODEV;
+
return 0;
err_free_prev:
@@ -399,7 +414,7 @@
/* Get MSI-X vectors */
wanted_irqs = 0;
list_for_each_entry(nn, &pf->ports, port_list)
- wanted_irqs += NFP_NET_NON_Q_VECTORS + nn->num_r_vecs;
+ wanted_irqs += NFP_NET_NON_Q_VECTORS + nn->dp.num_r_vecs;
pf->irq_entries = kcalloc(wanted_irqs, sizeof(*pf->irq_entries),
GFP_KERNEL);
if (!pf->irq_entries) {
@@ -408,7 +423,7 @@
}
num_irqs = nfp_net_irqs_alloc(pf->pdev, pf->irq_entries,
- NFP_NET_MIN_PORT_IRQS * pf->num_ports,
+ NFP_NET_MIN_PORT_IRQS * pf->num_netdevs,
wanted_irqs);
if (!num_irqs) {
nn_warn(nn, "Unable to allocate MSI-X Vectors. Exiting\n");
@@ -418,7 +433,7 @@
/* Distribute IRQs to ports */
irqs_left = num_irqs;
- ports_left = pf->num_ports;
+ ports_left = pf->num_netdevs;
list_for_each_entry(nn, &pf->ports, port_list) {
unsigned int n;
@@ -444,7 +459,7 @@
err_prev_deinit:
list_for_each_entry_continue_reverse(nn, &pf->ports, port_list) {
nfp_net_debugfs_dir_clean(&nn->debugfs_dir);
- nfp_net_netdev_clean(nn->netdev);
+ nfp_net_netdev_clean(nn->dp.netdev);
}
nfp_net_irqs_disable(pf->pdev);
err_vec_free:
@@ -454,6 +469,108 @@
return err;
}
+static void nfp_net_pci_remove_finish(struct nfp_pf *pf)
+{
+ nfp_net_debugfs_dir_clean(&pf->ddir);
+
+ nfp_net_irqs_disable(pf->pdev);
+ kfree(pf->irq_entries);
+
+ nfp_cpp_area_release_free(pf->rx_area);
+ nfp_cpp_area_release_free(pf->tx_area);
+ nfp_cpp_area_release_free(pf->ctrl_area);
+}
+
+static void nfp_net_refresh_netdevs(struct work_struct *work)
+{
+ struct nfp_pf *pf = container_of(work, struct nfp_pf,
+ port_refresh_work);
+ struct nfp_eth_table *eth_table;
+ struct nfp_net *nn, *next;
+
+ mutex_lock(&pf->port_lock);
+
+ /* Check for nfp_net_pci_remove() racing against us */
+ if (list_empty(&pf->ports))
+ goto out;
+
+ list_for_each_entry(nn, &pf->ports, port_list)
+ nfp_net_link_changed_read_clear(nn);
+
+ eth_table = nfp_eth_read_ports(pf->cpp);
+ if (!eth_table) {
+ nfp_err(pf->cpp, "Error refreshing port config!\n");
+ goto out;
+ }
+
+ rtnl_lock();
+ list_for_each_entry(nn, &pf->ports, port_list) {
+ if (!nn->eth_port)
+ continue;
+ nn->eth_port = nfp_net_find_port(eth_table,
+ nn->eth_port->eth_index);
+ }
+ rtnl_unlock();
+
+ kfree(pf->eth_tbl);
+ pf->eth_tbl = eth_table;
+
+ list_for_each_entry_safe(nn, next, &pf->ports, port_list) {
+ if (!nn->eth_port) {
+ nfp_warn(pf->cpp, "Warning: port not present after reconfig\n");
+ continue;
+ }
+ if (!nn->eth_port->override_changed)
+ continue;
+
+ nn_warn(nn, "Port config changed, unregistering. Reboot required before port will be operational again.\n");
+
+ nfp_net_debugfs_dir_clean(&nn->debugfs_dir);
+ nfp_net_netdev_clean(nn->dp.netdev);
+
+ list_del(&nn->port_list);
+ pf->num_netdevs--;
+ nfp_net_netdev_free(nn);
+ }
+
+ if (list_empty(&pf->ports))
+ nfp_net_pci_remove_finish(pf);
+out:
+ mutex_unlock(&pf->port_lock);
+}
+
+void nfp_net_refresh_port_table(struct nfp_net *nn)
+{
+ struct nfp_pf *pf = pci_get_drvdata(nn->pdev);
+
+ schedule_work(&pf->port_refresh_work);
+}
+
+int nfp_net_refresh_eth_port(struct nfp_net *nn)
+{
+ struct nfp_eth_table_port *eth_port;
+ struct nfp_eth_table *eth_table;
+
+ eth_table = nfp_eth_read_ports(nn->cpp);
+ if (!eth_table) {
+ nn_err(nn, "Error refreshing port state table!\n");
+ return -EIO;
+ }
+
+ eth_port = nfp_net_find_port(eth_table, nn->eth_port->eth_index);
+ if (!eth_port) {
+ nn_err(nn, "Error finding state of the port!\n");
+ kfree(eth_table);
+ return -EIO;
+ }
+
+ memcpy(nn->eth_port, eth_port, sizeof(*eth_port));
+
+ kfree(eth_table);
+
+ return 0;
+}
+
/*
* PCI device functions
*/
@@ -467,17 +584,23 @@
int stride;
int err;
+ INIT_WORK(&pf->port_refresh_work, nfp_net_refresh_netdevs);
+ mutex_init(&pf->port_lock);
+
/* Verify that the board has completed initialization */
if (!nfp_is_ready(pf->cpp)) {
nfp_err(pf->cpp, "NFP is not ready for NIC operation.\n");
return -EINVAL;
}
+ mutex_lock(&pf->port_lock);
pf->num_ports = nfp_net_pf_get_num_ports(pf);
ctrl_bar = nfp_net_pf_map_ctrl_bar(pf);
- if (!ctrl_bar)
- return pf->fw_loaded ? -EINVAL : -EPROBE_DEFER;
+ if (!ctrl_bar) {
+ err = pf->fw_loaded ? -EINVAL : -EPROBE_DEFER;
+ goto err_unlock;
+ }
nfp_net_get_fw_version(&fw_ver, ctrl_bar);
if (fw_ver.resv || fw_ver.class != NFP_NET_CFG_VERSION_CLASS_GENERIC) {
@@ -551,6 +674,8 @@
if (err)
goto err_clean_ddir;
+ mutex_unlock(&pf->port_lock);
+
return 0;
err_clean_ddir:
@@ -560,6 +685,8 @@
nfp_cpp_area_release_free(pf->tx_area);
err_ctrl_unmap:
nfp_cpp_area_release_free(pf->ctrl_area);
+err_unlock:
+ mutex_unlock(&pf->port_lock);
return err;
}
@@ -567,20 +694,21 @@
{
struct nfp_net *nn;
+ mutex_lock(&pf->port_lock);
+ if (list_empty(&pf->ports))
+ goto out;
+
list_for_each_entry(nn, &pf->ports, port_list) {
nfp_net_debugfs_dir_clean(&nn->debugfs_dir);
- nfp_net_netdev_clean(nn->netdev);
+ nfp_net_netdev_clean(nn->dp.netdev);
}
nfp_net_pf_free_netdevs(pf);
- nfp_net_debugfs_dir_clean(&pf->ddir);
+ nfp_net_pci_remove_finish(pf);
+out:
+ mutex_unlock(&pf->port_lock);
- nfp_net_irqs_disable(pf->pdev);
- kfree(pf->irq_entries);
-
- nfp_cpp_area_release_free(pf->rx_area);
- nfp_cpp_area_release_free(pf->tx_area);
- nfp_cpp_area_release_free(pf->ctrl_area);
+ cancel_work_sync(&pf->port_refresh_work);
}
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_offload.c b/drivers/net/ethernet/netronome/nfp/nfp_net_offload.c
index 18a851e..cc823df 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_offload.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_offload.c
@@ -58,7 +58,7 @@
spin_lock_bh(&nn->rx_filter_lock);
- if (nn->ctrl & NFP_NET_CFG_CTRL_BPF)
+ if (nn->dp.ctrl & NFP_NET_CFG_CTRL_BPF)
mod_timer(&nn->rx_filter_stats_timer,
jiffies + NFP_NET_STAT_POLL_IVL);
@@ -119,12 +119,12 @@
if (tc_no_actions(cls_bpf->exts))
return NN_ACT_DIRECT;
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
}
/* TC legacy mode */
if (!tc_single_action(cls_bpf->exts))
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
tcf_exts_to_list(cls_bpf->exts, &actions);
list_for_each_entry(a, &actions, list) {
@@ -132,11 +132,11 @@
return NN_ACT_TC_DROP;
if (is_tcf_mirred_egress_redirect(a) &&
- tcf_mirred_ifindex(a) == nn->netdev->ifindex)
+ tcf_mirred_ifindex(a) == nn->dp.netdev->ifindex)
return NN_ACT_TC_REDIR;
}
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
}
static int
@@ -152,7 +152,7 @@
int ret;
if (!IS_ENABLED(CONFIG_BPF_SYSCALL))
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
ret = nfp_net_bpf_get_act(nn, cls_bpf);
if (ret < 0)
@@ -160,16 +160,15 @@
act = ret;
max_mtu = nn_readb(nn, NFP_NET_CFG_BPF_INL_MTU) * 64 - 32;
- if (max_mtu < nn->netdev->mtu) {
+ if (max_mtu < nn->dp.netdev->mtu) {
nn_info(nn, "BPF offload not supported with MTU larger than HW packet split boundary\n");
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
}
start_off = nn_readw(nn, NFP_NET_CFG_BPF_START);
done_off = nn_readw(nn, NFP_NET_CFG_BPF_DONE);
- *code = dma_zalloc_coherent(&nn->pdev->dev, code_sz, dma_addr,
- GFP_KERNEL);
+ *code = dma_zalloc_coherent(nn->dp.dev, code_sz, dma_addr, GFP_KERNEL);
if (!*code)
return -ENOMEM;
@@ -181,7 +180,7 @@
return 0;
out:
- dma_free_coherent(&nn->pdev->dev, code_sz, *code, *dma_addr);
+ dma_free_coherent(nn->dp.dev, code_sz, *code, *dma_addr);
return ret;
}
@@ -194,7 +193,7 @@
u64 bpf_addr = dma_addr;
int err;
- nn->bpf_offload_skip_sw = !!(tc_flags & TCA_CLS_FLAGS_SKIP_SW);
+ nn->dp.bpf_offload_skip_sw = !!(tc_flags & TCA_CLS_FLAGS_SKIP_SW);
if (dense_mode)
bpf_addr |= NFP_NET_CFG_BPF_CFG_8CTX;
@@ -208,13 +207,13 @@
nn_err(nn, "FW command error while loading BPF: %d\n", err);
/* Enable passing packets through BPF function */
- nn->ctrl |= NFP_NET_CFG_CTRL_BPF;
- nn_writel(nn, NFP_NET_CFG_CTRL, nn->ctrl);
+ nn->dp.ctrl |= NFP_NET_CFG_CTRL_BPF;
+ nn_writel(nn, NFP_NET_CFG_CTRL, nn->dp.ctrl);
err = nfp_net_reconfig(nn, NFP_NET_CFG_UPDATE_GEN);
if (err)
nn_err(nn, "FW command error while enabling BPF: %d\n", err);
- dma_free_coherent(&nn->pdev->dev, code_sz, code, dma_addr);
+ dma_free_coherent(nn->dp.dev, code_sz, code, dma_addr);
nfp_net_bpf_stats_reset(nn);
mod_timer(&nn->rx_filter_stats_timer, jiffies + NFP_NET_STAT_POLL_IVL);
@@ -222,16 +221,16 @@
static int nfp_net_bpf_stop(struct nfp_net *nn)
{
- if (!(nn->ctrl & NFP_NET_CFG_CTRL_BPF))
+ if (!(nn->dp.ctrl & NFP_NET_CFG_CTRL_BPF))
return 0;
spin_lock_bh(&nn->rx_filter_lock);
- nn->ctrl &= ~NFP_NET_CFG_CTRL_BPF;
+ nn->dp.ctrl &= ~NFP_NET_CFG_CTRL_BPF;
spin_unlock_bh(&nn->rx_filter_lock);
- nn_writel(nn, NFP_NET_CFG_CTRL, nn->ctrl);
+ nn_writel(nn, NFP_NET_CFG_CTRL, nn->dp.ctrl);
del_timer_sync(&nn->rx_filter_stats_timer);
- nn->bpf_offload_skip_sw = 0;
+ nn->dp.bpf_offload_skip_sw = 0;
return nfp_net_reconfig(nn, NFP_NET_CFG_UPDATE_GEN);
}
@@ -255,7 +254,7 @@
* frames which didn't have BPF applied in the hardware should
* be fine if software fallback is available, though.
*/
- if (nn->bpf_offload_skip_sw)
+ if (nn->dp.bpf_offload_skip_sw)
return -EBUSY;
err = nfp_net_bpf_offload_prepare(nn, cls_bpf, &res, &code,
@@ -270,7 +269,7 @@
return 0;
case TC_CLSBPF_ADD:
- if (nn->ctrl & NFP_NET_CFG_CTRL_BPF)
+ if (nn->dp.ctrl & NFP_NET_CFG_CTRL_BPF)
return -EBUSY;
err = nfp_net_bpf_offload_prepare(nn, cls_bpf, &res, &code,
@@ -290,6 +289,6 @@
return nfp_net_bpf_stats_update(nn, cls_bpf);
default:
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
}
}
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c b/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c
index 39407f7..86e61be 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c
@@ -84,12 +84,12 @@
put_unaligned_be16(nn_readw(nn, NFP_NET_CFG_MACADDR + 6), &mac_addr[4]);
if (!is_valid_ether_addr(mac_addr)) {
- eth_hw_addr_random(nn->netdev);
+ eth_hw_addr_random(nn->dp.netdev);
return;
}
- ether_addr_copy(nn->netdev->dev_addr, mac_addr);
- ether_addr_copy(nn->netdev->perm_addr, mac_addr);
+ ether_addr_copy(nn->dp.netdev->dev_addr, mac_addr);
+ ether_addr_copy(nn->dp.netdev->perm_addr, mac_addr);
}
static int nfp_netvf_pci_probe(struct pci_dev *pdev,
@@ -210,8 +210,8 @@
vf->nn = nn;
nn->fw_ver = fw_ver;
- nn->ctrl_bar = ctrl_bar;
- nn->is_vf = 1;
+ nn->dp.ctrl_bar = ctrl_bar;
+ nn->dp.is_vf = 1;
nn->stride_tx = stride;
nn->stride_rx = stride;
@@ -268,7 +268,8 @@
num_irqs = nfp_net_irqs_alloc(pdev, vf->irq_entries,
NFP_NET_MIN_PORT_IRQS,
- NFP_NET_NON_Q_VECTORS + nn->num_r_vecs);
+ NFP_NET_NON_Q_VECTORS +
+ nn->dp.num_r_vecs);
if (!num_irqs) {
nn_warn(nn, "Unable to allocate MSI-X Vectors. Exiting\n");
err = -EIO;
@@ -282,7 +283,7 @@
*/
nn->me_freq_mhz = 1200;
- err = nfp_net_netdev_init(nn->netdev);
+ err = nfp_net_netdev_init(nn->dp.netdev);
if (err)
goto err_irqs_disable;
@@ -327,7 +328,7 @@
nfp_net_debugfs_dir_clean(&nn->debugfs_dir);
nfp_net_debugfs_dir_clean(&vf->ddir);
- nfp_net_netdev_clean(nn->netdev);
+ nfp_net_netdev_clean(nn->dp.netdev);
nfp_net_irqs_disable(pdev);
@@ -337,7 +338,7 @@
} else {
iounmap(vf->q_bar);
}
- iounmap(nn->ctrl_bar);
+ iounmap(nn->dp.ctrl_bar);
nfp_net_netdev_free(nn);
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp.h b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp.h
index 42cb720..4df2ce2 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp.h
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp.h
@@ -48,32 +48,26 @@
const char *nfp_hwinfo_lookup(struct nfp_cpp *cpp, const char *lookup);
-/* Implemented in nfp_nsp.c */
+/* Implemented in nfp_nsp.c, low level functions */
struct nfp_nsp;
-struct firmware;
-struct nfp_nsp *nfp_nsp_open(struct nfp_cpp *cpp);
-void nfp_nsp_close(struct nfp_nsp *state);
-u16 nfp_nsp_get_abi_ver_major(struct nfp_nsp *state);
-u16 nfp_nsp_get_abi_ver_minor(struct nfp_nsp *state);
-int nfp_nsp_wait(struct nfp_nsp *state);
-int nfp_nsp_device_soft_reset(struct nfp_nsp *state);
-int nfp_nsp_load_fw(struct nfp_nsp *state, const struct firmware *fw);
+struct nfp_cpp *nfp_nsp_cpp(struct nfp_nsp *state);
+bool nfp_nsp_config_modified(struct nfp_nsp *state);
+void nfp_nsp_config_set_modified(struct nfp_nsp *state, bool modified);
+void *nfp_nsp_config_entries(struct nfp_nsp *state);
+unsigned int nfp_nsp_config_idx(struct nfp_nsp *state);
+void nfp_nsp_config_set_state(struct nfp_nsp *state, void *entries,
+ unsigned int idx);
+void nfp_nsp_config_clear_state(struct nfp_nsp *state);
int nfp_nsp_read_eth_table(struct nfp_nsp *state, void *buf, unsigned int size);
int nfp_nsp_write_eth_table(struct nfp_nsp *state,
const void *buf, unsigned int size);
+int nfp_nsp_read_identify(struct nfp_nsp *state, void *buf, unsigned int size);
/* Implemented in nfp_resource.c */
-#define NFP_RESOURCE_TBL_TARGET NFP_CPP_TARGET_MU
-#define NFP_RESOURCE_TBL_BASE 0x8100000000ULL
-
-/* NFP Resource Table self-identifier */
-#define NFP_RESOURCE_TBL_NAME "nfp.res"
-#define NFP_RESOURCE_TBL_KEY 0x00000000 /* Special key for entry 0 */
-
-/* All other keys are CRC32-POSIX of the 8-byte identification string */
+/* All keys are CRC32-POSIX of the 8-byte identification string */
/* ARM/PCI vNIC Interfaces 0..3 */
#define NFP_RESOURCE_VNIC_PCI_0 "vnic.p0"
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp6000_pcie.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp6000_pcie.c
index 15cc3e7..43dc68e 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp6000_pcie.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp6000_pcie.c
@@ -217,7 +217,7 @@
#define TARGET_WIDTH_64 8
static int
-compute_bar(struct nfp6000_pcie *nfp, struct nfp_bar *bar,
+compute_bar(const struct nfp6000_pcie *nfp, const struct nfp_bar *bar,
u32 *bar_config, u64 *bar_base,
int tgt, int act, int tok, u64 offset, size_t size, int width)
{
@@ -410,35 +410,36 @@
/* Return EAGAIN if no resource is available */
static int
-find_unused_bar_noblock(struct nfp6000_pcie *nfp,
+find_unused_bar_noblock(const struct nfp6000_pcie *nfp,
int tgt, int act, int tok,
u64 offset, size_t size, int width)
{
- int n, invalid = 0;
+ int n, busy = 0;
for (n = 0; n < nfp->bars; n++) {
- struct nfp_bar *bar = &nfp->bar[n];
+ const struct nfp_bar *bar = &nfp->bar[n];
int err;
- if (bar->bitsize == 0) {
- invalid++;
- continue;
- }
-
- if (atomic_read(&bar->refcnt) != 0)
+ if (!bar->bitsize)
continue;
/* Just check to see if we can make it fit... */
err = compute_bar(nfp, bar, NULL, NULL,
tgt, act, tok, offset, size, width);
+ if (err)
+ continue;
- if (err < 0)
- invalid++;
- else
+ if (!atomic_read(&bar->refcnt))
return n;
+
+ busy++;
}
- return (n == invalid) ? -EINVAL : -EAGAIN;
+ if (WARN(!busy, "No suitable BAR found for request tgt:0x%x act:0x%x tok:0x%x off:0x%llx size:%zd width:%d\n",
+ tgt, act, tok, offset, size, width))
+ return -EINVAL;
+
+ return -EAGAIN;
}
static int
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_cppcore.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_cppcore.c
index 40108e6..e2abba4 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_cppcore.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_cppcore.c
@@ -65,39 +65,49 @@
u64 end;
};
-struct nfp_cpp_mutex {
- struct list_head list;
- struct nfp_cpp *cpp;
- int target;
- u16 usage;
- u16 depth;
- unsigned long long address;
- u32 key;
-};
-
+/**
+ * struct nfp_cpp - main nfpcore device structure
+ * Following fields are read-only after probe() exits or netdevs are spawned.
+ * @dev: embedded device structure
+ * @op: low-level implementation ops
+ * @priv: private data of the low-level implementation
+ * @model: chip model
+ * @interface: chip interface id we are using to reach it
+ * @serial: chip serial number
+ * @imb_cat_table: CPP Mapping Table
+ *
+ * Following fields can be used only in probe() or with rtnl held:
+ * @hwinfo: HWInfo database fetched from the device
+ * @rtsym: firmware run time symbols
+ *
+ * Following fields use explicit locking:
+ * @resource_list: NFP CPP resource list
+ * @resource_lock: protects @resource_list
+ *
+ * @area_cache_list: cached areas for cpp/xpb read/write speed up
+ * @area_cache_mutex: protects @area_cache_list
+ *
+ * @waitq: area wait queue
+ */
struct nfp_cpp {
struct device dev;
- void *priv; /* Private data of the low-level implementation */
+ void *priv;
u32 model;
u16 interface;
u8 serial[NFP_SERIAL_LEN];
const struct nfp_cpp_operations *op;
- struct list_head resource_list; /* NFP CPP resource list */
- struct list_head mutex_cache; /* Mutex cache */
+ struct list_head resource_list;
rwlock_t resource_lock;
wait_queue_head_t waitq;
- /* NFP6000 CPP Mapping Table */
u32 imb_cat_table[16];
- /* Cached areas for cpp/xpb readl/writel speedups */
- struct mutex area_cache_mutex; /* Lock for the area cache */
+ struct mutex area_cache_mutex;
struct list_head area_cache_list;
- /* Cached information */
void *hwinfo;
void *rtsym;
};
@@ -187,24 +197,6 @@
{
struct nfp_cpp_area_cache *cache, *ctmp;
struct nfp_cpp_resource *res, *rtmp;
- struct nfp_cpp_mutex *mutex, *mtmp;
-
- /* There should be no mutexes in the cache at this point. */
- WARN_ON(!list_empty(&cpp->mutex_cache));
- /* .. but if there are, unlock them and complain. */
- list_for_each_entry_safe(mutex, mtmp, &cpp->mutex_cache, list) {
- dev_err(cpp->dev.parent, "Dangling mutex: @%d::0x%llx, %d locks held by %d owners\n",
- mutex->target, (unsigned long long)mutex->address,
- mutex->depth, mutex->usage);
-
- /* Forcing an unlock */
- mutex->depth = 1;
- nfp_cpp_mutex_unlock(mutex);
-
- /* Forcing a free */
- mutex->usage = 1;
- nfp_cpp_mutex_free(mutex);
- }
/* Remove all caches */
list_for_each_entry_safe(cache, ctmp, &cpp->area_cache_list, entry) {
@@ -419,9 +411,43 @@
*/
void nfp_cpp_area_free(struct nfp_cpp_area *area)
{
+ if (atomic_read(&area->refcount))
+ nfp_warn(area->cpp, "Warning: freeing busy area\n");
nfp_cpp_area_put(area);
}
+static bool nfp_cpp_area_acquire_try(struct nfp_cpp_area *area, int *status)
+{
+ *status = area->cpp->op->area_acquire(area);
+
+ return *status != -EAGAIN;
+}
+
+static int __nfp_cpp_area_acquire(struct nfp_cpp_area *area)
+{
+ int err, status;
+
+ if (atomic_inc_return(&area->refcount) > 1)
+ return 0;
+
+ if (!area->cpp->op->area_acquire)
+ return 0;
+
+ err = wait_event_interruptible(area->cpp->waitq,
+ nfp_cpp_area_acquire_try(area, &status));
+ if (!err)
+ err = status;
+ if (err) {
+ nfp_warn(area->cpp, "Warning: area wait failed: %d\n", err);
+ atomic_dec(&area->refcount);
+ return err;
+ }
+
+ nfp_cpp_area_get(area);
+
+ return 0;
+}
+
/**
* nfp_cpp_area_acquire() - lock down a CPP area for access
* @area: CPP area handle
@@ -433,27 +459,13 @@
*/
int nfp_cpp_area_acquire(struct nfp_cpp_area *area)
{
+ int ret;
+
mutex_lock(&area->mutex);
- if (atomic_inc_return(&area->refcount) == 1) {
- int (*a_a)(struct nfp_cpp_area *);
-
- a_a = area->cpp->op->area_acquire;
- if (a_a) {
- int err;
-
- wait_event_interruptible(area->cpp->waitq,
- (err = a_a(area)) != -EAGAIN);
- if (err < 0) {
- atomic_dec(&area->refcount);
- mutex_unlock(&area->mutex);
- return err;
- }
- }
- }
+ ret = __nfp_cpp_area_acquire(area);
mutex_unlock(&area->mutex);
- nfp_cpp_area_get(area);
- return 0;
+ return ret;
}
/**
@@ -829,10 +841,7 @@
* the need for special case code below when
* checking against available cache size.
*/
- if (length == 0)
- return NULL;
-
- if (list_empty(&cpp->area_cache_list) || id == 0)
+ if (length == 0 || id == 0)
return NULL;
/* Remap from cpp_island to cpp_target */
@@ -840,10 +849,15 @@
if (err < 0)
return NULL;
- addr += *offset;
-
mutex_lock(&cpp->area_cache_mutex);
+ if (list_empty(&cpp->area_cache_list)) {
+ mutex_unlock(&cpp->area_cache_mutex);
+ return NULL;
+ }
+
+ addr += *offset;
+
/* See if we have a match */
list_for_each_entry(cache, &cpp->area_cache_list, entry) {
if (id == cache->id &&
@@ -937,12 +951,14 @@
return -ENOMEM;
err = nfp_cpp_area_acquire(area);
- if (err)
- goto out;
+ if (err) {
+ nfp_cpp_area_free(area);
+ return err;
+ }
}
err = nfp_cpp_area_read(area, offset, kernel_vaddr, length);
-out:
+
if (cache)
area_cache_put(cpp, cache);
else
@@ -979,13 +995,14 @@
return -ENOMEM;
err = nfp_cpp_area_acquire(area);
- if (err)
- goto out;
+ if (err) {
+ nfp_cpp_area_free(area);
+ return err;
+ }
}
err = nfp_cpp_area_write(area, offset, kernel_vaddr, length);
-out:
if (cache)
area_cache_put(cpp, cache);
else
@@ -1127,7 +1144,6 @@
rwlock_init(&cpp->resource_lock);
init_waitqueue_head(&cpp->waitq);
lockdep_set_class(&cpp->resource_lock, &nfp_cpp_resource_lock_key);
- INIT_LIST_HEAD(&cpp->mutex_cache);
INIT_LIST_HEAD(&cpp->resource_list);
INIT_LIST_HEAD(&cpp->area_cache_list);
mutex_init(&cpp->area_cache_mutex);
@@ -1425,322 +1441,3 @@
{
return &cpp_explicit[1];
}
-
-/* THIS FUNCTION IS NOT EXPORTED */
-static u32 nfp_mutex_locked(u16 interface)
-{
- return (u32)interface << 16 | 0x000f;
-}
-
-static u32 nfp_mutex_unlocked(u16 interface)
-{
- return (u32)interface << 16 | 0x0000;
-}
-
-static bool nfp_mutex_is_locked(u32 val)
-{
- return (val & 0xffff) == 0x000f;
-}
-
-static bool nfp_mutex_is_unlocked(u32 val)
-{
- return (val & 0xffff) == 0000;
-}
-
-/* If you need more than 65536 recursive locks, please rethink your code. */
-#define MUTEX_DEPTH_MAX 0xffff
-
-static int
-nfp_cpp_mutex_validate(u16 interface, int *target, unsigned long long address)
-{
- /* Not permitted on invalid interfaces */
- if (NFP_CPP_INTERFACE_TYPE_of(interface) ==
- NFP_CPP_INTERFACE_TYPE_INVALID)
- return -EINVAL;
-
- /* Address must be 64-bit aligned */
- if (address & 7)
- return -EINVAL;
-
- if (*target != NFP_CPP_TARGET_MU)
- return -EINVAL;
-
- return 0;
-}
-
-/**
- * nfp_cpp_mutex_init() - Initialize a mutex location
- * @cpp: NFP CPP handle
- * @target: NFP CPP target ID (ie NFP_CPP_TARGET_CLS or NFP_CPP_TARGET_MU)
- * @address: Offset into the address space of the NFP CPP target ID
- * @key: Unique 32-bit value for this mutex
- *
- * The CPP target:address must point to a 64-bit aligned location, and
- * will initialize 64 bits of data at the location.
- *
- * This creates the initial mutex state, as locked by this
- * nfp_cpp_interface().
- *
- * This function should only be called when setting up
- * the initial lock state upon boot-up of the system.
- *
- * Return: 0 on success, or -errno on failure
- */
-int nfp_cpp_mutex_init(struct nfp_cpp *cpp,
- int target, unsigned long long address, u32 key)
-{
- const u32 muw = NFP_CPP_ID(target, 4, 0); /* atomic_write */
- u16 interface = nfp_cpp_interface(cpp);
- int err;
-
- err = nfp_cpp_mutex_validate(interface, &target, address);
- if (err)
- return err;
-
- err = nfp_cpp_writel(cpp, muw, address + 4, key);
- if (err)
- return err;
-
- err = nfp_cpp_writel(cpp, muw, address, nfp_mutex_locked(interface));
- if (err)
- return err;
-
- return 0;
-}
-
-/**
- * nfp_cpp_mutex_alloc() - Create a mutex handle
- * @cpp: NFP CPP handle
- * @target: NFP CPP target ID (ie NFP_CPP_TARGET_CLS or NFP_CPP_TARGET_MU)
- * @address: Offset into the address space of the NFP CPP target ID
- * @key: 32-bit unique key (must match the key at this location)
- *
- * The CPP target:address must point to a 64-bit aligned location, and
- * reserve 64 bits of data at the location for use by the handle.
- *
- * Only target/address pairs that point to entities that support the
- * MU Atomic Engine's CmpAndSwap32 command are supported.
- *
- * Return: A non-NULL struct nfp_cpp_mutex * on success, NULL on failure.
- */
-struct nfp_cpp_mutex *nfp_cpp_mutex_alloc(struct nfp_cpp *cpp, int target,
- unsigned long long address, u32 key)
-{
- const u32 mur = NFP_CPP_ID(target, 3, 0); /* atomic_read */
- u16 interface = nfp_cpp_interface(cpp);
- struct nfp_cpp_mutex *mutex;
- int err;
- u32 tmp;
-
- err = nfp_cpp_mutex_validate(interface, &target, address);
- if (err)
- return NULL;
-
- /* Look for mutex on cache list */
- list_for_each_entry(mutex, &cpp->mutex_cache, list) {
- if (mutex->target == target && mutex->address == address) {
- mutex->usage++;
- return mutex;
- }
- }
-
- err = nfp_cpp_readl(cpp, mur, address + 4, &tmp);
- if (err < 0)
- return NULL;
-
- if (tmp != key)
- return NULL;
-
- mutex = kzalloc(sizeof(*mutex), GFP_KERNEL);
- if (!mutex)
- return NULL;
-
- mutex->cpp = cpp;
- mutex->target = target;
- mutex->address = address;
- mutex->key = key;
- mutex->depth = 0;
- mutex->usage = 1;
-
- /* Add mutex to cache list */
- list_add(&mutex->list, &cpp->mutex_cache);
-
- return mutex;
-}
-
-/**
- * nfp_cpp_mutex_free() - Free a mutex handle - does not alter the lock state
- * @mutex: NFP CPP Mutex handle
- */
-void nfp_cpp_mutex_free(struct nfp_cpp_mutex *mutex)
-{
- if (--mutex->usage)
- return;
-
- /* Remove mutex from cache */
- list_del(&mutex->list);
- kfree(mutex);
-}
-
-/**
- * nfp_cpp_mutex_lock() - Lock a mutex handle, using the NFP MU Atomic Engine
- * @mutex: NFP CPP Mutex handle
- *
- * Return: 0 on success, or -errno on failure
- */
-int nfp_cpp_mutex_lock(struct nfp_cpp_mutex *mutex)
-{
- unsigned long warn_at = jiffies + 15 * HZ;
- unsigned int timeout_ms = 1;
- int err;
-
- /* We can't use a waitqueue here, because the unlocker
- * might be on a separate CPU.
- *
- * So just wait for now.
- */
- for (;;) {
- err = nfp_cpp_mutex_trylock(mutex);
- if (err != -EBUSY)
- break;
-
- err = msleep_interruptible(timeout_ms);
- if (err != 0)
- return -ERESTARTSYS;
-
- if (time_is_before_eq_jiffies(warn_at)) {
- warn_at = jiffies + 60 * HZ;
- dev_warn(mutex->cpp->dev.parent,
- "Warning: waiting for NFP mutex [usage:%hd depth:%hd target:%d addr:%llx key:%08x]\n",
- mutex->usage, mutex->depth,
- mutex->target, mutex->address, mutex->key);
- }
- }
-
- return err;
-}
-
-/**
- * nfp_cpp_mutex_unlock() - Unlock a mutex handle, using the MU Atomic Engine
- * @mutex: NFP CPP Mutex handle
- *
- * Return: 0 on success, or -errno on failure
- */
-int nfp_cpp_mutex_unlock(struct nfp_cpp_mutex *mutex)
-{
- const u32 muw = NFP_CPP_ID(mutex->target, 4, 0); /* atomic_write */
- const u32 mur = NFP_CPP_ID(mutex->target, 3, 0); /* atomic_read */
- struct nfp_cpp *cpp = mutex->cpp;
- u32 key, value;
- u16 interface;
- int err;
-
- interface = nfp_cpp_interface(cpp);
-
- if (mutex->depth > 1) {
- mutex->depth--;
- return 0;
- }
-
- err = nfp_cpp_readl(mutex->cpp, mur, mutex->address + 4, &key);
- if (err < 0)
- return err;
-
- if (key != mutex->key)
- return -EPERM;
-
- err = nfp_cpp_readl(mutex->cpp, mur, mutex->address, &value);
- if (err < 0)
- return err;
-
- if (value != nfp_mutex_locked(interface))
- return -EACCES;
-
- err = nfp_cpp_writel(cpp, muw, mutex->address,
- nfp_mutex_unlocked(interface));
- if (err < 0)
- return err;
-
- mutex->depth = 0;
- return 0;
-}
-
-/**
- * nfp_cpp_mutex_trylock() - Attempt to lock a mutex handle
- * @mutex: NFP CPP Mutex handle
- *
- * Return: 0 if the lock succeeded, -errno on failure
- */
-int nfp_cpp_mutex_trylock(struct nfp_cpp_mutex *mutex)
-{
- const u32 muw = NFP_CPP_ID(mutex->target, 4, 0); /* atomic_write */
- const u32 mus = NFP_CPP_ID(mutex->target, 5, 3); /* test_set_imm */
- const u32 mur = NFP_CPP_ID(mutex->target, 3, 0); /* atomic_read */
- struct nfp_cpp *cpp = mutex->cpp;
- u32 key, value, tmp;
- int err;
-
- if (mutex->depth > 0) {
- if (mutex->depth == MUTEX_DEPTH_MAX)
- return -E2BIG;
- mutex->depth++;
- return 0;
- }
-
- /* Verify that the lock marker is not damaged */
- err = nfp_cpp_readl(cpp, mur, mutex->address + 4, &key);
- if (err < 0)
- return err;
-
- if (key != mutex->key)
- return -EPERM;
-
- /* Compare against the unlocked state, and if true,
- * write the interface id into the top 16 bits, and
- * mark as locked.
- */
- value = nfp_mutex_locked(nfp_cpp_interface(cpp));
-
- /* We use test_set_imm here, as it implies a read
- * of the current state, and sets the bits in the
- * bytemask of the command to 1s. Since the mutex
- * is guaranteed to be 64-bit aligned, the bytemask
- * of this 32-bit command is ensured to be 8'b00001111,
- * which implies that the lower 4 bits will be set to
- * ones regardless of the initial state.
- *
- * Since this is a 'Readback' operation, with no Pull
- * data, we can treat this as a normal Push (read)
- * atomic, which returns the original value.
- */
- err = nfp_cpp_readl(cpp, mus, mutex->address, &tmp);
- if (err < 0)
- return err;
-
- /* Was it unlocked? */
- if (nfp_mutex_is_unlocked(tmp)) {
- /* The read value can only be 0x....0000 in the unlocked state.
- * If there was another contending for this lock, then
- * the lock state would be 0x....000f
- */
-
- /* Write our owner ID into the lock
- * While not strictly necessary, this helps with
- * debug and bookkeeping.
- */
- err = nfp_cpp_writel(cpp, muw, mutex->address, value);
- if (err < 0)
- return err;
-
- mutex->depth = 1;
- return 0;
- }
-
- /* Already locked by us? Success! */
- if (tmp == value) {
- mutex->depth = 1;
- return 0;
- }
-
- return nfp_mutex_is_locked(tmp) ? -EBUSY : -EINVAL;
-}
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_mutex.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_mutex.c
new file mode 100644
index 0000000..8a99c18
--- /dev/null
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_mutex.c
@@ -0,0 +1,345 @@
+/*
+ * Copyright (C) 2015-2017 Netronome Systems, Inc.
+ *
+ * This software is dual licensed under the GNU General License Version 2,
+ * June 1991 as shown in the file COPYING in the top-level directory of this
+ * source tree or the BSD 2-Clause License provided below. You have the
+ * option to license this software under the complete terms of either license.
+ *
+ * The BSD 2-Clause License:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/jiffies.h>
+#include <linux/types.h>
+#include <linux/slab.h>
+#include <linux/wait.h>
+
+#include "nfp_cpp.h"
+#include "nfp6000/nfp6000.h"
+
+struct nfp_cpp_mutex {
+ struct nfp_cpp *cpp;
+ int target;
+ u16 depth;
+ unsigned long long address;
+ u32 key;
+};
+
+static u32 nfp_mutex_locked(u16 interface)
+{
+ return (u32)interface << 16 | 0x000f;
+}
+
+static u32 nfp_mutex_unlocked(u16 interface)
+{
+ return (u32)interface << 16 | 0x0000;
+}
+
+static bool nfp_mutex_is_locked(u32 val)
+{
+ return (val & 0xffff) == 0x000f;
+}
+
+static bool nfp_mutex_is_unlocked(u32 val)
+{
+ return (val & 0xffff) == 0000;
+}
+
+/* If you need more than 65536 recursive locks, please rethink your code. */
+#define NFP_MUTEX_DEPTH_MAX 0xffff
+
+static int
+nfp_cpp_mutex_validate(u16 interface, int *target, unsigned long long address)
+{
+ /* Not permitted on invalid interfaces */
+ if (NFP_CPP_INTERFACE_TYPE_of(interface) ==
+ NFP_CPP_INTERFACE_TYPE_INVALID)
+ return -EINVAL;
+
+ /* Address must be 64-bit aligned */
+ if (address & 7)
+ return -EINVAL;
+
+ if (*target != NFP_CPP_TARGET_MU)
+ return -EINVAL;
+
+ return 0;
+}
+
+/**
+ * nfp_cpp_mutex_init() - Initialize a mutex location
+ * @cpp: NFP CPP handle
+ * @target: NFP CPP target ID (ie NFP_CPP_TARGET_CLS or NFP_CPP_TARGET_MU)
+ * @address: Offset into the address space of the NFP CPP target ID
+ * @key: Unique 32-bit value for this mutex
+ *
+ * The CPP target:address must point to a 64-bit aligned location, and
+ * will initialize 64 bits of data at the location.
+ *
+ * This creates the initial mutex state, as locked by this
+ * nfp_cpp_interface().
+ *
+ * This function should only be called when setting up
+ * the initial lock state upon boot-up of the system.
+ *
+ * Return: 0 on success, or -errno on failure
+ */
+int nfp_cpp_mutex_init(struct nfp_cpp *cpp,
+ int target, unsigned long long address, u32 key)
+{
+ const u32 muw = NFP_CPP_ID(target, 4, 0); /* atomic_write */
+ u16 interface = nfp_cpp_interface(cpp);
+ int err;
+
+ err = nfp_cpp_mutex_validate(interface, &target, address);
+ if (err)
+ return err;
+
+ err = nfp_cpp_writel(cpp, muw, address + 4, key);
+ if (err)
+ return err;
+
+ err = nfp_cpp_writel(cpp, muw, address, nfp_mutex_locked(interface));
+ if (err)
+ return err;
+
+ return 0;
+}
+
+/**
+ * nfp_cpp_mutex_alloc() - Create a mutex handle
+ * @cpp: NFP CPP handle
+ * @target: NFP CPP target ID (ie NFP_CPP_TARGET_CLS or NFP_CPP_TARGET_MU)
+ * @address: Offset into the address space of the NFP CPP target ID
+ * @key: 32-bit unique key (must match the key at this location)
+ *
+ * The CPP target:address must point to a 64-bit aligned location, and
+ * reserve 64 bits of data at the location for use by the handle.
+ *
+ * Only target/address pairs that point to entities that support the
+ * MU Atomic Engine's CmpAndSwap32 command are supported.
+ *
+ * Return: A non-NULL struct nfp_cpp_mutex * on success, NULL on failure.
+ */
+struct nfp_cpp_mutex *nfp_cpp_mutex_alloc(struct nfp_cpp *cpp, int target,
+ unsigned long long address, u32 key)
+{
+ const u32 mur = NFP_CPP_ID(target, 3, 0); /* atomic_read */
+ u16 interface = nfp_cpp_interface(cpp);
+ struct nfp_cpp_mutex *mutex;
+ int err;
+ u32 tmp;
+
+ err = nfp_cpp_mutex_validate(interface, &target, address);
+ if (err)
+ return NULL;
+
+ err = nfp_cpp_readl(cpp, mur, address + 4, &tmp);
+ if (err < 0)
+ return NULL;
+
+ if (tmp != key)
+ return NULL;
+
+ mutex = kzalloc(sizeof(*mutex), GFP_KERNEL);
+ if (!mutex)
+ return NULL;
+
+ mutex->cpp = cpp;
+ mutex->target = target;
+ mutex->address = address;
+ mutex->key = key;
+ mutex->depth = 0;
+
+ return mutex;
+}
+
+/**
+ * nfp_cpp_mutex_free() - Free a mutex handle - does not alter the lock state
+ * @mutex: NFP CPP Mutex handle
+ */
+void nfp_cpp_mutex_free(struct nfp_cpp_mutex *mutex)
+{
+ kfree(mutex);
+}
+
+/**
+ * nfp_cpp_mutex_lock() - Lock a mutex handle, using the NFP MU Atomic Engine
+ * @mutex: NFP CPP Mutex handle
+ *
+ * Return: 0 on success, or -errno on failure
+ */
+int nfp_cpp_mutex_lock(struct nfp_cpp_mutex *mutex)
+{
+ unsigned long warn_at = jiffies + 15 * HZ;
+ unsigned int timeout_ms = 1;
+ int err;
+
+ /* We can't use a waitqueue here, because the unlocker
+ * might be on a separate CPU.
+ *
+ * So just wait for now.
+ */
+ for (;;) {
+ err = nfp_cpp_mutex_trylock(mutex);
+ if (err != -EBUSY)
+ break;
+
+ err = msleep_interruptible(timeout_ms);
+ if (err != 0)
+ return -ERESTARTSYS;
+
+ if (time_is_before_eq_jiffies(warn_at)) {
+ warn_at = jiffies + 60 * HZ;
+ nfp_warn(mutex->cpp,
+ "Warning: waiting for NFP mutex [depth:%hd target:%d addr:%llx key:%08x]\n",
+ mutex->depth,
+ mutex->target, mutex->address, mutex->key);
+ }
+ }
+
+ return err;
+}
+
+/**
+ * nfp_cpp_mutex_unlock() - Unlock a mutex handle, using the MU Atomic Engine
+ * @mutex: NFP CPP Mutex handle
+ *
+ * Return: 0 on success, or -errno on failure
+ */
+int nfp_cpp_mutex_unlock(struct nfp_cpp_mutex *mutex)
+{
+ const u32 muw = NFP_CPP_ID(mutex->target, 4, 0); /* atomic_write */
+ const u32 mur = NFP_CPP_ID(mutex->target, 3, 0); /* atomic_read */
+ struct nfp_cpp *cpp = mutex->cpp;
+ u32 key, value;
+ u16 interface;
+ int err;
+
+ interface = nfp_cpp_interface(cpp);
+
+ if (mutex->depth > 1) {
+ mutex->depth--;
+ return 0;
+ }
+
+ err = nfp_cpp_readl(mutex->cpp, mur, mutex->address + 4, &key);
+ if (err < 0)
+ return err;
+
+ if (key != mutex->key)
+ return -EPERM;
+
+ err = nfp_cpp_readl(mutex->cpp, mur, mutex->address, &value);
+ if (err < 0)
+ return err;
+
+ if (value != nfp_mutex_locked(interface))
+ return -EACCES;
+
+ err = nfp_cpp_writel(cpp, muw, mutex->address,
+ nfp_mutex_unlocked(interface));
+ if (err < 0)
+ return err;
+
+ mutex->depth = 0;
+ return 0;
+}
+
+/**
+ * nfp_cpp_mutex_trylock() - Attempt to lock a mutex handle
+ * @mutex: NFP CPP Mutex handle
+ *
+ * Return: 0 if the lock succeeded, -errno on failure
+ */
+int nfp_cpp_mutex_trylock(struct nfp_cpp_mutex *mutex)
+{
+ const u32 muw = NFP_CPP_ID(mutex->target, 4, 0); /* atomic_write */
+ const u32 mus = NFP_CPP_ID(mutex->target, 5, 3); /* test_set_imm */
+ const u32 mur = NFP_CPP_ID(mutex->target, 3, 0); /* atomic_read */
+ struct nfp_cpp *cpp = mutex->cpp;
+ u32 key, value, tmp;
+ int err;
+
+ if (mutex->depth > 0) {
+ if (mutex->depth == NFP_MUTEX_DEPTH_MAX)
+ return -E2BIG;
+ mutex->depth++;
+ return 0;
+ }
+
+ /* Verify that the lock marker is not damaged */
+ err = nfp_cpp_readl(cpp, mur, mutex->address + 4, &key);
+ if (err < 0)
+ return err;
+
+ if (key != mutex->key)
+ return -EPERM;
+
+ /* Compare against the unlocked state, and if true,
+ * write the interface id into the top 16 bits, and
+ * mark as locked.
+ */
+ value = nfp_mutex_locked(nfp_cpp_interface(cpp));
+
+ /* We use test_set_imm here, as it implies a read
+ * of the current state, and sets the bits in the
+ * bytemask of the command to 1s. Since the mutex
+ * is guaranteed to be 64-bit aligned, the bytemask
+ * of this 32-bit command is ensured to be 8'b00001111,
+ * which implies that the lower 4 bits will be set to
+ * ones regardless of the initial state.
+ *
+ * Since this is a 'Readback' operation, with no Pull
+ * data, we can treat this as a normal Push (read)
+ * atomic, which returns the original value.
+ */
+ err = nfp_cpp_readl(cpp, mus, mutex->address, &tmp);
+ if (err < 0)
+ return err;
+
+ /* Was it unlocked? */
+ if (nfp_mutex_is_unlocked(tmp)) {
+ /* The read value can only be 0x....0000 in the unlocked state.
+ * If there was another contending for this lock, then
+ * the lock state would be 0x....000f
+ */
+
+ /* Write our owner ID into the lock
+ * While not strictly necessary, this helps with
+ * debug and bookkeeping.
+ */
+ err = nfp_cpp_writel(cpp, muw, mutex->address, value);
+ if (err < 0)
+ return err;
+
+ mutex->depth = 1;
+ return 0;
+ }
+
+ return nfp_mutex_is_locked(tmp) ? -EBUSY : -EINVAL;
+}
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
index 34c5098..2fa9247 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.c
@@ -49,6 +49,7 @@
#include "nfp.h"
#include "nfp_cpp.h"
+#include "nfp_nsp.h"
/* Offsets relative to the CSR base */
#define NSP_STATUS 0x00
@@ -77,7 +78,7 @@
#define NSP_MAGIC 0xab10
#define NSP_MAJOR 0
-#define NSP_MINOR (__MAX_SPCODE - 1)
+#define NSP_MINOR 8
#define NSP_CODE_MAJOR GENMASK(15, 12)
#define NSP_CODE_MINOR GENMASK(11, 0)
@@ -92,8 +93,18 @@
SPCODE_FW_LOAD = 6, /* Load fw from buffer, len in option */
SPCODE_ETH_RESCAN = 7, /* Rescan ETHs, write ETH_TABLE to buf */
SPCODE_ETH_CONTROL = 8, /* Update media config from buffer */
+ SPCODE_NSP_IDENTIFY = 13, /* Read NSP version */
+};
- __MAX_SPCODE,
+static const struct {
+ int code;
+ const char *msg;
+} nsp_errors[] = {
+ { 6010, "could not map to phy for port" },
+ { 6011, "not an allowed rate/lanes for port" },
+ { 6012, "not an allowed rate/lanes for port" },
+ { 6013, "high/low error, change other port first" },
+ { 6014, "config not found in flash" },
};
struct nfp_nsp {
@@ -103,8 +114,63 @@
u16 major;
u16 minor;
} ver;
+
+ /* Eth table config state */
+ bool modified;
+ unsigned int idx;
+ void *entries;
};
+struct nfp_cpp *nfp_nsp_cpp(struct nfp_nsp *state)
+{
+ return state->cpp;
+}
+
+bool nfp_nsp_config_modified(struct nfp_nsp *state)
+{
+ return state->modified;
+}
+
+void nfp_nsp_config_set_modified(struct nfp_nsp *state, bool modified)
+{
+ state->modified = modified;
+}
+
+void *nfp_nsp_config_entries(struct nfp_nsp *state)
+{
+ return state->entries;
+}
+
+unsigned int nfp_nsp_config_idx(struct nfp_nsp *state)
+{
+ return state->idx;
+}
+
+void
+nfp_nsp_config_set_state(struct nfp_nsp *state, void *entries, unsigned int idx)
+{
+ state->entries = entries;
+ state->idx = idx;
+}
+
+void nfp_nsp_config_clear_state(struct nfp_nsp *state)
+{
+ state->entries = NULL;
+ state->idx = 0;
+}
+
+static void nfp_nsp_print_extended_error(struct nfp_nsp *state, u32 ret_val)
+{
+ int i;
+
+ if (!ret_val)
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(nsp_errors); i++)
+ if (ret_val == nsp_errors[i].code)
+ nfp_err(state->cpp, "err msg: %s\n", nsp_errors[i].msg);
+}
+
static int nfp_nsp_check(struct nfp_nsp *state)
{
struct nfp_cpp *cpp = state->cpp;
@@ -209,9 +275,8 @@
if ((*reg & mask) == val)
return 0;
- err = msleep_interruptible(100);
- if (err)
- return err;
+ if (msleep_interruptible(25))
+ return -ERESTARTSYS;
if (time_after(start_time, wait_until))
return -ETIMEDOUT;
@@ -228,7 +293,7 @@
*
* Return: 0 for success with no result
*
- * 1..255 for NSP completion with a result code
+ * positive value for NSP completion with a result code
*
* -EAGAIN if the NSP is not yet present
* -ENODEV if the NSP is not a supported model
@@ -239,7 +304,7 @@
static int nfp_nsp_command(struct nfp_nsp *state, u16 code, u32 option,
u32 buff_cpp, u64 buff_addr)
{
- u64 reg, nsp_base, nsp_buffer, nsp_status, nsp_command;
+ u64 reg, ret_val, nsp_base, nsp_buffer, nsp_status, nsp_command;
struct nfp_cpp *cpp = state->cpp;
u32 nsp_cpp;
int err;
@@ -292,18 +357,20 @@
return err;
}
+ err = nfp_cpp_readq(cpp, nsp_cpp, nsp_command, &ret_val);
+ if (err < 0)
+ return err;
+ ret_val = FIELD_GET(NSP_COMMAND_OPTION, ret_val);
+
err = FIELD_GET(NSP_STATUS_RESULT, reg);
if (err) {
- nfp_warn(cpp, "Result (error) code set: %d command: %d\n",
- -err, code);
+ nfp_warn(cpp, "Result (error) code set: %d (%d) command: %d\n",
+ -err, (int)ret_val, code);
+ nfp_nsp_print_extended_error(state, ret_val);
return -err;
}
- err = nfp_cpp_readq(cpp, nsp_cpp, nsp_command, ®);
- if (err < 0)
- return err;
-
- return FIELD_GET(NSP_COMMAND_OPTION, reg);
+ return ret_val;
}
static int nfp_nsp_command_buf(struct nfp_nsp *nsp, u16 code, u32 option,
@@ -380,9 +447,10 @@
if (err != -EAGAIN)
break;
- err = msleep_interruptible(100);
- if (err)
+ if (msleep_interruptible(25)) {
+ err = -ERESTARTSYS;
break;
+ }
if (time_after(start_time, wait_until)) {
err = -ETIMEDOUT;
@@ -424,3 +492,9 @@
return nfp_nsp_command_buf(state, SPCODE_ETH_CONTROL, size, buf, size,
NULL, 0);
}
+
+int nfp_nsp_read_identify(struct nfp_nsp *state, void *buf, unsigned int size)
+{
+ return nfp_nsp_command_buf(state, SPCODE_NSP_IDENTIFY, size, NULL, 0,
+ buf, size);
+}
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.h b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.h
new file mode 100644
index 0000000..36b21e4
--- /dev/null
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp.h
@@ -0,0 +1,174 @@
+/*
+ * Copyright (C) 2015-2017 Netronome Systems, Inc.
+ *
+ * This software is dual licensed under the GNU General License Version 2,
+ * June 1991 as shown in the file COPYING in the top-level directory of this
+ * source tree or the BSD 2-Clause License provided below. You have the
+ * option to license this software under the complete terms of either license.
+ *
+ * The BSD 2-Clause License:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#ifndef NSP_NSP_H
+#define NSP_NSP_H 1
+
+#include <linux/types.h>
+#include <linux/if_ether.h>
+
+struct firmware;
+struct nfp_cpp;
+struct nfp_nsp;
+
+struct nfp_nsp *nfp_nsp_open(struct nfp_cpp *cpp);
+void nfp_nsp_close(struct nfp_nsp *state);
+u16 nfp_nsp_get_abi_ver_major(struct nfp_nsp *state);
+u16 nfp_nsp_get_abi_ver_minor(struct nfp_nsp *state);
+int nfp_nsp_wait(struct nfp_nsp *state);
+int nfp_nsp_device_soft_reset(struct nfp_nsp *state);
+int nfp_nsp_load_fw(struct nfp_nsp *state, const struct firmware *fw);
+
+enum nfp_eth_interface {
+ NFP_INTERFACE_NONE = 0,
+ NFP_INTERFACE_SFP = 1,
+ NFP_INTERFACE_SFPP = 10,
+ NFP_INTERFACE_SFP28 = 28,
+ NFP_INTERFACE_QSFP = 40,
+ NFP_INTERFACE_CXP = 100,
+ NFP_INTERFACE_QSFP28 = 112,
+};
+
+enum nfp_eth_media {
+ NFP_MEDIA_DAC_PASSIVE = 0,
+ NFP_MEDIA_DAC_ACTIVE,
+ NFP_MEDIA_FIBRE,
+};
+
+enum nfp_eth_aneg {
+ NFP_ANEG_AUTO = 0,
+ NFP_ANEG_SEARCH,
+ NFP_ANEG_25G_CONSORTIUM,
+ NFP_ANEG_25G_IEEE,
+ NFP_ANEG_DISABLED,
+};
+
+/**
+ * struct nfp_eth_table - ETH table information
+ * @count: number of table entries
+ * @ports: table of ports
+ *
+ * @eth_index: port index according to legacy ethX numbering
+ * @index: chip-wide first channel index
+ * @nbi: NBI index
+ * @base: first channel index (within NBI)
+ * @lanes: number of channels
+ * @speed: interface speed (in Mbps)
+ * @interface: interface (module) plugged in
+ * @media: media type of the @interface
+ * @aneg: auto negotiation mode
+ * @mac_addr: interface MAC address
+ * @label_port: port id
+ * @label_subport: id of interface within port (for split ports)
+ * @enabled: is enabled?
+ * @tx_enabled: is TX enabled?
+ * @rx_enabled: is RX enabled?
+ * @override_changed: is media reconfig pending?
+ *
+ * @port_type: one of %PORT_* defines for ethtool
+ * @is_split: is interface part of a split port
+ */
+struct nfp_eth_table {
+ unsigned int count;
+ struct nfp_eth_table_port {
+ unsigned int eth_index;
+ unsigned int index;
+ unsigned int nbi;
+ unsigned int base;
+ unsigned int lanes;
+ unsigned int speed;
+
+ unsigned int interface;
+ enum nfp_eth_media media;
+
+ enum nfp_eth_aneg aneg;
+
+ u8 mac_addr[ETH_ALEN];
+
+ u8 label_port;
+ u8 label_subport;
+
+ bool enabled;
+ bool tx_enabled;
+ bool rx_enabled;
+
+ bool override_changed;
+
+ /* Computed fields */
+ u8 port_type;
+
+ bool is_split;
+ } ports[0];
+};
+
+struct nfp_eth_table *nfp_eth_read_ports(struct nfp_cpp *cpp);
+struct nfp_eth_table *
+__nfp_eth_read_ports(struct nfp_cpp *cpp, struct nfp_nsp *nsp);
+
+int nfp_eth_set_mod_enable(struct nfp_cpp *cpp, unsigned int idx, bool enable);
+int nfp_eth_set_configured(struct nfp_cpp *cpp, unsigned int idx,
+ bool configed);
+
+struct nfp_nsp *nfp_eth_config_start(struct nfp_cpp *cpp, unsigned int idx);
+int nfp_eth_config_commit_end(struct nfp_nsp *nsp);
+void nfp_eth_config_cleanup_end(struct nfp_nsp *nsp);
+
+int __nfp_eth_set_aneg(struct nfp_nsp *nsp, enum nfp_eth_aneg mode);
+int __nfp_eth_set_speed(struct nfp_nsp *nsp, unsigned int speed);
+int __nfp_eth_set_split(struct nfp_nsp *nsp, unsigned int lanes);
+
+/**
+ * struct nfp_nsp_identify - NSP static information
+ * @version: opaque version string
+ * @flags: version flags
+ * @br_primary: branch id of primary bootloader
+ * @br_secondary: branch id of secondary bootloader
+ * @br_nsp: branch id of NSP
+ * @primary: version of primarary bootloader
+ * @secondary: version id of secondary bootloader
+ * @nsp: version id of NSP
+ */
+struct nfp_nsp_identify {
+ char version[40];
+ u8 flags;
+ u8 br_primary;
+ u8 br_secondary;
+ u8 br_nsp;
+ u16 primary;
+ u16 secondary;
+ u16 nsp;
+};
+
+struct nfp_nsp_identify *__nfp_nsp_identify(struct nfp_nsp *nsp);
+
+#endif
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp_cmds.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp_cmds.c
new file mode 100644
index 0000000..e7a263d
--- /dev/null
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp_cmds.c
@@ -0,0 +1,89 @@
+/*
+ * Copyright (C) 2017 Netronome Systems, Inc.
+ *
+ * This software is dual licensed under the GNU General License Version 2,
+ * June 1991 as shown in the file COPYING in the top-level directory of this
+ * source tree or the BSD 2-Clause License provided below. You have the
+ * option to license this software under the complete terms of either license.
+ *
+ * The BSD 2-Clause License:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include <linux/kernel.h>
+#include <linux/slab.h>
+
+#include "nfp.h"
+#include "nfp_nsp.h"
+
+struct nsp_identify {
+ u8 version[40];
+ u8 flags;
+ u8 br_primary;
+ u8 br_secondary;
+ u8 br_nsp;
+ __le16 primary;
+ __le16 secondary;
+ __le16 nsp;
+ __le16 reserved;
+};
+
+struct nfp_nsp_identify *__nfp_nsp_identify(struct nfp_nsp *nsp)
+{
+ struct nfp_nsp_identify *nspi = NULL;
+ struct nsp_identify *ni;
+ int ret;
+
+ if (nfp_nsp_get_abi_ver_minor(nsp) < 15)
+ return NULL;
+
+ ni = kzalloc(sizeof(*ni), GFP_KERNEL);
+ if (!ni)
+ return NULL;
+
+ ret = nfp_nsp_read_identify(nsp, ni, sizeof(*ni));
+ if (ret < 0) {
+ nfp_err(nfp_nsp_cpp(nsp), "reading bsp version failed %d\n",
+ ret);
+ goto exit_free;
+ }
+
+ nspi = kzalloc(sizeof(*nspi), GFP_KERNEL);
+ if (!nspi)
+ goto exit_free;
+
+ memcpy(nspi->version, ni->version, sizeof(nspi->version));
+ nspi->version[sizeof(nspi->version) - 1] = '\0';
+ nspi->flags = ni->flags;
+ nspi->br_primary = ni->br_primary;
+ nspi->br_secondary = ni->br_secondary;
+ nspi->br_nsp = ni->br_nsp;
+ nspi->primary = le16_to_cpu(ni->primary);
+ nspi->secondary = le16_to_cpu(ni->secondary);
+ nspi->nsp = le16_to_cpu(ni->nsp);
+
+exit_free:
+ kfree(ni);
+ return nspi;
+}
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp_eth.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp_eth.c
index 1ece1f8..639438d 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp_eth.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp_eth.c
@@ -43,13 +43,13 @@
#include <linux/module.h>
#include "nfp.h"
-#include "nfp_nsp_eth.h"
+#include "nfp_nsp.h"
#include "nfp6000/nfp6000.h"
#define NSP_ETH_NBI_PORT_COUNT 24
#define NSP_ETH_MAX_COUNT (2 * NSP_ETH_NBI_PORT_COUNT)
#define NSP_ETH_TABLE_SIZE (NSP_ETH_MAX_COUNT * \
- sizeof(struct eth_table_entry))
+ sizeof(union eth_table_entry))
#define NSP_ETH_PORT_LANES GENMASK_ULL(3, 0)
#define NSP_ETH_PORT_INDEX GENMASK_ULL(15, 8)
@@ -58,14 +58,32 @@
#define NSP_ETH_PORT_LANES_MASK cpu_to_le64(NSP_ETH_PORT_LANES)
+#define NSP_ETH_STATE_CONFIGURED BIT_ULL(0)
#define NSP_ETH_STATE_ENABLED BIT_ULL(1)
#define NSP_ETH_STATE_TX_ENABLED BIT_ULL(2)
#define NSP_ETH_STATE_RX_ENABLED BIT_ULL(3)
#define NSP_ETH_STATE_RATE GENMASK_ULL(11, 8)
+#define NSP_ETH_STATE_INTERFACE GENMASK_ULL(19, 12)
+#define NSP_ETH_STATE_MEDIA GENMASK_ULL(21, 20)
+#define NSP_ETH_STATE_OVRD_CHNG BIT_ULL(22)
+#define NSP_ETH_STATE_ANEG GENMASK_ULL(25, 23)
+#define NSP_ETH_CTRL_CONFIGURED BIT_ULL(0)
#define NSP_ETH_CTRL_ENABLED BIT_ULL(1)
#define NSP_ETH_CTRL_TX_ENABLED BIT_ULL(2)
#define NSP_ETH_CTRL_RX_ENABLED BIT_ULL(3)
+#define NSP_ETH_CTRL_SET_RATE BIT_ULL(4)
+#define NSP_ETH_CTRL_SET_LANES BIT_ULL(5)
+#define NSP_ETH_CTRL_SET_ANEG BIT_ULL(6)
+
+enum nfp_eth_raw {
+ NSP_ETH_RAW_PORT = 0,
+ NSP_ETH_RAW_STATE,
+ NSP_ETH_RAW_MAC,
+ NSP_ETH_RAW_CONTROL,
+
+ NSP_ETH_NUM_RAW
+};
enum nfp_eth_rate {
RATE_INVALID = 0,
@@ -76,29 +94,49 @@
RATE_25G,
};
-struct eth_table_entry {
- __le64 port;
- __le64 state;
- u8 mac_addr[6];
- u8 resv[2];
- __le64 control;
+union eth_table_entry {
+ struct {
+ __le64 port;
+ __le64 state;
+ u8 mac_addr[6];
+ u8 resv[2];
+ __le64 control;
+ };
+ __le64 raw[NSP_ETH_NUM_RAW];
};
-static unsigned int nfp_eth_rate(enum nfp_eth_rate rate)
+static const struct {
+ enum nfp_eth_rate rate;
+ unsigned int speed;
+} nsp_eth_rate_tbl[] = {
+ { RATE_INVALID, 0, },
+ { RATE_10M, SPEED_10, },
+ { RATE_100M, SPEED_100, },
+ { RATE_1G, SPEED_1000, },
+ { RATE_10G, SPEED_10000, },
+ { RATE_25G, SPEED_25000, },
+};
+
+static unsigned int nfp_eth_rate2speed(enum nfp_eth_rate rate)
{
- unsigned int rate_xlate[] = {
- [RATE_INVALID] = 0,
- [RATE_10M] = SPEED_10,
- [RATE_100M] = SPEED_100,
- [RATE_1G] = SPEED_1000,
- [RATE_10G] = SPEED_10000,
- [RATE_25G] = SPEED_25000,
- };
+ int i;
- if (rate >= ARRAY_SIZE(rate_xlate))
- return 0;
+ for (i = 0; i < ARRAY_SIZE(nsp_eth_rate_tbl); i++)
+ if (nsp_eth_rate_tbl[i].rate == rate)
+ return nsp_eth_rate_tbl[i].speed;
- return rate_xlate[rate];
+ return 0;
+}
+
+static unsigned int nfp_eth_speed2rate(unsigned int speed)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(nsp_eth_rate_tbl); i++)
+ if (nsp_eth_rate_tbl[i].speed == speed)
+ return nsp_eth_rate_tbl[i].rate;
+
+ return RATE_INVALID;
}
static void nfp_eth_copy_mac_reverse(u8 *dst, const u8 *src)
@@ -110,8 +148,8 @@
}
static void
-nfp_eth_port_translate(const struct eth_table_entry *src, unsigned int index,
- struct nfp_eth_table_port *dst)
+nfp_eth_port_translate(struct nfp_nsp *nsp, const union eth_table_entry *src,
+ unsigned int index, struct nfp_eth_table_port *dst)
{
unsigned int rate;
u64 port, state;
@@ -129,14 +167,60 @@
dst->tx_enabled = FIELD_GET(NSP_ETH_STATE_TX_ENABLED, state);
dst->rx_enabled = FIELD_GET(NSP_ETH_STATE_RX_ENABLED, state);
- rate = nfp_eth_rate(FIELD_GET(NSP_ETH_STATE_RATE, state));
+ rate = nfp_eth_rate2speed(FIELD_GET(NSP_ETH_STATE_RATE, state));
dst->speed = dst->lanes * rate;
+ dst->interface = FIELD_GET(NSP_ETH_STATE_INTERFACE, state);
+ dst->media = FIELD_GET(NSP_ETH_STATE_MEDIA, state);
+
nfp_eth_copy_mac_reverse(dst->mac_addr, src->mac_addr);
- snprintf(dst->label, sizeof(dst->label) - 1, "%llu.%llu",
- FIELD_GET(NSP_ETH_PORT_PHYLABEL, port),
- FIELD_GET(NSP_ETH_PORT_LABEL, port));
+ dst->label_port = FIELD_GET(NSP_ETH_PORT_PHYLABEL, port);
+ dst->label_subport = FIELD_GET(NSP_ETH_PORT_LABEL, port);
+
+ if (nfp_nsp_get_abi_ver_minor(nsp) < 17)
+ return;
+
+ dst->override_changed = FIELD_GET(NSP_ETH_STATE_OVRD_CHNG, state);
+ dst->aneg = FIELD_GET(NSP_ETH_STATE_ANEG, state);
+}
+
+static void
+nfp_eth_mark_split_ports(struct nfp_cpp *cpp, struct nfp_eth_table *table)
+{
+ unsigned int i, j;
+
+ for (i = 0; i < table->count; i++)
+ for (j = 0; j < table->count; j++) {
+ if (i == j)
+ continue;
+ if (table->ports[i].label_port !=
+ table->ports[j].label_port)
+ continue;
+ if (table->ports[i].label_subport ==
+ table->ports[j].label_subport)
+ nfp_warn(cpp,
+ "Port %d subport %d is a duplicate\n",
+ table->ports[i].label_port,
+ table->ports[i].label_subport);
+
+ table->ports[i].is_split = true;
+ break;
+ }
+}
+
+static void
+nfp_eth_calc_port_type(struct nfp_cpp *cpp, struct nfp_eth_table_port *entry)
+{
+ if (entry->interface == NFP_INTERFACE_NONE) {
+ entry->port_type = PORT_NONE;
+ return;
+ }
+
+ if (entry->media == NFP_MEDIA_FIBRE)
+ entry->port_type = PORT_FIBRE;
+ else
+ entry->port_type = PORT_DA;
}
/**
@@ -166,10 +250,9 @@
struct nfp_eth_table *
__nfp_eth_read_ports(struct nfp_cpp *cpp, struct nfp_nsp *nsp)
{
- struct eth_table_entry *entries;
+ union eth_table_entry *entries;
struct nfp_eth_table *table;
- unsigned int cnt;
- int i, j, ret;
+ int i, j, ret, cnt = 0;
entries = kzalloc(NSP_ETH_TABLE_SIZE, GFP_KERNEL);
if (!entries)
@@ -178,34 +261,121 @@
ret = nfp_nsp_read_eth_table(nsp, entries, NSP_ETH_TABLE_SIZE);
if (ret < 0) {
nfp_err(cpp, "reading port table failed %d\n", ret);
- kfree(entries);
- return NULL;
+ goto err;
}
- /* Some versions of flash will give us 0 instead of port count */
- cnt = ret;
- if (!cnt) {
- for (i = 0; i < NSP_ETH_MAX_COUNT; i++)
- if (entries[i].port & NSP_ETH_PORT_LANES_MASK)
- cnt++;
+ for (i = 0; i < NSP_ETH_MAX_COUNT; i++)
+ if (entries[i].port & NSP_ETH_PORT_LANES_MASK)
+ cnt++;
+
+ /* Some versions of flash will give us 0 instead of port count.
+ * For those that give a port count, verify it against the value
+ * calculated above.
+ */
+ if (ret && ret != cnt) {
+ nfp_err(cpp, "table entry count reported (%d) does not match entries present (%d)\n",
+ ret, cnt);
+ goto err;
}
table = kzalloc(sizeof(*table) +
sizeof(struct nfp_eth_table_port) * cnt, GFP_KERNEL);
- if (!table) {
- kfree(entries);
- return NULL;
- }
+ if (!table)
+ goto err;
table->count = cnt;
for (i = 0, j = 0; i < NSP_ETH_MAX_COUNT; i++)
if (entries[i].port & NSP_ETH_PORT_LANES_MASK)
- nfp_eth_port_translate(&entries[i], i,
+ nfp_eth_port_translate(nsp, &entries[i], i,
&table->ports[j++]);
+ nfp_eth_mark_split_ports(cpp, table);
+ for (i = 0; i < table->count; i++)
+ nfp_eth_calc_port_type(cpp, &table->ports[i]);
+
kfree(entries);
return table;
+
+err:
+ kfree(entries);
+ return NULL;
+}
+
+struct nfp_nsp *nfp_eth_config_start(struct nfp_cpp *cpp, unsigned int idx)
+{
+ union eth_table_entry *entries;
+ struct nfp_nsp *nsp;
+ int ret;
+
+ entries = kzalloc(NSP_ETH_TABLE_SIZE, GFP_KERNEL);
+ if (!entries)
+ return ERR_PTR(-ENOMEM);
+
+ nsp = nfp_nsp_open(cpp);
+ if (IS_ERR(nsp)) {
+ kfree(entries);
+ return nsp;
+ }
+
+ ret = nfp_nsp_read_eth_table(nsp, entries, NSP_ETH_TABLE_SIZE);
+ if (ret < 0) {
+ nfp_err(cpp, "reading port table failed %d\n", ret);
+ goto err;
+ }
+
+ if (!(entries[idx].port & NSP_ETH_PORT_LANES_MASK)) {
+ nfp_warn(cpp, "trying to set port state on disabled port %d\n",
+ idx);
+ goto err;
+ }
+
+ nfp_nsp_config_set_state(nsp, entries, idx);
+ return nsp;
+
+err:
+ nfp_nsp_close(nsp);
+ kfree(entries);
+ return ERR_PTR(-EIO);
+}
+
+void nfp_eth_config_cleanup_end(struct nfp_nsp *nsp)
+{
+ union eth_table_entry *entries = nfp_nsp_config_entries(nsp);
+
+ nfp_nsp_config_set_modified(nsp, false);
+ nfp_nsp_config_clear_state(nsp);
+ nfp_nsp_close(nsp);
+ kfree(entries);
+}
+
+/**
+ * nfp_eth_config_commit_end() - perform recorded configuration changes
+ * @nsp: NFP NSP handle returned from nfp_eth_config_start()
+ *
+ * Perform the configuration which was requested with __nfp_eth_set_*()
+ * helpers and recorded in @nsp state. If device was already configured
+ * as requested or no __nfp_eth_set_*() operations were made no NSP command
+ * will be performed.
+ *
+ * Return:
+ * 0 - configuration successful;
+ * 1 - no changes were needed;
+ * -ERRNO - configuration failed.
+ */
+int nfp_eth_config_commit_end(struct nfp_nsp *nsp)
+{
+ union eth_table_entry *entries = nfp_nsp_config_entries(nsp);
+ int ret = 1;
+
+ if (nfp_nsp_config_modified(nsp)) {
+ ret = nfp_nsp_write_eth_table(nsp, entries, NSP_ETH_TABLE_SIZE);
+ ret = ret < 0 ? ret : 0;
+ }
+
+ nfp_eth_config_cleanup_end(nsp);
+
+ return ret;
}
/**
@@ -221,50 +391,158 @@
*/
int nfp_eth_set_mod_enable(struct nfp_cpp *cpp, unsigned int idx, bool enable)
{
- struct eth_table_entry *entries;
+ union eth_table_entry *entries;
struct nfp_nsp *nsp;
u64 reg;
- int ret;
- entries = kzalloc(NSP_ETH_TABLE_SIZE, GFP_KERNEL);
- if (!entries)
- return -ENOMEM;
-
- nsp = nfp_nsp_open(cpp);
- if (IS_ERR(nsp)) {
- kfree(entries);
+ nsp = nfp_eth_config_start(cpp, idx);
+ if (IS_ERR(nsp))
return PTR_ERR(nsp);
- }
- ret = nfp_nsp_read_eth_table(nsp, entries, NSP_ETH_TABLE_SIZE);
- if (ret < 0) {
- nfp_err(cpp, "reading port table failed %d\n", ret);
- goto exit_close_nsp;
- }
-
- if (!(entries[idx].port & NSP_ETH_PORT_LANES_MASK)) {
- nfp_warn(cpp, "trying to set port state on disabled port %d\n",
- idx);
- ret = -EINVAL;
- goto exit_close_nsp;
- }
+ entries = nfp_nsp_config_entries(nsp);
/* Check if we are already in requested state */
reg = le64_to_cpu(entries[idx].state);
- if (enable == FIELD_GET(NSP_ETH_CTRL_ENABLED, reg)) {
- ret = 0;
- goto exit_close_nsp;
+ if (enable != FIELD_GET(NSP_ETH_CTRL_ENABLED, reg)) {
+ reg = le64_to_cpu(entries[idx].control);
+ reg &= ~NSP_ETH_CTRL_ENABLED;
+ reg |= FIELD_PREP(NSP_ETH_CTRL_ENABLED, enable);
+ entries[idx].control = cpu_to_le64(reg);
+
+ nfp_nsp_config_set_modified(nsp, true);
}
- reg = le64_to_cpu(entries[idx].control);
- reg &= ~NSP_ETH_CTRL_ENABLED;
- reg |= FIELD_PREP(NSP_ETH_CTRL_ENABLED, enable);
- entries[idx].control = cpu_to_le64(reg);
+ return nfp_eth_config_commit_end(nsp);
+}
- ret = nfp_nsp_write_eth_table(nsp, entries, NSP_ETH_TABLE_SIZE);
-exit_close_nsp:
- nfp_nsp_close(nsp);
- kfree(entries);
+/**
+ * nfp_eth_set_configured() - set PHY module configured control bit
+ * @cpp: NFP CPP handle
+ * @idx: NFP chip-wide port index
+ * @configed: Desired state
+ *
+ * Set the ifup/ifdown state on the PHY.
+ *
+ * Return: 0 or -ERRNO.
+ */
+int nfp_eth_set_configured(struct nfp_cpp *cpp, unsigned int idx, bool configed)
+{
+ union eth_table_entry *entries;
+ struct nfp_nsp *nsp;
+ u64 reg;
- return ret < 0 ? ret : 0;
+ nsp = nfp_eth_config_start(cpp, idx);
+ if (IS_ERR(nsp))
+ return PTR_ERR(nsp);
+
+ entries = nfp_nsp_config_entries(nsp);
+
+ /* Check if we are already in requested state */
+ reg = le64_to_cpu(entries[idx].state);
+ if (configed != FIELD_GET(NSP_ETH_STATE_CONFIGURED, reg)) {
+ reg = le64_to_cpu(entries[idx].control);
+ reg &= ~NSP_ETH_CTRL_CONFIGURED;
+ reg |= FIELD_PREP(NSP_ETH_CTRL_CONFIGURED, configed);
+ entries[idx].control = cpu_to_le64(reg);
+
+ nfp_nsp_config_set_modified(nsp, true);
+ }
+
+ return nfp_eth_config_commit_end(nsp);
+}
+
+/* Force inline, FIELD_* macroes require masks to be compilation-time known */
+static __always_inline int
+nfp_eth_set_bit_config(struct nfp_nsp *nsp, unsigned int raw_idx,
+ const u64 mask, unsigned int val, const u64 ctrl_bit)
+{
+ union eth_table_entry *entries = nfp_nsp_config_entries(nsp);
+ unsigned int idx = nfp_nsp_config_idx(nsp);
+ u64 reg;
+
+ /* Note: set features were added in ABI 0.14 but the error
+ * codes were initially not populated correctly.
+ */
+ if (nfp_nsp_get_abi_ver_minor(nsp) < 17) {
+ nfp_err(nfp_nsp_cpp(nsp),
+ "set operations not supported, please update flash\n");
+ return -EOPNOTSUPP;
+ }
+
+ /* Check if we are already in requested state */
+ reg = le64_to_cpu(entries[idx].raw[raw_idx]);
+ if (val == FIELD_GET(mask, reg))
+ return 0;
+
+ reg &= ~mask;
+ reg |= FIELD_PREP(mask, val);
+ entries[idx].raw[raw_idx] = cpu_to_le64(reg);
+
+ entries[idx].control |= cpu_to_le64(ctrl_bit);
+
+ nfp_nsp_config_set_modified(nsp, true);
+
+ return 0;
+}
+
+/**
+ * __nfp_eth_set_aneg() - set PHY autonegotiation control bit
+ * @nsp: NFP NSP handle returned from nfp_eth_config_start()
+ * @mode: Desired autonegotiation mode
+ *
+ * Allow/disallow PHY module to advertise/perform autonegotiation.
+ * Will write to hwinfo overrides in the flash (persistent config).
+ *
+ * Return: 0 or -ERRNO.
+ */
+int __nfp_eth_set_aneg(struct nfp_nsp *nsp, enum nfp_eth_aneg mode)
+{
+ return nfp_eth_set_bit_config(nsp, NSP_ETH_RAW_STATE,
+ NSP_ETH_STATE_ANEG, mode,
+ NSP_ETH_CTRL_SET_ANEG);
+}
+
+/**
+ * __nfp_eth_set_speed() - set interface speed/rate
+ * @nsp: NFP NSP handle returned from nfp_eth_config_start()
+ * @speed: Desired speed (per lane)
+ *
+ * Set lane speed. Provided @speed value should be subport speed divided
+ * by number of lanes this subport is spanning (i.e. 10000 for 40G, 25000 for
+ * 50G, etc.)
+ * Will write to hwinfo overrides in the flash (persistent config).
+ *
+ * Return: 0 or -ERRNO.
+ */
+int __nfp_eth_set_speed(struct nfp_nsp *nsp, unsigned int speed)
+{
+ enum nfp_eth_rate rate;
+
+ rate = nfp_eth_speed2rate(speed);
+ if (rate == RATE_INVALID) {
+ nfp_warn(nfp_nsp_cpp(nsp),
+ "could not find matching lane rate for speed %u\n",
+ speed);
+ return -EINVAL;
+ }
+
+ return nfp_eth_set_bit_config(nsp, NSP_ETH_RAW_STATE,
+ NSP_ETH_STATE_RATE, rate,
+ NSP_ETH_CTRL_SET_RATE);
+}
+
+/**
+ * __nfp_eth_set_split() - set interface lane split
+ * @nsp: NFP NSP handle returned from nfp_eth_config_start()
+ * @lanes: Desired lanes per port
+ *
+ * Set number of lanes in the port.
+ * Will write to hwinfo overrides in the flash (persistent config).
+ *
+ * Return: 0 or -ERRNO.
+ */
+int __nfp_eth_set_split(struct nfp_nsp *nsp, unsigned int lanes)
+{
+ return nfp_eth_set_bit_config(nsp, NSP_ETH_RAW_PORT, NSP_ETH_PORT_LANES,
+ lanes, NSP_ETH_CTRL_SET_LANES);
}
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp_eth.h b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp_eth.h
deleted file mode 100644
index edf703d..0000000
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nsp_eth.h
+++ /dev/null
@@ -1,81 +0,0 @@
-/*
- * Copyright (C) 2015-2017 Netronome Systems, Inc.
- *
- * This software is dual licensed under the GNU General License Version 2,
- * June 1991 as shown in the file COPYING in the top-level directory of this
- * source tree or the BSD 2-Clause License provided below. You have the
- * option to license this software under the complete terms of either license.
- *
- * The BSD 2-Clause License:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * 1. Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * 2. Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#ifndef NSP_NSP_ETH_H
-#define NSP_NSP_ETH_H 1
-
-#include <linux/types.h>
-#include <linux/if_ether.h>
-
-/**
- * struct nfp_eth_table - ETH table information
- * @count: number of table entries
- * @ports: table of ports
- *
- * @eth_index: port index according to legacy ethX numbering
- * @index: chip-wide first channel index
- * @nbi: NBI index
- * @base: first channel index (within NBI)
- * @lanes: number of channels
- * @speed: interface speed (in Mbps)
- * @mac_addr: interface MAC address
- * @label: interface id string
- * @enabled: is enabled?
- * @tx_enabled: is TX enabled?
- * @rx_enabled: is RX enabled?
- */
-struct nfp_eth_table {
- unsigned int count;
- struct nfp_eth_table_port {
- unsigned int eth_index;
- unsigned int index;
- unsigned int nbi;
- unsigned int base;
- unsigned int lanes;
- unsigned int speed;
-
- u8 mac_addr[ETH_ALEN];
- char label[8];
-
- bool enabled;
- bool tx_enabled;
- bool rx_enabled;
- } ports[0];
-};
-
-struct nfp_eth_table *nfp_eth_read_ports(struct nfp_cpp *cpp);
-struct nfp_eth_table *
-__nfp_eth_read_ports(struct nfp_cpp *cpp, struct nfp_nsp *nsp);
-int nfp_eth_set_mod_enable(struct nfp_cpp *cpp, unsigned int idx, bool enable);
-
-#endif
diff --git a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_resource.c b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_resource.c
index a2850344..2d15a7c 100644
--- a/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_resource.c
+++ b/drivers/net/ethernet/netronome/nfp/nfpcore/nfp_resource.c
@@ -45,6 +45,13 @@
#include "nfp_cpp.h"
#include "nfp6000/nfp6000.h"
+#define NFP_RESOURCE_TBL_TARGET NFP_CPP_TARGET_MU
+#define NFP_RESOURCE_TBL_BASE 0x8100000000ULL
+
+/* NFP Resource Table self-identifier */
+#define NFP_RESOURCE_TBL_NAME "nfp.res"
+#define NFP_RESOURCE_TBL_KEY 0x00000000 /* Special key for entry 0 */
+
#define NFP_RESOURCE_ENTRY_NAME_SZ 8
/**
@@ -100,9 +107,11 @@
strncpy(name_pad, res->name, sizeof(name_pad));
/* Search for a matching entry */
- key = NFP_RESOURCE_TBL_KEY;
- if (memcmp(name_pad, NFP_RESOURCE_TBL_NAME "\0\0\0\0\0\0\0\0", 8))
- key = crc32_posix(name_pad, sizeof(name_pad));
+ if (!memcmp(name_pad, NFP_RESOURCE_TBL_NAME "\0\0\0\0\0\0\0\0", 8)) {
+ nfp_err(cpp, "Grabbing device lock not supported\n");
+ return -EOPNOTSUPP;
+ }
+ key = crc32_posix(name_pad, sizeof(name_pad));
for (i = 0; i < NFP_RESOURCE_TBL_ENTRIES; i++) {
u64 addr = NFP_RESOURCE_TBL_BASE +
diff --git a/drivers/net/ethernet/nuvoton/w90p910_ether.c b/drivers/net/ethernet/nuvoton/w90p910_ether.c
index 9709c8c..159564d 100644
--- a/drivers/net/ethernet/nuvoton/w90p910_ether.c
+++ b/drivers/net/ethernet/nuvoton/w90p910_ether.c
@@ -152,7 +152,6 @@
struct tran_pdesc *tdesc;
dma_addr_t rdesc_phys;
dma_addr_t tdesc_phys;
- struct net_device_stats stats;
struct platform_device *pdev;
struct resource *res;
struct sk_buff *skb;
@@ -584,15 +583,6 @@
return 0;
}
-static struct net_device_stats *w90p910_ether_stats(struct net_device *dev)
-{
- struct w90p910_ether *ether;
-
- ether = netdev_priv(dev);
-
- return ðer->stats;
-}
-
static int w90p910_send_frame(struct net_device *dev,
unsigned char *data, int length)
{
@@ -671,10 +661,10 @@
ether->finish_tx = 0;
if (txbd->sl & TXDS_TXCP) {
- ether->stats.tx_packets++;
- ether->stats.tx_bytes += txbd->sl & 0xFFFF;
+ dev->stats.tx_packets++;
+ dev->stats.tx_bytes += txbd->sl & 0xFFFF;
} else {
- ether->stats.tx_errors++;
+ dev->stats.tx_errors++;
}
txbd->sl = 0x0;
@@ -730,7 +720,7 @@
data = ether->rdesc->recv_buf[ether->cur_rx];
skb = netdev_alloc_skb(dev, length + 2);
if (!skb) {
- ether->stats.rx_dropped++;
+ dev->stats.rx_dropped++;
return;
}
@@ -738,24 +728,24 @@
skb_put(skb, length);
skb_copy_to_linear_data(skb, data, length);
skb->protocol = eth_type_trans(skb, dev);
- ether->stats.rx_packets++;
- ether->stats.rx_bytes += length;
+ dev->stats.rx_packets++;
+ dev->stats.rx_bytes += length;
netif_rx(skb);
} else {
- ether->stats.rx_errors++;
+ dev->stats.rx_errors++;
if (status & RXDS_RP) {
dev_err(&pdev->dev, "rx runt err\n");
- ether->stats.rx_length_errors++;
+ dev->stats.rx_length_errors++;
} else if (status & RXDS_CRCE) {
dev_err(&pdev->dev, "rx crc err\n");
- ether->stats.rx_crc_errors++;
+ dev->stats.rx_crc_errors++;
} else if (status & RXDS_ALIE) {
dev_err(&pdev->dev, "rx alignment err\n");
- ether->stats.rx_frame_errors++;
+ dev->stats.rx_frame_errors++;
} else if (status & RXDS_PTLE) {
dev_err(&pdev->dev, "rx longer err\n");
- ether->stats.rx_over_errors++;
+ dev->stats.rx_over_errors++;
}
}
@@ -912,7 +902,6 @@
.ndo_open = w90p910_ether_open,
.ndo_stop = w90p910_ether_close,
.ndo_start_xmit = w90p910_ether_start_xmit,
- .ndo_get_stats = w90p910_ether_stats,
.ndo_set_rx_mode = w90p910_ether_set_multicast_list,
.ndo_set_mac_address = w90p910_set_mac_address,
.ndo_do_ioctl = w90p910_ether_ioctl,
diff --git a/drivers/net/ethernet/qlogic/netxen/netxen_nic_init.c b/drivers/net/ethernet/qlogic/netxen/netxen_nic_init.c
index 7b43a3b..3dd9734 100644
--- a/drivers/net/ethernet/qlogic/netxen/netxen_nic_init.c
+++ b/drivers/net/ethernet/qlogic/netxen/netxen_nic_init.c
@@ -1375,13 +1375,8 @@
} while (--retries);
- if (!retries) {
- printk(KERN_ERR "Receive Peg initialization not "
- "complete, state: 0x%x.\n", val);
- return -EIO;
- }
-
- return 0;
+ pr_err("Receive Peg initialization not complete, state: 0x%x.\n", val);
+ return -EIO;
}
int netxen_init_firmware(struct netxen_adapter *adapter)
diff --git a/drivers/net/ethernet/qlogic/qed/qed.h b/drivers/net/ethernet/qlogic/qed/qed.h
index 00c17fa..2ab1aab 100644
--- a/drivers/net/ethernet/qlogic/qed/qed.h
+++ b/drivers/net/ethernet/qlogic/qed/qed.h
@@ -51,7 +51,19 @@
#include "qed_hsi.h"
extern const struct qed_common_ops qed_common_ops_pass;
-#define DRV_MODULE_VERSION "8.10.10.20"
+
+#define QED_MAJOR_VERSION 8
+#define QED_MINOR_VERSION 10
+#define QED_REVISION_VERSION 10
+#define QED_ENGINEERING_VERSION 21
+
+#define QED_VERSION \
+ ((QED_MAJOR_VERSION << 24) | (QED_MINOR_VERSION << 16) | \
+ (QED_REVISION_VERSION << 8) | QED_ENGINEERING_VERSION)
+
+#define STORM_FW_VERSION \
+ ((FW_MAJOR_VERSION << 24) | (FW_MINOR_VERSION << 16) | \
+ (FW_REVISION_VERSION << 8) | FW_ENGINEERING_VERSION)
#define MAX_HWFNS_PER_DEVICE (4)
#define NAME_SIZE 16
@@ -59,9 +71,8 @@
#define QED_WFQ_UNIT 100
-#define ISCSI_BDQ_ID(_port_id) (_port_id)
-#define FCOE_BDQ_ID(_port_id) ((_port_id) + 2)
#define QED_WID_SIZE (1024)
+#define QED_MIN_WIDS (4)
#define QED_PF_DEMS_SIZE (4)
/* cau states */
@@ -76,6 +87,15 @@
enum qed_mcp_protocol_type;
/* helpers */
+#define QED_MFW_GET_FIELD(name, field) \
+ (((name) & (field ## _MASK)) >> (field ## _SHIFT))
+
+#define QED_MFW_SET_FIELD(name, field, value) \
+ do { \
+ (name) &= ~((field ## _MASK) << (field ## _SHIFT)); \
+ (name) |= (((value) << (field ## _SHIFT)) & (field ## _MASK));\
+ } while (0)
+
static inline u32 qed_db_addr(u32 cid, u32 DEMS)
{
u32 db_addr = FIELD_VALUE(DB_LEGACY_ADDR_DEMS, DEMS) |
@@ -130,9 +150,35 @@
QED_TUNN_CLSS_MAC_VNI,
QED_TUNN_CLSS_INNER_MAC_VLAN,
QED_TUNN_CLSS_INNER_MAC_VNI,
+ QED_TUNN_CLSS_MAC_VLAN_DUAL_STAGE,
MAX_QED_TUNN_CLSS,
};
+struct qed_tunn_update_type {
+ bool b_update_mode;
+ bool b_mode_enabled;
+ enum qed_tunn_clss tun_cls;
+};
+
+struct qed_tunn_update_udp_port {
+ bool b_update_port;
+ u16 port;
+};
+
+struct qed_tunnel_info {
+ struct qed_tunn_update_type vxlan;
+ struct qed_tunn_update_type l2_geneve;
+ struct qed_tunn_update_type ip_geneve;
+ struct qed_tunn_update_type l2_gre;
+ struct qed_tunn_update_type ip_gre;
+
+ struct qed_tunn_update_udp_port vxlan_port;
+ struct qed_tunn_update_udp_port geneve_port;
+
+ bool b_update_rx_cls;
+ bool b_update_tx_cls;
+};
+
struct qed_tunn_start_params {
unsigned long tunn_mode;
u16 vxlan_udp_port;
@@ -198,6 +244,7 @@
QED_LL2_QUEUE,
QED_CMDQS_CQS,
QED_RDMA_STATS_QUEUE,
+ QED_BDQ,
QED_MAX_RESC,
};
@@ -205,8 +252,9 @@
QED_PF_L2_QUE,
QED_VF,
QED_RDMA_CNQ,
- QED_VF_L2_QUE,
+ QED_ISCSI_CQ,
QED_FCOE_CQ,
+ QED_VF_L2_QUE,
QED_MAX_FEATURES,
};
@@ -219,7 +267,9 @@
QED_PORT_MODE_DE_4X20G,
QED_PORT_MODE_DE_1X40G,
QED_PORT_MODE_DE_2X25G,
- QED_PORT_MODE_DE_1X25G
+ QED_PORT_MODE_DE_1X25G,
+ QED_PORT_MODE_DE_4X25G,
+ QED_PORT_MODE_DE_2X10G,
};
enum qed_dev_cap {
@@ -249,9 +299,14 @@
RESC_NUM(_p_hwfn, resc))
#define FEAT_NUM(_p_hwfn, resc) ((_p_hwfn)->hw_info.feat_num[resc])
- u8 num_tc;
+ /* Amount of traffic classes HW supports */
+ u8 num_hw_tc;
+
+ /* Amount of TCs which should be active according to DCBx or upper
+ * layer driver configuration.
+ */
+ u8 num_active_tc;
u8 offload_tc;
- u8 non_offload_tc;
u32 concrete_fid;
u16 opaque_fid;
@@ -314,15 +369,19 @@
struct init_qm_port_params *qm_port_params;
u16 start_pq;
u8 start_vport;
- u8 pure_lb_pq;
- u8 offload_pq;
- u8 pure_ack_pq;
- u8 ooo_pq;
- u8 vf_queues_offset;
+ u16 pure_lb_pq;
+ u16 offload_pq;
+ u16 low_latency_pq;
+ u16 pure_ack_pq;
+ u16 ooo_pq;
+ u16 first_vf_pq;
+ u16 first_mcos_pq;
+ u16 first_rl_pq;
u16 num_pqs;
u16 num_vf_pqs;
u8 num_vports;
u8 max_phys_tcs_per_port;
+ u8 ooo_tc;
bool pf_rl_en;
bool pf_wfq_en;
bool vport_rl_en;
@@ -353,6 +412,12 @@
u32 init_ops_size;
};
+#define DRV_MODULE_VERSION \
+ __stringify(QED_MAJOR_VERSION) "." \
+ __stringify(QED_MINOR_VERSION) "." \
+ __stringify(QED_REVISION_VERSION) "." \
+ __stringify(QED_ENGINEERING_VERSION)
+
struct qed_simd_fp_handler {
void *token;
void (*func)(void *);
@@ -364,7 +429,8 @@
#define IS_LEAD_HWFN(edev) (!((edev)->my_id))
u8 rel_pf_id; /* Relative to engine*/
u8 abs_pf_id;
-#define QED_PATH_ID(_p_hwfn) ((_p_hwfn)->abs_pf_id & 1)
+#define QED_PATH_ID(_p_hwfn) \
+ (QED_IS_K2((_p_hwfn)->cdev) ? 0 : ((_p_hwfn)->abs_pf_id & 1))
u8 port_id;
bool b_active;
@@ -409,6 +475,11 @@
struct qed_ptt *p_main_ptt;
struct qed_ptt *p_dpc_ptt;
+ /* PTP will be used only by the leading function.
+ * Usage of all PTP-apis should be synchronized as result.
+ */
+ struct qed_ptt *p_ptp_ptt;
+
struct qed_sb_sp_info *p_sp_sb;
struct qed_sb_attn_info *p_sb_attn;
@@ -455,6 +526,7 @@
struct dbg_tools_data dbg_info;
/* PWM region specific data */
+ u16 wid_count;
u32 dpi_size;
u32 dpi_count;
@@ -465,8 +537,8 @@
u8 dcbx_no_edpm;
u8 db_bar_no_edpm;
- /* p_ptp_ptt is valid for leading HWFN only */
- struct qed_ptt *p_ptp_ptt;
+ struct qed_ptt *p_arfs_ptt;
+
struct qed_simd_fp_handler simd_proto_handler[64];
#ifdef CONFIG_QED_SRIOV
@@ -523,9 +595,7 @@
u8 dp_level;
char name[NAME_SIZE];
- u8 type;
-#define QED_DEV_TYPE_BB (0 << 0)
-#define QED_DEV_TYPE_AH BIT(0)
+ enum qed_dev_type type;
/* Translate type/revision combo into the proper conditions */
#define QED_IS_BB(dev) ((dev)->type == QED_DEV_TYPE_BB)
#define QED_IS_BB_A0(dev) (QED_IS_BB(dev) && \
@@ -540,6 +610,9 @@
u16 vendor_id;
u16 device_id;
+#define QED_DEV_ID_MASK 0xff00
+#define QED_DEV_ID_MASK_BB 0x1600
+#define QED_DEV_ID_MASK_AH 0x8000
u16 chip_num;
#define CHIP_NUM_MASK 0xffff
@@ -606,9 +679,7 @@
/* SRIOV */
struct qed_hw_sriov_info *p_iov_info;
#define IS_QED_SRIOV(cdev) (!!(cdev)->p_iov_info)
-
- unsigned long tunn_mode;
-
+ struct qed_tunnel_info tunnel;
bool b_is_vf;
u32 drv_type;
struct qed_eth_stats *reset_stats;
@@ -652,12 +723,19 @@
u32 rdma_max_sge;
u32 rdma_max_inline;
u32 rdma_max_srq_sge;
+ u16 tunn_feature_mask;
};
-#define NUM_OF_VFS(dev) MAX_NUM_VFS_BB
-#define NUM_OF_L2_QUEUES(dev) MAX_NUM_L2_QUEUES_BB
-#define NUM_OF_SBS(dev) MAX_SB_PER_PATH_BB
-#define NUM_OF_ENG_PFS(dev) MAX_NUM_PFS_BB
+#define NUM_OF_VFS(dev) (QED_IS_BB(dev) ? MAX_NUM_VFS_BB \
+ : MAX_NUM_VFS_K2)
+#define NUM_OF_L2_QUEUES(dev) (QED_IS_BB(dev) ? MAX_NUM_L2_QUEUES_BB \
+ : MAX_NUM_L2_QUEUES_K2)
+#define NUM_OF_PORTS(dev) (QED_IS_BB(dev) ? MAX_NUM_PORTS_BB \
+ : MAX_NUM_PORTS_K2)
+#define NUM_OF_SBS(dev) (QED_IS_BB(dev) ? MAX_SB_PER_PATH_BB \
+ : MAX_SB_PER_PATH_K2)
+#define NUM_OF_ENG_PFS(dev) (QED_IS_BB(dev) ? MAX_NUM_PFS_BB \
+ : MAX_NUM_PFS_K2)
/**
* @brief qed_concrete_to_sw_fid - get the sw function id from
@@ -693,6 +771,26 @@
u32 min_pf_rate);
void qed_clean_wfq_db(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt);
+int qed_device_num_engines(struct qed_dev *cdev);
+int qed_device_get_port_id(struct qed_dev *cdev);
+
+#define QED_LEADING_HWFN(dev) (&dev->hwfns[0])
+
+/* Flags for indication of required queues */
+#define PQ_FLAGS_RLS (BIT(0))
+#define PQ_FLAGS_MCOS (BIT(1))
+#define PQ_FLAGS_LB (BIT(2))
+#define PQ_FLAGS_OOO (BIT(3))
+#define PQ_FLAGS_ACK (BIT(4))
+#define PQ_FLAGS_OFLD (BIT(5))
+#define PQ_FLAGS_VFS (BIT(6))
+#define PQ_FLAGS_LLT (BIT(7))
+
+/* physical queue index for cm context intialization */
+u16 qed_get_cm_pq_idx(struct qed_hwfn *p_hwfn, u32 pq_flags);
+u16 qed_get_cm_pq_idx_mcos(struct qed_hwfn *p_hwfn, u8 tc);
+u16 qed_get_cm_pq_idx_vf(struct qed_hwfn *p_hwfn, u16 vf);
+
#define QED_LEADING_HWFN(dev) (&dev->hwfns[0])
/* Other Linux specific common definitions */
@@ -721,5 +819,6 @@
enum qed_mcp_protocol_type type,
union qed_mcp_protocol_stats *stats);
int qed_slowpath_irq_req(struct qed_hwfn *hwfn);
+void qed_slowpath_irq_sync(struct qed_hwfn *p_hwfn);
#endif /* _QED_H */
diff --git a/drivers/net/ethernet/qlogic/qed/qed_cxt.c b/drivers/net/ethernet/qlogic/qed/qed_cxt.c
index 7e3a6fe..b3aaa98 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_cxt.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_cxt.c
@@ -71,8 +71,7 @@
#define TM_ALIGN BIT(TM_SHIFT)
#define TM_ELEM_SIZE 4
-/* For RoCE we configure to 64K to cover for RoCE max tasks 256K purpose. */
-#define ILT_DEFAULT_HW_P_SIZE (IS_ENABLED(CONFIG_QED_RDMA) ? 4 : 3)
+#define ILT_DEFAULT_HW_P_SIZE 4
#define ILT_PAGE_IN_BYTES(hw_p_size) (1U << ((hw_p_size) + 12))
#define ILT_CFG_REG(cli, reg) PSWRQ2_REG_ ## cli ## _ ## reg ## _RT_OFFSET
@@ -220,9 +219,6 @@
*/
u32 vf_count;
- /* total number of SRQ's for this hwfn */
- u32 srq_count;
-
/* Acquired CIDs */
struct qed_cid_acquired_map acquired[MAX_CONN_TYPES];
@@ -238,12 +234,17 @@
u32 t2_num_pages;
u64 first_free;
u64 last_free;
+
+ /* total number of SRQ's for this hwfn */
+ u32 srq_count;
+
+ /* Maximal number of L2 steering filters */
+ u32 arfs_count;
};
static bool src_proto(enum protocol_type type)
{
return type == PROTOCOLID_ISCSI ||
- type == PROTOCOLID_FCOE ||
- type == PROTOCOLID_ROCE;
+ type == PROTOCOLID_FCOE;
}
static bool tm_cid_proto(enum protocol_type type)
@@ -293,6 +294,9 @@
iids->pf_cids += p_mngr->conn_cfg[i].cid_count;
iids->per_vf_cids += p_mngr->conn_cfg[i].cids_per_vf;
}
+
+ /* Add L2 filtering filters in addition */
+ iids->pf_cids += p_mngr->arfs_count;
}
/* counts the iids for the Timers block configuration */
@@ -304,16 +308,34 @@
u32 per_vf_tids;
};
-static void qed_cxt_tm_iids(struct qed_cxt_mngr *p_mngr,
+static void qed_cxt_tm_iids(struct qed_hwfn *p_hwfn,
+ struct qed_cxt_mngr *p_mngr,
struct qed_tm_iids *iids)
{
- u32 i, j;
+ bool tm_vf_required = false;
+ bool tm_required = false;
+ int i, j;
- for (i = 0; i < MAX_CONN_TYPES; i++) {
+ /* Timers is a special case -> we don't count how many cids require
+ * timers but what's the max cid that will be used by the timer block.
+ * therefore we traverse in reverse order, and once we hit a protocol
+ * that requires the timers memory, we'll sum all the protocols up
+ * to that one.
+ */
+ for (i = MAX_CONN_TYPES - 1; i >= 0; i--) {
struct qed_conn_type_cfg *p_cfg = &p_mngr->conn_cfg[i];
- if (tm_cid_proto(i)) {
+ if (tm_cid_proto(i) || tm_required) {
+ if (p_cfg->cid_count)
+ tm_required = true;
+
iids->pf_cids += p_cfg->cid_count;
+ }
+
+ if (tm_cid_proto(i) || tm_vf_required) {
+ if (p_cfg->cids_per_vf)
+ tm_vf_required = true;
+
iids->per_vf_cids += p_cfg->cids_per_vf;
}
@@ -527,7 +549,22 @@
return lines_to_skip;
}
-int qed_cxt_cfg_ilt_compute(struct qed_hwfn *p_hwfn)
+static struct qed_ilt_client_cfg *qed_cxt_set_cli(struct qed_ilt_client_cfg
+ *p_cli)
+{
+ p_cli->active = false;
+ p_cli->first.val = 0;
+ p_cli->last.val = 0;
+ return p_cli;
+}
+
+static struct qed_ilt_cli_blk *qed_cxt_set_blk(struct qed_ilt_cli_blk *p_blk)
+{
+ p_blk->total_size = 0;
+ return p_blk;
+}
+
+int qed_cxt_cfg_ilt_compute(struct qed_hwfn *p_hwfn, u32 *line_count)
{
struct qed_cxt_mngr *p_mngr = p_hwfn->p_cxt_mngr;
u32 curr_line, total, i, task_size, line;
@@ -551,7 +588,8 @@
p_hwfn->my_id, p_hwfn->p_cxt_mngr->pf_start_line);
/* CDUC */
- p_cli = &p_mngr->clients[ILT_CLI_CDUC];
+ p_cli = qed_cxt_set_cli(&p_mngr->clients[ILT_CLI_CDUC]);
+
curr_line = p_mngr->pf_start_line;
/* CDUC PF */
@@ -560,7 +598,7 @@
/* get the counters for the CDUC and QM clients */
qed_cxt_cdu_iids(p_mngr, &cdu_iids);
- p_blk = &p_cli->pf_blks[CDUC_BLK];
+ p_blk = qed_cxt_set_blk(&p_cli->pf_blks[CDUC_BLK]);
total = cdu_iids.pf_cids * CONN_CXT_SIZE(p_hwfn);
@@ -574,7 +612,7 @@
ILT_CLI_CDUC);
/* CDUC VF */
- p_blk = &p_cli->vf_blks[CDUC_BLK];
+ p_blk = qed_cxt_set_blk(&p_cli->vf_blks[CDUC_BLK]);
total = cdu_iids.per_vf_cids * CONN_CXT_SIZE(p_hwfn);
qed_ilt_cli_blk_fill(p_cli, p_blk, curr_line,
@@ -588,7 +626,7 @@
ILT_CLI_CDUC);
/* CDUT PF */
- p_cli = &p_mngr->clients[ILT_CLI_CDUT];
+ p_cli = qed_cxt_set_cli(&p_mngr->clients[ILT_CLI_CDUT]);
p_cli->first.val = curr_line;
/* first the 'working' task memory */
@@ -597,7 +635,7 @@
if (!p_seg || p_seg->count == 0)
continue;
- p_blk = &p_cli->pf_blks[CDUT_SEG_BLK(i)];
+ p_blk = qed_cxt_set_blk(&p_cli->pf_blks[CDUT_SEG_BLK(i)]);
total = p_seg->count * p_mngr->task_type_size[p_seg->type];
qed_ilt_cli_blk_fill(p_cli, p_blk, curr_line, total,
p_mngr->task_type_size[p_seg->type]);
@@ -612,7 +650,8 @@
if (!p_seg || p_seg->count == 0)
continue;
- p_blk = &p_cli->pf_blks[CDUT_FL_SEG_BLK(i, PF)];
+ p_blk =
+ qed_cxt_set_blk(&p_cli->pf_blks[CDUT_FL_SEG_BLK(i, PF)]);
if (!p_seg->has_fl_mem) {
/* The segment is active (total size pf 'working'
@@ -657,7 +696,7 @@
/* 'working' memory */
total = p_seg->count * p_mngr->task_type_size[p_seg->type];
- p_blk = &p_cli->vf_blks[CDUT_SEG_BLK(0)];
+ p_blk = qed_cxt_set_blk(&p_cli->vf_blks[CDUT_SEG_BLK(0)]);
qed_ilt_cli_blk_fill(p_cli, p_blk,
curr_line, total,
p_mngr->task_type_size[p_seg->type]);
@@ -666,7 +705,8 @@
ILT_CLI_CDUT);
/* 'init' memory */
- p_blk = &p_cli->vf_blks[CDUT_FL_SEG_BLK(0, VF)];
+ p_blk =
+ qed_cxt_set_blk(&p_cli->vf_blks[CDUT_FL_SEG_BLK(0, VF)]);
if (!p_seg->has_fl_mem) {
/* see comment above */
line = p_cli->vf_blks[CDUT_SEG_BLK(0)].start_line;
@@ -694,8 +734,8 @@
}
/* QM */
- p_cli = &p_mngr->clients[ILT_CLI_QM];
- p_blk = &p_cli->pf_blks[0];
+ p_cli = qed_cxt_set_cli(&p_mngr->clients[ILT_CLI_QM]);
+ p_blk = qed_cxt_set_blk(&p_cli->pf_blks[0]);
qed_cxt_qm_iids(p_hwfn, &qm_iids);
total = qed_qm_pf_mem_size(p_hwfn->rel_pf_id, qm_iids.cids,
@@ -719,7 +759,7 @@
p_cli->pf_total_lines = curr_line - p_blk->start_line;
/* SRC */
- p_cli = &p_mngr->clients[ILT_CLI_SRC];
+ p_cli = qed_cxt_set_cli(&p_mngr->clients[ILT_CLI_SRC]);
qed_cxt_src_iids(p_mngr, &src_iids);
/* Both the PF and VFs searcher connections are stored in the per PF
@@ -733,7 +773,7 @@
total = roundup_pow_of_two(local_max);
- p_blk = &p_cli->pf_blks[0];
+ p_blk = qed_cxt_set_blk(&p_cli->pf_blks[0]);
qed_ilt_cli_blk_fill(p_cli, p_blk, curr_line,
total * sizeof(struct src_ent),
sizeof(struct src_ent));
@@ -744,11 +784,11 @@
}
/* TM PF */
- p_cli = &p_mngr->clients[ILT_CLI_TM];
- qed_cxt_tm_iids(p_mngr, &tm_iids);
+ p_cli = qed_cxt_set_cli(&p_mngr->clients[ILT_CLI_TM]);
+ qed_cxt_tm_iids(p_hwfn, p_mngr, &tm_iids);
total = tm_iids.pf_cids + tm_iids.pf_tids_total;
if (total) {
- p_blk = &p_cli->pf_blks[0];
+ p_blk = qed_cxt_set_blk(&p_cli->pf_blks[0]);
qed_ilt_cli_blk_fill(p_cli, p_blk, curr_line,
total * TM_ELEM_SIZE, TM_ELEM_SIZE);
@@ -760,14 +800,14 @@
/* TM VF */
total = tm_iids.per_vf_cids + tm_iids.per_vf_tids;
if (total) {
- p_blk = &p_cli->vf_blks[0];
+ p_blk = qed_cxt_set_blk(&p_cli->vf_blks[0]);
qed_ilt_cli_blk_fill(p_cli, p_blk, curr_line,
total * TM_ELEM_SIZE, TM_ELEM_SIZE);
qed_ilt_cli_adv_line(p_hwfn, p_cli, p_blk, &curr_line,
ILT_CLI_TM);
- p_cli->pf_total_lines = curr_line - p_blk->start_line;
+ p_cli->vf_total_lines = curr_line - p_blk->start_line;
for (i = 1; i < p_mngr->vf_count; i++)
qed_ilt_cli_adv_line(p_hwfn, p_cli, p_blk, &curr_line,
ILT_CLI_TM);
@@ -777,8 +817,8 @@
total = qed_cxt_get_srq_count(p_hwfn);
if (total) {
- p_cli = &p_mngr->clients[ILT_CLI_TSDM];
- p_blk = &p_cli->pf_blks[SRQ_BLK];
+ p_cli = qed_cxt_set_cli(&p_mngr->clients[ILT_CLI_TSDM]);
+ p_blk = qed_cxt_set_blk(&p_cli->pf_blks[SRQ_BLK]);
qed_ilt_cli_blk_fill(p_cli, p_blk, curr_line,
total * SRQ_CXT_SIZE, SRQ_CXT_SIZE);
@@ -787,13 +827,50 @@
p_cli->pf_total_lines = curr_line - p_blk->start_line;
}
+ *line_count = curr_line - p_hwfn->p_cxt_mngr->pf_start_line;
+
if (curr_line - p_hwfn->p_cxt_mngr->pf_start_line >
- RESC_NUM(p_hwfn, QED_ILT)) {
- DP_ERR(p_hwfn, "too many ilt lines...#lines=%d\n",
- curr_line - p_hwfn->p_cxt_mngr->pf_start_line);
+ RESC_NUM(p_hwfn, QED_ILT))
return -EINVAL;
+
+ return 0;
+}
+
+u32 qed_cxt_cfg_ilt_compute_excess(struct qed_hwfn *p_hwfn, u32 used_lines)
+{
+ struct qed_ilt_client_cfg *p_cli;
+ u32 excess_lines, available_lines;
+ struct qed_cxt_mngr *p_mngr;
+ u32 ilt_page_size, elem_size;
+ struct qed_tid_seg *p_seg;
+ int i;
+
+ available_lines = RESC_NUM(p_hwfn, QED_ILT);
+ excess_lines = used_lines - available_lines;
+
+ if (!excess_lines)
+ return 0;
+
+ if (p_hwfn->hw_info.personality != QED_PCI_ETH_ROCE)
+ return 0;
+
+ p_mngr = p_hwfn->p_cxt_mngr;
+ p_cli = &p_mngr->clients[ILT_CLI_CDUT];
+ ilt_page_size = ILT_PAGE_IN_BYTES(p_cli->p_size.val);
+
+ for (i = 0; i < NUM_TASK_PF_SEGMENTS; i++) {
+ p_seg = qed_cxt_tid_seg_info(p_hwfn, i);
+ if (!p_seg || p_seg->count == 0)
+ continue;
+
+ elem_size = p_mngr->task_type_size[p_seg->type];
+ if (!elem_size)
+ continue;
+
+ return (ilt_page_size / elem_size) * excess_lines;
}
+ DP_NOTICE(p_hwfn, "failed computing excess ILT lines\n");
return 0;
}
@@ -1127,7 +1204,7 @@
clients[ILT_CLI_TSDM].first.reg = ILT_CFG_REG(TSDM, FIRST_ILT);
clients[ILT_CLI_TSDM].last.reg = ILT_CFG_REG(TSDM, LAST_ILT);
clients[ILT_CLI_TSDM].p_size.reg = ILT_CFG_REG(TSDM, P_SIZE);
- /* default ILT page size for all clients is 32K */
+ /* default ILT page size for all clients is 64K */
for (i = 0; i < ILT_CLI_MAX; i++)
p_mngr->clients[i].p_size.val = ILT_DEFAULT_HW_P_SIZE;
@@ -1367,7 +1444,7 @@
}
}
-void qed_qm_init_pf(struct qed_hwfn *p_hwfn)
+void qed_qm_init_pf(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
{
struct qed_qm_pf_rt_init_params params;
struct qed_qm_info *qm_info = &p_hwfn->qm_info;
@@ -1393,22 +1470,15 @@
params.pq_params = qm_info->qm_pq_params;
params.vport_params = qm_info->qm_vport_params;
- qed_qm_pf_rt_init(p_hwfn, p_hwfn->p_main_ptt, ¶ms);
+ qed_qm_pf_rt_init(p_hwfn, p_ptt, ¶ms);
}
/* CM PF */
-static int qed_cm_init_pf(struct qed_hwfn *p_hwfn)
+void qed_cm_init_pf(struct qed_hwfn *p_hwfn)
{
- union qed_qm_pq_params pq_params;
- u16 pq;
-
/* XCM pure-LB queue */
- memset(&pq_params, 0, sizeof(pq_params));
- pq_params.core.tc = LB_TC;
- pq = qed_get_qm_pq(p_hwfn, PROTOCOLID_CORE, &pq_params);
- STORE_RT_REG(p_hwfn, XCM_REG_CON_PHY_Q3_RT_OFFSET, pq);
-
- return 0;
+ STORE_RT_REG(p_hwfn, XCM_REG_CON_PHY_Q3_RT_OFFSET,
+ qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_LB));
}
/* DQ PF */
@@ -1640,7 +1710,7 @@
u8 i;
memset(&tm_iids, 0, sizeof(tm_iids));
- qed_cxt_tm_iids(p_mngr, &tm_iids);
+ qed_cxt_tm_iids(p_hwfn, p_mngr, &tm_iids);
/* @@@TBD No pre-scan for now */
@@ -1758,9 +1828,9 @@
qed_prs_init_common(p_hwfn);
}
-void qed_cxt_hw_init_pf(struct qed_hwfn *p_hwfn)
+void qed_cxt_hw_init_pf(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
{
- qed_qm_init_pf(p_hwfn);
+ qed_qm_init_pf(p_hwfn, p_ptt);
qed_cm_init_pf(p_hwfn);
qed_dq_init_pf(p_hwfn);
qed_cdu_init_pf(p_hwfn);
@@ -1884,13 +1954,12 @@
}
static void qed_rdma_set_pf_params(struct qed_hwfn *p_hwfn,
- struct qed_rdma_pf_params *p_params)
+ struct qed_rdma_pf_params *p_params,
+ u32 num_tasks)
{
- u32 num_cons, num_tasks, num_qps, num_mrs, num_srqs;
+ u32 num_cons, num_qps, num_srqs;
enum protocol_type proto;
- num_mrs = min_t(u32, RDMA_MAX_TIDS, p_params->num_mrs);
- num_tasks = num_mrs; /* each mr uses a single task id */
num_srqs = min_t(u32, 32 * 1024, p_params->num_srqs);
switch (p_hwfn->hw_info.personality) {
@@ -1919,7 +1988,7 @@
}
}
-int qed_cxt_set_pf_params(struct qed_hwfn *p_hwfn)
+int qed_cxt_set_pf_params(struct qed_hwfn *p_hwfn, u32 rdma_tasks)
{
/* Set the number of required CORE connections */
u32 core_cids = 1; /* SPQ */
@@ -1931,9 +2000,10 @@
switch (p_hwfn->hw_info.personality) {
case QED_PCI_ETH_ROCE:
{
- qed_rdma_set_pf_params(p_hwfn,
- &p_hwfn->
- pf_params.rdma_pf_params);
+ qed_rdma_set_pf_params(p_hwfn,
+ &p_hwfn->
+ pf_params.rdma_pf_params,
+ rdma_tasks);
/* no need for break since RoCE coexist with Ethernet */
}
case QED_PCI_ETH:
@@ -1943,6 +2013,7 @@
qed_cxt_set_proto_cid_count(p_hwfn, PROTOCOLID_ETH,
p_params->num_cons, 1);
+ p_hwfn->p_cxt_mngr->arfs_count = p_params->num_arfs_filters;
break;
}
case QED_PCI_FCOE:
diff --git a/drivers/net/ethernet/qlogic/qed/qed_cxt.h b/drivers/net/ethernet/qlogic/qed/qed_cxt.h
index 8b01032..53ad532 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_cxt.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_cxt.h
@@ -105,19 +105,28 @@
* @brief qed_cxt_set_pf_params - Set the PF params for cxt init
*
* @param p_hwfn
- *
+ * @param rdma_tasks - requested maximum
* @return int
*/
-int qed_cxt_set_pf_params(struct qed_hwfn *p_hwfn);
+int qed_cxt_set_pf_params(struct qed_hwfn *p_hwfn, u32 rdma_tasks);
/**
* @brief qed_cxt_cfg_ilt_compute - compute ILT init parameters
*
* @param p_hwfn
+ * @param last_line
*
* @return int
*/
-int qed_cxt_cfg_ilt_compute(struct qed_hwfn *p_hwfn);
+int qed_cxt_cfg_ilt_compute(struct qed_hwfn *p_hwfn, u32 *last_line);
+
+/**
+ * @brief qed_cxt_cfg_ilt_compute_excess - how many lines can be decreased
+ *
+ * @param p_hwfn
+ * @param used_lines
+ */
+u32 qed_cxt_cfg_ilt_compute_excess(struct qed_hwfn *p_hwfn, u32 used_lines);
/**
* @brief qed_cxt_mngr_alloc - Allocate and init the context manager struct
@@ -163,19 +172,18 @@
/**
* @brief qed_cxt_hw_init_pf - Initailze ILT and DQ, PF phase, per path.
*
- *
- *
* @param p_hwfn
+ * @param p_ptt
*/
-void qed_cxt_hw_init_pf(struct qed_hwfn *p_hwfn);
+void qed_cxt_hw_init_pf(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt);
/**
* @brief qed_qm_init_pf - Initailze the QM PF phase, per path
*
* @param p_hwfn
+ * @param p_ptt
*/
-
-void qed_qm_init_pf(struct qed_hwfn *p_hwfn);
+void qed_qm_init_pf(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt);
/**
* @brief Reconfigures QM pf on the fly
diff --git a/drivers/net/ethernet/qlogic/qed/qed_dcbx.c b/drivers/net/ethernet/qlogic/qed/qed_dcbx.c
index a6e2bbe..d883ad5 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_dcbx.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_dcbx.c
@@ -64,11 +64,11 @@
((u32)(prio_tc_tbl >> ((7 - prio) * 4)) & 0x7)
static const struct qed_dcbx_app_metadata qed_dcbx_app_update[] = {
- {DCBX_PROTOCOL_ISCSI, "ISCSI", QED_PCI_DEFAULT},
- {DCBX_PROTOCOL_FCOE, "FCOE", QED_PCI_DEFAULT},
- {DCBX_PROTOCOL_ROCE, "ROCE", QED_PCI_DEFAULT},
- {DCBX_PROTOCOL_ROCE_V2, "ROCE_V2", QED_PCI_DEFAULT},
- {DCBX_PROTOCOL_ETH, "ETH", QED_PCI_ETH}
+ {DCBX_PROTOCOL_ISCSI, "ISCSI", QED_PCI_ISCSI},
+ {DCBX_PROTOCOL_FCOE, "FCOE", QED_PCI_FCOE},
+ {DCBX_PROTOCOL_ROCE, "ROCE", QED_PCI_ETH_ROCE},
+ {DCBX_PROTOCOL_ROCE_V2, "ROCE_V2", QED_PCI_ETH_ROCE},
+ {DCBX_PROTOCOL_ETH, "ETH", QED_PCI_ETH},
};
static bool qed_dcbx_app_ethtype(u32 app_info_bitmap)
@@ -183,7 +183,7 @@
"%s info: update %d, enable %d, prio %d, tc %d, num_tc %d\n",
qed_dcbx_app_update[i].name, p_data->arr[id].update,
p_data->arr[id].enable, p_data->arr[id].priority,
- p_data->arr[id].tc, p_hwfn->hw_info.num_tc);
+ p_data->arr[id].tc, p_hwfn->hw_info.num_active_tc);
}
}
@@ -204,12 +204,8 @@
p_data->arr[type].tc = tc;
/* QM reconf data */
- if (p_info->personality == personality) {
- if (personality == QED_PCI_ETH)
- p_info->non_offload_tc = tc;
- else
- p_info->offload_tc = tc;
- }
+ if (p_info->personality == personality)
+ p_info->offload_tc = tc;
}
/* Update app protocol data and hw_info fields with the TLV info */
@@ -275,8 +271,8 @@
struct dcbx_app_priority_entry *p_tbl,
u32 pri_tc_tbl, int count, u8 dcbx_version)
{
- u8 tc, priority_map;
enum dcbx_protocol_type type;
+ u8 tc, priority_map;
bool enable, ieee;
u16 protocol_id;
int priority;
@@ -376,7 +372,9 @@
if (rc)
return rc;
- p_info->num_tc = QED_MFW_GET_FIELD(p_ets->flags, DCBX_ETS_MAX_TCS);
+ p_info->num_active_tc = QED_MFW_GET_FIELD(p_ets->flags,
+ DCBX_ETS_MAX_TCS);
+ p_hwfn->qm_info.ooo_tc = QED_MFW_GET_FIELD(p_ets->flags, DCBX_OOO_TC);
data.pf_id = p_hwfn->rel_pf_id;
data.dcbx_enabled = !!dcbx_version;
@@ -558,8 +556,9 @@
p_params->pfc.prio[7] = !!(pfc_map & DCBX_PFC_PRI_EN_BITMAP_PRI_7);
DP_VERBOSE(p_hwfn, QED_MSG_DCB,
- "PFC params: willing %d, pfc_bitmap %d\n",
- p_params->pfc.willing, pfc_map);
+ "PFC params: willing %d, pfc_bitmap %u max_tc = %u enabled = %d\n",
+ p_params->pfc.willing, pfc_map, p_params->pfc.max_tc,
+ p_params->pfc.enabled);
}
static void
@@ -578,10 +577,10 @@
p_params->max_ets_tc = QED_MFW_GET_FIELD(p_ets->flags,
DCBX_ETS_MAX_TCS);
DP_VERBOSE(p_hwfn, QED_MSG_DCB,
- "ETS params: willing %d, ets_cbs %d pri_tc_tbl_0 %x max_ets_tc %d\n",
- p_params->ets_willing,
- p_params->ets_cbs,
- p_ets->pri_tc_tbl[0], p_params->max_ets_tc);
+ "ETS params: willing %d, enabled = %d ets_cbs %d pri_tc_tbl_0 %x max_ets_tc %d\n",
+ p_params->ets_willing, p_params->ets_enabled,
+ p_params->ets_cbs, p_ets->pri_tc_tbl[0],
+ p_params->max_ets_tc);
if (p_params->ets_enabled && !p_params->max_ets_tc) {
p_params->max_ets_tc = QED_MAX_PFC_PRIORITIES;
@@ -622,8 +621,7 @@
}
static void
-qed_dcbx_get_local_params(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt, struct qed_dcbx_get *params)
+qed_dcbx_get_local_params(struct qed_hwfn *p_hwfn, struct qed_dcbx_get *params)
{
struct dcbx_features *p_feat;
@@ -635,8 +633,7 @@
}
static void
-qed_dcbx_get_remote_params(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt, struct qed_dcbx_get *params)
+qed_dcbx_get_remote_params(struct qed_hwfn *p_hwfn, struct qed_dcbx_get *params)
{
struct dcbx_features *p_feat;
@@ -649,7 +646,6 @@
static void
qed_dcbx_get_operational_params(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt,
struct qed_dcbx_get *params)
{
struct qed_dcbx_operational_params *p_operational;
@@ -670,6 +666,7 @@
if (!enabled) {
p_operational->enabled = enabled;
p_operational->valid = false;
+ DP_VERBOSE(p_hwfn, QED_MSG_DCB, "Dcbx is disabled\n");
return;
}
@@ -683,8 +680,14 @@
DCBX_CONFIG_VERSION_CEE);
p_operational->cee = val;
- DP_VERBOSE(p_hwfn, QED_MSG_DCB, "Version support: ieee %d, cee %d\n",
- p_operational->ieee, p_operational->cee);
+ val = !!(QED_MFW_GET_FIELD(flags, DCBX_CONFIG_VERSION) ==
+ DCBX_CONFIG_VERSION_STATIC);
+ p_operational->local = val;
+
+ DP_VERBOSE(p_hwfn, QED_MSG_DCB,
+ "Version support: ieee %d, cee %d, static %d\n",
+ p_operational->ieee, p_operational->cee,
+ p_operational->local);
qed_dcbx_get_common_params(p_hwfn, &p_feat->app,
p_feat->app.app_pri_tbl, &p_feat->ets,
@@ -699,7 +702,6 @@
static void
qed_dcbx_get_local_lldp_params(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt,
struct qed_dcbx_get *params)
{
struct lldp_config_params_s *p_local;
@@ -714,7 +716,6 @@
static void
qed_dcbx_get_remote_lldp_params(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt,
struct qed_dcbx_get *params)
{
struct lldp_status_params_s *p_remote;
@@ -728,25 +729,24 @@
}
static int
-qed_dcbx_get_params(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt,
- struct qed_dcbx_get *p_params,
+qed_dcbx_get_params(struct qed_hwfn *p_hwfn, struct qed_dcbx_get *p_params,
enum qed_mib_read_type type)
{
switch (type) {
case QED_DCBX_REMOTE_MIB:
- qed_dcbx_get_remote_params(p_hwfn, p_ptt, p_params);
+ qed_dcbx_get_remote_params(p_hwfn, p_params);
break;
case QED_DCBX_LOCAL_MIB:
- qed_dcbx_get_local_params(p_hwfn, p_ptt, p_params);
+ qed_dcbx_get_local_params(p_hwfn, p_params);
break;
case QED_DCBX_OPERATIONAL_MIB:
- qed_dcbx_get_operational_params(p_hwfn, p_ptt, p_params);
+ qed_dcbx_get_operational_params(p_hwfn, p_params);
break;
case QED_DCBX_REMOTE_LLDP_MIB:
- qed_dcbx_get_remote_lldp_params(p_hwfn, p_ptt, p_params);
+ qed_dcbx_get_remote_lldp_params(p_hwfn, p_params);
break;
case QED_DCBX_LOCAL_LLDP_MIB:
- qed_dcbx_get_local_lldp_params(p_hwfn, p_ptt, p_params);
+ qed_dcbx_get_local_lldp_params(p_hwfn, p_params);
break;
default:
DP_ERR(p_hwfn, "MIB read err, unknown mib type %d\n", type);
@@ -904,7 +904,8 @@
qed_sp_pf_update(p_hwfn);
}
}
- qed_dcbx_get_params(p_hwfn, p_ptt, &p_hwfn->p_dcbx_info->get, type);
+
+ qed_dcbx_get_params(p_hwfn, &p_hwfn->p_dcbx_info->get, type);
qed_dcbx_aen(p_hwfn, type);
return rc;
@@ -912,17 +913,14 @@
int qed_dcbx_info_alloc(struct qed_hwfn *p_hwfn)
{
- int rc = 0;
-
p_hwfn->p_dcbx_info = kzalloc(sizeof(*p_hwfn->p_dcbx_info), GFP_KERNEL);
if (!p_hwfn->p_dcbx_info)
- rc = -ENOMEM;
+ return -ENOMEM;
- return rc;
+ return 0;
}
-void qed_dcbx_info_free(struct qed_hwfn *p_hwfn,
- struct qed_dcbx_info *p_dcbx_info)
+void qed_dcbx_info_free(struct qed_hwfn *p_hwfn)
{
kfree(p_hwfn->p_dcbx_info);
}
@@ -961,14 +959,9 @@
p_dcb_data = &p_dest->fcoe_dcb_data;
qed_dcbx_update_protocol_data(p_dcb_data, p_src, DCBX_PROTOCOL_FCOE);
p_dcb_data = &p_dest->roce_dcb_data;
-
- if (p_src->arr[DCBX_PROTOCOL_ROCE].update)
- qed_dcbx_update_protocol_data(p_dcb_data, p_src,
- DCBX_PROTOCOL_ROCE);
- if (p_src->arr[DCBX_PROTOCOL_ROCE_V2].update)
- qed_dcbx_update_protocol_data(p_dcb_data, p_src,
- DCBX_PROTOCOL_ROCE_V2);
-
+ qed_dcbx_update_protocol_data(p_dcb_data, p_src, DCBX_PROTOCOL_ROCE);
+ p_dcb_data = &p_dest->rroce_dcb_data;
+ qed_dcbx_update_protocol_data(p_dcb_data, p_src, DCBX_PROTOCOL_ROCE_V2);
p_dcb_data = &p_dest->iscsi_dcb_data;
qed_dcbx_update_protocol_data(p_dcb_data, p_src, DCBX_PROTOCOL_ISCSI);
p_dcb_data = &p_dest->eth_dcb_data;
@@ -994,7 +987,7 @@
if (rc)
goto out;
- rc = qed_dcbx_get_params(p_hwfn, p_ptt, p_get, type);
+ rc = qed_dcbx_get_params(p_hwfn, p_get, type);
out:
qed_ptt_release(p_hwfn, p_ptt);
@@ -1169,6 +1162,9 @@
local_admin->config = DCBX_CONFIG_VERSION_DISABLED;
}
+ DP_VERBOSE(p_hwfn, QED_MSG_DCB, "Dcbx version = %d\n",
+ local_admin->config);
+
if (params->override_flags & QED_DCBX_OVERRIDE_PFC_CFG)
qed_dcbx_set_pfc_data(p_hwfn, &local_admin->features.pfc,
¶ms->config.params);
@@ -1245,6 +1241,8 @@
p_hwfn->p_dcbx_info->set.ver_num |= DCBX_CONFIG_VERSION_CEE;
if (dcbx_info->operational.ieee)
p_hwfn->p_dcbx_info->set.ver_num |= DCBX_CONFIG_VERSION_IEEE;
+ if (dcbx_info->operational.local)
+ p_hwfn->p_dcbx_info->set.ver_num |= DCBX_CONFIG_VERSION_STATIC;
p_hwfn->p_dcbx_info->set.enabled = dcbx_info->operational.enabled;
memcpy(&p_hwfn->p_dcbx_info->set.config.params,
@@ -1794,8 +1792,9 @@
DP_VERBOSE(hwfn, QED_MSG_DCB, "new mode = %x\n", mode);
- if (!(mode & DCB_CAP_DCBX_VER_IEEE) && !(mode & DCB_CAP_DCBX_VER_CEE)) {
- DP_INFO(hwfn, "Allowed mode is cee, ieee or both\n");
+ if (!(mode & DCB_CAP_DCBX_VER_IEEE) &&
+ !(mode & DCB_CAP_DCBX_VER_CEE) && !(mode & DCB_CAP_DCBX_STATIC)) {
+ DP_INFO(hwfn, "Allowed modes are cee, ieee or static\n");
return 1;
}
@@ -1815,6 +1814,11 @@
dcbx_set.enabled = true;
}
+ if (mode & DCB_CAP_DCBX_STATIC) {
+ dcbx_set.ver_num |= DCBX_CONFIG_VERSION_STATIC;
+ dcbx_set.enabled = true;
+ }
+
ptt = qed_ptt_acquire(hwfn);
if (!ptt)
return 1;
@@ -1823,7 +1827,7 @@
qed_ptt_release(hwfn, ptt);
- return 0;
+ return rc;
}
static u8 qed_dcbnl_getfeatcfg(struct qed_dev *cdev, int featid, u8 *flags)
@@ -2202,15 +2206,46 @@
return qed_dcbnl_get_ieee_pfc(cdev, pfc, true);
}
+static int qed_get_sf_ieee_value(u8 selector, u8 *sf_ieee)
+{
+ switch (selector) {
+ case IEEE_8021QAZ_APP_SEL_ETHERTYPE:
+ *sf_ieee = QED_DCBX_SF_IEEE_ETHTYPE;
+ break;
+ case IEEE_8021QAZ_APP_SEL_STREAM:
+ *sf_ieee = QED_DCBX_SF_IEEE_TCP_PORT;
+ break;
+ case IEEE_8021QAZ_APP_SEL_DGRAM:
+ *sf_ieee = QED_DCBX_SF_IEEE_UDP_PORT;
+ break;
+ case IEEE_8021QAZ_APP_SEL_ANY:
+ *sf_ieee = QED_DCBX_SF_IEEE_TCP_UDP_PORT;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
static int qed_dcbnl_ieee_getapp(struct qed_dev *cdev, struct dcb_app *app)
{
struct qed_hwfn *hwfn = QED_LEADING_HWFN(cdev);
struct qed_dcbx_get *dcbx_info;
struct qed_app_entry *entry;
- bool ethtype;
u8 prio = 0;
+ u8 sf_ieee;
int i;
+ DP_VERBOSE(hwfn, QED_MSG_DCB, "selector = %d protocol = %d\n",
+ app->selector, app->protocol);
+
+ if (qed_get_sf_ieee_value(app->selector, &sf_ieee)) {
+ DP_INFO(cdev, "Invalid selector field value %d\n",
+ app->selector);
+ return -EINVAL;
+ }
+
dcbx_info = qed_dcbnl_get_dcbx(hwfn, QED_DCBX_OPERATIONAL_MIB);
if (!dcbx_info)
return -EINVAL;
@@ -2221,11 +2256,9 @@
return -EINVAL;
}
- /* ieee defines the selector field value for ethertype to be 1 */
- ethtype = !!((app->selector - 1) == DCB_APP_IDTYPE_ETHTYPE);
for (i = 0; i < QED_DCBX_MAX_APP_PROTOCOL; i++) {
entry = &dcbx_info->operational.params.app_entry[i];
- if ((entry->ethtype == ethtype) &&
+ if ((entry->sf_ieee == sf_ieee) &&
(entry->proto_id == app->protocol)) {
prio = entry->prio;
break;
@@ -2253,14 +2286,22 @@
struct qed_dcbx_set dcbx_set;
struct qed_app_entry *entry;
struct qed_ptt *ptt;
- bool ethtype;
+ u8 sf_ieee;
int rc, i;
+ DP_VERBOSE(hwfn, QED_MSG_DCB, "selector = %d protocol = %d pri = %d\n",
+ app->selector, app->protocol, app->priority);
if (app->priority < 0 || app->priority >= QED_MAX_PFC_PRIORITIES) {
DP_INFO(hwfn, "Invalid priority %d\n", app->priority);
return -EINVAL;
}
+ if (qed_get_sf_ieee_value(app->selector, &sf_ieee)) {
+ DP_INFO(cdev, "Invalid selector field value %d\n",
+ app->selector);
+ return -EINVAL;
+ }
+
dcbx_info = qed_dcbnl_get_dcbx(hwfn, QED_DCBX_OPERATIONAL_MIB);
if (!dcbx_info)
return -EINVAL;
@@ -2278,11 +2319,9 @@
if (rc)
return -EINVAL;
- /* ieee defines the selector field value for ethertype to be 1 */
- ethtype = !!((app->selector - 1) == DCB_APP_IDTYPE_ETHTYPE);
for (i = 0; i < QED_DCBX_MAX_APP_PROTOCOL; i++) {
entry = &dcbx_set.config.params.app_entry[i];
- if ((entry->ethtype == ethtype) &&
+ if ((entry->sf_ieee == sf_ieee) &&
(entry->proto_id == app->protocol))
break;
/* First empty slot */
@@ -2298,7 +2337,7 @@
}
dcbx_set.override_flags |= QED_DCBX_OVERRIDE_APP_CFG;
- dcbx_set.config.params.app_entry[i].ethtype = ethtype;
+ dcbx_set.config.params.app_entry[i].sf_ieee = sf_ieee;
dcbx_set.config.params.app_entry[i].proto_id = app->protocol;
dcbx_set.config.params.app_entry[i].prio = BIT(app->priority);
diff --git a/drivers/net/ethernet/qlogic/qed/qed_dcbx.h b/drivers/net/ethernet/qlogic/qed/qed_dcbx.h
index 0fabe97..414e262 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_dcbx.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_dcbx.h
@@ -85,9 +85,6 @@
enum qed_pci_personality personality;
};
-#define QED_MFW_GET_FIELD(name, field) \
- (((name) & (field ## _MASK)) >> (field ## _SHIFT))
-
struct qed_dcbx_info {
struct lldp_status_params_s lldp_remote[LLDP_MAX_LLDP_AGENTS];
struct lldp_config_params_s lldp_local[LLDP_MAX_LLDP_AGENTS];
@@ -122,7 +119,7 @@
struct qed_ptt *, enum qed_mib_read_type);
int qed_dcbx_info_alloc(struct qed_hwfn *p_hwfn);
-void qed_dcbx_info_free(struct qed_hwfn *, struct qed_dcbx_info *);
+void qed_dcbx_info_free(struct qed_hwfn *p_hwfn);
void qed_dcbx_set_pf_update_params(struct qed_dcbx_results *p_src,
struct pf_update_ramrod_data *p_dest);
diff --git a/drivers/net/ethernet/qlogic/qed/qed_debug.c b/drivers/net/ethernet/qlogic/qed/qed_debug.c
index 68f19ca..483241b 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_debug.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_debug.c
@@ -17,7 +17,6 @@
/* Chip IDs enum */
enum chip_ids {
- CHIP_RESERVED,
CHIP_BB_B0,
CHIP_K2,
MAX_CHIP_IDS
@@ -40,6 +39,7 @@
MEM_GROUP_BTB_RAM,
MEM_GROUP_RDIF_CTX,
MEM_GROUP_TDIF_CTX,
+ MEM_GROUP_CFC_MEM,
MEM_GROUP_CONN_CFC_MEM,
MEM_GROUP_TASK_CFC_MEM,
MEM_GROUP_CAU_PI,
@@ -72,6 +72,7 @@
"BTB_RAM",
"RDIF_CTX",
"TDIF_CTX",
+ "CFC_MEM",
"CONN_CFC_MEM",
"TASK_CFC_MEM",
"CAU_PI",
@@ -185,13 +186,16 @@
u32 size_in_dwords;
};
+struct chip_platform_defs {
+ u8 num_ports;
+ u8 num_pfs;
+ u8 num_vfs;
+};
+
/* Chip constant definitions */
struct chip_defs {
const char *name;
- struct {
- u8 num_ports;
- u8 num_pfs;
- } per_platform[MAX_PLATFORM_IDS];
+ struct chip_platform_defs per_platform[MAX_PLATFORM_IDS];
};
/* Platform constant definitions */
@@ -405,22 +409,23 @@
/***************************** Constant Arrays *******************************/
/* Debug arrays */
-static struct dbg_array s_dbg_arrays[MAX_BIN_DBG_BUFFER_TYPE] = { {NULL} };
+static struct dbg_array s_dbg_arrays[MAX_BIN_DBG_BUFFER_TYPE] = { {0} };
/* Chip constant definitions array */
static struct chip_defs s_chip_defs[MAX_CHIP_IDS] = {
- { "reserved", { {0, 0}, {0, 0}, {0, 0}, {0, 0} } },
{ "bb_b0",
- { {MAX_NUM_PORTS_BB, MAX_NUM_PFS_BB}, {0, 0}, {0, 0}, {0, 0} } },
- { "k2", { {MAX_NUM_PORTS_K2, MAX_NUM_PFS_K2}, {0, 0}, {0, 0}, {0, 0} } }
+ { {MAX_NUM_PORTS_BB, MAX_NUM_PFS_BB, MAX_NUM_VFS_BB}, {0, 0, 0},
+ {0, 0, 0}, {0, 0, 0} } },
+ { "k2",
+ { {MAX_NUM_PORTS_K2, MAX_NUM_PFS_K2, MAX_NUM_VFS_K2}, {0, 0, 0},
+ {0, 0, 0}, {0, 0, 0} } }
};
/* Storm constant definitions array */
static struct storm_defs s_storm_defs[] = {
/* Tstorm */
{'T', BLOCK_TSEM,
- {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT,
- DBG_BUS_CLIENT_RBCT}, true,
+ {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT}, true,
TSEM_REG_FAST_MEMORY,
TSEM_REG_DBG_FRAME_MODE, TSEM_REG_SLOW_DBG_ACTIVE,
TSEM_REG_SLOW_DBG_MODE, TSEM_REG_DBG_MODE1_CFG,
@@ -432,8 +437,7 @@
4, TCM_REG_SM_TASK_CTX},
/* Mstorm */
{'M', BLOCK_MSEM,
- {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT,
- DBG_BUS_CLIENT_RBCM}, false,
+ {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM}, false,
MSEM_REG_FAST_MEMORY,
MSEM_REG_DBG_FRAME_MODE, MSEM_REG_SLOW_DBG_ACTIVE,
MSEM_REG_SLOW_DBG_MODE, MSEM_REG_DBG_MODE1_CFG,
@@ -445,8 +449,7 @@
7, MCM_REG_SM_TASK_CTX},
/* Ustorm */
{'U', BLOCK_USEM,
- {DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU,
- DBG_BUS_CLIENT_RBCU}, false,
+ {DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU}, false,
USEM_REG_FAST_MEMORY,
USEM_REG_DBG_FRAME_MODE, USEM_REG_SLOW_DBG_ACTIVE,
USEM_REG_SLOW_DBG_MODE, USEM_REG_DBG_MODE1_CFG,
@@ -458,8 +461,7 @@
3, UCM_REG_SM_TASK_CTX},
/* Xstorm */
{'X', BLOCK_XSEM,
- {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX,
- DBG_BUS_CLIENT_RBCX}, false,
+ {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX}, false,
XSEM_REG_FAST_MEMORY,
XSEM_REG_DBG_FRAME_MODE, XSEM_REG_SLOW_DBG_ACTIVE,
XSEM_REG_SLOW_DBG_MODE, XSEM_REG_DBG_MODE1_CFG,
@@ -471,8 +473,7 @@
0, 0},
/* Ystorm */
{'Y', BLOCK_YSEM,
- {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX,
- DBG_BUS_CLIENT_RBCY}, false,
+ {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCY}, false,
YSEM_REG_FAST_MEMORY,
YSEM_REG_DBG_FRAME_MODE, YSEM_REG_SLOW_DBG_ACTIVE,
YSEM_REG_SLOW_DBG_MODE, YSEM_REG_DBG_MODE1_CFG,
@@ -484,8 +485,7 @@
12, YCM_REG_SM_TASK_CTX},
/* Pstorm */
{'P', BLOCK_PSEM,
- {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS,
- DBG_BUS_CLIENT_RBCS}, true,
+ {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS}, true,
PSEM_REG_FAST_MEMORY,
PSEM_REG_DBG_FRAME_MODE, PSEM_REG_SLOW_DBG_ACTIVE,
PSEM_REG_SLOW_DBG_MODE, PSEM_REG_DBG_MODE1_CFG,
@@ -499,8 +499,9 @@
/* Block definitions array */
static struct block_defs block_grc_defs = {
- "grc", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCN, DBG_BUS_CLIENT_RBCN, DBG_BUS_CLIENT_RBCN},
+ "grc",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCN, DBG_BUS_CLIENT_RBCN},
GRC_REG_DBG_SELECT, GRC_REG_DBG_DWORD_ENABLE,
GRC_REG_DBG_SHIFT, GRC_REG_DBG_FORCE_VALID,
GRC_REG_DBG_FORCE_FRAME,
@@ -508,29 +509,30 @@
};
static struct block_defs block_miscs_defs = {
- "miscs", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ "miscs", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
0, 0, 0, 0, 0,
false, false, MAX_DBG_RESET_REGS, 0
};
static struct block_defs block_misc_defs = {
- "misc", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ "misc", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
0, 0, 0, 0, 0,
false, false, MAX_DBG_RESET_REGS, 0
};
static struct block_defs block_dbu_defs = {
- "dbu", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ "dbu", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
0, 0, 0, 0, 0,
false, false, MAX_DBG_RESET_REGS, 0
};
static struct block_defs block_pglue_b_defs = {
- "pglue_b", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCH, DBG_BUS_CLIENT_RBCH, DBG_BUS_CLIENT_RBCH},
+ "pglue_b",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCH, DBG_BUS_CLIENT_RBCH},
PGLUE_B_REG_DBG_SELECT, PGLUE_B_REG_DBG_DWORD_ENABLE,
PGLUE_B_REG_DBG_SHIFT, PGLUE_B_REG_DBG_FORCE_VALID,
PGLUE_B_REG_DBG_FORCE_FRAME,
@@ -538,8 +540,9 @@
};
static struct block_defs block_cnig_defs = {
- "cnig", {false, false, true}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCW},
+ "cnig",
+ {false, true}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCW},
CNIG_REG_DBG_SELECT_K2, CNIG_REG_DBG_DWORD_ENABLE_K2,
CNIG_REG_DBG_SHIFT_K2, CNIG_REG_DBG_FORCE_VALID_K2,
CNIG_REG_DBG_FORCE_FRAME_K2,
@@ -547,15 +550,16 @@
};
static struct block_defs block_cpmu_defs = {
- "cpmu", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ "cpmu", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
0, 0, 0, 0, 0,
true, false, DBG_RESET_REG_MISCS_PL_HV, 8
};
static struct block_defs block_ncsi_defs = {
- "ncsi", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCZ, DBG_BUS_CLIENT_RBCZ, DBG_BUS_CLIENT_RBCZ},
+ "ncsi",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCZ, DBG_BUS_CLIENT_RBCZ},
NCSI_REG_DBG_SELECT, NCSI_REG_DBG_DWORD_ENABLE,
NCSI_REG_DBG_SHIFT, NCSI_REG_DBG_FORCE_VALID,
NCSI_REG_DBG_FORCE_FRAME,
@@ -563,15 +567,16 @@
};
static struct block_defs block_opte_defs = {
- "opte", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ "opte", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
0, 0, 0, 0, 0,
true, false, DBG_RESET_REG_MISCS_PL_HV, 4
};
static struct block_defs block_bmb_defs = {
- "bmb", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCZ, DBG_BUS_CLIENT_RBCZ, DBG_BUS_CLIENT_RBCB},
+ "bmb",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCZ, DBG_BUS_CLIENT_RBCB},
BMB_REG_DBG_SELECT, BMB_REG_DBG_DWORD_ENABLE,
BMB_REG_DBG_SHIFT, BMB_REG_DBG_FORCE_VALID,
BMB_REG_DBG_FORCE_FRAME,
@@ -579,8 +584,9 @@
};
static struct block_defs block_pcie_defs = {
- "pcie", {false, false, true}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCH},
+ "pcie",
+ {false, true}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCH},
PCIE_REG_DBG_COMMON_SELECT, PCIE_REG_DBG_COMMON_DWORD_ENABLE,
PCIE_REG_DBG_COMMON_SHIFT, PCIE_REG_DBG_COMMON_FORCE_VALID,
PCIE_REG_DBG_COMMON_FORCE_FRAME,
@@ -588,15 +594,16 @@
};
static struct block_defs block_mcp_defs = {
- "mcp", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ "mcp", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
0, 0, 0, 0, 0,
false, false, MAX_DBG_RESET_REGS, 0
};
static struct block_defs block_mcp2_defs = {
- "mcp2", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCZ, DBG_BUS_CLIENT_RBCZ, DBG_BUS_CLIENT_RBCZ},
+ "mcp2",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCZ, DBG_BUS_CLIENT_RBCZ},
MCP2_REG_DBG_SELECT, MCP2_REG_DBG_DWORD_ENABLE,
MCP2_REG_DBG_SHIFT, MCP2_REG_DBG_FORCE_VALID,
MCP2_REG_DBG_FORCE_FRAME,
@@ -604,8 +611,9 @@
};
static struct block_defs block_pswhst_defs = {
- "pswhst", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
+ "pswhst",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
PSWHST_REG_DBG_SELECT, PSWHST_REG_DBG_DWORD_ENABLE,
PSWHST_REG_DBG_SHIFT, PSWHST_REG_DBG_FORCE_VALID,
PSWHST_REG_DBG_FORCE_FRAME,
@@ -613,8 +621,9 @@
};
static struct block_defs block_pswhst2_defs = {
- "pswhst2", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
+ "pswhst2",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
PSWHST2_REG_DBG_SELECT, PSWHST2_REG_DBG_DWORD_ENABLE,
PSWHST2_REG_DBG_SHIFT, PSWHST2_REG_DBG_FORCE_VALID,
PSWHST2_REG_DBG_FORCE_FRAME,
@@ -622,8 +631,9 @@
};
static struct block_defs block_pswrd_defs = {
- "pswrd", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
+ "pswrd",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
PSWRD_REG_DBG_SELECT, PSWRD_REG_DBG_DWORD_ENABLE,
PSWRD_REG_DBG_SHIFT, PSWRD_REG_DBG_FORCE_VALID,
PSWRD_REG_DBG_FORCE_FRAME,
@@ -631,8 +641,9 @@
};
static struct block_defs block_pswrd2_defs = {
- "pswrd2", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
+ "pswrd2",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
PSWRD2_REG_DBG_SELECT, PSWRD2_REG_DBG_DWORD_ENABLE,
PSWRD2_REG_DBG_SHIFT, PSWRD2_REG_DBG_FORCE_VALID,
PSWRD2_REG_DBG_FORCE_FRAME,
@@ -640,8 +651,9 @@
};
static struct block_defs block_pswwr_defs = {
- "pswwr", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
+ "pswwr",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
PSWWR_REG_DBG_SELECT, PSWWR_REG_DBG_DWORD_ENABLE,
PSWWR_REG_DBG_SHIFT, PSWWR_REG_DBG_FORCE_VALID,
PSWWR_REG_DBG_FORCE_FRAME,
@@ -649,15 +661,16 @@
};
static struct block_defs block_pswwr2_defs = {
- "pswwr2", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ "pswwr2", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
0, 0, 0, 0, 0,
true, false, DBG_RESET_REG_MISC_PL_HV, 3
};
static struct block_defs block_pswrq_defs = {
- "pswrq", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
+ "pswrq",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
PSWRQ_REG_DBG_SELECT, PSWRQ_REG_DBG_DWORD_ENABLE,
PSWRQ_REG_DBG_SHIFT, PSWRQ_REG_DBG_FORCE_VALID,
PSWRQ_REG_DBG_FORCE_FRAME,
@@ -665,8 +678,9 @@
};
static struct block_defs block_pswrq2_defs = {
- "pswrq2", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
+ "pswrq2",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
PSWRQ2_REG_DBG_SELECT, PSWRQ2_REG_DBG_DWORD_ENABLE,
PSWRQ2_REG_DBG_SHIFT, PSWRQ2_REG_DBG_FORCE_VALID,
PSWRQ2_REG_DBG_FORCE_FRAME,
@@ -674,8 +688,9 @@
};
static struct block_defs block_pglcs_defs = {
- "pglcs", {false, false, true}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCH},
+ "pglcs",
+ {false, true}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCH},
PGLCS_REG_DBG_SELECT, PGLCS_REG_DBG_DWORD_ENABLE,
PGLCS_REG_DBG_SHIFT, PGLCS_REG_DBG_FORCE_VALID,
PGLCS_REG_DBG_FORCE_FRAME,
@@ -683,8 +698,9 @@
};
static struct block_defs block_ptu_defs = {
- "ptu", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
+ "ptu",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
PTU_REG_DBG_SELECT, PTU_REG_DBG_DWORD_ENABLE,
PTU_REG_DBG_SHIFT, PTU_REG_DBG_FORCE_VALID,
PTU_REG_DBG_FORCE_FRAME,
@@ -692,8 +708,9 @@
};
static struct block_defs block_dmae_defs = {
- "dmae", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
+ "dmae",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
DMAE_REG_DBG_SELECT, DMAE_REG_DBG_DWORD_ENABLE,
DMAE_REG_DBG_SHIFT, DMAE_REG_DBG_FORCE_VALID,
DMAE_REG_DBG_FORCE_FRAME,
@@ -701,8 +718,9 @@
};
static struct block_defs block_tcm_defs = {
- "tcm", {true, true, true}, true, DBG_TSTORM_ID,
- {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT},
+ "tcm",
+ {true, true}, true, DBG_TSTORM_ID,
+ {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT},
TCM_REG_DBG_SELECT, TCM_REG_DBG_DWORD_ENABLE,
TCM_REG_DBG_SHIFT, TCM_REG_DBG_FORCE_VALID,
TCM_REG_DBG_FORCE_FRAME,
@@ -710,8 +728,9 @@
};
static struct block_defs block_mcm_defs = {
- "mcm", {true, true, true}, true, DBG_MSTORM_ID,
- {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
+ "mcm",
+ {true, true}, true, DBG_MSTORM_ID,
+ {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
MCM_REG_DBG_SELECT, MCM_REG_DBG_DWORD_ENABLE,
MCM_REG_DBG_SHIFT, MCM_REG_DBG_FORCE_VALID,
MCM_REG_DBG_FORCE_FRAME,
@@ -719,8 +738,9 @@
};
static struct block_defs block_ucm_defs = {
- "ucm", {true, true, true}, true, DBG_USTORM_ID,
- {DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU},
+ "ucm",
+ {true, true}, true, DBG_USTORM_ID,
+ {DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU},
UCM_REG_DBG_SELECT, UCM_REG_DBG_DWORD_ENABLE,
UCM_REG_DBG_SHIFT, UCM_REG_DBG_FORCE_VALID,
UCM_REG_DBG_FORCE_FRAME,
@@ -728,8 +748,9 @@
};
static struct block_defs block_xcm_defs = {
- "xcm", {true, true, true}, true, DBG_XSTORM_ID,
- {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX},
+ "xcm",
+ {true, true}, true, DBG_XSTORM_ID,
+ {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX},
XCM_REG_DBG_SELECT, XCM_REG_DBG_DWORD_ENABLE,
XCM_REG_DBG_SHIFT, XCM_REG_DBG_FORCE_VALID,
XCM_REG_DBG_FORCE_FRAME,
@@ -737,8 +758,9 @@
};
static struct block_defs block_ycm_defs = {
- "ycm", {true, true, true}, true, DBG_YSTORM_ID,
- {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCY},
+ "ycm",
+ {true, true}, true, DBG_YSTORM_ID,
+ {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCY},
YCM_REG_DBG_SELECT, YCM_REG_DBG_DWORD_ENABLE,
YCM_REG_DBG_SHIFT, YCM_REG_DBG_FORCE_VALID,
YCM_REG_DBG_FORCE_FRAME,
@@ -746,8 +768,9 @@
};
static struct block_defs block_pcm_defs = {
- "pcm", {true, true, true}, true, DBG_PSTORM_ID,
- {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS},
+ "pcm",
+ {true, true}, true, DBG_PSTORM_ID,
+ {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS},
PCM_REG_DBG_SELECT, PCM_REG_DBG_DWORD_ENABLE,
PCM_REG_DBG_SHIFT, PCM_REG_DBG_FORCE_VALID,
PCM_REG_DBG_FORCE_FRAME,
@@ -755,8 +778,9 @@
};
static struct block_defs block_qm_defs = {
- "qm", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCQ},
+ "qm",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCQ},
QM_REG_DBG_SELECT, QM_REG_DBG_DWORD_ENABLE,
QM_REG_DBG_SHIFT, QM_REG_DBG_FORCE_VALID,
QM_REG_DBG_FORCE_FRAME,
@@ -764,8 +788,9 @@
};
static struct block_defs block_tm_defs = {
- "tm", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS},
+ "tm",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS},
TM_REG_DBG_SELECT, TM_REG_DBG_DWORD_ENABLE,
TM_REG_DBG_SHIFT, TM_REG_DBG_FORCE_VALID,
TM_REG_DBG_FORCE_FRAME,
@@ -773,8 +798,9 @@
};
static struct block_defs block_dorq_defs = {
- "dorq", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCY},
+ "dorq",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCY},
DORQ_REG_DBG_SELECT, DORQ_REG_DBG_DWORD_ENABLE,
DORQ_REG_DBG_SHIFT, DORQ_REG_DBG_FORCE_VALID,
DORQ_REG_DBG_FORCE_FRAME,
@@ -782,8 +808,9 @@
};
static struct block_defs block_brb_defs = {
- "brb", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCR, DBG_BUS_CLIENT_RBCR, DBG_BUS_CLIENT_RBCR},
+ "brb",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCR, DBG_BUS_CLIENT_RBCR},
BRB_REG_DBG_SELECT, BRB_REG_DBG_DWORD_ENABLE,
BRB_REG_DBG_SHIFT, BRB_REG_DBG_FORCE_VALID,
BRB_REG_DBG_FORCE_FRAME,
@@ -791,8 +818,9 @@
};
static struct block_defs block_src_defs = {
- "src", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCF, DBG_BUS_CLIENT_RBCF, DBG_BUS_CLIENT_RBCF},
+ "src",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCF, DBG_BUS_CLIENT_RBCF},
SRC_REG_DBG_SELECT, SRC_REG_DBG_DWORD_ENABLE,
SRC_REG_DBG_SHIFT, SRC_REG_DBG_FORCE_VALID,
SRC_REG_DBG_FORCE_FRAME,
@@ -800,8 +828,9 @@
};
static struct block_defs block_prs_defs = {
- "prs", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCR, DBG_BUS_CLIENT_RBCR, DBG_BUS_CLIENT_RBCR},
+ "prs",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCR, DBG_BUS_CLIENT_RBCR},
PRS_REG_DBG_SELECT, PRS_REG_DBG_DWORD_ENABLE,
PRS_REG_DBG_SHIFT, PRS_REG_DBG_FORCE_VALID,
PRS_REG_DBG_FORCE_FRAME,
@@ -809,8 +838,9 @@
};
static struct block_defs block_tsdm_defs = {
- "tsdm", {true, true, true}, true, DBG_TSTORM_ID,
- {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT},
+ "tsdm",
+ {true, true}, true, DBG_TSTORM_ID,
+ {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT},
TSDM_REG_DBG_SELECT, TSDM_REG_DBG_DWORD_ENABLE,
TSDM_REG_DBG_SHIFT, TSDM_REG_DBG_FORCE_VALID,
TSDM_REG_DBG_FORCE_FRAME,
@@ -818,8 +848,9 @@
};
static struct block_defs block_msdm_defs = {
- "msdm", {true, true, true}, true, DBG_MSTORM_ID,
- {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
+ "msdm",
+ {true, true}, true, DBG_MSTORM_ID,
+ {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
MSDM_REG_DBG_SELECT, MSDM_REG_DBG_DWORD_ENABLE,
MSDM_REG_DBG_SHIFT, MSDM_REG_DBG_FORCE_VALID,
MSDM_REG_DBG_FORCE_FRAME,
@@ -827,8 +858,9 @@
};
static struct block_defs block_usdm_defs = {
- "usdm", {true, true, true}, true, DBG_USTORM_ID,
- {DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU},
+ "usdm",
+ {true, true}, true, DBG_USTORM_ID,
+ {DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU},
USDM_REG_DBG_SELECT, USDM_REG_DBG_DWORD_ENABLE,
USDM_REG_DBG_SHIFT, USDM_REG_DBG_FORCE_VALID,
USDM_REG_DBG_FORCE_FRAME,
@@ -836,8 +868,9 @@
};
static struct block_defs block_xsdm_defs = {
- "xsdm", {true, true, true}, true, DBG_XSTORM_ID,
- {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX},
+ "xsdm",
+ {true, true}, true, DBG_XSTORM_ID,
+ {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX},
XSDM_REG_DBG_SELECT, XSDM_REG_DBG_DWORD_ENABLE,
XSDM_REG_DBG_SHIFT, XSDM_REG_DBG_FORCE_VALID,
XSDM_REG_DBG_FORCE_FRAME,
@@ -845,8 +878,9 @@
};
static struct block_defs block_ysdm_defs = {
- "ysdm", {true, true, true}, true, DBG_YSTORM_ID,
- {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCY},
+ "ysdm",
+ {true, true}, true, DBG_YSTORM_ID,
+ {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCY},
YSDM_REG_DBG_SELECT, YSDM_REG_DBG_DWORD_ENABLE,
YSDM_REG_DBG_SHIFT, YSDM_REG_DBG_FORCE_VALID,
YSDM_REG_DBG_FORCE_FRAME,
@@ -854,8 +888,9 @@
};
static struct block_defs block_psdm_defs = {
- "psdm", {true, true, true}, true, DBG_PSTORM_ID,
- {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS},
+ "psdm",
+ {true, true}, true, DBG_PSTORM_ID,
+ {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS},
PSDM_REG_DBG_SELECT, PSDM_REG_DBG_DWORD_ENABLE,
PSDM_REG_DBG_SHIFT, PSDM_REG_DBG_FORCE_VALID,
PSDM_REG_DBG_FORCE_FRAME,
@@ -863,8 +898,9 @@
};
static struct block_defs block_tsem_defs = {
- "tsem", {true, true, true}, true, DBG_TSTORM_ID,
- {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT},
+ "tsem",
+ {true, true}, true, DBG_TSTORM_ID,
+ {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT},
TSEM_REG_DBG_SELECT, TSEM_REG_DBG_DWORD_ENABLE,
TSEM_REG_DBG_SHIFT, TSEM_REG_DBG_FORCE_VALID,
TSEM_REG_DBG_FORCE_FRAME,
@@ -872,8 +908,9 @@
};
static struct block_defs block_msem_defs = {
- "msem", {true, true, true}, true, DBG_MSTORM_ID,
- {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
+ "msem",
+ {true, true}, true, DBG_MSTORM_ID,
+ {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
MSEM_REG_DBG_SELECT, MSEM_REG_DBG_DWORD_ENABLE,
MSEM_REG_DBG_SHIFT, MSEM_REG_DBG_FORCE_VALID,
MSEM_REG_DBG_FORCE_FRAME,
@@ -881,8 +918,9 @@
};
static struct block_defs block_usem_defs = {
- "usem", {true, true, true}, true, DBG_USTORM_ID,
- {DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU},
+ "usem",
+ {true, true}, true, DBG_USTORM_ID,
+ {DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU},
USEM_REG_DBG_SELECT, USEM_REG_DBG_DWORD_ENABLE,
USEM_REG_DBG_SHIFT, USEM_REG_DBG_FORCE_VALID,
USEM_REG_DBG_FORCE_FRAME,
@@ -890,8 +928,9 @@
};
static struct block_defs block_xsem_defs = {
- "xsem", {true, true, true}, true, DBG_XSTORM_ID,
- {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX},
+ "xsem",
+ {true, true}, true, DBG_XSTORM_ID,
+ {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX},
XSEM_REG_DBG_SELECT, XSEM_REG_DBG_DWORD_ENABLE,
XSEM_REG_DBG_SHIFT, XSEM_REG_DBG_FORCE_VALID,
XSEM_REG_DBG_FORCE_FRAME,
@@ -899,8 +938,9 @@
};
static struct block_defs block_ysem_defs = {
- "ysem", {true, true, true}, true, DBG_YSTORM_ID,
- {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCY},
+ "ysem",
+ {true, true}, true, DBG_YSTORM_ID,
+ {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCY},
YSEM_REG_DBG_SELECT, YSEM_REG_DBG_DWORD_ENABLE,
YSEM_REG_DBG_SHIFT, YSEM_REG_DBG_FORCE_VALID,
YSEM_REG_DBG_FORCE_FRAME,
@@ -908,8 +948,9 @@
};
static struct block_defs block_psem_defs = {
- "psem", {true, true, true}, true, DBG_PSTORM_ID,
- {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS},
+ "psem",
+ {true, true}, true, DBG_PSTORM_ID,
+ {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS},
PSEM_REG_DBG_SELECT, PSEM_REG_DBG_DWORD_ENABLE,
PSEM_REG_DBG_SHIFT, PSEM_REG_DBG_FORCE_VALID,
PSEM_REG_DBG_FORCE_FRAME,
@@ -917,8 +958,9 @@
};
static struct block_defs block_rss_defs = {
- "rss", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT},
+ "rss",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT},
RSS_REG_DBG_SELECT, RSS_REG_DBG_DWORD_ENABLE,
RSS_REG_DBG_SHIFT, RSS_REG_DBG_FORCE_VALID,
RSS_REG_DBG_FORCE_FRAME,
@@ -926,8 +968,9 @@
};
static struct block_defs block_tmld_defs = {
- "tmld", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
+ "tmld",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
TMLD_REG_DBG_SELECT, TMLD_REG_DBG_DWORD_ENABLE,
TMLD_REG_DBG_SHIFT, TMLD_REG_DBG_FORCE_VALID,
TMLD_REG_DBG_FORCE_FRAME,
@@ -935,8 +978,9 @@
};
static struct block_defs block_muld_defs = {
- "muld", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU},
+ "muld",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU},
MULD_REG_DBG_SELECT, MULD_REG_DBG_DWORD_ENABLE,
MULD_REG_DBG_SHIFT, MULD_REG_DBG_FORCE_VALID,
MULD_REG_DBG_FORCE_FRAME,
@@ -944,8 +988,9 @@
};
static struct block_defs block_yuld_defs = {
- "yuld", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU},
+ "yuld",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCU, DBG_BUS_CLIENT_RBCU},
YULD_REG_DBG_SELECT, YULD_REG_DBG_DWORD_ENABLE,
YULD_REG_DBG_SHIFT, YULD_REG_DBG_FORCE_VALID,
YULD_REG_DBG_FORCE_FRAME,
@@ -953,8 +998,9 @@
};
static struct block_defs block_xyld_defs = {
- "xyld", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX},
+ "xyld",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCX, DBG_BUS_CLIENT_RBCX},
XYLD_REG_DBG_SELECT, XYLD_REG_DBG_DWORD_ENABLE,
XYLD_REG_DBG_SHIFT, XYLD_REG_DBG_FORCE_VALID,
XYLD_REG_DBG_FORCE_FRAME,
@@ -962,8 +1008,9 @@
};
static struct block_defs block_prm_defs = {
- "prm", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
+ "prm",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
PRM_REG_DBG_SELECT, PRM_REG_DBG_DWORD_ENABLE,
PRM_REG_DBG_SHIFT, PRM_REG_DBG_FORCE_VALID,
PRM_REG_DBG_FORCE_FRAME,
@@ -971,8 +1018,9 @@
};
static struct block_defs block_pbf_pb1_defs = {
- "pbf_pb1", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCV},
+ "pbf_pb1",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCV},
PBF_PB1_REG_DBG_SELECT, PBF_PB1_REG_DBG_DWORD_ENABLE,
PBF_PB1_REG_DBG_SHIFT, PBF_PB1_REG_DBG_FORCE_VALID,
PBF_PB1_REG_DBG_FORCE_FRAME,
@@ -981,8 +1029,9 @@
};
static struct block_defs block_pbf_pb2_defs = {
- "pbf_pb2", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCV},
+ "pbf_pb2",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCV},
PBF_PB2_REG_DBG_SELECT, PBF_PB2_REG_DBG_DWORD_ENABLE,
PBF_PB2_REG_DBG_SHIFT, PBF_PB2_REG_DBG_FORCE_VALID,
PBF_PB2_REG_DBG_FORCE_FRAME,
@@ -991,8 +1040,9 @@
};
static struct block_defs block_rpb_defs = {
- "rpb", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
+ "rpb",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
RPB_REG_DBG_SELECT, RPB_REG_DBG_DWORD_ENABLE,
RPB_REG_DBG_SHIFT, RPB_REG_DBG_FORCE_VALID,
RPB_REG_DBG_FORCE_FRAME,
@@ -1000,8 +1050,9 @@
};
static struct block_defs block_btb_defs = {
- "btb", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCR, DBG_BUS_CLIENT_RBCR, DBG_BUS_CLIENT_RBCV},
+ "btb",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCR, DBG_BUS_CLIENT_RBCV},
BTB_REG_DBG_SELECT, BTB_REG_DBG_DWORD_ENABLE,
BTB_REG_DBG_SHIFT, BTB_REG_DBG_FORCE_VALID,
BTB_REG_DBG_FORCE_FRAME,
@@ -1009,8 +1060,9 @@
};
static struct block_defs block_pbf_defs = {
- "pbf", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCV},
+ "pbf",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCV},
PBF_REG_DBG_SELECT, PBF_REG_DBG_DWORD_ENABLE,
PBF_REG_DBG_SHIFT, PBF_REG_DBG_FORCE_VALID,
PBF_REG_DBG_FORCE_FRAME,
@@ -1018,8 +1070,9 @@
};
static struct block_defs block_rdif_defs = {
- "rdif", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
+ "rdif",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCT, DBG_BUS_CLIENT_RBCM},
RDIF_REG_DBG_SELECT, RDIF_REG_DBG_DWORD_ENABLE,
RDIF_REG_DBG_SHIFT, RDIF_REG_DBG_FORCE_VALID,
RDIF_REG_DBG_FORCE_FRAME,
@@ -1027,8 +1080,9 @@
};
static struct block_defs block_tdif_defs = {
- "tdif", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS},
+ "tdif",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCS, DBG_BUS_CLIENT_RBCS},
TDIF_REG_DBG_SELECT, TDIF_REG_DBG_DWORD_ENABLE,
TDIF_REG_DBG_SHIFT, TDIF_REG_DBG_FORCE_VALID,
TDIF_REG_DBG_FORCE_FRAME,
@@ -1036,8 +1090,9 @@
};
static struct block_defs block_cdu_defs = {
- "cdu", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCF, DBG_BUS_CLIENT_RBCF, DBG_BUS_CLIENT_RBCF},
+ "cdu",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCF, DBG_BUS_CLIENT_RBCF},
CDU_REG_DBG_SELECT, CDU_REG_DBG_DWORD_ENABLE,
CDU_REG_DBG_SHIFT, CDU_REG_DBG_FORCE_VALID,
CDU_REG_DBG_FORCE_FRAME,
@@ -1045,8 +1100,9 @@
};
static struct block_defs block_ccfc_defs = {
- "ccfc", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCF, DBG_BUS_CLIENT_RBCF, DBG_BUS_CLIENT_RBCF},
+ "ccfc",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCF, DBG_BUS_CLIENT_RBCF},
CCFC_REG_DBG_SELECT, CCFC_REG_DBG_DWORD_ENABLE,
CCFC_REG_DBG_SHIFT, CCFC_REG_DBG_FORCE_VALID,
CCFC_REG_DBG_FORCE_FRAME,
@@ -1054,8 +1110,9 @@
};
static struct block_defs block_tcfc_defs = {
- "tcfc", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCF, DBG_BUS_CLIENT_RBCF, DBG_BUS_CLIENT_RBCF},
+ "tcfc",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCF, DBG_BUS_CLIENT_RBCF},
TCFC_REG_DBG_SELECT, TCFC_REG_DBG_DWORD_ENABLE,
TCFC_REG_DBG_SHIFT, TCFC_REG_DBG_FORCE_VALID,
TCFC_REG_DBG_FORCE_FRAME,
@@ -1063,8 +1120,9 @@
};
static struct block_defs block_igu_defs = {
- "igu", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
+ "igu",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
IGU_REG_DBG_SELECT, IGU_REG_DBG_DWORD_ENABLE,
IGU_REG_DBG_SHIFT, IGU_REG_DBG_FORCE_VALID,
IGU_REG_DBG_FORCE_FRAME,
@@ -1072,8 +1130,9 @@
};
static struct block_defs block_cau_defs = {
- "cau", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
+ "cau",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCP, DBG_BUS_CLIENT_RBCP},
CAU_REG_DBG_SELECT, CAU_REG_DBG_DWORD_ENABLE,
CAU_REG_DBG_SHIFT, CAU_REG_DBG_FORCE_VALID,
CAU_REG_DBG_FORCE_FRAME,
@@ -1081,8 +1140,9 @@
};
static struct block_defs block_umac_defs = {
- "umac", {false, false, true}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCZ},
+ "umac",
+ {false, true}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCZ},
UMAC_REG_DBG_SELECT, UMAC_REG_DBG_DWORD_ENABLE,
UMAC_REG_DBG_SHIFT, UMAC_REG_DBG_FORCE_VALID,
UMAC_REG_DBG_FORCE_FRAME,
@@ -1090,22 +1150,23 @@
};
static struct block_defs block_xmac_defs = {
- "xmac", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ "xmac", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
0, 0, 0, 0, 0,
false, false, MAX_DBG_RESET_REGS, 0
};
static struct block_defs block_dbg_defs = {
- "dbg", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ "dbg", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
0, 0, 0, 0, 0,
true, true, DBG_RESET_REG_MISC_PL_PDA_VAUX, 3
};
static struct block_defs block_nig_defs = {
- "nig", {true, true, true}, false, 0,
- {DBG_BUS_CLIENT_RBCN, DBG_BUS_CLIENT_RBCN, DBG_BUS_CLIENT_RBCN},
+ "nig",
+ {true, true}, false, 0,
+ {DBG_BUS_CLIENT_RBCN, DBG_BUS_CLIENT_RBCN},
NIG_REG_DBG_SELECT, NIG_REG_DBG_DWORD_ENABLE,
NIG_REG_DBG_SHIFT, NIG_REG_DBG_FORCE_VALID,
NIG_REG_DBG_FORCE_FRAME,
@@ -1113,8 +1174,9 @@
};
static struct block_defs block_wol_defs = {
- "wol", {false, false, true}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCZ},
+ "wol",
+ {false, true}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCZ},
WOL_REG_DBG_SELECT, WOL_REG_DBG_DWORD_ENABLE,
WOL_REG_DBG_SHIFT, WOL_REG_DBG_FORCE_VALID,
WOL_REG_DBG_FORCE_FRAME,
@@ -1122,8 +1184,9 @@
};
static struct block_defs block_bmbn_defs = {
- "bmbn", {false, false, true}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCB},
+ "bmbn",
+ {false, true}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCB},
BMBN_REG_DBG_SELECT, BMBN_REG_DBG_DWORD_ENABLE,
BMBN_REG_DBG_SHIFT, BMBN_REG_DBG_FORCE_VALID,
BMBN_REG_DBG_FORCE_FRAME,
@@ -1131,15 +1194,16 @@
};
static struct block_defs block_ipc_defs = {
- "ipc", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ "ipc", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
0, 0, 0, 0, 0,
true, false, DBG_RESET_REG_MISCS_PL_UA, 8
};
static struct block_defs block_nwm_defs = {
- "nwm", {false, false, true}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCW},
+ "nwm",
+ {false, true}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCW},
NWM_REG_DBG_SELECT, NWM_REG_DBG_DWORD_ENABLE,
NWM_REG_DBG_SHIFT, NWM_REG_DBG_FORCE_VALID,
NWM_REG_DBG_FORCE_FRAME,
@@ -1147,22 +1211,29 @@
};
static struct block_defs block_nws_defs = {
- "nws", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
- 0, 0, 0, 0, 0,
+ "nws",
+ {false, true}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCW},
+ NWS_REG_DBG_SELECT, NWS_REG_DBG_DWORD_ENABLE,
+ NWS_REG_DBG_SHIFT, NWS_REG_DBG_FORCE_VALID,
+ NWS_REG_DBG_FORCE_FRAME,
true, false, DBG_RESET_REG_MISCS_PL_HV, 12
};
static struct block_defs block_ms_defs = {
- "ms", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
- 0, 0, 0, 0, 0,
+ "ms",
+ {false, true}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCZ},
+ MS_REG_DBG_SELECT, MS_REG_DBG_DWORD_ENABLE,
+ MS_REG_DBG_SHIFT, MS_REG_DBG_FORCE_VALID,
+ MS_REG_DBG_FORCE_FRAME,
true, false, DBG_RESET_REG_MISCS_PL_HV, 13
};
static struct block_defs block_phy_pcie_defs = {
- "phy_pcie", {false, false, true}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCH},
+ "phy_pcie",
+ {false, true}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, DBG_BUS_CLIENT_RBCH},
PCIE_REG_DBG_COMMON_SELECT, PCIE_REG_DBG_COMMON_DWORD_ENABLE,
PCIE_REG_DBG_COMMON_SHIFT, PCIE_REG_DBG_COMMON_FORCE_VALID,
PCIE_REG_DBG_COMMON_FORCE_FRAME,
@@ -1170,22 +1241,57 @@
};
static struct block_defs block_led_defs = {
- "led", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ "led", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
0, 0, 0, 0, 0,
- true, true, DBG_RESET_REG_MISCS_PL_HV, 14
+ true, false, DBG_RESET_REG_MISCS_PL_HV, 14
+};
+
+static struct block_defs block_avs_wrap_defs = {
+ "avs_wrap", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ 0, 0, 0, 0, 0,
+ true, false, DBG_RESET_REG_MISCS_PL_UA, 11
+};
+
+static struct block_defs block_rgfs_defs = {
+ "rgfs", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ 0, 0, 0, 0, 0,
+ false, false, MAX_DBG_RESET_REGS, 0
+};
+
+static struct block_defs block_tgfs_defs = {
+ "tgfs", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ 0, 0, 0, 0, 0,
+ false, false, MAX_DBG_RESET_REGS, 0
+};
+
+static struct block_defs block_ptld_defs = {
+ "ptld", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ 0, 0, 0, 0, 0,
+ false, false, MAX_DBG_RESET_REGS, 0
+};
+
+static struct block_defs block_ypld_defs = {
+ "ypld", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ 0, 0, 0, 0, 0,
+ false, false, MAX_DBG_RESET_REGS, 0
};
static struct block_defs block_misc_aeu_defs = {
- "misc_aeu", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ "misc_aeu", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
0, 0, 0, 0, 0,
false, false, MAX_DBG_RESET_REGS, 0
};
static struct block_defs block_bar0_map_defs = {
- "bar0_map", {false, false, false}, false, 0,
- {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
+ "bar0_map", {false, false}, false, 0,
+ {MAX_DBG_BUS_CLIENTS, MAX_DBG_BUS_CLIENTS},
0, 0, 0, 0, 0,
false, false, MAX_DBG_RESET_REGS, 0
};
@@ -1269,6 +1375,11 @@
&block_ms_defs,
&block_phy_pcie_defs,
&block_led_defs,
+ &block_avs_wrap_defs,
+ &block_rgfs_defs,
+ &block_tgfs_defs,
+ &block_ptld_defs,
+ &block_ypld_defs,
&block_misc_aeu_defs,
&block_bar0_map_defs,
};
@@ -1281,65 +1392,67 @@
};
static struct grc_param_defs s_grc_param_defs[] = {
- {{1, 1, 1}, 0, 1, false, 1, 1}, /* DBG_GRC_PARAM_DUMP_TSTORM */
- {{1, 1, 1}, 0, 1, false, 1, 1}, /* DBG_GRC_PARAM_DUMP_MSTORM */
- {{1, 1, 1}, 0, 1, false, 1, 1}, /* DBG_GRC_PARAM_DUMP_USTORM */
- {{1, 1, 1}, 0, 1, false, 1, 1}, /* DBG_GRC_PARAM_DUMP_XSTORM */
- {{1, 1, 1}, 0, 1, false, 1, 1}, /* DBG_GRC_PARAM_DUMP_YSTORM */
- {{1, 1, 1}, 0, 1, false, 1, 1}, /* DBG_GRC_PARAM_DUMP_PSTORM */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_REGS */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_RAM */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_PBUF */
- {{0, 0, 0}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_IOR */
- {{0, 0, 0}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_VFC */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_CM_CTX */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_ILT */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_RSS */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_CAU */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_QM */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_MCP */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_RESERVED */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_CFC */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_IGU */
- {{0, 0, 0}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_BRB */
- {{0, 0, 0}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_BTB */
- {{0, 0, 0}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_BMB */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_NIG */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_MULD */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_PRS */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_DMAE */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_TM */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_SDM */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_DIF */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_STATIC */
- {{0, 0, 0}, 0, 1, false, 0, 0}, /* DBG_GRC_PARAM_UNSTALL */
- {{MAX_LCIDS, MAX_LCIDS, MAX_LCIDS}, 1, MAX_LCIDS, false, MAX_LCIDS,
+ {{1, 1}, 0, 1, false, 1, 1}, /* DBG_GRC_PARAM_DUMP_TSTORM */
+ {{1, 1}, 0, 1, false, 1, 1}, /* DBG_GRC_PARAM_DUMP_MSTORM */
+ {{1, 1}, 0, 1, false, 1, 1}, /* DBG_GRC_PARAM_DUMP_USTORM */
+ {{1, 1}, 0, 1, false, 1, 1}, /* DBG_GRC_PARAM_DUMP_XSTORM */
+ {{1, 1}, 0, 1, false, 1, 1}, /* DBG_GRC_PARAM_DUMP_YSTORM */
+ {{1, 1}, 0, 1, false, 1, 1}, /* DBG_GRC_PARAM_DUMP_PSTORM */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_REGS */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_RAM */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_PBUF */
+ {{0, 0}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_IOR */
+ {{0, 0}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_VFC */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_CM_CTX */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_ILT */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_RSS */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_CAU */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_QM */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_MCP */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_RESERVED */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_CFC */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_IGU */
+ {{0, 0}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_BRB */
+ {{0, 0}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_BTB */
+ {{0, 0}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_BMB */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_NIG */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_MULD */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_PRS */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_DMAE */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_TM */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_SDM */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_DIF */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_STATIC */
+ {{0, 0}, 0, 1, false, 0, 0}, /* DBG_GRC_PARAM_UNSTALL */
+ {{MAX_LCIDS, MAX_LCIDS}, 1, MAX_LCIDS, false, MAX_LCIDS,
MAX_LCIDS}, /* DBG_GRC_PARAM_NUM_LCIDS */
- {{MAX_LTIDS, MAX_LTIDS, MAX_LTIDS}, 1, MAX_LTIDS, false, MAX_LTIDS,
+ {{MAX_LTIDS, MAX_LTIDS}, 1, MAX_LTIDS, false, MAX_LTIDS,
MAX_LTIDS}, /* DBG_GRC_PARAM_NUM_LTIDS */
- {{0, 0, 0}, 0, 1, true, 0, 0}, /* DBG_GRC_PARAM_EXCLUDE_ALL */
- {{0, 0, 0}, 0, 1, true, 0, 0}, /* DBG_GRC_PARAM_CRASH */
- {{0, 0, 0}, 0, 1, false, 1, 0}, /* DBG_GRC_PARAM_PARITY_SAFE */
- {{1, 1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_CM */
- {{1, 1, 1}, 0, 1, false, 0, 1} /* DBG_GRC_PARAM_DUMP_PHY */
+ {{0, 0}, 0, 1, true, 0, 0}, /* DBG_GRC_PARAM_EXCLUDE_ALL */
+ {{0, 0}, 0, 1, true, 0, 0}, /* DBG_GRC_PARAM_CRASH */
+ {{0, 0}, 0, 1, false, 1, 0}, /* DBG_GRC_PARAM_PARITY_SAFE */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_CM */
+ {{1, 1}, 0, 1, false, 0, 1}, /* DBG_GRC_PARAM_DUMP_PHY */
+ {{0, 0}, 0, 1, false, 0, 0}, /* DBG_GRC_PARAM_NO_MCP */
+ {{0, 0}, 0, 1, false, 0, 0} /* DBG_GRC_PARAM_NO_FW_VER */
};
static struct rss_mem_defs s_rss_mem_defs[] = {
{ "rss_mem_cid", "rss_cid", 0,
- {256, 256, 320},
- {32, 32, 32} },
+ {256, 320},
+ {32, 32} },
{ "rss_mem_key_msb", "rss_key", 1024,
- {128, 128, 208},
- {256, 256, 256} },
+ {128, 208},
+ {256, 256} },
{ "rss_mem_key_lsb", "rss_key", 2048,
- {128, 128, 208},
- {64, 64, 64} },
+ {128, 208},
+ {64, 64} },
{ "rss_mem_info", "rss_info", 3072,
- {128, 128, 208},
- {16, 16, 16} },
+ {128, 208},
+ {16, 16} },
{ "rss_mem_ind", "rss_ind", 4096,
- {(128 * 128), (128 * 128), (128 * 208)},
- {16, 16, 16} }
+ {(128 * 128), (128 * 208)},
+ {16, 16} }
};
static struct vfc_ram_defs s_vfc_ram_defs[] = {
@@ -1352,32 +1465,32 @@
static struct big_ram_defs s_big_ram_defs[] = {
{ "BRB", MEM_GROUP_BRB_MEM, MEM_GROUP_BRB_RAM, DBG_GRC_PARAM_DUMP_BRB,
BRB_REG_BIG_RAM_ADDRESS, BRB_REG_BIG_RAM_DATA,
- {4800, 4800, 5632} },
+ {4800, 5632} },
{ "BTB", MEM_GROUP_BTB_MEM, MEM_GROUP_BTB_RAM, DBG_GRC_PARAM_DUMP_BTB,
BTB_REG_BIG_RAM_ADDRESS, BTB_REG_BIG_RAM_DATA,
- {2880, 2880, 3680} },
+ {2880, 3680} },
{ "BMB", MEM_GROUP_BMB_MEM, MEM_GROUP_BMB_RAM, DBG_GRC_PARAM_DUMP_BMB,
BMB_REG_BIG_RAM_ADDRESS, BMB_REG_BIG_RAM_DATA,
- {1152, 1152, 1152} }
+ {1152, 1152} }
};
static struct reset_reg_defs s_reset_regs_defs[] = {
{ MISCS_REG_RESET_PL_UA, 0x0,
- {true, true, true} }, /* DBG_RESET_REG_MISCS_PL_UA */
+ {true, true} }, /* DBG_RESET_REG_MISCS_PL_UA */
{ MISCS_REG_RESET_PL_HV, 0x0,
- {true, true, true} }, /* DBG_RESET_REG_MISCS_PL_HV */
+ {true, true} }, /* DBG_RESET_REG_MISCS_PL_HV */
{ MISCS_REG_RESET_PL_HV_2, 0x0,
- {false, false, true} }, /* DBG_RESET_REG_MISCS_PL_HV_2 */
+ {false, true} }, /* DBG_RESET_REG_MISCS_PL_HV_2 */
{ MISC_REG_RESET_PL_UA, 0x0,
- {true, true, true} }, /* DBG_RESET_REG_MISC_PL_UA */
+ {true, true} }, /* DBG_RESET_REG_MISC_PL_UA */
{ MISC_REG_RESET_PL_HV, 0x0,
- {true, true, true} }, /* DBG_RESET_REG_MISC_PL_HV */
+ {true, true} }, /* DBG_RESET_REG_MISC_PL_HV */
{ MISC_REG_RESET_PL_PDA_VMAIN_1, 0x4404040,
- {true, true, true} }, /* DBG_RESET_REG_MISC_PL_PDA_VMAIN_1 */
+ {true, true} }, /* DBG_RESET_REG_MISC_PL_PDA_VMAIN_1 */
{ MISC_REG_RESET_PL_PDA_VMAIN_2, 0x7c00007,
- {true, true, true} }, /* DBG_RESET_REG_MISC_PL_PDA_VMAIN_2 */
+ {true, true} }, /* DBG_RESET_REG_MISC_PL_PDA_VMAIN_2 */
{ MISC_REG_RESET_PL_PDA_VAUX, 0x2,
- {true, true, true} }, /* DBG_RESET_REG_MISC_PL_PDA_VAUX */
+ {true, true} }, /* DBG_RESET_REG_MISC_PL_PDA_VAUX */
};
static struct phy_defs s_phy_defs[] = {
@@ -1410,6 +1523,26 @@
return dword;
}
+/* Returns the value of the specified GRC param */
+static u32 qed_grc_get_param(struct qed_hwfn *p_hwfn,
+ enum dbg_grc_params grc_param)
+{
+ struct dbg_tools_data *dev_data = &p_hwfn->dbg_info;
+
+ return dev_data->grc.param_val[grc_param];
+}
+
+/* Initializes the GRC parameters */
+static void qed_dbg_grc_init_params(struct qed_hwfn *p_hwfn)
+{
+ struct dbg_tools_data *dev_data = &p_hwfn->dbg_info;
+
+ if (!dev_data->grc.params_initialized) {
+ qed_dbg_grc_set_params_default(p_hwfn);
+ dev_data->grc.params_initialized = 1;
+ }
+}
+
/* Initializes debug data for the specified device */
static enum dbg_status qed_dbg_dev_init(struct qed_hwfn *p_hwfn,
struct qed_ptt *p_ptt)
@@ -1424,13 +1557,17 @@
dev_data->mode_enable[MODE_K2] = 1;
} else if (QED_IS_BB_B0(p_hwfn->cdev)) {
dev_data->chip_id = CHIP_BB_B0;
- dev_data->mode_enable[MODE_BB_B0] = 1;
+ dev_data->mode_enable[MODE_BB] = 1;
} else {
return DBG_STATUS_UNKNOWN_CHIP;
}
dev_data->platform_id = PLATFORM_ASIC;
dev_data->mode_enable[MODE_ASIC] = 1;
+
+ /* Initializes the GRC parameters */
+ qed_dbg_grc_init_params(p_hwfn);
+
dev_data->initialized = true;
return DBG_STATUS_OK;
}
@@ -1561,7 +1698,7 @@
int printed_chars;
u32 offset = 0;
- if (dump) {
+ if (dump && !qed_grc_get_param(p_hwfn, DBG_GRC_PARAM_NO_FW_VER)) {
/* Read FW image/version from PRAM in a non-reset SEMI */
bool found = false;
u8 storm_id;
@@ -1622,7 +1759,7 @@
{
char mfw_ver_str[16] = EMPTY_FW_VERSION_STR;
- if (dump) {
+ if (dump && !qed_grc_get_param(p_hwfn, DBG_GRC_PARAM_NO_FW_VER)) {
u32 global_section_offsize, global_section_addr, mfw_ver;
u32 public_data_addr, global_section_offsize_addr;
int printed_chars;
@@ -1683,15 +1820,13 @@
bool dump,
u8 num_specific_global_params)
{
+ u8 num_params = NUM_COMMON_GLOBAL_PARAMS + num_specific_global_params;
struct dbg_tools_data *dev_data = &p_hwfn->dbg_info;
u32 offset = 0;
/* Find platform string and dump global params section header */
offset += qed_dump_section_hdr(dump_buf + offset,
- dump,
- "global_params",
- NUM_COMMON_GLOBAL_PARAMS +
- num_specific_global_params);
+ dump, "global_params", num_params);
/* Store params */
offset += qed_dump_fw_ver_param(p_hwfn, p_ptt, dump_buf + offset, dump);
@@ -1815,37 +1950,6 @@
}
}
-/* Returns the value of the specified GRC param */
-static u32 qed_grc_get_param(struct qed_hwfn *p_hwfn,
- enum dbg_grc_params grc_param)
-{
- struct dbg_tools_data *dev_data = &p_hwfn->dbg_info;
-
- return dev_data->grc.param_val[grc_param];
-}
-
-/* Clear all GRC params */
-static void qed_dbg_grc_clear_params(struct qed_hwfn *p_hwfn)
-{
- struct dbg_tools_data *dev_data = &p_hwfn->dbg_info;
- u32 i;
-
- for (i = 0; i < MAX_DBG_GRC_PARAMS; i++)
- dev_data->grc.param_set_by_user[i] = 0;
-}
-
-/* Assign default GRC param values */
-static void qed_dbg_grc_set_params_default(struct qed_hwfn *p_hwfn)
-{
- struct dbg_tools_data *dev_data = &p_hwfn->dbg_info;
- u32 i;
-
- for (i = 0; i < MAX_DBG_GRC_PARAMS; i++)
- if (!dev_data->grc.param_set_by_user[i])
- dev_data->grc.param_val[i] =
- s_grc_param_defs[i].default_val[dev_data->chip_id];
-}
-
/* Returns true if the specified entity (indicated by GRC param) should be
* included in the dump, false otherwise.
*/
@@ -1971,7 +2075,7 @@
}
}
-/* Returns the attention name offsets of the specified block */
+/* Returns the attention block data of the specified block */
static const struct dbg_attn_block_type_data *
qed_get_block_attn_data(enum block_id block_id, enum dbg_attn_type attn_type)
{
@@ -2040,7 +2144,7 @@
* The following parameters are dumped:
* - 'count' = num_dumped_entries
* - 'split' = split_type
- * - 'id'i = split_id (dumped only if split_id >= 0)
+ * - 'id' = split_id (dumped only if split_id >= 0)
* - 'param_name' = param_val (user param, dumped only if param_name != NULL and
* param_val != NULL)
*/
@@ -2069,21 +2173,81 @@
return offset;
}
-/* Dumps GRC register/memory. Returns the dumped size in dwords. */
+/* Dumps the GRC registers in the specified address range.
+ * Returns the dumped size in dwords.
+ */
+static u32 qed_grc_dump_addr_range(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt, u32 *dump_buf,
+ bool dump, u32 addr, u32 len)
+{
+ u32 byte_addr = DWORDS_TO_BYTES(addr), offset = 0, i;
+
+ if (dump)
+ for (i = 0; i < len; i++, byte_addr += BYTES_IN_DWORD, offset++)
+ *(dump_buf + offset) = qed_rd(p_hwfn, p_ptt, byte_addr);
+ else
+ offset += len;
+ return offset;
+}
+
+/* Dumps GRC registers sequence header. Returns the dumped size in dwords. */
+static u32 qed_grc_dump_reg_entry_hdr(u32 *dump_buf, bool dump, u32 addr,
+ u32 len)
+{
+ if (dump)
+ *dump_buf = addr | (len << REG_DUMP_LEN_SHIFT);
+ return 1;
+}
+
+/* Dumps GRC registers sequence. Returns the dumped size in dwords. */
static u32 qed_grc_dump_reg_entry(struct qed_hwfn *p_hwfn,
struct qed_ptt *p_ptt, u32 *dump_buf,
bool dump, u32 addr, u32 len)
{
- u32 offset = 0, i;
+ u32 offset = 0;
+ offset += qed_grc_dump_reg_entry_hdr(dump_buf, dump, addr, len);
+ offset += qed_grc_dump_addr_range(p_hwfn,
+ p_ptt,
+ dump_buf + offset, dump, addr, len);
+ return offset;
+}
+
+/* Dumps GRC registers sequence with skip cycle.
+ * Returns the dumped size in dwords.
+ */
+static u32 qed_grc_dump_reg_entry_skip(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt, u32 *dump_buf,
+ bool dump, u32 addr, u32 total_len,
+ u32 read_len, u32 skip_len)
+{
+ u32 offset = 0, reg_offset = 0;
+
+ offset += qed_grc_dump_reg_entry_hdr(dump_buf, dump, addr, total_len);
if (dump) {
- *(dump_buf + offset++) = addr | (len << REG_DUMP_LEN_SHIFT);
- for (i = 0; i < len; i++, addr++, offset++)
- *(dump_buf + offset) = qed_rd(p_hwfn,
- p_ptt,
- DWORDS_TO_BYTES(addr));
+ while (reg_offset < total_len) {
+ u32 curr_len = min_t(u32,
+ read_len,
+ total_len - reg_offset);
+ offset += qed_grc_dump_addr_range(p_hwfn,
+ p_ptt,
+ dump_buf + offset,
+ dump, addr, curr_len);
+ reg_offset += curr_len;
+ addr += curr_len;
+ if (reg_offset < total_len) {
+ curr_len = min_t(u32,
+ skip_len,
+ total_len - skip_len);
+ memset(dump_buf + offset, 0,
+ DWORDS_TO_BYTES(curr_len));
+ offset += curr_len;
+ reg_offset += curr_len;
+ addr += curr_len;
+ }
+ }
} else {
- offset += len + 1;
+ offset += total_len;
}
return offset;
@@ -2124,14 +2288,17 @@
const struct dbg_dump_reg *reg =
(const struct dbg_dump_reg *)
&input_regs_arr.ptr[input_offset];
+ u32 addr, len;
+ addr = GET_FIELD(reg->data,
+ DBG_DUMP_REG_ADDRESS);
+ len = GET_FIELD(reg->data, DBG_DUMP_REG_LENGTH);
offset +=
- qed_grc_dump_reg_entry(p_hwfn, p_ptt,
- dump_buf + offset, dump,
- GET_FIELD(reg->data,
- DBG_DUMP_REG_ADDRESS),
- GET_FIELD(reg->data,
- DBG_DUMP_REG_LENGTH));
+ qed_grc_dump_reg_entry(p_hwfn, p_ptt,
+ dump_buf + offset,
+ dump,
+ addr,
+ len);
(*num_dumped_reg_entries)++;
}
} else {
@@ -2194,8 +2361,14 @@
const char *param_name, const char *param_val)
{
struct dbg_tools_data *dev_data = &p_hwfn->dbg_info;
+ struct chip_platform_defs *p_platform_defs;
u32 offset = 0, input_offset = 0;
- u8 port_id, pf_id;
+ struct chip_defs *p_chip_defs;
+ u8 port_id, pf_id, vf_id;
+ u16 fid;
+
+ p_chip_defs = &s_chip_defs[dev_data->chip_id];
+ p_platform_defs = &p_chip_defs->per_platform[dev_data->platform_id];
if (dump)
DP_VERBOSE(p_hwfn, QED_MSG_DEBUG, "Dumping registers...\n");
@@ -2214,7 +2387,6 @@
switch (split_type_id) {
case SPLIT_TYPE_NONE:
- case SPLIT_TYPE_VF:
offset += qed_grc_dump_split_data(p_hwfn,
p_ptt,
curr_input_regs_arr,
@@ -2227,10 +2399,7 @@
param_val);
break;
case SPLIT_TYPE_PORT:
- for (port_id = 0;
- port_id <
- s_chip_defs[dev_data->chip_id].
- per_platform[dev_data->platform_id].num_ports;
+ for (port_id = 0; port_id < p_platform_defs->num_ports;
port_id++) {
if (dump)
qed_port_pretend(p_hwfn, p_ptt,
@@ -2247,20 +2416,48 @@
break;
case SPLIT_TYPE_PF:
case SPLIT_TYPE_PORT_PF:
- for (pf_id = 0;
- pf_id <
- s_chip_defs[dev_data->chip_id].
- per_platform[dev_data->platform_id].num_pfs;
+ for (pf_id = 0; pf_id < p_platform_defs->num_pfs;
pf_id++) {
- if (dump)
- qed_fid_pretend(p_hwfn, p_ptt, pf_id);
- offset += qed_grc_dump_split_data(p_hwfn,
- p_ptt,
- curr_input_regs_arr,
- dump_buf + offset,
- dump, block_enable,
- "pf", pf_id, param_name,
- param_val);
+ u8 pfid_shift =
+ PXP_PRETEND_CONCRETE_FID_PFID_SHIFT;
+
+ if (dump) {
+ fid = pf_id << pfid_shift;
+ qed_fid_pretend(p_hwfn, p_ptt, fid);
+ }
+
+ offset +=
+ qed_grc_dump_split_data(p_hwfn, p_ptt,
+ curr_input_regs_arr,
+ dump_buf + offset,
+ dump, block_enable,
+ "pf", pf_id,
+ param_name,
+ param_val);
+ }
+ break;
+ case SPLIT_TYPE_VF:
+ for (vf_id = 0; vf_id < p_platform_defs->num_vfs;
+ vf_id++) {
+ u8 vfvalid_shift =
+ PXP_PRETEND_CONCRETE_FID_VFVALID_SHIFT;
+ u8 vfid_shift =
+ PXP_PRETEND_CONCRETE_FID_VFID_SHIFT;
+
+ if (dump) {
+ fid = BIT(vfvalid_shift) |
+ (vf_id << vfid_shift);
+ qed_fid_pretend(p_hwfn, p_ptt, fid);
+ }
+
+ offset +=
+ qed_grc_dump_split_data(p_hwfn, p_ptt,
+ curr_input_regs_arr,
+ dump_buf + offset,
+ dump, block_enable,
+ "vf", vf_id,
+ param_name,
+ param_val);
}
break;
default:
@@ -2271,8 +2468,11 @@
}
/* Pretend to original PF */
- if (dump)
- qed_fid_pretend(p_hwfn, p_ptt, p_hwfn->rel_pf_id);
+ if (dump) {
+ fid = p_hwfn->rel_pf_id << PXP_PRETEND_CONCRETE_FID_PFID_SHIFT;
+ qed_fid_pretend(p_hwfn, p_ptt, fid);
+ }
+
return offset;
}
@@ -2291,13 +2491,14 @@
/* Write reset registers */
for (i = 0; i < MAX_DBG_RESET_REGS; i++) {
if (s_reset_regs_defs[i].exists[dev_data->chip_id]) {
+ u32 addr = BYTES_TO_DWORDS(s_reset_regs_defs[i].addr);
+
offset += qed_grc_dump_reg_entry(p_hwfn,
p_ptt,
dump_buf + offset,
dump,
- BYTES_TO_DWORDS
- (s_reset_regs_defs
- [i].addr), 1);
+ addr,
+ 1);
num_regs++;
}
}
@@ -2339,6 +2540,7 @@
&attn_reg_arr[reg_idx];
u16 modes_buf_offset;
bool eval_mode;
+ u32 addr;
/* Check mode */
eval_mode = GET_FIELD(reg_data->mode.data,
@@ -2349,19 +2551,23 @@
if (!eval_mode ||
qed_is_mode_match(p_hwfn, &modes_buf_offset)) {
/* Mode match - read and dump registers */
- offset += qed_grc_dump_reg_entry(p_hwfn,
- p_ptt,
- dump_buf + offset,
- dump,
- reg_data->mask_address,
- 1);
- offset += qed_grc_dump_reg_entry(p_hwfn,
- p_ptt,
- dump_buf + offset,
- dump,
- GET_FIELD(reg_data->data,
- DBG_ATTN_REG_STS_ADDRESS),
- 1);
+ addr = reg_data->mask_address;
+ offset +=
+ qed_grc_dump_reg_entry(p_hwfn,
+ p_ptt,
+ dump_buf + offset,
+ dump,
+ addr,
+ 1);
+ addr = GET_FIELD(reg_data->data,
+ DBG_ATTN_REG_STS_ADDRESS);
+ offset +=
+ qed_grc_dump_reg_entry(p_hwfn,
+ p_ptt,
+ dump_buf + offset,
+ dump,
+ addr,
+ 1);
num_reg_entries += 2;
}
}
@@ -2369,18 +2575,21 @@
/* Write storm stall status registers */
for (storm_id = 0; storm_id < MAX_DBG_STORMS; storm_id++) {
+ u32 addr;
+
if (dev_data->block_in_reset[s_storm_defs[storm_id].block_id] &&
dump)
continue;
+ addr =
+ BYTES_TO_DWORDS(s_storm_defs[storm_id].sem_fast_mem_addr +
+ SEM_FAST_REG_STALLED);
offset += qed_grc_dump_reg_entry(p_hwfn,
- p_ptt,
- dump_buf + offset,
- dump,
- BYTES_TO_DWORDS(s_storm_defs[storm_id].
- sem_fast_mem_addr +
- SEM_FAST_REG_STALLED),
- 1);
+ p_ptt,
+ dump_buf + offset,
+ dump,
+ addr,
+ 1);
num_reg_entries++;
}
@@ -2392,11 +2601,47 @@
return offset;
}
+/* Dumps registers that can't be represented in the debug arrays */
+static u32 qed_grc_dump_special_regs(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ u32 *dump_buf, bool dump)
+{
+ u32 offset = 0, addr;
+
+ offset += qed_grc_dump_regs_hdr(dump_buf,
+ dump, 2, "eng", -1, NULL, NULL);
+
+ /* Dump R/TDIF_REG_DEBUG_ERROR_INFO_SIZE (every 8'th register should be
+ * skipped).
+ */
+ addr = BYTES_TO_DWORDS(RDIF_REG_DEBUG_ERROR_INFO);
+ offset += qed_grc_dump_reg_entry_skip(p_hwfn,
+ p_ptt,
+ dump_buf + offset,
+ dump,
+ addr,
+ RDIF_REG_DEBUG_ERROR_INFO_SIZE,
+ 7,
+ 1);
+ addr = BYTES_TO_DWORDS(TDIF_REG_DEBUG_ERROR_INFO);
+ offset +=
+ qed_grc_dump_reg_entry_skip(p_hwfn,
+ p_ptt,
+ dump_buf + offset,
+ dump,
+ addr,
+ TDIF_REG_DEBUG_ERROR_INFO_SIZE,
+ 7,
+ 1);
+
+ return offset;
+}
+
/* Dumps a GRC memory header (section and params).
* The following parameters are dumped:
* name - name is dumped only if it's not NULL.
- * addr - byte_addr is dumped only if name is NULL.
- * len - dword_len is always dumped.
+ * addr - addr is dumped only if name is NULL.
+ * len - len is always dumped.
* width - bit_width is dumped if it's not zero.
* packed - packed=1 is dumped if it's not false.
* mem_group - mem_group is always dumped.
@@ -2408,8 +2653,8 @@
u32 *dump_buf,
bool dump,
const char *name,
- u32 byte_addr,
- u32 dword_len,
+ u32 addr,
+ u32 len,
u32 bit_width,
bool packed,
const char *mem_group,
@@ -2419,7 +2664,7 @@
u32 offset = 0;
char buf[64];
- if (!dword_len)
+ if (!len)
DP_NOTICE(p_hwfn,
"Unexpected GRC Dump error: dumped memory size must be non-zero\n");
if (bit_width)
@@ -2446,20 +2691,21 @@
DP_VERBOSE(p_hwfn,
QED_MSG_DEBUG,
"Dumping %d registers from %s...\n",
- dword_len, buf);
+ len, buf);
} else {
/* Dump address */
offset += qed_dump_num_param(dump_buf + offset,
- dump, "addr", byte_addr);
- if (dump && dword_len > 64)
+ dump, "addr",
+ DWORDS_TO_BYTES(addr));
+ if (dump && len > 64)
DP_VERBOSE(p_hwfn,
QED_MSG_DEBUG,
"Dumping %d registers from address 0x%x...\n",
- dword_len, byte_addr);
+ len, (u32)DWORDS_TO_BYTES(addr));
}
/* Dump len */
- offset += qed_dump_num_param(dump_buf + offset, dump, "len", dword_len);
+ offset += qed_dump_num_param(dump_buf + offset, dump, "len", len);
/* Dump bit width */
if (bit_width)
@@ -2492,8 +2738,8 @@
u32 *dump_buf,
bool dump,
const char *name,
- u32 byte_addr,
- u32 dword_len,
+ u32 addr,
+ u32 len,
u32 bit_width,
bool packed,
const char *mem_group,
@@ -2505,21 +2751,14 @@
dump_buf + offset,
dump,
name,
- byte_addr,
- dword_len,
+ addr,
+ len,
bit_width,
packed,
mem_group, is_storm, storm_letter);
- if (dump) {
- u32 i;
-
- for (i = 0; i < dword_len;
- i++, byte_addr += BYTES_IN_DWORD, offset++)
- *(dump_buf + offset) = qed_rd(p_hwfn, p_ptt, byte_addr);
- } else {
- offset += dword_len;
- }
-
+ offset += qed_grc_dump_addr_range(p_hwfn,
+ p_ptt,
+ dump_buf + offset, dump, addr, len);
return offset;
}
@@ -2575,25 +2814,41 @@
if (qed_grc_is_mem_included(p_hwfn,
(enum block_id)cond_hdr->block_id,
mem_group_id)) {
- u32 mem_byte_addr =
- DWORDS_TO_BYTES(GET_FIELD(mem->dword0,
- DBG_DUMP_MEM_ADDRESS));
+ u32 mem_addr = GET_FIELD(mem->dword0,
+ DBG_DUMP_MEM_ADDRESS);
u32 mem_len = GET_FIELD(mem->dword1,
DBG_DUMP_MEM_LENGTH);
+ enum dbg_grc_params grc_param;
char storm_letter = 'a';
bool is_storm = false;
/* Update memory length for CCFC/TCFC memories
* according to number of LCIDs/LTIDs.
*/
- if (mem_group_id == MEM_GROUP_CONN_CFC_MEM)
+ if (mem_group_id == MEM_GROUP_CONN_CFC_MEM) {
+ if (mem_len % MAX_LCIDS != 0) {
+ DP_NOTICE(p_hwfn,
+ "Invalid CCFC connection memory size\n");
+ return 0;
+ }
+
+ grc_param = DBG_GRC_PARAM_NUM_LCIDS;
mem_len = qed_grc_get_param(p_hwfn,
- DBG_GRC_PARAM_NUM_LCIDS)
- * (mem_len / MAX_LCIDS);
- else if (mem_group_id == MEM_GROUP_TASK_CFC_MEM)
+ grc_param) *
+ (mem_len / MAX_LCIDS);
+ } else if (mem_group_id ==
+ MEM_GROUP_TASK_CFC_MEM) {
+ if (mem_len % MAX_LTIDS != 0) {
+ DP_NOTICE(p_hwfn,
+ "Invalid TCFC task memory size\n");
+ return 0;
+ }
+
+ grc_param = DBG_GRC_PARAM_NUM_LTIDS;
mem_len = qed_grc_get_param(p_hwfn,
- DBG_GRC_PARAM_NUM_LTIDS)
- * (mem_len / MAX_LTIDS);
+ grc_param) *
+ (mem_len / MAX_LTIDS);
+ }
/* If memory is associated with Storm, update
* Storm details.
@@ -2610,7 +2865,7 @@
/* Dump memory */
offset += qed_grc_dump_mem(p_hwfn, p_ptt,
dump_buf + offset, dump, NULL,
- mem_byte_addr, mem_len, 0,
+ mem_addr, mem_len, 0,
false,
s_mem_group_names[mem_group_id],
is_storm, storm_letter);
@@ -2799,29 +3054,31 @@
u32 offset = 0;
for (storm_id = 0; storm_id < MAX_DBG_STORMS; storm_id++) {
- if (qed_grc_is_storm_included(p_hwfn,
- (enum dbg_storms)storm_id)) {
- for (set_id = 0; set_id < NUM_IOR_SETS; set_id++) {
- u32 addr =
- s_storm_defs[storm_id].sem_fast_mem_addr +
- SEM_FAST_REG_STORM_REG_FILE +
- DWORDS_TO_BYTES(IOR_SET_OFFSET(set_id));
+ struct storm_defs *storm = &s_storm_defs[storm_id];
- buf[strlen(buf) - 1] = '0' + set_id;
- offset += qed_grc_dump_mem(p_hwfn,
- p_ptt,
- dump_buf + offset,
- dump,
- buf,
- addr,
- IORS_PER_SET,
- 32,
- false,
- "ior",
- true,
- s_storm_defs
- [storm_id].letter);
- }
+ if (!qed_grc_is_storm_included(p_hwfn,
+ (enum dbg_storms)storm_id))
+ continue;
+
+ for (set_id = 0; set_id < NUM_IOR_SETS; set_id++) {
+ u32 dwords, addr;
+
+ dwords = storm->sem_fast_mem_addr +
+ SEM_FAST_REG_STORM_REG_FILE;
+ addr = BYTES_TO_DWORDS(dwords) + IOR_SET_OFFSET(set_id);
+ buf[strlen(buf) - 1] = '0' + set_id;
+ offset += qed_grc_dump_mem(p_hwfn,
+ p_ptt,
+ dump_buf + offset,
+ dump,
+ buf,
+ addr,
+ IORS_PER_SET,
+ 32,
+ false,
+ "ior",
+ true,
+ storm->letter);
}
}
@@ -2990,34 +3247,39 @@
struct rss_mem_defs *rss_defs = &s_rss_mem_defs[rss_mem_id];
u32 num_entries = rss_defs->num_entries[dev_data->chip_id];
u32 entry_width = rss_defs->entry_width[dev_data->chip_id];
- u32 total_size = (num_entries * entry_width) / 32;
+ u32 total_dwords = (num_entries * entry_width) / 32;
+ u32 size = RSS_REG_RSS_RAM_DATA_SIZE;
bool packed = (entry_width == 16);
- u32 addr = rss_defs->addr;
- u32 i, j;
+ u32 rss_addr = rss_defs->addr;
+ u32 i, addr;
offset += qed_grc_dump_mem_hdr(p_hwfn,
dump_buf + offset,
dump,
rss_defs->mem_name,
- addr,
- total_size,
+ 0,
+ total_dwords,
entry_width,
packed,
rss_defs->type_name, false, 0);
if (!dump) {
- offset += total_size;
+ offset += total_dwords;
continue;
}
/* Dump RSS data */
- for (i = 0; i < BYTES_TO_DWORDS(total_size); i++, addr++) {
- qed_wr(p_hwfn, p_ptt, RSS_REG_RSS_RAM_ADDR, addr);
- for (j = 0; j < BYTES_IN_DWORD; j++, offset++)
- *(dump_buf + offset) =
- qed_rd(p_hwfn, p_ptt,
- RSS_REG_RSS_RAM_DATA +
- DWORDS_TO_BYTES(j));
+ for (i = 0; i < total_dwords;
+ i += RSS_REG_RSS_RAM_DATA_SIZE, rss_addr++) {
+ addr = BYTES_TO_DWORDS(RSS_REG_RSS_RAM_DATA);
+ qed_wr(p_hwfn, p_ptt, RSS_REG_RSS_RAM_ADDR, rss_addr);
+ offset += qed_grc_dump_addr_range(p_hwfn,
+ p_ptt,
+ dump_buf +
+ offset,
+ dump,
+ addr,
+ size);
}
}
@@ -3030,19 +3292,19 @@
u32 *dump_buf, bool dump, u8 big_ram_id)
{
struct dbg_tools_data *dev_data = &p_hwfn->dbg_info;
+ u32 total_blocks, ram_size, offset = 0, i;
char mem_name[12] = "???_BIG_RAM";
char type_name[8] = "???_RAM";
- u32 ram_size, total_blocks;
- u32 offset = 0, i, j;
+ struct big_ram_defs *big_ram;
- total_blocks =
- s_big_ram_defs[big_ram_id].num_of_blocks[dev_data->chip_id];
+ big_ram = &s_big_ram_defs[big_ram_id];
+ total_blocks = big_ram->num_of_blocks[dev_data->chip_id];
ram_size = total_blocks * BIG_RAM_BLOCK_SIZE_DWORDS;
- strncpy(type_name, s_big_ram_defs[big_ram_id].instance_name,
- strlen(s_big_ram_defs[big_ram_id].instance_name));
- strncpy(mem_name, s_big_ram_defs[big_ram_id].instance_name,
- strlen(s_big_ram_defs[big_ram_id].instance_name));
+ strncpy(type_name, big_ram->instance_name,
+ strlen(big_ram->instance_name));
+ strncpy(mem_name, big_ram->instance_name,
+ strlen(big_ram->instance_name));
/* Dump memory header */
offset += qed_grc_dump_mem_hdr(p_hwfn,
@@ -3059,13 +3321,17 @@
/* Read and dump Big RAM data */
for (i = 0; i < total_blocks / 2; i++) {
- qed_wr(p_hwfn, p_ptt, s_big_ram_defs[big_ram_id].addr_reg_addr,
- i);
- for (j = 0; j < 2 * BIG_RAM_BLOCK_SIZE_DWORDS; j++, offset++)
- *(dump_buf + offset) = qed_rd(p_hwfn, p_ptt,
- s_big_ram_defs[big_ram_id].
- data_reg_addr +
- DWORDS_TO_BYTES(j));
+ u32 addr, len;
+
+ qed_wr(p_hwfn, p_ptt, big_ram->addr_reg_addr, i);
+ addr = BYTES_TO_DWORDS(big_ram->data_reg_addr);
+ len = 2 * BIG_RAM_BLOCK_SIZE_DWORDS;
+ offset += qed_grc_dump_addr_range(p_hwfn,
+ p_ptt,
+ dump_buf + offset,
+ dump,
+ addr,
+ len);
}
return offset;
@@ -3075,11 +3341,11 @@
struct qed_ptt *p_ptt, u32 *dump_buf, bool dump)
{
bool block_enable[MAX_BLOCK_ID] = { 0 };
+ u32 offset = 0, addr;
bool halted = false;
- u32 offset = 0;
/* Halt MCP */
- if (dump) {
+ if (dump && !qed_grc_get_param(p_hwfn, DBG_GRC_PARAM_NO_MCP)) {
halted = !qed_mcp_halt(p_hwfn, p_ptt);
if (!halted)
DP_NOTICE(p_hwfn, "MCP halt failed!\n");
@@ -3091,7 +3357,7 @@
dump_buf + offset,
dump,
NULL,
- MCP_REG_SCRATCH,
+ BYTES_TO_DWORDS(MCP_REG_SCRATCH),
MCP_REG_SCRATCH_SIZE,
0, false, "MCP", false, 0);
@@ -3101,7 +3367,7 @@
dump_buf + offset,
dump,
NULL,
- MCP_REG_CPU_REG_FILE,
+ BYTES_TO_DWORDS(MCP_REG_CPU_REG_FILE),
MCP_REG_CPU_REG_FILE_SIZE,
0, false, "MCP", false, 0);
@@ -3115,12 +3381,13 @@
/* Dump required non-MCP registers */
offset += qed_grc_dump_regs_hdr(dump_buf + offset,
dump, 1, "eng", -1, "block", "MCP");
+ addr = BYTES_TO_DWORDS(MISC_REG_SHARED_MEM_ADDR);
offset += qed_grc_dump_reg_entry(p_hwfn,
p_ptt,
dump_buf + offset,
dump,
- BYTES_TO_DWORDS
- (MISC_REG_SHARED_MEM_ADDR), 1);
+ addr,
+ 1);
/* Release MCP */
if (halted && qed_mcp_resume(p_hwfn, p_ptt))
@@ -3212,7 +3479,7 @@
{
u32 block_dwords = NUM_DBG_BUS_LINES * STATIC_DEBUG_LINE_DWORDS;
struct dbg_tools_data *dev_data = &p_hwfn->dbg_info;
- u32 offset = 0, block_id, line_id, addr, i;
+ u32 offset = 0, block_id, line_id;
struct block_defs *p_block_defs;
if (dump) {
@@ -3255,6 +3522,8 @@
if (dump && !dev_data->block_in_reset[block_id]) {
u8 dbg_client_id =
p_block_defs->dbg_client_id[dev_data->chip_id];
+ u32 addr = BYTES_TO_DWORDS(DBG_REG_CALENDAR_OUT_DATA);
+ u32 len = STATIC_DEBUG_LINE_DWORDS;
/* Enable block's client */
qed_bus_enable_clients(p_hwfn, p_ptt,
@@ -3270,11 +3539,13 @@
0xf, 0, 0, 0);
/* Read debug line info */
- for (i = 0, addr = DBG_REG_CALENDAR_OUT_DATA;
- i < STATIC_DEBUG_LINE_DWORDS;
- i++, offset++, addr += BYTES_IN_DWORD)
- dump_buf[offset] = qed_rd(p_hwfn, p_ptt,
- addr);
+ offset +=
+ qed_grc_dump_addr_range(p_hwfn,
+ p_ptt,
+ dump_buf + offset,
+ dump,
+ addr,
+ len);
}
/* Disable block's client and debug output */
@@ -3311,14 +3582,8 @@
u8 i, port_mode = 0;
u32 offset = 0;
- /* Check if emulation platform */
*num_dumped_dwords = 0;
- /* Fill GRC parameters that were not set by the user with their default
- * value.
- */
- qed_dbg_grc_set_params_default(p_hwfn);
-
/* Find port mode */
if (dump) {
switch (qed_rd(p_hwfn, p_ptt, MISC_REG_PORT_MODE)) {
@@ -3370,15 +3635,14 @@
}
/* Disable all parities using MFW command */
- if (dump) {
+ if (dump && !qed_grc_get_param(p_hwfn, DBG_GRC_PARAM_NO_MCP)) {
parities_masked = !qed_mcp_mask_parities(p_hwfn, p_ptt, 1);
if (!parities_masked) {
+ DP_NOTICE(p_hwfn,
+ "Failed to mask parities using MFW\n");
if (qed_grc_get_param
(p_hwfn, DBG_GRC_PARAM_PARITY_SAFE))
return DBG_STATUS_MCP_COULD_NOT_MASK_PRTY;
- else
- DP_NOTICE(p_hwfn,
- "Failed to mask parities using MFW\n");
}
}
@@ -3409,6 +3673,11 @@
offset,
dump,
block_enable, NULL, NULL);
+
+ /* Dump special registers */
+ offset += qed_grc_dump_special_regs(p_hwfn,
+ p_ptt,
+ dump_buf + offset, dump);
}
/* Dump memories */
@@ -3583,9 +3852,9 @@
}
if (mode_match) {
- u32 grc_addr =
- DWORDS_TO_BYTES(GET_FIELD(reg->data,
- DBG_IDLE_CHK_INFO_REG_ADDRESS));
+ u32 addr =
+ GET_FIELD(reg->data,
+ DBG_IDLE_CHK_INFO_REG_ADDRESS);
/* Write register header */
struct dbg_idle_chk_result_reg_hdr *reg_hdr =
@@ -3597,16 +3866,19 @@
memset(reg_hdr, 0, sizeof(*reg_hdr));
reg_hdr->size = reg->size;
SET_FIELD(reg_hdr->data,
- DBG_IDLE_CHK_RESULT_REG_HDR_REG_ID,
- rule->num_cond_regs + reg_id);
+ DBG_IDLE_CHK_RESULT_REG_HDR_REG_ID,
+ rule->num_cond_regs + reg_id);
/* Write register values */
- for (i = 0; i < reg->size;
- i++, offset++, grc_addr += 4)
- dump_buf[offset] =
- qed_rd(p_hwfn, p_ptt, grc_addr);
- }
+ offset +=
+ qed_grc_dump_addr_range(p_hwfn,
+ p_ptt,
+ dump_buf + offset,
+ dump,
+ addr,
+ reg->size);
}
+ }
}
return offset;
@@ -3621,7 +3893,7 @@
{
struct dbg_tools_data *dev_data = &p_hwfn->dbg_info;
u32 cond_reg_values[IDLE_CHK_MAX_ENTRIES_SIZE];
- u32 i, j, offset = 0;
+ u32 i, offset = 0;
u16 entry_id;
u8 reg_id;
@@ -3664,73 +3936,83 @@
if (!check_rule && dump)
continue;
+ if (!dump) {
+ u32 entry_dump_size =
+ qed_idle_chk_dump_failure(p_hwfn,
+ p_ptt,
+ dump_buf + offset,
+ false,
+ rule->rule_id,
+ rule,
+ 0,
+ NULL);
+
+ offset += num_reg_entries * entry_dump_size;
+ (*num_failing_rules) += num_reg_entries;
+ continue;
+ }
+
/* Go over all register entries (number of entries is the same
* for all condition registers).
*/
for (entry_id = 0; entry_id < num_reg_entries; entry_id++) {
/* Read current entry of all condition registers */
- if (dump) {
- u32 next_reg_offset = 0;
+ u32 next_reg_offset = 0;
- for (reg_id = 0;
- reg_id < rule->num_cond_regs;
- reg_id++) {
- const struct dbg_idle_chk_cond_reg
- *reg = &cond_regs[reg_id];
+ for (reg_id = 0; reg_id < rule->num_cond_regs;
+ reg_id++) {
+ const struct dbg_idle_chk_cond_reg *reg =
+ &cond_regs[reg_id];
- /* Find GRC address (if it's a memory,
- * the address of the specific entry is
- * calculated).
- */
- u32 grc_addr =
- DWORDS_TO_BYTES(
- GET_FIELD(reg->data,
- DBG_IDLE_CHK_COND_REG_ADDRESS));
+ /* Find GRC address (if it's a memory,the
+ * address of the specific entry is calculated).
+ */
+ u32 addr =
+ GET_FIELD(reg->data,
+ DBG_IDLE_CHK_COND_REG_ADDRESS);
- if (reg->num_entries > 1 ||
- reg->start_entry > 0) {
- u32 padded_entry_size =
- reg->entry_size > 1 ?
- roundup_pow_of_two
- (reg->entry_size) : 1;
+ if (reg->num_entries > 1 ||
+ reg->start_entry > 0) {
+ u32 padded_entry_size =
+ reg->entry_size > 1 ?
+ roundup_pow_of_two(reg->entry_size) :
+ 1;
- grc_addr +=
- DWORDS_TO_BYTES(
- (reg->start_entry +
- entry_id)
- * padded_entry_size);
- }
-
- /* Read registers */
- if (next_reg_offset + reg->entry_size >=
- IDLE_CHK_MAX_ENTRIES_SIZE) {
- DP_NOTICE(p_hwfn,
- "idle check registers entry is too large\n");
- return 0;
- }
-
- for (j = 0; j < reg->entry_size;
- j++, next_reg_offset++,
- grc_addr += 4)
- cond_reg_values[next_reg_offset] =
- qed_rd(p_hwfn, p_ptt, grc_addr);
+ addr += (reg->start_entry + entry_id) *
+ padded_entry_size;
}
+
+ /* Read registers */
+ if (next_reg_offset + reg->entry_size >=
+ IDLE_CHK_MAX_ENTRIES_SIZE) {
+ DP_NOTICE(p_hwfn,
+ "idle check registers entry is too large\n");
+ return 0;
+ }
+
+ next_reg_offset +=
+ qed_grc_dump_addr_range(p_hwfn,
+ p_ptt,
+ cond_reg_values +
+ next_reg_offset,
+ dump, addr,
+ reg->entry_size);
}
/* Call rule's condition function - a return value of
* true indicates failure.
*/
if ((*cond_arr[rule->cond_id])(cond_reg_values,
- imm_values) || !dump) {
+ imm_values)) {
offset +=
- qed_idle_chk_dump_failure(p_hwfn,
- p_ptt,
- dump_buf + offset,
- dump,
- rule->rule_id,
- rule,
- entry_id,
- cond_reg_values);
+ qed_idle_chk_dump_failure(p_hwfn,
+ p_ptt,
+ dump_buf + offset,
+ dump,
+ rule->rule_id,
+ rule,
+ entry_id,
+ cond_reg_values);
(*num_failing_rules)++;
break;
}
@@ -3818,13 +4100,18 @@
struct mcp_file_att file_att;
/* Call NVRAM get file command */
- if (qed_mcp_nvm_rd_cmd(p_hwfn, p_ptt, DRV_MSG_CODE_NVM_GET_FILE_ATT,
- image_type, &ret_mcp_resp, &ret_mcp_param,
- &ret_txn_size, (u32 *)&file_att) != 0)
- return DBG_STATUS_NVRAM_GET_IMAGE_FAILED;
+ int nvm_result = qed_mcp_nvm_rd_cmd(p_hwfn,
+ p_ptt,
+ DRV_MSG_CODE_NVM_GET_FILE_ATT,
+ image_type,
+ &ret_mcp_resp,
+ &ret_mcp_param,
+ &ret_txn_size,
+ (u32 *)&file_att);
/* Check response */
- if ((ret_mcp_resp & FW_MSG_CODE_MASK) != FW_MSG_CODE_NVM_OK)
+ if (nvm_result ||
+ (ret_mcp_resp & FW_MSG_CODE_MASK) != FW_MSG_CODE_NVM_OK)
return DBG_STATUS_NVRAM_GET_IMAGE_FAILED;
/* Update return values */
@@ -3944,7 +4231,6 @@
u32 running_mfw_addr =
MCP_REG_SCRATCH + SECTION_OFFSET(spad_trace_offsize) +
QED_SECTION_SIZE(spad_trace_offsize) + trace_data_size_bytes;
- enum dbg_status status;
u32 nvram_image_type;
*running_bundle_id = qed_rd(p_hwfn, p_ptt, running_mfw_addr);
@@ -3955,30 +4241,12 @@
nvram_image_type =
(*running_bundle_id ==
DIR_ID_1) ? NVM_TYPE_MFW_TRACE1 : NVM_TYPE_MFW_TRACE2;
- status = qed_find_nvram_image(p_hwfn,
- p_ptt,
- nvram_image_type,
- trace_meta_offset_bytes,
- trace_meta_size_bytes);
- return status;
-}
-
-/* Reads the MCP Trace data from the specified GRC address into the specified
- * buffer.
- */
-static void qed_mcp_trace_read_data(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt,
- u32 grc_addr, u32 size_in_dwords, u32 *buf)
-{
- u32 i;
-
- DP_VERBOSE(p_hwfn,
- QED_MSG_DEBUG,
- "mcp_trace_read_data: reading trace data of size %d dwords from GRC address 0x%x\n",
- size_in_dwords, grc_addr);
- for (i = 0; i < size_in_dwords; i++, grc_addr += BYTES_IN_DWORD)
- buf[i] = qed_rd(p_hwfn, p_ptt, grc_addr);
+ return qed_find_nvram_image(p_hwfn,
+ p_ptt,
+ nvram_image_type,
+ trace_meta_offset_bytes,
+ trace_meta_size_bytes);
}
/* Reads the MCP Trace meta data (from NVRAM or buffer) into the specified
@@ -4034,11 +4302,14 @@
bool dump, u32 *num_dumped_dwords)
{
u32 trace_data_grc_addr, trace_data_size_bytes, trace_data_size_dwords;
- u32 trace_meta_size_dwords, running_bundle_id, offset = 0;
- u32 trace_meta_offset_bytes, trace_meta_size_bytes;
+ u32 trace_meta_size_dwords = 0, running_bundle_id, offset = 0;
+ u32 trace_meta_offset_bytes = 0, trace_meta_size_bytes = 0;
enum dbg_status status;
+ bool mcp_access;
int halted = 0;
+ mcp_access = !qed_grc_get_param(p_hwfn, DBG_GRC_PARAM_NO_MCP);
+
*num_dumped_dwords = 0;
/* Get trace data info */
@@ -4060,7 +4331,7 @@
* consistent if halt fails, MCP trace is taken anyway, with a small
* risk that it may be corrupt.
*/
- if (dump) {
+ if (dump && mcp_access) {
halted = !qed_mcp_halt(p_hwfn, p_ptt);
if (!halted)
DP_NOTICE(p_hwfn, "MCP halt failed!\n");
@@ -4078,13 +4349,12 @@
dump, "size", trace_data_size_dwords);
/* Read trace data from scratchpad into dump buffer */
- if (dump)
- qed_mcp_trace_read_data(p_hwfn,
- p_ptt,
- trace_data_grc_addr,
- trace_data_size_dwords,
- dump_buf + offset);
- offset += trace_data_size_dwords;
+ offset += qed_grc_dump_addr_range(p_hwfn,
+ p_ptt,
+ dump_buf + offset,
+ dump,
+ BYTES_TO_DWORDS(trace_data_grc_addr),
+ trace_data_size_dwords);
/* Resume MCP (only if halt succeeded) */
if (halted && qed_mcp_resume(p_hwfn, p_ptt) != 0)
@@ -4095,38 +4365,38 @@
dump, "mcp_trace_meta", 1);
/* Read trace meta info */
- status = qed_mcp_trace_get_meta_info(p_hwfn,
- p_ptt,
- trace_data_size_bytes,
- &running_bundle_id,
- &trace_meta_offset_bytes,
- &trace_meta_size_bytes);
- if (status != DBG_STATUS_OK)
- return status;
-
- /* Dump trace meta size param (trace_meta_size_bytes is always
- * dword-aligned).
- */
- trace_meta_size_dwords = BYTES_TO_DWORDS(trace_meta_size_bytes);
- offset += qed_dump_num_param(dump_buf + offset, dump, "size",
- trace_meta_size_dwords);
-
- /* Read trace meta image into dump buffer */
- if (dump) {
- status = qed_mcp_trace_read_meta(p_hwfn,
- p_ptt,
- trace_meta_offset_bytes,
- trace_meta_size_bytes,
- dump_buf + offset);
- if (status != DBG_STATUS_OK)
- return status;
+ if (mcp_access) {
+ status = qed_mcp_trace_get_meta_info(p_hwfn,
+ p_ptt,
+ trace_data_size_bytes,
+ &running_bundle_id,
+ &trace_meta_offset_bytes,
+ &trace_meta_size_bytes);
+ if (status == DBG_STATUS_OK)
+ trace_meta_size_dwords =
+ BYTES_TO_DWORDS(trace_meta_size_bytes);
}
- offset += trace_meta_size_dwords;
+ /* Dump trace meta size param */
+ offset += qed_dump_num_param(dump_buf + offset,
+ dump, "size", trace_meta_size_dwords);
+
+ /* Read trace meta image into dump buffer */
+ if (dump && trace_meta_size_dwords)
+ status = qed_mcp_trace_read_meta(p_hwfn,
+ p_ptt,
+ trace_meta_offset_bytes,
+ trace_meta_size_bytes,
+ dump_buf + offset);
+ if (status == DBG_STATUS_OK)
+ offset += trace_meta_size_dwords;
*num_dumped_dwords = offset;
- return DBG_STATUS_OK;
+ /* If no mcp access, indicate that the dump doesn't contain the meta
+ * data from NVRAM.
+ */
+ return mcp_access ? status : DBG_STATUS_NVRAM_GET_IMAGE_FAILED;
}
/* Dump GRC FIFO */
@@ -4311,9 +4581,10 @@
struct qed_ptt *p_ptt, u32 *dump_buf, bool dump)
{
struct dbg_tools_data *dev_data = &p_hwfn->dbg_info;
+ struct fw_asserts_ram_section *asserts;
char storm_letter_str[2] = "?";
struct fw_info fw_info;
- u32 offset = 0, i;
+ u32 offset = 0;
u8 storm_id;
/* Dump global params */
@@ -4323,8 +4594,8 @@
offset += qed_dump_str_param(dump_buf + offset,
dump, "dump-type", "fw-asserts");
for (storm_id = 0; storm_id < MAX_DBG_STORMS; storm_id++) {
- u32 fw_asserts_section_addr, next_list_idx_addr, next_list_idx,
- last_list_idx, element_addr;
+ u32 fw_asserts_section_addr, next_list_idx_addr, next_list_idx;
+ u32 last_list_idx, addr;
if (dev_data->block_in_reset[s_storm_defs[storm_id].block_id])
continue;
@@ -4332,6 +4603,8 @@
/* Read FW info for the current Storm */
qed_read_fw_info(p_hwfn, p_ptt, storm_id, &fw_info);
+ asserts = &fw_info.fw_asserts_section;
+
/* Dump FW Asserts section header and params */
storm_letter_str[0] = s_storm_defs[storm_id].letter;
offset += qed_dump_section_hdr(dump_buf + offset, dump,
@@ -4339,12 +4612,10 @@
offset += qed_dump_str_param(dump_buf + offset, dump, "storm",
storm_letter_str);
offset += qed_dump_num_param(dump_buf + offset, dump, "size",
- fw_info.fw_asserts_section.
- list_element_dword_size);
+ asserts->list_element_dword_size);
if (!dump) {
- offset += fw_info.fw_asserts_section.
- list_element_dword_size;
+ offset += asserts->list_element_dword_size;
continue;
}
@@ -4352,28 +4623,22 @@
fw_asserts_section_addr =
s_storm_defs[storm_id].sem_fast_mem_addr +
SEM_FAST_REG_INT_RAM +
- RAM_LINES_TO_BYTES(fw_info.fw_asserts_section.
- section_ram_line_offset);
+ RAM_LINES_TO_BYTES(asserts->section_ram_line_offset);
next_list_idx_addr =
fw_asserts_section_addr +
- DWORDS_TO_BYTES(fw_info.fw_asserts_section.
- list_next_index_dword_offset);
+ DWORDS_TO_BYTES(asserts->list_next_index_dword_offset);
next_list_idx = qed_rd(p_hwfn, p_ptt, next_list_idx_addr);
last_list_idx = (next_list_idx > 0
? next_list_idx
- : fw_info.fw_asserts_section.list_num_elements)
- - 1;
- element_addr =
- fw_asserts_section_addr +
- DWORDS_TO_BYTES(fw_info.fw_asserts_section.
- list_dword_offset) +
- last_list_idx *
- DWORDS_TO_BYTES(fw_info.fw_asserts_section.
- list_element_dword_size);
- for (i = 0;
- i < fw_info.fw_asserts_section.list_element_dword_size;
- i++, offset++, element_addr += BYTES_IN_DWORD)
- dump_buf[offset] = qed_rd(p_hwfn, p_ptt, element_addr);
+ : asserts->list_num_elements) - 1;
+ addr = BYTES_TO_DWORDS(fw_asserts_section_addr) +
+ asserts->list_dword_offset +
+ last_list_idx * asserts->list_element_dword_size;
+ offset +=
+ qed_grc_dump_addr_range(p_hwfn, p_ptt,
+ dump_buf + offset,
+ dump, addr,
+ asserts->list_element_dword_size);
}
/* Dump last section */
@@ -4386,13 +4651,10 @@
enum dbg_status qed_dbg_set_bin_ptr(const u8 * const bin_ptr)
{
/* Convert binary data to debug arrays */
- u32 num_of_buffers = *(u32 *)bin_ptr;
- struct bin_buffer_hdr *buf_array;
+ struct bin_buffer_hdr *buf_array = (struct bin_buffer_hdr *)bin_ptr;
u8 buf_id;
- buf_array = (struct bin_buffer_hdr *)((u32 *)bin_ptr + 1);
-
- for (buf_id = 0; buf_id < num_of_buffers; buf_id++) {
+ for (buf_id = 0; buf_id < MAX_BIN_DBG_BUFFER_TYPE; buf_id++) {
s_dbg_arrays[buf_id].ptr =
(u32 *)(bin_ptr + buf_array[buf_id].offset);
s_dbg_arrays[buf_id].size_in_dwords =
@@ -4402,6 +4664,17 @@
return DBG_STATUS_OK;
}
+/* Assign default GRC param values */
+void qed_dbg_grc_set_params_default(struct qed_hwfn *p_hwfn)
+{
+ struct dbg_tools_data *dev_data = &p_hwfn->dbg_info;
+ u32 i;
+
+ for (i = 0; i < MAX_DBG_GRC_PARAMS; i++)
+ dev_data->grc.param_val[i] =
+ s_grc_param_defs[i].default_val[dev_data->chip_id];
+}
+
enum dbg_status qed_dbg_grc_get_dump_buf_size(struct qed_hwfn *p_hwfn,
struct qed_ptt *p_ptt,
u32 *buf_size)
@@ -4441,8 +4714,9 @@
/* GRC Dump */
status = qed_grc_dump(p_hwfn, p_ptt, dump_buf, true, num_dumped_dwords);
- /* Clear all GRC params */
- qed_dbg_grc_clear_params(p_hwfn);
+ /* Revert GRC params to their default */
+ qed_dbg_grc_set_params_default(p_hwfn);
+
return status;
}
@@ -4495,6 +4769,10 @@
/* Idle Check Dump */
*num_dumped_dwords = qed_idle_chk_dump(p_hwfn, p_ptt, dump_buf, true);
+
+ /* Revert GRC params to their default */
+ qed_dbg_grc_set_params_default(p_hwfn);
+
return DBG_STATUS_OK;
}
@@ -4519,11 +4797,15 @@
u32 needed_buf_size_in_dwords;
enum dbg_status status;
- status = qed_dbg_mcp_trace_get_dump_buf_size(p_hwfn, p_ptt,
+ /* validate buffer size */
+ status =
+ qed_dbg_mcp_trace_get_dump_buf_size(p_hwfn, p_ptt,
&needed_buf_size_in_dwords);
- if (status != DBG_STATUS_OK)
+ if (status != DBG_STATUS_OK &&
+ status != DBG_STATUS_NVRAM_GET_IMAGE_FAILED)
return status;
+
if (buf_size_in_dwords < needed_buf_size_in_dwords)
return DBG_STATUS_DUMP_BUF_TOO_SMALL;
@@ -4531,8 +4813,13 @@
qed_update_blocks_reset_state(p_hwfn, p_ptt);
/* Perform dump */
- return qed_mcp_trace_dump(p_hwfn,
- p_ptt, dump_buf, true, num_dumped_dwords);
+ status = qed_mcp_trace_dump(p_hwfn,
+ p_ptt, dump_buf, true, num_dumped_dwords);
+
+ /* Revert GRC params to their default */
+ qed_dbg_grc_set_params_default(p_hwfn);
+
+ return status;
}
enum dbg_status qed_dbg_reg_fifo_get_dump_buf_size(struct qed_hwfn *p_hwfn,
@@ -4567,8 +4854,14 @@
/* Update reset state */
qed_update_blocks_reset_state(p_hwfn, p_ptt);
- return qed_reg_fifo_dump(p_hwfn,
- p_ptt, dump_buf, true, num_dumped_dwords);
+
+ status = qed_reg_fifo_dump(p_hwfn,
+ p_ptt, dump_buf, true, num_dumped_dwords);
+
+ /* Revert GRC params to their default */
+ qed_dbg_grc_set_params_default(p_hwfn);
+
+ return status;
}
enum dbg_status qed_dbg_igu_fifo_get_dump_buf_size(struct qed_hwfn *p_hwfn,
@@ -4603,8 +4896,13 @@
/* Update reset state */
qed_update_blocks_reset_state(p_hwfn, p_ptt);
- return qed_igu_fifo_dump(p_hwfn,
- p_ptt, dump_buf, true, num_dumped_dwords);
+
+ status = qed_igu_fifo_dump(p_hwfn,
+ p_ptt, dump_buf, true, num_dumped_dwords);
+ /* Revert GRC params to their default */
+ qed_dbg_grc_set_params_default(p_hwfn);
+
+ return status;
}
enum dbg_status
@@ -4641,9 +4939,16 @@
/* Update reset state */
qed_update_blocks_reset_state(p_hwfn, p_ptt);
- return qed_protection_override_dump(p_hwfn,
- p_ptt,
- dump_buf, true, num_dumped_dwords);
+
+ status = qed_protection_override_dump(p_hwfn,
+ p_ptt,
+ dump_buf,
+ true, num_dumped_dwords);
+
+ /* Revert GRC params to their default */
+ qed_dbg_grc_set_params_default(p_hwfn);
+
+ return status;
}
enum dbg_status qed_dbg_fw_asserts_get_dump_buf_size(struct qed_hwfn *p_hwfn,
@@ -5045,13 +5350,10 @@
enum dbg_status qed_dbg_user_set_bin_ptr(const u8 * const bin_ptr)
{
/* Convert binary data to debug arrays */
- u32 num_of_buffers = *(u32 *)bin_ptr;
- struct bin_buffer_hdr *buf_array;
+ struct bin_buffer_hdr *buf_array = (struct bin_buffer_hdr *)bin_ptr;
u8 buf_id;
- buf_array = (struct bin_buffer_hdr *)((u32 *)bin_ptr + 1);
-
- for (buf_id = 0; buf_id < num_of_buffers; buf_id++) {
+ for (buf_id = 0; buf_id < MAX_BIN_DBG_BUFFER_TYPE; buf_id++) {
s_dbg_arrays[buf_id].ptr =
(u32 *)(bin_ptr + buf_array[buf_id].offset);
s_dbg_arrays[buf_id].size_in_dwords =
@@ -5874,16 +6176,16 @@
results_offset +=
sprintf(qed_get_buf_ptr(results_buf,
results_offset),
- "raw: 0x%016llx, address: 0x%07llx, access: %-5s, pf: %2lld, vf: %s, port: %lld, privilege: %-3s, protection: %-12s, master: %-4s, errors: ",
+ "raw: 0x%016llx, address: 0x%07x, access: %-5s, pf: %2d, vf: %s, port: %d, privilege: %-3s, protection: %-12s, master: %-4s, errors: ",
elements[i].data,
- GET_FIELD(elements[i].data,
+ (u32)GET_FIELD(elements[i].data,
REG_FIFO_ELEMENT_ADDRESS) *
REG_FIFO_ELEMENT_ADDR_FACTOR,
s_access_strs[GET_FIELD(elements[i].data,
REG_FIFO_ELEMENT_ACCESS)],
- GET_FIELD(elements[i].data,
- REG_FIFO_ELEMENT_PF), vf_str,
- GET_FIELD(elements[i].data,
+ (u32)GET_FIELD(elements[i].data,
+ REG_FIFO_ELEMENT_PF), vf_str,
+ (u32)GET_FIELD(elements[i].data,
REG_FIFO_ELEMENT_PORT),
s_privilege_strs[GET_FIELD(elements[i].
data,
@@ -6189,13 +6491,13 @@
results_offset +=
sprintf(qed_get_buf_ptr(results_buf,
results_offset),
- "window %2d, address: 0x%07x, size: %7lld regs, read: %lld, write: %lld, read protection: %-12s, write protection: %-12s\n",
+ "window %2d, address: 0x%07x, size: %7d regs, read: %d, write: %d, read protection: %-12s, write protection: %-12s\n",
i, address,
- GET_FIELD(elements[i].data,
+ (u32)GET_FIELD(elements[i].data,
PROTECTION_OVERRIDE_ELEMENT_WINDOW_SIZE),
- GET_FIELD(elements[i].data,
+ (u32)GET_FIELD(elements[i].data,
PROTECTION_OVERRIDE_ELEMENT_READ),
- GET_FIELD(elements[i].data,
+ (u32)GET_FIELD(elements[i].data,
PROTECTION_OVERRIDE_ELEMENT_WRITE),
s_protection_strs[GET_FIELD(elements[i].data,
PROTECTION_OVERRIDE_ELEMENT_READ_PROTECTION)],
@@ -6508,7 +6810,7 @@
*/
rc = qed_features_lookup[feature_idx].get_size(p_hwfn, p_ptt,
&buf_size_dwords);
- if (rc != DBG_STATUS_OK)
+ if (rc != DBG_STATUS_OK && rc != DBG_STATUS_NVRAM_GET_IMAGE_FAILED)
return rc;
feature->buf_size = buf_size_dwords * sizeof(u32);
feature->dump_buf = vmalloc(feature->buf_size);
diff --git a/drivers/net/ethernet/qlogic/qed/qed_dev.c b/drivers/net/ethernet/qlogic/qed/qed_dev.c
index e518f91..5f31140 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_dev.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_dev.c
@@ -75,7 +75,8 @@
BAR_ID_1 /* Used for doorbells */
};
-static u32 qed_hw_bar_size(struct qed_hwfn *p_hwfn, enum BAR_ID bar_id)
+static u32 qed_hw_bar_size(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt, enum BAR_ID bar_id)
{
u32 bar_reg = (bar_id == BAR_ID_0 ?
PGLUE_B_REG_PF_BAR0_SIZE : PGLUE_B_REG_PF_BAR1_SIZE);
@@ -84,7 +85,7 @@
if (IS_VF(p_hwfn->cdev))
return 1 << 17;
- val = qed_rd(p_hwfn, p_hwfn->p_main_ptt, bar_reg);
+ val = qed_rd(p_hwfn, p_ptt, bar_reg);
if (val)
return 1 << (val + 15);
@@ -182,199 +183,573 @@
}
qed_iov_free(p_hwfn);
qed_dmae_info_free(p_hwfn);
- qed_dcbx_info_free(p_hwfn, p_hwfn->p_dcbx_info);
+ qed_dcbx_info_free(p_hwfn);
}
}
-static int qed_init_qm_info(struct qed_hwfn *p_hwfn, bool b_sleepable)
+/******************** QM initialization *******************/
+#define ACTIVE_TCS_BMAP 0x9f
+#define ACTIVE_TCS_BMAP_4PORT_K2 0xf
+
+/* determines the physical queue flags for a given PF. */
+static u32 qed_get_pq_flags(struct qed_hwfn *p_hwfn)
{
- u8 num_vports, vf_offset = 0, i, vport_id, num_ports, curr_queue = 0;
- struct qed_qm_info *qm_info = &p_hwfn->qm_info;
- struct init_qm_port_params *p_qm_port;
- bool init_rdma_offload_pq = false;
- bool init_pure_ack_pq = false;
- bool init_ooo_pq = false;
- u16 num_pqs, multi_cos_tcs = 1;
- u8 pf_wfq = qm_info->pf_wfq;
- u32 pf_rl = qm_info->pf_rl;
- u16 num_pf_rls = 0;
- u16 num_vfs = 0;
+ u32 flags;
-#ifdef CONFIG_QED_SRIOV
- if (p_hwfn->cdev->p_iov_info)
- num_vfs = p_hwfn->cdev->p_iov_info->total_vfs;
-#endif
- memset(qm_info, 0, sizeof(*qm_info));
+ /* common flags */
+ flags = PQ_FLAGS_LB;
- num_pqs = multi_cos_tcs + num_vfs + 1; /* The '1' is for pure-LB */
- num_vports = (u8)RESC_NUM(p_hwfn, QED_VPORT);
+ /* feature flags */
+ if (IS_QED_SRIOV(p_hwfn->cdev))
+ flags |= PQ_FLAGS_VFS;
- if (p_hwfn->hw_info.personality == QED_PCI_ETH_ROCE) {
- num_pqs++; /* for RoCE queue */
- init_rdma_offload_pq = true;
- /* we subtract num_vfs because each require a rate limiter,
- * and one default rate limiter
- */
- if (p_hwfn->pf_params.rdma_pf_params.enable_dcqcn)
- num_pf_rls = RESC_NUM(p_hwfn, QED_RL) - num_vfs - 1;
-
- num_pqs += num_pf_rls;
- qm_info->num_pf_rls = (u8) num_pf_rls;
- }
-
- if (p_hwfn->hw_info.personality == QED_PCI_ISCSI) {
- num_pqs += 2; /* for iSCSI pure-ACK / OOO queue */
- init_pure_ack_pq = true;
- init_ooo_pq = true;
- }
-
- /* Sanity checking that setup requires legal number of resources */
- if (num_pqs > RESC_NUM(p_hwfn, QED_PQ)) {
+ /* protocol flags */
+ switch (p_hwfn->hw_info.personality) {
+ case QED_PCI_ETH:
+ flags |= PQ_FLAGS_MCOS;
+ break;
+ case QED_PCI_FCOE:
+ flags |= PQ_FLAGS_OFLD;
+ break;
+ case QED_PCI_ISCSI:
+ flags |= PQ_FLAGS_ACK | PQ_FLAGS_OOO | PQ_FLAGS_OFLD;
+ break;
+ case QED_PCI_ETH_ROCE:
+ flags |= PQ_FLAGS_MCOS | PQ_FLAGS_OFLD | PQ_FLAGS_LLT;
+ break;
+ default:
DP_ERR(p_hwfn,
- "Need too many Physical queues - 0x%04x when only %04x are available\n",
- num_pqs, RESC_NUM(p_hwfn, QED_PQ));
- return -EINVAL;
+ "unknown personality %d\n", p_hwfn->hw_info.personality);
+ return 0;
}
- /* PQs will be arranged as follows: First per-TC PQ then pure-LB quete.
+ return flags;
+}
+
+/* Getters for resource amounts necessary for qm initialization */
+u8 qed_init_qm_get_num_tcs(struct qed_hwfn *p_hwfn)
+{
+ return p_hwfn->hw_info.num_hw_tc;
+}
+
+u16 qed_init_qm_get_num_vfs(struct qed_hwfn *p_hwfn)
+{
+ return IS_QED_SRIOV(p_hwfn->cdev) ?
+ p_hwfn->cdev->p_iov_info->total_vfs : 0;
+}
+
+#define NUM_DEFAULT_RLS 1
+
+u16 qed_init_qm_get_num_pf_rls(struct qed_hwfn *p_hwfn)
+{
+ u16 num_pf_rls, num_vfs = qed_init_qm_get_num_vfs(p_hwfn);
+
+ /* num RLs can't exceed resource amount of rls or vports */
+ num_pf_rls = (u16) min_t(u32, RESC_NUM(p_hwfn, QED_RL),
+ RESC_NUM(p_hwfn, QED_VPORT));
+
+ /* Make sure after we reserve there's something left */
+ if (num_pf_rls < num_vfs + NUM_DEFAULT_RLS)
+ return 0;
+
+ /* subtract rls necessary for VFs and one default one for the PF */
+ num_pf_rls -= num_vfs + NUM_DEFAULT_RLS;
+
+ return num_pf_rls;
+}
+
+u16 qed_init_qm_get_num_vports(struct qed_hwfn *p_hwfn)
+{
+ u32 pq_flags = qed_get_pq_flags(p_hwfn);
+
+ /* all pqs share the same vport, except for vfs and pf_rl pqs */
+ return (!!(PQ_FLAGS_RLS & pq_flags)) *
+ qed_init_qm_get_num_pf_rls(p_hwfn) +
+ (!!(PQ_FLAGS_VFS & pq_flags)) *
+ qed_init_qm_get_num_vfs(p_hwfn) + 1;
+}
+
+/* calc amount of PQs according to the requested flags */
+u16 qed_init_qm_get_num_pqs(struct qed_hwfn *p_hwfn)
+{
+ u32 pq_flags = qed_get_pq_flags(p_hwfn);
+
+ return (!!(PQ_FLAGS_RLS & pq_flags)) *
+ qed_init_qm_get_num_pf_rls(p_hwfn) +
+ (!!(PQ_FLAGS_MCOS & pq_flags)) *
+ qed_init_qm_get_num_tcs(p_hwfn) +
+ (!!(PQ_FLAGS_LB & pq_flags)) + (!!(PQ_FLAGS_OOO & pq_flags)) +
+ (!!(PQ_FLAGS_ACK & pq_flags)) + (!!(PQ_FLAGS_OFLD & pq_flags)) +
+ (!!(PQ_FLAGS_LLT & pq_flags)) +
+ (!!(PQ_FLAGS_VFS & pq_flags)) * qed_init_qm_get_num_vfs(p_hwfn);
+}
+
+/* initialize the top level QM params */
+static void qed_init_qm_params(struct qed_hwfn *p_hwfn)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
+ bool four_port;
+
+ /* pq and vport bases for this PF */
+ qm_info->start_pq = (u16) RESC_START(p_hwfn, QED_PQ);
+ qm_info->start_vport = (u8) RESC_START(p_hwfn, QED_VPORT);
+
+ /* rate limiting and weighted fair queueing are always enabled */
+ qm_info->vport_rl_en = 1;
+ qm_info->vport_wfq_en = 1;
+
+ /* TC config is different for AH 4 port */
+ four_port = p_hwfn->cdev->num_ports_in_engines == MAX_NUM_PORTS_K2;
+
+ /* in AH 4 port we have fewer TCs per port */
+ qm_info->max_phys_tcs_per_port = four_port ? NUM_PHYS_TCS_4PORT_K2 :
+ NUM_OF_PHYS_TCS;
+
+ /* unless MFW indicated otherwise, ooo_tc == 3 for
+ * AH 4-port and 4 otherwise.
*/
- qm_info->qm_pq_params = kcalloc(num_pqs,
- sizeof(struct init_qm_pq_params),
- b_sleepable ? GFP_KERNEL : GFP_ATOMIC);
- if (!qm_info->qm_pq_params)
- goto alloc_err;
+ if (!qm_info->ooo_tc)
+ qm_info->ooo_tc = four_port ? DCBX_TCP_OOO_K2_4PORT_TC :
+ DCBX_TCP_OOO_TC;
+}
- qm_info->qm_vport_params = kcalloc(num_vports,
- sizeof(struct init_qm_vport_params),
- b_sleepable ? GFP_KERNEL
- : GFP_ATOMIC);
- if (!qm_info->qm_vport_params)
- goto alloc_err;
+/* initialize qm vport params */
+static void qed_init_qm_vport_params(struct qed_hwfn *p_hwfn)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
+ u8 i;
- qm_info->qm_port_params = kcalloc(MAX_NUM_PORTS,
- sizeof(struct init_qm_port_params),
- b_sleepable ? GFP_KERNEL
- : GFP_ATOMIC);
- if (!qm_info->qm_port_params)
- goto alloc_err;
+ /* all vports participate in weighted fair queueing */
+ for (i = 0; i < qed_init_qm_get_num_vports(p_hwfn); i++)
+ qm_info->qm_vport_params[i].vport_wfq = 1;
+}
- qm_info->wfq_data = kcalloc(num_vports, sizeof(struct qed_wfq_data),
- b_sleepable ? GFP_KERNEL : GFP_ATOMIC);
- if (!qm_info->wfq_data)
- goto alloc_err;
-
- vport_id = (u8)RESC_START(p_hwfn, QED_VPORT);
-
- /* First init rate limited queues */
- for (curr_queue = 0; curr_queue < num_pf_rls; curr_queue++) {
- qm_info->qm_pq_params[curr_queue].vport_id = vport_id++;
- qm_info->qm_pq_params[curr_queue].tc_id =
- p_hwfn->hw_info.non_offload_tc;
- qm_info->qm_pq_params[curr_queue].wrr_group = 1;
- qm_info->qm_pq_params[curr_queue].rl_valid = 1;
- }
-
- /* First init per-TC PQs */
- for (i = 0; i < multi_cos_tcs; i++) {
- struct init_qm_pq_params *params =
- &qm_info->qm_pq_params[curr_queue++];
-
- if (p_hwfn->hw_info.personality == QED_PCI_ETH_ROCE ||
- p_hwfn->hw_info.personality == QED_PCI_ETH) {
- params->vport_id = vport_id;
- params->tc_id = p_hwfn->hw_info.non_offload_tc;
- params->wrr_group = 1;
- } else {
- params->vport_id = vport_id;
- params->tc_id = p_hwfn->hw_info.offload_tc;
- params->wrr_group = 1;
- }
- }
-
- /* Then init pure-LB PQ */
- qm_info->pure_lb_pq = curr_queue;
- qm_info->qm_pq_params[curr_queue].vport_id =
- (u8) RESC_START(p_hwfn, QED_VPORT);
- qm_info->qm_pq_params[curr_queue].tc_id = PURE_LB_TC;
- qm_info->qm_pq_params[curr_queue].wrr_group = 1;
- curr_queue++;
-
- qm_info->offload_pq = 0;
- if (init_rdma_offload_pq) {
- qm_info->offload_pq = curr_queue;
- qm_info->qm_pq_params[curr_queue].vport_id = vport_id;
- qm_info->qm_pq_params[curr_queue].tc_id =
- p_hwfn->hw_info.offload_tc;
- qm_info->qm_pq_params[curr_queue].wrr_group = 1;
- curr_queue++;
- }
-
- if (init_pure_ack_pq) {
- qm_info->pure_ack_pq = curr_queue;
- qm_info->qm_pq_params[curr_queue].vport_id = vport_id;
- qm_info->qm_pq_params[curr_queue].tc_id =
- p_hwfn->hw_info.offload_tc;
- qm_info->qm_pq_params[curr_queue].wrr_group = 1;
- curr_queue++;
- }
-
- if (init_ooo_pq) {
- qm_info->ooo_pq = curr_queue;
- qm_info->qm_pq_params[curr_queue].vport_id = vport_id;
- qm_info->qm_pq_params[curr_queue].tc_id = DCBX_ISCSI_OOO_TC;
- qm_info->qm_pq_params[curr_queue].wrr_group = 1;
- curr_queue++;
- }
-
- /* Then init per-VF PQs */
- vf_offset = curr_queue;
- for (i = 0; i < num_vfs; i++) {
- /* First vport is used by the PF */
- qm_info->qm_pq_params[curr_queue].vport_id = vport_id + i + 1;
- qm_info->qm_pq_params[curr_queue].tc_id =
- p_hwfn->hw_info.non_offload_tc;
- qm_info->qm_pq_params[curr_queue].wrr_group = 1;
- qm_info->qm_pq_params[curr_queue].rl_valid = 1;
- curr_queue++;
- }
-
- qm_info->vf_queues_offset = vf_offset;
- qm_info->num_pqs = num_pqs;
- qm_info->num_vports = num_vports;
-
+/* initialize qm port params */
+static void qed_init_qm_port_params(struct qed_hwfn *p_hwfn)
+{
/* Initialize qm port parameters */
- num_ports = p_hwfn->cdev->num_ports_in_engines;
+ u8 i, active_phys_tcs, num_ports = p_hwfn->cdev->num_ports_in_engines;
+
+ /* indicate how ooo and high pri traffic is dealt with */
+ active_phys_tcs = num_ports == MAX_NUM_PORTS_K2 ?
+ ACTIVE_TCS_BMAP_4PORT_K2 :
+ ACTIVE_TCS_BMAP;
+
for (i = 0; i < num_ports; i++) {
- p_qm_port = &qm_info->qm_port_params[i];
+ struct init_qm_port_params *p_qm_port =
+ &p_hwfn->qm_info.qm_port_params[i];
+
p_qm_port->active = 1;
- if (num_ports == 4)
- p_qm_port->active_phys_tcs = 0x7;
- else
- p_qm_port->active_phys_tcs = 0x9f;
+ p_qm_port->active_phys_tcs = active_phys_tcs;
p_qm_port->num_pbf_cmd_lines = PBF_MAX_CMD_LINES / num_ports;
p_qm_port->num_btb_blocks = BTB_MAX_BLOCKS / num_ports;
}
+}
- qm_info->max_phys_tcs_per_port = NUM_OF_PHYS_TCS;
+/* Reset the params which must be reset for qm init. QM init may be called as
+ * a result of flows other than driver load (e.g. dcbx renegotiation). Other
+ * params may be affected by the init but would simply recalculate to the same
+ * values. The allocations made for QM init, ports, vports, pqs and vfqs are not
+ * affected as these amounts stay the same.
+ */
+static void qed_init_qm_reset_params(struct qed_hwfn *p_hwfn)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
- qm_info->start_pq = (u16)RESC_START(p_hwfn, QED_PQ);
+ qm_info->num_pqs = 0;
+ qm_info->num_vports = 0;
+ qm_info->num_pf_rls = 0;
+ qm_info->num_vf_pqs = 0;
+ qm_info->first_vf_pq = 0;
+ qm_info->first_mcos_pq = 0;
+ qm_info->first_rl_pq = 0;
+}
+static void qed_init_qm_advance_vport(struct qed_hwfn *p_hwfn)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
+
+ qm_info->num_vports++;
+
+ if (qm_info->num_vports > qed_init_qm_get_num_vports(p_hwfn))
+ DP_ERR(p_hwfn,
+ "vport overflow! qm_info->num_vports %d, qm_init_get_num_vports() %d\n",
+ qm_info->num_vports, qed_init_qm_get_num_vports(p_hwfn));
+}
+
+/* initialize a single pq and manage qm_info resources accounting.
+ * The pq_init_flags param determines whether the PQ is rate limited
+ * (for VF or PF) and whether a new vport is allocated to the pq or not
+ * (i.e. vport will be shared).
+ */
+
+/* flags for pq init */
+#define PQ_INIT_SHARE_VPORT (1 << 0)
+#define PQ_INIT_PF_RL (1 << 1)
+#define PQ_INIT_VF_RL (1 << 2)
+
+/* defines for pq init */
+#define PQ_INIT_DEFAULT_WRR_GROUP 1
+#define PQ_INIT_DEFAULT_TC 0
+#define PQ_INIT_OFLD_TC (p_hwfn->hw_info.offload_tc)
+
+static void qed_init_qm_pq(struct qed_hwfn *p_hwfn,
+ struct qed_qm_info *qm_info,
+ u8 tc, u32 pq_init_flags)
+{
+ u16 pq_idx = qm_info->num_pqs, max_pq = qed_init_qm_get_num_pqs(p_hwfn);
+
+ if (pq_idx > max_pq)
+ DP_ERR(p_hwfn,
+ "pq overflow! pq %d, max pq %d\n", pq_idx, max_pq);
+
+ /* init pq params */
+ qm_info->qm_pq_params[pq_idx].vport_id = qm_info->start_vport +
+ qm_info->num_vports;
+ qm_info->qm_pq_params[pq_idx].tc_id = tc;
+ qm_info->qm_pq_params[pq_idx].wrr_group = PQ_INIT_DEFAULT_WRR_GROUP;
+ qm_info->qm_pq_params[pq_idx].rl_valid =
+ (pq_init_flags & PQ_INIT_PF_RL || pq_init_flags & PQ_INIT_VF_RL);
+
+ /* qm params accounting */
+ qm_info->num_pqs++;
+ if (!(pq_init_flags & PQ_INIT_SHARE_VPORT))
+ qm_info->num_vports++;
+
+ if (pq_init_flags & PQ_INIT_PF_RL)
+ qm_info->num_pf_rls++;
+
+ if (qm_info->num_vports > qed_init_qm_get_num_vports(p_hwfn))
+ DP_ERR(p_hwfn,
+ "vport overflow! qm_info->num_vports %d, qm_init_get_num_vports() %d\n",
+ qm_info->num_vports, qed_init_qm_get_num_vports(p_hwfn));
+
+ if (qm_info->num_pf_rls > qed_init_qm_get_num_pf_rls(p_hwfn))
+ DP_ERR(p_hwfn,
+ "rl overflow! qm_info->num_pf_rls %d, qm_init_get_num_pf_rls() %d\n",
+ qm_info->num_pf_rls, qed_init_qm_get_num_pf_rls(p_hwfn));
+}
+
+/* get pq index according to PQ_FLAGS */
+static u16 *qed_init_qm_get_idx_from_flags(struct qed_hwfn *p_hwfn,
+ u32 pq_flags)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
+
+ /* Can't have multiple flags set here */
+ if (bitmap_weight((unsigned long *)&pq_flags, sizeof(pq_flags)) > 1)
+ goto err;
+
+ switch (pq_flags) {
+ case PQ_FLAGS_RLS:
+ return &qm_info->first_rl_pq;
+ case PQ_FLAGS_MCOS:
+ return &qm_info->first_mcos_pq;
+ case PQ_FLAGS_LB:
+ return &qm_info->pure_lb_pq;
+ case PQ_FLAGS_OOO:
+ return &qm_info->ooo_pq;
+ case PQ_FLAGS_ACK:
+ return &qm_info->pure_ack_pq;
+ case PQ_FLAGS_OFLD:
+ return &qm_info->offload_pq;
+ case PQ_FLAGS_LLT:
+ return &qm_info->low_latency_pq;
+ case PQ_FLAGS_VFS:
+ return &qm_info->first_vf_pq;
+ default:
+ goto err;
+ }
+
+err:
+ DP_ERR(p_hwfn, "BAD pq flags %d\n", pq_flags);
+ return NULL;
+}
+
+/* save pq index in qm info */
+static void qed_init_qm_set_idx(struct qed_hwfn *p_hwfn,
+ u32 pq_flags, u16 pq_val)
+{
+ u16 *base_pq_idx = qed_init_qm_get_idx_from_flags(p_hwfn, pq_flags);
+
+ *base_pq_idx = p_hwfn->qm_info.start_pq + pq_val;
+}
+
+/* get tx pq index, with the PQ TX base already set (ready for context init) */
+u16 qed_get_cm_pq_idx(struct qed_hwfn *p_hwfn, u32 pq_flags)
+{
+ u16 *base_pq_idx = qed_init_qm_get_idx_from_flags(p_hwfn, pq_flags);
+
+ return *base_pq_idx + CM_TX_PQ_BASE;
+}
+
+u16 qed_get_cm_pq_idx_mcos(struct qed_hwfn *p_hwfn, u8 tc)
+{
+ u8 max_tc = qed_init_qm_get_num_tcs(p_hwfn);
+
+ if (tc > max_tc)
+ DP_ERR(p_hwfn, "tc %d must be smaller than %d\n", tc, max_tc);
+
+ return qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_MCOS) + tc;
+}
+
+u16 qed_get_cm_pq_idx_vf(struct qed_hwfn *p_hwfn, u16 vf)
+{
+ u16 max_vf = qed_init_qm_get_num_vfs(p_hwfn);
+
+ if (vf > max_vf)
+ DP_ERR(p_hwfn, "vf %d must be smaller than %d\n", vf, max_vf);
+
+ return qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_VFS) + vf;
+}
+
+u16 qed_get_cm_pq_idx_rl(struct qed_hwfn *p_hwfn, u8 rl)
+{
+ u16 max_rl = qed_init_qm_get_num_pf_rls(p_hwfn);
+
+ if (rl > max_rl)
+ DP_ERR(p_hwfn, "rl %d must be smaller than %d\n", rl, max_rl);
+
+ return qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_RLS) + rl;
+}
+
+/* Functions for creating specific types of pqs */
+static void qed_init_qm_lb_pq(struct qed_hwfn *p_hwfn)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
+
+ if (!(qed_get_pq_flags(p_hwfn) & PQ_FLAGS_LB))
+ return;
+
+ qed_init_qm_set_idx(p_hwfn, PQ_FLAGS_LB, qm_info->num_pqs);
+ qed_init_qm_pq(p_hwfn, qm_info, PURE_LB_TC, PQ_INIT_SHARE_VPORT);
+}
+
+static void qed_init_qm_ooo_pq(struct qed_hwfn *p_hwfn)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
+
+ if (!(qed_get_pq_flags(p_hwfn) & PQ_FLAGS_OOO))
+ return;
+
+ qed_init_qm_set_idx(p_hwfn, PQ_FLAGS_OOO, qm_info->num_pqs);
+ qed_init_qm_pq(p_hwfn, qm_info, qm_info->ooo_tc, PQ_INIT_SHARE_VPORT);
+}
+
+static void qed_init_qm_pure_ack_pq(struct qed_hwfn *p_hwfn)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
+
+ if (!(qed_get_pq_flags(p_hwfn) & PQ_FLAGS_ACK))
+ return;
+
+ qed_init_qm_set_idx(p_hwfn, PQ_FLAGS_ACK, qm_info->num_pqs);
+ qed_init_qm_pq(p_hwfn, qm_info, PQ_INIT_OFLD_TC, PQ_INIT_SHARE_VPORT);
+}
+
+static void qed_init_qm_offload_pq(struct qed_hwfn *p_hwfn)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
+
+ if (!(qed_get_pq_flags(p_hwfn) & PQ_FLAGS_OFLD))
+ return;
+
+ qed_init_qm_set_idx(p_hwfn, PQ_FLAGS_OFLD, qm_info->num_pqs);
+ qed_init_qm_pq(p_hwfn, qm_info, PQ_INIT_OFLD_TC, PQ_INIT_SHARE_VPORT);
+}
+
+static void qed_init_qm_low_latency_pq(struct qed_hwfn *p_hwfn)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
+
+ if (!(qed_get_pq_flags(p_hwfn) & PQ_FLAGS_LLT))
+ return;
+
+ qed_init_qm_set_idx(p_hwfn, PQ_FLAGS_LLT, qm_info->num_pqs);
+ qed_init_qm_pq(p_hwfn, qm_info, PQ_INIT_OFLD_TC, PQ_INIT_SHARE_VPORT);
+}
+
+static void qed_init_qm_mcos_pqs(struct qed_hwfn *p_hwfn)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
+ u8 tc_idx;
+
+ if (!(qed_get_pq_flags(p_hwfn) & PQ_FLAGS_MCOS))
+ return;
+
+ qed_init_qm_set_idx(p_hwfn, PQ_FLAGS_MCOS, qm_info->num_pqs);
+ for (tc_idx = 0; tc_idx < qed_init_qm_get_num_tcs(p_hwfn); tc_idx++)
+ qed_init_qm_pq(p_hwfn, qm_info, tc_idx, PQ_INIT_SHARE_VPORT);
+}
+
+static void qed_init_qm_vf_pqs(struct qed_hwfn *p_hwfn)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
+ u16 vf_idx, num_vfs = qed_init_qm_get_num_vfs(p_hwfn);
+
+ if (!(qed_get_pq_flags(p_hwfn) & PQ_FLAGS_VFS))
+ return;
+
+ qed_init_qm_set_idx(p_hwfn, PQ_FLAGS_VFS, qm_info->num_pqs);
qm_info->num_vf_pqs = num_vfs;
- qm_info->start_vport = (u8) RESC_START(p_hwfn, QED_VPORT);
+ for (vf_idx = 0; vf_idx < num_vfs; vf_idx++)
+ qed_init_qm_pq(p_hwfn,
+ qm_info, PQ_INIT_DEFAULT_TC, PQ_INIT_VF_RL);
+}
- for (i = 0; i < qm_info->num_vports; i++)
- qm_info->qm_vport_params[i].vport_wfq = 1;
+static void qed_init_qm_rl_pqs(struct qed_hwfn *p_hwfn)
+{
+ u16 pf_rls_idx, num_pf_rls = qed_init_qm_get_num_pf_rls(p_hwfn);
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
- qm_info->vport_rl_en = 1;
- qm_info->vport_wfq_en = 1;
- qm_info->pf_rl = pf_rl;
- qm_info->pf_wfq = pf_wfq;
+ if (!(qed_get_pq_flags(p_hwfn) & PQ_FLAGS_RLS))
+ return;
+
+ qed_init_qm_set_idx(p_hwfn, PQ_FLAGS_RLS, qm_info->num_pqs);
+ for (pf_rls_idx = 0; pf_rls_idx < num_pf_rls; pf_rls_idx++)
+ qed_init_qm_pq(p_hwfn, qm_info, PQ_INIT_OFLD_TC, PQ_INIT_PF_RL);
+}
+
+static void qed_init_qm_pq_params(struct qed_hwfn *p_hwfn)
+{
+ /* rate limited pqs, must come first (FW assumption) */
+ qed_init_qm_rl_pqs(p_hwfn);
+
+ /* pqs for multi cos */
+ qed_init_qm_mcos_pqs(p_hwfn);
+
+ /* pure loopback pq */
+ qed_init_qm_lb_pq(p_hwfn);
+
+ /* out of order pq */
+ qed_init_qm_ooo_pq(p_hwfn);
+
+ /* pure ack pq */
+ qed_init_qm_pure_ack_pq(p_hwfn);
+
+ /* pq for offloaded protocol */
+ qed_init_qm_offload_pq(p_hwfn);
+
+ /* low latency pq */
+ qed_init_qm_low_latency_pq(p_hwfn);
+
+ /* done sharing vports */
+ qed_init_qm_advance_vport(p_hwfn);
+
+ /* pqs for vfs */
+ qed_init_qm_vf_pqs(p_hwfn);
+}
+
+/* compare values of getters against resources amounts */
+static int qed_init_qm_sanity(struct qed_hwfn *p_hwfn)
+{
+ if (qed_init_qm_get_num_vports(p_hwfn) > RESC_NUM(p_hwfn, QED_VPORT)) {
+ DP_ERR(p_hwfn, "requested amount of vports exceeds resource\n");
+ return -EINVAL;
+ }
+
+ if (qed_init_qm_get_num_pqs(p_hwfn) > RESC_NUM(p_hwfn, QED_PQ)) {
+ DP_ERR(p_hwfn, "requested amount of pqs exceeds resource\n");
+ return -EINVAL;
+ }
return 0;
+}
-alloc_err:
- qed_qm_info_free(p_hwfn);
- return -ENOMEM;
+static void qed_dp_init_qm_params(struct qed_hwfn *p_hwfn)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
+ struct init_qm_vport_params *vport;
+ struct init_qm_port_params *port;
+ struct init_qm_pq_params *pq;
+ int i, tc;
+
+ /* top level params */
+ DP_VERBOSE(p_hwfn,
+ NETIF_MSG_HW,
+ "qm init top level params: start_pq %d, start_vport %d, pure_lb_pq %d, offload_pq %d, pure_ack_pq %d\n",
+ qm_info->start_pq,
+ qm_info->start_vport,
+ qm_info->pure_lb_pq,
+ qm_info->offload_pq, qm_info->pure_ack_pq);
+ DP_VERBOSE(p_hwfn,
+ NETIF_MSG_HW,
+ "ooo_pq %d, first_vf_pq %d, num_pqs %d, num_vf_pqs %d, num_vports %d, max_phys_tcs_per_port %d\n",
+ qm_info->ooo_pq,
+ qm_info->first_vf_pq,
+ qm_info->num_pqs,
+ qm_info->num_vf_pqs,
+ qm_info->num_vports, qm_info->max_phys_tcs_per_port);
+ DP_VERBOSE(p_hwfn,
+ NETIF_MSG_HW,
+ "pf_rl_en %d, pf_wfq_en %d, vport_rl_en %d, vport_wfq_en %d, pf_wfq %d, pf_rl %d, num_pf_rls %d, pq_flags %x\n",
+ qm_info->pf_rl_en,
+ qm_info->pf_wfq_en,
+ qm_info->vport_rl_en,
+ qm_info->vport_wfq_en,
+ qm_info->pf_wfq,
+ qm_info->pf_rl,
+ qm_info->num_pf_rls, qed_get_pq_flags(p_hwfn));
+
+ /* port table */
+ for (i = 0; i < p_hwfn->cdev->num_ports_in_engines; i++) {
+ port = &(qm_info->qm_port_params[i]);
+ DP_VERBOSE(p_hwfn,
+ NETIF_MSG_HW,
+ "port idx %d, active %d, active_phys_tcs %d, num_pbf_cmd_lines %d, num_btb_blocks %d, reserved %d\n",
+ i,
+ port->active,
+ port->active_phys_tcs,
+ port->num_pbf_cmd_lines,
+ port->num_btb_blocks, port->reserved);
+ }
+
+ /* vport table */
+ for (i = 0; i < qm_info->num_vports; i++) {
+ vport = &(qm_info->qm_vport_params[i]);
+ DP_VERBOSE(p_hwfn,
+ NETIF_MSG_HW,
+ "vport idx %d, vport_rl %d, wfq %d, first_tx_pq_id [ ",
+ qm_info->start_vport + i,
+ vport->vport_rl, vport->vport_wfq);
+ for (tc = 0; tc < NUM_OF_TCS; tc++)
+ DP_VERBOSE(p_hwfn,
+ NETIF_MSG_HW,
+ "%d ", vport->first_tx_pq_id[tc]);
+ DP_VERBOSE(p_hwfn, NETIF_MSG_HW, "]\n");
+ }
+
+ /* pq table */
+ for (i = 0; i < qm_info->num_pqs; i++) {
+ pq = &(qm_info->qm_pq_params[i]);
+ DP_VERBOSE(p_hwfn,
+ NETIF_MSG_HW,
+ "pq idx %d, vport_id %d, tc %d, wrr_grp %d, rl_valid %d\n",
+ qm_info->start_pq + i,
+ pq->vport_id,
+ pq->tc_id, pq->wrr_group, pq->rl_valid);
+ }
+}
+
+static void qed_init_qm_info(struct qed_hwfn *p_hwfn)
+{
+ /* reset params required for init run */
+ qed_init_qm_reset_params(p_hwfn);
+
+ /* init QM top level params */
+ qed_init_qm_params(p_hwfn);
+
+ /* init QM port params */
+ qed_init_qm_port_params(p_hwfn);
+
+ /* init QM vport params */
+ qed_init_qm_vport_params(p_hwfn);
+
+ /* init QM physical queue params */
+ qed_init_qm_pq_params(p_hwfn);
+
+ /* display all that init */
+ qed_dp_init_qm_params(p_hwfn);
}
/* This function reconfigures the QM pf on the fly.
@@ -391,17 +766,8 @@
bool b_rc;
int rc;
- /* qm_info is allocated in qed_init_qm_info() which is already called
- * from qed_resc_alloc() or previous call of qed_qm_reconf().
- * The allocated size may change each init, so we free it before next
- * allocation.
- */
- qed_qm_info_free(p_hwfn);
-
/* initialize qed's qm data structure */
- rc = qed_init_qm_info(p_hwfn, false);
- if (rc)
- return rc;
+ qed_init_qm_info(p_hwfn);
/* stop PF's qm queues */
spin_lock_bh(&qm_lock);
@@ -415,7 +781,7 @@
qed_init_clear_rt_data(p_hwfn);
/* prepare QM portion of runtime array */
- qed_qm_init_pf(p_hwfn);
+ qed_qm_init_pf(p_hwfn, p_ptt);
/* activate init tool on runtime array */
rc = qed_init_run(p_hwfn, p_ptt, PHASE_QM_PF, p_hwfn->rel_pf_id,
@@ -434,6 +800,47 @@
return 0;
}
+static int qed_alloc_qm_data(struct qed_hwfn *p_hwfn)
+{
+ struct qed_qm_info *qm_info = &p_hwfn->qm_info;
+ int rc;
+
+ rc = qed_init_qm_sanity(p_hwfn);
+ if (rc)
+ goto alloc_err;
+
+ qm_info->qm_pq_params = kzalloc(sizeof(*qm_info->qm_pq_params) *
+ qed_init_qm_get_num_pqs(p_hwfn),
+ GFP_KERNEL);
+ if (!qm_info->qm_pq_params)
+ goto alloc_err;
+
+ qm_info->qm_vport_params = kzalloc(sizeof(*qm_info->qm_vport_params) *
+ qed_init_qm_get_num_vports(p_hwfn),
+ GFP_KERNEL);
+ if (!qm_info->qm_vport_params)
+ goto alloc_err;
+
+ qm_info->qm_port_params = kzalloc(sizeof(*qm_info->qm_port_params) *
+ p_hwfn->cdev->num_ports_in_engines,
+ GFP_KERNEL);
+ if (!qm_info->qm_port_params)
+ goto alloc_err;
+
+ qm_info->wfq_data = kzalloc(sizeof(*qm_info->wfq_data) *
+ qed_init_qm_get_num_vports(p_hwfn),
+ GFP_KERNEL);
+ if (!qm_info->wfq_data)
+ goto alloc_err;
+
+ return 0;
+
+alloc_err:
+ DP_NOTICE(p_hwfn, "Failed to allocate memory for QM params\n");
+ qed_qm_info_free(p_hwfn);
+ return -ENOMEM;
+}
+
int qed_resc_alloc(struct qed_dev *cdev)
{
struct qed_iscsi_info *p_iscsi_info;
@@ -442,8 +849,10 @@
#ifdef CONFIG_QED_LL2
struct qed_ll2_info *p_ll2_info;
#endif
+ u32 rdma_tasks, excess_tasks;
struct qed_consq *p_consq;
struct qed_eq *p_eq;
+ u32 line_count;
int i, rc = 0;
if (IS_VF(cdev))
@@ -465,19 +874,44 @@
/* Set the HW cid/tid numbers (in the contest manager)
* Must be done prior to any further computations.
*/
- rc = qed_cxt_set_pf_params(p_hwfn);
+ rc = qed_cxt_set_pf_params(p_hwfn, RDMA_MAX_TIDS);
if (rc)
goto alloc_err;
- /* Prepare and process QM requirements */
- rc = qed_init_qm_info(p_hwfn, true);
+ rc = qed_alloc_qm_data(p_hwfn);
if (rc)
goto alloc_err;
+ /* init qm info */
+ qed_init_qm_info(p_hwfn);
+
/* Compute the ILT client partition */
- rc = qed_cxt_cfg_ilt_compute(p_hwfn);
- if (rc)
- goto alloc_err;
+ rc = qed_cxt_cfg_ilt_compute(p_hwfn, &line_count);
+ if (rc) {
+ DP_NOTICE(p_hwfn,
+ "too many ILT lines; re-computing with less lines\n");
+ /* In case there are not enough ILT lines we reduce the
+ * number of RDMA tasks and re-compute.
+ */
+ excess_tasks =
+ qed_cxt_cfg_ilt_compute_excess(p_hwfn, line_count);
+ if (!excess_tasks)
+ goto alloc_err;
+
+ rdma_tasks = RDMA_MAX_TIDS - excess_tasks;
+ rc = qed_cxt_set_pf_params(p_hwfn, rdma_tasks);
+ if (rc)
+ goto alloc_err;
+
+ rc = qed_cxt_cfg_ilt_compute(p_hwfn, &line_count);
+ if (rc) {
+ DP_ERR(p_hwfn,
+ "failed ILT compute. Requested too many lines: %u\n",
+ line_count);
+
+ goto alloc_err;
+ }
+ }
/* CID map / ILT shadow table / T2
* The talbes sizes are determined by the computations above
@@ -674,11 +1108,19 @@
return rc;
}
-static void qed_calc_hw_mode(struct qed_hwfn *p_hwfn)
+static int qed_calc_hw_mode(struct qed_hwfn *p_hwfn)
{
int hw_mode = 0;
- hw_mode = (1 << MODE_BB_B0);
+ if (QED_IS_BB_B0(p_hwfn->cdev)) {
+ hw_mode |= 1 << MODE_BB;
+ } else if (QED_IS_AH(p_hwfn->cdev)) {
+ hw_mode |= 1 << MODE_K2;
+ } else {
+ DP_NOTICE(p_hwfn, "Unknown chip type %#x\n",
+ p_hwfn->cdev->type);
+ return -EINVAL;
+ }
switch (p_hwfn->cdev->num_ports_in_engines) {
case 1:
@@ -693,7 +1135,7 @@
default:
DP_NOTICE(p_hwfn, "num_ports_in_engine = %d not supported\n",
p_hwfn->cdev->num_ports_in_engines);
- return;
+ return -EINVAL;
}
switch (p_hwfn->cdev->mf_mode) {
@@ -719,6 +1161,8 @@
DP_VERBOSE(p_hwfn, (NETIF_MSG_PROBE | NETIF_MSG_IFUP),
"Configuring function for hw_mode: 0x%08x\n",
p_hwfn->hw_info.hw_mode);
+
+ return 0;
}
/* Init run time data for all PFs on an engine. */
@@ -748,16 +1192,67 @@
}
}
+static void qed_init_cache_line_size(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt)
+{
+ u32 val, wr_mbs, cache_line_size;
+
+ val = qed_rd(p_hwfn, p_ptt, PSWRQ2_REG_WR_MBS0);
+ switch (val) {
+ case 0:
+ wr_mbs = 128;
+ break;
+ case 1:
+ wr_mbs = 256;
+ break;
+ case 2:
+ wr_mbs = 512;
+ break;
+ default:
+ DP_INFO(p_hwfn,
+ "Unexpected value of PSWRQ2_REG_WR_MBS0 [0x%x]. Avoid configuring PGLUE_B_REG_CACHE_LINE_SIZE.\n",
+ val);
+ return;
+ }
+
+ cache_line_size = min_t(u32, L1_CACHE_BYTES, wr_mbs);
+ switch (cache_line_size) {
+ case 32:
+ val = 0;
+ break;
+ case 64:
+ val = 1;
+ break;
+ case 128:
+ val = 2;
+ break;
+ case 256:
+ val = 3;
+ break;
+ default:
+ DP_INFO(p_hwfn,
+ "Unexpected value of cache line size [0x%x]. Avoid configuring PGLUE_B_REG_CACHE_LINE_SIZE.\n",
+ cache_line_size);
+ }
+
+ if (L1_CACHE_BYTES > wr_mbs)
+ DP_INFO(p_hwfn,
+ "The cache line size for padding is suboptimal for performance [OS cache line size 0x%x, wr mbs 0x%x]\n",
+ L1_CACHE_BYTES, wr_mbs);
+
+ STORE_RT_REG(p_hwfn, PGLUE_REG_B_CACHE_LINE_SIZE_RT_OFFSET, val);
+}
+
static int qed_hw_init_common(struct qed_hwfn *p_hwfn,
struct qed_ptt *p_ptt, int hw_mode)
{
struct qed_qm_info *qm_info = &p_hwfn->qm_info;
struct qed_qm_common_rt_init_params params;
struct qed_dev *cdev = p_hwfn->cdev;
+ u8 vf_id, max_num_vfs;
u16 num_pfs, pf_id;
u32 concrete_fid;
int rc = 0;
- u8 vf_id;
qed_init_cau_rt_data(cdev);
@@ -784,17 +1279,7 @@
qed_cxt_hw_init_common(p_hwfn);
- /* Close gate from NIG to BRB/Storm; By default they are open, but
- * we close them to prevent NIG from passing data to reset blocks.
- * Should have been done in the ENGINE phase, but init-tool lacks
- * proper port-pretend capabilities.
- */
- qed_wr(p_hwfn, p_ptt, NIG_REG_RX_BRB_OUT_EN, 0);
- qed_wr(p_hwfn, p_ptt, NIG_REG_STORM_OUT_EN, 0);
- qed_port_pretend(p_hwfn, p_ptt, p_hwfn->port_id ^ 1);
- qed_wr(p_hwfn, p_ptt, NIG_REG_RX_BRB_OUT_EN, 0);
- qed_wr(p_hwfn, p_ptt, NIG_REG_STORM_OUT_EN, 0);
- qed_port_unpretend(p_hwfn, p_ptt);
+ qed_init_cache_line_size(p_hwfn, p_ptt);
rc = qed_init_run(p_hwfn, p_ptt, PHASE_ENGINE, ANY_PHASE_ID, hw_mode);
if (rc)
@@ -814,7 +1299,8 @@
qed_fid_pretend(p_hwfn, p_ptt, p_hwfn->rel_pf_id);
}
- for (vf_id = 0; vf_id < MAX_NUM_VFS_BB; vf_id++) {
+ max_num_vfs = QED_IS_AH(cdev) ? MAX_NUM_VFS_K2 : MAX_NUM_VFS_BB;
+ for (vf_id = 0; vf_id < max_num_vfs; vf_id++) {
concrete_fid = qed_vfid_to_concrete(p_hwfn, vf_id);
qed_fid_pretend(p_hwfn, p_ptt, (u16) concrete_fid);
qed_wr(p_hwfn, p_ptt, CCFC_REG_STRONG_ENABLE_VF, 0x1);
@@ -832,17 +1318,15 @@
qed_hw_init_dpi_size(struct qed_hwfn *p_hwfn,
struct qed_ptt *p_ptt, u32 pwm_region_size, u32 n_cpus)
{
- u32 dpi_page_size_1, dpi_page_size_2, dpi_page_size;
- u32 dpi_bit_shift, dpi_count;
+ u32 dpi_bit_shift, dpi_count, dpi_page_size;
u32 min_dpis;
+ u32 n_wids;
/* Calculate DPI size */
- dpi_page_size_1 = QED_WID_SIZE * n_cpus;
- dpi_page_size_2 = max_t(u32, QED_WID_SIZE, PAGE_SIZE);
- dpi_page_size = max_t(u32, dpi_page_size_1, dpi_page_size_2);
- dpi_page_size = roundup_pow_of_two(dpi_page_size);
+ n_wids = max_t(u32, QED_MIN_WIDS, n_cpus);
+ dpi_page_size = QED_WID_SIZE * roundup_pow_of_two(n_wids);
+ dpi_page_size = (dpi_page_size + PAGE_SIZE - 1) & ~(PAGE_SIZE - 1);
dpi_bit_shift = ilog2(dpi_page_size / 4096);
-
dpi_count = pwm_region_size / dpi_page_size;
min_dpis = p_hwfn->pf_params.rdma_pf_params.min_dpis;
@@ -870,13 +1354,13 @@
{
u32 pwm_regsize, norm_regsize;
u32 non_pwm_conn, min_addr_reg1;
- u32 db_bar_size, n_cpus;
+ u32 db_bar_size, n_cpus = 1;
u32 roce_edpm_mode;
u32 pf_dems_shift;
int rc = 0;
u8 cond;
- db_bar_size = qed_hw_bar_size(p_hwfn, BAR_ID_1);
+ db_bar_size = qed_hw_bar_size(p_hwfn, p_ptt, BAR_ID_1);
if (p_hwfn->cdev->num_hwfns > 1)
db_bar_size /= 2;
@@ -931,6 +1415,8 @@
qed_rdma_dpm_bar(p_hwfn, p_ptt);
}
+ p_hwfn->wid_count = (u16) n_cpus;
+
DP_INFO(p_hwfn,
"doorbell bar: normal_region_size=%d, pwm_region_size=%d, dpi_size=%d, dpi_count=%d, roce_edpm=%s\n",
norm_regsize,
@@ -967,7 +1453,7 @@
static int qed_hw_init_pf(struct qed_hwfn *p_hwfn,
struct qed_ptt *p_ptt,
- struct qed_tunn_start_params *p_tunn,
+ struct qed_tunnel_info *p_tunn,
int hw_mode,
bool b_hw_start,
enum qed_int_mode int_mode,
@@ -987,7 +1473,7 @@
p_hwfn->qm_info.pf_rl = 100000;
}
- qed_cxt_hw_init_pf(p_hwfn);
+ qed_cxt_hw_init_pf(p_hwfn, p_ptt);
qed_int_igu_init_rt(p_hwfn);
@@ -1095,25 +1581,47 @@
p_hwfn->mcp_info->mfw_mb_cur, p_hwfn->mcp_info->mfw_mb_length);
}
-int qed_hw_init(struct qed_dev *cdev,
- struct qed_tunn_start_params *p_tunn,
- bool b_hw_start,
- enum qed_int_mode int_mode,
- bool allow_npar_tx_switch,
- const u8 *bin_fw_data)
+static void
+qed_fill_load_req_params(struct qed_load_req_params *p_load_req,
+ struct qed_drv_load_params *p_drv_load)
{
+ memset(p_load_req, 0, sizeof(*p_load_req));
+
+ p_load_req->drv_role = p_drv_load->is_crash_kernel ?
+ QED_DRV_ROLE_KDUMP : QED_DRV_ROLE_OS;
+ p_load_req->timeout_val = p_drv_load->mfw_timeout_val;
+ p_load_req->avoid_eng_reset = p_drv_load->avoid_eng_reset;
+ p_load_req->override_force_load = p_drv_load->override_force_load;
+}
+
+static int qed_vf_start(struct qed_hwfn *p_hwfn,
+ struct qed_hw_init_params *p_params)
+{
+ if (p_params->p_tunn) {
+ qed_vf_set_vf_start_tunn_update_param(p_params->p_tunn);
+ qed_vf_pf_tunnel_param_update(p_hwfn, p_params->p_tunn);
+ }
+
+ p_hwfn->b_int_enabled = 1;
+
+ return 0;
+}
+
+int qed_hw_init(struct qed_dev *cdev, struct qed_hw_init_params *p_params)
+{
+ struct qed_load_req_params load_req_params;
u32 load_code, param, drv_mb_param;
bool b_default_mtu = true;
struct qed_hwfn *p_hwfn;
int rc = 0, mfw_rc, i;
- if ((int_mode == QED_INT_MODE_MSI) && (cdev->num_hwfns > 1)) {
+ if ((p_params->int_mode == QED_INT_MODE_MSI) && (cdev->num_hwfns > 1)) {
DP_NOTICE(cdev, "MSI mode is not supported for CMT devices\n");
return -EINVAL;
}
if (IS_PF(cdev)) {
- rc = qed_init_fw_data(cdev, bin_fw_data);
+ rc = qed_init_fw_data(cdev, p_params->bin_fw_data);
if (rc)
return rc;
}
@@ -1128,26 +1636,32 @@
}
if (IS_VF(cdev)) {
- p_hwfn->b_int_enabled = 1;
+ qed_vf_start(p_hwfn, p_params);
continue;
}
/* Enable DMAE in PXP */
rc = qed_change_pci_hwfn(p_hwfn, p_hwfn->p_main_ptt, true);
- qed_calc_hw_mode(p_hwfn);
+ rc = qed_calc_hw_mode(p_hwfn);
+ if (rc)
+ return rc;
- rc = qed_mcp_load_req(p_hwfn, p_hwfn->p_main_ptt, &load_code);
+ qed_fill_load_req_params(&load_req_params,
+ p_params->p_drv_load_params);
+ rc = qed_mcp_load_req(p_hwfn, p_hwfn->p_main_ptt,
+ &load_req_params);
if (rc) {
- DP_NOTICE(p_hwfn, "Failed sending LOAD_REQ command\n");
+ DP_NOTICE(p_hwfn, "Failed sending a LOAD_REQ command\n");
return rc;
}
- qed_reset_mb_shadow(p_hwfn, p_hwfn->p_main_ptt);
-
+ load_code = load_req_params.load_code;
DP_VERBOSE(p_hwfn, QED_MSG_SP,
- "Load request was sent. Resp:0x%x, Load code: 0x%x\n",
- rc, load_code);
+ "Load request was sent. Load code: 0x%x\n",
+ load_code);
+
+ qed_reset_mb_shadow(p_hwfn, p_hwfn->p_main_ptt);
p_hwfn->first_on_engine = (load_code ==
FW_MSG_CODE_DRV_LOAD_ENGINE);
@@ -1168,11 +1682,15 @@
/* Fall into */
case FW_MSG_CODE_DRV_LOAD_FUNCTION:
rc = qed_hw_init_pf(p_hwfn, p_hwfn->p_main_ptt,
- p_tunn, p_hwfn->hw_info.hw_mode,
- b_hw_start, int_mode,
- allow_npar_tx_switch);
+ p_params->p_tunn,
+ p_hwfn->hw_info.hw_mode,
+ p_params->b_hw_start,
+ p_params->int_mode,
+ p_params->allow_npar_tx_switch);
break;
default:
+ DP_NOTICE(p_hwfn,
+ "Unexpected load code [0x%08x]", load_code);
rc = -EINVAL;
break;
}
@@ -1212,10 +1730,7 @@
if (IS_PF(cdev)) {
p_hwfn = QED_LEADING_HWFN(cdev);
- drv_mb_param = (FW_MAJOR_VERSION << 24) |
- (FW_MINOR_VERSION << 16) |
- (FW_REVISION_VERSION << 8) |
- (FW_ENGINEERING_VERSION);
+ drv_mb_param = STORM_FW_VERSION;
rc = qed_mcp_cmd(p_hwfn, p_hwfn->p_main_ptt,
DRV_MSG_CODE_OV_UPDATE_STORM_FW_VER,
drv_mb_param, &load_code, ¶m);
@@ -1290,27 +1805,53 @@
int qed_hw_stop(struct qed_dev *cdev)
{
- int rc = 0, t_rc;
+ struct qed_hwfn *p_hwfn;
+ struct qed_ptt *p_ptt;
+ int rc, rc2 = 0;
int j;
for_each_hwfn(cdev, j) {
- struct qed_hwfn *p_hwfn = &cdev->hwfns[j];
- struct qed_ptt *p_ptt = p_hwfn->p_main_ptt;
+ p_hwfn = &cdev->hwfns[j];
+ p_ptt = p_hwfn->p_main_ptt;
DP_VERBOSE(p_hwfn, NETIF_MSG_IFDOWN, "Stopping hw/fw\n");
if (IS_VF(cdev)) {
qed_vf_pf_int_cleanup(p_hwfn);
+ rc = qed_vf_pf_reset(p_hwfn);
+ if (rc) {
+ DP_NOTICE(p_hwfn,
+ "qed_vf_pf_reset failed. rc = %d.\n",
+ rc);
+ rc2 = -EINVAL;
+ }
continue;
}
/* mark the hw as uninitialized... */
p_hwfn->hw_init_done = false;
- rc = qed_sp_pf_stop(p_hwfn);
- if (rc)
+ /* Send unload command to MCP */
+ rc = qed_mcp_unload_req(p_hwfn, p_ptt);
+ if (rc) {
DP_NOTICE(p_hwfn,
- "Failed to close PF against FW. Continue to stop HW to prevent illegal host access by the device\n");
+ "Failed sending a UNLOAD_REQ command. rc = %d.\n",
+ rc);
+ rc2 = -EINVAL;
+ }
+
+ qed_slowpath_irq_sync(p_hwfn);
+
+ /* After this point no MFW attentions are expected, e.g. prevent
+ * race between pf stop and dcbx pf update.
+ */
+ rc = qed_sp_pf_stop(p_hwfn);
+ if (rc) {
+ DP_NOTICE(p_hwfn,
+ "Failed to close PF against FW [rc = %d]. Continue to stop HW to prevent illegal host access by the device.\n",
+ rc);
+ rc2 = -EINVAL;
+ }
qed_wr(p_hwfn, p_ptt,
NIG_REG_RX_LLH_BRB_GATE_DNTFWD_PERPF, 0x1);
@@ -1333,34 +1874,54 @@
/* Need to wait 1ms to guarantee SBs are cleared */
usleep_range(1000, 2000);
+
+ /* Disable PF in HW blocks */
+ qed_wr(p_hwfn, p_ptt, DORQ_REG_PF_DB_ENABLE, 0);
+ qed_wr(p_hwfn, p_ptt, QM_REG_PF_EN, 0);
+
+ qed_mcp_unload_done(p_hwfn, p_ptt);
+ if (rc) {
+ DP_NOTICE(p_hwfn,
+ "Failed sending a UNLOAD_DONE command. rc = %d.\n",
+ rc);
+ rc2 = -EINVAL;
+ }
}
if (IS_PF(cdev)) {
+ p_hwfn = QED_LEADING_HWFN(cdev);
+ p_ptt = QED_LEADING_HWFN(cdev)->p_main_ptt;
+
/* Disable DMAE in PXP - in CMT, this should only be done for
* first hw-function, and only after all transactions have
* stopped for all active hw-functions.
*/
- t_rc = qed_change_pci_hwfn(&cdev->hwfns[0],
- cdev->hwfns[0].p_main_ptt, false);
- if (t_rc != 0)
- rc = t_rc;
+ rc = qed_change_pci_hwfn(p_hwfn, p_ptt, false);
+ if (rc) {
+ DP_NOTICE(p_hwfn,
+ "qed_change_pci_hwfn failed. rc = %d.\n", rc);
+ rc2 = -EINVAL;
+ }
}
- return rc;
+ return rc2;
}
-void qed_hw_stop_fastpath(struct qed_dev *cdev)
+int qed_hw_stop_fastpath(struct qed_dev *cdev)
{
int j;
for_each_hwfn(cdev, j) {
struct qed_hwfn *p_hwfn = &cdev->hwfns[j];
- struct qed_ptt *p_ptt = p_hwfn->p_main_ptt;
+ struct qed_ptt *p_ptt;
if (IS_VF(cdev)) {
qed_vf_pf_int_cleanup(p_hwfn);
continue;
}
+ p_ptt = qed_ptt_acquire(p_hwfn);
+ if (!p_ptt)
+ return -EAGAIN;
DP_VERBOSE(p_hwfn,
NETIF_MSG_IFDOWN, "Shutting down the fastpath\n");
@@ -1378,100 +1939,28 @@
/* Need to wait 1ms to guarantee SBs are cleared */
usleep_range(1000, 2000);
- }
-}
-
-void qed_hw_start_fastpath(struct qed_hwfn *p_hwfn)
-{
- if (IS_VF(p_hwfn->cdev))
- return;
-
- /* Re-open incoming traffic */
- qed_wr(p_hwfn, p_hwfn->p_main_ptt,
- NIG_REG_RX_LLH_BRB_GATE_DNTFWD_PERPF, 0x0);
-}
-
-static int qed_reg_assert(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt, u32 reg, bool expected)
-{
- u32 assert_val = qed_rd(p_hwfn, p_ptt, reg);
-
- if (assert_val != expected) {
- DP_NOTICE(p_hwfn, "Value at address 0x%08x != 0x%08x\n",
- reg, expected);
- return -EINVAL;
+ qed_ptt_release(p_hwfn, p_ptt);
}
return 0;
}
-int qed_hw_reset(struct qed_dev *cdev)
+int qed_hw_start_fastpath(struct qed_hwfn *p_hwfn)
{
- int rc = 0;
- u32 unload_resp, unload_param;
- u32 wol_param;
- int i;
+ struct qed_ptt *p_ptt;
- switch (cdev->wol_config) {
- case QED_OV_WOL_DISABLED:
- wol_param = DRV_MB_PARAM_UNLOAD_WOL_DISABLED;
- break;
- case QED_OV_WOL_ENABLED:
- wol_param = DRV_MB_PARAM_UNLOAD_WOL_ENABLED;
- break;
- default:
- DP_NOTICE(cdev,
- "Unknown WoL configuration %02x\n", cdev->wol_config);
- /* Fallthrough */
- case QED_OV_WOL_DEFAULT:
- wol_param = DRV_MB_PARAM_UNLOAD_WOL_MCP;
- }
+ if (IS_VF(p_hwfn->cdev))
+ return 0;
- for_each_hwfn(cdev, i) {
- struct qed_hwfn *p_hwfn = &cdev->hwfns[i];
+ p_ptt = qed_ptt_acquire(p_hwfn);
+ if (!p_ptt)
+ return -EAGAIN;
- if (IS_VF(cdev)) {
- rc = qed_vf_pf_reset(p_hwfn);
- if (rc)
- return rc;
- continue;
- }
+ /* Re-open incoming traffic */
+ qed_wr(p_hwfn, p_ptt, NIG_REG_RX_LLH_BRB_GATE_DNTFWD_PERPF, 0x0);
+ qed_ptt_release(p_hwfn, p_ptt);
- DP_VERBOSE(p_hwfn, NETIF_MSG_IFDOWN, "Resetting hw/fw\n");
-
- /* Check for incorrect states */
- qed_reg_assert(p_hwfn, p_hwfn->p_main_ptt,
- QM_REG_USG_CNT_PF_TX, 0);
- qed_reg_assert(p_hwfn, p_hwfn->p_main_ptt,
- QM_REG_USG_CNT_PF_OTHER, 0);
-
- /* Disable PF in HW blocks */
- qed_wr(p_hwfn, p_hwfn->p_main_ptt, DORQ_REG_PF_DB_ENABLE, 0);
- qed_wr(p_hwfn, p_hwfn->p_main_ptt, QM_REG_PF_EN, 0);
- qed_wr(p_hwfn, p_hwfn->p_main_ptt,
- TCFC_REG_STRONG_ENABLE_PF, 0);
- qed_wr(p_hwfn, p_hwfn->p_main_ptt,
- CCFC_REG_STRONG_ENABLE_PF, 0);
-
- /* Send unload command to MCP */
- rc = qed_mcp_cmd(p_hwfn, p_hwfn->p_main_ptt,
- DRV_MSG_CODE_UNLOAD_REQ, wol_param,
- &unload_resp, &unload_param);
- if (rc) {
- DP_NOTICE(p_hwfn, "qed_hw_reset: UNLOAD_REQ failed\n");
- unload_resp = FW_MSG_CODE_DRV_UNLOAD_ENGINE;
- }
-
- rc = qed_mcp_cmd(p_hwfn, p_hwfn->p_main_ptt,
- DRV_MSG_CODE_UNLOAD_DONE,
- 0, &unload_resp, &unload_param);
- if (rc) {
- DP_NOTICE(p_hwfn, "qed_hw_reset: UNLOAD_DONE failed\n");
- return rc;
- }
- }
-
- return rc;
+ return 0;
}
/* Free hwfn memory and resources acquired in hw_hwfn_prepare */
@@ -1485,10 +1974,25 @@
static void qed_hw_hwfn_prepare(struct qed_hwfn *p_hwfn)
{
/* clear indirect access */
- qed_wr(p_hwfn, p_hwfn->p_main_ptt, PGLUE_B_REG_PGL_ADDR_88_F0, 0);
- qed_wr(p_hwfn, p_hwfn->p_main_ptt, PGLUE_B_REG_PGL_ADDR_8C_F0, 0);
- qed_wr(p_hwfn, p_hwfn->p_main_ptt, PGLUE_B_REG_PGL_ADDR_90_F0, 0);
- qed_wr(p_hwfn, p_hwfn->p_main_ptt, PGLUE_B_REG_PGL_ADDR_94_F0, 0);
+ if (QED_IS_AH(p_hwfn->cdev)) {
+ qed_wr(p_hwfn, p_hwfn->p_main_ptt,
+ PGLUE_B_REG_PGL_ADDR_E8_F0_K2, 0);
+ qed_wr(p_hwfn, p_hwfn->p_main_ptt,
+ PGLUE_B_REG_PGL_ADDR_EC_F0_K2, 0);
+ qed_wr(p_hwfn, p_hwfn->p_main_ptt,
+ PGLUE_B_REG_PGL_ADDR_F0_F0_K2, 0);
+ qed_wr(p_hwfn, p_hwfn->p_main_ptt,
+ PGLUE_B_REG_PGL_ADDR_F4_F0_K2, 0);
+ } else {
+ qed_wr(p_hwfn, p_hwfn->p_main_ptt,
+ PGLUE_B_REG_PGL_ADDR_88_F0_BB, 0);
+ qed_wr(p_hwfn, p_hwfn->p_main_ptt,
+ PGLUE_B_REG_PGL_ADDR_8C_F0_BB, 0);
+ qed_wr(p_hwfn, p_hwfn->p_main_ptt,
+ PGLUE_B_REG_PGL_ADDR_90_F0_BB, 0);
+ qed_wr(p_hwfn, p_hwfn->p_main_ptt,
+ PGLUE_B_REG_PGL_ADDR_94_F0_BB, 0);
+ }
/* Clean Previous errors if such exist */
qed_wr(p_hwfn, p_hwfn->p_main_ptt,
@@ -1522,7 +2026,7 @@
{
u32 *feat_num = p_hwfn->hw_info.feat_num;
struct qed_sb_cnt_info sb_cnt_info;
- int num_features = 1;
+ u32 non_l2_sbs = 0;
if (IS_ENABLED(CONFIG_QED_RDMA) &&
p_hwfn->hw_info.personality == QED_PCI_ETH_ROCE) {
@@ -1530,143 +2034,47 @@
* the status blocks equally between L2 / RoCE but with
* consideration as to how many l2 queues / cnqs we have.
*/
- num_features++;
-
feat_num[QED_RDMA_CNQ] =
- min_t(u32, RESC_NUM(p_hwfn, QED_SB) / num_features,
+ min_t(u32, RESC_NUM(p_hwfn, QED_SB) / 2,
RESC_NUM(p_hwfn, QED_RDMA_CNQ_RAM));
+
+ non_l2_sbs = feat_num[QED_RDMA_CNQ];
}
- feat_num[QED_PF_L2_QUE] = min_t(u32, RESC_NUM(p_hwfn, QED_SB) /
- num_features,
- RESC_NUM(p_hwfn, QED_L2_QUEUE));
+ if (p_hwfn->hw_info.personality == QED_PCI_ETH_ROCE ||
+ p_hwfn->hw_info.personality == QED_PCI_ETH) {
+ /* Start by allocating VF queues, then PF's */
+ memset(&sb_cnt_info, 0, sizeof(sb_cnt_info));
+ qed_int_get_num_sbs(p_hwfn, &sb_cnt_info);
+ feat_num[QED_VF_L2_QUE] = min_t(u32,
+ RESC_NUM(p_hwfn, QED_L2_QUEUE),
+ sb_cnt_info.sb_iov_cnt);
+ feat_num[QED_PF_L2_QUE] = min_t(u32,
+ RESC_NUM(p_hwfn, QED_SB) -
+ non_l2_sbs,
+ RESC_NUM(p_hwfn,
+ QED_L2_QUEUE) -
+ FEAT_NUM(p_hwfn,
+ QED_VF_L2_QUE));
+ }
- memset(&sb_cnt_info, 0, sizeof(sb_cnt_info));
- qed_int_get_num_sbs(p_hwfn, &sb_cnt_info);
- feat_num[QED_VF_L2_QUE] =
- min_t(u32,
- RESC_NUM(p_hwfn, QED_L2_QUEUE) -
- FEAT_NUM(p_hwfn, QED_PF_L2_QUE), sb_cnt_info.sb_iov_cnt);
-
+ if (p_hwfn->hw_info.personality == QED_PCI_ISCSI)
+ feat_num[QED_ISCSI_CQ] = min_t(u32, RESC_NUM(p_hwfn, QED_SB),
+ RESC_NUM(p_hwfn,
+ QED_CMDQS_CQS));
DP_VERBOSE(p_hwfn,
NETIF_MSG_PROBE,
- "#PF_L2_QUEUES=%d VF_L2_QUEUES=%d #ROCE_CNQ=%d #SBS=%d num_features=%d\n",
+ "#PF_L2_QUEUES=%d VF_L2_QUEUES=%d #ROCE_CNQ=%d ISCSI_CQ=%d #SBS=%d\n",
(int)FEAT_NUM(p_hwfn, QED_PF_L2_QUE),
(int)FEAT_NUM(p_hwfn, QED_VF_L2_QUE),
(int)FEAT_NUM(p_hwfn, QED_RDMA_CNQ),
- RESC_NUM(p_hwfn, QED_SB), num_features);
+ (int)FEAT_NUM(p_hwfn, QED_ISCSI_CQ),
+ RESC_NUM(p_hwfn, QED_SB));
}
-static enum resource_id_enum qed_hw_get_mfw_res_id(enum qed_resources res_id)
-{
- enum resource_id_enum mfw_res_id = RESOURCE_NUM_INVALID;
-
- switch (res_id) {
- case QED_SB:
- mfw_res_id = RESOURCE_NUM_SB_E;
- break;
- case QED_L2_QUEUE:
- mfw_res_id = RESOURCE_NUM_L2_QUEUE_E;
- break;
- case QED_VPORT:
- mfw_res_id = RESOURCE_NUM_VPORT_E;
- break;
- case QED_RSS_ENG:
- mfw_res_id = RESOURCE_NUM_RSS_ENGINES_E;
- break;
- case QED_PQ:
- mfw_res_id = RESOURCE_NUM_PQ_E;
- break;
- case QED_RL:
- mfw_res_id = RESOURCE_NUM_RL_E;
- break;
- case QED_MAC:
- case QED_VLAN:
- /* Each VFC resource can accommodate both a MAC and a VLAN */
- mfw_res_id = RESOURCE_VFC_FILTER_E;
- break;
- case QED_ILT:
- mfw_res_id = RESOURCE_ILT_E;
- break;
- case QED_LL2_QUEUE:
- mfw_res_id = RESOURCE_LL2_QUEUE_E;
- break;
- case QED_RDMA_CNQ_RAM:
- case QED_CMDQS_CQS:
- /* CNQ/CMDQS are the same resource */
- mfw_res_id = RESOURCE_CQS_E;
- break;
- case QED_RDMA_STATS_QUEUE:
- mfw_res_id = RESOURCE_RDMA_STATS_QUEUE_E;
- break;
- default:
- break;
- }
-
- return mfw_res_id;
-}
-
-static u32 qed_hw_get_dflt_resc_num(struct qed_hwfn *p_hwfn,
- enum qed_resources res_id)
-{
- u8 num_funcs = p_hwfn->num_funcs_on_engine;
- struct qed_sb_cnt_info sb_cnt_info;
- u32 dflt_resc_num = 0;
-
- switch (res_id) {
- case QED_SB:
- memset(&sb_cnt_info, 0, sizeof(sb_cnt_info));
- qed_int_get_num_sbs(p_hwfn, &sb_cnt_info);
- dflt_resc_num = sb_cnt_info.sb_cnt;
- break;
- case QED_L2_QUEUE:
- dflt_resc_num = MAX_NUM_L2_QUEUES_BB / num_funcs;
- break;
- case QED_VPORT:
- dflt_resc_num = MAX_NUM_VPORTS_BB / num_funcs;
- break;
- case QED_RSS_ENG:
- dflt_resc_num = ETH_RSS_ENGINE_NUM_BB / num_funcs;
- break;
- case QED_PQ:
- /* The granularity of the PQs is 8 */
- dflt_resc_num = MAX_QM_TX_QUEUES_BB / num_funcs;
- dflt_resc_num &= ~0x7;
- break;
- case QED_RL:
- dflt_resc_num = MAX_QM_GLOBAL_RLS / num_funcs;
- break;
- case QED_MAC:
- case QED_VLAN:
- /* Each VFC resource can accommodate both a MAC and a VLAN */
- dflt_resc_num = ETH_NUM_MAC_FILTERS / num_funcs;
- break;
- case QED_ILT:
- dflt_resc_num = PXP_NUM_ILT_RECORDS_BB / num_funcs;
- break;
- case QED_LL2_QUEUE:
- dflt_resc_num = MAX_NUM_LL2_RX_QUEUES / num_funcs;
- break;
- case QED_RDMA_CNQ_RAM:
- case QED_CMDQS_CQS:
- /* CNQ/CMDQS are the same resource */
- dflt_resc_num = NUM_OF_CMDQS_CQS / num_funcs;
- break;
- case QED_RDMA_STATS_QUEUE:
- dflt_resc_num = RDMA_NUM_STATISTIC_COUNTERS_BB / num_funcs;
- break;
- default:
- break;
- }
-
- return dflt_resc_num;
-}
-
-static const char *qed_hw_get_resc_name(enum qed_resources res_id)
+const char *qed_hw_get_resc_name(enum qed_resources res_id)
{
switch (res_id) {
- case QED_SB:
- return "SB";
case QED_L2_QUEUE:
return "L2_QUEUE";
case QED_VPORT:
@@ -1691,43 +2099,195 @@
return "CMDQS_CQS";
case QED_RDMA_STATS_QUEUE:
return "RDMA_STATS_QUEUE";
+ case QED_BDQ:
+ return "BDQ";
+ case QED_SB:
+ return "SB";
default:
return "UNKNOWN_RESOURCE";
}
}
-static int qed_hw_set_resc_info(struct qed_hwfn *p_hwfn,
- enum qed_resources res_id)
+static int
+__qed_hw_set_soft_resc_size(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ enum qed_resources res_id,
+ u32 resc_max_val, u32 *p_mcp_resp)
{
- u32 dflt_resc_num = 0, dflt_resc_start = 0, mcp_resp, mcp_param;
- u32 *p_resc_num, *p_resc_start;
- struct resource_info resc_info;
+ int rc;
+
+ rc = qed_mcp_set_resc_max_val(p_hwfn, p_ptt, res_id,
+ resc_max_val, p_mcp_resp);
+ if (rc) {
+ DP_NOTICE(p_hwfn,
+ "MFW response failure for a max value setting of resource %d [%s]\n",
+ res_id, qed_hw_get_resc_name(res_id));
+ return rc;
+ }
+
+ if (*p_mcp_resp != FW_MSG_CODE_RESOURCE_ALLOC_OK)
+ DP_INFO(p_hwfn,
+ "Failed to set the max value of resource %d [%s]. mcp_resp = 0x%08x.\n",
+ res_id, qed_hw_get_resc_name(res_id), *p_mcp_resp);
+
+ return 0;
+}
+
+static int
+qed_hw_set_soft_resc_size(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
+{
+ bool b_ah = QED_IS_AH(p_hwfn->cdev);
+ u32 resc_max_val, mcp_resp;
+ u8 res_id;
+ int rc;
+
+ for (res_id = 0; res_id < QED_MAX_RESC; res_id++) {
+ switch (res_id) {
+ case QED_LL2_QUEUE:
+ resc_max_val = MAX_NUM_LL2_RX_QUEUES;
+ break;
+ case QED_RDMA_CNQ_RAM:
+ /* No need for a case for QED_CMDQS_CQS since
+ * CNQ/CMDQS are the same resource.
+ */
+ resc_max_val = NUM_OF_CMDQS_CQS;
+ break;
+ case QED_RDMA_STATS_QUEUE:
+ resc_max_val = b_ah ? RDMA_NUM_STATISTIC_COUNTERS_K2
+ : RDMA_NUM_STATISTIC_COUNTERS_BB;
+ break;
+ case QED_BDQ:
+ resc_max_val = BDQ_NUM_RESOURCES;
+ break;
+ default:
+ continue;
+ }
+
+ rc = __qed_hw_set_soft_resc_size(p_hwfn, p_ptt, res_id,
+ resc_max_val, &mcp_resp);
+ if (rc)
+ return rc;
+
+ /* There's no point to continue to the next resource if the
+ * command is not supported by the MFW.
+ * We do continue if the command is supported but the resource
+ * is unknown to the MFW. Such a resource will be later
+ * configured with the default allocation values.
+ */
+ if (mcp_resp == FW_MSG_CODE_UNSUPPORTED)
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static
+int qed_hw_get_dflt_resc(struct qed_hwfn *p_hwfn,
+ enum qed_resources res_id,
+ u32 *p_resc_num, u32 *p_resc_start)
+{
+ u8 num_funcs = p_hwfn->num_funcs_on_engine;
+ bool b_ah = QED_IS_AH(p_hwfn->cdev);
+ struct qed_sb_cnt_info sb_cnt_info;
+
+ switch (res_id) {
+ case QED_L2_QUEUE:
+ *p_resc_num = (b_ah ? MAX_NUM_L2_QUEUES_K2 :
+ MAX_NUM_L2_QUEUES_BB) / num_funcs;
+ break;
+ case QED_VPORT:
+ *p_resc_num = (b_ah ? MAX_NUM_VPORTS_K2 :
+ MAX_NUM_VPORTS_BB) / num_funcs;
+ break;
+ case QED_RSS_ENG:
+ *p_resc_num = (b_ah ? ETH_RSS_ENGINE_NUM_K2 :
+ ETH_RSS_ENGINE_NUM_BB) / num_funcs;
+ break;
+ case QED_PQ:
+ *p_resc_num = (b_ah ? MAX_QM_TX_QUEUES_K2 :
+ MAX_QM_TX_QUEUES_BB) / num_funcs;
+ *p_resc_num &= ~0x7; /* The granularity of the PQs is 8 */
+ break;
+ case QED_RL:
+ *p_resc_num = MAX_QM_GLOBAL_RLS / num_funcs;
+ break;
+ case QED_MAC:
+ case QED_VLAN:
+ /* Each VFC resource can accommodate both a MAC and a VLAN */
+ *p_resc_num = ETH_NUM_MAC_FILTERS / num_funcs;
+ break;
+ case QED_ILT:
+ *p_resc_num = (b_ah ? PXP_NUM_ILT_RECORDS_K2 :
+ PXP_NUM_ILT_RECORDS_BB) / num_funcs;
+ break;
+ case QED_LL2_QUEUE:
+ *p_resc_num = MAX_NUM_LL2_RX_QUEUES / num_funcs;
+ break;
+ case QED_RDMA_CNQ_RAM:
+ case QED_CMDQS_CQS:
+ /* CNQ/CMDQS are the same resource */
+ *p_resc_num = NUM_OF_CMDQS_CQS / num_funcs;
+ break;
+ case QED_RDMA_STATS_QUEUE:
+ *p_resc_num = (b_ah ? RDMA_NUM_STATISTIC_COUNTERS_K2 :
+ RDMA_NUM_STATISTIC_COUNTERS_BB) / num_funcs;
+ break;
+ case QED_BDQ:
+ if (p_hwfn->hw_info.personality != QED_PCI_ISCSI &&
+ p_hwfn->hw_info.personality != QED_PCI_FCOE)
+ *p_resc_num = 0;
+ else
+ *p_resc_num = 1;
+ break;
+ case QED_SB:
+ memset(&sb_cnt_info, 0, sizeof(sb_cnt_info));
+ qed_int_get_num_sbs(p_hwfn, &sb_cnt_info);
+ *p_resc_num = sb_cnt_info.sb_cnt;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ switch (res_id) {
+ case QED_BDQ:
+ if (!*p_resc_num)
+ *p_resc_start = 0;
+ else if (p_hwfn->cdev->num_ports_in_engines == 4)
+ *p_resc_start = p_hwfn->port_id;
+ else if (p_hwfn->hw_info.personality == QED_PCI_ISCSI)
+ *p_resc_start = p_hwfn->port_id;
+ else if (p_hwfn->hw_info.personality == QED_PCI_FCOE)
+ *p_resc_start = p_hwfn->port_id + 2;
+ break;
+ default:
+ *p_resc_start = *p_resc_num * p_hwfn->enabled_func_idx;
+ break;
+ }
+
+ return 0;
+}
+
+static int __qed_hw_set_resc_info(struct qed_hwfn *p_hwfn,
+ enum qed_resources res_id)
+{
+ u32 dflt_resc_num = 0, dflt_resc_start = 0;
+ u32 mcp_resp, *p_resc_num, *p_resc_start;
int rc;
p_resc_num = &RESC_NUM(p_hwfn, res_id);
p_resc_start = &RESC_START(p_hwfn, res_id);
- /* Default values assumes that each function received equal share */
- dflt_resc_num = qed_hw_get_dflt_resc_num(p_hwfn, res_id);
- if (!dflt_resc_num) {
+ rc = qed_hw_get_dflt_resc(p_hwfn, res_id, &dflt_resc_num,
+ &dflt_resc_start);
+ if (rc) {
DP_ERR(p_hwfn,
"Failed to get default amount for resource %d [%s]\n",
res_id, qed_hw_get_resc_name(res_id));
- return -EINVAL;
- }
- dflt_resc_start = dflt_resc_num * p_hwfn->enabled_func_idx;
-
- memset(&resc_info, 0, sizeof(resc_info));
- resc_info.res_id = qed_hw_get_mfw_res_id(res_id);
- if (resc_info.res_id == RESOURCE_NUM_INVALID) {
- DP_ERR(p_hwfn,
- "Failed to match resource %d [%s] with the MFW resources\n",
- res_id, qed_hw_get_resc_name(res_id));
- return -EINVAL;
+ return rc;
}
- rc = qed_mcp_get_resc_info(p_hwfn, p_hwfn->p_main_ptt, &resc_info,
- &mcp_resp, &mcp_param);
+ rc = qed_mcp_get_resc_info(p_hwfn, p_hwfn->p_main_ptt, res_id,
+ &mcp_resp, p_resc_num, p_resc_start);
if (rc) {
DP_NOTICE(p_hwfn,
"MFW response failure for an allocation request for resource %d [%s]\n",
@@ -1740,13 +2300,12 @@
* - There is an internal error in the MFW while processing the request
* - The resource ID is unknown to the MFW
*/
- if (mcp_resp != FW_MSG_CODE_RESOURCE_ALLOC_OK &&
- mcp_resp != FW_MSG_CODE_RESOURCE_ALLOC_DEPRECATED) {
- DP_NOTICE(p_hwfn,
- "Resource %d [%s]: No allocation info was received [mcp_resp 0x%x]. Applying default values [num %d, start %d].\n",
- res_id,
- qed_hw_get_resc_name(res_id),
- mcp_resp, dflt_resc_num, dflt_resc_start);
+ if (mcp_resp != FW_MSG_CODE_RESOURCE_ALLOC_OK) {
+ DP_INFO(p_hwfn,
+ "Failed to receive allocation info for resource %d [%s]. mcp_resp = 0x%x. Applying default values [%d,%d].\n",
+ res_id,
+ qed_hw_get_resc_name(res_id),
+ mcp_resp, dflt_resc_num, dflt_resc_start);
*p_resc_num = dflt_resc_num;
*p_resc_start = dflt_resc_start;
goto out;
@@ -1754,13 +2313,9 @@
/* Special handling for status blocks; Would be revised in future */
if (res_id == QED_SB) {
- resc_info.size -= 1;
- resc_info.offset -= p_hwfn->enabled_func_idx;
+ *p_resc_num -= 1;
+ *p_resc_start -= p_hwfn->enabled_func_idx;
}
-
- *p_resc_num = resc_info.size;
- *p_resc_start = resc_info.offset;
-
out:
/* PQs have to divide by 8 [that's the HW granularity].
* Reduce number so it would fit.
@@ -1778,19 +2333,80 @@
return 0;
}
-static int qed_hw_get_resc(struct qed_hwfn *p_hwfn)
+static int qed_hw_set_resc_info(struct qed_hwfn *p_hwfn)
{
- u8 res_id;
int rc;
+ u8 res_id;
for (res_id = 0; res_id < QED_MAX_RESC; res_id++) {
- rc = qed_hw_set_resc_info(p_hwfn, res_id);
+ rc = __qed_hw_set_resc_info(p_hwfn, res_id);
if (rc)
return rc;
}
+ return 0;
+}
+
+static int qed_hw_get_resc(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
+{
+ struct qed_resc_unlock_params resc_unlock_params;
+ struct qed_resc_lock_params resc_lock_params;
+ bool b_ah = QED_IS_AH(p_hwfn->cdev);
+ u8 res_id;
+ int rc;
+
+ /* Setting the max values of the soft resources and the following
+ * resources allocation queries should be atomic. Since several PFs can
+ * run in parallel - a resource lock is needed.
+ * If either the resource lock or resource set value commands are not
+ * supported - skip the the max values setting, release the lock if
+ * needed, and proceed to the queries. Other failures, including a
+ * failure to acquire the lock, will cause this function to fail.
+ */
+ qed_mcp_resc_lock_default_init(&resc_lock_params, &resc_unlock_params,
+ QED_RESC_LOCK_RESC_ALLOC, false);
+
+ rc = qed_mcp_resc_lock(p_hwfn, p_ptt, &resc_lock_params);
+ if (rc && rc != -EINVAL) {
+ return rc;
+ } else if (rc == -EINVAL) {
+ DP_INFO(p_hwfn,
+ "Skip the max values setting of the soft resources since the resource lock is not supported by the MFW\n");
+ } else if (!rc && !resc_lock_params.b_granted) {
+ DP_NOTICE(p_hwfn,
+ "Failed to acquire the resource lock for the resource allocation commands\n");
+ return -EBUSY;
+ } else {
+ rc = qed_hw_set_soft_resc_size(p_hwfn, p_ptt);
+ if (rc && rc != -EINVAL) {
+ DP_NOTICE(p_hwfn,
+ "Failed to set the max values of the soft resources\n");
+ goto unlock_and_exit;
+ } else if (rc == -EINVAL) {
+ DP_INFO(p_hwfn,
+ "Skip the max values setting of the soft resources since it is not supported by the MFW\n");
+ rc = qed_mcp_resc_unlock(p_hwfn, p_ptt,
+ &resc_unlock_params);
+ if (rc)
+ DP_INFO(p_hwfn,
+ "Failed to release the resource lock for the resource allocation commands\n");
+ }
+ }
+
+ rc = qed_hw_set_resc_info(p_hwfn);
+ if (rc)
+ goto unlock_and_exit;
+
+ if (resc_lock_params.b_granted && !resc_unlock_params.b_released) {
+ rc = qed_mcp_resc_unlock(p_hwfn, p_ptt, &resc_unlock_params);
+ if (rc)
+ DP_INFO(p_hwfn,
+ "Failed to release the resource lock for the resource allocation commands\n");
+ }
+
/* Sanity for ILT */
- if ((RESC_END(p_hwfn, QED_ILT) > PXP_NUM_ILT_RECORDS_BB)) {
+ if ((b_ah && (RESC_END(p_hwfn, QED_ILT) > PXP_NUM_ILT_RECORDS_K2)) ||
+ (!b_ah && (RESC_END(p_hwfn, QED_ILT) > PXP_NUM_ILT_RECORDS_BB))) {
DP_NOTICE(p_hwfn, "Can't assign ILT pages [%08x,...,%08x]\n",
RESC_START(p_hwfn, QED_ILT),
RESC_END(p_hwfn, QED_ILT) - 1);
@@ -1799,8 +2415,6 @@
qed_hw_set_feat(p_hwfn);
- DP_VERBOSE(p_hwfn, NETIF_MSG_PROBE,
- "The numbers for each resource are:\n");
for (res_id = 0; res_id < QED_MAX_RESC; res_id++)
DP_VERBOSE(p_hwfn, NETIF_MSG_PROBE, "%s = %d start = %d\n",
qed_hw_get_resc_name(res_id),
@@ -1808,6 +2422,11 @@
RESC_START(p_hwfn, res_id));
return 0;
+
+unlock_and_exit:
+ if (resc_lock_params.b_granted && !resc_unlock_params.b_released)
+ qed_mcp_resc_unlock(p_hwfn, p_ptt, &resc_unlock_params);
+ return rc;
}
static int qed_hw_get_nvm_info(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
@@ -1860,9 +2479,15 @@
case NVM_CFG1_GLOB_NETWORK_PORT_MODE_2X25G:
p_hwfn->hw_info.port_mode = QED_PORT_MODE_DE_2X25G;
break;
+ case NVM_CFG1_GLOB_NETWORK_PORT_MODE_2X10G:
+ p_hwfn->hw_info.port_mode = QED_PORT_MODE_DE_2X10G;
+ break;
case NVM_CFG1_GLOB_NETWORK_PORT_MODE_1X25G:
p_hwfn->hw_info.port_mode = QED_PORT_MODE_DE_1X25G;
break;
+ case NVM_CFG1_GLOB_NETWORK_PORT_MODE_4X25G:
+ p_hwfn->hw_info.port_mode = QED_PORT_MODE_DE_4X25G;
+ break;
default:
DP_NOTICE(p_hwfn, "Unknown port mode in 0x%08x\n", core_cfg);
break;
@@ -1976,8 +2601,9 @@
{
u8 num_funcs, enabled_func_idx = p_hwfn->rel_pf_id;
u32 reg_function_hide, tmp, eng_mask, low_pfs_mask;
+ struct qed_dev *cdev = p_hwfn->cdev;
- num_funcs = MAX_NUM_PFS_BB;
+ num_funcs = QED_IS_AH(cdev) ? MAX_NUM_PFS_K2 : MAX_NUM_PFS_BB;
/* Bit 0 of MISCS_REG_FUNCTION_HIDE indicates whether the bypass values
* in the other bits are selected.
@@ -1990,12 +2616,17 @@
reg_function_hide = qed_rd(p_hwfn, p_ptt, MISCS_REG_FUNCTION_HIDE);
if (reg_function_hide & 0x1) {
- if (QED_PATH_ID(p_hwfn) && p_hwfn->cdev->num_hwfns == 1) {
- num_funcs = 0;
- eng_mask = 0xaaaa;
+ if (QED_IS_BB(cdev)) {
+ if (QED_PATH_ID(p_hwfn) && cdev->num_hwfns == 1) {
+ num_funcs = 0;
+ eng_mask = 0xaaaa;
+ } else {
+ num_funcs = 1;
+ eng_mask = 0x5554;
+ }
} else {
num_funcs = 1;
- eng_mask = 0x5554;
+ eng_mask = 0xfffe;
}
/* Get the number of the enabled functions on the engine */
@@ -2027,24 +2658,12 @@
p_hwfn->enabled_func_idx, p_hwfn->num_funcs_on_engine);
}
-static int
-qed_get_hw_info(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt,
- enum qed_pci_personality personality)
+static void qed_hw_info_port_num_bb(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt)
{
u32 port_mode;
- int rc;
- /* Since all information is common, only first hwfns should do this */
- if (IS_LEAD_HWFN(p_hwfn)) {
- rc = qed_iov_hw_info(p_hwfn);
- if (rc)
- return rc;
- }
-
- /* Read the port mode */
- port_mode = qed_rd(p_hwfn, p_ptt,
- CNIG_REG_NW_PORT_MODE_BB_B0);
+ port_mode = qed_rd(p_hwfn, p_ptt, CNIG_REG_NW_PORT_MODE_BB_B0);
if (port_mode < 3) {
p_hwfn->cdev->num_ports_in_engines = 1;
@@ -2057,6 +2676,54 @@
/* Default num_ports_in_engines to something */
p_hwfn->cdev->num_ports_in_engines = 1;
}
+}
+
+static void qed_hw_info_port_num_ah(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt)
+{
+ u32 port;
+ int i;
+
+ p_hwfn->cdev->num_ports_in_engines = 0;
+
+ for (i = 0; i < MAX_NUM_PORTS_K2; i++) {
+ port = qed_rd(p_hwfn, p_ptt,
+ CNIG_REG_NIG_PORT0_CONF_K2 + (i * 4));
+ if (port & 1)
+ p_hwfn->cdev->num_ports_in_engines++;
+ }
+
+ if (!p_hwfn->cdev->num_ports_in_engines) {
+ DP_NOTICE(p_hwfn, "All NIG ports are inactive\n");
+
+ /* Default num_ports_in_engine to something */
+ p_hwfn->cdev->num_ports_in_engines = 1;
+ }
+}
+
+static void qed_hw_info_port_num(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
+{
+ if (QED_IS_BB(p_hwfn->cdev))
+ qed_hw_info_port_num_bb(p_hwfn, p_ptt);
+ else
+ qed_hw_info_port_num_ah(p_hwfn, p_ptt);
+}
+
+static int
+qed_get_hw_info(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ enum qed_pci_personality personality)
+{
+ int rc;
+
+ /* Since all information is common, only first hwfns should do this */
+ if (IS_LEAD_HWFN(p_hwfn)) {
+ rc = qed_iov_hw_info(p_hwfn);
+ if (rc)
+ return rc;
+ }
+
+ qed_hw_info_port_num(p_hwfn, p_ptt);
qed_hw_get_nvm_info(p_hwfn, p_ptt);
@@ -2085,33 +2752,48 @@
p_hwfn->hw_info.personality = protocol;
}
+ p_hwfn->hw_info.num_hw_tc = NUM_PHYS_TCS_4PORT_K2;
+ p_hwfn->hw_info.num_active_tc = 1;
+
qed_get_num_funcs(p_hwfn, p_ptt);
if (qed_mcp_is_init(p_hwfn))
p_hwfn->hw_info.mtu = p_hwfn->mcp_info->func_info.mtu;
- return qed_hw_get_resc(p_hwfn);
+ return qed_hw_get_resc(p_hwfn, p_ptt);
}
-static int qed_get_dev_info(struct qed_dev *cdev)
+static int qed_get_dev_info(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
{
- struct qed_hwfn *p_hwfn = QED_LEADING_HWFN(cdev);
+ struct qed_dev *cdev = p_hwfn->cdev;
+ u16 device_id_mask;
u32 tmp;
/* Read Vendor Id / Device Id */
pci_read_config_word(cdev->pdev, PCI_VENDOR_ID, &cdev->vendor_id);
pci_read_config_word(cdev->pdev, PCI_DEVICE_ID, &cdev->device_id);
- cdev->chip_num = (u16)qed_rd(p_hwfn, p_hwfn->p_main_ptt,
- MISCS_REG_CHIP_NUM);
- cdev->chip_rev = (u16)qed_rd(p_hwfn, p_hwfn->p_main_ptt,
- MISCS_REG_CHIP_REV);
+ /* Determine type */
+ device_id_mask = cdev->device_id & QED_DEV_ID_MASK;
+ switch (device_id_mask) {
+ case QED_DEV_ID_MASK_BB:
+ cdev->type = QED_DEV_TYPE_BB;
+ break;
+ case QED_DEV_ID_MASK_AH:
+ cdev->type = QED_DEV_TYPE_AH;
+ break;
+ default:
+ DP_NOTICE(p_hwfn, "Unknown device id 0x%x\n", cdev->device_id);
+ return -EBUSY;
+ }
+
+ cdev->chip_num = (u16)qed_rd(p_hwfn, p_ptt, MISCS_REG_CHIP_NUM);
+ cdev->chip_rev = (u16)qed_rd(p_hwfn, p_ptt, MISCS_REG_CHIP_REV);
+
MASK_FIELD(CHIP_REV, cdev->chip_rev);
- cdev->type = QED_DEV_TYPE_BB;
/* Learn number of HW-functions */
- tmp = qed_rd(p_hwfn, p_hwfn->p_main_ptt,
- MISCS_REG_CMT_ENABLED_FOR_PAIR);
+ tmp = qed_rd(p_hwfn, p_ptt, MISCS_REG_CMT_ENABLED_FOR_PAIR);
if (tmp & (1 << p_hwfn->rel_pf_id)) {
DP_NOTICE(cdev->hwfns, "device in CMT mode\n");
@@ -2120,15 +2802,17 @@
cdev->num_hwfns = 1;
}
- cdev->chip_bond_id = qed_rd(p_hwfn, p_hwfn->p_main_ptt,
+ cdev->chip_bond_id = qed_rd(p_hwfn, p_ptt,
MISCS_REG_CHIP_TEST_REG) >> 4;
MASK_FIELD(CHIP_BOND_ID, cdev->chip_bond_id);
- cdev->chip_metal = (u16)qed_rd(p_hwfn, p_hwfn->p_main_ptt,
- MISCS_REG_CHIP_METAL);
+ cdev->chip_metal = (u16)qed_rd(p_hwfn, p_ptt, MISCS_REG_CHIP_METAL);
MASK_FIELD(CHIP_METAL, cdev->chip_metal);
DP_INFO(cdev->hwfns,
- "Chip details - Num: %04x Rev: %04x Bond id: %04x Metal: %04x\n",
+ "Chip details - %s %c%d, Num: %04x Rev: %04x Bond id: %04x Metal: %04x\n",
+ QED_IS_BB(cdev) ? "BB" : "AH",
+ 'A' + cdev->chip_rev,
+ (int)cdev->chip_metal,
cdev->chip_num, cdev->chip_rev,
cdev->chip_bond_id, cdev->chip_metal);
@@ -2174,7 +2858,7 @@
/* First hwfn learns basic information, e.g., number of hwfns */
if (!p_hwfn->my_id) {
- rc = qed_get_dev_info(p_hwfn->cdev);
+ rc = qed_get_dev_info(p_hwfn, p_hwfn->p_main_ptt);
if (rc)
goto err1;
}
@@ -2195,6 +2879,15 @@
goto err2;
}
+ /* Sending a mailbox to the MFW should be done after qed_get_hw_info()
+ * is called as it sets the ports number in an engine.
+ */
+ if (IS_LEAD_HWFN(p_hwfn)) {
+ rc = qed_mcp_initiate_pf_flr(p_hwfn, p_hwfn->p_main_ptt);
+ if (rc)
+ DP_NOTICE(p_hwfn, "Failed to initiate PF FLR\n");
+ }
+
/* Allocate the init RT array and initialize the init-ops engine */
rc = qed_init_alloc(p_hwfn);
if (rc)
@@ -2236,11 +2929,14 @@
u8 __iomem *addr;
/* adjust bar offset for second engine */
- addr = cdev->regview + qed_hw_bar_size(p_hwfn, BAR_ID_0) / 2;
+ addr = cdev->regview +
+ qed_hw_bar_size(p_hwfn, p_hwfn->p_main_ptt,
+ BAR_ID_0) / 2;
p_regview = addr;
- /* adjust doorbell bar offset for second engine */
- addr = cdev->doorbells + qed_hw_bar_size(p_hwfn, BAR_ID_1) / 2;
+ addr = cdev->doorbells +
+ qed_hw_bar_size(p_hwfn, p_hwfn->p_main_ptt,
+ BAR_ID_1) / 2;
p_doorbell = addr;
/* prepare second hw function */
@@ -3363,3 +4059,22 @@
memset(p_hwfn->qm_info.wfq_data, 0,
sizeof(*p_hwfn->qm_info.wfq_data) * p_hwfn->qm_info.num_vports);
}
+
+int qed_device_num_engines(struct qed_dev *cdev)
+{
+ return QED_IS_BB(cdev) ? 2 : 1;
+}
+
+static int qed_device_num_ports(struct qed_dev *cdev)
+{
+ /* in CMT always only one port */
+ if (cdev->num_hwfns > 1)
+ return 1;
+
+ return cdev->num_ports_in_engines * qed_device_num_engines(cdev);
+}
+
+int qed_device_get_port_id(struct qed_dev *cdev)
+{
+ return (QED_LEADING_HWFN(cdev)->abs_pf_id) % qed_device_num_ports(cdev);
+}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_dev_api.h b/drivers/net/ethernet/qlogic/qed/qed_dev_api.h
index 6812003..cefe3ee 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_dev_api.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_dev_api.h
@@ -82,26 +82,63 @@
*/
void qed_resc_setup(struct qed_dev *cdev);
+enum qed_override_force_load {
+ QED_OVERRIDE_FORCE_LOAD_NONE,
+ QED_OVERRIDE_FORCE_LOAD_ALWAYS,
+ QED_OVERRIDE_FORCE_LOAD_NEVER,
+};
+
+struct qed_drv_load_params {
+ /* Indicates whether the driver is running over a crash kernel.
+ * As part of the load request, this will be used for providing the
+ * driver role to the MFW.
+ * In case of a crash kernel over PDA - this should be set to false.
+ */
+ bool is_crash_kernel;
+
+ /* The timeout value that the MFW should use when locking the engine for
+ * the driver load process.
+ * A value of '0' means the default value, and '255' means no timeout.
+ */
+ u8 mfw_timeout_val;
+#define QED_LOAD_REQ_LOCK_TO_DEFAULT 0
+#define QED_LOAD_REQ_LOCK_TO_NONE 255
+
+ /* Avoid engine reset when first PF loads on it */
+ bool avoid_eng_reset;
+
+ /* Allow overriding the default force load behavior */
+ enum qed_override_force_load override_force_load;
+};
+
+struct qed_hw_init_params {
+ /* Tunneling parameters */
+ struct qed_tunnel_info *p_tunn;
+
+ bool b_hw_start;
+
+ /* Interrupt mode [msix, inta, etc.] to use */
+ enum qed_int_mode int_mode;
+
+ /* NPAR tx switching to be used for vports for tx-switching */
+ bool allow_npar_tx_switch;
+
+ /* Binary fw data pointer in binary fw file */
+ const u8 *bin_fw_data;
+
+ /* Driver load parameters */
+ struct qed_drv_load_params *p_drv_load_params;
+};
+
/**
* @brief qed_hw_init -
*
* @param cdev
- * @param p_tunn
- * @param b_hw_start
- * @param int_mode - interrupt mode [msix, inta, etc.] to use.
- * @param allow_npar_tx_switch - npar tx switching to be used
- * for vports configured for tx-switching.
- * @param bin_fw_data - binary fw data pointer in binary fw file.
- * Pass NULL if not using binary fw file.
+ * @param p_params
*
* @return int
*/
-int qed_hw_init(struct qed_dev *cdev,
- struct qed_tunn_start_params *p_tunn,
- bool b_hw_start,
- enum qed_int_mode int_mode,
- bool allow_npar_tx_switch,
- const u8 *bin_fw_data);
+int qed_hw_init(struct qed_dev *cdev, struct qed_hw_init_params *p_params);
/**
* @brief qed_hw_timers_stop_all - stop the timers HW block
@@ -128,26 +165,20 @@
*
* @param cdev
*
+ * @return int
*/
-void qed_hw_stop_fastpath(struct qed_dev *cdev);
+int qed_hw_stop_fastpath(struct qed_dev *cdev);
/**
* @brief qed_hw_start_fastpath -restart fastpath traffic,
* only if hw_stop_fastpath was called
*
- * @param cdev
- *
- */
-void qed_hw_start_fastpath(struct qed_hwfn *p_hwfn);
-
-/**
- * @brief qed_hw_reset -
- *
- * @param cdev
+ * @param p_hwfn
*
* @return int
*/
-int qed_hw_reset(struct qed_dev *cdev);
+int qed_hw_start_fastpath(struct qed_hwfn *p_hwfn);
+
/**
* @brief qed_hw_prepare -
@@ -441,4 +472,6 @@
*/
int qed_set_txq_coalesce(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt,
u16 coalesce, u8 qid, u16 sb_id);
+
+const char *qed_hw_get_resc_name(enum qed_resources res_id);
#endif
diff --git a/drivers/net/ethernet/qlogic/qed/qed_fcoe.c b/drivers/net/ethernet/qlogic/qed/qed_fcoe.c
index cbc8141..21a58ff 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_fcoe.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_fcoe.c
@@ -191,7 +191,7 @@
p_data->q_params.cq_sb_pi = fcoe_pf_params->gl_rq_pi;
p_data->q_params.cmdq_sb_pi = fcoe_pf_params->gl_cmd_pi;
- p_data->q_params.bdq_resource_id = FCOE_BDQ_ID(p_hwfn->port_id);
+ p_data->q_params.bdq_resource_id = (u8)RESC_START(p_hwfn, QED_BDQ);
DMA_REGPAIR_LE(p_data->q_params.bdq_pbl_base_address[BDQ_ID_RQ],
fcoe_pf_params->bdq_pbl_base_addr[BDQ_ID_RQ]);
@@ -241,7 +241,7 @@
struct fcoe_conn_offload_ramrod_data *p_data;
struct qed_spq_entry *p_ent = NULL;
struct qed_sp_init_data init_data;
- u16 pq_id = 0, tmp;
+ u16 physical_q0, tmp;
int rc;
/* Get SPQ entry */
@@ -261,9 +261,9 @@
p_data = &p_ramrod->offload_ramrod_data;
/* Transmission PQ is the first of the PF */
- pq_id = qed_get_qm_pq(p_hwfn, PROTOCOLID_FCOE, NULL);
- p_conn->physical_q0 = cpu_to_le16(pq_id);
- p_data->physical_q0 = cpu_to_le16(pq_id);
+ physical_q0 = qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_OFLD);
+ p_conn->physical_q0 = cpu_to_le16(physical_q0);
+ p_data->physical_q0 = cpu_to_le16(physical_q0);
p_data->conn_id = cpu_to_le16(p_conn->conn_id);
DMA_REGPAIR_LE(p_data->sq_pbl_addr, p_conn->sq_pbl_addr);
@@ -340,10 +340,10 @@
static int
qed_sp_fcoe_func_stop(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
enum spq_mode comp_mode,
struct qed_spq_comp_cb *p_comp_addr)
{
- struct qed_ptt *p_ptt = p_hwfn->p_main_ptt;
struct qed_spq_entry *p_ent = NULL;
struct qed_sp_init_data init_data;
u32 active_segs = 0;
@@ -512,19 +512,31 @@
static void __iomem *qed_fcoe_get_primary_bdq_prod(struct qed_hwfn *p_hwfn,
u8 bdq_id)
{
- u8 bdq_function_id = FCOE_BDQ_ID(p_hwfn->port_id);
-
- return (u8 __iomem *)p_hwfn->regview + GTT_BAR0_MAP_REG_MSDM_RAM +
- MSTORM_SCSI_BDQ_EXT_PROD_OFFSET(bdq_function_id, bdq_id);
+ if (RESC_NUM(p_hwfn, QED_BDQ)) {
+ return (u8 __iomem *)p_hwfn->regview +
+ GTT_BAR0_MAP_REG_MSDM_RAM +
+ MSTORM_SCSI_BDQ_EXT_PROD_OFFSET(RESC_START(p_hwfn,
+ QED_BDQ),
+ bdq_id);
+ } else {
+ DP_NOTICE(p_hwfn, "BDQ is not allocated!\n");
+ return NULL;
+ }
}
static void __iomem *qed_fcoe_get_secondary_bdq_prod(struct qed_hwfn *p_hwfn,
u8 bdq_id)
{
- u8 bdq_function_id = FCOE_BDQ_ID(p_hwfn->port_id);
-
- return (u8 __iomem *)p_hwfn->regview + GTT_BAR0_MAP_REG_TSDM_RAM +
- TSTORM_SCSI_BDQ_EXT_PROD_OFFSET(bdq_function_id, bdq_id);
+ if (RESC_NUM(p_hwfn, QED_BDQ)) {
+ return (u8 __iomem *)p_hwfn->regview +
+ GTT_BAR0_MAP_REG_TSDM_RAM +
+ TSTORM_SCSI_BDQ_EXT_PROD_OFFSET(RESC_START(p_hwfn,
+ QED_BDQ),
+ bdq_id);
+ } else {
+ DP_NOTICE(p_hwfn, "BDQ is not allocated!\n");
+ return NULL;
+ }
}
struct qed_fcoe_info *qed_fcoe_alloc(struct qed_hwfn *p_hwfn)
@@ -753,6 +765,7 @@
static int qed_fcoe_stop(struct qed_dev *cdev)
{
+ struct qed_ptt *p_ptt;
int rc;
if (!(cdev->flags & QED_FLAG_STORAGE_STARTED)) {
@@ -766,10 +779,15 @@
return -EINVAL;
}
+ p_ptt = qed_ptt_acquire(QED_LEADING_HWFN(cdev));
+ if (!p_ptt)
+ return -EAGAIN;
+
/* Stop the fcoe */
- rc = qed_sp_fcoe_func_stop(QED_LEADING_HWFN(cdev),
+ rc = qed_sp_fcoe_func_stop(QED_LEADING_HWFN(cdev), p_ptt,
QED_SPQ_MODE_EBLOCK, NULL);
cdev->flags &= ~QED_FLAG_STORAGE_STARTED;
+ qed_ptt_release(QED_LEADING_HWFN(cdev), p_ptt);
return rc;
}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_hsi.h b/drivers/net/ethernet/qlogic/qed/qed_hsi.h
index 37c2bfb..858a57a 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_hsi.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_hsi.h
@@ -574,6 +574,7 @@
CORE_EVENT_TX_QUEUE_STOP,
CORE_EVENT_RX_QUEUE_START,
CORE_EVENT_RX_QUEUE_STOP,
+ CORE_EVENT_RX_QUEUE_FLUSH,
MAX_CORE_EVENT_OPCODE
};
@@ -625,6 +626,7 @@
CORE_RAMROD_TX_QUEUE_START,
CORE_RAMROD_RX_QUEUE_STOP,
CORE_RAMROD_TX_QUEUE_STOP,
+ CORE_RAMROD_RX_QUEUE_FLUSH,
MAX_CORE_RAMROD_CMD_ID
};
@@ -698,7 +700,8 @@
u8 type;
u8 ramrod_cmd_id;
__le16 echo;
- __le32 reserved1[7];
+ struct core_rx_cqe_opaque_data opaque_data;
+ __le32 reserved1[5];
};
union core_rx_cqe_union {
@@ -735,45 +738,46 @@
__le16 reserved2[2];
};
-struct core_tx_bd_flags {
- u8 as_bitfield;
-#define CORE_TX_BD_FLAGS_FORCE_VLAN_MODE_MASK 0x1
-#define CORE_TX_BD_FLAGS_FORCE_VLAN_MODE_SHIFT 0
-#define CORE_TX_BD_FLAGS_VLAN_INSERTION_MASK 0x1
-#define CORE_TX_BD_FLAGS_VLAN_INSERTION_SHIFT 1
-#define CORE_TX_BD_FLAGS_START_BD_MASK 0x1
-#define CORE_TX_BD_FLAGS_START_BD_SHIFT 2
-#define CORE_TX_BD_FLAGS_IP_CSUM_MASK 0x1
-#define CORE_TX_BD_FLAGS_IP_CSUM_SHIFT 3
-#define CORE_TX_BD_FLAGS_L4_CSUM_MASK 0x1
-#define CORE_TX_BD_FLAGS_L4_CSUM_SHIFT 4
-#define CORE_TX_BD_FLAGS_IPV6_EXT_MASK 0x1
-#define CORE_TX_BD_FLAGS_IPV6_EXT_SHIFT 5
-#define CORE_TX_BD_FLAGS_L4_PROTOCOL_MASK 0x1
-#define CORE_TX_BD_FLAGS_L4_PROTOCOL_SHIFT 6
-#define CORE_TX_BD_FLAGS_L4_PSEUDO_CSUM_MODE_MASK 0x1
-#define CORE_TX_BD_FLAGS_L4_PSEUDO_CSUM_MODE_SHIFT 7
+struct core_tx_bd_data {
+ __le16 as_bitfield;
+#define CORE_TX_BD_DATA_FORCE_VLAN_MODE_MASK 0x1
+#define CORE_TX_BD_DATA_FORCE_VLAN_MODE_SHIFT 0
+#define CORE_TX_BD_DATA_VLAN_INSERTION_MASK 0x1
+#define CORE_TX_BD_DATA_VLAN_INSERTION_SHIFT 1
+#define CORE_TX_BD_DATA_START_BD_MASK 0x1
+#define CORE_TX_BD_DATA_START_BD_SHIFT 2
+#define CORE_TX_BD_DATA_IP_CSUM_MASK 0x1
+#define CORE_TX_BD_DATA_IP_CSUM_SHIFT 3
+#define CORE_TX_BD_DATA_L4_CSUM_MASK 0x1
+#define CORE_TX_BD_DATA_L4_CSUM_SHIFT 4
+#define CORE_TX_BD_DATA_IPV6_EXT_MASK 0x1
+#define CORE_TX_BD_DATA_IPV6_EXT_SHIFT 5
+#define CORE_TX_BD_DATA_L4_PROTOCOL_MASK 0x1
+#define CORE_TX_BD_DATA_L4_PROTOCOL_SHIFT 6
+#define CORE_TX_BD_DATA_L4_PSEUDO_CSUM_MODE_MASK 0x1
+#define CORE_TX_BD_DATA_L4_PSEUDO_CSUM_MODE_SHIFT 7
+#define CORE_TX_BD_DATA_NBDS_MASK 0xF
+#define CORE_TX_BD_DATA_NBDS_SHIFT 8
+#define CORE_TX_BD_DATA_ROCE_FLAV_MASK 0x1
+#define CORE_TX_BD_DATA_ROCE_FLAV_SHIFT 12
+#define CORE_TX_BD_DATA_IP_LEN_MASK 0x1
+#define CORE_TX_BD_DATA_IP_LEN_SHIFT 13
+#define CORE_TX_BD_DATA_RESERVED0_MASK 0x3
+#define CORE_TX_BD_DATA_RESERVED0_SHIFT 14
};
struct core_tx_bd {
struct regpair addr;
__le16 nbytes;
__le16 nw_vlan_or_lb_echo;
- u8 bitfield0;
-#define CORE_TX_BD_NBDS_MASK 0xF
-#define CORE_TX_BD_NBDS_SHIFT 0
-#define CORE_TX_BD_ROCE_FLAV_MASK 0x1
-#define CORE_TX_BD_ROCE_FLAV_SHIFT 4
-#define CORE_TX_BD_RESERVED0_MASK 0x7
-#define CORE_TX_BD_RESERVED0_SHIFT 5
- struct core_tx_bd_flags bd_flags;
+ struct core_tx_bd_data bd_data;
__le16 bitfield1;
#define CORE_TX_BD_L4_HDR_OFFSET_W_MASK 0x3FFF
#define CORE_TX_BD_L4_HDR_OFFSET_W_SHIFT 0
#define CORE_TX_BD_TX_DST_MASK 0x1
#define CORE_TX_BD_TX_DST_SHIFT 14
-#define CORE_TX_BD_RESERVED1_MASK 0x1
-#define CORE_TX_BD_RESERVED1_SHIFT 15
+#define CORE_TX_BD_RESERVED_MASK 0x1
+#define CORE_TX_BD_RESERVED_SHIFT 15
};
enum core_tx_dest {
@@ -800,6 +804,14 @@
__le32 reserved0[2];
};
+enum dcb_dhcp_update_flag {
+ DONT_UPDATE_DCB_DHCP,
+ UPDATE_DCB,
+ UPDATE_DSCP,
+ UPDATE_DCB_DSCP,
+ MAX_DCB_DHCP_UPDATE_FLAG
+};
+
struct eth_mstorm_per_pf_stat {
struct regpair gre_discard_pkts;
struct regpair vxlan_discard_pkts;
@@ -893,6 +905,12 @@
struct event_ring_next_addr next_addr;
};
+enum fw_flow_ctrl_mode {
+ flow_ctrl_pause,
+ flow_ctrl_pfc,
+ MAX_FW_FLOW_CTRL_MODE
+};
+
/* Major and Minor hsi Versions */
struct hsi_fp_ver_struct {
u8 minor_ver_arr[2];
@@ -921,6 +939,7 @@
ETH_EDPM_OUT_OF_SYNC,
ETH_TUNN_IPV6_EXT_NBD_ERR,
ETH_CONTROL_PACKET_VIOLATION,
+ ETH_ANTI_SPOOFING_ERR,
MAX_MALICIOUS_VF_ERROR_ID
};
@@ -1106,8 +1125,9 @@
struct regpair ll2_mac_filter_discard;
struct regpair ll2_conn_disabled_discard;
struct regpair iscsi_irregular_pkt;
- struct regpair reserved;
+ struct regpair fcoe_irregular_pkt;
struct regpair roce_irregular_pkt;
+ struct regpair reserved;
struct regpair eth_irregular_pkt;
struct regpair reserved1;
struct regpair preroce_irregular_pkt;
@@ -1648,6 +1668,11 @@
GRCBASE_MS = 0x6a0000,
GRCBASE_PHY_PCIE = 0x620000,
GRCBASE_LED = 0x6b8000,
+ GRCBASE_AVS_WRAP = 0x6b0000,
+ GRCBASE_RGFS = 0x19d0000,
+ GRCBASE_TGFS = 0x19e0000,
+ GRCBASE_PTLD = 0x19f0000,
+ GRCBASE_YPLD = 0x1a10000,
GRCBASE_MISC_AEU = 0x8000,
GRCBASE_BAR0_MAP = 0x1c00000,
MAX_BLOCK_ADDR
@@ -1732,6 +1757,11 @@
BLOCK_MS,
BLOCK_PHY_PCIE,
BLOCK_LED,
+ BLOCK_AVS_WRAP,
+ BLOCK_RGFS,
+ BLOCK_TGFS,
+ BLOCK_PTLD,
+ BLOCK_YPLD,
BLOCK_MISC_AEU,
BLOCK_BAR0_MAP,
MAX_BLOCK_ID
@@ -1783,9 +1813,9 @@
__le32 data;
#define DBG_ATTN_REG_RESULT_STS_ADDRESS_MASK 0xFFFFFF
#define DBG_ATTN_REG_RESULT_STS_ADDRESS_SHIFT 0
-#define DBG_ATTN_REG_RESULT_NUM_ATTN_IDX_MASK 0xFF
-#define DBG_ATTN_REG_RESULT_NUM_ATTN_IDX_SHIFT 24
- __le16 attn_idx_offset;
+#define DBG_ATTN_REG_RESULT_NUM_REG_ATTN_MASK 0xFF
+#define DBG_ATTN_REG_RESULT_NUM_REG_ATTN_SHIFT 24
+ __le16 block_attn_offset;
__le16 reserved;
__le32 sts_val;
__le32 mask_val;
@@ -1815,12 +1845,12 @@
/* Attention register */
struct dbg_attn_reg {
struct dbg_mode_hdr mode;
- __le16 attn_idx_offset;
+ __le16 block_attn_offset;
__le32 data;
#define DBG_ATTN_REG_STS_ADDRESS_MASK 0xFFFFFF
#define DBG_ATTN_REG_STS_ADDRESS_SHIFT 0
-#define DBG_ATTN_REG_NUM_ATTN_IDX_MASK 0xFF
-#define DBG_ATTN_REG_NUM_ATTN_IDX_SHIFT 24
+#define DBG_ATTN_REG_NUM_REG_ATTN_MASK 0xFF
+#define DBG_ATTN_REG_NUM_REG_ATTN_SHIFT 24
__le32 sts_clr_address;
__le32 mask_address;
};
@@ -2001,6 +2031,20 @@
MAX_DBG_BUS_CLIENTS
};
+enum dbg_bus_constraint_ops {
+ DBG_BUS_CONSTRAINT_OP_EQ,
+ DBG_BUS_CONSTRAINT_OP_NE,
+ DBG_BUS_CONSTRAINT_OP_LT,
+ DBG_BUS_CONSTRAINT_OP_LTC,
+ DBG_BUS_CONSTRAINT_OP_LE,
+ DBG_BUS_CONSTRAINT_OP_LEC,
+ DBG_BUS_CONSTRAINT_OP_GT,
+ DBG_BUS_CONSTRAINT_OP_GTC,
+ DBG_BUS_CONSTRAINT_OP_GE,
+ DBG_BUS_CONSTRAINT_OP_GEC,
+ MAX_DBG_BUS_CONSTRAINT_OPS
+};
+
/* Debug Bus memory address */
struct dbg_bus_mem_addr {
__le32 lo;
@@ -2092,10 +2136,18 @@
* DBG_BUS_TARGET_ID_PCI.
*/
__le16 reserved;
- struct dbg_bus_block_data blocks[80];/* Debug Bus data for each block */
+ struct dbg_bus_block_data blocks[88];/* Debug Bus data for each block */
struct dbg_bus_storm_data storms[6]; /* Debug Bus data for each block */
};
+enum dbg_bus_filter_types {
+ DBG_BUS_FILTER_TYPE_OFF,
+ DBG_BUS_FILTER_TYPE_PRE,
+ DBG_BUS_FILTER_TYPE_POST,
+ DBG_BUS_FILTER_TYPE_ON,
+ MAX_DBG_BUS_FILTER_TYPES
+};
+
/* Debug bus frame modes */
enum dbg_bus_frame_modes {
DBG_BUS_FRAME_MODE_0HW_4ST = 0, /* 0 HW dwords, 4 Storm dwords */
@@ -2104,6 +2156,40 @@
MAX_DBG_BUS_FRAME_MODES
};
+enum dbg_bus_input_types {
+ DBG_BUS_INPUT_TYPE_STORM,
+ DBG_BUS_INPUT_TYPE_BLOCK,
+ MAX_DBG_BUS_INPUT_TYPES
+};
+
+enum dbg_bus_other_engine_modes {
+ DBG_BUS_OTHER_ENGINE_MODE_NONE,
+ DBG_BUS_OTHER_ENGINE_MODE_DOUBLE_BW_TX,
+ DBG_BUS_OTHER_ENGINE_MODE_DOUBLE_BW_RX,
+ DBG_BUS_OTHER_ENGINE_MODE_CROSS_ENGINE_TX,
+ DBG_BUS_OTHER_ENGINE_MODE_CROSS_ENGINE_RX,
+ MAX_DBG_BUS_OTHER_ENGINE_MODES
+};
+
+enum dbg_bus_post_trigger_types {
+ DBG_BUS_POST_TRIGGER_RECORD,
+ DBG_BUS_POST_TRIGGER_DROP,
+ MAX_DBG_BUS_POST_TRIGGER_TYPES
+};
+
+enum dbg_bus_pre_trigger_types {
+ DBG_BUS_PRE_TRIGGER_START_FROM_ZERO,
+ DBG_BUS_PRE_TRIGGER_NUM_CHUNKS,
+ DBG_BUS_PRE_TRIGGER_DROP,
+ MAX_DBG_BUS_PRE_TRIGGER_TYPES
+};
+
+enum dbg_bus_semi_frame_modes {
+ DBG_BUS_SEMI_FRAME_MODE_0SLOW_4FAST = 0,
+ DBG_BUS_SEMI_FRAME_MODE_4SLOW_0FAST = 3,
+ MAX_DBG_BUS_SEMI_FRAME_MODES
+};
+
/* Debug bus states */
enum dbg_bus_states {
DBG_BUS_STATE_IDLE, /* debug bus idle state (not recording) */
@@ -2115,6 +2201,19 @@
MAX_DBG_BUS_STATES
};
+enum dbg_bus_storm_modes {
+ DBG_BUS_STORM_MODE_PRINTF,
+ DBG_BUS_STORM_MODE_PRAM_ADDR,
+ DBG_BUS_STORM_MODE_DRA_RW,
+ DBG_BUS_STORM_MODE_DRA_W,
+ DBG_BUS_STORM_MODE_LD_ST_ADDR,
+ DBG_BUS_STORM_MODE_DRA_FSM,
+ DBG_BUS_STORM_MODE_RH,
+ DBG_BUS_STORM_MODE_FOC,
+ DBG_BUS_STORM_MODE_EXT_STORE,
+ MAX_DBG_BUS_STORM_MODES
+};
+
/* Debug bus target IDs */
enum dbg_bus_targets {
/* records debug bus to DBG block internal buffer */
@@ -2128,13 +2227,10 @@
/* GRC Dump data */
struct dbg_grc_data {
- __le32 param_val[40]; /* Value of each GRC parameter. Array size must
- * match the enum dbg_grc_params.
- */
- u8 param_set_by_user[40]; /* Indicates for each GRC parameter if it was
- * set by the user (0/1). Array size must
- * match the enum dbg_grc_params.
- */
+ u8 params_initialized;
+ u8 reserved1;
+ __le16 reserved2;
+ __le32 param_val[48];
};
/* Debug GRC params */
@@ -2181,6 +2277,8 @@
DBG_GRC_PARAM_PARITY_SAFE,
DBG_GRC_PARAM_DUMP_CM, /* dump CM memories (0/1) */
DBG_GRC_PARAM_DUMP_PHY, /* dump PHY memories (0/1) */
+ DBG_GRC_PARAM_NO_MCP,
+ DBG_GRC_PARAM_NO_FW_VER,
MAX_DBG_GRC_PARAMS
};
@@ -2280,7 +2378,7 @@
struct dbg_bus_data bus; /* Debug Bus data */
struct idle_chk_data idle_chk; /* Idle Check data */
u8 mode_enable[40]; /* Indicates if a mode is enabled (0/1) */
- u8 block_in_reset[80]; /* Indicates if a block is in reset state (0/1).
+ u8 block_in_reset[88]; /* Indicates if a block is in reset state (0/1).
*/
u8 chip_id; /* Chip ID (from enum chip_ids) */
u8 platform_id; /* Platform ID (from enum platform_ids) */
@@ -2404,7 +2502,7 @@
enum init_modes {
MODE_RESERVED,
- MODE_BB_B0,
+ MODE_BB,
MODE_K2,
MODE_ASIC,
MODE_RESERVED2,
@@ -2418,7 +2516,6 @@
MODE_PORTS_PER_ENG_2,
MODE_PORTS_PER_ENG_4,
MODE_100G,
- MODE_40G,
MODE_RESERVED6,
MAX_INIT_MODES
};
@@ -2686,6 +2783,13 @@
*/
enum dbg_status qed_dbg_set_bin_ptr(const u8 * const bin_ptr);
/**
+ * @brief qed_dbg_grc_set_params_default - Reverts all GRC parameters to their
+ * default value.
+ *
+ * @param p_hwfn - HW device data
+ */
+void qed_dbg_grc_set_params_default(struct qed_hwfn *p_hwfn);
+/**
* @brief qed_dbg_grc_get_dump_buf_size - Returns the required buffer size for
* GRC Dump.
*
@@ -3369,6 +3473,11 @@
void qed_set_geneve_enable(struct qed_hwfn *p_hwfn,
struct qed_ptt *p_ptt,
bool eth_geneve_enable, bool ip_geneve_enable);
+void qed_set_rfs_mode_disable(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt, u16 pf_id);
+void qed_set_rfs_mode_enable(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt,
+ u16 pf_id, bool tcp, bool udp,
+ bool ipv4, bool ipv6);
#define YSTORM_FLOW_CONTROL_MODE_OFFSET (IRO[0].base)
#define YSTORM_FLOW_CONTROL_MODE_SIZE (IRO[0].size)
@@ -3418,7 +3527,7 @@
#define MSTORM_TPA_TIMEOUT_US_SIZE (IRO[21].size)
#define MSTORM_ETH_PF_STAT_OFFSET(pf_id) \
(IRO[22].base + ((pf_id) * IRO[22].m1))
-#define MSTORM_ETH_PF_STAT_SIZE (IRO[21].size)
+#define MSTORM_ETH_PF_STAT_SIZE (IRO[22].size)
#define USTORM_QUEUE_STAT_OFFSET(stat_counter_id) \
(IRO[23].base + ((stat_counter_id) * IRO[23].m1))
#define USTORM_QUEUE_STAT_SIZE (IRO[23].size)
@@ -3482,7 +3591,7 @@
static const struct iro iro_arr[47] = {
{0x0, 0x0, 0x0, 0x0, 0x8},
- {0x4cb0, 0x78, 0x0, 0x0, 0x78},
+ {0x4cb0, 0x80, 0x0, 0x0, 0x80},
{0x6318, 0x20, 0x0, 0x0, 0x20},
{0xb00, 0x8, 0x0, 0x0, 0x4},
{0xa80, 0x8, 0x0, 0x0, 0x4},
@@ -3521,13 +3630,13 @@
{0xd888, 0x38, 0x0, 0x0, 0x24},
{0x12c38, 0x10, 0x0, 0x0, 0x8},
{0x11aa0, 0x38, 0x0, 0x0, 0x18},
- {0xa8c0, 0x30, 0x0, 0x0, 0x10},
- {0x86f8, 0x28, 0x0, 0x0, 0x18},
+ {0xa8c0, 0x38, 0x0, 0x0, 0x10},
+ {0x86f8, 0x30, 0x0, 0x0, 0x18},
{0x101f8, 0x10, 0x0, 0x0, 0x10},
{0xdd08, 0x48, 0x0, 0x0, 0x38},
{0x10660, 0x20, 0x0, 0x0, 0x20},
{0x2b80, 0x80, 0x0, 0x0, 0x10},
- {0x5000, 0x10, 0x0, 0x0, 0x10},
+ {0x5020, 0x10, 0x0, 0x0, 0x10},
};
/* Runtime array offsets */
@@ -4595,6 +4704,12 @@
MAX_ETH_IPV4_FRAG_TYPE
};
+enum eth_ip_type {
+ ETH_IPV4,
+ ETH_IPV6,
+ MAX_ETH_IP_TYPE
+};
+
enum eth_ramrod_cmd_id {
ETH_RAMROD_UNUSED,
ETH_RAMROD_VPORT_START,
@@ -4752,6 +4867,18 @@
__le16 reserved2[3];
};
+enum gft_filter_update_action {
+ GFT_ADD_FILTER,
+ GFT_DELETE_FILTER,
+ MAX_GFT_FILTER_UPDATE_ACTION
+};
+
+enum gft_logic_filter_type {
+ GFT_FILTER_TYPE,
+ RFS_FILTER_TYPE,
+ MAX_GFT_LOGIC_FILTER_TYPE
+};
+
/* Ramrod data for rx queue start ramrod */
struct rx_queue_start_ramrod_data {
__le16 rx_queue_id;
@@ -4822,6 +4949,16 @@
__le32 tenant_id;
};
+struct rx_update_gft_filter_data {
+ struct regpair pkt_hdr_addr;
+ __le16 pkt_hdr_length;
+ __le16 rx_qid_or_action_icid;
+ u8 vport_id;
+ u8 filter_type;
+ u8 filter_action;
+ u8 reserved;
+};
+
/* Ramrod data for rx queue start ramrod */
struct tx_queue_start_ramrod_data {
__le16 sb_id;
@@ -4944,7 +5081,10 @@
u8 update_mtu_flg;
__le16 mtu;
- u8 reserved[2];
+ u8 update_ctl_frame_checks_en_flg;
+ u8 ctl_frame_mac_check_en;
+ u8 ctl_frame_ethtype_check_en;
+ u8 reserved[15];
};
struct vport_update_ramrod_mcast {
@@ -4962,6 +5102,652 @@
struct eth_vport_rss_config rss_config;
};
+struct gft_cam_line {
+ __le32 camline;
+#define GFT_CAM_LINE_VALID_MASK 0x1
+#define GFT_CAM_LINE_VALID_SHIFT 0
+#define GFT_CAM_LINE_DATA_MASK 0x3FFF
+#define GFT_CAM_LINE_DATA_SHIFT 1
+#define GFT_CAM_LINE_MASK_BITS_MASK 0x3FFF
+#define GFT_CAM_LINE_MASK_BITS_SHIFT 15
+#define GFT_CAM_LINE_RESERVED1_MASK 0x7
+#define GFT_CAM_LINE_RESERVED1_SHIFT 29
+};
+
+struct gft_cam_line_mapped {
+ __le32 camline;
+#define GFT_CAM_LINE_MAPPED_VALID_MASK 0x1
+#define GFT_CAM_LINE_MAPPED_VALID_SHIFT 0
+#define GFT_CAM_LINE_MAPPED_IP_VERSION_MASK 0x1
+#define GFT_CAM_LINE_MAPPED_IP_VERSION_SHIFT 1
+#define GFT_CAM_LINE_MAPPED_TUNNEL_IP_VERSION_MASK 0x1
+#define GFT_CAM_LINE_MAPPED_TUNNEL_IP_VERSION_SHIFT 2
+#define GFT_CAM_LINE_MAPPED_UPPER_PROTOCOL_TYPE_MASK 0xF
+#define GFT_CAM_LINE_MAPPED_UPPER_PROTOCOL_TYPE_SHIFT 3
+#define GFT_CAM_LINE_MAPPED_TUNNEL_TYPE_MASK 0xF
+#define GFT_CAM_LINE_MAPPED_TUNNEL_TYPE_SHIFT 7
+#define GFT_CAM_LINE_MAPPED_PF_ID_MASK 0xF
+#define GFT_CAM_LINE_MAPPED_PF_ID_SHIFT 11
+#define GFT_CAM_LINE_MAPPED_IP_VERSION_MASK_MASK 0x1
+#define GFT_CAM_LINE_MAPPED_IP_VERSION_MASK_SHIFT 15
+#define GFT_CAM_LINE_MAPPED_TUNNEL_IP_VERSION_MASK_MASK 0x1
+#define GFT_CAM_LINE_MAPPED_TUNNEL_IP_VERSION_MASK_SHIFT 16
+#define GFT_CAM_LINE_MAPPED_UPPER_PROTOCOL_TYPE_MASK_MASK 0xF
+#define GFT_CAM_LINE_MAPPED_UPPER_PROTOCOL_TYPE_MASK_SHIFT 17
+#define GFT_CAM_LINE_MAPPED_TUNNEL_TYPE_MASK_MASK 0xF
+#define GFT_CAM_LINE_MAPPED_TUNNEL_TYPE_MASK_SHIFT 21
+#define GFT_CAM_LINE_MAPPED_PF_ID_MASK_MASK 0xF
+#define GFT_CAM_LINE_MAPPED_PF_ID_MASK_SHIFT 25
+#define GFT_CAM_LINE_MAPPED_RESERVED1_MASK 0x7
+#define GFT_CAM_LINE_MAPPED_RESERVED1_SHIFT 29
+};
+
+union gft_cam_line_union {
+ struct gft_cam_line cam_line;
+ struct gft_cam_line_mapped cam_line_mapped;
+};
+
+enum gft_profile_ip_version {
+ GFT_PROFILE_IPV4 = 0,
+ GFT_PROFILE_IPV6 = 1,
+ MAX_GFT_PROFILE_IP_VERSION
+};
+
+enum gft_profile_upper_protocol_type {
+ GFT_PROFILE_ROCE_PROTOCOL = 0,
+ GFT_PROFILE_RROCE_PROTOCOL = 1,
+ GFT_PROFILE_FCOE_PROTOCOL = 2,
+ GFT_PROFILE_ICMP_PROTOCOL = 3,
+ GFT_PROFILE_ARP_PROTOCOL = 4,
+ GFT_PROFILE_USER_TCP_SRC_PORT_1_INNER = 5,
+ GFT_PROFILE_USER_TCP_DST_PORT_1_INNER = 6,
+ GFT_PROFILE_TCP_PROTOCOL = 7,
+ GFT_PROFILE_USER_UDP_DST_PORT_1_INNER = 8,
+ GFT_PROFILE_USER_UDP_DST_PORT_2_OUTER = 9,
+ GFT_PROFILE_UDP_PROTOCOL = 10,
+ GFT_PROFILE_USER_IP_1_INNER = 11,
+ GFT_PROFILE_USER_IP_2_OUTER = 12,
+ GFT_PROFILE_USER_ETH_1_INNER = 13,
+ GFT_PROFILE_USER_ETH_2_OUTER = 14,
+ GFT_PROFILE_RAW = 15,
+ MAX_GFT_PROFILE_UPPER_PROTOCOL_TYPE
+};
+
+struct gft_ram_line {
+ __le32 low32bits;
+#define GFT_RAM_LINE_VLAN_SELECT_MASK 0x3
+#define GFT_RAM_LINE_VLAN_SELECT_SHIFT 0
+#define GFT_RAM_LINE_TUNNEL_ENTROPHY_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_ENTROPHY_SHIFT 2
+#define GFT_RAM_LINE_TUNNEL_TTL_EQUAL_ONE_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_TTL_EQUAL_ONE_SHIFT 3
+#define GFT_RAM_LINE_TUNNEL_TTL_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_TTL_SHIFT 4
+#define GFT_RAM_LINE_TUNNEL_ETHERTYPE_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_ETHERTYPE_SHIFT 5
+#define GFT_RAM_LINE_TUNNEL_DST_PORT_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_DST_PORT_SHIFT 6
+#define GFT_RAM_LINE_TUNNEL_SRC_PORT_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_SRC_PORT_SHIFT 7
+#define GFT_RAM_LINE_TUNNEL_DSCP_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_DSCP_SHIFT 8
+#define GFT_RAM_LINE_TUNNEL_OVER_IP_PROTOCOL_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_OVER_IP_PROTOCOL_SHIFT 9
+#define GFT_RAM_LINE_TUNNEL_DST_IP_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_DST_IP_SHIFT 10
+#define GFT_RAM_LINE_TUNNEL_SRC_IP_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_SRC_IP_SHIFT 11
+#define GFT_RAM_LINE_TUNNEL_PRIORITY_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_PRIORITY_SHIFT 12
+#define GFT_RAM_LINE_TUNNEL_PROVIDER_VLAN_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_PROVIDER_VLAN_SHIFT 13
+#define GFT_RAM_LINE_TUNNEL_VLAN_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_VLAN_SHIFT 14
+#define GFT_RAM_LINE_TUNNEL_DST_MAC_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_DST_MAC_SHIFT 15
+#define GFT_RAM_LINE_TUNNEL_SRC_MAC_MASK 0x1
+#define GFT_RAM_LINE_TUNNEL_SRC_MAC_SHIFT 16
+#define GFT_RAM_LINE_TTL_EQUAL_ONE_MASK 0x1
+#define GFT_RAM_LINE_TTL_EQUAL_ONE_SHIFT 17
+#define GFT_RAM_LINE_TTL_MASK 0x1
+#define GFT_RAM_LINE_TTL_SHIFT 18
+#define GFT_RAM_LINE_ETHERTYPE_MASK 0x1
+#define GFT_RAM_LINE_ETHERTYPE_SHIFT 19
+#define GFT_RAM_LINE_RESERVED0_MASK 0x1
+#define GFT_RAM_LINE_RESERVED0_SHIFT 20
+#define GFT_RAM_LINE_TCP_FLAG_FIN_MASK 0x1
+#define GFT_RAM_LINE_TCP_FLAG_FIN_SHIFT 21
+#define GFT_RAM_LINE_TCP_FLAG_SYN_MASK 0x1
+#define GFT_RAM_LINE_TCP_FLAG_SYN_SHIFT 22
+#define GFT_RAM_LINE_TCP_FLAG_RST_MASK 0x1
+#define GFT_RAM_LINE_TCP_FLAG_RST_SHIFT 23
+#define GFT_RAM_LINE_TCP_FLAG_PSH_MASK 0x1
+#define GFT_RAM_LINE_TCP_FLAG_PSH_SHIFT 24
+#define GFT_RAM_LINE_TCP_FLAG_ACK_MASK 0x1
+#define GFT_RAM_LINE_TCP_FLAG_ACK_SHIFT 25
+#define GFT_RAM_LINE_TCP_FLAG_URG_MASK 0x1
+#define GFT_RAM_LINE_TCP_FLAG_URG_SHIFT 26
+#define GFT_RAM_LINE_TCP_FLAG_ECE_MASK 0x1
+#define GFT_RAM_LINE_TCP_FLAG_ECE_SHIFT 27
+#define GFT_RAM_LINE_TCP_FLAG_CWR_MASK 0x1
+#define GFT_RAM_LINE_TCP_FLAG_CWR_SHIFT 28
+#define GFT_RAM_LINE_TCP_FLAG_NS_MASK 0x1
+#define GFT_RAM_LINE_TCP_FLAG_NS_SHIFT 29
+#define GFT_RAM_LINE_DST_PORT_MASK 0x1
+#define GFT_RAM_LINE_DST_PORT_SHIFT 30
+#define GFT_RAM_LINE_SRC_PORT_MASK 0x1
+#define GFT_RAM_LINE_SRC_PORT_SHIFT 31
+ __le32 high32bits;
+#define GFT_RAM_LINE_DSCP_MASK 0x1
+#define GFT_RAM_LINE_DSCP_SHIFT 0
+#define GFT_RAM_LINE_OVER_IP_PROTOCOL_MASK 0x1
+#define GFT_RAM_LINE_OVER_IP_PROTOCOL_SHIFT 1
+#define GFT_RAM_LINE_DST_IP_MASK 0x1
+#define GFT_RAM_LINE_DST_IP_SHIFT 2
+#define GFT_RAM_LINE_SRC_IP_MASK 0x1
+#define GFT_RAM_LINE_SRC_IP_SHIFT 3
+#define GFT_RAM_LINE_PRIORITY_MASK 0x1
+#define GFT_RAM_LINE_PRIORITY_SHIFT 4
+#define GFT_RAM_LINE_PROVIDER_VLAN_MASK 0x1
+#define GFT_RAM_LINE_PROVIDER_VLAN_SHIFT 5
+#define GFT_RAM_LINE_VLAN_MASK 0x1
+#define GFT_RAM_LINE_VLAN_SHIFT 6
+#define GFT_RAM_LINE_DST_MAC_MASK 0x1
+#define GFT_RAM_LINE_DST_MAC_SHIFT 7
+#define GFT_RAM_LINE_SRC_MAC_MASK 0x1
+#define GFT_RAM_LINE_SRC_MAC_SHIFT 8
+#define GFT_RAM_LINE_TENANT_ID_MASK 0x1
+#define GFT_RAM_LINE_TENANT_ID_SHIFT 9
+#define GFT_RAM_LINE_RESERVED1_MASK 0x3FFFFF
+#define GFT_RAM_LINE_RESERVED1_SHIFT 10
+};
+
+struct mstorm_eth_conn_ag_ctx {
+ u8 byte0;
+ u8 byte1;
+ u8 flags0;
+#define MSTORM_ETH_CONN_AG_CTX_EXIST_IN_QM0_MASK 0x1
+#define MSTORM_ETH_CONN_AG_CTX_EXIST_IN_QM0_SHIFT 0
+#define MSTORM_ETH_CONN_AG_CTX_BIT1_MASK 0x1
+#define MSTORM_ETH_CONN_AG_CTX_BIT1_SHIFT 1
+#define MSTORM_ETH_CONN_AG_CTX_CF0_MASK 0x3
+#define MSTORM_ETH_CONN_AG_CTX_CF0_SHIFT 2
+#define MSTORM_ETH_CONN_AG_CTX_CF1_MASK 0x3
+#define MSTORM_ETH_CONN_AG_CTX_CF1_SHIFT 4
+#define MSTORM_ETH_CONN_AG_CTX_CF2_MASK 0x3
+#define MSTORM_ETH_CONN_AG_CTX_CF2_SHIFT 6
+ u8 flags1;
+#define MSTORM_ETH_CONN_AG_CTX_CF0EN_MASK 0x1
+#define MSTORM_ETH_CONN_AG_CTX_CF0EN_SHIFT 0
+#define MSTORM_ETH_CONN_AG_CTX_CF1EN_MASK 0x1
+#define MSTORM_ETH_CONN_AG_CTX_CF1EN_SHIFT 1
+#define MSTORM_ETH_CONN_AG_CTX_CF2EN_MASK 0x1
+#define MSTORM_ETH_CONN_AG_CTX_CF2EN_SHIFT 2
+#define MSTORM_ETH_CONN_AG_CTX_RULE0EN_MASK 0x1
+#define MSTORM_ETH_CONN_AG_CTX_RULE0EN_SHIFT 3
+#define MSTORM_ETH_CONN_AG_CTX_RULE1EN_MASK 0x1
+#define MSTORM_ETH_CONN_AG_CTX_RULE1EN_SHIFT 4
+#define MSTORM_ETH_CONN_AG_CTX_RULE2EN_MASK 0x1
+#define MSTORM_ETH_CONN_AG_CTX_RULE2EN_SHIFT 5
+#define MSTORM_ETH_CONN_AG_CTX_RULE3EN_MASK 0x1
+#define MSTORM_ETH_CONN_AG_CTX_RULE3EN_SHIFT 6
+#define MSTORM_ETH_CONN_AG_CTX_RULE4EN_MASK 0x1
+#define MSTORM_ETH_CONN_AG_CTX_RULE4EN_SHIFT 7
+ __le16 word0;
+ __le16 word1;
+ __le32 reg0;
+ __le32 reg1;
+};
+
+struct xstorm_eth_conn_agctxdq_ext_ldpart {
+ u8 reserved0;
+ u8 eth_state;
+ u8 flags0;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_EXIST_IN_QM0_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_EXIST_IN_QM0_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED1_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED1_SHIFT 1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED2_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED2_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_EXIST_IN_QM3_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_EXIST_IN_QM3_SHIFT 3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED3_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED3_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED4_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED4_SHIFT 5
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED5_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED5_SHIFT 6
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED6_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED6_SHIFT 7
+ u8 flags1;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED7_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED7_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED8_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED8_SHIFT 1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED9_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED9_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_BIT11_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_BIT11_SHIFT 3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_BIT12_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_BIT12_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_BIT13_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_BIT13_SHIFT 5
+#define XSTORMETHCONNAGCTXDQEXTLDPART_TX_RULE_ACTIVE_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_TX_RULE_ACTIVE_SHIFT 6
+#define XSTORMETHCONNAGCTXDQEXTLDPART_DQ_CF_ACTIVE_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_DQ_CF_ACTIVE_SHIFT 7
+ u8 flags2;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF0_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF0_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF1_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF1_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF2_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF2_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF3_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF3_SHIFT 6
+ u8 flags3;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF4_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF4_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF5_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF5_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF6_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF6_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF7_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF7_SHIFT 6
+ u8 flags4;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF8_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF8_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF9_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF9_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF10_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF10_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF11_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF11_SHIFT 6
+ u8 flags5;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF12_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF12_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF13_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF13_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF14_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF14_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF15_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF15_SHIFT 6
+ u8 flags6;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_GO_TO_BD_CONS_CF_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_GO_TO_BD_CONS_CF_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_MULTI_UNICAST_CF_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_MULTI_UNICAST_CF_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_DQ_CF_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_DQ_CF_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_TERMINATE_CF_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_TERMINATE_CF_SHIFT 6
+ u8 flags7;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_FLUSH_Q0_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_FLUSH_Q0_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED10_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED10_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_SLOW_PATH_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_SLOW_PATH_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF0EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF0EN_SHIFT 6
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF1EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF1EN_SHIFT 7
+ u8 flags8;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF2EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF2EN_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF3EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF3EN_SHIFT 1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF4EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF4EN_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF5EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF5EN_SHIFT 3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF6EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF6EN_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF7EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF7EN_SHIFT 5
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF8EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF8EN_SHIFT 6
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF9EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF9EN_SHIFT 7
+ u8 flags9;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF10EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF10EN_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF11EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF11EN_SHIFT 1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF12EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF12EN_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF13EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF13EN_SHIFT 3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF14EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF14EN_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF15EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_CF15EN_SHIFT 5
+#define XSTORMETHCONNAGCTXDQEXTLDPART_GO_TO_BD_CONS_CF_EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_GO_TO_BD_CONS_CF_EN_SHIFT 6
+#define XSTORMETHCONNAGCTXDQEXTLDPART_MULTI_UNICAST_CF_EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_MULTI_UNICAST_CF_EN_SHIFT 7
+ u8 flags10;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_DQ_CF_EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_DQ_CF_EN_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_TERMINATE_CF_EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_TERMINATE_CF_EN_SHIFT 1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_FLUSH_Q0_EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_FLUSH_Q0_EN_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED11_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED11_SHIFT 3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_SLOW_PATH_EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_SLOW_PATH_EN_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_TPH_ENABLE_EN_RESERVED_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_TPH_ENABLE_EN_RESERVED_SHIFT 5
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED12_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED12_SHIFT 6
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED13_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED13_SHIFT 7
+ u8 flags11;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED14_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED14_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED15_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RESERVED15_SHIFT 1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_TX_DEC_RULE_EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_TX_DEC_RULE_EN_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE5EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE5EN_SHIFT 3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE6EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE6EN_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE7EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE7EN_SHIFT 5
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED1_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED1_SHIFT 6
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE9EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE9EN_SHIFT 7
+ u8 flags12;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE10EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE10EN_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE11EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE11EN_SHIFT 1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED2_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED2_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED3_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED3_SHIFT 3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE14EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE14EN_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE15EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE15EN_SHIFT 5
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE16EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE16EN_SHIFT 6
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE17EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE17EN_SHIFT 7
+ u8 flags13;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE18EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE18EN_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE19EN_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_RULE19EN_SHIFT 1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED4_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED4_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED5_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED5_SHIFT 3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED6_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED6_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED7_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED7_SHIFT 5
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED8_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED8_SHIFT 6
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED9_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_A0_RESERVED9_SHIFT 7
+ u8 flags14;
+#define XSTORMETHCONNAGCTXDQEXTLDPART_EDPM_USE_EXT_HDR_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_EDPM_USE_EXT_HDR_SHIFT 0
+#define XSTORMETHCONNAGCTXDQEXTLDPART_EDPM_SEND_RAW_L3L4_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_EDPM_SEND_RAW_L3L4_SHIFT 1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_EDPM_INBAND_PROP_HDR_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_EDPM_INBAND_PROP_HDR_SHIFT 2
+#define XSTORMETHCONNAGCTXDQEXTLDPART_EDPM_SEND_EXT_TUNNEL_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_EDPM_SEND_EXT_TUNNEL_SHIFT 3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_L2_EDPM_ENABLE_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_L2_EDPM_ENABLE_SHIFT 4
+#define XSTORMETHCONNAGCTXDQEXTLDPART_ROCE_EDPM_ENABLE_MASK 0x1
+#define XSTORMETHCONNAGCTXDQEXTLDPART_ROCE_EDPM_ENABLE_SHIFT 5
+#define XSTORMETHCONNAGCTXDQEXTLDPART_TPH_ENABLE_MASK 0x3
+#define XSTORMETHCONNAGCTXDQEXTLDPART_TPH_ENABLE_SHIFT 6
+ u8 edpm_event_id;
+ __le16 physical_q0;
+ __le16 quota;
+ __le16 edpm_num_bds;
+ __le16 tx_bd_cons;
+ __le16 tx_bd_prod;
+ __le16 tx_class;
+ __le16 conn_dpi;
+ u8 byte3;
+ u8 byte4;
+ u8 byte5;
+ u8 byte6;
+ __le32 reg0;
+ __le32 reg1;
+ __le32 reg2;
+ __le32 reg3;
+ __le32 reg4;
+};
+
+struct xstorm_eth_hw_conn_ag_ctx {
+ u8 reserved0;
+ u8 eth_state;
+ u8 flags0;
+#define XSTORM_ETH_HW_CONN_AG_CTX_EXIST_IN_QM0_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_EXIST_IN_QM0_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED1_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED1_SHIFT 1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED2_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED2_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_EXIST_IN_QM3_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_EXIST_IN_QM3_SHIFT 3
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED3_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED3_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED4_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED4_SHIFT 5
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED5_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED5_SHIFT 6
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED6_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED6_SHIFT 7
+ u8 flags1;
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED7_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED7_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED8_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED8_SHIFT 1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED9_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED9_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_BIT11_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_BIT11_SHIFT 3
+#define XSTORM_ETH_HW_CONN_AG_CTX_BIT12_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_BIT12_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_BIT13_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_BIT13_SHIFT 5
+#define XSTORM_ETH_HW_CONN_AG_CTX_TX_RULE_ACTIVE_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_TX_RULE_ACTIVE_SHIFT 6
+#define XSTORM_ETH_HW_CONN_AG_CTX_DQ_CF_ACTIVE_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_DQ_CF_ACTIVE_SHIFT 7
+ u8 flags2;
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF0_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF0_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF1_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF1_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF2_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF2_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF3_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF3_SHIFT 6
+ u8 flags3;
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF4_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF4_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF5_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF5_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF6_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF6_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF7_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF7_SHIFT 6
+ u8 flags4;
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF8_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF8_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF9_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF9_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF10_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF10_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF11_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF11_SHIFT 6
+ u8 flags5;
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF12_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF12_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF13_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF13_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF14_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF14_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF15_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF15_SHIFT 6
+ u8 flags6;
+#define XSTORM_ETH_HW_CONN_AG_CTX_GO_TO_BD_CONS_CF_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_GO_TO_BD_CONS_CF_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_MULTI_UNICAST_CF_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_MULTI_UNICAST_CF_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_DQ_CF_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_DQ_CF_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_TERMINATE_CF_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_TERMINATE_CF_SHIFT 6
+ u8 flags7;
+#define XSTORM_ETH_HW_CONN_AG_CTX_FLUSH_Q0_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_FLUSH_Q0_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED10_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED10_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_SLOW_PATH_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_SLOW_PATH_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF0EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF0EN_SHIFT 6
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF1EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF1EN_SHIFT 7
+ u8 flags8;
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF2EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF2EN_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF3EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF3EN_SHIFT 1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF4EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF4EN_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF5EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF5EN_SHIFT 3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF6EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF6EN_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF7EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF7EN_SHIFT 5
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF8EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF8EN_SHIFT 6
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF9EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF9EN_SHIFT 7
+ u8 flags9;
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF10EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF10EN_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF11EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF11EN_SHIFT 1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF12EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF12EN_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF13EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF13EN_SHIFT 3
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF14EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF14EN_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF15EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_CF15EN_SHIFT 5
+#define XSTORM_ETH_HW_CONN_AG_CTX_GO_TO_BD_CONS_CF_EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_GO_TO_BD_CONS_CF_EN_SHIFT 6
+#define XSTORM_ETH_HW_CONN_AG_CTX_MULTI_UNICAST_CF_EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_MULTI_UNICAST_CF_EN_SHIFT 7
+ u8 flags10;
+#define XSTORM_ETH_HW_CONN_AG_CTX_DQ_CF_EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_DQ_CF_EN_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_TERMINATE_CF_EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_TERMINATE_CF_EN_SHIFT 1
+#define XSTORM_ETH_HW_CONN_AG_CTX_FLUSH_Q0_EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_FLUSH_Q0_EN_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED11_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED11_SHIFT 3
+#define XSTORM_ETH_HW_CONN_AG_CTX_SLOW_PATH_EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_SLOW_PATH_EN_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_TPH_ENABLE_EN_RESERVED_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_TPH_ENABLE_EN_RESERVED_SHIFT 5
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED12_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED12_SHIFT 6
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED13_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED13_SHIFT 7
+ u8 flags11;
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED14_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED14_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED15_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RESERVED15_SHIFT 1
+#define XSTORM_ETH_HW_CONN_AG_CTX_TX_DEC_RULE_EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_TX_DEC_RULE_EN_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE5EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE5EN_SHIFT 3
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE6EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE6EN_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE7EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE7EN_SHIFT 5
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED1_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED1_SHIFT 6
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE9EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE9EN_SHIFT 7
+ u8 flags12;
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE10EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE10EN_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE11EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE11EN_SHIFT 1
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED2_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED2_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED3_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED3_SHIFT 3
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE14EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE14EN_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE15EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE15EN_SHIFT 5
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE16EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE16EN_SHIFT 6
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE17EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE17EN_SHIFT 7
+ u8 flags13;
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE18EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE18EN_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE19EN_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_RULE19EN_SHIFT 1
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED4_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED4_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED5_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED5_SHIFT 3
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED6_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED6_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED7_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED7_SHIFT 5
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED8_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED8_SHIFT 6
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED9_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_A0_RESERVED9_SHIFT 7
+ u8 flags14;
+#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_USE_EXT_HDR_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_USE_EXT_HDR_SHIFT 0
+#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_SEND_RAW_L3L4_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_SEND_RAW_L3L4_SHIFT 1
+#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_INBAND_PROP_HDR_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_INBAND_PROP_HDR_SHIFT 2
+#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_SEND_EXT_TUNNEL_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_EDPM_SEND_EXT_TUNNEL_SHIFT 3
+#define XSTORM_ETH_HW_CONN_AG_CTX_L2_EDPM_ENABLE_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_L2_EDPM_ENABLE_SHIFT 4
+#define XSTORM_ETH_HW_CONN_AG_CTX_ROCE_EDPM_ENABLE_MASK 0x1
+#define XSTORM_ETH_HW_CONN_AG_CTX_ROCE_EDPM_ENABLE_SHIFT 5
+#define XSTORM_ETH_HW_CONN_AG_CTX_TPH_ENABLE_MASK 0x3
+#define XSTORM_ETH_HW_CONN_AG_CTX_TPH_ENABLE_SHIFT 6
+ u8 edpm_event_id;
+ __le16 physical_q0;
+ __le16 quota;
+ __le16 edpm_num_bds;
+ __le16 tx_bd_cons;
+ __le16 tx_bd_prod;
+ __le16 tx_class;
+ __le16 conn_dpi;
+};
+
struct mstorm_rdma_task_st_ctx {
struct regpair temp[4];
};
@@ -6165,7 +6951,7 @@
};
struct xstorm_roce_conn_st_ctx {
- struct regpair temp[22];
+ struct regpair temp[24];
};
struct tstorm_roce_conn_st_ctx {
@@ -6220,7 +7006,7 @@
__le16 mtu;
__le16 pd;
__le16 sq_num_pages;
- __le16 reseved2;
+ __le16 low_latency_phy_queue;
struct regpair sq_pbl_addr;
struct regpair orq_pbl_addr;
__le16 local_mac_addr[3];
@@ -6234,7 +7020,7 @@
u8 stats_counter_id;
u8 reserved3[7];
__le32 cq_cid;
- __le16 physical_queue0;
+ __le16 regular_latency_phy_queue;
__le16 dpi;
};
@@ -6282,15 +7068,16 @@
__le32 dst_gid[4];
struct regpair qp_handle_for_cqe;
struct regpair qp_handle_for_async;
- __le32 reserved2[2];
+ __le16 low_latency_phy_queue;
+ u8 reserved2[6];
__le32 cq_cid;
- __le16 physical_queue0;
+ __le16 regular_latency_phy_queue;
__le16 dpi;
};
struct roce_destroy_qp_req_output_params {
__le32 num_bound_mw;
- __le32 reserved;
+ __le32 cq_prod;
};
struct roce_destroy_qp_req_ramrod_data {
@@ -6299,7 +7086,7 @@
struct roce_destroy_qp_resp_output_params {
__le32 num_invalidated_mw;
- __le32 reserved;
+ __le32 cq_prod;
};
struct roce_destroy_qp_resp_ramrod_data {
@@ -7426,6 +8213,7 @@
u8 fcp_rsp_size;
__le16 mss;
struct regpair reserved;
+ __le16 min_frame_size;
u8 protection_info_flags;
#define YSTORM_FCOE_CONN_ST_CTX_SUPPORT_PROTECTION_MASK 0x1
#define YSTORM_FCOE_CONN_ST_CTX_SUPPORT_PROTECTION_SHIFT 0
@@ -7444,7 +8232,6 @@
#define YSTORM_FCOE_CONN_ST_CTX_RSRV_MASK 0x3F
#define YSTORM_FCOE_CONN_ST_CTX_RSRV_SHIFT 2
u8 fcp_xfer_size;
- u8 reserved3[2];
};
struct fcoe_vlan_fields {
@@ -8273,10 +9060,10 @@
#define XSTORM_ISCSI_CONN_AG_CTX_DQ_FLUSH_MASK 0x3
#define XSTORM_ISCSI_CONN_AG_CTX_DQ_FLUSH_SHIFT 6
u8 flags7;
-#define XSTORM_ISCSI_CONN_AG_CTX_FLUSH_Q0_MASK 0x3
-#define XSTORM_ISCSI_CONN_AG_CTX_FLUSH_Q0_SHIFT 0
-#define XSTORM_ISCSI_CONN_AG_CTX_FLUSH_Q1_MASK 0x3
-#define XSTORM_ISCSI_CONN_AG_CTX_FLUSH_Q1_SHIFT 2
+#define XSTORM_ISCSI_CONN_AG_CTX_MST_XCM_Q0_FLUSH_CF_MASK 0x3
+#define XSTORM_ISCSI_CONN_AG_CTX_MST_XCM_Q0_FLUSH_CF_SHIFT 0
+#define XSTORM_ISCSI_CONN_AG_CTX_UST_XCM_Q1_FLUSH_CF_MASK 0x3
+#define XSTORM_ISCSI_CONN_AG_CTX_UST_XCM_Q1_FLUSH_CF_SHIFT 2
#define XSTORM_ISCSI_CONN_AG_CTX_SLOW_PATH_MASK 0x3
#define XSTORM_ISCSI_CONN_AG_CTX_SLOW_PATH_SHIFT 4
#define XSTORM_ISCSI_CONN_AG_CTX_CF0EN_MASK 0x1
@@ -8322,10 +9109,10 @@
#define XSTORM_ISCSI_CONN_AG_CTX_CF18EN_SHIFT 0
#define XSTORM_ISCSI_CONN_AG_CTX_DQ_FLUSH_EN_MASK 0x1
#define XSTORM_ISCSI_CONN_AG_CTX_DQ_FLUSH_EN_SHIFT 1
-#define XSTORM_ISCSI_CONN_AG_CTX_FLUSH_Q0_EN_MASK 0x1
-#define XSTORM_ISCSI_CONN_AG_CTX_FLUSH_Q0_EN_SHIFT 2
-#define XSTORM_ISCSI_CONN_AG_CTX_FLUSH_Q1_EN_MASK 0x1
-#define XSTORM_ISCSI_CONN_AG_CTX_FLUSH_Q1_EN_SHIFT 3
+#define XSTORM_ISCSI_CONN_AG_CTX_MST_XCM_Q0_FLUSH_CF_EN_MASK 0x1
+#define XSTORM_ISCSI_CONN_AG_CTX_MST_XCM_Q0_FLUSH_CF_EN_SHIFT 2
+#define XSTORM_ISCSI_CONN_AG_CTX_UST_XCM_Q1_FLUSH_CF_EN_MASK 0x1
+#define XSTORM_ISCSI_CONN_AG_CTX_UST_XCM_Q1_FLUSH_CF_EN_SHIFT 3
#define XSTORM_ISCSI_CONN_AG_CTX_SLOW_PATH_EN_MASK 0x1
#define XSTORM_ISCSI_CONN_AG_CTX_SLOW_PATH_EN_SHIFT 4
#define XSTORM_ISCSI_CONN_AG_CTX_PROC_ONLY_CLEANUP_EN_MASK 0x1
@@ -8335,8 +9122,8 @@
#define XSTORM_ISCSI_CONN_AG_CTX_MORE_TO_SEND_DEC_RULE_EN_MASK 0x1
#define XSTORM_ISCSI_CONN_AG_CTX_MORE_TO_SEND_DEC_RULE_EN_SHIFT 7
u8 flags11;
-#define XSTORM_ISCSI_CONN_AG_CTX_RULE2EN_MASK 0x1
-#define XSTORM_ISCSI_CONN_AG_CTX_RULE2EN_SHIFT 0
+#define XSTORM_ISCSI_CONN_AG_CTX_TX_BLOCKED_EN_MASK 0x1
+#define XSTORM_ISCSI_CONN_AG_CTX_TX_BLOCKED_EN_SHIFT 0
#define XSTORM_ISCSI_CONN_AG_CTX_RULE3EN_MASK 0x1
#define XSTORM_ISCSI_CONN_AG_CTX_RULE3EN_SHIFT 1
#define XSTORM_ISCSI_CONN_AG_CTX_RESERVED3_MASK 0x1
@@ -8440,7 +9227,7 @@
__le32 reg10;
__le32 reg11;
__le32 exp_stat_sn;
- __le32 reg13;
+ __le32 ongoing_fast_rxmit_seq;
__le32 reg14;
__le32 reg15;
__le32 reg16;
@@ -8466,10 +9253,10 @@
#define TSTORM_ISCSI_CONN_AG_CTX_CF0_MASK 0x3
#define TSTORM_ISCSI_CONN_AG_CTX_CF0_SHIFT 6
u8 flags1;
-#define TSTORM_ISCSI_CONN_AG_CTX_CF1_MASK 0x3
-#define TSTORM_ISCSI_CONN_AG_CTX_CF1_SHIFT 0
-#define TSTORM_ISCSI_CONN_AG_CTX_CF2_MASK 0x3
-#define TSTORM_ISCSI_CONN_AG_CTX_CF2_SHIFT 2
+#define TSTORM_ISCSI_CONN_AG_CTX_P2T_FLUSH_CF_MASK 0x3
+#define TSTORM_ISCSI_CONN_AG_CTX_P2T_FLUSH_CF_SHIFT 0
+#define TSTORM_ISCSI_CONN_AG_CTX_M2T_FLUSH_CF_MASK 0x3
+#define TSTORM_ISCSI_CONN_AG_CTX_M2T_FLUSH_CF_SHIFT 2
#define TSTORM_ISCSI_CONN_AG_CTX_TIMER_STOP_ALL_MASK 0x3
#define TSTORM_ISCSI_CONN_AG_CTX_TIMER_STOP_ALL_SHIFT 4
#define TSTORM_ISCSI_CONN_AG_CTX_CF4_MASK 0x3
@@ -8490,10 +9277,10 @@
#define TSTORM_ISCSI_CONN_AG_CTX_CF10_SHIFT 2
#define TSTORM_ISCSI_CONN_AG_CTX_CF0EN_MASK 0x1
#define TSTORM_ISCSI_CONN_AG_CTX_CF0EN_SHIFT 4
-#define TSTORM_ISCSI_CONN_AG_CTX_CF1EN_MASK 0x1
-#define TSTORM_ISCSI_CONN_AG_CTX_CF1EN_SHIFT 5
-#define TSTORM_ISCSI_CONN_AG_CTX_CF2EN_MASK 0x1
-#define TSTORM_ISCSI_CONN_AG_CTX_CF2EN_SHIFT 6
+#define TSTORM_ISCSI_CONN_AG_CTX_P2T_FLUSH_CF_EN_MASK 0x1
+#define TSTORM_ISCSI_CONN_AG_CTX_P2T_FLUSH_CF_EN_SHIFT 5
+#define TSTORM_ISCSI_CONN_AG_CTX_M2T_FLUSH_CF_EN_MASK 0x1
+#define TSTORM_ISCSI_CONN_AG_CTX_M2T_FLUSH_CF_EN_SHIFT 6
#define TSTORM_ISCSI_CONN_AG_CTX_TIMER_STOP_ALL_EN_MASK 0x1
#define TSTORM_ISCSI_CONN_AG_CTX_TIMER_STOP_ALL_EN_SHIFT 7
u8 flags4;
@@ -8539,7 +9326,7 @@
__le32 reg6;
__le32 reg7;
__le32 reg8;
- u8 byte2;
+ u8 cid_offload_cnt;
u8 byte3;
__le16 word0;
};
@@ -8831,11 +9618,24 @@
u64 r511;
u64 r1023;
u64 r1518;
- u64 r1522;
- u64 r2047;
- u64 r4095;
- u64 r9216;
- u64 r16383;
+
+ union {
+ struct {
+ u64 r1522;
+ u64 r2047;
+ u64 r4095;
+ u64 r9216;
+ u64 r16383;
+ } bb0;
+ struct {
+ u64 unused1;
+ u64 r1519_to_max;
+ u64 unused2;
+ u64 unused3;
+ u64 unused4;
+ } ah0;
+ } u0;
+
u64 rfcs;
u64 rxcf;
u64 rxpf;
@@ -8852,14 +9652,36 @@
u64 t511;
u64 t1023;
u64 t1518;
- u64 t2047;
- u64 t4095;
- u64 t9216;
- u64 t16383;
+
+ union {
+ struct {
+ u64 t2047;
+ u64 t4095;
+ u64 t9216;
+ u64 t16383;
+ } bb1;
+ struct {
+ u64 t1519_to_max;
+ u64 unused6;
+ u64 unused7;
+ u64 unused8;
+ } ah1;
+ } u1;
+
u64 txpf;
u64 txpp;
- u64 tlpiec;
- u64 tncl;
+
+ union {
+ struct {
+ u64 tlpiec;
+ u64 tncl;
+ } bb2;
+ struct {
+ u64 unused9;
+ u64 unused10;
+ } ah2;
+ } u2;
+
u64 rbyte;
u64 rxuca;
u64 rxmca;
@@ -8943,12 +9765,12 @@
#define DCBX_ETS_CBS_SHIFT 3
#define DCBX_ETS_MAX_TCS_MASK 0x000000f0
#define DCBX_ETS_MAX_TCS_SHIFT 4
-#define DCBX_ISCSI_OOO_TC_MASK 0x00000f00
-#define DCBX_ISCSI_OOO_TC_SHIFT 8
+#define DCBX_OOO_TC_MASK 0x00000f00
+#define DCBX_OOO_TC_SHIFT 8
u32 pri_tc_tbl[1];
-#define DCBX_ISCSI_OOO_TC (4)
+#define DCBX_TCP_OOO_TC (4)
-#define NIG_ETS_ISCSI_OOO_CLIENT_OFFSET (DCBX_ISCSI_OOO_TC + 1)
+#define NIG_ETS_ISCSI_OOO_CLIENT_OFFSET (DCBX_TCP_OOO_TC + 1)
#define DCBX_CEE_STRICT_PRIORITY 0xf
u32 tc_bw_tbl[2];
u32 tc_tsa_tbl[2];
@@ -8957,6 +9779,9 @@
#define DCBX_ETS_TSA_ETS 2
};
+#define DCBX_TCP_OOO_TC (4)
+#define DCBX_TCP_OOO_K2_4PORT_TC (3)
+
struct dcbx_app_priority_entry {
u32 entry;
#define DCBX_APP_PRI_MAP_MASK 0x000000ff
@@ -9067,6 +9892,10 @@
struct public_global {
u32 max_path;
u32 max_ports;
+#define MODE_1P 1
+#define MODE_2P 2
+#define MODE_3P 3
+#define MODE_4P 4
u32 debug_mb_offset;
u32 phymod_dbg_mb_offset;
struct couple_mode_teaming cmt;
@@ -9248,9 +10077,11 @@
#define DRV_ID_PDA_COMP_VER_MASK 0x0000ffff
#define DRV_ID_PDA_COMP_VER_SHIFT 0
+#define LOAD_REQ_HSI_VERSION 2
#define DRV_ID_MCP_HSI_VER_MASK 0x00ff0000
#define DRV_ID_MCP_HSI_VER_SHIFT 16
-#define DRV_ID_MCP_HSI_VER_CURRENT (1 << DRV_ID_MCP_HSI_VER_SHIFT)
+#define DRV_ID_MCP_HSI_VER_CURRENT (LOAD_REQ_HSI_VERSION << \
+ DRV_ID_MCP_HSI_VER_SHIFT)
#define DRV_ID_DRV_TYPE_MASK 0x7f000000
#define DRV_ID_DRV_TYPE_SHIFT 24
@@ -9345,6 +10176,7 @@
RESOURCE_NUM_RSS_ENGINES_E = 14,
RESOURCE_LL2_QUEUE_E = 15,
RESOURCE_RDMA_STATS_QUEUE_E = 16,
+ RESOURCE_BDQ_E = 17,
RESOURCE_MAX_NUM,
RESOURCE_NUM_INVALID = 0xFFFFFFFF
};
@@ -9362,6 +10194,46 @@
#define RESOURCE_ELEMENT_STRICT (1 << 0)
};
+#define DRV_ROLE_NONE 0
+#define DRV_ROLE_PREBOOT 1
+#define DRV_ROLE_OS 2
+#define DRV_ROLE_KDUMP 3
+
+struct load_req_stc {
+ u32 drv_ver_0;
+ u32 drv_ver_1;
+ u32 fw_ver;
+ u32 misc0;
+#define LOAD_REQ_ROLE_MASK 0x000000FF
+#define LOAD_REQ_ROLE_SHIFT 0
+#define LOAD_REQ_LOCK_TO_MASK 0x0000FF00
+#define LOAD_REQ_LOCK_TO_SHIFT 8
+#define LOAD_REQ_LOCK_TO_DEFAULT 0
+#define LOAD_REQ_LOCK_TO_NONE 255
+#define LOAD_REQ_FORCE_MASK 0x000F0000
+#define LOAD_REQ_FORCE_SHIFT 16
+#define LOAD_REQ_FORCE_NONE 0
+#define LOAD_REQ_FORCE_PF 1
+#define LOAD_REQ_FORCE_ALL 2
+#define LOAD_REQ_FLAGS0_MASK 0x00F00000
+#define LOAD_REQ_FLAGS0_SHIFT 20
+#define LOAD_REQ_FLAGS0_AVOID_RESET (0x1 << 0)
+};
+
+struct load_rsp_stc {
+ u32 drv_ver_0;
+ u32 drv_ver_1;
+ u32 fw_ver;
+ u32 misc0;
+#define LOAD_RSP_ROLE_MASK 0x000000FF
+#define LOAD_RSP_ROLE_SHIFT 0
+#define LOAD_RSP_HSI_MASK 0x0000FF00
+#define LOAD_RSP_HSI_SHIFT 8
+#define LOAD_RSP_FLAGS0_MASK 0x000F0000
+#define LOAD_RSP_FLAGS0_SHIFT 16
+#define LOAD_RSP_FLAGS0_DRV_EXISTS (0x1 << 0)
+};
+
union drv_union_data {
u32 ver_str[MCP_DRV_VER_STR_SIZE_DWORD];
struct mcp_mac wol_mac;
@@ -9393,6 +10265,7 @@
#define DRV_MSG_CODE_LOAD_REQ 0x10000000
#define DRV_MSG_CODE_LOAD_DONE 0x11000000
#define DRV_MSG_CODE_INIT_HW 0x12000000
+#define DRV_MSG_CODE_CANCEL_LOAD_REQ 0x13000000
#define DRV_MSG_CODE_UNLOAD_REQ 0x20000000
#define DRV_MSG_CODE_UNLOAD_DONE 0x21000000
#define DRV_MSG_CODE_INIT_PHY 0x22000000
@@ -9405,12 +10278,14 @@
#define DRV_MSG_CODE_OV_UPDATE_DRIVER_STATE 0x31000000
#define DRV_MSG_CODE_BW_UPDATE_ACK 0x32000000
#define DRV_MSG_CODE_OV_UPDATE_MTU 0x33000000
+#define DRV_MSG_GET_RESOURCE_ALLOC_MSG 0x34000000
+#define DRV_MSG_SET_RESOURCE_VALUE_MSG 0x35000000
#define DRV_MSG_CODE_OV_UPDATE_WOL 0x38000000
#define DRV_MSG_CODE_OV_UPDATE_ESWITCH_MODE 0x39000000
#define DRV_MSG_CODE_BW_UPDATE_ACK 0x32000000
#define DRV_MSG_CODE_NIG_DRAIN 0x30000000
-#define DRV_MSG_GET_RESOURCE_ALLOC_MSG 0x34000000
+#define DRV_MSG_CODE_INITIATE_PF_FLR 0x02010000
#define DRV_MSG_CODE_VF_DISABLED_DONE 0xc0000000
#define DRV_MSG_CODE_CFG_VF_MSIX 0xc0010000
#define DRV_MSG_CODE_NVM_GET_FILE_ATT 0x00030000
@@ -9436,6 +10311,33 @@
#define DRV_MSG_CODE_BIST_TEST 0x001e0000
#define DRV_MSG_CODE_SET_LED_MODE 0x00200000
+#define DRV_MSG_CODE_RESOURCE_CMD 0x00230000
+
+#define RESOURCE_CMD_REQ_RESC_MASK 0x0000001F
+#define RESOURCE_CMD_REQ_RESC_SHIFT 0
+#define RESOURCE_CMD_REQ_OPCODE_MASK 0x000000E0
+#define RESOURCE_CMD_REQ_OPCODE_SHIFT 5
+#define RESOURCE_OPCODE_REQ 1
+#define RESOURCE_OPCODE_REQ_WO_AGING 2
+#define RESOURCE_OPCODE_REQ_W_AGING 3
+#define RESOURCE_OPCODE_RELEASE 4
+#define RESOURCE_OPCODE_FORCE_RELEASE 5
+#define RESOURCE_CMD_REQ_AGE_MASK 0x0000FF00
+#define RESOURCE_CMD_REQ_AGE_SHIFT 8
+
+#define RESOURCE_CMD_RSP_OWNER_MASK 0x000000FF
+#define RESOURCE_CMD_RSP_OWNER_SHIFT 0
+#define RESOURCE_CMD_RSP_OPCODE_MASK 0x00000700
+#define RESOURCE_CMD_RSP_OPCODE_SHIFT 8
+#define RESOURCE_OPCODE_GNT 1
+#define RESOURCE_OPCODE_BUSY 2
+#define RESOURCE_OPCODE_RELEASED 3
+#define RESOURCE_OPCODE_RELEASED_PREVIOUS 4
+#define RESOURCE_OPCODE_WRONG_OWNER 5
+#define RESOURCE_OPCODE_UNKNOWN_CMD 255
+
+#define RESOURCE_DUMP 0
+
#define DRV_MSG_CODE_GET_PF_RDMA_PROTOCOL 0x002b0000
#define DRV_MSG_CODE_OS_WOL 0x002e0000
@@ -9524,12 +10426,16 @@
u32 fw_mb_header;
#define FW_MSG_CODE_MASK 0xffff0000
+#define FW_MSG_CODE_UNSUPPORTED 0x00000000
#define FW_MSG_CODE_DRV_LOAD_ENGINE 0x10100000
#define FW_MSG_CODE_DRV_LOAD_PORT 0x10110000
#define FW_MSG_CODE_DRV_LOAD_FUNCTION 0x10120000
#define FW_MSG_CODE_DRV_LOAD_REFUSED_PDA 0x10200000
-#define FW_MSG_CODE_DRV_LOAD_REFUSED_HSI 0x10210000
+#define FW_MSG_CODE_DRV_LOAD_REFUSED_HSI_1 0x10210000
#define FW_MSG_CODE_DRV_LOAD_REFUSED_DIAG 0x10220000
+#define FW_MSG_CODE_DRV_LOAD_REFUSED_HSI 0x10230000
+#define FW_MSG_CODE_DRV_LOAD_REFUSED_REQUIRES_FORCE 0x10300000
+#define FW_MSG_CODE_DRV_LOAD_REFUSED_REJECT 0x10310000
#define FW_MSG_CODE_DRV_LOAD_DONE 0x11100000
#define FW_MSG_CODE_DRV_UNLOAD_ENGINE 0x20110000
#define FW_MSG_CODE_DRV_UNLOAD_PORT 0x20120000
@@ -9549,6 +10455,10 @@
#define FW_MSG_SEQ_NUMBER_MASK 0x0000ffff
u32 fw_mb_param;
+#define FW_MB_PARAM_RESOURCE_ALLOC_VERSION_MAJOR_MASK 0xFFFF0000
+#define FW_MB_PARAM_RESOURCE_ALLOC_VERSION_MAJOR_SHIFT 16
+#define FW_MB_PARAM_RESOURCE_ALLOC_VERSION_MINOR_MASK 0x0000FFFF
+#define FW_MB_PARAM_RESOURCE_ALLOC_VERSION_MINOR_SHIFT 0
/* get pf rdma protocol command responce */
#define FW_MB_PARAM_GET_PF_RDMA_NONE 0x0
@@ -9659,6 +10569,8 @@
#define NVM_CFG1_GLOB_NETWORK_PORT_MODE_2X25G 0xC
#define NVM_CFG1_GLOB_NETWORK_PORT_MODE_1X25G 0xD
#define NVM_CFG1_GLOB_NETWORK_PORT_MODE_4X25G 0xE
+#define NVM_CFG1_GLOB_NETWORK_PORT_MODE_2X10G 0xF
+
u32 e_lane_cfg1;
u32 e_lane_cfg2;
u32 f_lane_cfg1;
diff --git a/drivers/net/ethernet/qlogic/qed/qed_hw.c b/drivers/net/ethernet/qlogic/qed/qed_hw.c
index 899cad7..a05feb3 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_hw.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_hw.c
@@ -58,6 +58,7 @@
struct list_head list_entry;
unsigned int idx;
struct pxp_ptt_entry pxp;
+ u8 hwfn_id;
};
struct qed_ptt_pool {
@@ -79,6 +80,7 @@
p_pool->ptts[i].idx = i;
p_pool->ptts[i].pxp.offset = QED_BAR_INVALID_OFFSET;
p_pool->ptts[i].pxp.pretend.control = 0;
+ p_pool->ptts[i].hwfn_id = p_hwfn->my_id;
if (i >= RESERVED_PTT_MAX)
list_add(&p_pool->ptts[i].list_entry,
&p_pool->free_list);
@@ -193,6 +195,11 @@
offset = hw_addr - win_hw_addr;
+ if (p_ptt->hwfn_id != p_hwfn->my_id)
+ DP_NOTICE(p_hwfn,
+ "ptt[%d] of hwfn[%02x] is used by hwfn[%02x]!\n",
+ p_ptt->idx, p_ptt->hwfn_id, p_hwfn->my_id);
+
/* Verify the address is within the window */
if (hw_addr < win_hw_addr ||
offset >= PXP_EXTERNAL_BAR_PF_WINDOW_SINGLE_SIZE) {
@@ -800,55 +807,3 @@
return rc;
}
-u16 qed_get_qm_pq(struct qed_hwfn *p_hwfn,
- enum protocol_type proto, union qed_qm_pq_params *p_params)
-{
- u16 pq_id = 0;
-
- if ((proto == PROTOCOLID_CORE ||
- proto == PROTOCOLID_ETH ||
- proto == PROTOCOLID_ISCSI ||
- proto == PROTOCOLID_ROCE) && !p_params) {
- DP_NOTICE(p_hwfn,
- "Protocol %d received NULL PQ params\n", proto);
- return 0;
- }
-
- switch (proto) {
- case PROTOCOLID_CORE:
- if (p_params->core.tc == LB_TC)
- pq_id = p_hwfn->qm_info.pure_lb_pq;
- else if (p_params->core.tc == OOO_LB_TC)
- pq_id = p_hwfn->qm_info.ooo_pq;
- else
- pq_id = p_hwfn->qm_info.offload_pq;
- break;
- case PROTOCOLID_ETH:
- pq_id = p_params->eth.tc;
- if (p_params->eth.is_vf)
- pq_id += p_hwfn->qm_info.vf_queues_offset +
- p_params->eth.vf_id;
- break;
- case PROTOCOLID_ISCSI:
- if (p_params->iscsi.q_idx == 1)
- pq_id = p_hwfn->qm_info.pure_ack_pq;
- break;
- case PROTOCOLID_ROCE:
- if (p_params->roce.dcqcn)
- pq_id = p_params->roce.qpid;
- else
- pq_id = p_hwfn->qm_info.offload_pq;
- if (pq_id > p_hwfn->qm_info.num_pf_rls)
- pq_id = p_hwfn->qm_info.offload_pq;
- break;
- case PROTOCOLID_FCOE:
- pq_id = p_hwfn->qm_info.offload_pq;
- break;
- default:
- pq_id = 0;
- }
-
- pq_id = CM_TX_PQ_BASE + pq_id + RESC_START(p_hwfn, QED_PQ);
-
- return pq_id;
-}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_hw.h b/drivers/net/ethernet/qlogic/qed/qed_hw.h
index 9277264..f2505c6 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_hw.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_hw.h
@@ -297,9 +297,6 @@
} roce;
};
-u16 qed_get_qm_pq(struct qed_hwfn *p_hwfn,
- enum protocol_type proto, union qed_qm_pq_params *params);
-
int qed_init_fw_data(struct qed_dev *cdev,
const u8 *fw_data);
#endif
diff --git a/drivers/net/ethernet/qlogic/qed/qed_init_fw_funcs.c b/drivers/net/ethernet/qlogic/qed/qed_init_fw_funcs.c
index d891a68..67200c5 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_init_fw_funcs.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_init_fw_funcs.c
@@ -215,13 +215,6 @@
{
u32 qm_line_crd;
- /* In A0 - Limit the size of pbf queue so that only 511 commands with
- * the minimum size of 4 (FCoE minimum size)
- */
- bool is_bb_a0 = QED_IS_BB_A0(p_hwfn->cdev);
-
- if (is_bb_a0)
- cmdq_lines = min_t(u32, cmdq_lines, 1022);
qm_line_crd = QM_VOQ_LINE_CRD(cmdq_lines);
OVERWRITE_RT_REG(p_hwfn, PBF_CMDQ_LINES_RT_OFFSET(voq),
(u32)cmdq_lines);
@@ -343,13 +336,11 @@
u16 first_pq_group = p_params->start_pq / QM_PF_QUEUE_GROUP_SIZE;
u16 last_pq_group = (p_params->start_pq + num_pqs - 1) /
QM_PF_QUEUE_GROUP_SIZE;
- bool is_bb_a0 = QED_IS_BB_A0(p_hwfn->cdev);
u16 i, pq_id, pq_group;
/* a bit per Tx PQ indicating if the PQ is associated with a VF */
u32 tx_pq_vf_mask[MAX_QM_TX_QUEUES / QM_PF_QUEUE_GROUP_SIZE] = { 0 };
- u32 tx_pq_vf_mask_width = is_bb_a0 ? 32 : QM_PF_QUEUE_GROUP_SIZE;
- u32 num_tx_pq_vf_masks = MAX_QM_TX_QUEUES / tx_pq_vf_mask_width;
+ u32 num_tx_pq_vf_masks = MAX_QM_TX_QUEUES / QM_PF_QUEUE_GROUP_SIZE;
u32 pq_mem_4kb = QM_PQ_MEM_4KB(p_params->num_pf_cids);
u32 vport_pq_mem_4kb = QM_PQ_MEM_4KB(p_params->num_vf_cids);
u32 mem_addr_4kb = base_mem_addr_4kb;
@@ -371,6 +362,10 @@
bool is_vf_pq = (i >= p_params->num_pf_pqs);
struct qm_rf_pq_map tx_pq_map;
+ bool rl_valid = p_params->pq_params[i].rl_valid &&
+ (p_params->pq_params[i].vport_id <
+ MAX_QM_GLOBAL_RLS);
+
/* update first Tx PQ of VPORT/TC */
u8 vport_id_in_pf = p_params->pq_params[i].vport_id -
p_params->start_vport;
@@ -389,14 +384,18 @@
(p_params->pf_id <<
QM_WFQ_VP_PQ_PF_SHIFT));
}
+
+ if (p_params->pq_params[i].rl_valid && !rl_valid)
+ DP_NOTICE(p_hwfn,
+ "Invalid VPORT ID for rate limiter configuration");
/* fill PQ map entry */
memset(&tx_pq_map, 0, sizeof(tx_pq_map));
SET_FIELD(tx_pq_map.reg, QM_RF_PQ_MAP_PQ_VALID, 1);
- SET_FIELD(tx_pq_map.reg, QM_RF_PQ_MAP_RL_VALID,
- p_params->pq_params[i].rl_valid ? 1 : 0);
+ SET_FIELD(tx_pq_map.reg,
+ QM_RF_PQ_MAP_RL_VALID, rl_valid ? 1 : 0);
SET_FIELD(tx_pq_map.reg, QM_RF_PQ_MAP_VP_PQ_ID, first_tx_pq_id);
SET_FIELD(tx_pq_map.reg, QM_RF_PQ_MAP_RL_ID,
- p_params->pq_params[i].rl_valid ?
+ rl_valid ?
p_params->pq_params[i].vport_id : 0);
SET_FIELD(tx_pq_map.reg, QM_RF_PQ_MAP_VOQ, voq);
SET_FIELD(tx_pq_map.reg, QM_RF_PQ_MAP_WRR_WEIGHT_GROUP,
@@ -413,8 +412,9 @@
/* if PQ is associated with a VF, add indication
* to PQ VF mask
*/
- tx_pq_vf_mask[pq_id / tx_pq_vf_mask_width] |=
- (1 << (pq_id % tx_pq_vf_mask_width));
+ tx_pq_vf_mask[pq_id /
+ QM_PF_QUEUE_GROUP_SIZE] |=
+ BIT((pq_id % QM_PF_QUEUE_GROUP_SIZE));
mem_addr_4kb += vport_pq_mem_4kb;
} else {
mem_addr_4kb += pq_mem_4kb;
@@ -480,8 +480,8 @@
if (p_params->pf_id < MAX_NUM_PFS_BB)
crd_reg_offset = QM_REG_WFQPFCRD_RT_OFFSET;
else
- crd_reg_offset = QM_REG_WFQPFCRD_MSB_RT_OFFSET +
- (p_params->pf_id % MAX_NUM_PFS_BB);
+ crd_reg_offset = QM_REG_WFQPFCRD_MSB_RT_OFFSET;
+ crd_reg_offset += p_params->pf_id % MAX_NUM_PFS_BB;
inc_val = QM_WFQ_INC_VAL(p_params->pf_wfq);
if (!inc_val || inc_val > QM_WFQ_MAX_INC_VAL) {
@@ -498,11 +498,11 @@
QM_WFQ_CRD_REG_SIGN_BIT);
}
- STORE_RT_REG(p_hwfn, QM_REG_WFQPFWEIGHT_RT_OFFSET + p_params->pf_id,
- inc_val);
STORE_RT_REG(p_hwfn,
QM_REG_WFQPFUPPERBOUND_RT_OFFSET + p_params->pf_id,
QM_WFQ_UPPER_BOUND | QM_WFQ_CRD_REG_SIGN_BIT);
+ STORE_RT_REG(p_hwfn, QM_REG_WFQPFWEIGHT_RT_OFFSET + p_params->pf_id,
+ inc_val);
return 0;
}
@@ -576,6 +576,12 @@
{
u8 i, vport_id;
+ if (start_vport + num_vports >= MAX_QM_GLOBAL_RLS) {
+ DP_NOTICE(p_hwfn,
+ "Invalid VPORT ID for rate limiter configuration");
+ return -1;
+ }
+
/* go over all PF VPORTs */
for (i = 0, vport_id = start_vport; i < num_vports; i++, vport_id++) {
u32 inc_val = QM_RL_INC_VAL(vport_params[i].vport_rl);
@@ -785,6 +791,12 @@
{
u32 inc_val = QM_RL_INC_VAL(vport_rl);
+ if (vport_id >= MAX_QM_GLOBAL_RLS) {
+ DP_NOTICE(p_hwfn,
+ "Invalid VPORT ID for rate limiter configuration");
+ return -1;
+ }
+
if (inc_val > QM_RL_MAX_INC_VAL) {
DP_NOTICE(p_hwfn, "Invalid VPORT rate-limit configuration");
return -1;
@@ -940,12 +952,6 @@
eth_geneve_enable ? 1 : 0);
qed_wr(p_hwfn, p_ptt, NIG_REG_NGE_IP_ENABLE, ip_geneve_enable ? 1 : 0);
- /* comp ver */
- reg_val = (ip_geneve_enable || eth_geneve_enable) ? 1 : 0;
- qed_wr(p_hwfn, p_ptt, NIG_REG_NGE_COMP_VER, reg_val);
- qed_wr(p_hwfn, p_ptt, PBF_REG_NGE_COMP_VER, reg_val);
- qed_wr(p_hwfn, p_ptt, PRS_REG_NGE_COMP_VER, reg_val);
-
/* EDPM with geneve tunnel not supported in BB_B0 */
if (QED_IS_BB_B0(p_hwfn->cdev))
return;
@@ -955,3 +961,132 @@
qed_wr(p_hwfn, p_ptt, DORQ_REG_L2_EDPM_TUNNEL_NGE_IP_EN,
ip_geneve_enable ? 1 : 0);
}
+
+#define T_ETH_PACKET_MATCH_RFS_EVENTID 25
+#define PARSER_ETH_CONN_CM_HDR (0x0)
+#define CAM_LINE_SIZE sizeof(u32)
+#define RAM_LINE_SIZE sizeof(u64)
+#define REG_SIZE sizeof(u32)
+
+void qed_set_rfs_mode_disable(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt, u16 pf_id)
+{
+ union gft_cam_line_union camline;
+ struct gft_ram_line ramline;
+ u32 *p_ramline, i;
+
+ p_ramline = (u32 *)&ramline;
+
+ /*stop using gft logic */
+ qed_wr(p_hwfn, p_ptt, PRS_REG_SEARCH_GFT, 0);
+ qed_wr(p_hwfn, p_ptt, PRS_REG_CM_HDR_GFT, 0x0);
+ memset(&camline, 0, sizeof(union gft_cam_line_union));
+ qed_wr(p_hwfn, p_ptt, PRS_REG_GFT_CAM + CAM_LINE_SIZE * pf_id,
+ camline.cam_line_mapped.camline);
+ memset(&ramline, 0, sizeof(union gft_cam_line_union));
+
+ for (i = 0; i < RAM_LINE_SIZE / REG_SIZE; i++) {
+ u32 hw_addr = PRS_REG_GFT_PROFILE_MASK_RAM;
+
+ hw_addr += (RAM_LINE_SIZE * pf_id + i * REG_SIZE);
+
+ qed_wr(p_hwfn, p_ptt, hw_addr, *(p_ramline + i));
+ }
+}
+
+void qed_set_rfs_mode_enable(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt,
+ u16 pf_id, bool tcp, bool udp,
+ bool ipv4, bool ipv6)
+{
+ u32 rfs_cm_hdr_event_id, *p_ramline;
+ union gft_cam_line_union camline;
+ struct gft_ram_line ramline;
+ int i;
+
+ rfs_cm_hdr_event_id = qed_rd(p_hwfn, p_ptt, PRS_REG_CM_HDR_GFT);
+ p_ramline = (u32 *)&ramline;
+
+ if (!ipv6 && !ipv4)
+ DP_NOTICE(p_hwfn,
+ "set_rfs_mode_enable: must accept at least on of - ipv4 or ipv6");
+ if (!tcp && !udp)
+ DP_NOTICE(p_hwfn,
+ "set_rfs_mode_enable: must accept at least on of - udp or tcp");
+
+ rfs_cm_hdr_event_id |= T_ETH_PACKET_MATCH_RFS_EVENTID <<
+ PRS_REG_CM_HDR_GFT_EVENT_ID_SHIFT;
+ rfs_cm_hdr_event_id |= PARSER_ETH_CONN_CM_HDR <<
+ PRS_REG_CM_HDR_GFT_CM_HDR_SHIFT;
+ qed_wr(p_hwfn, p_ptt, PRS_REG_CM_HDR_GFT, rfs_cm_hdr_event_id);
+
+ /* Configure Registers for RFS mode */
+ qed_wr(p_hwfn, p_ptt, PRS_REG_SEARCH_GFT, 1);
+ qed_wr(p_hwfn, p_ptt, PRS_REG_LOAD_L2_FILTER, 0);
+ camline.cam_line_mapped.camline = 0;
+
+ /* cam line is now valid!! */
+ SET_FIELD(camline.cam_line_mapped.camline,
+ GFT_CAM_LINE_MAPPED_VALID, 1);
+
+ /* filters are per PF!! */
+ SET_FIELD(camline.cam_line_mapped.camline,
+ GFT_CAM_LINE_MAPPED_PF_ID_MASK, 1);
+ SET_FIELD(camline.cam_line_mapped.camline,
+ GFT_CAM_LINE_MAPPED_PF_ID, pf_id);
+ if (!(tcp && udp)) {
+ SET_FIELD(camline.cam_line_mapped.camline,
+ GFT_CAM_LINE_MAPPED_UPPER_PROTOCOL_TYPE_MASK, 1);
+ if (tcp)
+ SET_FIELD(camline.cam_line_mapped.camline,
+ GFT_CAM_LINE_MAPPED_UPPER_PROTOCOL_TYPE,
+ GFT_PROFILE_TCP_PROTOCOL);
+ else
+ SET_FIELD(camline.cam_line_mapped.camline,
+ GFT_CAM_LINE_MAPPED_UPPER_PROTOCOL_TYPE,
+ GFT_PROFILE_UDP_PROTOCOL);
+ }
+
+ if (!(ipv4 && ipv6)) {
+ SET_FIELD(camline.cam_line_mapped.camline,
+ GFT_CAM_LINE_MAPPED_IP_VERSION_MASK, 1);
+ if (ipv4)
+ SET_FIELD(camline.cam_line_mapped.camline,
+ GFT_CAM_LINE_MAPPED_IP_VERSION,
+ GFT_PROFILE_IPV4);
+ else
+ SET_FIELD(camline.cam_line_mapped.camline,
+ GFT_CAM_LINE_MAPPED_IP_VERSION,
+ GFT_PROFILE_IPV6);
+ }
+
+ /* write characteristics to cam */
+ qed_wr(p_hwfn, p_ptt, PRS_REG_GFT_CAM + CAM_LINE_SIZE * pf_id,
+ camline.cam_line_mapped.camline);
+ camline.cam_line_mapped.camline = qed_rd(p_hwfn, p_ptt,
+ PRS_REG_GFT_CAM +
+ CAM_LINE_SIZE * pf_id);
+
+ /* write line to RAM - compare to filter 4 tuple */
+ ramline.low32bits = 0;
+ ramline.high32bits = 0;
+ SET_FIELD(ramline.high32bits, GFT_RAM_LINE_DST_IP, 1);
+ SET_FIELD(ramline.high32bits, GFT_RAM_LINE_SRC_IP, 1);
+ SET_FIELD(ramline.low32bits, GFT_RAM_LINE_SRC_PORT, 1);
+ SET_FIELD(ramline.low32bits, GFT_RAM_LINE_DST_PORT, 1);
+
+ /* each iteration write to reg */
+ for (i = 0; i < RAM_LINE_SIZE / REG_SIZE; i++)
+ qed_wr(p_hwfn, p_ptt,
+ PRS_REG_GFT_PROFILE_MASK_RAM + RAM_LINE_SIZE * pf_id +
+ i * REG_SIZE, *(p_ramline + i));
+
+ /* set default profile so that no filter match will happen */
+ ramline.low32bits = 0xffff;
+ ramline.high32bits = 0xffff;
+
+ for (i = 0; i < RAM_LINE_SIZE / REG_SIZE; i++)
+ qed_wr(p_hwfn, p_ptt,
+ PRS_REG_GFT_PROFILE_MASK_RAM + RAM_LINE_SIZE *
+ PRS_GFT_CAM_LINES_NO_MATCH + i * REG_SIZE,
+ *(p_ramline + i));
+}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_init_ops.c b/drivers/net/ethernet/qlogic/qed/qed_init_ops.c
index 243b64e..4a2e7be 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_init_ops.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_init_ops.c
@@ -554,7 +554,7 @@
}
/* First Dword contains metadata and should be skipped */
- buf_hdr = (struct bin_buffer_hdr *)(data + sizeof(u32));
+ buf_hdr = (struct bin_buffer_hdr *)data;
offset = buf_hdr[BIN_BUF_INIT_FW_VER_INFO].offset;
fw->fw_ver_info = (struct fw_ver_info *)(data + offset);
diff --git a/drivers/net/ethernet/qlogic/qed/qed_int.c b/drivers/net/ethernet/qlogic/qed/qed_int.c
index 84310b6..0ed24d6 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_int.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_int.c
@@ -2500,8 +2500,9 @@
/* Configure pi coalescing if set */
if (p_hwfn->cdev->int_coalescing_mode == QED_COAL_MODE_ENABLE) {
+ u8 num_tc = p_hwfn->hw_info.num_hw_tc;
u8 timeset, timer_res;
- u8 num_tc = 1, i;
+ u8 i;
/* timeset = (coalesce >> timer-res), timeset is 7bit wide */
if (p_hwfn->cdev->rx_coalesce_usecs <= 0x7F)
diff --git a/drivers/net/ethernet/qlogic/qed/qed_iscsi.c b/drivers/net/ethernet/qlogic/qed/qed_iscsi.c
index 098766f..339c91d 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_iscsi.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_iscsi.c
@@ -181,6 +181,15 @@
p_params = &p_hwfn->pf_params.iscsi_pf_params;
p_queue = &p_init->q_params;
+ /* Sanity */
+ if (p_params->num_queues > p_hwfn->hw_info.feat_num[QED_ISCSI_CQ]) {
+ DP_ERR(p_hwfn,
+ "Cannot satisfy CQ amount. Queues requested %d, CQs available %d. Aborting function start\n",
+ p_params->num_queues,
+ p_hwfn->hw_info.resc_num[QED_ISCSI_CQ]);
+ return -EINVAL;
+ }
+
SET_FIELD(p_init->hdr.flags,
ISCSI_SLOW_PATH_HDR_LAYER_CODE, ISCSI_SLOW_PATH_LAYER_CODE);
p_init->hdr.op_code = ISCSI_RAMROD_CMD_ID_INIT_FUNC;
@@ -216,7 +225,7 @@
p_queue->cq_cmdq_sb_num_arr[i] = cpu_to_le16(val);
}
- p_queue->bdq_resource_id = ISCSI_BDQ_ID(p_hwfn->port_id);
+ p_queue->bdq_resource_id = (u8)RESC_START(p_hwfn, QED_BDQ);
DMA_REGPAIR_LE(p_queue->bdq_pbl_base_address[BDQ_ID_RQ],
p_params->bdq_pbl_base_addr[BDQ_ID_RQ]);
@@ -270,11 +279,10 @@
struct tcp_offload_params *p_tcp = NULL;
struct qed_spq_entry *p_ent = NULL;
struct qed_sp_init_data init_data;
- union qed_qm_pq_params pq_params;
- u16 pq0_id = 0, pq1_id = 0;
dma_addr_t r2tq_pbl_addr;
dma_addr_t xhq_pbl_addr;
dma_addr_t uhq_pbl_addr;
+ u16 physical_q;
int rc = 0;
u32 dval;
u16 wval;
@@ -297,16 +305,14 @@
p_ramrod = &p_ent->ramrod.iscsi_conn_offload;
/* Transmission PQ is the first of the PF */
- memset(&pq_params, 0, sizeof(pq_params));
- pq0_id = qed_get_qm_pq(p_hwfn, PROTOCOLID_ISCSI, &pq_params);
- p_conn->physical_q0 = cpu_to_le16(pq0_id);
- p_ramrod->iscsi.physical_q0 = cpu_to_le16(pq0_id);
+ physical_q = qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_OFLD);
+ p_conn->physical_q0 = cpu_to_le16(physical_q);
+ p_ramrod->iscsi.physical_q0 = cpu_to_le16(physical_q);
/* iSCSI Pure-ACK PQ */
- pq_params.iscsi.q_idx = 1;
- pq1_id = qed_get_qm_pq(p_hwfn, PROTOCOLID_ISCSI, &pq_params);
- p_conn->physical_q1 = cpu_to_le16(pq1_id);
- p_ramrod->iscsi.physical_q1 = cpu_to_le16(pq1_id);
+ physical_q = qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_ACK);
+ p_conn->physical_q1 = cpu_to_le16(physical_q);
+ p_ramrod->iscsi.physical_q1 = cpu_to_le16(physical_q);
p_ramrod->hdr.op_code = ISCSI_RAMROD_CMD_ID_OFFLOAD_CONN;
SET_FIELD(p_ramrod->hdr.flags, ISCSI_SLOW_PATH_HDR_LAYER_CODE,
@@ -593,21 +599,31 @@
static void __iomem *qed_iscsi_get_primary_bdq_prod(struct qed_hwfn *p_hwfn,
u8 bdq_id)
{
- u8 bdq_function_id = ISCSI_BDQ_ID(p_hwfn->port_id);
-
- return (u8 __iomem *)p_hwfn->regview + GTT_BAR0_MAP_REG_MSDM_RAM +
- MSTORM_SCSI_BDQ_EXT_PROD_OFFSET(bdq_function_id,
- bdq_id);
+ if (RESC_NUM(p_hwfn, QED_BDQ)) {
+ return (u8 __iomem *)p_hwfn->regview +
+ GTT_BAR0_MAP_REG_MSDM_RAM +
+ MSTORM_SCSI_BDQ_EXT_PROD_OFFSET(RESC_START(p_hwfn,
+ QED_BDQ),
+ bdq_id);
+ } else {
+ DP_NOTICE(p_hwfn, "BDQ is not allocated!\n");
+ return NULL;
+ }
}
static void __iomem *qed_iscsi_get_secondary_bdq_prod(struct qed_hwfn *p_hwfn,
u8 bdq_id)
{
- u8 bdq_function_id = ISCSI_BDQ_ID(p_hwfn->port_id);
-
- return (u8 __iomem *)p_hwfn->regview + GTT_BAR0_MAP_REG_TSDM_RAM +
- TSTORM_SCSI_BDQ_EXT_PROD_OFFSET(bdq_function_id,
- bdq_id);
+ if (RESC_NUM(p_hwfn, QED_BDQ)) {
+ return (u8 __iomem *)p_hwfn->regview +
+ GTT_BAR0_MAP_REG_TSDM_RAM +
+ TSTORM_SCSI_BDQ_EXT_PROD_OFFSET(RESC_START(p_hwfn,
+ QED_BDQ),
+ bdq_id);
+ } else {
+ DP_NOTICE(p_hwfn, "BDQ is not allocated!\n");
+ return NULL;
+ }
}
static int qed_iscsi_setup_connection(struct qed_hwfn *p_hwfn,
@@ -857,6 +873,8 @@
HILO_64_REGPAIR(tstats.iscsi_rx_bytes_cnt);
p_stats->iscsi_rx_packet_cnt =
HILO_64_REGPAIR(tstats.iscsi_rx_packet_cnt);
+ p_stats->iscsi_rx_new_ooo_isle_events_cnt =
+ HILO_64_REGPAIR(tstats.iscsi_rx_new_ooo_isle_events_cnt);
p_stats->iscsi_cmdq_threshold_cnt =
le32_to_cpu(tstats.iscsi_cmdq_threshold_cnt);
p_stats->iscsi_rq_threshold_cnt =
@@ -1003,6 +1021,8 @@
info->secondary_bdq_rq_addr =
qed_iscsi_get_secondary_bdq_prod(hwfn, BDQ_ID_RQ);
+ info->num_cqs = FEAT_NUM(hwfn, QED_ISCSI_CQ);
+
return rc;
}
@@ -1304,6 +1324,26 @@
return qed_iscsi_get_stats(QED_LEADING_HWFN(cdev), stats);
}
+void qed_get_protocol_stats_iscsi(struct qed_dev *cdev,
+ struct qed_mcp_iscsi_stats *stats)
+{
+ struct qed_iscsi_stats proto_stats;
+
+ /* Retrieve FW statistics */
+ memset(&proto_stats, 0, sizeof(proto_stats));
+ if (qed_iscsi_stats(cdev, &proto_stats)) {
+ DP_VERBOSE(cdev, QED_MSG_STORAGE,
+ "Failed to collect ISCSI statistics\n");
+ return;
+ }
+
+ /* Translate FW statistics into struct */
+ stats->rx_pdus = proto_stats.iscsi_rx_total_pdu_cnt;
+ stats->tx_pdus = proto_stats.iscsi_tx_total_pdu_cnt;
+ stats->rx_bytes = proto_stats.iscsi_rx_bytes_cnt;
+ stats->tx_bytes = proto_stats.iscsi_tx_bytes_cnt;
+}
+
static const struct qed_iscsi_ops qed_iscsi_ops_pass = {
.common = &qed_common_ops_pass,
.ll2 = &qed_ll2_ops_pass,
diff --git a/drivers/net/ethernet/qlogic/qed/qed_iscsi.h b/drivers/net/ethernet/qlogic/qed/qed_iscsi.h
index 20c187f..ae98f77 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_iscsi.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_iscsi.h
@@ -64,13 +64,25 @@
void qed_iscsi_free(struct qed_hwfn *p_hwfn,
struct qed_iscsi_info *p_iscsi_info);
+
+/**
+ * @brief - Fills provided statistics struct with statistics.
+ *
+ * @param cdev
+ * @param stats - points to struct that will be filled with statistics.
+ */
+void qed_get_protocol_stats_iscsi(struct qed_dev *cdev,
+ struct qed_mcp_iscsi_stats *stats);
#else /* IS_ENABLED(CONFIG_QED_ISCSI) */
static inline struct qed_iscsi_info *qed_iscsi_alloc(
struct qed_hwfn *p_hwfn) { return NULL; }
static inline void qed_iscsi_setup(struct qed_hwfn *p_hwfn,
struct qed_iscsi_info *p_iscsi_info) {}
static inline void qed_iscsi_free(struct qed_hwfn *p_hwfn,
- struct qed_iscsi_info *p_iscsi_info) {}
+ struct qed_iscsi_info *p_iscsi_info) {}
+static inline void
+qed_get_protocol_stats_iscsi(struct qed_dev *cdev,
+ struct qed_mcp_iscsi_stats *stats) {}
#endif /* IS_ENABLED(CONFIG_QED_ISCSI) */
#endif
diff --git a/drivers/net/ethernet/qlogic/qed/qed_l2.c b/drivers/net/ethernet/qlogic/qed/qed_l2.c
index df932be..746fed4 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_l2.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_l2.c
@@ -938,15 +938,12 @@
dma_addr_t pbl_addr,
u16 pbl_size, void __iomem **pp_doorbell)
{
- union qed_qm_pq_params pq_params;
int rc;
- memset(&pq_params, 0, sizeof(pq_params));
rc = qed_eth_txq_start_ramrod(p_hwfn, p_cid,
pbl_addr, pbl_size,
- qed_get_qm_pq(p_hwfn, PROTOCOLID_ETH,
- &pq_params));
+ qed_get_cm_pq_idx_mcos(p_hwfn, tc));
if (rc)
return rc;
@@ -1470,13 +1467,20 @@
memset(&pstats, 0, sizeof(pstats));
qed_memcpy_from(p_hwfn, p_ptt, &pstats, pstats_addr, pstats_len);
- p_stats->tx_ucast_bytes += HILO_64_REGPAIR(pstats.sent_ucast_bytes);
- p_stats->tx_mcast_bytes += HILO_64_REGPAIR(pstats.sent_mcast_bytes);
- p_stats->tx_bcast_bytes += HILO_64_REGPAIR(pstats.sent_bcast_bytes);
- p_stats->tx_ucast_pkts += HILO_64_REGPAIR(pstats.sent_ucast_pkts);
- p_stats->tx_mcast_pkts += HILO_64_REGPAIR(pstats.sent_mcast_pkts);
- p_stats->tx_bcast_pkts += HILO_64_REGPAIR(pstats.sent_bcast_pkts);
- p_stats->tx_err_drop_pkts += HILO_64_REGPAIR(pstats.error_drop_pkts);
+ p_stats->common.tx_ucast_bytes +=
+ HILO_64_REGPAIR(pstats.sent_ucast_bytes);
+ p_stats->common.tx_mcast_bytes +=
+ HILO_64_REGPAIR(pstats.sent_mcast_bytes);
+ p_stats->common.tx_bcast_bytes +=
+ HILO_64_REGPAIR(pstats.sent_bcast_bytes);
+ p_stats->common.tx_ucast_pkts +=
+ HILO_64_REGPAIR(pstats.sent_ucast_pkts);
+ p_stats->common.tx_mcast_pkts +=
+ HILO_64_REGPAIR(pstats.sent_mcast_pkts);
+ p_stats->common.tx_bcast_pkts +=
+ HILO_64_REGPAIR(pstats.sent_bcast_pkts);
+ p_stats->common.tx_err_drop_pkts +=
+ HILO_64_REGPAIR(pstats.error_drop_pkts);
}
static void __qed_get_vport_tstats(struct qed_hwfn *p_hwfn,
@@ -1502,10 +1506,10 @@
memset(&tstats, 0, sizeof(tstats));
qed_memcpy_from(p_hwfn, p_ptt, &tstats, tstats_addr, tstats_len);
- p_stats->mftag_filter_discards +=
- HILO_64_REGPAIR(tstats.mftag_filter_discard);
- p_stats->mac_filter_discards +=
- HILO_64_REGPAIR(tstats.eth_mac_filter_discard);
+ p_stats->common.mftag_filter_discards +=
+ HILO_64_REGPAIR(tstats.mftag_filter_discard);
+ p_stats->common.mac_filter_discards +=
+ HILO_64_REGPAIR(tstats.eth_mac_filter_discard);
}
static void __qed_get_vport_ustats_addrlen(struct qed_hwfn *p_hwfn,
@@ -1539,12 +1543,15 @@
memset(&ustats, 0, sizeof(ustats));
qed_memcpy_from(p_hwfn, p_ptt, &ustats, ustats_addr, ustats_len);
- p_stats->rx_ucast_bytes += HILO_64_REGPAIR(ustats.rcv_ucast_bytes);
- p_stats->rx_mcast_bytes += HILO_64_REGPAIR(ustats.rcv_mcast_bytes);
- p_stats->rx_bcast_bytes += HILO_64_REGPAIR(ustats.rcv_bcast_bytes);
- p_stats->rx_ucast_pkts += HILO_64_REGPAIR(ustats.rcv_ucast_pkts);
- p_stats->rx_mcast_pkts += HILO_64_REGPAIR(ustats.rcv_mcast_pkts);
- p_stats->rx_bcast_pkts += HILO_64_REGPAIR(ustats.rcv_bcast_pkts);
+ p_stats->common.rx_ucast_bytes +=
+ HILO_64_REGPAIR(ustats.rcv_ucast_bytes);
+ p_stats->common.rx_mcast_bytes +=
+ HILO_64_REGPAIR(ustats.rcv_mcast_bytes);
+ p_stats->common.rx_bcast_bytes +=
+ HILO_64_REGPAIR(ustats.rcv_bcast_bytes);
+ p_stats->common.rx_ucast_pkts += HILO_64_REGPAIR(ustats.rcv_ucast_pkts);
+ p_stats->common.rx_mcast_pkts += HILO_64_REGPAIR(ustats.rcv_mcast_pkts);
+ p_stats->common.rx_bcast_pkts += HILO_64_REGPAIR(ustats.rcv_bcast_pkts);
}
static void __qed_get_vport_mstats_addrlen(struct qed_hwfn *p_hwfn,
@@ -1578,23 +1585,26 @@
memset(&mstats, 0, sizeof(mstats));
qed_memcpy_from(p_hwfn, p_ptt, &mstats, mstats_addr, mstats_len);
- p_stats->no_buff_discards += HILO_64_REGPAIR(mstats.no_buff_discard);
- p_stats->packet_too_big_discard +=
- HILO_64_REGPAIR(mstats.packet_too_big_discard);
- p_stats->ttl0_discard += HILO_64_REGPAIR(mstats.ttl0_discard);
- p_stats->tpa_coalesced_pkts +=
- HILO_64_REGPAIR(mstats.tpa_coalesced_pkts);
- p_stats->tpa_coalesced_events +=
- HILO_64_REGPAIR(mstats.tpa_coalesced_events);
- p_stats->tpa_aborts_num += HILO_64_REGPAIR(mstats.tpa_aborts_num);
- p_stats->tpa_coalesced_bytes +=
- HILO_64_REGPAIR(mstats.tpa_coalesced_bytes);
+ p_stats->common.no_buff_discards +=
+ HILO_64_REGPAIR(mstats.no_buff_discard);
+ p_stats->common.packet_too_big_discard +=
+ HILO_64_REGPAIR(mstats.packet_too_big_discard);
+ p_stats->common.ttl0_discard += HILO_64_REGPAIR(mstats.ttl0_discard);
+ p_stats->common.tpa_coalesced_pkts +=
+ HILO_64_REGPAIR(mstats.tpa_coalesced_pkts);
+ p_stats->common.tpa_coalesced_events +=
+ HILO_64_REGPAIR(mstats.tpa_coalesced_events);
+ p_stats->common.tpa_aborts_num +=
+ HILO_64_REGPAIR(mstats.tpa_aborts_num);
+ p_stats->common.tpa_coalesced_bytes +=
+ HILO_64_REGPAIR(mstats.tpa_coalesced_bytes);
}
static void __qed_get_vport_port_stats(struct qed_hwfn *p_hwfn,
struct qed_ptt *p_ptt,
struct qed_eth_stats *p_stats)
{
+ struct qed_eth_stats_common *p_common = &p_stats->common;
struct port_stats port_stats;
int j;
@@ -1605,54 +1615,75 @@
offsetof(struct public_port, stats),
sizeof(port_stats));
- p_stats->rx_64_byte_packets += port_stats.eth.r64;
- p_stats->rx_65_to_127_byte_packets += port_stats.eth.r127;
- p_stats->rx_128_to_255_byte_packets += port_stats.eth.r255;
- p_stats->rx_256_to_511_byte_packets += port_stats.eth.r511;
- p_stats->rx_512_to_1023_byte_packets += port_stats.eth.r1023;
- p_stats->rx_1024_to_1518_byte_packets += port_stats.eth.r1518;
- p_stats->rx_1519_to_1522_byte_packets += port_stats.eth.r1522;
- p_stats->rx_1519_to_2047_byte_packets += port_stats.eth.r2047;
- p_stats->rx_2048_to_4095_byte_packets += port_stats.eth.r4095;
- p_stats->rx_4096_to_9216_byte_packets += port_stats.eth.r9216;
- p_stats->rx_9217_to_16383_byte_packets += port_stats.eth.r16383;
- p_stats->rx_crc_errors += port_stats.eth.rfcs;
- p_stats->rx_mac_crtl_frames += port_stats.eth.rxcf;
- p_stats->rx_pause_frames += port_stats.eth.rxpf;
- p_stats->rx_pfc_frames += port_stats.eth.rxpp;
- p_stats->rx_align_errors += port_stats.eth.raln;
- p_stats->rx_carrier_errors += port_stats.eth.rfcr;
- p_stats->rx_oversize_packets += port_stats.eth.rovr;
- p_stats->rx_jabbers += port_stats.eth.rjbr;
- p_stats->rx_undersize_packets += port_stats.eth.rund;
- p_stats->rx_fragments += port_stats.eth.rfrg;
- p_stats->tx_64_byte_packets += port_stats.eth.t64;
- p_stats->tx_65_to_127_byte_packets += port_stats.eth.t127;
- p_stats->tx_128_to_255_byte_packets += port_stats.eth.t255;
- p_stats->tx_256_to_511_byte_packets += port_stats.eth.t511;
- p_stats->tx_512_to_1023_byte_packets += port_stats.eth.t1023;
- p_stats->tx_1024_to_1518_byte_packets += port_stats.eth.t1518;
- p_stats->tx_1519_to_2047_byte_packets += port_stats.eth.t2047;
- p_stats->tx_2048_to_4095_byte_packets += port_stats.eth.t4095;
- p_stats->tx_4096_to_9216_byte_packets += port_stats.eth.t9216;
- p_stats->tx_9217_to_16383_byte_packets += port_stats.eth.t16383;
- p_stats->tx_pause_frames += port_stats.eth.txpf;
- p_stats->tx_pfc_frames += port_stats.eth.txpp;
- p_stats->tx_lpi_entry_count += port_stats.eth.tlpiec;
- p_stats->tx_total_collisions += port_stats.eth.tncl;
- p_stats->rx_mac_bytes += port_stats.eth.rbyte;
- p_stats->rx_mac_uc_packets += port_stats.eth.rxuca;
- p_stats->rx_mac_mc_packets += port_stats.eth.rxmca;
- p_stats->rx_mac_bc_packets += port_stats.eth.rxbca;
- p_stats->rx_mac_frames_ok += port_stats.eth.rxpok;
- p_stats->tx_mac_bytes += port_stats.eth.tbyte;
- p_stats->tx_mac_uc_packets += port_stats.eth.txuca;
- p_stats->tx_mac_mc_packets += port_stats.eth.txmca;
- p_stats->tx_mac_bc_packets += port_stats.eth.txbca;
- p_stats->tx_mac_ctrl_frames += port_stats.eth.txcf;
+ p_common->rx_64_byte_packets += port_stats.eth.r64;
+ p_common->rx_65_to_127_byte_packets += port_stats.eth.r127;
+ p_common->rx_128_to_255_byte_packets += port_stats.eth.r255;
+ p_common->rx_256_to_511_byte_packets += port_stats.eth.r511;
+ p_common->rx_512_to_1023_byte_packets += port_stats.eth.r1023;
+ p_common->rx_1024_to_1518_byte_packets += port_stats.eth.r1518;
+ p_common->rx_crc_errors += port_stats.eth.rfcs;
+ p_common->rx_mac_crtl_frames += port_stats.eth.rxcf;
+ p_common->rx_pause_frames += port_stats.eth.rxpf;
+ p_common->rx_pfc_frames += port_stats.eth.rxpp;
+ p_common->rx_align_errors += port_stats.eth.raln;
+ p_common->rx_carrier_errors += port_stats.eth.rfcr;
+ p_common->rx_oversize_packets += port_stats.eth.rovr;
+ p_common->rx_jabbers += port_stats.eth.rjbr;
+ p_common->rx_undersize_packets += port_stats.eth.rund;
+ p_common->rx_fragments += port_stats.eth.rfrg;
+ p_common->tx_64_byte_packets += port_stats.eth.t64;
+ p_common->tx_65_to_127_byte_packets += port_stats.eth.t127;
+ p_common->tx_128_to_255_byte_packets += port_stats.eth.t255;
+ p_common->tx_256_to_511_byte_packets += port_stats.eth.t511;
+ p_common->tx_512_to_1023_byte_packets += port_stats.eth.t1023;
+ p_common->tx_1024_to_1518_byte_packets += port_stats.eth.t1518;
+ p_common->tx_pause_frames += port_stats.eth.txpf;
+ p_common->tx_pfc_frames += port_stats.eth.txpp;
+ p_common->rx_mac_bytes += port_stats.eth.rbyte;
+ p_common->rx_mac_uc_packets += port_stats.eth.rxuca;
+ p_common->rx_mac_mc_packets += port_stats.eth.rxmca;
+ p_common->rx_mac_bc_packets += port_stats.eth.rxbca;
+ p_common->rx_mac_frames_ok += port_stats.eth.rxpok;
+ p_common->tx_mac_bytes += port_stats.eth.tbyte;
+ p_common->tx_mac_uc_packets += port_stats.eth.txuca;
+ p_common->tx_mac_mc_packets += port_stats.eth.txmca;
+ p_common->tx_mac_bc_packets += port_stats.eth.txbca;
+ p_common->tx_mac_ctrl_frames += port_stats.eth.txcf;
for (j = 0; j < 8; j++) {
- p_stats->brb_truncates += port_stats.brb.brb_truncate[j];
- p_stats->brb_discards += port_stats.brb.brb_discard[j];
+ p_common->brb_truncates += port_stats.brb.brb_truncate[j];
+ p_common->brb_discards += port_stats.brb.brb_discard[j];
+ }
+
+ if (QED_IS_BB(p_hwfn->cdev)) {
+ struct qed_eth_stats_bb *p_bb = &p_stats->bb;
+
+ p_bb->rx_1519_to_1522_byte_packets +=
+ port_stats.eth.u0.bb0.r1522;
+ p_bb->rx_1519_to_2047_byte_packets +=
+ port_stats.eth.u0.bb0.r2047;
+ p_bb->rx_2048_to_4095_byte_packets +=
+ port_stats.eth.u0.bb0.r4095;
+ p_bb->rx_4096_to_9216_byte_packets +=
+ port_stats.eth.u0.bb0.r9216;
+ p_bb->rx_9217_to_16383_byte_packets +=
+ port_stats.eth.u0.bb0.r16383;
+ p_bb->tx_1519_to_2047_byte_packets +=
+ port_stats.eth.u1.bb1.t2047;
+ p_bb->tx_2048_to_4095_byte_packets +=
+ port_stats.eth.u1.bb1.t4095;
+ p_bb->tx_4096_to_9216_byte_packets +=
+ port_stats.eth.u1.bb1.t9216;
+ p_bb->tx_9217_to_16383_byte_packets +=
+ port_stats.eth.u1.bb1.t16383;
+ p_bb->tx_lpi_entry_count += port_stats.eth.u2.bb2.tlpiec;
+ p_bb->tx_total_collisions += port_stats.eth.u2.bb2.tncl;
+ } else {
+ struct qed_eth_stats_ah *p_ah = &p_stats->ah;
+
+ p_ah->rx_1519_to_max_byte_packets +=
+ port_stats.eth.u0.ah0.r1519_to_max;
+ p_ah->tx_1519_to_max_byte_packets =
+ port_stats.eth.u1.ah1.t1519_to_max;
}
}
@@ -1768,6 +1799,84 @@
_qed_get_vport_stats(cdev, cdev->reset_stats);
}
+static void
+qed_arfs_mode_configure(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt,
+ struct qed_arfs_config_params *p_cfg_params)
+{
+ if (p_cfg_params->arfs_enable) {
+ qed_set_rfs_mode_enable(p_hwfn, p_ptt, p_hwfn->rel_pf_id,
+ p_cfg_params->tcp, p_cfg_params->udp,
+ p_cfg_params->ipv4, p_cfg_params->ipv6);
+ DP_VERBOSE(p_hwfn, QED_MSG_SP,
+ "tcp = %s, udp = %s, ipv4 = %s, ipv6 =%s\n",
+ p_cfg_params->tcp ? "Enable" : "Disable",
+ p_cfg_params->udp ? "Enable" : "Disable",
+ p_cfg_params->ipv4 ? "Enable" : "Disable",
+ p_cfg_params->ipv6 ? "Enable" : "Disable");
+ } else {
+ qed_set_rfs_mode_disable(p_hwfn, p_ptt, p_hwfn->rel_pf_id);
+ }
+
+ DP_VERBOSE(p_hwfn, QED_MSG_SP, "Configured ARFS mode : %s\n",
+ p_cfg_params->arfs_enable ? "Enable" : "Disable");
+}
+
+static int
+qed_configure_rfs_ntuple_filter(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt,
+ struct qed_spq_comp_cb *p_cb,
+ dma_addr_t p_addr, u16 length, u16 qid,
+ u8 vport_id, bool b_is_add)
+{
+ struct rx_update_gft_filter_data *p_ramrod = NULL;
+ struct qed_spq_entry *p_ent = NULL;
+ struct qed_sp_init_data init_data;
+ u16 abs_rx_q_id = 0;
+ u8 abs_vport_id = 0;
+ int rc = -EINVAL;
+
+ rc = qed_fw_vport(p_hwfn, vport_id, &abs_vport_id);
+ if (rc)
+ return rc;
+
+ rc = qed_fw_l2_queue(p_hwfn, qid, &abs_rx_q_id);
+ if (rc)
+ return rc;
+
+ /* Get SPQ entry */
+ memset(&init_data, 0, sizeof(init_data));
+ init_data.cid = qed_spq_get_cid(p_hwfn);
+
+ init_data.opaque_fid = p_hwfn->hw_info.opaque_fid;
+
+ if (p_cb) {
+ init_data.comp_mode = QED_SPQ_MODE_CB;
+ init_data.p_comp_data = p_cb;
+ } else {
+ init_data.comp_mode = QED_SPQ_MODE_EBLOCK;
+ }
+
+ rc = qed_sp_init_request(p_hwfn, &p_ent,
+ ETH_RAMROD_GFT_UPDATE_FILTER,
+ PROTOCOLID_ETH, &init_data);
+ if (rc)
+ return rc;
+
+ p_ramrod = &p_ent->ramrod.rx_update_gft;
+ DMA_REGPAIR_LE(p_ramrod->pkt_hdr_addr, p_addr);
+ p_ramrod->pkt_hdr_length = cpu_to_le16(length);
+ p_ramrod->rx_qid_or_action_icid = cpu_to_le16(abs_rx_q_id);
+ p_ramrod->vport_id = abs_vport_id;
+ p_ramrod->filter_type = RFS_FILTER_TYPE;
+ p_ramrod->filter_action = b_is_add ? GFT_ADD_FILTER : GFT_DELETE_FILTER;
+
+ DP_VERBOSE(p_hwfn, QED_MSG_SP,
+ "V[%0x], Q[%04x] - %s filter from 0x%llx [length %04xb]\n",
+ abs_vport_id, abs_rx_q_id,
+ b_is_add ? "Adding" : "Removing", (u64)p_addr, length);
+
+ return qed_spq_post(p_hwfn, p_ent, NULL);
+}
+
static int qed_fill_eth_dev_info(struct qed_dev *cdev,
struct qed_dev_eth_info *info)
{
@@ -1898,7 +2007,11 @@
return rc;
}
- qed_hw_start_fastpath(p_hwfn);
+ rc = qed_hw_start_fastpath(p_hwfn);
+ if (rc) {
+ DP_ERR(cdev, "Failed to start VPORT fastpath\n");
+ return rc;
+ }
DP_VERBOSE(cdev, (QED_MSG_SPQ | NETIF_MSG_IFUP),
"Started V-PORT %d with MTU %d\n",
@@ -2141,7 +2254,13 @@
#define QED_HW_STOP_RETRY_LIMIT (10)
static int qed_fastpath_stop(struct qed_dev *cdev)
{
- qed_hw_stop_fastpath(cdev);
+ int rc;
+
+ rc = qed_hw_stop_fastpath(cdev);
+ if (rc) {
+ DP_ERR(cdev, "Failed to stop Fastpath\n");
+ return rc;
+ }
return 0;
}
@@ -2166,31 +2285,46 @@
static int qed_tunn_configure(struct qed_dev *cdev,
struct qed_tunn_params *tunn_params)
{
- struct qed_tunn_update_params tunn_info;
+ struct qed_tunnel_info tunn_info;
int i, rc;
- if (IS_VF(cdev))
- return 0;
-
memset(&tunn_info, 0, sizeof(tunn_info));
- if (tunn_params->update_vxlan_port == 1) {
- tunn_info.update_vxlan_udp_port = 1;
- tunn_info.vxlan_udp_port = tunn_params->vxlan_port;
+ if (tunn_params->update_vxlan_port) {
+ tunn_info.vxlan_port.b_update_port = true;
+ tunn_info.vxlan_port.port = tunn_params->vxlan_port;
}
- if (tunn_params->update_geneve_port == 1) {
- tunn_info.update_geneve_udp_port = 1;
- tunn_info.geneve_udp_port = tunn_params->geneve_port;
+ if (tunn_params->update_geneve_port) {
+ tunn_info.geneve_port.b_update_port = true;
+ tunn_info.geneve_port.port = tunn_params->geneve_port;
}
for_each_hwfn(cdev, i) {
struct qed_hwfn *hwfn = &cdev->hwfns[i];
+ struct qed_tunnel_info *tun;
+
+ tun = &hwfn->cdev->tunnel;
rc = qed_sp_pf_update_tunn_cfg(hwfn, &tunn_info,
QED_SPQ_MODE_EBLOCK, NULL);
-
if (rc)
return rc;
+
+ if (IS_PF_SRIOV(hwfn)) {
+ u16 vxlan_port, geneve_port;
+ int j;
+
+ vxlan_port = tun->vxlan_port.port;
+ geneve_port = tun->geneve_port.port;
+
+ qed_for_each_vf(hwfn, j) {
+ qed_iov_bulletin_set_udp_ports(hwfn, j,
+ vxlan_port,
+ geneve_port);
+ }
+
+ qed_schedule_iov(hwfn, QED_IOV_WQ_BULLETIN_UPDATE_FLAG);
+ }
}
return 0;
@@ -2315,6 +2449,59 @@
}
}
+static int qed_configure_arfs_searcher(struct qed_dev *cdev, bool en_searcher)
+{
+ struct qed_hwfn *p_hwfn = QED_LEADING_HWFN(cdev);
+ struct qed_arfs_config_params arfs_config_params;
+
+ memset(&arfs_config_params, 0, sizeof(arfs_config_params));
+ arfs_config_params.tcp = true;
+ arfs_config_params.udp = true;
+ arfs_config_params.ipv4 = true;
+ arfs_config_params.ipv6 = true;
+ arfs_config_params.arfs_enable = en_searcher;
+
+ qed_arfs_mode_configure(p_hwfn, p_hwfn->p_arfs_ptt,
+ &arfs_config_params);
+ return 0;
+}
+
+static void
+qed_arfs_sp_response_handler(struct qed_hwfn *p_hwfn,
+ void *cookie, union event_ring_data *data,
+ u8 fw_return_code)
+{
+ struct qed_common_cb_ops *op = p_hwfn->cdev->protocol_ops.common;
+ void *dev = p_hwfn->cdev->ops_cookie;
+
+ op->arfs_filter_op(dev, cookie, fw_return_code);
+}
+
+static int qed_ntuple_arfs_filter_config(struct qed_dev *cdev, void *cookie,
+ dma_addr_t mapping, u16 length,
+ u16 vport_id, u16 rx_queue_id,
+ bool add_filter)
+{
+ struct qed_hwfn *p_hwfn = QED_LEADING_HWFN(cdev);
+ struct qed_spq_comp_cb cb;
+ int rc = -EINVAL;
+
+ cb.function = qed_arfs_sp_response_handler;
+ cb.cookie = cookie;
+
+ rc = qed_configure_rfs_ntuple_filter(p_hwfn, p_hwfn->p_arfs_ptt,
+ &cb, mapping, length, rx_queue_id,
+ vport_id, add_filter);
+ if (rc)
+ DP_NOTICE(p_hwfn,
+ "Failed to issue a-RFS filter configuration\n");
+ else
+ DP_VERBOSE(p_hwfn, NETIF_MSG_DRV,
+ "Successfully issued a-RFS filter configuration\n");
+
+ return rc;
+}
+
static int qed_fp_cqe_completion(struct qed_dev *dev,
u8 rss_id, struct eth_slow_path_rx_cqe *cqe)
{
@@ -2356,6 +2543,8 @@
.eth_cqe_completion = &qed_fp_cqe_completion,
.get_vport_stats = &qed_get_vport_stats,
.tunn_config = &qed_tunn_configure,
+ .ntuple_filter_config = &qed_ntuple_arfs_filter_config,
+ .configure_arfs_searcher = &qed_configure_arfs_searcher,
};
const struct qed_eth_ops *qed_get_eth_ops(void)
diff --git a/drivers/net/ethernet/qlogic/qed/qed_l2.h b/drivers/net/ethernet/qlogic/qed/qed_l2.h
index e763abd..6f44229 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_l2.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_l2.h
@@ -185,6 +185,14 @@
#define QED_ACCEPT_BCAST 0x20
};
+struct qed_arfs_config_params {
+ bool tcp;
+ bool udp;
+ bool ipv4;
+ bool ipv6;
+ bool arfs_enable;
+};
+
struct qed_sp_vport_update_params {
u16 opaque_fid;
u8 vport_id;
diff --git a/drivers/net/ethernet/qlogic/qed/qed_ll2.c b/drivers/net/ethernet/qlogic/qed/qed_ll2.c
index 0d3cef4..09c8641 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_ll2.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_ll2.c
@@ -597,7 +597,7 @@
u8 bd_flags = 0;
if (GET_FIELD(parse_flags, PARSING_AND_ERR_FLAGS_TAG8021QEXIST))
- SET_FIELD(bd_flags, CORE_TX_BD_FLAGS_VLAN_INSERTION, 1);
+ SET_FIELD(bd_flags, CORE_TX_BD_DATA_VLAN_INSERTION, 1);
return bd_flags;
}
@@ -758,8 +758,8 @@
p_buffer->placement_offset;
parse_flags = p_buffer->parse_flags;
bd_flags = qed_ll2_convert_rx_parse_to_tx_flags(parse_flags);
- SET_FIELD(bd_flags, CORE_TX_BD_FLAGS_FORCE_VLAN_MODE, 1);
- SET_FIELD(bd_flags, CORE_TX_BD_FLAGS_L4_PROTOCOL, 1);
+ SET_FIELD(bd_flags, CORE_TX_BD_DATA_FORCE_VLAN_MODE, 1);
+ SET_FIELD(bd_flags, CORE_TX_BD_DATA_L4_PROTOCOL, 1);
rc = qed_ll2_prepare_tx_packet(p_hwfn, p_ll2_conn->my_id, 1,
p_buffer->vlan, bd_flags,
@@ -1090,7 +1090,6 @@
struct core_tx_start_ramrod_data *p_ramrod = NULL;
struct qed_spq_entry *p_ent = NULL;
struct qed_sp_init_data init_data;
- union qed_qm_pq_params pq_params;
u16 pq_id = 0, pbl_size;
int rc = -EINVAL;
@@ -1127,9 +1126,18 @@
pbl_size = qed_chain_get_page_cnt(&p_tx->txq_chain);
p_ramrod->pbl_size = cpu_to_le16(pbl_size);
- memset(&pq_params, 0, sizeof(pq_params));
- pq_params.core.tc = p_ll2_conn->conn.tx_tc;
- pq_id = qed_get_qm_pq(p_hwfn, PROTOCOLID_CORE, &pq_params);
+ switch (p_ll2_conn->conn.tx_tc) {
+ case LB_TC:
+ pq_id = qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_LB);
+ break;
+ case OOO_LB_TC:
+ pq_id = qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_OOO);
+ break;
+ default:
+ pq_id = qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_OFLD);
+ break;
+ }
+
p_ramrod->qm_pq_id = cpu_to_le16(pq_id);
switch (conn_type) {
@@ -1400,13 +1408,21 @@
struct qed_ll2_info *p_ll2_conn;
struct qed_ll2_rx_queue *p_rx;
struct qed_ll2_tx_queue *p_tx;
+ struct qed_ptt *p_ptt;
int rc = -EINVAL;
u32 i, capacity;
u8 qid;
+ p_ptt = qed_ptt_acquire(p_hwfn);
+ if (!p_ptt)
+ return -EAGAIN;
+
p_ll2_conn = qed_ll2_handle_sanity_lock(p_hwfn, connection_handle);
- if (!p_ll2_conn)
- return -EINVAL;
+ if (!p_ll2_conn) {
+ rc = -EINVAL;
+ goto out;
+ }
+
p_rx = &p_ll2_conn->rx_queue;
p_tx = &p_ll2_conn->tx_queue;
@@ -1439,7 +1455,9 @@
p_tx->cur_completing_frag_num = 0;
*p_tx->p_fw_cons = 0;
- qed_cxt_acquire_cid(p_hwfn, PROTOCOLID_CORE, &p_ll2_conn->cid);
+ rc = qed_cxt_acquire_cid(p_hwfn, PROTOCOLID_CORE, &p_ll2_conn->cid);
+ if (rc)
+ goto out;
qid = p_hwfn->hw_info.resc_start[QED_LL2_QUEUE] + connection_handle;
p_ll2_conn->queue_id = qid;
@@ -1453,26 +1471,28 @@
rc = qed_ll2_establish_connection_rx(p_hwfn, p_ll2_conn);
if (rc)
- return rc;
+ goto out;
rc = qed_sp_ll2_tx_queue_start(p_hwfn, p_ll2_conn);
if (rc)
- return rc;
+ goto out;
if (p_hwfn->hw_info.personality != QED_PCI_ETH_ROCE)
- qed_wr(p_hwfn, p_hwfn->p_main_ptt, PRS_REG_USE_LIGHT_L2, 1);
+ qed_wr(p_hwfn, p_ptt, PRS_REG_USE_LIGHT_L2, 1);
qed_ll2_establish_connection_ooo(p_hwfn, p_ll2_conn);
if (p_ll2_conn->conn.conn_type == QED_LL2_TYPE_FCOE) {
- qed_llh_add_protocol_filter(p_hwfn, p_hwfn->p_main_ptt,
+ qed_llh_add_protocol_filter(p_hwfn, p_ptt,
0x8906, 0,
QED_LLH_FILTER_ETHERTYPE);
- qed_llh_add_protocol_filter(p_hwfn, p_hwfn->p_main_ptt,
+ qed_llh_add_protocol_filter(p_hwfn, p_ptt,
0x8914, 0,
QED_LLH_FILTER_ETHERTYPE);
}
+out:
+ qed_ptt_release(p_hwfn, p_ptt);
return rc;
}
@@ -1591,33 +1611,34 @@
p_tx->cur_send_frag_num++;
}
-static void qed_ll2_prepare_tx_packet_set_bd(struct qed_hwfn *p_hwfn,
- struct qed_ll2_info *p_ll2,
- struct qed_ll2_tx_packet *p_curp,
- u8 num_of_bds,
- enum core_tx_dest tx_dest,
- u16 vlan,
- u8 bd_flags,
- u16 l4_hdr_offset_w,
- enum core_roce_flavor_type type,
- dma_addr_t first_frag,
- u16 first_frag_len)
+static void
+qed_ll2_prepare_tx_packet_set_bd(struct qed_hwfn *p_hwfn,
+ struct qed_ll2_info *p_ll2,
+ struct qed_ll2_tx_packet *p_curp,
+ u8 num_of_bds,
+ enum core_tx_dest tx_dest,
+ u16 vlan,
+ u8 bd_flags,
+ u16 l4_hdr_offset_w,
+ enum core_roce_flavor_type roce_flavor,
+ dma_addr_t first_frag,
+ u16 first_frag_len)
{
struct qed_chain *p_tx_chain = &p_ll2->tx_queue.txq_chain;
u16 prod_idx = qed_chain_get_prod_idx(p_tx_chain);
struct core_tx_bd *start_bd = NULL;
- u16 frag_idx;
+ u16 bd_data = 0, frag_idx;
start_bd = (struct core_tx_bd *)qed_chain_produce(p_tx_chain);
start_bd->nw_vlan_or_lb_echo = cpu_to_le16(vlan);
SET_FIELD(start_bd->bitfield1, CORE_TX_BD_L4_HDR_OFFSET_W,
cpu_to_le16(l4_hdr_offset_w));
SET_FIELD(start_bd->bitfield1, CORE_TX_BD_TX_DST, tx_dest);
- start_bd->bd_flags.as_bitfield = bd_flags;
- start_bd->bd_flags.as_bitfield |= CORE_TX_BD_FLAGS_START_BD_MASK <<
- CORE_TX_BD_FLAGS_START_BD_SHIFT;
- SET_FIELD(start_bd->bitfield0, CORE_TX_BD_NBDS, num_of_bds);
- SET_FIELD(start_bd->bitfield0, CORE_TX_BD_ROCE_FLAV, type);
+ bd_data |= bd_flags;
+ SET_FIELD(bd_data, CORE_TX_BD_DATA_START_BD, 0x1);
+ SET_FIELD(bd_data, CORE_TX_BD_DATA_NBDS, num_of_bds);
+ SET_FIELD(bd_data, CORE_TX_BD_DATA_ROCE_FLAV, roce_flavor);
+ start_bd->bd_data.as_bitfield = cpu_to_le16(bd_data);
DMA_REGPAIR_LE(start_bd->addr, first_frag);
start_bd->nbytes = cpu_to_le16(first_frag_len);
@@ -1642,9 +1663,8 @@
struct core_tx_bd **p_bd = &p_curp->bds_set[frag_idx].txq_bd;
*p_bd = (struct core_tx_bd *)qed_chain_produce(p_tx_chain);
- (*p_bd)->bd_flags.as_bitfield = 0;
+ (*p_bd)->bd_data.as_bitfield = 0;
(*p_bd)->bitfield1 = 0;
- (*p_bd)->bitfield0 = 0;
p_curp->bds_set[frag_idx].tx_frag = 0;
p_curp->bds_set[frag_idx].frag_len = 0;
}
@@ -1823,23 +1843,30 @@
{
struct qed_ll2_info *p_ll2_conn = NULL;
int rc = -EINVAL;
+ struct qed_ptt *p_ptt;
+
+ p_ptt = qed_ptt_acquire(p_hwfn);
+ if (!p_ptt)
+ return -EAGAIN;
p_ll2_conn = qed_ll2_handle_sanity_lock(p_hwfn, connection_handle);
- if (!p_ll2_conn)
- return -EINVAL;
+ if (!p_ll2_conn) {
+ rc = -EINVAL;
+ goto out;
+ }
/* Stop Tx & Rx of connection, if needed */
if (QED_LL2_TX_REGISTERED(p_ll2_conn)) {
rc = qed_sp_ll2_tx_queue_stop(p_hwfn, p_ll2_conn);
if (rc)
- return rc;
+ goto out;
qed_ll2_txq_flush(p_hwfn, connection_handle);
}
if (QED_LL2_RX_REGISTERED(p_ll2_conn)) {
rc = qed_sp_ll2_rx_queue_stop(p_hwfn, p_ll2_conn);
if (rc)
- return rc;
+ goto out;
qed_ll2_rxq_flush(p_hwfn, connection_handle);
}
@@ -1847,14 +1874,16 @@
qed_ooo_release_all_isles(p_hwfn, p_hwfn->p_ooo_info);
if (p_ll2_conn->conn.conn_type == QED_LL2_TYPE_FCOE) {
- qed_llh_remove_protocol_filter(p_hwfn, p_hwfn->p_main_ptt,
+ qed_llh_remove_protocol_filter(p_hwfn, p_ptt,
0x8906, 0,
QED_LLH_FILTER_ETHERTYPE);
- qed_llh_remove_protocol_filter(p_hwfn, p_hwfn->p_main_ptt,
+ qed_llh_remove_protocol_filter(p_hwfn, p_ptt,
0x8914, 0,
QED_LLH_FILTER_ETHERTYPE);
}
+out:
+ qed_ptt_release(p_hwfn, p_ptt);
return rc;
}
@@ -2241,11 +2270,11 @@
/* Request HW to calculate IP csum */
if (!((vlan_get_protocol(skb) == htons(ETH_P_IPV6)) &&
ipv6_hdr(skb)->nexthdr == NEXTHDR_IPV6))
- flags |= BIT(CORE_TX_BD_FLAGS_IP_CSUM_SHIFT);
+ flags |= BIT(CORE_TX_BD_DATA_IP_CSUM_SHIFT);
if (skb_vlan_tag_present(skb)) {
vlan = skb_vlan_tag_get(skb);
- flags |= BIT(CORE_TX_BD_FLAGS_VLAN_INSERTION_SHIFT);
+ flags |= BIT(CORE_TX_BD_DATA_VLAN_INSERTION_SHIFT);
}
rc = qed_ll2_prepare_tx_packet(QED_LEADING_HWFN(cdev),
diff --git a/drivers/net/ethernet/qlogic/qed/qed_main.c b/drivers/net/ethernet/qlogic/qed/qed_main.c
index eef30a5..59992cf 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_main.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_main.c
@@ -45,6 +45,7 @@
#include <linux/ethtool.h>
#include <linux/etherdevice.h>
#include <linux/vmalloc.h>
+#include <linux/crash_dump.h>
#include <linux/qed/qed_if.h>
#include <linux/qed/qed_ll2_if.h>
@@ -54,6 +55,8 @@
#include "qed_dev_api.h"
#include "qed_ll2.h"
#include "qed_fcoe.h"
+#include "qed_iscsi.h"
+
#include "qed_mcp.h"
#include "qed_hw.h"
#include "qed_selftest.h"
@@ -227,10 +230,25 @@
int qed_fill_dev_info(struct qed_dev *cdev,
struct qed_dev_info *dev_info)
{
+ struct qed_tunnel_info *tun = &cdev->tunnel;
struct qed_ptt *ptt;
memset(dev_info, 0, sizeof(struct qed_dev_info));
+ if (tun->vxlan.tun_cls == QED_TUNN_CLSS_MAC_VLAN &&
+ tun->vxlan.b_mode_enabled)
+ dev_info->vxlan_enable = true;
+
+ if (tun->l2_gre.b_mode_enabled && tun->ip_gre.b_mode_enabled &&
+ tun->l2_gre.tun_cls == QED_TUNN_CLSS_MAC_VLAN &&
+ tun->ip_gre.tun_cls == QED_TUNN_CLSS_MAC_VLAN)
+ dev_info->gre_enable = true;
+
+ if (tun->l2_geneve.b_mode_enabled && tun->ip_geneve.b_mode_enabled &&
+ tun->l2_geneve.tun_cls == QED_TUNN_CLSS_MAC_VLAN &&
+ tun->ip_geneve.tun_cls == QED_TUNN_CLSS_MAC_VLAN)
+ dev_info->geneve_enable = true;
+
dev_info->num_hwfns = cdev->num_hwfns;
dev_info->pci_mem_start = cdev->pci_params.mem_start;
dev_info->pci_mem_end = cdev->pci_params.mem_end;
@@ -238,6 +256,7 @@
dev_info->rdma_supported = (cdev->hwfns[0].hw_info.personality ==
QED_PCI_ETH_ROCE);
dev_info->is_mf_default = IS_MF_DEFAULT(&cdev->hwfns[0]);
+ dev_info->dev_type = cdev->type;
ether_addr_copy(dev_info->hw_mac, cdev->hwfns[0].hw_info.hw_mac_addr);
if (IS_PF(cdev)) {
@@ -588,6 +607,19 @@
return rc;
}
+void qed_slowpath_irq_sync(struct qed_hwfn *p_hwfn)
+{
+ struct qed_dev *cdev = p_hwfn->cdev;
+ u8 id = p_hwfn->my_id;
+ u32 int_mode;
+
+ int_mode = cdev->int_params.out.int_mode;
+ if (int_mode == QED_INT_MODE_MSIX)
+ synchronize_irq(cdev->int_params.msix_table[id].vector);
+ else
+ synchronize_irq(cdev->pdev->irq);
+}
+
static void qed_slowpath_irq_free(struct qed_dev *cdev)
{
int i;
@@ -630,19 +662,6 @@
return rc;
}
-static int qed_nic_reset(struct qed_dev *cdev)
-{
- int rc;
-
- rc = qed_hw_reset(cdev);
- if (rc)
- return rc;
-
- qed_resc_free(cdev);
-
- return 0;
-}
-
static int qed_nic_setup(struct qed_dev *cdev)
{
int rc, i;
@@ -743,7 +762,8 @@
cdev->int_params.fp_msix_cnt = cdev->int_params.out.num_vectors -
cdev->num_hwfns;
- if (!IS_ENABLED(CONFIG_QED_RDMA))
+ if (!IS_ENABLED(CONFIG_QED_RDMA) ||
+ QED_LEADING_HWFN(cdev)->hw_info.personality != QED_PCI_ETH_ROCE)
return 0;
for_each_hwfn(cdev, i)
@@ -875,10 +895,12 @@
params->rdma_pf_params.num_qps = QED_ROCE_QPS;
params->rdma_pf_params.min_dpis = QED_ROCE_DPIS;
/* divide by 3 the MRs to avoid MF ILT overflow */
- params->rdma_pf_params.num_mrs = RDMA_MAX_TIDS;
params->rdma_pf_params.gl_pi = QED_ROCE_PROTOCOL_INDEX;
}
+ if (cdev->num_hwfns > 1 || IS_VF(cdev))
+ params->eth_pf_params.num_arfs_filters = 0;
+
/* In case we might support RDMA, don't allow qede to be greedy
* with the L2 contexts. Allow for 64 queues [rx, tx, xdp] per hwfn.
*/
@@ -900,11 +922,15 @@
static int qed_slowpath_start(struct qed_dev *cdev,
struct qed_slowpath_params *params)
{
- struct qed_tunn_start_params tunn_info;
+ struct qed_drv_load_params drv_load_params;
+ struct qed_hw_init_params hw_init_params;
struct qed_mcp_drv_version drv_version;
+ struct qed_tunnel_info tunn_info;
const u8 *data = NULL;
struct qed_hwfn *hwfn;
+#ifdef CONFIG_RFS_ACCEL
struct qed_ptt *p_ptt;
+#endif
int rc = -EINVAL;
if (qed_iov_wq_start(cdev))
@@ -920,13 +946,18 @@
goto err;
}
- p_ptt = qed_ptt_acquire(QED_LEADING_HWFN(cdev));
- if (p_ptt) {
- QED_LEADING_HWFN(cdev)->p_ptp_ptt = p_ptt;
- } else {
- DP_NOTICE(cdev, "Failed to acquire PTT for PTP\n");
- goto err;
+#ifdef CONFIG_RFS_ACCEL
+ if (cdev->num_hwfns == 1) {
+ p_ptt = qed_ptt_acquire(QED_LEADING_HWFN(cdev));
+ if (p_ptt) {
+ QED_LEADING_HWFN(cdev)->p_arfs_ptt = p_ptt;
+ } else {
+ DP_NOTICE(cdev,
+ "Failed to acquire PTT for aRFS\n");
+ goto err;
+ }
}
+#endif
}
cdev->rx_coalesce_usecs = QED_DEFAULT_RX_USECS;
@@ -953,27 +984,47 @@
qed_dbg_pf_init(cdev);
}
- memset(&tunn_info, 0, sizeof(tunn_info));
- tunn_info.tunn_mode |= 1 << QED_MODE_VXLAN_TUNN |
- 1 << QED_MODE_L2GRE_TUNN |
- 1 << QED_MODE_IPGRE_TUNN |
- 1 << QED_MODE_L2GENEVE_TUNN |
- 1 << QED_MODE_IPGENEVE_TUNN;
-
- tunn_info.tunn_clss_vxlan = QED_TUNN_CLSS_MAC_VLAN;
- tunn_info.tunn_clss_l2gre = QED_TUNN_CLSS_MAC_VLAN;
- tunn_info.tunn_clss_ipgre = QED_TUNN_CLSS_MAC_VLAN;
-
/* Start the slowpath */
- rc = qed_hw_init(cdev, &tunn_info, true,
- cdev->int_params.out.int_mode,
- true, data);
+ memset(&hw_init_params, 0, sizeof(hw_init_params));
+ memset(&tunn_info, 0, sizeof(tunn_info));
+ tunn_info.vxlan.b_mode_enabled = true;
+ tunn_info.l2_gre.b_mode_enabled = true;
+ tunn_info.ip_gre.b_mode_enabled = true;
+ tunn_info.l2_geneve.b_mode_enabled = true;
+ tunn_info.ip_geneve.b_mode_enabled = true;
+ tunn_info.vxlan.tun_cls = QED_TUNN_CLSS_MAC_VLAN;
+ tunn_info.l2_gre.tun_cls = QED_TUNN_CLSS_MAC_VLAN;
+ tunn_info.ip_gre.tun_cls = QED_TUNN_CLSS_MAC_VLAN;
+ tunn_info.l2_geneve.tun_cls = QED_TUNN_CLSS_MAC_VLAN;
+ tunn_info.ip_geneve.tun_cls = QED_TUNN_CLSS_MAC_VLAN;
+ hw_init_params.p_tunn = &tunn_info;
+ hw_init_params.b_hw_start = true;
+ hw_init_params.int_mode = cdev->int_params.out.int_mode;
+ hw_init_params.allow_npar_tx_switch = true;
+ hw_init_params.bin_fw_data = data;
+
+ memset(&drv_load_params, 0, sizeof(drv_load_params));
+ drv_load_params.is_crash_kernel = is_kdump_kernel();
+ drv_load_params.mfw_timeout_val = QED_LOAD_REQ_LOCK_TO_DEFAULT;
+ drv_load_params.avoid_eng_reset = false;
+ drv_load_params.override_force_load = QED_OVERRIDE_FORCE_LOAD_NONE;
+ hw_init_params.p_drv_load_params = &drv_load_params;
+
+ rc = qed_hw_init(cdev, &hw_init_params);
if (rc)
goto err2;
DP_INFO(cdev,
"HW initialization and function start completed successfully\n");
+ if (IS_PF(cdev)) {
+ cdev->tunn_feature_mask = (BIT(QED_MODE_VXLAN_TUNN) |
+ BIT(QED_MODE_L2GENEVE_TUNN) |
+ BIT(QED_MODE_IPGENEVE_TUNN) |
+ BIT(QED_MODE_L2GRE_TUNN) |
+ BIT(QED_MODE_IPGRE_TUNN));
+ }
+
/* Allocate LL2 interface if needed */
if (QED_LEADING_HWFN(cdev)->using_ll2) {
rc = qed_ll2_alloc_if(cdev);
@@ -1014,9 +1065,12 @@
if (IS_PF(cdev))
release_firmware(cdev->firmware);
- if (IS_PF(cdev) && QED_LEADING_HWFN(cdev)->p_ptp_ptt)
+#ifdef CONFIG_RFS_ACCEL
+ if (IS_PF(cdev) && (cdev->num_hwfns == 1) &&
+ QED_LEADING_HWFN(cdev)->p_arfs_ptt)
qed_ptt_release(QED_LEADING_HWFN(cdev),
- QED_LEADING_HWFN(cdev)->p_ptp_ptt);
+ QED_LEADING_HWFN(cdev)->p_arfs_ptt);
+#endif
qed_iov_wq_stop(cdev, false);
@@ -1031,8 +1085,11 @@
qed_ll2_dealloc_if(cdev);
if (IS_PF(cdev)) {
- qed_ptt_release(QED_LEADING_HWFN(cdev),
- QED_LEADING_HWFN(cdev)->p_ptp_ptt);
+#ifdef CONFIG_RFS_ACCEL
+ if (cdev->num_hwfns == 1)
+ qed_ptt_release(QED_LEADING_HWFN(cdev),
+ QED_LEADING_HWFN(cdev)->p_arfs_ptt);
+#endif
qed_free_stream_mem(cdev);
if (IS_QED_ETH_IF(cdev))
qed_sriov_disable(cdev, true);
@@ -1042,7 +1099,8 @@
}
qed_disable_msix(cdev);
- qed_nic_reset(cdev);
+
+ qed_resc_free(cdev);
qed_iov_wq_stop(cdev, true);
@@ -1653,13 +1711,18 @@
switch (type) {
case QED_MCP_LAN_STATS:
qed_get_vport_stats(cdev, ð_stats);
- stats->lan_stats.ucast_rx_pkts = eth_stats.rx_ucast_pkts;
- stats->lan_stats.ucast_tx_pkts = eth_stats.tx_ucast_pkts;
+ stats->lan_stats.ucast_rx_pkts =
+ eth_stats.common.rx_ucast_pkts;
+ stats->lan_stats.ucast_tx_pkts =
+ eth_stats.common.tx_ucast_pkts;
stats->lan_stats.fcs_err = -1;
break;
case QED_MCP_FCOE_STATS:
qed_get_protocol_stats_fcoe(cdev, &stats->fcoe_stats);
break;
+ case QED_MCP_ISCSI_STATS:
+ qed_get_protocol_stats_iscsi(cdev, &stats->iscsi_stats);
+ break;
default:
DP_ERR(cdev, "Invalid protocol type = %d\n", type);
return;
diff --git a/drivers/net/ethernet/qlogic/qed/qed_mcp.c b/drivers/net/ethernet/qlogic/qed/qed_mcp.c
index 87fde20..7266b36 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_mcp.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_mcp.c
@@ -111,12 +111,71 @@
}
}
+struct qed_mcp_cmd_elem {
+ struct list_head list;
+ struct qed_mcp_mb_params *p_mb_params;
+ u16 expected_seq_num;
+ bool b_is_completed;
+};
+
+/* Must be called while cmd_lock is acquired */
+static struct qed_mcp_cmd_elem *
+qed_mcp_cmd_add_elem(struct qed_hwfn *p_hwfn,
+ struct qed_mcp_mb_params *p_mb_params,
+ u16 expected_seq_num)
+{
+ struct qed_mcp_cmd_elem *p_cmd_elem = NULL;
+
+ p_cmd_elem = kzalloc(sizeof(*p_cmd_elem), GFP_ATOMIC);
+ if (!p_cmd_elem)
+ goto out;
+
+ p_cmd_elem->p_mb_params = p_mb_params;
+ p_cmd_elem->expected_seq_num = expected_seq_num;
+ list_add(&p_cmd_elem->list, &p_hwfn->mcp_info->cmd_list);
+out:
+ return p_cmd_elem;
+}
+
+/* Must be called while cmd_lock is acquired */
+static void qed_mcp_cmd_del_elem(struct qed_hwfn *p_hwfn,
+ struct qed_mcp_cmd_elem *p_cmd_elem)
+{
+ list_del(&p_cmd_elem->list);
+ kfree(p_cmd_elem);
+}
+
+/* Must be called while cmd_lock is acquired */
+static struct qed_mcp_cmd_elem *qed_mcp_cmd_get_elem(struct qed_hwfn *p_hwfn,
+ u16 seq_num)
+{
+ struct qed_mcp_cmd_elem *p_cmd_elem = NULL;
+
+ list_for_each_entry(p_cmd_elem, &p_hwfn->mcp_info->cmd_list, list) {
+ if (p_cmd_elem->expected_seq_num == seq_num)
+ return p_cmd_elem;
+ }
+
+ return NULL;
+}
+
int qed_mcp_free(struct qed_hwfn *p_hwfn)
{
if (p_hwfn->mcp_info) {
+ struct qed_mcp_cmd_elem *p_cmd_elem, *p_tmp;
+
kfree(p_hwfn->mcp_info->mfw_mb_cur);
kfree(p_hwfn->mcp_info->mfw_mb_shadow);
+
+ spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
+ list_for_each_entry_safe(p_cmd_elem,
+ p_tmp,
+ &p_hwfn->mcp_info->cmd_list, list) {
+ qed_mcp_cmd_del_elem(p_hwfn, p_cmd_elem);
+ }
+ spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
}
+
kfree(p_hwfn->mcp_info);
return 0;
@@ -160,7 +219,7 @@
p_info->drv_pulse_seq = DRV_MB_RD(p_hwfn, p_ptt, drv_pulse_mb) &
DRV_PULSE_SEQ_MASK;
- p_info->mcp_hist = (u16)qed_rd(p_hwfn, p_ptt, MISCS_REG_GENERIC_POR_0);
+ p_info->mcp_hist = qed_rd(p_hwfn, p_ptt, MISCS_REG_GENERIC_POR_0);
return 0;
}
@@ -176,6 +235,12 @@
goto err;
p_info = p_hwfn->mcp_info;
+ /* Initialize the MFW spinlock */
+ spin_lock_init(&p_info->cmd_lock);
+ spin_lock_init(&p_info->link_lock);
+
+ INIT_LIST_HEAD(&p_info->cmd_list);
+
if (qed_load_mcp_offsets(p_hwfn, p_ptt) != 0) {
DP_NOTICE(p_hwfn, "MCP is not initialized\n");
/* Do not free mcp_info here, since public_base indicate that
@@ -190,10 +255,6 @@
if (!p_info->mfw_mb_shadow || !p_info->mfw_mb_addr)
goto err;
- /* Initialize the MFW spinlock */
- spin_lock_init(&p_info->lock);
- spin_lock_init(&p_info->link_lock);
-
return 0;
err:
@@ -201,68 +262,39 @@
return -ENOMEM;
}
-/* Locks the MFW mailbox of a PF to ensure a single access.
- * The lock is achieved in most cases by holding a spinlock, causing other
- * threads to wait till a previous access is done.
- * In some cases (currently when a [UN]LOAD_REQ commands are sent), the single
- * access is achieved by setting a blocking flag, which will fail other
- * competing contexts to send their mailboxes.
- */
-static int qed_mcp_mb_lock(struct qed_hwfn *p_hwfn, u32 cmd)
+static void qed_mcp_reread_offsets(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt)
{
- spin_lock_bh(&p_hwfn->mcp_info->lock);
+ u32 generic_por_0 = qed_rd(p_hwfn, p_ptt, MISCS_REG_GENERIC_POR_0);
- /* The spinlock shouldn't be acquired when the mailbox command is
- * [UN]LOAD_REQ, since the engine is locked by the MFW, and a parallel
- * pending [UN]LOAD_REQ command of another PF together with a spinlock
- * (i.e. interrupts are disabled) - can lead to a deadlock.
- * It is assumed that for a single PF, no other mailbox commands can be
- * sent from another context while sending LOAD_REQ, and that any
- * parallel commands to UNLOAD_REQ can be cancelled.
+ /* Use MCP history register to check if MCP reset occurred between init
+ * time and now.
*/
- if (cmd == DRV_MSG_CODE_LOAD_DONE || cmd == DRV_MSG_CODE_UNLOAD_DONE)
- p_hwfn->mcp_info->block_mb_sending = false;
+ if (p_hwfn->mcp_info->mcp_hist != generic_por_0) {
+ DP_VERBOSE(p_hwfn,
+ QED_MSG_SP,
+ "Rereading MCP offsets [mcp_hist 0x%08x, generic_por_0 0x%08x]\n",
+ p_hwfn->mcp_info->mcp_hist, generic_por_0);
- if (p_hwfn->mcp_info->block_mb_sending) {
- DP_NOTICE(p_hwfn,
- "Trying to send a MFW mailbox command [0x%x] in parallel to [UN]LOAD_REQ. Aborting.\n",
- cmd);
- spin_unlock_bh(&p_hwfn->mcp_info->lock);
- return -EBUSY;
+ qed_load_mcp_offsets(p_hwfn, p_ptt);
+ qed_mcp_cmd_port_init(p_hwfn, p_ptt);
}
-
- if (cmd == DRV_MSG_CODE_LOAD_REQ || cmd == DRV_MSG_CODE_UNLOAD_REQ) {
- p_hwfn->mcp_info->block_mb_sending = true;
- spin_unlock_bh(&p_hwfn->mcp_info->lock);
- }
-
- return 0;
-}
-
-static void qed_mcp_mb_unlock(struct qed_hwfn *p_hwfn, u32 cmd)
-{
- if (cmd != DRV_MSG_CODE_LOAD_REQ && cmd != DRV_MSG_CODE_UNLOAD_REQ)
- spin_unlock_bh(&p_hwfn->mcp_info->lock);
}
int qed_mcp_reset(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
{
- u32 seq = ++p_hwfn->mcp_info->drv_mb_seq;
- u8 delay = CHIP_MCP_RESP_ITER_US;
- u32 org_mcp_reset_seq, cnt = 0;
+ u32 org_mcp_reset_seq, seq, delay = CHIP_MCP_RESP_ITER_US, cnt = 0;
int rc = 0;
- /* Ensure that only a single thread is accessing the mailbox at a
- * certain time.
- */
- rc = qed_mcp_mb_lock(p_hwfn, DRV_MSG_CODE_MCP_RESET);
- if (rc != 0)
- return rc;
+ /* Ensure that only a single thread is accessing the mailbox */
+ spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
+
+ org_mcp_reset_seq = qed_rd(p_hwfn, p_ptt, MISCS_REG_GENERIC_POR_0);
/* Set drv command along with the updated sequence */
- org_mcp_reset_seq = qed_rd(p_hwfn, p_ptt, MISCS_REG_GENERIC_POR_0);
- DRV_MB_WR(p_hwfn, p_ptt, drv_mb_header,
- (DRV_MSG_CODE_MCP_RESET | seq));
+ qed_mcp_reread_offsets(p_hwfn, p_ptt);
+ seq = ++p_hwfn->mcp_info->drv_mb_seq;
+ DRV_MB_WR(p_hwfn, p_ptt, drv_mb_header, (DRV_MSG_CODE_MCP_RESET | seq));
do {
/* Wait for MFW response */
@@ -281,72 +313,207 @@
rc = -EAGAIN;
}
- qed_mcp_mb_unlock(p_hwfn, DRV_MSG_CODE_MCP_RESET);
+ spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
return rc;
}
-static int qed_do_mcp_cmd(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt,
- u32 cmd,
- u32 param,
- u32 *o_mcp_resp,
- u32 *o_mcp_param)
+/* Must be called while cmd_lock is acquired */
+static bool qed_mcp_has_pending_cmd(struct qed_hwfn *p_hwfn)
{
- u8 delay = CHIP_MCP_RESP_ITER_US;
- u32 seq, cnt = 1, actual_mb_seq;
+ struct qed_mcp_cmd_elem *p_cmd_elem;
+
+ /* There is at most one pending command at a certain time, and if it
+ * exists - it is placed at the HEAD of the list.
+ */
+ if (!list_empty(&p_hwfn->mcp_info->cmd_list)) {
+ p_cmd_elem = list_first_entry(&p_hwfn->mcp_info->cmd_list,
+ struct qed_mcp_cmd_elem, list);
+ return !p_cmd_elem->b_is_completed;
+ }
+
+ return false;
+}
+
+/* Must be called while cmd_lock is acquired */
+static int
+qed_mcp_update_pending_cmd(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
+{
+ struct qed_mcp_mb_params *p_mb_params;
+ struct qed_mcp_cmd_elem *p_cmd_elem;
+ u32 mcp_resp;
+ u16 seq_num;
+
+ mcp_resp = DRV_MB_RD(p_hwfn, p_ptt, fw_mb_header);
+ seq_num = (u16)(mcp_resp & FW_MSG_SEQ_NUMBER_MASK);
+
+ /* Return if no new non-handled response has been received */
+ if (seq_num != p_hwfn->mcp_info->drv_mb_seq)
+ return -EAGAIN;
+
+ p_cmd_elem = qed_mcp_cmd_get_elem(p_hwfn, seq_num);
+ if (!p_cmd_elem) {
+ DP_ERR(p_hwfn,
+ "Failed to find a pending mailbox cmd that expects sequence number %d\n",
+ seq_num);
+ return -EINVAL;
+ }
+
+ p_mb_params = p_cmd_elem->p_mb_params;
+
+ /* Get the MFW response along with the sequence number */
+ p_mb_params->mcp_resp = mcp_resp;
+
+ /* Get the MFW param */
+ p_mb_params->mcp_param = DRV_MB_RD(p_hwfn, p_ptt, fw_mb_param);
+
+ /* Get the union data */
+ if (p_mb_params->p_data_dst != NULL && p_mb_params->data_dst_size) {
+ u32 union_data_addr = p_hwfn->mcp_info->drv_mb_addr +
+ offsetof(struct public_drv_mb,
+ union_data);
+ qed_memcpy_from(p_hwfn, p_ptt, p_mb_params->p_data_dst,
+ union_data_addr, p_mb_params->data_dst_size);
+ }
+
+ p_cmd_elem->b_is_completed = true;
+
+ return 0;
+}
+
+/* Must be called while cmd_lock is acquired */
+static void __qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ struct qed_mcp_mb_params *p_mb_params,
+ u16 seq_num)
+{
+ union drv_union_data union_data;
+ u32 union_data_addr;
+
+ /* Set the union data */
+ union_data_addr = p_hwfn->mcp_info->drv_mb_addr +
+ offsetof(struct public_drv_mb, union_data);
+ memset(&union_data, 0, sizeof(union_data));
+ if (p_mb_params->p_data_src != NULL && p_mb_params->data_src_size)
+ memcpy(&union_data, p_mb_params->p_data_src,
+ p_mb_params->data_src_size);
+ qed_memcpy_to(p_hwfn, p_ptt, union_data_addr, &union_data,
+ sizeof(union_data));
+
+ /* Set the drv param */
+ DRV_MB_WR(p_hwfn, p_ptt, drv_mb_param, p_mb_params->param);
+
+ /* Set the drv command along with the sequence number */
+ DRV_MB_WR(p_hwfn, p_ptt, drv_mb_header, (p_mb_params->cmd | seq_num));
+
+ DP_VERBOSE(p_hwfn, QED_MSG_SP,
+ "MFW mailbox: command 0x%08x param 0x%08x\n",
+ (p_mb_params->cmd | seq_num), p_mb_params->param);
+}
+
+static int
+_qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ struct qed_mcp_mb_params *p_mb_params,
+ u32 max_retries, u32 delay)
+{
+ struct qed_mcp_cmd_elem *p_cmd_elem;
+ u32 cnt = 0;
+ u16 seq_num;
int rc = 0;
- /* Get actual driver mailbox sequence */
- actual_mb_seq = DRV_MB_RD(p_hwfn, p_ptt, drv_mb_header) &
- DRV_MSG_SEQ_NUMBER_MASK;
-
- /* Use MCP history register to check if MCP reset occurred between
- * init time and now.
- */
- if (p_hwfn->mcp_info->mcp_hist !=
- qed_rd(p_hwfn, p_ptt, MISCS_REG_GENERIC_POR_0)) {
- DP_VERBOSE(p_hwfn, QED_MSG_SP, "Rereading MCP offsets\n");
- qed_load_mcp_offsets(p_hwfn, p_ptt);
- qed_mcp_cmd_port_init(p_hwfn, p_ptt);
- }
- seq = ++p_hwfn->mcp_info->drv_mb_seq;
-
- /* Set drv param */
- DRV_MB_WR(p_hwfn, p_ptt, drv_mb_param, param);
-
- /* Set drv command along with the updated sequence */
- DRV_MB_WR(p_hwfn, p_ptt, drv_mb_header, (cmd | seq));
-
- DP_VERBOSE(p_hwfn, QED_MSG_SP,
- "wrote command (%x) to MFW MB param 0x%08x\n",
- (cmd | seq), param);
-
+ /* Wait until the mailbox is non-occupied */
do {
- /* Wait for MFW response */
+ /* Exit the loop if there is no pending command, or if the
+ * pending command is completed during this iteration.
+ * The spinlock stays locked until the command is sent.
+ */
+
+ spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
+
+ if (!qed_mcp_has_pending_cmd(p_hwfn))
+ break;
+
+ rc = qed_mcp_update_pending_cmd(p_hwfn, p_ptt);
+ if (!rc)
+ break;
+ else if (rc != -EAGAIN)
+ goto err;
+
+ spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
udelay(delay);
- *o_mcp_resp = DRV_MB_RD(p_hwfn, p_ptt, fw_mb_header);
+ } while (++cnt < max_retries);
- /* Give the FW up to 5 second (500*10ms) */
- } while ((seq != (*o_mcp_resp & FW_MSG_SEQ_NUMBER_MASK)) &&
- (cnt++ < QED_DRV_MB_MAX_RETRIES));
-
- DP_VERBOSE(p_hwfn, QED_MSG_SP,
- "[after %d ms] read (%x) seq is (%x) from FW MB\n",
- cnt * delay, *o_mcp_resp, seq);
-
- /* Is this a reply to our command? */
- if (seq == (*o_mcp_resp & FW_MSG_SEQ_NUMBER_MASK)) {
- *o_mcp_resp &= FW_MSG_CODE_MASK;
- /* Get the MCP param */
- *o_mcp_param = DRV_MB_RD(p_hwfn, p_ptt, fw_mb_param);
- } else {
- /* FW BUG! */
- DP_ERR(p_hwfn, "MFW failed to respond [cmd 0x%x param 0x%x]\n",
- cmd, param);
- *o_mcp_resp = 0;
- rc = -EAGAIN;
+ if (cnt >= max_retries) {
+ DP_NOTICE(p_hwfn,
+ "The MFW mailbox is occupied by an uncompleted command. Failed to send command 0x%08x [param 0x%08x].\n",
+ p_mb_params->cmd, p_mb_params->param);
+ return -EAGAIN;
}
+
+ /* Send the mailbox command */
+ qed_mcp_reread_offsets(p_hwfn, p_ptt);
+ seq_num = ++p_hwfn->mcp_info->drv_mb_seq;
+ p_cmd_elem = qed_mcp_cmd_add_elem(p_hwfn, p_mb_params, seq_num);
+ if (!p_cmd_elem) {
+ rc = -ENOMEM;
+ goto err;
+ }
+
+ __qed_mcp_cmd_and_union(p_hwfn, p_ptt, p_mb_params, seq_num);
+ spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
+
+ /* Wait for the MFW response */
+ do {
+ /* Exit the loop if the command is already completed, or if the
+ * command is completed during this iteration.
+ * The spinlock stays locked until the list element is removed.
+ */
+
+ udelay(delay);
+ spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
+
+ if (p_cmd_elem->b_is_completed)
+ break;
+
+ rc = qed_mcp_update_pending_cmd(p_hwfn, p_ptt);
+ if (!rc)
+ break;
+ else if (rc != -EAGAIN)
+ goto err;
+
+ spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
+ } while (++cnt < max_retries);
+
+ if (cnt >= max_retries) {
+ DP_NOTICE(p_hwfn,
+ "The MFW failed to respond to command 0x%08x [param 0x%08x].\n",
+ p_mb_params->cmd, p_mb_params->param);
+
+ spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
+ qed_mcp_cmd_del_elem(p_hwfn, p_cmd_elem);
+ spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
+
+ return -EAGAIN;
+ }
+
+ qed_mcp_cmd_del_elem(p_hwfn, p_cmd_elem);
+ spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
+
+ DP_VERBOSE(p_hwfn,
+ QED_MSG_SP,
+ "MFW mailbox: response 0x%08x param 0x%08x [after %d.%03d ms]\n",
+ p_mb_params->mcp_resp,
+ p_mb_params->mcp_param,
+ (cnt * delay) / 1000, (cnt * delay) % 1000);
+
+ /* Clear the sequence number from the MFW response */
+ p_mb_params->mcp_resp &= FW_MSG_CODE_MASK;
+
+ return 0;
+
+err:
+ spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
return rc;
}
@@ -354,9 +521,9 @@
struct qed_ptt *p_ptt,
struct qed_mcp_mb_params *p_mb_params)
{
- u32 union_data_addr;
-
- int rc;
+ size_t union_data_size = sizeof(union drv_union_data);
+ u32 max_retries = QED_DRV_MB_MAX_RETRIES;
+ u32 delay = CHIP_MCP_RESP_ITER_US;
/* MCP not initialized */
if (!qed_mcp_is_init(p_hwfn)) {
@@ -364,33 +531,17 @@
return -EBUSY;
}
- union_data_addr = p_hwfn->mcp_info->drv_mb_addr +
- offsetof(struct public_drv_mb, union_data);
+ if (p_mb_params->data_src_size > union_data_size ||
+ p_mb_params->data_dst_size > union_data_size) {
+ DP_ERR(p_hwfn,
+ "The provided size is larger than the union data size [src_size %u, dst_size %u, union_data_size %zu]\n",
+ p_mb_params->data_src_size,
+ p_mb_params->data_dst_size, union_data_size);
+ return -EINVAL;
+ }
- /* Ensure that only a single thread is accessing the mailbox at a
- * certain time.
- */
- rc = qed_mcp_mb_lock(p_hwfn, p_mb_params->cmd);
- if (rc)
- return rc;
-
- if (p_mb_params->p_data_src != NULL)
- qed_memcpy_to(p_hwfn, p_ptt, union_data_addr,
- p_mb_params->p_data_src,
- sizeof(*p_mb_params->p_data_src));
-
- rc = qed_do_mcp_cmd(p_hwfn, p_ptt, p_mb_params->cmd,
- p_mb_params->param, &p_mb_params->mcp_resp,
- &p_mb_params->mcp_param);
-
- if (p_mb_params->p_data_dst != NULL)
- qed_memcpy_from(p_hwfn, p_ptt, p_mb_params->p_data_dst,
- union_data_addr,
- sizeof(*p_mb_params->p_data_dst));
-
- qed_mcp_mb_unlock(p_hwfn, p_mb_params->cmd);
-
- return rc;
+ return _qed_mcp_cmd_and_union(p_hwfn, p_ptt, p_mb_params, max_retries,
+ delay);
}
int qed_mcp_cmd(struct qed_hwfn *p_hwfn,
@@ -401,32 +552,12 @@
u32 *o_mcp_param)
{
struct qed_mcp_mb_params mb_params;
- union drv_union_data data_src;
int rc;
memset(&mb_params, 0, sizeof(mb_params));
- memset(&data_src, 0, sizeof(data_src));
mb_params.cmd = cmd;
mb_params.param = param;
- /* In case of UNLOAD_DONE, set the primary MAC */
- if ((cmd == DRV_MSG_CODE_UNLOAD_DONE) &&
- (p_hwfn->cdev->wol_config == QED_OV_WOL_ENABLED)) {
- u8 *p_mac = p_hwfn->cdev->wol_mac;
-
- data_src.wol_mac.mac_upper = p_mac[0] << 8 | p_mac[1];
- data_src.wol_mac.mac_lower = p_mac[2] << 24 | p_mac[3] << 16 |
- p_mac[4] << 8 | p_mac[5];
-
- DP_VERBOSE(p_hwfn,
- (QED_MSG_SP | NETIF_MSG_IFDOWN),
- "Setting WoL MAC: %pM --> [%08x,%08x]\n",
- p_mac, data_src.wol_mac.mac_upper,
- data_src.wol_mac.mac_lower);
-
- mb_params.p_data_src = &data_src;
- }
-
rc = qed_mcp_cmd_and_union(p_hwfn, p_ptt, &mb_params);
if (rc)
return rc;
@@ -445,13 +576,17 @@
u32 *o_mcp_param, u32 *o_txn_size, u32 *o_buf)
{
struct qed_mcp_mb_params mb_params;
- union drv_union_data union_data;
+ u8 raw_data[MCP_DRV_NVM_BUF_LEN];
int rc;
memset(&mb_params, 0, sizeof(mb_params));
mb_params.cmd = cmd;
mb_params.param = param;
- mb_params.p_data_dst = &union_data;
+ mb_params.p_data_dst = raw_data;
+
+ /* Use the maximal value since the actual one is part of the response */
+ mb_params.data_dst_size = MCP_DRV_NVM_BUF_LEN;
+
rc = qed_mcp_cmd_and_union(p_hwfn, p_ptt, &mb_params);
if (rc)
return rc;
@@ -460,55 +595,413 @@
*o_mcp_param = mb_params.mcp_param;
*o_txn_size = *o_mcp_param;
- memcpy(o_buf, &union_data.raw_data, *o_txn_size);
+ memcpy(o_buf, raw_data, *o_txn_size);
return 0;
}
-int qed_mcp_load_req(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt, u32 *p_load_code)
+static bool
+qed_mcp_can_force_load(u8 drv_role,
+ u8 exist_drv_role,
+ enum qed_override_force_load override_force_load)
{
- struct qed_dev *cdev = p_hwfn->cdev;
- struct qed_mcp_mb_params mb_params;
- union drv_union_data union_data;
+ bool can_force_load = false;
+
+ switch (override_force_load) {
+ case QED_OVERRIDE_FORCE_LOAD_ALWAYS:
+ can_force_load = true;
+ break;
+ case QED_OVERRIDE_FORCE_LOAD_NEVER:
+ can_force_load = false;
+ break;
+ default:
+ can_force_load = (drv_role == DRV_ROLE_OS &&
+ exist_drv_role == DRV_ROLE_PREBOOT) ||
+ (drv_role == DRV_ROLE_KDUMP &&
+ exist_drv_role == DRV_ROLE_OS);
+ break;
+ }
+
+ return can_force_load;
+}
+
+static int qed_mcp_cancel_load_req(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt)
+{
+ u32 resp = 0, param = 0;
int rc;
- memset(&mb_params, 0, sizeof(mb_params));
- /* Load Request */
- mb_params.cmd = DRV_MSG_CODE_LOAD_REQ;
- mb_params.param = PDA_COMP | DRV_ID_MCP_HSI_VER_CURRENT |
- cdev->drv_type;
- memcpy(&union_data.ver_str, cdev->ver_str, MCP_DRV_VER_STR_SIZE);
- mb_params.p_data_src = &union_data;
- rc = qed_mcp_cmd_and_union(p_hwfn, p_ptt, &mb_params);
+ rc = qed_mcp_cmd(p_hwfn, p_ptt, DRV_MSG_CODE_CANCEL_LOAD_REQ, 0,
+ &resp, ¶m);
+ if (rc)
+ DP_NOTICE(p_hwfn,
+ "Failed to send cancel load request, rc = %d\n", rc);
- /* if mcp fails to respond we must abort */
+ return rc;
+}
+
+#define CONFIG_QEDE_BITMAP_IDX BIT(0)
+#define CONFIG_QED_SRIOV_BITMAP_IDX BIT(1)
+#define CONFIG_QEDR_BITMAP_IDX BIT(2)
+#define CONFIG_QEDF_BITMAP_IDX BIT(4)
+#define CONFIG_QEDI_BITMAP_IDX BIT(5)
+#define CONFIG_QED_LL2_BITMAP_IDX BIT(6)
+
+static u32 qed_get_config_bitmap(void)
+{
+ u32 config_bitmap = 0x0;
+
+ if (IS_ENABLED(CONFIG_QEDE))
+ config_bitmap |= CONFIG_QEDE_BITMAP_IDX;
+
+ if (IS_ENABLED(CONFIG_QED_SRIOV))
+ config_bitmap |= CONFIG_QED_SRIOV_BITMAP_IDX;
+
+ if (IS_ENABLED(CONFIG_QED_RDMA))
+ config_bitmap |= CONFIG_QEDR_BITMAP_IDX;
+
+ if (IS_ENABLED(CONFIG_QED_FCOE))
+ config_bitmap |= CONFIG_QEDF_BITMAP_IDX;
+
+ if (IS_ENABLED(CONFIG_QED_ISCSI))
+ config_bitmap |= CONFIG_QEDI_BITMAP_IDX;
+
+ if (IS_ENABLED(CONFIG_QED_LL2))
+ config_bitmap |= CONFIG_QED_LL2_BITMAP_IDX;
+
+ return config_bitmap;
+}
+
+struct qed_load_req_in_params {
+ u8 hsi_ver;
+#define QED_LOAD_REQ_HSI_VER_DEFAULT 0
+#define QED_LOAD_REQ_HSI_VER_1 1
+ u32 drv_ver_0;
+ u32 drv_ver_1;
+ u32 fw_ver;
+ u8 drv_role;
+ u8 timeout_val;
+ u8 force_cmd;
+ bool avoid_eng_reset;
+};
+
+struct qed_load_req_out_params {
+ u32 load_code;
+ u32 exist_drv_ver_0;
+ u32 exist_drv_ver_1;
+ u32 exist_fw_ver;
+ u8 exist_drv_role;
+ u8 mfw_hsi_ver;
+ bool drv_exists;
+};
+
+static int
+__qed_mcp_load_req(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ struct qed_load_req_in_params *p_in_params,
+ struct qed_load_req_out_params *p_out_params)
+{
+ struct qed_mcp_mb_params mb_params;
+ struct load_req_stc load_req;
+ struct load_rsp_stc load_rsp;
+ u32 hsi_ver;
+ int rc;
+
+ memset(&load_req, 0, sizeof(load_req));
+ load_req.drv_ver_0 = p_in_params->drv_ver_0;
+ load_req.drv_ver_1 = p_in_params->drv_ver_1;
+ load_req.fw_ver = p_in_params->fw_ver;
+ QED_MFW_SET_FIELD(load_req.misc0, LOAD_REQ_ROLE, p_in_params->drv_role);
+ QED_MFW_SET_FIELD(load_req.misc0, LOAD_REQ_LOCK_TO,
+ p_in_params->timeout_val);
+ QED_MFW_SET_FIELD(load_req.misc0, LOAD_REQ_FORCE,
+ p_in_params->force_cmd);
+ QED_MFW_SET_FIELD(load_req.misc0, LOAD_REQ_FLAGS0,
+ p_in_params->avoid_eng_reset);
+
+ hsi_ver = (p_in_params->hsi_ver == QED_LOAD_REQ_HSI_VER_DEFAULT) ?
+ DRV_ID_MCP_HSI_VER_CURRENT :
+ (p_in_params->hsi_ver << DRV_ID_MCP_HSI_VER_SHIFT);
+
+ memset(&mb_params, 0, sizeof(mb_params));
+ mb_params.cmd = DRV_MSG_CODE_LOAD_REQ;
+ mb_params.param = PDA_COMP | hsi_ver | p_hwfn->cdev->drv_type;
+ mb_params.p_data_src = &load_req;
+ mb_params.data_src_size = sizeof(load_req);
+ mb_params.p_data_dst = &load_rsp;
+ mb_params.data_dst_size = sizeof(load_rsp);
+
+ DP_VERBOSE(p_hwfn, QED_MSG_SP,
+ "Load Request: param 0x%08x [init_hw %d, drv_type %d, hsi_ver %d, pda 0x%04x]\n",
+ mb_params.param,
+ QED_MFW_GET_FIELD(mb_params.param, DRV_ID_DRV_INIT_HW),
+ QED_MFW_GET_FIELD(mb_params.param, DRV_ID_DRV_TYPE),
+ QED_MFW_GET_FIELD(mb_params.param, DRV_ID_MCP_HSI_VER),
+ QED_MFW_GET_FIELD(mb_params.param, DRV_ID_PDA_COMP_VER));
+
+ if (p_in_params->hsi_ver != QED_LOAD_REQ_HSI_VER_1) {
+ DP_VERBOSE(p_hwfn, QED_MSG_SP,
+ "Load Request: drv_ver 0x%08x_0x%08x, fw_ver 0x%08x, misc0 0x%08x [role %d, timeout %d, force %d, flags0 0x%x]\n",
+ load_req.drv_ver_0,
+ load_req.drv_ver_1,
+ load_req.fw_ver,
+ load_req.misc0,
+ QED_MFW_GET_FIELD(load_req.misc0, LOAD_REQ_ROLE),
+ QED_MFW_GET_FIELD(load_req.misc0,
+ LOAD_REQ_LOCK_TO),
+ QED_MFW_GET_FIELD(load_req.misc0, LOAD_REQ_FORCE),
+ QED_MFW_GET_FIELD(load_req.misc0, LOAD_REQ_FLAGS0));
+ }
+
+ rc = qed_mcp_cmd_and_union(p_hwfn, p_ptt, &mb_params);
if (rc) {
- DP_ERR(p_hwfn, "MCP response failure, aborting\n");
+ DP_NOTICE(p_hwfn, "Failed to send load request, rc = %d\n", rc);
return rc;
}
- *p_load_code = mb_params.mcp_resp;
+ DP_VERBOSE(p_hwfn, QED_MSG_SP,
+ "Load Response: resp 0x%08x\n", mb_params.mcp_resp);
+ p_out_params->load_code = mb_params.mcp_resp;
- /* If MFW refused (e.g. other port is in diagnostic mode) we
- * must abort. This can happen in the following cases:
- * - Other port is in diagnostic mode
- * - Previously loaded function on the engine is not compliant with
- * the requester.
- * - MFW cannot cope with the requester's DRV_MFW_HSI_VERSION.
- * -
- */
- if (!(*p_load_code) ||
- ((*p_load_code) == FW_MSG_CODE_DRV_LOAD_REFUSED_HSI) ||
- ((*p_load_code) == FW_MSG_CODE_DRV_LOAD_REFUSED_PDA) ||
- ((*p_load_code) == FW_MSG_CODE_DRV_LOAD_REFUSED_DIAG)) {
- DP_ERR(p_hwfn, "MCP refused load request, aborting\n");
- return -EBUSY;
+ if (p_in_params->hsi_ver != QED_LOAD_REQ_HSI_VER_1 &&
+ p_out_params->load_code != FW_MSG_CODE_DRV_LOAD_REFUSED_HSI_1) {
+ DP_VERBOSE(p_hwfn,
+ QED_MSG_SP,
+ "Load Response: exist_drv_ver 0x%08x_0x%08x, exist_fw_ver 0x%08x, misc0 0x%08x [exist_role %d, mfw_hsi %d, flags0 0x%x]\n",
+ load_rsp.drv_ver_0,
+ load_rsp.drv_ver_1,
+ load_rsp.fw_ver,
+ load_rsp.misc0,
+ QED_MFW_GET_FIELD(load_rsp.misc0, LOAD_RSP_ROLE),
+ QED_MFW_GET_FIELD(load_rsp.misc0, LOAD_RSP_HSI),
+ QED_MFW_GET_FIELD(load_rsp.misc0, LOAD_RSP_FLAGS0));
+
+ p_out_params->exist_drv_ver_0 = load_rsp.drv_ver_0;
+ p_out_params->exist_drv_ver_1 = load_rsp.drv_ver_1;
+ p_out_params->exist_fw_ver = load_rsp.fw_ver;
+ p_out_params->exist_drv_role =
+ QED_MFW_GET_FIELD(load_rsp.misc0, LOAD_RSP_ROLE);
+ p_out_params->mfw_hsi_ver =
+ QED_MFW_GET_FIELD(load_rsp.misc0, LOAD_RSP_HSI);
+ p_out_params->drv_exists =
+ QED_MFW_GET_FIELD(load_rsp.misc0, LOAD_RSP_FLAGS0) &
+ LOAD_RSP_FLAGS0_DRV_EXISTS;
}
return 0;
}
+static int eocre_get_mfw_drv_role(struct qed_hwfn *p_hwfn,
+ enum qed_drv_role drv_role,
+ u8 *p_mfw_drv_role)
+{
+ switch (drv_role) {
+ case QED_DRV_ROLE_OS:
+ *p_mfw_drv_role = DRV_ROLE_OS;
+ break;
+ case QED_DRV_ROLE_KDUMP:
+ *p_mfw_drv_role = DRV_ROLE_KDUMP;
+ break;
+ default:
+ DP_ERR(p_hwfn, "Unexpected driver role %d\n", drv_role);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+enum qed_load_req_force {
+ QED_LOAD_REQ_FORCE_NONE,
+ QED_LOAD_REQ_FORCE_PF,
+ QED_LOAD_REQ_FORCE_ALL,
+};
+
+static void qed_get_mfw_force_cmd(struct qed_hwfn *p_hwfn,
+
+ enum qed_load_req_force force_cmd,
+ u8 *p_mfw_force_cmd)
+{
+ switch (force_cmd) {
+ case QED_LOAD_REQ_FORCE_NONE:
+ *p_mfw_force_cmd = LOAD_REQ_FORCE_NONE;
+ break;
+ case QED_LOAD_REQ_FORCE_PF:
+ *p_mfw_force_cmd = LOAD_REQ_FORCE_PF;
+ break;
+ case QED_LOAD_REQ_FORCE_ALL:
+ *p_mfw_force_cmd = LOAD_REQ_FORCE_ALL;
+ break;
+ }
+}
+
+int qed_mcp_load_req(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ struct qed_load_req_params *p_params)
+{
+ struct qed_load_req_out_params out_params;
+ struct qed_load_req_in_params in_params;
+ u8 mfw_drv_role, mfw_force_cmd;
+ int rc;
+
+ memset(&in_params, 0, sizeof(in_params));
+ in_params.hsi_ver = QED_LOAD_REQ_HSI_VER_DEFAULT;
+ in_params.drv_ver_0 = QED_VERSION;
+ in_params.drv_ver_1 = qed_get_config_bitmap();
+ in_params.fw_ver = STORM_FW_VERSION;
+ rc = eocre_get_mfw_drv_role(p_hwfn, p_params->drv_role, &mfw_drv_role);
+ if (rc)
+ return rc;
+
+ in_params.drv_role = mfw_drv_role;
+ in_params.timeout_val = p_params->timeout_val;
+ qed_get_mfw_force_cmd(p_hwfn,
+ QED_LOAD_REQ_FORCE_NONE, &mfw_force_cmd);
+
+ in_params.force_cmd = mfw_force_cmd;
+ in_params.avoid_eng_reset = p_params->avoid_eng_reset;
+
+ memset(&out_params, 0, sizeof(out_params));
+ rc = __qed_mcp_load_req(p_hwfn, p_ptt, &in_params, &out_params);
+ if (rc)
+ return rc;
+
+ /* First handle cases where another load request should/might be sent:
+ * - MFW expects the old interface [HSI version = 1]
+ * - MFW responds that a force load request is required
+ */
+ if (out_params.load_code == FW_MSG_CODE_DRV_LOAD_REFUSED_HSI_1) {
+ DP_INFO(p_hwfn,
+ "MFW refused a load request due to HSI > 1. Resending with HSI = 1\n");
+
+ in_params.hsi_ver = QED_LOAD_REQ_HSI_VER_1;
+ memset(&out_params, 0, sizeof(out_params));
+ rc = __qed_mcp_load_req(p_hwfn, p_ptt, &in_params, &out_params);
+ if (rc)
+ return rc;
+ } else if (out_params.load_code ==
+ FW_MSG_CODE_DRV_LOAD_REFUSED_REQUIRES_FORCE) {
+ if (qed_mcp_can_force_load(in_params.drv_role,
+ out_params.exist_drv_role,
+ p_params->override_force_load)) {
+ DP_INFO(p_hwfn,
+ "A force load is required [{role, fw_ver, drv_ver}: loading={%d, 0x%08x, x%08x_0x%08x}, existing={%d, 0x%08x, 0x%08x_0x%08x}]\n",
+ in_params.drv_role, in_params.fw_ver,
+ in_params.drv_ver_0, in_params.drv_ver_1,
+ out_params.exist_drv_role,
+ out_params.exist_fw_ver,
+ out_params.exist_drv_ver_0,
+ out_params.exist_drv_ver_1);
+
+ qed_get_mfw_force_cmd(p_hwfn,
+ QED_LOAD_REQ_FORCE_ALL,
+ &mfw_force_cmd);
+
+ in_params.force_cmd = mfw_force_cmd;
+ memset(&out_params, 0, sizeof(out_params));
+ rc = __qed_mcp_load_req(p_hwfn, p_ptt, &in_params,
+ &out_params);
+ if (rc)
+ return rc;
+ } else {
+ DP_NOTICE(p_hwfn,
+ "A force load is required [{role, fw_ver, drv_ver}: loading={%d, 0x%08x, x%08x_0x%08x}, existing={%d, 0x%08x, 0x%08x_0x%08x}] - Avoid\n",
+ in_params.drv_role, in_params.fw_ver,
+ in_params.drv_ver_0, in_params.drv_ver_1,
+ out_params.exist_drv_role,
+ out_params.exist_fw_ver,
+ out_params.exist_drv_ver_0,
+ out_params.exist_drv_ver_1);
+ DP_NOTICE(p_hwfn,
+ "Avoid sending a force load request to prevent disruption of active PFs\n");
+
+ qed_mcp_cancel_load_req(p_hwfn, p_ptt);
+ return -EBUSY;
+ }
+ }
+
+ /* Now handle the other types of responses.
+ * The "REFUSED_HSI_1" and "REFUSED_REQUIRES_FORCE" responses are not
+ * expected here after the additional revised load requests were sent.
+ */
+ switch (out_params.load_code) {
+ case FW_MSG_CODE_DRV_LOAD_ENGINE:
+ case FW_MSG_CODE_DRV_LOAD_PORT:
+ case FW_MSG_CODE_DRV_LOAD_FUNCTION:
+ if (out_params.mfw_hsi_ver != QED_LOAD_REQ_HSI_VER_1 &&
+ out_params.drv_exists) {
+ /* The role and fw/driver version match, but the PF is
+ * already loaded and has not been unloaded gracefully.
+ */
+ DP_NOTICE(p_hwfn,
+ "PF is already loaded\n");
+ return -EINVAL;
+ }
+ break;
+ default:
+ DP_NOTICE(p_hwfn,
+ "Unexpected refusal to load request [resp 0x%08x]. Aborting.\n",
+ out_params.load_code);
+ return -EBUSY;
+ }
+
+ p_params->load_code = out_params.load_code;
+
+ return 0;
+}
+
+int qed_mcp_unload_req(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
+{
+ u32 wol_param, mcp_resp, mcp_param;
+
+ switch (p_hwfn->cdev->wol_config) {
+ case QED_OV_WOL_DISABLED:
+ wol_param = DRV_MB_PARAM_UNLOAD_WOL_DISABLED;
+ break;
+ case QED_OV_WOL_ENABLED:
+ wol_param = DRV_MB_PARAM_UNLOAD_WOL_ENABLED;
+ break;
+ default:
+ DP_NOTICE(p_hwfn,
+ "Unknown WoL configuration %02x\n",
+ p_hwfn->cdev->wol_config);
+ /* Fallthrough */
+ case QED_OV_WOL_DEFAULT:
+ wol_param = DRV_MB_PARAM_UNLOAD_WOL_MCP;
+ }
+
+ return qed_mcp_cmd(p_hwfn, p_ptt, DRV_MSG_CODE_UNLOAD_REQ, wol_param,
+ &mcp_resp, &mcp_param);
+}
+
+int qed_mcp_unload_done(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
+{
+ struct qed_mcp_mb_params mb_params;
+ struct mcp_mac wol_mac;
+
+ memset(&mb_params, 0, sizeof(mb_params));
+ mb_params.cmd = DRV_MSG_CODE_UNLOAD_DONE;
+
+ /* Set the primary MAC if WoL is enabled */
+ if (p_hwfn->cdev->wol_config == QED_OV_WOL_ENABLED) {
+ u8 *p_mac = p_hwfn->cdev->wol_mac;
+
+ memset(&wol_mac, 0, sizeof(wol_mac));
+ wol_mac.mac_upper = p_mac[0] << 8 | p_mac[1];
+ wol_mac.mac_lower = p_mac[2] << 24 | p_mac[3] << 16 |
+ p_mac[4] << 8 | p_mac[5];
+
+ DP_VERBOSE(p_hwfn,
+ (QED_MSG_SP | NETIF_MSG_IFDOWN),
+ "Setting WoL MAC: %pM --> [%08x,%08x]\n",
+ p_mac, wol_mac.mac_upper, wol_mac.mac_lower);
+
+ mb_params.p_data_src = &wol_mac;
+ mb_params.data_src_size = sizeof(wol_mac);
+ }
+
+ return qed_mcp_cmd_and_union(p_hwfn, p_ptt, &mb_params);
+}
+
static void qed_mcp_handle_vf_flr(struct qed_hwfn *p_hwfn,
struct qed_ptt *p_ptt)
{
@@ -549,7 +1042,6 @@
u32 func_addr = SECTION_ADDR(mfw_func_offsize,
MCP_PF_ID(p_hwfn));
struct qed_mcp_mb_params mb_params;
- union drv_union_data union_data;
int rc;
int i;
@@ -560,8 +1052,8 @@
memset(&mb_params, 0, sizeof(mb_params));
mb_params.cmd = DRV_MSG_CODE_VF_DISABLED_DONE;
- memcpy(&union_data.ack_vf_disabled, vfs_to_ack, VF_MAX_STATIC / 8);
- mb_params.p_data_src = &union_data;
+ mb_params.p_data_src = vfs_to_ack;
+ mb_params.data_src_size = VF_MAX_STATIC / 8;
rc = qed_mcp_cmd_and_union(p_hwfn, p_ptt, &mb_params);
if (rc) {
DP_NOTICE(p_hwfn, "Failed to pass ACK for VF flr to MFW\n");
@@ -744,33 +1236,31 @@
{
struct qed_mcp_link_params *params = &p_hwfn->mcp_info->link_input;
struct qed_mcp_mb_params mb_params;
- union drv_union_data union_data;
- struct eth_phy_cfg *phy_cfg;
+ struct eth_phy_cfg phy_cfg;
int rc = 0;
u32 cmd;
/* Set the shmem configuration according to params */
- phy_cfg = &union_data.drv_phy_cfg;
- memset(phy_cfg, 0, sizeof(*phy_cfg));
+ memset(&phy_cfg, 0, sizeof(phy_cfg));
cmd = b_up ? DRV_MSG_CODE_INIT_PHY : DRV_MSG_CODE_LINK_RESET;
if (!params->speed.autoneg)
- phy_cfg->speed = params->speed.forced_speed;
- phy_cfg->pause |= (params->pause.autoneg) ? ETH_PAUSE_AUTONEG : 0;
- phy_cfg->pause |= (params->pause.forced_rx) ? ETH_PAUSE_RX : 0;
- phy_cfg->pause |= (params->pause.forced_tx) ? ETH_PAUSE_TX : 0;
- phy_cfg->adv_speed = params->speed.advertised_speeds;
- phy_cfg->loopback_mode = params->loopback_mode;
+ phy_cfg.speed = params->speed.forced_speed;
+ phy_cfg.pause |= (params->pause.autoneg) ? ETH_PAUSE_AUTONEG : 0;
+ phy_cfg.pause |= (params->pause.forced_rx) ? ETH_PAUSE_RX : 0;
+ phy_cfg.pause |= (params->pause.forced_tx) ? ETH_PAUSE_TX : 0;
+ phy_cfg.adv_speed = params->speed.advertised_speeds;
+ phy_cfg.loopback_mode = params->loopback_mode;
p_hwfn->b_drv_link_init = b_up;
if (b_up) {
DP_VERBOSE(p_hwfn, NETIF_MSG_LINK,
"Configuring Link: Speed 0x%08x, Pause 0x%08x, adv_speed 0x%08x, loopback 0x%08x, features 0x%08x\n",
- phy_cfg->speed,
- phy_cfg->pause,
- phy_cfg->adv_speed,
- phy_cfg->loopback_mode,
- phy_cfg->feature_config_flags);
+ phy_cfg.speed,
+ phy_cfg.pause,
+ phy_cfg.adv_speed,
+ phy_cfg.loopback_mode,
+ phy_cfg.feature_config_flags);
} else {
DP_VERBOSE(p_hwfn, NETIF_MSG_LINK,
"Resetting link\n");
@@ -778,7 +1268,8 @@
memset(&mb_params, 0, sizeof(mb_params));
mb_params.cmd = cmd;
- mb_params.p_data_src = &union_data;
+ mb_params.p_data_src = &phy_cfg;
+ mb_params.data_src_size = sizeof(phy_cfg);
rc = qed_mcp_cmd_and_union(p_hwfn, p_ptt, &mb_params);
/* if mcp fails to respond we must abort */
@@ -805,7 +1296,6 @@
enum qed_mcp_protocol_type stats_type;
union qed_mcp_protocol_stats stats;
struct qed_mcp_mb_params mb_params;
- union drv_union_data union_data;
u32 hsi_param;
switch (type) {
@@ -835,8 +1325,8 @@
memset(&mb_params, 0, sizeof(mb_params));
mb_params.cmd = DRV_MSG_CODE_GET_STATS;
mb_params.param = hsi_param;
- memcpy(&union_data, &stats, sizeof(stats));
- mb_params.p_data_src = &union_data;
+ mb_params.p_data_src = &stats;
+ mb_params.data_src_size = sizeof(stats);
qed_mcp_cmd_and_union(p_hwfn, p_ptt, &mb_params);
}
@@ -963,7 +1453,7 @@
qed_mcp_update_bw(p_hwfn, p_ptt);
break;
default:
- DP_NOTICE(p_hwfn, "Unimplemented MFW message %d\n", i);
+ DP_INFO(p_hwfn, "Unimplemented MFW message %d\n", i);
rc = -EINVAL;
}
}
@@ -1316,24 +1806,23 @@
struct qed_ptt *p_ptt,
struct qed_mcp_drv_version *p_ver)
{
- struct drv_version_stc *p_drv_version;
struct qed_mcp_mb_params mb_params;
- union drv_union_data union_data;
+ struct drv_version_stc drv_version;
__be32 val;
u32 i;
int rc;
- p_drv_version = &union_data.drv_version;
- p_drv_version->version = p_ver->version;
-
+ memset(&drv_version, 0, sizeof(drv_version));
+ drv_version.version = p_ver->version;
for (i = 0; i < (MCP_DRV_VER_STR_SIZE - 4) / sizeof(u32); i++) {
val = cpu_to_be32(*((u32 *)&p_ver->name[i * sizeof(u32)]));
- *(__be32 *)&p_drv_version->name[i * sizeof(u32)] = val;
+ *(__be32 *)&drv_version.name[i * sizeof(u32)] = val;
}
memset(&mb_params, 0, sizeof(mb_params));
mb_params.cmd = DRV_MSG_CODE_SET_VERSION;
- mb_params.p_data_src = &union_data;
+ mb_params.p_data_src = &drv_version;
+ mb_params.data_src_size = sizeof(drv_version);
rc = qed_mcp_cmd_and_union(p_hwfn, p_ptt, &mb_params);
if (rc)
DP_ERR(p_hwfn, "MCP response failure, aborting\n");
@@ -1450,7 +1939,7 @@
struct qed_ptt *p_ptt, u8 *mac)
{
struct qed_mcp_mb_params mb_params;
- union drv_union_data union_data;
+ u32 mfw_mac[2];
int rc;
memset(&mb_params, 0, sizeof(mb_params));
@@ -1458,8 +1947,17 @@
mb_params.param = DRV_MSG_CODE_VMAC_TYPE_MAC <<
DRV_MSG_CODE_VMAC_TYPE_SHIFT;
mb_params.param |= MCP_PF_ID(p_hwfn);
- ether_addr_copy(&union_data.raw_data[0], mac);
- mb_params.p_data_src = &union_data;
+
+ /* MCP is BE, and on LE platforms PCI would swap access to SHMEM
+ * in 32-bit granularity.
+ * So the MAC has to be set in native order [and not byte order],
+ * otherwise it would be read incorrectly by MFW after swap.
+ */
+ mfw_mac[0] = mac[0] << 24 | mac[1] << 16 | mac[2] << 8 | mac[3];
+ mfw_mac[1] = mac[4] << 24 | mac[5] << 16;
+
+ mb_params.p_data_src = (u8 *)mfw_mac;
+ mb_params.data_src_size = 8;
rc = qed_mcp_cmd_and_union(p_hwfn, p_ptt, &mb_params);
if (rc)
DP_ERR(p_hwfn, "Failed to send mac address, rc = %d\n", rc);
@@ -1724,52 +2222,426 @@
return rc;
}
-#define QED_RESC_ALLOC_VERSION_MAJOR 1
+static enum resource_id_enum qed_mcp_get_mfw_res_id(enum qed_resources res_id)
+{
+ enum resource_id_enum mfw_res_id = RESOURCE_NUM_INVALID;
+
+ switch (res_id) {
+ case QED_SB:
+ mfw_res_id = RESOURCE_NUM_SB_E;
+ break;
+ case QED_L2_QUEUE:
+ mfw_res_id = RESOURCE_NUM_L2_QUEUE_E;
+ break;
+ case QED_VPORT:
+ mfw_res_id = RESOURCE_NUM_VPORT_E;
+ break;
+ case QED_RSS_ENG:
+ mfw_res_id = RESOURCE_NUM_RSS_ENGINES_E;
+ break;
+ case QED_PQ:
+ mfw_res_id = RESOURCE_NUM_PQ_E;
+ break;
+ case QED_RL:
+ mfw_res_id = RESOURCE_NUM_RL_E;
+ break;
+ case QED_MAC:
+ case QED_VLAN:
+ /* Each VFC resource can accommodate both a MAC and a VLAN */
+ mfw_res_id = RESOURCE_VFC_FILTER_E;
+ break;
+ case QED_ILT:
+ mfw_res_id = RESOURCE_ILT_E;
+ break;
+ case QED_LL2_QUEUE:
+ mfw_res_id = RESOURCE_LL2_QUEUE_E;
+ break;
+ case QED_RDMA_CNQ_RAM:
+ case QED_CMDQS_CQS:
+ /* CNQ/CMDQS are the same resource */
+ mfw_res_id = RESOURCE_CQS_E;
+ break;
+ case QED_RDMA_STATS_QUEUE:
+ mfw_res_id = RESOURCE_RDMA_STATS_QUEUE_E;
+ break;
+ case QED_BDQ:
+ mfw_res_id = RESOURCE_BDQ_E;
+ break;
+ default:
+ break;
+ }
+
+ return mfw_res_id;
+}
+
+#define QED_RESC_ALLOC_VERSION_MAJOR 2
#define QED_RESC_ALLOC_VERSION_MINOR 0
#define QED_RESC_ALLOC_VERSION \
((QED_RESC_ALLOC_VERSION_MAJOR << \
DRV_MB_PARAM_RESOURCE_ALLOC_VERSION_MAJOR_SHIFT) | \
(QED_RESC_ALLOC_VERSION_MINOR << \
DRV_MB_PARAM_RESOURCE_ALLOC_VERSION_MINOR_SHIFT))
-int qed_mcp_get_resc_info(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt,
- struct resource_info *p_resc_info,
- u32 *p_mcp_resp, u32 *p_mcp_param)
+
+struct qed_resc_alloc_in_params {
+ u32 cmd;
+ enum qed_resources res_id;
+ u32 resc_max_val;
+};
+
+struct qed_resc_alloc_out_params {
+ u32 mcp_resp;
+ u32 mcp_param;
+ u32 resc_num;
+ u32 resc_start;
+ u32 vf_resc_num;
+ u32 vf_resc_start;
+ u32 flags;
+};
+
+static int
+qed_mcp_resc_allocation_msg(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ struct qed_resc_alloc_in_params *p_in_params,
+ struct qed_resc_alloc_out_params *p_out_params)
{
struct qed_mcp_mb_params mb_params;
- union drv_union_data union_data;
+ struct resource_info mfw_resc_info;
int rc;
+ memset(&mfw_resc_info, 0, sizeof(mfw_resc_info));
+
+ mfw_resc_info.res_id = qed_mcp_get_mfw_res_id(p_in_params->res_id);
+ if (mfw_resc_info.res_id == RESOURCE_NUM_INVALID) {
+ DP_ERR(p_hwfn,
+ "Failed to match resource %d [%s] with the MFW resources\n",
+ p_in_params->res_id,
+ qed_hw_get_resc_name(p_in_params->res_id));
+ return -EINVAL;
+ }
+
+ switch (p_in_params->cmd) {
+ case DRV_MSG_SET_RESOURCE_VALUE_MSG:
+ mfw_resc_info.size = p_in_params->resc_max_val;
+ /* Fallthrough */
+ case DRV_MSG_GET_RESOURCE_ALLOC_MSG:
+ break;
+ default:
+ DP_ERR(p_hwfn, "Unexpected resource alloc command [0x%08x]\n",
+ p_in_params->cmd);
+ return -EINVAL;
+ }
+
memset(&mb_params, 0, sizeof(mb_params));
- memset(&union_data, 0, sizeof(union_data));
- mb_params.cmd = DRV_MSG_GET_RESOURCE_ALLOC_MSG;
+ mb_params.cmd = p_in_params->cmd;
mb_params.param = QED_RESC_ALLOC_VERSION;
+ mb_params.p_data_src = &mfw_resc_info;
+ mb_params.data_src_size = sizeof(mfw_resc_info);
+ mb_params.p_data_dst = mb_params.p_data_src;
+ mb_params.data_dst_size = mb_params.data_src_size;
- /* Need to have a sufficient large struct, as the cmd_and_union
- * is going to do memcpy from and to it.
- */
- memcpy(&union_data.resource, p_resc_info, sizeof(*p_resc_info));
+ DP_VERBOSE(p_hwfn,
+ QED_MSG_SP,
+ "Resource message request: cmd 0x%08x, res_id %d [%s], hsi_version %d.%d, val 0x%x\n",
+ p_in_params->cmd,
+ p_in_params->res_id,
+ qed_hw_get_resc_name(p_in_params->res_id),
+ QED_MFW_GET_FIELD(mb_params.param,
+ DRV_MB_PARAM_RESOURCE_ALLOC_VERSION_MAJOR),
+ QED_MFW_GET_FIELD(mb_params.param,
+ DRV_MB_PARAM_RESOURCE_ALLOC_VERSION_MINOR),
+ p_in_params->resc_max_val);
- mb_params.p_data_src = &union_data;
- mb_params.p_data_dst = &union_data;
rc = qed_mcp_cmd_and_union(p_hwfn, p_ptt, &mb_params);
if (rc)
return rc;
- /* Copy the data back */
- memcpy(p_resc_info, &union_data.resource, sizeof(*p_resc_info));
- *p_mcp_resp = mb_params.mcp_resp;
- *p_mcp_param = mb_params.mcp_param;
+ p_out_params->mcp_resp = mb_params.mcp_resp;
+ p_out_params->mcp_param = mb_params.mcp_param;
+ p_out_params->resc_num = mfw_resc_info.size;
+ p_out_params->resc_start = mfw_resc_info.offset;
+ p_out_params->vf_resc_num = mfw_resc_info.vf_size;
+ p_out_params->vf_resc_start = mfw_resc_info.vf_offset;
+ p_out_params->flags = mfw_resc_info.flags;
DP_VERBOSE(p_hwfn,
QED_MSG_SP,
- "MFW resource_info: version 0x%x, res_id 0x%x, size 0x%x, offset 0x%x, vf_size 0x%x, vf_offset 0x%x, flags 0x%x\n",
- *p_mcp_param,
- p_resc_info->res_id,
- p_resc_info->size,
- p_resc_info->offset,
- p_resc_info->vf_size,
- p_resc_info->vf_offset, p_resc_info->flags);
+ "Resource message response: mfw_hsi_version %d.%d, num 0x%x, start 0x%x, vf_num 0x%x, vf_start 0x%x, flags 0x%08x\n",
+ QED_MFW_GET_FIELD(p_out_params->mcp_param,
+ FW_MB_PARAM_RESOURCE_ALLOC_VERSION_MAJOR),
+ QED_MFW_GET_FIELD(p_out_params->mcp_param,
+ FW_MB_PARAM_RESOURCE_ALLOC_VERSION_MINOR),
+ p_out_params->resc_num,
+ p_out_params->resc_start,
+ p_out_params->vf_resc_num,
+ p_out_params->vf_resc_start, p_out_params->flags);
return 0;
}
+
+int
+qed_mcp_set_resc_max_val(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ enum qed_resources res_id,
+ u32 resc_max_val, u32 *p_mcp_resp)
+{
+ struct qed_resc_alloc_out_params out_params;
+ struct qed_resc_alloc_in_params in_params;
+ int rc;
+
+ memset(&in_params, 0, sizeof(in_params));
+ in_params.cmd = DRV_MSG_SET_RESOURCE_VALUE_MSG;
+ in_params.res_id = res_id;
+ in_params.resc_max_val = resc_max_val;
+ memset(&out_params, 0, sizeof(out_params));
+ rc = qed_mcp_resc_allocation_msg(p_hwfn, p_ptt, &in_params,
+ &out_params);
+ if (rc)
+ return rc;
+
+ *p_mcp_resp = out_params.mcp_resp;
+
+ return 0;
+}
+
+int
+qed_mcp_get_resc_info(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ enum qed_resources res_id,
+ u32 *p_mcp_resp, u32 *p_resc_num, u32 *p_resc_start)
+{
+ struct qed_resc_alloc_out_params out_params;
+ struct qed_resc_alloc_in_params in_params;
+ int rc;
+
+ memset(&in_params, 0, sizeof(in_params));
+ in_params.cmd = DRV_MSG_GET_RESOURCE_ALLOC_MSG;
+ in_params.res_id = res_id;
+ memset(&out_params, 0, sizeof(out_params));
+ rc = qed_mcp_resc_allocation_msg(p_hwfn, p_ptt, &in_params,
+ &out_params);
+ if (rc)
+ return rc;
+
+ *p_mcp_resp = out_params.mcp_resp;
+
+ if (*p_mcp_resp == FW_MSG_CODE_RESOURCE_ALLOC_OK) {
+ *p_resc_num = out_params.resc_num;
+ *p_resc_start = out_params.resc_start;
+ }
+
+ return 0;
+}
+
+int qed_mcp_initiate_pf_flr(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
+{
+ u32 mcp_resp, mcp_param;
+
+ return qed_mcp_cmd(p_hwfn, p_ptt, DRV_MSG_CODE_INITIATE_PF_FLR, 0,
+ &mcp_resp, &mcp_param);
+}
+
+static int qed_mcp_resource_cmd(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ u32 param, u32 *p_mcp_resp, u32 *p_mcp_param)
+{
+ int rc;
+
+ rc = qed_mcp_cmd(p_hwfn, p_ptt, DRV_MSG_CODE_RESOURCE_CMD, param,
+ p_mcp_resp, p_mcp_param);
+ if (rc)
+ return rc;
+
+ if (*p_mcp_resp == FW_MSG_CODE_UNSUPPORTED) {
+ DP_INFO(p_hwfn,
+ "The resource command is unsupported by the MFW\n");
+ return -EINVAL;
+ }
+
+ if (*p_mcp_param == RESOURCE_OPCODE_UNKNOWN_CMD) {
+ u8 opcode = QED_MFW_GET_FIELD(param, RESOURCE_CMD_REQ_OPCODE);
+
+ DP_NOTICE(p_hwfn,
+ "The resource command is unknown to the MFW [param 0x%08x, opcode %d]\n",
+ param, opcode);
+ return -EINVAL;
+ }
+
+ return rc;
+}
+
+int
+__qed_mcp_resc_lock(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ struct qed_resc_lock_params *p_params)
+{
+ u32 param = 0, mcp_resp, mcp_param;
+ u8 opcode;
+ int rc;
+
+ switch (p_params->timeout) {
+ case QED_MCP_RESC_LOCK_TO_DEFAULT:
+ opcode = RESOURCE_OPCODE_REQ;
+ p_params->timeout = 0;
+ break;
+ case QED_MCP_RESC_LOCK_TO_NONE:
+ opcode = RESOURCE_OPCODE_REQ_WO_AGING;
+ p_params->timeout = 0;
+ break;
+ default:
+ opcode = RESOURCE_OPCODE_REQ_W_AGING;
+ break;
+ }
+
+ QED_MFW_SET_FIELD(param, RESOURCE_CMD_REQ_RESC, p_params->resource);
+ QED_MFW_SET_FIELD(param, RESOURCE_CMD_REQ_OPCODE, opcode);
+ QED_MFW_SET_FIELD(param, RESOURCE_CMD_REQ_AGE, p_params->timeout);
+
+ DP_VERBOSE(p_hwfn,
+ QED_MSG_SP,
+ "Resource lock request: param 0x%08x [age %d, opcode %d, resource %d]\n",
+ param, p_params->timeout, opcode, p_params->resource);
+
+ /* Attempt to acquire the resource */
+ rc = qed_mcp_resource_cmd(p_hwfn, p_ptt, param, &mcp_resp, &mcp_param);
+ if (rc)
+ return rc;
+
+ /* Analyze the response */
+ p_params->owner = QED_MFW_GET_FIELD(mcp_param, RESOURCE_CMD_RSP_OWNER);
+ opcode = QED_MFW_GET_FIELD(mcp_param, RESOURCE_CMD_RSP_OPCODE);
+
+ DP_VERBOSE(p_hwfn,
+ QED_MSG_SP,
+ "Resource lock response: mcp_param 0x%08x [opcode %d, owner %d]\n",
+ mcp_param, opcode, p_params->owner);
+
+ switch (opcode) {
+ case RESOURCE_OPCODE_GNT:
+ p_params->b_granted = true;
+ break;
+ case RESOURCE_OPCODE_BUSY:
+ p_params->b_granted = false;
+ break;
+ default:
+ DP_NOTICE(p_hwfn,
+ "Unexpected opcode in resource lock response [mcp_param 0x%08x, opcode %d]\n",
+ mcp_param, opcode);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+int
+qed_mcp_resc_lock(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt, struct qed_resc_lock_params *p_params)
+{
+ u32 retry_cnt = 0;
+ int rc;
+
+ do {
+ /* No need for an interval before the first iteration */
+ if (retry_cnt) {
+ if (p_params->sleep_b4_retry) {
+ u16 retry_interval_in_ms =
+ DIV_ROUND_UP(p_params->retry_interval,
+ 1000);
+
+ msleep(retry_interval_in_ms);
+ } else {
+ udelay(p_params->retry_interval);
+ }
+ }
+
+ rc = __qed_mcp_resc_lock(p_hwfn, p_ptt, p_params);
+ if (rc)
+ return rc;
+
+ if (p_params->b_granted)
+ break;
+ } while (retry_cnt++ < p_params->retry_num);
+
+ return 0;
+}
+
+int
+qed_mcp_resc_unlock(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ struct qed_resc_unlock_params *p_params)
+{
+ u32 param = 0, mcp_resp, mcp_param;
+ u8 opcode;
+ int rc;
+
+ opcode = p_params->b_force ? RESOURCE_OPCODE_FORCE_RELEASE
+ : RESOURCE_OPCODE_RELEASE;
+ QED_MFW_SET_FIELD(param, RESOURCE_CMD_REQ_RESC, p_params->resource);
+ QED_MFW_SET_FIELD(param, RESOURCE_CMD_REQ_OPCODE, opcode);
+
+ DP_VERBOSE(p_hwfn, QED_MSG_SP,
+ "Resource unlock request: param 0x%08x [opcode %d, resource %d]\n",
+ param, opcode, p_params->resource);
+
+ /* Attempt to release the resource */
+ rc = qed_mcp_resource_cmd(p_hwfn, p_ptt, param, &mcp_resp, &mcp_param);
+ if (rc)
+ return rc;
+
+ /* Analyze the response */
+ opcode = QED_MFW_GET_FIELD(mcp_param, RESOURCE_CMD_RSP_OPCODE);
+
+ DP_VERBOSE(p_hwfn, QED_MSG_SP,
+ "Resource unlock response: mcp_param 0x%08x [opcode %d]\n",
+ mcp_param, opcode);
+
+ switch (opcode) {
+ case RESOURCE_OPCODE_RELEASED_PREVIOUS:
+ DP_INFO(p_hwfn,
+ "Resource unlock request for an already released resource [%d]\n",
+ p_params->resource);
+ /* Fallthrough */
+ case RESOURCE_OPCODE_RELEASED:
+ p_params->b_released = true;
+ break;
+ case RESOURCE_OPCODE_WRONG_OWNER:
+ p_params->b_released = false;
+ break;
+ default:
+ DP_NOTICE(p_hwfn,
+ "Unexpected opcode in resource unlock response [mcp_param 0x%08x, opcode %d]\n",
+ mcp_param, opcode);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+void qed_mcp_resc_lock_default_init(struct qed_resc_lock_params *p_lock,
+ struct qed_resc_unlock_params *p_unlock,
+ enum qed_resc_lock
+ resource, bool b_is_permanent)
+{
+ if (p_lock) {
+ memset(p_lock, 0, sizeof(*p_lock));
+
+ /* Permanent resources don't require aging, and there's no
+ * point in trying to acquire them more than once since it's
+ * unexpected another entity would release them.
+ */
+ if (b_is_permanent) {
+ p_lock->timeout = QED_MCP_RESC_LOCK_TO_NONE;
+ } else {
+ p_lock->retry_num = QED_MCP_RESC_LOCK_RETRY_CNT_DFLT;
+ p_lock->retry_interval =
+ QED_MCP_RESC_LOCK_RETRY_VAL_DFLT;
+ p_lock->sleep_b4_retry = true;
+ }
+
+ p_lock->resource = resource;
+ }
+
+ if (p_unlock) {
+ memset(p_unlock, 0, sizeof(*p_unlock));
+ p_unlock->resource = resource;
+ }
+}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_mcp.h b/drivers/net/ethernet/qlogic/qed/qed_mcp.h
index 368e88d..5ae35d6 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_mcp.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_mcp.h
@@ -39,6 +39,7 @@
#include <linux/spinlock.h>
#include <linux/qed/qed_fcoe_if.h>
#include "qed_hsi.h"
+#include "qed_dev_api.h"
struct qed_mcp_link_speed_params {
bool autoneg;
@@ -479,14 +480,18 @@
rel_pfid)
#define MCP_PF_ID(p_hwfn) MCP_PF_ID_BY_REL(p_hwfn, (p_hwfn)->rel_pf_id)
-/* TODO - this is only correct as long as only BB is supported, and
- * no port-swapping is implemented; Afterwards we'll need to fix it.
- */
-#define MFW_PORT(_p_hwfn) ((_p_hwfn)->abs_pf_id % \
- ((_p_hwfn)->cdev->num_ports_in_engines * 2))
+#define MFW_PORT(_p_hwfn) ((_p_hwfn)->abs_pf_id % \
+ ((_p_hwfn)->cdev->num_ports_in_engines * \
+ qed_device_num_engines((_p_hwfn)->cdev)))
+
struct qed_mcp_info {
- /* Spinlock used for protecting the access to the MFW mailbox */
- spinlock_t lock;
+ /* List for mailbox commands which were sent and wait for a response */
+ struct list_head cmd_list;
+
+ /* Spinlock used for protecting the access to the mailbox commands list
+ * and the sending of the commands.
+ */
+ spinlock_t cmd_lock;
/* Spinlock used for syncing SW link-changes and link-changes
* originating from attention context.
@@ -506,14 +511,16 @@
u8 *mfw_mb_cur;
u8 *mfw_mb_shadow;
u16 mfw_mb_length;
- u16 mcp_hist;
+ u32 mcp_hist;
};
struct qed_mcp_mb_params {
u32 cmd;
u32 param;
- union drv_union_data *p_data_src;
- union drv_union_data *p_data_dst;
+ void *p_data_src;
+ u8 data_src_size;
+ void *p_data_dst;
+ u8 data_dst_size;
u32 mcp_resp;
u32 mcp_param;
};
@@ -564,27 +571,55 @@
int qed_mcp_handle_events(struct qed_hwfn *p_hwfn,
struct qed_ptt *p_ptt);
+enum qed_drv_role {
+ QED_DRV_ROLE_OS,
+ QED_DRV_ROLE_KDUMP,
+};
+
+struct qed_load_req_params {
+ /* Input params */
+ enum qed_drv_role drv_role;
+ u8 timeout_val;
+ bool avoid_eng_reset;
+ enum qed_override_force_load override_force_load;
+
+ /* Output params */
+ u32 load_code;
+};
+
/**
- * @brief Sends a LOAD_REQ to the MFW, and in case operation
- * succeed, returns whether this PF is the first on the
- * chip/engine/port or function. This function should be
- * called when driver is ready to accept MFW events after
- * Storms initializations are done.
+ * @brief Sends a LOAD_REQ to the MFW, and in case the operation succeeds,
+ * returns whether this PF is the first on the engine/port or function.
*
- * @param p_hwfn - hw function
- * @param p_ptt - PTT required for register access
- * @param p_load_code - The MCP response param containing one
- * of the following:
- * FW_MSG_CODE_DRV_LOAD_ENGINE
- * FW_MSG_CODE_DRV_LOAD_PORT
- * FW_MSG_CODE_DRV_LOAD_FUNCTION
- * @return int -
- * 0 - Operation was successul.
- * -EBUSY - Operation failed
+ * @param p_hwfn
+ * @param p_ptt
+ * @param p_params
+ *
+ * @return int - 0 - Operation was successful.
*/
int qed_mcp_load_req(struct qed_hwfn *p_hwfn,
struct qed_ptt *p_ptt,
- u32 *p_load_code);
+ struct qed_load_req_params *p_params);
+
+/**
+ * @brief Sends a UNLOAD_REQ message to the MFW
+ *
+ * @param p_hwfn
+ * @param p_ptt
+ *
+ * @return int - 0 - Operation was successful.
+ */
+int qed_mcp_unload_req(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt);
+
+/**
+ * @brief Sends a UNLOAD_DONE message to the MFW
+ *
+ * @param p_hwfn
+ * @param p_ptt
+ *
+ * @return int - 0 - Operation was successful.
+ */
+int qed_mcp_unload_done(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt);
/**
* @brief Read the MFW mailbox into Current buffer.
@@ -708,6 +743,41 @@
struct qed_ptt *p_ptt, u32 mask_parities);
/**
+ * @brief - Sets the MFW's max value for the given resource
+ *
+ * @param p_hwfn
+ * @param p_ptt
+ * @param res_id
+ * @param resc_max_val
+ * @param p_mcp_resp
+ *
+ * @return int - 0 - operation was successful.
+ */
+int
+qed_mcp_set_resc_max_val(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ enum qed_resources res_id,
+ u32 resc_max_val, u32 *p_mcp_resp);
+
+/**
+ * @brief - Gets the MFW allocation info for the given resource
+ *
+ * @param p_hwfn
+ * @param p_ptt
+ * @param res_id
+ * @param p_mcp_resp
+ * @param p_resc_num
+ * @param p_resc_start
+ *
+ * @return int - 0 - operation was successful.
+ */
+int
+qed_mcp_get_resc_info(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ enum qed_resources res_id,
+ u32 *p_mcp_resp, u32 *p_resc_num, u32 *p_resc_start);
+
+/**
* @brief Send eswitch mode to MFW
*
* @param p_hwfn
@@ -720,19 +790,106 @@
struct qed_ptt *p_ptt,
enum qed_ov_eswitch eswitch);
+#define QED_MCP_RESC_LOCK_MIN_VAL RESOURCE_DUMP
+#define QED_MCP_RESC_LOCK_MAX_VAL 31
+
+enum qed_resc_lock {
+ QED_RESC_LOCK_DBG_DUMP = QED_MCP_RESC_LOCK_MIN_VAL,
+ QED_RESC_LOCK_PTP_PORT0,
+ QED_RESC_LOCK_PTP_PORT1,
+ QED_RESC_LOCK_PTP_PORT2,
+ QED_RESC_LOCK_PTP_PORT3,
+ QED_RESC_LOCK_RESC_ALLOC = QED_MCP_RESC_LOCK_MAX_VAL,
+ QED_RESC_LOCK_RESC_INVALID
+};
+
/**
- * @brief - Gets the MFW allocation info for the given resource
+ * @brief - Initiates PF FLR
*
* @param p_hwfn
* @param p_ptt
- * @param p_resc_info - descriptor of requested resource
- * @param p_mcp_resp
- * @param p_mcp_param
*
* @return int - 0 - operation was successful.
*/
-int qed_mcp_get_resc_info(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt,
- struct resource_info *p_resc_info,
- u32 *p_mcp_resp, u32 *p_mcp_param);
+int qed_mcp_initiate_pf_flr(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt);
+struct qed_resc_lock_params {
+ /* Resource number [valid values are 0..31] */
+ u8 resource;
+
+ /* Lock timeout value in seconds [default, none or 1..254] */
+ u8 timeout;
+#define QED_MCP_RESC_LOCK_TO_DEFAULT 0
+#define QED_MCP_RESC_LOCK_TO_NONE 255
+
+ /* Number of times to retry locking */
+ u8 retry_num;
+#define QED_MCP_RESC_LOCK_RETRY_CNT_DFLT 10
+
+ /* The interval in usec between retries */
+ u16 retry_interval;
+#define QED_MCP_RESC_LOCK_RETRY_VAL_DFLT 10000
+
+ /* Use sleep or delay between retries */
+ bool sleep_b4_retry;
+
+ /* Will be set as true if the resource is free and granted */
+ bool b_granted;
+
+ /* Will be filled with the resource owner.
+ * [0..15 = PF0-15, 16 = MFW]
+ */
+ u8 owner;
+};
+
+/**
+ * @brief Acquires MFW generic resource lock
+ *
+ * @param p_hwfn
+ * @param p_ptt
+ * @param p_params
+ *
+ * @return int - 0 - operation was successful.
+ */
+int
+qed_mcp_resc_lock(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt, struct qed_resc_lock_params *p_params);
+
+struct qed_resc_unlock_params {
+ /* Resource number [valid values are 0..31] */
+ u8 resource;
+
+ /* Allow to release a resource even if belongs to another PF */
+ bool b_force;
+
+ /* Will be set as true if the resource is released */
+ bool b_released;
+};
+
+/**
+ * @brief Releases MFW generic resource lock
+ *
+ * @param p_hwfn
+ * @param p_ptt
+ * @param p_params
+ *
+ * @return int - 0 - operation was successful.
+ */
+int
+qed_mcp_resc_unlock(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ struct qed_resc_unlock_params *p_params);
+
+/**
+ * @brief - default initialization for lock/unlock resource structs
+ *
+ * @param p_lock - lock params struct to be initialized; Can be NULL
+ * @param p_unlock - unlock params struct to be initialized; Can be NULL
+ * @param resource - the requested resource
+ * @paral b_is_permanent - disable retries & aging when set
+ */
+void qed_mcp_resc_lock_default_init(struct qed_resc_lock_params *p_lock,
+ struct qed_resc_unlock_params *p_unlock,
+ enum qed_resc_lock
+ resource, bool b_is_permanent);
+
#endif
diff --git a/drivers/net/ethernet/qlogic/qed/qed_ooo.c b/drivers/net/ethernet/qlogic/qed/qed_ooo.c
index 378afce..db96670 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_ooo.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_ooo.c
@@ -41,6 +41,7 @@
#include "qed_iscsi.h"
#include "qed_ll2.h"
#include "qed_ooo.h"
+#include "qed_cxt.h"
static struct qed_ooo_archipelago
*qed_ooo_seek_archipelago(struct qed_hwfn *p_hwfn,
@@ -48,15 +49,18 @@
*p_ooo_info,
u32 cid)
{
- struct qed_ooo_archipelago *p_archipelago = NULL;
+ u32 idx = (cid & 0xffff) - p_ooo_info->cid_base;
+ struct qed_ooo_archipelago *p_archipelago;
- list_for_each_entry(p_archipelago,
- &p_ooo_info->archipelagos_list, list_entry) {
- if (p_archipelago->cid == cid)
- return p_archipelago;
- }
+ if (idx >= p_ooo_info->max_num_archipelagos)
+ return NULL;
- return NULL;
+ p_archipelago = &p_ooo_info->p_archipelagos_mem[idx];
+
+ if (list_empty(&p_archipelago->isles_list))
+ return NULL;
+
+ return p_archipelago;
}
static struct qed_ooo_isle *qed_ooo_seek_isle(struct qed_hwfn *p_hwfn,
@@ -97,8 +101,8 @@
struct qed_ooo_info *qed_ooo_alloc(struct qed_hwfn *p_hwfn)
{
+ u16 max_num_archipelagos = 0, cid_base;
struct qed_ooo_info *p_ooo_info;
- u16 max_num_archipelagos = 0;
u16 max_num_isles = 0;
u32 i;
@@ -110,6 +114,7 @@
max_num_archipelagos = p_hwfn->pf_params.iscsi_pf_params.num_cons;
max_num_isles = QED_MAX_NUM_ISLES + max_num_archipelagos;
+ cid_base = (u16)qed_cxt_get_proto_cid_start(p_hwfn, PROTOCOLID_ISCSI);
if (!max_num_archipelagos) {
DP_NOTICE(p_hwfn,
@@ -121,11 +126,12 @@
if (!p_ooo_info)
return NULL;
+ p_ooo_info->cid_base = cid_base;
+ p_ooo_info->max_num_archipelagos = max_num_archipelagos;
+
INIT_LIST_HEAD(&p_ooo_info->free_buffers_list);
INIT_LIST_HEAD(&p_ooo_info->ready_buffers_list);
INIT_LIST_HEAD(&p_ooo_info->free_isles_list);
- INIT_LIST_HEAD(&p_ooo_info->free_archipelagos_list);
- INIT_LIST_HEAD(&p_ooo_info->archipelagos_list);
p_ooo_info->p_isles_mem = kcalloc(max_num_isles,
sizeof(struct qed_ooo_isle),
@@ -146,11 +152,8 @@
if (!p_ooo_info->p_archipelagos_mem)
goto no_archipelagos_mem;
- for (i = 0; i < max_num_archipelagos; i++) {
+ for (i = 0; i < max_num_archipelagos; i++)
INIT_LIST_HEAD(&p_ooo_info->p_archipelagos_mem[i].isles_list);
- list_add_tail(&p_ooo_info->p_archipelagos_mem[i].list_entry,
- &p_ooo_info->free_archipelagos_list);
- }
p_ooo_info->ooo_history.p_cqes =
kcalloc(QED_MAX_NUM_OOO_HISTORY_ENTRIES,
@@ -178,21 +181,9 @@
struct qed_ooo_archipelago *p_archipelago;
struct qed_ooo_buffer *p_buffer;
struct qed_ooo_isle *p_isle;
- bool b_found = false;
- if (list_empty(&p_ooo_info->archipelagos_list))
- return;
-
- list_for_each_entry(p_archipelago,
- &p_ooo_info->archipelagos_list, list_entry) {
- if (p_archipelago->cid == cid) {
- list_del(&p_archipelago->list_entry);
- b_found = true;
- break;
- }
- }
-
- if (!b_found)
+ p_archipelago = qed_ooo_seek_archipelago(p_hwfn, p_ooo_info, cid);
+ if (!p_archipelago)
return;
while (!list_empty(&p_archipelago->isles_list)) {
@@ -216,27 +207,21 @@
list_add_tail(&p_isle->list_entry,
&p_ooo_info->free_isles_list);
}
-
- list_add_tail(&p_archipelago->list_entry,
- &p_ooo_info->free_archipelagos_list);
}
void qed_ooo_release_all_isles(struct qed_hwfn *p_hwfn,
struct qed_ooo_info *p_ooo_info)
{
- struct qed_ooo_archipelago *p_arch;
+ struct qed_ooo_archipelago *p_archipelago;
struct qed_ooo_buffer *p_buffer;
struct qed_ooo_isle *p_isle;
+ u32 i;
- while (!list_empty(&p_ooo_info->archipelagos_list)) {
- p_arch = list_first_entry(&p_ooo_info->archipelagos_list,
- struct qed_ooo_archipelago,
- list_entry);
+ for (i = 0; i < p_ooo_info->max_num_archipelagos; i++) {
+ p_archipelago = &(p_ooo_info->p_archipelagos_mem[i]);
- list_del(&p_arch->list_entry);
-
- while (!list_empty(&p_arch->isles_list)) {
- p_isle = list_first_entry(&p_arch->isles_list,
+ while (!list_empty(&p_archipelago->isles_list)) {
+ p_isle = list_first_entry(&p_archipelago->isles_list,
struct qed_ooo_isle,
list_entry);
@@ -258,8 +243,6 @@
list_add_tail(&p_isle->list_entry,
&p_ooo_info->free_isles_list);
}
- list_add_tail(&p_arch->list_entry,
- &p_ooo_info->free_archipelagos_list);
}
if (!list_empty(&p_ooo_info->ready_buffers_list))
list_splice_tail_init(&p_ooo_info->ready_buffers_list,
@@ -378,12 +361,6 @@
p_ooo_info->cur_isles_number--;
list_add(&p_isle->list_entry, &p_ooo_info->free_isles_list);
}
-
- if (list_empty(&p_archipelago->isles_list)) {
- list_del(&p_archipelago->list_entry);
- list_add(&p_archipelago->list_entry,
- &p_ooo_info->free_archipelagos_list);
- }
}
void qed_ooo_add_new_isle(struct qed_hwfn *p_hwfn,
@@ -426,28 +403,10 @@
return;
}
- if (!p_archipelago &&
- !list_empty(&p_ooo_info->free_archipelagos_list)) {
- p_archipelago =
- list_first_entry(&p_ooo_info->free_archipelagos_list,
- struct qed_ooo_archipelago, list_entry);
+ if (!p_archipelago) {
+ u32 idx = (cid & 0xffff) - p_ooo_info->cid_base;
- list_del(&p_archipelago->list_entry);
- if (!list_empty(&p_archipelago->isles_list)) {
- DP_NOTICE(p_hwfn,
- "Free OOO connection is not empty\n");
- INIT_LIST_HEAD(&p_archipelago->isles_list);
- }
- p_archipelago->cid = cid;
- list_add(&p_archipelago->list_entry,
- &p_ooo_info->archipelagos_list);
- } else if (!p_archipelago) {
- DP_NOTICE(p_hwfn, "No more free OOO connections\n");
- list_add(&p_isle->list_entry,
- &p_ooo_info->free_isles_list);
- list_add(&p_buffer->list_entry,
- &p_ooo_info->free_buffers_list);
- return;
+ p_archipelago = &p_ooo_info->p_archipelagos_mem[idx];
}
list_add(&p_buffer->list_entry, &p_isle->buffers_list);
@@ -517,11 +476,6 @@
} else {
list_splice_tail_init(&p_right_isle->buffers_list,
&p_ooo_info->ready_buffers_list);
- if (list_empty(&p_archipelago->isles_list)) {
- list_del(&p_archipelago->list_entry);
- list_add(&p_archipelago->list_entry,
- &p_ooo_info->free_archipelagos_list);
- }
}
list_add_tail(&p_right_isle->list_entry, &p_ooo_info->free_isles_list);
}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_ooo.h b/drivers/net/ethernet/qlogic/qed/qed_ooo.h
index 4f138fb..791ad0f 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_ooo.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_ooo.h
@@ -60,9 +60,7 @@
};
struct qed_ooo_archipelago {
- struct list_head list_entry;
struct list_head isles_list;
- u32 cid;
};
struct qed_ooo_history {
@@ -75,14 +73,14 @@
struct list_head free_buffers_list;
struct list_head ready_buffers_list;
struct list_head free_isles_list;
- struct list_head free_archipelagos_list;
- struct list_head archipelagos_list;
struct qed_ooo_archipelago *p_archipelagos_mem;
struct qed_ooo_isle *p_isles_mem;
struct qed_ooo_history ooo_history;
u32 cur_isles_number;
u32 max_isles_number;
u32 gen_isles_number;
+ u16 max_num_archipelagos;
+ u16 cid_base;
};
#if IS_ENABLED(CONFIG_QED_ISCSI)
diff --git a/drivers/net/ethernet/qlogic/qed/qed_ptp.c b/drivers/net/ethernet/qlogic/qed/qed_ptp.c
index d27aa85..434a164 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_ptp.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_ptp.c
@@ -34,7 +34,7 @@
#include "qed_dev_api.h"
#include "qed_hw.h"
#include "qed_l2.h"
-#include "qed_ptp.h"
+#include "qed_mcp.h"
#include "qed_reg_addr.h"
/* 16 nano second time quantas to wait before making a Drift adjustment */
@@ -45,6 +45,82 @@
#define QED_DRIFT_CNTR_DIRECTION_SHIFT 31
#define QED_TIMESTAMP_MASK BIT(16)
+static enum qed_resc_lock qed_ptcdev_to_resc(struct qed_hwfn *p_hwfn)
+{
+ switch (qed_device_get_port_id(p_hwfn->cdev)) {
+ case 0:
+ return QED_RESC_LOCK_PTP_PORT0;
+ case 1:
+ return QED_RESC_LOCK_PTP_PORT1;
+ case 2:
+ return QED_RESC_LOCK_PTP_PORT2;
+ case 3:
+ return QED_RESC_LOCK_PTP_PORT3;
+ default:
+ return QED_RESC_LOCK_RESC_INVALID;
+ }
+}
+
+static int qed_ptp_res_lock(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
+{
+ struct qed_resc_lock_params params;
+ enum qed_resc_lock resource;
+ int rc;
+
+ resource = qed_ptcdev_to_resc(p_hwfn);
+ if (resource == QED_RESC_LOCK_RESC_INVALID)
+ return -EINVAL;
+
+ qed_mcp_resc_lock_default_init(¶ms, NULL, resource, true);
+
+ rc = qed_mcp_resc_lock(p_hwfn, p_ptt, ¶ms);
+ if (rc && rc != -EINVAL) {
+ return rc;
+ } else if (rc == -EINVAL) {
+ /* MFW doesn't support resource locking, first PF on the port
+ * has lock ownership.
+ */
+ if (p_hwfn->abs_pf_id < p_hwfn->cdev->num_ports_in_engines)
+ return 0;
+
+ DP_INFO(p_hwfn, "PF doesn't have lock ownership\n");
+ return -EBUSY;
+ } else if (!rc && !params.b_granted) {
+ DP_INFO(p_hwfn, "Failed to acquire ptp resource lock\n");
+ return -EBUSY;
+ }
+
+ return rc;
+}
+
+static int qed_ptp_res_unlock(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
+{
+ struct qed_resc_unlock_params params;
+ enum qed_resc_lock resource;
+ int rc;
+
+ resource = qed_ptcdev_to_resc(p_hwfn);
+ if (resource == QED_RESC_LOCK_RESC_INVALID)
+ return -EINVAL;
+
+ qed_mcp_resc_lock_default_init(NULL, ¶ms, resource, true);
+
+ rc = qed_mcp_resc_unlock(p_hwfn, p_ptt, ¶ms);
+ if (rc == -EINVAL) {
+ /* MFW doesn't support locking, first PF has lock ownership */
+ if (p_hwfn->abs_pf_id < p_hwfn->cdev->num_ports_in_engines) {
+ rc = 0;
+ } else {
+ DP_INFO(p_hwfn, "PF doesn't have lock ownership\n");
+ return -EINVAL;
+ }
+ } else if (rc) {
+ DP_INFO(p_hwfn, "Failed to release the ptp resource lock\n");
+ }
+
+ return rc;
+}
+
/* Read Rx timestamp */
static int qed_ptp_hw_read_rx_ts(struct qed_dev *cdev, u64 *timestamp)
{
@@ -112,39 +188,73 @@
}
/* Filter PTP protocol packets that need to be timestamped */
-static int qed_ptp_hw_cfg_rx_filters(struct qed_dev *cdev,
- enum qed_ptp_filter_type type)
+static int qed_ptp_hw_cfg_filters(struct qed_dev *cdev,
+ enum qed_ptp_filter_type rx_type,
+ enum qed_ptp_hwtstamp_tx_type tx_type)
{
struct qed_hwfn *p_hwfn = QED_LEADING_HWFN(cdev);
struct qed_ptt *p_ptt = p_hwfn->p_ptp_ptt;
- u32 rule_mask, parm_mask;
+ u32 rule_mask, enable_cfg = 0x0;
- switch (type) {
- case QED_PTP_FILTER_L2_IPV4_IPV6:
- parm_mask = 0x6AA;
- rule_mask = 0x3EEE;
+ switch (rx_type) {
+ case QED_PTP_FILTER_NONE:
+ enable_cfg = 0x0;
+ rule_mask = 0x3FFF;
break;
- case QED_PTP_FILTER_L2:
- parm_mask = 0x6BF;
+ case QED_PTP_FILTER_ALL:
+ enable_cfg = 0x7;
+ rule_mask = 0x3CAA;
+ break;
+ case QED_PTP_FILTER_V1_L4_EVENT:
+ enable_cfg = 0x3;
+ rule_mask = 0x3FFA;
+ break;
+ case QED_PTP_FILTER_V1_L4_GEN:
+ enable_cfg = 0x3;
+ rule_mask = 0x3FFE;
+ break;
+ case QED_PTP_FILTER_V2_L4_EVENT:
+ enable_cfg = 0x5;
+ rule_mask = 0x3FAA;
+ break;
+ case QED_PTP_FILTER_V2_L4_GEN:
+ enable_cfg = 0x5;
+ rule_mask = 0x3FEE;
+ break;
+ case QED_PTP_FILTER_V2_L2_EVENT:
+ enable_cfg = 0x5;
+ rule_mask = 0x3CFF;
+ break;
+ case QED_PTP_FILTER_V2_L2_GEN:
+ enable_cfg = 0x5;
rule_mask = 0x3EFF;
break;
- case QED_PTP_FILTER_IPV4_IPV6:
- parm_mask = 0x7EA;
- rule_mask = 0x3FFE;
+ case QED_PTP_FILTER_V2_EVENT:
+ enable_cfg = 0x5;
+ rule_mask = 0x3CAA;
break;
- case QED_PTP_FILTER_IPV4:
- parm_mask = 0x7EE;
- rule_mask = 0x3FFE;
+ case QED_PTP_FILTER_V2_GEN:
+ enable_cfg = 0x5;
+ rule_mask = 0x3EEE;
break;
default:
- DP_INFO(p_hwfn, "Invalid PTP filter type %d\n", type);
+ DP_INFO(p_hwfn, "Invalid PTP filter type %d\n", rx_type);
return -EINVAL;
}
- qed_wr(p_hwfn, p_ptt, NIG_REG_LLH_PTP_PARAM_MASK, parm_mask);
+ qed_wr(p_hwfn, p_ptt, NIG_REG_LLH_PTP_PARAM_MASK, 0);
qed_wr(p_hwfn, p_ptt, NIG_REG_LLH_PTP_RULE_MASK, rule_mask);
+ qed_wr(p_hwfn, p_ptt, NIG_REG_RX_PTP_EN, enable_cfg);
- qed_wr(p_hwfn, p_ptt, NIG_REG_LLH_PTP_TO_HOST, 0x1);
+ if (tx_type == QED_PTP_HWTSTAMP_TX_OFF) {
+ qed_wr(p_hwfn, p_ptt, NIG_REG_TX_PTP_EN, 0x0);
+ qed_wr(p_hwfn, p_ptt, NIG_REG_TX_LLH_PTP_PARAM_MASK, 0x7FF);
+ qed_wr(p_hwfn, p_ptt, NIG_REG_TX_LLH_PTP_RULE_MASK, 0x3FFF);
+ } else {
+ qed_wr(p_hwfn, p_ptt, NIG_REG_TX_PTP_EN, enable_cfg);
+ qed_wr(p_hwfn, p_ptt, NIG_REG_TX_LLH_PTP_PARAM_MASK, 0);
+ qed_wr(p_hwfn, p_ptt, NIG_REG_TX_LLH_PTP_RULE_MASK, rule_mask);
+ }
/* Reset possibly old timestamps */
qed_wr(p_hwfn, p_ptt, NIG_REG_LLH_PTP_HOST_BUF_SEQID,
@@ -248,7 +358,25 @@
static int qed_ptp_hw_enable(struct qed_dev *cdev)
{
struct qed_hwfn *p_hwfn = QED_LEADING_HWFN(cdev);
- struct qed_ptt *p_ptt = p_hwfn->p_ptp_ptt;
+ struct qed_ptt *p_ptt;
+ int rc;
+
+ p_ptt = qed_ptt_acquire(p_hwfn);
+ if (!p_ptt) {
+ DP_NOTICE(p_hwfn, "Failed to acquire PTT for PTP\n");
+ return -EBUSY;
+ }
+
+ p_hwfn->p_ptp_ptt = p_ptt;
+
+ rc = qed_ptp_res_lock(p_hwfn, p_ptt);
+ if (rc) {
+ DP_INFO(p_hwfn,
+ "Couldn't acquire the resource lock, skip ptp enable for this PF\n");
+ qed_ptt_release(p_hwfn, p_ptt);
+ p_hwfn->p_ptp_ptt = NULL;
+ return rc;
+ }
/* Reset PTP event detection rules - will be configured in the IOCTL */
qed_wr(p_hwfn, p_ptt, NIG_REG_LLH_PTP_PARAM_MASK, 0x7FF);
@@ -262,12 +390,20 @@
qed_wr(p_hwfn, p_ptt, NIG_REG_TS_OUTPUT_ENABLE_PDA, 0x1);
/* Pause free running counter */
- qed_wr(p_hwfn, p_ptt, NIG_REG_TIMESYNC_GEN_REG_BB, 2);
+ if (QED_IS_BB_B0(p_hwfn->cdev))
+ qed_wr(p_hwfn, p_ptt, NIG_REG_TIMESYNC_GEN_REG_BB, 2);
+ if (QED_IS_AH(p_hwfn->cdev))
+ qed_wr(p_hwfn, p_ptt, NIG_REG_TSGEN_FREECNT_UPDATE_K2, 2);
qed_wr(p_hwfn, p_ptt, NIG_REG_TSGEN_FREE_CNT_VALUE_LSB, 0);
qed_wr(p_hwfn, p_ptt, NIG_REG_TSGEN_FREE_CNT_VALUE_MSB, 0);
/* Resume free running counter */
- qed_wr(p_hwfn, p_ptt, NIG_REG_TIMESYNC_GEN_REG_BB, 4);
+ if (QED_IS_BB_B0(p_hwfn->cdev))
+ qed_wr(p_hwfn, p_ptt, NIG_REG_TIMESYNC_GEN_REG_BB, 4);
+ if (QED_IS_AH(p_hwfn->cdev)) {
+ qed_wr(p_hwfn, p_ptt, NIG_REG_TSGEN_FREECNT_UPDATE_K2, 4);
+ qed_wr(p_hwfn, p_ptt, NIG_REG_PTP_LATCH_OSTS_PKT_TIME, 1);
+ }
/* Disable drift register */
qed_wr(p_hwfn, p_ptt, NIG_REG_TSGEN_DRIFT_CNTR_CONF, 0x0);
@@ -281,22 +417,13 @@
return 0;
}
-static int qed_ptp_hw_hwtstamp_tx_on(struct qed_dev *cdev)
-{
- struct qed_hwfn *p_hwfn = QED_LEADING_HWFN(cdev);
- struct qed_ptt *p_ptt = p_hwfn->p_ptp_ptt;
-
- qed_wr(p_hwfn, p_ptt, NIG_REG_TX_LLH_PTP_PARAM_MASK, 0x6AA);
- qed_wr(p_hwfn, p_ptt, NIG_REG_TX_LLH_PTP_RULE_MASK, 0x3EEE);
-
- return 0;
-}
-
static int qed_ptp_hw_disable(struct qed_dev *cdev)
{
struct qed_hwfn *p_hwfn = QED_LEADING_HWFN(cdev);
struct qed_ptt *p_ptt = p_hwfn->p_ptp_ptt;
+ qed_ptp_res_unlock(p_hwfn, p_ptt);
+
/* Reset PTP event detection rules */
qed_wr(p_hwfn, p_ptt, NIG_REG_LLH_PTP_PARAM_MASK, 0x7FF);
qed_wr(p_hwfn, p_ptt, NIG_REG_LLH_PTP_RULE_MASK, 0x3FFF);
@@ -308,12 +435,14 @@
qed_wr(p_hwfn, p_ptt, NIG_REG_RX_PTP_EN, 0x0);
qed_wr(p_hwfn, p_ptt, NIG_REG_TX_PTP_EN, 0x0);
+ qed_ptt_release(p_hwfn, p_ptt);
+ p_hwfn->p_ptp_ptt = NULL;
+
return 0;
}
const struct qed_eth_ptp_ops qed_ptp_ops_pass = {
- .hwtstamp_tx_on = qed_ptp_hw_hwtstamp_tx_on,
- .cfg_rx_filters = qed_ptp_hw_cfg_rx_filters,
+ .cfg_filters = qed_ptp_hw_cfg_filters,
.read_rx_ts = qed_ptp_hw_read_rx_ts,
.read_tx_ts = qed_ptp_hw_read_tx_ts,
.read_cc = qed_ptp_hw_read_cc,
diff --git a/drivers/net/ethernet/qlogic/qed/qed_ptp.h b/drivers/net/ethernet/qlogic/qed/qed_ptp.h
deleted file mode 100644
index 63c666d..0000000
--- a/drivers/net/ethernet/qlogic/qed/qed_ptp.h
+++ /dev/null
@@ -1,47 +0,0 @@
-/* QLogic qed NIC Driver
- * Copyright (c) 2015-2017 QLogic Corporation
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and /or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-#ifndef _QED_PTP_H
-#define _QED_PTP_H
-#include <linux/types.h>
-
-int qed_ptp_hwtstamp_tx_on(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt);
-int qed_ptp_cfg_rx_filters(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt,
- enum qed_ptp_filter_type type);
-int qed_ptp_read_rx_ts(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt, u64 *ts);
-int qed_ptp_read_tx_ts(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt, u64 *ts);
-int qed_ptp_read_cc(struct qed_hwfn *p_hwfn,
- struct qed_ptt *p_ptt, u64 *cycles);
-int qed_ptp_adjfreq(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt, s32 ppb);
-int qed_ptp_disable(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt);
-int qed_ptp_enable(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt);
-
-#endif
diff --git a/drivers/net/ethernet/qlogic/qed/qed_reg_addr.h b/drivers/net/ethernet/qlogic/qed/qed_reg_addr.h
index d59d9df..1ae73b2 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_reg_addr.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_reg_addr.h
@@ -160,13 +160,13 @@
0x2e0704UL
#define CCFC_REG_STRONG_ENABLE_PF \
0x2e0708UL
-#define PGLUE_B_REG_PGL_ADDR_88_F0 \
+#define PGLUE_B_REG_PGL_ADDR_88_F0_BB \
0x2aa404UL
-#define PGLUE_B_REG_PGL_ADDR_8C_F0 \
+#define PGLUE_B_REG_PGL_ADDR_8C_F0_BB \
0x2aa408UL
-#define PGLUE_B_REG_PGL_ADDR_90_F0 \
+#define PGLUE_B_REG_PGL_ADDR_90_F0_BB \
0x2aa40cUL
-#define PGLUE_B_REG_PGL_ADDR_94_F0 \
+#define PGLUE_B_REG_PGL_ADDR_94_F0_BB \
0x2aa410UL
#define PGLUE_B_REG_WAS_ERROR_PF_31_0_CLR \
0x2aa138UL
@@ -356,6 +356,10 @@
0x238804UL
#define RDIF_REG_STOP_ON_ERROR \
0x300040UL
+#define RDIF_REG_DEBUG_ERROR_INFO \
+ 0x300400UL
+#define RDIF_REG_DEBUG_ERROR_INFO_SIZE \
+ 64
#define SRC_REG_SOFT_RST \
0x23874cUL
#define TCFC_REG_ACTIVITY_COUNTER \
@@ -370,6 +374,10 @@
0x1700004UL
#define TDIF_REG_STOP_ON_ERROR \
0x310040UL
+#define TDIF_REG_DEBUG_ERROR_INFO \
+ 0x310400UL
+#define TDIF_REG_DEBUG_ERROR_INFO_SIZE \
+ 64
#define UCM_REG_INIT \
0x1280000UL
#define UMAC_REG_IPG_HD_BKP_CNTL_BB_B0 \
@@ -1236,6 +1244,26 @@
0x1901534UL
#define USEM_REG_DBG_FORCE_FRAME \
0x1901538UL
+#define NWS_REG_DBG_SELECT \
+ 0x700128UL
+#define NWS_REG_DBG_DWORD_ENABLE \
+ 0x70012cUL
+#define NWS_REG_DBG_SHIFT \
+ 0x700130UL
+#define NWS_REG_DBG_FORCE_VALID \
+ 0x700134UL
+#define NWS_REG_DBG_FORCE_FRAME \
+ 0x700138UL
+#define MS_REG_DBG_SELECT \
+ 0x6a0228UL
+#define MS_REG_DBG_DWORD_ENABLE \
+ 0x6a022cUL
+#define MS_REG_DBG_SHIFT \
+ 0x6a0230UL
+#define MS_REG_DBG_FORCE_VALID \
+ 0x6a0234UL
+#define MS_REG_DBG_FORCE_FRAME \
+ 0x6a0238UL
#define PCIE_REG_DBG_COMMON_SELECT \
0x054398UL
#define PCIE_REG_DBG_COMMON_DWORD_ENABLE \
@@ -1448,6 +1476,8 @@
0x000b48UL
#define RSS_REG_RSS_RAM_DATA \
0x238c20UL
+#define RSS_REG_RSS_RAM_DATA_SIZE \
+ 4
#define MISC_REG_BLOCK_256B_EN \
0x008c14UL
#define NWS_REG_NWS_CMU \
@@ -1520,4 +1550,22 @@
#define NIG_REG_TIMESYNC_GEN_REG_BB 0x500d00UL
#define NIG_REG_TSGEN_FREE_CNT_VALUE_LSB 0x5088a8UL
#define NIG_REG_TSGEN_FREE_CNT_VALUE_MSB 0x5088acUL
+#define NIG_REG_PTP_LATCH_OSTS_PKT_TIME 0x509040UL
+#define PSWRQ2_REG_WR_MBS0 0x240400UL
+
+#define PGLUE_B_REG_PGL_ADDR_E8_F0_K2 0x2aaf98UL
+#define PGLUE_B_REG_PGL_ADDR_EC_F0_K2 0x2aaf9cUL
+#define PGLUE_B_REG_PGL_ADDR_F0_F0_K2 0x2aafa0UL
+#define PGLUE_B_REG_PGL_ADDR_F4_F0_K2 0x2aafa4UL
+#define NIG_REG_TSGEN_FREECNT_UPDATE_K2 0x509008UL
+#define CNIG_REG_NIG_PORT0_CONF_K2 0x218200UL
+
+#define PRS_REG_SEARCH_GFT 0x1f11bcUL
+#define PRS_REG_CM_HDR_GFT 0x1f11c8UL
+#define PRS_REG_GFT_CAM 0x1f1100UL
+#define PRS_REG_GFT_PROFILE_MASK_RAM 0x1f1000UL
+#define PRS_REG_CM_HDR_GFT_EVENT_ID_SHIFT 0
+#define PRS_REG_CM_HDR_GFT_CM_HDR_SHIFT 8
+#define PRS_REG_LOAD_L2_FILTER 0x1f0198UL
+
#endif
diff --git a/drivers/net/ethernet/qlogic/qed/qed_roce.c b/drivers/net/ethernet/qlogic/qed/qed_roce.c
index d9ff6b2..56289d7 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_roce.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_roce.c
@@ -66,17 +66,31 @@
#include "qed_roce.h"
#include "qed_ll2.h"
-void qed_async_roce_event(struct qed_hwfn *p_hwfn,
- struct event_ring_entry *p_eqe)
-{
- struct qed_rdma_info *p_rdma_info = p_hwfn->p_rdma_info;
+static void qed_roce_free_real_icid(struct qed_hwfn *p_hwfn, u16 icid);
- p_rdma_info->events.affiliated_event(p_rdma_info->events.context,
- p_eqe->opcode, &p_eqe->data);
+void qed_roce_async_event(struct qed_hwfn *p_hwfn,
+ u8 fw_event_code, union rdma_eqe_data *rdma_data)
+{
+ if (fw_event_code == ROCE_ASYNC_EVENT_DESTROY_QP_DONE) {
+ u16 icid =
+ (u16)le32_to_cpu(rdma_data->rdma_destroy_qp_data.cid);
+
+ /* icid release in this async event can occur only if the icid
+ * was offloaded to the FW. In case it wasn't offloaded this is
+ * handled in qed_roce_sp_destroy_qp.
+ */
+ qed_roce_free_real_icid(p_hwfn, icid);
+ } else {
+ struct qed_rdma_events *events = &p_hwfn->p_rdma_info->events;
+
+ events->affiliated_event(p_hwfn->p_rdma_info->events.context,
+ fw_event_code,
+ &rdma_data->async_handle);
+ }
}
static int qed_rdma_bmap_alloc(struct qed_hwfn *p_hwfn,
- struct qed_bmap *bmap, u32 max_count)
+ struct qed_bmap *bmap, u32 max_count, char *name)
{
DP_VERBOSE(p_hwfn, QED_MSG_RDMA, "max_count = %08x\n", max_count);
@@ -90,43 +104,62 @@
return -ENOMEM;
}
- DP_VERBOSE(p_hwfn, QED_MSG_RDMA, "Allocated bitmap %p\n",
- bmap->bitmap);
+ snprintf(bmap->name, QED_RDMA_MAX_BMAP_NAME, "%s", name);
+
+ DP_VERBOSE(p_hwfn, QED_MSG_RDMA, "0\n");
return 0;
}
static int qed_rdma_bmap_alloc_id(struct qed_hwfn *p_hwfn,
struct qed_bmap *bmap, u32 *id_num)
{
- DP_VERBOSE(p_hwfn, QED_MSG_RDMA, "bmap = %p\n", bmap);
-
*id_num = find_first_zero_bit(bmap->bitmap, bmap->max_count);
-
- if (*id_num >= bmap->max_count) {
- DP_NOTICE(p_hwfn, "no id available max_count=%d\n",
- bmap->max_count);
+ if (*id_num >= bmap->max_count)
return -EINVAL;
- }
__set_bit(*id_num, bmap->bitmap);
+ DP_VERBOSE(p_hwfn, QED_MSG_RDMA, "%s bitmap: allocated id %d\n",
+ bmap->name, *id_num);
+
return 0;
}
+static void qed_bmap_set_id(struct qed_hwfn *p_hwfn,
+ struct qed_bmap *bmap, u32 id_num)
+{
+ if (id_num >= bmap->max_count)
+ return;
+
+ __set_bit(id_num, bmap->bitmap);
+}
+
static void qed_bmap_release_id(struct qed_hwfn *p_hwfn,
struct qed_bmap *bmap, u32 id_num)
{
bool b_acquired;
- DP_VERBOSE(p_hwfn, QED_MSG_RDMA, "id_num = %08x", id_num);
if (id_num >= bmap->max_count)
return;
b_acquired = test_and_clear_bit(id_num, bmap->bitmap);
if (!b_acquired) {
- DP_NOTICE(p_hwfn, "ID %d already released\n", id_num);
+ DP_NOTICE(p_hwfn, "%s bitmap: id %d already released\n",
+ bmap->name, id_num);
return;
}
+
+ DP_VERBOSE(p_hwfn, QED_MSG_RDMA, "%s bitmap: released id %d\n",
+ bmap->name, id_num);
+}
+
+static int qed_bmap_test_id(struct qed_hwfn *p_hwfn,
+ struct qed_bmap *bmap, u32 id_num)
+{
+ if (id_num >= bmap->max_count)
+ return -1;
+
+ return test_bit(id_num, bmap->bitmap);
}
static u32 qed_rdma_get_sb_id(void *p_hwfn, u32 rel_sb_id)
@@ -170,7 +203,8 @@
/* Queue zone lines are shared between RoCE and L2 in such a way that
* they can be used by each without obstructing the other.
*/
- p_rdma_info->queue_zone_base = (u16)FEAT_NUM(p_hwfn, QED_L2_QUEUE);
+ p_rdma_info->queue_zone_base = (u16)RESC_START(p_hwfn, QED_L2_QUEUE);
+ p_rdma_info->max_queue_zones = (u16)RESC_NUM(p_hwfn, QED_L2_QUEUE);
/* Allocate a struct with device params and fill it */
p_rdma_info->dev = kzalloc(sizeof(*p_rdma_info->dev), GFP_KERNEL);
@@ -191,7 +225,8 @@
}
/* Allocate bit map for pd's */
- rc = qed_rdma_bmap_alloc(p_hwfn, &p_rdma_info->pd_map, RDMA_MAX_PDS);
+ rc = qed_rdma_bmap_alloc(p_hwfn, &p_rdma_info->pd_map, RDMA_MAX_PDS,
+ "PD");
if (rc) {
DP_VERBOSE(p_hwfn, QED_MSG_RDMA,
"Failed to allocate pd_map, rc = %d\n",
@@ -201,7 +236,7 @@
/* Allocate DPI bitmap */
rc = qed_rdma_bmap_alloc(p_hwfn, &p_rdma_info->dpi_map,
- p_hwfn->dpi_count);
+ p_hwfn->dpi_count, "DPI");
if (rc) {
DP_VERBOSE(p_hwfn, QED_MSG_RDMA,
"Failed to allocate DPI bitmap, rc = %d\n", rc);
@@ -212,7 +247,7 @@
* twice the number of QPs.
*/
rc = qed_rdma_bmap_alloc(p_hwfn, &p_rdma_info->cq_map,
- p_rdma_info->num_qps * 2);
+ p_rdma_info->num_qps * 2, "CQ");
if (rc) {
DP_VERBOSE(p_hwfn, QED_MSG_RDMA,
"Failed to allocate cq bitmap, rc = %d\n", rc);
@@ -224,7 +259,7 @@
* The maximum number of CQs is bounded to twice the number of QPs.
*/
rc = qed_rdma_bmap_alloc(p_hwfn, &p_rdma_info->toggle_bits,
- p_rdma_info->num_qps * 2);
+ p_rdma_info->num_qps * 2, "Toggle");
if (rc) {
DP_VERBOSE(p_hwfn, QED_MSG_RDMA,
"Failed to allocate toogle bits, rc = %d\n", rc);
@@ -233,7 +268,7 @@
/* Allocate bitmap for itids */
rc = qed_rdma_bmap_alloc(p_hwfn, &p_rdma_info->tid_map,
- p_rdma_info->num_mrs);
+ p_rdma_info->num_mrs, "MR");
if (rc) {
DP_VERBOSE(p_hwfn, QED_MSG_RDMA,
"Failed to allocate itids bitmaps, rc = %d\n", rc);
@@ -241,16 +276,27 @@
}
/* Allocate bitmap for cids used for qps. */
- rc = qed_rdma_bmap_alloc(p_hwfn, &p_rdma_info->cid_map, num_cons);
+ rc = qed_rdma_bmap_alloc(p_hwfn, &p_rdma_info->cid_map, num_cons,
+ "CID");
if (rc) {
DP_VERBOSE(p_hwfn, QED_MSG_RDMA,
"Failed to allocate cid bitmap, rc = %d\n", rc);
goto free_tid_map;
}
+ /* Allocate bitmap for cids used for responders/requesters. */
+ rc = qed_rdma_bmap_alloc(p_hwfn, &p_rdma_info->real_cid_map, num_cons,
+ "REAL_CID");
+ if (rc) {
+ DP_VERBOSE(p_hwfn, QED_MSG_RDMA,
+ "Failed to allocate real cid bitmap, rc = %d\n", rc);
+ goto free_cid_map;
+ }
DP_VERBOSE(p_hwfn, QED_MSG_RDMA, "Allocation successful\n");
return 0;
+free_cid_map:
+ kfree(p_rdma_info->cid_map.bitmap);
free_tid_map:
kfree(p_rdma_info->tid_map.bitmap);
free_toggle_map:
@@ -271,16 +317,79 @@
return rc;
}
+static void qed_rdma_bmap_free(struct qed_hwfn *p_hwfn,
+ struct qed_bmap *bmap, bool check)
+{
+ int weight = bitmap_weight(bmap->bitmap, bmap->max_count);
+ int last_line = bmap->max_count / (64 * 8);
+ int last_item = last_line * 8 +
+ DIV_ROUND_UP(bmap->max_count % (64 * 8), 64);
+ u64 *pmap = (u64 *)bmap->bitmap;
+ int line, item, offset;
+ u8 str_last_line[200] = { 0 };
+
+ if (!weight || !check)
+ goto end;
+
+ DP_NOTICE(p_hwfn,
+ "%s bitmap not free - size=%d, weight=%d, 512 bits per line\n",
+ bmap->name, bmap->max_count, weight);
+
+ /* print aligned non-zero lines, if any */
+ for (item = 0, line = 0; line < last_line; line++, item += 8)
+ if (bitmap_weight((unsigned long *)&pmap[item], 64 * 8))
+ DP_NOTICE(p_hwfn,
+ "line 0x%04x: 0x%016llx 0x%016llx 0x%016llx 0x%016llx 0x%016llx 0x%016llx 0x%016llx 0x%016llx\n",
+ line,
+ pmap[item],
+ pmap[item + 1],
+ pmap[item + 2],
+ pmap[item + 3],
+ pmap[item + 4],
+ pmap[item + 5],
+ pmap[item + 6], pmap[item + 7]);
+
+ /* print last unaligned non-zero line, if any */
+ if ((bmap->max_count % (64 * 8)) &&
+ (bitmap_weight((unsigned long *)&pmap[item],
+ bmap->max_count - item * 64))) {
+ offset = sprintf(str_last_line, "line 0x%04x: ", line);
+ for (; item < last_item; item++)
+ offset += sprintf(str_last_line + offset,
+ "0x%016llx ", pmap[item]);
+ DP_NOTICE(p_hwfn, "%s\n", str_last_line);
+ }
+
+end:
+ kfree(bmap->bitmap);
+ bmap->bitmap = NULL;
+}
+
static void qed_rdma_resc_free(struct qed_hwfn *p_hwfn)
{
+ struct qed_bmap *rcid_map = &p_hwfn->p_rdma_info->real_cid_map;
struct qed_rdma_info *p_rdma_info = p_hwfn->p_rdma_info;
+ int wait_count = 0;
- kfree(p_rdma_info->cid_map.bitmap);
- kfree(p_rdma_info->tid_map.bitmap);
- kfree(p_rdma_info->toggle_bits.bitmap);
- kfree(p_rdma_info->cq_map.bitmap);
- kfree(p_rdma_info->dpi_map.bitmap);
- kfree(p_rdma_info->pd_map.bitmap);
+ /* when destroying a_RoCE QP the control is returned to the user after
+ * the synchronous part. The asynchronous part may take a little longer.
+ * We delay for a short while if an async destroy QP is still expected.
+ * Beyond the added delay we clear the bitmap anyway.
+ */
+ while (bitmap_weight(rcid_map->bitmap, rcid_map->max_count)) {
+ msleep(100);
+ if (wait_count++ > 20) {
+ DP_NOTICE(p_hwfn, "cid bitmap wait timed out\n");
+ break;
+ }
+ }
+
+ qed_rdma_bmap_free(p_hwfn, &p_hwfn->p_rdma_info->cid_map, 1);
+ qed_rdma_bmap_free(p_hwfn, &p_hwfn->p_rdma_info->pd_map, 1);
+ qed_rdma_bmap_free(p_hwfn, &p_hwfn->p_rdma_info->dpi_map, 1);
+ qed_rdma_bmap_free(p_hwfn, &p_hwfn->p_rdma_info->cq_map, 1);
+ qed_rdma_bmap_free(p_hwfn, &p_hwfn->p_rdma_info->toggle_bits, 0);
+ qed_rdma_bmap_free(p_hwfn, &p_hwfn->p_rdma_info->tid_map, 1);
kfree(p_rdma_info->port);
kfree(p_rdma_info->dev);
@@ -675,6 +784,7 @@
((out_params->dpi) * p_hwfn->dpi_size);
out_params->dpi_size = p_hwfn->dpi_size;
+ out_params->wid_count = p_hwfn->wid_count;
DP_VERBOSE(p_hwfn, QED_MSG_RDMA, "Adding user - done, rc = %d\n", rc);
return rc;
@@ -693,6 +803,8 @@
p_port->link_speed = p_hwfn->mcp_info->link_output.speed;
+ p_port->max_msg_size = RDMA_MAX_DATA_SIZE_IN_WQE;
+
return p_port;
}
@@ -724,6 +836,14 @@
u32 addr;
p_hwfn = (struct qed_hwfn *)rdma_cxt;
+
+ if (qz_offset > p_hwfn->p_rdma_info->max_queue_zones) {
+ DP_NOTICE(p_hwfn,
+ "queue zone offset %d is too large (max is %d)\n",
+ qz_offset, p_hwfn->p_rdma_info->max_queue_zones);
+ return;
+ }
+
qz_num = p_hwfn->p_rdma_info->queue_zone_base + qz_offset;
addr = GTT_BAR0_MAP_REG_USDM_RAM +
USTORM_COMMON_QUEUE_CONS_OFFSET(qz_num);
@@ -737,9 +857,12 @@
static int qed_fill_rdma_dev_info(struct qed_dev *cdev,
struct qed_dev_rdma_info *info)
{
+ struct qed_hwfn *p_hwfn = QED_LEADING_HWFN(cdev);
+
memset(info, 0, sizeof(*info));
info->rdma_type = QED_RDMA_TYPE_ROCE;
+ info->user_dpm_enabled = (p_hwfn->db_bar_no_edpm == 0);
qed_fill_dev_info(cdev, &info->common);
@@ -887,8 +1010,7 @@
/* Allocate icid */
spin_lock_bh(&p_info->lock);
- rc = qed_rdma_bmap_alloc_id(p_hwfn,
- &p_info->cq_map, &returned_id);
+ rc = qed_rdma_bmap_alloc_id(p_hwfn, &p_info->cq_map, &returned_id);
spin_unlock_bh(&p_info->lock);
if (rc) {
@@ -1080,6 +1202,14 @@
return flavor;
}
+void qed_roce_free_cid_pair(struct qed_hwfn *p_hwfn, u16 cid)
+{
+ spin_lock_bh(&p_hwfn->p_rdma_info->lock);
+ qed_bmap_release_id(p_hwfn, &p_hwfn->p_rdma_info->cid_map, cid);
+ qed_bmap_release_id(p_hwfn, &p_hwfn->p_rdma_info->cid_map, cid + 1);
+ spin_unlock_bh(&p_hwfn->p_rdma_info->lock);
+}
+
static int qed_roce_alloc_cid(struct qed_hwfn *p_hwfn, u16 *cid)
{
struct qed_rdma_info *p_rdma_info = p_hwfn->p_rdma_info;
@@ -1139,15 +1269,22 @@
return rc;
}
+static void qed_roce_set_real_cid(struct qed_hwfn *p_hwfn, u32 cid)
+{
+ spin_lock_bh(&p_hwfn->p_rdma_info->lock);
+ qed_bmap_set_id(p_hwfn, &p_hwfn->p_rdma_info->real_cid_map, cid);
+ spin_unlock_bh(&p_hwfn->p_rdma_info->lock);
+}
+
static int qed_roce_sp_create_responder(struct qed_hwfn *p_hwfn,
struct qed_rdma_qp *qp)
{
struct roce_create_qp_resp_ramrod_data *p_ramrod;
struct qed_sp_init_data init_data;
- union qed_qm_pq_params qm_params;
enum roce_flavor roce_flavor;
struct qed_spq_entry *p_ent;
- u16 physical_queue0 = 0;
+ u16 regular_latency_queue;
+ enum protocol_type proto;
int rc;
DP_VERBOSE(p_hwfn, QED_MSG_RDMA, "icid = %08x\n", qp->icid);
@@ -1229,15 +1366,16 @@
p_ramrod->qp_handle_for_async.lo = cpu_to_le32(qp->qp_handle_async.lo);
p_ramrod->qp_handle_for_cqe.hi = cpu_to_le32(qp->qp_handle.hi);
p_ramrod->qp_handle_for_cqe.lo = cpu_to_le32(qp->qp_handle.lo);
- p_ramrod->stats_counter_id = p_hwfn->rel_pf_id;
p_ramrod->cq_cid = cpu_to_le32((p_hwfn->hw_info.opaque_fid << 16) |
qp->rq_cq_id);
- memset(&qm_params, 0, sizeof(qm_params));
- qm_params.roce.qpid = qp->icid >> 1;
- physical_queue0 = qed_get_qm_pq(p_hwfn, PROTOCOLID_ROCE, &qm_params);
+ regular_latency_queue = qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_OFLD);
- p_ramrod->physical_queue0 = cpu_to_le16(physical_queue0);
+ p_ramrod->regular_latency_phy_queue =
+ cpu_to_le16(regular_latency_queue);
+ p_ramrod->low_latency_phy_queue =
+ cpu_to_le16(regular_latency_queue);
+
p_ramrod->dpi = cpu_to_le16(qp->dpi);
qed_rdma_set_fw_mac(p_ramrod->remote_mac_addr, qp->remote_mac_addr);
@@ -1253,13 +1391,19 @@
rc = qed_spq_post(p_hwfn, p_ent, NULL);
- DP_VERBOSE(p_hwfn, QED_MSG_RDMA, "rc = %d physical_queue0 = 0x%x\n",
- rc, physical_queue0);
+ DP_VERBOSE(p_hwfn, QED_MSG_RDMA,
+ "rc = %d regular physical queue = 0x%x\n", rc,
+ regular_latency_queue);
if (rc)
goto err;
qp->resp_offloaded = true;
+ qp->cq_prod = 0;
+
+ proto = p_hwfn->p_rdma_info->proto;
+ qed_roce_set_real_cid(p_hwfn, qp->icid -
+ qed_cxt_get_proto_cid_start(p_hwfn, proto));
return rc;
@@ -1277,10 +1421,10 @@
{
struct roce_create_qp_req_ramrod_data *p_ramrod;
struct qed_sp_init_data init_data;
- union qed_qm_pq_params qm_params;
enum roce_flavor roce_flavor;
struct qed_spq_entry *p_ent;
- u16 physical_queue0 = 0;
+ u16 regular_latency_queue;
+ enum protocol_type proto;
int rc;
DP_VERBOSE(p_hwfn, QED_MSG_RDMA, "icid = %08x\n", qp->icid);
@@ -1351,15 +1495,16 @@
p_ramrod->qp_handle_for_async.lo = cpu_to_le32(qp->qp_handle_async.lo);
p_ramrod->qp_handle_for_cqe.hi = cpu_to_le32(qp->qp_handle.hi);
p_ramrod->qp_handle_for_cqe.lo = cpu_to_le32(qp->qp_handle.lo);
- p_ramrod->stats_counter_id = p_hwfn->rel_pf_id;
- p_ramrod->cq_cid = cpu_to_le32((p_hwfn->hw_info.opaque_fid << 16) |
- qp->sq_cq_id);
+ p_ramrod->cq_cid =
+ cpu_to_le32((p_hwfn->hw_info.opaque_fid << 16) | qp->sq_cq_id);
- memset(&qm_params, 0, sizeof(qm_params));
- qm_params.roce.qpid = qp->icid >> 1;
- physical_queue0 = qed_get_qm_pq(p_hwfn, PROTOCOLID_ROCE, &qm_params);
+ regular_latency_queue = qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_OFLD);
- p_ramrod->physical_queue0 = cpu_to_le16(physical_queue0);
+ p_ramrod->regular_latency_phy_queue =
+ cpu_to_le16(regular_latency_queue);
+ p_ramrod->low_latency_phy_queue =
+ cpu_to_le16(regular_latency_queue);
+
p_ramrod->dpi = cpu_to_le16(qp->dpi);
qed_rdma_set_fw_mac(p_ramrod->remote_mac_addr, qp->remote_mac_addr);
@@ -1378,6 +1523,10 @@
goto err;
qp->req_offloaded = true;
+ proto = p_hwfn->p_rdma_info->proto;
+ qed_roce_set_real_cid(p_hwfn,
+ qp->icid + 1 -
+ qed_cxt_get_proto_cid_start(p_hwfn, proto));
return rc;
@@ -1577,7 +1726,8 @@
static int qed_roce_sp_destroy_qp_responder(struct qed_hwfn *p_hwfn,
struct qed_rdma_qp *qp,
- u32 *num_invalidated_mw)
+ u32 *num_invalidated_mw,
+ u32 *cq_prod)
{
struct roce_destroy_qp_resp_output_params *p_ramrod_res;
struct roce_destroy_qp_resp_ramrod_data *p_ramrod;
@@ -1588,8 +1738,22 @@
DP_VERBOSE(p_hwfn, QED_MSG_RDMA, "icid = %08x\n", qp->icid);
- if (!qp->resp_offloaded)
+ *num_invalidated_mw = 0;
+ *cq_prod = qp->cq_prod;
+
+ if (!qp->resp_offloaded) {
+ /* If a responder was never offload, we need to free the cids
+ * allocated in create_qp as a FW async event will never arrive
+ */
+ u32 cid;
+
+ cid = qp->icid -
+ qed_cxt_get_proto_cid_start(p_hwfn,
+ p_hwfn->p_rdma_info->proto);
+ qed_roce_free_cid_pair(p_hwfn, (u16)cid);
+
return 0;
+ }
/* Get SPQ entry */
memset(&init_data, 0, sizeof(init_data));
@@ -1624,6 +1788,8 @@
goto err;
*num_invalidated_mw = le32_to_cpu(p_ramrod_res->num_invalidated_mw);
+ *cq_prod = le32_to_cpu(p_ramrod_res->cq_prod);
+ qp->cq_prod = *cq_prod;
/* Free IRQ - only if ramrod succeeded, in case FW is still using it */
dma_free_coherent(&p_hwfn->cdev->pdev->dev,
@@ -1827,10 +1993,8 @@
out_params->draining = false;
- if (rq_err_state)
+ if (rq_err_state || sq_err_state)
qp->cur_state = QED_ROCE_QP_STATE_ERR;
- else if (sq_err_state)
- qp->cur_state = QED_ROCE_QP_STATE_SQE;
else if (sq_draining)
out_params->draining = true;
out_params->state = qp->cur_state;
@@ -1849,10 +2013,9 @@
static int qed_roce_destroy_qp(struct qed_hwfn *p_hwfn, struct qed_rdma_qp *qp)
{
- struct qed_rdma_info *p_rdma_info = p_hwfn->p_rdma_info;
u32 num_invalidated_mw = 0;
u32 num_bound_mw = 0;
- u32 start_cid;
+ u32 cq_prod;
int rc;
/* Destroys the specified QP */
@@ -1866,7 +2029,8 @@
if (qp->cur_state != QED_ROCE_QP_STATE_RESET) {
rc = qed_roce_sp_destroy_qp_responder(p_hwfn, qp,
- &num_invalidated_mw);
+ &num_invalidated_mw,
+ &cq_prod);
if (rc)
return rc;
@@ -1881,21 +2045,6 @@
"number of invalidate memory windows is different from bounded ones\n");
return -EINVAL;
}
-
- spin_lock_bh(&p_rdma_info->lock);
-
- start_cid = qed_cxt_get_proto_cid_start(p_hwfn,
- p_rdma_info->proto);
-
- /* Release responder's icid */
- qed_bmap_release_id(p_hwfn, &p_rdma_info->cid_map,
- qp->icid - start_cid);
-
- /* Release requester's icid */
- qed_bmap_release_id(p_hwfn, &p_rdma_info->cid_map,
- qp->icid + 1 - start_cid);
-
- spin_unlock_bh(&p_rdma_info->lock);
}
return 0;
@@ -2097,8 +2246,7 @@
params->modify_flags);
return rc;
- } else if (qp->cur_state == QED_ROCE_QP_STATE_ERR ||
- qp->cur_state == QED_ROCE_QP_STATE_SQE) {
+ } else if (qp->cur_state == QED_ROCE_QP_STATE_ERR) {
/* ->ERR */
rc = qed_roce_sp_modify_responder(p_hwfn, qp, true,
params->modify_flags);
@@ -2110,12 +2258,19 @@
return rc;
} else if (qp->cur_state == QED_ROCE_QP_STATE_RESET) {
/* Any state -> RESET */
+ u32 cq_prod;
- rc = qed_roce_sp_destroy_qp_responder(p_hwfn, qp,
- &num_invalidated_mw);
+ /* Send destroy responder ramrod */
+ rc = qed_roce_sp_destroy_qp_responder(p_hwfn,
+ qp,
+ &num_invalidated_mw,
+ &cq_prod);
+
if (rc)
return rc;
+ qp->cq_prod = cq_prod;
+
rc = qed_roce_sp_destroy_qp_requester(p_hwfn, qp,
&num_bound_mw);
@@ -2357,6 +2512,8 @@
}
rc = qed_spq_post(p_hwfn, p_ent, &fw_return_code);
+ if (rc)
+ return rc;
if (fw_return_code != RDMA_RETURN_OK) {
DP_NOTICE(p_hwfn, "fw_return_code = %d\n", fw_return_code);
@@ -2454,6 +2611,31 @@
return rc;
}
+static void qed_roce_free_real_icid(struct qed_hwfn *p_hwfn, u16 icid)
+{
+ struct qed_rdma_info *p_rdma_info = p_hwfn->p_rdma_info;
+ u32 start_cid, cid, xcid;
+
+ /* an even icid belongs to a responder while an odd icid belongs to a
+ * requester. The 'cid' received as an input can be either. We calculate
+ * the "partner" icid and call it xcid. Only if both are free then the
+ * "cid" map can be cleared.
+ */
+ start_cid = qed_cxt_get_proto_cid_start(p_hwfn, p_rdma_info->proto);
+ cid = icid - start_cid;
+ xcid = cid ^ 1;
+
+ spin_lock_bh(&p_rdma_info->lock);
+
+ qed_bmap_release_id(p_hwfn, &p_rdma_info->real_cid_map, cid);
+ if (qed_bmap_test_id(p_hwfn, &p_rdma_info->real_cid_map, xcid) == 0) {
+ qed_bmap_release_id(p_hwfn, &p_rdma_info->cid_map, cid);
+ qed_bmap_release_id(p_hwfn, &p_rdma_info->cid_map, xcid);
+ }
+
+ spin_unlock_bh(&p_hwfn->p_rdma_info->lock);
+}
+
static void *qed_rdma_get_rdma_ctx(struct qed_dev *cdev)
{
return QED_LEADING_HWFN(cdev);
@@ -2773,7 +2955,7 @@
: QED_LL2_RROCE;
if (pkt->roce_mode == ROCE_V2_IPV4)
- flags |= BIT(CORE_TX_BD_FLAGS_IP_CSUM_SHIFT);
+ flags |= BIT(CORE_TX_BD_DATA_IP_CSUM_SHIFT);
/* Tx header */
rc = qed_ll2_prepare_tx_packet(QED_LEADING_HWFN(cdev), roce_ll2->handle,
diff --git a/drivers/net/ethernet/qlogic/qed/qed_roce.h b/drivers/net/ethernet/qlogic/qed/qed_roce.h
index 36cf4b2..9742af5 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_roce.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_roce.h
@@ -67,9 +67,11 @@
QED_RDMA_TOGGLE_BIT_SET = 1
};
+#define QED_RDMA_MAX_BMAP_NAME (10)
struct qed_bmap {
unsigned long *bitmap;
u32 max_count;
+ char name[QED_RDMA_MAX_BMAP_NAME];
};
struct qed_rdma_info {
@@ -82,6 +84,7 @@
struct qed_bmap qp_map;
struct qed_bmap srq_map;
struct qed_bmap cid_map;
+ struct qed_bmap real_cid_map;
struct qed_bmap dpi_map;
struct qed_bmap toggle_bits;
struct qed_rdma_events events;
@@ -92,6 +95,7 @@
u32 num_qps;
u32 num_mrs;
u16 queue_zone_base;
+ u16 max_queue_zones;
enum protocol_type proto;
};
@@ -153,6 +157,7 @@
dma_addr_t irq_phys_addr;
u8 irq_num_pages;
bool resp_offloaded;
+ u32 cq_prod;
u8 remote_mac_addr[6];
u8 local_mac_addr[6];
@@ -163,8 +168,8 @@
#if IS_ENABLED(CONFIG_QED_RDMA)
void qed_rdma_dpm_bar(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt);
-void qed_async_roce_event(struct qed_hwfn *p_hwfn,
- struct event_ring_entry *p_eqe);
+void qed_roce_async_event(struct qed_hwfn *p_hwfn,
+ u8 fw_event_code, union rdma_eqe_data *rdma_data);
void qed_ll2b_complete_tx_gsi_packet(struct qed_hwfn *p_hwfn,
u8 connection_handle,
void *cookie,
@@ -187,7 +192,9 @@
u16 src_mac_addr_lo, bool b_last_packet);
#else
static inline void qed_rdma_dpm_bar(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt) {}
-static inline void qed_async_roce_event(struct qed_hwfn *p_hwfn, struct event_ring_entry *p_eqe) {}
+static inline void qed_roce_async_event(struct qed_hwfn *p_hwfn,
+ u8 fw_event_code,
+ union rdma_eqe_data *rdma_data) {}
static inline void qed_ll2b_complete_tx_gsi_packet(struct qed_hwfn *p_hwfn,
u8 connection_handle,
void *cookie,
diff --git a/drivers/net/ethernet/qlogic/qed/qed_sp.h b/drivers/net/ethernet/qlogic/qed/qed_sp.h
index 30393ff..3357bbe 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_sp.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_sp.h
@@ -84,6 +84,7 @@
struct tx_queue_stop_ramrod_data tx_queue_stop;
struct vport_start_ramrod_data vport_start;
struct vport_stop_ramrod_data vport_stop;
+ struct rx_update_gft_filter_data rx_update_gft;
struct vport_update_ramrod_data vport_update;
struct core_rx_start_ramrod_data core_rx_queue_start;
struct core_rx_stop_ramrod_data core_rx_queue_stop;
@@ -408,7 +409,7 @@
*/
int qed_sp_pf_start(struct qed_hwfn *p_hwfn,
- struct qed_tunn_start_params *p_tunn,
+ struct qed_tunnel_info *p_tunn,
enum qed_mf_mode mode, bool allow_npar_tx_switch);
/**
@@ -441,7 +442,7 @@
int qed_sp_pf_stop(struct qed_hwfn *p_hwfn);
int qed_sp_pf_update_tunn_cfg(struct qed_hwfn *p_hwfn,
- struct qed_tunn_update_params *p_tunn,
+ struct qed_tunnel_info *p_tunn,
enum spq_mode comp_mode,
struct qed_spq_comp_cb *p_comp_data);
/**
diff --git a/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c b/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c
index 6fb80f9..bc3694e 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c
@@ -111,7 +111,7 @@
return 0;
}
-static enum tunnel_clss qed_tunn_get_clss_type(u8 type)
+static enum tunnel_clss qed_tunn_clss_to_fw_clss(u8 type)
{
switch (type) {
case QED_TUNN_CLSS_MAC_VLAN:
@@ -122,206 +122,201 @@
return TUNNEL_CLSS_INNER_MAC_VLAN;
case QED_TUNN_CLSS_INNER_MAC_VNI:
return TUNNEL_CLSS_INNER_MAC_VNI;
+ case QED_TUNN_CLSS_MAC_VLAN_DUAL_STAGE:
+ return TUNNEL_CLSS_MAC_VLAN_DUAL_STAGE;
default:
return TUNNEL_CLSS_MAC_VLAN;
}
}
static void
-qed_tunn_set_pf_fix_tunn_mode(struct qed_hwfn *p_hwfn,
- struct qed_tunn_update_params *p_src,
- struct pf_update_tunnel_config *p_tunn_cfg)
+qed_set_pf_update_tunn_mode(struct qed_tunnel_info *p_tun,
+ struct qed_tunnel_info *p_src, bool b_pf_start)
{
- unsigned long cached_tunn_mode = p_hwfn->cdev->tunn_mode;
- unsigned long update_mask = p_src->tunn_mode_update_mask;
- unsigned long tunn_mode = p_src->tunn_mode;
- unsigned long new_tunn_mode = 0;
+ if (p_src->vxlan.b_update_mode || b_pf_start)
+ p_tun->vxlan.b_mode_enabled = p_src->vxlan.b_mode_enabled;
- if (test_bit(QED_MODE_L2GRE_TUNN, &update_mask)) {
- if (test_bit(QED_MODE_L2GRE_TUNN, &tunn_mode))
- __set_bit(QED_MODE_L2GRE_TUNN, &new_tunn_mode);
- } else {
- if (test_bit(QED_MODE_L2GRE_TUNN, &cached_tunn_mode))
- __set_bit(QED_MODE_L2GRE_TUNN, &new_tunn_mode);
+ if (p_src->l2_gre.b_update_mode || b_pf_start)
+ p_tun->l2_gre.b_mode_enabled = p_src->l2_gre.b_mode_enabled;
+
+ if (p_src->ip_gre.b_update_mode || b_pf_start)
+ p_tun->ip_gre.b_mode_enabled = p_src->ip_gre.b_mode_enabled;
+
+ if (p_src->l2_geneve.b_update_mode || b_pf_start)
+ p_tun->l2_geneve.b_mode_enabled =
+ p_src->l2_geneve.b_mode_enabled;
+
+ if (p_src->ip_geneve.b_update_mode || b_pf_start)
+ p_tun->ip_geneve.b_mode_enabled =
+ p_src->ip_geneve.b_mode_enabled;
+}
+
+static void qed_set_tunn_cls_info(struct qed_tunnel_info *p_tun,
+ struct qed_tunnel_info *p_src)
+{
+ enum tunnel_clss type;
+
+ p_tun->b_update_rx_cls = p_src->b_update_rx_cls;
+ p_tun->b_update_tx_cls = p_src->b_update_tx_cls;
+
+ type = qed_tunn_clss_to_fw_clss(p_src->vxlan.tun_cls);
+ p_tun->vxlan.tun_cls = type;
+ type = qed_tunn_clss_to_fw_clss(p_src->l2_gre.tun_cls);
+ p_tun->l2_gre.tun_cls = type;
+ type = qed_tunn_clss_to_fw_clss(p_src->ip_gre.tun_cls);
+ p_tun->ip_gre.tun_cls = type;
+ type = qed_tunn_clss_to_fw_clss(p_src->l2_geneve.tun_cls);
+ p_tun->l2_geneve.tun_cls = type;
+ type = qed_tunn_clss_to_fw_clss(p_src->ip_geneve.tun_cls);
+ p_tun->ip_geneve.tun_cls = type;
+}
+
+static void qed_set_tunn_ports(struct qed_tunnel_info *p_tun,
+ struct qed_tunnel_info *p_src)
+{
+ p_tun->geneve_port.b_update_port = p_src->geneve_port.b_update_port;
+ p_tun->vxlan_port.b_update_port = p_src->vxlan_port.b_update_port;
+
+ if (p_src->geneve_port.b_update_port)
+ p_tun->geneve_port.port = p_src->geneve_port.port;
+
+ if (p_src->vxlan_port.b_update_port)
+ p_tun->vxlan_port.port = p_src->vxlan_port.port;
+}
+
+static void
+__qed_set_ramrod_tunnel_param(u8 *p_tunn_cls, u8 *p_enable_tx_clas,
+ struct qed_tunn_update_type *tun_type)
+{
+ *p_tunn_cls = tun_type->tun_cls;
+
+ if (tun_type->b_mode_enabled)
+ *p_enable_tx_clas = 1;
+}
+
+static void
+qed_set_ramrod_tunnel_param(u8 *p_tunn_cls, u8 *p_enable_tx_clas,
+ struct qed_tunn_update_type *tun_type,
+ u8 *p_update_port, __le16 *p_port,
+ struct qed_tunn_update_udp_port *p_udp_port)
+{
+ __qed_set_ramrod_tunnel_param(p_tunn_cls, p_enable_tx_clas, tun_type);
+ if (p_udp_port->b_update_port) {
+ *p_update_port = 1;
+ *p_port = cpu_to_le16(p_udp_port->port);
}
-
- if (test_bit(QED_MODE_IPGRE_TUNN, &update_mask)) {
- if (test_bit(QED_MODE_IPGRE_TUNN, &tunn_mode))
- __set_bit(QED_MODE_IPGRE_TUNN, &new_tunn_mode);
- } else {
- if (test_bit(QED_MODE_IPGRE_TUNN, &cached_tunn_mode))
- __set_bit(QED_MODE_IPGRE_TUNN, &new_tunn_mode);
- }
-
- if (test_bit(QED_MODE_VXLAN_TUNN, &update_mask)) {
- if (test_bit(QED_MODE_VXLAN_TUNN, &tunn_mode))
- __set_bit(QED_MODE_VXLAN_TUNN, &new_tunn_mode);
- } else {
- if (test_bit(QED_MODE_VXLAN_TUNN, &cached_tunn_mode))
- __set_bit(QED_MODE_VXLAN_TUNN, &new_tunn_mode);
- }
-
- if (p_src->update_geneve_udp_port) {
- p_tunn_cfg->set_geneve_udp_port_flg = 1;
- p_tunn_cfg->geneve_udp_port =
- cpu_to_le16(p_src->geneve_udp_port);
- }
-
- if (test_bit(QED_MODE_L2GENEVE_TUNN, &update_mask)) {
- if (test_bit(QED_MODE_L2GENEVE_TUNN, &tunn_mode))
- __set_bit(QED_MODE_L2GENEVE_TUNN, &new_tunn_mode);
- } else {
- if (test_bit(QED_MODE_L2GENEVE_TUNN, &cached_tunn_mode))
- __set_bit(QED_MODE_L2GENEVE_TUNN, &new_tunn_mode);
- }
-
- if (test_bit(QED_MODE_IPGENEVE_TUNN, &update_mask)) {
- if (test_bit(QED_MODE_IPGENEVE_TUNN, &tunn_mode))
- __set_bit(QED_MODE_IPGENEVE_TUNN, &new_tunn_mode);
- } else {
- if (test_bit(QED_MODE_IPGENEVE_TUNN, &cached_tunn_mode))
- __set_bit(QED_MODE_IPGENEVE_TUNN, &new_tunn_mode);
- }
-
- p_src->tunn_mode = new_tunn_mode;
}
static void
qed_tunn_set_pf_update_params(struct qed_hwfn *p_hwfn,
- struct qed_tunn_update_params *p_src,
+ struct qed_tunnel_info *p_src,
struct pf_update_tunnel_config *p_tunn_cfg)
{
- unsigned long tunn_mode = p_src->tunn_mode;
- enum tunnel_clss type;
+ struct qed_tunnel_info *p_tun = &p_hwfn->cdev->tunnel;
- qed_tunn_set_pf_fix_tunn_mode(p_hwfn, p_src, p_tunn_cfg);
- p_tunn_cfg->update_rx_pf_clss = p_src->update_rx_pf_clss;
- p_tunn_cfg->update_tx_pf_clss = p_src->update_tx_pf_clss;
+ qed_set_pf_update_tunn_mode(p_tun, p_src, false);
+ qed_set_tunn_cls_info(p_tun, p_src);
+ qed_set_tunn_ports(p_tun, p_src);
- type = qed_tunn_get_clss_type(p_src->tunn_clss_vxlan);
- p_tunn_cfg->tunnel_clss_vxlan = type;
+ qed_set_ramrod_tunnel_param(&p_tunn_cfg->tunnel_clss_vxlan,
+ &p_tunn_cfg->tx_enable_vxlan,
+ &p_tun->vxlan,
+ &p_tunn_cfg->set_vxlan_udp_port_flg,
+ &p_tunn_cfg->vxlan_udp_port,
+ &p_tun->vxlan_port);
- type = qed_tunn_get_clss_type(p_src->tunn_clss_l2gre);
- p_tunn_cfg->tunnel_clss_l2gre = type;
+ qed_set_ramrod_tunnel_param(&p_tunn_cfg->tunnel_clss_l2geneve,
+ &p_tunn_cfg->tx_enable_l2geneve,
+ &p_tun->l2_geneve,
+ &p_tunn_cfg->set_geneve_udp_port_flg,
+ &p_tunn_cfg->geneve_udp_port,
+ &p_tun->geneve_port);
- type = qed_tunn_get_clss_type(p_src->tunn_clss_ipgre);
- p_tunn_cfg->tunnel_clss_ipgre = type;
+ __qed_set_ramrod_tunnel_param(&p_tunn_cfg->tunnel_clss_ipgeneve,
+ &p_tunn_cfg->tx_enable_ipgeneve,
+ &p_tun->ip_geneve);
- if (p_src->update_vxlan_udp_port) {
- p_tunn_cfg->set_vxlan_udp_port_flg = 1;
- p_tunn_cfg->vxlan_udp_port = cpu_to_le16(p_src->vxlan_udp_port);
- }
+ __qed_set_ramrod_tunnel_param(&p_tunn_cfg->tunnel_clss_l2gre,
+ &p_tunn_cfg->tx_enable_l2gre,
+ &p_tun->l2_gre);
- if (test_bit(QED_MODE_L2GRE_TUNN, &tunn_mode))
- p_tunn_cfg->tx_enable_l2gre = 1;
+ __qed_set_ramrod_tunnel_param(&p_tunn_cfg->tunnel_clss_ipgre,
+ &p_tunn_cfg->tx_enable_ipgre,
+ &p_tun->ip_gre);
- if (test_bit(QED_MODE_IPGRE_TUNN, &tunn_mode))
- p_tunn_cfg->tx_enable_ipgre = 1;
-
- if (test_bit(QED_MODE_VXLAN_TUNN, &tunn_mode))
- p_tunn_cfg->tx_enable_vxlan = 1;
-
- if (p_src->update_geneve_udp_port) {
- p_tunn_cfg->set_geneve_udp_port_flg = 1;
- p_tunn_cfg->geneve_udp_port =
- cpu_to_le16(p_src->geneve_udp_port);
- }
-
- if (test_bit(QED_MODE_L2GENEVE_TUNN, &tunn_mode))
- p_tunn_cfg->tx_enable_l2geneve = 1;
-
- if (test_bit(QED_MODE_IPGENEVE_TUNN, &tunn_mode))
- p_tunn_cfg->tx_enable_ipgeneve = 1;
-
- type = qed_tunn_get_clss_type(p_src->tunn_clss_l2geneve);
- p_tunn_cfg->tunnel_clss_l2geneve = type;
-
- type = qed_tunn_get_clss_type(p_src->tunn_clss_ipgeneve);
- p_tunn_cfg->tunnel_clss_ipgeneve = type;
+ p_tunn_cfg->update_rx_pf_clss = p_tun->b_update_rx_cls;
+ p_tunn_cfg->update_tx_pf_clss = p_tun->b_update_tx_cls;
}
static void qed_set_hw_tunn_mode(struct qed_hwfn *p_hwfn,
struct qed_ptt *p_ptt,
- unsigned long tunn_mode)
+ struct qed_tunnel_info *p_tun)
{
- u8 l2gre_enable = 0, ipgre_enable = 0, vxlan_enable = 0;
- u8 l2geneve_enable = 0, ipgeneve_enable = 0;
+ qed_set_gre_enable(p_hwfn, p_ptt, p_tun->l2_gre.b_mode_enabled,
+ p_tun->ip_gre.b_mode_enabled);
+ qed_set_vxlan_enable(p_hwfn, p_ptt, p_tun->vxlan.b_mode_enabled);
- if (test_bit(QED_MODE_L2GRE_TUNN, &tunn_mode))
- l2gre_enable = 1;
+ qed_set_geneve_enable(p_hwfn, p_ptt, p_tun->l2_geneve.b_mode_enabled,
+ p_tun->ip_geneve.b_mode_enabled);
+}
- if (test_bit(QED_MODE_IPGRE_TUNN, &tunn_mode))
- ipgre_enable = 1;
+static void qed_set_hw_tunn_mode_port(struct qed_hwfn *p_hwfn,
+ struct qed_tunnel_info *p_tunn)
+{
+ if (p_tunn->vxlan_port.b_update_port)
+ qed_set_vxlan_dest_port(p_hwfn, p_hwfn->p_main_ptt,
+ p_tunn->vxlan_port.port);
- if (test_bit(QED_MODE_VXLAN_TUNN, &tunn_mode))
- vxlan_enable = 1;
+ if (p_tunn->geneve_port.b_update_port)
+ qed_set_geneve_dest_port(p_hwfn, p_hwfn->p_main_ptt,
+ p_tunn->geneve_port.port);
- qed_set_gre_enable(p_hwfn, p_ptt, l2gre_enable, ipgre_enable);
- qed_set_vxlan_enable(p_hwfn, p_ptt, vxlan_enable);
-
- if (test_bit(QED_MODE_L2GENEVE_TUNN, &tunn_mode))
- l2geneve_enable = 1;
-
- if (test_bit(QED_MODE_IPGENEVE_TUNN, &tunn_mode))
- ipgeneve_enable = 1;
-
- qed_set_geneve_enable(p_hwfn, p_ptt, l2geneve_enable,
- ipgeneve_enable);
+ qed_set_hw_tunn_mode(p_hwfn, p_hwfn->p_main_ptt, p_tunn);
}
static void
qed_tunn_set_pf_start_params(struct qed_hwfn *p_hwfn,
- struct qed_tunn_start_params *p_src,
+ struct qed_tunnel_info *p_src,
struct pf_start_tunnel_config *p_tunn_cfg)
{
- unsigned long tunn_mode;
- enum tunnel_clss type;
+ struct qed_tunnel_info *p_tun = &p_hwfn->cdev->tunnel;
if (!p_src)
return;
- tunn_mode = p_src->tunn_mode;
- type = qed_tunn_get_clss_type(p_src->tunn_clss_vxlan);
- p_tunn_cfg->tunnel_clss_vxlan = type;
- type = qed_tunn_get_clss_type(p_src->tunn_clss_l2gre);
- p_tunn_cfg->tunnel_clss_l2gre = type;
- type = qed_tunn_get_clss_type(p_src->tunn_clss_ipgre);
- p_tunn_cfg->tunnel_clss_ipgre = type;
+ qed_set_pf_update_tunn_mode(p_tun, p_src, true);
+ qed_set_tunn_cls_info(p_tun, p_src);
+ qed_set_tunn_ports(p_tun, p_src);
- if (p_src->update_vxlan_udp_port) {
- p_tunn_cfg->set_vxlan_udp_port_flg = 1;
- p_tunn_cfg->vxlan_udp_port = cpu_to_le16(p_src->vxlan_udp_port);
- }
+ qed_set_ramrod_tunnel_param(&p_tunn_cfg->tunnel_clss_vxlan,
+ &p_tunn_cfg->tx_enable_vxlan,
+ &p_tun->vxlan,
+ &p_tunn_cfg->set_vxlan_udp_port_flg,
+ &p_tunn_cfg->vxlan_udp_port,
+ &p_tun->vxlan_port);
- if (test_bit(QED_MODE_L2GRE_TUNN, &tunn_mode))
- p_tunn_cfg->tx_enable_l2gre = 1;
+ qed_set_ramrod_tunnel_param(&p_tunn_cfg->tunnel_clss_l2geneve,
+ &p_tunn_cfg->tx_enable_l2geneve,
+ &p_tun->l2_geneve,
+ &p_tunn_cfg->set_geneve_udp_port_flg,
+ &p_tunn_cfg->geneve_udp_port,
+ &p_tun->geneve_port);
- if (test_bit(QED_MODE_IPGRE_TUNN, &tunn_mode))
- p_tunn_cfg->tx_enable_ipgre = 1;
+ __qed_set_ramrod_tunnel_param(&p_tunn_cfg->tunnel_clss_ipgeneve,
+ &p_tunn_cfg->tx_enable_ipgeneve,
+ &p_tun->ip_geneve);
- if (test_bit(QED_MODE_VXLAN_TUNN, &tunn_mode))
- p_tunn_cfg->tx_enable_vxlan = 1;
+ __qed_set_ramrod_tunnel_param(&p_tunn_cfg->tunnel_clss_l2gre,
+ &p_tunn_cfg->tx_enable_l2gre,
+ &p_tun->l2_gre);
- if (p_src->update_geneve_udp_port) {
- p_tunn_cfg->set_geneve_udp_port_flg = 1;
- p_tunn_cfg->geneve_udp_port =
- cpu_to_le16(p_src->geneve_udp_port);
- }
-
- if (test_bit(QED_MODE_L2GENEVE_TUNN, &tunn_mode))
- p_tunn_cfg->tx_enable_l2geneve = 1;
-
- if (test_bit(QED_MODE_IPGENEVE_TUNN, &tunn_mode))
- p_tunn_cfg->tx_enable_ipgeneve = 1;
-
- type = qed_tunn_get_clss_type(p_src->tunn_clss_l2geneve);
- p_tunn_cfg->tunnel_clss_l2geneve = type;
- type = qed_tunn_get_clss_type(p_src->tunn_clss_ipgeneve);
- p_tunn_cfg->tunnel_clss_ipgeneve = type;
+ __qed_set_ramrod_tunnel_param(&p_tunn_cfg->tunnel_clss_ipgre,
+ &p_tunn_cfg->tx_enable_ipgre,
+ &p_tun->ip_gre);
}
int qed_sp_pf_start(struct qed_hwfn *p_hwfn,
- struct qed_tunn_start_params *p_tunn,
+ struct qed_tunnel_info *p_tunn,
enum qed_mf_mode mode, bool allow_npar_tx_switch)
{
struct pf_start_ramrod_data *p_ramrod = NULL;
@@ -416,11 +411,8 @@
rc = qed_spq_post(p_hwfn, p_ent, NULL);
- if (p_tunn) {
- qed_set_hw_tunn_mode(p_hwfn, p_hwfn->p_main_ptt,
- p_tunn->tunn_mode);
- p_hwfn->cdev->tunn_mode = p_tunn->tunn_mode;
- }
+ if (p_tunn)
+ qed_set_hw_tunn_mode_port(p_hwfn, &p_hwfn->cdev->tunnel);
return rc;
}
@@ -451,7 +443,7 @@
/* Set pf update ramrod command params */
int qed_sp_pf_update_tunn_cfg(struct qed_hwfn *p_hwfn,
- struct qed_tunn_update_params *p_tunn,
+ struct qed_tunnel_info *p_tunn,
enum spq_mode comp_mode,
struct qed_spq_comp_cb *p_comp_data)
{
@@ -459,6 +451,12 @@
struct qed_sp_init_data init_data;
int rc = -EINVAL;
+ if (IS_VF(p_hwfn->cdev))
+ return qed_vf_pf_tunnel_param_update(p_hwfn, p_tunn);
+
+ if (!p_tunn)
+ return -EINVAL;
+
/* Get SPQ entry */
memset(&init_data, 0, sizeof(init_data));
init_data.cid = qed_spq_get_cid(p_hwfn);
@@ -479,15 +477,7 @@
if (rc)
return rc;
- if (p_tunn->update_vxlan_udp_port)
- qed_set_vxlan_dest_port(p_hwfn, p_hwfn->p_main_ptt,
- p_tunn->vxlan_udp_port);
- if (p_tunn->update_geneve_udp_port)
- qed_set_geneve_dest_port(p_hwfn, p_hwfn->p_main_ptt,
- p_tunn->geneve_udp_port);
-
- qed_set_hw_tunn_mode(p_hwfn, p_hwfn->p_main_ptt, p_tunn->tunn_mode);
- p_hwfn->cdev->tunn_mode = p_tunn->tunn_mode;
+ qed_set_hw_tunn_mode_port(p_hwfn, &p_hwfn->cdev->tunnel);
return rc;
}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_spq.c b/drivers/net/ethernet/qlogic/qed/qed_spq.c
index 645328a..f6423a1 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_spq.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_spq.c
@@ -119,6 +119,7 @@
u8 *p_fw_ret, bool skip_quick_poll)
{
struct qed_spq_comp_done *comp_done;
+ struct qed_ptt *p_ptt;
int rc;
/* A relatively short polling period w/o sleeping, to allow the FW to
@@ -135,8 +136,14 @@
if (!rc)
return 0;
+ p_ptt = qed_ptt_acquire(p_hwfn);
+ if (!p_ptt) {
+ DP_NOTICE(p_hwfn, "ptt, failed to acquire\n");
+ return -EAGAIN;
+ }
+
DP_INFO(p_hwfn, "Ramrod is stuck, requesting MCP drain\n");
- rc = qed_mcp_drain(p_hwfn, p_hwfn->p_main_ptt);
+ rc = qed_mcp_drain(p_hwfn, p_ptt);
if (rc) {
DP_NOTICE(p_hwfn, "MCP drain failed\n");
goto err;
@@ -145,15 +152,18 @@
/* Retry after drain */
rc = __qed_spq_block(p_hwfn, p_ent, p_fw_ret, true);
if (!rc)
- return 0;
+ goto out;
comp_done = (struct qed_spq_comp_done *)p_ent->comp_cb.cookie;
- if (comp_done->done == 1) {
+ if (comp_done->done == 1)
if (p_fw_ret)
*p_fw_ret = comp_done->fw_return_code;
- return 0;
- }
+out:
+ qed_ptt_release(p_hwfn, p_ptt);
+ return 0;
+
err:
+ qed_ptt_release(p_hwfn, p_ptt);
DP_NOTICE(p_hwfn,
"Ramrod is stuck [CID %08x cmd %02x protocol %02x echo %04x]\n",
le32_to_cpu(p_ent->elem.hdr.cid),
@@ -205,11 +215,10 @@
static void qed_spq_hw_initialize(struct qed_hwfn *p_hwfn,
struct qed_spq *p_spq)
{
- u16 pq;
- struct qed_cxt_info cxt_info;
- struct core_conn_context *p_cxt;
- union qed_qm_pq_params pq_params;
- int rc;
+ struct core_conn_context *p_cxt;
+ struct qed_cxt_info cxt_info;
+ u16 physical_q;
+ int rc;
cxt_info.iid = p_spq->cid;
@@ -231,10 +240,8 @@
XSTORM_CORE_CONN_AG_CTX_CONSOLID_PROD_CF_EN, 1);
/* QM physical queue */
- memset(&pq_params, 0, sizeof(pq_params));
- pq_params.core.tc = LB_TC;
- pq = qed_get_qm_pq(p_hwfn, PROTOCOLID_CORE, &pq_params);
- p_cxt->xstorm_ag_context.physical_q0 = cpu_to_le16(pq);
+ physical_q = qed_get_cm_pq_idx(p_hwfn, PQ_FLAGS_LB);
+ p_cxt->xstorm_ag_context.physical_q0 = cpu_to_le16(physical_q);
p_cxt->xstorm_st_context.spq_base_lo =
DMA_LO_LE(p_spq->chain.p_phys_addr);
@@ -296,9 +303,12 @@
struct event_ring_entry *p_eqe)
{
switch (p_eqe->protocol_id) {
+#if IS_ENABLED(CONFIG_QED_RDMA)
case PROTOCOLID_ROCE:
- qed_async_roce_event(p_hwfn, p_eqe);
+ qed_roce_async_event(p_hwfn, p_eqe->opcode,
+ &p_eqe->data.rdma_data);
return 0;
+#endif
case PROTOCOLID_COMMON:
return qed_sriov_eqe_event(p_hwfn,
p_eqe->opcode,
@@ -306,14 +316,6 @@
case PROTOCOLID_ISCSI:
if (!IS_ENABLED(CONFIG_QED_ISCSI))
return -EINVAL;
- if (p_eqe->opcode == ISCSI_EVENT_TYPE_ASYN_DELETE_OOO_ISLES) {
- u32 cid = le32_to_cpu(p_eqe->data.iscsi_info.cid);
-
- qed_ooo_release_connection_isles(p_hwfn,
- p_hwfn->p_ooo_info,
- cid);
- return 0;
- }
if (p_hwfn->p_iscsi_info->event_cb) {
struct qed_iscsi_info *p_iscsi = p_hwfn->p_iscsi_info;
diff --git a/drivers/net/ethernet/qlogic/qed/qed_sriov.c b/drivers/net/ethernet/qlogic/qed/qed_sriov.c
index 253c2bb..d5df29f 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_sriov.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_sriov.c
@@ -178,26 +178,59 @@
return vf;
}
-static bool qed_iov_validate_rxq(struct qed_hwfn *p_hwfn,
- struct qed_vf_info *p_vf, u16 rx_qid)
+enum qed_iov_validate_q_mode {
+ QED_IOV_VALIDATE_Q_NA,
+ QED_IOV_VALIDATE_Q_ENABLE,
+ QED_IOV_VALIDATE_Q_DISABLE,
+};
+
+static bool qed_iov_validate_queue_mode(struct qed_hwfn *p_hwfn,
+ struct qed_vf_info *p_vf,
+ u16 qid,
+ enum qed_iov_validate_q_mode mode,
+ bool b_is_tx)
{
- if (rx_qid >= p_vf->num_rxqs)
+ if (mode == QED_IOV_VALIDATE_Q_NA)
+ return true;
+
+ if ((b_is_tx && p_vf->vf_queues[qid].p_tx_cid) ||
+ (!b_is_tx && p_vf->vf_queues[qid].p_rx_cid))
+ return mode == QED_IOV_VALIDATE_Q_ENABLE;
+
+ /* In case we haven't found any valid cid, then its disabled */
+ return mode == QED_IOV_VALIDATE_Q_DISABLE;
+}
+
+static bool qed_iov_validate_rxq(struct qed_hwfn *p_hwfn,
+ struct qed_vf_info *p_vf,
+ u16 rx_qid,
+ enum qed_iov_validate_q_mode mode)
+{
+ if (rx_qid >= p_vf->num_rxqs) {
DP_VERBOSE(p_hwfn,
QED_MSG_IOV,
"VF[0x%02x] - can't touch Rx queue[%04x]; Only 0x%04x are allocated\n",
p_vf->abs_vf_id, rx_qid, p_vf->num_rxqs);
- return rx_qid < p_vf->num_rxqs;
+ return false;
+ }
+
+ return qed_iov_validate_queue_mode(p_hwfn, p_vf, rx_qid, mode, false);
}
static bool qed_iov_validate_txq(struct qed_hwfn *p_hwfn,
- struct qed_vf_info *p_vf, u16 tx_qid)
+ struct qed_vf_info *p_vf,
+ u16 tx_qid,
+ enum qed_iov_validate_q_mode mode)
{
- if (tx_qid >= p_vf->num_txqs)
+ if (tx_qid >= p_vf->num_txqs) {
DP_VERBOSE(p_hwfn,
QED_MSG_IOV,
"VF[0x%02x] - can't touch Tx queue[%04x]; Only 0x%04x are allocated\n",
p_vf->abs_vf_id, tx_qid, p_vf->num_txqs);
- return tx_qid < p_vf->num_txqs;
+ return false;
+ }
+
+ return qed_iov_validate_queue_mode(p_hwfn, p_vf, tx_qid, mode, true);
}
static bool qed_iov_validate_sb(struct qed_hwfn *p_hwfn,
@@ -217,6 +250,34 @@
return false;
}
+static bool qed_iov_validate_active_rxq(struct qed_hwfn *p_hwfn,
+ struct qed_vf_info *p_vf)
+{
+ u8 i;
+
+ for (i = 0; i < p_vf->num_rxqs; i++)
+ if (qed_iov_validate_queue_mode(p_hwfn, p_vf, i,
+ QED_IOV_VALIDATE_Q_ENABLE,
+ false))
+ return true;
+
+ return false;
+}
+
+static bool qed_iov_validate_active_txq(struct qed_hwfn *p_hwfn,
+ struct qed_vf_info *p_vf)
+{
+ u8 i;
+
+ for (i = 0; i < p_vf->num_txqs; i++)
+ if (qed_iov_validate_queue_mode(p_hwfn, p_vf, i,
+ QED_IOV_VALIDATE_Q_ENABLE,
+ true))
+ return true;
+
+ return false;
+}
+
static int qed_iov_post_vf_bulletin(struct qed_hwfn *p_hwfn,
int vfid, struct qed_ptt *p_ptt)
{
@@ -557,14 +618,30 @@
return 0;
}
- /* Calculate the first VF index - this is a bit tricky; Basically,
- * VFs start at offset 16 relative to PF0, and 2nd engine VFs begin
- * after the first engine's VFs.
+ /* First VF index based on offset is tricky:
+ * - If ARI is supported [likely], offset - (16 - pf_id) would
+ * provide the number for eng0. 2nd engine Vfs would begin
+ * after the first engine's VFs.
+ * - If !ARI, VFs would start on next device.
+ * so offset - (256 - pf_id) would provide the number.
+ * Utilize the fact that (256 - pf_id) is achieved only by later
+ * to diffrentiate between the two.
*/
- cdev->p_iov_info->first_vf_in_pf = p_hwfn->cdev->p_iov_info->offset +
- p_hwfn->abs_pf_id - 16;
- if (QED_PATH_ID(p_hwfn))
- cdev->p_iov_info->first_vf_in_pf -= MAX_NUM_VFS_BB;
+
+ if (p_hwfn->cdev->p_iov_info->offset < (256 - p_hwfn->abs_pf_id)) {
+ u32 first = p_hwfn->cdev->p_iov_info->offset +
+ p_hwfn->abs_pf_id - 16;
+
+ cdev->p_iov_info->first_vf_in_pf = first;
+
+ if (QED_PATH_ID(p_hwfn))
+ cdev->p_iov_info->first_vf_in_pf -= MAX_NUM_VFS_BB;
+ } else {
+ u32 first = p_hwfn->cdev->p_iov_info->offset +
+ p_hwfn->abs_pf_id - 256;
+
+ cdev->p_iov_info->first_vf_in_pf = first;
+ }
DP_VERBOSE(p_hwfn, QED_MSG_IOV,
"First VF in hwfn 0x%08x\n",
@@ -677,6 +754,11 @@
u32 igu_vf_conf = IGU_VF_CONF_FUNC_EN;
int rc;
+ /* It's possible VF was previously considered malicious -
+ * clear the indication even if we're only going to disable VF.
+ */
+ vf->b_malicious = false;
+
if (vf->to_disable)
return 0;
@@ -689,9 +771,6 @@
qed_iov_vf_igu_reset(p_hwfn, p_ptt, vf);
- /* It's possible VF was previously considered malicious */
- vf->b_malicious = false;
-
rc = qed_mcp_config_vf_msix(p_hwfn, p_ptt, vf->abs_vf_id, vf->num_sbs);
if (rc)
return rc;
@@ -1118,13 +1197,17 @@
(sizeof(union pfvf_tlvs) - sizeof(u64)) / 4,
¶ms);
- qed_dmae_host2host(p_hwfn, p_ptt, mbx->reply_phys,
- mbx->req_virt->first_tlv.reply_address,
- sizeof(u64) / 4, ¶ms);
-
+ /* Once PF copies the rc to the VF, the latter can continue
+ * and send an additional message. So we have to make sure the
+ * channel would be re-set to ready prior to that.
+ */
REG_WR(p_hwfn,
GTT_BAR0_MAP_REG_USDM_RAM +
USTORM_VF_PF_CHANNEL_READY_OFFSET(eng_vf_id), 1);
+
+ qed_dmae_host2host(p_hwfn, p_ptt, mbx->reply_phys,
+ mbx->req_virt->first_tlv.reply_address,
+ sizeof(u64) / 4, ¶ms);
}
static u16 qed_iov_vport_to_tlv(struct qed_hwfn *p_hwfn,
@@ -1733,6 +1816,8 @@
vf->state = VF_ENABLED;
start = &mbx->req_virt->start_vport;
+ qed_iov_enable_vf_traffic(p_hwfn, p_ptt, vf);
+
/* Initialize Status block in CAU */
for (sb_id = 0; sb_id < vf->num_sbs; sb_id++) {
if (!start->sb_addr[sb_id]) {
@@ -1746,7 +1831,6 @@
start->sb_addr[sb_id],
vf->igu_sbs[sb_id], vf->abs_vf_id, 1);
}
- qed_iov_enable_vf_traffic(p_hwfn, p_ptt, vf);
vf->mtu = start->mtu;
vf->shadow_config.inner_vlan_removal = start->inner_vlan_removal;
@@ -1803,6 +1887,16 @@
vf->vport_instance--;
vf->spoof_chk = false;
+ if ((qed_iov_validate_active_rxq(p_hwfn, vf)) ||
+ (qed_iov_validate_active_txq(p_hwfn, vf))) {
+ vf->b_malicious = true;
+ DP_NOTICE(p_hwfn,
+ "VF [%02x] - considered malicious; Unable to stop RX/TX queuess\n",
+ vf->abs_vf_id);
+ status = PFVF_STATUS_MALICIOUS;
+ goto out;
+ }
+
rc = qed_sp_vport_stop(p_hwfn, vf->opaque_fid, vf->vport_id);
if (rc) {
DP_ERR(p_hwfn, "qed_iov_vf_mbx_stop_vport returned error %d\n",
@@ -1814,6 +1908,7 @@
vf->configured_features = 0;
memset(&vf->shadow_config, 0, sizeof(vf->shadow_config));
+out:
qed_iov_prepare_resp(p_hwfn, p_ptt, vf, CHANNEL_TLV_VPORT_TEARDOWN,
sizeof(struct pfvf_def_resp_tlv), status);
}
@@ -1870,7 +1965,8 @@
req = &mbx->req_virt->start_rxq;
- if (!qed_iov_validate_rxq(p_hwfn, vf, req->rx_qid) ||
+ if (!qed_iov_validate_rxq(p_hwfn, vf, req->rx_qid,
+ QED_IOV_VALIDATE_Q_DISABLE) ||
!qed_iov_validate_sb(p_hwfn, vf, req->hw_sb))
goto out;
@@ -1923,6 +2019,220 @@
qed_iov_vf_mbx_start_rxq_resp(p_hwfn, p_ptt, vf, status, b_legacy_vf);
}
+static void
+qed_iov_pf_update_tun_response(struct pfvf_update_tunn_param_tlv *p_resp,
+ struct qed_tunnel_info *p_tun,
+ u16 tunn_feature_mask)
+{
+ p_resp->tunn_feature_mask = tunn_feature_mask;
+ p_resp->vxlan_mode = p_tun->vxlan.b_mode_enabled;
+ p_resp->l2geneve_mode = p_tun->l2_geneve.b_mode_enabled;
+ p_resp->ipgeneve_mode = p_tun->ip_geneve.b_mode_enabled;
+ p_resp->l2gre_mode = p_tun->l2_gre.b_mode_enabled;
+ p_resp->ipgre_mode = p_tun->l2_gre.b_mode_enabled;
+ p_resp->vxlan_clss = p_tun->vxlan.tun_cls;
+ p_resp->l2gre_clss = p_tun->l2_gre.tun_cls;
+ p_resp->ipgre_clss = p_tun->ip_gre.tun_cls;
+ p_resp->l2geneve_clss = p_tun->l2_geneve.tun_cls;
+ p_resp->ipgeneve_clss = p_tun->ip_geneve.tun_cls;
+ p_resp->geneve_udp_port = p_tun->geneve_port.port;
+ p_resp->vxlan_udp_port = p_tun->vxlan_port.port;
+}
+
+static void
+__qed_iov_pf_update_tun_param(struct vfpf_update_tunn_param_tlv *p_req,
+ struct qed_tunn_update_type *p_tun,
+ enum qed_tunn_mode mask, u8 tun_cls)
+{
+ if (p_req->tun_mode_update_mask & BIT(mask)) {
+ p_tun->b_update_mode = true;
+
+ if (p_req->tunn_mode & BIT(mask))
+ p_tun->b_mode_enabled = true;
+ }
+
+ p_tun->tun_cls = tun_cls;
+}
+
+static void
+qed_iov_pf_update_tun_param(struct vfpf_update_tunn_param_tlv *p_req,
+ struct qed_tunn_update_type *p_tun,
+ struct qed_tunn_update_udp_port *p_port,
+ enum qed_tunn_mode mask,
+ u8 tun_cls, u8 update_port, u16 port)
+{
+ if (update_port) {
+ p_port->b_update_port = true;
+ p_port->port = port;
+ }
+
+ __qed_iov_pf_update_tun_param(p_req, p_tun, mask, tun_cls);
+}
+
+static bool
+qed_iov_pf_validate_tunn_param(struct vfpf_update_tunn_param_tlv *p_req)
+{
+ bool b_update_requested = false;
+
+ if (p_req->tun_mode_update_mask || p_req->update_tun_cls ||
+ p_req->update_geneve_port || p_req->update_vxlan_port)
+ b_update_requested = true;
+
+ return b_update_requested;
+}
+
+static void qed_pf_validate_tunn_mode(struct qed_tunn_update_type *tun, int *rc)
+{
+ if (tun->b_update_mode && !tun->b_mode_enabled) {
+ tun->b_update_mode = false;
+ *rc = -EINVAL;
+ }
+}
+
+static int
+qed_pf_validate_modify_tunn_config(struct qed_hwfn *p_hwfn,
+ u16 *tun_features, bool *update,
+ struct qed_tunnel_info *tun_src)
+{
+ struct qed_eth_cb_ops *ops = p_hwfn->cdev->protocol_ops.eth;
+ struct qed_tunnel_info *tun = &p_hwfn->cdev->tunnel;
+ u16 bultn_vxlan_port, bultn_geneve_port;
+ void *cookie = p_hwfn->cdev->ops_cookie;
+ int i, rc = 0;
+
+ *tun_features = p_hwfn->cdev->tunn_feature_mask;
+ bultn_vxlan_port = tun->vxlan_port.port;
+ bultn_geneve_port = tun->geneve_port.port;
+ qed_pf_validate_tunn_mode(&tun_src->vxlan, &rc);
+ qed_pf_validate_tunn_mode(&tun_src->l2_geneve, &rc);
+ qed_pf_validate_tunn_mode(&tun_src->ip_geneve, &rc);
+ qed_pf_validate_tunn_mode(&tun_src->l2_gre, &rc);
+ qed_pf_validate_tunn_mode(&tun_src->ip_gre, &rc);
+
+ if ((tun_src->b_update_rx_cls || tun_src->b_update_tx_cls) &&
+ (tun_src->vxlan.tun_cls != QED_TUNN_CLSS_MAC_VLAN ||
+ tun_src->l2_geneve.tun_cls != QED_TUNN_CLSS_MAC_VLAN ||
+ tun_src->ip_geneve.tun_cls != QED_TUNN_CLSS_MAC_VLAN ||
+ tun_src->l2_gre.tun_cls != QED_TUNN_CLSS_MAC_VLAN ||
+ tun_src->ip_gre.tun_cls != QED_TUNN_CLSS_MAC_VLAN)) {
+ tun_src->b_update_rx_cls = false;
+ tun_src->b_update_tx_cls = false;
+ rc = -EINVAL;
+ }
+
+ if (tun_src->vxlan_port.b_update_port) {
+ if (tun_src->vxlan_port.port == tun->vxlan_port.port) {
+ tun_src->vxlan_port.b_update_port = false;
+ } else {
+ *update = true;
+ bultn_vxlan_port = tun_src->vxlan_port.port;
+ }
+ }
+
+ if (tun_src->geneve_port.b_update_port) {
+ if (tun_src->geneve_port.port == tun->geneve_port.port) {
+ tun_src->geneve_port.b_update_port = false;
+ } else {
+ *update = true;
+ bultn_geneve_port = tun_src->geneve_port.port;
+ }
+ }
+
+ qed_for_each_vf(p_hwfn, i) {
+ qed_iov_bulletin_set_udp_ports(p_hwfn, i, bultn_vxlan_port,
+ bultn_geneve_port);
+ }
+
+ qed_schedule_iov(p_hwfn, QED_IOV_WQ_BULLETIN_UPDATE_FLAG);
+ ops->ports_update(cookie, bultn_vxlan_port, bultn_geneve_port);
+
+ return rc;
+}
+
+static void qed_iov_vf_mbx_update_tunn_param(struct qed_hwfn *p_hwfn,
+ struct qed_ptt *p_ptt,
+ struct qed_vf_info *p_vf)
+{
+ struct qed_tunnel_info *p_tun = &p_hwfn->cdev->tunnel;
+ struct qed_iov_vf_mbx *mbx = &p_vf->vf_mbx;
+ struct pfvf_update_tunn_param_tlv *p_resp;
+ struct vfpf_update_tunn_param_tlv *p_req;
+ u8 status = PFVF_STATUS_SUCCESS;
+ bool b_update_required = false;
+ struct qed_tunnel_info tunn;
+ u16 tunn_feature_mask = 0;
+ int i, rc = 0;
+
+ mbx->offset = (u8 *)mbx->reply_virt;
+
+ memset(&tunn, 0, sizeof(tunn));
+ p_req = &mbx->req_virt->tunn_param_update;
+
+ if (!qed_iov_pf_validate_tunn_param(p_req)) {
+ DP_VERBOSE(p_hwfn, QED_MSG_IOV,
+ "No tunnel update requested by VF\n");
+ status = PFVF_STATUS_FAILURE;
+ goto send_resp;
+ }
+
+ tunn.b_update_rx_cls = p_req->update_tun_cls;
+ tunn.b_update_tx_cls = p_req->update_tun_cls;
+
+ qed_iov_pf_update_tun_param(p_req, &tunn.vxlan, &tunn.vxlan_port,
+ QED_MODE_VXLAN_TUNN, p_req->vxlan_clss,
+ p_req->update_vxlan_port,
+ p_req->vxlan_port);
+ qed_iov_pf_update_tun_param(p_req, &tunn.l2_geneve, &tunn.geneve_port,
+ QED_MODE_L2GENEVE_TUNN,
+ p_req->l2geneve_clss,
+ p_req->update_geneve_port,
+ p_req->geneve_port);
+ __qed_iov_pf_update_tun_param(p_req, &tunn.ip_geneve,
+ QED_MODE_IPGENEVE_TUNN,
+ p_req->ipgeneve_clss);
+ __qed_iov_pf_update_tun_param(p_req, &tunn.l2_gre,
+ QED_MODE_L2GRE_TUNN, p_req->l2gre_clss);
+ __qed_iov_pf_update_tun_param(p_req, &tunn.ip_gre,
+ QED_MODE_IPGRE_TUNN, p_req->ipgre_clss);
+
+ /* If PF modifies VF's req then it should
+ * still return an error in case of partial configuration
+ * or modified configuration as opposed to requested one.
+ */
+ rc = qed_pf_validate_modify_tunn_config(p_hwfn, &tunn_feature_mask,
+ &b_update_required, &tunn);
+
+ if (rc)
+ status = PFVF_STATUS_FAILURE;
+
+ /* If QED client is willing to update anything ? */
+ if (b_update_required) {
+ u16 geneve_port;
+
+ rc = qed_sp_pf_update_tunn_cfg(p_hwfn, &tunn,
+ QED_SPQ_MODE_EBLOCK, NULL);
+ if (rc)
+ status = PFVF_STATUS_FAILURE;
+
+ geneve_port = p_tun->geneve_port.port;
+ qed_for_each_vf(p_hwfn, i) {
+ qed_iov_bulletin_set_udp_ports(p_hwfn, i,
+ p_tun->vxlan_port.port,
+ geneve_port);
+ }
+ }
+
+send_resp:
+ p_resp = qed_add_tlv(p_hwfn, &mbx->offset,
+ CHANNEL_TLV_UPDATE_TUNN_PARAM, sizeof(*p_resp));
+
+ qed_iov_pf_update_tun_response(p_resp, p_tun, tunn_feature_mask);
+ qed_add_tlv(p_hwfn, &mbx->offset, CHANNEL_TLV_LIST_END,
+ sizeof(struct channel_list_end_tlv));
+
+ qed_iov_send_response(p_hwfn, p_ptt, p_vf, sizeof(*p_resp), status);
+}
+
static void qed_iov_vf_mbx_start_txq_resp(struct qed_hwfn *p_hwfn,
struct qed_ptt *p_ptt,
struct qed_vf_info *p_vf, u8 status)
@@ -1970,21 +2280,16 @@
struct qed_queue_start_common_params params;
struct qed_iov_vf_mbx *mbx = &vf->vf_mbx;
u8 status = PFVF_STATUS_NO_RESOURCE;
- union qed_qm_pq_params pq_params;
struct vfpf_start_txq_tlv *req;
struct qed_vf_q_info *p_queue;
int rc;
u16 pq;
- /* Prepare the parameters which would choose the right PQ */
- memset(&pq_params, 0, sizeof(pq_params));
- pq_params.eth.is_vf = 1;
- pq_params.eth.vf_id = vf->relative_vf_id;
-
memset(¶ms, 0, sizeof(params));
req = &mbx->req_virt->start_txq;
- if (!qed_iov_validate_txq(p_hwfn, vf, req->tx_qid) ||
+ if (!qed_iov_validate_txq(p_hwfn, vf, req->tx_qid,
+ QED_IOV_VALIDATE_Q_DISABLE) ||
!qed_iov_validate_sb(p_hwfn, vf, req->hw_sb))
goto out;
@@ -2004,7 +2309,7 @@
if (!p_queue->p_tx_cid)
goto out;
- pq = qed_get_qm_pq(p_hwfn, PROTOCOLID_ETH, &pq_params);
+ pq = qed_get_cm_pq_idx_vf(p_hwfn, vf->relative_vf_id);
rc = qed_eth_txq_start_ramrod(p_hwfn, p_queue->p_tx_cid,
req->pbl_addr, req->pbl_size, pq);
if (rc) {
@@ -2021,57 +2326,53 @@
static int qed_iov_vf_stop_rxqs(struct qed_hwfn *p_hwfn,
struct qed_vf_info *vf,
- u16 rxq_id, u8 num_rxqs, bool cqe_completion)
+ u16 rxq_id, bool cqe_completion)
{
struct qed_vf_q_info *p_queue;
int rc = 0;
- int qid;
- if (rxq_id + num_rxqs > ARRAY_SIZE(vf->vf_queues))
+ if (!qed_iov_validate_rxq(p_hwfn, vf, rxq_id,
+ QED_IOV_VALIDATE_Q_ENABLE)) {
+ DP_VERBOSE(p_hwfn,
+ QED_MSG_IOV,
+ "VF[%d] Tried Closing Rx 0x%04x which is inactive\n",
+ vf->relative_vf_id, rxq_id);
return -EINVAL;
-
- for (qid = rxq_id; qid < rxq_id + num_rxqs; qid++) {
- p_queue = &vf->vf_queues[qid];
-
- if (!p_queue->p_rx_cid)
- continue;
-
- rc = qed_eth_rx_queue_stop(p_hwfn,
- p_queue->p_rx_cid,
- false, cqe_completion);
- if (rc)
- return rc;
-
- vf->vf_queues[qid].p_rx_cid = NULL;
- vf->num_active_rxqs--;
}
- return rc;
+ p_queue = &vf->vf_queues[rxq_id];
+
+ rc = qed_eth_rx_queue_stop(p_hwfn,
+ p_queue->p_rx_cid,
+ false, cqe_completion);
+ if (rc)
+ return rc;
+
+ p_queue->p_rx_cid = NULL;
+ vf->num_active_rxqs--;
+
+ return 0;
}
static int qed_iov_vf_stop_txqs(struct qed_hwfn *p_hwfn,
- struct qed_vf_info *vf, u16 txq_id, u8 num_txqs)
+ struct qed_vf_info *vf, u16 txq_id)
{
- int rc = 0;
struct qed_vf_q_info *p_queue;
- int qid;
+ int rc = 0;
- if (txq_id + num_txqs > ARRAY_SIZE(vf->vf_queues))
+ if (!qed_iov_validate_txq(p_hwfn, vf, txq_id,
+ QED_IOV_VALIDATE_Q_ENABLE))
return -EINVAL;
- for (qid = txq_id; qid < txq_id + num_txqs; qid++) {
- p_queue = &vf->vf_queues[qid];
- if (!p_queue->p_tx_cid)
- continue;
+ p_queue = &vf->vf_queues[txq_id];
- rc = qed_eth_tx_queue_stop(p_hwfn, p_queue->p_tx_cid);
- if (rc)
- return rc;
+ rc = qed_eth_tx_queue_stop(p_hwfn, p_queue->p_tx_cid);
+ if (rc)
+ return rc;
- p_queue->p_tx_cid = NULL;
- }
+ p_queue->p_tx_cid = NULL;
- return rc;
+ return 0;
}
static void qed_iov_vf_mbx_stop_rxqs(struct qed_hwfn *p_hwfn,
@@ -2080,20 +2381,28 @@
{
u16 length = sizeof(struct pfvf_def_resp_tlv);
struct qed_iov_vf_mbx *mbx = &vf->vf_mbx;
- u8 status = PFVF_STATUS_SUCCESS;
+ u8 status = PFVF_STATUS_FAILURE;
struct vfpf_stop_rxqs_tlv *req;
int rc;
- /* We give the option of starting from qid != 0, in this case we
- * need to make sure that qid + num_qs doesn't exceed the actual
- * amount of queues that exist.
+ /* There has never been an official driver that used this interface
+ * for stopping multiple queues, and it is now considered deprecated.
+ * Validate this isn't used here.
*/
req = &mbx->req_virt->stop_rxqs;
- rc = qed_iov_vf_stop_rxqs(p_hwfn, vf, req->rx_qid,
- req->num_rxqs, req->cqe_completion);
- if (rc)
- status = PFVF_STATUS_FAILURE;
+ if (req->num_rxqs != 1) {
+ DP_VERBOSE(p_hwfn, QED_MSG_IOV,
+ "Odd; VF[%d] tried stopping multiple Rx queues\n",
+ vf->relative_vf_id);
+ status = PFVF_STATUS_NOT_SUPPORTED;
+ goto out;
+ }
+ rc = qed_iov_vf_stop_rxqs(p_hwfn, vf, req->rx_qid,
+ req->cqe_completion);
+ if (!rc)
+ status = PFVF_STATUS_SUCCESS;
+out:
qed_iov_prepare_resp(p_hwfn, p_ptt, vf, CHANNEL_TLV_STOP_RXQS,
length, status);
}
@@ -2104,19 +2413,27 @@
{
u16 length = sizeof(struct pfvf_def_resp_tlv);
struct qed_iov_vf_mbx *mbx = &vf->vf_mbx;
- u8 status = PFVF_STATUS_SUCCESS;
+ u8 status = PFVF_STATUS_FAILURE;
struct vfpf_stop_txqs_tlv *req;
int rc;
- /* We give the option of starting from qid != 0, in this case we
- * need to make sure that qid + num_qs doesn't exceed the actual
- * amount of queues that exist.
+ /* There has never been an official driver that used this interface
+ * for stopping multiple queues, and it is now considered deprecated.
+ * Validate this isn't used here.
*/
req = &mbx->req_virt->stop_txqs;
- rc = qed_iov_vf_stop_txqs(p_hwfn, vf, req->tx_qid, req->num_txqs);
- if (rc)
- status = PFVF_STATUS_FAILURE;
+ if (req->num_txqs != 1) {
+ DP_VERBOSE(p_hwfn, QED_MSG_IOV,
+ "Odd; VF[%d] tried stopping multiple Tx queues\n",
+ vf->relative_vf_id);
+ status = PFVF_STATUS_NOT_SUPPORTED;
+ goto out;
+ }
+ rc = qed_iov_vf_stop_txqs(p_hwfn, vf, req->tx_qid);
+ if (!rc)
+ status = PFVF_STATUS_SUCCESS;
+out:
qed_iov_prepare_resp(p_hwfn, p_ptt, vf, CHANNEL_TLV_STOP_TXQS,
length, status);
}
@@ -2141,22 +2458,17 @@
complete_event_flg = !!(req->flags & VFPF_RXQ_UPD_COMPLETE_EVENT_FLAG);
/* Validate inputs */
- if (req->num_rxqs + req->rx_qid > QED_MAX_VF_CHAINS_PER_PF ||
- !qed_iov_validate_rxq(p_hwfn, vf, req->rx_qid)) {
- DP_INFO(p_hwfn, "VF[%d]: Incorrect Rxqs [%04x, %02x]\n",
- vf->relative_vf_id, req->rx_qid, req->num_rxqs);
- goto out;
- }
-
- for (i = 0; i < req->num_rxqs; i++) {
- qid = req->rx_qid + i;
- if (!vf->vf_queues[qid].p_rx_cid) {
- DP_INFO(p_hwfn,
- "VF[%d] rx_qid = %d isn`t active!\n",
- vf->relative_vf_id, qid);
+ for (i = req->rx_qid; i < req->rx_qid + req->num_rxqs; i++)
+ if (!qed_iov_validate_rxq(p_hwfn, vf, i,
+ QED_IOV_VALIDATE_Q_ENABLE)) {
+ DP_INFO(p_hwfn, "VF[%d]: Incorrect Rxqs [%04x, %02x]\n",
+ vf->relative_vf_id, req->rx_qid, req->num_rxqs);
goto out;
}
+ /* Prepare the handlers */
+ for (i = 0; i < req->num_rxqs; i++) {
+ qid = req->rx_qid + i;
handlers[i] = vf->vf_queues[qid].p_rx_cid;
}
@@ -2372,7 +2684,8 @@
for (i = 0; i < table_size; i++) {
q_idx = p_rss_tlv->rss_ind_table[i];
- if (!qed_iov_validate_rxq(p_hwfn, vf, q_idx)) {
+ if (!qed_iov_validate_rxq(p_hwfn, vf, q_idx,
+ QED_IOV_VALIDATE_Q_ENABLE)) {
DP_VERBOSE(p_hwfn,
QED_MSG_IOV,
"VF[%d]: Omitting RSS due to wrong queue %04x\n",
@@ -2381,15 +2694,6 @@
goto out;
}
- if (!vf->vf_queues[q_idx].p_rx_cid) {
- DP_VERBOSE(p_hwfn,
- QED_MSG_IOV,
- "VF[%d]: Omitting RSS due to inactive queue %08x\n",
- vf->relative_vf_id, q_idx);
- b_reject = true;
- goto out;
- }
-
p_rss->rss_ind_table[i] = vf->vf_queues[q_idx].p_rx_cid;
}
@@ -3042,9 +3346,10 @@
return rc;
}
-int qed_iov_mark_vf_flr(struct qed_hwfn *p_hwfn, u32 *p_disabled_vfs)
+bool qed_iov_mark_vf_flr(struct qed_hwfn *p_hwfn, u32 *p_disabled_vfs)
{
- u16 i, found = 0;
+ bool found = false;
+ u16 i;
DP_VERBOSE(p_hwfn, QED_MSG_IOV, "Marking FLR-ed VFs\n");
for (i = 0; i < (VF_MAX_STATIC / 32); i++)
@@ -3054,7 +3359,7 @@
if (!p_hwfn->cdev->p_iov_info) {
DP_NOTICE(p_hwfn, "VF flr but no IOV\n");
- return 0;
+ return false;
}
/* Mark VFs */
@@ -3083,7 +3388,7 @@
* VF flr until ACKs, we're safe.
*/
p_flr[rel_vf_id / 64] |= 1ULL << (rel_vf_id % 64);
- found = 1;
+ found = true;
}
}
@@ -3184,6 +3489,9 @@
case CHANNEL_TLV_RELEASE:
qed_iov_vf_mbx_release(p_hwfn, p_ptt, p_vf);
break;
+ case CHANNEL_TLV_UPDATE_TUNN_PARAM:
+ qed_iov_vf_mbx_update_tunn_param(p_hwfn, p_ptt, p_vf);
+ break;
}
} else if (qed_iov_tlv_supported(mbx->first_tlv.tl.type)) {
DP_VERBOSE(p_hwfn, QED_MSG_IOV,
@@ -3289,11 +3597,17 @@
if (!p_vf)
return;
- DP_INFO(p_hwfn,
- "VF [%d] - Malicious behavior [%02x]\n",
- p_vf->abs_vf_id, p_data->err_id);
+ if (!p_vf->b_malicious) {
+ DP_NOTICE(p_hwfn,
+ "VF [%d] - Malicious behavior [%02x]\n",
+ p_vf->abs_vf_id, p_data->err_id);
- p_vf->b_malicious = true;
+ p_vf->b_malicious = true;
+ } else {
+ DP_INFO(p_hwfn,
+ "VF [%d] - Malicious behavior [%02x]\n",
+ p_vf->abs_vf_id, p_data->err_id);
+ }
}
int qed_sriov_eqe_event(struct qed_hwfn *p_hwfn,
@@ -3414,6 +3728,29 @@
qed_iov_configure_vport_forced(p_hwfn, vf_info, feature);
}
+void qed_iov_bulletin_set_udp_ports(struct qed_hwfn *p_hwfn,
+ int vfid, u16 vxlan_port, u16 geneve_port)
+{
+ struct qed_vf_info *vf_info;
+
+ vf_info = qed_iov_get_vf_info(p_hwfn, (u16)vfid, true);
+ if (!vf_info) {
+ DP_NOTICE(p_hwfn->cdev,
+ "Can not set udp ports, invalid vfid [%d]\n", vfid);
+ return;
+ }
+
+ if (vf_info->b_malicious) {
+ DP_VERBOSE(p_hwfn, QED_MSG_IOV,
+ "Can not set udp ports to malicious VF [%d]\n",
+ vfid);
+ return;
+ }
+
+ vf_info->bulletin.p_virt->vxlan_udp_port = vxlan_port;
+ vf_info->bulletin.p_virt->geneve_udp_port = geneve_port;
+}
+
static bool qed_iov_vf_has_vport_instance(struct qed_hwfn *p_hwfn, int vfid)
{
struct qed_vf_info *p_vf_info;
@@ -3842,6 +4179,7 @@
void qed_inform_vf_link_state(struct qed_hwfn *hwfn)
{
+ struct qed_hwfn *lead_hwfn = QED_LEADING_HWFN(hwfn->cdev);
struct qed_mcp_link_capabilities caps;
struct qed_mcp_link_params params;
struct qed_mcp_link_state link;
@@ -3858,9 +4196,15 @@
if (!vf_info)
continue;
- memcpy(¶ms, qed_mcp_get_link_params(hwfn), sizeof(params));
- memcpy(&link, qed_mcp_get_link_state(hwfn), sizeof(link));
- memcpy(&caps, qed_mcp_get_link_capabilities(hwfn),
+ /* Only hwfn0 is actually interested in the link speed.
+ * But since only it would receive an MFW indication of link,
+ * need to take configuration from it - otherwise things like
+ * rate limiting for hwfn1 VF would not work.
+ */
+ memcpy(¶ms, qed_mcp_get_link_params(lead_hwfn),
+ sizeof(params));
+ memcpy(&link, qed_mcp_get_link_state(lead_hwfn), sizeof(link));
+ memcpy(&caps, qed_mcp_get_link_capabilities(lead_hwfn),
sizeof(caps));
/* Modify link according to the VF's configured link state */
diff --git a/drivers/net/ethernet/qlogic/qed/qed_sriov.h b/drivers/net/ethernet/qlogic/qed/qed_sriov.h
index a896058..81a497c 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_sriov.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_sriov.h
@@ -270,6 +270,9 @@
*/
u16 qed_iov_get_next_active_vf(struct qed_hwfn *p_hwfn, u16 rel_vf_id);
+void qed_iov_bulletin_set_udp_ports(struct qed_hwfn *p_hwfn,
+ int vfid, u16 vxlan_port, u16 geneve_port);
+
/**
* @brief Read sriov related information and allocated resources
* reads from configuraiton space, shmem, etc.
@@ -348,9 +351,9 @@
* @param p_hwfn
* @param disabled_vfs - bitmask of all VFs on path that were FLRed
*
- * @return 1 iff one of the PF's vfs got FLRed. 0 otherwise.
+ * @return true iff one of the PF's vfs got FLRed. false otherwise.
*/
-int qed_iov_mark_vf_flr(struct qed_hwfn *p_hwfn, u32 *disabled_vfs);
+bool qed_iov_mark_vf_flr(struct qed_hwfn *p_hwfn, u32 *disabled_vfs);
/**
* @brief Search extended TLVs in request/reply buffer.
@@ -378,6 +381,12 @@
return MAX_NUM_VFS;
}
+static inline void
+qed_iov_bulletin_set_udp_ports(struct qed_hwfn *p_hwfn, int vfid,
+ u16 vxlan_port, u16 geneve_port)
+{
+}
+
static inline int qed_iov_hw_info(struct qed_hwfn *p_hwfn)
{
return 0;
@@ -407,10 +416,10 @@
return -EINVAL;
}
-static inline int qed_iov_mark_vf_flr(struct qed_hwfn *p_hwfn,
- u32 *disabled_vfs)
+static inline bool qed_iov_mark_vf_flr(struct qed_hwfn *p_hwfn,
+ u32 *disabled_vfs)
{
- return 0;
+ return false;
}
static inline void qed_iov_wq_stop(struct qed_dev *cdev, bool schedule_first)
diff --git a/drivers/net/ethernet/qlogic/qed/qed_vf.c b/drivers/net/ethernet/qlogic/qed/qed_vf.c
index 15d2855..11d71e5 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_vf.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_vf.c
@@ -134,14 +134,20 @@
}
if (!*done) {
- DP_VERBOSE(p_hwfn, QED_MSG_IOV,
- "VF <-- PF Timeout [Type %d]\n",
- p_req->first_tlv.tl.type);
+ DP_NOTICE(p_hwfn,
+ "VF <-- PF Timeout [Type %d]\n",
+ p_req->first_tlv.tl.type);
rc = -EBUSY;
} else {
- DP_VERBOSE(p_hwfn, QED_MSG_IOV,
- "PF response: %d [Type %d]\n",
- *done, p_req->first_tlv.tl.type);
+ if ((*done != PFVF_STATUS_SUCCESS) &&
+ (*done != PFVF_STATUS_NO_RESOURCE))
+ DP_NOTICE(p_hwfn,
+ "PF response: %d [Type %d]\n",
+ *done, p_req->first_tlv.tl.type);
+ else
+ DP_VERBOSE(p_hwfn, QED_MSG_IOV,
+ "PF response: %d [Type %d]\n",
+ *done, p_req->first_tlv.tl.type);
}
return rc;
@@ -228,7 +234,7 @@
/* send acquire request */
rc = qed_send_msg2pf(p_hwfn, &resp->hdr.status, sizeof(*resp));
if (rc)
- return rc;
+ goto exit;
/* copy acquire response from buffer to p_hwfn */
memcpy(&p_iov->acquire_resp, resp, sizeof(p_iov->acquire_resp));
@@ -412,6 +418,155 @@
#define MSTORM_QZONE_START(dev) (TSTORM_QZONE_START + \
(TSTORM_QZONE_SIZE * NUM_OF_L2_QUEUES(dev)))
+static void
+__qed_vf_prep_tunn_req_tlv(struct vfpf_update_tunn_param_tlv *p_req,
+ struct qed_tunn_update_type *p_src,
+ enum qed_tunn_clss mask, u8 *p_cls)
+{
+ if (p_src->b_update_mode) {
+ p_req->tun_mode_update_mask |= BIT(mask);
+
+ if (p_src->b_mode_enabled)
+ p_req->tunn_mode |= BIT(mask);
+ }
+
+ *p_cls = p_src->tun_cls;
+}
+
+static void
+qed_vf_prep_tunn_req_tlv(struct vfpf_update_tunn_param_tlv *p_req,
+ struct qed_tunn_update_type *p_src,
+ enum qed_tunn_clss mask,
+ u8 *p_cls, struct qed_tunn_update_udp_port *p_port,
+ u8 *p_update_port, u16 *p_udp_port)
+{
+ if (p_port->b_update_port) {
+ *p_update_port = 1;
+ *p_udp_port = p_port->port;
+ }
+
+ __qed_vf_prep_tunn_req_tlv(p_req, p_src, mask, p_cls);
+}
+
+void qed_vf_set_vf_start_tunn_update_param(struct qed_tunnel_info *p_tun)
+{
+ if (p_tun->vxlan.b_mode_enabled)
+ p_tun->vxlan.b_update_mode = true;
+ if (p_tun->l2_geneve.b_mode_enabled)
+ p_tun->l2_geneve.b_update_mode = true;
+ if (p_tun->ip_geneve.b_mode_enabled)
+ p_tun->ip_geneve.b_update_mode = true;
+ if (p_tun->l2_gre.b_mode_enabled)
+ p_tun->l2_gre.b_update_mode = true;
+ if (p_tun->ip_gre.b_mode_enabled)
+ p_tun->ip_gre.b_update_mode = true;
+
+ p_tun->b_update_rx_cls = true;
+ p_tun->b_update_tx_cls = true;
+}
+
+static void
+__qed_vf_update_tunn_param(struct qed_tunn_update_type *p_tun,
+ u16 feature_mask, u8 tunn_mode,
+ u8 tunn_cls, enum qed_tunn_mode val)
+{
+ if (feature_mask & BIT(val)) {
+ p_tun->b_mode_enabled = tunn_mode;
+ p_tun->tun_cls = tunn_cls;
+ } else {
+ p_tun->b_mode_enabled = false;
+ }
+}
+
+static void qed_vf_update_tunn_param(struct qed_hwfn *p_hwfn,
+ struct qed_tunnel_info *p_tun,
+ struct pfvf_update_tunn_param_tlv *p_resp)
+{
+ /* Update mode and classes provided by PF */
+ u16 feat_mask = p_resp->tunn_feature_mask;
+
+ __qed_vf_update_tunn_param(&p_tun->vxlan, feat_mask,
+ p_resp->vxlan_mode, p_resp->vxlan_clss,
+ QED_MODE_VXLAN_TUNN);
+ __qed_vf_update_tunn_param(&p_tun->l2_geneve, feat_mask,
+ p_resp->l2geneve_mode,
+ p_resp->l2geneve_clss,
+ QED_MODE_L2GENEVE_TUNN);
+ __qed_vf_update_tunn_param(&p_tun->ip_geneve, feat_mask,
+ p_resp->ipgeneve_mode,
+ p_resp->ipgeneve_clss,
+ QED_MODE_IPGENEVE_TUNN);
+ __qed_vf_update_tunn_param(&p_tun->l2_gre, feat_mask,
+ p_resp->l2gre_mode, p_resp->l2gre_clss,
+ QED_MODE_L2GRE_TUNN);
+ __qed_vf_update_tunn_param(&p_tun->ip_gre, feat_mask,
+ p_resp->ipgre_mode, p_resp->ipgre_clss,
+ QED_MODE_IPGRE_TUNN);
+ p_tun->geneve_port.port = p_resp->geneve_udp_port;
+ p_tun->vxlan_port.port = p_resp->vxlan_udp_port;
+
+ DP_VERBOSE(p_hwfn, QED_MSG_IOV,
+ "tunn mode: vxlan=0x%x, l2geneve=0x%x, ipgeneve=0x%x, l2gre=0x%x, ipgre=0x%x",
+ p_tun->vxlan.b_mode_enabled, p_tun->l2_geneve.b_mode_enabled,
+ p_tun->ip_geneve.b_mode_enabled,
+ p_tun->l2_gre.b_mode_enabled, p_tun->ip_gre.b_mode_enabled);
+}
+
+int qed_vf_pf_tunnel_param_update(struct qed_hwfn *p_hwfn,
+ struct qed_tunnel_info *p_src)
+{
+ struct qed_tunnel_info *p_tun = &p_hwfn->cdev->tunnel;
+ struct qed_vf_iov *p_iov = p_hwfn->vf_iov_info;
+ struct pfvf_update_tunn_param_tlv *p_resp;
+ struct vfpf_update_tunn_param_tlv *p_req;
+ int rc;
+
+ p_req = qed_vf_pf_prep(p_hwfn, CHANNEL_TLV_UPDATE_TUNN_PARAM,
+ sizeof(*p_req));
+
+ if (p_src->b_update_rx_cls && p_src->b_update_tx_cls)
+ p_req->update_tun_cls = 1;
+
+ qed_vf_prep_tunn_req_tlv(p_req, &p_src->vxlan, QED_MODE_VXLAN_TUNN,
+ &p_req->vxlan_clss, &p_src->vxlan_port,
+ &p_req->update_vxlan_port,
+ &p_req->vxlan_port);
+ qed_vf_prep_tunn_req_tlv(p_req, &p_src->l2_geneve,
+ QED_MODE_L2GENEVE_TUNN,
+ &p_req->l2geneve_clss, &p_src->geneve_port,
+ &p_req->update_geneve_port,
+ &p_req->geneve_port);
+ __qed_vf_prep_tunn_req_tlv(p_req, &p_src->ip_geneve,
+ QED_MODE_IPGENEVE_TUNN,
+ &p_req->ipgeneve_clss);
+ __qed_vf_prep_tunn_req_tlv(p_req, &p_src->l2_gre,
+ QED_MODE_L2GRE_TUNN, &p_req->l2gre_clss);
+ __qed_vf_prep_tunn_req_tlv(p_req, &p_src->ip_gre,
+ QED_MODE_IPGRE_TUNN, &p_req->ipgre_clss);
+
+ /* add list termination tlv */
+ qed_add_tlv(p_hwfn, &p_iov->offset,
+ CHANNEL_TLV_LIST_END,
+ sizeof(struct channel_list_end_tlv));
+
+ p_resp = &p_iov->pf2vf_reply->tunn_param_resp;
+ rc = qed_send_msg2pf(p_hwfn, &p_resp->hdr.status, sizeof(*p_resp));
+
+ if (rc)
+ goto exit;
+
+ if (p_resp->hdr.status != PFVF_STATUS_SUCCESS) {
+ DP_VERBOSE(p_hwfn, QED_MSG_IOV,
+ "Failed to update tunnel parameters\n");
+ rc = -EINVAL;
+ }
+
+ qed_vf_update_tunn_param(p_hwfn, p_tun, p_resp);
+exit:
+ qed_vf_pf_req_end(p_hwfn, rc);
+ return rc;
+}
+
int
qed_vf_pf_rxq_start(struct qed_hwfn *p_hwfn,
struct qed_queue_cid *p_cid,
@@ -1245,6 +1400,18 @@
return true;
}
+static void
+qed_vf_bulletin_get_udp_ports(struct qed_hwfn *p_hwfn,
+ u16 *p_vxlan_port, u16 *p_geneve_port)
+{
+ struct qed_bulletin_content *p_bulletin;
+
+ p_bulletin = &p_hwfn->vf_iov_info->bulletin_shadow;
+
+ *p_vxlan_port = p_bulletin->vxlan_udp_port;
+ *p_geneve_port = p_bulletin->geneve_udp_port;
+}
+
void qed_vf_get_fw_version(struct qed_hwfn *p_hwfn,
u16 *fw_major, u16 *fw_minor,
u16 *fw_rev, u16 *fw_eng)
@@ -1264,12 +1431,16 @@
struct qed_eth_cb_ops *ops = hwfn->cdev->protocol_ops.eth;
u8 mac[ETH_ALEN], is_mac_exist, is_mac_forced;
void *cookie = hwfn->cdev->ops_cookie;
+ u16 vxlan_port, geneve_port;
+ qed_vf_bulletin_get_udp_ports(hwfn, &vxlan_port, &geneve_port);
is_mac_exist = qed_vf_bulletin_get_forced_mac(hwfn, mac,
&is_mac_forced);
if (is_mac_exist && cookie)
ops->force_mac(cookie, mac, !!is_mac_forced);
+ ops->ports_update(cookie, vxlan_port, geneve_port);
+
/* Always update link configuration according to bulletin */
qed_link_update(hwfn);
}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_vf.h b/drivers/net/ethernet/qlogic/qed/qed_vf.h
index 7da0b16..34ac70b 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_vf.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_vf.h
@@ -275,6 +275,8 @@
struct vfpf_first_tlv first_tlv;
u16 rx_qid;
+
+ /* this field is deprecated and should *always* be set to '1' */
u8 num_rxqs;
u8 cqe_completion;
u8 padding[4];
@@ -285,6 +287,8 @@
struct vfpf_first_tlv first_tlv;
u16 tx_qid;
+
+ /* this field is deprecated and should *always* be set to '1' */
u8 num_txqs;
u8 padding[5];
};
@@ -425,6 +429,43 @@
u16 padding[3];
};
+/* tunnel update param tlv */
+struct vfpf_update_tunn_param_tlv {
+ struct vfpf_first_tlv first_tlv;
+
+ u8 tun_mode_update_mask;
+ u8 tunn_mode;
+ u8 update_tun_cls;
+ u8 vxlan_clss;
+ u8 l2gre_clss;
+ u8 ipgre_clss;
+ u8 l2geneve_clss;
+ u8 ipgeneve_clss;
+ u8 update_geneve_port;
+ u8 update_vxlan_port;
+ u16 geneve_port;
+ u16 vxlan_port;
+ u8 padding[2];
+};
+
+struct pfvf_update_tunn_param_tlv {
+ struct pfvf_tlv hdr;
+
+ u16 tunn_feature_mask;
+ u8 vxlan_mode;
+ u8 l2geneve_mode;
+ u8 ipgeneve_mode;
+ u8 l2gre_mode;
+ u8 ipgre_mode;
+ u8 vxlan_clss;
+ u8 l2gre_clss;
+ u8 ipgre_clss;
+ u8 l2geneve_clss;
+ u8 ipgeneve_clss;
+ u16 vxlan_udp_port;
+ u16 geneve_udp_port;
+};
+
struct tlv_buffer_size {
u8 tlv_buffer[TLV_BUFFER_SIZE];
};
@@ -440,6 +481,7 @@
struct vfpf_vport_start_tlv start_vport;
struct vfpf_vport_update_tlv vport_update;
struct vfpf_ucast_filter_tlv ucast_filter;
+ struct vfpf_update_tunn_param_tlv tunn_param_update;
struct channel_list_end_tlv list_end;
struct tlv_buffer_size tlv_buf_size;
};
@@ -449,6 +491,7 @@
struct pfvf_acquire_resp_tlv acquire_resp;
struct tlv_buffer_size tlv_buf_size;
struct pfvf_start_queue_resp_tlv queue_start;
+ struct pfvf_update_tunn_param_tlv tunn_param_resp;
};
enum qed_bulletin_bit {
@@ -509,7 +552,9 @@
u8 partner_rx_flow_ctrl_en;
u8 partner_adv_pause;
u8 sfp_tx_fault;
- u8 padding4[6];
+ u16 vxlan_udp_port;
+ u16 geneve_udp_port;
+ u8 padding4[2];
u32 speed;
u32 partner_adv_speed;
@@ -551,6 +596,7 @@
CHANNEL_TLV_VPORT_UPDATE_RSS,
CHANNEL_TLV_VPORT_UPDATE_ACCEPT_ANY_VLAN,
CHANNEL_TLV_VPORT_UPDATE_SGE_TPA,
+ CHANNEL_TLV_UPDATE_TUNN_PARAM,
CHANNEL_TLV_MAX,
/* Required for iterating over vport-update tlvs.
@@ -868,6 +914,9 @@
struct qed_bulletin_content *p_bulletin);
void qed_iov_vf_task(struct work_struct *work);
+void qed_vf_set_vf_start_tunn_update_param(struct qed_tunnel_info *p_tun);
+int qed_vf_pf_tunnel_param_update(struct qed_hwfn *p_hwfn,
+ struct qed_tunnel_info *p_tunn);
#else
static inline void qed_vf_get_link_params(struct qed_hwfn *p_hwfn,
struct qed_mcp_link_params *params)
@@ -1029,6 +1078,17 @@
static inline void qed_iov_vf_task(struct work_struct *work)
{
}
+
+static inline void
+qed_vf_set_vf_start_tunn_update_param(struct qed_tunnel_info *p_tun)
+{
+}
+
+static inline int qed_vf_pf_tunnel_param_update(struct qed_hwfn *p_hwfn,
+ struct qed_tunnel_info *p_tunn)
+{
+ return -EINVAL;
+}
#endif
#endif
diff --git a/drivers/net/ethernet/qlogic/qede/qede.h b/drivers/net/ethernet/qlogic/qede/qede.h
index f2aaef2..9b4f08b 100644
--- a/drivers/net/ethernet/qlogic/qede/qede.h
+++ b/drivers/net/ethernet/qlogic/qede/qede.h
@@ -41,6 +41,9 @@
#include <linux/mutex.h>
#include <linux/bpf.h>
#include <linux/io.h>
+#ifdef CONFIG_RFS_ACCEL
+#include <linux/cpu_rmap.h>
+#endif
#include <linux/qed/common_hsi.h>
#include <linux/qed/eth_common.h>
#include <linux/qed/qed_if.h>
@@ -50,7 +53,7 @@
#define QEDE_MAJOR_VERSION 8
#define QEDE_MINOR_VERSION 10
#define QEDE_REVISION_VERSION 10
-#define QEDE_ENGINEERING_VERSION 20
+#define QEDE_ENGINEERING_VERSION 21
#define DRV_MODULE_VERSION __stringify(QEDE_MAJOR_VERSION) "." \
__stringify(QEDE_MINOR_VERSION) "." \
__stringify(QEDE_REVISION_VERSION) "." \
@@ -58,7 +61,7 @@
#define DRV_MODULE_SYM qede
-struct qede_stats {
+struct qede_stats_common {
u64 no_buff_discards;
u64 packet_too_big_discard;
u64 ttl0_discard;
@@ -90,11 +93,6 @@
u64 rx_256_to_511_byte_packets;
u64 rx_512_to_1023_byte_packets;
u64 rx_1024_to_1518_byte_packets;
- u64 rx_1519_to_1522_byte_packets;
- u64 rx_1519_to_2047_byte_packets;
- u64 rx_2048_to_4095_byte_packets;
- u64 rx_4096_to_9216_byte_packets;
- u64 rx_9217_to_16383_byte_packets;
u64 rx_crc_errors;
u64 rx_mac_crtl_frames;
u64 rx_pause_frames;
@@ -111,17 +109,39 @@
u64 tx_256_to_511_byte_packets;
u64 tx_512_to_1023_byte_packets;
u64 tx_1024_to_1518_byte_packets;
+ u64 tx_pause_frames;
+ u64 tx_pfc_frames;
+ u64 brb_truncates;
+ u64 brb_discards;
+ u64 tx_mac_ctrl_frames;
+};
+
+struct qede_stats_bb {
+ u64 rx_1519_to_1522_byte_packets;
+ u64 rx_1519_to_2047_byte_packets;
+ u64 rx_2048_to_4095_byte_packets;
+ u64 rx_4096_to_9216_byte_packets;
+ u64 rx_9217_to_16383_byte_packets;
u64 tx_1519_to_2047_byte_packets;
u64 tx_2048_to_4095_byte_packets;
u64 tx_4096_to_9216_byte_packets;
u64 tx_9217_to_16383_byte_packets;
- u64 tx_pause_frames;
- u64 tx_pfc_frames;
u64 tx_lpi_entry_count;
u64 tx_total_collisions;
- u64 brb_truncates;
- u64 brb_discards;
- u64 tx_mac_ctrl_frames;
+};
+
+struct qede_stats_ah {
+ u64 rx_1519_to_max_byte_packets;
+ u64 tx_1519_to_max_byte_packets;
+};
+
+struct qede_stats {
+ struct qede_stats_common common;
+
+ union {
+ struct qede_stats_bb bb;
+ struct qede_stats_ah ah;
+ };
};
struct qede_vlan {
@@ -147,10 +167,11 @@
u32 dp_module;
u8 dp_level;
- u32 flags;
-#define QEDE_FLAG_IS_VF BIT(0)
+ unsigned long flags;
+#define QEDE_FLAG_IS_VF BIT(0)
#define IS_VF(edev) (!!((edev)->flags & QEDE_FLAG_IS_VF))
#define QEDE_TX_TIMESTAMPING_EN BIT(1)
+#define QEDE_FLAGS_PTP_TX_IN_PRORGESS BIT(2)
const struct qed_eth_ops *ops;
struct qede_ptp *ptp;
@@ -158,6 +179,10 @@
struct qed_dev_eth_info dev_info;
#define QEDE_MAX_RSS_CNT(edev) ((edev)->dev_info.num_queues)
#define QEDE_MAX_TSS_CNT(edev) ((edev)->dev_info.num_queues)
+#define QEDE_IS_BB(edev) \
+ ((edev)->dev_info.common.dev_type == QED_DEV_TYPE_BB)
+#define QEDE_IS_AH(edev) \
+ ((edev)->dev_info.common.dev_type == QED_DEV_TYPE_AH)
struct qede_fastpath *fp_array;
u8 req_num_tx;
@@ -216,7 +241,10 @@
u16 vxlan_dst_port;
u16 geneve_dst_port;
- bool wol_enabled;
+#ifdef CONFIG_RFS_ACCEL
+ struct qede_arfs *arfs;
+#endif
+ bool wol_enabled;
struct qede_rdma_dev rdma_info;
@@ -292,21 +320,24 @@
u8 data_direction;
u8 rxq_id;
+ /* Used once per each NAPI run */
+ u16 num_rx_buffers;
+
+ u16 rx_headroom;
+
u32 rx_buf_size;
u32 rx_buf_seg_size;
- u64 rcv_pkts;
-
struct sw_rx_data *sw_rx_ring;
struct qed_chain rx_bd_ring;
struct qed_chain rx_comp_ring ____cacheline_aligned;
- /* Used once per each NAPI run */
- u16 num_rx_buffers;
-
/* GRO */
struct qede_agg_info tpa_info[ETH_TPA_MAX_AGGS_NUM];
+ /* Used once per each NAPI run */
+ u64 rcv_pkts;
+
u64 rx_hw_errors;
u64 rx_alloc_errors;
u64 rx_ip_frags;
@@ -328,6 +359,11 @@
#define QEDE_TSO_SPLIT_BD BIT(0)
};
+struct sw_tx_xdp {
+ struct page *page;
+ dma_addr_t mapping;
+};
+
struct qede_tx_queue {
u8 is_xdp;
bool is_legacy;
@@ -351,11 +387,11 @@
#define QEDE_TXQ_IDX_TO_XDP(edev, idx) ((idx) + QEDE_MAX_TSS_CNT(edev))
/* Regular Tx requires skb + metadata for release purpose,
- * while XDP requires only the pages themselves.
+ * while XDP requires the pages and the mapped address.
*/
union {
struct sw_tx_bd *skbs;
- struct page **pages;
+ struct sw_tx_xdp *xdp;
} sw_tx_ring;
struct qed_chain tx_pbl;
@@ -407,8 +443,20 @@
#define QEDE_TUNN_CSUM_UNNECESSARY BIT(2)
#define QEDE_SP_RX_MODE 1
-#define QEDE_SP_VXLAN_PORT_CONFIG 2
-#define QEDE_SP_GENEVE_PORT_CONFIG 3
+
+#ifdef CONFIG_RFS_ACCEL
+int qede_rx_flow_steer(struct net_device *dev, const struct sk_buff *skb,
+ u16 rxq_index, u32 flow_id);
+void qede_process_arfs_filters(struct qede_dev *edev, bool free_fltr);
+void qede_poll_for_freeing_arfs_filters(struct qede_dev *edev);
+void qede_arfs_filter_op(void *dev, void *filter, u8 fw_rc);
+void qede_free_arfs(struct qede_dev *edev);
+int qede_alloc_arfs(struct qede_dev *edev);
+
+#define QEDE_SP_ARFS_CONFIG 4
+#define QEDE_SP_TASK_POLL_DELAY (5 * HZ)
+#define QEDE_RFS_MAX_FLTR 256
+#endif
struct qede_reload_args {
void (*func)(struct qede_dev *edev, struct qede_reload_args *args);
@@ -433,6 +481,7 @@
/* Filtering function definitions */
void qede_force_mac(void *dev, u8 *mac, bool forced);
+void qede_udp_ports_update(void *dev, u16 vxlan_port, u16 geneve_port);
int qede_set_mac_addr(struct net_device *ndev, void *p);
int qede_vlan_rx_add_vid(struct net_device *dev, __be16 proto, u16 vid);
diff --git a/drivers/net/ethernet/qlogic/qede/qede_dcbnl.c b/drivers/net/ethernet/qlogic/qede/qede_dcbnl.c
index 03e8c02..a9e7379 100644
--- a/drivers/net/ethernet/qlogic/qede/qede_dcbnl.c
+++ b/drivers/net/ethernet/qlogic/qede/qede_dcbnl.c
@@ -281,6 +281,11 @@
struct dcb_app *app)
{
struct qede_dev *edev = netdev_priv(netdev);
+ int err;
+
+ err = dcb_ieee_setapp(netdev, app);
+ if (err)
+ return err;
return edev->ops->dcb->ieee_setapp(edev->cdev, app);
}
diff --git a/drivers/net/ethernet/qlogic/qede/qede_ethtool.c b/drivers/net/ethernet/qlogic/qede/qede_ethtool.c
index 8979531..4dcfe96 100644
--- a/drivers/net/ethernet/qlogic/qede/qede_ethtool.c
+++ b/drivers/net/ethernet/qlogic/qede/qede_ethtool.c
@@ -75,16 +75,33 @@
QEDE_TQSTAT(stopped_cnt),
};
-#define QEDE_STAT_OFFSET(stat_name) (offsetof(struct qede_stats, stat_name))
-#define QEDE_STAT_STRING(stat_name) (#stat_name)
-#define _QEDE_STAT(stat_name, pf_only) \
- {QEDE_STAT_OFFSET(stat_name), QEDE_STAT_STRING(stat_name), pf_only}
-#define QEDE_PF_STAT(stat_name) _QEDE_STAT(stat_name, true)
-#define QEDE_STAT(stat_name) _QEDE_STAT(stat_name, false)
+#define QEDE_STAT_OFFSET(stat_name, type, base) \
+ (offsetof(type, stat_name) + (base))
+#define QEDE_STAT_STRING(stat_name) (#stat_name)
+#define _QEDE_STAT(stat_name, type, base, attr) \
+ {QEDE_STAT_OFFSET(stat_name, type, base), \
+ QEDE_STAT_STRING(stat_name), \
+ attr}
+#define QEDE_STAT(stat_name) \
+ _QEDE_STAT(stat_name, struct qede_stats_common, 0, 0x0)
+#define QEDE_PF_STAT(stat_name) \
+ _QEDE_STAT(stat_name, struct qede_stats_common, 0, \
+ BIT(QEDE_STAT_PF_ONLY))
+#define QEDE_PF_BB_STAT(stat_name) \
+ _QEDE_STAT(stat_name, struct qede_stats_bb, \
+ offsetof(struct qede_stats, bb), \
+ BIT(QEDE_STAT_PF_ONLY) | BIT(QEDE_STAT_BB_ONLY))
+#define QEDE_PF_AH_STAT(stat_name) \
+ _QEDE_STAT(stat_name, struct qede_stats_ah, \
+ offsetof(struct qede_stats, ah), \
+ BIT(QEDE_STAT_PF_ONLY) | BIT(QEDE_STAT_AH_ONLY))
static const struct {
u64 offset;
char string[ETH_GSTRING_LEN];
- bool pf_only;
+ unsigned long attr;
+#define QEDE_STAT_PF_ONLY 0
+#define QEDE_STAT_BB_ONLY 1
+#define QEDE_STAT_AH_ONLY 2
} qede_stats_arr[] = {
QEDE_STAT(rx_ucast_bytes),
QEDE_STAT(rx_mcast_bytes),
@@ -106,22 +123,23 @@
QEDE_PF_STAT(rx_256_to_511_byte_packets),
QEDE_PF_STAT(rx_512_to_1023_byte_packets),
QEDE_PF_STAT(rx_1024_to_1518_byte_packets),
- QEDE_PF_STAT(rx_1519_to_1522_byte_packets),
- QEDE_PF_STAT(rx_1519_to_2047_byte_packets),
- QEDE_PF_STAT(rx_2048_to_4095_byte_packets),
- QEDE_PF_STAT(rx_4096_to_9216_byte_packets),
- QEDE_PF_STAT(rx_9217_to_16383_byte_packets),
+ QEDE_PF_BB_STAT(rx_1519_to_1522_byte_packets),
+ QEDE_PF_BB_STAT(rx_1519_to_2047_byte_packets),
+ QEDE_PF_BB_STAT(rx_2048_to_4095_byte_packets),
+ QEDE_PF_BB_STAT(rx_4096_to_9216_byte_packets),
+ QEDE_PF_BB_STAT(rx_9217_to_16383_byte_packets),
+ QEDE_PF_AH_STAT(rx_1519_to_max_byte_packets),
QEDE_PF_STAT(tx_64_byte_packets),
QEDE_PF_STAT(tx_65_to_127_byte_packets),
QEDE_PF_STAT(tx_128_to_255_byte_packets),
QEDE_PF_STAT(tx_256_to_511_byte_packets),
QEDE_PF_STAT(tx_512_to_1023_byte_packets),
QEDE_PF_STAT(tx_1024_to_1518_byte_packets),
- QEDE_PF_STAT(tx_1519_to_2047_byte_packets),
- QEDE_PF_STAT(tx_2048_to_4095_byte_packets),
- QEDE_PF_STAT(tx_4096_to_9216_byte_packets),
- QEDE_PF_STAT(tx_9217_to_16383_byte_packets),
-
+ QEDE_PF_BB_STAT(tx_1519_to_2047_byte_packets),
+ QEDE_PF_BB_STAT(tx_2048_to_4095_byte_packets),
+ QEDE_PF_BB_STAT(tx_4096_to_9216_byte_packets),
+ QEDE_PF_BB_STAT(tx_9217_to_16383_byte_packets),
+ QEDE_PF_AH_STAT(tx_1519_to_max_byte_packets),
QEDE_PF_STAT(rx_mac_crtl_frames),
QEDE_PF_STAT(tx_mac_ctrl_frames),
QEDE_PF_STAT(rx_pause_frames),
@@ -136,8 +154,8 @@
QEDE_PF_STAT(rx_jabbers),
QEDE_PF_STAT(rx_undersize_packets),
QEDE_PF_STAT(rx_fragments),
- QEDE_PF_STAT(tx_lpi_entry_count),
- QEDE_PF_STAT(tx_total_collisions),
+ QEDE_PF_BB_STAT(tx_lpi_entry_count),
+ QEDE_PF_BB_STAT(tx_total_collisions),
QEDE_PF_STAT(brb_truncates),
QEDE_PF_STAT(brb_discards),
QEDE_STAT(no_buff_discards),
@@ -155,6 +173,12 @@
};
#define QEDE_NUM_STATS ARRAY_SIZE(qede_stats_arr)
+#define QEDE_STAT_IS_PF_ONLY(i) \
+ test_bit(QEDE_STAT_PF_ONLY, &qede_stats_arr[i].attr)
+#define QEDE_STAT_IS_BB_ONLY(i) \
+ test_bit(QEDE_STAT_BB_ONLY, &qede_stats_arr[i].attr)
+#define QEDE_STAT_IS_AH_ONLY(i) \
+ test_bit(QEDE_STAT_AH_ONLY, &qede_stats_arr[i].attr)
enum {
QEDE_PRI_FLAG_CMT,
@@ -213,6 +237,13 @@
}
}
+static bool qede_is_irrelevant_stat(struct qede_dev *edev, int stat_index)
+{
+ return (IS_VF(edev) && QEDE_STAT_IS_PF_ONLY(stat_index)) ||
+ (QEDE_IS_BB(edev) && QEDE_STAT_IS_AH_ONLY(stat_index)) ||
+ (QEDE_IS_AH(edev) && QEDE_STAT_IS_BB_ONLY(stat_index));
+}
+
static void qede_get_strings_stats(struct qede_dev *edev, u8 *buf)
{
struct qede_fastpath *fp;
@@ -234,7 +265,7 @@
/* Account for non-queue statistics */
for (i = 0; i < QEDE_NUM_STATS; i++) {
- if (IS_VF(edev) && qede_stats_arr[i].pf_only)
+ if (qede_is_irrelevant_stat(edev, i))
continue;
strcpy(buf, qede_stats_arr[i].string);
buf += ETH_GSTRING_LEN;
@@ -309,7 +340,7 @@
}
for (i = 0; i < QEDE_NUM_STATS; i++) {
- if (IS_VF(edev) && qede_stats_arr[i].pf_only)
+ if (qede_is_irrelevant_stat(edev, i))
continue;
*buf = *((u64 *)(((void *)&edev->stats) +
qede_stats_arr[i].offset));
@@ -323,17 +354,13 @@
static int qede_get_sset_count(struct net_device *dev, int stringset)
{
struct qede_dev *edev = netdev_priv(dev);
- int num_stats = QEDE_NUM_STATS;
+ int num_stats = QEDE_NUM_STATS, i;
switch (stringset) {
case ETH_SS_STATS:
- if (IS_VF(edev)) {
- int i;
-
- for (i = 0; i < QEDE_NUM_STATS; i++)
- if (qede_stats_arr[i].pf_only)
- num_stats--;
- }
+ for (i = 0; i < QEDE_NUM_STATS; i++)
+ if (qede_is_irrelevant_stat(edev, i))
+ num_stats--;
/* Account for the Regular Tx statistics */
num_stats += QEDE_TSS_COUNT(edev) * QEDE_NUM_TQSTATS;
diff --git a/drivers/net/ethernet/qlogic/qede/qede_filter.c b/drivers/net/ethernet/qlogic/qede/qede_filter.c
index 107c3fd..eb56520 100644
--- a/drivers/net/ethernet/qlogic/qede/qede_filter.c
+++ b/drivers/net/ethernet/qlogic/qede/qede_filter.c
@@ -38,6 +38,459 @@
#include <linux/qed/qed_if.h>
#include "qede.h"
+#ifdef CONFIG_RFS_ACCEL
+struct qede_arfs_tuple {
+ union {
+ __be32 src_ipv4;
+ struct in6_addr src_ipv6;
+ };
+ union {
+ __be32 dst_ipv4;
+ struct in6_addr dst_ipv6;
+ };
+ __be16 src_port;
+ __be16 dst_port;
+ __be16 eth_proto;
+ u8 ip_proto;
+};
+
+struct qede_arfs_fltr_node {
+#define QEDE_FLTR_VALID 0
+ unsigned long state;
+
+ /* pointer to aRFS packet buffer */
+ void *data;
+
+ /* dma map address of aRFS packet buffer */
+ dma_addr_t mapping;
+
+ /* length of aRFS packet buffer */
+ int buf_len;
+
+ /* tuples to hold from aRFS packet buffer */
+ struct qede_arfs_tuple tuple;
+
+ u32 flow_id;
+ u16 sw_id;
+ u16 rxq_id;
+ u16 next_rxq_id;
+ bool filter_op;
+ bool used;
+ struct hlist_node node;
+};
+
+struct qede_arfs {
+#define QEDE_ARFS_POLL_COUNT 100
+#define QEDE_RFS_FLW_BITSHIFT (4)
+#define QEDE_RFS_FLW_MASK ((1 << QEDE_RFS_FLW_BITSHIFT) - 1)
+ struct hlist_head arfs_hl_head[1 << QEDE_RFS_FLW_BITSHIFT];
+
+ /* lock for filter list access */
+ spinlock_t arfs_list_lock;
+ unsigned long *arfs_fltr_bmap;
+ int filter_count;
+ bool enable;
+};
+
+static void qede_configure_arfs_fltr(struct qede_dev *edev,
+ struct qede_arfs_fltr_node *n,
+ u16 rxq_id, bool add_fltr)
+{
+ const struct qed_eth_ops *op = edev->ops;
+
+ if (n->used)
+ return;
+
+ DP_VERBOSE(edev, NETIF_MSG_RX_STATUS,
+ "%s arfs filter flow_id=%d, sw_id=%d, src_port=%d, dst_port=%d, rxq=%d\n",
+ add_fltr ? "Adding" : "Deleting",
+ n->flow_id, n->sw_id, ntohs(n->tuple.src_port),
+ ntohs(n->tuple.dst_port), rxq_id);
+
+ n->used = true;
+ n->filter_op = add_fltr;
+ op->ntuple_filter_config(edev->cdev, n, n->mapping, n->buf_len, 0,
+ rxq_id, add_fltr);
+}
+
+static void
+qede_free_arfs_filter(struct qede_dev *edev, struct qede_arfs_fltr_node *fltr)
+{
+ kfree(fltr->data);
+ clear_bit(fltr->sw_id, edev->arfs->arfs_fltr_bmap);
+ kfree(fltr);
+}
+
+void qede_arfs_filter_op(void *dev, void *filter, u8 fw_rc)
+{
+ struct qede_arfs_fltr_node *fltr = filter;
+ struct qede_dev *edev = dev;
+
+ if (fw_rc) {
+ DP_NOTICE(edev,
+ "Failed arfs filter configuration fw_rc=%d, flow_id=%d, sw_id=%d, src_port=%d, dst_port=%d, rxq=%d\n",
+ fw_rc, fltr->flow_id, fltr->sw_id,
+ ntohs(fltr->tuple.src_port),
+ ntohs(fltr->tuple.dst_port), fltr->rxq_id);
+
+ spin_lock_bh(&edev->arfs->arfs_list_lock);
+
+ fltr->used = false;
+ clear_bit(QEDE_FLTR_VALID, &fltr->state);
+
+ spin_unlock_bh(&edev->arfs->arfs_list_lock);
+ return;
+ }
+
+ spin_lock_bh(&edev->arfs->arfs_list_lock);
+
+ fltr->used = false;
+
+ if (fltr->filter_op) {
+ set_bit(QEDE_FLTR_VALID, &fltr->state);
+ if (fltr->rxq_id != fltr->next_rxq_id)
+ qede_configure_arfs_fltr(edev, fltr, fltr->rxq_id,
+ false);
+ } else {
+ clear_bit(QEDE_FLTR_VALID, &fltr->state);
+ if (fltr->rxq_id != fltr->next_rxq_id) {
+ fltr->rxq_id = fltr->next_rxq_id;
+ qede_configure_arfs_fltr(edev, fltr,
+ fltr->rxq_id, true);
+ }
+ }
+
+ spin_unlock_bh(&edev->arfs->arfs_list_lock);
+}
+
+/* Should be called while qede_lock is held */
+void qede_process_arfs_filters(struct qede_dev *edev, bool free_fltr)
+{
+ int i;
+
+ for (i = 0; i <= QEDE_RFS_FLW_MASK; i++) {
+ struct hlist_node *temp;
+ struct hlist_head *head;
+ struct qede_arfs_fltr_node *fltr;
+
+ head = &edev->arfs->arfs_hl_head[i];
+
+ hlist_for_each_entry_safe(fltr, temp, head, node) {
+ bool del = false;
+
+ if (edev->state != QEDE_STATE_OPEN)
+ del = true;
+
+ spin_lock_bh(&edev->arfs->arfs_list_lock);
+
+ if ((!test_bit(QEDE_FLTR_VALID, &fltr->state) &&
+ !fltr->used) || free_fltr) {
+ hlist_del(&fltr->node);
+ dma_unmap_single(&edev->pdev->dev,
+ fltr->mapping,
+ fltr->buf_len, DMA_TO_DEVICE);
+ qede_free_arfs_filter(edev, fltr);
+ edev->arfs->filter_count--;
+ } else {
+ if ((rps_may_expire_flow(edev->ndev,
+ fltr->rxq_id,
+ fltr->flow_id,
+ fltr->sw_id) || del) &&
+ !free_fltr)
+ qede_configure_arfs_fltr(edev, fltr,
+ fltr->rxq_id,
+ false);
+ }
+
+ spin_unlock_bh(&edev->arfs->arfs_list_lock);
+ }
+ }
+
+ spin_lock_bh(&edev->arfs->arfs_list_lock);
+
+ if (!edev->arfs->filter_count) {
+ if (edev->arfs->enable) {
+ edev->arfs->enable = false;
+ edev->ops->configure_arfs_searcher(edev->cdev, false);
+ }
+ } else {
+ set_bit(QEDE_SP_ARFS_CONFIG, &edev->sp_flags);
+ schedule_delayed_work(&edev->sp_task,
+ QEDE_SP_TASK_POLL_DELAY);
+ }
+
+ spin_unlock_bh(&edev->arfs->arfs_list_lock);
+}
+
+/* This function waits until all aRFS filters get deleted and freed.
+ * On timeout it frees all filters forcefully.
+ */
+void qede_poll_for_freeing_arfs_filters(struct qede_dev *edev)
+{
+ int count = QEDE_ARFS_POLL_COUNT;
+
+ while (count) {
+ qede_process_arfs_filters(edev, false);
+
+ if (!edev->arfs->filter_count)
+ break;
+
+ msleep(100);
+ count--;
+ }
+
+ if (!count) {
+ DP_NOTICE(edev, "Timeout in polling for arfs filter free\n");
+
+ /* Something is terribly wrong, free forcefully */
+ qede_process_arfs_filters(edev, true);
+ }
+}
+
+int qede_alloc_arfs(struct qede_dev *edev)
+{
+ int i;
+
+ edev->arfs = vzalloc(sizeof(*edev->arfs));
+ if (!edev->arfs)
+ return -ENOMEM;
+
+ spin_lock_init(&edev->arfs->arfs_list_lock);
+
+ for (i = 0; i <= QEDE_RFS_FLW_MASK; i++)
+ INIT_HLIST_HEAD(&edev->arfs->arfs_hl_head[i]);
+
+ edev->ndev->rx_cpu_rmap = alloc_irq_cpu_rmap(QEDE_RSS_COUNT(edev));
+ if (!edev->ndev->rx_cpu_rmap) {
+ vfree(edev->arfs);
+ edev->arfs = NULL;
+ return -ENOMEM;
+ }
+
+ edev->arfs->arfs_fltr_bmap = vzalloc(BITS_TO_LONGS(QEDE_RFS_MAX_FLTR) *
+ sizeof(long));
+ if (!edev->arfs->arfs_fltr_bmap) {
+ free_irq_cpu_rmap(edev->ndev->rx_cpu_rmap);
+ edev->ndev->rx_cpu_rmap = NULL;
+ vfree(edev->arfs);
+ edev->arfs = NULL;
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+void qede_free_arfs(struct qede_dev *edev)
+{
+ if (!edev->arfs)
+ return;
+
+ if (edev->ndev->rx_cpu_rmap)
+ free_irq_cpu_rmap(edev->ndev->rx_cpu_rmap);
+
+ edev->ndev->rx_cpu_rmap = NULL;
+ vfree(edev->arfs->arfs_fltr_bmap);
+ edev->arfs->arfs_fltr_bmap = NULL;
+ vfree(edev->arfs);
+ edev->arfs = NULL;
+}
+
+static bool qede_compare_ip_addr(struct qede_arfs_fltr_node *tpos,
+ const struct sk_buff *skb)
+{
+ if (skb->protocol == htons(ETH_P_IP)) {
+ if (tpos->tuple.src_ipv4 == ip_hdr(skb)->saddr &&
+ tpos->tuple.dst_ipv4 == ip_hdr(skb)->daddr)
+ return true;
+ else
+ return false;
+ } else {
+ struct in6_addr *src = &tpos->tuple.src_ipv6;
+ u8 size = sizeof(struct in6_addr);
+
+ if (!memcmp(src, &ipv6_hdr(skb)->saddr, size) &&
+ !memcmp(&tpos->tuple.dst_ipv6, &ipv6_hdr(skb)->daddr, size))
+ return true;
+ else
+ return false;
+ }
+}
+
+static struct qede_arfs_fltr_node *
+qede_arfs_htbl_key_search(struct hlist_head *h, const struct sk_buff *skb,
+ __be16 src_port, __be16 dst_port, u8 ip_proto)
+{
+ struct qede_arfs_fltr_node *tpos;
+
+ hlist_for_each_entry(tpos, h, node)
+ if (tpos->tuple.ip_proto == ip_proto &&
+ tpos->tuple.eth_proto == skb->protocol &&
+ qede_compare_ip_addr(tpos, skb) &&
+ tpos->tuple.src_port == src_port &&
+ tpos->tuple.dst_port == dst_port)
+ return tpos;
+
+ return NULL;
+}
+
+static struct qede_arfs_fltr_node *
+qede_alloc_filter(struct qede_dev *edev, int min_hlen)
+{
+ struct qede_arfs_fltr_node *n;
+ int bit_id;
+
+ bit_id = find_first_zero_bit(edev->arfs->arfs_fltr_bmap,
+ QEDE_RFS_MAX_FLTR);
+
+ if (bit_id >= QEDE_RFS_MAX_FLTR)
+ return NULL;
+
+ n = kzalloc(sizeof(*n), GFP_ATOMIC);
+ if (!n)
+ return NULL;
+
+ n->data = kzalloc(min_hlen, GFP_ATOMIC);
+ if (!n->data) {
+ kfree(n);
+ return NULL;
+ }
+
+ n->sw_id = (u16)bit_id;
+ set_bit(bit_id, edev->arfs->arfs_fltr_bmap);
+ return n;
+}
+
+int qede_rx_flow_steer(struct net_device *dev, const struct sk_buff *skb,
+ u16 rxq_index, u32 flow_id)
+{
+ struct qede_dev *edev = netdev_priv(dev);
+ struct qede_arfs_fltr_node *n;
+ int min_hlen, rc, tp_offset;
+ struct ethhdr *eth;
+ __be16 *ports;
+ u16 tbl_idx;
+ u8 ip_proto;
+
+ if (skb->encapsulation)
+ return -EPROTONOSUPPORT;
+
+ if (skb->protocol != htons(ETH_P_IP) &&
+ skb->protocol != htons(ETH_P_IPV6))
+ return -EPROTONOSUPPORT;
+
+ if (skb->protocol == htons(ETH_P_IP)) {
+ ip_proto = ip_hdr(skb)->protocol;
+ tp_offset = sizeof(struct iphdr);
+ } else {
+ ip_proto = ipv6_hdr(skb)->nexthdr;
+ tp_offset = sizeof(struct ipv6hdr);
+ }
+
+ if (ip_proto != IPPROTO_TCP && ip_proto != IPPROTO_UDP)
+ return -EPROTONOSUPPORT;
+
+ ports = (__be16 *)(skb->data + tp_offset);
+ tbl_idx = skb_get_hash_raw(skb) & QEDE_RFS_FLW_MASK;
+
+ spin_lock_bh(&edev->arfs->arfs_list_lock);
+
+ n = qede_arfs_htbl_key_search(&edev->arfs->arfs_hl_head[tbl_idx],
+ skb, ports[0], ports[1], ip_proto);
+
+ if (n) {
+ /* Filter match */
+ n->next_rxq_id = rxq_index;
+
+ if (test_bit(QEDE_FLTR_VALID, &n->state)) {
+ if (n->rxq_id != rxq_index)
+ qede_configure_arfs_fltr(edev, n, n->rxq_id,
+ false);
+ } else {
+ if (!n->used) {
+ n->rxq_id = rxq_index;
+ qede_configure_arfs_fltr(edev, n, n->rxq_id,
+ true);
+ }
+ }
+
+ rc = n->sw_id;
+ goto ret_unlock;
+ }
+
+ min_hlen = ETH_HLEN + skb_headlen(skb);
+
+ n = qede_alloc_filter(edev, min_hlen);
+ if (!n) {
+ rc = -ENOMEM;
+ goto ret_unlock;
+ }
+
+ n->buf_len = min_hlen;
+ n->rxq_id = rxq_index;
+ n->next_rxq_id = rxq_index;
+ n->tuple.src_port = ports[0];
+ n->tuple.dst_port = ports[1];
+ n->flow_id = flow_id;
+
+ if (skb->protocol == htons(ETH_P_IP)) {
+ n->tuple.src_ipv4 = ip_hdr(skb)->saddr;
+ n->tuple.dst_ipv4 = ip_hdr(skb)->daddr;
+ } else {
+ memcpy(&n->tuple.src_ipv6, &ipv6_hdr(skb)->saddr,
+ sizeof(struct in6_addr));
+ memcpy(&n->tuple.dst_ipv6, &ipv6_hdr(skb)->daddr,
+ sizeof(struct in6_addr));
+ }
+
+ eth = (struct ethhdr *)n->data;
+ eth->h_proto = skb->protocol;
+ n->tuple.eth_proto = skb->protocol;
+ n->tuple.ip_proto = ip_proto;
+ memcpy(n->data + ETH_HLEN, skb->data, skb_headlen(skb));
+
+ n->mapping = dma_map_single(&edev->pdev->dev, n->data,
+ n->buf_len, DMA_TO_DEVICE);
+ if (dma_mapping_error(&edev->pdev->dev, n->mapping)) {
+ DP_NOTICE(edev, "Failed to map DMA memory for arfs\n");
+ qede_free_arfs_filter(edev, n);
+ rc = -ENOMEM;
+ goto ret_unlock;
+ }
+
+ INIT_HLIST_NODE(&n->node);
+ hlist_add_head(&n->node, &edev->arfs->arfs_hl_head[tbl_idx]);
+ edev->arfs->filter_count++;
+
+ if (edev->arfs->filter_count == 1 && !edev->arfs->enable) {
+ edev->ops->configure_arfs_searcher(edev->cdev, true);
+ edev->arfs->enable = true;
+ }
+
+ qede_configure_arfs_fltr(edev, n, n->rxq_id, true);
+
+ spin_unlock_bh(&edev->arfs->arfs_list_lock);
+
+ set_bit(QEDE_SP_ARFS_CONFIG, &edev->sp_flags);
+ schedule_delayed_work(&edev->sp_task, 0);
+ return n->sw_id;
+
+ret_unlock:
+ spin_unlock_bh(&edev->arfs->arfs_list_lock);
+ return rc;
+}
+#endif
+
+void qede_udp_ports_update(void *dev, u16 vxlan_port, u16 geneve_port)
+{
+ struct qede_dev *edev = dev;
+
+ if (edev->vxlan_dst_port != vxlan_port)
+ edev->vxlan_dst_port = 0;
+
+ if (edev->geneve_dst_port != geneve_port)
+ edev->geneve_dst_port = 0;
+}
+
void qede_force_mac(void *dev, u8 *mac, bool forced)
{
struct qede_dev *edev = dev;
@@ -441,69 +894,112 @@
void qede_udp_tunnel_add(struct net_device *dev, struct udp_tunnel_info *ti)
{
struct qede_dev *edev = netdev_priv(dev);
+ struct qed_tunn_params tunn_params;
u16 t_port = ntohs(ti->port);
+ int rc;
+
+ memset(&tunn_params, 0, sizeof(tunn_params));
switch (ti->type) {
case UDP_TUNNEL_TYPE_VXLAN:
+ if (!edev->dev_info.common.vxlan_enable)
+ return;
+
if (edev->vxlan_dst_port)
return;
- edev->vxlan_dst_port = t_port;
+ tunn_params.update_vxlan_port = 1;
+ tunn_params.vxlan_port = t_port;
- DP_VERBOSE(edev, QED_MSG_DEBUG, "Added vxlan port=%d\n",
- t_port);
+ __qede_lock(edev);
+ rc = edev->ops->tunn_config(edev->cdev, &tunn_params);
+ __qede_unlock(edev);
- set_bit(QEDE_SP_VXLAN_PORT_CONFIG, &edev->sp_flags);
+ if (!rc) {
+ edev->vxlan_dst_port = t_port;
+ DP_VERBOSE(edev, QED_MSG_DEBUG, "Added vxlan port=%d\n",
+ t_port);
+ } else {
+ DP_NOTICE(edev, "Failed to add vxlan UDP port=%d\n",
+ t_port);
+ }
+
break;
case UDP_TUNNEL_TYPE_GENEVE:
+ if (!edev->dev_info.common.geneve_enable)
+ return;
+
if (edev->geneve_dst_port)
return;
- edev->geneve_dst_port = t_port;
+ tunn_params.update_geneve_port = 1;
+ tunn_params.geneve_port = t_port;
- DP_VERBOSE(edev, QED_MSG_DEBUG, "Added geneve port=%d\n",
- t_port);
- set_bit(QEDE_SP_GENEVE_PORT_CONFIG, &edev->sp_flags);
+ __qede_lock(edev);
+ rc = edev->ops->tunn_config(edev->cdev, &tunn_params);
+ __qede_unlock(edev);
+
+ if (!rc) {
+ edev->geneve_dst_port = t_port;
+ DP_VERBOSE(edev, QED_MSG_DEBUG,
+ "Added geneve port=%d\n", t_port);
+ } else {
+ DP_NOTICE(edev, "Failed to add geneve UDP port=%d\n",
+ t_port);
+ }
+
break;
default:
return;
}
-
- schedule_delayed_work(&edev->sp_task, 0);
}
-void qede_udp_tunnel_del(struct net_device *dev, struct udp_tunnel_info *ti)
+void qede_udp_tunnel_del(struct net_device *dev,
+ struct udp_tunnel_info *ti)
{
struct qede_dev *edev = netdev_priv(dev);
+ struct qed_tunn_params tunn_params;
u16 t_port = ntohs(ti->port);
+ memset(&tunn_params, 0, sizeof(tunn_params));
+
switch (ti->type) {
case UDP_TUNNEL_TYPE_VXLAN:
if (t_port != edev->vxlan_dst_port)
return;
+ tunn_params.update_vxlan_port = 1;
+ tunn_params.vxlan_port = 0;
+
+ __qede_lock(edev);
+ edev->ops->tunn_config(edev->cdev, &tunn_params);
+ __qede_unlock(edev);
+
edev->vxlan_dst_port = 0;
DP_VERBOSE(edev, QED_MSG_DEBUG, "Deleted vxlan port=%d\n",
t_port);
- set_bit(QEDE_SP_VXLAN_PORT_CONFIG, &edev->sp_flags);
break;
case UDP_TUNNEL_TYPE_GENEVE:
if (t_port != edev->geneve_dst_port)
return;
+ tunn_params.update_geneve_port = 1;
+ tunn_params.geneve_port = 0;
+
+ __qede_lock(edev);
+ edev->ops->tunn_config(edev->cdev, &tunn_params);
+ __qede_unlock(edev);
+
edev->geneve_dst_port = 0;
DP_VERBOSE(edev, QED_MSG_DEBUG, "Deleted geneve port=%d\n",
t_port);
- set_bit(QEDE_SP_GENEVE_PORT_CONFIG, &edev->sp_flags);
break;
default:
return;
}
-
- schedule_delayed_work(&edev->sp_task, 0);
}
static void qede_xdp_reload_func(struct qede_dev *edev,
@@ -520,11 +1016,6 @@
{
struct qede_reload_args args;
- if (prog && prog->xdp_adjust_head) {
- DP_ERR(edev, "Does not support bpf_xdp_adjust_head()\n");
- return -EOPNOTSUPP;
- }
-
/* If we're called, there was already a bpf reference increment */
args.func = &qede_xdp_reload_func;
args.u.new_prog = prog;
@@ -537,6 +1028,11 @@
{
struct qede_dev *edev = netdev_priv(dev);
+ if (IS_VF(edev)) {
+ DP_NOTICE(edev, "VFs don't support XDP\n");
+ return -EOPNOTSUPP;
+ }
+
switch (xdp->command) {
case XDP_SETUP_PROG:
return qede_xdp_set(edev, xdp->prog);
diff --git a/drivers/net/ethernet/qlogic/qede/qede_fp.c b/drivers/net/ethernet/qlogic/qede/qede_fp.c
index 1e65038..7b6f41d 100644
--- a/drivers/net/ethernet/qlogic/qede/qede_fp.c
+++ b/drivers/net/ethernet/qlogic/qede/qede_fp.c
@@ -87,7 +87,8 @@
rx_bd = (struct eth_rx_bd *)qed_chain_produce(&rxq->rx_bd_ring);
WARN_ON(!rx_bd);
rx_bd->addr.hi = cpu_to_le32(upper_32_bits(mapping));
- rx_bd->addr.lo = cpu_to_le32(lower_32_bits(mapping));
+ rx_bd->addr.lo = cpu_to_le32(lower_32_bits(mapping) +
+ rxq->rx_headroom);
rxq->sw_rx_prod++;
rxq->filled_buffers++;
@@ -360,7 +361,8 @@
metadata->mapping + padding,
length, PCI_DMA_TODEVICE);
- txq->sw_tx_ring.pages[idx] = metadata->data;
+ txq->sw_tx_ring.xdp[idx].page = metadata->data;
+ txq->sw_tx_ring.xdp[idx].mapping = metadata->mapping;
txq->sw_tx_prod++;
/* Mark the fastpath for future XDP doorbell */
@@ -384,19 +386,19 @@
static void qede_xdp_tx_int(struct qede_dev *edev, struct qede_tx_queue *txq)
{
- struct eth_tx_1st_bd *bd;
- u16 hw_bd_cons;
+ u16 hw_bd_cons, idx;
hw_bd_cons = le16_to_cpu(*txq->hw_cons_ptr);
barrier();
while (hw_bd_cons != qed_chain_get_cons_idx(&txq->tx_pbl)) {
- bd = (struct eth_tx_1st_bd *)qed_chain_consume(&txq->tx_pbl);
+ qed_chain_consume(&txq->tx_pbl);
+ idx = txq->sw_tx_cons & NUM_TX_BDS_MAX;
- dma_unmap_single(&edev->pdev->dev, BD_UNMAP_ADDR(bd),
- PAGE_SIZE, DMA_BIDIRECTIONAL);
- __free_page(txq->sw_tx_ring.pages[txq->sw_tx_cons &
- NUM_TX_BDS_MAX]);
+ dma_unmap_page(&edev->pdev->dev,
+ txq->sw_tx_ring.xdp[idx].mapping,
+ PAGE_SIZE, DMA_BIDIRECTIONAL);
+ __free_page(txq->sw_tx_ring.xdp[idx].page);
txq->sw_tx_cons++;
txq->xmit_pkts++;
@@ -508,7 +510,8 @@
new_mapping = curr_prod->mapping + curr_prod->page_offset;
rx_bd_prod->addr.hi = cpu_to_le32(upper_32_bits(new_mapping));
- rx_bd_prod->addr.lo = cpu_to_le32(lower_32_bits(new_mapping));
+ rx_bd_prod->addr.lo = cpu_to_le32(lower_32_bits(new_mapping) +
+ rxq->rx_headroom);
rxq->sw_rx_prod++;
curr_cons->data = NULL;
@@ -624,7 +627,6 @@
__vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), vlan_tag);
napi_gro_receive(&fp->napi, skb);
- rxq->rcv_pkts++;
}
static void qede_set_gro_params(struct qede_dev *edev,
@@ -884,9 +886,9 @@
"Strange - TPA cont with more than a single len_list entry\n");
}
-static void qede_tpa_end(struct qede_dev *edev,
- struct qede_fastpath *fp,
- struct eth_fast_path_rx_tpa_end_cqe *cqe)
+static int qede_tpa_end(struct qede_dev *edev,
+ struct qede_fastpath *fp,
+ struct eth_fast_path_rx_tpa_end_cqe *cqe)
{
struct qede_rx_queue *rxq = fp->rxq;
struct qede_agg_info *tpa_info;
@@ -934,11 +936,12 @@
tpa_info->state = QEDE_AGG_STATE_NONE;
- return;
+ return 1;
err:
tpa_info->state = QEDE_AGG_STATE_NONE;
dev_kfree_skb_any(tpa_info->skb);
tpa_info->skb = NULL;
+ return 0;
}
static u8 qede_check_notunn_csum(u16 flag)
@@ -990,14 +993,15 @@
struct qede_rx_queue *rxq,
struct bpf_prog *prog,
struct sw_rx_data *bd,
- struct eth_fast_path_rx_reg_cqe *cqe)
+ struct eth_fast_path_rx_reg_cqe *cqe,
+ u16 *data_offset, u16 *len)
{
- u16 len = le16_to_cpu(cqe->len_on_first_bd);
struct xdp_buff xdp;
enum xdp_action act;
- xdp.data = page_address(bd->data) + cqe->placement_offset;
- xdp.data_end = xdp.data + len;
+ xdp.data_hard_start = page_address(bd->data);
+ xdp.data = xdp.data_hard_start + *data_offset;
+ xdp.data_end = xdp.data + *len;
/* Queues always have a full reset currently, so for the time
* being until there's atomic program replace just mark read
@@ -1007,6 +1011,10 @@
act = bpf_prog_run_xdp(prog, &xdp);
rcu_read_unlock();
+ /* Recalculate, as XDP might have changed the headers */
+ *data_offset = xdp.data - xdp.data_hard_start;
+ *len = xdp.data_end - xdp.data;
+
if (act == XDP_PASS)
return true;
@@ -1025,7 +1033,7 @@
/* Now if there's a transmission problem, we'd still have to
* throw current buffer, as replacement was already allocated.
*/
- if (qede_xdp_xmit(edev, fp, bd, cqe->placement_offset, len)) {
+ if (qede_xdp_xmit(edev, fp, bd, *data_offset, *len)) {
dma_unmap_page(rxq->dev, bd->mapping,
PAGE_SIZE, DMA_BIDIRECTIONAL);
__free_page(bd->data);
@@ -1052,7 +1060,7 @@
struct sw_rx_data *bd, u16 len,
u16 pad)
{
- unsigned int offset = bd->page_offset;
+ unsigned int offset = bd->page_offset + pad;
struct skb_frag_struct *frag;
struct page *page = bd->data;
unsigned int pull_len;
@@ -1069,7 +1077,7 @@
*/
if (len + pad <= edev->rx_copybreak) {
memcpy(skb_put(skb, len),
- page_address(page) + pad + offset, len);
+ page_address(page) + offset, len);
qede_reuse_page(rxq, bd);
goto out;
}
@@ -1077,7 +1085,7 @@
frag = &skb_shinfo(skb)->frags[0];
skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
- page, pad + offset, len, rxq->rx_buf_seg_size);
+ page, offset, len, rxq->rx_buf_seg_size);
va = skb_frag_address(frag);
pull_len = eth_get_headlen(va, QEDE_RX_HDR_SIZE);
@@ -1178,8 +1186,7 @@
qede_tpa_cont(edev, rxq, &cqe->fast_path_tpa_cont);
return 0;
case ETH_RX_CQE_TYPE_TPA_END:
- qede_tpa_end(edev, fp, &cqe->fast_path_tpa_end);
- return 1;
+ return qede_tpa_end(edev, fp, &cqe->fast_path_tpa_end);
default:
return 0;
}
@@ -1224,12 +1231,13 @@
fp_cqe = &cqe->fast_path_regular;
len = le16_to_cpu(fp_cqe->len_on_first_bd);
- pad = fp_cqe->placement_offset;
+ pad = fp_cqe->placement_offset + rxq->rx_headroom;
/* Run eBPF program if one is attached */
if (xdp_prog)
- if (!qede_rx_xdp(edev, fp, rxq, xdp_prog, bd, fp_cqe))
- return 1;
+ if (!qede_rx_xdp(edev, fp, rxq, xdp_prog, bd, fp_cqe,
+ &pad, &len))
+ return 0;
/* If this is an error packet then drop it */
flags = cqe->fast_path_regular.pars_flags.flags;
@@ -1290,8 +1298,8 @@
{
struct qede_rx_queue *rxq = fp->rxq;
struct qede_dev *edev = fp->edev;
+ int work_done = 0, rcv_pkts = 0;
u16 hw_comp_cons, sw_comp_cons;
- int work_done = 0;
hw_comp_cons = le16_to_cpu(*rxq->hw_cons_ptr);
sw_comp_cons = qed_chain_get_cons_idx(&rxq->rx_comp_ring);
@@ -1305,12 +1313,14 @@
/* Loop to complete all indicated BDs */
while ((sw_comp_cons != hw_comp_cons) && (work_done < budget)) {
- qede_rx_process_cqe(edev, fp, rxq);
+ rcv_pkts += qede_rx_process_cqe(edev, fp, rxq);
qed_chain_recycle_consumed(&rxq->rx_comp_ring);
sw_comp_cons = qed_chain_get_cons_idx(&rxq->rx_comp_ring);
work_done++;
}
+ rxq->rcv_pkts += rcv_pkts;
+
/* Allocate replacement buffers */
while (rxq->num_rx_buffers - rxq->filled_buffers)
if (qede_alloc_rx_buffer(rxq, false))
@@ -1687,13 +1697,24 @@
}
/* Disable offloads for geneve tunnels, as HW can't parse
- * the geneve header which has option length greater than 32B.
+ * the geneve header which has option length greater than 32b
+ * and disable offloads for the ports which are not offloaded.
*/
- if ((l4_proto == IPPROTO_UDP) &&
- ((skb_inner_mac_header(skb) -
- skb_transport_header(skb)) > QEDE_MAX_TUN_HDR_LEN))
- return features & ~(NETIF_F_CSUM_MASK |
- NETIF_F_GSO_MASK);
+ if (l4_proto == IPPROTO_UDP) {
+ struct qede_dev *edev = netdev_priv(dev);
+ u16 hdrlen, vxln_port, gnv_port;
+
+ hdrlen = QEDE_MAX_TUN_HDR_LEN;
+ vxln_port = edev->vxlan_dst_port;
+ gnv_port = edev->geneve_dst_port;
+
+ if ((skb_inner_mac_header(skb) -
+ skb_transport_header(skb)) > hdrlen ||
+ (ntohs(udp_hdr(skb)->dest) != vxln_port &&
+ ntohs(udp_hdr(skb)->dest) != gnv_port))
+ return features & ~(NETIF_F_CSUM_MASK |
+ NETIF_F_GSO_MASK);
+ }
}
return features;
diff --git a/drivers/net/ethernet/qlogic/qede/qede_main.c b/drivers/net/ethernet/qlogic/qede/qede_main.c
index 3a78c3f..b9ba23d 100644
--- a/drivers/net/ethernet/qlogic/qede/qede_main.c
+++ b/drivers/net/ethernet/qlogic/qede/qede_main.c
@@ -84,6 +84,8 @@
#define CHIP_NUM_57980S_50 0x1654
#define CHIP_NUM_57980S_25 0x1656
#define CHIP_NUM_57980S_IOV 0x1664
+#define CHIP_NUM_AH 0x8070
+#define CHIP_NUM_AH_IOV 0x8090
#ifndef PCI_DEVICE_ID_NX2_57980E
#define PCI_DEVICE_ID_57980S_40 CHIP_NUM_57980S_40
@@ -93,6 +95,9 @@
#define PCI_DEVICE_ID_57980S_50 CHIP_NUM_57980S_50
#define PCI_DEVICE_ID_57980S_25 CHIP_NUM_57980S_25
#define PCI_DEVICE_ID_57980S_IOV CHIP_NUM_57980S_IOV
+#define PCI_DEVICE_ID_AH CHIP_NUM_AH
+#define PCI_DEVICE_ID_AH_IOV CHIP_NUM_AH_IOV
+
#endif
enum qede_pci_private {
@@ -110,6 +115,10 @@
#ifdef CONFIG_QED_SRIOV
{PCI_VDEVICE(QLOGIC, PCI_DEVICE_ID_57980S_IOV), QEDE_PRIVATE_VF},
#endif
+ {PCI_VDEVICE(QLOGIC, PCI_DEVICE_ID_AH), QEDE_PRIVATE_PF},
+#ifdef CONFIG_QED_SRIOV
+ {PCI_VDEVICE(QLOGIC, PCI_DEVICE_ID_AH_IOV), QEDE_PRIVATE_VF},
+#endif
{ 0 }
};
@@ -216,9 +225,13 @@
static struct qed_eth_cb_ops qede_ll_ops = {
{
+#ifdef CONFIG_RFS_ACCEL
+ .arfs_filter_op = qede_arfs_filter_op,
+#endif
.link_update = qede_link_update,
},
.force_mac = qede_force_mac,
+ .ports_update = qede_udp_ports_update,
};
static int qede_netdev_event(struct notifier_block *this, unsigned long event,
@@ -314,122 +327,135 @@
void qede_fill_by_demand_stats(struct qede_dev *edev)
{
+ struct qede_stats_common *p_common = &edev->stats.common;
struct qed_eth_stats stats;
edev->ops->get_vport_stats(edev->cdev, &stats);
- edev->stats.no_buff_discards = stats.no_buff_discards;
- edev->stats.packet_too_big_discard = stats.packet_too_big_discard;
- edev->stats.ttl0_discard = stats.ttl0_discard;
- edev->stats.rx_ucast_bytes = stats.rx_ucast_bytes;
- edev->stats.rx_mcast_bytes = stats.rx_mcast_bytes;
- edev->stats.rx_bcast_bytes = stats.rx_bcast_bytes;
- edev->stats.rx_ucast_pkts = stats.rx_ucast_pkts;
- edev->stats.rx_mcast_pkts = stats.rx_mcast_pkts;
- edev->stats.rx_bcast_pkts = stats.rx_bcast_pkts;
- edev->stats.mftag_filter_discards = stats.mftag_filter_discards;
- edev->stats.mac_filter_discards = stats.mac_filter_discards;
- edev->stats.tx_ucast_bytes = stats.tx_ucast_bytes;
- edev->stats.tx_mcast_bytes = stats.tx_mcast_bytes;
- edev->stats.tx_bcast_bytes = stats.tx_bcast_bytes;
- edev->stats.tx_ucast_pkts = stats.tx_ucast_pkts;
- edev->stats.tx_mcast_pkts = stats.tx_mcast_pkts;
- edev->stats.tx_bcast_pkts = stats.tx_bcast_pkts;
- edev->stats.tx_err_drop_pkts = stats.tx_err_drop_pkts;
- edev->stats.coalesced_pkts = stats.tpa_coalesced_pkts;
- edev->stats.coalesced_events = stats.tpa_coalesced_events;
- edev->stats.coalesced_aborts_num = stats.tpa_aborts_num;
- edev->stats.non_coalesced_pkts = stats.tpa_not_coalesced_pkts;
- edev->stats.coalesced_bytes = stats.tpa_coalesced_bytes;
+ p_common->no_buff_discards = stats.common.no_buff_discards;
+ p_common->packet_too_big_discard = stats.common.packet_too_big_discard;
+ p_common->ttl0_discard = stats.common.ttl0_discard;
+ p_common->rx_ucast_bytes = stats.common.rx_ucast_bytes;
+ p_common->rx_mcast_bytes = stats.common.rx_mcast_bytes;
+ p_common->rx_bcast_bytes = stats.common.rx_bcast_bytes;
+ p_common->rx_ucast_pkts = stats.common.rx_ucast_pkts;
+ p_common->rx_mcast_pkts = stats.common.rx_mcast_pkts;
+ p_common->rx_bcast_pkts = stats.common.rx_bcast_pkts;
+ p_common->mftag_filter_discards = stats.common.mftag_filter_discards;
+ p_common->mac_filter_discards = stats.common.mac_filter_discards;
- edev->stats.rx_64_byte_packets = stats.rx_64_byte_packets;
- edev->stats.rx_65_to_127_byte_packets = stats.rx_65_to_127_byte_packets;
- edev->stats.rx_128_to_255_byte_packets =
- stats.rx_128_to_255_byte_packets;
- edev->stats.rx_256_to_511_byte_packets =
- stats.rx_256_to_511_byte_packets;
- edev->stats.rx_512_to_1023_byte_packets =
- stats.rx_512_to_1023_byte_packets;
- edev->stats.rx_1024_to_1518_byte_packets =
- stats.rx_1024_to_1518_byte_packets;
- edev->stats.rx_1519_to_1522_byte_packets =
- stats.rx_1519_to_1522_byte_packets;
- edev->stats.rx_1519_to_2047_byte_packets =
- stats.rx_1519_to_2047_byte_packets;
- edev->stats.rx_2048_to_4095_byte_packets =
- stats.rx_2048_to_4095_byte_packets;
- edev->stats.rx_4096_to_9216_byte_packets =
- stats.rx_4096_to_9216_byte_packets;
- edev->stats.rx_9217_to_16383_byte_packets =
- stats.rx_9217_to_16383_byte_packets;
- edev->stats.rx_crc_errors = stats.rx_crc_errors;
- edev->stats.rx_mac_crtl_frames = stats.rx_mac_crtl_frames;
- edev->stats.rx_pause_frames = stats.rx_pause_frames;
- edev->stats.rx_pfc_frames = stats.rx_pfc_frames;
- edev->stats.rx_align_errors = stats.rx_align_errors;
- edev->stats.rx_carrier_errors = stats.rx_carrier_errors;
- edev->stats.rx_oversize_packets = stats.rx_oversize_packets;
- edev->stats.rx_jabbers = stats.rx_jabbers;
- edev->stats.rx_undersize_packets = stats.rx_undersize_packets;
- edev->stats.rx_fragments = stats.rx_fragments;
- edev->stats.tx_64_byte_packets = stats.tx_64_byte_packets;
- edev->stats.tx_65_to_127_byte_packets = stats.tx_65_to_127_byte_packets;
- edev->stats.tx_128_to_255_byte_packets =
- stats.tx_128_to_255_byte_packets;
- edev->stats.tx_256_to_511_byte_packets =
- stats.tx_256_to_511_byte_packets;
- edev->stats.tx_512_to_1023_byte_packets =
- stats.tx_512_to_1023_byte_packets;
- edev->stats.tx_1024_to_1518_byte_packets =
- stats.tx_1024_to_1518_byte_packets;
- edev->stats.tx_1519_to_2047_byte_packets =
- stats.tx_1519_to_2047_byte_packets;
- edev->stats.tx_2048_to_4095_byte_packets =
- stats.tx_2048_to_4095_byte_packets;
- edev->stats.tx_4096_to_9216_byte_packets =
- stats.tx_4096_to_9216_byte_packets;
- edev->stats.tx_9217_to_16383_byte_packets =
- stats.tx_9217_to_16383_byte_packets;
- edev->stats.tx_pause_frames = stats.tx_pause_frames;
- edev->stats.tx_pfc_frames = stats.tx_pfc_frames;
- edev->stats.tx_lpi_entry_count = stats.tx_lpi_entry_count;
- edev->stats.tx_total_collisions = stats.tx_total_collisions;
- edev->stats.brb_truncates = stats.brb_truncates;
- edev->stats.brb_discards = stats.brb_discards;
- edev->stats.tx_mac_ctrl_frames = stats.tx_mac_ctrl_frames;
+ p_common->tx_ucast_bytes = stats.common.tx_ucast_bytes;
+ p_common->tx_mcast_bytes = stats.common.tx_mcast_bytes;
+ p_common->tx_bcast_bytes = stats.common.tx_bcast_bytes;
+ p_common->tx_ucast_pkts = stats.common.tx_ucast_pkts;
+ p_common->tx_mcast_pkts = stats.common.tx_mcast_pkts;
+ p_common->tx_bcast_pkts = stats.common.tx_bcast_pkts;
+ p_common->tx_err_drop_pkts = stats.common.tx_err_drop_pkts;
+ p_common->coalesced_pkts = stats.common.tpa_coalesced_pkts;
+ p_common->coalesced_events = stats.common.tpa_coalesced_events;
+ p_common->coalesced_aborts_num = stats.common.tpa_aborts_num;
+ p_common->non_coalesced_pkts = stats.common.tpa_not_coalesced_pkts;
+ p_common->coalesced_bytes = stats.common.tpa_coalesced_bytes;
+
+ p_common->rx_64_byte_packets = stats.common.rx_64_byte_packets;
+ p_common->rx_65_to_127_byte_packets =
+ stats.common.rx_65_to_127_byte_packets;
+ p_common->rx_128_to_255_byte_packets =
+ stats.common.rx_128_to_255_byte_packets;
+ p_common->rx_256_to_511_byte_packets =
+ stats.common.rx_256_to_511_byte_packets;
+ p_common->rx_512_to_1023_byte_packets =
+ stats.common.rx_512_to_1023_byte_packets;
+ p_common->rx_1024_to_1518_byte_packets =
+ stats.common.rx_1024_to_1518_byte_packets;
+ p_common->rx_crc_errors = stats.common.rx_crc_errors;
+ p_common->rx_mac_crtl_frames = stats.common.rx_mac_crtl_frames;
+ p_common->rx_pause_frames = stats.common.rx_pause_frames;
+ p_common->rx_pfc_frames = stats.common.rx_pfc_frames;
+ p_common->rx_align_errors = stats.common.rx_align_errors;
+ p_common->rx_carrier_errors = stats.common.rx_carrier_errors;
+ p_common->rx_oversize_packets = stats.common.rx_oversize_packets;
+ p_common->rx_jabbers = stats.common.rx_jabbers;
+ p_common->rx_undersize_packets = stats.common.rx_undersize_packets;
+ p_common->rx_fragments = stats.common.rx_fragments;
+ p_common->tx_64_byte_packets = stats.common.tx_64_byte_packets;
+ p_common->tx_65_to_127_byte_packets =
+ stats.common.tx_65_to_127_byte_packets;
+ p_common->tx_128_to_255_byte_packets =
+ stats.common.tx_128_to_255_byte_packets;
+ p_common->tx_256_to_511_byte_packets =
+ stats.common.tx_256_to_511_byte_packets;
+ p_common->tx_512_to_1023_byte_packets =
+ stats.common.tx_512_to_1023_byte_packets;
+ p_common->tx_1024_to_1518_byte_packets =
+ stats.common.tx_1024_to_1518_byte_packets;
+ p_common->tx_pause_frames = stats.common.tx_pause_frames;
+ p_common->tx_pfc_frames = stats.common.tx_pfc_frames;
+ p_common->brb_truncates = stats.common.brb_truncates;
+ p_common->brb_discards = stats.common.brb_discards;
+ p_common->tx_mac_ctrl_frames = stats.common.tx_mac_ctrl_frames;
+
+ if (QEDE_IS_BB(edev)) {
+ struct qede_stats_bb *p_bb = &edev->stats.bb;
+
+ p_bb->rx_1519_to_1522_byte_packets =
+ stats.bb.rx_1519_to_1522_byte_packets;
+ p_bb->rx_1519_to_2047_byte_packets =
+ stats.bb.rx_1519_to_2047_byte_packets;
+ p_bb->rx_2048_to_4095_byte_packets =
+ stats.bb.rx_2048_to_4095_byte_packets;
+ p_bb->rx_4096_to_9216_byte_packets =
+ stats.bb.rx_4096_to_9216_byte_packets;
+ p_bb->rx_9217_to_16383_byte_packets =
+ stats.bb.rx_9217_to_16383_byte_packets;
+ p_bb->tx_1519_to_2047_byte_packets =
+ stats.bb.tx_1519_to_2047_byte_packets;
+ p_bb->tx_2048_to_4095_byte_packets =
+ stats.bb.tx_2048_to_4095_byte_packets;
+ p_bb->tx_4096_to_9216_byte_packets =
+ stats.bb.tx_4096_to_9216_byte_packets;
+ p_bb->tx_9217_to_16383_byte_packets =
+ stats.bb.tx_9217_to_16383_byte_packets;
+ p_bb->tx_lpi_entry_count = stats.bb.tx_lpi_entry_count;
+ p_bb->tx_total_collisions = stats.bb.tx_total_collisions;
+ } else {
+ struct qede_stats_ah *p_ah = &edev->stats.ah;
+
+ p_ah->rx_1519_to_max_byte_packets =
+ stats.ah.rx_1519_to_max_byte_packets;
+ p_ah->tx_1519_to_max_byte_packets =
+ stats.ah.tx_1519_to_max_byte_packets;
+ }
}
static void qede_get_stats64(struct net_device *dev,
struct rtnl_link_stats64 *stats)
{
struct qede_dev *edev = netdev_priv(dev);
+ struct qede_stats_common *p_common;
qede_fill_by_demand_stats(edev);
+ p_common = &edev->stats.common;
- stats->rx_packets = edev->stats.rx_ucast_pkts +
- edev->stats.rx_mcast_pkts +
- edev->stats.rx_bcast_pkts;
- stats->tx_packets = edev->stats.tx_ucast_pkts +
- edev->stats.tx_mcast_pkts +
- edev->stats.tx_bcast_pkts;
+ stats->rx_packets = p_common->rx_ucast_pkts + p_common->rx_mcast_pkts +
+ p_common->rx_bcast_pkts;
+ stats->tx_packets = p_common->tx_ucast_pkts + p_common->tx_mcast_pkts +
+ p_common->tx_bcast_pkts;
- stats->rx_bytes = edev->stats.rx_ucast_bytes +
- edev->stats.rx_mcast_bytes +
- edev->stats.rx_bcast_bytes;
+ stats->rx_bytes = p_common->rx_ucast_bytes + p_common->rx_mcast_bytes +
+ p_common->rx_bcast_bytes;
+ stats->tx_bytes = p_common->tx_ucast_bytes + p_common->tx_mcast_bytes +
+ p_common->tx_bcast_bytes;
- stats->tx_bytes = edev->stats.tx_ucast_bytes +
- edev->stats.tx_mcast_bytes +
- edev->stats.tx_bcast_bytes;
+ stats->tx_errors = p_common->tx_err_drop_pkts;
+ stats->multicast = p_common->rx_mcast_pkts + p_common->rx_bcast_pkts;
- stats->tx_errors = edev->stats.tx_err_drop_pkts;
- stats->multicast = edev->stats.rx_mcast_pkts +
- edev->stats.rx_bcast_pkts;
+ stats->rx_fifo_errors = p_common->no_buff_discards;
- stats->rx_fifo_errors = edev->stats.no_buff_discards;
-
- stats->collisions = edev->stats.tx_total_collisions;
- stats->rx_crc_errors = edev->stats.rx_crc_errors;
- stats->rx_frame_errors = edev->stats.rx_align_errors;
+ if (QEDE_IS_BB(edev))
+ stats->collisions = edev->stats.bb.tx_total_collisions;
+ stats->rx_crc_errors = p_common->rx_crc_errors;
+ stats->rx_frame_errors = p_common->rx_align_errors;
}
#ifdef CONFIG_QED_SRIOV
@@ -532,6 +558,9 @@
.ndo_udp_tunnel_del = qede_udp_tunnel_del,
.ndo_features_check = qede_features_check,
.ndo_xdp = qede_xdp,
+#ifdef CONFIG_RFS_ACCEL
+ .ndo_rx_flow_steer = qede_rx_flow_steer,
+#endif
};
/* -------------------------------------------------------------------------
@@ -581,7 +610,8 @@
{
struct net_device *ndev = edev->ndev;
struct pci_dev *pdev = edev->pdev;
- u32 hw_features;
+ bool udp_tunnel_enable = false;
+ netdev_features_t hw_features;
pci_set_drvdata(pdev, ndev);
@@ -603,16 +633,33 @@
NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |
NETIF_F_TSO | NETIF_F_TSO6;
- /* Encap features*/
- hw_features |= NETIF_F_GSO_GRE | NETIF_F_GSO_UDP_TUNNEL |
- NETIF_F_TSO_ECN | NETIF_F_GSO_UDP_TUNNEL_CSUM |
- NETIF_F_GSO_GRE_CSUM;
- ndev->hw_enc_features = NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |
- NETIF_F_SG | NETIF_F_TSO | NETIF_F_TSO_ECN |
- NETIF_F_TSO6 | NETIF_F_GSO_GRE |
- NETIF_F_GSO_UDP_TUNNEL | NETIF_F_RXCSUM |
- NETIF_F_GSO_UDP_TUNNEL_CSUM |
- NETIF_F_GSO_GRE_CSUM;
+ if (!IS_VF(edev) && edev->dev_info.common.num_hwfns == 1)
+ hw_features |= NETIF_F_NTUPLE;
+
+ if (edev->dev_info.common.vxlan_enable ||
+ edev->dev_info.common.geneve_enable)
+ udp_tunnel_enable = true;
+
+ if (udp_tunnel_enable || edev->dev_info.common.gre_enable) {
+ hw_features |= NETIF_F_TSO_ECN;
+ ndev->hw_enc_features = NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |
+ NETIF_F_SG | NETIF_F_TSO |
+ NETIF_F_TSO_ECN | NETIF_F_TSO6 |
+ NETIF_F_RXCSUM;
+ }
+
+ if (udp_tunnel_enable) {
+ hw_features |= (NETIF_F_GSO_UDP_TUNNEL |
+ NETIF_F_GSO_UDP_TUNNEL_CSUM);
+ ndev->hw_enc_features |= (NETIF_F_GSO_UDP_TUNNEL |
+ NETIF_F_GSO_UDP_TUNNEL_CSUM);
+ }
+
+ if (edev->dev_info.common.gre_enable) {
+ hw_features |= (NETIF_F_GSO_GRE | NETIF_F_GSO_GRE_CSUM);
+ ndev->hw_enc_features |= (NETIF_F_GSO_GRE |
+ NETIF_F_GSO_GRE_CSUM);
+ }
ndev->vlan_features = hw_features | NETIF_F_RXHASH | NETIF_F_RXCSUM |
NETIF_F_HIGHDMA;
@@ -750,7 +797,6 @@
{
struct qede_dev *edev = container_of(work, struct qede_dev,
sp_task.work);
- struct qed_dev *cdev = edev->cdev;
__qede_lock(edev);
@@ -758,24 +804,12 @@
if (edev->state == QEDE_STATE_OPEN)
qede_config_rx_mode(edev->ndev);
- if (test_and_clear_bit(QEDE_SP_VXLAN_PORT_CONFIG, &edev->sp_flags)) {
- struct qed_tunn_params tunn_params;
-
- memset(&tunn_params, 0, sizeof(tunn_params));
- tunn_params.update_vxlan_port = 1;
- tunn_params.vxlan_port = edev->vxlan_dst_port;
- qed_ops->tunn_config(cdev, &tunn_params);
+#ifdef CONFIG_RFS_ACCEL
+ if (test_and_clear_bit(QEDE_SP_ARFS_CONFIG, &edev->sp_flags)) {
+ if (edev->state == QEDE_STATE_OPEN)
+ qede_process_arfs_filters(edev, false);
}
-
- if (test_and_clear_bit(QEDE_SP_GENEVE_PORT_CONFIG, &edev->sp_flags)) {
- struct qed_tunn_params tunn_params;
-
- memset(&tunn_params, 0, sizeof(tunn_params));
- tunn_params.update_geneve_port = 1;
- tunn_params.geneve_port = edev->geneve_dst_port;
- qed_ops->tunn_config(cdev, &tunn_params);
- }
-
+#endif
__qede_unlock(edev);
}
@@ -786,6 +820,9 @@
/* 64 rx + 64 tx + 64 XDP */
memset(&pf_params, 0, sizeof(struct qed_pf_params));
pf_params.eth_pf_params.num_cons = (MAX_SB_PER_PF_MIMD - 1) * 3;
+#ifdef CONFIG_RFS_ACCEL
+ pf_params.eth_pf_params.num_arfs_filters = QEDE_RFS_MAX_FLTR;
+#endif
qed_ops->common->update_pf_params(cdev, &pf_params);
}
@@ -870,13 +907,8 @@
edev->ops->common->set_id(cdev, edev->ndev->name, DRV_MODULE_VERSION);
/* PTP not supported on VFs */
- if (!is_vf) {
- rc = qede_ptp_register_phc(edev);
- if (rc) {
- DP_NOTICE(edev, "Cannot register PHC\n");
- goto err5;
- }
- }
+ if (!is_vf)
+ qede_ptp_enable(edev, true);
edev->ops->register_ops(cdev, &qede_ll_ops, edev);
@@ -891,8 +923,6 @@
return 0;
-err5:
- unregister_netdev(edev->ndev);
err4:
qede_roce_dev_remove(edev);
err3:
@@ -940,11 +970,10 @@
DP_INFO(edev, "Starting qede_remove\n");
+ unregister_netdev(ndev);
cancel_delayed_work_sync(&edev->sp_task);
- unregister_netdev(ndev);
-
- qede_ptp_remove(edev);
+ qede_ptp_disable(edev);
qede_roce_dev_remove(edev);
@@ -1165,9 +1194,11 @@
rxq->num_rx_buffers = edev->q_num_rx_buffers;
rxq->rx_buf_size = NET_IP_ALIGN + ETH_OVERHEAD + edev->ndev->mtu;
+ rxq->rx_headroom = edev->xdp_prog ? XDP_PACKET_HEADROOM : 0;
- if (rxq->rx_buf_size > PAGE_SIZE)
- rxq->rx_buf_size = PAGE_SIZE;
+ /* Make sure that the headroom and payload fit in a single page */
+ if (rxq->rx_buf_size + rxq->rx_headroom > PAGE_SIZE)
+ rxq->rx_buf_size = PAGE_SIZE - rxq->rx_headroom;
/* Segment size to spilt a page in multiple equal parts,
* unless XDP is used in which case we'd use the entire page.
@@ -1229,7 +1260,7 @@
{
/* Free the parallel SW ring */
if (txq->is_xdp)
- kfree(txq->sw_tx_ring.pages);
+ kfree(txq->sw_tx_ring.xdp);
else
kfree(txq->sw_tx_ring.skbs);
@@ -1247,9 +1278,9 @@
/* Allocate the parallel driver ring for Tx buffers */
if (txq->is_xdp) {
- size = sizeof(*txq->sw_tx_ring.pages) * TX_RING_SIZE;
- txq->sw_tx_ring.pages = kzalloc(size, GFP_KERNEL);
- if (!txq->sw_tx_ring.pages)
+ size = sizeof(*txq->sw_tx_ring.xdp) * TX_RING_SIZE;
+ txq->sw_tx_ring.xdp = kzalloc(size, GFP_KERNEL);
+ if (!txq->sw_tx_ring.xdp)
goto err;
} else {
size = sizeof(*txq->sw_tx_ring.skbs) * TX_RING_SIZE;
@@ -1466,6 +1497,18 @@
}
for (i = 0; i < QEDE_QUEUE_CNT(edev); i++) {
+#ifdef CONFIG_RFS_ACCEL
+ struct qede_fastpath *fp = &edev->fp_array[i];
+
+ if (edev->ndev->rx_cpu_rmap && (fp->type & QEDE_FASTPATH_RX)) {
+ rc = irq_cpu_rmap_add(edev->ndev->rx_cpu_rmap,
+ edev->int_info.msix[i].vector);
+ if (rc) {
+ DP_ERR(edev, "Failed to add CPU rmap\n");
+ qede_free_arfs(edev);
+ }
+ }
+#endif
rc = request_irq(edev->int_info.msix[i].vector,
qede_msix_fp_int, 0, edev->fp_array[i].name,
&edev->fp_array[i]);
@@ -1827,8 +1870,6 @@
qede_roce_dev_event_close(edev);
edev->state = QEDE_STATE_CLOSED;
- qede_ptp_stop(edev);
-
/* Close OS Tx */
netif_tx_disable(edev->ndev);
netif_carrier_off(edev->ndev);
@@ -1847,7 +1888,12 @@
qede_vlan_mark_nonconfigured(edev);
edev->ops->fastpath_stop(edev->cdev);
-
+#ifdef CONFIG_RFS_ACCEL
+ if (!IS_VF(edev) && edev->dev_info.common.num_hwfns == 1) {
+ qede_poll_for_freeing_arfs_filters(edev);
+ qede_free_arfs(edev);
+ }
+#endif
/* Release the interrupts */
qede_sync_free_irqs(edev);
edev->ops->common->set_fp_int(edev->cdev, 0);
@@ -1899,6 +1945,13 @@
if (rc)
goto err2;
+#ifdef CONFIG_RFS_ACCEL
+ if (!IS_VF(edev) && edev->dev_info.common.num_hwfns == 1) {
+ rc = qede_alloc_arfs(edev);
+ if (rc)
+ DP_NOTICE(edev, "aRFS memory allocation failed\n");
+ }
+#endif
qede_napi_add_enable(edev);
DP_INFO(edev, "Napi added and enabled\n");
@@ -1925,13 +1978,10 @@
qede_roce_dev_event_open(edev);
- qede_ptp_start(edev, (mode == QEDE_LOAD_NORMAL));
-
edev->state = QEDE_STATE_OPEN;
DP_INFO(edev, "Ending successfully qede load\n");
-
goto out;
err4:
qede_sync_free_irqs(edev);
diff --git a/drivers/net/ethernet/qlogic/qede/qede_ptp.c b/drivers/net/ethernet/qlogic/qede/qede_ptp.c
index 2e62dec..24f06e2 100644
--- a/drivers/net/ethernet/qlogic/qede/qede_ptp.c
+++ b/drivers/net/ethernet/qlogic/qede/qede_ptp.c
@@ -181,6 +181,7 @@
skb_tstamp_tx(ptp->tx_skb, &shhwtstamps);
dev_kfree_skb_any(ptp->tx_skb);
ptp->tx_skb = NULL;
+ clear_bit_unlock(QEDE_FLAGS_PTP_TX_IN_PRORGESS, &edev->flags);
DP_VERBOSE(edev, QED_MSG_DEBUG,
"Tx timestamp, timestamp cycles = %llu, ns = %llu\n",
@@ -206,23 +207,10 @@
return phc_cycles;
}
-static void qede_ptp_init_cc(struct qede_dev *edev)
-{
- struct qede_ptp *ptp;
-
- ptp = edev->ptp;
- if (!ptp)
- return;
-
- memset(&ptp->cc, 0, sizeof(ptp->cc));
- ptp->cc.read = qede_ptp_read_cc;
- ptp->cc.mask = CYCLECOUNTER_MASK(64);
- ptp->cc.shift = 0;
- ptp->cc.mult = 1;
-}
-
static int qede_ptp_cfg_filters(struct qede_dev *edev)
{
+ enum qed_ptp_hwtstamp_tx_type tx_type = QED_PTP_HWTSTAMP_TX_ON;
+ enum qed_ptp_filter_type rx_filter = QED_PTP_FILTER_NONE;
struct qede_ptp *ptp = edev->ptp;
if (!ptp)
@@ -236,7 +224,12 @@
switch (ptp->tx_type) {
case HWTSTAMP_TX_ON:
edev->flags |= QEDE_TX_TIMESTAMPING_EN;
- ptp->ops->hwtstamp_tx_on(edev->cdev);
+ tx_type = QED_PTP_HWTSTAMP_TX_ON;
+ break;
+
+ case HWTSTAMP_TX_OFF:
+ edev->flags &= ~QEDE_TX_TIMESTAMPING_EN;
+ tx_type = QED_PTP_HWTSTAMP_TX_OFF;
break;
case HWTSTAMP_TX_ONESTEP_SYNC:
@@ -247,42 +240,57 @@
spin_lock_bh(&ptp->lock);
switch (ptp->rx_filter) {
case HWTSTAMP_FILTER_NONE:
+ rx_filter = QED_PTP_FILTER_NONE;
break;
case HWTSTAMP_FILTER_ALL:
case HWTSTAMP_FILTER_SOME:
ptp->rx_filter = HWTSTAMP_FILTER_NONE;
+ rx_filter = QED_PTP_FILTER_ALL;
break;
case HWTSTAMP_FILTER_PTP_V1_L4_EVENT:
+ ptp->rx_filter = HWTSTAMP_FILTER_PTP_V1_L4_EVENT;
+ rx_filter = QED_PTP_FILTER_V1_L4_EVENT;
+ break;
case HWTSTAMP_FILTER_PTP_V1_L4_SYNC:
case HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ:
ptp->rx_filter = HWTSTAMP_FILTER_PTP_V1_L4_EVENT;
/* Initialize PTP detection for UDP/IPv4 events */
- ptp->ops->cfg_rx_filters(edev->cdev, QED_PTP_FILTER_IPV4);
+ rx_filter = QED_PTP_FILTER_V1_L4_GEN;
break;
case HWTSTAMP_FILTER_PTP_V2_L4_EVENT:
+ ptp->rx_filter = HWTSTAMP_FILTER_PTP_V2_L4_EVENT;
+ rx_filter = QED_PTP_FILTER_V2_L4_EVENT;
+ break;
case HWTSTAMP_FILTER_PTP_V2_L4_SYNC:
case HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ:
ptp->rx_filter = HWTSTAMP_FILTER_PTP_V2_L4_EVENT;
/* Initialize PTP detection for UDP/IPv4 or UDP/IPv6 events */
- ptp->ops->cfg_rx_filters(edev->cdev, QED_PTP_FILTER_IPV4_IPV6);
+ rx_filter = QED_PTP_FILTER_V2_L4_GEN;
break;
case HWTSTAMP_FILTER_PTP_V2_L2_EVENT:
+ ptp->rx_filter = HWTSTAMP_FILTER_PTP_V2_L2_EVENT;
+ rx_filter = QED_PTP_FILTER_V2_L2_EVENT;
+ break;
case HWTSTAMP_FILTER_PTP_V2_L2_SYNC:
case HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ:
ptp->rx_filter = HWTSTAMP_FILTER_PTP_V2_L2_EVENT;
/* Initialize PTP detection L2 events */
- ptp->ops->cfg_rx_filters(edev->cdev, QED_PTP_FILTER_L2);
+ rx_filter = QED_PTP_FILTER_V2_L2_GEN;
break;
case HWTSTAMP_FILTER_PTP_V2_EVENT:
+ ptp->rx_filter = HWTSTAMP_FILTER_PTP_V2_EVENT;
+ rx_filter = QED_PTP_FILTER_V2_EVENT;
+ break;
case HWTSTAMP_FILTER_PTP_V2_SYNC:
case HWTSTAMP_FILTER_PTP_V2_DELAY_REQ:
ptp->rx_filter = HWTSTAMP_FILTER_PTP_V2_EVENT;
/* Initialize PTP detection L2, UDP/IPv4 or UDP/IPv6 events */
- ptp->ops->cfg_rx_filters(edev->cdev,
- QED_PTP_FILTER_L2_IPV4_IPV6);
+ rx_filter = QED_PTP_FILTER_V2_GEN;
break;
}
+ ptp->ops->cfg_filters(edev->cdev, rx_filter, tx_type);
+
spin_unlock_bh(&ptp->lock);
return 0;
@@ -324,61 +332,6 @@
sizeof(config)) ? -EFAULT : 0;
}
-/* Called during load, to initialize PTP-related stuff */
-static void qede_ptp_init(struct qede_dev *edev, bool init_tc)
-{
- struct qede_ptp *ptp;
- int rc;
-
- ptp = edev->ptp;
- if (!ptp)
- return;
-
- spin_lock_init(&ptp->lock);
-
- /* Configure PTP in HW */
- rc = ptp->ops->enable(edev->cdev);
- if (rc) {
- DP_ERR(edev, "Stopping PTP initialization\n");
- return;
- }
-
- /* Init work queue for Tx timestamping */
- INIT_WORK(&ptp->work, qede_ptp_task);
-
- /* Init cyclecounter and timecounter. This is done only in the first
- * load. If done in every load, PTP application will fail when doing
- * unload / load (e.g. MTU change) while it is running.
- */
- if (init_tc) {
- qede_ptp_init_cc(edev);
- timecounter_init(&ptp->tc, &ptp->cc,
- ktime_to_ns(ktime_get_real()));
- }
-
- DP_VERBOSE(edev, QED_MSG_DEBUG, "PTP initialization is successful\n");
-}
-
-void qede_ptp_start(struct qede_dev *edev, bool init_tc)
-{
- qede_ptp_init(edev, init_tc);
- qede_ptp_cfg_filters(edev);
-}
-
-void qede_ptp_remove(struct qede_dev *edev)
-{
- struct qede_ptp *ptp;
-
- ptp = edev->ptp;
- if (ptp && ptp->clock) {
- ptp_clock_unregister(ptp->clock);
- ptp->clock = NULL;
- }
-
- kfree(ptp);
- edev->ptp = NULL;
-}
-
int qede_ptp_get_ts_info(struct qede_dev *edev, struct ethtool_ts_info *info)
{
struct qede_ptp *ptp = edev->ptp;
@@ -417,8 +370,7 @@
return 0;
}
-/* Called during unload, to stop PTP-related stuff */
-void qede_ptp_stop(struct qede_dev *edev)
+void qede_ptp_disable(struct qede_dev *edev)
{
struct qede_ptp *ptp;
@@ -426,6 +378,11 @@
if (!ptp)
return;
+ if (ptp->clock) {
+ ptp_clock_unregister(ptp->clock);
+ ptp->clock = NULL;
+ }
+
/* Cancel PTP work queue. Should be done after the Tx queues are
* drained to prevent additional scheduling.
*/
@@ -439,11 +396,54 @@
spin_lock_bh(&ptp->lock);
ptp->ops->disable(edev->cdev);
spin_unlock_bh(&ptp->lock);
+
+ kfree(ptp);
+ edev->ptp = NULL;
}
-int qede_ptp_register_phc(struct qede_dev *edev)
+static int qede_ptp_init(struct qede_dev *edev, bool init_tc)
{
struct qede_ptp *ptp;
+ int rc;
+
+ ptp = edev->ptp;
+ if (!ptp)
+ return -EINVAL;
+
+ spin_lock_init(&ptp->lock);
+
+ /* Configure PTP in HW */
+ rc = ptp->ops->enable(edev->cdev);
+ if (rc) {
+ DP_INFO(edev, "PTP HW enable failed\n");
+ return rc;
+ }
+
+ /* Init work queue for Tx timestamping */
+ INIT_WORK(&ptp->work, qede_ptp_task);
+
+ /* Init cyclecounter and timecounter. This is done only in the first
+ * load. If done in every load, PTP application will fail when doing
+ * unload / load (e.g. MTU change) while it is running.
+ */
+ if (init_tc) {
+ memset(&ptp->cc, 0, sizeof(ptp->cc));
+ ptp->cc.read = qede_ptp_read_cc;
+ ptp->cc.mask = CYCLECOUNTER_MASK(64);
+ ptp->cc.shift = 0;
+ ptp->cc.mult = 1;
+
+ timecounter_init(&ptp->tc, &ptp->cc,
+ ktime_to_ns(ktime_get_real()));
+ }
+
+ return rc;
+}
+
+int qede_ptp_enable(struct qede_dev *edev, bool init_tc)
+{
+ struct qede_ptp *ptp;
+ int rc;
ptp = kzalloc(sizeof(*ptp), GFP_KERNEL);
if (!ptp) {
@@ -454,14 +454,19 @@
ptp->edev = edev;
ptp->ops = edev->ops->ptp;
if (!ptp->ops) {
- kfree(ptp);
- edev->ptp = NULL;
- DP_ERR(edev, "PTP clock registeration failed\n");
- return -EIO;
+ DP_INFO(edev, "PTP enable failed\n");
+ rc = -EIO;
+ goto err1;
}
edev->ptp = ptp;
+ rc = qede_ptp_init(edev, init_tc);
+ if (rc)
+ goto err1;
+
+ qede_ptp_cfg_filters(edev);
+
/* Fill the ptp_clock_info struct and register PTP clock */
ptp->clock_info.owner = THIS_MODULE;
snprintf(ptp->clock_info.name, 16, "%s", edev->ndev->name);
@@ -478,13 +483,21 @@
ptp->clock = ptp_clock_register(&ptp->clock_info, &edev->pdev->dev);
if (IS_ERR(ptp->clock)) {
- ptp->clock = NULL;
- kfree(ptp);
- edev->ptp = NULL;
+ rc = -EINVAL;
DP_ERR(edev, "PTP clock registeration failed\n");
+ goto err2;
}
return 0;
+
+err2:
+ qede_ptp_disable(edev);
+ ptp->clock = NULL;
+err1:
+ kfree(ptp);
+ edev->ptp = NULL;
+
+ return rc;
}
void qede_ptp_tx_ts(struct qede_dev *edev, struct sk_buff *skb)
@@ -495,6 +508,9 @@
if (!ptp)
return;
+ if (test_and_set_bit_lock(QEDE_FLAGS_PTP_TX_IN_PRORGESS, &edev->flags))
+ return;
+
if (unlikely(!(edev->flags & QEDE_TX_TIMESTAMPING_EN))) {
DP_NOTICE(edev,
"Tx timestamping was not enabled, this packet will not be timestamped\n");
diff --git a/drivers/net/ethernet/qlogic/qede/qede_ptp.h b/drivers/net/ethernet/qlogic/qede/qede_ptp.h
index f328f9b..691a14c 100644
--- a/drivers/net/ethernet/qlogic/qede/qede_ptp.h
+++ b/drivers/net/ethernet/qlogic/qede/qede_ptp.h
@@ -40,10 +40,8 @@
void qede_ptp_rx_ts(struct qede_dev *edev, struct sk_buff *skb);
void qede_ptp_tx_ts(struct qede_dev *edev, struct sk_buff *skb);
int qede_ptp_hw_ts(struct qede_dev *edev, struct ifreq *req);
-void qede_ptp_start(struct qede_dev *edev, bool init_tc);
-void qede_ptp_stop(struct qede_dev *edev);
-void qede_ptp_remove(struct qede_dev *edev);
-int qede_ptp_register_phc(struct qede_dev *edev);
+void qede_ptp_disable(struct qede_dev *edev);
+int qede_ptp_enable(struct qede_dev *edev, bool init_tc);
int qede_ptp_get_ts_info(struct qede_dev *edev, struct ethtool_ts_info *ts);
static inline void qede_ptp_record_rx_ts(struct qede_dev *edev,
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_sriov_common.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_sriov_common.c
index d710705..2f656f3 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_sriov_common.c
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_sriov_common.c
@@ -128,6 +128,8 @@
return 0;
pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
+ if (!pos)
+ return 0;
pci_read_config_word(dev, pos + PCI_SRIOV_VF_OFFSET, &offset);
pci_read_config_word(dev, pos + PCI_SRIOV_VF_STRIDE, &stride);
diff --git a/drivers/net/ethernet/qlogic/qlge/qlge_main.c b/drivers/net/ethernet/qlogic/qlge/qlge_main.c
index e9e6470..1188d42 100644
--- a/drivers/net/ethernet/qlogic/qlge/qlge_main.c
+++ b/drivers/net/ethernet/qlogic/qlge/qlge_main.c
@@ -4686,7 +4686,8 @@
/*
* Set up the operating parameters.
*/
- qdev->workqueue = alloc_ordered_workqueue(ndev->name, WQ_MEM_RECLAIM);
+ qdev->workqueue = alloc_ordered_workqueue("%s", WQ_MEM_RECLAIM,
+ ndev->name);
INIT_DELAYED_WORK(&qdev->asic_reset_work, ql_asic_reset_work);
INIT_DELAYED_WORK(&qdev->mpi_reset_work, ql_mpi_reset_work);
INIT_DELAYED_WORK(&qdev->mpi_work, ql_mpi_work);
diff --git a/drivers/net/ethernet/qualcomm/emac/emac-sgmii-qdf2400.c b/drivers/net/ethernet/qualcomm/emac/emac-sgmii-qdf2400.c
index f62c215..7116be4 100644
--- a/drivers/net/ethernet/qualcomm/emac/emac-sgmii-qdf2400.c
+++ b/drivers/net/ethernet/qualcomm/emac/emac-sgmii-qdf2400.c
@@ -26,6 +26,7 @@
/* SGMII digital lane registers */
#define EMAC_SGMII_LN_DRVR_CTRL0 0x000C
+#define EMAC_SGMII_LN_DRVR_CTRL1 0x0010
#define EMAC_SGMII_LN_DRVR_TAP_EN 0x0018
#define EMAC_SGMII_LN_TX_MARGINING 0x001C
#define EMAC_SGMII_LN_TX_PRE 0x0020
@@ -48,6 +49,7 @@
#define EMAC_SGMII_LN_RX_EN_SIGNAL 0x02AC
#define EMAC_SGMII_LN_RX_MISC_CNTRL0 0x02B8
#define EMAC_SGMII_LN_DRVR_LOGIC_CLKDIV 0x02C8
+#define EMAC_SGMII_LN_RX_RESECODE_OFFSET 0x02CC
/* SGMII digital lane register values */
#define UCDR_STEP_BY_TWO_MODE0 BIT(7)
@@ -73,6 +75,8 @@
#define CML_GEAR_MODE(x) (((x) & 7) << 3)
#define CML2CMOS_IBOOST_MODE(x) ((x) & 7)
+#define RESCODE_OFFSET(x) ((x) & 0x1f)
+
#define MIXER_LOADB_MODE(x) (((x) & 0xf) << 2)
#define MIXER_DATARATE_MODE(x) ((x) & 3)
@@ -159,6 +163,8 @@
{EMAC_SGMII_LN_PARALLEL_RATE, PARALLEL_RATE_MODE0(1)},
{EMAC_SGMII_LN_TX_BAND_MODE, BAND_MODE0(1)},
{EMAC_SGMII_LN_RX_BAND, BAND_MODE0(2)},
+ {EMAC_SGMII_LN_DRVR_CTRL1, RESCODE_OFFSET(7)},
+ {EMAC_SGMII_LN_RX_RESECODE_OFFSET, RESCODE_OFFSET(9)},
{EMAC_SGMII_LN_LANE_MODE, LANE_MODE(26)},
{EMAC_SGMII_LN_RX_RCVR_PATH1_MODE0, CDR_PD_SEL_MODE0(2) |
EN_DLL_MODE0 | EN_IQ_DCC_MODE0 | EN_IQCAL_MODE0},
diff --git a/drivers/net/ethernet/qualcomm/emac/emac-sgmii.c b/drivers/net/ethernet/qualcomm/emac/emac-sgmii.c
index 040b289..18c184e 100644
--- a/drivers/net/ethernet/qualcomm/emac/emac-sgmii.c
+++ b/drivers/net/ethernet/qualcomm/emac/emac-sgmii.c
@@ -13,6 +13,7 @@
/* Qualcomm Technologies, Inc. EMAC SGMII Controller driver.
*/
+#include <linux/interrupt.h>
#include <linux/iopoll.h>
#include <linux/acpi.h>
#include <linux/of_device.h>
diff --git a/drivers/net/ethernet/realtek/8139cp.c b/drivers/net/ethernet/realtek/8139cp.c
index 672f6b6..72233ab 100644
--- a/drivers/net/ethernet/realtek/8139cp.c
+++ b/drivers/net/ethernet/realtek/8139cp.c
@@ -1406,27 +1406,29 @@
}
}
-static int cp_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int cp_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct cp_private *cp = netdev_priv(dev);
int rc;
unsigned long flags;
spin_lock_irqsave(&cp->lock, flags);
- rc = mii_ethtool_gset(&cp->mii_if, cmd);
+ rc = mii_ethtool_get_link_ksettings(&cp->mii_if, cmd);
spin_unlock_irqrestore(&cp->lock, flags);
return rc;
}
-static int cp_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int cp_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct cp_private *cp = netdev_priv(dev);
int rc;
unsigned long flags;
spin_lock_irqsave(&cp->lock, flags);
- rc = mii_ethtool_sset(&cp->mii_if, cmd);
+ rc = mii_ethtool_set_link_ksettings(&cp->mii_if, cmd);
spin_unlock_irqrestore(&cp->lock, flags);
return rc;
@@ -1578,8 +1580,6 @@
.get_drvinfo = cp_get_drvinfo,
.get_regs_len = cp_get_regs_len,
.get_sset_count = cp_get_sset_count,
- .get_settings = cp_get_settings,
- .set_settings = cp_set_settings,
.nway_reset = cp_nway_reset,
.get_link = ethtool_op_get_link,
.get_msglevel = cp_get_msglevel,
@@ -1593,6 +1593,8 @@
.get_eeprom = cp_get_eeprom,
.set_eeprom = cp_set_eeprom,
.get_ringparam = cp_get_ringparam,
+ .get_link_ksettings = cp_get_link_ksettings,
+ .set_link_ksettings = cp_set_link_ksettings,
};
static int cp_ioctl (struct net_device *dev, struct ifreq *rq, int cmd)
diff --git a/drivers/net/ethernet/realtek/8139too.c b/drivers/net/ethernet/realtek/8139too.c
index 8963175..ca22f28 100644
--- a/drivers/net/ethernet/realtek/8139too.c
+++ b/drivers/net/ethernet/realtek/8139too.c
@@ -2384,21 +2384,23 @@
strlcpy(info->bus_info, pci_name(tp->pci_dev), sizeof(info->bus_info));
}
-static int rtl8139_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int rtl8139_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct rtl8139_private *tp = netdev_priv(dev);
spin_lock_irq(&tp->lock);
- mii_ethtool_gset(&tp->mii, cmd);
+ mii_ethtool_get_link_ksettings(&tp->mii, cmd);
spin_unlock_irq(&tp->lock);
return 0;
}
-static int rtl8139_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int rtl8139_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct rtl8139_private *tp = netdev_priv(dev);
int rc;
spin_lock_irq(&tp->lock);
- rc = mii_ethtool_sset(&tp->mii, cmd);
+ rc = mii_ethtool_set_link_ksettings(&tp->mii, cmd);
spin_unlock_irq(&tp->lock);
return rc;
}
@@ -2480,8 +2482,6 @@
static const struct ethtool_ops rtl8139_ethtool_ops = {
.get_drvinfo = rtl8139_get_drvinfo,
- .get_settings = rtl8139_get_settings,
- .set_settings = rtl8139_set_settings,
.get_regs_len = rtl8139_get_regs_len,
.get_regs = rtl8139_get_regs,
.nway_reset = rtl8139_nway_reset,
@@ -2493,6 +2493,8 @@
.get_strings = rtl8139_get_strings,
.get_sset_count = rtl8139_get_sset_count,
.get_ethtool_stats = rtl8139_get_ethtool_stats,
+ .get_link_ksettings = rtl8139_get_link_ksettings,
+ .set_link_ksettings = rtl8139_set_link_ksettings,
};
static int netdev_ioctl(struct net_device *dev, struct ifreq *rq, int cmd)
diff --git a/drivers/net/ethernet/realtek/r8169.c b/drivers/net/ethernet/realtek/r8169.c
index 81f18a8..0a8f281 100644
--- a/drivers/net/ethernet/realtek/r8169.c
+++ b/drivers/net/ethernet/realtek/r8169.c
@@ -817,7 +817,8 @@
} csi_ops;
int (*set_speed)(struct net_device *, u8 aneg, u16 sp, u8 dpx, u32 adv);
- int (*get_settings)(struct net_device *, struct ethtool_cmd *);
+ int (*get_link_ksettings)(struct net_device *,
+ struct ethtool_link_ksettings *);
void (*phy_reset_enable)(struct rtl8169_private *tp);
void (*hw_start)(struct net_device *);
unsigned int (*phy_reset_pending)(struct rtl8169_private *tp);
@@ -2115,41 +2116,49 @@
__vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), swab16(opts2 & 0xffff));
}
-static int rtl8169_gset_tbi(struct net_device *dev, struct ethtool_cmd *cmd)
+static int rtl8169_get_link_ksettings_tbi(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct rtl8169_private *tp = netdev_priv(dev);
void __iomem *ioaddr = tp->mmio_addr;
u32 status;
+ u32 supported, advertising;
- cmd->supported =
+ supported =
SUPPORTED_1000baseT_Full | SUPPORTED_Autoneg | SUPPORTED_FIBRE;
- cmd->port = PORT_FIBRE;
- cmd->transceiver = XCVR_INTERNAL;
+ cmd->base.port = PORT_FIBRE;
status = RTL_R32(TBICSR);
- cmd->advertising = (status & TBINwEnable) ? ADVERTISED_Autoneg : 0;
- cmd->autoneg = !!(status & TBINwEnable);
+ advertising = (status & TBINwEnable) ? ADVERTISED_Autoneg : 0;
+ cmd->base.autoneg = !!(status & TBINwEnable);
- ethtool_cmd_speed_set(cmd, SPEED_1000);
- cmd->duplex = DUPLEX_FULL; /* Always set */
+ cmd->base.speed = SPEED_1000;
+ cmd->base.duplex = DUPLEX_FULL; /* Always set */
+
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
+ advertising);
return 0;
}
-static int rtl8169_gset_xmii(struct net_device *dev, struct ethtool_cmd *cmd)
+static int rtl8169_get_link_ksettings_xmii(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct rtl8169_private *tp = netdev_priv(dev);
- return mii_ethtool_gset(&tp->mii, cmd);
+ return mii_ethtool_get_link_ksettings(&tp->mii, cmd);
}
-static int rtl8169_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int rtl8169_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct rtl8169_private *tp = netdev_priv(dev);
int rc;
rtl_lock_work(tp);
- rc = tp->get_settings(dev, cmd);
+ rc = tp->get_link_ksettings(dev, cmd);
rtl_unlock_work(tp);
return rc;
@@ -2356,7 +2365,6 @@
.get_drvinfo = rtl8169_get_drvinfo,
.get_regs_len = rtl8169_get_regs_len,
.get_link = ethtool_op_get_link,
- .get_settings = rtl8169_get_settings,
.set_settings = rtl8169_set_settings,
.get_msglevel = rtl8169_get_msglevel,
.set_msglevel = rtl8169_set_msglevel,
@@ -2368,6 +2376,7 @@
.get_ethtool_stats = rtl8169_get_ethtool_stats,
.get_ts_info = ethtool_op_get_ts_info,
.nway_reset = rtl8169_nway_reset,
+ .get_link_ksettings = rtl8169_get_link_ksettings,
};
static void rtl8169_get_mac_version(struct rtl8169_private *tp,
@@ -8351,14 +8360,14 @@
if (rtl_tbi_enabled(tp)) {
tp->set_speed = rtl8169_set_speed_tbi;
- tp->get_settings = rtl8169_gset_tbi;
+ tp->get_link_ksettings = rtl8169_get_link_ksettings_tbi;
tp->phy_reset_enable = rtl8169_tbi_reset_enable;
tp->phy_reset_pending = rtl8169_tbi_reset_pending;
tp->link_ok = rtl8169_tbi_link_ok;
tp->do_ioctl = rtl_tbi_ioctl;
} else {
tp->set_speed = rtl8169_set_speed_xmii;
- tp->get_settings = rtl8169_gset_xmii;
+ tp->get_link_ksettings = rtl8169_get_link_ksettings_xmii;
tp->phy_reset_enable = rtl8169_xmii_reset_enable;
tp->phy_reset_pending = rtl8169_xmii_reset_pending;
tp->link_ok = rtl8169_xmii_link_ok;
@@ -8444,9 +8453,7 @@
tp->opts1_mask = (tp->mac_version != RTL_GIGA_MAC_VER_01) ?
~(RxBOVF | RxFOVF) : ~0;
- init_timer(&tp->timer);
- tp->timer.data = (unsigned long) dev;
- tp->timer.function = rtl8169_phy_timer;
+ setup_timer(&tp->timer, rtl8169_phy_timer, (unsigned long)dev);
tp->rtl_fw = RTL_FIRMWARE_UNKNOWN;
diff --git a/drivers/net/ethernet/renesas/ravb_main.c b/drivers/net/ethernet/renesas/ravb_main.c
index 8cfc4a5..3cd7989 100644
--- a/drivers/net/ethernet/renesas/ravb_main.c
+++ b/drivers/net/ethernet/renesas/ravb_main.c
@@ -1516,11 +1516,12 @@
spin_unlock_irqrestore(&priv->lock, flags);
return NETDEV_TX_BUSY;
}
- entry = priv->cur_tx[q] % (priv->num_tx_ring[q] * NUM_TX_DESC);
- priv->tx_skb[q][entry / NUM_TX_DESC] = skb;
if (skb_put_padto(skb, ETH_ZLEN))
- goto drop;
+ goto exit;
+
+ entry = priv->cur_tx[q] % (priv->num_tx_ring[q] * NUM_TX_DESC);
+ priv->tx_skb[q][entry / NUM_TX_DESC] = skb;
buffer = PTR_ALIGN(priv->tx_align[q], DPTR_ALIGN) +
entry / NUM_TX_DESC * DPTR_ALIGN;
diff --git a/drivers/net/ethernet/rocker/rocker_main.c b/drivers/net/ethernet/rocker/rocker_main.c
index 0f63a44..bab1361 100644
--- a/drivers/net/ethernet/rocker/rocker_main.c
+++ b/drivers/net/ethernet/rocker/rocker_main.c
@@ -33,6 +33,7 @@
#include <net/rtnetlink.h>
#include <net/netevent.h>
#include <net/arp.h>
+#include <net/fib_rules.h>
#include <linux/io-64-nonatomic-lo-hi.h>
#include <generated/utsrelease.h>
@@ -1115,7 +1116,7 @@
const struct rocker_desc_info *desc_info,
void *priv)
{
- struct ethtool_cmd *ecmd = priv;
+ struct ethtool_link_ksettings *ecmd = priv;
const struct rocker_tlv *attrs[ROCKER_TLV_CMD_MAX + 1];
const struct rocker_tlv *info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_MAX + 1];
u32 speed;
@@ -1137,13 +1138,14 @@
duplex = rocker_tlv_get_u8(info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_DUPLEX]);
autoneg = rocker_tlv_get_u8(info_attrs[ROCKER_TLV_CMD_PORT_SETTINGS_AUTONEG]);
- ecmd->transceiver = XCVR_INTERNAL;
- ecmd->supported = SUPPORTED_TP;
- ecmd->phy_address = 0xff;
- ecmd->port = PORT_TP;
- ethtool_cmd_speed_set(ecmd, speed);
- ecmd->duplex = duplex ? DUPLEX_FULL : DUPLEX_HALF;
- ecmd->autoneg = autoneg ? AUTONEG_ENABLE : AUTONEG_DISABLE;
+ ethtool_link_ksettings_zero_link_mode(ecmd, supported);
+ ethtool_link_ksettings_add_link_mode(ecmd, supported, TP);
+
+ ecmd->base.phy_address = 0xff;
+ ecmd->base.port = PORT_TP;
+ ecmd->base.speed = speed;
+ ecmd->base.duplex = duplex ? DUPLEX_FULL : DUPLEX_HALF;
+ ecmd->base.autoneg = autoneg ? AUTONEG_ENABLE : AUTONEG_DISABLE;
return 0;
}
@@ -1250,7 +1252,7 @@
struct rocker_desc_info *desc_info,
void *priv)
{
- struct ethtool_cmd *ecmd = priv;
+ struct ethtool_link_ksettings *ecmd = priv;
struct rocker_tlv *cmd_info;
if (rocker_tlv_put_u16(desc_info, ROCKER_TLV_CMD_TYPE,
@@ -1263,13 +1265,13 @@
rocker_port->pport))
return -EMSGSIZE;
if (rocker_tlv_put_u32(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_SPEED,
- ethtool_cmd_speed(ecmd)))
+ ecmd->base.speed))
return -EMSGSIZE;
if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_DUPLEX,
- ecmd->duplex))
+ ecmd->base.duplex))
return -EMSGSIZE;
if (rocker_tlv_put_u8(desc_info, ROCKER_TLV_CMD_PORT_SETTINGS_AUTONEG,
- ecmd->autoneg))
+ ecmd->base.autoneg))
return -EMSGSIZE;
rocker_tlv_nest_end(desc_info, cmd_info);
return 0;
@@ -1347,8 +1349,9 @@
return 0;
}
-static int rocker_cmd_get_port_settings_ethtool(struct rocker_port *rocker_port,
- struct ethtool_cmd *ecmd)
+static int
+rocker_cmd_get_port_settings_ethtool(struct rocker_port *rocker_port,
+ struct ethtool_link_ksettings *ecmd)
{
return rocker_cmd_exec(rocker_port, false,
rocker_cmd_get_port_settings_prep, NULL,
@@ -1373,12 +1376,17 @@
rocker_cmd_get_port_settings_mode_proc, p_mode);
}
-static int rocker_cmd_set_port_settings_ethtool(struct rocker_port *rocker_port,
- struct ethtool_cmd *ecmd)
+static int
+rocker_cmd_set_port_settings_ethtool(struct rocker_port *rocker_port,
+ const struct ethtool_link_ksettings *ecmd)
{
+ struct ethtool_link_ksettings copy_ecmd;
+
+ memcpy(©_ecmd, ecmd, sizeof(copy_ecmd));
+
return rocker_cmd_exec(rocker_port, false,
rocker_cmd_set_port_settings_ethtool_prep,
- ecmd, NULL, NULL);
+ ©_ecmd, NULL, NULL);
}
static int rocker_cmd_set_port_settings_macaddr(struct rocker_port *rocker_port,
@@ -2168,7 +2176,10 @@
struct rocker_fib_event_work {
struct work_struct work;
- struct fib_entry_notifier_info fen_info;
+ union {
+ struct fib_entry_notifier_info fen_info;
+ struct fib_rule_notifier_info fr_info;
+ };
struct rocker *rocker;
unsigned long event;
};
@@ -2178,6 +2189,7 @@
struct rocker_fib_event_work *fib_work =
container_of(work, struct rocker_fib_event_work, work);
struct rocker *rocker = fib_work->rocker;
+ struct fib_rule *rule;
int err;
/* Protect internal structures from changes */
@@ -2195,7 +2207,10 @@
break;
case FIB_EVENT_RULE_ADD: /* fall through */
case FIB_EVENT_RULE_DEL:
- rocker_world_fib4_abort(rocker);
+ rule = fib_work->fr_info.rule;
+ if (!fib4_rule_default(rule))
+ rocker_world_fib4_abort(rocker);
+ fib_rule_put(rule);
break;
}
rtnl_unlock();
@@ -2226,6 +2241,11 @@
*/
fib_info_hold(fib_work->fen_info.fi);
break;
+ case FIB_EVENT_RULE_ADD: /* fall through */
+ case FIB_EVENT_RULE_DEL:
+ memcpy(&fib_work->fr_info, ptr, sizeof(fib_work->fr_info));
+ fib_rule_get(fib_work->fr_info.rule);
+ break;
}
queue_work(rocker->rocker_owq, &fib_work->work);
@@ -2237,16 +2257,18 @@
* ethtool interface
********************/
-static int rocker_port_get_settings(struct net_device *dev,
- struct ethtool_cmd *ecmd)
+static int
+rocker_port_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *ecmd)
{
struct rocker_port *rocker_port = netdev_priv(dev);
return rocker_cmd_get_port_settings_ethtool(rocker_port, ecmd);
}
-static int rocker_port_set_settings(struct net_device *dev,
- struct ethtool_cmd *ecmd)
+static int
+rocker_port_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *ecmd)
{
struct rocker_port *rocker_port = netdev_priv(dev);
@@ -2388,13 +2410,13 @@
}
static const struct ethtool_ops rocker_port_ethtool_ops = {
- .get_settings = rocker_port_get_settings,
- .set_settings = rocker_port_set_settings,
.get_drvinfo = rocker_port_get_drvinfo,
.get_link = ethtool_op_get_link,
.get_strings = rocker_port_get_strings,
.get_ethtool_stats = rocker_port_get_stats,
.get_sset_count = rocker_port_get_sset_count,
+ .get_link_ksettings = rocker_port_get_link_ksettings,
+ .set_link_ksettings = rocker_port_set_link_ksettings,
};
/*****************
diff --git a/drivers/net/ethernet/sfc/ef10.c b/drivers/net/ethernet/sfc/ef10.c
index c60c2d4..78efb28 100644
--- a/drivers/net/ethernet/sfc/ef10.c
+++ b/drivers/net/ethernet/sfc/ef10.c
@@ -119,6 +119,7 @@
bool mc_promisc;
/* Whether in multicast promiscuous mode when last changed */
bool mc_promisc_last;
+ bool mc_overflow; /* Too many MC addrs; should always imply mc_promisc */
bool vlan_filter;
struct list_head vlan_list;
};
@@ -5058,6 +5059,7 @@
struct netdev_hw_addr *mc;
unsigned int i, addr_count;
+ table->mc_overflow = false;
table->mc_promisc = !!(net_dev->flags & (IFF_PROMISC | IFF_ALLMULTI));
addr_count = netdev_mc_count(net_dev);
@@ -5065,6 +5067,7 @@
netdev_for_each_mc_addr(mc, net_dev) {
if (i >= EFX_EF10_FILTER_DEV_MC_MAX) {
table->mc_promisc = true;
+ table->mc_overflow = true;
break;
}
ether_addr_copy(table->dev_mc_list[i].addr, mc->addr);
@@ -5469,12 +5472,15 @@
}
} else {
/* If we failed to insert promiscuous filters, don't
- * rollback. Regardless, also insert the mc_list
+ * rollback. Regardless, also insert the mc_list,
+ * unless it's incomplete due to overflow
*/
efx_ef10_filter_insert_def(efx, vlan,
EFX_ENCAP_TYPE_NONE,
true, false);
- efx_ef10_filter_insert_addr_list(efx, vlan, true, false);
+ if (!table->mc_overflow)
+ efx_ef10_filter_insert_addr_list(efx, vlan,
+ true, false);
}
} else {
/* If any filters failed to insert, rollback and fall back to
diff --git a/drivers/net/ethernet/sfc/efx.h b/drivers/net/ethernet/sfc/efx.h
index ee146624..a0c52e3 100644
--- a/drivers/net/ethernet/sfc/efx.h
+++ b/drivers/net/ethernet/sfc/efx.h
@@ -74,7 +74,10 @@
#define EFX_RXQ_MIN_ENT 128U
#define EFX_TXQ_MIN_ENT(efx) (2 * efx_tx_max_skb_descs(efx))
-#define EFX_TXQ_MAX_ENT(efx) (EFX_WORKAROUND_35388(efx) ? \
+/* All EF10 architecture NICs steal one bit of the DMAQ size for various
+ * other purposes when counting TxQ entries, so we halve the queue size.
+ */
+#define EFX_TXQ_MAX_ENT(efx) (EFX_WORKAROUND_EF10(efx) ? \
EFX_MAX_DMAQ_SIZE / 2 : EFX_MAX_DMAQ_SIZE)
static inline bool efx_rss_enabled(struct efx_nic *efx)
diff --git a/drivers/net/ethernet/sfc/falcon/tx.c b/drivers/net/ethernet/sfc/falcon/tx.c
index 104fb15..f6daf09 100644
--- a/drivers/net/ethernet/sfc/falcon/tx.c
+++ b/drivers/net/ethernet/sfc/falcon/tx.c
@@ -437,11 +437,13 @@
if (ntc->type != TC_SETUP_MQPRIO)
return -EINVAL;
- num_tc = ntc->tc;
+ num_tc = ntc->mqprio->num_tc;
if (ef4_nic_rev(efx) < EF4_REV_FALCON_B0 || num_tc > EF4_MAX_TX_TC)
return -EINVAL;
+ ntc->mqprio->hw = TC_MQPRIO_HW_OFFLOAD_TCS;
+
if (num_tc == net_dev->num_tc)
return 0;
diff --git a/drivers/net/ethernet/sfc/tx.c b/drivers/net/ethernet/sfc/tx.c
index ff88d60..3bdf87f 100644
--- a/drivers/net/ethernet/sfc/tx.c
+++ b/drivers/net/ethernet/sfc/tx.c
@@ -665,11 +665,13 @@
if (ntc->type != TC_SETUP_MQPRIO)
return -EINVAL;
- num_tc = ntc->tc;
+ num_tc = ntc->mqprio->num_tc;
if (num_tc > EFX_MAX_TX_TC)
return -EINVAL;
+ ntc->mqprio->hw = TC_MQPRIO_HW_OFFLOAD_TCS;
+
if (num_tc == net_dev->num_tc)
return 0;
diff --git a/drivers/net/ethernet/sfc/workarounds.h b/drivers/net/ethernet/sfc/workarounds.h
index 103f827..c67fa18 100644
--- a/drivers/net/ethernet/sfc/workarounds.h
+++ b/drivers/net/ethernet/sfc/workarounds.h
@@ -16,6 +16,7 @@
*/
#define EFX_WORKAROUND_SIENA(efx) (efx_nic_rev(efx) == EFX_REV_SIENA_A0)
+#define EFX_WORKAROUND_EF10(efx) (efx_nic_rev(efx) >= EFX_REV_HUNT_A0)
#define EFX_WORKAROUND_10G(efx) 1
/* Bit-bashed I2C reads cause performance drop */
diff --git a/drivers/net/ethernet/sgi/ioc3-eth.c b/drivers/net/ethernet/sgi/ioc3-eth.c
index 57e6cef..52ead55 100644
--- a/drivers/net/ethernet/sgi/ioc3-eth.c
+++ b/drivers/net/ethernet/sgi/ioc3-eth.c
@@ -1558,25 +1558,27 @@
strlcpy(info->bus_info, pci_name(ip->pdev), sizeof(info->bus_info));
}
-static int ioc3_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int ioc3_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct ioc3_private *ip = netdev_priv(dev);
int rc;
spin_lock_irq(&ip->ioc3_lock);
- rc = mii_ethtool_gset(&ip->mii, cmd);
+ rc = mii_ethtool_get_link_ksettings(&ip->mii, cmd);
spin_unlock_irq(&ip->ioc3_lock);
return rc;
}
-static int ioc3_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int ioc3_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct ioc3_private *ip = netdev_priv(dev);
int rc;
spin_lock_irq(&ip->ioc3_lock);
- rc = mii_ethtool_sset(&ip->mii, cmd);
+ rc = mii_ethtool_set_link_ksettings(&ip->mii, cmd);
spin_unlock_irq(&ip->ioc3_lock);
return rc;
@@ -1608,10 +1610,10 @@
static const struct ethtool_ops ioc3_ethtool_ops = {
.get_drvinfo = ioc3_get_drvinfo,
- .get_settings = ioc3_get_settings,
- .set_settings = ioc3_set_settings,
.nway_reset = ioc3_nway_reset,
.get_link = ioc3_get_link,
+ .get_link_ksettings = ioc3_get_link_ksettings,
+ .set_link_ksettings = ioc3_set_link_ksettings,
};
static int ioc3_ioctl(struct net_device *dev, struct ifreq *rq, int cmd)
diff --git a/drivers/net/ethernet/silan/sc92031.c b/drivers/net/ethernet/silan/sc92031.c
index 6c2e2b3..751c818 100644
--- a/drivers/net/ethernet/silan/sc92031.c
+++ b/drivers/net/ethernet/silan/sc92031.c
@@ -1122,14 +1122,16 @@
}
#endif
-static int sc92031_ethtool_get_settings(struct net_device *dev,
- struct ethtool_cmd *cmd)
+static int
+sc92031_ethtool_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct sc92031_priv *priv = netdev_priv(dev);
void __iomem *port_base = priv->port_base;
u8 phy_address;
u32 phy_ctrl;
u16 output_status;
+ u32 supported, advertising;
spin_lock_bh(&priv->lock);
@@ -1142,68 +1144,77 @@
spin_unlock_bh(&priv->lock);
- cmd->supported = SUPPORTED_10baseT_Half | SUPPORTED_10baseT_Full
+ supported = SUPPORTED_10baseT_Half | SUPPORTED_10baseT_Full
| SUPPORTED_100baseT_Half | SUPPORTED_100baseT_Full
| SUPPORTED_Autoneg | SUPPORTED_TP | SUPPORTED_MII;
- cmd->advertising = ADVERTISED_TP | ADVERTISED_MII;
+ advertising = ADVERTISED_TP | ADVERTISED_MII;
if ((phy_ctrl & (PhyCtrlDux | PhyCtrlSpd100 | PhyCtrlSpd10))
== (PhyCtrlDux | PhyCtrlSpd100 | PhyCtrlSpd10))
- cmd->advertising |= ADVERTISED_Autoneg;
+ advertising |= ADVERTISED_Autoneg;
if ((phy_ctrl & PhyCtrlSpd10) == PhyCtrlSpd10)
- cmd->advertising |= ADVERTISED_10baseT_Half;
+ advertising |= ADVERTISED_10baseT_Half;
if ((phy_ctrl & (PhyCtrlSpd10 | PhyCtrlDux))
== (PhyCtrlSpd10 | PhyCtrlDux))
- cmd->advertising |= ADVERTISED_10baseT_Full;
+ advertising |= ADVERTISED_10baseT_Full;
if ((phy_ctrl & PhyCtrlSpd100) == PhyCtrlSpd100)
- cmd->advertising |= ADVERTISED_100baseT_Half;
+ advertising |= ADVERTISED_100baseT_Half;
if ((phy_ctrl & (PhyCtrlSpd100 | PhyCtrlDux))
== (PhyCtrlSpd100 | PhyCtrlDux))
- cmd->advertising |= ADVERTISED_100baseT_Full;
+ advertising |= ADVERTISED_100baseT_Full;
if (phy_ctrl & PhyCtrlAne)
- cmd->advertising |= ADVERTISED_Autoneg;
+ advertising |= ADVERTISED_Autoneg;
- ethtool_cmd_speed_set(cmd,
- (output_status & 0x2) ? SPEED_100 : SPEED_10);
- cmd->duplex = (output_status & 0x4) ? DUPLEX_FULL : DUPLEX_HALF;
- cmd->port = PORT_MII;
- cmd->phy_address = phy_address;
- cmd->transceiver = XCVR_INTERNAL;
- cmd->autoneg = (phy_ctrl & PhyCtrlAne) ? AUTONEG_ENABLE : AUTONEG_DISABLE;
+ cmd->base.speed = (output_status & 0x2) ? SPEED_100 : SPEED_10;
+ cmd->base.duplex = (output_status & 0x4) ? DUPLEX_FULL : DUPLEX_HALF;
+ cmd->base.port = PORT_MII;
+ cmd->base.phy_address = phy_address;
+ cmd->base.autoneg = (phy_ctrl & PhyCtrlAne) ?
+ AUTONEG_ENABLE : AUTONEG_DISABLE;
+
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
+ advertising);
return 0;
}
-static int sc92031_ethtool_set_settings(struct net_device *dev,
- struct ethtool_cmd *cmd)
+static int
+sc92031_ethtool_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct sc92031_priv *priv = netdev_priv(dev);
void __iomem *port_base = priv->port_base;
- u32 speed = ethtool_cmd_speed(cmd);
+ u32 speed = cmd->base.speed;
u32 phy_ctrl;
u32 old_phy_ctrl;
+ u32 advertising;
+
+ ethtool_convert_link_mode_to_legacy_u32(&advertising,
+ cmd->link_modes.advertising);
if (!(speed == SPEED_10 || speed == SPEED_100))
return -EINVAL;
- if (!(cmd->duplex == DUPLEX_HALF || cmd->duplex == DUPLEX_FULL))
+ if (!(cmd->base.duplex == DUPLEX_HALF ||
+ cmd->base.duplex == DUPLEX_FULL))
return -EINVAL;
- if (!(cmd->port == PORT_MII))
+ if (!(cmd->base.port == PORT_MII))
return -EINVAL;
- if (!(cmd->phy_address == 0x1f))
+ if (!(cmd->base.phy_address == 0x1f))
return -EINVAL;
- if (!(cmd->transceiver == XCVR_INTERNAL))
- return -EINVAL;
- if (!(cmd->autoneg == AUTONEG_DISABLE || cmd->autoneg == AUTONEG_ENABLE))
+ if (!(cmd->base.autoneg == AUTONEG_DISABLE ||
+ cmd->base.autoneg == AUTONEG_ENABLE))
return -EINVAL;
- if (cmd->autoneg == AUTONEG_ENABLE) {
- if (!(cmd->advertising & (ADVERTISED_Autoneg
+ if (cmd->base.autoneg == AUTONEG_ENABLE) {
+ if (!(advertising & (ADVERTISED_Autoneg
| ADVERTISED_100baseT_Full
| ADVERTISED_100baseT_Half
| ADVERTISED_10baseT_Full
@@ -1213,15 +1224,15 @@
phy_ctrl = PhyCtrlAne;
// FIXME: I'm not sure what the original code was trying to do
- if (cmd->advertising & ADVERTISED_Autoneg)
+ if (advertising & ADVERTISED_Autoneg)
phy_ctrl |= PhyCtrlDux | PhyCtrlSpd100 | PhyCtrlSpd10;
- if (cmd->advertising & ADVERTISED_100baseT_Full)
+ if (advertising & ADVERTISED_100baseT_Full)
phy_ctrl |= PhyCtrlDux | PhyCtrlSpd100;
- if (cmd->advertising & ADVERTISED_100baseT_Half)
+ if (advertising & ADVERTISED_100baseT_Half)
phy_ctrl |= PhyCtrlSpd100;
- if (cmd->advertising & ADVERTISED_10baseT_Full)
+ if (advertising & ADVERTISED_10baseT_Full)
phy_ctrl |= PhyCtrlSpd10 | PhyCtrlDux;
- if (cmd->advertising & ADVERTISED_10baseT_Half)
+ if (advertising & ADVERTISED_10baseT_Half)
phy_ctrl |= PhyCtrlSpd10;
} else {
// FIXME: Whole branch guessed
@@ -1232,7 +1243,7 @@
else /* cmd->speed == SPEED_100 */
phy_ctrl |= PhyCtrlSpd100;
- if (cmd->duplex == DUPLEX_FULL)
+ if (cmd->base.duplex == DUPLEX_FULL)
phy_ctrl |= PhyCtrlDux;
}
@@ -1368,8 +1379,6 @@
}
static const struct ethtool_ops sc92031_ethtool_ops = {
- .get_settings = sc92031_ethtool_get_settings,
- .set_settings = sc92031_ethtool_set_settings,
.get_wol = sc92031_ethtool_get_wol,
.set_wol = sc92031_ethtool_set_wol,
.nway_reset = sc92031_ethtool_nway_reset,
@@ -1377,6 +1386,8 @@
.get_strings = sc92031_ethtool_get_strings,
.get_sset_count = sc92031_ethtool_get_sset_count,
.get_ethtool_stats = sc92031_ethtool_get_ethtool_stats,
+ .get_link_ksettings = sc92031_ethtool_get_link_ksettings,
+ .set_link_ksettings = sc92031_ethtool_set_link_ksettings,
};
diff --git a/drivers/net/ethernet/sis/sis190.c b/drivers/net/ethernet/sis/sis190.c
index 210e35d..02da106 100644
--- a/drivers/net/ethernet/sis/sis190.c
+++ b/drivers/net/ethernet/sis/sis190.c
@@ -1734,18 +1734,20 @@
BMCR_ANENABLE | BMCR_ANRESTART | BMCR_RESET);
}
-static int sis190_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int sis190_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct sis190_private *tp = netdev_priv(dev);
- return mii_ethtool_gset(&tp->mii_if, cmd);
+ return mii_ethtool_get_link_ksettings(&tp->mii_if, cmd);
}
-static int sis190_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int sis190_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct sis190_private *tp = netdev_priv(dev);
- return mii_ethtool_sset(&tp->mii_if, cmd);
+ return mii_ethtool_set_link_ksettings(&tp->mii_if, cmd);
}
static void sis190_get_drvinfo(struct net_device *dev,
@@ -1797,8 +1799,6 @@
}
static const struct ethtool_ops sis190_ethtool_ops = {
- .get_settings = sis190_get_settings,
- .set_settings = sis190_set_settings,
.get_drvinfo = sis190_get_drvinfo,
.get_regs_len = sis190_get_regs_len,
.get_regs = sis190_get_regs,
@@ -1806,6 +1806,8 @@
.get_msglevel = sis190_get_msglevel,
.set_msglevel = sis190_set_msglevel,
.nway_reset = sis190_nway_reset,
+ .get_link_ksettings = sis190_get_link_ksettings,
+ .set_link_ksettings = sis190_set_link_ksettings,
};
static int sis190_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
diff --git a/drivers/net/ethernet/sis/sis900.c b/drivers/net/ethernet/sis/sis900.c
index 1b6f617..40bd883 100644
--- a/drivers/net/ethernet/sis/sis900.c
+++ b/drivers/net/ethernet/sis/sis900.c
@@ -2035,23 +2035,23 @@
return mii_link_ok(&sis_priv->mii_info);
}
-static int sis900_get_settings(struct net_device *net_dev,
- struct ethtool_cmd *cmd)
+static int sis900_get_link_ksettings(struct net_device *net_dev,
+ struct ethtool_link_ksettings *cmd)
{
struct sis900_private *sis_priv = netdev_priv(net_dev);
spin_lock_irq(&sis_priv->lock);
- mii_ethtool_gset(&sis_priv->mii_info, cmd);
+ mii_ethtool_get_link_ksettings(&sis_priv->mii_info, cmd);
spin_unlock_irq(&sis_priv->lock);
return 0;
}
-static int sis900_set_settings(struct net_device *net_dev,
- struct ethtool_cmd *cmd)
+static int sis900_set_link_ksettings(struct net_device *net_dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct sis900_private *sis_priv = netdev_priv(net_dev);
int rt;
spin_lock_irq(&sis_priv->lock);
- rt = mii_ethtool_sset(&sis_priv->mii_info, cmd);
+ rt = mii_ethtool_set_link_ksettings(&sis_priv->mii_info, cmd);
spin_unlock_irq(&sis_priv->lock);
return rt;
}
@@ -2129,11 +2129,11 @@
.get_msglevel = sis900_get_msglevel,
.set_msglevel = sis900_set_msglevel,
.get_link = sis900_get_link,
- .get_settings = sis900_get_settings,
- .set_settings = sis900_set_settings,
.nway_reset = sis900_nway_reset,
.get_wol = sis900_get_wol,
- .set_wol = sis900_set_wol
+ .set_wol = sis900_set_wol,
+ .get_link_ksettings = sis900_get_link_ksettings,
+ .set_link_ksettings = sis900_set_link_ksettings,
};
/**
diff --git a/drivers/net/ethernet/smsc/epic100.c b/drivers/net/ethernet/smsc/epic100.c
index 5f27371..db6dcb0 100644
--- a/drivers/net/ethernet/smsc/epic100.c
+++ b/drivers/net/ethernet/smsc/epic100.c
@@ -1387,25 +1387,27 @@
strlcpy(info->bus_info, pci_name(np->pci_dev), sizeof(info->bus_info));
}
-static int netdev_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int netdev_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct epic_private *np = netdev_priv(dev);
int rc;
spin_lock_irq(&np->lock);
- rc = mii_ethtool_gset(&np->mii, cmd);
+ rc = mii_ethtool_get_link_ksettings(&np->mii, cmd);
spin_unlock_irq(&np->lock);
return rc;
}
-static int netdev_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int netdev_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct epic_private *np = netdev_priv(dev);
int rc;
spin_lock_irq(&np->lock);
- rc = mii_ethtool_sset(&np->mii, cmd);
+ rc = mii_ethtool_set_link_ksettings(&np->mii, cmd);
spin_unlock_irq(&np->lock);
return rc;
@@ -1460,14 +1462,14 @@
static const struct ethtool_ops netdev_ethtool_ops = {
.get_drvinfo = netdev_get_drvinfo,
- .get_settings = netdev_get_settings,
- .set_settings = netdev_set_settings,
.nway_reset = netdev_nway_reset,
.get_link = netdev_get_link,
.get_msglevel = netdev_get_msglevel,
.set_msglevel = netdev_set_msglevel,
.begin = ethtool_begin,
- .complete = ethtool_complete
+ .complete = ethtool_complete,
+ .get_link_ksettings = netdev_get_link_ksettings,
+ .set_link_ksettings = netdev_set_link_ksettings,
};
static int netdev_ioctl(struct net_device *dev, struct ifreq *rq, int cmd)
diff --git a/drivers/net/ethernet/smsc/smc911x.c b/drivers/net/ethernet/smsc/smc911x.c
index 4f19c61..36307d3 100644
--- a/drivers/net/ethernet/smsc/smc911x.c
+++ b/drivers/net/ethernet/smsc/smc911x.c
@@ -1446,40 +1446,40 @@
* Ethtool support
*/
static int
-smc911x_ethtool_getsettings(struct net_device *dev, struct ethtool_cmd *cmd)
+smc911x_ethtool_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct smc911x_local *lp = netdev_priv(dev);
int ret, status;
unsigned long flags;
+ u32 supported;
DBG(SMC_DEBUG_FUNC, dev, "--> %s\n", __func__);
- cmd->maxtxpkt = 1;
- cmd->maxrxpkt = 1;
if (lp->phy_type != 0) {
spin_lock_irqsave(&lp->lock, flags);
- ret = mii_ethtool_gset(&lp->mii, cmd);
+ ret = mii_ethtool_get_link_ksettings(&lp->mii, cmd);
spin_unlock_irqrestore(&lp->lock, flags);
} else {
- cmd->supported = SUPPORTED_10baseT_Half |
+ supported = SUPPORTED_10baseT_Half |
SUPPORTED_10baseT_Full |
SUPPORTED_TP | SUPPORTED_AUI;
if (lp->ctl_rspeed == 10)
- ethtool_cmd_speed_set(cmd, SPEED_10);
+ cmd->base.speed = SPEED_10;
else if (lp->ctl_rspeed == 100)
- ethtool_cmd_speed_set(cmd, SPEED_100);
+ cmd->base.speed = SPEED_100;
- cmd->autoneg = AUTONEG_DISABLE;
- if (lp->mii.phy_id==1)
- cmd->transceiver = XCVR_INTERNAL;
- else
- cmd->transceiver = XCVR_EXTERNAL;
- cmd->port = 0;
+ cmd->base.autoneg = AUTONEG_DISABLE;
+ cmd->base.port = 0;
SMC_GET_PHY_SPECIAL(lp, lp->mii.phy_id, status);
- cmd->duplex =
+ cmd->base.duplex =
(status & (PHY_SPECIAL_SPD_10FULL_ | PHY_SPECIAL_SPD_100FULL_)) ?
DUPLEX_FULL : DUPLEX_HALF;
+
+ ethtool_convert_legacy_u32_to_link_mode(
+ cmd->link_modes.supported, supported);
+
ret = 0;
}
@@ -1487,7 +1487,8 @@
}
static int
-smc911x_ethtool_setsettings(struct net_device *dev, struct ethtool_cmd *cmd)
+smc911x_ethtool_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct smc911x_local *lp = netdev_priv(dev);
int ret;
@@ -1495,16 +1496,18 @@
if (lp->phy_type != 0) {
spin_lock_irqsave(&lp->lock, flags);
- ret = mii_ethtool_sset(&lp->mii, cmd);
+ ret = mii_ethtool_set_link_ksettings(&lp->mii, cmd);
spin_unlock_irqrestore(&lp->lock, flags);
} else {
- if (cmd->autoneg != AUTONEG_DISABLE ||
- cmd->speed != SPEED_10 ||
- (cmd->duplex != DUPLEX_HALF && cmd->duplex != DUPLEX_FULL) ||
- (cmd->port != PORT_TP && cmd->port != PORT_AUI))
+ if (cmd->base.autoneg != AUTONEG_DISABLE ||
+ cmd->base.speed != SPEED_10 ||
+ (cmd->base.duplex != DUPLEX_HALF &&
+ cmd->base.duplex != DUPLEX_FULL) ||
+ (cmd->base.port != PORT_TP &&
+ cmd->base.port != PORT_AUI))
return -EINVAL;
- lp->ctl_rfduplx = cmd->duplex == DUPLEX_FULL;
+ lp->ctl_rfduplx = cmd->base.duplex == DUPLEX_FULL;
ret = 0;
}
@@ -1686,8 +1689,6 @@
}
static const struct ethtool_ops smc911x_ethtool_ops = {
- .get_settings = smc911x_ethtool_getsettings,
- .set_settings = smc911x_ethtool_setsettings,
.get_drvinfo = smc911x_ethtool_getdrvinfo,
.get_msglevel = smc911x_ethtool_getmsglevel,
.set_msglevel = smc911x_ethtool_setmsglevel,
@@ -1698,6 +1699,8 @@
.get_eeprom_len = smc911x_ethtool_geteeprom_len,
.get_eeprom = smc911x_ethtool_geteeprom,
.set_eeprom = smc911x_ethtool_seteeprom,
+ .get_link_ksettings = smc911x_ethtool_get_link_ksettings,
+ .set_link_ksettings = smc911x_ethtool_set_link_ksettings,
};
/*
diff --git a/drivers/net/ethernet/smsc/smc91c92_cs.c b/drivers/net/ethernet/smsc/smc91c92_cs.c
index 97280da..976aa87 100644
--- a/drivers/net/ethernet/smsc/smc91c92_cs.c
+++ b/drivers/net/ethernet/smsc/smc91c92_cs.c
@@ -1843,56 +1843,60 @@
}
}
-static int smc_netdev_get_ecmd(struct net_device *dev, struct ethtool_cmd *ecmd)
+static int smc_netdev_get_ecmd(struct net_device *dev,
+ struct ethtool_link_ksettings *ecmd)
{
- u16 tmp;
- unsigned int ioaddr = dev->base_addr;
+ u16 tmp;
+ unsigned int ioaddr = dev->base_addr;
+ u32 supported;
- ecmd->supported = (SUPPORTED_TP | SUPPORTED_AUI |
- SUPPORTED_10baseT_Half | SUPPORTED_10baseT_Full);
-
- SMC_SELECT_BANK(1);
- tmp = inw(ioaddr + CONFIG);
- ecmd->port = (tmp & CFG_AUI_SELECT) ? PORT_AUI : PORT_TP;
- ecmd->transceiver = XCVR_INTERNAL;
- ethtool_cmd_speed_set(ecmd, SPEED_10);
- ecmd->phy_address = ioaddr + MGMT;
+ supported = (SUPPORTED_TP | SUPPORTED_AUI |
+ SUPPORTED_10baseT_Half | SUPPORTED_10baseT_Full);
- SMC_SELECT_BANK(0);
- tmp = inw(ioaddr + TCR);
- ecmd->duplex = (tmp & TCR_FDUPLX) ? DUPLEX_FULL : DUPLEX_HALF;
+ SMC_SELECT_BANK(1);
+ tmp = inw(ioaddr + CONFIG);
+ ecmd->base.port = (tmp & CFG_AUI_SELECT) ? PORT_AUI : PORT_TP;
+ ecmd->base.speed = SPEED_10;
+ ecmd->base.phy_address = ioaddr + MGMT;
- return 0;
+ SMC_SELECT_BANK(0);
+ tmp = inw(ioaddr + TCR);
+ ecmd->base.duplex = (tmp & TCR_FDUPLX) ? DUPLEX_FULL : DUPLEX_HALF;
+
+ ethtool_convert_legacy_u32_to_link_mode(ecmd->link_modes.supported,
+ supported);
+
+ return 0;
}
-static int smc_netdev_set_ecmd(struct net_device *dev, struct ethtool_cmd *ecmd)
+static int smc_netdev_set_ecmd(struct net_device *dev,
+ const struct ethtool_link_ksettings *ecmd)
{
- u16 tmp;
- unsigned int ioaddr = dev->base_addr;
+ u16 tmp;
+ unsigned int ioaddr = dev->base_addr;
- if (ethtool_cmd_speed(ecmd) != SPEED_10)
- return -EINVAL;
- if (ecmd->duplex != DUPLEX_HALF && ecmd->duplex != DUPLEX_FULL)
- return -EINVAL;
- if (ecmd->port != PORT_TP && ecmd->port != PORT_AUI)
- return -EINVAL;
- if (ecmd->transceiver != XCVR_INTERNAL)
- return -EINVAL;
+ if (ecmd->base.speed != SPEED_10)
+ return -EINVAL;
+ if (ecmd->base.duplex != DUPLEX_HALF &&
+ ecmd->base.duplex != DUPLEX_FULL)
+ return -EINVAL;
+ if (ecmd->base.port != PORT_TP && ecmd->base.port != PORT_AUI)
+ return -EINVAL;
- if (ecmd->port == PORT_AUI)
- smc_set_xcvr(dev, 1);
- else
- smc_set_xcvr(dev, 0);
+ if (ecmd->base.port == PORT_AUI)
+ smc_set_xcvr(dev, 1);
+ else
+ smc_set_xcvr(dev, 0);
- SMC_SELECT_BANK(0);
- tmp = inw(ioaddr + TCR);
- if (ecmd->duplex == DUPLEX_FULL)
- tmp |= TCR_FDUPLX;
- else
- tmp &= ~TCR_FDUPLX;
- outw(tmp, ioaddr + TCR);
-
- return 0;
+ SMC_SELECT_BANK(0);
+ tmp = inw(ioaddr + TCR);
+ if (ecmd->base.duplex == DUPLEX_FULL)
+ tmp |= TCR_FDUPLX;
+ else
+ tmp &= ~TCR_FDUPLX;
+ outw(tmp, ioaddr + TCR);
+
+ return 0;
}
static int check_if_running(struct net_device *dev)
@@ -1908,7 +1912,8 @@
strlcpy(info->version, DRV_VERSION, sizeof(info->version));
}
-static int smc_get_settings(struct net_device *dev, struct ethtool_cmd *ecmd)
+static int smc_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *ecmd)
{
struct smc_private *smc = netdev_priv(dev);
unsigned int ioaddr = dev->base_addr;
@@ -1919,7 +1924,7 @@
spin_lock_irqsave(&smc->lock, flags);
SMC_SELECT_BANK(3);
if (smc->cfg & CFG_MII_SELECT)
- ret = mii_ethtool_gset(&smc->mii_if, ecmd);
+ ret = mii_ethtool_get_link_ksettings(&smc->mii_if, ecmd);
else
ret = smc_netdev_get_ecmd(dev, ecmd);
SMC_SELECT_BANK(saved_bank);
@@ -1927,7 +1932,8 @@
return ret;
}
-static int smc_set_settings(struct net_device *dev, struct ethtool_cmd *ecmd)
+static int smc_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *ecmd)
{
struct smc_private *smc = netdev_priv(dev);
unsigned int ioaddr = dev->base_addr;
@@ -1938,7 +1944,7 @@
spin_lock_irqsave(&smc->lock, flags);
SMC_SELECT_BANK(3);
if (smc->cfg & CFG_MII_SELECT)
- ret = mii_ethtool_sset(&smc->mii_if, ecmd);
+ ret = mii_ethtool_set_link_ksettings(&smc->mii_if, ecmd);
else
ret = smc_netdev_set_ecmd(dev, ecmd);
SMC_SELECT_BANK(saved_bank);
@@ -1982,10 +1988,10 @@
static const struct ethtool_ops ethtool_ops = {
.begin = check_if_running,
.get_drvinfo = smc_get_drvinfo,
- .get_settings = smc_get_settings,
- .set_settings = smc_set_settings,
.get_link = smc_get_link,
.nway_reset = smc_nway_reset,
+ .get_link_ksettings = smc_get_link_ksettings,
+ .set_link_ksettings = smc_set_link_ksettings,
};
static int smc_ioctl (struct net_device *dev, struct ifreq *rq, int cmd)
diff --git a/drivers/net/ethernet/smsc/smsc911x.c b/drivers/net/ethernet/smsc/smsc911x.c
index fa5ca09..ea1bbc3 100644
--- a/drivers/net/ethernet/smsc/smsc911x.c
+++ b/drivers/net/ethernet/smsc/smsc911x.c
@@ -25,7 +25,7 @@
* LAN9215, LAN9216, LAN9217, LAN9218
* LAN9210, LAN9211
* LAN9220, LAN9221
- * LAN89218
+ * LAN89218,LAN9250
*
*/
@@ -1450,6 +1450,8 @@
unsigned int timeout;
unsigned int temp;
int ret;
+ unsigned int reset_offset = HW_CFG;
+ unsigned int reset_mask = HW_CFG_SRST_;
/*
* Make sure to power-up the PHY chip before doing a reset, otherwise
@@ -1476,15 +1478,23 @@
}
}
+ if ((pdata->idrev & 0xFFFF0000) == LAN9250) {
+ /* special reset for LAN9250 */
+ reset_offset = RESET_CTL;
+ reset_mask = RESET_CTL_DIGITAL_RST_;
+ }
+
/* Reset the LAN911x */
- smsc911x_reg_write(pdata, HW_CFG, HW_CFG_SRST_);
+ smsc911x_reg_write(pdata, reset_offset, reset_mask);
+
+ /* verify reset bit is cleared */
timeout = 10;
do {
udelay(10);
- temp = smsc911x_reg_read(pdata, HW_CFG);
- } while ((--timeout) && (temp & HW_CFG_SRST_));
+ temp = smsc911x_reg_read(pdata, reset_offset);
+ } while ((--timeout) && (temp & reset_mask));
- if (unlikely(temp & HW_CFG_SRST_)) {
+ if (unlikely(temp & reset_mask)) {
SMSC_WARN(pdata, drv, "Failed to complete reset");
return -EIO;
}
@@ -2253,28 +2263,29 @@
pdata->idrev = smsc911x_reg_read(pdata, ID_REV);
switch (pdata->idrev & 0xFFFF0000) {
- case 0x01180000:
- case 0x01170000:
- case 0x01160000:
- case 0x01150000:
- case 0x218A0000:
+ case LAN9118:
+ case LAN9117:
+ case LAN9116:
+ case LAN9115:
+ case LAN89218:
/* LAN911[5678] family */
pdata->generation = pdata->idrev & 0x0000FFFF;
break;
- case 0x118A0000:
- case 0x117A0000:
- case 0x116A0000:
- case 0x115A0000:
+ case LAN9218:
+ case LAN9217:
+ case LAN9216:
+ case LAN9215:
/* LAN921[5678] family */
pdata->generation = 3;
break;
- case 0x92100000:
- case 0x92110000:
- case 0x92200000:
- case 0x92210000:
- /* LAN9210/LAN9211/LAN9220/LAN9221 */
+ case LAN9210:
+ case LAN9211:
+ case LAN9220:
+ case LAN9221:
+ case LAN9250:
+ /* LAN9210/LAN9211/LAN9220/LAN9221/LAN9250 */
pdata->generation = 4;
break;
diff --git a/drivers/net/ethernet/smsc/smsc911x.h b/drivers/net/ethernet/smsc/smsc911x.h
index 54d6489..8d75508 100644
--- a/drivers/net/ethernet/smsc/smsc911x.h
+++ b/drivers/net/ethernet/smsc/smsc911x.h
@@ -20,6 +20,22 @@
#ifndef __SMSC911X_H__
#define __SMSC911X_H__
+/*Chip ID*/
+#define LAN9115 0x01150000
+#define LAN9116 0x01160000
+#define LAN9117 0x01170000
+#define LAN9118 0x01180000
+#define LAN9215 0x115A0000
+#define LAN9216 0x116A0000
+#define LAN9217 0x117A0000
+#define LAN9218 0x118A0000
+#define LAN9210 0x92100000
+#define LAN9211 0x92110000
+#define LAN9220 0x92200000
+#define LAN9221 0x92210000
+#define LAN9250 0x92500000
+#define LAN89218 0x218A0000
+
#define TX_FIFO_LOW_THRESHOLD ((u32)1600)
#define SMSC911X_EEPROM_SIZE ((u32)128)
#define USE_DEBUG 0
@@ -303,6 +319,9 @@
#define E2P_DATA_EEPROM_DATA_ 0x000000FF
#define LAN_REGISTER_EXTENT 0x00000100
+#define RESET_CTL 0x1F8
+#define RESET_CTL_DIGITAL_RST_ 0x00000001
+
/*
* MAC Control and Status Register (Indirect Address)
* Offset (through the MAC_CSR CMD and DATA port)
diff --git a/drivers/net/ethernet/stmicro/stmmac/chain_mode.c b/drivers/net/ethernet/stmicro/stmmac/chain_mode.c
index 01a8c02..e93c40b 100644
--- a/drivers/net/ethernet/stmicro/stmmac/chain_mode.c
+++ b/drivers/net/ethernet/stmicro/stmmac/chain_mode.c
@@ -26,12 +26,15 @@
static int stmmac_jumbo_frm(void *p, struct sk_buff *skb, int csum)
{
- struct stmmac_priv *priv = (struct stmmac_priv *)p;
- unsigned int entry = priv->cur_tx;
- struct dma_desc *desc = priv->dma_tx + entry;
+ struct stmmac_tx_queue *tx_q = (struct stmmac_tx_queue *)p;
unsigned int nopaged_len = skb_headlen(skb);
+ struct stmmac_priv *priv = tx_q->priv_data;
+ unsigned int entry = tx_q->cur_tx;
unsigned int bmax, des2;
unsigned int i = 1, len;
+ struct dma_desc *desc;
+
+ desc = tx_q->dma_tx + entry;
if (priv->plat->enh_desc)
bmax = BUF_SIZE_8KiB;
@@ -45,16 +48,16 @@
desc->des2 = cpu_to_le32(des2);
if (dma_mapping_error(priv->device, des2))
return -1;
- priv->tx_skbuff_dma[entry].buf = des2;
- priv->tx_skbuff_dma[entry].len = bmax;
+ tx_q->tx_skbuff_dma[entry].buf = des2;
+ tx_q->tx_skbuff_dma[entry].len = bmax;
/* do not close the descriptor and do not set own bit */
priv->hw->desc->prepare_tx_desc(desc, 1, bmax, csum, STMMAC_CHAIN_MODE,
- 0, false);
+ 0, false, skb->len);
while (len != 0) {
- priv->tx_skbuff[entry] = NULL;
+ tx_q->tx_skbuff[entry] = NULL;
entry = STMMAC_GET_ENTRY(entry, DMA_TX_SIZE);
- desc = priv->dma_tx + entry;
+ desc = tx_q->dma_tx + entry;
if (len > bmax) {
des2 = dma_map_single(priv->device,
@@ -63,11 +66,11 @@
desc->des2 = cpu_to_le32(des2);
if (dma_mapping_error(priv->device, des2))
return -1;
- priv->tx_skbuff_dma[entry].buf = des2;
- priv->tx_skbuff_dma[entry].len = bmax;
+ tx_q->tx_skbuff_dma[entry].buf = des2;
+ tx_q->tx_skbuff_dma[entry].len = bmax;
priv->hw->desc->prepare_tx_desc(desc, 0, bmax, csum,
STMMAC_CHAIN_MODE, 1,
- false);
+ false, skb->len);
len -= bmax;
i++;
} else {
@@ -77,17 +80,17 @@
desc->des2 = cpu_to_le32(des2);
if (dma_mapping_error(priv->device, des2))
return -1;
- priv->tx_skbuff_dma[entry].buf = des2;
- priv->tx_skbuff_dma[entry].len = len;
+ tx_q->tx_skbuff_dma[entry].buf = des2;
+ tx_q->tx_skbuff_dma[entry].len = len;
/* last descriptor can be set now */
priv->hw->desc->prepare_tx_desc(desc, 0, len, csum,
STMMAC_CHAIN_MODE, 1,
- true);
+ true, skb->len);
len = 0;
}
}
- priv->cur_tx = entry;
+ tx_q->cur_tx = entry;
return entry;
}
@@ -136,32 +139,34 @@
static void stmmac_refill_desc3(void *priv_ptr, struct dma_desc *p)
{
- struct stmmac_priv *priv = (struct stmmac_priv *)priv_ptr;
+ struct stmmac_rx_queue *rx_q = (struct stmmac_rx_queue *)priv_ptr;
+ struct stmmac_priv *priv = rx_q->priv_data;
if (priv->hwts_rx_en && !priv->extend_desc)
/* NOTE: Device will overwrite des3 with timestamp value if
* 1588-2002 time stamping is enabled, hence reinitialize it
* to keep explicit chaining in the descriptor.
*/
- p->des3 = cpu_to_le32((unsigned int)(priv->dma_rx_phy +
- (((priv->dirty_rx) + 1) %
+ p->des3 = cpu_to_le32((unsigned int)(rx_q->dma_rx_phy +
+ (((rx_q->dirty_rx) + 1) %
DMA_RX_SIZE) *
sizeof(struct dma_desc)));
}
static void stmmac_clean_desc3(void *priv_ptr, struct dma_desc *p)
{
- struct stmmac_priv *priv = (struct stmmac_priv *)priv_ptr;
- unsigned int entry = priv->dirty_tx;
+ struct stmmac_tx_queue *tx_q = (struct stmmac_tx_queue *)priv_ptr;
+ struct stmmac_priv *priv = tx_q->priv_data;
+ unsigned int entry = tx_q->dirty_tx;
- if (priv->tx_skbuff_dma[entry].last_segment && !priv->extend_desc &&
+ if (tx_q->tx_skbuff_dma[entry].last_segment && !priv->extend_desc &&
priv->hwts_tx_en)
/* NOTE: Device will overwrite des3 with timestamp value if
* 1588-2002 time stamping is enabled, hence reinitialize it
* to keep explicit chaining in the descriptor.
*/
- p->des3 = cpu_to_le32((unsigned int)((priv->dma_tx_phy +
- ((priv->dirty_tx + 1) % DMA_TX_SIZE))
+ p->des3 = cpu_to_le32((unsigned int)((tx_q->dma_tx_phy +
+ ((tx_q->dirty_tx + 1) % DMA_TX_SIZE))
* sizeof(struct dma_desc)));
}
diff --git a/drivers/net/ethernet/stmicro/stmmac/common.h b/drivers/net/ethernet/stmicro/stmmac/common.h
index 04d9245..b7ce3fb 100644
--- a/drivers/net/ethernet/stmicro/stmmac/common.h
+++ b/drivers/net/ethernet/stmicro/stmmac/common.h
@@ -246,6 +246,15 @@
#define STMMAC_TX_MAX_FRAMES 256
#define STMMAC_TX_FRAMES 64
+/* Packets types */
+enum packets_types {
+ PACKET_AVCPQ = 0x1, /* AV Untagged Control packets */
+ PACKET_PTPQ = 0x2, /* PTP Packets */
+ PACKET_DCBCPQ = 0x3, /* DCB Control Packets */
+ PACKET_UPQ = 0x4, /* Untagged Packets */
+ PACKET_MCBCQ = 0x5, /* Multicast & Broadcast Packets */
+};
+
/* Rx IPC status */
enum rx_frame_status {
good_frame = 0x0,
@@ -324,6 +333,9 @@
unsigned int number_tx_queues;
/* Alternate (enhanced) DESC mode */
unsigned int enh_desc;
+ /* TX and RX FIFO sizes */
+ unsigned int tx_fifo_size;
+ unsigned int rx_fifo_size;
};
/* GMAC TX FIFO is 8K, Rx FIFO is 16K */
@@ -361,7 +373,7 @@
/* Invoked by the xmit function to prepare the tx descriptor */
void (*prepare_tx_desc) (struct dma_desc *p, int is_fs, int len,
bool csum_flag, int mode, bool tx_own,
- bool ls);
+ bool ls, unsigned int tot_pkt_len);
void (*prepare_tso_tx_desc)(struct dma_desc *p, int is_fs, int len1,
int len2, bool tx_own, bool ls,
unsigned int tcphdrlen,
@@ -413,6 +425,14 @@
int (*reset)(void __iomem *ioaddr);
void (*init)(void __iomem *ioaddr, struct stmmac_dma_cfg *dma_cfg,
u32 dma_tx, u32 dma_rx, int atds);
+ void (*init_chan)(void __iomem *ioaddr,
+ struct stmmac_dma_cfg *dma_cfg, u32 chan);
+ void (*init_rx_chan)(void __iomem *ioaddr,
+ struct stmmac_dma_cfg *dma_cfg,
+ u32 dma_rx_phy, u32 chan);
+ void (*init_tx_chan)(void __iomem *ioaddr,
+ struct stmmac_dma_cfg *dma_cfg,
+ u32 dma_tx_phy, u32 chan);
/* Configure the AXI Bus Mode Register */
void (*axi)(void __iomem *ioaddr, struct stmmac_axi *axi);
/* Dump DMA registers */
@@ -421,25 +441,28 @@
* An invalid value enables the store-and-forward mode */
void (*dma_mode)(void __iomem *ioaddr, int txmode, int rxmode,
int rxfifosz);
+ void (*dma_rx_mode)(void __iomem *ioaddr, int mode, u32 channel,
+ int fifosz);
+ void (*dma_tx_mode)(void __iomem *ioaddr, int mode, u32 channel);
/* To track extra statistic (if supported) */
void (*dma_diagnostic_fr) (void *data, struct stmmac_extra_stats *x,
void __iomem *ioaddr);
void (*enable_dma_transmission) (void __iomem *ioaddr);
- void (*enable_dma_irq) (void __iomem *ioaddr);
- void (*disable_dma_irq) (void __iomem *ioaddr);
- void (*start_tx) (void __iomem *ioaddr);
- void (*stop_tx) (void __iomem *ioaddr);
- void (*start_rx) (void __iomem *ioaddr);
- void (*stop_rx) (void __iomem *ioaddr);
+ void (*enable_dma_irq)(void __iomem *ioaddr, u32 chan);
+ void (*disable_dma_irq)(void __iomem *ioaddr, u32 chan);
+ void (*start_tx)(void __iomem *ioaddr, u32 chan);
+ void (*stop_tx)(void __iomem *ioaddr, u32 chan);
+ void (*start_rx)(void __iomem *ioaddr, u32 chan);
+ void (*stop_rx)(void __iomem *ioaddr, u32 chan);
int (*dma_interrupt) (void __iomem *ioaddr,
- struct stmmac_extra_stats *x);
+ struct stmmac_extra_stats *x, u32 chan);
/* If supported then get the optional core features */
void (*get_hw_feature)(void __iomem *ioaddr,
struct dma_features *dma_cap);
/* Program the HW RX Watchdog */
- void (*rx_watchdog) (void __iomem *ioaddr, u32 riwt);
- void (*set_tx_ring_len)(void __iomem *ioaddr, u32 len);
- void (*set_rx_ring_len)(void __iomem *ioaddr, u32 len);
+ void (*rx_watchdog)(void __iomem *ioaddr, u32 riwt, u32 number_chan);
+ void (*set_tx_ring_len)(void __iomem *ioaddr, u32 len, u32 chan);
+ void (*set_rx_ring_len)(void __iomem *ioaddr, u32 len, u32 chan);
void (*set_rx_tail_ptr)(void __iomem *ioaddr, u32 tail_ptr, u32 chan);
void (*set_tx_tail_ptr)(void __iomem *ioaddr, u32 tail_ptr, u32 chan);
void (*enable_tso)(void __iomem *ioaddr, bool en, u32 chan);
@@ -451,20 +474,44 @@
struct stmmac_ops {
/* MAC core initialization */
void (*core_init)(struct mac_device_info *hw, int mtu);
+ /* Enable the MAC RX/TX */
+ void (*set_mac)(void __iomem *ioaddr, bool enable);
/* Enable and verify that the IPC module is supported */
int (*rx_ipc)(struct mac_device_info *hw);
/* Enable RX Queues */
- void (*rx_queue_enable)(struct mac_device_info *hw, u32 queue);
+ void (*rx_queue_enable)(struct mac_device_info *hw, u8 mode, u32 queue);
+ /* RX Queues Priority */
+ void (*rx_queue_prio)(struct mac_device_info *hw, u32 prio, u32 queue);
+ /* TX Queues Priority */
+ void (*tx_queue_prio)(struct mac_device_info *hw, u32 prio, u32 queue);
+ /* RX Queues Routing */
+ void (*rx_queue_routing)(struct mac_device_info *hw, u8 packet,
+ u32 queue);
+ /* Program RX Algorithms */
+ void (*prog_mtl_rx_algorithms)(struct mac_device_info *hw, u32 rx_alg);
+ /* Program TX Algorithms */
+ void (*prog_mtl_tx_algorithms)(struct mac_device_info *hw, u32 tx_alg);
+ /* Set MTL TX queues weight */
+ void (*set_mtl_tx_queue_weight)(struct mac_device_info *hw,
+ u32 weight, u32 queue);
+ /* RX MTL queue to RX dma mapping */
+ void (*map_mtl_to_dma)(struct mac_device_info *hw, u32 queue, u32 chan);
+ /* Configure AV Algorithm */
+ void (*config_cbs)(struct mac_device_info *hw, u32 send_slope,
+ u32 idle_slope, u32 high_credit, u32 low_credit,
+ u32 queue);
/* Dump MAC registers */
void (*dump_regs)(struct mac_device_info *hw, u32 *reg_space);
/* Handle extra events on specific interrupts hw dependent */
int (*host_irq_status)(struct mac_device_info *hw,
struct stmmac_extra_stats *x);
+ /* Handle MTL interrupts */
+ int (*host_mtl_irq_status)(struct mac_device_info *hw, u32 chan);
/* Multicast filter setting */
void (*set_filter)(struct mac_device_info *hw, struct net_device *dev);
/* Flow control setting */
void (*flow_ctrl)(struct mac_device_info *hw, unsigned int duplex,
- unsigned int fc, unsigned int pause_time);
+ unsigned int fc, unsigned int pause_time, u32 tx_cnt);
/* Set power management mode (e.g. magic frame) */
void (*pmt)(struct mac_device_info *hw, unsigned long mode);
/* Set/Get Unicast MAC addresses */
@@ -477,7 +524,8 @@
void (*reset_eee_mode)(struct mac_device_info *hw);
void (*set_eee_timer)(struct mac_device_info *hw, int ls, int tw);
void (*set_eee_pls)(struct mac_device_info *hw, int link);
- void (*debug)(void __iomem *ioaddr, struct stmmac_extra_stats *x);
+ void (*debug)(void __iomem *ioaddr, struct stmmac_extra_stats *x,
+ u32 rx_queues, u32 tx_queues);
/* PCS calls */
void (*pcs_ctrl_ane)(void __iomem *ioaddr, bool ane, bool srgmi_ral,
bool loopback);
@@ -547,6 +595,11 @@
unsigned int ps;
};
+struct stmmac_rx_routing {
+ u32 reg_mask;
+ u32 reg_shift;
+};
+
struct mac_device_info *dwmac1000_setup(void __iomem *ioaddr, int mcbins,
int perfect_uc_entries,
int *synopsys_id);
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac-dwc-qos-eth.c b/drivers/net/ethernet/stmicro/stmmac/dwmac-dwc-qos-eth.c
index 1a3fa3d..dd6a2f9 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac-dwc-qos-eth.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac-dwc-qos-eth.c
@@ -14,16 +14,34 @@
#include <linux/clk.h>
#include <linux/clk-provider.h>
#include <linux/device.h>
+#include <linux/gpio/consumer.h>
#include <linux/ethtool.h>
#include <linux/io.h>
+#include <linux/iopoll.h>
#include <linux/ioport.h>
#include <linux/module.h>
+#include <linux/of_device.h>
#include <linux/of_net.h>
#include <linux/mfd/syscon.h>
#include <linux/platform_device.h>
+#include <linux/reset.h>
#include <linux/stmmac.h>
#include "stmmac_platform.h"
+#include "dwmac4.h"
+
+struct tegra_eqos {
+ struct device *dev;
+ void __iomem *regs;
+
+ struct reset_control *rst;
+ struct clk *clk_master;
+ struct clk *clk_slave;
+ struct clk *clk_tx;
+ struct clk *clk_rx;
+
+ struct gpio_desc *reset;
+};
static int dwc_eth_dwmac_config_dt(struct platform_device *pdev,
struct plat_stmmacenet_data *plat_dat)
@@ -106,13 +124,309 @@
return 0;
}
+static void *dwc_qos_probe(struct platform_device *pdev,
+ struct plat_stmmacenet_data *plat_dat,
+ struct stmmac_resources *stmmac_res)
+{
+ int err;
+
+ plat_dat->stmmac_clk = devm_clk_get(&pdev->dev, "apb_pclk");
+ if (IS_ERR(plat_dat->stmmac_clk)) {
+ dev_err(&pdev->dev, "apb_pclk clock not found.\n");
+ return ERR_CAST(plat_dat->stmmac_clk);
+ }
+
+ err = clk_prepare_enable(plat_dat->stmmac_clk);
+ if (err < 0) {
+ dev_err(&pdev->dev, "failed to enable apb_pclk clock: %d\n",
+ err);
+ return ERR_PTR(err);
+ }
+
+ plat_dat->pclk = devm_clk_get(&pdev->dev, "phy_ref_clk");
+ if (IS_ERR(plat_dat->pclk)) {
+ dev_err(&pdev->dev, "phy_ref_clk clock not found.\n");
+ err = PTR_ERR(plat_dat->pclk);
+ goto disable;
+ }
+
+ err = clk_prepare_enable(plat_dat->pclk);
+ if (err < 0) {
+ dev_err(&pdev->dev, "failed to enable phy_ref clock: %d\n",
+ err);
+ goto disable;
+ }
+
+ return NULL;
+
+disable:
+ clk_disable_unprepare(plat_dat->stmmac_clk);
+ return ERR_PTR(err);
+}
+
+static int dwc_qos_remove(struct platform_device *pdev)
+{
+ struct net_device *ndev = platform_get_drvdata(pdev);
+ struct stmmac_priv *priv = netdev_priv(ndev);
+
+ clk_disable_unprepare(priv->plat->pclk);
+ clk_disable_unprepare(priv->plat->stmmac_clk);
+
+ return 0;
+}
+
+#define SDMEMCOMPPADCTRL 0x8800
+#define SDMEMCOMPPADCTRL_PAD_E_INPUT_OR_E_PWRD BIT(31)
+
+#define AUTO_CAL_CONFIG 0x8804
+#define AUTO_CAL_CONFIG_START BIT(31)
+#define AUTO_CAL_CONFIG_ENABLE BIT(29)
+
+#define AUTO_CAL_STATUS 0x880c
+#define AUTO_CAL_STATUS_ACTIVE BIT(31)
+
+static void tegra_eqos_fix_speed(void *priv, unsigned int speed)
+{
+ struct tegra_eqos *eqos = priv;
+ unsigned long rate = 125000000;
+ bool needs_calibration = false;
+ u32 value;
+ int err;
+
+ switch (speed) {
+ case SPEED_1000:
+ needs_calibration = true;
+ rate = 125000000;
+ break;
+
+ case SPEED_100:
+ needs_calibration = true;
+ rate = 25000000;
+ break;
+
+ case SPEED_10:
+ rate = 2500000;
+ break;
+
+ default:
+ dev_err(eqos->dev, "invalid speed %u\n", speed);
+ break;
+ }
+
+ if (needs_calibration) {
+ /* calibrate */
+ value = readl(eqos->regs + SDMEMCOMPPADCTRL);
+ value |= SDMEMCOMPPADCTRL_PAD_E_INPUT_OR_E_PWRD;
+ writel(value, eqos->regs + SDMEMCOMPPADCTRL);
+
+ udelay(1);
+
+ value = readl(eqos->regs + AUTO_CAL_CONFIG);
+ value |= AUTO_CAL_CONFIG_START | AUTO_CAL_CONFIG_ENABLE;
+ writel(value, eqos->regs + AUTO_CAL_CONFIG);
+
+ err = readl_poll_timeout_atomic(eqos->regs + AUTO_CAL_STATUS,
+ value,
+ value & AUTO_CAL_STATUS_ACTIVE,
+ 1, 10);
+ if (err < 0) {
+ dev_err(eqos->dev, "calibration did not start\n");
+ goto failed;
+ }
+
+ err = readl_poll_timeout_atomic(eqos->regs + AUTO_CAL_STATUS,
+ value,
+ (value & AUTO_CAL_STATUS_ACTIVE) == 0,
+ 20, 200);
+ if (err < 0) {
+ dev_err(eqos->dev, "calibration didn't finish\n");
+ goto failed;
+ }
+
+ failed:
+ value = readl(eqos->regs + SDMEMCOMPPADCTRL);
+ value &= ~SDMEMCOMPPADCTRL_PAD_E_INPUT_OR_E_PWRD;
+ writel(value, eqos->regs + SDMEMCOMPPADCTRL);
+ } else {
+ value = readl(eqos->regs + AUTO_CAL_CONFIG);
+ value &= ~AUTO_CAL_CONFIG_ENABLE;
+ writel(value, eqos->regs + AUTO_CAL_CONFIG);
+ }
+
+ err = clk_set_rate(eqos->clk_tx, rate);
+ if (err < 0)
+ dev_err(eqos->dev, "failed to set TX rate: %d\n", err);
+}
+
+static int tegra_eqos_init(struct platform_device *pdev, void *priv)
+{
+ struct tegra_eqos *eqos = priv;
+ unsigned long rate;
+ u32 value;
+
+ rate = clk_get_rate(eqos->clk_slave);
+
+ value = (rate / 1000000) - 1;
+ writel(value, eqos->regs + GMAC_1US_TIC_COUNTER);
+
+ return 0;
+}
+
+static void *tegra_eqos_probe(struct platform_device *pdev,
+ struct plat_stmmacenet_data *data,
+ struct stmmac_resources *res)
+{
+ struct tegra_eqos *eqos;
+ int err;
+
+ eqos = devm_kzalloc(&pdev->dev, sizeof(*eqos), GFP_KERNEL);
+ if (!eqos) {
+ err = -ENOMEM;
+ goto error;
+ }
+
+ eqos->dev = &pdev->dev;
+ eqos->regs = res->addr;
+
+ eqos->clk_master = devm_clk_get(&pdev->dev, "master_bus");
+ if (IS_ERR(eqos->clk_master)) {
+ err = PTR_ERR(eqos->clk_master);
+ goto error;
+ }
+
+ err = clk_prepare_enable(eqos->clk_master);
+ if (err < 0)
+ goto error;
+
+ eqos->clk_slave = devm_clk_get(&pdev->dev, "slave_bus");
+ if (IS_ERR(eqos->clk_slave)) {
+ err = PTR_ERR(eqos->clk_slave);
+ goto disable_master;
+ }
+
+ data->stmmac_clk = eqos->clk_slave;
+
+ err = clk_prepare_enable(eqos->clk_slave);
+ if (err < 0)
+ goto disable_master;
+
+ eqos->clk_rx = devm_clk_get(&pdev->dev, "rx");
+ if (IS_ERR(eqos->clk_rx)) {
+ err = PTR_ERR(eqos->clk_rx);
+ goto disable_slave;
+ }
+
+ err = clk_prepare_enable(eqos->clk_rx);
+ if (err < 0)
+ goto disable_slave;
+
+ eqos->clk_tx = devm_clk_get(&pdev->dev, "tx");
+ if (IS_ERR(eqos->clk_tx)) {
+ err = PTR_ERR(eqos->clk_tx);
+ goto disable_rx;
+ }
+
+ err = clk_prepare_enable(eqos->clk_tx);
+ if (err < 0)
+ goto disable_rx;
+
+ eqos->reset = devm_gpiod_get(&pdev->dev, "phy-reset", GPIOD_OUT_HIGH);
+ if (IS_ERR(eqos->reset)) {
+ err = PTR_ERR(eqos->reset);
+ goto disable_tx;
+ }
+
+ usleep_range(2000, 4000);
+ gpiod_set_value(eqos->reset, 0);
+
+ eqos->rst = devm_reset_control_get(&pdev->dev, "eqos");
+ if (IS_ERR(eqos->rst)) {
+ err = PTR_ERR(eqos->rst);
+ goto reset_phy;
+ }
+
+ err = reset_control_assert(eqos->rst);
+ if (err < 0)
+ goto reset_phy;
+
+ usleep_range(2000, 4000);
+
+ err = reset_control_deassert(eqos->rst);
+ if (err < 0)
+ goto reset_phy;
+
+ usleep_range(2000, 4000);
+
+ data->fix_mac_speed = tegra_eqos_fix_speed;
+ data->init = tegra_eqos_init;
+ data->bsp_priv = eqos;
+
+ err = tegra_eqos_init(pdev, eqos);
+ if (err < 0)
+ goto reset;
+
+out:
+ return eqos;
+
+reset:
+ reset_control_assert(eqos->rst);
+reset_phy:
+ gpiod_set_value(eqos->reset, 1);
+disable_tx:
+ clk_disable_unprepare(eqos->clk_tx);
+disable_rx:
+ clk_disable_unprepare(eqos->clk_rx);
+disable_slave:
+ clk_disable_unprepare(eqos->clk_slave);
+disable_master:
+ clk_disable_unprepare(eqos->clk_master);
+error:
+ eqos = ERR_PTR(err);
+ goto out;
+}
+
+static int tegra_eqos_remove(struct platform_device *pdev)
+{
+ struct tegra_eqos *eqos = get_stmmac_bsp_priv(&pdev->dev);
+
+ reset_control_assert(eqos->rst);
+ gpiod_set_value(eqos->reset, 1);
+ clk_disable_unprepare(eqos->clk_tx);
+ clk_disable_unprepare(eqos->clk_rx);
+ clk_disable_unprepare(eqos->clk_slave);
+ clk_disable_unprepare(eqos->clk_master);
+
+ return 0;
+}
+
+struct dwc_eth_dwmac_data {
+ void *(*probe)(struct platform_device *pdev,
+ struct plat_stmmacenet_data *data,
+ struct stmmac_resources *res);
+ int (*remove)(struct platform_device *pdev);
+};
+
+static const struct dwc_eth_dwmac_data dwc_qos_data = {
+ .probe = dwc_qos_probe,
+ .remove = dwc_qos_remove,
+};
+
+static const struct dwc_eth_dwmac_data tegra_eqos_data = {
+ .probe = tegra_eqos_probe,
+ .remove = tegra_eqos_remove,
+};
+
static int dwc_eth_dwmac_probe(struct platform_device *pdev)
{
+ const struct dwc_eth_dwmac_data *data;
struct plat_stmmacenet_data *plat_dat;
struct stmmac_resources stmmac_res;
struct resource *res;
+ void *priv;
int ret;
+ data = of_device_get_match_data(&pdev->dev);
+
memset(&stmmac_res, 0, sizeof(struct stmmac_resources));
/**
@@ -138,39 +452,26 @@
if (IS_ERR(plat_dat))
return PTR_ERR(plat_dat);
- plat_dat->stmmac_clk = devm_clk_get(&pdev->dev, "apb_pclk");
- if (IS_ERR(plat_dat->stmmac_clk)) {
- dev_err(&pdev->dev, "apb_pclk clock not found.\n");
- ret = PTR_ERR(plat_dat->stmmac_clk);
- plat_dat->stmmac_clk = NULL;
- goto err_remove_config_dt;
+ priv = data->probe(pdev, plat_dat, &stmmac_res);
+ if (IS_ERR(priv)) {
+ ret = PTR_ERR(priv);
+ dev_err(&pdev->dev, "failed to probe subdriver: %d\n", ret);
+ goto remove_config;
}
- clk_prepare_enable(plat_dat->stmmac_clk);
-
- plat_dat->pclk = devm_clk_get(&pdev->dev, "phy_ref_clk");
- if (IS_ERR(plat_dat->pclk)) {
- dev_err(&pdev->dev, "phy_ref_clk clock not found.\n");
- ret = PTR_ERR(plat_dat->pclk);
- plat_dat->pclk = NULL;
- goto err_out_clk_dis_phy;
- }
- clk_prepare_enable(plat_dat->pclk);
ret = dwc_eth_dwmac_config_dt(pdev, plat_dat);
if (ret)
- goto err_out_clk_dis_aper;
+ goto remove;
ret = stmmac_dvr_probe(&pdev->dev, plat_dat, &stmmac_res);
if (ret)
- goto err_out_clk_dis_aper;
+ goto remove;
- return 0;
+ return ret;
-err_out_clk_dis_aper:
- clk_disable_unprepare(plat_dat->pclk);
-err_out_clk_dis_phy:
- clk_disable_unprepare(plat_dat->stmmac_clk);
-err_remove_config_dt:
+remove:
+ data->remove(pdev);
+remove_config:
stmmac_remove_config_dt(pdev, plat_dat);
return ret;
@@ -178,11 +479,29 @@
static int dwc_eth_dwmac_remove(struct platform_device *pdev)
{
- return stmmac_pltfr_remove(pdev);
+ struct net_device *ndev = platform_get_drvdata(pdev);
+ struct stmmac_priv *priv = netdev_priv(ndev);
+ const struct dwc_eth_dwmac_data *data;
+ int err;
+
+ data = of_device_get_match_data(&pdev->dev);
+
+ err = stmmac_dvr_remove(&pdev->dev);
+ if (err < 0)
+ dev_err(&pdev->dev, "failed to remove platform: %d\n", err);
+
+ err = data->remove(pdev);
+ if (err < 0)
+ dev_err(&pdev->dev, "failed to remove subdriver: %d\n", err);
+
+ stmmac_remove_config_dt(pdev, priv->plat);
+
+ return err;
}
static const struct of_device_id dwc_eth_dwmac_match[] = {
- { .compatible = "snps,dwc-qos-ethernet-4.10", },
+ { .compatible = "snps,dwc-qos-ethernet-4.10", .data = &dwc_qos_data },
+ { .compatible = "nvidia,tegra186-eqos", .data = &tegra_eqos_data },
{ }
};
MODULE_DEVICE_TABLE(of, dwc_eth_dwmac_match);
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac-rk.c b/drivers/net/ethernet/stmicro/stmmac/dwmac-rk.c
index e5db6ac..f0df519 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac-rk.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac-rk.c
@@ -74,6 +74,10 @@
#define GRF_BIT(nr) (BIT(nr) | BIT(nr+16))
#define GRF_CLR_BIT(nr) (BIT(nr+16))
+#define DELAY_ENABLE(soc, tx, rx) \
+ (((tx) ? soc##_GMAC_TXCLK_DLY_ENABLE : soc##_GMAC_TXCLK_DLY_DISABLE) | \
+ ((rx) ? soc##_GMAC_RXCLK_DLY_ENABLE : soc##_GMAC_RXCLK_DLY_DISABLE))
+
#define RK3228_GRF_MAC_CON0 0x0900
#define RK3228_GRF_MAC_CON1 0x0904
@@ -115,8 +119,7 @@
regmap_write(bsp_priv->grf, RK3228_GRF_MAC_CON1,
RK3228_GMAC_PHY_INTF_SEL_RGMII |
RK3228_GMAC_RMII_MODE_CLR |
- RK3228_GMAC_RXCLK_DLY_ENABLE |
- RK3228_GMAC_TXCLK_DLY_ENABLE);
+ DELAY_ENABLE(RK3228, tx_delay, rx_delay));
regmap_write(bsp_priv->grf, RK3228_GRF_MAC_CON0,
RK3228_GMAC_CLK_RX_DL_CFG(rx_delay) |
@@ -232,8 +235,7 @@
RK3288_GMAC_PHY_INTF_SEL_RGMII |
RK3288_GMAC_RMII_MODE_CLR);
regmap_write(bsp_priv->grf, RK3288_GRF_SOC_CON3,
- RK3288_GMAC_RXCLK_DLY_ENABLE |
- RK3288_GMAC_TXCLK_DLY_ENABLE |
+ DELAY_ENABLE(RK3288, tx_delay, rx_delay) |
RK3288_GMAC_CLK_RX_DL_CFG(rx_delay) |
RK3288_GMAC_CLK_TX_DL_CFG(tx_delay));
}
@@ -460,8 +462,7 @@
RK3366_GMAC_PHY_INTF_SEL_RGMII |
RK3366_GMAC_RMII_MODE_CLR);
regmap_write(bsp_priv->grf, RK3366_GRF_SOC_CON7,
- RK3366_GMAC_RXCLK_DLY_ENABLE |
- RK3366_GMAC_TXCLK_DLY_ENABLE |
+ DELAY_ENABLE(RK3366, tx_delay, rx_delay) |
RK3366_GMAC_CLK_RX_DL_CFG(rx_delay) |
RK3366_GMAC_CLK_TX_DL_CFG(tx_delay));
}
@@ -572,8 +573,7 @@
RK3368_GMAC_PHY_INTF_SEL_RGMII |
RK3368_GMAC_RMII_MODE_CLR);
regmap_write(bsp_priv->grf, RK3368_GRF_SOC_CON16,
- RK3368_GMAC_RXCLK_DLY_ENABLE |
- RK3368_GMAC_TXCLK_DLY_ENABLE |
+ DELAY_ENABLE(RK3368, tx_delay, rx_delay) |
RK3368_GMAC_CLK_RX_DL_CFG(rx_delay) |
RK3368_GMAC_CLK_TX_DL_CFG(tx_delay));
}
@@ -684,8 +684,7 @@
RK3399_GMAC_PHY_INTF_SEL_RGMII |
RK3399_GMAC_RMII_MODE_CLR);
regmap_write(bsp_priv->grf, RK3399_GRF_SOC_CON6,
- RK3399_GMAC_RXCLK_DLY_ENABLE |
- RK3399_GMAC_TXCLK_DLY_ENABLE |
+ DELAY_ENABLE(RK3399, tx_delay, rx_delay) |
RK3399_GMAC_CLK_RX_DL_CFG(rx_delay) |
RK3399_GMAC_CLK_TX_DL_CFG(tx_delay));
}
@@ -985,14 +984,29 @@
return ret;
/*rmii or rgmii*/
- if (bsp_priv->phy_iface == PHY_INTERFACE_MODE_RGMII) {
+ switch (bsp_priv->phy_iface) {
+ case PHY_INTERFACE_MODE_RGMII:
dev_info(dev, "init for RGMII\n");
bsp_priv->ops->set_to_rgmii(bsp_priv, bsp_priv->tx_delay,
bsp_priv->rx_delay);
- } else if (bsp_priv->phy_iface == PHY_INTERFACE_MODE_RMII) {
+ break;
+ case PHY_INTERFACE_MODE_RGMII_ID:
+ dev_info(dev, "init for RGMII_ID\n");
+ bsp_priv->ops->set_to_rgmii(bsp_priv, 0, 0);
+ break;
+ case PHY_INTERFACE_MODE_RGMII_RXID:
+ dev_info(dev, "init for RGMII_RXID\n");
+ bsp_priv->ops->set_to_rgmii(bsp_priv, bsp_priv->tx_delay, 0);
+ break;
+ case PHY_INTERFACE_MODE_RGMII_TXID:
+ dev_info(dev, "init for RGMII_TXID\n");
+ bsp_priv->ops->set_to_rgmii(bsp_priv, 0, bsp_priv->rx_delay);
+ break;
+ case PHY_INTERFACE_MODE_RMII:
dev_info(dev, "init for RMII\n");
bsp_priv->ops->set_to_rmii(bsp_priv);
- } else {
+ break;
+ default:
dev_err(dev, "NO interface defined!\n");
}
@@ -1022,12 +1036,19 @@
struct rk_priv_data *bsp_priv = priv;
struct device *dev = &bsp_priv->pdev->dev;
- if (bsp_priv->phy_iface == PHY_INTERFACE_MODE_RGMII)
+ switch (bsp_priv->phy_iface) {
+ case PHY_INTERFACE_MODE_RGMII:
+ case PHY_INTERFACE_MODE_RGMII_ID:
+ case PHY_INTERFACE_MODE_RGMII_RXID:
+ case PHY_INTERFACE_MODE_RGMII_TXID:
bsp_priv->ops->set_rgmii_speed(bsp_priv, speed);
- else if (bsp_priv->phy_iface == PHY_INTERFACE_MODE_RMII)
+ break;
+ case PHY_INTERFACE_MODE_RMII:
bsp_priv->ops->set_rmii_speed(bsp_priv, speed);
- else
+ break;
+ default:
dev_err(dev, "unsupported interface %d", bsp_priv->phy_iface);
+ }
}
static int rk_gmac_probe(struct platform_device *pdev)
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac1000_core.c b/drivers/net/ethernet/stmicro/stmmac/dwmac1000_core.c
index 19b9b30..f3d9305 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac1000_core.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac1000_core.c
@@ -216,7 +216,8 @@
static void dwmac1000_flow_ctrl(struct mac_device_info *hw, unsigned int duplex,
- unsigned int fc, unsigned int pause_time)
+ unsigned int fc, unsigned int pause_time,
+ u32 tx_cnt)
{
void __iomem *ioaddr = hw->pcsr;
/* Set flow such that DZPQ in Mac Register 6 is 0,
@@ -412,7 +413,8 @@
dwmac_get_adv_lp(ioaddr, GMAC_PCS_BASE, adv);
}
-static void dwmac1000_debug(void __iomem *ioaddr, struct stmmac_extra_stats *x)
+static void dwmac1000_debug(void __iomem *ioaddr, struct stmmac_extra_stats *x,
+ u32 rx_queues, u32 tx_queues)
{
u32 value = readl(ioaddr + GMAC_DEBUG);
@@ -488,6 +490,7 @@
static const struct stmmac_ops dwmac1000_ops = {
.core_init = dwmac1000_core_init,
+ .set_mac = stmmac_set_mac,
.rx_ipc = dwmac1000_rx_ipc_enable,
.dump_regs = dwmac1000_dump_regs,
.host_irq_status = dwmac1000_irq_status,
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac1000_dma.c b/drivers/net/ethernet/stmicro/stmmac/dwmac1000_dma.c
index d3654a4..471a9aa 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac1000_dma.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac1000_dma.c
@@ -247,7 +247,8 @@
dma_cap->enh_desc = (hw_cap & DMA_HW_FEAT_ENHDESSEL) >> 24;
}
-static void dwmac1000_rx_watchdog(void __iomem *ioaddr, u32 riwt)
+static void dwmac1000_rx_watchdog(void __iomem *ioaddr, u32 riwt,
+ u32 number_chan)
{
writel(riwt, ioaddr + DMA_RX_WATCHDOG);
}
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac100_core.c b/drivers/net/ethernet/stmicro/stmmac/dwmac100_core.c
index e370cce..1b36091 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac100_core.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac100_core.c
@@ -131,7 +131,8 @@
}
static void dwmac100_flow_ctrl(struct mac_device_info *hw, unsigned int duplex,
- unsigned int fc, unsigned int pause_time)
+ unsigned int fc, unsigned int pause_time,
+ u32 tx_cnt)
{
void __iomem *ioaddr = hw->pcsr;
unsigned int flow = MAC_FLOW_CTRL_ENABLE;
@@ -149,6 +150,7 @@
static const struct stmmac_ops dwmac100_ops = {
.core_init = dwmac100_core_init,
+ .set_mac = stmmac_set_mac,
.rx_ipc = dwmac100_rx_ipc_enable,
.dump_regs = dwmac100_dump_mac_regs,
.host_irq_status = dwmac100_irq_status,
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac4.h b/drivers/net/ethernet/stmicro/stmmac/dwmac4.h
index db45134..d74cedf 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac4.h
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac4.h
@@ -22,9 +22,15 @@
#define GMAC_HASH_TAB_32_63 0x00000014
#define GMAC_RX_FLOW_CTRL 0x00000090
#define GMAC_QX_TX_FLOW_CTRL(x) (0x70 + x * 4)
+#define GMAC_TXQ_PRTY_MAP0 0x98
+#define GMAC_TXQ_PRTY_MAP1 0x9C
#define GMAC_RXQ_CTRL0 0x000000a0
+#define GMAC_RXQ_CTRL1 0x000000a4
+#define GMAC_RXQ_CTRL2 0x000000a8
+#define GMAC_RXQ_CTRL3 0x000000ac
#define GMAC_INT_STATUS 0x000000b0
#define GMAC_INT_EN 0x000000b4
+#define GMAC_1US_TIC_COUNTER 0x000000dc
#define GMAC_PCS_BASE 0x000000e0
#define GMAC_PHYIF_CONTROL_STATUS 0x000000f8
#define GMAC_PMT 0x000000c0
@@ -38,6 +44,22 @@
#define GMAC_ADDR_HIGH(reg) (0x300 + reg * 8)
#define GMAC_ADDR_LOW(reg) (0x304 + reg * 8)
+/* RX Queues Routing */
+#define GMAC_RXQCTRL_AVCPQ_MASK GENMASK(2, 0)
+#define GMAC_RXQCTRL_AVCPQ_SHIFT 0
+#define GMAC_RXQCTRL_PTPQ_MASK GENMASK(6, 4)
+#define GMAC_RXQCTRL_PTPQ_SHIFT 4
+#define GMAC_RXQCTRL_DCBCPQ_MASK GENMASK(10, 8)
+#define GMAC_RXQCTRL_DCBCPQ_SHIFT 8
+#define GMAC_RXQCTRL_UPQ_MASK GENMASK(14, 12)
+#define GMAC_RXQCTRL_UPQ_SHIFT 12
+#define GMAC_RXQCTRL_MCBCQ_MASK GENMASK(18, 16)
+#define GMAC_RXQCTRL_MCBCQ_SHIFT 16
+#define GMAC_RXQCTRL_MCBCQEN BIT(20)
+#define GMAC_RXQCTRL_MCBCQEN_SHIFT 20
+#define GMAC_RXQCTRL_TACPQE BIT(21)
+#define GMAC_RXQCTRL_TACPQE_SHIFT 21
+
/* MAC Packet Filtering */
#define GMAC_PACKET_FILTER_PR BIT(0)
#define GMAC_PACKET_FILTER_HMC BIT(2)
@@ -53,6 +75,14 @@
/* MAC Flow Control RX */
#define GMAC_RX_FLOW_CTRL_RFE BIT(0)
+/* RX Queues Priorities */
+#define GMAC_RXQCTRL_PSRQX_MASK(x) GENMASK(7 + ((x) * 8), 0 + ((x) * 8))
+#define GMAC_RXQCTRL_PSRQX_SHIFT(x) ((x) * 8)
+
+/* TX Queues Priorities */
+#define GMAC_TXQCTRL_PSTQX_MASK(x) GENMASK(7 + ((x) * 8), 0 + ((x) * 8))
+#define GMAC_TXQCTRL_PSTQX_SHIFT(x) ((x) * 8)
+
/* MAC Flow Control TX */
#define GMAC_TX_FLOW_CTRL_TFE BIT(1)
#define GMAC_TX_FLOW_CTRL_PT_SHIFT 16
@@ -148,6 +178,8 @@
/* MAC HW features1 bitmap */
#define GMAC_HW_FEAT_AVSEL BIT(20)
#define GMAC_HW_TSOEN BIT(18)
+#define GMAC_HW_TXFIFOSIZE GENMASK(10, 6)
+#define GMAC_HW_RXFIFOSIZE GENMASK(4, 0)
/* MAC HW features2 bitmap */
#define GMAC_HW_FEAT_TXCHCNT GENMASK(21, 18)
@@ -161,8 +193,25 @@
#define GMAC_HI_REG_AE BIT(31)
/* MTL registers */
+#define MTL_OPERATION_MODE 0x00000c00
+#define MTL_OPERATION_SCHALG_MASK GENMASK(6, 5)
+#define MTL_OPERATION_SCHALG_WRR (0x0 << 5)
+#define MTL_OPERATION_SCHALG_WFQ (0x1 << 5)
+#define MTL_OPERATION_SCHALG_DWRR (0x2 << 5)
+#define MTL_OPERATION_SCHALG_SP (0x3 << 5)
+#define MTL_OPERATION_RAA BIT(2)
+#define MTL_OPERATION_RAA_SP (0x0 << 2)
+#define MTL_OPERATION_RAA_WSP (0x1 << 2)
+
#define MTL_INT_STATUS 0x00000c20
-#define MTL_INT_Q0 BIT(0)
+#define MTL_INT_QX(x) BIT(x)
+
+#define MTL_RXQ_DMA_MAP0 0x00000c30 /* queue 0 to 3 */
+#define MTL_RXQ_DMA_MAP1 0x00000c34 /* queue 4 to 7 */
+#define MTL_RXQ_DMA_Q04MDMACH_MASK GENMASK(3, 0)
+#define MTL_RXQ_DMA_Q04MDMACH(x) ((x) << 0)
+#define MTL_RXQ_DMA_QXMDMACH_MASK(x) GENMASK(11 + (8 * ((x) - 1)), 8 * (x))
+#define MTL_RXQ_DMA_QXMDMACH(chan, q) ((chan) << (8 * (q)))
#define MTL_CHAN_BASE_ADDR 0x00000d00
#define MTL_CHAN_BASE_OFFSET 0x40
@@ -180,6 +229,7 @@
#define MTL_OP_MODE_TSF BIT(1)
#define MTL_OP_MODE_TQS_MASK GENMASK(24, 16)
+#define MTL_OP_MODE_TQS_SHIFT 16
#define MTL_OP_MODE_TTC_MASK 0x70
#define MTL_OP_MODE_TTC_SHIFT 4
@@ -193,6 +243,17 @@
#define MTL_OP_MODE_TTC_384 (6 << MTL_OP_MODE_TTC_SHIFT)
#define MTL_OP_MODE_TTC_512 (7 << MTL_OP_MODE_TTC_SHIFT)
+#define MTL_OP_MODE_RQS_MASK GENMASK(29, 20)
+#define MTL_OP_MODE_RQS_SHIFT 20
+
+#define MTL_OP_MODE_RFD_MASK GENMASK(19, 14)
+#define MTL_OP_MODE_RFD_SHIFT 14
+
+#define MTL_OP_MODE_RFA_MASK GENMASK(13, 8)
+#define MTL_OP_MODE_RFA_SHIFT 8
+
+#define MTL_OP_MODE_EHFC BIT(7)
+
#define MTL_OP_MODE_RTC_MASK 0x18
#define MTL_OP_MODE_RTC_SHIFT 3
@@ -201,6 +262,46 @@
#define MTL_OP_MODE_RTC_96 (2 << MTL_OP_MODE_RTC_SHIFT)
#define MTL_OP_MODE_RTC_128 (3 << MTL_OP_MODE_RTC_SHIFT)
+/* MTL ETS Control register */
+#define MTL_ETS_CTRL_BASE_ADDR 0x00000d10
+#define MTL_ETS_CTRL_BASE_OFFSET 0x40
+#define MTL_ETSX_CTRL_BASE_ADDR(x) (MTL_ETS_CTRL_BASE_ADDR + \
+ ((x) * MTL_ETS_CTRL_BASE_OFFSET))
+
+#define MTL_ETS_CTRL_CC BIT(3)
+#define MTL_ETS_CTRL_AVALG BIT(2)
+
+/* MTL Queue Quantum Weight */
+#define MTL_TXQ_WEIGHT_BASE_ADDR 0x00000d18
+#define MTL_TXQ_WEIGHT_BASE_OFFSET 0x40
+#define MTL_TXQX_WEIGHT_BASE_ADDR(x) (MTL_TXQ_WEIGHT_BASE_ADDR + \
+ ((x) * MTL_TXQ_WEIGHT_BASE_OFFSET))
+#define MTL_TXQ_WEIGHT_ISCQW_MASK GENMASK(20, 0)
+
+/* MTL sendSlopeCredit register */
+#define MTL_SEND_SLP_CRED_BASE_ADDR 0x00000d1c
+#define MTL_SEND_SLP_CRED_OFFSET 0x40
+#define MTL_SEND_SLP_CREDX_BASE_ADDR(x) (MTL_SEND_SLP_CRED_BASE_ADDR + \
+ ((x) * MTL_SEND_SLP_CRED_OFFSET))
+
+#define MTL_SEND_SLP_CRED_SSC_MASK GENMASK(13, 0)
+
+/* MTL hiCredit register */
+#define MTL_HIGH_CRED_BASE_ADDR 0x00000d20
+#define MTL_HIGH_CRED_OFFSET 0x40
+#define MTL_HIGH_CREDX_BASE_ADDR(x) (MTL_HIGH_CRED_BASE_ADDR + \
+ ((x) * MTL_HIGH_CRED_OFFSET))
+
+#define MTL_HIGH_CRED_HC_MASK GENMASK(28, 0)
+
+/* MTL loCredit register */
+#define MTL_LOW_CRED_BASE_ADDR 0x00000d24
+#define MTL_LOW_CRED_OFFSET 0x40
+#define MTL_LOW_CREDX_BASE_ADDR(x) (MTL_LOW_CRED_BASE_ADDR + \
+ ((x) * MTL_LOW_CRED_OFFSET))
+
+#define MTL_HIGH_CRED_LC_MASK GENMASK(28, 0)
+
/* MTL debug */
#define MTL_DEBUG_TXSTSFSTS BIT(5)
#define MTL_DEBUG_TXFSTS BIT(4)
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac4_core.c b/drivers/net/ethernet/stmicro/stmmac/dwmac4_core.c
index 1e79e65..48793f2 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac4_core.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac4_core.c
@@ -59,17 +59,211 @@
writel(value, ioaddr + GMAC_INT_EN);
}
-static void dwmac4_rx_queue_enable(struct mac_device_info *hw, u32 queue)
+static void dwmac4_rx_queue_enable(struct mac_device_info *hw,
+ u8 mode, u32 queue)
{
void __iomem *ioaddr = hw->pcsr;
u32 value = readl(ioaddr + GMAC_RXQ_CTRL0);
value &= GMAC_RX_QUEUE_CLEAR(queue);
- value |= GMAC_RX_AV_QUEUE_ENABLE(queue);
+ if (mode == MTL_QUEUE_AVB)
+ value |= GMAC_RX_AV_QUEUE_ENABLE(queue);
+ else if (mode == MTL_QUEUE_DCB)
+ value |= GMAC_RX_DCB_QUEUE_ENABLE(queue);
writel(value, ioaddr + GMAC_RXQ_CTRL0);
}
+static void dwmac4_rx_queue_priority(struct mac_device_info *hw,
+ u32 prio, u32 queue)
+{
+ void __iomem *ioaddr = hw->pcsr;
+ u32 base_register;
+ u32 value;
+
+ base_register = (queue < 4) ? GMAC_RXQ_CTRL2 : GMAC_RXQ_CTRL3;
+
+ value = readl(ioaddr + base_register);
+
+ value &= ~GMAC_RXQCTRL_PSRQX_MASK(queue);
+ value |= (prio << GMAC_RXQCTRL_PSRQX_SHIFT(queue)) &
+ GMAC_RXQCTRL_PSRQX_MASK(queue);
+ writel(value, ioaddr + base_register);
+}
+
+static void dwmac4_tx_queue_priority(struct mac_device_info *hw,
+ u32 prio, u32 queue)
+{
+ void __iomem *ioaddr = hw->pcsr;
+ u32 base_register;
+ u32 value;
+
+ base_register = (queue < 4) ? GMAC_TXQ_PRTY_MAP0 : GMAC_TXQ_PRTY_MAP1;
+
+ value = readl(ioaddr + base_register);
+
+ value &= ~GMAC_TXQCTRL_PSTQX_MASK(queue);
+ value |= (prio << GMAC_TXQCTRL_PSTQX_SHIFT(queue)) &
+ GMAC_TXQCTRL_PSTQX_MASK(queue);
+
+ writel(value, ioaddr + base_register);
+}
+
+static void dwmac4_tx_queue_routing(struct mac_device_info *hw,
+ u8 packet, u32 queue)
+{
+ void __iomem *ioaddr = hw->pcsr;
+ u32 value;
+
+ const struct stmmac_rx_routing route_possibilities[] = {
+ { GMAC_RXQCTRL_AVCPQ_MASK, GMAC_RXQCTRL_AVCPQ_SHIFT },
+ { GMAC_RXQCTRL_PTPQ_MASK, GMAC_RXQCTRL_PTPQ_SHIFT },
+ { GMAC_RXQCTRL_DCBCPQ_MASK, GMAC_RXQCTRL_DCBCPQ_SHIFT },
+ { GMAC_RXQCTRL_UPQ_MASK, GMAC_RXQCTRL_UPQ_SHIFT },
+ { GMAC_RXQCTRL_MCBCQ_MASK, GMAC_RXQCTRL_MCBCQ_SHIFT },
+ };
+
+ value = readl(ioaddr + GMAC_RXQ_CTRL1);
+
+ /* routing configuration */
+ value &= ~route_possibilities[packet - 1].reg_mask;
+ value |= (queue << route_possibilities[packet-1].reg_shift) &
+ route_possibilities[packet - 1].reg_mask;
+
+ /* some packets require extra ops */
+ if (packet == PACKET_AVCPQ) {
+ value &= ~GMAC_RXQCTRL_TACPQE;
+ value |= 0x1 << GMAC_RXQCTRL_TACPQE_SHIFT;
+ } else if (packet == PACKET_MCBCQ) {
+ value &= ~GMAC_RXQCTRL_MCBCQEN;
+ value |= 0x1 << GMAC_RXQCTRL_MCBCQEN_SHIFT;
+ }
+
+ writel(value, ioaddr + GMAC_RXQ_CTRL1);
+}
+
+static void dwmac4_prog_mtl_rx_algorithms(struct mac_device_info *hw,
+ u32 rx_alg)
+{
+ void __iomem *ioaddr = hw->pcsr;
+ u32 value = readl(ioaddr + MTL_OPERATION_MODE);
+
+ value &= ~MTL_OPERATION_RAA;
+ switch (rx_alg) {
+ case MTL_RX_ALGORITHM_SP:
+ value |= MTL_OPERATION_RAA_SP;
+ break;
+ case MTL_RX_ALGORITHM_WSP:
+ value |= MTL_OPERATION_RAA_WSP;
+ break;
+ default:
+ break;
+ }
+
+ writel(value, ioaddr + MTL_OPERATION_MODE);
+}
+
+static void dwmac4_prog_mtl_tx_algorithms(struct mac_device_info *hw,
+ u32 tx_alg)
+{
+ void __iomem *ioaddr = hw->pcsr;
+ u32 value = readl(ioaddr + MTL_OPERATION_MODE);
+
+ value &= ~MTL_OPERATION_SCHALG_MASK;
+ switch (tx_alg) {
+ case MTL_TX_ALGORITHM_WRR:
+ value |= MTL_OPERATION_SCHALG_WRR;
+ break;
+ case MTL_TX_ALGORITHM_WFQ:
+ value |= MTL_OPERATION_SCHALG_WFQ;
+ break;
+ case MTL_TX_ALGORITHM_DWRR:
+ value |= MTL_OPERATION_SCHALG_DWRR;
+ break;
+ case MTL_TX_ALGORITHM_SP:
+ value |= MTL_OPERATION_SCHALG_SP;
+ break;
+ default:
+ break;
+ }
+}
+
+static void dwmac4_set_mtl_tx_queue_weight(struct mac_device_info *hw,
+ u32 weight, u32 queue)
+{
+ void __iomem *ioaddr = hw->pcsr;
+ u32 value = readl(ioaddr + MTL_TXQX_WEIGHT_BASE_ADDR(queue));
+
+ value &= ~MTL_TXQ_WEIGHT_ISCQW_MASK;
+ value |= weight & MTL_TXQ_WEIGHT_ISCQW_MASK;
+ writel(value, ioaddr + MTL_TXQX_WEIGHT_BASE_ADDR(queue));
+}
+
+static void dwmac4_map_mtl_dma(struct mac_device_info *hw, u32 queue, u32 chan)
+{
+ void __iomem *ioaddr = hw->pcsr;
+ u32 value;
+
+ if (queue < 4)
+ value = readl(ioaddr + MTL_RXQ_DMA_MAP0);
+ else
+ value = readl(ioaddr + MTL_RXQ_DMA_MAP1);
+
+ if (queue == 0 || queue == 4) {
+ value &= ~MTL_RXQ_DMA_Q04MDMACH_MASK;
+ value |= MTL_RXQ_DMA_Q04MDMACH(chan);
+ } else {
+ value &= ~MTL_RXQ_DMA_QXMDMACH_MASK(queue);
+ value |= MTL_RXQ_DMA_QXMDMACH(chan, queue);
+ }
+
+ if (queue < 4)
+ writel(value, ioaddr + MTL_RXQ_DMA_MAP0);
+ else
+ writel(value, ioaddr + MTL_RXQ_DMA_MAP1);
+}
+
+static void dwmac4_config_cbs(struct mac_device_info *hw,
+ u32 send_slope, u32 idle_slope,
+ u32 high_credit, u32 low_credit, u32 queue)
+{
+ void __iomem *ioaddr = hw->pcsr;
+ u32 value;
+
+ pr_debug("Queue %d configured as AVB. Parameters:\n", queue);
+ pr_debug("\tsend_slope: 0x%08x\n", send_slope);
+ pr_debug("\tidle_slope: 0x%08x\n", idle_slope);
+ pr_debug("\thigh_credit: 0x%08x\n", high_credit);
+ pr_debug("\tlow_credit: 0x%08x\n", low_credit);
+
+ /* enable AV algorithm */
+ value = readl(ioaddr + MTL_ETSX_CTRL_BASE_ADDR(queue));
+ value |= MTL_ETS_CTRL_AVALG;
+ value |= MTL_ETS_CTRL_CC;
+ writel(value, ioaddr + MTL_ETSX_CTRL_BASE_ADDR(queue));
+
+ /* configure send slope */
+ value = readl(ioaddr + MTL_SEND_SLP_CREDX_BASE_ADDR(queue));
+ value &= ~MTL_SEND_SLP_CRED_SSC_MASK;
+ value |= send_slope & MTL_SEND_SLP_CRED_SSC_MASK;
+ writel(value, ioaddr + MTL_SEND_SLP_CREDX_BASE_ADDR(queue));
+
+ /* configure idle slope (same register as tx weight) */
+ dwmac4_set_mtl_tx_queue_weight(hw, idle_slope, queue);
+
+ /* configure high credit */
+ value = readl(ioaddr + MTL_HIGH_CREDX_BASE_ADDR(queue));
+ value &= ~MTL_HIGH_CRED_HC_MASK;
+ value |= high_credit & MTL_HIGH_CRED_HC_MASK;
+ writel(value, ioaddr + MTL_HIGH_CREDX_BASE_ADDR(queue));
+
+ /* configure high credit */
+ value = readl(ioaddr + MTL_LOW_CREDX_BASE_ADDR(queue));
+ value &= ~MTL_HIGH_CRED_LC_MASK;
+ value |= low_credit & MTL_HIGH_CRED_LC_MASK;
+ writel(value, ioaddr + MTL_LOW_CREDX_BASE_ADDR(queue));
+}
+
static void dwmac4_dump_regs(struct mac_device_info *hw, u32 *reg_space)
{
void __iomem *ioaddr = hw->pcsr;
@@ -251,11 +445,12 @@
}
static void dwmac4_flow_ctrl(struct mac_device_info *hw, unsigned int duplex,
- unsigned int fc, unsigned int pause_time)
+ unsigned int fc, unsigned int pause_time,
+ u32 tx_cnt)
{
void __iomem *ioaddr = hw->pcsr;
- u32 channel = STMMAC_CHAN0; /* FIXME */
unsigned int flow = 0;
+ u32 queue = 0;
pr_debug("GMAC Flow-Control:\n");
if (fc & FLOW_RX) {
@@ -265,13 +460,18 @@
}
if (fc & FLOW_TX) {
pr_debug("\tTransmit Flow-Control ON\n");
- flow |= GMAC_TX_FLOW_CTRL_TFE;
- writel(flow, ioaddr + GMAC_QX_TX_FLOW_CTRL(channel));
- if (duplex) {
+ if (duplex)
pr_debug("\tduplex mode: PAUSE %d\n", pause_time);
- flow |= (pause_time << GMAC_TX_FLOW_CTRL_PT_SHIFT);
- writel(flow, ioaddr + GMAC_QX_TX_FLOW_CTRL(channel));
+
+ for (queue = 0; queue < tx_cnt; queue++) {
+ flow |= GMAC_TX_FLOW_CTRL_TFE;
+
+ if (duplex)
+ flow |=
+ (pause_time << GMAC_TX_FLOW_CTRL_PT_SHIFT);
+
+ writel(flow, ioaddr + GMAC_QX_TX_FLOW_CTRL(queue));
}
}
}
@@ -325,11 +525,34 @@
}
}
+static int dwmac4_irq_mtl_status(struct mac_device_info *hw, u32 chan)
+{
+ void __iomem *ioaddr = hw->pcsr;
+ u32 mtl_int_qx_status;
+ int ret = 0;
+
+ mtl_int_qx_status = readl(ioaddr + MTL_INT_STATUS);
+
+ /* Check MTL Interrupt */
+ if (mtl_int_qx_status & MTL_INT_QX(chan)) {
+ /* read Queue x Interrupt status */
+ u32 status = readl(ioaddr + MTL_CHAN_INT_CTRL(chan));
+
+ if (status & MTL_RX_OVERFLOW_INT) {
+ /* clear Interrupt */
+ writel(status | MTL_RX_OVERFLOW_INT,
+ ioaddr + MTL_CHAN_INT_CTRL(chan));
+ ret = CORE_IRQ_MTL_RX_OVERFLOW;
+ }
+ }
+
+ return ret;
+}
+
static int dwmac4_irq_status(struct mac_device_info *hw,
struct stmmac_extra_stats *x)
{
void __iomem *ioaddr = hw->pcsr;
- u32 mtl_int_qx_status;
u32 intr_status;
int ret = 0;
@@ -348,20 +571,6 @@
x->irq_receive_pmt_irq_n++;
}
- mtl_int_qx_status = readl(ioaddr + MTL_INT_STATUS);
- /* Check MTL Interrupt: Currently only one queue is used: Q0. */
- if (mtl_int_qx_status & MTL_INT_Q0) {
- /* read Queue 0 Interrupt status */
- u32 status = readl(ioaddr + MTL_CHAN_INT_CTRL(STMMAC_CHAN0));
-
- if (status & MTL_RX_OVERFLOW_INT) {
- /* clear Interrupt */
- writel(status | MTL_RX_OVERFLOW_INT,
- ioaddr + MTL_CHAN_INT_CTRL(STMMAC_CHAN0));
- ret = CORE_IRQ_MTL_RX_OVERFLOW;
- }
- }
-
dwmac_pcs_isr(ioaddr, GMAC_PCS_BASE, intr_status, x);
if (intr_status & PCS_RGSMIIIS_IRQ)
dwmac4_phystatus(ioaddr, x);
@@ -369,64 +578,69 @@
return ret;
}
-static void dwmac4_debug(void __iomem *ioaddr, struct stmmac_extra_stats *x)
+static void dwmac4_debug(void __iomem *ioaddr, struct stmmac_extra_stats *x,
+ u32 rx_queues, u32 tx_queues)
{
u32 value;
+ u32 queue;
- /* Currently only channel 0 is supported */
- value = readl(ioaddr + MTL_CHAN_TX_DEBUG(STMMAC_CHAN0));
+ for (queue = 0; queue < tx_queues; queue++) {
+ value = readl(ioaddr + MTL_CHAN_TX_DEBUG(queue));
- if (value & MTL_DEBUG_TXSTSFSTS)
- x->mtl_tx_status_fifo_full++;
- if (value & MTL_DEBUG_TXFSTS)
- x->mtl_tx_fifo_not_empty++;
- if (value & MTL_DEBUG_TWCSTS)
- x->mmtl_fifo_ctrl++;
- if (value & MTL_DEBUG_TRCSTS_MASK) {
- u32 trcsts = (value & MTL_DEBUG_TRCSTS_MASK)
- >> MTL_DEBUG_TRCSTS_SHIFT;
- if (trcsts == MTL_DEBUG_TRCSTS_WRITE)
- x->mtl_tx_fifo_read_ctrl_write++;
- else if (trcsts == MTL_DEBUG_TRCSTS_TXW)
- x->mtl_tx_fifo_read_ctrl_wait++;
- else if (trcsts == MTL_DEBUG_TRCSTS_READ)
- x->mtl_tx_fifo_read_ctrl_read++;
- else
- x->mtl_tx_fifo_read_ctrl_idle++;
+ if (value & MTL_DEBUG_TXSTSFSTS)
+ x->mtl_tx_status_fifo_full++;
+ if (value & MTL_DEBUG_TXFSTS)
+ x->mtl_tx_fifo_not_empty++;
+ if (value & MTL_DEBUG_TWCSTS)
+ x->mmtl_fifo_ctrl++;
+ if (value & MTL_DEBUG_TRCSTS_MASK) {
+ u32 trcsts = (value & MTL_DEBUG_TRCSTS_MASK)
+ >> MTL_DEBUG_TRCSTS_SHIFT;
+ if (trcsts == MTL_DEBUG_TRCSTS_WRITE)
+ x->mtl_tx_fifo_read_ctrl_write++;
+ else if (trcsts == MTL_DEBUG_TRCSTS_TXW)
+ x->mtl_tx_fifo_read_ctrl_wait++;
+ else if (trcsts == MTL_DEBUG_TRCSTS_READ)
+ x->mtl_tx_fifo_read_ctrl_read++;
+ else
+ x->mtl_tx_fifo_read_ctrl_idle++;
+ }
+ if (value & MTL_DEBUG_TXPAUSED)
+ x->mac_tx_in_pause++;
}
- if (value & MTL_DEBUG_TXPAUSED)
- x->mac_tx_in_pause++;
- value = readl(ioaddr + MTL_CHAN_RX_DEBUG(STMMAC_CHAN0));
+ for (queue = 0; queue < rx_queues; queue++) {
+ value = readl(ioaddr + MTL_CHAN_RX_DEBUG(queue));
- if (value & MTL_DEBUG_RXFSTS_MASK) {
- u32 rxfsts = (value & MTL_DEBUG_RXFSTS_MASK)
- >> MTL_DEBUG_RRCSTS_SHIFT;
+ if (value & MTL_DEBUG_RXFSTS_MASK) {
+ u32 rxfsts = (value & MTL_DEBUG_RXFSTS_MASK)
+ >> MTL_DEBUG_RRCSTS_SHIFT;
- if (rxfsts == MTL_DEBUG_RXFSTS_FULL)
- x->mtl_rx_fifo_fill_level_full++;
- else if (rxfsts == MTL_DEBUG_RXFSTS_AT)
- x->mtl_rx_fifo_fill_above_thresh++;
- else if (rxfsts == MTL_DEBUG_RXFSTS_BT)
- x->mtl_rx_fifo_fill_below_thresh++;
- else
- x->mtl_rx_fifo_fill_level_empty++;
+ if (rxfsts == MTL_DEBUG_RXFSTS_FULL)
+ x->mtl_rx_fifo_fill_level_full++;
+ else if (rxfsts == MTL_DEBUG_RXFSTS_AT)
+ x->mtl_rx_fifo_fill_above_thresh++;
+ else if (rxfsts == MTL_DEBUG_RXFSTS_BT)
+ x->mtl_rx_fifo_fill_below_thresh++;
+ else
+ x->mtl_rx_fifo_fill_level_empty++;
+ }
+ if (value & MTL_DEBUG_RRCSTS_MASK) {
+ u32 rrcsts = (value & MTL_DEBUG_RRCSTS_MASK) >>
+ MTL_DEBUG_RRCSTS_SHIFT;
+
+ if (rrcsts == MTL_DEBUG_RRCSTS_FLUSH)
+ x->mtl_rx_fifo_read_ctrl_flush++;
+ else if (rrcsts == MTL_DEBUG_RRCSTS_RSTAT)
+ x->mtl_rx_fifo_read_ctrl_read_data++;
+ else if (rrcsts == MTL_DEBUG_RRCSTS_RDATA)
+ x->mtl_rx_fifo_read_ctrl_status++;
+ else
+ x->mtl_rx_fifo_read_ctrl_idle++;
+ }
+ if (value & MTL_DEBUG_RWCSTS)
+ x->mtl_rx_fifo_ctrl_active++;
}
- if (value & MTL_DEBUG_RRCSTS_MASK) {
- u32 rrcsts = (value & MTL_DEBUG_RRCSTS_MASK) >>
- MTL_DEBUG_RRCSTS_SHIFT;
-
- if (rrcsts == MTL_DEBUG_RRCSTS_FLUSH)
- x->mtl_rx_fifo_read_ctrl_flush++;
- else if (rrcsts == MTL_DEBUG_RRCSTS_RSTAT)
- x->mtl_rx_fifo_read_ctrl_read_data++;
- else if (rrcsts == MTL_DEBUG_RRCSTS_RDATA)
- x->mtl_rx_fifo_read_ctrl_status++;
- else
- x->mtl_rx_fifo_read_ctrl_idle++;
- }
- if (value & MTL_DEBUG_RWCSTS)
- x->mtl_rx_fifo_ctrl_active++;
/* GMAC debug */
value = readl(ioaddr + GMAC_DEBUG);
@@ -455,10 +669,51 @@
static const struct stmmac_ops dwmac4_ops = {
.core_init = dwmac4_core_init,
+ .set_mac = stmmac_set_mac,
.rx_ipc = dwmac4_rx_ipc_enable,
.rx_queue_enable = dwmac4_rx_queue_enable,
+ .rx_queue_prio = dwmac4_rx_queue_priority,
+ .tx_queue_prio = dwmac4_tx_queue_priority,
+ .rx_queue_routing = dwmac4_tx_queue_routing,
+ .prog_mtl_rx_algorithms = dwmac4_prog_mtl_rx_algorithms,
+ .prog_mtl_tx_algorithms = dwmac4_prog_mtl_tx_algorithms,
+ .set_mtl_tx_queue_weight = dwmac4_set_mtl_tx_queue_weight,
+ .map_mtl_to_dma = dwmac4_map_mtl_dma,
+ .config_cbs = dwmac4_config_cbs,
.dump_regs = dwmac4_dump_regs,
.host_irq_status = dwmac4_irq_status,
+ .host_mtl_irq_status = dwmac4_irq_mtl_status,
+ .flow_ctrl = dwmac4_flow_ctrl,
+ .pmt = dwmac4_pmt,
+ .set_umac_addr = dwmac4_set_umac_addr,
+ .get_umac_addr = dwmac4_get_umac_addr,
+ .set_eee_mode = dwmac4_set_eee_mode,
+ .reset_eee_mode = dwmac4_reset_eee_mode,
+ .set_eee_timer = dwmac4_set_eee_timer,
+ .set_eee_pls = dwmac4_set_eee_pls,
+ .pcs_ctrl_ane = dwmac4_ctrl_ane,
+ .pcs_rane = dwmac4_rane,
+ .pcs_get_adv_lp = dwmac4_get_adv_lp,
+ .debug = dwmac4_debug,
+ .set_filter = dwmac4_set_filter,
+};
+
+static const struct stmmac_ops dwmac410_ops = {
+ .core_init = dwmac4_core_init,
+ .set_mac = stmmac_dwmac4_set_mac,
+ .rx_ipc = dwmac4_rx_ipc_enable,
+ .rx_queue_enable = dwmac4_rx_queue_enable,
+ .rx_queue_prio = dwmac4_rx_queue_priority,
+ .tx_queue_prio = dwmac4_tx_queue_priority,
+ .rx_queue_routing = dwmac4_tx_queue_routing,
+ .prog_mtl_rx_algorithms = dwmac4_prog_mtl_rx_algorithms,
+ .prog_mtl_tx_algorithms = dwmac4_prog_mtl_tx_algorithms,
+ .set_mtl_tx_queue_weight = dwmac4_set_mtl_tx_queue_weight,
+ .map_mtl_to_dma = dwmac4_map_mtl_dma,
+ .config_cbs = dwmac4_config_cbs,
+ .dump_regs = dwmac4_dump_regs,
+ .host_irq_status = dwmac4_irq_status,
+ .host_mtl_irq_status = dwmac4_irq_mtl_status,
.flow_ctrl = dwmac4_flow_ctrl,
.pmt = dwmac4_pmt,
.set_umac_addr = dwmac4_set_umac_addr,
@@ -492,8 +747,6 @@
if (mac->multicast_filter_bins)
mac->mcast_bits_log2 = ilog2(mac->multicast_filter_bins);
- mac->mac = &dwmac4_ops;
-
mac->link.port = GMAC_CONFIG_PS;
mac->link.duplex = GMAC_CONFIG_DM;
mac->link.speed = GMAC_CONFIG_FES;
@@ -514,5 +767,10 @@
else
mac->dma = &dwmac4_dma_ops;
+ if (*synopsys_id >= DWMAC_CORE_4_00)
+ mac->mac = &dwmac410_ops;
+ else
+ mac->mac = &dwmac4_ops;
+
return mac;
}
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c b/drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c
index 843ec69..aa64764 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c
@@ -304,12 +304,13 @@
static void dwmac4_rd_prepare_tx_desc(struct dma_desc *p, int is_fs, int len,
bool csum_flag, int mode, bool tx_own,
- bool ls)
+ bool ls, unsigned int tot_pkt_len)
{
unsigned int tdes3 = le32_to_cpu(p->des3);
p->des2 |= cpu_to_le32(len & TDES2_BUFFER1_SIZE_MASK);
+ tdes3 |= tot_pkt_len & TDES3_PACKET_SIZE_MASK;
if (is_fs)
tdes3 |= TDES3_FIRST_DESCRIPTOR;
else
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac4_dma.c b/drivers/net/ethernet/stmicro/stmmac/dwmac4_dma.c
index f97b0d5d..eec8463 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac4_dma.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac4_dma.c
@@ -71,36 +71,48 @@
writel(value, ioaddr + DMA_SYS_BUS_MODE);
}
-static void dwmac4_dma_init_channel(void __iomem *ioaddr,
- struct stmmac_dma_cfg *dma_cfg,
- u32 dma_tx_phy, u32 dma_rx_phy,
- u32 channel)
+void dwmac4_dma_init_rx_chan(void __iomem *ioaddr,
+ struct stmmac_dma_cfg *dma_cfg,
+ u32 dma_rx_phy, u32 chan)
{
u32 value;
- int txpbl = dma_cfg->txpbl ?: dma_cfg->pbl;
- int rxpbl = dma_cfg->rxpbl ?: dma_cfg->pbl;
+ u32 rxpbl = dma_cfg->rxpbl ?: dma_cfg->pbl;
- /* set PBL for each channels. Currently we affect same configuration
- * on each channel
- */
- value = readl(ioaddr + DMA_CHAN_CONTROL(channel));
+ value = readl(ioaddr + DMA_CHAN_RX_CONTROL(chan));
+ value = value | (rxpbl << DMA_BUS_MODE_RPBL_SHIFT);
+ writel(value, ioaddr + DMA_CHAN_RX_CONTROL(chan));
+
+ writel(dma_rx_phy, ioaddr + DMA_CHAN_RX_BASE_ADDR(chan));
+}
+
+void dwmac4_dma_init_tx_chan(void __iomem *ioaddr,
+ struct stmmac_dma_cfg *dma_cfg,
+ u32 dma_tx_phy, u32 chan)
+{
+ u32 value;
+ u32 txpbl = dma_cfg->txpbl ?: dma_cfg->pbl;
+
+ value = readl(ioaddr + DMA_CHAN_TX_CONTROL(chan));
+ value = value | (txpbl << DMA_BUS_MODE_PBL_SHIFT);
+ writel(value, ioaddr + DMA_CHAN_TX_CONTROL(chan));
+
+ writel(dma_tx_phy, ioaddr + DMA_CHAN_TX_BASE_ADDR(chan));
+}
+
+void dwmac4_dma_init_channel(void __iomem *ioaddr,
+ struct stmmac_dma_cfg *dma_cfg, u32 chan)
+{
+ u32 value;
+
+ /* common channel control register config */
+ value = readl(ioaddr + DMA_CHAN_CONTROL(chan));
if (dma_cfg->pblx8)
value = value | DMA_BUS_MODE_PBL;
- writel(value, ioaddr + DMA_CHAN_CONTROL(channel));
-
- value = readl(ioaddr + DMA_CHAN_TX_CONTROL(channel));
- value = value | (txpbl << DMA_BUS_MODE_PBL_SHIFT);
- writel(value, ioaddr + DMA_CHAN_TX_CONTROL(channel));
-
- value = readl(ioaddr + DMA_CHAN_RX_CONTROL(channel));
- value = value | (rxpbl << DMA_BUS_MODE_RPBL_SHIFT);
- writel(value, ioaddr + DMA_CHAN_RX_CONTROL(channel));
+ writel(value, ioaddr + DMA_CHAN_CONTROL(chan));
/* Mask interrupts by writing to CSR7 */
- writel(DMA_CHAN_INTR_DEFAULT_MASK, ioaddr + DMA_CHAN_INTR_ENA(channel));
-
- writel(dma_tx_phy, ioaddr + DMA_CHAN_TX_BASE_ADDR(channel));
- writel(dma_rx_phy, ioaddr + DMA_CHAN_RX_BASE_ADDR(channel));
+ writel(DMA_CHAN_INTR_DEFAULT_MASK,
+ ioaddr + DMA_CHAN_INTR_ENA(chan));
}
static void dwmac4_dma_init(void __iomem *ioaddr,
@@ -108,7 +120,6 @@
u32 dma_tx, u32 dma_rx, int atds)
{
u32 value = readl(ioaddr + DMA_SYS_BUS_MODE);
- int i;
/* Set the Fixed burst mode */
if (dma_cfg->fixed_burst)
@@ -122,9 +133,6 @@
value |= DMA_SYS_BUS_AAL;
writel(value, ioaddr + DMA_SYS_BUS_MODE);
-
- for (i = 0; i < DMA_CHANNEL_NB_MAX; i++)
- dwmac4_dma_init_channel(ioaddr, dma_cfg, dma_tx, dma_rx, i);
}
static void _dwmac4_dump_dma_regs(void __iomem *ioaddr, u32 channel,
@@ -174,46 +182,121 @@
_dwmac4_dump_dma_regs(ioaddr, i, reg_space);
}
-static void dwmac4_rx_watchdog(void __iomem *ioaddr, u32 riwt)
+static void dwmac4_rx_watchdog(void __iomem *ioaddr, u32 riwt, u32 number_chan)
{
- int i;
+ u32 chan;
- for (i = 0; i < DMA_CHANNEL_NB_MAX; i++)
- writel(riwt, ioaddr + DMA_CHAN_RX_WATCHDOG(i));
+ for (chan = 0; chan < number_chan; chan++)
+ writel(riwt, ioaddr + DMA_CHAN_RX_WATCHDOG(chan));
}
-static void dwmac4_dma_chan_op_mode(void __iomem *ioaddr, int txmode,
- int rxmode, u32 channel)
+static void dwmac4_dma_rx_chan_op_mode(void __iomem *ioaddr, int mode,
+ u32 channel, int fifosz)
{
- u32 mtl_tx_op, mtl_rx_op, mtl_rx_int;
+ unsigned int rqs = fifosz / 256 - 1;
+ u32 mtl_rx_op, mtl_rx_int;
- /* Following code only done for channel 0, other channels not yet
- * supported.
- */
- mtl_tx_op = readl(ioaddr + MTL_CHAN_TX_OP_MODE(channel));
+ mtl_rx_op = readl(ioaddr + MTL_CHAN_RX_OP_MODE(channel));
- if (txmode == SF_DMA_MODE) {
+ if (mode == SF_DMA_MODE) {
+ pr_debug("GMAC: enable RX store and forward mode\n");
+ mtl_rx_op |= MTL_OP_MODE_RSF;
+ } else {
+ pr_debug("GMAC: disable RX SF mode (threshold %d)\n", mode);
+ mtl_rx_op &= ~MTL_OP_MODE_RSF;
+ mtl_rx_op &= MTL_OP_MODE_RTC_MASK;
+ if (mode <= 32)
+ mtl_rx_op |= MTL_OP_MODE_RTC_32;
+ else if (mode <= 64)
+ mtl_rx_op |= MTL_OP_MODE_RTC_64;
+ else if (mode <= 96)
+ mtl_rx_op |= MTL_OP_MODE_RTC_96;
+ else
+ mtl_rx_op |= MTL_OP_MODE_RTC_128;
+ }
+
+ mtl_rx_op &= ~MTL_OP_MODE_RQS_MASK;
+ mtl_rx_op |= rqs << MTL_OP_MODE_RQS_SHIFT;
+
+ /* enable flow control only if each channel gets 4 KiB or more FIFO */
+ if (fifosz >= 4096) {
+ unsigned int rfd, rfa;
+
+ mtl_rx_op |= MTL_OP_MODE_EHFC;
+
+ /* Set Threshold for Activating Flow Control to min 2 frames,
+ * i.e. 1500 * 2 = 3000 bytes.
+ *
+ * Set Threshold for Deactivating Flow Control to min 1 frame,
+ * i.e. 1500 bytes.
+ */
+ switch (fifosz) {
+ case 4096:
+ /* This violates the above formula because of FIFO size
+ * limit therefore overflow may occur in spite of this.
+ */
+ rfd = 0x03; /* Full-2.5K */
+ rfa = 0x01; /* Full-1.5K */
+ break;
+
+ case 8192:
+ rfd = 0x06; /* Full-4K */
+ rfa = 0x0a; /* Full-6K */
+ break;
+
+ case 16384:
+ rfd = 0x06; /* Full-4K */
+ rfa = 0x12; /* Full-10K */
+ break;
+
+ default:
+ rfd = 0x06; /* Full-4K */
+ rfa = 0x1e; /* Full-16K */
+ break;
+ }
+
+ mtl_rx_op &= ~MTL_OP_MODE_RFD_MASK;
+ mtl_rx_op |= rfd << MTL_OP_MODE_RFD_SHIFT;
+
+ mtl_rx_op &= ~MTL_OP_MODE_RFA_MASK;
+ mtl_rx_op |= rfa << MTL_OP_MODE_RFA_SHIFT;
+ }
+
+ writel(mtl_rx_op, ioaddr + MTL_CHAN_RX_OP_MODE(channel));
+
+ /* Enable MTL RX overflow */
+ mtl_rx_int = readl(ioaddr + MTL_CHAN_INT_CTRL(channel));
+ writel(mtl_rx_int | MTL_RX_OVERFLOW_INT_EN,
+ ioaddr + MTL_CHAN_INT_CTRL(channel));
+}
+
+static void dwmac4_dma_tx_chan_op_mode(void __iomem *ioaddr, int mode,
+ u32 channel)
+{
+ u32 mtl_tx_op = readl(ioaddr + MTL_CHAN_TX_OP_MODE(channel));
+
+ if (mode == SF_DMA_MODE) {
pr_debug("GMAC: enable TX store and forward mode\n");
/* Transmit COE type 2 cannot be done in cut-through mode. */
mtl_tx_op |= MTL_OP_MODE_TSF;
} else {
- pr_debug("GMAC: disabling TX SF (threshold %d)\n", txmode);
+ pr_debug("GMAC: disabling TX SF (threshold %d)\n", mode);
mtl_tx_op &= ~MTL_OP_MODE_TSF;
mtl_tx_op &= MTL_OP_MODE_TTC_MASK;
/* Set the transmit threshold */
- if (txmode <= 32)
+ if (mode <= 32)
mtl_tx_op |= MTL_OP_MODE_TTC_32;
- else if (txmode <= 64)
+ else if (mode <= 64)
mtl_tx_op |= MTL_OP_MODE_TTC_64;
- else if (txmode <= 96)
+ else if (mode <= 96)
mtl_tx_op |= MTL_OP_MODE_TTC_96;
- else if (txmode <= 128)
+ else if (mode <= 128)
mtl_tx_op |= MTL_OP_MODE_TTC_128;
- else if (txmode <= 192)
+ else if (mode <= 192)
mtl_tx_op |= MTL_OP_MODE_TTC_192;
- else if (txmode <= 256)
+ else if (mode <= 256)
mtl_tx_op |= MTL_OP_MODE_TTC_256;
- else if (txmode <= 384)
+ else if (mode <= 384)
mtl_tx_op |= MTL_OP_MODE_TTC_384;
else
mtl_tx_op |= MTL_OP_MODE_TTC_512;
@@ -230,39 +313,6 @@
*/
mtl_tx_op |= MTL_OP_MODE_TXQEN | MTL_OP_MODE_TQS_MASK;
writel(mtl_tx_op, ioaddr + MTL_CHAN_TX_OP_MODE(channel));
-
- mtl_rx_op = readl(ioaddr + MTL_CHAN_RX_OP_MODE(channel));
-
- if (rxmode == SF_DMA_MODE) {
- pr_debug("GMAC: enable RX store and forward mode\n");
- mtl_rx_op |= MTL_OP_MODE_RSF;
- } else {
- pr_debug("GMAC: disable RX SF mode (threshold %d)\n", rxmode);
- mtl_rx_op &= ~MTL_OP_MODE_RSF;
- mtl_rx_op &= MTL_OP_MODE_RTC_MASK;
- if (rxmode <= 32)
- mtl_rx_op |= MTL_OP_MODE_RTC_32;
- else if (rxmode <= 64)
- mtl_rx_op |= MTL_OP_MODE_RTC_64;
- else if (rxmode <= 96)
- mtl_rx_op |= MTL_OP_MODE_RTC_96;
- else
- mtl_rx_op |= MTL_OP_MODE_RTC_128;
- }
-
- writel(mtl_rx_op, ioaddr + MTL_CHAN_RX_OP_MODE(channel));
-
- /* Enable MTL RX overflow */
- mtl_rx_int = readl(ioaddr + MTL_CHAN_INT_CTRL(channel));
- writel(mtl_rx_int | MTL_RX_OVERFLOW_INT_EN,
- ioaddr + MTL_CHAN_INT_CTRL(channel));
-}
-
-static void dwmac4_dma_operation_mode(void __iomem *ioaddr, int txmode,
- int rxmode, int rxfifosz)
-{
- /* Only Channel 0 is actually configured and used */
- dwmac4_dma_chan_op_mode(ioaddr, txmode, rxmode, 0);
}
static void dwmac4_get_hw_feature(void __iomem *ioaddr,
@@ -294,6 +344,11 @@
hw_cap = readl(ioaddr + GMAC_HW_FEATURE1);
dma_cap->av = (hw_cap & GMAC_HW_FEAT_AVSEL) >> 20;
dma_cap->tsoen = (hw_cap & GMAC_HW_TSOEN) >> 18;
+ /* RX and TX FIFO sizes are encoded as log2(n / 128). Undo that by
+ * shifting and store the sizes in bytes.
+ */
+ dma_cap->tx_fifo_size = 128 << ((hw_cap & GMAC_HW_TXFIFOSIZE) >> 6);
+ dma_cap->rx_fifo_size = 128 << ((hw_cap & GMAC_HW_RXFIFOSIZE) >> 0);
/* MAC HW feature2 */
hw_cap = readl(ioaddr + GMAC_HW_FEATURE2);
/* TX and RX number of channels */
@@ -332,9 +387,13 @@
const struct stmmac_dma_ops dwmac4_dma_ops = {
.reset = dwmac4_dma_reset,
.init = dwmac4_dma_init,
+ .init_chan = dwmac4_dma_init_channel,
+ .init_rx_chan = dwmac4_dma_init_rx_chan,
+ .init_tx_chan = dwmac4_dma_init_tx_chan,
.axi = dwmac4_dma_axi,
.dump_regs = dwmac4_dump_dma_regs,
- .dma_mode = dwmac4_dma_operation_mode,
+ .dma_rx_mode = dwmac4_dma_rx_chan_op_mode,
+ .dma_tx_mode = dwmac4_dma_tx_chan_op_mode,
.enable_dma_irq = dwmac4_enable_dma_irq,
.disable_dma_irq = dwmac4_disable_dma_irq,
.start_tx = dwmac4_dma_start_tx,
@@ -354,9 +413,13 @@
const struct stmmac_dma_ops dwmac410_dma_ops = {
.reset = dwmac4_dma_reset,
.init = dwmac4_dma_init,
+ .init_chan = dwmac4_dma_init_channel,
+ .init_rx_chan = dwmac4_dma_init_rx_chan,
+ .init_tx_chan = dwmac4_dma_init_tx_chan,
.axi = dwmac4_dma_axi,
.dump_regs = dwmac4_dump_dma_regs,
- .dma_mode = dwmac4_dma_operation_mode,
+ .dma_rx_mode = dwmac4_dma_rx_chan_op_mode,
+ .dma_tx_mode = dwmac4_dma_tx_chan_op_mode,
.enable_dma_irq = dwmac410_enable_dma_irq,
.disable_dma_irq = dwmac4_disable_dma_irq,
.start_tx = dwmac4_dma_start_tx,
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac4_dma.h b/drivers/net/ethernet/stmicro/stmmac/dwmac4_dma.h
index 1b06df7..8474bf9 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac4_dma.h
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac4_dma.h
@@ -185,17 +185,17 @@
int dwmac4_dma_reset(void __iomem *ioaddr);
void dwmac4_enable_dma_transmission(void __iomem *ioaddr, u32 tail_ptr);
-void dwmac4_enable_dma_irq(void __iomem *ioaddr);
-void dwmac410_enable_dma_irq(void __iomem *ioaddr);
-void dwmac4_disable_dma_irq(void __iomem *ioaddr);
-void dwmac4_dma_start_tx(void __iomem *ioaddr);
-void dwmac4_dma_stop_tx(void __iomem *ioaddr);
-void dwmac4_dma_start_rx(void __iomem *ioaddr);
-void dwmac4_dma_stop_rx(void __iomem *ioaddr);
+void dwmac4_enable_dma_irq(void __iomem *ioaddr, u32 chan);
+void dwmac410_enable_dma_irq(void __iomem *ioaddr, u32 chan);
+void dwmac4_disable_dma_irq(void __iomem *ioaddr, u32 chan);
+void dwmac4_dma_start_tx(void __iomem *ioaddr, u32 chan);
+void dwmac4_dma_stop_tx(void __iomem *ioaddr, u32 chan);
+void dwmac4_dma_start_rx(void __iomem *ioaddr, u32 chan);
+void dwmac4_dma_stop_rx(void __iomem *ioaddr, u32 chan);
int dwmac4_dma_interrupt(void __iomem *ioaddr,
- struct stmmac_extra_stats *x);
-void dwmac4_set_rx_ring_len(void __iomem *ioaddr, u32 len);
-void dwmac4_set_tx_ring_len(void __iomem *ioaddr, u32 len);
+ struct stmmac_extra_stats *x, u32 chan);
+void dwmac4_set_rx_ring_len(void __iomem *ioaddr, u32 len, u32 chan);
+void dwmac4_set_tx_ring_len(void __iomem *ioaddr, u32 len, u32 chan);
void dwmac4_set_rx_tail_ptr(void __iomem *ioaddr, u32 tail_ptr, u32 chan);
void dwmac4_set_tx_tail_ptr(void __iomem *ioaddr, u32 tail_ptr, u32 chan);
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac4_lib.c b/drivers/net/ethernet/stmicro/stmmac/dwmac4_lib.c
index c7326d5..49f5687 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac4_lib.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac4_lib.c
@@ -37,96 +37,96 @@
void dwmac4_set_rx_tail_ptr(void __iomem *ioaddr, u32 tail_ptr, u32 chan)
{
- writel(tail_ptr, ioaddr + DMA_CHAN_RX_END_ADDR(0));
+ writel(tail_ptr, ioaddr + DMA_CHAN_RX_END_ADDR(chan));
}
void dwmac4_set_tx_tail_ptr(void __iomem *ioaddr, u32 tail_ptr, u32 chan)
{
- writel(tail_ptr, ioaddr + DMA_CHAN_TX_END_ADDR(0));
+ writel(tail_ptr, ioaddr + DMA_CHAN_TX_END_ADDR(chan));
}
-void dwmac4_dma_start_tx(void __iomem *ioaddr)
+void dwmac4_dma_start_tx(void __iomem *ioaddr, u32 chan)
{
- u32 value = readl(ioaddr + DMA_CHAN_TX_CONTROL(STMMAC_CHAN0));
+ u32 value = readl(ioaddr + DMA_CHAN_TX_CONTROL(chan));
value |= DMA_CONTROL_ST;
- writel(value, ioaddr + DMA_CHAN_TX_CONTROL(STMMAC_CHAN0));
+ writel(value, ioaddr + DMA_CHAN_TX_CONTROL(chan));
value = readl(ioaddr + GMAC_CONFIG);
value |= GMAC_CONFIG_TE;
writel(value, ioaddr + GMAC_CONFIG);
}
-void dwmac4_dma_stop_tx(void __iomem *ioaddr)
+void dwmac4_dma_stop_tx(void __iomem *ioaddr, u32 chan)
{
- u32 value = readl(ioaddr + DMA_CHAN_TX_CONTROL(STMMAC_CHAN0));
+ u32 value = readl(ioaddr + DMA_CHAN_TX_CONTROL(chan));
value &= ~DMA_CONTROL_ST;
- writel(value, ioaddr + DMA_CHAN_TX_CONTROL(STMMAC_CHAN0));
+ writel(value, ioaddr + DMA_CHAN_TX_CONTROL(chan));
value = readl(ioaddr + GMAC_CONFIG);
value &= ~GMAC_CONFIG_TE;
writel(value, ioaddr + GMAC_CONFIG);
}
-void dwmac4_dma_start_rx(void __iomem *ioaddr)
+void dwmac4_dma_start_rx(void __iomem *ioaddr, u32 chan)
{
- u32 value = readl(ioaddr + DMA_CHAN_RX_CONTROL(STMMAC_CHAN0));
+ u32 value = readl(ioaddr + DMA_CHAN_RX_CONTROL(chan));
value |= DMA_CONTROL_SR;
- writel(value, ioaddr + DMA_CHAN_RX_CONTROL(STMMAC_CHAN0));
+ writel(value, ioaddr + DMA_CHAN_RX_CONTROL(chan));
value = readl(ioaddr + GMAC_CONFIG);
value |= GMAC_CONFIG_RE;
writel(value, ioaddr + GMAC_CONFIG);
}
-void dwmac4_dma_stop_rx(void __iomem *ioaddr)
+void dwmac4_dma_stop_rx(void __iomem *ioaddr, u32 chan)
{
- u32 value = readl(ioaddr + DMA_CHAN_RX_CONTROL(STMMAC_CHAN0));
+ u32 value = readl(ioaddr + DMA_CHAN_RX_CONTROL(chan));
value &= ~DMA_CONTROL_SR;
- writel(value, ioaddr + DMA_CHAN_RX_CONTROL(STMMAC_CHAN0));
+ writel(value, ioaddr + DMA_CHAN_RX_CONTROL(chan));
value = readl(ioaddr + GMAC_CONFIG);
value &= ~GMAC_CONFIG_RE;
writel(value, ioaddr + GMAC_CONFIG);
}
-void dwmac4_set_tx_ring_len(void __iomem *ioaddr, u32 len)
+void dwmac4_set_tx_ring_len(void __iomem *ioaddr, u32 len, u32 chan)
{
- writel(len, ioaddr + DMA_CHAN_TX_RING_LEN(STMMAC_CHAN0));
+ writel(len, ioaddr + DMA_CHAN_TX_RING_LEN(chan));
}
-void dwmac4_set_rx_ring_len(void __iomem *ioaddr, u32 len)
+void dwmac4_set_rx_ring_len(void __iomem *ioaddr, u32 len, u32 chan)
{
- writel(len, ioaddr + DMA_CHAN_RX_RING_LEN(STMMAC_CHAN0));
+ writel(len, ioaddr + DMA_CHAN_RX_RING_LEN(chan));
}
-void dwmac4_enable_dma_irq(void __iomem *ioaddr)
+void dwmac4_enable_dma_irq(void __iomem *ioaddr, u32 chan)
{
writel(DMA_CHAN_INTR_DEFAULT_MASK, ioaddr +
- DMA_CHAN_INTR_ENA(STMMAC_CHAN0));
+ DMA_CHAN_INTR_ENA(chan));
}
-void dwmac410_enable_dma_irq(void __iomem *ioaddr)
+void dwmac410_enable_dma_irq(void __iomem *ioaddr, u32 chan)
{
writel(DMA_CHAN_INTR_DEFAULT_MASK_4_10,
- ioaddr + DMA_CHAN_INTR_ENA(STMMAC_CHAN0));
+ ioaddr + DMA_CHAN_INTR_ENA(chan));
}
-void dwmac4_disable_dma_irq(void __iomem *ioaddr)
+void dwmac4_disable_dma_irq(void __iomem *ioaddr, u32 chan)
{
- writel(0, ioaddr + DMA_CHAN_INTR_ENA(STMMAC_CHAN0));
+ writel(0, ioaddr + DMA_CHAN_INTR_ENA(chan));
}
int dwmac4_dma_interrupt(void __iomem *ioaddr,
- struct stmmac_extra_stats *x)
+ struct stmmac_extra_stats *x, u32 chan)
{
int ret = 0;
- u32 intr_status = readl(ioaddr + DMA_CHAN_STATUS(0));
+ u32 intr_status = readl(ioaddr + DMA_CHAN_STATUS(chan));
/* ABNORMAL interrupts */
if (unlikely(intr_status & DMA_CHAN_STATUS_AIS)) {
@@ -153,7 +153,7 @@
if (likely(intr_status & DMA_CHAN_STATUS_RI)) {
u32 value;
- value = readl(ioaddr + DMA_CHAN_INTR_ENA(STMMAC_CHAN0));
+ value = readl(ioaddr + DMA_CHAN_INTR_ENA(chan));
/* to schedule NAPI on real RIE event. */
if (likely(value & DMA_CHAN_INTR_ENA_RIE)) {
x->rx_normal_irq_n++;
@@ -172,7 +172,7 @@
* status [21-0] expect reserved bits [5-3]
*/
writel((intr_status & 0x3fffc7),
- ioaddr + DMA_CHAN_STATUS(STMMAC_CHAN0));
+ ioaddr + DMA_CHAN_STATUS(chan));
return ret;
}
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac_dma.h b/drivers/net/ethernet/stmicro/stmmac/dwmac_dma.h
index 56e485f..9091df8 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac_dma.h
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac_dma.h
@@ -137,13 +137,14 @@
#define DMA_CONTROL_FTF 0x00100000 /* Flush transmit FIFO */
void dwmac_enable_dma_transmission(void __iomem *ioaddr);
-void dwmac_enable_dma_irq(void __iomem *ioaddr);
-void dwmac_disable_dma_irq(void __iomem *ioaddr);
-void dwmac_dma_start_tx(void __iomem *ioaddr);
-void dwmac_dma_stop_tx(void __iomem *ioaddr);
-void dwmac_dma_start_rx(void __iomem *ioaddr);
-void dwmac_dma_stop_rx(void __iomem *ioaddr);
-int dwmac_dma_interrupt(void __iomem *ioaddr, struct stmmac_extra_stats *x);
+void dwmac_enable_dma_irq(void __iomem *ioaddr, u32 chan);
+void dwmac_disable_dma_irq(void __iomem *ioaddr, u32 chan);
+void dwmac_dma_start_tx(void __iomem *ioaddr, u32 chan);
+void dwmac_dma_stop_tx(void __iomem *ioaddr, u32 chan);
+void dwmac_dma_start_rx(void __iomem *ioaddr, u32 chan);
+void dwmac_dma_stop_rx(void __iomem *ioaddr, u32 chan);
+int dwmac_dma_interrupt(void __iomem *ioaddr, struct stmmac_extra_stats *x,
+ u32 chan);
int dwmac_dma_reset(void __iomem *ioaddr);
#endif /* __DWMAC_DMA_H__ */
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac_lib.c b/drivers/net/ethernet/stmicro/stmmac/dwmac_lib.c
index e60bfca..38f9430 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac_lib.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac_lib.c
@@ -47,38 +47,38 @@
writel(1, ioaddr + DMA_XMT_POLL_DEMAND);
}
-void dwmac_enable_dma_irq(void __iomem *ioaddr)
+void dwmac_enable_dma_irq(void __iomem *ioaddr, u32 chan)
{
writel(DMA_INTR_DEFAULT_MASK, ioaddr + DMA_INTR_ENA);
}
-void dwmac_disable_dma_irq(void __iomem *ioaddr)
+void dwmac_disable_dma_irq(void __iomem *ioaddr, u32 chan)
{
writel(0, ioaddr + DMA_INTR_ENA);
}
-void dwmac_dma_start_tx(void __iomem *ioaddr)
+void dwmac_dma_start_tx(void __iomem *ioaddr, u32 chan)
{
u32 value = readl(ioaddr + DMA_CONTROL);
value |= DMA_CONTROL_ST;
writel(value, ioaddr + DMA_CONTROL);
}
-void dwmac_dma_stop_tx(void __iomem *ioaddr)
+void dwmac_dma_stop_tx(void __iomem *ioaddr, u32 chan)
{
u32 value = readl(ioaddr + DMA_CONTROL);
value &= ~DMA_CONTROL_ST;
writel(value, ioaddr + DMA_CONTROL);
}
-void dwmac_dma_start_rx(void __iomem *ioaddr)
+void dwmac_dma_start_rx(void __iomem *ioaddr, u32 chan)
{
u32 value = readl(ioaddr + DMA_CONTROL);
value |= DMA_CONTROL_SR;
writel(value, ioaddr + DMA_CONTROL);
}
-void dwmac_dma_stop_rx(void __iomem *ioaddr)
+void dwmac_dma_stop_rx(void __iomem *ioaddr, u32 chan)
{
u32 value = readl(ioaddr + DMA_CONTROL);
value &= ~DMA_CONTROL_SR;
@@ -156,7 +156,7 @@
#endif
int dwmac_dma_interrupt(void __iomem *ioaddr,
- struct stmmac_extra_stats *x)
+ struct stmmac_extra_stats *x, u32 chan)
{
int ret = 0;
/* read the status register (CSR5) */
diff --git a/drivers/net/ethernet/stmicro/stmmac/enh_desc.c b/drivers/net/ethernet/stmicro/stmmac/enh_desc.c
index 323b59e..7546b36 100644
--- a/drivers/net/ethernet/stmicro/stmmac/enh_desc.c
+++ b/drivers/net/ethernet/stmicro/stmmac/enh_desc.c
@@ -315,7 +315,7 @@
static void enh_desc_prepare_tx_desc(struct dma_desc *p, int is_fs, int len,
bool csum_flag, int mode, bool tx_own,
- bool ls)
+ bool ls, unsigned int tot_pkt_len)
{
unsigned int tdes0 = le32_to_cpu(p->des0);
diff --git a/drivers/net/ethernet/stmicro/stmmac/norm_desc.c b/drivers/net/ethernet/stmicro/stmmac/norm_desc.c
index efb818e..f817f8f 100644
--- a/drivers/net/ethernet/stmicro/stmmac/norm_desc.c
+++ b/drivers/net/ethernet/stmicro/stmmac/norm_desc.c
@@ -191,7 +191,7 @@
static void ndesc_prepare_tx_desc(struct dma_desc *p, int is_fs, int len,
bool csum_flag, int mode, bool tx_own,
- bool ls)
+ bool ls, unsigned int tot_pkt_len)
{
unsigned int tdes1 = le32_to_cpu(p->des1);
diff --git a/drivers/net/ethernet/stmicro/stmmac/ring_mode.c b/drivers/net/ethernet/stmicro/stmmac/ring_mode.c
index 452f256..28e4b5d 100644
--- a/drivers/net/ethernet/stmicro/stmmac/ring_mode.c
+++ b/drivers/net/ethernet/stmicro/stmmac/ring_mode.c
@@ -26,16 +26,17 @@
static int stmmac_jumbo_frm(void *p, struct sk_buff *skb, int csum)
{
- struct stmmac_priv *priv = (struct stmmac_priv *)p;
- unsigned int entry = priv->cur_tx;
- struct dma_desc *desc;
+ struct stmmac_tx_queue *tx_q = (struct stmmac_tx_queue *)p;
unsigned int nopaged_len = skb_headlen(skb);
+ struct stmmac_priv *priv = tx_q->priv_data;
+ unsigned int entry = tx_q->cur_tx;
unsigned int bmax, len, des2;
+ struct dma_desc *desc;
if (priv->extend_desc)
- desc = (struct dma_desc *)(priv->dma_etx + entry);
+ desc = (struct dma_desc *)(tx_q->dma_etx + entry);
else
- desc = priv->dma_tx + entry;
+ desc = tx_q->dma_tx + entry;
if (priv->plat->enh_desc)
bmax = BUF_SIZE_8KiB;
@@ -52,48 +53,51 @@
if (dma_mapping_error(priv->device, des2))
return -1;
- priv->tx_skbuff_dma[entry].buf = des2;
- priv->tx_skbuff_dma[entry].len = bmax;
- priv->tx_skbuff_dma[entry].is_jumbo = true;
+ tx_q->tx_skbuff_dma[entry].buf = des2;
+ tx_q->tx_skbuff_dma[entry].len = bmax;
+ tx_q->tx_skbuff_dma[entry].is_jumbo = true;
desc->des3 = cpu_to_le32(des2 + BUF_SIZE_4KiB);
priv->hw->desc->prepare_tx_desc(desc, 1, bmax, csum,
- STMMAC_RING_MODE, 0, false);
- priv->tx_skbuff[entry] = NULL;
+ STMMAC_RING_MODE, 0,
+ false, skb->len);
+ tx_q->tx_skbuff[entry] = NULL;
entry = STMMAC_GET_ENTRY(entry, DMA_TX_SIZE);
if (priv->extend_desc)
- desc = (struct dma_desc *)(priv->dma_etx + entry);
+ desc = (struct dma_desc *)(tx_q->dma_etx + entry);
else
- desc = priv->dma_tx + entry;
+ desc = tx_q->dma_tx + entry;
des2 = dma_map_single(priv->device, skb->data + bmax, len,
DMA_TO_DEVICE);
desc->des2 = cpu_to_le32(des2);
if (dma_mapping_error(priv->device, des2))
return -1;
- priv->tx_skbuff_dma[entry].buf = des2;
- priv->tx_skbuff_dma[entry].len = len;
- priv->tx_skbuff_dma[entry].is_jumbo = true;
+ tx_q->tx_skbuff_dma[entry].buf = des2;
+ tx_q->tx_skbuff_dma[entry].len = len;
+ tx_q->tx_skbuff_dma[entry].is_jumbo = true;
desc->des3 = cpu_to_le32(des2 + BUF_SIZE_4KiB);
priv->hw->desc->prepare_tx_desc(desc, 0, len, csum,
- STMMAC_RING_MODE, 1, true);
+ STMMAC_RING_MODE, 1,
+ true, skb->len);
} else {
des2 = dma_map_single(priv->device, skb->data,
nopaged_len, DMA_TO_DEVICE);
desc->des2 = cpu_to_le32(des2);
if (dma_mapping_error(priv->device, des2))
return -1;
- priv->tx_skbuff_dma[entry].buf = des2;
- priv->tx_skbuff_dma[entry].len = nopaged_len;
- priv->tx_skbuff_dma[entry].is_jumbo = true;
+ tx_q->tx_skbuff_dma[entry].buf = des2;
+ tx_q->tx_skbuff_dma[entry].len = nopaged_len;
+ tx_q->tx_skbuff_dma[entry].is_jumbo = true;
desc->des3 = cpu_to_le32(des2 + BUF_SIZE_4KiB);
priv->hw->desc->prepare_tx_desc(desc, 1, nopaged_len, csum,
- STMMAC_RING_MODE, 0, true);
+ STMMAC_RING_MODE, 0,
+ true, skb->len);
}
- priv->cur_tx = entry;
+ tx_q->cur_tx = entry;
return entry;
}
@@ -125,12 +129,13 @@
static void stmmac_clean_desc3(void *priv_ptr, struct dma_desc *p)
{
- struct stmmac_priv *priv = (struct stmmac_priv *)priv_ptr;
- unsigned int entry = priv->dirty_tx;
+ struct stmmac_tx_queue *tx_q = (struct stmmac_tx_queue *)priv_ptr;
+ struct stmmac_priv *priv = tx_q->priv_data;
+ unsigned int entry = tx_q->dirty_tx;
/* des3 is only used for jumbo frames tx or time stamping */
- if (unlikely(priv->tx_skbuff_dma[entry].is_jumbo ||
- (priv->tx_skbuff_dma[entry].last_segment &&
+ if (unlikely(tx_q->tx_skbuff_dma[entry].is_jumbo ||
+ (tx_q->tx_skbuff_dma[entry].last_segment &&
!priv->extend_desc && priv->hwts_tx_en)))
p->des3 = 0;
}
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac.h b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
index cd8fb61..33efe70 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac.h
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
@@ -46,38 +46,51 @@
bool is_jumbo;
};
-struct stmmac_priv {
- /* Frequently used values are kept adjacent for cache effect */
+/* Frequently used values are kept adjacent for cache effect */
+struct stmmac_tx_queue {
+ u32 queue_index;
+ struct stmmac_priv *priv_data;
struct dma_extended_desc *dma_etx ____cacheline_aligned_in_smp;
struct dma_desc *dma_tx;
struct sk_buff **tx_skbuff;
+ struct stmmac_tx_info *tx_skbuff_dma;
unsigned int cur_tx;
unsigned int dirty_tx;
+ dma_addr_t dma_tx_phy;
+ u32 tx_tail_addr;
+};
+
+struct stmmac_rx_queue {
+ u32 queue_index;
+ struct stmmac_priv *priv_data;
+ struct dma_extended_desc *dma_erx;
+ struct dma_desc *dma_rx ____cacheline_aligned_in_smp;
+ struct sk_buff **rx_skbuff;
+ dma_addr_t *rx_skbuff_dma;
+ unsigned int cur_rx;
+ unsigned int dirty_rx;
+ u32 rx_zeroc_thresh;
+ dma_addr_t dma_rx_phy;
+ u32 rx_tail_addr;
+ struct napi_struct napi ____cacheline_aligned_in_smp;
+};
+
+struct stmmac_priv {
+ /* Frequently used values are kept adjacent for cache effect */
u32 tx_count_frames;
u32 tx_coal_frames;
u32 tx_coal_timer;
- struct stmmac_tx_info *tx_skbuff_dma;
- dma_addr_t dma_tx_phy;
+
int tx_coalesce;
int hwts_tx_en;
bool tx_path_in_lpi_mode;
struct timer_list txtimer;
bool tso;
- struct dma_desc *dma_rx ____cacheline_aligned_in_smp;
- struct dma_extended_desc *dma_erx;
- struct sk_buff **rx_skbuff;
- unsigned int cur_rx;
- unsigned int dirty_rx;
unsigned int dma_buf_sz;
unsigned int rx_copybreak;
- unsigned int rx_zeroc_thresh;
u32 rx_riwt;
int hwts_rx_en;
- dma_addr_t *rx_skbuff_dma;
- dma_addr_t dma_rx_phy;
-
- struct napi_struct napi ____cacheline_aligned_in_smp;
void __iomem *ioaddr;
struct net_device *dev;
@@ -85,6 +98,12 @@
struct mac_device_info *hw;
spinlock_t lock;
+ /* RX Queue */
+ struct stmmac_rx_queue rx_queue[MTL_MAX_RX_QUEUES];
+
+ /* TX Queue */
+ struct stmmac_tx_queue tx_queue[MTL_MAX_TX_QUEUES];
+
int oldlink;
int speed;
int oldduplex;
@@ -119,8 +138,6 @@
spinlock_t ptp_lock;
void __iomem *mmcaddr;
void __iomem *ptpaddr;
- u32 rx_tail_addr;
- u32 tx_tail_addr;
u32 mss;
#ifdef CONFIG_DEBUG_FS
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c
index 85d6411..16808e4 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c
@@ -481,6 +481,7 @@
struct ethtool_pauseparam *pause)
{
struct stmmac_priv *priv = netdev_priv(netdev);
+ u32 tx_cnt = priv->plat->tx_queues_to_use;
struct phy_device *phy = netdev->phydev;
int new_pause = FLOW_OFF;
@@ -511,7 +512,7 @@
}
priv->hw->mac->flow_ctrl(priv->hw, phy->duplex, priv->flow_ctrl,
- priv->pause);
+ priv->pause, tx_cnt);
return 0;
}
@@ -519,6 +520,8 @@
struct ethtool_stats *dummy, u64 *data)
{
struct stmmac_priv *priv = netdev_priv(dev);
+ u32 rx_queues_count = priv->plat->rx_queues_to_use;
+ u32 tx_queues_count = priv->plat->tx_queues_to_use;
int i, j = 0;
/* Update the DMA HW counters for dwmac10/100 */
@@ -549,7 +552,8 @@
if ((priv->hw->mac->debug) &&
(priv->synopsys_id >= DWMAC_CORE_3_50))
priv->hw->mac->debug(priv->ioaddr,
- (void *)&priv->xstats);
+ (void *)&priv->xstats,
+ rx_queues_count, tx_queues_count);
}
for (i = 0; i < STMMAC_STATS_LEN; i++) {
char *p = (char *)priv + stmmac_gstrings_stats[i].stat_offset;
@@ -726,6 +730,7 @@
struct ethtool_coalesce *ec)
{
struct stmmac_priv *priv = netdev_priv(dev);
+ u32 rx_cnt = priv->plat->rx_queues_to_use;
unsigned int rx_riwt;
/* Check not supported parameters */
@@ -764,7 +769,7 @@
priv->tx_coal_frames = ec->tx_max_coalesced_frames;
priv->tx_coal_timer = ec->tx_coalesce_usecs;
priv->rx_riwt = rx_riwt;
- priv->hw->dma->rx_watchdog(priv->ioaddr, priv->rx_riwt);
+ priv->hw->dma->rx_watchdog(priv->ioaddr, priv->rx_riwt, rx_cnt);
return 0;
}
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index 4498a38..cd8c601 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -139,6 +139,64 @@
}
/**
+ * stmmac_disable_all_queues - Disable all queues
+ * @priv: driver private structure
+ */
+static void stmmac_disable_all_queues(struct stmmac_priv *priv)
+{
+ u32 rx_queues_cnt = priv->plat->rx_queues_to_use;
+ u32 queue;
+
+ for (queue = 0; queue < rx_queues_cnt; queue++) {
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+
+ napi_disable(&rx_q->napi);
+ }
+}
+
+/**
+ * stmmac_enable_all_queues - Enable all queues
+ * @priv: driver private structure
+ */
+static void stmmac_enable_all_queues(struct stmmac_priv *priv)
+{
+ u32 rx_queues_cnt = priv->plat->rx_queues_to_use;
+ u32 queue;
+
+ for (queue = 0; queue < rx_queues_cnt; queue++) {
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+
+ napi_enable(&rx_q->napi);
+ }
+}
+
+/**
+ * stmmac_stop_all_queues - Stop all queues
+ * @priv: driver private structure
+ */
+static void stmmac_stop_all_queues(struct stmmac_priv *priv)
+{
+ u32 tx_queues_cnt = priv->plat->tx_queues_to_use;
+ u32 queue;
+
+ for (queue = 0; queue < tx_queues_cnt; queue++)
+ netif_tx_stop_queue(netdev_get_tx_queue(priv->dev, queue));
+}
+
+/**
+ * stmmac_start_all_queues - Start all queues
+ * @priv: driver private structure
+ */
+static void stmmac_start_all_queues(struct stmmac_priv *priv)
+{
+ u32 tx_queues_cnt = priv->plat->tx_queues_to_use;
+ u32 queue;
+
+ for (queue = 0; queue < tx_queues_cnt; queue++)
+ netif_tx_start_queue(netdev_get_tx_queue(priv->dev, queue));
+}
+
+/**
* stmmac_clk_csr_set - dynamically set the MDC clock
* @priv: driver private structure
* Description: this is to dynamically set the MDC clock according to the csr
@@ -185,26 +243,33 @@
print_hex_dump_bytes("", DUMP_PREFIX_OFFSET, buf, len);
}
-static inline u32 stmmac_tx_avail(struct stmmac_priv *priv)
+static inline u32 stmmac_tx_avail(struct stmmac_priv *priv, u32 queue)
{
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
u32 avail;
- if (priv->dirty_tx > priv->cur_tx)
- avail = priv->dirty_tx - priv->cur_tx - 1;
+ if (tx_q->dirty_tx > tx_q->cur_tx)
+ avail = tx_q->dirty_tx - tx_q->cur_tx - 1;
else
- avail = DMA_TX_SIZE - priv->cur_tx + priv->dirty_tx - 1;
+ avail = DMA_TX_SIZE - tx_q->cur_tx + tx_q->dirty_tx - 1;
return avail;
}
-static inline u32 stmmac_rx_dirty(struct stmmac_priv *priv)
+/**
+ * stmmac_rx_dirty - Get RX queue dirty
+ * @priv: driver private structure
+ * @queue: RX queue index
+ */
+static inline u32 stmmac_rx_dirty(struct stmmac_priv *priv, u32 queue)
{
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
u32 dirty;
- if (priv->dirty_rx <= priv->cur_rx)
- dirty = priv->cur_rx - priv->dirty_rx;
+ if (rx_q->dirty_rx <= rx_q->cur_rx)
+ dirty = rx_q->cur_rx - rx_q->dirty_rx;
else
- dirty = DMA_RX_SIZE - priv->dirty_rx + priv->cur_rx;
+ dirty = DMA_RX_SIZE - rx_q->dirty_rx + rx_q->cur_rx;
return dirty;
}
@@ -232,9 +297,19 @@
*/
static void stmmac_enable_eee_mode(struct stmmac_priv *priv)
{
+ u32 tx_cnt = priv->plat->tx_queues_to_use;
+ u32 queue;
+
+ /* check if all TX queues have the work finished */
+ for (queue = 0; queue < tx_cnt; queue++) {
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
+
+ if (tx_q->dirty_tx != tx_q->cur_tx)
+ return; /* still unfinished work */
+ }
+
/* Check and enter in LPI mode */
- if ((priv->dirty_tx == priv->cur_tx) &&
- (priv->tx_path_in_lpi_mode == false))
+ if (!priv->tx_path_in_lpi_mode)
priv->hw->mac->set_eee_mode(priv->hw,
priv->plat->en_tx_lpi_clockgating);
}
@@ -673,6 +748,19 @@
}
/**
+ * stmmac_mac_flow_ctrl - Configure flow control in all queues
+ * @priv: driver private structure
+ * Description: It is used for configuring the flow control in all queues
+ */
+static void stmmac_mac_flow_ctrl(struct stmmac_priv *priv, u32 duplex)
+{
+ u32 tx_cnt = priv->plat->tx_queues_to_use;
+
+ priv->hw->mac->flow_ctrl(priv->hw, duplex, priv->flow_ctrl,
+ priv->pause, tx_cnt);
+}
+
+/**
* stmmac_adjust_link - adjusts the link parameters
* @dev: net device structure
* Description: this is the helper called by the physical abstraction layer
@@ -687,7 +775,6 @@
struct phy_device *phydev = dev->phydev;
unsigned long flags;
int new_state = 0;
- unsigned int fc = priv->flow_ctrl, pause_time = priv->pause;
if (!phydev)
return;
@@ -709,8 +796,7 @@
}
/* Flow Control operation */
if (phydev->pause)
- priv->hw->mac->flow_ctrl(priv->hw, phydev->duplex,
- fc, pause_time);
+ stmmac_mac_flow_ctrl(priv, phydev->duplex);
if (phydev->speed != priv->speed) {
new_state = 1;
@@ -878,22 +964,56 @@
return 0;
}
+static void stmmac_display_rx_rings(struct stmmac_priv *priv)
+{
+ u32 rx_cnt = priv->plat->rx_queues_to_use;
+ void *head_rx;
+ u32 queue;
+
+ /* Display RX rings */
+ for (queue = 0; queue < rx_cnt; queue++) {
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+
+ pr_info("\tRX Queue %u rings\n", queue);
+
+ if (priv->extend_desc)
+ head_rx = (void *)rx_q->dma_erx;
+ else
+ head_rx = (void *)rx_q->dma_rx;
+
+ /* Display RX ring */
+ priv->hw->desc->display_ring(head_rx, DMA_RX_SIZE, true);
+ }
+}
+
+static void stmmac_display_tx_rings(struct stmmac_priv *priv)
+{
+ u32 tx_cnt = priv->plat->tx_queues_to_use;
+ void *head_tx;
+ u32 queue;
+
+ /* Display TX rings */
+ for (queue = 0; queue < tx_cnt; queue++) {
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
+
+ pr_info("\tTX Queue %d rings\n", queue);
+
+ if (priv->extend_desc)
+ head_tx = (void *)tx_q->dma_etx;
+ else
+ head_tx = (void *)tx_q->dma_tx;
+
+ priv->hw->desc->display_ring(head_tx, DMA_TX_SIZE, false);
+ }
+}
+
static void stmmac_display_rings(struct stmmac_priv *priv)
{
- void *head_rx, *head_tx;
+ /* Display RX ring */
+ stmmac_display_rx_rings(priv);
- if (priv->extend_desc) {
- head_rx = (void *)priv->dma_erx;
- head_tx = (void *)priv->dma_etx;
- } else {
- head_rx = (void *)priv->dma_rx;
- head_tx = (void *)priv->dma_tx;
- }
-
- /* Display Rx ring */
- priv->hw->desc->display_ring(head_rx, DMA_RX_SIZE, true);
- /* Display Tx ring */
- priv->hw->desc->display_ring(head_tx, DMA_TX_SIZE, false);
+ /* Display TX ring */
+ stmmac_display_tx_rings(priv);
}
static int stmmac_set_bfsize(int mtu, int bufsize)
@@ -913,34 +1033,72 @@
}
/**
+ * stmmac_clear_rx_descriptors - clear RX descriptors
+ * @priv: driver private structure
+ * @queue: RX queue index
+ * Description: this function is called to clear the RX descriptors
+ * in case of both basic and extended descriptors are used.
+ */
+static void stmmac_clear_rx_descriptors(struct stmmac_priv *priv, u32 queue)
+{
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+ int i;
+
+ /* Clear the RX descriptors */
+ for (i = 0; i < DMA_RX_SIZE; i++)
+ if (priv->extend_desc)
+ priv->hw->desc->init_rx_desc(&rx_q->dma_erx[i].basic,
+ priv->use_riwt, priv->mode,
+ (i == DMA_RX_SIZE - 1));
+ else
+ priv->hw->desc->init_rx_desc(&rx_q->dma_rx[i],
+ priv->use_riwt, priv->mode,
+ (i == DMA_RX_SIZE - 1));
+}
+
+/**
+ * stmmac_clear_tx_descriptors - clear tx descriptors
+ * @priv: driver private structure
+ * @queue: TX queue index.
+ * Description: this function is called to clear the TX descriptors
+ * in case of both basic and extended descriptors are used.
+ */
+static void stmmac_clear_tx_descriptors(struct stmmac_priv *priv, u32 queue)
+{
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
+ int i;
+
+ /* Clear the TX descriptors */
+ for (i = 0; i < DMA_TX_SIZE; i++)
+ if (priv->extend_desc)
+ priv->hw->desc->init_tx_desc(&tx_q->dma_etx[i].basic,
+ priv->mode,
+ (i == DMA_TX_SIZE - 1));
+ else
+ priv->hw->desc->init_tx_desc(&tx_q->dma_tx[i],
+ priv->mode,
+ (i == DMA_TX_SIZE - 1));
+}
+
+/**
* stmmac_clear_descriptors - clear descriptors
* @priv: driver private structure
- * Description: this function is called to clear the tx and rx descriptors
+ * Description: this function is called to clear the TX and RX descriptors
* in case of both basic and extended descriptors are used.
*/
static void stmmac_clear_descriptors(struct stmmac_priv *priv)
{
- int i;
+ u32 rx_queue_cnt = priv->plat->rx_queues_to_use;
+ u32 tx_queue_cnt = priv->plat->tx_queues_to_use;
+ u32 queue;
- /* Clear the Rx/Tx descriptors */
- for (i = 0; i < DMA_RX_SIZE; i++)
- if (priv->extend_desc)
- priv->hw->desc->init_rx_desc(&priv->dma_erx[i].basic,
- priv->use_riwt, priv->mode,
- (i == DMA_RX_SIZE - 1));
- else
- priv->hw->desc->init_rx_desc(&priv->dma_rx[i],
- priv->use_riwt, priv->mode,
- (i == DMA_RX_SIZE - 1));
- for (i = 0; i < DMA_TX_SIZE; i++)
- if (priv->extend_desc)
- priv->hw->desc->init_tx_desc(&priv->dma_etx[i].basic,
- priv->mode,
- (i == DMA_TX_SIZE - 1));
- else
- priv->hw->desc->init_tx_desc(&priv->dma_tx[i],
- priv->mode,
- (i == DMA_TX_SIZE - 1));
+ /* Clear the RX descriptors */
+ for (queue = 0; queue < rx_queue_cnt; queue++)
+ stmmac_clear_rx_descriptors(priv, queue);
+
+ /* Clear the TX descriptors */
+ for (queue = 0; queue < tx_queue_cnt; queue++)
+ stmmac_clear_tx_descriptors(priv, queue);
}
/**
@@ -948,13 +1106,15 @@
* @priv: driver private structure
* @p: descriptor pointer
* @i: descriptor index
- * @flags: gfp flag.
+ * @flags: gfp flag
+ * @queue: RX queue index
* Description: this function is called to allocate a receive buffer, perform
* the DMA mapping and init the descriptor.
*/
static int stmmac_init_rx_buffers(struct stmmac_priv *priv, struct dma_desc *p,
- int i, gfp_t flags)
+ int i, gfp_t flags, u32 queue)
{
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
struct sk_buff *skb;
skb = __netdev_alloc_skb_ip_align(priv->dev, priv->dma_buf_sz, flags);
@@ -963,20 +1123,20 @@
"%s: Rx init fails; skb is NULL\n", __func__);
return -ENOMEM;
}
- priv->rx_skbuff[i] = skb;
- priv->rx_skbuff_dma[i] = dma_map_single(priv->device, skb->data,
+ rx_q->rx_skbuff[i] = skb;
+ rx_q->rx_skbuff_dma[i] = dma_map_single(priv->device, skb->data,
priv->dma_buf_sz,
DMA_FROM_DEVICE);
- if (dma_mapping_error(priv->device, priv->rx_skbuff_dma[i])) {
+ if (dma_mapping_error(priv->device, rx_q->rx_skbuff_dma[i])) {
netdev_err(priv->dev, "%s: DMA mapping error\n", __func__);
dev_kfree_skb_any(skb);
return -EINVAL;
}
if (priv->synopsys_id >= DWMAC_CORE_4_00)
- p->des0 = cpu_to_le32(priv->rx_skbuff_dma[i]);
+ p->des0 = cpu_to_le32(rx_q->rx_skbuff_dma[i]);
else
- p->des2 = cpu_to_le32(priv->rx_skbuff_dma[i]);
+ p->des2 = cpu_to_le32(rx_q->rx_skbuff_dma[i]);
if ((priv->hw->mode->init_desc3) &&
(priv->dma_buf_sz == BUF_SIZE_16KiB))
@@ -985,14 +1145,209 @@
return 0;
}
-static void stmmac_free_rx_buffers(struct stmmac_priv *priv, int i)
+/**
+ * stmmac_free_rx_buffer - free RX dma buffers
+ * @priv: private structure
+ * @queue: RX queue index
+ * @i: buffer index.
+ */
+static void stmmac_free_rx_buffer(struct stmmac_priv *priv, u32 queue, int i)
{
- if (priv->rx_skbuff[i]) {
- dma_unmap_single(priv->device, priv->rx_skbuff_dma[i],
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+
+ if (rx_q->rx_skbuff[i]) {
+ dma_unmap_single(priv->device, rx_q->rx_skbuff_dma[i],
priv->dma_buf_sz, DMA_FROM_DEVICE);
- dev_kfree_skb_any(priv->rx_skbuff[i]);
+ dev_kfree_skb_any(rx_q->rx_skbuff[i]);
}
- priv->rx_skbuff[i] = NULL;
+ rx_q->rx_skbuff[i] = NULL;
+}
+
+/**
+ * stmmac_free_tx_buffer - free RX dma buffers
+ * @priv: private structure
+ * @queue: RX queue index
+ * @i: buffer index.
+ */
+static void stmmac_free_tx_buffer(struct stmmac_priv *priv, u32 queue, int i)
+{
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
+
+ if (tx_q->tx_skbuff_dma[i].buf) {
+ if (tx_q->tx_skbuff_dma[i].map_as_page)
+ dma_unmap_page(priv->device,
+ tx_q->tx_skbuff_dma[i].buf,
+ tx_q->tx_skbuff_dma[i].len,
+ DMA_TO_DEVICE);
+ else
+ dma_unmap_single(priv->device,
+ tx_q->tx_skbuff_dma[i].buf,
+ tx_q->tx_skbuff_dma[i].len,
+ DMA_TO_DEVICE);
+ }
+
+ if (tx_q->tx_skbuff[i]) {
+ dev_kfree_skb_any(tx_q->tx_skbuff[i]);
+ tx_q->tx_skbuff[i] = NULL;
+ tx_q->tx_skbuff_dma[i].buf = 0;
+ tx_q->tx_skbuff_dma[i].map_as_page = false;
+ }
+}
+
+/**
+ * init_dma_rx_desc_rings - init the RX descriptor rings
+ * @dev: net device structure
+ * @flags: gfp flag.
+ * Description: this function initializes the DMA RX descriptors
+ * and allocates the socket buffers. It supports the chained and ring
+ * modes.
+ */
+static int init_dma_rx_desc_rings(struct net_device *dev, gfp_t flags)
+{
+ struct stmmac_priv *priv = netdev_priv(dev);
+ u32 rx_count = priv->plat->rx_queues_to_use;
+ unsigned int bfsize = 0;
+ int ret = -ENOMEM;
+ u32 queue;
+ int i;
+
+ if (priv->hw->mode->set_16kib_bfsize)
+ bfsize = priv->hw->mode->set_16kib_bfsize(dev->mtu);
+
+ if (bfsize < BUF_SIZE_16KiB)
+ bfsize = stmmac_set_bfsize(dev->mtu, priv->dma_buf_sz);
+
+ priv->dma_buf_sz = bfsize;
+
+ /* RX INITIALIZATION */
+ netif_dbg(priv, probe, priv->dev,
+ "SKB addresses:\nskb\t\tskb data\tdma data\n");
+
+ for (queue = 0; queue < rx_count; queue++) {
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+
+ netif_dbg(priv, probe, priv->dev,
+ "(%s) dma_rx_phy=0x%08x\n", __func__,
+ (u32)rx_q->dma_rx_phy);
+
+ for (i = 0; i < DMA_RX_SIZE; i++) {
+ struct dma_desc *p;
+
+ if (priv->extend_desc)
+ p = &((rx_q->dma_erx + i)->basic);
+ else
+ p = rx_q->dma_rx + i;
+
+ ret = stmmac_init_rx_buffers(priv, p, i, flags,
+ queue);
+ if (ret)
+ goto err_init_rx_buffers;
+
+ netif_dbg(priv, probe, priv->dev, "[%p]\t[%p]\t[%x]\n",
+ rx_q->rx_skbuff[i], rx_q->rx_skbuff[i]->data,
+ (unsigned int)rx_q->rx_skbuff_dma[i]);
+ }
+
+ rx_q->cur_rx = 0;
+ rx_q->dirty_rx = (unsigned int)(i - DMA_RX_SIZE);
+
+ stmmac_clear_rx_descriptors(priv, queue);
+
+ /* Setup the chained descriptor addresses */
+ if (priv->mode == STMMAC_CHAIN_MODE) {
+ if (priv->extend_desc)
+ priv->hw->mode->init(rx_q->dma_erx,
+ rx_q->dma_rx_phy,
+ DMA_RX_SIZE, 1);
+ else
+ priv->hw->mode->init(rx_q->dma_rx,
+ rx_q->dma_rx_phy,
+ DMA_RX_SIZE, 0);
+ }
+ }
+
+ buf_sz = bfsize;
+
+ return 0;
+
+err_init_rx_buffers:
+ while (queue >= 0) {
+ while (--i >= 0)
+ stmmac_free_rx_buffer(priv, queue, i);
+
+ if (queue == 0)
+ break;
+
+ i = DMA_RX_SIZE;
+ queue--;
+ }
+
+ return ret;
+}
+
+/**
+ * init_dma_tx_desc_rings - init the TX descriptor rings
+ * @dev: net device structure.
+ * Description: this function initializes the DMA TX descriptors
+ * and allocates the socket buffers. It supports the chained and ring
+ * modes.
+ */
+static int init_dma_tx_desc_rings(struct net_device *dev)
+{
+ struct stmmac_priv *priv = netdev_priv(dev);
+ u32 tx_queue_cnt = priv->plat->tx_queues_to_use;
+ u32 queue;
+ int i;
+
+ for (queue = 0; queue < tx_queue_cnt; queue++) {
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
+
+ netif_dbg(priv, probe, priv->dev,
+ "(%s) dma_tx_phy=0x%08x\n", __func__,
+ (u32)tx_q->dma_tx_phy);
+
+ /* Setup the chained descriptor addresses */
+ if (priv->mode == STMMAC_CHAIN_MODE) {
+ if (priv->extend_desc)
+ priv->hw->mode->init(tx_q->dma_etx,
+ tx_q->dma_tx_phy,
+ DMA_TX_SIZE, 1);
+ else
+ priv->hw->mode->init(tx_q->dma_tx,
+ tx_q->dma_tx_phy,
+ DMA_TX_SIZE, 0);
+ }
+
+ for (i = 0; i < DMA_TX_SIZE; i++) {
+ struct dma_desc *p;
+ if (priv->extend_desc)
+ p = &((tx_q->dma_etx + i)->basic);
+ else
+ p = tx_q->dma_tx + i;
+
+ if (priv->synopsys_id >= DWMAC_CORE_4_00) {
+ p->des0 = 0;
+ p->des1 = 0;
+ p->des2 = 0;
+ p->des3 = 0;
+ } else {
+ p->des2 = 0;
+ }
+
+ tx_q->tx_skbuff_dma[i].buf = 0;
+ tx_q->tx_skbuff_dma[i].map_as_page = false;
+ tx_q->tx_skbuff_dma[i].len = 0;
+ tx_q->tx_skbuff_dma[i].last_segment = false;
+ tx_q->tx_skbuff[i] = NULL;
+ }
+
+ tx_q->dirty_tx = 0;
+ tx_q->cur_tx = 0;
+
+ netdev_tx_reset_queue(netdev_get_tx_queue(priv->dev, queue));
+ }
+
+ return 0;
}
/**
@@ -1005,134 +1360,234 @@
*/
static int init_dma_desc_rings(struct net_device *dev, gfp_t flags)
{
- int i;
struct stmmac_priv *priv = netdev_priv(dev);
- unsigned int bfsize = 0;
- int ret = -ENOMEM;
+ int ret;
- if (priv->hw->mode->set_16kib_bfsize)
- bfsize = priv->hw->mode->set_16kib_bfsize(dev->mtu);
+ ret = init_dma_rx_desc_rings(dev, flags);
+ if (ret)
+ return ret;
- if (bfsize < BUF_SIZE_16KiB)
- bfsize = stmmac_set_bfsize(dev->mtu, priv->dma_buf_sz);
-
- priv->dma_buf_sz = bfsize;
-
- netif_dbg(priv, probe, priv->dev,
- "(%s) dma_rx_phy=0x%08x dma_tx_phy=0x%08x\n",
- __func__, (u32)priv->dma_rx_phy, (u32)priv->dma_tx_phy);
-
- /* RX INITIALIZATION */
- netif_dbg(priv, probe, priv->dev,
- "SKB addresses:\nskb\t\tskb data\tdma data\n");
-
- for (i = 0; i < DMA_RX_SIZE; i++) {
- struct dma_desc *p;
- if (priv->extend_desc)
- p = &((priv->dma_erx + i)->basic);
- else
- p = priv->dma_rx + i;
-
- ret = stmmac_init_rx_buffers(priv, p, i, flags);
- if (ret)
- goto err_init_rx_buffers;
-
- netif_dbg(priv, probe, priv->dev, "[%p]\t[%p]\t[%x]\n",
- priv->rx_skbuff[i], priv->rx_skbuff[i]->data,
- (unsigned int)priv->rx_skbuff_dma[i]);
- }
- priv->cur_rx = 0;
- priv->dirty_rx = (unsigned int)(i - DMA_RX_SIZE);
- buf_sz = bfsize;
-
- /* Setup the chained descriptor addresses */
- if (priv->mode == STMMAC_CHAIN_MODE) {
- if (priv->extend_desc) {
- priv->hw->mode->init(priv->dma_erx, priv->dma_rx_phy,
- DMA_RX_SIZE, 1);
- priv->hw->mode->init(priv->dma_etx, priv->dma_tx_phy,
- DMA_TX_SIZE, 1);
- } else {
- priv->hw->mode->init(priv->dma_rx, priv->dma_rx_phy,
- DMA_RX_SIZE, 0);
- priv->hw->mode->init(priv->dma_tx, priv->dma_tx_phy,
- DMA_TX_SIZE, 0);
- }
- }
-
- /* TX INITIALIZATION */
- for (i = 0; i < DMA_TX_SIZE; i++) {
- struct dma_desc *p;
- if (priv->extend_desc)
- p = &((priv->dma_etx + i)->basic);
- else
- p = priv->dma_tx + i;
-
- if (priv->synopsys_id >= DWMAC_CORE_4_00) {
- p->des0 = 0;
- p->des1 = 0;
- p->des2 = 0;
- p->des3 = 0;
- } else {
- p->des2 = 0;
- }
-
- priv->tx_skbuff_dma[i].buf = 0;
- priv->tx_skbuff_dma[i].map_as_page = false;
- priv->tx_skbuff_dma[i].len = 0;
- priv->tx_skbuff_dma[i].last_segment = false;
- priv->tx_skbuff[i] = NULL;
- }
-
- priv->dirty_tx = 0;
- priv->cur_tx = 0;
- netdev_reset_queue(priv->dev);
+ ret = init_dma_tx_desc_rings(dev);
stmmac_clear_descriptors(priv);
if (netif_msg_hw(priv))
stmmac_display_rings(priv);
- return 0;
-err_init_rx_buffers:
- while (--i >= 0)
- stmmac_free_rx_buffers(priv, i);
return ret;
}
-static void dma_free_rx_skbufs(struct stmmac_priv *priv)
+/**
+ * dma_free_rx_skbufs - free RX dma buffers
+ * @priv: private structure
+ * @queue: RX queue index
+ */
+static void dma_free_rx_skbufs(struct stmmac_priv *priv, u32 queue)
{
int i;
for (i = 0; i < DMA_RX_SIZE; i++)
- stmmac_free_rx_buffers(priv, i);
+ stmmac_free_rx_buffer(priv, queue, i);
}
-static void dma_free_tx_skbufs(struct stmmac_priv *priv)
+/**
+ * dma_free_tx_skbufs - free TX dma buffers
+ * @priv: private structure
+ * @queue: TX queue index
+ */
+static void dma_free_tx_skbufs(struct stmmac_priv *priv, u32 queue)
{
int i;
- for (i = 0; i < DMA_TX_SIZE; i++) {
- if (priv->tx_skbuff_dma[i].buf) {
- if (priv->tx_skbuff_dma[i].map_as_page)
- dma_unmap_page(priv->device,
- priv->tx_skbuff_dma[i].buf,
- priv->tx_skbuff_dma[i].len,
- DMA_TO_DEVICE);
- else
- dma_unmap_single(priv->device,
- priv->tx_skbuff_dma[i].buf,
- priv->tx_skbuff_dma[i].len,
- DMA_TO_DEVICE);
- }
+ for (i = 0; i < DMA_TX_SIZE; i++)
+ stmmac_free_tx_buffer(priv, queue, i);
+}
- if (priv->tx_skbuff[i]) {
- dev_kfree_skb_any(priv->tx_skbuff[i]);
- priv->tx_skbuff[i] = NULL;
- priv->tx_skbuff_dma[i].buf = 0;
- priv->tx_skbuff_dma[i].map_as_page = false;
+/**
+ * free_dma_rx_desc_resources - free RX dma desc resources
+ * @priv: private structure
+ */
+static void free_dma_rx_desc_resources(struct stmmac_priv *priv)
+{
+ u32 rx_count = priv->plat->rx_queues_to_use;
+ u32 queue;
+
+ /* Free RX queue resources */
+ for (queue = 0; queue < rx_count; queue++) {
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+
+ /* Release the DMA RX socket buffers */
+ dma_free_rx_skbufs(priv, queue);
+
+ /* Free DMA regions of consistent memory previously allocated */
+ if (!priv->extend_desc)
+ dma_free_coherent(priv->device,
+ DMA_RX_SIZE * sizeof(struct dma_desc),
+ rx_q->dma_rx, rx_q->dma_rx_phy);
+ else
+ dma_free_coherent(priv->device, DMA_RX_SIZE *
+ sizeof(struct dma_extended_desc),
+ rx_q->dma_erx, rx_q->dma_rx_phy);
+
+ kfree(rx_q->rx_skbuff_dma);
+ kfree(rx_q->rx_skbuff);
+ }
+}
+
+/**
+ * free_dma_tx_desc_resources - free TX dma desc resources
+ * @priv: private structure
+ */
+static void free_dma_tx_desc_resources(struct stmmac_priv *priv)
+{
+ u32 tx_count = priv->plat->tx_queues_to_use;
+ u32 queue = 0;
+
+ /* Free TX queue resources */
+ for (queue = 0; queue < tx_count; queue++) {
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
+
+ /* Release the DMA TX socket buffers */
+ dma_free_tx_skbufs(priv, queue);
+
+ /* Free DMA regions of consistent memory previously allocated */
+ if (!priv->extend_desc)
+ dma_free_coherent(priv->device,
+ DMA_TX_SIZE * sizeof(struct dma_desc),
+ tx_q->dma_tx, tx_q->dma_tx_phy);
+ else
+ dma_free_coherent(priv->device, DMA_TX_SIZE *
+ sizeof(struct dma_extended_desc),
+ tx_q->dma_etx, tx_q->dma_tx_phy);
+
+ kfree(tx_q->tx_skbuff_dma);
+ kfree(tx_q->tx_skbuff);
+ }
+}
+
+/**
+ * alloc_dma_rx_desc_resources - alloc RX resources.
+ * @priv: private structure
+ * Description: according to which descriptor can be used (extend or basic)
+ * this function allocates the resources for TX and RX paths. In case of
+ * reception, for example, it pre-allocated the RX socket buffer in order to
+ * allow zero-copy mechanism.
+ */
+static int alloc_dma_rx_desc_resources(struct stmmac_priv *priv)
+{
+ u32 rx_count = priv->plat->rx_queues_to_use;
+ int ret = -ENOMEM;
+ u32 queue;
+
+ /* RX queues buffers and DMA */
+ for (queue = 0; queue < rx_count; queue++) {
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+
+ rx_q->queue_index = queue;
+ rx_q->priv_data = priv;
+
+ rx_q->rx_skbuff_dma = kmalloc_array(DMA_RX_SIZE,
+ sizeof(dma_addr_t),
+ GFP_KERNEL);
+ if (!rx_q->rx_skbuff_dma)
+ return -ENOMEM;
+
+ rx_q->rx_skbuff = kmalloc_array(DMA_RX_SIZE,
+ sizeof(struct sk_buff *),
+ GFP_KERNEL);
+ if (!rx_q->rx_skbuff)
+ goto err_dma;
+
+ if (priv->extend_desc) {
+ rx_q->dma_erx = dma_zalloc_coherent(priv->device,
+ DMA_RX_SIZE *
+ sizeof(struct
+ dma_extended_desc),
+ &rx_q->dma_rx_phy,
+ GFP_KERNEL);
+ if (!rx_q->dma_erx)
+ goto err_dma;
+
+ } else {
+ rx_q->dma_rx = dma_zalloc_coherent(priv->device,
+ DMA_RX_SIZE *
+ sizeof(struct
+ dma_desc),
+ &rx_q->dma_rx_phy,
+ GFP_KERNEL);
+ if (!rx_q->dma_rx)
+ goto err_dma;
}
}
+
+ return 0;
+
+err_dma:
+ free_dma_rx_desc_resources(priv);
+
+ return ret;
+}
+
+/**
+ * alloc_dma_tx_desc_resources - alloc TX resources.
+ * @priv: private structure
+ * Description: according to which descriptor can be used (extend or basic)
+ * this function allocates the resources for TX and RX paths. In case of
+ * reception, for example, it pre-allocated the RX socket buffer in order to
+ * allow zero-copy mechanism.
+ */
+static int alloc_dma_tx_desc_resources(struct stmmac_priv *priv)
+{
+ u32 tx_count = priv->plat->tx_queues_to_use;
+ int ret = -ENOMEM;
+ u32 queue;
+
+ /* TX queues buffers and DMA */
+ for (queue = 0; queue < tx_count; queue++) {
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
+
+ tx_q->queue_index = queue;
+ tx_q->priv_data = priv;
+
+ tx_q->tx_skbuff_dma = kmalloc_array(DMA_TX_SIZE,
+ sizeof(*tx_q->tx_skbuff_dma),
+ GFP_KERNEL);
+ if (!tx_q->tx_skbuff_dma)
+ return -ENOMEM;
+
+ tx_q->tx_skbuff = kmalloc_array(DMA_TX_SIZE,
+ sizeof(struct sk_buff *),
+ GFP_KERNEL);
+ if (!tx_q->tx_skbuff)
+ goto err_dma_buffers;
+
+ if (priv->extend_desc) {
+ tx_q->dma_etx = dma_zalloc_coherent(priv->device,
+ DMA_TX_SIZE *
+ sizeof(struct
+ dma_extended_desc),
+ &tx_q->dma_tx_phy,
+ GFP_KERNEL);
+ if (!tx_q->dma_etx)
+ goto err_dma_buffers;
+ } else {
+ tx_q->dma_tx = dma_zalloc_coherent(priv->device,
+ DMA_TX_SIZE *
+ sizeof(struct
+ dma_desc),
+ &tx_q->dma_tx_phy,
+ GFP_KERNEL);
+ if (!tx_q->dma_tx)
+ goto err_dma_buffers;
+ }
+ }
+
+ return 0;
+
+err_dma_buffers:
+ free_dma_tx_desc_resources(priv);
+
+ return ret;
}
/**
@@ -1145,108 +1600,28 @@
*/
static int alloc_dma_desc_resources(struct stmmac_priv *priv)
{
- int ret = -ENOMEM;
+ /* RX Allocation */
+ int ret = alloc_dma_rx_desc_resources(priv);
- priv->rx_skbuff_dma = kmalloc_array(DMA_RX_SIZE, sizeof(dma_addr_t),
- GFP_KERNEL);
- if (!priv->rx_skbuff_dma)
- return -ENOMEM;
+ if (ret)
+ return ret;
- priv->rx_skbuff = kmalloc_array(DMA_RX_SIZE, sizeof(struct sk_buff *),
- GFP_KERNEL);
- if (!priv->rx_skbuff)
- goto err_rx_skbuff;
+ ret = alloc_dma_tx_desc_resources(priv);
- priv->tx_skbuff_dma = kmalloc_array(DMA_TX_SIZE,
- sizeof(*priv->tx_skbuff_dma),
- GFP_KERNEL);
- if (!priv->tx_skbuff_dma)
- goto err_tx_skbuff_dma;
-
- priv->tx_skbuff = kmalloc_array(DMA_TX_SIZE, sizeof(struct sk_buff *),
- GFP_KERNEL);
- if (!priv->tx_skbuff)
- goto err_tx_skbuff;
-
- if (priv->extend_desc) {
- priv->dma_erx = dma_zalloc_coherent(priv->device, DMA_RX_SIZE *
- sizeof(struct
- dma_extended_desc),
- &priv->dma_rx_phy,
- GFP_KERNEL);
- if (!priv->dma_erx)
- goto err_dma;
-
- priv->dma_etx = dma_zalloc_coherent(priv->device, DMA_TX_SIZE *
- sizeof(struct
- dma_extended_desc),
- &priv->dma_tx_phy,
- GFP_KERNEL);
- if (!priv->dma_etx) {
- dma_free_coherent(priv->device, DMA_RX_SIZE *
- sizeof(struct dma_extended_desc),
- priv->dma_erx, priv->dma_rx_phy);
- goto err_dma;
- }
- } else {
- priv->dma_rx = dma_zalloc_coherent(priv->device, DMA_RX_SIZE *
- sizeof(struct dma_desc),
- &priv->dma_rx_phy,
- GFP_KERNEL);
- if (!priv->dma_rx)
- goto err_dma;
-
- priv->dma_tx = dma_zalloc_coherent(priv->device, DMA_TX_SIZE *
- sizeof(struct dma_desc),
- &priv->dma_tx_phy,
- GFP_KERNEL);
- if (!priv->dma_tx) {
- dma_free_coherent(priv->device, DMA_RX_SIZE *
- sizeof(struct dma_desc),
- priv->dma_rx, priv->dma_rx_phy);
- goto err_dma;
- }
- }
-
- return 0;
-
-err_dma:
- kfree(priv->tx_skbuff);
-err_tx_skbuff:
- kfree(priv->tx_skbuff_dma);
-err_tx_skbuff_dma:
- kfree(priv->rx_skbuff);
-err_rx_skbuff:
- kfree(priv->rx_skbuff_dma);
return ret;
}
+/**
+ * free_dma_desc_resources - free dma desc resources
+ * @priv: private structure
+ */
static void free_dma_desc_resources(struct stmmac_priv *priv)
{
- /* Release the DMA TX/RX socket buffers */
- dma_free_rx_skbufs(priv);
- dma_free_tx_skbufs(priv);
+ /* Release the DMA RX socket buffers */
+ free_dma_rx_desc_resources(priv);
- /* Free DMA regions of consistent memory previously allocated */
- if (!priv->extend_desc) {
- dma_free_coherent(priv->device,
- DMA_TX_SIZE * sizeof(struct dma_desc),
- priv->dma_tx, priv->dma_tx_phy);
- dma_free_coherent(priv->device,
- DMA_RX_SIZE * sizeof(struct dma_desc),
- priv->dma_rx, priv->dma_rx_phy);
- } else {
- dma_free_coherent(priv->device, DMA_TX_SIZE *
- sizeof(struct dma_extended_desc),
- priv->dma_etx, priv->dma_tx_phy);
- dma_free_coherent(priv->device, DMA_RX_SIZE *
- sizeof(struct dma_extended_desc),
- priv->dma_erx, priv->dma_rx_phy);
- }
- kfree(priv->rx_skbuff_dma);
- kfree(priv->rx_skbuff);
- kfree(priv->tx_skbuff_dma);
- kfree(priv->tx_skbuff);
+ /* Release the DMA TX socket buffers */
+ free_dma_tx_desc_resources(priv);
}
/**
@@ -1256,19 +1631,104 @@
*/
static void stmmac_mac_enable_rx_queues(struct stmmac_priv *priv)
{
- int rx_count = priv->dma_cap.number_rx_queues;
- int queue = 0;
+ u32 rx_queues_count = priv->plat->rx_queues_to_use;
+ int queue;
+ u8 mode;
- /* If GMAC does not have multiple queues, then this is not necessary*/
- if (rx_count == 1)
- return;
+ for (queue = 0; queue < rx_queues_count; queue++) {
+ mode = priv->plat->rx_queues_cfg[queue].mode_to_use;
+ priv->hw->mac->rx_queue_enable(priv->hw, mode, queue);
+ }
+}
- /**
- * If the core is synthesized with multiple rx queues / multiple
- * dma channels, then rx queues will be disabled by default.
- * For now only rx queue 0 is enabled.
- */
- priv->hw->mac->rx_queue_enable(priv->hw, queue);
+/**
+ * stmmac_start_rx_dma - start RX DMA channel
+ * @priv: driver private structure
+ * @chan: RX channel index
+ * Description:
+ * This starts a RX DMA channel
+ */
+static void stmmac_start_rx_dma(struct stmmac_priv *priv, u32 chan)
+{
+ netdev_dbg(priv->dev, "DMA RX processes started in channel %d\n", chan);
+ priv->hw->dma->start_rx(priv->ioaddr, chan);
+}
+
+/**
+ * stmmac_start_tx_dma - start TX DMA channel
+ * @priv: driver private structure
+ * @chan: TX channel index
+ * Description:
+ * This starts a TX DMA channel
+ */
+static void stmmac_start_tx_dma(struct stmmac_priv *priv, u32 chan)
+{
+ netdev_dbg(priv->dev, "DMA TX processes started in channel %d\n", chan);
+ priv->hw->dma->start_tx(priv->ioaddr, chan);
+}
+
+/**
+ * stmmac_stop_rx_dma - stop RX DMA channel
+ * @priv: driver private structure
+ * @chan: RX channel index
+ * Description:
+ * This stops a RX DMA channel
+ */
+static void stmmac_stop_rx_dma(struct stmmac_priv *priv, u32 chan)
+{
+ netdev_dbg(priv->dev, "DMA RX processes stopped in channel %d\n", chan);
+ priv->hw->dma->stop_rx(priv->ioaddr, chan);
+}
+
+/**
+ * stmmac_stop_tx_dma - stop TX DMA channel
+ * @priv: driver private structure
+ * @chan: TX channel index
+ * Description:
+ * This stops a TX DMA channel
+ */
+static void stmmac_stop_tx_dma(struct stmmac_priv *priv, u32 chan)
+{
+ netdev_dbg(priv->dev, "DMA TX processes stopped in channel %d\n", chan);
+ priv->hw->dma->stop_tx(priv->ioaddr, chan);
+}
+
+/**
+ * stmmac_start_all_dma - start all RX and TX DMA channels
+ * @priv: driver private structure
+ * Description:
+ * This starts all the RX and TX DMA channels
+ */
+static void stmmac_start_all_dma(struct stmmac_priv *priv)
+{
+ u32 rx_channels_count = priv->plat->rx_queues_to_use;
+ u32 tx_channels_count = priv->plat->tx_queues_to_use;
+ u32 chan = 0;
+
+ for (chan = 0; chan < rx_channels_count; chan++)
+ stmmac_start_rx_dma(priv, chan);
+
+ for (chan = 0; chan < tx_channels_count; chan++)
+ stmmac_start_tx_dma(priv, chan);
+}
+
+/**
+ * stmmac_stop_all_dma - stop all RX and TX DMA channels
+ * @priv: driver private structure
+ * Description:
+ * This stops the RX and TX DMA channels
+ */
+static void stmmac_stop_all_dma(struct stmmac_priv *priv)
+{
+ u32 rx_channels_count = priv->plat->rx_queues_to_use;
+ u32 tx_channels_count = priv->plat->tx_queues_to_use;
+ u32 chan = 0;
+
+ for (chan = 0; chan < rx_channels_count; chan++)
+ stmmac_stop_rx_dma(priv, chan);
+
+ for (chan = 0; chan < tx_channels_count; chan++)
+ stmmac_stop_tx_dma(priv, chan);
}
/**
@@ -1279,11 +1739,20 @@
*/
static void stmmac_dma_operation_mode(struct stmmac_priv *priv)
{
+ u32 rx_channels_count = priv->plat->rx_queues_to_use;
+ u32 tx_channels_count = priv->plat->tx_queues_to_use;
int rxfifosz = priv->plat->rx_fifo_size;
+ u32 txmode = 0;
+ u32 rxmode = 0;
+ u32 chan = 0;
- if (priv->plat->force_thresh_dma_mode)
- priv->hw->dma->dma_mode(priv->ioaddr, tc, tc, rxfifosz);
- else if (priv->plat->force_sf_dma_mode || priv->plat->tx_coe) {
+ if (rxfifosz == 0)
+ rxfifosz = priv->dma_cap.rx_fifo_size;
+
+ if (priv->plat->force_thresh_dma_mode) {
+ txmode = tc;
+ rxmode = tc;
+ } else if (priv->plat->force_sf_dma_mode || priv->plat->tx_coe) {
/*
* In case of GMAC, SF mode can be enabled
* to perform the TX COE in HW. This depends on:
@@ -1291,37 +1760,53 @@
* 2) There is no bugged Jumbo frame support
* that needs to not insert csum in the TDES.
*/
- priv->hw->dma->dma_mode(priv->ioaddr, SF_DMA_MODE, SF_DMA_MODE,
- rxfifosz);
+ txmode = SF_DMA_MODE;
+ rxmode = SF_DMA_MODE;
priv->xstats.threshold = SF_DMA_MODE;
- } else
- priv->hw->dma->dma_mode(priv->ioaddr, tc, SF_DMA_MODE,
+ } else {
+ txmode = tc;
+ rxmode = SF_DMA_MODE;
+ }
+
+ /* configure all channels */
+ if (priv->synopsys_id >= DWMAC_CORE_4_00) {
+ for (chan = 0; chan < rx_channels_count; chan++)
+ priv->hw->dma->dma_rx_mode(priv->ioaddr, rxmode, chan,
+ rxfifosz);
+
+ for (chan = 0; chan < tx_channels_count; chan++)
+ priv->hw->dma->dma_tx_mode(priv->ioaddr, txmode, chan);
+ } else {
+ priv->hw->dma->dma_mode(priv->ioaddr, txmode, rxmode,
rxfifosz);
+ }
}
/**
* stmmac_tx_clean - to manage the transmission completion
* @priv: driver private structure
+ * @queue: TX queue index
* Description: it reclaims the transmit resources after transmission completes.
*/
-static void stmmac_tx_clean(struct stmmac_priv *priv)
+static void stmmac_tx_clean(struct stmmac_priv *priv, u32 queue)
{
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
unsigned int bytes_compl = 0, pkts_compl = 0;
- unsigned int entry = priv->dirty_tx;
+ unsigned int entry = tx_q->dirty_tx;
netif_tx_lock(priv->dev);
priv->xstats.tx_clean++;
- while (entry != priv->cur_tx) {
- struct sk_buff *skb = priv->tx_skbuff[entry];
+ while (entry != tx_q->cur_tx) {
+ struct sk_buff *skb = tx_q->tx_skbuff[entry];
struct dma_desc *p;
int status;
if (priv->extend_desc)
- p = (struct dma_desc *)(priv->dma_etx + entry);
+ p = (struct dma_desc *)(tx_q->dma_etx + entry);
else
- p = priv->dma_tx + entry;
+ p = tx_q->dma_tx + entry;
status = priv->hw->desc->tx_status(&priv->dev->stats,
&priv->xstats, p,
@@ -1342,48 +1827,51 @@
stmmac_get_tx_hwtstamp(priv, p, skb);
}
- if (likely(priv->tx_skbuff_dma[entry].buf)) {
- if (priv->tx_skbuff_dma[entry].map_as_page)
+ if (likely(tx_q->tx_skbuff_dma[entry].buf)) {
+ if (tx_q->tx_skbuff_dma[entry].map_as_page)
dma_unmap_page(priv->device,
- priv->tx_skbuff_dma[entry].buf,
- priv->tx_skbuff_dma[entry].len,
+ tx_q->tx_skbuff_dma[entry].buf,
+ tx_q->tx_skbuff_dma[entry].len,
DMA_TO_DEVICE);
else
dma_unmap_single(priv->device,
- priv->tx_skbuff_dma[entry].buf,
- priv->tx_skbuff_dma[entry].len,
+ tx_q->tx_skbuff_dma[entry].buf,
+ tx_q->tx_skbuff_dma[entry].len,
DMA_TO_DEVICE);
- priv->tx_skbuff_dma[entry].buf = 0;
- priv->tx_skbuff_dma[entry].len = 0;
- priv->tx_skbuff_dma[entry].map_as_page = false;
+ tx_q->tx_skbuff_dma[entry].buf = 0;
+ tx_q->tx_skbuff_dma[entry].len = 0;
+ tx_q->tx_skbuff_dma[entry].map_as_page = false;
}
if (priv->hw->mode->clean_desc3)
- priv->hw->mode->clean_desc3(priv, p);
+ priv->hw->mode->clean_desc3(tx_q, p);
- priv->tx_skbuff_dma[entry].last_segment = false;
- priv->tx_skbuff_dma[entry].is_jumbo = false;
+ tx_q->tx_skbuff_dma[entry].last_segment = false;
+ tx_q->tx_skbuff_dma[entry].is_jumbo = false;
if (likely(skb != NULL)) {
pkts_compl++;
bytes_compl += skb->len;
dev_consume_skb_any(skb);
- priv->tx_skbuff[entry] = NULL;
+ tx_q->tx_skbuff[entry] = NULL;
}
priv->hw->desc->release_tx_desc(p, priv->mode);
entry = STMMAC_GET_ENTRY(entry, DMA_TX_SIZE);
}
- priv->dirty_tx = entry;
+ tx_q->dirty_tx = entry;
- netdev_completed_queue(priv->dev, pkts_compl, bytes_compl);
+ netdev_tx_completed_queue(netdev_get_tx_queue(priv->dev, queue),
+ pkts_compl, bytes_compl);
- if (unlikely(netif_queue_stopped(priv->dev) &&
- stmmac_tx_avail(priv) > STMMAC_TX_THRESH)) {
+ if (unlikely(netif_tx_queue_stopped(netdev_get_tx_queue(priv->dev,
+ queue))) &&
+ stmmac_tx_avail(priv, queue) > STMMAC_TX_THRESH) {
+
netif_dbg(priv, tx_done, priv->dev,
"%s: restart transmit\n", __func__);
- netif_wake_queue(priv->dev);
+ netif_tx_wake_queue(netdev_get_tx_queue(priv->dev, queue));
}
if ((priv->eee_enabled) && (!priv->tx_path_in_lpi_mode)) {
@@ -1393,45 +1881,76 @@
netif_tx_unlock(priv->dev);
}
-static inline void stmmac_enable_dma_irq(struct stmmac_priv *priv)
+static inline void stmmac_enable_dma_irq(struct stmmac_priv *priv, u32 chan)
{
- priv->hw->dma->enable_dma_irq(priv->ioaddr);
+ priv->hw->dma->enable_dma_irq(priv->ioaddr, chan);
}
-static inline void stmmac_disable_dma_irq(struct stmmac_priv *priv)
+static inline void stmmac_disable_dma_irq(struct stmmac_priv *priv, u32 chan)
{
- priv->hw->dma->disable_dma_irq(priv->ioaddr);
+ priv->hw->dma->disable_dma_irq(priv->ioaddr, chan);
}
/**
* stmmac_tx_err - to manage the tx error
* @priv: driver private structure
+ * @chan: channel index
* Description: it cleans the descriptors and restarts the transmission
* in case of transmission errors.
*/
-static void stmmac_tx_err(struct stmmac_priv *priv)
+static void stmmac_tx_err(struct stmmac_priv *priv, u32 chan)
{
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[chan];
int i;
- netif_stop_queue(priv->dev);
- priv->hw->dma->stop_tx(priv->ioaddr);
- dma_free_tx_skbufs(priv);
+ netif_tx_stop_queue(netdev_get_tx_queue(priv->dev, chan));
+
+ stmmac_stop_tx_dma(priv, chan);
+ dma_free_tx_skbufs(priv, chan);
for (i = 0; i < DMA_TX_SIZE; i++)
if (priv->extend_desc)
- priv->hw->desc->init_tx_desc(&priv->dma_etx[i].basic,
+ priv->hw->desc->init_tx_desc(&tx_q->dma_etx[i].basic,
priv->mode,
(i == DMA_TX_SIZE - 1));
else
- priv->hw->desc->init_tx_desc(&priv->dma_tx[i],
+ priv->hw->desc->init_tx_desc(&tx_q->dma_tx[i],
priv->mode,
(i == DMA_TX_SIZE - 1));
- priv->dirty_tx = 0;
- priv->cur_tx = 0;
- netdev_reset_queue(priv->dev);
- priv->hw->dma->start_tx(priv->ioaddr);
+ tx_q->dirty_tx = 0;
+ tx_q->cur_tx = 0;
+ netdev_tx_reset_queue(netdev_get_tx_queue(priv->dev, chan));
+ stmmac_start_tx_dma(priv, chan);
priv->dev->stats.tx_errors++;
- netif_wake_queue(priv->dev);
+ netif_tx_wake_queue(netdev_get_tx_queue(priv->dev, chan));
+}
+
+/**
+ * stmmac_set_dma_operation_mode - Set DMA operation mode by channel
+ * @priv: driver private structure
+ * @txmode: TX operating mode
+ * @rxmode: RX operating mode
+ * @chan: channel index
+ * Description: it is used for configuring of the DMA operation mode in
+ * runtime in order to program the tx/rx DMA thresholds or Store-And-Forward
+ * mode.
+ */
+static void stmmac_set_dma_operation_mode(struct stmmac_priv *priv, u32 txmode,
+ u32 rxmode, u32 chan)
+{
+ int rxfifosz = priv->plat->rx_fifo_size;
+
+ if (rxfifosz == 0)
+ rxfifosz = priv->dma_cap.rx_fifo_size;
+
+ if (priv->synopsys_id >= DWMAC_CORE_4_00) {
+ priv->hw->dma->dma_rx_mode(priv->ioaddr, rxmode, chan,
+ rxfifosz);
+ priv->hw->dma->dma_tx_mode(priv->ioaddr, txmode, chan);
+ } else {
+ priv->hw->dma->dma_mode(priv->ioaddr, txmode, rxmode,
+ rxfifosz);
+ }
}
/**
@@ -1443,31 +1962,43 @@
*/
static void stmmac_dma_interrupt(struct stmmac_priv *priv)
{
+ u32 tx_channel_count = priv->plat->tx_queues_to_use;
int status;
- int rxfifosz = priv->plat->rx_fifo_size;
+ u32 chan;
- status = priv->hw->dma->dma_interrupt(priv->ioaddr, &priv->xstats);
- if (likely((status & handle_rx)) || (status & handle_tx)) {
- if (likely(napi_schedule_prep(&priv->napi))) {
- stmmac_disable_dma_irq(priv);
- __napi_schedule(&priv->napi);
+ for (chan = 0; chan < tx_channel_count; chan++) {
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[chan];
+
+ status = priv->hw->dma->dma_interrupt(priv->ioaddr,
+ &priv->xstats, chan);
+ if (likely((status & handle_rx)) || (status & handle_tx)) {
+ if (likely(napi_schedule_prep(&rx_q->napi))) {
+ stmmac_disable_dma_irq(priv, chan);
+ __napi_schedule(&rx_q->napi);
+ }
+ }
+
+ if (unlikely(status & tx_hard_error_bump_tc)) {
+ /* Try to bump up the dma threshold on this failure */
+ if (unlikely(priv->xstats.threshold != SF_DMA_MODE) &&
+ (tc <= 256)) {
+ tc += 64;
+ if (priv->plat->force_thresh_dma_mode)
+ stmmac_set_dma_operation_mode(priv,
+ tc,
+ tc,
+ chan);
+ else
+ stmmac_set_dma_operation_mode(priv,
+ tc,
+ SF_DMA_MODE,
+ chan);
+ priv->xstats.threshold = tc;
+ }
+ } else if (unlikely(status == tx_hard_error)) {
+ stmmac_tx_err(priv, chan);
}
}
- if (unlikely(status & tx_hard_error_bump_tc)) {
- /* Try to bump up the dma threshold on this failure */
- if (unlikely(priv->xstats.threshold != SF_DMA_MODE) &&
- (tc <= 256)) {
- tc += 64;
- if (priv->plat->force_thresh_dma_mode)
- priv->hw->dma->dma_mode(priv->ioaddr, tc, tc,
- rxfifosz);
- else
- priv->hw->dma->dma_mode(priv->ioaddr, tc,
- SF_DMA_MODE, rxfifosz);
- priv->xstats.threshold = tc;
- }
- } else if (unlikely(status == tx_hard_error))
- stmmac_tx_err(priv);
}
/**
@@ -1574,6 +2105,13 @@
*/
static int stmmac_init_dma_engine(struct stmmac_priv *priv)
{
+ u32 rx_channels_count = priv->plat->rx_queues_to_use;
+ u32 tx_channels_count = priv->plat->tx_queues_to_use;
+ struct stmmac_rx_queue *rx_q;
+ struct stmmac_tx_queue *tx_q;
+ u32 dummy_dma_rx_phy = 0;
+ u32 dummy_dma_tx_phy = 0;
+ u32 chan = 0;
int atds = 0;
int ret = 0;
@@ -1591,19 +2129,49 @@
return ret;
}
- priv->hw->dma->init(priv->ioaddr, priv->plat->dma_cfg,
- priv->dma_tx_phy, priv->dma_rx_phy, atds);
-
if (priv->synopsys_id >= DWMAC_CORE_4_00) {
- priv->rx_tail_addr = priv->dma_rx_phy +
- (DMA_RX_SIZE * sizeof(struct dma_desc));
- priv->hw->dma->set_rx_tail_ptr(priv->ioaddr, priv->rx_tail_addr,
- STMMAC_CHAN0);
+ /* DMA Configuration */
+ priv->hw->dma->init(priv->ioaddr, priv->plat->dma_cfg,
+ dummy_dma_tx_phy, dummy_dma_rx_phy, atds);
- priv->tx_tail_addr = priv->dma_tx_phy +
- (DMA_TX_SIZE * sizeof(struct dma_desc));
- priv->hw->dma->set_tx_tail_ptr(priv->ioaddr, priv->tx_tail_addr,
- STMMAC_CHAN0);
+ /* DMA RX Channel Configuration */
+ for (chan = 0; chan < rx_channels_count; chan++) {
+ rx_q = &priv->rx_queue[chan];
+
+ priv->hw->dma->init_rx_chan(priv->ioaddr,
+ priv->plat->dma_cfg,
+ rx_q->dma_rx_phy, chan);
+
+ rx_q->rx_tail_addr = rx_q->dma_rx_phy +
+ (DMA_RX_SIZE * sizeof(struct dma_desc));
+ priv->hw->dma->set_rx_tail_ptr(priv->ioaddr,
+ rx_q->rx_tail_addr,
+ chan);
+ }
+
+ /* DMA TX Channel Configuration */
+ for (chan = 0; chan < tx_channels_count; chan++) {
+ tx_q = &priv->tx_queue[chan];
+
+ priv->hw->dma->init_chan(priv->ioaddr,
+ priv->plat->dma_cfg,
+ chan);
+
+ priv->hw->dma->init_tx_chan(priv->ioaddr,
+ priv->plat->dma_cfg,
+ tx_q->dma_tx_phy, chan);
+
+ tx_q->tx_tail_addr = tx_q->dma_tx_phy +
+ (DMA_TX_SIZE * sizeof(struct dma_desc));
+ priv->hw->dma->set_tx_tail_ptr(priv->ioaddr,
+ tx_q->tx_tail_addr,
+ chan);
+ }
+ } else {
+ rx_q = &priv->rx_queue[chan];
+ tx_q = &priv->tx_queue[chan];
+ priv->hw->dma->init(priv->ioaddr, priv->plat->dma_cfg,
+ tx_q->dma_tx_phy, rx_q->dma_rx_phy, atds);
}
if (priv->plat->axi && priv->hw->dma->axi)
@@ -1621,8 +2189,12 @@
static void stmmac_tx_timer(unsigned long data)
{
struct stmmac_priv *priv = (struct stmmac_priv *)data;
+ u32 tx_queues_count = priv->plat->tx_queues_to_use;
+ u32 queue;
- stmmac_tx_clean(priv);
+ /* let's scan all the tx queues */
+ for (queue = 0; queue < tx_queues_count; queue++)
+ stmmac_tx_clean(priv, queue);
}
/**
@@ -1644,6 +2216,196 @@
add_timer(&priv->txtimer);
}
+static void stmmac_set_rings_length(struct stmmac_priv *priv)
+{
+ u32 rx_channels_count = priv->plat->rx_queues_to_use;
+ u32 tx_channels_count = priv->plat->tx_queues_to_use;
+ u32 chan;
+
+ /* set TX ring length */
+ if (priv->hw->dma->set_tx_ring_len) {
+ for (chan = 0; chan < tx_channels_count; chan++)
+ priv->hw->dma->set_tx_ring_len(priv->ioaddr,
+ (DMA_TX_SIZE - 1), chan);
+ }
+
+ /* set RX ring length */
+ if (priv->hw->dma->set_rx_ring_len) {
+ for (chan = 0; chan < rx_channels_count; chan++)
+ priv->hw->dma->set_rx_ring_len(priv->ioaddr,
+ (DMA_RX_SIZE - 1), chan);
+ }
+}
+
+/**
+ * stmmac_set_tx_queue_weight - Set TX queue weight
+ * @priv: driver private structure
+ * Description: It is used for setting TX queues weight
+ */
+static void stmmac_set_tx_queue_weight(struct stmmac_priv *priv)
+{
+ u32 tx_queues_count = priv->plat->tx_queues_to_use;
+ u32 weight;
+ u32 queue;
+
+ for (queue = 0; queue < tx_queues_count; queue++) {
+ weight = priv->plat->tx_queues_cfg[queue].weight;
+ priv->hw->mac->set_mtl_tx_queue_weight(priv->hw, weight, queue);
+ }
+}
+
+/**
+ * stmmac_configure_cbs - Configure CBS in TX queue
+ * @priv: driver private structure
+ * Description: It is used for configuring CBS in AVB TX queues
+ */
+static void stmmac_configure_cbs(struct stmmac_priv *priv)
+{
+ u32 tx_queues_count = priv->plat->tx_queues_to_use;
+ u32 mode_to_use;
+ u32 queue;
+
+ /* queue 0 is reserved for legacy traffic */
+ for (queue = 1; queue < tx_queues_count; queue++) {
+ mode_to_use = priv->plat->tx_queues_cfg[queue].mode_to_use;
+ if (mode_to_use == MTL_QUEUE_DCB)
+ continue;
+
+ priv->hw->mac->config_cbs(priv->hw,
+ priv->plat->tx_queues_cfg[queue].send_slope,
+ priv->plat->tx_queues_cfg[queue].idle_slope,
+ priv->plat->tx_queues_cfg[queue].high_credit,
+ priv->plat->tx_queues_cfg[queue].low_credit,
+ queue);
+ }
+}
+
+/**
+ * stmmac_rx_queue_dma_chan_map - Map RX queue to RX dma channel
+ * @priv: driver private structure
+ * Description: It is used for mapping RX queues to RX dma channels
+ */
+static void stmmac_rx_queue_dma_chan_map(struct stmmac_priv *priv)
+{
+ u32 rx_queues_count = priv->plat->rx_queues_to_use;
+ u32 queue;
+ u32 chan;
+
+ for (queue = 0; queue < rx_queues_count; queue++) {
+ chan = priv->plat->rx_queues_cfg[queue].chan;
+ priv->hw->mac->map_mtl_to_dma(priv->hw, queue, chan);
+ }
+}
+
+/**
+ * stmmac_mac_config_rx_queues_prio - Configure RX Queue priority
+ * @priv: driver private structure
+ * Description: It is used for configuring the RX Queue Priority
+ */
+static void stmmac_mac_config_rx_queues_prio(struct stmmac_priv *priv)
+{
+ u32 rx_queues_count = priv->plat->rx_queues_to_use;
+ u32 queue;
+ u32 prio;
+
+ for (queue = 0; queue < rx_queues_count; queue++) {
+ if (!priv->plat->rx_queues_cfg[queue].use_prio)
+ continue;
+
+ prio = priv->plat->rx_queues_cfg[queue].prio;
+ priv->hw->mac->rx_queue_prio(priv->hw, prio, queue);
+ }
+}
+
+/**
+ * stmmac_mac_config_tx_queues_prio - Configure TX Queue priority
+ * @priv: driver private structure
+ * Description: It is used for configuring the TX Queue Priority
+ */
+static void stmmac_mac_config_tx_queues_prio(struct stmmac_priv *priv)
+{
+ u32 tx_queues_count = priv->plat->tx_queues_to_use;
+ u32 queue;
+ u32 prio;
+
+ for (queue = 0; queue < tx_queues_count; queue++) {
+ if (!priv->plat->tx_queues_cfg[queue].use_prio)
+ continue;
+
+ prio = priv->plat->tx_queues_cfg[queue].prio;
+ priv->hw->mac->tx_queue_prio(priv->hw, prio, queue);
+ }
+}
+
+/**
+ * stmmac_mac_config_rx_queues_routing - Configure RX Queue Routing
+ * @priv: driver private structure
+ * Description: It is used for configuring the RX queue routing
+ */
+static void stmmac_mac_config_rx_queues_routing(struct stmmac_priv *priv)
+{
+ u32 rx_queues_count = priv->plat->rx_queues_to_use;
+ u32 queue;
+ u8 packet;
+
+ for (queue = 0; queue < rx_queues_count; queue++) {
+ /* no specific packet type routing specified for the queue */
+ if (priv->plat->rx_queues_cfg[queue].pkt_route == 0x0)
+ continue;
+
+ packet = priv->plat->rx_queues_cfg[queue].pkt_route;
+ priv->hw->mac->rx_queue_prio(priv->hw, packet, queue);
+ }
+}
+
+/**
+ * stmmac_mtl_configuration - Configure MTL
+ * @priv: driver private structure
+ * Description: It is used for configurring MTL
+ */
+static void stmmac_mtl_configuration(struct stmmac_priv *priv)
+{
+ u32 rx_queues_count = priv->plat->rx_queues_to_use;
+ u32 tx_queues_count = priv->plat->tx_queues_to_use;
+
+ if (tx_queues_count > 1 && priv->hw->mac->set_mtl_tx_queue_weight)
+ stmmac_set_tx_queue_weight(priv);
+
+ /* Configure MTL RX algorithms */
+ if (rx_queues_count > 1 && priv->hw->mac->prog_mtl_rx_algorithms)
+ priv->hw->mac->prog_mtl_rx_algorithms(priv->hw,
+ priv->plat->rx_sched_algorithm);
+
+ /* Configure MTL TX algorithms */
+ if (tx_queues_count > 1 && priv->hw->mac->prog_mtl_tx_algorithms)
+ priv->hw->mac->prog_mtl_tx_algorithms(priv->hw,
+ priv->plat->tx_sched_algorithm);
+
+ /* Configure CBS in AVB TX queues */
+ if (tx_queues_count > 1 && priv->hw->mac->config_cbs)
+ stmmac_configure_cbs(priv);
+
+ /* Map RX MTL to DMA channels */
+ if (priv->hw->mac->map_mtl_to_dma)
+ stmmac_rx_queue_dma_chan_map(priv);
+
+ /* Enable MAC RX Queues */
+ if (priv->hw->mac->rx_queue_enable)
+ stmmac_mac_enable_rx_queues(priv);
+
+ /* Set RX priorities */
+ if (rx_queues_count > 1 && priv->hw->mac->rx_queue_prio)
+ stmmac_mac_config_rx_queues_prio(priv);
+
+ /* Set TX priorities */
+ if (tx_queues_count > 1 && priv->hw->mac->tx_queue_prio)
+ stmmac_mac_config_tx_queues_prio(priv);
+
+ /* Set RX routing */
+ if (rx_queues_count > 1 && priv->hw->mac->rx_queue_routing)
+ stmmac_mac_config_rx_queues_routing(priv);
+}
+
/**
* stmmac_hw_setup - setup mac in a usable state.
* @dev : pointer to the device structure.
@@ -1659,6 +2421,9 @@
static int stmmac_hw_setup(struct net_device *dev, bool init_ptp)
{
struct stmmac_priv *priv = netdev_priv(dev);
+ u32 rx_cnt = priv->plat->rx_queues_to_use;
+ u32 tx_cnt = priv->plat->tx_queues_to_use;
+ u32 chan;
int ret;
/* DMA initialization and SW reset */
@@ -1688,9 +2453,9 @@
/* Initialize the MAC Core */
priv->hw->mac->core_init(priv->hw, dev->mtu);
- /* Initialize MAC RX Queues */
- if (priv->hw->mac->rx_queue_enable)
- stmmac_mac_enable_rx_queues(priv);
+ /* Initialize MTL*/
+ if (priv->synopsys_id >= DWMAC_CORE_4_00)
+ stmmac_mtl_configuration(priv);
ret = priv->hw->mac->rx_ipc(priv->hw);
if (!ret) {
@@ -1700,10 +2465,7 @@
}
/* Enable the MAC Rx/Tx */
- if (priv->synopsys_id >= DWMAC_CORE_4_00)
- stmmac_dwmac4_set_mac(priv->ioaddr, true);
- else
- stmmac_set_mac(priv->ioaddr, true);
+ priv->hw->mac->set_mac(priv->ioaddr, true);
/* Set the HW DMA mode and the COE */
stmmac_dma_operation_mode(priv);
@@ -1711,6 +2473,10 @@
stmmac_mmc_setup(priv);
if (init_ptp) {
+ ret = clk_prepare_enable(priv->plat->clk_ptp_ref);
+ if (ret < 0)
+ netdev_warn(priv->dev, "failed to enable PTP reference clock: %d\n", ret);
+
ret = stmmac_init_ptp(priv);
if (ret == -EOPNOTSUPP)
netdev_warn(priv->dev, "PTP not supported by HW\n");
@@ -1725,35 +2491,37 @@
__func__);
#endif
/* Start the ball rolling... */
- netdev_dbg(priv->dev, "DMA RX/TX processes started...\n");
- priv->hw->dma->start_tx(priv->ioaddr);
- priv->hw->dma->start_rx(priv->ioaddr);
+ stmmac_start_all_dma(priv);
priv->tx_lpi_timer = STMMAC_DEFAULT_TWT_LS;
if ((priv->use_riwt) && (priv->hw->dma->rx_watchdog)) {
priv->rx_riwt = MAX_DMA_RIWT;
- priv->hw->dma->rx_watchdog(priv->ioaddr, MAX_DMA_RIWT);
+ priv->hw->dma->rx_watchdog(priv->ioaddr, MAX_DMA_RIWT, rx_cnt);
}
if (priv->hw->pcs && priv->hw->mac->pcs_ctrl_ane)
priv->hw->mac->pcs_ctrl_ane(priv->hw, 1, priv->hw->ps, 0);
- /* set TX ring length */
- if (priv->hw->dma->set_tx_ring_len)
- priv->hw->dma->set_tx_ring_len(priv->ioaddr,
- (DMA_TX_SIZE - 1));
- /* set RX ring length */
- if (priv->hw->dma->set_rx_ring_len)
- priv->hw->dma->set_rx_ring_len(priv->ioaddr,
- (DMA_RX_SIZE - 1));
+ /* set TX and RX rings length */
+ stmmac_set_rings_length(priv);
+
/* Enable TSO */
- if (priv->tso)
- priv->hw->dma->enable_tso(priv->ioaddr, 1, STMMAC_CHAN0);
+ if (priv->tso) {
+ for (chan = 0; chan < tx_cnt; chan++)
+ priv->hw->dma->enable_tso(priv->ioaddr, 1, chan);
+ }
return 0;
}
+static void stmmac_hw_teardown(struct net_device *dev)
+{
+ struct stmmac_priv *priv = netdev_priv(dev);
+
+ clk_disable_unprepare(priv->plat->clk_ptp_ref);
+}
+
/**
* stmmac_open - open entry point of the driver
* @dev : pointer to the device structure.
@@ -1821,7 +2589,7 @@
netdev_err(priv->dev,
"%s: ERROR: allocating the IRQ %d (error: %d)\n",
__func__, dev->irq, ret);
- goto init_error;
+ goto irq_error;
}
/* Request the Wake IRQ in case of another line is used for WoL */
@@ -1848,8 +2616,8 @@
}
}
- napi_enable(&priv->napi);
- netif_start_queue(dev);
+ stmmac_enable_all_queues(priv);
+ stmmac_start_all_queues(priv);
return 0;
@@ -1858,7 +2626,12 @@
free_irq(priv->wol_irq, dev);
wolirq_error:
free_irq(dev->irq, dev);
+irq_error:
+ if (dev->phydev)
+ phy_stop(dev->phydev);
+ del_timer_sync(&priv->txtimer);
+ stmmac_hw_teardown(dev);
init_error:
free_dma_desc_resources(priv);
dma_desc_error:
@@ -1887,9 +2660,9 @@
phy_disconnect(dev->phydev);
}
- netif_stop_queue(dev);
+ stmmac_stop_all_queues(priv);
- napi_disable(&priv->napi);
+ stmmac_disable_all_queues(priv);
del_timer_sync(&priv->txtimer);
@@ -1901,14 +2674,13 @@
free_irq(priv->lpi_irq, dev);
/* Stop TX/RX DMA and clear the descriptors */
- priv->hw->dma->stop_tx(priv->ioaddr);
- priv->hw->dma->stop_rx(priv->ioaddr);
+ stmmac_stop_all_dma(priv);
/* Release and free the Rx/Tx resources */
free_dma_desc_resources(priv);
/* Disable the MAC Rx/Tx */
- stmmac_set_mac(priv->ioaddr, false);
+ priv->hw->mac->set_mac(priv->ioaddr, false);
netif_carrier_off(dev);
@@ -1927,22 +2699,24 @@
* @des: buffer start address
* @total_len: total length to fill in descriptors
* @last_segmant: condition for the last descriptor
+ * @queue: TX queue index
* Description:
* This function fills descriptor and request new descriptors according to
* buffer length to fill
*/
static void stmmac_tso_allocator(struct stmmac_priv *priv, unsigned int des,
- int total_len, bool last_segment)
+ int total_len, bool last_segment, u32 queue)
{
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
struct dma_desc *desc;
- int tmp_len;
u32 buff_size;
+ int tmp_len;
tmp_len = total_len;
while (tmp_len > 0) {
- priv->cur_tx = STMMAC_GET_ENTRY(priv->cur_tx, DMA_TX_SIZE);
- desc = priv->dma_tx + priv->cur_tx;
+ tx_q->cur_tx = STMMAC_GET_ENTRY(tx_q->cur_tx, DMA_TX_SIZE);
+ desc = tx_q->dma_tx + tx_q->cur_tx;
desc->des0 = cpu_to_le32(des + (total_len - tmp_len));
buff_size = tmp_len >= TSO_MAX_BUFF_SIZE ?
@@ -1986,23 +2760,28 @@
*/
static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
{
- u32 pay_len, mss;
- int tmp_pay_len = 0;
+ struct dma_desc *desc, *first, *mss_desc = NULL;
struct stmmac_priv *priv = netdev_priv(dev);
int nfrags = skb_shinfo(skb)->nr_frags;
+ u32 queue = skb_get_queue_mapping(skb);
unsigned int first_entry, des;
- struct dma_desc *desc, *first, *mss_desc = NULL;
+ struct stmmac_tx_queue *tx_q;
+ int tmp_pay_len = 0;
+ u32 pay_len, mss;
u8 proto_hdr_len;
int i;
+ tx_q = &priv->tx_queue[queue];
+
/* Compute header lengths */
proto_hdr_len = skb_transport_offset(skb) + tcp_hdrlen(skb);
/* Desc availability based on threshold should be enough safe */
- if (unlikely(stmmac_tx_avail(priv) <
+ if (unlikely(stmmac_tx_avail(priv, queue) <
(((skb->len - proto_hdr_len) / TSO_MAX_BUFF_SIZE + 1)))) {
- if (!netif_queue_stopped(dev)) {
- netif_stop_queue(dev);
+ if (!netif_tx_queue_stopped(netdev_get_tx_queue(dev, queue))) {
+ netif_tx_stop_queue(netdev_get_tx_queue(priv->dev,
+ queue));
/* This is a hard error, log it. */
netdev_err(priv->dev,
"%s: Tx Ring full when queue awake\n",
@@ -2017,10 +2796,10 @@
/* set new MSS value if needed */
if (mss != priv->mss) {
- mss_desc = priv->dma_tx + priv->cur_tx;
+ mss_desc = tx_q->dma_tx + tx_q->cur_tx;
priv->hw->desc->set_mss(mss_desc, mss);
priv->mss = mss;
- priv->cur_tx = STMMAC_GET_ENTRY(priv->cur_tx, DMA_TX_SIZE);
+ tx_q->cur_tx = STMMAC_GET_ENTRY(tx_q->cur_tx, DMA_TX_SIZE);
}
if (netif_msg_tx_queued(priv)) {
@@ -2030,9 +2809,9 @@
skb->data_len);
}
- first_entry = priv->cur_tx;
+ first_entry = tx_q->cur_tx;
- desc = priv->dma_tx + first_entry;
+ desc = tx_q->dma_tx + first_entry;
first = desc;
/* first descriptor: fill Headers on Buf1 */
@@ -2041,9 +2820,9 @@
if (dma_mapping_error(priv->device, des))
goto dma_map_err;
- priv->tx_skbuff_dma[first_entry].buf = des;
- priv->tx_skbuff_dma[first_entry].len = skb_headlen(skb);
- priv->tx_skbuff[first_entry] = skb;
+ tx_q->tx_skbuff_dma[first_entry].buf = des;
+ tx_q->tx_skbuff_dma[first_entry].len = skb_headlen(skb);
+ tx_q->tx_skbuff[first_entry] = skb;
first->des0 = cpu_to_le32(des);
@@ -2054,7 +2833,7 @@
/* If needed take extra descriptors to fill the remaining payload */
tmp_pay_len = pay_len - TSO_MAX_BUFF_SIZE;
- stmmac_tso_allocator(priv, des, tmp_pay_len, (nfrags == 0));
+ stmmac_tso_allocator(priv, des, tmp_pay_len, (nfrags == 0), queue);
/* Prepare fragments */
for (i = 0; i < nfrags; i++) {
@@ -2063,24 +2842,26 @@
des = skb_frag_dma_map(priv->device, frag, 0,
skb_frag_size(frag),
DMA_TO_DEVICE);
+ if (dma_mapping_error(priv->device, des))
+ goto dma_map_err;
stmmac_tso_allocator(priv, des, skb_frag_size(frag),
- (i == nfrags - 1));
+ (i == nfrags - 1), queue);
- priv->tx_skbuff_dma[priv->cur_tx].buf = des;
- priv->tx_skbuff_dma[priv->cur_tx].len = skb_frag_size(frag);
- priv->tx_skbuff[priv->cur_tx] = NULL;
- priv->tx_skbuff_dma[priv->cur_tx].map_as_page = true;
+ tx_q->tx_skbuff_dma[tx_q->cur_tx].buf = des;
+ tx_q->tx_skbuff_dma[tx_q->cur_tx].len = skb_frag_size(frag);
+ tx_q->tx_skbuff[tx_q->cur_tx] = NULL;
+ tx_q->tx_skbuff_dma[tx_q->cur_tx].map_as_page = true;
}
- priv->tx_skbuff_dma[priv->cur_tx].last_segment = true;
+ tx_q->tx_skbuff_dma[tx_q->cur_tx].last_segment = true;
- priv->cur_tx = STMMAC_GET_ENTRY(priv->cur_tx, DMA_TX_SIZE);
+ tx_q->cur_tx = STMMAC_GET_ENTRY(tx_q->cur_tx, DMA_TX_SIZE);
- if (unlikely(stmmac_tx_avail(priv) <= (MAX_SKB_FRAGS + 1))) {
+ if (unlikely(stmmac_tx_avail(priv, queue) <= (MAX_SKB_FRAGS + 1))) {
netif_dbg(priv, hw, priv->dev, "%s: stop transmitted packets\n",
__func__);
- netif_stop_queue(dev);
+ netif_tx_stop_queue(netdev_get_tx_queue(priv->dev, queue));
}
dev->stats.tx_bytes += skb->len;
@@ -2112,7 +2893,7 @@
priv->hw->desc->prepare_tso_tx_desc(first, 1,
proto_hdr_len,
pay_len,
- 1, priv->tx_skbuff_dma[first_entry].last_segment,
+ 1, tx_q->tx_skbuff_dma[first_entry].last_segment,
tcp_hdrlen(skb) / 4, (skb->len - proto_hdr_len));
/* If context desc is used to change MSS */
@@ -2127,20 +2908,20 @@
if (netif_msg_pktdata(priv)) {
pr_info("%s: curr=%d dirty=%d f=%d, e=%d, f_p=%p, nfrags %d\n",
- __func__, priv->cur_tx, priv->dirty_tx, first_entry,
- priv->cur_tx, first, nfrags);
+ __func__, tx_q->cur_tx, tx_q->dirty_tx, first_entry,
+ tx_q->cur_tx, first, nfrags);
- priv->hw->desc->display_ring((void *)priv->dma_tx, DMA_TX_SIZE,
+ priv->hw->desc->display_ring((void *)tx_q->dma_tx, DMA_TX_SIZE,
0);
pr_info(">>> frame to be transmitted: ");
print_pkt(skb->data, skb_headlen(skb));
}
- netdev_sent_queue(dev, skb->len);
+ netdev_tx_sent_queue(netdev_get_tx_queue(dev, queue), skb->len);
- priv->hw->dma->set_tx_tail_ptr(priv->ioaddr, priv->tx_tail_addr,
- STMMAC_CHAN0);
+ priv->hw->dma->set_tx_tail_ptr(priv->ioaddr, tx_q->tx_tail_addr,
+ queue);
return NETDEV_TX_OK;
@@ -2164,21 +2945,26 @@
struct stmmac_priv *priv = netdev_priv(dev);
unsigned int nopaged_len = skb_headlen(skb);
int i, csum_insertion = 0, is_jumbo = 0;
+ u32 queue = skb_get_queue_mapping(skb);
int nfrags = skb_shinfo(skb)->nr_frags;
unsigned int entry, first_entry;
struct dma_desc *desc, *first;
+ struct stmmac_tx_queue *tx_q;
unsigned int enh_desc;
unsigned int des;
+ tx_q = &priv->tx_queue[queue];
+
/* Manage oversized TCP frames for GMAC4 device */
if (skb_is_gso(skb) && priv->tso) {
if (ip_hdr(skb)->protocol == IPPROTO_TCP)
return stmmac_tso_xmit(skb, dev);
}
- if (unlikely(stmmac_tx_avail(priv) < nfrags + 1)) {
- if (!netif_queue_stopped(dev)) {
- netif_stop_queue(dev);
+ if (unlikely(stmmac_tx_avail(priv, queue) < nfrags + 1)) {
+ if (!netif_tx_queue_stopped(netdev_get_tx_queue(dev, queue))) {
+ netif_tx_stop_queue(netdev_get_tx_queue(priv->dev,
+ queue));
/* This is a hard error, log it. */
netdev_err(priv->dev,
"%s: Tx Ring full when queue awake\n",
@@ -2190,19 +2976,19 @@
if (priv->tx_path_in_lpi_mode)
stmmac_disable_eee_mode(priv);
- entry = priv->cur_tx;
+ entry = tx_q->cur_tx;
first_entry = entry;
csum_insertion = (skb->ip_summed == CHECKSUM_PARTIAL);
if (likely(priv->extend_desc))
- desc = (struct dma_desc *)(priv->dma_etx + entry);
+ desc = (struct dma_desc *)(tx_q->dma_etx + entry);
else
- desc = priv->dma_tx + entry;
+ desc = tx_q->dma_tx + entry;
first = desc;
- priv->tx_skbuff[first_entry] = skb;
+ tx_q->tx_skbuff[first_entry] = skb;
enh_desc = priv->plat->enh_desc;
/* To program the descriptors according to the size of the frame */
@@ -2211,7 +2997,7 @@
if (unlikely(is_jumbo) && likely(priv->synopsys_id <
DWMAC_CORE_4_00)) {
- entry = priv->hw->mode->jumbo_frm(priv, skb, csum_insertion);
+ entry = priv->hw->mode->jumbo_frm(tx_q, skb, csum_insertion);
if (unlikely(entry < 0))
goto dma_map_err;
}
@@ -2224,48 +3010,49 @@
entry = STMMAC_GET_ENTRY(entry, DMA_TX_SIZE);
if (likely(priv->extend_desc))
- desc = (struct dma_desc *)(priv->dma_etx + entry);
+ desc = (struct dma_desc *)(tx_q->dma_etx + entry);
else
- desc = priv->dma_tx + entry;
+ desc = tx_q->dma_tx + entry;
des = skb_frag_dma_map(priv->device, frag, 0, len,
DMA_TO_DEVICE);
if (dma_mapping_error(priv->device, des))
goto dma_map_err; /* should reuse desc w/o issues */
- priv->tx_skbuff[entry] = NULL;
+ tx_q->tx_skbuff[entry] = NULL;
- priv->tx_skbuff_dma[entry].buf = des;
+ tx_q->tx_skbuff_dma[entry].buf = des;
if (unlikely(priv->synopsys_id >= DWMAC_CORE_4_00))
desc->des0 = cpu_to_le32(des);
else
desc->des2 = cpu_to_le32(des);
- priv->tx_skbuff_dma[entry].map_as_page = true;
- priv->tx_skbuff_dma[entry].len = len;
- priv->tx_skbuff_dma[entry].last_segment = last_segment;
+ tx_q->tx_skbuff_dma[entry].map_as_page = true;
+ tx_q->tx_skbuff_dma[entry].len = len;
+ tx_q->tx_skbuff_dma[entry].last_segment = last_segment;
/* Prepare the descriptor and set the own bit too */
priv->hw->desc->prepare_tx_desc(desc, 0, len, csum_insertion,
- priv->mode, 1, last_segment);
+ priv->mode, 1, last_segment,
+ skb->len);
}
entry = STMMAC_GET_ENTRY(entry, DMA_TX_SIZE);
- priv->cur_tx = entry;
+ tx_q->cur_tx = entry;
if (netif_msg_pktdata(priv)) {
void *tx_head;
netdev_dbg(priv->dev,
"%s: curr=%d dirty=%d f=%d, e=%d, first=%p, nfrags=%d",
- __func__, priv->cur_tx, priv->dirty_tx, first_entry,
+ __func__, tx_q->cur_tx, tx_q->dirty_tx, first_entry,
entry, first, nfrags);
if (priv->extend_desc)
- tx_head = (void *)priv->dma_etx;
+ tx_head = (void *)tx_q->dma_etx;
else
- tx_head = (void *)priv->dma_tx;
+ tx_head = (void *)tx_q->dma_tx;
priv->hw->desc->display_ring(tx_head, DMA_TX_SIZE, false);
@@ -2273,10 +3060,10 @@
print_pkt(skb->data, skb->len);
}
- if (unlikely(stmmac_tx_avail(priv) <= (MAX_SKB_FRAGS + 1))) {
+ if (unlikely(stmmac_tx_avail(priv, queue) <= (MAX_SKB_FRAGS + 1))) {
netif_dbg(priv, hw, priv->dev, "%s: stop transmitted packets\n",
__func__);
- netif_stop_queue(dev);
+ netif_tx_stop_queue(netdev_get_tx_queue(priv->dev, queue));
}
dev->stats.tx_bytes += skb->len;
@@ -2311,14 +3098,14 @@
if (dma_mapping_error(priv->device, des))
goto dma_map_err;
- priv->tx_skbuff_dma[first_entry].buf = des;
+ tx_q->tx_skbuff_dma[first_entry].buf = des;
if (unlikely(priv->synopsys_id >= DWMAC_CORE_4_00))
first->des0 = cpu_to_le32(des);
else
first->des2 = cpu_to_le32(des);
- priv->tx_skbuff_dma[first_entry].len = nopaged_len;
- priv->tx_skbuff_dma[first_entry].last_segment = last_segment;
+ tx_q->tx_skbuff_dma[first_entry].len = nopaged_len;
+ tx_q->tx_skbuff_dma[first_entry].last_segment = last_segment;
if (unlikely((skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) &&
priv->hwts_tx_en)) {
@@ -2330,7 +3117,7 @@
/* Prepare the first descriptor setting the OWN bit too */
priv->hw->desc->prepare_tx_desc(first, 1, nopaged_len,
csum_insertion, priv->mode, 1,
- last_segment);
+ last_segment, skb->len);
/* The own bit must be the latest setting done when prepare the
* descriptor and then barrier is needed to make sure that
@@ -2339,13 +3126,13 @@
dma_wmb();
}
- netdev_sent_queue(dev, skb->len);
+ netdev_tx_sent_queue(netdev_get_tx_queue(dev, queue), skb->len);
if (priv->synopsys_id < DWMAC_CORE_4_00)
priv->hw->dma->enable_dma_transmission(priv->ioaddr);
else
- priv->hw->dma->set_tx_tail_ptr(priv->ioaddr, priv->tx_tail_addr,
- STMMAC_CHAN0);
+ priv->hw->dma->set_tx_tail_ptr(priv->ioaddr, tx_q->tx_tail_addr,
+ queue);
return NETDEV_TX_OK;
@@ -2373,9 +3160,9 @@
}
-static inline int stmmac_rx_threshold_count(struct stmmac_priv *priv)
+static inline int stmmac_rx_threshold_count(struct stmmac_rx_queue *rx_q)
{
- if (priv->rx_zeroc_thresh < STMMAC_RX_THRESH)
+ if (rx_q->rx_zeroc_thresh < STMMAC_RX_THRESH)
return 0;
return 1;
@@ -2384,30 +3171,33 @@
/**
* stmmac_rx_refill - refill used skb preallocated buffers
* @priv: driver private structure
+ * @queue: RX queue index
* Description : this is to reallocate the skb for the reception process
* that is based on zero-copy.
*/
-static inline void stmmac_rx_refill(struct stmmac_priv *priv)
+static inline void stmmac_rx_refill(struct stmmac_priv *priv, u32 queue)
{
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+ int dirty = stmmac_rx_dirty(priv, queue);
+ unsigned int entry = rx_q->dirty_rx;
+
int bfsize = priv->dma_buf_sz;
- unsigned int entry = priv->dirty_rx;
- int dirty = stmmac_rx_dirty(priv);
while (dirty-- > 0) {
struct dma_desc *p;
if (priv->extend_desc)
- p = (struct dma_desc *)(priv->dma_erx + entry);
+ p = (struct dma_desc *)(rx_q->dma_erx + entry);
else
- p = priv->dma_rx + entry;
+ p = rx_q->dma_rx + entry;
- if (likely(priv->rx_skbuff[entry] == NULL)) {
+ if (likely(!rx_q->rx_skbuff[entry])) {
struct sk_buff *skb;
skb = netdev_alloc_skb_ip_align(priv->dev, bfsize);
if (unlikely(!skb)) {
/* so for a while no zero-copy! */
- priv->rx_zeroc_thresh = STMMAC_RX_THRESH;
+ rx_q->rx_zeroc_thresh = STMMAC_RX_THRESH;
if (unlikely(net_ratelimit()))
dev_err(priv->device,
"fail to alloc skb entry %d\n",
@@ -2415,28 +3205,28 @@
break;
}
- priv->rx_skbuff[entry] = skb;
- priv->rx_skbuff_dma[entry] =
+ rx_q->rx_skbuff[entry] = skb;
+ rx_q->rx_skbuff_dma[entry] =
dma_map_single(priv->device, skb->data, bfsize,
DMA_FROM_DEVICE);
if (dma_mapping_error(priv->device,
- priv->rx_skbuff_dma[entry])) {
+ rx_q->rx_skbuff_dma[entry])) {
netdev_err(priv->dev, "Rx DMA map failed\n");
dev_kfree_skb(skb);
break;
}
if (unlikely(priv->synopsys_id >= DWMAC_CORE_4_00)) {
- p->des0 = cpu_to_le32(priv->rx_skbuff_dma[entry]);
+ p->des0 = cpu_to_le32(rx_q->rx_skbuff_dma[entry]);
p->des1 = 0;
} else {
- p->des2 = cpu_to_le32(priv->rx_skbuff_dma[entry]);
+ p->des2 = cpu_to_le32(rx_q->rx_skbuff_dma[entry]);
}
if (priv->hw->mode->refill_desc3)
- priv->hw->mode->refill_desc3(priv, p);
+ priv->hw->mode->refill_desc3(rx_q, p);
- if (priv->rx_zeroc_thresh > 0)
- priv->rx_zeroc_thresh--;
+ if (rx_q->rx_zeroc_thresh > 0)
+ rx_q->rx_zeroc_thresh--;
netif_dbg(priv, rx_status, priv->dev,
"refill entry #%d\n", entry);
@@ -2452,31 +3242,33 @@
entry = STMMAC_GET_ENTRY(entry, DMA_RX_SIZE);
}
- priv->dirty_rx = entry;
+ rx_q->dirty_rx = entry;
}
/**
* stmmac_rx - manage the receive process
* @priv: driver private structure
- * @limit: napi bugget.
+ * @limit: napi bugget
+ * @queue: RX queue index.
* Description : this the function called by the napi poll method.
* It gets all the frames inside the ring.
*/
-static int stmmac_rx(struct stmmac_priv *priv, int limit)
+static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
{
- unsigned int entry = priv->cur_rx;
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+ unsigned int entry = rx_q->cur_rx;
+ int coe = priv->hw->rx_csum;
unsigned int next_entry;
unsigned int count = 0;
- int coe = priv->hw->rx_csum;
if (netif_msg_rx_status(priv)) {
void *rx_head;
netdev_dbg(priv->dev, "%s: descriptor ring:\n", __func__);
if (priv->extend_desc)
- rx_head = (void *)priv->dma_erx;
+ rx_head = (void *)rx_q->dma_erx;
else
- rx_head = (void *)priv->dma_rx;
+ rx_head = (void *)rx_q->dma_rx;
priv->hw->desc->display_ring(rx_head, DMA_RX_SIZE, true);
}
@@ -2486,9 +3278,9 @@
struct dma_desc *np;
if (priv->extend_desc)
- p = (struct dma_desc *)(priv->dma_erx + entry);
+ p = (struct dma_desc *)(rx_q->dma_erx + entry);
else
- p = priv->dma_rx + entry;
+ p = rx_q->dma_rx + entry;
/* read the status of the incoming frame */
status = priv->hw->desc->rx_status(&priv->dev->stats,
@@ -2499,20 +3291,20 @@
count++;
- priv->cur_rx = STMMAC_GET_ENTRY(priv->cur_rx, DMA_RX_SIZE);
- next_entry = priv->cur_rx;
+ rx_q->cur_rx = STMMAC_GET_ENTRY(rx_q->cur_rx, DMA_RX_SIZE);
+ next_entry = rx_q->cur_rx;
if (priv->extend_desc)
- np = (struct dma_desc *)(priv->dma_erx + next_entry);
+ np = (struct dma_desc *)(rx_q->dma_erx + next_entry);
else
- np = priv->dma_rx + next_entry;
+ np = rx_q->dma_rx + next_entry;
prefetch(np);
if ((priv->extend_desc) && (priv->hw->desc->rx_extended_status))
priv->hw->desc->rx_extended_status(&priv->dev->stats,
&priv->xstats,
- priv->dma_erx +
+ rx_q->dma_erx +
entry);
if (unlikely(status == discard_frame)) {
priv->dev->stats.rx_errors++;
@@ -2522,9 +3314,9 @@
* them in stmmac_rx_refill() function so that
* device can reuse it.
*/
- priv->rx_skbuff[entry] = NULL;
+ rx_q->rx_skbuff[entry] = NULL;
dma_unmap_single(priv->device,
- priv->rx_skbuff_dma[entry],
+ rx_q->rx_skbuff_dma[entry],
priv->dma_buf_sz,
DMA_FROM_DEVICE);
}
@@ -2572,7 +3364,7 @@
*/
if (unlikely(!priv->plat->has_gmac4 &&
((frame_len < priv->rx_copybreak) ||
- stmmac_rx_threshold_count(priv)))) {
+ stmmac_rx_threshold_count(rx_q)))) {
skb = netdev_alloc_skb_ip_align(priv->dev,
frame_len);
if (unlikely(!skb)) {
@@ -2584,21 +3376,21 @@
}
dma_sync_single_for_cpu(priv->device,
- priv->rx_skbuff_dma
+ rx_q->rx_skbuff_dma
[entry], frame_len,
DMA_FROM_DEVICE);
skb_copy_to_linear_data(skb,
- priv->
+ rx_q->
rx_skbuff[entry]->data,
frame_len);
skb_put(skb, frame_len);
dma_sync_single_for_device(priv->device,
- priv->rx_skbuff_dma
+ rx_q->rx_skbuff_dma
[entry], frame_len,
DMA_FROM_DEVICE);
} else {
- skb = priv->rx_skbuff[entry];
+ skb = rx_q->rx_skbuff[entry];
if (unlikely(!skb)) {
netdev_err(priv->dev,
"%s: Inconsistent Rx chain\n",
@@ -2607,12 +3399,12 @@
break;
}
prefetch(skb->data - NET_IP_ALIGN);
- priv->rx_skbuff[entry] = NULL;
- priv->rx_zeroc_thresh++;
+ rx_q->rx_skbuff[entry] = NULL;
+ rx_q->rx_zeroc_thresh++;
skb_put(skb, frame_len);
dma_unmap_single(priv->device,
- priv->rx_skbuff_dma[entry],
+ rx_q->rx_skbuff_dma[entry],
priv->dma_buf_sz,
DMA_FROM_DEVICE);
}
@@ -2634,7 +3426,7 @@
else
skb->ip_summed = CHECKSUM_UNNECESSARY;
- napi_gro_receive(&priv->napi, skb);
+ napi_gro_receive(&rx_q->napi, skb);
priv->dev->stats.rx_packets++;
priv->dev->stats.rx_bytes += frame_len;
@@ -2642,7 +3434,7 @@
entry = next_entry;
}
- stmmac_rx_refill(priv);
+ stmmac_rx_refill(priv, queue);
priv->xstats.rx_pkt_n += count;
@@ -2659,16 +3451,24 @@
*/
static int stmmac_poll(struct napi_struct *napi, int budget)
{
- struct stmmac_priv *priv = container_of(napi, struct stmmac_priv, napi);
+ struct stmmac_rx_queue *rx_q =
+ container_of(napi, struct stmmac_rx_queue, napi);
+ struct stmmac_priv *priv = rx_q->priv_data;
+ u32 tx_count = priv->plat->tx_queues_to_use;
+ u32 chan = rx_q->queue_index;
int work_done = 0;
+ u32 queue;
priv->xstats.napi_poll++;
- stmmac_tx_clean(priv);
- work_done = stmmac_rx(priv, budget);
+ /* check all the queues */
+ for (queue = 0; queue < tx_count; queue++)
+ stmmac_tx_clean(priv, queue);
+
+ work_done = stmmac_rx(priv, budget, rx_q->queue_index);
if (work_done < budget) {
napi_complete_done(napi, work_done);
- stmmac_enable_dma_irq(priv);
+ stmmac_enable_dma_irq(priv, chan);
}
return work_done;
}
@@ -2684,9 +3484,12 @@
static void stmmac_tx_timeout(struct net_device *dev)
{
struct stmmac_priv *priv = netdev_priv(dev);
+ u32 tx_count = priv->plat->tx_queues_to_use;
+ u32 chan;
/* Clear Tx resources and restart transmitting again */
- stmmac_tx_err(priv);
+ for (chan = 0; chan < tx_count; chan++)
+ stmmac_tx_err(priv, chan);
}
/**
@@ -2795,6 +3598,12 @@
{
struct net_device *dev = (struct net_device *)dev_id;
struct stmmac_priv *priv = netdev_priv(dev);
+ u32 rx_cnt = priv->plat->rx_queues_to_use;
+ u32 tx_cnt = priv->plat->tx_queues_to_use;
+ u32 queues_count;
+ u32 queue;
+
+ queues_count = (rx_cnt > tx_cnt) ? rx_cnt : tx_cnt;
if (priv->irq_wake)
pm_wakeup_event(priv->device, 0);
@@ -2808,16 +3617,30 @@
if ((priv->plat->has_gmac) || (priv->plat->has_gmac4)) {
int status = priv->hw->mac->host_irq_status(priv->hw,
&priv->xstats);
+
if (unlikely(status)) {
/* For LPI we need to save the tx status */
if (status & CORE_IRQ_TX_PATH_IN_LPI_MODE)
priv->tx_path_in_lpi_mode = true;
if (status & CORE_IRQ_TX_PATH_EXIT_LPI_MODE)
priv->tx_path_in_lpi_mode = false;
- if (status & CORE_IRQ_MTL_RX_OVERFLOW && priv->hw->dma->set_rx_tail_ptr)
- priv->hw->dma->set_rx_tail_ptr(priv->ioaddr,
- priv->rx_tail_addr,
- STMMAC_CHAN0);
+ }
+
+ if (priv->synopsys_id >= DWMAC_CORE_4_00) {
+ for (queue = 0; queue < queues_count; queue++) {
+ struct stmmac_rx_queue *rx_q =
+ &priv->rx_queue[queue];
+
+ status |=
+ priv->hw->mac->host_mtl_irq_status(priv->hw,
+ queue);
+
+ if (status & CORE_IRQ_MTL_RX_OVERFLOW &&
+ priv->hw->dma->set_rx_tail_ptr)
+ priv->hw->dma->set_rx_tail_ptr(priv->ioaddr,
+ rx_q->rx_tail_addr,
+ queue);
+ }
}
/* PCS link status */
@@ -2915,17 +3738,40 @@
{
struct net_device *dev = seq->private;
struct stmmac_priv *priv = netdev_priv(dev);
+ u32 rx_count = priv->plat->rx_queues_to_use;
+ u32 tx_count = priv->plat->tx_queues_to_use;
+ u32 queue;
- if (priv->extend_desc) {
- seq_printf(seq, "Extended RX descriptor ring:\n");
- sysfs_display_ring((void *)priv->dma_erx, DMA_RX_SIZE, 1, seq);
- seq_printf(seq, "Extended TX descriptor ring:\n");
- sysfs_display_ring((void *)priv->dma_etx, DMA_TX_SIZE, 1, seq);
- } else {
- seq_printf(seq, "RX descriptor ring:\n");
- sysfs_display_ring((void *)priv->dma_rx, DMA_RX_SIZE, 0, seq);
- seq_printf(seq, "TX descriptor ring:\n");
- sysfs_display_ring((void *)priv->dma_tx, DMA_TX_SIZE, 0, seq);
+ for (queue = 0; queue < rx_count; queue++) {
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+
+ seq_printf(seq, "RX Queue %d:\n", queue);
+
+ if (priv->extend_desc) {
+ seq_printf(seq, "Extended descriptor ring:\n");
+ sysfs_display_ring((void *)rx_q->dma_erx,
+ DMA_RX_SIZE, 1, seq);
+ } else {
+ seq_printf(seq, "Descriptor ring:\n");
+ sysfs_display_ring((void *)rx_q->dma_rx,
+ DMA_RX_SIZE, 0, seq);
+ }
+ }
+
+ for (queue = 0; queue < tx_count; queue++) {
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
+
+ seq_printf(seq, "TX Queue %d:\n", queue);
+
+ if (priv->extend_desc) {
+ seq_printf(seq, "Extended descriptor ring:\n");
+ sysfs_display_ring((void *)tx_q->dma_etx,
+ DMA_TX_SIZE, 1, seq);
+ } else {
+ seq_printf(seq, "Descriptor ring:\n");
+ sysfs_display_ring((void *)tx_q->dma_tx,
+ DMA_TX_SIZE, 0, seq);
+ }
}
return 0;
@@ -3208,11 +4054,14 @@
struct plat_stmmacenet_data *plat_dat,
struct stmmac_resources *res)
{
- int ret = 0;
struct net_device *ndev = NULL;
struct stmmac_priv *priv;
+ int ret = 0;
+ u32 queue;
- ndev = alloc_etherdev(sizeof(struct stmmac_priv));
+ ndev = alloc_etherdev_mqs(sizeof(struct stmmac_priv),
+ MTL_MAX_TX_QUEUES,
+ MTL_MAX_RX_QUEUES);
if (!ndev)
return -ENOMEM;
@@ -3254,6 +4103,10 @@
if (ret)
goto error_hw_init;
+ /* Configure real RX and TX queues */
+ netif_set_real_num_rx_queues(ndev, priv->plat->rx_queues_to_use);
+ netif_set_real_num_tx_queues(ndev, priv->plat->tx_queues_to_use);
+
ndev->netdev_ops = &stmmac_netdev_ops;
ndev->hw_features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |
@@ -3303,7 +4156,12 @@
"Enable RX Mitigation via HW Watchdog Timer\n");
}
- netif_napi_add(ndev, &priv->napi, stmmac_poll, 64);
+ for (queue = 0; queue < priv->plat->rx_queues_to_use; queue++) {
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+
+ netif_napi_add(ndev, &rx_q->napi, stmmac_poll,
+ (8 * priv->plat->rx_queues_to_use));
+ }
spin_lock_init(&priv->lock);
@@ -3348,7 +4206,11 @@
priv->hw->pcs != STMMAC_PCS_RTBI)
stmmac_mdio_unregister(ndev);
error_mdio_register:
- netif_napi_del(&priv->napi);
+ for (queue = 0; queue < priv->plat->rx_queues_to_use; queue++) {
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+
+ netif_napi_del(&rx_q->napi);
+ }
error_hw_init:
free_netdev(ndev);
@@ -3369,10 +4231,9 @@
netdev_info(priv->dev, "%s: removing driver", __func__);
- priv->hw->dma->stop_rx(priv->ioaddr);
- priv->hw->dma->stop_tx(priv->ioaddr);
+ stmmac_stop_all_dma(priv);
- stmmac_set_mac(priv->ioaddr, false);
+ priv->hw->mac->set_mac(priv->ioaddr, false);
netif_carrier_off(ndev);
unregister_netdev(ndev);
if (priv->plat->stmmac_rst)
@@ -3411,20 +4272,19 @@
spin_lock_irqsave(&priv->lock, flags);
netif_device_detach(ndev);
- netif_stop_queue(ndev);
+ stmmac_stop_all_queues(priv);
- napi_disable(&priv->napi);
+ stmmac_disable_all_queues(priv);
/* Stop TX/RX DMA */
- priv->hw->dma->stop_tx(priv->ioaddr);
- priv->hw->dma->stop_rx(priv->ioaddr);
+ stmmac_stop_all_dma(priv);
/* Enable Power down mode by programming the PMT regs */
if (device_may_wakeup(priv->device)) {
priv->hw->mac->pmt(priv->hw, priv->wolopts);
priv->irq_wake = 1;
} else {
- stmmac_set_mac(priv->ioaddr, false);
+ priv->hw->mac->set_mac(priv->ioaddr, false);
pinctrl_pm_select_sleep_state(priv->device);
/* Disable clock in case of PWM is off */
clk_disable(priv->plat->pclk);
@@ -3440,6 +4300,31 @@
EXPORT_SYMBOL_GPL(stmmac_suspend);
/**
+ * stmmac_reset_queues_param - reset queue parameters
+ * @dev: device pointer
+ */
+static void stmmac_reset_queues_param(struct stmmac_priv *priv)
+{
+ u32 rx_cnt = priv->plat->rx_queues_to_use;
+ u32 tx_cnt = priv->plat->tx_queues_to_use;
+ u32 queue;
+
+ for (queue = 0; queue < rx_cnt; queue++) {
+ struct stmmac_rx_queue *rx_q = &priv->rx_queue[queue];
+
+ rx_q->cur_rx = 0;
+ rx_q->dirty_rx = 0;
+ }
+
+ for (queue = 0; queue < tx_cnt; queue++) {
+ struct stmmac_tx_queue *tx_q = &priv->tx_queue[queue];
+
+ tx_q->cur_tx = 0;
+ tx_q->dirty_tx = 0;
+ }
+}
+
+/**
* stmmac_resume - resume callback
* @dev: device pointer
* Description: when resume this function is invoked to setup the DMA and CORE
@@ -3479,10 +4364,8 @@
spin_lock_irqsave(&priv->lock, flags);
- priv->cur_rx = 0;
- priv->dirty_rx = 0;
- priv->dirty_tx = 0;
- priv->cur_tx = 0;
+ stmmac_reset_queues_param(priv);
+
/* reset private mss value to force mss context settings at
* next tso xmit (only used for gmac4).
*/
@@ -3494,9 +4377,9 @@
stmmac_init_tx_coalesce(priv);
stmmac_set_rx_mode(ndev);
- napi_enable(&priv->napi);
+ stmmac_enable_all_queues(priv);
- netif_start_queue(ndev);
+ stmmac_start_all_queues(priv);
spin_unlock_irqrestore(&priv->lock, flags);
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c
index 5c9e4622..39be967 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c
@@ -32,6 +32,7 @@
*/
struct stmmac_pci_dmi_data {
const char *name;
+ const char *asset_tag;
unsigned int func;
int phy_addr;
};
@@ -46,6 +47,7 @@
static int stmmac_pci_find_phy_addr(struct stmmac_pci_info *info)
{
const char *name = dmi_get_system_info(DMI_BOARD_NAME);
+ const char *asset_tag = dmi_get_system_info(DMI_BOARD_ASSET_TAG);
unsigned int func = PCI_FUNC(info->pdev->devfn);
struct stmmac_pci_dmi_data *dmi;
@@ -57,8 +59,12 @@
return 1;
for (dmi = info->dmi; dmi->name && *dmi->name; dmi++) {
- if (!strcmp(dmi->name, name) && dmi->func == func)
+ if (!strcmp(dmi->name, name) && dmi->func == func) {
+ /* If asset tag is provided, match on it as well. */
+ if (dmi->asset_tag && strcmp(dmi->asset_tag, asset_tag))
+ continue;
return dmi->phy_addr;
+ }
}
return -ENODEV;
@@ -88,6 +94,17 @@
/* Set the maxmtu to a default of JUMBO_LEN */
plat->maxmtu = JUMBO_LEN;
+
+ /* Set default number of RX and TX queues to use */
+ plat->tx_queues_to_use = 1;
+ plat->rx_queues_to_use = 1;
+
+ /* Disable Priority config by default */
+ plat->tx_queues_cfg[0].use_prio = false;
+ plat->rx_queues_cfg[0].use_prio = false;
+
+ /* Disable RX queues routing by default */
+ plat->rx_queues_cfg[0].pkt_route = 0x0;
}
static int quark_default_data(struct plat_stmmacenet_data *plat,
@@ -142,6 +159,24 @@
.func = 6,
.phy_addr = 1,
},
+ {
+ .name = "SIMATIC IOT2000",
+ .asset_tag = "6ES7647-0AA00-0YA2",
+ .func = 6,
+ .phy_addr = 1,
+ },
+ {
+ .name = "SIMATIC IOT2000",
+ .asset_tag = "6ES7647-0AA00-1YA2",
+ .func = 6,
+ .phy_addr = 1,
+ },
+ {
+ .name = "SIMATIC IOT2000",
+ .asset_tag = "6ES7647-0AA00-1YA2",
+ .func = 7,
+ .phy_addr = 1,
+ },
{}
};
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c
index 433a842..7fc3a1e 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c
@@ -108,7 +108,7 @@
if (!np)
return NULL;
- axi = kzalloc(sizeof(*axi), GFP_KERNEL);
+ axi = devm_kzalloc(&pdev->dev, sizeof(*axi), GFP_KERNEL);
if (!axi) {
of_node_put(np);
return ERR_PTR(-ENOMEM);
@@ -132,6 +132,155 @@
}
/**
+ * stmmac_mtl_setup - parse DT parameters for multiple queues configuration
+ * @pdev: platform device
+ */
+static void stmmac_mtl_setup(struct platform_device *pdev,
+ struct plat_stmmacenet_data *plat)
+{
+ struct device_node *q_node;
+ struct device_node *rx_node;
+ struct device_node *tx_node;
+ u8 queue = 0;
+
+ /* For backwards-compatibility with device trees that don't have any
+ * snps,mtl-rx-config or snps,mtl-tx-config properties, we fall back
+ * to one RX and TX queues each.
+ */
+ plat->rx_queues_to_use = 1;
+ plat->tx_queues_to_use = 1;
+
+ rx_node = of_parse_phandle(pdev->dev.of_node, "snps,mtl-rx-config", 0);
+ if (!rx_node)
+ return;
+
+ tx_node = of_parse_phandle(pdev->dev.of_node, "snps,mtl-tx-config", 0);
+ if (!tx_node) {
+ of_node_put(rx_node);
+ return;
+ }
+
+ /* Processing RX queues common config */
+ if (of_property_read_u8(rx_node, "snps,rx-queues-to-use",
+ &plat->rx_queues_to_use))
+ plat->rx_queues_to_use = 1;
+
+ if (of_property_read_bool(rx_node, "snps,rx-sched-sp"))
+ plat->rx_sched_algorithm = MTL_RX_ALGORITHM_SP;
+ else if (of_property_read_bool(rx_node, "snps,rx-sched-wsp"))
+ plat->rx_sched_algorithm = MTL_RX_ALGORITHM_WSP;
+ else
+ plat->rx_sched_algorithm = MTL_RX_ALGORITHM_SP;
+
+ /* Processing individual RX queue config */
+ for_each_child_of_node(rx_node, q_node) {
+ if (queue >= plat->rx_queues_to_use)
+ break;
+
+ if (of_property_read_bool(q_node, "snps,dcb-algorithm"))
+ plat->rx_queues_cfg[queue].mode_to_use = MTL_QUEUE_DCB;
+ else if (of_property_read_bool(q_node, "snps,avb-algorithm"))
+ plat->rx_queues_cfg[queue].mode_to_use = MTL_QUEUE_AVB;
+ else
+ plat->rx_queues_cfg[queue].mode_to_use = MTL_QUEUE_DCB;
+
+ if (of_property_read_u8(q_node, "snps,map-to-dma-channel",
+ &plat->rx_queues_cfg[queue].chan))
+ plat->rx_queues_cfg[queue].chan = queue;
+ /* TODO: Dynamic mapping to be included in the future */
+
+ if (of_property_read_u32(q_node, "snps,priority",
+ &plat->rx_queues_cfg[queue].prio)) {
+ plat->rx_queues_cfg[queue].prio = 0;
+ plat->rx_queues_cfg[queue].use_prio = false;
+ } else {
+ plat->rx_queues_cfg[queue].use_prio = true;
+ }
+
+ /* RX queue specific packet type routing */
+ if (of_property_read_bool(q_node, "snps,route-avcp"))
+ plat->rx_queues_cfg[queue].pkt_route = PACKET_AVCPQ;
+ else if (of_property_read_bool(q_node, "snps,route-ptp"))
+ plat->rx_queues_cfg[queue].pkt_route = PACKET_PTPQ;
+ else if (of_property_read_bool(q_node, "snps,route-dcbcp"))
+ plat->rx_queues_cfg[queue].pkt_route = PACKET_DCBCPQ;
+ else if (of_property_read_bool(q_node, "snps,route-up"))
+ plat->rx_queues_cfg[queue].pkt_route = PACKET_UPQ;
+ else if (of_property_read_bool(q_node, "snps,route-multi-broad"))
+ plat->rx_queues_cfg[queue].pkt_route = PACKET_MCBCQ;
+ else
+ plat->rx_queues_cfg[queue].pkt_route = 0x0;
+
+ queue++;
+ }
+
+ /* Processing TX queues common config */
+ if (of_property_read_u8(tx_node, "snps,tx-queues-to-use",
+ &plat->tx_queues_to_use))
+ plat->tx_queues_to_use = 1;
+
+ if (of_property_read_bool(tx_node, "snps,tx-sched-wrr"))
+ plat->tx_sched_algorithm = MTL_TX_ALGORITHM_WRR;
+ else if (of_property_read_bool(tx_node, "snps,tx-sched-wfq"))
+ plat->tx_sched_algorithm = MTL_TX_ALGORITHM_WFQ;
+ else if (of_property_read_bool(tx_node, "snps,tx-sched-dwrr"))
+ plat->tx_sched_algorithm = MTL_TX_ALGORITHM_DWRR;
+ else if (of_property_read_bool(tx_node, "snps,tx-sched-sp"))
+ plat->tx_sched_algorithm = MTL_TX_ALGORITHM_SP;
+ else
+ plat->tx_sched_algorithm = MTL_TX_ALGORITHM_SP;
+
+ queue = 0;
+
+ /* Processing individual TX queue config */
+ for_each_child_of_node(tx_node, q_node) {
+ if (queue >= plat->tx_queues_to_use)
+ break;
+
+ if (of_property_read_u8(q_node, "snps,weight",
+ &plat->tx_queues_cfg[queue].weight))
+ plat->tx_queues_cfg[queue].weight = 0x10 + queue;
+
+ if (of_property_read_bool(q_node, "snps,dcb-algorithm")) {
+ plat->tx_queues_cfg[queue].mode_to_use = MTL_QUEUE_DCB;
+ } else if (of_property_read_bool(q_node,
+ "snps,avb-algorithm")) {
+ plat->tx_queues_cfg[queue].mode_to_use = MTL_QUEUE_AVB;
+
+ /* Credit Base Shaper parameters used by AVB */
+ if (of_property_read_u32(q_node, "snps,send_slope",
+ &plat->tx_queues_cfg[queue].send_slope))
+ plat->tx_queues_cfg[queue].send_slope = 0x0;
+ if (of_property_read_u32(q_node, "snps,idle_slope",
+ &plat->tx_queues_cfg[queue].idle_slope))
+ plat->tx_queues_cfg[queue].idle_slope = 0x0;
+ if (of_property_read_u32(q_node, "snps,high_credit",
+ &plat->tx_queues_cfg[queue].high_credit))
+ plat->tx_queues_cfg[queue].high_credit = 0x0;
+ if (of_property_read_u32(q_node, "snps,low_credit",
+ &plat->tx_queues_cfg[queue].low_credit))
+ plat->tx_queues_cfg[queue].low_credit = 0x0;
+ } else {
+ plat->tx_queues_cfg[queue].mode_to_use = MTL_QUEUE_DCB;
+ }
+
+ if (of_property_read_u32(q_node, "snps,priority",
+ &plat->tx_queues_cfg[queue].prio)) {
+ plat->tx_queues_cfg[queue].prio = 0;
+ plat->tx_queues_cfg[queue].use_prio = false;
+ } else {
+ plat->tx_queues_cfg[queue].use_prio = true;
+ }
+
+ queue++;
+ }
+
+ of_node_put(rx_node);
+ of_node_put(tx_node);
+ of_node_put(q_node);
+}
+
+/**
* stmmac_dt_phy - parse device-tree driver parameters to allocate PHY resources
* @plat: driver data platform structure
* @np: device tree node
@@ -340,6 +489,8 @@
plat->axi = stmmac_axi_setup(pdev);
+ stmmac_mtl_setup(pdev, plat);
+
/* clock setup */
plat->stmmac_clk = devm_clk_get(&pdev->dev,
STMMAC_RESOURCE_NAME);
@@ -359,13 +510,12 @@
clk_prepare_enable(plat->pclk);
/* Fall-back to main clock in case of no PTP ref is passed */
- plat->clk_ptp_ref = devm_clk_get(&pdev->dev, "clk_ptp_ref");
+ plat->clk_ptp_ref = devm_clk_get(&pdev->dev, "ptp_ref");
if (IS_ERR(plat->clk_ptp_ref)) {
plat->clk_ptp_rate = clk_get_rate(plat->stmmac_clk);
plat->clk_ptp_ref = NULL;
dev_warn(&pdev->dev, "PTP uses main clock\n");
} else {
- clk_prepare_enable(plat->clk_ptp_ref);
plat->clk_ptp_rate = clk_get_rate(plat->clk_ptp_ref);
dev_dbg(&pdev->dev, "PTP rate %d\n", plat->clk_ptp_rate);
}
diff --git a/drivers/net/ethernet/sun/cassini.c b/drivers/net/ethernet/sun/cassini.c
index 0e8e89f..382993c 100644
--- a/drivers/net/ethernet/sun/cassini.c
+++ b/drivers/net/ethernet/sun/cassini.c
@@ -691,7 +691,8 @@
}
/* Must be invoked under cp->lock */
-static void cas_begin_auto_negotiation(struct cas *cp, struct ethtool_cmd *ep)
+static void cas_begin_auto_negotiation(struct cas *cp,
+ const struct ethtool_link_ksettings *ep)
{
u16 ctl;
#if 1
@@ -704,16 +705,16 @@
if (!ep)
goto start_aneg;
lcntl = cp->link_cntl;
- if (ep->autoneg == AUTONEG_ENABLE)
+ if (ep->base.autoneg == AUTONEG_ENABLE) {
cp->link_cntl = BMCR_ANENABLE;
- else {
- u32 speed = ethtool_cmd_speed(ep);
+ } else {
+ u32 speed = ep->base.speed;
cp->link_cntl = 0;
if (speed == SPEED_100)
cp->link_cntl |= BMCR_SPEED100;
else if (speed == SPEED_1000)
cp->link_cntl |= CAS_BMCR_SPEED1000;
- if (ep->duplex == DUPLEX_FULL)
+ if (ep->base.duplex == DUPLEX_FULL)
cp->link_cntl |= BMCR_FULLDPLX;
}
#if 1
@@ -4528,19 +4529,21 @@
strlcpy(info->bus_info, pci_name(cp->pdev), sizeof(info->bus_info));
}
-static int cas_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int cas_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct cas *cp = netdev_priv(dev);
u16 bmcr;
int full_duplex, speed, pause;
unsigned long flags;
enum link_state linkstate = link_up;
+ u32 supported, advertising;
- cmd->advertising = 0;
- cmd->supported = SUPPORTED_Autoneg;
+ advertising = 0;
+ supported = SUPPORTED_Autoneg;
if (cp->cas_flags & CAS_FLAG_1000MB_CAP) {
- cmd->supported |= SUPPORTED_1000baseT_Full;
- cmd->advertising |= ADVERTISED_1000baseT_Full;
+ supported |= SUPPORTED_1000baseT_Full;
+ advertising |= ADVERTISED_1000baseT_Full;
}
/* Record PHY settings if HW is on. */
@@ -4548,17 +4551,15 @@
bmcr = 0;
linkstate = cp->lstate;
if (CAS_PHY_MII(cp->phy_type)) {
- cmd->port = PORT_MII;
- cmd->transceiver = (cp->cas_flags & CAS_FLAG_SATURN) ?
- XCVR_INTERNAL : XCVR_EXTERNAL;
- cmd->phy_address = cp->phy_addr;
- cmd->advertising |= ADVERTISED_TP | ADVERTISED_MII |
+ cmd->base.port = PORT_MII;
+ cmd->base.phy_address = cp->phy_addr;
+ advertising |= ADVERTISED_TP | ADVERTISED_MII |
ADVERTISED_10baseT_Half |
ADVERTISED_10baseT_Full |
ADVERTISED_100baseT_Half |
ADVERTISED_100baseT_Full;
- cmd->supported |=
+ supported |=
(SUPPORTED_10baseT_Half |
SUPPORTED_10baseT_Full |
SUPPORTED_100baseT_Half |
@@ -4574,11 +4575,10 @@
}
} else {
- cmd->port = PORT_FIBRE;
- cmd->transceiver = XCVR_INTERNAL;
- cmd->phy_address = 0;
- cmd->supported |= SUPPORTED_FIBRE;
- cmd->advertising |= ADVERTISED_FIBRE;
+ cmd->base.port = PORT_FIBRE;
+ cmd->base.phy_address = 0;
+ supported |= SUPPORTED_FIBRE;
+ advertising |= ADVERTISED_FIBRE;
if (cp->hw_running) {
/* pcs uses the same bits as mii */
@@ -4590,21 +4590,20 @@
spin_unlock_irqrestore(&cp->lock, flags);
if (bmcr & BMCR_ANENABLE) {
- cmd->advertising |= ADVERTISED_Autoneg;
- cmd->autoneg = AUTONEG_ENABLE;
- ethtool_cmd_speed_set(cmd, ((speed == 10) ?
+ advertising |= ADVERTISED_Autoneg;
+ cmd->base.autoneg = AUTONEG_ENABLE;
+ cmd->base.speed = ((speed == 10) ?
SPEED_10 :
((speed == 1000) ?
- SPEED_1000 : SPEED_100)));
- cmd->duplex = full_duplex ? DUPLEX_FULL : DUPLEX_HALF;
+ SPEED_1000 : SPEED_100));
+ cmd->base.duplex = full_duplex ? DUPLEX_FULL : DUPLEX_HALF;
} else {
- cmd->autoneg = AUTONEG_DISABLE;
- ethtool_cmd_speed_set(cmd, ((bmcr & CAS_BMCR_SPEED1000) ?
+ cmd->base.autoneg = AUTONEG_DISABLE;
+ cmd->base.speed = ((bmcr & CAS_BMCR_SPEED1000) ?
SPEED_1000 :
((bmcr & BMCR_SPEED100) ?
- SPEED_100 : SPEED_10)));
- cmd->duplex =
- (bmcr & BMCR_FULLDPLX) ?
+ SPEED_100 : SPEED_10));
+ cmd->base.duplex = (bmcr & BMCR_FULLDPLX) ?
DUPLEX_FULL : DUPLEX_HALF;
}
if (linkstate != link_up) {
@@ -4619,39 +4618,46 @@
* settings that we configured.
*/
if (cp->link_cntl & BMCR_ANENABLE) {
- ethtool_cmd_speed_set(cmd, 0);
- cmd->duplex = 0xff;
+ cmd->base.speed = 0;
+ cmd->base.duplex = 0xff;
} else {
- ethtool_cmd_speed_set(cmd, SPEED_10);
+ cmd->base.speed = SPEED_10;
if (cp->link_cntl & BMCR_SPEED100) {
- ethtool_cmd_speed_set(cmd, SPEED_100);
+ cmd->base.speed = SPEED_100;
} else if (cp->link_cntl & CAS_BMCR_SPEED1000) {
- ethtool_cmd_speed_set(cmd, SPEED_1000);
+ cmd->base.speed = SPEED_1000;
}
- cmd->duplex = (cp->link_cntl & BMCR_FULLDPLX)?
+ cmd->base.duplex = (cp->link_cntl & BMCR_FULLDPLX) ?
DUPLEX_FULL : DUPLEX_HALF;
}
}
+
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
+ advertising);
+
return 0;
}
-static int cas_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int cas_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct cas *cp = netdev_priv(dev);
unsigned long flags;
- u32 speed = ethtool_cmd_speed(cmd);
+ u32 speed = cmd->base.speed;
/* Verify the settings we care about. */
- if (cmd->autoneg != AUTONEG_ENABLE &&
- cmd->autoneg != AUTONEG_DISABLE)
+ if (cmd->base.autoneg != AUTONEG_ENABLE &&
+ cmd->base.autoneg != AUTONEG_DISABLE)
return -EINVAL;
- if (cmd->autoneg == AUTONEG_DISABLE &&
+ if (cmd->base.autoneg == AUTONEG_DISABLE &&
((speed != SPEED_1000 &&
speed != SPEED_100 &&
speed != SPEED_10) ||
- (cmd->duplex != DUPLEX_HALF &&
- cmd->duplex != DUPLEX_FULL)))
+ (cmd->base.duplex != DUPLEX_HALF &&
+ cmd->base.duplex != DUPLEX_FULL)))
return -EINVAL;
/* Apply settings and restart link process. */
@@ -4753,8 +4759,6 @@
static const struct ethtool_ops cas_ethtool_ops = {
.get_drvinfo = cas_get_drvinfo,
- .get_settings = cas_get_settings,
- .set_settings = cas_set_settings,
.nway_reset = cas_nway_reset,
.get_link = cas_get_link,
.get_msglevel = cas_get_msglevel,
@@ -4764,6 +4768,8 @@
.get_sset_count = cas_get_sset_count,
.get_strings = cas_get_strings,
.get_ethtool_stats = cas_get_ethtool_stats,
+ .get_link_ksettings = cas_get_link_ksettings,
+ .set_link_ksettings = cas_set_link_ksettings,
};
static int cas_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
diff --git a/drivers/net/ethernet/sun/ldmvsw.c b/drivers/net/ethernet/sun/ldmvsw.c
index 89952de..5a90fed 100644
--- a/drivers/net/ethernet/sun/ldmvsw.c
+++ b/drivers/net/ethernet/sun/ldmvsw.c
@@ -1,6 +1,6 @@
/* ldmvsw.c: Sun4v LDOM Virtual Switch Driver.
*
- * Copyright (C) 2016 Oracle. All rights reserved.
+ * Copyright (C) 2016-2017 Oracle. All rights reserved.
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
@@ -41,8 +41,8 @@
static u8 vsw_port_hwaddr[ETH_ALEN] = {0xFE, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
#define DRV_MODULE_NAME "ldmvsw"
-#define DRV_MODULE_VERSION "1.1"
-#define DRV_MODULE_RELDATE "February 3, 2017"
+#define DRV_MODULE_VERSION "1.2"
+#define DRV_MODULE_RELDATE "March 4, 2017"
static char version[] =
DRV_MODULE_NAME " " DRV_MODULE_VERSION " (" DRV_MODULE_RELDATE ")";
@@ -123,6 +123,20 @@
return sunvnet_set_rx_mode_common(dev, port->vp);
}
+int ldmvsw_open(struct net_device *dev)
+{
+ struct vnet_port *port = netdev_priv(dev);
+ struct vio_driver_state *vio = &port->vio;
+
+ /* reset the channel */
+ vio_link_state_change(vio, LDC_EVENT_RESET);
+ vnet_port_reset(port);
+ vio_port_up(vio);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(ldmvsw_open);
+
#ifdef CONFIG_NET_POLL_CONTROLLER
static void vsw_poll_controller(struct net_device *dev)
{
@@ -133,7 +147,7 @@
#endif
static const struct net_device_ops vsw_ops = {
- .ndo_open = sunvnet_open_common,
+ .ndo_open = ldmvsw_open,
.ndo_stop = sunvnet_close_common,
.ndo_set_rx_mode = vsw_set_rx_mode,
.ndo_set_mac_address = sunvnet_set_mac_addr_common,
@@ -365,6 +379,11 @@
napi_enable(&port->napi);
vio_port_up(&port->vio);
+ /* assure no carrier until we receive an LDC_EVENT_UP,
+ * even if the vsw config script tries to force us up
+ */
+ netif_carrier_off(dev);
+
netdev_info(dev, "LDOM vsw-port %pM\n", dev->dev_addr);
pr_info("%s: PORT ( remote-mac %pM%s )\n", dev->name,
diff --git a/drivers/net/ethernet/sun/niu.c b/drivers/net/ethernet/sun/niu.c
index 5797805..2dcca24 100644
--- a/drivers/net/ethernet/sun/niu.c
+++ b/drivers/net/ethernet/sun/niu.c
@@ -6813,7 +6813,8 @@
sizeof(info->bus_info));
}
-static int niu_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int niu_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct niu *np = netdev_priv(dev);
struct niu_link_config *lp;
@@ -6821,28 +6822,30 @@
lp = &np->link_config;
memset(cmd, 0, sizeof(*cmd));
- cmd->phy_address = np->phy_addr;
- cmd->supported = lp->supported;
- cmd->advertising = lp->active_advertising;
- cmd->autoneg = lp->active_autoneg;
- ethtool_cmd_speed_set(cmd, lp->active_speed);
- cmd->duplex = lp->active_duplex;
- cmd->port = (np->flags & NIU_FLAGS_FIBER) ? PORT_FIBRE : PORT_TP;
- cmd->transceiver = (np->flags & NIU_FLAGS_XCVR_SERDES) ?
- XCVR_EXTERNAL : XCVR_INTERNAL;
+ cmd->base.phy_address = np->phy_addr;
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ lp->supported);
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
+ lp->active_advertising);
+ cmd->base.autoneg = lp->active_autoneg;
+ cmd->base.speed = lp->active_speed;
+ cmd->base.duplex = lp->active_duplex;
+ cmd->base.port = (np->flags & NIU_FLAGS_FIBER) ? PORT_FIBRE : PORT_TP;
return 0;
}
-static int niu_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int niu_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct niu *np = netdev_priv(dev);
struct niu_link_config *lp = &np->link_config;
- lp->advertising = cmd->advertising;
- lp->speed = ethtool_cmd_speed(cmd);
- lp->duplex = cmd->duplex;
- lp->autoneg = cmd->autoneg;
+ ethtool_convert_link_mode_to_legacy_u32(&lp->advertising,
+ cmd->link_modes.advertising);
+ lp->speed = cmd->base.speed;
+ lp->duplex = cmd->base.duplex;
+ lp->autoneg = cmd->base.autoneg;
return niu_init_link(np);
}
@@ -7902,14 +7905,14 @@
.nway_reset = niu_nway_reset,
.get_eeprom_len = niu_get_eeprom_len,
.get_eeprom = niu_get_eeprom,
- .get_settings = niu_get_settings,
- .set_settings = niu_set_settings,
.get_strings = niu_get_strings,
.get_sset_count = niu_get_sset_count,
.get_ethtool_stats = niu_get_ethtool_stats,
.set_phys_id = niu_set_phys_id,
.get_rxnfc = niu_get_nfc,
.set_rxnfc = niu_set_nfc,
+ .get_link_ksettings = niu_get_link_ksettings,
+ .set_link_ksettings = niu_set_link_ksettings,
};
static int niu_ldg_assign_ldn(struct niu *np, struct niu_parent *parent,
diff --git a/drivers/net/ethernet/sun/sunbmac.c b/drivers/net/ethernet/sun/sunbmac.c
index c4caf48..3189722 100644
--- a/drivers/net/ethernet/sun/sunbmac.c
+++ b/drivers/net/ethernet/sun/sunbmac.c
@@ -169,7 +169,7 @@
static void bigmac_get_counters(struct bigmac *bp, void __iomem *bregs)
{
- struct net_device_stats *stats = &bp->enet_stats;
+ struct net_device_stats *stats = &bp->dev->stats;
stats->rx_crc_errors += sbus_readl(bregs + BMAC_RCRCECTR);
sbus_writel(0, bregs + BMAC_RCRCECTR);
@@ -774,8 +774,8 @@
if (this->tx_flags & TXD_OWN)
break;
skb = bp->tx_skbs[elem];
- bp->enet_stats.tx_packets++;
- bp->enet_stats.tx_bytes += skb->len;
+ dev->stats.tx_packets++;
+ dev->stats.tx_bytes += skb->len;
dma_unmap_single(&bp->bigmac_op->dev,
this->tx_addr, skb->len,
DMA_TO_DEVICE);
@@ -811,12 +811,12 @@
/* Check for errors. */
if (len < ETH_ZLEN) {
- bp->enet_stats.rx_errors++;
- bp->enet_stats.rx_length_errors++;
+ bp->dev->stats.rx_errors++;
+ bp->dev->stats.rx_length_errors++;
drop_it:
/* Return it to the BigMAC. */
- bp->enet_stats.rx_dropped++;
+ bp->dev->stats.rx_dropped++;
this->rx_flags =
(RXD_OWN | ((RX_BUF_ALLOC_SIZE - 34) & RXD_LENGTH));
goto next;
@@ -875,8 +875,8 @@
/* No checksums done by the BigMAC ;-( */
skb->protocol = eth_type_trans(skb, bp->dev);
netif_rx(skb);
- bp->enet_stats.rx_packets++;
- bp->enet_stats.rx_bytes += len;
+ bp->dev->stats.rx_packets++;
+ bp->dev->stats.rx_bytes += len;
next:
elem = NEXT_RX(elem);
this = &rxbase[elem];
@@ -987,7 +987,7 @@
struct bigmac *bp = netdev_priv(dev);
bigmac_get_counters(bp, bp->bregs);
- return &bp->enet_stats;
+ return &dev->stats;
}
static void bigmac_set_multicast(struct net_device *dev)
diff --git a/drivers/net/ethernet/sun/sunbmac.h b/drivers/net/ethernet/sun/sunbmac.h
index 532fc56..ee56930 100644
--- a/drivers/net/ethernet/sun/sunbmac.h
+++ b/drivers/net/ethernet/sun/sunbmac.h
@@ -311,7 +311,6 @@
enum bigmac_timer_state timer_state;
unsigned int timer_ticks;
- struct net_device_stats enet_stats;
struct platform_device *qec_op;
struct platform_device *bigmac_op;
struct net_device *dev;
diff --git a/drivers/net/ethernet/sun/sungem.c b/drivers/net/ethernet/sun/sungem.c
index 5c5952e..fa607d0 100644
--- a/drivers/net/ethernet/sun/sungem.c
+++ b/drivers/net/ethernet/sun/sungem.c
@@ -1250,12 +1250,18 @@
// XXX dbl check what that function should do when called on PCS PHY
-static void gem_begin_auto_negotiation(struct gem *gp, struct ethtool_cmd *ep)
+static void gem_begin_auto_negotiation(struct gem *gp,
+ const struct ethtool_link_ksettings *ep)
{
u32 advertise, features;
int autoneg;
int speed;
int duplex;
+ u32 advertising;
+
+ if (ep)
+ ethtool_convert_link_mode_to_legacy_u32(
+ &advertising, ep->link_modes.advertising);
if (gp->phy_type != phy_mii_mdio0 &&
gp->phy_type != phy_mii_mdio1)
@@ -1278,13 +1284,13 @@
/* Setup link parameters */
if (!ep)
goto start_aneg;
- if (ep->autoneg == AUTONEG_ENABLE) {
- advertise = ep->advertising;
+ if (ep->base.autoneg == AUTONEG_ENABLE) {
+ advertise = advertising;
autoneg = 1;
} else {
autoneg = 0;
- speed = ethtool_cmd_speed(ep);
- duplex = ep->duplex;
+ speed = ep->base.speed;
+ duplex = ep->base.duplex;
}
start_aneg:
@@ -2515,85 +2521,96 @@
strlcpy(info->bus_info, pci_name(gp->pdev), sizeof(info->bus_info));
}
-static int gem_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int gem_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct gem *gp = netdev_priv(dev);
+ u32 supported, advertising;
if (gp->phy_type == phy_mii_mdio0 ||
gp->phy_type == phy_mii_mdio1) {
if (gp->phy_mii.def)
- cmd->supported = gp->phy_mii.def->features;
+ supported = gp->phy_mii.def->features;
else
- cmd->supported = (SUPPORTED_10baseT_Half |
+ supported = (SUPPORTED_10baseT_Half |
SUPPORTED_10baseT_Full);
/* XXX hardcoded stuff for now */
- cmd->port = PORT_MII;
- cmd->transceiver = XCVR_EXTERNAL;
- cmd->phy_address = 0; /* XXX fixed PHYAD */
+ cmd->base.port = PORT_MII;
+ cmd->base.phy_address = 0; /* XXX fixed PHYAD */
/* Return current PHY settings */
- cmd->autoneg = gp->want_autoneg;
- ethtool_cmd_speed_set(cmd, gp->phy_mii.speed);
- cmd->duplex = gp->phy_mii.duplex;
- cmd->advertising = gp->phy_mii.advertising;
+ cmd->base.autoneg = gp->want_autoneg;
+ cmd->base.speed = gp->phy_mii.speed;
+ cmd->base.duplex = gp->phy_mii.duplex;
+ advertising = gp->phy_mii.advertising;
/* If we started with a forced mode, we don't have a default
* advertise set, we need to return something sensible so
* userland can re-enable autoneg properly.
*/
- if (cmd->advertising == 0)
- cmd->advertising = cmd->supported;
+ if (advertising == 0)
+ advertising = supported;
} else { // XXX PCS ?
- cmd->supported =
+ supported =
(SUPPORTED_10baseT_Half | SUPPORTED_10baseT_Full |
SUPPORTED_100baseT_Half | SUPPORTED_100baseT_Full |
SUPPORTED_Autoneg);
- cmd->advertising = cmd->supported;
- ethtool_cmd_speed_set(cmd, 0);
- cmd->duplex = cmd->port = cmd->phy_address =
- cmd->transceiver = cmd->autoneg = 0;
+ advertising = supported;
+ cmd->base.speed = 0;
+ cmd->base.duplex = 0;
+ cmd->base.port = 0;
+ cmd->base.phy_address = 0;
+ cmd->base.autoneg = 0;
/* serdes means usually a Fibre connector, with most fixed */
if (gp->phy_type == phy_serdes) {
- cmd->port = PORT_FIBRE;
- cmd->supported = (SUPPORTED_1000baseT_Half |
+ cmd->base.port = PORT_FIBRE;
+ supported = (SUPPORTED_1000baseT_Half |
SUPPORTED_1000baseT_Full |
SUPPORTED_FIBRE | SUPPORTED_Autoneg |
SUPPORTED_Pause | SUPPORTED_Asym_Pause);
- cmd->advertising = cmd->supported;
- cmd->transceiver = XCVR_INTERNAL;
+ advertising = supported;
if (gp->lstate == link_up)
- ethtool_cmd_speed_set(cmd, SPEED_1000);
- cmd->duplex = DUPLEX_FULL;
- cmd->autoneg = 1;
+ cmd->base.speed = SPEED_1000;
+ cmd->base.duplex = DUPLEX_FULL;
+ cmd->base.autoneg = 1;
}
}
- cmd->maxtxpkt = cmd->maxrxpkt = 0;
+
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
+ advertising);
return 0;
}
-static int gem_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int gem_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct gem *gp = netdev_priv(dev);
- u32 speed = ethtool_cmd_speed(cmd);
+ u32 speed = cmd->base.speed;
+ u32 advertising;
+
+ ethtool_convert_link_mode_to_legacy_u32(&advertising,
+ cmd->link_modes.advertising);
/* Verify the settings we care about. */
- if (cmd->autoneg != AUTONEG_ENABLE &&
- cmd->autoneg != AUTONEG_DISABLE)
+ if (cmd->base.autoneg != AUTONEG_ENABLE &&
+ cmd->base.autoneg != AUTONEG_DISABLE)
return -EINVAL;
- if (cmd->autoneg == AUTONEG_ENABLE &&
- cmd->advertising == 0)
+ if (cmd->base.autoneg == AUTONEG_ENABLE &&
+ advertising == 0)
return -EINVAL;
- if (cmd->autoneg == AUTONEG_DISABLE &&
+ if (cmd->base.autoneg == AUTONEG_DISABLE &&
((speed != SPEED_1000 &&
speed != SPEED_100 &&
speed != SPEED_10) ||
- (cmd->duplex != DUPLEX_HALF &&
- cmd->duplex != DUPLEX_FULL)))
+ (cmd->base.duplex != DUPLEX_HALF &&
+ cmd->base.duplex != DUPLEX_FULL)))
return -EINVAL;
/* Apply settings and restart link process. */
@@ -2666,13 +2683,13 @@
static const struct ethtool_ops gem_ethtool_ops = {
.get_drvinfo = gem_get_drvinfo,
.get_link = ethtool_op_get_link,
- .get_settings = gem_get_settings,
- .set_settings = gem_set_settings,
.nway_reset = gem_nway_reset,
.get_msglevel = gem_get_msglevel,
.set_msglevel = gem_set_msglevel,
.get_wol = gem_get_wol,
.set_wol = gem_set_wol,
+ .get_link_ksettings = gem_get_link_ksettings,
+ .set_link_ksettings = gem_set_link_ksettings,
};
static int gem_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
diff --git a/drivers/net/ethernet/sun/sunhme.c b/drivers/net/ethernet/sun/sunhme.c
index 72ff05c..9e983e1 100644
--- a/drivers/net/ethernet/sun/sunhme.c
+++ b/drivers/net/ethernet/sun/sunhme.c
@@ -933,7 +933,7 @@
/* hp->happy_lock must be held */
static void happy_meal_get_counters(struct happy_meal *hp, void __iomem *bregs)
{
- struct net_device_stats *stats = &hp->net_stats;
+ struct net_device_stats *stats = &hp->dev->stats;
stats->rx_crc_errors += hme_read32(hp, bregs + BMAC_RCRCECTR);
hme_write32(hp, bregs + BMAC_RCRCECTR, 0);
@@ -1294,9 +1294,10 @@
}
/* hp->happy_lock must be held */
-static void happy_meal_begin_auto_negotiation(struct happy_meal *hp,
- void __iomem *tregs,
- struct ethtool_cmd *ep)
+static void
+happy_meal_begin_auto_negotiation(struct happy_meal *hp,
+ void __iomem *tregs,
+ const struct ethtool_link_ksettings *ep)
{
int timeout;
@@ -1309,7 +1310,7 @@
/* XXX Check BMSR_ANEGCAPABLE, should not be necessary though. */
hp->sw_advertise = happy_meal_tcvr_read(hp, tregs, MII_ADVERTISE);
- if (ep == NULL || ep->autoneg == AUTONEG_ENABLE) {
+ if (!ep || ep->base.autoneg == AUTONEG_ENABLE) {
/* Advertise everything we can support. */
if (hp->sw_bmsr & BMSR_10HALF)
hp->sw_advertise |= (ADVERTISE_10HALF);
@@ -1384,14 +1385,14 @@
/* Disable auto-negotiation in BMCR, enable the duplex and
* speed setting, init the timer state machine, and fire it off.
*/
- if (ep == NULL || ep->autoneg == AUTONEG_ENABLE) {
+ if (!ep || ep->base.autoneg == AUTONEG_ENABLE) {
hp->sw_bmcr = BMCR_SPEED100;
} else {
- if (ethtool_cmd_speed(ep) == SPEED_100)
+ if (ep->base.speed == SPEED_100)
hp->sw_bmcr = BMCR_SPEED100;
else
hp->sw_bmcr = 0;
- if (ep->duplex == DUPLEX_FULL)
+ if (ep->base.duplex == DUPLEX_FULL)
hp->sw_bmcr |= BMCR_FULLDPLX;
}
happy_meal_tcvr_write(hp, tregs, MII_BMCR, hp->sw_bmcr);
@@ -1856,7 +1857,7 @@
if (status & GREG_STAT_TXLERR)
printk("LateError ");
if (status & GREG_STAT_TXPERR)
- printk("ParityErro ");
+ printk("ParityError ");
if (status & GREG_STAT_TXTERR)
printk("TagBotch ");
printk("]\n");
@@ -1946,7 +1947,7 @@
break;
}
hp->tx_skbs[elem] = NULL;
- hp->net_stats.tx_bytes += skb->len;
+ dev->stats.tx_bytes += skb->len;
for (frag = 0; frag <= skb_shinfo(skb)->nr_frags; frag++) {
dma_addr = hme_read_desc32(hp, &this->tx_addr);
@@ -1963,7 +1964,7 @@
}
dev_kfree_skb_irq(skb);
- hp->net_stats.tx_packets++;
+ dev->stats.tx_packets++;
}
hp->tx_old = elem;
TXD((">"));
@@ -2008,17 +2009,17 @@
/* Check for errors. */
if ((len < ETH_ZLEN) || (flags & RXFLAG_OVERFLOW)) {
RXD(("ERR(%08x)]", flags));
- hp->net_stats.rx_errors++;
+ dev->stats.rx_errors++;
if (len < ETH_ZLEN)
- hp->net_stats.rx_length_errors++;
+ dev->stats.rx_length_errors++;
if (len & (RXFLAG_OVERFLOW >> 16)) {
- hp->net_stats.rx_over_errors++;
- hp->net_stats.rx_fifo_errors++;
+ dev->stats.rx_over_errors++;
+ dev->stats.rx_fifo_errors++;
}
/* Return it to the Happy meal. */
drop_it:
- hp->net_stats.rx_dropped++;
+ dev->stats.rx_dropped++;
hme_write_rxd(hp, this,
(RXFLAG_OWN|((RX_BUF_ALLOC_SIZE-RX_OFFSET)<<16)),
dma_addr);
@@ -2083,8 +2084,8 @@
skb->protocol = eth_type_trans(skb, dev);
netif_rx(skb);
- hp->net_stats.rx_packets++;
- hp->net_stats.rx_bytes += len;
+ dev->stats.rx_packets++;
+ dev->stats.rx_bytes += len;
next:
elem = NEXT_RX(elem);
this = &rxbase[elem];
@@ -2395,7 +2396,7 @@
happy_meal_get_counters(hp, hp->bigmacregs);
spin_unlock_irq(&hp->happy_lock);
- return &hp->net_stats;
+ return &dev->stats;
}
static void happy_meal_set_multicast(struct net_device *dev)
@@ -2434,20 +2435,21 @@
}
/* Ethtool support... */
-static int hme_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int hme_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct happy_meal *hp = netdev_priv(dev);
u32 speed;
+ u32 supported;
- cmd->supported =
+ supported =
(SUPPORTED_10baseT_Half | SUPPORTED_10baseT_Full |
SUPPORTED_100baseT_Half | SUPPORTED_100baseT_Full |
SUPPORTED_Autoneg | SUPPORTED_TP | SUPPORTED_MII);
/* XXX hardcoded stuff for now */
- cmd->port = PORT_TP; /* XXX no MII support */
- cmd->transceiver = XCVR_INTERNAL; /* XXX no external xcvr support */
- cmd->phy_address = 0; /* XXX fixed PHYAD */
+ cmd->base.port = PORT_TP; /* XXX no MII support */
+ cmd->base.phy_address = 0; /* XXX fixed PHYAD */
/* Record PHY settings. */
spin_lock_irq(&hp->happy_lock);
@@ -2456,41 +2458,45 @@
spin_unlock_irq(&hp->happy_lock);
if (hp->sw_bmcr & BMCR_ANENABLE) {
- cmd->autoneg = AUTONEG_ENABLE;
+ cmd->base.autoneg = AUTONEG_ENABLE;
speed = ((hp->sw_lpa & (LPA_100HALF | LPA_100FULL)) ?
SPEED_100 : SPEED_10);
if (speed == SPEED_100)
- cmd->duplex =
+ cmd->base.duplex =
(hp->sw_lpa & (LPA_100FULL)) ?
DUPLEX_FULL : DUPLEX_HALF;
else
- cmd->duplex =
+ cmd->base.duplex =
(hp->sw_lpa & (LPA_10FULL)) ?
DUPLEX_FULL : DUPLEX_HALF;
} else {
- cmd->autoneg = AUTONEG_DISABLE;
+ cmd->base.autoneg = AUTONEG_DISABLE;
speed = (hp->sw_bmcr & BMCR_SPEED100) ? SPEED_100 : SPEED_10;
- cmd->duplex =
+ cmd->base.duplex =
(hp->sw_bmcr & BMCR_FULLDPLX) ?
DUPLEX_FULL : DUPLEX_HALF;
}
- ethtool_cmd_speed_set(cmd, speed);
+ cmd->base.speed = speed;
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+
return 0;
}
-static int hme_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int hme_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct happy_meal *hp = netdev_priv(dev);
/* Verify the settings we care about. */
- if (cmd->autoneg != AUTONEG_ENABLE &&
- cmd->autoneg != AUTONEG_DISABLE)
+ if (cmd->base.autoneg != AUTONEG_ENABLE &&
+ cmd->base.autoneg != AUTONEG_DISABLE)
return -EINVAL;
- if (cmd->autoneg == AUTONEG_DISABLE &&
- ((ethtool_cmd_speed(cmd) != SPEED_100 &&
- ethtool_cmd_speed(cmd) != SPEED_10) ||
- (cmd->duplex != DUPLEX_HALF &&
- cmd->duplex != DUPLEX_FULL)))
+ if (cmd->base.autoneg == AUTONEG_DISABLE &&
+ ((cmd->base.speed != SPEED_100 &&
+ cmd->base.speed != SPEED_10) ||
+ (cmd->base.duplex != DUPLEX_HALF &&
+ cmd->base.duplex != DUPLEX_FULL)))
return -EINVAL;
/* Ok, do it to it. */
@@ -2537,10 +2543,10 @@
}
static const struct ethtool_ops hme_ethtool_ops = {
- .get_settings = hme_get_settings,
- .set_settings = hme_set_settings,
.get_drvinfo = hme_get_drvinfo,
.get_link = hme_get_link,
+ .get_link_ksettings = hme_get_link_ksettings,
+ .set_link_ksettings = hme_set_link_ksettings,
};
static int hme_version_printed;
diff --git a/drivers/net/ethernet/sun/sunhme.h b/drivers/net/ethernet/sun/sunhme.h
index 4a8d5b1..3af540a 100644
--- a/drivers/net/ethernet/sun/sunhme.h
+++ b/drivers/net/ethernet/sun/sunhme.h
@@ -418,8 +418,6 @@
int rx_new, tx_new, rx_old, tx_old;
- struct net_device_stats net_stats; /* Statistical counters */
-
#if defined(CONFIG_SBUS) && defined(CONFIG_PCI)
u32 (*read32)(void __iomem *);
void (*write32)(void __iomem *, u32);
diff --git a/drivers/net/ethernet/sun/sunvnet.c b/drivers/net/ethernet/sun/sunvnet.c
index 4cc2571..0b95105 100644
--- a/drivers/net/ethernet/sun/sunvnet.c
+++ b/drivers/net/ethernet/sun/sunvnet.c
@@ -1,7 +1,7 @@
/* sunvnet.c: Sun LDOM Virtual Network Driver.
*
* Copyright (C) 2007, 2008 David S. Miller <davem@davemloft.net>
- * Copyright (C) 2016 Oracle. All rights reserved.
+ * Copyright (C) 2016-2017 Oracle. All rights reserved.
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
@@ -77,11 +77,125 @@
vp->msg_enable = value;
}
+static const struct {
+ const char string[ETH_GSTRING_LEN];
+} ethtool_stats_keys[] = {
+ { "rx_packets" },
+ { "tx_packets" },
+ { "rx_bytes" },
+ { "tx_bytes" },
+ { "rx_errors" },
+ { "tx_errors" },
+ { "rx_dropped" },
+ { "tx_dropped" },
+ { "multicast" },
+ { "rx_length_errors" },
+ { "rx_frame_errors" },
+ { "rx_missed_errors" },
+ { "tx_carrier_errors" },
+ { "nports" },
+};
+
+static int vnet_get_sset_count(struct net_device *dev, int sset)
+{
+ struct vnet *vp = (struct vnet *)netdev_priv(dev);
+
+ switch (sset) {
+ case ETH_SS_STATS:
+ return ARRAY_SIZE(ethtool_stats_keys)
+ + (NUM_VNET_PORT_STATS * vp->nports);
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+
+static void vnet_get_strings(struct net_device *dev, u32 stringset, u8 *buf)
+{
+ struct vnet *vp = (struct vnet *)netdev_priv(dev);
+ struct vnet_port *port;
+ char *p = (char *)buf;
+
+ switch (stringset) {
+ case ETH_SS_STATS:
+ memcpy(buf, ðtool_stats_keys, sizeof(ethtool_stats_keys));
+ p += sizeof(ethtool_stats_keys);
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(port, &vp->port_list, list) {
+ snprintf(p, ETH_GSTRING_LEN, "p%u.%s-%pM",
+ port->q_index, port->switch_port ? "s" : "q",
+ port->raddr);
+ p += ETH_GSTRING_LEN;
+ snprintf(p, ETH_GSTRING_LEN, "p%u.rx_packets",
+ port->q_index);
+ p += ETH_GSTRING_LEN;
+ snprintf(p, ETH_GSTRING_LEN, "p%u.tx_packets",
+ port->q_index);
+ p += ETH_GSTRING_LEN;
+ snprintf(p, ETH_GSTRING_LEN, "p%u.rx_bytes",
+ port->q_index);
+ p += ETH_GSTRING_LEN;
+ snprintf(p, ETH_GSTRING_LEN, "p%u.tx_bytes",
+ port->q_index);
+ p += ETH_GSTRING_LEN;
+ snprintf(p, ETH_GSTRING_LEN, "p%u.event_up",
+ port->q_index);
+ p += ETH_GSTRING_LEN;
+ snprintf(p, ETH_GSTRING_LEN, "p%u.event_reset",
+ port->q_index);
+ p += ETH_GSTRING_LEN;
+ }
+ rcu_read_unlock();
+ break;
+ default:
+ WARN_ON(1);
+ break;
+ }
+}
+
+static void vnet_get_ethtool_stats(struct net_device *dev,
+ struct ethtool_stats *estats, u64 *data)
+{
+ struct vnet *vp = (struct vnet *)netdev_priv(dev);
+ struct vnet_port *port;
+ int i = 0;
+
+ data[i++] = dev->stats.rx_packets;
+ data[i++] = dev->stats.tx_packets;
+ data[i++] = dev->stats.rx_bytes;
+ data[i++] = dev->stats.tx_bytes;
+ data[i++] = dev->stats.rx_errors;
+ data[i++] = dev->stats.tx_errors;
+ data[i++] = dev->stats.rx_dropped;
+ data[i++] = dev->stats.tx_dropped;
+ data[i++] = dev->stats.multicast;
+ data[i++] = dev->stats.rx_length_errors;
+ data[i++] = dev->stats.rx_frame_errors;
+ data[i++] = dev->stats.rx_missed_errors;
+ data[i++] = dev->stats.tx_carrier_errors;
+ data[i++] = vp->nports;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(port, &vp->port_list, list) {
+ data[i++] = port->q_index;
+ data[i++] = port->stats.rx_packets;
+ data[i++] = port->stats.tx_packets;
+ data[i++] = port->stats.rx_bytes;
+ data[i++] = port->stats.tx_bytes;
+ data[i++] = port->stats.event_up;
+ data[i++] = port->stats.event_reset;
+ }
+ rcu_read_unlock();
+}
+
static const struct ethtool_ops vnet_ethtool_ops = {
.get_drvinfo = vnet_get_drvinfo,
.get_msglevel = vnet_get_msglevel,
.set_msglevel = vnet_set_msglevel,
.get_link = ethtool_op_get_link,
+ .get_sset_count = vnet_get_sset_count,
+ .get_strings = vnet_get_strings,
+ .get_ethtool_stats = vnet_get_ethtool_stats,
};
static LIST_HEAD(vnet_list);
diff --git a/drivers/net/ethernet/sun/sunvnet_common.c b/drivers/net/ethernet/sun/sunvnet_common.c
index fa2d11c..9e86833 100644
--- a/drivers/net/ethernet/sun/sunvnet_common.c
+++ b/drivers/net/ethernet/sun/sunvnet_common.c
@@ -1,7 +1,7 @@
/* sunvnet.c: Sun LDOM Virtual Network Driver.
*
* Copyright (C) 2007, 2008 David S. Miller <davem@davemloft.net>
- * Copyright (C) 2016 Oracle. All rights reserved.
+ * Copyright (C) 2016-2017 Oracle. All rights reserved.
*/
#include <linux/module.h>
@@ -43,7 +43,6 @@
MODULE_VERSION("1.1");
static int __vnet_tx_trigger(struct vnet_port *port, u32 start);
-static void vnet_port_reset(struct vnet_port *port);
static inline u32 vnet_tx_dring_avail(struct vio_dring_state *dr)
{
@@ -410,8 +409,12 @@
skb->ip_summed = port->switch_port ? CHECKSUM_NONE : CHECKSUM_PARTIAL;
+ if (unlikely(is_multicast_ether_addr(eth_hdr(skb)->h_dest)))
+ dev->stats.multicast++;
dev->stats.rx_packets++;
dev->stats.rx_bytes += len;
+ port->stats.rx_packets++;
+ port->stats.rx_bytes += len;
napi_gro_receive(&port->napi, skb);
return 0;
@@ -747,6 +750,13 @@
/* RESET takes precedent over any other event */
if (port->rx_event & LDC_EVENT_RESET) {
+ /* a link went down */
+
+ if (port->vsw == 1) {
+ netif_tx_stop_all_queues(dev);
+ netif_carrier_off(dev);
+ }
+
vio_link_state_change(vio, LDC_EVENT_RESET);
vnet_port_reset(port);
vio_port_up(vio);
@@ -762,12 +772,21 @@
maybe_tx_wakeup(port);
port->rx_event = 0;
+ port->stats.event_reset++;
return 0;
}
if (port->rx_event & LDC_EVENT_UP) {
+ /* a link came up */
+
+ if (port->vsw == 1) {
+ netif_carrier_on(port->dev);
+ netif_tx_start_all_queues(port->dev);
+ }
+
vio_link_state_change(vio, LDC_EVENT_UP);
port->rx_event = 0;
+ port->stats.event_up++;
return 0;
}
@@ -1417,6 +1436,8 @@
dev->stats.tx_packets++;
dev->stats.tx_bytes += port->tx_bufs[txi].skb->len;
+ port->stats.tx_packets++;
+ port->stats.tx_bytes += port->tx_bufs[txi].skb->len;
dr->prod = (dr->prod + 1) & (VNET_TX_RING_SIZE - 1);
if (unlikely(vnet_tx_dring_avail(dr) < 1)) {
@@ -1631,7 +1652,7 @@
}
EXPORT_SYMBOL_GPL(sunvnet_port_free_tx_bufs_common);
-static void vnet_port_reset(struct vnet_port *port)
+void vnet_port_reset(struct vnet_port *port)
{
del_timer(&port->clean_timer);
sunvnet_port_free_tx_bufs_common(port);
@@ -1639,6 +1660,7 @@
port->tso = (port->vsw == 0); /* no tso in vsw, misbehaves in bridge */
port->tsolen = 0;
}
+EXPORT_SYMBOL_GPL(vnet_port_reset);
static int vnet_port_alloc_tx_ring(struct vnet_port *port)
{
@@ -1708,20 +1730,32 @@
void sunvnet_port_add_txq_common(struct vnet_port *port)
{
struct vnet *vp = port->vp;
- int n;
+ int smallest = 0;
+ int i;
- n = vp->nports++;
- n = n & (VNET_MAX_TXQS - 1);
- port->q_index = n;
- netif_tx_wake_queue(netdev_get_tx_queue(VNET_PORT_TO_NET_DEVICE(port),
- port->q_index));
+ /* find the first least-used q
+ * When there are more ldoms than q's, we start to
+ * double up on ports per queue.
+ */
+ for (i = 0; i < VNET_MAX_TXQS; i++) {
+ if (vp->q_used[i] == 0) {
+ smallest = i;
+ break;
+ }
+ if (vp->q_used[i] < vp->q_used[smallest])
+ smallest = i;
+ }
+
+ vp->nports++;
+ vp->q_used[smallest]++;
+ port->q_index = smallest;
}
EXPORT_SYMBOL_GPL(sunvnet_port_add_txq_common);
void sunvnet_port_rm_txq_common(struct vnet_port *port)
{
port->vp->nports--;
- netif_tx_stop_queue(netdev_get_tx_queue(VNET_PORT_TO_NET_DEVICE(port),
- port->q_index));
+ port->vp->q_used[port->q_index]--;
+ port->q_index = 0;
}
EXPORT_SYMBOL_GPL(sunvnet_port_rm_txq_common);
diff --git a/drivers/net/ethernet/sun/sunvnet_common.h b/drivers/net/ethernet/sun/sunvnet_common.h
index ce5c824..b20d6fa 100644
--- a/drivers/net/ethernet/sun/sunvnet_common.h
+++ b/drivers/net/ethernet/sun/sunvnet_common.h
@@ -35,6 +35,19 @@
struct vnet;
+struct vnet_port_stats {
+ /* keep them all the same size */
+ u32 rx_bytes;
+ u32 tx_bytes;
+ u32 rx_packets;
+ u32 tx_packets;
+ u32 event_up;
+ u32 event_reset;
+ u32 q_placeholder;
+};
+
+#define NUM_VNET_PORT_STATS (sizeof(struct vnet_port_stats) / sizeof(u32))
+
/* Structure to describe a vnet-port or vsw-port in the MD.
* If the vsw bit is set, this structure represents a vswitch
* port, and the net_device can be found from ->dev. If the
@@ -44,6 +57,8 @@
struct vnet_port {
struct vio_driver_state vio;
+ struct vnet_port_stats stats;
+
struct hlist_node hash;
u8 raddr[ETH_ALEN];
unsigned switch_port:1;
@@ -97,22 +112,15 @@
};
struct vnet {
- /* Protects port_list and port_hash. */
- spinlock_t lock;
-
+ spinlock_t lock; /* Protects port_list and port_hash. */
struct net_device *dev;
-
u32 msg_enable;
-
+ u8 q_used[VNET_MAX_TXQS];
struct list_head port_list;
-
struct hlist_head port_hash[VNET_PORT_HASH_SIZE];
-
struct vnet_mcast_entry *mcast_list;
-
struct list_head list;
u64 local_mac;
-
int nports;
};
@@ -139,6 +147,7 @@
void sunvnet_handshake_complete_common(struct vio_driver_state *vio);
int sunvnet_poll_common(struct napi_struct *napi, int budget);
void sunvnet_port_free_tx_bufs_common(struct vnet_port *port);
+void vnet_port_reset(struct vnet_port *port);
bool sunvnet_port_is_up_common(struct vnet_port *vnet);
void sunvnet_port_add_txq_common(struct vnet_port *port);
void sunvnet_port_rm_txq_common(struct vnet_port *port);
diff --git a/drivers/net/ethernet/synopsys/Kconfig b/drivers/net/ethernet/synopsys/Kconfig
new file mode 100644
index 0000000..a950388
--- /dev/null
+++ b/drivers/net/ethernet/synopsys/Kconfig
@@ -0,0 +1,41 @@
+#
+# Synopsys network device configuration
+#
+
+config NET_VENDOR_SYNOPSYS
+ bool "Synopsys devices"
+ default y
+ ---help---
+ If you have a network (Ethernet) device belonging to this class, say Y.
+
+ Note that the answer to this question doesn't directly affect the
+ kernel: saying N will just cause the configurator to skip all
+ the questions about Synopsys devices. If you say Y, you will be asked
+ for your specific device in the following questions.
+
+if NET_VENDOR_SYNOPSYS
+
+config DWC_XLGMAC
+ tristate "Synopsys DWC Enterprise Ethernet (XLGMAC) driver support"
+ depends on HAS_IOMEM && HAS_DMA
+ select BITREVERSE
+ select CRC32
+ ---help---
+ This driver supports the Synopsys DesignWare Cores Enterprise
+ Ethernet (dwc-xlgmac).
+
+if DWC_XLGMAC
+
+config DWC_XLGMAC_PCI
+ tristate "XLGMAC PCI bus support"
+ depends on DWC_XLGMAC && PCI
+ ---help---
+ This selects the pci bus support for the dwc-xlgmac driver.
+ This driver was tested on Synopsys XLGMAC IP Prototyping Kit.
+
+ If you have a controller with this interface, say Y or M here.
+ If unsure, say N.
+
+endif # DWC_XLGMAC
+
+endif # NET_VENDOR_SYNOPSYS
diff --git a/drivers/net/ethernet/synopsys/Makefile b/drivers/net/ethernet/synopsys/Makefile
new file mode 100644
index 0000000..0ad0191
--- /dev/null
+++ b/drivers/net/ethernet/synopsys/Makefile
@@ -0,0 +1,10 @@
+#
+# Makefile for the Synopsys network device drivers.
+#
+
+obj-$(CONFIG_DWC_XLGMAC) += dwc-xlgmac.o
+dwc-xlgmac-objs := dwc-xlgmac-net.o dwc-xlgmac-desc.o \
+ dwc-xlgmac-hw.o dwc-xlgmac-common.o \
+ dwc-xlgmac-ethtool.o
+
+dwc-xlgmac-$(CONFIG_DWC_XLGMAC_PCI) += dwc-xlgmac-pci.o
diff --git a/drivers/net/ethernet/synopsys/dwc-xlgmac-common.c b/drivers/net/ethernet/synopsys/dwc-xlgmac-common.c
new file mode 100644
index 0000000..d655a42
--- /dev/null
+++ b/drivers/net/ethernet/synopsys/dwc-xlgmac-common.c
@@ -0,0 +1,737 @@
+/* Synopsys DesignWare Core Enterprise Ethernet (XLGMAC) Driver
+ *
+ * Copyright (c) 2017 Synopsys, Inc. (www.synopsys.com)
+ *
+ * This program is dual-licensed; you may select either version 2 of
+ * the GNU General Public License ("GPL") or BSD license ("BSD").
+ *
+ * This Synopsys DWC XLGMAC software driver and associated documentation
+ * (hereinafter the "Software") is an unsupported proprietary work of
+ * Synopsys, Inc. unless otherwise expressly agreed to in writing between
+ * Synopsys and you. The Software IS NOT an item of Licensed Software or a
+ * Licensed Product under any End User Software License Agreement or
+ * Agreement for Licensed Products with Synopsys or any supplement thereto.
+ * Synopsys is a registered trademark of Synopsys, Inc. Other names included
+ * in the SOFTWARE may be the trademarks of their respective owners.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include "dwc-xlgmac.h"
+#include "dwc-xlgmac-reg.h"
+
+MODULE_LICENSE("Dual BSD/GPL");
+
+static int debug = -1;
+module_param(debug, int, 0644);
+MODULE_PARM_DESC(debug, "DWC ethernet debug level (0=none,...,16=all)");
+static const u32 default_msg_level = (NETIF_MSG_LINK | NETIF_MSG_IFDOWN |
+ NETIF_MSG_IFUP);
+
+static unsigned char dev_addr[6] = {0, 0x55, 0x7b, 0xb5, 0x7d, 0xf7};
+
+static void xlgmac_read_mac_addr(struct xlgmac_pdata *pdata)
+{
+ struct net_device *netdev = pdata->netdev;
+
+ /* Currently it uses a static mac address for test */
+ memcpy(pdata->mac_addr, dev_addr, netdev->addr_len);
+}
+
+static void xlgmac_default_config(struct xlgmac_pdata *pdata)
+{
+ pdata->tx_osp_mode = DMA_OSP_ENABLE;
+ pdata->tx_sf_mode = MTL_TSF_ENABLE;
+ pdata->rx_sf_mode = MTL_RSF_DISABLE;
+ pdata->pblx8 = DMA_PBL_X8_ENABLE;
+ pdata->tx_pbl = DMA_PBL_32;
+ pdata->rx_pbl = DMA_PBL_32;
+ pdata->tx_threshold = MTL_TX_THRESHOLD_128;
+ pdata->rx_threshold = MTL_RX_THRESHOLD_128;
+ pdata->tx_pause = 1;
+ pdata->rx_pause = 1;
+ pdata->phy_speed = SPEED_25000;
+ pdata->sysclk_rate = XLGMAC_SYSCLOCK;
+
+ strlcpy(pdata->drv_name, XLGMAC_DRV_NAME, sizeof(pdata->drv_name));
+ strlcpy(pdata->drv_ver, XLGMAC_DRV_VERSION, sizeof(pdata->drv_ver));
+}
+
+static void xlgmac_init_all_ops(struct xlgmac_pdata *pdata)
+{
+ xlgmac_init_desc_ops(&pdata->desc_ops);
+ xlgmac_init_hw_ops(&pdata->hw_ops);
+}
+
+static int xlgmac_init(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_hw_ops *hw_ops = &pdata->hw_ops;
+ struct net_device *netdev = pdata->netdev;
+ unsigned int i;
+ int ret;
+
+ /* Set default configuration data */
+ xlgmac_default_config(pdata);
+
+ /* Set irq, base_addr, MAC address, */
+ netdev->irq = pdata->dev_irq;
+ netdev->base_addr = (unsigned long)pdata->mac_regs;
+ xlgmac_read_mac_addr(pdata);
+ memcpy(netdev->dev_addr, pdata->mac_addr, netdev->addr_len);
+
+ /* Set all the function pointers */
+ xlgmac_init_all_ops(pdata);
+
+ /* Issue software reset to device */
+ hw_ops->exit(pdata);
+
+ /* Populate the hardware features */
+ xlgmac_get_all_hw_features(pdata);
+ xlgmac_print_all_hw_features(pdata);
+
+ /* TODO: Set the PHY mode to XLGMII */
+
+ /* Set the DMA mask */
+ ret = dma_set_mask_and_coherent(pdata->dev,
+ DMA_BIT_MASK(pdata->hw_feat.dma_width));
+ if (ret) {
+ dev_err(pdata->dev, "dma_set_mask_and_coherent failed\n");
+ return ret;
+ }
+
+ /* Channel and ring params initializtion
+ * pdata->channel_count;
+ * pdata->tx_ring_count;
+ * pdata->rx_ring_count;
+ * pdata->tx_desc_count;
+ * pdata->rx_desc_count;
+ */
+ BUILD_BUG_ON_NOT_POWER_OF_2(XLGMAC_TX_DESC_CNT);
+ pdata->tx_desc_count = XLGMAC_TX_DESC_CNT;
+ if (pdata->tx_desc_count & (pdata->tx_desc_count - 1)) {
+ dev_err(pdata->dev, "tx descriptor count (%d) is not valid\n",
+ pdata->tx_desc_count);
+ ret = -EINVAL;
+ return ret;
+ }
+ BUILD_BUG_ON_NOT_POWER_OF_2(XLGMAC_RX_DESC_CNT);
+ pdata->rx_desc_count = XLGMAC_RX_DESC_CNT;
+ if (pdata->rx_desc_count & (pdata->rx_desc_count - 1)) {
+ dev_err(pdata->dev, "rx descriptor count (%d) is not valid\n",
+ pdata->rx_desc_count);
+ ret = -EINVAL;
+ return ret;
+ }
+
+ pdata->tx_ring_count = min_t(unsigned int, num_online_cpus(),
+ pdata->hw_feat.tx_ch_cnt);
+ pdata->tx_ring_count = min_t(unsigned int, pdata->tx_ring_count,
+ pdata->hw_feat.tx_q_cnt);
+ pdata->tx_q_count = pdata->tx_ring_count;
+ ret = netif_set_real_num_tx_queues(netdev, pdata->tx_q_count);
+ if (ret) {
+ dev_err(pdata->dev, "error setting real tx queue count\n");
+ return ret;
+ }
+
+ pdata->rx_ring_count = min_t(unsigned int,
+ netif_get_num_default_rss_queues(),
+ pdata->hw_feat.rx_ch_cnt);
+ pdata->rx_ring_count = min_t(unsigned int, pdata->rx_ring_count,
+ pdata->hw_feat.rx_q_cnt);
+ pdata->rx_q_count = pdata->rx_ring_count;
+ ret = netif_set_real_num_rx_queues(netdev, pdata->rx_q_count);
+ if (ret) {
+ dev_err(pdata->dev, "error setting real rx queue count\n");
+ return ret;
+ }
+
+ pdata->channel_count =
+ max_t(unsigned int, pdata->tx_ring_count, pdata->rx_ring_count);
+
+ /* Initialize RSS hash key and lookup table */
+ netdev_rss_key_fill(pdata->rss_key, sizeof(pdata->rss_key));
+
+ for (i = 0; i < XLGMAC_RSS_MAX_TABLE_SIZE; i++)
+ pdata->rss_table[i] = XLGMAC_SET_REG_BITS(
+ pdata->rss_table[i],
+ MAC_RSSDR_DMCH_POS,
+ MAC_RSSDR_DMCH_LEN,
+ i % pdata->rx_ring_count);
+
+ pdata->rss_options = XLGMAC_SET_REG_BITS(
+ pdata->rss_options,
+ MAC_RSSCR_IP2TE_POS,
+ MAC_RSSCR_IP2TE_LEN, 1);
+ pdata->rss_options = XLGMAC_SET_REG_BITS(
+ pdata->rss_options,
+ MAC_RSSCR_TCP4TE_POS,
+ MAC_RSSCR_TCP4TE_LEN, 1);
+ pdata->rss_options = XLGMAC_SET_REG_BITS(
+ pdata->rss_options,
+ MAC_RSSCR_UDP4TE_POS,
+ MAC_RSSCR_UDP4TE_LEN, 1);
+
+ /* Set device operations */
+ netdev->netdev_ops = xlgmac_get_netdev_ops();
+ netdev->ethtool_ops = xlgmac_get_ethtool_ops();
+
+ /* Set device features */
+ if (pdata->hw_feat.tso) {
+ netdev->hw_features = NETIF_F_TSO;
+ netdev->hw_features |= NETIF_F_TSO6;
+ netdev->hw_features |= NETIF_F_SG;
+ netdev->hw_features |= NETIF_F_IP_CSUM;
+ netdev->hw_features |= NETIF_F_IPV6_CSUM;
+ } else if (pdata->hw_feat.tx_coe) {
+ netdev->hw_features = NETIF_F_IP_CSUM;
+ netdev->hw_features |= NETIF_F_IPV6_CSUM;
+ }
+
+ if (pdata->hw_feat.rx_coe) {
+ netdev->hw_features |= NETIF_F_RXCSUM;
+ netdev->hw_features |= NETIF_F_GRO;
+ }
+
+ if (pdata->hw_feat.rss)
+ netdev->hw_features |= NETIF_F_RXHASH;
+
+ netdev->vlan_features |= netdev->hw_features;
+
+ netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_RX;
+ if (pdata->hw_feat.sa_vlan_ins)
+ netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_TX;
+ if (pdata->hw_feat.vlhash)
+ netdev->hw_features |= NETIF_F_HW_VLAN_CTAG_FILTER;
+
+ netdev->features |= netdev->hw_features;
+ pdata->netdev_features = netdev->features;
+
+ netdev->priv_flags |= IFF_UNICAST_FLT;
+
+ /* Use default watchdog timeout */
+ netdev->watchdog_timeo = 0;
+
+ /* Tx coalesce parameters initialization */
+ pdata->tx_usecs = XLGMAC_INIT_DMA_TX_USECS;
+ pdata->tx_frames = XLGMAC_INIT_DMA_TX_FRAMES;
+
+ /* Rx coalesce parameters initialization */
+ pdata->rx_riwt = hw_ops->usec_to_riwt(pdata, XLGMAC_INIT_DMA_RX_USECS);
+ pdata->rx_usecs = XLGMAC_INIT_DMA_RX_USECS;
+ pdata->rx_frames = XLGMAC_INIT_DMA_RX_FRAMES;
+
+ return 0;
+}
+
+int xlgmac_drv_probe(struct device *dev, struct xlgmac_resources *res)
+{
+ struct xlgmac_pdata *pdata;
+ struct net_device *netdev;
+ int ret;
+
+ netdev = alloc_etherdev_mq(sizeof(struct xlgmac_pdata),
+ XLGMAC_MAX_DMA_CHANNELS);
+
+ if (!netdev) {
+ dev_err(dev, "alloc_etherdev failed\n");
+ return -ENOMEM;
+ }
+
+ SET_NETDEV_DEV(netdev, dev);
+ dev_set_drvdata(dev, netdev);
+ pdata = netdev_priv(netdev);
+ pdata->dev = dev;
+ pdata->netdev = netdev;
+
+ pdata->dev_irq = res->irq;
+ pdata->mac_regs = res->addr;
+
+ mutex_init(&pdata->rss_mutex);
+ pdata->msg_enable = netif_msg_init(debug, default_msg_level);
+
+ ret = xlgmac_init(pdata);
+ if (ret) {
+ dev_err(dev, "xlgmac init failed\n");
+ goto err_free_netdev;
+ }
+
+ ret = register_netdev(netdev);
+ if (ret) {
+ dev_err(dev, "net device registration failed\n");
+ goto err_free_netdev;
+ }
+
+ return 0;
+
+err_free_netdev:
+ free_netdev(netdev);
+
+ return ret;
+}
+
+int xlgmac_drv_remove(struct device *dev)
+{
+ struct net_device *netdev = dev_get_drvdata(dev);
+
+ unregister_netdev(netdev);
+ free_netdev(netdev);
+
+ return 0;
+}
+
+void xlgmac_dump_tx_desc(struct xlgmac_pdata *pdata,
+ struct xlgmac_ring *ring,
+ unsigned int idx,
+ unsigned int count,
+ unsigned int flag)
+{
+ struct xlgmac_desc_data *desc_data;
+ struct xlgmac_dma_desc *dma_desc;
+
+ while (count--) {
+ desc_data = XLGMAC_GET_DESC_DATA(ring, idx);
+ dma_desc = desc_data->dma_desc;
+
+ netdev_dbg(pdata->netdev, "TX: dma_desc=%p, dma_desc_addr=%pad\n",
+ desc_data->dma_desc, &desc_data->dma_desc_addr);
+ netdev_dbg(pdata->netdev,
+ "TX_NORMAL_DESC[%d %s] = %08x:%08x:%08x:%08x\n", idx,
+ (flag == 1) ? "QUEUED FOR TX" : "TX BY DEVICE",
+ le32_to_cpu(dma_desc->desc0),
+ le32_to_cpu(dma_desc->desc1),
+ le32_to_cpu(dma_desc->desc2),
+ le32_to_cpu(dma_desc->desc3));
+
+ idx++;
+ }
+}
+
+void xlgmac_dump_rx_desc(struct xlgmac_pdata *pdata,
+ struct xlgmac_ring *ring,
+ unsigned int idx)
+{
+ struct xlgmac_desc_data *desc_data;
+ struct xlgmac_dma_desc *dma_desc;
+
+ desc_data = XLGMAC_GET_DESC_DATA(ring, idx);
+ dma_desc = desc_data->dma_desc;
+
+ netdev_dbg(pdata->netdev, "RX: dma_desc=%p, dma_desc_addr=%pad\n",
+ desc_data->dma_desc, &desc_data->dma_desc_addr);
+ netdev_dbg(pdata->netdev,
+ "RX_NORMAL_DESC[%d RX BY DEVICE] = %08x:%08x:%08x:%08x\n",
+ idx,
+ le32_to_cpu(dma_desc->desc0),
+ le32_to_cpu(dma_desc->desc1),
+ le32_to_cpu(dma_desc->desc2),
+ le32_to_cpu(dma_desc->desc3));
+}
+
+void xlgmac_print_pkt(struct net_device *netdev,
+ struct sk_buff *skb, bool tx_rx)
+{
+ struct ethhdr *eth = (struct ethhdr *)skb->data;
+ unsigned char *buf = skb->data;
+ unsigned char buffer[128];
+ unsigned int i, j;
+
+ netdev_dbg(netdev, "\n************** SKB dump ****************\n");
+
+ netdev_dbg(netdev, "%s packet of %d bytes\n",
+ (tx_rx ? "TX" : "RX"), skb->len);
+
+ netdev_dbg(netdev, "Dst MAC addr: %pM\n", eth->h_dest);
+ netdev_dbg(netdev, "Src MAC addr: %pM\n", eth->h_source);
+ netdev_dbg(netdev, "Protocol: %#06hx\n", ntohs(eth->h_proto));
+
+ for (i = 0, j = 0; i < skb->len;) {
+ j += snprintf(buffer + j, sizeof(buffer) - j, "%02hhx",
+ buf[i++]);
+
+ if ((i % 32) == 0) {
+ netdev_dbg(netdev, " %#06x: %s\n", i - 32, buffer);
+ j = 0;
+ } else if ((i % 16) == 0) {
+ buffer[j++] = ' ';
+ buffer[j++] = ' ';
+ } else if ((i % 4) == 0) {
+ buffer[j++] = ' ';
+ }
+ }
+ if (i % 32)
+ netdev_dbg(netdev, " %#06x: %s\n", i - (i % 32), buffer);
+
+ netdev_dbg(netdev, "\n************** SKB dump ****************\n");
+}
+
+void xlgmac_get_all_hw_features(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_hw_features *hw_feat = &pdata->hw_feat;
+ unsigned int mac_hfr0, mac_hfr1, mac_hfr2;
+
+ mac_hfr0 = readl(pdata->mac_regs + MAC_HWF0R);
+ mac_hfr1 = readl(pdata->mac_regs + MAC_HWF1R);
+ mac_hfr2 = readl(pdata->mac_regs + MAC_HWF2R);
+
+ memset(hw_feat, 0, sizeof(*hw_feat));
+
+ hw_feat->version = readl(pdata->mac_regs + MAC_VR);
+
+ /* Hardware feature register 0 */
+ hw_feat->phyifsel = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_PHYIFSEL_POS,
+ MAC_HWF0R_PHYIFSEL_LEN);
+ hw_feat->vlhash = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_VLHASH_POS,
+ MAC_HWF0R_VLHASH_LEN);
+ hw_feat->sma = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_SMASEL_POS,
+ MAC_HWF0R_SMASEL_LEN);
+ hw_feat->rwk = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_RWKSEL_POS,
+ MAC_HWF0R_RWKSEL_LEN);
+ hw_feat->mgk = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_MGKSEL_POS,
+ MAC_HWF0R_MGKSEL_LEN);
+ hw_feat->mmc = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_MMCSEL_POS,
+ MAC_HWF0R_MMCSEL_LEN);
+ hw_feat->aoe = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_ARPOFFSEL_POS,
+ MAC_HWF0R_ARPOFFSEL_LEN);
+ hw_feat->ts = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_TSSEL_POS,
+ MAC_HWF0R_TSSEL_LEN);
+ hw_feat->eee = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_EEESEL_POS,
+ MAC_HWF0R_EEESEL_LEN);
+ hw_feat->tx_coe = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_TXCOESEL_POS,
+ MAC_HWF0R_TXCOESEL_LEN);
+ hw_feat->rx_coe = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_RXCOESEL_POS,
+ MAC_HWF0R_RXCOESEL_LEN);
+ hw_feat->addn_mac = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_ADDMACADRSEL_POS,
+ MAC_HWF0R_ADDMACADRSEL_LEN);
+ hw_feat->ts_src = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_TSSTSSEL_POS,
+ MAC_HWF0R_TSSTSSEL_LEN);
+ hw_feat->sa_vlan_ins = XLGMAC_GET_REG_BITS(mac_hfr0,
+ MAC_HWF0R_SAVLANINS_POS,
+ MAC_HWF0R_SAVLANINS_LEN);
+
+ /* Hardware feature register 1 */
+ hw_feat->rx_fifo_size = XLGMAC_GET_REG_BITS(mac_hfr1,
+ MAC_HWF1R_RXFIFOSIZE_POS,
+ MAC_HWF1R_RXFIFOSIZE_LEN);
+ hw_feat->tx_fifo_size = XLGMAC_GET_REG_BITS(mac_hfr1,
+ MAC_HWF1R_TXFIFOSIZE_POS,
+ MAC_HWF1R_TXFIFOSIZE_LEN);
+ hw_feat->adv_ts_hi = XLGMAC_GET_REG_BITS(mac_hfr1,
+ MAC_HWF1R_ADVTHWORD_POS,
+ MAC_HWF1R_ADVTHWORD_LEN);
+ hw_feat->dma_width = XLGMAC_GET_REG_BITS(mac_hfr1,
+ MAC_HWF1R_ADDR64_POS,
+ MAC_HWF1R_ADDR64_LEN);
+ hw_feat->dcb = XLGMAC_GET_REG_BITS(mac_hfr1,
+ MAC_HWF1R_DCBEN_POS,
+ MAC_HWF1R_DCBEN_LEN);
+ hw_feat->sph = XLGMAC_GET_REG_BITS(mac_hfr1,
+ MAC_HWF1R_SPHEN_POS,
+ MAC_HWF1R_SPHEN_LEN);
+ hw_feat->tso = XLGMAC_GET_REG_BITS(mac_hfr1,
+ MAC_HWF1R_TSOEN_POS,
+ MAC_HWF1R_TSOEN_LEN);
+ hw_feat->dma_debug = XLGMAC_GET_REG_BITS(mac_hfr1,
+ MAC_HWF1R_DBGMEMA_POS,
+ MAC_HWF1R_DBGMEMA_LEN);
+ hw_feat->rss = XLGMAC_GET_REG_BITS(mac_hfr1,
+ MAC_HWF1R_RSSEN_POS,
+ MAC_HWF1R_RSSEN_LEN);
+ hw_feat->tc_cnt = XLGMAC_GET_REG_BITS(mac_hfr1,
+ MAC_HWF1R_NUMTC_POS,
+ MAC_HWF1R_NUMTC_LEN);
+ hw_feat->hash_table_size = XLGMAC_GET_REG_BITS(mac_hfr1,
+ MAC_HWF1R_HASHTBLSZ_POS,
+ MAC_HWF1R_HASHTBLSZ_LEN);
+ hw_feat->l3l4_filter_num = XLGMAC_GET_REG_BITS(mac_hfr1,
+ MAC_HWF1R_L3L4FNUM_POS,
+ MAC_HWF1R_L3L4FNUM_LEN);
+
+ /* Hardware feature register 2 */
+ hw_feat->rx_q_cnt = XLGMAC_GET_REG_BITS(mac_hfr2,
+ MAC_HWF2R_RXQCNT_POS,
+ MAC_HWF2R_RXQCNT_LEN);
+ hw_feat->tx_q_cnt = XLGMAC_GET_REG_BITS(mac_hfr2,
+ MAC_HWF2R_TXQCNT_POS,
+ MAC_HWF2R_TXQCNT_LEN);
+ hw_feat->rx_ch_cnt = XLGMAC_GET_REG_BITS(mac_hfr2,
+ MAC_HWF2R_RXCHCNT_POS,
+ MAC_HWF2R_RXCHCNT_LEN);
+ hw_feat->tx_ch_cnt = XLGMAC_GET_REG_BITS(mac_hfr2,
+ MAC_HWF2R_TXCHCNT_POS,
+ MAC_HWF2R_TXCHCNT_LEN);
+ hw_feat->pps_out_num = XLGMAC_GET_REG_BITS(mac_hfr2,
+ MAC_HWF2R_PPSOUTNUM_POS,
+ MAC_HWF2R_PPSOUTNUM_LEN);
+ hw_feat->aux_snap_num = XLGMAC_GET_REG_BITS(mac_hfr2,
+ MAC_HWF2R_AUXSNAPNUM_POS,
+ MAC_HWF2R_AUXSNAPNUM_LEN);
+
+ /* Translate the Hash Table size into actual number */
+ switch (hw_feat->hash_table_size) {
+ case 0:
+ break;
+ case 1:
+ hw_feat->hash_table_size = 64;
+ break;
+ case 2:
+ hw_feat->hash_table_size = 128;
+ break;
+ case 3:
+ hw_feat->hash_table_size = 256;
+ break;
+ }
+
+ /* Translate the address width setting into actual number */
+ switch (hw_feat->dma_width) {
+ case 0:
+ hw_feat->dma_width = 32;
+ break;
+ case 1:
+ hw_feat->dma_width = 40;
+ break;
+ case 2:
+ hw_feat->dma_width = 48;
+ break;
+ default:
+ hw_feat->dma_width = 32;
+ }
+
+ /* The Queue, Channel and TC counts are zero based so increment them
+ * to get the actual number
+ */
+ hw_feat->rx_q_cnt++;
+ hw_feat->tx_q_cnt++;
+ hw_feat->rx_ch_cnt++;
+ hw_feat->tx_ch_cnt++;
+ hw_feat->tc_cnt++;
+}
+
+void xlgmac_print_all_hw_features(struct xlgmac_pdata *pdata)
+{
+ char *str = NULL;
+
+ XLGMAC_PR("\n");
+ XLGMAC_PR("=====================================================\n");
+ XLGMAC_PR("\n");
+ XLGMAC_PR("HW support following features\n");
+ XLGMAC_PR("\n");
+ /* HW Feature Register0 */
+ XLGMAC_PR("VLAN Hash Filter Selected : %s\n",
+ pdata->hw_feat.vlhash ? "YES" : "NO");
+ XLGMAC_PR("SMA (MDIO) Interface : %s\n",
+ pdata->hw_feat.sma ? "YES" : "NO");
+ XLGMAC_PR("PMT Remote Wake-up Packet Enable : %s\n",
+ pdata->hw_feat.rwk ? "YES" : "NO");
+ XLGMAC_PR("PMT Magic Packet Enable : %s\n",
+ pdata->hw_feat.mgk ? "YES" : "NO");
+ XLGMAC_PR("RMON/MMC Module Enable : %s\n",
+ pdata->hw_feat.mmc ? "YES" : "NO");
+ XLGMAC_PR("ARP Offload Enabled : %s\n",
+ pdata->hw_feat.aoe ? "YES" : "NO");
+ XLGMAC_PR("IEEE 1588-2008 Timestamp Enabled : %s\n",
+ pdata->hw_feat.ts ? "YES" : "NO");
+ XLGMAC_PR("Energy Efficient Ethernet Enabled : %s\n",
+ pdata->hw_feat.eee ? "YES" : "NO");
+ XLGMAC_PR("Transmit Checksum Offload Enabled : %s\n",
+ pdata->hw_feat.tx_coe ? "YES" : "NO");
+ XLGMAC_PR("Receive Checksum Offload Enabled : %s\n",
+ pdata->hw_feat.rx_coe ? "YES" : "NO");
+ XLGMAC_PR("Additional MAC Addresses 1-31 Selected : %s\n",
+ pdata->hw_feat.addn_mac ? "YES" : "NO");
+
+ switch (pdata->hw_feat.ts_src) {
+ case 0:
+ str = "RESERVED";
+ break;
+ case 1:
+ str = "INTERNAL";
+ break;
+ case 2:
+ str = "EXTERNAL";
+ break;
+ case 3:
+ str = "BOTH";
+ break;
+ }
+ XLGMAC_PR("Timestamp System Time Source : %s\n", str);
+
+ XLGMAC_PR("Source Address or VLAN Insertion Enable : %s\n",
+ pdata->hw_feat.sa_vlan_ins ? "YES" : "NO");
+
+ /* HW Feature Register1 */
+ switch (pdata->hw_feat.rx_fifo_size) {
+ case 0:
+ str = "128 bytes";
+ break;
+ case 1:
+ str = "256 bytes";
+ break;
+ case 2:
+ str = "512 bytes";
+ break;
+ case 3:
+ str = "1 KBytes";
+ break;
+ case 4:
+ str = "2 KBytes";
+ break;
+ case 5:
+ str = "4 KBytes";
+ break;
+ case 6:
+ str = "8 KBytes";
+ break;
+ case 7:
+ str = "16 KBytes";
+ break;
+ case 8:
+ str = "32 kBytes";
+ break;
+ case 9:
+ str = "64 KBytes";
+ break;
+ case 10:
+ str = "128 KBytes";
+ break;
+ case 11:
+ str = "256 KBytes";
+ break;
+ default:
+ str = "RESERVED";
+ }
+ XLGMAC_PR("MTL Receive FIFO Size : %s\n", str);
+
+ switch (pdata->hw_feat.tx_fifo_size) {
+ case 0:
+ str = "128 bytes";
+ break;
+ case 1:
+ str = "256 bytes";
+ break;
+ case 2:
+ str = "512 bytes";
+ break;
+ case 3:
+ str = "1 KBytes";
+ break;
+ case 4:
+ str = "2 KBytes";
+ break;
+ case 5:
+ str = "4 KBytes";
+ break;
+ case 6:
+ str = "8 KBytes";
+ break;
+ case 7:
+ str = "16 KBytes";
+ break;
+ case 8:
+ str = "32 kBytes";
+ break;
+ case 9:
+ str = "64 KBytes";
+ break;
+ case 10:
+ str = "128 KBytes";
+ break;
+ case 11:
+ str = "256 KBytes";
+ break;
+ default:
+ str = "RESERVED";
+ }
+ XLGMAC_PR("MTL Transmit FIFO Size : %s\n", str);
+
+ XLGMAC_PR("IEEE 1588 High Word Register Enable : %s\n",
+ pdata->hw_feat.adv_ts_hi ? "YES" : "NO");
+ XLGMAC_PR("Address width : %u\n",
+ pdata->hw_feat.dma_width);
+ XLGMAC_PR("DCB Feature Enable : %s\n",
+ pdata->hw_feat.dcb ? "YES" : "NO");
+ XLGMAC_PR("Split Header Feature Enable : %s\n",
+ pdata->hw_feat.sph ? "YES" : "NO");
+ XLGMAC_PR("TCP Segmentation Offload Enable : %s\n",
+ pdata->hw_feat.tso ? "YES" : "NO");
+ XLGMAC_PR("DMA Debug Registers Enabled : %s\n",
+ pdata->hw_feat.dma_debug ? "YES" : "NO");
+ XLGMAC_PR("RSS Feature Enabled : %s\n",
+ pdata->hw_feat.rss ? "YES" : "NO");
+ XLGMAC_PR("Number of Traffic classes : %u\n",
+ (pdata->hw_feat.tc_cnt));
+ XLGMAC_PR("Hash Table Size : %u\n",
+ pdata->hw_feat.hash_table_size);
+ XLGMAC_PR("Total number of L3 or L4 Filters : %u\n",
+ pdata->hw_feat.l3l4_filter_num);
+
+ /* HW Feature Register2 */
+ XLGMAC_PR("Number of MTL Receive Queues : %u\n",
+ pdata->hw_feat.rx_q_cnt);
+ XLGMAC_PR("Number of MTL Transmit Queues : %u\n",
+ pdata->hw_feat.tx_q_cnt);
+ XLGMAC_PR("Number of DMA Receive Channels : %u\n",
+ pdata->hw_feat.rx_ch_cnt);
+ XLGMAC_PR("Number of DMA Transmit Channels : %u\n",
+ pdata->hw_feat.tx_ch_cnt);
+
+ switch (pdata->hw_feat.pps_out_num) {
+ case 0:
+ str = "No PPS output";
+ break;
+ case 1:
+ str = "1 PPS output";
+ break;
+ case 2:
+ str = "2 PPS output";
+ break;
+ case 3:
+ str = "3 PPS output";
+ break;
+ case 4:
+ str = "4 PPS output";
+ break;
+ default:
+ str = "RESERVED";
+ }
+ XLGMAC_PR("Number of PPS Outputs : %s\n", str);
+
+ switch (pdata->hw_feat.aux_snap_num) {
+ case 0:
+ str = "No auxiliary input";
+ break;
+ case 1:
+ str = "1 auxiliary input";
+ break;
+ case 2:
+ str = "2 auxiliary input";
+ break;
+ case 3:
+ str = "3 auxiliary input";
+ break;
+ case 4:
+ str = "4 auxiliary input";
+ break;
+ default:
+ str = "RESERVED";
+ }
+ XLGMAC_PR("Number of Auxiliary Snapshot Inputs : %s", str);
+
+ XLGMAC_PR("\n");
+ XLGMAC_PR("=====================================================\n");
+ XLGMAC_PR("\n");
+}
diff --git a/drivers/net/ethernet/synopsys/dwc-xlgmac-desc.c b/drivers/net/ethernet/synopsys/dwc-xlgmac-desc.c
new file mode 100644
index 0000000..e9672b1
--- /dev/null
+++ b/drivers/net/ethernet/synopsys/dwc-xlgmac-desc.c
@@ -0,0 +1,644 @@
+/* Synopsys DesignWare Core Enterprise Ethernet (XLGMAC) Driver
+ *
+ * Copyright (c) 2017 Synopsys, Inc. (www.synopsys.com)
+ *
+ * This program is dual-licensed; you may select either version 2 of
+ * the GNU General Public License ("GPL") or BSD license ("BSD").
+ *
+ * This Synopsys DWC XLGMAC software driver and associated documentation
+ * (hereinafter the "Software") is an unsupported proprietary work of
+ * Synopsys, Inc. unless otherwise expressly agreed to in writing between
+ * Synopsys and you. The Software IS NOT an item of Licensed Software or a
+ * Licensed Product under any End User Software License Agreement or
+ * Agreement for Licensed Products with Synopsys or any supplement thereto.
+ * Synopsys is a registered trademark of Synopsys, Inc. Other names included
+ * in the SOFTWARE may be the trademarks of their respective owners.
+ */
+
+#include "dwc-xlgmac.h"
+#include "dwc-xlgmac-reg.h"
+
+static void xlgmac_unmap_desc_data(struct xlgmac_pdata *pdata,
+ struct xlgmac_desc_data *desc_data)
+{
+ if (desc_data->skb_dma) {
+ if (desc_data->mapped_as_page) {
+ dma_unmap_page(pdata->dev, desc_data->skb_dma,
+ desc_data->skb_dma_len, DMA_TO_DEVICE);
+ } else {
+ dma_unmap_single(pdata->dev, desc_data->skb_dma,
+ desc_data->skb_dma_len, DMA_TO_DEVICE);
+ }
+ desc_data->skb_dma = 0;
+ desc_data->skb_dma_len = 0;
+ }
+
+ if (desc_data->skb) {
+ dev_kfree_skb_any(desc_data->skb);
+ desc_data->skb = NULL;
+ }
+
+ if (desc_data->rx.hdr.pa.pages)
+ put_page(desc_data->rx.hdr.pa.pages);
+
+ if (desc_data->rx.hdr.pa_unmap.pages) {
+ dma_unmap_page(pdata->dev, desc_data->rx.hdr.pa_unmap.pages_dma,
+ desc_data->rx.hdr.pa_unmap.pages_len,
+ DMA_FROM_DEVICE);
+ put_page(desc_data->rx.hdr.pa_unmap.pages);
+ }
+
+ if (desc_data->rx.buf.pa.pages)
+ put_page(desc_data->rx.buf.pa.pages);
+
+ if (desc_data->rx.buf.pa_unmap.pages) {
+ dma_unmap_page(pdata->dev, desc_data->rx.buf.pa_unmap.pages_dma,
+ desc_data->rx.buf.pa_unmap.pages_len,
+ DMA_FROM_DEVICE);
+ put_page(desc_data->rx.buf.pa_unmap.pages);
+ }
+
+ memset(&desc_data->tx, 0, sizeof(desc_data->tx));
+ memset(&desc_data->rx, 0, sizeof(desc_data->rx));
+
+ desc_data->mapped_as_page = 0;
+
+ if (desc_data->state_saved) {
+ desc_data->state_saved = 0;
+ desc_data->state.skb = NULL;
+ desc_data->state.len = 0;
+ desc_data->state.error = 0;
+ }
+}
+
+static void xlgmac_free_ring(struct xlgmac_pdata *pdata,
+ struct xlgmac_ring *ring)
+{
+ struct xlgmac_desc_data *desc_data;
+ unsigned int i;
+
+ if (!ring)
+ return;
+
+ if (ring->desc_data_head) {
+ for (i = 0; i < ring->dma_desc_count; i++) {
+ desc_data = XLGMAC_GET_DESC_DATA(ring, i);
+ xlgmac_unmap_desc_data(pdata, desc_data);
+ }
+
+ kfree(ring->desc_data_head);
+ ring->desc_data_head = NULL;
+ }
+
+ if (ring->rx_hdr_pa.pages) {
+ dma_unmap_page(pdata->dev, ring->rx_hdr_pa.pages_dma,
+ ring->rx_hdr_pa.pages_len, DMA_FROM_DEVICE);
+ put_page(ring->rx_hdr_pa.pages);
+
+ ring->rx_hdr_pa.pages = NULL;
+ ring->rx_hdr_pa.pages_len = 0;
+ ring->rx_hdr_pa.pages_offset = 0;
+ ring->rx_hdr_pa.pages_dma = 0;
+ }
+
+ if (ring->rx_buf_pa.pages) {
+ dma_unmap_page(pdata->dev, ring->rx_buf_pa.pages_dma,
+ ring->rx_buf_pa.pages_len, DMA_FROM_DEVICE);
+ put_page(ring->rx_buf_pa.pages);
+
+ ring->rx_buf_pa.pages = NULL;
+ ring->rx_buf_pa.pages_len = 0;
+ ring->rx_buf_pa.pages_offset = 0;
+ ring->rx_buf_pa.pages_dma = 0;
+ }
+
+ if (ring->dma_desc_head) {
+ dma_free_coherent(pdata->dev,
+ (sizeof(struct xlgmac_dma_desc) *
+ ring->dma_desc_count),
+ ring->dma_desc_head,
+ ring->dma_desc_head_addr);
+ ring->dma_desc_head = NULL;
+ }
+}
+
+static int xlgmac_init_ring(struct xlgmac_pdata *pdata,
+ struct xlgmac_ring *ring,
+ unsigned int dma_desc_count)
+{
+ if (!ring)
+ return 0;
+
+ /* Descriptors */
+ ring->dma_desc_count = dma_desc_count;
+ ring->dma_desc_head = dma_alloc_coherent(pdata->dev,
+ (sizeof(struct xlgmac_dma_desc) *
+ dma_desc_count),
+ &ring->dma_desc_head_addr,
+ GFP_KERNEL);
+ if (!ring->dma_desc_head)
+ return -ENOMEM;
+
+ /* Array of descriptor data */
+ ring->desc_data_head = kcalloc(dma_desc_count,
+ sizeof(struct xlgmac_desc_data),
+ GFP_KERNEL);
+ if (!ring->desc_data_head)
+ return -ENOMEM;
+
+ netif_dbg(pdata, drv, pdata->netdev,
+ "dma_desc_head=%p, dma_desc_head_addr=%pad, desc_data_head=%p\n",
+ ring->dma_desc_head,
+ &ring->dma_desc_head_addr,
+ ring->desc_data_head);
+
+ return 0;
+}
+
+static void xlgmac_free_rings(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+
+ if (!pdata->channel_head)
+ return;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ xlgmac_free_ring(pdata, channel->tx_ring);
+ xlgmac_free_ring(pdata, channel->rx_ring);
+ }
+}
+
+static int xlgmac_alloc_rings(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+ int ret;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ netif_dbg(pdata, drv, pdata->netdev, "%s - Tx ring:\n",
+ channel->name);
+
+ ret = xlgmac_init_ring(pdata, channel->tx_ring,
+ pdata->tx_desc_count);
+
+ if (ret) {
+ netdev_alert(pdata->netdev,
+ "error initializing Tx ring");
+ goto err_init_ring;
+ }
+
+ netif_dbg(pdata, drv, pdata->netdev, "%s - Rx ring:\n",
+ channel->name);
+
+ ret = xlgmac_init_ring(pdata, channel->rx_ring,
+ pdata->rx_desc_count);
+ if (ret) {
+ netdev_alert(pdata->netdev,
+ "error initializing Rx ring\n");
+ goto err_init_ring;
+ }
+ }
+
+ return 0;
+
+err_init_ring:
+ xlgmac_free_rings(pdata);
+
+ return ret;
+}
+
+static void xlgmac_free_channels(struct xlgmac_pdata *pdata)
+{
+ if (!pdata->channel_head)
+ return;
+
+ kfree(pdata->channel_head->tx_ring);
+ pdata->channel_head->tx_ring = NULL;
+
+ kfree(pdata->channel_head->rx_ring);
+ pdata->channel_head->rx_ring = NULL;
+
+ kfree(pdata->channel_head);
+
+ pdata->channel_head = NULL;
+ pdata->channel_count = 0;
+}
+
+static int xlgmac_alloc_channels(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel_head, *channel;
+ struct xlgmac_ring *tx_ring, *rx_ring;
+ int ret = -ENOMEM;
+ unsigned int i;
+
+ channel_head = kcalloc(pdata->channel_count,
+ sizeof(struct xlgmac_channel), GFP_KERNEL);
+ if (!channel_head)
+ return ret;
+
+ netif_dbg(pdata, drv, pdata->netdev,
+ "channel_head=%p\n", channel_head);
+
+ tx_ring = kcalloc(pdata->tx_ring_count, sizeof(struct xlgmac_ring),
+ GFP_KERNEL);
+ if (!tx_ring)
+ goto err_tx_ring;
+
+ rx_ring = kcalloc(pdata->rx_ring_count, sizeof(struct xlgmac_ring),
+ GFP_KERNEL);
+ if (!rx_ring)
+ goto err_rx_ring;
+
+ for (i = 0, channel = channel_head; i < pdata->channel_count;
+ i++, channel++) {
+ snprintf(channel->name, sizeof(channel->name), "channel-%u", i);
+ channel->pdata = pdata;
+ channel->queue_index = i;
+ channel->dma_regs = pdata->mac_regs + DMA_CH_BASE +
+ (DMA_CH_INC * i);
+
+ if (pdata->per_channel_irq) {
+ /* Get the per DMA interrupt */
+ ret = pdata->channel_irq[i];
+ if (ret < 0) {
+ netdev_err(pdata->netdev,
+ "get_irq %u failed\n",
+ i + 1);
+ goto err_irq;
+ }
+ channel->dma_irq = ret;
+ }
+
+ if (i < pdata->tx_ring_count)
+ channel->tx_ring = tx_ring++;
+
+ if (i < pdata->rx_ring_count)
+ channel->rx_ring = rx_ring++;
+
+ netif_dbg(pdata, drv, pdata->netdev,
+ "%s: dma_regs=%p, tx_ring=%p, rx_ring=%p\n",
+ channel->name, channel->dma_regs,
+ channel->tx_ring, channel->rx_ring);
+ }
+
+ pdata->channel_head = channel_head;
+
+ return 0;
+
+err_irq:
+ kfree(rx_ring);
+
+err_rx_ring:
+ kfree(tx_ring);
+
+err_tx_ring:
+ kfree(channel_head);
+
+ return ret;
+}
+
+static void xlgmac_free_channels_and_rings(struct xlgmac_pdata *pdata)
+{
+ xlgmac_free_rings(pdata);
+
+ xlgmac_free_channels(pdata);
+}
+
+static int xlgmac_alloc_channels_and_rings(struct xlgmac_pdata *pdata)
+{
+ int ret;
+
+ ret = xlgmac_alloc_channels(pdata);
+ if (ret)
+ goto err_alloc;
+
+ ret = xlgmac_alloc_rings(pdata);
+ if (ret)
+ goto err_alloc;
+
+ return 0;
+
+err_alloc:
+ xlgmac_free_channels_and_rings(pdata);
+
+ return ret;
+}
+
+static int xlgmac_alloc_pages(struct xlgmac_pdata *pdata,
+ struct xlgmac_page_alloc *pa,
+ gfp_t gfp, int order)
+{
+ struct page *pages = NULL;
+ dma_addr_t pages_dma;
+
+ /* Try to obtain pages, decreasing order if necessary */
+ gfp |= __GFP_COLD | __GFP_COMP | __GFP_NOWARN;
+ while (order >= 0) {
+ pages = alloc_pages(gfp, order);
+ if (pages)
+ break;
+
+ order--;
+ }
+ if (!pages)
+ return -ENOMEM;
+
+ /* Map the pages */
+ pages_dma = dma_map_page(pdata->dev, pages, 0,
+ PAGE_SIZE << order, DMA_FROM_DEVICE);
+ if (dma_mapping_error(pdata->dev, pages_dma)) {
+ put_page(pages);
+ return -ENOMEM;
+ }
+
+ pa->pages = pages;
+ pa->pages_len = PAGE_SIZE << order;
+ pa->pages_offset = 0;
+ pa->pages_dma = pages_dma;
+
+ return 0;
+}
+
+static void xlgmac_set_buffer_data(struct xlgmac_buffer_data *bd,
+ struct xlgmac_page_alloc *pa,
+ unsigned int len)
+{
+ get_page(pa->pages);
+ bd->pa = *pa;
+
+ bd->dma_base = pa->pages_dma;
+ bd->dma_off = pa->pages_offset;
+ bd->dma_len = len;
+
+ pa->pages_offset += len;
+ if ((pa->pages_offset + len) > pa->pages_len) {
+ /* This data descriptor is responsible for unmapping page(s) */
+ bd->pa_unmap = *pa;
+
+ /* Get a new allocation next time */
+ pa->pages = NULL;
+ pa->pages_len = 0;
+ pa->pages_offset = 0;
+ pa->pages_dma = 0;
+ }
+}
+
+static int xlgmac_map_rx_buffer(struct xlgmac_pdata *pdata,
+ struct xlgmac_ring *ring,
+ struct xlgmac_desc_data *desc_data)
+{
+ int order, ret;
+
+ if (!ring->rx_hdr_pa.pages) {
+ ret = xlgmac_alloc_pages(pdata, &ring->rx_hdr_pa,
+ GFP_ATOMIC, 0);
+ if (ret)
+ return ret;
+ }
+
+ if (!ring->rx_buf_pa.pages) {
+ order = max_t(int, PAGE_ALLOC_COSTLY_ORDER - 1, 0);
+ ret = xlgmac_alloc_pages(pdata, &ring->rx_buf_pa,
+ GFP_ATOMIC, order);
+ if (ret)
+ return ret;
+ }
+
+ /* Set up the header page info */
+ xlgmac_set_buffer_data(&desc_data->rx.hdr, &ring->rx_hdr_pa,
+ XLGMAC_SKB_ALLOC_SIZE);
+
+ /* Set up the buffer page info */
+ xlgmac_set_buffer_data(&desc_data->rx.buf, &ring->rx_buf_pa,
+ pdata->rx_buf_size);
+
+ return 0;
+}
+
+static void xlgmac_tx_desc_init(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_hw_ops *hw_ops = &pdata->hw_ops;
+ struct xlgmac_desc_data *desc_data;
+ struct xlgmac_dma_desc *dma_desc;
+ struct xlgmac_channel *channel;
+ struct xlgmac_ring *ring;
+ dma_addr_t dma_desc_addr;
+ unsigned int i, j;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ ring = channel->tx_ring;
+ if (!ring)
+ break;
+
+ dma_desc = ring->dma_desc_head;
+ dma_desc_addr = ring->dma_desc_head_addr;
+
+ for (j = 0; j < ring->dma_desc_count; j++) {
+ desc_data = XLGMAC_GET_DESC_DATA(ring, j);
+
+ desc_data->dma_desc = dma_desc;
+ desc_data->dma_desc_addr = dma_desc_addr;
+
+ dma_desc++;
+ dma_desc_addr += sizeof(struct xlgmac_dma_desc);
+ }
+
+ ring->cur = 0;
+ ring->dirty = 0;
+ memset(&ring->tx, 0, sizeof(ring->tx));
+
+ hw_ops->tx_desc_init(channel);
+ }
+}
+
+static void xlgmac_rx_desc_init(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_hw_ops *hw_ops = &pdata->hw_ops;
+ struct xlgmac_desc_data *desc_data;
+ struct xlgmac_dma_desc *dma_desc;
+ struct xlgmac_channel *channel;
+ struct xlgmac_ring *ring;
+ dma_addr_t dma_desc_addr;
+ unsigned int i, j;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ ring = channel->rx_ring;
+ if (!ring)
+ break;
+
+ dma_desc = ring->dma_desc_head;
+ dma_desc_addr = ring->dma_desc_head_addr;
+
+ for (j = 0; j < ring->dma_desc_count; j++) {
+ desc_data = XLGMAC_GET_DESC_DATA(ring, j);
+
+ desc_data->dma_desc = dma_desc;
+ desc_data->dma_desc_addr = dma_desc_addr;
+
+ if (xlgmac_map_rx_buffer(pdata, ring, desc_data))
+ break;
+
+ dma_desc++;
+ dma_desc_addr += sizeof(struct xlgmac_dma_desc);
+ }
+
+ ring->cur = 0;
+ ring->dirty = 0;
+
+ hw_ops->rx_desc_init(channel);
+ }
+}
+
+static int xlgmac_map_tx_skb(struct xlgmac_channel *channel,
+ struct sk_buff *skb)
+{
+ struct xlgmac_pdata *pdata = channel->pdata;
+ struct xlgmac_ring *ring = channel->tx_ring;
+ unsigned int start_index, cur_index;
+ struct xlgmac_desc_data *desc_data;
+ unsigned int offset, datalen, len;
+ struct xlgmac_pkt_info *pkt_info;
+ struct skb_frag_struct *frag;
+ unsigned int tso, vlan;
+ dma_addr_t skb_dma;
+ unsigned int i;
+
+ offset = 0;
+ start_index = ring->cur;
+ cur_index = ring->cur;
+
+ pkt_info = &ring->pkt_info;
+ pkt_info->desc_count = 0;
+ pkt_info->length = 0;
+
+ tso = XLGMAC_GET_REG_BITS(pkt_info->attributes,
+ TX_PACKET_ATTRIBUTES_TSO_ENABLE_POS,
+ TX_PACKET_ATTRIBUTES_TSO_ENABLE_LEN);
+ vlan = XLGMAC_GET_REG_BITS(pkt_info->attributes,
+ TX_PACKET_ATTRIBUTES_VLAN_CTAG_POS,
+ TX_PACKET_ATTRIBUTES_VLAN_CTAG_LEN);
+
+ /* Save space for a context descriptor if needed */
+ if ((tso && (pkt_info->mss != ring->tx.cur_mss)) ||
+ (vlan && (pkt_info->vlan_ctag != ring->tx.cur_vlan_ctag)))
+ cur_index++;
+ desc_data = XLGMAC_GET_DESC_DATA(ring, cur_index);
+
+ if (tso) {
+ /* Map the TSO header */
+ skb_dma = dma_map_single(pdata->dev, skb->data,
+ pkt_info->header_len, DMA_TO_DEVICE);
+ if (dma_mapping_error(pdata->dev, skb_dma)) {
+ netdev_alert(pdata->netdev, "dma_map_single failed\n");
+ goto err_out;
+ }
+ desc_data->skb_dma = skb_dma;
+ desc_data->skb_dma_len = pkt_info->header_len;
+ netif_dbg(pdata, tx_queued, pdata->netdev,
+ "skb header: index=%u, dma=%pad, len=%u\n",
+ cur_index, &skb_dma, pkt_info->header_len);
+
+ offset = pkt_info->header_len;
+
+ pkt_info->length += pkt_info->header_len;
+
+ cur_index++;
+ desc_data = XLGMAC_GET_DESC_DATA(ring, cur_index);
+ }
+
+ /* Map the (remainder of the) packet */
+ for (datalen = skb_headlen(skb) - offset; datalen; ) {
+ len = min_t(unsigned int, datalen, XLGMAC_TX_MAX_BUF_SIZE);
+
+ skb_dma = dma_map_single(pdata->dev, skb->data + offset, len,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(pdata->dev, skb_dma)) {
+ netdev_alert(pdata->netdev, "dma_map_single failed\n");
+ goto err_out;
+ }
+ desc_data->skb_dma = skb_dma;
+ desc_data->skb_dma_len = len;
+ netif_dbg(pdata, tx_queued, pdata->netdev,
+ "skb data: index=%u, dma=%pad, len=%u\n",
+ cur_index, &skb_dma, len);
+
+ datalen -= len;
+ offset += len;
+
+ pkt_info->length += len;
+
+ cur_index++;
+ desc_data = XLGMAC_GET_DESC_DATA(ring, cur_index);
+ }
+
+ for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
+ netif_dbg(pdata, tx_queued, pdata->netdev,
+ "mapping frag %u\n", i);
+
+ frag = &skb_shinfo(skb)->frags[i];
+ offset = 0;
+
+ for (datalen = skb_frag_size(frag); datalen; ) {
+ len = min_t(unsigned int, datalen,
+ XLGMAC_TX_MAX_BUF_SIZE);
+
+ skb_dma = skb_frag_dma_map(pdata->dev, frag, offset,
+ len, DMA_TO_DEVICE);
+ if (dma_mapping_error(pdata->dev, skb_dma)) {
+ netdev_alert(pdata->netdev,
+ "skb_frag_dma_map failed\n");
+ goto err_out;
+ }
+ desc_data->skb_dma = skb_dma;
+ desc_data->skb_dma_len = len;
+ desc_data->mapped_as_page = 1;
+ netif_dbg(pdata, tx_queued, pdata->netdev,
+ "skb frag: index=%u, dma=%pad, len=%u\n",
+ cur_index, &skb_dma, len);
+
+ datalen -= len;
+ offset += len;
+
+ pkt_info->length += len;
+
+ cur_index++;
+ desc_data = XLGMAC_GET_DESC_DATA(ring, cur_index);
+ }
+ }
+
+ /* Save the skb address in the last entry. We always have some data
+ * that has been mapped so desc_data is always advanced past the last
+ * piece of mapped data - use the entry pointed to by cur_index - 1.
+ */
+ desc_data = XLGMAC_GET_DESC_DATA(ring, cur_index - 1);
+ desc_data->skb = skb;
+
+ /* Save the number of descriptor entries used */
+ pkt_info->desc_count = cur_index - start_index;
+
+ return pkt_info->desc_count;
+
+err_out:
+ while (start_index < cur_index) {
+ desc_data = XLGMAC_GET_DESC_DATA(ring, start_index++);
+ xlgmac_unmap_desc_data(pdata, desc_data);
+ }
+
+ return 0;
+}
+
+void xlgmac_init_desc_ops(struct xlgmac_desc_ops *desc_ops)
+{
+ desc_ops->alloc_channles_and_rings = xlgmac_alloc_channels_and_rings;
+ desc_ops->free_channels_and_rings = xlgmac_free_channels_and_rings;
+ desc_ops->map_tx_skb = xlgmac_map_tx_skb;
+ desc_ops->map_rx_buffer = xlgmac_map_rx_buffer;
+ desc_ops->unmap_desc_data = xlgmac_unmap_desc_data;
+ desc_ops->tx_desc_init = xlgmac_tx_desc_init;
+ desc_ops->rx_desc_init = xlgmac_rx_desc_init;
+}
diff --git a/drivers/net/ethernet/synopsys/dwc-xlgmac-ethtool.c b/drivers/net/ethernet/synopsys/dwc-xlgmac-ethtool.c
new file mode 100644
index 0000000..fde7221
--- /dev/null
+++ b/drivers/net/ethernet/synopsys/dwc-xlgmac-ethtool.c
@@ -0,0 +1,275 @@
+/* Synopsys DesignWare Core Enterprise Ethernet (XLGMAC) Driver
+ *
+ * Copyright (c) 2017 Synopsys, Inc. (www.synopsys.com)
+ *
+ * This program is dual-licensed; you may select either version 2 of
+ * the GNU General Public License ("GPL") or BSD license ("BSD").
+ *
+ * This Synopsys DWC XLGMAC software driver and associated documentation
+ * (hereinafter the "Software") is an unsupported proprietary work of
+ * Synopsys, Inc. unless otherwise expressly agreed to in writing between
+ * Synopsys and you. The Software IS NOT an item of Licensed Software or a
+ * Licensed Product under any End User Software License Agreement or
+ * Agreement for Licensed Products with Synopsys or any supplement thereto.
+ * Synopsys is a registered trademark of Synopsys, Inc. Other names included
+ * in the SOFTWARE may be the trademarks of their respective owners.
+ */
+
+#include <linux/ethtool.h>
+#include <linux/kernel.h>
+#include <linux/netdevice.h>
+
+#include "dwc-xlgmac.h"
+#include "dwc-xlgmac-reg.h"
+
+struct xlgmac_stats_desc {
+ char stat_string[ETH_GSTRING_LEN];
+ int stat_offset;
+};
+
+#define XLGMAC_STAT(str, var) \
+ { \
+ str, \
+ offsetof(struct xlgmac_pdata, stats.var), \
+ }
+
+static const struct xlgmac_stats_desc xlgmac_gstring_stats[] = {
+ /* MMC TX counters */
+ XLGMAC_STAT("tx_bytes", txoctetcount_gb),
+ XLGMAC_STAT("tx_bytes_good", txoctetcount_g),
+ XLGMAC_STAT("tx_packets", txframecount_gb),
+ XLGMAC_STAT("tx_packets_good", txframecount_g),
+ XLGMAC_STAT("tx_unicast_packets", txunicastframes_gb),
+ XLGMAC_STAT("tx_broadcast_packets", txbroadcastframes_gb),
+ XLGMAC_STAT("tx_broadcast_packets_good", txbroadcastframes_g),
+ XLGMAC_STAT("tx_multicast_packets", txmulticastframes_gb),
+ XLGMAC_STAT("tx_multicast_packets_good", txmulticastframes_g),
+ XLGMAC_STAT("tx_vlan_packets_good", txvlanframes_g),
+ XLGMAC_STAT("tx_64_byte_packets", tx64octets_gb),
+ XLGMAC_STAT("tx_65_to_127_byte_packets", tx65to127octets_gb),
+ XLGMAC_STAT("tx_128_to_255_byte_packets", tx128to255octets_gb),
+ XLGMAC_STAT("tx_256_to_511_byte_packets", tx256to511octets_gb),
+ XLGMAC_STAT("tx_512_to_1023_byte_packets", tx512to1023octets_gb),
+ XLGMAC_STAT("tx_1024_to_max_byte_packets", tx1024tomaxoctets_gb),
+ XLGMAC_STAT("tx_underflow_errors", txunderflowerror),
+ XLGMAC_STAT("tx_pause_frames", txpauseframes),
+
+ /* MMC RX counters */
+ XLGMAC_STAT("rx_bytes", rxoctetcount_gb),
+ XLGMAC_STAT("rx_bytes_good", rxoctetcount_g),
+ XLGMAC_STAT("rx_packets", rxframecount_gb),
+ XLGMAC_STAT("rx_unicast_packets_good", rxunicastframes_g),
+ XLGMAC_STAT("rx_broadcast_packets_good", rxbroadcastframes_g),
+ XLGMAC_STAT("rx_multicast_packets_good", rxmulticastframes_g),
+ XLGMAC_STAT("rx_vlan_packets", rxvlanframes_gb),
+ XLGMAC_STAT("rx_64_byte_packets", rx64octets_gb),
+ XLGMAC_STAT("rx_65_to_127_byte_packets", rx65to127octets_gb),
+ XLGMAC_STAT("rx_128_to_255_byte_packets", rx128to255octets_gb),
+ XLGMAC_STAT("rx_256_to_511_byte_packets", rx256to511octets_gb),
+ XLGMAC_STAT("rx_512_to_1023_byte_packets", rx512to1023octets_gb),
+ XLGMAC_STAT("rx_1024_to_max_byte_packets", rx1024tomaxoctets_gb),
+ XLGMAC_STAT("rx_undersize_packets_good", rxundersize_g),
+ XLGMAC_STAT("rx_oversize_packets_good", rxoversize_g),
+ XLGMAC_STAT("rx_crc_errors", rxcrcerror),
+ XLGMAC_STAT("rx_crc_errors_small_packets", rxrunterror),
+ XLGMAC_STAT("rx_crc_errors_giant_packets", rxjabbererror),
+ XLGMAC_STAT("rx_length_errors", rxlengtherror),
+ XLGMAC_STAT("rx_out_of_range_errors", rxoutofrangetype),
+ XLGMAC_STAT("rx_fifo_overflow_errors", rxfifooverflow),
+ XLGMAC_STAT("rx_watchdog_errors", rxwatchdogerror),
+ XLGMAC_STAT("rx_pause_frames", rxpauseframes),
+
+ /* Extra counters */
+ XLGMAC_STAT("tx_tso_packets", tx_tso_packets),
+ XLGMAC_STAT("rx_split_header_packets", rx_split_header_packets),
+ XLGMAC_STAT("tx_process_stopped", tx_process_stopped),
+ XLGMAC_STAT("rx_process_stopped", rx_process_stopped),
+ XLGMAC_STAT("tx_buffer_unavailable", tx_buffer_unavailable),
+ XLGMAC_STAT("rx_buffer_unavailable", rx_buffer_unavailable),
+ XLGMAC_STAT("fatal_bus_error", fatal_bus_error),
+ XLGMAC_STAT("tx_vlan_packets", tx_vlan_packets),
+ XLGMAC_STAT("rx_vlan_packets", rx_vlan_packets),
+ XLGMAC_STAT("napi_poll_isr", napi_poll_isr),
+ XLGMAC_STAT("napi_poll_txtimer", napi_poll_txtimer),
+};
+
+#define XLGMAC_STATS_COUNT ARRAY_SIZE(xlgmac_gstring_stats)
+
+static void xlgmac_ethtool_get_drvinfo(struct net_device *netdev,
+ struct ethtool_drvinfo *drvinfo)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ u32 ver = pdata->hw_feat.version;
+ u32 snpsver, devid, userver;
+
+ strlcpy(drvinfo->driver, pdata->drv_name, sizeof(drvinfo->driver));
+ strlcpy(drvinfo->version, pdata->drv_ver, sizeof(drvinfo->version));
+ strlcpy(drvinfo->bus_info, dev_name(pdata->dev),
+ sizeof(drvinfo->bus_info));
+ /* S|SNPSVER: Synopsys-defined Version
+ * D|DEVID: Indicates the Device family
+ * U|USERVER: User-defined Version
+ */
+ snpsver = XLGMAC_GET_REG_BITS(ver, MAC_VR_SNPSVER_POS,
+ MAC_VR_SNPSVER_LEN);
+ devid = XLGMAC_GET_REG_BITS(ver, MAC_VR_DEVID_POS,
+ MAC_VR_DEVID_LEN);
+ userver = XLGMAC_GET_REG_BITS(ver, MAC_VR_USERVER_POS,
+ MAC_VR_USERVER_LEN);
+ snprintf(drvinfo->fw_version, sizeof(drvinfo->fw_version),
+ "S.D.U: %x.%x.%x", snpsver, devid, userver);
+}
+
+static u32 xlgmac_ethtool_get_msglevel(struct net_device *netdev)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+
+ return pdata->msg_enable;
+}
+
+static void xlgmac_ethtool_set_msglevel(struct net_device *netdev,
+ u32 msglevel)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+
+ pdata->msg_enable = msglevel;
+}
+
+static void xlgmac_ethtool_get_channels(struct net_device *netdev,
+ struct ethtool_channels *channel)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+
+ channel->max_rx = XLGMAC_MAX_DMA_CHANNELS;
+ channel->max_tx = XLGMAC_MAX_DMA_CHANNELS;
+ channel->rx_count = pdata->rx_q_count;
+ channel->tx_count = pdata->tx_q_count;
+}
+
+static int xlgmac_ethtool_get_coalesce(struct net_device *netdev,
+ struct ethtool_coalesce *ec)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+
+ memset(ec, 0, sizeof(struct ethtool_coalesce));
+ ec->rx_coalesce_usecs = pdata->rx_usecs;
+ ec->rx_max_coalesced_frames = pdata->rx_frames;
+ ec->tx_max_coalesced_frames = pdata->tx_frames;
+
+ return 0;
+}
+
+static int xlgmac_ethtool_set_coalesce(struct net_device *netdev,
+ struct ethtool_coalesce *ec)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ struct xlgmac_hw_ops *hw_ops = &pdata->hw_ops;
+ unsigned int rx_frames, rx_riwt, rx_usecs;
+ unsigned int tx_frames;
+
+ /* Check for not supported parameters */
+ if ((ec->rx_coalesce_usecs_irq) || (ec->rx_max_coalesced_frames_irq) ||
+ (ec->tx_coalesce_usecs) || (ec->tx_coalesce_usecs_high) ||
+ (ec->tx_max_coalesced_frames_irq) || (ec->tx_coalesce_usecs_irq) ||
+ (ec->stats_block_coalesce_usecs) || (ec->pkt_rate_low) ||
+ (ec->use_adaptive_rx_coalesce) || (ec->use_adaptive_tx_coalesce) ||
+ (ec->rx_max_coalesced_frames_low) || (ec->rx_coalesce_usecs_low) ||
+ (ec->tx_coalesce_usecs_low) || (ec->tx_max_coalesced_frames_low) ||
+ (ec->pkt_rate_high) || (ec->rx_coalesce_usecs_high) ||
+ (ec->rx_max_coalesced_frames_high) ||
+ (ec->tx_max_coalesced_frames_high) ||
+ (ec->rate_sample_interval))
+ return -EOPNOTSUPP;
+
+ rx_usecs = ec->rx_coalesce_usecs;
+ rx_riwt = hw_ops->usec_to_riwt(pdata, rx_usecs);
+ rx_frames = ec->rx_max_coalesced_frames;
+ tx_frames = ec->tx_max_coalesced_frames;
+
+ if ((rx_riwt > XLGMAC_MAX_DMA_RIWT) ||
+ (rx_riwt < XLGMAC_MIN_DMA_RIWT) ||
+ (rx_frames > pdata->rx_desc_count))
+ return -EINVAL;
+
+ if (tx_frames > pdata->tx_desc_count)
+ return -EINVAL;
+
+ pdata->rx_riwt = rx_riwt;
+ pdata->rx_usecs = rx_usecs;
+ pdata->rx_frames = rx_frames;
+ hw_ops->config_rx_coalesce(pdata);
+
+ pdata->tx_frames = tx_frames;
+ hw_ops->config_tx_coalesce(pdata);
+
+ return 0;
+}
+
+static void xlgmac_ethtool_get_strings(struct net_device *netdev,
+ u32 stringset, u8 *data)
+{
+ int i;
+
+ switch (stringset) {
+ case ETH_SS_STATS:
+ for (i = 0; i < XLGMAC_STATS_COUNT; i++) {
+ memcpy(data, xlgmac_gstring_stats[i].stat_string,
+ ETH_GSTRING_LEN);
+ data += ETH_GSTRING_LEN;
+ }
+ break;
+ default:
+ WARN_ON(1);
+ break;
+ }
+}
+
+static int xlgmac_ethtool_get_sset_count(struct net_device *netdev,
+ int stringset)
+{
+ int ret;
+
+ switch (stringset) {
+ case ETH_SS_STATS:
+ ret = XLGMAC_STATS_COUNT;
+ break;
+
+ default:
+ ret = -EOPNOTSUPP;
+ }
+
+ return ret;
+}
+
+static void xlgmac_ethtool_get_ethtool_stats(struct net_device *netdev,
+ struct ethtool_stats *stats,
+ u64 *data)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ u8 *stat;
+ int i;
+
+ pdata->hw_ops.read_mmc_stats(pdata);
+ for (i = 0; i < XLGMAC_STATS_COUNT; i++) {
+ stat = (u8 *)pdata + xlgmac_gstring_stats[i].stat_offset;
+ *data++ = *(u64 *)stat;
+ }
+}
+
+static const struct ethtool_ops xlgmac_ethtool_ops = {
+ .get_drvinfo = xlgmac_ethtool_get_drvinfo,
+ .get_link = ethtool_op_get_link,
+ .get_msglevel = xlgmac_ethtool_get_msglevel,
+ .set_msglevel = xlgmac_ethtool_set_msglevel,
+ .get_channels = xlgmac_ethtool_get_channels,
+ .get_coalesce = xlgmac_ethtool_get_coalesce,
+ .set_coalesce = xlgmac_ethtool_set_coalesce,
+ .get_strings = xlgmac_ethtool_get_strings,
+ .get_sset_count = xlgmac_ethtool_get_sset_count,
+ .get_ethtool_stats = xlgmac_ethtool_get_ethtool_stats,
+};
+
+const struct ethtool_ops *xlgmac_get_ethtool_ops(void)
+{
+ return &xlgmac_ethtool_ops;
+}
diff --git a/drivers/net/ethernet/synopsys/dwc-xlgmac-hw.c b/drivers/net/ethernet/synopsys/dwc-xlgmac-hw.c
new file mode 100644
index 0000000..458a784
--- /dev/null
+++ b/drivers/net/ethernet/synopsys/dwc-xlgmac-hw.c
@@ -0,0 +1,3147 @@
+/* Synopsys DesignWare Core Enterprise Ethernet (XLGMAC) Driver
+ *
+ * Copyright (c) 2017 Synopsys, Inc. (www.synopsys.com)
+ *
+ * This program is dual-licensed; you may select either version 2 of
+ * the GNU General Public License ("GPL") or BSD license ("BSD").
+ *
+ * This Synopsys DWC XLGMAC software driver and associated documentation
+ * (hereinafter the "Software") is an unsupported proprietary work of
+ * Synopsys, Inc. unless otherwise expressly agreed to in writing between
+ * Synopsys and you. The Software IS NOT an item of Licensed Software or a
+ * Licensed Product under any End User Software License Agreement or
+ * Agreement for Licensed Products with Synopsys or any supplement thereto.
+ * Synopsys is a registered trademark of Synopsys, Inc. Other names included
+ * in the SOFTWARE may be the trademarks of their respective owners.
+ */
+
+#include <linux/phy.h>
+#include <linux/mdio.h>
+#include <linux/clk.h>
+#include <linux/bitrev.h>
+#include <linux/crc32.h>
+#include <linux/dcbnl.h>
+
+#include "dwc-xlgmac.h"
+#include "dwc-xlgmac-reg.h"
+
+static int xlgmac_tx_complete(struct xlgmac_dma_desc *dma_desc)
+{
+ return !XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ TX_NORMAL_DESC3_OWN_POS,
+ TX_NORMAL_DESC3_OWN_LEN);
+}
+
+static int xlgmac_disable_rx_csum(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = readl(pdata->mac_regs + MAC_RCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RCR_IPC_POS,
+ MAC_RCR_IPC_LEN, 0);
+ writel(regval, pdata->mac_regs + MAC_RCR);
+
+ return 0;
+}
+
+static int xlgmac_enable_rx_csum(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = readl(pdata->mac_regs + MAC_RCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RCR_IPC_POS,
+ MAC_RCR_IPC_LEN, 1);
+ writel(regval, pdata->mac_regs + MAC_RCR);
+
+ return 0;
+}
+
+static int xlgmac_set_mac_address(struct xlgmac_pdata *pdata, u8 *addr)
+{
+ unsigned int mac_addr_hi, mac_addr_lo;
+
+ mac_addr_hi = (addr[5] << 8) | (addr[4] << 0);
+ mac_addr_lo = (addr[3] << 24) | (addr[2] << 16) |
+ (addr[1] << 8) | (addr[0] << 0);
+
+ writel(mac_addr_hi, pdata->mac_regs + MAC_MACA0HR);
+ writel(mac_addr_lo, pdata->mac_regs + MAC_MACA0LR);
+
+ return 0;
+}
+
+static void xlgmac_set_mac_reg(struct xlgmac_pdata *pdata,
+ struct netdev_hw_addr *ha,
+ unsigned int *mac_reg)
+{
+ unsigned int mac_addr_hi, mac_addr_lo;
+ u8 *mac_addr;
+
+ mac_addr_lo = 0;
+ mac_addr_hi = 0;
+
+ if (ha) {
+ mac_addr = (u8 *)&mac_addr_lo;
+ mac_addr[0] = ha->addr[0];
+ mac_addr[1] = ha->addr[1];
+ mac_addr[2] = ha->addr[2];
+ mac_addr[3] = ha->addr[3];
+ mac_addr = (u8 *)&mac_addr_hi;
+ mac_addr[0] = ha->addr[4];
+ mac_addr[1] = ha->addr[5];
+
+ netif_dbg(pdata, drv, pdata->netdev,
+ "adding mac address %pM at %#x\n",
+ ha->addr, *mac_reg);
+
+ mac_addr_hi = XLGMAC_SET_REG_BITS(mac_addr_hi,
+ MAC_MACA1HR_AE_POS,
+ MAC_MACA1HR_AE_LEN,
+ 1);
+ }
+
+ writel(mac_addr_hi, pdata->mac_regs + *mac_reg);
+ *mac_reg += MAC_MACA_INC;
+ writel(mac_addr_lo, pdata->mac_regs + *mac_reg);
+ *mac_reg += MAC_MACA_INC;
+}
+
+static int xlgmac_enable_rx_vlan_stripping(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = readl(pdata->mac_regs + MAC_VLANTR);
+ /* Put the VLAN tag in the Rx descriptor */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_VLANTR_EVLRXS_POS,
+ MAC_VLANTR_EVLRXS_LEN, 1);
+ /* Don't check the VLAN type */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_VLANTR_DOVLTC_POS,
+ MAC_VLANTR_DOVLTC_LEN, 1);
+ /* Check only C-TAG (0x8100) packets */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_VLANTR_ERSVLM_POS,
+ MAC_VLANTR_ERSVLM_LEN, 0);
+ /* Don't consider an S-TAG (0x88A8) packet as a VLAN packet */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_VLANTR_ESVL_POS,
+ MAC_VLANTR_ESVL_LEN, 0);
+ /* Enable VLAN tag stripping */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_VLANTR_EVLS_POS,
+ MAC_VLANTR_EVLS_LEN, 0x3);
+ writel(regval, pdata->mac_regs + MAC_VLANTR);
+
+ return 0;
+}
+
+static int xlgmac_disable_rx_vlan_stripping(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = readl(pdata->mac_regs + MAC_VLANTR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_VLANTR_EVLS_POS,
+ MAC_VLANTR_EVLS_LEN, 0);
+ writel(regval, pdata->mac_regs + MAC_VLANTR);
+
+ return 0;
+}
+
+static int xlgmac_enable_rx_vlan_filtering(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = readl(pdata->mac_regs + MAC_PFR);
+ /* Enable VLAN filtering */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_PFR_VTFE_POS,
+ MAC_PFR_VTFE_LEN, 1);
+ writel(regval, pdata->mac_regs + MAC_PFR);
+
+ regval = readl(pdata->mac_regs + MAC_VLANTR);
+ /* Enable VLAN Hash Table filtering */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_VLANTR_VTHM_POS,
+ MAC_VLANTR_VTHM_LEN, 1);
+ /* Disable VLAN tag inverse matching */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_VLANTR_VTIM_POS,
+ MAC_VLANTR_VTIM_LEN, 0);
+ /* Only filter on the lower 12-bits of the VLAN tag */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_VLANTR_ETV_POS,
+ MAC_VLANTR_ETV_LEN, 1);
+ /* In order for the VLAN Hash Table filtering to be effective,
+ * the VLAN tag identifier in the VLAN Tag Register must not
+ * be zero. Set the VLAN tag identifier to "1" to enable the
+ * VLAN Hash Table filtering. This implies that a VLAN tag of
+ * 1 will always pass filtering.
+ */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_VLANTR_VL_POS,
+ MAC_VLANTR_VL_LEN, 1);
+ writel(regval, pdata->mac_regs + MAC_VLANTR);
+
+ return 0;
+}
+
+static int xlgmac_disable_rx_vlan_filtering(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = readl(pdata->mac_regs + MAC_PFR);
+ /* Disable VLAN filtering */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_PFR_VTFE_POS,
+ MAC_PFR_VTFE_LEN, 0);
+ writel(regval, pdata->mac_regs + MAC_PFR);
+
+ return 0;
+}
+
+static u32 xlgmac_vid_crc32_le(__le16 vid_le)
+{
+ unsigned char *data = (unsigned char *)&vid_le;
+ unsigned char data_byte = 0;
+ u32 poly = 0xedb88320;
+ u32 crc = ~0;
+ u32 temp = 0;
+ int i, bits;
+
+ bits = get_bitmask_order(VLAN_VID_MASK);
+ for (i = 0; i < bits; i++) {
+ if ((i % 8) == 0)
+ data_byte = data[i / 8];
+
+ temp = ((crc & 1) ^ data_byte) & 1;
+ crc >>= 1;
+ data_byte >>= 1;
+
+ if (temp)
+ crc ^= poly;
+ }
+
+ return crc;
+}
+
+static int xlgmac_update_vlan_hash_table(struct xlgmac_pdata *pdata)
+{
+ u16 vlan_hash_table = 0;
+ __le16 vid_le;
+ u32 regval;
+ u32 crc;
+ u16 vid;
+
+ /* Generate the VLAN Hash Table value */
+ for_each_set_bit(vid, pdata->active_vlans, VLAN_N_VID) {
+ /* Get the CRC32 value of the VLAN ID */
+ vid_le = cpu_to_le16(vid);
+ crc = bitrev32(~xlgmac_vid_crc32_le(vid_le)) >> 28;
+
+ vlan_hash_table |= (1 << crc);
+ }
+
+ regval = readl(pdata->mac_regs + MAC_VLANHTR);
+ /* Set the VLAN Hash Table filtering register */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_VLANHTR_VLHT_POS,
+ MAC_VLANHTR_VLHT_LEN, vlan_hash_table);
+ writel(regval, pdata->mac_regs + MAC_VLANHTR);
+
+ return 0;
+}
+
+static int xlgmac_set_promiscuous_mode(struct xlgmac_pdata *pdata,
+ unsigned int enable)
+{
+ unsigned int val = enable ? 1 : 0;
+ u32 regval;
+
+ regval = XLGMAC_GET_REG_BITS(readl(pdata->mac_regs + MAC_PFR),
+ MAC_PFR_PR_POS, MAC_PFR_PR_LEN);
+ if (regval == val)
+ return 0;
+
+ netif_dbg(pdata, drv, pdata->netdev, "%s promiscuous mode\n",
+ enable ? "entering" : "leaving");
+
+ regval = readl(pdata->mac_regs + MAC_PFR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_PFR_PR_POS,
+ MAC_PFR_PR_LEN, val);
+ writel(regval, pdata->mac_regs + MAC_PFR);
+
+ /* Hardware will still perform VLAN filtering in promiscuous mode */
+ if (enable) {
+ xlgmac_disable_rx_vlan_filtering(pdata);
+ } else {
+ if (pdata->netdev->features & NETIF_F_HW_VLAN_CTAG_FILTER)
+ xlgmac_enable_rx_vlan_filtering(pdata);
+ }
+
+ return 0;
+}
+
+static int xlgmac_set_all_multicast_mode(struct xlgmac_pdata *pdata,
+ unsigned int enable)
+{
+ unsigned int val = enable ? 1 : 0;
+ u32 regval;
+
+ regval = XLGMAC_GET_REG_BITS(readl(pdata->mac_regs + MAC_PFR),
+ MAC_PFR_PM_POS, MAC_PFR_PM_LEN);
+ if (regval == val)
+ return 0;
+
+ netif_dbg(pdata, drv, pdata->netdev, "%s allmulti mode\n",
+ enable ? "entering" : "leaving");
+
+ regval = readl(pdata->mac_regs + MAC_PFR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_PFR_PM_POS,
+ MAC_PFR_PM_LEN, val);
+ writel(regval, pdata->mac_regs + MAC_PFR);
+
+ return 0;
+}
+
+static void xlgmac_set_mac_addn_addrs(struct xlgmac_pdata *pdata)
+{
+ struct net_device *netdev = pdata->netdev;
+ struct netdev_hw_addr *ha;
+ unsigned int addn_macs;
+ unsigned int mac_reg;
+
+ mac_reg = MAC_MACA1HR;
+ addn_macs = pdata->hw_feat.addn_mac;
+
+ if (netdev_uc_count(netdev) > addn_macs) {
+ xlgmac_set_promiscuous_mode(pdata, 1);
+ } else {
+ netdev_for_each_uc_addr(ha, netdev) {
+ xlgmac_set_mac_reg(pdata, ha, &mac_reg);
+ addn_macs--;
+ }
+
+ if (netdev_mc_count(netdev) > addn_macs) {
+ xlgmac_set_all_multicast_mode(pdata, 1);
+ } else {
+ netdev_for_each_mc_addr(ha, netdev) {
+ xlgmac_set_mac_reg(pdata, ha, &mac_reg);
+ addn_macs--;
+ }
+ }
+ }
+
+ /* Clear remaining additional MAC address entries */
+ while (addn_macs--)
+ xlgmac_set_mac_reg(pdata, NULL, &mac_reg);
+}
+
+static void xlgmac_set_mac_hash_table(struct xlgmac_pdata *pdata)
+{
+ unsigned int hash_table_shift, hash_table_count;
+ u32 hash_table[XLGMAC_MAC_HASH_TABLE_SIZE];
+ struct net_device *netdev = pdata->netdev;
+ struct netdev_hw_addr *ha;
+ unsigned int hash_reg;
+ unsigned int i;
+ u32 crc;
+
+ hash_table_shift = 26 - (pdata->hw_feat.hash_table_size >> 7);
+ hash_table_count = pdata->hw_feat.hash_table_size / 32;
+ memset(hash_table, 0, sizeof(hash_table));
+
+ /* Build the MAC Hash Table register values */
+ netdev_for_each_uc_addr(ha, netdev) {
+ crc = bitrev32(~crc32_le(~0, ha->addr, ETH_ALEN));
+ crc >>= hash_table_shift;
+ hash_table[crc >> 5] |= (1 << (crc & 0x1f));
+ }
+
+ netdev_for_each_mc_addr(ha, netdev) {
+ crc = bitrev32(~crc32_le(~0, ha->addr, ETH_ALEN));
+ crc >>= hash_table_shift;
+ hash_table[crc >> 5] |= (1 << (crc & 0x1f));
+ }
+
+ /* Set the MAC Hash Table registers */
+ hash_reg = MAC_HTR0;
+ for (i = 0; i < hash_table_count; i++) {
+ writel(hash_table[i], pdata->mac_regs + hash_reg);
+ hash_reg += MAC_HTR_INC;
+ }
+}
+
+static int xlgmac_add_mac_addresses(struct xlgmac_pdata *pdata)
+{
+ if (pdata->hw_feat.hash_table_size)
+ xlgmac_set_mac_hash_table(pdata);
+ else
+ xlgmac_set_mac_addn_addrs(pdata);
+
+ return 0;
+}
+
+static void xlgmac_config_mac_address(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ xlgmac_set_mac_address(pdata, pdata->netdev->dev_addr);
+
+ /* Filtering is done using perfect filtering and hash filtering */
+ if (pdata->hw_feat.hash_table_size) {
+ regval = readl(pdata->mac_regs + MAC_PFR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_PFR_HPF_POS,
+ MAC_PFR_HPF_LEN, 1);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_PFR_HUC_POS,
+ MAC_PFR_HUC_LEN, 1);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_PFR_HMC_POS,
+ MAC_PFR_HMC_LEN, 1);
+ writel(regval, pdata->mac_regs + MAC_PFR);
+ }
+}
+
+static void xlgmac_config_jumbo_enable(struct xlgmac_pdata *pdata)
+{
+ unsigned int val;
+ u32 regval;
+
+ val = (pdata->netdev->mtu > XLGMAC_STD_PACKET_MTU) ? 1 : 0;
+
+ regval = readl(pdata->mac_regs + MAC_RCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RCR_JE_POS,
+ MAC_RCR_JE_LEN, val);
+ writel(regval, pdata->mac_regs + MAC_RCR);
+}
+
+static void xlgmac_config_checksum_offload(struct xlgmac_pdata *pdata)
+{
+ if (pdata->netdev->features & NETIF_F_RXCSUM)
+ xlgmac_enable_rx_csum(pdata);
+ else
+ xlgmac_disable_rx_csum(pdata);
+}
+
+static void xlgmac_config_vlan_support(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = readl(pdata->mac_regs + MAC_VLANIR);
+ /* Indicate that VLAN Tx CTAGs come from context descriptors */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_VLANIR_CSVL_POS,
+ MAC_VLANIR_CSVL_LEN, 0);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_VLANIR_VLTI_POS,
+ MAC_VLANIR_VLTI_LEN, 1);
+ writel(regval, pdata->mac_regs + MAC_VLANIR);
+
+ /* Set the current VLAN Hash Table register value */
+ xlgmac_update_vlan_hash_table(pdata);
+
+ if (pdata->netdev->features & NETIF_F_HW_VLAN_CTAG_FILTER)
+ xlgmac_enable_rx_vlan_filtering(pdata);
+ else
+ xlgmac_disable_rx_vlan_filtering(pdata);
+
+ if (pdata->netdev->features & NETIF_F_HW_VLAN_CTAG_RX)
+ xlgmac_enable_rx_vlan_stripping(pdata);
+ else
+ xlgmac_disable_rx_vlan_stripping(pdata);
+}
+
+static int xlgmac_config_rx_mode(struct xlgmac_pdata *pdata)
+{
+ struct net_device *netdev = pdata->netdev;
+ unsigned int pr_mode, am_mode;
+
+ pr_mode = ((netdev->flags & IFF_PROMISC) != 0);
+ am_mode = ((netdev->flags & IFF_ALLMULTI) != 0);
+
+ xlgmac_set_promiscuous_mode(pdata, pr_mode);
+ xlgmac_set_all_multicast_mode(pdata, am_mode);
+
+ xlgmac_add_mac_addresses(pdata);
+
+ return 0;
+}
+
+static void xlgmac_prepare_tx_stop(struct xlgmac_pdata *pdata,
+ struct xlgmac_channel *channel)
+{
+ unsigned int tx_dsr, tx_pos, tx_qidx;
+ unsigned long tx_timeout;
+ unsigned int tx_status;
+
+ /* Calculate the status register to read and the position within */
+ if (channel->queue_index < DMA_DSRX_FIRST_QUEUE) {
+ tx_dsr = DMA_DSR0;
+ tx_pos = (channel->queue_index * DMA_DSR_Q_LEN) +
+ DMA_DSR0_TPS_START;
+ } else {
+ tx_qidx = channel->queue_index - DMA_DSRX_FIRST_QUEUE;
+
+ tx_dsr = DMA_DSR1 + ((tx_qidx / DMA_DSRX_QPR) * DMA_DSRX_INC);
+ tx_pos = ((tx_qidx % DMA_DSRX_QPR) * DMA_DSR_Q_LEN) +
+ DMA_DSRX_TPS_START;
+ }
+
+ /* The Tx engine cannot be stopped if it is actively processing
+ * descriptors. Wait for the Tx engine to enter the stopped or
+ * suspended state. Don't wait forever though...
+ */
+ tx_timeout = jiffies + (XLGMAC_DMA_STOP_TIMEOUT * HZ);
+ while (time_before(jiffies, tx_timeout)) {
+ tx_status = readl(pdata->mac_regs + tx_dsr);
+ tx_status = XLGMAC_GET_REG_BITS(tx_status, tx_pos,
+ DMA_DSR_TPS_LEN);
+ if ((tx_status == DMA_TPS_STOPPED) ||
+ (tx_status == DMA_TPS_SUSPENDED))
+ break;
+
+ usleep_range(500, 1000);
+ }
+
+ if (!time_before(jiffies, tx_timeout))
+ netdev_info(pdata->netdev,
+ "timed out waiting for Tx DMA channel %u to stop\n",
+ channel->queue_index);
+}
+
+static void xlgmac_enable_tx(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+ u32 regval;
+
+ /* Enable each Tx DMA channel */
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->tx_ring)
+ break;
+
+ regval = readl(XLGMAC_DMA_REG(channel, DMA_CH_TCR));
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_CH_TCR_ST_POS,
+ DMA_CH_TCR_ST_LEN, 1);
+ writel(regval, XLGMAC_DMA_REG(channel, DMA_CH_TCR));
+ }
+
+ /* Enable each Tx queue */
+ for (i = 0; i < pdata->tx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_TQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_TQOMR_TXQEN_POS,
+ MTL_Q_TQOMR_TXQEN_LEN,
+ MTL_Q_ENABLED);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_TQOMR));
+ }
+
+ /* Enable MAC Tx */
+ regval = readl(pdata->mac_regs + MAC_TCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_TCR_TE_POS,
+ MAC_TCR_TE_LEN, 1);
+ writel(regval, pdata->mac_regs + MAC_TCR);
+}
+
+static void xlgmac_disable_tx(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+ u32 regval;
+
+ /* Prepare for Tx DMA channel stop */
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->tx_ring)
+ break;
+
+ xlgmac_prepare_tx_stop(pdata, channel);
+ }
+
+ /* Disable MAC Tx */
+ regval = readl(pdata->mac_regs + MAC_TCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_TCR_TE_POS,
+ MAC_TCR_TE_LEN, 0);
+ writel(regval, pdata->mac_regs + MAC_TCR);
+
+ /* Disable each Tx queue */
+ for (i = 0; i < pdata->tx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_TQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_TQOMR_TXQEN_POS,
+ MTL_Q_TQOMR_TXQEN_LEN, 0);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_TQOMR));
+ }
+
+ /* Disable each Tx DMA channel */
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->tx_ring)
+ break;
+
+ regval = readl(XLGMAC_DMA_REG(channel, DMA_CH_TCR));
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_CH_TCR_ST_POS,
+ DMA_CH_TCR_ST_LEN, 0);
+ writel(regval, XLGMAC_DMA_REG(channel, DMA_CH_TCR));
+ }
+}
+
+static void xlgmac_prepare_rx_stop(struct xlgmac_pdata *pdata,
+ unsigned int queue)
+{
+ unsigned int rx_status, prxq, rxqsts;
+ unsigned long rx_timeout;
+
+ /* The Rx engine cannot be stopped if it is actively processing
+ * packets. Wait for the Rx queue to empty the Rx fifo. Don't
+ * wait forever though...
+ */
+ rx_timeout = jiffies + (XLGMAC_DMA_STOP_TIMEOUT * HZ);
+ while (time_before(jiffies, rx_timeout)) {
+ rx_status = readl(XLGMAC_MTL_REG(pdata, queue, MTL_Q_RQDR));
+ prxq = XLGMAC_GET_REG_BITS(rx_status, MTL_Q_RQDR_PRXQ_POS,
+ MTL_Q_RQDR_PRXQ_LEN);
+ rxqsts = XLGMAC_GET_REG_BITS(rx_status, MTL_Q_RQDR_RXQSTS_POS,
+ MTL_Q_RQDR_RXQSTS_LEN);
+ if ((prxq == 0) && (rxqsts == 0))
+ break;
+
+ usleep_range(500, 1000);
+ }
+
+ if (!time_before(jiffies, rx_timeout))
+ netdev_info(pdata->netdev,
+ "timed out waiting for Rx queue %u to empty\n",
+ queue);
+}
+
+static void xlgmac_enable_rx(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int regval, i;
+
+ /* Enable each Rx DMA channel */
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->rx_ring)
+ break;
+
+ regval = readl(XLGMAC_DMA_REG(channel, DMA_CH_RCR));
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_CH_RCR_SR_POS,
+ DMA_CH_RCR_SR_LEN, 1);
+ writel(regval, XLGMAC_DMA_REG(channel, DMA_CH_RCR));
+ }
+
+ /* Enable each Rx queue */
+ regval = 0;
+ for (i = 0; i < pdata->rx_q_count; i++)
+ regval |= (0x02 << (i << 1));
+ writel(regval, pdata->mac_regs + MAC_RQC0R);
+
+ /* Enable MAC Rx */
+ regval = readl(pdata->mac_regs + MAC_RCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RCR_DCRCC_POS,
+ MAC_RCR_DCRCC_LEN, 1);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RCR_CST_POS,
+ MAC_RCR_CST_LEN, 1);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RCR_ACS_POS,
+ MAC_RCR_ACS_LEN, 1);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RCR_RE_POS,
+ MAC_RCR_RE_LEN, 1);
+ writel(regval, pdata->mac_regs + MAC_RCR);
+}
+
+static void xlgmac_disable_rx(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+ u32 regval;
+
+ /* Disable MAC Rx */
+ regval = readl(pdata->mac_regs + MAC_RCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RCR_DCRCC_POS,
+ MAC_RCR_DCRCC_LEN, 0);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RCR_CST_POS,
+ MAC_RCR_CST_LEN, 0);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RCR_ACS_POS,
+ MAC_RCR_ACS_LEN, 0);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RCR_RE_POS,
+ MAC_RCR_RE_LEN, 0);
+ writel(regval, pdata->mac_regs + MAC_RCR);
+
+ /* Prepare for Rx DMA channel stop */
+ for (i = 0; i < pdata->rx_q_count; i++)
+ xlgmac_prepare_rx_stop(pdata, i);
+
+ /* Disable each Rx queue */
+ writel(0, pdata->mac_regs + MAC_RQC0R);
+
+ /* Disable each Rx DMA channel */
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->rx_ring)
+ break;
+
+ regval = readl(XLGMAC_DMA_REG(channel, DMA_CH_RCR));
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_CH_RCR_SR_POS,
+ DMA_CH_RCR_SR_LEN, 0);
+ writel(regval, XLGMAC_DMA_REG(channel, DMA_CH_RCR));
+ }
+}
+
+static void xlgmac_tx_start_xmit(struct xlgmac_channel *channel,
+ struct xlgmac_ring *ring)
+{
+ struct xlgmac_pdata *pdata = channel->pdata;
+ struct xlgmac_desc_data *desc_data;
+
+ /* Make sure everything is written before the register write */
+ wmb();
+
+ /* Issue a poll command to Tx DMA by writing address
+ * of next immediate free descriptor
+ */
+ desc_data = XLGMAC_GET_DESC_DATA(ring, ring->cur);
+ writel(lower_32_bits(desc_data->dma_desc_addr),
+ XLGMAC_DMA_REG(channel, DMA_CH_TDTR_LO));
+
+ /* Start the Tx timer */
+ if (pdata->tx_usecs && !channel->tx_timer_active) {
+ channel->tx_timer_active = 1;
+ mod_timer(&channel->tx_timer,
+ jiffies + usecs_to_jiffies(pdata->tx_usecs));
+ }
+
+ ring->tx.xmit_more = 0;
+}
+
+static void xlgmac_dev_xmit(struct xlgmac_channel *channel)
+{
+ struct xlgmac_pdata *pdata = channel->pdata;
+ struct xlgmac_ring *ring = channel->tx_ring;
+ unsigned int tso_context, vlan_context;
+ struct xlgmac_desc_data *desc_data;
+ struct xlgmac_dma_desc *dma_desc;
+ struct xlgmac_pkt_info *pkt_info;
+ unsigned int csum, tso, vlan;
+ int start_index = ring->cur;
+ int cur_index = ring->cur;
+ unsigned int tx_set_ic;
+ int i;
+
+ pkt_info = &ring->pkt_info;
+ csum = XLGMAC_GET_REG_BITS(pkt_info->attributes,
+ TX_PACKET_ATTRIBUTES_CSUM_ENABLE_POS,
+ TX_PACKET_ATTRIBUTES_CSUM_ENABLE_LEN);
+ tso = XLGMAC_GET_REG_BITS(pkt_info->attributes,
+ TX_PACKET_ATTRIBUTES_TSO_ENABLE_POS,
+ TX_PACKET_ATTRIBUTES_TSO_ENABLE_LEN);
+ vlan = XLGMAC_GET_REG_BITS(pkt_info->attributes,
+ TX_PACKET_ATTRIBUTES_VLAN_CTAG_POS,
+ TX_PACKET_ATTRIBUTES_VLAN_CTAG_LEN);
+
+ if (tso && (pkt_info->mss != ring->tx.cur_mss))
+ tso_context = 1;
+ else
+ tso_context = 0;
+
+ if (vlan && (pkt_info->vlan_ctag != ring->tx.cur_vlan_ctag))
+ vlan_context = 1;
+ else
+ vlan_context = 0;
+
+ /* Determine if an interrupt should be generated for this Tx:
+ * Interrupt:
+ * - Tx frame count exceeds the frame count setting
+ * - Addition of Tx frame count to the frame count since the
+ * last interrupt was set exceeds the frame count setting
+ * No interrupt:
+ * - No frame count setting specified (ethtool -C ethX tx-frames 0)
+ * - Addition of Tx frame count to the frame count since the
+ * last interrupt was set does not exceed the frame count setting
+ */
+ ring->coalesce_count += pkt_info->tx_packets;
+ if (!pdata->tx_frames)
+ tx_set_ic = 0;
+ else if (pkt_info->tx_packets > pdata->tx_frames)
+ tx_set_ic = 1;
+ else if ((ring->coalesce_count % pdata->tx_frames) <
+ pkt_info->tx_packets)
+ tx_set_ic = 1;
+ else
+ tx_set_ic = 0;
+
+ desc_data = XLGMAC_GET_DESC_DATA(ring, cur_index);
+ dma_desc = desc_data->dma_desc;
+
+ /* Create a context descriptor if this is a TSO pkt_info */
+ if (tso_context || vlan_context) {
+ if (tso_context) {
+ netif_dbg(pdata, tx_queued, pdata->netdev,
+ "TSO context descriptor, mss=%u\n",
+ pkt_info->mss);
+
+ /* Set the MSS size */
+ dma_desc->desc2 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc2,
+ TX_CONTEXT_DESC2_MSS_POS,
+ TX_CONTEXT_DESC2_MSS_LEN,
+ pkt_info->mss);
+
+ /* Mark it as a CONTEXT descriptor */
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_CONTEXT_DESC3_CTXT_POS,
+ TX_CONTEXT_DESC3_CTXT_LEN,
+ 1);
+
+ /* Indicate this descriptor contains the MSS */
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_CONTEXT_DESC3_TCMSSV_POS,
+ TX_CONTEXT_DESC3_TCMSSV_LEN,
+ 1);
+
+ ring->tx.cur_mss = pkt_info->mss;
+ }
+
+ if (vlan_context) {
+ netif_dbg(pdata, tx_queued, pdata->netdev,
+ "VLAN context descriptor, ctag=%u\n",
+ pkt_info->vlan_ctag);
+
+ /* Mark it as a CONTEXT descriptor */
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_CONTEXT_DESC3_CTXT_POS,
+ TX_CONTEXT_DESC3_CTXT_LEN,
+ 1);
+
+ /* Set the VLAN tag */
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_CONTEXT_DESC3_VT_POS,
+ TX_CONTEXT_DESC3_VT_LEN,
+ pkt_info->vlan_ctag);
+
+ /* Indicate this descriptor contains the VLAN tag */
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_CONTEXT_DESC3_VLTV_POS,
+ TX_CONTEXT_DESC3_VLTV_LEN,
+ 1);
+
+ ring->tx.cur_vlan_ctag = pkt_info->vlan_ctag;
+ }
+
+ cur_index++;
+ desc_data = XLGMAC_GET_DESC_DATA(ring, cur_index);
+ dma_desc = desc_data->dma_desc;
+ }
+
+ /* Update buffer address (for TSO this is the header) */
+ dma_desc->desc0 = cpu_to_le32(lower_32_bits(desc_data->skb_dma));
+ dma_desc->desc1 = cpu_to_le32(upper_32_bits(desc_data->skb_dma));
+
+ /* Update the buffer length */
+ dma_desc->desc2 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc2,
+ TX_NORMAL_DESC2_HL_B1L_POS,
+ TX_NORMAL_DESC2_HL_B1L_LEN,
+ desc_data->skb_dma_len);
+
+ /* VLAN tag insertion check */
+ if (vlan) {
+ dma_desc->desc2 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc2,
+ TX_NORMAL_DESC2_VTIR_POS,
+ TX_NORMAL_DESC2_VTIR_LEN,
+ TX_NORMAL_DESC2_VLAN_INSERT);
+ pdata->stats.tx_vlan_packets++;
+ }
+
+ /* Timestamp enablement check */
+ if (XLGMAC_GET_REG_BITS(pkt_info->attributes,
+ TX_PACKET_ATTRIBUTES_PTP_POS,
+ TX_PACKET_ATTRIBUTES_PTP_LEN))
+ dma_desc->desc2 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc2,
+ TX_NORMAL_DESC2_TTSE_POS,
+ TX_NORMAL_DESC2_TTSE_LEN,
+ 1);
+
+ /* Mark it as First Descriptor */
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_FD_POS,
+ TX_NORMAL_DESC3_FD_LEN,
+ 1);
+
+ /* Mark it as a NORMAL descriptor */
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_CTXT_POS,
+ TX_NORMAL_DESC3_CTXT_LEN,
+ 0);
+
+ /* Set OWN bit if not the first descriptor */
+ if (cur_index != start_index)
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_OWN_POS,
+ TX_NORMAL_DESC3_OWN_LEN,
+ 1);
+
+ if (tso) {
+ /* Enable TSO */
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_TSE_POS,
+ TX_NORMAL_DESC3_TSE_LEN, 1);
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_TCPPL_POS,
+ TX_NORMAL_DESC3_TCPPL_LEN,
+ pkt_info->tcp_payload_len);
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_TCPHDRLEN_POS,
+ TX_NORMAL_DESC3_TCPHDRLEN_LEN,
+ pkt_info->tcp_header_len / 4);
+
+ pdata->stats.tx_tso_packets++;
+ } else {
+ /* Enable CRC and Pad Insertion */
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_CPC_POS,
+ TX_NORMAL_DESC3_CPC_LEN, 0);
+
+ /* Enable HW CSUM */
+ if (csum)
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_CIC_POS,
+ TX_NORMAL_DESC3_CIC_LEN,
+ 0x3);
+
+ /* Set the total length to be transmitted */
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_FL_POS,
+ TX_NORMAL_DESC3_FL_LEN,
+ pkt_info->length);
+ }
+
+ for (i = cur_index - start_index + 1; i < pkt_info->desc_count; i++) {
+ cur_index++;
+ desc_data = XLGMAC_GET_DESC_DATA(ring, cur_index);
+ dma_desc = desc_data->dma_desc;
+
+ /* Update buffer address */
+ dma_desc->desc0 =
+ cpu_to_le32(lower_32_bits(desc_data->skb_dma));
+ dma_desc->desc1 =
+ cpu_to_le32(upper_32_bits(desc_data->skb_dma));
+
+ /* Update the buffer length */
+ dma_desc->desc2 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc2,
+ TX_NORMAL_DESC2_HL_B1L_POS,
+ TX_NORMAL_DESC2_HL_B1L_LEN,
+ desc_data->skb_dma_len);
+
+ /* Set OWN bit */
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_OWN_POS,
+ TX_NORMAL_DESC3_OWN_LEN, 1);
+
+ /* Mark it as NORMAL descriptor */
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_CTXT_POS,
+ TX_NORMAL_DESC3_CTXT_LEN, 0);
+
+ /* Enable HW CSUM */
+ if (csum)
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_CIC_POS,
+ TX_NORMAL_DESC3_CIC_LEN,
+ 0x3);
+ }
+
+ /* Set LAST bit for the last descriptor */
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_LD_POS,
+ TX_NORMAL_DESC3_LD_LEN, 1);
+
+ /* Set IC bit based on Tx coalescing settings */
+ if (tx_set_ic)
+ dma_desc->desc2 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc2,
+ TX_NORMAL_DESC2_IC_POS,
+ TX_NORMAL_DESC2_IC_LEN, 1);
+
+ /* Save the Tx info to report back during cleanup */
+ desc_data->tx.packets = pkt_info->tx_packets;
+ desc_data->tx.bytes = pkt_info->tx_bytes;
+
+ /* In case the Tx DMA engine is running, make sure everything
+ * is written to the descriptor(s) before setting the OWN bit
+ * for the first descriptor
+ */
+ dma_wmb();
+
+ /* Set OWN bit for the first descriptor */
+ desc_data = XLGMAC_GET_DESC_DATA(ring, start_index);
+ dma_desc = desc_data->dma_desc;
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ TX_NORMAL_DESC3_OWN_POS,
+ TX_NORMAL_DESC3_OWN_LEN, 1);
+
+ if (netif_msg_tx_queued(pdata))
+ xlgmac_dump_tx_desc(pdata, ring, start_index,
+ pkt_info->desc_count, 1);
+
+ /* Make sure ownership is written to the descriptor */
+ smp_wmb();
+
+ ring->cur = cur_index + 1;
+ if (!pkt_info->skb->xmit_more ||
+ netif_xmit_stopped(netdev_get_tx_queue(pdata->netdev,
+ channel->queue_index)))
+ xlgmac_tx_start_xmit(channel, ring);
+ else
+ ring->tx.xmit_more = 1;
+
+ XLGMAC_PR("%s: descriptors %u to %u written\n",
+ channel->name, start_index & (ring->dma_desc_count - 1),
+ (ring->cur - 1) & (ring->dma_desc_count - 1));
+}
+
+static void xlgmac_get_rx_tstamp(struct xlgmac_pkt_info *pkt_info,
+ struct xlgmac_dma_desc *dma_desc)
+{
+ u32 tsa, tsd;
+ u64 nsec;
+
+ tsa = XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ RX_CONTEXT_DESC3_TSA_POS,
+ RX_CONTEXT_DESC3_TSA_LEN);
+ tsd = XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ RX_CONTEXT_DESC3_TSD_POS,
+ RX_CONTEXT_DESC3_TSD_LEN);
+ if (tsa && !tsd) {
+ nsec = le32_to_cpu(dma_desc->desc1);
+ nsec <<= 32;
+ nsec |= le32_to_cpu(dma_desc->desc0);
+ if (nsec != 0xffffffffffffffffULL) {
+ pkt_info->rx_tstamp = nsec;
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_RX_TSTAMP_POS,
+ RX_PACKET_ATTRIBUTES_RX_TSTAMP_LEN,
+ 1);
+ }
+ }
+}
+
+static void xlgmac_tx_desc_reset(struct xlgmac_desc_data *desc_data)
+{
+ struct xlgmac_dma_desc *dma_desc = desc_data->dma_desc;
+
+ /* Reset the Tx descriptor
+ * Set buffer 1 (lo) address to zero
+ * Set buffer 1 (hi) address to zero
+ * Reset all other control bits (IC, TTSE, B2L & B1L)
+ * Reset all other control bits (OWN, CTXT, FD, LD, CPC, CIC, etc)
+ */
+ dma_desc->desc0 = 0;
+ dma_desc->desc1 = 0;
+ dma_desc->desc2 = 0;
+ dma_desc->desc3 = 0;
+
+ /* Make sure ownership is written to the descriptor */
+ dma_wmb();
+}
+
+static void xlgmac_tx_desc_init(struct xlgmac_channel *channel)
+{
+ struct xlgmac_ring *ring = channel->tx_ring;
+ struct xlgmac_desc_data *desc_data;
+ int start_index = ring->cur;
+ int i;
+
+ /* Initialze all descriptors */
+ for (i = 0; i < ring->dma_desc_count; i++) {
+ desc_data = XLGMAC_GET_DESC_DATA(ring, i);
+
+ /* Initialize Tx descriptor */
+ xlgmac_tx_desc_reset(desc_data);
+ }
+
+ /* Update the total number of Tx descriptors */
+ writel(ring->dma_desc_count - 1, XLGMAC_DMA_REG(channel, DMA_CH_TDRLR));
+
+ /* Update the starting address of descriptor ring */
+ desc_data = XLGMAC_GET_DESC_DATA(ring, start_index);
+ writel(upper_32_bits(desc_data->dma_desc_addr),
+ XLGMAC_DMA_REG(channel, DMA_CH_TDLR_HI));
+ writel(lower_32_bits(desc_data->dma_desc_addr),
+ XLGMAC_DMA_REG(channel, DMA_CH_TDLR_LO));
+}
+
+static void xlgmac_rx_desc_reset(struct xlgmac_pdata *pdata,
+ struct xlgmac_desc_data *desc_data,
+ unsigned int index)
+{
+ struct xlgmac_dma_desc *dma_desc = desc_data->dma_desc;
+ unsigned int rx_frames = pdata->rx_frames;
+ unsigned int rx_usecs = pdata->rx_usecs;
+ dma_addr_t hdr_dma, buf_dma;
+ unsigned int inte;
+
+ if (!rx_usecs && !rx_frames) {
+ /* No coalescing, interrupt for every descriptor */
+ inte = 1;
+ } else {
+ /* Set interrupt based on Rx frame coalescing setting */
+ if (rx_frames && !((index + 1) % rx_frames))
+ inte = 1;
+ else
+ inte = 0;
+ }
+
+ /* Reset the Rx descriptor
+ * Set buffer 1 (lo) address to header dma address (lo)
+ * Set buffer 1 (hi) address to header dma address (hi)
+ * Set buffer 2 (lo) address to buffer dma address (lo)
+ * Set buffer 2 (hi) address to buffer dma address (hi) and
+ * set control bits OWN and INTE
+ */
+ hdr_dma = desc_data->rx.hdr.dma_base + desc_data->rx.hdr.dma_off;
+ buf_dma = desc_data->rx.buf.dma_base + desc_data->rx.buf.dma_off;
+ dma_desc->desc0 = cpu_to_le32(lower_32_bits(hdr_dma));
+ dma_desc->desc1 = cpu_to_le32(upper_32_bits(hdr_dma));
+ dma_desc->desc2 = cpu_to_le32(lower_32_bits(buf_dma));
+ dma_desc->desc3 = cpu_to_le32(upper_32_bits(buf_dma));
+
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ RX_NORMAL_DESC3_INTE_POS,
+ RX_NORMAL_DESC3_INTE_LEN,
+ inte);
+
+ /* Since the Rx DMA engine is likely running, make sure everything
+ * is written to the descriptor(s) before setting the OWN bit
+ * for the descriptor
+ */
+ dma_wmb();
+
+ dma_desc->desc3 = XLGMAC_SET_REG_BITS_LE(
+ dma_desc->desc3,
+ RX_NORMAL_DESC3_OWN_POS,
+ RX_NORMAL_DESC3_OWN_LEN,
+ 1);
+
+ /* Make sure ownership is written to the descriptor */
+ dma_wmb();
+}
+
+static void xlgmac_rx_desc_init(struct xlgmac_channel *channel)
+{
+ struct xlgmac_pdata *pdata = channel->pdata;
+ struct xlgmac_ring *ring = channel->rx_ring;
+ unsigned int start_index = ring->cur;
+ struct xlgmac_desc_data *desc_data;
+ unsigned int i;
+
+ /* Initialize all descriptors */
+ for (i = 0; i < ring->dma_desc_count; i++) {
+ desc_data = XLGMAC_GET_DESC_DATA(ring, i);
+
+ /* Initialize Rx descriptor */
+ xlgmac_rx_desc_reset(pdata, desc_data, i);
+ }
+
+ /* Update the total number of Rx descriptors */
+ writel(ring->dma_desc_count - 1, XLGMAC_DMA_REG(channel, DMA_CH_RDRLR));
+
+ /* Update the starting address of descriptor ring */
+ desc_data = XLGMAC_GET_DESC_DATA(ring, start_index);
+ writel(upper_32_bits(desc_data->dma_desc_addr),
+ XLGMAC_DMA_REG(channel, DMA_CH_RDLR_HI));
+ writel(lower_32_bits(desc_data->dma_desc_addr),
+ XLGMAC_DMA_REG(channel, DMA_CH_RDLR_LO));
+
+ /* Update the Rx Descriptor Tail Pointer */
+ desc_data = XLGMAC_GET_DESC_DATA(ring, start_index +
+ ring->dma_desc_count - 1);
+ writel(lower_32_bits(desc_data->dma_desc_addr),
+ XLGMAC_DMA_REG(channel, DMA_CH_RDTR_LO));
+}
+
+static int xlgmac_is_context_desc(struct xlgmac_dma_desc *dma_desc)
+{
+ /* Rx and Tx share CTXT bit, so check TDES3.CTXT bit */
+ return XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ TX_NORMAL_DESC3_CTXT_POS,
+ TX_NORMAL_DESC3_CTXT_LEN);
+}
+
+static int xlgmac_is_last_desc(struct xlgmac_dma_desc *dma_desc)
+{
+ /* Rx and Tx share LD bit, so check TDES3.LD bit */
+ return XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ TX_NORMAL_DESC3_LD_POS,
+ TX_NORMAL_DESC3_LD_LEN);
+}
+
+static int xlgmac_disable_tx_flow_control(struct xlgmac_pdata *pdata)
+{
+ unsigned int max_q_count, q_count;
+ unsigned int reg, regval;
+ unsigned int i;
+
+ /* Clear MTL flow control */
+ for (i = 0; i < pdata->rx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_RQOMR_EHFC_POS,
+ MTL_Q_RQOMR_EHFC_LEN, 0);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ }
+
+ /* Clear MAC flow control */
+ max_q_count = XLGMAC_MAX_FLOW_CONTROL_QUEUES;
+ q_count = min_t(unsigned int, pdata->tx_q_count, max_q_count);
+ reg = MAC_Q0TFCR;
+ for (i = 0; i < q_count; i++) {
+ regval = readl(pdata->mac_regs + reg);
+ regval = XLGMAC_SET_REG_BITS(regval,
+ MAC_Q0TFCR_TFE_POS,
+ MAC_Q0TFCR_TFE_LEN,
+ 0);
+ writel(regval, pdata->mac_regs + reg);
+
+ reg += MAC_QTFCR_INC;
+ }
+
+ return 0;
+}
+
+static int xlgmac_enable_tx_flow_control(struct xlgmac_pdata *pdata)
+{
+ unsigned int max_q_count, q_count;
+ unsigned int reg, regval;
+ unsigned int i;
+
+ /* Set MTL flow control */
+ for (i = 0; i < pdata->rx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_RQOMR_EHFC_POS,
+ MTL_Q_RQOMR_EHFC_LEN, 1);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ }
+
+ /* Set MAC flow control */
+ max_q_count = XLGMAC_MAX_FLOW_CONTROL_QUEUES;
+ q_count = min_t(unsigned int, pdata->tx_q_count, max_q_count);
+ reg = MAC_Q0TFCR;
+ for (i = 0; i < q_count; i++) {
+ regval = readl(pdata->mac_regs + reg);
+
+ /* Enable transmit flow control */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_Q0TFCR_TFE_POS,
+ MAC_Q0TFCR_TFE_LEN, 1);
+ /* Set pause time */
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_Q0TFCR_PT_POS,
+ MAC_Q0TFCR_PT_LEN, 0xffff);
+
+ writel(regval, pdata->mac_regs + reg);
+
+ reg += MAC_QTFCR_INC;
+ }
+
+ return 0;
+}
+
+static int xlgmac_disable_rx_flow_control(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = readl(pdata->mac_regs + MAC_RFCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RFCR_RFE_POS,
+ MAC_RFCR_RFE_LEN, 0);
+ writel(regval, pdata->mac_regs + MAC_RFCR);
+
+ return 0;
+}
+
+static int xlgmac_enable_rx_flow_control(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = readl(pdata->mac_regs + MAC_RFCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RFCR_RFE_POS,
+ MAC_RFCR_RFE_LEN, 1);
+ writel(regval, pdata->mac_regs + MAC_RFCR);
+
+ return 0;
+}
+
+static int xlgmac_config_tx_flow_control(struct xlgmac_pdata *pdata)
+{
+ if (pdata->tx_pause)
+ xlgmac_enable_tx_flow_control(pdata);
+ else
+ xlgmac_disable_tx_flow_control(pdata);
+
+ return 0;
+}
+
+static int xlgmac_config_rx_flow_control(struct xlgmac_pdata *pdata)
+{
+ if (pdata->rx_pause)
+ xlgmac_enable_rx_flow_control(pdata);
+ else
+ xlgmac_disable_rx_flow_control(pdata);
+
+ return 0;
+}
+
+static int xlgmac_config_rx_coalesce(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+ u32 regval;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->rx_ring)
+ break;
+
+ regval = readl(XLGMAC_DMA_REG(channel, DMA_CH_RIWT));
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_CH_RIWT_RWT_POS,
+ DMA_CH_RIWT_RWT_LEN,
+ pdata->rx_riwt);
+ writel(regval, XLGMAC_DMA_REG(channel, DMA_CH_RIWT));
+ }
+
+ return 0;
+}
+
+static void xlgmac_config_flow_control(struct xlgmac_pdata *pdata)
+{
+ xlgmac_config_tx_flow_control(pdata);
+ xlgmac_config_rx_flow_control(pdata);
+}
+
+static void xlgmac_config_rx_fep_enable(struct xlgmac_pdata *pdata)
+{
+ unsigned int i;
+ u32 regval;
+
+ for (i = 0; i < pdata->rx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_RQOMR_FEP_POS,
+ MTL_Q_RQOMR_FEP_LEN, 1);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ }
+}
+
+static void xlgmac_config_rx_fup_enable(struct xlgmac_pdata *pdata)
+{
+ unsigned int i;
+ u32 regval;
+
+ for (i = 0; i < pdata->rx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_RQOMR_FUP_POS,
+ MTL_Q_RQOMR_FUP_LEN, 1);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ }
+}
+
+static int xlgmac_config_tx_coalesce(struct xlgmac_pdata *pdata)
+{
+ return 0;
+}
+
+static void xlgmac_config_rx_buffer_size(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+ u32 regval;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->rx_ring)
+ break;
+
+ regval = readl(XLGMAC_DMA_REG(channel, DMA_CH_RCR));
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_CH_RCR_RBSZ_POS,
+ DMA_CH_RCR_RBSZ_LEN,
+ pdata->rx_buf_size);
+ writel(regval, XLGMAC_DMA_REG(channel, DMA_CH_RCR));
+ }
+}
+
+static void xlgmac_config_tso_mode(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+ u32 regval;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->tx_ring)
+ break;
+
+ if (pdata->hw_feat.tso) {
+ regval = readl(XLGMAC_DMA_REG(channel, DMA_CH_TCR));
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_CH_TCR_TSE_POS,
+ DMA_CH_TCR_TSE_LEN, 1);
+ writel(regval, XLGMAC_DMA_REG(channel, DMA_CH_TCR));
+ }
+ }
+}
+
+static void xlgmac_config_sph_mode(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+ u32 regval;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->rx_ring)
+ break;
+
+ regval = readl(XLGMAC_DMA_REG(channel, DMA_CH_CR));
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_CH_CR_SPH_POS,
+ DMA_CH_CR_SPH_LEN, 1);
+ writel(regval, XLGMAC_DMA_REG(channel, DMA_CH_CR));
+ }
+
+ regval = readl(pdata->mac_regs + MAC_RCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RCR_HDSMS_POS,
+ MAC_RCR_HDSMS_LEN,
+ XLGMAC_SPH_HDSMS_SIZE);
+ writel(regval, pdata->mac_regs + MAC_RCR);
+}
+
+static unsigned int xlgmac_usec_to_riwt(struct xlgmac_pdata *pdata,
+ unsigned int usec)
+{
+ unsigned long rate;
+ unsigned int ret;
+
+ rate = pdata->sysclk_rate;
+
+ /* Convert the input usec value to the watchdog timer value. Each
+ * watchdog timer value is equivalent to 256 clock cycles.
+ * Calculate the required value as:
+ * ( usec * ( system_clock_mhz / 10^6 ) / 256
+ */
+ ret = (usec * (rate / 1000000)) / 256;
+
+ return ret;
+}
+
+static unsigned int xlgmac_riwt_to_usec(struct xlgmac_pdata *pdata,
+ unsigned int riwt)
+{
+ unsigned long rate;
+ unsigned int ret;
+
+ rate = pdata->sysclk_rate;
+
+ /* Convert the input watchdog timer value to the usec value. Each
+ * watchdog timer value is equivalent to 256 clock cycles.
+ * Calculate the required value as:
+ * ( riwt * 256 ) / ( system_clock_mhz / 10^6 )
+ */
+ ret = (riwt * 256) / (rate / 1000000);
+
+ return ret;
+}
+
+static int xlgmac_config_rx_threshold(struct xlgmac_pdata *pdata,
+ unsigned int val)
+{
+ unsigned int i;
+ u32 regval;
+
+ for (i = 0; i < pdata->rx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_RQOMR_RTC_POS,
+ MTL_Q_RQOMR_RTC_LEN, val);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ }
+
+ return 0;
+}
+
+static void xlgmac_config_mtl_mode(struct xlgmac_pdata *pdata)
+{
+ unsigned int i;
+ u32 regval;
+
+ /* Set Tx to weighted round robin scheduling algorithm */
+ regval = readl(pdata->mac_regs + MTL_OMR);
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_OMR_ETSALG_POS,
+ MTL_OMR_ETSALG_LEN, MTL_ETSALG_WRR);
+ writel(regval, pdata->mac_regs + MTL_OMR);
+
+ /* Set Tx traffic classes to use WRR algorithm with equal weights */
+ for (i = 0; i < pdata->hw_feat.tc_cnt; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_TC_ETSCR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_TC_ETSCR_TSA_POS,
+ MTL_TC_ETSCR_TSA_LEN, MTL_TSA_ETS);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_TC_ETSCR));
+
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_TC_QWR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_TC_QWR_QW_POS,
+ MTL_TC_QWR_QW_LEN, 1);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_TC_QWR));
+ }
+
+ /* Set Rx to strict priority algorithm */
+ regval = readl(pdata->mac_regs + MTL_OMR);
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_OMR_RAA_POS,
+ MTL_OMR_RAA_LEN, MTL_RAA_SP);
+ writel(regval, pdata->mac_regs + MTL_OMR);
+}
+
+static void xlgmac_config_queue_mapping(struct xlgmac_pdata *pdata)
+{
+ unsigned int ppq, ppq_extra, prio, prio_queues;
+ unsigned int qptc, qptc_extra, queue;
+ unsigned int reg, regval;
+ unsigned int mask;
+ unsigned int i, j;
+
+ /* Map the MTL Tx Queues to Traffic Classes
+ * Note: Tx Queues >= Traffic Classes
+ */
+ qptc = pdata->tx_q_count / pdata->hw_feat.tc_cnt;
+ qptc_extra = pdata->tx_q_count % pdata->hw_feat.tc_cnt;
+
+ for (i = 0, queue = 0; i < pdata->hw_feat.tc_cnt; i++) {
+ for (j = 0; j < qptc; j++) {
+ netif_dbg(pdata, drv, pdata->netdev,
+ "TXq%u mapped to TC%u\n", queue, i);
+ regval = readl(XLGMAC_MTL_REG(pdata, queue,
+ MTL_Q_TQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval,
+ MTL_Q_TQOMR_Q2TCMAP_POS,
+ MTL_Q_TQOMR_Q2TCMAP_LEN,
+ i);
+ writel(regval, XLGMAC_MTL_REG(pdata, queue,
+ MTL_Q_TQOMR));
+ queue++;
+ }
+
+ if (i < qptc_extra) {
+ netif_dbg(pdata, drv, pdata->netdev,
+ "TXq%u mapped to TC%u\n", queue, i);
+ regval = readl(XLGMAC_MTL_REG(pdata, queue,
+ MTL_Q_TQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval,
+ MTL_Q_TQOMR_Q2TCMAP_POS,
+ MTL_Q_TQOMR_Q2TCMAP_LEN,
+ i);
+ writel(regval, XLGMAC_MTL_REG(pdata, queue,
+ MTL_Q_TQOMR));
+ queue++;
+ }
+ }
+
+ /* Map the 8 VLAN priority values to available MTL Rx queues */
+ prio_queues = min_t(unsigned int, IEEE_8021QAZ_MAX_TCS,
+ pdata->rx_q_count);
+ ppq = IEEE_8021QAZ_MAX_TCS / prio_queues;
+ ppq_extra = IEEE_8021QAZ_MAX_TCS % prio_queues;
+
+ reg = MAC_RQC2R;
+ regval = 0;
+ for (i = 0, prio = 0; i < prio_queues;) {
+ mask = 0;
+ for (j = 0; j < ppq; j++) {
+ netif_dbg(pdata, drv, pdata->netdev,
+ "PRIO%u mapped to RXq%u\n", prio, i);
+ mask |= (1 << prio);
+ prio++;
+ }
+
+ if (i < ppq_extra) {
+ netif_dbg(pdata, drv, pdata->netdev,
+ "PRIO%u mapped to RXq%u\n", prio, i);
+ mask |= (1 << prio);
+ prio++;
+ }
+
+ regval |= (mask << ((i++ % MAC_RQC2_Q_PER_REG) << 3));
+
+ if ((i % MAC_RQC2_Q_PER_REG) && (i != prio_queues))
+ continue;
+
+ writel(regval, pdata->mac_regs + reg);
+ reg += MAC_RQC2_INC;
+ regval = 0;
+ }
+
+ /* Configure one to one, MTL Rx queue to DMA Rx channel mapping
+ * ie Q0 <--> CH0, Q1 <--> CH1 ... Q11 <--> CH11
+ */
+ reg = MTL_RQDCM0R;
+ regval = readl(pdata->mac_regs + reg);
+ regval |= (MTL_RQDCM0R_Q0MDMACH | MTL_RQDCM0R_Q1MDMACH |
+ MTL_RQDCM0R_Q2MDMACH | MTL_RQDCM0R_Q3MDMACH);
+ writel(regval, pdata->mac_regs + reg);
+
+ reg += MTL_RQDCM_INC;
+ regval = readl(pdata->mac_regs + reg);
+ regval |= (MTL_RQDCM1R_Q4MDMACH | MTL_RQDCM1R_Q5MDMACH |
+ MTL_RQDCM1R_Q6MDMACH | MTL_RQDCM1R_Q7MDMACH);
+ writel(regval, pdata->mac_regs + reg);
+
+ reg += MTL_RQDCM_INC;
+ regval = readl(pdata->mac_regs + reg);
+ regval |= (MTL_RQDCM2R_Q8MDMACH | MTL_RQDCM2R_Q9MDMACH |
+ MTL_RQDCM2R_Q10MDMACH | MTL_RQDCM2R_Q11MDMACH);
+ writel(regval, pdata->mac_regs + reg);
+}
+
+static unsigned int xlgmac_calculate_per_queue_fifo(
+ unsigned int fifo_size,
+ unsigned int queue_count)
+{
+ unsigned int q_fifo_size;
+ unsigned int p_fifo;
+
+ /* Calculate the configured fifo size */
+ q_fifo_size = 1 << (fifo_size + 7);
+
+ /* The configured value may not be the actual amount of fifo RAM */
+ q_fifo_size = min_t(unsigned int, XLGMAC_MAX_FIFO, q_fifo_size);
+
+ q_fifo_size = q_fifo_size / queue_count;
+
+ /* Each increment in the queue fifo size represents 256 bytes of
+ * fifo, with 0 representing 256 bytes. Distribute the fifo equally
+ * between the queues.
+ */
+ p_fifo = q_fifo_size / 256;
+ if (p_fifo)
+ p_fifo--;
+
+ return p_fifo;
+}
+
+static void xlgmac_config_tx_fifo_size(struct xlgmac_pdata *pdata)
+{
+ unsigned int fifo_size;
+ unsigned int i;
+ u32 regval;
+
+ fifo_size = xlgmac_calculate_per_queue_fifo(
+ pdata->hw_feat.tx_fifo_size,
+ pdata->tx_q_count);
+
+ for (i = 0; i < pdata->tx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_TQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_TQOMR_TQS_POS,
+ MTL_Q_TQOMR_TQS_LEN, fifo_size);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_TQOMR));
+ }
+
+ netif_info(pdata, drv, pdata->netdev,
+ "%d Tx hardware queues, %d byte fifo per queue\n",
+ pdata->tx_q_count, ((fifo_size + 1) * 256));
+}
+
+static void xlgmac_config_rx_fifo_size(struct xlgmac_pdata *pdata)
+{
+ unsigned int fifo_size;
+ unsigned int i;
+ u32 regval;
+
+ fifo_size = xlgmac_calculate_per_queue_fifo(
+ pdata->hw_feat.rx_fifo_size,
+ pdata->rx_q_count);
+
+ for (i = 0; i < pdata->rx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_RQOMR_RQS_POS,
+ MTL_Q_RQOMR_RQS_LEN, fifo_size);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ }
+
+ netif_info(pdata, drv, pdata->netdev,
+ "%d Rx hardware queues, %d byte fifo per queue\n",
+ pdata->rx_q_count, ((fifo_size + 1) * 256));
+}
+
+static void xlgmac_config_flow_control_threshold(struct xlgmac_pdata *pdata)
+{
+ unsigned int i;
+ u32 regval;
+
+ for (i = 0; i < pdata->rx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_RQFCR));
+ /* Activate flow control when less than 4k left in fifo */
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_RQFCR_RFA_POS,
+ MTL_Q_RQFCR_RFA_LEN, 2);
+ /* De-activate flow control when more than 6k left in fifo */
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_RQFCR_RFD_POS,
+ MTL_Q_RQFCR_RFD_LEN, 4);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_RQFCR));
+ }
+}
+
+static int xlgmac_config_tx_threshold(struct xlgmac_pdata *pdata,
+ unsigned int val)
+{
+ unsigned int i;
+ u32 regval;
+
+ for (i = 0; i < pdata->tx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_TQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_TQOMR_TTC_POS,
+ MTL_Q_TQOMR_TTC_LEN, val);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_TQOMR));
+ }
+
+ return 0;
+}
+
+static int xlgmac_config_rsf_mode(struct xlgmac_pdata *pdata,
+ unsigned int val)
+{
+ unsigned int i;
+ u32 regval;
+
+ for (i = 0; i < pdata->rx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_RQOMR_RSF_POS,
+ MTL_Q_RQOMR_RSF_LEN, val);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_RQOMR));
+ }
+
+ return 0;
+}
+
+static int xlgmac_config_tsf_mode(struct xlgmac_pdata *pdata,
+ unsigned int val)
+{
+ unsigned int i;
+ u32 regval;
+
+ for (i = 0; i < pdata->tx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_TQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_TQOMR_TSF_POS,
+ MTL_Q_TQOMR_TSF_LEN, val);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_TQOMR));
+ }
+
+ return 0;
+}
+
+static int xlgmac_config_osp_mode(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+ u32 regval;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->tx_ring)
+ break;
+
+ regval = readl(XLGMAC_DMA_REG(channel, DMA_CH_TCR));
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_CH_TCR_OSP_POS,
+ DMA_CH_TCR_OSP_LEN,
+ pdata->tx_osp_mode);
+ writel(regval, XLGMAC_DMA_REG(channel, DMA_CH_TCR));
+ }
+
+ return 0;
+}
+
+static int xlgmac_config_pblx8(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+ u32 regval;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ regval = readl(XLGMAC_DMA_REG(channel, DMA_CH_CR));
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_CH_CR_PBLX8_POS,
+ DMA_CH_CR_PBLX8_LEN,
+ pdata->pblx8);
+ writel(regval, XLGMAC_DMA_REG(channel, DMA_CH_CR));
+ }
+
+ return 0;
+}
+
+static int xlgmac_get_tx_pbl_val(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = readl(XLGMAC_DMA_REG(pdata->channel_head, DMA_CH_TCR));
+ regval = XLGMAC_GET_REG_BITS(regval, DMA_CH_TCR_PBL_POS,
+ DMA_CH_TCR_PBL_LEN);
+ return regval;
+}
+
+static int xlgmac_config_tx_pbl_val(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+ u32 regval;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->tx_ring)
+ break;
+
+ regval = readl(XLGMAC_DMA_REG(channel, DMA_CH_TCR));
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_CH_TCR_PBL_POS,
+ DMA_CH_TCR_PBL_LEN,
+ pdata->tx_pbl);
+ writel(regval, XLGMAC_DMA_REG(channel, DMA_CH_TCR));
+ }
+
+ return 0;
+}
+
+static int xlgmac_get_rx_pbl_val(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = readl(XLGMAC_DMA_REG(pdata->channel_head, DMA_CH_RCR));
+ regval = XLGMAC_GET_REG_BITS(regval, DMA_CH_RCR_PBL_POS,
+ DMA_CH_RCR_PBL_LEN);
+ return regval;
+}
+
+static int xlgmac_config_rx_pbl_val(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+ u32 regval;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->rx_ring)
+ break;
+
+ regval = readl(XLGMAC_DMA_REG(channel, DMA_CH_RCR));
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_CH_RCR_PBL_POS,
+ DMA_CH_RCR_PBL_LEN,
+ pdata->rx_pbl);
+ writel(regval, XLGMAC_DMA_REG(channel, DMA_CH_RCR));
+ }
+
+ return 0;
+}
+
+static u64 xlgmac_mmc_read(struct xlgmac_pdata *pdata, unsigned int reg_lo)
+{
+ bool read_hi;
+ u64 val;
+
+ switch (reg_lo) {
+ /* These registers are always 64 bit */
+ case MMC_TXOCTETCOUNT_GB_LO:
+ case MMC_TXOCTETCOUNT_G_LO:
+ case MMC_RXOCTETCOUNT_GB_LO:
+ case MMC_RXOCTETCOUNT_G_LO:
+ read_hi = true;
+ break;
+
+ default:
+ read_hi = false;
+ }
+
+ val = (u64)readl(pdata->mac_regs + reg_lo);
+
+ if (read_hi)
+ val |= ((u64)readl(pdata->mac_regs + reg_lo + 4) << 32);
+
+ return val;
+}
+
+static void xlgmac_tx_mmc_int(struct xlgmac_pdata *pdata)
+{
+ unsigned int mmc_isr = readl(pdata->mac_regs + MMC_TISR);
+ struct xlgmac_stats *stats = &pdata->stats;
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TXOCTETCOUNT_GB_POS,
+ MMC_TISR_TXOCTETCOUNT_GB_LEN))
+ stats->txoctetcount_gb +=
+ xlgmac_mmc_read(pdata, MMC_TXOCTETCOUNT_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TXFRAMECOUNT_GB_POS,
+ MMC_TISR_TXFRAMECOUNT_GB_LEN))
+ stats->txframecount_gb +=
+ xlgmac_mmc_read(pdata, MMC_TXFRAMECOUNT_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TXBROADCASTFRAMES_G_POS,
+ MMC_TISR_TXBROADCASTFRAMES_G_LEN))
+ stats->txbroadcastframes_g +=
+ xlgmac_mmc_read(pdata, MMC_TXBROADCASTFRAMES_G_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TXMULTICASTFRAMES_G_POS,
+ MMC_TISR_TXMULTICASTFRAMES_G_LEN))
+ stats->txmulticastframes_g +=
+ xlgmac_mmc_read(pdata, MMC_TXMULTICASTFRAMES_G_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TX64OCTETS_GB_POS,
+ MMC_TISR_TX64OCTETS_GB_LEN))
+ stats->tx64octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_TX64OCTETS_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TX65TO127OCTETS_GB_POS,
+ MMC_TISR_TX65TO127OCTETS_GB_LEN))
+ stats->tx65to127octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_TX65TO127OCTETS_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TX128TO255OCTETS_GB_POS,
+ MMC_TISR_TX128TO255OCTETS_GB_LEN))
+ stats->tx128to255octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_TX128TO255OCTETS_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TX256TO511OCTETS_GB_POS,
+ MMC_TISR_TX256TO511OCTETS_GB_LEN))
+ stats->tx256to511octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_TX256TO511OCTETS_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TX512TO1023OCTETS_GB_POS,
+ MMC_TISR_TX512TO1023OCTETS_GB_LEN))
+ stats->tx512to1023octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_TX512TO1023OCTETS_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TX1024TOMAXOCTETS_GB_POS,
+ MMC_TISR_TX1024TOMAXOCTETS_GB_LEN))
+ stats->tx1024tomaxoctets_gb +=
+ xlgmac_mmc_read(pdata, MMC_TX1024TOMAXOCTETS_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TXUNICASTFRAMES_GB_POS,
+ MMC_TISR_TXUNICASTFRAMES_GB_LEN))
+ stats->txunicastframes_gb +=
+ xlgmac_mmc_read(pdata, MMC_TXUNICASTFRAMES_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TXMULTICASTFRAMES_GB_POS,
+ MMC_TISR_TXMULTICASTFRAMES_GB_LEN))
+ stats->txmulticastframes_gb +=
+ xlgmac_mmc_read(pdata, MMC_TXMULTICASTFRAMES_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TXBROADCASTFRAMES_GB_POS,
+ MMC_TISR_TXBROADCASTFRAMES_GB_LEN))
+ stats->txbroadcastframes_g +=
+ xlgmac_mmc_read(pdata, MMC_TXBROADCASTFRAMES_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TXUNDERFLOWERROR_POS,
+ MMC_TISR_TXUNDERFLOWERROR_LEN))
+ stats->txunderflowerror +=
+ xlgmac_mmc_read(pdata, MMC_TXUNDERFLOWERROR_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TXOCTETCOUNT_G_POS,
+ MMC_TISR_TXOCTETCOUNT_G_LEN))
+ stats->txoctetcount_g +=
+ xlgmac_mmc_read(pdata, MMC_TXOCTETCOUNT_G_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TXFRAMECOUNT_G_POS,
+ MMC_TISR_TXFRAMECOUNT_G_LEN))
+ stats->txframecount_g +=
+ xlgmac_mmc_read(pdata, MMC_TXFRAMECOUNT_G_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TXPAUSEFRAMES_POS,
+ MMC_TISR_TXPAUSEFRAMES_LEN))
+ stats->txpauseframes +=
+ xlgmac_mmc_read(pdata, MMC_TXPAUSEFRAMES_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_TISR_TXVLANFRAMES_G_POS,
+ MMC_TISR_TXVLANFRAMES_G_LEN))
+ stats->txvlanframes_g +=
+ xlgmac_mmc_read(pdata, MMC_TXVLANFRAMES_G_LO);
+}
+
+static void xlgmac_rx_mmc_int(struct xlgmac_pdata *pdata)
+{
+ unsigned int mmc_isr = readl(pdata->mac_regs + MMC_RISR);
+ struct xlgmac_stats *stats = &pdata->stats;
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXFRAMECOUNT_GB_POS,
+ MMC_RISR_RXFRAMECOUNT_GB_LEN))
+ stats->rxframecount_gb +=
+ xlgmac_mmc_read(pdata, MMC_RXFRAMECOUNT_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXOCTETCOUNT_GB_POS,
+ MMC_RISR_RXOCTETCOUNT_GB_LEN))
+ stats->rxoctetcount_gb +=
+ xlgmac_mmc_read(pdata, MMC_RXOCTETCOUNT_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXOCTETCOUNT_G_POS,
+ MMC_RISR_RXOCTETCOUNT_G_LEN))
+ stats->rxoctetcount_g +=
+ xlgmac_mmc_read(pdata, MMC_RXOCTETCOUNT_G_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXBROADCASTFRAMES_G_POS,
+ MMC_RISR_RXBROADCASTFRAMES_G_LEN))
+ stats->rxbroadcastframes_g +=
+ xlgmac_mmc_read(pdata, MMC_RXBROADCASTFRAMES_G_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXMULTICASTFRAMES_G_POS,
+ MMC_RISR_RXMULTICASTFRAMES_G_LEN))
+ stats->rxmulticastframes_g +=
+ xlgmac_mmc_read(pdata, MMC_RXMULTICASTFRAMES_G_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXCRCERROR_POS,
+ MMC_RISR_RXCRCERROR_LEN))
+ stats->rxcrcerror +=
+ xlgmac_mmc_read(pdata, MMC_RXCRCERROR_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXRUNTERROR_POS,
+ MMC_RISR_RXRUNTERROR_LEN))
+ stats->rxrunterror +=
+ xlgmac_mmc_read(pdata, MMC_RXRUNTERROR);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXJABBERERROR_POS,
+ MMC_RISR_RXJABBERERROR_LEN))
+ stats->rxjabbererror +=
+ xlgmac_mmc_read(pdata, MMC_RXJABBERERROR);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXUNDERSIZE_G_POS,
+ MMC_RISR_RXUNDERSIZE_G_LEN))
+ stats->rxundersize_g +=
+ xlgmac_mmc_read(pdata, MMC_RXUNDERSIZE_G);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXOVERSIZE_G_POS,
+ MMC_RISR_RXOVERSIZE_G_LEN))
+ stats->rxoversize_g +=
+ xlgmac_mmc_read(pdata, MMC_RXOVERSIZE_G);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RX64OCTETS_GB_POS,
+ MMC_RISR_RX64OCTETS_GB_LEN))
+ stats->rx64octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_RX64OCTETS_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RX65TO127OCTETS_GB_POS,
+ MMC_RISR_RX65TO127OCTETS_GB_LEN))
+ stats->rx65to127octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_RX65TO127OCTETS_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RX128TO255OCTETS_GB_POS,
+ MMC_RISR_RX128TO255OCTETS_GB_LEN))
+ stats->rx128to255octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_RX128TO255OCTETS_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RX256TO511OCTETS_GB_POS,
+ MMC_RISR_RX256TO511OCTETS_GB_LEN))
+ stats->rx256to511octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_RX256TO511OCTETS_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RX512TO1023OCTETS_GB_POS,
+ MMC_RISR_RX512TO1023OCTETS_GB_LEN))
+ stats->rx512to1023octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_RX512TO1023OCTETS_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RX1024TOMAXOCTETS_GB_POS,
+ MMC_RISR_RX1024TOMAXOCTETS_GB_LEN))
+ stats->rx1024tomaxoctets_gb +=
+ xlgmac_mmc_read(pdata, MMC_RX1024TOMAXOCTETS_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXUNICASTFRAMES_G_POS,
+ MMC_RISR_RXUNICASTFRAMES_G_LEN))
+ stats->rxunicastframes_g +=
+ xlgmac_mmc_read(pdata, MMC_RXUNICASTFRAMES_G_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXLENGTHERROR_POS,
+ MMC_RISR_RXLENGTHERROR_LEN))
+ stats->rxlengtherror +=
+ xlgmac_mmc_read(pdata, MMC_RXLENGTHERROR_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXOUTOFRANGETYPE_POS,
+ MMC_RISR_RXOUTOFRANGETYPE_LEN))
+ stats->rxoutofrangetype +=
+ xlgmac_mmc_read(pdata, MMC_RXOUTOFRANGETYPE_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXPAUSEFRAMES_POS,
+ MMC_RISR_RXPAUSEFRAMES_LEN))
+ stats->rxpauseframes +=
+ xlgmac_mmc_read(pdata, MMC_RXPAUSEFRAMES_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXFIFOOVERFLOW_POS,
+ MMC_RISR_RXFIFOOVERFLOW_LEN))
+ stats->rxfifooverflow +=
+ xlgmac_mmc_read(pdata, MMC_RXFIFOOVERFLOW_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXVLANFRAMES_GB_POS,
+ MMC_RISR_RXVLANFRAMES_GB_LEN))
+ stats->rxvlanframes_gb +=
+ xlgmac_mmc_read(pdata, MMC_RXVLANFRAMES_GB_LO);
+
+ if (XLGMAC_GET_REG_BITS(mmc_isr,
+ MMC_RISR_RXWATCHDOGERROR_POS,
+ MMC_RISR_RXWATCHDOGERROR_LEN))
+ stats->rxwatchdogerror +=
+ xlgmac_mmc_read(pdata, MMC_RXWATCHDOGERROR);
+}
+
+static void xlgmac_read_mmc_stats(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_stats *stats = &pdata->stats;
+ u32 regval;
+
+ /* Freeze counters */
+ regval = readl(pdata->mac_regs + MMC_CR);
+ regval = XLGMAC_SET_REG_BITS(regval, MMC_CR_MCF_POS,
+ MMC_CR_MCF_LEN, 1);
+ writel(regval, pdata->mac_regs + MMC_CR);
+
+ stats->txoctetcount_gb +=
+ xlgmac_mmc_read(pdata, MMC_TXOCTETCOUNT_GB_LO);
+
+ stats->txframecount_gb +=
+ xlgmac_mmc_read(pdata, MMC_TXFRAMECOUNT_GB_LO);
+
+ stats->txbroadcastframes_g +=
+ xlgmac_mmc_read(pdata, MMC_TXBROADCASTFRAMES_G_LO);
+
+ stats->txmulticastframes_g +=
+ xlgmac_mmc_read(pdata, MMC_TXMULTICASTFRAMES_G_LO);
+
+ stats->tx64octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_TX64OCTETS_GB_LO);
+
+ stats->tx65to127octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_TX65TO127OCTETS_GB_LO);
+
+ stats->tx128to255octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_TX128TO255OCTETS_GB_LO);
+
+ stats->tx256to511octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_TX256TO511OCTETS_GB_LO);
+
+ stats->tx512to1023octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_TX512TO1023OCTETS_GB_LO);
+
+ stats->tx1024tomaxoctets_gb +=
+ xlgmac_mmc_read(pdata, MMC_TX1024TOMAXOCTETS_GB_LO);
+
+ stats->txunicastframes_gb +=
+ xlgmac_mmc_read(pdata, MMC_TXUNICASTFRAMES_GB_LO);
+
+ stats->txmulticastframes_gb +=
+ xlgmac_mmc_read(pdata, MMC_TXMULTICASTFRAMES_GB_LO);
+
+ stats->txbroadcastframes_g +=
+ xlgmac_mmc_read(pdata, MMC_TXBROADCASTFRAMES_GB_LO);
+
+ stats->txunderflowerror +=
+ xlgmac_mmc_read(pdata, MMC_TXUNDERFLOWERROR_LO);
+
+ stats->txoctetcount_g +=
+ xlgmac_mmc_read(pdata, MMC_TXOCTETCOUNT_G_LO);
+
+ stats->txframecount_g +=
+ xlgmac_mmc_read(pdata, MMC_TXFRAMECOUNT_G_LO);
+
+ stats->txpauseframes +=
+ xlgmac_mmc_read(pdata, MMC_TXPAUSEFRAMES_LO);
+
+ stats->txvlanframes_g +=
+ xlgmac_mmc_read(pdata, MMC_TXVLANFRAMES_G_LO);
+
+ stats->rxframecount_gb +=
+ xlgmac_mmc_read(pdata, MMC_RXFRAMECOUNT_GB_LO);
+
+ stats->rxoctetcount_gb +=
+ xlgmac_mmc_read(pdata, MMC_RXOCTETCOUNT_GB_LO);
+
+ stats->rxoctetcount_g +=
+ xlgmac_mmc_read(pdata, MMC_RXOCTETCOUNT_G_LO);
+
+ stats->rxbroadcastframes_g +=
+ xlgmac_mmc_read(pdata, MMC_RXBROADCASTFRAMES_G_LO);
+
+ stats->rxmulticastframes_g +=
+ xlgmac_mmc_read(pdata, MMC_RXMULTICASTFRAMES_G_LO);
+
+ stats->rxcrcerror +=
+ xlgmac_mmc_read(pdata, MMC_RXCRCERROR_LO);
+
+ stats->rxrunterror +=
+ xlgmac_mmc_read(pdata, MMC_RXRUNTERROR);
+
+ stats->rxjabbererror +=
+ xlgmac_mmc_read(pdata, MMC_RXJABBERERROR);
+
+ stats->rxundersize_g +=
+ xlgmac_mmc_read(pdata, MMC_RXUNDERSIZE_G);
+
+ stats->rxoversize_g +=
+ xlgmac_mmc_read(pdata, MMC_RXOVERSIZE_G);
+
+ stats->rx64octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_RX64OCTETS_GB_LO);
+
+ stats->rx65to127octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_RX65TO127OCTETS_GB_LO);
+
+ stats->rx128to255octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_RX128TO255OCTETS_GB_LO);
+
+ stats->rx256to511octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_RX256TO511OCTETS_GB_LO);
+
+ stats->rx512to1023octets_gb +=
+ xlgmac_mmc_read(pdata, MMC_RX512TO1023OCTETS_GB_LO);
+
+ stats->rx1024tomaxoctets_gb +=
+ xlgmac_mmc_read(pdata, MMC_RX1024TOMAXOCTETS_GB_LO);
+
+ stats->rxunicastframes_g +=
+ xlgmac_mmc_read(pdata, MMC_RXUNICASTFRAMES_G_LO);
+
+ stats->rxlengtherror +=
+ xlgmac_mmc_read(pdata, MMC_RXLENGTHERROR_LO);
+
+ stats->rxoutofrangetype +=
+ xlgmac_mmc_read(pdata, MMC_RXOUTOFRANGETYPE_LO);
+
+ stats->rxpauseframes +=
+ xlgmac_mmc_read(pdata, MMC_RXPAUSEFRAMES_LO);
+
+ stats->rxfifooverflow +=
+ xlgmac_mmc_read(pdata, MMC_RXFIFOOVERFLOW_LO);
+
+ stats->rxvlanframes_gb +=
+ xlgmac_mmc_read(pdata, MMC_RXVLANFRAMES_GB_LO);
+
+ stats->rxwatchdogerror +=
+ xlgmac_mmc_read(pdata, MMC_RXWATCHDOGERROR);
+
+ /* Un-freeze counters */
+ regval = readl(pdata->mac_regs + MMC_CR);
+ regval = XLGMAC_SET_REG_BITS(regval, MMC_CR_MCF_POS,
+ MMC_CR_MCF_LEN, 0);
+ writel(regval, pdata->mac_regs + MMC_CR);
+}
+
+static void xlgmac_config_mmc(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = readl(pdata->mac_regs + MMC_CR);
+ /* Set counters to reset on read */
+ regval = XLGMAC_SET_REG_BITS(regval, MMC_CR_ROR_POS,
+ MMC_CR_ROR_LEN, 1);
+ /* Reset the counters */
+ regval = XLGMAC_SET_REG_BITS(regval, MMC_CR_CR_POS,
+ MMC_CR_CR_LEN, 1);
+ writel(regval, pdata->mac_regs + MMC_CR);
+}
+
+static int xlgmac_write_rss_reg(struct xlgmac_pdata *pdata, unsigned int type,
+ unsigned int index, unsigned int val)
+{
+ unsigned int wait;
+ int ret = 0;
+ u32 regval;
+
+ mutex_lock(&pdata->rss_mutex);
+
+ regval = XLGMAC_GET_REG_BITS(readl(pdata->mac_regs + MAC_RSSAR),
+ MAC_RSSAR_OB_POS, MAC_RSSAR_OB_LEN);
+ if (regval) {
+ ret = -EBUSY;
+ goto unlock;
+ }
+
+ writel(val, pdata->mac_regs + MAC_RSSDR);
+
+ regval = readl(pdata->mac_regs + MAC_RSSAR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RSSAR_RSSIA_POS,
+ MAC_RSSAR_RSSIA_LEN, index);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RSSAR_ADDRT_POS,
+ MAC_RSSAR_ADDRT_LEN, type);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RSSAR_CT_POS,
+ MAC_RSSAR_CT_LEN, 0);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RSSAR_OB_POS,
+ MAC_RSSAR_OB_LEN, 1);
+ writel(regval, pdata->mac_regs + MAC_RSSAR);
+
+ wait = 1000;
+ while (wait--) {
+ regval = XLGMAC_GET_REG_BITS(readl(pdata->mac_regs + MAC_RSSAR),
+ MAC_RSSAR_OB_POS,
+ MAC_RSSAR_OB_LEN);
+ if (!regval)
+ goto unlock;
+
+ usleep_range(1000, 1500);
+ }
+
+ ret = -EBUSY;
+
+unlock:
+ mutex_unlock(&pdata->rss_mutex);
+
+ return ret;
+}
+
+static int xlgmac_write_rss_hash_key(struct xlgmac_pdata *pdata)
+{
+ unsigned int key_regs = sizeof(pdata->rss_key) / sizeof(u32);
+ unsigned int *key = (unsigned int *)&pdata->rss_key;
+ int ret;
+
+ while (key_regs--) {
+ ret = xlgmac_write_rss_reg(pdata, XLGMAC_RSS_HASH_KEY_TYPE,
+ key_regs, *key++);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int xlgmac_write_rss_lookup_table(struct xlgmac_pdata *pdata)
+{
+ unsigned int i;
+ int ret;
+
+ for (i = 0; i < ARRAY_SIZE(pdata->rss_table); i++) {
+ ret = xlgmac_write_rss_reg(pdata,
+ XLGMAC_RSS_LOOKUP_TABLE_TYPE, i,
+ pdata->rss_table[i]);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int xlgmac_set_rss_hash_key(struct xlgmac_pdata *pdata, const u8 *key)
+{
+ memcpy(pdata->rss_key, key, sizeof(pdata->rss_key));
+
+ return xlgmac_write_rss_hash_key(pdata);
+}
+
+static int xlgmac_set_rss_lookup_table(struct xlgmac_pdata *pdata,
+ const u32 *table)
+{
+ unsigned int i;
+ u32 tval;
+
+ for (i = 0; i < ARRAY_SIZE(pdata->rss_table); i++) {
+ tval = table[i];
+ pdata->rss_table[i] = XLGMAC_SET_REG_BITS(
+ pdata->rss_table[i],
+ MAC_RSSDR_DMCH_POS,
+ MAC_RSSDR_DMCH_LEN,
+ tval);
+ }
+
+ return xlgmac_write_rss_lookup_table(pdata);
+}
+
+static int xlgmac_enable_rss(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+ int ret;
+
+ if (!pdata->hw_feat.rss)
+ return -EOPNOTSUPP;
+
+ /* Program the hash key */
+ ret = xlgmac_write_rss_hash_key(pdata);
+ if (ret)
+ return ret;
+
+ /* Program the lookup table */
+ ret = xlgmac_write_rss_lookup_table(pdata);
+ if (ret)
+ return ret;
+
+ /* Set the RSS options */
+ writel(pdata->rss_options, pdata->mac_regs + MAC_RSSCR);
+
+ /* Enable RSS */
+ regval = readl(pdata->mac_regs + MAC_RSSCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RSSCR_RSSE_POS,
+ MAC_RSSCR_RSSE_LEN, 1);
+ writel(regval, pdata->mac_regs + MAC_RSSCR);
+
+ return 0;
+}
+
+static int xlgmac_disable_rss(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ if (!pdata->hw_feat.rss)
+ return -EOPNOTSUPP;
+
+ regval = readl(pdata->mac_regs + MAC_RSSCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_RSSCR_RSSE_POS,
+ MAC_RSSCR_RSSE_LEN, 0);
+ writel(regval, pdata->mac_regs + MAC_RSSCR);
+
+ return 0;
+}
+
+static void xlgmac_config_rss(struct xlgmac_pdata *pdata)
+{
+ int ret;
+
+ if (!pdata->hw_feat.rss)
+ return;
+
+ if (pdata->netdev->features & NETIF_F_RXHASH)
+ ret = xlgmac_enable_rss(pdata);
+ else
+ ret = xlgmac_disable_rss(pdata);
+
+ if (ret)
+ netdev_err(pdata->netdev,
+ "error configuring RSS, RSS disabled\n");
+}
+
+static void xlgmac_enable_dma_interrupts(struct xlgmac_pdata *pdata)
+{
+ unsigned int dma_ch_isr, dma_ch_ier;
+ struct xlgmac_channel *channel;
+ unsigned int i;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ /* Clear all the interrupts which are set */
+ dma_ch_isr = readl(XLGMAC_DMA_REG(channel, DMA_CH_SR));
+ writel(dma_ch_isr, XLGMAC_DMA_REG(channel, DMA_CH_SR));
+
+ /* Clear all interrupt enable bits */
+ dma_ch_ier = 0;
+
+ /* Enable following interrupts
+ * NIE - Normal Interrupt Summary Enable
+ * AIE - Abnormal Interrupt Summary Enable
+ * FBEE - Fatal Bus Error Enable
+ */
+ dma_ch_ier = XLGMAC_SET_REG_BITS(dma_ch_ier,
+ DMA_CH_IER_NIE_POS,
+ DMA_CH_IER_NIE_LEN, 1);
+ dma_ch_ier = XLGMAC_SET_REG_BITS(dma_ch_ier,
+ DMA_CH_IER_AIE_POS,
+ DMA_CH_IER_AIE_LEN, 1);
+ dma_ch_ier = XLGMAC_SET_REG_BITS(dma_ch_ier,
+ DMA_CH_IER_FBEE_POS,
+ DMA_CH_IER_FBEE_LEN, 1);
+
+ if (channel->tx_ring) {
+ /* Enable the following Tx interrupts
+ * TIE - Transmit Interrupt Enable (unless using
+ * per channel interrupts)
+ */
+ if (!pdata->per_channel_irq)
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier,
+ DMA_CH_IER_TIE_POS,
+ DMA_CH_IER_TIE_LEN,
+ 1);
+ }
+ if (channel->rx_ring) {
+ /* Enable following Rx interrupts
+ * RBUE - Receive Buffer Unavailable Enable
+ * RIE - Receive Interrupt Enable (unless using
+ * per channel interrupts)
+ */
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier,
+ DMA_CH_IER_RBUE_POS,
+ DMA_CH_IER_RBUE_LEN,
+ 1);
+ if (!pdata->per_channel_irq)
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier,
+ DMA_CH_IER_RIE_POS,
+ DMA_CH_IER_RIE_LEN,
+ 1);
+ }
+
+ writel(dma_ch_isr, XLGMAC_DMA_REG(channel, DMA_CH_IER));
+ }
+}
+
+static void xlgmac_enable_mtl_interrupts(struct xlgmac_pdata *pdata)
+{
+ unsigned int q_count, i;
+ unsigned int mtl_q_isr;
+
+ q_count = max(pdata->hw_feat.tx_q_cnt, pdata->hw_feat.rx_q_cnt);
+ for (i = 0; i < q_count; i++) {
+ /* Clear all the interrupts which are set */
+ mtl_q_isr = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_ISR));
+ writel(mtl_q_isr, XLGMAC_MTL_REG(pdata, i, MTL_Q_ISR));
+
+ /* No MTL interrupts to be enabled */
+ writel(0, XLGMAC_MTL_REG(pdata, i, MTL_Q_IER));
+ }
+}
+
+static void xlgmac_enable_mac_interrupts(struct xlgmac_pdata *pdata)
+{
+ unsigned int mac_ier = 0;
+ u32 regval;
+
+ /* Enable Timestamp interrupt */
+ mac_ier = XLGMAC_SET_REG_BITS(mac_ier, MAC_IER_TSIE_POS,
+ MAC_IER_TSIE_LEN, 1);
+
+ writel(mac_ier, pdata->mac_regs + MAC_IER);
+
+ /* Enable all counter interrupts */
+ regval = readl(pdata->mac_regs + MMC_RIER);
+ regval = XLGMAC_SET_REG_BITS(regval, MMC_RIER_ALL_INTERRUPTS_POS,
+ MMC_RIER_ALL_INTERRUPTS_LEN, 0xffffffff);
+ writel(regval, pdata->mac_regs + MMC_RIER);
+ regval = readl(pdata->mac_regs + MMC_TIER);
+ regval = XLGMAC_SET_REG_BITS(regval, MMC_TIER_ALL_INTERRUPTS_POS,
+ MMC_TIER_ALL_INTERRUPTS_LEN, 0xffffffff);
+ writel(regval, pdata->mac_regs + MMC_TIER);
+}
+
+static int xlgmac_set_xlgmii_25000_speed(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = XLGMAC_GET_REG_BITS(readl(pdata->mac_regs + MAC_TCR),
+ MAC_TCR_SS_POS, MAC_TCR_SS_LEN);
+ if (regval == 0x1)
+ return 0;
+
+ regval = readl(pdata->mac_regs + MAC_TCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_TCR_SS_POS,
+ MAC_TCR_SS_LEN, 0x1);
+ writel(regval, pdata->mac_regs + MAC_TCR);
+
+ return 0;
+}
+
+static int xlgmac_set_xlgmii_40000_speed(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = XLGMAC_GET_REG_BITS(readl(pdata->mac_regs + MAC_TCR),
+ MAC_TCR_SS_POS, MAC_TCR_SS_LEN);
+ if (regval == 0)
+ return 0;
+
+ regval = readl(pdata->mac_regs + MAC_TCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_TCR_SS_POS,
+ MAC_TCR_SS_LEN, 0);
+ writel(regval, pdata->mac_regs + MAC_TCR);
+
+ return 0;
+}
+
+static int xlgmac_set_xlgmii_50000_speed(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = XLGMAC_GET_REG_BITS(readl(pdata->mac_regs + MAC_TCR),
+ MAC_TCR_SS_POS, MAC_TCR_SS_LEN);
+ if (regval == 0x2)
+ return 0;
+
+ regval = readl(pdata->mac_regs + MAC_TCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_TCR_SS_POS,
+ MAC_TCR_SS_LEN, 0x2);
+ writel(regval, pdata->mac_regs + MAC_TCR);
+
+ return 0;
+}
+
+static int xlgmac_set_xlgmii_100000_speed(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = XLGMAC_GET_REG_BITS(readl(pdata->mac_regs + MAC_TCR),
+ MAC_TCR_SS_POS, MAC_TCR_SS_LEN);
+ if (regval == 0x3)
+ return 0;
+
+ regval = readl(pdata->mac_regs + MAC_TCR);
+ regval = XLGMAC_SET_REG_BITS(regval, MAC_TCR_SS_POS,
+ MAC_TCR_SS_LEN, 0x3);
+ writel(regval, pdata->mac_regs + MAC_TCR);
+
+ return 0;
+}
+
+static void xlgmac_config_mac_speed(struct xlgmac_pdata *pdata)
+{
+ switch (pdata->phy_speed) {
+ case SPEED_100000:
+ xlgmac_set_xlgmii_100000_speed(pdata);
+ break;
+
+ case SPEED_50000:
+ xlgmac_set_xlgmii_50000_speed(pdata);
+ break;
+
+ case SPEED_40000:
+ xlgmac_set_xlgmii_40000_speed(pdata);
+ break;
+
+ case SPEED_25000:
+ xlgmac_set_xlgmii_25000_speed(pdata);
+ break;
+ }
+}
+
+static int xlgmac_dev_read(struct xlgmac_channel *channel)
+{
+ struct xlgmac_pdata *pdata = channel->pdata;
+ struct xlgmac_ring *ring = channel->rx_ring;
+ struct net_device *netdev = pdata->netdev;
+ struct xlgmac_desc_data *desc_data;
+ struct xlgmac_dma_desc *dma_desc;
+ struct xlgmac_pkt_info *pkt_info;
+ unsigned int err, etlt, l34t;
+
+ desc_data = XLGMAC_GET_DESC_DATA(ring, ring->cur);
+ dma_desc = desc_data->dma_desc;
+ pkt_info = &ring->pkt_info;
+
+ /* Check for data availability */
+ if (XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ RX_NORMAL_DESC3_OWN_POS,
+ RX_NORMAL_DESC3_OWN_LEN))
+ return 1;
+
+ /* Make sure descriptor fields are read after reading the OWN bit */
+ dma_rmb();
+
+ if (netif_msg_rx_status(pdata))
+ xlgmac_dump_rx_desc(pdata, ring, ring->cur);
+
+ if (XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ RX_NORMAL_DESC3_CTXT_POS,
+ RX_NORMAL_DESC3_CTXT_LEN)) {
+ /* Timestamp Context Descriptor */
+ xlgmac_get_rx_tstamp(pkt_info, dma_desc);
+
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_CONTEXT_POS,
+ RX_PACKET_ATTRIBUTES_CONTEXT_LEN,
+ 1);
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_CONTEXT_NEXT_POS,
+ RX_PACKET_ATTRIBUTES_CONTEXT_NEXT_LEN,
+ 0);
+ return 0;
+ }
+
+ /* Normal Descriptor, be sure Context Descriptor bit is off */
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_CONTEXT_POS,
+ RX_PACKET_ATTRIBUTES_CONTEXT_LEN,
+ 0);
+
+ /* Indicate if a Context Descriptor is next */
+ if (XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ RX_NORMAL_DESC3_CDA_POS,
+ RX_NORMAL_DESC3_CDA_LEN))
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_CONTEXT_NEXT_POS,
+ RX_PACKET_ATTRIBUTES_CONTEXT_NEXT_LEN,
+ 1);
+
+ /* Get the header length */
+ if (XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ RX_NORMAL_DESC3_FD_POS,
+ RX_NORMAL_DESC3_FD_LEN)) {
+ desc_data->rx.hdr_len = XLGMAC_GET_REG_BITS_LE(dma_desc->desc2,
+ RX_NORMAL_DESC2_HL_POS,
+ RX_NORMAL_DESC2_HL_LEN);
+ if (desc_data->rx.hdr_len)
+ pdata->stats.rx_split_header_packets++;
+ }
+
+ /* Get the RSS hash */
+ if (XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ RX_NORMAL_DESC3_RSV_POS,
+ RX_NORMAL_DESC3_RSV_LEN)) {
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_RSS_HASH_POS,
+ RX_PACKET_ATTRIBUTES_RSS_HASH_LEN,
+ 1);
+
+ pkt_info->rss_hash = le32_to_cpu(dma_desc->desc1);
+
+ l34t = XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ RX_NORMAL_DESC3_L34T_POS,
+ RX_NORMAL_DESC3_L34T_LEN);
+ switch (l34t) {
+ case RX_DESC3_L34T_IPV4_TCP:
+ case RX_DESC3_L34T_IPV4_UDP:
+ case RX_DESC3_L34T_IPV6_TCP:
+ case RX_DESC3_L34T_IPV6_UDP:
+ pkt_info->rss_hash_type = PKT_HASH_TYPE_L4;
+ break;
+ default:
+ pkt_info->rss_hash_type = PKT_HASH_TYPE_L3;
+ }
+ }
+
+ /* Get the pkt_info length */
+ desc_data->rx.len = XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ RX_NORMAL_DESC3_PL_POS,
+ RX_NORMAL_DESC3_PL_LEN);
+
+ if (!XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ RX_NORMAL_DESC3_LD_POS,
+ RX_NORMAL_DESC3_LD_LEN)) {
+ /* Not all the data has been transferred for this pkt_info */
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_INCOMPLETE_POS,
+ RX_PACKET_ATTRIBUTES_INCOMPLETE_LEN,
+ 1);
+ return 0;
+ }
+
+ /* This is the last of the data for this pkt_info */
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_INCOMPLETE_POS,
+ RX_PACKET_ATTRIBUTES_INCOMPLETE_LEN,
+ 0);
+
+ /* Set checksum done indicator as appropriate */
+ if (netdev->features & NETIF_F_RXCSUM)
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_CSUM_DONE_POS,
+ RX_PACKET_ATTRIBUTES_CSUM_DONE_LEN,
+ 1);
+
+ /* Check for errors (only valid in last descriptor) */
+ err = XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ RX_NORMAL_DESC3_ES_POS,
+ RX_NORMAL_DESC3_ES_LEN);
+ etlt = XLGMAC_GET_REG_BITS_LE(dma_desc->desc3,
+ RX_NORMAL_DESC3_ETLT_POS,
+ RX_NORMAL_DESC3_ETLT_LEN);
+ netif_dbg(pdata, rx_status, netdev, "err=%u, etlt=%#x\n", err, etlt);
+
+ if (!err || !etlt) {
+ /* No error if err is 0 or etlt is 0 */
+ if ((etlt == 0x09) &&
+ (netdev->features & NETIF_F_HW_VLAN_CTAG_RX)) {
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_VLAN_CTAG_POS,
+ RX_PACKET_ATTRIBUTES_VLAN_CTAG_LEN,
+ 1);
+ pkt_info->vlan_ctag =
+ XLGMAC_GET_REG_BITS_LE(dma_desc->desc0,
+ RX_NORMAL_DESC0_OVT_POS,
+ RX_NORMAL_DESC0_OVT_LEN);
+ netif_dbg(pdata, rx_status, netdev, "vlan-ctag=%#06x\n",
+ pkt_info->vlan_ctag);
+ }
+ } else {
+ if ((etlt == 0x05) || (etlt == 0x06))
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_CSUM_DONE_POS,
+ RX_PACKET_ATTRIBUTES_CSUM_DONE_LEN,
+ 0);
+ else
+ pkt_info->errors = XLGMAC_SET_REG_BITS(
+ pkt_info->errors,
+ RX_PACKET_ERRORS_FRAME_POS,
+ RX_PACKET_ERRORS_FRAME_LEN,
+ 1);
+ }
+
+ XLGMAC_PR("%s - descriptor=%u (cur=%d)\n", channel->name,
+ ring->cur & (ring->dma_desc_count - 1), ring->cur);
+
+ return 0;
+}
+
+static int xlgmac_enable_int(struct xlgmac_channel *channel,
+ enum xlgmac_int int_id)
+{
+ unsigned int dma_ch_ier;
+
+ dma_ch_ier = readl(XLGMAC_DMA_REG(channel, DMA_CH_IER));
+
+ switch (int_id) {
+ case XLGMAC_INT_DMA_CH_SR_TI:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_TIE_POS,
+ DMA_CH_IER_TIE_LEN, 1);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_TPS:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_TXSE_POS,
+ DMA_CH_IER_TXSE_LEN, 1);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_TBU:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_TBUE_POS,
+ DMA_CH_IER_TBUE_LEN, 1);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_RI:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_RIE_POS,
+ DMA_CH_IER_RIE_LEN, 1);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_RBU:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_RBUE_POS,
+ DMA_CH_IER_RBUE_LEN, 1);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_RPS:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_RSE_POS,
+ DMA_CH_IER_RSE_LEN, 1);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_TI_RI:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_TIE_POS,
+ DMA_CH_IER_TIE_LEN, 1);
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_RIE_POS,
+ DMA_CH_IER_RIE_LEN, 1);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_FBE:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_FBEE_POS,
+ DMA_CH_IER_FBEE_LEN, 1);
+ break;
+ case XLGMAC_INT_DMA_ALL:
+ dma_ch_ier |= channel->saved_ier;
+ break;
+ default:
+ return -1;
+ }
+
+ writel(dma_ch_ier, XLGMAC_DMA_REG(channel, DMA_CH_IER));
+
+ return 0;
+}
+
+static int xlgmac_disable_int(struct xlgmac_channel *channel,
+ enum xlgmac_int int_id)
+{
+ unsigned int dma_ch_ier;
+
+ dma_ch_ier = readl(XLGMAC_DMA_REG(channel, DMA_CH_IER));
+
+ switch (int_id) {
+ case XLGMAC_INT_DMA_CH_SR_TI:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_TIE_POS,
+ DMA_CH_IER_TIE_LEN, 0);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_TPS:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_TXSE_POS,
+ DMA_CH_IER_TXSE_LEN, 0);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_TBU:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_TBUE_POS,
+ DMA_CH_IER_TBUE_LEN, 0);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_RI:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_RIE_POS,
+ DMA_CH_IER_RIE_LEN, 0);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_RBU:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_RBUE_POS,
+ DMA_CH_IER_RBUE_LEN, 0);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_RPS:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_RSE_POS,
+ DMA_CH_IER_RSE_LEN, 0);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_TI_RI:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_TIE_POS,
+ DMA_CH_IER_TIE_LEN, 0);
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_RIE_POS,
+ DMA_CH_IER_RIE_LEN, 0);
+ break;
+ case XLGMAC_INT_DMA_CH_SR_FBE:
+ dma_ch_ier = XLGMAC_SET_REG_BITS(
+ dma_ch_ier, DMA_CH_IER_FBEE_POS,
+ DMA_CH_IER_FBEE_LEN, 0);
+ break;
+ case XLGMAC_INT_DMA_ALL:
+ channel->saved_ier = dma_ch_ier & XLGMAC_DMA_INTERRUPT_MASK;
+ dma_ch_ier &= ~XLGMAC_DMA_INTERRUPT_MASK;
+ break;
+ default:
+ return -1;
+ }
+
+ writel(dma_ch_ier, XLGMAC_DMA_REG(channel, DMA_CH_IER));
+
+ return 0;
+}
+
+static int xlgmac_flush_tx_queues(struct xlgmac_pdata *pdata)
+{
+ unsigned int i, count;
+ u32 regval;
+
+ for (i = 0; i < pdata->tx_q_count; i++) {
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_TQOMR));
+ regval = XLGMAC_SET_REG_BITS(regval, MTL_Q_TQOMR_FTQ_POS,
+ MTL_Q_TQOMR_FTQ_LEN, 1);
+ writel(regval, XLGMAC_MTL_REG(pdata, i, MTL_Q_TQOMR));
+ }
+
+ /* Poll Until Poll Condition */
+ for (i = 0; i < pdata->tx_q_count; i++) {
+ count = 2000;
+ regval = readl(XLGMAC_MTL_REG(pdata, i, MTL_Q_TQOMR));
+ regval = XLGMAC_GET_REG_BITS(regval, MTL_Q_TQOMR_FTQ_POS,
+ MTL_Q_TQOMR_FTQ_LEN);
+ while (--count && regval)
+ usleep_range(500, 600);
+
+ if (!count)
+ return -EBUSY;
+ }
+
+ return 0;
+}
+
+static void xlgmac_config_dma_bus(struct xlgmac_pdata *pdata)
+{
+ u32 regval;
+
+ regval = readl(pdata->mac_regs + DMA_SBMR);
+ /* Set enhanced addressing mode */
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_SBMR_EAME_POS,
+ DMA_SBMR_EAME_LEN, 1);
+ /* Set the System Bus mode */
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_SBMR_UNDEF_POS,
+ DMA_SBMR_UNDEF_LEN, 1);
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_SBMR_BLEN_256_POS,
+ DMA_SBMR_BLEN_256_LEN, 1);
+ writel(regval, pdata->mac_regs + DMA_SBMR);
+}
+
+static int xlgmac_hw_init(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_desc_ops *desc_ops = &pdata->desc_ops;
+ int ret;
+
+ /* Flush Tx queues */
+ ret = xlgmac_flush_tx_queues(pdata);
+ if (ret)
+ return ret;
+
+ /* Initialize DMA related features */
+ xlgmac_config_dma_bus(pdata);
+ xlgmac_config_osp_mode(pdata);
+ xlgmac_config_pblx8(pdata);
+ xlgmac_config_tx_pbl_val(pdata);
+ xlgmac_config_rx_pbl_val(pdata);
+ xlgmac_config_rx_coalesce(pdata);
+ xlgmac_config_tx_coalesce(pdata);
+ xlgmac_config_rx_buffer_size(pdata);
+ xlgmac_config_tso_mode(pdata);
+ xlgmac_config_sph_mode(pdata);
+ xlgmac_config_rss(pdata);
+ desc_ops->tx_desc_init(pdata);
+ desc_ops->rx_desc_init(pdata);
+ xlgmac_enable_dma_interrupts(pdata);
+
+ /* Initialize MTL related features */
+ xlgmac_config_mtl_mode(pdata);
+ xlgmac_config_queue_mapping(pdata);
+ xlgmac_config_tsf_mode(pdata, pdata->tx_sf_mode);
+ xlgmac_config_rsf_mode(pdata, pdata->rx_sf_mode);
+ xlgmac_config_tx_threshold(pdata, pdata->tx_threshold);
+ xlgmac_config_rx_threshold(pdata, pdata->rx_threshold);
+ xlgmac_config_tx_fifo_size(pdata);
+ xlgmac_config_rx_fifo_size(pdata);
+ xlgmac_config_flow_control_threshold(pdata);
+ xlgmac_config_rx_fep_enable(pdata);
+ xlgmac_config_rx_fup_enable(pdata);
+ xlgmac_enable_mtl_interrupts(pdata);
+
+ /* Initialize MAC related features */
+ xlgmac_config_mac_address(pdata);
+ xlgmac_config_rx_mode(pdata);
+ xlgmac_config_jumbo_enable(pdata);
+ xlgmac_config_flow_control(pdata);
+ xlgmac_config_mac_speed(pdata);
+ xlgmac_config_checksum_offload(pdata);
+ xlgmac_config_vlan_support(pdata);
+ xlgmac_config_mmc(pdata);
+ xlgmac_enable_mac_interrupts(pdata);
+
+ return 0;
+}
+
+static int xlgmac_hw_exit(struct xlgmac_pdata *pdata)
+{
+ unsigned int count = 2000;
+ u32 regval;
+
+ /* Issue a software reset */
+ regval = readl(pdata->mac_regs + DMA_MR);
+ regval = XLGMAC_SET_REG_BITS(regval, DMA_MR_SWR_POS,
+ DMA_MR_SWR_LEN, 1);
+ writel(regval, pdata->mac_regs + DMA_MR);
+ usleep_range(10, 15);
+
+ /* Poll Until Poll Condition */
+ while (--count &&
+ XLGMAC_GET_REG_BITS(readl(pdata->mac_regs + DMA_MR),
+ DMA_MR_SWR_POS, DMA_MR_SWR_LEN))
+ usleep_range(500, 600);
+
+ if (!count)
+ return -EBUSY;
+
+ return 0;
+}
+
+void xlgmac_init_hw_ops(struct xlgmac_hw_ops *hw_ops)
+{
+ hw_ops->init = xlgmac_hw_init;
+ hw_ops->exit = xlgmac_hw_exit;
+
+ hw_ops->tx_complete = xlgmac_tx_complete;
+
+ hw_ops->enable_tx = xlgmac_enable_tx;
+ hw_ops->disable_tx = xlgmac_disable_tx;
+ hw_ops->enable_rx = xlgmac_enable_rx;
+ hw_ops->disable_rx = xlgmac_disable_rx;
+
+ hw_ops->dev_xmit = xlgmac_dev_xmit;
+ hw_ops->dev_read = xlgmac_dev_read;
+ hw_ops->enable_int = xlgmac_enable_int;
+ hw_ops->disable_int = xlgmac_disable_int;
+
+ hw_ops->set_mac_address = xlgmac_set_mac_address;
+ hw_ops->config_rx_mode = xlgmac_config_rx_mode;
+ hw_ops->enable_rx_csum = xlgmac_enable_rx_csum;
+ hw_ops->disable_rx_csum = xlgmac_disable_rx_csum;
+
+ /* For MII speed configuration */
+ hw_ops->set_xlgmii_25000_speed = xlgmac_set_xlgmii_25000_speed;
+ hw_ops->set_xlgmii_40000_speed = xlgmac_set_xlgmii_40000_speed;
+ hw_ops->set_xlgmii_50000_speed = xlgmac_set_xlgmii_50000_speed;
+ hw_ops->set_xlgmii_100000_speed = xlgmac_set_xlgmii_100000_speed;
+
+ /* For descriptor related operation */
+ hw_ops->tx_desc_init = xlgmac_tx_desc_init;
+ hw_ops->rx_desc_init = xlgmac_rx_desc_init;
+ hw_ops->tx_desc_reset = xlgmac_tx_desc_reset;
+ hw_ops->rx_desc_reset = xlgmac_rx_desc_reset;
+ hw_ops->is_last_desc = xlgmac_is_last_desc;
+ hw_ops->is_context_desc = xlgmac_is_context_desc;
+ hw_ops->tx_start_xmit = xlgmac_tx_start_xmit;
+
+ /* For Flow Control */
+ hw_ops->config_tx_flow_control = xlgmac_config_tx_flow_control;
+ hw_ops->config_rx_flow_control = xlgmac_config_rx_flow_control;
+
+ /* For Vlan related config */
+ hw_ops->enable_rx_vlan_stripping = xlgmac_enable_rx_vlan_stripping;
+ hw_ops->disable_rx_vlan_stripping = xlgmac_disable_rx_vlan_stripping;
+ hw_ops->enable_rx_vlan_filtering = xlgmac_enable_rx_vlan_filtering;
+ hw_ops->disable_rx_vlan_filtering = xlgmac_disable_rx_vlan_filtering;
+ hw_ops->update_vlan_hash_table = xlgmac_update_vlan_hash_table;
+
+ /* For RX coalescing */
+ hw_ops->config_rx_coalesce = xlgmac_config_rx_coalesce;
+ hw_ops->config_tx_coalesce = xlgmac_config_tx_coalesce;
+ hw_ops->usec_to_riwt = xlgmac_usec_to_riwt;
+ hw_ops->riwt_to_usec = xlgmac_riwt_to_usec;
+
+ /* For RX and TX threshold config */
+ hw_ops->config_rx_threshold = xlgmac_config_rx_threshold;
+ hw_ops->config_tx_threshold = xlgmac_config_tx_threshold;
+
+ /* For RX and TX Store and Forward Mode config */
+ hw_ops->config_rsf_mode = xlgmac_config_rsf_mode;
+ hw_ops->config_tsf_mode = xlgmac_config_tsf_mode;
+
+ /* For TX DMA Operating on Second Frame config */
+ hw_ops->config_osp_mode = xlgmac_config_osp_mode;
+
+ /* For RX and TX PBL config */
+ hw_ops->config_rx_pbl_val = xlgmac_config_rx_pbl_val;
+ hw_ops->get_rx_pbl_val = xlgmac_get_rx_pbl_val;
+ hw_ops->config_tx_pbl_val = xlgmac_config_tx_pbl_val;
+ hw_ops->get_tx_pbl_val = xlgmac_get_tx_pbl_val;
+ hw_ops->config_pblx8 = xlgmac_config_pblx8;
+
+ /* For MMC statistics support */
+ hw_ops->tx_mmc_int = xlgmac_tx_mmc_int;
+ hw_ops->rx_mmc_int = xlgmac_rx_mmc_int;
+ hw_ops->read_mmc_stats = xlgmac_read_mmc_stats;
+
+ /* For Receive Side Scaling */
+ hw_ops->enable_rss = xlgmac_enable_rss;
+ hw_ops->disable_rss = xlgmac_disable_rss;
+ hw_ops->set_rss_hash_key = xlgmac_set_rss_hash_key;
+ hw_ops->set_rss_lookup_table = xlgmac_set_rss_lookup_table;
+}
diff --git a/drivers/net/ethernet/synopsys/dwc-xlgmac-net.c b/drivers/net/ethernet/synopsys/dwc-xlgmac-net.c
new file mode 100644
index 0000000..3b91257
--- /dev/null
+++ b/drivers/net/ethernet/synopsys/dwc-xlgmac-net.c
@@ -0,0 +1,1350 @@
+/* Synopsys DesignWare Core Enterprise Ethernet (XLGMAC) Driver
+ *
+ * Copyright (c) 2017 Synopsys, Inc. (www.synopsys.com)
+ *
+ * This program is dual-licensed; you may select either version 2 of
+ * the GNU General Public License ("GPL") or BSD license ("BSD").
+ *
+ * This Synopsys DWC XLGMAC software driver and associated documentation
+ * (hereinafter the "Software") is an unsupported proprietary work of
+ * Synopsys, Inc. unless otherwise expressly agreed to in writing between
+ * Synopsys and you. The Software IS NOT an item of Licensed Software or a
+ * Licensed Product under any End User Software License Agreement or
+ * Agreement for Licensed Products with Synopsys or any supplement thereto.
+ * Synopsys is a registered trademark of Synopsys, Inc. Other names included
+ * in the SOFTWARE may be the trademarks of their respective owners.
+ */
+
+#include <linux/netdevice.h>
+#include <linux/tcp.h>
+
+#include "dwc-xlgmac.h"
+#include "dwc-xlgmac-reg.h"
+
+static int xlgmac_one_poll(struct napi_struct *, int);
+static int xlgmac_all_poll(struct napi_struct *, int);
+
+static inline unsigned int xlgmac_tx_avail_desc(struct xlgmac_ring *ring)
+{
+ return (ring->dma_desc_count - (ring->cur - ring->dirty));
+}
+
+static inline unsigned int xlgmac_rx_dirty_desc(struct xlgmac_ring *ring)
+{
+ return (ring->cur - ring->dirty);
+}
+
+static int xlgmac_maybe_stop_tx_queue(
+ struct xlgmac_channel *channel,
+ struct xlgmac_ring *ring,
+ unsigned int count)
+{
+ struct xlgmac_pdata *pdata = channel->pdata;
+
+ if (count > xlgmac_tx_avail_desc(ring)) {
+ netif_info(pdata, drv, pdata->netdev,
+ "Tx queue stopped, not enough descriptors available\n");
+ netif_stop_subqueue(pdata->netdev, channel->queue_index);
+ ring->tx.queue_stopped = 1;
+
+ /* If we haven't notified the hardware because of xmit_more
+ * support, tell it now
+ */
+ if (ring->tx.xmit_more)
+ pdata->hw_ops.tx_start_xmit(channel, ring);
+
+ return NETDEV_TX_BUSY;
+ }
+
+ return 0;
+}
+
+static void xlgmac_prep_vlan(struct sk_buff *skb,
+ struct xlgmac_pkt_info *pkt_info)
+{
+ if (skb_vlan_tag_present(skb))
+ pkt_info->vlan_ctag = skb_vlan_tag_get(skb);
+}
+
+static int xlgmac_prep_tso(struct sk_buff *skb,
+ struct xlgmac_pkt_info *pkt_info)
+{
+ int ret;
+
+ if (!XLGMAC_GET_REG_BITS(pkt_info->attributes,
+ TX_PACKET_ATTRIBUTES_TSO_ENABLE_POS,
+ TX_PACKET_ATTRIBUTES_TSO_ENABLE_LEN))
+ return 0;
+
+ ret = skb_cow_head(skb, 0);
+ if (ret)
+ return ret;
+
+ pkt_info->header_len = skb_transport_offset(skb) + tcp_hdrlen(skb);
+ pkt_info->tcp_header_len = tcp_hdrlen(skb);
+ pkt_info->tcp_payload_len = skb->len - pkt_info->header_len;
+ pkt_info->mss = skb_shinfo(skb)->gso_size;
+
+ XLGMAC_PR("header_len=%u\n", pkt_info->header_len);
+ XLGMAC_PR("tcp_header_len=%u, tcp_payload_len=%u\n",
+ pkt_info->tcp_header_len, pkt_info->tcp_payload_len);
+ XLGMAC_PR("mss=%u\n", pkt_info->mss);
+
+ /* Update the number of packets that will ultimately be transmitted
+ * along with the extra bytes for each extra packet
+ */
+ pkt_info->tx_packets = skb_shinfo(skb)->gso_segs;
+ pkt_info->tx_bytes += (pkt_info->tx_packets - 1) * pkt_info->header_len;
+
+ return 0;
+}
+
+static int xlgmac_is_tso(struct sk_buff *skb)
+{
+ if (skb->ip_summed != CHECKSUM_PARTIAL)
+ return 0;
+
+ if (!skb_is_gso(skb))
+ return 0;
+
+ return 1;
+}
+
+static void xlgmac_prep_tx_pkt(struct xlgmac_pdata *pdata,
+ struct xlgmac_ring *ring,
+ struct sk_buff *skb,
+ struct xlgmac_pkt_info *pkt_info)
+{
+ struct skb_frag_struct *frag;
+ unsigned int context_desc;
+ unsigned int len;
+ unsigned int i;
+
+ pkt_info->skb = skb;
+
+ context_desc = 0;
+ pkt_info->desc_count = 0;
+
+ pkt_info->tx_packets = 1;
+ pkt_info->tx_bytes = skb->len;
+
+ if (xlgmac_is_tso(skb)) {
+ /* TSO requires an extra descriptor if mss is different */
+ if (skb_shinfo(skb)->gso_size != ring->tx.cur_mss) {
+ context_desc = 1;
+ pkt_info->desc_count++;
+ }
+
+ /* TSO requires an extra descriptor for TSO header */
+ pkt_info->desc_count++;
+
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ TX_PACKET_ATTRIBUTES_TSO_ENABLE_POS,
+ TX_PACKET_ATTRIBUTES_TSO_ENABLE_LEN,
+ 1);
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ TX_PACKET_ATTRIBUTES_CSUM_ENABLE_POS,
+ TX_PACKET_ATTRIBUTES_CSUM_ENABLE_LEN,
+ 1);
+ } else if (skb->ip_summed == CHECKSUM_PARTIAL)
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ TX_PACKET_ATTRIBUTES_CSUM_ENABLE_POS,
+ TX_PACKET_ATTRIBUTES_CSUM_ENABLE_LEN,
+ 1);
+
+ if (skb_vlan_tag_present(skb)) {
+ /* VLAN requires an extra descriptor if tag is different */
+ if (skb_vlan_tag_get(skb) != ring->tx.cur_vlan_ctag)
+ /* We can share with the TSO context descriptor */
+ if (!context_desc) {
+ context_desc = 1;
+ pkt_info->desc_count++;
+ }
+
+ pkt_info->attributes = XLGMAC_SET_REG_BITS(
+ pkt_info->attributes,
+ TX_PACKET_ATTRIBUTES_VLAN_CTAG_POS,
+ TX_PACKET_ATTRIBUTES_VLAN_CTAG_LEN,
+ 1);
+ }
+
+ for (len = skb_headlen(skb); len;) {
+ pkt_info->desc_count++;
+ len -= min_t(unsigned int, len, XLGMAC_TX_MAX_BUF_SIZE);
+ }
+
+ for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
+ frag = &skb_shinfo(skb)->frags[i];
+ for (len = skb_frag_size(frag); len; ) {
+ pkt_info->desc_count++;
+ len -= min_t(unsigned int, len, XLGMAC_TX_MAX_BUF_SIZE);
+ }
+ }
+}
+
+static int xlgmac_calc_rx_buf_size(struct net_device *netdev, unsigned int mtu)
+{
+ unsigned int rx_buf_size;
+
+ if (mtu > XLGMAC_JUMBO_PACKET_MTU) {
+ netdev_alert(netdev, "MTU exceeds maximum supported value\n");
+ return -EINVAL;
+ }
+
+ rx_buf_size = mtu + ETH_HLEN + ETH_FCS_LEN + VLAN_HLEN;
+ rx_buf_size = clamp_val(rx_buf_size, XLGMAC_RX_MIN_BUF_SIZE, PAGE_SIZE);
+
+ rx_buf_size = (rx_buf_size + XLGMAC_RX_BUF_ALIGN - 1) &
+ ~(XLGMAC_RX_BUF_ALIGN - 1);
+
+ return rx_buf_size;
+}
+
+static void xlgmac_enable_rx_tx_ints(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_hw_ops *hw_ops = &pdata->hw_ops;
+ struct xlgmac_channel *channel;
+ enum xlgmac_int int_id;
+ unsigned int i;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (channel->tx_ring && channel->rx_ring)
+ int_id = XLGMAC_INT_DMA_CH_SR_TI_RI;
+ else if (channel->tx_ring)
+ int_id = XLGMAC_INT_DMA_CH_SR_TI;
+ else if (channel->rx_ring)
+ int_id = XLGMAC_INT_DMA_CH_SR_RI;
+ else
+ continue;
+
+ hw_ops->enable_int(channel, int_id);
+ }
+}
+
+static void xlgmac_disable_rx_tx_ints(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_hw_ops *hw_ops = &pdata->hw_ops;
+ struct xlgmac_channel *channel;
+ enum xlgmac_int int_id;
+ unsigned int i;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (channel->tx_ring && channel->rx_ring)
+ int_id = XLGMAC_INT_DMA_CH_SR_TI_RI;
+ else if (channel->tx_ring)
+ int_id = XLGMAC_INT_DMA_CH_SR_TI;
+ else if (channel->rx_ring)
+ int_id = XLGMAC_INT_DMA_CH_SR_RI;
+ else
+ continue;
+
+ hw_ops->disable_int(channel, int_id);
+ }
+}
+
+static irqreturn_t xlgmac_isr(int irq, void *data)
+{
+ unsigned int dma_isr, dma_ch_isr, mac_isr;
+ struct xlgmac_pdata *pdata = data;
+ struct xlgmac_channel *channel;
+ struct xlgmac_hw_ops *hw_ops;
+ unsigned int i, ti, ri;
+
+ hw_ops = &pdata->hw_ops;
+
+ /* The DMA interrupt status register also reports MAC and MTL
+ * interrupts. So for polling mode, we just need to check for
+ * this register to be non-zero
+ */
+ dma_isr = readl(pdata->mac_regs + DMA_ISR);
+ if (!dma_isr)
+ return IRQ_HANDLED;
+
+ netif_dbg(pdata, intr, pdata->netdev, "DMA_ISR=%#010x\n", dma_isr);
+
+ for (i = 0; i < pdata->channel_count; i++) {
+ if (!(dma_isr & (1 << i)))
+ continue;
+
+ channel = pdata->channel_head + i;
+
+ dma_ch_isr = readl(XLGMAC_DMA_REG(channel, DMA_CH_SR));
+ netif_dbg(pdata, intr, pdata->netdev, "DMA_CH%u_ISR=%#010x\n",
+ i, dma_ch_isr);
+
+ /* The TI or RI interrupt bits may still be set even if using
+ * per channel DMA interrupts. Check to be sure those are not
+ * enabled before using the private data napi structure.
+ */
+ ti = XLGMAC_GET_REG_BITS(dma_ch_isr, DMA_CH_SR_TI_POS,
+ DMA_CH_SR_TI_LEN);
+ ri = XLGMAC_GET_REG_BITS(dma_ch_isr, DMA_CH_SR_RI_POS,
+ DMA_CH_SR_RI_LEN);
+ if (!pdata->per_channel_irq && (ti || ri)) {
+ if (napi_schedule_prep(&pdata->napi)) {
+ /* Disable Tx and Rx interrupts */
+ xlgmac_disable_rx_tx_ints(pdata);
+
+ pdata->stats.napi_poll_isr++;
+ /* Turn on polling */
+ __napi_schedule_irqoff(&pdata->napi);
+ }
+ }
+
+ if (XLGMAC_GET_REG_BITS(dma_ch_isr, DMA_CH_SR_TPS_POS,
+ DMA_CH_SR_TPS_LEN))
+ pdata->stats.tx_process_stopped++;
+
+ if (XLGMAC_GET_REG_BITS(dma_ch_isr, DMA_CH_SR_RPS_POS,
+ DMA_CH_SR_RPS_LEN))
+ pdata->stats.rx_process_stopped++;
+
+ if (XLGMAC_GET_REG_BITS(dma_ch_isr, DMA_CH_SR_TBU_POS,
+ DMA_CH_SR_TBU_LEN))
+ pdata->stats.tx_buffer_unavailable++;
+
+ if (XLGMAC_GET_REG_BITS(dma_ch_isr, DMA_CH_SR_RBU_POS,
+ DMA_CH_SR_RBU_LEN))
+ pdata->stats.rx_buffer_unavailable++;
+
+ /* Restart the device on a Fatal Bus Error */
+ if (XLGMAC_GET_REG_BITS(dma_ch_isr, DMA_CH_SR_FBE_POS,
+ DMA_CH_SR_FBE_LEN)) {
+ pdata->stats.fatal_bus_error++;
+ schedule_work(&pdata->restart_work);
+ }
+
+ /* Clear all interrupt signals */
+ writel(dma_ch_isr, XLGMAC_DMA_REG(channel, DMA_CH_SR));
+ }
+
+ if (XLGMAC_GET_REG_BITS(dma_isr, DMA_ISR_MACIS_POS,
+ DMA_ISR_MACIS_LEN)) {
+ mac_isr = readl(pdata->mac_regs + MAC_ISR);
+
+ if (XLGMAC_GET_REG_BITS(mac_isr, MAC_ISR_MMCTXIS_POS,
+ MAC_ISR_MMCTXIS_LEN))
+ hw_ops->tx_mmc_int(pdata);
+
+ if (XLGMAC_GET_REG_BITS(mac_isr, MAC_ISR_MMCRXIS_POS,
+ MAC_ISR_MMCRXIS_LEN))
+ hw_ops->rx_mmc_int(pdata);
+ }
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t xlgmac_dma_isr(int irq, void *data)
+{
+ struct xlgmac_channel *channel = data;
+
+ /* Per channel DMA interrupts are enabled, so we use the per
+ * channel napi structure and not the private data napi structure
+ */
+ if (napi_schedule_prep(&channel->napi)) {
+ /* Disable Tx and Rx interrupts */
+ disable_irq_nosync(channel->dma_irq);
+
+ /* Turn on polling */
+ __napi_schedule_irqoff(&channel->napi);
+ }
+
+ return IRQ_HANDLED;
+}
+
+static void xlgmac_tx_timer(unsigned long data)
+{
+ struct xlgmac_channel *channel = (struct xlgmac_channel *)data;
+ struct xlgmac_pdata *pdata = channel->pdata;
+ struct napi_struct *napi;
+
+ napi = (pdata->per_channel_irq) ? &channel->napi : &pdata->napi;
+
+ if (napi_schedule_prep(napi)) {
+ /* Disable Tx and Rx interrupts */
+ if (pdata->per_channel_irq)
+ disable_irq_nosync(channel->dma_irq);
+ else
+ xlgmac_disable_rx_tx_ints(pdata);
+
+ pdata->stats.napi_poll_txtimer++;
+ /* Turn on polling */
+ __napi_schedule(napi);
+ }
+
+ channel->tx_timer_active = 0;
+}
+
+static void xlgmac_init_timers(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->tx_ring)
+ break;
+
+ setup_timer(&channel->tx_timer, xlgmac_tx_timer,
+ (unsigned long)channel);
+ }
+}
+
+static void xlgmac_stop_timers(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->tx_ring)
+ break;
+
+ del_timer_sync(&channel->tx_timer);
+ }
+}
+
+static void xlgmac_napi_enable(struct xlgmac_pdata *pdata, unsigned int add)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+
+ if (pdata->per_channel_irq) {
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (add)
+ netif_napi_add(pdata->netdev, &channel->napi,
+ xlgmac_one_poll,
+ NAPI_POLL_WEIGHT);
+
+ napi_enable(&channel->napi);
+ }
+ } else {
+ if (add)
+ netif_napi_add(pdata->netdev, &pdata->napi,
+ xlgmac_all_poll, NAPI_POLL_WEIGHT);
+
+ napi_enable(&pdata->napi);
+ }
+}
+
+static void xlgmac_napi_disable(struct xlgmac_pdata *pdata, unsigned int del)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+
+ if (pdata->per_channel_irq) {
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ napi_disable(&channel->napi);
+
+ if (del)
+ netif_napi_del(&channel->napi);
+ }
+ } else {
+ napi_disable(&pdata->napi);
+
+ if (del)
+ netif_napi_del(&pdata->napi);
+ }
+}
+
+static int xlgmac_request_irqs(struct xlgmac_pdata *pdata)
+{
+ struct net_device *netdev = pdata->netdev;
+ struct xlgmac_channel *channel;
+ unsigned int i;
+ int ret;
+
+ ret = devm_request_irq(pdata->dev, pdata->dev_irq, xlgmac_isr,
+ IRQF_SHARED, netdev->name, pdata);
+ if (ret) {
+ netdev_alert(netdev, "error requesting irq %d\n",
+ pdata->dev_irq);
+ return ret;
+ }
+
+ if (!pdata->per_channel_irq)
+ return 0;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ snprintf(channel->dma_irq_name,
+ sizeof(channel->dma_irq_name) - 1,
+ "%s-TxRx-%u", netdev_name(netdev),
+ channel->queue_index);
+
+ ret = devm_request_irq(pdata->dev, channel->dma_irq,
+ xlgmac_dma_isr, 0,
+ channel->dma_irq_name, channel);
+ if (ret) {
+ netdev_alert(netdev, "error requesting irq %d\n",
+ channel->dma_irq);
+ goto err_irq;
+ }
+ }
+
+ return 0;
+
+err_irq:
+ /* Using an unsigned int, 'i' will go to UINT_MAX and exit */
+ for (i--, channel--; i < pdata->channel_count; i--, channel--)
+ devm_free_irq(pdata->dev, channel->dma_irq, channel);
+
+ devm_free_irq(pdata->dev, pdata->dev_irq, pdata);
+
+ return ret;
+}
+
+static void xlgmac_free_irqs(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_channel *channel;
+ unsigned int i;
+
+ devm_free_irq(pdata->dev, pdata->dev_irq, pdata);
+
+ if (!pdata->per_channel_irq)
+ return;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++)
+ devm_free_irq(pdata->dev, channel->dma_irq, channel);
+}
+
+static void xlgmac_free_tx_data(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_desc_ops *desc_ops = &pdata->desc_ops;
+ struct xlgmac_desc_data *desc_data;
+ struct xlgmac_channel *channel;
+ struct xlgmac_ring *ring;
+ unsigned int i, j;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ ring = channel->tx_ring;
+ if (!ring)
+ break;
+
+ for (j = 0; j < ring->dma_desc_count; j++) {
+ desc_data = XLGMAC_GET_DESC_DATA(ring, j);
+ desc_ops->unmap_desc_data(pdata, desc_data);
+ }
+ }
+}
+
+static void xlgmac_free_rx_data(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_desc_ops *desc_ops = &pdata->desc_ops;
+ struct xlgmac_desc_data *desc_data;
+ struct xlgmac_channel *channel;
+ struct xlgmac_ring *ring;
+ unsigned int i, j;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ ring = channel->rx_ring;
+ if (!ring)
+ break;
+
+ for (j = 0; j < ring->dma_desc_count; j++) {
+ desc_data = XLGMAC_GET_DESC_DATA(ring, j);
+ desc_ops->unmap_desc_data(pdata, desc_data);
+ }
+ }
+}
+
+static int xlgmac_start(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_hw_ops *hw_ops = &pdata->hw_ops;
+ struct net_device *netdev = pdata->netdev;
+ int ret;
+
+ hw_ops->init(pdata);
+ xlgmac_napi_enable(pdata, 1);
+
+ ret = xlgmac_request_irqs(pdata);
+ if (ret)
+ goto err_napi;
+
+ hw_ops->enable_tx(pdata);
+ hw_ops->enable_rx(pdata);
+ netif_tx_start_all_queues(netdev);
+
+ return 0;
+
+err_napi:
+ xlgmac_napi_disable(pdata, 1);
+ hw_ops->exit(pdata);
+
+ return ret;
+}
+
+static void xlgmac_stop(struct xlgmac_pdata *pdata)
+{
+ struct xlgmac_hw_ops *hw_ops = &pdata->hw_ops;
+ struct net_device *netdev = pdata->netdev;
+ struct xlgmac_channel *channel;
+ struct netdev_queue *txq;
+ unsigned int i;
+
+ netif_tx_stop_all_queues(netdev);
+ xlgmac_stop_timers(pdata);
+ hw_ops->disable_tx(pdata);
+ hw_ops->disable_rx(pdata);
+ xlgmac_free_irqs(pdata);
+ xlgmac_napi_disable(pdata, 1);
+ hw_ops->exit(pdata);
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ if (!channel->tx_ring)
+ continue;
+
+ txq = netdev_get_tx_queue(netdev, channel->queue_index);
+ netdev_tx_reset_queue(txq);
+ }
+}
+
+static void xlgmac_restart_dev(struct xlgmac_pdata *pdata)
+{
+ /* If not running, "restart" will happen on open */
+ if (!netif_running(pdata->netdev))
+ return;
+
+ xlgmac_stop(pdata);
+
+ xlgmac_free_tx_data(pdata);
+ xlgmac_free_rx_data(pdata);
+
+ xlgmac_start(pdata);
+}
+
+static void xlgmac_restart(struct work_struct *work)
+{
+ struct xlgmac_pdata *pdata = container_of(work,
+ struct xlgmac_pdata,
+ restart_work);
+
+ rtnl_lock();
+
+ xlgmac_restart_dev(pdata);
+
+ rtnl_unlock();
+}
+
+static int xlgmac_open(struct net_device *netdev)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ struct xlgmac_desc_ops *desc_ops;
+ int ret;
+
+ desc_ops = &pdata->desc_ops;
+
+ /* TODO: Initialize the phy */
+
+ /* Calculate the Rx buffer size before allocating rings */
+ ret = xlgmac_calc_rx_buf_size(netdev, netdev->mtu);
+ if (ret < 0)
+ return ret;
+ pdata->rx_buf_size = ret;
+
+ /* Allocate the channels and rings */
+ ret = desc_ops->alloc_channles_and_rings(pdata);
+ if (ret)
+ return ret;
+
+ INIT_WORK(&pdata->restart_work, xlgmac_restart);
+ xlgmac_init_timers(pdata);
+
+ ret = xlgmac_start(pdata);
+ if (ret)
+ goto err_channels_and_rings;
+
+ return 0;
+
+err_channels_and_rings:
+ desc_ops->free_channels_and_rings(pdata);
+
+ return ret;
+}
+
+static int xlgmac_close(struct net_device *netdev)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ struct xlgmac_desc_ops *desc_ops;
+
+ desc_ops = &pdata->desc_ops;
+
+ /* Stop the device */
+ xlgmac_stop(pdata);
+
+ /* Free the channels and rings */
+ desc_ops->free_channels_and_rings(pdata);
+
+ return 0;
+}
+
+static void xlgmac_tx_timeout(struct net_device *netdev)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+
+ netdev_warn(netdev, "tx timeout, device restarting\n");
+ schedule_work(&pdata->restart_work);
+}
+
+static int xlgmac_xmit(struct sk_buff *skb, struct net_device *netdev)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ struct xlgmac_pkt_info *tx_pkt_info;
+ struct xlgmac_desc_ops *desc_ops;
+ struct xlgmac_channel *channel;
+ struct xlgmac_hw_ops *hw_ops;
+ struct netdev_queue *txq;
+ struct xlgmac_ring *ring;
+ int ret;
+
+ desc_ops = &pdata->desc_ops;
+ hw_ops = &pdata->hw_ops;
+
+ XLGMAC_PR("skb->len = %d\n", skb->len);
+
+ channel = pdata->channel_head + skb->queue_mapping;
+ txq = netdev_get_tx_queue(netdev, channel->queue_index);
+ ring = channel->tx_ring;
+ tx_pkt_info = &ring->pkt_info;
+
+ if (skb->len == 0) {
+ netif_err(pdata, tx_err, netdev,
+ "empty skb received from stack\n");
+ dev_kfree_skb_any(skb);
+ return NETDEV_TX_OK;
+ }
+
+ /* Prepare preliminary packet info for TX */
+ memset(tx_pkt_info, 0, sizeof(*tx_pkt_info));
+ xlgmac_prep_tx_pkt(pdata, ring, skb, tx_pkt_info);
+
+ /* Check that there are enough descriptors available */
+ ret = xlgmac_maybe_stop_tx_queue(channel, ring,
+ tx_pkt_info->desc_count);
+ if (ret)
+ return ret;
+
+ ret = xlgmac_prep_tso(skb, tx_pkt_info);
+ if (ret) {
+ netif_err(pdata, tx_err, netdev,
+ "error processing TSO packet\n");
+ dev_kfree_skb_any(skb);
+ return ret;
+ }
+ xlgmac_prep_vlan(skb, tx_pkt_info);
+
+ if (!desc_ops->map_tx_skb(channel, skb)) {
+ dev_kfree_skb_any(skb);
+ return NETDEV_TX_OK;
+ }
+
+ /* Report on the actual number of bytes (to be) sent */
+ netdev_tx_sent_queue(txq, tx_pkt_info->tx_bytes);
+
+ /* Configure required descriptor fields for transmission */
+ hw_ops->dev_xmit(channel);
+
+ if (netif_msg_pktdata(pdata))
+ xlgmac_print_pkt(netdev, skb, true);
+
+ /* Stop the queue in advance if there may not be enough descriptors */
+ xlgmac_maybe_stop_tx_queue(channel, ring, XLGMAC_TX_MAX_DESC_NR);
+
+ return NETDEV_TX_OK;
+}
+
+static void xlgmac_get_stats64(struct net_device *netdev,
+ struct rtnl_link_stats64 *s)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ struct xlgmac_stats *pstats = &pdata->stats;
+
+ pdata->hw_ops.read_mmc_stats(pdata);
+
+ s->rx_packets = pstats->rxframecount_gb;
+ s->rx_bytes = pstats->rxoctetcount_gb;
+ s->rx_errors = pstats->rxframecount_gb -
+ pstats->rxbroadcastframes_g -
+ pstats->rxmulticastframes_g -
+ pstats->rxunicastframes_g;
+ s->multicast = pstats->rxmulticastframes_g;
+ s->rx_length_errors = pstats->rxlengtherror;
+ s->rx_crc_errors = pstats->rxcrcerror;
+ s->rx_fifo_errors = pstats->rxfifooverflow;
+
+ s->tx_packets = pstats->txframecount_gb;
+ s->tx_bytes = pstats->txoctetcount_gb;
+ s->tx_errors = pstats->txframecount_gb - pstats->txframecount_g;
+ s->tx_dropped = netdev->stats.tx_dropped;
+}
+
+static int xlgmac_set_mac_address(struct net_device *netdev, void *addr)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ struct xlgmac_hw_ops *hw_ops = &pdata->hw_ops;
+ struct sockaddr *saddr = addr;
+
+ if (!is_valid_ether_addr(saddr->sa_data))
+ return -EADDRNOTAVAIL;
+
+ memcpy(netdev->dev_addr, saddr->sa_data, netdev->addr_len);
+
+ hw_ops->set_mac_address(pdata, netdev->dev_addr);
+
+ return 0;
+}
+
+static int xlgmac_ioctl(struct net_device *netdev,
+ struct ifreq *ifreq, int cmd)
+{
+ if (!netif_running(netdev))
+ return -ENODEV;
+
+ return 0;
+}
+
+static int xlgmac_change_mtu(struct net_device *netdev, int mtu)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ int ret;
+
+ ret = xlgmac_calc_rx_buf_size(netdev, mtu);
+ if (ret < 0)
+ return ret;
+
+ pdata->rx_buf_size = ret;
+ netdev->mtu = mtu;
+
+ xlgmac_restart_dev(pdata);
+
+ return 0;
+}
+
+static int xlgmac_vlan_rx_add_vid(struct net_device *netdev,
+ __be16 proto,
+ u16 vid)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ struct xlgmac_hw_ops *hw_ops = &pdata->hw_ops;
+
+ set_bit(vid, pdata->active_vlans);
+ hw_ops->update_vlan_hash_table(pdata);
+
+ return 0;
+}
+
+static int xlgmac_vlan_rx_kill_vid(struct net_device *netdev,
+ __be16 proto,
+ u16 vid)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ struct xlgmac_hw_ops *hw_ops = &pdata->hw_ops;
+
+ clear_bit(vid, pdata->active_vlans);
+ hw_ops->update_vlan_hash_table(pdata);
+
+ return 0;
+}
+
+#ifdef CONFIG_NET_POLL_CONTROLLER
+static void xlgmac_poll_controller(struct net_device *netdev)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ struct xlgmac_channel *channel;
+ unsigned int i;
+
+ if (pdata->per_channel_irq) {
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++)
+ xlgmac_dma_isr(channel->dma_irq, channel);
+ } else {
+ disable_irq(pdata->dev_irq);
+ xlgmac_isr(pdata->dev_irq, pdata);
+ enable_irq(pdata->dev_irq);
+ }
+}
+#endif /* CONFIG_NET_POLL_CONTROLLER */
+
+static int xlgmac_set_features(struct net_device *netdev,
+ netdev_features_t features)
+{
+ netdev_features_t rxhash, rxcsum, rxvlan, rxvlan_filter;
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ struct xlgmac_hw_ops *hw_ops = &pdata->hw_ops;
+ int ret = 0;
+
+ rxhash = pdata->netdev_features & NETIF_F_RXHASH;
+ rxcsum = pdata->netdev_features & NETIF_F_RXCSUM;
+ rxvlan = pdata->netdev_features & NETIF_F_HW_VLAN_CTAG_RX;
+ rxvlan_filter = pdata->netdev_features & NETIF_F_HW_VLAN_CTAG_FILTER;
+
+ if ((features & NETIF_F_RXHASH) && !rxhash)
+ ret = hw_ops->enable_rss(pdata);
+ else if (!(features & NETIF_F_RXHASH) && rxhash)
+ ret = hw_ops->disable_rss(pdata);
+ if (ret)
+ return ret;
+
+ if ((features & NETIF_F_RXCSUM) && !rxcsum)
+ hw_ops->enable_rx_csum(pdata);
+ else if (!(features & NETIF_F_RXCSUM) && rxcsum)
+ hw_ops->disable_rx_csum(pdata);
+
+ if ((features & NETIF_F_HW_VLAN_CTAG_RX) && !rxvlan)
+ hw_ops->enable_rx_vlan_stripping(pdata);
+ else if (!(features & NETIF_F_HW_VLAN_CTAG_RX) && rxvlan)
+ hw_ops->disable_rx_vlan_stripping(pdata);
+
+ if ((features & NETIF_F_HW_VLAN_CTAG_FILTER) && !rxvlan_filter)
+ hw_ops->enable_rx_vlan_filtering(pdata);
+ else if (!(features & NETIF_F_HW_VLAN_CTAG_FILTER) && rxvlan_filter)
+ hw_ops->disable_rx_vlan_filtering(pdata);
+
+ pdata->netdev_features = features;
+
+ return 0;
+}
+
+static void xlgmac_set_rx_mode(struct net_device *netdev)
+{
+ struct xlgmac_pdata *pdata = netdev_priv(netdev);
+ struct xlgmac_hw_ops *hw_ops = &pdata->hw_ops;
+
+ hw_ops->config_rx_mode(pdata);
+}
+
+static const struct net_device_ops xlgmac_netdev_ops = {
+ .ndo_open = xlgmac_open,
+ .ndo_stop = xlgmac_close,
+ .ndo_start_xmit = xlgmac_xmit,
+ .ndo_tx_timeout = xlgmac_tx_timeout,
+ .ndo_get_stats64 = xlgmac_get_stats64,
+ .ndo_change_mtu = xlgmac_change_mtu,
+ .ndo_set_mac_address = xlgmac_set_mac_address,
+ .ndo_validate_addr = eth_validate_addr,
+ .ndo_do_ioctl = xlgmac_ioctl,
+ .ndo_vlan_rx_add_vid = xlgmac_vlan_rx_add_vid,
+ .ndo_vlan_rx_kill_vid = xlgmac_vlan_rx_kill_vid,
+#ifdef CONFIG_NET_POLL_CONTROLLER
+ .ndo_poll_controller = xlgmac_poll_controller,
+#endif
+ .ndo_set_features = xlgmac_set_features,
+ .ndo_set_rx_mode = xlgmac_set_rx_mode,
+};
+
+const struct net_device_ops *xlgmac_get_netdev_ops(void)
+{
+ return &xlgmac_netdev_ops;
+}
+
+static void xlgmac_rx_refresh(struct xlgmac_channel *channel)
+{
+ struct xlgmac_pdata *pdata = channel->pdata;
+ struct xlgmac_ring *ring = channel->rx_ring;
+ struct xlgmac_desc_data *desc_data;
+ struct xlgmac_desc_ops *desc_ops;
+ struct xlgmac_hw_ops *hw_ops;
+
+ desc_ops = &pdata->desc_ops;
+ hw_ops = &pdata->hw_ops;
+
+ while (ring->dirty != ring->cur) {
+ desc_data = XLGMAC_GET_DESC_DATA(ring, ring->dirty);
+
+ /* Reset desc_data values */
+ desc_ops->unmap_desc_data(pdata, desc_data);
+
+ if (desc_ops->map_rx_buffer(pdata, ring, desc_data))
+ break;
+
+ hw_ops->rx_desc_reset(pdata, desc_data, ring->dirty);
+
+ ring->dirty++;
+ }
+
+ /* Make sure everything is written before the register write */
+ wmb();
+
+ /* Update the Rx Tail Pointer Register with address of
+ * the last cleaned entry
+ */
+ desc_data = XLGMAC_GET_DESC_DATA(ring, ring->dirty - 1);
+ writel(lower_32_bits(desc_data->dma_desc_addr),
+ XLGMAC_DMA_REG(channel, DMA_CH_RDTR_LO));
+}
+
+static struct sk_buff *xlgmac_create_skb(struct xlgmac_pdata *pdata,
+ struct napi_struct *napi,
+ struct xlgmac_desc_data *desc_data,
+ unsigned int len)
+{
+ unsigned int copy_len;
+ struct sk_buff *skb;
+ u8 *packet;
+
+ skb = napi_alloc_skb(napi, desc_data->rx.hdr.dma_len);
+ if (!skb)
+ return NULL;
+
+ /* Start with the header buffer which may contain just the header
+ * or the header plus data
+ */
+ dma_sync_single_range_for_cpu(pdata->dev, desc_data->rx.hdr.dma_base,
+ desc_data->rx.hdr.dma_off,
+ desc_data->rx.hdr.dma_len,
+ DMA_FROM_DEVICE);
+
+ packet = page_address(desc_data->rx.hdr.pa.pages) +
+ desc_data->rx.hdr.pa.pages_offset;
+ copy_len = (desc_data->rx.hdr_len) ? desc_data->rx.hdr_len : len;
+ copy_len = min(desc_data->rx.hdr.dma_len, copy_len);
+ skb_copy_to_linear_data(skb, packet, copy_len);
+ skb_put(skb, copy_len);
+
+ len -= copy_len;
+ if (len) {
+ /* Add the remaining data as a frag */
+ dma_sync_single_range_for_cpu(pdata->dev,
+ desc_data->rx.buf.dma_base,
+ desc_data->rx.buf.dma_off,
+ desc_data->rx.buf.dma_len,
+ DMA_FROM_DEVICE);
+
+ skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
+ desc_data->rx.buf.pa.pages,
+ desc_data->rx.buf.pa.pages_offset,
+ len, desc_data->rx.buf.dma_len);
+ desc_data->rx.buf.pa.pages = NULL;
+ }
+
+ return skb;
+}
+
+static int xlgmac_tx_poll(struct xlgmac_channel *channel)
+{
+ struct xlgmac_pdata *pdata = channel->pdata;
+ struct xlgmac_ring *ring = channel->tx_ring;
+ struct net_device *netdev = pdata->netdev;
+ unsigned int tx_packets = 0, tx_bytes = 0;
+ struct xlgmac_desc_data *desc_data;
+ struct xlgmac_dma_desc *dma_desc;
+ struct xlgmac_desc_ops *desc_ops;
+ struct xlgmac_hw_ops *hw_ops;
+ struct netdev_queue *txq;
+ int processed = 0;
+ unsigned int cur;
+
+ desc_ops = &pdata->desc_ops;
+ hw_ops = &pdata->hw_ops;
+
+ /* Nothing to do if there isn't a Tx ring for this channel */
+ if (!ring)
+ return 0;
+
+ cur = ring->cur;
+
+ /* Be sure we get ring->cur before accessing descriptor data */
+ smp_rmb();
+
+ txq = netdev_get_tx_queue(netdev, channel->queue_index);
+
+ while ((processed < XLGMAC_TX_DESC_MAX_PROC) &&
+ (ring->dirty != cur)) {
+ desc_data = XLGMAC_GET_DESC_DATA(ring, ring->dirty);
+ dma_desc = desc_data->dma_desc;
+
+ if (!hw_ops->tx_complete(dma_desc))
+ break;
+
+ /* Make sure descriptor fields are read after reading
+ * the OWN bit
+ */
+ dma_rmb();
+
+ if (netif_msg_tx_done(pdata))
+ xlgmac_dump_tx_desc(pdata, ring, ring->dirty, 1, 0);
+
+ if (hw_ops->is_last_desc(dma_desc)) {
+ tx_packets += desc_data->tx.packets;
+ tx_bytes += desc_data->tx.bytes;
+ }
+
+ /* Free the SKB and reset the descriptor for re-use */
+ desc_ops->unmap_desc_data(pdata, desc_data);
+ hw_ops->tx_desc_reset(desc_data);
+
+ processed++;
+ ring->dirty++;
+ }
+
+ if (!processed)
+ return 0;
+
+ netdev_tx_completed_queue(txq, tx_packets, tx_bytes);
+
+ if ((ring->tx.queue_stopped == 1) &&
+ (xlgmac_tx_avail_desc(ring) > XLGMAC_TX_DESC_MIN_FREE)) {
+ ring->tx.queue_stopped = 0;
+ netif_tx_wake_queue(txq);
+ }
+
+ XLGMAC_PR("processed=%d\n", processed);
+
+ return processed;
+}
+
+static int xlgmac_rx_poll(struct xlgmac_channel *channel, int budget)
+{
+ struct xlgmac_pdata *pdata = channel->pdata;
+ struct xlgmac_ring *ring = channel->rx_ring;
+ struct net_device *netdev = pdata->netdev;
+ unsigned int len, dma_desc_len, max_len;
+ unsigned int context_next, context;
+ struct xlgmac_desc_data *desc_data;
+ struct xlgmac_pkt_info *pkt_info;
+ unsigned int incomplete, error;
+ struct xlgmac_hw_ops *hw_ops;
+ unsigned int received = 0;
+ struct napi_struct *napi;
+ struct sk_buff *skb;
+ int packet_count = 0;
+
+ hw_ops = &pdata->hw_ops;
+
+ /* Nothing to do if there isn't a Rx ring for this channel */
+ if (!ring)
+ return 0;
+
+ incomplete = 0;
+ context_next = 0;
+
+ napi = (pdata->per_channel_irq) ? &channel->napi : &pdata->napi;
+
+ desc_data = XLGMAC_GET_DESC_DATA(ring, ring->cur);
+ pkt_info = &ring->pkt_info;
+ while (packet_count < budget) {
+ /* First time in loop see if we need to restore state */
+ if (!received && desc_data->state_saved) {
+ skb = desc_data->state.skb;
+ error = desc_data->state.error;
+ len = desc_data->state.len;
+ } else {
+ memset(pkt_info, 0, sizeof(*pkt_info));
+ skb = NULL;
+ error = 0;
+ len = 0;
+ }
+
+read_again:
+ desc_data = XLGMAC_GET_DESC_DATA(ring, ring->cur);
+
+ if (xlgmac_rx_dirty_desc(ring) > XLGMAC_RX_DESC_MAX_DIRTY)
+ xlgmac_rx_refresh(channel);
+
+ if (hw_ops->dev_read(channel))
+ break;
+
+ received++;
+ ring->cur++;
+
+ incomplete = XLGMAC_GET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_INCOMPLETE_POS,
+ RX_PACKET_ATTRIBUTES_INCOMPLETE_LEN);
+ context_next = XLGMAC_GET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_CONTEXT_NEXT_POS,
+ RX_PACKET_ATTRIBUTES_CONTEXT_NEXT_LEN);
+ context = XLGMAC_GET_REG_BITS(
+ pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_CONTEXT_POS,
+ RX_PACKET_ATTRIBUTES_CONTEXT_LEN);
+
+ /* Earlier error, just drain the remaining data */
+ if ((incomplete || context_next) && error)
+ goto read_again;
+
+ if (error || pkt_info->errors) {
+ if (pkt_info->errors)
+ netif_err(pdata, rx_err, netdev,
+ "error in received packet\n");
+ dev_kfree_skb(skb);
+ goto next_packet;
+ }
+
+ if (!context) {
+ /* Length is cumulative, get this descriptor's length */
+ dma_desc_len = desc_data->rx.len - len;
+ len += dma_desc_len;
+
+ if (dma_desc_len && !skb) {
+ skb = xlgmac_create_skb(pdata, napi, desc_data,
+ dma_desc_len);
+ if (!skb)
+ error = 1;
+ } else if (dma_desc_len) {
+ dma_sync_single_range_for_cpu(
+ pdata->dev,
+ desc_data->rx.buf.dma_base,
+ desc_data->rx.buf.dma_off,
+ desc_data->rx.buf.dma_len,
+ DMA_FROM_DEVICE);
+
+ skb_add_rx_frag(
+ skb, skb_shinfo(skb)->nr_frags,
+ desc_data->rx.buf.pa.pages,
+ desc_data->rx.buf.pa.pages_offset,
+ dma_desc_len,
+ desc_data->rx.buf.dma_len);
+ desc_data->rx.buf.pa.pages = NULL;
+ }
+ }
+
+ if (incomplete || context_next)
+ goto read_again;
+
+ if (!skb)
+ goto next_packet;
+
+ /* Be sure we don't exceed the configured MTU */
+ max_len = netdev->mtu + ETH_HLEN;
+ if (!(netdev->features & NETIF_F_HW_VLAN_CTAG_RX) &&
+ (skb->protocol == htons(ETH_P_8021Q)))
+ max_len += VLAN_HLEN;
+
+ if (skb->len > max_len) {
+ netif_err(pdata, rx_err, netdev,
+ "packet length exceeds configured MTU\n");
+ dev_kfree_skb(skb);
+ goto next_packet;
+ }
+
+ if (netif_msg_pktdata(pdata))
+ xlgmac_print_pkt(netdev, skb, false);
+
+ skb_checksum_none_assert(skb);
+ if (XLGMAC_GET_REG_BITS(pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_CSUM_DONE_POS,
+ RX_PACKET_ATTRIBUTES_CSUM_DONE_LEN))
+ skb->ip_summed = CHECKSUM_UNNECESSARY;
+
+ if (XLGMAC_GET_REG_BITS(pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_VLAN_CTAG_POS,
+ RX_PACKET_ATTRIBUTES_VLAN_CTAG_LEN)) {
+ __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q),
+ pkt_info->vlan_ctag);
+ pdata->stats.rx_vlan_packets++;
+ }
+
+ if (XLGMAC_GET_REG_BITS(pkt_info->attributes,
+ RX_PACKET_ATTRIBUTES_RSS_HASH_POS,
+ RX_PACKET_ATTRIBUTES_RSS_HASH_LEN))
+ skb_set_hash(skb, pkt_info->rss_hash,
+ pkt_info->rss_hash_type);
+
+ skb->dev = netdev;
+ skb->protocol = eth_type_trans(skb, netdev);
+ skb_record_rx_queue(skb, channel->queue_index);
+
+ napi_gro_receive(napi, skb);
+
+next_packet:
+ packet_count++;
+ }
+
+ /* Check if we need to save state before leaving */
+ if (received && (incomplete || context_next)) {
+ desc_data = XLGMAC_GET_DESC_DATA(ring, ring->cur);
+ desc_data->state_saved = 1;
+ desc_data->state.skb = skb;
+ desc_data->state.len = len;
+ desc_data->state.error = error;
+ }
+
+ XLGMAC_PR("packet_count = %d\n", packet_count);
+
+ return packet_count;
+}
+
+static int xlgmac_one_poll(struct napi_struct *napi, int budget)
+{
+ struct xlgmac_channel *channel = container_of(napi,
+ struct xlgmac_channel,
+ napi);
+ int processed = 0;
+
+ XLGMAC_PR("budget=%d\n", budget);
+
+ /* Cleanup Tx ring first */
+ xlgmac_tx_poll(channel);
+
+ /* Process Rx ring next */
+ processed = xlgmac_rx_poll(channel, budget);
+
+ /* If we processed everything, we are done */
+ if (processed < budget) {
+ /* Turn off polling */
+ napi_complete_done(napi, processed);
+
+ /* Enable Tx and Rx interrupts */
+ enable_irq(channel->dma_irq);
+ }
+
+ XLGMAC_PR("received = %d\n", processed);
+
+ return processed;
+}
+
+static int xlgmac_all_poll(struct napi_struct *napi, int budget)
+{
+ struct xlgmac_pdata *pdata = container_of(napi,
+ struct xlgmac_pdata,
+ napi);
+ struct xlgmac_channel *channel;
+ int processed, last_processed;
+ int ring_budget;
+ unsigned int i;
+
+ XLGMAC_PR("budget=%d\n", budget);
+
+ processed = 0;
+ ring_budget = budget / pdata->rx_ring_count;
+ do {
+ last_processed = processed;
+
+ channel = pdata->channel_head;
+ for (i = 0; i < pdata->channel_count; i++, channel++) {
+ /* Cleanup Tx ring first */
+ xlgmac_tx_poll(channel);
+
+ /* Process Rx ring next */
+ if (ring_budget > (budget - processed))
+ ring_budget = budget - processed;
+ processed += xlgmac_rx_poll(channel, ring_budget);
+ }
+ } while ((processed < budget) && (processed != last_processed));
+
+ /* If we processed everything, we are done */
+ if (processed < budget) {
+ /* Turn off polling */
+ napi_complete_done(napi, processed);
+
+ /* Enable Tx and Rx interrupts */
+ xlgmac_enable_rx_tx_ints(pdata);
+ }
+
+ XLGMAC_PR("received = %d\n", processed);
+
+ return processed;
+}
diff --git a/drivers/net/ethernet/synopsys/dwc-xlgmac-pci.c b/drivers/net/ethernet/synopsys/dwc-xlgmac-pci.c
new file mode 100644
index 0000000..386bafe
--- /dev/null
+++ b/drivers/net/ethernet/synopsys/dwc-xlgmac-pci.c
@@ -0,0 +1,78 @@
+/* Synopsys DesignWare Core Enterprise Ethernet (XLGMAC) Driver
+ *
+ * Copyright (c) 2017 Synopsys, Inc. (www.synopsys.com)
+ *
+ * This program is dual-licensed; you may select either version 2 of
+ * the GNU General Public License ("GPL") or BSD license ("BSD").
+ *
+ * This Synopsys DWC XLGMAC software driver and associated documentation
+ * (hereinafter the "Software") is an unsupported proprietary work of
+ * Synopsys, Inc. unless otherwise expressly agreed to in writing between
+ * Synopsys and you. The Software IS NOT an item of Licensed Software or a
+ * Licensed Product under any End User Software License Agreement or
+ * Agreement for Licensed Products with Synopsys or any supplement thereto.
+ * Synopsys is a registered trademark of Synopsys, Inc. Other names included
+ * in the SOFTWARE may be the trademarks of their respective owners.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+
+#include "dwc-xlgmac.h"
+#include "dwc-xlgmac-reg.h"
+
+static int xlgmac_probe(struct pci_dev *pcidev, const struct pci_device_id *id)
+{
+ struct device *dev = &pcidev->dev;
+ struct xlgmac_resources res;
+ int i, ret;
+
+ ret = pcim_enable_device(pcidev);
+ if (ret) {
+ dev_err(dev, "ERROR: failed to enable device\n");
+ return ret;
+ }
+
+ for (i = 0; i <= PCI_STD_RESOURCE_END; i++) {
+ if (pci_resource_len(pcidev, i) == 0)
+ continue;
+ ret = pcim_iomap_regions(pcidev, BIT(i), XLGMAC_DRV_NAME);
+ if (ret)
+ return ret;
+ break;
+ }
+
+ pci_set_master(pcidev);
+
+ memset(&res, 0, sizeof(res));
+ res.irq = pcidev->irq;
+ res.addr = pcim_iomap_table(pcidev)[i];
+
+ return xlgmac_drv_probe(&pcidev->dev, &res);
+}
+
+static void xlgmac_remove(struct pci_dev *pcidev)
+{
+ xlgmac_drv_remove(&pcidev->dev);
+}
+
+static const struct pci_device_id xlgmac_pci_tbl[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_SYNOPSYS, 0x7302) },
+ { 0 }
+};
+MODULE_DEVICE_TABLE(pci, xlgmac_pci_tbl);
+
+static struct pci_driver xlgmac_pci_driver = {
+ .name = XLGMAC_DRV_NAME,
+ .id_table = xlgmac_pci_tbl,
+ .probe = xlgmac_probe,
+ .remove = xlgmac_remove,
+};
+
+module_pci_driver(xlgmac_pci_driver);
+
+MODULE_DESCRIPTION(XLGMAC_DRV_DESC);
+MODULE_VERSION(XLGMAC_DRV_VERSION);
+MODULE_AUTHOR("Jie Deng <jiedeng@synopsys.com>");
+MODULE_LICENSE("Dual BSD/GPL");
diff --git a/drivers/net/ethernet/synopsys/dwc-xlgmac-reg.h b/drivers/net/ethernet/synopsys/dwc-xlgmac-reg.h
new file mode 100644
index 0000000..3754f22
--- /dev/null
+++ b/drivers/net/ethernet/synopsys/dwc-xlgmac-reg.h
@@ -0,0 +1,744 @@
+/* Synopsys DesignWare Core Enterprise Ethernet (XLGMAC) Driver
+ *
+ * Copyright (c) 2017 Synopsys, Inc. (www.synopsys.com)
+ *
+ * This program is dual-licensed; you may select either version 2 of
+ * the GNU General Public License ("GPL") or BSD license ("BSD").
+ *
+ * This Synopsys DWC XLGMAC software driver and associated documentation
+ * (hereinafter the "Software") is an unsupported proprietary work of
+ * Synopsys, Inc. unless otherwise expressly agreed to in writing between
+ * Synopsys and you. The Software IS NOT an item of Licensed Software or a
+ * Licensed Product under any End User Software License Agreement or
+ * Agreement for Licensed Products with Synopsys or any supplement thereto.
+ * Synopsys is a registered trademark of Synopsys, Inc. Other names included
+ * in the SOFTWARE may be the trademarks of their respective owners.
+ */
+
+#ifndef __DWC_XLGMAC_REG_H__
+#define __DWC_XLGMAC_REG_H__
+
+/* MAC register offsets */
+#define MAC_TCR 0x0000
+#define MAC_RCR 0x0004
+#define MAC_PFR 0x0008
+#define MAC_HTR0 0x0010
+#define MAC_VLANTR 0x0050
+#define MAC_VLANHTR 0x0058
+#define MAC_VLANIR 0x0060
+#define MAC_Q0TFCR 0x0070
+#define MAC_RFCR 0x0090
+#define MAC_RQC0R 0x00a0
+#define MAC_RQC1R 0x00a4
+#define MAC_RQC2R 0x00a8
+#define MAC_RQC3R 0x00ac
+#define MAC_ISR 0x00b0
+#define MAC_IER 0x00b4
+#define MAC_VR 0x0110
+#define MAC_HWF0R 0x011c
+#define MAC_HWF1R 0x0120
+#define MAC_HWF2R 0x0124
+#define MAC_MACA0HR 0x0300
+#define MAC_MACA0LR 0x0304
+#define MAC_MACA1HR 0x0308
+#define MAC_MACA1LR 0x030c
+#define MAC_RSSCR 0x0c80
+#define MAC_RSSAR 0x0c88
+#define MAC_RSSDR 0x0c8c
+
+#define MAC_QTFCR_INC 4
+#define MAC_MACA_INC 4
+#define MAC_HTR_INC 4
+#define MAC_RQC2_INC 4
+#define MAC_RQC2_Q_PER_REG 4
+
+/* MAC register entry bit positions and sizes */
+#define MAC_HWF0R_ADDMACADRSEL_POS 18
+#define MAC_HWF0R_ADDMACADRSEL_LEN 5
+#define MAC_HWF0R_ARPOFFSEL_POS 9
+#define MAC_HWF0R_ARPOFFSEL_LEN 1
+#define MAC_HWF0R_EEESEL_POS 13
+#define MAC_HWF0R_EEESEL_LEN 1
+#define MAC_HWF0R_PHYIFSEL_POS 1
+#define MAC_HWF0R_PHYIFSEL_LEN 2
+#define MAC_HWF0R_MGKSEL_POS 7
+#define MAC_HWF0R_MGKSEL_LEN 1
+#define MAC_HWF0R_MMCSEL_POS 8
+#define MAC_HWF0R_MMCSEL_LEN 1
+#define MAC_HWF0R_RWKSEL_POS 6
+#define MAC_HWF0R_RWKSEL_LEN 1
+#define MAC_HWF0R_RXCOESEL_POS 16
+#define MAC_HWF0R_RXCOESEL_LEN 1
+#define MAC_HWF0R_SAVLANINS_POS 27
+#define MAC_HWF0R_SAVLANINS_LEN 1
+#define MAC_HWF0R_SMASEL_POS 5
+#define MAC_HWF0R_SMASEL_LEN 1
+#define MAC_HWF0R_TSSEL_POS 12
+#define MAC_HWF0R_TSSEL_LEN 1
+#define MAC_HWF0R_TSSTSSEL_POS 25
+#define MAC_HWF0R_TSSTSSEL_LEN 2
+#define MAC_HWF0R_TXCOESEL_POS 14
+#define MAC_HWF0R_TXCOESEL_LEN 1
+#define MAC_HWF0R_VLHASH_POS 4
+#define MAC_HWF0R_VLHASH_LEN 1
+#define MAC_HWF1R_ADDR64_POS 14
+#define MAC_HWF1R_ADDR64_LEN 2
+#define MAC_HWF1R_ADVTHWORD_POS 13
+#define MAC_HWF1R_ADVTHWORD_LEN 1
+#define MAC_HWF1R_DBGMEMA_POS 19
+#define MAC_HWF1R_DBGMEMA_LEN 1
+#define MAC_HWF1R_DCBEN_POS 16
+#define MAC_HWF1R_DCBEN_LEN 1
+#define MAC_HWF1R_HASHTBLSZ_POS 24
+#define MAC_HWF1R_HASHTBLSZ_LEN 3
+#define MAC_HWF1R_L3L4FNUM_POS 27
+#define MAC_HWF1R_L3L4FNUM_LEN 4
+#define MAC_HWF1R_NUMTC_POS 21
+#define MAC_HWF1R_NUMTC_LEN 3
+#define MAC_HWF1R_RSSEN_POS 20
+#define MAC_HWF1R_RSSEN_LEN 1
+#define MAC_HWF1R_RXFIFOSIZE_POS 0
+#define MAC_HWF1R_RXFIFOSIZE_LEN 5
+#define MAC_HWF1R_SPHEN_POS 17
+#define MAC_HWF1R_SPHEN_LEN 1
+#define MAC_HWF1R_TSOEN_POS 18
+#define MAC_HWF1R_TSOEN_LEN 1
+#define MAC_HWF1R_TXFIFOSIZE_POS 6
+#define MAC_HWF1R_TXFIFOSIZE_LEN 5
+#define MAC_HWF2R_AUXSNAPNUM_POS 28
+#define MAC_HWF2R_AUXSNAPNUM_LEN 3
+#define MAC_HWF2R_PPSOUTNUM_POS 24
+#define MAC_HWF2R_PPSOUTNUM_LEN 3
+#define MAC_HWF2R_RXCHCNT_POS 12
+#define MAC_HWF2R_RXCHCNT_LEN 4
+#define MAC_HWF2R_RXQCNT_POS 0
+#define MAC_HWF2R_RXQCNT_LEN 4
+#define MAC_HWF2R_TXCHCNT_POS 18
+#define MAC_HWF2R_TXCHCNT_LEN 4
+#define MAC_HWF2R_TXQCNT_POS 6
+#define MAC_HWF2R_TXQCNT_LEN 4
+#define MAC_IER_TSIE_POS 12
+#define MAC_IER_TSIE_LEN 1
+#define MAC_ISR_MMCRXIS_POS 9
+#define MAC_ISR_MMCRXIS_LEN 1
+#define MAC_ISR_MMCTXIS_POS 10
+#define MAC_ISR_MMCTXIS_LEN 1
+#define MAC_ISR_PMTIS_POS 4
+#define MAC_ISR_PMTIS_LEN 1
+#define MAC_ISR_TSIS_POS 12
+#define MAC_ISR_TSIS_LEN 1
+#define MAC_MACA1HR_AE_POS 31
+#define MAC_MACA1HR_AE_LEN 1
+#define MAC_PFR_HMC_POS 2
+#define MAC_PFR_HMC_LEN 1
+#define MAC_PFR_HPF_POS 10
+#define MAC_PFR_HPF_LEN 1
+#define MAC_PFR_HUC_POS 1
+#define MAC_PFR_HUC_LEN 1
+#define MAC_PFR_PM_POS 4
+#define MAC_PFR_PM_LEN 1
+#define MAC_PFR_PR_POS 0
+#define MAC_PFR_PR_LEN 1
+#define MAC_PFR_VTFE_POS 16
+#define MAC_PFR_VTFE_LEN 1
+#define MAC_Q0TFCR_PT_POS 16
+#define MAC_Q0TFCR_PT_LEN 16
+#define MAC_Q0TFCR_TFE_POS 1
+#define MAC_Q0TFCR_TFE_LEN 1
+#define MAC_RCR_ACS_POS 1
+#define MAC_RCR_ACS_LEN 1
+#define MAC_RCR_CST_POS 2
+#define MAC_RCR_CST_LEN 1
+#define MAC_RCR_DCRCC_POS 3
+#define MAC_RCR_DCRCC_LEN 1
+#define MAC_RCR_HDSMS_POS 12
+#define MAC_RCR_HDSMS_LEN 3
+#define MAC_RCR_IPC_POS 9
+#define MAC_RCR_IPC_LEN 1
+#define MAC_RCR_JE_POS 8
+#define MAC_RCR_JE_LEN 1
+#define MAC_RCR_LM_POS 10
+#define MAC_RCR_LM_LEN 1
+#define MAC_RCR_RE_POS 0
+#define MAC_RCR_RE_LEN 1
+#define MAC_RFCR_PFCE_POS 8
+#define MAC_RFCR_PFCE_LEN 1
+#define MAC_RFCR_RFE_POS 0
+#define MAC_RFCR_RFE_LEN 1
+#define MAC_RFCR_UP_POS 1
+#define MAC_RFCR_UP_LEN 1
+#define MAC_RQC0R_RXQ0EN_POS 0
+#define MAC_RQC0R_RXQ0EN_LEN 2
+#define MAC_RSSAR_ADDRT_POS 2
+#define MAC_RSSAR_ADDRT_LEN 1
+#define MAC_RSSAR_CT_POS 1
+#define MAC_RSSAR_CT_LEN 1
+#define MAC_RSSAR_OB_POS 0
+#define MAC_RSSAR_OB_LEN 1
+#define MAC_RSSAR_RSSIA_POS 8
+#define MAC_RSSAR_RSSIA_LEN 8
+#define MAC_RSSCR_IP2TE_POS 1
+#define MAC_RSSCR_IP2TE_LEN 1
+#define MAC_RSSCR_RSSE_POS 0
+#define MAC_RSSCR_RSSE_LEN 1
+#define MAC_RSSCR_TCP4TE_POS 2
+#define MAC_RSSCR_TCP4TE_LEN 1
+#define MAC_RSSCR_UDP4TE_POS 3
+#define MAC_RSSCR_UDP4TE_LEN 1
+#define MAC_RSSDR_DMCH_POS 0
+#define MAC_RSSDR_DMCH_LEN 4
+#define MAC_TCR_SS_POS 28
+#define MAC_TCR_SS_LEN 3
+#define MAC_TCR_TE_POS 0
+#define MAC_TCR_TE_LEN 1
+#define MAC_VLANHTR_VLHT_POS 0
+#define MAC_VLANHTR_VLHT_LEN 16
+#define MAC_VLANIR_VLTI_POS 20
+#define MAC_VLANIR_VLTI_LEN 1
+#define MAC_VLANIR_CSVL_POS 19
+#define MAC_VLANIR_CSVL_LEN 1
+#define MAC_VLANTR_DOVLTC_POS 20
+#define MAC_VLANTR_DOVLTC_LEN 1
+#define MAC_VLANTR_ERSVLM_POS 19
+#define MAC_VLANTR_ERSVLM_LEN 1
+#define MAC_VLANTR_ESVL_POS 18
+#define MAC_VLANTR_ESVL_LEN 1
+#define MAC_VLANTR_ETV_POS 16
+#define MAC_VLANTR_ETV_LEN 1
+#define MAC_VLANTR_EVLS_POS 21
+#define MAC_VLANTR_EVLS_LEN 2
+#define MAC_VLANTR_EVLRXS_POS 24
+#define MAC_VLANTR_EVLRXS_LEN 1
+#define MAC_VLANTR_VL_POS 0
+#define MAC_VLANTR_VL_LEN 16
+#define MAC_VLANTR_VTHM_POS 25
+#define MAC_VLANTR_VTHM_LEN 1
+#define MAC_VLANTR_VTIM_POS 17
+#define MAC_VLANTR_VTIM_LEN 1
+#define MAC_VR_DEVID_POS 8
+#define MAC_VR_DEVID_LEN 8
+#define MAC_VR_SNPSVER_POS 0
+#define MAC_VR_SNPSVER_LEN 8
+#define MAC_VR_USERVER_POS 16
+#define MAC_VR_USERVER_LEN 8
+
+/* MMC register offsets */
+#define MMC_CR 0x0800
+#define MMC_RISR 0x0804
+#define MMC_TISR 0x0808
+#define MMC_RIER 0x080c
+#define MMC_TIER 0x0810
+#define MMC_TXOCTETCOUNT_GB_LO 0x0814
+#define MMC_TXFRAMECOUNT_GB_LO 0x081c
+#define MMC_TXBROADCASTFRAMES_G_LO 0x0824
+#define MMC_TXMULTICASTFRAMES_G_LO 0x082c
+#define MMC_TX64OCTETS_GB_LO 0x0834
+#define MMC_TX65TO127OCTETS_GB_LO 0x083c
+#define MMC_TX128TO255OCTETS_GB_LO 0x0844
+#define MMC_TX256TO511OCTETS_GB_LO 0x084c
+#define MMC_TX512TO1023OCTETS_GB_LO 0x0854
+#define MMC_TX1024TOMAXOCTETS_GB_LO 0x085c
+#define MMC_TXUNICASTFRAMES_GB_LO 0x0864
+#define MMC_TXMULTICASTFRAMES_GB_LO 0x086c
+#define MMC_TXBROADCASTFRAMES_GB_LO 0x0874
+#define MMC_TXUNDERFLOWERROR_LO 0x087c
+#define MMC_TXOCTETCOUNT_G_LO 0x0884
+#define MMC_TXFRAMECOUNT_G_LO 0x088c
+#define MMC_TXPAUSEFRAMES_LO 0x0894
+#define MMC_TXVLANFRAMES_G_LO 0x089c
+#define MMC_RXFRAMECOUNT_GB_LO 0x0900
+#define MMC_RXOCTETCOUNT_GB_LO 0x0908
+#define MMC_RXOCTETCOUNT_G_LO 0x0910
+#define MMC_RXBROADCASTFRAMES_G_LO 0x0918
+#define MMC_RXMULTICASTFRAMES_G_LO 0x0920
+#define MMC_RXCRCERROR_LO 0x0928
+#define MMC_RXRUNTERROR 0x0930
+#define MMC_RXJABBERERROR 0x0934
+#define MMC_RXUNDERSIZE_G 0x0938
+#define MMC_RXOVERSIZE_G 0x093c
+#define MMC_RX64OCTETS_GB_LO 0x0940
+#define MMC_RX65TO127OCTETS_GB_LO 0x0948
+#define MMC_RX128TO255OCTETS_GB_LO 0x0950
+#define MMC_RX256TO511OCTETS_GB_LO 0x0958
+#define MMC_RX512TO1023OCTETS_GB_LO 0x0960
+#define MMC_RX1024TOMAXOCTETS_GB_LO 0x0968
+#define MMC_RXUNICASTFRAMES_G_LO 0x0970
+#define MMC_RXLENGTHERROR_LO 0x0978
+#define MMC_RXOUTOFRANGETYPE_LO 0x0980
+#define MMC_RXPAUSEFRAMES_LO 0x0988
+#define MMC_RXFIFOOVERFLOW_LO 0x0990
+#define MMC_RXVLANFRAMES_GB_LO 0x0998
+#define MMC_RXWATCHDOGERROR 0x09a0
+
+/* MMC register entry bit positions and sizes */
+#define MMC_CR_CR_POS 0
+#define MMC_CR_CR_LEN 1
+#define MMC_CR_CSR_POS 1
+#define MMC_CR_CSR_LEN 1
+#define MMC_CR_ROR_POS 2
+#define MMC_CR_ROR_LEN 1
+#define MMC_CR_MCF_POS 3
+#define MMC_CR_MCF_LEN 1
+#define MMC_CR_MCT_POS 4
+#define MMC_CR_MCT_LEN 2
+#define MMC_RIER_ALL_INTERRUPTS_POS 0
+#define MMC_RIER_ALL_INTERRUPTS_LEN 23
+#define MMC_RISR_RXFRAMECOUNT_GB_POS 0
+#define MMC_RISR_RXFRAMECOUNT_GB_LEN 1
+#define MMC_RISR_RXOCTETCOUNT_GB_POS 1
+#define MMC_RISR_RXOCTETCOUNT_GB_LEN 1
+#define MMC_RISR_RXOCTETCOUNT_G_POS 2
+#define MMC_RISR_RXOCTETCOUNT_G_LEN 1
+#define MMC_RISR_RXBROADCASTFRAMES_G_POS 3
+#define MMC_RISR_RXBROADCASTFRAMES_G_LEN 1
+#define MMC_RISR_RXMULTICASTFRAMES_G_POS 4
+#define MMC_RISR_RXMULTICASTFRAMES_G_LEN 1
+#define MMC_RISR_RXCRCERROR_POS 5
+#define MMC_RISR_RXCRCERROR_LEN 1
+#define MMC_RISR_RXRUNTERROR_POS 6
+#define MMC_RISR_RXRUNTERROR_LEN 1
+#define MMC_RISR_RXJABBERERROR_POS 7
+#define MMC_RISR_RXJABBERERROR_LEN 1
+#define MMC_RISR_RXUNDERSIZE_G_POS 8
+#define MMC_RISR_RXUNDERSIZE_G_LEN 1
+#define MMC_RISR_RXOVERSIZE_G_POS 9
+#define MMC_RISR_RXOVERSIZE_G_LEN 1
+#define MMC_RISR_RX64OCTETS_GB_POS 10
+#define MMC_RISR_RX64OCTETS_GB_LEN 1
+#define MMC_RISR_RX65TO127OCTETS_GB_POS 11
+#define MMC_RISR_RX65TO127OCTETS_GB_LEN 1
+#define MMC_RISR_RX128TO255OCTETS_GB_POS 12
+#define MMC_RISR_RX128TO255OCTETS_GB_LEN 1
+#define MMC_RISR_RX256TO511OCTETS_GB_POS 13
+#define MMC_RISR_RX256TO511OCTETS_GB_LEN 1
+#define MMC_RISR_RX512TO1023OCTETS_GB_POS 14
+#define MMC_RISR_RX512TO1023OCTETS_GB_LEN 1
+#define MMC_RISR_RX1024TOMAXOCTETS_GB_POS 15
+#define MMC_RISR_RX1024TOMAXOCTETS_GB_LEN 1
+#define MMC_RISR_RXUNICASTFRAMES_G_POS 16
+#define MMC_RISR_RXUNICASTFRAMES_G_LEN 1
+#define MMC_RISR_RXLENGTHERROR_POS 17
+#define MMC_RISR_RXLENGTHERROR_LEN 1
+#define MMC_RISR_RXOUTOFRANGETYPE_POS 18
+#define MMC_RISR_RXOUTOFRANGETYPE_LEN 1
+#define MMC_RISR_RXPAUSEFRAMES_POS 19
+#define MMC_RISR_RXPAUSEFRAMES_LEN 1
+#define MMC_RISR_RXFIFOOVERFLOW_POS 20
+#define MMC_RISR_RXFIFOOVERFLOW_LEN 1
+#define MMC_RISR_RXVLANFRAMES_GB_POS 21
+#define MMC_RISR_RXVLANFRAMES_GB_LEN 1
+#define MMC_RISR_RXWATCHDOGERROR_POS 22
+#define MMC_RISR_RXWATCHDOGERROR_LEN 1
+#define MMC_TIER_ALL_INTERRUPTS_POS 0
+#define MMC_TIER_ALL_INTERRUPTS_LEN 18
+#define MMC_TISR_TXOCTETCOUNT_GB_POS 0
+#define MMC_TISR_TXOCTETCOUNT_GB_LEN 1
+#define MMC_TISR_TXFRAMECOUNT_GB_POS 1
+#define MMC_TISR_TXFRAMECOUNT_GB_LEN 1
+#define MMC_TISR_TXBROADCASTFRAMES_G_POS 2
+#define MMC_TISR_TXBROADCASTFRAMES_G_LEN 1
+#define MMC_TISR_TXMULTICASTFRAMES_G_POS 3
+#define MMC_TISR_TXMULTICASTFRAMES_G_LEN 1
+#define MMC_TISR_TX64OCTETS_GB_POS 4
+#define MMC_TISR_TX64OCTETS_GB_LEN 1
+#define MMC_TISR_TX65TO127OCTETS_GB_POS 5
+#define MMC_TISR_TX65TO127OCTETS_GB_LEN 1
+#define MMC_TISR_TX128TO255OCTETS_GB_POS 6
+#define MMC_TISR_TX128TO255OCTETS_GB_LEN 1
+#define MMC_TISR_TX256TO511OCTETS_GB_POS 7
+#define MMC_TISR_TX256TO511OCTETS_GB_LEN 1
+#define MMC_TISR_TX512TO1023OCTETS_GB_POS 8
+#define MMC_TISR_TX512TO1023OCTETS_GB_LEN 1
+#define MMC_TISR_TX1024TOMAXOCTETS_GB_POS 9
+#define MMC_TISR_TX1024TOMAXOCTETS_GB_LEN 1
+#define MMC_TISR_TXUNICASTFRAMES_GB_POS 10
+#define MMC_TISR_TXUNICASTFRAMES_GB_LEN 1
+#define MMC_TISR_TXMULTICASTFRAMES_GB_POS 11
+#define MMC_TISR_TXMULTICASTFRAMES_GB_LEN 1
+#define MMC_TISR_TXBROADCASTFRAMES_GB_POS 12
+#define MMC_TISR_TXBROADCASTFRAMES_GB_LEN 1
+#define MMC_TISR_TXUNDERFLOWERROR_POS 13
+#define MMC_TISR_TXUNDERFLOWERROR_LEN 1
+#define MMC_TISR_TXOCTETCOUNT_G_POS 14
+#define MMC_TISR_TXOCTETCOUNT_G_LEN 1
+#define MMC_TISR_TXFRAMECOUNT_G_POS 15
+#define MMC_TISR_TXFRAMECOUNT_G_LEN 1
+#define MMC_TISR_TXPAUSEFRAMES_POS 16
+#define MMC_TISR_TXPAUSEFRAMES_LEN 1
+#define MMC_TISR_TXVLANFRAMES_G_POS 17
+#define MMC_TISR_TXVLANFRAMES_G_LEN 1
+
+/* MTL register offsets */
+#define MTL_OMR 0x1000
+#define MTL_FDDR 0x1010
+#define MTL_RQDCM0R 0x1030
+
+#define MTL_RQDCM_INC 4
+#define MTL_RQDCM_Q_PER_REG 4
+
+/* MTL register entry bit positions and sizes */
+#define MTL_OMR_ETSALG_POS 5
+#define MTL_OMR_ETSALG_LEN 2
+#define MTL_OMR_RAA_POS 2
+#define MTL_OMR_RAA_LEN 1
+
+/* MTL queue register offsets
+ * Multiple queues can be active. The first queue has registers
+ * that begin at 0x1100. Each subsequent queue has registers that
+ * are accessed using an offset of 0x80 from the previous queue.
+ */
+#define MTL_Q_BASE 0x1100
+#define MTL_Q_INC 0x80
+
+#define MTL_Q_TQOMR 0x00
+#define MTL_Q_RQOMR 0x40
+#define MTL_Q_RQDR 0x48
+#define MTL_Q_RQFCR 0x50
+#define MTL_Q_IER 0x70
+#define MTL_Q_ISR 0x74
+
+/* MTL queue register entry bit positions and sizes */
+#define MTL_Q_RQDR_PRXQ_POS 16
+#define MTL_Q_RQDR_PRXQ_LEN 14
+#define MTL_Q_RQDR_RXQSTS_POS 4
+#define MTL_Q_RQDR_RXQSTS_LEN 2
+#define MTL_Q_RQFCR_RFA_POS 1
+#define MTL_Q_RQFCR_RFA_LEN 6
+#define MTL_Q_RQFCR_RFD_POS 17
+#define MTL_Q_RQFCR_RFD_LEN 6
+#define MTL_Q_RQOMR_EHFC_POS 7
+#define MTL_Q_RQOMR_EHFC_LEN 1
+#define MTL_Q_RQOMR_RQS_POS 16
+#define MTL_Q_RQOMR_RQS_LEN 9
+#define MTL_Q_RQOMR_RSF_POS 5
+#define MTL_Q_RQOMR_RSF_LEN 1
+#define MTL_Q_RQOMR_FEP_POS 4
+#define MTL_Q_RQOMR_FEP_LEN 1
+#define MTL_Q_RQOMR_FUP_POS 3
+#define MTL_Q_RQOMR_FUP_LEN 1
+#define MTL_Q_RQOMR_RTC_POS 0
+#define MTL_Q_RQOMR_RTC_LEN 2
+#define MTL_Q_TQOMR_FTQ_POS 0
+#define MTL_Q_TQOMR_FTQ_LEN 1
+#define MTL_Q_TQOMR_Q2TCMAP_POS 8
+#define MTL_Q_TQOMR_Q2TCMAP_LEN 3
+#define MTL_Q_TQOMR_TQS_POS 16
+#define MTL_Q_TQOMR_TQS_LEN 10
+#define MTL_Q_TQOMR_TSF_POS 1
+#define MTL_Q_TQOMR_TSF_LEN 1
+#define MTL_Q_TQOMR_TTC_POS 4
+#define MTL_Q_TQOMR_TTC_LEN 3
+#define MTL_Q_TQOMR_TXQEN_POS 2
+#define MTL_Q_TQOMR_TXQEN_LEN 2
+
+/* MTL queue register value */
+#define MTL_RSF_DISABLE 0x00
+#define MTL_RSF_ENABLE 0x01
+#define MTL_TSF_DISABLE 0x00
+#define MTL_TSF_ENABLE 0x01
+
+#define MTL_RX_THRESHOLD_64 0x00
+#define MTL_RX_THRESHOLD_96 0x02
+#define MTL_RX_THRESHOLD_128 0x03
+#define MTL_TX_THRESHOLD_64 0x00
+#define MTL_TX_THRESHOLD_96 0x02
+#define MTL_TX_THRESHOLD_128 0x03
+#define MTL_TX_THRESHOLD_192 0x04
+#define MTL_TX_THRESHOLD_256 0x05
+#define MTL_TX_THRESHOLD_384 0x06
+#define MTL_TX_THRESHOLD_512 0x07
+
+#define MTL_ETSALG_WRR 0x00
+#define MTL_ETSALG_WFQ 0x01
+#define MTL_ETSALG_DWRR 0x02
+#define MTL_RAA_SP 0x00
+#define MTL_RAA_WSP 0x01
+
+#define MTL_Q_DISABLED 0x00
+#define MTL_Q_ENABLED 0x02
+
+#define MTL_RQDCM0R_Q0MDMACH 0x0
+#define MTL_RQDCM0R_Q1MDMACH 0x00000100
+#define MTL_RQDCM0R_Q2MDMACH 0x00020000
+#define MTL_RQDCM0R_Q3MDMACH 0x03000000
+#define MTL_RQDCM1R_Q4MDMACH 0x00000004
+#define MTL_RQDCM1R_Q5MDMACH 0x00000500
+#define MTL_RQDCM1R_Q6MDMACH 0x00060000
+#define MTL_RQDCM1R_Q7MDMACH 0x07000000
+#define MTL_RQDCM2R_Q8MDMACH 0x00000008
+#define MTL_RQDCM2R_Q9MDMACH 0x00000900
+#define MTL_RQDCM2R_Q10MDMACH 0x000A0000
+#define MTL_RQDCM2R_Q11MDMACH 0x0B000000
+
+/* MTL traffic class register offsets
+ * Multiple traffic classes can be active. The first class has registers
+ * that begin at 0x1100. Each subsequent queue has registers that
+ * are accessed using an offset of 0x80 from the previous queue.
+ */
+#define MTL_TC_BASE MTL_Q_BASE
+#define MTL_TC_INC MTL_Q_INC
+
+#define MTL_TC_ETSCR 0x10
+#define MTL_TC_ETSSR 0x14
+#define MTL_TC_QWR 0x18
+
+/* MTL traffic class register entry bit positions and sizes */
+#define MTL_TC_ETSCR_TSA_POS 0
+#define MTL_TC_ETSCR_TSA_LEN 2
+#define MTL_TC_QWR_QW_POS 0
+#define MTL_TC_QWR_QW_LEN 21
+
+/* MTL traffic class register value */
+#define MTL_TSA_SP 0x00
+#define MTL_TSA_ETS 0x02
+
+/* DMA register offsets */
+#define DMA_MR 0x3000
+#define DMA_SBMR 0x3004
+#define DMA_ISR 0x3008
+#define DMA_DSR0 0x3020
+#define DMA_DSR1 0x3024
+
+/* DMA register entry bit positions and sizes */
+#define DMA_ISR_MACIS_POS 17
+#define DMA_ISR_MACIS_LEN 1
+#define DMA_ISR_MTLIS_POS 16
+#define DMA_ISR_MTLIS_LEN 1
+#define DMA_MR_SWR_POS 0
+#define DMA_MR_SWR_LEN 1
+#define DMA_SBMR_EAME_POS 11
+#define DMA_SBMR_EAME_LEN 1
+#define DMA_SBMR_BLEN_64_POS 5
+#define DMA_SBMR_BLEN_64_LEN 1
+#define DMA_SBMR_BLEN_128_POS 6
+#define DMA_SBMR_BLEN_128_LEN 1
+#define DMA_SBMR_BLEN_256_POS 7
+#define DMA_SBMR_BLEN_256_LEN 1
+#define DMA_SBMR_UNDEF_POS 0
+#define DMA_SBMR_UNDEF_LEN 1
+
+/* DMA register values */
+#define DMA_DSR_RPS_LEN 4
+#define DMA_DSR_TPS_LEN 4
+#define DMA_DSR_Q_LEN (DMA_DSR_RPS_LEN + DMA_DSR_TPS_LEN)
+#define DMA_DSR0_TPS_START 12
+#define DMA_DSRX_FIRST_QUEUE 3
+#define DMA_DSRX_INC 4
+#define DMA_DSRX_QPR 4
+#define DMA_DSRX_TPS_START 4
+#define DMA_TPS_STOPPED 0x00
+#define DMA_TPS_SUSPENDED 0x06
+
+/* DMA channel register offsets
+ * Multiple channels can be active. The first channel has registers
+ * that begin at 0x3100. Each subsequent channel has registers that
+ * are accessed using an offset of 0x80 from the previous channel.
+ */
+#define DMA_CH_BASE 0x3100
+#define DMA_CH_INC 0x80
+
+#define DMA_CH_CR 0x00
+#define DMA_CH_TCR 0x04
+#define DMA_CH_RCR 0x08
+#define DMA_CH_TDLR_HI 0x10
+#define DMA_CH_TDLR_LO 0x14
+#define DMA_CH_RDLR_HI 0x18
+#define DMA_CH_RDLR_LO 0x1c
+#define DMA_CH_TDTR_LO 0x24
+#define DMA_CH_RDTR_LO 0x2c
+#define DMA_CH_TDRLR 0x30
+#define DMA_CH_RDRLR 0x34
+#define DMA_CH_IER 0x38
+#define DMA_CH_RIWT 0x3c
+#define DMA_CH_SR 0x60
+
+/* DMA channel register entry bit positions and sizes */
+#define DMA_CH_CR_PBLX8_POS 16
+#define DMA_CH_CR_PBLX8_LEN 1
+#define DMA_CH_CR_SPH_POS 24
+#define DMA_CH_CR_SPH_LEN 1
+#define DMA_CH_IER_AIE_POS 15
+#define DMA_CH_IER_AIE_LEN 1
+#define DMA_CH_IER_FBEE_POS 12
+#define DMA_CH_IER_FBEE_LEN 1
+#define DMA_CH_IER_NIE_POS 16
+#define DMA_CH_IER_NIE_LEN 1
+#define DMA_CH_IER_RBUE_POS 7
+#define DMA_CH_IER_RBUE_LEN 1
+#define DMA_CH_IER_RIE_POS 6
+#define DMA_CH_IER_RIE_LEN 1
+#define DMA_CH_IER_RSE_POS 8
+#define DMA_CH_IER_RSE_LEN 1
+#define DMA_CH_IER_TBUE_POS 2
+#define DMA_CH_IER_TBUE_LEN 1
+#define DMA_CH_IER_TIE_POS 0
+#define DMA_CH_IER_TIE_LEN 1
+#define DMA_CH_IER_TXSE_POS 1
+#define DMA_CH_IER_TXSE_LEN 1
+#define DMA_CH_RCR_PBL_POS 16
+#define DMA_CH_RCR_PBL_LEN 6
+#define DMA_CH_RCR_RBSZ_POS 1
+#define DMA_CH_RCR_RBSZ_LEN 14
+#define DMA_CH_RCR_SR_POS 0
+#define DMA_CH_RCR_SR_LEN 1
+#define DMA_CH_RIWT_RWT_POS 0
+#define DMA_CH_RIWT_RWT_LEN 8
+#define DMA_CH_SR_FBE_POS 12
+#define DMA_CH_SR_FBE_LEN 1
+#define DMA_CH_SR_RBU_POS 7
+#define DMA_CH_SR_RBU_LEN 1
+#define DMA_CH_SR_RI_POS 6
+#define DMA_CH_SR_RI_LEN 1
+#define DMA_CH_SR_RPS_POS 8
+#define DMA_CH_SR_RPS_LEN 1
+#define DMA_CH_SR_TBU_POS 2
+#define DMA_CH_SR_TBU_LEN 1
+#define DMA_CH_SR_TI_POS 0
+#define DMA_CH_SR_TI_LEN 1
+#define DMA_CH_SR_TPS_POS 1
+#define DMA_CH_SR_TPS_LEN 1
+#define DMA_CH_TCR_OSP_POS 4
+#define DMA_CH_TCR_OSP_LEN 1
+#define DMA_CH_TCR_PBL_POS 16
+#define DMA_CH_TCR_PBL_LEN 6
+#define DMA_CH_TCR_ST_POS 0
+#define DMA_CH_TCR_ST_LEN 1
+#define DMA_CH_TCR_TSE_POS 12
+#define DMA_CH_TCR_TSE_LEN 1
+
+/* DMA channel register values */
+#define DMA_OSP_DISABLE 0x00
+#define DMA_OSP_ENABLE 0x01
+#define DMA_PBL_1 1
+#define DMA_PBL_2 2
+#define DMA_PBL_4 4
+#define DMA_PBL_8 8
+#define DMA_PBL_16 16
+#define DMA_PBL_32 32
+#define DMA_PBL_64 64
+#define DMA_PBL_128 128
+#define DMA_PBL_256 256
+#define DMA_PBL_X8_DISABLE 0x00
+#define DMA_PBL_X8_ENABLE 0x01
+
+/* Descriptor/Packet entry bit positions and sizes */
+#define RX_PACKET_ERRORS_CRC_POS 2
+#define RX_PACKET_ERRORS_CRC_LEN 1
+#define RX_PACKET_ERRORS_FRAME_POS 3
+#define RX_PACKET_ERRORS_FRAME_LEN 1
+#define RX_PACKET_ERRORS_LENGTH_POS 0
+#define RX_PACKET_ERRORS_LENGTH_LEN 1
+#define RX_PACKET_ERRORS_OVERRUN_POS 1
+#define RX_PACKET_ERRORS_OVERRUN_LEN 1
+
+#define RX_PACKET_ATTRIBUTES_CSUM_DONE_POS 0
+#define RX_PACKET_ATTRIBUTES_CSUM_DONE_LEN 1
+#define RX_PACKET_ATTRIBUTES_VLAN_CTAG_POS 1
+#define RX_PACKET_ATTRIBUTES_VLAN_CTAG_LEN 1
+#define RX_PACKET_ATTRIBUTES_INCOMPLETE_POS 2
+#define RX_PACKET_ATTRIBUTES_INCOMPLETE_LEN 1
+#define RX_PACKET_ATTRIBUTES_CONTEXT_NEXT_POS 3
+#define RX_PACKET_ATTRIBUTES_CONTEXT_NEXT_LEN 1
+#define RX_PACKET_ATTRIBUTES_CONTEXT_POS 4
+#define RX_PACKET_ATTRIBUTES_CONTEXT_LEN 1
+#define RX_PACKET_ATTRIBUTES_RX_TSTAMP_POS 5
+#define RX_PACKET_ATTRIBUTES_RX_TSTAMP_LEN 1
+#define RX_PACKET_ATTRIBUTES_RSS_HASH_POS 6
+#define RX_PACKET_ATTRIBUTES_RSS_HASH_LEN 1
+
+#define RX_NORMAL_DESC0_OVT_POS 0
+#define RX_NORMAL_DESC0_OVT_LEN 16
+#define RX_NORMAL_DESC2_HL_POS 0
+#define RX_NORMAL_DESC2_HL_LEN 10
+#define RX_NORMAL_DESC3_CDA_POS 27
+#define RX_NORMAL_DESC3_CDA_LEN 1
+#define RX_NORMAL_DESC3_CTXT_POS 30
+#define RX_NORMAL_DESC3_CTXT_LEN 1
+#define RX_NORMAL_DESC3_ES_POS 15
+#define RX_NORMAL_DESC3_ES_LEN 1
+#define RX_NORMAL_DESC3_ETLT_POS 16
+#define RX_NORMAL_DESC3_ETLT_LEN 4
+#define RX_NORMAL_DESC3_FD_POS 29
+#define RX_NORMAL_DESC3_FD_LEN 1
+#define RX_NORMAL_DESC3_INTE_POS 30
+#define RX_NORMAL_DESC3_INTE_LEN 1
+#define RX_NORMAL_DESC3_L34T_POS 20
+#define RX_NORMAL_DESC3_L34T_LEN 4
+#define RX_NORMAL_DESC3_LD_POS 28
+#define RX_NORMAL_DESC3_LD_LEN 1
+#define RX_NORMAL_DESC3_OWN_POS 31
+#define RX_NORMAL_DESC3_OWN_LEN 1
+#define RX_NORMAL_DESC3_PL_POS 0
+#define RX_NORMAL_DESC3_PL_LEN 14
+#define RX_NORMAL_DESC3_RSV_POS 26
+#define RX_NORMAL_DESC3_RSV_LEN 1
+
+#define RX_DESC3_L34T_IPV4_TCP 1
+#define RX_DESC3_L34T_IPV4_UDP 2
+#define RX_DESC3_L34T_IPV4_ICMP 3
+#define RX_DESC3_L34T_IPV6_TCP 9
+#define RX_DESC3_L34T_IPV6_UDP 10
+#define RX_DESC3_L34T_IPV6_ICMP 11
+
+#define RX_CONTEXT_DESC3_TSA_POS 4
+#define RX_CONTEXT_DESC3_TSA_LEN 1
+#define RX_CONTEXT_DESC3_TSD_POS 6
+#define RX_CONTEXT_DESC3_TSD_LEN 1
+
+#define TX_PACKET_ATTRIBUTES_CSUM_ENABLE_POS 0
+#define TX_PACKET_ATTRIBUTES_CSUM_ENABLE_LEN 1
+#define TX_PACKET_ATTRIBUTES_TSO_ENABLE_POS 1
+#define TX_PACKET_ATTRIBUTES_TSO_ENABLE_LEN 1
+#define TX_PACKET_ATTRIBUTES_VLAN_CTAG_POS 2
+#define TX_PACKET_ATTRIBUTES_VLAN_CTAG_LEN 1
+#define TX_PACKET_ATTRIBUTES_PTP_POS 3
+#define TX_PACKET_ATTRIBUTES_PTP_LEN 1
+
+#define TX_CONTEXT_DESC2_MSS_POS 0
+#define TX_CONTEXT_DESC2_MSS_LEN 15
+#define TX_CONTEXT_DESC3_CTXT_POS 30
+#define TX_CONTEXT_DESC3_CTXT_LEN 1
+#define TX_CONTEXT_DESC3_TCMSSV_POS 26
+#define TX_CONTEXT_DESC3_TCMSSV_LEN 1
+#define TX_CONTEXT_DESC3_VLTV_POS 16
+#define TX_CONTEXT_DESC3_VLTV_LEN 1
+#define TX_CONTEXT_DESC3_VT_POS 0
+#define TX_CONTEXT_DESC3_VT_LEN 16
+
+#define TX_NORMAL_DESC2_HL_B1L_POS 0
+#define TX_NORMAL_DESC2_HL_B1L_LEN 14
+#define TX_NORMAL_DESC2_IC_POS 31
+#define TX_NORMAL_DESC2_IC_LEN 1
+#define TX_NORMAL_DESC2_TTSE_POS 30
+#define TX_NORMAL_DESC2_TTSE_LEN 1
+#define TX_NORMAL_DESC2_VTIR_POS 14
+#define TX_NORMAL_DESC2_VTIR_LEN 2
+#define TX_NORMAL_DESC3_CIC_POS 16
+#define TX_NORMAL_DESC3_CIC_LEN 2
+#define TX_NORMAL_DESC3_CPC_POS 26
+#define TX_NORMAL_DESC3_CPC_LEN 2
+#define TX_NORMAL_DESC3_CTXT_POS 30
+#define TX_NORMAL_DESC3_CTXT_LEN 1
+#define TX_NORMAL_DESC3_FD_POS 29
+#define TX_NORMAL_DESC3_FD_LEN 1
+#define TX_NORMAL_DESC3_FL_POS 0
+#define TX_NORMAL_DESC3_FL_LEN 15
+#define TX_NORMAL_DESC3_LD_POS 28
+#define TX_NORMAL_DESC3_LD_LEN 1
+#define TX_NORMAL_DESC3_OWN_POS 31
+#define TX_NORMAL_DESC3_OWN_LEN 1
+#define TX_NORMAL_DESC3_TCPHDRLEN_POS 19
+#define TX_NORMAL_DESC3_TCPHDRLEN_LEN 4
+#define TX_NORMAL_DESC3_TCPPL_POS 0
+#define TX_NORMAL_DESC3_TCPPL_LEN 18
+#define TX_NORMAL_DESC3_TSE_POS 18
+#define TX_NORMAL_DESC3_TSE_LEN 1
+
+#define TX_NORMAL_DESC2_VLAN_INSERT 0x2
+
+#define XLGMAC_MTL_REG(pdata, n, reg) \
+ ((pdata)->mac_regs + MTL_Q_BASE + ((n) * MTL_Q_INC) + (reg))
+
+#define XLGMAC_DMA_REG(channel, reg) ((channel)->dma_regs + (reg))
+
+#endif /* __DWC_XLGMAC_REG_H__ */
diff --git a/drivers/net/ethernet/synopsys/dwc-xlgmac.h b/drivers/net/ethernet/synopsys/dwc-xlgmac.h
new file mode 100644
index 0000000..cab3e40
--- /dev/null
+++ b/drivers/net/ethernet/synopsys/dwc-xlgmac.h
@@ -0,0 +1,660 @@
+/* Synopsys DesignWare Core Enterprise Ethernet (XLGMAC) Driver
+ *
+ * Copyright (c) 2017 Synopsys, Inc. (www.synopsys.com)
+ *
+ * This program is dual-licensed; you may select either version 2 of
+ * the GNU General Public License ("GPL") or BSD license ("BSD").
+ *
+ * This Synopsys DWC XLGMAC software driver and associated documentation
+ * (hereinafter the "Software") is an unsupported proprietary work of
+ * Synopsys, Inc. unless otherwise expressly agreed to in writing between
+ * Synopsys and you. The Software IS NOT an item of Licensed Software or a
+ * Licensed Product under any End User Software License Agreement or
+ * Agreement for Licensed Products with Synopsys or any supplement thereto.
+ * Synopsys is a registered trademark of Synopsys, Inc. Other names included
+ * in the SOFTWARE may be the trademarks of their respective owners.
+ */
+
+#ifndef __DWC_XLGMAC_H__
+#define __DWC_XLGMAC_H__
+
+#include <linux/dma-mapping.h>
+#include <linux/netdevice.h>
+#include <linux/workqueue.h>
+#include <linux/phy.h>
+#include <linux/if_vlan.h>
+#include <linux/bitops.h>
+#include <linux/timecounter.h>
+
+#define XLGMAC_DRV_NAME "dwc-xlgmac"
+#define XLGMAC_DRV_VERSION "1.0.0"
+#define XLGMAC_DRV_DESC "Synopsys DWC XLGMAC Driver"
+
+/* Descriptor related parameters */
+#define XLGMAC_TX_DESC_CNT 1024
+#define XLGMAC_TX_DESC_MIN_FREE (XLGMAC_TX_DESC_CNT >> 3)
+#define XLGMAC_TX_DESC_MAX_PROC (XLGMAC_TX_DESC_CNT >> 1)
+#define XLGMAC_RX_DESC_CNT 1024
+#define XLGMAC_RX_DESC_MAX_DIRTY (XLGMAC_RX_DESC_CNT >> 3)
+
+/* Descriptors required for maximum contiguous TSO/GSO packet */
+#define XLGMAC_TX_MAX_SPLIT ((GSO_MAX_SIZE / XLGMAC_TX_MAX_BUF_SIZE) + 1)
+
+/* Maximum possible descriptors needed for a SKB */
+#define XLGMAC_TX_MAX_DESC_NR (MAX_SKB_FRAGS + XLGMAC_TX_MAX_SPLIT + 2)
+
+#define XLGMAC_TX_MAX_BUF_SIZE (0x3fff & ~(64 - 1))
+#define XLGMAC_RX_MIN_BUF_SIZE (ETH_FRAME_LEN + ETH_FCS_LEN + VLAN_HLEN)
+#define XLGMAC_RX_BUF_ALIGN 64
+
+/* Maximum Size for Splitting the Header Data
+ * Keep in sync with SKB_ALLOC_SIZE
+ * 3'b000: 64 bytes, 3'b001: 128 bytes
+ * 3'b010: 256 bytes, 3'b011: 512 bytes
+ * 3'b100: 1023 bytes , 3'b101'3'b111: Reserved
+ */
+#define XLGMAC_SPH_HDSMS_SIZE 3
+#define XLGMAC_SKB_ALLOC_SIZE 512
+
+#define XLGMAC_MAX_FIFO 81920
+
+#define XLGMAC_MAX_DMA_CHANNELS 16
+#define XLGMAC_DMA_STOP_TIMEOUT 5
+#define XLGMAC_DMA_INTERRUPT_MASK 0x31c7
+
+/* Default coalescing parameters */
+#define XLGMAC_INIT_DMA_TX_USECS 1000
+#define XLGMAC_INIT_DMA_TX_FRAMES 25
+#define XLGMAC_INIT_DMA_RX_USECS 30
+#define XLGMAC_INIT_DMA_RX_FRAMES 25
+#define XLGMAC_MAX_DMA_RIWT 0xff
+#define XLGMAC_MIN_DMA_RIWT 0x01
+
+/* Flow control queue count */
+#define XLGMAC_MAX_FLOW_CONTROL_QUEUES 8
+
+/* System clock is 125 MHz */
+#define XLGMAC_SYSCLOCK 125000000
+
+/* Maximum MAC address hash table size (256 bits = 8 bytes) */
+#define XLGMAC_MAC_HASH_TABLE_SIZE 8
+
+/* Receive Side Scaling */
+#define XLGMAC_RSS_HASH_KEY_SIZE 40
+#define XLGMAC_RSS_MAX_TABLE_SIZE 256
+#define XLGMAC_RSS_LOOKUP_TABLE_TYPE 0
+#define XLGMAC_RSS_HASH_KEY_TYPE 1
+
+#define XLGMAC_STD_PACKET_MTU 1500
+#define XLGMAC_JUMBO_PACKET_MTU 9000
+
+/* Helper macro for descriptor handling
+ * Always use XLGMAC_GET_DESC_DATA to access the descriptor data
+ */
+#define XLGMAC_GET_DESC_DATA(ring, idx) ({ \
+ typeof(ring) _ring = (ring); \
+ ((_ring)->desc_data_head + \
+ ((idx) & ((_ring)->dma_desc_count - 1))); \
+})
+
+#define XLGMAC_GET_REG_BITS(var, pos, len) ({ \
+ typeof(pos) _pos = (pos); \
+ typeof(len) _len = (len); \
+ ((var) & GENMASK(_pos + _len - 1, _pos)) >> (_pos); \
+})
+
+#define XLGMAC_GET_REG_BITS_LE(var, pos, len) ({ \
+ typeof(pos) _pos = (pos); \
+ typeof(len) _len = (len); \
+ typeof(var) _var = le32_to_cpu((var)); \
+ ((_var) & GENMASK(_pos + _len - 1, _pos)) >> (_pos); \
+})
+
+#define XLGMAC_SET_REG_BITS(var, pos, len, val) ({ \
+ typeof(var) _var = (var); \
+ typeof(pos) _pos = (pos); \
+ typeof(len) _len = (len); \
+ typeof(val) _val = (val); \
+ _val = (_val << _pos) & GENMASK(_pos + _len - 1, _pos); \
+ _var = (_var & ~GENMASK(_pos + _len - 1, _pos)) | _val; \
+})
+
+#define XLGMAC_SET_REG_BITS_LE(var, pos, len, val) ({ \
+ typeof(var) _var = (var); \
+ typeof(pos) _pos = (pos); \
+ typeof(len) _len = (len); \
+ typeof(val) _val = (val); \
+ _val = (_val << _pos) & GENMASK(_pos + _len - 1, _pos); \
+ _var = (_var & ~GENMASK(_pos + _len - 1, _pos)) | _val; \
+ cpu_to_le32(_var); \
+})
+
+struct xlgmac_pdata;
+
+enum xlgmac_int {
+ XLGMAC_INT_DMA_CH_SR_TI,
+ XLGMAC_INT_DMA_CH_SR_TPS,
+ XLGMAC_INT_DMA_CH_SR_TBU,
+ XLGMAC_INT_DMA_CH_SR_RI,
+ XLGMAC_INT_DMA_CH_SR_RBU,
+ XLGMAC_INT_DMA_CH_SR_RPS,
+ XLGMAC_INT_DMA_CH_SR_TI_RI,
+ XLGMAC_INT_DMA_CH_SR_FBE,
+ XLGMAC_INT_DMA_ALL,
+};
+
+struct xlgmac_stats {
+ /* MMC TX counters */
+ u64 txoctetcount_gb;
+ u64 txframecount_gb;
+ u64 txbroadcastframes_g;
+ u64 txmulticastframes_g;
+ u64 tx64octets_gb;
+ u64 tx65to127octets_gb;
+ u64 tx128to255octets_gb;
+ u64 tx256to511octets_gb;
+ u64 tx512to1023octets_gb;
+ u64 tx1024tomaxoctets_gb;
+ u64 txunicastframes_gb;
+ u64 txmulticastframes_gb;
+ u64 txbroadcastframes_gb;
+ u64 txunderflowerror;
+ u64 txoctetcount_g;
+ u64 txframecount_g;
+ u64 txpauseframes;
+ u64 txvlanframes_g;
+
+ /* MMC RX counters */
+ u64 rxframecount_gb;
+ u64 rxoctetcount_gb;
+ u64 rxoctetcount_g;
+ u64 rxbroadcastframes_g;
+ u64 rxmulticastframes_g;
+ u64 rxcrcerror;
+ u64 rxrunterror;
+ u64 rxjabbererror;
+ u64 rxundersize_g;
+ u64 rxoversize_g;
+ u64 rx64octets_gb;
+ u64 rx65to127octets_gb;
+ u64 rx128to255octets_gb;
+ u64 rx256to511octets_gb;
+ u64 rx512to1023octets_gb;
+ u64 rx1024tomaxoctets_gb;
+ u64 rxunicastframes_g;
+ u64 rxlengtherror;
+ u64 rxoutofrangetype;
+ u64 rxpauseframes;
+ u64 rxfifooverflow;
+ u64 rxvlanframes_gb;
+ u64 rxwatchdogerror;
+
+ /* Extra counters */
+ u64 tx_tso_packets;
+ u64 rx_split_header_packets;
+ u64 tx_process_stopped;
+ u64 rx_process_stopped;
+ u64 tx_buffer_unavailable;
+ u64 rx_buffer_unavailable;
+ u64 fatal_bus_error;
+ u64 tx_vlan_packets;
+ u64 rx_vlan_packets;
+ u64 napi_poll_isr;
+ u64 napi_poll_txtimer;
+};
+
+struct xlgmac_ring_buf {
+ struct sk_buff *skb;
+ dma_addr_t skb_dma;
+ unsigned int skb_len;
+};
+
+/* Common Tx and Rx DMA hardware descriptor */
+struct xlgmac_dma_desc {
+ __le32 desc0;
+ __le32 desc1;
+ __le32 desc2;
+ __le32 desc3;
+};
+
+/* Page allocation related values */
+struct xlgmac_page_alloc {
+ struct page *pages;
+ unsigned int pages_len;
+ unsigned int pages_offset;
+
+ dma_addr_t pages_dma;
+};
+
+/* Ring entry buffer data */
+struct xlgmac_buffer_data {
+ struct xlgmac_page_alloc pa;
+ struct xlgmac_page_alloc pa_unmap;
+
+ dma_addr_t dma_base;
+ unsigned long dma_off;
+ unsigned int dma_len;
+};
+
+/* Tx-related desc data */
+struct xlgmac_tx_desc_data {
+ unsigned int packets; /* BQL packet count */
+ unsigned int bytes; /* BQL byte count */
+};
+
+/* Rx-related desc data */
+struct xlgmac_rx_desc_data {
+ struct xlgmac_buffer_data hdr; /* Header locations */
+ struct xlgmac_buffer_data buf; /* Payload locations */
+
+ unsigned short hdr_len; /* Length of received header */
+ unsigned short len; /* Length of received packet */
+};
+
+struct xlgmac_pkt_info {
+ struct sk_buff *skb;
+
+ unsigned int attributes;
+
+ unsigned int errors;
+
+ /* descriptors needed for this packet */
+ unsigned int desc_count;
+ unsigned int length;
+
+ unsigned int tx_packets;
+ unsigned int tx_bytes;
+
+ unsigned int header_len;
+ unsigned int tcp_header_len;
+ unsigned int tcp_payload_len;
+ unsigned short mss;
+
+ unsigned short vlan_ctag;
+
+ u64 rx_tstamp;
+
+ u32 rss_hash;
+ enum pkt_hash_types rss_hash_type;
+};
+
+struct xlgmac_desc_data {
+ /* dma_desc: Virtual address of descriptor
+ * dma_desc_addr: DMA address of descriptor
+ */
+ struct xlgmac_dma_desc *dma_desc;
+ dma_addr_t dma_desc_addr;
+
+ /* skb: Virtual address of SKB
+ * skb_dma: DMA address of SKB data
+ * skb_dma_len: Length of SKB DMA area
+ */
+ struct sk_buff *skb;
+ dma_addr_t skb_dma;
+ unsigned int skb_dma_len;
+
+ /* Tx/Rx -related data */
+ struct xlgmac_tx_desc_data tx;
+ struct xlgmac_rx_desc_data rx;
+
+ unsigned int mapped_as_page;
+
+ /* Incomplete receive save location. If the budget is exhausted
+ * or the last descriptor (last normal descriptor or a following
+ * context descriptor) has not been DMA'd yet the current state
+ * of the receive processing needs to be saved.
+ */
+ unsigned int state_saved;
+ struct {
+ struct sk_buff *skb;
+ unsigned int len;
+ unsigned int error;
+ } state;
+};
+
+struct xlgmac_ring {
+ /* Per packet related information */
+ struct xlgmac_pkt_info pkt_info;
+
+ /* Virtual/DMA addresses of DMA descriptor list and the total count */
+ struct xlgmac_dma_desc *dma_desc_head;
+ dma_addr_t dma_desc_head_addr;
+ unsigned int dma_desc_count;
+
+ /* Array of descriptor data corresponding the DMA descriptor
+ * (always use the XLGMAC_GET_DESC_DATA macro to access this data)
+ */
+ struct xlgmac_desc_data *desc_data_head;
+
+ /* Page allocation for RX buffers */
+ struct xlgmac_page_alloc rx_hdr_pa;
+ struct xlgmac_page_alloc rx_buf_pa;
+
+ /* Ring index values
+ * cur - Tx: index of descriptor to be used for current transfer
+ * Rx: index of descriptor to check for packet availability
+ * dirty - Tx: index of descriptor to check for transfer complete
+ * Rx: index of descriptor to check for buffer reallocation
+ */
+ unsigned int cur;
+ unsigned int dirty;
+
+ /* Coalesce frame count used for interrupt bit setting */
+ unsigned int coalesce_count;
+
+ union {
+ struct {
+ unsigned int xmit_more;
+ unsigned int queue_stopped;
+ unsigned short cur_mss;
+ unsigned short cur_vlan_ctag;
+ } tx;
+ };
+} ____cacheline_aligned;
+
+struct xlgmac_channel {
+ char name[16];
+
+ /* Address of private data area for device */
+ struct xlgmac_pdata *pdata;
+
+ /* Queue index and base address of queue's DMA registers */
+ unsigned int queue_index;
+ void __iomem *dma_regs;
+
+ /* Per channel interrupt irq number */
+ int dma_irq;
+ char dma_irq_name[IFNAMSIZ + 32];
+
+ /* Netdev related settings */
+ struct napi_struct napi;
+
+ unsigned int saved_ier;
+
+ unsigned int tx_timer_active;
+ struct timer_list tx_timer;
+
+ struct xlgmac_ring *tx_ring;
+ struct xlgmac_ring *rx_ring;
+} ____cacheline_aligned;
+
+struct xlgmac_desc_ops {
+ int (*alloc_channles_and_rings)(struct xlgmac_pdata *pdata);
+ void (*free_channels_and_rings)(struct xlgmac_pdata *pdata);
+ int (*map_tx_skb)(struct xlgmac_channel *channel,
+ struct sk_buff *skb);
+ int (*map_rx_buffer)(struct xlgmac_pdata *pdata,
+ struct xlgmac_ring *ring,
+ struct xlgmac_desc_data *desc_data);
+ void (*unmap_desc_data)(struct xlgmac_pdata *pdata,
+ struct xlgmac_desc_data *desc_data);
+ void (*tx_desc_init)(struct xlgmac_pdata *pdata);
+ void (*rx_desc_init)(struct xlgmac_pdata *pdata);
+};
+
+struct xlgmac_hw_ops {
+ int (*init)(struct xlgmac_pdata *pdata);
+ int (*exit)(struct xlgmac_pdata *pdata);
+
+ int (*tx_complete)(struct xlgmac_dma_desc *dma_desc);
+
+ void (*enable_tx)(struct xlgmac_pdata *pdata);
+ void (*disable_tx)(struct xlgmac_pdata *pdata);
+ void (*enable_rx)(struct xlgmac_pdata *pdata);
+ void (*disable_rx)(struct xlgmac_pdata *pdata);
+
+ int (*enable_int)(struct xlgmac_channel *channel,
+ enum xlgmac_int int_id);
+ int (*disable_int)(struct xlgmac_channel *channel,
+ enum xlgmac_int int_id);
+ void (*dev_xmit)(struct xlgmac_channel *channel);
+ int (*dev_read)(struct xlgmac_channel *channel);
+
+ int (*set_mac_address)(struct xlgmac_pdata *pdata, u8 *addr);
+ int (*config_rx_mode)(struct xlgmac_pdata *pdata);
+ int (*enable_rx_csum)(struct xlgmac_pdata *pdata);
+ int (*disable_rx_csum)(struct xlgmac_pdata *pdata);
+
+ /* For MII speed configuration */
+ int (*set_xlgmii_25000_speed)(struct xlgmac_pdata *pdata);
+ int (*set_xlgmii_40000_speed)(struct xlgmac_pdata *pdata);
+ int (*set_xlgmii_50000_speed)(struct xlgmac_pdata *pdata);
+ int (*set_xlgmii_100000_speed)(struct xlgmac_pdata *pdata);
+
+ /* For descriptor related operation */
+ void (*tx_desc_init)(struct xlgmac_channel *channel);
+ void (*rx_desc_init)(struct xlgmac_channel *channel);
+ void (*tx_desc_reset)(struct xlgmac_desc_data *desc_data);
+ void (*rx_desc_reset)(struct xlgmac_pdata *pdata,
+ struct xlgmac_desc_data *desc_data,
+ unsigned int index);
+ int (*is_last_desc)(struct xlgmac_dma_desc *dma_desc);
+ int (*is_context_desc)(struct xlgmac_dma_desc *dma_desc);
+ void (*tx_start_xmit)(struct xlgmac_channel *channel,
+ struct xlgmac_ring *ring);
+
+ /* For Flow Control */
+ int (*config_tx_flow_control)(struct xlgmac_pdata *pdata);
+ int (*config_rx_flow_control)(struct xlgmac_pdata *pdata);
+
+ /* For Vlan related config */
+ int (*enable_rx_vlan_stripping)(struct xlgmac_pdata *pdata);
+ int (*disable_rx_vlan_stripping)(struct xlgmac_pdata *pdata);
+ int (*enable_rx_vlan_filtering)(struct xlgmac_pdata *pdata);
+ int (*disable_rx_vlan_filtering)(struct xlgmac_pdata *pdata);
+ int (*update_vlan_hash_table)(struct xlgmac_pdata *pdata);
+
+ /* For RX coalescing */
+ int (*config_rx_coalesce)(struct xlgmac_pdata *pdata);
+ int (*config_tx_coalesce)(struct xlgmac_pdata *pdata);
+ unsigned int (*usec_to_riwt)(struct xlgmac_pdata *pdata,
+ unsigned int usec);
+ unsigned int (*riwt_to_usec)(struct xlgmac_pdata *pdata,
+ unsigned int riwt);
+
+ /* For RX and TX threshold config */
+ int (*config_rx_threshold)(struct xlgmac_pdata *pdata,
+ unsigned int val);
+ int (*config_tx_threshold)(struct xlgmac_pdata *pdata,
+ unsigned int val);
+
+ /* For RX and TX Store and Forward Mode config */
+ int (*config_rsf_mode)(struct xlgmac_pdata *pdata,
+ unsigned int val);
+ int (*config_tsf_mode)(struct xlgmac_pdata *pdata,
+ unsigned int val);
+
+ /* For TX DMA Operate on Second Frame config */
+ int (*config_osp_mode)(struct xlgmac_pdata *pdata);
+
+ /* For RX and TX PBL config */
+ int (*config_rx_pbl_val)(struct xlgmac_pdata *pdata);
+ int (*get_rx_pbl_val)(struct xlgmac_pdata *pdata);
+ int (*config_tx_pbl_val)(struct xlgmac_pdata *pdata);
+ int (*get_tx_pbl_val)(struct xlgmac_pdata *pdata);
+ int (*config_pblx8)(struct xlgmac_pdata *pdata);
+
+ /* For MMC statistics */
+ void (*rx_mmc_int)(struct xlgmac_pdata *pdata);
+ void (*tx_mmc_int)(struct xlgmac_pdata *pdata);
+ void (*read_mmc_stats)(struct xlgmac_pdata *pdata);
+
+ /* For Receive Side Scaling */
+ int (*enable_rss)(struct xlgmac_pdata *pdata);
+ int (*disable_rss)(struct xlgmac_pdata *pdata);
+ int (*set_rss_hash_key)(struct xlgmac_pdata *pdata,
+ const u8 *key);
+ int (*set_rss_lookup_table)(struct xlgmac_pdata *pdata,
+ const u32 *table);
+};
+
+/* This structure contains flags that indicate what hardware features
+ * or configurations are present in the device.
+ */
+struct xlgmac_hw_features {
+ /* HW Version */
+ unsigned int version;
+
+ /* HW Feature Register0 */
+ unsigned int phyifsel; /* PHY interface support */
+ unsigned int vlhash; /* VLAN Hash Filter */
+ unsigned int sma; /* SMA(MDIO) Interface */
+ unsigned int rwk; /* PMT remote wake-up packet */
+ unsigned int mgk; /* PMT magic packet */
+ unsigned int mmc; /* RMON module */
+ unsigned int aoe; /* ARP Offload */
+ unsigned int ts; /* IEEE 1588-2008 Advanced Timestamp */
+ unsigned int eee; /* Energy Efficient Ethernet */
+ unsigned int tx_coe; /* Tx Checksum Offload */
+ unsigned int rx_coe; /* Rx Checksum Offload */
+ unsigned int addn_mac; /* Additional MAC Addresses */
+ unsigned int ts_src; /* Timestamp Source */
+ unsigned int sa_vlan_ins; /* Source Address or VLAN Insertion */
+
+ /* HW Feature Register1 */
+ unsigned int rx_fifo_size; /* MTL Receive FIFO Size */
+ unsigned int tx_fifo_size; /* MTL Transmit FIFO Size */
+ unsigned int adv_ts_hi; /* Advance Timestamping High Word */
+ unsigned int dma_width; /* DMA width */
+ unsigned int dcb; /* DCB Feature */
+ unsigned int sph; /* Split Header Feature */
+ unsigned int tso; /* TCP Segmentation Offload */
+ unsigned int dma_debug; /* DMA Debug Registers */
+ unsigned int rss; /* Receive Side Scaling */
+ unsigned int tc_cnt; /* Number of Traffic Classes */
+ unsigned int hash_table_size; /* Hash Table Size */
+ unsigned int l3l4_filter_num; /* Number of L3-L4 Filters */
+
+ /* HW Feature Register2 */
+ unsigned int rx_q_cnt; /* Number of MTL Receive Queues */
+ unsigned int tx_q_cnt; /* Number of MTL Transmit Queues */
+ unsigned int rx_ch_cnt; /* Number of DMA Receive Channels */
+ unsigned int tx_ch_cnt; /* Number of DMA Transmit Channels */
+ unsigned int pps_out_num; /* Number of PPS outputs */
+ unsigned int aux_snap_num; /* Number of Aux snapshot inputs */
+};
+
+struct xlgmac_resources {
+ void __iomem *addr;
+ int irq;
+};
+
+struct xlgmac_pdata {
+ struct net_device *netdev;
+ struct device *dev;
+
+ struct xlgmac_hw_ops hw_ops;
+ struct xlgmac_desc_ops desc_ops;
+
+ /* Device statistics */
+ struct xlgmac_stats stats;
+
+ u32 msg_enable;
+
+ /* MAC registers base */
+ void __iomem *mac_regs;
+
+ /* Hardware features of the device */
+ struct xlgmac_hw_features hw_feat;
+
+ struct work_struct restart_work;
+
+ /* Rings for Tx/Rx on a DMA channel */
+ struct xlgmac_channel *channel_head;
+ unsigned int channel_count;
+ unsigned int tx_ring_count;
+ unsigned int rx_ring_count;
+ unsigned int tx_desc_count;
+ unsigned int rx_desc_count;
+ unsigned int tx_q_count;
+ unsigned int rx_q_count;
+
+ /* Tx/Rx common settings */
+ unsigned int pblx8;
+
+ /* Tx settings */
+ unsigned int tx_sf_mode;
+ unsigned int tx_threshold;
+ unsigned int tx_pbl;
+ unsigned int tx_osp_mode;
+
+ /* Rx settings */
+ unsigned int rx_sf_mode;
+ unsigned int rx_threshold;
+ unsigned int rx_pbl;
+
+ /* Tx coalescing settings */
+ unsigned int tx_usecs;
+ unsigned int tx_frames;
+
+ /* Rx coalescing settings */
+ unsigned int rx_riwt;
+ unsigned int rx_usecs;
+ unsigned int rx_frames;
+
+ /* Current Rx buffer size */
+ unsigned int rx_buf_size;
+
+ /* Flow control settings */
+ unsigned int tx_pause;
+ unsigned int rx_pause;
+
+ /* Device interrupt number */
+ int dev_irq;
+ unsigned int per_channel_irq;
+ int channel_irq[XLGMAC_MAX_DMA_CHANNELS];
+
+ /* Netdev related settings */
+ unsigned char mac_addr[ETH_ALEN];
+ netdev_features_t netdev_features;
+ struct napi_struct napi;
+
+ /* Filtering support */
+ unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)];
+
+ /* Device clocks */
+ unsigned long sysclk_rate;
+
+ /* RSS addressing mutex */
+ struct mutex rss_mutex;
+
+ /* Receive Side Scaling settings */
+ u8 rss_key[XLGMAC_RSS_HASH_KEY_SIZE];
+ u32 rss_table[XLGMAC_RSS_MAX_TABLE_SIZE];
+ u32 rss_options;
+
+ int phy_speed;
+
+ char drv_name[32];
+ char drv_ver[32];
+};
+
+void xlgmac_init_desc_ops(struct xlgmac_desc_ops *desc_ops);
+void xlgmac_init_hw_ops(struct xlgmac_hw_ops *hw_ops);
+const struct net_device_ops *xlgmac_get_netdev_ops(void);
+const struct ethtool_ops *xlgmac_get_ethtool_ops(void);
+void xlgmac_dump_tx_desc(struct xlgmac_pdata *pdata,
+ struct xlgmac_ring *ring,
+ unsigned int idx,
+ unsigned int count,
+ unsigned int flag);
+void xlgmac_dump_rx_desc(struct xlgmac_pdata *pdata,
+ struct xlgmac_ring *ring,
+ unsigned int idx);
+void xlgmac_print_pkt(struct net_device *netdev,
+ struct sk_buff *skb, bool tx_rx);
+void xlgmac_get_all_hw_features(struct xlgmac_pdata *pdata);
+void xlgmac_print_all_hw_features(struct xlgmac_pdata *pdata);
+int xlgmac_drv_probe(struct device *dev,
+ struct xlgmac_resources *res);
+int xlgmac_drv_remove(struct device *dev);
+
+/* For debug prints */
+#ifdef XLGMAC_DEBUG
+#define XLGMAC_PR(fmt, args...) \
+ pr_alert("[%s,%d]:" fmt, __func__, __LINE__, ## args)
+#else
+#define XLGMAC_PR(x...) do { } while (0)
+#endif
+
+#endif /* __DWC_XLGMAC_H__ */
diff --git a/drivers/net/ethernet/tehuti/tehuti.c b/drivers/net/ethernet/tehuti/tehuti.c
index f864fd0..711fbbb 100644
--- a/drivers/net/ethernet/tehuti/tehuti.c
+++ b/drivers/net/ethernet/tehuti/tehuti.c
@@ -2124,33 +2124,26 @@
};
/*
- * bdx_get_settings - get device-specific settings
+ * bdx_get_link_ksettings - get device-specific settings
* @netdev
* @ecmd
*/
-static int bdx_get_settings(struct net_device *netdev, struct ethtool_cmd *ecmd)
+static int bdx_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *ecmd)
{
- u32 rdintcm;
- u32 tdintcm;
- struct bdx_priv *priv = netdev_priv(netdev);
+ ethtool_link_ksettings_zero_link_mode(ecmd, supported);
+ ethtool_link_ksettings_add_link_mode(ecmd, supported,
+ 10000baseT_Full);
+ ethtool_link_ksettings_add_link_mode(ecmd, supported, FIBRE);
+ ethtool_link_ksettings_zero_link_mode(ecmd, advertising);
+ ethtool_link_ksettings_add_link_mode(ecmd, advertising,
+ 10000baseT_Full);
+ ethtool_link_ksettings_add_link_mode(ecmd, advertising, FIBRE);
- rdintcm = priv->rdintcm;
- tdintcm = priv->tdintcm;
-
- ecmd->supported = (SUPPORTED_10000baseT_Full | SUPPORTED_FIBRE);
- ecmd->advertising = (ADVERTISED_10000baseT_Full | ADVERTISED_FIBRE);
- ethtool_cmd_speed_set(ecmd, SPEED_10000);
- ecmd->duplex = DUPLEX_FULL;
- ecmd->port = PORT_FIBRE;
- ecmd->transceiver = XCVR_EXTERNAL; /* what does it mean? */
- ecmd->autoneg = AUTONEG_DISABLE;
-
- /* PCK_TH measures in multiples of FIFO bytes
- We translate to packets */
- ecmd->maxtxpkt =
- ((GET_PCK_TH(tdintcm) * PCK_TH_MULT) / BDX_TXF_DESC_SZ);
- ecmd->maxrxpkt =
- ((GET_PCK_TH(rdintcm) * PCK_TH_MULT) / sizeof(struct rxf_desc));
+ ecmd->base.speed = SPEED_10000;
+ ecmd->base.duplex = DUPLEX_FULL;
+ ecmd->base.port = PORT_FIBRE;
+ ecmd->base.autoneg = AUTONEG_DISABLE;
return 0;
}
@@ -2384,7 +2377,6 @@
static void bdx_set_ethtool_ops(struct net_device *netdev)
{
static const struct ethtool_ops bdx_ethtool_ops = {
- .get_settings = bdx_get_settings,
.get_drvinfo = bdx_get_drvinfo,
.get_link = ethtool_op_get_link,
.get_coalesce = bdx_get_coalesce,
@@ -2394,6 +2386,7 @@
.get_strings = bdx_get_strings,
.get_sset_count = bdx_get_sset_count,
.get_ethtool_stats = bdx_get_ethtool_stats,
+ .get_link_ksettings = bdx_get_link_ksettings,
};
netdev->ethtool_ops = &bdx_ethtool_ops;
diff --git a/drivers/net/ethernet/ti/Kconfig b/drivers/net/ethernet/ti/Kconfig
index 9e63195..48a541e 100644
--- a/drivers/net/ethernet/ti/Kconfig
+++ b/drivers/net/ethernet/ti/Kconfig
@@ -76,7 +76,7 @@
config TI_CPTS
bool "TI Common Platform Time Sync (CPTS) Support"
depends on TI_CPSW || TI_KEYSTONE_NETCP
- depends on PTP_1588_CLOCK
+ depends on POSIX_TIMERS
---help---
This driver supports the Common Platform Time Sync unit of
the CPSW Ethernet Switch and Keystone 2 1g/10g Switch Subsystem.
@@ -87,6 +87,8 @@
tristate
depends on TI_CPTS
default y if TI_CPSW=y || TI_KEYSTONE_NETCP=y
+ select NET_PTP_CLASSIFY
+ imply PTP_1588_CLOCK
default m
config TI_KEYSTONE_NETCP
diff --git a/drivers/net/ethernet/ti/netcp_core.c b/drivers/net/ethernet/ti/netcp_core.c
index 7c7ae08..729a7da 100644
--- a/drivers/net/ethernet/ti/netcp_core.c
+++ b/drivers/net/ethernet/ti/netcp_core.c
@@ -1134,7 +1134,6 @@
u32 buf_len = skb_frag_size(frag);
dma_addr_t desc_dma;
u32 desc_dma_32;
- u32 pkt_info;
dma_addr = dma_map_page(dev, page, page_offset, buf_len,
DMA_TO_DEVICE);
@@ -1151,9 +1150,6 @@
}
desc_dma = knav_pool_desc_virt_to_dma(netcp->tx_pool, ndesc);
- pkt_info =
- (netcp->tx_compl_qid & KNAV_DMA_DESC_RETQ_MASK) <<
- KNAV_DMA_DESC_RETQ_SHIFT;
set_pkt_info(dma_addr, buf_len, 0, ndesc);
desc_dma_32 = (u32)desc_dma;
set_words(&desc_dma_32, 1, &pdesc->next_desc);
@@ -1882,6 +1878,7 @@
static int netcp_setup_tc(struct net_device *dev, u32 handle, __be16 proto,
struct tc_to_netdev *tc)
{
+ u8 num_tc;
int i;
/* setup tc must be called under rtnl lock */
@@ -1890,15 +1887,18 @@
if (tc->type != TC_SETUP_MQPRIO)
return -EINVAL;
+ tc->mqprio->hw = TC_MQPRIO_HW_OFFLOAD_TCS;
+ num_tc = tc->mqprio->num_tc;
+
/* Sanity-check the number of traffic classes requested */
if ((dev->real_num_tx_queues <= 1) ||
- (dev->real_num_tx_queues < tc->tc))
+ (dev->real_num_tx_queues < num_tc))
return -EINVAL;
/* Configure traffic class to queue mappings */
- if (tc->tc) {
- netdev_set_num_tc(dev, tc->tc);
- for (i = 0; i < tc->tc; i++)
+ if (num_tc) {
+ netdev_set_num_tc(dev, num_tc);
+ for (i = 0; i < num_tc; i++)
netdev_set_tc_queue(dev, i, 1, i);
} else {
netdev_reset_tc(dev);
diff --git a/drivers/net/ethernet/ti/netcp_ethss.c b/drivers/net/ethernet/ti/netcp_ethss.c
index eece3e2..897176f 100644
--- a/drivers/net/ethernet/ti/netcp_ethss.c
+++ b/drivers/net/ethernet/ti/netcp_ethss.c
@@ -3048,8 +3048,7 @@
for_each_child_of_node(node, port) {
slave = devm_kzalloc(dev, sizeof(*slave), GFP_KERNEL);
if (!slave) {
- dev_err(dev,
- "memomry alloc failed for secondary port(%s), skipping...\n",
+ dev_err(dev, "memory alloc failed for secondary port(%s), skipping...\n",
port->name);
continue;
}
diff --git a/drivers/net/ethernet/toshiba/ps3_gelic_net.c b/drivers/net/ethernet/toshiba/ps3_gelic_net.c
index 7201331..fa6a065 100644
--- a/drivers/net/ethernet/toshiba/ps3_gelic_net.c
+++ b/drivers/net/ethernet/toshiba/ps3_gelic_net.c
@@ -1206,61 +1206,68 @@
strlcpy(info->version, DRV_VERSION, sizeof(info->version));
}
-static int gelic_ether_get_settings(struct net_device *netdev,
- struct ethtool_cmd *cmd)
+static int gelic_ether_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
{
struct gelic_card *card = netdev_card(netdev);
+ u32 supported, advertising;
gelic_card_get_ether_port_status(card, 0);
if (card->ether_port_status & GELIC_LV1_ETHER_FULL_DUPLEX)
- cmd->duplex = DUPLEX_FULL;
+ cmd->base.duplex = DUPLEX_FULL;
else
- cmd->duplex = DUPLEX_HALF;
+ cmd->base.duplex = DUPLEX_HALF;
switch (card->ether_port_status & GELIC_LV1_ETHER_SPEED_MASK) {
case GELIC_LV1_ETHER_SPEED_10:
- ethtool_cmd_speed_set(cmd, SPEED_10);
+ cmd->base.speed = SPEED_10;
break;
case GELIC_LV1_ETHER_SPEED_100:
- ethtool_cmd_speed_set(cmd, SPEED_100);
+ cmd->base.speed = SPEED_100;
break;
case GELIC_LV1_ETHER_SPEED_1000:
- ethtool_cmd_speed_set(cmd, SPEED_1000);
+ cmd->base.speed = SPEED_1000;
break;
default:
pr_info("%s: speed unknown\n", __func__);
- ethtool_cmd_speed_set(cmd, SPEED_10);
+ cmd->base.speed = SPEED_10;
break;
}
- cmd->supported = SUPPORTED_TP | SUPPORTED_Autoneg |
+ supported = SUPPORTED_TP | SUPPORTED_Autoneg |
SUPPORTED_10baseT_Half | SUPPORTED_10baseT_Full |
SUPPORTED_100baseT_Half | SUPPORTED_100baseT_Full |
SUPPORTED_1000baseT_Full;
- cmd->advertising = cmd->supported;
+ advertising = supported;
if (card->link_mode & GELIC_LV1_ETHER_AUTO_NEG) {
- cmd->autoneg = AUTONEG_ENABLE;
+ cmd->base.autoneg = AUTONEG_ENABLE;
} else {
- cmd->autoneg = AUTONEG_DISABLE;
- cmd->advertising &= ~ADVERTISED_Autoneg;
+ cmd->base.autoneg = AUTONEG_DISABLE;
+ advertising &= ~ADVERTISED_Autoneg;
}
- cmd->port = PORT_TP;
+ cmd->base.port = PORT_TP;
+
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
+ advertising);
return 0;
}
-static int gelic_ether_set_settings(struct net_device *netdev,
- struct ethtool_cmd *cmd)
+static int
+gelic_ether_set_link_ksettings(struct net_device *netdev,
+ const struct ethtool_link_ksettings *cmd)
{
struct gelic_card *card = netdev_card(netdev);
u64 mode;
int ret;
- if (cmd->autoneg == AUTONEG_ENABLE) {
+ if (cmd->base.autoneg == AUTONEG_ENABLE) {
mode = GELIC_LV1_ETHER_AUTO_NEG;
} else {
- switch (cmd->speed) {
+ switch (cmd->base.speed) {
case SPEED_10:
mode = GELIC_LV1_ETHER_SPEED_10;
break;
@@ -1273,9 +1280,9 @@
default:
return -EINVAL;
}
- if (cmd->duplex == DUPLEX_FULL)
+ if (cmd->base.duplex == DUPLEX_FULL) {
mode |= GELIC_LV1_ETHER_FULL_DUPLEX;
- else if (cmd->speed == SPEED_1000) {
+ } else if (cmd->base.speed == SPEED_1000) {
pr_info("1000 half duplex is not supported.\n");
return -EINVAL;
}
@@ -1370,11 +1377,11 @@
static const struct ethtool_ops gelic_ether_ethtool_ops = {
.get_drvinfo = gelic_net_get_drvinfo,
- .get_settings = gelic_ether_get_settings,
- .set_settings = gelic_ether_set_settings,
.get_link = ethtool_op_get_link,
.get_wol = gelic_net_get_wol,
.set_wol = gelic_net_set_wol,
+ .get_link_ksettings = gelic_ether_get_link_ksettings,
+ .set_link_ksettings = gelic_ether_set_link_ksettings,
};
/**
diff --git a/drivers/net/ethernet/toshiba/spider_net_ethtool.c b/drivers/net/ethernet/toshiba/spider_net_ethtool.c
index ffe5193..16bd036 100644
--- a/drivers/net/ethernet/toshiba/spider_net_ethtool.c
+++ b/drivers/net/ethernet/toshiba/spider_net_ethtool.c
@@ -47,19 +47,23 @@
};
static int
-spider_net_ethtool_get_settings(struct net_device *netdev,
- struct ethtool_cmd *cmd)
+spider_net_ethtool_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
{
struct spider_net_card *card;
card = netdev_priv(netdev);
- cmd->supported = (SUPPORTED_1000baseT_Full |
- SUPPORTED_FIBRE);
- cmd->advertising = (ADVERTISED_1000baseT_Full |
- ADVERTISED_FIBRE);
- cmd->port = PORT_FIBRE;
- ethtool_cmd_speed_set(cmd, card->phy.speed);
- cmd->duplex = DUPLEX_FULL;
+ ethtool_link_ksettings_zero_link_mode(cmd, supported);
+ ethtool_link_ksettings_add_link_mode(cmd, supported, 1000baseT_Full);
+ ethtool_link_ksettings_add_link_mode(cmd, supported, FIBRE);
+
+ ethtool_link_ksettings_zero_link_mode(cmd, advertising);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, 1000baseT_Full);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, FIBRE);
+
+ cmd->base.port = PORT_FIBRE;
+ cmd->base.speed = card->phy.speed;
+ cmd->base.duplex = DUPLEX_FULL;
return 0;
}
@@ -166,7 +170,6 @@
}
const struct ethtool_ops spider_net_ethtool_ops = {
- .get_settings = spider_net_ethtool_get_settings,
.get_drvinfo = spider_net_ethtool_get_drvinfo,
.get_wol = spider_net_ethtool_get_wol,
.get_msglevel = spider_net_ethtool_get_msglevel,
@@ -177,5 +180,6 @@
.get_strings = spider_net_get_strings,
.get_sset_count = spider_net_get_sset_count,
.get_ethtool_stats = spider_net_get_ethtool_stats,
+ .get_link_ksettings = spider_net_ethtool_get_link_ksettings,
};
diff --git a/drivers/net/ethernet/toshiba/tc35815.c b/drivers/net/ethernet/toshiba/tc35815.c
index a45f98f..3dadee1 100644
--- a/drivers/net/ethernet/toshiba/tc35815.c
+++ b/drivers/net/ethernet/toshiba/tc35815.c
@@ -1017,8 +1017,8 @@
BUG_ON(lp->tx_skbs[i].skb != skb);
#endif
if (skb) {
- dev_kfree_skb(skb);
pci_unmap_single(lp->pci_dev, lp->tx_skbs[i].skb_dma, skb->len, PCI_DMA_TODEVICE);
+ dev_kfree_skb(skb);
lp->tx_skbs[i].skb = NULL;
lp->tx_skbs[i].skb_dma = 0;
}
diff --git a/drivers/net/ethernet/tundra/tsi108_eth.c b/drivers/net/ethernet/tundra/tsi108_eth.c
index c558399..5ac6eaa 100644
--- a/drivers/net/ethernet/tundra/tsi108_eth.c
+++ b/drivers/net/ethernet/tundra/tsi108_eth.c
@@ -1499,27 +1499,29 @@
TSI_WRITE(TSI108_EC_INTMASK, ~0);
}
-static int tsi108_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int tsi108_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct tsi108_prv_data *data = netdev_priv(dev);
unsigned long flags;
int rc;
spin_lock_irqsave(&data->txlock, flags);
- rc = mii_ethtool_gset(&data->mii_if, cmd);
+ rc = mii_ethtool_get_link_ksettings(&data->mii_if, cmd);
spin_unlock_irqrestore(&data->txlock, flags);
return rc;
}
-static int tsi108_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int tsi108_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct tsi108_prv_data *data = netdev_priv(dev);
unsigned long flags;
int rc;
spin_lock_irqsave(&data->txlock, flags);
- rc = mii_ethtool_sset(&data->mii_if, cmd);
+ rc = mii_ethtool_set_link_ksettings(&data->mii_if, cmd);
spin_unlock_irqrestore(&data->txlock, flags);
return rc;
@@ -1535,8 +1537,8 @@
static const struct ethtool_ops tsi108_ethtool_ops = {
.get_link = ethtool_op_get_link,
- .get_settings = tsi108_get_settings,
- .set_settings = tsi108_set_settings,
+ .get_link_ksettings = tsi108_get_link_ksettings,
+ .set_link_ksettings = tsi108_set_link_ksettings,
};
static const struct net_device_ops tsi108_netdev_ops = {
diff --git a/drivers/net/ethernet/via/via-rhine.c b/drivers/net/ethernet/via/via-rhine.c
index c068c58..4cf41f7 100644
--- a/drivers/net/ethernet/via/via-rhine.c
+++ b/drivers/net/ethernet/via/via-rhine.c
@@ -2303,25 +2303,27 @@
strlcpy(info->bus_info, dev_name(hwdev), sizeof(info->bus_info));
}
-static int netdev_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int netdev_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct rhine_private *rp = netdev_priv(dev);
int rc;
mutex_lock(&rp->task_lock);
- rc = mii_ethtool_gset(&rp->mii_if, cmd);
+ rc = mii_ethtool_get_link_ksettings(&rp->mii_if, cmd);
mutex_unlock(&rp->task_lock);
return rc;
}
-static int netdev_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int netdev_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct rhine_private *rp = netdev_priv(dev);
int rc;
mutex_lock(&rp->task_lock);
- rc = mii_ethtool_sset(&rp->mii_if, cmd);
+ rc = mii_ethtool_set_link_ksettings(&rp->mii_if, cmd);
rhine_set_carrier(&rp->mii_if);
mutex_unlock(&rp->task_lock);
@@ -2391,14 +2393,14 @@
static const struct ethtool_ops netdev_ethtool_ops = {
.get_drvinfo = netdev_get_drvinfo,
- .get_settings = netdev_get_settings,
- .set_settings = netdev_set_settings,
.nway_reset = netdev_nway_reset,
.get_link = netdev_get_link,
.get_msglevel = netdev_get_msglevel,
.set_msglevel = netdev_set_msglevel,
.get_wol = rhine_get_wol,
.set_wol = rhine_set_wol,
+ .get_link_ksettings = netdev_get_link_ksettings,
+ .set_link_ksettings = netdev_set_link_ksettings,
};
static int netdev_ioctl(struct net_device *dev, struct ifreq *rq, int cmd)
diff --git a/drivers/net/ethernet/via/via-velocity.c b/drivers/net/ethernet/via/via-velocity.c
index d088788..ef9538e 100644
--- a/drivers/net/ethernet/via/via-velocity.c
+++ b/drivers/net/ethernet/via/via-velocity.c
@@ -3291,15 +3291,17 @@
velocity_set_power_state(vptr, PCI_D3hot);
}
-static int velocity_get_settings(struct net_device *dev,
- struct ethtool_cmd *cmd)
+static int velocity_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct velocity_info *vptr = netdev_priv(dev);
struct mac_regs __iomem *regs = vptr->mac_regs;
u32 status;
+ u32 supported, advertising;
+
status = check_connection_type(vptr->mac_regs);
- cmd->supported = SUPPORTED_TP |
+ supported = SUPPORTED_TP |
SUPPORTED_Autoneg |
SUPPORTED_10baseT_Half |
SUPPORTED_10baseT_Full |
@@ -3308,9 +3310,9 @@
SUPPORTED_1000baseT_Half |
SUPPORTED_1000baseT_Full;
- cmd->advertising = ADVERTISED_TP | ADVERTISED_Autoneg;
+ advertising = ADVERTISED_TP | ADVERTISED_Autoneg;
if (vptr->options.spd_dpx == SPD_DPX_AUTO) {
- cmd->advertising |=
+ advertising |=
ADVERTISED_10baseT_Half |
ADVERTISED_10baseT_Full |
ADVERTISED_100baseT_Half |
@@ -3320,19 +3322,19 @@
} else {
switch (vptr->options.spd_dpx) {
case SPD_DPX_1000_FULL:
- cmd->advertising |= ADVERTISED_1000baseT_Full;
+ advertising |= ADVERTISED_1000baseT_Full;
break;
case SPD_DPX_100_HALF:
- cmd->advertising |= ADVERTISED_100baseT_Half;
+ advertising |= ADVERTISED_100baseT_Half;
break;
case SPD_DPX_100_FULL:
- cmd->advertising |= ADVERTISED_100baseT_Full;
+ advertising |= ADVERTISED_100baseT_Full;
break;
case SPD_DPX_10_HALF:
- cmd->advertising |= ADVERTISED_10baseT_Half;
+ advertising |= ADVERTISED_10baseT_Half;
break;
case SPD_DPX_10_FULL:
- cmd->advertising |= ADVERTISED_10baseT_Full;
+ advertising |= ADVERTISED_10baseT_Full;
break;
default:
break;
@@ -3340,30 +3342,35 @@
}
if (status & VELOCITY_SPEED_1000)
- ethtool_cmd_speed_set(cmd, SPEED_1000);
+ cmd->base.speed = SPEED_1000;
else if (status & VELOCITY_SPEED_100)
- ethtool_cmd_speed_set(cmd, SPEED_100);
+ cmd->base.speed = SPEED_100;
else
- ethtool_cmd_speed_set(cmd, SPEED_10);
+ cmd->base.speed = SPEED_10;
- cmd->autoneg = (status & VELOCITY_AUTONEG_ENABLE) ? AUTONEG_ENABLE : AUTONEG_DISABLE;
- cmd->port = PORT_TP;
- cmd->transceiver = XCVR_INTERNAL;
- cmd->phy_address = readb(®s->MIIADR) & 0x1F;
+ cmd->base.autoneg = (status & VELOCITY_AUTONEG_ENABLE) ?
+ AUTONEG_ENABLE : AUTONEG_DISABLE;
+ cmd->base.port = PORT_TP;
+ cmd->base.phy_address = readb(®s->MIIADR) & 0x1F;
if (status & VELOCITY_DUPLEX_FULL)
- cmd->duplex = DUPLEX_FULL;
+ cmd->base.duplex = DUPLEX_FULL;
else
- cmd->duplex = DUPLEX_HALF;
+ cmd->base.duplex = DUPLEX_HALF;
+
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.supported,
+ supported);
+ ethtool_convert_legacy_u32_to_link_mode(cmd->link_modes.advertising,
+ advertising);
return 0;
}
-static int velocity_set_settings(struct net_device *dev,
- struct ethtool_cmd *cmd)
+static int velocity_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct velocity_info *vptr = netdev_priv(dev);
- u32 speed = ethtool_cmd_speed(cmd);
+ u32 speed = cmd->base.speed;
u32 curr_status;
u32 new_status = 0;
int ret = 0;
@@ -3371,11 +3378,12 @@
curr_status = check_connection_type(vptr->mac_regs);
curr_status &= (~VELOCITY_LINK_FAIL);
- new_status |= ((cmd->autoneg) ? VELOCITY_AUTONEG_ENABLE : 0);
+ new_status |= ((cmd->base.autoneg) ? VELOCITY_AUTONEG_ENABLE : 0);
new_status |= ((speed == SPEED_1000) ? VELOCITY_SPEED_1000 : 0);
new_status |= ((speed == SPEED_100) ? VELOCITY_SPEED_100 : 0);
new_status |= ((speed == SPEED_10) ? VELOCITY_SPEED_10 : 0);
- new_status |= ((cmd->duplex == DUPLEX_FULL) ? VELOCITY_DUPLEX_FULL : 0);
+ new_status |= ((cmd->base.duplex == DUPLEX_FULL) ?
+ VELOCITY_DUPLEX_FULL : 0);
if ((new_status & VELOCITY_AUTONEG_ENABLE) &&
(new_status != (curr_status | VELOCITY_AUTONEG_ENABLE))) {
@@ -3644,8 +3652,6 @@
}
static const struct ethtool_ops velocity_ethtool_ops = {
- .get_settings = velocity_get_settings,
- .set_settings = velocity_set_settings,
.get_drvinfo = velocity_get_drvinfo,
.get_wol = velocity_ethtool_get_wol,
.set_wol = velocity_ethtool_set_wol,
@@ -3658,7 +3664,9 @@
.get_coalesce = velocity_get_coalesce,
.set_coalesce = velocity_set_coalesce,
.begin = velocity_ethtool_up,
- .complete = velocity_ethtool_down
+ .complete = velocity_ethtool_down,
+ .get_link_ksettings = velocity_get_link_ksettings,
+ .set_link_ksettings = velocity_set_link_ksettings,
};
#if defined(CONFIG_PM) && defined(CONFIG_INET)
diff --git a/drivers/net/ethernet/wiznet/w5100.c b/drivers/net/ethernet/wiznet/w5100.c
index f90267f..2bdfb39 100644
--- a/drivers/net/ethernet/wiznet/w5100.c
+++ b/drivers/net/ethernet/wiznet/w5100.c
@@ -1152,7 +1152,8 @@
if (err < 0)
goto err_register;
- priv->xfer_wq = alloc_workqueue(netdev_name(ndev), WQ_MEM_RECLAIM, 0);
+ priv->xfer_wq = alloc_workqueue("%s", WQ_MEM_RECLAIM, 0,
+ netdev_name(ndev));
if (!priv->xfer_wq) {
err = -ENOMEM;
goto err_wq;
diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
index b96e969..33c595f 100644
--- a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
+++ b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c
@@ -301,7 +301,7 @@
if (address)
memcpy(ndev->dev_addr, address, ETH_ALEN);
if (!is_valid_ether_addr(ndev->dev_addr))
- eth_random_addr(ndev->dev_addr);
+ eth_hw_addr_random(ndev);
/* Set up unicast MAC address filter set its mac address */
axienet_iow(lp, XAE_UAW0_OFFSET,
diff --git a/drivers/net/fjes/fjes_ethtool.c b/drivers/net/fjes/fjes_ethtool.c
index 6575f88..7d10171 100644
--- a/drivers/net/fjes/fjes_ethtool.c
+++ b/drivers/net/fjes/fjes_ethtool.c
@@ -175,16 +175,15 @@
"platform:%s", plat_dev->name);
}
-static int fjes_get_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+static int fjes_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *ecmd)
{
- ecmd->supported = 0;
- ecmd->advertising = 0;
- ecmd->duplex = DUPLEX_FULL;
- ecmd->autoneg = AUTONEG_DISABLE;
- ecmd->transceiver = XCVR_DUMMY1;
- ecmd->port = PORT_NONE;
- ethtool_cmd_speed_set(ecmd, 20000); /* 20Gb/s */
+ ethtool_link_ksettings_zero_link_mode(ecmd, supported);
+ ethtool_link_ksettings_zero_link_mode(ecmd, advertising);
+ ecmd->base.duplex = DUPLEX_FULL;
+ ecmd->base.autoneg = AUTONEG_DISABLE;
+ ecmd->base.port = PORT_NONE;
+ ecmd->base.speed = 20000; /* 20Gb/s */
return 0;
}
@@ -296,7 +295,6 @@
}
static const struct ethtool_ops fjes_ethtool_ops = {
- .get_settings = fjes_get_settings,
.get_drvinfo = fjes_get_drvinfo,
.get_ethtool_stats = fjes_get_ethtool_stats,
.get_strings = fjes_get_strings,
@@ -306,6 +304,7 @@
.set_dump = fjes_set_dump,
.get_dump_flag = fjes_get_dump_flag,
.get_dump_data = fjes_get_dump_data,
+ .get_link_ksettings = fjes_get_link_ksettings,
};
void fjes_set_ethtool_ops(struct net_device *netdev)
diff --git a/drivers/net/geneve.c b/drivers/net/geneve.c
index 7074b40..dec5d56 100644
--- a/drivers/net/geneve.c
+++ b/drivers/net/geneve.c
@@ -1244,7 +1244,7 @@
metadata = true;
if (data[IFLA_GENEVE_UDP_CSUM] &&
- !nla_get_u8(data[IFLA_GENEVE_UDP_CSUM]))
+ nla_get_u8(data[IFLA_GENEVE_UDP_CSUM]))
info.key.tun_flags |= TUNNEL_CSUM;
if (data[IFLA_GENEVE_UDP_ZERO_CSUM6_TX] &&
diff --git a/drivers/net/gtp.c b/drivers/net/gtp.c
index 8969874..4fea1b3 100644
--- a/drivers/net/gtp.c
+++ b/drivers/net/gtp.c
@@ -56,7 +56,10 @@
u16 af;
struct in_addr ms_addr_ip4;
- struct in_addr sgsn_addr_ip4;
+ struct in_addr peer_addr_ip4;
+
+ struct sock *sk;
+ struct net_device *dev;
atomic_t tx_seq;
struct rcu_head rcu_head;
@@ -66,11 +69,12 @@
struct gtp_dev {
struct list_head list;
- struct socket *sock0;
- struct socket *sock1u;
+ struct sock *sk0;
+ struct sock *sk1u;
struct net_device *dev;
+ unsigned int role;
unsigned int hash_size;
struct hlist_head *tid_hash;
struct hlist_head *addr_hash;
@@ -84,6 +88,8 @@
static u32 gtp_h_initval;
+static void pdp_context_delete(struct pdp_ctx *pctx);
+
static inline u32 gtp0_hashfn(u64 tid)
{
u32 *tid32 = (u32 *) &tid;
@@ -149,8 +155,8 @@
return NULL;
}
-static bool gtp_check_src_ms_ipv4(struct sk_buff *skb, struct pdp_ctx *pctx,
- unsigned int hdrlen)
+static bool gtp_check_ms_ipv4(struct sk_buff *skb, struct pdp_ctx *pctx,
+ unsigned int hdrlen, unsigned int role)
{
struct iphdr *iph;
@@ -159,25 +165,62 @@
iph = (struct iphdr *)(skb->data + hdrlen);
- return iph->saddr == pctx->ms_addr_ip4.s_addr;
+ if (role == GTP_ROLE_SGSN)
+ return iph->daddr == pctx->ms_addr_ip4.s_addr;
+ else
+ return iph->saddr == pctx->ms_addr_ip4.s_addr;
}
-/* Check if the inner IP source address in this packet is assigned to any
+/* Check if the inner IP address in this packet is assigned to any
* existing mobile subscriber.
*/
-static bool gtp_check_src_ms(struct sk_buff *skb, struct pdp_ctx *pctx,
- unsigned int hdrlen)
+static bool gtp_check_ms(struct sk_buff *skb, struct pdp_ctx *pctx,
+ unsigned int hdrlen, unsigned int role)
{
switch (ntohs(skb->protocol)) {
case ETH_P_IP:
- return gtp_check_src_ms_ipv4(skb, pctx, hdrlen);
+ return gtp_check_ms_ipv4(skb, pctx, hdrlen, role);
}
return false;
}
+static int gtp_rx(struct pdp_ctx *pctx, struct sk_buff *skb,
+ unsigned int hdrlen, unsigned int role)
+{
+ struct pcpu_sw_netstats *stats;
+
+ if (!gtp_check_ms(skb, pctx, hdrlen, role)) {
+ netdev_dbg(pctx->dev, "No PDP ctx for this MS\n");
+ return 1;
+ }
+
+ /* Get rid of the GTP + UDP headers. */
+ if (iptunnel_pull_header(skb, hdrlen, skb->protocol,
+ !net_eq(sock_net(pctx->sk), dev_net(pctx->dev))))
+ return -1;
+
+ netdev_dbg(pctx->dev, "forwarding packet from GGSN to uplink\n");
+
+ /* Now that the UDP and the GTP header have been removed, set up the
+ * new network header. This is required by the upper layer to
+ * calculate the transport header.
+ */
+ skb_reset_network_header(skb);
+
+ skb->dev = pctx->dev;
+
+ stats = this_cpu_ptr(pctx->dev->tstats);
+ u64_stats_update_begin(&stats->syncp);
+ stats->rx_packets++;
+ stats->rx_bytes += skb->len;
+ u64_stats_update_end(&stats->syncp);
+
+ netif_rx(skb);
+ return 0;
+}
+
/* 1 means pass up to the stack, -1 means drop and 0 means decapsulated. */
-static int gtp0_udp_encap_recv(struct gtp_dev *gtp, struct sk_buff *skb,
- bool xnet)
+static int gtp0_udp_encap_recv(struct gtp_dev *gtp, struct sk_buff *skb)
{
unsigned int hdrlen = sizeof(struct udphdr) +
sizeof(struct gtp0_header);
@@ -201,17 +244,10 @@
return 1;
}
- if (!gtp_check_src_ms(skb, pctx, hdrlen)) {
- netdev_dbg(gtp->dev, "No PDP ctx for this MS\n");
- return 1;
- }
-
- /* Get rid of the GTP + UDP headers. */
- return iptunnel_pull_header(skb, hdrlen, skb->protocol, xnet);
+ return gtp_rx(pctx, skb, hdrlen, gtp->role);
}
-static int gtp1u_udp_encap_recv(struct gtp_dev *gtp, struct sk_buff *skb,
- bool xnet)
+static int gtp1u_udp_encap_recv(struct gtp_dev *gtp, struct sk_buff *skb)
{
unsigned int hdrlen = sizeof(struct udphdr) +
sizeof(struct gtp1_header);
@@ -250,28 +286,7 @@
return 1;
}
- if (!gtp_check_src_ms(skb, pctx, hdrlen)) {
- netdev_dbg(gtp->dev, "No PDP ctx for this MS\n");
- return 1;
- }
-
- /* Get rid of the GTP + UDP headers. */
- return iptunnel_pull_header(skb, hdrlen, skb->protocol, xnet);
-}
-
-static void gtp_encap_disable(struct gtp_dev *gtp)
-{
- if (gtp->sock0 && gtp->sock0->sk) {
- udp_sk(gtp->sock0->sk)->encap_type = 0;
- rcu_assign_sk_user_data(gtp->sock0->sk, NULL);
- }
- if (gtp->sock1u && gtp->sock1u->sk) {
- udp_sk(gtp->sock1u->sk)->encap_type = 0;
- rcu_assign_sk_user_data(gtp->sock1u->sk, NULL);
- }
-
- gtp->sock0 = NULL;
- gtp->sock1u = NULL;
+ return gtp_rx(pctx, skb, hdrlen, gtp->role);
}
static void gtp_encap_destroy(struct sock *sk)
@@ -279,8 +294,25 @@
struct gtp_dev *gtp;
gtp = rcu_dereference_sk_user_data(sk);
- if (gtp)
- gtp_encap_disable(gtp);
+ if (gtp) {
+ udp_sk(sk)->encap_type = 0;
+ rcu_assign_sk_user_data(sk, NULL);
+ sock_put(sk);
+ }
+}
+
+static void gtp_encap_disable_sock(struct sock *sk)
+{
+ if (!sk)
+ return;
+
+ gtp_encap_destroy(sk);
+}
+
+static void gtp_encap_disable(struct gtp_dev *gtp)
+{
+ gtp_encap_disable_sock(gtp->sk0);
+ gtp_encap_disable_sock(gtp->sk1u);
}
/* UDP encapsulation receive handler. See net/ipv4/udp.c.
@@ -288,10 +320,8 @@
*/
static int gtp_encap_recv(struct sock *sk, struct sk_buff *skb)
{
- struct pcpu_sw_netstats *stats;
struct gtp_dev *gtp;
- bool xnet;
- int ret;
+ int ret = 0;
gtp = rcu_dereference_sk_user_data(sk);
if (!gtp)
@@ -299,16 +329,14 @@
netdev_dbg(gtp->dev, "encap_recv sk=%p\n", sk);
- xnet = !net_eq(sock_net(sk), dev_net(gtp->dev));
-
switch (udp_sk(sk)->encap_type) {
case UDP_ENCAP_GTP0:
netdev_dbg(gtp->dev, "received GTP0 packet\n");
- ret = gtp0_udp_encap_recv(gtp, skb, xnet);
+ ret = gtp0_udp_encap_recv(gtp, skb);
break;
case UDP_ENCAP_GTP1U:
netdev_dbg(gtp->dev, "received GTP1U packet\n");
- ret = gtp1u_udp_encap_recv(gtp, skb, xnet);
+ ret = gtp1u_udp_encap_recv(gtp, skb);
break;
default:
ret = -1; /* Shouldn't happen. */
@@ -317,33 +345,17 @@
switch (ret) {
case 1:
netdev_dbg(gtp->dev, "pass up to the process\n");
- return 1;
+ break;
case 0:
- netdev_dbg(gtp->dev, "forwarding packet from GGSN to uplink\n");
break;
case -1:
netdev_dbg(gtp->dev, "GTP packet has been dropped\n");
kfree_skb(skb);
- return 0;
+ ret = 0;
+ break;
}
- /* Now that the UDP and the GTP header have been removed, set up the
- * new network header. This is required by the upper layer to
- * calculate the transport header.
- */
- skb_reset_network_header(skb);
-
- skb->dev = gtp->dev;
-
- stats = this_cpu_ptr(gtp->dev->tstats);
- u64_stats_update_begin(&stats->syncp);
- stats->rx_packets++;
- stats->rx_bytes += skb->len;
- u64_stats_update_end(&stats->syncp);
-
- netif_rx(skb);
-
- return 0;
+ return ret;
}
static int gtp_dev_init(struct net_device *dev)
@@ -367,8 +379,9 @@
free_percpu(dev->tstats);
}
-static struct rtable *ip4_route_output_gtp(struct net *net, struct flowi4 *fl4,
- const struct sock *sk, __be32 daddr)
+static struct rtable *ip4_route_output_gtp(struct flowi4 *fl4,
+ const struct sock *sk,
+ __be32 daddr)
{
memset(fl4, 0, sizeof(*fl4));
fl4->flowi4_oif = sk->sk_bound_dev_if;
@@ -377,7 +390,7 @@
fl4->flowi4_tos = RT_CONN_FLAGS(sk);
fl4->flowi4_proto = sk->sk_protocol;
- return ip_route_output_key(net, fl4);
+ return ip_route_output_key(sock_net(sk), fl4);
}
static inline void gtp0_push_header(struct sk_buff *skb, struct pdp_ctx *pctx)
@@ -466,7 +479,6 @@
struct rtable *rt;
struct flowi4 fl4;
struct iphdr *iph;
- struct sock *sk;
__be16 df;
int mtu;
@@ -474,7 +486,11 @@
* Prepend PDP header with TEI/TID from PDP ctx.
*/
iph = ip_hdr(skb);
- pctx = ipv4_pdp_find(gtp, iph->daddr);
+ if (gtp->role == GTP_ROLE_SGSN)
+ pctx = ipv4_pdp_find(gtp, iph->saddr);
+ else
+ pctx = ipv4_pdp_find(gtp, iph->daddr);
+
if (!pctx) {
netdev_dbg(dev, "no PDP ctx found for %pI4, skip\n",
&iph->daddr);
@@ -482,40 +498,17 @@
}
netdev_dbg(dev, "found PDP context %p\n", pctx);
- switch (pctx->gtp_version) {
- case GTP_V0:
- if (gtp->sock0)
- sk = gtp->sock0->sk;
- else
- sk = NULL;
- break;
- case GTP_V1:
- if (gtp->sock1u)
- sk = gtp->sock1u->sk;
- else
- sk = NULL;
- break;
- default:
- return -ENOENT;
- }
-
- if (!sk) {
- netdev_dbg(dev, "no userspace socket is available, skip\n");
- return -ENOENT;
- }
-
- rt = ip4_route_output_gtp(sock_net(sk), &fl4, gtp->sock0->sk,
- pctx->sgsn_addr_ip4.s_addr);
+ rt = ip4_route_output_gtp(&fl4, pctx->sk, pctx->peer_addr_ip4.s_addr);
if (IS_ERR(rt)) {
netdev_dbg(dev, "no route to SSGN %pI4\n",
- &pctx->sgsn_addr_ip4.s_addr);
+ &pctx->peer_addr_ip4.s_addr);
dev->stats.tx_carrier_errors++;
goto err;
}
if (rt->dst.dev == dev) {
netdev_dbg(dev, "circular route to SSGN %pI4\n",
- &pctx->sgsn_addr_ip4.s_addr);
+ &pctx->peer_addr_ip4.s_addr);
dev->stats.collisions++;
goto err_rt;
}
@@ -550,7 +543,7 @@
goto err_rt;
}
- gtp_set_pktinfo_ipv4(pktinfo, sk, iph, pctx, rt, &fl4, dev);
+ gtp_set_pktinfo_ipv4(pktinfo, pctx->sk, iph, pctx, rt, &fl4, dev);
gtp_push_header(skb, pktinfo);
return 0;
@@ -640,27 +633,23 @@
static int gtp_hashtable_new(struct gtp_dev *gtp, int hsize);
static void gtp_hashtable_free(struct gtp_dev *gtp);
-static int gtp_encap_enable(struct net_device *dev, struct gtp_dev *gtp,
- int fd_gtp0, int fd_gtp1);
+static int gtp_encap_enable(struct gtp_dev *gtp, struct nlattr *data[]);
static int gtp_newlink(struct net *src_net, struct net_device *dev,
struct nlattr *tb[], struct nlattr *data[])
{
- int hashsize, err, fd0, fd1;
struct gtp_dev *gtp;
struct gtp_net *gn;
+ int hashsize, err;
- if (!data[IFLA_GTP_FD0] || !data[IFLA_GTP_FD1])
+ if (!data[IFLA_GTP_FD0] && !data[IFLA_GTP_FD1])
return -EINVAL;
gtp = netdev_priv(dev);
- fd0 = nla_get_u32(data[IFLA_GTP_FD0]);
- fd1 = nla_get_u32(data[IFLA_GTP_FD1]);
-
- err = gtp_encap_enable(dev, gtp, fd0, fd1);
+ err = gtp_encap_enable(gtp, data);
if (err < 0)
- goto out_err;
+ return err;
if (!data[IFLA_GTP_PDP_HASHSIZE])
hashsize = 1024;
@@ -688,7 +677,6 @@
gtp_hashtable_free(gtp);
out_encap:
gtp_encap_disable(gtp);
-out_err:
return err;
}
@@ -706,6 +694,7 @@
[IFLA_GTP_FD0] = { .type = NLA_U32 },
[IFLA_GTP_FD1] = { .type = NLA_U32 },
[IFLA_GTP_PDP_HASHSIZE] = { .type = NLA_U32 },
+ [IFLA_GTP_ROLE] = { .type = NLA_U32 },
};
static int gtp_validate(struct nlattr *tb[], struct nlattr *data[])
@@ -747,21 +736,6 @@
.fill_info = gtp_fill_info,
};
-static struct net *gtp_genl_get_net(struct net *src_net, struct nlattr *tb[])
-{
- struct net *net;
-
- /* Examine the link attributes and figure out which network namespace
- * we are talking about.
- */
- if (tb[GTPA_NET_NS_FD])
- net = get_net_ns_by_fd(nla_get_u32(tb[GTPA_NET_NS_FD]));
- else
- net = get_net(src_net);
-
- return net;
-}
-
static int gtp_hashtable_new(struct gtp_dev *gtp, int hsize)
{
int i;
@@ -791,93 +765,127 @@
struct pdp_ctx *pctx;
int i;
- for (i = 0; i < gtp->hash_size; i++) {
- hlist_for_each_entry_rcu(pctx, >p->tid_hash[i], hlist_tid) {
- hlist_del_rcu(&pctx->hlist_tid);
- hlist_del_rcu(&pctx->hlist_addr);
- kfree_rcu(pctx, rcu_head);
- }
- }
+ for (i = 0; i < gtp->hash_size; i++)
+ hlist_for_each_entry_rcu(pctx, >p->tid_hash[i], hlist_tid)
+ pdp_context_delete(pctx);
+
synchronize_rcu();
kfree(gtp->addr_hash);
kfree(gtp->tid_hash);
}
-static int gtp_encap_enable(struct net_device *dev, struct gtp_dev *gtp,
- int fd_gtp0, int fd_gtp1)
+static struct sock *gtp_encap_enable_socket(int fd, int type,
+ struct gtp_dev *gtp)
{
struct udp_tunnel_sock_cfg tuncfg = {NULL};
- struct socket *sock0, *sock1u;
+ struct socket *sock;
+ struct sock *sk;
int err;
- netdev_dbg(dev, "enable gtp on %d, %d\n", fd_gtp0, fd_gtp1);
+ pr_debug("enable gtp on %d, %d\n", fd, type);
- sock0 = sockfd_lookup(fd_gtp0, &err);
- if (sock0 == NULL) {
- netdev_dbg(dev, "socket fd=%d not found (gtp0)\n", fd_gtp0);
- return -ENOENT;
+ sock = sockfd_lookup(fd, &err);
+ if (!sock) {
+ pr_debug("gtp socket fd=%d not found\n", fd);
+ return NULL;
}
- if (sock0->sk->sk_protocol != IPPROTO_UDP) {
- netdev_dbg(dev, "socket fd=%d not UDP\n", fd_gtp0);
- err = -EINVAL;
- goto err1;
+ if (sock->sk->sk_protocol != IPPROTO_UDP) {
+ pr_debug("socket fd=%d not UDP\n", fd);
+ sk = ERR_PTR(-EINVAL);
+ goto out_sock;
}
- sock1u = sockfd_lookup(fd_gtp1, &err);
- if (sock1u == NULL) {
- netdev_dbg(dev, "socket fd=%d not found (gtp1u)\n", fd_gtp1);
- err = -ENOENT;
- goto err1;
+ if (rcu_dereference_sk_user_data(sock->sk)) {
+ sk = ERR_PTR(-EBUSY);
+ goto out_sock;
}
- if (sock1u->sk->sk_protocol != IPPROTO_UDP) {
- netdev_dbg(dev, "socket fd=%d not UDP\n", fd_gtp1);
- err = -EINVAL;
- goto err2;
- }
-
- netdev_dbg(dev, "enable gtp on %p, %p\n", sock0, sock1u);
-
- gtp->sock0 = sock0;
- gtp->sock1u = sock1u;
+ sk = sock->sk;
+ sock_hold(sk);
tuncfg.sk_user_data = gtp;
+ tuncfg.encap_type = type;
tuncfg.encap_rcv = gtp_encap_recv;
tuncfg.encap_destroy = gtp_encap_destroy;
- tuncfg.encap_type = UDP_ENCAP_GTP0;
- setup_udp_tunnel_sock(sock_net(gtp->sock0->sk), gtp->sock0, &tuncfg);
+ setup_udp_tunnel_sock(sock_net(sock->sk), sock, &tuncfg);
- tuncfg.encap_type = UDP_ENCAP_GTP1U;
- setup_udp_tunnel_sock(sock_net(gtp->sock1u->sk), gtp->sock1u, &tuncfg);
-
- err = 0;
-err2:
- sockfd_put(sock1u);
-err1:
- sockfd_put(sock0);
- return err;
+out_sock:
+ sockfd_put(sock);
+ return sk;
}
-static struct net_device *gtp_find_dev(struct net *net, int ifindex)
+static int gtp_encap_enable(struct gtp_dev *gtp, struct nlattr *data[])
{
- struct gtp_net *gn = net_generic(net, gtp_net_id);
- struct gtp_dev *gtp;
+ struct sock *sk1u = NULL;
+ struct sock *sk0 = NULL;
+ unsigned int role = GTP_ROLE_GGSN;
- list_for_each_entry_rcu(gtp, &gn->gtp_dev_list, list) {
- if (ifindex == gtp->dev->ifindex)
- return gtp->dev;
+ if (data[IFLA_GTP_FD0]) {
+ u32 fd0 = nla_get_u32(data[IFLA_GTP_FD0]);
+
+ sk0 = gtp_encap_enable_socket(fd0, UDP_ENCAP_GTP0, gtp);
+ if (IS_ERR(sk0))
+ return PTR_ERR(sk0);
}
- return NULL;
+
+ if (data[IFLA_GTP_FD1]) {
+ u32 fd1 = nla_get_u32(data[IFLA_GTP_FD1]);
+
+ sk1u = gtp_encap_enable_socket(fd1, UDP_ENCAP_GTP1U, gtp);
+ if (IS_ERR(sk1u)) {
+ if (sk0)
+ gtp_encap_disable_sock(sk0);
+ return PTR_ERR(sk1u);
+ }
+ }
+
+ if (data[IFLA_GTP_ROLE]) {
+ role = nla_get_u32(data[IFLA_GTP_ROLE]);
+ if (role > GTP_ROLE_SGSN)
+ return -EINVAL;
+ }
+
+ gtp->sk0 = sk0;
+ gtp->sk1u = sk1u;
+ gtp->role = role;
+
+ return 0;
+}
+
+static struct gtp_dev *gtp_find_dev(struct net *src_net, struct nlattr *nla[])
+{
+ struct gtp_dev *gtp = NULL;
+ struct net_device *dev;
+ struct net *net;
+
+ /* Examine the link attributes and figure out which network namespace
+ * we are talking about.
+ */
+ if (nla[GTPA_NET_NS_FD])
+ net = get_net_ns_by_fd(nla_get_u32(nla[GTPA_NET_NS_FD]));
+ else
+ net = get_net(src_net);
+
+ if (IS_ERR(net))
+ return NULL;
+
+ /* Check if there's an existing gtpX device to configure */
+ dev = dev_get_by_index_rcu(net, nla_get_u32(nla[GTPA_LINK]));
+ if (dev->netdev_ops == >p_netdev_ops)
+ gtp = netdev_priv(dev);
+
+ put_net(net);
+ return gtp;
}
static void ipv4_pdp_fill(struct pdp_ctx *pctx, struct genl_info *info)
{
pctx->gtp_version = nla_get_u32(info->attrs[GTPA_VERSION]);
pctx->af = AF_INET;
- pctx->sgsn_addr_ip4.s_addr =
- nla_get_be32(info->attrs[GTPA_SGSN_ADDRESS]);
+ pctx->peer_addr_ip4.s_addr =
+ nla_get_be32(info->attrs[GTPA_PEER_ADDRESS]);
pctx->ms_addr_ip4.s_addr =
nla_get_be32(info->attrs[GTPA_MS_ADDRESS]);
@@ -899,9 +907,10 @@
}
}
-static int ipv4_pdp_add(struct net_device *dev, struct genl_info *info)
+static int ipv4_pdp_add(struct gtp_dev *gtp, struct sock *sk,
+ struct genl_info *info)
{
- struct gtp_dev *gtp = netdev_priv(dev);
+ struct net_device *dev = gtp->dev;
u32 hash_ms, hash_tid = 0;
struct pdp_ctx *pctx;
bool found = false;
@@ -940,6 +949,9 @@
if (pctx == NULL)
return -ENOMEM;
+ sock_hold(sk);
+ pctx->sk = sk;
+ pctx->dev = gtp->dev;
ipv4_pdp_fill(pctx, info);
atomic_set(&pctx->tx_seq, 0);
@@ -963,31 +975,50 @@
switch (pctx->gtp_version) {
case GTP_V0:
netdev_dbg(dev, "GTPv0-U: new PDP ctx id=%llx ssgn=%pI4 ms=%pI4 (pdp=%p)\n",
- pctx->u.v0.tid, &pctx->sgsn_addr_ip4,
+ pctx->u.v0.tid, &pctx->peer_addr_ip4,
&pctx->ms_addr_ip4, pctx);
break;
case GTP_V1:
netdev_dbg(dev, "GTPv1-U: new PDP ctx id=%x/%x ssgn=%pI4 ms=%pI4 (pdp=%p)\n",
pctx->u.v1.i_tei, pctx->u.v1.o_tei,
- &pctx->sgsn_addr_ip4, &pctx->ms_addr_ip4, pctx);
+ &pctx->peer_addr_ip4, &pctx->ms_addr_ip4, pctx);
break;
}
return 0;
}
+static void pdp_context_free(struct rcu_head *head)
+{
+ struct pdp_ctx *pctx = container_of(head, struct pdp_ctx, rcu_head);
+
+ sock_put(pctx->sk);
+ kfree(pctx);
+}
+
+static void pdp_context_delete(struct pdp_ctx *pctx)
+{
+ hlist_del_rcu(&pctx->hlist_tid);
+ hlist_del_rcu(&pctx->hlist_addr);
+ call_rcu(&pctx->rcu_head, pdp_context_free);
+}
+
static int gtp_genl_new_pdp(struct sk_buff *skb, struct genl_info *info)
{
- struct net_device *dev;
- struct net *net;
+ unsigned int version;
+ struct gtp_dev *gtp;
+ struct sock *sk;
+ int err;
if (!info->attrs[GTPA_VERSION] ||
!info->attrs[GTPA_LINK] ||
- !info->attrs[GTPA_SGSN_ADDRESS] ||
+ !info->attrs[GTPA_PEER_ADDRESS] ||
!info->attrs[GTPA_MS_ADDRESS])
return -EINVAL;
- switch (nla_get_u32(info->attrs[GTPA_VERSION])) {
+ version = nla_get_u32(info->attrs[GTPA_VERSION]);
+
+ switch (version) {
case GTP_V0:
if (!info->attrs[GTPA_TID] ||
!info->attrs[GTPA_FLOW])
@@ -1003,77 +1034,101 @@
return -EINVAL;
}
- net = gtp_genl_get_net(sock_net(skb->sk), info->attrs);
- if (IS_ERR(net))
- return PTR_ERR(net);
+ rcu_read_lock();
- /* Check if there's an existing gtpX device to configure */
- dev = gtp_find_dev(net, nla_get_u32(info->attrs[GTPA_LINK]));
- if (dev == NULL) {
- put_net(net);
- return -ENODEV;
+ gtp = gtp_find_dev(sock_net(skb->sk), info->attrs);
+ if (!gtp) {
+ err = -ENODEV;
+ goto out_unlock;
}
- put_net(net);
- return ipv4_pdp_add(dev, info);
+ if (version == GTP_V0)
+ sk = gtp->sk0;
+ else if (version == GTP_V1)
+ sk = gtp->sk1u;
+ else
+ sk = NULL;
+
+ if (!sk) {
+ err = -ENODEV;
+ goto out_unlock;
+ }
+
+ err = ipv4_pdp_add(gtp, sk, info);
+
+out_unlock:
+ rcu_read_unlock();
+ return err;
+}
+
+static struct pdp_ctx *gtp_find_pdp_by_link(struct net *net,
+ struct nlattr *nla[])
+{
+ struct gtp_dev *gtp;
+
+ gtp = gtp_find_dev(net, nla);
+ if (!gtp)
+ return ERR_PTR(-ENODEV);
+
+ if (nla[GTPA_MS_ADDRESS]) {
+ __be32 ip = nla_get_be32(nla[GTPA_MS_ADDRESS]);
+
+ return ipv4_pdp_find(gtp, ip);
+ } else if (nla[GTPA_VERSION]) {
+ u32 gtp_version = nla_get_u32(nla[GTPA_VERSION]);
+
+ if (gtp_version == GTP_V0 && nla[GTPA_TID])
+ return gtp0_pdp_find(gtp, nla_get_u64(nla[GTPA_TID]));
+ else if (gtp_version == GTP_V1 && nla[GTPA_I_TEI])
+ return gtp1_pdp_find(gtp, nla_get_u32(nla[GTPA_I_TEI]));
+ }
+
+ return ERR_PTR(-EINVAL);
+}
+
+static struct pdp_ctx *gtp_find_pdp(struct net *net, struct nlattr *nla[])
+{
+ struct pdp_ctx *pctx;
+
+ if (nla[GTPA_LINK])
+ pctx = gtp_find_pdp_by_link(net, nla);
+ else
+ pctx = ERR_PTR(-EINVAL);
+
+ if (!pctx)
+ pctx = ERR_PTR(-ENOENT);
+
+ return pctx;
}
static int gtp_genl_del_pdp(struct sk_buff *skb, struct genl_info *info)
{
- struct net_device *dev;
struct pdp_ctx *pctx;
- struct gtp_dev *gtp;
- struct net *net;
+ int err = 0;
- if (!info->attrs[GTPA_VERSION] ||
- !info->attrs[GTPA_LINK])
+ if (!info->attrs[GTPA_VERSION])
return -EINVAL;
- net = gtp_genl_get_net(sock_net(skb->sk), info->attrs);
- if (IS_ERR(net))
- return PTR_ERR(net);
+ rcu_read_lock();
- /* Check if there's an existing gtpX device to configure */
- dev = gtp_find_dev(net, nla_get_u32(info->attrs[GTPA_LINK]));
- if (dev == NULL) {
- put_net(net);
- return -ENODEV;
+ pctx = gtp_find_pdp(sock_net(skb->sk), info->attrs);
+ if (IS_ERR(pctx)) {
+ err = PTR_ERR(pctx);
+ goto out_unlock;
}
- put_net(net);
-
- gtp = netdev_priv(dev);
-
- switch (nla_get_u32(info->attrs[GTPA_VERSION])) {
- case GTP_V0:
- if (!info->attrs[GTPA_TID])
- return -EINVAL;
- pctx = gtp0_pdp_find(gtp, nla_get_u64(info->attrs[GTPA_TID]));
- break;
- case GTP_V1:
- if (!info->attrs[GTPA_I_TEI])
- return -EINVAL;
- pctx = gtp1_pdp_find(gtp, nla_get_u64(info->attrs[GTPA_I_TEI]));
- break;
-
- default:
- return -EINVAL;
- }
-
- if (pctx == NULL)
- return -ENOENT;
if (pctx->gtp_version == GTP_V0)
- netdev_dbg(dev, "GTPv0-U: deleting tunnel id = %llx (pdp %p)\n",
+ netdev_dbg(pctx->dev, "GTPv0-U: deleting tunnel id = %llx (pdp %p)\n",
pctx->u.v0.tid, pctx);
else if (pctx->gtp_version == GTP_V1)
- netdev_dbg(dev, "GTPv1-U: deleting tunnel id = %x/%x (pdp %p)\n",
+ netdev_dbg(pctx->dev, "GTPv1-U: deleting tunnel id = %x/%x (pdp %p)\n",
pctx->u.v1.i_tei, pctx->u.v1.o_tei, pctx);
- hlist_del_rcu(&pctx->hlist_tid);
- hlist_del_rcu(&pctx->hlist_addr);
- kfree_rcu(pctx, rcu_head);
+ pdp_context_delete(pctx);
- return 0;
+out_unlock:
+ rcu_read_unlock();
+ return err;
}
static struct genl_family gtp_genl_family;
@@ -1089,7 +1144,7 @@
goto nlmsg_failure;
if (nla_put_u32(skb, GTPA_VERSION, pctx->gtp_version) ||
- nla_put_be32(skb, GTPA_SGSN_ADDRESS, pctx->sgsn_addr_ip4.s_addr) ||
+ nla_put_be32(skb, GTPA_PEER_ADDRESS, pctx->peer_addr_ip4.s_addr) ||
nla_put_be32(skb, GTPA_MS_ADDRESS, pctx->ms_addr_ip4.s_addr))
goto nla_put_failure;
@@ -1117,59 +1172,17 @@
static int gtp_genl_get_pdp(struct sk_buff *skb, struct genl_info *info)
{
struct pdp_ctx *pctx = NULL;
- struct net_device *dev;
struct sk_buff *skb2;
- struct gtp_dev *gtp;
- u32 gtp_version;
- struct net *net;
int err;
- if (!info->attrs[GTPA_VERSION] ||
- !info->attrs[GTPA_LINK])
+ if (!info->attrs[GTPA_VERSION])
return -EINVAL;
- gtp_version = nla_get_u32(info->attrs[GTPA_VERSION]);
- switch (gtp_version) {
- case GTP_V0:
- case GTP_V1:
- break;
- default:
- return -EINVAL;
- }
-
- net = gtp_genl_get_net(sock_net(skb->sk), info->attrs);
- if (IS_ERR(net))
- return PTR_ERR(net);
-
- /* Check if there's an existing gtpX device to configure */
- dev = gtp_find_dev(net, nla_get_u32(info->attrs[GTPA_LINK]));
- if (dev == NULL) {
- put_net(net);
- return -ENODEV;
- }
- put_net(net);
-
- gtp = netdev_priv(dev);
-
rcu_read_lock();
- if (gtp_version == GTP_V0 &&
- info->attrs[GTPA_TID]) {
- u64 tid = nla_get_u64(info->attrs[GTPA_TID]);
- pctx = gtp0_pdp_find(gtp, tid);
- } else if (gtp_version == GTP_V1 &&
- info->attrs[GTPA_I_TEI]) {
- u32 tid = nla_get_u32(info->attrs[GTPA_I_TEI]);
-
- pctx = gtp1_pdp_find(gtp, tid);
- } else if (info->attrs[GTPA_MS_ADDRESS]) {
- __be32 ip = nla_get_be32(info->attrs[GTPA_MS_ADDRESS]);
-
- pctx = ipv4_pdp_find(gtp, ip);
- }
-
- if (pctx == NULL) {
- err = -ENOENT;
+ pctx = gtp_find_pdp(sock_net(skb->sk), info->attrs);
+ if (IS_ERR(pctx)) {
+ err = PTR_ERR(pctx);
goto err_unlock;
}
@@ -1242,7 +1255,7 @@
[GTPA_LINK] = { .type = NLA_U32, },
[GTPA_VERSION] = { .type = NLA_U32, },
[GTPA_TID] = { .type = NLA_U64, },
- [GTPA_SGSN_ADDRESS] = { .type = NLA_U32, },
+ [GTPA_PEER_ADDRESS] = { .type = NLA_U32, },
[GTPA_MS_ADDRESS] = { .type = NLA_U32, },
[GTPA_FLOW] = { .type = NLA_U16, },
[GTPA_NET_NS_FD] = { .type = NLA_U32, },
diff --git a/drivers/net/hyperv/hyperv_net.h b/drivers/net/hyperv/hyperv_net.h
index f9f3dba..262b2ea 100644
--- a/drivers/net/hyperv/hyperv_net.h
+++ b/drivers/net/hyperv/hyperv_net.h
@@ -196,6 +196,7 @@
const struct ndis_tcp_ip_checksum_info *csum_info,
const struct ndis_pkt_8021q_info *vlan);
void netvsc_channel_cb(void *context);
+int netvsc_poll(struct napi_struct *napi, int budget);
int rndis_filter_open(struct netvsc_device *nvdev);
int rndis_filter_close(struct netvsc_device *nvdev);
int rndis_filter_device_add(struct hv_device *dev,
@@ -632,7 +633,7 @@
#define NETVSC_PACKET_SIZE 4096
-#define VRSS_SEND_TAB_SIZE 16
+#define VRSS_SEND_TAB_SIZE 16 /* must be power of 2 */
#define VRSS_CHANNEL_MAX 64
#define VRSS_CHANNEL_DEFAULT 8
@@ -685,7 +686,7 @@
/* point back to our device context */
struct hv_device *device_ctx;
/* netvsc_device */
- struct netvsc_device *nvdev;
+ struct netvsc_device __rcu *nvdev;
/* reconfigure work */
struct delayed_work dwork;
/* last reconfig time */
@@ -707,9 +708,6 @@
u32 speed;
struct netvsc_ethtool_stats eth_stats;
- /* the device is going away */
- bool start_remove;
-
/* State to manage the associated VF interface. */
struct net_device __rcu *vf_netdev;
@@ -722,6 +720,8 @@
/* Per channel data */
struct netvsc_channel {
struct vmbus_channel *channel;
+ const struct vmpacket_descriptor *desc;
+ struct napi_struct napi;
struct multi_send_data msd;
struct multi_recv_comp mrc;
atomic_t queue_sends;
@@ -751,7 +751,6 @@
u32 send_section_cnt;
u32 send_section_size;
unsigned long *send_section_map;
- int map_words;
/* Used for NetVSP initialization protocol */
struct completion channel_init_wait;
@@ -761,8 +760,8 @@
u32 max_chn;
u32 num_chn;
- spinlock_t sc_lock; /* Protects num_sc_offered variable */
- u32 num_sc_offered;
+
+ refcount_t sc_offered;
/* Holds rndis device info */
void *extension;
@@ -777,6 +776,8 @@
atomic_t open_cnt;
struct netvsc_channel chan_table[VRSS_CHANNEL_MAX];
+
+ struct rcu_head rcu;
};
static inline struct netvsc_device *
@@ -1425,9 +1426,6 @@
((void *) rndis_msg)
-#define __struct_bcount(x)
-
-
#define RNDIS_HEADER_SIZE (sizeof(struct rndis_message) - \
sizeof(union rndis_message_container))
diff --git a/drivers/net/hyperv/netvsc.c b/drivers/net/hyperv/netvsc.c
index 8dd0b87..15749d3 100644
--- a/drivers/net/hyperv/netvsc.c
+++ b/drivers/net/hyperv/netvsc.c
@@ -80,8 +80,10 @@
return net_device;
}
-static void free_netvsc_device(struct netvsc_device *nvdev)
+static void free_netvsc_device(struct rcu_head *head)
{
+ struct netvsc_device *nvdev
+ = container_of(head, struct netvsc_device, rcu);
int i;
for (i = 0; i < VRSS_CHANNEL_MAX; i++)
@@ -90,14 +92,9 @@
kfree(nvdev);
}
-
-static inline bool netvsc_channel_idle(const struct netvsc_device *net_device,
- u16 q_idx)
+static void free_netvsc_device_rcu(struct netvsc_device *nvdev)
{
- const struct netvsc_channel *nvchan = &net_device->chan_table[q_idx];
-
- return atomic_read(&net_device->num_outstanding_recvs) == 0 &&
- atomic_read(&nvchan->queue_sends) == 0;
+ call_rcu(&nvdev->rcu, free_netvsc_device);
}
static struct netvsc_device *get_outbound_net_device(struct hv_device *device)
@@ -138,6 +135,13 @@
sizeof(struct nvsp_message),
(unsigned long)revoke_packet,
VM_PKT_DATA_INBAND, 0);
+ /* If the failure is because the channel is rescinded;
+ * ignore the failure since we cannot send on a rescinded
+ * channel. This would allow us to properly cleanup
+ * even when the channel is rescinded.
+ */
+ if (device->channel->rescind)
+ ret = 0;
/*
* If we failed here, we might as well return and
* have a leak rather than continue and a bugchk
@@ -198,6 +202,15 @@
sizeof(struct nvsp_message),
(unsigned long)revoke_packet,
VM_PKT_DATA_INBAND, 0);
+
+ /* If the failure is because the channel is rescinded;
+ * ignore the failure since we cannot send on a rescinded
+ * channel. This would allow us to properly cleanup
+ * even when the channel is rescinded.
+ */
+ if (device->channel->rescind)
+ ret = 0;
+
/* If we failed here, we might as well return and
* have a leak rather than continue and a bugchk
*/
@@ -236,6 +249,7 @@
struct netvsc_device *net_device;
struct nvsp_message *init_packet;
struct net_device *ndev;
+ size_t map_words;
int node;
net_device = get_outbound_net_device(device);
@@ -401,11 +415,9 @@
net_device->send_section_size, net_device->send_section_cnt);
/* Setup state for managing the send buffer. */
- net_device->map_words = DIV_ROUND_UP(net_device->send_section_cnt,
- BITS_PER_LONG);
+ map_words = DIV_ROUND_UP(net_device->send_section_cnt, BITS_PER_LONG);
- net_device->send_section_map = kcalloc(net_device->map_words,
- sizeof(ulong), GFP_KERNEL);
+ net_device->send_section_map = kcalloc(map_words, sizeof(ulong), GFP_KERNEL);
if (net_device->send_section_map == NULL) {
ret = -ENOMEM;
goto cleanup;
@@ -556,10 +568,11 @@
struct net_device *ndev = hv_get_drvdata(device);
struct net_device_context *net_device_ctx = netdev_priv(ndev);
struct netvsc_device *net_device = net_device_ctx->nvdev;
+ int i;
netvsc_disconnect_vsp(device);
- net_device_ctx->nvdev = NULL;
+ RCU_INIT_POINTER(net_device_ctx->nvdev, NULL);
/*
* At this point, no one should be accessing net_device
@@ -570,8 +583,12 @@
/* Now, we can close the channel safely */
vmbus_close(device->channel);
+ /* And dissassociate NAPI context from device */
+ for (i = 0; i < net_device->num_chn; i++)
+ netif_napi_del(&net_device->chan_table[i].napi);
+
/* Release all resources */
- free_netvsc_device(net_device);
+ free_netvsc_device_rcu(net_device);
}
#define RING_AVAIL_PERCENT_HIWATER 20
@@ -600,11 +617,11 @@
static void netvsc_send_tx_complete(struct netvsc_device *net_device,
struct vmbus_channel *incoming_channel,
struct hv_device *device,
- struct vmpacket_descriptor *packet)
+ const struct vmpacket_descriptor *desc,
+ int budget)
{
- struct sk_buff *skb = (struct sk_buff *)(unsigned long)packet->trans_id;
+ struct sk_buff *skb = (struct sk_buff *)(unsigned long)desc->trans_id;
struct net_device *ndev = hv_get_drvdata(device);
- struct net_device_context *net_device_ctx = netdev_priv(ndev);
struct vmbus_channel *channel = device->channel;
u16 q_idx = 0;
int queue_sends;
@@ -628,7 +645,7 @@
tx_stats->bytes += packet->total_bytes;
u64_stats_update_end(&tx_stats->syncp);
- dev_consume_skb_any(skb);
+ napi_consume_skb(skb, budget);
}
queue_sends =
@@ -638,7 +655,6 @@
wake_up(&net_device->wait_drain);
if (netif_tx_queue_stopped(netdev_get_tx_queue(ndev, q_idx)) &&
- !net_device_ctx->start_remove &&
(hv_ringbuf_avail_percent(&channel->outbound) > RING_AVAIL_PERCENT_HIWATER ||
queue_sends < 1))
netif_tx_wake_queue(netdev_get_tx_queue(ndev, q_idx));
@@ -647,14 +663,12 @@
static void netvsc_send_completion(struct netvsc_device *net_device,
struct vmbus_channel *incoming_channel,
struct hv_device *device,
- struct vmpacket_descriptor *packet)
+ const struct vmpacket_descriptor *desc,
+ int budget)
{
- struct nvsp_message *nvsp_packet;
+ struct nvsp_message *nvsp_packet = hv_pkt_data(desc);
struct net_device *ndev = hv_get_drvdata(device);
- nvsp_packet = (struct nvsp_message *)((unsigned long)packet +
- (packet->offset8 << 3));
-
switch (nvsp_packet->hdr.msg_type) {
case NVSP_MSG_TYPE_INIT_COMPLETE:
case NVSP_MSG1_TYPE_SEND_RECV_BUF_COMPLETE:
@@ -668,7 +682,7 @@
case NVSP_MSG1_TYPE_SEND_RNDIS_PKT_COMPLETE:
netvsc_send_tx_complete(net_device, incoming_channel,
- device, packet);
+ device, desc, budget);
break;
default:
@@ -683,7 +697,7 @@
unsigned long *map_addr = net_device->send_section_map;
unsigned int i;
- for_each_clear_bit(i, map_addr, net_device->map_words) {
+ for_each_clear_bit(i, map_addr, net_device->send_section_cnt) {
if (sync_test_and_set_bit(i, map_addr) == 0)
return i;
}
@@ -710,8 +724,7 @@
packet->page_buf_cnt;
/* Add padding */
- if (skb && skb->xmit_more && remain &&
- !packet->cp_partial) {
+ if (skb->xmit_more && remain && !packet->cp_partial) {
padding = net_device->pkt_align - remain;
rndis_msg->msg_len += padding;
packet->total_data_buflen += padding;
@@ -869,9 +882,7 @@
if (msdp->pkt)
msd_len = msdp->pkt->total_data_buflen;
- try_batch = (skb != NULL) && msd_len > 0 && msdp->count <
- net_device->max_pkt;
-
+ try_batch = msd_len > 0 && msdp->count < net_device->max_pkt;
if (try_batch && msd_len + pktlen + net_device->pkt_align <
net_device->send_section_size) {
section_index = msdp->pkt->send_buf_index;
@@ -881,7 +892,7 @@
section_index = msdp->pkt->send_buf_index;
packet->cp_partial = true;
- } else if ((skb != NULL) && pktlen + net_device->pkt_align <
+ } else if (pktlen + net_device->pkt_align <
net_device->send_section_size) {
section_index = netvsc_get_next_send_section(net_device);
if (section_index != NETVSC_INVALID_INDEX) {
@@ -1066,28 +1077,29 @@
return rcd;
}
-static void netvsc_receive(struct net_device *ndev,
+static int netvsc_receive(struct net_device *ndev,
struct netvsc_device *net_device,
struct net_device_context *net_device_ctx,
struct hv_device *device,
struct vmbus_channel *channel,
- struct vmtransfer_page_packet_header *vmxferpage_packet,
+ const struct vmpacket_descriptor *desc,
struct nvsp_message *nvsp)
{
+ const struct vmtransfer_page_packet_header *vmxferpage_packet
+ = container_of(desc, const struct vmtransfer_page_packet_header, d);
+ u16 q_idx = channel->offermsg.offer.sub_channel_index;
char *recv_buf = net_device->recv_buf;
u32 status = NVSP_STAT_SUCCESS;
int i;
int count = 0;
int ret;
- struct recv_comp_data *rcd;
- u16 q_idx = channel->offermsg.offer.sub_channel_index;
/* Make sure this is a valid nvsp packet */
if (unlikely(nvsp->hdr.msg_type != NVSP_MSG1_TYPE_SEND_RNDIS_PKT)) {
netif_err(net_device_ctx, rx_err, ndev,
"Unknown nvsp packet type received %u\n",
nvsp->hdr.msg_type);
- return;
+ return 0;
}
if (unlikely(vmxferpage_packet->xfer_pageset_id != NETVSC_RECEIVE_BUFFER_ID)) {
@@ -1095,7 +1107,7 @@
"Invalid xfer page set id - expecting %x got %x\n",
NETVSC_RECEIVE_BUFFER_ID,
vmxferpage_packet->xfer_pageset_id);
- return;
+ return 0;
}
count = vmxferpage_packet->range_cnt;
@@ -1111,26 +1123,26 @@
channel, data, buflen);
}
- if (!net_device->chan_table[q_idx].mrc.buf) {
+ if (net_device->chan_table[q_idx].mrc.buf) {
+ struct recv_comp_data *rcd;
+
+ rcd = get_recv_comp_slot(net_device, channel, q_idx);
+ if (rcd) {
+ rcd->tid = vmxferpage_packet->d.trans_id;
+ rcd->status = status;
+ } else {
+ netdev_err(ndev, "Recv_comp full buf q:%hd, tid:%llx\n",
+ q_idx, vmxferpage_packet->d.trans_id);
+ }
+ } else {
ret = netvsc_send_recv_completion(channel,
vmxferpage_packet->d.trans_id,
status);
if (ret)
netdev_err(ndev, "Recv_comp q:%hd, tid:%llx, err:%d\n",
q_idx, vmxferpage_packet->d.trans_id, ret);
- return;
}
-
- rcd = get_recv_comp_slot(net_device, channel, q_idx);
-
- if (!rcd) {
- netdev_err(ndev, "Recv_comp full buf q:%hd, tid:%llx\n",
- q_idx, vmxferpage_packet->d.trans_id);
- return;
- }
-
- rcd->tid = vmxferpage_packet->d.trans_id;
- rcd->status = status;
+ return count;
}
static void netvsc_send_table(struct hv_device *hdev,
@@ -1176,28 +1188,25 @@
}
}
-static void netvsc_process_raw_pkt(struct hv_device *device,
- struct vmbus_channel *channel,
- struct netvsc_device *net_device,
- struct net_device *ndev,
- u64 request_id,
- struct vmpacket_descriptor *desc)
+static int netvsc_process_raw_pkt(struct hv_device *device,
+ struct vmbus_channel *channel,
+ struct netvsc_device *net_device,
+ struct net_device *ndev,
+ const struct vmpacket_descriptor *desc,
+ int budget)
{
struct net_device_context *net_device_ctx = netdev_priv(ndev);
- struct nvsp_message *nvmsg
- = (struct nvsp_message *)((unsigned long)desc
- + (desc->offset8 << 3));
+ struct nvsp_message *nvmsg = hv_pkt_data(desc);
switch (desc->type) {
case VM_PKT_COMP:
- netvsc_send_completion(net_device, channel, device, desc);
+ netvsc_send_completion(net_device, channel, device,
+ desc, budget);
break;
case VM_PKT_DATA_USING_XFER_PAGES:
- netvsc_receive(ndev, net_device, net_device_ctx,
- device, channel,
- (struct vmtransfer_page_packet_header *)desc,
- nvmsg);
+ return netvsc_receive(ndev, net_device, net_device_ctx,
+ device, channel, desc, nvmsg);
break;
case VM_PKT_DATA_INBAND:
@@ -1206,53 +1215,74 @@
default:
netdev_err(ndev, "unhandled packet type %d, tid %llx\n",
- desc->type, request_id);
+ desc->type, desc->trans_id);
break;
}
+
+ return 0;
}
-void netvsc_channel_cb(void *context)
+static struct hv_device *netvsc_channel_to_device(struct vmbus_channel *channel)
{
- struct vmbus_channel *channel = context;
+ struct vmbus_channel *primary = channel->primary_channel;
+
+ return primary ? primary->device_obj : channel->device_obj;
+}
+
+/* Network processing softirq
+ * Process data in incoming ring buffer from host
+ * Stops when ring is empty or budget is met or exceeded.
+ */
+int netvsc_poll(struct napi_struct *napi, int budget)
+{
+ struct netvsc_channel *nvchan
+ = container_of(napi, struct netvsc_channel, napi);
+ struct vmbus_channel *channel = nvchan->channel;
+ struct hv_device *device = netvsc_channel_to_device(channel);
u16 q_idx = channel->offermsg.offer.sub_channel_index;
- struct hv_device *device;
- struct netvsc_device *net_device;
- struct vmpacket_descriptor *desc;
- struct net_device *ndev;
- bool need_to_commit = false;
+ struct net_device *ndev = hv_get_drvdata(device);
+ struct netvsc_device *net_device = net_device_to_netvsc_device(ndev);
+ int work_done = 0;
- if (channel->primary_channel != NULL)
- device = channel->primary_channel->device_obj;
- else
- device = channel->device_obj;
+ /* If starting a new interval */
+ if (!nvchan->desc)
+ nvchan->desc = hv_pkt_iter_first(channel);
- ndev = hv_get_drvdata(device);
- if (unlikely(!ndev))
- return;
-
- net_device = net_device_to_netvsc_device(ndev);
- if (unlikely(!net_device))
- return;
-
- if (unlikely(net_device->destroy &&
- netvsc_channel_idle(net_device, q_idx)))
- return;
-
- /* commit_rd_index() -> hv_signal_on_read() needs this. */
- init_cached_read_index(channel);
-
- while ((desc = get_next_pkt_raw(channel)) != NULL) {
- netvsc_process_raw_pkt(device, channel, net_device,
- ndev, desc->trans_id, desc);
-
- put_pkt_raw(channel, desc);
- need_to_commit = true;
+ while (nvchan->desc && work_done < budget) {
+ work_done += netvsc_process_raw_pkt(device, channel, net_device,
+ ndev, nvchan->desc, budget);
+ nvchan->desc = hv_pkt_iter_next(channel, nvchan->desc);
}
- if (need_to_commit)
- commit_rd_index(channel);
+ /* If receive ring was exhausted
+ * and not doing busy poll
+ * then re-enable host interrupts
+ * and reschedule if ring is not empty.
+ */
+ if (work_done < budget &&
+ napi_complete_done(napi, work_done) &&
+ hv_end_read(&channel->inbound) != 0)
+ napi_reschedule(napi);
netvsc_chk_recv_comp(net_device, channel, q_idx);
+
+ /* Driver may overshoot since multiple packets per descriptor */
+ return min(work_done, budget);
+}
+
+/* Call back when data is available in host ring buffer.
+ * Processing is deferred until network softirq (NAPI)
+ */
+void netvsc_channel_cb(void *context)
+{
+ struct netvsc_channel *nvchan = context;
+
+ if (napi_schedule_prep(&nvchan->napi)) {
+ /* disable interupts from host */
+ hv_begin_read(&nvchan->channel->inbound);
+
+ __napi_schedule(&nvchan->napi);
+ }
}
/*
@@ -1274,10 +1304,29 @@
net_device->ring_size = ring_size;
+ /* Because the device uses NAPI, all the interrupt batching and
+ * control is done via Net softirq, not the channel handling
+ */
+ set_channel_read_mode(device->channel, HV_CALL_ISR);
+
+ /* If we're reopening the device we may have multiple queues, fill the
+ * chn_table with the default channel to use it before subchannels are
+ * opened.
+ * Initialize the channel state before we open;
+ * we can be interrupted as soon as we open the channel.
+ */
+
+ for (i = 0; i < VRSS_CHANNEL_MAX; i++) {
+ struct netvsc_channel *nvchan = &net_device->chan_table[i];
+
+ nvchan->channel = device->channel;
+ }
+
/* Open the channel */
ret = vmbus_open(device->channel, ring_size * PAGE_SIZE,
ring_size * PAGE_SIZE, NULL, 0,
- netvsc_channel_cb, device->channel);
+ netvsc_channel_cb,
+ net_device->chan_table);
if (ret != 0) {
netdev_err(ndev, "unable to open channel: %d\n", ret);
@@ -1287,19 +1336,15 @@
/* Channel is opened */
netdev_dbg(ndev, "hv_netvsc channel opened successfully\n");
- /* If we're reopening the device we may have multiple queues, fill the
- * chn_table with the default channel to use it before subchannels are
- * opened.
- */
- for (i = 0; i < VRSS_CHANNEL_MAX; i++)
- net_device->chan_table[i].channel = device->channel;
+ /* Enable NAPI handler for init callbacks */
+ netif_napi_add(ndev, &net_device->chan_table[0].napi,
+ netvsc_poll, NAPI_POLL_WEIGHT);
+ napi_enable(&net_device->chan_table[0].napi);
/* Writing nvdev pointer unlocks netvsc_send(), make sure chn_table is
* populated.
*/
- wmb();
-
- net_device_ctx->nvdev = net_device;
+ rcu_assign_pointer(net_device_ctx->nvdev, net_device);
/* Connect with the NetVsp */
ret = netvsc_connect_vsp(device);
@@ -1312,11 +1357,13 @@
return ret;
close:
+ netif_napi_del(&net_device->chan_table[0].napi);
+
/* Now, we can close the channel safely */
vmbus_close(device->channel);
cleanup:
- free_netvsc_device(net_device);
+ free_netvsc_device(&net_device->rcu);
return ret;
}
diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c
index 5ede87f..4421a6d 100644
--- a/drivers/net/hyperv/netvsc_drv.c
+++ b/drivers/net/hyperv/netvsc_drv.c
@@ -62,7 +62,7 @@
container_of(w, struct net_device_context, work);
struct hv_device *device_obj = ndevctx->device_ctx;
struct net_device *ndev = hv_get_drvdata(device_obj);
- struct netvsc_device *nvdev = ndevctx->nvdev;
+ struct netvsc_device *nvdev = rcu_dereference(ndevctx->nvdev);
struct rndis_device *rdev;
if (!nvdev)
@@ -116,7 +116,7 @@
static int netvsc_close(struct net_device *net)
{
struct net_device_context *net_device_ctx = netdev_priv(net);
- struct netvsc_device *nvdev = net_device_ctx->nvdev;
+ struct netvsc_device *nvdev = rtnl_dereference(net_device_ctx->nvdev);
int ret;
u32 aread, awrite, i, msec = 10, retry = 0, retry_max = 20;
struct vmbus_channel *chn;
@@ -191,6 +191,54 @@
return ppi;
}
+/* Azure hosts don't support non-TCP port numbers in hashing yet. We compute
+ * hash for non-TCP traffic with only IP numbers.
+ */
+static inline u32 netvsc_get_hash(struct sk_buff *skb, struct sock *sk)
+{
+ struct flow_keys flow;
+ u32 hash;
+ static u32 hashrnd __read_mostly;
+
+ net_get_random_once(&hashrnd, sizeof(hashrnd));
+
+ if (!skb_flow_dissect_flow_keys(skb, &flow, 0))
+ return 0;
+
+ if (flow.basic.ip_proto == IPPROTO_TCP) {
+ return skb_get_hash(skb);
+ } else {
+ if (flow.basic.n_proto == htons(ETH_P_IP))
+ hash = jhash2((u32 *)&flow.addrs.v4addrs, 2, hashrnd);
+ else if (flow.basic.n_proto == htons(ETH_P_IPV6))
+ hash = jhash2((u32 *)&flow.addrs.v6addrs, 8, hashrnd);
+ else
+ hash = 0;
+
+ skb_set_hash(skb, hash, PKT_HASH_TYPE_L3);
+ }
+
+ return hash;
+}
+
+static inline int netvsc_get_tx_queue(struct net_device *ndev,
+ struct sk_buff *skb, int old_idx)
+{
+ const struct net_device_context *ndc = netdev_priv(ndev);
+ struct sock *sk = skb->sk;
+ int q_idx;
+
+ q_idx = ndc->tx_send_table[netvsc_get_hash(skb, sk) &
+ (VRSS_SEND_TAB_SIZE - 1)];
+
+ /* If queue index changed record the new value */
+ if (q_idx != old_idx &&
+ sk && sk_fullsock(sk) && rcu_access_pointer(sk->sk_dst_cache))
+ sk_tx_queue_set(sk, q_idx);
+
+ return q_idx;
+}
+
/*
* Select queue for transmit.
*
@@ -205,24 +253,22 @@
static u16 netvsc_select_queue(struct net_device *ndev, struct sk_buff *skb,
void *accel_priv, select_queue_fallback_t fallback)
{
- struct net_device_context *net_device_ctx = netdev_priv(ndev);
unsigned int num_tx_queues = ndev->real_num_tx_queues;
- struct sock *sk = skb->sk;
- int q_idx = sk_tx_queue_get(sk);
+ int q_idx = sk_tx_queue_get(skb->sk);
- if (q_idx < 0 || skb->ooo_okay || q_idx >= num_tx_queues) {
- u16 hash = __skb_tx_hash(ndev, skb, VRSS_SEND_TAB_SIZE);
- int new_idx;
-
- new_idx = net_device_ctx->tx_send_table[hash] % num_tx_queues;
-
- if (q_idx != new_idx && sk &&
- sk_fullsock(sk) && rcu_access_pointer(sk->sk_dst_cache))
- sk_tx_queue_set(sk, new_idx);
-
- q_idx = new_idx;
+ if (q_idx < 0 || skb->ooo_okay) {
+ /* If forwarding a packet, we use the recorded queue when
+ * available for better cache locality.
+ */
+ if (skb_rx_queue_recorded(skb))
+ q_idx = skb_get_rx_queue(skb);
+ else
+ q_idx = netvsc_get_tx_queue(ndev, skb, q_idx);
}
+ while (unlikely(q_idx >= num_tx_queues))
+ q_idx -= num_tx_queues;
+
return q_idx;
}
@@ -584,13 +630,14 @@
}
static struct sk_buff *netvsc_alloc_recv_skb(struct net_device *net,
+ struct napi_struct *napi,
const struct ndis_tcp_ip_checksum_info *csum_info,
const struct ndis_pkt_8021q_info *vlan,
void *data, u32 buflen)
{
struct sk_buff *skb;
- skb = netdev_alloc_skb_ip_align(net, buflen);
+ skb = napi_alloc_skb(napi, buflen);
if (!skb)
return skb;
@@ -636,12 +683,12 @@
const struct ndis_pkt_8021q_info *vlan)
{
struct net_device_context *net_device_ctx = netdev_priv(net);
- struct netvsc_device *net_device = net_device_ctx->nvdev;
+ struct netvsc_device *net_device;
+ u16 q_idx = channel->offermsg.offer.sub_channel_index;
+ struct netvsc_channel *nvchan;
struct net_device *vf_netdev;
struct sk_buff *skb;
struct netvsc_stats *rx_stats;
- u16 q_idx = channel->offermsg.offer.sub_channel_index;
-
if (net->reg_state != NETREG_REGISTERED)
return NVSP_STAT_FAIL;
@@ -654,13 +701,20 @@
* interface in the guest.
*/
rcu_read_lock();
+ net_device = rcu_dereference(net_device_ctx->nvdev);
+ if (unlikely(!net_device))
+ goto drop;
+
+ nvchan = &net_device->chan_table[q_idx];
vf_netdev = rcu_dereference(net_device_ctx->vf_netdev);
if (vf_netdev && (vf_netdev->flags & IFF_UP))
net = vf_netdev;
/* Allocate a skb - TODO direct I/O to pages? */
- skb = netvsc_alloc_recv_skb(net, csum_info, vlan, data, len);
+ skb = netvsc_alloc_recv_skb(net, &nvchan->napi,
+ csum_info, vlan, data, len);
if (unlikely(!skb)) {
+drop:
++net->stats.rx_dropped;
rcu_read_unlock();
return NVSP_STAT_FAIL;
@@ -674,7 +728,7 @@
* on the synthetic device because modifying the VF device
* statistics will not work correctly.
*/
- rx_stats = &net_device->chan_table[q_idx].rx_stats;
+ rx_stats = &nvchan->rx_stats;
u64_stats_update_begin(&rx_stats->syncp);
rx_stats->packets++;
rx_stats->bytes += len;
@@ -685,12 +739,7 @@
++rx_stats->multicast;
u64_stats_update_end(&rx_stats->syncp);
- /*
- * Pass the skb back up. Network stack will deallocate the skb when it
- * is done.
- * TODO - use NAPI?
- */
- netif_receive_skb(skb);
+ napi_gro_receive(&nvchan->napi, skb);
rcu_read_unlock();
return 0;
@@ -707,7 +756,7 @@
struct ethtool_channels *channel)
{
struct net_device_context *net_device_ctx = netdev_priv(net);
- struct netvsc_device *nvdev = net_device_ctx->nvdev;
+ struct netvsc_device *nvdev = rtnl_dereference(net_device_ctx->nvdev);
if (nvdev) {
channel->max_combined = nvdev->max_chn;
@@ -744,8 +793,9 @@
{
struct net_device_context *net_device_ctx = netdev_priv(net);
struct hv_device *dev = net_device_ctx->device_ctx;
- struct netvsc_device *nvdev = net_device_ctx->nvdev;
+ struct netvsc_device *nvdev = rtnl_dereference(net_device_ctx->nvdev);
unsigned int count = channels->combined_count;
+ bool was_running;
int ret;
/* We do not support separate count for rx, tx, or other */
@@ -756,7 +806,7 @@
if (count > net->num_tx_queues || count > net->num_rx_queues)
return -EINVAL;
- if (net_device_ctx->start_remove || !nvdev || nvdev->destroy)
+ if (!nvdev || nvdev->destroy)
return -ENODEV;
if (nvdev->nvsp_version < NVSP_PROTOCOL_VERSION_5)
@@ -765,11 +815,13 @@
if (count > nvdev->max_chn)
return -EINVAL;
- ret = netvsc_close(net);
- if (ret)
- return ret;
+ was_running = netif_running(net);
+ if (was_running) {
+ ret = netvsc_close(net);
+ if (ret)
+ return ret;
+ }
- net_device_ctx->start_remove = true;
rndis_filter_device_remove(dev, nvdev);
ret = netvsc_set_queues(net, dev, count);
@@ -778,8 +830,8 @@
else
netvsc_set_queues(net, dev, nvdev->num_chn);
- netvsc_open(net);
- net_device_ctx->start_remove = false;
+ if (was_running)
+ ret = netvsc_open(net);
/* We may have missed link change notifications */
schedule_delayed_work(&net_device_ctx->dwork, 0);
@@ -787,18 +839,19 @@
return ret;
}
-static bool netvsc_validate_ethtool_ss_cmd(const struct ethtool_cmd *cmd)
+static bool
+netvsc_validate_ethtool_ss_cmd(const struct ethtool_link_ksettings *cmd)
{
- struct ethtool_cmd diff1 = *cmd;
- struct ethtool_cmd diff2 = {};
+ struct ethtool_link_ksettings diff1 = *cmd;
+ struct ethtool_link_ksettings diff2 = {};
- ethtool_cmd_speed_set(&diff1, 0);
- diff1.duplex = 0;
+ diff1.base.speed = 0;
+ diff1.base.duplex = 0;
/* advertising and cmd are usually set */
- diff1.advertising = 0;
- diff1.cmd = 0;
+ ethtool_link_ksettings_zero_link_mode(&diff1, advertising);
+ diff1.base.cmd = 0;
/* We set port to PORT_OTHER */
- diff2.port = PORT_OTHER;
+ diff2.base.port = PORT_OTHER;
return !memcmp(&diff1, &diff2, sizeof(diff1));
}
@@ -808,33 +861,35 @@
struct net_device_context *ndc = netdev_priv(dev);
ndc->speed = SPEED_UNKNOWN;
- ndc->duplex = DUPLEX_UNKNOWN;
+ ndc->duplex = DUPLEX_FULL;
}
-static int netvsc_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int netvsc_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct net_device_context *ndc = netdev_priv(dev);
- ethtool_cmd_speed_set(cmd, ndc->speed);
- cmd->duplex = ndc->duplex;
- cmd->port = PORT_OTHER;
+ cmd->base.speed = ndc->speed;
+ cmd->base.duplex = ndc->duplex;
+ cmd->base.port = PORT_OTHER;
return 0;
}
-static int netvsc_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int netvsc_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct net_device_context *ndc = netdev_priv(dev);
u32 speed;
- speed = ethtool_cmd_speed(cmd);
+ speed = cmd->base.speed;
if (!ethtool_validate_speed(speed) ||
- !ethtool_validate_duplex(cmd->duplex) ||
+ !ethtool_validate_duplex(cmd->base.duplex) ||
!netvsc_validate_ethtool_ss_cmd(cmd))
return -EINVAL;
ndc->speed = speed;
- ndc->duplex = cmd->duplex;
+ ndc->duplex = cmd->base.duplex;
return 0;
}
@@ -842,24 +897,27 @@
static int netvsc_change_mtu(struct net_device *ndev, int mtu)
{
struct net_device_context *ndevctx = netdev_priv(ndev);
- struct netvsc_device *nvdev = ndevctx->nvdev;
+ struct netvsc_device *nvdev = rtnl_dereference(ndevctx->nvdev);
struct hv_device *hdev = ndevctx->device_ctx;
struct netvsc_device_info device_info;
- int ret;
+ bool was_running;
+ int ret = 0;
- if (ndevctx->start_remove || !nvdev || nvdev->destroy)
+ if (!nvdev || nvdev->destroy)
return -ENODEV;
- ret = netvsc_close(ndev);
- if (ret)
- goto out;
+ was_running = netif_running(ndev);
+ if (was_running) {
+ ret = netvsc_close(ndev);
+ if (ret)
+ return ret;
+ }
memset(&device_info, 0, sizeof(device_info));
device_info.ring_size = ring_size;
device_info.num_chn = nvdev->num_chn;
device_info.max_num_vrss_chns = nvdev->num_chn;
- ndevctx->start_remove = true;
rndis_filter_device_remove(hdev, nvdev);
/* 'nvdev' has been freed in rndis_filter_device_remove() ->
@@ -872,9 +930,8 @@
rndis_filter_device_add(hdev, &device_info);
-out:
- netvsc_open(ndev);
- ndevctx->start_remove = false;
+ if (was_running)
+ ret = netvsc_open(ndev);
/* We may have missed link change notifications */
schedule_delayed_work(&ndevctx->dwork, 0);
@@ -886,7 +943,7 @@
struct rtnl_link_stats64 *t)
{
struct net_device_context *ndev_ctx = netdev_priv(net);
- struct netvsc_device *nvdev = ndev_ctx->nvdev;
+ struct netvsc_device *nvdev = rcu_dereference_rtnl(ndev_ctx->nvdev);
int i;
if (!nvdev)
@@ -971,7 +1028,10 @@
static int netvsc_get_sset_count(struct net_device *dev, int string_set)
{
struct net_device_context *ndc = netdev_priv(dev);
- struct netvsc_device *nvdev = ndc->nvdev;
+ struct netvsc_device *nvdev = rcu_dereference(ndc->nvdev);
+
+ if (!nvdev)
+ return -ENODEV;
switch (string_set) {
case ETH_SS_STATS:
@@ -985,13 +1045,16 @@
struct ethtool_stats *stats, u64 *data)
{
struct net_device_context *ndc = netdev_priv(dev);
- struct netvsc_device *nvdev = ndc->nvdev;
+ struct netvsc_device *nvdev = rcu_dereference(ndc->nvdev);
const void *nds = &ndc->eth_stats;
const struct netvsc_stats *qstats;
unsigned int start;
u64 packets, bytes;
int i, j;
+ if (!nvdev)
+ return;
+
for (i = 0; i < NETVSC_GLOBAL_STATS_LEN; i++)
data[i] = *(unsigned long *)(nds + netvsc_stats[i].offset);
@@ -1020,10 +1083,13 @@
static void netvsc_get_strings(struct net_device *dev, u32 stringset, u8 *data)
{
struct net_device_context *ndc = netdev_priv(dev);
- struct netvsc_device *nvdev = ndc->nvdev;
+ struct netvsc_device *nvdev = rcu_dereference(ndc->nvdev);
u8 *p = data;
int i;
+ if (!nvdev)
+ return;
+
switch (stringset) {
case ETH_SS_STATS:
for (i = 0; i < ARRAY_SIZE(netvsc_stats); i++)
@@ -1075,7 +1141,10 @@
u32 *rules)
{
struct net_device_context *ndc = netdev_priv(dev);
- struct netvsc_device *nvdev = ndc->nvdev;
+ struct netvsc_device *nvdev = rcu_dereference(ndc->nvdev);
+
+ if (!nvdev)
+ return -ENODEV;
switch (info->cmd) {
case ETHTOOL_GRXRINGS:
@@ -1111,13 +1180,17 @@
u8 *hfunc)
{
struct net_device_context *ndc = netdev_priv(dev);
- struct netvsc_device *ndev = ndc->nvdev;
- struct rndis_device *rndis_dev = ndev->extension;
+ struct netvsc_device *ndev = rcu_dereference(ndc->nvdev);
+ struct rndis_device *rndis_dev;
int i;
+ if (!ndev)
+ return -ENODEV;
+
if (hfunc)
*hfunc = ETH_RSS_HASH_TOP; /* Toeplitz */
+ rndis_dev = ndev->extension;
if (indir) {
for (i = 0; i < ITAB_NUM; i++)
indir[i] = rndis_dev->ind_table[i];
@@ -1133,13 +1206,17 @@
const u8 *key, const u8 hfunc)
{
struct net_device_context *ndc = netdev_priv(dev);
- struct netvsc_device *ndev = ndc->nvdev;
- struct rndis_device *rndis_dev = ndev->extension;
+ struct netvsc_device *ndev = rtnl_dereference(ndc->nvdev);
+ struct rndis_device *rndis_dev;
int i;
+ if (!ndev)
+ return -ENODEV;
+
if (hfunc != ETH_RSS_HASH_NO_CHANGE && hfunc != ETH_RSS_HASH_TOP)
return -EOPNOTSUPP;
+ rndis_dev = ndev->extension;
if (indir) {
for (i = 0; i < ITAB_NUM; i++)
if (indir[i] >= dev->num_rx_queues)
@@ -1168,13 +1245,13 @@
.get_channels = netvsc_get_channels,
.set_channels = netvsc_set_channels,
.get_ts_info = ethtool_op_get_ts_info,
- .get_settings = netvsc_get_settings,
- .set_settings = netvsc_set_settings,
.get_rxnfc = netvsc_get_rxnfc,
.get_rxfh_key_size = netvsc_get_rxfh_key_size,
.get_rxfh_indir_size = netvsc_rss_indir_size,
.get_rxfh = netvsc_get_rxfh,
.set_rxfh = netvsc_set_rxfh,
+ .get_link_ksettings = netvsc_get_link_ksettings,
+ .set_link_ksettings = netvsc_set_link_ksettings,
};
static const struct net_device_ops device_ops = {
@@ -1210,10 +1287,10 @@
unsigned long flags, next_reconfig, delay;
rtnl_lock();
- if (ndev_ctx->start_remove)
+ net_device = rtnl_dereference(ndev_ctx->nvdev);
+ if (!net_device)
goto out_unlock;
- net_device = ndev_ctx->nvdev;
rdev = net_device->extension;
next_reconfig = ndev_ctx->last_reconfig + LINKCHANGE_INT;
@@ -1354,7 +1431,7 @@
return NOTIFY_DONE;
net_device_ctx = netdev_priv(ndev);
- netvsc_dev = net_device_ctx->nvdev;
+ netvsc_dev = rtnl_dereference(net_device_ctx->nvdev);
if (!netvsc_dev || rtnl_dereference(net_device_ctx->vf_netdev))
return NOTIFY_DONE;
@@ -1380,7 +1457,7 @@
return NOTIFY_DONE;
net_device_ctx = netdev_priv(ndev);
- netvsc_dev = net_device_ctx->nvdev;
+ netvsc_dev = rtnl_dereference(net_device_ctx->nvdev);
netdev_info(ndev, "VF up: %s\n", vf_netdev->name);
@@ -1414,7 +1491,7 @@
return NOTIFY_DONE;
net_device_ctx = netdev_priv(ndev);
- netvsc_dev = net_device_ctx->nvdev;
+ netvsc_dev = rtnl_dereference(net_device_ctx->nvdev);
netdev_info(ndev, "VF down: %s\n", vf_netdev->name);
netvsc_switch_datapath(ndev, false);
@@ -1474,8 +1551,6 @@
hv_set_drvdata(dev, net);
- net_device_ctx->start_remove = false;
-
INIT_DELAYED_WORK(&net_device_ctx->dwork, netvsc_link_change);
INIT_WORK(&net_device_ctx->work, do_set_multicast);
@@ -1492,8 +1567,7 @@
/* Notify the netvsc driver of the new device */
memset(&device_info, 0, sizeof(device_info));
device_info.ring_size = ring_size;
- device_info.max_num_vrss_chns = min_t(u32, VRSS_CHANNEL_DEFAULT,
- num_online_cpus());
+ device_info.num_chn = VRSS_CHANNEL_DEFAULT;
ret = rndis_filter_device_add(dev, &device_info);
if (ret != 0) {
netdev_err(net, "unable to add netvsc device (ret %d)\n", ret);
@@ -1509,6 +1583,7 @@
NETIF_F_HW_VLAN_CTAG_TX | NETIF_F_HW_VLAN_CTAG_RX;
net->vlan_features = net->features;
+ /* RCU not necessary here, device not registered */
nvdev = net_device_ctx->nvdev;
netif_set_real_num_tx_queues(net, nvdev->num_chn);
netif_set_real_num_rx_queues(net, nvdev->num_chn);
@@ -1544,26 +1619,20 @@
ndev_ctx = netdev_priv(net);
- /* Avoid racing with netvsc_change_mtu()/netvsc_set_channels()
- * removing the device.
- */
- rtnl_lock();
- ndev_ctx->start_remove = true;
- rtnl_unlock();
+ netif_device_detach(net);
cancel_delayed_work_sync(&ndev_ctx->dwork);
cancel_work_sync(&ndev_ctx->work);
- /* Stop outbound asap */
- netif_tx_disable(net);
-
- unregister_netdev(net);
-
/*
* Call to the vsc driver to let it know that the device is being
- * removed
+ * removed. Also blocks mtu and channel changes.
*/
+ rtnl_lock();
rndis_filter_device_remove(dev, ndev_ctx->nvdev);
+ rtnl_unlock();
+
+ unregister_netdev(net);
hv_set_drvdata(dev, NULL);
diff --git a/drivers/net/hyperv/rndis_filter.c b/drivers/net/hyperv/rndis_filter.c
index 19356f5..ab92c3c 100644
--- a/drivers/net/hyperv/rndis_filter.c
+++ b/drivers/net/hyperv/rndis_filter.c
@@ -819,16 +819,14 @@
{
struct rndis_request *request;
struct rndis_set_request *set;
- struct rndis_set_complete *set_complete;
int ret;
request = get_rndis_request(dev, RNDIS_MSG_SET,
RNDIS_MESSAGE_SIZE(struct rndis_set_request) +
sizeof(u32));
- if (!request) {
- ret = -ENOMEM;
- goto cleanup;
- }
+ if (!request)
+ return -ENOMEM;
+
/* Setup the rndis set */
set = &request->request_msg.msg.set_req;
@@ -840,15 +838,11 @@
&new_filter, sizeof(u32));
ret = rndis_filter_send_request(dev, request);
- if (ret != 0)
- goto cleanup;
+ if (ret == 0)
+ wait_for_completion(&request->wait_event);
- wait_for_completion(&request->wait_event);
+ put_rndis_request(dev, request);
- set_complete = &request->response_msg.msg.set_complete;
-cleanup:
- if (request)
- put_rndis_request(dev, request);
return ret;
}
@@ -926,8 +920,6 @@
struct rndis_halt_request *halt;
struct net_device_context *net_device_ctx = netdev_priv(dev->ndev);
struct netvsc_device *nvdev = net_device_ctx->nvdev;
- struct hv_device *hdev = net_device_ctx->device_ctx;
- ulong flags;
/* Attempt to do a rndis device halt */
request = get_rndis_request(dev, RNDIS_MSG_HALT,
@@ -945,9 +937,10 @@
dev->state = RNDIS_DEV_UNINITIALIZED;
cleanup:
- spin_lock_irqsave(&hdev->channel->inbound_lock, flags);
nvdev->destroy = true;
- spin_unlock_irqrestore(&hdev->channel->inbound_lock, flags);
+
+ /* Force flag to be ordered before waiting */
+ wmb();
/* Wait for all send completions */
wait_event(nvdev->wait_drain, netvsc_device_idle(nvdev));
@@ -996,26 +989,38 @@
hv_get_drvdata(new_sc->primary_channel->device_obj);
struct netvsc_device *nvscdev = net_device_to_netvsc_device(ndev);
u16 chn_index = new_sc->offermsg.offer.sub_channel_index;
+ struct netvsc_channel *nvchan;
int ret;
- unsigned long flags;
if (chn_index >= nvscdev->num_chn)
return;
- nvscdev->chan_table[chn_index].mrc.buf
+ nvchan = nvscdev->chan_table + chn_index;
+ nvchan->mrc.buf
= vzalloc(NETVSC_RECVSLOT_MAX * sizeof(struct recv_comp_data));
+ if (!nvchan->mrc.buf)
+ return;
+
+ /* Because the device uses NAPI, all the interrupt batching and
+ * control is done via Net softirq, not the channel handling
+ */
+ set_channel_read_mode(new_sc, HV_CALL_ISR);
+
+ /* Set the channel before opening.*/
+ nvchan->channel = new_sc;
+ netif_napi_add(ndev, &nvchan->napi,
+ netvsc_poll, NAPI_POLL_WEIGHT);
+
ret = vmbus_open(new_sc, nvscdev->ring_size * PAGE_SIZE,
nvscdev->ring_size * PAGE_SIZE, NULL, 0,
- netvsc_channel_cb, new_sc);
-
+ netvsc_channel_cb, nvchan);
if (ret == 0)
- nvscdev->chan_table[chn_index].channel = new_sc;
+ napi_enable(&nvchan->napi);
+ else
+ netdev_err(ndev, "sub channel open failed (%d)\n", ret);
- spin_lock_irqsave(&nvscdev->sc_lock, flags);
- nvscdev->num_sc_offered--;
- spin_unlock_irqrestore(&nvscdev->sc_lock, flags);
- if (nvscdev->num_sc_offered == 0)
+ if (refcount_dec_and_test(&nvscdev->sc_offered))
complete(&nvscdev->channel_init_wait);
}
@@ -1032,12 +1037,9 @@
struct ndis_recv_scale_cap rsscap;
u32 rsscap_size = sizeof(struct ndis_recv_scale_cap);
unsigned int gso_max_size = GSO_MAX_SIZE;
- u32 mtu, size;
- u32 num_rss_qs;
- u32 sc_delta;
+ u32 mtu, size, num_rss_qs;
const struct cpumask *node_cpu_mask;
u32 num_possible_rss_qs;
- unsigned long flags;
int i, ret;
rndis_device = get_rndis_device();
@@ -1060,7 +1062,7 @@
net_device->max_chn = 1;
net_device->num_chn = 1;
- spin_lock_init(&net_device->sc_lock);
+ refcount_set(&net_device->sc_offered, 0);
net_device->extension = rndis_device;
rndis_device->ndev = net;
@@ -1174,34 +1176,30 @@
if (ret || rsscap.num_recv_que < 2)
goto out;
- net_device->max_chn = min_t(u32, VRSS_CHANNEL_MAX, rsscap.num_recv_que);
-
- num_rss_qs = min(device_info->max_num_vrss_chns, net_device->max_chn);
-
/*
* We will limit the VRSS channels to the number CPUs in the NUMA node
* the primary channel is currently bound to.
+ *
+ * This also guarantees that num_possible_rss_qs <= num_online_cpus
*/
node_cpu_mask = cpumask_of_node(cpu_to_node(dev->channel->target_cpu));
- num_possible_rss_qs = cpumask_weight(node_cpu_mask);
+ num_possible_rss_qs = min_t(u32, cpumask_weight(node_cpu_mask),
+ rsscap.num_recv_que);
+
+ net_device->max_chn = min_t(u32, VRSS_CHANNEL_MAX, num_possible_rss_qs);
/* We will use the given number of channels if available. */
- if (device_info->num_chn && device_info->num_chn < net_device->max_chn)
- net_device->num_chn = device_info->num_chn;
- else
- net_device->num_chn = min(num_possible_rss_qs, num_rss_qs);
-
- num_rss_qs = net_device->num_chn - 1;
+ net_device->num_chn = min(net_device->max_chn, device_info->num_chn);
for (i = 0; i < ITAB_NUM; i++)
rndis_device->ind_table[i] = ethtool_rxfh_indir_default(i,
net_device->num_chn);
- net_device->num_sc_offered = num_rss_qs;
+ num_rss_qs = net_device->num_chn - 1;
+ if (num_rss_qs == 0)
+ return 0;
- if (net_device->num_chn == 1)
- goto out;
-
+ refcount_set(&net_device->sc_offered, num_rss_qs);
vmbus_set_sc_create_callback(dev->channel, netvsc_sc_open);
init_packet = &net_device->channel_init_pkt;
@@ -1217,32 +1215,23 @@
VMBUS_DATA_PACKET_FLAG_COMPLETION_REQUESTED);
if (ret)
goto out;
- wait_for_completion(&net_device->channel_init_wait);
- if (init_packet->msg.v5_msg.subchn_comp.status !=
- NVSP_STAT_SUCCESS) {
+ if (init_packet->msg.v5_msg.subchn_comp.status != NVSP_STAT_SUCCESS) {
ret = -ENODEV;
goto out;
}
+ wait_for_completion(&net_device->channel_init_wait);
+
net_device->num_chn = 1 +
init_packet->msg.v5_msg.subchn_comp.num_subchannels;
- ret = rndis_filter_set_rss_param(rndis_device, netvsc_hash_key,
- net_device->num_chn);
-
- /*
- * Set the number of sub-channels to be received.
- */
- spin_lock_irqsave(&net_device->sc_lock, flags);
- sc_delta = num_rss_qs - (net_device->num_chn - 1);
- net_device->num_sc_offered -= sc_delta;
- spin_unlock_irqrestore(&net_device->sc_lock, flags);
-
+ /* ignore failues from setting rss parameters, still have channels */
+ rndis_filter_set_rss_param(rndis_device, netvsc_hash_key,
+ net_device->num_chn);
out:
if (ret) {
net_device->max_chn = 1;
net_device->num_chn = 1;
- net_device->num_sc_offered = 0;
}
return 0; /* return 0 because primary channel can be used alone */
@@ -1257,12 +1246,6 @@
{
struct rndis_device *rndis_dev = net_dev->extension;
- /* If not all subchannel offers are complete, wait for them until
- * completion to avoid race.
- */
- if (net_dev->num_sc_offered > 0)
- wait_for_completion(&net_dev->channel_init_wait);
-
/* Halt and release the rndis device */
rndis_filter_halt_device(rndis_dev);
diff --git a/drivers/net/ieee802154/Kconfig b/drivers/net/ieee802154/Kconfig
index 3057a8d..303ba41 100644
--- a/drivers/net/ieee802154/Kconfig
+++ b/drivers/net/ieee802154/Kconfig
@@ -82,3 +82,25 @@
This driver can also be built as a module. To do so, say M here.
the module will be called 'adf7242'.
+
+config IEEE802154_CA8210
+ tristate "Cascoda CA8210 transceiver driver"
+ depends on IEEE802154_DRIVERS && MAC802154
+ depends on COMMON_CLK
+ depends on SPI
+ ---help---
+ Say Y here to enable the CA8210 SPI 802.15.4 wireless
+ controller.
+
+ This driver can also be built as a module. To do so, say M here.
+ the module will be called 'ca8210'.
+
+config IEEE802154_CA8210_DEBUGFS
+ bool "CA8210 debugfs interface"
+ depends on IEEE802154_CA8210
+ depends on DEBUG_FS
+ ---help---
+ This option compiles debugfs code for the ca8210 driver. This
+ exposes a debugfs node for each CA8210 instance which allows
+ direct use of the Cascoda API, exposing the 802.15.4 MAC
+ management entities.
diff --git a/drivers/net/ieee802154/Makefile b/drivers/net/ieee802154/Makefile
index 3a923d3..8374bb4 100644
--- a/drivers/net/ieee802154/Makefile
+++ b/drivers/net/ieee802154/Makefile
@@ -4,3 +4,4 @@
obj-$(CONFIG_IEEE802154_CC2520) += cc2520.o
obj-$(CONFIG_IEEE802154_ATUSB) += atusb.o
obj-$(CONFIG_IEEE802154_ADF7242) += adf7242.o
+obj-$(CONFIG_IEEE802154_CA8210) += ca8210.o
diff --git a/drivers/net/ieee802154/ca8210.c b/drivers/net/ieee802154/ca8210.c
new file mode 100644
index 0000000..25fd3b0
--- /dev/null
+++ b/drivers/net/ieee802154/ca8210.c
@@ -0,0 +1,3242 @@
+/*
+ * http://www.cascoda.com/products/ca-821x/
+ * Copyright (c) 2016, Cascoda, Ltd.
+ * All rights reserved.
+ *
+ * This code is dual-licensed under both GPLv2 and 3-clause BSD. What follows is
+ * the license notice for both respectively.
+ *
+ *******************************************************************************
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ *******************************************************************************
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright notice,
+ * this list of conditions and the following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above copyright notice,
+ * this list of conditions and the following disclaimer in the documentation
+ * and/or other materials provided with the distribution.
+ *
+ * 3. Neither the name of the copyright holder nor the names of its contributors
+ * may be used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <linux/cdev.h>
+#include <linux/clk-provider.h>
+#include <linux/debugfs.h>
+#include <linux/delay.h>
+#include <linux/gpio.h>
+#include <linux/ieee802154.h>
+#include <linux/kfifo.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/of_gpio.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/poll.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <linux/spi/spi.h>
+#include <linux/spinlock.h>
+#include <linux/string.h>
+#include <linux/workqueue.h>
+
+#include <net/ieee802154_netdev.h>
+#include <net/mac802154.h>
+
+#define DRIVER_NAME "ca8210"
+
+/* external clock frequencies */
+#define ONE_MHZ 1000000
+#define TWO_MHZ (2 * ONE_MHZ)
+#define FOUR_MHZ (4 * ONE_MHZ)
+#define EIGHT_MHZ (8 * ONE_MHZ)
+#define SIXTEEN_MHZ (16 * ONE_MHZ)
+
+/* spi constants */
+#define CA8210_SPI_BUF_SIZE 256
+#define CA8210_SYNC_TIMEOUT 1000 /* Timeout for synchronous commands [ms] */
+
+/* test interface constants */
+#define CA8210_TEST_INT_FILE_NAME "ca8210_test"
+#define CA8210_TEST_INT_FIFO_SIZE 256
+
+/* MAC status enumerations */
+#define MAC_SUCCESS (0x00)
+#define MAC_ERROR (0x01)
+#define MAC_CANCELLED (0x02)
+#define MAC_READY_FOR_POLL (0x03)
+#define MAC_COUNTER_ERROR (0xDB)
+#define MAC_IMPROPER_KEY_TYPE (0xDC)
+#define MAC_IMPROPER_SECURITY_LEVEL (0xDD)
+#define MAC_UNSUPPORTED_LEGACY (0xDE)
+#define MAC_UNSUPPORTED_SECURITY (0xDF)
+#define MAC_BEACON_LOST (0xE0)
+#define MAC_CHANNEL_ACCESS_FAILURE (0xE1)
+#define MAC_DENIED (0xE2)
+#define MAC_DISABLE_TRX_FAILURE (0xE3)
+#define MAC_SECURITY_ERROR (0xE4)
+#define MAC_FRAME_TOO_LONG (0xE5)
+#define MAC_INVALID_GTS (0xE6)
+#define MAC_INVALID_HANDLE (0xE7)
+#define MAC_INVALID_PARAMETER (0xE8)
+#define MAC_NO_ACK (0xE9)
+#define MAC_NO_BEACON (0xEA)
+#define MAC_NO_DATA (0xEB)
+#define MAC_NO_SHORT_ADDRESS (0xEC)
+#define MAC_OUT_OF_CAP (0xED)
+#define MAC_PAN_ID_CONFLICT (0xEE)
+#define MAC_REALIGNMENT (0xEF)
+#define MAC_TRANSACTION_EXPIRED (0xF0)
+#define MAC_TRANSACTION_OVERFLOW (0xF1)
+#define MAC_TX_ACTIVE (0xF2)
+#define MAC_UNAVAILABLE_KEY (0xF3)
+#define MAC_UNSUPPORTED_ATTRIBUTE (0xF4)
+#define MAC_INVALID_ADDRESS (0xF5)
+#define MAC_ON_TIME_TOO_LONG (0xF6)
+#define MAC_PAST_TIME (0xF7)
+#define MAC_TRACKING_OFF (0xF8)
+#define MAC_INVALID_INDEX (0xF9)
+#define MAC_LIMIT_REACHED (0xFA)
+#define MAC_READ_ONLY (0xFB)
+#define MAC_SCAN_IN_PROGRESS (0xFC)
+#define MAC_SUPERFRAME_OVERLAP (0xFD)
+#define MAC_SYSTEM_ERROR (0xFF)
+
+/* HWME attribute IDs */
+#define HWME_EDTHRESHOLD (0x04)
+#define HWME_EDVALUE (0x06)
+#define HWME_SYSCLKOUT (0x0F)
+#define HWME_LQILIMIT (0x11)
+
+/* TDME attribute IDs */
+#define TDME_CHANNEL (0x00)
+#define TDME_ATM_CONFIG (0x06)
+
+#define MAX_HWME_ATTRIBUTE_SIZE 16
+#define MAX_TDME_ATTRIBUTE_SIZE 2
+
+/* PHY/MAC PIB Attribute Enumerations */
+#define PHY_CURRENT_CHANNEL (0x00)
+#define PHY_TRANSMIT_POWER (0x02)
+#define PHY_CCA_MODE (0x03)
+#define MAC_ASSOCIATION_PERMIT (0x41)
+#define MAC_AUTO_REQUEST (0x42)
+#define MAC_BATT_LIFE_EXT (0x43)
+#define MAC_BATT_LIFE_EXT_PERIODS (0x44)
+#define MAC_BEACON_PAYLOAD (0x45)
+#define MAC_BEACON_PAYLOAD_LENGTH (0x46)
+#define MAC_BEACON_ORDER (0x47)
+#define MAC_GTS_PERMIT (0x4d)
+#define MAC_MAX_CSMA_BACKOFFS (0x4e)
+#define MAC_MIN_BE (0x4f)
+#define MAC_PAN_ID (0x50)
+#define MAC_PROMISCUOUS_MODE (0x51)
+#define MAC_RX_ON_WHEN_IDLE (0x52)
+#define MAC_SHORT_ADDRESS (0x53)
+#define MAC_SUPERFRAME_ORDER (0x54)
+#define MAC_ASSOCIATED_PAN_COORD (0x56)
+#define MAC_MAX_BE (0x57)
+#define MAC_MAX_FRAME_RETRIES (0x59)
+#define MAC_RESPONSE_WAIT_TIME (0x5A)
+#define MAC_SECURITY_ENABLED (0x5D)
+
+#define MAC_AUTO_REQUEST_SECURITY_LEVEL (0x78)
+#define MAC_AUTO_REQUEST_KEY_ID_MODE (0x79)
+
+#define NS_IEEE_ADDRESS (0xFF) /* Non-standard IEEE address */
+
+/* MAC Address Mode Definitions */
+#define MAC_MODE_NO_ADDR (0x00)
+#define MAC_MODE_SHORT_ADDR (0x02)
+#define MAC_MODE_LONG_ADDR (0x03)
+
+/* MAC constants */
+#define MAX_BEACON_OVERHEAD (75)
+#define MAX_BEACON_PAYLOAD_LENGTH (IEEE802154_MTU - MAX_BEACON_OVERHEAD)
+
+#define MAX_ATTRIBUTE_SIZE (122)
+#define MAX_DATA_SIZE (114)
+
+#define CA8210_VALID_CHANNELS (0x07FFF800)
+
+/* MAC workarounds for V1.1 and MPW silicon (V0.x) */
+#define CA8210_MAC_WORKAROUNDS (0)
+#define CA8210_MAC_MPW (0)
+
+/* memory manipulation macros */
+#define LS_BYTE(x) ((u8)((x) & 0xFF))
+#define MS_BYTE(x) ((u8)(((x) >> 8) & 0xFF))
+
+/* message ID codes in SPI commands */
+/* downstream */
+#define MCPS_DATA_REQUEST (0x00)
+#define MLME_ASSOCIATE_REQUEST (0x02)
+#define MLME_ASSOCIATE_RESPONSE (0x03)
+#define MLME_DISASSOCIATE_REQUEST (0x04)
+#define MLME_GET_REQUEST (0x05)
+#define MLME_ORPHAN_RESPONSE (0x06)
+#define MLME_RESET_REQUEST (0x07)
+#define MLME_RX_ENABLE_REQUEST (0x08)
+#define MLME_SCAN_REQUEST (0x09)
+#define MLME_SET_REQUEST (0x0A)
+#define MLME_START_REQUEST (0x0B)
+#define MLME_POLL_REQUEST (0x0D)
+#define HWME_SET_REQUEST (0x0E)
+#define HWME_GET_REQUEST (0x0F)
+#define TDME_SETSFR_REQUEST (0x11)
+#define TDME_GETSFR_REQUEST (0x12)
+#define TDME_SET_REQUEST (0x14)
+/* upstream */
+#define MCPS_DATA_INDICATION (0x00)
+#define MCPS_DATA_CONFIRM (0x01)
+#define MLME_RESET_CONFIRM (0x0A)
+#define MLME_SET_CONFIRM (0x0E)
+#define MLME_START_CONFIRM (0x0F)
+#define HWME_SET_CONFIRM (0x12)
+#define HWME_GET_CONFIRM (0x13)
+#define HWME_WAKEUP_INDICATION (0x15)
+#define TDME_SETSFR_CONFIRM (0x17)
+
+/* SPI command IDs */
+/* bit indicating a confirm or indication from slave to master */
+#define SPI_S2M (0x20)
+/* bit indicating a synchronous message */
+#define SPI_SYN (0x40)
+
+/* SPI command definitions */
+#define SPI_IDLE (0xFF)
+#define SPI_NACK (0xF0)
+
+#define SPI_MCPS_DATA_REQUEST (MCPS_DATA_REQUEST)
+#define SPI_MCPS_DATA_INDICATION (MCPS_DATA_INDICATION + SPI_S2M)
+#define SPI_MCPS_DATA_CONFIRM (MCPS_DATA_CONFIRM + SPI_S2M)
+
+#define SPI_MLME_ASSOCIATE_REQUEST (MLME_ASSOCIATE_REQUEST)
+#define SPI_MLME_RESET_REQUEST (MLME_RESET_REQUEST + SPI_SYN)
+#define SPI_MLME_SET_REQUEST (MLME_SET_REQUEST + SPI_SYN)
+#define SPI_MLME_START_REQUEST (MLME_START_REQUEST + SPI_SYN)
+#define SPI_MLME_RESET_CONFIRM (MLME_RESET_CONFIRM + SPI_S2M + SPI_SYN)
+#define SPI_MLME_SET_CONFIRM (MLME_SET_CONFIRM + SPI_S2M + SPI_SYN)
+#define SPI_MLME_START_CONFIRM (MLME_START_CONFIRM + SPI_S2M + SPI_SYN)
+
+#define SPI_HWME_SET_REQUEST (HWME_SET_REQUEST + SPI_SYN)
+#define SPI_HWME_GET_REQUEST (HWME_GET_REQUEST + SPI_SYN)
+#define SPI_HWME_SET_CONFIRM (HWME_SET_CONFIRM + SPI_S2M + SPI_SYN)
+#define SPI_HWME_GET_CONFIRM (HWME_GET_CONFIRM + SPI_S2M + SPI_SYN)
+#define SPI_HWME_WAKEUP_INDICATION (HWME_WAKEUP_INDICATION + SPI_S2M)
+
+#define SPI_TDME_SETSFR_REQUEST (TDME_SETSFR_REQUEST + SPI_SYN)
+#define SPI_TDME_SET_REQUEST (TDME_SET_REQUEST + SPI_SYN)
+#define SPI_TDME_SETSFR_CONFIRM (TDME_SETSFR_CONFIRM + SPI_S2M + SPI_SYN)
+
+/* TDME SFR addresses */
+/* Page 0 */
+#define CA8210_SFR_PACFG (0xB1)
+#define CA8210_SFR_MACCON (0xD8)
+#define CA8210_SFR_PACFGIB (0xFE)
+/* Page 1 */
+#define CA8210_SFR_LOTXCAL (0xBF)
+#define CA8210_SFR_PTHRH (0xD1)
+#define CA8210_SFR_PRECFG (0xD3)
+#define CA8210_SFR_LNAGX40 (0xE1)
+#define CA8210_SFR_LNAGX41 (0xE2)
+#define CA8210_SFR_LNAGX42 (0xE3)
+#define CA8210_SFR_LNAGX43 (0xE4)
+#define CA8210_SFR_LNAGX44 (0xE5)
+#define CA8210_SFR_LNAGX45 (0xE6)
+#define CA8210_SFR_LNAGX46 (0xE7)
+#define CA8210_SFR_LNAGX47 (0xE9)
+
+#define PACFGIB_DEFAULT_CURRENT (0x3F)
+#define PTHRH_DEFAULT_THRESHOLD (0x5A)
+#define LNAGX40_DEFAULT_GAIN (0x29) /* 10dB */
+#define LNAGX41_DEFAULT_GAIN (0x54) /* 21dB */
+#define LNAGX42_DEFAULT_GAIN (0x6C) /* 27dB */
+#define LNAGX43_DEFAULT_GAIN (0x7A) /* 30dB */
+#define LNAGX44_DEFAULT_GAIN (0x84) /* 33dB */
+#define LNAGX45_DEFAULT_GAIN (0x8B) /* 34dB */
+#define LNAGX46_DEFAULT_GAIN (0x92) /* 36dB */
+#define LNAGX47_DEFAULT_GAIN (0x96) /* 37dB */
+
+#define CA8210_IOCTL_HARD_RESET (0x00)
+
+/* Structs/Enums */
+
+/**
+ * struct cas_control - spi transfer structure
+ * @msg: spi_message for each exchange
+ * @transfer: spi_transfer for each exchange
+ * @tx_buf: source array for transmission
+ * @tx_in_buf: array storing bytes received during transmission
+ * @priv: pointer to private data
+ *
+ * This structure stores all the necessary data passed around during a single
+ * spi exchange.
+ */
+struct cas_control {
+ struct spi_message msg;
+ struct spi_transfer transfer;
+
+ u8 tx_buf[CA8210_SPI_BUF_SIZE];
+ u8 tx_in_buf[CA8210_SPI_BUF_SIZE];
+
+ struct ca8210_priv *priv;
+};
+
+/**
+ * struct ca8210_test - ca8210 test interface structure
+ * @ca8210_dfs_spi_int: pointer to the entry in the debug fs for this device
+ * @up_fifo: fifo for upstream messages
+ *
+ * This structure stores all the data pertaining to the debug interface
+ */
+struct ca8210_test {
+ struct dentry *ca8210_dfs_spi_int;
+ struct kfifo up_fifo;
+ wait_queue_head_t readq;
+};
+
+/**
+ * struct ca8210_priv - ca8210 private data structure
+ * @spi: pointer to the ca8210 spi device object
+ * @hw: pointer to the ca8210 ieee802154_hw object
+ * @hw_registered: true if hw has been registered with ieee802154
+ * @lock: spinlock protecting the private data area
+ * @mlme_workqueue: workqueue for triggering MLME Reset
+ * @irq_workqueue: workqueue for irq processing
+ * @tx_skb: current socket buffer to transmit
+ * @nextmsduhandle: msdu handle to pass to the 15.4 MAC layer for the
+ * next transmission
+ * @clk: external clock provided by the ca8210
+ * @last_dsn: sequence number of last data packet received, for
+ * resend detection
+ * @test: test interface data section for this instance
+ * @async_tx_pending: true if an asynchronous transmission was started and
+ * is not complete
+ * @sync_command_response: pointer to buffer to fill with sync response
+ * @ca8210_is_awake: nonzero if ca8210 is initialised, ready for comms
+ * @sync_down: counts number of downstream synchronous commands
+ * @sync_up: counts number of upstream synchronous commands
+ * @spi_transfer_complete completion object for a single spi_transfer
+ * @sync_exchange_complete completion object for a complete synchronous API
+ * exchange
+ * @promiscuous whether the ca8210 is in promiscuous mode or not
+ * @retries: records how many times the current pending spi
+ * transfer has been retried
+ */
+struct ca8210_priv {
+ struct spi_device *spi;
+ struct ieee802154_hw *hw;
+ bool hw_registered;
+ spinlock_t lock;
+ struct workqueue_struct *mlme_workqueue;
+ struct workqueue_struct *irq_workqueue;
+ struct sk_buff *tx_skb;
+ u8 nextmsduhandle;
+ struct clk *clk;
+ int last_dsn;
+ struct ca8210_test test;
+ bool async_tx_pending;
+ u8 *sync_command_response;
+ struct completion ca8210_is_awake;
+ int sync_down, sync_up;
+ struct completion spi_transfer_complete, sync_exchange_complete;
+ bool promiscuous;
+ int retries;
+};
+
+/**
+ * struct work_priv_container - link between a work object and the relevant
+ * device's private data
+ * @work: work object being executed
+ * @priv: device's private data section
+ *
+ */
+struct work_priv_container {
+ struct work_struct work;
+ struct ca8210_priv *priv;
+};
+
+/**
+ * struct ca8210_platform_data - ca8210 platform data structure
+ * @extclockenable: true if the external clock is to be enabled
+ * @extclockfreq: frequency of the external clock
+ * @extclockgpio: ca8210 output gpio of the external clock
+ * @gpio_reset: gpio number of ca8210 reset line
+ * @gpio_irq: gpio number of ca8210 interrupt line
+ * @irq_id: identifier for the ca8210 irq
+ *
+ */
+struct ca8210_platform_data {
+ bool extclockenable;
+ unsigned int extclockfreq;
+ unsigned int extclockgpio;
+ int gpio_reset;
+ int gpio_irq;
+ int irq_id;
+};
+
+/**
+ * struct fulladdr - full MAC addressing information structure
+ * @mode: address mode (none, short, extended)
+ * @pan_id: 16-bit LE pan id
+ * @address: LE address, variable length as specified by mode
+ *
+ */
+struct fulladdr {
+ u8 mode;
+ u8 pan_id[2];
+ u8 address[8];
+};
+
+/**
+ * union macaddr: generic MAC address container
+ * @short_addr: 16-bit short address
+ * @ieee_address: 64-bit extended address as LE byte array
+ *
+ */
+union macaddr {
+ u16 short_address;
+ u8 ieee_address[8];
+};
+
+/**
+ * struct secspec: security specification for SAP commands
+ * @security_level: 0-7, controls level of authentication & encryption
+ * @key_id_mode: 0-3, specifies how to obtain key
+ * @key_source: extended key retrieval data
+ * @key_index: single-byte key identifier
+ *
+ */
+struct secspec {
+ u8 security_level;
+ u8 key_id_mode;
+ u8 key_source[8];
+ u8 key_index;
+};
+
+/* downlink functions parameter set definitions */
+struct mcps_data_request_pset {
+ u8 src_addr_mode;
+ struct fulladdr dst;
+ u8 msdu_length;
+ u8 msdu_handle;
+ u8 tx_options;
+ u8 msdu[MAX_DATA_SIZE];
+};
+
+struct mlme_set_request_pset {
+ u8 pib_attribute;
+ u8 pib_attribute_index;
+ u8 pib_attribute_length;
+ u8 pib_attribute_value[MAX_ATTRIBUTE_SIZE];
+};
+
+struct hwme_set_request_pset {
+ u8 hw_attribute;
+ u8 hw_attribute_length;
+ u8 hw_attribute_value[MAX_HWME_ATTRIBUTE_SIZE];
+};
+
+struct hwme_get_request_pset {
+ u8 hw_attribute;
+};
+
+struct tdme_setsfr_request_pset {
+ u8 sfr_page;
+ u8 sfr_address;
+ u8 sfr_value;
+};
+
+/* uplink functions parameter set definitions */
+struct hwme_set_confirm_pset {
+ u8 status;
+ u8 hw_attribute;
+};
+
+struct hwme_get_confirm_pset {
+ u8 status;
+ u8 hw_attribute;
+ u8 hw_attribute_length;
+ u8 hw_attribute_value[MAX_HWME_ATTRIBUTE_SIZE];
+};
+
+struct tdme_setsfr_confirm_pset {
+ u8 status;
+ u8 sfr_page;
+ u8 sfr_address;
+};
+
+struct mac_message {
+ u8 command_id;
+ u8 length;
+ union {
+ struct mcps_data_request_pset data_req;
+ struct mlme_set_request_pset set_req;
+ struct hwme_set_request_pset hwme_set_req;
+ struct hwme_get_request_pset hwme_get_req;
+ struct tdme_setsfr_request_pset tdme_set_sfr_req;
+ struct hwme_set_confirm_pset hwme_set_cnf;
+ struct hwme_get_confirm_pset hwme_get_cnf;
+ struct tdme_setsfr_confirm_pset tdme_set_sfr_cnf;
+ u8 u8param;
+ u8 status;
+ u8 payload[148];
+ } pdata;
+};
+
+union pa_cfg_sfr {
+ struct {
+ u8 bias_current_trim : 3;
+ u8 /* reserved */ : 1;
+ u8 buffer_capacitor_trim : 3;
+ u8 boost : 1;
+ };
+ u8 paib;
+};
+
+struct preamble_cfg_sfr {
+ u8 timeout_symbols : 3;
+ u8 acquisition_symbols : 3;
+ u8 search_symbols : 2;
+};
+
+static int (*cascoda_api_upstream)(
+ const u8 *buf,
+ size_t len,
+ void *device_ref
+);
+
+/**
+ * link_to_linux_err() - Translates an 802.15.4 return code into the closest
+ * linux error
+ * @link_status: 802.15.4 status code
+ *
+ * Return: 0 or Linux error code
+ */
+static int link_to_linux_err(int link_status)
+{
+ if (link_status < 0) {
+ /* status is already a Linux code */
+ return link_status;
+ }
+ switch (link_status) {
+ case MAC_SUCCESS:
+ case MAC_REALIGNMENT:
+ return 0;
+ case MAC_IMPROPER_KEY_TYPE:
+ return -EKEYREJECTED;
+ case MAC_IMPROPER_SECURITY_LEVEL:
+ case MAC_UNSUPPORTED_LEGACY:
+ case MAC_DENIED:
+ return -EACCES;
+ case MAC_BEACON_LOST:
+ case MAC_NO_ACK:
+ case MAC_NO_BEACON:
+ return -ENETUNREACH;
+ case MAC_CHANNEL_ACCESS_FAILURE:
+ case MAC_TX_ACTIVE:
+ case MAC_SCAN_IN_PROGRESS:
+ return -EBUSY;
+ case MAC_DISABLE_TRX_FAILURE:
+ case MAC_OUT_OF_CAP:
+ return -EAGAIN;
+ case MAC_FRAME_TOO_LONG:
+ return -EMSGSIZE;
+ case MAC_INVALID_GTS:
+ case MAC_PAST_TIME:
+ return -EBADSLT;
+ case MAC_INVALID_HANDLE:
+ return -EBADMSG;
+ case MAC_INVALID_PARAMETER:
+ case MAC_UNSUPPORTED_ATTRIBUTE:
+ case MAC_ON_TIME_TOO_LONG:
+ case MAC_INVALID_INDEX:
+ return -EINVAL;
+ case MAC_NO_DATA:
+ return -ENODATA;
+ case MAC_NO_SHORT_ADDRESS:
+ return -EFAULT;
+ case MAC_PAN_ID_CONFLICT:
+ return -EADDRINUSE;
+ case MAC_TRANSACTION_EXPIRED:
+ return -ETIME;
+ case MAC_TRANSACTION_OVERFLOW:
+ return -ENOBUFS;
+ case MAC_UNAVAILABLE_KEY:
+ return -ENOKEY;
+ case MAC_INVALID_ADDRESS:
+ return -ENXIO;
+ case MAC_TRACKING_OFF:
+ case MAC_SUPERFRAME_OVERLAP:
+ return -EREMOTEIO;
+ case MAC_LIMIT_REACHED:
+ return -EDQUOT;
+ case MAC_READ_ONLY:
+ return -EROFS;
+ default:
+ return -EPROTO;
+ }
+}
+
+/**
+ * ca8210_test_int_driver_write() - Writes a message to the test interface to be
+ * read by the userspace
+ * @buf: Buffer containing upstream message
+ * @len: length of message to write
+ * @spi: SPI device of message originator
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_test_int_driver_write(
+ const u8 *buf,
+ size_t len,
+ void *spi
+)
+{
+ struct ca8210_priv *priv = spi_get_drvdata(spi);
+ struct ca8210_test *test = &priv->test;
+ char *fifo_buffer;
+ int i;
+
+ dev_dbg(
+ &priv->spi->dev,
+ "test_interface: Buffering upstream message:\n"
+ );
+ for (i = 0; i < len; i++)
+ dev_dbg(&priv->spi->dev, "%#03x\n", buf[i]);
+
+ fifo_buffer = kmalloc(len, GFP_KERNEL);
+ if (!fifo_buffer)
+ return -ENOMEM;
+ memcpy(fifo_buffer, buf, len);
+ kfifo_in(&test->up_fifo, &fifo_buffer, 4);
+ wake_up_interruptible(&priv->test.readq);
+
+ return 0;
+}
+
+/* SPI Operation */
+
+static int ca8210_net_rx(
+ struct ieee802154_hw *hw,
+ u8 *command,
+ size_t len
+);
+static u8 mlme_reset_request_sync(
+ u8 set_default_pib,
+ void *device_ref
+);
+static int ca8210_spi_transfer(
+ struct spi_device *spi,
+ const u8 *buf,
+ size_t len
+);
+
+/**
+ * ca8210_reset_send() - Hard resets the ca8210 for a given time
+ * @spi: Pointer to target ca8210 spi device
+ * @ms: Milliseconds to hold the reset line low for
+ */
+static void ca8210_reset_send(struct spi_device *spi, unsigned int ms)
+{
+ struct ca8210_platform_data *pdata = spi->dev.platform_data;
+ struct ca8210_priv *priv = spi_get_drvdata(spi);
+ long status;
+
+ gpio_set_value(pdata->gpio_reset, 0);
+ reinit_completion(&priv->ca8210_is_awake);
+ msleep(ms);
+ gpio_set_value(pdata->gpio_reset, 1);
+ priv->promiscuous = false;
+
+ /* Wait until wakeup indication seen */
+ status = wait_for_completion_interruptible_timeout(
+ &priv->ca8210_is_awake,
+ msecs_to_jiffies(CA8210_SYNC_TIMEOUT)
+ );
+ if (status == 0) {
+ dev_crit(
+ &spi->dev,
+ "Fatal: No wakeup from ca8210 after reset!\n"
+ );
+ }
+
+ dev_dbg(&spi->dev, "Reset the device\n");
+}
+
+/**
+ * ca8210_mlme_reset_worker() - Resets the MLME, Called when the MAC OVERFLOW
+ * condition happens.
+ * @work: Pointer to work being executed
+ */
+static void ca8210_mlme_reset_worker(struct work_struct *work)
+{
+ struct work_priv_container *wpc = container_of(
+ work,
+ struct work_priv_container,
+ work
+ );
+ struct ca8210_priv *priv = wpc->priv;
+
+ mlme_reset_request_sync(0, priv->spi);
+ kfree(wpc);
+}
+
+/**
+ * ca8210_rx_done() - Calls various message dispatches responding to a received
+ * command
+ * @arg: Pointer to the cas_control object for the relevant spi transfer
+ *
+ * Presents a received SAP command from the ca8210 to the Cascoda EVBME, test
+ * interface and network driver.
+ */
+static void ca8210_rx_done(struct cas_control *cas_ctl)
+{
+ u8 *buf;
+ u8 len;
+ struct work_priv_container *mlme_reset_wpc;
+ struct ca8210_priv *priv = cas_ctl->priv;
+
+ buf = cas_ctl->tx_in_buf;
+ len = buf[1] + 2;
+ if (len > CA8210_SPI_BUF_SIZE) {
+ dev_crit(
+ &priv->spi->dev,
+ "Received packet len (%d) erroneously long\n",
+ len
+ );
+ goto finish;
+ }
+
+ if (buf[0] & SPI_SYN) {
+ if (priv->sync_command_response) {
+ memcpy(priv->sync_command_response, buf, len);
+ complete(&priv->sync_exchange_complete);
+ } else {
+ if (cascoda_api_upstream)
+ cascoda_api_upstream(buf, len, priv->spi);
+ priv->sync_up++;
+ }
+ } else {
+ if (cascoda_api_upstream)
+ cascoda_api_upstream(buf, len, priv->spi);
+ }
+
+ ca8210_net_rx(priv->hw, buf, len);
+ if (buf[0] == SPI_MCPS_DATA_CONFIRM) {
+ if (buf[3] == MAC_TRANSACTION_OVERFLOW) {
+ dev_info(
+ &priv->spi->dev,
+ "Waiting for transaction overflow to stabilise...\n");
+ msleep(2000);
+ dev_info(
+ &priv->spi->dev,
+ "Resetting MAC...\n");
+
+ mlme_reset_wpc = kmalloc(sizeof(*mlme_reset_wpc),
+ GFP_KERNEL);
+ if (!mlme_reset_wpc)
+ goto finish;
+ INIT_WORK(
+ &mlme_reset_wpc->work,
+ ca8210_mlme_reset_worker
+ );
+ mlme_reset_wpc->priv = priv;
+ queue_work(priv->mlme_workqueue, &mlme_reset_wpc->work);
+ }
+ } else if (buf[0] == SPI_HWME_WAKEUP_INDICATION) {
+ dev_notice(
+ &priv->spi->dev,
+ "Wakeup indication received, reason:\n"
+ );
+ switch (buf[2]) {
+ case 0:
+ dev_notice(
+ &priv->spi->dev,
+ "Transceiver woken up from Power Up / System Reset\n"
+ );
+ break;
+ case 1:
+ dev_notice(
+ &priv->spi->dev,
+ "Watchdog Timer Time-Out\n"
+ );
+ break;
+ case 2:
+ dev_notice(
+ &priv->spi->dev,
+ "Transceiver woken up from Power-Off by Sleep Timer Time-Out\n");
+ break;
+ case 3:
+ dev_notice(
+ &priv->spi->dev,
+ "Transceiver woken up from Power-Off by GPIO Activity\n"
+ );
+ break;
+ case 4:
+ dev_notice(
+ &priv->spi->dev,
+ "Transceiver woken up from Standby by Sleep Timer Time-Out\n"
+ );
+ break;
+ case 5:
+ dev_notice(
+ &priv->spi->dev,
+ "Transceiver woken up from Standby by GPIO Activity\n"
+ );
+ break;
+ case 6:
+ dev_notice(
+ &priv->spi->dev,
+ "Sleep-Timer Time-Out in Active Mode\n"
+ );
+ break;
+ default:
+ dev_warn(&priv->spi->dev, "Wakeup reason unknown\n");
+ break;
+ }
+ complete(&priv->ca8210_is_awake);
+ }
+
+finish:;
+}
+
+static int ca8210_remove(struct spi_device *spi_device);
+
+/**
+ * ca8210_spi_transfer_complete() - Called when a single spi transfer has
+ * completed
+ * @context: Pointer to the cas_control object for the finished transfer
+ */
+static void ca8210_spi_transfer_complete(void *context)
+{
+ struct cas_control *cas_ctl = context;
+ struct ca8210_priv *priv = cas_ctl->priv;
+ bool duplex_rx = false;
+ int i;
+ u8 retry_buffer[CA8210_SPI_BUF_SIZE];
+
+ if (
+ cas_ctl->tx_in_buf[0] == SPI_NACK ||
+ (cas_ctl->tx_in_buf[0] == SPI_IDLE &&
+ cas_ctl->tx_in_buf[1] == SPI_NACK)
+ ) {
+ /* ca8210 is busy */
+ dev_info(&priv->spi->dev, "ca8210 was busy during attempted write\n");
+ if (cas_ctl->tx_buf[0] == SPI_IDLE) {
+ dev_warn(
+ &priv->spi->dev,
+ "IRQ servicing NACKd, dropping transfer\n"
+ );
+ kfree(cas_ctl);
+ return;
+ }
+ if (priv->retries > 3) {
+ dev_err(&priv->spi->dev, "too many retries!\n");
+ kfree(cas_ctl);
+ ca8210_remove(priv->spi);
+ return;
+ }
+ memcpy(retry_buffer, cas_ctl->tx_buf, CA8210_SPI_BUF_SIZE);
+ kfree(cas_ctl);
+ ca8210_spi_transfer(
+ priv->spi,
+ retry_buffer,
+ CA8210_SPI_BUF_SIZE
+ );
+ priv->retries++;
+ dev_info(&priv->spi->dev, "retried spi write\n");
+ return;
+ } else if (
+ cas_ctl->tx_in_buf[0] != SPI_IDLE &&
+ cas_ctl->tx_in_buf[0] != SPI_NACK
+ ) {
+ duplex_rx = true;
+ }
+
+ if (duplex_rx) {
+ dev_dbg(&priv->spi->dev, "READ CMD DURING TX\n");
+ for (i = 0; i < cas_ctl->tx_in_buf[1] + 2; i++)
+ dev_dbg(
+ &priv->spi->dev,
+ "%#03x\n",
+ cas_ctl->tx_in_buf[i]
+ );
+ ca8210_rx_done(cas_ctl);
+ }
+ complete(&priv->spi_transfer_complete);
+ kfree(cas_ctl);
+ priv->retries = 0;
+}
+
+/**
+ * ca8210_spi_transfer() - Initiate duplex spi transfer with ca8210
+ * @spi: Pointer to spi device for transfer
+ * @buf: Octet array to send
+ * @len: length of the buffer being sent
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_spi_transfer(
+ struct spi_device *spi,
+ const u8 *buf,
+ size_t len
+)
+{
+ int i, status = 0;
+ struct ca8210_priv *priv = spi_get_drvdata(spi);
+ struct cas_control *cas_ctl;
+
+ if (!spi) {
+ dev_crit(
+ &spi->dev,
+ "NULL spi device passed to ca8210_spi_transfer\n"
+ );
+ return -ENODEV;
+ }
+
+ reinit_completion(&priv->spi_transfer_complete);
+
+ dev_dbg(&spi->dev, "ca8210_spi_transfer called\n");
+
+ cas_ctl = kmalloc(sizeof(*cas_ctl), GFP_ATOMIC);
+ if (!cas_ctl)
+ return -ENOMEM;
+
+ cas_ctl->priv = priv;
+ memset(cas_ctl->tx_buf, SPI_IDLE, CA8210_SPI_BUF_SIZE);
+ memset(cas_ctl->tx_in_buf, SPI_IDLE, CA8210_SPI_BUF_SIZE);
+ memcpy(cas_ctl->tx_buf, buf, len);
+
+ for (i = 0; i < len; i++)
+ dev_dbg(&spi->dev, "%#03x\n", cas_ctl->tx_buf[i]);
+
+ spi_message_init(&cas_ctl->msg);
+
+ cas_ctl->transfer.tx_nbits = 1; /* 1 MOSI line */
+ cas_ctl->transfer.rx_nbits = 1; /* 1 MISO line */
+ cas_ctl->transfer.speed_hz = 0; /* Use device setting */
+ cas_ctl->transfer.bits_per_word = 0; /* Use device setting */
+ cas_ctl->transfer.tx_buf = cas_ctl->tx_buf;
+ cas_ctl->transfer.rx_buf = cas_ctl->tx_in_buf;
+ cas_ctl->transfer.delay_usecs = 0;
+ cas_ctl->transfer.cs_change = 0;
+ cas_ctl->transfer.len = sizeof(struct mac_message);
+ cas_ctl->msg.complete = ca8210_spi_transfer_complete;
+ cas_ctl->msg.context = cas_ctl;
+
+ spi_message_add_tail(
+ &cas_ctl->transfer,
+ &cas_ctl->msg
+ );
+
+ status = spi_async(spi, &cas_ctl->msg);
+ if (status < 0) {
+ dev_crit(
+ &spi->dev,
+ "status %d from spi_sync in write\n",
+ status
+ );
+ }
+
+ return status;
+}
+
+/**
+ * ca8210_spi_exchange() - Exchange API/SAP commands with the radio
+ * @buf: Octet array of command being sent downstream
+ * @len: length of buf
+ * @response: buffer for storing synchronous response
+ * @device_ref: spi_device pointer for ca8210
+ *
+ * Effectively calls ca8210_spi_transfer to write buf[] to the spi, then for
+ * synchronous commands waits for the corresponding response to be read from
+ * the spi before returning. The response is written to the response parameter.
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_spi_exchange(
+ const u8 *buf,
+ size_t len,
+ u8 *response,
+ void *device_ref
+)
+{
+ int status = 0;
+ struct spi_device *spi = device_ref;
+ struct ca8210_priv *priv = spi->dev.driver_data;
+ long wait_remaining;
+
+ if ((buf[0] & SPI_SYN) && response) { /* if sync wait for confirm */
+ reinit_completion(&priv->sync_exchange_complete);
+ priv->sync_command_response = response;
+ }
+
+ do {
+ reinit_completion(&priv->spi_transfer_complete);
+ status = ca8210_spi_transfer(priv->spi, buf, len);
+ if (status) {
+ dev_warn(
+ &spi->dev,
+ "spi write failed, returned %d\n",
+ status
+ );
+ if (status == -EBUSY)
+ continue;
+ if (((buf[0] & SPI_SYN) && response))
+ complete(&priv->sync_exchange_complete);
+ goto cleanup;
+ }
+
+ wait_remaining = wait_for_completion_interruptible_timeout(
+ &priv->spi_transfer_complete,
+ msecs_to_jiffies(1000)
+ );
+ if (wait_remaining == -ERESTARTSYS) {
+ status = -ERESTARTSYS;
+ } else if (wait_remaining == 0) {
+ dev_err(
+ &spi->dev,
+ "SPI downstream transfer timed out!\n"
+ );
+ status = -ETIME;
+ goto cleanup;
+ }
+ } while (status < 0);
+
+ if (!((buf[0] & SPI_SYN) && response))
+ goto cleanup;
+
+ wait_remaining = wait_for_completion_interruptible_timeout(
+ &priv->sync_exchange_complete,
+ msecs_to_jiffies(CA8210_SYNC_TIMEOUT)
+ );
+ if (wait_remaining == -ERESTARTSYS) {
+ status = -ERESTARTSYS;
+ } else if (wait_remaining == 0) {
+ dev_err(
+ &spi->dev,
+ "Synchronous confirm timeout\n"
+ );
+ status = -ETIME;
+ }
+
+cleanup:
+ priv->sync_command_response = NULL;
+ return status;
+}
+
+/**
+ * ca8210_interrupt_handler() - Called when an irq is received from the ca8210
+ * @irq: Id of the irq being handled
+ * @dev_id: Pointer passed by the system, pointing to the ca8210's private data
+ *
+ * This function is called when the irq line from the ca8210 is asserted,
+ * signifying that the ca8210 has a message to send upstream to us. Starts the
+ * asynchronous spi read.
+ *
+ * Return: irq return code
+ */
+static irqreturn_t ca8210_interrupt_handler(int irq, void *dev_id)
+{
+ struct ca8210_priv *priv = dev_id;
+ int status;
+
+ dev_dbg(&priv->spi->dev, "irq: Interrupt occurred\n");
+ do {
+ status = ca8210_spi_transfer(priv->spi, NULL, 0);
+ if (status && (status != -EBUSY)) {
+ dev_warn(
+ &priv->spi->dev,
+ "spi read failed, returned %d\n",
+ status
+ );
+ }
+ } while (status == -EBUSY);
+ return IRQ_HANDLED;
+}
+
+static int (*cascoda_api_downstream)(
+ const u8 *buf,
+ size_t len,
+ u8 *response,
+ void *device_ref
+) = ca8210_spi_exchange;
+
+/* Cascoda API / 15.4 SAP Primitives */
+
+/**
+ * tdme_setsfr_request_sync() - TDME_SETSFR_request/confirm according to API
+ * @sfr_page: SFR Page
+ * @sfr_address: SFR Address
+ * @sfr_value: SFR Value
+ * @device_ref: Nondescript pointer to target device
+ *
+ * Return: 802.15.4 status code of TDME-SETSFR.confirm
+ */
+static u8 tdme_setsfr_request_sync(
+ u8 sfr_page,
+ u8 sfr_address,
+ u8 sfr_value,
+ void *device_ref
+)
+{
+ int ret;
+ struct mac_message command, response;
+ struct spi_device *spi = device_ref;
+
+ command.command_id = SPI_TDME_SETSFR_REQUEST;
+ command.length = 3;
+ command.pdata.tdme_set_sfr_req.sfr_page = sfr_page;
+ command.pdata.tdme_set_sfr_req.sfr_address = sfr_address;
+ command.pdata.tdme_set_sfr_req.sfr_value = sfr_value;
+ response.command_id = SPI_IDLE;
+ ret = cascoda_api_downstream(
+ &command.command_id,
+ command.length + 2,
+ &response.command_id,
+ device_ref
+ );
+ if (ret) {
+ dev_crit(&spi->dev, "cascoda_api_downstream returned %d", ret);
+ return MAC_SYSTEM_ERROR;
+ }
+
+ if (response.command_id != SPI_TDME_SETSFR_CONFIRM) {
+ dev_crit(
+ &spi->dev,
+ "sync response to SPI_TDME_SETSFR_REQUEST was not SPI_TDME_SETSFR_CONFIRM, it was %d\n",
+ response.command_id
+ );
+ return MAC_SYSTEM_ERROR;
+ }
+
+ return response.pdata.tdme_set_sfr_cnf.status;
+}
+
+/**
+ * tdme_chipinit() - TDME Chip Register Default Initialisation Macro
+ * @device_ref: Nondescript pointer to target device
+ *
+ * Return: 802.15.4 status code of API calls
+ */
+static u8 tdme_chipinit(void *device_ref)
+{
+ u8 status = MAC_SUCCESS;
+ u8 sfr_address;
+ struct spi_device *spi = device_ref;
+ struct preamble_cfg_sfr pre_cfg_value = {
+ .timeout_symbols = 3,
+ .acquisition_symbols = 3,
+ .search_symbols = 1,
+ };
+ /* LNA Gain Settings */
+ status = tdme_setsfr_request_sync(
+ 1, (sfr_address = CA8210_SFR_LNAGX40),
+ LNAGX40_DEFAULT_GAIN, device_ref);
+ if (status)
+ goto finish;
+ status = tdme_setsfr_request_sync(
+ 1, (sfr_address = CA8210_SFR_LNAGX41),
+ LNAGX41_DEFAULT_GAIN, device_ref);
+ if (status)
+ goto finish;
+ status = tdme_setsfr_request_sync(
+ 1, (sfr_address = CA8210_SFR_LNAGX42),
+ LNAGX42_DEFAULT_GAIN, device_ref);
+ if (status)
+ goto finish;
+ status = tdme_setsfr_request_sync(
+ 1, (sfr_address = CA8210_SFR_LNAGX43),
+ LNAGX43_DEFAULT_GAIN, device_ref);
+ if (status)
+ goto finish;
+ status = tdme_setsfr_request_sync(
+ 1, (sfr_address = CA8210_SFR_LNAGX44),
+ LNAGX44_DEFAULT_GAIN, device_ref);
+ if (status)
+ goto finish;
+ status = tdme_setsfr_request_sync(
+ 1, (sfr_address = CA8210_SFR_LNAGX45),
+ LNAGX45_DEFAULT_GAIN, device_ref);
+ if (status)
+ goto finish;
+ status = tdme_setsfr_request_sync(
+ 1, (sfr_address = CA8210_SFR_LNAGX46),
+ LNAGX46_DEFAULT_GAIN, device_ref);
+ if (status)
+ goto finish;
+ status = tdme_setsfr_request_sync(
+ 1, (sfr_address = CA8210_SFR_LNAGX47),
+ LNAGX47_DEFAULT_GAIN, device_ref);
+ if (status)
+ goto finish;
+ /* Preamble Timing Config */
+ status = tdme_setsfr_request_sync(
+ 1, (sfr_address = CA8210_SFR_PRECFG),
+ *((u8 *)&pre_cfg_value), device_ref);
+ if (status)
+ goto finish;
+ /* Preamble Threshold High */
+ status = tdme_setsfr_request_sync(
+ 1, (sfr_address = CA8210_SFR_PTHRH),
+ PTHRH_DEFAULT_THRESHOLD, device_ref);
+ if (status)
+ goto finish;
+ /* Tx Output Power 8 dBm */
+ status = tdme_setsfr_request_sync(
+ 0, (sfr_address = CA8210_SFR_PACFGIB),
+ PACFGIB_DEFAULT_CURRENT, device_ref);
+ if (status)
+ goto finish;
+
+finish:
+ if (status != MAC_SUCCESS) {
+ dev_err(
+ &spi->dev,
+ "failed to set sfr at %#03x, status = %#03x\n",
+ sfr_address,
+ status
+ );
+ }
+ return status;
+}
+
+/**
+ * tdme_channelinit() - TDME Channel Register Default Initialisation Macro (Tx)
+ * @channel: 802.15.4 channel to initialise chip for
+ * @device_ref: Nondescript pointer to target device
+ *
+ * Return: 802.15.4 status code of API calls
+ */
+static u8 tdme_channelinit(u8 channel, void *device_ref)
+{
+ /* Transceiver front-end local oscillator tx two-point calibration
+ * value. Tuned for the hardware.
+ */
+ u8 txcalval;
+
+ if (channel >= 25)
+ txcalval = 0xA7;
+ else if (channel >= 23)
+ txcalval = 0xA8;
+ else if (channel >= 22)
+ txcalval = 0xA9;
+ else if (channel >= 20)
+ txcalval = 0xAA;
+ else if (channel >= 17)
+ txcalval = 0xAB;
+ else if (channel >= 16)
+ txcalval = 0xAC;
+ else if (channel >= 14)
+ txcalval = 0xAD;
+ else if (channel >= 12)
+ txcalval = 0xAE;
+ else
+ txcalval = 0xAF;
+
+ return tdme_setsfr_request_sync(
+ 1,
+ CA8210_SFR_LOTXCAL,
+ txcalval,
+ device_ref
+ ); /* LO Tx Cal */
+}
+
+/**
+ * tdme_checkpibattribute() - Checks Attribute Values that are not checked in
+ * MAC
+ * @pib_attribute: Attribute Number
+ * @pib_attribute_length: Attribute length
+ * @pib_attribute_value: Pointer to Attribute Value
+ * @device_ref: Nondescript pointer to target device
+ *
+ * Return: 802.15.4 status code of checks
+ */
+static u8 tdme_checkpibattribute(
+ u8 pib_attribute,
+ u8 pib_attribute_length,
+ const void *pib_attribute_value
+)
+{
+ u8 status = MAC_SUCCESS;
+ u8 value;
+
+ value = *((u8 *)pib_attribute_value);
+
+ switch (pib_attribute) {
+ /* PHY */
+ case PHY_TRANSMIT_POWER:
+ if (value > 0x3F)
+ status = MAC_INVALID_PARAMETER;
+ break;
+ case PHY_CCA_MODE:
+ if (value > 0x03)
+ status = MAC_INVALID_PARAMETER;
+ break;
+ /* MAC */
+ case MAC_BATT_LIFE_EXT_PERIODS:
+ if ((value < 6) || (value > 41))
+ status = MAC_INVALID_PARAMETER;
+ break;
+ case MAC_BEACON_PAYLOAD:
+ if (pib_attribute_length > MAX_BEACON_PAYLOAD_LENGTH)
+ status = MAC_INVALID_PARAMETER;
+ break;
+ case MAC_BEACON_PAYLOAD_LENGTH:
+ if (value > MAX_BEACON_PAYLOAD_LENGTH)
+ status = MAC_INVALID_PARAMETER;
+ break;
+ case MAC_BEACON_ORDER:
+ if (value > 15)
+ status = MAC_INVALID_PARAMETER;
+ break;
+ case MAC_MAX_BE:
+ if ((value < 3) || (value > 8))
+ status = MAC_INVALID_PARAMETER;
+ break;
+ case MAC_MAX_CSMA_BACKOFFS:
+ if (value > 5)
+ status = MAC_INVALID_PARAMETER;
+ break;
+ case MAC_MAX_FRAME_RETRIES:
+ if (value > 7)
+ status = MAC_INVALID_PARAMETER;
+ break;
+ case MAC_MIN_BE:
+ if (value > 8)
+ status = MAC_INVALID_PARAMETER;
+ break;
+ case MAC_RESPONSE_WAIT_TIME:
+ if ((value < 2) || (value > 64))
+ status = MAC_INVALID_PARAMETER;
+ break;
+ case MAC_SUPERFRAME_ORDER:
+ if (value > 15)
+ status = MAC_INVALID_PARAMETER;
+ break;
+ /* boolean */
+ case MAC_ASSOCIATED_PAN_COORD:
+ case MAC_ASSOCIATION_PERMIT:
+ case MAC_AUTO_REQUEST:
+ case MAC_BATT_LIFE_EXT:
+ case MAC_GTS_PERMIT:
+ case MAC_PROMISCUOUS_MODE:
+ case MAC_RX_ON_WHEN_IDLE:
+ case MAC_SECURITY_ENABLED:
+ if (value > 1)
+ status = MAC_INVALID_PARAMETER;
+ break;
+ /* MAC SEC */
+ case MAC_AUTO_REQUEST_SECURITY_LEVEL:
+ if (value > 7)
+ status = MAC_INVALID_PARAMETER;
+ break;
+ case MAC_AUTO_REQUEST_KEY_ID_MODE:
+ if (value > 3)
+ status = MAC_INVALID_PARAMETER;
+ break;
+ default:
+ break;
+ }
+
+ return status;
+}
+
+/**
+ * tdme_settxpower() - Sets the tx power for MLME_SET phyTransmitPower
+ * @txp: Transmit Power
+ * @device_ref: Nondescript pointer to target device
+ *
+ * Normalised to 802.15.4 Definition (6-bit, signed):
+ * Bit 7-6: not used
+ * Bit 5-0: tx power (-32 - +31 dB)
+ *
+ * Return: 802.15.4 status code of api calls
+ */
+static u8 tdme_settxpower(u8 txp, void *device_ref)
+{
+ u8 status;
+ s8 txp_val;
+ u8 txp_ext;
+ union pa_cfg_sfr pa_cfg_val;
+
+ /* extend from 6 to 8 bit */
+ txp_ext = 0x3F & txp;
+ if (txp_ext & 0x20)
+ txp_ext += 0xC0;
+ txp_val = (s8)txp_ext;
+
+ if (CA8210_MAC_MPW) {
+ if (txp_val > 0) {
+ /* 8 dBm: ptrim = 5, itrim = +3 => +4 dBm */
+ pa_cfg_val.bias_current_trim = 3;
+ pa_cfg_val.buffer_capacitor_trim = 5;
+ pa_cfg_val.boost = 1;
+ } else {
+ /* 0 dBm: ptrim = 7, itrim = +3 => -6 dBm */
+ pa_cfg_val.bias_current_trim = 3;
+ pa_cfg_val.buffer_capacitor_trim = 7;
+ pa_cfg_val.boost = 0;
+ }
+ /* write PACFG */
+ status = tdme_setsfr_request_sync(
+ 0,
+ CA8210_SFR_PACFG,
+ pa_cfg_val.paib,
+ device_ref
+ );
+ } else {
+ /* Look-Up Table for Setting Current and Frequency Trim values
+ * for desired Output Power
+ */
+ if (txp_val > 8) {
+ pa_cfg_val.paib = 0x3F;
+ } else if (txp_val == 8) {
+ pa_cfg_val.paib = 0x32;
+ } else if (txp_val == 7) {
+ pa_cfg_val.paib = 0x22;
+ } else if (txp_val == 6) {
+ pa_cfg_val.paib = 0x18;
+ } else if (txp_val == 5) {
+ pa_cfg_val.paib = 0x10;
+ } else if (txp_val == 4) {
+ pa_cfg_val.paib = 0x0C;
+ } else if (txp_val == 3) {
+ pa_cfg_val.paib = 0x08;
+ } else if (txp_val == 2) {
+ pa_cfg_val.paib = 0x05;
+ } else if (txp_val == 1) {
+ pa_cfg_val.paib = 0x03;
+ } else if (txp_val == 0) {
+ pa_cfg_val.paib = 0x01;
+ } else { /* < 0 */
+ pa_cfg_val.paib = 0x00;
+ }
+ /* write PACFGIB */
+ status = tdme_setsfr_request_sync(
+ 0,
+ CA8210_SFR_PACFGIB,
+ pa_cfg_val.paib,
+ device_ref
+ );
+ }
+
+ return status;
+}
+
+/**
+ * mcps_data_request() - mcps_data_request (Send Data) according to API Spec
+ * @src_addr_mode: Source Addressing Mode
+ * @dst_address_mode: Destination Addressing Mode
+ * @dst_pan_id: Destination PAN ID
+ * @dst_addr: Pointer to Destination Address
+ * @msdu_length: length of Data
+ * @msdu: Pointer to Data
+ * @msdu_handle: Handle of Data
+ * @tx_options: Tx Options Bit Field
+ * @security: Pointer to Security Structure or NULL
+ * @device_ref: Nondescript pointer to target device
+ *
+ * Return: 802.15.4 status code of action
+ */
+static u8 mcps_data_request(
+ u8 src_addr_mode,
+ u8 dst_address_mode,
+ u16 dst_pan_id,
+ union macaddr *dst_addr,
+ u8 msdu_length,
+ u8 *msdu,
+ u8 msdu_handle,
+ u8 tx_options,
+ struct secspec *security,
+ void *device_ref
+)
+{
+ struct secspec *psec;
+ struct mac_message command;
+
+ command.command_id = SPI_MCPS_DATA_REQUEST;
+ command.pdata.data_req.src_addr_mode = src_addr_mode;
+ command.pdata.data_req.dst.mode = dst_address_mode;
+ if (dst_address_mode != MAC_MODE_NO_ADDR) {
+ command.pdata.data_req.dst.pan_id[0] = LS_BYTE(dst_pan_id);
+ command.pdata.data_req.dst.pan_id[1] = MS_BYTE(dst_pan_id);
+ if (dst_address_mode == MAC_MODE_SHORT_ADDR) {
+ command.pdata.data_req.dst.address[0] = LS_BYTE(
+ dst_addr->short_address
+ );
+ command.pdata.data_req.dst.address[1] = MS_BYTE(
+ dst_addr->short_address
+ );
+ } else { /* MAC_MODE_LONG_ADDR*/
+ memcpy(
+ command.pdata.data_req.dst.address,
+ dst_addr->ieee_address,
+ 8
+ );
+ }
+ }
+ command.pdata.data_req.msdu_length = msdu_length;
+ command.pdata.data_req.msdu_handle = msdu_handle;
+ command.pdata.data_req.tx_options = tx_options;
+ memcpy(command.pdata.data_req.msdu, msdu, msdu_length);
+ psec = (struct secspec *)(command.pdata.data_req.msdu + msdu_length);
+ command.length = sizeof(struct mcps_data_request_pset) -
+ MAX_DATA_SIZE + msdu_length;
+ if (!security || (security->security_level == 0)) {
+ psec->security_level = 0;
+ command.length += 1;
+ } else {
+ *psec = *security;
+ command.length += sizeof(struct secspec);
+ }
+
+ if (ca8210_spi_transfer(device_ref, &command.command_id,
+ command.length + 2))
+ return MAC_SYSTEM_ERROR;
+
+ return MAC_SUCCESS;
+}
+
+/**
+ * mlme_reset_request_sync() - MLME_RESET_request/confirm according to API Spec
+ * @set_default_pib: Set defaults in PIB
+ * @device_ref: Nondescript pointer to target device
+ *
+ * Return: 802.15.4 status code of MLME-RESET.confirm
+ */
+static u8 mlme_reset_request_sync(
+ u8 set_default_pib,
+ void *device_ref
+)
+{
+ u8 status;
+ struct mac_message command, response;
+ struct spi_device *spi = device_ref;
+
+ command.command_id = SPI_MLME_RESET_REQUEST;
+ command.length = 1;
+ command.pdata.u8param = set_default_pib;
+
+ if (cascoda_api_downstream(
+ &command.command_id,
+ command.length + 2,
+ &response.command_id,
+ device_ref)) {
+ dev_err(&spi->dev, "cascoda_api_downstream failed\n");
+ return MAC_SYSTEM_ERROR;
+ }
+
+ if (response.command_id != SPI_MLME_RESET_CONFIRM)
+ return MAC_SYSTEM_ERROR;
+
+ status = response.pdata.status;
+
+ /* reset COORD Bit for Channel Filtering as Coordinator */
+ if (CA8210_MAC_WORKAROUNDS && set_default_pib && (!status)) {
+ status = tdme_setsfr_request_sync(
+ 0,
+ CA8210_SFR_MACCON,
+ 0,
+ device_ref
+ );
+ }
+
+ return status;
+}
+
+/**
+ * mlme_set_request_sync() - MLME_SET_request/confirm according to API Spec
+ * @pib_attribute: Attribute Number
+ * @pib_attribute_index: Index within Attribute if an Array
+ * @pib_attribute_length: Attribute length
+ * @pib_attribute_value: Pointer to Attribute Value
+ * @device_ref: Nondescript pointer to target device
+ *
+ * Return: 802.15.4 status code of MLME-SET.confirm
+ */
+static u8 mlme_set_request_sync(
+ u8 pib_attribute,
+ u8 pib_attribute_index,
+ u8 pib_attribute_length,
+ const void *pib_attribute_value,
+ void *device_ref
+)
+{
+ u8 status;
+ struct mac_message command, response;
+
+ /* pre-check the validity of pib_attribute values that are not checked
+ * in MAC
+ */
+ if (tdme_checkpibattribute(
+ pib_attribute, pib_attribute_length, pib_attribute_value)) {
+ return MAC_INVALID_PARAMETER;
+ }
+
+ if (pib_attribute == PHY_CURRENT_CHANNEL) {
+ status = tdme_channelinit(
+ *((u8 *)pib_attribute_value),
+ device_ref
+ );
+ if (status)
+ return status;
+ }
+
+ if (pib_attribute == PHY_TRANSMIT_POWER) {
+ return tdme_settxpower(
+ *((u8 *)pib_attribute_value),
+ device_ref
+ );
+ }
+
+ command.command_id = SPI_MLME_SET_REQUEST;
+ command.length = sizeof(struct mlme_set_request_pset) -
+ MAX_ATTRIBUTE_SIZE + pib_attribute_length;
+ command.pdata.set_req.pib_attribute = pib_attribute;
+ command.pdata.set_req.pib_attribute_index = pib_attribute_index;
+ command.pdata.set_req.pib_attribute_length = pib_attribute_length;
+ memcpy(
+ command.pdata.set_req.pib_attribute_value,
+ pib_attribute_value,
+ pib_attribute_length
+ );
+
+ if (cascoda_api_downstream(
+ &command.command_id,
+ command.length + 2,
+ &response.command_id,
+ device_ref)) {
+ return MAC_SYSTEM_ERROR;
+ }
+
+ if (response.command_id != SPI_MLME_SET_CONFIRM)
+ return MAC_SYSTEM_ERROR;
+
+ return response.pdata.status;
+}
+
+/**
+ * hwme_set_request_sync() - HWME_SET_request/confirm according to API Spec
+ * @hw_attribute: Attribute Number
+ * @hw_attribute_length: Attribute length
+ * @hw_attribute_value: Pointer to Attribute Value
+ * @device_ref: Nondescript pointer to target device
+ *
+ * Return: 802.15.4 status code of HWME-SET.confirm
+ */
+static u8 hwme_set_request_sync(
+ u8 hw_attribute,
+ u8 hw_attribute_length,
+ u8 *hw_attribute_value,
+ void *device_ref
+)
+{
+ struct mac_message command, response;
+
+ command.command_id = SPI_HWME_SET_REQUEST;
+ command.length = 2 + hw_attribute_length;
+ command.pdata.hwme_set_req.hw_attribute = hw_attribute;
+ command.pdata.hwme_set_req.hw_attribute_length = hw_attribute_length;
+ memcpy(
+ command.pdata.hwme_set_req.hw_attribute_value,
+ hw_attribute_value,
+ hw_attribute_length
+ );
+
+ if (cascoda_api_downstream(
+ &command.command_id,
+ command.length + 2,
+ &response.command_id,
+ device_ref)) {
+ return MAC_SYSTEM_ERROR;
+ }
+
+ if (response.command_id != SPI_HWME_SET_CONFIRM)
+ return MAC_SYSTEM_ERROR;
+
+ return response.pdata.hwme_set_cnf.status;
+}
+
+/**
+ * hwme_get_request_sync() - HWME_GET_request/confirm according to API Spec
+ * @hw_attribute: Attribute Number
+ * @hw_attribute_length: Attribute length
+ * @hw_attribute_value: Pointer to Attribute Value
+ * @device_ref: Nondescript pointer to target device
+ *
+ * Return: 802.15.4 status code of HWME-GET.confirm
+ */
+static u8 hwme_get_request_sync(
+ u8 hw_attribute,
+ u8 *hw_attribute_length,
+ u8 *hw_attribute_value,
+ void *device_ref
+)
+{
+ struct mac_message command, response;
+
+ command.command_id = SPI_HWME_GET_REQUEST;
+ command.length = 1;
+ command.pdata.hwme_get_req.hw_attribute = hw_attribute;
+
+ if (cascoda_api_downstream(
+ &command.command_id,
+ command.length + 2,
+ &response.command_id,
+ device_ref)) {
+ return MAC_SYSTEM_ERROR;
+ }
+
+ if (response.command_id != SPI_HWME_GET_CONFIRM)
+ return MAC_SYSTEM_ERROR;
+
+ if (response.pdata.hwme_get_cnf.status == MAC_SUCCESS) {
+ *hw_attribute_length =
+ response.pdata.hwme_get_cnf.hw_attribute_length;
+ memcpy(
+ hw_attribute_value,
+ response.pdata.hwme_get_cnf.hw_attribute_value,
+ *hw_attribute_length
+ );
+ }
+
+ return response.pdata.hwme_get_cnf.status;
+}
+
+/* Network driver operation */
+
+/**
+ * ca8210_async_xmit_complete() - Called to announce that an asynchronous
+ * transmission has finished
+ * @hw: ieee802154_hw of ca8210 that has finished exchange
+ * @msduhandle: Identifier of transmission that has completed
+ * @status: Returned 802.15.4 status code of the transmission
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_async_xmit_complete(
+ struct ieee802154_hw *hw,
+ u8 msduhandle,
+ u8 status)
+{
+ struct ca8210_priv *priv = hw->priv;
+
+ if (priv->nextmsduhandle != msduhandle) {
+ dev_err(
+ &priv->spi->dev,
+ "Unexpected msdu_handle on data confirm, Expected %d, got %d\n",
+ priv->nextmsduhandle,
+ msduhandle
+ );
+ return -EIO;
+ }
+
+ priv->async_tx_pending = false;
+ priv->nextmsduhandle++;
+
+ if (status) {
+ dev_err(
+ &priv->spi->dev,
+ "Link transmission unsuccessful, status = %d\n",
+ status
+ );
+ if (status != MAC_TRANSACTION_OVERFLOW) {
+ ieee802154_wake_queue(priv->hw);
+ return 0;
+ }
+ }
+ ieee802154_xmit_complete(priv->hw, priv->tx_skb, true);
+
+ return 0;
+}
+
+/**
+ * ca8210_skb_rx() - Contructs a properly framed socket buffer from a received
+ * MCPS_DATA_indication
+ * @hw: ieee802154_hw that MCPS_DATA_indication was received by
+ * @len: length of MCPS_DATA_indication
+ * @data_ind: Octet array of MCPS_DATA_indication
+ *
+ * Called by the spi driver whenever a SAP command is received, this function
+ * will ascertain whether the command is of interest to the network driver and
+ * take necessary action.
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_skb_rx(
+ struct ieee802154_hw *hw,
+ size_t len,
+ u8 *data_ind
+)
+{
+ struct ieee802154_hdr hdr;
+ int msdulen;
+ int hlen;
+ u8 mpdulinkquality = data_ind[23];
+ struct sk_buff *skb;
+ struct ca8210_priv *priv = hw->priv;
+
+ /* Allocate mtu size buffer for every rx packet */
+ skb = dev_alloc_skb(IEEE802154_MTU + sizeof(hdr));
+ if (!skb) {
+ dev_crit(&priv->spi->dev, "dev_alloc_skb failed\n");
+ return -ENOMEM;
+ }
+ skb_reserve(skb, sizeof(hdr));
+
+ msdulen = data_ind[22]; /* msdu_length */
+ if (msdulen > IEEE802154_MTU) {
+ dev_err(
+ &priv->spi->dev,
+ "received erroneously large msdu length!\n"
+ );
+ kfree_skb(skb);
+ return -EMSGSIZE;
+ }
+ dev_dbg(&priv->spi->dev, "skb buffer length = %d\n", msdulen);
+
+ if (priv->promiscuous)
+ goto copy_payload;
+
+ /* Populate hdr */
+ hdr.sec.level = data_ind[29 + msdulen];
+ dev_dbg(&priv->spi->dev, "security level: %#03x\n", hdr.sec.level);
+ if (hdr.sec.level > 0) {
+ hdr.sec.key_id_mode = data_ind[30 + msdulen];
+ memcpy(&hdr.sec.extended_src, &data_ind[31 + msdulen], 8);
+ hdr.sec.key_id = data_ind[39 + msdulen];
+ }
+ hdr.source.mode = data_ind[0];
+ dev_dbg(&priv->spi->dev, "srcAddrMode: %#03x\n", hdr.source.mode);
+ hdr.source.pan_id = *(u16 *)&data_ind[1];
+ dev_dbg(&priv->spi->dev, "srcPanId: %#06x\n", hdr.source.pan_id);
+ memcpy(&hdr.source.extended_addr, &data_ind[3], 8);
+ hdr.dest.mode = data_ind[11];
+ dev_dbg(&priv->spi->dev, "dstAddrMode: %#03x\n", hdr.dest.mode);
+ hdr.dest.pan_id = *(u16 *)&data_ind[12];
+ dev_dbg(&priv->spi->dev, "dstPanId: %#06x\n", hdr.dest.pan_id);
+ memcpy(&hdr.dest.extended_addr, &data_ind[14], 8);
+
+ /* Fill in FC implicitly */
+ hdr.fc.type = 1; /* Data frame */
+ if (hdr.sec.level)
+ hdr.fc.security_enabled = 1;
+ else
+ hdr.fc.security_enabled = 0;
+ if (data_ind[1] != data_ind[12] || data_ind[2] != data_ind[13])
+ hdr.fc.intra_pan = 1;
+ else
+ hdr.fc.intra_pan = 0;
+ hdr.fc.dest_addr_mode = hdr.dest.mode;
+ hdr.fc.source_addr_mode = hdr.source.mode;
+
+ /* Add hdr to front of buffer */
+ hlen = ieee802154_hdr_push(skb, &hdr);
+
+ if (hlen < 0) {
+ dev_crit(&priv->spi->dev, "failed to push mac hdr onto skb!\n");
+ kfree_skb(skb);
+ return hlen;
+ }
+
+ skb_reset_mac_header(skb);
+ skb->mac_len = hlen;
+
+copy_payload:
+ /* Add <msdulen> bytes of space to the back of the buffer */
+ /* Copy msdu to skb */
+ memcpy(skb_put(skb, msdulen), &data_ind[29], msdulen);
+
+ ieee802154_rx_irqsafe(hw, skb, mpdulinkquality);
+ return 0;
+}
+
+/**
+ * ca8210_net_rx() - Acts upon received SAP commands relevant to the network
+ * driver
+ * @hw: ieee802154_hw that command was received by
+ * @command: Octet array of received command
+ * @len: length of the received command
+ *
+ * Called by the spi driver whenever a SAP command is received, this function
+ * will ascertain whether the command is of interest to the network driver and
+ * take necessary action.
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_net_rx(struct ieee802154_hw *hw, u8 *command, size_t len)
+{
+ struct ca8210_priv *priv = hw->priv;
+ unsigned long flags;
+ u8 status;
+
+ dev_dbg(&priv->spi->dev, "ca8210_net_rx(), CmdID = %d\n", command[0]);
+
+ if (command[0] == SPI_MCPS_DATA_INDICATION) {
+ /* Received data */
+ spin_lock_irqsave(&priv->lock, flags);
+ if (command[26] == priv->last_dsn) {
+ dev_dbg(
+ &priv->spi->dev,
+ "DSN %d resend received, ignoring...\n",
+ command[26]
+ );
+ spin_unlock_irqrestore(&priv->lock, flags);
+ return 0;
+ }
+ priv->last_dsn = command[26];
+ spin_unlock_irqrestore(&priv->lock, flags);
+ return ca8210_skb_rx(hw, len - 2, command + 2);
+ } else if (command[0] == SPI_MCPS_DATA_CONFIRM) {
+ status = command[3];
+ if (priv->async_tx_pending) {
+ return ca8210_async_xmit_complete(
+ hw,
+ command[2],
+ status
+ );
+ }
+ }
+
+ return 0;
+}
+
+/**
+ * ca8210_skb_tx() - Transmits a given socket buffer using the ca8210
+ * @skb: Socket buffer to transmit
+ * @msduhandle: Data identifier to pass to the 802.15.4 MAC
+ * @priv: Pointer to private data section of target ca8210
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_skb_tx(
+ struct sk_buff *skb,
+ u8 msduhandle,
+ struct ca8210_priv *priv
+)
+{
+ int status;
+ struct ieee802154_hdr header = { 0 };
+ struct secspec secspec;
+ unsigned int mac_len;
+
+ dev_dbg(&priv->spi->dev, "ca8210_skb_tx() called\n");
+
+ /* Get addressing info from skb - ieee802154 layer creates a full
+ * packet
+ */
+ mac_len = ieee802154_hdr_peek_addrs(skb, &header);
+
+ secspec.security_level = header.sec.level;
+ secspec.key_id_mode = header.sec.key_id_mode;
+ if (secspec.key_id_mode == 2)
+ memcpy(secspec.key_source, &header.sec.short_src, 4);
+ else if (secspec.key_id_mode == 3)
+ memcpy(secspec.key_source, &header.sec.extended_src, 8);
+ secspec.key_index = header.sec.key_id;
+
+ /* Pass to Cascoda API */
+ status = mcps_data_request(
+ header.source.mode,
+ header.dest.mode,
+ header.dest.pan_id,
+ (union macaddr *)&header.dest.extended_addr,
+ skb->len - mac_len,
+ &skb->data[mac_len],
+ msduhandle,
+ header.fc.ack_request,
+ &secspec,
+ priv->spi
+ );
+ return link_to_linux_err(status);
+}
+
+/**
+ * ca8210_start() - Starts the network driver
+ * @hw: ieee802154_hw of ca8210 being started
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_start(struct ieee802154_hw *hw)
+{
+ int status;
+ u8 rx_on_when_idle;
+ u8 lqi_threshold = 0;
+ struct ca8210_priv *priv = hw->priv;
+
+ priv->last_dsn = -1;
+ /* Turn receiver on when idle for now just to test rx */
+ rx_on_when_idle = 1;
+ status = mlme_set_request_sync(
+ MAC_RX_ON_WHEN_IDLE,
+ 0,
+ 1,
+ &rx_on_when_idle,
+ priv->spi
+ );
+ if (status) {
+ dev_crit(
+ &priv->spi->dev,
+ "Setting rx_on_when_idle failed, status = %d\n",
+ status
+ );
+ return link_to_linux_err(status);
+ }
+ status = hwme_set_request_sync(
+ HWME_LQILIMIT,
+ 1,
+ &lqi_threshold,
+ priv->spi
+ );
+ if (status) {
+ dev_crit(
+ &priv->spi->dev,
+ "Setting lqilimit failed, status = %d\n",
+ status
+ );
+ return link_to_linux_err(status);
+ }
+
+ return 0;
+}
+
+/**
+ * ca8210_stop() - Stops the network driver
+ * @hw: ieee802154_hw of ca8210 being stopped
+ *
+ * Return: 0 or linux error code
+ */
+static void ca8210_stop(struct ieee802154_hw *hw)
+{
+}
+
+/**
+ * ca8210_xmit_async() - Asynchronously transmits a given socket buffer using
+ * the ca8210
+ * @hw: ieee802154_hw of ca8210 to transmit from
+ * @skb: Socket buffer to transmit
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_xmit_async(struct ieee802154_hw *hw, struct sk_buff *skb)
+{
+ struct ca8210_priv *priv = hw->priv;
+ int status;
+
+ dev_dbg(&priv->spi->dev, "calling ca8210_xmit_async()\n");
+
+ priv->tx_skb = skb;
+ priv->async_tx_pending = true;
+ status = ca8210_skb_tx(skb, priv->nextmsduhandle, priv);
+ return status;
+}
+
+/**
+ * ca8210_get_ed() - Returns the measured energy on the current channel at this
+ * instant in time
+ * @hw: ieee802154_hw of target ca8210
+ * @level: Measured Energy Detect level
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_get_ed(struct ieee802154_hw *hw, u8 *level)
+{
+ u8 lenvar;
+ struct ca8210_priv *priv = hw->priv;
+
+ return link_to_linux_err(
+ hwme_get_request_sync(HWME_EDVALUE, &lenvar, level, priv->spi)
+ );
+}
+
+/**
+ * ca8210_set_channel() - Sets the current operating 802.15.4 channel of the
+ * ca8210
+ * @hw: ieee802154_hw of target ca8210
+ * @page: Channel page to set
+ * @channel: Channel number to set
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_set_channel(
+ struct ieee802154_hw *hw,
+ u8 page,
+ u8 channel
+)
+{
+ u8 status;
+ struct ca8210_priv *priv = hw->priv;
+
+ status = mlme_set_request_sync(
+ PHY_CURRENT_CHANNEL,
+ 0,
+ 1,
+ &channel,
+ priv->spi
+ );
+ if (status) {
+ dev_err(
+ &priv->spi->dev,
+ "error setting channel, MLME-SET.confirm status = %d\n",
+ status
+ );
+ }
+ return link_to_linux_err(status);
+}
+
+/**
+ * ca8210_set_hw_addr_filt() - Sets the address filtering parameters of the
+ * ca8210
+ * @hw: ieee802154_hw of target ca8210
+ * @filt: Filtering parameters
+ * @changed: Bitmap representing which parameters to change
+ *
+ * Effectively just sets the actual addressing information identifying this node
+ * as all filtering is performed by the ca8210 as detailed in the IEEE 802.15.4
+ * 2006 specification.
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_set_hw_addr_filt(
+ struct ieee802154_hw *hw,
+ struct ieee802154_hw_addr_filt *filt,
+ unsigned long changed
+)
+{
+ u8 status = 0;
+ struct ca8210_priv *priv = hw->priv;
+
+ if (changed & IEEE802154_AFILT_PANID_CHANGED) {
+ status = mlme_set_request_sync(
+ MAC_PAN_ID,
+ 0,
+ 2,
+ &filt->pan_id, priv->spi
+ );
+ if (status) {
+ dev_err(
+ &priv->spi->dev,
+ "error setting pan id, MLME-SET.confirm status = %d",
+ status
+ );
+ return link_to_linux_err(status);
+ }
+ }
+ if (changed & IEEE802154_AFILT_SADDR_CHANGED) {
+ status = mlme_set_request_sync(
+ MAC_SHORT_ADDRESS,
+ 0,
+ 2,
+ &filt->short_addr, priv->spi
+ );
+ if (status) {
+ dev_err(
+ &priv->spi->dev,
+ "error setting short address, MLME-SET.confirm status = %d",
+ status
+ );
+ return link_to_linux_err(status);
+ }
+ }
+ if (changed & IEEE802154_AFILT_IEEEADDR_CHANGED) {
+ status = mlme_set_request_sync(
+ NS_IEEE_ADDRESS,
+ 0,
+ 8,
+ &filt->ieee_addr,
+ priv->spi
+ );
+ if (status) {
+ dev_err(
+ &priv->spi->dev,
+ "error setting ieee address, MLME-SET.confirm status = %d",
+ status
+ );
+ return link_to_linux_err(status);
+ }
+ }
+ /* TODO: Should use MLME_START to set coord bit? */
+ return 0;
+}
+
+/**
+ * ca8210_set_tx_power() - Sets the transmit power of the ca8210
+ * @hw: ieee802154_hw of target ca8210
+ * @mbm: Transmit power in mBm (dBm*100)
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_set_tx_power(struct ieee802154_hw *hw, s32 mbm)
+{
+ struct ca8210_priv *priv = hw->priv;
+
+ mbm /= 100;
+ return link_to_linux_err(
+ mlme_set_request_sync(PHY_TRANSMIT_POWER, 0, 1, &mbm, priv->spi)
+ );
+}
+
+/**
+ * ca8210_set_cca_mode() - Sets the clear channel assessment mode of the ca8210
+ * @hw: ieee802154_hw of target ca8210
+ * @cca: CCA mode to set
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_set_cca_mode(
+ struct ieee802154_hw *hw,
+ const struct wpan_phy_cca *cca
+)
+{
+ u8 status;
+ u8 cca_mode;
+ struct ca8210_priv *priv = hw->priv;
+
+ cca_mode = cca->mode & 3;
+ if (cca_mode == 3 && cca->opt == NL802154_CCA_OPT_ENERGY_CARRIER_OR) {
+ /* cca_mode 0 == CS OR ED, 3 == CS AND ED */
+ cca_mode = 0;
+ }
+ status = mlme_set_request_sync(
+ PHY_CCA_MODE,
+ 0,
+ 1,
+ &cca_mode,
+ priv->spi
+ );
+ if (status) {
+ dev_err(
+ &priv->spi->dev,
+ "error setting cca mode, MLME-SET.confirm status = %d",
+ status
+ );
+ }
+ return link_to_linux_err(status);
+}
+
+/**
+ * ca8210_set_cca_ed_level() - Sets the CCA ED level of the ca8210
+ * @hw: ieee802154_hw of target ca8210
+ * @level: ED level to set (in mbm)
+ *
+ * Sets the minimum threshold of measured energy above which the ca8210 will
+ * back off and retry a transmission.
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_set_cca_ed_level(struct ieee802154_hw *hw, s32 level)
+{
+ u8 status;
+ u8 ed_threshold = (level / 100) * 2 + 256;
+ struct ca8210_priv *priv = hw->priv;
+
+ status = hwme_set_request_sync(
+ HWME_EDTHRESHOLD,
+ 1,
+ &ed_threshold,
+ priv->spi
+ );
+ if (status) {
+ dev_err(
+ &priv->spi->dev,
+ "error setting ed threshold, HWME-SET.confirm status = %d",
+ status
+ );
+ }
+ return link_to_linux_err(status);
+}
+
+/**
+ * ca8210_set_csma_params() - Sets the CSMA parameters of the ca8210
+ * @hw: ieee802154_hw of target ca8210
+ * @min_be: Minimum backoff exponent when backing off a transmission
+ * @max_be: Maximum backoff exponent when backing off a transmission
+ * @retries: Number of times to retry after backing off
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_set_csma_params(
+ struct ieee802154_hw *hw,
+ u8 min_be,
+ u8 max_be,
+ u8 retries
+)
+{
+ u8 status;
+ struct ca8210_priv *priv = hw->priv;
+
+ status = mlme_set_request_sync(MAC_MIN_BE, 0, 1, &min_be, priv->spi);
+ if (status) {
+ dev_err(
+ &priv->spi->dev,
+ "error setting min be, MLME-SET.confirm status = %d",
+ status
+ );
+ return link_to_linux_err(status);
+ }
+ status = mlme_set_request_sync(MAC_MAX_BE, 0, 1, &max_be, priv->spi);
+ if (status) {
+ dev_err(
+ &priv->spi->dev,
+ "error setting max be, MLME-SET.confirm status = %d",
+ status
+ );
+ return link_to_linux_err(status);
+ }
+ status = mlme_set_request_sync(
+ MAC_MAX_CSMA_BACKOFFS,
+ 0,
+ 1,
+ &retries,
+ priv->spi
+ );
+ if (status) {
+ dev_err(
+ &priv->spi->dev,
+ "error setting max csma backoffs, MLME-SET.confirm status = %d",
+ status
+ );
+ }
+ return link_to_linux_err(status);
+}
+
+/**
+ * ca8210_set_frame_retries() - Sets the maximum frame retries of the ca8210
+ * @hw: ieee802154_hw of target ca8210
+ * @retries: Number of retries
+ *
+ * Sets the number of times to retry a transmission if no acknowledgment was
+ * was received from the other end when one was requested.
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_set_frame_retries(struct ieee802154_hw *hw, s8 retries)
+{
+ u8 status;
+ struct ca8210_priv *priv = hw->priv;
+
+ status = mlme_set_request_sync(
+ MAC_MAX_FRAME_RETRIES,
+ 0,
+ 1,
+ &retries,
+ priv->spi
+ );
+ if (status) {
+ dev_err(
+ &priv->spi->dev,
+ "error setting frame retries, MLME-SET.confirm status = %d",
+ status
+ );
+ }
+ return link_to_linux_err(status);
+}
+
+static int ca8210_set_promiscuous_mode(struct ieee802154_hw *hw, const bool on)
+{
+ u8 status;
+ struct ca8210_priv *priv = hw->priv;
+
+ status = mlme_set_request_sync(
+ MAC_PROMISCUOUS_MODE,
+ 0,
+ 1,
+ (const void*)&on,
+ priv->spi
+ );
+ if (status) {
+ dev_err(
+ &priv->spi->dev,
+ "error setting promiscuous mode, MLME-SET.confirm status = %d",
+ status
+ );
+ } else {
+ priv->promiscuous = on;
+ }
+ return link_to_linux_err(status);
+}
+
+static const struct ieee802154_ops ca8210_phy_ops = {
+ .start = ca8210_start,
+ .stop = ca8210_stop,
+ .xmit_async = ca8210_xmit_async,
+ .ed = ca8210_get_ed,
+ .set_channel = ca8210_set_channel,
+ .set_hw_addr_filt = ca8210_set_hw_addr_filt,
+ .set_txpower = ca8210_set_tx_power,
+ .set_cca_mode = ca8210_set_cca_mode,
+ .set_cca_ed_level = ca8210_set_cca_ed_level,
+ .set_csma_params = ca8210_set_csma_params,
+ .set_frame_retries = ca8210_set_frame_retries,
+ .set_promiscuous_mode = ca8210_set_promiscuous_mode
+};
+
+/* Test/EVBME Interface */
+
+/**
+ * ca8210_test_int_open() - Opens the test interface to the userspace
+ * @inodp: inode representation of file interface
+ * @filp: file interface
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_test_int_open(struct inode *inodp, struct file *filp)
+{
+ struct ca8210_priv *priv = inodp->i_private;
+
+ filp->private_data = priv;
+ return 0;
+}
+
+/**
+ * ca8210_test_check_upstream() - Checks a command received from the upstream
+ * testing interface for required action
+ * @buf: Buffer containing command to check
+ * @device_ref: Nondescript pointer to target device
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_test_check_upstream(u8 *buf, void *device_ref)
+{
+ int ret;
+ u8 response[CA8210_SPI_BUF_SIZE];
+
+ if (buf[0] == SPI_MLME_SET_REQUEST) {
+ ret = tdme_checkpibattribute(buf[2], buf[4], buf + 5);
+ if (ret) {
+ response[0] = SPI_MLME_SET_CONFIRM;
+ response[1] = 3;
+ response[2] = MAC_INVALID_PARAMETER;
+ response[3] = buf[2];
+ response[4] = buf[3];
+ if (cascoda_api_upstream)
+ cascoda_api_upstream(response, 5, device_ref);
+ return ret;
+ }
+ }
+ if (buf[0] == SPI_MLME_ASSOCIATE_REQUEST) {
+ return tdme_channelinit(buf[2], device_ref);
+ } else if (buf[0] == SPI_MLME_START_REQUEST) {
+ return tdme_channelinit(buf[4], device_ref);
+ } else if (
+ (buf[0] == SPI_MLME_SET_REQUEST) &&
+ (buf[2] == PHY_CURRENT_CHANNEL)
+ ) {
+ return tdme_channelinit(buf[5], device_ref);
+ } else if (
+ (buf[0] == SPI_TDME_SET_REQUEST) &&
+ (buf[2] == TDME_CHANNEL)
+ ) {
+ return tdme_channelinit(buf[4], device_ref);
+ } else if (
+ (CA8210_MAC_WORKAROUNDS) &&
+ (buf[0] == SPI_MLME_RESET_REQUEST) &&
+ (buf[2] == 1)
+ ) {
+ /* reset COORD Bit for Channel Filtering as Coordinator */
+ return tdme_setsfr_request_sync(
+ 0,
+ CA8210_SFR_MACCON,
+ 0,
+ device_ref
+ );
+ }
+ return 0;
+} /* End of EVBMECheckSerialCommand() */
+
+/**
+ * ca8210_test_int_user_write() - Called by a process in userspace to send a
+ * message to the ca8210 drivers
+ * @filp: file interface
+ * @in_buf: Buffer containing message to write
+ * @len: length of message
+ * @off: file offset
+ *
+ * Return: 0 or linux error code
+ */
+static ssize_t ca8210_test_int_user_write(
+ struct file *filp,
+ const char __user *in_buf,
+ size_t len,
+ loff_t *off
+)
+{
+ int ret;
+ struct ca8210_priv *priv = filp->private_data;
+ u8 command[CA8210_SPI_BUF_SIZE];
+
+ if (len > CA8210_SPI_BUF_SIZE) {
+ dev_warn(
+ &priv->spi->dev,
+ "userspace requested erroneously long write (%zu)\n",
+ len
+ );
+ return -EMSGSIZE;
+ }
+
+ ret = copy_from_user(command, in_buf, len);
+ if (ret) {
+ dev_err(
+ &priv->spi->dev,
+ "%d bytes could not be copied from userspace\n",
+ ret
+ );
+ return -EIO;
+ }
+
+ ret = ca8210_test_check_upstream(command, priv->spi);
+ if (ret == 0) {
+ ret = ca8210_spi_exchange(
+ command,
+ command[1] + 2,
+ NULL,
+ priv->spi
+ );
+ if (ret < 0) {
+ /* effectively 0 bytes were written successfully */
+ dev_err(
+ &priv->spi->dev,
+ "spi exchange failed\n"
+ );
+ return ret;
+ }
+ if (command[0] & SPI_SYN)
+ priv->sync_down++;
+ }
+
+ return len;
+}
+
+/**
+ * ca8210_test_int_user_read() - Called by a process in userspace to read a
+ * message from the ca8210 drivers
+ * @filp: file interface
+ * @buf: Buffer to write message to
+ * @len: length of message to read (ignored)
+ * @offp: file offset
+ *
+ * If the O_NONBLOCK flag was set when opening the file then this function will
+ * not block, i.e. it will return if the fifo is empty. Otherwise the function
+ * will block, i.e. wait until new data arrives.
+ *
+ * Return: number of bytes read
+ */
+static ssize_t ca8210_test_int_user_read(
+ struct file *filp,
+ char __user *buf,
+ size_t len,
+ loff_t *offp
+)
+{
+ int i, cmdlen;
+ struct ca8210_priv *priv = filp->private_data;
+ unsigned char *fifo_buffer;
+ unsigned long bytes_not_copied;
+
+ if (filp->f_flags & O_NONBLOCK) {
+ /* Non-blocking mode */
+ if (kfifo_is_empty(&priv->test.up_fifo))
+ return 0;
+ } else {
+ /* Blocking mode */
+ wait_event_interruptible(
+ priv->test.readq,
+ !kfifo_is_empty(&priv->test.up_fifo)
+ );
+ }
+
+ if (kfifo_out(&priv->test.up_fifo, &fifo_buffer, 4) != 4) {
+ dev_err(
+ &priv->spi->dev,
+ "test_interface: Wrong number of elements popped from upstream fifo\n"
+ );
+ return 0;
+ }
+ cmdlen = fifo_buffer[1];
+ bytes_not_copied = cmdlen + 2;
+
+ bytes_not_copied = copy_to_user(buf, fifo_buffer, bytes_not_copied);
+ if (bytes_not_copied > 0) {
+ dev_err(
+ &priv->spi->dev,
+ "%lu bytes could not be copied to user space!\n",
+ bytes_not_copied
+ );
+ }
+
+ dev_dbg(&priv->spi->dev, "test_interface: Cmd len = %d\n", cmdlen);
+
+ dev_dbg(&priv->spi->dev, "test_interface: Read\n");
+ for (i = 0; i < cmdlen + 2; i++)
+ dev_dbg(&priv->spi->dev, "%#03x\n", fifo_buffer[i]);
+
+ kfree(fifo_buffer);
+
+ return cmdlen + 2;
+}
+
+/**
+ * ca8210_test_int_ioctl() - Called by a process in userspace to enact an
+ * arbitrary action
+ * @filp: file interface
+ * @ioctl_num: which action to enact
+ * @ioctl_param: arbitrary parameter for the action
+ *
+ * Return: status
+ */
+static long ca8210_test_int_ioctl(
+ struct file *filp,
+ unsigned int ioctl_num,
+ unsigned long ioctl_param
+)
+{
+ struct ca8210_priv *priv = filp->private_data;
+
+ switch (ioctl_num) {
+ case CA8210_IOCTL_HARD_RESET:
+ ca8210_reset_send(priv->spi, ioctl_param);
+ break;
+ default:
+ break;
+ }
+ return 0;
+}
+
+/**
+ * ca8210_test_int_poll() - Called by a process in userspace to determine which
+ * actions are currently possible for the file
+ * @filp: file interface
+ * @ptable: poll table
+ *
+ * Return: set of poll return flags
+ */
+static unsigned int ca8210_test_int_poll(
+ struct file *filp,
+ struct poll_table_struct *ptable
+)
+{
+ unsigned int return_flags = 0;
+ struct ca8210_priv *priv = filp->private_data;
+
+ poll_wait(filp, &priv->test.readq, ptable);
+ if (!kfifo_is_empty(&priv->test.up_fifo))
+ return_flags |= (POLLIN | POLLRDNORM);
+ if (wait_event_interruptible(
+ priv->test.readq,
+ !kfifo_is_empty(&priv->test.up_fifo))) {
+ return POLLERR;
+ }
+ return return_flags;
+}
+
+static const struct file_operations test_int_fops = {
+ .read = ca8210_test_int_user_read,
+ .write = ca8210_test_int_user_write,
+ .open = ca8210_test_int_open,
+ .release = NULL,
+ .unlocked_ioctl = ca8210_test_int_ioctl,
+ .poll = ca8210_test_int_poll
+};
+
+/* Init/Deinit */
+
+/**
+ * ca8210_get_platform_data() - Populate a ca8210_platform_data object
+ * @spi_device: Pointer to ca8210 spi device object to get data for
+ * @pdata: Pointer to ca8210_platform_data object to populate
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_get_platform_data(
+ struct spi_device *spi_device,
+ struct ca8210_platform_data *pdata
+)
+{
+ int ret = 0;
+
+ if (!spi_device->dev.of_node)
+ return -EINVAL;
+
+ pdata->extclockenable = of_property_read_bool(
+ spi_device->dev.of_node,
+ "extclock-enable"
+ );
+ if (pdata->extclockenable) {
+ ret = of_property_read_u32(
+ spi_device->dev.of_node,
+ "extclock-freq",
+ &pdata->extclockfreq
+ );
+ if (ret < 0)
+ return ret;
+
+ ret = of_property_read_u32(
+ spi_device->dev.of_node,
+ "extclock-gpio",
+ &pdata->extclockgpio
+ );
+ }
+
+ return ret;
+}
+
+/**
+ * ca8210_config_extern_clk() - Configure the external clock provided by the
+ * ca8210
+ * @pdata: Pointer to ca8210_platform_data containing clock parameters
+ * @spi: Pointer to target ca8210 spi device
+ * @on: True to turn the clock on, false to turn off
+ *
+ * The external clock is configured with a frequency and output pin taken from
+ * the platform data.
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_config_extern_clk(
+ struct ca8210_platform_data *pdata,
+ struct spi_device *spi,
+ bool on
+)
+{
+ u8 clkparam[2];
+
+ if (on) {
+ dev_info(&spi->dev, "Switching external clock on\n");
+ switch (pdata->extclockfreq) {
+ case SIXTEEN_MHZ:
+ clkparam[0] = 1;
+ break;
+ case EIGHT_MHZ:
+ clkparam[0] = 2;
+ break;
+ case FOUR_MHZ:
+ clkparam[0] = 3;
+ break;
+ case TWO_MHZ:
+ clkparam[0] = 4;
+ break;
+ case ONE_MHZ:
+ clkparam[0] = 5;
+ break;
+ default:
+ dev_crit(&spi->dev, "Invalid extclock-freq\n");
+ return -EINVAL;
+ }
+ clkparam[1] = pdata->extclockgpio;
+ } else {
+ dev_info(&spi->dev, "Switching external clock off\n");
+ clkparam[0] = 0; /* off */
+ clkparam[1] = 0;
+ }
+ return link_to_linux_err(
+ hwme_set_request_sync(HWME_SYSCLKOUT, 2, clkparam, spi)
+ );
+}
+
+/**
+ * ca8210_register_ext_clock() - Register ca8210's external clock with kernel
+ * @spi: Pointer to target ca8210 spi device
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_register_ext_clock(struct spi_device *spi)
+{
+ struct device_node *np = spi->dev.of_node;
+ struct ca8210_priv *priv = spi_get_drvdata(spi);
+ struct ca8210_platform_data *pdata = spi->dev.platform_data;
+ int ret = 0;
+
+ if (!np)
+ return -EFAULT;
+
+ priv->clk = clk_register_fixed_rate(
+ &spi->dev,
+ np->name,
+ NULL,
+ 0,
+ pdata->extclockfreq
+ );
+
+ if (IS_ERR(priv->clk)) {
+ dev_crit(&spi->dev, "Failed to register external clk\n");
+ return PTR_ERR(priv->clk);
+ }
+ ret = of_clk_add_provider(np, of_clk_src_simple_get, priv->clk);
+ if (ret) {
+ clk_unregister(priv->clk);
+ dev_crit(
+ &spi->dev,
+ "Failed to register external clock as clock provider\n"
+ );
+ } else {
+ dev_info(&spi->dev, "External clock set as clock provider\n");
+ }
+
+ return ret;
+}
+
+/**
+ * ca8210_unregister_ext_clock() - Unregister ca8210's external clock with
+ * kernel
+ * @spi: Pointer to target ca8210 spi device
+ */
+static void ca8210_unregister_ext_clock(struct spi_device *spi)
+{
+ struct ca8210_priv *priv = spi_get_drvdata(spi);
+
+ if (!priv->clk)
+ return
+
+ of_clk_del_provider(spi->dev.of_node);
+ clk_unregister(priv->clk);
+ dev_info(&spi->dev, "External clock unregistered\n");
+}
+
+/**
+ * ca8210_reset_init() - Initialise the reset input to the ca8210
+ * @spi: Pointer to target ca8210 spi device
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_reset_init(struct spi_device *spi)
+{
+ int ret;
+ struct ca8210_platform_data *pdata = spi->dev.platform_data;
+
+ pdata->gpio_reset = of_get_named_gpio(
+ spi->dev.of_node,
+ "reset-gpio",
+ 0
+ );
+
+ ret = gpio_direction_output(pdata->gpio_reset, 1);
+ if (ret < 0) {
+ dev_crit(
+ &spi->dev,
+ "Reset GPIO %d did not set to output mode\n",
+ pdata->gpio_reset
+ );
+ }
+
+ return ret;
+}
+
+/**
+ * ca8210_interrupt_init() - Initialise the irq output from the ca8210
+ * @spi: Pointer to target ca8210 spi device
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_interrupt_init(struct spi_device *spi)
+{
+ int ret;
+ struct ca8210_platform_data *pdata = spi->dev.platform_data;
+
+ pdata->gpio_irq = of_get_named_gpio(
+ spi->dev.of_node,
+ "irq-gpio",
+ 0
+ );
+
+ pdata->irq_id = gpio_to_irq(pdata->gpio_irq);
+ if (pdata->irq_id < 0) {
+ dev_crit(
+ &spi->dev,
+ "Could not get irq for gpio pin %d\n",
+ pdata->gpio_irq
+ );
+ gpio_free(pdata->gpio_irq);
+ return pdata->irq_id;
+ }
+
+ ret = request_irq(
+ pdata->irq_id,
+ ca8210_interrupt_handler,
+ IRQF_TRIGGER_FALLING,
+ "ca8210-irq",
+ spi_get_drvdata(spi)
+ );
+ if (ret) {
+ dev_crit(&spi->dev, "request_irq %d failed\n", pdata->irq_id);
+ gpio_unexport(pdata->gpio_irq);
+ gpio_free(pdata->gpio_irq);
+ }
+
+ return ret;
+}
+
+/**
+ * ca8210_dev_com_init() - Initialise the spi communication component
+ * @priv: Pointer to private data structure
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_dev_com_init(struct ca8210_priv *priv)
+{
+ priv->mlme_workqueue = alloc_ordered_workqueue(
+ "MLME work queue",
+ WQ_UNBOUND
+ );
+ if (!priv->mlme_workqueue) {
+ dev_crit(&priv->spi->dev, "alloc of mlme_workqueue failed!\n");
+ return -ENOMEM;
+ }
+
+ priv->irq_workqueue = alloc_ordered_workqueue(
+ "ca8210 irq worker",
+ WQ_UNBOUND
+ );
+ if (!priv->irq_workqueue) {
+ dev_crit(&priv->spi->dev, "alloc of irq_workqueue failed!\n");
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+/**
+ * ca8210_dev_com_clear() - Deinitialise the spi communication component
+ * @priv: Pointer to private data structure
+ */
+static void ca8210_dev_com_clear(struct ca8210_priv *priv)
+{
+ flush_workqueue(priv->mlme_workqueue);
+ destroy_workqueue(priv->mlme_workqueue);
+ flush_workqueue(priv->irq_workqueue);
+ destroy_workqueue(priv->irq_workqueue);
+}
+
+#define CA8210_MAX_TX_POWERS (9)
+static const s32 ca8210_tx_powers[CA8210_MAX_TX_POWERS] = {
+ 800, 700, 600, 500, 400, 300, 200, 100, 0
+};
+
+#define CA8210_MAX_ED_LEVELS (21)
+static const s32 ca8210_ed_levels[CA8210_MAX_ED_LEVELS] = {
+ -10300, -10250, -10200, -10150, -10100, -10050, -10000, -9950, -9900,
+ -9850, -9800, -9750, -9700, -9650, -9600, -9550, -9500, -9450, -9400,
+ -9350, -9300
+};
+
+/**
+ * ca8210_hw_setup() - Populate the ieee802154_hw phy attributes with the
+ * ca8210's defaults
+ * @ca8210_hw: Pointer to ieee802154_hw to populate
+ */
+static void ca8210_hw_setup(struct ieee802154_hw *ca8210_hw)
+{
+ /* Support channels 11-26 */
+ ca8210_hw->phy->supported.channels[0] = CA8210_VALID_CHANNELS;
+ ca8210_hw->phy->supported.tx_powers_size = CA8210_MAX_TX_POWERS;
+ ca8210_hw->phy->supported.tx_powers = ca8210_tx_powers;
+ ca8210_hw->phy->supported.cca_ed_levels_size = CA8210_MAX_ED_LEVELS;
+ ca8210_hw->phy->supported.cca_ed_levels = ca8210_ed_levels;
+ ca8210_hw->phy->current_channel = 18;
+ ca8210_hw->phy->current_page = 0;
+ ca8210_hw->phy->transmit_power = 800;
+ ca8210_hw->phy->cca.mode = NL802154_CCA_ENERGY_CARRIER;
+ ca8210_hw->phy->cca.opt = NL802154_CCA_OPT_ENERGY_CARRIER_AND;
+ ca8210_hw->phy->cca_ed_level = -9800;
+ ca8210_hw->phy->symbol_duration = 16;
+ ca8210_hw->phy->lifs_period = 40;
+ ca8210_hw->phy->sifs_period = 12;
+ ca8210_hw->flags =
+ IEEE802154_HW_AFILT |
+ IEEE802154_HW_OMIT_CKSUM |
+ IEEE802154_HW_FRAME_RETRIES |
+ IEEE802154_HW_PROMISCUOUS |
+ IEEE802154_HW_CSMA_PARAMS;
+ ca8210_hw->phy->flags =
+ WPAN_PHY_FLAG_TXPOWER |
+ WPAN_PHY_FLAG_CCA_ED_LEVEL |
+ WPAN_PHY_FLAG_CCA_MODE;
+}
+
+/**
+ * ca8210_test_interface_init() - Initialise the test file interface
+ * @priv: Pointer to private data structure
+ *
+ * Provided as an alternative to the standard linux network interface, the test
+ * interface exposes a file in the filesystem (ca8210_test) that allows
+ * 802.15.4 SAP Commands and Cascoda EVBME commands to be sent directly to
+ * the stack.
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_test_interface_init(struct ca8210_priv *priv)
+{
+ struct ca8210_test *test = &priv->test;
+ char node_name[32];
+
+ snprintf(
+ node_name,
+ sizeof(node_name),
+ "ca8210@%d_%d",
+ priv->spi->master->bus_num,
+ priv->spi->chip_select
+ );
+
+ test->ca8210_dfs_spi_int = debugfs_create_file(
+ node_name,
+ 0600, /* S_IRUSR | S_IWUSR */
+ NULL,
+ priv,
+ &test_int_fops
+ );
+ if (IS_ERR(test->ca8210_dfs_spi_int)) {
+ dev_err(
+ &priv->spi->dev,
+ "Error %ld when creating debugfs node\n",
+ PTR_ERR(test->ca8210_dfs_spi_int)
+ );
+ return PTR_ERR(test->ca8210_dfs_spi_int);
+ }
+ debugfs_create_symlink("ca8210", NULL, node_name);
+ init_waitqueue_head(&test->readq);
+ return kfifo_alloc(
+ &test->up_fifo,
+ CA8210_TEST_INT_FIFO_SIZE,
+ GFP_KERNEL
+ );
+}
+
+/**
+ * ca8210_test_interface_clear() - Deinitialise the test file interface
+ * @priv: Pointer to private data structure
+ */
+static void ca8210_test_interface_clear(struct ca8210_priv *priv)
+{
+ struct ca8210_test *test = &priv->test;
+
+ if (!IS_ERR(test->ca8210_dfs_spi_int))
+ debugfs_remove(test->ca8210_dfs_spi_int);
+ kfifo_free(&test->up_fifo);
+ dev_info(&priv->spi->dev, "Test interface removed\n");
+}
+
+/**
+ * ca8210_remove() - Shut down a ca8210 upon being disconnected
+ * @priv: Pointer to private data structure
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_remove(struct spi_device *spi_device)
+{
+ struct ca8210_priv *priv;
+ struct ca8210_platform_data *pdata;
+
+ dev_info(&spi_device->dev, "Removing ca8210\n");
+
+ pdata = spi_device->dev.platform_data;
+ if (pdata) {
+ if (pdata->extclockenable) {
+ ca8210_unregister_ext_clock(spi_device);
+ ca8210_config_extern_clk(pdata, spi_device, 0);
+ }
+ free_irq(pdata->irq_id, spi_device->dev.driver_data);
+ kfree(pdata);
+ spi_device->dev.platform_data = NULL;
+ }
+ /* get spi_device private data */
+ priv = spi_get_drvdata(spi_device);
+ if (priv) {
+ dev_info(
+ &spi_device->dev,
+ "sync_down = %d, sync_up = %d\n",
+ priv->sync_down,
+ priv->sync_up
+ );
+ ca8210_dev_com_clear(spi_device->dev.driver_data);
+ if (priv->hw) {
+ if (priv->hw_registered)
+ ieee802154_unregister_hw(priv->hw);
+ ieee802154_free_hw(priv->hw);
+ priv->hw = NULL;
+ dev_info(
+ &spi_device->dev,
+ "Unregistered & freed ieee802154_hw.\n"
+ );
+ }
+ if (IS_ENABLED(CONFIG_IEEE802154_CA8210_DEBUGFS))
+ ca8210_test_interface_clear(priv);
+ }
+
+ return 0;
+}
+
+/**
+ * ca8210_probe() - Set up a connected ca8210 upon being detected by the system
+ * @priv: Pointer to private data structure
+ *
+ * Return: 0 or linux error code
+ */
+static int ca8210_probe(struct spi_device *spi_device)
+{
+ struct ca8210_priv *priv;
+ struct ieee802154_hw *hw;
+ struct ca8210_platform_data *pdata;
+ int ret;
+
+ dev_info(&spi_device->dev, "Inserting ca8210\n");
+
+ /* allocate ieee802154_hw and private data */
+ hw = ieee802154_alloc_hw(sizeof(struct ca8210_priv), &ca8210_phy_ops);
+ if (!hw) {
+ dev_crit(&spi_device->dev, "ieee802154_alloc_hw failed\n");
+ ret = -ENOMEM;
+ goto error;
+ }
+
+ priv = hw->priv;
+ priv->hw = hw;
+ priv->spi = spi_device;
+ hw->parent = &spi_device->dev;
+ spin_lock_init(&priv->lock);
+ priv->async_tx_pending = false;
+ priv->hw_registered = false;
+ priv->sync_up = 0;
+ priv->sync_down = 0;
+ priv->promiscuous = false;
+ priv->retries = 0;
+ init_completion(&priv->ca8210_is_awake);
+ init_completion(&priv->spi_transfer_complete);
+ init_completion(&priv->sync_exchange_complete);
+ spi_set_drvdata(priv->spi, priv);
+ if (IS_ENABLED(CONFIG_IEEE802154_CA8210_DEBUGFS)) {
+ cascoda_api_upstream = ca8210_test_int_driver_write;
+ ca8210_test_interface_init(priv);
+ } else {
+ cascoda_api_upstream = NULL;
+ }
+ ca8210_hw_setup(hw);
+ ieee802154_random_extended_addr(&hw->phy->perm_extended_addr);
+
+ pdata = kmalloc(sizeof(*pdata), GFP_KERNEL);
+ if (!pdata) {
+ dev_crit(
+ &spi_device->dev,
+ "Could not allocate platform data\n"
+ );
+ ret = -ENOMEM;
+ goto error;
+ }
+
+ ret = ca8210_get_platform_data(priv->spi, pdata);
+ if (ret) {
+ dev_crit(&spi_device->dev, "ca8210_get_platform_data failed\n");
+ goto error;
+ }
+ priv->spi->dev.platform_data = pdata;
+
+ ret = ca8210_dev_com_init(priv);
+ if (ret) {
+ dev_crit(&spi_device->dev, "ca8210_dev_com_init failed\n");
+ goto error;
+ }
+ ret = ca8210_reset_init(priv->spi);
+ if (ret) {
+ dev_crit(&spi_device->dev, "ca8210_reset_init failed\n");
+ goto error;
+ }
+
+ ret = ca8210_interrupt_init(priv->spi);
+ if (ret) {
+ dev_crit(&spi_device->dev, "ca8210_interrupt_init failed\n");
+ goto error;
+ }
+
+ msleep(100);
+
+ ca8210_reset_send(priv->spi, 1);
+
+ ret = tdme_chipinit(priv->spi);
+ if (ret) {
+ dev_crit(&spi_device->dev, "tdme_chipinit failed\n");
+ goto error;
+ }
+
+ if (pdata->extclockenable) {
+ ret = ca8210_config_extern_clk(pdata, priv->spi, 1);
+ if (ret) {
+ dev_crit(
+ &spi_device->dev,
+ "ca8210_config_extern_clk failed\n"
+ );
+ goto error;
+ }
+ ret = ca8210_register_ext_clock(priv->spi);
+ if (ret) {
+ dev_crit(
+ &spi_device->dev,
+ "ca8210_register_ext_clock failed\n"
+ );
+ goto error;
+ }
+ }
+
+ ret = ieee802154_register_hw(hw);
+ if (ret) {
+ dev_crit(&spi_device->dev, "ieee802154_register_hw failed\n");
+ goto error;
+ }
+ priv->hw_registered = true;
+
+ return 0;
+error:
+ msleep(100); /* wait for pending spi transfers to complete */
+ ca8210_remove(spi_device);
+ return link_to_linux_err(ret);
+}
+
+static const struct of_device_id ca8210_of_ids[] = {
+ {.compatible = "cascoda,ca8210", },
+ {},
+};
+MODULE_DEVICE_TABLE(of, ca8210_of_ids);
+
+static struct spi_driver ca8210_spi_driver = {
+ .driver = {
+ .name = DRIVER_NAME,
+ .owner = THIS_MODULE,
+ .of_match_table = of_match_ptr(ca8210_of_ids),
+ },
+ .probe = ca8210_probe,
+ .remove = ca8210_remove
+};
+
+module_spi_driver(ca8210_spi_driver);
+
+MODULE_AUTHOR("Harry Morris <h.morris@cascoda.com>");
+MODULE_DESCRIPTION("CA-8210 SoftMAC driver");
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_VERSION("1.0");
diff --git a/drivers/net/ieee802154/mrf24j40.c b/drivers/net/ieee802154/mrf24j40.c
index 7b131f8..bd63289 100644
--- a/drivers/net/ieee802154/mrf24j40.c
+++ b/drivers/net/ieee802154/mrf24j40.c
@@ -18,6 +18,7 @@
#include <linux/spi/spi.h>
#include <linux/interrupt.h>
#include <linux/module.h>
+#include <linux/of.h>
#include <linux/regmap.h>
#include <linux/ieee802154.h>
#include <linux/irq.h>
diff --git a/drivers/net/ipvlan/ipvlan.h b/drivers/net/ipvlan/ipvlan.h
index 800a46c..7919369 100644
--- a/drivers/net/ipvlan/ipvlan.h
+++ b/drivers/net/ipvlan/ipvlan.h
@@ -26,6 +26,7 @@
#include <linux/netfilter.h>
#include <net/ip.h>
#include <net/ip6_route.h>
+#include <net/netns/generic.h>
#include <net/rtnetlink.h>
#include <net/route.h>
#include <net/addrconf.h>
@@ -91,6 +92,7 @@
struct ipvl_port {
struct net_device *dev;
+ possible_net_t pnet;
struct hlist_head hlhead[IPVLAN_HASH_SIZE];
struct list_head ipvlans;
u16 mode;
diff --git a/drivers/net/ipvlan/ipvlan_main.c b/drivers/net/ipvlan/ipvlan_main.c
index aa8575c..618ed88 100644
--- a/drivers/net/ipvlan/ipvlan_main.c
+++ b/drivers/net/ipvlan/ipvlan_main.c
@@ -9,7 +9,11 @@
#include "ipvlan.h"
-static u32 ipvl_nf_hook_refcnt = 0;
+static unsigned int ipvlan_netid __read_mostly;
+
+struct ipvlan_netns {
+ unsigned int ipvl_nf_hook_refcnt;
+};
static struct nf_hook_ops ipvl_nfops[] __read_mostly = {
{
@@ -35,28 +39,34 @@
ipvlan->dev->mtu = dev->mtu;
}
-static int ipvlan_register_nf_hook(void)
+static int ipvlan_register_nf_hook(struct net *net)
{
+ struct ipvlan_netns *vnet = net_generic(net, ipvlan_netid);
int err = 0;
- if (!ipvl_nf_hook_refcnt) {
- err = _nf_register_hooks(ipvl_nfops, ARRAY_SIZE(ipvl_nfops));
+ if (!vnet->ipvl_nf_hook_refcnt) {
+ err = nf_register_net_hooks(net, ipvl_nfops,
+ ARRAY_SIZE(ipvl_nfops));
if (!err)
- ipvl_nf_hook_refcnt = 1;
+ vnet->ipvl_nf_hook_refcnt = 1;
} else {
- ipvl_nf_hook_refcnt++;
+ vnet->ipvl_nf_hook_refcnt++;
}
return err;
}
-static void ipvlan_unregister_nf_hook(void)
+static void ipvlan_unregister_nf_hook(struct net *net)
{
- WARN_ON(!ipvl_nf_hook_refcnt);
+ struct ipvlan_netns *vnet = net_generic(net, ipvlan_netid);
- ipvl_nf_hook_refcnt--;
- if (!ipvl_nf_hook_refcnt)
- _nf_unregister_hooks(ipvl_nfops, ARRAY_SIZE(ipvl_nfops));
+ if (WARN_ON(!vnet->ipvl_nf_hook_refcnt))
+ return;
+
+ vnet->ipvl_nf_hook_refcnt--;
+ if (!vnet->ipvl_nf_hook_refcnt)
+ nf_unregister_net_hooks(net, ipvl_nfops,
+ ARRAY_SIZE(ipvl_nfops));
}
static int ipvlan_set_port_mode(struct ipvl_port *port, u16 nval)
@@ -69,7 +79,7 @@
if (port->mode != nval) {
if (nval == IPVLAN_MODE_L3S) {
/* New mode is L3S */
- err = ipvlan_register_nf_hook();
+ err = ipvlan_register_nf_hook(read_pnet(&port->pnet));
if (!err) {
mdev->l3mdev_ops = &ipvl_l3mdev_ops;
mdev->priv_flags |= IFF_L3MDEV_MASTER;
@@ -78,7 +88,7 @@
} else if (port->mode == IPVLAN_MODE_L3S) {
/* Old mode was L3S */
mdev->priv_flags &= ~IFF_L3MDEV_MASTER;
- ipvlan_unregister_nf_hook();
+ ipvlan_unregister_nf_hook(read_pnet(&port->pnet));
mdev->l3mdev_ops = NULL;
}
list_for_each_entry(ipvlan, &port->ipvlans, pnode) {
@@ -111,6 +121,7 @@
if (!port)
return -ENOMEM;
+ write_pnet(&port->pnet, dev_net(dev));
port->dev = dev;
port->mode = IPVLAN_MODE_L3;
INIT_LIST_HEAD(&port->ipvlans);
@@ -142,7 +153,7 @@
dev->priv_flags &= ~IFF_IPVLAN_MASTER;
if (port->mode == IPVLAN_MODE_L3S) {
dev->priv_flags &= ~IFF_L3MDEV_MASTER;
- ipvlan_unregister_nf_hook();
+ ipvlan_unregister_nf_hook(dev_net(dev));
dev->l3mdev_ops = NULL;
}
netdev_rx_handler_unregister(dev);
@@ -673,6 +684,24 @@
ipvlan->dev);
break;
+ case NETDEV_REGISTER: {
+ struct net *oldnet, *newnet = dev_net(dev);
+ struct ipvlan_netns *old_vnet;
+
+ oldnet = read_pnet(&port->pnet);
+ if (net_eq(newnet, oldnet))
+ break;
+
+ write_pnet(&port->pnet, newnet);
+
+ old_vnet = net_generic(oldnet, ipvlan_netid);
+ if (!old_vnet->ipvl_nf_hook_refcnt)
+ break;
+
+ ipvlan_register_nf_hook(newnet);
+ ipvlan_unregister_nf_hook(oldnet);
+ break;
+ }
case NETDEV_UNREGISTER:
if (dev->reg_state != NETREG_UNREGISTERING)
break;
@@ -854,6 +883,23 @@
.notifier_call = ipvlan_addr6_event,
};
+static void ipvlan_ns_exit(struct net *net)
+{
+ struct ipvlan_netns *vnet = net_generic(net, ipvlan_netid);
+
+ if (WARN_ON_ONCE(vnet->ipvl_nf_hook_refcnt)) {
+ vnet->ipvl_nf_hook_refcnt = 0;
+ nf_unregister_net_hooks(net, ipvl_nfops,
+ ARRAY_SIZE(ipvl_nfops));
+ }
+}
+
+static struct pernet_operations ipvlan_net_ops = {
+ .id = &ipvlan_netid,
+ .size = sizeof(struct ipvlan_netns),
+ .exit = ipvlan_ns_exit,
+};
+
static int __init ipvlan_init_module(void)
{
int err;
@@ -863,10 +909,16 @@
register_inet6addr_notifier(&ipvlan_addr6_notifier_block);
register_inetaddr_notifier(&ipvlan_addr4_notifier_block);
- err = ipvlan_link_register(&ipvlan_link_ops);
+ err = register_pernet_subsys(&ipvlan_net_ops);
if (err < 0)
goto error;
+ err = ipvlan_link_register(&ipvlan_link_ops);
+ if (err < 0) {
+ unregister_pernet_subsys(&ipvlan_net_ops);
+ goto error;
+ }
+
return 0;
error:
unregister_inetaddr_notifier(&ipvlan_addr4_notifier_block);
@@ -878,6 +930,7 @@
static void __exit ipvlan_cleanup_module(void)
{
rtnl_link_unregister(&ipvlan_link_ops);
+ unregister_pernet_subsys(&ipvlan_net_ops);
unregister_netdevice_notifier(&ipvlan_notifier_block);
unregister_inetaddr_notifier(&ipvlan_addr4_notifier_block);
unregister_inet6addr_notifier(&ipvlan_addr6_notifier_block);
diff --git a/drivers/net/loopback.c b/drivers/net/loopback.c
index b23b719..224f65c 100644
--- a/drivers/net/loopback.c
+++ b/drivers/net/loopback.c
@@ -13,7 +13,7 @@
*
* Alan Cox : Fixed oddments for NET3.014
* Alan Cox : Rejig for NET3.029 snap #3
- * Alan Cox : Fixed NET3.029 bugs and sped up
+ * Alan Cox : Fixed NET3.029 bugs and sped up
* Larry McVoy : Tiny tweak to double performance
* Alan Cox : Backed out LMV's tweak - the linux mm
* can't take it...
@@ -41,7 +41,7 @@
#include <linux/in.h>
#include <linux/uaccess.h>
-#include <asm/io.h>
+#include <linux/io.h>
#include <linux/inet.h>
#include <linux/netdevice.h>
@@ -55,6 +55,7 @@
#include <linux/ip.h>
#include <linux/tcp.h>
#include <linux/percpu.h>
+#include <linux/net_tstamp.h>
#include <net/net_namespace.h>
#include <linux/u64_stats_sync.h>
@@ -64,8 +65,7 @@
struct u64_stats_sync syncp;
};
-/*
- * The higher levels take care of making this non-reentrant (it's
+/* The higher levels take care of making this non-reentrant (it's
* called with bh's disabled).
*/
static netdev_tx_t loopback_xmit(struct sk_buff *skb,
@@ -74,6 +74,7 @@
struct pcpu_lstats *lb_stats;
int len;
+ skb_tx_timestamp(skb);
skb_orphan(skb);
/* Before queueing this packet to netif_rx(),
@@ -129,8 +130,21 @@
return 1;
}
+static int loopback_get_ts_info(struct net_device *netdev,
+ struct ethtool_ts_info *ts_info)
+{
+ ts_info->so_timestamping = SOF_TIMESTAMPING_TX_SOFTWARE |
+ SOF_TIMESTAMPING_RX_SOFTWARE |
+ SOF_TIMESTAMPING_SOFTWARE;
+
+ ts_info->phc_index = -1;
+
+ return 0;
+};
+
static const struct ethtool_ops loopback_ethtool_ops = {
.get_link = always_on,
+ .get_ts_info = loopback_get_ts_info,
};
static int loopback_dev_init(struct net_device *dev)
@@ -149,14 +163,13 @@
}
static const struct net_device_ops loopback_ops = {
- .ndo_init = loopback_dev_init,
- .ndo_start_xmit= loopback_xmit,
+ .ndo_init = loopback_dev_init,
+ .ndo_start_xmit = loopback_xmit,
.ndo_get_stats64 = loopback_get_stats64,
.ndo_set_mac_address = eth_mac_addr,
};
-/*
- * The loopback device is special. There is only one instance
+/* The loopback device is special. There is only one instance
* per network namespace.
*/
static void loopback_setup(struct net_device *dev)
@@ -170,7 +183,7 @@
dev->priv_flags |= IFF_LIVE_ADDR_CHANGE | IFF_NO_QUEUE;
netif_keep_dst(dev);
dev->hw_features = NETIF_F_GSO_SOFTWARE;
- dev->features = NETIF_F_SG | NETIF_F_FRAGLIST
+ dev->features = NETIF_F_SG | NETIF_F_FRAGLIST
| NETIF_F_GSO_SOFTWARE
| NETIF_F_HW_CSUM
| NETIF_F_RXCSUM
@@ -206,7 +219,6 @@
net->loopback_dev = dev;
return 0;
-
out_free_netdev:
free_netdev(dev);
out:
@@ -217,5 +229,5 @@
/* Registered in net/core/dev.c */
struct pernet_operations __net_initdata loopback_net_ops = {
- .init = loopback_net_init,
+ .init = loopback_net_init,
};
diff --git a/drivers/net/macsec.c b/drivers/net/macsec.c
index ff0a5ed..cdc347b 100644
--- a/drivers/net/macsec.c
+++ b/drivers/net/macsec.c
@@ -617,7 +617,8 @@
static struct aead_request *macsec_alloc_req(struct crypto_aead *tfm,
unsigned char **iv,
- struct scatterlist **sg)
+ struct scatterlist **sg,
+ int num_frags)
{
size_t size, iv_offset, sg_offset;
struct aead_request *req;
@@ -629,7 +630,7 @@
size = ALIGN(size, __alignof__(struct scatterlist));
sg_offset = size;
- size += sizeof(struct scatterlist) * (MAX_SKB_FRAGS + 1);
+ size += sizeof(struct scatterlist) * num_frags;
tmp = kmalloc(size, GFP_ATOMIC);
if (!tmp)
@@ -649,6 +650,7 @@
{
int ret;
struct scatterlist *sg;
+ struct sk_buff *trailer;
unsigned char *iv;
struct ethhdr *eth;
struct macsec_eth_header *hh;
@@ -723,7 +725,14 @@
return ERR_PTR(-EINVAL);
}
- req = macsec_alloc_req(tx_sa->key.tfm, &iv, &sg);
+ ret = skb_cow_data(skb, 0, &trailer);
+ if (unlikely(ret < 0)) {
+ macsec_txsa_put(tx_sa);
+ kfree_skb(skb);
+ return ERR_PTR(ret);
+ }
+
+ req = macsec_alloc_req(tx_sa->key.tfm, &iv, &sg, ret);
if (!req) {
macsec_txsa_put(tx_sa);
kfree_skb(skb);
@@ -732,7 +741,7 @@
macsec_fill_iv(iv, secy->sci, pn);
- sg_init_table(sg, MAX_SKB_FRAGS + 1);
+ sg_init_table(sg, ret);
skb_to_sgvec(skb, sg, 0, skb->len);
if (tx_sc->encrypt) {
@@ -917,6 +926,7 @@
{
int ret;
struct scatterlist *sg;
+ struct sk_buff *trailer;
unsigned char *iv;
struct aead_request *req;
struct macsec_eth_header *hdr;
@@ -927,7 +937,12 @@
if (!skb)
return ERR_PTR(-ENOMEM);
- req = macsec_alloc_req(rx_sa->key.tfm, &iv, &sg);
+ ret = skb_cow_data(skb, 0, &trailer);
+ if (unlikely(ret < 0)) {
+ kfree_skb(skb);
+ return ERR_PTR(ret);
+ }
+ req = macsec_alloc_req(rx_sa->key.tfm, &iv, &sg, ret);
if (!req) {
kfree_skb(skb);
return ERR_PTR(-ENOMEM);
@@ -936,7 +951,7 @@
hdr = (struct macsec_eth_header *)skb->data;
macsec_fill_iv(iv, sci, ntohl(hdr->packet_number));
- sg_init_table(sg, MAX_SKB_FRAGS + 1);
+ sg_init_table(sg, ret);
skb_to_sgvec(skb, sg, 0, skb->len);
if (hdr->tci_an & MACSEC_TCI_E) {
@@ -1590,8 +1605,9 @@
if (!attrs[MACSEC_ATTR_SA_CONFIG])
return -EINVAL;
- if (nla_parse_nested(tb_sa, MACSEC_SA_ATTR_MAX, attrs[MACSEC_ATTR_SA_CONFIG],
- macsec_genl_sa_policy))
+ if (nla_parse_nested(tb_sa, MACSEC_SA_ATTR_MAX,
+ attrs[MACSEC_ATTR_SA_CONFIG],
+ macsec_genl_sa_policy, NULL))
return -EINVAL;
return 0;
@@ -1602,8 +1618,9 @@
if (!attrs[MACSEC_ATTR_RXSC_CONFIG])
return -EINVAL;
- if (nla_parse_nested(tb_rxsc, MACSEC_RXSC_ATTR_MAX, attrs[MACSEC_ATTR_RXSC_CONFIG],
- macsec_genl_rxsc_policy))
+ if (nla_parse_nested(tb_rxsc, MACSEC_RXSC_ATTR_MAX,
+ attrs[MACSEC_ATTR_RXSC_CONFIG],
+ macsec_genl_rxsc_policy, NULL))
return -EINVAL;
return 0;
diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
index 9261722..b34eaaa 100644
--- a/drivers/net/macvlan.c
+++ b/drivers/net/macvlan.c
@@ -1139,6 +1139,7 @@
static void macvlan_port_destroy(struct net_device *dev)
{
struct macvlan_port *port = macvlan_port_get_rtnl(dev);
+ struct sk_buff *skb;
dev->priv_flags &= ~IFF_MACVLAN_PORT;
netdev_rx_handler_unregister(dev);
@@ -1147,7 +1148,15 @@
* but we need to cancel it and purge left skbs if any.
*/
cancel_work_sync(&port->bc_work);
- __skb_queue_purge(&port->bc_queue);
+
+ while ((skb = __skb_dequeue(&port->bc_queue))) {
+ const struct macvlan_dev *src = MACVLAN_SKB_CB(skb)->src;
+
+ if (src)
+ dev_put(src->dev);
+
+ kfree_skb(skb);
+ }
kfree(port);
}
diff --git a/drivers/net/ntb_netdev.c b/drivers/net/ntb_netdev.c
index 36877ba..4daf3d0 100644
--- a/drivers/net/ntb_netdev.c
+++ b/drivers/net/ntb_netdev.c
@@ -372,18 +372,19 @@
strlcpy(info->bus_info, pci_name(dev->pdev), sizeof(info->bus_info));
}
-static int ntb_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int ntb_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
- cmd->supported = SUPPORTED_Backplane;
- cmd->advertising = ADVERTISED_Backplane;
- ethtool_cmd_speed_set(cmd, SPEED_UNKNOWN);
- cmd->duplex = DUPLEX_FULL;
- cmd->port = PORT_OTHER;
- cmd->phy_address = 0;
- cmd->transceiver = XCVR_DUMMY1;
- cmd->autoneg = AUTONEG_ENABLE;
- cmd->maxtxpkt = 0;
- cmd->maxrxpkt = 0;
+ ethtool_link_ksettings_zero_link_mode(cmd, supported);
+ ethtool_link_ksettings_add_link_mode(cmd, supported, Backplane);
+ ethtool_link_ksettings_zero_link_mode(cmd, advertising);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, Backplane);
+
+ cmd->base.speed = SPEED_UNKNOWN;
+ cmd->base.duplex = DUPLEX_FULL;
+ cmd->base.port = PORT_OTHER;
+ cmd->base.phy_address = 0;
+ cmd->base.autoneg = AUTONEG_ENABLE;
return 0;
}
@@ -391,7 +392,7 @@
static const struct ethtool_ops ntb_ethtool_ops = {
.get_drvinfo = ntb_get_drvinfo,
.get_link = ethtool_op_get_link,
- .get_settings = ntb_get_settings,
+ .get_link_ksettings = ntb_get_link_ksettings,
};
static const struct ntb_queue_handlers ntb_netdev_handlers = {
diff --git a/drivers/net/phy/Kconfig b/drivers/net/phy/Kconfig
index 8dbd59b..60ffc9d 100644
--- a/drivers/net/phy/Kconfig
+++ b/drivers/net/phy/Kconfig
@@ -2,33 +2,12 @@
# PHY Layer Configuration
#
-menuconfig PHYLIB
- tristate "PHY Device support and infrastructure"
- depends on NETDEVICES
+menuconfig MDIO_DEVICE
+ tristate "MDIO bus device drivers"
help
- Ethernet controllers are usually attached to PHY
- devices. This option provides infrastructure for
- managing PHY devices.
+ MDIO devices and driver infrastructure code.
-if PHYLIB
-
-config SWPHY
- bool
-
-config LED_TRIGGER_PHY
- bool "Support LED triggers for tracking link state"
- depends on LEDS_TRIGGERS
- ---help---
- Adds support for a set of LED trigger events per-PHY. Link
- state change will trigger the events, for consumption by an
- LED class driver. There are triggers for each link speed currently
- supported by the phy, and are of the form:
- <mii bus id>:<phy>:<speed>
-
- Where speed is in the form:
- <Speed in megabits>Mbps or <Speed in gigabits>Gbps
-
-comment "MDIO bus device drivers"
+if MDIO_DEVICE
config MDIO_BCM_IPROC
tristate "Broadcom iProc MDIO bus controller"
@@ -40,7 +19,7 @@
config MDIO_BCM_UNIMAC
tristate "Broadcom UniMAC MDIO bus controller"
- depends on HAS_IOMEM
+ depends on HAS_IOMEM && OF_MDIO
help
This module provides a driver for the Broadcom UniMAC MDIO busses.
This hardware can be found in the Broadcom GENET Ethernet MAC
@@ -49,6 +28,7 @@
config MDIO_BITBANG
tristate "Bitbanged MDIO buses"
+ depends on !(MDIO_DEVICE=y && PHYLIB=m)
help
This module implements the MDIO bus protocol in software,
for use by low level drivers that export the ability to
@@ -160,6 +140,36 @@
This module provides a driver for the MDIO busses found in the
APM X-Gene SoC's.
+endif
+
+menuconfig PHYLIB
+ tristate "PHY Device support and infrastructure"
+ depends on NETDEVICES
+ select MDIO_DEVICE
+ help
+ Ethernet controllers are usually attached to PHY
+ devices. This option provides infrastructure for
+ managing PHY devices.
+
+if PHYLIB
+
+config SWPHY
+ bool
+
+config LED_TRIGGER_PHY
+ bool "Support LED triggers for tracking link state"
+ depends on LEDS_TRIGGERS
+ ---help---
+ Adds support for a set of LED trigger events per-PHY. Link
+ state change will trigger the events, for consumption by an
+ LED class driver. There are triggers for each link speed currently
+ supported by the phy, and are of the form:
+ <mii bus id>:<phy>:<speed>
+
+ Where speed is in the form:
+ <Speed in megabits>Mbps or <Speed in gigabits>Gbps
+
+
comment "MII PHY device drivers"
config AMD_PHY
diff --git a/drivers/net/phy/Makefile b/drivers/net/phy/Makefile
index 407b0b6..e36db9a 100644
--- a/drivers/net/phy/Makefile
+++ b/drivers/net/phy/Makefile
@@ -1,7 +1,20 @@
# Makefile for Linux PHY drivers and MDIO bus drivers
-libphy-y := phy.o phy_device.o mdio_bus.o mdio_device.o \
- mdio-boardinfo.o
+libphy-y := phy.o phy-core.o phy_device.o
+mdio-bus-y += mdio_bus.o mdio_device.o
+
+ifdef CONFIG_MDIO_DEVICE
+obj-y += mdio-boardinfo.o
+endif
+
+# PHYLIB implies MDIO_DEVICE, in that case, we have a bunch of circular
+# dependencies that does not make it possible to split mdio-bus objects into a
+# dedicated loadable module, so we bundle them all together into libphy.ko
+ifdef CONFIG_PHYLIB
+libphy-y += $(mdio-bus-y)
+else
+obj-$(CONFIG_MDIO_DEVICE) += mdio-bus.o
+endif
libphy-$(CONFIG_SWPHY) += swphy.o
libphy-$(CONFIG_LED_TRIGGER_PHY) += phy_led_triggers.o
diff --git a/drivers/net/phy/bcm-phy-lib.c b/drivers/net/phy/bcm-phy-lib.c
index ab9ad68..171010e 100644
--- a/drivers/net/phy/bcm-phy-lib.c
+++ b/drivers/net/phy/bcm-phy-lib.c
@@ -1,5 +1,5 @@
/*
- * Copyright (C) 2015 Broadcom Corporation
+ * Copyright (C) 2015-2017 Broadcom
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License as
@@ -201,8 +201,7 @@
int val;
/* Enable EEE at PHY level */
- val = phy_read_mmd_indirect(phydev, BRCM_CL45VEN_EEE_CONTROL,
- MDIO_MMD_AN);
+ val = phy_read_mmd(phydev, MDIO_MMD_AN, BRCM_CL45VEN_EEE_CONTROL);
if (val < 0)
return val;
@@ -211,22 +210,19 @@
else
val &= ~(LPI_FEATURE_EN | LPI_FEATURE_EN_DIG1000X);
- phy_write_mmd_indirect(phydev, BRCM_CL45VEN_EEE_CONTROL,
- MDIO_MMD_AN, (u32)val);
+ phy_write_mmd(phydev, MDIO_MMD_AN, BRCM_CL45VEN_EEE_CONTROL, (u32)val);
/* Advertise EEE */
- val = phy_read_mmd_indirect(phydev, BCM_CL45VEN_EEE_ADV,
- MDIO_MMD_AN);
+ val = phy_read_mmd(phydev, MDIO_MMD_AN, BCM_CL45VEN_EEE_ADV);
if (val < 0)
return val;
if (enable)
- val |= (MDIO_AN_EEE_ADV_100TX | MDIO_AN_EEE_ADV_1000T);
+ val |= (MDIO_EEE_100TX | MDIO_EEE_1000T);
else
- val &= ~(MDIO_AN_EEE_ADV_100TX | MDIO_AN_EEE_ADV_1000T);
+ val &= ~(MDIO_EEE_100TX | MDIO_EEE_1000T);
- phy_write_mmd_indirect(phydev, BCM_CL45VEN_EEE_ADV,
- MDIO_MMD_AN, (u32)val);
+ phy_write_mmd(phydev, MDIO_MMD_AN, BCM_CL45VEN_EEE_ADV, (u32)val);
return 0;
}
diff --git a/drivers/net/phy/bcm7xxx.c b/drivers/net/phy/bcm7xxx.c
index d1c2614..caa9f6e 100644
--- a/drivers/net/phy/bcm7xxx.c
+++ b/drivers/net/phy/bcm7xxx.c
@@ -1,7 +1,7 @@
/*
* Broadcom BCM7xxx internal transceivers support.
*
- * Copyright (C) 2014, Broadcom Corporation
+ * Copyright (C) 2014-2017 Broadcom
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
@@ -19,7 +19,7 @@
/* Broadcom BCM7xxx internal PHY registers */
-/* 40nm only register definitions */
+/* EPHY only register definitions */
#define MII_BCM7XXX_100TX_AUX_CTL 0x10
#define MII_BCM7XXX_100TX_FALSE_CAR 0x13
#define MII_BCM7XXX_100TX_DISC 0x14
@@ -27,6 +27,19 @@
#define MII_BCM7XXX_64CLK_MDIO BIT(12)
#define MII_BCM7XXX_TEST 0x1f
#define MII_BCM7XXX_SHD_MODE_2 BIT(2)
+#define MII_BCM7XXX_SHD_2_ADDR_CTRL 0xe
+#define MII_BCM7XXX_SHD_2_CTRL_STAT 0xf
+#define MII_BCM7XXX_SHD_2_BIAS_TRIM 0x1a
+#define MII_BCM7XXX_SHD_3_AN_EEE_ADV 0x3
+#define MII_BCM7XXX_SHD_3_PCS_CTRL_2 0x6
+#define MII_BCM7XXX_PCS_CTRL_2_DEF 0x4400
+#define MII_BCM7XXX_SHD_3_AN_STAT 0xb
+#define MII_BCM7XXX_AN_NULL_MSG_EN BIT(0)
+#define MII_BCM7XXX_AN_EEE_EN BIT(1)
+#define MII_BCM7XXX_SHD_3_EEE_THRESH 0xe
+#define MII_BCM7XXX_EEE_THRESH_DEF 0x50
+#define MII_BCM7XXX_SHD_3_TL4 0x23
+#define MII_BCM7XXX_TL4_RST_MSK (BIT(2) | BIT(1))
/* 28nm only register definitions */
#define MISC_ADDR(base, channel) base, channel
@@ -286,6 +299,181 @@
return v;
}
+static int bcm7xxx_28nm_ephy_01_afe_config_init(struct phy_device *phydev)
+{
+ int ret;
+
+ /* set shadow mode 2 */
+ ret = phy_set_clr_bits(phydev, MII_BCM7XXX_TEST,
+ MII_BCM7XXX_SHD_MODE_2, 0);
+ if (ret < 0)
+ return ret;
+
+ /* Set current trim values INT_trim = -1, Ext_trim =0 */
+ ret = phy_write(phydev, MII_BCM7XXX_SHD_2_BIAS_TRIM, 0x3BE0);
+ if (ret < 0)
+ goto reset_shadow_mode;
+
+ /* Cal reset */
+ ret = phy_write(phydev, MII_BCM7XXX_SHD_2_ADDR_CTRL,
+ MII_BCM7XXX_SHD_3_TL4);
+ if (ret < 0)
+ goto reset_shadow_mode;
+ ret = phy_set_clr_bits(phydev, MII_BCM7XXX_SHD_2_CTRL_STAT,
+ MII_BCM7XXX_TL4_RST_MSK, 0);
+ if (ret < 0)
+ goto reset_shadow_mode;
+
+ /* Cal reset disable */
+ ret = phy_write(phydev, MII_BCM7XXX_SHD_2_ADDR_CTRL,
+ MII_BCM7XXX_SHD_3_TL4);
+ if (ret < 0)
+ goto reset_shadow_mode;
+ ret = phy_set_clr_bits(phydev, MII_BCM7XXX_SHD_2_CTRL_STAT,
+ 0, MII_BCM7XXX_TL4_RST_MSK);
+ if (ret < 0)
+ goto reset_shadow_mode;
+
+reset_shadow_mode:
+ /* reset shadow mode 2 */
+ ret = phy_set_clr_bits(phydev, MII_BCM7XXX_TEST, 0,
+ MII_BCM7XXX_SHD_MODE_2);
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+/* The 28nm EPHY does not support Clause 45 (MMD) used by bcm-phy-lib */
+static int bcm7xxx_28nm_ephy_apd_enable(struct phy_device *phydev)
+{
+ int ret;
+
+ /* set shadow mode 1 */
+ ret = phy_set_clr_bits(phydev, MII_BRCM_FET_BRCMTEST,
+ MII_BRCM_FET_BT_SRE, 0);
+ if (ret < 0)
+ return ret;
+
+ /* Enable auto-power down */
+ ret = phy_set_clr_bits(phydev, MII_BRCM_FET_SHDW_AUXSTAT2,
+ MII_BRCM_FET_SHDW_AS2_APDE, 0);
+ if (ret < 0)
+ return ret;
+
+ /* reset shadow mode 1 */
+ ret = phy_set_clr_bits(phydev, MII_BRCM_FET_BRCMTEST, 0,
+ MII_BRCM_FET_BT_SRE);
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+static int bcm7xxx_28nm_ephy_eee_enable(struct phy_device *phydev)
+{
+ int ret;
+
+ /* set shadow mode 2 */
+ ret = phy_set_clr_bits(phydev, MII_BCM7XXX_TEST,
+ MII_BCM7XXX_SHD_MODE_2, 0);
+ if (ret < 0)
+ return ret;
+
+ /* Advertise supported modes */
+ ret = phy_write(phydev, MII_BCM7XXX_SHD_2_ADDR_CTRL,
+ MII_BCM7XXX_SHD_3_AN_EEE_ADV);
+ if (ret < 0)
+ goto reset_shadow_mode;
+ ret = phy_write(phydev, MII_BCM7XXX_SHD_2_CTRL_STAT,
+ MDIO_EEE_100TX);
+ if (ret < 0)
+ goto reset_shadow_mode;
+
+ /* Restore Defaults */
+ ret = phy_write(phydev, MII_BCM7XXX_SHD_2_ADDR_CTRL,
+ MII_BCM7XXX_SHD_3_PCS_CTRL_2);
+ if (ret < 0)
+ goto reset_shadow_mode;
+ ret = phy_write(phydev, MII_BCM7XXX_SHD_2_CTRL_STAT,
+ MII_BCM7XXX_PCS_CTRL_2_DEF);
+ if (ret < 0)
+ goto reset_shadow_mode;
+
+ ret = phy_write(phydev, MII_BCM7XXX_SHD_2_ADDR_CTRL,
+ MII_BCM7XXX_SHD_3_EEE_THRESH);
+ if (ret < 0)
+ goto reset_shadow_mode;
+ ret = phy_write(phydev, MII_BCM7XXX_SHD_2_CTRL_STAT,
+ MII_BCM7XXX_EEE_THRESH_DEF);
+ if (ret < 0)
+ goto reset_shadow_mode;
+
+ /* Enable EEE autonegotiation */
+ ret = phy_write(phydev, MII_BCM7XXX_SHD_2_ADDR_CTRL,
+ MII_BCM7XXX_SHD_3_AN_STAT);
+ if (ret < 0)
+ goto reset_shadow_mode;
+ ret = phy_write(phydev, MII_BCM7XXX_SHD_2_CTRL_STAT,
+ (MII_BCM7XXX_AN_NULL_MSG_EN | MII_BCM7XXX_AN_EEE_EN));
+ if (ret < 0)
+ goto reset_shadow_mode;
+
+reset_shadow_mode:
+ /* reset shadow mode 2 */
+ ret = phy_set_clr_bits(phydev, MII_BCM7XXX_TEST, 0,
+ MII_BCM7XXX_SHD_MODE_2);
+ if (ret < 0)
+ return ret;
+
+ /* Restart autoneg */
+ phy_write(phydev, MII_BMCR,
+ (BMCR_SPEED100 | BMCR_ANENABLE | BMCR_ANRESTART));
+
+ return 0;
+}
+
+static int bcm7xxx_28nm_ephy_config_init(struct phy_device *phydev)
+{
+ u8 rev = phydev->phy_id & ~phydev->drv->phy_id_mask;
+ int ret = 0;
+
+ pr_info_once("%s: %s PHY revision: 0x%02x\n",
+ phydev_name(phydev), phydev->drv->name, rev);
+
+ /* Dummy read to a register to workaround a possible issue upon reset
+ * where the internal inverter may not allow the first MDIO transaction
+ * to pass the MDIO management controller and make us return 0xffff for
+ * such reads.
+ */
+ phy_read(phydev, MII_BMSR);
+
+ /* Apply AFE software work-around if necessary */
+ if (rev == 0x01) {
+ ret = bcm7xxx_28nm_ephy_01_afe_config_init(phydev);
+ if (ret)
+ return ret;
+ }
+
+ ret = bcm7xxx_28nm_ephy_eee_enable(phydev);
+ if (ret)
+ return ret;
+
+ return bcm7xxx_28nm_ephy_apd_enable(phydev);
+}
+
+static int bcm7xxx_28nm_ephy_resume(struct phy_device *phydev)
+{
+ int ret;
+
+ /* Re-apply workarounds coming out suspend/resume */
+ ret = bcm7xxx_28nm_ephy_config_init(phydev);
+ if (ret)
+ return ret;
+
+ return genphy_config_aneg(phydev);
+}
+
static int bcm7xxx_config_init(struct phy_device *phydev)
{
int ret;
@@ -434,6 +622,23 @@
.probe = bcm7xxx_28nm_probe, \
}
+#define BCM7XXX_28NM_EPHY(_oui, _name) \
+{ \
+ .phy_id = (_oui), \
+ .phy_id_mask = 0xfffffff0, \
+ .name = _name, \
+ .features = PHY_BASIC_FEATURES, \
+ .flags = PHY_IS_INTERNAL, \
+ .config_init = bcm7xxx_28nm_ephy_config_init, \
+ .config_aneg = genphy_config_aneg, \
+ .read_status = genphy_read_status, \
+ .resume = bcm7xxx_28nm_ephy_resume, \
+ .get_sset_count = bcm_phy_get_sset_count, \
+ .get_strings = bcm_phy_get_strings, \
+ .get_stats = bcm7xxx_28nm_get_phy_stats, \
+ .probe = bcm7xxx_28nm_probe, \
+}
+
#define BCM7XXX_40NM_EPHY(_oui, _name) \
{ \
.phy_id = (_oui), \
@@ -450,6 +655,9 @@
static struct phy_driver bcm7xxx_driver[] = {
BCM7XXX_28NM_GPHY(PHY_ID_BCM7250, "Broadcom BCM7250"),
+ BCM7XXX_28NM_EPHY(PHY_ID_BCM7260, "Broadcom BCM7260"),
+ BCM7XXX_28NM_EPHY(PHY_ID_BCM7268, "Broadcom BCM7268"),
+ BCM7XXX_28NM_EPHY(PHY_ID_BCM7271, "Broadcom BCM7271"),
BCM7XXX_28NM_GPHY(PHY_ID_BCM7278, "Broadcom BCM7278"),
BCM7XXX_28NM_GPHY(PHY_ID_BCM7364, "Broadcom BCM7364"),
BCM7XXX_28NM_GPHY(PHY_ID_BCM7366, "Broadcom BCM7366"),
@@ -466,6 +674,9 @@
static struct mdio_device_id __maybe_unused bcm7xxx_tbl[] = {
{ PHY_ID_BCM7250, 0xfffffff0, },
+ { PHY_ID_BCM7260, 0xfffffff0, },
+ { PHY_ID_BCM7268, 0xfffffff0, },
+ { PHY_ID_BCM7271, 0xfffffff0, },
{ PHY_ID_BCM7278, 0xfffffff0, },
{ PHY_ID_BCM7364, 0xfffffff0, },
{ PHY_ID_BCM7366, 0xfffffff0, },
diff --git a/drivers/net/phy/broadcom.c b/drivers/net/phy/broadcom.c
index 9cd8b27..a32dc5d 100644
--- a/drivers/net/phy/broadcom.c
+++ b/drivers/net/phy/broadcom.c
@@ -74,27 +74,40 @@
return 0;
}
-static int bcm54810_config(struct phy_device *phydev)
+static int bcm5481x_config(struct phy_device *phydev)
{
int rc, val;
- val = bcm_phy_read_exp(phydev, BCM54810_EXP_BROADREACH_LRE_MISC_CTL);
- val &= ~BCM54810_EXP_BROADREACH_LRE_MISC_CTL_EN;
- rc = bcm_phy_write_exp(phydev, BCM54810_EXP_BROADREACH_LRE_MISC_CTL,
- val);
- if (rc < 0)
- return rc;
-
+ /* handling PHY's internal RX clock delay */
val = bcm54xx_auxctl_read(phydev, MII_BCM54XX_AUXCTL_SHDWSEL_MISC);
- val &= ~MII_BCM54XX_AUXCTL_SHDWSEL_MISC_RGMII_SKEW_EN;
val |= MII_BCM54XX_AUXCTL_MISC_WREN;
+ if (phydev->interface == PHY_INTERFACE_MODE_RGMII ||
+ phydev->interface == PHY_INTERFACE_MODE_RGMII_TXID) {
+ /* Disable RGMII RXC-RXD skew */
+ val &= ~MII_BCM54XX_AUXCTL_SHDWSEL_MISC_RGMII_SKEW_EN;
+ }
+ if (phydev->interface == PHY_INTERFACE_MODE_RGMII_ID ||
+ phydev->interface == PHY_INTERFACE_MODE_RGMII_RXID) {
+ /* Enable RGMII RXC-RXD skew */
+ val |= MII_BCM54XX_AUXCTL_SHDWSEL_MISC_RGMII_SKEW_EN;
+ }
rc = bcm54xx_auxctl_write(phydev, MII_BCM54XX_AUXCTL_SHDWSEL_MISC,
val);
if (rc < 0)
return rc;
+ /* handling PHY's internal TX clock delay */
val = bcm_phy_read_shadow(phydev, BCM54810_SHD_CLK_CTL);
- val &= ~BCM54810_SHD_CLK_CTL_GTXCLK_EN;
+ if (phydev->interface == PHY_INTERFACE_MODE_RGMII ||
+ phydev->interface == PHY_INTERFACE_MODE_RGMII_RXID) {
+ /* Disable internal TX clock delay */
+ val &= ~BCM54810_SHD_CLK_CTL_GTXCLK_EN;
+ }
+ if (phydev->interface == PHY_INTERFACE_MODE_RGMII_ID ||
+ phydev->interface == PHY_INTERFACE_MODE_RGMII_TXID) {
+ /* Enable internal TX clock delay */
+ val |= BCM54810_SHD_CLK_CTL_GTXCLK_EN;
+ }
rc = bcm_phy_write_shadow(phydev, BCM54810_SHD_CLK_CTL, val);
if (rc < 0)
return rc;
@@ -244,7 +257,7 @@
static int bcm54xx_config_init(struct phy_device *phydev)
{
- int reg, err;
+ int reg, err, val;
reg = phy_read(phydev, MII_BCM54XX_ECR);
if (reg < 0)
@@ -283,8 +296,14 @@
if (err)
return err;
} else if (BRCM_PHY_MODEL(phydev) == PHY_ID_BCM54810) {
- err = bcm54810_config(phydev);
- if (err)
+ /* For BCM54810, we need to disable BroadR-Reach function */
+ val = bcm_phy_read_exp(phydev,
+ BCM54810_EXP_BROADREACH_LRE_MISC_CTL);
+ val &= ~BCM54810_EXP_BROADREACH_LRE_MISC_CTL_EN;
+ err = bcm_phy_write_exp(phydev,
+ BCM54810_EXP_BROADREACH_LRE_MISC_CTL,
+ val);
+ if (err < 0)
return err;
}
@@ -392,29 +411,7 @@
ret = genphy_config_aneg(phydev);
/* Then we can set up the delay. */
- if (phydev->interface == PHY_INTERFACE_MODE_RGMII_RXID) {
- u16 reg;
-
- /*
- * There is no BCM5481 specification available, so down
- * here is everything we know about "register 0x18". This
- * at least helps BCM5481 to successfully receive packets
- * on MPC8360E-RDK board. Peter Barada <peterb@logicpd.com>
- * says: "This sets delay between the RXD and RXC signals
- * instead of using trace lengths to achieve timing".
- */
-
- /* Set RDX clk delay. */
- reg = 0x7 | (0x7 << 12);
- phy_write(phydev, 0x18, reg);
-
- reg = phy_read(phydev, 0x18);
- /* Set RDX-RXC skew. */
- reg |= (1 << 8);
- /* Write bits 14:0. */
- reg |= (1 << 15);
- phy_write(phydev, 0x18, reg);
- }
+ bcm5481x_config(phydev);
if (of_property_read_bool(np, "enet-phy-lane-swap")) {
/* Lane Swap - Undocumented register...magic! */
diff --git a/drivers/net/phy/dp83848.c b/drivers/net/phy/dp83848.c
index a10d0e7..3de4fe4 100644
--- a/drivers/net/phy/dp83848.c
+++ b/drivers/net/phy/dp83848.c
@@ -115,5 +115,5 @@
module_phy_driver(dp83848_driver);
MODULE_DESCRIPTION("Texas Instruments DP83848 PHY driver");
-MODULE_AUTHOR("Andrew F. Davis <afd@ti.com");
+MODULE_AUTHOR("Andrew F. Davis <afd@ti.com>");
MODULE_LICENSE("GPL");
diff --git a/drivers/net/phy/dp83867.c b/drivers/net/phy/dp83867.c
index 1986553..b57f20e 100644
--- a/drivers/net/phy/dp83867.c
+++ b/drivers/net/phy/dp83867.c
@@ -133,14 +133,14 @@
(struct dp83867_private *)phydev->priv;
u16 val;
- val = phy_read_mmd_indirect(phydev, DP83867_CFG4, DP83867_DEVADDR);
+ val = phy_read_mmd(phydev, DP83867_DEVADDR, DP83867_CFG4);
if (dp83867->port_mirroring == DP83867_PORT_MIRROING_EN)
val |= DP83867_CFG4_PORT_MIRROR_EN;
else
val &= ~DP83867_CFG4_PORT_MIRROR_EN;
- phy_write_mmd_indirect(phydev, DP83867_CFG4, DP83867_DEVADDR, val);
+ phy_write_mmd(phydev, DP83867_DEVADDR, DP83867_CFG4, val);
return 0;
}
@@ -231,8 +231,7 @@
* register's bit 11 (marked as RESERVED).
*/
- bs = phy_read_mmd_indirect(phydev, DP83867_STRAP_STS1,
- DP83867_DEVADDR);
+ bs = phy_read_mmd(phydev, DP83867_DEVADDR, DP83867_STRAP_STS1);
if (bs & DP83867_STRAP_STS1_RESERVED)
val &= ~DP83867_PHYCR_RESERVED_MASK;
@@ -243,8 +242,7 @@
if ((phydev->interface >= PHY_INTERFACE_MODE_RGMII_ID) &&
(phydev->interface <= PHY_INTERFACE_MODE_RGMII_RXID)) {
- val = phy_read_mmd_indirect(phydev, DP83867_RGMIICTL,
- DP83867_DEVADDR);
+ val = phy_read_mmd(phydev, DP83867_DEVADDR, DP83867_RGMIICTL);
if (phydev->interface == PHY_INTERFACE_MODE_RGMII_ID)
val |= (DP83867_RGMII_TX_CLK_DELAY_EN | DP83867_RGMII_RX_CLK_DELAY_EN);
@@ -255,25 +253,24 @@
if (phydev->interface == PHY_INTERFACE_MODE_RGMII_RXID)
val |= DP83867_RGMII_RX_CLK_DELAY_EN;
- phy_write_mmd_indirect(phydev, DP83867_RGMIICTL,
- DP83867_DEVADDR, val);
+ phy_write_mmd(phydev, DP83867_DEVADDR, DP83867_RGMIICTL, val);
delay = (dp83867->rx_id_delay |
(dp83867->tx_id_delay << DP83867_RGMII_TX_CLK_DELAY_SHIFT));
- phy_write_mmd_indirect(phydev, DP83867_RGMIIDCTL,
- DP83867_DEVADDR, delay);
+ phy_write_mmd(phydev, DP83867_DEVADDR, DP83867_RGMIIDCTL,
+ delay);
if (dp83867->io_impedance >= 0) {
- val = phy_read_mmd_indirect(phydev, DP83867_IO_MUX_CFG,
- DP83867_DEVADDR);
+ val = phy_read_mmd(phydev, DP83867_DEVADDR,
+ DP83867_IO_MUX_CFG);
val &= ~DP83867_IO_MUX_CFG_IO_IMPEDANCE_CTRL;
val |= dp83867->io_impedance &
DP83867_IO_MUX_CFG_IO_IMPEDANCE_CTRL;
- phy_write_mmd_indirect(phydev, DP83867_IO_MUX_CFG,
- DP83867_DEVADDR, val);
+ phy_write_mmd(phydev, DP83867_DEVADDR,
+ DP83867_IO_MUX_CFG, val);
}
}
diff --git a/drivers/net/phy/intel-xway.c b/drivers/net/phy/intel-xway.c
index b1fd7bb..55f8c52 100644
--- a/drivers/net/phy/intel-xway.c
+++ b/drivers/net/phy/intel-xway.c
@@ -166,13 +166,13 @@
/* Clear all pending interrupts */
phy_read(phydev, XWAY_MDIO_ISTAT);
- phy_write_mmd_indirect(phydev, XWAY_MMD_LEDCH, MDIO_MMD_VEND2,
- XWAY_MMD_LEDCH_NACS_NONE |
- XWAY_MMD_LEDCH_SBF_F02HZ |
- XWAY_MMD_LEDCH_FBF_F16HZ);
- phy_write_mmd_indirect(phydev, XWAY_MMD_LEDCL, MDIO_MMD_VEND2,
- XWAY_MMD_LEDCH_CBLINK_NONE |
- XWAY_MMD_LEDCH_SCAN_NONE);
+ phy_write_mmd(phydev, MDIO_MMD_VEND2, XWAY_MMD_LEDCH,
+ XWAY_MMD_LEDCH_NACS_NONE |
+ XWAY_MMD_LEDCH_SBF_F02HZ |
+ XWAY_MMD_LEDCH_FBF_F16HZ);
+ phy_write_mmd(phydev, MDIO_MMD_VEND2, XWAY_MMD_LEDCL,
+ XWAY_MMD_LEDCH_CBLINK_NONE |
+ XWAY_MMD_LEDCH_SCAN_NONE);
/**
* In most cases only one LED is connected to this phy, so
@@ -183,12 +183,12 @@
ledxh = XWAY_MMD_LEDxH_BLINKF_NONE | XWAY_MMD_LEDxH_CON_LINK10XX;
ledxl = XWAY_MMD_LEDxL_PULSE_TXACT | XWAY_MMD_LEDxL_PULSE_RXACT |
XWAY_MMD_LEDxL_BLINKS_NONE;
- phy_write_mmd_indirect(phydev, XWAY_MMD_LED0H, MDIO_MMD_VEND2, ledxh);
- phy_write_mmd_indirect(phydev, XWAY_MMD_LED0L, MDIO_MMD_VEND2, ledxl);
- phy_write_mmd_indirect(phydev, XWAY_MMD_LED1H, MDIO_MMD_VEND2, ledxh);
- phy_write_mmd_indirect(phydev, XWAY_MMD_LED1L, MDIO_MMD_VEND2, ledxl);
- phy_write_mmd_indirect(phydev, XWAY_MMD_LED2H, MDIO_MMD_VEND2, ledxh);
- phy_write_mmd_indirect(phydev, XWAY_MMD_LED2L, MDIO_MMD_VEND2, ledxl);
+ phy_write_mmd(phydev, MDIO_MMD_VEND2, XWAY_MMD_LED0H, ledxh);
+ phy_write_mmd(phydev, MDIO_MMD_VEND2, XWAY_MMD_LED0L, ledxl);
+ phy_write_mmd(phydev, MDIO_MMD_VEND2, XWAY_MMD_LED1H, ledxh);
+ phy_write_mmd(phydev, MDIO_MMD_VEND2, XWAY_MMD_LED1L, ledxl);
+ phy_write_mmd(phydev, MDIO_MMD_VEND2, XWAY_MMD_LED2H, ledxh);
+ phy_write_mmd(phydev, MDIO_MMD_VEND2, XWAY_MMD_LED2L, ledxl);
return 0;
}
diff --git a/drivers/net/phy/mdio-bcm-unimac.c b/drivers/net/phy/mdio-bcm-unimac.c
index 8c73b2e..3439523 100644
--- a/drivers/net/phy/mdio-bcm-unimac.c
+++ b/drivers/net/phy/mdio-bcm-unimac.c
@@ -1,7 +1,7 @@
/*
* Broadcom UniMAC MDIO bus controller driver
*
- * Copyright (C) 2014, Broadcom Corporation
+ * Copyright (C) 2014-2017 Broadcom
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -228,6 +228,7 @@
}
static const struct of_device_id unimac_mdio_ids[] = {
+ { .compatible = "brcm,genet-mdio-v5", },
{ .compatible = "brcm,genet-mdio-v4", },
{ .compatible = "brcm,genet-mdio-v3", },
{ .compatible = "brcm,genet-mdio-v2", },
diff --git a/drivers/net/phy/mdio-boardinfo.c b/drivers/net/phy/mdio-boardinfo.c
index 61941e2..1861f38 100644
--- a/drivers/net/phy/mdio-boardinfo.c
+++ b/drivers/net/phy/mdio-boardinfo.c
@@ -24,10 +24,12 @@
* @mdiodev: MDIO device pointer
* Context: can sleep
*/
-void mdiobus_setup_mdiodev_from_board_info(struct mii_bus *bus)
+void mdiobus_setup_mdiodev_from_board_info(struct mii_bus *bus,
+ int (*cb)
+ (struct mii_bus *bus,
+ struct mdio_board_info *bi))
{
struct mdio_board_entry *be;
- struct mdio_device *mdiodev;
struct mdio_board_info *bi;
int ret;
@@ -38,23 +40,14 @@
if (strcmp(bus->id, bi->bus_id))
continue;
- mdiodev = mdio_device_create(bus, bi->mdio_addr);
- if (IS_ERR(mdiodev))
+ ret = cb(bus, bi);
+ if (ret)
continue;
- strncpy(mdiodev->modalias, bi->modalias,
- sizeof(mdiodev->modalias));
- mdiodev->bus_match = mdio_device_bus_match;
- mdiodev->dev.platform_data = (void *)bi->platform_data;
-
- ret = mdio_device_register(mdiodev);
- if (ret) {
- mdio_device_free(mdiodev);
- continue;
- }
}
mutex_unlock(&mdio_board_lock);
}
+EXPORT_SYMBOL(mdiobus_setup_mdiodev_from_board_info);
/**
* mdio_register_board_info - register MDIO devices for a given board
diff --git a/drivers/net/phy/mdio-boardinfo.h b/drivers/net/phy/mdio-boardinfo.h
index 00f9816..3a7f1439 100644
--- a/drivers/net/phy/mdio-boardinfo.h
+++ b/drivers/net/phy/mdio-boardinfo.h
@@ -14,6 +14,9 @@
struct mdio_board_info board_info;
};
-void mdiobus_setup_mdiodev_from_board_info(struct mii_bus *bus);
+void mdiobus_setup_mdiodev_from_board_info(struct mii_bus *bus,
+ int (*cb)
+ (struct mii_bus *bus,
+ struct mdio_board_info *bi));
#endif /* __MDIO_BOARD_INFO_H */
diff --git a/drivers/net/phy/mdio-xgene.c b/drivers/net/phy/mdio-xgene.c
index f095051..3e2ac07 100644
--- a/drivers/net/phy/mdio-xgene.c
+++ b/drivers/net/phy/mdio-xgene.c
@@ -229,7 +229,7 @@
val = SET_VAL(HSTPHYADX, phy_id) | SET_VAL(HSTREGADX, reg) |
SET_VAL(HSTMIIMWRDAT, data);
- xgene_enet_wr_mdio_csr(addr, MIIM_FIELD_ADDR, data);
+ xgene_enet_wr_mdio_csr(addr, MIIM_FIELD_ADDR, val);
val = HSTLDCMD | SET_VAL(HSTMIIMCMD, MIIM_CMD_LEGACY_WRITE);
xgene_enet_wr_mdio_csr(addr, MIIM_COMMAND_ADDR, val);
diff --git a/drivers/net/phy/mdio_bus.c b/drivers/net/phy/mdio_bus.c
index fa7d51f..a898e5c 100644
--- a/drivers/net/phy/mdio_bus.c
+++ b/drivers/net/phy/mdio_bus.c
@@ -22,8 +22,11 @@
#include <linux/init.h>
#include <linux/delay.h>
#include <linux/device.h>
+#include <linux/gpio.h>
+#include <linux/gpio/consumer.h>
#include <linux/of_device.h>
#include <linux/of_mdio.h>
+#include <linux/of_gpio.h>
#include <linux/netdevice.h>
#include <linux/etherdevice.h>
#include <linux/skbuff.h>
@@ -290,6 +293,36 @@
#endif
/**
+ * mdiobus_create_device_from_board_info - create a full MDIO device given
+ * a mdio_board_info structure
+ * @bus: MDIO bus to create the devices on
+ * @bi: mdio_board_info structure describing the devices
+ *
+ * Returns 0 on success or < 0 on error.
+ */
+static int mdiobus_create_device(struct mii_bus *bus,
+ struct mdio_board_info *bi)
+{
+ struct mdio_device *mdiodev;
+ int ret = 0;
+
+ mdiodev = mdio_device_create(bus, bi->mdio_addr);
+ if (IS_ERR(mdiodev))
+ return -ENODEV;
+
+ strncpy(mdiodev->modalias, bi->modalias,
+ sizeof(mdiodev->modalias));
+ mdiodev->bus_match = mdio_device_bus_match;
+ mdiodev->dev.platform_data = (void *)bi->platform_data;
+
+ ret = mdio_device_register(mdiodev);
+ if (ret)
+ mdio_device_free(mdiodev);
+
+ return ret;
+}
+
+/**
* __mdiobus_register - bring up all the PHYs on a given bus and attach them to bus
* @bus: target mii_bus
* @owner: module containing bus accessor functions
@@ -307,6 +340,7 @@
{
struct mdio_device *mdiodev;
int i, err;
+ struct gpio_desc *gpiod;
if (NULL == bus || NULL == bus->name ||
NULL == bus->read || NULL == bus->write)
@@ -333,6 +367,35 @@
if (bus->reset)
bus->reset(bus);
+ /* de-assert bus level PHY GPIO resets */
+ if (bus->num_reset_gpios > 0) {
+ bus->reset_gpiod = devm_kcalloc(&bus->dev,
+ bus->num_reset_gpios,
+ sizeof(struct gpio_desc *),
+ GFP_KERNEL);
+ if (!bus->reset_gpiod)
+ return -ENOMEM;
+ }
+
+ for (i = 0; i < bus->num_reset_gpios; i++) {
+ gpiod = devm_gpiod_get_index(&bus->dev, "reset", i,
+ GPIOD_OUT_LOW);
+ if (IS_ERR(gpiod)) {
+ err = PTR_ERR(gpiod);
+ if (err != -ENOENT) {
+ dev_err(&bus->dev,
+ "mii_bus %s couldn't get reset GPIO\n",
+ bus->id);
+ return err;
+ }
+ } else {
+ bus->reset_gpiod[i] = gpiod;
+ gpiod_set_value_cansleep(gpiod, 1);
+ udelay(bus->reset_delay_us);
+ gpiod_set_value_cansleep(gpiod, 0);
+ }
+ }
+
for (i = 0; i < PHY_MAX_ADDR; i++) {
if ((bus->phy_mask & (1 << i)) == 0) {
struct phy_device *phydev;
@@ -345,7 +408,7 @@
}
}
- mdiobus_setup_mdiodev_from_board_info(bus);
+ mdiobus_setup_mdiodev_from_board_info(bus, mdiobus_create_device);
bus->state = MDIOBUS_REGISTERED;
pr_info("%s: probed\n", bus->name);
@@ -360,6 +423,13 @@
mdiodev->device_remove(mdiodev);
mdiodev->device_free(mdiodev);
}
+
+ /* Put PHYs in RESET to save power */
+ for (i = 0; i < bus->num_reset_gpios; i++) {
+ if (bus->reset_gpiod[i])
+ gpiod_set_value_cansleep(bus->reset_gpiod[i], 1);
+ }
+
device_del(&bus->dev);
return err;
}
@@ -381,6 +451,13 @@
mdiodev->device_remove(mdiodev);
mdiodev->device_free(mdiodev);
}
+
+ /* Put PHYs in RESET to save power */
+ for (i = 0; i < bus->num_reset_gpios; i++) {
+ if (bus->reset_gpiod[i])
+ gpiod_set_value_cansleep(bus->reset_gpiod[i], 1);
+ }
+
device_del(&bus->dev);
}
EXPORT_SYMBOL(mdiobus_unregister);
@@ -648,9 +725,18 @@
return ret;
}
+EXPORT_SYMBOL_GPL(mdio_bus_init);
+#if IS_ENABLED(CONFIG_PHYLIB)
void mdio_bus_exit(void)
{
class_unregister(&mdio_bus_class);
bus_unregister(&mdio_bus_type);
}
+EXPORT_SYMBOL_GPL(mdio_bus_exit);
+#else
+module_init(mdio_bus_init);
+/* no module_exit, intentional */
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("MDIO bus/device layer");
+#endif
diff --git a/drivers/net/phy/micrel.c b/drivers/net/phy/micrel.c
index 1326d99..6a5fd18 100644
--- a/drivers/net/phy/micrel.c
+++ b/drivers/net/phy/micrel.c
@@ -297,17 +297,6 @@
if (priv->led_mode >= 0)
kszphy_setup_led(phydev, type->led_mode_reg, priv->led_mode);
- if (phy_interrupt_is_valid(phydev)) {
- int ctl = phy_read(phydev, MII_BMCR);
-
- if (ctl < 0)
- return ctl;
-
- ret = phy_write(phydev, MII_BMCR, ctl & ~BMCR_ANENABLE);
- if (ret < 0)
- return ret;
- }
-
return 0;
}
@@ -637,8 +626,7 @@
* MMD extended PHY registers.
*/
static int
-ksz9021_rd_mmd_phyreg(struct phy_device *phydev, int ptrad, int devnum,
- int regnum)
+ksz9021_rd_mmd_phyreg(struct phy_device *phydev, int devad, u16 regnum)
{
return -1;
}
@@ -646,10 +634,10 @@
/* This routine does nothing since the Micrel ksz9021 does not support
* standard IEEE MMD extended PHY registers.
*/
-static void
-ksz9021_wr_mmd_phyreg(struct phy_device *phydev, int ptrad, int devnum,
- int regnum, u32 val)
+static int
+ksz9021_wr_mmd_phyreg(struct phy_device *phydev, int devad, u16 regnum, u16 val)
{
+ return -1;
}
static int kszphy_get_sset_count(struct phy_device *phydev)
@@ -957,8 +945,8 @@
.get_stats = kszphy_get_stats,
.suspend = genphy_suspend,
.resume = genphy_resume,
- .read_mmd_indirect = ksz9021_rd_mmd_phyreg,
- .write_mmd_indirect = ksz9021_wr_mmd_phyreg,
+ .read_mmd = ksz9021_rd_mmd_phyreg,
+ .write_mmd = ksz9021_wr_mmd_phyreg,
}, {
.phy_id = PHY_ID_KSZ9031,
.phy_id_mask = MICREL_PHY_ID_MASK,
diff --git a/drivers/net/phy/microchip.c b/drivers/net/phy/microchip.c
index 324fbf6..2b2f543 100644
--- a/drivers/net/phy/microchip.c
+++ b/drivers/net/phy/microchip.c
@@ -78,9 +78,8 @@
priv->wolopts = 0;
/* these values can be used to identify internal PHY */
- priv->chip_id = phy_read_mmd_indirect(phydev, LAN88XX_MMD3_CHIP_ID, 3);
- priv->chip_rev = phy_read_mmd_indirect(phydev, LAN88XX_MMD3_CHIP_REV,
- 3);
+ priv->chip_id = phy_read_mmd(phydev, 3, LAN88XX_MMD3_CHIP_ID);
+ priv->chip_rev = phy_read_mmd(phydev, 3, LAN88XX_MMD3_CHIP_REV);
phydev->priv = priv;
diff --git a/drivers/net/phy/phy-core.c b/drivers/net/phy/phy-core.c
new file mode 100644
index 0000000..6739b73
--- /dev/null
+++ b/drivers/net/phy/phy-core.c
@@ -0,0 +1,101 @@
+/*
+ * Core PHY library, taken from phy.c
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ */
+#include <linux/export.h>
+#include <linux/phy.h>
+
+static void mmd_phy_indirect(struct mii_bus *bus, int phy_addr, int devad,
+ u16 regnum)
+{
+ /* Write the desired MMD Devad */
+ bus->write(bus, phy_addr, MII_MMD_CTRL, devad);
+
+ /* Write the desired MMD register address */
+ bus->write(bus, phy_addr, MII_MMD_DATA, regnum);
+
+ /* Select the Function : DATA with no post increment */
+ bus->write(bus, phy_addr, MII_MMD_CTRL, devad | MII_MMD_CTRL_NOINCR);
+}
+
+/**
+ * phy_read_mmd - Convenience function for reading a register
+ * from an MMD on a given PHY.
+ * @phydev: The phy_device struct
+ * @devad: The MMD to read from (0..31)
+ * @regnum: The register on the MMD to read (0..65535)
+ *
+ * Same rules as for phy_read();
+ */
+int phy_read_mmd(struct phy_device *phydev, int devad, u32 regnum)
+{
+ int val;
+
+ if (regnum > (u16)~0 || devad > 32)
+ return -EINVAL;
+
+ if (phydev->drv->read_mmd) {
+ val = phydev->drv->read_mmd(phydev, devad, regnum);
+ } else if (phydev->is_c45) {
+ u32 addr = MII_ADDR_C45 | (devad << 16) | (regnum & 0xffff);
+
+ val = mdiobus_read(phydev->mdio.bus, phydev->mdio.addr, addr);
+ } else {
+ struct mii_bus *bus = phydev->mdio.bus;
+ int phy_addr = phydev->mdio.addr;
+
+ mutex_lock(&bus->mdio_lock);
+ mmd_phy_indirect(bus, phy_addr, devad, regnum);
+
+ /* Read the content of the MMD's selected register */
+ val = bus->read(bus, phy_addr, MII_MMD_DATA);
+ mutex_unlock(&bus->mdio_lock);
+ }
+ return val;
+}
+EXPORT_SYMBOL(phy_read_mmd);
+
+/**
+ * phy_write_mmd - Convenience function for writing a register
+ * on an MMD on a given PHY.
+ * @phydev: The phy_device struct
+ * @devad: The MMD to read from
+ * @regnum: The register on the MMD to read
+ * @val: value to write to @regnum
+ *
+ * Same rules as for phy_write();
+ */
+int phy_write_mmd(struct phy_device *phydev, int devad, u32 regnum, u16 val)
+{
+ int ret;
+
+ if (regnum > (u16)~0 || devad > 32)
+ return -EINVAL;
+
+ if (phydev->drv->write_mmd) {
+ ret = phydev->drv->write_mmd(phydev, devad, regnum, val);
+ } else if (phydev->is_c45) {
+ u32 addr = MII_ADDR_C45 | (devad << 16) | (regnum & 0xffff);
+
+ ret = mdiobus_write(phydev->mdio.bus, phydev->mdio.addr,
+ addr, val);
+ } else {
+ struct mii_bus *bus = phydev->mdio.bus;
+ int phy_addr = phydev->mdio.addr;
+
+ mutex_lock(&bus->mdio_lock);
+ mmd_phy_indirect(bus, phy_addr, devad, regnum);
+
+ /* Write the data into MMD's selected register */
+ bus->write(bus, phy_addr, MII_MMD_DATA, val);
+ mutex_unlock(&bus->mdio_lock);
+
+ ret = 0;
+ }
+ return ret;
+}
+EXPORT_SYMBOL(phy_write_mmd);
diff --git a/drivers/net/phy/phy.c b/drivers/net/phy/phy.c
index a2bfc82..82ab8fb 100644
--- a/drivers/net/phy/phy.c
+++ b/drivers/net/phy/phy.c
@@ -50,8 +50,22 @@
return "1Gbps";
case SPEED_2500:
return "2.5Gbps";
+ case SPEED_5000:
+ return "5Gbps";
case SPEED_10000:
return "10Gbps";
+ case SPEED_20000:
+ return "20Gbps";
+ case SPEED_25000:
+ return "25Gbps";
+ case SPEED_40000:
+ return "40Gbps";
+ case SPEED_50000:
+ return "50Gbps";
+ case SPEED_56000:
+ return "56Gbps";
+ case SPEED_100000:
+ return "100Gbps";
case SPEED_UNKNOWN:
return "Unknown";
default:
@@ -162,7 +176,9 @@
u32 setting;
};
-/* A mapping of all SUPPORTED settings to speed/duplex */
+/* A mapping of all SUPPORTED settings to speed/duplex. This table
+ * must be grouped by speed and sorted in descending match priority
+ * - iow, descending speed. */
static const struct phy_setting settings[] = {
{
.speed = SPEED_10000,
@@ -221,45 +237,70 @@
},
};
-#define MAX_NUM_SETTINGS ARRAY_SIZE(settings)
-
/**
- * phy_find_setting - find a PHY settings array entry that matches speed & duplex
+ * phy_lookup_setting - lookup a PHY setting
* @speed: speed to match
* @duplex: duplex to match
+ * @feature: allowed link modes
+ * @exact: an exact match is required
*
- * Description: Searches the settings array for the setting which
- * matches the desired speed and duplex, and returns the index
- * of that setting. Returns the index of the last setting if
- * none of the others match.
+ * Search the settings array for a setting that matches the speed and
+ * duplex, and which is supported.
+ *
+ * If @exact is unset, either an exact match or %NULL for no match will
+ * be returned.
+ *
+ * If @exact is set, an exact match, the fastest supported setting at
+ * or below the specified speed, the slowest supported setting, or if
+ * they all fail, %NULL will be returned.
*/
-static inline unsigned int phy_find_setting(int speed, int duplex)
+static const struct phy_setting *
+phy_lookup_setting(int speed, int duplex, u32 features, bool exact)
{
- unsigned int idx = 0;
+ const struct phy_setting *p, *match = NULL, *last = NULL;
+ int i;
- while (idx < ARRAY_SIZE(settings) &&
- (settings[idx].speed != speed || settings[idx].duplex != duplex))
- idx++;
+ for (i = 0, p = settings; i < ARRAY_SIZE(settings); i++, p++) {
+ if (p->setting & features) {
+ last = p;
+ if (p->speed == speed && p->duplex == duplex) {
+ /* Exact match for speed and duplex */
+ match = p;
+ break;
+ } else if (!exact) {
+ if (!match && p->speed <= speed)
+ /* Candidate */
+ match = p;
- return idx < MAX_NUM_SETTINGS ? idx : MAX_NUM_SETTINGS - 1;
+ if (p->speed < speed)
+ break;
+ }
+ }
+ }
+
+ if (!match && !exact)
+ match = last;
+
+ return match;
}
/**
- * phy_find_valid - find a PHY setting that matches the requested features mask
- * @idx: The first index in settings[] to search
- * @features: A mask of the valid settings
+ * phy_find_valid - find a PHY setting that matches the requested parameters
+ * @speed: desired speed
+ * @duplex: desired duplex
+ * @supported: mask of supported link modes
*
- * Description: Returns the index of the first valid setting less
- * than or equal to the one pointed to by idx, as determined by
- * the mask in features. Returns the index of the last setting
- * if nothing else matches.
+ * Locate a supported phy setting that is, in priority order:
+ * - an exact match for the specified speed and duplex mode
+ * - a match for the specified speed, or slower speed
+ * - the slowest supported speed
+ * Returns the matched phy_setting entry, or %NULL if no supported phy
+ * settings were found.
*/
-static inline unsigned int phy_find_valid(unsigned int idx, u32 features)
+static const struct phy_setting *
+phy_find_valid(int speed, int duplex, u32 supported)
{
- while (idx < MAX_NUM_SETTINGS && !(settings[idx].setting & features))
- idx++;
-
- return idx < MAX_NUM_SETTINGS ? idx : MAX_NUM_SETTINGS - 1;
+ return phy_lookup_setting(speed, duplex, supported, false);
}
/**
@@ -279,20 +320,11 @@
unsigned int count = 0;
unsigned int idx = 0;
- while (idx < MAX_NUM_SETTINGS && count < size) {
- idx = phy_find_valid(idx, phy->supported);
-
- if (!(settings[idx].setting & phy->supported))
- break;
-
+ for (idx = 0; idx < ARRAY_SIZE(settings) && count < size; idx++)
/* Assumes settings are grouped by speed */
- if ((count == 0) ||
- (speeds[count - 1] != settings[idx].speed)) {
- speeds[count] = settings[idx].speed;
- count++;
- }
- idx++;
- }
+ if ((settings[idx].setting & phy->supported) &&
+ (count == 0 || speeds[count - 1] != settings[idx].speed))
+ speeds[count++] = settings[idx].speed;
return count;
}
@@ -308,12 +340,7 @@
*/
static inline bool phy_check_valid(int speed, int duplex, u32 features)
{
- unsigned int idx;
-
- idx = phy_find_valid(phy_find_setting(speed, duplex), features);
-
- return settings[idx].speed == speed && settings[idx].duplex == duplex &&
- (settings[idx].setting & features);
+ return !!phy_lookup_setting(speed, duplex, features, true);
}
/**
@@ -326,18 +353,22 @@
*/
static void phy_sanitize_settings(struct phy_device *phydev)
{
+ const struct phy_setting *setting;
u32 features = phydev->supported;
- unsigned int idx;
/* Sanitize settings based on PHY capabilities */
if ((features & SUPPORTED_Autoneg) == 0)
phydev->autoneg = AUTONEG_DISABLE;
- idx = phy_find_valid(phy_find_setting(phydev->speed, phydev->duplex),
- features);
-
- phydev->speed = settings[idx].speed;
- phydev->duplex = settings[idx].duplex;
+ setting = phy_find_valid(phydev->speed, phydev->duplex, features);
+ if (setting) {
+ phydev->speed = setting->speed;
+ phydev->duplex = setting->duplex;
+ } else {
+ /* We failed to find anything (no supported speeds?) */
+ phydev->speed = SPEED_UNKNOWN;
+ phydev->duplex = DUPLEX_UNKNOWN;
+ }
}
/**
@@ -591,16 +622,18 @@
EXPORT_SYMBOL(phy_mii_ioctl);
/**
- * phy_start_aneg - start auto-negotiation for this PHY device
+ * phy_start_aneg_priv - start auto-negotiation for this PHY device
* @phydev: the phy_device struct
+ * @sync: indicate whether we should wait for the workqueue cancelation
*
* Description: Sanitizes the settings (if we're not autonegotiating
* them), and then calls the driver's config_aneg function.
* If the PHYCONTROL Layer is operating, we change the state to
* reflect the beginning of Auto-negotiation or forcing.
*/
-int phy_start_aneg(struct phy_device *phydev)
+static int phy_start_aneg_priv(struct phy_device *phydev, bool sync)
{
+ bool trigger = 0;
int err;
if (!phydev->drv)
@@ -628,10 +661,40 @@
}
}
+ /* Re-schedule a PHY state machine to check PHY status because
+ * negotiation may already be done and aneg interrupt may not be
+ * generated.
+ */
+ if (phy_interrupt_is_valid(phydev) && (phydev->state == PHY_AN)) {
+ err = phy_aneg_done(phydev);
+ if (err > 0) {
+ trigger = true;
+ err = 0;
+ }
+ }
+
out_unlock:
mutex_unlock(&phydev->lock);
+
+ if (trigger)
+ phy_trigger_machine(phydev, sync);
+
return err;
}
+
+/**
+ * phy_start_aneg - start auto-negotiation for this PHY device
+ * @phydev: the phy_device struct
+ *
+ * Description: Sanitizes the settings (if we're not autonegotiating
+ * them), and then calls the driver's config_aneg function.
+ * If the PHYCONTROL Layer is operating, we change the state to
+ * reflect the beginning of Auto-negotiation or forcing.
+ */
+int phy_start_aneg(struct phy_device *phydev)
+{
+ return phy_start_aneg_priv(phydev, true);
+}
EXPORT_SYMBOL(phy_start_aneg);
/**
@@ -659,7 +722,7 @@
* state machine runs.
*/
-static void phy_trigger_machine(struct phy_device *phydev, bool sync)
+void phy_trigger_machine(struct phy_device *phydev, bool sync)
{
if (sync)
cancel_delayed_work_sync(&phydev->state_queue);
@@ -1154,7 +1217,7 @@
mutex_unlock(&phydev->lock);
if (needs_aneg)
- err = phy_start_aneg(phydev);
+ err = phy_start_aneg_priv(phydev, false);
else if (do_suspend)
phy_suspend(phydev);
@@ -1192,91 +1255,6 @@
}
EXPORT_SYMBOL(phy_mac_interrupt);
-static inline void mmd_phy_indirect(struct mii_bus *bus, int prtad, int devad,
- int addr)
-{
- /* Write the desired MMD Devad */
- bus->write(bus, addr, MII_MMD_CTRL, devad);
-
- /* Write the desired MMD register address */
- bus->write(bus, addr, MII_MMD_DATA, prtad);
-
- /* Select the Function : DATA with no post increment */
- bus->write(bus, addr, MII_MMD_CTRL, (devad | MII_MMD_CTRL_NOINCR));
-}
-
-/**
- * phy_read_mmd_indirect - reads data from the MMD registers
- * @phydev: The PHY device bus
- * @prtad: MMD Address
- * @devad: MMD DEVAD
- *
- * Description: it reads data from the MMD registers (clause 22 to access to
- * clause 45) of the specified phy address.
- * To read these register we have:
- * 1) Write reg 13 // DEVAD
- * 2) Write reg 14 // MMD Address
- * 3) Write reg 13 // MMD Data Command for MMD DEVAD
- * 3) Read reg 14 // Read MMD data
- */
-int phy_read_mmd_indirect(struct phy_device *phydev, int prtad, int devad)
-{
- struct phy_driver *phydrv = phydev->drv;
- int addr = phydev->mdio.addr;
- int value = -1;
-
- if (!phydrv->read_mmd_indirect) {
- struct mii_bus *bus = phydev->mdio.bus;
-
- mutex_lock(&bus->mdio_lock);
- mmd_phy_indirect(bus, prtad, devad, addr);
-
- /* Read the content of the MMD's selected register */
- value = bus->read(bus, addr, MII_MMD_DATA);
- mutex_unlock(&bus->mdio_lock);
- } else {
- value = phydrv->read_mmd_indirect(phydev, prtad, devad, addr);
- }
- return value;
-}
-EXPORT_SYMBOL(phy_read_mmd_indirect);
-
-/**
- * phy_write_mmd_indirect - writes data to the MMD registers
- * @phydev: The PHY device
- * @prtad: MMD Address
- * @devad: MMD DEVAD
- * @data: data to write in the MMD register
- *
- * Description: Write data from the MMD registers of the specified
- * phy address.
- * To write these register we have:
- * 1) Write reg 13 // DEVAD
- * 2) Write reg 14 // MMD Address
- * 3) Write reg 13 // MMD Data Command for MMD DEVAD
- * 3) Write reg 14 // Write MMD data
- */
-void phy_write_mmd_indirect(struct phy_device *phydev, int prtad,
- int devad, u32 data)
-{
- struct phy_driver *phydrv = phydev->drv;
- int addr = phydev->mdio.addr;
-
- if (!phydrv->write_mmd_indirect) {
- struct mii_bus *bus = phydev->mdio.bus;
-
- mutex_lock(&bus->mdio_lock);
- mmd_phy_indirect(bus, prtad, devad, addr);
-
- /* Write the data into MMD's selected register */
- bus->write(bus, addr, MII_MMD_DATA, data);
- mutex_unlock(&bus->mdio_lock);
- } else {
- phydrv->write_mmd_indirect(phydev, prtad, devad, addr, data);
- }
-}
-EXPORT_SYMBOL(phy_write_mmd_indirect);
-
/**
* phy_init_eee - init and check the EEE feature
* @phydev: target phy_device struct
@@ -1293,15 +1271,8 @@
return -EIO;
/* According to 802.3az,the EEE is supported only in full duplex-mode.
- * Also EEE feature is active when core is operating with MII, GMII
- * or RGMII (all kinds). Internal PHYs are also allowed to proceed and
- * should return an error if they do not support EEE.
*/
- if ((phydev->duplex == DUPLEX_FULL) &&
- ((phydev->interface == PHY_INTERFACE_MODE_MII) ||
- (phydev->interface == PHY_INTERFACE_MODE_GMII) ||
- phy_interface_is_rgmii(phydev) ||
- phy_is_internal(phydev))) {
+ if (phydev->duplex == DUPLEX_FULL) {
int eee_lp, eee_cap, eee_adv;
u32 lp, cap, adv;
int status;
@@ -1312,8 +1283,7 @@
return status;
/* First check if the EEE ability is supported */
- eee_cap = phy_read_mmd_indirect(phydev, MDIO_PCS_EEE_ABLE,
- MDIO_MMD_PCS);
+ eee_cap = phy_read_mmd(phydev, MDIO_MMD_PCS, MDIO_PCS_EEE_ABLE);
if (eee_cap <= 0)
goto eee_exit_err;
@@ -1324,13 +1294,11 @@
/* Check which link settings negotiated and verify it in
* the EEE advertising registers.
*/
- eee_lp = phy_read_mmd_indirect(phydev, MDIO_AN_EEE_LPABLE,
- MDIO_MMD_AN);
+ eee_lp = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_AN_EEE_LPABLE);
if (eee_lp <= 0)
goto eee_exit_err;
- eee_adv = phy_read_mmd_indirect(phydev, MDIO_AN_EEE_ADV,
- MDIO_MMD_AN);
+ eee_adv = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_AN_EEE_ADV);
if (eee_adv <= 0)
goto eee_exit_err;
@@ -1343,14 +1311,12 @@
/* Configure the PHY to stop receiving xMII
* clock while it is signaling LPI.
*/
- int val = phy_read_mmd_indirect(phydev, MDIO_CTRL1,
- MDIO_MMD_PCS);
+ int val = phy_read_mmd(phydev, MDIO_MMD_PCS, MDIO_CTRL1);
if (val < 0)
return val;
val |= MDIO_PCS_CTRL1_CLKSTOP_EN;
- phy_write_mmd_indirect(phydev, MDIO_CTRL1,
- MDIO_MMD_PCS, val);
+ phy_write_mmd(phydev, MDIO_MMD_PCS, MDIO_CTRL1, val);
}
return 0; /* EEE supported */
@@ -1372,7 +1338,7 @@
if (!phydev->drv)
return -EIO;
- return phy_read_mmd_indirect(phydev, MDIO_PCS_EEE_WK_ERR, MDIO_MMD_PCS);
+ return phy_read_mmd(phydev, MDIO_MMD_PCS, MDIO_PCS_EEE_WK_ERR);
}
EXPORT_SYMBOL(phy_get_eee_err);
@@ -1392,19 +1358,19 @@
return -EIO;
/* Get Supported EEE */
- val = phy_read_mmd_indirect(phydev, MDIO_PCS_EEE_ABLE, MDIO_MMD_PCS);
+ val = phy_read_mmd(phydev, MDIO_MMD_PCS, MDIO_PCS_EEE_ABLE);
if (val < 0)
return val;
data->supported = mmd_eee_cap_to_ethtool_sup_t(val);
/* Get advertisement EEE */
- val = phy_read_mmd_indirect(phydev, MDIO_AN_EEE_ADV, MDIO_MMD_AN);
+ val = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_AN_EEE_ADV);
if (val < 0)
return val;
data->advertised = mmd_eee_adv_to_ethtool_adv_t(val);
/* Get LP advertisement EEE */
- val = phy_read_mmd_indirect(phydev, MDIO_AN_EEE_LPABLE, MDIO_MMD_AN);
+ val = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_AN_EEE_LPABLE);
if (val < 0)
return val;
data->lp_advertised = mmd_eee_adv_to_ethtool_adv_t(val);
@@ -1422,15 +1388,37 @@
*/
int phy_ethtool_set_eee(struct phy_device *phydev, struct ethtool_eee *data)
{
- int val = ethtool_adv_to_mmd_eee_adv_t(data->advertised);
+ int cap, old_adv, adv, ret;
if (!phydev->drv)
return -EIO;
- /* Mask prohibited EEE modes */
- val &= ~phydev->eee_broken_modes;
+ /* Get Supported EEE */
+ cap = phy_read_mmd(phydev, MDIO_MMD_PCS, MDIO_PCS_EEE_ABLE);
+ if (cap < 0)
+ return cap;
- phy_write_mmd_indirect(phydev, MDIO_AN_EEE_ADV, MDIO_MMD_AN, val);
+ old_adv = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_AN_EEE_ADV);
+ if (old_adv < 0)
+ return old_adv;
+
+ adv = ethtool_adv_to_mmd_eee_adv_t(data->advertised) & cap;
+
+ /* Mask prohibited EEE modes */
+ adv &= ~phydev->eee_broken_modes;
+
+ if (old_adv != adv) {
+ ret = phy_write_mmd(phydev, MDIO_MMD_AN, MDIO_AN_EEE_ADV, adv);
+ if (ret < 0)
+ return ret;
+
+ /* Restart autonegotiation so the new modes get sent to the
+ * link partner.
+ */
+ ret = genphy_restart_aneg(phydev);
+ if (ret < 0)
+ return ret;
+ }
return 0;
}
diff --git a/drivers/net/phy/phy_device.c b/drivers/net/phy/phy_device.c
index 5198ccf..1219eea 100644
--- a/drivers/net/phy/phy_device.c
+++ b/drivers/net/phy/phy_device.c
@@ -1217,7 +1217,7 @@
* supported by the phy. If we read 0, EEE is not advertised
* In both case, we don't need to continue
*/
- adv = phy_read_mmd_indirect(phydev, MDIO_AN_EEE_ADV, MDIO_MMD_AN);
+ adv = phy_read_mmd(phydev, MDIO_MMD_AN, MDIO_AN_EEE_ADV);
if (adv <= 0)
return 0;
@@ -1228,7 +1228,7 @@
if (old_adv == adv)
return 0;
- phy_write_mmd_indirect(phydev, MDIO_AN_EEE_ADV, MDIO_MMD_AN, adv);
+ phy_write_mmd(phydev, MDIO_MMD_AN, MDIO_AN_EEE_ADV, adv);
return 1;
}
diff --git a/drivers/net/phy/smsc.c b/drivers/net/phy/smsc.c
index fb32eaf..cef6967 100644
--- a/drivers/net/phy/smsc.c
+++ b/drivers/net/phy/smsc.c
@@ -20,6 +20,7 @@
#include <linux/module.h>
#include <linux/mii.h>
#include <linux/ethtool.h>
+#include <linux/of.h>
#include <linux/phy.h>
#include <linux/netdevice.h>
#include <linux/smscphy.h>
diff --git a/drivers/net/team/team.c b/drivers/net/team/team.c
index f8c81f1..6c5d5ef 100644
--- a/drivers/net/team/team.c
+++ b/drivers/net/team/team.c
@@ -2361,8 +2361,10 @@
hdr = genlmsg_put(skb, portid, seq, &team_nl_family, flags | NLM_F_MULTI,
TEAM_CMD_OPTIONS_GET);
- if (!hdr)
+ if (!hdr) {
+ nlmsg_free(skb);
return -EMSGSIZE;
+ }
if (nla_put_u32(skb, TEAM_ATTR_TEAM_IFINDEX, team->dev->ifindex))
goto nla_put_failure;
@@ -2474,7 +2476,8 @@
goto team_put;
}
err = nla_parse_nested(opt_attrs, TEAM_ATTR_OPTION_MAX,
- nl_option, team_nl_option_policy);
+ nl_option, team_nl_option_policy,
+ info->extack);
if (err)
goto team_put;
if (!opt_attrs[TEAM_ATTR_OPTION_NAME] ||
@@ -2634,8 +2637,10 @@
hdr = genlmsg_put(skb, portid, seq, &team_nl_family, flags | NLM_F_MULTI,
TEAM_CMD_PORT_LIST_GET);
- if (!hdr)
+ if (!hdr) {
+ nlmsg_free(skb);
return -EMSGSIZE;
+ }
if (nla_put_u32(skb, TEAM_ATTR_TEAM_IFINDEX, team->dev->ifindex))
goto nla_put_failure;
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index cc88cd7..bbd707b 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -2444,18 +2444,16 @@
/* ethtool interface */
-static int tun_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int tun_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
- cmd->supported = 0;
- cmd->advertising = 0;
- ethtool_cmd_speed_set(cmd, SPEED_10);
- cmd->duplex = DUPLEX_FULL;
- cmd->port = PORT_TP;
- cmd->phy_address = 0;
- cmd->transceiver = XCVR_INTERNAL;
- cmd->autoneg = AUTONEG_DISABLE;
- cmd->maxtxpkt = 0;
- cmd->maxrxpkt = 0;
+ ethtool_link_ksettings_zero_link_mode(cmd, supported);
+ ethtool_link_ksettings_zero_link_mode(cmd, advertising);
+ cmd->base.speed = SPEED_10;
+ cmd->base.duplex = DUPLEX_FULL;
+ cmd->base.port = PORT_TP;
+ cmd->base.phy_address = 0;
+ cmd->base.autoneg = AUTONEG_DISABLE;
return 0;
}
@@ -2518,7 +2516,6 @@
}
static const struct ethtool_ops tun_ethtool_ops = {
- .get_settings = tun_get_settings,
.get_drvinfo = tun_get_drvinfo,
.get_msglevel = tun_get_msglevel,
.set_msglevel = tun_set_msglevel,
@@ -2526,6 +2523,7 @@
.get_ts_info = ethtool_op_get_ts_info,
.get_coalesce = tun_get_coalesce,
.set_coalesce = tun_set_coalesce,
+ .get_link_ksettings = tun_get_link_ksettings,
};
static int tun_queue_resize(struct tun_struct *tun)
diff --git a/drivers/net/usb/Kconfig b/drivers/net/usb/Kconfig
index 3dd490f5..f28bd74 100644
--- a/drivers/net/usb/Kconfig
+++ b/drivers/net/usb/Kconfig
@@ -369,7 +369,7 @@
optionally with LEDs that indicate traffic
config USB_NET_PLUSB
- tristate "Prolific PL-2301/2302/25A1 based cables"
+ tristate "Prolific PL-2301/2302/25A1/27A1 based cables"
# if the handshake/init/reset problems, from original 'plusb',
# are ever resolved ... then remove "experimental"
depends on USB_USBNET
diff --git a/drivers/net/usb/asix_devices.c b/drivers/net/usb/asix_devices.c
index 0dd5106..a3aa0a2 100644
--- a/drivers/net/usb/asix_devices.c
+++ b/drivers/net/usb/asix_devices.c
@@ -136,9 +136,9 @@
.get_eeprom_len = asix_get_eeprom_len,
.get_eeprom = asix_get_eeprom,
.set_eeprom = asix_set_eeprom,
- .get_settings = usbnet_get_settings,
- .set_settings = usbnet_set_settings,
.nway_reset = usbnet_nway_reset,
+ .get_link_ksettings = usbnet_get_link_ksettings,
+ .set_link_ksettings = usbnet_set_link_ksettings,
};
static void ax88172_set_multicast(struct net_device *net)
@@ -206,6 +206,7 @@
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
.ndo_change_mtu = usbnet_change_mtu,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
.ndo_do_ioctl = asix_ioctl,
@@ -301,9 +302,9 @@
.get_eeprom_len = asix_get_eeprom_len,
.get_eeprom = asix_get_eeprom,
.set_eeprom = asix_set_eeprom,
- .get_settings = usbnet_get_settings,
- .set_settings = usbnet_set_settings,
.nway_reset = usbnet_nway_reset,
+ .get_link_ksettings = usbnet_get_link_ksettings,
+ .set_link_ksettings = usbnet_set_link_ksettings,
};
static int ax88772_link_reset(struct usbnet *dev)
@@ -591,6 +592,7 @@
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
.ndo_change_mtu = usbnet_change_mtu,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_set_mac_address = asix_set_mac_address,
.ndo_validate_addr = eth_validate_addr,
.ndo_do_ioctl = asix_ioctl,
@@ -775,9 +777,9 @@
.get_eeprom_len = asix_get_eeprom_len,
.get_eeprom = asix_get_eeprom,
.set_eeprom = asix_set_eeprom,
- .get_settings = usbnet_get_settings,
- .set_settings = usbnet_set_settings,
.nway_reset = usbnet_nway_reset,
+ .get_link_ksettings = usbnet_get_link_ksettings,
+ .set_link_ksettings = usbnet_set_link_ksettings,
};
static int marvell_phy_init(struct usbnet *dev)
@@ -1044,6 +1046,7 @@
.ndo_stop = usbnet_stop,
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_set_mac_address = asix_set_mac_address,
.ndo_validate_addr = eth_validate_addr,
.ndo_set_rx_mode = asix_set_multicast,
diff --git a/drivers/net/usb/ax88172a.c b/drivers/net/usb/ax88172a.c
index 6308386..501576f 100644
--- a/drivers/net/usb/ax88172a.c
+++ b/drivers/net/usb/ax88172a.c
@@ -143,6 +143,7 @@
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
.ndo_change_mtu = usbnet_change_mtu,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_set_mac_address = asix_set_mac_address,
.ndo_validate_addr = eth_validate_addr,
.ndo_do_ioctl = ax88172a_ioctl,
diff --git a/drivers/net/usb/ax88179_178a.c b/drivers/net/usb/ax88179_178a.c
index a3a7db0..51cf600 100644
--- a/drivers/net/usb/ax88179_178a.c
+++ b/drivers/net/usb/ax88179_178a.c
@@ -620,16 +620,18 @@
return 0;
}
-static int ax88179_get_settings(struct net_device *net, struct ethtool_cmd *cmd)
+static int ax88179_get_link_ksettings(struct net_device *net,
+ struct ethtool_link_ksettings *cmd)
{
struct usbnet *dev = netdev_priv(net);
- return mii_ethtool_gset(&dev->mii, cmd);
+ return mii_ethtool_get_link_ksettings(&dev->mii, cmd);
}
-static int ax88179_set_settings(struct net_device *net, struct ethtool_cmd *cmd)
+static int ax88179_set_link_ksettings(struct net_device *net,
+ const struct ethtool_link_ksettings *cmd)
{
struct usbnet *dev = netdev_priv(net);
- return mii_ethtool_sset(&dev->mii, cmd);
+ return mii_ethtool_set_link_ksettings(&dev->mii, cmd);
}
static int
@@ -826,11 +828,11 @@
.set_wol = ax88179_set_wol,
.get_eeprom_len = ax88179_get_eeprom_len,
.get_eeprom = ax88179_get_eeprom,
- .get_settings = ax88179_get_settings,
- .set_settings = ax88179_set_settings,
.get_eee = ax88179_get_eee,
.set_eee = ax88179_set_eee,
.nway_reset = usbnet_nway_reset,
+ .get_link_ksettings = ax88179_get_link_ksettings,
+ .set_link_ksettings = ax88179_set_link_ksettings,
};
static void ax88179_set_multicast(struct net_device *net)
@@ -957,6 +959,7 @@
.ndo_stop = usbnet_stop,
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_change_mtu = ax88179_change_mtu,
.ndo_set_mac_address = ax88179_set_mac_addr,
.ndo_validate_addr = eth_validate_addr,
diff --git a/drivers/net/usb/catc.c b/drivers/net/usb/catc.c
index 0acc9b64..fce92f0 100644
--- a/drivers/net/usb/catc.c
+++ b/drivers/net/usb/catc.c
@@ -688,29 +688,34 @@
usb_make_path(catc->usbdev, info->bus_info, sizeof(info->bus_info));
}
-static int catc_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int catc_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct catc *catc = netdev_priv(dev);
if (!catc->is_f5u011)
return -EOPNOTSUPP;
- cmd->supported = SUPPORTED_10baseT_Half | SUPPORTED_TP;
- cmd->advertising = ADVERTISED_10baseT_Half | ADVERTISED_TP;
- ethtool_cmd_speed_set(cmd, SPEED_10);
- cmd->duplex = DUPLEX_HALF;
- cmd->port = PORT_TP;
- cmd->phy_address = 0;
- cmd->transceiver = XCVR_INTERNAL;
- cmd->autoneg = AUTONEG_DISABLE;
- cmd->maxtxpkt = 1;
- cmd->maxrxpkt = 1;
+ ethtool_link_ksettings_zero_link_mode(cmd, supported);
+ ethtool_link_ksettings_add_link_mode(cmd, supported, 10baseT_Half);
+ ethtool_link_ksettings_add_link_mode(cmd, supported, TP);
+
+ ethtool_link_ksettings_zero_link_mode(cmd, advertising);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, 10baseT_Half);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, TP);
+
+ cmd->base.speed = SPEED_10;
+ cmd->base.duplex = DUPLEX_HALF;
+ cmd->base.port = PORT_TP;
+ cmd->base.phy_address = 0;
+ cmd->base.autoneg = AUTONEG_DISABLE;
+
return 0;
}
static const struct ethtool_ops ops = {
.get_drvinfo = catc_get_drvinfo,
- .get_settings = catc_get_settings,
- .get_link = ethtool_op_get_link
+ .get_link = ethtool_op_get_link,
+ .get_link_ksettings = catc_get_link_ksettings,
};
/*
diff --git a/drivers/net/usb/cdc_mbim.c b/drivers/net/usb/cdc_mbim.c
index 3a98f37..a6b997c 100644
--- a/drivers/net/usb/cdc_mbim.c
+++ b/drivers/net/usb/cdc_mbim.c
@@ -100,6 +100,7 @@
.ndo_stop = usbnet_stop,
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_change_mtu = cdc_ncm_change_mtu,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
diff --git a/drivers/net/usb/cdc_ncm.c b/drivers/net/usb/cdc_ncm.c
index f317984..bb3f71f 100644
--- a/drivers/net/usb/cdc_ncm.c
+++ b/drivers/net/usb/cdc_ncm.c
@@ -131,8 +131,6 @@
static void cdc_ncm_update_rxtx_max(struct usbnet *dev, u32 new_rx, u32 new_tx);
static const struct ethtool_ops cdc_ncm_ethtool_ops = {
- .get_settings = usbnet_get_settings,
- .set_settings = usbnet_set_settings,
.get_link = usbnet_get_link,
.nway_reset = usbnet_nway_reset,
.get_drvinfo = usbnet_get_drvinfo,
@@ -142,6 +140,8 @@
.get_sset_count = cdc_ncm_get_sset_count,
.get_strings = cdc_ncm_get_strings,
.get_ethtool_stats = cdc_ncm_get_ethtool_stats,
+ .get_link_ksettings = usbnet_get_link_ksettings,
+ .set_link_ksettings = usbnet_set_link_ksettings,
};
static u32 cdc_ncm_check_rx_max(struct usbnet *dev, u32 new_rx)
@@ -753,6 +753,7 @@
.ndo_stop = usbnet_stop,
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_change_mtu = cdc_ncm_change_mtu,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
diff --git a/drivers/net/usb/dm9601.c b/drivers/net/usb/dm9601.c
index 0b4bdd3..b91f92e 100644
--- a/drivers/net/usb/dm9601.c
+++ b/drivers/net/usb/dm9601.c
@@ -281,9 +281,9 @@
.set_msglevel = usbnet_set_msglevel,
.get_eeprom_len = dm9601_get_eeprom_len,
.get_eeprom = dm9601_get_eeprom,
- .get_settings = usbnet_get_settings,
- .set_settings = usbnet_set_settings,
.nway_reset = usbnet_nway_reset,
+ .get_link_ksettings = usbnet_get_link_ksettings,
+ .set_link_ksettings = usbnet_set_link_ksettings,
};
static void dm9601_set_multicast(struct net_device *net)
@@ -343,6 +343,7 @@
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
.ndo_change_mtu = usbnet_change_mtu,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_validate_addr = eth_validate_addr,
.ndo_do_ioctl = dm9601_ioctl,
.ndo_set_rx_mode = dm9601_set_multicast,
diff --git a/drivers/net/usb/hso.c b/drivers/net/usb/hso.c
index 4f2e814..00067a0 100644
--- a/drivers/net/usb/hso.c
+++ b/drivers/net/usb/hso.c
@@ -2534,13 +2534,6 @@
SET_NETDEV_DEV(net, &interface->dev);
SET_NETDEV_DEVTYPE(net, &hso_type);
- /* registering our net device */
- result = register_netdev(net);
- if (result) {
- dev_err(&interface->dev, "Failed to register device\n");
- goto exit;
- }
-
/* start allocating */
for (i = 0; i < MUX_BULK_RX_BUF_COUNT; i++) {
hso_net->mux_bulk_rx_urb_pool[i] = usb_alloc_urb(0, GFP_KERNEL);
@@ -2560,6 +2553,13 @@
add_net_device(hso_dev);
+ /* registering our net device */
+ result = register_netdev(net);
+ if (result) {
+ dev_err(&interface->dev, "Failed to register device\n");
+ goto exit;
+ }
+
hso_log_port(hso_dev);
hso_create_rfkill(hso_dev, interface);
@@ -3279,9 +3279,9 @@
pr_info("unloaded\n");
tty_unregister_driver(tty_drv);
- put_tty_driver(tty_drv);
/* deregister the usb driver */
usb_deregister(&hso_driver);
+ put_tty_driver(tty_drv);
}
/* Module definitions */
diff --git a/drivers/net/usb/int51x1.c b/drivers/net/usb/int51x1.c
index 4ff70b2..5a43b77 100644
--- a/drivers/net/usb/int51x1.c
+++ b/drivers/net/usb/int51x1.c
@@ -144,6 +144,7 @@
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
.ndo_change_mtu = usbnet_change_mtu,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
.ndo_set_rx_mode = int51x1_set_multicast,
diff --git a/drivers/net/usb/kaweth.c b/drivers/net/usb/kaweth.c
index 2a2c3ed..37fb621 100644
--- a/drivers/net/usb/kaweth.c
+++ b/drivers/net/usb/kaweth.c
@@ -245,8 +245,6 @@
__u16 packet_filter_bitmap;
struct kaweth_ethernet_configuration configuration;
-
- struct net_device_stats stats;
};
/****************************************************************
@@ -598,7 +596,7 @@
struct sk_buff *skb;
if (unlikely(status == -EPIPE)) {
- kaweth->stats.rx_errors++;
+ net->stats.rx_errors++;
kaweth->end = 1;
wake_up(&kaweth->term_wait);
dev_dbg(dev, "Status was -EPIPE.\n");
@@ -613,12 +611,12 @@
}
if (unlikely(status == -EPROTO || status == -ETIME ||
status == -EILSEQ)) {
- kaweth->stats.rx_errors++;
+ net->stats.rx_errors++;
dev_dbg(dev, "Status was -EPROTO, -ETIME, or -EILSEQ.\n");
return;
}
if (unlikely(status == -EOVERFLOW)) {
- kaweth->stats.rx_errors++;
+ net->stats.rx_errors++;
dev_dbg(dev, "Status was -EOVERFLOW.\n");
}
spin_lock(&kaweth->device_lock);
@@ -663,8 +661,8 @@
netif_rx(skb);
- kaweth->stats.rx_packets++;
- kaweth->stats.rx_bytes += pkt_len;
+ net->stats.rx_packets++;
+ net->stats.rx_bytes += pkt_len;
}
kaweth_resubmit_rx_urb(kaweth, GFP_ATOMIC);
@@ -804,7 +802,7 @@
/* We now decide whether we can put our special header into the sk_buff */
if (skb_cow_head(skb, 2)) {
- kaweth->stats.tx_errors++;
+ net->stats.tx_errors++;
netif_start_queue(net);
spin_unlock_irq(&kaweth->device_lock);
dev_kfree_skb_any(skb);
@@ -828,15 +826,15 @@
{
dev_warn(&net->dev, "kaweth failed tx_urb %d\n", res);
skip:
- kaweth->stats.tx_errors++;
+ net->stats.tx_errors++;
netif_start_queue(net);
dev_kfree_skb_irq(skb);
}
else
{
- kaweth->stats.tx_packets++;
- kaweth->stats.tx_bytes += skb->len;
+ net->stats.tx_packets++;
+ net->stats.tx_bytes += skb->len;
}
spin_unlock_irq(&kaweth->device_lock);
@@ -906,15 +904,6 @@
}
/****************************************************************
- * kaweth_netdev_stats
- ****************************************************************/
-static struct net_device_stats *kaweth_netdev_stats(struct net_device *dev)
-{
- struct kaweth_device *kaweth = netdev_priv(dev);
- return &kaweth->stats;
-}
-
-/****************************************************************
* kaweth_tx_timeout
****************************************************************/
static void kaweth_tx_timeout(struct net_device *net)
@@ -922,7 +911,7 @@
struct kaweth_device *kaweth = netdev_priv(net);
dev_warn(&net->dev, "%s: Tx timed out. Resetting.\n", net->name);
- kaweth->stats.tx_errors++;
+ net->stats.tx_errors++;
netif_trans_update(net);
usb_unlink_urb(kaweth->tx_urb);
@@ -975,7 +964,6 @@
.ndo_start_xmit = kaweth_start_xmit,
.ndo_tx_timeout = kaweth_tx_timeout,
.ndo_set_rx_mode = kaweth_set_rx_mode,
- .ndo_get_stats = kaweth_netdev_stats,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
};
diff --git a/drivers/net/usb/lan78xx.c b/drivers/net/usb/lan78xx.c
index 636f48f..9eff97a 100644
--- a/drivers/net/usb/lan78xx.c
+++ b/drivers/net/usb/lan78xx.c
@@ -29,6 +29,7 @@
#include <linux/ip.h>
#include <linux/ipv6.h>
#include <linux/mdio.h>
+#include <linux/phy.h>
#include <net/ip6_checksum.h>
#include <linux/interrupt.h>
#include <linux/irqdomain.h>
@@ -1952,10 +1953,10 @@
struct lan78xx_net *dev = netdev_priv(phydev->attached_dev);
/* LED2/PME_N/IRQ_N/RGMII_ID pin to IRQ_N mode */
- buf = phy_read_mmd_indirect(phydev, 0x8010, 3);
+ buf = phy_read_mmd(phydev, MDIO_MMD_PCS, 0x8010);
buf &= ~0x1800;
buf |= 0x0800;
- phy_write_mmd_indirect(phydev, 0x8010, 3, buf);
+ phy_write_mmd(phydev, MDIO_MMD_PCS, 0x8010, buf);
/* RGMII MAC TXC Delay Enable */
ret = lan78xx_write_reg(dev, MAC_RGMII_ID,
@@ -1975,11 +1976,11 @@
/* Micrel9301RNX PHY configuration */
/* RGMII Control Signal Pad Skew */
- phy_write_mmd_indirect(phydev, 4, 2, 0x0077);
+ phy_write_mmd(phydev, MDIO_MMD_WIS, 4, 0x0077);
/* RGMII RX Data Pad Skew */
- phy_write_mmd_indirect(phydev, 5, 2, 0x7777);
+ phy_write_mmd(phydev, MDIO_MMD_WIS, 5, 0x7777);
/* RGMII RX Clock Pad Skew */
- phy_write_mmd_indirect(phydev, 8, 2, 0x1FF);
+ phy_write_mmd(phydev, MDIO_MMD_WIS, 8, 0x1FF);
dev->interface = PHY_INTERFACE_MODE_RGMII_RXID;
diff --git a/drivers/net/usb/mcs7830.c b/drivers/net/usb/mcs7830.c
index 4f345bd..5a47e55 100644
--- a/drivers/net/usb/mcs7830.c
+++ b/drivers/net/usb/mcs7830.c
@@ -464,9 +464,9 @@
.get_link = usbnet_get_link,
.get_msglevel = usbnet_get_msglevel,
.set_msglevel = usbnet_set_msglevel,
- .get_settings = usbnet_get_settings,
- .set_settings = usbnet_set_settings,
.nway_reset = usbnet_nway_reset,
+ .get_link_ksettings = usbnet_get_link_ksettings,
+ .set_link_ksettings = usbnet_set_link_ksettings,
};
static const struct net_device_ops mcs7830_netdev_ops = {
@@ -475,6 +475,7 @@
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
.ndo_change_mtu = usbnet_change_mtu,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_validate_addr = eth_validate_addr,
.ndo_do_ioctl = mcs7830_ioctl,
.ndo_set_rx_mode = mcs7830_set_multicast,
diff --git a/drivers/net/usb/pegasus.c b/drivers/net/usb/pegasus.c
index 3667448..6514c86 100644
--- a/drivers/net/usb/pegasus.c
+++ b/drivers/net/usb/pegasus.c
@@ -501,13 +501,13 @@
if (rx_status & 0x1e) {
netif_dbg(pegasus, rx_err, net,
"RX packet error %x\n", rx_status);
- pegasus->stats.rx_errors++;
+ net->stats.rx_errors++;
if (rx_status & 0x06) /* long or runt */
- pegasus->stats.rx_length_errors++;
+ net->stats.rx_length_errors++;
if (rx_status & 0x08)
- pegasus->stats.rx_crc_errors++;
+ net->stats.rx_crc_errors++;
if (rx_status & 0x10) /* extra bits */
- pegasus->stats.rx_frame_errors++;
+ net->stats.rx_frame_errors++;
goto goon;
}
if (pegasus->chip == 0x8513) {
@@ -535,8 +535,8 @@
skb_put(pegasus->rx_skb, pkt_len);
pegasus->rx_skb->protocol = eth_type_trans(pegasus->rx_skb, net);
netif_rx(pegasus->rx_skb);
- pegasus->stats.rx_packets++;
- pegasus->stats.rx_bytes += pkt_len;
+ net->stats.rx_packets++;
+ net->stats.rx_bytes += pkt_len;
if (pegasus->flags & PEGASUS_UNPLUG)
return;
@@ -670,13 +670,13 @@
/* byte 0 == tx_status1, reg 2B */
if (d[0] & (TX_UNDERRUN|EXCESSIVE_COL
|LATE_COL|JABBER_TIMEOUT)) {
- pegasus->stats.tx_errors++;
+ net->stats.tx_errors++;
if (d[0] & TX_UNDERRUN)
- pegasus->stats.tx_fifo_errors++;
+ net->stats.tx_fifo_errors++;
if (d[0] & (EXCESSIVE_COL | JABBER_TIMEOUT))
- pegasus->stats.tx_aborted_errors++;
+ net->stats.tx_aborted_errors++;
if (d[0] & LATE_COL)
- pegasus->stats.tx_window_errors++;
+ net->stats.tx_window_errors++;
}
/* d[5].LINK_STATUS lies on some adapters.
@@ -685,7 +685,7 @@
*/
/* bytes 3-4 == rx_lostpkt, reg 2E/2F */
- pegasus->stats.rx_missed_errors += ((d[3] & 0x7f) << 8) | d[4];
+ net->stats.rx_missed_errors += ((d[3] & 0x7f) << 8) | d[4];
}
res = usb_submit_urb(urb, GFP_ATOMIC);
@@ -701,7 +701,7 @@
pegasus_t *pegasus = netdev_priv(net);
netif_warn(pegasus, timer, net, "tx timeout\n");
usb_unlink_urb(pegasus->tx_urb);
- pegasus->stats.tx_errors++;
+ net->stats.tx_errors++;
}
static netdev_tx_t pegasus_start_xmit(struct sk_buff *skb,
@@ -731,23 +731,18 @@
netif_device_detach(pegasus->net);
break;
default:
- pegasus->stats.tx_errors++;
+ net->stats.tx_errors++;
netif_start_queue(net);
}
} else {
- pegasus->stats.tx_packets++;
- pegasus->stats.tx_bytes += skb->len;
+ net->stats.tx_packets++;
+ net->stats.tx_bytes += skb->len;
}
dev_kfree_skb(skb);
return NETDEV_TX_OK;
}
-static struct net_device_stats *pegasus_netdev_stats(struct net_device *dev)
-{
- return &((pegasus_t *) netdev_priv(dev))->stats;
-}
-
static inline void disable_net_traffic(pegasus_t *pegasus)
{
__le16 tmp = cpu_to_le16(0);
@@ -953,20 +948,22 @@
}
static int
-pegasus_get_settings(struct net_device *dev, struct ethtool_cmd *ecmd)
+pegasus_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *ecmd)
{
pegasus_t *pegasus;
pegasus = netdev_priv(dev);
- mii_ethtool_gset(&pegasus->mii, ecmd);
+ mii_ethtool_get_link_ksettings(&pegasus->mii, ecmd);
return 0;
}
static int
-pegasus_set_settings(struct net_device *dev, struct ethtool_cmd *ecmd)
+pegasus_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *ecmd)
{
pegasus_t *pegasus = netdev_priv(dev);
- return mii_ethtool_sset(&pegasus->mii, ecmd);
+ return mii_ethtool_set_link_ksettings(&pegasus->mii, ecmd);
}
static int pegasus_nway_reset(struct net_device *dev)
@@ -995,14 +992,14 @@
static const struct ethtool_ops ops = {
.get_drvinfo = pegasus_get_drvinfo,
- .get_settings = pegasus_get_settings,
- .set_settings = pegasus_set_settings,
.nway_reset = pegasus_nway_reset,
.get_link = pegasus_get_link,
.get_msglevel = pegasus_get_msglevel,
.set_msglevel = pegasus_set_msglevel,
.get_wol = pegasus_get_wol,
.set_wol = pegasus_set_wol,
+ .get_link_ksettings = pegasus_get_link_ksettings,
+ .set_link_ksettings = pegasus_set_link_ksettings,
};
static int pegasus_ioctl(struct net_device *net, struct ifreq *rq, int cmd)
@@ -1292,7 +1289,6 @@
.ndo_do_ioctl = pegasus_ioctl,
.ndo_start_xmit = pegasus_start_xmit,
.ndo_set_rx_mode = pegasus_set_multicast,
- .ndo_get_stats = pegasus_netdev_stats,
.ndo_tx_timeout = pegasus_tx_timeout,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
diff --git a/drivers/net/usb/pegasus.h b/drivers/net/usb/pegasus.h
index d156462..9b7ea9c 100644
--- a/drivers/net/usb/pegasus.h
+++ b/drivers/net/usb/pegasus.h
@@ -83,7 +83,6 @@
struct usb_device *usb;
struct usb_interface *intf;
struct net_device *net;
- struct net_device_stats stats;
struct mii_if_info mii;
unsigned flags;
unsigned features;
diff --git a/drivers/net/usb/plusb.c b/drivers/net/usb/plusb.c
index 22e1a9a..6fe5937 100644
--- a/drivers/net/usb/plusb.c
+++ b/drivers/net/usb/plusb.c
@@ -102,7 +102,7 @@
}
static const struct driver_info prolific_info = {
- .description = "Prolific PL-2301/PL-2302/PL-25A1",
+ .description = "Prolific PL-2301/PL-2302/PL-25A1/PL-27A1",
.flags = FLAG_POINTTOPOINT | FLAG_NO_SETINT,
/* some PL-2302 versions seem to fail usb_set_interface() */
.reset = pl_reset,
@@ -139,6 +139,17 @@
* Host-to-Host Cable
*/
.driver_info = (unsigned long) &prolific_info,
+
+},
+
+/* super speed cables */
+{
+ USB_DEVICE(0x067b, 0x27a1), /* PL-27A1, no eeprom
+ * also: goobay Active USB 3.0
+ * Data Link,
+ * Unitek Y-3501
+ */
+ .driver_info = (unsigned long) &prolific_info,
},
{ }, // END
@@ -158,5 +169,5 @@
module_usb_driver(plusb_driver);
MODULE_AUTHOR("David Brownell");
-MODULE_DESCRIPTION("Prolific PL-2301/2302/25A1 USB Host to Host Link Driver");
+MODULE_DESCRIPTION("Prolific PL-2301/2302/25A1/27A1 USB Host to Host Link Driver");
MODULE_LICENSE("GPL");
diff --git a/drivers/net/usb/qmi_wwan.c b/drivers/net/usb/qmi_wwan.c
index 2474618..d716576 100644
--- a/drivers/net/usb/qmi_wwan.c
+++ b/drivers/net/usb/qmi_wwan.c
@@ -58,12 +58,198 @@
enum qmi_wwan_flags {
QMI_WWAN_FLAG_RAWIP = 1 << 0,
+ QMI_WWAN_FLAG_MUX = 1 << 1,
};
enum qmi_wwan_quirks {
QMI_WWAN_QUIRK_DTR = 1 << 0, /* needs "set DTR" request */
};
+struct qmimux_hdr {
+ u8 pad;
+ u8 mux_id;
+ __be16 pkt_len;
+};
+
+struct qmimux_priv {
+ struct net_device *real_dev;
+ u8 mux_id;
+};
+
+static int qmimux_open(struct net_device *dev)
+{
+ struct qmimux_priv *priv = netdev_priv(dev);
+ struct net_device *real_dev = priv->real_dev;
+
+ if (!(priv->real_dev->flags & IFF_UP))
+ return -ENETDOWN;
+
+ if (netif_carrier_ok(real_dev))
+ netif_carrier_on(dev);
+ return 0;
+}
+
+static int qmimux_stop(struct net_device *dev)
+{
+ netif_carrier_off(dev);
+ return 0;
+}
+
+static netdev_tx_t qmimux_start_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+ struct qmimux_priv *priv = netdev_priv(dev);
+ unsigned int len = skb->len;
+ struct qmimux_hdr *hdr;
+
+ hdr = (struct qmimux_hdr *)skb_push(skb, sizeof(struct qmimux_hdr));
+ hdr->pad = 0;
+ hdr->mux_id = priv->mux_id;
+ hdr->pkt_len = cpu_to_be16(len);
+ skb->dev = priv->real_dev;
+ return dev_queue_xmit(skb);
+}
+
+static const struct net_device_ops qmimux_netdev_ops = {
+ .ndo_open = qmimux_open,
+ .ndo_stop = qmimux_stop,
+ .ndo_start_xmit = qmimux_start_xmit,
+};
+
+static void qmimux_setup(struct net_device *dev)
+{
+ dev->header_ops = NULL; /* No header */
+ dev->type = ARPHRD_NONE;
+ dev->hard_header_len = 0;
+ dev->addr_len = 0;
+ dev->flags = IFF_POINTOPOINT | IFF_NOARP | IFF_MULTICAST;
+ dev->netdev_ops = &qmimux_netdev_ops;
+ dev->destructor = free_netdev;
+}
+
+static struct net_device *qmimux_find_dev(struct usbnet *dev, u8 mux_id)
+{
+ struct qmimux_priv *priv;
+ struct list_head *iter;
+ struct net_device *ldev;
+
+ rcu_read_lock();
+ netdev_for_each_upper_dev_rcu(dev->net, ldev, iter) {
+ priv = netdev_priv(ldev);
+ if (priv->mux_id == mux_id) {
+ rcu_read_unlock();
+ return ldev;
+ }
+ }
+ rcu_read_unlock();
+ return NULL;
+}
+
+static bool qmimux_has_slaves(struct usbnet *dev)
+{
+ return !list_empty(&dev->net->adj_list.upper);
+}
+
+static int qmimux_rx_fixup(struct usbnet *dev, struct sk_buff *skb)
+{
+ unsigned int len, offset = sizeof(struct qmimux_hdr);
+ struct qmimux_hdr *hdr;
+ struct net_device *net;
+ struct sk_buff *skbn;
+
+ while (offset < skb->len) {
+ hdr = (struct qmimux_hdr *)skb->data;
+ len = be16_to_cpu(hdr->pkt_len);
+
+ /* drop the packet, bogus length */
+ if (offset + len > skb->len)
+ return 0;
+
+ /* control packet, we do not know what to do */
+ if (hdr->pad & 0x80)
+ goto skip;
+
+ net = qmimux_find_dev(dev, hdr->mux_id);
+ if (!net)
+ goto skip;
+ skbn = netdev_alloc_skb(net, len);
+ if (!skbn)
+ return 0;
+ skbn->dev = net;
+
+ switch (skb->data[offset] & 0xf0) {
+ case 0x40:
+ skbn->protocol = htons(ETH_P_IP);
+ break;
+ case 0x60:
+ skbn->protocol = htons(ETH_P_IPV6);
+ break;
+ default:
+ /* not ip - do not know what to do */
+ goto skip;
+ }
+
+ memcpy(skb_put(skbn, len), skb->data + offset, len);
+ if (netif_rx(skbn) != NET_RX_SUCCESS)
+ return 0;
+
+skip:
+ offset += len + sizeof(struct qmimux_hdr);
+ }
+ return 1;
+}
+
+static int qmimux_register_device(struct net_device *real_dev, u8 mux_id)
+{
+ struct net_device *new_dev;
+ struct qmimux_priv *priv;
+ int err;
+
+ new_dev = alloc_netdev(sizeof(struct qmimux_priv),
+ "qmimux%d", NET_NAME_UNKNOWN, qmimux_setup);
+ if (!new_dev)
+ return -ENOBUFS;
+
+ dev_net_set(new_dev, dev_net(real_dev));
+ priv = netdev_priv(new_dev);
+ priv->mux_id = mux_id;
+ priv->real_dev = real_dev;
+
+ err = register_netdevice(new_dev);
+ if (err < 0)
+ goto out_free_newdev;
+
+ /* Account for reference in struct qmimux_priv_priv */
+ dev_hold(real_dev);
+
+ err = netdev_upper_dev_link(real_dev, new_dev);
+ if (err)
+ goto out_unregister_netdev;
+
+ netif_stacked_transfer_operstate(real_dev, new_dev);
+
+ return 0;
+
+out_unregister_netdev:
+ unregister_netdevice(new_dev);
+ dev_put(real_dev);
+
+out_free_newdev:
+ free_netdev(new_dev);
+ return err;
+}
+
+static void qmimux_unregister_device(struct net_device *dev)
+{
+ struct qmimux_priv *priv = netdev_priv(dev);
+ struct net_device *real_dev = priv->real_dev;
+
+ netdev_upper_dev_unlink(real_dev, dev);
+ unregister_netdevice(dev);
+
+ /* Get rid of the reference to real_dev */
+ dev_put(real_dev);
+}
+
static void qmi_wwan_netdev_setup(struct net_device *net)
{
struct usbnet *dev = netdev_priv(net);
@@ -137,10 +323,114 @@
return ret;
}
+static ssize_t add_mux_show(struct device *d, struct device_attribute *attr, char *buf)
+{
+ struct net_device *dev = to_net_dev(d);
+ struct qmimux_priv *priv;
+ struct list_head *iter;
+ struct net_device *ldev;
+ ssize_t count = 0;
+
+ rcu_read_lock();
+ netdev_for_each_upper_dev_rcu(dev, ldev, iter) {
+ priv = netdev_priv(ldev);
+ count += scnprintf(&buf[count], PAGE_SIZE - count,
+ "0x%02x\n", priv->mux_id);
+ }
+ rcu_read_unlock();
+ return count;
+}
+
+static ssize_t add_mux_store(struct device *d, struct device_attribute *attr, const char *buf, size_t len)
+{
+ struct usbnet *dev = netdev_priv(to_net_dev(d));
+ struct qmi_wwan_state *info = (void *)&dev->data;
+ u8 mux_id;
+ int ret;
+
+ if (kstrtou8(buf, 0, &mux_id))
+ return -EINVAL;
+
+ /* mux_id [1 - 0x7f] range empirically found */
+ if (mux_id < 1 || mux_id > 0x7f)
+ return -EINVAL;
+
+ if (!rtnl_trylock())
+ return restart_syscall();
+
+ if (qmimux_find_dev(dev, mux_id)) {
+ netdev_err(dev->net, "mux_id already present\n");
+ ret = -EINVAL;
+ goto err;
+ }
+
+ /* we don't want to modify a running netdev */
+ if (netif_running(dev->net)) {
+ netdev_err(dev->net, "Cannot change a running device\n");
+ ret = -EBUSY;
+ goto err;
+ }
+
+ ret = qmimux_register_device(dev->net, mux_id);
+ if (!ret) {
+ info->flags |= QMI_WWAN_FLAG_MUX;
+ ret = len;
+ }
+err:
+ rtnl_unlock();
+ return ret;
+}
+
+static ssize_t del_mux_show(struct device *d, struct device_attribute *attr, char *buf)
+{
+ return add_mux_show(d, attr, buf);
+}
+
+static ssize_t del_mux_store(struct device *d, struct device_attribute *attr, const char *buf, size_t len)
+{
+ struct usbnet *dev = netdev_priv(to_net_dev(d));
+ struct qmi_wwan_state *info = (void *)&dev->data;
+ struct net_device *del_dev;
+ u8 mux_id;
+ int ret = 0;
+
+ if (kstrtou8(buf, 0, &mux_id))
+ return -EINVAL;
+
+ if (!rtnl_trylock())
+ return restart_syscall();
+
+ /* we don't want to modify a running netdev */
+ if (netif_running(dev->net)) {
+ netdev_err(dev->net, "Cannot change a running device\n");
+ ret = -EBUSY;
+ goto err;
+ }
+
+ del_dev = qmimux_find_dev(dev, mux_id);
+ if (!del_dev) {
+ netdev_err(dev->net, "mux_id not present\n");
+ ret = -EINVAL;
+ goto err;
+ }
+ qmimux_unregister_device(del_dev);
+
+ if (!qmimux_has_slaves(dev))
+ info->flags &= ~QMI_WWAN_FLAG_MUX;
+ ret = len;
+err:
+ rtnl_unlock();
+ return ret;
+}
+
static DEVICE_ATTR_RW(raw_ip);
+static DEVICE_ATTR_RW(add_mux);
+static DEVICE_ATTR_RW(del_mux);
static struct attribute *qmi_wwan_sysfs_attrs[] = {
&dev_attr_raw_ip.attr,
+ &dev_attr_add_mux.attr,
+ &dev_attr_del_mux.attr,
NULL,
};
@@ -184,6 +474,9 @@
if (skb->len < dev->net->hard_header_len)
return 0;
+ if (info->flags & QMI_WWAN_FLAG_MUX)
+ return qmimux_rx_fixup(dev, skb);
+
switch (skb->data[0] & 0xf0) {
case 0x40:
proto = htons(ETH_P_IP);
@@ -249,6 +542,7 @@
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
.ndo_change_mtu = usbnet_change_mtu,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_set_mac_address = qmi_wwan_mac_addr,
.ndo_validate_addr = eth_validate_addr,
};
@@ -907,6 +1201,7 @@
{QMI_FIXED_INTF(0x2357, 0x0201, 4)}, /* TP-LINK HSUPA Modem MA180 */
{QMI_FIXED_INTF(0x2357, 0x9000, 4)}, /* TP-LINK MA260 */
{QMI_QUIRK_SET_DTR(0x1bc7, 0x1040, 2)}, /* Telit LE922A */
+ {QMI_FIXED_INTF(0x1bc7, 0x1100, 3)}, /* Telit ME910 */
{QMI_FIXED_INTF(0x1bc7, 0x1200, 5)}, /* Telit LE920 */
{QMI_QUIRK_SET_DTR(0x1bc7, 0x1201, 2)}, /* Telit LE920, LE920A4 */
{QMI_FIXED_INTF(0x1c9e, 0x9b01, 3)}, /* XS Stick W100-2 from 4G Systems */
@@ -1036,11 +1331,33 @@
return usbnet_probe(intf, id);
}
+static void qmi_wwan_disconnect(struct usb_interface *intf)
+{
+ struct usbnet *dev = usb_get_intfdata(intf);
+ struct qmi_wwan_state *info = (void *)&dev->data;
+ struct list_head *iter;
+ struct net_device *ldev;
+
+ if (info->flags & QMI_WWAN_FLAG_MUX) {
+ if (!rtnl_trylock()) {
+ restart_syscall();
+ return;
+ }
+ rcu_read_lock();
+ netdev_for_each_upper_dev_rcu(dev->net, ldev, iter)
+ qmimux_unregister_device(ldev);
+ rcu_read_unlock();
+ rtnl_unlock();
+ info->flags &= ~QMI_WWAN_FLAG_MUX;
+ }
+ usbnet_disconnect(intf);
+}
+
static struct usb_driver qmi_wwan_driver = {
.name = "qmi_wwan",
.id_table = products,
.probe = qmi_wwan_probe,
- .disconnect = usbnet_disconnect,
+ .disconnect = qmi_wwan_disconnect,
.suspend = qmi_wwan_suspend,
.resume = qmi_wwan_resume,
.reset_resume = qmi_wwan_resume,
diff --git a/drivers/net/usb/r8152.c b/drivers/net/usb/r8152.c
index 07f788c..ddc62cb 100644
--- a/drivers/net/usb/r8152.c
+++ b/drivers/net/usb/r8152.c
@@ -1766,6 +1766,7 @@
unsigned long flags;
struct list_head *cursor, *next, rx_queue;
int ret = 0, work_done = 0;
+ struct napi_struct *napi = &tp->napi;
if (!skb_queue_empty(&tp->rx_queue)) {
while (work_done < budget) {
@@ -1778,7 +1779,7 @@
break;
pkt_len = skb->len;
- napi_gro_receive(&tp->napi, skb);
+ napi_gro_receive(napi, skb);
work_done++;
stats->rx_packets++;
stats->rx_bytes += pkt_len;
@@ -1828,7 +1829,7 @@
pkt_len -= CRC_SIZE;
rx_data += sizeof(struct rx_desc);
- skb = napi_alloc_skb(&tp->napi, pkt_len);
+ skb = napi_alloc_skb(napi, pkt_len);
if (!skb) {
stats->rx_dropped++;
goto find_next_rx;
@@ -1840,7 +1841,7 @@
skb->protocol = eth_type_trans(skb, netdev);
rtl_rx_vlan_tag(rx_desc, skb);
if (work_done < budget) {
- napi_gro_receive(&tp->napi, skb);
+ napi_gro_receive(napi, skb);
work_done++;
stats->rx_packets++;
stats->rx_bytes += pkt_len;
@@ -3156,6 +3157,7 @@
static void set_carrier(struct r8152 *tp)
{
struct net_device *netdev = tp->netdev;
+ struct napi_struct *napi = &tp->napi;
u8 speed;
speed = rtl8152_get_speed(tp);
@@ -3165,7 +3167,7 @@
tp->rtl_ops.enable(tp);
set_bit(RTL8152_SET_RX_MODE, &tp->flags);
netif_stop_queue(netdev);
- napi_disable(&tp->napi);
+ napi_disable(napi);
netif_carrier_on(netdev);
rtl_start_rx(tp);
napi_enable(&tp->napi);
@@ -3178,9 +3180,9 @@
} else {
if (netif_carrier_ok(netdev)) {
netif_carrier_off(netdev);
- napi_disable(&tp->napi);
+ napi_disable(napi);
tp->rtl_ops.disable(tp);
- napi_enable(&tp->napi);
+ napi_enable(napi);
netif_info(tp, link, netdev, "carrier off\n");
}
}
@@ -3642,11 +3644,13 @@
tp->rtl_ops.autosuspend_en(tp, true);
if (netif_carrier_ok(netdev)) {
- napi_disable(&tp->napi);
+ struct napi_struct *napi = &tp->napi;
+
+ napi_disable(napi);
rtl_stop_rx(tp);
rxdy_gated_en(tp, false);
ocp_write_dword(tp, MCU_TYPE_PLA, PLA_RCR, rcr);
- napi_enable(&tp->napi);
+ napi_enable(napi);
}
}
@@ -3662,12 +3666,14 @@
netif_device_detach(netdev);
if (netif_running(netdev) && test_bit(WORK_ENABLE, &tp->flags)) {
+ struct napi_struct *napi = &tp->napi;
+
clear_bit(WORK_ENABLE, &tp->flags);
usb_kill_urb(tp->intr_urb);
- napi_disable(&tp->napi);
+ napi_disable(napi);
cancel_delayed_work_sync(&tp->schedule);
tp->rtl_ops.down(tp);
- napi_enable(&tp->napi);
+ napi_enable(napi);
}
return ret;
@@ -3693,45 +3699,46 @@
static int rtl8152_resume(struct usb_interface *intf)
{
struct r8152 *tp = usb_get_intfdata(intf);
+ struct net_device *netdev = tp->netdev;
mutex_lock(&tp->control);
if (!test_bit(SELECTIVE_SUSPEND, &tp->flags)) {
tp->rtl_ops.init(tp);
queue_delayed_work(system_long_wq, &tp->hw_phy_work, 0);
- netif_device_attach(tp->netdev);
+ netif_device_attach(netdev);
}
- if (netif_running(tp->netdev) && tp->netdev->flags & IFF_UP) {
+ if (netif_running(netdev) && netdev->flags & IFF_UP) {
if (test_bit(SELECTIVE_SUSPEND, &tp->flags)) {
- tp->rtl_ops.autosuspend_en(tp, false);
- napi_disable(&tp->napi);
- set_bit(WORK_ENABLE, &tp->flags);
+ struct napi_struct *napi = &tp->napi;
- if (netif_carrier_ok(tp->netdev)) {
+ tp->rtl_ops.autosuspend_en(tp, false);
+ napi_disable(napi);
+ set_bit(WORK_ENABLE, &tp->flags);
+ if (netif_carrier_ok(netdev)) {
if (rtl8152_get_speed(tp) & LINK_STATUS) {
rtl_start_rx(tp);
} else {
- netif_carrier_off(tp->netdev);
+ netif_carrier_off(netdev);
tp->rtl_ops.disable(tp);
- netif_info(tp, link, tp->netdev,
+ netif_info(tp, link, netdev,
"linking down\n");
}
}
-
- napi_enable(&tp->napi);
+ napi_enable(napi);
clear_bit(SELECTIVE_SUSPEND, &tp->flags);
smp_mb__after_atomic();
if (!list_empty(&tp->rx_done))
napi_schedule(&tp->napi);
} else {
tp->rtl_ops.up(tp);
- netif_carrier_off(tp->netdev);
+ netif_carrier_off(netdev);
set_bit(WORK_ENABLE, &tp->flags);
}
usb_submit_urb(tp->intr_urb, GFP_KERNEL);
} else if (test_bit(SELECTIVE_SUSPEND, &tp->flags)) {
- if (tp->netdev->flags & IFF_UP)
+ if (netdev->flags & IFF_UP)
tp->rtl_ops.autosuspend_en(tp, false);
clear_bit(SELECTIVE_SUSPEND, &tp->flags);
}
@@ -3819,7 +3826,8 @@
}
static
-int rtl8152_get_settings(struct net_device *netdev, struct ethtool_cmd *cmd)
+int rtl8152_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
{
struct r8152 *tp = netdev_priv(netdev);
int ret;
@@ -3833,7 +3841,7 @@
mutex_lock(&tp->control);
- ret = mii_ethtool_gset(&tp->mii, cmd);
+ ret = mii_ethtool_get_link_ksettings(&tp->mii, cmd);
mutex_unlock(&tp->control);
@@ -3843,7 +3851,8 @@
return ret;
}
-static int rtl8152_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int rtl8152_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct r8152 *tp = netdev_priv(dev);
int ret;
@@ -3854,11 +3863,12 @@
mutex_lock(&tp->control);
- ret = rtl8152_set_speed(tp, cmd->autoneg, cmd->speed, cmd->duplex);
+ ret = rtl8152_set_speed(tp, cmd->base.autoneg, cmd->base.speed,
+ cmd->base.duplex);
if (!ret) {
- tp->autoneg = cmd->autoneg;
- tp->speed = cmd->speed;
- tp->duplex = cmd->duplex;
+ tp->autoneg = cmd->base.autoneg;
+ tp->speed = cmd->base.speed;
+ tp->duplex = cmd->base.duplex;
}
mutex_unlock(&tp->control);
@@ -4136,8 +4146,6 @@
static const struct ethtool_ops ops = {
.get_drvinfo = rtl8152_get_drvinfo,
- .get_settings = rtl8152_get_settings,
- .set_settings = rtl8152_set_settings,
.get_link = ethtool_op_get_link,
.nway_reset = rtl8152_nway_reset,
.get_msglevel = rtl8152_get_msglevel,
@@ -4151,6 +4159,8 @@
.set_coalesce = rtl8152_set_coalesce,
.get_eee = rtl_ethtool_get_eee,
.set_eee = rtl_ethtool_set_eee,
+ .get_link_ksettings = rtl8152_get_link_ksettings,
+ .set_link_ksettings = rtl8152_set_link_ksettings,
};
static int rtl8152_ioctl(struct net_device *netdev, struct ifreq *rq, int cmd)
@@ -4249,44 +4259,6 @@
.ndo_features_check = rtl8152_features_check,
};
-static void r8152b_get_version(struct r8152 *tp)
-{
- u32 ocp_data;
- u16 version;
-
- ocp_data = ocp_read_word(tp, MCU_TYPE_PLA, PLA_TCR1);
- version = (u16)(ocp_data & VERSION_MASK);
-
- switch (version) {
- case 0x4c00:
- tp->version = RTL_VER_01;
- break;
- case 0x4c10:
- tp->version = RTL_VER_02;
- break;
- case 0x5c00:
- tp->version = RTL_VER_03;
- tp->mii.supports_gmii = 1;
- break;
- case 0x5c10:
- tp->version = RTL_VER_04;
- tp->mii.supports_gmii = 1;
- break;
- case 0x5c20:
- tp->version = RTL_VER_05;
- tp->mii.supports_gmii = 1;
- break;
- case 0x5c30:
- tp->version = RTL_VER_06;
- tp->mii.supports_gmii = 1;
- break;
- default:
- netif_info(tp, probe, tp->netdev,
- "Unknown version 0x%04x\n", version);
- break;
- }
-}
-
static void rtl8152_unload(struct r8152 *tp)
{
if (test_bit(RTL8152_UNPLUG, &tp->flags))
@@ -4351,14 +4323,66 @@
return ret;
}
+static u8 rtl_get_version(struct usb_interface *intf)
+{
+ struct usb_device *udev = interface_to_usbdev(intf);
+ u32 ocp_data = 0;
+ __le32 *tmp;
+ u8 version;
+ int ret;
+
+ tmp = kmalloc(sizeof(*tmp), GFP_KERNEL);
+ if (!tmp)
+ return 0;
+
+ ret = usb_control_msg(udev, usb_rcvctrlpipe(udev, 0),
+ RTL8152_REQ_GET_REGS, RTL8152_REQT_READ,
+ PLA_TCR0, MCU_TYPE_PLA, tmp, sizeof(*tmp), 500);
+ if (ret > 0)
+ ocp_data = (__le32_to_cpu(*tmp) >> 16) & VERSION_MASK;
+
+ kfree(tmp);
+
+ switch (ocp_data) {
+ case 0x4c00:
+ version = RTL_VER_01;
+ break;
+ case 0x4c10:
+ version = RTL_VER_02;
+ break;
+ case 0x5c00:
+ version = RTL_VER_03;
+ break;
+ case 0x5c10:
+ version = RTL_VER_04;
+ break;
+ case 0x5c20:
+ version = RTL_VER_05;
+ break;
+ case 0x5c30:
+ version = RTL_VER_06;
+ break;
+ default:
+ version = RTL_VER_UNKNOWN;
+ dev_info(&intf->dev, "Unknown version 0x%04x\n", ocp_data);
+ break;
+ }
+
+ return version;
+}
+
static int rtl8152_probe(struct usb_interface *intf,
const struct usb_device_id *id)
{
struct usb_device *udev = interface_to_usbdev(intf);
+ u8 version = rtl_get_version(intf);
struct r8152 *tp;
struct net_device *netdev;
int ret;
+ if (version == RTL_VER_UNKNOWN)
+ return -ENODEV;
+
if (udev->actconfig->desc.bConfigurationValue != 1) {
usb_driver_set_configuration(udev, 1);
return -ENODEV;
@@ -4378,8 +4402,18 @@
tp->udev = udev;
tp->netdev = netdev;
tp->intf = intf;
+ tp->version = version;
- r8152b_get_version(tp);
+ switch (version) {
+ case RTL_VER_01:
+ case RTL_VER_02:
+ tp->mii.supports_gmii = 0;
+ break;
+ default:
+ tp->mii.supports_gmii = 1;
+ break;
+ }
+
ret = rtl_ops_init(tp);
if (ret)
goto out;
diff --git a/drivers/net/usb/rndis_host.c b/drivers/net/usb/rndis_host.c
index c5b2113..e96e2e5 100644
--- a/drivers/net/usb/rndis_host.c
+++ b/drivers/net/usb/rndis_host.c
@@ -291,6 +291,7 @@
.ndo_stop = usbnet_stop,
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
};
diff --git a/drivers/net/usb/rtl8150.c b/drivers/net/usb/rtl8150.c
index c81c791..daaa88a 100644
--- a/drivers/net/usb/rtl8150.c
+++ b/drivers/net/usb/rtl8150.c
@@ -791,47 +791,52 @@
usb_make_path(dev->udev, info->bus_info, sizeof(info->bus_info));
}
-static int rtl8150_get_settings(struct net_device *netdev, struct ethtool_cmd *ecmd)
+static int rtl8150_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *ecmd)
{
rtl8150_t *dev = netdev_priv(netdev);
short lpa, bmcr;
+ u32 supported;
- ecmd->supported = (SUPPORTED_10baseT_Half |
+ supported = (SUPPORTED_10baseT_Half |
SUPPORTED_10baseT_Full |
SUPPORTED_100baseT_Half |
SUPPORTED_100baseT_Full |
SUPPORTED_Autoneg |
SUPPORTED_TP | SUPPORTED_MII);
- ecmd->port = PORT_TP;
- ecmd->transceiver = XCVR_INTERNAL;
- ecmd->phy_address = dev->phy;
+ ecmd->base.port = PORT_TP;
+ ecmd->base.phy_address = dev->phy;
get_registers(dev, BMCR, 2, &bmcr);
get_registers(dev, ANLP, 2, &lpa);
if (bmcr & BMCR_ANENABLE) {
u32 speed = ((lpa & (LPA_100HALF | LPA_100FULL)) ?
SPEED_100 : SPEED_10);
- ethtool_cmd_speed_set(ecmd, speed);
- ecmd->autoneg = AUTONEG_ENABLE;
+ ecmd->base.speed = speed;
+ ecmd->base.autoneg = AUTONEG_ENABLE;
if (speed == SPEED_100)
- ecmd->duplex = (lpa & LPA_100FULL) ?
+ ecmd->base.duplex = (lpa & LPA_100FULL) ?
DUPLEX_FULL : DUPLEX_HALF;
else
- ecmd->duplex = (lpa & LPA_10FULL) ?
+ ecmd->base.duplex = (lpa & LPA_10FULL) ?
DUPLEX_FULL : DUPLEX_HALF;
} else {
- ecmd->autoneg = AUTONEG_DISABLE;
- ethtool_cmd_speed_set(ecmd, ((bmcr & BMCR_SPEED100) ?
- SPEED_100 : SPEED_10));
- ecmd->duplex = (bmcr & BMCR_FULLDPLX) ?
+ ecmd->base.autoneg = AUTONEG_DISABLE;
+ ecmd->base.speed = ((bmcr & BMCR_SPEED100) ?
+ SPEED_100 : SPEED_10);
+ ecmd->base.duplex = (bmcr & BMCR_FULLDPLX) ?
DUPLEX_FULL : DUPLEX_HALF;
}
+
+ ethtool_convert_legacy_u32_to_link_mode(ecmd->link_modes.supported,
+ supported);
+
return 0;
}
static const struct ethtool_ops ops = {
.get_drvinfo = rtl8150_get_drvinfo,
- .get_settings = rtl8150_get_settings,
- .get_link = ethtool_op_get_link
+ .get_link = ethtool_op_get_link,
+ .get_link_ksettings = rtl8150_get_link_ksettings,
};
static int rtl8150_ioctl(struct net_device *netdev, struct ifreq *rq, int cmd)
diff --git a/drivers/net/usb/sierra_net.c b/drivers/net/usb/sierra_net.c
index ac69f28..2110ab3 100644
--- a/drivers/net/usb/sierra_net.c
+++ b/drivers/net/usb/sierra_net.c
@@ -199,6 +199,7 @@
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
.ndo_change_mtu = usbnet_change_mtu,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
};
@@ -648,9 +649,9 @@
.get_link = sierra_net_get_link,
.get_msglevel = usbnet_get_msglevel,
.set_msglevel = usbnet_set_msglevel,
- .get_settings = usbnet_get_settings,
- .set_settings = usbnet_set_settings,
.nway_reset = usbnet_nway_reset,
+ .get_link_ksettings = usbnet_get_link_ksettings,
+ .set_link_ksettings = usbnet_set_link_ksettings,
};
static int sierra_net_get_fw_attr(struct usbnet *dev, u16 *datap)
diff --git a/drivers/net/usb/smsc75xx.c b/drivers/net/usb/smsc75xx.c
index 190de9a..d0a1137 100644
--- a/drivers/net/usb/smsc75xx.c
+++ b/drivers/net/usb/smsc75xx.c
@@ -743,13 +743,13 @@
.get_drvinfo = usbnet_get_drvinfo,
.get_msglevel = usbnet_get_msglevel,
.set_msglevel = usbnet_set_msglevel,
- .get_settings = usbnet_get_settings,
- .set_settings = usbnet_set_settings,
.get_eeprom_len = smsc75xx_ethtool_get_eeprom_len,
.get_eeprom = smsc75xx_ethtool_get_eeprom,
.set_eeprom = smsc75xx_ethtool_set_eeprom,
.get_wol = smsc75xx_ethtool_get_wol,
.set_wol = smsc75xx_ethtool_set_wol,
+ .get_link_ksettings = usbnet_get_link_ksettings,
+ .set_link_ksettings = usbnet_set_link_ksettings,
};
static int smsc75xx_ioctl(struct net_device *netdev, struct ifreq *rq, int cmd)
@@ -1381,6 +1381,7 @@
.ndo_stop = usbnet_stop,
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_change_mtu = smsc75xx_change_mtu,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
diff --git a/drivers/net/usb/smsc95xx.c b/drivers/net/usb/smsc95xx.c
index 5f19fb0..765400b 100644
--- a/drivers/net/usb/smsc95xx.c
+++ b/drivers/net/usb/smsc95xx.c
@@ -33,7 +33,7 @@
#include "smsc95xx.h"
#define SMSC_CHIPNAME "smsc95xx"
-#define SMSC_DRIVER_VERSION "1.0.5"
+#define SMSC_DRIVER_VERSION "1.0.6"
#define HS_USB_PKT_SIZE (512)
#define FS_USB_PKT_SIZE (64)
#define DEFAULT_HS_BURST_CAP_SIZE (16 * 1024 + 5 * HS_USB_PKT_SIZE)
@@ -853,32 +853,32 @@
pdata->mdix_ctrl = mdix_ctrl;
}
-static int smsc95xx_get_settings(struct net_device *net,
- struct ethtool_cmd *cmd)
+static int smsc95xx_get_link_ksettings(struct net_device *net,
+ struct ethtool_link_ksettings *cmd)
{
struct usbnet *dev = netdev_priv(net);
struct smsc95xx_priv *pdata = (struct smsc95xx_priv *)(dev->data[0]);
int retval;
- retval = usbnet_get_settings(net, cmd);
+ retval = usbnet_get_link_ksettings(net, cmd);
- cmd->eth_tp_mdix = pdata->mdix_ctrl;
- cmd->eth_tp_mdix_ctrl = pdata->mdix_ctrl;
+ cmd->base.eth_tp_mdix = pdata->mdix_ctrl;
+ cmd->base.eth_tp_mdix_ctrl = pdata->mdix_ctrl;
return retval;
}
-static int smsc95xx_set_settings(struct net_device *net,
- struct ethtool_cmd *cmd)
+static int smsc95xx_set_link_ksettings(struct net_device *net,
+ const struct ethtool_link_ksettings *cmd)
{
struct usbnet *dev = netdev_priv(net);
struct smsc95xx_priv *pdata = (struct smsc95xx_priv *)(dev->data[0]);
int retval;
- if (pdata->mdix_ctrl != cmd->eth_tp_mdix_ctrl)
- set_mdix_status(net, cmd->eth_tp_mdix_ctrl);
+ if (pdata->mdix_ctrl != cmd->base.eth_tp_mdix_ctrl)
+ set_mdix_status(net, cmd->base.eth_tp_mdix_ctrl);
- retval = usbnet_set_settings(net, cmd);
+ retval = usbnet_set_link_ksettings(net, cmd);
return retval;
}
@@ -889,8 +889,6 @@
.get_drvinfo = usbnet_get_drvinfo,
.get_msglevel = usbnet_get_msglevel,
.set_msglevel = usbnet_set_msglevel,
- .get_settings = smsc95xx_get_settings,
- .set_settings = smsc95xx_set_settings,
.get_eeprom_len = smsc95xx_ethtool_get_eeprom_len,
.get_eeprom = smsc95xx_ethtool_get_eeprom,
.set_eeprom = smsc95xx_ethtool_set_eeprom,
@@ -898,6 +896,8 @@
.get_regs = smsc95xx_ethtool_getregs,
.get_wol = smsc95xx_ethtool_get_wol,
.set_wol = smsc95xx_ethtool_set_wol,
+ .get_link_ksettings = smsc95xx_get_link_ksettings,
+ .set_link_ksettings = smsc95xx_set_link_ksettings,
};
static int smsc95xx_ioctl(struct net_device *netdev, struct ifreq *rq, int cmd)
@@ -1248,6 +1248,7 @@
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
.ndo_change_mtu = usbnet_change_mtu,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
.ndo_do_ioctl = smsc95xx_ioctl,
@@ -1498,7 +1499,7 @@
if (ret < 0)
return ret;
- if (val & 0xFFFF) {
+ if (val & RX_FIFO_INF_USED_) {
netdev_info(dev->net, "rx fifo not empty in autosuspend\n");
return -EBUSY;
}
diff --git a/drivers/net/usb/smsc95xx.h b/drivers/net/usb/smsc95xx.h
index 29a4d9e..cfc704f 100644
--- a/drivers/net/usb/smsc95xx.h
+++ b/drivers/net/usb/smsc95xx.h
@@ -21,218 +21,280 @@
#define _SMSC95XX_H
/* Tx command words */
-#define TX_CMD_A_DATA_OFFSET_ (0x001F0000)
-#define TX_CMD_A_FIRST_SEG_ (0x00002000)
-#define TX_CMD_A_LAST_SEG_ (0x00001000)
-#define TX_CMD_A_BUF_SIZE_ (0x000007FF)
+#define TX_CMD_A_DATA_OFFSET_ (0x001F0000) /* Data Start Offset */
+#define TX_CMD_A_FIRST_SEG_ (0x00002000) /* First Segment */
+#define TX_CMD_A_LAST_SEG_ (0x00001000) /* Last Segment */
+#define TX_CMD_A_BUF_SIZE_ (0x000007FF) /* Buffer Size */
-#define TX_CMD_B_CSUM_ENABLE (0x00004000)
-#define TX_CMD_B_ADD_CRC_DISABLE_ (0x00002000)
-#define TX_CMD_B_DISABLE_PADDING_ (0x00001000)
-#define TX_CMD_B_PKT_BYTE_LENGTH_ (0x000007FF)
+#define TX_CMD_B_CSUM_ENABLE (0x00004000) /* TX Checksum Enable */
+#define TX_CMD_B_ADD_CRC_DIS_ (0x00002000) /* Add CRC Disable */
+#define TX_CMD_B_DIS_PADDING_ (0x00001000) /* Disable Frame Padding */
+#define TX_CMD_B_FRAME_LENGTH_ (0x000007FF) /* Frame Length (bytes) */
/* Rx status word */
-#define RX_STS_FF_ (0x40000000) /* Filter Fail */
-#define RX_STS_FL_ (0x3FFF0000) /* Frame Length */
-#define RX_STS_ES_ (0x00008000) /* Error Summary */
-#define RX_STS_BF_ (0x00002000) /* Broadcast Frame */
-#define RX_STS_LE_ (0x00001000) /* Length Error */
-#define RX_STS_RF_ (0x00000800) /* Runt Frame */
-#define RX_STS_MF_ (0x00000400) /* Multicast Frame */
-#define RX_STS_TL_ (0x00000080) /* Frame too long */
-#define RX_STS_CS_ (0x00000040) /* Collision Seen */
-#define RX_STS_FT_ (0x00000020) /* Frame Type */
-#define RX_STS_RW_ (0x00000010) /* Receive Watchdog */
-#define RX_STS_ME_ (0x00000008) /* Mii Error */
-#define RX_STS_DB_ (0x00000004) /* Dribbling */
-#define RX_STS_CRC_ (0x00000002) /* CRC Error */
+#define RX_STS_FF_ (0x40000000) /* Filter Fail */
+#define RX_STS_FL_ (0x3FFF0000) /* Frame Length */
+#define RX_STS_ES_ (0x00008000) /* Error Summary */
+#define RX_STS_BF_ (0x00002000) /* Broadcast Frame */
+#define RX_STS_LE_ (0x00001000) /* Length Error */
+#define RX_STS_RF_ (0x00000800) /* Runt Frame */
+#define RX_STS_MF_ (0x00000400) /* Multicast Frame */
+#define RX_STS_TL_ (0x00000080) /* Frame too long */
+#define RX_STS_CS_ (0x00000040) /* Collision Seen */
+#define RX_STS_FT_ (0x00000020) /* Frame Type */
+#define RX_STS_RW_ (0x00000010) /* Receive Watchdog */
+#define RX_STS_ME_ (0x00000008) /* MII Error */
+#define RX_STS_DB_ (0x00000004) /* Dribbling */
+#define RX_STS_CRC_ (0x00000002) /* CRC Error */
-/* SCSRs */
-#define ID_REV (0x00)
-#define ID_REV_CHIP_ID_MASK_ (0xFFFF0000)
-#define ID_REV_CHIP_REV_MASK_ (0x0000FFFF)
-#define ID_REV_CHIP_ID_9500_ (0x9500)
-#define ID_REV_CHIP_ID_9500A_ (0x9E00)
-#define ID_REV_CHIP_ID_9512_ (0xEC00)
-#define ID_REV_CHIP_ID_9530_ (0x9530)
-#define ID_REV_CHIP_ID_89530_ (0x9E08)
-#define ID_REV_CHIP_ID_9730_ (0x9730)
+/* SCSRs - System Control and Status Registers */
+/* Device ID and Revision Register */
+#define ID_REV (0x00)
+#define ID_REV_CHIP_ID_MASK_ (0xFFFF0000)
+#define ID_REV_CHIP_REV_MASK_ (0x0000FFFF)
+#define ID_REV_CHIP_ID_9500_ (0x9500)
+#define ID_REV_CHIP_ID_9500A_ (0x9E00)
+#define ID_REV_CHIP_ID_9512_ (0xEC00)
+#define ID_REV_CHIP_ID_9530_ (0x9530)
+#define ID_REV_CHIP_ID_89530_ (0x9E08)
+#define ID_REV_CHIP_ID_9730_ (0x9730)
-#define INT_STS (0x08)
-#define INT_STS_TX_STOP_ (0x00020000)
-#define INT_STS_RX_STOP_ (0x00010000)
-#define INT_STS_PHY_INT_ (0x00008000)
-#define INT_STS_TXE_ (0x00004000)
-#define INT_STS_TDFU_ (0x00002000)
-#define INT_STS_TDFO_ (0x00001000)
-#define INT_STS_RXDF_ (0x00000800)
-#define INT_STS_GPIOS_ (0x000007FF)
-#define INT_STS_CLEAR_ALL_ (0xFFFFFFFF)
+/* Interrupt Status Register */
+#define INT_STS (0x08)
+#define INT_STS_MAC_RTO_ (0x00040000) /* MAC Reset Time Out */
+#define INT_STS_TX_STOP_ (0x00020000) /* TX Stopped */
+#define INT_STS_RX_STOP_ (0x00010000) /* RX Stopped */
+#define INT_STS_PHY_INT_ (0x00008000) /* PHY Interrupt */
+#define INT_STS_TXE_ (0x00004000) /* Transmitter Error */
+#define INT_STS_TDFU_ (0x00002000) /* TX Data FIFO Underrun */
+#define INT_STS_TDFO_ (0x00001000) /* TX Data FIFO Overrun */
+#define INT_STS_RXDF_ (0x00000800) /* RX Dropped Frame */
+#define INT_STS_GPIOS_ (0x000007FF) /* GPIOs Interrupts */
+#define INT_STS_CLEAR_ALL_ (0xFFFFFFFF)
-#define RX_CFG (0x0C)
-#define RX_FIFO_FLUSH_ (0x00000001)
+/* Receive Configuration Register */
+#define RX_CFG (0x0C)
+#define RX_FIFO_FLUSH_ (0x00000001) /* Receive FIFO Flush */
-#define TX_CFG (0x10)
-#define TX_CFG_ON_ (0x00000004)
-#define TX_CFG_STOP_ (0x00000002)
-#define TX_CFG_FIFO_FLUSH_ (0x00000001)
+/* Transmit Configuration Register */
+#define TX_CFG (0x10)
+#define TX_CFG_ON_ (0x00000004) /* Transmitter Enable */
+#define TX_CFG_STOP_ (0x00000002) /* Stop Transmitter */
+#define TX_CFG_FIFO_FLUSH_ (0x00000001) /* Transmit FIFO Flush */
-#define HW_CFG (0x14)
-#define HW_CFG_BIR_ (0x00001000)
-#define HW_CFG_LEDB_ (0x00000800)
-#define HW_CFG_RXDOFF_ (0x00000600)
-#define HW_CFG_DRP_ (0x00000040)
-#define HW_CFG_MEF_ (0x00000020)
-#define HW_CFG_LRST_ (0x00000008)
-#define HW_CFG_PSEL_ (0x00000004)
-#define HW_CFG_BCE_ (0x00000002)
-#define HW_CFG_SRST_ (0x00000001)
+/* Hardware Configuration Register */
+#define HW_CFG (0x14)
+#define HW_CFG_BIR_ (0x00001000) /* Bulk In Empty Response */
+#define HW_CFG_LEDB_ (0x00000800) /* Activity LED 80ms Bypass */
+#define HW_CFG_RXDOFF_ (0x00000600) /* RX Data Offset */
+#define HW_CFG_SBP_ (0x00000100) /* Stall Bulk Out Pipe Dis. */
+#define HW_CFG_IME_ (0x00000080) /* Internal MII Visi. Enable */
+#define HW_CFG_DRP_ (0x00000040) /* Discard Errored RX Frame */
+#define HW_CFG_MEF_ (0x00000020) /* Mult. ETH Frames/USB pkt */
+#define HW_CFG_ETC_ (0x00000010) /* EEPROM Timeout Control */
+#define HW_CFG_LRST_ (0x00000008) /* Soft Lite Reset */
+#define HW_CFG_PSEL_ (0x00000004) /* External PHY Select */
+#define HW_CFG_BCE_ (0x00000002) /* Burst Cap Enable */
+#define HW_CFG_SRST_ (0x00000001) /* Soft Reset */
-#define RX_FIFO_INF (0x18)
+/* Receive FIFO Information Register */
+#define RX_FIFO_INF (0x18)
+#define RX_FIFO_INF_USED_ (0x0000FFFF) /* RX Data FIFO Used Space */
-#define PM_CTRL (0x20)
-#define PM_CTL_RES_CLR_WKP_STS (0x00000200)
-#define PM_CTL_DEV_RDY_ (0x00000080)
-#define PM_CTL_SUS_MODE_ (0x00000060)
-#define PM_CTL_SUS_MODE_0 (0x00000000)
-#define PM_CTL_SUS_MODE_1 (0x00000020)
-#define PM_CTL_SUS_MODE_2 (0x00000040)
-#define PM_CTL_SUS_MODE_3 (0x00000060)
-#define PM_CTL_PHY_RST_ (0x00000010)
-#define PM_CTL_WOL_EN_ (0x00000008)
-#define PM_CTL_ED_EN_ (0x00000004)
-#define PM_CTL_WUPS_ (0x00000003)
-#define PM_CTL_WUPS_NO_ (0x00000000)
-#define PM_CTL_WUPS_ED_ (0x00000001)
-#define PM_CTL_WUPS_WOL_ (0x00000002)
-#define PM_CTL_WUPS_MULTI_ (0x00000003)
+/* Transmit FIFO Information Register */
+#define TX_FIFO_INF (0x1C)
+#define TX_FIFO_INF_FREE_ (0x0000FFFF) /* TX Data FIFO Free Space */
-#define LED_GPIO_CFG (0x24)
-#define LED_GPIO_CFG_SPD_LED (0x01000000)
-#define LED_GPIO_CFG_LNK_LED (0x00100000)
-#define LED_GPIO_CFG_FDX_LED (0x00010000)
+/* Power Management Control Register */
+#define PM_CTRL (0x20)
+#define PM_CTL_RES_CLR_WKP_STS (0x00000200) /* Resume Clears Wakeup STS */
+#define PM_CTL_RES_CLR_WKP_EN (0x00000100) /* Resume Clears Wkp Enables */
+#define PM_CTL_DEV_RDY_ (0x00000080) /* Device Ready */
+#define PM_CTL_SUS_MODE_ (0x00000060) /* Suspend Mode */
+#define PM_CTL_SUS_MODE_0 (0x00000000)
+#define PM_CTL_SUS_MODE_1 (0x00000020)
+#define PM_CTL_SUS_MODE_2 (0x00000040)
+#define PM_CTL_SUS_MODE_3 (0x00000060)
+#define PM_CTL_PHY_RST_ (0x00000010) /* PHY Reset */
+#define PM_CTL_WOL_EN_ (0x00000008) /* Wake On Lan Enable */
+#define PM_CTL_ED_EN_ (0x00000004) /* Energy Detect Enable */
+#define PM_CTL_WUPS_ (0x00000003) /* Wake Up Status */
+#define PM_CTL_WUPS_NO_ (0x00000000) /* No Wake Up Event Detected */
+#define PM_CTL_WUPS_ED_ (0x00000001) /* Energy Detect */
+#define PM_CTL_WUPS_WOL_ (0x00000002) /* Wake On Lan */
+#define PM_CTL_WUPS_MULTI_ (0x00000003) /* Multiple Events Occurred */
-#define GPIO_CFG (0x28)
+/* LED General Purpose IO Configuration Register */
+#define LED_GPIO_CFG (0x24)
+#define LED_GPIO_CFG_SPD_LED (0x01000000) /* GPIOz as Speed LED */
+#define LED_GPIO_CFG_LNK_LED (0x00100000) /* GPIOy as Link LED */
+#define LED_GPIO_CFG_FDX_LED (0x00010000) /* GPIOx as Full Duplex LED */
-#define AFC_CFG (0x2C)
+/* General Purpose IO Configuration Register */
+#define GPIO_CFG (0x28)
+/* Automatic Flow Control Configuration Register */
+#define AFC_CFG (0x2C)
+#define AFC_CFG_HI_ (0x00FF0000) /* Auto Flow Ctrl High Level */
+#define AFC_CFG_LO_ (0x0000FF00) /* Auto Flow Ctrl Low Level */
+#define AFC_CFG_BACK_DUR_ (0x000000F0) /* Back Pressure Duration */
+#define AFC_CFG_FC_MULT_ (0x00000008) /* Flow Ctrl on Mcast Frame */
+#define AFC_CFG_FC_BRD_ (0x00000004) /* Flow Ctrl on Bcast Frame */
+#define AFC_CFG_FC_ADD_ (0x00000002) /* Flow Ctrl on Addr. Decode */
+#define AFC_CFG_FC_ANY_ (0x00000001) /* Flow Ctrl on Any Frame */
/* Hi watermark = 15.5Kb (~10 mtu pkts) */
/* low watermark = 3k (~2 mtu pkts) */
/* backpressure duration = ~ 350us */
/* Apply FC on any frame. */
-#define AFC_CFG_DEFAULT (0x00F830A1)
+#define AFC_CFG_DEFAULT (0x00F830A1)
-#define E2P_CMD (0x30)
-#define E2P_CMD_BUSY_ (0x80000000)
-#define E2P_CMD_MASK_ (0x70000000)
-#define E2P_CMD_READ_ (0x00000000)
-#define E2P_CMD_EWDS_ (0x10000000)
-#define E2P_CMD_EWEN_ (0x20000000)
-#define E2P_CMD_WRITE_ (0x30000000)
-#define E2P_CMD_WRAL_ (0x40000000)
-#define E2P_CMD_ERASE_ (0x50000000)
-#define E2P_CMD_ERAL_ (0x60000000)
-#define E2P_CMD_RELOAD_ (0x70000000)
-#define E2P_CMD_TIMEOUT_ (0x00000400)
-#define E2P_CMD_LOADED_ (0x00000200)
-#define E2P_CMD_ADDR_ (0x000001FF)
+/* EEPROM Command Register */
+#define E2P_CMD (0x30)
+#define E2P_CMD_BUSY_ (0x80000000) /* E2P Controller Busy */
+#define E2P_CMD_MASK_ (0x70000000) /* Command Mask (see below) */
+#define E2P_CMD_READ_ (0x00000000) /* Read Location */
+#define E2P_CMD_EWDS_ (0x10000000) /* Erase/Write Disable */
+#define E2P_CMD_EWEN_ (0x20000000) /* Erase/Write Enable */
+#define E2P_CMD_WRITE_ (0x30000000) /* Write Location */
+#define E2P_CMD_WRAL_ (0x40000000) /* Write All */
+#define E2P_CMD_ERASE_ (0x50000000) /* Erase Location */
+#define E2P_CMD_ERAL_ (0x60000000) /* Erase All */
+#define E2P_CMD_RELOAD_ (0x70000000) /* Data Reload */
+#define E2P_CMD_TIMEOUT_ (0x00000400) /* Set if no resp within 30ms */
+#define E2P_CMD_LOADED_ (0x00000200) /* Valid EEPROM found */
+#define E2P_CMD_ADDR_ (0x000001FF) /* Byte aligned address */
-#define MAX_EEPROM_SIZE (512)
+#define MAX_EEPROM_SIZE (512)
-#define E2P_DATA (0x34)
-#define E2P_DATA_MASK_ (0x000000FF)
+/* EEPROM Data Register */
+#define E2P_DATA (0x34)
+#define E2P_DATA_MASK_ (0x000000FF) /* EEPROM Data Mask */
-#define BURST_CAP (0x38)
+/* Burst Cap Register */
+#define BURST_CAP (0x38)
+#define BURST_CAP_MASK_ (0x000000FF) /* Max burst sent by the UTX */
+/* Configuration Straps Status Register */
#define STRAP_STATUS (0x3C)
-#define STRAP_STATUS_PWR_SEL_ (0x00000020)
-#define STRAP_STATUS_AMDIX_EN_ (0x00000010)
-#define STRAP_STATUS_PORT_SWAP_ (0x00000008)
-#define STRAP_STATUS_EEP_SIZE_ (0x00000004)
-#define STRAP_STATUS_RMT_WKP_ (0x00000002)
-#define STRAP_STATUS_EEP_DISABLE_ (0x00000001)
+#define STRAP_STATUS_PWR_SEL_ (0x00000020) /* Device self-powered */
+#define STRAP_STATUS_AMDIX_EN_ (0x00000010) /* Auto-MDIX Enabled */
+#define STRAP_STATUS_PORT_SWAP_ (0x00000008) /* USBD+/USBD- Swapped */
+#define STRAP_STATUS_EEP_SIZE_ (0x00000004) /* EEPROM Size */
+#define STRAP_STATUS_RMT_WKP_ (0x00000002) /* Remote Wkp supported */
+#define STRAP_STATUS_EEP_DISABLE_ (0x00000001) /* EEPROM Disabled */
-#define GPIO_WAKE (0x64)
+/* Data Port Select Register */
+#define DP_SEL (0x40)
-#define INT_EP_CTL (0x68)
-#define INT_EP_CTL_INTEP_ (0x80000000)
-#define INT_EP_CTL_MACRTO_ (0x00080000)
-#define INT_EP_CTL_TX_STOP_ (0x00020000)
-#define INT_EP_CTL_RX_STOP_ (0x00010000)
-#define INT_EP_CTL_PHY_INT_ (0x00008000)
-#define INT_EP_CTL_TXE_ (0x00004000)
-#define INT_EP_CTL_TDFU_ (0x00002000)
-#define INT_EP_CTL_TDFO_ (0x00001000)
-#define INT_EP_CTL_RXDF_ (0x00000800)
-#define INT_EP_CTL_GPIOS_ (0x000007FF)
+/* Data Port Command Register */
+#define DP_CMD (0x44)
-#define BULK_IN_DLY (0x6C)
+/* Data Port Address Register */
+#define DP_ADDR (0x48)
-/* MAC CSRs */
-#define MAC_CR (0x100)
-#define MAC_CR_RXALL_ (0x80000000)
-#define MAC_CR_RCVOWN_ (0x00800000)
-#define MAC_CR_LOOPBK_ (0x00200000)
-#define MAC_CR_FDPX_ (0x00100000)
-#define MAC_CR_MCPAS_ (0x00080000)
-#define MAC_CR_PRMS_ (0x00040000)
-#define MAC_CR_INVFILT_ (0x00020000)
-#define MAC_CR_PASSBAD_ (0x00010000)
-#define MAC_CR_HFILT_ (0x00008000)
-#define MAC_CR_HPFILT_ (0x00002000)
-#define MAC_CR_LCOLL_ (0x00001000)
-#define MAC_CR_BCAST_ (0x00000800)
-#define MAC_CR_DISRTY_ (0x00000400)
-#define MAC_CR_PADSTR_ (0x00000100)
-#define MAC_CR_BOLMT_MASK (0x000000C0)
-#define MAC_CR_DFCHK_ (0x00000020)
-#define MAC_CR_TXEN_ (0x00000008)
-#define MAC_CR_RXEN_ (0x00000004)
+/* Data Port Data 0 Register */
+#define DP_DATA0 (0x4C)
-#define ADDRH (0x104)
+/* Data Port Data 1 Register */
+#define DP_DATA1 (0x50)
-#define ADDRL (0x108)
+/* General Purpose IO Wake Enable and Polarity Register */
+#define GPIO_WAKE (0x64)
-#define HASHH (0x10C)
+/* Interrupt Endpoint Control Register */
+#define INT_EP_CTL (0x68)
+#define INT_EP_CTL_INTEP_ (0x80000000) /* Always TX Interrupt PKT */
+#define INT_EP_CTL_MAC_RTO_ (0x00080000) /* MAC Reset Time Out */
+#define INT_EP_CTL_RX_FIFO_ (0x00040000) /* RX FIFO Has Frame */
+#define INT_EP_CTL_TX_STOP_ (0x00020000) /* TX Stopped */
+#define INT_EP_CTL_RX_STOP_ (0x00010000) /* RX Stopped */
+#define INT_EP_CTL_PHY_INT_ (0x00008000) /* PHY Interrupt */
+#define INT_EP_CTL_TXE_ (0x00004000) /* TX Error */
+#define INT_EP_CTL_TDFU_ (0x00002000) /* TX Data FIFO Underrun */
+#define INT_EP_CTL_TDFO_ (0x00001000) /* TX Data FIFO Overrun */
+#define INT_EP_CTL_RXDF_ (0x00000800) /* RX Dropped Frame */
+#define INT_EP_CTL_GPIOS_ (0x000007FF) /* GPIOs Interrupt Enable */
-#define HASHL (0x110)
+/* Bulk In Delay Register (units of 16.667ns, until ~1092µs) */
+#define BULK_IN_DLY (0x6C)
-#define MII_ADDR (0x114)
-#define MII_WRITE_ (0x02)
-#define MII_BUSY_ (0x01)
-#define MII_READ_ (0x00) /* ~of MII Write bit */
+/* MAC CSRs - MAC Control and Status Registers */
+/* MAC Control Register */
+#define MAC_CR (0x100)
+#define MAC_CR_RXALL_ (0x80000000) /* Receive All Mode */
+#define MAC_CR_RCVOWN_ (0x00800000) /* Disable Receive Own */
+#define MAC_CR_LOOPBK_ (0x00200000) /* Loopback Operation Mode */
+#define MAC_CR_FDPX_ (0x00100000) /* Full Duplex Mode */
+#define MAC_CR_MCPAS_ (0x00080000) /* Pass All Multicast */
+#define MAC_CR_PRMS_ (0x00040000) /* Promiscuous Mode */
+#define MAC_CR_INVFILT_ (0x00020000) /* Inverse Filtering */
+#define MAC_CR_PASSBAD_ (0x00010000) /* Pass Bad Frames */
+#define MAC_CR_HFILT_ (0x00008000) /* Hash Only Filtering Mode */
+#define MAC_CR_HPFILT_ (0x00002000) /* Hash/Perfect Filt. Mode */
+#define MAC_CR_LCOLL_ (0x00001000) /* Late Collision Control */
+#define MAC_CR_BCAST_ (0x00000800) /* Disable Broadcast Frames */
+#define MAC_CR_DISRTY_ (0x00000400) /* Disable Retry */
+#define MAC_CR_PADSTR_ (0x00000100) /* Automatic Pad Stripping */
+#define MAC_CR_BOLMT_MASK (0x000000C0) /* BackOff Limit */
+#define MAC_CR_DFCHK_ (0x00000020) /* Deferral Check */
+#define MAC_CR_TXEN_ (0x00000008) /* Transmitter Enable */
+#define MAC_CR_RXEN_ (0x00000004) /* Receiver Enable */
-#define MII_DATA (0x118)
+/* MAC Address High Register */
+#define ADDRH (0x104)
-#define FLOW (0x11C)
-#define FLOW_FCPT_ (0xFFFF0000)
-#define FLOW_FCPASS_ (0x00000004)
-#define FLOW_FCEN_ (0x00000002)
-#define FLOW_FCBSY_ (0x00000001)
+/* MAC Address Low Register */
+#define ADDRL (0x108)
-#define VLAN1 (0x120)
+/* Multicast Hash Table High Register */
+#define HASHH (0x10C)
-#define VLAN2 (0x124)
+/* Multicast Hash Table Low Register */
+#define HASHL (0x110)
-#define WUFF (0x128)
-#define LAN9500_WUFF_NUM (4)
-#define LAN9500A_WUFF_NUM (8)
+/* MII Access Register */
+#define MII_ADDR (0x114)
+#define MII_WRITE_ (0x02)
+#define MII_BUSY_ (0x01)
+#define MII_READ_ (0x00) /* ~of MII Write bit */
-#define WUCSR (0x12C)
-#define WUCSR_WFF_PTR_RST_ (0x80000000)
-#define WUCSR_GUE_ (0x00000200)
-#define WUCSR_WUFR_ (0x00000040)
-#define WUCSR_MPR_ (0x00000020)
-#define WUCSR_WAKE_EN_ (0x00000004)
-#define WUCSR_MPEN_ (0x00000002)
+/* MII Data Register */
+#define MII_DATA (0x118)
-#define COE_CR (0x130)
-#define Tx_COE_EN_ (0x00010000)
-#define Rx_COE_MODE_ (0x00000002)
-#define Rx_COE_EN_ (0x00000001)
+/* Flow Control Register */
+#define FLOW (0x11C)
+#define FLOW_FCPT_ (0xFFFF0000) /* Pause Time */
+#define FLOW_FCPASS_ (0x00000004) /* Pass Control Frames */
+#define FLOW_FCEN_ (0x00000002) /* Flow Control Enable */
+#define FLOW_FCBSY_ (0x00000001) /* Flow Control Busy */
-/* Vendor-specific PHY Definitions */
+/* VLAN1 Tag Register */
+#define VLAN1 (0x120)
+/* VLAN2 Tag Register */
+#define VLAN2 (0x124)
+
+/* Wake Up Frame Filter Register */
+#define WUFF (0x128)
+#define LAN9500_WUFF_NUM (4)
+#define LAN9500A_WUFF_NUM (8)
+
+/* Wake Up Control and Status Register */
+#define WUCSR (0x12C)
+#define WUCSR_WFF_PTR_RST_ (0x80000000) /* WFrame Filter Pointer Rst */
+#define WUCSR_GUE_ (0x00000200) /* Global Unicast Enable */
+#define WUCSR_WUFR_ (0x00000040) /* Wakeup Frame Received */
+#define WUCSR_MPR_ (0x00000020) /* Magic Packet Received */
+#define WUCSR_WAKE_EN_ (0x00000004) /* Wakeup Frame Enable */
+#define WUCSR_MPEN_ (0x00000002) /* Magic Packet Enable */
+
+/* Checksum Offload Engine Control Register */
+#define COE_CR (0x130)
+#define Tx_COE_EN_ (0x00010000) /* TX Csum Offload Enable */
+#define Rx_COE_MODE_ (0x00000002) /* RX Csum Offload Mode */
+#define Rx_COE_EN_ (0x00000001) /* RX Csum Offload Enable */
+
+/* Vendor-specific PHY Definitions (via MII access) */
/* EDPD NLP / crossover time configuration (LAN9500A only) */
#define PHY_EDPD_CONFIG (16)
#define PHY_EDPD_CONFIG_TX_NLP_EN_ ((u16)0x8000)
@@ -255,17 +317,20 @@
#define MODE_CTRL_STS_EDPWRDOWN_ ((u16)0x2000)
#define MODE_CTRL_STS_ENERGYON_ ((u16)0x0002)
+/* Control/Status Indication Register */
#define SPECIAL_CTRL_STS (27)
#define SPECIAL_CTRL_STS_OVRRD_AMDIX_ ((u16)0x8000)
#define SPECIAL_CTRL_STS_AMDIX_ENABLE_ ((u16)0x4000)
#define SPECIAL_CTRL_STS_AMDIX_STATE_ ((u16)0x2000)
+/* Interrupt Source Register */
#define PHY_INT_SRC (29)
#define PHY_INT_SRC_ENERGY_ON_ ((u16)0x0080)
#define PHY_INT_SRC_ANEG_COMP_ ((u16)0x0040)
#define PHY_INT_SRC_REMOTE_FAULT_ ((u16)0x0020)
#define PHY_INT_SRC_LINK_DOWN_ ((u16)0x0010)
+/* Interrupt Mask Register */
#define PHY_INT_MASK (30)
#define PHY_INT_MASK_ENERGY_ON_ ((u16)0x0080)
#define PHY_INT_MASK_ANEG_COMP_ ((u16)0x0040)
@@ -273,7 +338,7 @@
#define PHY_INT_MASK_LINK_DOWN_ ((u16)0x0010)
#define PHY_INT_MASK_DEFAULT_ (PHY_INT_MASK_ANEG_COMP_ | \
PHY_INT_MASK_LINK_DOWN_)
-
+/* PHY Special Control/Status Register */
#define PHY_SPECIAL (31)
#define PHY_SPECIAL_SPD_ ((u16)0x001C)
#define PHY_SPECIAL_SPD_10HALF_ ((u16)0x0004)
@@ -287,12 +352,13 @@
#define USB_VENDOR_REQUEST_GET_STATS 0xA2
/* Interrupt Endpoint status word bitfields */
-#define INT_ENP_TX_STOP_ ((u32)BIT(17))
-#define INT_ENP_RX_STOP_ ((u32)BIT(16))
-#define INT_ENP_PHY_INT_ ((u32)BIT(15))
-#define INT_ENP_TXE_ ((u32)BIT(14))
-#define INT_ENP_TDFU_ ((u32)BIT(13))
-#define INT_ENP_TDFO_ ((u32)BIT(12))
-#define INT_ENP_RXDF_ ((u32)BIT(11))
+#define INT_ENP_MAC_RTO_ ((u32)BIT(18)) /* MAC Reset Time Out */
+#define INT_ENP_TX_STOP_ ((u32)BIT(17)) /* TX Stopped */
+#define INT_ENP_RX_STOP_ ((u32)BIT(16)) /* RX Stopped */
+#define INT_ENP_PHY_INT_ ((u32)BIT(15)) /* PHY Interrupt */
+#define INT_ENP_TXE_ ((u32)BIT(14)) /* TX Error */
+#define INT_ENP_TDFU_ ((u32)BIT(13)) /* TX FIFO Underrun */
+#define INT_ENP_TDFO_ ((u32)BIT(12)) /* TX FIFO Overrun */
+#define INT_ENP_RXDF_ ((u32)BIT(11)) /* RX Dropped Frame */
#endif /* _SMSC95XX_H */
diff --git a/drivers/net/usb/sr9700.c b/drivers/net/usb/sr9700.c
index aadfe1d..2d316c1 100644
--- a/drivers/net/usb/sr9700.c
+++ b/drivers/net/usb/sr9700.c
@@ -249,9 +249,9 @@
.set_msglevel = usbnet_set_msglevel,
.get_eeprom_len = sr9700_get_eeprom_len,
.get_eeprom = sr9700_get_eeprom,
- .get_settings = usbnet_get_settings,
- .set_settings = usbnet_set_settings,
.nway_reset = usbnet_nway_reset,
+ .get_link_ksettings = usbnet_get_link_ksettings,
+ .set_link_ksettings = usbnet_set_link_ksettings,
};
static void sr9700_set_multicast(struct net_device *netdev)
@@ -308,6 +308,7 @@
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
.ndo_change_mtu = usbnet_change_mtu,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_validate_addr = eth_validate_addr,
.ndo_do_ioctl = sr9700_ioctl,
.ndo_set_rx_mode = sr9700_set_multicast,
diff --git a/drivers/net/usb/sr9800.c b/drivers/net/usb/sr9800.c
index a50df0d..9277a0f 100644
--- a/drivers/net/usb/sr9800.c
+++ b/drivers/net/usb/sr9800.c
@@ -524,9 +524,9 @@
.set_wol = sr_set_wol,
.get_eeprom_len = sr_get_eeprom_len,
.get_eeprom = sr_get_eeprom,
- .get_settings = usbnet_get_settings,
- .set_settings = usbnet_set_settings,
.nway_reset = usbnet_nway_reset,
+ .get_link_ksettings = usbnet_get_link_ksettings,
+ .set_link_ksettings = usbnet_set_link_ksettings,
};
static int sr9800_link_reset(struct usbnet *dev)
@@ -679,6 +679,7 @@
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
.ndo_change_mtu = usbnet_change_mtu,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_set_mac_address = sr_set_mac_address,
.ndo_validate_addr = eth_validate_addr,
.ndo_do_ioctl = sr_ioctl,
diff --git a/drivers/net/usb/usbnet.c b/drivers/net/usb/usbnet.c
index 4532448..79048e7 100644
--- a/drivers/net/usb/usbnet.c
+++ b/drivers/net/usb/usbnet.c
@@ -82,8 +82,6 @@
// randomly generated ethernet address
static u8 node_id [ETH_ALEN];
-static const char driver_name [] = "usbnet";
-
/* use ethtool to change the level for any given device */
static int msg_level = -1;
module_param (msg_level, int, 0);
@@ -316,6 +314,7 @@
*/
void usbnet_skb_return (struct usbnet *dev, struct sk_buff *skb)
{
+ struct pcpu_sw_netstats *stats64 = this_cpu_ptr(dev->stats64);
int status;
if (test_bit(EVENT_RX_PAUSED, &dev->flags)) {
@@ -327,8 +326,10 @@
if (skb->protocol == 0)
skb->protocol = eth_type_trans (skb, dev->net);
- dev->net->stats.rx_packets++;
- dev->net->stats.rx_bytes += skb->len;
+ u64_stats_update_begin(&stats64->syncp);
+ stats64->rx_packets++;
+ stats64->rx_bytes += skb->len;
+ u64_stats_update_end(&stats64->syncp);
netif_dbg(dev, rx_status, dev->net, "< rx, len %zu, type 0x%x\n",
skb->len + sizeof (struct ethhdr), skb->protocol);
@@ -947,18 +948,20 @@
* they'll probably want to use this base set.
*/
-int usbnet_get_settings (struct net_device *net, struct ethtool_cmd *cmd)
+int usbnet_get_link_ksettings(struct net_device *net,
+ struct ethtool_link_ksettings *cmd)
{
struct usbnet *dev = netdev_priv(net);
if (!dev->mii.mdio_read)
return -EOPNOTSUPP;
- return mii_ethtool_gset(&dev->mii, cmd);
+ return mii_ethtool_get_link_ksettings(&dev->mii, cmd);
}
-EXPORT_SYMBOL_GPL(usbnet_get_settings);
+EXPORT_SYMBOL_GPL(usbnet_get_link_ksettings);
-int usbnet_set_settings (struct net_device *net, struct ethtool_cmd *cmd)
+int usbnet_set_link_ksettings(struct net_device *net,
+ const struct ethtool_link_ksettings *cmd)
{
struct usbnet *dev = netdev_priv(net);
int retval;
@@ -966,7 +969,7 @@
if (!dev->mii.mdio_write)
return -EOPNOTSUPP;
- retval = mii_ethtool_sset(&dev->mii, cmd);
+ retval = mii_ethtool_set_link_ksettings(&dev->mii, cmd);
/* link speed/duplex might have changed */
if (dev->driver_info->link_reset)
@@ -976,9 +979,39 @@
usbnet_update_max_qlen(dev);
return retval;
-
}
-EXPORT_SYMBOL_GPL(usbnet_set_settings);
+EXPORT_SYMBOL_GPL(usbnet_set_link_ksettings);
+
+void usbnet_get_stats64(struct net_device *net, struct rtnl_link_stats64 *stats)
+{
+ struct usbnet *dev = netdev_priv(net);
+ unsigned int start;
+ int cpu;
+
+ netdev_stats_to_stats64(stats, &net->stats);
+
+ for_each_possible_cpu(cpu) {
+ struct pcpu_sw_netstats *stats64;
+ u64 rx_packets, rx_bytes;
+ u64 tx_packets, tx_bytes;
+
+ stats64 = per_cpu_ptr(dev->stats64, cpu);
+
+ do {
+ start = u64_stats_fetch_begin_irq(&stats64->syncp);
+ rx_packets = stats64->rx_packets;
+ rx_bytes = stats64->rx_bytes;
+ tx_packets = stats64->tx_packets;
+ tx_bytes = stats64->tx_bytes;
+ } while (u64_stats_fetch_retry_irq(&stats64->syncp, start));
+
+ stats->rx_packets += rx_packets;
+ stats->rx_bytes += rx_bytes;
+ stats->tx_packets += tx_packets;
+ stats->tx_bytes += tx_bytes;
+ }
+}
+EXPORT_SYMBOL_GPL(usbnet_get_stats64);
u32 usbnet_get_link (struct net_device *net)
{
@@ -1038,14 +1071,14 @@
/* drivers may override default ethtool_ops in their bind() routine */
static const struct ethtool_ops usbnet_ethtool_ops = {
- .get_settings = usbnet_get_settings,
- .set_settings = usbnet_set_settings,
.get_link = usbnet_get_link,
.nway_reset = usbnet_nway_reset,
.get_drvinfo = usbnet_get_drvinfo,
.get_msglevel = usbnet_get_msglevel,
.set_msglevel = usbnet_set_msglevel,
.get_ts_info = ethtool_op_get_ts_info,
+ .get_link_ksettings = usbnet_get_link_ksettings,
+ .set_link_ksettings = usbnet_set_link_ksettings,
};
/*-------------------------------------------------------------------------*/
@@ -1211,8 +1244,12 @@
struct usbnet *dev = entry->dev;
if (urb->status == 0) {
- dev->net->stats.tx_packets += entry->packets;
- dev->net->stats.tx_bytes += entry->length;
+ struct pcpu_sw_netstats *stats64 = this_cpu_ptr(dev->stats64);
+
+ u64_stats_update_begin(&stats64->syncp);
+ stats64->tx_packets += entry->packets;
+ stats64->tx_bytes += entry->length;
+ u64_stats_update_end(&stats64->syncp);
} else {
dev->net->stats.tx_errors++;
@@ -1569,6 +1606,7 @@
usb_free_urb(dev->interrupt);
kfree(dev->padding_pkt);
+ free_percpu(dev->stats64);
free_netdev(net);
}
EXPORT_SYMBOL_GPL(usbnet_disconnect);
@@ -1580,6 +1618,7 @@
.ndo_tx_timeout = usbnet_tx_timeout,
.ndo_set_rx_mode = usbnet_set_rx_mode,
.ndo_change_mtu = usbnet_change_mtu,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
};
@@ -1641,6 +1680,11 @@
dev->intf = udev;
dev->driver_info = info;
dev->driver_name = name;
+
+ dev->stats64 = netdev_alloc_pcpu_stats(struct pcpu_sw_netstats);
+ if (!dev->stats64)
+ goto out0;
+
dev->msg_enable = netif_msg_init (msg_level, NETIF_MSG_DRV
| NETIF_MSG_PROBE | NETIF_MSG_LINK);
init_waitqueue_head(&dev->wait);
@@ -1780,6 +1824,8 @@
*/
cancel_work_sync(&dev->kevent);
del_timer_sync(&dev->delay);
+ free_percpu(dev->stats64);
+out0:
free_netdev(net);
out:
return status;
diff --git a/drivers/net/veth.c b/drivers/net/veth.c
index 8c39d6d..38f0f03 100644
--- a/drivers/net/veth.c
+++ b/drivers/net/veth.c
@@ -45,18 +45,13 @@
{ "peer_ifindex" },
};
-static int veth_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int veth_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
- cmd->supported = 0;
- cmd->advertising = 0;
- ethtool_cmd_speed_set(cmd, SPEED_10000);
- cmd->duplex = DUPLEX_FULL;
- cmd->port = PORT_TP;
- cmd->phy_address = 0;
- cmd->transceiver = XCVR_INTERNAL;
- cmd->autoneg = AUTONEG_DISABLE;
- cmd->maxtxpkt = 0;
- cmd->maxrxpkt = 0;
+ cmd->base.speed = SPEED_10000;
+ cmd->base.duplex = DUPLEX_FULL;
+ cmd->base.port = PORT_TP;
+ cmd->base.autoneg = AUTONEG_DISABLE;
return 0;
}
@@ -95,12 +90,12 @@
}
static const struct ethtool_ops veth_ethtool_ops = {
- .get_settings = veth_get_settings,
.get_drvinfo = veth_get_drvinfo,
.get_link = ethtool_op_get_link,
.get_strings = veth_get_strings,
.get_sset_count = veth_get_sset_count,
.get_ethtool_stats = veth_get_ethtool_stats,
+ .get_link_ksettings = veth_get_link_ksettings,
};
static netdev_tx_t veth_xmit(struct sk_buff *skb, struct net_device *dev)
@@ -373,7 +368,8 @@
ifmp = nla_data(nla_peer);
err = rtnl_nla_parse_ifla(peer_tb,
nla_data(nla_peer) + sizeof(struct ifinfomsg),
- nla_len(nla_peer) - sizeof(struct ifinfomsg));
+ nla_len(nla_peer) - sizeof(struct ifinfomsg),
+ NULL);
if (err < 0)
return err;
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index f365846..1c6d392 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -33,9 +33,10 @@
static int napi_weight = NAPI_POLL_WEIGHT;
module_param(napi_weight, int, 0444);
-static bool csum = true, gso = true;
+static bool csum = true, gso = true, napi_tx;
module_param(csum, bool, 0444);
module_param(gso, bool, 0444);
+module_param(napi_tx, bool, 0644);
/* FIXME: MTU in config. */
#define GOOD_PACKET_LEN (ETH_HLEN + VLAN_HLEN + ETH_DATA_LEN)
@@ -86,6 +87,8 @@
/* Name of the send queue: output.$index */
char name[40];
+
+ struct napi_struct napi;
};
/* Internal representation of a receive virtqueue */
@@ -239,15 +242,39 @@
return p;
}
+static void virtqueue_napi_schedule(struct napi_struct *napi,
+ struct virtqueue *vq)
+{
+ if (napi_schedule_prep(napi)) {
+ virtqueue_disable_cb(vq);
+ __napi_schedule(napi);
+ }
+}
+
+static void virtqueue_napi_complete(struct napi_struct *napi,
+ struct virtqueue *vq, int processed)
+{
+ int opaque;
+
+ opaque = virtqueue_enable_cb_prepare(vq);
+ if (napi_complete_done(napi, processed) &&
+ unlikely(virtqueue_poll(vq, opaque)))
+ virtqueue_napi_schedule(napi, vq);
+}
+
static void skb_xmit_done(struct virtqueue *vq)
{
struct virtnet_info *vi = vq->vdev->priv;
+ struct napi_struct *napi = &vi->sq[vq2txq(vq)].napi;
/* Suppress further interrupts. */
virtqueue_disable_cb(vq);
- /* We were probably waiting for more output buffers. */
- netif_wake_subqueue(vi->dev, vq2txq(vq));
+ if (napi->weight)
+ virtqueue_napi_schedule(napi, vq);
+ else
+ /* We were probably waiting for more output buffers. */
+ netif_wake_subqueue(vi->dev, vq2txq(vq));
}
static unsigned int mergeable_ctx_to_buf_truesize(unsigned long mrg_ctx)
@@ -936,27 +963,44 @@
struct virtnet_info *vi = rvq->vdev->priv;
struct receive_queue *rq = &vi->rq[vq2rxq(rvq)];
- /* Schedule NAPI, Suppress further interrupts if successful. */
- if (napi_schedule_prep(&rq->napi)) {
- virtqueue_disable_cb(rvq);
- __napi_schedule(&rq->napi);
- }
+ virtqueue_napi_schedule(&rq->napi, rvq);
}
-static void virtnet_napi_enable(struct receive_queue *rq)
+static void virtnet_napi_enable(struct virtqueue *vq, struct napi_struct *napi)
{
- napi_enable(&rq->napi);
+ napi_enable(napi);
/* If all buffers were filled by other side before we napi_enabled, we
- * won't get another interrupt, so process any outstanding packets
- * now. virtnet_poll wants re-enable the queue, so we disable here.
- * We synchronize against interrupts via NAPI_STATE_SCHED */
- if (napi_schedule_prep(&rq->napi)) {
- virtqueue_disable_cb(rq->vq);
- local_bh_disable();
- __napi_schedule(&rq->napi);
- local_bh_enable();
+ * won't get another interrupt, so process any outstanding packets now.
+ * Call local_bh_enable after to trigger softIRQ processing.
+ */
+ local_bh_disable();
+ virtqueue_napi_schedule(napi, vq);
+ local_bh_enable();
+}
+
+static void virtnet_napi_tx_enable(struct virtnet_info *vi,
+ struct virtqueue *vq,
+ struct napi_struct *napi)
+{
+ if (!napi->weight)
+ return;
+
+ /* Tx napi touches cachelines on the cpu handling tx interrupts. Only
+ * enable the feature if this is likely affine with the transmit path.
+ */
+ if (!vi->affinity_hint_set) {
+ napi->weight = 0;
+ return;
}
+
+ return virtnet_napi_enable(vq, napi);
+}
+
+static void virtnet_napi_tx_disable(struct napi_struct *napi)
+{
+ if (napi->weight)
+ napi_disable(napi);
}
static void refill_work(struct work_struct *work)
@@ -971,7 +1015,7 @@
napi_disable(&rq->napi);
still_empty = !try_fill_recv(vi, rq, GFP_KERNEL);
- virtnet_napi_enable(rq);
+ virtnet_napi_enable(rq->vq, &rq->napi);
/* In theory, this can happen: if we don't get any buffers in
* we will *never* try to fill again.
@@ -1007,45 +1051,6 @@
return received;
}
-static int virtnet_poll(struct napi_struct *napi, int budget)
-{
- struct receive_queue *rq =
- container_of(napi, struct receive_queue, napi);
- unsigned int r, received;
-
- received = virtnet_receive(rq, budget);
-
- /* Out of packets? */
- if (received < budget) {
- r = virtqueue_enable_cb_prepare(rq->vq);
- if (napi_complete_done(napi, received)) {
- if (unlikely(virtqueue_poll(rq->vq, r)) &&
- napi_schedule_prep(napi)) {
- virtqueue_disable_cb(rq->vq);
- __napi_schedule(napi);
- }
- }
- }
-
- return received;
-}
-
-static int virtnet_open(struct net_device *dev)
-{
- struct virtnet_info *vi = netdev_priv(dev);
- int i;
-
- for (i = 0; i < vi->max_queue_pairs; i++) {
- if (i < vi->curr_queue_pairs)
- /* Make sure we have some buffers: if oom use wq. */
- if (!try_fill_recv(vi, &vi->rq[i], GFP_KERNEL))
- schedule_delayed_work(&vi->refill, 0);
- virtnet_napi_enable(&vi->rq[i]);
- }
-
- return 0;
-}
-
static void free_old_xmit_skbs(struct send_queue *sq)
{
struct sk_buff *skb;
@@ -1076,6 +1081,77 @@
u64_stats_update_end(&stats->tx_syncp);
}
+static void virtnet_poll_cleantx(struct receive_queue *rq)
+{
+ struct virtnet_info *vi = rq->vq->vdev->priv;
+ unsigned int index = vq2rxq(rq->vq);
+ struct send_queue *sq = &vi->sq[index];
+ struct netdev_queue *txq = netdev_get_tx_queue(vi->dev, index);
+
+ if (!sq->napi.weight)
+ return;
+
+ if (__netif_tx_trylock(txq)) {
+ free_old_xmit_skbs(sq);
+ __netif_tx_unlock(txq);
+ }
+
+ if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS)
+ netif_tx_wake_queue(txq);
+}
+
+static int virtnet_poll(struct napi_struct *napi, int budget)
+{
+ struct receive_queue *rq =
+ container_of(napi, struct receive_queue, napi);
+ unsigned int received;
+
+ virtnet_poll_cleantx(rq);
+
+ received = virtnet_receive(rq, budget);
+
+ /* Out of packets? */
+ if (received < budget)
+ virtqueue_napi_complete(napi, rq->vq, received);
+
+ return received;
+}
+
+static int virtnet_open(struct net_device *dev)
+{
+ struct virtnet_info *vi = netdev_priv(dev);
+ int i;
+
+ for (i = 0; i < vi->max_queue_pairs; i++) {
+ if (i < vi->curr_queue_pairs)
+ /* Make sure we have some buffers: if oom use wq. */
+ if (!try_fill_recv(vi, &vi->rq[i], GFP_KERNEL))
+ schedule_delayed_work(&vi->refill, 0);
+ virtnet_napi_enable(vi->rq[i].vq, &vi->rq[i].napi);
+ virtnet_napi_tx_enable(vi, vi->sq[i].vq, &vi->sq[i].napi);
+ }
+
+ return 0;
+}
+
+static int virtnet_poll_tx(struct napi_struct *napi, int budget)
+{
+ struct send_queue *sq = container_of(napi, struct send_queue, napi);
+ struct virtnet_info *vi = sq->vq->vdev->priv;
+ struct netdev_queue *txq = netdev_get_tx_queue(vi->dev, vq2txq(sq->vq));
+
+ __netif_tx_lock(txq, raw_smp_processor_id());
+ free_old_xmit_skbs(sq);
+ __netif_tx_unlock(txq);
+
+ virtqueue_napi_complete(napi, sq->vq, 0);
+
+ if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS)
+ netif_tx_wake_queue(txq);
+
+ return 0;
+}
+
static int xmit_skb(struct send_queue *sq, struct sk_buff *skb)
{
struct virtio_net_hdr_mrg_rxbuf *hdr;
@@ -1125,10 +1201,14 @@
int err;
struct netdev_queue *txq = netdev_get_tx_queue(dev, qnum);
bool kick = !skb->xmit_more;
+ bool use_napi = sq->napi.weight;
/* Free up any pending old buffers before queueing new ones. */
free_old_xmit_skbs(sq);
+ if (use_napi && kick)
+ virtqueue_enable_cb_delayed(sq->vq);
+
/* timestamp packet in software */
skb_tx_timestamp(skb);
@@ -1147,8 +1227,10 @@
}
/* Don't wait up for transmitted skbs to be freed. */
- skb_orphan(skb);
- nf_reset(skb);
+ if (!use_napi) {
+ skb_orphan(skb);
+ nf_reset(skb);
+ }
/* If running out of space, stop queue to avoid getting packets that we
* are then unable to transmit.
@@ -1162,7 +1244,8 @@
*/
if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
netif_stop_subqueue(dev, qnum);
- if (unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
+ if (!use_napi &&
+ unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
/* More just got used, free them then recheck. */
free_old_xmit_skbs(sq);
if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) {
@@ -1366,8 +1449,10 @@
/* Make sure refill_work doesn't re-enable napi! */
cancel_delayed_work_sync(&vi->refill);
- for (i = 0; i < vi->max_queue_pairs; i++)
+ for (i = 0; i < vi->max_queue_pairs; i++) {
napi_disable(&vi->rq[i].napi);
+ virtnet_napi_tx_disable(&vi->sq[i].napi);
+ }
return 0;
}
@@ -1636,47 +1721,57 @@
}
/* Check if the user is trying to change anything besides speed/duplex */
-static bool virtnet_validate_ethtool_cmd(const struct ethtool_cmd *cmd)
+static bool
+virtnet_validate_ethtool_cmd(const struct ethtool_link_ksettings *cmd)
{
- struct ethtool_cmd diff1 = *cmd;
- struct ethtool_cmd diff2 = {};
+ struct ethtool_link_ksettings diff1 = *cmd;
+ struct ethtool_link_ksettings diff2 = {};
/* cmd is always set so we need to clear it, validate the port type
* and also without autonegotiation we can ignore advertising
*/
- ethtool_cmd_speed_set(&diff1, 0);
- diff2.port = PORT_OTHER;
- diff1.advertising = 0;
- diff1.duplex = 0;
- diff1.cmd = 0;
+ diff1.base.speed = 0;
+ diff2.base.port = PORT_OTHER;
+ ethtool_link_ksettings_zero_link_mode(&diff1, advertising);
+ diff1.base.duplex = 0;
+ diff1.base.cmd = 0;
+ diff1.base.link_mode_masks_nwords = 0;
- return !memcmp(&diff1, &diff2, sizeof(diff1));
+ return !memcmp(&diff1.base, &diff2.base, sizeof(diff1.base)) &&
+ bitmap_empty(diff1.link_modes.supported,
+ __ETHTOOL_LINK_MODE_MASK_NBITS) &&
+ bitmap_empty(diff1.link_modes.advertising,
+ __ETHTOOL_LINK_MODE_MASK_NBITS) &&
+ bitmap_empty(diff1.link_modes.lp_advertising,
+ __ETHTOOL_LINK_MODE_MASK_NBITS);
}
-static int virtnet_set_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int virtnet_set_link_ksettings(struct net_device *dev,
+ const struct ethtool_link_ksettings *cmd)
{
struct virtnet_info *vi = netdev_priv(dev);
u32 speed;
- speed = ethtool_cmd_speed(cmd);
+ speed = cmd->base.speed;
/* don't allow custom speed and duplex */
if (!ethtool_validate_speed(speed) ||
- !ethtool_validate_duplex(cmd->duplex) ||
+ !ethtool_validate_duplex(cmd->base.duplex) ||
!virtnet_validate_ethtool_cmd(cmd))
return -EINVAL;
vi->speed = speed;
- vi->duplex = cmd->duplex;
+ vi->duplex = cmd->base.duplex;
return 0;
}
-static int virtnet_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
+static int virtnet_get_link_ksettings(struct net_device *dev,
+ struct ethtool_link_ksettings *cmd)
{
struct virtnet_info *vi = netdev_priv(dev);
- ethtool_cmd_speed_set(cmd, vi->speed);
- cmd->duplex = vi->duplex;
- cmd->port = PORT_OTHER;
+ cmd->base.speed = vi->speed;
+ cmd->base.duplex = vi->duplex;
+ cmd->base.port = PORT_OTHER;
return 0;
}
@@ -1696,8 +1791,8 @@
.set_channels = virtnet_set_channels,
.get_channels = virtnet_get_channels,
.get_ts_info = ethtool_op_get_ts_info,
- .get_settings = virtnet_get_settings,
- .set_settings = virtnet_set_settings,
+ .get_link_ksettings = virtnet_get_link_ksettings,
+ .set_link_ksettings = virtnet_set_link_ksettings,
};
static void virtnet_freeze_down(struct virtio_device *vdev)
@@ -1712,8 +1807,10 @@
cancel_delayed_work_sync(&vi->refill);
if (netif_running(vi->dev)) {
- for (i = 0; i < vi->max_queue_pairs; i++)
+ for (i = 0; i < vi->max_queue_pairs; i++) {
napi_disable(&vi->rq[i].napi);
+ virtnet_napi_tx_disable(&vi->sq[i].napi);
+ }
}
}
@@ -1736,8 +1833,11 @@
if (!try_fill_recv(vi, &vi->rq[i], GFP_KERNEL))
schedule_delayed_work(&vi->refill, 0);
- for (i = 0; i < vi->max_queue_pairs; i++)
- virtnet_napi_enable(&vi->rq[i]);
+ for (i = 0; i < vi->max_queue_pairs; i++) {
+ virtnet_napi_enable(vi->rq[i].vq, &vi->rq[i].napi);
+ virtnet_napi_tx_enable(vi, vi->sq[i].vq,
+ &vi->sq[i].napi);
+ }
}
netif_device_attach(vi->dev);
@@ -1778,7 +1878,8 @@
return ret;
}
-static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog)
+static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog,
+ struct netlink_ext_ack *extack)
{
unsigned long int max_sz = PAGE_SIZE - sizeof(struct padded_vnet_hdr);
struct virtnet_info *vi = netdev_priv(dev);
@@ -1790,16 +1891,17 @@
virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_TSO6) ||
virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_ECN) ||
virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_UFO)) {
- netdev_warn(dev, "can't set XDP while host is implementing LRO, disable LRO first\n");
+ NL_SET_ERR_MSG_MOD(extack, "Can't set XDP while host is implementing LRO, disable LRO first");
return -EOPNOTSUPP;
}
if (vi->mergeable_rx_bufs && !vi->any_header_sg) {
- netdev_warn(dev, "XDP expects header/data in single page, any_header_sg required\n");
+ NL_SET_ERR_MSG_MOD(extack, "XDP expects header/data in single page, any_header_sg required");
return -EINVAL;
}
if (dev->mtu > max_sz) {
+ NL_SET_ERR_MSG_MOD(extack, "MTU too large to enable XDP");
netdev_warn(dev, "XDP requires MTU less than %lu\n", max_sz);
return -EINVAL;
}
@@ -1810,6 +1912,7 @@
/* XDP requires extra queues for XDP_TX */
if (curr_qp + xdp_qp > vi->max_queue_pairs) {
+ NL_SET_ERR_MSG_MOD(extack, "Too few free TX rings available");
netdev_warn(dev, "request %i queues but max is %i\n",
curr_qp + xdp_qp, vi->max_queue_pairs);
return -ENOMEM;
@@ -1871,7 +1974,7 @@
{
switch (xdp->command) {
case XDP_SETUP_PROG:
- return virtnet_xdp_set(dev, xdp->prog);
+ return virtnet_xdp_set(dev, xdp->prog, xdp->extack);
case XDP_QUERY_PROG:
xdp->prog_attached = virtnet_xdp_query(dev);
return 0;
@@ -1942,6 +2045,7 @@
for (i = 0; i < vi->max_queue_pairs; i++) {
napi_hash_del(&vi->rq[i].napi);
netif_napi_del(&vi->rq[i].napi);
+ netif_napi_del(&vi->sq[i].napi);
}
/* We called napi_hash_del() before netif_napi_del(),
@@ -2127,6 +2231,8 @@
vi->rq[i].pages = NULL;
netif_napi_add(vi->dev, &vi->rq[i].napi, virtnet_poll,
napi_weight);
+ netif_tx_napi_add(vi->dev, &vi->sq[i].napi, virtnet_poll_tx,
+ napi_tx ? napi_weight : 0);
sg_init_table(vi->rq[i].sg, ARRAY_SIZE(vi->rq[i].sg));
ewma_pkt_len_init(&vi->rq[i].mrg_avg_pkt_len);
diff --git a/drivers/net/vmxnet3/vmxnet3_ethtool.c b/drivers/net/vmxnet3/vmxnet3_ethtool.c
index f88ffaf..2ff2731 100644
--- a/drivers/net/vmxnet3/vmxnet3_ethtool.c
+++ b/drivers/net/vmxnet3/vmxnet3_ethtool.c
@@ -471,22 +471,25 @@
static int
-vmxnet3_get_settings(struct net_device *netdev, struct ethtool_cmd *ecmd)
+vmxnet3_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *ecmd)
{
struct vmxnet3_adapter *adapter = netdev_priv(netdev);
- ecmd->supported = SUPPORTED_10000baseT_Full | SUPPORTED_1000baseT_Full |
- SUPPORTED_TP;
- ecmd->advertising = ADVERTISED_TP;
- ecmd->port = PORT_TP;
- ecmd->transceiver = XCVR_INTERNAL;
+ ethtool_link_ksettings_zero_link_mode(ecmd, supported);
+ ethtool_link_ksettings_add_link_mode(ecmd, supported, 10000baseT_Full);
+ ethtool_link_ksettings_add_link_mode(ecmd, supported, 1000baseT_Full);
+ ethtool_link_ksettings_add_link_mode(ecmd, supported, TP);
+ ethtool_link_ksettings_zero_link_mode(ecmd, advertising);
+ ethtool_link_ksettings_add_link_mode(ecmd, advertising, TP);
+ ecmd->base.port = PORT_TP;
if (adapter->link_speed) {
- ethtool_cmd_speed_set(ecmd, adapter->link_speed);
- ecmd->duplex = DUPLEX_FULL;
+ ecmd->base.speed = adapter->link_speed;
+ ecmd->base.duplex = DUPLEX_FULL;
} else {
- ethtool_cmd_speed_set(ecmd, SPEED_UNKNOWN);
- ecmd->duplex = DUPLEX_UNKNOWN;
+ ecmd->base.speed = SPEED_UNKNOWN;
+ ecmd->base.duplex = DUPLEX_UNKNOWN;
}
return 0;
}
@@ -880,7 +883,6 @@
}
static const struct ethtool_ops vmxnet3_ethtool_ops = {
- .get_settings = vmxnet3_get_settings,
.get_drvinfo = vmxnet3_get_drvinfo,
.get_regs_len = vmxnet3_get_regs_len,
.get_regs = vmxnet3_get_regs,
@@ -900,6 +902,7 @@
.get_rxfh = vmxnet3_get_rss,
.set_rxfh = vmxnet3_set_rss,
#endif
+ .get_link_ksettings = vmxnet3_get_link_ksettings,
};
void vmxnet3_set_ethtool_ops(struct net_device *netdev)
diff --git a/drivers/net/vrf.c b/drivers/net/vrf.c
index 7d909c8..ceda586 100644
--- a/drivers/net/vrf.c
+++ b/drivers/net/vrf.c
@@ -104,6 +104,23 @@
}
}
+/* by default VRF devices do not have a qdisc and are expected
+ * to be created with only a single queue.
+ */
+static bool qdisc_tx_is_default(const struct net_device *dev)
+{
+ struct netdev_queue *txq;
+ struct Qdisc *qdisc;
+
+ if (dev->num_tx_queues > 1)
+ return false;
+
+ txq = netdev_get_tx_queue(dev, 0);
+ qdisc = rcu_access_pointer(txq->qdisc);
+
+ return !qdisc->enqueue;
+}
+
/* Local traffic destined to local address. Reinsert the packet to rx
* path, similar to loopback handling.
*/
@@ -357,6 +374,29 @@
return ret;
}
+static int vrf_finish_direct(struct net *net, struct sock *sk,
+ struct sk_buff *skb)
+{
+ struct net_device *vrf_dev = skb->dev;
+
+ if (!list_empty(&vrf_dev->ptype_all) &&
+ likely(skb_headroom(skb) >= ETH_HLEN)) {
+ struct ethhdr *eth = (struct ethhdr *)skb_push(skb, ETH_HLEN);
+
+ ether_addr_copy(eth->h_source, vrf_dev->dev_addr);
+ eth_zero_addr(eth->h_dest);
+ eth->h_proto = skb->protocol;
+
+ rcu_read_lock_bh();
+ dev_queue_xmit_nit(skb, vrf_dev);
+ rcu_read_unlock_bh();
+
+ skb_pull(skb, ETH_HLEN);
+ }
+
+ return 1;
+}
+
#if IS_ENABLED(CONFIG_IPV6)
/* modelled after ip6_finish_output2 */
static int vrf_finish_output6(struct net *net, struct sock *sk,
@@ -405,18 +445,13 @@
* packet to go through device based features such as qdisc, netfilter
* hooks and packet sockets with skb->dev set to vrf device.
*/
-static struct sk_buff *vrf_ip6_out(struct net_device *vrf_dev,
- struct sock *sk,
- struct sk_buff *skb)
+static struct sk_buff *vrf_ip6_out_redirect(struct net_device *vrf_dev,
+ struct sk_buff *skb)
{
struct net_vrf *vrf = netdev_priv(vrf_dev);
struct dst_entry *dst = NULL;
struct rt6_info *rt6;
- /* don't divert link scope packets */
- if (rt6_need_strict(&ipv6_hdr(skb)->daddr))
- return skb;
-
rcu_read_lock();
rt6 = rcu_dereference(vrf->rt6);
@@ -438,6 +473,55 @@
return skb;
}
+static int vrf_output6_direct(struct net *net, struct sock *sk,
+ struct sk_buff *skb)
+{
+ skb->protocol = htons(ETH_P_IPV6);
+
+ return NF_HOOK_COND(NFPROTO_IPV6, NF_INET_POST_ROUTING,
+ net, sk, skb, NULL, skb->dev,
+ vrf_finish_direct,
+ !(IPCB(skb)->flags & IPSKB_REROUTED));
+}
+
+static struct sk_buff *vrf_ip6_out_direct(struct net_device *vrf_dev,
+ struct sock *sk,
+ struct sk_buff *skb)
+{
+ struct net *net = dev_net(vrf_dev);
+ int err;
+
+ skb->dev = vrf_dev;
+
+ err = nf_hook(NFPROTO_IPV6, NF_INET_LOCAL_OUT, net, sk,
+ skb, NULL, vrf_dev, vrf_output6_direct);
+
+ if (likely(err == 1))
+ err = vrf_output6_direct(net, sk, skb);
+
+ /* reset skb device */
+ if (likely(err == 1))
+ nf_reset(skb);
+ else
+ skb = NULL;
+
+ return skb;
+}
+
+static struct sk_buff *vrf_ip6_out(struct net_device *vrf_dev,
+ struct sock *sk,
+ struct sk_buff *skb)
+{
+ /* don't divert link scope packets */
+ if (rt6_need_strict(&ipv6_hdr(skb)->daddr))
+ return skb;
+
+ if (qdisc_tx_is_default(vrf_dev))
+ return vrf_ip6_out_direct(vrf_dev, sk, skb);
+
+ return vrf_ip6_out_redirect(vrf_dev, skb);
+}
+
/* holding rtnl */
static void vrf_rt6_release(struct net_device *dev, struct net_vrf *vrf)
{
@@ -609,18 +693,13 @@
* packet to go through device based features such as qdisc, netfilter
* hooks and packet sockets with skb->dev set to vrf device.
*/
-static struct sk_buff *vrf_ip_out(struct net_device *vrf_dev,
- struct sock *sk,
- struct sk_buff *skb)
+static struct sk_buff *vrf_ip_out_redirect(struct net_device *vrf_dev,
+ struct sk_buff *skb)
{
struct net_vrf *vrf = netdev_priv(vrf_dev);
struct dst_entry *dst = NULL;
struct rtable *rth;
- /* don't divert multicast */
- if (ipv4_is_multicast(ip_hdr(skb)->daddr))
- return skb;
-
rcu_read_lock();
rth = rcu_dereference(vrf->rth);
@@ -642,6 +721,55 @@
return skb;
}
+static int vrf_output_direct(struct net *net, struct sock *sk,
+ struct sk_buff *skb)
+{
+ skb->protocol = htons(ETH_P_IP);
+
+ return NF_HOOK_COND(NFPROTO_IPV4, NF_INET_POST_ROUTING,
+ net, sk, skb, NULL, skb->dev,
+ vrf_finish_direct,
+ !(IPCB(skb)->flags & IPSKB_REROUTED));
+}
+
+static struct sk_buff *vrf_ip_out_direct(struct net_device *vrf_dev,
+ struct sock *sk,
+ struct sk_buff *skb)
+{
+ struct net *net = dev_net(vrf_dev);
+ int err;
+
+ skb->dev = vrf_dev;
+
+ err = nf_hook(NFPROTO_IPV4, NF_INET_LOCAL_OUT, net, sk,
+ skb, NULL, vrf_dev, vrf_output_direct);
+
+ if (likely(err == 1))
+ err = vrf_output_direct(net, sk, skb);
+
+ /* reset skb device */
+ if (likely(err == 1))
+ nf_reset(skb);
+ else
+ skb = NULL;
+
+ return skb;
+}
+
+static struct sk_buff *vrf_ip_out(struct net_device *vrf_dev,
+ struct sock *sk,
+ struct sk_buff *skb)
+{
+ /* don't divert multicast */
+ if (ipv4_is_multicast(ip_hdr(skb)->daddr))
+ return skb;
+
+ if (qdisc_tx_is_default(vrf_dev))
+ return vrf_ip_out_direct(vrf_dev, sk, skb);
+
+ return vrf_ip_out_redirect(vrf_dev, skb);
+}
+
/* called with rcu lock held */
static struct sk_buff *vrf_l3_out(struct net_device *vrf_dev,
struct sock *sk,
@@ -749,14 +877,24 @@
{
int ret;
- ret = netdev_master_upper_dev_link(port_dev, dev, NULL, NULL);
- if (ret < 0)
- return ret;
+ /* do not allow loopback device to be enslaved to a VRF.
+ * The vrf device acts as the loopback for the vrf.
+ */
+ if (port_dev == dev_net(dev)->loopback_dev)
+ return -EOPNOTSUPP;
port_dev->priv_flags |= IFF_L3MDEV_SLAVE;
+ ret = netdev_master_upper_dev_link(port_dev, dev, NULL, NULL);
+ if (ret < 0)
+ goto err;
+
cycle_netdev(port_dev);
return 0;
+
+err:
+ port_dev->priv_flags &= ~IFF_L3MDEV_SLAVE;
+ return ret;
}
static int vrf_add_slave(struct net_device *dev, struct net_device *port_dev)
@@ -978,9 +1116,11 @@
skb->dev = vrf_dev;
skb->skb_iif = vrf_dev->ifindex;
- skb_push(skb, skb->mac_len);
- dev_queue_xmit_nit(skb, vrf_dev);
- skb_pull(skb, skb->mac_len);
+ if (!list_empty(&vrf_dev->ptype_all)) {
+ skb_push(skb, skb->mac_len);
+ dev_queue_xmit_nit(skb, vrf_dev);
+ skb_pull(skb, skb->mac_len);
+ }
IP6CB(skb)->flags |= IP6SKB_L3SLAVE;
}
@@ -1021,9 +1161,11 @@
vrf_rx_stats(vrf_dev, skb->len);
- skb_push(skb, skb->mac_len);
- dev_queue_xmit_nit(skb, vrf_dev);
- skb_pull(skb, skb->mac_len);
+ if (!list_empty(&vrf_dev->ptype_all)) {
+ skb_push(skb, skb->mac_len);
+ dev_queue_xmit_nit(skb, vrf_dev);
+ skb_pull(skb, skb->mac_len);
+ }
skb = vrf_rcv_nfhook(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, vrf_dev);
out:
@@ -1146,11 +1288,11 @@
/* fib_nl_{new,del}rule handling looks for net from skb->sk */
skb->sk = dev_net(dev)->rtnl;
if (add_it) {
- err = fib_nl_newrule(skb, nlh);
+ err = fib_nl_newrule(skb, nlh, NULL);
if (err == -EEXIST)
err = 0;
} else {
- err = fib_nl_delrule(skb, nlh);
+ err = fib_nl_delrule(skb, nlh, NULL);
if (err == -ENOENT)
err = 0;
}
diff --git a/drivers/net/vsockmon.c b/drivers/net/vsockmon.c
new file mode 100644
index 0000000..7f0136f
--- /dev/null
+++ b/drivers/net/vsockmon.c
@@ -0,0 +1,170 @@
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/if_arp.h>
+#include <net/rtnetlink.h>
+#include <net/sock.h>
+#include <net/af_vsock.h>
+#include <uapi/linux/vsockmon.h>
+#include <linux/virtio_vsock.h>
+
+/* Virtio transport max packet size plus header */
+#define DEFAULT_MTU (VIRTIO_VSOCK_MAX_PKT_BUF_SIZE + \
+ sizeof(struct af_vsockmon_hdr))
+
+struct pcpu_lstats {
+ u64 rx_packets;
+ u64 rx_bytes;
+ struct u64_stats_sync syncp;
+};
+
+static int vsockmon_dev_init(struct net_device *dev)
+{
+ dev->lstats = netdev_alloc_pcpu_stats(struct pcpu_lstats);
+ if (!dev->lstats)
+ return -ENOMEM;
+ return 0;
+}
+
+static void vsockmon_dev_uninit(struct net_device *dev)
+{
+ free_percpu(dev->lstats);
+}
+
+struct vsockmon {
+ struct vsock_tap vt;
+};
+
+static int vsockmon_open(struct net_device *dev)
+{
+ struct vsockmon *vsockmon = netdev_priv(dev);
+
+ vsockmon->vt.dev = dev;
+ vsockmon->vt.module = THIS_MODULE;
+ return vsock_add_tap(&vsockmon->vt);
+}
+
+static int vsockmon_close(struct net_device *dev)
+{
+ struct vsockmon *vsockmon = netdev_priv(dev);
+
+ return vsock_remove_tap(&vsockmon->vt);
+}
+
+static netdev_tx_t vsockmon_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+ int len = skb->len;
+ struct pcpu_lstats *stats = this_cpu_ptr(dev->lstats);
+
+ u64_stats_update_begin(&stats->syncp);
+ stats->rx_bytes += len;
+ stats->rx_packets++;
+ u64_stats_update_end(&stats->syncp);
+
+ dev_kfree_skb(skb);
+
+ return NETDEV_TX_OK;
+}
+
+static void
+vsockmon_get_stats64(struct net_device *dev, struct rtnl_link_stats64 *stats)
+{
+ int i;
+ u64 bytes = 0, packets = 0;
+
+ for_each_possible_cpu(i) {
+ const struct pcpu_lstats *vstats;
+ u64 tbytes, tpackets;
+ unsigned int start;
+
+ vstats = per_cpu_ptr(dev->lstats, i);
+
+ do {
+ start = u64_stats_fetch_begin_irq(&vstats->syncp);
+ tbytes = vstats->rx_bytes;
+ tpackets = vstats->rx_packets;
+ } while (u64_stats_fetch_retry_irq(&vstats->syncp, start));
+
+ packets += tpackets;
+ bytes += tbytes;
+ }
+
+ stats->rx_packets = packets;
+ stats->tx_packets = 0;
+
+ stats->rx_bytes = bytes;
+ stats->tx_bytes = 0;
+}
+
+static int vsockmon_is_valid_mtu(int new_mtu)
+{
+ return new_mtu >= (int)sizeof(struct af_vsockmon_hdr);
+}
+
+static int vsockmon_change_mtu(struct net_device *dev, int new_mtu)
+{
+ if (!vsockmon_is_valid_mtu(new_mtu))
+ return -EINVAL;
+
+ dev->mtu = new_mtu;
+ return 0;
+}
+
+static const struct net_device_ops vsockmon_ops = {
+ .ndo_init = vsockmon_dev_init,
+ .ndo_uninit = vsockmon_dev_uninit,
+ .ndo_open = vsockmon_open,
+ .ndo_stop = vsockmon_close,
+ .ndo_start_xmit = vsockmon_xmit,
+ .ndo_get_stats64 = vsockmon_get_stats64,
+ .ndo_change_mtu = vsockmon_change_mtu,
+};
+
+static u32 always_on(struct net_device *dev)
+{
+ return 1;
+}
+
+static const struct ethtool_ops vsockmon_ethtool_ops = {
+ .get_link = always_on,
+};
+
+static void vsockmon_setup(struct net_device *dev)
+{
+ dev->type = ARPHRD_VSOCKMON;
+ dev->priv_flags |= IFF_NO_QUEUE;
+
+ dev->netdev_ops = &vsockmon_ops;
+ dev->ethtool_ops = &vsockmon_ethtool_ops;
+ dev->destructor = free_netdev;
+
+ dev->features = NETIF_F_SG | NETIF_F_FRAGLIST |
+ NETIF_F_HIGHDMA | NETIF_F_LLTX;
+
+ dev->flags = IFF_NOARP;
+
+ dev->mtu = DEFAULT_MTU;
+}
+
+static struct rtnl_link_ops vsockmon_link_ops __read_mostly = {
+ .kind = "vsockmon",
+ .priv_size = sizeof(struct vsockmon),
+ .setup = vsockmon_setup,
+};
+
+static __init int vsockmon_register(void)
+{
+ return rtnl_link_register(&vsockmon_link_ops);
+}
+
+static __exit void vsockmon_unregister(void)
+{
+ rtnl_link_unregister(&vsockmon_link_ops);
+}
+
+module_init(vsockmon_register);
+module_exit(vsockmon_unregister);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Gerard Garcia <ggarcia@deic.uab.cat>");
+MODULE_DESCRIPTION("Vsock monitoring device. Based on nlmon device.");
+MODULE_ALIAS_RTNL_LINK("vsockmon");
diff --git a/drivers/net/vxlan.c b/drivers/net/vxlan.c
index bdb6ae1..328b471 100644
--- a/drivers/net/vxlan.c
+++ b/drivers/net/vxlan.c
@@ -276,9 +276,9 @@
send_eth = send_ip = true;
if (type == RTM_GETNEIGH) {
- ndm->ndm_family = AF_INET;
send_ip = !vxlan_addr_any(&rdst->remote_ip);
send_eth = !is_zero_ether_addr(fdb->eth_addr);
+ ndm->ndm_family = send_ip ? rdst->remote_ip.sa.sa_family : AF_INET;
} else
ndm->ndm_family = AF_BRIDGE;
ndm->ndm_state = fdb->state;
@@ -1515,7 +1515,7 @@
int ns_olen;
int i, len;
- if (dev == NULL)
+ if (dev == NULL || !pskb_may_pull(request, request->len))
return NULL;
len = LL_RESERVED_SPACE(dev) + sizeof(struct ipv6hdr) +
@@ -1530,10 +1530,11 @@
skb_push(reply, sizeof(struct ethhdr));
skb_reset_mac_header(reply);
- ns = (struct nd_msg *)skb_transport_header(request);
+ ns = (struct nd_msg *)(ipv6_hdr(request) + 1);
daddr = eth_hdr(request)->h_source;
- ns_olen = request->len - skb_transport_offset(request) - sizeof(*ns);
+ ns_olen = request->len - skb_network_offset(request) -
+ sizeof(struct ipv6hdr) - sizeof(*ns);
for (i = 0; i < ns_olen-1; i += (ns->opt[i+1]<<3)) {
if (ns->opt[i] == ND_OPT_SOURCE_LL_ADDR) {
daddr = ns->opt + i + sizeof(struct nd_opt_hdr);
@@ -1604,10 +1605,13 @@
if (!in6_dev)
goto out;
+ if (!pskb_may_pull(skb, sizeof(struct ipv6hdr) + sizeof(struct nd_msg)))
+ goto out;
+
iphdr = ipv6_hdr(skb);
daddr = &iphdr->daddr;
- msg = (struct nd_msg *)skb_transport_header(skb);
+ msg = (struct nd_msg *)(iphdr + 1);
if (msg->icmph.icmp6_code != 0 ||
msg->icmph.icmp6_type != NDISC_NEIGHBOUR_SOLICITATION)
goto out;
@@ -2242,16 +2246,13 @@
if (ntohs(eth->h_proto) == ETH_P_ARP)
return arp_reduce(dev, skb, vni);
#if IS_ENABLED(CONFIG_IPV6)
- else if (ntohs(eth->h_proto) == ETH_P_IPV6 &&
- pskb_may_pull(skb, sizeof(struct ipv6hdr)
- + sizeof(struct nd_msg)) &&
- ipv6_hdr(skb)->nexthdr == IPPROTO_ICMPV6) {
- struct nd_msg *msg;
-
- msg = (struct nd_msg *)skb_transport_header(skb);
- if (msg->icmph.icmp6_code == 0 &&
- msg->icmph.icmp6_type == NDISC_NEIGHBOUR_SOLICITATION)
- return neigh_reduce(dev, skb, vni);
+ else if (ntohs(eth->h_proto) == ETH_P_IPV6) {
+ struct ipv6hdr *hdr, _hdr;
+ if ((hdr = skb_header_pointer(skb,
+ skb_network_offset(skb),
+ sizeof(_hdr), &_hdr)) &&
+ hdr->nexthdr == IPPROTO_ICMPV6)
+ return neigh_reduce(dev, skb, vni);
}
#endif
}
@@ -2322,6 +2323,9 @@
if (f->state & (NUD_PERMANENT | NUD_NOARP))
continue;
+ if (f->flags & NTF_EXT_LEARNED)
+ continue;
+
timeout = f->used + vxlan->cfg.age_interval * HZ;
if (time_before_eq(timeout, jiffies)) {
netdev_dbg(vxlan->dev,
@@ -2754,8 +2758,6 @@
sock = vxlan_create_sock(net, ipv6, port, flags);
if (IS_ERR(sock)) {
- pr_info("Cannot bind port %d, err=%ld\n", ntohs(port),
- PTR_ERR(sock));
kfree(vs);
return ERR_CAST(sock);
}
@@ -2818,17 +2820,21 @@
static int vxlan_sock_add(struct vxlan_dev *vxlan)
{
- bool ipv6 = vxlan->flags & VXLAN_F_IPV6;
bool metadata = vxlan->flags & VXLAN_F_COLLECT_METADATA;
+ bool ipv6 = vxlan->flags & VXLAN_F_IPV6 || metadata;
+ bool ipv4 = !ipv6 || metadata;
int ret = 0;
RCU_INIT_POINTER(vxlan->vn4_sock, NULL);
#if IS_ENABLED(CONFIG_IPV6)
RCU_INIT_POINTER(vxlan->vn6_sock, NULL);
- if (ipv6 || metadata)
+ if (ipv6) {
ret = __vxlan_sock_add(vxlan, true);
+ if (ret < 0 && ret != -EAFNOSUPPORT)
+ ipv4 = false;
+ }
#endif
- if (!ret && (!ipv6 || metadata))
+ if (ipv4)
ret = __vxlan_sock_add(vxlan, false);
if (ret < 0)
vxlan_sock_release(vxlan);
@@ -2923,6 +2929,11 @@
return -EINVAL;
}
+ if (lowerdev) {
+ dev->gso_max_size = lowerdev->gso_max_size;
+ dev->gso_max_segs = lowerdev->gso_max_segs;
+ }
+
if (conf->mtu) {
int max_mtu = ETH_MAX_MTU;
diff --git a/drivers/net/wan/pc300too.c b/drivers/net/wan/pc300too.c
index e1dd1ec..b9b934b 100644
--- a/drivers/net/wan/pc300too.c
+++ b/drivers/net/wan/pc300too.c
@@ -346,6 +346,7 @@
card->rambase == NULL) {
pr_err("ioremap() failed\n");
pc300_pci_remove_one(pdev);
+ return -ENOMEM;
}
/* PLX PCI 9050 workaround for local configuration register read bug */
diff --git a/drivers/net/wireless/admtek/adm8211.c b/drivers/net/wireless/admtek/adm8211.c
index 098c814..ed626f56 100644
--- a/drivers/net/wireless/admtek/adm8211.c
+++ b/drivers/net/wireless/admtek/adm8211.c
@@ -1917,6 +1917,8 @@
dev->wiphy->bands[NL80211_BAND_2GHZ] = &priv->band;
+ wiphy_ext_feature_set(dev->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
err = ieee80211_register_hw(dev);
if (err) {
printk(KERN_ERR "%s (adm8211): Cannot register device\n",
diff --git a/drivers/net/wireless/ath/ar5523/ar5523.c b/drivers/net/wireless/ath/ar5523/ar5523.c
index 7a60d2e..f2f4ccf 100644
--- a/drivers/net/wireless/ath/ar5523/ar5523.c
+++ b/drivers/net/wireless/ath/ar5523/ar5523.c
@@ -1689,6 +1689,8 @@
if (error)
goto out_cancel_rx_cmd;
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
usb_set_intfdata(intf, hw);
error = ieee80211_register_hw(hw);
diff --git a/drivers/net/wireless/ath/ath10k/ahb.c b/drivers/net/wireless/ath/ath10k/ahb.c
index 45226db..da770af 100644
--- a/drivers/net/wireless/ath/ath10k/ahb.c
+++ b/drivers/net/wireless/ath/ath10k/ahb.c
@@ -640,6 +640,7 @@
{
ath10k_dbg(ar, ATH10K_DBG_BOOT, "boot ahb hif start\n");
+ napi_enable(&ar->napi);
ath10k_ce_enable_interrupts(ar);
ath10k_pci_enable_legacy_irq(ar);
@@ -692,7 +693,6 @@
ath10k_err(ar, "could not wake up target CPU: %d\n", ret);
goto err_ce_deinit;
}
- napi_enable(&ar->napi);
return 0;
diff --git a/drivers/net/wireless/ath/ath10k/bmi.c b/drivers/net/wireless/ath/ath10k/bmi.c
index 2872d34..abeee20 100644
--- a/drivers/net/wireless/ath/ath10k/bmi.c
+++ b/drivers/net/wireless/ath/ath10k/bmi.c
@@ -19,12 +19,21 @@
#include "hif.h"
#include "debug.h"
#include "htc.h"
+#include "hw.h"
void ath10k_bmi_start(struct ath10k *ar)
{
+ int ret;
+
ath10k_dbg(ar, ATH10K_DBG_BMI, "bmi start\n");
ar->bmi.done_sent = false;
+
+ /* Enable hardware clock to speed up firmware download */
+ if (ar->hw_params.hw_ops->enable_pll_clk) {
+ ret = ar->hw_params.hw_ops->enable_pll_clk(ar);
+ ath10k_dbg(ar, ATH10K_DBG_BMI, "bmi enable pll ret %d\n", ret);
+ }
}
int ath10k_bmi_done(struct ath10k *ar)
@@ -129,6 +138,69 @@
return 0;
}
+int ath10k_bmi_write_soc_reg(struct ath10k *ar, u32 address, u32 reg_val)
+{
+ struct bmi_cmd cmd;
+ u32 cmdlen = sizeof(cmd.id) + sizeof(cmd.write_soc_reg);
+ int ret;
+
+ ath10k_dbg(ar, ATH10K_DBG_BMI,
+ "bmi write soc register 0x%08x val 0x%08x\n",
+ address, reg_val);
+
+ if (ar->bmi.done_sent) {
+ ath10k_warn(ar, "bmi write soc register command in progress\n");
+ return -EBUSY;
+ }
+
+ cmd.id = __cpu_to_le32(BMI_WRITE_SOC_REGISTER);
+ cmd.write_soc_reg.addr = __cpu_to_le32(address);
+ cmd.write_soc_reg.value = __cpu_to_le32(reg_val);
+
+ ret = ath10k_hif_exchange_bmi_msg(ar, &cmd, cmdlen, NULL, NULL);
+ if (ret) {
+ ath10k_warn(ar, "Unable to write soc register to device: %d\n",
+ ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+int ath10k_bmi_read_soc_reg(struct ath10k *ar, u32 address, u32 *reg_val)
+{
+ struct bmi_cmd cmd;
+ union bmi_resp resp;
+ u32 cmdlen = sizeof(cmd.id) + sizeof(cmd.read_soc_reg);
+ u32 resplen = sizeof(resp.read_soc_reg);
+ int ret;
+
+ ath10k_dbg(ar, ATH10K_DBG_BMI, "bmi read soc register 0x%08x\n",
+ address);
+
+ if (ar->bmi.done_sent) {
+ ath10k_warn(ar, "bmi read soc register command in progress\n");
+ return -EBUSY;
+ }
+
+ cmd.id = __cpu_to_le32(BMI_READ_SOC_REGISTER);
+ cmd.read_soc_reg.addr = __cpu_to_le32(address);
+
+ ret = ath10k_hif_exchange_bmi_msg(ar, &cmd, cmdlen, &resp, &resplen);
+ if (ret) {
+ ath10k_warn(ar, "Unable to read soc register from device: %d\n",
+ ret);
+ return ret;
+ }
+
+ *reg_val = __le32_to_cpu(resp.read_soc_reg.value);
+
+ ath10k_dbg(ar, ATH10K_DBG_BMI, "bmi read soc register value 0x%08x\n",
+ *reg_val);
+
+ return 0;
+}
+
int ath10k_bmi_write_memory(struct ath10k *ar,
u32 address, const void *buffer, u32 length)
{
diff --git a/drivers/net/wireless/ath/ath10k/bmi.h b/drivers/net/wireless/ath/ath10k/bmi.h
index 7d3231a..cc45b63 100644
--- a/drivers/net/wireless/ath/ath10k/bmi.h
+++ b/drivers/net/wireless/ath/ath10k/bmi.h
@@ -176,7 +176,8 @@
} rompatch_uninstall;
struct {
/* 0 = nothing executed
- * otherwise = NVRAM segment return value */
+ * otherwise = NVRAM segment return value
+ */
__le32 result;
} nvram_process;
u8 payload[BMI_MAX_CMDBUF_SIZE];
@@ -232,4 +233,6 @@
int ath10k_bmi_lz_data(struct ath10k *ar, const void *buffer, u32 length);
int ath10k_bmi_fast_download(struct ath10k *ar, u32 address,
const void *buffer, u32 length);
+int ath10k_bmi_read_soc_reg(struct ath10k *ar, u32 address, u32 *reg_val);
+int ath10k_bmi_write_soc_reg(struct ath10k *ar, u32 address, u32 reg_val);
#endif /* _BMI_H_ */
diff --git a/drivers/net/wireless/ath/ath10k/ce.c b/drivers/net/wireless/ath/ath10k/ce.c
index 4045657..ee1090c 100644
--- a/drivers/net/wireless/ath/ath10k/ce.c
+++ b/drivers/net/wireless/ath/ath10k/ce.c
@@ -261,8 +261,7 @@
}
/*
- * Guts of ath10k_ce_send, used by both ath10k_ce_send and
- * ath10k_ce_sendlist_send.
+ * Guts of ath10k_ce_send.
* The caller takes responsibility for any needed locking.
*/
int ath10k_ce_send_nolock(struct ath10k_ce_pipe *ce_state,
@@ -1052,7 +1051,7 @@
*/
BUILD_BUG_ON(2 * TARGET_NUM_MSDU_DESC >
(CE_HTT_H2T_MSG_SRC_NENTRIES - 1));
- BUILD_BUG_ON(2 * TARGET_10X_NUM_MSDU_DESC >
+ BUILD_BUG_ON(2 * TARGET_10_4_NUM_MSDU_DESC_PFC >
(CE_HTT_H2T_MSG_SRC_NENTRIES - 1));
BUILD_BUG_ON(2 * TARGET_TLV_NUM_MSDU_DESC >
(CE_HTT_H2T_MSG_SRC_NENTRIES - 1));
diff --git a/drivers/net/wireless/ath/ath10k/core.c b/drivers/net/wireless/ath/ath10k/core.c
index 0a8e29e..5a06389 100644
--- a/drivers/net/wireless/ath/ath10k/core.c
+++ b/drivers/net/wireless/ath/ath10k/core.c
@@ -71,6 +71,7 @@
},
.hw_ops = &qca988x_ops,
.decap_align_bytes = 4,
+ .spectral_bin_discard = 0,
},
{
.id = QCA9887_HW_1_0_VERSION,
@@ -91,6 +92,7 @@
},
.hw_ops = &qca988x_ops,
.decap_align_bytes = 4,
+ .spectral_bin_discard = 0,
},
{
.id = QCA6174_HW_2_1_VERSION,
@@ -110,6 +112,7 @@
},
.hw_ops = &qca988x_ops,
.decap_align_bytes = 4,
+ .spectral_bin_discard = 0,
},
{
.id = QCA6174_HW_2_1_VERSION,
@@ -129,6 +132,7 @@
},
.hw_ops = &qca988x_ops,
.decap_align_bytes = 4,
+ .spectral_bin_discard = 0,
},
{
.id = QCA6174_HW_3_0_VERSION,
@@ -148,6 +152,7 @@
},
.hw_ops = &qca988x_ops,
.decap_align_bytes = 4,
+ .spectral_bin_discard = 0,
},
{
.id = QCA6174_HW_3_2_VERSION,
@@ -166,8 +171,11 @@
.board_size = QCA6174_BOARD_DATA_SZ,
.board_ext_size = QCA6174_BOARD_EXT_DATA_SZ,
},
- .hw_ops = &qca988x_ops,
+ .hw_ops = &qca6174_ops,
+ .hw_clk = qca6174_clk,
+ .target_cpu_freq = 176000000,
.decap_align_bytes = 4,
+ .spectral_bin_discard = 0,
},
{
.id = QCA99X0_HW_2_0_DEV_VERSION,
@@ -193,6 +201,7 @@
.sw_decrypt_mcast_mgmt = true,
.hw_ops = &qca99x0_ops,
.decap_align_bytes = 1,
+ .spectral_bin_discard = 4,
},
{
.id = QCA9984_HW_1_0_DEV_VERSION,
@@ -219,6 +228,7 @@
.sw_decrypt_mcast_mgmt = true,
.hw_ops = &qca99x0_ops,
.decap_align_bytes = 1,
+ .spectral_bin_discard = 12,
},
{
.id = QCA9888_HW_2_0_DEV_VERSION,
@@ -244,6 +254,7 @@
.sw_decrypt_mcast_mgmt = true,
.hw_ops = &qca99x0_ops,
.decap_align_bytes = 1,
+ .spectral_bin_discard = 12,
},
{
.id = QCA9377_HW_1_0_DEV_VERSION,
@@ -263,6 +274,7 @@
},
.hw_ops = &qca988x_ops,
.decap_align_bytes = 4,
+ .spectral_bin_discard = 0,
},
{
.id = QCA9377_HW_1_1_DEV_VERSION,
@@ -280,8 +292,11 @@
.board_size = QCA9377_BOARD_DATA_SZ,
.board_ext_size = QCA9377_BOARD_EXT_DATA_SZ,
},
- .hw_ops = &qca988x_ops,
+ .hw_ops = &qca6174_ops,
+ .hw_clk = qca6174_clk,
+ .target_cpu_freq = 176000000,
.decap_align_bytes = 4,
+ .spectral_bin_discard = 0,
},
{
.id = QCA4019_HW_1_0_DEV_VERSION,
@@ -308,6 +323,7 @@
.sw_decrypt_mcast_mgmt = true,
.hw_ops = &qca99x0_ops,
.decap_align_bytes = 1,
+ .spectral_bin_discard = 4,
},
};
@@ -1623,6 +1639,13 @@
wake_up(&ar->wmi.tx_credits_wq);
wake_up(&ar->peer_mapping_wq);
+ /* TODO: We can have one instance of cancelling coverage_class_work by
+ * moving it to ath10k_halt(), so that both stop() and restart() would
+ * call that but it takes conf_mutex() and if we call cancel_work_sync()
+ * with conf_mutex it will deadlock.
+ */
+ cancel_work_sync(&ar->set_coverage_class_work);
+
mutex_lock(&ar->conf_mutex);
switch (ar->state) {
@@ -1634,7 +1657,8 @@
break;
case ATH10K_STATE_OFF:
/* this can happen if driver is being unloaded
- * or if the crash happens during FW probing */
+ * or if the crash happens during FW probing
+ */
ath10k_warn(ar, "cannot restart a device that hasn't been started\n");
break;
case ATH10K_STATE_RESTARTING:
@@ -2162,7 +2186,8 @@
/* mac80211 manages fw/hw initialization through start/stop hooks. However in
* order to know what hw capabilities should be advertised to mac80211 it is
* necessary to load the firmware (and tear it down immediately since start
- * hook will try to init it again) before registering */
+ * hook will try to init it again) before registering
+ */
static int ath10k_core_probe_fw(struct ath10k *ar)
{
struct bmi_target_info target_info;
@@ -2356,7 +2381,8 @@
/* We must unregister from mac80211 before we stop HTC and HIF.
* Otherwise we will fail to submit commands to FW and mac80211 will be
- * unhappy about callback failures. */
+ * unhappy about callback failures.
+ */
ath10k_mac_unregister(ar);
ath10k_testmode_destroy(ar);
diff --git a/drivers/net/wireless/ath/ath10k/core.h b/drivers/net/wireless/ath/ath10k/core.h
index 88d14be..bf09151 100644
--- a/drivers/net/wireless/ath/ath10k/core.h
+++ b/drivers/net/wireless/ath/ath10k/core.h
@@ -501,14 +501,16 @@
* stopped in ath10k_core_restart() work holding conf_mutex. The state
* RESTARTED means that the device is up and mac80211 has started hw
* reconfiguration. Once mac80211 is done with the reconfiguration we
- * set the state to STATE_ON in reconfig_complete(). */
+ * set the state to STATE_ON in reconfig_complete().
+ */
ATH10K_STATE_RESTARTING,
ATH10K_STATE_RESTARTED,
/* The device has crashed while restarting hw. This state is like ON
* but commands are blocked in HTC and -ECOMM response is given. This
* prevents completion timeouts and makes the driver more responsive to
- * userspace commands. This is also prevents recursive recovery. */
+ * userspace commands. This is also prevents recursive recovery.
+ */
ATH10K_STATE_WEDGED,
/* factory tests */
@@ -775,6 +777,8 @@
u32 num_rf_chains;
u32 max_spatial_stream;
/* protected by conf_mutex */
+ u32 low_5ghz_chan;
+ u32 high_5ghz_chan;
bool ani_enabled;
bool p2p;
@@ -918,7 +922,8 @@
struct work_struct restart_work;
/* cycle count is reported twice for each visited channel during scan.
- * access protected by data_lock */
+ * access protected by data_lock
+ */
u32 survey_last_rx_clear_count;
u32 survey_last_cycle_count;
struct survey_info survey[ATH10K_NUM_CHANS];
diff --git a/drivers/net/wireless/ath/ath10k/debug.c b/drivers/net/wireless/ath/ath10k/debug.c
index fb0ade3..4cd2a0f 100644
--- a/drivers/net/wireless/ath/ath10k/debug.c
+++ b/drivers/net/wireless/ath/ath10k/debug.c
@@ -249,9 +249,6 @@
mutex_lock(&ar->conf_mutex);
- if (len > buf_len)
- len = buf_len;
-
spin_lock_bh(&ar->data_lock);
for (i = 0; i < WMI_SERVICE_MAX; i++) {
enabled = test_bit(i, ar->wmi.svc_map);
@@ -1819,7 +1816,7 @@
tpc_stats->num_tx_chain,
tpc_stats->rate_max);
- for (j = 0; j < tpc_stats->num_tx_chain ; j++) {
+ for (j = 0; j < WMI_TPC_FLAG; j++) {
switch (j) {
case WMI_TPC_TABLE_TYPE_CDD:
if (tpc_stats->flag[j] == ATH10K_TPC_TABLE_TYPE_FLAG) {
@@ -1985,7 +1982,8 @@
/* Must not use _sync to avoid deadlock, we do that in
* ath10k_debug_destroy(). The check for htt_stats_mask is to avoid
- * warning from del_timer(). */
+ * warning from del_timer().
+ */
if (ar->debug.htt_stats_mask != 0)
cancel_delayed_work(&ar->debug.htt_stats_dwork);
@@ -1997,6 +1995,15 @@
size_t count, loff_t *ppos)
{
struct ath10k *ar = file->private_data;
+ struct ath10k_vif *arvif;
+
+ /* Just check for for the first vif alone, as all the vifs will be
+ * sharing the same channel and if the channel is disabled, all the
+ * vifs will share the same 'is_started' state.
+ */
+ arvif = list_first_entry(&ar->arvifs, typeof(*arvif), list);
+ if (!arvif->is_started)
+ return -EINVAL;
ieee80211_radar_detected(ar->hw);
@@ -2437,86 +2444,82 @@
init_completion(&ar->debug.tpc_complete);
init_completion(&ar->debug.fw_stats_complete);
- debugfs_create_file("fw_stats", S_IRUSR, ar->debug.debugfs_phy, ar,
+ debugfs_create_file("fw_stats", 0400, ar->debug.debugfs_phy, ar,
&fops_fw_stats);
- debugfs_create_file("fw_reset_stats", S_IRUSR, ar->debug.debugfs_phy,
- ar, &fops_fw_reset_stats);
+ debugfs_create_file("fw_reset_stats", 0400, ar->debug.debugfs_phy, ar,
+ &fops_fw_reset_stats);
- debugfs_create_file("wmi_services", S_IRUSR, ar->debug.debugfs_phy, ar,
+ debugfs_create_file("wmi_services", 0400, ar->debug.debugfs_phy, ar,
&fops_wmi_services);
- debugfs_create_file("simulate_fw_crash", S_IRUSR | S_IWUSR,
- ar->debug.debugfs_phy, ar, &fops_simulate_fw_crash);
+ debugfs_create_file("simulate_fw_crash", 0600, ar->debug.debugfs_phy, ar,
+ &fops_simulate_fw_crash);
- debugfs_create_file("fw_crash_dump", S_IRUSR, ar->debug.debugfs_phy,
- ar, &fops_fw_crash_dump);
+ debugfs_create_file("fw_crash_dump", 0400, ar->debug.debugfs_phy, ar,
+ &fops_fw_crash_dump);
- debugfs_create_file("reg_addr", S_IRUSR | S_IWUSR,
- ar->debug.debugfs_phy, ar, &fops_reg_addr);
+ debugfs_create_file("reg_addr", 0600, ar->debug.debugfs_phy, ar,
+ &fops_reg_addr);
- debugfs_create_file("reg_value", S_IRUSR | S_IWUSR,
- ar->debug.debugfs_phy, ar, &fops_reg_value);
+ debugfs_create_file("reg_value", 0600, ar->debug.debugfs_phy, ar,
+ &fops_reg_value);
- debugfs_create_file("mem_value", S_IRUSR | S_IWUSR,
- ar->debug.debugfs_phy, ar, &fops_mem_value);
+ debugfs_create_file("mem_value", 0600, ar->debug.debugfs_phy, ar,
+ &fops_mem_value);
- debugfs_create_file("chip_id", S_IRUSR, ar->debug.debugfs_phy,
- ar, &fops_chip_id);
+ debugfs_create_file("chip_id", 0400, ar->debug.debugfs_phy, ar,
+ &fops_chip_id);
- debugfs_create_file("htt_stats_mask", S_IRUSR | S_IWUSR,
- ar->debug.debugfs_phy, ar, &fops_htt_stats_mask);
+ debugfs_create_file("htt_stats_mask", 0600, ar->debug.debugfs_phy, ar,
+ &fops_htt_stats_mask);
- debugfs_create_file("htt_max_amsdu_ampdu", S_IRUSR | S_IWUSR,
- ar->debug.debugfs_phy, ar,
+ debugfs_create_file("htt_max_amsdu_ampdu", 0600, ar->debug.debugfs_phy, ar,
&fops_htt_max_amsdu_ampdu);
- debugfs_create_file("fw_dbglog", S_IRUSR | S_IWUSR,
- ar->debug.debugfs_phy, ar, &fops_fw_dbglog);
+ debugfs_create_file("fw_dbglog", 0600, ar->debug.debugfs_phy, ar,
+ &fops_fw_dbglog);
- debugfs_create_file("cal_data", S_IRUSR, ar->debug.debugfs_phy,
- ar, &fops_cal_data);
+ debugfs_create_file("cal_data", 0400, ar->debug.debugfs_phy, ar,
+ &fops_cal_data);
- debugfs_create_file("ani_enable", S_IRUSR | S_IWUSR,
- ar->debug.debugfs_phy, ar, &fops_ani_enable);
+ debugfs_create_file("ani_enable", 0600, ar->debug.debugfs_phy, ar,
+ &fops_ani_enable);
- debugfs_create_file("nf_cal_period", S_IRUSR | S_IWUSR,
- ar->debug.debugfs_phy, ar, &fops_nf_cal_period);
+ debugfs_create_file("nf_cal_period", 0600, ar->debug.debugfs_phy, ar,
+ &fops_nf_cal_period);
if (IS_ENABLED(CONFIG_ATH10K_DFS_CERTIFIED)) {
- debugfs_create_file("dfs_simulate_radar", S_IWUSR,
- ar->debug.debugfs_phy, ar,
- &fops_simulate_radar);
+ debugfs_create_file("dfs_simulate_radar", 0200, ar->debug.debugfs_phy,
+ ar, &fops_simulate_radar);
- debugfs_create_bool("dfs_block_radar_events", S_IWUSR,
+ debugfs_create_bool("dfs_block_radar_events", 0200,
ar->debug.debugfs_phy,
&ar->dfs_block_radar_events);
- debugfs_create_file("dfs_stats", S_IRUSR,
- ar->debug.debugfs_phy, ar,
+ debugfs_create_file("dfs_stats", 0400, ar->debug.debugfs_phy, ar,
&fops_dfs_stats);
}
- debugfs_create_file("pktlog_filter", S_IRUGO | S_IWUSR,
- ar->debug.debugfs_phy, ar, &fops_pktlog_filter);
+ debugfs_create_file("pktlog_filter", 0644, ar->debug.debugfs_phy, ar,
+ &fops_pktlog_filter);
- debugfs_create_file("quiet_period", S_IRUGO | S_IWUSR,
- ar->debug.debugfs_phy, ar, &fops_quiet_period);
+ debugfs_create_file("quiet_period", 0644, ar->debug.debugfs_phy, ar,
+ &fops_quiet_period);
- debugfs_create_file("tpc_stats", S_IRUSR,
- ar->debug.debugfs_phy, ar, &fops_tpc_stats);
+ debugfs_create_file("tpc_stats", 0400, ar->debug.debugfs_phy, ar,
+ &fops_tpc_stats);
if (test_bit(WMI_SERVICE_COEX_GPIO, ar->wmi.svc_map))
- debugfs_create_file("btcoex", S_IRUGO | S_IWUSR,
- ar->debug.debugfs_phy, ar, &fops_btcoex);
+ debugfs_create_file("btcoex", 0644, ar->debug.debugfs_phy, ar,
+ &fops_btcoex);
if (test_bit(WMI_SERVICE_PEER_STATS, ar->wmi.svc_map))
- debugfs_create_file("peer_stats", S_IRUGO | S_IWUSR,
- ar->debug.debugfs_phy, ar,
+ debugfs_create_file("peer_stats", 0644, ar->debug.debugfs_phy, ar,
&fops_peer_stats);
- debugfs_create_file("fw_checksums", S_IRUSR,
- ar->debug.debugfs_phy, ar, &fops_fw_checksums);
+ debugfs_create_file("fw_checksums", 0400, ar->debug.debugfs_phy, ar,
+ &fops_fw_checksums);
return 0;
}
diff --git a/drivers/net/wireless/ath/ath10k/debugfs_sta.c b/drivers/net/wireless/ath/ath10k/debugfs_sta.c
index 7353e7e..d59ac6b 100644
--- a/drivers/net/wireless/ath/ath10k/debugfs_sta.c
+++ b/drivers/net/wireless/ath/ath10k/debugfs_sta.c
@@ -372,11 +372,10 @@
void ath10k_sta_add_debugfs(struct ieee80211_hw *hw, struct ieee80211_vif *vif,
struct ieee80211_sta *sta, struct dentry *dir)
{
- debugfs_create_file("aggr_mode", S_IRUGO | S_IWUSR, dir, sta,
- &fops_aggr_mode);
- debugfs_create_file("addba", S_IWUSR, dir, sta, &fops_addba);
- debugfs_create_file("addba_resp", S_IWUSR, dir, sta, &fops_addba_resp);
- debugfs_create_file("delba", S_IWUSR, dir, sta, &fops_delba);
+ debugfs_create_file("aggr_mode", 0644, dir, sta, &fops_aggr_mode);
+ debugfs_create_file("addba", 0200, dir, sta, &fops_addba);
+ debugfs_create_file("addba_resp", 0200, dir, sta, &fops_addba_resp);
+ debugfs_create_file("delba", 0200, dir, sta, &fops_delba);
debugfs_create_file("peer_debug_trigger", 0600, dir, sta,
&fops_peer_debug_trigger);
}
diff --git a/drivers/net/wireless/ath/ath10k/hif.h b/drivers/net/wireless/ath/ath10k/hif.h
index b2566b0..6679dd9 100644
--- a/drivers/net/wireless/ath/ath10k/hif.h
+++ b/drivers/net/wireless/ath/ath10k/hif.h
@@ -54,7 +54,8 @@
int (*start)(struct ath10k *ar);
/* Clean up what start() did. This does not revert to BMI phase. If
- * desired so, call power_down() and power_up() */
+ * desired so, call power_down() and power_up()
+ */
void (*stop)(struct ath10k *ar);
int (*map_service_to_pipe)(struct ath10k *ar, u16 service_id,
@@ -82,7 +83,8 @@
int (*power_up)(struct ath10k *ar);
/* Power down the device and free up resources. stop() must be called
- * before this if start() was called earlier */
+ * before this if start() was called earlier
+ */
void (*power_down)(struct ath10k *ar);
int (*suspend)(struct ath10k *ar);
diff --git a/drivers/net/wireless/ath/ath10k/htc.c b/drivers/net/wireless/ath/ath10k/htc.c
index 9f6a915..b7669b2 100644
--- a/drivers/net/wireless/ath/ath10k/htc.c
+++ b/drivers/net/wireless/ath/ath10k/htc.c
@@ -119,6 +119,9 @@
credits = DIV_ROUND_UP(skb->len, htc->target_credit_size);
spin_lock_bh(&htc->tx_lock);
if (ep->tx_credits < credits) {
+ ath10k_dbg(ar, ATH10K_DBG_HTC,
+ "htc insufficient credits ep %d required %d available %d\n",
+ eid, credits, ep->tx_credits);
spin_unlock_bh(&htc->tx_lock);
ret = -EAGAIN;
goto err_pull;
@@ -419,7 +422,8 @@
struct sk_buff *skb)
{
/* This is unexpected. FW is not supposed to send regular rx on this
- * endpoint. */
+ * endpoint.
+ */
ath10k_warn(ar, "unexpected htc rx\n");
kfree_skb(skb);
}
diff --git a/drivers/net/wireless/ath/ath10k/htt.h b/drivers/net/wireless/ath/ath10k/htt.h
index 90c2f72..6305308 100644
--- a/drivers/net/wireless/ath/ath10k/htt.h
+++ b/drivers/net/wireless/ath/ath10k/htt.h
@@ -51,7 +51,8 @@
HTT_H2T_MSG_TYPE_FRAG_DESC_BANK_CFG = 6,
/* This command is used for sending management frames in HTT < 3.0.
- * HTT >= 3.0 uses TX_FRM for everything. */
+ * HTT >= 3.0 uses TX_FRM for everything.
+ */
HTT_H2T_MSG_TYPE_MGMT_TX = 7,
HTT_H2T_MSG_TYPE_TX_FETCH_RESP = 11,
@@ -910,7 +911,8 @@
/* payload consists of 2 lists:
* a) num_ints * sizeof(__le32)
- * b) num_chars * sizeof(u8) aligned to 4bytes */
+ * b) num_chars * sizeof(u8) aligned to 4bytes
+ */
u8 payload[0];
} __packed;
@@ -1307,7 +1309,8 @@
} __packed;
/* real is 16 but it wouldn't fit in the max htt message size
- * so we use a conservatively safe value for now */
+ * so we use a conservatively safe value for now
+ */
#define HTT_FRAG_DESC_BANK_MAX 4
#define HTT_FRAG_DESC_BANK_CFG_INFO_PDEV_ID_MASK 0x03
@@ -1684,12 +1687,14 @@
DECLARE_KFIFO_PTR(txdone_fifo, struct htt_tx_done);
/* set if host-fw communication goes haywire
- * used to avoid further failures */
+ * used to avoid further failures
+ */
bool rx_confused;
atomic_t num_mpdus_ready;
/* This is used to group tx/rx completions separately and process them
- * in batches to reduce cache stalls */
+ * in batches to reduce cache stalls
+ */
struct sk_buff_head rx_compl_q;
struct sk_buff_head rx_in_ord_compl_q;
struct sk_buff_head tx_fetch_ind_q;
@@ -1725,11 +1730,13 @@
/* This structure layout is programmed via rx ring setup
* so that FW knows how to transfer the rx descriptor to the host.
- * Buffers like this are placed on the rx ring. */
+ * Buffers like this are placed on the rx ring.
+ */
struct htt_rx_desc {
union {
/* This field is filled on the host using the msdu buffer
- * from htt_rx_indication */
+ * from htt_rx_indication
+ */
struct fw_rx_desc_base fw_desc;
u32 pad;
} __packed;
@@ -1760,7 +1767,8 @@
#define HTT_RX_MSDU_SIZE (HTT_RX_BUF_SIZE - (int)sizeof(struct htt_rx_desc))
/* Refill a bunch of RX buffers for each refill round so that FW/HW can handle
- * aggregated traffic more nicely. */
+ * aggregated traffic more nicely.
+ */
#define ATH10K_HTT_MAX_NUM_REFILL 100
/*
diff --git a/drivers/net/wireless/ath/ath10k/htt_rx.c b/drivers/net/wireless/ath/ath10k/htt_rx.c
index 02a3fc8..84b6067 100644
--- a/drivers/net/wireless/ath/ath10k/htt_rx.c
+++ b/drivers/net/wireless/ath/ath10k/htt_rx.c
@@ -177,7 +177,8 @@
* automatically balances load wrt to CPU power.
*
* This probably comes at a cost of lower maximum throughput but
- * improves the average and stability. */
+ * improves the average and stability.
+ */
spin_lock_bh(&htt->rx_ring.lock);
num_deficit = htt->rx_ring.fill_level - htt->rx_ring.fill_cnt;
num_to_fill = min(ATH10K_HTT_MAX_NUM_REFILL, num_deficit);
@@ -304,7 +305,8 @@
rx_desc = (struct htt_rx_desc *)msdu->data;
/* FIXME: we must report msdu payload since this is what caller
- * expects now */
+ * expects now
+ */
skb_put(msdu, offsetof(struct htt_rx_desc, msdu_payload));
skb_pull(msdu, offsetof(struct htt_rx_desc, msdu_payload));
@@ -630,16 +632,17 @@
sgi = (info3 >> 7) & 1;
status->rate_idx = mcs;
- status->flag |= RX_FLAG_HT;
+ status->encoding = RX_ENC_HT;
if (sgi)
- status->flag |= RX_FLAG_SHORT_GI;
+ status->enc_flags |= RX_ENC_FLAG_SHORT_GI;
if (bw)
- status->flag |= RX_FLAG_40MHZ;
+ status->bw = RATE_INFO_BW_40;
break;
case HTT_RX_VHT:
case HTT_RX_VHT_WITH_TXBF:
/* VHT-SIG-A1 in info2, VHT-SIG-A2 in info3
- TODO check this */
+ * TODO check this
+ */
bw = info2 & 3;
sgi = info3 & 1;
group_id = (info2 >> 4) & 0x3F;
@@ -686,10 +689,10 @@
}
status->rate_idx = mcs;
- status->vht_nss = nss;
+ status->nss = nss;
if (sgi)
- status->flag |= RX_FLAG_SHORT_GI;
+ status->enc_flags |= RX_ENC_FLAG_SHORT_GI;
switch (bw) {
/* 20MHZ */
@@ -697,18 +700,18 @@
break;
/* 40MHZ */
case 1:
- status->flag |= RX_FLAG_40MHZ;
+ status->bw = RATE_INFO_BW_40;
break;
/* 80MHZ */
case 2:
- status->vht_flag |= RX_VHT_FLAG_80MHZ;
+ status->bw = RATE_INFO_BW_80;
break;
case 3:
- status->vht_flag |= RX_VHT_FLAG_160MHZ;
+ status->bw = RATE_INFO_BW_160;
break;
}
- status->flag |= RX_FLAG_VHT;
+ status->encoding = RX_ENC_VHT;
break;
default:
break;
@@ -871,13 +874,10 @@
/* New PPDU starts so clear out the old per-PPDU status. */
status->freq = 0;
status->rate_idx = 0;
- status->vht_nss = 0;
- status->vht_flag &= ~RX_VHT_FLAG_80MHZ;
- status->flag &= ~(RX_FLAG_HT |
- RX_FLAG_VHT |
- RX_FLAG_SHORT_GI |
- RX_FLAG_40MHZ |
- RX_FLAG_MACTIME_END);
+ status->nss = 0;
+ status->encoding = RX_ENC_LEGACY;
+ status->bw = RATE_INFO_BW_20;
+ status->flag &= ~RX_FLAG_MACTIME_END;
status->flag |= RX_FLAG_NO_SIGNAL_VAL;
ath10k_htt_rx_h_signal(ar, status, rxd);
@@ -930,7 +930,7 @@
*status = *rx_status;
ath10k_dbg(ar, ATH10K_DBG_DATA,
- "rx skb %pK len %u peer %pM %s %s sn %u %s%s%s%s%s%s %srate_idx %u vht_nss %u freq %u band %u flag 0x%llx fcs-err %i mic-err %i amsdu-more %i\n",
+ "rx skb %pK len %u peer %pM %s %s sn %u %s%s%s%s%s%s %srate_idx %u vht_nss %u freq %u band %u flag 0x%x fcs-err %i mic-err %i amsdu-more %i\n",
skb,
skb->len,
ieee80211_get_SA(hdr),
@@ -938,16 +938,15 @@
is_multicast_ether_addr(ieee80211_get_DA(hdr)) ?
"mcast" : "ucast",
(__le16_to_cpu(hdr->seq_ctrl) & IEEE80211_SCTL_SEQ) >> 4,
- (status->flag & (RX_FLAG_HT | RX_FLAG_VHT)) == 0 ?
- "legacy" : "",
- status->flag & RX_FLAG_HT ? "ht" : "",
- status->flag & RX_FLAG_VHT ? "vht" : "",
- status->flag & RX_FLAG_40MHZ ? "40" : "",
- status->vht_flag & RX_VHT_FLAG_80MHZ ? "80" : "",
- status->vht_flag & RX_VHT_FLAG_160MHZ ? "160" : "",
- status->flag & RX_FLAG_SHORT_GI ? "sgi " : "",
+ (status->encoding == RX_ENC_LEGACY) ? "legacy" : "",
+ (status->encoding == RX_ENC_HT) ? "ht" : "",
+ (status->encoding == RX_ENC_VHT) ? "vht" : "",
+ (status->bw == RATE_INFO_BW_40) ? "40" : "",
+ (status->bw == RATE_INFO_BW_80) ? "80" : "",
+ (status->bw == RATE_INFO_BW_160) ? "160" : "",
+ status->enc_flags & RX_ENC_FLAG_SHORT_GI ? "sgi " : "",
status->rate_idx,
- status->vht_nss,
+ status->nss,
status->freq,
status->band, status->flag,
!!(status->flag & RX_FLAG_FAILED_FCS_CRC),
diff --git a/drivers/net/wireless/ath/ath10k/htt_tx.c b/drivers/net/wireless/ath/ath10k/htt_tx.c
index 86b427f..685faac 100644
--- a/drivers/net/wireless/ath/ath10k/htt_tx.c
+++ b/drivers/net/wireless/ath/ath10k/htt_tx.c
@@ -526,7 +526,8 @@
memset(req, 0, sizeof(*req));
/* currently we support only max 8 bit masks so no need to worry
- * about endian support */
+ * about endian support
+ */
req->upload_types[0] = mask;
req->reset_types[0] = mask;
req->stat_type = HTT_STATS_REQ_CFG_STAT_TYPE_INVALID;
@@ -1008,7 +1009,8 @@
* There is simply no point in pushing HTT TX_FRM through HTC tx path
* as it's a waste of resources. By bypassing HTC it is possible to
* avoid extra memory allocations, compress data structures and thus
- * improve performance. */
+ * improve performance.
+ */
txbuf->htc_hdr.eid = htt->eid;
txbuf->htc_hdr.len = __cpu_to_le16(sizeof(txbuf->cmd_hdr) +
diff --git a/drivers/net/wireless/ath/ath10k/hw.c b/drivers/net/wireless/ath/ath10k/hw.c
index d9f37ee..c866ab5 100644
--- a/drivers/net/wireless/ath/ath10k/hw.c
+++ b/drivers/net/wireless/ath/ath10k/hw.c
@@ -19,6 +19,7 @@
#include "hw.h"
#include "hif.h"
#include "wmi-ops.h"
+#include "bmi.h"
const struct ath10k_hw_regs qca988x_regs = {
.rtc_soc_base_address = 0x00004000,
@@ -72,6 +73,9 @@
.pcie_intr_fw_mask = 0x00000400,
.pcie_intr_ce_mask_all = 0x0007f800,
.pcie_intr_clr_address = 0x00000014,
+ .cpu_pll_init_address = 0x00404020,
+ .cpu_speed_address = 0x00404024,
+ .core_clk_div_address = 0x00404028,
};
const struct ath10k_hw_regs qca99x0_regs = {
@@ -187,6 +191,73 @@
.ce_desc_meta_data_lsb = 4,
};
+const struct ath10k_hw_clk_params qca6174_clk[ATH10K_HW_REFCLK_COUNT] = {
+ {
+ .refclk = 48000000,
+ .div = 0xe,
+ .rnfrac = 0x2aaa8,
+ .settle_time = 2400,
+ .refdiv = 0,
+ .outdiv = 1,
+ },
+ {
+ .refclk = 19200000,
+ .div = 0x24,
+ .rnfrac = 0x2aaa8,
+ .settle_time = 960,
+ .refdiv = 0,
+ .outdiv = 1,
+ },
+ {
+ .refclk = 24000000,
+ .div = 0x1d,
+ .rnfrac = 0x15551,
+ .settle_time = 1200,
+ .refdiv = 0,
+ .outdiv = 1,
+ },
+ {
+ .refclk = 26000000,
+ .div = 0x1b,
+ .rnfrac = 0x4ec4,
+ .settle_time = 1300,
+ .refdiv = 0,
+ .outdiv = 1,
+ },
+ {
+ .refclk = 37400000,
+ .div = 0x12,
+ .rnfrac = 0x34b49,
+ .settle_time = 1870,
+ .refdiv = 0,
+ .outdiv = 1,
+ },
+ {
+ .refclk = 38400000,
+ .div = 0x12,
+ .rnfrac = 0x15551,
+ .settle_time = 1920,
+ .refdiv = 0,
+ .outdiv = 1,
+ },
+ {
+ .refclk = 40000000,
+ .div = 0x12,
+ .rnfrac = 0x26665,
+ .settle_time = 2000,
+ .refdiv = 0,
+ .outdiv = 1,
+ },
+ {
+ .refclk = 52000000,
+ .div = 0x1b,
+ .rnfrac = 0x4ec4,
+ .settle_time = 2600,
+ .refdiv = 0,
+ .outdiv = 1,
+ },
+};
+
void ath10k_hw_fill_survey_time(struct ath10k *ar, struct survey_info *survey,
u32 cc, u32 rcc, u32 cc_prev, u32 rcc_prev)
{
@@ -361,6 +432,195 @@
mutex_unlock(&ar->conf_mutex);
}
+/**
+ * ath10k_hw_qca6174_enable_pll_clock() - enable the qca6174 hw pll clock
+ * @ar: the ath10k blob
+ *
+ * This function is very hardware specific, the clock initialization
+ * steps is very sensitive and could lead to unknown crash, so they
+ * should be done in sequence.
+ *
+ * *** Be aware if you planned to refactor them. ***
+ *
+ * Return: 0 if successfully enable the pll, otherwise EINVAL
+ */
+static int ath10k_hw_qca6174_enable_pll_clock(struct ath10k *ar)
+{
+ int ret, wait_limit;
+ u32 clk_div_addr, pll_init_addr, speed_addr;
+ u32 addr, reg_val, mem_val;
+ struct ath10k_hw_params *hw;
+ const struct ath10k_hw_clk_params *hw_clk;
+
+ hw = &ar->hw_params;
+
+ if (ar->regs->core_clk_div_address == 0 ||
+ ar->regs->cpu_pll_init_address == 0 ||
+ ar->regs->cpu_speed_address == 0)
+ return -EINVAL;
+
+ clk_div_addr = ar->regs->core_clk_div_address;
+ pll_init_addr = ar->regs->cpu_pll_init_address;
+ speed_addr = ar->regs->cpu_speed_address;
+
+ /* Read efuse register to find out the right hw clock configuration */
+ addr = (RTC_SOC_BASE_ADDRESS | EFUSE_OFFSET);
+ ret = ath10k_bmi_read_soc_reg(ar, addr, ®_val);
+ if (ret)
+ return -EINVAL;
+
+ /* sanitize if the hw refclk index is out of the boundary */
+ if (MS(reg_val, EFUSE_XTAL_SEL) > ATH10K_HW_REFCLK_COUNT)
+ return -EINVAL;
+
+ hw_clk = &hw->hw_clk[MS(reg_val, EFUSE_XTAL_SEL)];
+
+ /* Set the rnfrac and outdiv params to bb_pll register */
+ addr = (RTC_SOC_BASE_ADDRESS | BB_PLL_CONFIG_OFFSET);
+ ret = ath10k_bmi_read_soc_reg(ar, addr, ®_val);
+ if (ret)
+ return -EINVAL;
+
+ reg_val &= ~(BB_PLL_CONFIG_FRAC_MASK | BB_PLL_CONFIG_OUTDIV_MASK);
+ reg_val |= (SM(hw_clk->rnfrac, BB_PLL_CONFIG_FRAC) |
+ SM(hw_clk->outdiv, BB_PLL_CONFIG_OUTDIV));
+ ret = ath10k_bmi_write_soc_reg(ar, addr, reg_val);
+ if (ret)
+ return -EINVAL;
+
+ /* Set the correct settle time value to pll_settle register */
+ addr = (RTC_WMAC_BASE_ADDRESS | WLAN_PLL_SETTLE_OFFSET);
+ ret = ath10k_bmi_read_soc_reg(ar, addr, ®_val);
+ if (ret)
+ return -EINVAL;
+
+ reg_val &= ~WLAN_PLL_SETTLE_TIME_MASK;
+ reg_val |= SM(hw_clk->settle_time, WLAN_PLL_SETTLE_TIME);
+ ret = ath10k_bmi_write_soc_reg(ar, addr, reg_val);
+ if (ret)
+ return -EINVAL;
+
+ /* Set the clock_ctrl div to core_clk_ctrl register */
+ addr = (RTC_SOC_BASE_ADDRESS | SOC_CORE_CLK_CTRL_OFFSET);
+ ret = ath10k_bmi_read_soc_reg(ar, addr, ®_val);
+ if (ret)
+ return -EINVAL;
+
+ reg_val &= ~SOC_CORE_CLK_CTRL_DIV_MASK;
+ reg_val |= SM(1, SOC_CORE_CLK_CTRL_DIV);
+ ret = ath10k_bmi_write_soc_reg(ar, addr, reg_val);
+ if (ret)
+ return -EINVAL;
+
+ /* Set the clock_div register */
+ mem_val = 1;
+ ret = ath10k_bmi_write_memory(ar, clk_div_addr, &mem_val,
+ sizeof(mem_val));
+ if (ret)
+ return -EINVAL;
+
+ /* Configure the pll_control register */
+ addr = (RTC_WMAC_BASE_ADDRESS | WLAN_PLL_CONTROL_OFFSET);
+ ret = ath10k_bmi_read_soc_reg(ar, addr, ®_val);
+ if (ret)
+ return -EINVAL;
+
+ reg_val |= (SM(hw_clk->refdiv, WLAN_PLL_CONTROL_REFDIV) |
+ SM(hw_clk->div, WLAN_PLL_CONTROL_DIV) |
+ SM(1, WLAN_PLL_CONTROL_NOPWD));
+ ret = ath10k_bmi_write_soc_reg(ar, addr, reg_val);
+ if (ret)
+ return -EINVAL;
+
+ /* busy wait (max 1s) the rtc_sync status register indicate ready */
+ wait_limit = 100000;
+ addr = (RTC_WMAC_BASE_ADDRESS | RTC_SYNC_STATUS_OFFSET);
+ do {
+ ret = ath10k_bmi_read_soc_reg(ar, addr, ®_val);
+ if (ret)
+ return -EINVAL;
+
+ if (!MS(reg_val, RTC_SYNC_STATUS_PLL_CHANGING))
+ break;
+
+ wait_limit--;
+ udelay(10);
+
+ } while (wait_limit > 0);
+
+ if (MS(reg_val, RTC_SYNC_STATUS_PLL_CHANGING))
+ return -EINVAL;
+
+ /* Unset the pll_bypass in pll_control register */
+ addr = (RTC_WMAC_BASE_ADDRESS | WLAN_PLL_CONTROL_OFFSET);
+ ret = ath10k_bmi_read_soc_reg(ar, addr, ®_val);
+ if (ret)
+ return -EINVAL;
+
+ reg_val &= ~WLAN_PLL_CONTROL_BYPASS_MASK;
+ reg_val |= SM(0, WLAN_PLL_CONTROL_BYPASS);
+ ret = ath10k_bmi_write_soc_reg(ar, addr, reg_val);
+ if (ret)
+ return -EINVAL;
+
+ /* busy wait (max 1s) the rtc_sync status register indicate ready */
+ wait_limit = 100000;
+ addr = (RTC_WMAC_BASE_ADDRESS | RTC_SYNC_STATUS_OFFSET);
+ do {
+ ret = ath10k_bmi_read_soc_reg(ar, addr, ®_val);
+ if (ret)
+ return -EINVAL;
+
+ if (!MS(reg_val, RTC_SYNC_STATUS_PLL_CHANGING))
+ break;
+
+ wait_limit--;
+ udelay(10);
+
+ } while (wait_limit > 0);
+
+ if (MS(reg_val, RTC_SYNC_STATUS_PLL_CHANGING))
+ return -EINVAL;
+
+ /* Enable the hardware cpu clock register */
+ addr = (RTC_SOC_BASE_ADDRESS | SOC_CPU_CLOCK_OFFSET);
+ ret = ath10k_bmi_read_soc_reg(ar, addr, ®_val);
+ if (ret)
+ return -EINVAL;
+
+ reg_val &= ~SOC_CPU_CLOCK_STANDARD_MASK;
+ reg_val |= SM(1, SOC_CPU_CLOCK_STANDARD);
+ ret = ath10k_bmi_write_soc_reg(ar, addr, reg_val);
+ if (ret)
+ return -EINVAL;
+
+ /* unset the nopwd from pll_control register */
+ addr = (RTC_WMAC_BASE_ADDRESS | WLAN_PLL_CONTROL_OFFSET);
+ ret = ath10k_bmi_read_soc_reg(ar, addr, ®_val);
+ if (ret)
+ return -EINVAL;
+
+ reg_val &= ~WLAN_PLL_CONTROL_NOPWD_MASK;
+ ret = ath10k_bmi_write_soc_reg(ar, addr, reg_val);
+ if (ret)
+ return -EINVAL;
+
+ /* enable the pll_init register */
+ mem_val = 1;
+ ret = ath10k_bmi_write_memory(ar, pll_init_addr, &mem_val,
+ sizeof(mem_val));
+ if (ret)
+ return -EINVAL;
+
+ /* set the target clock frequency to speed register */
+ ret = ath10k_bmi_write_memory(ar, speed_addr, &hw->target_cpu_freq,
+ sizeof(hw->target_cpu_freq));
+ if (ret)
+ return -EINVAL;
+
+ return 0;
+}
+
const struct ath10k_hw_ops qca988x_ops = {
.set_coverage_class = ath10k_hw_qca988x_set_coverage_class,
};
@@ -374,3 +634,8 @@
const struct ath10k_hw_ops qca99x0_ops = {
.rx_desc_get_l3_pad_bytes = ath10k_qca99x0_rx_desc_get_l3_pad_bytes,
};
+
+const struct ath10k_hw_ops qca6174_ops = {
+ .set_coverage_class = ath10k_hw_qca988x_set_coverage_class,
+ .enable_pll_clk = ath10k_hw_qca6174_enable_pll_clock,
+};
diff --git a/drivers/net/wireless/ath/ath10k/hw.h b/drivers/net/wireless/ath/ath10k/hw.h
index f0fda0f..5b1e90b 100644
--- a/drivers/net/wireless/ath/ath10k/hw.h
+++ b/drivers/net/wireless/ath/ath10k/hw.h
@@ -129,7 +129,7 @@
#define QCA4019_HW_1_0_PATCH_LOAD_ADDR 0x1234
#define ATH10K_FW_FILE_BASE "firmware"
-#define ATH10K_FW_API_MAX 5
+#define ATH10K_FW_API_MAX 6
#define ATH10K_FW_API_MIN 2
#define ATH10K_FW_API2_FILE "firmware-2.bin"
@@ -141,6 +141,9 @@
/* HTT id conflict fix for management frames over HTT */
#define ATH10K_FW_API5_FILE "firmware-5.bin"
+/* the firmware-6.bin blob */
+#define ATH10K_FW_API6_FILE "firmware-6.bin"
+
#define ATH10K_FW_UTF_FILE "utf.bin"
#define ATH10K_FW_UTF_API2_FILE "utf-2.bin"
@@ -255,6 +258,9 @@
u32 pcie_intr_fw_mask;
u32 pcie_intr_ce_mask_all;
u32 pcie_intr_clr_address;
+ u32 cpu_pll_init_address;
+ u32 cpu_speed_address;
+ u32 core_clk_div_address;
};
extern const struct ath10k_hw_regs qca988x_regs;
@@ -293,7 +299,8 @@
* - raw appears in nwifi decap, raw and nwifi appear in ethernet decap
* - raw have FCS, nwifi doesn't
* - ethernet frames have 802.11 header decapped and parts (base hdr, cipher
- * param, llc/snap) are aligned to 4byte boundaries each */
+ * param, llc/snap) are aligned to 4byte boundaries each
+ */
enum ath10k_hw_txrx_mode {
ATH10K_HW_TXRX_RAW = 0,
@@ -363,6 +370,30 @@
ATH10K_HW_CC_WRAP_SHIFTED_EACH = 2,
};
+enum ath10k_hw_refclk_speed {
+ ATH10K_HW_REFCLK_UNKNOWN = -1,
+ ATH10K_HW_REFCLK_48_MHZ = 0,
+ ATH10K_HW_REFCLK_19_2_MHZ = 1,
+ ATH10K_HW_REFCLK_24_MHZ = 2,
+ ATH10K_HW_REFCLK_26_MHZ = 3,
+ ATH10K_HW_REFCLK_37_4_MHZ = 4,
+ ATH10K_HW_REFCLK_38_4_MHZ = 5,
+ ATH10K_HW_REFCLK_40_MHZ = 6,
+ ATH10K_HW_REFCLK_52_MHZ = 7,
+
+ /* must be the last one */
+ ATH10K_HW_REFCLK_COUNT,
+};
+
+struct ath10k_hw_clk_params {
+ u32 refclk;
+ u32 div;
+ u32 rnfrac;
+ u32 settle_time;
+ u32 refdiv;
+ u32 outdiv;
+};
+
struct ath10k_hw_params {
u32 id;
u16 dev_id;
@@ -416,6 +447,13 @@
/* Number of bytes used for alignment in rx_hdr_status of rx desc. */
int decap_align_bytes;
+
+ /* hw specific clock control parameters */
+ const struct ath10k_hw_clk_params *hw_clk;
+ int target_cpu_freq;
+
+ /* Number of bytes to be discarded for each FFT sample */
+ int spectral_bin_discard;
};
struct htt_rx_desc;
@@ -424,10 +462,14 @@
struct ath10k_hw_ops {
int (*rx_desc_get_l3_pad_bytes)(struct htt_rx_desc *rxd);
void (*set_coverage_class)(struct ath10k *ar, s16 value);
+ int (*enable_pll_clk)(struct ath10k *ar);
};
extern const struct ath10k_hw_ops qca988x_ops;
extern const struct ath10k_hw_ops qca99x0_ops;
+extern const struct ath10k_hw_ops qca6174_ops;
+
+extern const struct ath10k_hw_clk_params qca6174_clk[];
static inline int
ath10k_rx_desc_get_l3_pad_bytes(struct ath10k_hw_params *hw,
@@ -847,4 +889,38 @@
#define WAVE1_PHYCLK_USEC_MASK 0x0000007F
#define WAVE1_PHYCLK_USEC_LSB 0
+/* qca6174 PLL offset/mask */
+#define SOC_CORE_CLK_CTRL_OFFSET 0x00000114
+#define SOC_CORE_CLK_CTRL_DIV_LSB 0
+#define SOC_CORE_CLK_CTRL_DIV_MASK 0x00000007
+
+#define EFUSE_OFFSET 0x0000032c
+#define EFUSE_XTAL_SEL_LSB 8
+#define EFUSE_XTAL_SEL_MASK 0x00000700
+
+#define BB_PLL_CONFIG_OFFSET 0x000002f4
+#define BB_PLL_CONFIG_FRAC_LSB 0
+#define BB_PLL_CONFIG_FRAC_MASK 0x0003ffff
+#define BB_PLL_CONFIG_OUTDIV_LSB 18
+#define BB_PLL_CONFIG_OUTDIV_MASK 0x001c0000
+
+#define WLAN_PLL_SETTLE_OFFSET 0x0018
+#define WLAN_PLL_SETTLE_TIME_LSB 0
+#define WLAN_PLL_SETTLE_TIME_MASK 0x000007ff
+
+#define WLAN_PLL_CONTROL_OFFSET 0x0014
+#define WLAN_PLL_CONTROL_DIV_LSB 0
+#define WLAN_PLL_CONTROL_DIV_MASK 0x000003ff
+#define WLAN_PLL_CONTROL_REFDIV_LSB 10
+#define WLAN_PLL_CONTROL_REFDIV_MASK 0x00003c00
+#define WLAN_PLL_CONTROL_BYPASS_LSB 16
+#define WLAN_PLL_CONTROL_BYPASS_MASK 0x00010000
+#define WLAN_PLL_CONTROL_NOPWD_LSB 18
+#define WLAN_PLL_CONTROL_NOPWD_MASK 0x00040000
+
+#define RTC_SYNC_STATUS_OFFSET 0x0244
+#define RTC_SYNC_STATUS_PLL_CHANGING_LSB 5
+#define RTC_SYNC_STATUS_PLL_CHANGING_MASK 0x00000020
+/* qca6174 PLL offset/mask end */
+
#endif /* _HW_H_ */
diff --git a/drivers/net/wireless/ath/ath10k/mac.c b/drivers/net/wireless/ath/ath10k/mac.c
index 3029f25..4674ff3 100644
--- a/drivers/net/wireless/ath/ath10k/mac.c
+++ b/drivers/net/wireless/ath/ath10k/mac.c
@@ -457,7 +457,8 @@
for (;;) {
/* since ath10k_install_key we can't hold data_lock all the
- * time, so we try to remove the keys incrementally */
+ * time, so we try to remove the keys incrementally
+ */
spin_lock_bh(&ar->data_lock);
i = 0;
list_for_each_entry(peer, &ar->peers, list) {
@@ -609,7 +610,8 @@
case 2:
case 3:
/* Our lower layer calculations limit our precision to
- 1 microsecond */
+ * 1 microsecond
+ */
return 1;
case 4:
return 2;
@@ -978,7 +980,8 @@
arg.channel.band_center_freq2 = chandef->center_freq2;
/* TODO setup this dynamically, what in case we
- don't have any vifs? */
+ * don't have any vifs?
+ */
arg.channel.mode = chan_to_phymode(chandef);
arg.channel.chan_radar =
!!(channel->flags & IEEE80211_CHAN_RADAR);
@@ -2373,9 +2376,10 @@
}
/* TODO setup this based on STA listen interval and
- beacon interval. Currently we don't know
- sta->listen_interval - mac80211 patch required.
- Currently use 10 seconds */
+ * beacon interval. Currently we don't know
+ * sta->listen_interval - mac80211 patch required.
+ * Currently use 10 seconds
+ */
ret = ath10k_wmi_set_ap_ps_param(ar, arvif->vdev_id, sta->addr,
WMI_AP_PS_PEER_PARAM_AGEOUT_TIME,
10);
@@ -2451,6 +2455,8 @@
enum nl80211_band band;
const u16 *vht_mcs_mask;
u8 ampdu_factor;
+ u8 max_nss, vht_mcs;
+ int i;
if (WARN_ON(ath10k_mac_vif_chan(vif, &def)))
return;
@@ -2478,7 +2484,8 @@
/* Workaround: Some Netgear/Linksys 11ac APs set Rx A-MPDU factor to
* zero in VHT IE. Using it would result in degraded throughput.
* arg->peer_max_mpdu at this point contains HT max_mpdu so keep
- * it if VHT max_mpdu is smaller. */
+ * it if VHT max_mpdu is smaller.
+ */
arg->peer_max_mpdu = max(arg->peer_max_mpdu,
(1U << (IEEE80211_HT_MAX_AMPDU_FACTOR +
ampdu_factor)) - 1);
@@ -2489,6 +2496,18 @@
if (sta->bandwidth == IEEE80211_STA_RX_BW_160)
arg->peer_flags |= ar->wmi.peer_flags->bw160;
+ /* Calculate peer NSS capability from VHT capabilities if STA
+ * supports VHT.
+ */
+ for (i = 0, max_nss = 0, vht_mcs = 0; i < NL80211_VHT_NSS_MAX; i++) {
+ vht_mcs = __le16_to_cpu(vht_cap->vht_mcs.rx_mcs_map) >>
+ (2 * i) & 3;
+
+ if ((vht_mcs != IEEE80211_VHT_MCS_NOT_SUPPORTED) &&
+ vht_mcs_mask[i])
+ max_nss = i + 1;
+ }
+ arg->peer_num_spatial_streams = min(sta->rx_nss, max_nss);
arg->peer_vht_rates.rx_max_rate =
__le16_to_cpu(vht_cap->vht_mcs.rx_highest);
arg->peer_vht_rates.rx_mcs_set =
@@ -2779,7 +2798,8 @@
}
/* ap_sta must be accessed only within rcu section which must be left
- * before calling ath10k_setup_peer_smps() which might sleep. */
+ * before calling ath10k_setup_peer_smps() which might sleep.
+ */
ht_cap = ap_sta->ht_cap;
vht_cap = ap_sta->vht_cap;
@@ -3050,7 +3070,8 @@
/* FIXME: why use only legacy modes, why not any
* HT/VHT modes? Would that even make any
- * difference? */
+ * difference?
+ */
if (channel->band == NL80211_BAND_2GHZ)
ch->mode = MODE_11G;
else
@@ -3114,7 +3135,8 @@
}
/* Target allows setting up per-band regdomain but ath_common provides
- * a combined one only */
+ * a combined one only
+ */
ret = ath10k_wmi_pdev_set_regdomain(ar,
regpair->reg_domain,
regpair->reg_domain, /* 2ghz */
@@ -3126,6 +3148,21 @@
ath10k_warn(ar, "failed to set pdev regdomain: %d\n", ret);
}
+static void ath10k_mac_update_channel_list(struct ath10k *ar,
+ struct ieee80211_supported_band *band)
+{
+ int i;
+
+ if (ar->low_5ghz_chan && ar->high_5ghz_chan) {
+ for (i = 0; i < band->n_channels; i++) {
+ if (band->channels[i].center_freq < ar->low_5ghz_chan ||
+ band->channels[i].center_freq > ar->high_5ghz_chan)
+ band->channels[i].flags |=
+ IEEE80211_CHAN_DISABLED;
+ }
+ }
+}
+
static void ath10k_reg_notifier(struct wiphy *wiphy,
struct regulatory_request *request)
{
@@ -3149,6 +3186,10 @@
if (ar->state == ATH10K_STATE_ON)
ath10k_regd_update(ar);
mutex_unlock(&ar->conf_mutex);
+
+ if (ar->phy_capability & WHAL_WLAN_11A_CAPABILITY)
+ ath10k_mac_update_channel_list(ar,
+ ar->hw->wiphy->bands[NL80211_BAND_5GHZ]);
}
/***************/
@@ -3644,7 +3685,8 @@
* never transmitted. We delete the peer upon tx completion.
* It is unlikely that a peer for offchannel tx will already be
* present. However it may be in some rare cases so account for that.
- * Otherwise we might remove a legitimate peer and break stuff. */
+ * Otherwise we might remove a legitimate peer and break stuff.
+ */
for (;;) {
skb = skb_dequeue(&ar->offchan_tx_queue);
@@ -4684,6 +4726,7 @@
}
mutex_unlock(&ar->conf_mutex);
+ cancel_work_sync(&ar->set_coverage_class_work);
cancel_delayed_work_sync(&ar->scan.timeout);
cancel_work_sync(&ar->restart_work);
}
@@ -5683,7 +5726,8 @@
}
/* the peer should not disappear in mid-way (unless FW goes awry) since
- * we already hold conf_mutex. we just make sure its there now. */
+ * we already hold conf_mutex. we just make sure its there now.
+ */
spin_lock_bh(&ar->data_lock);
peer = ath10k_peer_find(ar, arvif->vdev_id, peer_addr);
spin_unlock_bh(&ar->data_lock);
@@ -5695,8 +5739,7 @@
ret = -EOPNOTSUPP;
goto exit;
} else {
- /* if the peer doesn't exist there is no key to disable
- * anymore */
+ /* if the peer doesn't exist there is no key to disable anymore */
goto exit;
}
}
@@ -6555,7 +6598,8 @@
long time_left;
/* mac80211 doesn't care if we really xmit queued frames or not
- * we'll collect those frames either way if we stop/delete vdevs */
+ * we'll collect those frames either way if we stop/delete vdevs
+ */
if (drop)
return;
@@ -6606,7 +6650,8 @@
mutex_lock(&ar->conf_mutex);
/* If device failed to restart it will be in a different state, e.g.
- * ATH10K_STATE_WEDGED */
+ * ATH10K_STATE_WEDGED
+ */
if (ar->state == ATH10K_STATE_RESTARTED) {
ath10k_info(ar, "device successfully recovered\n");
ar->state = ATH10K_STATE_ON;
@@ -7129,7 +7174,7 @@
lockdep_assert_held(&ar->data_lock);
WARN_ON(ctx && vifs);
- WARN_ON(vifs && n_vifs != 1);
+ WARN_ON(vifs && !n_vifs);
/* FIXME: Sort of an optimization and a workaround. Peers and vifs are
* on a linked list now. Doing a lookup peer -> vif -> chanctx for each
@@ -8248,6 +8293,8 @@
ar->hw->wiphy->cipher_suites = cipher_suites;
ar->hw->wiphy->n_cipher_suites = ARRAY_SIZE(cipher_suites);
+ wiphy_ext_feature_set(ar->hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
ret = ieee80211_register_hw(ar->hw);
if (ret) {
ath10k_err(ar, "failed to register ieee80211: %d\n", ret);
diff --git a/drivers/net/wireless/ath/ath10k/pci.c b/drivers/net/wireless/ath/ath10k/pci.c
index 6094372..1e9806f 100644
--- a/drivers/net/wireless/ath/ath10k/pci.c
+++ b/drivers/net/wireless/ath/ath10k/pci.c
@@ -720,14 +720,16 @@
{
/* IMPORTANT: INTR_CLR register has to be set after
* INTR_ENABLE is set to 0, otherwise interrupt can not be
- * really cleared. */
+ * really cleared.
+ */
ath10k_pci_write32(ar, SOC_CORE_BASE_ADDRESS + PCIE_INTR_ENABLE_ADDRESS,
0);
ath10k_pci_write32(ar, SOC_CORE_BASE_ADDRESS + PCIE_INTR_CLR_ADDRESS,
PCIE_INTR_FIRMWARE_MASK | PCIE_INTR_CE_MASK_ALL);
/* IMPORTANT: this extra read transaction is required to
- * flush the posted write buffer. */
+ * flush the posted write buffer.
+ */
(void)ath10k_pci_read32(ar, SOC_CORE_BASE_ADDRESS +
PCIE_INTR_ENABLE_ADDRESS);
}
@@ -739,7 +741,8 @@
PCIE_INTR_FIRMWARE_MASK | PCIE_INTR_CE_MASK_ALL);
/* IMPORTANT: this extra read transaction is required to
- * flush the posted write buffer. */
+ * flush the posted write buffer.
+ */
(void)ath10k_pci_read32(ar, SOC_CORE_BASE_ADDRESS +
PCIE_INTR_ENABLE_ADDRESS);
}
@@ -970,12 +973,6 @@
}
remaining_bytes -= nbytes;
-
- if (ret) {
- ath10k_warn(ar, "failed to read diag value at 0x%x: %d\n",
- address, ret);
- break;
- }
memcpy(data, data_buf, nbytes);
address += nbytes;
@@ -2914,7 +2911,8 @@
* host won't know when target writes BAR to CORE_CTRL.
* This write might get lost if target has NOT written BAR.
* For now, fix the race by repeating the write in below
- * synchronization checking. */
+ * synchronization checking.
+ */
ar_pci->oper_irq_mode = ATH10K_PCI_IRQ_LEGACY;
ath10k_pci_write32(ar, SOC_CORE_BASE_ADDRESS + PCIE_INTR_ENABLE_ADDRESS,
@@ -3430,6 +3428,7 @@
/* QCA6174 3.1 firmware files */
MODULE_FIRMWARE(QCA6174_HW_3_0_FW_DIR "/" ATH10K_FW_API4_FILE);
MODULE_FIRMWARE(QCA6174_HW_3_0_FW_DIR "/" ATH10K_FW_API5_FILE);
+MODULE_FIRMWARE(QCA6174_HW_3_0_FW_DIR "/" ATH10K_FW_API6_FILE);
MODULE_FIRMWARE(QCA6174_HW_3_0_FW_DIR "/" QCA6174_HW_3_0_BOARD_DATA_FILE);
MODULE_FIRMWARE(QCA6174_HW_3_0_FW_DIR "/" ATH10K_BOARD_API2_FILE);
diff --git a/drivers/net/wireless/ath/ath10k/rx_desc.h b/drivers/net/wireless/ath/ath10k/rx_desc.h
index 034e7a5..c1022a1 100644
--- a/drivers/net/wireless/ath/ath10k/rx_desc.h
+++ b/drivers/net/wireless/ath/ath10k/rx_desc.h
@@ -439,19 +439,22 @@
* c) A-MSDU subframe header (14 bytes) if appliable
* d) LLC/SNAP (RFC1042, 8 bytes)
*
- * In case of A-MSDU only first frame in sequence contains (a) and (b). */
+ * In case of A-MSDU only first frame in sequence contains (a) and (b).
+ */
enum rx_msdu_decap_format {
RX_MSDU_DECAP_RAW = 0,
/* Note: QoS frames are reported as non-QoS. The rx_hdr_status in
- * htt_rx_desc contains the original decapped 802.11 header. */
+ * htt_rx_desc contains the original decapped 802.11 header.
+ */
RX_MSDU_DECAP_NATIVE_WIFI = 1,
/* Payload contains an ethernet header (struct ethhdr). */
RX_MSDU_DECAP_ETHERNET2_DIX = 2,
/* Payload contains two 48-bit addresses and 2-byte length (14 bytes
- * total), followed by an RFC1042 header (8 bytes). */
+ * total), followed by an RFC1042 header (8 bytes).
+ */
RX_MSDU_DECAP_8023_SNAP_LLC = 3
};
@@ -867,7 +870,7 @@
*
* reserved_9
* Reserved: HW should fill with 0, FW should ignore.
-*/
+ */
#define RX_PPDU_END_FLAGS_PHY_ERR (1 << 0)
#define RX_PPDU_END_FLAGS_RX_LOCATION (1 << 1)
@@ -1207,7 +1210,7 @@
* Every time HW sets this bit in memory FW/SW must clear this
* bit in memory. FW will initialize all the ppdu_done dword
* to 0.
-*/
+ */
#define FW_RX_DESC_INFO0_DISCARD (1 << 0)
#define FW_RX_DESC_INFO0_FORWARD (1 << 1)
diff --git a/drivers/net/wireless/ath/ath10k/spectral.c b/drivers/net/wireless/ath/ath10k/spectral.c
index c061d69..dd9cc09 100644
--- a/drivers/net/wireless/ath/ath10k/spectral.c
+++ b/drivers/net/wireless/ath/ath10k/spectral.c
@@ -56,6 +56,21 @@
return max_exp;
}
+static inline size_t ath10k_spectral_fix_bin_size(struct ath10k *ar,
+ size_t bin_len)
+{
+ /* some chipsets reports bin size as 2^n bytes + 'm' bytes in
+ * report mode 2. First 2^n bytes carries inband tones and last
+ * 'm' bytes carries band edge detection data mainly used in
+ * radar detection purpose. Strip last 'm' bytes to make bin size
+ * as a valid one. 'm' can take possible values of 4, 12.
+ */
+ if (!is_power_of_2(bin_len))
+ bin_len -= ar->hw_params.spectral_bin_discard;
+
+ return bin_len;
+}
+
int ath10k_spectral_process_fft(struct ath10k *ar,
struct wmi_phyerr_ev_arg *phyerr,
const struct phyerr_fft_report *fftr,
@@ -70,18 +85,11 @@
fft_sample = (struct fft_sample_ath10k *)&buf;
+ bin_len = ath10k_spectral_fix_bin_size(ar, bin_len);
+
if (bin_len < 64 || bin_len > SPECTRAL_ATH10K_MAX_NUM_BINS)
return -EINVAL;
- /* qca99x0 reports bin size as 68 bytes (64 bytes + 4 bytes) in
- * report mode 2. First 64 bytes carries inband tones (-32 to +31)
- * and last 4 byte carries band edge detection data (+32) mainly
- * used in radar detection purpose. Strip last 4 byte to make bin
- * size is valid one.
- */
- if (bin_len == 68)
- bin_len -= 4;
-
reg0 = __le32_to_cpu(fftr->reg0);
reg1 = __le32_to_cpu(fftr->reg1);
@@ -536,15 +544,15 @@
1140, 2500,
&rfs_spec_scan_cb, NULL);
debugfs_create_file("spectral_scan_ctl",
- S_IRUSR | S_IWUSR,
+ 0600,
ar->debug.debugfs_phy, ar,
&fops_spec_scan_ctl);
debugfs_create_file("spectral_count",
- S_IRUSR | S_IWUSR,
+ 0600,
ar->debug.debugfs_phy, ar,
&fops_spectral_count);
debugfs_create_file("spectral_bins",
- S_IRUSR | S_IWUSR,
+ 0600,
ar->debug.debugfs_phy, ar,
&fops_spectral_bins);
diff --git a/drivers/net/wireless/ath/ath10k/targaddrs.h b/drivers/net/wireless/ath/ath10k/targaddrs.h
index a47cab4..cbac9e42 100644
--- a/drivers/net/wireless/ath/ath10k/targaddrs.h
+++ b/drivers/net/wireless/ath/ath10k/targaddrs.h
@@ -268,13 +268,13 @@
#define HI_OPTION_FW_BRIDGE_SHIFT 0x04
/*
-Fw Mode/SubMode Mask
-|-----------------------------------------------------------------------------|
-| SUB | SUB | SUB | SUB | | | | |
-|MODE[3] | MODE[2] | MODE[1] | MODE[0] | MODE[3] | MODE[2] | MODE[1] | MODE[0]|
-| (2) | (2) | (2) | (2) | (2) | (2) | (2) | (2) |
-|-----------------------------------------------------------------------------|
-*/
+ * Fw Mode/SubMode Mask
+ *-----------------------------------------------------------------------------
+ * SUB | SUB | SUB | SUB | | | |
+ *MODE[3] | MODE[2] | MODE[1] | MODE[0] | MODE[3] | MODE[2] | MODE[1] | MODE[0]
+ * (2) | (2) | (2) | (2) | (2) | (2) | (2) | (2)
+ *-----------------------------------------------------------------------------
+ */
#define HI_OPTION_FW_MODE_BITS 0x2
#define HI_OPTION_FW_MODE_MASK 0x3
#define HI_OPTION_FW_MODE_SHIFT 0xC
@@ -428,8 +428,9 @@
#define HI_PWR_SAVE_LPL_ENABLED 0x1
/*b1-b3 reserved*/
/*b4-b5 : dev0 LPL type : 0 - none
- 1- Reduce Pwr Search
- 2- Reduce Pwr Listen*/
+ * 1- Reduce Pwr Search
+ * 2- Reduce Pwr Listen
+ */
/*b6-b7 : dev1 LPL type and so on for Max 8 devices*/
#define HI_PWR_SAVE_LPL_DEV0_LSB 4
#define HI_PWR_SAVE_LPL_DEV_MASK 0x3
diff --git a/drivers/net/wireless/ath/ath10k/testmode.c b/drivers/net/wireless/ath/ath10k/testmode.c
index 8bb36c18..d856462 100644
--- a/drivers/net/wireless/ath/ath10k/testmode.c
+++ b/drivers/net/wireless/ath/ath10k/testmode.c
@@ -420,8 +420,8 @@
struct nlattr *tb[ATH10K_TM_ATTR_MAX + 1];
int ret;
- ret = nla_parse(tb, ATH10K_TM_ATTR_MAX, data, len,
- ath10k_tm_policy);
+ ret = nla_parse(tb, ATH10K_TM_ATTR_MAX, data, len, ath10k_tm_policy,
+ NULL);
if (ret)
return ret;
diff --git a/drivers/net/wireless/ath/ath10k/thermal.c b/drivers/net/wireless/ath/ath10k/thermal.c
index 0a47269..87948af 100644
--- a/drivers/net/wireless/ath/ath10k/thermal.c
+++ b/drivers/net/wireless/ath/ath10k/thermal.c
@@ -124,7 +124,7 @@
complete(&ar->thermal.wmi_sync);
}
-static SENSOR_DEVICE_ATTR(temp1_input, S_IRUGO, ath10k_thermal_show_temp,
+static SENSOR_DEVICE_ATTR(temp1_input, 0444, ath10k_thermal_show_temp,
NULL, 0);
static struct attribute *ath10k_hwmon_attrs[] = {
@@ -191,7 +191,8 @@
return 0;
/* Avoid linking error on devm_hwmon_device_register_with_groups, I
- * guess linux/hwmon.h is missing proper stubs. */
+ * guess linux/hwmon.h is missing proper stubs.
+ */
if (!IS_REACHABLE(CONFIG_HWMON))
return 0;
diff --git a/drivers/net/wireless/ath/ath10k/txrx.c b/drivers/net/wireless/ath/ath10k/txrx.c
index 9852c5d..d4986f6 100644
--- a/drivers/net/wireless/ath/ath10k/txrx.c
+++ b/drivers/net/wireless/ath/ath10k/txrx.c
@@ -34,7 +34,8 @@
/* If the original wait_for_completion() timed out before
* {data,mgmt}_tx_completed() was called then we could complete
* offchan_tx_completed for a different skb. Prevent this by using
- * offchan_tx_skb. */
+ * offchan_tx_skb.
+ */
spin_lock_bh(&ar->data_lock);
if (ar->offchan_tx_skb != skb) {
ath10k_warn(ar, "completed old offchannel frame\n");
diff --git a/drivers/net/wireless/ath/ath10k/wmi-ops.h b/drivers/net/wireless/ath/ath10k/wmi-ops.h
index c7956e1..2fc3f24 100644
--- a/drivers/net/wireless/ath/ath10k/wmi-ops.h
+++ b/drivers/net/wireless/ath/ath10k/wmi-ops.h
@@ -390,7 +390,8 @@
return ret;
/* FIXME There's no ACK event for Management Tx. This probably
- * shouldn't be called here either. */
+ * shouldn't be called here either.
+ */
info->flags |= IEEE80211_TX_STAT_ACK;
ieee80211_tx_status_irqsafe(ar->hw, msdu);
diff --git a/drivers/net/wireless/ath/ath10k/wmi.c b/drivers/net/wireless/ath/ath10k/wmi.c
index 2f1743e..6afc8d2 100644
--- a/drivers/net/wireless/ath/ath10k/wmi.c
+++ b/drivers/net/wireless/ath/ath10k/wmi.c
@@ -3210,7 +3210,8 @@
tim_len = tim_info->tim_len ? __le32_to_cpu(tim_info->tim_len) : 1;
/* if next SWBA has no tim_changed the tim_bitmap is garbage.
- * we must copy the bitmap upon change and reuse it later */
+ * we must copy the bitmap upon change and reuse it later
+ */
if (__le32_to_cpu(tim_info->tim_changed)) {
int i;
@@ -3529,7 +3530,8 @@
* before telling mac80211 to decrement CSA counter
*
* Once CSA counter is completed stop sending beacons until
- * actual channel switch is done */
+ * actual channel switch is done
+ */
if (arvif->vif->csa_active &&
ieee80211_csa_is_complete(arvif->vif)) {
ieee80211_csa_finish(arvif->vif);
@@ -3643,6 +3645,11 @@
spin_lock_bh(&ar->data_lock);
ch = ar->rx_channel;
+
+ /* fetch target operating channel during channel change */
+ if (!ch)
+ ch = ar->tgt_oper_chan;
+
spin_unlock_bh(&ar->data_lock);
if (!ch) {
@@ -3686,7 +3693,8 @@
ATH10K_DFS_STAT_INC(ar, radar_detected);
/* Control radar events reporting in debugfs file
- dfs_block_radar_events */
+ * dfs_block_radar_events
+ */
if (ar->dfs_block_radar_events) {
ath10k_info(ar, "DFS Radar detected, but ignored as requested\n");
return;
@@ -4593,6 +4601,8 @@
arg->phy_capab = ev->phy_capability;
arg->num_rf_chains = ev->num_rf_chains;
arg->eeprom_rd = ev->hal_reg_capabilities.eeprom_rd;
+ arg->low_5ghz_chan = ev->hal_reg_capabilities.low_5ghz_chan;
+ arg->high_5ghz_chan = ev->hal_reg_capabilities.high_5ghz_chan;
arg->num_mem_reqs = ev->num_mem_reqs;
arg->service_map = ev->wmi_service_bitmap;
arg->service_map_len = sizeof(ev->wmi_service_bitmap);
@@ -4629,6 +4639,8 @@
arg->phy_capab = ev->phy_capability;
arg->num_rf_chains = ev->num_rf_chains;
arg->eeprom_rd = ev->hal_reg_capabilities.eeprom_rd;
+ arg->low_5ghz_chan = ev->hal_reg_capabilities.low_5ghz_chan;
+ arg->high_5ghz_chan = ev->hal_reg_capabilities.high_5ghz_chan;
arg->num_mem_reqs = ev->num_mem_reqs;
arg->service_map = ev->wmi_service_bitmap;
arg->service_map_len = sizeof(ev->wmi_service_bitmap);
@@ -4682,6 +4694,8 @@
ar->phy_capability = __le32_to_cpu(arg.phy_capab);
ar->num_rf_chains = __le32_to_cpu(arg.num_rf_chains);
ar->hw_eeprom_rd = __le32_to_cpu(arg.eeprom_rd);
+ ar->low_5ghz_chan = __le32_to_cpu(arg.low_5ghz_chan);
+ ar->high_5ghz_chan = __le32_to_cpu(arg.high_5ghz_chan);
ath10k_dbg_dump(ar, ATH10K_DBG_WMI, NULL, "wmi svc: ",
arg.service_map, arg.service_map_len);
@@ -4758,9 +4772,10 @@
num_units = ar->max_num_peers + 1;
} else if (num_unit_info & NUM_UNITS_IS_NUM_PEERS) {
/* number of units to allocate is number of
- * peers, 1 extra for self peer on target */
- /* this needs to be tied, host and target
- * can get out of sync */
+ * peers, 1 extra for self peer on target
+ * this needs to be tied, host and target
+ * can get out of sync
+ */
num_units = ar->max_num_peers + 1;
} else if (num_unit_info & NUM_UNITS_IS_NUM_VDEVS) {
num_units = ar->max_num_vdevs + 1;
diff --git a/drivers/net/wireless/ath/ath10k/wmi.h b/drivers/net/wireless/ath/ath10k/wmi.h
index 386aa51..1b4865a 100644
--- a/drivers/net/wireless/ath/ath10k/wmi.h
+++ b/drivers/net/wireless/ath/ath10k/wmi.h
@@ -1038,7 +1038,8 @@
WMI_STA_UAPSD_AUTO_TRIG_CMDID,
/* STA Keep alive parameter configuration,
- Requires WMI_SERVICE_STA_KEEP_ALIVE */
+ * Requires WMI_SERVICE_STA_KEEP_ALIVE
+ */
WMI_STA_KEEPALIVE_CMD,
/* misc command group */
@@ -1774,7 +1775,8 @@
break;
/* no default handler to allow compiler to check that the
- * enum is fully handled */
+ * enum is fully handled
+ */
};
return "<unknown>";
@@ -2974,7 +2976,8 @@
/* When set, DFS channels will not be scanned */
#define WMI_SCAN_BYPASS_DFS_CHN 0x40
/* Different FW scan engine may choose to bail out on errors.
- * Allow the driver to have influence over that. */
+ * Allow the driver to have influence over that.
+ */
#define WMI_SCAN_CONTINUE_ON_ERROR 0x80
/* WMI_SCAN_CLASS_MASK must be the same value as IEEE80211_SCAN_CLASS_MASK */
@@ -3182,7 +3185,7 @@
struct phyerr_radar_report {
__le32 reg0; /* RADAR_REPORT_REG0_* */
- __le32 reg1; /* REDAR_REPORT_REG1_* */
+ __le32 reg1; /* RADAR_REPORT_REG1_* */
} __packed;
#define RADAR_REPORT_REG0_PULSE_IS_CHIRP_MASK 0x80000000
@@ -4447,14 +4450,16 @@
/* values for vdev_start_request flags */
/*
* Indicates that AP VDEV uses hidden ssid. only valid for
- * AP/GO */
+ * AP/GO
+ */
#define WMI_VDEV_START_HIDDEN_SSID (1 << 0)
/*
* Indicates if robust management frame/management frame
* protection is enabled. For GO/AP vdevs, it indicates that
* it may support station/client associations with RMF enabled.
* For STA/client vdevs, it indicates that sta will
- * associate with AP with RMF enabled. */
+ * associate with AP with RMF enabled.
+ */
#define WMI_VDEV_START_PMF_ENABLED (1 << 1)
struct wmi_p2p_noa_descriptor {
@@ -4814,7 +4819,8 @@
* An associated STA is considered unresponsive if there is no recent
* TX/RX activity and downlink frames are buffered for it. Once a STA
* exceeds the maximum unresponsive time, the AP will send a
- * WMI_STA_KICKOUT event to the host so the STA can be deleted. */
+ * WMI_STA_KICKOUT event to the host so the STA can be deleted.
+ */
WMI_VDEV_PARAM_AP_KEEPALIVE_MAX_UNRESPONSIVE_TIME_SECS,
/* Enable NAWDS : MCAST INSPECT Enable, NAWDS Flag set */
@@ -4941,7 +4947,8 @@
* An associated STA is considered unresponsive if there is no recent
* TX/RX activity and downlink frames are buffered for it. Once a STA
* exceeds the maximum unresponsive time, the AP will send a
- * WMI_10X_STA_KICKOUT event to the host so the STA can be deleted. */
+ * WMI_10X_STA_KICKOUT event to the host so the STA can be deleted.
+ */
WMI_10X_VDEV_PARAM_AP_KEEPALIVE_MAX_UNRESPONSIVE_TIME_SECS,
/* Enable NAWDS : MCAST INSPECT Enable, NAWDS Flag set */
@@ -5605,12 +5612,14 @@
struct wmi_p2p_noa_info {
/* Bit 0 - Flag to indicate an update in NOA schedule
- Bits 7-1 - Reserved */
+ * Bits 7-1 - Reserved
+ */
u8 changed;
/* NOA index */
u8 index;
/* Bit 0 - Opp PS state of the AP
- Bits 1-7 - Ctwindow in TUs */
+ * Bits 1-7 - Ctwindow in TUs
+ */
u8 ctwindow_oppps;
/* Number of NOA descriptors */
u8 num_descriptors;
@@ -6000,7 +6009,8 @@
struct wmi_common_peer_assoc_complete_cmd cmd;
/* HT Operation Element of the peer. Five bytes packed in 2
- * INT32 array and filled from lsb to msb. */
+ * INT32 array and filled from lsb to msb.
+ */
__le32 peer_ht_info[2];
} __packed;
@@ -6335,6 +6345,8 @@
__le32 num_rf_chains;
__le32 eeprom_rd;
__le32 num_mem_reqs;
+ __le32 low_5ghz_chan;
+ __le32 high_5ghz_chan;
const __le32 *service_map;
size_t service_map_len;
const struct wlan_host_mem_req *mem_reqs[WMI_MAX_MEM_REQS];
diff --git a/drivers/net/wireless/ath/ath5k/base.c b/drivers/net/wireless/ath/ath5k/base.c
index d98fd42..527afcf 100644
--- a/drivers/net/wireless/ath/ath5k/base.c
+++ b/drivers/net/wireless/ath/ath5k/base.c
@@ -1414,10 +1414,10 @@
rxs->flag |= ath5k_rx_decrypted(ah, skb, rs);
switch (ah->ah_bwmode) {
case AR5K_BWMODE_5MHZ:
- rxs->flag |= RX_FLAG_5MHZ;
+ rxs->bw = RATE_INFO_BW_5;
break;
case AR5K_BWMODE_10MHZ:
- rxs->flag |= RX_FLAG_10MHZ;
+ rxs->bw = RATE_INFO_BW_10;
break;
default:
break;
@@ -1425,7 +1425,7 @@
if (rs->rs_rate ==
ah->sbands[ah->curchan->band].bitrates[rxs->rate_idx].hw_value_short)
- rxs->flag |= RX_FLAG_SHORTPRE;
+ rxs->enc_flags |= RX_ENC_FLAG_SHORTPRE;
trace_ath5k_rx(ah, skb);
@@ -2564,6 +2564,8 @@
hw->extra_tx_headroom = 2;
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
/*
* Mark the device as detached to avoid processing
* interrupts until setup is complete.
diff --git a/drivers/net/wireless/ath/ath6kl/cfg80211.c b/drivers/net/wireless/ath/ath6kl/cfg80211.c
index 363b30a..414b5b5 100644
--- a/drivers/net/wireless/ath/ath6kl/cfg80211.c
+++ b/drivers/net/wireless/ath/ath6kl/cfg80211.c
@@ -102,10 +102,8 @@
};
static struct ieee80211_channel ath6kl_5ghz_a_channels[] = {
- CHAN5G(34, 0), CHAN5G(36, 0),
- CHAN5G(38, 0), CHAN5G(40, 0),
- CHAN5G(42, 0), CHAN5G(44, 0),
- CHAN5G(46, 0), CHAN5G(48, 0),
+ CHAN5G(36, 0), CHAN5G(40, 0),
+ CHAN5G(44, 0), CHAN5G(48, 0),
CHAN5G(52, 0), CHAN5G(56, 0),
CHAN5G(60, 0), CHAN5G(64, 0),
CHAN5G(100, 0), CHAN5G(104, 0),
@@ -171,7 +169,7 @@
if (!stopped)
return;
- cfg80211_sched_scan_stopped(ar->wiphy);
+ cfg80211_sched_scan_stopped(ar->wiphy, 0);
}
static int ath6kl_set_wpa_version(struct ath6kl_vif *vif,
@@ -808,9 +806,15 @@
WLAN_STATUS_SUCCESS, GFP_KERNEL);
cfg80211_put_bss(ar->wiphy, bss);
} else if (vif->sme_state == SME_CONNECTED) {
+ struct cfg80211_roam_info roam_info = {
+ .bss = bss,
+ .req_ie = assoc_req_ie,
+ .req_ie_len = assoc_req_len,
+ .resp_ie = assoc_resp_ie,
+ .resp_ie_len = assoc_resp_len,
+ };
/* inform roam event to cfg80211 */
- cfg80211_roamed_bss(vif->ndev, bss, assoc_req_ie, assoc_req_len,
- assoc_resp_ie, assoc_resp_len, GFP_KERNEL);
+ cfg80211_roamed(vif->ndev, &roam_info, GFP_KERNEL);
}
}
@@ -1505,7 +1509,6 @@
const char *name,
unsigned char name_assign_type,
enum nl80211_iftype type,
- u32 *flags,
struct vif_params *params)
{
struct ath6kl *ar = wiphy_priv(wiphy);
@@ -1552,7 +1555,7 @@
static int ath6kl_cfg80211_change_iface(struct wiphy *wiphy,
struct net_device *ndev,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct ath6kl_vif *vif = netdev_priv(ndev);
@@ -3355,7 +3358,7 @@
}
static int ath6kl_cfg80211_sscan_stop(struct wiphy *wiphy,
- struct net_device *dev)
+ struct net_device *dev, u64 reqid)
{
struct ath6kl_vif *vif = netdev_priv(dev);
bool stopped;
@@ -3976,7 +3979,7 @@
WIPHY_FLAG_AP_PROBE_RESP_OFFLOAD;
if (test_bit(ATH6KL_FW_CAPABILITY_SCHED_SCAN_V2, ar->fw_capabilities))
- ar->wiphy->flags |= WIPHY_FLAG_SUPPORTS_SCHED_SCAN;
+ ar->wiphy->max_sched_scan_reqs = 1;
if (test_bit(ATH6KL_FW_CAPABILITY_INACTIVITY_TIMEOUT,
ar->fw_capabilities))
diff --git a/drivers/net/wireless/ath/ath6kl/debug.h b/drivers/net/wireless/ath/ath6kl/debug.h
index 0614393..9429757 100644
--- a/drivers/net/wireless/ath/ath6kl/debug.h
+++ b/drivers/net/wireless/ath/ath6kl/debug.h
@@ -63,6 +63,7 @@
#ifdef CONFIG_ATH6KL_DEBUG
+__printf(2, 3)
void ath6kl_dbg(enum ATH6K_DEBUG_MASK mask, const char *fmt, ...);
void ath6kl_dbg_dump(enum ATH6K_DEBUG_MASK mask,
const char *msg, const char *prefix,
@@ -83,6 +84,7 @@
void ath6kl_debug_cleanup(struct ath6kl *ar);
#else
+__printf(2, 3)
static inline void ath6kl_dbg(enum ATH6K_DEBUG_MASK dbg_mask,
const char *fmt, ...)
{
diff --git a/drivers/net/wireless/ath/ath6kl/htc_pipe.c b/drivers/net/wireless/ath/ath6kl/htc_pipe.c
index ca1a18c..d127a08 100644
--- a/drivers/net/wireless/ath/ath6kl/htc_pipe.c
+++ b/drivers/net/wireless/ath/ath6kl/htc_pipe.c
@@ -995,7 +995,7 @@
if (netlen < (payload_len + HTC_HDR_LENGTH)) {
ath6kl_dbg(ATH6KL_DBG_HTC,
- "HTC Rx: insufficient length, got:%d expected =%u\n",
+ "HTC Rx: insufficient length, got:%d expected =%zu\n",
netlen, payload_len + HTC_HDR_LENGTH);
status = -EINVAL;
goto free_skb;
diff --git a/drivers/net/wireless/ath/ath6kl/testmode.c b/drivers/net/wireless/ath/ath6kl/testmode.c
index d67170e..d8dcacd 100644
--- a/drivers/net/wireless/ath/ath6kl/testmode.c
+++ b/drivers/net/wireless/ath/ath6kl/testmode.c
@@ -74,8 +74,8 @@
int err, buf_len;
void *buf;
- err = nla_parse(tb, ATH6KL_TM_ATTR_MAX, data, len,
- ath6kl_tm_policy);
+ err = nla_parse(tb, ATH6KL_TM_ATTR_MAX, data, len, ath6kl_tm_policy,
+ NULL);
if (err)
return err;
diff --git a/drivers/net/wireless/ath/ath6kl/wmi.c b/drivers/net/wireless/ath/ath6kl/wmi.c
index 84a6d12..bfc20b4 100644
--- a/drivers/net/wireless/ath/ath6kl/wmi.c
+++ b/drivers/net/wireless/ath/ath6kl/wmi.c
@@ -1082,7 +1082,7 @@
{
struct ath6kl_vif *vif = (struct ath6kl_vif *) ptr;
- cfg80211_sched_scan_results(vif->ar->wiphy);
+ cfg80211_sched_scan_results(vif->ar->wiphy, 0);
}
static int ath6kl_wmi_bssinfo_event_rx(struct wmi *wmi, u8 *datap, int len,
@@ -1596,7 +1596,7 @@
rate = le32_to_cpu(ev->rate);
pkts = le32_to_cpu(ev->pkts);
- ath6kl_dbg(ATH6KL_DBG_WMI, "TXE notify event: peer %pM rate %d% pkts %d intvl %ds\n",
+ ath6kl_dbg(ATH6KL_DBG_WMI, "TXE notify event: peer %pM rate %d%% pkts %d intvl %ds\n",
vif->bssid, rate, pkts, vif->txe_intvl);
cfg80211_cqm_txe_notify(vif->ndev, vif->bssid, pkts,
diff --git a/drivers/net/wireless/ath/ath9k/ar9003_mac.c b/drivers/net/wireless/ath/ath9k/ar9003_mac.c
index cc5bb0a..68fcbe0 100644
--- a/drivers/net/wireless/ath/ath9k/ar9003_mac.c
+++ b/drivers/net/wireless/ath/ath9k/ar9003_mac.c
@@ -494,7 +494,8 @@
rxs->rs_status = 0;
rxs->rs_flags = 0;
- rxs->flag = 0;
+ rxs->enc_flags = 0;
+ rxs->bw = RATE_INFO_BW_20;
rxs->rs_datalen = rxsp->status2 & AR_DataLen;
rxs->rs_tstamp = rxsp->status3;
@@ -520,8 +521,8 @@
rxs->rs_isaggr = (rxsp->status11 & AR_RxAggr) ? 1 : 0;
rxs->rs_moreaggr = (rxsp->status11 & AR_RxMoreAggr) ? 1 : 0;
rxs->rs_antenna = (MS(rxsp->status4, AR_RxAntenna) & 0x7);
- rxs->flag |= (rxsp->status4 & AR_GI) ? RX_FLAG_SHORT_GI : 0;
- rxs->flag |= (rxsp->status4 & AR_2040) ? RX_FLAG_40MHZ : 0;
+ rxs->enc_flags |= (rxsp->status4 & AR_GI) ? RX_ENC_FLAG_SHORT_GI : 0;
+ rxs->enc_flags |= (rxsp->status4 & AR_2040) ? RX_ENC_FLAG_40MHZ : 0;
rxs->evm0 = rxsp->status6;
rxs->evm1 = rxsp->status7;
diff --git a/drivers/net/wireless/ath/ath9k/calib.c b/drivers/net/wireless/ath/ath9k/calib.c
index 0f71146..13ab6bc 100644
--- a/drivers/net/wireless/ath/ath9k/calib.c
+++ b/drivers/net/wireless/ath/ath9k/calib.c
@@ -254,7 +254,9 @@
if ((i >= AR5416_MAX_CHAINS) && !IS_CHAN_HT40(chan))
continue;
- if (h)
+ if (ah->nf_override)
+ nfval = ah->nf_override;
+ else if (h)
nfval = h[i].privNF;
else
nfval = default_nf;
@@ -348,6 +350,7 @@
return 0;
}
+EXPORT_SYMBOL(ath9k_hw_loadnf);
static void ath9k_hw_nf_sanitize(struct ath_hw *ah, s16 *nf)
diff --git a/drivers/net/wireless/ath/ath9k/common-spectral.c b/drivers/net/wireless/ath/ath9k/common-spectral.c
index 0ffa23a..5e77fe1 100644
--- a/drivers/net/wireless/ath/ath9k/common-spectral.c
+++ b/drivers/net/wireless/ath/ath9k/common-spectral.c
@@ -742,6 +742,9 @@
return;
}
+ if (!spec_priv->spec_config.enabled)
+ return;
+
ath_ps_ops(common)->wakeup(common);
rxfilter = ath9k_hw_getrxfilter(ah);
ath9k_hw_setrxfilter(ah, rxfilter |
diff --git a/drivers/net/wireless/ath/ath9k/common.c b/drivers/net/wireless/ath/ath9k/common.c
index b80e08b..c67d0e0 100644
--- a/drivers/net/wireless/ath/ath9k/common.c
+++ b/drivers/net/wireless/ath/ath9k/common.c
@@ -181,14 +181,15 @@
sband = hw->wiphy->bands[band];
if (IS_CHAN_QUARTER_RATE(ah->curchan))
- rxs->flag |= RX_FLAG_5MHZ;
+ rxs->bw = RATE_INFO_BW_5;
else if (IS_CHAN_HALF_RATE(ah->curchan))
- rxs->flag |= RX_FLAG_10MHZ;
+ rxs->bw = RATE_INFO_BW_10;
if (rx_stats->rs_rate & 0x80) {
/* HT rate */
- rxs->flag |= RX_FLAG_HT;
- rxs->flag |= rx_stats->flag;
+ rxs->encoding = RX_ENC_HT;
+ rxs->enc_flags |= rx_stats->enc_flags;
+ rxs->bw = rx_stats->bw;
rxs->rate_idx = rx_stats->rs_rate & 0x7f;
return 0;
}
@@ -199,7 +200,7 @@
return 0;
}
if (sband->bitrates[i].hw_value_short == rx_stats->rs_rate) {
- rxs->flag |= RX_FLAG_SHORTPRE;
+ rxs->enc_flags |= RX_ENC_FLAG_SHORTPRE;
rxs->rate_idx = i;
return 0;
}
diff --git a/drivers/net/wireless/ath/ath9k/debug.c b/drivers/net/wireless/ath/ath9k/debug.c
index 43930c3..2e64977 100644
--- a/drivers/net/wireless/ath/ath9k/debug.c
+++ b/drivers/net/wireless/ath/ath9k/debug.c
@@ -1191,6 +1191,65 @@
.llseek = default_llseek,
};
+static ssize_t read_file_nf_override(struct file *file,
+ char __user *user_buf,
+ size_t count, loff_t *ppos)
+{
+ struct ath_softc *sc = file->private_data;
+ struct ath_hw *ah = sc->sc_ah;
+ char buf[32];
+ unsigned int len;
+
+ if (ah->nf_override == 0)
+ len = sprintf(buf, "off\n");
+ else
+ len = sprintf(buf, "%d\n", ah->nf_override);
+
+ return simple_read_from_buffer(user_buf, count, ppos, buf, len);
+}
+
+static ssize_t write_file_nf_override(struct file *file,
+ const char __user *user_buf,
+ size_t count, loff_t *ppos)
+{
+ struct ath_softc *sc = file->private_data;
+ struct ath_hw *ah = sc->sc_ah;
+ long val;
+ char buf[32];
+ ssize_t len;
+
+ len = min(count, sizeof(buf) - 1);
+ if (copy_from_user(buf, user_buf, len))
+ return -EFAULT;
+
+ buf[len] = '\0';
+ if (strncmp("off", buf, 3) == 0)
+ val = 0;
+ else if (kstrtol(buf, 0, &val))
+ return -EINVAL;
+
+ if (val > 0)
+ return -EINVAL;
+
+ if (val < -120)
+ return -EINVAL;
+
+ ah->nf_override = val;
+
+ if (ah->curchan)
+ ath9k_hw_loadnf(ah, ah->curchan);
+
+ return count;
+}
+
+static const struct file_operations fops_nf_override = {
+ .read = read_file_nf_override,
+ .write = write_file_nf_override,
+ .open = simple_open,
+ .owner = THIS_MODULE,
+ .llseek = default_llseek,
+};
+
/* Ethtool support for get-stats */
#define AMKSTR(nm) #nm "_BE", #nm "_BK", #nm "_VI", #nm "_VO"
@@ -1402,5 +1461,8 @@
debugfs_create_u16("airtime_flags", S_IRUSR | S_IWUSR,
sc->debug.debugfs_phy, &sc->airtime_flags);
+ debugfs_create_file("nf_override", S_IRUSR | S_IWUSR,
+ sc->debug.debugfs_phy, sc, &fops_nf_override);
+
return 0;
}
diff --git a/drivers/net/wireless/ath/ath9k/debug_sta.c b/drivers/net/wireless/ath/ath9k/debug_sta.c
index 524cbf13..efc692e 100644
--- a/drivers/net/wireless/ath/ath9k/debug_sta.c
+++ b/drivers/net/wireless/ath/ath9k/debug_sta.c
@@ -116,12 +116,12 @@
if (rxs->rate_idx >= ARRAY_SIZE(rstats->ht_stats))
goto exit;
- if (rxs->flag & RX_FLAG_40MHZ)
+ if ((rxs->bw == RATE_INFO_BW_40))
rstats->ht_stats[rxs->rate_idx].ht40_cnt++;
else
rstats->ht_stats[rxs->rate_idx].ht20_cnt++;
- if (rxs->flag & RX_FLAG_SHORT_GI)
+ if (rxs->enc_flags & RX_ENC_FLAG_SHORT_GI)
rstats->ht_stats[rxs->rate_idx].sgi_cnt++;
else
rstats->ht_stats[rxs->rate_idx].lgi_cnt++;
@@ -130,7 +130,7 @@
}
if (IS_CCK_RATE(rs->rs_rate)) {
- if (rxs->flag & RX_FLAG_SHORTPRE)
+ if (rxs->enc_flags & RX_ENC_FLAG_SHORTPRE)
rstats->cck_stats[rxs->rate_idx].cck_sp_cnt++;
else
rstats->cck_stats[rxs->rate_idx].cck_lp_cnt++;
diff --git a/drivers/net/wireless/ath/ath9k/eeprom.c b/drivers/net/wireless/ath/ath9k/eeprom.c
index fb80ec8..6ccf248 100644
--- a/drivers/net/wireless/ath/ath9k/eeprom.c
+++ b/drivers/net/wireless/ath/ath9k/eeprom.c
@@ -112,7 +112,7 @@
static bool ath9k_hw_nvram_read_array(u16 *blob, size_t blob_size,
off_t offset, u16 *data)
{
- if (offset > blob_size)
+ if (offset >= blob_size)
return false;
*data = blob[offset];
diff --git a/drivers/net/wireless/ath/ath9k/eeprom.h b/drivers/net/wireless/ath/ath9k/eeprom.h
index 30bf722..31390af 100644
--- a/drivers/net/wireless/ath/ath9k/eeprom.h
+++ b/drivers/net/wireless/ath/ath9k/eeprom.h
@@ -106,7 +106,7 @@
#define AR9285_RDEXT_DEFAULT 0x1F
#define ATH9K_POW_SM(_r, _s) (((_r) & 0x3f) << (_s))
-#define FREQ2FBIN(x, y) ((y) ? ((x) - 2300) : (((x) - 4800) / 5))
+#define FREQ2FBIN(x, y) (u8)((y) ? ((x) - 2300) : (((x) - 4800) / 5))
#define FBIN2FREQ(x, y) ((y) ? (2300 + x) : (4800 + 5 * x))
#define ath9k_hw_use_flash(_ah) (!(_ah->ah_flags & AH_USE_EEPROM))
diff --git a/drivers/net/wireless/ath/ath9k/hif_usb.c b/drivers/net/wireless/ath/ath9k/hif_usb.c
index de2d212..12aa8ab 100644
--- a/drivers/net/wireless/ath/ath9k/hif_usb.c
+++ b/drivers/net/wireless/ath/ath9k/hif_usb.c
@@ -37,6 +37,7 @@
{ USB_DEVICE(0x0cf3, 0xb002) }, /* Ubiquiti WifiStation */
{ USB_DEVICE(0x057c, 0x8403) }, /* AVM FRITZ!WLAN 11N v2 USB */
{ USB_DEVICE(0x0471, 0x209e) }, /* Philips (or NXP) PTA01 */
+ { USB_DEVICE(0x1eda, 0x2315) }, /* AirTies */
{ USB_DEVICE(0x0cf3, 0x7015),
.driver_info = AR9287_USB }, /* Atheros */
@@ -1219,6 +1220,9 @@
u8 bulk_out_ep;
int r;
+ if (iface_desc->desc.bNumEndpoints < 2)
+ return -ENODEV;
+
/* Find bulk out endpoint */
for (r = 1; r >= 0; r--) {
endpoint = &iface_desc->endpoint[r].desc;
diff --git a/drivers/net/wireless/ath/ath9k/htc_drv_init.c b/drivers/net/wireless/ath/ath9k/htc_drv_init.c
index b65c1b6..defacc6 100644
--- a/drivers/net/wireless/ath/ath9k/htc_drv_init.c
+++ b/drivers/net/wireless/ath/ath9k/htc_drv_init.c
@@ -780,6 +780,8 @@
}
SET_IEEE80211_PERM_ADDR(hw, common->macaddr);
+
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
}
static int ath9k_init_firmware_version(struct ath9k_htc_priv *priv)
diff --git a/drivers/net/wireless/ath/ath9k/htc_drv_txrx.c b/drivers/net/wireless/ath/ath9k/htc_drv_txrx.c
index f333ef1..b38a586 100644
--- a/drivers/net/wireless/ath/ath9k/htc_drv_txrx.c
+++ b/drivers/net/wireless/ath/ath9k/htc_drv_txrx.c
@@ -929,11 +929,12 @@
static inline void convert_htc_flag(struct ath_rx_status *rx_stats,
struct ath_htc_rx_status *rxstatus)
{
- rx_stats->flag = 0;
+ rx_stats->enc_flags = 0;
+ rx_stats->bw = RATE_INFO_BW_20;
if (rxstatus->rs_flags & ATH9K_RX_2040)
- rx_stats->flag |= RX_FLAG_40MHZ;
+ rx_stats->bw = RATE_INFO_BW_40;
if (rxstatus->rs_flags & ATH9K_RX_GI)
- rx_stats->flag |= RX_FLAG_SHORT_GI;
+ rx_stats->enc_flags |= RX_ENC_FLAG_SHORT_GI;
}
static void rx_status_htc_to_ath(struct ath_rx_status *rx_stats,
diff --git a/drivers/net/wireless/ath/ath9k/hw.h b/drivers/net/wireless/ath/ath9k/hw.h
index 9cbca12..4ac7082 100644
--- a/drivers/net/wireless/ath/ath9k/hw.h
+++ b/drivers/net/wireless/ath/ath9k/hw.h
@@ -803,6 +803,7 @@
u32 rfkill_gpio;
u32 rfkill_polarity;
u32 ah_flags;
+ s16 nf_override;
bool reset_power_on;
bool htc_reset_init;
diff --git a/drivers/net/wireless/ath/ath9k/init.c b/drivers/net/wireless/ath/ath9k/init.c
index fa4b3cc..fd9a618 100644
--- a/drivers/net/wireless/ath/ath9k/init.c
+++ b/drivers/net/wireless/ath/ath9k/init.c
@@ -955,6 +955,8 @@
ath9k_cmn_reload_chainmask(ah);
SET_IEEE80211_PERM_ADDR(hw, common->macaddr);
+
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
}
int ath9k_init_device(u16 devid, struct ath_softc *sc,
diff --git a/drivers/net/wireless/ath/ath9k/mac.c b/drivers/net/wireless/ath/ath9k/mac.c
index d937c39..6128c2b 100644
--- a/drivers/net/wireless/ath/ath9k/mac.c
+++ b/drivers/net/wireless/ath/ath9k/mac.c
@@ -535,7 +535,8 @@
rs->rs_status = 0;
rs->rs_flags = 0;
- rs->flag = 0;
+ rs->enc_flags = 0;
+ rs->bw = RATE_INFO_BW_20;
rs->rs_datalen = ads.ds_rxstatus1 & AR_DataLen;
rs->rs_tstamp = ads.AR_RcvTimestamp;
@@ -577,15 +578,15 @@
rs->rs_antenna = MS(ads.ds_rxstatus3, AR_RxAntenna);
/* directly mapped flags for ieee80211_rx_status */
- rs->flag |=
- (ads.ds_rxstatus3 & AR_GI) ? RX_FLAG_SHORT_GI : 0;
- rs->flag |=
- (ads.ds_rxstatus3 & AR_2040) ? RX_FLAG_40MHZ : 0;
+ rs->enc_flags |=
+ (ads.ds_rxstatus3 & AR_GI) ? RX_ENC_FLAG_SHORT_GI : 0;
+ rs->enc_flags |=
+ (ads.ds_rxstatus3 & AR_2040) ? RX_ENC_FLAG_40MHZ : 0;
if (AR_SREV_9280_20_OR_LATER(ah))
- rs->flag |=
+ rs->enc_flags |=
(ads.ds_rxstatus3 & AR_STBC) ?
/* we can only Nss=1 STBC */
- (1 << RX_FLAG_STBC_SHIFT) : 0;
+ (1 << RX_ENC_FLAG_STBC_SHIFT) : 0;
if (ads.ds_rxstatus8 & AR_PreDelimCRCErr)
rs->rs_flags |= ATH9K_RX_DELIM_CRC_PRE;
diff --git a/drivers/net/wireless/ath/ath9k/mac.h b/drivers/net/wireless/ath/ath9k/mac.h
index 770fc11..fd6aa49 100644
--- a/drivers/net/wireless/ath/ath9k/mac.h
+++ b/drivers/net/wireless/ath/ath9k/mac.h
@@ -16,6 +16,7 @@
#ifndef MAC_H
#define MAC_H
+#include <net/cfg80211.h>
#define set11nTries(_series, _index) \
(SM((_series)[_index].Tries, AR_XmitDataTries##_index))
@@ -143,7 +144,8 @@
u32 evm2;
u32 evm3;
u32 evm4;
- u32 flag; /* see enum mac80211_rx_flags */
+ u16 enc_flags;
+ enum rate_info_bw bw;
};
struct ath_htc_rx_status {
diff --git a/drivers/net/wireless/ath/ath9k/pci.c b/drivers/net/wireless/ath/ath9k/pci.c
index aff473d..7b7627f 100644
--- a/drivers/net/wireless/ath/ath9k/pci.c
+++ b/drivers/net/wireless/ath/ath9k/pci.c
@@ -383,6 +383,11 @@
0x10CF, /* Fujitsu */
0x1783),
.driver_data = ATH9K_PCI_WOW },
+ { PCI_DEVICE_SUB(PCI_VENDOR_ID_ATHEROS,
+ 0x0034,
+ PCI_VENDOR_ID_DELL,
+ 0x020B),
+ .driver_data = ATH9K_PCI_WOW },
/* Killer Wireless (2x2) */
{ PCI_DEVICE_SUB(PCI_VENDOR_ID_ATHEROS,
diff --git a/drivers/net/wireless/ath/ath9k/recv.c b/drivers/net/wireless/ath/ath9k/recv.c
index d79837f..2197aee 100644
--- a/drivers/net/wireless/ath/ath9k/recv.c
+++ b/drivers/net/wireless/ath/ath9k/recv.c
@@ -1037,11 +1037,11 @@
rxs = IEEE80211_SKB_RXCB(skb);
- is_sgi = !!(rxs->flag & RX_FLAG_SHORT_GI);
- is_40 = !!(rxs->flag & RX_FLAG_40MHZ);
- is_sp = !!(rxs->flag & RX_FLAG_SHORTPRE);
+ is_sgi = !!(rxs->enc_flags & RX_ENC_FLAG_SHORT_GI);
+ is_40 = !!(rxs->bw == RATE_INFO_BW_40);
+ is_sp = !!(rxs->enc_flags & RX_ENC_FLAG_SHORTPRE);
- if (!!(rxs->flag & RX_FLAG_HT)) {
+ if (!!(rxs->encoding == RX_ENC_HT)) {
/* MCS rates */
airtime += ath_pkt_duration(sc, rxs->rate_idx, len,
diff --git a/drivers/net/wireless/ath/carl9170/main.c b/drivers/net/wireless/ath/carl9170/main.c
index ffb22a0..988c885 100644
--- a/drivers/net/wireless/ath/carl9170/main.c
+++ b/drivers/net/wireless/ath/carl9170/main.c
@@ -1874,6 +1874,8 @@
for (i = 0; i < ARRAY_SIZE(ar->noise); i++)
ar->noise[i] = -95; /* ATH_DEFAULT_NOISE_FLOOR */
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
return ar;
err_nomem:
diff --git a/drivers/net/wireless/ath/carl9170/rx.c b/drivers/net/wireless/ath/carl9170/rx.c
index 0c34c87..b216672 100644
--- a/drivers/net/wireless/ath/carl9170/rx.c
+++ b/drivers/net/wireless/ath/carl9170/rx.c
@@ -358,7 +358,7 @@
switch (mac->status & AR9170_RX_STATUS_MODULATION) {
case AR9170_RX_STATUS_MODULATION_CCK:
if (mac->status & AR9170_RX_STATUS_SHORT_PREAMBLE)
- status->flag |= RX_FLAG_SHORTPRE;
+ status->enc_flags |= RX_ENC_FLAG_SHORTPRE;
switch (head->plcp[0]) {
case AR9170_RX_PHY_RATE_CCK_1M:
status->rate_idx = 0;
@@ -423,12 +423,12 @@
case AR9170_RX_STATUS_MODULATION_HT:
if (head->plcp[3] & 0x80)
- status->flag |= RX_FLAG_40MHZ;
+ status->bw = RATE_INFO_BW_40;
if (head->plcp[6] & 0x80)
- status->flag |= RX_FLAG_SHORT_GI;
+ status->enc_flags |= RX_ENC_FLAG_SHORT_GI;
status->rate_idx = clamp(0, 75, head->plcp[3] & 0x7f);
- status->flag |= RX_FLAG_HT;
+ status->encoding = RX_ENC_HT;
break;
default:
diff --git a/drivers/net/wireless/ath/regd.c b/drivers/net/wireless/ath/regd.c
index 43afa83..e25bfdf 100644
--- a/drivers/net/wireless/ath/regd.c
+++ b/drivers/net/wireless/ath/regd.c
@@ -254,8 +254,12 @@
EXPORT_SYMBOL(ath_is_49ghz_allowed);
/* Frequency is one where radar detection is required */
-static bool ath_is_radar_freq(u16 center_freq)
+static bool ath_is_radar_freq(u16 center_freq,
+ struct ath_regulatory *reg)
+
{
+ if (reg->country_code == CTRY_INDIA)
+ return (center_freq >= 5500 && center_freq <= 5700);
return (center_freq >= 5260 && center_freq <= 5700);
}
@@ -306,7 +310,7 @@
enum nl80211_reg_initiator initiator,
struct ieee80211_channel *ch)
{
- if (ath_is_radar_freq(ch->center_freq) ||
+ if (ath_is_radar_freq(ch->center_freq, reg) ||
(ch->flags & IEEE80211_CHAN_RADAR))
return;
@@ -395,8 +399,9 @@
}
}
-/* Always apply Radar/DFS rules on freq range 5260 MHz - 5700 MHz */
-static void ath_reg_apply_radar_flags(struct wiphy *wiphy)
+/* Always apply Radar/DFS rules on freq range 5500 MHz - 5700 MHz */
+static void ath_reg_apply_radar_flags(struct wiphy *wiphy,
+ struct ath_regulatory *reg)
{
struct ieee80211_supported_band *sband;
struct ieee80211_channel *ch;
@@ -409,7 +414,7 @@
for (i = 0; i < sband->n_channels; i++) {
ch = &sband->channels[i];
- if (!ath_is_radar_freq(ch->center_freq))
+ if (!ath_is_radar_freq(ch->center_freq, reg))
continue;
/* We always enable radar detection/DFS on this
* frequency range. Additionally we also apply on
@@ -506,7 +511,7 @@
struct ath_common *common = container_of(reg, struct ath_common,
regulatory);
/* We always apply this */
- ath_reg_apply_radar_flags(wiphy);
+ ath_reg_apply_radar_flags(wiphy, reg);
/*
* This would happen when we have sent a custom regulatory request
@@ -654,7 +659,7 @@
}
wiphy_apply_custom_regulatory(wiphy, regd);
- ath_reg_apply_radar_flags(wiphy);
+ ath_reg_apply_radar_flags(wiphy, reg);
ath_reg_apply_world_flags(wiphy, NL80211_REGDOM_SET_BY_DRIVER, reg);
return 0;
}
diff --git a/drivers/net/wireless/ath/wcn36xx/Kconfig b/drivers/net/wireless/ath/wcn36xx/Kconfig
index 4b83e87..20bf967 100644
--- a/drivers/net/wireless/ath/wcn36xx/Kconfig
+++ b/drivers/net/wireless/ath/wcn36xx/Kconfig
@@ -2,7 +2,7 @@
tristate "Qualcomm Atheros WCN3660/3680 support"
depends on MAC80211 && HAS_DMA
depends on QCOM_WCNSS_CTRL || QCOM_WCNSS_CTRL=n
- depends on QCOM_SMD || QCOM_SMD=n
+ depends on RPMSG || RPMSG=n
---help---
This module adds support for wireless adapters based on
Qualcomm Atheros WCN3660 and WCN3680 mobile chipsets.
diff --git a/drivers/net/wireless/ath/wcn36xx/main.c b/drivers/net/wireless/ath/wcn36xx/main.c
index 7a0c2e7..d5e993d 100644
--- a/drivers/net/wireless/ath/wcn36xx/main.c
+++ b/drivers/net/wireless/ath/wcn36xx/main.c
@@ -22,7 +22,7 @@
#include <linux/of_address.h>
#include <linux/of_device.h>
#include <linux/of_irq.h>
-#include <linux/soc/qcom/smd.h>
+#include <linux/rpmsg.h>
#include <linux/soc/qcom/smem_state.h>
#include <linux/soc/qcom/wcnss_ctrl.h>
#include "wcn36xx.h"
@@ -337,10 +337,10 @@
wcn36xx_smd_stop(wcn);
out_free_smd_buf:
kfree(wcn->hal_buf);
-out_free_dxe_pool:
- wcn36xx_dxe_free_mem_pools(wcn);
out_free_dxe_ctl:
wcn36xx_dxe_free_ctl_blks(wcn);
+out_free_dxe_pool:
+ wcn36xx_dxe_free_mem_pools(wcn);
out_smd_close:
wcn36xx_smd_close(wcn);
out_err:
@@ -1112,6 +1112,9 @@
wcn->hw->sta_data_size = sizeof(struct wcn36xx_sta);
wcn->hw->vif_data_size = sizeof(struct wcn36xx_vif);
+ wiphy_ext_feature_set(wcn->hw->wiphy,
+ NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
return ret;
}
@@ -1218,15 +1221,13 @@
INIT_WORK(&wcn->scan_work, wcn36xx_hw_scan_worker);
- wcn->smd_channel = qcom_wcnss_open_channel(wcnss, "WLAN_CTRL", wcn36xx_smd_rsp_process);
+ wcn->smd_channel = qcom_wcnss_open_channel(wcnss, "WLAN_CTRL", wcn36xx_smd_rsp_process, hw);
if (IS_ERR(wcn->smd_channel)) {
wcn36xx_err("failed to open WLAN_CTRL channel\n");
ret = PTR_ERR(wcn->smd_channel);
goto out_wq;
}
- qcom_smd_set_drvdata(wcn->smd_channel, hw);
-
addr = of_get_property(pdev->dev.of_node, "local-mac-address", &ret);
if (addr && ret != ETH_ALEN) {
wcn36xx_err("invalid local-mac-address\n");
diff --git a/drivers/net/wireless/ath/wcn36xx/smd.c b/drivers/net/wireless/ath/wcn36xx/smd.c
index 1c2966f..9c6590d 100644
--- a/drivers/net/wireless/ath/wcn36xx/smd.c
+++ b/drivers/net/wireless/ath/wcn36xx/smd.c
@@ -19,7 +19,7 @@
#include <linux/etherdevice.h>
#include <linux/firmware.h>
#include <linux/bitops.h>
-#include <linux/soc/qcom/smd.h>
+#include <linux/rpmsg.h>
#include "smd.h"
struct wcn36xx_cfg_val {
@@ -254,7 +254,7 @@
init_completion(&wcn->hal_rsp_compl);
start = jiffies;
- ret = qcom_smd_send(wcn->smd_channel, wcn->hal_buf, len);
+ ret = rpmsg_send(wcn->smd_channel, wcn->hal_buf, len);
if (ret) {
wcn36xx_err("HAL TX failed\n");
goto out;
@@ -2205,11 +2205,11 @@
return ret;
}
-int wcn36xx_smd_rsp_process(struct qcom_smd_channel *channel,
- const void *buf, size_t len)
+int wcn36xx_smd_rsp_process(struct rpmsg_device *rpdev,
+ void *buf, int len, void *priv, u32 addr)
{
const struct wcn36xx_hal_msg_header *msg_header = buf;
- struct ieee80211_hw *hw = qcom_smd_get_drvdata(channel);
+ struct ieee80211_hw *hw = priv;
struct wcn36xx *wcn = hw->priv;
struct wcn36xx_hal_ind_msg *msg_ind;
wcn36xx_dbg_dump(WCN36XX_DBG_SMD_DUMP, "SMD <<< ", buf, len);
diff --git a/drivers/net/wireless/ath/wcn36xx/smd.h b/drivers/net/wireless/ath/wcn36xx/smd.h
index 8892ccd..013fc95 100644
--- a/drivers/net/wireless/ath/wcn36xx/smd.h
+++ b/drivers/net/wireless/ath/wcn36xx/smd.h
@@ -51,7 +51,7 @@
};
struct wcn36xx;
-struct qcom_smd_channel;
+struct rpmsg_device;
int wcn36xx_smd_open(struct wcn36xx *wcn);
void wcn36xx_smd_close(struct wcn36xx *wcn);
@@ -129,8 +129,8 @@
int wcn36xx_smd_update_cfg(struct wcn36xx *wcn, u32 cfg_id, u32 value);
-int wcn36xx_smd_rsp_process(struct qcom_smd_channel *channel,
- const void *buf, size_t len);
+int wcn36xx_smd_rsp_process(struct rpmsg_device *rpdev,
+ void *buf, int len, void *priv, u32 addr);
int wcn36xx_smd_set_mc_list(struct wcn36xx *wcn,
struct ieee80211_vif *vif,
diff --git a/drivers/net/wireless/ath/wcn36xx/txrx.c b/drivers/net/wireless/ath/wcn36xx/txrx.c
index 8c387a0..22304ed 100644
--- a/drivers/net/wireless/ath/wcn36xx/txrx.c
+++ b/drivers/net/wireless/ath/wcn36xx/txrx.c
@@ -68,7 +68,7 @@
RX_FLAG_MMIC_STRIPPED |
RX_FLAG_DECRYPTED;
- wcn36xx_dbg(WCN36XX_DBG_RX, "status.flags=%llx\n", status.flag);
+ wcn36xx_dbg(WCN36XX_DBG_RX, "status.flags=%x\n", status.flag);
memcpy(IEEE80211_SKB_RXCB(skb), &status, sizeof(status));
diff --git a/drivers/net/wireless/ath/wcn36xx/wcn36xx.h b/drivers/net/wireless/ath/wcn36xx/wcn36xx.h
index 7423998..b52b4da9 100644
--- a/drivers/net/wireless/ath/wcn36xx/wcn36xx.h
+++ b/drivers/net/wireless/ath/wcn36xx/wcn36xx.h
@@ -195,7 +195,7 @@
void __iomem *ccu_base;
void __iomem *dxe_base;
- struct qcom_smd_channel *smd_channel;
+ struct rpmsg_endpoint *smd_channel;
struct qcom_smem_state *tx_enable_state;
unsigned tx_enable_state_bit;
diff --git a/drivers/net/wireless/ath/wil6210/cfg80211.c b/drivers/net/wireless/ath/wil6210/cfg80211.c
index 83155b5..fdaa99c 100644
--- a/drivers/net/wireless/ath/wil6210/cfg80211.c
+++ b/drivers/net/wireless/ath/wil6210/cfg80211.c
@@ -178,9 +178,8 @@
BIT(NL80211_STA_INFO_RX_DROP_MISC) |
BIT(NL80211_STA_INFO_TX_FAILED);
- sinfo->txrate.flags = RATE_INFO_FLAGS_MCS | RATE_INFO_FLAGS_60G;
+ sinfo->txrate.flags = RATE_INFO_FLAGS_60G;
sinfo->txrate.mcs = le16_to_cpu(reply.evt.bf_mcs);
- sinfo->rxrate.flags = RATE_INFO_FLAGS_MCS | RATE_INFO_FLAGS_60G;
sinfo->rxrate.mcs = stats->last_mcs_rx;
sinfo->rx_bytes = stats->rx_bytes;
sinfo->rx_packets = stats->rx_packets;
@@ -256,7 +255,7 @@
wil_cfg80211_add_iface(struct wiphy *wiphy, const char *name,
unsigned char name_assign_type,
enum nl80211_iftype type,
- u32 *flags, struct vif_params *params)
+ struct vif_params *params)
{
struct wil6210_priv *wil = wiphy_to_wil(wiphy);
struct net_device *ndev = wil_to_ndev(wil);
@@ -307,7 +306,7 @@
static int wil_cfg80211_change_iface(struct wiphy *wiphy,
struct net_device *ndev,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct wil6210_priv *wil = wiphy_to_wil(wiphy);
@@ -334,11 +333,8 @@
case NL80211_IFTYPE_P2P_GO:
break;
case NL80211_IFTYPE_MONITOR:
- if (flags)
- wil->monitor_flags = *flags;
- else
- wil->monitor_flags = 0;
-
+ if (params->flags)
+ wil->monitor_flags = params->flags;
break;
default:
return -EOPNOTSUPP;
@@ -390,22 +386,23 @@
}
mutex_unlock(&wil->p2p_wdev_mutex);
- /* social scan on P2P_DEVICE is handled as p2p search */
- if (wdev->iftype == NL80211_IFTYPE_P2P_DEVICE &&
- wil_p2p_is_social_scan(request)) {
+ if (wdev->iftype == NL80211_IFTYPE_P2P_DEVICE) {
if (!wil->p2p.p2p_dev_started) {
wil_err(wil, "P2P search requested on stopped P2P device\n");
rc = -EIO;
goto out;
}
- wil->scan_request = request;
- wil->radio_wdev = wdev;
- rc = wil_p2p_search(wil, request);
- if (rc) {
- wil->radio_wdev = wil_to_wdev(wil);
- wil->scan_request = NULL;
+ /* social scan on P2P_DEVICE is handled as p2p search */
+ if (wil_p2p_is_social_scan(request)) {
+ wil->scan_request = request;
+ wil->radio_wdev = wdev;
+ rc = wil_p2p_search(wil, request);
+ if (rc) {
+ wil->radio_wdev = wil_to_wdev(wil);
+ wil->scan_request = NULL;
+ }
+ goto out;
}
- goto out;
}
(void)wil_p2p_stop_discovery(wil);
@@ -415,9 +412,9 @@
for (i = 0; i < request->n_ssids; i++) {
wil_dbg_misc(wil, "SSID[%d]", i);
- print_hex_dump_bytes("SSID ", DUMP_PREFIX_OFFSET,
- request->ssids[i].ssid,
- request->ssids[i].ssid_len);
+ wil_hex_dump_misc("SSID ", DUMP_PREFIX_OFFSET, 16, 1,
+ request->ssids[i].ssid,
+ request->ssids[i].ssid_len, true);
}
if (request->n_ssids)
@@ -454,8 +451,8 @@
}
if (request->ie_len)
- print_hex_dump_bytes("Scan IE ", DUMP_PREFIX_OFFSET,
- request->ie, request->ie_len);
+ wil_hex_dump_misc("Scan IE ", DUMP_PREFIX_OFFSET, 16, 1,
+ request->ie, request->ie_len, true);
else
wil_dbg_misc(wil, "Scan has no IE's\n");
@@ -679,6 +676,8 @@
rc = wmi_send(wil, WMI_CONNECT_CMDID, &conn, sizeof(conn));
if (rc == 0) {
netif_carrier_on(ndev);
+ wil6210_bus_request(wil, WIL_MAX_BUS_REQUEST_KBPS);
+ wil->bss = bss;
/* Connect can take lots of time */
mod_timer(&wil->connect_timer,
jiffies + msecs_to_jiffies(2000));
@@ -707,6 +706,7 @@
return 0;
}
+ wil->locally_generated_disc = true;
rc = wmi_call(wil, WMI_DISCONNECT_CMDID, NULL, 0,
WMI_DISCONNECT_EVENTID, NULL, 0,
WIL6210_DISCONNECT_TO_MS);
@@ -760,7 +760,8 @@
*/
wil_dbg_misc(wil, "mgmt_tx\n");
- print_hex_dump_bytes("mgmt tx frame ", DUMP_PREFIX_OFFSET, buf, len);
+ wil_hex_dump_misc("mgmt tx frame ", DUMP_PREFIX_OFFSET, 16, 1, buf,
+ len, true);
cmd = kmalloc(sizeof(*cmd) + len, GFP_KERNEL);
if (!cmd) {
@@ -1093,18 +1094,18 @@
static void wil_print_bcon_data(struct cfg80211_beacon_data *b)
{
- print_hex_dump_bytes("head ", DUMP_PREFIX_OFFSET,
- b->head, b->head_len);
- print_hex_dump_bytes("tail ", DUMP_PREFIX_OFFSET,
- b->tail, b->tail_len);
- print_hex_dump_bytes("BCON IE ", DUMP_PREFIX_OFFSET,
- b->beacon_ies, b->beacon_ies_len);
- print_hex_dump_bytes("PROBE ", DUMP_PREFIX_OFFSET,
- b->probe_resp, b->probe_resp_len);
- print_hex_dump_bytes("PROBE IE ", DUMP_PREFIX_OFFSET,
- b->proberesp_ies, b->proberesp_ies_len);
- print_hex_dump_bytes("ASSOC IE ", DUMP_PREFIX_OFFSET,
- b->assocresp_ies, b->assocresp_ies_len);
+ wil_hex_dump_misc("head ", DUMP_PREFIX_OFFSET, 16, 1,
+ b->head, b->head_len, true);
+ wil_hex_dump_misc("tail ", DUMP_PREFIX_OFFSET, 16, 1,
+ b->tail, b->tail_len, true);
+ wil_hex_dump_misc("BCON IE ", DUMP_PREFIX_OFFSET, 16, 1,
+ b->beacon_ies, b->beacon_ies_len, true);
+ wil_hex_dump_misc("PROBE ", DUMP_PREFIX_OFFSET, 16, 1,
+ b->probe_resp, b->probe_resp_len, true);
+ wil_hex_dump_misc("PROBE IE ", DUMP_PREFIX_OFFSET, 16, 1,
+ b->proberesp_ies, b->proberesp_ies_len, true);
+ wil_hex_dump_misc("ASSOC IE ", DUMP_PREFIX_OFFSET, 16, 1,
+ b->assocresp_ies, b->assocresp_ies_len, true);
}
/* internal functions for device reset and starting AP */
@@ -1198,6 +1199,7 @@
wil->pbss = pbss;
netif_carrier_on(ndev);
+ wil6210_bus_request(wil, WIL_MAX_BUS_REQUEST_KBPS);
rc = wmi_pcp_start(wil, bi, wmi_nettype, chan, hidden_ssid, is_go);
if (rc)
@@ -1213,6 +1215,7 @@
wmi_pcp_stop(wil);
err_pcp_start:
netif_carrier_off(ndev);
+ wil6210_bus_request(wil, WIL_DEFAULT_BUS_REQUEST_KBPS);
out:
mutex_unlock(&wil->mutex);
return rc;
@@ -1298,8 +1301,8 @@
wil_dbg_misc(wil, "BI %d DTIM %d\n", info->beacon_interval,
info->dtim_period);
wil_dbg_misc(wil, "PBSS %d\n", info->pbss);
- print_hex_dump_bytes("SSID ", DUMP_PREFIX_OFFSET,
- info->ssid, info->ssid_len);
+ wil_hex_dump_misc("SSID ", DUMP_PREFIX_OFFSET, 16, 1,
+ info->ssid, info->ssid_len, true);
wil_print_bcon_data(bcon);
wil_print_crypto(wil, crypto);
@@ -1319,6 +1322,7 @@
wil_dbg_misc(wil, "stop_ap\n");
netif_carrier_off(ndev);
+ wil6210_bus_request(wil, WIL_DEFAULT_BUS_REQUEST_KBPS);
wil_set_recovery_state(wil, fw_recovery_idle);
mutex_lock(&wil->mutex);
@@ -1555,12 +1559,6 @@
{
struct wil6210_priv *wil = wiphy_to_wil(wiphy);
enum wmi_ps_profile_type ps_profile;
- int rc;
-
- if (!test_bit(WMI_FW_CAPABILITY_PS_CONFIG, wil->fw_capabilities)) {
- wil_err(wil, "set_power_mgmt not supported\n");
- return -EOPNOTSUPP;
- }
wil_dbg_misc(wil, "enabled=%d, timeout=%d\n",
enabled, timeout);
@@ -1570,11 +1568,7 @@
else
ps_profile = WMI_PS_PROFILE_TYPE_PS_DISABLED;
- rc = wmi_ps_dev_profile_cfg(wil, ps_profile);
- if (rc)
- wil_err(wil, "wmi_ps_dev_profile_cfg failed (%d)\n", rc);
-
- return rc;
+ return wil_ps_update(wil, ps_profile);
}
static const struct cfg80211_ops wil_cfg80211_ops = {
diff --git a/drivers/net/wireless/ath/wil6210/debugfs.c b/drivers/net/wireless/ath/wil6210/debugfs.c
index 3e8cdf1..5648ebb 100644
--- a/drivers/net/wireless/ath/wil6210/debugfs.c
+++ b/drivers/net/wireless/ath/wil6210/debugfs.c
@@ -524,9 +524,8 @@
if (!buf)
return -ENOMEM;
- wil_memcpy_fromio_halp_vote(wil_blob->wil, buf,
- (const volatile void __iomem *)
- wil_blob->blob.data + pos, count);
+ wil_memcpy_fromio_32(buf, (const void __iomem *)
+ wil_blob->blob.data + pos, count);
ret = copy_to_user(user_buf, buf, count);
kfree(buf);
diff --git a/drivers/net/wireless/ath/wil6210/fw_inc.c b/drivers/net/wireless/ath/wil6210/fw_inc.c
index f490158..e01acac 100644
--- a/drivers/net/wireless/ath/wil6210/fw_inc.c
+++ b/drivers/net/wireless/ath/wil6210/fw_inc.c
@@ -554,5 +554,7 @@
rc = request_firmware(&fw, name, wil_to_dev(wil));
if (!rc)
release_firmware(fw);
- return rc != -ENOENT;
+ else
+ wil_dbg_fw(wil, "<%s> not available: %d\n", name, rc);
+ return !rc;
}
diff --git a/drivers/net/wireless/ath/wil6210/main.c b/drivers/net/wireless/ath/wil6210/main.c
index efb1f59..3208679 100644
--- a/drivers/net/wireless/ath/wil6210/main.c
+++ b/drivers/net/wireless/ath/wil6210/main.c
@@ -30,8 +30,8 @@
module_param(debug_fw, bool, 0444);
MODULE_PARM_DESC(debug_fw, " do not perform card reset. For FW debug");
-static bool oob_mode;
-module_param(oob_mode, bool, 0444);
+static u8 oob_mode;
+module_param(oob_mode, byte, 0444);
MODULE_PARM_DESC(oob_mode,
" enable out of the box (OOB) mode in FW, for diagnostics and certification");
@@ -130,17 +130,15 @@
u32 *d = dst;
const volatile u32 __iomem *s = src;
- /* size_t is unsigned, if (count%4 != 0) it will wrap */
- for (count += 4; count > 4; count -= 4)
+ for (; count >= 4; count -= 4)
*d++ = __raw_readl(s++);
-}
-void wil_memcpy_fromio_halp_vote(struct wil6210_priv *wil, void *dst,
- const volatile void __iomem *src, size_t count)
-{
- wil_halp_vote(wil);
- wil_memcpy_fromio_32(dst, src, count);
- wil_halp_unvote(wil);
+ if (unlikely(count)) {
+ /* count can be 1..3 */
+ u32 tmp = __raw_readl(s);
+
+ memcpy(d, &tmp, count);
+ }
}
void wil_memcpy_toio_32(volatile void __iomem *dst, const void *src,
@@ -149,17 +147,16 @@
volatile u32 __iomem *d = dst;
const u32 *s = src;
- for (count += 4; count > 4; count -= 4)
+ for (; count >= 4; count -= 4)
__raw_writel(*s++, d++);
-}
-void wil_memcpy_toio_halp_vote(struct wil6210_priv *wil,
- volatile void __iomem *dst,
- const void *src, size_t count)
-{
- wil_halp_vote(wil);
- wil_memcpy_toio_32(dst, src, count);
- wil_halp_unvote(wil);
+ if (unlikely(count)) {
+ /* count can be 1..3 */
+ u32 tmp = 0;
+
+ memcpy(&tmp, s, count);
+ __raw_writel(tmp, d);
+ }
}
static void wil_disconnect_cid(struct wil6210_priv *wil, int cid,
@@ -274,15 +271,20 @@
wil_bcast_fini(wil);
wil_update_net_queues_bh(wil, NULL, true);
netif_carrier_off(ndev);
+ wil6210_bus_request(wil, WIL_DEFAULT_BUS_REQUEST_KBPS);
if (test_bit(wil_status_fwconnected, wil->status)) {
clear_bit(wil_status_fwconnected, wil->status);
cfg80211_disconnected(ndev, reason_code,
- NULL, 0, false, GFP_KERNEL);
+ NULL, 0,
+ wil->locally_generated_disc,
+ GFP_KERNEL);
+ wil->locally_generated_disc = false;
} else if (test_bit(wil_status_fwconnecting, wil->status)) {
cfg80211_connect_result(ndev, bssid, NULL, 0, NULL, 0,
WLAN_STATUS_UNSPECIFIED_FAILURE,
GFP_KERNEL);
+ wil->bss = NULL;
}
clear_bit(wil_status_fwconnecting, wil->status);
break;
@@ -304,10 +306,34 @@
{
struct wil6210_priv *wil = container_of(work,
struct wil6210_priv, disconnect_worker);
+ struct net_device *ndev = wil_to_ndev(wil);
+ int rc;
+ struct {
+ struct wmi_cmd_hdr wmi;
+ struct wmi_disconnect_event evt;
+ } __packed reply;
- mutex_lock(&wil->mutex);
- _wil6210_disconnect(wil, NULL, WLAN_REASON_UNSPECIFIED, false);
- mutex_unlock(&wil->mutex);
+ if (test_bit(wil_status_fwconnected, wil->status))
+ /* connect succeeded after all */
+ return;
+
+ if (!test_bit(wil_status_fwconnecting, wil->status))
+ /* already disconnected */
+ return;
+
+ rc = wmi_call(wil, WMI_DISCONNECT_CMDID, NULL, 0,
+ WMI_DISCONNECT_EVENTID, &reply, sizeof(reply),
+ WIL6210_DISCONNECT_TO_MS);
+ if (rc) {
+ wil_err(wil, "disconnect error %d\n", rc);
+ return;
+ }
+
+ wil_update_net_queues_bh(wil, NULL, true);
+ netif_carrier_off(ndev);
+ cfg80211_connect_result(ndev, NULL, NULL, 0, NULL, 0,
+ WLAN_STATUS_UNSPECIFIED_FAILURE, GFP_KERNEL);
+ clear_bit(wil_status_fwconnecting, wil->status);
}
static void wil_connect_timer_fn(ulong x)
@@ -547,6 +573,9 @@
if (rx_ring_overflow_thrsh == WIL6210_RX_HIGH_TRSH_INIT)
rx_ring_overflow_thrsh = WIL6210_RX_HIGH_TRSH_DEFAULT;
+
+ wil->ps_profile = WMI_PS_PROFILE_TYPE_DEFAULT;
+
return 0;
out_wmi_wq:
@@ -555,6 +584,12 @@
return -EAGAIN;
}
+void wil6210_bus_request(struct wil6210_priv *wil, u32 kbps)
+{
+ if (wil->platform_ops.bus_request)
+ wil->platform_ops.bus_request(wil->platform_handle, kbps);
+}
+
/**
* wil6210_disconnect - disconnect one connection
* @wil: driver context
@@ -607,13 +642,25 @@
wil_w(wil, RGF_USER_USER_CPU_0, 1);
}
-static void wil_set_oob_mode(struct wil6210_priv *wil, bool enable)
+static void wil_set_oob_mode(struct wil6210_priv *wil, u8 mode)
{
- wil_info(wil, "enable=%d\n", enable);
- if (enable)
+ wil_info(wil, "oob_mode to %d\n", mode);
+ switch (mode) {
+ case 0:
+ wil_c(wil, RGF_USER_USAGE_6, BIT_USER_OOB_MODE |
+ BIT_USER_OOB_R2_MODE);
+ break;
+ case 1:
+ wil_c(wil, RGF_USER_USAGE_6, BIT_USER_OOB_R2_MODE);
wil_s(wil, RGF_USER_USAGE_6, BIT_USER_OOB_MODE);
- else
+ break;
+ case 2:
wil_c(wil, RGF_USER_USAGE_6, BIT_USER_OOB_MODE);
+ wil_s(wil, RGF_USER_USAGE_6, BIT_USER_OOB_R2_MODE);
+ break;
+ default:
+ wil_err(wil, "invalid oob_mode: %d\n", mode);
+ }
}
static int wil_target_reset(struct wil6210_priv *wil)
@@ -856,6 +903,24 @@
}
}
+int wil_ps_update(struct wil6210_priv *wil, enum wmi_ps_profile_type ps_profile)
+{
+ int rc;
+
+ if (!test_bit(WMI_FW_CAPABILITY_PS_CONFIG, wil->fw_capabilities)) {
+ wil_err(wil, "set_power_mgmt not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ rc = wmi_ps_dev_profile_cfg(wil, ps_profile);
+ if (rc)
+ wil_err(wil, "wmi_ps_dev_profile_cfg failed (%d)\n", rc);
+ else
+ wil->ps_profile = ps_profile;
+
+ return rc;
+}
+
/*
* We reset all the structures, and we reset the UMAC.
* After calling this routine, you're expected to reload
@@ -901,15 +966,15 @@
/* Disable device led before reset*/
wmi_led_cfg(wil, false);
+ mutex_lock(&wil->p2p_wdev_mutex);
+ wil_abort_scan(wil, false);
+ mutex_unlock(&wil->p2p_wdev_mutex);
+
/* prevent NAPI from being scheduled and prevent wmi commands */
mutex_lock(&wil->wmi_mutex);
bitmap_zero(wil->status, wil_status_last);
mutex_unlock(&wil->wmi_mutex);
- mutex_lock(&wil->p2p_wdev_mutex);
- wil_abort_scan(wil, false);
- mutex_unlock(&wil->p2p_wdev_mutex);
-
wil_mask_irq(wil);
wmi_event_flush(wil);
@@ -986,6 +1051,9 @@
return rc;
}
+ if (wil->ps_profile != WMI_PS_PROFILE_TYPE_DEFAULT)
+ wil_ps_update(wil, wil->ps_profile);
+
wil_collect_fw_info(wil);
if (wil->platform_ops.notify) {
@@ -1066,9 +1134,7 @@
napi_enable(&wil->napi_tx);
set_bit(wil_status_napi_en, wil->status);
- if (wil->platform_ops.bus_request)
- wil->platform_ops.bus_request(wil->platform_handle,
- WIL_MAX_BUS_REQUEST_KBPS);
+ wil6210_bus_request(wil, WIL_DEFAULT_BUS_REQUEST_KBPS);
return 0;
}
@@ -1092,8 +1158,7 @@
set_bit(wil_status_resetting, wil->status);
- if (wil->platform_ops.bus_request)
- wil->platform_ops.bus_request(wil->platform_handle, 0);
+ wil6210_bus_request(wil, 0);
wil_disable_irq(wil);
if (test_and_clear_bit(wil_status_napi_en, wil->status)) {
@@ -1154,6 +1219,7 @@
wil->halp.ref_cnt);
if (++wil->halp.ref_cnt == 1) {
+ reinit_completion(&wil->halp.comp);
wil6210_set_halp(wil);
rc = wait_for_completion_timeout(&wil->halp.comp, to_jiffies);
if (!rc) {
diff --git a/drivers/net/wireless/ath/wil6210/pcie_bus.c b/drivers/net/wireless/ath/wil6210/pcie_bus.c
index 874c787..b38515f 100644
--- a/drivers/net/wireless/ath/wil6210/pcie_bus.c
+++ b/drivers/net/wireless/ath/wil6210/pcie_bus.c
@@ -211,6 +211,7 @@
dev_err(dev, "wil_if_alloc failed: %d\n", rc);
return rc;
}
+
wil->pdev = pdev;
pci_set_drvdata(pdev, wil);
/* rollback to if_free */
@@ -224,6 +225,21 @@
}
/* rollback to err_plat */
+ /* device supports 48bit addresses */
+ rc = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(48));
+ if (rc) {
+ dev_err(dev, "dma_set_mask_and_coherent(48) failed: %d\n", rc);
+ rc = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
+ if (rc) {
+ dev_err(dev,
+ "dma_set_mask_and_coherent(32) failed: %d\n",
+ rc);
+ goto err_plat;
+ }
+ } else {
+ wil->use_extended_dma_addr = 1;
+ }
+
rc = pci_enable_device(pdev);
if (rc) {
wil_err(wil,
diff --git a/drivers/net/wireless/ath/wil6210/pm.c b/drivers/net/wireless/ath/wil6210/pm.c
index a0acb2d..2ae4fe8 100644
--- a/drivers/net/wireless/ath/wil6210/pm.c
+++ b/drivers/net/wireless/ath/wil6210/pm.c
@@ -71,6 +71,11 @@
wil_dbg_pm(wil, "suspend: %s\n", is_runtime ? "runtime" : "system");
+ if (test_bit(wil_status_suspended, wil->status)) {
+ wil_dbg_pm(wil, "trying to suspend while suspended\n");
+ return 0;
+ }
+
/* if netif up, hardware is alive, shut it down */
if (ndev->flags & IFF_UP) {
rc = wil_down(wil);
@@ -80,12 +85,24 @@
}
}
- if (wil->platform_ops.suspend)
+ /* Disable PCIe IRQ to prevent sporadic IRQs when PCIe is suspending */
+ wil_dbg_pm(wil, "Disabling PCIe IRQ before suspending\n");
+ wil_disable_irq(wil);
+
+ if (wil->platform_ops.suspend) {
rc = wil->platform_ops.suspend(wil->platform_handle);
+ if (rc) {
+ wil_enable_irq(wil);
+ goto out;
+ }
+ }
+
+ set_bit(wil_status_suspended, wil->status);
out:
wil_dbg_pm(wil, "suspend: %s => %d\n",
is_runtime ? "runtime" : "system", rc);
+
return rc;
}
@@ -104,12 +121,18 @@
}
}
+ wil_dbg_pm(wil, "Enabling PCIe IRQ\n");
+ wil_enable_irq(wil);
+
/* if netif up, bring hardware up
* During open(), IFF_UP set after actual device method
- * invocation. This prevent recursive call to wil_up()
+ * invocation. This prevent recursive call to wil_up().
+ * wil_status_suspended will be cleared in wil_reset
*/
if (ndev->flags & IFF_UP)
rc = wil_up(wil);
+ else
+ clear_bit(wil_status_suspended, wil->status);
out:
wil_dbg_pm(wil, "resume: %s => %d\n",
diff --git a/drivers/net/wireless/ath/wil6210/pmc.c b/drivers/net/wireless/ath/wil6210/pmc.c
index 3ff4f4c..2e301b6 100644
--- a/drivers/net/wireless/ath/wil6210/pmc.c
+++ b/drivers/net/wireless/ath/wil6210/pmc.c
@@ -107,13 +107,28 @@
/* Allocate pring buffer and descriptors.
* vring->va should be aligned on its size rounded up to power of 2
- * This is granted by the dma_alloc_coherent
+ * This is granted by the dma_alloc_coherent.
+ *
+ * HW has limitation that all vrings addresses must share the same
+ * upper 16 msb bits part of 48 bits address. To workaround that,
+ * if we are using 48 bit addresses switch to 32 bit allocation
+ * before allocating vring memory.
+ *
+ * There's no check for the return value of dma_set_mask_and_coherent,
+ * since we assume if we were able to set the mask during
+ * initialization in this system it will not fail if we set it again
*/
+ if (wil->use_extended_dma_addr)
+ dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
+
pmc->pring_va = dma_alloc_coherent(dev,
sizeof(struct vring_tx_desc) * num_descriptors,
&pmc->pring_pa,
GFP_KERNEL);
+ if (wil->use_extended_dma_addr)
+ dma_set_mask_and_coherent(dev, DMA_BIT_MASK(48));
+
wil_dbg_misc(wil,
"pmc_alloc: allocated pring %p => %pad. %zd x %d = total %zd bytes\n",
pmc->pring_va, &pmc->pring_pa,
@@ -185,7 +200,7 @@
release_pmc_skbs:
wil_err(wil, "exit on error: Releasing skbs...\n");
- for (i = 0; pmc->descriptors[i].va && i < num_descriptors; i++) {
+ for (i = 0; i < num_descriptors && pmc->descriptors[i].va; i++) {
dma_free_coherent(dev,
descriptor_size,
pmc->descriptors[i].va,
@@ -268,7 +283,7 @@
int i;
for (i = 0;
- pmc->descriptors[i].va && i < pmc->num_descriptors; i++) {
+ i < pmc->num_descriptors && pmc->descriptors[i].va; i++) {
dma_free_coherent(dev,
pmc->descriptor_size,
pmc->descriptors[i].va,
diff --git a/drivers/net/wireless/ath/wil6210/rx_reorder.c b/drivers/net/wireless/ath/wil6210/rx_reorder.c
index 7404b6f..a43cffc 100644
--- a/drivers/net/wireless/ath/wil6210/rx_reorder.c
+++ b/drivers/net/wireless/ath/wil6210/rx_reorder.c
@@ -343,8 +343,16 @@
wil_err(wil, "BACK requested unsupported ba_policy == 1\n");
status = WLAN_STATUS_INVALID_QOS_PARAM;
}
- if (status == WLAN_STATUS_SUCCESS)
- agg_wsize = wil_agg_size(wil, req_agg_wsize);
+ if (status == WLAN_STATUS_SUCCESS) {
+ if (req_agg_wsize == 0) {
+ wil_dbg_misc(wil, "Suggest BACK wsize %d\n",
+ WIL_MAX_AGG_WSIZE);
+ agg_wsize = WIL_MAX_AGG_WSIZE;
+ } else {
+ agg_wsize = min_t(u16,
+ WIL_MAX_AGG_WSIZE, req_agg_wsize);
+ }
+ }
rc = wmi_addba_rx_resp(wil, cid, tid, dialog_token, status,
agg_amsdu, agg_wsize, agg_timeout);
diff --git a/drivers/net/wireless/ath/wil6210/txrx.c b/drivers/net/wireless/ath/wil6210/txrx.c
index 072182e..edab4c0 100644
--- a/drivers/net/wireless/ath/wil6210/txrx.c
+++ b/drivers/net/wireless/ath/wil6210/txrx.c
@@ -37,6 +37,10 @@
module_param(rx_align_2, bool, 0444);
MODULE_PARM_DESC(rx_align_2, " align Rx buffers on 4*n+2, default - no");
+bool rx_large_buf;
+module_param(rx_large_buf, bool, 0444);
+MODULE_PARM_DESC(rx_large_buf, " allocate 8KB RX buffers, default - no");
+
static inline uint wil_rx_snaplen(void)
{
return rx_align_2 ? 6 : 0;
@@ -123,15 +127,32 @@
vring->va = NULL;
return -ENOMEM;
}
+
/* vring->va should be aligned on its size rounded up to power of 2
- * This is granted by the dma_alloc_coherent
+ * This is granted by the dma_alloc_coherent.
+ *
+ * HW has limitation that all vrings addresses must share the same
+ * upper 16 msb bits part of 48 bits address. To workaround that,
+ * if we are using 48 bit addresses switch to 32 bit allocation
+ * before allocating vring memory.
+ *
+ * There's no check for the return value of dma_set_mask_and_coherent,
+ * since we assume if we were able to set the mask during
+ * initialization in this system it will not fail if we set it again
*/
+ if (wil->use_extended_dma_addr)
+ dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
+
vring->va = dma_alloc_coherent(dev, sz, &vring->pa, GFP_KERNEL);
if (!vring->va) {
kfree(vring->ctx);
vring->ctx = NULL;
return -ENOMEM;
}
+
+ if (wil->use_extended_dma_addr)
+ dma_set_mask_and_coherent(dev, DMA_BIT_MASK(48));
+
/* initially, all descriptors are SW owned
* For Tx and Rx, ownership bit is at the same location, thus
* we can use any
@@ -238,7 +259,7 @@
u32 i, int headroom)
{
struct device *dev = wil_to_dev(wil);
- unsigned int sz = mtu_max + ETH_HLEN + wil_rx_snaplen();
+ unsigned int sz = wil->rx_buf_len + ETH_HLEN + wil_rx_snaplen();
struct vring_rx_desc dd, *d = ⅆ
volatile struct vring_rx_desc *_d = &vring->va[i].rx;
dma_addr_t pa;
@@ -402,7 +423,7 @@
struct sk_buff *skb;
dma_addr_t pa;
unsigned int snaplen = wil_rx_snaplen();
- unsigned int sz = mtu_max + ETH_HLEN + snaplen;
+ unsigned int sz = wil->rx_buf_len + ETH_HLEN + snaplen;
u16 dmalen;
u8 ftype;
int cid;
@@ -763,6 +784,20 @@
wil_rx_refill(wil, v->size);
}
+static void wil_rx_buf_len_init(struct wil6210_priv *wil)
+{
+ wil->rx_buf_len = rx_large_buf ?
+ WIL_MAX_ETH_MTU : TXRX_BUF_LEN_DEFAULT - WIL_MAX_MPDU_OVERHEAD;
+ if (mtu_max > wil->rx_buf_len) {
+ /* do not allow RX buffers to be smaller than mtu_max, for
+ * backward compatibility (mtu_max parameter was also used
+ * to support receiving large packets)
+ */
+ wil_info(wil, "Override RX buffer to mtu_max(%d)\n", mtu_max);
+ wil->rx_buf_len = mtu_max;
+ }
+}
+
int wil_rx_init(struct wil6210_priv *wil, u16 size)
{
struct vring *vring = &wil->vring_rx;
@@ -775,6 +810,8 @@
return -EINVAL;
}
+ wil_rx_buf_len_init(wil);
+
vring->size = size;
rc = wil_vring_alloc(wil, vring);
if (rc)
diff --git a/drivers/net/wireless/ath/wil6210/wil6210.h b/drivers/net/wireless/ath/wil6210/wil6210.h
index 085a2db..b00c803 100644
--- a/drivers/net/wireless/ath/wil6210/wil6210.h
+++ b/drivers/net/wireless/ath/wil6210/wil6210.h
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2012-2016 Qualcomm Atheros, Inc.
+ * Copyright (c) 2012-2017 Qualcomm Atheros, Inc.
*
* Permission to use, copy, modify, and/or distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
@@ -32,6 +32,7 @@
extern int agg_wsize;
extern u32 vring_idle_trsh;
extern bool rx_align_2;
+extern bool rx_large_buf;
extern bool debug_fw;
extern bool disable_ap_sme;
@@ -40,6 +41,7 @@
#define WIL_FW_NAME_SPARROW_PLUS "wil6210_sparrow_plus.fw" /* code Sparrow D0 */
#define WIL_BOARD_FILE_NAME "wil6210.brd" /* board & radio parameters */
+#define WIL_DEFAULT_BUS_REQUEST_KBPS 128000 /* ~1Gbps */
#define WIL_MAX_BUS_REQUEST_KBPS 800000 /* ~6.1Gbps */
/**
@@ -139,6 +141,7 @@
#define RGF_USER_USAGE_1 (0x880004)
#define RGF_USER_USAGE_6 (0x880018)
#define BIT_USER_OOB_MODE BIT(31)
+ #define BIT_USER_OOB_R2_MODE BIT(30)
#define RGF_USER_HW_MACHINE_STATE (0x8801dc)
#define HW_MACHINE_BOOT_DONE (0x3fffffd)
#define RGF_USER_USER_CPU_0 (0x8801e0)
@@ -409,6 +412,7 @@
wil_status_irqen, /* FIXME: interrupts enabled - for debug */
wil_status_napi_en, /* NAPI enabled protected by wil->mutex */
wil_status_resetting, /* reset in progress */
+ wil_status_suspended, /* suspend completed, device is suspended */
wil_status_last /* keep last */
};
@@ -612,6 +616,8 @@
u16 channel; /* relevant in AP mode */
int sinfo_gen;
u32 ap_isolate; /* no intra-BSS communication */
+ struct cfg80211_bss *bss; /* connected bss, relevant in STA mode */
+ int locally_generated_disc; /* relevant in STA mode */
/* interrupt moderation */
u32 tx_max_burst_duration;
u32 tx_interframe_timeout;
@@ -652,11 +658,13 @@
struct work_struct probe_client_worker;
/* DMA related */
struct vring vring_rx;
+ unsigned int rx_buf_len;
struct vring vring_tx[WIL6210_MAX_TX_RINGS];
struct vring_tx_data vring_tx_data[WIL6210_MAX_TX_RINGS];
u8 vring2cid_tid[WIL6210_MAX_TX_RINGS][2]; /* [0] - CID, [1] - TID */
struct wil_sta_info sta[WIL6210_MAX_CID];
int bcast_vring;
+ bool use_extended_dma_addr; /* indicates whether we are using 48 bits */
/* scan */
struct cfg80211_scan_request *scan_request;
@@ -686,6 +694,8 @@
/* High Access Latency Policy voting */
struct wil_halp halp;
+ enum wmi_ps_profile_type ps_profile;
+
#ifdef CONFIG_PM
#ifdef CONFIG_PM_SLEEP
struct notifier_block pm_notify;
@@ -764,6 +774,12 @@
print_hex_dump_debug("DBG[ WMI]" prefix_str,\
prefix_type, rowsize, \
groupsize, buf, len, ascii)
+
+#define wil_hex_dump_misc(prefix_str, prefix_type, rowsize, \
+ groupsize, buf, len, ascii) \
+ print_hex_dump_debug("DBG[MISC]" prefix_str,\
+ prefix_type, rowsize, \
+ groupsize, buf, len, ascii)
#else /* defined(CONFIG_DYNAMIC_DEBUG) */
static inline
void wil_hex_dump_txrx(const char *prefix_str, int prefix_type, int rowsize,
@@ -776,18 +792,18 @@
int groupsize, const void *buf, size_t len, bool ascii)
{
}
+
+static inline
+void wil_hex_dump_misc(const char *prefix_str, int prefix_type, int rowsize,
+ int groupsize, const void *buf, size_t len, bool ascii)
+{
+}
#endif /* defined(CONFIG_DYNAMIC_DEBUG) */
void wil_memcpy_fromio_32(void *dst, const volatile void __iomem *src,
size_t count);
void wil_memcpy_toio_32(volatile void __iomem *dst, const void *src,
size_t count);
-void wil_memcpy_fromio_halp_vote(struct wil6210_priv *wil, void *dst,
- const volatile void __iomem *src,
- size_t count);
-void wil_memcpy_toio_halp_vote(struct wil6210_priv *wil,
- volatile void __iomem *dst,
- const void *src, size_t count);
void *wil_if_alloc(struct device *dev);
void wil_if_free(struct wil6210_priv *wil);
@@ -795,6 +811,8 @@
void wil_if_remove(struct wil6210_priv *wil);
int wil_priv_init(struct wil6210_priv *wil);
void wil_priv_deinit(struct wil6210_priv *wil);
+int wil_ps_update(struct wil6210_priv *wil,
+ enum wmi_ps_profile_type ps_profile);
int wil_reset(struct wil6210_priv *wil, bool no_fw);
void wil_fw_error_recovery(struct wil6210_priv *wil);
void wil_set_recovery_state(struct wil6210_priv *wil, int state);
@@ -899,7 +917,7 @@
int wmi_led_cfg(struct wil6210_priv *wil, bool enable);
int wmi_abort_scan(struct wil6210_priv *wil);
void wil_abort_scan(struct wil6210_priv *wil, bool sync);
-
+void wil6210_bus_request(struct wil6210_priv *wil, u32 kbps);
void wil6210_disconnect(struct wil6210_priv *wil, const u8 *bssid,
u16 reason_code, bool from_event);
void wil_probe_client_flush(struct wil6210_priv *wil);
diff --git a/drivers/net/wireless/ath/wil6210/wmi.c b/drivers/net/wireless/ath/wil6210/wmi.c
index 1f22c19..814c356 100644
--- a/drivers/net/wireless/ath/wil6210/wmi.c
+++ b/drivers/net/wireless/ath/wil6210/wmi.c
@@ -518,16 +518,16 @@
assoc_resp_ielen = 0;
}
- mutex_lock(&wil->mutex);
if (test_bit(wil_status_resetting, wil->status) ||
!test_bit(wil_status_fwready, wil->status)) {
wil_err(wil, "status_resetting, cancel connect event, CID %d\n",
evt->cid);
- mutex_unlock(&wil->mutex);
/* no need for cleanup, wil_reset will do that */
return;
}
+ mutex_lock(&wil->mutex);
+
if ((wdev->iftype == NL80211_IFTYPE_STATION) ||
(wdev->iftype == NL80211_IFTYPE_P2P_CLIENT)) {
if (!test_bit(wil_status_fwconnecting, wil->status)) {
@@ -565,6 +565,7 @@
(wdev->iftype == NL80211_IFTYPE_P2P_CLIENT)) {
if (rc) {
netif_carrier_off(ndev);
+ wil6210_bus_request(wil, WIL_DEFAULT_BUS_REQUEST_KBPS);
wil_err(wil, "cfg80211_connect_result with failure\n");
cfg80211_connect_result(ndev, evt->bssid, NULL, 0,
NULL, 0,
@@ -572,12 +573,16 @@
GFP_KERNEL);
goto out;
} else {
- cfg80211_connect_result(ndev, evt->bssid,
- assoc_req_ie, assoc_req_ielen,
- assoc_resp_ie, assoc_resp_ielen,
- WLAN_STATUS_SUCCESS,
- GFP_KERNEL);
+ struct wiphy *wiphy = wil_to_wiphy(wil);
+
+ cfg80211_ref_bss(wiphy, wil->bss);
+ cfg80211_connect_bss(ndev, evt->bssid, wil->bss,
+ assoc_req_ie, assoc_req_ielen,
+ assoc_resp_ie, assoc_resp_ielen,
+ WLAN_STATUS_SUCCESS, GFP_KERNEL,
+ NL80211_TIMEOUT_UNSPECIFIED);
}
+ wil->bss = NULL;
} else if ((wdev->iftype == NL80211_IFTYPE_AP) ||
(wdev->iftype == NL80211_IFTYPE_P2P_GO)) {
if (rc) {
@@ -626,6 +631,13 @@
wil->sinfo_gen++;
+ if (test_bit(wil_status_resetting, wil->status) ||
+ !test_bit(wil_status_fwready, wil->status)) {
+ wil_err(wil, "status_resetting, cancel disconnect event\n");
+ /* no need for cleanup, wil_reset will do that */
+ return;
+ }
+
mutex_lock(&wil->mutex);
wil6210_disconnect(wil, evt->bssid, reason_code, true);
mutex_unlock(&wil->mutex);
@@ -1393,7 +1405,8 @@
struct wmi_cfg_rx_chain_cmd cmd = {
.action = WMI_RX_CHAIN_ADD,
.rx_sw_ring = {
- .max_mpdu_size = cpu_to_le16(wil_mtu2macbuf(mtu_max)),
+ .max_mpdu_size = cpu_to_le16(
+ wil_mtu2macbuf(wil->rx_buf_len)),
.ring_mem_base = cpu_to_le64(vring->pa),
.ring_size = cpu_to_le16(vring->size),
},
@@ -1492,6 +1505,7 @@
wil_dbg_wmi(wil, "disconnect_sta: (%pM, reason %d)\n", mac, reason);
+ wil->locally_generated_disc = true;
if (del_sta) {
ether_addr_copy(del_sta_cmd.dst_mac, mac);
rc = wmi_call(wil, WMI_DEL_STA_CMDID, &del_sta_cmd,
@@ -1733,14 +1747,19 @@
void wmi_event_flush(struct wil6210_priv *wil)
{
+ ulong flags;
struct pending_wmi_event *evt, *t;
wil_dbg_wmi(wil, "event_flush\n");
+ spin_lock_irqsave(&wil->wmi_ev_lock, flags);
+
list_for_each_entry_safe(evt, t, &wil->pending_wmi_ev, list) {
list_del(&evt->list);
kfree(evt);
}
+
+ spin_unlock_irqrestore(&wil->wmi_ev_lock, flags);
}
static bool wmi_evt_call_handler(struct wil6210_priv *wil, int id,
diff --git a/drivers/net/wireless/ath/wil6210/wmi.h b/drivers/net/wireless/ath/wil6210/wmi.h
index 7c9fee5..f7f5f4f 100644
--- a/drivers/net/wireless/ath/wil6210/wmi.h
+++ b/drivers/net/wireless/ath/wil6210/wmi.h
@@ -58,6 +58,7 @@
WMI_FW_CAPABILITY_MGMT_RETRY_LIMIT = 3,
WMI_FW_CAPABILITY_DISABLE_AP_SME = 4,
WMI_FW_CAPABILITY_WMI_ONLY = 5,
+ WMI_FW_CAPABILITY_THERMAL_THROTTLING = 7,
WMI_FW_CAPABILITY_MAX,
};
@@ -142,8 +143,6 @@
WMI_MAINTAIN_RESUME_CMDID = 0x851,
WMI_RS_MGMT_CMDID = 0x852,
WMI_RF_MGMT_CMDID = 0x853,
- WMI_THERMAL_THROTTLING_CTRL_CMDID = 0x854,
- WMI_THERMAL_THROTTLING_GET_STATUS_CMDID = 0x855,
WMI_OTP_READ_CMDID = 0x856,
WMI_OTP_WRITE_CMDID = 0x857,
WMI_LED_CFG_CMDID = 0x858,
@@ -192,6 +191,8 @@
WMI_GET_MGMT_RETRY_LIMIT_CMDID = 0x931,
WMI_NEW_STA_CMDID = 0x935,
WMI_DEL_STA_CMDID = 0x936,
+ WMI_SET_THERMAL_THROTTLING_CFG_CMDID = 0x940,
+ WMI_GET_THERMAL_THROTTLING_CFG_CMDID = 0x941,
WMI_TOF_SESSION_START_CMDID = 0x991,
WMI_TOF_GET_CAPABILITIES_CMDID = 0x992,
WMI_TOF_SET_LCR_CMDID = 0x993,
@@ -438,16 +439,6 @@
__le32 rf_mgmt_type;
} __packed;
-/* WMI_THERMAL_THROTTLING_CTRL_CMDID */
-#define THERMAL_THROTTLING_USE_DEFAULT_MAX_TXOP_LENGTH (0xFFFFFFFF)
-
-/* WMI_THERMAL_THROTTLING_CTRL_CMDID */
-struct wmi_thermal_throttling_ctrl_cmd {
- __le32 time_on_usec;
- __le32 time_off_usec;
- __le32 max_txop_length_usec;
-} __packed;
-
/* WMI_RF_RX_TEST_CMDID */
struct wmi_rf_rx_test_cmd {
__le32 sector;
@@ -549,7 +540,7 @@
u8 hidden_ssid;
u8 is_go;
u8 reserved0[5];
- /* abft_len override if non-0 */
+ /* A-BFT length override if non-0 */
u8 abft_len;
u8 disable_ap_sme;
u8 network_type;
@@ -910,6 +901,39 @@
u8 reserved[3];
} __packed;
+/* Zones: HIGH, MAX, CRITICAL */
+#define WMI_NUM_OF_TT_ZONES (3)
+
+struct wmi_tt_zone_limits {
+ /* Above this temperature this zone is active */
+ u8 temperature_high;
+ /* Below this temperature the adjacent lower zone is active */
+ u8 temperature_low;
+ u8 reserved[2];
+} __packed;
+
+/* Struct used for both configuration and status commands of thermal
+ * throttling
+ */
+struct wmi_tt_data {
+ /* Enable/Disable TT algorithm for baseband */
+ u8 bb_enabled;
+ u8 reserved0[3];
+ /* Define zones for baseband */
+ struct wmi_tt_zone_limits bb_zones[WMI_NUM_OF_TT_ZONES];
+ /* Enable/Disable TT algorithm for radio */
+ u8 rf_enabled;
+ u8 reserved1[3];
+ /* Define zones for all radio chips */
+ struct wmi_tt_zone_limits rf_zones[WMI_NUM_OF_TT_ZONES];
+} __packed;
+
+/* WMI_SET_THERMAL_THROTTLING_CFG_CMDID */
+struct wmi_set_thermal_throttling_cfg_cmd {
+ /* Command data */
+ struct wmi_tt_data tt_data;
+} __packed;
+
/* WMI_NEW_STA_CMDID */
struct wmi_new_sta_cmd {
u8 dst_mac[WMI_MAC_LEN];
@@ -1040,7 +1064,6 @@
WMI_BF_RXSS_MGMT_DONE_EVENTID = 0x1839,
WMI_RS_MGMT_DONE_EVENTID = 0x1852,
WMI_RF_MGMT_STATUS_EVENTID = 0x1853,
- WMI_THERMAL_THROTTLING_STATUS_EVENTID = 0x1855,
WMI_BF_SM_MGMT_DONE_EVENTID = 0x1838,
WMI_RX_MGMT_PACKET_EVENTID = 0x1840,
WMI_TX_MGMT_PACKET_EVENTID = 0x1841,
@@ -1090,6 +1113,8 @@
WMI_BRP_SET_ANT_LIMIT_EVENTID = 0x1924,
WMI_SET_MGMT_RETRY_LIMIT_EVENTID = 0x1930,
WMI_GET_MGMT_RETRY_LIMIT_EVENTID = 0x1931,
+ WMI_SET_THERMAL_THROTTLING_CFG_EVENTID = 0x1940,
+ WMI_GET_THERMAL_THROTTLING_CFG_EVENTID = 0x1941,
WMI_TOF_SESSION_END_EVENTID = 0x1991,
WMI_TOF_GET_CAPABILITIES_EVENTID = 0x1992,
WMI_TOF_SET_LCR_EVENTID = 0x1993,
@@ -1133,13 +1158,6 @@
__le32 rf_status;
} __packed;
-/* WMI_THERMAL_THROTTLING_STATUS_EVENTID */
-struct wmi_thermal_throttling_status_event {
- __le32 time_on_usec;
- __le32 time_off_usec;
- __le32 max_txop_length_usec;
-} __packed;
-
/* WMI_GET_STATUS_DONE_EVENTID */
struct wmi_get_status_done_event {
__le32 is_associated;
@@ -2206,6 +2224,19 @@
__le32 aoa_supported_types;
} __packed;
+/* WMI_SET_THERMAL_THROTTLING_CFG_EVENTID */
+struct wmi_set_thermal_throttling_cfg_event {
+ /* wmi_fw_status */
+ u8 status;
+ u8 reserved[3];
+} __packed;
+
+/* WMI_GET_THERMAL_THROTTLING_CFG_EVENTID */
+struct wmi_get_thermal_throttling_cfg_event {
+ /* Status data */
+ struct wmi_tt_data tt_data;
+} __packed;
+
enum wmi_tof_session_end_status {
WMI_TOF_SESSION_END_NO_ERROR = 0x00,
WMI_TOF_SESSION_END_FAIL = 0x01,
diff --git a/drivers/net/wireless/atmel/at76c50x-usb.c b/drivers/net/wireless/atmel/at76c50x-usb.c
index 0e18067..09defbc 100644
--- a/drivers/net/wireless/atmel/at76c50x-usb.c
+++ b/drivers/net/wireless/atmel/at76c50x-usb.c
@@ -2377,6 +2377,8 @@
wiphy->hw_version = priv->board_type;
+ wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
ret = ieee80211_register_hw(priv->hw);
if (ret) {
printk(KERN_ERR "cannot register mac80211 hw (status %d)!\n",
diff --git a/drivers/net/wireless/atmel/atmel.c b/drivers/net/wireless/atmel/atmel.c
index e12f623..27b110d 100644
--- a/drivers/net/wireless/atmel/atmel.c
+++ b/drivers/net/wireless/atmel/atmel.c
@@ -513,7 +513,7 @@
} station_state;
int operating_mode, power_mode;
- time_t last_qual;
+ unsigned long last_qual;
int beacons_this_sec;
int channel;
int reg_domain, config_reg_domain;
diff --git a/drivers/net/wireless/broadcom/b43/main.c b/drivers/net/wireless/broadcom/b43/main.c
index 52f3541..d23aac7 100644
--- a/drivers/net/wireless/broadcom/b43/main.c
+++ b/drivers/net/wireless/broadcom/b43/main.c
@@ -5598,6 +5598,8 @@
hw->wiphy->flags |= WIPHY_FLAG_IBSS_RSN;
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
wl->hw_registred = false;
hw->max_rates = 2;
SET_IEEE80211_DEV(hw, dev->dev);
diff --git a/drivers/net/wireless/broadcom/b43/xmit.c b/drivers/net/wireless/broadcom/b43/xmit.c
index b068d5a..1b9c191 100644
--- a/drivers/net/wireless/broadcom/b43/xmit.c
+++ b/drivers/net/wireless/broadcom/b43/xmit.c
@@ -694,7 +694,7 @@
if (unlikely(phystat0 & (B43_RX_PHYST0_PLCPHCF | B43_RX_PHYST0_PLCPFV)))
status.flag |= RX_FLAG_FAILED_PLCP_CRC;
if (phystat0 & B43_RX_PHYST0_SHORTPRMBL)
- status.flag |= RX_FLAG_SHORTPRE;
+ status.enc_flags |= RX_ENC_FLAG_SHORTPRE;
if (macstat & B43_RX_MAC_DECERR) {
/* Decryption with the given key failed.
* Drop the packet. We also won't be able to decrypt it with
diff --git a/drivers/net/wireless/broadcom/b43legacy/main.c b/drivers/net/wireless/broadcom/b43legacy/main.c
index cdafebb..f1e3dad 100644
--- a/drivers/net/wireless/broadcom/b43legacy/main.c
+++ b/drivers/net/wireless/broadcom/b43legacy/main.c
@@ -3850,6 +3850,8 @@
else
SET_IEEE80211_PERM_ADDR(hw, sprom->il0mac);
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
/* Get and initialize struct b43legacy_wl */
wl = hw_to_b43legacy_wl(hw);
memset(wl, 0, sizeof(*wl));
diff --git a/drivers/net/wireless/broadcom/brcm80211/Kconfig b/drivers/net/wireless/broadcom/brcm80211/Kconfig
index ab42b1f..9d99eb4 100644
--- a/drivers/net/wireless/broadcom/brcm80211/Kconfig
+++ b/drivers/net/wireless/broadcom/brcm80211/Kconfig
@@ -18,14 +18,14 @@
module, the driver will be called brcmsmac.ko.
config BRCMFMAC
- tristate "Broadcom IEEE802.11n embedded FullMAC WLAN driver"
+ tristate "Broadcom FullMAC WLAN driver"
depends on CFG80211
select BRCMUTIL
---help---
- This module adds support for embedded wireless adapters based on
- Broadcom IEEE802.11n FullMAC chipsets. It has to work with at least
- one of the bus interface support. If you choose to build a module,
- it'll be called brcmfmac.ko.
+ This module adds support for wireless adapters based on Broadcom
+ FullMAC chipsets. It has to work with at least one of the bus
+ interface support. If you choose to build a module, it'll be called
+ brcmfmac.ko.
config BRCMFMAC_PROTO_BCDC
bool
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/Makefile b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/Makefile
index 0383ba5..1f5a9b9 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/Makefile
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/Makefile
@@ -25,7 +25,6 @@
chip.o \
fwil.o \
fweh.o \
- fwsignal.o \
p2p.o \
proto.o \
common.o \
@@ -36,7 +35,8 @@
vendor.o \
pno.o
brcmfmac-$(CONFIG_BRCMFMAC_PROTO_BCDC) += \
- bcdc.o
+ bcdc.o \
+ fwsignal.o
brcmfmac-$(CONFIG_BRCMFMAC_PROTO_MSGBUF) += \
commonring.o \
flowring.o \
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.c
index 384b187..9f2d0b0 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.c
@@ -103,9 +103,17 @@
u8 bus_header[BUS_HEADER_LEN];
struct brcmf_proto_bcdc_dcmd msg;
unsigned char buf[BRCMF_DCMD_MAXLEN];
+ struct brcmf_fws_info *fws;
};
+struct brcmf_fws_info *drvr_to_fws(struct brcmf_pub *drvr)
+{
+ struct brcmf_bcdc *bcdc = drvr->proto->pd;
+
+ return bcdc->fws;
+}
+
static int
brcmf_proto_bcdc_msg(struct brcmf_pub *drvr, int ifidx, uint cmd, void *buf,
uint len, bool set)
@@ -330,8 +338,9 @@
struct sk_buff *skb)
{
struct brcmf_if *ifp = brcmf_get_ifp(drvr, ifidx);
+ struct brcmf_bcdc *bcdc = drvr->proto->pd;
- if (!brcmf_fws_queue_skbs(drvr->fws))
+ if (!brcmf_fws_queue_skbs(bcdc->fws))
return brcmf_proto_txdata(drvr, ifidx, 0, skb);
return brcmf_fws_process_skb(ifp, skb);
@@ -345,6 +354,36 @@
return brcmf_bus_txdata(drvr->bus_if, pktbuf);
}
+void brcmf_proto_bcdc_txflowblock(struct device *dev, bool state)
+{
+ struct brcmf_bus *bus_if = dev_get_drvdata(dev);
+ struct brcmf_pub *drvr = bus_if->drvr;
+
+ brcmf_dbg(TRACE, "Enter\n");
+
+ brcmf_fws_bus_blocked(drvr, state);
+}
+
+void
+brcmf_proto_bcdc_txcomplete(struct device *dev, struct sk_buff *txp,
+ bool success)
+{
+ struct brcmf_bus *bus_if = dev_get_drvdata(dev);
+ struct brcmf_bcdc *bcdc = bus_if->drvr->proto->pd;
+ struct brcmf_if *ifp;
+
+ /* await txstatus signal for firmware if active */
+ if (brcmf_fws_fc_active(bcdc->fws)) {
+ if (!success)
+ brcmf_fws_bustxfail(bcdc->fws, txp);
+ } else {
+ if (brcmf_proto_bcdc_hdrpull(bus_if->drvr, false, txp, &ifp))
+ brcmu_pkt_buf_free_skb(txp);
+ else
+ brcmf_txfinalize(ifp, txp, success);
+ }
+}
+
static void
brcmf_proto_bcdc_configure_addr_mode(struct brcmf_pub *drvr, int ifidx,
enum proto_addr_mode addr_mode)
@@ -369,6 +408,38 @@
brcmf_fws_rxreorder(ifp, skb);
}
+static void
+brcmf_proto_bcdc_add_if(struct brcmf_if *ifp)
+{
+ brcmf_fws_add_interface(ifp);
+}
+
+static void
+brcmf_proto_bcdc_del_if(struct brcmf_if *ifp)
+{
+ brcmf_fws_del_interface(ifp);
+}
+
+static void
+brcmf_proto_bcdc_reset_if(struct brcmf_if *ifp)
+{
+ brcmf_fws_reset_interface(ifp);
+}
+
+static int
+brcmf_proto_bcdc_init_done(struct brcmf_pub *drvr)
+{
+ struct brcmf_bcdc *bcdc = drvr->proto->pd;
+ struct brcmf_fws_info *fws;
+
+ fws = brcmf_fws_attach(drvr);
+ if (IS_ERR(fws))
+ return PTR_ERR(fws);
+
+ bcdc->fws = fws;
+ return 0;
+}
+
int brcmf_proto_bcdc_attach(struct brcmf_pub *drvr)
{
struct brcmf_bcdc *bcdc;
@@ -392,6 +463,10 @@
drvr->proto->delete_peer = brcmf_proto_bcdc_delete_peer;
drvr->proto->add_tdls_peer = brcmf_proto_bcdc_add_tdls_peer;
drvr->proto->rxreorder = brcmf_proto_bcdc_rxreorder;
+ drvr->proto->add_if = brcmf_proto_bcdc_add_if;
+ drvr->proto->del_if = brcmf_proto_bcdc_del_if;
+ drvr->proto->reset_if = brcmf_proto_bcdc_reset_if;
+ drvr->proto->init_done = brcmf_proto_bcdc_init_done;
drvr->proto->pd = bcdc;
drvr->hdrlen += BCDC_HEADER_LEN + BRCMF_PROT_FW_SIGNAL_MAX_TXBYTES;
@@ -406,6 +481,9 @@
void brcmf_proto_bcdc_detach(struct brcmf_pub *drvr)
{
- kfree(drvr->proto->pd);
+ struct brcmf_bcdc *bcdc = drvr->proto->pd;
+
drvr->proto->pd = NULL;
+ brcmf_fws_detach(bcdc->fws);
+ kfree(bcdc);
}
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.h
index 6003179..3b0e9ef 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.h
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcdc.h
@@ -19,6 +19,10 @@
#ifdef CONFIG_BRCMFMAC_PROTO_BCDC
int brcmf_proto_bcdc_attach(struct brcmf_pub *drvr);
void brcmf_proto_bcdc_detach(struct brcmf_pub *drvr);
+void brcmf_proto_bcdc_txflowblock(struct device *dev, bool state);
+void brcmf_proto_bcdc_txcomplete(struct device *dev, struct sk_buff *txp,
+ bool success);
+struct brcmf_fws_info *drvr_to_fws(struct brcmf_pub *drvr);
#else
static inline int brcmf_proto_bcdc_attach(struct brcmf_pub *drvr) { return 0; }
static inline void brcmf_proto_bcdc_detach(struct brcmf_pub *drvr) {}
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c
index 5bc2ba2..9b970dc 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c
@@ -21,6 +21,7 @@
#include <linux/pci_ids.h>
#include <linux/sched.h>
#include <linux/completion.h>
+#include <linux/interrupt.h>
#include <linux/scatterlist.h>
#include <linux/mmc/sdio.h>
#include <linux/mmc/core.h>
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bus.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bus.h
index 76693df..b55c329 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bus.h
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bus.h
@@ -229,11 +229,6 @@
void brcmf_detach(struct device *dev);
/* Indication from bus module that dongle should be reset */
void brcmf_dev_reset(struct device *dev);
-/* Indication from bus module to change flow-control state */
-void brcmf_txflowblock(struct device *dev, bool state);
-
-/* Notify the bus has transferred the tx packet to firmware */
-void brcmf_txcomplete(struct device *dev, struct sk_buff *txp, bool success);
/* Configure the "global" bus state used by upper layers */
void brcmf_bus_change_state(struct brcmf_bus *bus, enum brcmf_bus_state state);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c
index 944b83c..cd1d673 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c
@@ -575,12 +575,11 @@
*
* @wiphy: wiphy device of new interface.
* @name: name of the new interface.
- * @flags: not used.
* @params: contains mac address for AP device.
*/
static
struct wireless_dev *brcmf_ap_add_vif(struct wiphy *wiphy, const char *name,
- u32 *flags, struct vif_params *params)
+ struct vif_params *params)
{
struct brcmf_cfg80211_info *cfg = wiphy_to_cfg(wiphy);
struct brcmf_if *ifp = netdev_priv(cfg_to_ndev(cfg));
@@ -653,7 +652,6 @@
const char *name,
unsigned char name_assign_type,
enum nl80211_iftype type,
- u32 *flags,
struct vif_params *params)
{
struct wireless_dev *wdev;
@@ -674,12 +672,12 @@
case NL80211_IFTYPE_MESH_POINT:
return ERR_PTR(-EOPNOTSUPP);
case NL80211_IFTYPE_AP:
- wdev = brcmf_ap_add_vif(wiphy, name, flags, params);
+ wdev = brcmf_ap_add_vif(wiphy, name, params);
break;
case NL80211_IFTYPE_P2P_CLIENT:
case NL80211_IFTYPE_P2P_GO:
case NL80211_IFTYPE_P2P_DEVICE:
- wdev = brcmf_p2p_add_vif(wiphy, name, name_assign_type, type, flags, params);
+ wdev = brcmf_p2p_add_vif(wiphy, name, name_assign_type, type, params);
break;
case NL80211_IFTYPE_UNSPECIFIED:
default:
@@ -764,7 +762,7 @@
brcmf_dbg(SCAN, "scheduled scan completed\n");
cfg->internal_escan = false;
if (!aborted)
- cfg80211_sched_scan_results(cfg_to_wiphy(cfg));
+ cfg80211_sched_scan_results(cfg_to_wiphy(cfg), 0);
} else if (scan_request) {
struct cfg80211_scan_info info = {
.aborted = aborted,
@@ -858,7 +856,7 @@
static s32
brcmf_cfg80211_change_iface(struct wiphy *wiphy, struct net_device *ndev,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct brcmf_cfg80211_info *cfg = wiphy_priv(wiphy);
@@ -3097,6 +3095,9 @@
status = e->status;
+ if (status == BRCMF_E_STATUS_ABORT)
+ goto exit;
+
if (!test_bit(BRCMF_SCAN_STATUS_BUSY, &cfg->scan_status)) {
brcmf_err("scan not ready, bsscfgidx=%d\n", ifp->bsscfgidx);
return -EPERM;
@@ -3213,7 +3214,7 @@
{
struct ieee80211_channel *chan;
enum nl80211_band band;
- int freq;
+ int freq, i;
if (channel <= CH_MAX_2G_CHANNEL)
band = NL80211_BAND_2GHZ;
@@ -3228,10 +3229,22 @@
if (!chan)
return -EINVAL;
- req->channels[req->n_channels++] = chan;
- memcpy(req->ssids[req->n_ssids].ssid, ssid, ssid_len);
- req->ssids[req->n_ssids++].ssid_len = ssid_len;
+ for (i = 0; i < req->n_channels; i++) {
+ if (req->channels[i] == chan)
+ break;
+ }
+ if (i == req->n_channels)
+ req->channels[req->n_channels++] = chan;
+ for (i = 0; i < req->n_ssids; i++) {
+ if (req->ssids[i].ssid_len == ssid_len &&
+ !memcmp(req->ssids[i].ssid, ssid, ssid_len))
+ break;
+ }
+ if (i == req->n_ssids) {
+ memcpy(req->ssids[req->n_ssids].ssid, ssid, ssid_len);
+ req->ssids[req->n_ssids++].ssid_len = ssid_len;
+ }
return 0;
}
@@ -3297,6 +3310,7 @@
struct brcmf_pno_scanresults_le *pfn_result;
u32 result_count;
u32 status;
+ u32 datalen;
brcmf_dbg(SCAN, "Enter\n");
@@ -3323,6 +3337,14 @@
brcmf_err("FALSE PNO Event. (pfn_count == 0)\n");
goto out_err;
}
+
+ netinfo_start = brcmf_get_netinfo_array(pfn_result);
+ datalen = e->datalen - ((void *)netinfo_start - (void *)pfn_result);
+ if (datalen < result_count * sizeof(*netinfo)) {
+ brcmf_err("insufficient event data\n");
+ goto out_err;
+ }
+
request = brcmf_alloc_internal_escan_request(wiphy,
result_count);
if (!request) {
@@ -3330,17 +3352,11 @@
goto out_err;
}
- netinfo_start = brcmf_get_netinfo_array(pfn_result);
-
for (i = 0; i < result_count; i++) {
netinfo = &netinfo_start[i];
- if (!netinfo) {
- brcmf_err("Invalid netinfo ptr. index: %d\n",
- i);
- err = -EINVAL;
- goto out_err;
- }
+ if (netinfo->SSID_len > IEEE80211_MAX_SSID_LEN)
+ netinfo->SSID_len = IEEE80211_MAX_SSID_LEN;
brcmf_dbg(SCAN, "SSID:%.32s Channel:%d\n",
netinfo->SSID, netinfo->channel);
err = brcmf_internal_escan_add_info(request,
@@ -3356,7 +3372,7 @@
goto free_req;
out_err:
- cfg80211_sched_scan_stopped(wiphy);
+ cfg80211_sched_scan_stopped(wiphy, 0);
free_req:
kfree(request);
return err;
@@ -3389,7 +3405,7 @@
}
static int brcmf_cfg80211_sched_scan_stop(struct wiphy *wiphy,
- struct net_device *ndev)
+ struct net_device *ndev, u64 reqid)
{
struct brcmf_cfg80211_info *cfg = wiphy_to_cfg(wiphy);
struct brcmf_if *ifp = netdev_priv(ndev);
@@ -3591,7 +3607,7 @@
cfg->wowl.pre_pmmode);
cfg->wowl.active = false;
if (cfg->wowl.nd_enabled) {
- brcmf_cfg80211_sched_scan_stop(cfg->wiphy, ifp->ndev);
+ brcmf_cfg80211_sched_scan_stop(cfg->wiphy, ifp->ndev, 0);
brcmf_fweh_unregister(cfg->pub, BRCMF_E_PFN_NET_FOUND);
brcmf_fweh_register(cfg->pub, BRCMF_E_PFN_NET_FOUND,
brcmf_notify_sched_scan_results);
@@ -3675,7 +3691,7 @@
/* Stop scheduled scan */
if (brcmf_feat_is_enabled(ifp, BRCMF_FEAT_PNO))
- brcmf_cfg80211_sched_scan_stop(wiphy, ndev);
+ brcmf_cfg80211_sched_scan_stop(wiphy, ndev, 0);
/* end any scanning */
if (test_bit(BRCMF_SCAN_STATUS_BUSY, &cfg->scan_status))
@@ -5343,6 +5359,7 @@
struct ieee80211_supported_band *band;
struct brcmf_bss_info_le *bi;
struct brcmu_chan ch;
+ struct cfg80211_roam_info roam_info = {};
u32 freq;
s32 err = 0;
u8 *buf;
@@ -5381,9 +5398,15 @@
done:
kfree(buf);
- cfg80211_roamed(ndev, notify_channel, (u8 *)profile->bssid,
- conn_info->req_ie, conn_info->req_ie_len,
- conn_info->resp_ie, conn_info->resp_ie_len, GFP_KERNEL);
+
+ roam_info.channel = notify_channel;
+ roam_info.bssid = profile->bssid;
+ roam_info.req_ie = conn_info->req_ie;
+ roam_info.req_ie_len = conn_info->req_ie_len;
+ roam_info.resp_ie = conn_info->resp_ie;
+ roam_info.resp_ie_len = conn_info->resp_ie_len;
+
+ cfg80211_roamed(ndev, &roam_info, GFP_KERNEL);
brcmf_dbg(CONN, "Report roaming result\n");
set_bit(BRCMF_VIF_STATUS_CONNECTED, &ifp->vif->sme_state);
@@ -6358,11 +6381,11 @@
static void brcmf_wiphy_pno_params(struct wiphy *wiphy)
{
/* scheduled scan settings */
+ wiphy->max_sched_scan_reqs = 1;
wiphy->max_sched_scan_ssids = BRCMF_PNO_MAX_PFN_COUNT;
wiphy->max_match_sets = BRCMF_PNO_MAX_PFN_COUNT;
wiphy->max_sched_scan_ie_len = BRCMF_SCAN_IE_LEN_MAX;
wiphy->max_sched_scan_plan_interval = BRCMF_PNO_SCHED_SCAN_MAX_PERIOD;
- wiphy->flags |= WIPHY_FLAG_SUPPORTS_SCHED_SCAN;
}
#ifdef CONFIG_PM
@@ -6450,7 +6473,8 @@
BIT(NL80211_BSS_SELECT_ATTR_BAND_PREF) |
BIT(NL80211_BSS_SELECT_ATTR_RSSI_ADJUST);
- wiphy->flags |= WIPHY_FLAG_PS_ON_BY_DEFAULT |
+ wiphy->flags |= WIPHY_FLAG_NETNS_OK |
+ WIPHY_FLAG_PS_ON_BY_DEFAULT |
WIPHY_FLAG_OFFCHAN_TX |
WIPHY_FLAG_HAS_REMAIN_ON_CHANNEL;
if (brcmf_feat_is_enabled(ifp, BRCMF_FEAT_TDLS))
@@ -6549,7 +6573,7 @@
if (err)
goto default_conf_out;
err = brcmf_cfg80211_change_iface(wdev->wiphy, ndev, wdev->iftype,
- NULL, NULL);
+ NULL);
if (err)
goto default_conf_out;
@@ -6736,6 +6760,10 @@
s32 err;
int i;
+ /* The country code gets set to "00" by default at boot, ignore */
+ if (req->alpha2[0] == '0' && req->alpha2[1] == '0')
+ return;
+
/* ignore non-ISO3166 country codes */
for (i = 0; i < sizeof(req->alpha2); i++)
if (req->alpha2[i] < 'A' || req->alpha2[i] > 'Z') {
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/common.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/common.c
index 33b133f..7a2b495 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/common.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/common.c
@@ -161,7 +161,7 @@
strsep(&ptr, "\n");
/* Print fw version info */
- brcmf_err("Firmware version = %s\n", buf);
+ brcmf_info("Firmware version = %s\n", buf);
/* locate firmware version number for ethtool */
ptr = strrchr(buf, ' ') + 1;
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
index 60da86a..a3d8236 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
@@ -32,7 +32,6 @@
#include "p2p.h"
#include "cfg80211.h"
#include "fwil.h"
-#include "fwsignal.h"
#include "feature.h"
#include "proto.h"
#include "pcie.h"
@@ -198,7 +197,7 @@
int ret;
struct brcmf_if *ifp = netdev_priv(ndev);
struct brcmf_pub *drvr = ifp->drvr;
- struct ethhdr *eh = (struct ethhdr *)(skb->data);
+ struct ethhdr *eh;
brcmf_dbg(DATA, "Enter, bsscfgidx=%d\n", ifp->bsscfgidx);
@@ -211,22 +210,13 @@
goto done;
}
- /* Make sure there's enough room for any header */
- if (skb_headroom(skb) < drvr->hdrlen) {
- struct sk_buff *skb2;
-
- brcmf_dbg(INFO, "%s: insufficient headroom\n",
+ /* Make sure there's enough writable headroom*/
+ ret = skb_cow_head(skb, drvr->hdrlen);
+ if (ret < 0) {
+ brcmf_err("%s: skb_cow_head failed\n",
brcmf_ifname(ifp));
- drvr->bus_if->tx_realloc++;
- skb2 = skb_realloc_headroom(skb, drvr->hdrlen);
dev_kfree_skb(skb);
- skb = skb2;
- if (skb == NULL) {
- brcmf_err("%s: skb_realloc_headroom failed\n",
- brcmf_ifname(ifp));
- ret = -ENOMEM;
- goto done;
- }
+ goto done;
}
/* validate length for ether packet */
@@ -236,6 +226,8 @@
goto done;
}
+ eh = (struct ethhdr *)(skb->data);
+
if (eh->h_proto == htons(ETH_P_PAE))
atomic_inc(&ifp->pend_8021x_cnt);
@@ -283,16 +275,6 @@
spin_unlock_irqrestore(&ifp->netif_stop_lock, flags);
}
-void brcmf_txflowblock(struct device *dev, bool state)
-{
- struct brcmf_bus *bus_if = dev_get_drvdata(dev);
- struct brcmf_pub *drvr = bus_if->drvr;
-
- brcmf_dbg(TRACE, "Enter\n");
-
- brcmf_fws_bus_blocked(drvr, state);
-}
-
void brcmf_netif_rx(struct brcmf_if *ifp, struct sk_buff *skb)
{
if (skb->pkt_type == PACKET_MULTICAST)
@@ -393,24 +375,6 @@
brcmu_pkt_buf_free_skb(txp);
}
-void brcmf_txcomplete(struct device *dev, struct sk_buff *txp, bool success)
-{
- struct brcmf_bus *bus_if = dev_get_drvdata(dev);
- struct brcmf_pub *drvr = bus_if->drvr;
- struct brcmf_if *ifp;
-
- /* await txstatus signal for firmware if active */
- if (brcmf_fws_fc_active(drvr->fws)) {
- if (!success)
- brcmf_fws_bustxfail(drvr->fws, txp);
- } else {
- if (brcmf_proto_hdrpull(drvr, false, txp, &ifp))
- brcmu_pkt_buf_free_skb(txp);
- else
- brcmf_txfinalize(ifp, txp, success);
- }
-}
-
static void brcmf_ethtool_get_drvinfo(struct net_device *ndev,
struct ethtool_drvinfo *info)
{
@@ -504,8 +468,9 @@
ndev->needed_headroom += drvr->hdrlen;
ndev->ethtool_ops = &brcmf_ethtool_ops;
- /* set the mac address */
+ /* set the mac address & netns */
memcpy(ndev->dev_addr, ifp->mac_addr, ETH_ALEN);
+ dev_net_set(ndev, wiphy_net(cfg_to_wiphy(drvr->config)));
INIT_WORK(&ifp->multicast_work, _brcmf_set_multicast_list);
INIT_WORK(&ifp->ndoffload_work, _brcmf_update_ndtable);
@@ -734,10 +699,28 @@
return;
brcmf_dbg(TRACE, "Enter, bsscfgidx=%d, ifidx=%d\n", ifp->bsscfgidx,
ifp->ifidx);
- brcmf_fws_del_interface(ifp);
+ brcmf_proto_del_if(ifp->drvr, ifp);
brcmf_del_if(ifp->drvr, ifp->bsscfgidx, rtnl_locked);
}
+static int brcmf_psm_watchdog_notify(struct brcmf_if *ifp,
+ const struct brcmf_event_msg *evtmsg,
+ void *data)
+{
+ int err;
+
+ brcmf_dbg(TRACE, "enter: bsscfgidx=%d\n", ifp->bsscfgidx);
+
+ brcmf_err("PSM's watchdog has fired!\n");
+
+ err = brcmf_debug_create_memdump(ifp->drvr->bus_if, data,
+ evtmsg->datalen);
+ if (err)
+ brcmf_err("Failed to get memory dump, %d\n", err);
+
+ return err;
+}
+
#ifdef CONFIG_INET
#define ARPOL_MAX_ENTRIES 8
static int brcmf_inetaddr_changed(struct notifier_block *nb,
@@ -917,6 +900,10 @@
goto fail;
}
+ /* Attach to events important for core code */
+ brcmf_fweh_register(drvr, BRCMF_E_PSM_WATCHDOG,
+ brcmf_psm_watchdog_notify);
+
/* attach firmware event handler */
brcmf_fweh_attach(drvr);
@@ -992,11 +979,11 @@
}
brcmf_feat_attach(drvr);
- ret = brcmf_fws_init(drvr);
+ ret = brcmf_proto_init_done(drvr);
if (ret < 0)
goto fail;
- brcmf_fws_add_interface(ifp);
+ brcmf_proto_add_if(drvr, ifp);
drvr->config = brcmf_cfg80211_attach(drvr, bus_if->dev,
drvr->settings->p2p_enable);
@@ -1040,10 +1027,6 @@
brcmf_cfg80211_detach(drvr->config);
drvr->config = NULL;
}
- if (drvr->fws) {
- brcmf_fws_del_interface(ifp);
- brcmf_fws_deinit(drvr);
- }
brcmf_net_detach(ifp->ndev, false);
if (p2p_ifp)
brcmf_net_detach(p2p_ifp->ndev, false);
@@ -1109,8 +1092,6 @@
brcmf_cfg80211_detach(drvr->config);
- brcmf_fws_deinit(drvr);
-
brcmf_bus_stop(drvr->bus_if);
brcmf_proto_detach(drvr);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.h
index 6aecd8d..a4dd313 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.h
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.h
@@ -127,8 +127,6 @@
struct brcmf_fweh_info fweh;
- struct brcmf_fws_info *fws;
-
struct brcmf_ampdu_rx_reorder
*reorder_flows[BRCMF_AMPDU_RX_REORDER_MAXFLOWS];
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/debug.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/debug.c
index f4644cf..1447a83 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/debug.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/debug.c
@@ -27,8 +27,8 @@
static struct dentry *root_folder;
-static int brcmf_debug_create_memdump(struct brcmf_bus *bus, const void *data,
- size_t len)
+int brcmf_debug_create_memdump(struct brcmf_bus *bus, const void *data,
+ size_t len)
{
void *dump;
size_t ramsize;
@@ -54,24 +54,6 @@
return 0;
}
-static int brcmf_debug_psm_watchdog_notify(struct brcmf_if *ifp,
- const struct brcmf_event_msg *evtmsg,
- void *data)
-{
- int err;
-
- brcmf_dbg(TRACE, "enter: bsscfgidx=%d\n", ifp->bsscfgidx);
-
- brcmf_err("PSM's watchdog has fired!\n");
-
- err = brcmf_debug_create_memdump(ifp->drvr->bus_if, data,
- evtmsg->datalen);
- if (err)
- brcmf_err("Failed to get memory dump, %d\n", err);
-
- return err;
-}
-
void brcmf_debugfs_init(void)
{
root_folder = debugfs_create_dir(KBUILD_MODNAME, NULL);
@@ -99,9 +81,7 @@
if (IS_ERR(drvr->dbgfs_dir))
return PTR_ERR(drvr->dbgfs_dir);
-
- return brcmf_fweh_register(drvr, BRCMF_E_PSM_WATCHDOG,
- brcmf_debug_psm_watchdog_notify);
+ return 0;
}
void brcmf_debug_detach(struct brcmf_pub *drvr)
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/debug.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/debug.h
index 0661261..fe264a5 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/debug.h
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/debug.h
@@ -59,6 +59,10 @@
} while (0)
#if defined(DEBUG) || defined(CONFIG_BRCM_TRACING)
+
+/* For debug/tracing purposes treat info messages as errors */
+#define brcmf_info brcmf_err
+
__printf(3, 4)
void __brcmf_dbg(u32 level, const char *func, const char *fmt, ...);
#define brcmf_dbg(level, fmt, ...) \
@@ -77,6 +81,11 @@
#else /* defined(DEBUG) || defined(CONFIG_BRCM_TRACING) */
+#define brcmf_info(fmt, ...) \
+ do { \
+ pr_info("%s: " fmt, __func__, ##__VA_ARGS__); \
+ } while (0)
+
#define brcmf_dbg(level, fmt, ...) no_printk(fmt, ##__VA_ARGS__)
#define BRCMF_DATA_ON() 0
@@ -99,6 +108,7 @@
extern int brcmf_msg_level;
+struct brcmf_bus;
struct brcmf_pub;
#ifdef DEBUG
void brcmf_debugfs_init(void);
@@ -108,6 +118,8 @@
struct dentry *brcmf_debugfs_get_devdir(struct brcmf_pub *drvr);
int brcmf_debugfs_add_entry(struct brcmf_pub *drvr, const char *fn,
int (*read_fn)(struct seq_file *seq, void *data));
+int brcmf_debug_create_memdump(struct brcmf_bus *bus, const void *data,
+ size_t len);
#else
static inline void brcmf_debugfs_init(void)
{
@@ -128,6 +140,12 @@
{
return 0;
}
+static inline
+int brcmf_debug_create_memdump(struct brcmf_bus *bus, const void *data,
+ size_t len)
+{
+ return 0;
+}
#endif
#endif /* BRCMFMAC_DEBUG_H */
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fweh.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fweh.c
index c79306b..4eb1e1c 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fweh.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fweh.c
@@ -22,9 +22,9 @@
#include "core.h"
#include "debug.h"
#include "tracepoint.h"
-#include "fwsignal.h"
#include "fweh.h"
#include "fwil.h"
+#include "proto.h"
/**
* struct brcmf_fweh_queue_item - event item on event queue.
@@ -172,14 +172,14 @@
if (IS_ERR(ifp))
return;
if (!is_p2pdev)
- brcmf_fws_add_interface(ifp);
+ brcmf_proto_add_if(drvr, ifp);
if (!drvr->fweh.evt_handler[BRCMF_E_IF])
if (brcmf_net_attach(ifp, false) < 0)
return;
}
if (ifp && ifevent->action == BRCMF_E_IF_CHANGE)
- brcmf_fws_reset_interface(ifp);
+ brcmf_proto_reset_if(drvr, ifp);
err = brcmf_fweh_call_event_handler(ifp, emsg->event_code, emsg, data);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.c
index 5f1a592..72373e5 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.c
@@ -36,6 +36,7 @@
#include "p2p.h"
#include "cfg80211.h"
#include "proto.h"
+#include "bcdc.h"
#include "common.h"
/**
@@ -1586,7 +1587,7 @@
const struct brcmf_event_msg *e,
void *data)
{
- struct brcmf_fws_info *fws = ifp->drvr->fws;
+ struct brcmf_fws_info *fws = drvr_to_fws(ifp->drvr);
int i;
u8 *credits = data;
@@ -1617,7 +1618,7 @@
const struct brcmf_event_msg *e,
void *data)
{
- struct brcmf_fws_info *fws = ifp->drvr->fws;
+ struct brcmf_fws_info *fws = drvr_to_fws(ifp->drvr);
if (fws) {
brcmf_fws_lock(fws);
@@ -1826,7 +1827,7 @@
void brcmf_fws_hdrpull(struct brcmf_if *ifp, s16 siglen, struct sk_buff *skb)
{
struct brcmf_skb_reorder_data *rd;
- struct brcmf_fws_info *fws = ifp->drvr->fws;
+ struct brcmf_fws_info *fws = drvr_to_fws(ifp->drvr);
u8 *signal_data;
s16 data_len;
u8 type;
@@ -2091,8 +2092,7 @@
int brcmf_fws_process_skb(struct brcmf_if *ifp, struct sk_buff *skb)
{
- struct brcmf_pub *drvr = ifp->drvr;
- struct brcmf_fws_info *fws = drvr->fws;
+ struct brcmf_fws_info *fws = drvr_to_fws(ifp->drvr);
struct brcmf_skbuff_cb *skcb = brcmf_skbcb(skb);
struct ethhdr *eh = (struct ethhdr *)(skb->data);
int fifo = BRCMF_FWS_FIFO_BCMC;
@@ -2142,10 +2142,10 @@
void brcmf_fws_add_interface(struct brcmf_if *ifp)
{
- struct brcmf_fws_info *fws = ifp->drvr->fws;
+ struct brcmf_fws_info *fws = drvr_to_fws(ifp->drvr);
struct brcmf_fws_mac_descriptor *entry;
- if (!ifp->ndev)
+ if (!ifp->ndev || fws->fcmode == BRCMF_FWS_FCMODE_NONE)
return;
entry = &fws->desc.iface[ifp->ifidx];
@@ -2160,16 +2160,17 @@
void brcmf_fws_del_interface(struct brcmf_if *ifp)
{
struct brcmf_fws_mac_descriptor *entry = ifp->fws_desc;
+ struct brcmf_fws_info *fws = drvr_to_fws(ifp->drvr);
if (!entry)
return;
- brcmf_fws_lock(ifp->drvr->fws);
+ brcmf_fws_lock(fws);
ifp->fws_desc = NULL;
brcmf_dbg(TRACE, "deleting %s\n", entry->name);
brcmf_fws_macdesc_deinit(entry);
- brcmf_fws_cleanup(ifp->drvr->fws, ifp->ifidx);
- brcmf_fws_unlock(ifp->drvr->fws);
+ brcmf_fws_cleanup(fws, ifp->ifidx);
+ brcmf_fws_unlock(fws);
}
static void brcmf_fws_dequeue_worker(struct work_struct *worker)
@@ -2243,7 +2244,7 @@
static int brcmf_debugfs_fws_stats_read(struct seq_file *seq, void *data)
{
struct brcmf_bus *bus_if = dev_get_drvdata(seq->private);
- struct brcmf_fws_stats *fwstats = &bus_if->drvr->fws->stats;
+ struct brcmf_fws_stats *fwstats = &(drvr_to_fws(bus_if->drvr)->stats);
seq_printf(seq,
"header_pulls: %u\n"
@@ -2308,7 +2309,7 @@
}
#endif
-int brcmf_fws_init(struct brcmf_pub *drvr)
+struct brcmf_fws_info *brcmf_fws_attach(struct brcmf_pub *drvr)
{
struct brcmf_fws_info *fws;
struct brcmf_if *ifp;
@@ -2316,17 +2317,15 @@
int rc;
u32 mode;
- drvr->fws = kzalloc(sizeof(*(drvr->fws)), GFP_KERNEL);
- if (!drvr->fws) {
+ fws = kzalloc(sizeof(*fws), GFP_KERNEL);
+ if (!fws) {
rc = -ENOMEM;
goto fail;
}
- fws = drvr->fws;
-
spin_lock_init(&fws->spinlock);
- /* set linkage back */
+ /* store drvr reference */
fws->drvr = drvr;
fws->fcmode = drvr->settings->fcmode;
@@ -2334,7 +2333,7 @@
(fws->fcmode == BRCMF_FWS_FCMODE_NONE)) {
fws->avoid_queueing = true;
brcmf_dbg(INFO, "FWS queueing will be avoided\n");
- return 0;
+ return fws;
}
fws->fws_wq = create_singlethread_workqueue("brcmf_fws_wq");
@@ -2396,6 +2395,7 @@
brcmf_fws_hanger_init(&fws->hanger);
brcmf_fws_macdesc_init(&fws->desc.other, NULL, 0);
brcmf_fws_macdesc_set_name(fws, &fws->desc.other);
+ brcmf_dbg(INFO, "added %s\n", fws->desc.other.name);
brcmu_pktq_init(&fws->desc.other.psq, BRCMF_FWS_PSQ_PREC_COUNT,
BRCMF_FWS_PSQ_LEN);
@@ -2405,27 +2405,24 @@
brcmf_dbg(INFO, "%s bdcv2 tlv signaling [%x]\n",
fws->fw_signals ? "enabled" : "disabled", tlv);
- return 0;
+ return fws;
fail:
- brcmf_fws_deinit(drvr);
- return rc;
+ brcmf_fws_detach(fws);
+ return ERR_PTR(rc);
}
-void brcmf_fws_deinit(struct brcmf_pub *drvr)
+void brcmf_fws_detach(struct brcmf_fws_info *fws)
{
- struct brcmf_fws_info *fws = drvr->fws;
-
if (!fws)
return;
- if (drvr->fws->fws_wq)
- destroy_workqueue(drvr->fws->fws_wq);
+ if (fws->fws_wq)
+ destroy_workqueue(fws->fws_wq);
/* cleanup */
brcmf_fws_lock(fws);
brcmf_fws_cleanup(fws, -1);
- drvr->fws = NULL;
brcmf_fws_unlock(fws);
/* free top structure */
@@ -2461,7 +2458,7 @@
void brcmf_fws_bus_blocked(struct brcmf_pub *drvr, bool flow_blocked)
{
- struct brcmf_fws_info *fws = drvr->fws;
+ struct brcmf_fws_info *fws = drvr_to_fws(drvr);
struct brcmf_if *ifp;
int i;
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.h
index 96df660..ba07bd9 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.h
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/fwsignal.h
@@ -18,8 +18,8 @@
#ifndef FWSIGNAL_H_
#define FWSIGNAL_H_
-int brcmf_fws_init(struct brcmf_pub *drvr);
-void brcmf_fws_deinit(struct brcmf_pub *drvr);
+struct brcmf_fws_info *brcmf_fws_attach(struct brcmf_pub *drvr);
+void brcmf_fws_detach(struct brcmf_fws_info *fws);
bool brcmf_fws_queue_skbs(struct brcmf_fws_info *fws);
bool brcmf_fws_fc_active(struct brcmf_fws_info *fws);
void brcmf_fws_hdrpull(struct brcmf_if *ifp, s16 siglen, struct sk_buff *skb);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c
index 85d949e..aa299c4 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c
@@ -2141,12 +2141,11 @@
* @name: name of the new interface.
* @name_assign_type: origin of the interface name
* @type: nl80211 interface type.
- * @flags: not used.
* @params: contains mac address for P2P device.
*/
struct wireless_dev *brcmf_p2p_add_vif(struct wiphy *wiphy, const char *name,
unsigned char name_assign_type,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct brcmf_cfg80211_info *cfg = wiphy_to_cfg(wiphy);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.h
index 8ce9447..0e8b34d2 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.h
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.h
@@ -150,7 +150,7 @@
void brcmf_p2p_detach(struct brcmf_p2p_info *p2p);
struct wireless_dev *brcmf_p2p_add_vif(struct wiphy *wiphy, const char *name,
unsigned char name_assign_type,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params);
int brcmf_p2p_del_vif(struct wiphy *wiphy, struct wireless_dev *wdev);
int brcmf_p2p_ifchange(struct brcmf_cfg80211_info *cfg,
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c
index 6fae4cf..f36b96d 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c
@@ -1877,6 +1877,7 @@
BRCMF_PCIE_MBDATA_TIMEOUT);
if (!devinfo->mbdata_completed) {
brcmf_err("Timeout on response for entering D3 substate\n");
+ brcmf_bus_change_state(bus, BRCMF_BUS_UP);
return -EIO;
}
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/pno.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/pno.c
index 9a25e79..6c3bde8 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/pno.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/pno.c
@@ -182,7 +182,6 @@
int brcmf_pno_start_sched_scan(struct brcmf_if *ifp,
struct cfg80211_sched_scan_request *req)
{
- struct brcmu_d11inf *d11inf;
struct brcmf_pno_config_le pno_cfg;
struct cfg80211_ssid *ssid;
u16 chan;
@@ -209,7 +208,6 @@
}
/* configure channels to use */
- d11inf = &ifp->drvr->config->d11inf;
for (i = 0; i < req->n_channels; i++) {
chan = req->channels[i]->hw_value;
pno_cfg.channel_list[i] = cpu_to_le16(chan);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/proto.h b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/proto.h
index 34b59fe..2404f8a 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/proto.h
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/proto.h
@@ -44,6 +44,10 @@
void (*add_tdls_peer)(struct brcmf_pub *drvr, int ifidx,
u8 peer[ETH_ALEN]);
void (*rxreorder)(struct brcmf_if *ifp, struct sk_buff *skb);
+ void (*add_if)(struct brcmf_if *ifp);
+ void (*del_if)(struct brcmf_if *ifp);
+ void (*reset_if)(struct brcmf_if *ifp);
+ int (*init_done)(struct brcmf_pub *drvr);
void *pd;
};
@@ -118,4 +122,36 @@
ifp->drvr->proto->rxreorder(ifp, skb);
}
+static inline void
+brcmf_proto_add_if(struct brcmf_pub *drvr, struct brcmf_if *ifp)
+{
+ if (!drvr->proto->add_if)
+ return;
+ drvr->proto->add_if(ifp);
+}
+
+static inline void
+brcmf_proto_del_if(struct brcmf_pub *drvr, struct brcmf_if *ifp)
+{
+ if (!drvr->proto->del_if)
+ return;
+ drvr->proto->del_if(ifp);
+}
+
+static inline void
+brcmf_proto_reset_if(struct brcmf_pub *drvr, struct brcmf_if *ifp)
+{
+ if (!drvr->proto->reset_if)
+ return;
+ drvr->proto->reset_if(ifp);
+}
+
+static inline int
+brcmf_proto_init_done(struct brcmf_pub *drvr)
+{
+ if (!drvr->proto->init_done)
+ return 0;
+ return drvr->proto->init_done(drvr);
+}
+
#endif /* BRCMFMAC_PROTO_H */
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c
index 6568946..fc64b89 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c
@@ -44,6 +44,7 @@
#include "firmware.h"
#include "core.h"
#include "common.h"
+#include "bcdc.h"
#define DCMD_RESP_TIMEOUT msecs_to_jiffies(2500)
#define CTL_DONE_TIMEOUT msecs_to_jiffies(2500)
@@ -539,7 +540,11 @@
/* Limit on rounding up frames */
static const uint max_roundup = 512;
+#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
+#define ALIGNMENT 8
+#else
#define ALIGNMENT 4
+#endif
enum brcmf_sdio_frmtype {
BRCMF_SDIO_FT_NORMAL,
@@ -2265,7 +2270,8 @@
bus->tx_seq = (bus->tx_seq + pktq->qlen) % SDPCM_SEQ_WRAP;
skb_queue_walk_safe(pktq, pkt_next, tmp) {
__skb_unlink(pkt_next, pktq);
- brcmf_txcomplete(bus->sdiodev->dev, pkt_next, ret == 0);
+ brcmf_proto_bcdc_txcomplete(bus->sdiodev->dev, pkt_next,
+ ret == 0);
}
return ret;
}
@@ -2328,7 +2334,7 @@
if ((bus->sdiodev->state == BRCMF_SDIOD_DATA) &&
bus->txoff && (pktq_len(&bus->txq) < TXLOW)) {
bus->txoff = false;
- brcmf_txflowblock(bus->sdiodev->dev, false);
+ brcmf_proto_bcdc_txflowblock(bus->sdiodev->dev, false);
}
return cnt;
@@ -2753,7 +2759,7 @@
if (pktq_len(&bus->txq) >= TXHI) {
bus->txoff = true;
- brcmf_txflowblock(dev, true);
+ brcmf_proto_bcdc_txflowblock(dev, true);
}
spin_unlock_bh(&bus->txq_lock);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/usb.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/usb.c
index d93ebbd..e4d545f 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/usb.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/usb.c
@@ -29,6 +29,7 @@
#include "usb.h"
#include "core.h"
#include "common.h"
+#include "bcdc.h"
#define IOCTL_RESP_TIMEOUT msecs_to_jiffies(2000)
@@ -482,13 +483,13 @@
req->skb);
brcmf_usb_del_fromq(devinfo, req);
- brcmf_txcomplete(devinfo->dev, req->skb, urb->status == 0);
+ brcmf_proto_bcdc_txcomplete(devinfo->dev, req->skb, urb->status == 0);
req->skb = NULL;
brcmf_usb_enq(devinfo, &devinfo->tx_freeq, req, &devinfo->tx_freecount);
spin_lock_irqsave(&devinfo->tx_flowblock_lock, flags);
if (devinfo->tx_freecount > devinfo->tx_high_watermark &&
devinfo->tx_flowblock) {
- brcmf_txflowblock(devinfo->dev, false);
+ brcmf_proto_bcdc_txflowblock(devinfo->dev, false);
devinfo->tx_flowblock = false;
}
spin_unlock_irqrestore(&devinfo->tx_flowblock_lock, flags);
@@ -635,7 +636,7 @@
spin_lock_irqsave(&devinfo->tx_flowblock_lock, flags);
if (devinfo->tx_freecount < devinfo->tx_low_watermark &&
!devinfo->tx_flowblock) {
- brcmf_txflowblock(dev, true);
+ brcmf_proto_bcdc_txflowblock(dev, true);
devinfo->tx_flowblock = true;
}
spin_unlock_irqrestore(&devinfo->tx_flowblock_lock, flags);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmsmac/mac80211_if.c b/drivers/net/wireless/broadcom/brcm80211/brcmsmac/mac80211_if.c
index 7c2a9a9..ddfdfe1 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmsmac/mac80211_if.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmsmac/mac80211_if.c
@@ -1082,6 +1082,8 @@
* hw->wiphy->flags |= WIPHY_FLAG_AP_PROBE_RESP_OFFLOAD;
*/
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
hw->rate_control_algorithm = "minstrel_ht";
hw->sta_data_size = 0;
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmsmac/main.c b/drivers/net/wireless/broadcom/brcm80211/brcmsmac/main.c
index c2a938b..0a14942 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmsmac/main.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmsmac/main.c
@@ -7092,9 +7092,9 @@
rspec = brcms_c_compute_rspec(rxh, plcp);
if (is_mcs_rate(rspec)) {
rx_status->rate_idx = rspec & RSPEC_RATE_MASK;
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
if (rspec_is40mhz(rspec))
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
} else {
switch (rspec2rate(rspec)) {
case BRCM_RATE_1M:
@@ -7149,9 +7149,9 @@
/* Determine short preamble and rate_idx */
if (is_cck_rate(rspec)) {
if (rxh->PhyRxStatus_0 & PRXS0_SHORTH)
- rx_status->flag |= RX_FLAG_SHORTPRE;
+ rx_status->enc_flags |= RX_ENC_FLAG_SHORTPRE;
} else if (is_ofdm_rate(rspec)) {
- rx_status->flag |= RX_FLAG_SHORTPRE;
+ rx_status->enc_flags |= RX_ENC_FLAG_SHORTPRE;
} else {
brcms_err(wlc->hw->d11core, "%s: Unknown modulation\n",
__func__);
@@ -7159,7 +7159,7 @@
}
if (plcp3_issgi(plcp[3]))
- rx_status->flag |= RX_FLAG_SHORT_GI;
+ rx_status->enc_flags |= RX_ENC_FLAG_SHORT_GI;
if (rxh->RxStatus1 & RXS_DECERR) {
rx_status->flag |= RX_FLAG_FAILED_PLCP_CRC;
diff --git a/drivers/net/wireless/intel/ipw2x00/ipw2200.c b/drivers/net/wireless/intel/ipw2x00/ipw2200.c
index 5ef3c5c..bbc579b 100644
--- a/drivers/net/wireless/intel/ipw2x00/ipw2200.c
+++ b/drivers/net/wireless/intel/ipw2x00/ipw2200.c
@@ -3539,9 +3539,6 @@
fw_img = &fw->data[le32_to_cpu(fw->boot_size) +
le32_to_cpu(fw->ucode_size)];
- if (rc < 0)
- goto error;
-
if (!priv->rxq)
priv->rxq = ipw_rx_queue_alloc(priv);
else
diff --git a/drivers/net/wireless/intel/iwlegacy/3945-mac.c b/drivers/net/wireless/intel/iwlegacy/3945-mac.c
index e8e6511..38bf403 100644
--- a/drivers/net/wireless/intel/iwlegacy/3945-mac.c
+++ b/drivers/net/wireless/intel/iwlegacy/3945-mac.c
@@ -3592,6 +3592,8 @@
il_leds_init(il);
+ wiphy_ext_feature_set(il->hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
ret = ieee80211_register_hw(il->hw);
if (ret) {
IL_ERR("Failed to register hw (error %d)\n", ret);
diff --git a/drivers/net/wireless/intel/iwlegacy/3945-rs.c b/drivers/net/wireless/intel/iwlegacy/3945-rs.c
index 03ad9b8..b2f35df 100644
--- a/drivers/net/wireless/intel/iwlegacy/3945-rs.c
+++ b/drivers/net/wireless/intel/iwlegacy/3945-rs.c
@@ -656,7 +656,7 @@
rate_mask = sta->supp_rates[sband->band];
/* get user max rate if set */
- max_rate_idx = txrc->max_rate_idx;
+ max_rate_idx = fls(txrc->rate_idx_mask) - 1;
if (sband->band == NL80211_BAND_5GHZ && max_rate_idx != -1)
max_rate_idx += IL_FIRST_OFDM_RATE;
if (max_rate_idx < 0 || max_rate_idx >= RATE_COUNT)
diff --git a/drivers/net/wireless/intel/iwlegacy/3945.c b/drivers/net/wireless/intel/iwlegacy/3945.c
index 4db327a..080ea81 100644
--- a/drivers/net/wireless/intel/iwlegacy/3945.c
+++ b/drivers/net/wireless/intel/iwlegacy/3945.c
@@ -570,7 +570,7 @@
/* set the preamble flag if appropriate */
if (rx_hdr->phy_flags & RX_RES_PHY_FLAGS_SHORT_PREAMBLE_MSK)
- rx_status.flag |= RX_FLAG_SHORTPRE;
+ rx_status.enc_flags |= RX_ENC_FLAG_SHORTPRE;
if ((unlikely(rx_stats->phy_count > 20))) {
D_DROP("dsp size out of range [0,20]: %d\n",
diff --git a/drivers/net/wireless/intel/iwlegacy/4965-mac.c b/drivers/net/wireless/intel/iwlegacy/4965-mac.c
index 2781f57..5d5faa3 100644
--- a/drivers/net/wireless/intel/iwlegacy/4965-mac.c
+++ b/drivers/net/wireless/intel/iwlegacy/4965-mac.c
@@ -728,15 +728,15 @@
/* set the preamble flag if appropriate */
if (phy_res->phy_flags & RX_RES_PHY_FLAGS_SHORT_PREAMBLE_MSK)
- rx_status.flag |= RX_FLAG_SHORTPRE;
+ rx_status.enc_flags |= RX_ENC_FLAG_SHORTPRE;
/* Set up the HT phy flags */
if (rate_n_flags & RATE_MCS_HT_MSK)
- rx_status.flag |= RX_FLAG_HT;
+ rx_status.encoding = RX_ENC_HT;
if (rate_n_flags & RATE_MCS_HT40_MSK)
- rx_status.flag |= RX_FLAG_40MHZ;
+ rx_status.enc_flags |= RX_ENC_FLAG_40MHZ;
if (rate_n_flags & RATE_MCS_SGI_MSK)
- rx_status.flag |= RX_FLAG_SHORT_GI;
+ rx_status.enc_flags |= RX_ENC_FLAG_SHORT_GI;
if (phy_res->phy_flags & RX_RES_PHY_FLAGS_AGG_MSK) {
/* We know which subframes of an A-MPDU belong
@@ -5799,6 +5799,8 @@
il_leds_init(il);
+ wiphy_ext_feature_set(il->hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
ret = ieee80211_register_hw(il->hw);
if (ret) {
IL_ERR("Failed to register hw (error %d)\n", ret);
diff --git a/drivers/net/wireless/intel/iwlegacy/4965-rs.c b/drivers/net/wireless/intel/iwlegacy/4965-rs.c
index a867ae7..c055f6d 100644
--- a/drivers/net/wireless/intel/iwlegacy/4965-rs.c
+++ b/drivers/net/wireless/intel/iwlegacy/4965-rs.c
@@ -2211,7 +2211,7 @@
/* Get max rate if user set max rate */
if (lq_sta) {
- lq_sta->max_rate_idx = txrc->max_rate_idx;
+ lq_sta->max_rate_idx = fls(txrc->rate_idx_mask) - 1;
if (sband->band == NL80211_BAND_5GHZ &&
lq_sta->max_rate_idx != -1)
lq_sta->max_rate_idx += IL_FIRST_OFDM_RATE;
diff --git a/drivers/net/wireless/intel/iwlwifi/Makefile b/drivers/net/wireless/intel/iwlwifi/Makefile
index 92e6118..411cb91 100644
--- a/drivers/net/wireless/intel/iwlwifi/Makefile
+++ b/drivers/net/wireless/intel/iwlwifi/Makefile
@@ -7,6 +7,7 @@
iwlwifi-objs += iwl-eeprom-read.o iwl-eeprom-parse.o
iwlwifi-objs += iwl-phy-db.o iwl-nvm-parse.o
iwlwifi-objs += pcie/drv.o pcie/rx.o pcie/tx.o pcie/trans.o
+iwlwifi-objs += pcie/ctxt-info.o pcie/trans-gen2.o pcie/tx-gen2.o
iwlwifi-$(CONFIG_IWLDVM) += iwl-1000.o iwl-2000.o iwl-5000.o iwl-6000.o
iwlwifi-$(CONFIG_IWLMVM) += iwl-7000.o iwl-8000.o iwl-9000.o iwl-a000.o
iwlwifi-objs += iwl-trans.o
diff --git a/drivers/net/wireless/intel/iwlwifi/dvm/lib.c b/drivers/net/wireless/intel/iwlwifi/dvm/lib.c
index 6c2d6da..74e52f7 100644
--- a/drivers/net/wireless/intel/iwlwifi/dvm/lib.c
+++ b/drivers/net/wireless/intel/iwlwifi/dvm/lib.c
@@ -180,7 +180,7 @@
goto done;
}
IWL_DEBUG_INFO(priv, "wait transmit/flush all frames\n");
- iwl_trans_wait_tx_queue_empty(priv->trans, 0xffffffff);
+ iwl_trans_wait_tx_queues_empty(priv->trans, 0xffffffff);
done:
ieee80211_wake_queues(priv->hw);
mutex_unlock(&priv->mutex);
diff --git a/drivers/net/wireless/intel/iwlwifi/dvm/mac80211.c b/drivers/net/wireless/intel/iwlwifi/dvm/mac80211.c
index 2a04d0c..444c743 100644
--- a/drivers/net/wireless/intel/iwlwifi/dvm/mac80211.c
+++ b/drivers/net/wireless/intel/iwlwifi/dvm/mac80211.c
@@ -213,6 +213,8 @@
iwl_leds_init(priv);
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
ret = ieee80211_register_hw(priv->hw);
if (ret) {
IWL_ERR(priv, "Failed to register hw (error %d)\n", ret);
@@ -1143,7 +1145,7 @@
}
IWL_DEBUG_TX_QUEUES(priv, "wait transmit/flush all frames\n");
- iwl_trans_wait_tx_queue_empty(priv->trans, scd_queues);
+ iwl_trans_wait_tx_queues_empty(priv->trans, scd_queues);
done:
mutex_unlock(&priv->mutex);
IWL_DEBUG_MAC80211(priv, "leave\n");
diff --git a/drivers/net/wireless/intel/iwlwifi/dvm/rs.c b/drivers/net/wireless/intel/iwlwifi/dvm/rs.c
index ff44ebc..ddcd8c2 100644
--- a/drivers/net/wireless/intel/iwlwifi/dvm/rs.c
+++ b/drivers/net/wireless/intel/iwlwifi/dvm/rs.c
@@ -2720,7 +2720,7 @@
/* Get max rate if user set max rate */
if (lq_sta) {
- lq_sta->max_rate_idx = txrc->max_rate_idx;
+ lq_sta->max_rate_idx = fls(txrc->rate_idx_mask) - 1;
if ((sband->band == NL80211_BAND_5GHZ) &&
(lq_sta->max_rate_idx != -1))
lq_sta->max_rate_idx += IWL_FIRST_OFDM_RATE;
diff --git a/drivers/net/wireless/intel/iwlwifi/dvm/rx.c b/drivers/net/wireless/intel/iwlwifi/dvm/rx.c
index dfa2041..1ee1ba9 100644
--- a/drivers/net/wireless/intel/iwlwifi/dvm/rx.c
+++ b/drivers/net/wireless/intel/iwlwifi/dvm/rx.c
@@ -873,7 +873,7 @@
/* set the preamble flag if appropriate */
if (phy_res->phy_flags & RX_RES_PHY_FLAGS_SHORT_PREAMBLE_MSK)
- rx_status.flag |= RX_FLAG_SHORTPRE;
+ rx_status.enc_flags |= RX_ENC_FLAG_SHORTPRE;
if (phy_res->phy_flags & RX_RES_PHY_FLAGS_AGG_MSK) {
/*
@@ -887,13 +887,13 @@
/* Set up the HT phy flags */
if (rate_n_flags & RATE_MCS_HT_MSK)
- rx_status.flag |= RX_FLAG_HT;
+ rx_status.encoding = RX_ENC_HT;
if (rate_n_flags & RATE_MCS_HT40_MSK)
- rx_status.flag |= RX_FLAG_40MHZ;
+ rx_status.enc_flags |= RX_ENC_FLAG_40MHZ;
if (rate_n_flags & RATE_MCS_SGI_MSK)
- rx_status.flag |= RX_FLAG_SHORT_GI;
+ rx_status.enc_flags |= RX_ENC_FLAG_SHORT_GI;
if (rate_n_flags & RATE_MCS_GF_MSK)
- rx_status.flag |= RX_FLAG_HT_GF;
+ rx_status.enc_flags |= RX_ENC_FLAG_HT_GF;
iwlagn_pass_packet_to_mac80211(priv, header, len, ampdu_status,
rxb, &rx_status);
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-7000.c b/drivers/net/wireless/intel/iwlwifi/iwl-7000.c
index a72e586..3b3e076 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-7000.c
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-7000.c
@@ -73,8 +73,8 @@
/* Highest firmware API version supported */
#define IWL7260_UCODE_API_MAX 17
#define IWL7265_UCODE_API_MAX 17
-#define IWL7265D_UCODE_API_MAX 28
-#define IWL3168_UCODE_API_MAX 28
+#define IWL7265D_UCODE_API_MAX 29
+#define IWL3168_UCODE_API_MAX 29
/* Lowest firmware API version supported */
#define IWL7260_UCODE_API_MIN 17
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-8000.c b/drivers/net/wireless/intel/iwlwifi/iwl-8000.c
index b7953bf..b9718c0 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-8000.c
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-8000.c
@@ -70,8 +70,8 @@
#include "iwl-agn-hw.h"
/* Highest firmware API version supported */
-#define IWL8000_UCODE_API_MAX 28
-#define IWL8265_UCODE_API_MAX 28
+#define IWL8000_UCODE_API_MAX 30
+#define IWL8265_UCODE_API_MAX 30
/* Lowest firmware API version supported */
#define IWL8000_UCODE_API_MIN 17
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-9000.c b/drivers/net/wireless/intel/iwlwifi/iwl-9000.c
index a5f0c0b..110ceef 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-9000.c
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-9000.c
@@ -5,7 +5,7 @@
*
* GPL LICENSE SUMMARY
*
- * Copyright(c) 2015-2016 Intel Deutschland GmbH
+ * Copyright(c) 2015-2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -18,7 +18,7 @@
*
* BSD LICENSE
*
- * Copyright(c) 2015-2016 Intel Deutschland GmbH
+ * Copyright(c) 2015-2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -55,10 +55,10 @@
#include "iwl-agn-hw.h"
/* Highest firmware API version supported */
-#define IWL9000_UCODE_API_MAX 28
+#define IWL9000_UCODE_API_MAX 30
/* Lowest firmware API version supported */
-#define IWL9000_UCODE_API_MIN 17
+#define IWL9000_UCODE_API_MIN 30
/* NVM versions */
#define IWL9000_NVM_VERSION 0x0a1d
@@ -73,14 +73,14 @@
#define IWL9000_SMEM_LEN 0x68000
#define IWL9000_FW_PRE "iwlwifi-9000-pu-a0-jf-a0-"
-#define IWL9260_FW_PRE "iwlwifi-9260-th-a0-jf-a0-"
-#define IWL9000LC_FW_PRE "iwlwifi-9000-pu-a0-lc-a0-"
+#define IWL9260A_FW_PRE "iwlwifi-9260-th-a0-jf-a0-"
+#define IWL9260B_FW_PRE "iwlwifi-9260-th-b0-jf-b0-"
#define IWL9000_MODULE_FIRMWARE(api) \
IWL9000_FW_PRE "-" __stringify(api) ".ucode"
-#define IWL9260_MODULE_FIRMWARE(api) \
- IWL9260_FW_PRE "-" __stringify(api) ".ucode"
-#define IWL9000LC_MODULE_FIRMWARE(api) \
- IWL9000LC_FW_PRE "-" __stringify(api) ".ucode"
+#define IWL9260A_MODULE_FIRMWARE(api) \
+ IWL9260A_FW_PRE "-" __stringify(api) ".ucode"
+#define IWL9260B_MODULE_FIRMWARE(api) \
+ IWL9260B_FW_PRE "-" __stringify(api) ".ucode"
#define NVM_HW_SECTION_NUM_FAMILY_9000 10
@@ -148,7 +148,8 @@
const struct iwl_cfg iwl9160_2ac_cfg = {
.name = "Intel(R) Dual Band Wireless AC 9160",
- .fw_name_pre = IWL9260_FW_PRE,
+ .fw_name_pre = IWL9260A_FW_PRE,
+ .fw_name_pre_next_step = IWL9260B_FW_PRE,
IWL_DEVICE_9000,
.ht_params = &iwl9000_ht_params,
.nvm_ver = IWL9000_NVM_VERSION,
@@ -158,7 +159,8 @@
const struct iwl_cfg iwl9260_2ac_cfg = {
.name = "Intel(R) Dual Band Wireless AC 9260",
- .fw_name_pre = IWL9260_FW_PRE,
+ .fw_name_pre = IWL9260A_FW_PRE,
+ .fw_name_pre_next_step = IWL9260B_FW_PRE,
IWL_DEVICE_9000,
.ht_params = &iwl9000_ht_params,
.nvm_ver = IWL9000_NVM_VERSION,
@@ -168,7 +170,8 @@
const struct iwl_cfg iwl9270_2ac_cfg = {
.name = "Intel(R) Dual Band Wireless AC 9270",
- .fw_name_pre = IWL9260_FW_PRE,
+ .fw_name_pre = IWL9260A_FW_PRE,
+ .fw_name_pre_next_step = IWL9260B_FW_PRE,
IWL_DEVICE_9000,
.ht_params = &iwl9000_ht_params,
.nvm_ver = IWL9000_NVM_VERSION,
@@ -198,21 +201,6 @@
.integrated = true,
};
-/*
- * TODO the struct below is for internal testing only this should be
- * removed by EO 2016~
- */
-const struct iwl_cfg iwl9000lc_2ac_cfg = {
- .name = "Intel(R) Dual Band Wireless AC 9000",
- .fw_name_pre = IWL9000LC_FW_PRE,
- IWL_DEVICE_9000,
- .ht_params = &iwl9000_ht_params,
- .nvm_ver = IWL9000_NVM_VERSION,
- .nvm_calib_ver = IWL9000_TX_POWER_VERSION,
- .max_ht_ampdu_exponent = IEEE80211_HT_MAX_AMPDU_64K,
- .integrated = true,
-};
-
MODULE_FIRMWARE(IWL9000_MODULE_FIRMWARE(IWL9000_UCODE_API_MAX));
-MODULE_FIRMWARE(IWL9260_MODULE_FIRMWARE(IWL9000_UCODE_API_MAX));
-MODULE_FIRMWARE(IWL9000LC_MODULE_FIRMWARE(IWL9000_UCODE_API_MAX));
+MODULE_FIRMWARE(IWL9260A_MODULE_FIRMWARE(IWL9000_UCODE_API_MAX));
+MODULE_FIRMWARE(IWL9260B_MODULE_FIRMWARE(IWL9000_UCODE_API_MAX));
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-a000.c b/drivers/net/wireless/intel/iwlwifi/iwl-a000.c
index 15dd7f6..c648cfb 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-a000.c
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-a000.c
@@ -5,7 +5,7 @@
*
* GPL LICENSE SUMMARY
*
- * Copyright(c) 2015-2016 Intel Deutschland GmbH
+ * Copyright(c) 2015-2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -18,7 +18,7 @@
*
* BSD LICENSE
*
- * Copyright(c) 2015-2016 Intel Deutschland GmbH
+ * Copyright(c) 2015-2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -55,7 +55,7 @@
#include "iwl-agn-hw.h"
/* Highest firmware API version supported */
-#define IWL_A000_UCODE_API_MAX 28
+#define IWL_A000_UCODE_API_MAX 30
/* Lowest firmware API version supported */
#define IWL_A000_UCODE_API_MIN 24
@@ -65,15 +65,16 @@
#define IWL_A000_TX_POWER_VERSION 0xffff /* meaningless */
/* Memory offsets and lengths */
-#define IWL_A000_DCCM_OFFSET 0x800000
-#define IWL_A000_DCCM_LEN 0x18000
+#define IWL_A000_DCCM_OFFSET 0x800000 /* LMAC1 */
+#define IWL_A000_DCCM_LEN 0x10000 /* LMAC1 */
#define IWL_A000_DCCM2_OFFSET 0x880000
#define IWL_A000_DCCM2_LEN 0x8000
#define IWL_A000_SMEM_OFFSET 0x400000
-#define IWL_A000_SMEM_LEN 0x68000
+#define IWL_A000_SMEM_LEN 0xD0000
-#define IWL_A000_JF_FW_PRE "iwlwifi-Qu-a0-jf-b0-"
-#define IWL_A000_HR_FW_PRE "iwlwifi-Qu-a0-hr-a0-"
+#define IWL_A000_JF_FW_PRE "iwlwifi-Qu-a0-jf-b0-"
+#define IWL_A000_HR_FW_PRE "iwlwifi-Qu-a0-hr-a0-"
+#define IWL_A000_HR_CDB_FW_PRE "iwlwifi-QuIcp-a0-hrcdb-a0-"
#define IWL_A000_HR_MODULE_FIRMWARE(api) \
IWL_A000_HR_FW_PRE "-" __stringify(api) ".ucode"
@@ -84,7 +85,7 @@
static const struct iwl_base_params iwl_a000_base_params = {
.eeprom_size = OTP_LOW_IMAGE_SIZE_FAMILY_A000,
- .num_of_queues = 31,
+ .num_of_queues = 512,
.shadow_ram_support = true,
.led_compensation = 57,
.wd_timeout = IWL_LONG_WD_TIMEOUT,
@@ -121,7 +122,8 @@
.vht_mu_mimo_supported = true, \
.mac_addr_from_csr = true, \
.use_tfh = true, \
- .rf_id = true
+ .rf_id = true, \
+ .gen2 = true
const struct iwl_cfg iwla000_2ac_cfg_hr = {
.name = "Intel(R) Dual Band Wireless AC a000",
@@ -133,6 +135,17 @@
.max_ht_ampdu_exponent = IEEE80211_HT_MAX_AMPDU_64K,
};
+const struct iwl_cfg iwla000_2ac_cfg_hr_cdb = {
+ .name = "Intel(R) Dual Band Wireless AC a000",
+ .fw_name_pre = IWL_A000_HR_CDB_FW_PRE,
+ IWL_DEVICE_A000,
+ .ht_params = &iwl_a000_ht_params,
+ .nvm_ver = IWL_A000_NVM_VERSION,
+ .nvm_calib_ver = IWL_A000_TX_POWER_VERSION,
+ .max_ht_ampdu_exponent = IEEE80211_HT_MAX_AMPDU_64K,
+ .cdb = true,
+};
+
const struct iwl_cfg iwla000_2ac_cfg_jf = {
.name = "Intel(R) Dual Band Wireless AC a000",
.fw_name_pre = IWL_A000_JF_FW_PRE,
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-config.h b/drivers/net/wireless/intel/iwlwifi/iwl-config.h
index 94f8a51..a12197e 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-config.h
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-config.h
@@ -6,7 +6,7 @@
* GPL LICENSE SUMMARY
*
* Copyright(c) 2007 - 2014 Intel Corporation. All rights reserved.
- * Copyright (C) 2016 Intel Deutschland GmbH
+ * Copyright (C) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -32,7 +32,7 @@
* BSD LICENSE
*
* Copyright(c) 2005 - 2014 Intel Corporation. All rights reserved.
- * Copyright (C) 2016 Intel Deutschland GmbH
+ * Copyright (C) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -90,16 +90,6 @@
IWL_DEVICE_FAMILY_8000,
};
-static inline bool iwl_has_secure_boot(u32 hw_rev,
- enum iwl_device_family family)
-{
- /* return 1 only for family 8000 B0 */
- if ((family == IWL_DEVICE_FAMILY_8000) && (hw_rev & 0xC))
- return true;
-
- return false;
-}
-
/*
* LED mode
* IWL_LED_DEFAULT: use device default
@@ -180,7 +170,7 @@
apmg_wake_up_wa:1,
scd_chain_ext_wa:1;
- u8 num_of_queues; /* def: HW dependent */
+ u16 num_of_queues; /* def: HW dependent */
u8 max_ll_items;
u8 led_compensation;
@@ -283,6 +273,8 @@
* @fw_name_pre: Firmware filename prefix. The api version and extension
* (.ucode) will be added to filename before loading from disk. The
* filename is constructed as fw_name_pre<api>.ucode.
+ * @fw_name_pre_next_step: same as @fw_name_pre, only for next step
+ * (if supported)
* @ucode_api_max: Highest version of uCode API supported by driver.
* @ucode_api_min: Lowest version of uCode API supported by driver.
* @max_inst_size: The maximal length of the fw inst section
@@ -321,6 +313,8 @@
* @vht_mu_mimo_supported: VHT MU-MIMO support
* @rf_id: need to read rf_id to determine the firmware image
* @integrated: discrete or integrated
+ * @gen2: a000 and on transport operation
+ * @cdb: CDB support
*
* We enable the driver to be backward compatible wrt. hardware features.
* API differences in uCode shouldn't be handled here but through TLVs
@@ -330,6 +324,7 @@
/* params specific to an individual device within a device family */
const char *name;
const char *fw_name_pre;
+ const char *fw_name_pre_next_step;
/* params not likely to change within a device family */
const struct iwl_base_params *base_params;
/* params likely to change within a device family */
@@ -365,7 +360,9 @@
vht_mu_mimo_supported:1,
rf_id:1,
integrated:1,
- use_tfh:1;
+ use_tfh:1,
+ gen2:1,
+ cdb:1;
u8 valid_tx_ant;
u8 valid_rx_ant;
u8 non_shared_ant;
@@ -449,13 +446,13 @@
extern const struct iwl_cfg iwl8260_2ac_sdio_cfg;
extern const struct iwl_cfg iwl8265_2ac_sdio_cfg;
extern const struct iwl_cfg iwl4165_2ac_sdio_cfg;
-extern const struct iwl_cfg iwl9000lc_2ac_cfg;
extern const struct iwl_cfg iwl9160_2ac_cfg;
extern const struct iwl_cfg iwl9260_2ac_cfg;
extern const struct iwl_cfg iwl9270_2ac_cfg;
extern const struct iwl_cfg iwl9460_2ac_cfg;
extern const struct iwl_cfg iwl9560_2ac_cfg;
extern const struct iwl_cfg iwla000_2ac_cfg_hr;
+extern const struct iwl_cfg iwla000_2ac_cfg_hr_cdb;
extern const struct iwl_cfg iwla000_2ac_cfg_jf;
#endif /* CONFIG_IWLMVM */
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-context-info.h b/drivers/net/wireless/intel/iwlwifi/iwl-context-info.h
new file mode 100644
index 0000000..b870c09
--- /dev/null
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-context-info.h
@@ -0,0 +1,203 @@
+/******************************************************************************
+ *
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2017 Intel Deutschland GmbH
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Deutschland GmbH
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ *****************************************************************************/
+
+#ifndef __iwl_context_info_file_h__
+#define __iwl_context_info_file_h__
+
+/* maximmum number of DRAM map entries supported by FW */
+#define IWL_MAX_DRAM_ENTRY 64
+#define CSR_CTXT_INFO_BA 0x40
+
+/**
+ * enum iwl_context_info_flags - Context information control flags
+ * @IWL_CTXT_INFO_AUTO_FUNC_INIT: If set, FW will not wait before interrupting
+ * the init done for driver command that configures several system modes
+ * @IWL_CTXT_INFO_EARLY_DEBUG: enable early debug
+ * @IWL_CTXT_INFO_ENABLE_CDMP: enable core dump
+ * @IWL_CTXT_INFO_RB_SIZE_4K: Use 4K RB size (the default is 2K)
+ * @IWL_CTXT_INFO_RB_CB_SIZE_POS: position of the RBD Cyclic Buffer Size
+ * exponent, the actual size is 2**value, valid sizes are 8-2048.
+ * The value is four bits long. Maximum valid exponent is 12
+ * @IWL_CTXT_INFO_TFD_FORMAT_LONG: use long TFD Format (the
+ * default is short format - not supported by the driver)
+ */
+enum iwl_context_info_flags {
+ IWL_CTXT_INFO_AUTO_FUNC_INIT = BIT(0),
+ IWL_CTXT_INFO_EARLY_DEBUG = BIT(1),
+ IWL_CTXT_INFO_ENABLE_CDMP = BIT(2),
+ IWL_CTXT_INFO_RB_SIZE_4K = BIT(3),
+ IWL_CTXT_INFO_RB_CB_SIZE_POS = 4,
+ IWL_CTXT_INFO_TFD_FORMAT_LONG = BIT(8),
+};
+
+/*
+ * struct iwl_context_info_version - version structure
+ * @mac_id: SKU and revision id
+ * @version: context information version id
+ * @size: the size of the context information in DWs
+ */
+struct iwl_context_info_version {
+ __le16 mac_id;
+ __le16 version;
+ __le16 size;
+ __le16 reserved;
+} __packed;
+
+/*
+ * struct iwl_context_info_control - version structure
+ * @control_flags: context information flags see &enum iwl_context_info_flags
+ */
+struct iwl_context_info_control {
+ __le32 control_flags;
+ __le32 reserved;
+} __packed;
+
+/*
+ * struct iwl_context_info_dram - images DRAM map
+ * each entry in the map represents a DRAM chunk of up to 32 KB
+ * @umac_img: UMAC image DRAM map
+ * @lmac_img: LMAC image DRAM map
+ * @virtual_img: paged image DRAM map
+ */
+struct iwl_context_info_dram {
+ __le64 umac_img[IWL_MAX_DRAM_ENTRY];
+ __le64 lmac_img[IWL_MAX_DRAM_ENTRY];
+ __le64 virtual_img[IWL_MAX_DRAM_ENTRY];
+} __packed;
+
+/*
+ * struct iwl_context_info_rbd_cfg - RBDs configuration
+ * @free_rbd_addr: default queue free RB CB base address
+ * @used_rbd_addr: default queue used RB CB base address
+ * @status_wr_ptr: default queue used RB status write pointer
+ */
+struct iwl_context_info_rbd_cfg {
+ __le64 free_rbd_addr;
+ __le64 used_rbd_addr;
+ __le64 status_wr_ptr;
+} __packed;
+
+/*
+ * struct iwl_context_info_hcmd_cfg - command queue configuration
+ * @cmd_queue_addr: address of command queue
+ * @cmd_queue_size: number of entries
+ */
+struct iwl_context_info_hcmd_cfg {
+ __le64 cmd_queue_addr;
+ u8 cmd_queue_size;
+ u8 reserved[7];
+} __packed;
+
+/*
+ * struct iwl_context_info_dump_cfg - Core Dump configuration
+ * @core_dump_addr: core dump (debug DRAM address) start address
+ * @core_dump_size: size, in DWs
+ */
+struct iwl_context_info_dump_cfg {
+ __le64 core_dump_addr;
+ __le32 core_dump_size;
+ __le32 reserved;
+} __packed;
+
+/*
+ * struct iwl_context_info_pnvm_cfg - platform NVM data configuration
+ * @platform_nvm_addr: Platform NVM data start address
+ * @platform_nvm_size: size in DWs
+ */
+struct iwl_context_info_pnvm_cfg {
+ __le64 platform_nvm_addr;
+ __le32 platform_nvm_size;
+ __le32 reserved;
+} __packed;
+
+/*
+ * struct iwl_context_info_early_dbg_cfg - early debug configuration for
+ * dumping DRAM addresses
+ * @early_debug_addr: early debug start address
+ * @early_debug_size: size in DWs
+ */
+struct iwl_context_info_early_dbg_cfg {
+ __le64 early_debug_addr;
+ __le32 early_debug_size;
+ __le32 reserved;
+} __packed;
+
+/*
+ * struct iwl_context_info - device INIT configuration
+ * @version: version information of context info and HW
+ * @control: control flags of FH configurations
+ * @rbd_cfg: default RX queue configuration
+ * @hcmd_cfg: command queue configuration
+ * @dump_cfg: core dump data
+ * @edbg_cfg: early debug configuration
+ * @pnvm_cfg: platform nvm configuration
+ * @dram: firmware image addresses in DRAM
+ */
+struct iwl_context_info {
+ struct iwl_context_info_version version;
+ struct iwl_context_info_control control;
+ __le64 reserved0;
+ struct iwl_context_info_rbd_cfg rbd_cfg;
+ struct iwl_context_info_hcmd_cfg hcmd_cfg;
+ __le32 reserved1[4];
+ struct iwl_context_info_dump_cfg dump_cfg;
+ struct iwl_context_info_early_dbg_cfg edbg_cfg;
+ struct iwl_context_info_pnvm_cfg pnvm_cfg;
+ __le32 reserved2[16];
+ struct iwl_context_info_dram dram;
+ __le32 reserved3[16];
+} __packed;
+
+int iwl_pcie_ctxt_info_init(struct iwl_trans *trans, const struct fw_img *fw);
+void iwl_pcie_ctxt_info_free(struct iwl_trans *trans);
+void iwl_pcie_ctxt_info_free_paging(struct iwl_trans *trans);
+
+#endif /* __iwl_context_info_file_h__ */
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-csr.h b/drivers/net/wireless/intel/iwlwifi/iwl-csr.h
index 4ee3b62..fa120fb 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-csr.h
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-csr.h
@@ -348,7 +348,6 @@
/* RF_ID value */
#define CSR_HW_RF_ID_TYPE_JF (0x00105000)
-#define CSR_HW_RF_ID_TYPE_LC (0x00101000)
#define CSR_HW_RF_ID_TYPE_HR (0x00109000)
/* EEPROM REG */
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-drv.c b/drivers/net/wireless/intel/iwlwifi/iwl-drv.c
index be466a0..5cfacb0 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-drv.c
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-drv.c
@@ -7,7 +7,7 @@
*
* Copyright(c) 2007 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -34,7 +34,7 @@
*
* Copyright(c) 2005 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -211,24 +211,46 @@
static int iwl_request_firmware(struct iwl_drv *drv, bool first)
{
- const char *name_pre = drv->trans->cfg->fw_name_pre;
+ const struct iwl_cfg *cfg = drv->trans->cfg;
char tag[8];
+ const char *fw_pre_name;
+
+ if (drv->trans->cfg->device_family == IWL_DEVICE_FAMILY_8000 &&
+ CSR_HW_REV_STEP(drv->trans->hw_rev) == SILICON_B_STEP)
+ fw_pre_name = cfg->fw_name_pre_next_step;
+ else
+ fw_pre_name = cfg->fw_name_pre;
if (first) {
- drv->fw_index = drv->trans->cfg->ucode_api_max;
+ drv->fw_index = cfg->ucode_api_max;
sprintf(tag, "%d", drv->fw_index);
} else {
drv->fw_index--;
sprintf(tag, "%d", drv->fw_index);
}
- if (drv->fw_index < drv->trans->cfg->ucode_api_min) {
+ if (drv->fw_index < cfg->ucode_api_min) {
IWL_ERR(drv, "no suitable firmware found!\n");
+
+ if (cfg->ucode_api_min == cfg->ucode_api_max) {
+ IWL_ERR(drv, "%s%d is required\n", fw_pre_name,
+ cfg->ucode_api_max);
+ } else {
+ IWL_ERR(drv, "minimum version required: %s%d\n",
+ fw_pre_name,
+ cfg->ucode_api_min);
+ IWL_ERR(drv, "maximum version supported: %s%d\n",
+ fw_pre_name,
+ cfg->ucode_api_max);
+ }
+
+ IWL_ERR(drv,
+ "check git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git\n");
return -ENOENT;
}
snprintf(drv->firmware_name, sizeof(drv->firmware_name), "%s%s.ucode",
- name_pre, tag);
+ fw_pre_name, tag);
IWL_DEBUG_INFO(drv, "attempting to load firmware '%s'\n",
drv->firmware_name);
@@ -1260,7 +1282,7 @@
pieces = kzalloc(sizeof(*pieces), GFP_KERNEL);
if (!pieces)
- return;
+ goto out_free_fw;
if (!ucode_raw)
goto try_again;
@@ -1472,7 +1494,7 @@
* or hangs loading.
*/
if (load_module) {
- err = request_module("%s", op->name);
+ request_module("%s", op->name);
#ifdef CONFIG_IWLWIFI_OPMODE_MODULAR
if (err)
IWL_ERR(drv,
@@ -1490,17 +1512,18 @@
goto free;
out_free_fw:
- IWL_ERR(drv, "failed to allocate pci memory\n");
iwl_dealloc_ucode(drv);
release_firmware(ucode_raw);
out_unbind:
complete(&drv->request_firmware_complete);
device_release_driver(drv->trans->dev);
free:
- for (i = 0; i < ARRAY_SIZE(pieces->img); i++)
- kfree(pieces->img[i].sec);
- kfree(pieces->dbg_mem_tlv);
- kfree(pieces);
+ if (pieces) {
+ for (i = 0; i < ARRAY_SIZE(pieces->img); i++)
+ kfree(pieces->img[i].sec);
+ kfree(pieces->dbg_mem_tlv);
+ kfree(pieces);
+ }
}
struct iwl_drv *iwl_drv_start(struct iwl_trans *trans)
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-fh.h b/drivers/net/wireless/intel/iwlwifi/iwl-fh.h
index 33ef537..62f9fe9 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-fh.h
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-fh.h
@@ -614,6 +614,8 @@
#define RX_POOL_SIZE (MQ_RX_NUM_RBDS + \
IWL_MAX_RX_HW_QUEUES * \
(RX_CLAIM_REQ_ALLOC - RX_POST_REQ_ALLOC))
+/* cb size is the exponent */
+#define RX_QUEUE_CB_SIZE(x) ilog2(x)
#define RX_QUEUE_SIZE 256
#define RX_QUEUE_MASK 255
@@ -639,6 +641,8 @@
#define TFD_QUEUE_SIZE_MAX (256)
+/* cb size is the exponent - 3 */
+#define TFD_QUEUE_CB_SIZE(x) (ilog2(x) - 3)
#define TFD_QUEUE_SIZE_BC_DUP (64)
#define TFD_QUEUE_BC_SIZE (TFD_QUEUE_SIZE_MAX + TFD_QUEUE_SIZE_BC_DUP)
#define IWL_TX_DMA_MASK DMA_BIT_MASK(36)
@@ -647,7 +651,7 @@
static inline u8 iwl_get_dma_hi_addr(dma_addr_t addr)
{
- return (sizeof(addr) > sizeof(u32) ? (addr >> 16) >> 16 : 0) & 0xF;
+ return (sizeof(addr) > sizeof(u32) ? upper_32_bits(addr) : 0) & 0xF;
}
/**
* struct iwl_tfd_tb transmit buffer descriptor within transmit frame descriptor
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-fw-file.h b/drivers/net/wireless/intel/iwlwifi/iwl-fw-file.h
index d01701e..44419e8 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-fw-file.h
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-fw-file.h
@@ -241,6 +241,9 @@
* iteration complete notification, and the timestamp reported for RX
* received during scan, are reported in TSF of the mac specified in the
* scan request.
+ * @IWL_UCODE_TLV_API_TKIP_MIC_KEYS: This ucode supports version 2 of
+ * ADD_MODIFY_STA_KEY_API_S_VER_2.
+ * @IWL_UCODE_TLV_API_STA_TYPE: This ucode supports station type assignement.
*
* @NUM_IWL_UCODE_TLV_API: number of bits used
*/
@@ -250,6 +253,8 @@
IWL_UCODE_TLV_API_LQ_SS_PARAMS = (__force iwl_ucode_tlv_api_t)18,
IWL_UCODE_TLV_API_NEW_VERSION = (__force iwl_ucode_tlv_api_t)20,
IWL_UCODE_TLV_API_SCAN_TSF_REPORT = (__force iwl_ucode_tlv_api_t)28,
+ IWL_UCODE_TLV_API_TKIP_MIC_KEYS = (__force iwl_ucode_tlv_api_t)29,
+ IWL_UCODE_TLV_API_STA_TYPE = (__force iwl_ucode_tlv_api_t)30,
NUM_IWL_UCODE_TLV_API
#ifdef __CHECKER__
@@ -344,6 +349,8 @@
IWL_UCODE_TLV_CAPA_BT_COEX_RRC = (__force iwl_ucode_tlv_capa_t)30,
IWL_UCODE_TLV_CAPA_GSCAN_SUPPORT = (__force iwl_ucode_tlv_capa_t)31,
IWL_UCODE_TLV_CAPA_STA_PM_NOTIF = (__force iwl_ucode_tlv_capa_t)38,
+ IWL_UCODE_TLV_CAPA_BINDING_CDB_SUPPORT = (__force iwl_ucode_tlv_capa_t)39,
+ IWL_UCODE_TLV_CAPA_CDB_SUPPORT = (__force iwl_ucode_tlv_capa_t)40,
IWL_UCODE_TLV_CAPA_EXTENDED_DTS_MEASURE = (__force iwl_ucode_tlv_capa_t)64,
IWL_UCODE_TLV_CAPA_SHORT_PM_TIMEOUTS = (__force iwl_ucode_tlv_capa_t)65,
IWL_UCODE_TLV_CAPA_BT_MPLUT_SUPPORT = (__force iwl_ucode_tlv_capa_t)67,
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-io.c b/drivers/net/wireless/intel/iwlwifi/iwl-io.c
index a9f69fd..9c8b09c 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-io.c
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-io.c
@@ -54,8 +54,8 @@
void iwl_write64(struct iwl_trans *trans, u64 ofs, u64 val)
{
trace_iwlwifi_dev_iowrite64(trans->dev, ofs, val);
- iwl_trans_write32(trans, ofs, val & 0xffffffff);
- iwl_trans_write32(trans, ofs + 4, val >> 32);
+ iwl_trans_write32(trans, ofs, lower_32_bits(val));
+ iwl_trans_write32(trans, ofs + 4, upper_32_bits(val));
}
IWL_EXPORT_SYMBOL(iwl_write64);
@@ -246,6 +246,9 @@
DEVICE_SET_NMI_VAL_DRV);
iwl_write_prph(trans, DEVICE_SET_NMI_REG,
DEVICE_SET_NMI_VAL_HW);
+ } else if (trans->cfg->gen2) {
+ iwl_write_prph(trans, UREG_NIC_SET_NMI_DRIVER,
+ DEVICE_SET_NMI_8000_VAL);
} else {
iwl_write_prph(trans, DEVICE_SET_NMI_8000_REG,
DEVICE_SET_NMI_8000_VAL);
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-notif-wait.c b/drivers/net/wireless/intel/iwlwifi/iwl-notif-wait.c
index 88f260d..68412ff 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-notif-wait.c
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-notif-wait.c
@@ -76,8 +76,8 @@
}
IWL_EXPORT_SYMBOL(iwl_notification_wait_init);
-void iwl_notification_wait_notify(struct iwl_notif_wait_data *notif_wait,
- struct iwl_rx_packet *pkt)
+bool iwl_notification_wait(struct iwl_notif_wait_data *notif_wait,
+ struct iwl_rx_packet *pkt)
{
bool triggered = false;
@@ -118,13 +118,11 @@
}
}
spin_unlock(¬if_wait->notif_wait_lock);
-
}
- if (triggered)
- wake_up_all(¬if_wait->notif_waitq);
+ return triggered;
}
-IWL_EXPORT_SYMBOL(iwl_notification_wait_notify);
+IWL_EXPORT_SYMBOL(iwl_notification_wait);
void iwl_abort_notification_waits(struct iwl_notif_wait_data *notif_wait)
{
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-notif-wait.h b/drivers/net/wireless/intel/iwlwifi/iwl-notif-wait.h
index 0f9995e..368884b 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-notif-wait.h
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-notif-wait.h
@@ -6,7 +6,7 @@
* GPL LICENSE SUMMARY
*
* Copyright(c) 2007 - 2014 Intel Corporation. All rights reserved.
- * Copyright(c) 2015 Intel Deutschland GmbH
+ * Copyright(c) 2015 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -32,6 +32,7 @@
* BSD LICENSE
*
* Copyright(c) 2005 - 2014 Intel Corporation. All rights reserved.
+ * Copyright(c) 2015 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -89,10 +90,10 @@
*
* This structure is not used directly, to wait for a
* notification declare it on the stack, and call
- * iwlagn_init_notification_wait() with appropriate
+ * iwl_init_notification_wait() with appropriate
* parameters. Then do whatever will cause the ucode
* to notify the driver, and to wait for that then
- * call iwlagn_wait_notification().
+ * call iwl_wait_notification().
*
* Each notification is one-shot. If at some point we
* need to support multi-shot notifications (which
@@ -114,10 +115,24 @@
/* caller functions */
void iwl_notification_wait_init(struct iwl_notif_wait_data *notif_data);
-void iwl_notification_wait_notify(struct iwl_notif_wait_data *notif_data,
- struct iwl_rx_packet *pkt);
+bool iwl_notification_wait(struct iwl_notif_wait_data *notif_data,
+ struct iwl_rx_packet *pkt);
void iwl_abort_notification_waits(struct iwl_notif_wait_data *notif_data);
+static inline void
+iwl_notification_notify(struct iwl_notif_wait_data *notif_data)
+{
+ wake_up_all(¬if_data->notif_waitq);
+}
+
+static inline void
+iwl_notification_wait_notify(struct iwl_notif_wait_data *notif_data,
+ struct iwl_rx_packet *pkt)
+{
+ if (iwl_notification_wait(notif_data, pkt))
+ iwl_notification_notify(notif_data);
+}
+
/* user functions */
void __acquires(wait_entry)
iwl_init_notification_wait(struct iwl_notif_wait_data *notif_data,
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.c b/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.c
index 3bd6fc1..721ae6b 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.c
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.c
@@ -7,7 +7,7 @@
*
* Copyright(c) 2008 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -34,6 +34,7 @@
*
* Copyright(c) 2005 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -438,25 +439,16 @@
vht_cap->vht_mcs.tx_mcs_map = vht_cap->vht_mcs.rx_mcs_map;
}
-static void iwl_init_sbands(struct device *dev, const struct iwl_cfg *cfg,
- struct iwl_nvm_data *data,
- const __le16 *ch_section,
- u8 tx_chains, u8 rx_chains, bool lar_supported)
+void iwl_init_sbands(struct device *dev, const struct iwl_cfg *cfg,
+ struct iwl_nvm_data *data, const __le16 *nvm_ch_flags,
+ u8 tx_chains, u8 rx_chains, bool lar_supported)
{
int n_channels;
int n_used = 0;
struct ieee80211_supported_band *sband;
- if (cfg->device_family != IWL_DEVICE_FAMILY_8000)
- n_channels = iwl_init_channel_map(
- dev, cfg, data,
- &ch_section[NVM_CHANNELS], lar_supported);
- else
- n_channels = iwl_init_channel_map(
- dev, cfg, data,
- &ch_section[NVM_CHANNELS_FAMILY_8000],
- lar_supported);
-
+ n_channels = iwl_init_channel_map(dev, cfg, data, nvm_ch_flags,
+ lar_supported);
sband = &data->bands[NL80211_BAND_2GHZ];
sband->band = NL80211_BAND_2GHZ;
sband->bitrates = &iwl_cfg80211_rates[RATES_24_OFFS];
@@ -482,6 +474,7 @@
IWL_ERR_DEV(dev, "NVM: used only %d of %d channels\n",
n_used, n_channels);
}
+IWL_EXPORT_SYMBOL(iwl_init_sbands);
static int iwl_get_sku(const struct iwl_cfg *cfg, const __le16 *nvm_sw,
const __le16 *phy_sku)
@@ -559,8 +552,8 @@
dest[5] = hw_addr[0];
}
-static void iwl_set_hw_address_from_csr(struct iwl_trans *trans,
- struct iwl_nvm_data *data)
+void iwl_set_hw_address_from_csr(struct iwl_trans *trans,
+ struct iwl_nvm_data *data)
{
__le32 mac_addr0 = cpu_to_le32(iwl_read32(trans, CSR_MAC_ADDR0_STRAP));
__le32 mac_addr1 = cpu_to_le32(iwl_read32(trans, CSR_MAC_ADDR1_STRAP));
@@ -578,6 +571,7 @@
iwl_flip_hw_address(mac_addr0, mac_addr1, data->hw_addr);
}
+IWL_EXPORT_SYMBOL(iwl_set_hw_address_from_csr);
static void iwl_set_hw_address_family_8000(struct iwl_trans *trans,
const struct iwl_cfg *cfg,
@@ -718,7 +712,7 @@
data->xtal_calib[0] = *(nvm_calib + XTAL_CALIB);
data->xtal_calib[1] = *(nvm_calib + XTAL_CALIB + 1);
lar_enabled = true;
- ch_section = nvm_sw;
+ ch_section = &nvm_sw[NVM_CHANNELS];
} else {
u16 lar_offset = data->nvm_version < 0xE39 ?
NVM_LAR_OFFSET_FAMILY_8000_OLD :
@@ -728,7 +722,7 @@
data->lar_enabled = !!(lar_config &
NVM_LAR_ENABLED_FAMILY_8000);
lar_enabled = data->lar_enabled;
- ch_section = regulatory;
+ ch_section = ®ulatory[NVM_CHANNELS_FAMILY_8000];
}
/* If no valid mac address was found - bail out */
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.h b/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.h
index 7249e5b..3fd6506 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.h
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-nvm-parse.h
@@ -6,7 +6,7 @@
* GPL LICENSE SUMMARY
*
* Copyright(c) 2008 - 2015 Intel Corporation. All rights reserved.
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -32,6 +32,7 @@
* BSD LICENSE
*
* Copyright(c) 2005 - 2014 Intel Corporation. All rights reserved.
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -82,6 +83,19 @@
u8 tx_chains, u8 rx_chains, bool lar_fw_supported);
/**
+ * iwl_set_hw_address_from_csr - sets HW address for 9000 devices and on
+ */
+void iwl_set_hw_address_from_csr(struct iwl_trans *trans,
+ struct iwl_nvm_data *data);
+
+/**
+ * iwl_init_sbands - parse and set all channel profiles
+ */
+void iwl_init_sbands(struct device *dev, const struct iwl_cfg *cfg,
+ struct iwl_nvm_data *data, const __le16 *nvm_ch_flags,
+ u8 tx_chains, u8 rx_chains, bool lar_supported);
+
+/**
* iwl_parse_mcc_info - parse MCC (mobile country code) info coming from FW
*
* This function parses the regulatory channel data received as a
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-prph.h b/drivers/net/wireless/intel/iwlwifi/iwl-prph.h
index 406ef30..306bc96 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-prph.h
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-prph.h
@@ -114,6 +114,7 @@
#define DEVICE_SET_NMI_VAL_DRV BIT(7)
#define DEVICE_SET_NMI_8000_REG 0x00a01c24
#define DEVICE_SET_NMI_8000_VAL 0x1000000
+#define UREG_NIC_SET_NMI_DRIVER 0x00a05c10
/* Shared registers (0x0..0x3ff, via target indirect or periphery */
#define SHR_BASE 0x00a10000
@@ -294,9 +295,6 @@
/*********************** END TX SCHEDULER *************************************/
-/* tcp checksum offload */
-#define RX_EN_CSUM (0x00a00d88)
-
/* Oscillator clock */
#define OSC_CLK (0xa04068)
#define OSC_CLK_FORCE_CONTROL (0x8)
@@ -309,6 +307,7 @@
* Note this address is cleared after MAC reset.
*/
#define UREG_UCODE_LOAD_STATUS (0xa05c40)
+#define UREG_CPU_INIT_RUN (0xa05c44)
#define LMPM_SECURE_UCODE_LOAD_CPU1_HDR_ADDR (0x1E78)
#define LMPM_SECURE_UCODE_LOAD_CPU2_HDR_ADDR (0x1E7C)
@@ -316,6 +315,8 @@
#define LMPM_SECURE_CPU1_HDR_MEM_SPACE (0x420000)
#define LMPM_SECURE_CPU2_HDR_MEM_SPACE (0x420400)
+#define LMAC2_PRPH_OFFSET (0x100000)
+
/* Rx FIFO */
#define RXF_SIZE_ADDR (0xa00c88)
#define RXF_RD_D_SPACE (0xa00c40)
@@ -378,6 +379,7 @@
#define RADIO_REG_SYS_MANUAL_DFT_0 0xAD4078
#define RFIC_REG_RD 0xAD0470
#define WFPM_CTRL_REG 0xA03030
+#define WFPM_GP2 0xA030B4
enum {
ENABLE_WFPM = BIT(31),
WFPM_AUX_CTL_AUX_IF_MAC_OWNER_MSK = 0x80000000,
@@ -398,6 +400,8 @@
#define PREG_AUX_BUS_WPROT_0 0xA04CC0
#define SB_CPU_1_STATUS 0xA01E30
#define SB_CPU_2_STATUS 0xA01E34
+#define UMAG_SB_CPU_1_STATUS 0xA038C0
+#define UMAG_SB_CPU_2_STATUS 0xA038C4
/* FW chicken bits */
#define LMPM_CHICK 0xA01FF8
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-trans.c b/drivers/net/wireless/intel/iwlwifi/iwl-trans.c
index d42cab2..0bde26b 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-trans.c
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-trans.c
@@ -70,8 +70,7 @@
struct iwl_trans *iwl_trans_alloc(unsigned int priv_size,
struct device *dev,
const struct iwl_cfg *cfg,
- const struct iwl_trans_ops *ops,
- size_t dev_cmd_headroom)
+ const struct iwl_trans_ops *ops)
{
struct iwl_trans *trans;
#ifdef CONFIG_LOCKDEP
@@ -90,15 +89,13 @@
trans->dev = dev;
trans->cfg = cfg;
trans->ops = ops;
- trans->dev_cmd_headroom = dev_cmd_headroom;
trans->num_rx_queues = 1;
snprintf(trans->dev_cmd_pool_name, sizeof(trans->dev_cmd_pool_name),
"iwl_cmd_pool:%s", dev_name(trans->dev));
trans->dev_cmd_pool =
kmem_cache_create(trans->dev_cmd_pool_name,
- sizeof(struct iwl_device_cmd)
- + trans->dev_cmd_headroom,
+ sizeof(struct iwl_device_cmd),
sizeof(void *),
SLAB_HWCACHE_ALIGN,
NULL);
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-trans.h b/drivers/net/wireless/intel/iwlwifi/iwl-trans.h
index 0296124..0ebfdbb 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-trans.h
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-trans.h
@@ -7,7 +7,7 @@
*
* Copyright(c) 2007 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -34,7 +34,7 @@
*
* Copyright(c) 2005 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -396,7 +396,10 @@
* currently supports
*/
#define IWL_MAX_HW_QUEUES 32
+#define IWL_MAX_TVQM_QUEUES 512
+
#define IWL_MAX_TID_COUNT 8
+#define IWL_MGMT_TID 15
#define IWL_FRAME_LIMIT 64
#define IWL_MAX_RX_HW_QUEUES 16
@@ -530,6 +533,44 @@
int frame_limit;
};
+/* Available options for &struct iwl_tx_queue_cfg_cmd */
+enum iwl_tx_queue_cfg_actions {
+ TX_QUEUE_CFG_ENABLE_QUEUE = BIT(0),
+ TX_QUEUE_CFG_TFD_SHORT_FORMAT = BIT(1),
+};
+
+/**
+ * struct iwl_tx_queue_cfg_cmd - txq hw scheduler config command
+ * @sta_id: station id
+ * @tid: tid of the queue
+ * @flags: Bit 0 - on enable, off - disable, Bit 1 - short TFD format
+ * @cb_size: size of TFD cyclic buffer. Value is exponent - 3.
+ * Minimum value 0 (8 TFDs), maximum value 5 (256 TFDs)
+ * @byte_cnt_addr: address of byte count table
+ * @tfdq_addr: address of TFD circular buffer
+ */
+struct iwl_tx_queue_cfg_cmd {
+ u8 sta_id;
+ u8 tid;
+ __le16 flags;
+ __le32 cb_size;
+ __le64 byte_cnt_addr;
+ __le64 tfdq_addr;
+} __packed; /* TX_QUEUE_CFG_CMD_API_S_VER_2 */
+
+/**
+ * struct iwl_tx_queue_cfg_rsp - response to txq hw scheduler config
+ * @queue_number: queue number assigned to this RA -TID
+ * @flags: set on failure
+ * @write_pointer: initial value for write pointer
+ */
+struct iwl_tx_queue_cfg_rsp {
+ __le16 queue_number;
+ __le16 flags;
+ __le16 write_pointer;
+ __le16 reserved;
+} __packed; /* TX_QUEUE_CFG_RSP_API_S_VER_2 */
+
/**
* struct iwl_trans_ops - transport specific operations
*
@@ -640,13 +681,17 @@
unsigned int queue_wdg_timeout);
void (*txq_disable)(struct iwl_trans *trans, int queue,
bool configure_scd);
+ /* a000 functions */
+ int (*txq_alloc)(struct iwl_trans *trans,
+ struct iwl_tx_queue_cfg_cmd *cmd,
+ int cmd_id,
+ unsigned int queue_wdg_timeout);
+ void (*txq_free)(struct iwl_trans *trans, int queue);
void (*txq_set_shared_mode)(struct iwl_trans *trans, u32 txq_id,
bool shared);
- dma_addr_t (*get_txq_byte_table)(struct iwl_trans *trans, int txq_id);
-
- int (*wait_tx_queue_empty)(struct iwl_trans *trans, u32 txq_bm);
+ int (*wait_tx_queues_empty)(struct iwl_trans *trans, u32 txq_bm);
void (*freeze_txq_timer)(struct iwl_trans *trans, unsigned long txqs,
bool freeze);
void (*block_txq_ptrs)(struct iwl_trans *trans, bool block);
@@ -774,9 +819,6 @@
* the transport must set this before calling iwl_drv_start()
* @dev_cmd_pool: pool for Tx cmd allocation - for internal use only.
* The user should use iwl_trans_{alloc,free}_tx_cmd.
- * @dev_cmd_headroom: room needed for the transport's private use before the
- * device_cmd for Tx - for internal use only
- * The user should use iwl_trans_{alloc,free}_tx_cmd.
* @rx_mpdu_cmd: MPDU RX command ID, must be assigned by opmode before
* starting the firmware, used for tracing
* @rx_mpdu_cmd_hdr_size: used for tracing, amount of data before the
@@ -827,7 +869,6 @@
/* The following fields are internal only */
struct kmem_cache *dev_cmd_pool;
- size_t dev_cmd_headroom;
char dev_cmd_pool_name[50];
struct dentry *dbgfs_dir;
@@ -1000,13 +1041,13 @@
static inline struct iwl_device_cmd *
iwl_trans_alloc_tx_cmd(struct iwl_trans *trans)
{
- u8 *dev_cmd_ptr = kmem_cache_alloc(trans->dev_cmd_pool, GFP_ATOMIC);
+ struct iwl_device_cmd *dev_cmd_ptr =
+ kmem_cache_alloc(trans->dev_cmd_pool, GFP_ATOMIC);
if (unlikely(dev_cmd_ptr == NULL))
return NULL;
- return (struct iwl_device_cmd *)
- (dev_cmd_ptr + trans->dev_cmd_headroom);
+ return dev_cmd_ptr;
}
int iwl_trans_send_cmd(struct iwl_trans *trans, struct iwl_host_cmd *cmd);
@@ -1014,9 +1055,7 @@
static inline void iwl_trans_free_tx_cmd(struct iwl_trans *trans,
struct iwl_device_cmd *dev_cmd)
{
- u8 *dev_cmd_ptr = (u8 *)dev_cmd - trans->dev_cmd_headroom;
-
- kmem_cache_free(trans->dev_cmd_pool, dev_cmd_ptr);
+ kmem_cache_free(trans->dev_cmd_pool, dev_cmd);
}
static inline int iwl_trans_tx(struct iwl_trans *trans, struct sk_buff *skb,
@@ -1065,6 +1104,34 @@
trans->ops->txq_enable(trans, queue, ssn, cfg, queue_wdg_timeout);
}
+static inline void
+iwl_trans_txq_free(struct iwl_trans *trans, int queue)
+{
+ if (WARN_ON_ONCE(!trans->ops->txq_free))
+ return;
+
+ trans->ops->txq_free(trans, queue);
+}
+
+static inline int
+iwl_trans_txq_alloc(struct iwl_trans *trans,
+ struct iwl_tx_queue_cfg_cmd *cmd,
+ int cmd_id,
+ unsigned int queue_wdg_timeout)
+{
+ might_sleep();
+
+ if (WARN_ON_ONCE(!trans->ops->txq_alloc))
+ return -ENOTSUPP;
+
+ if (WARN_ON_ONCE(trans->state != IWL_TRANS_FW_ALIVE)) {
+ IWL_ERR(trans, "%s bad state = %d\n", __func__, trans->state);
+ return -EIO;
+ }
+
+ return trans->ops->txq_alloc(trans, cmd, cmd_id, queue_wdg_timeout);
+}
+
static inline void iwl_trans_txq_set_shared_mode(struct iwl_trans *trans,
int queue, bool shared_mode)
{
@@ -1072,15 +1139,6 @@
trans->ops->txq_set_shared_mode(trans, queue, shared_mode);
}
-static inline dma_addr_t iwl_trans_get_txq_byte_table(struct iwl_trans *trans,
- int queue)
-{
- /* we should never be called if the trans doesn't support it */
- BUG_ON(!trans->ops->get_txq_byte_table);
-
- return trans->ops->get_txq_byte_table(trans, queue);
-}
-
static inline void iwl_trans_txq_enable(struct iwl_trans *trans, int queue,
int fifo, int sta_id, int tid,
int frame_limit, u16 ssn,
@@ -1137,15 +1195,15 @@
trans->ops->block_txq_ptrs(trans, block);
}
-static inline int iwl_trans_wait_tx_queue_empty(struct iwl_trans *trans,
- u32 txqs)
+static inline int iwl_trans_wait_tx_queues_empty(struct iwl_trans *trans,
+ u32 txqs)
{
if (WARN_ON_ONCE(trans->state != IWL_TRANS_FW_ALIVE)) {
IWL_ERR(trans, "%s bad state = %d\n", __func__, trans->state);
return -EIO;
}
- return trans->ops->wait_tx_queue_empty(trans, txqs);
+ return trans->ops->wait_tx_queues_empty(trans, txqs);
}
static inline void iwl_trans_write8(struct iwl_trans *trans, u32 ofs, u8 val)
@@ -1248,8 +1306,7 @@
struct iwl_trans *iwl_trans_alloc(unsigned int priv_size,
struct device *dev,
const struct iwl_cfg *cfg,
- const struct iwl_trans_ops *ops,
- size_t dev_cmd_headroom);
+ const struct iwl_trans_ops *ops);
void iwl_trans_free(struct iwl_trans *trans);
/*****************************************************
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/binding.c b/drivers/net/wireless/intel/iwlwifi/mvm/binding.c
index 7cb68f6..75d35f6 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/binding.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/binding.c
@@ -6,6 +6,7 @@
* GPL LICENSE SUMMARY
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
+ * Copyright(c) 2016 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -31,6 +32,7 @@
* BSD LICENSE
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
+ * Copyright(c) 2016 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -82,9 +84,22 @@
struct iwl_mvm_phy_ctxt *phyctxt = data->phyctxt;
int i, ret;
u32 status;
+ int size;
memset(&cmd, 0, sizeof(cmd));
+ if (fw_has_capa(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_CAPA_BINDING_CDB_SUPPORT)) {
+ size = sizeof(cmd);
+ if (phyctxt->channel->band == NL80211_BAND_2GHZ ||
+ !iwl_mvm_is_cdb_supported(mvm))
+ cmd.lmac_id = cpu_to_le32(IWL_LMAC_24G_INDEX);
+ else
+ cmd.lmac_id = cpu_to_le32(IWL_LMAC_5G_INDEX);
+ } else {
+ size = IWL_BINDING_CMD_SIZE_V1;
+ }
+
cmd.id_and_color = cpu_to_le32(FW_CMD_ID_AND_COLOR(phyctxt->id,
phyctxt->color));
cmd.action = cpu_to_le32(action);
@@ -99,7 +114,7 @@
status = 0;
ret = iwl_mvm_send_cmd_pdu_status(mvm, BINDING_CONTEXT_CMD,
- sizeof(cmd), &cmd, &status);
+ size, &cmd, &status);
if (ret) {
IWL_ERR(mvm, "Failed to send binding (action:%d): %d\n",
action, ret);
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/coex.c b/drivers/net/wireless/intel/iwlwifi/mvm/coex.c
index 5bdb6c2..49b4418 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/coex.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/coex.c
@@ -756,7 +756,7 @@
* Rssi update while not associated - can happen since the statistics
* are handled asynchronously
*/
- if (mvmvif->ap_sta_id == IWL_MVM_STATION_COUNT)
+ if (mvmvif->ap_sta_id == IWL_MVM_INVALID_STA)
return;
/* No BT - reports should be disabled */
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/d3.c b/drivers/net/wireless/intel/iwlwifi/mvm/d3.c
index c7eb198..119a3bd 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/d3.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/d3.c
@@ -665,6 +665,19 @@
struct iwl_binding_cmd binding_cmd = {};
struct iwl_time_quota_cmd quota_cmd = {};
u32 status;
+ int size;
+
+ if (fw_has_capa(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_CAPA_BINDING_CDB_SUPPORT)) {
+ size = sizeof(binding_cmd);
+ if (mvmvif->phy_ctxt->channel->band == NL80211_BAND_2GHZ ||
+ !iwl_mvm_is_cdb_supported(mvm))
+ binding_cmd.lmac_id = cpu_to_le32(IWL_LMAC_24G_INDEX);
+ else
+ binding_cmd.lmac_id = cpu_to_le32(IWL_LMAC_5G_INDEX);
+ } else {
+ size = IWL_BINDING_CMD_SIZE_V1;
+ }
/* add back the PHY */
if (WARN_ON(!mvmvif->phy_ctxt))
@@ -711,8 +724,7 @@
status = 0;
ret = iwl_mvm_send_cmd_pdu_status(mvm, BINDING_CONTEXT_CMD,
- sizeof(binding_cmd), &binding_cmd,
- &status);
+ size, &binding_cmd, &status);
if (ret) {
IWL_ERR(mvm, "Failed to add binding: %d\n", ret);
return ret;
@@ -986,7 +998,9 @@
goto out;
}
- if (key_data.use_tkip) {
+ if (key_data.use_tkip &&
+ !fw_has_api(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_API_TKIP_MIC_KEYS)) {
ret = iwl_mvm_send_cmd_pdu(mvm,
WOWLAN_TKIP_PARAM,
cmd_flags, sizeof(tkip_cmd),
@@ -1194,7 +1208,7 @@
mvmvif = iwl_mvm_vif_from_mac80211(vif);
- if (mvmvif->ap_sta_id == IWL_MVM_STATION_COUNT) {
+ if (mvmvif->ap_sta_id == IWL_MVM_INVALID_STA) {
/* if we're not associated, this must be netdetect */
if (!wowlan->nd_config) {
ret = 1;
@@ -2102,6 +2116,10 @@
*/
iwl_mvm_update_changed_regdom(mvm);
+ if (!unified_image)
+ /* Re-configure default SAR profile */
+ iwl_mvm_sar_select_profile(mvm, 1, 1);
+
if (mvm->net_detect) {
/* If this is a non-unified image, we restart the FW,
* so no need to stop the netdetect scan. If that
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/debugfs-vif.c b/drivers/net/wireless/intel/iwlwifi/mvm/debugfs-vif.c
index f4d75ff..5d475b4 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/debugfs-vif.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/debugfs-vif.c
@@ -280,7 +280,7 @@
mvmvif->queue_params[i].uapsd);
if (vif->type == NL80211_IFTYPE_STATION &&
- ap_sta_id != IWL_MVM_STATION_COUNT) {
+ ap_sta_id != IWL_MVM_INVALID_STA) {
struct iwl_mvm_sta *mvm_sta;
mvm_sta = iwl_mvm_sta_from_staid_protected(mvm, ap_sta_id);
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/debugfs.c b/drivers/net/wireless/intel/iwlwifi/mvm/debugfs.c
index 077bfd8..4028466 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/debugfs.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/debugfs.c
@@ -330,7 +330,7 @@
mutex_lock(&mvm->mutex);
- for (i = 0; i < IWL_MVM_STATION_COUNT; i++) {
+ for (i = 0; i < ARRAY_SIZE(mvm->fw_id_to_mac_id); i++) {
pos += scnprintf(buf + pos, bufsz - pos, "%.2d: ", i);
sta = rcu_dereference_protected(mvm->fw_id_to_mac_id[i],
lockdep_is_held(&mvm->mutex));
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-mac.h b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-mac.h
index 480a54a..970b030 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-mac.h
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-mac.h
@@ -73,7 +73,9 @@
#define NUM_MAC_INDEX (NUM_MAC_INDEX_DRIVER + 1)
#define NUM_MAC_INDEX_CDB (NUM_MAC_INDEX_DRIVER + 2)
-#define IWL_MVM_STATION_COUNT 16
+#define IWL_MVM_STATION_COUNT 16
+#define IWL_MVM_INVALID_STA 0xFF
+
#define IWL_MVM_TDLS_STA_COUNT 4
enum iwl_ac {
@@ -155,7 +157,8 @@
* @bi_reciprocal: 2^32 / bi
* @dtim_interval: dtim transmit time in TU
* @dtim_reciprocal: 2^32 / dtim_interval
- * @mcast_qid: queue ID for multicast traffic
+ * @mcast_qid: queue ID for multicast traffic.
+ * NOTE: obsolete from VER2 and on
* @beacon_template: beacon template ID
*/
struct iwl_mac_data_ap {
@@ -167,7 +170,7 @@
__le32 dtim_reciprocal;
__le32 mcast_qid;
__le32 beacon_template;
-} __packed; /* AP_MAC_DATA_API_S_VER_1 */
+} __packed; /* AP_MAC_DATA_API_S_VER_2 */
/**
* struct iwl_mac_data_ibss - configuration data for IBSS MAC context
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-power.h b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-power.h
index 3fa43d1..750510a 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-power.h
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-power.h
@@ -7,7 +7,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
- * Copyright(c) 2015 - 2016 Intel Deutschland GmbH
+ * Copyright(c) 2015 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -34,7 +34,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
- * Copyright(c) 2015 - 2016 Intel Deutschland GmbH
+ * Copyright(c) 2015 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -351,6 +351,45 @@
u8 reserved[3];
} __packed; /* TX_REDUCED_POWER_API_S_VER_4 */
+#define IWL_NUM_GEO_PROFILES 3
+
+/**
+ * enum iwl_geo_per_chain_offset_operation - type of operation
+ * @IWL_PER_CHAIN_OFFSET_SET_TABLES: send the tables from the host to the FW.
+ * @IWL_PER_CHAIN_OFFSET_GET_CURRENT_TABLE: retrieve the last configured table.
+ */
+enum iwl_geo_per_chain_offset_operation {
+ IWL_PER_CHAIN_OFFSET_SET_TABLES,
+ IWL_PER_CHAIN_OFFSET_GET_CURRENT_TABLE,
+}; /* GEO_TX_POWER_LIMIT FLAGS TYPE */
+
+/**
+ * struct iwl_per_chain_offset - embedded struct for GEO_TX_POWER_LIMIT.
+ * @max_tx_power: maximum allowed tx power.
+ * @chain_a: tx power offset for chain a.
+ * @chain_b: tx power offset for chain b.
+ */
+struct iwl_per_chain_offset {
+ __le16 max_tx_power;
+ u8 chain_a;
+ u8 chain_b;
+} __packed; /* PER_CHAIN_LIMIT_OFFSET_PER_CHAIN_S_VER_1 */
+
+struct iwl_per_chain_offset_group {
+ struct iwl_per_chain_offset lb;
+ struct iwl_per_chain_offset hb;
+} __packed; /* PER_CHAIN_LIMIT_OFFSET_GROUP_S_VER_1 */
+
+/**
+ * struct iwl_geo_tx_power_profile_cmd - struct for GEO_TX_POWER_LIMIT cmd.
+ * @ops: operations, value from &enum iwl_geo_per_chain_offset_operation
+ * @table: offset profile per band.
+ */
+struct iwl_geo_tx_power_profiles_cmd {
+ __le32 ops;
+ struct iwl_per_chain_offset_group table[IWL_NUM_GEO_PROFILES];
+} __packed; /* GEO_TX_POWER_LIMIT */
+
/**
* struct iwl_beacon_filter_cmd
* REPLY_BEACON_FILTERING_CMD = 0xd2 (command)
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-rs.h b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-rs.h
index ad9cc03..1b7d265 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-rs.h
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-rs.h
@@ -6,6 +6,7 @@
* GPL LICENSE SUMMARY
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
+ * Copyright(c) 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -31,6 +32,7 @@
* BSD LICENSE
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
+ * Copyright(c) 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -227,6 +229,9 @@
*/
#define RATE_LEGACY_RATE_MSK 0xff
+/* Bit 10 - OFDM HE */
+#define RATE_MCS_OFDM_HE_POS 10
+#define RATE_MCS_OFDM_HE_MSK BIT(RATE_MCS_OFDM_HE_POS)
/*
* Bit 11-12: (0) 20MHz, (1) 40MHz, (2) 80MHz, (3) 160MHz
@@ -255,18 +260,29 @@
#define RATE_MCS_ANT_MSK RATE_MCS_ANT_ABC_MSK
#define RATE_MCS_ANT_NUM 3
-/* Bit 17-18: (0) SS, (1) SS*2 */
+/* Bit 17: (0) SS, (1) SS*2 */
#define RATE_MCS_STBC_POS 17
-#define RATE_MCS_HT_STBC_MSK (3 << RATE_MCS_STBC_POS)
-#define RATE_MCS_VHT_STBC_MSK (1 << RATE_MCS_STBC_POS)
+#define RATE_MCS_STBC_MSK BIT(RATE_MCS_STBC_POS)
+
+/* Bit 18: OFDM-HE dual carrier mode */
+#define RATE_HE_DUAL_CARRIER_MODE 18
+#define RATE_HE_DUAL_CARRIER_MODE_MSK BIT(RATE_HE_DUAL_CARRIER_MODE)
/* Bit 19: (0) Beamforming is off, (1) Beamforming is on */
#define RATE_MCS_BF_POS 19
#define RATE_MCS_BF_MSK (1 << RATE_MCS_BF_POS)
-/* Bit 20: (0) ZLF is off, (1) ZLF is on */
-#define RATE_MCS_ZLF_POS 20
-#define RATE_MCS_ZLF_MSK (1 << RATE_MCS_ZLF_POS)
+/*
+ * Bit 20-21: HE guard interval and LTF type.
+ * (0) 1xLTF+1.6us, (1) 2xLTF+0.8us,
+ * (2) 2xLTF+1.6us, (3) 4xLTF+3.2us
+ */
+#define RATE_MCS_HE_GI_LTF_POS 20
+#define RATE_MCS_HE_GI_LTF_MSK (3 << RATE_MCS_HE_GI_LTF_POS)
+
+/* Bit 22-23: HE type. (0) SU, (1) SU_EXT, (2) MU, (3) trigger based */
+#define RATE_MCS_HE_TYPE_POS 22
+#define RATE_MCS_HE_TYPE_MSK (3 << RATE_MCS_HE_TYPE_POS)
/* Bit 24-25: (0) 20MHz (no dup), (1) 2x20MHz, (2) 4x20MHz, 3 8x20MHz */
#define RATE_MCS_DUP_POS 24
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-scan.h b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-scan.h
index c78a0c4..3178eb9 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-scan.h
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-scan.h
@@ -516,7 +516,7 @@
* scan_config_channel_flag
* @channel_array: default supported channels
*/
-struct iwl_scan_config {
+struct iwl_scan_config_v1 {
__le32 flags;
__le32 tx_chains;
__le32 rx_chains;
@@ -532,7 +532,7 @@
#define SCAN_TWO_LMACS 2
-struct iwl_scan_config_cdb {
+struct iwl_scan_config {
__le32 flags;
__le32 tx_chains;
__le32 rx_chains;
@@ -669,7 +669,7 @@
u8 n_channels;
__le16 reserved;
u8 data[];
- } no_cdb; /* SCAN_REQUEST_CMD_UMAC_API_S_VER_1 */
+ } v1; /* SCAN_REQUEST_CMD_UMAC_API_S_VER_1 */
struct {
__le32 max_out_time[SCAN_TWO_LMACS];
__le32 suspend_time[SCAN_TWO_LMACS];
@@ -679,13 +679,13 @@
u8 n_channels;
__le16 reserved;
u8 data[];
- } cdb; /* SCAN_REQUEST_CMD_UMAC_API_S_VER_5 */
+ } v6; /* SCAN_REQUEST_CMD_UMAC_API_S_VER_6 */
};
} __packed;
-#define IWL_SCAN_REQ_UMAC_SIZE_CDB sizeof(struct iwl_scan_req_umac)
-#define IWL_SCAN_REQ_UMAC_SIZE (sizeof(struct iwl_scan_req_umac) - \
- 2 * sizeof(__le32))
+#define IWL_SCAN_REQ_UMAC_SIZE sizeof(struct iwl_scan_req_umac)
+#define IWL_SCAN_REQ_UMAC_SIZE_V1 (sizeof(struct iwl_scan_req_umac) - \
+ 2 * sizeof(__le32))
/**
* struct iwl_umac_scan_abort
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-sta.h b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-sta.h
index 3b5150e..421b9dd 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-sta.h
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-sta.h
@@ -7,7 +7,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -34,7 +34,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -179,7 +179,7 @@
* enum iwl_sta_modify_flag - indicate to the fw what flag are being changed
* @STA_MODIFY_QUEUE_REMOVAL: this command removes a queue
* @STA_MODIFY_TID_DISABLE_TX: this command modifies %tid_disable_tx
- * @STA_MODIFY_UAPSD_ACS: this command modifies %uapsd_trigger_acs
+ * @STA_MODIFY_UAPSD_ACS: this command modifies %uapsd_acs
* @STA_MODIFY_ADD_BA_TID: this command modifies %add_immediate_ba_tid
* @STA_MODIFY_REMOVE_BA_TID: this command modifies %remove_immediate_ba_tid
* @STA_MODIFY_SLEEPING_STA_TX_COUNT: this command modifies %sleep_tx_count
@@ -214,20 +214,6 @@
STA_SLEEP_STATE_MOREDATA = BIT(2),
};
-/* STA ID and color bits definitions */
-#define STA_ID_SEED (0x0f)
-#define STA_ID_POS (0)
-#define STA_ID_MSK (STA_ID_SEED << STA_ID_POS)
-
-#define STA_COLOR_SEED (0x7)
-#define STA_COLOR_POS (4)
-#define STA_COLOR_MSK (STA_COLOR_SEED << STA_COLOR_POS)
-
-#define STA_ID_N_COLOR_GET_COLOR(id_n_color) \
- (((id_n_color) & STA_COLOR_MSK) >> STA_COLOR_POS)
-#define STA_ID_N_COLOR_GET_ID(id_n_color) \
- (((id_n_color) & STA_ID_MSK) >> STA_ID_POS)
-
#define STA_KEY_MAX_NUM (16)
#define STA_KEY_IDX_INVALID (0xff)
#define STA_KEY_MAX_DATA_KEY_NUM (4)
@@ -324,6 +310,24 @@
} __packed; /* ADD_STA_CMD_API_S_VER_7 */
/**
+ * enum iwl_sta_type - FW station types
+ * ( REPLY_ADD_STA = 0x18 )
+ * @IWL_STA_LINK: Link station - normal RX and TX traffic.
+ * @IWL_STA_GENERAL_PURPOSE: General purpose. In AP mode used for beacons
+ * and probe responses.
+ * @IWL_STA_MULTICAST: multicast traffic,
+ * @IWL_STA_TDLS_LINK: TDLS link station
+ * @IWL_STA_AUX_ACTIVITY: auxilary station (scan, ROC and so on).
+ */
+enum iwl_sta_type {
+ IWL_STA_LINK,
+ IWL_STA_GENERAL_PURPOSE,
+ IWL_STA_MULTICAST,
+ IWL_STA_TDLS_LINK,
+ IWL_STA_AUX_ACTIVITY,
+};
+
+/**
* struct iwl_mvm_add_sta_cmd - Add/modify a station in the fw's sta table.
* ( REPLY_ADD_STA = 0x18 )
* @add_modify: 1: modify existing, 0: add new station
@@ -347,14 +351,17 @@
* @sleep_tx_count: number of packets to transmit to station even though it is
* asleep. Used to synchronise PS-poll and u-APSD responses while ucode
* keeps track of STA sleep state.
+ * @station_type: type of this station. See &enum iwl_sta_type.
* @sleep_state_flags: Look at %iwl_sta_sleep_flag.
* @assoc_id: assoc_id to be sent in VHT PLCP (9-bit), for grp use 0, for AP
* mac-addr.
* @beamform_flags: beam forming controls
- * @tfd_queue_msk: tfd queues used by this station
+ * @tfd_queue_msk: tfd queues used by this station.
+ * Obselete for new TX API (9 and above).
* @rx_ba_window: aggregation window size
- * @scd_queue_bank: queue bank in used. Each bank contains 32 queues. 0 means
- * that the queues used by this station are in the first 32.
+ * @sp_length: the size of the SP as it appears in the WME IE
+ * @uapsd_acs: 4 LS bits are trigger enabled ACs, 4 MS bits are the deliver
+ * enabled ACs.
*
* The device contains an internal table of per-station information, with info
* on security keys, aggregation parameters, and Tx rates for initial Tx
@@ -379,38 +386,61 @@
u8 remove_immediate_ba_tid;
__le16 add_immediate_ba_ssn;
__le16 sleep_tx_count;
- __le16 sleep_state_flags;
+ u8 sleep_state_flags;
+ u8 station_type;
__le16 assoc_id;
__le16 beamform_flags;
__le32 tfd_queue_msk;
__le16 rx_ba_window;
- u8 scd_queue_bank;
- u8 uapsd_trigger_acs;
-} __packed; /* ADD_STA_CMD_API_S_VER_8 */
+ u8 sp_length;
+ u8 uapsd_acs;
+} __packed; /* ADD_STA_CMD_API_S_VER_10 */
/**
- * struct iwl_mvm_add_sta_key_cmd - add/modify sta key
+ * struct iwl_mvm_add_sta_key_common - add/modify sta key common part
* ( REPLY_ADD_STA_KEY = 0x17 )
* @sta_id: index of station in uCode's station table
* @key_offset: key offset in key storage
* @key_flags: type %iwl_sta_key_flag
* @key: key material data
* @rx_secur_seq_cnt: RX security sequence counter for the key
- * @tkip_rx_tsc_byte2: TSC[2] for key mix ph1 detection
- * @tkip_rx_ttak: 10-byte unicast TKIP TTAK for Rx
*/
-struct iwl_mvm_add_sta_key_cmd {
+struct iwl_mvm_add_sta_key_common {
u8 sta_id;
u8 key_offset;
__le16 key_flags;
u8 key[32];
u8 rx_secur_seq_cnt[16];
+} __packed;
+
+/**
+ * struct iwl_mvm_add_sta_key_cmd_v1 - add/modify sta key
+ * @common: see &struct iwl_mvm_add_sta_key_common
+ * @tkip_rx_tsc_byte2: TSC[2] for key mix ph1 detection
+ * @tkip_rx_ttak: 10-byte unicast TKIP TTAK for Rx
+ */
+struct iwl_mvm_add_sta_key_cmd_v1 {
+ struct iwl_mvm_add_sta_key_common common;
u8 tkip_rx_tsc_byte2;
u8 reserved;
__le16 tkip_rx_ttak[5];
} __packed; /* ADD_MODIFY_STA_KEY_API_S_VER_1 */
/**
+ * struct iwl_mvm_add_sta_key_cmd - add/modify sta key
+ * @common: see &struct iwl_mvm_add_sta_key_common
+ * @rx_mic_key: TKIP RX unicast or multicast key
+ * @tx_mic_key: TKIP TX key
+ * @transmit_seq_cnt: TSC, transmit packet number
+ */
+struct iwl_mvm_add_sta_key_cmd {
+ struct iwl_mvm_add_sta_key_common common;
+ __le64 rx_mic_key;
+ __le64 tx_mic_key;
+ __le64 transmit_seq_cnt;
+} __packed; /* ADD_MODIFY_STA_KEY_API_S_VER_2 */
+
+/**
* enum iwl_mvm_add_sta_rsp_status - status in the response to ADD_STA command
* @ADD_STA_SUCCESS: operation was executed successfully
* @ADD_STA_STATIONS_OVERLOAD: no room left in the fw's station table
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-tx.h b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-tx.h
index b38cc07..81b9891 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-tx.h
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api-tx.h
@@ -6,7 +6,7 @@
* GPL LICENSE SUMMARY
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -32,6 +32,7 @@
* BSD LICENSE
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -124,6 +125,20 @@
}; /* TX_FLAGS_BITS_API_S_VER_1 */
/**
+ * enum iwl_tx_cmd_flags - bitmasks for tx_flags in TX command for a000
+ * @IWL_TX_FLAGS_CMD_RATE: use rate from the TX command
+ * @IWL_TX_FLAGS_ENCRYPT_DIS: frame should not be encrypted, even if it belongs
+ * to a secured STA
+ * @IWL_TX_FLAGS_HIGH_PRI: high priority frame (like EAPOL) - can affect rate
+ * selection, retry limits and BT kill
+ */
+enum iwl_tx_cmd_flags {
+ IWL_TX_FLAGS_CMD_RATE = BIT(0),
+ IWL_TX_FLAGS_ENCRYPT_DIS = BIT(1),
+ IWL_TX_FLAGS_HIGH_PRI = BIT(2),
+}; /* TX_FLAGS_BITS_API_S_VER_3 */
+
+/**
* enum iwl_tx_pm_timeouts - pm timeout values in TX command
* @PM_FRAME_NONE: no need to suspend sleep mode
* @PM_FRAME_MGMT: fw suspend sleep mode for 100TU
@@ -159,7 +174,7 @@
TX_CMD_SEC_EXT = 0x04,
TX_CMD_SEC_GCMP = 0x05,
TX_CMD_SEC_KEY128 = 0x08,
- TX_CMD_SEC_KEY_FROM_TABLE = 0x08,
+ TX_CMD_SEC_KEY_FROM_TABLE = 0x10,
};
/* TODO: how does these values are OK with only 16 bit variable??? */
@@ -301,6 +316,31 @@
struct ieee80211_hdr hdr[0];
} __packed; /* TX_CMD_API_S_VER_6 */
+struct iwl_dram_sec_info {
+ __le32 pn_low;
+ __le16 pn_high;
+ __le16 aux_info;
+} __packed; /* DRAM_SEC_INFO_API_S_VER_1 */
+
+/**
+ * struct iwl_tx_cmd_gen2 - TX command struct to FW for a000 devices
+ * ( TX_CMD = 0x1c )
+ * @len: in bytes of the payload, see below for details
+ * @offload_assist: TX offload configuration
+ * @tx_flags: combination of &iwl_tx_cmd_flags
+ * @dram_info: FW internal DRAM storage
+ * @rate_n_flags: rate for *all* Tx attempts, if TX_CMD_FLG_STA_RATE_MSK is
+ * cleared. Combination of RATE_MCS_*
+ */
+struct iwl_tx_cmd_gen2 {
+ __le16 len;
+ __le16 offload_assist;
+ __le32 flags;
+ struct iwl_dram_sec_info dram_info;
+ __le32 rate_n_flags;
+ struct ieee80211_hdr hdr[0];
+} __packed; /* TX_CMD_API_S_VER_7 */
+
/*
* TX response related data
*/
@@ -508,9 +548,11 @@
* @tlc_info: TLC rate info
* @ra_tid: bits [3:0] = ra, bits [7:4] = tid
* @frame_ctrl: frame control
+ * @tx_queue: TX queue for this response
* @status: for non-agg: frame status TX_STATUS_*
* for agg: status of 1st frame, AGG_TX_STATE_*; other frame status fields
* follow this one, up to frame_count.
+ * For version 6 TX response isn't received for aggregation at all.
*
* After the array of statuses comes the SSN of the SCD. Look at
* %iwl_mvm_get_scd_ssn for more details.
@@ -537,9 +579,17 @@
u8 tlc_info;
u8 ra_tid;
__le16 frame_ctrl;
-
- struct agg_tx_status status;
-} __packed; /* TX_RSP_API_S_VER_3 */
+ union {
+ struct {
+ struct agg_tx_status status;
+ } v3;/* TX_RSP_API_S_VER_3 */
+ struct {
+ __le16 tx_queue;
+ __le16 reserved2;
+ struct agg_tx_status status;
+ } v6;
+ };
+} __packed; /* TX_RSP_API_S_VER_6 */
/**
* struct iwl_mvm_ba_notif - notifies about reception of BA
@@ -579,11 +629,14 @@
* struct iwl_mvm_compressed_ba_tfd - progress of a TFD queue
* @q_num: TFD queue number
* @tfd_index: Index of first un-acked frame in the TFD queue
+ * @scd_queue: For debug only - the physical queue the TFD queue is bound to
*/
struct iwl_mvm_compressed_ba_tfd {
- u8 q_num;
- u8 reserved;
+ __le16 q_num;
__le16 tfd_index;
+ u8 scd_queue;
+ u8 reserved;
+ __le16 reserved2;
} __packed; /* COMPRESSED_BA_TFD_API_S_VER_1 */
/**
@@ -635,6 +688,10 @@
* @tx_rate: the rate the aggregation was sent at
* @tfd_cnt: number of TFD-Q elements
* @ra_tid_cnt: number of RATID-Q elements
+ * @ba_tfd: array of TFD queue status updates. See &iwl_mvm_compressed_ba_tfd
+ * for details.
+ * @ra_tid: array of RA-TID queue status updates. For debug purposes only. See
+ * &iwl_mvm_compressed_ba_ratid for more details.
*/
struct iwl_mvm_compressed_ba_notif {
__le32 flags;
@@ -646,6 +703,7 @@
__le16 query_frame_cnt;
__le16 txed;
__le16 done;
+ __le16 reserved;
__le32 wireless_time;
__le32 tx_rate;
__le16 tfd_cnt;
@@ -754,25 +812,6 @@
__le16 reserved;
} __packed; /* TX_PATH_FLUSH_CMD_API_S_VER_1 */
-/**
- * iwl_mvm_get_scd_ssn - returns the SSN of the SCD
- * @tx_resp: the Tx response from the fw (agg or non-agg)
- *
- * When the fw sends an AMPDU, it fetches the MPDUs one after the other. Since
- * it can't know that everything will go well until the end of the AMPDU, it
- * can't know in advance the number of MPDUs that will be sent in the current
- * batch. This is why it writes the agg Tx response while it fetches the MPDUs.
- * Hence, it can't know in advance what the SSN of the SCD will be at the end
- * of the batch. This is why the SSN of the SCD is written at the end of the
- * whole struct at a variable offset. This function knows how to cope with the
- * variable offset and returns the SSN of the SCD.
- */
-static inline u32 iwl_mvm_get_scd_ssn(struct iwl_mvm_tx_resp *tx_resp)
-{
- return le32_to_cpup((__le32 *)&tx_resp->status +
- tx_resp->frame_count) & 0xfff;
-}
-
/* Available options for the SCD_QUEUE_CFG HCMD */
enum iwl_scd_cfg_actions {
SCD_CFG_DISABLE_QUEUE = 0x0,
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api.h b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api.h
index cf2b836..f545c5f 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/fw-api.h
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/fw-api.h
@@ -7,7 +7,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -34,7 +34,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -320,12 +320,14 @@
CMD_DTS_MEASUREMENT_TRIGGER_WIDE = 0x0,
CTDP_CONFIG_CMD = 0x03,
TEMP_REPORTING_THRESHOLDS_CMD = 0x04,
+ GEO_TX_POWER_LIMIT = 0x05,
CT_KILL_NOTIFICATION = 0xFE,
DTS_MEASUREMENT_NOTIF_WIDE = 0xFF,
};
enum iwl_system_subcmd_ids {
SHARED_MEM_CFG_CMD = 0x0,
+ INIT_EXTENDED_CFG_CMD = 0x03,
};
enum iwl_data_path_subcmd_ids {
@@ -345,9 +347,10 @@
NVM_ACCESS_COMPLETE = 0x0,
};
-enum iwl_fmac_debug_cmds {
+enum iwl_debug_cmds {
LMAC_RD_WR = 0x0,
UMAC_RD_WR = 0x1,
+ MFU_ASSERT_DUMP_NTF = 0xFE,
};
/* command groups */
@@ -673,10 +676,8 @@
/* Common PHY, MAC and Bindings definitions */
-
#define MAX_MACS_IN_BINDING (3)
#define MAX_BINDINGS (4)
-#define AUX_BINDING_INDEX (3)
/* Used to extract ID and color from the context dword */
#define FW_CTXT_ID_POS (0)
@@ -689,7 +690,7 @@
(_color << FW_CTXT_COLOR_POS))
/* Possible actions on PHYs, MACs and Bindings */
-enum {
+enum iwl_phy_ctxt_action {
FW_CTXT_ACTION_STUB = 0,
FW_CTXT_ACTION_ADD,
FW_CTXT_ACTION_MODIFY,
@@ -960,6 +961,7 @@
* @action: action to perform, one of FW_CTXT_ACTION_*
* @macs: array of MAC id and colors which belong to the binding
* @phy: PHY id and color which belongs to the binding
+ * @lmac_id: the lmac id the binding belongs to
*/
struct iwl_binding_cmd {
/* COMMON_INDEX_HDR_API_S_VER_1 */
@@ -968,7 +970,13 @@
/* BINDING_DATA_API_S_VER_1 */
__le32 macs[MAX_MACS_IN_BINDING];
__le32 phy;
-} __packed; /* BINDING_CMD_API_S_VER_1 */
+ /* BINDING_CMD_API_S_VER_1 */
+ __le32 lmac_id;
+} __packed; /* BINDING_CMD_API_S_VER_2 */
+
+#define IWL_BINDING_CMD_SIZE_V1 offsetof(struct iwl_binding_cmd, lmac_id)
+#define IWL_LMAC_24G_INDEX 0
+#define IWL_LMAC_5G_INDEX 1
/* The maximal number of fragments in the FW's schedule session */
#define IWL_MVM_MAX_QUOTA 128
@@ -990,6 +998,9 @@
* struct iwl_time_quota_cmd - configuration of time quota between bindings
* ( TIME_QUOTA_CMD = 0x2c )
* @quotas: allocations per binding
+ * Note: on non-CDB the fourth one is the auxilary mac and is
+ * essentially zero.
+ * On CDB the fourth one is a regular binding.
*/
struct iwl_time_quota_cmd {
struct iwl_time_quota_data quotas[MAX_BINDINGS];
@@ -1231,6 +1242,25 @@
} __packed; /*MFU_LOADER_NTFY_API_S_VER_2*/
/**
+ * struct iwl_mfu_assert_dump_notif - mfuart dump logs
+ * ( MFU_ASSERT_DUMP_NTF = 0xfe )
+ * @assert_id: mfuart assert id that cause the notif
+ * @curr_reset_num: number of asserts since uptime
+ * @index_num: current chunk id
+ * @parts_num: total number of chunks
+ * @data_size: number of data bytes sent
+ * @data: data buffer
+ */
+struct iwl_mfu_assert_dump_notif {
+ __le32 assert_id;
+ __le32 curr_reset_num;
+ __le16 index_num;
+ __le16 parts_num;
+ __le32 data_size;
+ __le32 data[0];
+} __packed; /*MFU_DUMP_ASSERT_API_S_VER_1*/
+
+/**
* struct iwl_set_calib_default_cmd - set default value for calibration.
* ( SET_CALIB_DEFAULT_CMD = 0x8e )
* @calib_index: the calibration to set value for
@@ -1998,19 +2028,48 @@
__le32 internal_txfifo_size[TX_FIFO_INTERNAL_MAX_NUM];
} __packed; /* SHARED_MEM_ALLOC_API_S_VER_2 */
+/**
+ * struct iwl_shared_mem_lmac_cfg - LMAC shared memory configuration
+ *
+ * @txfifo_addr: start addr of TXF0 (excluding the context table 0.5KB)
+ * @txfifo_size: size of TX FIFOs
+ * @rxfifo1_addr: RXF1 addr
+ * @rxfifo1_size: RXF1 size
+ */
+struct iwl_shared_mem_lmac_cfg {
+ __le32 txfifo_addr;
+ __le32 txfifo_size[TX_FIFO_MAX_NUM];
+ __le32 rxfifo1_addr;
+ __le32 rxfifo1_size;
+
+} __packed; /* SHARED_MEM_ALLOC_LMAC_API_S_VER_1 */
+
+/**
+ * Shared memory configuration information from the FW
+ *
+ * @shared_mem_addr: shared memory address
+ * @shared_mem_size: shared memory size
+ * @sample_buff_addr: internal sample (mon/adc) buff addr
+ * @sample_buff_size: internal sample buff size
+ * @rxfifo2_addr: start addr of RXF2
+ * @rxfifo2_size: size of RXF2
+ * @page_buff_addr: used by UMAC and performance debug (page miss analysis),
+ * when paging is not supported this should be 0
+ * @page_buff_size: size of %page_buff_addr
+ * @lmac_num: number of LMACs (1 or 2)
+ * @lmac_smem: per - LMAC smem data
+ */
struct iwl_shared_mem_cfg {
__le32 shared_mem_addr;
__le32 shared_mem_size;
__le32 sample_buff_addr;
__le32 sample_buff_size;
- __le32 txfifo_addr;
- __le32 txfifo_size[TX_FIFO_MAX_NUM];
- __le32 rxfifo_size[RX_FIFO_MAX_NUM];
+ __le32 rxfifo2_addr;
+ __le32 rxfifo2_size;
__le32 page_buff_addr;
__le32 page_buff_size;
- __le32 rxfifo_addr;
- __le32 internal_txfifo_addr;
- __le32 internal_txfifo_size[TX_FIFO_INTERNAL_MAX_NUM];
+ __le32 lmac_num;
+ struct iwl_shared_mem_lmac_cfg lmac_smem[2];
} __packed; /* SHARED_MEM_ALLOC_API_S_VER_3 */
/**
@@ -2178,4 +2237,26 @@
__le32 reserved;
} __packed; /* NVM_ACCESS_COMPLETE_CMD_API_S_VER_1 */
+/**
+ * enum iwl_extended_cfg_flag - commands driver may send before
+ * finishing init flow
+ * @IWL_INIT_DEBUG_CFG: driver is going to send debug config command
+ * @IWL_INIT_NVM: driver is going to send NVM_ACCESS commands
+ * @IWL_INIT_PHY: driver is going to send PHY_DB commands
+ */
+enum iwl_extended_cfg_flags {
+ IWL_INIT_DEBUG_CFG,
+ IWL_INIT_NVM,
+ IWL_INIT_PHY,
+};
+
+/**
+ * struct iwl_extended_cfg_cmd - mark what commands ucode should wait for
+ * before finishing init flows
+ * @init_flags: values from iwl_extended_cfg_flags
+ */
+struct iwl_init_extended_cfg_cmd {
+ __le32 init_flags;
+} __packed; /* INIT_EXTENDED_CFG_CMD_API_S_VER_1 */
+
#endif /* __fw_api_h__ */
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/fw-dbg.c b/drivers/net/wireless/intel/iwlwifi/mvm/fw-dbg.c
index a027b11..7b86a4f 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/fw-dbg.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/fw-dbg.c
@@ -7,7 +7,7 @@
*
* Copyright(c) 2008 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2015 - 2016 Intel Deutschland GmbH
+ * Copyright(c) 2015 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -32,7 +32,7 @@
*
* Copyright(c) 2005 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2015 - 2016 Intel Deutschland GmbH
+ * Copyright(c) 2015 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -99,10 +99,120 @@
iwl_trans_release_nic_access(mvm->trans, &flags);
}
+static void iwl_mvm_dump_rxf(struct iwl_mvm *mvm,
+ struct iwl_fw_error_dump_data **dump_data,
+ int size, u32 offset, int fifo_num)
+{
+ struct iwl_fw_error_dump_fifo *fifo_hdr;
+ u32 *fifo_data;
+ u32 fifo_len;
+ int i;
+
+ fifo_hdr = (void *)(*dump_data)->data;
+ fifo_data = (void *)fifo_hdr->data;
+ fifo_len = size;
+
+ /* No need to try to read the data if the length is 0 */
+ if (fifo_len == 0)
+ return;
+
+ /* Add a TLV for the RXF */
+ (*dump_data)->type = cpu_to_le32(IWL_FW_ERROR_DUMP_RXF);
+ (*dump_data)->len = cpu_to_le32(fifo_len + sizeof(*fifo_hdr));
+
+ fifo_hdr->fifo_num = cpu_to_le32(fifo_num);
+ fifo_hdr->available_bytes =
+ cpu_to_le32(iwl_trans_read_prph(mvm->trans,
+ RXF_RD_D_SPACE + offset));
+ fifo_hdr->wr_ptr =
+ cpu_to_le32(iwl_trans_read_prph(mvm->trans,
+ RXF_RD_WR_PTR + offset));
+ fifo_hdr->rd_ptr =
+ cpu_to_le32(iwl_trans_read_prph(mvm->trans,
+ RXF_RD_RD_PTR + offset));
+ fifo_hdr->fence_ptr =
+ cpu_to_le32(iwl_trans_read_prph(mvm->trans,
+ RXF_RD_FENCE_PTR + offset));
+ fifo_hdr->fence_mode =
+ cpu_to_le32(iwl_trans_read_prph(mvm->trans,
+ RXF_SET_FENCE_MODE + offset));
+
+ /* Lock fence */
+ iwl_trans_write_prph(mvm->trans, RXF_SET_FENCE_MODE + offset, 0x1);
+ /* Set fence pointer to the same place like WR pointer */
+ iwl_trans_write_prph(mvm->trans, RXF_LD_WR2FENCE + offset, 0x1);
+ /* Set fence offset */
+ iwl_trans_write_prph(mvm->trans,
+ RXF_LD_FENCE_OFFSET_ADDR + offset, 0x0);
+
+ /* Read FIFO */
+ fifo_len /= sizeof(u32); /* Size in DWORDS */
+ for (i = 0; i < fifo_len; i++)
+ fifo_data[i] = iwl_trans_read_prph(mvm->trans,
+ RXF_FIFO_RD_FENCE_INC +
+ offset);
+ *dump_data = iwl_fw_error_next_data(*dump_data);
+}
+
+static void iwl_mvm_dump_txf(struct iwl_mvm *mvm,
+ struct iwl_fw_error_dump_data **dump_data,
+ int size, u32 offset, int fifo_num)
+{
+ struct iwl_fw_error_dump_fifo *fifo_hdr;
+ u32 *fifo_data;
+ u32 fifo_len;
+ int i;
+
+ fifo_hdr = (void *)(*dump_data)->data;
+ fifo_data = (void *)fifo_hdr->data;
+ fifo_len = size;
+
+ /* No need to try to read the data if the length is 0 */
+ if (fifo_len == 0)
+ return;
+
+ /* Add a TLV for the FIFO */
+ (*dump_data)->type = cpu_to_le32(IWL_FW_ERROR_DUMP_TXF);
+ (*dump_data)->len = cpu_to_le32(fifo_len + sizeof(*fifo_hdr));
+
+ fifo_hdr->fifo_num = cpu_to_le32(fifo_num);
+ fifo_hdr->available_bytes =
+ cpu_to_le32(iwl_trans_read_prph(mvm->trans,
+ TXF_FIFO_ITEM_CNT + offset));
+ fifo_hdr->wr_ptr =
+ cpu_to_le32(iwl_trans_read_prph(mvm->trans,
+ TXF_WR_PTR + offset));
+ fifo_hdr->rd_ptr =
+ cpu_to_le32(iwl_trans_read_prph(mvm->trans,
+ TXF_RD_PTR + offset));
+ fifo_hdr->fence_ptr =
+ cpu_to_le32(iwl_trans_read_prph(mvm->trans,
+ TXF_FENCE_PTR + offset));
+ fifo_hdr->fence_mode =
+ cpu_to_le32(iwl_trans_read_prph(mvm->trans,
+ TXF_LOCK_FENCE + offset));
+
+ /* Set the TXF_READ_MODIFY_ADDR to TXF_WR_PTR */
+ iwl_trans_write_prph(mvm->trans, TXF_READ_MODIFY_ADDR + offset,
+ TXF_WR_PTR + offset);
+
+ /* Dummy-read to advance the read pointer to the head */
+ iwl_trans_read_prph(mvm->trans, TXF_READ_MODIFY_DATA + offset);
+
+ /* Read FIFO */
+ fifo_len /= sizeof(u32); /* Size in DWORDS */
+ for (i = 0; i < fifo_len; i++)
+ fifo_data[i] = iwl_trans_read_prph(mvm->trans,
+ TXF_READ_MODIFY_DATA +
+ offset);
+ *dump_data = iwl_fw_error_next_data(*dump_data);
+}
+
static void iwl_mvm_dump_fifos(struct iwl_mvm *mvm,
struct iwl_fw_error_dump_data **dump_data)
{
struct iwl_fw_error_dump_fifo *fifo_hdr;
+ struct iwl_mvm_shared_mem_cfg *cfg = &mvm->smem_cfg;
u32 *fifo_data;
u32 fifo_len;
unsigned long flags;
@@ -111,126 +221,47 @@
if (!iwl_trans_grab_nic_access(mvm->trans, &flags))
return;
- /* Pull RXF data from all RXFs */
- for (i = 0; i < ARRAY_SIZE(mvm->shared_mem_cfg.rxfifo_size); i++) {
- /*
- * Keep aside the additional offset that might be needed for
- * next RXF
- */
- u32 offset_diff = RXF_DIFF_FROM_PREV * i;
+ /* Pull RXF1 */
+ iwl_mvm_dump_rxf(mvm, dump_data, cfg->lmac[0].rxfifo1_size, 0, 0);
+ /* Pull RXF2 */
+ iwl_mvm_dump_rxf(mvm, dump_data, cfg->rxfifo2_size,
+ RXF_DIFF_FROM_PREV, 1);
+ /* Pull LMAC2 RXF1 */
+ if (mvm->smem_cfg.num_lmacs > 1)
+ iwl_mvm_dump_rxf(mvm, dump_data, cfg->lmac[1].rxfifo1_size,
+ LMAC2_PRPH_OFFSET, 2);
- fifo_hdr = (void *)(*dump_data)->data;
- fifo_data = (void *)fifo_hdr->data;
- fifo_len = mvm->shared_mem_cfg.rxfifo_size[i];
-
- /* No need to try to read the data if the length is 0 */
- if (fifo_len == 0)
- continue;
-
- /* Add a TLV for the RXF */
- (*dump_data)->type = cpu_to_le32(IWL_FW_ERROR_DUMP_RXF);
- (*dump_data)->len = cpu_to_le32(fifo_len + sizeof(*fifo_hdr));
-
- fifo_hdr->fifo_num = cpu_to_le32(i);
- fifo_hdr->available_bytes =
- cpu_to_le32(iwl_trans_read_prph(mvm->trans,
- RXF_RD_D_SPACE +
- offset_diff));
- fifo_hdr->wr_ptr =
- cpu_to_le32(iwl_trans_read_prph(mvm->trans,
- RXF_RD_WR_PTR +
- offset_diff));
- fifo_hdr->rd_ptr =
- cpu_to_le32(iwl_trans_read_prph(mvm->trans,
- RXF_RD_RD_PTR +
- offset_diff));
- fifo_hdr->fence_ptr =
- cpu_to_le32(iwl_trans_read_prph(mvm->trans,
- RXF_RD_FENCE_PTR +
- offset_diff));
- fifo_hdr->fence_mode =
- cpu_to_le32(iwl_trans_read_prph(mvm->trans,
- RXF_SET_FENCE_MODE +
- offset_diff));
-
- /* Lock fence */
- iwl_trans_write_prph(mvm->trans,
- RXF_SET_FENCE_MODE + offset_diff, 0x1);
- /* Set fence pointer to the same place like WR pointer */
- iwl_trans_write_prph(mvm->trans,
- RXF_LD_WR2FENCE + offset_diff, 0x1);
- /* Set fence offset */
- iwl_trans_write_prph(mvm->trans,
- RXF_LD_FENCE_OFFSET_ADDR + offset_diff,
- 0x0);
-
- /* Read FIFO */
- fifo_len /= sizeof(u32); /* Size in DWORDS */
- for (j = 0; j < fifo_len; j++)
- fifo_data[j] = iwl_trans_read_prph(mvm->trans,
- RXF_FIFO_RD_FENCE_INC +
- offset_diff);
- *dump_data = iwl_fw_error_next_data(*dump_data);
- }
-
- /* Pull TXF data from all TXFs */
- for (i = 0; i < ARRAY_SIZE(mvm->shared_mem_cfg.txfifo_size); i++) {
+ /* Pull TXF data from LMAC1 */
+ for (i = 0; i < mvm->smem_cfg.num_txfifo_entries; i++) {
/* Mark the number of TXF we're pulling now */
iwl_trans_write_prph(mvm->trans, TXF_LARC_NUM, i);
+ iwl_mvm_dump_txf(mvm, dump_data, cfg->lmac[0].txfifo_size[i],
+ 0, i);
+ }
- fifo_hdr = (void *)(*dump_data)->data;
- fifo_data = (void *)fifo_hdr->data;
- fifo_len = mvm->shared_mem_cfg.txfifo_size[i];
-
- /* No need to try to read the data if the length is 0 */
- if (fifo_len == 0)
- continue;
-
- /* Add a TLV for the FIFO */
- (*dump_data)->type = cpu_to_le32(IWL_FW_ERROR_DUMP_TXF);
- (*dump_data)->len = cpu_to_le32(fifo_len + sizeof(*fifo_hdr));
-
- fifo_hdr->fifo_num = cpu_to_le32(i);
- fifo_hdr->available_bytes =
- cpu_to_le32(iwl_trans_read_prph(mvm->trans,
- TXF_FIFO_ITEM_CNT));
- fifo_hdr->wr_ptr =
- cpu_to_le32(iwl_trans_read_prph(mvm->trans,
- TXF_WR_PTR));
- fifo_hdr->rd_ptr =
- cpu_to_le32(iwl_trans_read_prph(mvm->trans,
- TXF_RD_PTR));
- fifo_hdr->fence_ptr =
- cpu_to_le32(iwl_trans_read_prph(mvm->trans,
- TXF_FENCE_PTR));
- fifo_hdr->fence_mode =
- cpu_to_le32(iwl_trans_read_prph(mvm->trans,
- TXF_LOCK_FENCE));
-
- /* Set the TXF_READ_MODIFY_ADDR to TXF_WR_PTR */
- iwl_trans_write_prph(mvm->trans, TXF_READ_MODIFY_ADDR,
- TXF_WR_PTR);
-
- /* Dummy-read to advance the read pointer to the head */
- iwl_trans_read_prph(mvm->trans, TXF_READ_MODIFY_DATA);
-
- /* Read FIFO */
- fifo_len /= sizeof(u32); /* Size in DWORDS */
- for (j = 0; j < fifo_len; j++)
- fifo_data[j] = iwl_trans_read_prph(mvm->trans,
- TXF_READ_MODIFY_DATA);
- *dump_data = iwl_fw_error_next_data(*dump_data);
+ /* Pull TXF data from LMAC2 */
+ if (mvm->smem_cfg.num_lmacs > 1) {
+ for (i = 0; i < mvm->smem_cfg.num_txfifo_entries; i++) {
+ /* Mark the number of TXF we're pulling now */
+ iwl_trans_write_prph(mvm->trans,
+ TXF_LARC_NUM + LMAC2_PRPH_OFFSET,
+ i);
+ iwl_mvm_dump_txf(mvm, dump_data,
+ cfg->lmac[1].txfifo_size[i],
+ LMAC2_PRPH_OFFSET,
+ i + cfg->num_txfifo_entries);
+ }
}
if (fw_has_capa(&mvm->fw->ucode_capa,
IWL_UCODE_TLV_CAPA_EXTEND_SHARED_MEM_CFG)) {
/* Pull UMAC internal TXF data from all TXFs */
for (i = 0;
- i < ARRAY_SIZE(mvm->shared_mem_cfg.internal_txfifo_size);
+ i < ARRAY_SIZE(mvm->smem_cfg.internal_txfifo_size);
i++) {
fifo_hdr = (void *)(*dump_data)->data;
fifo_data = (void *)fifo_hdr->data;
- fifo_len = mvm->shared_mem_cfg.internal_txfifo_size[i];
+ fifo_len = mvm->smem_cfg.internal_txfifo_size[i];
/* No need to try to read the data if the length is 0 */
if (fifo_len == 0)
@@ -246,7 +277,7 @@
/* Mark the number of TXF we're pulling now */
iwl_trans_write_prph(mvm->trans, TXF_CPU2_NUM, i +
- ARRAY_SIZE(mvm->shared_mem_cfg.txfifo_size));
+ mvm->smem_cfg.num_txfifo_entries);
fifo_hdr->available_bytes =
cpu_to_le32(iwl_trans_read_prph(mvm->trans,
@@ -553,31 +584,45 @@
/* reading RXF/TXF sizes */
if (test_bit(STATUS_FW_ERROR, &mvm->trans->status)) {
- struct iwl_mvm_shared_mem_cfg *mem_cfg = &mvm->shared_mem_cfg;
+ struct iwl_mvm_shared_mem_cfg *mem_cfg = &mvm->smem_cfg;
fifo_data_len = 0;
- /* Count RXF size */
- for (i = 0; i < ARRAY_SIZE(mem_cfg->rxfifo_size); i++) {
- if (!mem_cfg->rxfifo_size[i])
- continue;
-
+ /* Count RXF2 size */
+ if (mem_cfg->rxfifo2_size) {
/* Add header info */
- fifo_data_len += mem_cfg->rxfifo_size[i] +
+ fifo_data_len += mem_cfg->rxfifo2_size +
sizeof(*dump_data) +
sizeof(struct iwl_fw_error_dump_fifo);
}
- for (i = 0; i < mem_cfg->num_txfifo_entries; i++) {
- if (!mem_cfg->txfifo_size[i])
+ /* Count RXF1 sizes */
+ for (i = 0; i < mem_cfg->num_lmacs; i++) {
+ if (!mem_cfg->lmac[i].rxfifo1_size)
continue;
/* Add header info */
- fifo_data_len += mem_cfg->txfifo_size[i] +
+ fifo_data_len += mem_cfg->lmac[i].rxfifo1_size +
sizeof(*dump_data) +
sizeof(struct iwl_fw_error_dump_fifo);
}
+ /* Count TXF sizes */
+ for (i = 0; i < mem_cfg->num_lmacs; i++) {
+ int j;
+
+ for (j = 0; j < mem_cfg->num_txfifo_entries; j++) {
+ if (!mem_cfg->lmac[i].txfifo_size[j])
+ continue;
+
+ /* Add header info */
+ fifo_data_len +=
+ mem_cfg->lmac[i].txfifo_size[j] +
+ sizeof(*dump_data) +
+ sizeof(struct iwl_fw_error_dump_fifo);
+ }
+ }
+
if (fw_has_capa(&mvm->fw->ucode_capa,
IWL_UCODE_TLV_CAPA_EXTEND_SHARED_MEM_CFG)) {
for (i = 0;
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/fw.c b/drivers/net/wireless/intel/iwlwifi/mvm/fw.c
index 45cb4f4..e6c9528 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/fw.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/fw.c
@@ -7,7 +7,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -34,6 +34,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -271,6 +272,27 @@
return 0;
}
+void iwl_mvm_mfu_assert_dump_notif(struct iwl_mvm *mvm,
+ struct iwl_rx_cmd_buffer *rxb)
+{
+ struct iwl_rx_packet *pkt = rxb_addr(rxb);
+ struct iwl_mfu_assert_dump_notif *mfu_dump_notif = (void *)pkt->data;
+ __le32 *dump_data = mfu_dump_notif->data;
+ int n_words = le32_to_cpu(mfu_dump_notif->data_size) / sizeof(__le32);
+ int i;
+
+ if (mfu_dump_notif->index_num == 0)
+ IWL_INFO(mvm, "MFUART assert id 0x%x occurred\n",
+ le32_to_cpu(mfu_dump_notif->assert_id));
+
+ for (i = 0; i < n_words; i++)
+ IWL_DEBUG_INFO(mvm,
+ "MFUART assert dump, dword %u: 0x%08x\n",
+ le16_to_cpu(mfu_dump_notif->index_num) *
+ n_words + i,
+ le32_to_cpu(dump_data[i]));
+}
+
static int iwl_alloc_fw_paging_mem(struct iwl_mvm *mvm,
const struct fw_img *image)
{
@@ -617,11 +639,18 @@
ret = iwl_wait_notification(&mvm->notif_wait, &alive_wait,
MVM_UCODE_ALIVE_TIMEOUT);
if (ret) {
- if (mvm->trans->cfg->device_family == IWL_DEVICE_FAMILY_8000)
+ struct iwl_trans *trans = mvm->trans;
+
+ if (trans->cfg->gen2)
IWL_ERR(mvm,
"SecBoot CPU1 Status: 0x%x, CPU2 Status: 0x%x\n",
- iwl_read_prph(mvm->trans, SB_CPU_1_STATUS),
- iwl_read_prph(mvm->trans, SB_CPU_2_STATUS));
+ iwl_read_prph(trans, UMAG_SB_CPU_1_STATUS),
+ iwl_read_prph(trans, UMAG_SB_CPU_2_STATUS));
+ else if (trans->cfg->device_family == IWL_DEVICE_FAMILY_8000)
+ IWL_ERR(mvm,
+ "SecBoot CPU1 Status: 0x%x, CPU2 Status: 0x%x\n",
+ iwl_read_prph(trans, SB_CPU_1_STATUS),
+ iwl_read_prph(trans, SB_CPU_2_STATUS));
mvm->cur_ucode = old_type;
return ret;
}
@@ -669,6 +698,82 @@
return 0;
}
+static int iwl_run_unified_mvm_ucode(struct iwl_mvm *mvm, bool read_nvm)
+{
+ struct iwl_notification_wait init_wait;
+ struct iwl_nvm_access_complete_cmd nvm_complete = {};
+ struct iwl_init_extended_cfg_cmd init_cfg = {
+ .init_flags = cpu_to_le32(BIT(IWL_INIT_NVM)),
+ };
+ static const u16 init_complete[] = {
+ INIT_COMPLETE_NOTIF,
+ };
+ int ret;
+
+ lockdep_assert_held(&mvm->mutex);
+
+ iwl_init_notification_wait(&mvm->notif_wait,
+ &init_wait,
+ init_complete,
+ ARRAY_SIZE(init_complete),
+ iwl_wait_init_complete,
+ NULL);
+
+ /* Will also start the device */
+ ret = iwl_mvm_load_ucode_wait_alive(mvm, IWL_UCODE_REGULAR);
+ if (ret) {
+ IWL_ERR(mvm, "Failed to start RT ucode: %d\n", ret);
+ goto error;
+ }
+
+ /* Send init config command to mark that we are sending NVM access
+ * commands
+ */
+ ret = iwl_mvm_send_cmd_pdu(mvm, WIDE_ID(SYSTEM_GROUP,
+ INIT_EXTENDED_CFG_CMD), 0,
+ sizeof(init_cfg), &init_cfg);
+ if (ret) {
+ IWL_ERR(mvm, "Failed to run init config command: %d\n",
+ ret);
+ goto error;
+ }
+
+ /* Read the NVM only at driver load time, no need to do this twice */
+ if (read_nvm) {
+ /* Read nvm */
+ ret = iwl_nvm_init(mvm, true);
+ if (ret) {
+ IWL_ERR(mvm, "Failed to read NVM: %d\n", ret);
+ goto error;
+ }
+ }
+
+ /* In case we read the NVM from external file, load it to the NIC */
+ if (mvm->nvm_file_name)
+ iwl_mvm_load_nvm_to_nic(mvm);
+
+ ret = iwl_nvm_check_version(mvm->nvm_data, mvm->trans);
+ if (WARN_ON(ret))
+ goto error;
+
+ ret = iwl_mvm_send_cmd_pdu(mvm, WIDE_ID(REGULATORY_AND_NVM_GROUP,
+ NVM_ACCESS_COMPLETE), 0,
+ sizeof(nvm_complete), &nvm_complete);
+ if (ret) {
+ IWL_ERR(mvm, "Failed to run complete NVM access: %d\n",
+ ret);
+ goto error;
+ }
+
+ /* We wait for the INIT complete notification */
+ return iwl_wait_notification(&mvm->notif_wait, &init_wait,
+ MVM_UCODE_ALIVE_TIMEOUT);
+
+error:
+ iwl_remove_notification(&mvm->notif_wait, &init_wait);
+ return ret;
+}
+
static int iwl_send_phy_cfg_cmd(struct iwl_mvm *mvm)
{
struct iwl_phy_cfg_cmd phy_cfg_cmd;
@@ -697,6 +802,9 @@
};
int ret;
+ if (iwl_mvm_has_new_tx_api(mvm))
+ return iwl_run_unified_mvm_ucode(mvm, true);
+
lockdep_assert_held(&mvm->mutex);
if (WARN_ON_ONCE(mvm->calibrating))
@@ -803,97 +911,31 @@
return ret;
}
-int iwl_run_unified_mvm_ucode(struct iwl_mvm *mvm, bool read_nvm)
-{
- struct iwl_notification_wait init_wait;
- struct iwl_nvm_access_complete_cmd nvm_complete = {};
- static const u16 init_complete[] = {
- INIT_COMPLETE_NOTIF,
- };
- int ret;
-
- lockdep_assert_held(&mvm->mutex);
-
- iwl_init_notification_wait(&mvm->notif_wait,
- &init_wait,
- init_complete,
- ARRAY_SIZE(init_complete),
- iwl_wait_init_complete,
- NULL);
-
- /* Will also start the device */
- ret = iwl_mvm_load_ucode_wait_alive(mvm, IWL_UCODE_REGULAR);
- if (ret) {
- IWL_ERR(mvm, "Failed to start RT ucode: %d\n", ret);
- goto error;
- }
-
- /* TODO: remove when integrating context info */
- ret = iwl_mvm_init_paging(mvm);
- if (ret) {
- IWL_ERR(mvm, "Failed to init paging: %d\n",
- ret);
- goto error;
- }
-
- /* Read the NVM only at driver load time, no need to do this twice */
- if (read_nvm) {
- /* Read nvm */
- ret = iwl_nvm_init(mvm, true);
- if (ret) {
- IWL_ERR(mvm, "Failed to read NVM: %d\n", ret);
- goto error;
- }
- }
-
- /* In case we read the NVM from external file, load it to the NIC */
- if (mvm->nvm_file_name)
- iwl_mvm_load_nvm_to_nic(mvm);
-
- ret = iwl_nvm_check_version(mvm->nvm_data, mvm->trans);
- if (WARN_ON(ret))
- goto error;
-
- ret = iwl_mvm_send_cmd_pdu(mvm, WIDE_ID(REGULATORY_AND_NVM_GROUP,
- NVM_ACCESS_COMPLETE), 0,
- sizeof(nvm_complete), &nvm_complete);
- if (ret) {
- IWL_ERR(mvm, "Failed to run complete NVM access: %d\n",
- ret);
- goto error;
- }
-
- /* We wait for the INIT complete notification */
- return iwl_wait_notification(&mvm->notif_wait, &init_wait,
- MVM_UCODE_ALIVE_TIMEOUT);
-
-error:
- iwl_remove_notification(&mvm->notif_wait, &init_wait);
- return ret;
-}
-
static void iwl_mvm_parse_shared_mem_a000(struct iwl_mvm *mvm,
struct iwl_rx_packet *pkt)
{
struct iwl_shared_mem_cfg *mem_cfg = (void *)pkt->data;
- int i;
+ int i, lmac;
+ int lmac_num = le32_to_cpu(mem_cfg->lmac_num);
- mvm->shared_mem_cfg.num_txfifo_entries =
- ARRAY_SIZE(mvm->shared_mem_cfg.txfifo_size);
- for (i = 0; i < ARRAY_SIZE(mem_cfg->txfifo_size); i++)
- mvm->shared_mem_cfg.txfifo_size[i] =
- le32_to_cpu(mem_cfg->txfifo_size[i]);
- for (i = 0; i < ARRAY_SIZE(mvm->shared_mem_cfg.rxfifo_size); i++)
- mvm->shared_mem_cfg.rxfifo_size[i] =
- le32_to_cpu(mem_cfg->rxfifo_size[i]);
+ if (WARN_ON(lmac_num > ARRAY_SIZE(mem_cfg->lmac_smem)))
+ return;
- BUILD_BUG_ON(sizeof(mvm->shared_mem_cfg.internal_txfifo_size) !=
- sizeof(mem_cfg->internal_txfifo_size));
+ mvm->smem_cfg.num_lmacs = lmac_num;
+ mvm->smem_cfg.num_txfifo_entries =
+ ARRAY_SIZE(mem_cfg->lmac_smem[0].txfifo_size);
+ mvm->smem_cfg.rxfifo2_size = le32_to_cpu(mem_cfg->rxfifo2_size);
- for (i = 0; i < ARRAY_SIZE(mvm->shared_mem_cfg.internal_txfifo_size);
- i++)
- mvm->shared_mem_cfg.internal_txfifo_size[i] =
- le32_to_cpu(mem_cfg->internal_txfifo_size[i]);
+ for (lmac = 0; lmac < lmac_num; lmac++) {
+ struct iwl_shared_mem_lmac_cfg *lmac_cfg =
+ &mem_cfg->lmac_smem[lmac];
+
+ for (i = 0; i < ARRAY_SIZE(lmac_cfg->txfifo_size); i++)
+ mvm->smem_cfg.lmac[lmac].txfifo_size[i] =
+ le32_to_cpu(lmac_cfg->txfifo_size[i]);
+ mvm->smem_cfg.lmac[lmac].rxfifo1_size =
+ le32_to_cpu(lmac_cfg->rxfifo1_size);
+ }
}
static void iwl_mvm_parse_shared_mem(struct iwl_mvm *mvm,
@@ -902,25 +944,27 @@
struct iwl_shared_mem_cfg_v1 *mem_cfg = (void *)pkt->data;
int i;
- mvm->shared_mem_cfg.num_txfifo_entries =
- ARRAY_SIZE(mvm->shared_mem_cfg.txfifo_size);
+ mvm->smem_cfg.num_lmacs = 1;
+
+ mvm->smem_cfg.num_txfifo_entries = ARRAY_SIZE(mem_cfg->txfifo_size);
for (i = 0; i < ARRAY_SIZE(mem_cfg->txfifo_size); i++)
- mvm->shared_mem_cfg.txfifo_size[i] =
+ mvm->smem_cfg.lmac[0].txfifo_size[i] =
le32_to_cpu(mem_cfg->txfifo_size[i]);
- for (i = 0; i < ARRAY_SIZE(mvm->shared_mem_cfg.rxfifo_size); i++)
- mvm->shared_mem_cfg.rxfifo_size[i] =
- le32_to_cpu(mem_cfg->rxfifo_size[i]);
+
+ mvm->smem_cfg.lmac[0].rxfifo1_size =
+ le32_to_cpu(mem_cfg->rxfifo_size[0]);
+ mvm->smem_cfg.rxfifo2_size = le32_to_cpu(mem_cfg->rxfifo_size[1]);
/* new API has more data, from rxfifo_addr field and on */
if (fw_has_capa(&mvm->fw->ucode_capa,
IWL_UCODE_TLV_CAPA_EXTEND_SHARED_MEM_CFG)) {
- BUILD_BUG_ON(sizeof(mvm->shared_mem_cfg.internal_txfifo_size) !=
+ BUILD_BUG_ON(sizeof(mvm->smem_cfg.internal_txfifo_size) !=
sizeof(mem_cfg->internal_txfifo_size));
for (i = 0;
- i < ARRAY_SIZE(mvm->shared_mem_cfg.internal_txfifo_size);
+ i < ARRAY_SIZE(mvm->smem_cfg.internal_txfifo_size);
i++)
- mvm->shared_mem_cfg.internal_txfifo_size[i] =
+ mvm->smem_cfg.internal_txfifo_size[i] =
le32_to_cpu(mem_cfg->internal_txfifo_size[i]);
}
}
@@ -969,85 +1013,94 @@
sizeof(cmd), &cmd);
}
-#define ACPI_WRDS_METHOD "WRDS"
-#define ACPI_WRDS_WIFI (0x07)
-#define ACPI_WRDS_TABLE_SIZE 10
-
-struct iwl_mvm_sar_table {
- bool enabled;
- u8 values[ACPI_WRDS_TABLE_SIZE];
-};
-
#ifdef CONFIG_ACPI
-static int iwl_mvm_sar_get_wrds(struct iwl_mvm *mvm, union acpi_object *wrds,
- struct iwl_mvm_sar_table *sar_table)
+#define ACPI_WRDS_METHOD "WRDS"
+#define ACPI_EWRD_METHOD "EWRD"
+#define ACPI_WGDS_METHOD "WGDS"
+#define ACPI_WIFI_DOMAIN (0x07)
+#define ACPI_WRDS_WIFI_DATA_SIZE (IWL_MVM_SAR_TABLE_SIZE + 2)
+#define ACPI_EWRD_WIFI_DATA_SIZE ((IWL_MVM_SAR_PROFILE_NUM - 1) * \
+ IWL_MVM_SAR_TABLE_SIZE + 3)
+#define ACPI_WGDS_WIFI_DATA_SIZE 18
+#define ACPI_WGDS_NUM_BANDS 2
+#define ACPI_WGDS_TABLE_SIZE 3
+
+static int iwl_mvm_sar_set_profile(struct iwl_mvm *mvm,
+ union acpi_object *table,
+ struct iwl_mvm_sar_profile *profile,
+ bool enabled)
{
- union acpi_object *data_pkg;
- u32 i;
+ int i;
- /* We need at least two packages, one for the revision and one
- * for the data itself. Also check that the revision is valid
- * (i.e. it is an integer set to 0).
- */
- if (wrds->type != ACPI_TYPE_PACKAGE ||
- wrds->package.count < 2 ||
- wrds->package.elements[0].type != ACPI_TYPE_INTEGER ||
- wrds->package.elements[0].integer.value != 0) {
- IWL_DEBUG_RADIO(mvm, "Unsupported wrds structure\n");
- return -EINVAL;
- }
+ profile->enabled = enabled;
- /* loop through all the packages to find the one for WiFi */
- for (i = 1; i < wrds->package.count; i++) {
- union acpi_object *domain;
-
- data_pkg = &wrds->package.elements[i];
-
- /* Skip anything that is not a package with the right
- * amount of elements (i.e. domain_type,
- * enabled/disabled plus the sar table size.
- */
- if (data_pkg->type != ACPI_TYPE_PACKAGE ||
- data_pkg->package.count != ACPI_WRDS_TABLE_SIZE + 2)
- continue;
-
- domain = &data_pkg->package.elements[0];
- if (domain->type == ACPI_TYPE_INTEGER &&
- domain->integer.value == ACPI_WRDS_WIFI)
- break;
-
- data_pkg = NULL;
- }
-
- if (!data_pkg)
- return -ENOENT;
-
- if (data_pkg->package.elements[1].type != ACPI_TYPE_INTEGER)
- return -EINVAL;
-
- sar_table->enabled = !!(data_pkg->package.elements[1].integer.value);
-
- for (i = 0; i < ACPI_WRDS_TABLE_SIZE; i++) {
- union acpi_object *entry;
-
- entry = &data_pkg->package.elements[i + 2];
- if ((entry->type != ACPI_TYPE_INTEGER) ||
- (entry->integer.value > U8_MAX))
+ for (i = 0; i < IWL_MVM_SAR_TABLE_SIZE; i++) {
+ if ((table[i].type != ACPI_TYPE_INTEGER) ||
+ (table[i].integer.value > U8_MAX))
return -EINVAL;
- sar_table->values[i] = entry->integer.value;
+ profile->table[i] = table[i].integer.value;
}
return 0;
}
-static int iwl_mvm_sar_get_table(struct iwl_mvm *mvm,
- struct iwl_mvm_sar_table *sar_table)
+static union acpi_object *iwl_mvm_sar_find_wifi_pkg(struct iwl_mvm *mvm,
+ union acpi_object *data,
+ int data_size)
{
+ int i;
+ union acpi_object *wifi_pkg;
+
+ /*
+ * We need at least two packages, one for the revision and one
+ * for the data itself. Also check that the revision is valid
+ * (i.e. it is an integer set to 0).
+ */
+ if (data->type != ACPI_TYPE_PACKAGE ||
+ data->package.count < 2 ||
+ data->package.elements[0].type != ACPI_TYPE_INTEGER ||
+ data->package.elements[0].integer.value != 0) {
+ IWL_DEBUG_RADIO(mvm, "Unsupported packages structure\n");
+ return ERR_PTR(-EINVAL);
+ }
+
+ /* loop through all the packages to find the one for WiFi */
+ for (i = 1; i < data->package.count; i++) {
+ union acpi_object *domain;
+
+ wifi_pkg = &data->package.elements[i];
+
+ /* Skip anything that is not a package with the right
+ * amount of elements (i.e. domain_type,
+ * enabled/disabled plus the actual data size.
+ */
+ if (wifi_pkg->type != ACPI_TYPE_PACKAGE ||
+ wifi_pkg->package.count != data_size)
+ continue;
+
+ domain = &wifi_pkg->package.elements[0];
+ if (domain->type == ACPI_TYPE_INTEGER &&
+ domain->integer.value == ACPI_WIFI_DOMAIN)
+ break;
+
+ wifi_pkg = NULL;
+ }
+
+ if (!wifi_pkg)
+ return ERR_PTR(-ENOENT);
+
+ return wifi_pkg;
+}
+
+static int iwl_mvm_sar_get_wrds_table(struct iwl_mvm *mvm)
+{
+ union acpi_object *wifi_pkg, *table;
acpi_handle root_handle;
acpi_handle handle;
struct acpi_buffer wrds = {ACPI_ALLOCATE_BUFFER, NULL};
acpi_status status;
+ bool enabled;
int ret;
root_handle = ACPI_HANDLE(mvm->dev);
@@ -1072,62 +1125,307 @@
return -ENOENT;
}
- ret = iwl_mvm_sar_get_wrds(mvm, wrds.pointer, sar_table);
- kfree(wrds.pointer);
+ wifi_pkg = iwl_mvm_sar_find_wifi_pkg(mvm, wrds.pointer,
+ ACPI_WRDS_WIFI_DATA_SIZE);
+ if (IS_ERR(wifi_pkg)) {
+ ret = PTR_ERR(wifi_pkg);
+ goto out_free;
+ }
+ if (wifi_pkg->package.elements[1].type != ACPI_TYPE_INTEGER) {
+ ret = -EINVAL;
+ goto out_free;
+ }
+
+ enabled = !!(wifi_pkg->package.elements[1].integer.value);
+
+ /* position of the actual table */
+ table = &wifi_pkg->package.elements[2];
+
+ /* The profile from WRDS is officially profile 1, but goes
+ * into sar_profiles[0] (because we don't have a profile 0).
+ */
+ ret = iwl_mvm_sar_set_profile(mvm, table, &mvm->sar_profiles[0],
+ enabled);
+
+out_free:
+ kfree(wrds.pointer);
return ret;
}
-#else /* CONFIG_ACPI */
-static int iwl_mvm_sar_get_table(struct iwl_mvm *mvm,
- struct iwl_mvm_sar_table *sar_table)
-{
- return -ENOENT;
-}
-#endif /* CONFIG_ACPI */
-static int iwl_mvm_sar_init(struct iwl_mvm *mvm)
+static int iwl_mvm_sar_get_ewrd_table(struct iwl_mvm *mvm)
{
- struct iwl_mvm_sar_table sar_table;
+ union acpi_object *wifi_pkg;
+ acpi_handle root_handle;
+ acpi_handle handle;
+ struct acpi_buffer ewrd = {ACPI_ALLOCATE_BUFFER, NULL};
+ acpi_status status;
+ bool enabled;
+ int i, n_profiles, ret;
+
+ root_handle = ACPI_HANDLE(mvm->dev);
+ if (!root_handle) {
+ IWL_DEBUG_RADIO(mvm,
+ "Could not retrieve root port ACPI handle\n");
+ return -ENOENT;
+ }
+
+ /* Get the method's handle */
+ status = acpi_get_handle(root_handle, (acpi_string)ACPI_EWRD_METHOD,
+ &handle);
+ if (ACPI_FAILURE(status)) {
+ IWL_DEBUG_RADIO(mvm, "EWRD method not found\n");
+ return -ENOENT;
+ }
+
+ /* Call EWRD with no arguments */
+ status = acpi_evaluate_object(handle, NULL, NULL, &ewrd);
+ if (ACPI_FAILURE(status)) {
+ IWL_DEBUG_RADIO(mvm, "EWRD invocation failed (0x%x)\n", status);
+ return -ENOENT;
+ }
+
+ wifi_pkg = iwl_mvm_sar_find_wifi_pkg(mvm, ewrd.pointer,
+ ACPI_EWRD_WIFI_DATA_SIZE);
+ if (IS_ERR(wifi_pkg)) {
+ ret = PTR_ERR(wifi_pkg);
+ goto out_free;
+ }
+
+ if ((wifi_pkg->package.elements[1].type != ACPI_TYPE_INTEGER) ||
+ (wifi_pkg->package.elements[2].type != ACPI_TYPE_INTEGER)) {
+ ret = -EINVAL;
+ goto out_free;
+ }
+
+ enabled = !!(wifi_pkg->package.elements[1].integer.value);
+ n_profiles = wifi_pkg->package.elements[2].integer.value;
+
+ /* in case of BIOS bug */
+ if (n_profiles <= 0) {
+ ret = -EINVAL;
+ goto out_free;
+ }
+
+ for (i = 0; i < n_profiles; i++) {
+ /* the tables start at element 3 */
+ static int pos = 3;
+
+ /* The EWRD profiles officially go from 2 to 4, but we
+ * save them in sar_profiles[1-3] (because we don't
+ * have profile 0). So in the array we start from 1.
+ */
+ ret = iwl_mvm_sar_set_profile(mvm,
+ &wifi_pkg->package.elements[pos],
+ &mvm->sar_profiles[i + 1],
+ enabled);
+ if (ret < 0)
+ break;
+
+ /* go to the next table */
+ pos += IWL_MVM_SAR_TABLE_SIZE;
+ }
+
+out_free:
+ kfree(ewrd.pointer);
+ return ret;
+}
+
+static int iwl_mvm_sar_get_wgds_table(struct iwl_mvm *mvm,
+ struct iwl_mvm_geo_table *geo_table)
+{
+ union acpi_object *wifi_pkg;
+ acpi_handle root_handle;
+ acpi_handle handle;
+ struct acpi_buffer wgds = {ACPI_ALLOCATE_BUFFER, NULL};
+ acpi_status status;
+ int i, ret;
+
+ root_handle = ACPI_HANDLE(mvm->dev);
+ if (!root_handle) {
+ IWL_DEBUG_RADIO(mvm,
+ "Could not retrieve root port ACPI handle\n");
+ return -ENOENT;
+ }
+
+ /* Get the method's handle */
+ status = acpi_get_handle(root_handle, (acpi_string)ACPI_WGDS_METHOD,
+ &handle);
+ if (ACPI_FAILURE(status)) {
+ IWL_DEBUG_RADIO(mvm, "WGDS method not found\n");
+ return -ENOENT;
+ }
+
+ /* Call WGDS with no arguments */
+ status = acpi_evaluate_object(handle, NULL, NULL, &wgds);
+ if (ACPI_FAILURE(status)) {
+ IWL_DEBUG_RADIO(mvm, "WGDS invocation failed (0x%x)\n", status);
+ return -ENOENT;
+ }
+
+ wifi_pkg = iwl_mvm_sar_find_wifi_pkg(mvm, wgds.pointer,
+ ACPI_WGDS_WIFI_DATA_SIZE);
+ if (IS_ERR(wifi_pkg)) {
+ ret = PTR_ERR(wifi_pkg);
+ goto out_free;
+ }
+
+ for (i = 0; i < ACPI_WGDS_WIFI_DATA_SIZE; i++) {
+ union acpi_object *entry;
+
+ entry = &wifi_pkg->package.elements[i + 1];
+ if ((entry->type != ACPI_TYPE_INTEGER) ||
+ (entry->integer.value > U8_MAX))
+ return -EINVAL;
+
+ geo_table->values[i] = entry->integer.value;
+ }
+ ret = 0;
+out_free:
+ kfree(wgds.pointer);
+ return ret;
+}
+
+int iwl_mvm_sar_select_profile(struct iwl_mvm *mvm, int prof_a, int prof_b)
+{
struct iwl_dev_tx_power_cmd cmd = {
.v3.set_mode = cpu_to_le32(IWL_TX_POWER_MODE_SET_CHAINS),
};
- int ret, i, j, idx;
+ int i, j, idx;
+ int profs[IWL_NUM_CHAIN_LIMITS] = { prof_a, prof_b };
int len = sizeof(cmd);
+ BUILD_BUG_ON(IWL_NUM_CHAIN_LIMITS < 2);
+ BUILD_BUG_ON(IWL_NUM_CHAIN_LIMITS * IWL_NUM_SUB_BANDS !=
+ IWL_MVM_SAR_TABLE_SIZE);
+
if (!fw_has_capa(&mvm->fw->ucode_capa, IWL_UCODE_TLV_CAPA_TX_POWER_ACK))
len = sizeof(cmd.v3);
- ret = iwl_mvm_sar_get_table(mvm, &sar_table);
+ for (i = 0; i < IWL_NUM_CHAIN_LIMITS; i++) {
+ struct iwl_mvm_sar_profile *prof;
+
+ /* don't allow SAR to be disabled (profile 0 means disable) */
+ if (profs[i] == 0)
+ return -EPERM;
+
+ /* we are off by one, so allow up to IWL_MVM_SAR_PROFILE_NUM */
+ if (profs[i] > IWL_MVM_SAR_PROFILE_NUM)
+ return -EINVAL;
+
+ /* profiles go from 1 to 4, so decrement to access the array */
+ prof = &mvm->sar_profiles[profs[i] - 1];
+
+ /* if the profile is disabled, do nothing */
+ if (!prof->enabled) {
+ IWL_DEBUG_RADIO(mvm, "SAR profile %d is disabled.\n",
+ profs[i]);
+ /* if one of the profiles is disabled, we fail all */
+ return -ENOENT;
+ }
+
+ IWL_DEBUG_RADIO(mvm, " Chain[%d]:\n", i);
+ for (j = 0; j < IWL_NUM_SUB_BANDS; j++) {
+ idx = (i * IWL_NUM_SUB_BANDS) + j;
+ cmd.v3.per_chain_restriction[i][j] =
+ cpu_to_le16(prof->table[idx]);
+ IWL_DEBUG_RADIO(mvm, " Band[%d] = %d * .125dBm\n",
+ j, prof->table[idx]);
+ }
+ }
+
+ IWL_DEBUG_RADIO(mvm, "Sending REDUCE_TX_POWER_CMD per chain\n");
+
+ return iwl_mvm_send_cmd_pdu(mvm, REDUCE_TX_POWER_CMD, 0, len, &cmd);
+}
+
+static int iwl_mvm_sar_geo_init(struct iwl_mvm *mvm)
+{
+ struct iwl_mvm_geo_table geo_table;
+ struct iwl_geo_tx_power_profiles_cmd cmd = {
+ .ops = cpu_to_le32(IWL_PER_CHAIN_OFFSET_SET_TABLES),
+ };
+ int ret, i, j, idx;
+ u16 cmd_wide_id = WIDE_ID(PHY_OPS_GROUP, GEO_TX_POWER_LIMIT);
+
+ ret = iwl_mvm_sar_get_wgds_table(mvm, &geo_table);
if (ret < 0) {
IWL_DEBUG_RADIO(mvm,
- "SAR BIOS table invalid or unavailable. (%d)\n",
+ "Geo SAR BIOS table invalid or unavailable. (%d)\n",
ret);
/* we don't fail if the table is not available */
return 0;
}
- if (!sar_table.enabled)
- return 0;
+ IWL_DEBUG_RADIO(mvm, "Sending GEO_TX_POWER_LIMIT\n");
- IWL_DEBUG_RADIO(mvm, "Sending REDUCE_TX_POWER_CMD per chain\n");
+ BUILD_BUG_ON(IWL_NUM_GEO_PROFILES * ACPI_WGDS_NUM_BANDS *
+ ACPI_WGDS_TABLE_SIZE != ACPI_WGDS_WIFI_DATA_SIZE);
- BUILD_BUG_ON(IWL_NUM_CHAIN_LIMITS * IWL_NUM_SUB_BANDS !=
- ACPI_WRDS_TABLE_SIZE);
+ for (i = 0; i < IWL_NUM_GEO_PROFILES; i++) {
+ struct iwl_per_chain_offset *chain =
+ (struct iwl_per_chain_offset *)&cmd.table[i];
- for (i = 0; i < IWL_NUM_CHAIN_LIMITS; i++) {
- IWL_DEBUG_RADIO(mvm, " Chain[%d]:\n", i);
- for (j = 0; j < IWL_NUM_SUB_BANDS; j++) {
- idx = (i * IWL_NUM_SUB_BANDS) + j;
- cmd.v3.per_chain_restriction[i][j] =
- cpu_to_le16(sar_table.values[idx]);
- IWL_DEBUG_RADIO(mvm, " Band[%d] = %d * .125dBm\n",
- j, sar_table.values[idx]);
+ for (j = 0; j < ACPI_WGDS_NUM_BANDS; j++) {
+ u8 *value;
+
+ idx = i * ACPI_WGDS_NUM_BANDS * ACPI_WGDS_TABLE_SIZE +
+ j * ACPI_WGDS_TABLE_SIZE;
+ value = &geo_table.values[idx];
+ chain[j].max_tx_power = cpu_to_le16(value[0]);
+ chain[j].chain_a = value[1];
+ chain[j].chain_b = value[2];
+ IWL_DEBUG_RADIO(mvm,
+ "SAR geographic profile[%d] Band[%d]: chain A = %d chain B = %d max_tx_power = %d\n",
+ i, j, value[1], value[2], value[0]);
}
}
+ return iwl_mvm_send_cmd_pdu(mvm, cmd_wide_id, 0, sizeof(cmd), &cmd);
+}
- ret = iwl_mvm_send_cmd_pdu(mvm, REDUCE_TX_POWER_CMD, 0, len, &cmd);
- if (ret)
- IWL_ERR(mvm, "failed to set per-chain TX power: %d\n", ret);
+#else /* CONFIG_ACPI */
+static int iwl_mvm_sar_get_wrds_table(struct iwl_mvm *mvm)
+{
+ return -ENOENT;
+}
+
+static int iwl_mvm_sar_get_ewrd_table(struct iwl_mvm *mvm)
+{
+ return -ENOENT;
+}
+
+static int iwl_mvm_sar_geo_init(struct iwl_mvm *mvm)
+{
+ return 0;
+}
+#endif /* CONFIG_ACPI */
+
+static int iwl_mvm_sar_init(struct iwl_mvm *mvm)
+{
+ int ret;
+
+ ret = iwl_mvm_sar_get_wrds_table(mvm);
+ if (ret < 0) {
+ IWL_DEBUG_RADIO(mvm,
+ "WRDS SAR BIOS table invalid or unavailable. (%d)\n",
+ ret);
+ /* if not available, don't fail and don't bother with EWRD */
+ return 0;
+ }
+
+ ret = iwl_mvm_sar_get_ewrd_table(mvm);
+ /* if EWRD is not available, we can still use WRDS, so don't fail */
+ if (ret < 0)
+ IWL_DEBUG_RADIO(mvm,
+ "EWRD SAR BIOS table invalid or unavailable. (%d)\n",
+ ret);
+
+ /* choose profile 1 (WRDS) as default for both chains */
+ ret = iwl_mvm_sar_select_profile(mvm, 1, 1);
+
+ /* if we don't have profile 0 from BIOS, just skip it */
+ if (ret == -ENOENT)
+ return 0;
return ret;
}
@@ -1219,7 +1517,8 @@
}
/* Init RSS configuration */
- if (iwl_mvm_has_new_rx_api(mvm)) {
+ /* TODO - remove a000 disablement when we have RXQ config API */
+ if (iwl_mvm_has_new_rx_api(mvm) && !iwl_mvm_has_new_tx_api(mvm)) {
ret = iwl_send_rss_cfg_cmd(mvm);
if (ret) {
IWL_ERR(mvm, "Failed to configure RSS queues: %d\n",
@@ -1229,10 +1528,10 @@
}
/* init the fw <-> mac80211 STA mapping */
- for (i = 0; i < IWL_MVM_STATION_COUNT; i++)
+ for (i = 0; i < ARRAY_SIZE(mvm->fw_id_to_mac_id); i++)
RCU_INIT_POINTER(mvm->fw_id_to_mac_id[i], NULL);
- mvm->tdls_cs.peer.sta_id = IWL_MVM_STATION_COUNT;
+ mvm->tdls_cs.peer.sta_id = IWL_MVM_INVALID_STA;
/* reset quota debouncing buffer - 0xff will yield invalid data */
memset(&mvm->last_quota_cmd, 0xff, sizeof(mvm->last_quota_cmd));
@@ -1313,10 +1612,6 @@
goto error;
}
- if (iwl_mvm_is_csum_supported(mvm) &&
- mvm->cfg->features & NETIF_F_RXCSUM)
- iwl_trans_write_prph(mvm->trans, RX_EN_CSUM, 0x3);
-
/* allow FW/transport low power modes if not during restart */
if (!test_bit(IWL_MVM_STATUS_IN_HW_RESTART, &mvm->status))
iwl_mvm_unref(mvm, IWL_MVM_REF_UCODE_DOWN);
@@ -1325,6 +1620,10 @@
if (ret)
goto error;
+ ret = iwl_mvm_sar_geo_init(mvm);
+ if (ret)
+ goto error;
+
IWL_DEBUG_INFO(mvm, "RT uCode started.\n");
return 0;
error:
@@ -1362,7 +1661,7 @@
goto error;
/* init the fw <-> mac80211 STA mapping */
- for (i = 0; i < IWL_MVM_STATION_COUNT; i++)
+ for (i = 0; i < ARRAY_SIZE(mvm->fw_id_to_mac_id); i++)
RCU_INIT_POINTER(mvm->fw_id_to_mac_id[i], NULL);
/* Add auxiliary station for scanning */
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c b/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c
index c5734e1..0f1831b 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c
@@ -467,13 +467,19 @@
queue = IWL_MVM_DQA_GCAST_QUEUE;
}
+ /*
+ * For TVQM this will be overwritten later with the FW assigned
+ * queue value (when queue is enabled).
+ */
+ mvmvif->cab_queue = queue;
vif->cab_queue = queue;
} else {
vif->cab_queue = IEEE80211_INVAL_HW_QUEUE;
}
- mvmvif->bcast_sta.sta_id = IWL_MVM_STATION_COUNT;
- mvmvif->ap_sta_id = IWL_MVM_STATION_COUNT;
+ mvmvif->bcast_sta.sta_id = IWL_MVM_INVALID_STA;
+ mvmvif->mcast_sta.sta_id = IWL_MVM_INVALID_STA;
+ mvmvif->ap_sta_id = IWL_MVM_INVALID_STA;
for (i = 0; i < NUM_IWL_MVM_SMPS_REQ; i++)
mvmvif->smps_requests[i] = IEEE80211_SMPS_AUTOMATIC;
@@ -901,7 +907,7 @@
/* Allocate sniffer station */
ret = iwl_mvm_allocate_int_sta(mvm, &mvm->snif_sta, tfd_queue_msk,
- vif->type);
+ vif->type, IWL_STA_GENERAL_PURPOSE);
if (ret)
return ret;
@@ -1222,7 +1228,9 @@
cpu_to_le32(iwl_mvm_reciprocal(vif->bss_conf.beacon_int *
vif->bss_conf.dtim_period));
- ctxt_ap->mcast_qid = cpu_to_le32(vif->cab_queue);
+ if (!fw_has_api(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_API_STA_TYPE))
+ ctxt_ap->mcast_qid = cpu_to_le32(vif->cab_queue);
/*
* Only set the beacon time when the MAC is being added, when we
@@ -1442,6 +1450,7 @@
struct iwl_mvm_tx_resp *beacon_notify_hdr;
struct ieee80211_vif *csa_vif;
struct ieee80211_vif *tx_blocked_vif;
+ struct agg_tx_status *agg_status;
u16 status;
lockdep_assert_held(&mvm->mutex);
@@ -1449,7 +1458,8 @@
beacon_notify_hdr = &beacon->beacon_notify_hdr;
mvm->ap_last_beacon_gp2 = le32_to_cpu(beacon->gp2);
- status = le16_to_cpu(beacon_notify_hdr->status.status) & TX_STATUS_MSK;
+ agg_status = iwl_mvm_get_agg_status(mvm, beacon_notify_hdr);
+ status = le16_to_cpu(agg_status->status) & TX_STATUS_MSK;
IWL_DEBUG_RX(mvm,
"beacon status %#x retries:%d tsf:0x%16llX gp2:0x%X rate:%d\n",
status, beacon_notify_hdr->failure_frame,
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c b/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c
index 486dcce..a67aa1f 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c
@@ -6,8 +6,8 @@
* GPL LICENSE SUMMARY
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
- * Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -33,7 +33,8 @@
* BSD LICENSE
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
- * Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
+ * Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -619,7 +620,7 @@
else
hw->wiphy->flags &= ~WIPHY_FLAG_PS_ON_BY_DEFAULT;
- hw->wiphy->flags |= WIPHY_FLAG_SUPPORTS_SCHED_SCAN;
+ hw->wiphy->max_sched_scan_reqs = 1;
hw->wiphy->max_sched_scan_ssids = PROBE_OPTION_MAX;
hw->wiphy->max_match_sets = IWL_SCAN_MAX_PROFILES;
/* we create the 802.11 header and zero length SSID IE. */
@@ -766,7 +767,7 @@
goto out;
mvmsta = iwl_mvm_sta_from_mac80211(sta);
- if (mvmsta->sta_id == IWL_MVM_STATION_COUNT ||
+ if (mvmsta->sta_id == IWL_MVM_INVALID_STA ||
mvmsta->sta_id != mvm->d0i3_ap_sta_id)
goto out;
@@ -1010,7 +1011,7 @@
struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
mvmvif->uploaded = false;
- mvmvif->ap_sta_id = IWL_MVM_STATION_COUNT;
+ mvmvif->ap_sta_id = IWL_MVM_INVALID_STA;
spin_lock_bh(&mvm->time_event_lock);
iwl_mvm_te_clear_data(mvm, &mvmvif->time_event_data);
@@ -1053,7 +1054,7 @@
ieee80211_iterate_interfaces(mvm->hw, 0, iwl_mvm_cleanup_iterator, mvm);
mvm->p2p_device_vif = NULL;
- mvm->d0i3_ap_sta_id = IWL_MVM_STATION_COUNT;
+ mvm->d0i3_ap_sta_id = IWL_MVM_INVALID_STA;
iwl_mvm_reset_phy_ctxts(mvm);
memset(mvm->fw_key_table, 0, sizeof(mvm->fw_key_table));
@@ -1351,6 +1352,18 @@
goto out_release;
}
+ if (iwl_mvm_is_dqa_supported(mvm)) {
+ /*
+ * Only queue for this station is the mcast queue,
+ * which shouldn't be in TFD mask anyway
+ */
+ ret = iwl_mvm_allocate_int_sta(mvm, &mvmvif->mcast_sta,
+ 0, vif->type,
+ IWL_STA_MULTICAST);
+ if (ret)
+ goto out_release;
+ }
+
iwl_mvm_vif_dbgfs_register(mvm, vif);
goto out_unlock;
}
@@ -1465,7 +1478,7 @@
* already marked as draining, so to complete the draining, we
* just need to wait until the transport is empty.
*/
- iwl_trans_wait_tx_queue_empty(mvm->trans, tfd_msk);
+ iwl_trans_wait_tx_queues_empty(mvm->trans, tfd_msk);
}
if (vif->type == NL80211_IFTYPE_P2P_DEVICE) {
@@ -1516,6 +1529,7 @@
mvm->noa_duration = 0;
}
#endif
+ iwl_mvm_dealloc_int_sta(mvm, &mvmvif->mcast_sta);
iwl_mvm_dealloc_bcast_sta(mvm, vif);
goto out_release;
}
@@ -1952,7 +1966,7 @@
IWL_MVM_SMPS_REQ_PROT,
IEEE80211_SMPS_DYNAMIC);
}
- } else if (mvmvif->ap_sta_id != IWL_MVM_STATION_COUNT) {
+ } else if (mvmvif->ap_sta_id != IWL_MVM_INVALID_STA) {
/*
* If update fails - SF might be running in associated
* mode while disassociated - which is forbidden.
@@ -1966,8 +1980,8 @@
IWL_ERR(mvm, "failed to remove AP station\n");
if (mvm->d0i3_ap_sta_id == mvmvif->ap_sta_id)
- mvm->d0i3_ap_sta_id = IWL_MVM_STATION_COUNT;
- mvmvif->ap_sta_id = IWL_MVM_STATION_COUNT;
+ mvm->d0i3_ap_sta_id = IWL_MVM_INVALID_STA;
+ mvmvif->ap_sta_id = IWL_MVM_INVALID_STA;
/* remove quota for this interface */
ret = iwl_mvm_update_quotas(mvm, false, NULL);
if (ret)
@@ -2098,11 +2112,15 @@
if (ret)
goto out_remove;
+ ret = iwl_mvm_add_mcast_sta(mvm, vif);
+ if (ret)
+ goto out_unbind;
+
/* Send the bcast station. At this stage the TBTT and DTIM time events
* are added and applied to the scheduler */
ret = iwl_mvm_send_add_bcast_sta(mvm, vif);
if (ret)
- goto out_unbind;
+ goto out_rm_mcast;
/* must be set before quota calculations */
mvmvif->ap_ibss_active = true;
@@ -2132,6 +2150,8 @@
iwl_mvm_power_update_mac(mvm);
mvmvif->ap_ibss_active = false;
iwl_mvm_send_rm_bcast_sta(mvm, vif);
+out_rm_mcast:
+ iwl_mvm_rm_mcast_sta(mvm, vif);
out_unbind:
iwl_mvm_binding_remove_vif(mvm, vif);
out_remove:
@@ -2177,7 +2197,20 @@
iwl_mvm_mac_ctxt_changed(mvm, mvm->p2p_device_vif, false, NULL);
iwl_mvm_update_quotas(mvm, false, NULL);
+
+ /*
+ * This is not very nice, but the simplest:
+ * For older FWs removing the mcast sta before the bcast station may
+ * cause assert 0x2b00.
+ * This is fixed in later FW (which will stop beaconing when removing
+ * bcast station).
+ * So make the order of removal depend on the TLV
+ */
+ if (!fw_has_api(&mvm->fw->ucode_capa, IWL_UCODE_TLV_API_STA_TYPE))
+ iwl_mvm_rm_mcast_sta(mvm, vif);
iwl_mvm_send_rm_bcast_sta(mvm, vif);
+ if (fw_has_api(&mvm->fw->ucode_capa, IWL_UCODE_TLV_API_STA_TYPE))
+ iwl_mvm_rm_mcast_sta(mvm, vif);
iwl_mvm_binding_remove_vif(mvm, vif);
iwl_mvm_power_update_mac(mvm);
@@ -2343,6 +2376,9 @@
tid_data->state != IWL_EMPTYING_HW_QUEUE_DELBA)
continue;
+ if (tid_data->txq_id == IWL_MVM_INVALID_QUEUE)
+ continue;
+
__set_bit(tid_data->txq_id, &txqs);
if (iwl_mvm_tid_queued(tid_data) == 0)
@@ -2368,7 +2404,7 @@
*/
break;
case STA_NOTIFY_AWAKE:
- if (WARN_ON(mvmsta->sta_id == IWL_MVM_STATION_COUNT))
+ if (WARN_ON(mvmsta->sta_id == IWL_MVM_INVALID_STA))
break;
if (txqs)
@@ -3711,7 +3747,8 @@
int err;
u32 noa_duration;
- err = nla_parse(tb, IWL_MVM_TM_ATTR_MAX, data, len, iwl_mvm_tm_policy);
+ err = nla_parse(tb, IWL_MVM_TM_ATTR_MAX, data, len, iwl_mvm_tm_policy,
+ NULL);
if (err)
return err;
@@ -3938,7 +3975,7 @@
mvmvif = iwl_mvm_vif_from_mac80211(vif);
/* flush the AP-station and all TDLS peers */
- for (i = 0; i < IWL_MVM_STATION_COUNT; i++) {
+ for (i = 0; i < ARRAY_SIZE(mvm->fw_id_to_mac_id); i++) {
sta = rcu_dereference_protected(mvm->fw_id_to_mac_id[i],
lockdep_is_held(&mvm->mutex));
if (IS_ERR_OR_NULL(sta))
@@ -3964,7 +4001,7 @@
/* this can take a while, and we may need/want other operations
* to succeed while doing this, so do it without the mutex held
*/
- iwl_trans_wait_tx_queue_empty(mvm->trans, msk);
+ iwl_trans_wait_tx_queues_empty(mvm->trans, msk);
}
}
@@ -4195,7 +4232,8 @@
lockdep_assert_held(&mvm->mutex);
- if (!iwl_mvm_has_new_rx_api(mvm))
+ /* TODO - remove a000 disablement when we have RXQ config API */
+ if (!iwl_mvm_has_new_rx_api(mvm) || iwl_mvm_has_new_tx_api(mvm))
return;
notif->cookie = mvm->queue_sync_cookie;
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h b/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h
index 73a2165..4e74a6b 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h
@@ -7,7 +7,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -34,7 +34,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -380,6 +380,8 @@
bool associated;
u8 ap_assoc_sta_count;
+ u16 cab_queue;
+
bool uploaded;
bool ap_ibss_active;
bool pm_enabled;
@@ -407,6 +409,7 @@
struct iwl_mvm_time_event_data hs_time_event_data;
struct iwl_mvm_int_sta bcast_sta;
+ struct iwl_mvm_int_sta mcast_sta;
/*
* Assigned while mac80211 has the interface in a channel context,
@@ -603,10 +606,15 @@
IWL_MVM_TDLS_SW_ACTIVE,
};
+#define MAX_NUM_LMAC 2
struct iwl_mvm_shared_mem_cfg {
+ int num_lmacs;
int num_txfifo_entries;
- u32 txfifo_size[TX_FIFO_MAX_NUM];
- u32 rxfifo_size[RX_FIFO_MAX_NUM];
+ struct {
+ u32 txfifo_size[TX_FIFO_MAX_NUM];
+ u32 rxfifo1_size;
+ } lmac[MAX_NUM_LMAC];
+ u32 rxfifo2_size;
u32 internal_txfifo_addr;
u32 internal_txfifo_size[TX_FIFO_INTERNAL_MAX_NUM];
};
@@ -625,6 +633,7 @@
* @reorder_timer: timer for frames are in the reorder buffer. For AMSDU
* it is the time of last received sub-frame
* @removed: prevent timer re-arming
+ * @valid: reordering is valid for this queue
* @lock: protect reorder buffer internal state
* @mvm: mvm pointer, needed for frame timer context
*/
@@ -640,6 +649,7 @@
unsigned long reorder_time[IEEE80211_MAX_AMPDU_BUF];
struct timer_list reorder_timer;
bool removed;
+ bool valid;
spinlock_t lock;
struct iwl_mvm *mvm;
} ____cacheline_aligned_in_smp;
@@ -707,8 +717,25 @@
};
#define IWL_MVM_DQA_QUEUE_TIMEOUT (5 * HZ)
+#define IWL_MVM_INVALID_QUEUE 0xFFFF
+
#define IWL_MVM_NUM_CIPHERS 10
+#ifdef CONFIG_ACPI
+#define IWL_MVM_SAR_TABLE_SIZE 10
+#define IWL_MVM_SAR_PROFILE_NUM 4
+#define IWL_MVM_GEO_TABLE_SIZE 18
+
+struct iwl_mvm_sar_profile {
+ bool enabled;
+ u8 table[IWL_MVM_SAR_TABLE_SIZE];
+};
+
+struct iwl_mvm_geo_table {
+ u8 values[IWL_MVM_GEO_TABLE_SIZE];
+};
+#endif
+
struct iwl_mvm {
/* for logger access */
struct device *dev;
@@ -761,9 +788,9 @@
u64 on_time_scan;
} radio_stats, accu_radio_stats;
+ u8 hw_queue_to_mac80211[IWL_MAX_TVQM_QUEUES];
+
struct {
- /* Map to HW queue */
- u32 hw_queue_to_mac80211;
u8 hw_queue_refcount;
u8 ra_sta_id; /* The RA this queue is mapped to, if exists */
bool reserved; /* Is this the TXQ reserved for a STA */
@@ -975,7 +1002,10 @@
#endif
/* Tx queues */
- u8 aux_queue;
+ u16 aux_queue;
+ u16 probe_queue;
+ u16 p2p_dev_queue;
+
u8 first_agg_queue;
u8 last_agg_queue;
@@ -1018,7 +1048,7 @@
} peer;
} tdls_cs;
- struct iwl_mvm_shared_mem_cfg shared_mem_cfg;
+ struct iwl_mvm_shared_mem_cfg smem_cfg;
u32 ciphers[IWL_MVM_NUM_CIPHERS];
struct ieee80211_cipher_scheme cs[IWL_UCODE_MAX_CS];
@@ -1035,6 +1065,9 @@
bool drop_bcn_ap_mode;
struct delayed_work cs_tx_unblock_dwork;
+#ifdef CONFIG_ACPI
+ struct iwl_mvm_sar_profile sar_profiles[IWL_MVM_SAR_PROFILE_NUM];
+#endif
};
/* Extract MVM priv from op_mode and _hw */
@@ -1222,13 +1255,25 @@
{
/*
* TODO:
- * The issue of how to determine CDB support is still not well defined.
- * It may be that it will be for all next HW devices and it may be per
- * FW compilation and it may also differ between different devices.
- * For now take a ride on the new TX API and get back to it when
- * it is well defined.
+ * The issue of how to determine CDB APIs and usage is still not fully
+ * defined.
+ * There is a compilation for CDB and non-CDB FW, but there may
+ * be also runtime check.
+ * For now there is a TLV for checking compilation mode, but a
+ * runtime check will also have to be here - once defined.
*/
- return iwl_mvm_has_new_tx_api(mvm);
+ return fw_has_capa(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_CAPA_CDB_SUPPORT);
+}
+
+static inline struct agg_tx_status*
+iwl_mvm_get_agg_status(struct iwl_mvm *mvm,
+ struct iwl_mvm_tx_resp *tx_resp)
+{
+ if (iwl_mvm_has_new_tx_api(mvm))
+ return &tx_resp->v6.status;
+ else
+ return &tx_resp->v3.status;
}
static inline bool iwl_mvm_is_tt_in_fw(struct iwl_mvm *mvm)
@@ -1271,7 +1316,6 @@
******************/
/* uCode */
int iwl_run_init_mvm_ucode(struct iwl_mvm *mvm, bool read_nvm);
-int iwl_run_unified_mvm_ucode(struct iwl_mvm *mvm, bool read_nvm);
/* Utils */
int iwl_mvm_legacy_rate_to_mac80211_idx(u32 rate_n_flags,
@@ -1389,6 +1433,8 @@
void iwl_mvm_rx_queue_notif(struct iwl_mvm *mvm, struct iwl_rx_cmd_buffer *rxb,
int queue);
void iwl_mvm_rx_tx_cmd(struct iwl_mvm *mvm, struct iwl_rx_cmd_buffer *rxb);
+void iwl_mvm_mfu_assert_dump_notif(struct iwl_mvm *mvm,
+ struct iwl_rx_cmd_buffer *rxb);
void iwl_mvm_rx_ba_notif(struct iwl_mvm *mvm, struct iwl_rx_cmd_buffer *rxb);
void iwl_mvm_rx_ant_coupling_notif(struct iwl_mvm *mvm,
struct iwl_rx_cmd_buffer *rxb);
@@ -1668,6 +1714,9 @@
void iwl_mvm_enable_txq(struct iwl_mvm *mvm, int queue, int mac80211_queue,
u16 ssn, const struct iwl_trans_txq_scd_cfg *cfg,
unsigned int wdg_timeout);
+int iwl_mvm_tvqm_enable_txq(struct iwl_mvm *mvm, int mac80211_queue,
+ u8 sta_id, u8 tid, unsigned int timeout);
+
/*
* Disable a TXQ.
* Note that in non-DQA mode the %mac80211_queue and %tid params are ignored.
@@ -1701,7 +1750,8 @@
static inline void iwl_mvm_stop_device(struct iwl_mvm *mvm)
{
- iwl_free_fw_paging(mvm);
+ if (!iwl_mvm_has_new_tx_api(mvm))
+ iwl_free_fw_paging(mvm);
mvm->ucode_loaded = false;
iwl_trans_stop_device(mvm->trans);
}
@@ -1781,6 +1831,7 @@
u32 size);
void iwl_mvm_reorder_timer_expired(unsigned long data);
struct ieee80211_vif *iwl_mvm_get_bss_vif(struct iwl_mvm *mvm);
+bool iwl_mvm_is_vif_assoc(struct iwl_mvm *mvm);
void iwl_mvm_inactivity_check(struct iwl_mvm *mvm);
@@ -1797,4 +1848,14 @@
u32 duration, u32 timeout);
bool iwl_mvm_lqm_active(struct iwl_mvm *mvm);
+#ifdef CONFIG_ACPI
+int iwl_mvm_sar_select_profile(struct iwl_mvm *mvm, int prof_a, int prof_b);
+#else
+static inline
+int iwl_mvm_sar_select_profile(struct iwl_mvm *mvm, int prof_a, int prof_b)
+{
+ return -ENOENT;
+}
+#endif /* CONFIG_ACPI */
+
#endif /* __IWL_MVM_H__ */
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/nvm.c b/drivers/net/wireless/intel/iwlwifi/mvm/nvm.c
index eade099..283c41d 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/nvm.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/nvm.c
@@ -7,7 +7,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -34,7 +34,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -817,6 +817,11 @@
lockdep_assert_held(&mvm->mutex);
+ if (iwl_mvm_is_vif_assoc(mvm) && notif->source_id == MCC_SOURCE_WIFI) {
+ IWL_DEBUG_LAR(mvm, "Ignore mcc update while associated\n");
+ return;
+ }
+
if (WARN_ON_ONCE(!iwl_mvm_is_lar_supported(mvm)))
return;
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/ops.c b/drivers/net/wireless/intel/iwlwifi/mvm/ops.c
index 4cd72d4..9ffff6e 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/ops.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/ops.c
@@ -302,6 +302,8 @@
RX_HANDLER_SYNC),
RX_HANDLER(TOF_NOTIFICATION, iwl_mvm_tof_resp_handler,
RX_HANDLER_ASYNC_LOCKED),
+ RX_HANDLER_GRP(DEBUG_GROUP, MFU_ASSERT_DUMP_NTF,
+ iwl_mvm_mfu_assert_dump_notif, RX_HANDLER_SYNC),
RX_HANDLER_GRP(PROT_OFFLOAD_GROUP, STORED_BEACON_NTF,
iwl_mvm_rx_stored_beacon_notif, RX_HANDLER_SYNC),
RX_HANDLER_GRP(DATA_PATH_GROUP, MU_GROUP_MGMT_NOTIF,
@@ -426,6 +428,7 @@
*/
static const struct iwl_hcmd_names iwl_mvm_system_names[] = {
HCMD_NAME(SHARED_MEM_CFG_CMD),
+ HCMD_NAME(INIT_EXTENDED_CFG_CMD),
};
/* Please keep this array *SORTED* by hex value.
@@ -444,6 +447,7 @@
HCMD_NAME(CMD_DTS_MEASUREMENT_TRIGGER_WIDE),
HCMD_NAME(CTDP_CONFIG_CMD),
HCMD_NAME(TEMP_REPORTING_THRESHOLDS_CMD),
+ HCMD_NAME(GEO_TX_POWER_LIMIT),
HCMD_NAME(CT_KILL_NOTIFICATION),
HCMD_NAME(DTS_MEASUREMENT_NOTIF_WIDE),
};
@@ -452,6 +456,7 @@
* Access is done through binary search
*/
static const struct iwl_hcmd_names iwl_mvm_data_path_names[] = {
+ HCMD_NAME(DQA_ENABLE_CMD),
HCMD_NAME(UPDATE_MU_GROUPS_CMD),
HCMD_NAME(TRIGGER_RX_QUEUES_NOTIF_CMD),
HCMD_NAME(STA_PM_NOTIF),
@@ -462,6 +467,13 @@
/* Please keep this array *SORTED* by hex value.
* Access is done through binary search
*/
+static const struct iwl_hcmd_names iwl_mvm_debug_names[] = {
+ HCMD_NAME(MFU_ASSERT_DUMP_NTF),
+};
+
+/* Please keep this array *SORTED* by hex value.
+ * Access is done through binary search
+ */
static const struct iwl_hcmd_names iwl_mvm_prot_offload_names[] = {
HCMD_NAME(STORED_BEACON_NTF),
};
@@ -602,6 +614,8 @@
}
} else {
mvm->aux_queue = IWL_MVM_DQA_AUX_QUEUE;
+ mvm->probe_queue = IWL_MVM_DQA_AP_PROBE_RESP_QUEUE;
+ mvm->p2p_dev_queue = IWL_MVM_DQA_P2P_DEVICE_QUEUE;
mvm->first_agg_queue = IWL_MVM_DQA_MIN_DATA_QUEUE;
mvm->last_agg_queue = IWL_MVM_DQA_MAX_DATA_QUEUE;
}
@@ -732,10 +746,7 @@
mutex_lock(&mvm->mutex);
iwl_mvm_ref(mvm, IWL_MVM_REF_INIT_UCODE);
- if (iwl_mvm_has_new_tx_api(mvm))
- err = iwl_run_unified_mvm_ucode(mvm, true);
- else
- err = iwl_run_init_mvm_ucode(mvm, true);
+ err = iwl_run_init_mvm_ucode(mvm, true);
if (!err || !iwlmvm_mod_params.init_dbg)
iwl_mvm_stop_device(mvm);
iwl_mvm_unref(mvm, IWL_MVM_REF_INIT_UCODE);
@@ -1033,7 +1044,7 @@
unsigned long mq;
spin_lock_bh(&mvm->queue_info_lock);
- mq = mvm->queue_info[hw_queue].hw_queue_to_mac80211;
+ mq = mvm->hw_queue_to_mac80211[hw_queue];
spin_unlock_bh(&mvm->queue_info_lock);
iwl_mvm_stop_mac_queues(mvm, mq);
@@ -1063,7 +1074,7 @@
unsigned long mq;
spin_lock_bh(&mvm->queue_info_lock);
- mq = mvm->queue_info[hw_queue].hw_queue_to_mac80211;
+ mq = mvm->hw_queue_to_mac80211[hw_queue];
spin_unlock_bh(&mvm->queue_info_lock);
iwl_mvm_start_mac_queues(mvm, mq);
@@ -1256,7 +1267,7 @@
u8 tid;
if (WARN_ON(vif->type != NL80211_IFTYPE_STATION ||
- mvmvif->ap_sta_id == IWL_MVM_STATION_COUNT))
+ mvmvif->ap_sta_id == IWL_MVM_INVALID_STA))
return false;
mvmsta = iwl_mvm_sta_from_staid_rcu(mvm, mvmvif->ap_sta_id);
@@ -1344,7 +1355,7 @@
struct ieee80211_sta *ap_sta;
struct iwl_mvm_sta *mvm_ap_sta;
- if (iter_data->ap_sta_id == IWL_MVM_STATION_COUNT)
+ if (iter_data->ap_sta_id == IWL_MVM_INVALID_STA)
return;
rcu_read_lock();
@@ -1414,7 +1425,7 @@
mvm->d0i3_offloading = !d0i3_iter_data.disable_offloading;
} else {
WARN_ON_ONCE(d0i3_iter_data.vif_count > 1);
- mvm->d0i3_ap_sta_id = IWL_MVM_STATION_COUNT;
+ mvm->d0i3_ap_sta_id = IWL_MVM_INVALID_STA;
mvm->d0i3_offloading = false;
}
@@ -1427,7 +1438,7 @@
return ret;
/* configure wowlan configuration only if needed */
- if (mvm->d0i3_ap_sta_id != IWL_MVM_STATION_COUNT) {
+ if (mvm->d0i3_ap_sta_id != IWL_MVM_INVALID_STA) {
/* wake on beacons only if beacon storing isn't supported */
if (!fw_has_capa(&mvm->fw->ucode_capa,
IWL_UCODE_TLV_CAPA_BEACON_STORING))
@@ -1504,7 +1515,7 @@
spin_lock_bh(&mvm->d0i3_tx_lock);
- if (mvm->d0i3_ap_sta_id == IWL_MVM_STATION_COUNT)
+ if (mvm->d0i3_ap_sta_id == IWL_MVM_INVALID_STA)
goto out;
IWL_DEBUG_RPM(mvm, "re-enqueue packets\n");
@@ -1542,7 +1553,7 @@
}
clear_bit(IWL_MVM_STATUS_IN_D0I3, &mvm->status);
wake_up(&mvm->d0i3_exit_waitq);
- mvm->d0i3_ap_sta_id = IWL_MVM_STATION_COUNT;
+ mvm->d0i3_ap_sta_id = IWL_MVM_INVALID_STA;
if (wake_queues)
ieee80211_wake_queues(mvm->hw);
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/phy-ctxt.c b/drivers/net/wireless/intel/iwlwifi/mvm/phy-ctxt.c
index 9513883..d59efe8 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/phy-ctxt.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/phy-ctxt.c
@@ -7,6 +7,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
+ * Copyright(c) 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -250,12 +251,30 @@
struct cfg80211_chan_def *chandef,
u8 chains_static, u8 chains_dynamic)
{
+ enum iwl_phy_ctxt_action action = FW_CTXT_ACTION_MODIFY;
+
lockdep_assert_held(&mvm->mutex);
+ /* In CDB mode we cannot modify PHY context between bands so... */
+ if (iwl_mvm_has_new_tx_api(mvm) &&
+ ctxt->channel->band != chandef->chan->band) {
+ int ret;
+
+ /* ... remove it here ...*/
+ ret = iwl_mvm_phy_ctxt_apply(mvm, ctxt, chandef,
+ chains_static, chains_dynamic,
+ FW_CTXT_ACTION_REMOVE, 0);
+ if (ret)
+ return ret;
+
+ /* ... and proceed to add it again */
+ action = FW_CTXT_ACTION_ADD;
+ }
+
ctxt->channel = chandef->chan;
return iwl_mvm_phy_ctxt_apply(mvm, ctxt, chandef,
chains_static, chains_dynamic,
- FW_CTXT_ACTION_MODIFY, 0);
+ action, 0);
}
void iwl_mvm_phy_ctxt_unref(struct iwl_mvm *mvm, struct iwl_mvm_phy_ctxt *ctxt)
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/rs.c b/drivers/net/wireless/intel/iwlwifi/mvm/rs.c
index ce907c5..7788eef 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/rs.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/rs.c
@@ -826,7 +826,7 @@
if (is_siso(rate) && rate->stbc) {
/* To enable STBC we need to set both a flag and ANT_AB */
ucode_rate |= RATE_MCS_ANT_AB_MSK;
- ucode_rate |= RATE_MCS_VHT_STBC_MSK;
+ ucode_rate |= RATE_MCS_STBC_MSK;
}
ucode_rate |= rate->bw;
@@ -873,7 +873,7 @@
rate->sgi = true;
if (ucode_rate & RATE_MCS_LDPC_MSK)
rate->ldpc = true;
- if (ucode_rate & RATE_MCS_VHT_STBC_MSK)
+ if (ucode_rate & RATE_MCS_STBC_MSK)
rate->stbc = true;
if (ucode_rate & RATE_MCS_BF_MSK)
rate->bfer = true;
@@ -3641,13 +3641,12 @@
bw = "BAD BW";
}
- return sprintf(buf, "%s | ANT: %s BW: %s MCS: %d NSS: %d %s%s%s%s%s\n",
+ return sprintf(buf, "%s | ANT: %s BW: %s MCS: %d NSS: %d %s%s%s%s\n",
type, rs_pretty_ant(ant), bw, mcs, nss,
(rate & RATE_MCS_SGI_MSK) ? "SGI " : "NGI ",
- (rate & RATE_MCS_HT_STBC_MSK) ? "STBC " : "",
+ (rate & RATE_MCS_STBC_MSK) ? "STBC " : "",
(rate & RATE_MCS_LDPC_MSK) ? "LDPC " : "",
- (rate & RATE_MCS_BF_MSK) ? "BF " : "",
- (rate & RATE_MCS_ZLF_MSK) ? "ZLF " : "");
+ (rate & RATE_MCS_BF_MSK) ? "BF " : "");
}
/**
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/rx.c b/drivers/net/wireless/intel/iwlwifi/mvm/rx.c
index 20473df..fd1dd06 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/rx.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/rx.c
@@ -7,7 +7,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -104,7 +104,20 @@
u8 crypt_len,
struct iwl_rx_cmd_buffer *rxb)
{
- unsigned int hdrlen, fraglen;
+ unsigned int hdrlen = ieee80211_hdrlen(hdr->frame_control);
+ unsigned int fraglen;
+
+ /*
+ * The 'hdrlen' (plus the 8 bytes for the SNAP and the crypt_len,
+ * but those are all multiples of 4 long) all goes away, but we
+ * want the *end* of it, which is going to be the start of the IP
+ * header, to be aligned when it gets pulled in.
+ * The beginning of the skb->data is aligned on at least a 4-byte
+ * boundary after allocation. Everything here is aligned at least
+ * on a 2-byte boundary so we can just take hdrlen & 3 and pad by
+ * the result.
+ */
+ skb_reserve(skb, hdrlen & 3);
/* If frame is small enough to fit in skb->head, pull it completely.
* If not, only pull ieee80211_hdr (including crypto if present, and
@@ -118,8 +131,7 @@
* If the latter changes (there are efforts in the standards group
* to do so) we should revisit this and ieee80211_data_to_8023().
*/
- hdrlen = (len <= skb_tailroom(skb)) ? len :
- sizeof(*hdr) + crypt_len + 8;
+ hdrlen = (len <= skb_tailroom(skb)) ? len : hdrlen + crypt_len + 8;
memcpy(skb_put(skb, hdrlen), hdr, hdrlen);
fraglen = len - hdrlen;
@@ -339,7 +351,7 @@
id >>= RX_MDPU_RES_STATUS_STA_ID_SHIFT;
- if (!WARN_ON_ONCE(id >= IWL_MVM_STATION_COUNT)) {
+ if (!WARN_ON_ONCE(id >= ARRAY_SIZE(mvm->fw_id_to_mac_id))) {
sta = rcu_dereference(mvm->fw_id_to_mac_id[id]);
if (IS_ERR(sta))
sta = NULL;
@@ -398,7 +410,7 @@
/* set the preamble flag if appropriate */
if (phy_info->phy_flags & cpu_to_le16(RX_RES_PHY_FLAGS_SHORT_PREAMBLE))
- rx_status->flag |= RX_FLAG_SHORTPRE;
+ rx_status->enc_flags |= RX_ENC_FLAG_SHORTPRE;
if (phy_info->phy_flags & cpu_to_le16(RX_RES_PHY_FLAGS_AGG)) {
/*
@@ -415,42 +427,49 @@
case RATE_MCS_CHAN_WIDTH_20:
break;
case RATE_MCS_CHAN_WIDTH_40:
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
break;
case RATE_MCS_CHAN_WIDTH_80:
- rx_status->vht_flag |= RX_VHT_FLAG_80MHZ;
+ rx_status->bw = RATE_INFO_BW_80;
break;
case RATE_MCS_CHAN_WIDTH_160:
- rx_status->vht_flag |= RX_VHT_FLAG_160MHZ;
+ rx_status->bw = RATE_INFO_BW_160;
break;
}
if (rate_n_flags & RATE_MCS_SGI_MSK)
- rx_status->flag |= RX_FLAG_SHORT_GI;
+ rx_status->enc_flags |= RX_ENC_FLAG_SHORT_GI;
if (rate_n_flags & RATE_HT_MCS_GF_MSK)
- rx_status->flag |= RX_FLAG_HT_GF;
+ rx_status->enc_flags |= RX_ENC_FLAG_HT_GF;
if (rate_n_flags & RATE_MCS_LDPC_MSK)
- rx_status->flag |= RX_FLAG_LDPC;
+ rx_status->enc_flags |= RX_ENC_FLAG_LDPC;
if (rate_n_flags & RATE_MCS_HT_MSK) {
- u8 stbc = (rate_n_flags & RATE_MCS_HT_STBC_MSK) >>
+ u8 stbc = (rate_n_flags & RATE_MCS_STBC_MSK) >>
RATE_MCS_STBC_POS;
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
rx_status->rate_idx = rate_n_flags & RATE_HT_MCS_INDEX_MSK;
- rx_status->flag |= stbc << RX_FLAG_STBC_SHIFT;
+ rx_status->enc_flags |= stbc << RX_ENC_FLAG_STBC_SHIFT;
} else if (rate_n_flags & RATE_MCS_VHT_MSK) {
- u8 stbc = (rate_n_flags & RATE_MCS_VHT_STBC_MSK) >>
+ u8 stbc = (rate_n_flags & RATE_MCS_STBC_MSK) >>
RATE_MCS_STBC_POS;
- rx_status->vht_nss =
+ rx_status->nss =
((rate_n_flags & RATE_VHT_MCS_NSS_MSK) >>
RATE_VHT_MCS_NSS_POS) + 1;
rx_status->rate_idx = rate_n_flags & RATE_VHT_MCS_RATE_CODE_MSK;
- rx_status->flag |= RX_FLAG_VHT;
- rx_status->flag |= stbc << RX_FLAG_STBC_SHIFT;
+ rx_status->encoding = RX_ENC_VHT;
+ rx_status->enc_flags |= stbc << RX_ENC_FLAG_STBC_SHIFT;
if (rate_n_flags & RATE_MCS_BF_MSK)
- rx_status->vht_flag |= RX_VHT_FLAG_BF;
+ rx_status->enc_flags |= RX_ENC_FLAG_BF;
} else {
- rx_status->rate_idx =
- iwl_mvm_legacy_rate_to_mac80211_idx(rate_n_flags,
- rx_status->band);
+ int rate = iwl_mvm_legacy_rate_to_mac80211_idx(rate_n_flags,
+ rx_status->band);
+
+ if (WARN(rate < 0 || rate > 0xFF,
+ "Invalid rate flags 0x%x, band %d,\n",
+ rate_n_flags, rx_status->band)) {
+ kfree_skb(skb);
+ return;
+ }
+ rx_status->rate_idx = rate;
}
#ifdef CONFIG_IWLWIFI_DEBUGFS
@@ -637,6 +656,9 @@
.mvm = mvm,
};
int expected_size;
+ int i;
+ u8 *energy;
+ __le32 *bytes, *air_time;
if (iwl_mvm_is_cdb_supported(mvm))
expected_size = sizeof(*stats);
@@ -645,8 +667,11 @@
else
expected_size = sizeof(struct iwl_notif_statistics_v10);
- if (iwl_rx_packet_payload_len(pkt) != expected_size)
- goto invalid;
+ if (iwl_rx_packet_payload_len(pkt) != expected_size) {
+ IWL_ERR(mvm, "received invalid statistics size (%d)!\n",
+ iwl_rx_packet_payload_len(pkt));
+ return;
+ }
data.mac_id = stats->rx.general.mac_id;
data.beacon_filter_average_energy =
@@ -662,38 +687,6 @@
le64_to_cpu(stats->general.common.on_time_scan);
data.general = &stats->general;
- if (iwl_mvm_has_new_rx_api(mvm)) {
- int i;
- u8 *energy;
- __le32 *bytes, *air_time;
-
- if (!iwl_mvm_is_cdb_supported(mvm)) {
- struct iwl_notif_statistics_v11 *v11 =
- (void *)&pkt->data;
-
- energy = (void *)&v11->load_stats.avg_energy;
- bytes = (void *)&v11->load_stats.byte_count;
- air_time = (void *)&v11->load_stats.air_time;
- } else {
- energy = (void *)&stats->load_stats.avg_energy;
- bytes = (void *)&stats->load_stats.byte_count;
- air_time = (void *)&stats->load_stats.air_time;
- }
-
- rcu_read_lock();
- for (i = 0; i < IWL_MVM_STATION_COUNT; i++) {
- struct iwl_mvm_sta *sta;
-
- if (!energy[i])
- continue;
-
- sta = iwl_mvm_sta_from_staid_rcu(mvm, i);
- if (!sta)
- continue;
- sta->avg_energy = energy[i];
- }
- rcu_read_unlock();
- }
iwl_mvm_rx_stats_check_trigger(mvm, pkt);
@@ -701,10 +694,36 @@
IEEE80211_IFACE_ITER_NORMAL,
iwl_mvm_stat_iterator,
&data);
- return;
- invalid:
- IWL_ERR(mvm, "received invalid statistics size (%d)!\n",
- iwl_rx_packet_payload_len(pkt));
+
+ if (!iwl_mvm_has_new_rx_api(mvm))
+ return;
+
+ if (!iwl_mvm_is_cdb_supported(mvm)) {
+ struct iwl_notif_statistics_v11 *v11 =
+ (void *)&pkt->data;
+
+ energy = (void *)&v11->load_stats.avg_energy;
+ bytes = (void *)&v11->load_stats.byte_count;
+ air_time = (void *)&v11->load_stats.air_time;
+ } else {
+ energy = (void *)&stats->load_stats.avg_energy;
+ bytes = (void *)&stats->load_stats.byte_count;
+ air_time = (void *)&stats->load_stats.air_time;
+ }
+
+ rcu_read_lock();
+ for (i = 0; i < ARRAY_SIZE(mvm->fw_id_to_mac_id); i++) {
+ struct iwl_mvm_sta *sta;
+
+ if (!energy[i])
+ continue;
+
+ sta = iwl_mvm_sta_from_staid_rcu(mvm, i);
+ if (!sta)
+ continue;
+ sta->avg_energy = energy[i];
+ }
+ rcu_read_unlock();
}
void iwl_mvm_rx_statistics(struct iwl_mvm *mvm, struct iwl_rx_cmd_buffer *rxb)
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c b/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c
index d79e9c2..966cd75 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c
@@ -7,7 +7,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2015 - 2016 Intel Deutschland GmbH
+ * Copyright(c) 2015 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -29,7 +29,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2015 - 2016 Intel Deutschland GmbH
+ * Copyright(c) 2015 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -462,6 +462,7 @@
int i;
u16 sn = 0, index = 0;
bool expired = false;
+ bool cont = false;
spin_lock(&buf->lock);
@@ -473,12 +474,21 @@
for (i = 0; i < buf->buf_size ; i++) {
index = (buf->head_sn + i) % buf->buf_size;
- if (skb_queue_empty(&buf->entries[index]))
+ if (skb_queue_empty(&buf->entries[index])) {
+ /*
+ * If there is a hole and the next frame didn't expire
+ * we want to break and not advance SN
+ */
+ cont = false;
continue;
- if (!time_after(jiffies, buf->reorder_time[index] +
- RX_REORDER_BUF_TIMEOUT_MQ))
+ }
+ if (!cont && !time_after(jiffies, buf->reorder_time[index] +
+ RX_REORDER_BUF_TIMEOUT_MQ))
break;
+
expired = true;
+ /* continue until next hole after this expired frames */
+ cont = true;
sn = ieee80211_sn_add(buf->head_sn, i + 1);
}
@@ -626,9 +636,13 @@
return false;
baid_data = rcu_dereference(mvm->baid_map[baid]);
- if (WARN(!baid_data,
- "Received baid %d, but no data exists for this BAID\n", baid))
+ if (!baid_data) {
+ WARN(!(reorder & IWL_RX_MPDU_REORDER_BA_OLD_SN),
+ "Received baid %d, but no data exists for this BAID\n",
+ baid);
return false;
+ }
+
if (WARN(tid != baid_data->tid || mvm_sta->sta_id != baid_data->sta_id,
"baid 0x%x is mapped to sta:%d tid:%d, but was received for sta:%d tid:%d\n",
baid, baid_data->sta_id, baid_data->tid, mvm_sta->sta_id,
@@ -643,6 +657,14 @@
spin_lock_bh(&buffer->lock);
+ if (!buffer->valid) {
+ if (reorder & IWL_RX_MPDU_REORDER_BA_OLD_SN) {
+ spin_unlock_bh(&buffer->lock);
+ return false;
+ }
+ buffer->valid = true;
+ }
+
if (ieee80211_is_back_req(hdr->frame_control)) {
iwl_mvm_release_frames(mvm, sta, napi, buffer, nssn);
goto drop;
@@ -727,7 +749,8 @@
return true;
}
-static void iwl_mvm_agg_rx_received(struct iwl_mvm *mvm, u8 baid)
+static void iwl_mvm_agg_rx_received(struct iwl_mvm *mvm,
+ u32 reorder_data, u8 baid)
{
unsigned long now = jiffies;
unsigned long timeout;
@@ -736,8 +759,10 @@
rcu_read_lock();
data = rcu_dereference(mvm->baid_map[baid]);
- if (WARN_ON(!data))
+ if (!data) {
+ WARN_ON(!(reorder_data & IWL_RX_MPDU_REORDER_BA_OLD_SN));
goto out;
+ }
if (!data->timeout)
goto out;
@@ -799,7 +824,7 @@
}
/* set the preamble flag if appropriate */
if (phy_info & IWL_RX_MPDU_PHY_SHORT_PREAMBLE)
- rx_status->flag |= RX_FLAG_SHORTPRE;
+ rx_status->enc_flags |= RX_ENC_FLAG_SHORTPRE;
if (likely(!(phy_info & IWL_RX_MPDU_PHY_TSF_OVERLOAD))) {
rx_status->mactime = le64_to_cpu(desc->tsf_on_air_rise);
@@ -831,7 +856,7 @@
if (le16_to_cpu(desc->status) & IWL_RX_MPDU_STATUS_SRC_STA_FOUND) {
u8 id = desc->sta_id_flags & IWL_RX_MPDU_SIF_STA_ID_MASK;
- if (!WARN_ON_ONCE(id >= IWL_MVM_STATION_COUNT)) {
+ if (!WARN_ON_ONCE(id >= ARRAY_SIZE(mvm->fw_id_to_mac_id))) {
sta = rcu_dereference(mvm->fw_id_to_mac_id[id]);
if (IS_ERR(sta))
sta = NULL;
@@ -893,26 +918,39 @@
if (iwl_mvm_is_nonagg_dup(sta, queue, rx_status, hdr, desc)) {
kfree_skb(skb);
- rcu_read_unlock();
- return;
+ goto out;
}
/*
* Our hardware de-aggregates AMSDUs but copies the mac header
* as it to the de-aggregated MPDUs. We need to turn off the
* AMSDU bit in the QoS control ourselves.
+ * In addition, HW reverses addr3 and addr4 - reverse it back.
*/
if ((desc->mac_flags2 & IWL_RX_MPDU_MFLG2_AMSDU) &&
!WARN_ON(!ieee80211_is_data_qos(hdr->frame_control))) {
+ int i;
u8 *qc = ieee80211_get_qos_ctl(hdr);
+ u8 mac_addr[ETH_ALEN];
*qc &= ~IEEE80211_QOS_CTL_A_MSDU_PRESENT;
- if (!(desc->amsdu_info &
- IWL_RX_MPDU_AMSDU_LAST_SUBFRAME))
- rx_status->flag |= RX_FLAG_AMSDU_MORE;
+
+ for (i = 0; i < ETH_ALEN; i++)
+ mac_addr[i] = hdr->addr3[ETH_ALEN - i - 1];
+ ether_addr_copy(hdr->addr3, mac_addr);
+
+ if (ieee80211_has_a4(hdr->frame_control)) {
+ for (i = 0; i < ETH_ALEN; i++)
+ mac_addr[i] =
+ hdr->addr4[ETH_ALEN - i - 1];
+ ether_addr_copy(hdr->addr4, mac_addr);
+ }
}
- if (baid != IWL_RX_REORDER_DATA_INVALID_BAID)
- iwl_mvm_agg_rx_received(mvm, baid);
+ if (baid != IWL_RX_REORDER_DATA_INVALID_BAID) {
+ u32 reorder_data = le32_to_cpu(desc->reorder_data);
+
+ iwl_mvm_agg_rx_received(mvm, reorder_data, baid);
+ }
}
/* Set up the HT phy flags */
@@ -920,42 +958,50 @@
case RATE_MCS_CHAN_WIDTH_20:
break;
case RATE_MCS_CHAN_WIDTH_40:
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
break;
case RATE_MCS_CHAN_WIDTH_80:
- rx_status->vht_flag |= RX_VHT_FLAG_80MHZ;
+ rx_status->bw = RATE_INFO_BW_80;
break;
case RATE_MCS_CHAN_WIDTH_160:
- rx_status->vht_flag |= RX_VHT_FLAG_160MHZ;
+ rx_status->bw = RATE_INFO_BW_160;
break;
}
if (rate_n_flags & RATE_MCS_SGI_MSK)
- rx_status->flag |= RX_FLAG_SHORT_GI;
+ rx_status->enc_flags |= RX_ENC_FLAG_SHORT_GI;
if (rate_n_flags & RATE_HT_MCS_GF_MSK)
- rx_status->flag |= RX_FLAG_HT_GF;
+ rx_status->enc_flags |= RX_ENC_FLAG_HT_GF;
if (rate_n_flags & RATE_MCS_LDPC_MSK)
- rx_status->flag |= RX_FLAG_LDPC;
+ rx_status->enc_flags |= RX_ENC_FLAG_LDPC;
if (rate_n_flags & RATE_MCS_HT_MSK) {
- u8 stbc = (rate_n_flags & RATE_MCS_HT_STBC_MSK) >>
+ u8 stbc = (rate_n_flags & RATE_MCS_STBC_MSK) >>
RATE_MCS_STBC_POS;
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
rx_status->rate_idx = rate_n_flags & RATE_HT_MCS_INDEX_MSK;
- rx_status->flag |= stbc << RX_FLAG_STBC_SHIFT;
+ rx_status->enc_flags |= stbc << RX_ENC_FLAG_STBC_SHIFT;
} else if (rate_n_flags & RATE_MCS_VHT_MSK) {
- u8 stbc = (rate_n_flags & RATE_MCS_VHT_STBC_MSK) >>
+ u8 stbc = (rate_n_flags & RATE_MCS_STBC_MSK) >>
RATE_MCS_STBC_POS;
- rx_status->vht_nss =
+ rx_status->nss =
((rate_n_flags & RATE_VHT_MCS_NSS_MSK) >>
RATE_VHT_MCS_NSS_POS) + 1;
rx_status->rate_idx = rate_n_flags & RATE_VHT_MCS_RATE_CODE_MSK;
- rx_status->flag |= RX_FLAG_VHT;
- rx_status->flag |= stbc << RX_FLAG_STBC_SHIFT;
+ rx_status->encoding = RX_ENC_VHT;
+ rx_status->enc_flags |= stbc << RX_ENC_FLAG_STBC_SHIFT;
if (rate_n_flags & RATE_MCS_BF_MSK)
- rx_status->vht_flag |= RX_VHT_FLAG_BF;
+ rx_status->enc_flags |= RX_ENC_FLAG_BF;
} else {
- rx_status->rate_idx =
- iwl_mvm_legacy_rate_to_mac80211_idx(rate_n_flags,
- rx_status->band);
+ int rate = iwl_mvm_legacy_rate_to_mac80211_idx(rate_n_flags,
+ rx_status->band);
+
+ if (WARN(rate < 0 || rate > 0xFF,
+ "Invalid rate flags 0x%x, band %d,\n",
+ rate_n_flags, rx_status->band)) {
+ kfree_skb(skb);
+ goto out;
+ }
+ rx_status->rate_idx = rate;
+
}
/* management stuff on default queue */
@@ -974,6 +1020,7 @@
iwl_mvm_create_skb(skb, hdr, len, crypt_len, rxb);
if (!iwl_mvm_reorder(mvm, napi, queue, sta, skb, desc))
iwl_mvm_pass_packet_to_mac80211(mvm, napi, skb, queue, sta);
+out:
rcu_read_unlock();
}
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/scan.c b/drivers/net/wireless/intel/iwlwifi/mvm/scan.c
index 0a64efa..8d1b994 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/scan.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/scan.c
@@ -7,7 +7,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -34,7 +34,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -81,44 +81,30 @@
IWL_MVM_TRAFFIC_HIGH,
};
+#define IWL_SCAN_DWELL_ACTIVE 10
+#define IWL_SCAN_DWELL_PASSIVE 110
+#define IWL_SCAN_DWELL_FRAGMENTED 44
+#define IWL_SCAN_DWELL_EXTENDED 90
+
struct iwl_mvm_scan_timing_params {
- u32 dwell_active;
- u32 dwell_passive;
- u32 dwell_fragmented;
- u32 dwell_extended;
u32 suspend_time;
u32 max_out_time;
};
static struct iwl_mvm_scan_timing_params scan_timing[] = {
[IWL_SCAN_TYPE_UNASSOC] = {
- .dwell_active = 10,
- .dwell_passive = 110,
- .dwell_fragmented = 44,
- .dwell_extended = 90,
.suspend_time = 0,
.max_out_time = 0,
},
[IWL_SCAN_TYPE_WILD] = {
- .dwell_active = 10,
- .dwell_passive = 110,
- .dwell_fragmented = 44,
- .dwell_extended = 90,
.suspend_time = 30,
.max_out_time = 120,
},
[IWL_SCAN_TYPE_MILD] = {
- .dwell_active = 10,
- .dwell_passive = 110,
- .dwell_fragmented = 44,
- .dwell_extended = 90,
.suspend_time = 120,
.max_out_time = 120,
},
[IWL_SCAN_TYPE_FRAGMENTED] = {
- .dwell_active = 10,
- .dwell_passive = 110,
- .dwell_fragmented = 44,
.suspend_time = 95,
.max_out_time = 44,
},
@@ -294,34 +280,15 @@
return max_ie_len;
}
-static u8 *iwl_mvm_dump_channel_list(struct iwl_scan_results_notif *res,
- int num_res, u8 *buf, size_t buf_size)
-{
- int i;
- u8 *pos = buf, *end = buf + buf_size;
-
- for (i = 0; pos < end && i < num_res; i++)
- pos += snprintf(pos, end - pos, " %u", res[i].channel);
-
- /* terminate the string in case the buffer was too short */
- *(buf + buf_size - 1) = '\0';
-
- return buf;
-}
-
void iwl_mvm_rx_lmac_scan_iter_complete_notif(struct iwl_mvm *mvm,
struct iwl_rx_cmd_buffer *rxb)
{
struct iwl_rx_packet *pkt = rxb_addr(rxb);
struct iwl_lmac_scan_complete_notif *notif = (void *)pkt->data;
- u8 buf[256];
IWL_DEBUG_SCAN(mvm,
- "Scan offload iteration complete: status=0x%x scanned channels=%d channels list: %s\n",
- notif->status, notif->scanned_channels,
- iwl_mvm_dump_channel_list(notif->results,
- notif->scanned_channels, buf,
- sizeof(buf)));
+ "Scan offload iteration complete: status=0x%x scanned channels=%d\n",
+ notif->status, notif->scanned_channels);
if (mvm->sched_scan_pass_all == SCHED_SCAN_PASS_ALL_FOUND) {
IWL_DEBUG_SCAN(mvm, "Pass all scheduled scan results found\n");
@@ -743,10 +710,10 @@
struct iwl_scan_req_lmac *cmd,
struct iwl_mvm_scan_params *params)
{
- cmd->active_dwell = scan_timing[params->type].dwell_active;
- cmd->passive_dwell = scan_timing[params->type].dwell_passive;
- cmd->fragmented_dwell = scan_timing[params->type].dwell_fragmented;
- cmd->extended_dwell = scan_timing[params->type].dwell_extended;
+ cmd->active_dwell = IWL_SCAN_DWELL_ACTIVE;
+ cmd->passive_dwell = IWL_SCAN_DWELL_PASSIVE;
+ cmd->fragmented_dwell = IWL_SCAN_DWELL_FRAGMENTED;
+ cmd->extended_dwell = IWL_SCAN_DWELL_EXTENDED;
cmd->max_out_time = cpu_to_le32(scan_timing[params->type].max_out_time);
cmd->suspend_time = cpu_to_le32(scan_timing[params->type].suspend_time);
cmd->scan_prio = cpu_to_le32(IWL_SCAN_PRIORITY_EXT_6);
@@ -944,13 +911,12 @@
}
static void iwl_mvm_fill_scan_dwell(struct iwl_mvm *mvm,
- struct iwl_scan_dwell *dwell,
- struct iwl_mvm_scan_timing_params *timing)
+ struct iwl_scan_dwell *dwell)
{
- dwell->active = timing->dwell_active;
- dwell->passive = timing->dwell_passive;
- dwell->fragmented = timing->dwell_fragmented;
- dwell->extended = timing->dwell_extended;
+ dwell->active = IWL_SCAN_DWELL_ACTIVE;
+ dwell->passive = IWL_SCAN_DWELL_PASSIVE;
+ dwell->fragmented = IWL_SCAN_DWELL_FRAGMENTED;
+ dwell->extended = IWL_SCAN_DWELL_EXTENDED;
}
static void iwl_mvm_fill_channels(struct iwl_mvm *mvm, u8 *channels)
@@ -966,6 +932,29 @@
channels[j] = band->channels[i].hw_value;
}
+static void iwl_mvm_fill_scan_config_v1(struct iwl_mvm *mvm, void *config,
+ u32 flags, u8 channel_flags)
+{
+ enum iwl_mvm_scan_type type = iwl_mvm_get_scan_type(mvm, false);
+ struct iwl_scan_config_v1 *cfg = config;
+
+ cfg->flags = cpu_to_le32(flags);
+ cfg->tx_chains = cpu_to_le32(iwl_mvm_get_valid_tx_ant(mvm));
+ cfg->rx_chains = cpu_to_le32(iwl_mvm_scan_rx_ant(mvm));
+ cfg->legacy_rates = iwl_mvm_scan_config_rates(mvm);
+ cfg->out_of_channel_time = cpu_to_le32(scan_timing[type].max_out_time);
+ cfg->suspend_time = cpu_to_le32(scan_timing[type].suspend_time);
+
+ iwl_mvm_fill_scan_dwell(mvm, &cfg->dwell);
+
+ memcpy(&cfg->mac_addr, &mvm->addresses[0].addr, ETH_ALEN);
+
+ cfg->bcast_sta_id = mvm->aux_sta.sta_id;
+ cfg->channel_flags = channel_flags;
+
+ iwl_mvm_fill_channels(mvm, cfg->channel_array);
+}
+
static void iwl_mvm_fill_scan_config(struct iwl_mvm *mvm, void *config,
u32 flags, u8 channel_flags)
{
@@ -976,37 +965,18 @@
cfg->tx_chains = cpu_to_le32(iwl_mvm_get_valid_tx_ant(mvm));
cfg->rx_chains = cpu_to_le32(iwl_mvm_scan_rx_ant(mvm));
cfg->legacy_rates = iwl_mvm_scan_config_rates(mvm);
- cfg->out_of_channel_time = cpu_to_le32(scan_timing[type].max_out_time);
- cfg->suspend_time = cpu_to_le32(scan_timing[type].suspend_time);
-
- iwl_mvm_fill_scan_dwell(mvm, &cfg->dwell, &scan_timing[type]);
-
- memcpy(&cfg->mac_addr, &mvm->addresses[0].addr, ETH_ALEN);
-
- cfg->bcast_sta_id = mvm->aux_sta.sta_id;
- cfg->channel_flags = channel_flags;
-
- iwl_mvm_fill_channels(mvm, cfg->channel_array);
-}
-
-static void iwl_mvm_fill_scan_config_cdb(struct iwl_mvm *mvm, void *config,
- u32 flags, u8 channel_flags)
-{
- enum iwl_mvm_scan_type type = iwl_mvm_get_scan_type(mvm, false);
- struct iwl_scan_config_cdb *cfg = config;
-
- cfg->flags = cpu_to_le32(flags);
- cfg->tx_chains = cpu_to_le32(iwl_mvm_get_valid_tx_ant(mvm));
- cfg->rx_chains = cpu_to_le32(iwl_mvm_scan_rx_ant(mvm));
- cfg->legacy_rates = iwl_mvm_scan_config_rates(mvm);
cfg->out_of_channel_time[0] =
cpu_to_le32(scan_timing[type].max_out_time);
- cfg->out_of_channel_time[1] =
- cpu_to_le32(scan_timing[type].max_out_time);
cfg->suspend_time[0] = cpu_to_le32(scan_timing[type].suspend_time);
- cfg->suspend_time[1] = cpu_to_le32(scan_timing[type].suspend_time);
- iwl_mvm_fill_scan_dwell(mvm, &cfg->dwell, &scan_timing[type]);
+ if (iwl_mvm_is_cdb_supported(mvm)) {
+ cfg->suspend_time[1] =
+ cpu_to_le32(scan_timing[type].suspend_time);
+ cfg->out_of_channel_time[1] =
+ cpu_to_le32(scan_timing[type].max_out_time);
+ }
+
+ iwl_mvm_fill_scan_dwell(mvm, &cfg->dwell);
memcpy(&cfg->mac_addr, &mvm->addresses[0].addr, ETH_ALEN);
@@ -1033,16 +1003,13 @@
if (WARN_ON(num_channels > mvm->fw->ucode_capa.n_scan_channels))
return -ENOBUFS;
- if (type == mvm->scan_type) {
- IWL_DEBUG_SCAN(mvm,
- "Ignoring UMAC scan config of the same type\n");
+ if (type == mvm->scan_type)
return 0;
- }
- if (iwl_mvm_is_cdb_supported(mvm))
- cmd_size = sizeof(struct iwl_scan_config_cdb);
- else
+ if (iwl_mvm_has_new_tx_api(mvm))
cmd_size = sizeof(struct iwl_scan_config);
+ else
+ cmd_size = sizeof(struct iwl_scan_config_v1);
cmd_size += mvm->fw->ucode_capa.n_scan_channels;
cfg = kzalloc(cmd_size, GFP_KERNEL);
@@ -1068,13 +1035,13 @@
IWL_CHANNEL_FLAG_EBS_ADD |
IWL_CHANNEL_FLAG_PRE_SCAN_PASSIVE2ACTIVE;
- if (iwl_mvm_is_cdb_supported(mvm)) {
+ if (iwl_mvm_has_new_tx_api(mvm)) {
flags |= (type == IWL_SCAN_TYPE_FRAGMENTED) ?
SCAN_CONFIG_FLAG_SET_LMAC2_FRAGMENTED :
SCAN_CONFIG_FLAG_CLEAR_LMAC2_FRAGMENTED;
- iwl_mvm_fill_scan_config_cdb(mvm, cfg, flags, channel_flags);
- } else {
iwl_mvm_fill_scan_config(mvm, cfg, flags, channel_flags);
+ } else {
+ iwl_mvm_fill_scan_config_v1(mvm, cfg, flags, channel_flags);
}
cmd.data[0] = cfg;
@@ -1113,22 +1080,26 @@
cmd->passive_dwell = params->measurement_dwell;
cmd->extended_dwell = params->measurement_dwell;
} else {
- cmd->active_dwell = timing->dwell_active;
- cmd->passive_dwell = timing->dwell_passive;
- cmd->extended_dwell = timing->dwell_extended;
+ cmd->active_dwell = IWL_SCAN_DWELL_ACTIVE;
+ cmd->passive_dwell = IWL_SCAN_DWELL_PASSIVE;
+ cmd->extended_dwell = IWL_SCAN_DWELL_EXTENDED;
}
- cmd->fragmented_dwell = timing->dwell_fragmented;
+ cmd->fragmented_dwell = IWL_SCAN_DWELL_FRAGMENTED;
- if (iwl_mvm_is_cdb_supported(mvm)) {
- cmd->cdb.max_out_time[0] = cpu_to_le32(timing->max_out_time);
- cmd->cdb.suspend_time[0] = cpu_to_le32(timing->suspend_time);
- cmd->cdb.max_out_time[1] = cpu_to_le32(timing->max_out_time);
- cmd->cdb.suspend_time[1] = cpu_to_le32(timing->suspend_time);
- cmd->cdb.scan_priority = cpu_to_le32(IWL_SCAN_PRIORITY_EXT_6);
+ if (iwl_mvm_has_new_tx_api(mvm)) {
+ cmd->v6.scan_priority = cpu_to_le32(IWL_SCAN_PRIORITY_EXT_6);
+ cmd->v6.max_out_time[0] = cpu_to_le32(timing->max_out_time);
+ cmd->v6.suspend_time[0] = cpu_to_le32(timing->suspend_time);
+ if (iwl_mvm_is_cdb_supported(mvm)) {
+ cmd->v6.max_out_time[1] =
+ cpu_to_le32(timing->max_out_time);
+ cmd->v6.suspend_time[1] =
+ cpu_to_le32(timing->suspend_time);
+ }
} else {
- cmd->no_cdb.max_out_time = cpu_to_le32(timing->max_out_time);
- cmd->no_cdb.suspend_time = cpu_to_le32(timing->suspend_time);
- cmd->no_cdb.scan_priority =
+ cmd->v1.max_out_time = cpu_to_le32(timing->max_out_time);
+ cmd->v1.suspend_time = cpu_to_le32(timing->suspend_time);
+ cmd->v1.scan_priority =
cpu_to_le32(IWL_SCAN_PRIORITY_EXT_6);
}
@@ -1207,8 +1178,8 @@
int type)
{
struct iwl_scan_req_umac *cmd = mvm->scan_cmd;
- void *cmd_data = iwl_mvm_is_cdb_supported(mvm) ?
- (void *)&cmd->cdb.data : (void *)&cmd->no_cdb.data;
+ void *cmd_data = iwl_mvm_has_new_tx_api(mvm) ?
+ (void *)&cmd->v6.data : (void *)&cmd->v1.data;
struct iwl_scan_req_umac_tail *sec_part = cmd_data +
sizeof(struct iwl_scan_channel_cfg_umac) *
mvm->fw->ucode_capa.n_scan_channels;
@@ -1245,12 +1216,12 @@
IWL_SCAN_CHANNEL_FLAG_EBS_ACCURATE |
IWL_SCAN_CHANNEL_FLAG_CACHE_ADD;
- if (iwl_mvm_is_cdb_supported(mvm)) {
- cmd->cdb.channel_flags = channel_flags;
- cmd->cdb.n_channels = params->n_channels;
+ if (iwl_mvm_has_new_tx_api(mvm)) {
+ cmd->v6.channel_flags = channel_flags;
+ cmd->v6.n_channels = params->n_channels;
} else {
- cmd->no_cdb.channel_flags = channel_flags;
- cmd->no_cdb.n_channels = params->n_channels;
+ cmd->v1.channel_flags = channel_flags;
+ cmd->v1.n_channels = params->n_channels;
}
iwl_scan_build_ssids(params, sec_part->direct_scan, &ssid_bitmap);
@@ -1607,16 +1578,12 @@
{
struct iwl_rx_packet *pkt = rxb_addr(rxb);
struct iwl_umac_scan_iter_complete_notif *notif = (void *)pkt->data;
- u8 buf[256];
mvm->scan_start = le64_to_cpu(notif->start_tsf);
IWL_DEBUG_SCAN(mvm,
- "UMAC Scan iteration complete: status=0x%x scanned_channels=%d channels list: %s\n",
- notif->status, notif->scanned_channels,
- iwl_mvm_dump_channel_list(notif->results,
- notif->scanned_channels, buf,
- sizeof(buf)));
+ "UMAC Scan iteration complete: status=0x%x scanned_channels=%d\n",
+ notif->status, notif->scanned_channels);
if (mvm->sched_scan_pass_all == SCHED_SCAN_PASS_ALL_FOUND) {
IWL_DEBUG_SCAN(mvm, "Pass all scheduled scan results found\n");
@@ -1692,10 +1659,10 @@
int iwl_mvm_scan_size(struct iwl_mvm *mvm)
{
- int base_size = IWL_SCAN_REQ_UMAC_SIZE;
+ int base_size = IWL_SCAN_REQ_UMAC_SIZE_V1;
- if (iwl_mvm_is_cdb_supported(mvm))
- base_size = IWL_SCAN_REQ_UMAC_SIZE_CDB;
+ if (iwl_mvm_has_new_tx_api(mvm))
+ base_size = IWL_SCAN_REQ_UMAC_SIZE;
if (fw_has_capa(&mvm->fw->ucode_capa, IWL_UCODE_TLV_CAPA_UMAC_SCAN))
return base_size +
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/sf.c b/drivers/net/wireless/intel/iwlwifi/mvm/sf.c
index 101fb04..539b06b 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/sf.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/sf.c
@@ -235,7 +235,7 @@
iwl_mvm_fill_sf_command(mvm, &sf_cmd, NULL);
break;
case SF_FULL_ON:
- if (sta_id == IWL_MVM_STATION_COUNT) {
+ if (sta_id == IWL_MVM_INVALID_STA) {
IWL_ERR(mvm,
"No station: Cannot switch SF to FULL_ON\n");
return -EINVAL;
@@ -276,12 +276,12 @@
bool remove_vif)
{
enum iwl_sf_state new_state;
- u8 sta_id = IWL_MVM_STATION_COUNT;
+ u8 sta_id = IWL_MVM_INVALID_STA;
struct iwl_mvm_vif *mvmvif = NULL;
struct iwl_mvm_active_iface_iterator_data data = {
.ignore_vif = changed_vif,
.sta_vif_state = SF_UNINIT,
- .sta_vif_ap_sta_id = IWL_MVM_STATION_COUNT,
+ .sta_vif_ap_sta_id = IWL_MVM_INVALID_STA,
};
/*
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/sta.c b/drivers/net/wireless/intel/iwlwifi/mvm/sta.c
index 9d28db7..f5c786d 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/sta.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/sta.c
@@ -7,7 +7,7 @@
*
* Copyright(c) 2012 - 2015 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -34,7 +34,7 @@
*
* Copyright(c) 2012 - 2015 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -77,9 +77,11 @@
*/
static inline int iwl_mvm_add_sta_cmd_size(struct iwl_mvm *mvm)
{
- return iwl_mvm_has_new_rx_api(mvm) ?
- sizeof(struct iwl_mvm_add_sta_cmd) :
- sizeof(struct iwl_mvm_add_sta_cmd_v7);
+ if (iwl_mvm_has_new_rx_api(mvm) ||
+ fw_has_api(&mvm->fw->ucode_capa, IWL_UCODE_TLV_API_STA_TYPE))
+ return sizeof(struct iwl_mvm_add_sta_cmd);
+ else
+ return sizeof(struct iwl_mvm_add_sta_cmd_v7);
}
static int iwl_mvm_find_free_sta_id(struct iwl_mvm *mvm,
@@ -98,7 +100,7 @@
reserved_ids = BIT(0);
/* Don't take rcu_read_lock() since we are protected by mvm->mutex */
- for (sta_id = 0; sta_id < IWL_MVM_STATION_COUNT; sta_id++) {
+ for (sta_id = 0; sta_id < ARRAY_SIZE(mvm->fw_id_to_mac_id); sta_id++) {
if (BIT(sta_id) & reserved_ids)
continue;
@@ -106,7 +108,7 @@
lockdep_is_held(&mvm->mutex)))
return sta_id;
}
- return IWL_MVM_STATION_COUNT;
+ return IWL_MVM_INVALID_STA;
}
/* send station add/update command to firmware */
@@ -126,12 +128,21 @@
u32 status;
u32 agg_size = 0, mpdu_dens = 0;
+ if (fw_has_api(&mvm->fw->ucode_capa, IWL_UCODE_TLV_API_STA_TYPE))
+ add_sta_cmd.station_type = mvm_sta->sta_type;
+
if (!update || (flags & STA_MODIFY_QUEUES)) {
- add_sta_cmd.tfd_queue_msk = cpu_to_le32(mvm_sta->tfd_queue_msk);
memcpy(&add_sta_cmd.addr, sta->addr, ETH_ALEN);
- if (flags & STA_MODIFY_QUEUES)
- add_sta_cmd.modify_mask |= STA_MODIFY_QUEUES;
+ if (!iwl_mvm_has_new_tx_api(mvm)) {
+ add_sta_cmd.tfd_queue_msk =
+ cpu_to_le32(mvm_sta->tfd_queue_msk);
+
+ if (flags & STA_MODIFY_QUEUES)
+ add_sta_cmd.modify_mask |= STA_MODIFY_QUEUES;
+ } else {
+ WARN_ON(flags & STA_MODIFY_QUEUES);
+ }
}
switch (sta->bandwidth) {
@@ -209,13 +220,15 @@
add_sta_cmd.modify_mask |= STA_MODIFY_UAPSD_ACS;
if (sta->uapsd_queues & IEEE80211_WMM_IE_STA_QOSINFO_AC_BK)
- add_sta_cmd.uapsd_trigger_acs |= BIT(AC_BK);
+ add_sta_cmd.uapsd_acs |= BIT(AC_BK);
if (sta->uapsd_queues & IEEE80211_WMM_IE_STA_QOSINFO_AC_BE)
- add_sta_cmd.uapsd_trigger_acs |= BIT(AC_BE);
+ add_sta_cmd.uapsd_acs |= BIT(AC_BE);
if (sta->uapsd_queues & IEEE80211_WMM_IE_STA_QOSINFO_AC_VI)
- add_sta_cmd.uapsd_trigger_acs |= BIT(AC_VI);
+ add_sta_cmd.uapsd_acs |= BIT(AC_VI);
if (sta->uapsd_queues & IEEE80211_WMM_IE_STA_QOSINFO_AC_VO)
- add_sta_cmd.uapsd_trigger_acs |= BIT(AC_VO);
+ add_sta_cmd.uapsd_acs |= BIT(AC_VO);
+ add_sta_cmd.uapsd_acs |= add_sta_cmd.uapsd_acs << 4;
+ add_sta_cmd.sp_length = sta->max_sp ? sta->max_sp * 2 : 128;
}
status = ADD_STA_SUCCESS;
@@ -337,6 +350,9 @@
u8 sta_id;
int ret;
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return -EINVAL;
+
spin_lock_bh(&mvm->queue_info_lock);
sta_id = mvm->queue_info[queue].ra_sta_id;
spin_unlock_bh(&mvm->queue_info_lock);
@@ -387,6 +403,9 @@
lockdep_assert_held(&mvm->mutex);
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return -EINVAL;
+
spin_lock_bh(&mvm->queue_info_lock);
sta_id = mvm->queue_info[queue].ra_sta_id;
tid_bitmap = mvm->queue_info[queue].tid_bitmap;
@@ -426,6 +445,9 @@
lockdep_assert_held(&mvm->mutex);
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return -EINVAL;
+
spin_lock_bh(&mvm->queue_info_lock);
sta_id = mvm->queue_info[queue].ra_sta_id;
tid_bitmap = mvm->queue_info[queue].tid_bitmap;
@@ -447,7 +469,7 @@
for_each_set_bit(tid, &tid_bitmap, IWL_MAX_TID_COUNT + 1) {
if (mvmsta->tid_data[tid].state == IWL_AGG_ON)
disable_agg_tids |= BIT(tid);
- mvmsta->tid_data[tid].txq_id = IEEE80211_INVAL_HW_QUEUE;
+ mvmsta->tid_data[tid].txq_id = IWL_MVM_INVALID_QUEUE;
}
mvmsta->tfd_queue_msk &= ~BIT(queue); /* Don't use this queue anymore */
@@ -468,6 +490,9 @@
lockdep_assert_held(&mvm->mutex);
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return -EINVAL;
+
spin_lock_bh(&mvm->queue_info_lock);
txq_curr_ac = mvm->queue_info[queue].mac80211_ac;
sta_id = mvm->queue_info[queue].ra_sta_id;
@@ -475,6 +500,8 @@
spin_unlock_bh(&mvm->queue_info_lock);
mvmsta = iwl_mvm_sta_from_staid_protected(mvm, sta_id);
+ if (WARN_ON(!mvmsta))
+ return -EINVAL;
disable_agg_tids = iwl_mvm_remove_sta_queue_marking(mvm, queue);
/* Disable the queue */
@@ -512,6 +539,8 @@
int i;
lockdep_assert_held(&mvm->queue_info_lock);
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return -EINVAL;
memset(&ac_to_queue, IEEE80211_INVAL_HW_QUEUE, sizeof(ac_to_queue));
@@ -596,6 +625,9 @@
unsigned long mq;
int ret;
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return -EINVAL;
+
/*
* If the AC is lower than current one - FIFO needs to be redirected to
* the lowest one of the streams in the queue. Check if this is needed
@@ -617,7 +649,7 @@
cmd.sta_id = mvm->queue_info[queue].ra_sta_id;
cmd.tx_fifo = iwl_mvm_ac_to_tx_fifo[mvm->queue_info[queue].mac80211_ac];
cmd.tid = mvm->queue_info[queue].txq_tid;
- mq = mvm->queue_info[queue].hw_queue_to_mac80211;
+ mq = mvm->hw_queue_to_mac80211[queue];
shared_queue = (mvm->queue_info[queue].hw_queue_refcount > 1);
spin_unlock_bh(&mvm->queue_info_lock);
@@ -626,7 +658,7 @@
/* Stop MAC queues and wait for this queue to empty */
iwl_mvm_stop_mac_queues(mvm, mq);
- ret = iwl_trans_wait_tx_queue_empty(mvm->trans, BIT(queue));
+ ret = iwl_trans_wait_tx_queues_empty(mvm->trans, BIT(queue));
if (ret) {
IWL_ERR(mvm, "Error draining queue %d before reconfig\n",
queue);
@@ -677,6 +709,37 @@
return ret;
}
+static int iwl_mvm_sta_alloc_queue_tvqm(struct iwl_mvm *mvm,
+ struct ieee80211_sta *sta, u8 ac,
+ int tid)
+{
+ struct iwl_mvm_sta *mvmsta = iwl_mvm_sta_from_mac80211(sta);
+ unsigned int wdg_timeout =
+ iwl_mvm_get_wd_timeout(mvm, mvmsta->vif, false, false);
+ u8 mac_queue = mvmsta->vif->hw_queue[ac];
+ int queue = -1;
+
+ lockdep_assert_held(&mvm->mutex);
+
+ IWL_DEBUG_TX_QUEUES(mvm,
+ "Allocating queue for sta %d on tid %d\n",
+ mvmsta->sta_id, tid);
+ queue = iwl_mvm_tvqm_enable_txq(mvm, mac_queue, mvmsta->sta_id, tid,
+ wdg_timeout);
+ if (queue < 0)
+ return queue;
+
+ IWL_DEBUG_TX_QUEUES(mvm, "Allocated queue is %d\n", queue);
+
+ spin_lock_bh(&mvmsta->lock);
+ mvmsta->tid_data[tid].txq_id = queue;
+ mvmsta->tid_data[tid].is_tid_active = true;
+ mvmsta->tfd_queue_msk |= BIT(queue);
+ spin_unlock_bh(&mvmsta->lock);
+
+ return 0;
+}
+
static int iwl_mvm_sta_alloc_queue(struct iwl_mvm *mvm,
struct ieee80211_sta *sta, u8 ac, int tid,
struct ieee80211_hdr *hdr)
@@ -702,6 +765,9 @@
lockdep_assert_held(&mvm->mutex);
+ if (iwl_mvm_has_new_tx_api(mvm))
+ return iwl_mvm_sta_alloc_queue_tvqm(mvm, sta, ac, tid);
+
spin_lock_bh(&mvmsta->lock);
tfd_queue_mask = mvmsta->tfd_queue_msk;
spin_unlock_bh(&mvmsta->lock);
@@ -880,6 +946,9 @@
lockdep_assert_held(&mvm->mutex);
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return;
+
spin_lock_bh(&mvm->queue_info_lock);
tid_bitmap = mvm->queue_info[queue].tid_bitmap;
spin_unlock_bh(&mvm->queue_info_lock);
@@ -917,6 +986,10 @@
int ssn;
int ret = true;
+ /* queue sharing is disabled on new TX path */
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return;
+
lockdep_assert_held(&mvm->mutex);
spin_lock_bh(&mvm->queue_info_lock);
@@ -1014,7 +1087,7 @@
ac = iwl_mvm_tid_to_ac_queue(tid);
mac_queue = IEEE80211_SKB_CB(skb)->hw_queue;
- if (tid_data->txq_id == IEEE80211_INVAL_HW_QUEUE &&
+ if (tid_data->txq_id == IWL_MVM_INVALID_QUEUE &&
iwl_mvm_sta_alloc_queue(mvm, sta, ac, tid, hdr)) {
IWL_ERR(mvm,
"Can't alloc TXQ for sta %d tid %d - dropping frame\n",
@@ -1059,8 +1132,12 @@
mutex_lock(&mvm->mutex);
+ /* No queue reconfiguration in TVQM mode */
+ if (iwl_mvm_has_new_tx_api(mvm))
+ goto alloc_queues;
+
/* Reconfigure queues requiring reconfiguation */
- for (queue = 0; queue < IWL_MAX_HW_QUEUES; queue++) {
+ for (queue = 0; queue < ARRAY_SIZE(mvm->queue_info); queue++) {
bool reconfig;
bool change_owner;
@@ -1088,6 +1165,7 @@
iwl_mvm_change_queue_owner(mvm, queue);
}
+alloc_queues:
/* Go over all stations with deferred traffic */
for_each_set_bit(sta_id, mvm->sta_deferred_frames,
IWL_MVM_STATION_COUNT) {
@@ -1116,6 +1194,10 @@
int queue;
bool using_inactive_queue = false, same_sta = false;
+ /* queue reserving is disabled on new TX path */
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return 0;
+
/*
* Check for inactive queues, so we don't reach a situation where we
* can't add a STA due to a shortage in queues that doesn't really exist
@@ -1191,7 +1273,7 @@
int ac;
u8 mac_queue;
- if (txq_id == IEEE80211_INVAL_HW_QUEUE)
+ if (txq_id == IWL_MVM_INVALID_QUEUE)
continue;
skb_queue_head_init(&tid_data->deferred_tx_frames);
@@ -1199,20 +1281,31 @@
ac = tid_to_mac80211_ac[i];
mac_queue = mvm_sta->vif->hw_queue[ac];
- cfg.tid = i;
- cfg.fifo = iwl_mvm_ac_to_tx_fifo[ac];
- cfg.aggregate = (txq_id >= IWL_MVM_DQA_MIN_DATA_QUEUE ||
- txq_id == IWL_MVM_DQA_BSS_CLIENT_QUEUE);
+ if (iwl_mvm_has_new_tx_api(mvm)) {
+ IWL_DEBUG_TX_QUEUES(mvm,
+ "Re-mapping sta %d tid %d\n",
+ mvm_sta->sta_id, i);
+ txq_id = iwl_mvm_tvqm_enable_txq(mvm, mac_queue,
+ mvm_sta->sta_id,
+ i, wdg_timeout);
+ tid_data->txq_id = txq_id;
+ } else {
+ u16 seq = IEEE80211_SEQ_TO_SN(tid_data->seq_number);
- IWL_DEBUG_TX_QUEUES(mvm,
- "Re-mapping sta %d tid %d to queue %d\n",
- mvm_sta->sta_id, i, txq_id);
+ cfg.tid = i;
+ cfg.fifo = iwl_mvm_ac_to_tx_fifo[ac];
+ cfg.aggregate = (txq_id >= IWL_MVM_DQA_MIN_DATA_QUEUE ||
+ txq_id ==
+ IWL_MVM_DQA_BSS_CLIENT_QUEUE);
- iwl_mvm_enable_txq(mvm, txq_id, mac_queue,
- IEEE80211_SEQ_TO_SN(tid_data->seq_number),
- &cfg, wdg_timeout);
+ IWL_DEBUG_TX_QUEUES(mvm,
+ "Re-mapping sta %d tid %d to queue %d\n",
+ mvm_sta->sta_id, i, txq_id);
- mvm->queue_info[txq_id].status = IWL_MVM_QUEUE_READY;
+ iwl_mvm_enable_txq(mvm, txq_id, mac_queue, seq, &cfg,
+ wdg_timeout);
+ mvm->queue_info[txq_id].status = IWL_MVM_QUEUE_READY;
+ }
}
atomic_set(&mvm->pending_frames[mvm_sta->sta_id], 0);
@@ -1235,7 +1328,7 @@
else
sta_id = mvm_sta->sta_id;
- if (sta_id == IWL_MVM_STATION_COUNT)
+ if (sta_id == IWL_MVM_INVALID_STA)
return -ENOSPC;
spin_lock_init(&mvm_sta->lock);
@@ -1254,6 +1347,7 @@
mvm_sta->max_agg_bufsize = LINK_QUAL_AGG_FRAME_LIMIT_DEF;
mvm_sta->tx_protection = 0;
mvm_sta->tt_tx_protection = false;
+ mvm_sta->sta_type = sta->tdls ? IWL_STA_TDLS_LINK : IWL_STA_LINK;
/* HW restart, don't assume the memory has been zeroed */
atomic_set(&mvm->pending_frames[sta_id], 0);
@@ -1287,7 +1381,7 @@
* Mark all queues for this STA as unallocated and defer TX
* frames until the queue is allocated
*/
- mvm_sta->tid_data[i].txq_id = IEEE80211_INVAL_HW_QUEUE;
+ mvm_sta->tid_data[i].txq_id = IWL_MVM_INVALID_QUEUE;
skb_queue_head_init(&mvm_sta->tid_data[i].deferred_tx_frames);
}
mvm_sta->deferred_traffic_tid_map = 0;
@@ -1303,7 +1397,7 @@
mvm_sta->dup_data = dup_data;
}
- if (iwl_mvm_is_dqa_supported(mvm)) {
+ if (iwl_mvm_is_dqa_supported(mvm) && !iwl_mvm_has_new_tx_api(mvm)) {
ret = iwl_mvm_reserve_sta_stream(mvm, sta,
ieee80211_vif_type_p2p(vif));
if (ret)
@@ -1317,10 +1411,10 @@
if (vif->type == NL80211_IFTYPE_STATION) {
if (!sta->tdls) {
- WARN_ON(mvmvif->ap_sta_id != IWL_MVM_STATION_COUNT);
+ WARN_ON(mvmvif->ap_sta_id != IWL_MVM_INVALID_STA);
mvmvif->ap_sta_id = sta_id;
} else {
- WARN_ON(mvmvif->ap_sta_id == IWL_MVM_STATION_COUNT);
+ WARN_ON(mvmvif->ap_sta_id == IWL_MVM_INVALID_STA);
}
}
@@ -1486,13 +1580,13 @@
lockdep_assert_held(&mvm->mutex);
for (i = 0; i < ARRAY_SIZE(mvm_sta->tid_data); i++) {
- if (mvm_sta->tid_data[i].txq_id == IEEE80211_INVAL_HW_QUEUE)
+ if (mvm_sta->tid_data[i].txq_id == IWL_MVM_INVALID_QUEUE)
continue;
ac = iwl_mvm_tid_to_ac_queue(i);
iwl_mvm_disable_txq(mvm, mvm_sta->tid_data[i].txq_id,
vif->hw_queue[ac], i, 0);
- mvm_sta->tid_data[i].txq_id = IEEE80211_INVAL_HW_QUEUE;
+ mvm_sta->tid_data[i].txq_id = IWL_MVM_INVALID_QUEUE;
}
}
@@ -1520,8 +1614,8 @@
ret = iwl_mvm_flush_tx_path(mvm, mvm_sta->tfd_queue_msk, 0);
if (ret)
return ret;
- ret = iwl_trans_wait_tx_queue_empty(mvm->trans,
- mvm_sta->tfd_queue_msk);
+ ret = iwl_trans_wait_tx_queues_empty(mvm->trans,
+ mvm_sta->tfd_queue_msk);
if (ret)
return ret;
ret = iwl_mvm_drain_sta(mvm, mvm_sta, false);
@@ -1571,11 +1665,11 @@
return ret;
/* unassoc - go ahead - remove the AP STA now */
- mvmvif->ap_sta_id = IWL_MVM_STATION_COUNT;
+ mvmvif->ap_sta_id = IWL_MVM_INVALID_STA;
/* clear d0i3_ap_sta_id if no longer relevant */
if (mvm->d0i3_ap_sta_id == sta_id)
- mvm->d0i3_ap_sta_id = IWL_MVM_STATION_COUNT;
+ mvm->d0i3_ap_sta_id = IWL_MVM_INVALID_STA;
}
}
@@ -1584,7 +1678,7 @@
* before the STA is removed.
*/
if (WARN_ON_ONCE(mvm->tdls_cs.peer.sta_id == sta_id)) {
- mvm->tdls_cs.peer.sta_id = IWL_MVM_STATION_COUNT;
+ mvm->tdls_cs.peer.sta_id = IWL_MVM_INVALID_STA;
cancel_delayed_work(&mvm->tdls_cs.dwork);
}
@@ -1637,27 +1731,28 @@
int iwl_mvm_allocate_int_sta(struct iwl_mvm *mvm,
struct iwl_mvm_int_sta *sta,
- u32 qmask, enum nl80211_iftype iftype)
+ u32 qmask, enum nl80211_iftype iftype,
+ enum iwl_sta_type type)
{
if (!test_bit(IWL_MVM_STATUS_IN_HW_RESTART, &mvm->status)) {
sta->sta_id = iwl_mvm_find_free_sta_id(mvm, iftype);
- if (WARN_ON_ONCE(sta->sta_id == IWL_MVM_STATION_COUNT))
+ if (WARN_ON_ONCE(sta->sta_id == IWL_MVM_INVALID_STA))
return -ENOSPC;
}
sta->tfd_queue_msk = qmask;
+ sta->type = type;
/* put a non-NULL value so iterating over the stations won't stop */
rcu_assign_pointer(mvm->fw_id_to_mac_id[sta->sta_id], ERR_PTR(-EINVAL));
return 0;
}
-static void iwl_mvm_dealloc_int_sta(struct iwl_mvm *mvm,
- struct iwl_mvm_int_sta *sta)
+void iwl_mvm_dealloc_int_sta(struct iwl_mvm *mvm, struct iwl_mvm_int_sta *sta)
{
RCU_INIT_POINTER(mvm->fw_id_to_mac_id[sta->sta_id], NULL);
memset(sta, 0, sizeof(struct iwl_mvm_int_sta));
- sta->sta_id = IWL_MVM_STATION_COUNT;
+ sta->sta_id = IWL_MVM_INVALID_STA;
}
static int iwl_mvm_add_int_sta_common(struct iwl_mvm *mvm,
@@ -1675,8 +1770,11 @@
cmd.sta_id = sta->sta_id;
cmd.mac_id_n_color = cpu_to_le32(FW_CMD_ID_AND_COLOR(mac_id,
color));
+ if (fw_has_api(&mvm->fw->ucode_capa, IWL_UCODE_TLV_API_STA_TYPE))
+ cmd.station_type = sta->type;
- cmd.tfd_queue_msk = cpu_to_le32(sta->tfd_queue_msk);
+ if (!iwl_mvm_has_new_tx_api(mvm))
+ cmd.tfd_queue_msk = cpu_to_le32(sta->tfd_queue_msk);
cmd.tid_disable_tx = cpu_to_le16(0xffff);
if (addr)
@@ -1701,27 +1799,19 @@
return ret;
}
-int iwl_mvm_add_aux_sta(struct iwl_mvm *mvm)
+static void iwl_mvm_enable_aux_queue(struct iwl_mvm *mvm)
{
unsigned int wdg_timeout = iwlmvm_mod_params.tfd_q_hang_detect ?
mvm->cfg->base_params->wd_timeout :
IWL_WATCHDOG_DISABLED;
- int ret;
- lockdep_assert_held(&mvm->mutex);
-
- /* Map Aux queue to fifo - needs to happen before adding Aux station */
- if (!iwl_mvm_is_dqa_supported(mvm))
- iwl_mvm_enable_ac_txq(mvm, mvm->aux_queue, mvm->aux_queue,
- IWL_MVM_TX_FIFO_MCAST, 0, wdg_timeout);
-
- /* Allocate aux station and assign to it the aux queue */
- ret = iwl_mvm_allocate_int_sta(mvm, &mvm->aux_sta, BIT(mvm->aux_queue),
- NL80211_IFTYPE_UNSPECIFIED);
- if (ret)
- return ret;
-
- if (iwl_mvm_is_dqa_supported(mvm)) {
+ if (iwl_mvm_has_new_tx_api(mvm)) {
+ int queue = iwl_mvm_tvqm_enable_txq(mvm, mvm->aux_queue,
+ mvm->aux_sta.sta_id,
+ IWL_MAX_TID_COUNT,
+ wdg_timeout);
+ mvm->aux_queue = queue;
+ } else if (iwl_mvm_is_dqa_supported(mvm)) {
struct iwl_trans_txq_scd_cfg cfg = {
.fifo = IWL_MVM_TX_FIFO_MCAST,
.sta_id = mvm->aux_sta.sta_id,
@@ -1732,14 +1822,44 @@
iwl_mvm_enable_txq(mvm, mvm->aux_queue, mvm->aux_queue, 0, &cfg,
wdg_timeout);
+ } else {
+ iwl_mvm_enable_ac_txq(mvm, mvm->aux_queue, mvm->aux_queue,
+ IWL_MVM_TX_FIFO_MCAST, 0, wdg_timeout);
}
+}
+
+int iwl_mvm_add_aux_sta(struct iwl_mvm *mvm)
+{
+ int ret;
+
+ lockdep_assert_held(&mvm->mutex);
+
+ /* Allocate aux station and assign to it the aux queue */
+ ret = iwl_mvm_allocate_int_sta(mvm, &mvm->aux_sta, BIT(mvm->aux_queue),
+ NL80211_IFTYPE_UNSPECIFIED,
+ IWL_STA_AUX_ACTIVITY);
+ if (ret)
+ return ret;
+
+ /* Map Aux queue to fifo - needs to happen before adding Aux station */
+ if (!iwl_mvm_has_new_tx_api(mvm))
+ iwl_mvm_enable_aux_queue(mvm);
ret = iwl_mvm_add_int_sta_common(mvm, &mvm->aux_sta, NULL,
MAC_INDEX_AUX, 0);
-
- if (ret)
+ if (ret) {
iwl_mvm_dealloc_int_sta(mvm, &mvm->aux_sta);
- return ret;
+ return ret;
+ }
+
+ /*
+ * For a000 firmware and on we cannot add queue to a station unknown
+ * to firmware so enable queue here - after the station was added
+ */
+ if (iwl_mvm_has_new_tx_api(mvm))
+ iwl_mvm_enable_aux_queue(mvm);
+
+ return 0;
}
int iwl_mvm_add_snif_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif)
@@ -1790,39 +1910,39 @@
struct iwl_mvm_int_sta *bsta = &mvmvif->bcast_sta;
static const u8 _baddr[] = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF};
const u8 *baddr = _baddr;
+ int queue;
int ret;
+ unsigned int wdg_timeout =
+ iwl_mvm_get_wd_timeout(mvm, vif, false, false);
+ struct iwl_trans_txq_scd_cfg cfg = {
+ .fifo = IWL_MVM_TX_FIFO_VO,
+ .sta_id = mvmvif->bcast_sta.sta_id,
+ .tid = IWL_MAX_TID_COUNT,
+ .aggregate = false,
+ .frame_limit = IWL_FRAME_LIMIT,
+ };
lockdep_assert_held(&mvm->mutex);
- if (iwl_mvm_is_dqa_supported(mvm)) {
- struct iwl_trans_txq_scd_cfg cfg = {
- .fifo = IWL_MVM_TX_FIFO_VO,
- .sta_id = mvmvif->bcast_sta.sta_id,
- .tid = IWL_MAX_TID_COUNT,
- .aggregate = false,
- .frame_limit = IWL_FRAME_LIMIT,
- };
- unsigned int wdg_timeout =
- iwl_mvm_get_wd_timeout(mvm, vif, false, false);
- int queue;
-
+ if (iwl_mvm_is_dqa_supported(mvm) && !iwl_mvm_has_new_tx_api(mvm)) {
if (vif->type == NL80211_IFTYPE_AP ||
vif->type == NL80211_IFTYPE_ADHOC)
- queue = IWL_MVM_DQA_AP_PROBE_RESP_QUEUE;
+ queue = mvm->probe_queue;
else if (vif->type == NL80211_IFTYPE_P2P_DEVICE)
- queue = IWL_MVM_DQA_P2P_DEVICE_QUEUE;
+ queue = mvm->p2p_dev_queue;
else if (WARN(1, "Missing required TXQ for adding bcast STA\n"))
return -EINVAL;
- iwl_mvm_enable_txq(mvm, queue, vif->hw_queue[0], 0, &cfg,
- wdg_timeout);
bsta->tfd_queue_msk |= BIT(queue);
+
+ iwl_mvm_enable_txq(mvm, queue, vif->hw_queue[0], 0,
+ &cfg, wdg_timeout);
}
if (vif->type == NL80211_IFTYPE_ADHOC)
baddr = vif->bss_conf.bssid;
- if (WARN_ON_ONCE(bsta->sta_id == IWL_MVM_STATION_COUNT))
+ if (WARN_ON_ONCE(bsta->sta_id == IWL_MVM_INVALID_STA))
return -ENOSPC;
ret = iwl_mvm_add_int_sta_common(mvm, bsta, baddr,
@@ -1831,27 +1951,21 @@
return ret;
/*
- * In AP vif type, we also need to enable the cab_queue. However, we
- * have to enable it after the ADD_STA command is sent, otherwise the
- * FW will throw an assert once we send the ADD_STA command (it'll
- * detect a mismatch in the tfd_queue_msk, as we can't add the
- * enabled-cab_queue to the mask)
+ * For a000 firmware and on we cannot add queue to a station unknown
+ * to firmware so enable queue here - after the station was added
*/
- if (iwl_mvm_is_dqa_supported(mvm) &&
- (vif->type == NL80211_IFTYPE_AP ||
- vif->type == NL80211_IFTYPE_ADHOC)) {
- struct iwl_trans_txq_scd_cfg cfg = {
- .fifo = IWL_MVM_TX_FIFO_MCAST,
- .sta_id = mvmvif->bcast_sta.sta_id,
- .tid = IWL_MAX_TID_COUNT,
- .aggregate = false,
- .frame_limit = IWL_FRAME_LIMIT,
- };
- unsigned int wdg_timeout =
- iwl_mvm_get_wd_timeout(mvm, vif, false, false);
+ if (iwl_mvm_has_new_tx_api(mvm)) {
+ queue = iwl_mvm_tvqm_enable_txq(mvm, vif->hw_queue[0],
+ bsta->sta_id,
+ IWL_MAX_TID_COUNT,
+ wdg_timeout);
- iwl_mvm_enable_txq(mvm, vif->cab_queue, vif->cab_queue,
- 0, &cfg, wdg_timeout);
+ if (vif->type == NL80211_IFTYPE_AP)
+ mvm->probe_queue = queue;
+ else if (vif->type == NL80211_IFTYPE_P2P_DEVICE)
+ mvm->p2p_dev_queue = queue;
+
+ bsta->tfd_queue_msk |= BIT(queue);
}
return 0;
@@ -1869,24 +1983,18 @@
iwl_mvm_disable_txq(mvm, vif->cab_queue, vif->cab_queue,
IWL_MAX_TID_COUNT, 0);
- if (mvmvif->bcast_sta.tfd_queue_msk &
- BIT(IWL_MVM_DQA_AP_PROBE_RESP_QUEUE)) {
- iwl_mvm_disable_txq(mvm,
- IWL_MVM_DQA_AP_PROBE_RESP_QUEUE,
+ if (mvmvif->bcast_sta.tfd_queue_msk & BIT(mvm->probe_queue)) {
+ iwl_mvm_disable_txq(mvm, mvm->probe_queue,
vif->hw_queue[0], IWL_MAX_TID_COUNT,
0);
- mvmvif->bcast_sta.tfd_queue_msk &=
- ~BIT(IWL_MVM_DQA_AP_PROBE_RESP_QUEUE);
+ mvmvif->bcast_sta.tfd_queue_msk &= ~BIT(mvm->probe_queue);
}
- if (mvmvif->bcast_sta.tfd_queue_msk &
- BIT(IWL_MVM_DQA_P2P_DEVICE_QUEUE)) {
- iwl_mvm_disable_txq(mvm,
- IWL_MVM_DQA_P2P_DEVICE_QUEUE,
+ if (mvmvif->bcast_sta.tfd_queue_msk & BIT(mvm->p2p_dev_queue)) {
+ iwl_mvm_disable_txq(mvm, mvm->p2p_dev_queue,
vif->hw_queue[0], IWL_MAX_TID_COUNT,
0);
- mvmvif->bcast_sta.tfd_queue_msk &=
- ~BIT(IWL_MVM_DQA_P2P_DEVICE_QUEUE);
+ mvmvif->bcast_sta.tfd_queue_msk &= ~BIT(mvm->p2p_dev_queue);
}
}
@@ -1928,7 +2036,8 @@
}
return iwl_mvm_allocate_int_sta(mvm, &mvmvif->bcast_sta, qmask,
- ieee80211_vif_type_p2p(vif));
+ ieee80211_vif_type_p2p(vif),
+ IWL_STA_GENERAL_PURPOSE);
}
/* Allocate a new station entry for the broadcast station to the given vif,
@@ -1982,6 +2091,101 @@
return ret;
}
+/*
+ * Allocate a new station entry for the multicast station to the given vif,
+ * and send it to the FW.
+ * Note that each AP/GO mac should have its own multicast station.
+ *
+ * @mvm: the mvm component
+ * @vif: the interface to which the multicast station is added
+ */
+int iwl_mvm_add_mcast_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif)
+{
+ struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
+ struct iwl_mvm_int_sta *msta = &mvmvif->mcast_sta;
+ static const u8 _maddr[] = {0x03, 0x00, 0x00, 0x00, 0x00, 0x00};
+ const u8 *maddr = _maddr;
+ struct iwl_trans_txq_scd_cfg cfg = {
+ .fifo = IWL_MVM_TX_FIFO_MCAST,
+ .sta_id = msta->sta_id,
+ .tid = IWL_MAX_TID_COUNT,
+ .aggregate = false,
+ .frame_limit = IWL_FRAME_LIMIT,
+ };
+ unsigned int timeout = iwl_mvm_get_wd_timeout(mvm, vif, false, false);
+ int ret;
+
+ lockdep_assert_held(&mvm->mutex);
+
+ if (!iwl_mvm_is_dqa_supported(mvm))
+ return 0;
+
+ if (WARN_ON(vif->type != NL80211_IFTYPE_AP))
+ return -ENOTSUPP;
+
+ /*
+ * While in previous FWs we had to exclude cab queue from TFD queue
+ * mask, now it is needed as any other queue.
+ */
+ if (!iwl_mvm_has_new_tx_api(mvm) &&
+ fw_has_api(&mvm->fw->ucode_capa, IWL_UCODE_TLV_API_STA_TYPE)) {
+ iwl_mvm_enable_txq(mvm, vif->cab_queue, vif->cab_queue, 0,
+ &cfg, timeout);
+ msta->tfd_queue_msk |= BIT(vif->cab_queue);
+ }
+ ret = iwl_mvm_add_int_sta_common(mvm, msta, maddr,
+ mvmvif->id, mvmvif->color);
+ if (ret) {
+ iwl_mvm_dealloc_int_sta(mvm, msta);
+ return ret;
+ }
+
+ /*
+ * Enable cab queue after the ADD_STA command is sent.
+ * This is needed for a000 firmware which won't accept SCD_QUEUE_CFG
+ * command with unknown station id, and for FW that doesn't support
+ * station API since the cab queue is not included in the
+ * tfd_queue_mask.
+ */
+ if (iwl_mvm_has_new_tx_api(mvm)) {
+ int queue = iwl_mvm_tvqm_enable_txq(mvm, vif->cab_queue,
+ msta->sta_id,
+ IWL_MAX_TID_COUNT,
+ timeout);
+ mvmvif->cab_queue = queue;
+ } else if (!fw_has_api(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_API_STA_TYPE)) {
+ iwl_mvm_enable_txq(mvm, vif->cab_queue, vif->cab_queue, 0,
+ &cfg, timeout);
+ }
+
+ return 0;
+}
+
+/*
+ * Send the FW a request to remove the station from it's internal data
+ * structures, and in addition remove it from the local data structure.
+ */
+int iwl_mvm_rm_mcast_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif)
+{
+ struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif);
+ int ret;
+
+ lockdep_assert_held(&mvm->mutex);
+
+ if (!iwl_mvm_is_dqa_supported(mvm))
+ return 0;
+
+ iwl_mvm_disable_txq(mvm, mvmvif->cab_queue, vif->cab_queue,
+ IWL_MAX_TID_COUNT, 0);
+
+ ret = iwl_mvm_rm_sta_common(mvm, mvmvif->mcast_sta.sta_id);
+ if (ret)
+ IWL_WARN(mvm, "Failed sending remove station\n");
+
+ return ret;
+}
+
#define IWL_MAX_RX_BA_SESSIONS 16
static void iwl_mvm_sync_rxq_del_ba(struct iwl_mvm *mvm, u8 baid)
@@ -2059,6 +2263,7 @@
reorder_buf->mvm = mvm;
reorder_buf->queue = i;
reorder_buf->sta_id = sta_id;
+ reorder_buf->valid = false;
for (j = 0; j < reorder_buf->buf_size; j++)
__skb_queue_head_init(&reorder_buf->entries[j]);
}
@@ -2226,7 +2431,9 @@
cmd.mac_id_n_color = cpu_to_le32(mvm_sta->mac_id_n_color);
cmd.sta_id = mvm_sta->sta_id;
cmd.add_modify = STA_MODE_MODIFY;
- cmd.modify_mask = STA_MODIFY_QUEUES | STA_MODIFY_TID_DISABLE_TX;
+ if (!iwl_mvm_has_new_tx_api(mvm))
+ cmd.modify_mask = STA_MODIFY_QUEUES;
+ cmd.modify_mask |= STA_MODIFY_TID_DISABLE_TX;
cmd.tfd_queue_msk = cpu_to_le32(mvm_sta->tfd_queue_msk);
cmd.tid_disable_tx = cpu_to_le16(mvm_sta->tid_disable_agg);
@@ -2310,10 +2517,18 @@
* one and mark it as reserved
* 3. In DQA mode, but no traffic yet on this TID: same treatment as in
* non-DQA mode, since the TXQ hasn't yet been allocated
+ * Don't support case 3 for new TX path as it is not expected to happen
+ * and aggregation will be offloaded soon anyway
*/
txq_id = mvmsta->tid_data[tid].txq_id;
- if (iwl_mvm_is_dqa_supported(mvm) &&
- unlikely(mvm->queue_info[txq_id].status == IWL_MVM_QUEUE_SHARED)) {
+ if (iwl_mvm_has_new_tx_api(mvm)) {
+ if (txq_id == IWL_MVM_INVALID_QUEUE) {
+ ret = -ENXIO;
+ goto release_locks;
+ }
+ } else if (iwl_mvm_is_dqa_supported(mvm) &&
+ unlikely(mvm->queue_info[txq_id].status ==
+ IWL_MVM_QUEUE_SHARED)) {
ret = -ENXIO;
IWL_DEBUG_TX_QUEUES(mvm,
"Can't start tid %d agg on shared queue!\n",
@@ -2409,6 +2624,20 @@
tid_data->amsdu_in_ampdu_allowed = amsdu;
spin_unlock_bh(&mvmsta->lock);
+ if (iwl_mvm_has_new_tx_api(mvm)) {
+ /*
+ * If no queue iwl_mvm_sta_tx_agg_start() would have failed so
+ * no need to check queue's status
+ */
+ if (buf_size < mvmsta->max_agg_bufsize)
+ return -ENOTSUPP;
+
+ ret = iwl_mvm_sta_tx_agg(mvm, sta, tid, queue, true);
+ if (ret)
+ return -EIO;
+ goto out;
+ }
+
cfg.fifo = iwl_mvm_ac_to_tx_fifo[tid_to_mac80211_ac[tid]];
spin_lock_bh(&mvm->queue_info_lock);
@@ -2430,8 +2659,8 @@
* If reconfiguring an existing queue, it first must be
* drained
*/
- ret = iwl_trans_wait_tx_queue_empty(mvm->trans,
- BIT(queue));
+ ret = iwl_trans_wait_tx_queues_empty(mvm->trans,
+ BIT(queue));
if (ret) {
IWL_ERR(mvm,
"Error draining queue before reconfig\n");
@@ -2466,6 +2695,7 @@
mvm->queue_info[queue].status = IWL_MVM_QUEUE_READY;
spin_unlock_bh(&mvm->queue_info_lock);
+out:
/*
* Even though in theory the peer could have different
* aggregation reorder buffer sizes for different sessions,
@@ -2483,6 +2713,27 @@
return iwl_mvm_send_lq_cmd(mvm, &mvmsta->lq_sta.lq, false);
}
+static void iwl_mvm_unreserve_agg_queue(struct iwl_mvm *mvm,
+ struct iwl_mvm_sta *mvmsta,
+ u16 txq_id)
+{
+ if (iwl_mvm_has_new_tx_api(mvm))
+ return;
+
+ spin_lock_bh(&mvm->queue_info_lock);
+ /*
+ * The TXQ is marked as reserved only if no traffic came through yet
+ * This means no traffic has been sent on this TID (agg'd or not), so
+ * we no longer have use for the queue. Since it hasn't even been
+ * allocated through iwl_mvm_enable_txq, so we can just mark it back as
+ * free.
+ */
+ if (mvm->queue_info[txq_id].status == IWL_MVM_QUEUE_RESERVED)
+ mvm->queue_info[txq_id].status = IWL_MVM_QUEUE_FREE;
+
+ spin_unlock_bh(&mvm->queue_info_lock);
+}
+
int iwl_mvm_sta_tx_agg_stop(struct iwl_mvm *mvm, struct ieee80211_vif *vif,
struct ieee80211_sta *sta, u16 tid)
{
@@ -2509,18 +2760,7 @@
mvmsta->agg_tids &= ~BIT(tid);
- spin_lock_bh(&mvm->queue_info_lock);
- /*
- * The TXQ is marked as reserved only if no traffic came through yet
- * This means no traffic has been sent on this TID (agg'd or not), so
- * we no longer have use for the queue. Since it hasn't even been
- * allocated through iwl_mvm_enable_txq, so we can just mark it back as
- * free.
- */
- if (mvm->queue_info[txq_id].status == IWL_MVM_QUEUE_RESERVED)
- mvm->queue_info[txq_id].status = IWL_MVM_QUEUE_FREE;
-
- spin_unlock_bh(&mvm->queue_info_lock);
+ iwl_mvm_unreserve_agg_queue(mvm, mvmsta, txq_id);
switch (tid_data->state) {
case IWL_AGG_ON:
@@ -2600,24 +2840,14 @@
mvmsta->agg_tids &= ~BIT(tid);
spin_unlock_bh(&mvmsta->lock);
- spin_lock_bh(&mvm->queue_info_lock);
- /*
- * The TXQ is marked as reserved only if no traffic came through yet
- * This means no traffic has been sent on this TID (agg'd or not), so
- * we no longer have use for the queue. Since it hasn't even been
- * allocated through iwl_mvm_enable_txq, so we can just mark it back as
- * free.
- */
- if (mvm->queue_info[txq_id].status == IWL_MVM_QUEUE_RESERVED)
- mvm->queue_info[txq_id].status = IWL_MVM_QUEUE_FREE;
- spin_unlock_bh(&mvm->queue_info_lock);
+ iwl_mvm_unreserve_agg_queue(mvm, mvmsta, txq_id);
if (old_state >= IWL_AGG_ON) {
iwl_mvm_drain_sta(mvm, mvmsta, true);
if (iwl_mvm_flush_tx_path(mvm, BIT(txq_id), 0))
IWL_ERR(mvm, "Couldn't flush the AGG queue\n");
- iwl_trans_wait_tx_queue_empty(mvm->trans,
- mvmsta->tfd_queue_msk);
+ iwl_trans_wait_tx_queues_empty(mvm->trans,
+ mvmsta->tfd_queue_msk);
iwl_mvm_drain_sta(mvm, mvmsta, false);
iwl_mvm_sta_tx_agg(mvm, sta, tid, txq_id, false);
@@ -2675,7 +2905,7 @@
* station ID, then use AP's station ID.
*/
if (vif->type == NL80211_IFTYPE_STATION &&
- mvmvif->ap_sta_id != IWL_MVM_STATION_COUNT) {
+ mvmvif->ap_sta_id != IWL_MVM_INVALID_STA) {
u8 sta_id = mvmvif->ap_sta_id;
sta = rcu_dereference_check(mvm->fw_id_to_mac_id[sta_id],
@@ -2697,68 +2927,97 @@
static int iwl_mvm_send_sta_key(struct iwl_mvm *mvm,
struct iwl_mvm_sta *mvm_sta,
- struct ieee80211_key_conf *keyconf, bool mcast,
+ struct ieee80211_key_conf *key, bool mcast,
u32 tkip_iv32, u16 *tkip_p1k, u32 cmd_flags,
u8 key_offset)
{
- struct iwl_mvm_add_sta_key_cmd cmd = {};
+ union {
+ struct iwl_mvm_add_sta_key_cmd_v1 cmd_v1;
+ struct iwl_mvm_add_sta_key_cmd cmd;
+ } u = {};
__le16 key_flags;
int ret;
u32 status;
u16 keyidx;
- int i;
- u8 sta_id = mvm_sta->sta_id;
+ u64 pn = 0;
+ int i, size;
+ bool new_api = fw_has_api(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_API_TKIP_MIC_KEYS);
- keyidx = (keyconf->keyidx << STA_KEY_FLG_KEYID_POS) &
+ keyidx = (key->keyidx << STA_KEY_FLG_KEYID_POS) &
STA_KEY_FLG_KEYID_MSK;
key_flags = cpu_to_le16(keyidx);
key_flags |= cpu_to_le16(STA_KEY_FLG_WEP_KEY_MAP);
- switch (keyconf->cipher) {
+ switch (key->cipher) {
case WLAN_CIPHER_SUITE_TKIP:
key_flags |= cpu_to_le16(STA_KEY_FLG_TKIP);
- cmd.tkip_rx_tsc_byte2 = tkip_iv32;
- for (i = 0; i < 5; i++)
- cmd.tkip_rx_ttak[i] = cpu_to_le16(tkip_p1k[i]);
- memcpy(cmd.key, keyconf->key, keyconf->keylen);
+ if (new_api) {
+ memcpy((void *)&u.cmd.tx_mic_key,
+ &key->key[NL80211_TKIP_DATA_OFFSET_TX_MIC_KEY],
+ IWL_MIC_KEY_SIZE);
+
+ memcpy((void *)&u.cmd.rx_mic_key,
+ &key->key[NL80211_TKIP_DATA_OFFSET_RX_MIC_KEY],
+ IWL_MIC_KEY_SIZE);
+ pn = atomic64_read(&key->tx_pn);
+
+ } else {
+ u.cmd_v1.tkip_rx_tsc_byte2 = tkip_iv32;
+ for (i = 0; i < 5; i++)
+ u.cmd_v1.tkip_rx_ttak[i] =
+ cpu_to_le16(tkip_p1k[i]);
+ }
+ memcpy(u.cmd.common.key, key->key, key->keylen);
break;
case WLAN_CIPHER_SUITE_CCMP:
key_flags |= cpu_to_le16(STA_KEY_FLG_CCM);
- memcpy(cmd.key, keyconf->key, keyconf->keylen);
+ memcpy(u.cmd.common.key, key->key, key->keylen);
+ if (new_api)
+ pn = atomic64_read(&key->tx_pn);
break;
case WLAN_CIPHER_SUITE_WEP104:
key_flags |= cpu_to_le16(STA_KEY_FLG_WEP_13BYTES);
/* fall through */
case WLAN_CIPHER_SUITE_WEP40:
key_flags |= cpu_to_le16(STA_KEY_FLG_WEP);
- memcpy(cmd.key + 3, keyconf->key, keyconf->keylen);
+ memcpy(u.cmd.common.key + 3, key->key, key->keylen);
break;
case WLAN_CIPHER_SUITE_GCMP_256:
key_flags |= cpu_to_le16(STA_KEY_FLG_KEY_32BYTES);
/* fall through */
case WLAN_CIPHER_SUITE_GCMP:
key_flags |= cpu_to_le16(STA_KEY_FLG_GCMP);
- memcpy(cmd.key, keyconf->key, keyconf->keylen);
+ memcpy(u.cmd.common.key, key->key, key->keylen);
+ if (new_api)
+ pn = atomic64_read(&key->tx_pn);
break;
default:
key_flags |= cpu_to_le16(STA_KEY_FLG_EXT);
- memcpy(cmd.key, keyconf->key, keyconf->keylen);
+ memcpy(u.cmd.common.key, key->key, key->keylen);
}
if (mcast)
key_flags |= cpu_to_le16(STA_KEY_MULTICAST);
- cmd.key_offset = key_offset;
- cmd.key_flags = key_flags;
- cmd.sta_id = sta_id;
+ u.cmd.common.key_offset = key_offset;
+ u.cmd.common.key_flags = key_flags;
+ u.cmd.common.sta_id = mvm_sta->sta_id;
+
+ if (new_api) {
+ u.cmd.transmit_seq_cnt = cpu_to_le64(pn);
+ size = sizeof(u.cmd);
+ } else {
+ size = sizeof(u.cmd_v1);
+ }
status = ADD_STA_SUCCESS;
if (cmd_flags & CMD_ASYNC)
- ret = iwl_mvm_send_cmd_pdu(mvm, ADD_STA_KEY, CMD_ASYNC,
- sizeof(cmd), &cmd);
+ ret = iwl_mvm_send_cmd_pdu(mvm, ADD_STA_KEY, CMD_ASYNC, size,
+ &u.cmd);
else
- ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA_KEY, sizeof(cmd),
- &cmd, &status);
+ ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA_KEY, size,
+ &u.cmd, &status);
switch (status) {
case ADD_STA_SUCCESS:
@@ -2858,7 +3117,7 @@
return sta->addr;
if (vif->type == NL80211_IFTYPE_STATION &&
- mvmvif->ap_sta_id != IWL_MVM_STATION_COUNT) {
+ mvmvif->ap_sta_id != IWL_MVM_INVALID_STA) {
u8 sta_id = mvmvif->ap_sta_id;
sta = rcu_dereference_protected(mvm->fw_id_to_mac_id[sta_id],
lockdep_is_held(&mvm->mutex));
@@ -2911,9 +3170,14 @@
struct ieee80211_key_conf *keyconf,
bool mcast)
{
- struct iwl_mvm_add_sta_key_cmd cmd = {};
+ union {
+ struct iwl_mvm_add_sta_key_cmd_v1 cmd_v1;
+ struct iwl_mvm_add_sta_key_cmd cmd;
+ } u = {};
+ bool new_api = fw_has_api(&mvm->fw->ucode_capa,
+ IWL_UCODE_TLV_API_TKIP_MIC_KEYS);
__le16 key_flags;
- int ret;
+ int ret, size;
u32 status;
key_flags = cpu_to_le16((keyconf->keyidx << STA_KEY_FLG_KEYID_POS) &
@@ -2924,13 +3188,19 @@
if (mcast)
key_flags |= cpu_to_le16(STA_KEY_MULTICAST);
- cmd.key_flags = key_flags;
- cmd.key_offset = keyconf->hw_key_idx;
- cmd.sta_id = sta_id;
+ /*
+ * The fields assigned here are in the same location at the start
+ * of the command, so we can do this union trick.
+ */
+ u.cmd.common.key_flags = key_flags;
+ u.cmd.common.key_offset = keyconf->hw_key_idx;
+ u.cmd.common.sta_id = sta_id;
+
+ size = new_api ? sizeof(u.cmd) : sizeof(u.cmd_v1);
status = ADD_STA_SUCCESS;
- ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA_KEY, sizeof(cmd),
- &cmd, &status);
+ ret = iwl_mvm_send_cmd_pdu_status(mvm, ADD_STA_KEY, size, &u.cmd,
+ &status);
switch (status) {
case ADD_STA_SUCCESS:
@@ -3044,7 +3314,7 @@
{
bool mcast = !(keyconf->flags & IEEE80211_KEY_FLAG_PAIRWISE);
struct iwl_mvm_sta *mvm_sta;
- u8 sta_id = IWL_MVM_STATION_COUNT;
+ u8 sta_id = IWL_MVM_INVALID_STA;
int ret, i;
lockdep_assert_held(&mvm->mutex);
@@ -3207,13 +3477,13 @@
/* Note: this is ignored by firmware not supporting GO uAPSD */
if (more_data)
- cmd.sleep_state_flags |= cpu_to_le16(STA_SLEEP_STATE_MOREDATA);
+ cmd.sleep_state_flags |= STA_SLEEP_STATE_MOREDATA;
if (reason == IEEE80211_FRAME_RELEASE_PSPOLL) {
mvmsta->next_status_eosp = true;
- cmd.sleep_state_flags |= cpu_to_le16(STA_SLEEP_STATE_PS_POLL);
+ cmd.sleep_state_flags |= STA_SLEEP_STATE_PS_POLL;
} else {
- cmd.sleep_state_flags |= cpu_to_le16(STA_SLEEP_STATE_UAPSD);
+ cmd.sleep_state_flags |= STA_SLEEP_STATE_UAPSD;
}
/* block the Tx queues until the FW updated the sleep Tx count */
@@ -3290,6 +3560,27 @@
spin_unlock_bh(&mvm_sta->lock);
}
+static void iwl_mvm_int_sta_modify_disable_tx(struct iwl_mvm *mvm,
+ struct iwl_mvm_vif *mvmvif,
+ struct iwl_mvm_int_sta *sta,
+ bool disable)
+{
+ u32 id = FW_CMD_ID_AND_COLOR(mvmvif->id, mvmvif->color);
+ struct iwl_mvm_add_sta_cmd cmd = {
+ .add_modify = STA_MODE_MODIFY,
+ .sta_id = sta->sta_id,
+ .station_flags = disable ? cpu_to_le32(STA_FLG_DISABLE_TX) : 0,
+ .station_flags_msk = cpu_to_le32(STA_FLG_DISABLE_TX),
+ .mac_id_n_color = cpu_to_le32(id),
+ };
+ int ret;
+
+ ret = iwl_mvm_send_cmd_pdu(mvm, ADD_STA, 0,
+ iwl_mvm_add_sta_cmd_size(mvm), &cmd);
+ if (ret)
+ IWL_ERR(mvm, "Failed to send ADD_STA command (%d)\n", ret);
+}
+
void iwl_mvm_modify_all_sta_disable_tx(struct iwl_mvm *mvm,
struct iwl_mvm_vif *mvmvif,
bool disable)
@@ -3301,7 +3592,7 @@
lockdep_assert_held(&mvm->mutex);
/* Block/unblock all the stations of the given mvmvif */
- for (i = 0; i < IWL_MVM_STATION_COUNT; i++) {
+ for (i = 0; i < ARRAY_SIZE(mvm->fw_id_to_mac_id); i++) {
sta = rcu_dereference_protected(mvm->fw_id_to_mac_id[i],
lockdep_is_held(&mvm->mutex));
if (IS_ERR_OR_NULL(sta))
@@ -3314,6 +3605,22 @@
iwl_mvm_sta_modify_disable_tx_ap(mvm, sta, disable);
}
+
+ if (!fw_has_api(&mvm->fw->ucode_capa, IWL_UCODE_TLV_API_STA_TYPE))
+ return;
+
+ /* Need to block/unblock also multicast station */
+ if (mvmvif->mcast_sta.sta_id != IWL_MVM_INVALID_STA)
+ iwl_mvm_int_sta_modify_disable_tx(mvm, mvmvif,
+ &mvmvif->mcast_sta, disable);
+
+ /*
+ * Only unblock the broadcast station (FW blocks it for immediate
+ * quiet, not the driver)
+ */
+ if (!disable && mvmvif->bcast_sta.sta_id != IWL_MVM_INVALID_STA)
+ iwl_mvm_int_sta_modify_disable_tx(mvm, mvmvif,
+ &mvmvif->bcast_sta, disable);
}
void iwl_mvm_csa_client_absent(struct iwl_mvm *mvm, struct ieee80211_vif *vif)
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/sta.h b/drivers/net/wireless/intel/iwlwifi/mvm/sta.h
index 1927ce6..2716cb5 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/sta.h
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/sta.h
@@ -380,6 +380,7 @@
* @tid_disable_agg: bitmap: if bit(tid) is set, the fw won't send ampdus for
* tid.
* @max_agg_bufsize: the maximal size of the AGG buffer for this station
+ * @sta_type: station type
* @bt_reduced_txpower: is reduced tx power enabled for this station
* @next_status_eosp: the next reclaimed packet is a PS-Poll response and
* we need to signal the EOSP
@@ -416,6 +417,7 @@
u32 mac_id_n_color;
u16 tid_disable_agg;
u8 max_agg_bufsize;
+ enum iwl_sta_type sta_type;
bool bt_reduced_txpower;
bool next_status_eosp;
spinlock_t lock;
@@ -453,10 +455,12 @@
* struct iwl_mvm_int_sta - representation of an internal station (auxiliary or
* broadcast)
* @sta_id: the index of the station in the fw (will be replaced by id_n_color)
+ * @type: station type
* @tfd_queue_msk: the tfd queues used by the station
*/
struct iwl_mvm_int_sta {
u32 sta_id;
+ enum iwl_sta_type type;
u32 tfd_queue_msk;
};
@@ -532,10 +536,14 @@
int iwl_mvm_add_bcast_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif);
int iwl_mvm_send_rm_bcast_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif);
int iwl_mvm_rm_bcast_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif);
+int iwl_mvm_add_mcast_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif);
+int iwl_mvm_rm_mcast_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif);
int iwl_mvm_allocate_int_sta(struct iwl_mvm *mvm,
struct iwl_mvm_int_sta *sta,
- u32 qmask, enum nl80211_iftype iftype);
+ u32 qmask, enum nl80211_iftype iftype,
+ enum iwl_sta_type type);
void iwl_mvm_dealloc_bcast_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif);
+void iwl_mvm_dealloc_int_sta(struct iwl_mvm *mvm, struct iwl_mvm_int_sta *sta);
int iwl_mvm_add_snif_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif);
int iwl_mvm_rm_snif_sta(struct iwl_mvm *mvm, struct ieee80211_vif *vif);
void iwl_mvm_dealloc_snif_sta(struct iwl_mvm *mvm);
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/tdls.c b/drivers/net/wireless/intel/iwlwifi/mvm/tdls.c
index 9f160fc..df7cd87 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/tdls.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/tdls.c
@@ -6,6 +6,7 @@
* GPL LICENSE SUMMARY
*
* Copyright(c) 2014 Intel Mobile Communications GmbH
+ * Copyright(c) 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -31,6 +32,7 @@
* BSD LICENSE
*
* Copyright(c) 2014 Intel Mobile Communications GmbH
+ * Copyright(c) 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -78,7 +80,7 @@
lockdep_assert_held(&mvm->mutex);
- for (i = 0; i < IWL_MVM_STATION_COUNT; i++) {
+ for (i = 0; i < ARRAY_SIZE(mvm->fw_id_to_mac_id); i++) {
sta = rcu_dereference_protected(mvm->fw_id_to_mac_id[i],
lockdep_is_held(&mvm->mutex));
if (!sta || IS_ERR(sta) || !sta->tdls)
@@ -101,7 +103,7 @@
lockdep_assert_held(&mvm->mutex);
- for (i = 0; i < IWL_MVM_STATION_COUNT; i++) {
+ for (i = 0; i < ARRAY_SIZE(mvm->fw_id_to_mac_id); i++) {
sta = rcu_dereference_protected(mvm->fw_id_to_mac_id[i],
lockdep_is_held(&mvm->mutex));
if (!sta || IS_ERR(sta) || !sta->tdls)
@@ -145,7 +147,7 @@
/* populate TDLS peer data */
cnt = 0;
- for (i = 0; i < IWL_MVM_STATION_COUNT; i++) {
+ for (i = 0; i < ARRAY_SIZE(mvm->fw_id_to_mac_id); i++) {
sta = rcu_dereference_protected(mvm->fw_id_to_mac_id[i],
lockdep_is_held(&mvm->mutex));
if (IS_ERR_OR_NULL(sta) || !sta->tdls)
@@ -251,7 +253,7 @@
iwl_read_prph(mvm->trans, DEVICE_SYSTEM_TIME_REG);
if (state == IWL_MVM_TDLS_SW_IDLE)
- mvm->tdls_cs.cur_sta_id = IWL_MVM_STATION_COUNT;
+ mvm->tdls_cs.cur_sta_id = IWL_MVM_INVALID_STA;
}
void iwl_mvm_rx_tdls_notif(struct iwl_mvm *mvm, struct iwl_rx_cmd_buffer *rxb)
@@ -305,7 +307,7 @@
/* get the existing peer if it's there */
if (mvm->tdls_cs.state != IWL_MVM_TDLS_SW_IDLE &&
- mvm->tdls_cs.cur_sta_id != IWL_MVM_STATION_COUNT) {
+ mvm->tdls_cs.cur_sta_id != IWL_MVM_INVALID_STA) {
struct ieee80211_sta *sta = rcu_dereference_protected(
mvm->fw_id_to_mac_id[mvm->tdls_cs.cur_sta_id],
lockdep_is_held(&mvm->mutex));
@@ -523,7 +525,7 @@
iwl_mvm_tdls_update_cs_state(mvm, IWL_MVM_TDLS_SW_IDLE);
/* station might be gone, in that case do nothing */
- if (mvm->tdls_cs.peer.sta_id == IWL_MVM_STATION_COUNT)
+ if (mvm->tdls_cs.peer.sta_id == IWL_MVM_INVALID_STA)
goto out;
sta = rcu_dereference_protected(
@@ -573,7 +575,7 @@
sta->addr, chandef->chan->center_freq, chandef->width);
/* we only support a single peer for channel switching */
- if (mvm->tdls_cs.peer.sta_id != IWL_MVM_STATION_COUNT) {
+ if (mvm->tdls_cs.peer.sta_id != IWL_MVM_INVALID_STA) {
IWL_DEBUG_TDLS(mvm,
"Existing peer. Can't start switch with %pM\n",
sta->addr);
@@ -633,7 +635,7 @@
IWL_DEBUG_TDLS(mvm, "TDLS cancel channel switch with %pM\n", sta->addr);
/* we only support a single peer for channel switching */
- if (mvm->tdls_cs.peer.sta_id == IWL_MVM_STATION_COUNT) {
+ if (mvm->tdls_cs.peer.sta_id == IWL_MVM_INVALID_STA) {
IWL_DEBUG_TDLS(mvm, "No ch switch peer - %pM\n", sta->addr);
goto out;
}
@@ -654,7 +656,7 @@
mvm->tdls_cs.state != IWL_MVM_TDLS_SW_IDLE)
wait_for_phy = true;
- mvm->tdls_cs.peer.sta_id = IWL_MVM_STATION_COUNT;
+ mvm->tdls_cs.peer.sta_id = IWL_MVM_INVALID_STA;
dev_kfree_skb(mvm->tdls_cs.peer.skb);
mvm->tdls_cs.peer.skb = NULL;
@@ -697,7 +699,7 @@
if (params->action_code == WLAN_TDLS_CHANNEL_SWITCH_RESPONSE &&
params->status != 0 &&
mvm->tdls_cs.state == IWL_MVM_TDLS_SW_REQ_SENT &&
- mvm->tdls_cs.cur_sta_id != IWL_MVM_STATION_COUNT) {
+ mvm->tdls_cs.cur_sta_id != IWL_MVM_INVALID_STA) {
struct ieee80211_sta *cur_sta;
/* make sure it's the same peer */
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/tof.c b/drivers/net/wireless/intel/iwlwifi/mvm/tof.c
index a1947d6..16ce8a5 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/tof.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/tof.c
@@ -80,7 +80,7 @@
if (IWL_MVM_TOF_IS_RESPONDER) {
tof_data->responder_cfg.sub_grp_cmd_id =
cpu_to_le32(TOF_RESPONDER_CONFIG_CMD);
- tof_data->responder_cfg.sta_id = IWL_MVM_STATION_COUNT;
+ tof_data->responder_cfg.sta_id = IWL_MVM_INVALID_STA;
}
#endif
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/tt.c b/drivers/net/wireless/intel/iwlwifi/mvm/tt.c
index bec7d9c..f9cbd19 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/tt.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/tt.c
@@ -356,7 +356,7 @@
struct iwl_mvm_sta *mvmsta;
int i, err;
- for (i = 0; i < IWL_MVM_STATION_COUNT; i++) {
+ for (i = 0; i < ARRAY_SIZE(mvm->fw_id_to_mac_id); i++) {
mvmsta = iwl_mvm_sta_from_staid_protected(mvm, i);
if (!mvmsta)
continue;
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/tx.c b/drivers/net/wireless/intel/iwlwifi/mvm/tx.c
index 1ba0a6f..bcaceb64 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/tx.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/tx.c
@@ -475,6 +475,39 @@
memset(dev_cmd, 0, sizeof(*dev_cmd));
dev_cmd->hdr.cmd = TX_CMD;
+
+ if (iwl_mvm_has_new_tx_api(mvm)) {
+ struct iwl_tx_cmd_gen2 *cmd = (void *)dev_cmd->payload;
+ u16 offload_assist = iwl_mvm_tx_csum(mvm, skb, hdr, info);
+
+ /* padding is inserted later in transport */
+ /* FIXME - check for AMSDU may need to be removed */
+ if (ieee80211_hdrlen(hdr->frame_control) % 4 &&
+ !(offload_assist & BIT(TX_CMD_OFFLD_AMSDU)))
+ offload_assist |= BIT(TX_CMD_OFFLD_PAD);
+
+ cmd->offload_assist |= cpu_to_le16(offload_assist);
+
+ /* Total # bytes to be transmitted */
+ cmd->len = cpu_to_le16((u16)skb->len);
+
+ /* Copy MAC header from skb into command buffer */
+ memcpy(cmd->hdr, hdr, hdrlen);
+
+ if (!info->control.hw_key)
+ cmd->flags |= cpu_to_le32(IWL_TX_FLAGS_ENCRYPT_DIS);
+
+ /* For data packets rate info comes from the fw */
+ if (ieee80211_is_data(hdr->frame_control) && sta)
+ goto out;
+
+ cmd->flags |= cpu_to_le32(IWL_TX_FLAGS_CMD_RATE);
+ cmd->rate_n_flags =
+ cpu_to_le32(iwl_mvm_get_tx_rate(mvm, info, sta));
+
+ goto out;
+ }
+
tx_cmd = (struct iwl_tx_cmd *)dev_cmd->payload;
if (info->control.hw_key)
@@ -484,6 +517,10 @@
iwl_mvm_set_tx_cmd_rate(mvm, tx_cmd, info, sta, hdr->frame_control);
+ /* Copy MAC header from skb into command buffer */
+ memcpy(tx_cmd->hdr, hdr, hdrlen);
+
+out:
return dev_cmd;
}
@@ -514,21 +551,21 @@
*/
if (ieee80211_is_probe_resp(fc) || ieee80211_is_auth(fc) ||
ieee80211_is_deauth(fc))
- return IWL_MVM_DQA_AP_PROBE_RESP_QUEUE;
+ return mvm->probe_queue;
if (info->hw_queue == info->control.vif->cab_queue)
return info->hw_queue;
WARN_ONCE(info->control.vif->type != NL80211_IFTYPE_ADHOC,
"fc=0x%02x", le16_to_cpu(fc));
- return IWL_MVM_DQA_AP_PROBE_RESP_QUEUE;
+ return mvm->probe_queue;
case NL80211_IFTYPE_P2P_DEVICE:
if (ieee80211_is_mgmt(fc))
- return IWL_MVM_DQA_P2P_DEVICE_QUEUE;
+ return mvm->p2p_dev_queue;
if (info->hw_queue == info->control.vif->cab_queue)
return info->hw_queue;
WARN_ON_ONCE(1);
- return IWL_MVM_DQA_P2P_DEVICE_QUEUE;
+ return mvm->p2p_dev_queue;
default:
WARN_ONCE(1, "Not a ctrl vif, no available queue\n");
return -1;
@@ -541,7 +578,6 @@
struct ieee80211_tx_info *skb_info = IEEE80211_SKB_CB(skb);
struct ieee80211_tx_info info;
struct iwl_device_cmd *dev_cmd;
- struct iwl_tx_cmd *tx_cmd;
u8 sta_id;
int hdrlen = ieee80211_hdrlen(hdr->frame_control);
int queue;
@@ -594,11 +630,13 @@
if (queue < 0)
return -1;
+ if (queue == info.control.vif->cab_queue)
+ queue = mvmvif->cab_queue;
} else if (info.control.vif->type == NL80211_IFTYPE_STATION &&
is_multicast_ether_addr(hdr->addr1)) {
u8 ap_sta_id = ACCESS_ONCE(mvmvif->ap_sta_id);
- if (ap_sta_id != IWL_MVM_STATION_COUNT)
+ if (ap_sta_id != IWL_MVM_INVALID_STA)
sta_id = ap_sta_id;
} else if (iwl_mvm_is_dqa_supported(mvm) &&
info.control.vif->type == NL80211_IFTYPE_STATION &&
@@ -616,11 +654,6 @@
/* From now on, we cannot access info->control */
iwl_mvm_skb_prepare_status(skb, dev_cmd);
- tx_cmd = (struct iwl_tx_cmd *)dev_cmd->payload;
-
- /* Copy MAC header from skb into command buffer */
- memcpy(tx_cmd->hdr, hdr, hdrlen);
-
if (iwl_trans_tx(mvm->trans, skb, dev_cmd, queue)) {
iwl_trans_free_tx_cmd(mvm->trans, dev_cmd);
return -1;
@@ -713,7 +746,7 @@
* fifo to be able to send bursts.
*/
max_amsdu_len = min_t(unsigned int, max_amsdu_len,
- mvm->shared_mem_cfg.txfifo_size[txf] - 256);
+ mvm->smem_cfg.lmac[0].txfifo_size[txf] - 256);
if (unlikely(dbg_max_amsdu_len))
max_amsdu_len = min_t(unsigned int, max_amsdu_len,
@@ -862,6 +895,9 @@
unsigned long now = jiffies;
int tid;
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return false;
+
for_each_set_bit(tid, &queue_tid_bitmap, IWL_MAX_TID_COUNT + 1) {
if (time_before(mvm->queue_info[txq_id].last_frame_time[tid] +
IWL_MVM_DQA_QUEUE_TIMEOUT, now))
@@ -881,11 +917,10 @@
struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data;
struct iwl_mvm_sta *mvmsta;
struct iwl_device_cmd *dev_cmd;
- struct iwl_tx_cmd *tx_cmd;
__le16 fc;
u16 seq_number = 0;
u8 tid = IWL_MAX_TID_COUNT;
- u8 txq_id = info->hw_queue;
+ u16 txq_id = info->hw_queue;
bool is_ampdu = false;
int hdrlen;
@@ -896,7 +931,7 @@
if (WARN_ON_ONCE(!mvmsta))
return -1;
- if (WARN_ON_ONCE(mvmsta->sta_id == IWL_MVM_STATION_COUNT))
+ if (WARN_ON_ONCE(mvmsta->sta_id == IWL_MVM_INVALID_STA))
return -1;
dev_cmd = iwl_mvm_set_tx_params(mvm, skb, info, hdrlen,
@@ -904,8 +939,6 @@
if (!dev_cmd)
goto drop;
- tx_cmd = (struct iwl_tx_cmd *)dev_cmd->payload;
-
/*
* we handle that entirely ourselves -- for uAPSD the firmware
* will always send a notification, and for PS-Poll responses
@@ -926,18 +959,27 @@
if (WARN_ON_ONCE(tid >= IWL_MAX_TID_COUNT))
goto drop_unlock_sta;
- seq_number = mvmsta->tid_data[tid].seq_number;
- seq_number &= IEEE80211_SCTL_SEQ;
- hdr->seq_ctrl &= cpu_to_le16(IEEE80211_SCTL_FRAG);
- hdr->seq_ctrl |= cpu_to_le16(seq_number);
is_ampdu = info->flags & IEEE80211_TX_CTL_AMPDU;
if (WARN_ON_ONCE(is_ampdu &&
mvmsta->tid_data[tid].state != IWL_AGG_ON))
goto drop_unlock_sta;
+
+ seq_number = mvmsta->tid_data[tid].seq_number;
+ seq_number &= IEEE80211_SCTL_SEQ;
+
+ if (!iwl_mvm_has_new_tx_api(mvm)) {
+ struct iwl_tx_cmd *tx_cmd = (void *)dev_cmd->payload;
+
+ hdr->seq_ctrl &= cpu_to_le16(IEEE80211_SCTL_FRAG);
+ hdr->seq_ctrl |= cpu_to_le16(seq_number);
+ /* update the tx_cmd hdr as it was already copied */
+ tx_cmd->hdr->seq_ctrl = hdr->seq_ctrl;
+ }
}
if (iwl_mvm_is_dqa_supported(mvm) || is_ampdu)
txq_id = mvmsta->tid_data[tid].txq_id;
+
if (sta->tdls && !iwl_mvm_is_dqa_supported(mvm)) {
/* default to TID 0 for non-QoS packets */
u8 tdls_tid = tid == IWL_MAX_TID_COUNT ? 0 : tid;
@@ -945,17 +987,14 @@
txq_id = mvmsta->hw_queue[tid_to_mac80211_ac[tdls_tid]];
}
- /* Copy MAC header from skb into command buffer */
- memcpy(tx_cmd->hdr, hdr, hdrlen);
-
WARN_ON_ONCE(info->flags & IEEE80211_TX_CTL_SEND_AFTER_DTIM);
/* Check if TXQ needs to be allocated or re-activated */
- if (unlikely(txq_id == IEEE80211_INVAL_HW_QUEUE ||
+ if (unlikely(txq_id == IWL_MVM_INVALID_QUEUE ||
!mvmsta->tid_data[tid].is_tid_active) &&
iwl_mvm_is_dqa_supported(mvm)) {
/* If TXQ needs to be allocated... */
- if (txq_id == IEEE80211_INVAL_HW_QUEUE) {
+ if (txq_id == IWL_MVM_INVALID_QUEUE) {
iwl_mvm_tx_add_stream(mvm, mvmsta, tid, skb);
/*
@@ -967,6 +1006,9 @@
return 0;
}
+ /* queue should always be active in new TX path */
+ WARN_ON(iwl_mvm_has_new_tx_api(mvm));
+
/* If we are here - TXQ exists and needs to be re-activated */
spin_lock(&mvm->queue_info_lock);
mvm->queue_info[txq_id].status = IWL_MVM_QUEUE_READY;
@@ -977,7 +1019,7 @@
txq_id);
}
- if (iwl_mvm_is_dqa_supported(mvm)) {
+ if (iwl_mvm_is_dqa_supported(mvm) && !iwl_mvm_has_new_tx_api(mvm)) {
/* Keep track of the time of the last frame for this RA/TID */
mvm->queue_info[txq_id].last_frame_time[tid] = jiffies;
@@ -1036,7 +1078,7 @@
if (WARN_ON_ONCE(!mvmsta))
return -1;
- if (WARN_ON_ONCE(mvmsta->sta_id == IWL_MVM_STATION_COUNT))
+ if (WARN_ON_ONCE(mvmsta->sta_id == IWL_MVM_INVALID_STA))
return -1;
memcpy(&info, skb->cb, sizeof(info));
@@ -1245,6 +1287,26 @@
}
}
+/**
+ * iwl_mvm_get_scd_ssn - returns the SSN of the SCD
+ * @tx_resp: the Tx response from the fw (agg or non-agg)
+ *
+ * When the fw sends an AMPDU, it fetches the MPDUs one after the other. Since
+ * it can't know that everything will go well until the end of the AMPDU, it
+ * can't know in advance the number of MPDUs that will be sent in the current
+ * batch. This is why it writes the agg Tx response while it fetches the MPDUs.
+ * Hence, it can't know in advance what the SSN of the SCD will be at the end
+ * of the batch. This is why the SSN of the SCD is written at the end of the
+ * whole struct at a variable offset. This function knows how to cope with the
+ * variable offset and returns the SSN of the SCD.
+ */
+static inline u32 iwl_mvm_get_scd_ssn(struct iwl_mvm *mvm,
+ struct iwl_mvm_tx_resp *tx_resp)
+{
+ return le32_to_cpup((__le32 *)iwl_mvm_get_agg_status(mvm, tx_resp) +
+ tx_resp->frame_count) & 0xfff;
+}
+
static void iwl_mvm_rx_tx_cmd_single(struct iwl_mvm *mvm,
struct iwl_rx_packet *pkt)
{
@@ -1254,8 +1316,10 @@
struct iwl_mvm_tx_resp *tx_resp = (void *)pkt->data;
int sta_id = IWL_MVM_TX_RES_GET_RA(tx_resp->ra_tid);
int tid = IWL_MVM_TX_RES_GET_TID(tx_resp->ra_tid);
- u32 status = le16_to_cpu(tx_resp->status.status);
- u16 ssn = iwl_mvm_get_scd_ssn(tx_resp);
+ struct agg_tx_status *agg_status =
+ iwl_mvm_get_agg_status(mvm, tx_resp);
+ u32 status = le16_to_cpu(agg_status->status);
+ u16 ssn = iwl_mvm_get_scd_ssn(mvm, tx_resp);
struct iwl_mvm_sta *mvmsta;
struct sk_buff_head skbs;
u8 skb_freed = 0;
@@ -1264,6 +1328,9 @@
__skb_queue_head_init(&skbs);
+ if (iwl_mvm_has_new_tx_api(mvm))
+ txq_id = le16_to_cpu(tx_resp->v6.tx_queue);
+
seq_ctl = le16_to_cpu(tx_resp->seq_ctl);
/* we can free until ssn % q.n_bd not inclusive */
@@ -1388,7 +1455,7 @@
if (!IS_ERR(sta)) {
mvmsta = iwl_mvm_sta_from_mac80211(sta);
- if (tid != IWL_TID_NON_QOS) {
+ if (tid != IWL_TID_NON_QOS && tid != IWL_MGMT_TID) {
struct iwl_mvm_tid_data *tid_data =
&mvmsta->tid_data[tid];
bool send_eosp_ndp = false;
@@ -1520,7 +1587,8 @@
struct iwl_rx_packet *pkt)
{
struct iwl_mvm_tx_resp *tx_resp = (void *)pkt->data;
- struct agg_tx_status *frame_status = &tx_resp->status;
+ struct agg_tx_status *frame_status =
+ iwl_mvm_get_agg_status(mvm, tx_resp);
int i;
for (i = 0; i < tx_resp->frame_count; i++) {
@@ -1722,6 +1790,9 @@
ba_info.status.status_driver_data[0] =
(void *)(uintptr_t)ba_res->reduced_txp;
+ if (!le16_to_cpu(ba_res->tfd_cnt))
+ goto out;
+
/*
* TODO:
* When supporting multi TID aggregations - we need to move
@@ -1730,12 +1801,16 @@
* This will go together with SN and AddBA offload and cannot
* be handled properly for now.
*/
- WARN_ON(le16_to_cpu(ba_res->tfd_cnt) != 1);
- iwl_mvm_tx_reclaim(mvm, sta_id, ba_res->ra_tid[0].tid,
- (int)ba_res->tfd[0].q_num,
+ WARN_ON(le16_to_cpu(ba_res->ra_tid_cnt) != 1);
+ tid = ba_res->ra_tid[0].tid;
+ if (tid == IWL_MGMT_TID)
+ tid = IWL_MAX_TID_COUNT;
+ iwl_mvm_tx_reclaim(mvm, sta_id, tid,
+ (int)(le16_to_cpu(ba_res->tfd[0].q_num)),
le16_to_cpu(ba_res->tfd[0].tfd_index),
&ba_info, le32_to_cpu(ba_res->tx_rate));
+out:
IWL_DEBUG_TX_REPLY(mvm,
"BA_NOTIFICATION Received from sta_id = %d, flags %x, sent:%d, acked:%d\n",
sta_id, le32_to_cpu(ba_res->flags),
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/utils.c b/drivers/net/wireless/intel/iwlwifi/mvm/utils.c
index dedea96..8f4f176 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/utils.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/utils.c
@@ -7,7 +7,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
- * Copyright (C) 2015 Intel Deutschland GmbH
+ * Copyright (C) 2015 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -34,6 +34,7 @@
*
* Copyright(c) 2012 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2014 Intel Mobile Communications GmbH
+ * Copyright (C) 2015 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -591,6 +592,10 @@
lockdep_assert_held(&mvm->queue_info_lock);
+ /* This should not be hit with new TX path */
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return -ENOSPC;
+
/* Start by looking for a free queue */
for (i = minq; i <= maxq; i++)
if (mvm->queue_info[i].hw_queue_refcount == 0 &&
@@ -627,6 +632,9 @@
};
int ret;
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return -EINVAL;
+
spin_lock_bh(&mvm->queue_info_lock);
if (WARN(mvm->queue_info[queue].hw_queue_refcount == 0,
"Trying to reconfig unallocated queue %d\n", queue)) {
@@ -644,50 +652,94 @@
return ret;
}
-void iwl_mvm_enable_txq(struct iwl_mvm *mvm, int queue, int mac80211_queue,
- u16 ssn, const struct iwl_trans_txq_scd_cfg *cfg,
- unsigned int wdg_timeout)
+static bool iwl_mvm_update_txq_mapping(struct iwl_mvm *mvm, int queue,
+ int mac80211_queue, u8 sta_id, u8 tid)
{
bool enable_queue = true;
spin_lock_bh(&mvm->queue_info_lock);
/* Make sure this TID isn't already enabled */
- if (mvm->queue_info[queue].tid_bitmap & BIT(cfg->tid)) {
+ if (mvm->queue_info[queue].tid_bitmap & BIT(tid)) {
spin_unlock_bh(&mvm->queue_info_lock);
IWL_ERR(mvm, "Trying to enable TXQ %d with existing TID %d\n",
- queue, cfg->tid);
- return;
+ queue, tid);
+ return false;
}
/* Update mappings and refcounts */
if (mvm->queue_info[queue].hw_queue_refcount > 0)
enable_queue = false;
- mvm->queue_info[queue].hw_queue_to_mac80211 |= BIT(mac80211_queue);
+ mvm->hw_queue_to_mac80211[queue] |= BIT(mac80211_queue);
+
mvm->queue_info[queue].hw_queue_refcount++;
- mvm->queue_info[queue].tid_bitmap |= BIT(cfg->tid);
- mvm->queue_info[queue].ra_sta_id = cfg->sta_id;
+ mvm->queue_info[queue].tid_bitmap |= BIT(tid);
+ mvm->queue_info[queue].ra_sta_id = sta_id;
if (enable_queue) {
- if (cfg->tid != IWL_MAX_TID_COUNT)
+ if (tid != IWL_MAX_TID_COUNT)
mvm->queue_info[queue].mac80211_ac =
- tid_to_mac80211_ac[cfg->tid];
+ tid_to_mac80211_ac[tid];
else
mvm->queue_info[queue].mac80211_ac = IEEE80211_AC_VO;
- mvm->queue_info[queue].txq_tid = cfg->tid;
+ mvm->queue_info[queue].txq_tid = tid;
}
IWL_DEBUG_TX_QUEUES(mvm,
"Enabling TXQ #%d refcount=%d (mac80211 map:0x%x)\n",
queue, mvm->queue_info[queue].hw_queue_refcount,
- mvm->queue_info[queue].hw_queue_to_mac80211);
+ mvm->hw_queue_to_mac80211[queue]);
spin_unlock_bh(&mvm->queue_info_lock);
+ return enable_queue;
+}
+
+int iwl_mvm_tvqm_enable_txq(struct iwl_mvm *mvm, int mac80211_queue,
+ u8 sta_id, u8 tid, unsigned int timeout)
+{
+ struct iwl_tx_queue_cfg_cmd cmd = {
+ .flags = cpu_to_le16(TX_QUEUE_CFG_ENABLE_QUEUE),
+ .sta_id = sta_id,
+ .tid = tid,
+ };
+ int queue;
+
+ if (cmd.tid == IWL_MAX_TID_COUNT)
+ cmd.tid = IWL_MGMT_TID;
+ queue = iwl_trans_txq_alloc(mvm->trans, (void *)&cmd,
+ SCD_QUEUE_CFG, timeout);
+
+ if (queue < 0) {
+ IWL_DEBUG_TX_QUEUES(mvm,
+ "Failed allocating TXQ for sta %d tid %d, ret: %d\n",
+ sta_id, tid, queue);
+ return queue;
+ }
+
+ IWL_DEBUG_TX_QUEUES(mvm, "Enabling TXQ #%d for sta %d tid %d\n",
+ queue, sta_id, tid);
+
+ mvm->hw_queue_to_mac80211[queue] |= BIT(mac80211_queue);
+ IWL_DEBUG_TX_QUEUES(mvm,
+ "Enabling TXQ #%d (mac80211 map:0x%x)\n",
+ queue, mvm->hw_queue_to_mac80211[queue]);
+
+ return queue;
+}
+
+void iwl_mvm_enable_txq(struct iwl_mvm *mvm, int queue, int mac80211_queue,
+ u16 ssn, const struct iwl_trans_txq_scd_cfg *cfg,
+ unsigned int wdg_timeout)
+{
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return;
+
/* Send the enabling command if we need to */
- if (enable_queue) {
+ if (iwl_mvm_update_txq_mapping(mvm, queue, mac80211_queue,
+ cfg->sta_id, cfg->tid)) {
struct iwl_scd_txq_cfg_cmd cmd = {
.scd_queue = queue,
.action = SCD_CFG_ENABLE_QUEUE,
@@ -701,7 +753,8 @@
iwl_trans_txq_enable_cfg(mvm->trans, queue, ssn, NULL,
wdg_timeout);
- WARN(iwl_mvm_send_cmd_pdu(mvm, SCD_QUEUE_CFG, 0, sizeof(cmd),
+ WARN(iwl_mvm_send_cmd_pdu(mvm, SCD_QUEUE_CFG, 0,
+ sizeof(struct iwl_scd_txq_cfg_cmd),
&cmd),
"Failed to configure queue %d on FIFO %d\n", queue,
cfg->fifo);
@@ -718,6 +771,16 @@
bool remove_mac_queue = true;
int ret;
+ if (iwl_mvm_has_new_tx_api(mvm)) {
+ spin_lock_bh(&mvm->queue_info_lock);
+ mvm->hw_queue_to_mac80211[queue] &= ~BIT(mac80211_queue);
+ spin_unlock_bh(&mvm->queue_info_lock);
+
+ iwl_trans_txq_free(mvm->trans, queue);
+
+ return 0;
+ }
+
spin_lock_bh(&mvm->queue_info_lock);
if (WARN_ON(mvm->queue_info[queue].hw_queue_refcount == 0)) {
@@ -744,7 +807,7 @@
}
if (remove_mac_queue)
- mvm->queue_info[queue].hw_queue_to_mac80211 &=
+ mvm->hw_queue_to_mac80211[queue] &=
~BIT(mac80211_queue);
mvm->queue_info[queue].hw_queue_refcount--;
@@ -757,7 +820,7 @@
"Disabling TXQ #%d refcount=%d (mac80211 map:0x%x)\n",
queue,
mvm->queue_info[queue].hw_queue_refcount,
- mvm->queue_info[queue].hw_queue_to_mac80211);
+ mvm->hw_queue_to_mac80211[queue]);
/* If the queue is still enabled - nothing left to do in this func */
if (cmd.action == SCD_CFG_ENABLE_QUEUE) {
@@ -771,16 +834,16 @@
/* Make sure queue info is correct even though we overwrite it */
WARN(mvm->queue_info[queue].hw_queue_refcount ||
mvm->queue_info[queue].tid_bitmap ||
- mvm->queue_info[queue].hw_queue_to_mac80211,
+ mvm->hw_queue_to_mac80211[queue],
"TXQ #%d info out-of-sync - refcount=%d, mac map=0x%x, tid=0x%x\n",
queue, mvm->queue_info[queue].hw_queue_refcount,
- mvm->queue_info[queue].hw_queue_to_mac80211,
+ mvm->hw_queue_to_mac80211[queue],
mvm->queue_info[queue].tid_bitmap);
/* If we are here - the queue is freed and we can zero out these vals */
mvm->queue_info[queue].hw_queue_refcount = 0;
mvm->queue_info[queue].tid_bitmap = 0;
- mvm->queue_info[queue].hw_queue_to_mac80211 = 0;
+ mvm->hw_queue_to_mac80211[queue] = 0;
/* Regardless if this is a reserved TXQ for a STA - mark it as false */
mvm->queue_info[queue].reserved = false;
@@ -789,11 +852,11 @@
iwl_trans_txq_disable(mvm->trans, queue, false);
ret = iwl_mvm_send_cmd_pdu(mvm, SCD_QUEUE_CFG, flags,
- sizeof(cmd), &cmd);
+ sizeof(struct iwl_scd_txq_cfg_cmd), &cmd);
+
if (ret)
IWL_ERR(mvm, "Failed to disable queue %d (ret=%d)\n",
queue, ret);
-
return ret;
}
@@ -816,7 +879,7 @@
.data = { lq, },
};
- if (WARN_ON(lq->sta_id == IWL_MVM_STATION_COUNT))
+ if (WARN_ON(lq->sta_id == IWL_MVM_INVALID_STA))
return -EINVAL;
return iwl_mvm_send_cmd(mvm, &cmd);
@@ -1006,6 +1069,35 @@
return bss_iter_data.vif;
}
+struct iwl_sta_iter_data {
+ bool assoc;
+};
+
+static void iwl_mvm_sta_iface_iterator(void *_data, u8 *mac,
+ struct ieee80211_vif *vif)
+{
+ struct iwl_sta_iter_data *data = _data;
+
+ if (vif->type != NL80211_IFTYPE_STATION)
+ return;
+
+ if (vif->bss_conf.assoc)
+ data->assoc = true;
+}
+
+bool iwl_mvm_is_vif_assoc(struct iwl_mvm *mvm)
+{
+ struct iwl_sta_iter_data data = {
+ .assoc = false,
+ };
+
+ ieee80211_iterate_active_interfaces_atomic(mvm->hw,
+ IEEE80211_IFACE_ITER_NORMAL,
+ iwl_mvm_sta_iface_iterator,
+ &data);
+ return data.assoc;
+}
+
unsigned int iwl_mvm_get_wd_timeout(struct iwl_mvm *mvm,
struct ieee80211_vif *vif,
bool tdls, bool cmd_q)
@@ -1088,6 +1180,9 @@
lockdep_assert_held(&mvmsta->lock);
lockdep_assert_held(&mvm->queue_info_lock);
+ if (WARN_ON(iwl_mvm_has_new_tx_api(mvm)))
+ return;
+
/* Go over all non-active TIDs, incl. IWL_MAX_TID_COUNT (for mgmt) */
for_each_set_bit(tid, &tid_bitmap, IWL_MAX_TID_COUNT + 1) {
/* If some TFDs are still queued - don't mark TID as inactive */
@@ -1114,8 +1209,8 @@
for_each_set_bit(tid, &tid_bitmap, IWL_MAX_TID_COUNT + 1) {
int mac_queue = mvmsta->vif->hw_queue[tid_to_mac80211_ac[tid]];
- mvmsta->tid_data[tid].txq_id = IEEE80211_INVAL_HW_QUEUE;
- mvm->queue_info[queue].hw_queue_to_mac80211 &= ~BIT(mac_queue);
+ mvmsta->tid_data[tid].txq_id = IWL_MVM_INVALID_QUEUE;
+ mvm->hw_queue_to_mac80211[queue] &= ~BIT(mac_queue);
mvm->queue_info[queue].hw_queue_refcount--;
mvm->queue_info[queue].tid_bitmap &= ~BIT(tid);
mvmsta->tid_data[tid].is_tid_active = false;
@@ -1135,7 +1230,7 @@
*/
tid_bitmap = mvm->queue_info[queue].tid_bitmap;
for_each_set_bit(tid, &tid_bitmap, IWL_MAX_TID_COUNT + 1) {
- mvm->queue_info[queue].hw_queue_to_mac80211 |=
+ mvm->hw_queue_to_mac80211[queue] |=
BIT(mvmsta->vif->hw_queue[tid_to_mac80211_ac[tid]]);
}
@@ -1154,6 +1249,9 @@
unsigned long now = jiffies;
int i;
+ if (iwl_mvm_has_new_tx_api(mvm))
+ return;
+
spin_lock_bh(&mvm->queue_info_lock);
for (i = 0; i < IWL_MAX_HW_QUEUES; i++)
if (mvm->queue_info[i].hw_queue_refcount > 0)
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/ctxt-info.c b/drivers/net/wireless/intel/iwlwifi/pcie/ctxt-info.c
new file mode 100644
index 0000000..b1f43397
--- /dev/null
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/ctxt-info.c
@@ -0,0 +1,281 @@
+/******************************************************************************
+ *
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2017 Intel Deutschland GmbH
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Deutschland GmbH
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ *****************************************************************************/
+
+#include "iwl-trans.h"
+#include "iwl-fh.h"
+#include "iwl-context-info.h"
+#include "internal.h"
+#include "iwl-prph.h"
+
+static int iwl_pcie_get_num_sections(const struct fw_img *fw,
+ int start)
+{
+ int i = 0;
+
+ while (start < fw->num_sec &&
+ fw->sec[start].offset != CPU1_CPU2_SEPARATOR_SECTION &&
+ fw->sec[start].offset != PAGING_SEPARATOR_SECTION) {
+ start++;
+ i++;
+ }
+
+ return i;
+}
+
+static int iwl_pcie_ctxt_info_alloc_dma(struct iwl_trans *trans,
+ const struct fw_desc *sec,
+ struct iwl_dram_data *dram)
+{
+ dram->block = dma_alloc_coherent(trans->dev, sec->len,
+ &dram->physical,
+ GFP_KERNEL);
+ if (!dram->block)
+ return -ENOMEM;
+
+ dram->size = sec->len;
+ memcpy(dram->block, sec->data, sec->len);
+
+ return 0;
+}
+
+static void iwl_pcie_ctxt_info_free_fw_img(struct iwl_trans *trans)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct iwl_self_init_dram *dram = &trans_pcie->init_dram;
+ int i;
+
+ if (!dram->fw) {
+ WARN_ON(dram->fw_cnt);
+ return;
+ }
+
+ for (i = 0; i < dram->fw_cnt; i++)
+ dma_free_coherent(trans->dev, dram->fw[i].size,
+ dram->fw[i].block, dram->fw[i].physical);
+
+ kfree(dram->fw);
+ dram->fw_cnt = 0;
+}
+
+void iwl_pcie_ctxt_info_free_paging(struct iwl_trans *trans)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct iwl_self_init_dram *dram = &trans_pcie->init_dram;
+ int i;
+
+ if (!dram->paging) {
+ WARN_ON(dram->paging_cnt);
+ return;
+ }
+
+ /* free paging*/
+ for (i = 0; i < dram->paging_cnt; i++)
+ dma_free_coherent(trans->dev, dram->paging[i].size,
+ dram->paging[i].block,
+ dram->paging[i].physical);
+
+ kfree(dram->paging);
+ dram->paging_cnt = 0;
+}
+
+static int iwl_pcie_ctxt_info_init_fw_sec(struct iwl_trans *trans,
+ const struct fw_img *fw,
+ struct iwl_context_info *ctxt_info)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct iwl_self_init_dram *dram = &trans_pcie->init_dram;
+ struct iwl_context_info_dram *ctxt_dram = &ctxt_info->dram;
+ int i, ret, lmac_cnt, umac_cnt, paging_cnt;
+
+ lmac_cnt = iwl_pcie_get_num_sections(fw, 0);
+ /* add 1 due to separator */
+ umac_cnt = iwl_pcie_get_num_sections(fw, lmac_cnt + 1);
+ /* add 2 due to separators */
+ paging_cnt = iwl_pcie_get_num_sections(fw, lmac_cnt + umac_cnt + 2);
+
+ dram->fw = kcalloc(umac_cnt + lmac_cnt, sizeof(*dram->fw), GFP_KERNEL);
+ if (!dram->fw)
+ return -ENOMEM;
+ dram->paging = kcalloc(paging_cnt, sizeof(*dram->paging), GFP_KERNEL);
+ if (!dram->paging)
+ return -ENOMEM;
+
+ /* initialize lmac sections */
+ for (i = 0; i < lmac_cnt; i++) {
+ ret = iwl_pcie_ctxt_info_alloc_dma(trans, &fw->sec[i],
+ &dram->fw[dram->fw_cnt]);
+ if (ret)
+ return ret;
+ ctxt_dram->lmac_img[i] =
+ cpu_to_le64(dram->fw[dram->fw_cnt].physical);
+ dram->fw_cnt++;
+ }
+
+ /* initialize umac sections */
+ for (i = 0; i < umac_cnt; i++) {
+ /* access FW with +1 to make up for lmac separator */
+ ret = iwl_pcie_ctxt_info_alloc_dma(trans,
+ &fw->sec[dram->fw_cnt + 1],
+ &dram->fw[dram->fw_cnt]);
+ if (ret)
+ return ret;
+ ctxt_dram->umac_img[i] =
+ cpu_to_le64(dram->fw[dram->fw_cnt].physical);
+ dram->fw_cnt++;
+ }
+
+ /*
+ * Initialize paging.
+ * Paging memory isn't stored in dram->fw as the umac and lmac - it is
+ * stored separately.
+ * This is since the timing of its release is different -
+ * while fw memory can be released on alive, the paging memory can be
+ * freed only when the device goes down.
+ * Given that, the logic here in accessing the fw image is a bit
+ * different - fw_cnt isn't changing so loop counter is added to it.
+ */
+ for (i = 0; i < paging_cnt; i++) {
+ /* access FW with +2 to make up for lmac & umac separators */
+ int fw_idx = dram->fw_cnt + i + 2;
+
+ ret = iwl_pcie_ctxt_info_alloc_dma(trans, &fw->sec[fw_idx],
+ &dram->paging[i]);
+ if (ret)
+ return ret;
+
+ ctxt_dram->virtual_img[i] =
+ cpu_to_le64(dram->paging[i].physical);
+ dram->paging_cnt++;
+ }
+
+ return 0;
+}
+
+int iwl_pcie_ctxt_info_init(struct iwl_trans *trans,
+ const struct fw_img *fw)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct iwl_context_info *ctxt_info;
+ struct iwl_context_info_rbd_cfg *rx_cfg;
+ u32 control_flags = 0;
+ int ret;
+
+ ctxt_info = dma_alloc_coherent(trans->dev, sizeof(*ctxt_info),
+ &trans_pcie->ctxt_info_dma_addr,
+ GFP_KERNEL);
+ if (!ctxt_info)
+ return -ENOMEM;
+
+ ctxt_info->version.version = 0;
+ ctxt_info->version.mac_id =
+ cpu_to_le16((u16)iwl_read32(trans, CSR_HW_REV));
+ /* size is in DWs */
+ ctxt_info->version.size = cpu_to_le16(sizeof(*ctxt_info) / 4);
+
+ BUILD_BUG_ON(RX_QUEUE_CB_SIZE(MQ_RX_TABLE_SIZE) > 0xF);
+ control_flags = IWL_CTXT_INFO_RB_SIZE_4K |
+ IWL_CTXT_INFO_TFD_FORMAT_LONG |
+ RX_QUEUE_CB_SIZE(MQ_RX_TABLE_SIZE) <<
+ IWL_CTXT_INFO_RB_CB_SIZE_POS;
+ ctxt_info->control.control_flags = cpu_to_le32(control_flags);
+
+ /* initialize RX default queue */
+ rx_cfg = &ctxt_info->rbd_cfg;
+ rx_cfg->free_rbd_addr = cpu_to_le64(trans_pcie->rxq->bd_dma);
+ rx_cfg->used_rbd_addr = cpu_to_le64(trans_pcie->rxq->used_bd_dma);
+ rx_cfg->status_wr_ptr = cpu_to_le64(trans_pcie->rxq->rb_stts_dma);
+
+ /* initialize TX command queue */
+ ctxt_info->hcmd_cfg.cmd_queue_addr =
+ cpu_to_le64(trans_pcie->txq[trans_pcie->cmd_queue]->dma_addr);
+ ctxt_info->hcmd_cfg.cmd_queue_size =
+ TFD_QUEUE_CB_SIZE(TFD_QUEUE_SIZE_MAX);
+
+ /* allocate ucode sections in dram and set addresses */
+ ret = iwl_pcie_ctxt_info_init_fw_sec(trans, fw, ctxt_info);
+ if (ret) {
+ dma_free_coherent(trans->dev, sizeof(*trans_pcie->ctxt_info),
+ ctxt_info, trans_pcie->ctxt_info_dma_addr);
+ return ret;
+ }
+
+ trans_pcie->ctxt_info = ctxt_info;
+
+ iwl_enable_interrupts(trans);
+
+ /* Configure debug, if exists */
+ if (trans->dbg_dest_tlv)
+ iwl_pcie_apply_destination(trans);
+
+ /* kick FW self load */
+ iwl_write64(trans, CSR_CTXT_INFO_BA, trans_pcie->ctxt_info_dma_addr);
+ iwl_write_prph(trans, UREG_CPU_INIT_RUN, 1);
+
+ /* Context info will be released upon alive or failure to get one */
+
+ return 0;
+}
+
+void iwl_pcie_ctxt_info_free(struct iwl_trans *trans)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+
+ if (!trans_pcie->ctxt_info)
+ return;
+
+ dma_free_coherent(trans->dev, sizeof(*trans_pcie->ctxt_info),
+ trans_pcie->ctxt_info,
+ trans_pcie->ctxt_info_dma_addr);
+ trans_pcie->ctxt_info_dma_addr = 0;
+ trans_pcie->ctxt_info = NULL;
+
+ iwl_pcie_ctxt_info_free_fw_img(trans);
+}
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/drv.c b/drivers/net/wireless/intel/iwlwifi/pcie/drv.c
index ba8a81c..e51760e 100644
--- a/drivers/net/wireless/intel/iwlwifi/pcie/drv.c
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/drv.c
@@ -501,6 +501,10 @@
{IWL_PCI_DEVICE(0x24FD, 0x0930, iwl8265_2ac_cfg)},
{IWL_PCI_DEVICE(0x24FD, 0x0950, iwl8265_2ac_cfg)},
{IWL_PCI_DEVICE(0x24FD, 0x0850, iwl8265_2ac_cfg)},
+ {IWL_PCI_DEVICE(0x24FD, 0x1014, iwl8265_2ac_cfg)},
+ {IWL_PCI_DEVICE(0x24FD, 0x3E02, iwl8275_2ac_cfg)},
+ {IWL_PCI_DEVICE(0x24FD, 0x3E01, iwl8275_2ac_cfg)},
+ {IWL_PCI_DEVICE(0x24FD, 0x1012, iwl8275_2ac_cfg)},
{IWL_PCI_DEVICE(0x24FD, 0x0012, iwl8275_2ac_cfg)},
/* 9000 Series */
@@ -533,7 +537,8 @@
{IWL_PCI_DEVICE(0xA370, 0x1030, iwl9560_2ac_cfg)},
/* a000 Series */
- {IWL_PCI_DEVICE(0x2720, 0x0A10, iwla000_2ac_cfg_hr)},
+ {IWL_PCI_DEVICE(0x2720, 0x0A10, iwla000_2ac_cfg_hr_cdb)},
+ {IWL_PCI_DEVICE(0x2722, 0x0A10, iwla000_2ac_cfg_hr)},
#endif /* CONFIG_IWLMVM */
{0}
@@ -667,18 +672,11 @@
iwl_trans->cfg = cfg_7265d;
}
- if (iwl_trans->cfg->rf_id) {
- if (cfg == &iwl9460_2ac_cfg &&
- iwl_trans->hw_rf_id == CSR_HW_RF_ID_TYPE_LC) {
- cfg = &iwl9000lc_2ac_cfg;
- iwl_trans->cfg = cfg;
- }
-
- if (cfg == &iwla000_2ac_cfg_hr &&
- iwl_trans->hw_rf_id == CSR_HW_RF_ID_TYPE_JF) {
- cfg = &iwla000_2ac_cfg_jf;
- iwl_trans->cfg = cfg;
- }
+ if (iwl_trans->cfg->rf_id &&
+ (cfg == &iwla000_2ac_cfg_hr || cfg == &iwla000_2ac_cfg_hr_cdb) &&
+ iwl_trans->hw_rf_id == CSR_HW_RF_ID_TYPE_JF) {
+ cfg = &iwla000_2ac_cfg_jf;
+ iwl_trans->cfg = cfg;
}
#endif
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/internal.h b/drivers/net/wireless/intel/iwlwifi/pcie/internal.h
index 1093730..fd4faaa 100644
--- a/drivers/net/wireless/intel/iwlwifi/pcie/internal.h
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/internal.h
@@ -2,7 +2,7 @@
*
* Copyright(c) 2003 - 2015 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* Portions of this file are derived from the ipw3945 project, as well
* as portions of the ieee80211 subsystem header files.
@@ -205,11 +205,11 @@
* into the buffer regardless of whether it should be mapped or not.
* This indicates how big the first TB must be to include the scratch buffer
* and the assigned PN.
- * Since PN location is 16 bytes at offset 24, it's 40 now.
+ * Since PN location is 8 bytes at offset 12, it's 20 now.
* If we make it bigger then allocations will be bigger and copy slower, so
* that's probably not useful.
*/
-#define IWL_FIRST_TB_SIZE 40
+#define IWL_FIRST_TB_SIZE 20
#define IWL_FIRST_TB_SIZE_ALIGN ALIGN(IWL_FIRST_TB_SIZE, 64)
struct iwl_pcie_txq_entry {
@@ -237,11 +237,11 @@
* @stuck_timer: timer that fires if queue gets stuck
* @trans_pcie: pointer back to transport (for timer)
* @need_update: indicates need to update read/write index
- * @active: stores if queue is active
* @ampdu: true if this queue is an ampdu queue for an specific RA/TID
* @wd_timeout: queue watchdog timeout (jiffies) - per queue
* @frozen: tx stuck queue timer is frozen
* @frozen_expiry_remainder: remember how long until the timer fires
+ * @bc_tbl: byte count table of the queue (relevant only for gen2 transport)
* @write_ptr: 1-st empty entry (index) host_w
* @read_ptr: last used entry (index) host_r
* @dma_addr: physical addr for BD's
@@ -277,11 +277,11 @@
struct iwl_trans_pcie *trans_pcie;
bool need_update;
bool frozen;
- u8 active;
bool ampdu;
int block;
unsigned long wd_timeout;
struct sk_buff_head overflow_q;
+ struct iwl_dma_ptr bc_tbl;
int write_ptr;
int read_ptr;
@@ -315,11 +315,43 @@
};
/**
+ * struct iwl_dram_data
+ * @physical: page phy pointer
+ * @block: pointer to the allocated block/page
+ * @size: size of the block/page
+ */
+struct iwl_dram_data {
+ dma_addr_t physical;
+ void *block;
+ int size;
+};
+
+/**
+ * struct iwl_self_init_dram - dram data used by self init process
+ * @fw: lmac and umac dram data
+ * @fw_cnt: total number of items in array
+ * @paging: paging dram data
+ * @paging_cnt: total number of items in array
+ */
+struct iwl_self_init_dram {
+ struct iwl_dram_data *fw;
+ int fw_cnt;
+ struct iwl_dram_data *paging;
+ int paging_cnt;
+};
+
+/**
* struct iwl_trans_pcie - PCIe transport specific data
* @rxq: all the RX queue data
* @rx_pool: initial pool of iwl_rx_mem_buffer for all the queues
* @global_table: table mapping received VID from hw to rxb
* @rba: allocator for RX replenishing
+ * @ctxt_info: context information for FW self init
+ * @ctxt_info_dma_addr: dma addr of context information
+ * @init_dram: DRAM data of firmware image (including paging).
+ * Context information addresses will be taken from here.
+ * This is driver's local copy for keeping track of size and
+ * count for allocating and freeing the memory.
* @trans: pointer to the generic transport area
* @scd_base_addr: scheduler sram base address in SRAM
* @scd_bc_tbls: pointer to the byte count table of the scheduler
@@ -357,6 +389,9 @@
struct iwl_rx_mem_buffer rx_pool[RX_POOL_SIZE];
struct iwl_rx_mem_buffer *global_table[RX_POOL_SIZE];
struct iwl_rb_allocator rba;
+ struct iwl_context_info *ctxt_info;
+ dma_addr_t ctxt_info_dma_addr;
+ struct iwl_self_init_dram init_dram;
struct iwl_trans *trans;
struct net_device napi_dev;
@@ -378,9 +413,10 @@
struct iwl_dma_ptr scd_bc_tbls;
struct iwl_dma_ptr kw;
- struct iwl_txq *txq;
- unsigned long queue_used[BITS_TO_LONGS(IWL_MAX_HW_QUEUES)];
- unsigned long queue_stopped[BITS_TO_LONGS(IWL_MAX_HW_QUEUES)];
+ struct iwl_txq *txq_memory;
+ struct iwl_txq *txq[IWL_MAX_TVQM_QUEUES];
+ unsigned long queue_used[BITS_TO_LONGS(IWL_MAX_TVQM_QUEUES)];
+ unsigned long queue_stopped[BITS_TO_LONGS(IWL_MAX_TVQM_QUEUES)];
/* PCI bus related data */
struct pci_dev *pci_dev;
@@ -454,6 +490,7 @@
* RX
******************************************************/
int iwl_pcie_rx_init(struct iwl_trans *trans);
+int iwl_pcie_gen2_rx_init(struct iwl_trans *trans);
irqreturn_t iwl_pcie_msix_isr(int irq, void *data);
irqreturn_t iwl_pcie_irq_handler(int irq, void *dev_id);
irqreturn_t iwl_pcie_irq_msix_handler(int irq, void *dev_id);
@@ -474,6 +511,7 @@
* TX / HCMD
******************************************************/
int iwl_pcie_tx_init(struct iwl_trans *trans);
+int iwl_pcie_gen2_tx_init(struct iwl_trans *trans);
void iwl_pcie_tx_start(struct iwl_trans *trans, u32 scd_base_addr);
int iwl_pcie_tx_stop(struct iwl_trans *trans);
void iwl_pcie_tx_free(struct iwl_trans *trans);
@@ -484,7 +522,6 @@
bool configure_scd);
void iwl_trans_pcie_txq_set_shared_mode(struct iwl_trans *trans, u32 txq_id,
bool shared_mode);
-dma_addr_t iwl_trans_pcie_get_txq_byte_table(struct iwl_trans *trans, int txq);
void iwl_trans_pcie_log_scd_error(struct iwl_trans *trans,
struct iwl_txq *txq);
int iwl_trans_pcie_tx(struct iwl_trans *trans, struct sk_buff *skb,
@@ -616,6 +653,12 @@
}
}
+static inline void *iwl_pcie_get_tfd(struct iwl_trans_pcie *trans_pcie,
+ struct iwl_txq *txq, int idx)
+{
+ return txq->tfds + trans_pcie->tfd_size * idx;
+}
+
static inline void iwl_enable_rfkill_int(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
@@ -719,4 +762,41 @@
void iwl_pcie_enable_rx_wake(struct iwl_trans *trans, bool enable);
+/* common functions that are used by gen2 transport */
+void iwl_pcie_apm_config(struct iwl_trans *trans);
+int iwl_pcie_prepare_card_hw(struct iwl_trans *trans);
+void iwl_pcie_synchronize_irqs(struct iwl_trans *trans);
+bool iwl_trans_check_hw_rf_kill(struct iwl_trans *trans);
+void iwl_pcie_txq_free_tfd(struct iwl_trans *trans, struct iwl_txq *txq);
+int iwl_queue_space(const struct iwl_txq *q);
+int iwl_pcie_apm_stop_master(struct iwl_trans *trans);
+void iwl_pcie_conf_msix_hw(struct iwl_trans_pcie *trans_pcie);
+int iwl_pcie_txq_init(struct iwl_trans *trans, struct iwl_txq *txq,
+ int slots_num, bool cmd_queue);
+int iwl_pcie_txq_alloc(struct iwl_trans *trans,
+ struct iwl_txq *txq, int slots_num, bool cmd_queue);
+int iwl_pcie_alloc_dma_ptr(struct iwl_trans *trans,
+ struct iwl_dma_ptr *ptr, size_t size);
+void iwl_pcie_free_dma_ptr(struct iwl_trans *trans, struct iwl_dma_ptr *ptr);
+void iwl_pcie_apply_destination(struct iwl_trans *trans);
+
+/* transport gen 2 exported functions */
+int iwl_trans_pcie_gen2_start_fw(struct iwl_trans *trans,
+ const struct fw_img *fw, bool run_in_rfkill);
+void iwl_trans_pcie_gen2_fw_alive(struct iwl_trans *trans, u32 scd_addr);
+int iwl_trans_pcie_dyn_txq_alloc(struct iwl_trans *trans,
+ struct iwl_tx_queue_cfg_cmd *cmd,
+ int cmd_id,
+ unsigned int timeout);
+void iwl_trans_pcie_dyn_txq_free(struct iwl_trans *trans, int queue);
+int iwl_trans_pcie_gen2_tx(struct iwl_trans *trans, struct sk_buff *skb,
+ struct iwl_device_cmd *dev_cmd, int txq_id);
+int iwl_trans_pcie_gen2_send_hcmd(struct iwl_trans *trans,
+ struct iwl_host_cmd *cmd);
+void iwl_trans_pcie_gen2_stop_device(struct iwl_trans *trans,
+ bool low_power);
+void _iwl_trans_pcie_gen2_stop_device(struct iwl_trans *trans, bool low_power);
+void iwl_pcie_gen2_txq_unmap(struct iwl_trans *trans, int txq_id);
+void iwl_pcie_gen2_tx_free(struct iwl_trans *trans);
+void iwl_pcie_gen2_tx_stop(struct iwl_trans *trans);
#endif /* __iwl_trans_int_pcie_h__ */
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
index de94dfd..1da2de2 100644
--- a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
@@ -2,7 +2,7 @@
*
* Copyright(c) 2003 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* Portions of this file are derived from the ipw3945 project, as well
* as portions of the ieee80211 subsystem header files.
@@ -880,7 +880,7 @@
return 0;
}
-int iwl_pcie_rx_init(struct iwl_trans *trans)
+static int _iwl_pcie_rx_init(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
struct iwl_rxq *def_rxq;
@@ -958,20 +958,40 @@
iwl_pcie_rxq_alloc_rbs(trans, GFP_KERNEL, def_rxq);
+ return 0;
+}
+
+int iwl_pcie_rx_init(struct iwl_trans *trans)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ int ret = _iwl_pcie_rx_init(trans);
+
+ if (ret)
+ return ret;
+
if (trans->cfg->mq_rx_supported)
iwl_pcie_rx_mq_hw_init(trans);
else
- iwl_pcie_rx_hw_init(trans, def_rxq);
+ iwl_pcie_rx_hw_init(trans, trans_pcie->rxq);
- iwl_pcie_rxq_restock(trans, def_rxq);
+ iwl_pcie_rxq_restock(trans, trans_pcie->rxq);
- spin_lock(&def_rxq->lock);
- iwl_pcie_rxq_inc_wr_ptr(trans, def_rxq);
- spin_unlock(&def_rxq->lock);
+ spin_lock(&trans_pcie->rxq->lock);
+ iwl_pcie_rxq_inc_wr_ptr(trans, trans_pcie->rxq);
+ spin_unlock(&trans_pcie->rxq->lock);
return 0;
}
+int iwl_pcie_gen2_rx_init(struct iwl_trans *trans)
+{
+ /*
+ * We don't configure the RFH.
+ * Restock will be done at alive, after firmware configured the RFH.
+ */
+ return _iwl_pcie_rx_init(trans);
+}
+
void iwl_pcie_rx_free(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
@@ -1074,7 +1094,7 @@
bool emergency)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_txq *txq = &trans_pcie->txq[trans_pcie->cmd_queue];
+ struct iwl_txq *txq = trans_pcie->txq[trans_pcie->cmd_queue];
bool page_stolen = false;
int max_len = PAGE_SIZE << trans_pcie->rx_page_order;
u32 offset = 0;
@@ -1127,7 +1147,7 @@
* Ucode should set SEQ_RX_FRAME bit if ucode-originated,
* but apparently a few don't get set; catch them here. */
reclaim = !(pkt->hdr.sequence & SEQ_RX_FRAME);
- if (reclaim) {
+ if (reclaim && !pkt->hdr.group_id) {
int i;
for (i = 0; i < trans_pcie->n_no_reclaim_cmds; i++) {
@@ -1393,17 +1413,17 @@
return;
}
- iwl_pcie_dump_csr(trans);
- iwl_dump_fh(trans, NULL);
-
local_bh_disable();
/* The STATUS_FW_ERROR bit is set in this function. This must happen
* before we wake up the command caller, to ensure a proper cleanup. */
iwl_trans_fw_error(trans);
local_bh_enable();
- for (i = 0; i < trans->cfg->base_params->num_of_queues; i++)
- del_timer(&trans_pcie->txq[i].stuck_timer);
+ for (i = 0; i < trans->cfg->base_params->num_of_queues; i++) {
+ if (!trans_pcie->txq[i])
+ continue;
+ del_timer(&trans_pcie->txq[i]->stuck_timer);
+ }
clear_bit(STATUS_SYNC_HCMD_ACTIVE, &trans->status);
wake_up(&trans_pcie->wait_command_queue);
@@ -1597,6 +1617,13 @@
if (inta & CSR_INT_BIT_ALIVE) {
IWL_DEBUG_ISR(trans, "Alive interrupt\n");
isr_stats->alive++;
+ if (trans->cfg->gen2) {
+ /*
+ * We can restock, since firmware configured
+ * the RFH
+ */
+ iwl_pcie_rxmq_restock(trans, trans_pcie->rxq);
+ }
}
}
@@ -1933,6 +1960,10 @@
if (inta_hw & MSIX_HW_INT_CAUSES_REG_ALIVE) {
IWL_DEBUG_ISR(trans, "Alive interrupt\n");
isr_stats->alive++;
+ if (trans->cfg->gen2) {
+ /* We can restock, since firmware configured the RFH */
+ iwl_pcie_rxmq_restock(trans, trans_pcie->rxq);
+ }
}
/* uCode wakes up after power-down sleep */
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c b/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c
new file mode 100644
index 0000000..ac60a28
--- /dev/null
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/trans-gen2.c
@@ -0,0 +1,374 @@
+/******************************************************************************
+ *
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2017 Intel Deutschland GmbH
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Deutschland GmbH
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ *****************************************************************************/
+#include "iwl-trans.h"
+#include "iwl-context-info.h"
+#include "internal.h"
+
+/*
+ * Start up NIC's basic functionality after it has been reset
+ * (e.g. after platform boot, or shutdown via iwl_pcie_apm_stop())
+ * NOTE: This does not load uCode nor start the embedded processor
+ */
+static int iwl_pcie_gen2_apm_init(struct iwl_trans *trans)
+{
+ int ret = 0;
+
+ IWL_DEBUG_INFO(trans, "Init card's basic functions\n");
+
+ /*
+ * Use "set_bit" below rather than "write", to preserve any hardware
+ * bits already set by default after reset.
+ */
+
+ /*
+ * Disable L0s without affecting L1;
+ * don't wait for ICH L0s (ICH bug W/A)
+ */
+ iwl_set_bit(trans, CSR_GIO_CHICKEN_BITS,
+ CSR_GIO_CHICKEN_BITS_REG_BIT_L1A_NO_L0S_RX);
+
+ /* Set FH wait threshold to maximum (HW error during stress W/A) */
+ iwl_set_bit(trans, CSR_DBG_HPET_MEM_REG, CSR_DBG_HPET_MEM_REG_VAL);
+
+ /*
+ * Enable HAP INTA (interrupt from management bus) to
+ * wake device's PCI Express link L1a -> L0s
+ */
+ iwl_set_bit(trans, CSR_HW_IF_CONFIG_REG,
+ CSR_HW_IF_CONFIG_REG_BIT_HAP_WAKE_L1A);
+
+ iwl_pcie_apm_config(trans);
+
+ /*
+ * Set "initialization complete" bit to move adapter from
+ * D0U* --> D0A* (powered-up active) state.
+ */
+ iwl_set_bit(trans, CSR_GP_CNTRL, CSR_GP_CNTRL_REG_FLAG_INIT_DONE);
+
+ /*
+ * Wait for clock stabilization; once stabilized, access to
+ * device-internal resources is supported, e.g. iwl_write_prph()
+ * and accesses to uCode SRAM.
+ */
+ ret = iwl_poll_bit(trans, CSR_GP_CNTRL,
+ CSR_GP_CNTRL_REG_FLAG_MAC_CLOCK_READY,
+ CSR_GP_CNTRL_REG_FLAG_MAC_CLOCK_READY, 25000);
+ if (ret < 0) {
+ IWL_DEBUG_INFO(trans, "Failed to init the card\n");
+ return ret;
+ }
+
+ set_bit(STATUS_DEVICE_ENABLED, &trans->status);
+
+ return 0;
+}
+
+static void iwl_pcie_gen2_apm_stop(struct iwl_trans *trans, bool op_mode_leave)
+{
+ IWL_DEBUG_INFO(trans, "Stop card, put in low power state\n");
+
+ if (op_mode_leave) {
+ if (!test_bit(STATUS_DEVICE_ENABLED, &trans->status))
+ iwl_pcie_gen2_apm_init(trans);
+
+ /* inform ME that we are leaving */
+ iwl_set_bit(trans, CSR_DBG_LINK_PWR_MGMT_REG,
+ CSR_RESET_LINK_PWR_MGMT_DISABLED);
+ iwl_set_bit(trans, CSR_HW_IF_CONFIG_REG,
+ CSR_HW_IF_CONFIG_REG_PREPARE |
+ CSR_HW_IF_CONFIG_REG_ENABLE_PME);
+ mdelay(1);
+ iwl_clear_bit(trans, CSR_DBG_LINK_PWR_MGMT_REG,
+ CSR_RESET_LINK_PWR_MGMT_DISABLED);
+ mdelay(5);
+ }
+
+ clear_bit(STATUS_DEVICE_ENABLED, &trans->status);
+
+ /* Stop device's DMA activity */
+ iwl_pcie_apm_stop_master(trans);
+
+ /* Reset the entire device */
+ iwl_set_bit(trans, CSR_RESET, CSR_RESET_REG_FLAG_SW_RESET);
+ usleep_range(1000, 2000);
+
+ /*
+ * Clear "initialization complete" bit to move adapter from
+ * D0A* (powered-up Active) --> D0U* (Uninitialized) state.
+ */
+ iwl_clear_bit(trans, CSR_GP_CNTRL, CSR_GP_CNTRL_REG_FLAG_INIT_DONE);
+}
+
+void _iwl_trans_pcie_gen2_stop_device(struct iwl_trans *trans, bool low_power)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ bool hw_rfkill, was_hw_rfkill;
+
+ lockdep_assert_held(&trans_pcie->mutex);
+
+ if (trans_pcie->is_down)
+ return;
+
+ trans_pcie->is_down = true;
+
+ was_hw_rfkill = iwl_is_rfkill_set(trans);
+
+ /* tell the device to stop sending interrupts */
+ iwl_disable_interrupts(trans);
+
+ /* device going down, Stop using ICT table */
+ iwl_pcie_disable_ict(trans);
+
+ /*
+ * If a HW restart happens during firmware loading,
+ * then the firmware loading might call this function
+ * and later it might be called again due to the
+ * restart. So don't process again if the device is
+ * already dead.
+ */
+ if (test_and_clear_bit(STATUS_DEVICE_ENABLED, &trans->status)) {
+ IWL_DEBUG_INFO(trans,
+ "DEVICE_ENABLED bit was set and is now cleared\n");
+ iwl_pcie_gen2_tx_stop(trans);
+ iwl_pcie_rx_stop(trans);
+ }
+
+ iwl_pcie_ctxt_info_free_paging(trans);
+ iwl_pcie_ctxt_info_free(trans);
+
+ /* Make sure (redundant) we've released our request to stay awake */
+ iwl_clear_bit(trans, CSR_GP_CNTRL,
+ CSR_GP_CNTRL_REG_FLAG_MAC_ACCESS_REQ);
+
+ /* Stop the device, and put it in low power state */
+ iwl_pcie_gen2_apm_stop(trans, false);
+
+ /* stop and reset the on-board processor */
+ iwl_write32(trans, CSR_RESET, CSR_RESET_REG_FLAG_SW_RESET);
+ usleep_range(1000, 2000);
+
+ /*
+ * Upon stop, the IVAR table gets erased, so msi-x won't
+ * work. This causes a bug in RF-KILL flows, since the interrupt
+ * that enables radio won't fire on the correct irq, and the
+ * driver won't be able to handle the interrupt.
+ * Configure the IVAR table again after reset.
+ */
+ iwl_pcie_conf_msix_hw(trans_pcie);
+
+ /*
+ * Upon stop, the APM issues an interrupt if HW RF kill is set.
+ * This is a bug in certain verions of the hardware.
+ * Certain devices also keep sending HW RF kill interrupt all
+ * the time, unless the interrupt is ACKed even if the interrupt
+ * should be masked. Re-ACK all the interrupts here.
+ */
+ iwl_disable_interrupts(trans);
+
+ /* clear all status bits */
+ clear_bit(STATUS_SYNC_HCMD_ACTIVE, &trans->status);
+ clear_bit(STATUS_INT_ENABLED, &trans->status);
+ clear_bit(STATUS_TPOWER_PMI, &trans->status);
+ clear_bit(STATUS_RFKILL, &trans->status);
+
+ /*
+ * Even if we stop the HW, we still want the RF kill
+ * interrupt
+ */
+ iwl_enable_rfkill_int(trans);
+
+ /*
+ * Check again since the RF kill state may have changed while
+ * all the interrupts were disabled, in this case we couldn't
+ * receive the RF kill interrupt and update the state in the
+ * op_mode.
+ * Don't call the op_mode if the rkfill state hasn't changed.
+ * This allows the op_mode to call stop_device from the rfkill
+ * notification without endless recursion. Under very rare
+ * circumstances, we might have a small recursion if the rfkill
+ * state changed exactly now while we were called from stop_device.
+ * This is very unlikely but can happen and is supported.
+ */
+ hw_rfkill = iwl_is_rfkill_set(trans);
+ if (hw_rfkill)
+ set_bit(STATUS_RFKILL, &trans->status);
+ else
+ clear_bit(STATUS_RFKILL, &trans->status);
+ if (hw_rfkill != was_hw_rfkill)
+ iwl_trans_pcie_rf_kill(trans, hw_rfkill);
+
+ /* re-take ownership to prevent other users from stealing the device */
+ iwl_pcie_prepare_card_hw(trans);
+}
+
+void iwl_trans_pcie_gen2_stop_device(struct iwl_trans *trans, bool low_power)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+
+ mutex_lock(&trans_pcie->mutex);
+ _iwl_trans_pcie_gen2_stop_device(trans, low_power);
+ mutex_unlock(&trans_pcie->mutex);
+}
+
+static int iwl_pcie_gen2_nic_init(struct iwl_trans *trans)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+
+ /* TODO: most of the logic can be removed in A0 - but not in Z0 */
+ spin_lock(&trans_pcie->irq_lock);
+ iwl_pcie_gen2_apm_init(trans);
+ spin_unlock(&trans_pcie->irq_lock);
+
+ iwl_op_mode_nic_config(trans->op_mode);
+
+ /* Allocate the RX queue, or reset if it is already allocated */
+ if (iwl_pcie_gen2_rx_init(trans))
+ return -ENOMEM;
+
+ /* Allocate or reset and init all Tx and Command queues */
+ if (iwl_pcie_gen2_tx_init(trans))
+ return -ENOMEM;
+
+ /* enable shadow regs in HW */
+ iwl_set_bit(trans, CSR_MAC_SHADOW_REG_CTRL, 0x800FFFFF);
+ IWL_DEBUG_INFO(trans, "Enabling shadow registers in device\n");
+
+ return 0;
+}
+
+void iwl_trans_pcie_gen2_fw_alive(struct iwl_trans *trans, u32 scd_addr)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+
+ iwl_pcie_reset_ict(trans);
+
+ /* make sure all queue are not stopped/used */
+ memset(trans_pcie->queue_stopped, 0, sizeof(trans_pcie->queue_stopped));
+ memset(trans_pcie->queue_used, 0, sizeof(trans_pcie->queue_used));
+
+ /* now that we got alive we can free the fw image & the context info.
+ * paging memory cannot be freed included since FW will still use it
+ */
+ iwl_pcie_ctxt_info_free(trans);
+}
+
+int iwl_trans_pcie_gen2_start_fw(struct iwl_trans *trans,
+ const struct fw_img *fw, bool run_in_rfkill)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ bool hw_rfkill;
+ int ret;
+
+ /* This may fail if AMT took ownership of the device */
+ if (iwl_pcie_prepare_card_hw(trans)) {
+ IWL_WARN(trans, "Exit HW not ready\n");
+ ret = -EIO;
+ goto out;
+ }
+
+ iwl_enable_rfkill_int(trans);
+
+ iwl_write32(trans, CSR_INT, 0xFFFFFFFF);
+
+ /*
+ * We enabled the RF-Kill interrupt and the handler may very
+ * well be running. Disable the interrupts to make sure no other
+ * interrupt can be fired.
+ */
+ iwl_disable_interrupts(trans);
+
+ /* Make sure it finished running */
+ iwl_pcie_synchronize_irqs(trans);
+
+ mutex_lock(&trans_pcie->mutex);
+
+ /* If platform's RF_KILL switch is NOT set to KILL */
+ hw_rfkill = iwl_trans_check_hw_rf_kill(trans);
+ if (hw_rfkill && !run_in_rfkill) {
+ ret = -ERFKILL;
+ goto out;
+ }
+
+ /* Someone called stop_device, don't try to start_fw */
+ if (trans_pcie->is_down) {
+ IWL_WARN(trans,
+ "Can't start_fw since the HW hasn't been started\n");
+ ret = -EIO;
+ goto out;
+ }
+
+ /* make sure rfkill handshake bits are cleared */
+ iwl_write32(trans, CSR_UCODE_DRV_GP1_CLR, CSR_UCODE_SW_BIT_RFKILL);
+ iwl_write32(trans, CSR_UCODE_DRV_GP1_CLR,
+ CSR_UCODE_DRV_GP1_BIT_CMD_BLOCKED);
+
+ /* clear (again), then enable host interrupts */
+ iwl_write32(trans, CSR_INT, 0xFFFFFFFF);
+
+ ret = iwl_pcie_gen2_nic_init(trans);
+ if (ret) {
+ IWL_ERR(trans, "Unable to init nic\n");
+ goto out;
+ }
+
+ ret = iwl_pcie_ctxt_info_init(trans, fw);
+ if (ret)
+ goto out;
+
+ /* re-check RF-Kill state since we may have missed the interrupt */
+ hw_rfkill = iwl_trans_check_hw_rf_kill(trans);
+ if (hw_rfkill && !run_in_rfkill)
+ ret = -ERFKILL;
+
+out:
+ mutex_unlock(&trans_pcie->mutex);
+ return ret;
+}
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/trans.c b/drivers/net/wireless/intel/iwlwifi/pcie/trans.c
index 7f05fc5..70acf85 100644
--- a/drivers/net/wireless/intel/iwlwifi/pcie/trans.c
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/trans.c
@@ -7,7 +7,7 @@
*
* Copyright(c) 2007 - 2015 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
@@ -34,7 +34,7 @@
*
* Copyright(c) 2005 - 2015 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -201,7 +201,7 @@
/* PCI registers */
#define PCI_CFG_RETRY_TIMEOUT 0x041
-static void iwl_pcie_apm_config(struct iwl_trans *trans)
+void iwl_pcie_apm_config(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
u16 lctl;
@@ -448,7 +448,7 @@
~SHR_APMG_XTAL_CFG_XTAL_ON_REQ);
}
-static int iwl_pcie_apm_stop_master(struct iwl_trans *trans)
+int iwl_pcie_apm_stop_master(struct iwl_trans *trans)
{
int ret = 0;
@@ -567,7 +567,7 @@
}
/* Note: returns standard 0/-ERROR code */
-static int iwl_pcie_prepare_card_hw(struct iwl_trans *trans)
+int iwl_pcie_prepare_card_hw(struct iwl_trans *trans)
{
int ret;
int t = 0;
@@ -636,29 +636,6 @@
FH_TCSR_TX_CONFIG_REG_VAL_CIRQ_HOST_ENDTFD);
}
-static void iwl_pcie_load_firmware_chunk_tfh(struct iwl_trans *trans,
- u32 dst_addr, dma_addr_t phy_addr,
- u32 byte_cnt)
-{
- /* Stop DMA channel */
- iwl_write32(trans, TFH_SRV_DMA_CHNL0_CTRL, 0);
-
- /* Configure SRAM address */
- iwl_write32(trans, TFH_SRV_DMA_CHNL0_SRAM_ADDR,
- dst_addr);
-
- /* Configure DRAM address - 64 bit */
- iwl_write64(trans, TFH_SRV_DMA_CHNL0_DRAM_ADDR, phy_addr);
-
- /* Configure byte count to transfer */
- iwl_write32(trans, TFH_SRV_DMA_CHNL0_BC, byte_cnt);
-
- /* Enable the DRAM2SRAM to start */
- iwl_write32(trans, TFH_SRV_DMA_CHNL0_CTRL, TFH_SRV_DMA_SNOOP |
- TFH_SRV_DMA_TO_DRIVER |
- TFH_SRV_DMA_START);
-}
-
static int iwl_pcie_load_firmware_chunk(struct iwl_trans *trans,
u32 dst_addr, dma_addr_t phy_addr,
u32 byte_cnt)
@@ -672,12 +649,8 @@
if (!iwl_trans_grab_nic_access(trans, &flags))
return -EIO;
- if (trans->cfg->use_tfh)
- iwl_pcie_load_firmware_chunk_tfh(trans, dst_addr, phy_addr,
- byte_cnt);
- else
- iwl_pcie_load_firmware_chunk_fh(trans, dst_addr, phy_addr,
- byte_cnt);
+ iwl_pcie_load_firmware_chunk_fh(trans, dst_addr, phy_addr,
+ byte_cnt);
iwl_trans_release_nic_access(trans, &flags);
ret = wait_event_timeout(trans_pcie->ucode_write_waitq,
@@ -747,47 +720,6 @@
return ret;
}
-/*
- * Driver Takes the ownership on secure machine before FW load
- * and prevent race with the BT load.
- * W/A for ROM bug. (should be remove in the next Si step)
- */
-static int iwl_pcie_rsa_race_bug_wa(struct iwl_trans *trans)
-{
- u32 val, loop = 1000;
-
- /*
- * Check the RSA semaphore is accessible.
- * If the HW isn't locked and the rsa semaphore isn't accessible,
- * we are in trouble.
- */
- val = iwl_read_prph(trans, PREG_AUX_BUS_WPROT_0);
- if (val & (BIT(1) | BIT(17))) {
- IWL_DEBUG_INFO(trans,
- "can't access the RSA semaphore it is write protected\n");
- return 0;
- }
-
- /* take ownership on the AUX IF */
- iwl_write_prph(trans, WFPM_CTRL_REG, WFPM_AUX_CTL_AUX_IF_MAC_OWNER_MSK);
- iwl_write_prph(trans, AUX_MISC_MASTER1_EN, AUX_MISC_MASTER1_EN_SBE_MSK);
-
- do {
- iwl_write_prph(trans, AUX_MISC_MASTER1_SMPHR_STATUS, 0x1);
- val = iwl_read_prph(trans, AUX_MISC_MASTER1_SMPHR_STATUS);
- if (val == 0x1) {
- iwl_write_prph(trans, RSA_ENABLE, 0);
- return 0;
- }
-
- udelay(10);
- loop--;
- } while (loop > 0);
-
- IWL_ERR(trans, "Failed to take ownership on secure machine\n");
- return -EIO;
-}
-
static int iwl_pcie_load_cpu_sections_8000(struct iwl_trans *trans,
const struct fw_img *image,
int cpu,
@@ -828,15 +760,10 @@
return ret;
/* Notify ucode of loaded section number and status */
- if (trans->cfg->use_tfh) {
- val = iwl_read_prph(trans, UREG_UCODE_LOAD_STATUS);
- val = val | (sec_num << shift_param);
- iwl_write_prph(trans, UREG_UCODE_LOAD_STATUS, val);
- } else {
- val = iwl_read_direct32(trans, FH_UCODE_LOAD_STATUS);
- val = val | (sec_num << shift_param);
- iwl_write_direct32(trans, FH_UCODE_LOAD_STATUS, val);
- }
+ val = iwl_read_direct32(trans, FH_UCODE_LOAD_STATUS);
+ val = val | (sec_num << shift_param);
+ iwl_write_direct32(trans, FH_UCODE_LOAD_STATUS, val);
+
sec_num = (sec_num << 1) | 0x1;
}
@@ -904,7 +831,7 @@
return 0;
}
-static void iwl_pcie_apply_destination(struct iwl_trans *trans)
+void iwl_pcie_apply_destination(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
const struct iwl_fw_dbg_dest_tlv *dest = trans->dbg_dest_tlv;
@@ -1042,10 +969,15 @@
if (trans->dbg_dest_tlv)
iwl_pcie_apply_destination(trans);
- /* TODO: remove in the next Si step */
- ret = iwl_pcie_rsa_race_bug_wa(trans);
- if (ret)
- return ret;
+ IWL_DEBUG_POWER(trans, "Original WFPM value = 0x%08X\n",
+ iwl_read_prph(trans, WFPM_GP2));
+
+ /*
+ * Set default value. On resume reading the values that were
+ * zeored can provide debug data on the resume flow.
+ * This is for debugging only and has no functional impact.
+ */
+ iwl_write_prph(trans, WFPM_GP2, 0x01010101);
/* configure the ucode to be ready to get the secured image */
/* release CPU reset */
@@ -1062,7 +994,7 @@
&first_ucode_section);
}
-static bool iwl_trans_check_hw_rf_kill(struct iwl_trans *trans)
+bool iwl_trans_check_hw_rf_kill(struct iwl_trans *trans)
{
bool hw_rfkill = iwl_is_rfkill_set(trans);
@@ -1147,7 +1079,7 @@
iwl_write8(trans, CSR_MSIX_RX_IVAR(1), val);
}
-static void iwl_pcie_conf_msix_hw(struct iwl_trans_pcie *trans_pcie)
+void iwl_pcie_conf_msix_hw(struct iwl_trans_pcie *trans_pcie)
{
struct iwl_trans *trans = trans_pcie->trans;
@@ -1299,7 +1231,7 @@
iwl_pcie_prepare_card_hw(trans);
}
-static void iwl_pcie_synchronize_irqs(struct iwl_trans *trans)
+void iwl_pcie_synchronize_irqs(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
@@ -1423,8 +1355,12 @@
lockdep_assert_held(&trans_pcie->mutex);
- if (iwl_op_mode_hw_rf_kill(trans->op_mode, state))
- _iwl_trans_pcie_stop_device(trans, true);
+ if (iwl_op_mode_hw_rf_kill(trans->op_mode, state)) {
+ if (trans->cfg->gen2)
+ _iwl_trans_pcie_gen2_stop_device(trans, true);
+ else
+ _iwl_trans_pcie_stop_device(trans, true);
+ }
}
static void iwl_trans_pcie_d3_suspend(struct iwl_trans *trans, bool test,
@@ -1527,6 +1463,9 @@
}
}
+ IWL_DEBUG_POWER(trans, "WFPM value upon resume = 0x%08X\n",
+ iwl_read_prph(trans, WFPM_GP2));
+
val = iwl_read32(trans, CSR_RESET);
if (val & CSR_RESET_REG_FLAG_NEVO_RESET)
*status = IWL_D3_STATUS_RESET;
@@ -1828,7 +1767,10 @@
iwl_pcie_synchronize_irqs(trans);
- iwl_pcie_tx_free(trans);
+ if (trans->cfg->gen2)
+ iwl_pcie_gen2_tx_free(trans);
+ else
+ iwl_pcie_tx_free(trans);
iwl_pcie_rx_free(trans);
if (trans_pcie->msix_enabled) {
@@ -1998,7 +1940,7 @@
int queue;
for_each_set_bit(queue, &txqs, BITS_PER_LONG) {
- struct iwl_txq *txq = &trans_pcie->txq[queue];
+ struct iwl_txq *txq = trans_pcie->txq[queue];
unsigned long now;
spin_lock_bh(&txq->lock);
@@ -2050,7 +1992,7 @@
int i;
for (i = 0; i < trans->cfg->base_params->num_of_queues; i++) {
- struct iwl_txq *txq = &trans_pcie->txq[i];
+ struct iwl_txq *txq = trans_pcie->txq[i];
if (i == trans_pcie->cmd_queue)
continue;
@@ -2075,48 +2017,32 @@
void iwl_trans_pcie_log_scd_error(struct iwl_trans *trans, struct iwl_txq *txq)
{
- struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- u32 scd_sram_addr;
- u8 buf[16];
- int cnt;
+ u32 txq_id = txq->id;
+ u32 status;
+ bool active;
+ u8 fifo;
- IWL_ERR(trans, "Current SW read_ptr %d write_ptr %d\n",
- txq->read_ptr, txq->write_ptr);
-
- if (trans->cfg->use_tfh)
+ if (trans->cfg->use_tfh) {
+ IWL_ERR(trans, "Queue %d is stuck %d %d\n", txq_id,
+ txq->read_ptr, txq->write_ptr);
/* TODO: access new SCD registers and dump them */
return;
-
- scd_sram_addr = trans_pcie->scd_base_addr +
- SCD_TX_STTS_QUEUE_OFFSET(txq->id);
- iwl_trans_read_mem_bytes(trans, scd_sram_addr, buf, sizeof(buf));
-
- iwl_print_hex_error(trans, buf, sizeof(buf));
-
- for (cnt = 0; cnt < FH_TCSR_CHNL_NUM; cnt++)
- IWL_ERR(trans, "FH TRBs(%d) = 0x%08x\n", cnt,
- iwl_read_direct32(trans, FH_TX_TRB_REG(cnt)));
-
- for (cnt = 0; cnt < trans->cfg->base_params->num_of_queues; cnt++) {
- u32 status = iwl_read_prph(trans, SCD_QUEUE_STATUS_BITS(cnt));
- u8 fifo = (status >> SCD_QUEUE_STTS_REG_POS_TXF) & 0x7;
- bool active = !!(status & BIT(SCD_QUEUE_STTS_REG_POS_ACTIVE));
- u32 tbl_dw =
- iwl_trans_read_mem32(trans, trans_pcie->scd_base_addr +
- SCD_TRANS_TBL_OFFSET_QUEUE(cnt));
-
- if (cnt & 0x1)
- tbl_dw = (tbl_dw & 0xFFFF0000) >> 16;
- else
- tbl_dw = tbl_dw & 0x0000FFFF;
-
- IWL_ERR(trans,
- "Q %d is %sactive and mapped to fifo %d ra_tid 0x%04x [%d,%d]\n",
- cnt, active ? "" : "in", fifo, tbl_dw,
- iwl_read_prph(trans, SCD_QUEUE_RDPTR(cnt)) &
- (TFD_QUEUE_SIZE_MAX - 1),
- iwl_read_prph(trans, SCD_QUEUE_WRPTR(cnt)));
}
+
+ status = iwl_read_prph(trans, SCD_QUEUE_STATUS_BITS(txq_id));
+ fifo = (status >> SCD_QUEUE_STTS_REG_POS_TXF) & 0x7;
+ active = !!(status & BIT(SCD_QUEUE_STTS_REG_POS_ACTIVE));
+
+ IWL_ERR(trans,
+ "Queue %d is %sactive on fifo %d and stuck for %u ms. SW [%d, %d] HW [%d, %d] FH TRB=0x0%x\n",
+ txq_id, active ? "" : "in", fifo,
+ jiffies_to_msecs(txq->wd_timeout),
+ txq->read_ptr, txq->write_ptr,
+ iwl_read_prph(trans, SCD_QUEUE_RDPTR(txq_id)) &
+ (TFD_QUEUE_SIZE_MAX - 1),
+ iwl_read_prph(trans, SCD_QUEUE_WRPTR(txq_id)) &
+ (TFD_QUEUE_SIZE_MAX - 1),
+ iwl_read_direct32(trans, FH_TX_TRB_REG(fifo)));
}
static int iwl_trans_pcie_wait_txq_empty(struct iwl_trans *trans, u32 txq_bm)
@@ -2139,7 +2065,7 @@
continue;
IWL_DEBUG_TX_QUEUES(trans, "Emptying queue %d...\n", cnt);
- txq = &trans_pcie->txq[cnt];
+ txq = trans_pcie->txq[cnt];
wr_ptr = ACCESS_ONCE(txq->write_ptr);
while (txq->read_ptr != ACCESS_ONCE(txq->write_ptr) &&
@@ -2330,7 +2256,7 @@
bufsz = sizeof(char) * 75 * trans->cfg->base_params->num_of_queues;
- if (!trans_pcie->txq)
+ if (!trans_pcie->txq_memory)
return -EAGAIN;
buf = kzalloc(bufsz, GFP_KERNEL);
@@ -2338,7 +2264,7 @@
return -ENOMEM;
for (cnt = 0; cnt < trans->cfg->base_params->num_of_queues; cnt++) {
- txq = &trans_pcie->txq[cnt];
+ txq = trans_pcie->txq[cnt];
pos += scnprintf(buf + pos, bufsz - pos,
"hwq %.2d: read=%u write=%u use=%d stop=%d need_update=%d frozen=%d%s\n",
cnt, txq->read_ptr, txq->write_ptr,
@@ -2755,7 +2681,7 @@
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
struct iwl_fw_error_dump_data *data;
- struct iwl_txq *cmdq = &trans_pcie->txq[trans_pcie->cmd_queue];
+ struct iwl_txq *cmdq = trans_pcie->txq[trans_pcie->cmd_queue];
struct iwl_fw_error_dump_txcmd *txcmd;
struct iwl_trans_dump_data *dump_data;
u32 len, num_rbs;
@@ -2890,21 +2816,43 @@
}
#endif /* CONFIG_PM_SLEEP */
+#define IWL_TRANS_COMMON_OPS \
+ .op_mode_leave = iwl_trans_pcie_op_mode_leave, \
+ .write8 = iwl_trans_pcie_write8, \
+ .write32 = iwl_trans_pcie_write32, \
+ .read32 = iwl_trans_pcie_read32, \
+ .read_prph = iwl_trans_pcie_read_prph, \
+ .write_prph = iwl_trans_pcie_write_prph, \
+ .read_mem = iwl_trans_pcie_read_mem, \
+ .write_mem = iwl_trans_pcie_write_mem, \
+ .configure = iwl_trans_pcie_configure, \
+ .set_pmi = iwl_trans_pcie_set_pmi, \
+ .grab_nic_access = iwl_trans_pcie_grab_nic_access, \
+ .release_nic_access = iwl_trans_pcie_release_nic_access, \
+ .set_bits_mask = iwl_trans_pcie_set_bits_mask, \
+ .ref = iwl_trans_pcie_ref, \
+ .unref = iwl_trans_pcie_unref, \
+ .dump_data = iwl_trans_pcie_dump_data, \
+ .wait_tx_queues_empty = iwl_trans_pcie_wait_txq_empty, \
+ .d3_suspend = iwl_trans_pcie_d3_suspend, \
+ .d3_resume = iwl_trans_pcie_d3_resume
+
+#ifdef CONFIG_PM_SLEEP
+#define IWL_TRANS_PM_OPS \
+ .suspend = iwl_trans_pcie_suspend, \
+ .resume = iwl_trans_pcie_resume,
+#else
+#define IWL_TRANS_PM_OPS
+#endif /* CONFIG_PM_SLEEP */
+
static const struct iwl_trans_ops trans_ops_pcie = {
+ IWL_TRANS_COMMON_OPS,
+ IWL_TRANS_PM_OPS
.start_hw = iwl_trans_pcie_start_hw,
- .op_mode_leave = iwl_trans_pcie_op_mode_leave,
.fw_alive = iwl_trans_pcie_fw_alive,
.start_fw = iwl_trans_pcie_start_fw,
.stop_device = iwl_trans_pcie_stop_device,
- .d3_suspend = iwl_trans_pcie_d3_suspend,
- .d3_resume = iwl_trans_pcie_d3_resume,
-
-#ifdef CONFIG_PM_SLEEP
- .suspend = iwl_trans_pcie_suspend,
- .resume = iwl_trans_pcie_resume,
-#endif /* CONFIG_PM_SLEEP */
-
.send_cmd = iwl_trans_pcie_send_hcmd,
.tx = iwl_trans_pcie_tx,
@@ -2913,31 +2861,27 @@
.txq_disable = iwl_trans_pcie_txq_disable,
.txq_enable = iwl_trans_pcie_txq_enable,
- .get_txq_byte_table = iwl_trans_pcie_get_txq_byte_table,
-
.txq_set_shared_mode = iwl_trans_pcie_txq_set_shared_mode,
- .wait_tx_queue_empty = iwl_trans_pcie_wait_txq_empty,
.freeze_txq_timer = iwl_trans_pcie_freeze_txq_timer,
.block_txq_ptrs = iwl_trans_pcie_block_txq_ptrs,
+};
- .write8 = iwl_trans_pcie_write8,
- .write32 = iwl_trans_pcie_write32,
- .read32 = iwl_trans_pcie_read32,
- .read_prph = iwl_trans_pcie_read_prph,
- .write_prph = iwl_trans_pcie_write_prph,
- .read_mem = iwl_trans_pcie_read_mem,
- .write_mem = iwl_trans_pcie_write_mem,
- .configure = iwl_trans_pcie_configure,
- .set_pmi = iwl_trans_pcie_set_pmi,
- .grab_nic_access = iwl_trans_pcie_grab_nic_access,
- .release_nic_access = iwl_trans_pcie_release_nic_access,
- .set_bits_mask = iwl_trans_pcie_set_bits_mask,
+static const struct iwl_trans_ops trans_ops_pcie_gen2 = {
+ IWL_TRANS_COMMON_OPS,
+ IWL_TRANS_PM_OPS
+ .start_hw = iwl_trans_pcie_start_hw,
+ .fw_alive = iwl_trans_pcie_gen2_fw_alive,
+ .start_fw = iwl_trans_pcie_gen2_start_fw,
+ .stop_device = iwl_trans_pcie_gen2_stop_device,
- .ref = iwl_trans_pcie_ref,
- .unref = iwl_trans_pcie_unref,
+ .send_cmd = iwl_trans_pcie_gen2_send_hcmd,
- .dump_data = iwl_trans_pcie_dump_data,
+ .tx = iwl_trans_pcie_gen2_tx,
+ .reclaim = iwl_trans_pcie_reclaim,
+
+ .txq_alloc = iwl_trans_pcie_dyn_txq_alloc,
+ .txq_free = iwl_trans_pcie_dyn_txq_free,
};
struct iwl_trans *iwl_trans_pcie_alloc(struct pci_dev *pdev,
@@ -2952,8 +2896,12 @@
if (ret)
return ERR_PTR(ret);
- trans = iwl_trans_alloc(sizeof(struct iwl_trans_pcie),
- &pdev->dev, cfg, &trans_ops_pcie, 0);
+ if (cfg->gen2)
+ trans = iwl_trans_alloc(sizeof(struct iwl_trans_pcie),
+ &pdev->dev, cfg, &trans_ops_pcie_gen2);
+ else
+ trans = iwl_trans_alloc(sizeof(struct iwl_trans_pcie),
+ &pdev->dev, cfg, &trans_ops_pcie);
if (!trans)
return ERR_PTR(-ENOMEM);
@@ -3028,7 +2976,6 @@
* PCI Tx retries from interfering with C3 CPU state */
pci_write_config_byte(pdev, PCI_CFG_RETRY_TIMEOUT, 0x00);
- trans->dev = &pdev->dev;
trans_pcie->pci_dev = pdev;
iwl_disable_interrupts(trans);
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/tx-gen2.c b/drivers/net/wireless/intel/iwlwifi/pcie/tx-gen2.c
new file mode 100644
index 0000000..9fb46a6f
--- /dev/null
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/tx-gen2.c
@@ -0,0 +1,1018 @@
+/******************************************************************************
+ *
+ * This file is provided under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2017 Intel Deutschland GmbH
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2017 Intel Deutschland GmbH
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ *****************************************************************************/
+#include <linux/pm_runtime.h>
+
+#include "iwl-debug.h"
+#include "iwl-csr.h"
+#include "iwl-io.h"
+#include "internal.h"
+#include "mvm/fw-api.h"
+
+ /*
+ * iwl_pcie_gen2_tx_stop - Stop all Tx DMA channels
+ */
+void iwl_pcie_gen2_tx_stop(struct iwl_trans *trans)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ int txq_id;
+
+ /*
+ * This function can be called before the op_mode disabled the
+ * queues. This happens when we have an rfkill interrupt.
+ * Since we stop Tx altogether - mark the queues as stopped.
+ */
+ memset(trans_pcie->queue_stopped, 0, sizeof(trans_pcie->queue_stopped));
+ memset(trans_pcie->queue_used, 0, sizeof(trans_pcie->queue_used));
+
+ /* Unmap DMA from host system and free skb's */
+ for (txq_id = 0; txq_id < ARRAY_SIZE(trans_pcie->txq); txq_id++) {
+ if (!trans_pcie->txq[txq_id])
+ continue;
+ iwl_pcie_gen2_txq_unmap(trans, txq_id);
+ }
+}
+
+/*
+ * iwl_pcie_txq_update_byte_tbl - Set up entry in Tx byte-count array
+ */
+static void iwl_pcie_gen2_update_byte_tbl(struct iwl_txq *txq, u16 byte_cnt,
+ int num_tbs)
+{
+ struct iwlagn_scd_bc_tbl *scd_bc_tbl = txq->bc_tbl.addr;
+ int write_ptr = txq->write_ptr;
+ u8 filled_tfd_size, num_fetch_chunks;
+ u16 len = byte_cnt;
+ __le16 bc_ent;
+
+ len = DIV_ROUND_UP(len, 4);
+
+ if (WARN_ON(len > 0xFFF || write_ptr >= TFD_QUEUE_SIZE_MAX))
+ return;
+
+ filled_tfd_size = offsetof(struct iwl_tfh_tfd, tbs) +
+ num_tbs * sizeof(struct iwl_tfh_tb);
+ /*
+ * filled_tfd_size contains the number of filled bytes in the TFD.
+ * Dividing it by 64 will give the number of chunks to fetch
+ * to SRAM- 0 for one chunk, 1 for 2 and so on.
+ * If, for example, TFD contains only 3 TBs then 32 bytes
+ * of the TFD are used, and only one chunk of 64 bytes should
+ * be fetched
+ */
+ num_fetch_chunks = DIV_ROUND_UP(filled_tfd_size, 64) - 1;
+
+ bc_ent = cpu_to_le16(len | (num_fetch_chunks << 12));
+ scd_bc_tbl->tfd_offset[write_ptr] = bc_ent;
+}
+
+/*
+ * iwl_pcie_gen2_txq_inc_wr_ptr - Send new write index to hardware
+ */
+static void iwl_pcie_gen2_txq_inc_wr_ptr(struct iwl_trans *trans,
+ struct iwl_txq *txq)
+{
+ lockdep_assert_held(&txq->lock);
+
+ IWL_DEBUG_TX(trans, "Q:%d WR: 0x%x\n", txq->id, txq->write_ptr);
+
+ /*
+ * if not in power-save mode, uCode will never sleep when we're
+ * trying to tx (during RFKILL, we're not trying to tx).
+ */
+ iwl_write32(trans, HBUS_TARG_WRPTR, txq->write_ptr | (txq->id << 16));
+}
+
+static u8 iwl_pcie_gen2_get_num_tbs(struct iwl_trans *trans,
+ struct iwl_tfh_tfd *tfd)
+{
+ return le16_to_cpu(tfd->num_tbs) & 0x1f;
+}
+
+static void iwl_pcie_gen2_tfd_unmap(struct iwl_trans *trans,
+ struct iwl_cmd_meta *meta,
+ struct iwl_tfh_tfd *tfd)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ int i, num_tbs;
+
+ /* Sanity check on number of chunks */
+ num_tbs = iwl_pcie_gen2_get_num_tbs(trans, tfd);
+
+ if (num_tbs >= trans_pcie->max_tbs) {
+ IWL_ERR(trans, "Too many chunks: %i\n", num_tbs);
+ return;
+ }
+
+ /* first TB is never freed - it's the bidirectional DMA data */
+ for (i = 1; i < num_tbs; i++) {
+ if (meta->tbs & BIT(i))
+ dma_unmap_page(trans->dev,
+ le64_to_cpu(tfd->tbs[i].addr),
+ le16_to_cpu(tfd->tbs[i].tb_len),
+ DMA_TO_DEVICE);
+ else
+ dma_unmap_single(trans->dev,
+ le64_to_cpu(tfd->tbs[i].addr),
+ le16_to_cpu(tfd->tbs[i].tb_len),
+ DMA_TO_DEVICE);
+ }
+
+ tfd->num_tbs = 0;
+}
+
+static void iwl_pcie_gen2_free_tfd(struct iwl_trans *trans, struct iwl_txq *txq)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+
+ /* rd_ptr is bounded by TFD_QUEUE_SIZE_MAX and
+ * idx is bounded by n_window
+ */
+ int rd_ptr = txq->read_ptr;
+ int idx = get_cmd_index(txq, rd_ptr);
+
+ lockdep_assert_held(&txq->lock);
+
+ /* We have only q->n_window txq->entries, but we use
+ * TFD_QUEUE_SIZE_MAX tfds
+ */
+ iwl_pcie_gen2_tfd_unmap(trans, &txq->entries[idx].meta,
+ iwl_pcie_get_tfd(trans_pcie, txq, rd_ptr));
+
+ /* free SKB */
+ if (txq->entries) {
+ struct sk_buff *skb;
+
+ skb = txq->entries[idx].skb;
+
+ /* Can be called from irqs-disabled context
+ * If skb is not NULL, it means that the whole queue is being
+ * freed and that the queue is not empty - free the skb
+ */
+ if (skb) {
+ iwl_op_mode_free_skb(trans->op_mode, skb);
+ txq->entries[idx].skb = NULL;
+ }
+ }
+}
+
+static int iwl_pcie_gen2_set_tb(struct iwl_trans *trans,
+ struct iwl_tfh_tfd *tfd, dma_addr_t addr,
+ u16 len)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ int idx = iwl_pcie_gen2_get_num_tbs(trans, tfd);
+ struct iwl_tfh_tb *tb = &tfd->tbs[idx];
+
+ /* Each TFD can point to a maximum max_tbs Tx buffers */
+ if (le16_to_cpu(tfd->num_tbs) >= trans_pcie->max_tbs) {
+ IWL_ERR(trans, "Error can not send more than %d chunks\n",
+ trans_pcie->max_tbs);
+ return -EINVAL;
+ }
+
+ put_unaligned_le64(addr, &tb->addr);
+ tb->tb_len = cpu_to_le16(len);
+
+ tfd->num_tbs = cpu_to_le16(idx + 1);
+
+ return idx;
+}
+
+static
+struct iwl_tfh_tfd *iwl_pcie_gen2_build_tfd(struct iwl_trans *trans,
+ struct iwl_txq *txq,
+ struct iwl_device_cmd *dev_cmd,
+ struct sk_buff *skb,
+ struct iwl_cmd_meta *out_meta)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data;
+ struct iwl_tfh_tfd *tfd =
+ iwl_pcie_get_tfd(trans_pcie, txq, txq->write_ptr);
+ dma_addr_t tb_phys;
+ int i, len, tb1_len, tb2_len, hdr_len;
+ void *tb1_addr;
+
+ memset(tfd, 0, sizeof(*tfd));
+
+ tb_phys = iwl_pcie_get_first_tb_dma(txq, txq->write_ptr);
+ /* The first TB points to bi-directional DMA data */
+ memcpy(&txq->first_tb_bufs[txq->write_ptr], &dev_cmd->hdr,
+ IWL_FIRST_TB_SIZE);
+
+ iwl_pcie_gen2_set_tb(trans, tfd, tb_phys, IWL_FIRST_TB_SIZE);
+
+ /* there must be data left over for TB1 or this code must be changed */
+ BUILD_BUG_ON(sizeof(struct iwl_tx_cmd_gen2) < IWL_FIRST_TB_SIZE);
+
+ /*
+ * The second TB (tb1) points to the remainder of the TX command
+ * and the 802.11 header - dword aligned size
+ * (This calculation modifies the TX command, so do it before the
+ * setup of the first TB)
+ */
+ len = sizeof(struct iwl_tx_cmd_gen2) + sizeof(struct iwl_cmd_header) +
+ ieee80211_hdrlen(hdr->frame_control) - IWL_FIRST_TB_SIZE;
+
+ tb1_len = ALIGN(len, 4);
+
+ /* map the data for TB1 */
+ tb1_addr = ((u8 *)&dev_cmd->hdr) + IWL_FIRST_TB_SIZE;
+ tb_phys = dma_map_single(trans->dev, tb1_addr, tb1_len, DMA_TO_DEVICE);
+ if (unlikely(dma_mapping_error(trans->dev, tb_phys)))
+ goto out_err;
+ iwl_pcie_gen2_set_tb(trans, tfd, tb_phys, tb1_len);
+
+ /* set up TFD's third entry to point to remainder of skb's head */
+ hdr_len = ieee80211_hdrlen(hdr->frame_control);
+ tb2_len = skb_headlen(skb) - hdr_len;
+
+ if (tb2_len > 0) {
+ tb_phys = dma_map_single(trans->dev, skb->data + hdr_len,
+ tb2_len, DMA_TO_DEVICE);
+ if (unlikely(dma_mapping_error(trans->dev, tb_phys)))
+ goto out_err;
+ iwl_pcie_gen2_set_tb(trans, tfd, tb_phys, tb2_len);
+ }
+
+ /* set up the remaining entries to point to the data */
+ for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
+ const skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
+ int tb_idx;
+
+ if (!skb_frag_size(frag))
+ continue;
+
+ tb_phys = skb_frag_dma_map(trans->dev, frag, 0,
+ skb_frag_size(frag), DMA_TO_DEVICE);
+
+ if (unlikely(dma_mapping_error(trans->dev, tb_phys)))
+ goto out_err;
+ tb_idx = iwl_pcie_gen2_set_tb(trans, tfd, tb_phys,
+ skb_frag_size(frag));
+
+ out_meta->tbs |= BIT(tb_idx);
+ }
+
+ trace_iwlwifi_dev_tx(trans->dev, skb, tfd, sizeof(*tfd), &dev_cmd->hdr,
+ IWL_FIRST_TB_SIZE + tb1_len,
+ skb->data + hdr_len, tb2_len);
+ trace_iwlwifi_dev_tx_data(trans->dev, skb, hdr_len,
+ skb->len - hdr_len);
+
+ return tfd;
+
+out_err:
+ iwl_pcie_gen2_tfd_unmap(trans, out_meta, tfd);
+ return NULL;
+}
+
+int iwl_trans_pcie_gen2_tx(struct iwl_trans *trans, struct sk_buff *skb,
+ struct iwl_device_cmd *dev_cmd, int txq_id)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct iwl_tx_cmd_gen2 *tx_cmd = (void *)dev_cmd->payload;
+ struct iwl_cmd_meta *out_meta;
+ struct iwl_txq *txq = trans_pcie->txq[txq_id];
+ void *tfd;
+
+ if (WARN_ONCE(!test_bit(txq_id, trans_pcie->queue_used),
+ "TX on unused queue %d\n", txq_id))
+ return -EINVAL;
+
+ if (skb_is_nonlinear(skb) &&
+ skb_shinfo(skb)->nr_frags > IWL_PCIE_MAX_FRAGS(trans_pcie) &&
+ __skb_linearize(skb))
+ return -ENOMEM;
+
+ spin_lock(&txq->lock);
+
+ /* Set up driver data for this TFD */
+ txq->entries[txq->write_ptr].skb = skb;
+ txq->entries[txq->write_ptr].cmd = dev_cmd;
+
+ dev_cmd->hdr.sequence =
+ cpu_to_le16((u16)(QUEUE_TO_SEQ(txq_id) |
+ INDEX_TO_SEQ(txq->write_ptr)));
+
+ /* Set up first empty entry in queue's array of Tx/cmd buffers */
+ out_meta = &txq->entries[txq->write_ptr].meta;
+ out_meta->flags = 0;
+
+ tfd = iwl_pcie_gen2_build_tfd(trans, txq, dev_cmd, skb, out_meta);
+ if (!tfd) {
+ spin_unlock(&txq->lock);
+ return -1;
+ }
+
+ /* Set up entry for this TFD in Tx byte-count array */
+ iwl_pcie_gen2_update_byte_tbl(txq, le16_to_cpu(tx_cmd->len),
+ iwl_pcie_gen2_get_num_tbs(trans, tfd));
+
+ /* start timer if queue currently empty */
+ if (txq->read_ptr == txq->write_ptr) {
+ if (txq->wd_timeout)
+ mod_timer(&txq->stuck_timer, jiffies + txq->wd_timeout);
+ IWL_DEBUG_RPM(trans, "Q: %d first tx - take ref\n", txq->id);
+ iwl_trans_ref(trans);
+ }
+
+ /* Tell device the write index *just past* this latest filled TFD */
+ txq->write_ptr = iwl_queue_inc_wrap(txq->write_ptr);
+ iwl_pcie_gen2_txq_inc_wr_ptr(trans, txq);
+ if (iwl_queue_space(txq) < txq->high_mark)
+ iwl_stop_queue(trans, txq);
+
+ /*
+ * At this point the frame is "transmitted" successfully
+ * and we will get a TX status notification eventually.
+ */
+ spin_unlock(&txq->lock);
+ return 0;
+}
+
+/*************** HOST COMMAND QUEUE FUNCTIONS *****/
+
+/*
+ * iwl_pcie_gen2_enqueue_hcmd - enqueue a uCode command
+ * @priv: device private data point
+ * @cmd: a pointer to the ucode command structure
+ *
+ * The function returns < 0 values to indicate the operation
+ * failed. On success, it returns the index (>= 0) of command in the
+ * command queue.
+ */
+static int iwl_pcie_gen2_enqueue_hcmd(struct iwl_trans *trans,
+ struct iwl_host_cmd *cmd)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct iwl_txq *txq = trans_pcie->txq[trans_pcie->cmd_queue];
+ struct iwl_device_cmd *out_cmd;
+ struct iwl_cmd_meta *out_meta;
+ unsigned long flags;
+ void *dup_buf = NULL;
+ dma_addr_t phys_addr;
+ int idx, i, cmd_pos;
+ u16 copy_size, cmd_size, tb0_size;
+ bool had_nocopy = false;
+ u8 group_id = iwl_cmd_groupid(cmd->id);
+ const u8 *cmddata[IWL_MAX_CMD_TBS_PER_TFD];
+ u16 cmdlen[IWL_MAX_CMD_TBS_PER_TFD];
+ struct iwl_tfh_tfd *tfd =
+ iwl_pcie_get_tfd(trans_pcie, txq, txq->write_ptr);
+
+ memset(tfd, 0, sizeof(*tfd));
+
+ copy_size = sizeof(struct iwl_cmd_header_wide);
+ cmd_size = sizeof(struct iwl_cmd_header_wide);
+
+ for (i = 0; i < IWL_MAX_CMD_TBS_PER_TFD; i++) {
+ cmddata[i] = cmd->data[i];
+ cmdlen[i] = cmd->len[i];
+
+ if (!cmd->len[i])
+ continue;
+
+ /* need at least IWL_FIRST_TB_SIZE copied */
+ if (copy_size < IWL_FIRST_TB_SIZE) {
+ int copy = IWL_FIRST_TB_SIZE - copy_size;
+
+ if (copy > cmdlen[i])
+ copy = cmdlen[i];
+ cmdlen[i] -= copy;
+ cmddata[i] += copy;
+ copy_size += copy;
+ }
+
+ if (cmd->dataflags[i] & IWL_HCMD_DFL_NOCOPY) {
+ had_nocopy = true;
+ if (WARN_ON(cmd->dataflags[i] & IWL_HCMD_DFL_DUP)) {
+ idx = -EINVAL;
+ goto free_dup_buf;
+ }
+ } else if (cmd->dataflags[i] & IWL_HCMD_DFL_DUP) {
+ /*
+ * This is also a chunk that isn't copied
+ * to the static buffer so set had_nocopy.
+ */
+ had_nocopy = true;
+
+ /* only allowed once */
+ if (WARN_ON(dup_buf)) {
+ idx = -EINVAL;
+ goto free_dup_buf;
+ }
+
+ dup_buf = kmemdup(cmddata[i], cmdlen[i],
+ GFP_ATOMIC);
+ if (!dup_buf)
+ return -ENOMEM;
+ } else {
+ /* NOCOPY must not be followed by normal! */
+ if (WARN_ON(had_nocopy)) {
+ idx = -EINVAL;
+ goto free_dup_buf;
+ }
+ copy_size += cmdlen[i];
+ }
+ cmd_size += cmd->len[i];
+ }
+
+ /*
+ * If any of the command structures end up being larger than the
+ * TFD_MAX_PAYLOAD_SIZE and they aren't dynamically allocated into
+ * separate TFDs, then we will need to increase the size of the buffers
+ */
+ if (WARN(copy_size > TFD_MAX_PAYLOAD_SIZE,
+ "Command %s (%#x) is too large (%d bytes)\n",
+ iwl_get_cmd_string(trans, cmd->id), cmd->id, copy_size)) {
+ idx = -EINVAL;
+ goto free_dup_buf;
+ }
+
+ spin_lock_bh(&txq->lock);
+
+ if (iwl_queue_space(txq) < ((cmd->flags & CMD_ASYNC) ? 2 : 1)) {
+ spin_unlock_bh(&txq->lock);
+
+ IWL_ERR(trans, "No space in command queue\n");
+ iwl_op_mode_cmd_queue_full(trans->op_mode);
+ idx = -ENOSPC;
+ goto free_dup_buf;
+ }
+
+ idx = get_cmd_index(txq, txq->write_ptr);
+ out_cmd = txq->entries[idx].cmd;
+ out_meta = &txq->entries[idx].meta;
+
+ /* re-initialize to NULL */
+ memset(out_meta, 0, sizeof(*out_meta));
+ if (cmd->flags & CMD_WANT_SKB)
+ out_meta->source = cmd;
+
+ /* set up the header */
+ out_cmd->hdr_wide.cmd = iwl_cmd_opcode(cmd->id);
+ out_cmd->hdr_wide.group_id = group_id;
+ out_cmd->hdr_wide.version = iwl_cmd_version(cmd->id);
+ out_cmd->hdr_wide.length =
+ cpu_to_le16(cmd_size - sizeof(struct iwl_cmd_header_wide));
+ out_cmd->hdr_wide.reserved = 0;
+ out_cmd->hdr_wide.sequence =
+ cpu_to_le16(QUEUE_TO_SEQ(trans_pcie->cmd_queue) |
+ INDEX_TO_SEQ(txq->write_ptr));
+
+ cmd_pos = sizeof(struct iwl_cmd_header_wide);
+ copy_size = sizeof(struct iwl_cmd_header_wide);
+
+ /* and copy the data that needs to be copied */
+ for (i = 0; i < IWL_MAX_CMD_TBS_PER_TFD; i++) {
+ int copy;
+
+ if (!cmd->len[i])
+ continue;
+
+ /* copy everything if not nocopy/dup */
+ if (!(cmd->dataflags[i] & (IWL_HCMD_DFL_NOCOPY |
+ IWL_HCMD_DFL_DUP))) {
+ copy = cmd->len[i];
+
+ memcpy((u8 *)out_cmd + cmd_pos, cmd->data[i], copy);
+ cmd_pos += copy;
+ copy_size += copy;
+ continue;
+ }
+
+ /*
+ * Otherwise we need at least IWL_FIRST_TB_SIZE copied
+ * in total (for bi-directional DMA), but copy up to what
+ * we can fit into the payload for debug dump purposes.
+ */
+ copy = min_t(int, TFD_MAX_PAYLOAD_SIZE - cmd_pos, cmd->len[i]);
+
+ memcpy((u8 *)out_cmd + cmd_pos, cmd->data[i], copy);
+ cmd_pos += copy;
+
+ /* However, treat copy_size the proper way, we need it below */
+ if (copy_size < IWL_FIRST_TB_SIZE) {
+ copy = IWL_FIRST_TB_SIZE - copy_size;
+
+ if (copy > cmd->len[i])
+ copy = cmd->len[i];
+ copy_size += copy;
+ }
+ }
+
+ IWL_DEBUG_HC(trans,
+ "Sending command %s (%.2x.%.2x), seq: 0x%04X, %d bytes at %d[%d]:%d\n",
+ iwl_get_cmd_string(trans, cmd->id), group_id,
+ out_cmd->hdr.cmd, le16_to_cpu(out_cmd->hdr.sequence),
+ cmd_size, txq->write_ptr, idx, trans_pcie->cmd_queue);
+
+ /* start the TFD with the minimum copy bytes */
+ tb0_size = min_t(int, copy_size, IWL_FIRST_TB_SIZE);
+ memcpy(&txq->first_tb_bufs[idx], &out_cmd->hdr, tb0_size);
+ iwl_pcie_gen2_set_tb(trans, tfd, iwl_pcie_get_first_tb_dma(txq, idx),
+ tb0_size);
+
+ /* map first command fragment, if any remains */
+ if (copy_size > tb0_size) {
+ phys_addr = dma_map_single(trans->dev,
+ ((u8 *)&out_cmd->hdr) + tb0_size,
+ copy_size - tb0_size,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(trans->dev, phys_addr)) {
+ idx = -ENOMEM;
+ iwl_pcie_gen2_tfd_unmap(trans, out_meta, tfd);
+ goto out;
+ }
+ iwl_pcie_gen2_set_tb(trans, tfd, phys_addr,
+ copy_size - tb0_size);
+ }
+
+ /* map the remaining (adjusted) nocopy/dup fragments */
+ for (i = 0; i < IWL_MAX_CMD_TBS_PER_TFD; i++) {
+ const void *data = cmddata[i];
+
+ if (!cmdlen[i])
+ continue;
+ if (!(cmd->dataflags[i] & (IWL_HCMD_DFL_NOCOPY |
+ IWL_HCMD_DFL_DUP)))
+ continue;
+ if (cmd->dataflags[i] & IWL_HCMD_DFL_DUP)
+ data = dup_buf;
+ phys_addr = dma_map_single(trans->dev, (void *)data,
+ cmdlen[i], DMA_TO_DEVICE);
+ if (dma_mapping_error(trans->dev, phys_addr)) {
+ idx = -ENOMEM;
+ iwl_pcie_gen2_tfd_unmap(trans, out_meta, tfd);
+ goto out;
+ }
+ iwl_pcie_gen2_set_tb(trans, tfd, phys_addr, cmdlen[i]);
+ }
+
+ BUILD_BUG_ON(IWL_TFH_NUM_TBS > sizeof(out_meta->tbs) * BITS_PER_BYTE);
+ out_meta->flags = cmd->flags;
+ if (WARN_ON_ONCE(txq->entries[idx].free_buf))
+ kzfree(txq->entries[idx].free_buf);
+ txq->entries[idx].free_buf = dup_buf;
+
+ trace_iwlwifi_dev_hcmd(trans->dev, cmd, cmd_size, &out_cmd->hdr_wide);
+
+ /* start timer if queue currently empty */
+ if (txq->read_ptr == txq->write_ptr && txq->wd_timeout)
+ mod_timer(&txq->stuck_timer, jiffies + txq->wd_timeout);
+
+ spin_lock_irqsave(&trans_pcie->reg_lock, flags);
+ if (!(cmd->flags & CMD_SEND_IN_IDLE) &&
+ !trans_pcie->ref_cmd_in_flight) {
+ trans_pcie->ref_cmd_in_flight = true;
+ IWL_DEBUG_RPM(trans, "set ref_cmd_in_flight - ref\n");
+ iwl_trans_ref(trans);
+ }
+ /* Increment and update queue's write index */
+ txq->write_ptr = iwl_queue_inc_wrap(txq->write_ptr);
+ iwl_pcie_gen2_txq_inc_wr_ptr(trans, txq);
+ spin_unlock_irqrestore(&trans_pcie->reg_lock, flags);
+
+out:
+ spin_unlock_bh(&txq->lock);
+free_dup_buf:
+ if (idx < 0)
+ kfree(dup_buf);
+ return idx;
+}
+
+#define HOST_COMPLETE_TIMEOUT (2 * HZ)
+
+static int iwl_pcie_gen2_send_hcmd_sync(struct iwl_trans *trans,
+ struct iwl_host_cmd *cmd)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ const char *cmd_str = iwl_get_cmd_string(trans, cmd->id);
+ struct iwl_txq *txq = trans_pcie->txq[trans_pcie->cmd_queue];
+ int cmd_idx;
+ int ret;
+
+ IWL_DEBUG_INFO(trans, "Attempting to send sync command %s\n", cmd_str);
+
+ if (WARN(test_and_set_bit(STATUS_SYNC_HCMD_ACTIVE,
+ &trans->status),
+ "Command %s: a command is already active!\n", cmd_str))
+ return -EIO;
+
+ IWL_DEBUG_INFO(trans, "Setting HCMD_ACTIVE for command %s\n", cmd_str);
+
+ if (pm_runtime_suspended(&trans_pcie->pci_dev->dev)) {
+ ret = wait_event_timeout(trans_pcie->d0i3_waitq,
+ pm_runtime_active(&trans_pcie->pci_dev->dev),
+ msecs_to_jiffies(IWL_TRANS_IDLE_TIMEOUT));
+ if (!ret) {
+ IWL_ERR(trans, "Timeout exiting D0i3 before hcmd\n");
+ return -ETIMEDOUT;
+ }
+ }
+
+ cmd_idx = iwl_pcie_gen2_enqueue_hcmd(trans, cmd);
+ if (cmd_idx < 0) {
+ ret = cmd_idx;
+ clear_bit(STATUS_SYNC_HCMD_ACTIVE, &trans->status);
+ IWL_ERR(trans, "Error sending %s: enqueue_hcmd failed: %d\n",
+ cmd_str, ret);
+ return ret;
+ }
+
+ ret = wait_event_timeout(trans_pcie->wait_command_queue,
+ !test_bit(STATUS_SYNC_HCMD_ACTIVE,
+ &trans->status),
+ HOST_COMPLETE_TIMEOUT);
+ if (!ret) {
+ IWL_ERR(trans, "Error sending %s: time out after %dms.\n",
+ cmd_str, jiffies_to_msecs(HOST_COMPLETE_TIMEOUT));
+
+ IWL_ERR(trans, "Current CMD queue read_ptr %d write_ptr %d\n",
+ txq->read_ptr, txq->write_ptr);
+
+ clear_bit(STATUS_SYNC_HCMD_ACTIVE, &trans->status);
+ IWL_DEBUG_INFO(trans, "Clearing HCMD_ACTIVE for command %s\n",
+ cmd_str);
+ ret = -ETIMEDOUT;
+
+ iwl_force_nmi(trans);
+ iwl_trans_fw_error(trans);
+
+ goto cancel;
+ }
+
+ if (test_bit(STATUS_FW_ERROR, &trans->status)) {
+ IWL_ERR(trans, "FW error in SYNC CMD %s\n", cmd_str);
+ dump_stack();
+ ret = -EIO;
+ goto cancel;
+ }
+
+ if (!(cmd->flags & CMD_SEND_IN_RFKILL) &&
+ test_bit(STATUS_RFKILL, &trans->status)) {
+ IWL_DEBUG_RF_KILL(trans, "RFKILL in SYNC CMD... no rsp\n");
+ ret = -ERFKILL;
+ goto cancel;
+ }
+
+ if ((cmd->flags & CMD_WANT_SKB) && !cmd->resp_pkt) {
+ IWL_ERR(trans, "Error: Response NULL in '%s'\n", cmd_str);
+ ret = -EIO;
+ goto cancel;
+ }
+
+ return 0;
+
+cancel:
+ if (cmd->flags & CMD_WANT_SKB) {
+ /*
+ * Cancel the CMD_WANT_SKB flag for the cmd in the
+ * TX cmd queue. Otherwise in case the cmd comes
+ * in later, it will possibly set an invalid
+ * address (cmd->meta.source).
+ */
+ txq->entries[cmd_idx].meta.flags &= ~CMD_WANT_SKB;
+ }
+
+ if (cmd->resp_pkt) {
+ iwl_free_resp(cmd);
+ cmd->resp_pkt = NULL;
+ }
+
+ return ret;
+}
+
+int iwl_trans_pcie_gen2_send_hcmd(struct iwl_trans *trans,
+ struct iwl_host_cmd *cmd)
+{
+ if (!(cmd->flags & CMD_SEND_IN_RFKILL) &&
+ test_bit(STATUS_RFKILL, &trans->status)) {
+ IWL_DEBUG_RF_KILL(trans, "Dropping CMD 0x%x: RF KILL\n",
+ cmd->id);
+ return -ERFKILL;
+ }
+
+ if (cmd->flags & CMD_ASYNC) {
+ int ret;
+
+ /* An asynchronous command can not expect an SKB to be set. */
+ if (WARN_ON(cmd->flags & CMD_WANT_SKB))
+ return -EINVAL;
+
+ ret = iwl_pcie_gen2_enqueue_hcmd(trans, cmd);
+ if (ret < 0) {
+ IWL_ERR(trans,
+ "Error sending %s: enqueue_hcmd failed: %d\n",
+ iwl_get_cmd_string(trans, cmd->id), ret);
+ return ret;
+ }
+ return 0;
+ }
+
+ return iwl_pcie_gen2_send_hcmd_sync(trans, cmd);
+}
+
+/*
+ * iwl_pcie_gen2_txq_unmap - Unmap any remaining DMA mappings and free skb's
+ */
+void iwl_pcie_gen2_txq_unmap(struct iwl_trans *trans, int txq_id)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct iwl_txq *txq = trans_pcie->txq[txq_id];
+
+ spin_lock_bh(&txq->lock);
+ while (txq->write_ptr != txq->read_ptr) {
+ IWL_DEBUG_TX_REPLY(trans, "Q %d Free %d\n",
+ txq_id, txq->read_ptr);
+
+ iwl_pcie_gen2_free_tfd(trans, txq);
+ txq->read_ptr = iwl_queue_inc_wrap(txq->read_ptr);
+
+ if (txq->read_ptr == txq->write_ptr) {
+ unsigned long flags;
+
+ spin_lock_irqsave(&trans_pcie->reg_lock, flags);
+ if (txq_id != trans_pcie->cmd_queue) {
+ IWL_DEBUG_RPM(trans, "Q %d - last tx freed\n",
+ txq->id);
+ iwl_trans_unref(trans);
+ } else if (trans_pcie->ref_cmd_in_flight) {
+ trans_pcie->ref_cmd_in_flight = false;
+ IWL_DEBUG_RPM(trans,
+ "clear ref_cmd_in_flight\n");
+ iwl_trans_unref(trans);
+ }
+ spin_unlock_irqrestore(&trans_pcie->reg_lock, flags);
+ }
+ }
+ spin_unlock_bh(&txq->lock);
+
+ /* just in case - this queue may have been stopped */
+ iwl_wake_queue(trans, txq);
+}
+
+static void iwl_pcie_gen2_txq_free_memory(struct iwl_trans *trans,
+ struct iwl_txq *txq)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct device *dev = trans->dev;
+
+ /* De-alloc circular buffer of TFDs */
+ if (txq->tfds) {
+ dma_free_coherent(dev,
+ trans_pcie->tfd_size * TFD_QUEUE_SIZE_MAX,
+ txq->tfds, txq->dma_addr);
+ dma_free_coherent(dev,
+ sizeof(*txq->first_tb_bufs) * txq->n_window,
+ txq->first_tb_bufs, txq->first_tb_dma);
+ }
+
+ kfree(txq->entries);
+ iwl_pcie_free_dma_ptr(trans, &txq->bc_tbl);
+ kfree(txq);
+}
+
+/*
+ * iwl_pcie_txq_free - Deallocate DMA queue.
+ * @txq: Transmit queue to deallocate.
+ *
+ * Empty queue by removing and destroying all BD's.
+ * Free all buffers.
+ * 0-fill, but do not free "txq" descriptor structure.
+ */
+static void iwl_pcie_gen2_txq_free(struct iwl_trans *trans, int txq_id)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct iwl_txq *txq = trans_pcie->txq[txq_id];
+ int i;
+
+ if (WARN_ON(!txq))
+ return;
+
+ iwl_pcie_gen2_txq_unmap(trans, txq_id);
+
+ /* De-alloc array of command/tx buffers */
+ if (txq_id == trans_pcie->cmd_queue)
+ for (i = 0; i < txq->n_window; i++) {
+ kzfree(txq->entries[i].cmd);
+ kzfree(txq->entries[i].free_buf);
+ }
+ del_timer_sync(&txq->stuck_timer);
+
+ iwl_pcie_gen2_txq_free_memory(trans, txq);
+
+ trans_pcie->txq[txq_id] = NULL;
+
+ clear_bit(txq_id, trans_pcie->queue_used);
+}
+
+int iwl_trans_pcie_dyn_txq_alloc(struct iwl_trans *trans,
+ struct iwl_tx_queue_cfg_cmd *cmd,
+ int cmd_id,
+ unsigned int timeout)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct iwl_tx_queue_cfg_rsp *rsp;
+ struct iwl_txq *txq;
+ struct iwl_host_cmd hcmd = {
+ .id = cmd_id,
+ .len = { sizeof(*cmd) },
+ .data = { cmd, },
+ .flags = CMD_WANT_SKB,
+ };
+ int ret, qid;
+
+ txq = kzalloc(sizeof(*txq), GFP_KERNEL);
+ if (!txq)
+ return -ENOMEM;
+ ret = iwl_pcie_alloc_dma_ptr(trans, &txq->bc_tbl,
+ sizeof(struct iwlagn_scd_bc_tbl));
+ if (ret) {
+ IWL_ERR(trans, "Scheduler BC Table allocation failed\n");
+ kfree(txq);
+ return -ENOMEM;
+ }
+
+ ret = iwl_pcie_txq_alloc(trans, txq, TFD_TX_CMD_SLOTS, false);
+ if (ret) {
+ IWL_ERR(trans, "Tx queue alloc failed\n");
+ goto error;
+ }
+ ret = iwl_pcie_txq_init(trans, txq, TFD_TX_CMD_SLOTS, false);
+ if (ret) {
+ IWL_ERR(trans, "Tx queue init failed\n");
+ goto error;
+ }
+
+ txq->wd_timeout = msecs_to_jiffies(timeout);
+
+ cmd->tfdq_addr = cpu_to_le64(txq->dma_addr);
+ cmd->byte_cnt_addr = cpu_to_le64(txq->bc_tbl.dma);
+ cmd->cb_size = cpu_to_le32(TFD_QUEUE_CB_SIZE(TFD_QUEUE_SIZE_MAX));
+
+ ret = iwl_trans_send_cmd(trans, &hcmd);
+ if (ret)
+ goto error;
+
+ if (WARN_ON(iwl_rx_packet_payload_len(hcmd.resp_pkt) != sizeof(*rsp))) {
+ ret = -EINVAL;
+ goto error;
+ }
+
+ rsp = (void *)hcmd.resp_pkt->data;
+ qid = le16_to_cpu(rsp->queue_number);
+
+ if (qid > ARRAY_SIZE(trans_pcie->txq)) {
+ WARN_ONCE(1, "queue index %d unsupported", qid);
+ ret = -EIO;
+ goto error;
+ }
+
+ if (test_and_set_bit(qid, trans_pcie->queue_used)) {
+ WARN_ONCE(1, "queue %d already used", qid);
+ ret = -EIO;
+ goto error;
+ }
+
+ txq->id = qid;
+ trans_pcie->txq[qid] = txq;
+
+ /* Place first TFD at index corresponding to start sequence number */
+ txq->read_ptr = le16_to_cpu(rsp->write_pointer);
+ txq->write_ptr = le16_to_cpu(rsp->write_pointer);
+ iwl_write_direct32(trans, HBUS_TARG_WRPTR,
+ (txq->write_ptr) | (qid << 16));
+ IWL_DEBUG_TX_QUEUES(trans, "Activate queue %d\n", qid);
+
+ return qid;
+
+error:
+ iwl_pcie_gen2_txq_free_memory(trans, txq);
+ return ret;
+}
+
+void iwl_trans_pcie_dyn_txq_free(struct iwl_trans *trans, int queue)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+
+ /*
+ * Upon HW Rfkill - we stop the device, and then stop the queues
+ * in the op_mode. Just for the sake of the simplicity of the op_mode,
+ * allow the op_mode to call txq_disable after it already called
+ * stop_device.
+ */
+ if (!test_and_clear_bit(queue, trans_pcie->queue_used)) {
+ WARN_ONCE(test_bit(STATUS_DEVICE_ENABLED, &trans->status),
+ "queue %d not used", queue);
+ return;
+ }
+
+ iwl_pcie_gen2_txq_unmap(trans, queue);
+
+ IWL_DEBUG_TX_QUEUES(trans, "Deactivate queue %d\n", queue);
+}
+
+void iwl_pcie_gen2_tx_free(struct iwl_trans *trans)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ int i;
+
+ memset(trans_pcie->queue_used, 0, sizeof(trans_pcie->queue_used));
+
+ /* Free all TX queues */
+ for (i = 0; i < ARRAY_SIZE(trans_pcie->txq); i++) {
+ if (!trans_pcie->txq[i])
+ continue;
+
+ iwl_pcie_gen2_txq_free(trans, i);
+ }
+}
+
+int iwl_pcie_gen2_tx_init(struct iwl_trans *trans)
+{
+ struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct iwl_txq *cmd_queue;
+ int txq_id = trans_pcie->cmd_queue, ret;
+
+ /* alloc and init the command queue */
+ if (!trans_pcie->txq[txq_id]) {
+ cmd_queue = kzalloc(sizeof(*cmd_queue), GFP_KERNEL);
+ if (!cmd_queue) {
+ IWL_ERR(trans, "Not enough memory for command queue\n");
+ return -ENOMEM;
+ }
+ trans_pcie->txq[txq_id] = cmd_queue;
+ ret = iwl_pcie_txq_alloc(trans, cmd_queue, TFD_CMD_SLOTS, true);
+ if (ret) {
+ IWL_ERR(trans, "Tx %d queue init failed\n", txq_id);
+ goto error;
+ }
+ } else {
+ cmd_queue = trans_pcie->txq[txq_id];
+ }
+
+ ret = iwl_pcie_txq_init(trans, cmd_queue, TFD_CMD_SLOTS, true);
+ if (ret) {
+ IWL_ERR(trans, "Tx %d queue alloc failed\n", txq_id);
+ goto error;
+ }
+ trans_pcie->txq[txq_id]->id = txq_id;
+ set_bit(txq_id, trans_pcie->queue_used);
+
+ return 0;
+
+error:
+ iwl_pcie_gen2_tx_free(trans);
+ return ret;
+}
+
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/tx.c b/drivers/net/wireless/intel/iwlwifi/pcie/tx.c
index 911cf98..386950a 100644
--- a/drivers/net/wireless/intel/iwlwifi/pcie/tx.c
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/tx.c
@@ -2,7 +2,7 @@
*
* Copyright(c) 2003 - 2014 Intel Corporation. All rights reserved.
* Copyright(c) 2013 - 2015 Intel Mobile Communications GmbH
- * Copyright(c) 2016 Intel Deutschland GmbH
+ * Copyright(c) 2016 - 2017 Intel Deutschland GmbH
*
* Portions of this file are derived from the ipw3945 project, as well
* as portions of the ieee80211 subsystem header files.
@@ -71,7 +71,7 @@
*
***************************************************/
-static int iwl_queue_space(const struct iwl_txq *q)
+int iwl_queue_space(const struct iwl_txq *q)
{
unsigned int max;
unsigned int used;
@@ -102,10 +102,9 @@
/*
* iwl_queue_init - Initialize queue's high/low-water and read/write indexes
*/
-static int iwl_queue_init(struct iwl_txq *q, int slots_num, u32 id)
+static int iwl_queue_init(struct iwl_txq *q, int slots_num)
{
q->n_window = slots_num;
- q->id = id;
/* slots_num must be power-of-two size, otherwise
* get_cmd_index is broken. */
@@ -126,8 +125,8 @@
return 0;
}
-static int iwl_pcie_alloc_dma_ptr(struct iwl_trans *trans,
- struct iwl_dma_ptr *ptr, size_t size)
+int iwl_pcie_alloc_dma_ptr(struct iwl_trans *trans,
+ struct iwl_dma_ptr *ptr, size_t size)
{
if (WARN_ON(ptr->addr))
return -EINVAL;
@@ -140,8 +139,7 @@
return 0;
}
-static void iwl_pcie_free_dma_ptr(struct iwl_trans *trans,
- struct iwl_dma_ptr *ptr)
+void iwl_pcie_free_dma_ptr(struct iwl_trans *trans, struct iwl_dma_ptr *ptr)
{
if (unlikely(!ptr->addr))
return;
@@ -164,9 +162,6 @@
}
spin_unlock(&txq->lock);
- IWL_ERR(trans, "Queue %d stuck for %u ms.\n", txq->id,
- jiffies_to_msecs(txq->wd_timeout));
-
iwl_trans_pcie_log_scd_error(trans, txq);
iwl_force_nmi(trans);
@@ -188,6 +183,7 @@
__le16 bc_ent;
struct iwl_tx_cmd *tx_cmd =
(void *)txq->entries[txq->write_ptr].cmd->payload;
+ u8 sta_id = tx_cmd->sta_id;
scd_bc_tbl = trans_pcie->scd_bc_tbls.addr;
@@ -210,26 +206,7 @@
if (WARN_ON(len > 0xFFF || write_ptr >= TFD_QUEUE_SIZE_MAX))
return;
- if (trans->cfg->use_tfh) {
- u8 filled_tfd_size = offsetof(struct iwl_tfh_tfd, tbs) +
- num_tbs * sizeof(struct iwl_tfh_tb);
- /*
- * filled_tfd_size contains the number of filled bytes in the
- * TFD.
- * Dividing it by 64 will give the number of chunks to fetch
- * to SRAM- 0 for one chunk, 1 for 2 and so on.
- * If, for example, TFD contains only 3 TBs then 32 bytes
- * of the TFD are used, and only one chunk of 64 bytes should
- * be fetched
- */
- u8 num_fetch_chunks = DIV_ROUND_UP(filled_tfd_size, 64) - 1;
-
- bc_ent = cpu_to_le16(len | (num_fetch_chunks << 12));
- } else {
- u8 sta_id = tx_cmd->sta_id;
-
- bc_ent = cpu_to_le16(len | (sta_id << 12));
- }
+ bc_ent = cpu_to_le16(len | (sta_id << 12));
scd_bc_tbl[txq_id].tfd_offset[write_ptr] = bc_ent;
@@ -319,23 +296,17 @@
int i;
for (i = 0; i < trans->cfg->base_params->num_of_queues; i++) {
- struct iwl_txq *txq = &trans_pcie->txq[i];
+ struct iwl_txq *txq = trans_pcie->txq[i];
spin_lock_bh(&txq->lock);
- if (trans_pcie->txq[i].need_update) {
+ if (txq->need_update) {
iwl_pcie_txq_inc_wr_ptr(trans, txq);
- trans_pcie->txq[i].need_update = false;
+ txq->need_update = false;
}
spin_unlock_bh(&txq->lock);
}
}
-static inline void *iwl_pcie_get_tfd(struct iwl_trans_pcie *trans_pcie,
- struct iwl_txq *txq, int idx)
-{
- return txq->tfds + trans_pcie->tfd_size * idx;
-}
-
static inline dma_addr_t iwl_pcie_tfd_tb_get_addr(struct iwl_trans *trans,
void *_tfd, u8 idx)
{
@@ -368,28 +339,17 @@
static inline void iwl_pcie_tfd_set_tb(struct iwl_trans *trans, void *tfd,
u8 idx, dma_addr_t addr, u16 len)
{
- if (trans->cfg->use_tfh) {
- struct iwl_tfh_tfd *tfd_fh = (void *)tfd;
- struct iwl_tfh_tb *tb = &tfd_fh->tbs[idx];
+ struct iwl_tfd *tfd_fh = (void *)tfd;
+ struct iwl_tfd_tb *tb = &tfd_fh->tbs[idx];
- put_unaligned_le64(addr, &tb->addr);
- tb->tb_len = cpu_to_le16(len);
+ u16 hi_n_len = len << 4;
- tfd_fh->num_tbs = cpu_to_le16(idx + 1);
- } else {
- struct iwl_tfd *tfd_fh = (void *)tfd;
- struct iwl_tfd_tb *tb = &tfd_fh->tbs[idx];
+ put_unaligned_le32(addr, &tb->lo);
+ hi_n_len |= iwl_get_dma_hi_addr(addr);
- u16 hi_n_len = len << 4;
+ tb->hi_n_len = cpu_to_le16(hi_n_len);
- put_unaligned_le32(addr, &tb->lo);
- if (sizeof(dma_addr_t) > sizeof(u32))
- hi_n_len |= ((addr >> 16) >> 16) & 0xF;
-
- tb->hi_n_len = cpu_to_le16(hi_n_len);
-
- tfd_fh->num_tbs = idx + 1;
- }
+ tfd_fh->num_tbs = idx + 1;
}
static inline u8 iwl_pcie_tfd_get_num_tbs(struct iwl_trans *trans, void *_tfd)
@@ -460,7 +420,7 @@
* Does NOT advance any TFD circular buffer read/write indexes
* Does NOT free the TFD itself (which is within circular buffer)
*/
-static void iwl_pcie_txq_free_tfd(struct iwl_trans *trans, struct iwl_txq *txq)
+void iwl_pcie_txq_free_tfd(struct iwl_trans *trans, struct iwl_txq *txq)
{
/* rd_ptr is bounded by TFD_QUEUE_SIZE_MAX and
* idx is bounded by n_window
@@ -522,9 +482,8 @@
return num_tbs;
}
-static int iwl_pcie_txq_alloc(struct iwl_trans *trans,
- struct iwl_txq *txq, int slots_num,
- u32 txq_id)
+int iwl_pcie_txq_alloc(struct iwl_trans *trans, struct iwl_txq *txq,
+ int slots_num, bool cmd_queue)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
size_t tfd_sz = trans_pcie->tfd_size * TFD_QUEUE_SIZE_MAX;
@@ -547,7 +506,7 @@
if (!txq->entries)
goto error;
- if (txq_id == trans_pcie->cmd_queue)
+ if (cmd_queue)
for (i = 0; i < slots_num; i++) {
txq->entries[i].cmd =
kmalloc(sizeof(struct iwl_device_cmd),
@@ -573,13 +532,11 @@
if (!txq->first_tb_bufs)
goto err_free_tfds;
- txq->id = txq_id;
-
return 0;
err_free_tfds:
dma_free_coherent(trans->dev, tfd_sz, txq->tfds, txq->dma_addr);
error:
- if (txq->entries && txq_id == trans_pcie->cmd_queue)
+ if (txq->entries && cmd_queue)
for (i = 0; i < slots_num; i++)
kfree(txq->entries[i].cmd);
kfree(txq->entries);
@@ -589,10 +546,9 @@
}
-static int iwl_pcie_txq_init(struct iwl_trans *trans, struct iwl_txq *txq,
- int slots_num, u32 txq_id)
+int iwl_pcie_txq_init(struct iwl_trans *trans, struct iwl_txq *txq,
+ int slots_num, bool cmd_queue)
{
- struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
int ret;
txq->need_update = false;
@@ -602,13 +558,13 @@
BUILD_BUG_ON(TFD_QUEUE_SIZE_MAX & (TFD_QUEUE_SIZE_MAX - 1));
/* Initialize queue's high/low-water marks, and head/tail indexes */
- ret = iwl_queue_init(txq, slots_num, txq_id);
+ ret = iwl_queue_init(txq, slots_num);
if (ret)
return ret;
spin_lock_init(&txq->lock);
- if (txq_id == trans_pcie->cmd_queue) {
+ if (cmd_queue) {
static struct lock_class_key iwl_pcie_cmd_queue_lock_class;
lockdep_set_class(&txq->lock, &iwl_pcie_cmd_queue_lock_class);
@@ -616,18 +572,6 @@
__skb_queue_head_init(&txq->overflow_q);
- /*
- * Tell nic where to find circular buffer of Tx Frame Descriptors for
- * given Tx queue, and enable the DMA channel used for that queue.
- * Circular buffer (TFD queue in DRAM) physical base address */
- if (trans->cfg->use_tfh)
- iwl_write_direct64(trans,
- FH_MEM_CBBC_QUEUE(trans, txq_id),
- txq->dma_addr);
- else
- iwl_write_direct32(trans, FH_MEM_CBBC_QUEUE(trans, txq_id),
- txq->dma_addr >> 8);
-
return 0;
}
@@ -672,7 +616,7 @@
static void iwl_pcie_txq_unmap(struct iwl_trans *trans, int txq_id)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_txq *txq = &trans_pcie->txq[txq_id];
+ struct iwl_txq *txq = trans_pcie->txq[txq_id];
spin_lock_bh(&txq->lock);
while (txq->write_ptr != txq->read_ptr) {
@@ -704,7 +648,6 @@
spin_unlock_irqrestore(&trans_pcie->reg_lock, flags);
}
}
- txq->active = false;
while (!skb_queue_empty(&txq->overflow_q)) {
struct sk_buff *skb = __skb_dequeue(&txq->overflow_q);
@@ -729,7 +672,7 @@
static void iwl_pcie_txq_free(struct iwl_trans *trans, int txq_id)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_txq *txq = &trans_pcie->txq[txq_id];
+ struct iwl_txq *txq = trans_pcie->txq[txq_id];
struct device *dev = trans->dev;
int i;
@@ -780,9 +723,6 @@
memset(trans_pcie->queue_stopped, 0, sizeof(trans_pcie->queue_stopped));
memset(trans_pcie->queue_used, 0, sizeof(trans_pcie->queue_used));
- if (trans->cfg->use_tfh)
- return;
-
trans_pcie->scd_base_addr =
iwl_read_prph(trans, SCD_SRAM_BASE_ADDR);
@@ -832,9 +772,16 @@
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
int txq_id;
+ /*
+ * we should never get here in gen2 trans mode return early to avoid
+ * having invalid accesses
+ */
+ if (WARN_ON_ONCE(trans->cfg->gen2))
+ return;
+
for (txq_id = 0; txq_id < trans->cfg->base_params->num_of_queues;
txq_id++) {
- struct iwl_txq *txq = &trans_pcie->txq[txq_id];
+ struct iwl_txq *txq = trans_pcie->txq[txq_id];
if (trans->cfg->use_tfh)
iwl_write_direct64(trans,
FH_MEM_CBBC_QUEUE(trans, txq_id),
@@ -914,7 +861,7 @@
memset(trans_pcie->queue_used, 0, sizeof(trans_pcie->queue_used));
/* This can happen: start_hw, stop_device */
- if (!trans_pcie->txq)
+ if (!trans_pcie->txq_memory)
return 0;
/* Unmap DMA from host system and free skb's */
@@ -935,15 +882,20 @@
int txq_id;
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ memset(trans_pcie->queue_used, 0, sizeof(trans_pcie->queue_used));
+
/* Tx queues */
- if (trans_pcie->txq) {
+ if (trans_pcie->txq_memory) {
for (txq_id = 0;
- txq_id < trans->cfg->base_params->num_of_queues; txq_id++)
+ txq_id < trans->cfg->base_params->num_of_queues;
+ txq_id++) {
iwl_pcie_txq_free(trans, txq_id);
+ trans_pcie->txq[txq_id] = NULL;
+ }
}
- kfree(trans_pcie->txq);
- trans_pcie->txq = NULL;
+ kfree(trans_pcie->txq_memory);
+ trans_pcie->txq_memory = NULL;
iwl_pcie_free_dma_ptr(trans, &trans_pcie->kw);
@@ -965,7 +917,7 @@
/*It is not allowed to alloc twice, so warn when this happens.
* We cannot rely on the previous allocation, so free and fail */
- if (WARN_ON(trans_pcie->txq)) {
+ if (WARN_ON(trans_pcie->txq_memory)) {
ret = -EINVAL;
goto error;
}
@@ -984,9 +936,9 @@
goto error;
}
- trans_pcie->txq = kcalloc(trans->cfg->base_params->num_of_queues,
- sizeof(struct iwl_txq), GFP_KERNEL);
- if (!trans_pcie->txq) {
+ trans_pcie->txq_memory = kcalloc(trans->cfg->base_params->num_of_queues,
+ sizeof(struct iwl_txq), GFP_KERNEL);
+ if (!trans_pcie->txq_memory) {
IWL_ERR(trans, "Not enough memory for txq\n");
ret = -ENOMEM;
goto error;
@@ -995,14 +947,17 @@
/* Alloc and init all Tx queues, including the command queue (#4/#9) */
for (txq_id = 0; txq_id < trans->cfg->base_params->num_of_queues;
txq_id++) {
- slots_num = (txq_id == trans_pcie->cmd_queue) ?
- TFD_CMD_SLOTS : TFD_TX_CMD_SLOTS;
- ret = iwl_pcie_txq_alloc(trans, &trans_pcie->txq[txq_id],
- slots_num, txq_id);
+ bool cmd_queue = (txq_id == trans_pcie->cmd_queue);
+
+ slots_num = cmd_queue ? TFD_CMD_SLOTS : TFD_TX_CMD_SLOTS;
+ trans_pcie->txq[txq_id] = &trans_pcie->txq_memory[txq_id];
+ ret = iwl_pcie_txq_alloc(trans, trans_pcie->txq[txq_id],
+ slots_num, cmd_queue);
if (ret) {
IWL_ERR(trans, "Tx %d queue alloc failed\n", txq_id);
goto error;
}
+ trans_pcie->txq[txq_id]->id = txq_id;
}
return 0;
@@ -1012,6 +967,7 @@
return ret;
}
+
int iwl_pcie_tx_init(struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
@@ -1019,7 +975,7 @@
int txq_id, slots_num;
bool alloc = false;
- if (!trans_pcie->txq) {
+ if (!trans_pcie->txq_memory) {
ret = iwl_pcie_tx_alloc(trans);
if (ret)
goto error;
@@ -1040,22 +996,24 @@
/* Alloc and init all Tx queues, including the command queue (#4/#9) */
for (txq_id = 0; txq_id < trans->cfg->base_params->num_of_queues;
txq_id++) {
- slots_num = (txq_id == trans_pcie->cmd_queue) ?
- TFD_CMD_SLOTS : TFD_TX_CMD_SLOTS;
- ret = iwl_pcie_txq_init(trans, &trans_pcie->txq[txq_id],
- slots_num, txq_id);
+ bool cmd_queue = (txq_id == trans_pcie->cmd_queue);
+
+ slots_num = cmd_queue ? TFD_CMD_SLOTS : TFD_TX_CMD_SLOTS;
+ ret = iwl_pcie_txq_init(trans, trans_pcie->txq[txq_id],
+ slots_num, cmd_queue);
if (ret) {
IWL_ERR(trans, "Tx %d queue init failed\n", txq_id);
goto error;
}
- }
- if (trans->cfg->use_tfh) {
- iwl_write_direct32(trans, TFH_TRANSFER_MODE,
- TFH_TRANSFER_MAX_PENDING_REQ |
- TFH_CHUNK_SIZE_128 |
- TFH_CHUNK_SPLIT_MODE);
- return 0;
+ /*
+ * Tell nic where to find circular buffer of TFDs for a
+ * given Tx queue, and enable the DMA channel used for that
+ * queue.
+ * Circular buffer (TFD queue in DRAM) physical base address
+ */
+ iwl_write_direct32(trans, FH_MEM_CBBC_QUEUE(trans, txq_id),
+ trans_pcie->txq[txq_id]->dma_addr >> 8);
}
iwl_set_bits_prph(trans, SCD_GP_CTRL, SCD_GP_CTRL_AUTO_ACTIVE_MODE);
@@ -1100,7 +1058,7 @@
struct sk_buff_head *skbs)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_txq *txq = &trans_pcie->txq[txq_id];
+ struct iwl_txq *txq = trans_pcie->txq[txq_id];
int tfd_num = ssn & (TFD_QUEUE_SIZE_MAX - 1);
int last_to_free;
@@ -1110,7 +1068,7 @@
spin_lock_bh(&txq->lock);
- if (!txq->active) {
+ if (!test_bit(txq_id, trans_pcie->queue_used)) {
IWL_DEBUG_TX_QUEUES(trans, "Q %d inactive - ignoring idx %d\n",
txq_id, ssn);
goto out;
@@ -1257,7 +1215,7 @@
static void iwl_pcie_cmdq_reclaim(struct iwl_trans *trans, int txq_id, int idx)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_txq *txq = &trans_pcie->txq[txq_id];
+ struct iwl_txq *txq = trans_pcie->txq[txq_id];
unsigned long flags;
int nfreed = 0;
@@ -1324,15 +1282,12 @@
unsigned int wdg_timeout)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_txq *txq = &trans_pcie->txq[txq_id];
+ struct iwl_txq *txq = trans_pcie->txq[txq_id];
int fifo = -1;
if (test_and_set_bit(txq_id, trans_pcie->queue_used))
WARN_ONCE(1, "queue %d already used - expect issues", txq_id);
- if (cfg && trans->cfg->use_tfh)
- WARN_ONCE(1, "Expected no calls to SCD configuration");
-
txq->wd_timeout = msecs_to_jiffies(wdg_timeout);
if (cfg) {
@@ -1414,27 +1369,17 @@
"Activate queue %d WrPtr: %d\n",
txq_id, ssn & 0xff);
}
-
- txq->active = true;
}
void iwl_trans_pcie_txq_set_shared_mode(struct iwl_trans *trans, u32 txq_id,
bool shared_mode)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_txq *txq = &trans_pcie->txq[txq_id];
+ struct iwl_txq *txq = trans_pcie->txq[txq_id];
txq->ampdu = !shared_mode;
}
-dma_addr_t iwl_trans_pcie_get_txq_byte_table(struct iwl_trans *trans, int txq)
-{
- struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
-
- return trans_pcie->scd_bc_tbls.dma +
- txq * sizeof(struct iwlagn_scd_bc_tbl);
-}
-
void iwl_trans_pcie_txq_disable(struct iwl_trans *trans, int txq_id,
bool configure_scd)
{
@@ -1443,8 +1388,8 @@
SCD_TX_STTS_QUEUE_OFFSET(txq_id);
static const u32 zero_val[4] = {};
- trans_pcie->txq[txq_id].frozen_expiry_remainder = 0;
- trans_pcie->txq[txq_id].frozen = false;
+ trans_pcie->txq[txq_id]->frozen_expiry_remainder = 0;
+ trans_pcie->txq[txq_id]->frozen = false;
/*
* Upon HW Rfkill - we stop the device, and then stop the queues
@@ -1458,9 +1403,6 @@
return;
}
- if (configure_scd && trans->cfg->use_tfh)
- WARN_ONCE(1, "Expected no calls to SCD configuration");
-
if (configure_scd) {
iwl_scd_txq_set_inactive(trans, txq_id);
@@ -1469,7 +1411,7 @@
}
iwl_pcie_txq_unmap(trans, txq_id);
- trans_pcie->txq[txq_id].ampdu = false;
+ trans_pcie->txq[txq_id]->ampdu = false;
IWL_DEBUG_TX_QUEUES(trans, "Deactivate queue %d\n", txq_id);
}
@@ -1489,7 +1431,7 @@
struct iwl_host_cmd *cmd)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_txq *txq = &trans_pcie->txq[trans_pcie->cmd_queue];
+ struct iwl_txq *txq = trans_pcie->txq[trans_pcie->cmd_queue];
struct iwl_device_cmd *out_cmd;
struct iwl_cmd_meta *out_meta;
unsigned long flags;
@@ -1774,16 +1716,15 @@
struct iwl_device_cmd *cmd;
struct iwl_cmd_meta *meta;
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- struct iwl_txq *txq = &trans_pcie->txq[trans_pcie->cmd_queue];
+ struct iwl_txq *txq = trans_pcie->txq[trans_pcie->cmd_queue];
/* If a Tx command is being handled and it isn't in the actual
* command queue then there a command routing bug has been introduced
* in the queue management code. */
if (WARN(txq_id != trans_pcie->cmd_queue,
"wrong command queue %d (should be %d), sequence 0x%X readp=%d writep=%d\n",
- txq_id, trans_pcie->cmd_queue, sequence,
- trans_pcie->txq[trans_pcie->cmd_queue].read_ptr,
- trans_pcie->txq[trans_pcie->cmd_queue].write_ptr)) {
+ txq_id, trans_pcie->cmd_queue, sequence, txq->read_ptr,
+ txq->write_ptr)) {
iwl_print_hex_error(trans, pkt, 32);
return;
}
@@ -1867,6 +1808,7 @@
struct iwl_host_cmd *cmd)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
+ struct iwl_txq *txq = trans_pcie->txq[trans_pcie->cmd_queue];
int cmd_idx;
int ret;
@@ -1907,8 +1849,6 @@
&trans->status),
HOST_COMPLETE_TIMEOUT);
if (!ret) {
- struct iwl_txq *txq = &trans_pcie->txq[trans_pcie->cmd_queue];
-
IWL_ERR(trans, "Error sending %s: time out after %dms.\n",
iwl_get_cmd_string(trans, cmd->id),
jiffies_to_msecs(HOST_COMPLETE_TIMEOUT));
@@ -1959,8 +1899,7 @@
* in later, it will possibly set an invalid
* address (cmd->meta.source).
*/
- trans_pcie->txq[trans_pcie->cmd_queue].
- entries[cmd_idx].meta.flags &= ~CMD_WANT_SKB;
+ txq->entries[cmd_idx].meta.flags &= ~CMD_WANT_SKB;
}
if (cmd->resp_pkt) {
@@ -2314,7 +2253,7 @@
u16 wifi_seq;
bool amsdu;
- txq = &trans_pcie->txq[txq_id];
+ txq = trans_pcie->txq[txq_id];
if (WARN_ONCE(!test_bit(txq_id, trans_pcie->queue_used),
"TX on unused queue %d\n", txq_id))
diff --git a/drivers/net/wireless/intersil/orinoco/cfg.c b/drivers/net/wireless/intersil/orinoco/cfg.c
index 7aa4706..b2d5ec8 100644
--- a/drivers/net/wireless/intersil/orinoco/cfg.c
+++ b/drivers/net/wireless/intersil/orinoco/cfg.c
@@ -97,7 +97,7 @@
}
static int orinoco_change_vif(struct wiphy *wiphy, struct net_device *dev,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct orinoco_private *priv = wiphy_priv(wiphy);
diff --git a/drivers/net/wireless/intersil/orinoco/main.c b/drivers/net/wireless/intersil/orinoco/main.c
index 28cf974..d9128bb 100644
--- a/drivers/net/wireless/intersil/orinoco/main.c
+++ b/drivers/net/wireless/intersil/orinoco/main.c
@@ -2283,7 +2283,7 @@
priv->ndev = dev;
/* Report what we've done */
- dev_dbg(priv->dev, "Registerred interface %s.\n", dev->name);
+ dev_dbg(priv->dev, "Registered interface %s.\n", dev->name);
return 0;
diff --git a/drivers/net/wireless/intersil/orinoco/orinoco_usb.c b/drivers/net/wireless/intersil/orinoco/orinoco_usb.c
index 98e1380..132f5fb 100644
--- a/drivers/net/wireless/intersil/orinoco/orinoco_usb.c
+++ b/drivers/net/wireless/intersil/orinoco/orinoco_usb.c
@@ -769,18 +769,31 @@
static inline int ezusb_8051_cpucs(struct ezusb_priv *upriv, int reset)
{
- u8 res_val = reset; /* avoid argument promotion */
+ int ret;
+ u8 *res_val = NULL;
if (!upriv->udev) {
err("%s: !upriv->udev", __func__);
return -EFAULT;
}
- return usb_control_msg(upriv->udev,
+
+ res_val = kmalloc(sizeof(*res_val), GFP_KERNEL);
+
+ if (!res_val)
+ return -ENOMEM;
+
+ *res_val = reset; /* avoid argument promotion */
+
+ ret = usb_control_msg(upriv->udev,
usb_sndctrlpipe(upriv->udev, 0),
EZUSB_REQUEST_FW_TRANS,
USB_TYPE_VENDOR | USB_RECIP_DEVICE |
- USB_DIR_OUT, EZUSB_CPUCS_REG, 0, &res_val,
- sizeof(res_val), DEF_TIMEOUT);
+ USB_DIR_OUT, EZUSB_CPUCS_REG, 0, res_val,
+ sizeof(*res_val), DEF_TIMEOUT);
+
+ kfree(res_val);
+
+ return ret;
}
static int ezusb_firmware_download(struct ezusb_priv *upriv,
diff --git a/drivers/net/wireless/intersil/p54/txrx.c b/drivers/net/wireless/intersil/p54/txrx.c
index 1af7da0..5e1c91a 100644
--- a/drivers/net/wireless/intersil/p54/txrx.c
+++ b/drivers/net/wireless/intersil/p54/txrx.c
@@ -352,7 +352,7 @@
rx_status->signal = p54_rssi_to_dbm(priv, hdr->rssi);
if (hdr->rate & 0x10)
- rx_status->flag |= RX_FLAG_SHORTPRE;
+ rx_status->enc_flags |= RX_ENC_FLAG_SHORTPRE;
if (priv->hw->conf.chandef.chan->band == NL80211_BAND_5GHZ)
rx_status->rate_idx = (rate < 4) ? 0 : rate - 4;
else
diff --git a/drivers/net/wireless/mac80211_hwsim.c b/drivers/net/wireless/mac80211_hwsim.c
index 50c219f..87444af 100644
--- a/drivers/net/wireless/mac80211_hwsim.c
+++ b/drivers/net/wireless/mac80211_hwsim.c
@@ -350,6 +350,7 @@
CHAN5G(5785), /* Channel 157 */
CHAN5G(5805), /* Channel 161 */
CHAN5G(5825), /* Channel 165 */
+ CHAN5G(5845), /* Channel 169 */
};
static const struct ieee80211_rate hwsim_rates[] = {
@@ -389,7 +390,7 @@
u32 val;
err = nla_parse(tb, QCA_WLAN_VENDOR_ATTR_MAX, data, data_len,
- hwsim_vendor_test_policy);
+ hwsim_vendor_test_policy, NULL);
if (err)
return err;
if (!tb[QCA_WLAN_VENDOR_ATTR_TEST])
@@ -525,6 +526,11 @@
struct ieee80211_vif *hw_scan_vif;
int scan_chan_idx;
u8 scan_addr[ETH_ALEN];
+ struct {
+ struct ieee80211_channel *channel;
+ unsigned long next_start, start, end;
+ } survey_data[ARRAY_SIZE(hwsim_channels_2ghz) +
+ ARRAY_SIZE(hwsim_channels_5ghz)];
struct ieee80211_channel *channel;
u64 beacon_int /* beacon interval in us */;
@@ -552,8 +558,6 @@
/* wmediumd portid responsible for netgroup of this radio */
u32 wmediumd;
- int power_level;
-
/* difference between this hw's clock and the real clock, in usecs */
s64 tsf_offset;
s64 bcn_delta;
@@ -1188,20 +1192,22 @@
if (info->control.rates[0].flags & IEEE80211_TX_RC_VHT_MCS) {
rx_status.rate_idx =
ieee80211_rate_get_vht_mcs(&info->control.rates[0]);
- rx_status.vht_nss =
+ rx_status.nss =
ieee80211_rate_get_vht_nss(&info->control.rates[0]);
- rx_status.flag |= RX_FLAG_VHT;
+ rx_status.encoding = RX_ENC_VHT;
} else {
rx_status.rate_idx = info->control.rates[0].idx;
if (info->control.rates[0].flags & IEEE80211_TX_RC_MCS)
- rx_status.flag |= RX_FLAG_HT;
+ rx_status.encoding = RX_ENC_HT;
}
if (info->control.rates[0].flags & IEEE80211_TX_RC_40_MHZ_WIDTH)
- rx_status.flag |= RX_FLAG_40MHZ;
+ rx_status.enc_flags |= RX_ENC_FLAG_40MHZ;
if (info->control.rates[0].flags & IEEE80211_TX_RC_SHORT_GI)
- rx_status.flag |= RX_FLAG_SHORT_GI;
+ rx_status.enc_flags |= RX_ENC_FLAG_SHORT_GI;
/* TODO: simulate real signal strength (and optional packet loss) */
- rx_status.signal = data->power_level - 50;
+ rx_status.signal = -50;
+ if (info->control.vif)
+ rx_status.signal += info->control.vif->bss_conf.txpower;
if (data->ps != PS_DISABLED)
hdr->frame_control |= cpu_to_le16(IEEE80211_FCTL_PM);
@@ -1576,6 +1582,7 @@
[IEEE80211_SMPS_STATIC] = "static",
[IEEE80211_SMPS_DYNAMIC] = "dynamic",
};
+ int idx;
if (conf->chandef.chan)
wiphy_debug(hw->wiphy,
@@ -1598,11 +1605,34 @@
data->idle = !!(conf->flags & IEEE80211_CONF_IDLE);
- data->channel = conf->chandef.chan;
+ WARN_ON(conf->chandef.chan && data->use_chanctx);
- WARN_ON(data->channel && data->use_chanctx);
+ mutex_lock(&data->mutex);
+ if (data->scanning && conf->chandef.chan) {
+ for (idx = 0; idx < ARRAY_SIZE(data->survey_data); idx++) {
+ if (data->survey_data[idx].channel == data->channel) {
+ data->survey_data[idx].start =
+ data->survey_data[idx].next_start;
+ data->survey_data[idx].end = jiffies;
+ break;
+ }
+ }
- data->power_level = conf->power_level;
+ data->channel = conf->chandef.chan;
+
+ for (idx = 0; idx < ARRAY_SIZE(data->survey_data); idx++) {
+ if (data->survey_data[idx].channel &&
+ data->survey_data[idx].channel != data->channel)
+ continue;
+ data->survey_data[idx].channel = data->channel;
+ data->survey_data[idx].next_start = jiffies;
+ break;
+ }
+ } else {
+ data->channel = conf->chandef.chan;
+ }
+ mutex_unlock(&data->mutex);
+
if (!data->started || !data->beacon_int)
tasklet_hrtimer_cancel(&data->beacon_timer);
else if (!hrtimer_is_queued(&data->beacon_timer.timer)) {
@@ -1787,28 +1817,37 @@
return 0;
}
-static int mac80211_hwsim_get_survey(
- struct ieee80211_hw *hw, int idx,
- struct survey_info *survey)
+static int mac80211_hwsim_get_survey(struct ieee80211_hw *hw, int idx,
+ struct survey_info *survey)
{
- struct ieee80211_conf *conf = &hw->conf;
+ struct mac80211_hwsim_data *hwsim = hw->priv;
- wiphy_debug(hw->wiphy, "%s (idx=%d)\n", __func__, idx);
-
- if (idx != 0)
+ if (idx < 0 || idx >= ARRAY_SIZE(hwsim->survey_data))
return -ENOENT;
- /* Current channel */
- survey->channel = conf->chandef.chan;
+ mutex_lock(&hwsim->mutex);
+ survey->channel = hwsim->survey_data[idx].channel;
+ if (!survey->channel) {
+ mutex_unlock(&hwsim->mutex);
+ return -ENOENT;
+ }
/*
- * Magically conjured noise level --- this is only ok for simulated hardware.
+ * Magically conjured dummy values --- this is only ok for simulated hardware.
*
- * A real driver which cannot determine the real channel noise MUST NOT
- * report any noise, especially not a magically conjured one :-)
+ * A real driver which cannot determine real values noise MUST NOT
+ * report any, especially not a magically conjured ones :-)
*/
- survey->filled = SURVEY_INFO_NOISE_DBM;
+ survey->filled = SURVEY_INFO_NOISE_DBM |
+ SURVEY_INFO_TIME |
+ SURVEY_INFO_TIME_BUSY;
survey->noise = -92;
+ survey->time =
+ jiffies_to_msecs(hwsim->survey_data[idx].end -
+ hwsim->survey_data[idx].start);
+ /* report 12.5% of channel time is used */
+ survey->time_busy = survey->time/8;
+ mutex_unlock(&hwsim->mutex);
return 0;
}
@@ -1852,7 +1891,7 @@
int err, ps;
err = nla_parse(tb, HWSIM_TM_ATTR_MAX, data, len,
- hwsim_testmode_policy);
+ hwsim_testmode_policy, NULL);
if (err)
return err;
@@ -1986,6 +2025,10 @@
}
ieee80211_queue_delayed_work(hwsim->hw, &hwsim->hw_scan,
msecs_to_jiffies(dwell));
+ hwsim->survey_data[hwsim->scan_chan_idx].channel = hwsim->tmp_chan;
+ hwsim->survey_data[hwsim->scan_chan_idx].start = jiffies;
+ hwsim->survey_data[hwsim->scan_chan_idx].end =
+ jiffies + msecs_to_jiffies(dwell);
hwsim->scan_chan_idx++;
mutex_unlock(&hwsim->mutex);
}
@@ -2011,6 +2054,7 @@
hw_req->req.mac_addr_mask);
else
memcpy(hwsim->scan_addr, vif->addr, ETH_ALEN);
+ memset(hwsim->survey_data, 0, sizeof(hwsim->survey_data));
mutex_unlock(&hwsim->mutex);
wiphy_debug(hw->wiphy, "hwsim hw_scan request\n");
@@ -2057,6 +2101,7 @@
memcpy(hwsim->scan_addr, mac_addr, ETH_ALEN);
hwsim->scanning = true;
+ memset(hwsim->survey_data, 0, sizeof(hwsim->survey_data));
out:
mutex_unlock(&hwsim->mutex);
@@ -2207,7 +2252,6 @@
"d_tx_failed",
"d_ps_mode",
"d_group",
- "d_tx_power",
};
#define MAC80211_HWSIM_SSTATS_LEN ARRAY_SIZE(mac80211_hwsim_gstrings_stats)
@@ -2244,7 +2288,6 @@
data[i++] = ar->tx_failed;
data[i++] = ar->ps;
data[i++] = ar->group;
- data[i++] = ar->power_level;
WARN_ON(i != MAC80211_HWSIM_SSTATS_LEN);
}
@@ -2438,6 +2481,9 @@
goto failed;
}
+ /* ieee80211_alloc_hw_nm may have used a default name */
+ param->hwname = wiphy_name(hw->wiphy);
+
if (info)
net = genl_info_net(info);
else
@@ -2645,6 +2691,8 @@
if (param->no_vif)
ieee80211_hw_set(hw, NO_AUTO_VIF);
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
err = ieee80211_register_hw(hw);
if (err < 0) {
printk(KERN_DEBUG "mac80211_hwsim: ieee80211_register_hw failed (%d)\n",
diff --git a/drivers/net/wireless/mac80211_hwsim.h b/drivers/net/wireless/mac80211_hwsim.h
index 39f2246..3f5eda5 100644
--- a/drivers/net/wireless/mac80211_hwsim.h
+++ b/drivers/net/wireless/mac80211_hwsim.h
@@ -57,12 +57,12 @@
* @HWSIM_CMD_REGISTER: request to register and received all broadcasted
* frames by any mac80211_hwsim radio device.
* @HWSIM_CMD_FRAME: send/receive a broadcasted frame from/to kernel/user
- * space, uses:
+ * space, uses:
* %HWSIM_ATTR_ADDR_TRANSMITTER, %HWSIM_ATTR_ADDR_RECEIVER,
* %HWSIM_ATTR_FRAME, %HWSIM_ATTR_FLAGS, %HWSIM_ATTR_RX_RATE,
* %HWSIM_ATTR_SIGNAL, %HWSIM_ATTR_COOKIE, %HWSIM_ATTR_FREQ (optional)
* @HWSIM_CMD_TX_INFO_FRAME: Transmission info report from user space to
- * kernel, uses:
+ * kernel, uses:
* %HWSIM_ATTR_ADDR_TRANSMITTER, %HWSIM_ATTR_FLAGS,
* %HWSIM_ATTR_TX_INFO, %HWSIM_ATTR_SIGNAL, %HWSIM_ATTR_COOKIE
* @HWSIM_CMD_NEW_RADIO: create a new radio with the given parameters,
diff --git a/drivers/net/wireless/marvell/libertas/cfg.c b/drivers/net/wireless/marvell/libertas/cfg.c
index 3f97acb..a0463fe 100644
--- a/drivers/net/wireless/marvell/libertas/cfg.c
+++ b/drivers/net/wireless/marvell/libertas/cfg.c
@@ -1657,7 +1657,7 @@
*/
static int lbs_change_intf(struct wiphy *wiphy, struct net_device *dev,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct lbs_private *priv = wiphy_priv(wiphy);
diff --git a/drivers/net/wireless/marvell/libertas/if_spi.c b/drivers/net/wireless/marvell/libertas/if_spi.c
index c3a53cd..7b4955c 100644
--- a/drivers/net/wireless/marvell/libertas/if_spi.c
+++ b/drivers/net/wireless/marvell/libertas/if_spi.c
@@ -1181,6 +1181,10 @@
/* Initialize interrupt handling stuff. */
card->workqueue = alloc_workqueue("libertas_spi", WQ_MEM_RECLAIM, 0);
+ if (!card->workqueue) {
+ err = -ENOMEM;
+ goto remove_card;
+ }
INIT_WORK(&card->packet_work, if_spi_host_to_card_worker);
INIT_WORK(&card->resume_work, if_spi_resume_worker);
@@ -1209,6 +1213,7 @@
free_irq(spi->irq, card);
terminate_workqueue:
destroy_workqueue(card->workqueue);
+remove_card:
lbs_remove_card(priv); /* will call free_netdev */
free_card:
free_if_spi_card(card);
diff --git a/drivers/net/wireless/marvell/libertas_tf/main.c b/drivers/net/wireless/marvell/libertas_tf/main.c
index 54e426c..d803331 100644
--- a/drivers/net/wireless/marvell/libertas_tf/main.c
+++ b/drivers/net/wireless/marvell/libertas_tf/main.c
@@ -641,6 +641,8 @@
BIT(NL80211_IFTYPE_ADHOC);
skb_queue_head_init(&priv->bc_ps_buf);
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
SET_IEEE80211_DEV(hw, dmdev);
INIT_WORK(&priv->cmd_work, lbtf_cmd_work);
diff --git a/drivers/net/wireless/marvell/mwifiex/11h.c b/drivers/net/wireless/marvell/mwifiex/11h.c
index 43dccd5..366eb49 100644
--- a/drivers/net/wireless/marvell/mwifiex/11h.c
+++ b/drivers/net/wireless/marvell/mwifiex/11h.c
@@ -153,7 +153,8 @@
cmd->command = cpu_to_le16(HostCmd_CMD_CHAN_REPORT_REQUEST);
cmd->size = cpu_to_le16(S_DS_GEN);
- le16_add_cpu(&cmd->size, sizeof(struct host_cmd_ds_chan_rpt_req));
+ le16_unaligned_add_cpu(&cmd->size,
+ sizeof(struct host_cmd_ds_chan_rpt_req));
cr_req->chan_desc.start_freq = cpu_to_le16(MWIFIEX_A_BAND_START_FREQ);
cr_req->chan_desc.chan_num = radar_params->chandef->chan->hw_value;
diff --git a/drivers/net/wireless/marvell/mwifiex/cfg80211.c b/drivers/net/wireless/marvell/mwifiex/cfg80211.c
index 1e3bd43..7ec06bf 100644
--- a/drivers/net/wireless/marvell/mwifiex/cfg80211.c
+++ b/drivers/net/wireless/marvell/mwifiex/cfg80211.c
@@ -594,6 +594,24 @@
return 0;
}
+static void mwifiex_reg_apply_radar_flags(struct wiphy *wiphy)
+{
+ struct ieee80211_supported_band *sband;
+ struct ieee80211_channel *chan;
+ unsigned int i;
+
+ if (!wiphy->bands[NL80211_BAND_5GHZ])
+ return;
+ sband = wiphy->bands[NL80211_BAND_5GHZ];
+
+ for (i = 0; i < sband->n_channels; i++) {
+ chan = &sband->channels[i];
+ if ((!(chan->flags & IEEE80211_CHAN_DISABLED)) &&
+ (chan->flags & IEEE80211_CHAN_RADAR))
+ chan->flags |= IEEE80211_CHAN_NO_IR;
+ }
+}
+
/*
* CFG802.11 regulatory domain callback function.
*
@@ -613,6 +631,7 @@
mwifiex_dbg(adapter, INFO,
"info: cfg80211 regulatory domain callback for %c%c\n",
request->alpha2[0], request->alpha2[1]);
+ mwifiex_reg_apply_radar_flags(wiphy);
switch (request->initiator) {
case NL80211_REGDOM_SET_BY_DRIVER:
@@ -916,7 +935,7 @@
static int
mwifiex_change_vif_to_p2p(struct net_device *dev,
enum nl80211_iftype curr_iftype,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct mwifiex_private *priv;
@@ -988,7 +1007,7 @@
static int
mwifiex_change_vif_to_sta_adhoc(struct net_device *dev,
enum nl80211_iftype curr_iftype,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct mwifiex_private *priv;
@@ -1047,7 +1066,7 @@
static int
mwifiex_change_vif_to_ap(struct net_device *dev,
enum nl80211_iftype curr_iftype,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct mwifiex_private *priv;
@@ -1103,7 +1122,7 @@
static int
mwifiex_cfg80211_change_virtual_intf(struct wiphy *wiphy,
struct net_device *dev,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct mwifiex_private *priv = mwifiex_netdev_get_priv(dev);
@@ -1124,10 +1143,10 @@
case NL80211_IFTYPE_P2P_CLIENT:
case NL80211_IFTYPE_P2P_GO:
return mwifiex_change_vif_to_p2p(dev, curr_iftype,
- type, flags, params);
+ type, params);
case NL80211_IFTYPE_AP:
return mwifiex_change_vif_to_ap(dev, curr_iftype, type,
- flags, params);
+ params);
case NL80211_IFTYPE_UNSPECIFIED:
mwifiex_dbg(priv->adapter, INFO,
"%s: kept type as IBSS\n", dev->name);
@@ -1154,10 +1173,10 @@
case NL80211_IFTYPE_P2P_CLIENT:
case NL80211_IFTYPE_P2P_GO:
return mwifiex_change_vif_to_p2p(dev, curr_iftype,
- type, flags, params);
+ type, params);
case NL80211_IFTYPE_AP:
return mwifiex_change_vif_to_ap(dev, curr_iftype, type,
- flags, params);
+ params);
case NL80211_IFTYPE_UNSPECIFIED:
mwifiex_dbg(priv->adapter, INFO,
"%s: kept type as STA\n", dev->name);
@@ -1175,13 +1194,12 @@
case NL80211_IFTYPE_ADHOC:
case NL80211_IFTYPE_STATION:
return mwifiex_change_vif_to_sta_adhoc(dev, curr_iftype,
- type, flags,
- params);
+ type, params);
break;
case NL80211_IFTYPE_P2P_CLIENT:
case NL80211_IFTYPE_P2P_GO:
return mwifiex_change_vif_to_p2p(dev, curr_iftype,
- type, flags, params);
+ type, params);
case NL80211_IFTYPE_UNSPECIFIED:
mwifiex_dbg(priv->adapter, INFO,
"%s: kept type as AP\n", dev->name);
@@ -1214,14 +1232,13 @@
if (mwifiex_cfg80211_deinit_p2p(priv))
return -EFAULT;
return mwifiex_change_vif_to_sta_adhoc(dev, curr_iftype,
- type, flags,
- params);
+ type, params);
break;
case NL80211_IFTYPE_AP:
if (mwifiex_cfg80211_deinit_p2p(priv))
return -EFAULT;
return mwifiex_change_vif_to_ap(dev, curr_iftype, type,
- flags, params);
+ params);
case NL80211_IFTYPE_UNSPECIFIED:
mwifiex_dbg(priv->adapter, INFO,
"%s: kept type as P2P\n", dev->name);
@@ -2036,7 +2053,7 @@
struct mwifiex_private *priv = mwifiex_netdev_get_priv(dev);
if (!mwifiex_stop_bg_scan(priv))
- cfg80211_sched_scan_stopped_rtnl(priv->wdev.wiphy);
+ cfg80211_sched_scan_stopped_rtnl(priv->wdev.wiphy, 0);
if (mwifiex_deauthenticate(priv, NULL))
return -EFAULT;
@@ -2304,7 +2321,7 @@
(int)sme->ssid_len, (char *)sme->ssid, sme->bssid);
if (!mwifiex_stop_bg_scan(priv))
- cfg80211_sched_scan_stopped_rtnl(priv->wdev.wiphy);
+ cfg80211_sched_scan_stopped_rtnl(priv->wdev.wiphy, 0);
ret = mwifiex_cfg80211_assoc(priv, sme->ssid_len, sme->ssid, sme->bssid,
priv->bss_mode, sme->channel, sme, 0);
@@ -2513,7 +2530,7 @@
priv->scan_block = false;
if (!mwifiex_stop_bg_scan(priv))
- cfg80211_sched_scan_stopped_rtnl(priv->wdev.wiphy);
+ cfg80211_sched_scan_stopped_rtnl(priv->wdev.wiphy, 0);
user_scan_cfg = kzalloc(sizeof(*user_scan_cfg), GFP_KERNEL);
if (!user_scan_cfg)
@@ -2528,9 +2545,11 @@
priv->random_mac[i] |= get_random_int() &
~(request->mac_addr_mask[i]);
}
+ ether_addr_copy(user_scan_cfg->random_mac, priv->random_mac);
+ } else {
+ eth_zero_addr(priv->random_mac);
}
- ether_addr_copy(user_scan_cfg->random_mac, priv->random_mac);
user_scan_cfg->num_ssids = request->n_ssids;
user_scan_cfg->ssid_list = request->ssids;
@@ -2701,7 +2720,7 @@
* previous bgscan configuration in the firmware
*/
static int mwifiex_cfg80211_sched_scan_stop(struct wiphy *wiphy,
- struct net_device *dev)
+ struct net_device *dev, u64 reqid)
{
struct mwifiex_private *priv = mwifiex_netdev_get_priv(dev);
@@ -2822,7 +2841,6 @@
const char *name,
unsigned char name_assign_type,
enum nl80211_iftype type,
- u32 *flags,
struct vif_params *params)
{
struct mwifiex_adapter *adapter = mwifiex_cfg80211_get_adapter(wiphy);
@@ -3997,8 +4015,8 @@
if (!priv)
return -EINVAL;
- err = nla_parse(tb, MWIFIEX_TM_ATTR_MAX, data, len,
- mwifiex_tm_policy);
+ err = nla_parse(tb, MWIFIEX_TM_ATTR_MAX, data, len, mwifiex_tm_policy,
+ NULL);
if (err)
return err;
@@ -4279,7 +4297,6 @@
wiphy->flags |= WIPHY_FLAG_HAVE_AP_SME |
WIPHY_FLAG_AP_PROBE_RESP_OFFLOAD |
WIPHY_FLAG_AP_UAPSD |
- WIPHY_FLAG_SUPPORTS_SCHED_SCAN |
WIPHY_FLAG_HAS_REMAIN_ON_CHANNEL |
WIPHY_FLAG_HAS_CHANNEL_SWITCH |
WIPHY_FLAG_PS_ON_BY_DEFAULT;
@@ -4298,6 +4315,7 @@
NL80211_PROBE_RESP_OFFLOAD_SUPPORT_WPS2 |
NL80211_PROBE_RESP_OFFLOAD_SUPPORT_P2P;
+ wiphy->max_sched_scan_reqs = 1;
wiphy->max_sched_scan_ssids = MWIFIEX_MAX_SSID_LIST_LENGTH;
wiphy->max_sched_scan_ie_len = MWIFIEX_MAX_VSIE_LEN;
wiphy->max_match_sets = MWIFIEX_MAX_SSID_LIST_LENGTH;
diff --git a/drivers/net/wireless/marvell/mwifiex/cmdevt.c b/drivers/net/wireless/marvell/mwifiex/cmdevt.c
index 25a7475..0c3b217 100644
--- a/drivers/net/wireless/marvell/mwifiex/cmdevt.c
+++ b/drivers/net/wireless/marvell/mwifiex/cmdevt.c
@@ -242,7 +242,7 @@
mwifiex_dbg(adapter, CMD,
"cmd: DNLD_CMD: %#x, act %#x, len %d, seqno %#x\n",
cmd_code,
- le16_to_cpu(*(__le16 *)((u8 *)host_cmd + S_DS_GEN)),
+ get_unaligned_le16((u8 *)host_cmd + S_DS_GEN),
cmd_size, le16_to_cpu(host_cmd->seq_num));
mwifiex_dbg_dump(adapter, CMD_D, "cmd buffer:", host_cmd, cmd_size);
@@ -286,7 +286,7 @@
(adapter->dbg.last_cmd_index + 1) % DBG_CMD_NUM;
adapter->dbg.last_cmd_id[adapter->dbg.last_cmd_index] = cmd_code;
adapter->dbg.last_cmd_act[adapter->dbg.last_cmd_index] =
- le16_to_cpu(*(__le16 *) ((u8 *) host_cmd + S_DS_GEN));
+ get_unaligned_le16((u8 *)host_cmd + S_DS_GEN);
/* Clear BSS_NO_BITS from HostCmd */
cmd_code &= HostCmd_CMD_ID_MASK;
diff --git a/drivers/net/wireless/marvell/mwifiex/fw.h b/drivers/net/wireless/marvell/mwifiex/fw.h
index cb6a1a8..6cf9ab9 100644
--- a/drivers/net/wireless/marvell/mwifiex/fw.h
+++ b/drivers/net/wireless/marvell/mwifiex/fw.h
@@ -31,17 +31,35 @@
u8 llc_ctrl;
u8 snap_oui[3];
__be16 snap_type;
-};
+} __packed;
struct rx_packet_hdr {
struct ethhdr eth803_hdr;
struct rfc_1042_hdr rfc1042_hdr;
-};
+} __packed;
struct tx_packet_hdr {
struct ethhdr eth803_hdr;
struct rfc_1042_hdr rfc1042_hdr;
-};
+} __packed;
+
+struct mwifiex_fw_header {
+ __le32 dnld_cmd;
+ __le32 base_addr;
+ __le32 data_length;
+ __le32 crc;
+} __packed;
+
+struct mwifiex_fw_data {
+ struct mwifiex_fw_header header;
+ __le32 seq_num;
+ u8 data[1];
+} __packed;
+
+#define MWIFIEX_FW_DNLD_CMD_1 0x1
+#define MWIFIEX_FW_DNLD_CMD_5 0x5
+#define MWIFIEX_FW_DNLD_CMD_6 0x6
+#define MWIFIEX_FW_DNLD_CMD_7 0x7
#define B_SUPPORTED_RATES 5
#define G_SUPPORTED_RATES 9
@@ -707,7 +725,7 @@
u8 reserved1[2];
u8 tx_token_id;
u8 reserved[2];
-};
+} __packed;
struct uap_rxpd {
u8 bss_type;
@@ -723,7 +741,7 @@
u8 ht_info;
u8 reserved[3];
u8 flags;
-};
+} __packed;
struct mwifiex_fw_chan_stats {
u8 chan_num;
@@ -987,7 +1005,7 @@
__le16 adhoc_wake_period;
__le16 mode;
__le16 delay_to_ps;
-};
+} __packed;
#define HS_DEF_WAKE_INTERVAL 100
#define HS_DEF_INACTIVITY_TIMEOUT 50
@@ -996,7 +1014,7 @@
struct mwifiex_ie_types_header header;
__le32 hs_wake_int;
__le32 hs_inact_timeout;
-};
+} __packed;
#define BITMAP_AUTO_DS 0x01
#define BITMAP_STA_PS 0x10
@@ -1062,7 +1080,7 @@
__le16 nbcn;
__le16 reserved[9];
long long reserved_1;
-};
+} __packed;
struct host_cmd_ds_802_11_rssi_info_rsp {
__le16 action;
@@ -1077,12 +1095,12 @@
__le16 bcn_rssi_avg;
__le16 bcn_nf_avg;
long long tsf_bcn;
-};
+} __packed;
struct host_cmd_ds_802_11_mac_address {
__le16 action;
u8 mac_addr[ETH_ALEN];
-};
+} __packed;
struct host_cmd_ds_mac_control {
__le32 action;
@@ -1230,7 +1248,7 @@
__le32 wep_icv_err_cnt[4];
__le32 bcn_rcv_cnt;
__le32 bcn_miss_cnt;
-};
+} __packed;
/* Enumeration for rate format */
enum _mwifiex_rate_format {
@@ -1368,12 +1386,12 @@
__le16 tx_ant_mode;
__le16 action_rx;
__le16 rx_ant_mode;
-};
+} __packed;
struct host_cmd_ds_rf_ant_siso {
__le16 action;
__le16 ant_mode;
-};
+} __packed;
struct host_cmd_ds_tdls_oper {
__le16 tdls_action;
@@ -1383,13 +1401,13 @@
struct mwifiex_tdls_config {
__le16 enable;
-};
+} __packed;
struct mwifiex_tdls_config_cs_params {
u8 unit_time;
u8 thr_otherlink;
u8 thr_directlink;
-};
+} __packed;
struct mwifiex_tdls_init_cs_params {
u8 peer_mac[ETH_ALEN];
@@ -1404,7 +1422,7 @@
struct mwifiex_tdls_stop_cs_params {
u8 peer_mac[ETH_ALEN];
-};
+} __packed;
struct host_cmd_ds_tdls_config {
__le16 tdls_action;
@@ -1709,7 +1727,7 @@
struct mwifiex_ie_types_wmm_param_set {
struct mwifiex_ie_types_header header;
u8 wmm_ie[1];
-};
+} __packed;
struct mwifiex_ie_types_mgmt_frame {
struct mwifiex_ie_types_header header;
@@ -1834,7 +1852,7 @@
__le16 reserved;
__le32 addr;
__le32 value;
-};
+} __packed;
struct mwifiex_ie_types_qos_info {
struct mwifiex_ie_types_header header;
diff --git a/drivers/net/wireless/marvell/mwifiex/ie.c b/drivers/net/wireless/marvell/mwifiex/ie.c
index c488c30..922e3d6 100644
--- a/drivers/net/wireless/marvell/mwifiex/ie.c
+++ b/drivers/net/wireless/marvell/mwifiex/ie.c
@@ -131,9 +131,10 @@
sizeof(struct mwifiex_ie));
}
- le16_add_cpu(&ie_list->len,
- le16_to_cpu(priv->mgmt_ie[index].ie_length) +
- MWIFIEX_IE_HDR_SIZE);
+ le16_unaligned_add_cpu(&ie_list->len,
+ le16_to_cpu(
+ priv->mgmt_ie[index].ie_length) +
+ MWIFIEX_IE_HDR_SIZE);
input_len -= tlv_len + MWIFIEX_IE_HDR_SIZE;
}
@@ -172,21 +173,21 @@
le16_to_cpu(beacon_ie->ie_length);
memcpy(pos, beacon_ie, len);
pos += len;
- le16_add_cpu(&ap_custom_ie->len, len);
+ le16_unaligned_add_cpu(&ap_custom_ie->len, len);
}
if (pr_ie) {
len = sizeof(struct mwifiex_ie) - IEEE_MAX_IE_SIZE +
le16_to_cpu(pr_ie->ie_length);
memcpy(pos, pr_ie, len);
pos += len;
- le16_add_cpu(&ap_custom_ie->len, len);
+ le16_unaligned_add_cpu(&ap_custom_ie->len, len);
}
if (ar_ie) {
len = sizeof(struct mwifiex_ie) - IEEE_MAX_IE_SIZE +
le16_to_cpu(ar_ie->ie_length);
memcpy(pos, ar_ie, len);
pos += len;
- le16_add_cpu(&ap_custom_ie->len, len);
+ le16_unaligned_add_cpu(&ap_custom_ie->len, len);
}
ret = mwifiex_update_autoindex_ies(priv, ap_custom_ie);
@@ -242,7 +243,7 @@
vs_ie = (struct ieee_types_header *)vendor_ie;
memcpy(ie->ie_buffer + le16_to_cpu(ie->ie_length),
vs_ie, vs_ie->len + 2);
- le16_add_cpu(&ie->ie_length, vs_ie->len + 2);
+ le16_unaligned_add_cpu(&ie->ie_length, vs_ie->len + 2);
ie->mgmt_subtype_mask = cpu_to_le16(mask);
ie->ie_index = cpu_to_le16(MWIFIEX_AUTO_IDX_MASK);
}
diff --git a/drivers/net/wireless/marvell/mwifiex/ioctl.h b/drivers/net/wireless/marvell/mwifiex/ioctl.h
index 536ab83..48e154e 100644
--- a/drivers/net/wireless/marvell/mwifiex/ioctl.h
+++ b/drivers/net/wireless/marvell/mwifiex/ioctl.h
@@ -91,6 +91,8 @@
#define MWIFIEX_TDLS_DEF_QOS_CAPAB 0xf
#define MWIFIEX_PRIO_BK 2
#define MWIFIEX_PRIO_VI 5
+#define MWIFIEX_SUPPORTED_CHANNELS 2
+#define MWIFIEX_OPERATING_CLASSES 16
struct mwifiex_uap_bss_param {
u8 channel;
diff --git a/drivers/net/wireless/marvell/mwifiex/main.c b/drivers/net/wireless/marvell/mwifiex/main.c
index b62e03d..dd87b9f 100644
--- a/drivers/net/wireless/marvell/mwifiex/main.c
+++ b/drivers/net/wireless/marvell/mwifiex/main.c
@@ -17,6 +17,8 @@
* this warranty disclaimer.
*/
+#include <linux/suspend.h>
+
#include "main.h"
#include "wmm.h"
#include "cfg80211.h"
@@ -147,7 +149,6 @@
kfree(adapter->regd);
- vfree(adapter->chan_stats);
kfree(adapter);
return 0;
}
@@ -511,7 +512,7 @@
* - Download the correct firmware to card
* - Issue the init commands to firmware
*/
-static void mwifiex_fw_dpc(const struct firmware *firmware, void *context)
+static int _mwifiex_fw_dpc(const struct firmware *firmware, void *context)
{
int ret;
char fmt[64];
@@ -594,7 +595,7 @@
rtnl_lock();
/* Create station interface by default */
wdev = mwifiex_add_virtual_intf(adapter->wiphy, "mlan%d", NET_NAME_ENUM,
- NL80211_IFTYPE_STATION, NULL, NULL);
+ NL80211_IFTYPE_STATION, NULL);
if (IS_ERR(wdev)) {
mwifiex_dbg(adapter, ERROR,
"cannot create default STA interface\n");
@@ -604,7 +605,7 @@
if (driver_mode & MWIFIEX_DRIVER_MODE_UAP) {
wdev = mwifiex_add_virtual_intf(adapter->wiphy, "uap%d", NET_NAME_ENUM,
- NL80211_IFTYPE_AP, NULL, NULL);
+ NL80211_IFTYPE_AP, NULL);
if (IS_ERR(wdev)) {
mwifiex_dbg(adapter, ERROR,
"cannot create AP interface\n");
@@ -615,8 +616,7 @@
if (driver_mode & MWIFIEX_DRIVER_MODE_P2P) {
wdev = mwifiex_add_virtual_intf(adapter->wiphy, "p2p%d", NET_NAME_ENUM,
- NL80211_IFTYPE_P2P_CLIENT, NULL,
- NULL);
+ NL80211_IFTYPE_P2P_CLIENT, NULL);
if (IS_ERR(wdev)) {
mwifiex_dbg(adapter, ERROR,
"cannot create p2p client interface\n");
@@ -631,6 +631,7 @@
goto done;
err_add_intf:
+ vfree(adapter->chan_stats);
wiphy_unregister(adapter->wiphy);
wiphy_free(adapter->wiphy);
err_init_fw:
@@ -664,11 +665,18 @@
mwifiex_free_adapter(adapter);
/* Tell all current and future waiters we're finished */
complete_all(fw_done);
- return;
+
+ return init_failed ? -EIO : 0;
+}
+
+static void mwifiex_fw_dpc(const struct firmware *firmware, void *context)
+{
+ _mwifiex_fw_dpc(firmware, context);
}
/*
- * This function initializes the hardware and gets firmware.
+ * This function gets the firmware and (if called asynchronously) kicks off the
+ * HW init when done.
*/
static int mwifiex_init_hw_fw(struct mwifiex_adapter *adapter,
bool req_fw_nowait)
@@ -691,20 +699,15 @@
ret = request_firmware_nowait(THIS_MODULE, 1, adapter->fw_name,
adapter->dev, GFP_KERNEL, adapter,
mwifiex_fw_dpc);
- if (ret < 0)
- mwifiex_dbg(adapter, ERROR,
- "request_firmware_nowait error %d\n", ret);
} else {
ret = request_firmware(&adapter->firmware,
adapter->fw_name,
adapter->dev);
- if (ret < 0)
- mwifiex_dbg(adapter, ERROR,
- "request_firmware error %d\n", ret);
- else
- mwifiex_fw_dpc(adapter->firmware, (void *)adapter);
}
+ if (ret < 0)
+ mwifiex_dbg(adapter, ERROR, "request_firmware%s error %d\n",
+ req_fw_nowait ? "_nowait" : "", ret);
return ret;
}
@@ -745,7 +748,7 @@
mwifiex_dbg(priv->adapter, INFO,
"aborting bgscan on ndo_stop\n");
mwifiex_stop_bg_scan(priv);
- cfg80211_sched_scan_stopped(priv->wdev.wiphy);
+ cfg80211_sched_scan_stopped(priv->wdev.wiphy, 0);
}
return 0;
@@ -1413,6 +1416,7 @@
mwifiex_del_virtual_intf(adapter->wiphy, &priv->wdev);
rtnl_unlock();
}
+ vfree(adapter->chan_stats);
mwifiex_dbg(adapter, INFO, "%s, successful\n", __func__);
exit_return:
@@ -1426,6 +1430,8 @@
int
mwifiex_reinit_sw(struct mwifiex_adapter *adapter)
{
+ int ret;
+
mwifiex_init_lock_list(adapter);
if (adapter->if_ops.up_dev)
adapter->if_ops.up_dev(adapter);
@@ -1435,6 +1441,7 @@
init_waitqueue_head(&adapter->init_wait_q);
adapter->is_suspended = false;
adapter->hs_activated = false;
+ adapter->is_cmd_timedout = 0;
init_waitqueue_head(&adapter->hs_activate_wait_q);
init_waitqueue_head(&adapter->cmd_wait_q.wait);
adapter->cmd_wait_q.status = 0;
@@ -1472,9 +1479,15 @@
"%s: firmware init failed\n", __func__);
goto err_init_fw;
}
+
+ /* _mwifiex_fw_dpc() does its own cleanup */
+ ret = _mwifiex_fw_dpc(adapter->firmware, adapter);
+ if (ret) {
+ pr_err("Failed to bring up adapter: %d\n", ret);
+ return ret;
+ }
mwifiex_dbg(adapter, INFO, "%s, successful\n", __func__);
- complete_all(adapter->fw_done);
return 0;
err_init_fw:
@@ -1502,14 +1515,13 @@
{
struct mwifiex_adapter *adapter = priv;
- if (adapter->irq_wakeup >= 0) {
- dev_dbg(adapter->dev, "%s: wake by wifi", __func__);
- adapter->wake_by_wifi = true;
- disable_irq_nosync(irq);
- }
+ dev_dbg(adapter->dev, "%s: wake by wifi", __func__);
+ adapter->wake_by_wifi = true;
+ disable_irq_nosync(irq);
/* Notify PM core we are wakeup source */
pm_wakeup_event(adapter->dev, 0);
+ pm_system_wakeup();
return IRQ_HANDLED;
}
@@ -1714,6 +1726,7 @@
mwifiex_del_virtual_intf(adapter->wiphy, &priv->wdev);
rtnl_unlock();
}
+ vfree(adapter->chan_stats);
wiphy_unregister(adapter->wiphy);
wiphy_free(adapter->wiphy);
@@ -1742,7 +1755,7 @@
struct va_format vaf;
va_list args;
- if (!adapter->dev || !(adapter->debug_mask & mask))
+ if (!(adapter->debug_mask & mask))
return;
va_start(args, fmt);
@@ -1750,7 +1763,10 @@
vaf.fmt = fmt;
vaf.va = &args;
- dev_info(adapter->dev, "%pV", &vaf);
+ if (adapter->dev)
+ dev_info(adapter->dev, "%pV", &vaf);
+ else
+ pr_info("%pV", &vaf);
va_end(args);
}
diff --git a/drivers/net/wireless/marvell/mwifiex/main.h b/drivers/net/wireless/marvell/mwifiex/main.h
index 5c82972..bb2a467 100644
--- a/drivers/net/wireless/marvell/mwifiex/main.h
+++ b/drivers/net/wireless/marvell/mwifiex/main.h
@@ -1359,7 +1359,7 @@
*/
static inline bool mwifiex_is_skb_mgmt_frame(struct sk_buff *skb)
{
- return (le32_to_cpu(*(__le32 *)skb->data) == PKT_TYPE_MGMT);
+ return (get_unaligned_le32(skb->data) == PKT_TYPE_MGMT);
}
/* This function retrieves channel closed for operation by Channel
@@ -1529,7 +1529,6 @@
const char *name,
unsigned char name_assign_type,
enum nl80211_iftype type,
- u32 *flags,
struct vif_params *params);
int mwifiex_del_virtual_intf(struct wiphy *wiphy, struct wireless_dev *wdev);
diff --git a/drivers/net/wireless/marvell/mwifiex/pcie.c b/drivers/net/wireless/marvell/mwifiex/pcie.c
index b8c990d..ac62bce 100644
--- a/drivers/net/wireless/marvell/mwifiex/pcie.c
+++ b/drivers/net/wireless/marvell/mwifiex/pcie.c
@@ -119,7 +119,7 @@
*/
static bool mwifiex_pcie_ok_to_access_hw(struct mwifiex_adapter *adapter)
{
- u32 *cookie_addr;
+ u32 cookie_value;
struct pcie_service_card *card = adapter->card;
const struct mwifiex_pcie_card_reg *reg = card->pcie.reg;
@@ -127,11 +127,11 @@
return true;
if (card->sleep_cookie_vbase) {
- cookie_addr = (u32 *)card->sleep_cookie_vbase;
+ cookie_value = get_unaligned_le32(card->sleep_cookie_vbase);
mwifiex_dbg(adapter, INFO,
"info: ACCESS_HW: sleep cookie=0x%x\n",
- *cookie_addr);
- if (*cookie_addr == FW_AWAKE_COOKIE)
+ cookie_value);
+ if (cookie_value == FW_AWAKE_COOKIE)
return true;
}
@@ -294,8 +294,6 @@
if (!adapter || !adapter->priv_num)
return;
- cancel_work_sync(&card->work);
-
reg = card->pcie.reg;
if (reg)
ret = mwifiex_read_reg(adapter, reg->fw_status, &fw_status);
@@ -350,22 +348,16 @@
static void mwifiex_pcie_reset_notify(struct pci_dev *pdev, bool prepare)
{
- struct mwifiex_adapter *adapter;
- struct pcie_service_card *card;
+ struct pcie_service_card *card = pci_get_drvdata(pdev);
+ struct mwifiex_adapter *adapter = card->adapter;
+ int ret;
- if (!pdev) {
- pr_err("%s: PCIe device is not specified\n", __func__);
+ if (!adapter) {
+ dev_err(&pdev->dev, "%s: adapter structure is not valid\n",
+ __func__);
return;
}
- card = (struct pcie_service_card *)pci_get_drvdata(pdev);
- if (!card || !card->adapter) {
- pr_err("%s: Card or adapter structure is not valid (%ld)\n",
- __func__, (long)card);
- return;
- }
-
- adapter = card->adapter;
mwifiex_dbg(adapter, INFO,
"%s: vendor=0x%4.04x device=0x%4.04x rev=%d %s\n",
__func__, pdev->vendor, pdev->device,
@@ -379,13 +371,19 @@
*/
mwifiex_shutdown_sw(adapter);
adapter->surprise_removed = true;
+ clear_bit(MWIFIEX_IFACE_WORK_DEVICE_DUMP, &card->work_flags);
+ clear_bit(MWIFIEX_IFACE_WORK_CARD_RESET, &card->work_flags);
} else {
/* Kernel stores and restores PCIe function context before and
* after performing FLR respectively. Reconfigure the software
* and firmware including firmware redownload
*/
adapter->surprise_removed = false;
- mwifiex_reinit_sw(adapter);
+ ret = mwifiex_reinit_sw(adapter);
+ if (ret) {
+ dev_err(&pdev->dev, "reinit failed: %d\n", ret);
+ return;
+ }
}
mwifiex_dbg(adapter, INFO, "%s, successful\n", __func__);
}
@@ -447,7 +445,7 @@
sizeof(sleep_cookie),
PCI_DMA_FROMDEVICE);
buffer = cmdrsp->data;
- sleep_cookie = READ_ONCE(*(u32 *)buffer);
+ sleep_cookie = get_unaligned_le32(buffer);
if (sleep_cookie == MWIFIEX_DEF_SLEEP_COOKIE) {
mwifiex_dbg(adapter, INFO,
@@ -1039,6 +1037,7 @@
if (card && card->cmd_buf) {
mwifiex_unmap_pci_memory(adapter, card->cmd_buf,
PCI_DMA_TODEVICE);
+ dev_kfree_skb_any(card->cmd_buf);
}
return 0;
}
@@ -1049,6 +1048,7 @@
static int mwifiex_pcie_alloc_sleep_cookie_buf(struct mwifiex_adapter *adapter)
{
struct pcie_service_card *card = adapter->card;
+ u32 tmp;
card->sleep_cookie_vbase = pci_alloc_consistent(card->dev, sizeof(u32),
&card->sleep_cookie_pbase);
@@ -1058,11 +1058,12 @@
return -ENOMEM;
}
/* Init val of Sleep Cookie */
- *(u32 *)card->sleep_cookie_vbase = FW_AWAKE_COOKIE;
+ tmp = FW_AWAKE_COOKIE;
+ put_unaligned(tmp, card->sleep_cookie_vbase);
mwifiex_dbg(adapter, INFO,
"alloc_scook: sleep cookie=0x%x\n",
- *((u32 *)card->sleep_cookie_vbase));
+ get_unaligned(card->sleep_cookie_vbase));
return 0;
}
@@ -1223,7 +1224,6 @@
dma_addr_t buf_pa;
struct mwifiex_pcie_buf_desc *desc = NULL;
struct mwifiex_pfu_buf_desc *desc2 = NULL;
- __le16 *tmp;
if (!(skb->data && skb->len)) {
mwifiex_dbg(adapter, ERROR,
@@ -1244,10 +1244,8 @@
adapter->data_sent = true;
payload = skb->data;
- tmp = (__le16 *)&payload[0];
- *tmp = cpu_to_le16((u16)skb->len);
- tmp = (__le16 *)&payload[2];
- *tmp = cpu_to_le16(MWIFIEX_TYPE_DATA);
+ put_unaligned_le16((u16)skb->len, payload + 0);
+ put_unaligned_le16(MWIFIEX_TYPE_DATA, payload + 2);
if (mwifiex_map_pci_memory(adapter, skb, skb->len,
PCI_DMA_TODEVICE))
@@ -1376,7 +1374,6 @@
(card->rxbd_rdptr & reg->rx_rollover_ind))) {
struct sk_buff *skb_data;
u16 rx_len;
- __le16 pkt_len;
rd_index = card->rxbd_rdptr & reg->rx_mask;
skb_data = card->rx_buf_list[rd_index];
@@ -1393,8 +1390,7 @@
/* Get data length from interface header -
* first 2 bytes for len, next 2 bytes is for type
*/
- pkt_len = *((__le16 *)skb_data->data);
- rx_len = le16_to_cpu(pkt_len);
+ rx_len = get_unaligned_le16(skb_data->data);
if (WARN_ON(rx_len <= INTF_HEADER_LEN ||
rx_len > MWIFIEX_RX_DATA_BUF_SIZE)) {
mwifiex_dbg(adapter, ERROR,
@@ -1601,13 +1597,18 @@
adapter->cmd_sent = true;
- *(__le16 *)&payload[0] = cpu_to_le16((u16)skb->len);
- *(__le16 *)&payload[2] = cpu_to_le16(MWIFIEX_TYPE_CMD);
+ put_unaligned_le16((u16)skb->len, &payload[0]);
+ put_unaligned_le16(MWIFIEX_TYPE_CMD, &payload[2]);
if (mwifiex_map_pci_memory(adapter, skb, skb->len, PCI_DMA_TODEVICE))
return -1;
card->cmd_buf = skb;
+ /*
+ * Need to keep a reference, since core driver might free up this
+ * buffer before we've unmapped it.
+ */
+ skb_get(skb);
/* To send a command, the driver will:
1. Write the 64bit physical address of the data buffer to
@@ -1694,7 +1695,6 @@
struct sk_buff *skb = card->cmdrsp_buf;
int count = 0;
u16 rx_len;
- __le16 pkt_len;
mwifiex_dbg(adapter, CMD,
"info: Rx CMD Response\n");
@@ -1711,11 +1711,11 @@
if (card->cmd_buf) {
mwifiex_unmap_pci_memory(adapter, card->cmd_buf,
PCI_DMA_TODEVICE);
+ dev_kfree_skb_any(card->cmd_buf);
card->cmd_buf = NULL;
}
- pkt_len = *((__le16 *)skb->data);
- rx_len = le16_to_cpu(pkt_len);
+ rx_len = get_unaligned_le16(skb->data);
skb_put(skb, MWIFIEX_UPLD_SIZE - skb->len);
skb_trim(skb, rx_len);
@@ -1856,7 +1856,7 @@
desc = card->evtbd_ring[rdptr];
memset(desc, 0, sizeof(*desc));
- event = *(u32 *) &skb_cmd->data[INTF_HEADER_LEN];
+ event = get_unaligned_le32(&skb_cmd->data[INTF_HEADER_LEN]);
adapter->event_cause = event;
/* The first 4bytes will be the event transfer header
len is 2 bytes followed by type which is 2 bytes */
@@ -1965,6 +1965,94 @@
return ret;
}
+/* Combo firmware image is a combination of
+ * (1) combo crc heaer, start with CMD5
+ * (2) bluetooth image, start with CMD7, end with CMD6, data wrapped in CMD1.
+ * (3) wifi image.
+ *
+ * This function bypass the header and bluetooth part, return
+ * the offset of tail wifi-only part.
+ */
+
+static int mwifiex_extract_wifi_fw(struct mwifiex_adapter *adapter,
+ const void *firmware, u32 firmware_len) {
+ const struct mwifiex_fw_data *fwdata;
+ u32 offset = 0, data_len, dnld_cmd;
+ int ret = 0;
+ bool cmd7_before = false;
+
+ while (1) {
+ /* Check for integer and buffer overflow */
+ if (offset + sizeof(fwdata->header) < sizeof(fwdata->header) ||
+ offset + sizeof(fwdata->header) >= firmware_len) {
+ mwifiex_dbg(adapter, ERROR,
+ "extract wifi-only fw failure!\n");
+ ret = -1;
+ goto done;
+ }
+
+ fwdata = firmware + offset;
+ dnld_cmd = le32_to_cpu(fwdata->header.dnld_cmd);
+ data_len = le32_to_cpu(fwdata->header.data_length);
+
+ /* Skip past header */
+ offset += sizeof(fwdata->header);
+
+ switch (dnld_cmd) {
+ case MWIFIEX_FW_DNLD_CMD_1:
+ if (!cmd7_before) {
+ mwifiex_dbg(adapter, ERROR,
+ "no cmd7 before cmd1!\n");
+ ret = -1;
+ goto done;
+ }
+ if (offset + data_len < data_len) {
+ mwifiex_dbg(adapter, ERROR, "bad FW parse\n");
+ ret = -1;
+ goto done;
+ }
+ offset += data_len;
+ break;
+ case MWIFIEX_FW_DNLD_CMD_5:
+ /* Check for integer overflow */
+ if (offset + data_len < data_len) {
+ mwifiex_dbg(adapter, ERROR, "bad FW parse\n");
+ ret = -1;
+ goto done;
+ }
+ offset += data_len;
+ break;
+ case MWIFIEX_FW_DNLD_CMD_6:
+ /* Check for integer overflow */
+ if (offset + data_len < data_len) {
+ mwifiex_dbg(adapter, ERROR, "bad FW parse\n");
+ ret = -1;
+ goto done;
+ }
+ offset += data_len;
+ if (offset >= firmware_len) {
+ mwifiex_dbg(adapter, ERROR,
+ "extract wifi-only fw failure!\n");
+ ret = -1;
+ } else {
+ ret = offset;
+ }
+ goto done;
+ case MWIFIEX_FW_DNLD_CMD_7:
+ cmd7_before = true;
+ break;
+ default:
+ mwifiex_dbg(adapter, ERROR, "unknown dnld_cmd %d\n",
+ dnld_cmd);
+ ret = -1;
+ goto done;
+ }
+ }
+
+done:
+ return ret;
+}
+
/*
* This function downloads the firmware to the card.
*
@@ -1980,7 +2068,7 @@
u32 firmware_len = fw->fw_len;
u32 offset = 0;
struct sk_buff *skb;
- u32 txlen, tx_blocks = 0, tries, len;
+ u32 txlen, tx_blocks = 0, tries, len, val;
u32 block_retry_cnt = 0;
struct pcie_service_card *card = adapter->card;
const struct mwifiex_pcie_card_reg *reg = card->pcie.reg;
@@ -2007,6 +2095,24 @@
goto done;
}
+ ret = mwifiex_read_reg(adapter, PCIE_SCRATCH_13_REG, &val);
+ if (ret) {
+ mwifiex_dbg(adapter, FATAL, "Failed to read scratch register 13\n");
+ goto done;
+ }
+
+ /* PCIE FLR case: extract wifi part from combo firmware*/
+ if (val == MWIFIEX_PCIE_FLR_HAPPENS) {
+ ret = mwifiex_extract_wifi_fw(adapter, firmware, firmware_len);
+ if (ret < 0) {
+ mwifiex_dbg(adapter, ERROR, "Failed to extract wifi fw\n");
+ goto done;
+ }
+ offset = ret;
+ mwifiex_dbg(adapter, MSG,
+ "info: dnld wifi firmware from %d bytes\n", offset);
+ }
+
/* Perform firmware data transfer */
do {
u32 ireg_intr = 0;
@@ -2503,8 +2609,8 @@
struct pcie_service_card *card = adapter->card;
const struct mwifiex_pcie_card_reg *reg = card->pcie.reg;
int pcie_scratch_reg[] = {PCIE_SCRATCH_12_REG,
- PCIE_SCRATCH_13_REG,
- PCIE_SCRATCH_14_REG};
+ PCIE_SCRATCH_14_REG,
+ PCIE_SCRATCH_15_REG};
if (!p)
return 0;
@@ -2874,6 +2980,8 @@
int ret;
u32 fw_status;
+ cancel_work_sync(&card->work);
+
ret = mwifiex_read_reg(adapter, reg->fw_status, &fw_status);
if (fw_status == FIRMWARE_READY_PCIE) {
mwifiex_dbg(adapter, INFO,
@@ -3077,12 +3185,6 @@
struct pci_dev *pdev = card->dev;
const struct mwifiex_pcie_card_reg *reg = card->pcie.reg;
- /* Bluetooth is not on pcie interface. Download Wifi only firmware
- * during pcie FLR, so that bluetooth part of firmware which is
- * already running doesn't get affected.
- */
- strcpy(adapter->fw_name, PCIE8997_DEFAULT_WIFIFW_NAME);
-
/* tx_buf_size might be changed to 3584 by firmware during
* data transfer, we should reset it to default size.
*/
diff --git a/drivers/net/wireless/marvell/mwifiex/pcie.h b/drivers/net/wireless/marvell/mwifiex/pcie.h
index 00e8ee5..f7ce9b6 100644
--- a/drivers/net/wireless/marvell/mwifiex/pcie.h
+++ b/drivers/net/wireless/marvell/mwifiex/pcie.h
@@ -35,7 +35,6 @@
#define PCIE8897_B0_FW_NAME "mrvl/pcie8897_uapsta.bin"
#define PCIEUART8997_FW_NAME_V4 "mrvl/pcieuart8997_combo_v4.bin"
#define PCIEUSB8997_FW_NAME_V4 "mrvl/pcieusb8997_combo_v4.bin"
-#define PCIE8997_DEFAULT_WIFIFW_NAME "mrvl/pcie8997_wlan_v4.bin"
#define PCIE_VENDOR_ID_MARVELL (0x11ab)
#define PCIE_VENDOR_ID_V2_MARVELL (0x1b4b)
@@ -77,8 +76,9 @@
#define PCIE_SCRATCH_10_REG 0xCE8
#define PCIE_SCRATCH_11_REG 0xCEC
#define PCIE_SCRATCH_12_REG 0xCF0
-#define PCIE_SCRATCH_13_REG 0xCF8
-#define PCIE_SCRATCH_14_REG 0xCFC
+#define PCIE_SCRATCH_13_REG 0xCF4
+#define PCIE_SCRATCH_14_REG 0xCF8
+#define PCIE_SCRATCH_15_REG 0xCFC
#define PCIE_RD_DATA_PTR_Q0_Q1 0xC08C
#define PCIE_WR_DATA_PTR_Q0_Q1 0xC05C
@@ -119,6 +119,8 @@
#define MWIFIEX_SLEEP_COOKIE_SIZE 4
#define MWIFIEX_MAX_DELAY_COUNT 100
+#define MWIFIEX_PCIE_FLR_HAPPENS 0xFEDCBABA
+
struct mwifiex_pcie_card_reg {
u16 cmd_addr_lo;
u16 cmd_addr_hi;
@@ -217,8 +219,8 @@
.ring_tx_start_ptr = MWIFIEX_BD_FLAG_TX_START_PTR,
.pfu_enabled = 1,
.sleep_cookie = 0,
- .fw_dump_ctrl = 0xcf4,
- .fw_dump_start = 0xcf8,
+ .fw_dump_ctrl = PCIE_SCRATCH_13_REG,
+ .fw_dump_start = PCIE_SCRATCH_14_REG,
.fw_dump_end = 0xcff,
.fw_dump_host_ready = 0xee,
.fw_dump_read_done = 0xfe,
@@ -254,8 +256,8 @@
.ring_tx_start_ptr = MWIFIEX_BD_FLAG_TX_START_PTR,
.pfu_enabled = 1,
.sleep_cookie = 0,
- .fw_dump_ctrl = 0xcf4,
- .fw_dump_start = 0xcf8,
+ .fw_dump_ctrl = PCIE_SCRATCH_13_REG,
+ .fw_dump_start = PCIE_SCRATCH_14_REG,
.fw_dump_end = 0xcff,
.fw_dump_host_ready = 0xcc,
.fw_dump_read_done = 0xdd,
diff --git a/drivers/net/wireless/marvell/mwifiex/scan.c b/drivers/net/wireless/marvell/mwifiex/scan.c
index 1816916..ce6936d 100644
--- a/drivers/net/wireless/marvell/mwifiex/scan.c
+++ b/drivers/net/wireless/marvell/mwifiex/scan.c
@@ -691,8 +691,9 @@
/* Increment the TLV header length by the size
appended */
- le16_add_cpu(&chan_tlv_out->header.len,
- sizeof(chan_tlv_out->chan_scan_param));
+ le16_unaligned_add_cpu(&chan_tlv_out->header.len,
+ sizeof(
+ chan_tlv_out->chan_scan_param));
/*
* The tlv buffer length is set to the number of bytes
@@ -859,6 +860,7 @@
*scan_current_only = false;
if (user_scan_in) {
+ u8 tmpaddr[ETH_ALEN];
/* Default the ssid_filter flag to TRUE, set false under
certain wildcard conditions and qualified by the existence
@@ -883,8 +885,10 @@
user_scan_in->specific_bssid,
sizeof(scan_cfg_out->specific_bssid));
+ memcpy(tmpaddr, scan_cfg_out->specific_bssid, ETH_ALEN);
+
if (adapter->ext_scan &&
- !is_zero_ether_addr(scan_cfg_out->specific_bssid)) {
+ !is_zero_ether_addr(tmpaddr)) {
bssid_tlv =
(struct mwifiex_ie_types_bssid_list *)tlv_pos;
bssid_tlv->header.type = cpu_to_le16(TLV_TYPE_BSSID);
@@ -947,8 +951,9 @@
* truncate scan results. That is not an issue with an SSID
* or BSSID filter applied to the scan results in the firmware.
*/
+ memcpy(tmpaddr, scan_cfg_out->specific_bssid, ETH_ALEN);
if ((i && ssid_filter) ||
- !is_zero_ether_addr(scan_cfg_out->specific_bssid))
+ !is_zero_ether_addr(tmpaddr))
*filtered_scan = true;
if (user_scan_in->scan_chan_gap) {
@@ -989,10 +994,15 @@
* If a specific BSSID or SSID is used, the number of channels in the
* scan command will be increased to the absolute maximum.
*/
- if (*filtered_scan)
+ if (*filtered_scan) {
*max_chan_per_scan = MWIFIEX_MAX_CHANNELS_PER_SPECIFIC_SCAN;
- else
- *max_chan_per_scan = MWIFIEX_DEF_CHANNELS_PER_SCAN_CMD;
+ } else {
+ if (!priv->media_connected)
+ *max_chan_per_scan = MWIFIEX_DEF_CHANNELS_PER_SCAN_CMD;
+ else
+ *max_chan_per_scan =
+ MWIFIEX_DEF_CHANNELS_PER_SCAN_CMD / 2;
+ }
if (adapter->ext_scan) {
bss_mode = (struct mwifiex_ie_types_bss_mode *)tlv_pos;
@@ -1742,7 +1752,7 @@
if (*bytes_left >= sizeof(beacon_size)) {
/* Extract & convert beacon size from command buffer */
- beacon_size = le16_to_cpu(*(__le16 *)(*bss_info));
+ beacon_size = get_unaligned_le16((*bss_info));
*bytes_left -= sizeof(beacon_size);
*bss_info += sizeof(beacon_size);
}
@@ -2369,8 +2379,9 @@
temp_chan = chan_list_tlv->chan_scan_param + chan_idx;
/* Increment the TLV header length by size appended */
- le16_add_cpu(&chan_list_tlv->header.len,
- sizeof(chan_list_tlv->chan_scan_param));
+ le16_unaligned_add_cpu(&chan_list_tlv->header.len,
+ sizeof(
+ chan_list_tlv->chan_scan_param));
temp_chan->chan_number =
bgscan_cfg_in->chan_list[chan_idx].chan_number;
@@ -2407,8 +2418,8 @@
mwifiex_bgscan_create_channel_list(priv, bgscan_cfg_in,
chan_list_tlv->
chan_scan_param);
- le16_add_cpu(&chan_list_tlv->header.len,
- chan_num *
+ le16_unaligned_add_cpu(&chan_list_tlv->header.len,
+ chan_num *
sizeof(chan_list_tlv->chan_scan_param[0]));
}
@@ -2432,7 +2443,7 @@
/* Append vendor specific IE TLV */
mwifiex_cmd_append_vsie_tlv(priv, MWIFIEX_VSIE_MASK_BGSCAN, &tlv_pos);
- le16_add_cpu(&cmd->size, tlv_pos - bgscan_config->tlv);
+ le16_unaligned_add_cpu(&cmd->size, tlv_pos - bgscan_config->tlv);
return 0;
}
diff --git a/drivers/net/wireless/marvell/mwifiex/sdio.c b/drivers/net/wireless/marvell/mwifiex/sdio.c
index a4b356d..0af1c67 100644
--- a/drivers/net/wireless/marvell/mwifiex/sdio.c
+++ b/drivers/net/wireless/marvell/mwifiex/sdio.c
@@ -387,8 +387,6 @@
if (!adapter || !adapter->priv_num)
return;
- cancel_work_sync(&card->work);
-
mwifiex_dbg(adapter, INFO, "info: SDIO func num=%d\n", func->num);
ret = mwifiex_sdio_read_fw_status(adapter, &firmware_stat);
@@ -943,7 +941,7 @@
return -1;
}
- nb = le16_to_cpu(*(__le16 *) (buffer));
+ nb = get_unaligned_le16((buffer));
if (nb > npayload) {
mwifiex_dbg(adapter, ERROR,
"%s: invalid packet, nb=%d npayload=%d\n",
@@ -951,7 +949,7 @@
return -1;
}
- *type = le16_to_cpu(*(__le16 *) (buffer + 2));
+ *type = get_unaligned_le16((buffer + 2));
return ret;
}
@@ -1139,7 +1137,8 @@
__func__, blk_num, blk_size, total_pkt_len);
break;
}
- pkt_len = le16_to_cpu(*(__le16 *)(data + SDIO_HEADER_OFFSET));
+ pkt_len = get_unaligned_le16((data +
+ SDIO_HEADER_OFFSET));
if ((pkt_len + SDIO_HEADER_OFFSET) > blk_size) {
mwifiex_dbg(adapter, ERROR,
"%s: error in pkt_len,\t"
@@ -1172,10 +1171,11 @@
struct sk_buff *skb, u32 upld_typ)
{
u8 *cmd_buf;
- __le16 *curr_ptr = (__le16 *)skb->data;
- u16 pkt_len = le16_to_cpu(*curr_ptr);
+ u16 pkt_len;
struct mwifiex_rxinfo *rx_info;
+ pkt_len = get_unaligned_le16(skb->data);
+
if (upld_typ != MWIFIEX_TYPE_AGGR_DATA) {
skb_trim(skb, pkt_len);
skb_pull(skb, INTF_HEADER_LEN);
@@ -1235,7 +1235,7 @@
case MWIFIEX_TYPE_EVENT:
mwifiex_dbg(adapter, EVENT,
"info: --- Rx: Event ---\n");
- adapter->event_cause = le32_to_cpu(*(__le32 *) skb->data);
+ adapter->event_cause = get_unaligned_le32(skb->data);
if ((skb->len > 0) && (skb->len < MAX_EVENT_SIZE))
memcpy(adapter->event_body,
@@ -1380,7 +1380,7 @@
}
if (card->mpa_rx.pkt_cnt == 1)
- mport = adapter->ioport + port;
+ mport = adapter->ioport + card->mpa_rx.start_port;
if (mwifiex_read_data_sync(adapter, card->mpa_rx.buf,
card->mpa_rx.buf_len, mport, 1))
@@ -1392,8 +1392,8 @@
u32 *len_arr = card->mpa_rx.len_arr;
/* get curr PKT len & type */
- pkt_len = le16_to_cpu(*(__le16 *) &curr_ptr[0]);
- pkt_type = le16_to_cpu(*(__le16 *) &curr_ptr[2]);
+ pkt_len = get_unaligned_le16(&curr_ptr[0]);
+ pkt_type = get_unaligned_le16(&curr_ptr[2]);
/* copy pkt to deaggr buf */
skb_deaggr = mwifiex_alloc_dma_align_buf(len_arr[pind],
@@ -1813,7 +1813,7 @@
}
if (card->mpa_tx.pkt_cnt == 1)
- mport = adapter->ioport + port;
+ mport = adapter->ioport + card->mpa_tx.start_port;
ret = mwifiex_write_data_to_card(adapter, card->mpa_tx.buf,
card->mpa_tx.buf_len, mport);
@@ -1874,8 +1874,9 @@
/* Allocate buffer and copy payload */
blk_size = MWIFIEX_SDIO_BLOCK_SIZE;
buf_block_len = (pkt_len + blk_size - 1) / blk_size;
- *(__le16 *)&payload[0] = cpu_to_le16((u16)pkt_len);
- *(__le16 *)&payload[2] = cpu_to_le16(type);
+ put_unaligned_le16((u16)pkt_len, payload + 0);
+ put_unaligned_le16((u32)type, payload + 2);
+
/*
* This is SDIO specific header
@@ -2155,6 +2156,8 @@
{
struct sdio_mmc_card *card = adapter->card;
+ cancel_work_sync(&card->work);
+
kfree(card->mp_regs);
kfree(card->mpa_rx.skb_arr);
kfree(card->mpa_rx.len_arr);
@@ -2193,6 +2196,7 @@
{
struct sdio_mmc_card *card = adapter->card;
struct sdio_func *func = card->func;
+ int ret;
mwifiex_shutdown_sw(adapter);
@@ -2207,7 +2211,9 @@
clear_bit(MWIFIEX_IFACE_WORK_DEVICE_DUMP, &card->work_flags);
clear_bit(MWIFIEX_IFACE_WORK_CARD_RESET, &card->work_flags);
- mwifiex_reinit_sw(adapter);
+ ret = mwifiex_reinit_sw(adapter);
+ if (ret)
+ dev_err(&func->dev, "reinit failed: %d\n", ret);
}
/* This function read/write firmware */
diff --git a/drivers/net/wireless/marvell/mwifiex/sta_cmd.c b/drivers/net/wireless/marvell/mwifiex/sta_cmd.c
index 2f1f4d1..83916c1 100644
--- a/drivers/net/wireless/marvell/mwifiex/sta_cmd.c
+++ b/drivers/net/wireless/marvell/mwifiex/sta_cmd.c
@@ -126,19 +126,19 @@
if (cmd_action == HostCmd_ACT_GEN_GET) {
snmp_mib->query_type = cpu_to_le16(HostCmd_ACT_GEN_GET);
snmp_mib->buf_size = cpu_to_le16(MAX_SNMP_BUF_SIZE);
- le16_add_cpu(&cmd->size, MAX_SNMP_BUF_SIZE);
+ le16_unaligned_add_cpu(&cmd->size, MAX_SNMP_BUF_SIZE);
} else if (cmd_action == HostCmd_ACT_GEN_SET) {
snmp_mib->query_type = cpu_to_le16(HostCmd_ACT_GEN_SET);
snmp_mib->buf_size = cpu_to_le16(sizeof(u16));
- *((__le16 *) (snmp_mib->value)) = cpu_to_le16(*ul_temp);
- le16_add_cpu(&cmd->size, sizeof(u16));
+ put_unaligned_le16(*ul_temp, snmp_mib->value);
+ le16_unaligned_add_cpu(&cmd->size, sizeof(u16));
}
mwifiex_dbg(priv->adapter, CMD,
"cmd: SNMP_CMD: Action=0x%x, OID=0x%x,\t"
"OIDSize=0x%x, Value=0x%x\n",
cmd_action, cmd_oid, le16_to_cpu(snmp_mib->buf_size),
- le16_to_cpu(*(__le16 *)snmp_mib->value));
+ get_unaligned_le16(snmp_mib->value));
return 0;
}
@@ -1357,8 +1357,9 @@
subsc_evt_cfg->bcn_l_rssi_cfg.evt_freq);
pos += sizeof(struct mwifiex_ie_types_rssi_threshold);
- le16_add_cpu(&cmd->size,
- sizeof(struct mwifiex_ie_types_rssi_threshold));
+ le16_unaligned_add_cpu(&cmd->size,
+ sizeof(
+ struct mwifiex_ie_types_rssi_threshold));
}
if (event_bitmap & BITMASK_BCN_RSSI_HIGH) {
@@ -1378,8 +1379,9 @@
subsc_evt_cfg->bcn_h_rssi_cfg.evt_freq);
pos += sizeof(struct mwifiex_ie_types_rssi_threshold);
- le16_add_cpu(&cmd->size,
- sizeof(struct mwifiex_ie_types_rssi_threshold));
+ le16_unaligned_add_cpu(&cmd->size,
+ sizeof(
+ struct mwifiex_ie_types_rssi_threshold));
}
return 0;
@@ -1398,7 +1400,7 @@
filter = &mef_entry->filter[i];
if (!filter->filt_type)
break;
- *(__le32 *)stack_ptr = cpu_to_le32((u32)filter->repeat);
+ put_unaligned_le32((u32)filter->repeat, stack_ptr);
stack_ptr += 4;
*stack_ptr = TYPE_DNUM;
stack_ptr += 1;
@@ -1410,8 +1412,7 @@
stack_ptr += 1;
*stack_ptr = TYPE_BYTESEQ;
stack_ptr += 1;
-
- *(__le32 *)stack_ptr = cpu_to_le32((u32)filter->offset);
+ put_unaligned_le32((u32)filter->offset, stack_ptr);
stack_ptr += 4;
*stack_ptr = TYPE_DNUM;
stack_ptr += 1;
@@ -1683,14 +1684,15 @@
sizeof(u8) + sizeof(u8));
/* Add the rule length to the command size*/
- le16_add_cpu(&cmd->size, le16_to_cpu(rule->header.len) +
- sizeof(struct mwifiex_ie_types_header));
+ le16_unaligned_add_cpu(&cmd->size,
+ le16_to_cpu(rule->header.len) +
+ sizeof(struct mwifiex_ie_types_header));
rule = (void *)((u8 *)rule->params + length);
}
/* Add sizeof action, num_of_rules to total command length */
- le16_add_cpu(&cmd->size, sizeof(u16) + sizeof(u16));
+ le16_unaligned_add_cpu(&cmd->size, sizeof(u16) + sizeof(u16));
return 0;
}
@@ -1708,7 +1710,7 @@
cmd->command = cpu_to_le16(HostCmd_CMD_TDLS_CONFIG);
cmd->size = cpu_to_le16(S_DS_GEN);
tdls_config->tdls_action = cpu_to_le16(cmd_action);
- le16_add_cpu(&cmd->size, sizeof(tdls_config->tdls_action));
+ le16_unaligned_add_cpu(&cmd->size, sizeof(tdls_config->tdls_action));
switch (cmd_action) {
case ACT_TDLS_CS_ENABLE_CONFIG:
@@ -1735,7 +1737,7 @@
return -ENOTSUPP;
}
- le16_add_cpu(&cmd->size, len);
+ le16_unaligned_add_cpu(&cmd->size, len);
return 0;
}
@@ -1759,7 +1761,8 @@
cmd->command = cpu_to_le16(HostCmd_CMD_TDLS_OPER);
cmd->size = cpu_to_le16(S_DS_GEN);
- le16_add_cpu(&cmd->size, sizeof(struct host_cmd_ds_tdls_oper));
+ le16_unaligned_add_cpu(&cmd->size,
+ sizeof(struct host_cmd_ds_tdls_oper));
tdls_oper->reason = 0;
memcpy(tdls_oper->peer_mac, oper->peer_mac, ETH_ALEN);
@@ -1783,7 +1786,7 @@
return -ENODATA;
}
- *(__le16 *)pos = cpu_to_le16(params->capability);
+ put_unaligned_le16(params->capability, pos);
config_len += sizeof(params->capability);
qos_info = params->uapsd_queues | (params->max_sp << 5);
@@ -1861,7 +1864,7 @@
return -ENOTSUPP;
}
- le16_add_cpu(&cmd->size, config_len);
+ le16_unaligned_add_cpu(&cmd->size, config_len);
return 0;
}
@@ -2032,7 +2035,7 @@
case HostCmd_CMD_VERSION_EXT:
cmd_ptr->command = cpu_to_le16(cmd_no);
cmd_ptr->params.verext.version_str_sel =
- (u8) (*((u32 *) data_buf));
+ (u8)(get_unaligned((u32 *)data_buf));
memcpy(&cmd_ptr->params, data_buf,
sizeof(struct host_cmd_ds_version_ext));
cmd_ptr->size =
@@ -2043,7 +2046,8 @@
case HostCmd_CMD_MGMT_FRAME_REG:
cmd_ptr->command = cpu_to_le16(cmd_no);
cmd_ptr->params.reg_mask.action = cpu_to_le16(cmd_action);
- cmd_ptr->params.reg_mask.mask = cpu_to_le32(*(u32 *)data_buf);
+ cmd_ptr->params.reg_mask.mask = cpu_to_le32(
+ get_unaligned((u32 *)data_buf));
cmd_ptr->size =
cpu_to_le16(sizeof(struct host_cmd_ds_mgmt_frame_reg) +
S_DS_GEN);
@@ -2063,7 +2067,8 @@
case HostCmd_CMD_P2P_MODE_CFG:
cmd_ptr->command = cpu_to_le16(cmd_no);
cmd_ptr->params.mode_cfg.action = cpu_to_le16(cmd_action);
- cmd_ptr->params.mode_cfg.mode = cpu_to_le16(*(u16 *)data_buf);
+ cmd_ptr->params.mode_cfg.mode = cpu_to_le16(
+ get_unaligned((u16 *)data_buf));
cmd_ptr->size =
cpu_to_le16(sizeof(struct host_cmd_ds_p2p_mode_cfg) +
S_DS_GEN);
@@ -2359,8 +2364,7 @@
if (ret)
return -1;
- if (!disable_auto_ds &&
- first_sta && priv->adapter->iface_type != MWIFIEX_USB &&
+ if (!disable_auto_ds && first_sta &&
priv->bss_type != MWIFIEX_BSS_TYPE_UAP) {
/* Enable auto deep sleep */
auto_ds.auto_ds = DEEP_SLEEP_ON;
diff --git a/drivers/net/wireless/marvell/mwifiex/sta_cmdresp.c b/drivers/net/wireless/marvell/mwifiex/sta_cmdresp.c
index 8548027a..f1d1f56 100644
--- a/drivers/net/wireless/marvell/mwifiex/sta_cmdresp.c
+++ b/drivers/net/wireless/marvell/mwifiex/sta_cmdresp.c
@@ -183,7 +183,7 @@
"query_type = %#x, buf size = %#x\n",
oid, query_type, le16_to_cpu(smib->buf_size));
if (query_type == HostCmd_ACT_GEN_GET) {
- ul_temp = le16_to_cpu(*((__le16 *) (smib->value)));
+ ul_temp = get_unaligned_le16(smib->value);
if (data_buf)
*data_buf = ul_temp;
switch (oid) {
@@ -741,7 +741,7 @@
struct host_cmd_ds_p2p_mode_cfg *mode_cfg = &resp->params.mode_cfg;
if (data_buf)
- *((u16 *)data_buf) = le16_to_cpu(mode_cfg->mode);
+ put_unaligned_le16(le16_to_cpu(mode_cfg->mode), data_buf);
return 0;
}
@@ -1201,7 +1201,7 @@
break;
case HostCmd_CMD_802_11_BG_SCAN_QUERY:
ret = mwifiex_ret_802_11_scan(priv, resp);
- cfg80211_sched_scan_results(priv->wdev.wiphy);
+ cfg80211_sched_scan_results(priv->wdev.wiphy, 0);
mwifiex_dbg(adapter, CMD,
"info: CMD_RESP: BG_SCAN result is ready!\n");
break;
diff --git a/drivers/net/wireless/marvell/mwifiex/sta_event.c b/drivers/net/wireless/marvell/mwifiex/sta_event.c
index d63d163..839df8a 100644
--- a/drivers/net/wireless/marvell/mwifiex/sta_event.c
+++ b/drivers/net/wireless/marvell/mwifiex/sta_event.c
@@ -670,7 +670,7 @@
adapter->dbg.num_event_deauth++;
if (priv->media_connected) {
reason_code =
- le16_to_cpu(*(__le16 *)adapter->event_body);
+ get_unaligned_le16(adapter->event_body);
mwifiex_reset_connect_state(priv, reason_code, true);
}
break;
@@ -685,7 +685,7 @@
adapter->dbg.num_event_disassoc++;
if (priv->media_connected) {
reason_code =
- le16_to_cpu(*(__le16 *)adapter->event_body);
+ get_unaligned_le16(adapter->event_body);
mwifiex_reset_connect_state(priv, reason_code, true);
}
break;
@@ -695,7 +695,7 @@
adapter->dbg.num_event_link_lost++;
if (priv->media_connected) {
reason_code =
- le16_to_cpu(*(__le16 *)adapter->event_body);
+ get_unaligned_le16(adapter->event_body);
mwifiex_reset_connect_state(priv, reason_code, true);
}
break;
@@ -793,7 +793,7 @@
case EVENT_BG_SCAN_STOPPED:
dev_dbg(adapter->dev, "event: BGS_STOPPED\n");
- cfg80211_sched_scan_stopped(priv->wdev.wiphy);
+ cfg80211_sched_scan_stopped(priv->wdev.wiphy, 0);
if (priv->sched_scanning)
priv->sched_scanning = false;
break;
@@ -923,7 +923,7 @@
adapter->event_body);
break;
case EVENT_AMSDU_AGGR_CTRL:
- ctrl = le16_to_cpu(*(__le16 *)adapter->event_body);
+ ctrl = get_unaligned_le16(adapter->event_body);
mwifiex_dbg(adapter, EVENT,
"event: AMSDU_AGGR_CTRL %d\n", ctrl);
diff --git a/drivers/net/wireless/marvell/mwifiex/sta_ioctl.c b/drivers/net/wireless/marvell/mwifiex/sta_ioctl.c
index 1532ac9..42997e0 100644
--- a/drivers/net/wireless/marvell/mwifiex/sta_ioctl.c
+++ b/drivers/net/wireless/marvell/mwifiex/sta_ioctl.c
@@ -560,7 +560,7 @@
#endif
mwifiex_dbg(adapter, CMD, "aborting bgscan!\n");
mwifiex_stop_bg_scan(priv);
- cfg80211_sched_scan_stopped(priv->wdev.wiphy);
+ cfg80211_sched_scan_stopped(priv->wdev.wiphy, 0);
#ifdef CONFIG_PM
}
#endif
diff --git a/drivers/net/wireless/marvell/mwifiex/tdls.c b/drivers/net/wireless/marvell/mwifiex/tdls.c
index df9704de0..7d0d3ff 100644
--- a/drivers/net/wireless/marvell/mwifiex/tdls.c
+++ b/drivers/net/wireless/marvell/mwifiex/tdls.c
@@ -349,7 +349,7 @@
chan_bw = IEEE80211_VHT_CHANWIDTH_USE_HT;
break;
}
- vht_oper->center_freq_seg1_idx =
+ vht_oper->center_freq_seg0_idx =
mwifiex_get_center_freq_index(priv, BAND_AAC,
bss_desc->channel,
chan_bw);
@@ -431,6 +431,41 @@
*buf++ = qosinfo; /* U-APSD no in use */
}
+static void mwifiex_tdls_add_bss_co_2040(struct sk_buff *skb)
+{
+ struct ieee_types_bss_co_2040 *bssco;
+
+ bssco = (void *)skb_put(skb, sizeof(struct ieee_types_bss_co_2040));
+ bssco->ieee_hdr.element_id = WLAN_EID_BSS_COEX_2040;
+ bssco->ieee_hdr.len = sizeof(struct ieee_types_bss_co_2040) -
+ sizeof(struct ieee_types_header);
+ bssco->bss_2040co = 0x01;
+}
+
+static void mwifiex_tdls_add_supported_chan(struct sk_buff *skb)
+{
+ struct ieee_types_generic *supp_chan;
+ u8 chan_supp[] = {1, 11};
+
+ supp_chan = (void *)skb_put(skb, (sizeof(struct ieee_types_header) +
+ sizeof(chan_supp)));
+ supp_chan->ieee_hdr.element_id = WLAN_EID_SUPPORTED_CHANNELS;
+ supp_chan->ieee_hdr.len = sizeof(chan_supp);
+ memcpy(supp_chan->data, chan_supp, sizeof(chan_supp));
+}
+
+static void mwifiex_tdls_add_oper_class(struct sk_buff *skb)
+{
+ struct ieee_types_generic *reg_class;
+ u8 rc_list[] = {1,
+ 1, 2, 3, 4, 12, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33};
+ reg_class = (void *)skb_put(skb, (sizeof(struct ieee_types_header) +
+ sizeof(rc_list)));
+ reg_class->ieee_hdr.element_id = WLAN_EID_SUPPORTED_REGULATORY_CLASSES;
+ reg_class->ieee_hdr.len = sizeof(rc_list);
+ memcpy(reg_class->data, rc_list, sizeof(rc_list));
+}
+
static int mwifiex_prep_tdls_encap_data(struct mwifiex_private *priv,
const u8 *peer, u8 action_code,
u8 dialog_token,
@@ -484,7 +519,9 @@
}
mwifiex_tdls_add_ext_capab(priv, skb);
- mwifiex_tdls_add_qos_capab(skb);
+ mwifiex_tdls_add_bss_co_2040(skb);
+ mwifiex_tdls_add_supported_chan(skb);
+ mwifiex_tdls_add_oper_class(skb);
mwifiex_add_wmm_info_ie(priv, skb, 0);
break;
@@ -522,7 +559,9 @@
}
mwifiex_tdls_add_ext_capab(priv, skb);
- mwifiex_tdls_add_qos_capab(skb);
+ mwifiex_tdls_add_bss_co_2040(skb);
+ mwifiex_tdls_add_supported_chan(skb);
+ mwifiex_tdls_add_oper_class(skb);
mwifiex_add_wmm_info_ie(priv, skb, 0);
break;
@@ -612,6 +651,9 @@
sizeof(struct ieee_types_bss_co_2040) +
sizeof(struct ieee80211_ht_operation) +
sizeof(struct ieee80211_tdls_lnkie) +
+ (2 * (sizeof(struct ieee_types_header))) +
+ MWIFIEX_SUPPORTED_CHANNELS +
+ MWIFIEX_OPERATING_CLASSES +
sizeof(struct ieee80211_wmm_param_ie) +
extra_ies_len;
@@ -760,7 +802,10 @@
}
mwifiex_tdls_add_ext_capab(priv, skb);
+ mwifiex_tdls_add_bss_co_2040(skb);
+ mwifiex_tdls_add_supported_chan(skb);
mwifiex_tdls_add_qos_capab(skb);
+ mwifiex_tdls_add_oper_class(skb);
break;
default:
mwifiex_dbg(priv->adapter, ERROR, "Unknown TDLS action frame type\n");
@@ -857,7 +902,7 @@
struct mwifiex_sta_node *sta_ptr;
u8 *peer, *pos, *end;
u8 i, action, basic;
- __le16 cap = 0;
+ u16 cap = 0;
int ie_len = 0;
if (len < (sizeof(struct ethhdr) + 3))
@@ -879,7 +924,7 @@
pos = buf + sizeof(struct ethhdr) + 4;
/* payload 1+ category 1 + action 1 + dialog 1 */
- cap = cpu_to_le16(*(u16 *)pos);
+ cap = get_unaligned_le16(pos);
ie_len = len - sizeof(struct ethhdr) - TDLS_REQ_FIX_LEN;
pos += 2;
break;
@@ -889,7 +934,7 @@
return;
/* payload 1+ category 1 + action 1 + dialog 1 + status code 2*/
pos = buf + sizeof(struct ethhdr) + 6;
- cap = cpu_to_le16(*(u16 *)pos);
+ cap = get_unaligned_le16(pos);
ie_len = len - sizeof(struct ethhdr) - TDLS_RESP_FIX_LEN;
pos += 2;
break;
@@ -909,7 +954,7 @@
if (!sta_ptr)
return;
- sta_ptr->tdls_cap.capab = cap;
+ sta_ptr->tdls_cap.capab = cpu_to_le16(cap);
for (end = pos + ie_len; pos + 1 < end; pos += 2 + pos[1]) {
if (pos + 2 + pos[1] > end)
@@ -969,7 +1014,7 @@
case WLAN_EID_AID:
if (priv->adapter->is_hw_11ac_capable)
sta_ptr->tdls_cap.aid =
- le16_to_cpu(*(__le16 *)(pos + 2));
+ get_unaligned_le16((pos + 2));
default:
break;
}
diff --git a/drivers/net/wireless/marvell/mwifiex/uap_event.c b/drivers/net/wireless/marvell/mwifiex/uap_event.c
index d24eca3..e10b2a5 100644
--- a/drivers/net/wireless/marvell/mwifiex/uap_event.c
+++ b/drivers/net/wireless/marvell/mwifiex/uap_event.c
@@ -202,7 +202,7 @@
"AP EVENT: event id: %#x\n", eventcause);
break;
case EVENT_AMSDU_AGGR_CTRL:
- ctrl = le16_to_cpu(*(__le16 *)adapter->event_body);
+ ctrl = get_unaligned_le16(adapter->event_body);
mwifiex_dbg(adapter, EVENT,
"event: AMSDU_AGGR_CTRL %d\n", ctrl);
diff --git a/drivers/net/wireless/marvell/mwifiex/usb.c b/drivers/net/wireless/marvell/mwifiex/usb.c
index 9cf3334..2f7705c 100644
--- a/drivers/net/wireless/marvell/mwifiex/usb.c
+++ b/drivers/net/wireless/marvell/mwifiex/usb.c
@@ -306,9 +306,17 @@
}
}
- usb_fill_bulk_urb(ctx->urb, card->udev,
- usb_rcvbulkpipe(card->udev, ctx->ep), ctx->skb->data,
- size, mwifiex_usb_rx_complete, (void *)ctx);
+ if (card->rx_cmd_ep == ctx->ep &&
+ card->rx_cmd_ep_type == USB_ENDPOINT_XFER_INT)
+ usb_fill_int_urb(ctx->urb, card->udev,
+ usb_rcvintpipe(card->udev, ctx->ep),
+ ctx->skb->data, size, mwifiex_usb_rx_complete,
+ (void *)ctx, card->rx_cmd_interval);
+ else
+ usb_fill_bulk_urb(ctx->urb, card->udev,
+ usb_rcvbulkpipe(card->udev, ctx->ep),
+ ctx->skb->data, size, mwifiex_usb_rx_complete,
+ (void *)ctx);
if (card->rx_cmd_ep == ctx->ep)
atomic_inc(&card->rx_cmd_urb_pending);
@@ -424,10 +432,13 @@
epd = &iface_desc->endpoint[i].desc;
if (usb_endpoint_dir_in(epd) &&
usb_endpoint_num(epd) == MWIFIEX_USB_EP_CMD_EVENT &&
- usb_endpoint_xfer_bulk(epd)) {
- pr_debug("info: bulk IN: max pkt size: %d, addr: %d\n",
+ (usb_endpoint_xfer_bulk(epd) ||
+ usb_endpoint_xfer_int(epd))) {
+ card->rx_cmd_ep_type = usb_endpoint_type(epd);
+ card->rx_cmd_interval = epd->bInterval;
+ pr_debug("info: Rx CMD/EVT:: max pkt size: %d, addr: %d, ep_type: %d\n",
le16_to_cpu(epd->wMaxPacketSize),
- epd->bEndpointAddress);
+ epd->bEndpointAddress, card->rx_cmd_ep_type);
card->rx_cmd_ep = usb_endpoint_num(epd);
atomic_set(&card->rx_cmd_urb_pending, 0);
}
@@ -461,10 +472,16 @@
}
if (usb_endpoint_dir_out(epd) &&
usb_endpoint_num(epd) == MWIFIEX_USB_EP_CMD_EVENT &&
- usb_endpoint_xfer_bulk(epd)) {
+ (usb_endpoint_xfer_bulk(epd) ||
+ usb_endpoint_xfer_int(epd))) {
+ card->tx_cmd_ep_type = usb_endpoint_type(epd);
+ card->tx_cmd_interval = epd->bInterval;
pr_debug("info: bulk OUT: max pkt size: %d, addr: %d\n",
le16_to_cpu(epd->wMaxPacketSize),
epd->bEndpointAddress);
+ pr_debug("info: Tx CMD:: max pkt size: %d, addr: %d, ep_type: %d\n",
+ le16_to_cpu(epd->wMaxPacketSize),
+ epd->bEndpointAddress, card->tx_cmd_ep_type);
card->tx_cmd_ep = usb_endpoint_num(epd);
atomic_set(&card->tx_cmd_urb_pending, 0);
card->bulk_out_maxpktsize =
@@ -884,9 +901,17 @@
context->skb = skb;
tx_urb = context->urb;
- usb_fill_bulk_urb(tx_urb, card->udev, usb_sndbulkpipe(card->udev, ep),
- data, skb->len, mwifiex_usb_tx_complete,
- (void *)context);
+ if (ep == card->tx_cmd_ep &&
+ card->tx_cmd_ep_type == USB_ENDPOINT_XFER_INT)
+ usb_fill_int_urb(tx_urb, card->udev,
+ usb_sndintpipe(card->udev, ep), data,
+ skb->len, mwifiex_usb_tx_complete,
+ (void *)context, card->tx_cmd_interval);
+ else
+ usb_fill_bulk_urb(tx_urb, card->udev,
+ usb_sndbulkpipe(card->udev, ep), data,
+ skb->len, mwifiex_usb_tx_complete,
+ (void *)context);
tx_urb->transfer_flags |= URB_ZERO_PACKET;
diff --git a/drivers/net/wireless/marvell/mwifiex/usb.h b/drivers/net/wireless/marvell/mwifiex/usb.h
index e5f204e..e36bd63 100644
--- a/drivers/net/wireless/marvell/mwifiex/usb.h
+++ b/drivers/net/wireless/marvell/mwifiex/usb.h
@@ -90,6 +90,10 @@
struct urb_context tx_cmd;
u8 mc_resync_flag;
struct usb_tx_data_port port[MWIFIEX_TX_DATA_PORT];
+ int rx_cmd_ep_type;
+ u8 rx_cmd_interval;
+ int tx_cmd_ep_type;
+ u8 tx_cmd_interval;
};
struct fw_header {
@@ -102,12 +106,12 @@
struct fw_sync_header {
__le32 cmd;
__le32 seq_num;
-};
+} __packed;
struct fw_data {
struct fw_header fw_hdr;
__le32 seq_num;
u8 data[1];
-};
+} __packed;
#endif /*_MWIFIEX_USB_H */
diff --git a/drivers/net/wireless/marvell/mwifiex/util.c b/drivers/net/wireless/marvell/mwifiex/util.c
index b1ab8da..0cd68ff 100644
--- a/drivers/net/wireless/marvell/mwifiex/util.c
+++ b/drivers/net/wireless/marvell/mwifiex/util.c
@@ -274,13 +274,13 @@
val = *((u8 *)addr);
break;
case 2:
- val = *((u16 *)addr);
+ val = get_unaligned((u16 *)addr);
break;
case 4:
- val = *((u32 *)addr);
+ val = get_unaligned((u32 *)addr);
break;
case 8:
- val = *((long long *)addr);
+ val = get_unaligned((long long *)addr);
break;
default:
val = -1;
diff --git a/drivers/net/wireless/marvell/mwifiex/util.h b/drivers/net/wireless/marvell/mwifiex/util.h
index b541d66..c386992 100644
--- a/drivers/net/wireless/marvell/mwifiex/util.h
+++ b/drivers/net/wireless/marvell/mwifiex/util.h
@@ -93,4 +93,9 @@
int mwifiex_debug_info_to_buffer(struct mwifiex_private *priv, char *buf,
struct mwifiex_debug_info *info);
+static inline void le16_unaligned_add_cpu(__le16 *var, u16 val)
+{
+ put_unaligned_le16(get_unaligned_le16(var) + val, var);
+}
+
#endif /* !_MWIFIEX_UTIL_H_ */
diff --git a/drivers/net/wireless/marvell/mwl8k.c b/drivers/net/wireless/marvell/mwl8k.c
index b1b400b..e813b2c 100644
--- a/drivers/net/wireless/marvell/mwl8k.c
+++ b/drivers/net/wireless/marvell/mwl8k.c
@@ -994,9 +994,9 @@
*noise = -rxd->noise_floor;
if (rxd->rate & MWL8K_AP_RATE_INFO_MCS_FORMAT) {
- status->flag |= RX_FLAG_HT;
+ status->encoding = RX_ENC_HT;
if (rxd->rate & MWL8K_AP_RATE_INFO_40MHZ)
- status->flag |= RX_FLAG_40MHZ;
+ status->bw = RATE_INFO_BW_40;
status->rate_idx = MWL8K_AP_RATE_INFO_RATEID(rxd->rate);
} else {
int i;
@@ -1011,7 +1011,7 @@
if (rxd->channel > 14) {
status->band = NL80211_BAND_5GHZ;
- if (!(status->flag & RX_FLAG_HT))
+ if (!(status->encoding == RX_ENC_HT))
status->rate_idx -= 5;
} else {
status->band = NL80211_BAND_2GHZ;
@@ -1109,17 +1109,17 @@
status->rate_idx = MWL8K_STA_RATE_INFO_RATEID(rate_info);
if (rate_info & MWL8K_STA_RATE_INFO_SHORTPRE)
- status->flag |= RX_FLAG_SHORTPRE;
+ status->enc_flags |= RX_ENC_FLAG_SHORTPRE;
if (rate_info & MWL8K_STA_RATE_INFO_40MHZ)
- status->flag |= RX_FLAG_40MHZ;
+ status->bw = RATE_INFO_BW_40;
if (rate_info & MWL8K_STA_RATE_INFO_SHORTGI)
- status->flag |= RX_FLAG_SHORT_GI;
+ status->enc_flags |= RX_ENC_FLAG_SHORT_GI;
if (rate_info & MWL8K_STA_RATE_INFO_MCS_FORMAT)
- status->flag |= RX_FLAG_HT;
+ status->encoding = RX_ENC_HT;
if (rxd->channel > 14) {
status->band = NL80211_BAND_5GHZ;
- if (!(status->flag & RX_FLAG_HT))
+ if (!(status->encoding == RX_ENC_HT))
status->rate_idx -= 5;
} else {
status->band = NL80211_BAND_2GHZ;
@@ -6144,6 +6144,8 @@
if (priv->sta_macids_supported || priv->device_info->fw_image_sta)
hw->wiphy->interface_modes |= BIT(NL80211_IFTYPE_STATION);
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
rc = ieee80211_register_hw(hw);
if (rc) {
wiphy_err(hw->wiphy, "Cannot register device\n");
diff --git a/drivers/net/wireless/mediatek/mt7601u/init.c b/drivers/net/wireless/mediatek/mt7601u/init.c
index a6e9017..d3b611a 100644
--- a/drivers/net/wireless/mediatek/mt7601u/init.c
+++ b/drivers/net/wireless/mediatek/mt7601u/init.c
@@ -615,6 +615,8 @@
wiphy->features |= NL80211_FEATURE_ACTIVE_MONITOR;
wiphy->interface_modes = BIT(NL80211_IFTYPE_STATION);
+ wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
ret = mt76_init_sband_2g(dev);
if (ret)
return ret;
diff --git a/drivers/net/wireless/mediatek/mt7601u/mac.c b/drivers/net/wireless/mediatek/mt7601u/mac.c
index 3c57639..d6dc59b 100644
--- a/drivers/net/wireless/mediatek/mt7601u/mac.c
+++ b/drivers/net/wireless/mediatek/mt7601u/mac.c
@@ -401,7 +401,7 @@
case MT_PHY_TYPE_CCK:
if (idx >= 8) {
idx -= 8;
- status->flag |= RX_FLAG_SHORTPRE;
+ status->enc_flags |= RX_ENC_FLAG_SHORTPRE;
}
if (WARN_ON(idx >= 4))
@@ -410,10 +410,10 @@
status->rate_idx = idx;
return;
case MT_PHY_TYPE_HT_GF:
- status->flag |= RX_FLAG_HT_GF;
+ status->enc_flags |= RX_ENC_FLAG_HT_GF;
/* fall through */
case MT_PHY_TYPE_HT:
- status->flag |= RX_FLAG_HT;
+ status->encoding = RX_ENC_HT;
status->rate_idx = idx;
break;
default:
@@ -422,13 +422,13 @@
}
if (rate & MT_RXWI_RATE_SGI)
- status->flag |= RX_FLAG_SHORT_GI;
+ status->enc_flags |= RX_ENC_FLAG_SHORT_GI;
if (rate & MT_RXWI_RATE_STBC)
- status->flag |= 1 << RX_FLAG_STBC_SHIFT;
+ status->enc_flags |= 1 << RX_ENC_FLAG_STBC_SHIFT;
if (rate & MT_RXWI_RATE_BW)
- status->flag |= RX_FLAG_40MHZ;
+ status->bw = RATE_INFO_BW_40;
}
static void
diff --git a/drivers/net/wireless/mediatek/mt7601u/mcu.c b/drivers/net/wireless/mediatek/mt7601u/mcu.c
index dbdfb3f..a9f5f39 100644
--- a/drivers/net/wireless/mediatek/mt7601u/mcu.c
+++ b/drivers/net/wireless/mediatek/mt7601u/mcu.c
@@ -66,8 +66,10 @@
WARN_ON(len % 4); /* if length is not divisible by 4 we need to pad */
skb = alloc_skb(len + MT_DMA_HDR_LEN + 4, GFP_KERNEL);
- skb_reserve(skb, MT_DMA_HDR_LEN);
- memcpy(skb_put(skb, len), data, len);
+ if (skb) {
+ skb_reserve(skb, MT_DMA_HDR_LEN);
+ memcpy(skb_put(skb, len), data, len);
+ }
return skb;
}
@@ -170,6 +172,8 @@
};
skb = mt7601u_mcu_msg_alloc(dev, &msg, sizeof(msg));
+ if (!skb)
+ return -ENOMEM;
return mt7601u_mcu_msg_send(dev, skb, CMD_FUN_SET_OP, func == 5);
}
@@ -205,6 +209,8 @@
};
skb = mt7601u_mcu_msg_alloc(dev, &msg, sizeof(msg));
+ if (!skb)
+ return -ENOMEM;
return mt7601u_mcu_msg_send(dev, skb, CMD_CALIBRATION_OP, true);
}
diff --git a/drivers/net/wireless/ralink/rt2x00/Kconfig b/drivers/net/wireless/ralink/rt2x00/Kconfig
index de62f5d..a1d1cfe 100644
--- a/drivers/net/wireless/ralink/rt2x00/Kconfig
+++ b/drivers/net/wireless/ralink/rt2x00/Kconfig
@@ -201,7 +201,7 @@
config RT2800SOC
tristate "Ralink WiSoC support"
- depends on SOC_RT288X || SOC_RT305X
+ depends on SOC_RT288X || SOC_RT305X || SOC_MT7620
select RT2X00_LIB_SOC
select RT2X00_LIB_MMIO
select RT2X00_LIB_CRYPTO
diff --git a/drivers/net/wireless/ralink/rt2x00/rt2800.h b/drivers/net/wireless/ralink/rt2x00/rt2800.h
index 256496b..6a8c93f 100644
--- a/drivers/net/wireless/ralink/rt2x00/rt2800.h
+++ b/drivers/net/wireless/ralink/rt2x00/rt2800.h
@@ -79,6 +79,7 @@
#define RF5372 0x5372
#define RF5390 0x5390
#define RF5392 0x5392
+#define RF7620 0x7620
/*
* Chipset revisions.
@@ -639,6 +640,24 @@
#define RF_CSR_CFG_BUSY FIELD32(0x00020000)
/*
+ * MT7620 RF registers (reversed order)
+ */
+#define RF_CSR_CFG_DATA_MT7620 FIELD32(0x0000ff00)
+#define RF_CSR_CFG_REGNUM_MT7620 FIELD32(0x03ff0000)
+#define RF_CSR_CFG_WRITE_MT7620 FIELD32(0x00000010)
+#define RF_CSR_CFG_BUSY_MT7620 FIELD32(0x00000001)
+
+/* undocumented registers for calibration of new MAC */
+#define RF_CONTROL0 0x0518
+#define RF_BYPASS0 0x051c
+#define RF_CONTROL1 0x0520
+#define RF_BYPASS1 0x0524
+#define RF_CONTROL2 0x0528
+#define RF_BYPASS2 0x052c
+#define RF_CONTROL3 0x0530
+#define RF_BYPASS3 0x0534
+
+/*
* EFUSE_CSR: RT30x0 EEPROM
*/
#define EFUSE_CTRL 0x0580
@@ -1022,6 +1041,16 @@
#define AUTOWAKEUP_CFG_AUTOWAKE FIELD32(0x00008000)
/*
+ * MIMO_PS_CFG: MIMO Power-save Configuration
+ */
+#define MIMO_PS_CFG 0x1210
+#define MIMO_PS_CFG_MMPS_BB_EN FIELD32(0x00000001)
+#define MIMO_PS_CFG_MMPS_RX_ANT_NUM FIELD32(0x00000006)
+#define MIMO_PS_CFG_MMPS_RF_EN FIELD32(0x00000008)
+#define MIMO_PS_CFG_RX_STBY_POL FIELD32(0x00000010)
+#define MIMO_PS_CFG_RX_RX_STBY0 FIELD32(0x00000020)
+
+/*
* EDCA_AC0_CFG:
*/
#define EDCA_AC0_CFG 0x1300
@@ -1095,6 +1124,12 @@
#define TX_PWR_CFG_0_OFDM6_CH1 FIELD32(0x00f00000)
#define TX_PWR_CFG_0_OFDM12_CH0 FIELD32(0x0f000000)
#define TX_PWR_CFG_0_OFDM12_CH1 FIELD32(0xf0000000)
+/* bits for new 2T devices */
+#define TX_PWR_CFG_0B_1MBS_2MBS FIELD32(0x000000ff)
+#define TX_PWR_CFG_0B_5MBS_11MBS FIELD32(0x0000ff00)
+#define TX_PWR_CFG_0B_6MBS_9MBS FIELD32(0x00ff0000)
+#define TX_PWR_CFG_0B_12MBS_18MBS FIELD32(0xff000000)
+
/*
* TX_PWR_CFG_1:
@@ -1117,6 +1152,11 @@
#define TX_PWR_CFG_1_MCS0_CH1 FIELD32(0x00f00000)
#define TX_PWR_CFG_1_MCS2_CH0 FIELD32(0x0f000000)
#define TX_PWR_CFG_1_MCS2_CH1 FIELD32(0xf0000000)
+/* bits for new 2T devices */
+#define TX_PWR_CFG_1B_24MBS_36MBS FIELD32(0x000000ff)
+#define TX_PWR_CFG_1B_48MBS FIELD32(0x0000ff00)
+#define TX_PWR_CFG_1B_MCS0_MCS1 FIELD32(0x00ff0000)
+#define TX_PWR_CFG_1B_MCS2_MCS3 FIELD32(0xff000000)
/*
* TX_PWR_CFG_2:
@@ -1139,6 +1179,11 @@
#define TX_PWR_CFG_2_MCS8_CH1 FIELD32(0x00f00000)
#define TX_PWR_CFG_2_MCS10_CH0 FIELD32(0x0f000000)
#define TX_PWR_CFG_2_MCS10_CH1 FIELD32(0xf0000000)
+/* bits for new 2T devices */
+#define TX_PWR_CFG_2B_MCS4_MCS5 FIELD32(0x000000ff)
+#define TX_PWR_CFG_2B_MCS6_MCS7 FIELD32(0x0000ff00)
+#define TX_PWR_CFG_2B_MCS8_MCS9 FIELD32(0x00ff0000)
+#define TX_PWR_CFG_2B_MCS10_MCS11 FIELD32(0xff000000)
/*
* TX_PWR_CFG_3:
@@ -1161,6 +1206,11 @@
#define TX_PWR_CFG_3_STBC0_CH1 FIELD32(0x00f00000)
#define TX_PWR_CFG_3_STBC2_CH0 FIELD32(0x0f000000)
#define TX_PWR_CFG_3_STBC2_CH1 FIELD32(0xf0000000)
+/* bits for new 2T devices */
+#define TX_PWR_CFG_3B_MCS12_MCS13 FIELD32(0x000000ff)
+#define TX_PWR_CFG_3B_MCS14 FIELD32(0x0000ff00)
+#define TX_PWR_CFG_3B_STBC_MCS0_MCS1 FIELD32(0x00ff0000)
+#define TX_PWR_CFG_3B_STBC_MCS2_MSC3 FIELD32(0xff000000)
/*
* TX_PWR_CFG_4:
@@ -1171,10 +1221,13 @@
#define TX_PWR_CFG_4_UKNOWN7 FIELD32(0x00000f00)
#define TX_PWR_CFG_4_UKNOWN8 FIELD32(0x0000f000)
/* bits for 3T devices */
-#define TX_PWR_CFG_3_STBC4_CH0 FIELD32(0x0000000f)
-#define TX_PWR_CFG_3_STBC4_CH1 FIELD32(0x000000f0)
-#define TX_PWR_CFG_3_STBC6_CH0 FIELD32(0x00000f00)
-#define TX_PWR_CFG_3_STBC6_CH1 FIELD32(0x0000f000)
+#define TX_PWR_CFG_4_STBC4_CH0 FIELD32(0x0000000f)
+#define TX_PWR_CFG_4_STBC4_CH1 FIELD32(0x000000f0)
+#define TX_PWR_CFG_4_STBC6_CH0 FIELD32(0x00000f00)
+#define TX_PWR_CFG_4_STBC6_CH1 FIELD32(0x0000f000)
+/* bits for new 2T devices */
+#define TX_PWR_CFG_4B_STBC_MCS4_MCS5 FIELD32(0x000000ff)
+#define TX_PWR_CFG_4B_STBC_MCS6 FIELD32(0x0000ff00)
/*
* TX_PIN_CFG:
@@ -1201,6 +1254,8 @@
#define TX_PIN_CFG_RFTR_POL FIELD32(0x00020000)
#define TX_PIN_CFG_TRSW_EN FIELD32(0x00040000)
#define TX_PIN_CFG_TRSW_POL FIELD32(0x00080000)
+#define TX_PIN_CFG_RFRX_EN FIELD32(0x00100000)
+#define TX_PIN_CFG_RFRX_POL FIELD32(0x00200000)
#define TX_PIN_CFG_PA_PE_A2_EN FIELD32(0x01000000)
#define TX_PIN_CFG_PA_PE_G2_EN FIELD32(0x02000000)
#define TX_PIN_CFG_PA_PE_A2_POL FIELD32(0x04000000)
@@ -1547,6 +1602,95 @@
#define TX_PWR_CFG_4_EXT_STBC4_CH2 FIELD32(0x0000000f)
#define TX_PWR_CFG_4_EXT_STBC6_CH2 FIELD32(0x00000f00)
+/* TXn_RF_GAIN_CORRECT: RF Gain Correction for each RF_ALC[3:2]
+ * Unit: 0.1 dB, Range: -3.2 dB to 3.1 dB
+ */
+#define TX0_RF_GAIN_CORRECT 0x13a0
+#define TX0_RF_GAIN_CORRECT_GAIN_CORR_0 FIELD32(0x0000003f)
+#define TX0_RF_GAIN_CORRECT_GAIN_CORR_1 FIELD32(0x00003f00)
+#define TX0_RF_GAIN_CORRECT_GAIN_CORR_2 FIELD32(0x003f0000)
+#define TX0_RF_GAIN_CORRECT_GAIN_CORR_3 FIELD32(0x3f000000)
+
+#define TX1_RF_GAIN_CORRECT 0x13a4
+#define TX1_RF_GAIN_CORRECT_GAIN_CORR_0 FIELD32(0x0000003f)
+#define TX1_RF_GAIN_CORRECT_GAIN_CORR_1 FIELD32(0x00003f00)
+#define TX1_RF_GAIN_CORRECT_GAIN_CORR_2 FIELD32(0x003f0000)
+#define TX1_RF_GAIN_CORRECT_GAIN_CORR_3 FIELD32(0x3f000000)
+
+/* TXn_RF_GAIN_ATTEN: TXn RF Gain Attenuation Level
+ * Format: 7-bit, signed value
+ * Unit: 0.5 dB, Range: -20 dB to -5 dB
+ */
+#define TX0_RF_GAIN_ATTEN 0x13a8
+#define TX0_RF_GAIN_ATTEN_LEVEL_0 FIELD32(0x0000007f)
+#define TX0_RF_GAIN_ATTEN_LEVEL_1 FIELD32(0x00007f00)
+#define TX0_RF_GAIN_ATTEN_LEVEL_2 FIELD32(0x007f0000)
+#define TX0_RF_GAIN_ATTEN_LEVEL_3 FIELD32(0x7f000000)
+#define TX1_RF_GAIN_ATTEN 0x13ac
+#define TX1_RF_GAIN_ATTEN_LEVEL_0 FIELD32(0x0000007f)
+#define TX1_RF_GAIN_ATTEN_LEVEL_1 FIELD32(0x00007f00)
+#define TX1_RF_GAIN_ATTEN_LEVEL_2 FIELD32(0x007f0000)
+#define TX1_RF_GAIN_ATTEN_LEVEL_3 FIELD32(0x7f000000)
+
+/* TX_ALC_CFG_0: TX Automatic Level Control Configuration 0
+ * TX_ALC_LIMIT_n: TXn upper limit
+ * TX_ALC_CH_INIT_n: TXn channel initial transmission gain
+ * Unit: 0.5 dB, Range: 0 to 23.5 dB
+ */
+#define TX_ALC_CFG_0 0x13b0
+#define TX_ALC_CFG_0_CH_INIT_0 FIELD32(0x0000003f)
+#define TX_ALC_CFG_0_CH_INIT_1 FIELD32(0x00003f00)
+#define TX_ALC_CFG_0_LIMIT_0 FIELD32(0x003f0000)
+#define TX_ALC_CFG_0_LIMIT_1 FIELD32(0x3f000000)
+
+/* TX_ALC_CFG_1: TX Automatic Level Control Configuration 1
+ * TX_TEMP_COMP: TX Power Temperature Compensation
+ * Unit: 0.5 dB, Range: -10 dB to 10 dB
+ * TXn_GAIN_FINE: TXn Gain Fine Adjustment
+ * Unit: 0.1 dB, Range: -0.8 dB to 0.7 dB
+ * RF_TOS_DLY: Sets the RF_TOS_EN assertion delay after
+ * deassertion of PA_PE.
+ * Unit: 0.25 usec
+ * TXn_RF_GAIN_ATTEN: TXn RF gain attentuation selector
+ * RF_TOS_TIMEOUT: time-out value for RF_TOS_ENABLE
+ * deassertion if RF_TOS_DONE is missing.
+ * Unit: 0.25 usec
+ * RF_TOS_ENABLE: TX offset calibration enable
+ * ROS_BUSY_EN: RX offset calibration busy enable
+ */
+#define TX_ALC_CFG_1 0x13b4
+#define TX_ALC_CFG_1_TX_TEMP_COMP FIELD32(0x0000003f)
+#define TX_ALC_CFG_1_TX0_GAIN_FINE FIELD32(0x00000f00)
+#define TX_ALC_CFG_1_TX1_GAIN_FINE FIELD32(0x0000f000)
+#define TX_ALC_CFG_1_RF_TOS_DLY FIELD32(0x00070000)
+#define TX_ALC_CFG_1_TX0_RF_GAIN_ATTEN FIELD32(0x00300000)
+#define TX_ALC_CFG_1_TX1_RF_GAIN_ATTEN FIELD32(0x00c00000)
+#define TX_ALC_CFG_1_RF_TOS_TIMEOUT FIELD32(0x3f000000)
+#define TX_ALC_CFG_1_RF_TOS_ENABLE FIELD32(0x40000000)
+#define TX_ALC_CFG_1_ROS_BUSY_EN FIELD32(0x80000000)
+
+/* TXn_BB_GAIN_ATTEN: TXn RF Gain Attenuation Level
+ * Format: 5-bit signed values
+ * Unit: 0.5 dB, Range: -8 dB to 7 dB
+ */
+#define TX0_BB_GAIN_ATTEN 0x13c0
+#define TX0_BB_GAIN_ATTEN_LEVEL_0 FIELD32(0x0000001f)
+#define TX0_BB_GAIN_ATTEN_LEVEL_1 FIELD32(0x00001f00)
+#define TX0_BB_GAIN_ATTEN_LEVEL_2 FIELD32(0x001f0000)
+#define TX0_BB_GAIN_ATTEN_LEVEL_3 FIELD32(0x1f000000)
+#define TX1_BB_GAIN_ATTEN 0x13c4
+#define TX1_BB_GAIN_ATTEN_LEVEL_0 FIELD32(0x0000001f)
+#define TX1_BB_GAIN_ATTEN_LEVEL_1 FIELD32(0x00001f00)
+#define TX1_BB_GAIN_ATTEN_LEVEL_2 FIELD32(0x001f0000)
+#define TX1_BB_GAIN_ATTEN_LEVEL_3 FIELD32(0x1f000000)
+
+/* TX_ALC_VGA3: TX Automatic Level Correction Variable Gain Amplifier 3 */
+#define TX_ALC_VGA3 0x13c8
+#define TX_ALC_VGA3_TX0_ALC_VGA3 FIELD32(0x0000001f)
+#define TX_ALC_VGA3_TX1_ALC_VGA3 FIELD32(0x00001f00)
+#define TX_ALC_VGA3_TX0_ALC_VGA2 FIELD32(0x001f0000)
+#define TX_ALC_VGA3_TX1_ALC_VGA2 FIELD32(0x1f000000)
+
/* TX_PWR_CFG_7 */
#define TX_PWR_CFG_7 0x13d4
#define TX_PWR_CFG_7_OFDM54_CH0 FIELD32(0x0000000f)
@@ -1555,6 +1699,10 @@
#define TX_PWR_CFG_7_MCS7_CH0 FIELD32(0x000f0000)
#define TX_PWR_CFG_7_MCS7_CH1 FIELD32(0x00f00000)
#define TX_PWR_CFG_7_MCS7_CH2 FIELD32(0x0f000000)
+/* bits for new 2T devices */
+#define TX_PWR_CFG_7B_54MBS FIELD32(0x000000ff)
+#define TX_PWR_CFG_7B_MCS7 FIELD32(0x00ff0000)
+
/* TX_PWR_CFG_8 */
#define TX_PWR_CFG_8 0x13d8
@@ -1564,12 +1712,17 @@
#define TX_PWR_CFG_8_MCS23_CH0 FIELD32(0x000f0000)
#define TX_PWR_CFG_8_MCS23_CH1 FIELD32(0x00f00000)
#define TX_PWR_CFG_8_MCS23_CH2 FIELD32(0x0f000000)
+/* bits for new 2T devices */
+#define TX_PWR_CFG_8B_MCS15 FIELD32(0x000000ff)
+
/* TX_PWR_CFG_9 */
#define TX_PWR_CFG_9 0x13dc
#define TX_PWR_CFG_9_STBC7_CH0 FIELD32(0x0000000f)
#define TX_PWR_CFG_9_STBC7_CH1 FIELD32(0x000000f0)
#define TX_PWR_CFG_9_STBC7_CH2 FIELD32(0x00000f00)
+/* bits for new 2T devices */
+#define TX_PWR_CFG_9B_STBC_MCS7 FIELD32(0x000000ff)
/*
* RX_FILTER_CFG: RX configuration register.
@@ -1760,6 +1913,8 @@
#define TX_STA_FIFO_WCID FIELD32(0x0000ff00)
#define TX_STA_FIFO_SUCCESS_RATE FIELD32(0xffff0000)
#define TX_STA_FIFO_MCS FIELD32(0x007f0000)
+#define TX_STA_FIFO_BW FIELD32(0x00800000)
+#define TX_STA_FIFO_SGI FIELD32(0x01000000)
#define TX_STA_FIFO_PHYMODE FIELD32(0xc0000000)
/*
@@ -2135,11 +2290,14 @@
#define RFCSR1_TX1_PD FIELD8(0x20)
#define RFCSR1_RX2_PD FIELD8(0x40)
#define RFCSR1_TX2_PD FIELD8(0x80)
+#define RFCSR1_TX2_EN_MT7620 FIELD8(0x02)
/*
* RFCSR 2:
*/
#define RFCSR2_RESCAL_EN FIELD8(0x80)
+#define RFCSR2_RX2_EN_MT7620 FIELD8(0x02)
+#define RFCSR2_TX2_EN_MT7620 FIELD8(0x20)
/*
* RFCSR 3:
@@ -2158,6 +2316,12 @@
#define RFCSR3_BIT5 FIELD8(0x20)
/*
+ * RFCSR 4:
+ * VCOCAL_EN used by MT7620
+ */
+#define RFCSR4_VCOCAL_EN FIELD8(0x80)
+
+/*
* FRCSR 5:
*/
#define RFCSR5_R1 FIELD8(0x0c)
@@ -2212,6 +2376,7 @@
*/
#define RFCSR13_TX_POWER FIELD8(0x1f)
#define RFCSR13_DR0 FIELD8(0xe0)
+#define RFCSR13_RDIV_MT7620 FIELD8(0x03)
/*
* RFCSR 15:
@@ -2222,6 +2387,8 @@
* RFCSR 16:
*/
#define RFCSR16_TXMIXER_GAIN FIELD8(0x07)
+#define RFCSR16_RF_PLL_FREQ_SEL_MT7620 FIELD8(0x0F)
+#define RFCSR16_SDM_MODE_MT7620 FIELD8(0xE0)
/*
* RFCSR 17:
@@ -2234,6 +2401,8 @@
/* RFCSR 18 */
#define RFCSR18_XO_TUNE_BYPASS FIELD8(0x40)
+/* RFCSR 19 */
+#define RFCSR19_K FIELD8(0x03)
/*
* RFCSR 20:
@@ -2244,11 +2413,14 @@
* RFCSR 21:
*/
#define RFCSR21_RX_LO2_EN FIELD8(0x08)
+#define RFCSR21_BIT1 FIELD8(0x01)
+#define RFCSR21_BIT8 FIELD8(0x80)
/*
* RFCSR 22:
*/
#define RFCSR22_BASEBAND_LOOPBACK FIELD8(0x01)
+#define RFCSR22_FREQPLAN_D_MT7620 FIELD8(0x07)
/*
* RFCSR 23:
@@ -2271,6 +2443,11 @@
#define RFCSR27_R4 FIELD8(0x40)
/*
+ * RFCSR 28:
+ */
+#define RFCSR28_CH11_HT40 FIELD8(0x04)
+
+/*
* RFCSR 29:
*/
#define RFCSR29_ADC6_TEST FIELD8(0x01)
@@ -2331,6 +2508,7 @@
*/
#define RFCSR42_BIT1 FIELD8(0x01)
#define RFCSR42_BIT4 FIELD8(0x08)
+#define RFCSR42_TX2_EN_MT7620 FIELD8(0x40)
/*
* RFCSR 49:
@@ -2433,6 +2611,7 @@
EEPROM_TSSI_BOUND_BG5,
EEPROM_TXPOWER_A1,
EEPROM_TXPOWER_A2,
+ EEPROM_TXPOWER_INIT,
EEPROM_TSSI_BOUND_A1,
EEPROM_TSSI_BOUND_A2,
EEPROM_TSSI_BOUND_A3,
@@ -2987,29 +3166,4 @@
*/
#define BCN_TBTT_OFFSET 64
-/*
- * Hardware has 255 WCID table entries. First 32 entries are reserved for
- * shared keys. Since parts of the pairwise key table might be shared with
- * the beacon frame buffers 6 & 7 we could only use the first 222 entries.
- */
-#define WCID_START 33
-#define WCID_END 222
-#define STA_IDS_SIZE (WCID_END - WCID_START + 2)
-
-/*
- * RT2800 driver data structure
- */
-struct rt2800_drv_data {
- u8 calibration_bw20;
- u8 calibration_bw40;
- u8 bbp25;
- u8 bbp26;
- u8 txmixer_gain_24g;
- u8 txmixer_gain_5g;
- u8 max_psdu;
- unsigned int tbtt_tick;
- unsigned int ampdu_factor_cnt[4];
- DECLARE_BITMAP(sta_ids, STA_IDS_SIZE);
-};
-
#endif /* RT2800_H */
diff --git a/drivers/net/wireless/ralink/rt2x00/rt2800lib.c b/drivers/net/wireless/ralink/rt2x00/rt2800lib.c
index 8223a15..d11c7b2 100644
--- a/drivers/net/wireless/ralink/rt2x00/rt2800lib.c
+++ b/drivers/net/wireless/ralink/rt2x00/rt2800lib.c
@@ -59,6 +59,9 @@
rt2800_regbusy_read((__dev), BBP_CSR_CFG, BBP_CSR_CFG_BUSY, (__reg))
#define WAIT_FOR_RFCSR(__dev, __reg) \
rt2800_regbusy_read((__dev), RF_CSR_CFG, RF_CSR_CFG_BUSY, (__reg))
+#define WAIT_FOR_RFCSR_MT7620(__dev, __reg) \
+ rt2800_regbusy_read((__dev), RF_CSR_CFG, RF_CSR_CFG_BUSY_MT7620, \
+ (__reg))
#define WAIT_FOR_RF(__dev, __reg) \
rt2800_regbusy_read((__dev), RF_CSR_CFG0, RF_CSR_CFG0_BUSY, (__reg))
#define WAIT_FOR_MCU(__dev, __reg) \
@@ -150,19 +153,56 @@
* Wait until the RFCSR becomes available, afterwards we
* can safely write the new data into the register.
*/
- if (WAIT_FOR_RFCSR(rt2x00dev, ®)) {
- reg = 0;
- rt2x00_set_field32(®, RF_CSR_CFG_DATA, value);
- rt2x00_set_field32(®, RF_CSR_CFG_REGNUM, word);
- rt2x00_set_field32(®, RF_CSR_CFG_WRITE, 1);
- rt2x00_set_field32(®, RF_CSR_CFG_BUSY, 1);
+ switch (rt2x00dev->chip.rt) {
+ case RT6352:
+ if (WAIT_FOR_RFCSR_MT7620(rt2x00dev, ®)) {
+ reg = 0;
+ rt2x00_set_field32(®, RF_CSR_CFG_DATA_MT7620, value);
+ rt2x00_set_field32(®, RF_CSR_CFG_REGNUM_MT7620,
+ word);
+ rt2x00_set_field32(®, RF_CSR_CFG_WRITE_MT7620, 1);
+ rt2x00_set_field32(®, RF_CSR_CFG_BUSY_MT7620, 1);
- rt2800_register_write_lock(rt2x00dev, RF_CSR_CFG, reg);
+ rt2800_register_write_lock(rt2x00dev, RF_CSR_CFG, reg);
+ }
+ break;
+
+ default:
+ if (WAIT_FOR_RFCSR(rt2x00dev, ®)) {
+ reg = 0;
+ rt2x00_set_field32(®, RF_CSR_CFG_DATA, value);
+ rt2x00_set_field32(®, RF_CSR_CFG_REGNUM, word);
+ rt2x00_set_field32(®, RF_CSR_CFG_WRITE, 1);
+ rt2x00_set_field32(®, RF_CSR_CFG_BUSY, 1);
+
+ rt2800_register_write_lock(rt2x00dev, RF_CSR_CFG, reg);
+ }
+ break;
}
mutex_unlock(&rt2x00dev->csr_mutex);
}
+static void rt2800_rfcsr_write_bank(struct rt2x00_dev *rt2x00dev, const u8 bank,
+ const unsigned int reg, const u8 value)
+{
+ rt2800_rfcsr_write(rt2x00dev, (reg | (bank << 6)), value);
+}
+
+static void rt2800_rfcsr_write_chanreg(struct rt2x00_dev *rt2x00dev,
+ const unsigned int reg, const u8 value)
+{
+ rt2800_rfcsr_write_bank(rt2x00dev, 4, reg, value);
+ rt2800_rfcsr_write_bank(rt2x00dev, 6, reg, value);
+}
+
+static void rt2800_rfcsr_write_dccal(struct rt2x00_dev *rt2x00dev,
+ const unsigned int reg, const u8 value)
+{
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, reg, value);
+ rt2800_rfcsr_write_bank(rt2x00dev, 7, reg, value);
+}
+
static void rt2800_rfcsr_read(struct rt2x00_dev *rt2x00dev,
const unsigned int word, u8 *value)
{
@@ -178,22 +218,48 @@
* doesn't become available in time, reg will be 0xffffffff
* which means we return 0xff to the caller.
*/
- if (WAIT_FOR_RFCSR(rt2x00dev, ®)) {
- reg = 0;
- rt2x00_set_field32(®, RF_CSR_CFG_REGNUM, word);
- rt2x00_set_field32(®, RF_CSR_CFG_WRITE, 0);
- rt2x00_set_field32(®, RF_CSR_CFG_BUSY, 1);
+ switch (rt2x00dev->chip.rt) {
+ case RT6352:
+ if (WAIT_FOR_RFCSR_MT7620(rt2x00dev, ®)) {
+ reg = 0;
+ rt2x00_set_field32(®, RF_CSR_CFG_REGNUM_MT7620,
+ word);
+ rt2x00_set_field32(®, RF_CSR_CFG_WRITE_MT7620, 0);
+ rt2x00_set_field32(®, RF_CSR_CFG_BUSY_MT7620, 1);
- rt2800_register_write_lock(rt2x00dev, RF_CSR_CFG, reg);
+ rt2800_register_write_lock(rt2x00dev, RF_CSR_CFG, reg);
- WAIT_FOR_RFCSR(rt2x00dev, ®);
+ WAIT_FOR_RFCSR_MT7620(rt2x00dev, ®);
+ }
+
+ *value = rt2x00_get_field32(reg, RF_CSR_CFG_DATA_MT7620);
+ break;
+
+ default:
+ if (WAIT_FOR_RFCSR(rt2x00dev, ®)) {
+ reg = 0;
+ rt2x00_set_field32(®, RF_CSR_CFG_REGNUM, word);
+ rt2x00_set_field32(®, RF_CSR_CFG_WRITE, 0);
+ rt2x00_set_field32(®, RF_CSR_CFG_BUSY, 1);
+
+ rt2800_register_write_lock(rt2x00dev, RF_CSR_CFG, reg);
+
+ WAIT_FOR_RFCSR(rt2x00dev, ®);
+ }
+
+ *value = rt2x00_get_field32(reg, RF_CSR_CFG_DATA);
+ break;
}
- *value = rt2x00_get_field32(reg, RF_CSR_CFG_DATA);
-
mutex_unlock(&rt2x00dev->csr_mutex);
}
+static void rt2800_rfcsr_read_bank(struct rt2x00_dev *rt2x00dev, const u8 bank,
+ const unsigned int reg, u8 *value)
+{
+ rt2800_rfcsr_read(rt2x00dev, (reg | (bank << 6)), value);
+}
+
static void rt2800_rf_write(struct rt2x00_dev *rt2x00dev,
const unsigned int word, const u32 value)
{
@@ -250,6 +316,7 @@
[EEPROM_TSSI_BOUND_BG5] = 0x003b,
[EEPROM_TXPOWER_A1] = 0x003c,
[EEPROM_TXPOWER_A2] = 0x0053,
+ [EEPROM_TXPOWER_INIT] = 0x0068,
[EEPROM_TSSI_BOUND_A1] = 0x006a,
[EEPROM_TSSI_BOUND_A2] = 0x006b,
[EEPROM_TSSI_BOUND_A3] = 0x006c,
@@ -524,6 +591,7 @@
break;
case RT5592:
+ case RT6352:
*txwi_size = TXWI_DESC_SIZE_5WORDS;
*rxwi_size = RXWI_DESC_SIZE_6WORDS;
break;
@@ -821,10 +889,10 @@
rt2x00_desc_read(rxwi, 1, &word);
if (rt2x00_get_field32(word, RXWI_W1_SHORT_GI))
- rxdesc->flags |= RX_FLAG_SHORT_GI;
+ rxdesc->enc_flags |= RX_ENC_FLAG_SHORT_GI;
if (rt2x00_get_field32(word, RXWI_W1_BW))
- rxdesc->flags |= RX_FLAG_40MHZ;
+ rxdesc->bw = RATE_INFO_BW_40;
/*
* Detect RX rate, always use MCS as signal type.
@@ -852,14 +920,49 @@
}
EXPORT_SYMBOL_GPL(rt2800_process_rxwi);
-void rt2800_txdone_entry(struct queue_entry *entry, u32 status, __le32 *txwi)
+static void rt2800_rate_from_status(struct skb_frame_desc *skbdesc,
+ u32 status, enum nl80211_band band)
+{
+ u8 flags = 0;
+ u8 idx = rt2x00_get_field32(status, TX_STA_FIFO_MCS);
+
+ switch (rt2x00_get_field32(status, TX_STA_FIFO_PHYMODE)) {
+ case RATE_MODE_HT_GREENFIELD:
+ flags |= IEEE80211_TX_RC_GREEN_FIELD;
+ /* fall through */
+ case RATE_MODE_HT_MIX:
+ flags |= IEEE80211_TX_RC_MCS;
+ break;
+ case RATE_MODE_OFDM:
+ if (band == NL80211_BAND_2GHZ)
+ idx += 4;
+ break;
+ case RATE_MODE_CCK:
+ if (idx >= 8)
+ idx -= 8;
+ break;
+ }
+
+ if (rt2x00_get_field32(status, TX_STA_FIFO_BW))
+ flags |= IEEE80211_TX_RC_40_MHZ_WIDTH;
+
+ if (rt2x00_get_field32(status, TX_STA_FIFO_SGI))
+ flags |= IEEE80211_TX_RC_SHORT_GI;
+
+ skbdesc->tx_rate_idx = idx;
+ skbdesc->tx_rate_flags = flags;
+}
+
+void rt2800_txdone_entry(struct queue_entry *entry, u32 status, __le32 *txwi,
+ bool match)
{
struct rt2x00_dev *rt2x00dev = entry->queue->rt2x00dev;
+ struct rt2800_drv_data *drv_data = rt2x00dev->drv_data;
struct skb_frame_desc *skbdesc = get_skb_frame_desc(entry->skb);
struct txdone_entry_desc txdesc;
u32 word;
u16 mcs, real_mcs;
- int aggr, ampdu;
+ int aggr, ampdu, wcid, ack_req;
/*
* Obtain the status about this packet.
@@ -872,6 +975,8 @@
real_mcs = rt2x00_get_field32(status, TX_STA_FIFO_MCS);
aggr = rt2x00_get_field32(status, TX_STA_FIFO_TX_AGGRE);
+ wcid = rt2x00_get_field32(status, TX_STA_FIFO_WCID);
+ ack_req = rt2x00_get_field32(status, TX_STA_FIFO_TX_ACK_REQUIRED);
/*
* If a frame was meant to be sent as a single non-aggregated MPDU
@@ -888,15 +993,22 @@
* Hence, replace the requested rate with the real tx rate to not
* confuse the rate control algortihm by providing clearly wrong
* data.
- */
- if (unlikely(aggr == 1 && ampdu == 0 && real_mcs != mcs)) {
- skbdesc->tx_rate_idx = real_mcs;
+ *
+ * FIXME: if we do not find matching entry, we tell that frame was
+ * posted without any retries. We need to find a way to fix that
+ * and provide retry count.
+ */
+ if (unlikely((aggr == 1 && ampdu == 0 && real_mcs != mcs)) || !match) {
+ rt2800_rate_from_status(skbdesc, status, rt2x00dev->curr_band);
mcs = real_mcs;
}
if (aggr == 1 || ampdu == 1)
__set_bit(TXDONE_AMPDU, &txdesc.flags);
+ if (!ack_req)
+ __set_bit(TXDONE_NO_ACK_REQ, &txdesc.flags);
+
/*
* Ralink has a retry mechanism using a global fallback
* table. We setup this fallback table to try the immediate
@@ -928,7 +1040,18 @@
if (txdesc.retry)
__set_bit(TXDONE_FALLBACK, &txdesc.flags);
- rt2x00lib_txdone(entry, &txdesc);
+ if (!match) {
+ /* RCU assures non-null sta will not be freed by mac80211. */
+ rcu_read_lock();
+ if (likely(wcid >= WCID_START && wcid <= WCID_END))
+ skbdesc->sta = drv_data->wcid_to_sta[wcid - WCID_START];
+ else
+ skbdesc->sta = NULL;
+ rt2x00lib_txdone_nomatch(entry, &txdesc);
+ rcu_read_unlock();
+ } else {
+ rt2x00lib_txdone(entry, &txdesc);
+ }
}
EXPORT_SYMBOL_GPL(rt2800_txdone_entry);
@@ -1468,6 +1591,7 @@
return 0;
__set_bit(wcid - WCID_START, drv_data->sta_ids);
+ drv_data->wcid_to_sta[wcid - WCID_START] = sta;
/*
* Clean up WCID attributes and write STA address to the device.
@@ -1498,6 +1622,7 @@
* get renewed when the WCID is reused.
*/
rt2800_config_wcid(rt2x00dev, NULL, wcid);
+ drv_data->wcid_to_sta[wcid - WCID_START] = NULL;
__clear_bit(wcid - WCID_START, drv_data->sta_ids);
return 0;
@@ -2753,7 +2878,8 @@
rt2800_rfcsr_write(rt2x00dev, 59,
r59_nonbt_rev[idx]);
} else if (rt2x00_rt(rt2x00dev, RT5390) ||
- rt2x00_rt(rt2x00dev, RT5392)) {
+ rt2x00_rt(rt2x00dev, RT5392) ||
+ rt2x00_rt(rt2x00dev, RT6352)) {
static const char r59_non_bt[] = {0x8f, 0x8f,
0x8f, 0x8f, 0x8f, 0x8f, 0x8f, 0x8d,
0x8a, 0x88, 0x88, 0x87, 0x87, 0x86};
@@ -3047,6 +3173,244 @@
rt2800_bbp_write(rt2x00dev, 196, (rf->channel <= 14) ? 0x19 : 0x7F);
}
+static void rt2800_config_channel_rf7620(struct rt2x00_dev *rt2x00dev,
+ struct ieee80211_conf *conf,
+ struct rf_channel *rf,
+ struct channel_info *info)
+{
+ struct rt2800_drv_data *drv_data = rt2x00dev->drv_data;
+ u8 rx_agc_fc, tx_agc_fc;
+ u8 rfcsr;
+
+ /* Frequeny plan setting */
+ /* Rdiv setting (set 0x03 if Xtal==20)
+ * R13[1:0]
+ */
+ rt2800_rfcsr_read(rt2x00dev, 13, &rfcsr);
+ rt2x00_set_field8(&rfcsr, RFCSR13_RDIV_MT7620,
+ rt2800_clk_is_20mhz(rt2x00dev) ? 3 : 0);
+ rt2800_rfcsr_write(rt2x00dev, 13, rfcsr);
+
+ /* N setting
+ * R20[7:0] in rf->rf1
+ * R21[0] always 0
+ */
+ rt2800_rfcsr_read(rt2x00dev, 20, &rfcsr);
+ rfcsr = (rf->rf1 & 0x00ff);
+ rt2800_rfcsr_write(rt2x00dev, 20, rfcsr);
+
+ rt2800_rfcsr_read(rt2x00dev, 21, &rfcsr);
+ rt2x00_set_field8(&rfcsr, RFCSR21_BIT1, 0);
+ rt2800_rfcsr_write(rt2x00dev, 21, rfcsr);
+
+ /* K setting (always 0)
+ * R16[3:0] (RF PLL freq selection)
+ */
+ rt2800_rfcsr_read(rt2x00dev, 16, &rfcsr);
+ rt2x00_set_field8(&rfcsr, RFCSR16_RF_PLL_FREQ_SEL_MT7620, 0);
+ rt2800_rfcsr_write(rt2x00dev, 16, rfcsr);
+
+ /* D setting (always 0)
+ * R22[2:0] (D=15, R22[2:0]=<111>)
+ */
+ rt2800_rfcsr_read(rt2x00dev, 22, &rfcsr);
+ rt2x00_set_field8(&rfcsr, RFCSR22_FREQPLAN_D_MT7620, 0);
+ rt2800_rfcsr_write(rt2x00dev, 22, rfcsr);
+
+ /* Ksd setting
+ * Ksd: R17<7:0> in rf->rf2
+ * R18<7:0> in rf->rf3
+ * R19<1:0> in rf->rf4
+ */
+ rt2800_rfcsr_read(rt2x00dev, 17, &rfcsr);
+ rfcsr = rf->rf2;
+ rt2800_rfcsr_write(rt2x00dev, 17, rfcsr);
+
+ rt2800_rfcsr_read(rt2x00dev, 18, &rfcsr);
+ rfcsr = rf->rf3;
+ rt2800_rfcsr_write(rt2x00dev, 18, rfcsr);
+
+ rt2800_rfcsr_read(rt2x00dev, 19, &rfcsr);
+ rt2x00_set_field8(&rfcsr, RFCSR19_K, rf->rf4);
+ rt2800_rfcsr_write(rt2x00dev, 19, rfcsr);
+
+ /* Default: XO=20MHz , SDM mode */
+ rt2800_rfcsr_read(rt2x00dev, 16, &rfcsr);
+ rt2x00_set_field8(&rfcsr, RFCSR16_SDM_MODE_MT7620, 0x80);
+ rt2800_rfcsr_write(rt2x00dev, 16, rfcsr);
+
+ rt2800_rfcsr_read(rt2x00dev, 21, &rfcsr);
+ rt2x00_set_field8(&rfcsr, RFCSR21_BIT8, 1);
+ rt2800_rfcsr_write(rt2x00dev, 21, rfcsr);
+
+ rt2800_rfcsr_read(rt2x00dev, 1, &rfcsr);
+ rt2x00_set_field8(&rfcsr, RFCSR1_TX2_EN_MT7620,
+ rt2x00dev->default_ant.tx_chain_num != 1);
+ rt2800_rfcsr_write(rt2x00dev, 1, rfcsr);
+
+ rt2800_rfcsr_read(rt2x00dev, 2, &rfcsr);
+ rt2x00_set_field8(&rfcsr, RFCSR2_TX2_EN_MT7620,
+ rt2x00dev->default_ant.tx_chain_num != 1);
+ rt2x00_set_field8(&rfcsr, RFCSR2_RX2_EN_MT7620,
+ rt2x00dev->default_ant.rx_chain_num != 1);
+ rt2800_rfcsr_write(rt2x00dev, 2, rfcsr);
+
+ rt2800_rfcsr_read(rt2x00dev, 42, &rfcsr);
+ rt2x00_set_field8(&rfcsr, RFCSR42_TX2_EN_MT7620,
+ rt2x00dev->default_ant.tx_chain_num != 1);
+ rt2800_rfcsr_write(rt2x00dev, 42, rfcsr);
+
+ /* RF for DC Cal BW */
+ if (conf_is_ht40(conf)) {
+ rt2800_rfcsr_write_dccal(rt2x00dev, 6, 0x10);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 7, 0x10);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 8, 0x04);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 58, 0x10);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 59, 0x10);
+ } else {
+ rt2800_rfcsr_write_dccal(rt2x00dev, 6, 0x20);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 7, 0x20);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 8, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 58, 0x20);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 59, 0x20);
+ }
+
+ if (conf_is_ht40(conf)) {
+ rt2800_rfcsr_write_dccal(rt2x00dev, 58, 0x08);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 59, 0x08);
+ } else {
+ rt2800_rfcsr_write_dccal(rt2x00dev, 58, 0x28);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 59, 0x28);
+ }
+
+ rt2800_rfcsr_read(rt2x00dev, 28, &rfcsr);
+ rt2x00_set_field8(&rfcsr, RFCSR28_CH11_HT40,
+ conf_is_ht40(conf) && (rf->channel == 11));
+ rt2800_rfcsr_write(rt2x00dev, 28, rfcsr);
+
+ if (!test_bit(DEVICE_STATE_SCANNING, &rt2x00dev->flags)) {
+ if (conf_is_ht40(conf)) {
+ rx_agc_fc = drv_data->rx_calibration_bw40;
+ tx_agc_fc = drv_data->tx_calibration_bw40;
+ } else {
+ rx_agc_fc = drv_data->rx_calibration_bw20;
+ tx_agc_fc = drv_data->tx_calibration_bw20;
+ }
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 6, &rfcsr);
+ rfcsr &= (~0x3F);
+ rfcsr |= rx_agc_fc;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 6, rfcsr);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 7, &rfcsr);
+ rfcsr &= (~0x3F);
+ rfcsr |= rx_agc_fc;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 7, rfcsr);
+ rt2800_rfcsr_read_bank(rt2x00dev, 7, 6, &rfcsr);
+ rfcsr &= (~0x3F);
+ rfcsr |= rx_agc_fc;
+ rt2800_rfcsr_write_bank(rt2x00dev, 7, 6, rfcsr);
+ rt2800_rfcsr_read_bank(rt2x00dev, 7, 7, &rfcsr);
+ rfcsr &= (~0x3F);
+ rfcsr |= rx_agc_fc;
+ rt2800_rfcsr_write_bank(rt2x00dev, 7, 7, rfcsr);
+
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 58, &rfcsr);
+ rfcsr &= (~0x3F);
+ rfcsr |= tx_agc_fc;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 58, rfcsr);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 59, &rfcsr);
+ rfcsr &= (~0x3F);
+ rfcsr |= tx_agc_fc;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 59, rfcsr);
+ rt2800_rfcsr_read_bank(rt2x00dev, 7, 58, &rfcsr);
+ rfcsr &= (~0x3F);
+ rfcsr |= tx_agc_fc;
+ rt2800_rfcsr_write_bank(rt2x00dev, 7, 58, rfcsr);
+ rt2800_rfcsr_read_bank(rt2x00dev, 7, 59, &rfcsr);
+ rfcsr &= (~0x3F);
+ rfcsr |= tx_agc_fc;
+ rt2800_rfcsr_write_bank(rt2x00dev, 7, 59, rfcsr);
+ }
+}
+
+static void rt2800_config_alc(struct rt2x00_dev *rt2x00dev,
+ struct ieee80211_channel *chan,
+ int power_level) {
+ u16 eeprom, target_power, max_power;
+ u32 mac_sys_ctrl, mac_status;
+ u32 reg;
+ u8 bbp;
+ int i;
+
+ /* hardware unit is 0.5dBm, limited to 23.5dBm */
+ power_level *= 2;
+ if (power_level > 0x2f)
+ power_level = 0x2f;
+
+ max_power = chan->max_power * 2;
+ if (max_power > 0x2f)
+ max_power = 0x2f;
+
+ rt2800_register_read(rt2x00dev, TX_ALC_CFG_0, ®);
+ rt2x00_set_field32(®, TX_ALC_CFG_0_CH_INIT_0, power_level);
+ rt2x00_set_field32(®, TX_ALC_CFG_0_CH_INIT_1, power_level);
+ rt2x00_set_field32(®, TX_ALC_CFG_0_LIMIT_0, max_power);
+ rt2x00_set_field32(®, TX_ALC_CFG_0_LIMIT_1, max_power);
+
+ rt2800_eeprom_read(rt2x00dev, EEPROM_NIC_CONF1, &eeprom);
+ if (rt2x00_get_field16(eeprom, EEPROM_NIC_CONF1_INTERNAL_TX_ALC)) {
+ /* init base power by eeprom target power */
+ rt2800_eeprom_read(rt2x00dev, EEPROM_TXPOWER_INIT,
+ &target_power);
+ rt2x00_set_field32(®, TX_ALC_CFG_0_CH_INIT_0, target_power);
+ rt2x00_set_field32(®, TX_ALC_CFG_0_CH_INIT_1, target_power);
+ }
+ rt2800_register_write(rt2x00dev, TX_ALC_CFG_0, reg);
+
+ rt2800_register_read(rt2x00dev, TX_ALC_CFG_1, ®);
+ rt2x00_set_field32(®, TX_ALC_CFG_1_TX_TEMP_COMP, 0);
+ rt2800_register_write(rt2x00dev, TX_ALC_CFG_1, reg);
+
+ /* Save MAC SYS CTRL registers */
+ rt2800_register_read(rt2x00dev, MAC_SYS_CTRL, &mac_sys_ctrl);
+ /* Disable Tx/Rx */
+ rt2800_register_write(rt2x00dev, MAC_SYS_CTRL, 0);
+ /* Check MAC Tx/Rx idle */
+ for (i = 0; i < 10000; i++) {
+ rt2800_register_read(rt2x00dev, MAC_STATUS_CFG,
+ &mac_status);
+ if (mac_status & 0x3)
+ usleep_range(50, 200);
+ else
+ break;
+ }
+
+ if (i == 10000)
+ rt2x00_warn(rt2x00dev, "Wait MAC Status to MAX !!!\n");
+
+ if (chan->center_freq > 2457) {
+ rt2800_bbp_read(rt2x00dev, 30, &bbp);
+ bbp = 0x40;
+ rt2800_bbp_write(rt2x00dev, 30, bbp);
+ rt2800_rfcsr_write(rt2x00dev, 39, 0);
+ if (rt2x00_has_cap_external_lna_bg(rt2x00dev))
+ rt2800_rfcsr_write(rt2x00dev, 42, 0xfb);
+ else
+ rt2800_rfcsr_write(rt2x00dev, 42, 0x7b);
+ } else {
+ rt2800_bbp_read(rt2x00dev, 30, &bbp);
+ bbp = 0x1f;
+ rt2800_bbp_write(rt2x00dev, 30, bbp);
+ rt2800_rfcsr_write(rt2x00dev, 39, 0x80);
+ if (rt2x00_has_cap_external_lna_bg(rt2x00dev))
+ rt2800_rfcsr_write(rt2x00dev, 42, 0xdb);
+ else
+ rt2800_rfcsr_write(rt2x00dev, 42, 0x5b);
+ }
+ rt2800_register_write(rt2x00dev, MAC_SYS_CTRL, mac_sys_ctrl);
+
+ rt2800_vco_calibration(rt2x00dev);
+}
+
static void rt2800_bbp_write_with_rx_chain(struct rt2x00_dev *rt2x00dev,
const unsigned int word,
const u8 value)
@@ -3171,7 +3535,7 @@
struct channel_info *info)
{
u32 reg;
- unsigned int tx_pin;
+ u32 tx_pin;
u8 bbp, rfcsr;
info->default_power1 = rt2800_txpower_to_dev(rt2x00dev, rf->channel,
@@ -3216,6 +3580,9 @@
case RF5592:
rt2800_config_channel_rf55xx(rt2x00dev, conf, rf, info);
break;
+ case RF7620:
+ rt2800_config_channel_rf7620(rt2x00dev, conf, rf, info);
+ break;
default:
rt2800_config_channel_rf2xxx(rt2x00dev, conf, rf, info);
}
@@ -3290,7 +3657,8 @@
if (rf->channel <= 14) {
if (!rt2x00_rt(rt2x00dev, RT5390) &&
- !rt2x00_rt(rt2x00dev, RT5392)) {
+ !rt2x00_rt(rt2x00dev, RT5392) &&
+ !rt2x00_rt(rt2x00dev, RT6352)) {
if (rt2x00_has_cap_external_lna_bg(rt2x00dev)) {
rt2800_bbp_write(rt2x00dev, 82, 0x62);
rt2800_bbp_write(rt2x00dev, 75, 0x46);
@@ -3310,7 +3678,7 @@
rt2800_bbp_write(rt2x00dev, 82, 0x94);
else if (rt2x00_rt(rt2x00dev, RT3593))
rt2800_bbp_write(rt2x00dev, 82, 0x82);
- else
+ else if (!rt2x00_rt(rt2x00dev, RT6352))
rt2800_bbp_write(rt2x00dev, 82, 0xf2);
if (rt2x00_rt(rt2x00dev, RT3593))
@@ -3331,7 +3699,7 @@
if (rt2x00_rt(rt2x00dev, RT3572))
rt2800_rfcsr_write(rt2x00dev, 8, 0);
- tx_pin = 0;
+ rt2800_register_read(rt2x00dev, TX_PIN_CFG, &tx_pin);
switch (rt2x00dev->default_ant.tx_chain_num) {
case 3:
@@ -3380,6 +3748,7 @@
rt2x00_set_field32(&tx_pin, TX_PIN_CFG_RFTR_EN, 1);
rt2x00_set_field32(&tx_pin, TX_PIN_CFG_TRSW_EN, 1);
+ rt2x00_set_field32(&tx_pin, TX_PIN_CFG_RFRX_EN, 1); /* mt7620 */
rt2800_register_write(rt2x00dev, TX_PIN_CFG, tx_pin);
@@ -3438,12 +3807,26 @@
usleep_range(1000, 1500);
}
- if (rt2x00_rt(rt2x00dev, RT5592)) {
+ if (rt2x00_rt(rt2x00dev, RT5592) || rt2x00_rt(rt2x00dev, RT6352)) {
+ reg = 0x10;
+ if (!conf_is_ht40(conf)) {
+ if (rt2x00_rt(rt2x00dev, RT6352) &&
+ rt2x00_has_cap_external_lna_bg(rt2x00dev)) {
+ reg |= 0x5;
+ } else {
+ reg |= 0xa;
+ }
+ }
rt2800_bbp_write(rt2x00dev, 195, 141);
- rt2800_bbp_write(rt2x00dev, 196, conf_is_ht40(conf) ? 0x10 : 0x1a);
+ rt2800_bbp_write(rt2x00dev, 196, reg);
/* AGC init */
- reg = (rf->channel <= 14 ? 0x1c : 0x24) + 2 * rt2x00dev->lna_gain;
+ if (rt2x00_rt(rt2x00dev, RT6352))
+ reg = 0x04;
+ else
+ reg = rf->channel <= 14 ? 0x1c : 0x24;
+
+ reg += 2 * rt2x00dev->lna_gain;
rt2800_bbp_write_with_rx_chain(rt2x00dev, 66, reg);
rt2800_iq_calibrate(rt2x00dev, rf->channel);
@@ -4125,6 +4508,128 @@
(unsigned long) regs[i]);
}
+static void rt2800_config_txpower_rt6352(struct rt2x00_dev *rt2x00dev,
+ struct ieee80211_channel *chan,
+ int power_level)
+{
+ u32 reg, pwreg;
+ u16 eeprom;
+ u32 data, gdata;
+ u8 t, i;
+ enum nl80211_band band = chan->band;
+ int delta;
+
+ /* Warn user if bw_comp is set in EEPROM */
+ delta = rt2800_get_txpower_bw_comp(rt2x00dev, band);
+
+ if (delta)
+ rt2x00_warn(rt2x00dev, "ignoring EEPROM HT40 power delta: %d\n",
+ delta);
+
+ /* populate TX_PWR_CFG_0 up to TX_PWR_CFG_4 from EEPROM for HT20, limit
+ * value to 0x3f and replace 0x20 by 0x21 as this is what the vendor
+ * driver does as well, though it looks kinda wrong.
+ * Maybe some misunderstanding of what a signed 8-bit value is? Maybe
+ * the hardware has a problem handling 0x20, and as the code initially
+ * used a fixed offset between HT20 and HT40 rates they had to work-
+ * around that issue and most likely just forgot about it later on.
+ * Maybe we should use rt2800_get_txpower_bw_comp() here as well,
+ * however, the corresponding EEPROM value is not respected by the
+ * vendor driver, so maybe this is rather being taken care of the
+ * TXALC and the driver doesn't need to handle it...?
+ * Though this is all very awkward, just do as they did, as that's what
+ * board vendors expected when they populated the EEPROM...
+ */
+ for (i = 0; i < 5; i++) {
+ rt2800_eeprom_read_from_array(rt2x00dev, EEPROM_TXPOWER_BYRATE,
+ i * 2, &eeprom);
+
+ data = eeprom;
+
+ t = eeprom & 0x3f;
+ if (t == 32)
+ t++;
+
+ gdata = t;
+
+ t = (eeprom & 0x3f00) >> 8;
+ if (t == 32)
+ t++;
+
+ gdata |= (t << 8);
+
+ rt2800_eeprom_read_from_array(rt2x00dev, EEPROM_TXPOWER_BYRATE,
+ (i * 2) + 1, &eeprom);
+
+ t = eeprom & 0x3f;
+ if (t == 32)
+ t++;
+
+ gdata |= (t << 16);
+
+ t = (eeprom & 0x3f00) >> 8;
+ if (t == 32)
+ t++;
+
+ gdata |= (t << 24);
+ data |= (eeprom << 16);
+
+ if (!test_bit(CONFIG_CHANNEL_HT40, &rt2x00dev->flags)) {
+ /* HT20 */
+ if (data != 0xffffffff)
+ rt2800_register_write(rt2x00dev,
+ TX_PWR_CFG_0 + (i * 4),
+ data);
+ } else {
+ /* HT40 */
+ if (gdata != 0xffffffff)
+ rt2800_register_write(rt2x00dev,
+ TX_PWR_CFG_0 + (i * 4),
+ gdata);
+ }
+ }
+
+ /* Aparently Ralink ran out of space in the BYRATE calibration section
+ * of the EERPOM which is copied to the corresponding TX_PWR_CFG_x
+ * registers. As recent 2T chips use 8-bit instead of 4-bit values for
+ * power-offsets more space would be needed. Ralink decided to keep the
+ * EEPROM layout untouched and rather have some shared values covering
+ * multiple bitrates.
+ * Populate the registers not covered by the EEPROM in the same way the
+ * vendor driver does.
+ */
+
+ /* For OFDM 54MBS use value from OFDM 48MBS */
+ pwreg = 0;
+ rt2800_register_read(rt2x00dev, TX_PWR_CFG_1, ®);
+ t = rt2x00_get_field32(reg, TX_PWR_CFG_1B_48MBS);
+ rt2x00_set_field32(&pwreg, TX_PWR_CFG_7B_54MBS, t);
+
+ /* For MCS 7 use value from MCS 6 */
+ rt2800_register_read(rt2x00dev, TX_PWR_CFG_2, ®);
+ t = rt2x00_get_field32(reg, TX_PWR_CFG_2B_MCS6_MCS7);
+ rt2x00_set_field32(&pwreg, TX_PWR_CFG_7B_MCS7, t);
+ rt2800_register_write(rt2x00dev, TX_PWR_CFG_7, pwreg);
+
+ /* For MCS 15 use value from MCS 14 */
+ pwreg = 0;
+ rt2800_register_read(rt2x00dev, TX_PWR_CFG_3, ®);
+ t = rt2x00_get_field32(reg, TX_PWR_CFG_3B_MCS14);
+ rt2x00_set_field32(&pwreg, TX_PWR_CFG_8B_MCS15, t);
+ rt2800_register_write(rt2x00dev, TX_PWR_CFG_8, pwreg);
+
+ /* For STBC MCS 7 use value from STBC MCS 6 */
+ pwreg = 0;
+ rt2800_register_read(rt2x00dev, TX_PWR_CFG_4, ®);
+ t = rt2x00_get_field32(reg, TX_PWR_CFG_4B_STBC_MCS6);
+ rt2x00_set_field32(&pwreg, TX_PWR_CFG_9B_STBC_MCS7, t);
+ rt2800_register_write(rt2x00dev, TX_PWR_CFG_9, pwreg);
+
+ rt2800_config_alc(rt2x00dev, chan, power_level);
+
+ /* TODO: temperature compensation code! */
+}
+
/*
* We configure transmit power using MAC TX_PWR_CFG_{0,...,N} registers and
* BBP R1 register. TX_PWR_CFG_X allow to configure per rate TX power values,
@@ -4321,6 +4826,8 @@
{
if (rt2x00_rt(rt2x00dev, RT3593))
rt2800_config_txpower_rt3593(rt2x00dev, chan, power_level);
+ else if (rt2x00_rt(rt2x00dev, RT6352))
+ rt2800_config_txpower_rt6352(rt2x00dev, chan, power_level);
else
rt2800_config_txpower_rt28xx(rt2x00dev, chan, power_level);
}
@@ -4336,6 +4843,7 @@
{
u32 tx_pin;
u8 rfcsr;
+ unsigned long min_sleep = 0;
/*
* A voltage-controlled oscillator(VCO) is an electronic oscillator
@@ -4374,6 +4882,15 @@
rt2800_rfcsr_read(rt2x00dev, 3, &rfcsr);
rt2x00_set_field8(&rfcsr, RFCSR3_VCOCAL_EN, 1);
rt2800_rfcsr_write(rt2x00dev, 3, rfcsr);
+ min_sleep = 1000;
+ break;
+ case RF7620:
+ rt2800_rfcsr_write(rt2x00dev, 5, 0x40);
+ rt2800_rfcsr_write(rt2x00dev, 4, 0x0C);
+ rt2800_rfcsr_read(rt2x00dev, 4, &rfcsr);
+ rt2x00_set_field8(&rfcsr, RFCSR4_VCOCAL_EN, 1);
+ rt2800_rfcsr_write(rt2x00dev, 4, rfcsr);
+ min_sleep = 2000;
break;
default:
WARN_ONCE(1, "Not supported RF chipet %x for VCO recalibration",
@@ -4381,7 +4898,8 @@
return;
}
- usleep_range(1000, 1500);
+ if (min_sleep > 0)
+ usleep_range(min_sleep, min_sleep * 2);
rt2800_register_read(rt2x00dev, TX_PIN_CFG, &tx_pin);
if (rt2x00dev->rf_channel <= 14) {
@@ -4413,6 +4931,42 @@
}
rt2800_register_write(rt2x00dev, TX_PIN_CFG, tx_pin);
+ if (rt2x00_rt(rt2x00dev, RT6352)) {
+ if (rt2x00dev->default_ant.rx_chain_num == 1) {
+ rt2800_bbp_write(rt2x00dev, 91, 0x07);
+ rt2800_bbp_write(rt2x00dev, 95, 0x1A);
+ rt2800_bbp_write(rt2x00dev, 195, 128);
+ rt2800_bbp_write(rt2x00dev, 196, 0xA0);
+ rt2800_bbp_write(rt2x00dev, 195, 170);
+ rt2800_bbp_write(rt2x00dev, 196, 0x12);
+ rt2800_bbp_write(rt2x00dev, 195, 171);
+ rt2800_bbp_write(rt2x00dev, 196, 0x10);
+ } else {
+ rt2800_bbp_write(rt2x00dev, 91, 0x06);
+ rt2800_bbp_write(rt2x00dev, 95, 0x9A);
+ rt2800_bbp_write(rt2x00dev, 195, 128);
+ rt2800_bbp_write(rt2x00dev, 196, 0xE0);
+ rt2800_bbp_write(rt2x00dev, 195, 170);
+ rt2800_bbp_write(rt2x00dev, 196, 0x30);
+ rt2800_bbp_write(rt2x00dev, 195, 171);
+ rt2800_bbp_write(rt2x00dev, 196, 0x30);
+ }
+
+ if (rt2x00_has_cap_external_lna_bg(rt2x00dev)) {
+ rt2800_bbp_write(rt2x00dev, 75, 0x68);
+ rt2800_bbp_write(rt2x00dev, 76, 0x4C);
+ rt2800_bbp_write(rt2x00dev, 79, 0x1C);
+ rt2800_bbp_write(rt2x00dev, 80, 0x0C);
+ rt2800_bbp_write(rt2x00dev, 82, 0xB6);
+ }
+
+ /* On 11A, We should delay and wait RF/BBP to be stable
+ * and the appropriate time should be 1000 micro seconds
+ * 2005/06/05 - On 11G, we also need this delay time.
+ * Otherwise it's difficult to pass the WHQL.
+ */
+ usleep_range(1000, 1500);
+ }
}
EXPORT_SYMBOL_GPL(rt2800_vco_calibration);
@@ -4511,7 +5065,8 @@
rt2x00_rt(rt2x00dev, RT3593) ||
rt2x00_rt(rt2x00dev, RT5390) ||
rt2x00_rt(rt2x00dev, RT5392) ||
- rt2x00_rt(rt2x00dev, RT5592))
+ rt2x00_rt(rt2x00dev, RT5592) ||
+ rt2x00_rt(rt2x00dev, RT6352))
vgc = 0x1c + (2 * rt2x00dev->lna_gain);
else
vgc = 0x2e + rt2x00dev->lna_gain;
@@ -4738,7 +5293,8 @@
0x00000000);
}
} else if (rt2x00_rt(rt2x00dev, RT5390) ||
- rt2x00_rt(rt2x00dev, RT5392)) {
+ rt2x00_rt(rt2x00dev, RT5392) ||
+ rt2x00_rt(rt2x00dev, RT6352)) {
rt2800_register_write(rt2x00dev, TX_SW_CFG0, 0x00000404);
rt2800_register_write(rt2x00dev, TX_SW_CFG1, 0x00080606);
rt2800_register_write(rt2x00dev, TX_SW_CFG2, 0x00000000);
@@ -4748,6 +5304,24 @@
rt2800_register_write(rt2x00dev, TX_SW_CFG2, 0x00000000);
} else if (rt2x00_rt(rt2x00dev, RT5350)) {
rt2800_register_write(rt2x00dev, TX_SW_CFG0, 0x00000404);
+ } else if (rt2x00_rt(rt2x00dev, RT6352)) {
+ rt2800_register_write(rt2x00dev, TX_SW_CFG0, 0x00000401);
+ rt2800_register_write(rt2x00dev, TX_SW_CFG1, 0x000C0000);
+ rt2800_register_write(rt2x00dev, TX_SW_CFG2, 0x00000000);
+ rt2800_register_write(rt2x00dev, MIMO_PS_CFG, 0x00000002);
+ rt2800_register_write(rt2x00dev, TX_PIN_CFG, 0x00150F0F);
+ rt2800_register_write(rt2x00dev, TX_ALC_VGA3, 0x06060606);
+ rt2800_register_write(rt2x00dev, TX0_BB_GAIN_ATTEN, 0x0);
+ rt2800_register_write(rt2x00dev, TX1_BB_GAIN_ATTEN, 0x0);
+ rt2800_register_write(rt2x00dev, TX0_RF_GAIN_ATTEN, 0x6C6C666C);
+ rt2800_register_write(rt2x00dev, TX1_RF_GAIN_ATTEN, 0x6C6C666C);
+ rt2800_register_write(rt2x00dev, TX0_RF_GAIN_CORRECT,
+ 0x3630363A);
+ rt2800_register_write(rt2x00dev, TX1_RF_GAIN_CORRECT,
+ 0x3630363A);
+ rt2800_register_read(rt2x00dev, TX_ALC_CFG_1, ®);
+ rt2x00_set_field32(®, TX_ALC_CFG_1_ROS_BUSY_EN, 0);
+ rt2800_register_write(rt2x00dev, TX_ALC_CFG_1, reg);
} else {
rt2800_register_write(rt2x00dev, TX_SW_CFG0, 0x00000000);
rt2800_register_write(rt2x00dev, TX_SW_CFG1, 0x00080606);
@@ -5729,6 +6303,231 @@
rt2800_bbp_write(rt2x00dev, 103, 0xc0);
}
+static void rt2800_bbp_glrt_write(struct rt2x00_dev *rt2x00dev,
+ const u8 reg, const u8 value)
+{
+ rt2800_bbp_write(rt2x00dev, 195, reg);
+ rt2800_bbp_write(rt2x00dev, 196, value);
+}
+
+static void rt2800_bbp_dcoc_write(struct rt2x00_dev *rt2x00dev,
+ const u8 reg, const u8 value)
+{
+ rt2800_bbp_write(rt2x00dev, 158, reg);
+ rt2800_bbp_write(rt2x00dev, 159, value);
+}
+
+static void rt2800_bbp_dcoc_read(struct rt2x00_dev *rt2x00dev,
+ const u8 reg, u8 *value)
+{
+ rt2800_bbp_write(rt2x00dev, 158, reg);
+ rt2800_bbp_read(rt2x00dev, 159, value);
+}
+
+static void rt2800_init_bbp_6352(struct rt2x00_dev *rt2x00dev)
+{
+ u8 bbp;
+
+ /* Apply Maximum Likelihood Detection (MLD) for 2 stream case */
+ rt2800_bbp_read(rt2x00dev, 105, &bbp);
+ rt2x00_set_field8(&bbp, BBP105_MLD,
+ rt2x00dev->default_ant.rx_chain_num == 2);
+ rt2800_bbp_write(rt2x00dev, 105, bbp);
+
+ /* Avoid data loss and CRC errors */
+ rt2800_bbp4_mac_if_ctrl(rt2x00dev);
+
+ /* Fix I/Q swap issue */
+ rt2800_bbp_read(rt2x00dev, 1, &bbp);
+ bbp |= 0x04;
+ rt2800_bbp_write(rt2x00dev, 1, bbp);
+
+ /* BBP for G band */
+ rt2800_bbp_write(rt2x00dev, 3, 0x08);
+ rt2800_bbp_write(rt2x00dev, 4, 0x00); /* rt2800_bbp4_mac_if_ctrl? */
+ rt2800_bbp_write(rt2x00dev, 6, 0x08);
+ rt2800_bbp_write(rt2x00dev, 14, 0x09);
+ rt2800_bbp_write(rt2x00dev, 15, 0xFF);
+ rt2800_bbp_write(rt2x00dev, 16, 0x01);
+ rt2800_bbp_write(rt2x00dev, 20, 0x06);
+ rt2800_bbp_write(rt2x00dev, 21, 0x00);
+ rt2800_bbp_write(rt2x00dev, 22, 0x00);
+ rt2800_bbp_write(rt2x00dev, 27, 0x00);
+ rt2800_bbp_write(rt2x00dev, 28, 0x00);
+ rt2800_bbp_write(rt2x00dev, 30, 0x00);
+ rt2800_bbp_write(rt2x00dev, 31, 0x48);
+ rt2800_bbp_write(rt2x00dev, 47, 0x40);
+ rt2800_bbp_write(rt2x00dev, 62, 0x00);
+ rt2800_bbp_write(rt2x00dev, 63, 0x00);
+ rt2800_bbp_write(rt2x00dev, 64, 0x00);
+ rt2800_bbp_write(rt2x00dev, 65, 0x2C);
+ rt2800_bbp_write(rt2x00dev, 66, 0x1C);
+ rt2800_bbp_write(rt2x00dev, 67, 0x20);
+ rt2800_bbp_write(rt2x00dev, 68, 0xDD);
+ rt2800_bbp_write(rt2x00dev, 69, 0x10);
+ rt2800_bbp_write(rt2x00dev, 70, 0x05);
+ rt2800_bbp_write(rt2x00dev, 73, 0x18);
+ rt2800_bbp_write(rt2x00dev, 74, 0x0F);
+ rt2800_bbp_write(rt2x00dev, 75, 0x60);
+ rt2800_bbp_write(rt2x00dev, 76, 0x44);
+ rt2800_bbp_write(rt2x00dev, 77, 0x59);
+ rt2800_bbp_write(rt2x00dev, 78, 0x1E);
+ rt2800_bbp_write(rt2x00dev, 79, 0x1C);
+ rt2800_bbp_write(rt2x00dev, 80, 0x0C);
+ rt2800_bbp_write(rt2x00dev, 81, 0x3A);
+ rt2800_bbp_write(rt2x00dev, 82, 0xB6);
+ rt2800_bbp_write(rt2x00dev, 83, 0x9A);
+ rt2800_bbp_write(rt2x00dev, 84, 0x9A);
+ rt2800_bbp_write(rt2x00dev, 86, 0x38);
+ rt2800_bbp_write(rt2x00dev, 88, 0x90);
+ rt2800_bbp_write(rt2x00dev, 91, 0x04);
+ rt2800_bbp_write(rt2x00dev, 92, 0x02);
+ rt2800_bbp_write(rt2x00dev, 95, 0x9A);
+ rt2800_bbp_write(rt2x00dev, 96, 0x00);
+ rt2800_bbp_write(rt2x00dev, 103, 0xC0);
+ rt2800_bbp_write(rt2x00dev, 104, 0x92);
+ /* FIXME BBP105 owerwrite */
+ rt2800_bbp_write(rt2x00dev, 105, 0x3C);
+ rt2800_bbp_write(rt2x00dev, 106, 0x12);
+ rt2800_bbp_write(rt2x00dev, 109, 0x00);
+ rt2800_bbp_write(rt2x00dev, 134, 0x10);
+ rt2800_bbp_write(rt2x00dev, 135, 0xA6);
+ rt2800_bbp_write(rt2x00dev, 137, 0x04);
+ rt2800_bbp_write(rt2x00dev, 142, 0x30);
+ rt2800_bbp_write(rt2x00dev, 143, 0xF7);
+ rt2800_bbp_write(rt2x00dev, 160, 0xEC);
+ rt2800_bbp_write(rt2x00dev, 161, 0xC4);
+ rt2800_bbp_write(rt2x00dev, 162, 0x77);
+ rt2800_bbp_write(rt2x00dev, 163, 0xF9);
+ rt2800_bbp_write(rt2x00dev, 164, 0x00);
+ rt2800_bbp_write(rt2x00dev, 165, 0x00);
+ rt2800_bbp_write(rt2x00dev, 186, 0x00);
+ rt2800_bbp_write(rt2x00dev, 187, 0x00);
+ rt2800_bbp_write(rt2x00dev, 188, 0x00);
+ rt2800_bbp_write(rt2x00dev, 186, 0x00);
+ rt2800_bbp_write(rt2x00dev, 187, 0x01);
+ rt2800_bbp_write(rt2x00dev, 188, 0x00);
+ rt2800_bbp_write(rt2x00dev, 189, 0x00);
+
+ rt2800_bbp_write(rt2x00dev, 91, 0x06);
+ rt2800_bbp_write(rt2x00dev, 92, 0x04);
+ rt2800_bbp_write(rt2x00dev, 93, 0x54);
+ rt2800_bbp_write(rt2x00dev, 99, 0x50);
+ rt2800_bbp_write(rt2x00dev, 148, 0x84);
+ rt2800_bbp_write(rt2x00dev, 167, 0x80);
+ rt2800_bbp_write(rt2x00dev, 178, 0xFF);
+ rt2800_bbp_write(rt2x00dev, 106, 0x13);
+
+ /* BBP for G band GLRT function (BBP_128 ~ BBP_221) */
+ rt2800_bbp_glrt_write(rt2x00dev, 0, 0x00);
+ rt2800_bbp_glrt_write(rt2x00dev, 1, 0x14);
+ rt2800_bbp_glrt_write(rt2x00dev, 2, 0x20);
+ rt2800_bbp_glrt_write(rt2x00dev, 3, 0x0A);
+ rt2800_bbp_glrt_write(rt2x00dev, 10, 0x16);
+ rt2800_bbp_glrt_write(rt2x00dev, 11, 0x06);
+ rt2800_bbp_glrt_write(rt2x00dev, 12, 0x02);
+ rt2800_bbp_glrt_write(rt2x00dev, 13, 0x07);
+ rt2800_bbp_glrt_write(rt2x00dev, 14, 0x05);
+ rt2800_bbp_glrt_write(rt2x00dev, 15, 0x09);
+ rt2800_bbp_glrt_write(rt2x00dev, 16, 0x20);
+ rt2800_bbp_glrt_write(rt2x00dev, 17, 0x08);
+ rt2800_bbp_glrt_write(rt2x00dev, 18, 0x4A);
+ rt2800_bbp_glrt_write(rt2x00dev, 19, 0x00);
+ rt2800_bbp_glrt_write(rt2x00dev, 20, 0x00);
+ rt2800_bbp_glrt_write(rt2x00dev, 128, 0xE0);
+ rt2800_bbp_glrt_write(rt2x00dev, 129, 0x1F);
+ rt2800_bbp_glrt_write(rt2x00dev, 130, 0x4F);
+ rt2800_bbp_glrt_write(rt2x00dev, 131, 0x32);
+ rt2800_bbp_glrt_write(rt2x00dev, 132, 0x08);
+ rt2800_bbp_glrt_write(rt2x00dev, 133, 0x28);
+ rt2800_bbp_glrt_write(rt2x00dev, 134, 0x19);
+ rt2800_bbp_glrt_write(rt2x00dev, 135, 0x0A);
+ rt2800_bbp_glrt_write(rt2x00dev, 138, 0x16);
+ rt2800_bbp_glrt_write(rt2x00dev, 139, 0x10);
+ rt2800_bbp_glrt_write(rt2x00dev, 140, 0x10);
+ rt2800_bbp_glrt_write(rt2x00dev, 141, 0x1A);
+ rt2800_bbp_glrt_write(rt2x00dev, 142, 0x36);
+ rt2800_bbp_glrt_write(rt2x00dev, 143, 0x2C);
+ rt2800_bbp_glrt_write(rt2x00dev, 144, 0x26);
+ rt2800_bbp_glrt_write(rt2x00dev, 145, 0x24);
+ rt2800_bbp_glrt_write(rt2x00dev, 146, 0x42);
+ rt2800_bbp_glrt_write(rt2x00dev, 147, 0x40);
+ rt2800_bbp_glrt_write(rt2x00dev, 148, 0x30);
+ rt2800_bbp_glrt_write(rt2x00dev, 149, 0x29);
+ rt2800_bbp_glrt_write(rt2x00dev, 150, 0x4C);
+ rt2800_bbp_glrt_write(rt2x00dev, 151, 0x46);
+ rt2800_bbp_glrt_write(rt2x00dev, 152, 0x3D);
+ rt2800_bbp_glrt_write(rt2x00dev, 153, 0x40);
+ rt2800_bbp_glrt_write(rt2x00dev, 154, 0x3E);
+ rt2800_bbp_glrt_write(rt2x00dev, 155, 0x38);
+ rt2800_bbp_glrt_write(rt2x00dev, 156, 0x3D);
+ rt2800_bbp_glrt_write(rt2x00dev, 157, 0x2F);
+ rt2800_bbp_glrt_write(rt2x00dev, 158, 0x3C);
+ rt2800_bbp_glrt_write(rt2x00dev, 159, 0x34);
+ rt2800_bbp_glrt_write(rt2x00dev, 160, 0x2C);
+ rt2800_bbp_glrt_write(rt2x00dev, 161, 0x2F);
+ rt2800_bbp_glrt_write(rt2x00dev, 162, 0x3C);
+ rt2800_bbp_glrt_write(rt2x00dev, 163, 0x35);
+ rt2800_bbp_glrt_write(rt2x00dev, 164, 0x2E);
+ rt2800_bbp_glrt_write(rt2x00dev, 165, 0x2F);
+ rt2800_bbp_glrt_write(rt2x00dev, 166, 0x49);
+ rt2800_bbp_glrt_write(rt2x00dev, 167, 0x41);
+ rt2800_bbp_glrt_write(rt2x00dev, 168, 0x36);
+ rt2800_bbp_glrt_write(rt2x00dev, 169, 0x39);
+ rt2800_bbp_glrt_write(rt2x00dev, 170, 0x30);
+ rt2800_bbp_glrt_write(rt2x00dev, 171, 0x30);
+ rt2800_bbp_glrt_write(rt2x00dev, 172, 0x0E);
+ rt2800_bbp_glrt_write(rt2x00dev, 173, 0x0D);
+ rt2800_bbp_glrt_write(rt2x00dev, 174, 0x28);
+ rt2800_bbp_glrt_write(rt2x00dev, 175, 0x21);
+ rt2800_bbp_glrt_write(rt2x00dev, 176, 0x1C);
+ rt2800_bbp_glrt_write(rt2x00dev, 177, 0x16);
+ rt2800_bbp_glrt_write(rt2x00dev, 178, 0x50);
+ rt2800_bbp_glrt_write(rt2x00dev, 179, 0x4A);
+ rt2800_bbp_glrt_write(rt2x00dev, 180, 0x43);
+ rt2800_bbp_glrt_write(rt2x00dev, 181, 0x50);
+ rt2800_bbp_glrt_write(rt2x00dev, 182, 0x10);
+ rt2800_bbp_glrt_write(rt2x00dev, 183, 0x10);
+ rt2800_bbp_glrt_write(rt2x00dev, 184, 0x10);
+ rt2800_bbp_glrt_write(rt2x00dev, 185, 0x10);
+ rt2800_bbp_glrt_write(rt2x00dev, 200, 0x7D);
+ rt2800_bbp_glrt_write(rt2x00dev, 201, 0x14);
+ rt2800_bbp_glrt_write(rt2x00dev, 202, 0x32);
+ rt2800_bbp_glrt_write(rt2x00dev, 203, 0x2C);
+ rt2800_bbp_glrt_write(rt2x00dev, 204, 0x36);
+ rt2800_bbp_glrt_write(rt2x00dev, 205, 0x4C);
+ rt2800_bbp_glrt_write(rt2x00dev, 206, 0x43);
+ rt2800_bbp_glrt_write(rt2x00dev, 207, 0x2C);
+ rt2800_bbp_glrt_write(rt2x00dev, 208, 0x2E);
+ rt2800_bbp_glrt_write(rt2x00dev, 209, 0x36);
+ rt2800_bbp_glrt_write(rt2x00dev, 210, 0x30);
+ rt2800_bbp_glrt_write(rt2x00dev, 211, 0x6E);
+
+ /* BBP for G band DCOC function */
+ rt2800_bbp_dcoc_write(rt2x00dev, 140, 0x0C);
+ rt2800_bbp_dcoc_write(rt2x00dev, 141, 0x00);
+ rt2800_bbp_dcoc_write(rt2x00dev, 142, 0x10);
+ rt2800_bbp_dcoc_write(rt2x00dev, 143, 0x10);
+ rt2800_bbp_dcoc_write(rt2x00dev, 144, 0x10);
+ rt2800_bbp_dcoc_write(rt2x00dev, 145, 0x10);
+ rt2800_bbp_dcoc_write(rt2x00dev, 146, 0x08);
+ rt2800_bbp_dcoc_write(rt2x00dev, 147, 0x40);
+ rt2800_bbp_dcoc_write(rt2x00dev, 148, 0x04);
+ rt2800_bbp_dcoc_write(rt2x00dev, 149, 0x04);
+ rt2800_bbp_dcoc_write(rt2x00dev, 150, 0x08);
+ rt2800_bbp_dcoc_write(rt2x00dev, 151, 0x08);
+ rt2800_bbp_dcoc_write(rt2x00dev, 152, 0x03);
+ rt2800_bbp_dcoc_write(rt2x00dev, 153, 0x03);
+ rt2800_bbp_dcoc_write(rt2x00dev, 154, 0x03);
+ rt2800_bbp_dcoc_write(rt2x00dev, 155, 0x02);
+ rt2800_bbp_dcoc_write(rt2x00dev, 156, 0x40);
+ rt2800_bbp_dcoc_write(rt2x00dev, 157, 0x40);
+ rt2800_bbp_dcoc_write(rt2x00dev, 158, 0x64);
+ rt2800_bbp_dcoc_write(rt2x00dev, 159, 0x64);
+
+ rt2800_bbp4_mac_if_ctrl(rt2x00dev);
+}
+
static void rt2800_init_bbp(struct rt2x00_dev *rt2x00dev)
{
unsigned int i;
@@ -5773,6 +6572,9 @@
case RT5592:
rt2800_init_bbp_5592(rt2x00dev);
return;
+ case RT6352:
+ rt2800_init_bbp_6352(rt2x00dev);
+ break;
}
for (i = 0; i < EEPROM_BBP_SIZE; i++) {
@@ -6228,9 +7030,9 @@
static void rt2800_init_rfcsr_3352(struct rt2x00_dev *rt2x00dev)
{
- int tx0_int_pa = test_bit(CAPABILITY_INTERNAL_PA_TX0,
+ int tx0_ext_pa = test_bit(CAPABILITY_EXTERNAL_PA_TX0,
&rt2x00dev->cap_flags);
- int tx1_int_pa = test_bit(CAPABILITY_INTERNAL_PA_TX1,
+ int tx1_ext_pa = test_bit(CAPABILITY_EXTERNAL_PA_TX1,
&rt2x00dev->cap_flags);
u8 rfcsr;
@@ -6270,9 +7072,9 @@
rt2800_rfcsr_write(rt2x00dev, 32, 0x80);
rt2800_rfcsr_write(rt2x00dev, 33, 0x00);
rfcsr = 0x01;
- if (!tx0_int_pa)
+ if (tx0_ext_pa)
rt2x00_set_field8(&rfcsr, RFCSR34_TX0_EXT_PA, 1);
- if (!tx1_int_pa)
+ if (tx1_ext_pa)
rt2x00_set_field8(&rfcsr, RFCSR34_TX1_EXT_PA, 1);
rt2800_rfcsr_write(rt2x00dev, 34, rfcsr);
rt2800_rfcsr_write(rt2x00dev, 35, 0x03);
@@ -6282,13 +7084,13 @@
rt2800_rfcsr_write(rt2x00dev, 39, 0xc5);
rt2800_rfcsr_write(rt2x00dev, 40, 0x33);
rfcsr = 0x52;
- if (tx0_int_pa) {
+ if (!tx0_ext_pa) {
rt2x00_set_field8(&rfcsr, RFCSR41_BIT1, 1);
rt2x00_set_field8(&rfcsr, RFCSR41_BIT4, 1);
}
rt2800_rfcsr_write(rt2x00dev, 41, rfcsr);
rfcsr = 0x52;
- if (tx1_int_pa) {
+ if (!tx1_ext_pa) {
rt2x00_set_field8(&rfcsr, RFCSR42_BIT1, 1);
rt2x00_set_field8(&rfcsr, RFCSR42_BIT4, 1);
}
@@ -6301,19 +7103,19 @@
rt2800_rfcsr_write(rt2x00dev, 48, 0x14);
rt2800_rfcsr_write(rt2x00dev, 49, 0x00);
rfcsr = 0x2d;
- if (!tx0_int_pa)
+ if (tx0_ext_pa)
rt2x00_set_field8(&rfcsr, RFCSR50_TX0_EXT_PA, 1);
- if (!tx1_int_pa)
+ if (tx1_ext_pa)
rt2x00_set_field8(&rfcsr, RFCSR50_TX1_EXT_PA, 1);
rt2800_rfcsr_write(rt2x00dev, 50, rfcsr);
- rt2800_rfcsr_write(rt2x00dev, 51, (tx0_int_pa ? 0x7f : 0x52));
- rt2800_rfcsr_write(rt2x00dev, 52, (tx0_int_pa ? 0x00 : 0xc0));
- rt2800_rfcsr_write(rt2x00dev, 53, (tx0_int_pa ? 0x52 : 0xd2));
- rt2800_rfcsr_write(rt2x00dev, 54, (tx0_int_pa ? 0x1b : 0xc0));
- rt2800_rfcsr_write(rt2x00dev, 55, (tx1_int_pa ? 0x7f : 0x52));
- rt2800_rfcsr_write(rt2x00dev, 56, (tx1_int_pa ? 0x00 : 0xc0));
- rt2800_rfcsr_write(rt2x00dev, 57, (tx0_int_pa ? 0x52 : 0x49));
- rt2800_rfcsr_write(rt2x00dev, 58, (tx1_int_pa ? 0x1b : 0xc0));
+ rt2800_rfcsr_write(rt2x00dev, 51, (tx0_ext_pa ? 0x52 : 0x7f));
+ rt2800_rfcsr_write(rt2x00dev, 52, (tx0_ext_pa ? 0xc0 : 0x00));
+ rt2800_rfcsr_write(rt2x00dev, 53, (tx0_ext_pa ? 0xd2 : 0x52));
+ rt2800_rfcsr_write(rt2x00dev, 54, (tx0_ext_pa ? 0xc0 : 0x1b));
+ rt2800_rfcsr_write(rt2x00dev, 55, (tx1_ext_pa ? 0x52 : 0x7f));
+ rt2800_rfcsr_write(rt2x00dev, 56, (tx1_ext_pa ? 0xc0 : 0x00));
+ rt2800_rfcsr_write(rt2x00dev, 57, (tx0_ext_pa ? 0x49 : 0x52));
+ rt2800_rfcsr_write(rt2x00dev, 58, (tx1_ext_pa ? 0xc0 : 0x1b));
rt2800_rfcsr_write(rt2x00dev, 59, 0x00);
rt2800_rfcsr_write(rt2x00dev, 60, 0x00);
rt2800_rfcsr_write(rt2x00dev, 61, 0x00);
@@ -6844,6 +7646,617 @@
rt2800_led_open_drain_enable(rt2x00dev);
}
+static void rt2800_bbp_core_soft_reset(struct rt2x00_dev *rt2x00dev,
+ bool set_bw, bool is_ht40)
+{
+ u8 bbp_val;
+
+ rt2800_bbp_read(rt2x00dev, 21, &bbp_val);
+ bbp_val |= 0x1;
+ rt2800_bbp_write(rt2x00dev, 21, bbp_val);
+ usleep_range(100, 200);
+
+ if (set_bw) {
+ rt2800_bbp_read(rt2x00dev, 4, &bbp_val);
+ rt2x00_set_field8(&bbp_val, BBP4_BANDWIDTH, 2 * is_ht40);
+ rt2800_bbp_write(rt2x00dev, 4, bbp_val);
+ usleep_range(100, 200);
+ }
+
+ rt2800_bbp_read(rt2x00dev, 21, &bbp_val);
+ bbp_val &= (~0x1);
+ rt2800_bbp_write(rt2x00dev, 21, bbp_val);
+ usleep_range(100, 200);
+}
+
+static int rt2800_rf_lp_config(struct rt2x00_dev *rt2x00dev, bool btxcal)
+{
+ u8 rf_val;
+
+ if (btxcal)
+ rt2800_register_write(rt2x00dev, RF_CONTROL0, 0x04);
+ else
+ rt2800_register_write(rt2x00dev, RF_CONTROL0, 0x02);
+
+ rt2800_register_write(rt2x00dev, RF_BYPASS0, 0x06);
+
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 17, &rf_val);
+ rf_val |= 0x80;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 17, rf_val);
+
+ if (btxcal) {
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 18, 0xC1);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 19, 0x20);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 20, 0x02);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 3, &rf_val);
+ rf_val &= (~0x3F);
+ rf_val |= 0x3F;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 3, rf_val);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 4, &rf_val);
+ rf_val &= (~0x3F);
+ rf_val |= 0x3F;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 4, rf_val);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 5, 0x31);
+ } else {
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 18, 0xF1);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 19, 0x18);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 20, 0x02);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 3, &rf_val);
+ rf_val &= (~0x3F);
+ rf_val |= 0x34;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 3, rf_val);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 4, &rf_val);
+ rf_val &= (~0x3F);
+ rf_val |= 0x34;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 4, rf_val);
+ }
+
+ return 0;
+}
+
+static char rt2800_lp_tx_filter_bw_cal(struct rt2x00_dev *rt2x00dev)
+{
+ unsigned int cnt;
+ u8 bbp_val;
+ char cal_val;
+
+ rt2800_bbp_dcoc_write(rt2x00dev, 0, 0x82);
+
+ cnt = 0;
+ do {
+ usleep_range(500, 2000);
+ rt2800_bbp_read(rt2x00dev, 159, &bbp_val);
+ if (bbp_val == 0x02 || cnt == 20)
+ break;
+
+ cnt++;
+ } while (cnt < 20);
+
+ rt2800_bbp_dcoc_read(rt2x00dev, 0x39, &bbp_val);
+ cal_val = bbp_val & 0x7F;
+ if (cal_val >= 0x40)
+ cal_val -= 128;
+
+ return cal_val;
+}
+
+static void rt2800_bw_filter_calibration(struct rt2x00_dev *rt2x00dev,
+ bool btxcal)
+{
+ struct rt2800_drv_data *drv_data = rt2x00dev->drv_data;
+ u8 tx_agc_fc = 0, rx_agc_fc = 0, cmm_agc_fc;
+ u8 filter_target;
+ u8 tx_filter_target_20m = 0x09, tx_filter_target_40m = 0x02;
+ u8 rx_filter_target_20m = 0x27, rx_filter_target_40m = 0x31;
+ int loop = 0, is_ht40, cnt;
+ u8 bbp_val, rf_val;
+ char cal_r32_init, cal_r32_val, cal_diff;
+ u8 saverfb5r00, saverfb5r01, saverfb5r03, saverfb5r04, saverfb5r05;
+ u8 saverfb5r06, saverfb5r07;
+ u8 saverfb5r08, saverfb5r17, saverfb5r18, saverfb5r19, saverfb5r20;
+ u8 saverfb5r37, saverfb5r38, saverfb5r39, saverfb5r40, saverfb5r41;
+ u8 saverfb5r42, saverfb5r43, saverfb5r44, saverfb5r45, saverfb5r46;
+ u8 saverfb5r58, saverfb5r59;
+ u8 savebbp159r0, savebbp159r2, savebbpr23;
+ u32 MAC_RF_CONTROL0, MAC_RF_BYPASS0;
+
+ /* Save MAC registers */
+ rt2800_register_read(rt2x00dev, RF_CONTROL0, &MAC_RF_CONTROL0);
+ rt2800_register_read(rt2x00dev, RF_BYPASS0, &MAC_RF_BYPASS0);
+
+ /* save BBP registers */
+ rt2800_bbp_read(rt2x00dev, 23, &savebbpr23);
+
+ rt2800_bbp_dcoc_read(rt2x00dev, 0, &savebbp159r0);
+ rt2800_bbp_dcoc_read(rt2x00dev, 2, &savebbp159r2);
+
+ /* Save RF registers */
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 0, &saverfb5r00);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 1, &saverfb5r01);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 3, &saverfb5r03);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 4, &saverfb5r04);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 5, &saverfb5r05);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 6, &saverfb5r06);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 7, &saverfb5r07);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 8, &saverfb5r08);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 17, &saverfb5r17);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 18, &saverfb5r18);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 19, &saverfb5r19);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 20, &saverfb5r20);
+
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 37, &saverfb5r37);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 38, &saverfb5r38);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 39, &saverfb5r39);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 40, &saverfb5r40);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 41, &saverfb5r41);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 42, &saverfb5r42);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 43, &saverfb5r43);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 44, &saverfb5r44);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 45, &saverfb5r45);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 46, &saverfb5r46);
+
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 58, &saverfb5r58);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 59, &saverfb5r59);
+
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 0, &rf_val);
+ rf_val |= 0x3;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 0, rf_val);
+
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 1, &rf_val);
+ rf_val |= 0x1;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 1, rf_val);
+
+ cnt = 0;
+ do {
+ usleep_range(500, 2000);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 1, &rf_val);
+ if (((rf_val & 0x1) == 0x00) || (cnt == 40))
+ break;
+ cnt++;
+ } while (cnt < 40);
+
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 0, &rf_val);
+ rf_val &= (~0x3);
+ rf_val |= 0x1;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 0, rf_val);
+
+ /* I-3 */
+ rt2800_bbp_read(rt2x00dev, 23, &bbp_val);
+ bbp_val &= (~0x1F);
+ bbp_val |= 0x10;
+ rt2800_bbp_write(rt2x00dev, 23, bbp_val);
+
+ do {
+ /* I-4,5,6,7,8,9 */
+ if (loop == 0) {
+ is_ht40 = false;
+
+ if (btxcal)
+ filter_target = tx_filter_target_20m;
+ else
+ filter_target = rx_filter_target_20m;
+ } else {
+ is_ht40 = true;
+
+ if (btxcal)
+ filter_target = tx_filter_target_40m;
+ else
+ filter_target = rx_filter_target_40m;
+ }
+
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 8, &rf_val);
+ rf_val &= (~0x04);
+ if (loop == 1)
+ rf_val |= 0x4;
+
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 8, rf_val);
+
+ rt2800_bbp_core_soft_reset(rt2x00dev, true, is_ht40);
+
+ rt2800_rf_lp_config(rt2x00dev, btxcal);
+ if (btxcal) {
+ tx_agc_fc = 0;
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 58, &rf_val);
+ rf_val &= (~0x7F);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 58, rf_val);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 59, &rf_val);
+ rf_val &= (~0x7F);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 59, rf_val);
+ } else {
+ rx_agc_fc = 0;
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 6, &rf_val);
+ rf_val &= (~0x7F);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 6, rf_val);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 7, &rf_val);
+ rf_val &= (~0x7F);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 7, rf_val);
+ }
+
+ usleep_range(1000, 2000);
+
+ rt2800_bbp_dcoc_read(rt2x00dev, 2, &bbp_val);
+ bbp_val &= (~0x6);
+ rt2800_bbp_dcoc_write(rt2x00dev, 2, bbp_val);
+
+ rt2800_bbp_core_soft_reset(rt2x00dev, false, is_ht40);
+
+ cal_r32_init = rt2800_lp_tx_filter_bw_cal(rt2x00dev);
+
+ rt2800_bbp_dcoc_read(rt2x00dev, 2, &bbp_val);
+ bbp_val |= 0x6;
+ rt2800_bbp_dcoc_write(rt2x00dev, 2, bbp_val);
+do_cal:
+ if (btxcal) {
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 58, &rf_val);
+ rf_val &= (~0x7F);
+ rf_val |= tx_agc_fc;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 58, rf_val);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 59, &rf_val);
+ rf_val &= (~0x7F);
+ rf_val |= tx_agc_fc;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 59, rf_val);
+ } else {
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 6, &rf_val);
+ rf_val &= (~0x7F);
+ rf_val |= rx_agc_fc;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 6, rf_val);
+ rt2800_rfcsr_read_bank(rt2x00dev, 5, 7, &rf_val);
+ rf_val &= (~0x7F);
+ rf_val |= rx_agc_fc;
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 7, rf_val);
+ }
+
+ usleep_range(500, 1000);
+
+ rt2800_bbp_core_soft_reset(rt2x00dev, false, is_ht40);
+
+ cal_r32_val = rt2800_lp_tx_filter_bw_cal(rt2x00dev);
+
+ cal_diff = cal_r32_init - cal_r32_val;
+
+ if (btxcal)
+ cmm_agc_fc = tx_agc_fc;
+ else
+ cmm_agc_fc = rx_agc_fc;
+
+ if (((cal_diff > filter_target) && (cmm_agc_fc == 0)) ||
+ ((cal_diff < filter_target) && (cmm_agc_fc == 0x3f))) {
+ if (btxcal)
+ tx_agc_fc = 0;
+ else
+ rx_agc_fc = 0;
+ } else if ((cal_diff <= filter_target) && (cmm_agc_fc < 0x3f)) {
+ if (btxcal)
+ tx_agc_fc++;
+ else
+ rx_agc_fc++;
+ goto do_cal;
+ }
+
+ if (btxcal) {
+ if (loop == 0)
+ drv_data->tx_calibration_bw20 = tx_agc_fc;
+ else
+ drv_data->tx_calibration_bw40 = tx_agc_fc;
+ } else {
+ if (loop == 0)
+ drv_data->rx_calibration_bw20 = rx_agc_fc;
+ else
+ drv_data->rx_calibration_bw40 = rx_agc_fc;
+ }
+
+ loop++;
+ } while (loop <= 1);
+
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 0, saverfb5r00);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 1, saverfb5r01);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 3, saverfb5r03);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 4, saverfb5r04);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 5, saverfb5r05);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 6, saverfb5r06);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 7, saverfb5r07);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 8, saverfb5r08);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 17, saverfb5r17);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 18, saverfb5r18);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 19, saverfb5r19);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 20, saverfb5r20);
+
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 37, saverfb5r37);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 38, saverfb5r38);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 39, saverfb5r39);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 40, saverfb5r40);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 41, saverfb5r41);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 42, saverfb5r42);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 43, saverfb5r43);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 44, saverfb5r44);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 45, saverfb5r45);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 46, saverfb5r46);
+
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 58, saverfb5r58);
+ rt2800_rfcsr_write_bank(rt2x00dev, 5, 59, saverfb5r59);
+
+ rt2800_bbp_write(rt2x00dev, 23, savebbpr23);
+
+ rt2800_bbp_dcoc_write(rt2x00dev, 0, savebbp159r0);
+ rt2800_bbp_dcoc_write(rt2x00dev, 2, savebbp159r2);
+
+ rt2800_bbp_read(rt2x00dev, 4, &bbp_val);
+ rt2x00_set_field8(&bbp_val, BBP4_BANDWIDTH,
+ 2 * test_bit(CONFIG_CHANNEL_HT40, &rt2x00dev->flags));
+ rt2800_bbp_write(rt2x00dev, 4, bbp_val);
+
+ rt2800_register_write(rt2x00dev, RF_CONTROL0, MAC_RF_CONTROL0);
+ rt2800_register_write(rt2x00dev, RF_BYPASS0, MAC_RF_BYPASS0);
+}
+
+static void rt2800_init_rfcsr_6352(struct rt2x00_dev *rt2x00dev)
+{
+ /* Initialize RF central register to default value */
+ rt2800_rfcsr_write(rt2x00dev, 0, 0x02);
+ rt2800_rfcsr_write(rt2x00dev, 1, 0x03);
+ rt2800_rfcsr_write(rt2x00dev, 2, 0x33);
+ rt2800_rfcsr_write(rt2x00dev, 3, 0xFF);
+ rt2800_rfcsr_write(rt2x00dev, 4, 0x0C);
+ rt2800_rfcsr_write(rt2x00dev, 5, 0x40);
+ rt2800_rfcsr_write(rt2x00dev, 6, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 7, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 8, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 9, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 10, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 11, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 12, rt2x00dev->freq_offset);
+ rt2800_rfcsr_write(rt2x00dev, 13, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 14, 0x40);
+ rt2800_rfcsr_write(rt2x00dev, 15, 0x22);
+ rt2800_rfcsr_write(rt2x00dev, 16, 0x4C);
+ rt2800_rfcsr_write(rt2x00dev, 17, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 18, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 19, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 20, 0xA0);
+ rt2800_rfcsr_write(rt2x00dev, 21, 0x12);
+ rt2800_rfcsr_write(rt2x00dev, 22, 0x07);
+ rt2800_rfcsr_write(rt2x00dev, 23, 0x13);
+ rt2800_rfcsr_write(rt2x00dev, 24, 0xFE);
+ rt2800_rfcsr_write(rt2x00dev, 25, 0x24);
+ rt2800_rfcsr_write(rt2x00dev, 26, 0x7A);
+ rt2800_rfcsr_write(rt2x00dev, 27, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 28, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 29, 0x05);
+ rt2800_rfcsr_write(rt2x00dev, 30, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 31, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 32, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 33, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 34, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 35, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 36, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 37, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 38, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 39, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 40, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 41, 0xD0);
+ rt2800_rfcsr_write(rt2x00dev, 42, 0x5B);
+ rt2800_rfcsr_write(rt2x00dev, 43, 0x00);
+
+ rt2800_rfcsr_write(rt2x00dev, 11, 0x21);
+ if (rt2800_clk_is_20mhz(rt2x00dev))
+ rt2800_rfcsr_write(rt2x00dev, 13, 0x03);
+ else
+ rt2800_rfcsr_write(rt2x00dev, 13, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 14, 0x7C);
+ rt2800_rfcsr_write(rt2x00dev, 16, 0x80);
+ rt2800_rfcsr_write(rt2x00dev, 17, 0x99);
+ rt2800_rfcsr_write(rt2x00dev, 18, 0x99);
+ rt2800_rfcsr_write(rt2x00dev, 19, 0x09);
+ rt2800_rfcsr_write(rt2x00dev, 20, 0x50);
+ rt2800_rfcsr_write(rt2x00dev, 21, 0xB0);
+ rt2800_rfcsr_write(rt2x00dev, 22, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 23, 0x06);
+ rt2800_rfcsr_write(rt2x00dev, 24, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 25, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 26, 0x5D);
+ rt2800_rfcsr_write(rt2x00dev, 27, 0x00);
+ rt2800_rfcsr_write(rt2x00dev, 28, 0x61);
+ rt2800_rfcsr_write(rt2x00dev, 29, 0xB5);
+ rt2800_rfcsr_write(rt2x00dev, 43, 0x02);
+
+ rt2800_rfcsr_write(rt2x00dev, 28, 0x62);
+ rt2800_rfcsr_write(rt2x00dev, 29, 0xAD);
+ rt2800_rfcsr_write(rt2x00dev, 39, 0x80);
+
+ /* Initialize RF channel register to default value */
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 0, 0x03);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 1, 0x00);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 2, 0x00);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 3, 0x00);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 4, 0x00);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 5, 0x08);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 6, 0x00);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 7, 0x51);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 8, 0x53);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 9, 0x16);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 10, 0x61);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 11, 0x53);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 12, 0x22);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 13, 0x3D);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 14, 0x06);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 15, 0x13);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 16, 0x22);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 17, 0x27);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 18, 0x02);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 19, 0xA7);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 20, 0x01);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 21, 0x52);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 22, 0x80);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 23, 0xB3);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 24, 0x00);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 25, 0x00);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 26, 0x00);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 27, 0x00);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 28, 0x5C);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 29, 0x6B);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 30, 0x6B);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 31, 0x31);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 32, 0x5D);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 33, 0x00);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 34, 0xE6);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 35, 0x55);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 36, 0x00);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 37, 0xBB);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 38, 0xB3);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 39, 0xB3);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 40, 0x03);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 41, 0x00);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 42, 0x00);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 43, 0xB3);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 44, 0xD3);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 45, 0xD5);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 46, 0x07);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 47, 0x68);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 48, 0xEF);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 49, 0x1C);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 54, 0x07);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 55, 0xA8);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 56, 0x85);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 57, 0x10);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 58, 0x07);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 59, 0x6A);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 60, 0x85);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 61, 0x10);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 62, 0x1C);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 63, 0x00);
+
+ rt2800_rfcsr_write_bank(rt2x00dev, 6, 45, 0xC5);
+
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 9, 0x47);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 10, 0x71);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 11, 0x33);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 14, 0x0E);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 17, 0x23);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 19, 0xA4);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 20, 0x02);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 21, 0x12);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 28, 0x1C);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 29, 0xEB);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 32, 0x7D);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 34, 0xD6);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 36, 0x08);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 38, 0xB4);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 43, 0xD3);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 44, 0xB3);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 45, 0xD5);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 46, 0x27);
+ rt2800_rfcsr_write_bank(rt2x00dev, 4, 47, 0x67);
+ rt2800_rfcsr_write_bank(rt2x00dev, 6, 47, 0x69);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 48, 0xFF);
+ rt2800_rfcsr_write_bank(rt2x00dev, 4, 54, 0x27);
+ rt2800_rfcsr_write_bank(rt2x00dev, 6, 54, 0x20);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 55, 0x66);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 56, 0xFF);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 57, 0x1C);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 58, 0x20);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 59, 0x6B);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 60, 0xF7);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 61, 0x09);
+
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 10, 0x51);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 14, 0x06);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 19, 0xA7);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 28, 0x2C);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 55, 0x64);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 8, 0x51);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 9, 0x36);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 11, 0x53);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 14, 0x16);
+
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 47, 0x6C);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 48, 0xFC);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 49, 0x1F);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 54, 0x27);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 55, 0x66);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 59, 0x6B);
+
+ /* Initialize RF channel register for DRQFN */
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 43, 0xD3);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 44, 0xE3);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 45, 0xE5);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 47, 0x28);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 55, 0x68);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 56, 0xF7);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 58, 0x02);
+ rt2800_rfcsr_write_chanreg(rt2x00dev, 60, 0xC7);
+
+ /* Initialize RF DC calibration register to default value */
+ rt2800_rfcsr_write_dccal(rt2x00dev, 0, 0x47);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 1, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 2, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 3, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 4, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 5, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 6, 0x10);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 7, 0x10);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 8, 0x04);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 9, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 10, 0x07);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 11, 0x01);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 12, 0x07);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 13, 0x07);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 14, 0x07);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 15, 0x20);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 16, 0x22);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 17, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 18, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 19, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 20, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 21, 0xF1);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 22, 0x11);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 23, 0x02);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 24, 0x41);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 25, 0x20);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 26, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 27, 0xD7);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 28, 0xA2);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 29, 0x20);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 30, 0x49);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 31, 0x20);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 32, 0x04);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 33, 0xF1);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 34, 0xA1);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 35, 0x01);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 41, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 42, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 43, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 44, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 45, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 46, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 47, 0x3E);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 48, 0x3D);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 49, 0x3E);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 50, 0x3D);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 51, 0x3E);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 52, 0x3D);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 53, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 54, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 55, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 56, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 57, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 58, 0x10);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 59, 0x10);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 60, 0x0A);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 61, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 62, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 63, 0x00);
+
+ rt2800_rfcsr_write_dccal(rt2x00dev, 3, 0x08);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 4, 0x04);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 5, 0x20);
+
+ rt2800_rfcsr_write_dccal(rt2x00dev, 5, 0x00);
+ rt2800_rfcsr_write_dccal(rt2x00dev, 17, 0x7C);
+
+ rt2800_bw_filter_calibration(rt2x00dev, true);
+ rt2800_bw_filter_calibration(rt2x00dev, false);
+}
+
static void rt2800_init_rfcsr(struct rt2x00_dev *rt2x00dev)
{
if (rt2800_is_305x_soc(rt2x00dev)) {
@@ -6884,6 +8297,9 @@
case RT5592:
rt2800_init_rfcsr_5592(rt2x00dev);
break;
+ case RT6352:
+ rt2800_init_rfcsr_6352(rt2x00dev);
+ break;
}
}
@@ -7250,7 +8666,8 @@
*/
if (rt2x00_rt(rt2x00dev, RT3290) ||
rt2x00_rt(rt2x00dev, RT5390) ||
- rt2x00_rt(rt2x00dev, RT5392))
+ rt2x00_rt(rt2x00dev, RT5392) ||
+ rt2x00_rt(rt2x00dev, RT6352))
rt2800_eeprom_read(rt2x00dev, EEPROM_CHIP_ID, &rf);
else if (rt2x00_rt(rt2x00dev, RT3352))
rf = RF3322;
@@ -7282,6 +8699,7 @@
case RF5390:
case RF5392:
case RF5592:
+ case RF7620:
break;
default:
rt2x00_err(rt2x00dev, "Invalid RF chipset 0x%04x detected\n",
@@ -7382,13 +8800,13 @@
rt2800_eeprom_read(rt2x00dev, EEPROM_NIC_CONF1, &eeprom);
if (rt2x00_rt(rt2x00dev, RT3352)) {
- if (!rt2x00_get_field16(eeprom,
+ if (rt2x00_get_field16(eeprom,
EEPROM_NIC_CONF1_EXTERNAL_TX0_PA_3352))
- __set_bit(CAPABILITY_INTERNAL_PA_TX0,
+ __set_bit(CAPABILITY_EXTERNAL_PA_TX0,
&rt2x00dev->cap_flags);
- if (!rt2x00_get_field16(eeprom,
+ if (rt2x00_get_field16(eeprom,
EEPROM_NIC_CONF1_EXTERNAL_TX1_PA_3352))
- __set_bit(CAPABILITY_INTERNAL_PA_TX1,
+ __set_bit(CAPABILITY_EXTERNAL_PA_TX1,
&rt2x00dev->cap_flags);
}
@@ -7689,6 +9107,23 @@
{196, 83, 0, 12, 1},
};
+static const struct rf_channel rf_vals_7620[] = {
+ {1, 0x50, 0x99, 0x99, 1},
+ {2, 0x50, 0x44, 0x44, 2},
+ {3, 0x50, 0xEE, 0xEE, 2},
+ {4, 0x50, 0x99, 0x99, 3},
+ {5, 0x51, 0x44, 0x44, 0},
+ {6, 0x51, 0xEE, 0xEE, 0},
+ {7, 0x51, 0x99, 0x99, 1},
+ {8, 0x51, 0x44, 0x44, 2},
+ {9, 0x51, 0xEE, 0xEE, 2},
+ {10, 0x51, 0x99, 0x99, 3},
+ {11, 0x52, 0x44, 0x44, 0},
+ {12, 0x52, 0xEE, 0xEE, 0},
+ {13, 0x52, 0x99, 0x99, 1},
+ {14, 0x52, 0x33, 0x33, 3},
+};
+
static int rt2800_probe_hw_mode(struct rt2x00_dev *rt2x00dev)
{
struct hw_mode_spec *spec = &rt2x00dev->spec;
@@ -7792,6 +9227,11 @@
spec->channels = rf_vals_3x;
break;
+ case RF7620:
+ spec->num_channels = ARRAY_SIZE(rf_vals_7620);
+ spec->channels = rf_vals_7620;
+ break;
+
case RF3052:
case RF3053:
spec->num_channels = ARRAY_SIZE(rf_vals_3x);
@@ -7923,6 +9363,7 @@
case RF5390:
case RF5392:
case RF5592:
+ case RF7620:
__set_bit(CAPABILITY_VCO_RECALIBRATION, &rt2x00dev->cap_flags);
break;
}
@@ -7967,6 +9408,9 @@
return -ENODEV;
}
+ if (rt == RT5390 && rt2x00_is_soc(rt2x00dev))
+ rt = RT6352;
+
rt2x00_set_rt(rt2x00dev, rt, rev);
return 0;
diff --git a/drivers/net/wireless/ralink/rt2x00/rt2800lib.h b/drivers/net/wireless/ralink/rt2x00/rt2800lib.h
index 0a8b4df..f357531 100644
--- a/drivers/net/wireless/ralink/rt2x00/rt2800lib.h
+++ b/drivers/net/wireless/ralink/rt2x00/rt2800lib.h
@@ -20,6 +20,34 @@
#ifndef RT2800LIB_H
#define RT2800LIB_H
+/*
+ * Hardware has 255 WCID table entries. First 32 entries are reserved for
+ * shared keys. Since parts of the pairwise key table might be shared with
+ * the beacon frame buffers 6 & 7 we could only use the first 222 entries.
+ */
+#define WCID_START 33
+#define WCID_END 222
+#define STA_IDS_SIZE (WCID_END - WCID_START + 2)
+
+/* RT2800 driver data structure */
+struct rt2800_drv_data {
+ u8 calibration_bw20;
+ u8 calibration_bw40;
+ char rx_calibration_bw20;
+ char rx_calibration_bw40;
+ char tx_calibration_bw20;
+ char tx_calibration_bw40;
+ u8 bbp25;
+ u8 bbp26;
+ u8 txmixer_gain_24g;
+ u8 txmixer_gain_5g;
+ u8 max_psdu;
+ unsigned int tbtt_tick;
+ unsigned int ampdu_factor_cnt[4];
+ DECLARE_BITMAP(sta_ids, STA_IDS_SIZE);
+ struct ieee80211_sta *wcid_to_sta[STA_IDS_SIZE];
+};
+
struct rt2800_ops {
void (*register_read)(struct rt2x00_dev *rt2x00dev,
const unsigned int offset, u32 *value);
@@ -167,7 +195,8 @@
struct txentry_desc *txdesc);
void rt2800_process_rxwi(struct queue_entry *entry, struct rxdone_entry_desc *txdesc);
-void rt2800_txdone_entry(struct queue_entry *entry, u32 status, __le32* txwi);
+void rt2800_txdone_entry(struct queue_entry *entry, u32 status, __le32 *txwi,
+ bool match);
void rt2800_write_beacon(struct queue_entry *entry, struct txentry_desc *txdesc);
void rt2800_clear_beacon(struct queue_entry *entry);
diff --git a/drivers/net/wireless/ralink/rt2x00/rt2800mmio.c b/drivers/net/wireless/ralink/rt2x00/rt2800mmio.c
index de4790b..3ab3b53 100644
--- a/drivers/net/wireless/ralink/rt2x00/rt2800mmio.c
+++ b/drivers/net/wireless/ralink/rt2x00/rt2800mmio.c
@@ -239,7 +239,7 @@
{
if (test_bit(ENTRY_DATA_STATUS_SET, &entry->flags)) {
rt2800_txdone_entry(entry, entry->status,
- rt2800mmio_get_txwi(entry));
+ rt2800mmio_get_txwi(entry), true);
return false;
}
diff --git a/drivers/net/wireless/ralink/rt2x00/rt2800usb.c b/drivers/net/wireless/ralink/rt2x00/rt2800usb.c
index 205a7b8..f11e3f5 100644
--- a/drivers/net/wireless/ralink/rt2x00/rt2800usb.c
+++ b/drivers/net/wireless/ralink/rt2x00/rt2800usb.c
@@ -501,8 +501,7 @@
/*
* TX control handlers
*/
-static enum txdone_entry_desc_flags
-rt2800usb_txdone_entry_check(struct queue_entry *entry, u32 reg)
+static bool rt2800usb_txdone_entry_check(struct queue_entry *entry, u32 reg)
{
__le32 *txwi;
u32 word;
@@ -515,7 +514,7 @@
* frame.
*/
if (test_bit(ENTRY_DATA_IO_FAILED, &entry->flags))
- return TXDONE_FAILURE;
+ return false;
wcid = rt2x00_get_field32(reg, TX_STA_FIFO_WCID);
ack = rt2x00_get_field32(reg, TX_STA_FIFO_TX_ACK_REQUIRED);
@@ -537,10 +536,10 @@
rt2x00_dbg(entry->queue->rt2x00dev,
"TX status report missed for queue %d entry %d\n",
entry->queue->qid, entry->entry_idx);
- return TXDONE_UNKNOWN;
+ return false;
}
- return TXDONE_SUCCESS;
+ return true;
}
static void rt2800usb_txdone(struct rt2x00_dev *rt2x00dev)
@@ -549,7 +548,7 @@
struct queue_entry *entry;
u32 reg;
u8 qid;
- enum txdone_entry_desc_flags done_status;
+ bool match;
while (kfifo_get(&rt2x00dev->txstatus_fifo, ®)) {
/*
@@ -574,11 +573,8 @@
break;
}
- done_status = rt2800usb_txdone_entry_check(entry, reg);
- if (likely(done_status == TXDONE_SUCCESS))
- rt2800_txdone_entry(entry, reg, rt2800usb_get_txwi(entry));
- else
- rt2x00lib_txdone_noinfo(entry, done_status);
+ match = rt2800usb_txdone_entry_check(entry, reg);
+ rt2800_txdone_entry(entry, reg, rt2800usb_get_txwi(entry), match);
}
}
diff --git a/drivers/net/wireless/ralink/rt2x00/rt2x00.h b/drivers/net/wireless/ralink/rt2x00/rt2x00.h
index 3407878..1bc353e 100644
--- a/drivers/net/wireless/ralink/rt2x00/rt2x00.h
+++ b/drivers/net/wireless/ralink/rt2x00/rt2x00.h
@@ -174,6 +174,7 @@
#define RT5390 0x5390 /* 2.4GHz */
#define RT5392 0x5392 /* 2.4GHz */
#define RT5592 0x5592
+#define RT6352 0x6352 /* WSOC 2.4GHz */
u16 rf;
u16 rev;
@@ -718,8 +719,8 @@
CAPABILITY_DOUBLE_ANTENNA,
CAPABILITY_BT_COEXIST,
CAPABILITY_VCO_RECALIBRATION,
- CAPABILITY_INTERNAL_PA_TX0,
- CAPABILITY_INTERNAL_PA_TX1,
+ CAPABILITY_EXTERNAL_PA_TX0,
+ CAPABILITY_EXTERNAL_PA_TX1,
};
/*
@@ -1396,7 +1397,7 @@
* rt2x00debug_dump_frame - Dump a frame to userspace through debugfs.
* @rt2x00dev: Pointer to &struct rt2x00_dev.
* @type: The type of frame that is being dumped.
- * @skb: The skb containing the frame to be dumped.
+ * @entry: The queue entry containing the frame to be dumped.
*/
#ifdef CONFIG_RT2X00_LIB_DEBUGFS
void rt2x00debug_dump_frame(struct rt2x00_dev *rt2x00dev,
@@ -1425,6 +1426,8 @@
void rt2x00lib_dmadone(struct queue_entry *entry);
void rt2x00lib_txdone(struct queue_entry *entry,
struct txdone_entry_desc *txdesc);
+void rt2x00lib_txdone_nomatch(struct queue_entry *entry,
+ struct txdone_entry_desc *txdesc);
void rt2x00lib_txdone_noinfo(struct queue_entry *entry, u32 status);
void rt2x00lib_rxdone(struct queue_entry *entry, gfp_t gfp);
diff --git a/drivers/net/wireless/ralink/rt2x00/rt2x00dev.c b/drivers/net/wireless/ralink/rt2x00/rt2x00dev.c
index dd66781..357c094 100644
--- a/drivers/net/wireless/ralink/rt2x00/rt2x00dev.c
+++ b/drivers/net/wireless/ralink/rt2x00/rt2x00dev.c
@@ -313,15 +313,172 @@
return ret;
}
+static void rt2x00lib_fill_tx_status(struct rt2x00_dev *rt2x00dev,
+ struct ieee80211_tx_info *tx_info,
+ struct skb_frame_desc *skbdesc,
+ struct txdone_entry_desc *txdesc,
+ bool success)
+{
+ u8 rate_idx, rate_flags, retry_rates;
+ int i;
+
+ rate_idx = skbdesc->tx_rate_idx;
+ rate_flags = skbdesc->tx_rate_flags;
+ retry_rates = test_bit(TXDONE_FALLBACK, &txdesc->flags) ?
+ (txdesc->retry + 1) : 1;
+
+ /*
+ * Initialize TX status
+ */
+ memset(&tx_info->status, 0, sizeof(tx_info->status));
+ tx_info->status.ack_signal = 0;
+
+ /*
+ * Frame was send with retries, hardware tried
+ * different rates to send out the frame, at each
+ * retry it lowered the rate 1 step except when the
+ * lowest rate was used.
+ */
+ for (i = 0; i < retry_rates && i < IEEE80211_TX_MAX_RATES; i++) {
+ tx_info->status.rates[i].idx = rate_idx - i;
+ tx_info->status.rates[i].flags = rate_flags;
+
+ if (rate_idx - i == 0) {
+ /*
+ * The lowest rate (index 0) was used until the
+ * number of max retries was reached.
+ */
+ tx_info->status.rates[i].count = retry_rates - i;
+ i++;
+ break;
+ }
+ tx_info->status.rates[i].count = 1;
+ }
+ if (i < (IEEE80211_TX_MAX_RATES - 1))
+ tx_info->status.rates[i].idx = -1; /* terminate */
+
+ if (test_bit(TXDONE_NO_ACK_REQ, &txdesc->flags))
+ tx_info->flags |= IEEE80211_TX_CTL_NO_ACK;
+
+ if (!(tx_info->flags & IEEE80211_TX_CTL_NO_ACK)) {
+ if (success)
+ tx_info->flags |= IEEE80211_TX_STAT_ACK;
+ else
+ rt2x00dev->low_level_stats.dot11ACKFailureCount++;
+ }
+
+ /*
+ * Every single frame has it's own tx status, hence report
+ * every frame as ampdu of size 1.
+ *
+ * TODO: if we can find out how many frames were aggregated
+ * by the hw we could provide the real ampdu_len to mac80211
+ * which would allow the rc algorithm to better decide on
+ * which rates are suitable.
+ */
+ if (test_bit(TXDONE_AMPDU, &txdesc->flags) ||
+ tx_info->flags & IEEE80211_TX_CTL_AMPDU) {
+ tx_info->flags |= IEEE80211_TX_STAT_AMPDU |
+ IEEE80211_TX_CTL_AMPDU;
+ tx_info->status.ampdu_len = 1;
+ tx_info->status.ampdu_ack_len = success ? 1 : 0;
+
+ if (!success)
+ tx_info->flags |= IEEE80211_TX_STAT_AMPDU_NO_BACK;
+ }
+
+ if (rate_flags & IEEE80211_TX_RC_USE_RTS_CTS) {
+ if (success)
+ rt2x00dev->low_level_stats.dot11RTSSuccessCount++;
+ else
+ rt2x00dev->low_level_stats.dot11RTSFailureCount++;
+ }
+}
+
+static void rt2x00lib_clear_entry(struct rt2x00_dev *rt2x00dev,
+ struct queue_entry *entry)
+{
+ /*
+ * Make this entry available for reuse.
+ */
+ entry->skb = NULL;
+ entry->flags = 0;
+
+ rt2x00dev->ops->lib->clear_entry(entry);
+
+ rt2x00queue_index_inc(entry, Q_INDEX_DONE);
+
+ /*
+ * If the data queue was below the threshold before the txdone
+ * handler we must make sure the packet queue in the mac80211 stack
+ * is reenabled when the txdone handler has finished. This has to be
+ * serialized with rt2x00mac_tx(), otherwise we can wake up queue
+ * before it was stopped.
+ */
+ spin_lock_bh(&entry->queue->tx_lock);
+ if (!rt2x00queue_threshold(entry->queue))
+ rt2x00queue_unpause_queue(entry->queue);
+ spin_unlock_bh(&entry->queue->tx_lock);
+}
+
+void rt2x00lib_txdone_nomatch(struct queue_entry *entry,
+ struct txdone_entry_desc *txdesc)
+{
+ struct rt2x00_dev *rt2x00dev = entry->queue->rt2x00dev;
+ struct skb_frame_desc *skbdesc = get_skb_frame_desc(entry->skb);
+ struct ieee80211_tx_info txinfo = {};
+ bool success;
+
+ /*
+ * Unmap the skb.
+ */
+ rt2x00queue_unmap_skb(entry);
+
+ /*
+ * Signal that the TX descriptor is no longer in the skb.
+ */
+ skbdesc->flags &= ~SKBDESC_DESC_IN_SKB;
+
+ /*
+ * Send frame to debugfs immediately, after this call is completed
+ * we are going to overwrite the skb->cb array.
+ */
+ rt2x00debug_dump_frame(rt2x00dev, DUMP_FRAME_TXDONE, entry);
+
+ /*
+ * Determine if the frame has been successfully transmitted and
+ * remove BARs from our check list while checking for their
+ * TX status.
+ */
+ success =
+ rt2x00lib_txdone_bar_status(entry) ||
+ test_bit(TXDONE_SUCCESS, &txdesc->flags);
+
+ if (!test_bit(TXDONE_UNKNOWN, &txdesc->flags)) {
+ /*
+ * Update TX statistics.
+ */
+ rt2x00dev->link.qual.tx_success += success;
+ rt2x00dev->link.qual.tx_failed += !success;
+
+ rt2x00lib_fill_tx_status(rt2x00dev, &txinfo, skbdesc, txdesc,
+ success);
+ ieee80211_tx_status_noskb(rt2x00dev->hw, skbdesc->sta, &txinfo);
+ }
+
+ dev_kfree_skb_any(entry->skb);
+ rt2x00lib_clear_entry(rt2x00dev, entry);
+}
+EXPORT_SYMBOL_GPL(rt2x00lib_txdone_nomatch);
+
void rt2x00lib_txdone(struct queue_entry *entry,
struct txdone_entry_desc *txdesc)
{
struct rt2x00_dev *rt2x00dev = entry->queue->rt2x00dev;
struct ieee80211_tx_info *tx_info = IEEE80211_SKB_CB(entry->skb);
struct skb_frame_desc *skbdesc = get_skb_frame_desc(entry->skb);
- unsigned int header_length, i;
- u8 rate_idx, rate_flags, retry_rates;
u8 skbdesc_flags = skbdesc->flags;
+ unsigned int header_length;
bool success;
/*
@@ -381,73 +538,7 @@
rt2x00dev->link.qual.tx_success += success;
rt2x00dev->link.qual.tx_failed += !success;
- rate_idx = skbdesc->tx_rate_idx;
- rate_flags = skbdesc->tx_rate_flags;
- retry_rates = test_bit(TXDONE_FALLBACK, &txdesc->flags) ?
- (txdesc->retry + 1) : 1;
-
- /*
- * Initialize TX status
- */
- memset(&tx_info->status, 0, sizeof(tx_info->status));
- tx_info->status.ack_signal = 0;
-
- /*
- * Frame was send with retries, hardware tried
- * different rates to send out the frame, at each
- * retry it lowered the rate 1 step except when the
- * lowest rate was used.
- */
- for (i = 0; i < retry_rates && i < IEEE80211_TX_MAX_RATES; i++) {
- tx_info->status.rates[i].idx = rate_idx - i;
- tx_info->status.rates[i].flags = rate_flags;
-
- if (rate_idx - i == 0) {
- /*
- * The lowest rate (index 0) was used until the
- * number of max retries was reached.
- */
- tx_info->status.rates[i].count = retry_rates - i;
- i++;
- break;
- }
- tx_info->status.rates[i].count = 1;
- }
- if (i < (IEEE80211_TX_MAX_RATES - 1))
- tx_info->status.rates[i].idx = -1; /* terminate */
-
- if (!(tx_info->flags & IEEE80211_TX_CTL_NO_ACK)) {
- if (success)
- tx_info->flags |= IEEE80211_TX_STAT_ACK;
- else
- rt2x00dev->low_level_stats.dot11ACKFailureCount++;
- }
-
- /*
- * Every single frame has it's own tx status, hence report
- * every frame as ampdu of size 1.
- *
- * TODO: if we can find out how many frames were aggregated
- * by the hw we could provide the real ampdu_len to mac80211
- * which would allow the rc algorithm to better decide on
- * which rates are suitable.
- */
- if (test_bit(TXDONE_AMPDU, &txdesc->flags) ||
- tx_info->flags & IEEE80211_TX_CTL_AMPDU) {
- tx_info->flags |= IEEE80211_TX_STAT_AMPDU;
- tx_info->status.ampdu_len = 1;
- tx_info->status.ampdu_ack_len = success ? 1 : 0;
-
- if (!success)
- tx_info->flags |= IEEE80211_TX_STAT_AMPDU_NO_BACK;
- }
-
- if (rate_flags & IEEE80211_TX_RC_USE_RTS_CTS) {
- if (success)
- rt2x00dev->low_level_stats.dot11RTSSuccessCount++;
- else
- rt2x00dev->low_level_stats.dot11RTSFailureCount++;
- }
+ rt2x00lib_fill_tx_status(rt2x00dev, tx_info, skbdesc, txdesc, success);
/*
* Only send the status report to mac80211 when it's a frame
@@ -460,30 +551,11 @@
ieee80211_tx_status(rt2x00dev->hw, entry->skb);
else
ieee80211_tx_status_ni(rt2x00dev->hw, entry->skb);
- } else
+ } else {
dev_kfree_skb_any(entry->skb);
+ }
- /*
- * Make this entry available for reuse.
- */
- entry->skb = NULL;
- entry->flags = 0;
-
- rt2x00dev->ops->lib->clear_entry(entry);
-
- rt2x00queue_index_inc(entry, Q_INDEX_DONE);
-
- /*
- * If the data queue was below the threshold before the txdone
- * handler we must make sure the packet queue in the mac80211 stack
- * is reenabled when the txdone handler has finished. This has to be
- * serialized with rt2x00mac_tx(), otherwise we can wake up queue
- * before it was stopped.
- */
- spin_lock_bh(&entry->queue->tx_lock);
- if (!rt2x00queue_threshold(entry->queue))
- rt2x00queue_unpause_queue(entry->queue);
- spin_unlock_bh(&entry->queue->tx_lock);
+ rt2x00lib_clear_entry(rt2x00dev, entry);
}
EXPORT_SYMBOL_GPL(rt2x00lib_txdone);
@@ -753,7 +825,7 @@
rate_idx = rt2x00lib_rxdone_read_signal(rt2x00dev, &rxdesc);
if (rxdesc.rate_mode == RATE_MODE_HT_MIX ||
rxdesc.rate_mode == RATE_MODE_HT_GREENFIELD)
- rxdesc.flags |= RX_FLAG_HT;
+ rxdesc.encoding = RX_ENC_HT;
/*
* Check if this is a beacon, and more frames have been
@@ -793,6 +865,9 @@
rx_status->rate_idx = rate_idx;
rx_status->signal = rxdesc.rssi;
rx_status->flag = rxdesc.flags;
+ rx_status->enc_flags = rxdesc.enc_flags;
+ rx_status->encoding = rxdesc.encoding;
+ rx_status->bw = rxdesc.bw;
rx_status->antenna = rt2x00dev->link.ant.active.rx;
ieee80211_rx_ni(rt2x00dev->hw, entry->skb);
@@ -1384,6 +1459,9 @@
rt2x00dev->hw->wiphy->flags |= WIPHY_FLAG_IBSS_RSN;
+ wiphy_ext_feature_set(rt2x00dev->hw->wiphy,
+ NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
/*
* Initialize ieee80211 structure.
*/
diff --git a/drivers/net/wireless/ralink/rt2x00/rt2x00queue.c b/drivers/net/wireless/ralink/rt2x00/rt2x00queue.c
index e1660b9..a2c1ca5 100644
--- a/drivers/net/wireless/ralink/rt2x00/rt2x00queue.c
+++ b/drivers/net/wireless/ralink/rt2x00/rt2x00queue.c
@@ -372,15 +372,16 @@
/*
* Determine IFS values
- * - Use TXOP_BACKOFF for management frames except beacons
+ * - Use TXOP_BACKOFF for probe and management frames except beacons
* - Use TXOP_SIFS for fragment bursts
* - Use TXOP_HTTXOP for everything else
*
* Note: rt2800 devices won't use CTS protection (if used)
* for frames not transmitted with TXOP_HTTXOP
*/
- if (ieee80211_is_mgmt(hdr->frame_control) &&
- !ieee80211_is_beacon(hdr->frame_control))
+ if ((ieee80211_is_mgmt(hdr->frame_control) &&
+ !ieee80211_is_beacon(hdr->frame_control)) ||
+ (tx_info->flags & IEEE80211_TX_CTL_RATE_CTRL_PROBE))
txdesc->u.ht.txop = TXOP_BACKOFF;
else if (!(tx_info->flags & IEEE80211_TX_CTL_FIRST_FRAGMENT))
txdesc->u.ht.txop = TXOP_SIFS;
diff --git a/drivers/net/wireless/ralink/rt2x00/rt2x00queue.h b/drivers/net/wireless/ralink/rt2x00/rt2x00queue.h
index 22d1881..6055f36 100644
--- a/drivers/net/wireless/ralink/rt2x00/rt2x00queue.h
+++ b/drivers/net/wireless/ralink/rt2x00/rt2x00queue.h
@@ -102,7 +102,7 @@
* of the scope of the skb->data pointer.
* @iv: IV/EIV data used during encryption/decryption.
* @skb_dma: (PCI-only) the DMA address associated with the sk buffer.
- * @entry: The entry to which this sk buffer belongs.
+ * @sta: The station where sk buffer was sent.
*/
struct skb_frame_desc {
u8 flags;
@@ -116,6 +116,7 @@
__le32 iv[2];
dma_addr_t skb_dma;
+ struct ieee80211_sta *sta;
};
/**
@@ -183,6 +184,9 @@
int flags;
int dev_flags;
u16 rate_mode;
+ u16 enc_flags;
+ enum mac80211_rx_encoding encoding;
+ enum rate_info_bw bw;
u8 cipher;
u8 cipher_status;
@@ -214,6 +218,7 @@
TXDONE_FAILURE,
TXDONE_EXCESSIVE_RETRY,
TXDONE_AMPDU,
+ TXDONE_NO_ACK_REQ,
};
/**
diff --git a/drivers/net/wireless/realtek/rtl818x/rtl8180/dev.c b/drivers/net/wireless/realtek/rtl818x/rtl8180/dev.c
index e895a84..225c1c8 100644
--- a/drivers/net/wireless/realtek/rtl818x/rtl8180/dev.c
+++ b/drivers/net/wireless/realtek/rtl818x/rtl8180/dev.c
@@ -315,7 +315,7 @@
rx_status.mactime = tsft;
rx_status.flag |= RX_FLAG_MACTIME_START;
if (flags & RTL818X_RX_DESC_FLAG_SPLCP)
- rx_status.flag |= RX_FLAG_SHORTPRE;
+ rx_status.enc_flags |= RX_ENC_FLAG_SHORTPRE;
if (flags & RTL818X_RX_DESC_FLAG_CRC32_ERR)
rx_status.flag |= RX_FLAG_FAILED_FCS_CRC;
@@ -1877,6 +1877,8 @@
else
ieee80211_hw_set(dev, SIGNAL_UNSPEC);
+ wiphy_ext_feature_set(dev->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
rtl8180_eeprom_read(priv);
switch (priv->rf_type) {
diff --git a/drivers/net/wireless/realtek/rtl818x/rtl8187/dev.c b/drivers/net/wireless/realtek/rtl818x/rtl8187/dev.c
index 231f84d..35fe991 100644
--- a/drivers/net/wireless/realtek/rtl818x/rtl8187/dev.c
+++ b/drivers/net/wireless/realtek/rtl818x/rtl8187/dev.c
@@ -389,7 +389,7 @@
rx_status.band = dev->conf.chandef.chan->band;
rx_status.flag |= RX_FLAG_MACTIME_START;
if (flags & RTL818X_RX_DESC_FLAG_SPLCP)
- rx_status.flag |= RX_FLAG_SHORTPRE;
+ rx_status.enc_flags |= RX_ENC_FLAG_SHORTPRE;
if (flags & RTL818X_RX_DESC_FLAG_CRC32_ERR)
rx_status.flag |= RX_FLAG_FAILED_FCS_CRC;
memcpy(IEEE80211_SKB_RXCB(skb), &rx_status, sizeof(rx_status));
@@ -946,8 +946,7 @@
(7 << 13 /* RX FIFO threshold NONE */) |
(7 << 10 /* MAX RX DMA */) |
RTL818X_RX_CONF_RX_AUTORESETPHY |
- RTL818X_RX_CONF_ONLYERLPKT |
- RTL818X_RX_CONF_MULTICAST;
+ RTL818X_RX_CONF_ONLYERLPKT;
priv->rx_conf = reg;
rtl818x_iowrite32(priv, &priv->map->RX_CONF, reg);
@@ -1319,12 +1318,11 @@
priv->rx_conf ^= RTL818X_RX_CONF_FCS;
if (changed_flags & FIF_CONTROL)
priv->rx_conf ^= RTL818X_RX_CONF_CTRL;
- if (changed_flags & FIF_OTHER_BSS)
- priv->rx_conf ^= RTL818X_RX_CONF_MONITOR;
- if (*total_flags & FIF_ALLMULTI || multicast > 0)
- priv->rx_conf |= RTL818X_RX_CONF_MULTICAST;
+ if (*total_flags & FIF_OTHER_BSS ||
+ *total_flags & FIF_ALLMULTI || multicast > 0)
+ priv->rx_conf |= RTL818X_RX_CONF_MONITOR;
else
- priv->rx_conf &= ~RTL818X_RX_CONF_MULTICAST;
+ priv->rx_conf &= ~RTL818X_RX_CONF_MONITOR;
*total_flags = 0;
@@ -1332,10 +1330,10 @@
*total_flags |= FIF_FCSFAIL;
if (priv->rx_conf & RTL818X_RX_CONF_CTRL)
*total_flags |= FIF_CONTROL;
- if (priv->rx_conf & RTL818X_RX_CONF_MONITOR)
+ if (priv->rx_conf & RTL818X_RX_CONF_MONITOR) {
*total_flags |= FIF_OTHER_BSS;
- if (priv->rx_conf & RTL818X_RX_CONF_MULTICAST)
*total_flags |= FIF_ALLMULTI;
+ }
rtl818x_iowrite32_async(priv, &priv->map->RX_CONF, priv->rx_conf);
}
@@ -1609,6 +1607,8 @@
dev->wiphy->interface_modes = BIT(NL80211_IFTYPE_STATION) |
BIT(NL80211_IFTYPE_ADHOC) ;
+ wiphy_ext_feature_set(dev->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
if ((id->driver_info == DEVICE_RTL8187) && priv->is_rtl8187b)
printk(KERN_INFO "rtl8187: inconsistency between id with OEM"
" info!\n");
diff --git a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c
index e544dd1..39d5631 100644
--- a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c
+++ b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c
@@ -5041,7 +5041,7 @@
u32 rxmcs)
{
if (phy_stats->sgi_en)
- rx_status->flag |= RX_FLAG_SHORT_GI;
+ rx_status->enc_flags |= RX_ENC_FLAG_SHORT_GI;
if (rxmcs < DESC_RATE_6M) {
/*
@@ -5267,10 +5267,10 @@
if (rx_desc->crc32)
rx_status->flag |= RX_FLAG_FAILED_FCS_CRC;
if (rx_desc->bw)
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
if (rx_desc->rxht) {
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
rx_status->rate_idx = rx_desc->rxmcs - DESC_RATE_MCS0;
} else {
rx_status->rate_idx = rx_desc->rxmcs;
@@ -5337,10 +5337,10 @@
if (rx_desc->crc32)
rx_status->flag |= RX_FLAG_FAILED_FCS_CRC;
if (rx_desc->bw)
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
if (rx_desc->rxmcs >= DESC_RATE_MCS0) {
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
rx_status->rate_idx = rx_desc->rxmcs - DESC_RATE_MCS0;
} else {
rx_status->rate_idx = rx_desc->rxmcs;
@@ -6135,6 +6135,8 @@
ieee80211_hw_set(hw, HAS_RATE_CONTROL);
ieee80211_hw_set(hw, AMPDU_AGGREGATION);
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
ret = ieee80211_register_hw(priv->hw);
if (ret) {
dev_err(&udev->dev, "%s: Failed to register: %i\n",
diff --git a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8192e2ant.c b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8192e2ant.c
index ffa1f43..57e633d 100644
--- a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8192e2ant.c
+++ b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8192e2ant.c
@@ -44,7 +44,7 @@
static struct coex_sta_8192e_2ant glcoex_sta_8192e_2ant;
static struct coex_sta_8192e_2ant *coex_sta = &glcoex_sta_8192e_2ant;
-static const char *const GLBtInfoSrc8192e2Ant[] = {
+static const char *const glbt_info_src_8192e_2ant[] = {
"BT Info[wifi fw]",
"BT Info[bt rsp]",
"BT Info[bt auto report]",
@@ -57,31 +57,31 @@
* local function proto type if needed
**************************************************************/
/**************************************************************
- * local function start with halbtc8192e2ant_
+ * local function start with btc8192e2ant_
**************************************************************/
-static u8 halbtc8192e2ant_btrssi_state(struct btc_coexist *btcoexist,
- u8 level_num, u8 rssi_thresh,
- u8 rssi_thresh1)
+static u8 btc8192e2ant_bt_rssi_state(struct btc_coexist *btcoexist,
+ u8 level_num, u8 rssi_thresh,
+ u8 rssi_thresh1)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- int btrssi = 0;
- u8 btrssi_state = coex_sta->pre_bt_rssi_state;
+ int bt_rssi = 0;
+ u8 bt_rssi_state = coex_sta->pre_bt_rssi_state;
- btrssi = coex_sta->bt_rssi;
+ bt_rssi = coex_sta->bt_rssi;
if (level_num == 2) {
if ((coex_sta->pre_bt_rssi_state == BTC_RSSI_STATE_LOW) ||
(coex_sta->pre_bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- if (btrssi >=
+ if (bt_rssi >=
(rssi_thresh + BTC_RSSI_COEX_THRESH_TOL_8192E_2ANT))
- btrssi_state = BTC_RSSI_STATE_HIGH;
+ bt_rssi_state = BTC_RSSI_STATE_HIGH;
else
- btrssi_state = BTC_RSSI_STATE_STAY_LOW;
+ bt_rssi_state = BTC_RSSI_STATE_STAY_LOW;
} else {
- if (btrssi < rssi_thresh)
- btrssi_state = BTC_RSSI_STATE_LOW;
+ if (bt_rssi < rssi_thresh)
+ bt_rssi_state = BTC_RSSI_STATE_LOW;
else
- btrssi_state = BTC_RSSI_STATE_STAY_HIGH;
+ bt_rssi_state = BTC_RSSI_STATE_STAY_HIGH;
}
} else if (level_num == 3) {
if (rssi_thresh > rssi_thresh1) {
@@ -89,62 +89,63 @@
"[BTCoex], BT Rssi thresh error!!\n");
return coex_sta->pre_bt_rssi_state;
}
+
if ((coex_sta->pre_bt_rssi_state == BTC_RSSI_STATE_LOW) ||
(coex_sta->pre_bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- if (btrssi >=
+ if (bt_rssi >=
(rssi_thresh + BTC_RSSI_COEX_THRESH_TOL_8192E_2ANT))
- btrssi_state = BTC_RSSI_STATE_MEDIUM;
+ bt_rssi_state = BTC_RSSI_STATE_MEDIUM;
else
- btrssi_state = BTC_RSSI_STATE_STAY_LOW;
+ bt_rssi_state = BTC_RSSI_STATE_STAY_LOW;
} else if ((coex_sta->pre_bt_rssi_state ==
BTC_RSSI_STATE_MEDIUM) ||
(coex_sta->pre_bt_rssi_state ==
BTC_RSSI_STATE_STAY_MEDIUM)) {
- if (btrssi >= (rssi_thresh1 +
+ if (bt_rssi >= (rssi_thresh1 +
BTC_RSSI_COEX_THRESH_TOL_8192E_2ANT))
- btrssi_state = BTC_RSSI_STATE_HIGH;
- else if (btrssi < rssi_thresh)
- btrssi_state = BTC_RSSI_STATE_LOW;
+ bt_rssi_state = BTC_RSSI_STATE_HIGH;
+ else if (bt_rssi < rssi_thresh)
+ bt_rssi_state = BTC_RSSI_STATE_LOW;
else
- btrssi_state = BTC_RSSI_STATE_STAY_MEDIUM;
+ bt_rssi_state = BTC_RSSI_STATE_STAY_MEDIUM;
} else {
- if (btrssi < rssi_thresh1)
- btrssi_state = BTC_RSSI_STATE_MEDIUM;
+ if (bt_rssi < rssi_thresh1)
+ bt_rssi_state = BTC_RSSI_STATE_MEDIUM;
else
- btrssi_state = BTC_RSSI_STATE_STAY_HIGH;
+ bt_rssi_state = BTC_RSSI_STATE_STAY_HIGH;
}
}
- coex_sta->pre_bt_rssi_state = btrssi_state;
+ coex_sta->pre_bt_rssi_state = bt_rssi_state;
- return btrssi_state;
+ return bt_rssi_state;
}
-static u8 halbtc8192e2ant_wifirssi_state(struct btc_coexist *btcoexist,
- u8 index, u8 level_num, u8 rssi_thresh,
- u8 rssi_thresh1)
+static u8 btc8192e2ant_wifi_rssi_state(struct btc_coexist *btcoexist,
+ u8 index, u8 level_num, u8 rssi_thresh,
+ u8 rssi_thresh1)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- int wifirssi = 0;
- u8 wifirssi_state = coex_sta->pre_wifi_rssi_state[index];
+ int wifi_rssi = 0;
+ u8 wifi_rssi_state = coex_sta->pre_wifi_rssi_state[index];
- btcoexist->btc_get(btcoexist, BTC_GET_S4_WIFI_RSSI, &wifirssi);
+ btcoexist->btc_get(btcoexist, BTC_GET_S4_WIFI_RSSI, &wifi_rssi);
if (level_num == 2) {
if ((coex_sta->pre_wifi_rssi_state[index] ==
BTC_RSSI_STATE_LOW) ||
(coex_sta->pre_wifi_rssi_state[index] ==
BTC_RSSI_STATE_STAY_LOW)) {
- if (wifirssi >= (rssi_thresh +
- BTC_RSSI_COEX_THRESH_TOL_8192E_2ANT))
- wifirssi_state = BTC_RSSI_STATE_HIGH;
+ if (wifi_rssi >=
+ (rssi_thresh + BTC_RSSI_COEX_THRESH_TOL_8192E_2ANT))
+ wifi_rssi_state = BTC_RSSI_STATE_HIGH;
else
- wifirssi_state = BTC_RSSI_STATE_STAY_LOW;
+ wifi_rssi_state = BTC_RSSI_STATE_STAY_LOW;
} else {
- if (wifirssi < rssi_thresh)
- wifirssi_state = BTC_RSSI_STATE_LOW;
+ if (wifi_rssi < rssi_thresh)
+ wifi_rssi_state = BTC_RSSI_STATE_LOW;
else
- wifirssi_state = BTC_RSSI_STATE_STAY_HIGH;
+ wifi_rssi_state = BTC_RSSI_STATE_STAY_HIGH;
}
} else if (level_num == 3) {
if (rssi_thresh > rssi_thresh1) {
@@ -157,36 +158,37 @@
BTC_RSSI_STATE_LOW) ||
(coex_sta->pre_wifi_rssi_state[index] ==
BTC_RSSI_STATE_STAY_LOW)) {
- if (wifirssi >= (rssi_thresh +
- BTC_RSSI_COEX_THRESH_TOL_8192E_2ANT))
- wifirssi_state = BTC_RSSI_STATE_MEDIUM;
+ if (wifi_rssi >=
+ (rssi_thresh + BTC_RSSI_COEX_THRESH_TOL_8192E_2ANT))
+ wifi_rssi_state = BTC_RSSI_STATE_MEDIUM;
else
- wifirssi_state = BTC_RSSI_STATE_STAY_LOW;
+ wifi_rssi_state = BTC_RSSI_STATE_STAY_LOW;
} else if ((coex_sta->pre_wifi_rssi_state[index] ==
BTC_RSSI_STATE_MEDIUM) ||
(coex_sta->pre_wifi_rssi_state[index] ==
BTC_RSSI_STATE_STAY_MEDIUM)) {
- if (wifirssi >= (rssi_thresh1 +
+ if (wifi_rssi >= (rssi_thresh1 +
BTC_RSSI_COEX_THRESH_TOL_8192E_2ANT))
- wifirssi_state = BTC_RSSI_STATE_HIGH;
- else if (wifirssi < rssi_thresh)
- wifirssi_state = BTC_RSSI_STATE_LOW;
+ wifi_rssi_state = BTC_RSSI_STATE_HIGH;
+ else if (wifi_rssi < rssi_thresh)
+ wifi_rssi_state = BTC_RSSI_STATE_LOW;
else
- wifirssi_state = BTC_RSSI_STATE_STAY_MEDIUM;
+ wifi_rssi_state = BTC_RSSI_STATE_STAY_MEDIUM;
} else {
- if (wifirssi < rssi_thresh1)
- wifirssi_state = BTC_RSSI_STATE_MEDIUM;
+ if (wifi_rssi < rssi_thresh1)
+ wifi_rssi_state = BTC_RSSI_STATE_MEDIUM;
else
- wifirssi_state = BTC_RSSI_STATE_STAY_HIGH;
+ wifi_rssi_state = BTC_RSSI_STATE_STAY_HIGH;
}
}
- coex_sta->pre_wifi_rssi_state[index] = wifirssi_state;
+ coex_sta->pre_wifi_rssi_state[index] = wifi_rssi_state;
- return wifirssi_state;
+ return wifi_rssi_state;
}
-static void btc8192e2ant_monitor_bt_enable_dis(struct btc_coexist *btcoexist)
+static void btc8192e2ant_monitor_bt_enable_disable(struct btc_coexist
+ *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
static bool pre_bt_disabled;
@@ -236,57 +238,57 @@
}
}
-static u32 halbtc8192e2ant_decidera_mask(struct btc_coexist *btcoexist,
- u8 sstype, u32 ra_masktype)
+static u32 btc8192e2ant_decide_ra_mask(struct btc_coexist *btcoexist,
+ u8 ss_type, u32 ra_mask_type)
{
- u32 disra_mask = 0x0;
+ u32 dis_ra_mask = 0x0;
- switch (ra_masktype) {
+ switch (ra_mask_type) {
case 0: /* normal mode */
- if (sstype == 2)
- disra_mask = 0x0; /* enable 2ss */
+ if (ss_type == 2)
+ dis_ra_mask = 0x0; /* enable 2ss */
else
- disra_mask = 0xfff00000;/* disable 2ss */
+ dis_ra_mask = 0xfff00000; /* disable 2ss */
break;
case 1: /* disable cck 1/2 */
- if (sstype == 2)
- disra_mask = 0x00000003;/* enable 2ss */
+ if (ss_type == 2)
+ dis_ra_mask = 0x00000003; /* enable 2ss */
else
- disra_mask = 0xfff00003;/* disable 2ss */
+ dis_ra_mask = 0xfff00003; /* disable 2ss */
break;
case 2: /* disable cck 1/2/5.5, ofdm 6/9/12/18/24, mcs 0/1/2/3/4 */
- if (sstype == 2)
- disra_mask = 0x0001f1f7;/* enable 2ss */
+ if (ss_type == 2)
+ dis_ra_mask = 0x0001f1f7; /* enable 2ss */
else
- disra_mask = 0xfff1f1f7;/* disable 2ss */
+ dis_ra_mask = 0xfff1f1f7; /* disable 2ss */
break;
default:
break;
}
- return disra_mask;
+ return dis_ra_mask;
}
-static void halbtc8192e2ant_Updatera_mask(struct btc_coexist *btcoexist,
- bool force_exec, u32 dis_ratemask)
+static void btc8192e2ant_update_ra_mask(struct btc_coexist *btcoexist,
+ bool force_exec, u32 dis_rate_mask)
{
- coex_dm->curra_mask = dis_ratemask;
+ coex_dm->cur_ra_mask = dis_rate_mask;
- if (force_exec || (coex_dm->prera_mask != coex_dm->curra_mask))
- btcoexist->btc_set(btcoexist, BTC_SET_ACT_UPDATE_ra_mask,
- &coex_dm->curra_mask);
- coex_dm->prera_mask = coex_dm->curra_mask;
+ if (force_exec || (coex_dm->pre_ra_mask != coex_dm->cur_ra_mask))
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_UPDATE_RAMASK,
+ &coex_dm->cur_ra_mask);
+ coex_dm->pre_ra_mask = coex_dm->cur_ra_mask;
}
-static void btc8192e2ant_autorate_fallback_retry(struct btc_coexist *btcoexist,
- bool force_exec, u8 type)
+static void btc8192e2ant_auto_rate_fallback_retry(struct btc_coexist *btcoexist,
+ bool force_exec, u8 type)
{
- bool wifi_under_bmode = false;
+ bool wifi_under_b_mode = false;
- coex_dm->cur_arfrtype = type;
+ coex_dm->cur_arfr_type = type;
- if (force_exec || (coex_dm->pre_arfrtype != coex_dm->cur_arfrtype)) {
- switch (coex_dm->cur_arfrtype) {
+ if (force_exec || (coex_dm->pre_arfr_type != coex_dm->cur_arfr_type)) {
+ switch (coex_dm->cur_arfr_type) {
case 0: /* normal mode */
btcoexist->btc_write_4byte(btcoexist, 0x430,
coex_dm->backup_arfr_cnt1);
@@ -296,8 +298,8 @@
case 1:
btcoexist->btc_get(btcoexist,
BTC_GET_BL_WIFI_UNDER_B_MODE,
- &wifi_under_bmode);
- if (wifi_under_bmode) {
+ &wifi_under_b_mode);
+ if (wifi_under_b_mode) {
btcoexist->btc_write_4byte(btcoexist, 0x430,
0x0);
btcoexist->btc_write_4byte(btcoexist, 0x434,
@@ -314,46 +316,45 @@
}
}
- coex_dm->pre_arfrtype = coex_dm->cur_arfrtype;
+ coex_dm->pre_arfr_type = coex_dm->cur_arfr_type;
}
-static void halbtc8192e2ant_retrylimit(struct btc_coexist *btcoexist,
- bool force_exec, u8 type)
+static void btc8192e2ant_retry_limit(struct btc_coexist *btcoexist,
+ bool force_exec, u8 type)
{
- coex_dm->cur_retrylimit_type = type;
+ coex_dm->cur_retry_limit_type = type;
- if (force_exec || (coex_dm->pre_retrylimit_type !=
- coex_dm->cur_retrylimit_type)) {
- switch (coex_dm->cur_retrylimit_type) {
+ if (force_exec || (coex_dm->pre_retry_limit_type !=
+ coex_dm->cur_retry_limit_type)) {
+ switch (coex_dm->cur_retry_limit_type) {
case 0: /* normal mode */
- btcoexist->btc_write_2byte(btcoexist, 0x42a,
- coex_dm->backup_retrylimit);
- break;
+ btcoexist->btc_write_2byte(btcoexist, 0x42a,
+ coex_dm->backup_retry_limit);
+ break;
case 1: /* retry limit = 8 */
- btcoexist->btc_write_2byte(btcoexist, 0x42a,
- 0x0808);
- break;
+ btcoexist->btc_write_2byte(btcoexist, 0x42a, 0x0808);
+ break;
default:
- break;
+ break;
}
}
- coex_dm->pre_retrylimit_type = coex_dm->cur_retrylimit_type;
+ coex_dm->pre_retry_limit_type = coex_dm->cur_retry_limit_type;
}
-static void halbtc8192e2ant_ampdu_maxtime(struct btc_coexist *btcoexist,
- bool force_exec, u8 type)
+static void btc8192e2ant_ampdu_maxtime(struct btc_coexist *btcoexist,
+ bool force_exec, u8 type)
{
- coex_dm->cur_ampdutime_type = type;
+ coex_dm->cur_ampdu_time_type = type;
- if (force_exec || (coex_dm->pre_ampdutime_type !=
- coex_dm->cur_ampdutime_type)) {
- switch (coex_dm->cur_ampdutime_type) {
+ if (force_exec || (coex_dm->pre_ampdu_time_type !=
+ coex_dm->cur_ampdu_time_type)) {
+ switch (coex_dm->cur_ampdu_time_type) {
case 0: /* normal mode */
btcoexist->btc_write_1byte(btcoexist, 0x456,
coex_dm->backup_ampdu_maxtime);
break;
- case 1: /* AMPDU timw = 0x38 * 32us */
+ case 1: /* AMPDU time = 0x38 * 32us */
btcoexist->btc_write_1byte(btcoexist, 0x456, 0x38);
break;
default:
@@ -361,30 +362,30 @@
}
}
- coex_dm->pre_ampdutime_type = coex_dm->cur_ampdutime_type;
+ coex_dm->pre_ampdu_time_type = coex_dm->cur_ampdu_time_type;
}
-static void halbtc8192e2ant_limited_tx(struct btc_coexist *btcoexist,
- bool force_exec, u8 ra_masktype,
- u8 arfr_type, u8 retrylimit_type,
- u8 ampdutime_type)
+static void btc8192e2ant_limited_tx(struct btc_coexist *btcoexist,
+ bool force_exec, u8 ra_mask_type,
+ u8 arfr_type, u8 retry_limit_type,
+ u8 ampdu_time_type)
{
- u32 disra_mask = 0x0;
+ u32 dis_ra_mask = 0x0;
- coex_dm->curra_masktype = ra_masktype;
- disra_mask = halbtc8192e2ant_decidera_mask(btcoexist,
- coex_dm->cur_sstype,
- ra_masktype);
- halbtc8192e2ant_Updatera_mask(btcoexist, force_exec, disra_mask);
-btc8192e2ant_autorate_fallback_retry(btcoexist, force_exec, arfr_type);
- halbtc8192e2ant_retrylimit(btcoexist, force_exec, retrylimit_type);
- halbtc8192e2ant_ampdu_maxtime(btcoexist, force_exec, ampdutime_type);
+ coex_dm->cur_ra_mask_type = ra_mask_type;
+ dis_ra_mask =
+ btc8192e2ant_decide_ra_mask(btcoexist, coex_dm->cur_ss_type,
+ ra_mask_type);
+ btc8192e2ant_update_ra_mask(btcoexist, force_exec, dis_ra_mask);
+ btc8192e2ant_auto_rate_fallback_retry(btcoexist, force_exec, arfr_type);
+ btc8192e2ant_retry_limit(btcoexist, force_exec, retry_limit_type);
+ btc8192e2ant_ampdu_maxtime(btcoexist, force_exec, ampdu_time_type);
}
-static void halbtc8192e2ant_limited_rx(struct btc_coexist *btcoexist,
- bool force_exec, bool rej_ap_agg_pkt,
- bool bt_ctrl_agg_buf_size,
- u8 agg_buf_size)
+static void btc8192e2ant_limited_rx(struct btc_coexist *btcoexist,
+ bool force_exec, bool rej_ap_agg_pkt,
+ bool bt_ctrl_agg_buf_size,
+ u8 agg_buf_size)
{
bool reject_rx_agg = rej_ap_agg_pkt;
bool bt_ctrl_rx_agg_size = bt_ctrl_agg_buf_size;
@@ -406,7 +407,7 @@
btcoexist->btc_set(btcoexist, BTC_SET_ACT_AGGREGATE_CTRL, NULL);
}
-static void halbtc8192e2ant_monitor_bt_ctr(struct btc_coexist *btcoexist)
+static void btc8192e2ant_monitor_bt_ctr(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u32 reg_hp_txrx, reg_lp_txrx, u32tmp;
@@ -417,11 +418,11 @@
u32tmp = btcoexist->btc_read_4byte(btcoexist, reg_hp_txrx);
reg_hp_tx = u32tmp & MASKLWORD;
- reg_hp_rx = (u32tmp & MASKHWORD)>>16;
+ reg_hp_rx = (u32tmp & MASKHWORD) >> 16;
u32tmp = btcoexist->btc_read_4byte(btcoexist, reg_lp_txrx);
reg_lp_tx = u32tmp & MASKLWORD;
- reg_lp_rx = (u32tmp & MASKHWORD)>>16;
+ reg_lp_rx = (u32tmp & MASKHWORD) >> 16;
coex_sta->high_priority_tx = reg_hp_tx;
coex_sta->high_priority_rx = reg_hp_rx;
@@ -439,14 +440,14 @@
btcoexist->btc_write_1byte(btcoexist, 0x76e, 0xc);
}
-static void halbtc8192e2ant_querybt_info(struct btc_coexist *btcoexist)
+static void btc8192e2ant_query_bt_info(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[1] = {0};
coex_sta->c2h_bt_info_req_sent = true;
- h2c_parameter[0] |= BIT0; /* trigger */
+ h2c_parameter[0] |= BIT0; /* trigger */
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Query Bt Info, FW write 0x61 = 0x%x\n",
@@ -455,12 +456,12 @@
btcoexist->btc_fill_h2c(btcoexist, 0x61, 1, h2c_parameter);
}
-static void halbtc8192e2ant_update_btlink_info(struct btc_coexist *btcoexist)
+static void btc8192e2ant_update_bt_link_info(struct btc_coexist *btcoexist)
{
struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
- bool bt_hson = false;
+ bool bt_hs_on = false;
- btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hson);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
bt_link_info->bt_link_exist = coex_sta->bt_link_exist;
bt_link_info->sco_exist = coex_sta->sco_exist;
@@ -469,7 +470,7 @@
bt_link_info->hid_exist = coex_sta->hid_exist;
/* work around for HS mode. */
- if (bt_hson) {
+ if (bt_hs_on) {
bt_link_info->pan_exist = true;
bt_link_info->bt_link_exist = true;
}
@@ -511,16 +512,16 @@
bt_link_info->hid_only = false;
}
-static u8 halbtc8192e2ant_action_algorithm(struct btc_coexist *btcoexist)
+static u8 btc8192e2ant_action_algorithm(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
struct btc_stack_info *stack_info = &btcoexist->stack_info;
- bool bt_hson = false;
+ bool bt_hs_on = false;
u8 algorithm = BT_8192E_2ANT_COEX_ALGO_UNDEFINED;
- u8 numdiffprofile = 0;
+ u8 num_of_diff_profile = 0;
- btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hson);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
if (!bt_link_info->bt_link_exist) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -529,15 +530,15 @@
}
if (bt_link_info->sco_exist)
- numdiffprofile++;
+ num_of_diff_profile++;
if (bt_link_info->hid_exist)
- numdiffprofile++;
+ num_of_diff_profile++;
if (bt_link_info->pan_exist)
- numdiffprofile++;
+ num_of_diff_profile++;
if (bt_link_info->a2dp_exist)
- numdiffprofile++;
+ num_of_diff_profile++;
- if (numdiffprofile == 1) {
+ if (num_of_diff_profile == 1) {
if (bt_link_info->sco_exist) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"SCO only\n");
@@ -552,7 +553,7 @@
"A2DP only\n");
algorithm = BT_8192E_2ANT_COEX_ALGO_A2DP;
} else if (bt_link_info->pan_exist) {
- if (bt_hson) {
+ if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"PAN(HS) only\n");
@@ -567,7 +568,7 @@
}
}
}
- } else if (numdiffprofile == 2) {
+ } else if (num_of_diff_profile == 2) {
if (bt_link_info->sco_exist) {
if (bt_link_info->hid_exist) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -578,7 +579,7 @@
"SCO + A2DP ==> SCO\n");
algorithm = BT_8192E_2ANT_COEX_ALGO_PANEDR_HID;
} else if (bt_link_info->pan_exist) {
- if (bt_hson) {
+ if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"SCO + PAN(HS)\n");
@@ -609,7 +610,7 @@
}
} else if (bt_link_info->hid_exist &&
bt_link_info->pan_exist) {
- if (bt_hson) {
+ if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"HID + PAN(HS)\n");
@@ -623,7 +624,7 @@
}
} else if (bt_link_info->pan_exist &&
bt_link_info->a2dp_exist) {
- if (bt_hson) {
+ if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"A2DP + PAN(HS)\n");
@@ -638,7 +639,7 @@
}
}
}
- } else if (numdiffprofile == 3) {
+ } else if (num_of_diff_profile == 3) {
if (bt_link_info->sco_exist) {
if (bt_link_info->hid_exist &&
bt_link_info->a2dp_exist) {
@@ -647,7 +648,7 @@
algorithm = BT_8192E_2ANT_COEX_ALGO_PANEDR_HID;
} else if (bt_link_info->hid_exist &&
bt_link_info->pan_exist) {
- if (bt_hson) {
+ if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"SCO + HID + PAN(HS)\n");
@@ -661,7 +662,7 @@
}
} else if (bt_link_info->pan_exist &&
bt_link_info->a2dp_exist) {
- if (bt_hson) {
+ if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"SCO + A2DP + PAN(HS)\n");
@@ -678,7 +679,7 @@
if (bt_link_info->hid_exist &&
bt_link_info->pan_exist &&
bt_link_info->a2dp_exist) {
- if (bt_hson) {
+ if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"HID + A2DP + PAN(HS)\n");
@@ -693,12 +694,12 @@
}
}
}
- } else if (numdiffprofile >= 3) {
+ } else if (num_of_diff_profile >= 3) {
if (bt_link_info->sco_exist) {
if (bt_link_info->hid_exist &&
bt_link_info->pan_exist &&
bt_link_info->a2dp_exist) {
- if (bt_hson) {
+ if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"ErrorSCO+HID+A2DP+PAN(HS)\n");
@@ -717,8 +718,8 @@
return algorithm;
}
-static void halbtc8192e2ant_setfw_dac_swinglevel(struct btc_coexist *btcoexist,
- u8 dac_swinglvl)
+static void btc8192e2ant_set_fw_dac_swing_level(struct btc_coexist *btcoexist,
+ u8 dac_swing_lvl)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[1] = {0};
@@ -726,81 +727,81 @@
/* There are several type of dacswing
* 0x18/ 0x10/ 0xc/ 0x8/ 0x4/ 0x6
*/
- h2c_parameter[0] = dac_swinglvl;
+ h2c_parameter[0] = dac_swing_lvl;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Set Dac Swing Level = 0x%x\n", dac_swinglvl);
+ "[BTCoex], Set Dac Swing Level = 0x%x\n", dac_swing_lvl);
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], FW write 0x64 = 0x%x\n", h2c_parameter[0]);
btcoexist->btc_fill_h2c(btcoexist, 0x64, 1, h2c_parameter);
}
-static void halbtc8192e2ant_set_fwdec_btpwr(struct btc_coexist *btcoexist,
- u8 dec_btpwr_lvl)
+static void btc8192e2ant_set_fw_dec_bt_pwr(struct btc_coexist *btcoexist,
+ u8 dec_bt_pwr_lvl)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[1] = {0};
- h2c_parameter[0] = dec_btpwr_lvl;
+ h2c_parameter[0] = dec_bt_pwr_lvl;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex] decrease Bt Power level = %d, FW write 0x62 = 0x%x\n",
- dec_btpwr_lvl, h2c_parameter[0]);
+ dec_bt_pwr_lvl, h2c_parameter[0]);
btcoexist->btc_fill_h2c(btcoexist, 0x62, 1, h2c_parameter);
}
-static void halbtc8192e2ant_dec_btpwr(struct btc_coexist *btcoexist,
- bool force_exec, u8 dec_btpwr_lvl)
+static void btc8192e2ant_dec_bt_pwr(struct btc_coexist *btcoexist,
+ bool force_exec, u8 dec_bt_pwr_lvl)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], %s Dec BT power level = %d\n",
- force_exec ? "force to" : "", dec_btpwr_lvl);
- coex_dm->cur_dec_bt_pwr = dec_btpwr_lvl;
+ force_exec ? "force to" : "", dec_bt_pwr_lvl);
+ coex_dm->cur_dec_bt_pwr = dec_bt_pwr_lvl;
if (!force_exec) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], preBtDecPwrLvl=%d, curBtDecPwrLvl=%d\n",
coex_dm->pre_dec_bt_pwr, coex_dm->cur_dec_bt_pwr);
}
- halbtc8192e2ant_set_fwdec_btpwr(btcoexist, coex_dm->cur_dec_bt_pwr);
+ btc8192e2ant_set_fw_dec_bt_pwr(btcoexist, coex_dm->cur_dec_bt_pwr);
coex_dm->pre_dec_bt_pwr = coex_dm->cur_dec_bt_pwr;
}
-static void halbtc8192e2ant_set_bt_autoreport(struct btc_coexist *btcoexist,
- bool enable_autoreport)
+static void btc8192e2ant_set_bt_auto_report(struct btc_coexist *btcoexist,
+ bool enable_auto_report)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[1] = {0};
h2c_parameter[0] = 0;
- if (enable_autoreport)
+ if (enable_auto_report)
h2c_parameter[0] |= BIT0;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], BT FW auto report : %s, FW write 0x68 = 0x%x\n",
- (enable_autoreport ? "Enabled!!" : "Disabled!!"),
+ (enable_auto_report ? "Enabled!!" : "Disabled!!"),
h2c_parameter[0]);
btcoexist->btc_fill_h2c(btcoexist, 0x68, 1, h2c_parameter);
}
-static void halbtc8192e2ant_bt_autoreport(struct btc_coexist *btcoexist,
- bool force_exec,
- bool enable_autoreport)
+static void btc8192e2ant_bt_auto_report(struct btc_coexist *btcoexist,
+ bool force_exec,
+ bool enable_auto_report)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], %s BT Auto report = %s\n",
(force_exec ? "force to" : ""),
- ((enable_autoreport) ? "Enabled" : "Disabled"));
- coex_dm->cur_bt_auto_report = enable_autoreport;
+ ((enable_auto_report) ? "Enabled" : "Disabled"));
+ coex_dm->cur_bt_auto_report = enable_auto_report;
if (!force_exec) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -811,21 +812,21 @@
if (coex_dm->pre_bt_auto_report == coex_dm->cur_bt_auto_report)
return;
}
- halbtc8192e2ant_set_bt_autoreport(btcoexist,
- coex_dm->cur_bt_auto_report);
+ btc8192e2ant_set_bt_auto_report(btcoexist,
+ coex_dm->cur_bt_auto_report);
coex_dm->pre_bt_auto_report = coex_dm->cur_bt_auto_report;
}
-static void halbtc8192e2ant_fw_dac_swinglvl(struct btc_coexist *btcoexist,
- bool force_exec, u8 fw_dac_swinglvl)
+static void btc8192e2ant_fw_dac_swing_lvl(struct btc_coexist *btcoexist,
+ bool force_exec, u8 fw_dac_swing_lvl)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], %s set FW Dac Swing level = %d\n",
- (force_exec ? "force to" : ""), fw_dac_swinglvl);
- coex_dm->cur_fw_dac_swing_lvl = fw_dac_swinglvl;
+ (force_exec ? "force to" : ""), fw_dac_swing_lvl);
+ coex_dm->cur_fw_dac_swing_lvl = fw_dac_swing_lvl;
if (!force_exec) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -838,8 +839,8 @@
return;
}
- halbtc8192e2ant_setfw_dac_swinglevel(btcoexist,
- coex_dm->cur_fw_dac_swing_lvl);
+ btc8192e2ant_set_fw_dac_swing_level(btcoexist,
+ coex_dm->cur_fw_dac_swing_lvl);
coex_dm->pre_fw_dac_swing_lvl = coex_dm->cur_fw_dac_swing_lvl;
}
@@ -869,8 +870,8 @@
}
}
-static void halbtc8192e2ant_rf_shrink(struct btc_coexist *btcoexist,
- bool force_exec, bool rx_rf_shrink_on)
+static void btc8192e2ant_rf_shrink(struct btc_coexist *btcoexist,
+ bool force_exec, bool rx_rf_shrink_on)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -896,8 +897,8 @@
coex_dm->pre_rf_rx_lpf_shrink = coex_dm->cur_rf_rx_lpf_shrink;
}
-static void halbtc8192e2ant_set_dac_swingreg(struct btc_coexist *btcoexist,
- u32 level)
+static void btc8192e2ant_set_dac_swing_reg(struct btc_coexist *btcoexist,
+ u32 level)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 val = (u8)level;
@@ -907,28 +908,28 @@
btcoexist->btc_write_1byte_bitmask(btcoexist, 0x883, 0x3e, val);
}
-static void btc8192e2ant_setsw_full_swing(struct btc_coexist *btcoexist,
- bool sw_dac_swingon,
- u32 sw_dac_swinglvl)
+static void btc8192e2ant_set_sw_full_swing(struct btc_coexist *btcoexist,
+ bool sw_dac_swing_on,
+ u32 sw_dac_swing_lvl)
{
- if (sw_dac_swingon)
- halbtc8192e2ant_set_dac_swingreg(btcoexist, sw_dac_swinglvl);
+ if (sw_dac_swing_on)
+ btc8192e2ant_set_dac_swing_reg(btcoexist, sw_dac_swing_lvl);
else
- halbtc8192e2ant_set_dac_swingreg(btcoexist, 0x18);
+ btc8192e2ant_set_dac_swing_reg(btcoexist, 0x18);
}
-static void halbtc8192e2ant_DacSwing(struct btc_coexist *btcoexist,
- bool force_exec, bool dac_swingon,
- u32 dac_swinglvl)
+static void btc8192e2ant_dac_swing(struct btc_coexist *btcoexist,
+ bool force_exec, bool dac_swing_on,
+ u32 dac_swing_lvl)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], %s turn DacSwing=%s, dac_swinglvl = 0x%x\n",
+ "[BTCoex], %s turn DacSwing=%s, dac_swing_lvl = 0x%x\n",
(force_exec ? "force to" : ""),
- ((dac_swingon) ? "ON" : "OFF"), dac_swinglvl);
- coex_dm->cur_dac_swing_on = dac_swingon;
- coex_dm->cur_dac_swing_lvl = dac_swinglvl;
+ ((dac_swing_on) ? "ON" : "OFF"), dac_swing_lvl);
+ coex_dm->cur_dac_swing_on = dac_swing_on;
+ coex_dm->cur_dac_swing_lvl = dac_swing_lvl;
if (!force_exec) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -945,14 +946,14 @@
return;
}
mdelay(30);
- btc8192e2ant_setsw_full_swing(btcoexist, dac_swingon, dac_swinglvl);
+ btc8192e2ant_set_sw_full_swing(btcoexist, dac_swing_on, dac_swing_lvl);
coex_dm->pre_dac_swing_on = coex_dm->cur_dac_swing_on;
coex_dm->pre_dac_swing_lvl = coex_dm->cur_dac_swing_lvl;
}
-static void halbtc8192e2ant_set_agc_table(struct btc_coexist *btcoexist,
- bool agc_table_en)
+static void btc8192e2ant_set_agc_table(struct btc_coexist *btcoexist,
+ bool agc_table_en)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -978,8 +979,8 @@
}
}
-static void halbtc8192e2ant_AgcTable(struct btc_coexist *btcoexist,
- bool force_exec, bool agc_table_en)
+static void btc8192e2ant_agc_table(struct btc_coexist *btcoexist,
+ bool force_exec, bool agc_table_en)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -998,14 +999,14 @@
if (coex_dm->pre_agc_table_en == coex_dm->cur_agc_table_en)
return;
}
- halbtc8192e2ant_set_agc_table(btcoexist, agc_table_en);
+ btc8192e2ant_set_agc_table(btcoexist, agc_table_en);
coex_dm->pre_agc_table_en = coex_dm->cur_agc_table_en;
}
-static void halbtc8192e2ant_set_coex_table(struct btc_coexist *btcoexist,
- u32 val0x6c0, u32 val0x6c4,
- u32 val0x6c8, u8 val0x6cc)
+static void btc8192e2ant_set_coex_table(struct btc_coexist *btcoexist,
+ u32 val0x6c0, u32 val0x6c4,
+ u32 val0x6c8, u8 val0x6cc)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -1026,10 +1027,9 @@
btcoexist->btc_write_1byte(btcoexist, 0x6cc, val0x6cc);
}
-static void halbtc8192e2ant_coex_table(struct btc_coexist *btcoexist,
- bool force_exec,
- u32 val0x6c0, u32 val0x6c4,
- u32 val0x6c8, u8 val0x6cc)
+static void btc8192e2ant_coex_table(struct btc_coexist *btcoexist,
+ bool force_exec, u32 val0x6c0, u32 val0x6c4,
+ u32 val0x6c8, u8 val0x6cc)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -1064,8 +1064,8 @@
(coex_dm->pre_val0x6cc == coex_dm->cur_val0x6cc))
return;
}
- halbtc8192e2ant_set_coex_table(btcoexist, val0x6c0, val0x6c4,
- val0x6c8, val0x6cc);
+ btc8192e2ant_set_coex_table(btcoexist, val0x6c0, val0x6c4, val0x6c8,
+ val0x6cc);
coex_dm->pre_val0x6c0 = coex_dm->cur_val0x6c0;
coex_dm->pre_val0x6c4 = coex_dm->cur_val0x6c4;
@@ -1073,37 +1073,37 @@
coex_dm->pre_val0x6cc = coex_dm->cur_val0x6cc;
}
-static void btc8192e2ant_coex_tbl_w_type(struct btc_coexist *btcoexist,
- bool force_exec, u8 type)
+static void btc8192e2ant_coex_table_with_type(struct btc_coexist *btcoexist,
+ bool force_exec, u8 type)
{
switch (type) {
case 0:
- halbtc8192e2ant_coex_table(btcoexist, force_exec, 0x55555555,
- 0x5a5a5a5a, 0xffffff, 0x3);
+ btc8192e2ant_coex_table(btcoexist, force_exec, 0x55555555,
+ 0x5a5a5a5a, 0xffffff, 0x3);
break;
case 1:
- halbtc8192e2ant_coex_table(btcoexist, force_exec, 0x5a5a5a5a,
- 0x5a5a5a5a, 0xffffff, 0x3);
+ btc8192e2ant_coex_table(btcoexist, force_exec, 0x5a5a5a5a,
+ 0x5a5a5a5a, 0xffffff, 0x3);
break;
case 2:
- halbtc8192e2ant_coex_table(btcoexist, force_exec, 0x55555555,
- 0x5ffb5ffb, 0xffffff, 0x3);
+ btc8192e2ant_coex_table(btcoexist, force_exec, 0x55555555,
+ 0x5ffb5ffb, 0xffffff, 0x3);
break;
case 3:
- halbtc8192e2ant_coex_table(btcoexist, force_exec, 0xdfffdfff,
- 0x5fdb5fdb, 0xffffff, 0x3);
+ btc8192e2ant_coex_table(btcoexist, force_exec, 0xdfffdfff,
+ 0x5fdb5fdb, 0xffffff, 0x3);
break;
case 4:
- halbtc8192e2ant_coex_table(btcoexist, force_exec, 0xdfffdfff,
- 0x5ffb5ffb, 0xffffff, 0x3);
+ btc8192e2ant_coex_table(btcoexist, force_exec, 0xdfffdfff,
+ 0x5ffb5ffb, 0xffffff, 0x3);
break;
default:
break;
}
}
-static void halbtc8192e2ant_set_fw_ignore_wlanact(struct btc_coexist *btcoexist,
- bool enable)
+static void btc8192e2ant_set_fw_ignore_wlan_act(struct btc_coexist *btcoexist,
+ bool enable)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[1] = {0};
@@ -1118,8 +1118,8 @@
btcoexist->btc_fill_h2c(btcoexist, 0x63, 1, h2c_parameter);
}
-static void halbtc8192e2ant_IgnoreWlanAct(struct btc_coexist *btcoexist,
- bool force_exec, bool enable)
+static void btc8192e2ant_ignore_wlan_act(struct btc_coexist *btcoexist,
+ bool force_exec, bool enable)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -1140,12 +1140,12 @@
coex_dm->cur_ignore_wlan_act)
return;
}
- halbtc8192e2ant_set_fw_ignore_wlanact(btcoexist, enable);
+ btc8192e2ant_set_fw_ignore_wlan_act(btcoexist, enable);
coex_dm->pre_ignore_wlan_act = coex_dm->cur_ignore_wlan_act;
}
-static void halbtc8192e2ant_SetFwPstdma(struct btc_coexist *btcoexist, u8 byte1,
+static void btc8192e2ant_set_fw_ps_tdma(struct btc_coexist *btcoexist, u8 byte1,
u8 byte2, u8 byte3, u8 byte4, u8 byte5)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -1173,24 +1173,24 @@
btcoexist->btc_fill_h2c(btcoexist, 0x60, 5, h2c_parameter);
}
-static void btc8192e2ant_sw_mec1(struct btc_coexist *btcoexist,
- bool shrink_rx_lpf, bool low_penalty_ra,
- bool limited_dig, bool btlan_constrain)
+static void btc8192e2ant_sw_mechanism1(struct btc_coexist *btcoexist,
+ bool shrink_rx_lpf, bool low_penalty_ra,
+ bool limited_dig, bool btlan_constrain)
{
- halbtc8192e2ant_rf_shrink(btcoexist, NORMAL_EXEC, shrink_rx_lpf);
+ btc8192e2ant_rf_shrink(btcoexist, NORMAL_EXEC, shrink_rx_lpf);
}
-static void btc8192e2ant_sw_mec2(struct btc_coexist *btcoexist,
- bool agc_table_shift, bool adc_backoff,
- bool sw_dac_swing, u32 dac_swinglvl)
+static void btc8192e2ant_sw_mechanism2(struct btc_coexist *btcoexist,
+ bool agc_table_shift, bool adc_backoff,
+ bool sw_dac_swing, u32 dac_swing_lvl)
{
- halbtc8192e2ant_AgcTable(btcoexist, NORMAL_EXEC, agc_table_shift);
- halbtc8192e2ant_DacSwing(btcoexist, NORMAL_EXEC, sw_dac_swing,
- dac_swinglvl);
+ btc8192e2ant_agc_table(btcoexist, NORMAL_EXEC, agc_table_shift);
+ btc8192e2ant_dac_swing(btcoexist, NORMAL_EXEC, sw_dac_swing,
+ dac_swing_lvl);
}
-static void halbtc8192e2ant_ps_tdma(struct btc_coexist *btcoexist,
- bool force_exec, bool turn_on, u8 type)
+static void btc8192e2ant_ps_tdma(struct btc_coexist *btcoexist,
+ bool force_exec, bool turn_on, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -1217,91 +1217,91 @@
switch (type) {
case 1:
default:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x1a,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1a,
0x1a, 0xe1, 0x90);
break;
case 2:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x12,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x12,
0x12, 0xe1, 0x90);
break;
case 3:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x1c,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1c,
0x3, 0xf1, 0x90);
break;
case 4:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x10,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x10,
0x3, 0xf1, 0x90);
break;
case 5:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x1a,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1a,
0x1a, 0x60, 0x90);
break;
case 6:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x12,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x12,
0x12, 0x60, 0x90);
break;
case 7:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x1c,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1c,
0x3, 0x70, 0x90);
break;
case 8:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xa3, 0x10,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xa3, 0x10,
0x3, 0x70, 0x90);
break;
case 9:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x1a,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1a,
0x1a, 0xe1, 0x10);
break;
case 10:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x12,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x12,
0x12, 0xe1, 0x10);
break;
case 11:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x1c,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1c,
0x3, 0xf1, 0x10);
break;
case 12:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x10,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x10,
0x3, 0xf1, 0x10);
break;
case 13:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x1a,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1a,
0x1a, 0xe0, 0x10);
break;
case 14:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x12,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x12,
0x12, 0xe0, 0x10);
break;
case 15:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x1c,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1c,
0x3, 0xf0, 0x10);
break;
case 16:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x12,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x12,
0x3, 0xf0, 0x10);
break;
case 17:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0x61, 0x20,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0x61, 0x20,
0x03, 0x10, 0x10);
break;
case 18:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x5,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x5,
0x5, 0xe1, 0x90);
break;
case 19:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x25,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x25,
0x25, 0xe1, 0x90);
break;
case 20:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x25,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x25,
0x25, 0x60, 0x90);
break;
case 21:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x15,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x15,
0x03, 0x70, 0x90);
break;
case 71:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0xe3, 0x1a,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1a,
0x1a, 0xe1, 0x90);
break;
}
@@ -1310,12 +1310,12 @@
switch (type) {
default:
case 0:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0x8, 0x0, 0x0,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0x8, 0x0, 0x0,
0x0, 0x0);
btcoexist->btc_write_1byte(btcoexist, 0x92c, 0x4);
break;
case 1:
- halbtc8192e2ant_SetFwPstdma(btcoexist, 0x0, 0x0, 0x0,
+ btc8192e2ant_set_fw_ps_tdma(btcoexist, 0x0, 0x0, 0x0,
0x8, 0x0);
mdelay(5);
btcoexist->btc_write_1byte(btcoexist, 0x92c, 0x20);
@@ -1328,22 +1328,22 @@
coex_dm->pre_ps_tdma = coex_dm->cur_ps_tdma;
}
-static void halbtc8192e2ant_set_switch_sstype(struct btc_coexist *btcoexist,
- u8 sstype)
+static void btc8192e2ant_set_switch_ss_type(struct btc_coexist *btcoexist,
+ u8 ss_type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 mimops = BTC_MIMO_PS_DYNAMIC;
- u32 disra_mask = 0x0;
+ u32 dis_ra_mask = 0x0;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], REAL set SS Type = %d\n", sstype);
+ "[BTCoex], REAL set SS Type = %d\n", ss_type);
- disra_mask = halbtc8192e2ant_decidera_mask(btcoexist, sstype,
- coex_dm->curra_masktype);
- halbtc8192e2ant_Updatera_mask(btcoexist, FORCE_EXEC, disra_mask);
+ dis_ra_mask = btc8192e2ant_decide_ra_mask(btcoexist, ss_type,
+ coex_dm->cur_ra_mask_type);
+ btc8192e2ant_update_ra_mask(btcoexist, FORCE_EXEC, dis_ra_mask);
- if (sstype == 1) {
- halbtc8192e2ant_ps_tdma(btcoexist, FORCE_EXEC, false, 1);
+ if (ss_type == 1) {
+ btc8192e2ant_ps_tdma(btcoexist, FORCE_EXEC, false, 1);
/* switch ofdm path */
btcoexist->btc_write_1byte(btcoexist, 0xc04, 0x11);
btcoexist->btc_write_1byte(btcoexist, 0xd04, 0x1);
@@ -1352,8 +1352,8 @@
btcoexist->btc_write_1byte_bitmask(btcoexist, 0xe77, 0x4, 0x1);
btcoexist->btc_write_1byte(btcoexist, 0xa07, 0x81);
mimops = BTC_MIMO_PS_STATIC;
- } else if (sstype == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist, FORCE_EXEC, false, 0);
+ } else if (ss_type == 2) {
+ btc8192e2ant_ps_tdma(btcoexist, FORCE_EXEC, false, 0);
btcoexist->btc_write_1byte(btcoexist, 0xc04, 0x33);
btcoexist->btc_write_1byte(btcoexist, 0xd04, 0x3);
btcoexist->btc_write_4byte(btcoexist, 0x90c, 0x81121313);
@@ -1365,89 +1365,89 @@
btcoexist->btc_set(btcoexist, BTC_SET_ACT_SEND_MIMO_PS, &mimops);
}
-static void halbtc8192e2ant_switch_sstype(struct btc_coexist *btcoexist,
- bool force_exec, u8 new_sstype)
+static void btc8192e2ant_switch_ss_type(struct btc_coexist *btcoexist,
+ bool force_exec, u8 new_ss_type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], %s Switch SS Type = %d\n",
- (force_exec ? "force to" : ""), new_sstype);
- coex_dm->cur_sstype = new_sstype;
+ (force_exec ? "force to" : ""), new_ss_type);
+ coex_dm->cur_ss_type = new_ss_type;
if (!force_exec) {
- if (coex_dm->pre_sstype == coex_dm->cur_sstype)
+ if (coex_dm->pre_ss_type == coex_dm->cur_ss_type)
return;
}
- halbtc8192e2ant_set_switch_sstype(btcoexist, coex_dm->cur_sstype);
+ btc8192e2ant_set_switch_ss_type(btcoexist, coex_dm->cur_ss_type);
- coex_dm->pre_sstype = coex_dm->cur_sstype;
+ coex_dm->pre_ss_type = coex_dm->cur_ss_type;
}
-static void halbtc8192e2ant_coex_alloff(struct btc_coexist *btcoexist)
+static void btc8192e2ant_coex_all_off(struct btc_coexist *btcoexist)
{
/* fw all off */
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, NORMAL_EXEC, 6);
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
/* sw all off */
- btc8192e2ant_sw_mec1(btcoexist, false, false, false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false, false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false, false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false, false, 0x18);
/* hw all off */
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
}
-static void halbtc8192e2ant_init_coex_dm(struct btc_coexist *btcoexist)
+static void btc8192e2ant_init_coex_dm(struct btc_coexist *btcoexist)
{
/* force to reset coex mechanism */
- halbtc8192e2ant_ps_tdma(btcoexist, FORCE_EXEC, false, 1);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, FORCE_EXEC, 6);
- halbtc8192e2ant_dec_btpwr(btcoexist, FORCE_EXEC, 0);
+ btc8192e2ant_ps_tdma(btcoexist, FORCE_EXEC, false, 1);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, FORCE_EXEC, 6);
+ btc8192e2ant_dec_bt_pwr(btcoexist, FORCE_EXEC, 0);
- btc8192e2ant_coex_tbl_w_type(btcoexist, FORCE_EXEC, 0);
- halbtc8192e2ant_switch_sstype(btcoexist, FORCE_EXEC, 2);
+ btc8192e2ant_coex_table_with_type(btcoexist, FORCE_EXEC, 0);
+ btc8192e2ant_switch_ss_type(btcoexist, FORCE_EXEC, 2);
- btc8192e2ant_sw_mec1(btcoexist, false, false, false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false, false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false, false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false, false, 0x18);
}
-static void halbtc8192e2ant_action_bt_inquiry(struct btc_coexist *btcoexist)
+static void btc8192e2ant_action_bt_inquiry(struct btc_coexist *btcoexist)
{
bool low_pwr_disable = true;
btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
&low_pwr_disable);
- halbtc8192e2ant_switch_sstype(btcoexist, NORMAL_EXEC, 1);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 1);
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 2);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 3);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, NORMAL_EXEC, 6);
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 3);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
- btc8192e2ant_sw_mec1(btcoexist, false, false, false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false, false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false, false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false, false, 0x18);
}
-static bool halbtc8192e2ant_is_common_action(struct btc_coexist *btcoexist)
+static bool btc8192e2ant_is_common_action(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
bool common = false, wifi_connected = false, wifi_busy = false;
- bool bt_hson = false, low_pwr_disable = false;
+ bool bt_hs_on = false, low_pwr_disable = false;
- btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hson);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
&wifi_connected);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_BUSY, &wifi_busy);
if (bt_link_info->sco_exist || bt_link_info->hid_exist)
- halbtc8192e2ant_limited_tx(btcoexist, NORMAL_EXEC, 1, 0, 0, 0);
+ btc8192e2ant_limited_tx(btcoexist, NORMAL_EXEC, 1, 0, 0, 0);
else
- halbtc8192e2ant_limited_tx(btcoexist, NORMAL_EXEC, 0, 0, 0, 0);
+ btc8192e2ant_limited_tx(btcoexist, NORMAL_EXEC, 0, 0, 0, 0);
if (!wifi_connected) {
low_pwr_disable = false;
@@ -1461,26 +1461,24 @@
coex_dm->bt_status) ||
(BT_8192E_2ANT_BT_STATUS_CONNECTED_IDLE ==
coex_dm->bt_status)) {
- halbtc8192e2ant_switch_sstype(btcoexist,
- NORMAL_EXEC, 2);
- btc8192e2ant_coex_tbl_w_type(btcoexist,
- NORMAL_EXEC, 1);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- false, 0);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 1);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 0);
} else {
- halbtc8192e2ant_switch_sstype(btcoexist,
- NORMAL_EXEC, 1);
- btc8192e2ant_coex_tbl_w_type(btcoexist,
- NORMAL_EXEC, 0);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- false, 1);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 1);
+ btc8192e2ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 0);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
}
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, NORMAL_EXEC, 6);
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
- btc8192e2ant_sw_mec1(btcoexist, false, false, false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false, false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false, false,
+ false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false, false,
+ 0x18);
common = true;
} else {
@@ -1494,20 +1492,18 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Wifi connected + BT non connected-idle!!\n");
- halbtc8192e2ant_switch_sstype(btcoexist,
- NORMAL_EXEC, 2);
- btc8192e2ant_coex_tbl_w_type(btcoexist,
- NORMAL_EXEC, 1);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- false, 0);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist,
- NORMAL_EXEC, 6);
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 1);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 0);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist,
+ NORMAL_EXEC, 6);
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
- btc8192e2ant_sw_mec1(btcoexist, false, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
common = true;
} else if (BT_8192E_2ANT_BT_STATUS_CONNECTED_IDLE ==
@@ -1517,25 +1513,25 @@
BTC_SET_ACT_DISABLE_LOW_POWER,
&low_pwr_disable);
- if (bt_hson)
+ if (bt_hs_on)
return false;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Wifi connected + BT connected-idle!!\n");
- halbtc8192e2ant_switch_sstype(btcoexist,
- NORMAL_EXEC, 2);
- btc8192e2ant_coex_tbl_w_type(btcoexist,
- NORMAL_EXEC, 1);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- false, 0);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist,
- NORMAL_EXEC, 6);
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_switch_ss_type(btcoexist,
+ NORMAL_EXEC, 2);
+ btc8192e2ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 1);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
+ false, 0);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist,
+ NORMAL_EXEC, 6);
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
- btc8192e2ant_sw_mec1(btcoexist, true, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
common = true;
} else {
@@ -1552,20 +1548,21 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Wifi Connected-Idle + BT Busy!!\n");
- halbtc8192e2ant_switch_sstype(btcoexist,
- NORMAL_EXEC, 1);
- btc8192e2ant_coex_tbl_w_type(btcoexist,
- NORMAL_EXEC, 2);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 21);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist,
- NORMAL_EXEC, 6);
- halbtc8192e2ant_dec_btpwr(btcoexist,
- NORMAL_EXEC, 0);
- btc8192e2ant_sw_mec1(btcoexist, false,
- false, false, false);
- btc8192e2ant_sw_mec2(btcoexist, false,
- false, false, 0x18);
+ btc8192e2ant_switch_ss_type(btcoexist,
+ NORMAL_EXEC, 1);
+ btc8192e2ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC,
+ 2);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
+ true, 21);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist,
+ NORMAL_EXEC, 6);
+ btc8192e2ant_dec_bt_pwr(btcoexist,
+ NORMAL_EXEC, 0);
+ btc8192e2ant_sw_mechanism1(btcoexist, false,
+ false, false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false,
+ false, false, 0x18);
common = true;
}
}
@@ -1573,588 +1570,9 @@
return common;
}
-static void btc8192e_int1(struct btc_coexist *btcoexist, bool tx_pause,
- int result)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- if (tx_pause) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 1\n");
-
- if (coex_dm->cur_ps_tdma == 71) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 5);
- coex_dm->tdma_adj_type = 5;
- } else if (coex_dm->cur_ps_tdma == 1) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 5);
- coex_dm->tdma_adj_type = 5;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 4) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- }
- if (coex_dm->cur_ps_tdma == 9) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 13);
- coex_dm->tdma_adj_type = 13;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 12) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
-
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 5) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- } else if (coex_dm->cur_ps_tdma == 13) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 8) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 5);
- coex_dm->tdma_adj_type = 5;
- } else if (coex_dm->cur_ps_tdma == 16) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 13);
- coex_dm->tdma_adj_type = 13;
- }
- }
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 0\n");
- if (coex_dm->cur_ps_tdma == 5) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 71);
- coex_dm->tdma_adj_type = 71;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 8) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- }
- if (coex_dm->cur_ps_tdma == 13) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 16) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
-
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 71) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 1);
- coex_dm->tdma_adj_type = 1;
- } else if (coex_dm->cur_ps_tdma == 1) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- } else if (coex_dm->cur_ps_tdma == 9) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 4) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 1);
- coex_dm->tdma_adj_type = 1;
- } else if (coex_dm->cur_ps_tdma == 1) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 71);
- coex_dm->tdma_adj_type = 71;
- } else if (coex_dm->cur_ps_tdma == 12) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
- }
- }
- }
-}
-
-static void btc8192e_int2(struct btc_coexist *btcoexist, bool tx_pause,
- int result)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- if (tx_pause) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 1\n");
- if (coex_dm->cur_ps_tdma == 1) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 4) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- }
- if (coex_dm->cur_ps_tdma == 9) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 12) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 5) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- } else if (coex_dm->cur_ps_tdma == 13) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 8) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 16) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- }
- }
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 0\n");
- if (coex_dm->cur_ps_tdma == 5) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 8) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- }
- if (coex_dm->cur_ps_tdma == 13) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 16) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 1) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- } else if (coex_dm->cur_ps_tdma == 9) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 4) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 12) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- }
- }
- }
-}
-
-static void btc8192e_int3(struct btc_coexist *btcoexist, bool tx_pause,
- int result)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- if (tx_pause) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 1\n");
- if (coex_dm->cur_ps_tdma == 1) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 4) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- }
- if (coex_dm->cur_ps_tdma == 9) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 12) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 5) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- } else if (coex_dm->cur_ps_tdma == 13) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 8) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 16) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- }
- }
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 0\n");
- if (coex_dm->cur_ps_tdma == 5) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 8) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- }
- if (coex_dm->cur_ps_tdma == 13) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 16) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 1) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- } else if (coex_dm->cur_ps_tdma == 9) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 4) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 12) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- }
- }
- }
-}
-
-static void halbtc8192e2ant_tdma_duration_adjust(struct btc_coexist *btcoexist,
- bool sco_hid, bool tx_pause,
- u8 max_interval)
+static void btc8192e2ant_tdma_duration_adjust(struct btc_coexist *btcoexist,
+ bool sco_hid, bool tx_pause,
+ u8 max_interval)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
static int up, dn, m, n, wait_cnt;
@@ -2174,72 +1592,72 @@
if (sco_hid) {
if (tx_pause) {
if (max_interval == 1) {
- halbtc8192e2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 13);
+ btc8192e2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 13);
coex_dm->tdma_adj_type = 13;
} else if (max_interval == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 14);
+ btc8192e2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 14);
coex_dm->tdma_adj_type = 14;
} else {
- halbtc8192e2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 15);
+ btc8192e2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 15);
coex_dm->tdma_adj_type = 15;
}
} else {
if (max_interval == 1) {
- halbtc8192e2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 9);
+ btc8192e2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 9);
coex_dm->tdma_adj_type = 9;
} else if (max_interval == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 10);
+ btc8192e2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 10);
coex_dm->tdma_adj_type = 10;
} else {
- halbtc8192e2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 11);
+ btc8192e2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 11);
coex_dm->tdma_adj_type = 11;
}
}
} else {
if (tx_pause) {
if (max_interval == 1) {
- halbtc8192e2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 5);
+ btc8192e2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 5);
coex_dm->tdma_adj_type = 5;
} else if (max_interval == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 6);
+ btc8192e2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 6);
coex_dm->tdma_adj_type = 6;
} else {
- halbtc8192e2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 7);
+ btc8192e2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 7);
coex_dm->tdma_adj_type = 7;
}
} else {
if (max_interval == 1) {
- halbtc8192e2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 1);
+ btc8192e2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 1);
coex_dm->tdma_adj_type = 1;
} else if (max_interval == 2) {
- halbtc8192e2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 2);
+ btc8192e2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 2);
coex_dm->tdma_adj_type = 2;
} else {
- halbtc8192e2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 3);
+ btc8192e2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 3);
coex_dm->tdma_adj_type = 3;
}
}
@@ -2322,12 +1740,6 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], max Interval = %d\n", max_interval);
- if (max_interval == 1)
- btc8192e_int1(btcoexist, tx_pause, result);
- else if (max_interval == 2)
- btc8192e_int2(btcoexist, tx_pause, result);
- else if (max_interval == 3)
- btc8192e_int3(btcoexist, tx_pause, result);
}
/* if current PsTdma not match with
@@ -2348,9 +1760,8 @@
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_ROAM, &roam);
if (!scan && !link && !roam)
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true,
- coex_dm->tdma_adj_type);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
+ true, coex_dm->tdma_adj_type);
else
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], roaming/link/scan is under progress, will adjust next time!!!\n");
@@ -2358,583 +1769,578 @@
}
/* SCO only or SCO+PAN(HS) */
-static void halbtc8192e2ant_action_sco(struct btc_coexist *btcoexist)
+static void btc8192e2ant_action_sco(struct btc_coexist *btcoexist)
{
- u8 wifirssi_state, btrssi_state = BTC_RSSI_STATE_STAY_LOW;
+ u8 wifi_rssi_state, bt_rssi_state = BTC_RSSI_STATE_STAY_LOW;
u32 wifi_bw;
- wifirssi_state = halbtc8192e2ant_wifirssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state = btc8192e2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
- halbtc8192e2ant_switch_sstype(btcoexist, NORMAL_EXEC, 1);
- halbtc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 1);
+ btc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, NORMAL_EXEC, 6);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 4);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 4);
- btrssi_state = halbtc8192e2ant_btrssi_state(btcoexist, 3, 34, 42);
+ bt_rssi_state = btc8192e2ant_bt_rssi_state(btcoexist, 3, 34, 42);
- if ((btrssi_state == BTC_RSSI_STATE_LOW) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 13);
- } else if ((btrssi_state == BTC_RSSI_STATE_MEDIUM) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 2);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 9);
- } else if ((btrssi_state == BTC_RSSI_STATE_HIGH) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 4);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 9);
+ if ((bt_rssi_state == BTC_RSSI_STATE_LOW) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 13);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_MEDIUM) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 9);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 4);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 9);
}
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
/* sw mechanism */
if (BTC_WIFI_BW_HT40 == wifi_bw) {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, true, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x6);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x6);
} else {
- btc8192e2ant_sw_mec1(btcoexist, true, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x6);
+ btc8192e2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x6);
}
} else {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, false, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x6);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x6);
} else {
- btc8192e2ant_sw_mec1(btcoexist, false, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x6);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x6);
}
}
}
-static void halbtc8192e2ant_action_sco_pan(struct btc_coexist *btcoexist)
+static void btc8192e2ant_action_sco_pan(struct btc_coexist *btcoexist)
{
- u8 wifirssi_state, btrssi_state = BTC_RSSI_STATE_STAY_LOW;
+ u8 wifi_rssi_state, bt_rssi_state = BTC_RSSI_STATE_STAY_LOW;
u32 wifi_bw;
- wifirssi_state = halbtc8192e2ant_wifirssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state = btc8192e2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
- halbtc8192e2ant_switch_sstype(btcoexist, NORMAL_EXEC, 1);
- halbtc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 1);
+ btc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, NORMAL_EXEC, 6);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 4);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 4);
- btrssi_state = halbtc8192e2ant_btrssi_state(btcoexist, 3, 34, 42);
+ bt_rssi_state = btc8192e2ant_bt_rssi_state(btcoexist, 3, 34, 42);
- if ((btrssi_state == BTC_RSSI_STATE_LOW) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
- } else if ((btrssi_state == BTC_RSSI_STATE_MEDIUM) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 2);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 10);
- } else if ((btrssi_state == BTC_RSSI_STATE_HIGH) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 4);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 10);
+ if ((bt_rssi_state == BTC_RSSI_STATE_LOW) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_MEDIUM) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 10);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 4);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 10);
}
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
/* sw mechanism */
if (BTC_WIFI_BW_HT40 == wifi_bw) {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, true, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x6);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x6);
} else {
- btc8192e2ant_sw_mec1(btcoexist, true, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x6);
+ btc8192e2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x6);
}
} else {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, false, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x6);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x6);
} else {
- btc8192e2ant_sw_mec1(btcoexist, false, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x6);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x6);
}
}
}
-static void halbtc8192e2ant_action_hid(struct btc_coexist *btcoexist)
+static void btc8192e2ant_action_hid(struct btc_coexist *btcoexist)
{
- u8 wifirssi_state, btrssi_state = BTC_RSSI_STATE_HIGH;
+ u8 wifi_rssi_state, bt_rssi_state = BTC_RSSI_STATE_HIGH;
u32 wifi_bw;
- wifirssi_state = halbtc8192e2ant_wifirssi_state(btcoexist, 0, 2, 15, 0);
- btrssi_state = halbtc8192e2ant_btrssi_state(btcoexist, 3, 34, 42);
+ wifi_rssi_state = btc8192e2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8192e2ant_bt_rssi_state(btcoexist, 3, 34, 42);
- halbtc8192e2ant_switch_sstype(btcoexist, NORMAL_EXEC, 1);
- halbtc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 1);
+ btc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, NORMAL_EXEC, 6);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 3);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 3);
- if ((btrssi_state == BTC_RSSI_STATE_LOW) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 13);
- } else if ((btrssi_state == BTC_RSSI_STATE_MEDIUM) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 2);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 9);
- } else if ((btrssi_state == BTC_RSSI_STATE_HIGH) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 4);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 9);
+ if ((bt_rssi_state == BTC_RSSI_STATE_LOW) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 13);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_MEDIUM) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 9);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 4);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 9);
}
/* sw mechanism */
if (BTC_WIFI_BW_HT40 == wifi_bw) {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, true, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, true, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, false, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, false, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
/* A2DP only / PAN(EDR) only/ A2DP+PAN(HS) */
-static void halbtc8192e2ant_action_a2dp(struct btc_coexist *btcoexist)
+static void btc8192e2ant_action_a2dp(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- u8 wifirssi_state, btrssi_state = BTC_RSSI_STATE_HIGH;
+ u8 wifi_rssi_state, bt_rssi_state = BTC_RSSI_STATE_HIGH;
u32 wifi_bw;
bool long_dist = false;
- wifirssi_state = halbtc8192e2ant_wifirssi_state(btcoexist, 0, 2, 15, 0);
- btrssi_state = halbtc8192e2ant_btrssi_state(btcoexist, 3, 34, 42);
+ wifi_rssi_state = btc8192e2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8192e2ant_bt_rssi_state(btcoexist, 3, 34, 42);
- if ((btrssi_state == BTC_RSSI_STATE_LOW ||
- btrssi_state == BTC_RSSI_STATE_STAY_LOW) &&
- (wifirssi_state == BTC_RSSI_STATE_LOW ||
- wifirssi_state == BTC_RSSI_STATE_STAY_LOW)) {
+ if ((bt_rssi_state == BTC_RSSI_STATE_LOW ||
+ bt_rssi_state == BTC_RSSI_STATE_STAY_LOW) &&
+ (wifi_rssi_state == BTC_RSSI_STATE_LOW ||
+ wifi_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], A2dp, wifi/bt rssi both LOW!!\n");
long_dist = true;
}
if (long_dist) {
- halbtc8192e2ant_switch_sstype(btcoexist, NORMAL_EXEC, 2);
- halbtc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, true,
- 0x4);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, true,
+ 0x4);
} else {
- halbtc8192e2ant_switch_sstype(btcoexist, NORMAL_EXEC, 1);
- halbtc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false,
- 0x8);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 1);
+ btc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false,
+ 0x8);
}
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, NORMAL_EXEC, 6);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
if (long_dist)
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
else
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
if (long_dist) {
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 17);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 17);
coex_dm->auto_tdma_adjust = false;
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
} else {
- if ((btrssi_state == BTC_RSSI_STATE_LOW) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, false,
- true, 1);
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
- } else if ((btrssi_state == BTC_RSSI_STATE_MEDIUM) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, false,
- false, 1);
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 2);
- } else if ((btrssi_state == BTC_RSSI_STATE_HIGH) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, false,
- false, 1);
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 4);
+ if ((bt_rssi_state == BTC_RSSI_STATE_LOW) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
+ btc8192e2ant_tdma_duration_adjust(btcoexist, false,
+ true, 1);
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_MEDIUM) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
+ btc8192e2ant_tdma_duration_adjust(btcoexist, false,
+ false, 1);
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_tdma_duration_adjust(btcoexist, false,
+ false, 1);
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 4);
}
}
/* sw mechanism */
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
if (BTC_WIFI_BW_HT40 == wifi_bw) {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, true, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, true, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, false, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, false, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
-static void halbtc8192e2ant_action_a2dp_pan_hs(struct btc_coexist *btcoexist)
+static void btc8192e2ant_action_a2dp_pan_hs(struct btc_coexist *btcoexist)
{
- u8 wifirssi_state, btrssi_state = BTC_RSSI_STATE_HIGH;
+ u8 wifi_rssi_state, bt_rssi_state = BTC_RSSI_STATE_HIGH;
u32 wifi_bw;
- wifirssi_state = halbtc8192e2ant_wifirssi_state(btcoexist, 0, 2, 15, 0);
- btrssi_state = halbtc8192e2ant_btrssi_state(btcoexist, 3, 34, 42);
+ wifi_rssi_state = btc8192e2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8192e2ant_bt_rssi_state(btcoexist, 3, 34, 42);
- halbtc8192e2ant_switch_sstype(btcoexist, NORMAL_EXEC, 1);
- halbtc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 1);
+ btc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, NORMAL_EXEC, 6);
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
- if ((btrssi_state == BTC_RSSI_STATE_LOW) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, false, true, 2);
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
- } else if ((btrssi_state == BTC_RSSI_STATE_MEDIUM) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, false,
- false, 2);
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 2);
- } else if ((btrssi_state == BTC_RSSI_STATE_HIGH) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, false,
- false, 2);
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 4);
+ if ((bt_rssi_state == BTC_RSSI_STATE_LOW) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
+ btc8192e2ant_tdma_duration_adjust(btcoexist, false, true, 2);
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_MEDIUM) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
+ btc8192e2ant_tdma_duration_adjust(btcoexist, false, false, 2);
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_tdma_duration_adjust(btcoexist, false, false, 2);
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 4);
}
/* sw mechanism */
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
if (BTC_WIFI_BW_HT40 == wifi_bw) {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, true, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- true, 0x6);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ true, 0x6);
} else {
- btc8192e2ant_sw_mec1(btcoexist, true, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- true, 0x6);
+ btc8192e2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ true, 0x6);
}
} else {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, false, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- true, 0x6);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ true, 0x6);
} else {
- btc8192e2ant_sw_mec1(btcoexist, false, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- true, 0x6);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ true, 0x6);
}
}
}
-static void halbtc8192e2ant_action_pan_edr(struct btc_coexist *btcoexist)
+static void btc8192e2ant_action_pan_edr(struct btc_coexist *btcoexist)
{
- u8 wifirssi_state, btrssi_state = BTC_RSSI_STATE_HIGH;
+ u8 wifi_rssi_state, bt_rssi_state = BTC_RSSI_STATE_HIGH;
u32 wifi_bw;
- wifirssi_state = halbtc8192e2ant_wifirssi_state(btcoexist, 0, 2, 15, 0);
- btrssi_state = halbtc8192e2ant_btrssi_state(btcoexist, 3, 34, 42);
+ wifi_rssi_state = btc8192e2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8192e2ant_bt_rssi_state(btcoexist, 3, 34, 42);
- halbtc8192e2ant_switch_sstype(btcoexist, NORMAL_EXEC, 1);
- halbtc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 1);
+ btc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, NORMAL_EXEC, 6);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
- if ((btrssi_state == BTC_RSSI_STATE_LOW) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 5);
- } else if ((btrssi_state == BTC_RSSI_STATE_MEDIUM) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 2);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 1);
- } else if ((btrssi_state == BTC_RSSI_STATE_HIGH) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 4);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 1);
+ if ((bt_rssi_state == BTC_RSSI_STATE_LOW) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 5);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_MEDIUM) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 1);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 4);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 1);
}
/* sw mechanism */
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
if (BTC_WIFI_BW_HT40 == wifi_bw) {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, true, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, true, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, false, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, false, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
/* PAN(HS) only */
-static void halbtc8192e2ant_action_pan_hs(struct btc_coexist *btcoexist)
+static void btc8192e2ant_action_pan_hs(struct btc_coexist *btcoexist)
{
- u8 wifirssi_state, btrssi_state = BTC_RSSI_STATE_HIGH;
+ u8 wifi_rssi_state, bt_rssi_state = BTC_RSSI_STATE_HIGH;
u32 wifi_bw;
- wifirssi_state = halbtc8192e2ant_wifirssi_state(btcoexist, 0, 2, 15, 0);
- btrssi_state = halbtc8192e2ant_btrssi_state(btcoexist, 3, 34, 42);
+ wifi_rssi_state = btc8192e2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8192e2ant_bt_rssi_state(btcoexist, 3, 34, 42);
- halbtc8192e2ant_switch_sstype(btcoexist, NORMAL_EXEC, 1);
- halbtc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 1);
+ btc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, NORMAL_EXEC, 6);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
- if ((btrssi_state == BTC_RSSI_STATE_LOW) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
- } else if ((btrssi_state == BTC_RSSI_STATE_MEDIUM) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 2);
- } else if ((btrssi_state == BTC_RSSI_STATE_HIGH) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 4);
+ if ((bt_rssi_state == BTC_RSSI_STATE_LOW) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_MEDIUM) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 4);
}
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
if (BTC_WIFI_BW_HT40 == wifi_bw) {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, true, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, true, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, false, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, false, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
/* PAN(EDR)+A2DP */
-static void halbtc8192e2ant_action_pan_edr_a2dp(struct btc_coexist *btcoexist)
+static void btc8192e2ant_action_pan_edr_a2dp(struct btc_coexist *btcoexist)
{
- u8 wifirssi_state, btrssi_state = BTC_RSSI_STATE_HIGH;
+ u8 wifi_rssi_state, bt_rssi_state = BTC_RSSI_STATE_HIGH;
u32 wifi_bw;
- wifirssi_state = halbtc8192e2ant_wifirssi_state(btcoexist, 0, 2, 15, 0);
- btrssi_state = halbtc8192e2ant_btrssi_state(btcoexist, 3, 34, 42);
+ wifi_rssi_state = btc8192e2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8192e2ant_bt_rssi_state(btcoexist, 3, 34, 42);
- halbtc8192e2ant_switch_sstype(btcoexist, NORMAL_EXEC, 1);
- halbtc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 1);
+ btc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, NORMAL_EXEC, 6);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- if ((btrssi_state == BTC_RSSI_STATE_LOW) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, false, true, 3);
- } else if ((btrssi_state == BTC_RSSI_STATE_MEDIUM) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 2);
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, false,
- false, 3);
- } else if ((btrssi_state == BTC_RSSI_STATE_HIGH) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 4);
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, false,
- false, 3);
+ if ((bt_rssi_state == BTC_RSSI_STATE_LOW) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_tdma_duration_adjust(btcoexist, false, true, 3);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_MEDIUM) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_tdma_duration_adjust(btcoexist, false, false, 3);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 4);
+ btc8192e2ant_tdma_duration_adjust(btcoexist, false, false, 3);
}
/* sw mechanism */
if (BTC_WIFI_BW_HT40 == wifi_bw) {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, true, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, true, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, false, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, false, false,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
-static void halbtc8192e2ant_action_pan_edr_hid(struct btc_coexist *btcoexist)
+static void btc8192e2ant_action_pan_edr_hid(struct btc_coexist *btcoexist)
{
- u8 wifirssi_state, btrssi_state = BTC_RSSI_STATE_HIGH;
+ u8 wifi_rssi_state, bt_rssi_state = BTC_RSSI_STATE_HIGH;
u32 wifi_bw;
- wifirssi_state = halbtc8192e2ant_wifirssi_state(btcoexist, 0, 2, 15, 0);
- btrssi_state = halbtc8192e2ant_btrssi_state(btcoexist, 3, 34, 42);
+ wifi_rssi_state = btc8192e2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8192e2ant_bt_rssi_state(btcoexist, 3, 34, 42);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- halbtc8192e2ant_switch_sstype(btcoexist, NORMAL_EXEC, 1);
- halbtc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 1);
+ btc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, NORMAL_EXEC, 6);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 3);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 3);
- if ((btrssi_state == BTC_RSSI_STATE_LOW) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
- } else if ((btrssi_state == BTC_RSSI_STATE_MEDIUM) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 2);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- } else if ((btrssi_state == BTC_RSSI_STATE_HIGH) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 4);
- halbtc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
+ if ((bt_rssi_state == BTC_RSSI_STATE_LOW) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_MEDIUM) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 10);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 4);
+ btc8192e2ant_ps_tdma(btcoexist, NORMAL_EXEC,
+ true, 10);
}
/* sw mechanism */
if (BTC_WIFI_BW_HT40 == wifi_bw) {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, true, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, true, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, false, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, false, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
@@ -2942,125 +2348,125 @@
/* HID+A2DP+PAN(EDR) */
static void btc8192e2ant_action_hid_a2dp_pan_edr(struct btc_coexist *btcoexist)
{
- u8 wifirssi_state, btrssi_state = BTC_RSSI_STATE_HIGH;
+ u8 wifi_rssi_state, bt_rssi_state = BTC_RSSI_STATE_HIGH;
u32 wifi_bw;
- wifirssi_state = halbtc8192e2ant_wifirssi_state(btcoexist, 0, 2, 15, 0);
- btrssi_state = halbtc8192e2ant_btrssi_state(btcoexist, 3, 34, 42);
+ wifi_rssi_state = btc8192e2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8192e2ant_bt_rssi_state(btcoexist, 3, 34, 42);
- halbtc8192e2ant_switch_sstype(btcoexist, NORMAL_EXEC, 1);
- halbtc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 1);
+ btc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
- halbtc8192e2ant_fw_dac_swinglvl(btcoexist, NORMAL_EXEC, 6);
+ btc8192e2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 3);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 3);
- if ((btrssi_state == BTC_RSSI_STATE_LOW) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, true, true, 3);
- } else if ((btrssi_state == BTC_RSSI_STATE_MEDIUM) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 2);
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, true, false, 3);
- } else if ((btrssi_state == BTC_RSSI_STATE_HIGH) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 4);
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, true, false, 3);
+ if ((bt_rssi_state == BTC_RSSI_STATE_LOW) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_tdma_duration_adjust(btcoexist, true, true, 3);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_MEDIUM) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_tdma_duration_adjust(btcoexist, true, false, 3);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 4);
+ btc8192e2ant_tdma_duration_adjust(btcoexist, true, false, 3);
}
/* sw mechanism */
if (BTC_WIFI_BW_HT40 == wifi_bw) {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, true, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, true, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, false, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, false, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
-static void halbtc8192e2ant_action_hid_a2dp(struct btc_coexist *btcoexist)
+static void btc8192e2ant_action_hid_a2dp(struct btc_coexist *btcoexist)
{
- u8 wifirssi_state, btrssi_state = BTC_RSSI_STATE_HIGH;
+ u8 wifi_rssi_state, bt_rssi_state = BTC_RSSI_STATE_HIGH;
u32 wifi_bw;
- wifirssi_state = halbtc8192e2ant_wifirssi_state(btcoexist, 0, 2, 15, 0);
- btrssi_state = halbtc8192e2ant_btrssi_state(btcoexist, 3, 34, 42);
+ wifi_rssi_state = btc8192e2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8192e2ant_bt_rssi_state(btcoexist, 3, 34, 42);
- halbtc8192e2ant_switch_sstype(btcoexist, NORMAL_EXEC, 1);
- halbtc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8192e2ant_switch_ss_type(btcoexist, NORMAL_EXEC, 1);
+ btc8192e2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- btc8192e2ant_coex_tbl_w_type(btcoexist, NORMAL_EXEC, 3);
+ btc8192e2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 3);
- if ((btrssi_state == BTC_RSSI_STATE_LOW) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 0);
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, true, true, 2);
- } else if ((btrssi_state == BTC_RSSI_STATE_MEDIUM) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 2);
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, true, false, 2);
- } else if ((btrssi_state == BTC_RSSI_STATE_HIGH) ||
- (btrssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8192e2ant_dec_btpwr(btcoexist, NORMAL_EXEC, 4);
- halbtc8192e2ant_tdma_duration_adjust(btcoexist, true, false, 2);
+ if ((bt_rssi_state == BTC_RSSI_STATE_LOW) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ btc8192e2ant_tdma_duration_adjust(btcoexist, true, true, 2);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_MEDIUM) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_MEDIUM)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ btc8192e2ant_tdma_duration_adjust(btcoexist, true, false, 2);
+ } else if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 4);
+ btc8192e2ant_tdma_duration_adjust(btcoexist, true, false, 2);
}
/* sw mechanism */
if (BTC_WIFI_BW_HT40 == wifi_bw) {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, true, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, true, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- if ((wifirssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifirssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8192e2ant_sw_mec1(btcoexist, false, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, true, false,
- false, 0x18);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8192e2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8192e2ant_sw_mec1(btcoexist, false, true,
- false, false);
- btc8192e2ant_sw_mec2(btcoexist, false, false,
- false, 0x18);
+ btc8192e2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8192e2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
-static void halbtc8192e2ant_run_coexist_mechanism(struct btc_coexist *btcoexist)
+static void btc8192e2ant_run_coexist_mechanism(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 algorithm = 0;
@@ -3080,12 +2486,12 @@
return;
}
- algorithm = halbtc8192e2ant_action_algorithm(btcoexist);
+ algorithm = btc8192e2ant_action_algorithm(btcoexist);
if (coex_sta->c2h_bt_inquiry_page &&
(BT_8192E_2ANT_COEX_ALGO_PANHS != algorithm)) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], BT is under inquiry/page scan !!\n");
- halbtc8192e2ant_action_bt_inquiry(btcoexist);
+ btc8192e2ant_action_bt_inquiry(btcoexist);
return;
}
@@ -3093,7 +2499,7 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Algorithm = %d\n", coex_dm->cur_algorithm);
- if (halbtc8192e2ant_is_common_action(btcoexist)) {
+ if (btc8192e2ant_is_common_action(btcoexist)) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action 2-Ant common\n");
coex_dm->auto_tdma_adjust = false;
@@ -3109,47 +2515,47 @@
case BT_8192E_2ANT_COEX_ALGO_SCO:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Action 2-Ant, algorithm = SCO\n");
- halbtc8192e2ant_action_sco(btcoexist);
+ btc8192e2ant_action_sco(btcoexist);
break;
case BT_8192E_2ANT_COEX_ALGO_SCO_PAN:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Action 2-Ant, algorithm = SCO+PAN(EDR)\n");
- halbtc8192e2ant_action_sco_pan(btcoexist);
+ btc8192e2ant_action_sco_pan(btcoexist);
break;
case BT_8192E_2ANT_COEX_ALGO_HID:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Action 2-Ant, algorithm = HID\n");
- halbtc8192e2ant_action_hid(btcoexist);
+ btc8192e2ant_action_hid(btcoexist);
break;
case BT_8192E_2ANT_COEX_ALGO_A2DP:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Action 2-Ant, algorithm = A2DP\n");
- halbtc8192e2ant_action_a2dp(btcoexist);
+ btc8192e2ant_action_a2dp(btcoexist);
break;
case BT_8192E_2ANT_COEX_ALGO_A2DP_PANHS:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Action 2-Ant, algorithm = A2DP+PAN(HS)\n");
- halbtc8192e2ant_action_a2dp_pan_hs(btcoexist);
+ btc8192e2ant_action_a2dp_pan_hs(btcoexist);
break;
case BT_8192E_2ANT_COEX_ALGO_PANEDR:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Action 2-Ant, algorithm = PAN(EDR)\n");
- halbtc8192e2ant_action_pan_edr(btcoexist);
+ btc8192e2ant_action_pan_edr(btcoexist);
break;
case BT_8192E_2ANT_COEX_ALGO_PANHS:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Action 2-Ant, algorithm = HS mode\n");
- halbtc8192e2ant_action_pan_hs(btcoexist);
+ btc8192e2ant_action_pan_hs(btcoexist);
break;
case BT_8192E_2ANT_COEX_ALGO_PANEDR_A2DP:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Action 2-Ant, algorithm = PAN+A2DP\n");
- halbtc8192e2ant_action_pan_edr_a2dp(btcoexist);
+ btc8192e2ant_action_pan_edr_a2dp(btcoexist);
break;
case BT_8192E_2ANT_COEX_ALGO_PANEDR_HID:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Action 2-Ant, algorithm = PAN(EDR)+HID\n");
- halbtc8192e2ant_action_pan_edr_hid(btcoexist);
+ btc8192e2ant_action_pan_edr_hid(btcoexist);
break;
case BT_8192E_2ANT_COEX_ALGO_HID_A2DP_PANEDR:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -3159,20 +2565,20 @@
case BT_8192E_2ANT_COEX_ALGO_HID_A2DP:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Action 2-Ant, algorithm = HID+A2DP\n");
- halbtc8192e2ant_action_hid_a2dp(btcoexist);
+ btc8192e2ant_action_hid_a2dp(btcoexist);
break;
default:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"Action 2-Ant, algorithm = unknown!!\n");
- /* halbtc8192e2ant_coex_alloff(btcoexist); */
+ /* btc8192e2ant_coex_all_off(btcoexist); */
break;
}
coex_dm->pre_algorithm = coex_dm->cur_algorithm;
}
}
-static void halbtc8192e2ant_init_hwconfig(struct btc_coexist *btcoexist,
- bool backup)
+static void btc8192e2ant_init_hwconfig(struct btc_coexist *btcoexist,
+ bool backup)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u16 u16tmp = 0;
@@ -3191,7 +2597,7 @@
0x430);
coex_dm->backup_arfr_cnt2 = btcoexist->btc_read_4byte(btcoexist,
0x434);
- coex_dm->backup_retrylimit = btcoexist->btc_read_2byte(
+ coex_dm->backup_retry_limit = btcoexist->btc_read_2byte(
btcoexist,
0x42a);
coex_dm->backup_ampdu_maxtime = btcoexist->btc_read_1byte(
@@ -3209,7 +2615,7 @@
else
btcoexist->btc_write_4byte(btcoexist, 0x64, 0x30030004);
- btc8192e2ant_coex_tbl_w_type(btcoexist, FORCE_EXEC, 0);
+ btc8192e2ant_coex_table_with_type(btcoexist, FORCE_EXEC, 0);
/* antenna switch control parameter */
btcoexist->btc_write_4byte(btcoexist, 0x858, 0x55555555);
@@ -3232,7 +2638,7 @@
u16tmp |= BIT9;
btcoexist->btc_write_2byte(btcoexist, 0x40, u16tmp);
- /* enable PTA I2C mailbox */
+ /* enable PTA I2C mailbox */
u8tmp = btcoexist->btc_read_1byte(btcoexist, 0x101);
u8tmp |= BIT4;
btcoexist->btc_write_1byte(btcoexist, 0x101, u8tmp);
@@ -3247,29 +2653,25 @@
btcoexist->btc_write_1byte(btcoexist, 0x7, u8tmp);
}
-/*************************************************************
- * work around function start with wa_halbtc8192e2ant_
- *************************************************************/
-
/************************************************************
- * extern function start with EXhalbtc8192e2ant_
+ * extern function start with ex_btc8192e2ant_
************************************************************/
-void ex_halbtc8192e2ant_init_hwconfig(struct btc_coexist *btcoexist)
+void ex_btc8192e2ant_init_hwconfig(struct btc_coexist *btcoexist)
{
- halbtc8192e2ant_init_hwconfig(btcoexist, true);
+ btc8192e2ant_init_hwconfig(btcoexist, true);
}
-void ex_halbtc8192e2ant_init_coex_dm(struct btc_coexist *btcoexist)
+void ex_btc8192e2ant_init_coex_dm(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Coex Mechanism Init!!\n");
- halbtc8192e2ant_init_coex_dm(btcoexist);
+ btc8192e2ant_init_coex_dm(btcoexist);
}
-void ex_halbtc8192e2ant_display_coex_info(struct btc_coexist *btcoexist)
+void ex_btc8192e2ant_display_coex_info(struct btc_coexist *btcoexist)
{
struct btc_board_info *board_info = &btcoexist->board_info;
struct btc_stack_info *stack_info = &btcoexist->stack_info;
@@ -3278,8 +2680,8 @@
u16 u16tmp[4];
u32 u32tmp[4];
bool roam = false, scan = false, link = false, wifi_under_5g = false;
- bool bt_hson = false, wifi_busy = false;
- int wifirssi = 0, bt_hs_rssi = 0;
+ bool bt_hs_on = false, wifi_busy = false;
+ int wifi_rssi = 0, bt_hs_rssi = 0;
u32 wifi_bw, wifi_traffic_dir;
u8 wifi_dot11_chnl, wifi_hs_chnl;
u32 fw_ver = 0, bt_patch_ver = 0;
@@ -3316,21 +2718,21 @@
glcoex_ver_date_8192e_2ant, glcoex_ver_8192e_2ant,
fw_ver, bt_patch_ver, bt_patch_ver);
- btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hson);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
btcoexist->btc_get(btcoexist, BTC_GET_U1_WIFI_DOT11_CHNL,
&wifi_dot11_chnl);
btcoexist->btc_get(btcoexist, BTC_GET_U1_WIFI_HS_CHNL, &wifi_hs_chnl);
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG, "\r\n %-35s = %d / %d(%d)",
"Dot11 channel / HsMode(HsChnl)",
- wifi_dot11_chnl, bt_hson, wifi_hs_chnl);
+ wifi_dot11_chnl, bt_hs_on, wifi_hs_chnl);
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG, "\r\n %-35s = %3ph ",
"H2C Wifi inform bt chnl Info", coex_dm->wifi_chnl_info);
- btcoexist->btc_get(btcoexist, BTC_GET_S4_WIFI_RSSI, &wifirssi);
+ btcoexist->btc_get(btcoexist, BTC_GET_S4_WIFI_RSSI, &wifi_rssi);
btcoexist->btc_get(btcoexist, BTC_GET_S4_HS_RSSI, &bt_hs_rssi);
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG, "\r\n %-35s = %d/ %d",
- "Wifi rssi/ HS rssi", wifirssi, bt_hs_rssi);
+ "Wifi rssi/ HS rssi", wifi_rssi, bt_hs_rssi);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_SCAN, &scan);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_LINK, &link);
@@ -3377,7 +2779,7 @@
if (coex_sta->bt_info_c2h_cnt[i]) {
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG,
"\r\n %-35s = %7ph(%d)",
- GLBtInfoSrc8192e2Ant[i],
+ glbt_info_src_8192e_2ant[i],
coex_sta->bt_info_c2h[i],
coex_sta->bt_info_c2h_cnt[i]);
}
@@ -3390,7 +2792,7 @@
btcoexist->btc_disp_dbg_msg(btcoexist, BTC_DBG_DISP_FW_PWR_MODE_CMD);
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG, "\r\n %-35s = 0x%x ", "SS Type",
- coex_dm->cur_sstype);
+ coex_dm->cur_ss_type);
/* Sw mechanism */
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG, "\r\n %-35s",
@@ -3429,7 +2831,7 @@
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG, "\r\n %-35s = 0x%x/0x%x/0x%x/0x%x",
"backup ARFR1/ARFR2/RL/AMaxTime", coex_dm->backup_arfr_cnt1,
- coex_dm->backup_arfr_cnt2, coex_dm->backup_retrylimit,
+ coex_dm->backup_arfr_cnt2, coex_dm->backup_retry_limit,
coex_dm->backup_ampdu_maxtime);
u32tmp[0] = btcoexist->btc_read_4byte(btcoexist, 0x430);
@@ -3485,12 +2887,12 @@
"0x774(lp rx[31:16]/tx[15:0])",
coex_sta->low_priority_rx, coex_sta->low_priority_tx);
#if (BT_AUTO_REPORT_ONLY_8192E_2ANT == 1)
- halbtc8192e2ant_monitor_bt_ctr(btcoexist);
+ btc8192e2ant_monitor_bt_ctr(btcoexist);
#endif
btcoexist->btc_disp_dbg_msg(btcoexist, BTC_DBG_DISP_COEX_STATISTICS);
}
-void ex_halbtc8192e2ant_ips_notify(struct btc_coexist *btcoexist, u8 type)
+void ex_btc8192e2ant_ips_notify(struct btc_coexist *btcoexist, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -3498,7 +2900,7 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], IPS ENTER notify\n");
coex_sta->under_ips = true;
- halbtc8192e2ant_coex_alloff(btcoexist);
+ btc8192e2ant_coex_all_off(btcoexist);
} else if (BTC_IPS_LEAVE == type) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], IPS LEAVE notify\n");
@@ -3506,7 +2908,7 @@
}
}
-void ex_halbtc8192e2ant_lps_notify(struct btc_coexist *btcoexist, u8 type)
+void ex_btc8192e2ant_lps_notify(struct btc_coexist *btcoexist, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -3521,7 +2923,7 @@
}
}
-void ex_halbtc8192e2ant_scan_notify(struct btc_coexist *btcoexist, u8 type)
+void ex_btc8192e2ant_scan_notify(struct btc_coexist *btcoexist, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -3533,7 +2935,7 @@
"[BTCoex], SCAN FINISH notify\n");
}
-void ex_halbtc8192e2ant_connect_notify(struct btc_coexist *btcoexist, u8 type)
+void ex_btc8192e2ant_connect_notify(struct btc_coexist *btcoexist, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -3545,8 +2947,8 @@
"[BTCoex], CONNECT FINISH notify\n");
}
-void ex_halbtc8192e2ant_media_status_notify(struct btc_coexist *btcoexist,
- u8 type)
+void ex_btc8192e2ant_media_status_notify(struct btc_coexist *btcoexist,
+ u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[3] = {0};
@@ -3591,8 +2993,8 @@
btcoexist->btc_fill_h2c(btcoexist, 0x66, 3, h2c_parameter);
}
-void ex_halbtc8192e2ant_special_packet_notify(struct btc_coexist *btcoexist,
- u8 type)
+void ex_btc8192e2ant_special_packet_notify(struct btc_coexist *btcoexist,
+ u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -3601,8 +3003,8 @@
"[BTCoex], DHCP Packet notify\n");
}
-void ex_halbtc8192e2ant_bt_info_notify(struct btc_coexist *btcoexist,
- u8 *tmp_buf, u8 length)
+void ex_btc8192e2ant_bt_info_notify(struct btc_coexist *btcoexist,
+ u8 *tmp_buf, u8 length)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 bt_info = 0;
@@ -3633,7 +3035,8 @@
}
if (BT_INFO_SRC_8192E_2ANT_WIFI_FW != rsp_source) {
- coex_sta->bt_retry_cnt = /* [3:0] */
+ /* [3:0] */
+ coex_sta->bt_retry_cnt =
coex_sta->bt_info_c2h[rsp_source][2] & 0xf;
coex_sta->bt_rssi =
@@ -3651,11 +3054,11 @@
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
&wifi_connected);
if (wifi_connected)
- ex_halbtc8192e2ant_media_status_notify(
+ ex_btc8192e2ant_media_status_notify(
btcoexist,
BTC_MEDIA_CONNECT);
else
- ex_halbtc8192e2ant_media_status_notify(
+ ex_btc8192e2ant_media_status_notify(
btcoexist,
BTC_MEDIA_DISCONNECT);
}
@@ -3665,9 +3068,9 @@
!btcoexist->stop_coex_dm) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"bit3, BT NOT ignore Wlan active!\n");
- halbtc8192e2ant_IgnoreWlanAct(btcoexist,
- FORCE_EXEC,
- false);
+ btc8192e2ant_ignore_wlan_act(btcoexist,
+ FORCE_EXEC,
+ false);
}
} else {
/* BT already NOT ignore Wlan active,
@@ -3679,8 +3082,8 @@
if ((coex_sta->bt_info_ext & BIT4)) {
/* BT auto report already enabled, do nothing */
} else {
- halbtc8192e2ant_bt_autoreport(btcoexist, FORCE_EXEC,
- true);
+ btc8192e2ant_bt_auto_report(btcoexist, FORCE_EXEC,
+ true);
}
#endif
}
@@ -3718,9 +3121,9 @@
coex_sta->sco_exist = false;
}
- halbtc8192e2ant_update_btlink_info(btcoexist);
+ btc8192e2ant_update_bt_link_info(btcoexist);
- if (!(bt_info&BT_INFO_8192E_2ANT_B_CONNECTION)) {
+ if (!(bt_info & BT_INFO_8192E_2ANT_B_CONNECTION)) {
coex_dm->bt_status = BT_8192E_2ANT_BT_STATUS_NON_CONNECTED_IDLE;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], BT Non-Connected idle!!!\n");
@@ -3728,12 +3131,12 @@
coex_dm->bt_status = BT_8192E_2ANT_BT_STATUS_CONNECTED_IDLE;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], bt_infoNotify(), BT Connected-idle!!!\n");
- } else if ((bt_info&BT_INFO_8192E_2ANT_B_SCO_ESCO) ||
- (bt_info&BT_INFO_8192E_2ANT_B_SCO_BUSY)) {
+ } else if ((bt_info & BT_INFO_8192E_2ANT_B_SCO_ESCO) ||
+ (bt_info & BT_INFO_8192E_2ANT_B_SCO_BUSY)) {
coex_dm->bt_status = BT_8192E_2ANT_BT_STATUS_SCO_BUSY;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], bt_infoNotify(), BT SCO busy!!!\n");
- } else if (bt_info&BT_INFO_8192E_2ANT_B_ACL_BUSY) {
+ } else if (bt_info & BT_INFO_8192E_2ANT_B_ACL_BUSY) {
coex_dm->bt_status = BT_8192E_2ANT_BT_STATUS_ACL_BUSY;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], bt_infoNotify(), BT ACL busy!!!\n");
@@ -3758,12 +3161,7 @@
coex_dm->limited_dig = limited_dig;
btcoexist->btc_set(btcoexist, BTC_SET_BL_BT_LIMITED_DIG, &limited_dig);
- halbtc8192e2ant_run_coexist_mechanism(btcoexist);
-}
-
-void ex_halbtc8192e2ant_stack_operation_notify(struct btc_coexist *btcoexist,
- u8 type)
-{
+ btc8192e2ant_run_coexist_mechanism(btcoexist);
}
void ex_halbtc8192e2ant_halt_notify(struct btc_coexist *btcoexist)
@@ -3772,11 +3170,11 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD, "[BTCoex], Halt notify\n");
- halbtc8192e2ant_IgnoreWlanAct(btcoexist, FORCE_EXEC, true);
- ex_halbtc8192e2ant_media_status_notify(btcoexist, BTC_MEDIA_DISCONNECT);
+ btc8192e2ant_ignore_wlan_act(btcoexist, FORCE_EXEC, true);
+ ex_btc8192e2ant_media_status_notify(btcoexist, BTC_MEDIA_DISCONNECT);
}
-void ex_halbtc8192e2ant_periodical(struct btc_coexist *btcoexist)
+void ex_btc8192e2ant_periodical(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
static u8 dis_ver_info_cnt;
@@ -3810,12 +3208,12 @@
}
#if (BT_AUTO_REPORT_ONLY_8192E_2ANT == 0)
- halbtc8192e2ant_querybt_info(btcoexist);
- halbtc8192e2ant_monitor_bt_ctr(btcoexist);
- btc8192e2ant_monitor_bt_enable_dis(btcoexist);
+ btc8192e2ant_query_bt_info(btcoexist);
+ btc8192e2ant_monitor_bt_ctr(btcoexist);
+ btc8192e2ant_monitor_bt_enable_disable(btcoexist);
#else
- if (halbtc8192e2ant_iswifi_status_changed(btcoexist) ||
+ if (btc8192e2ant_is_wifi_status_changed(btcoexist) ||
coex_dm->auto_tdma_adjust)
- halbtc8192e2ant_run_coexist_mechanism(btcoexist);
+ btc8192e2ant_run_coexist_mechanism(btcoexist);
#endif
}
diff --git a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8192e2ant.h b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8192e2ant.h
index 75e1f7d..fc0fa87 100644
--- a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8192e2ant.h
+++ b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8192e2ant.h
@@ -116,7 +116,7 @@
u32 backup_arfr_cnt1; /* Auto Rate Fallback Retry cnt */
u32 backup_arfr_cnt2; /* Auto Rate Fallback Retry cnt */
- u16 backup_retrylimit;
+ u16 backup_retry_limit;
u8 backup_ampdu_maxtime;
/* algorithm related */
@@ -125,18 +125,18 @@
u8 bt_status;
u8 wifi_chnl_info[3];
- u8 pre_sstype;
- u8 cur_sstype;
+ u8 pre_ss_type;
+ u8 cur_ss_type;
- u32 prera_mask;
- u32 curra_mask;
- u8 curra_masktype;
- u8 pre_arfrtype;
- u8 cur_arfrtype;
- u8 pre_retrylimit_type;
- u8 cur_retrylimit_type;
- u8 pre_ampdutime_type;
- u8 cur_ampdutime_type;
+ u32 pre_ra_mask;
+ u32 cur_ra_mask;
+ u8 cur_ra_mask_type;
+ u8 pre_arfr_type;
+ u8 cur_arfr_type;
+ u8 pre_retry_limit_type;
+ u8 cur_retry_limit_type;
+ u8 pre_ampdu_time_type;
+ u8 cur_ampdu_time_type;
};
struct coex_sta_8192e_2ant {
diff --git a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8723b1ant.c b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8723b1ant.c
index d67bbfb..2003c8c 100644
--- a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8723b1ant.c
+++ b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8723b1ant.c
@@ -45,7 +45,7 @@
static struct coex_sta_8723b_1ant glcoex_sta_8723b_1ant;
static struct coex_sta_8723b_1ant *coex_sta = &glcoex_sta_8723b_1ant;
-static const char *const GLBtInfoSrc8723b1Ant[] = {
+static const char *const glbt_info_src_8723b_1ant[] = {
"BT Info[wifi fw]",
"BT Info[bt rsp]",
"BT Info[bt auto report]",
@@ -60,188 +60,6 @@
/***************************************************************
* local function start with halbtc8723b1ant_
***************************************************************/
-static u8 halbtc8723b1ant_bt_rssi_state(struct btc_coexist *btcoexist,
- u8 level_num, u8 rssi_thresh,
- u8 rssi_thresh1)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- s32 bt_rssi = 0;
- u8 bt_rssi_state = coex_sta->pre_bt_rssi_state;
-
- bt_rssi = coex_sta->bt_rssi;
-
- if (level_num == 2) {
- if ((coex_sta->pre_bt_rssi_state == BTC_RSSI_STATE_LOW) ||
- (coex_sta->pre_bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- if (bt_rssi >= rssi_thresh +
- BTC_RSSI_COEX_THRESH_TOL_8723B_1ANT) {
- bt_rssi_state = BTC_RSSI_STATE_HIGH;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Rssi state switch to High\n");
- } else {
- bt_rssi_state = BTC_RSSI_STATE_STAY_LOW;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Rssi state stay at Low\n");
- }
- } else {
- if (bt_rssi < rssi_thresh) {
- bt_rssi_state = BTC_RSSI_STATE_LOW;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Rssi state switch to Low\n");
- } else {
- bt_rssi_state = BTC_RSSI_STATE_STAY_HIGH;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Rssi state stay at High\n");
- }
- }
- } else if (level_num == 3) {
- if (rssi_thresh > rssi_thresh1) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Rssi thresh error!!\n");
- return coex_sta->pre_bt_rssi_state;
- }
-
- if ((coex_sta->pre_bt_rssi_state == BTC_RSSI_STATE_LOW) ||
- (coex_sta->pre_bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- if (bt_rssi >= rssi_thresh +
- BTC_RSSI_COEX_THRESH_TOL_8723B_1ANT) {
- bt_rssi_state = BTC_RSSI_STATE_MEDIUM;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Rssi state switch to Medium\n");
- } else {
- bt_rssi_state = BTC_RSSI_STATE_STAY_LOW;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Rssi state stay at Low\n");
- }
- } else if ((coex_sta->pre_bt_rssi_state ==
- BTC_RSSI_STATE_MEDIUM) ||
- (coex_sta->pre_bt_rssi_state ==
- BTC_RSSI_STATE_STAY_MEDIUM)) {
- if (bt_rssi >= rssi_thresh1 +
- BTC_RSSI_COEX_THRESH_TOL_8723B_1ANT) {
- bt_rssi_state = BTC_RSSI_STATE_HIGH;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Rssi state switch to High\n");
- } else if (bt_rssi < rssi_thresh) {
- bt_rssi_state = BTC_RSSI_STATE_LOW;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Rssi state switch to Low\n");
- } else {
- bt_rssi_state = BTC_RSSI_STATE_STAY_MEDIUM;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Rssi state stay at Medium\n");
- }
- } else {
- if (bt_rssi < rssi_thresh1) {
- bt_rssi_state = BTC_RSSI_STATE_MEDIUM;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Rssi state switch to Medium\n");
- } else {
- bt_rssi_state = BTC_RSSI_STATE_STAY_HIGH;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Rssi state stay at High\n");
- }
- }
- }
-
- coex_sta->pre_bt_rssi_state = bt_rssi_state;
-
- return bt_rssi_state;
-}
-
-static u8 halbtc8723b1ant_wifi_rssi_state(struct btc_coexist *btcoexist,
- u8 index, u8 level_num,
- u8 rssi_thresh, u8 rssi_thresh1)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- s32 wifi_rssi = 0;
- u8 wifi_rssi_state = coex_sta->pre_wifi_rssi_state[index];
-
- btcoexist->btc_get(btcoexist,
- BTC_GET_S4_WIFI_RSSI, &wifi_rssi);
-
- if (level_num == 2) {
- if ((coex_sta->pre_wifi_rssi_state[index] ==
- BTC_RSSI_STATE_LOW) ||
- (coex_sta->pre_wifi_rssi_state[index] ==
- BTC_RSSI_STATE_STAY_LOW)) {
- if (wifi_rssi >= rssi_thresh +
- BTC_RSSI_COEX_THRESH_TOL_8723B_1ANT) {
- wifi_rssi_state = BTC_RSSI_STATE_HIGH;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], wifi RSSI state switch to High\n");
- } else {
- wifi_rssi_state = BTC_RSSI_STATE_STAY_LOW;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], wifi RSSI state stay at Low\n");
- }
- } else {
- if (wifi_rssi < rssi_thresh) {
- wifi_rssi_state = BTC_RSSI_STATE_LOW;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], wifi RSSI state switch to Low\n");
- } else {
- wifi_rssi_state = BTC_RSSI_STATE_STAY_HIGH;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], wifi RSSI state stay at High\n");
- }
- }
- } else if (level_num == 3) {
- if (rssi_thresh > rssi_thresh1) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], wifi RSSI thresh error!!\n");
- return coex_sta->pre_wifi_rssi_state[index];
- }
-
- if ((coex_sta->pre_wifi_rssi_state[index] ==
- BTC_RSSI_STATE_LOW) ||
- (coex_sta->pre_wifi_rssi_state[index] ==
- BTC_RSSI_STATE_STAY_LOW)) {
- if (wifi_rssi >= rssi_thresh +
- BTC_RSSI_COEX_THRESH_TOL_8723B_1ANT) {
- wifi_rssi_state = BTC_RSSI_STATE_MEDIUM;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], wifi RSSI state switch to Medium\n");
- } else {
- wifi_rssi_state = BTC_RSSI_STATE_STAY_LOW;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], wifi RSSI state stay at Low\n");
- }
- } else if ((coex_sta->pre_wifi_rssi_state[index] ==
- BTC_RSSI_STATE_MEDIUM) ||
- (coex_sta->pre_wifi_rssi_state[index] ==
- BTC_RSSI_STATE_STAY_MEDIUM)) {
- if (wifi_rssi >= rssi_thresh1 +
- BTC_RSSI_COEX_THRESH_TOL_8723B_1ANT) {
- wifi_rssi_state = BTC_RSSI_STATE_HIGH;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], wifi RSSI state switch to High\n");
- } else if (wifi_rssi < rssi_thresh) {
- wifi_rssi_state = BTC_RSSI_STATE_LOW;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], wifi RSSI state switch to Low\n");
- } else {
- wifi_rssi_state = BTC_RSSI_STATE_STAY_MEDIUM;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], wifi RSSI state stay at Medium\n");
- }
- } else {
- if (wifi_rssi < rssi_thresh1) {
- wifi_rssi_state = BTC_RSSI_STATE_MEDIUM;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], wifi RSSI state switch to Medium\n");
- } else {
- wifi_rssi_state = BTC_RSSI_STATE_STAY_HIGH;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], wifi RSSI state stay at High\n");
- }
- }
- }
-
- coex_sta->pre_wifi_rssi_state[index] = wifi_rssi_state;
-
- return wifi_rssi_state;
-}
static void halbtc8723b1ant_updatera_mask(struct btc_coexist *btcoexist,
bool force_exec, u32 dis_rate_mask)
@@ -249,7 +67,7 @@
coex_dm->curra_mask = dis_rate_mask;
if (force_exec || (coex_dm->prera_mask != coex_dm->curra_mask))
- btcoexist->btc_set(btcoexist, BTC_SET_ACT_UPDATE_ra_mask,
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_UPDATE_RAMASK,
&coex_dm->curra_mask);
coex_dm->prera_mask = coex_dm->curra_mask;
@@ -326,15 +144,14 @@
coex_dm->cur_ampdu_time_type)) {
switch (coex_dm->cur_ampdu_time_type) {
case 0: /* normal mode */
- btcoexist->btc_write_1byte(btcoexist, 0x456,
- coex_dm->backup_ampdu_max_time);
- break;
+ btcoexist->btc_write_1byte(btcoexist, 0x456,
+ coex_dm->backup_ampdu_max_time);
+ break;
case 1: /* AMPDU timw = 0x38 * 32us */
- btcoexist->btc_write_1byte(btcoexist,
- 0x456, 0x38);
- break;
+ btcoexist->btc_write_1byte(btcoexist, 0x456, 0x38);
+ break;
default:
- break;
+ break;
}
}
@@ -354,7 +171,7 @@
halbtc8723b1ant_updatera_mask(btcoexist, force_exec,
0x00000003);
break;
- /* disable cck 1/2/5.5, ofdm 6/9/12/18/24, mcs 0/1/2/3/4*/
+ /* disable cck 1/2/5.5, ofdm 6/9/12/18/24, mcs 0/1/2/3/4 */
case 2:
halbtc8723b1ant_updatera_mask(btcoexist, force_exec,
0x0001f1f7);
@@ -426,7 +243,8 @@
coex_sta->c2h_bt_info_req_sent = true;
- h2c_parameter[0] |= BIT0; /* trigger*/
+ /* trigger */
+ h2c_parameter[0] |= BIT0;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Query Bt Info, FW write 0x61 = 0x%x\n",
@@ -515,202 +333,6 @@
bt_link_info->hid_only = false;
}
-static u8 halbtc8723b1ant_action_algorithm(struct btc_coexist *btcoexist)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
- bool bt_hs_on = false;
- u8 algorithm = BT_8723B_1ANT_COEX_ALGO_UNDEFINED;
- u8 numdiffprofile = 0;
-
- btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
-
- if (!bt_link_info->bt_link_exist) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], No BT link exists!!!\n");
- return algorithm;
- }
-
- if (bt_link_info->sco_exist)
- numdiffprofile++;
- if (bt_link_info->hid_exist)
- numdiffprofile++;
- if (bt_link_info->pan_exist)
- numdiffprofile++;
- if (bt_link_info->a2dp_exist)
- numdiffprofile++;
-
- if (numdiffprofile == 1) {
- if (bt_link_info->sco_exist) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Profile = SCO only\n");
- algorithm = BT_8723B_1ANT_COEX_ALGO_SCO;
- } else {
- if (bt_link_info->hid_exist) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Profile = HID only\n");
- algorithm = BT_8723B_1ANT_COEX_ALGO_HID;
- } else if (bt_link_info->a2dp_exist) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Profile = A2DP only\n");
- algorithm = BT_8723B_1ANT_COEX_ALGO_A2DP;
- } else if (bt_link_info->pan_exist) {
- if (bt_hs_on) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = PAN(HS) only\n");
- algorithm =
- BT_8723B_1ANT_COEX_ALGO_PANHS;
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = PAN(EDR) only\n");
- algorithm =
- BT_8723B_1ANT_COEX_ALGO_PANEDR;
- }
- }
- }
- } else if (numdiffprofile == 2) {
- if (bt_link_info->sco_exist) {
- if (bt_link_info->hid_exist) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Profile = SCO + HID\n");
- algorithm = BT_8723B_1ANT_COEX_ALGO_HID;
- } else if (bt_link_info->a2dp_exist) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Profile = SCO + A2DP ==> SCO\n");
- algorithm = BT_8723B_1ANT_COEX_ALGO_SCO;
- } else if (bt_link_info->pan_exist) {
- if (bt_hs_on) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = SCO + PAN(HS)\n");
- algorithm = BT_8723B_1ANT_COEX_ALGO_SCO;
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = SCO + PAN(EDR)\n");
- algorithm =
- BT_8723B_1ANT_COEX_ALGO_PANEDR_HID;
- }
- }
- } else {
- if (bt_link_info->hid_exist &&
- bt_link_info->a2dp_exist) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Profile = HID + A2DP\n");
- algorithm = BT_8723B_1ANT_COEX_ALGO_HID_A2DP;
- } else if (bt_link_info->hid_exist &&
- bt_link_info->pan_exist) {
- if (bt_hs_on) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = HID + PAN(HS)\n");
- algorithm =
- BT_8723B_1ANT_COEX_ALGO_HID_A2DP;
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = HID + PAN(EDR)\n");
- algorithm =
- BT_8723B_1ANT_COEX_ALGO_PANEDR_HID;
- }
- } else if (bt_link_info->pan_exist &&
- bt_link_info->a2dp_exist) {
- if (bt_hs_on) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = A2DP + PAN(HS)\n");
- algorithm =
- BT_8723B_1ANT_COEX_ALGO_A2DP_PANHS;
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = A2DP + PAN(EDR)\n");
- algorithm =
- BT_8723B_1ANT_COEX_ALGO_PANEDR_A2DP;
- }
- }
- }
- } else if (numdiffprofile == 3) {
- if (bt_link_info->sco_exist) {
- if (bt_link_info->hid_exist &&
- bt_link_info->a2dp_exist) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT Profile = SCO + HID + A2DP ==> HID\n");
- algorithm = BT_8723B_1ANT_COEX_ALGO_HID;
- } else if (bt_link_info->hid_exist &&
- bt_link_info->pan_exist) {
- if (bt_hs_on) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = SCO + HID + PAN(HS)\n");
- algorithm =
- BT_8723B_1ANT_COEX_ALGO_HID_A2DP;
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = SCO + HID + PAN(EDR)\n");
- algorithm =
- BT_8723B_1ANT_COEX_ALGO_PANEDR_HID;
- }
- } else if (bt_link_info->pan_exist &&
- bt_link_info->a2dp_exist) {
- if (bt_hs_on) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = SCO + A2DP + PAN(HS)\n");
- algorithm = BT_8723B_1ANT_COEX_ALGO_SCO;
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = SCO + A2DP + PAN(EDR) ==> HID\n");
- algorithm =
- BT_8723B_1ANT_COEX_ALGO_PANEDR_HID;
- }
- }
- } else {
- if (bt_link_info->hid_exist &&
- bt_link_info->pan_exist &&
- bt_link_info->a2dp_exist) {
- if (bt_hs_on) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = HID + A2DP + PAN(HS)\n");
- algorithm =
- BT_8723B_1ANT_COEX_ALGO_HID_A2DP;
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = HID + A2DP + PAN(EDR)\n");
- algorithm =
- BT_8723B_1ANT_COEX_ALGO_HID_A2DP_PANEDR;
- }
- }
- }
- } else if (numdiffprofile >= 3) {
- if (bt_link_info->sco_exist) {
- if (bt_link_info->hid_exist &&
- bt_link_info->pan_exist &&
- bt_link_info->a2dp_exist) {
- if (bt_hs_on) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], Error!!! BT Profile = SCO + HID + A2DP + PAN(HS)\n");
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST,
- DBG_LOUD,
- "[BTCoex], BT Profile = SCO + HID + A2DP + PAN(EDR)==>PAN(EDR)+HID\n");
- algorithm =
- BT_8723B_1ANT_COEX_ALGO_PANEDR_HID;
- }
- }
- }
- }
-
- return algorithm;
-}
-
static void btc8723b1ant_set_sw_pen_tx_rate_adapt(struct btc_coexist *btcoexist,
bool low_penalty_ra)
{
@@ -721,11 +343,11 @@
if (low_penalty_ra) {
h2c_parameter[1] |= BIT0;
- /*normal rate except MCS7/6/5, OFDM54/48/36 */
+ /* normal rate except MCS7/6/5, OFDM54/48/36 */
h2c_parameter[2] = 0x00;
- h2c_parameter[3] = 0xf7; /*MCS7 or OFDM54 */
- h2c_parameter[4] = 0xf8; /*MCS6 or OFDM48 */
- h2c_parameter[5] = 0xf9; /*MCS5 or OFDM36 */
+ h2c_parameter[3] = 0xf7; /* MCS7 or OFDM54 */
+ h2c_parameter[4] = 0xf8; /* MCS6 or OFDM48 */
+ h2c_parameter[5] = 0xf9; /* MCS5 or OFDM36 */
}
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -846,8 +468,9 @@
}
}
-static void halbtc8723b1ant_SetFwIgnoreWlanAct(struct btc_coexist *btcoexist,
- bool enable)
+static void
+halbtc8723b1ant_set_fw_ignore_wlan_act(struct btc_coexist *btcoexist,
+ bool enable)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[1] = {0};
@@ -882,7 +505,7 @@
coex_dm->cur_ignore_wlan_act)
return;
}
- halbtc8723b1ant_SetFwIgnoreWlanAct(btcoexist, enable);
+ halbtc8723b1ant_set_fw_ignore_wlan_act(btcoexist, enable);
coex_dm->pre_ignore_wlan_act = coex_dm->cur_ignore_wlan_act;
}
@@ -944,9 +567,9 @@
btcoexist->btc_set(btcoexist, BTC_SET_U1_RPWM_VAL, &rpwm);
}
-static void halbtc8723b1ant_LpsRpwm(struct btc_coexist *btcoexist,
- bool force_exec,
- u8 lps_val, u8 rpwm_val)
+static void halbtc8723b1ant_lps_rpwm(struct btc_coexist *btcoexist,
+ bool force_exec,
+ u8 lps_val, u8 rpwm_val)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -987,9 +610,9 @@
halbtc8723b1ant_low_penalty_ra(btcoexist, NORMAL_EXEC, low_penalty_ra);
}
-static void halbtc8723b1ant_SetAntPath(struct btc_coexist *btcoexist,
- u8 ant_pos_type, bool init_hw_cfg,
- bool wifi_off)
+static void halbtc8723b1ant_set_ant_path(struct btc_coexist *btcoexist,
+ u8 ant_pos_type, bool init_hw_cfg,
+ bool wifi_off)
{
struct btc_board_info *board_info = &btcoexist->board_info;
u32 fw_ver = 0, u32tmp = 0;
@@ -1028,7 +651,7 @@
if (use_ext_switch) {
if (init_hw_cfg) {
/* 0x4c[23] = 0, 0x4c[24] = 1
- * Antenna control by WL/BT
+ * Antenna control by WL/BT
*/
u32tmp = btcoexist->btc_read_4byte(btcoexist, 0x4c);
u32tmp &= ~BIT23;
@@ -1037,35 +660,36 @@
if (board_info->btdm_ant_pos ==
BTC_ANTENNA_AT_MAIN_PORT) {
- /* Main Ant to BT for IPS case 0x4c[23] = 1 */
+ /* Main Ant to BT for IPS case 0x4c[23] = 1 */
btcoexist->btc_write_1byte_bitmask(btcoexist,
0x64, 0x1,
0x1);
- /*tell firmware "no antenna inverse"*/
+ /* tell firmware "no antenna inverse" */
h2c_parameter[0] = 0;
h2c_parameter[1] = 1; /*ext switch type*/
btcoexist->btc_fill_h2c(btcoexist, 0x65, 2,
h2c_parameter);
} else {
- /*Aux Ant to BT for IPS case 0x4c[23] = 1 */
+ /* Aux Ant to BT for IPS case 0x4c[23] = 1 */
btcoexist->btc_write_1byte_bitmask(btcoexist,
0x64, 0x1,
0x0);
- /*tell firmware "antenna inverse"*/
+ /* tell firmware "antenna inverse" */
h2c_parameter[0] = 1;
- h2c_parameter[1] = 1; /*ext switch type*/
+ h2c_parameter[1] = 1; /* ext switch type */
btcoexist->btc_fill_h2c(btcoexist, 0x65, 2,
h2c_parameter);
}
}
- /* fixed internal switch first*/
- /* fixed internal switch S1->WiFi, S0->BT*/
+ /* fixed internal switch first
+ * fixed internal switch S1->WiFi, S0->BT
+ */
if (board_info->btdm_ant_pos == BTC_ANTENNA_AT_MAIN_PORT)
btcoexist->btc_write_2byte(btcoexist, 0x948, 0x0);
- else/* fixed internal switch S0->WiFi, S1->BT*/
+ else /* fixed internal switch S0->WiFi, S1->BT */
btcoexist->btc_write_2byte(btcoexist, 0x948, 0x280);
/* ext switch setting */
@@ -1108,7 +732,7 @@
} else {
if (init_hw_cfg) {
- /* 0x4c[23] = 1, 0x4c[24] = 0 Antenna control by 0x64*/
+ /* 0x4c[23] = 1, 0x4c[24] = 0 Antenna control by 0x64 */
u32tmp = btcoexist->btc_read_4byte(btcoexist, 0x4c);
u32tmp |= BIT23;
u32tmp &= ~BIT24;
@@ -1116,41 +740,42 @@
if (board_info->btdm_ant_pos ==
BTC_ANTENNA_AT_MAIN_PORT) {
- /*Main Ant to WiFi for IPS case 0x4c[23] = 1*/
+ /* Main Ant to WiFi for IPS case 0x4c[23] = 1 */
btcoexist->btc_write_1byte_bitmask(btcoexist,
0x64, 0x1,
0x0);
- /*tell firmware "no antenna inverse"*/
+ /* tell firmware "no antenna inverse" */
h2c_parameter[0] = 0;
- h2c_parameter[1] = 0; /*internal switch type*/
+ h2c_parameter[1] = 0; /* internal switch type */
btcoexist->btc_fill_h2c(btcoexist, 0x65, 2,
h2c_parameter);
} else {
- /*Aux Ant to BT for IPS case 0x4c[23] = 1*/
+ /* Aux Ant to BT for IPS case 0x4c[23] = 1 */
btcoexist->btc_write_1byte_bitmask(btcoexist,
0x64, 0x1,
0x1);
- /*tell firmware "antenna inverse"*/
+ /* tell firmware "antenna inverse" */
h2c_parameter[0] = 1;
- h2c_parameter[1] = 0; /*internal switch type*/
+ h2c_parameter[1] = 0; /* internal switch type */
btcoexist->btc_fill_h2c(btcoexist, 0x65, 2,
h2c_parameter);
}
}
- /* fixed external switch first*/
- /*Main->WiFi, Aux->BT*/
+ /* fixed external switch first
+ * Main->WiFi, Aux->BT
+ */
if (board_info->btdm_ant_pos ==
BTC_ANTENNA_AT_MAIN_PORT)
btcoexist->btc_write_1byte_bitmask(btcoexist, 0x92c,
0x3, 0x1);
- else/*Main->BT, Aux->WiFi */
+ else /* Main->BT, Aux->WiFi */
btcoexist->btc_write_1byte_bitmask(btcoexist, 0x92c,
0x3, 0x2);
- /* internal switch setting*/
+ /* internal switch setting */
switch (ant_pos_type) {
case BTC_ANT_PATH_WIFI:
if (board_info->btdm_ant_pos ==
@@ -1365,7 +990,7 @@
halbtc8723b1ant_set_fw_ps_tdma(btcoexist, 0xd3, 0x12,
0x3, 0x14, 0x50);
break;
- /* SoftAP only with no sta associated,BT disable ,
+ /* SoftAP only with no sta associated, BT disable,
* TDMA mode for power saving
* here softap mode screen off will cost 70-80mA for phone
*/
@@ -1376,24 +1001,29 @@
}
} else {
switch (type) {
- case 8: /*PTA Control */
+ case 8: /* PTA Control */
halbtc8723b1ant_set_fw_ps_tdma(btcoexist, 0x8, 0x0,
0x0, 0x0, 0x0);
- halbtc8723b1ant_SetAntPath(btcoexist, BTC_ANT_PATH_PTA,
- false, false);
+ halbtc8723b1ant_set_ant_path(btcoexist,
+ BTC_ANT_PATH_PTA,
+ false, false);
break;
case 0:
- default: /*Software control, Antenna at BT side */
+ default:
+ /* Software control, Antenna at BT side */
halbtc8723b1ant_set_fw_ps_tdma(btcoexist, 0x0, 0x0,
0x0, 0x0, 0x0);
- halbtc8723b1ant_SetAntPath(btcoexist, BTC_ANT_PATH_BT,
- false, false);
+ halbtc8723b1ant_set_ant_path(btcoexist,
+ BTC_ANT_PATH_BT,
+ false, false);
break;
- case 9: /*Software control, Antenna at WiFi side */
+ case 9:
+ /* Software control, Antenna at WiFi side */
halbtc8723b1ant_set_fw_ps_tdma(btcoexist, 0x0, 0x0,
0x0, 0x0, 0x0);
- halbtc8723b1ant_SetAntPath(btcoexist, BTC_ANT_PATH_WIFI,
- false, false);
+ halbtc8723b1ant_set_ant_path(btcoexist,
+ BTC_ANT_PATH_WIFI,
+ false, false);
break;
}
}
@@ -1407,247 +1037,15 @@
coex_dm->pre_ps_tdma = coex_dm->cur_ps_tdma;
}
-static bool halbtc8723b1ant_is_common_action(struct btc_coexist *btcoexist)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- bool commom = false, wifi_connected = false;
- bool wifi_busy = false;
-
- btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
- &wifi_connected);
- btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_BUSY, &wifi_busy);
-
- if (!wifi_connected &&
- BT_8723B_1ANT_BT_STATUS_NON_CONNECTED_IDLE == coex_dm->bt_status) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi non connected-idle + BT non connected-idle!!\n");
- halbtc8723b1ant_sw_mechanism(btcoexist, false);
- commom = true;
- } else if (wifi_connected &&
- (BT_8723B_1ANT_BT_STATUS_NON_CONNECTED_IDLE ==
- coex_dm->bt_status)) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi connected + BT non connected-idle!!\n");
- halbtc8723b1ant_sw_mechanism(btcoexist, false);
- commom = true;
- } else if (!wifi_connected &&
- (BT_8723B_1ANT_BT_STATUS_CONNECTED_IDLE ==
- coex_dm->bt_status)) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi non connected-idle + BT connected-idle!!\n");
- halbtc8723b1ant_sw_mechanism(btcoexist, false);
- commom = true;
- } else if (wifi_connected &&
- (BT_8723B_1ANT_BT_STATUS_CONNECTED_IDLE ==
- coex_dm->bt_status)) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi connected + BT connected-idle!!\n");
- halbtc8723b1ant_sw_mechanism(btcoexist, false);
- commom = true;
- } else if (!wifi_connected &&
- (BT_8723B_1ANT_BT_STATUS_CONNECTED_IDLE !=
- coex_dm->bt_status)) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi non connected-idle + BT Busy!!\n");
- halbtc8723b1ant_sw_mechanism(btcoexist, false);
- commom = true;
- } else {
- if (wifi_busy)
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi Connected-Busy + BT Busy!!\n");
- else
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi Connected-Idle + BT Busy!!\n");
-
- commom = false;
- }
-
- return commom;
-}
-
-static void btc8723b1ant_tdma_dur_adj_for_acl(struct btc_coexist *btcoexist,
- u8 wifi_status)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- static s32 up, dn, m, n, wait_count;
- /* 0: no change, +1: increase WiFi duration,
- * -1: decrease WiFi duration
- */
- s32 result;
- u8 retry_count = 0, bt_info_ext;
- bool wifi_busy = false;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TdmaDurationAdjustForAcl()\n");
-
- if (BT_8723B_1ANT_WIFI_STATUS_CONNECTED_BUSY == wifi_status)
- wifi_busy = true;
- else
- wifi_busy = false;
-
- if ((BT_8723B_1ANT_WIFI_STATUS_NON_CONNECTED_ASSO_AUTH_SCAN ==
- wifi_status) ||
- (BT_8723B_1ANT_WIFI_STATUS_CONNECTED_SCAN == wifi_status) ||
- (BT_8723B_1ANT_WIFI_STATUS_CONNECTED_SPECIAL_PKT == wifi_status)) {
- if (coex_dm->cur_ps_tdma != 1 && coex_dm->cur_ps_tdma != 2 &&
- coex_dm->cur_ps_tdma != 3 && coex_dm->cur_ps_tdma != 9) {
- halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
-
- up = 0;
- dn = 0;
- m = 1;
- n = 3;
- result = 0;
- wait_count = 0;
- }
- return;
- }
-
- if (!coex_dm->auto_tdma_adjust) {
- coex_dm->auto_tdma_adjust = true;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], first run TdmaDurationAdjust()!!\n");
-
- halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 2);
- coex_dm->tdma_adj_type = 2;
-
- up = 0;
- dn = 0;
- m = 1;
- n = 3;
- result = 0;
- wait_count = 0;
- } else {
- /*accquire the BT TRx retry count from BT_Info byte2 */
- retry_count = coex_sta->bt_retry_cnt;
- bt_info_ext = coex_sta->bt_info_ext;
- result = 0;
- wait_count++;
- /* no retry in the last 2-second duration */
- if (retry_count == 0) {
- up++;
- dn--;
-
- if (dn <= 0)
- dn = 0;
-
- if (up >= n) {
- wait_count = 0;
- n = 3;
- up = 0;
- dn = 0;
- result = 1;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Increase wifi duration!!\n");
- }
- } else if (retry_count <= 3) {
- up--;
- dn++;
-
- if (up <= 0)
- up = 0;
-
- if (dn == 2) {
- if (wait_count <= 2)
- m++;
- else
- m = 1;
-
- if (m >= 20)
- m = 20;
-
- n = 3 * m;
- up = 0;
- dn = 0;
- wait_count = 0;
- result = -1;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Decrease wifi duration for retryCounter<3!!\n");
- }
- } else {
- if (wait_count == 1)
- m++;
- else
- m = 1;
-
- if (m >= 20)
- m = 20;
-
- n = 3 * m;
- up = 0;
- dn = 0;
- wait_count = 0;
- result = -1;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Decrease wifi duration for retryCounter>3!!\n");
- }
-
- if (result == -1) {
- if ((BT_INFO_8723B_1ANT_A2DP_BASIC_RATE(bt_info_ext)) &&
- ((coex_dm->cur_ps_tdma == 1) ||
- (coex_dm->cur_ps_tdma == 2))) {
- halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
- } else if (coex_dm->cur_ps_tdma == 1) {
- halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
- } else if (coex_dm->cur_ps_tdma == 9) {
- halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- }
- } else if (result == 1) {
- if ((BT_INFO_8723B_1ANT_A2DP_BASIC_RATE(bt_info_ext)) &&
- ((coex_dm->cur_ps_tdma == 1) ||
- (coex_dm->cur_ps_tdma == 2))) {
- halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
- } else if (coex_dm->cur_ps_tdma == 9) {
- halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 1);
- coex_dm->tdma_adj_type = 1;
- }
- } else { /*no change */
- /*if busy / idle change */
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex],********* TDMA(on, %d) ********\n",
- coex_dm->cur_ps_tdma);
- }
-
- if (coex_dm->cur_ps_tdma != 1 && coex_dm->cur_ps_tdma != 2 &&
- coex_dm->cur_ps_tdma != 9 && coex_dm->cur_ps_tdma != 11) {
- /* recover to previous adjust type */
- halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC, true,
- coex_dm->tdma_adj_type);
- }
- }
-}
-
-static void btc8723b1ant_pstdmachkpwrsave(struct btc_coexist *btcoexist,
- bool new_ps_state)
+static void halbtc8723b1ant_ps_tdma_chk_pwr_save(struct btc_coexist *btcoexist,
+ bool new_ps_state)
{
u8 lps_mode = 0x0;
btcoexist->btc_get(btcoexist, BTC_GET_U1_LPS_MODE, &lps_mode);
- if (lps_mode) { /* already under LPS state */
+ if (lps_mode) {
+ /* already under LPS state */
if (new_ps_state) {
/* keep state under LPS, do nothing. */
} else {
@@ -1655,7 +1053,8 @@
halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
false, 0);
}
- } else { /* NO PS state */
+ } else {
+ /* NO PS state */
if (new_ps_state) {
/* will enter LPS state, turn off psTdma first */
halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
@@ -1681,18 +1080,18 @@
btcoexist->btc_set(btcoexist, BTC_SET_ACT_NORMAL_LPS, NULL);
break;
case BTC_PS_LPS_ON:
- btc8723b1ant_pstdmachkpwrsave(btcoexist, true);
- halbtc8723b1ant_LpsRpwm(btcoexist, NORMAL_EXEC, lps_val,
- rpwm_val);
- /* when coex force to enter LPS, do not enter 32k low power. */
+ halbtc8723b1ant_ps_tdma_chk_pwr_save(btcoexist, true);
+ halbtc8723b1ant_lps_rpwm(btcoexist, NORMAL_EXEC, lps_val,
+ rpwm_val);
+ /* when coex force to enter LPS, do not enter 32k low power */
low_pwr_disable = true;
btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
&low_pwr_disable);
- /* power save must executed before psTdma. */
+ /* power save must executed before psTdma */
btcoexist->btc_set(btcoexist, BTC_SET_ACT_ENTER_LPS, NULL);
break;
case BTC_PS_LPS_OFF:
- btc8723b1ant_pstdmachkpwrsave(btcoexist, false);
+ halbtc8723b1ant_ps_tdma_chk_pwr_save(btcoexist, false);
btcoexist->btc_set(btcoexist, BTC_SET_ACT_LEAVE_LPS, NULL);
break;
default:
@@ -1700,66 +1099,6 @@
}
}
-/***************************************************
- *
- * Software Coex Mechanism start
- *
- ***************************************************/
-/* SCO only or SCO+PAN(HS) */
-static void halbtc8723b1ant_action_sco(struct btc_coexist *btcoexist)
-{
- halbtc8723b1ant_sw_mechanism(btcoexist, true);
-}
-
-static void halbtc8723b1ant_action_hid(struct btc_coexist *btcoexist)
-{
- halbtc8723b1ant_sw_mechanism(btcoexist, true);
-}
-
-/*A2DP only / PAN(EDR) only/ A2DP+PAN(HS) */
-static void halbtc8723b1ant_action_a2dp(struct btc_coexist *btcoexist)
-{
- halbtc8723b1ant_sw_mechanism(btcoexist, false);
-}
-
-static void halbtc8723b1ant_action_a2dp_pan_hs(struct btc_coexist *btcoexist)
-{
- halbtc8723b1ant_sw_mechanism(btcoexist, false);
-}
-
-static void halbtc8723b1ant_action_pan_edr(struct btc_coexist *btcoexist)
-{
- halbtc8723b1ant_sw_mechanism(btcoexist, false);
-}
-
-/* PAN(HS) only */
-static void halbtc8723b1ant_action_pan_hs(struct btc_coexist *btcoexist)
-{
- halbtc8723b1ant_sw_mechanism(btcoexist, false);
-}
-
-/*PAN(EDR)+A2DP */
-static void halbtc8723b1ant_action_pan_edr_a2dp(struct btc_coexist *btcoexist)
-{
- halbtc8723b1ant_sw_mechanism(btcoexist, false);
-}
-
-static void halbtc8723b1ant_action_pan_edr_hid(struct btc_coexist *btcoexist)
-{
- halbtc8723b1ant_sw_mechanism(btcoexist, true);
-}
-
-/* HID+A2DP+PAN(EDR) */
-static void btc8723b1ant_action_hid_a2dp_pan_edr(struct btc_coexist *btcoexist)
-{
- halbtc8723b1ant_sw_mechanism(btcoexist, true);
-}
-
-static void halbtc8723b1ant_action_hid_a2dp(struct btc_coexist *btcoexist)
-{
- halbtc8723b1ant_sw_mechanism(btcoexist, true);
-}
-
/*****************************************************
*
* Non-Software Coex Mechanism start
@@ -1826,11 +1165,11 @@
&wifi_connected);
/* tdma and coex table */
-
if (bt_link_info->sco_exist) {
halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 5);
halbtc8723b1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
- } else { /* HID */
+ } else {
+ /* HID */
halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 6);
halbtc8723b1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 5);
}
@@ -1840,30 +1179,21 @@
struct btc_coexist *btcoexist,
u8 wifi_status)
{
- u8 bt_rssi_state;
-
struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
- bt_rssi_state = halbtc8723b1ant_bt_rssi_state(btcoexist, 2, 28, 0);
- if (bt_link_info->hid_only) { /*HID */
+ if (bt_link_info->hid_only) { /* HID */
btc8723b1ant_act_bt_sco_hid_only_busy(btcoexist, wifi_status);
coex_dm->auto_tdma_adjust = false;
return;
- } else if (bt_link_info->a2dp_only) { /*A2DP */
- if (BT_8723B_1ANT_WIFI_STATUS_CONNECTED_IDLE == wifi_status) {
+ } else if (bt_link_info->a2dp_only) { /* A2DP */
+ if (wifi_status == BT_8723B_1ANT_WIFI_STATUS_CONNECTED_IDLE) {
halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
false, 8);
halbtc8723b1ant_coex_table_with_type(btcoexist,
NORMAL_EXEC, 2);
coex_dm->auto_tdma_adjust = false;
- } else if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b1ant_tdma_dur_adj_for_acl(btcoexist,
- wifi_status);
- halbtc8723b1ant_coex_table_with_type(btcoexist,
- NORMAL_EXEC, 1);
- } else { /*for low BT RSSI */
+ } else { /* for low BT RSSI */
halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
true, 11);
halbtc8723b1ant_coex_table_with_type(btcoexist,
@@ -1871,18 +1201,18 @@
coex_dm->auto_tdma_adjust = false;
}
} else if (bt_link_info->hid_exist &&
- bt_link_info->a2dp_exist) { /*HID+A2DP */
- halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
+ bt_link_info->a2dp_exist) { /* HID + A2DP */
+ halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
coex_dm->auto_tdma_adjust = false;
halbtc8723b1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 6);
- /*PAN(OPP,FTP), HID+PAN(OPP,FTP) */
+ /* PAN(OPP,FTP), HID + PAN(OPP,FTP) */
} else if (bt_link_info->pan_only ||
(bt_link_info->hid_exist && bt_link_info->pan_exist)) {
halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 3);
halbtc8723b1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 6);
coex_dm->auto_tdma_adjust = false;
- /*A2DP+PAN(OPP,FTP), HID+A2DP+PAN(OPP,FTP)*/
+ /* A2DP + PAN(OPP,FTP), HID + A2DP + PAN(OPP,FTP) */
} else if ((bt_link_info->a2dp_exist && bt_link_info->pan_exist) ||
(bt_link_info->hid_exist && bt_link_info->a2dp_exist &&
bt_link_info->pan_exist)) {
@@ -1907,57 +1237,59 @@
halbtc8723b1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
}
-static void btc8723b1ant_action_wifi_not_conn_scan(struct btc_coexist *btcoex)
+static void
+btc8723b1ant_action_wifi_not_conn_scan(struct btc_coexist *btcoexist)
{
- struct btc_bt_link_info *bt_link_info = &btcoex->bt_link_info;
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
- halbtc8723b1ant_power_save_state(btcoex, BTC_PS_WIFI_NATIVE,
+ halbtc8723b1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
0x0, 0x0);
/* tdma and coex table */
if (BT_8723B_1ANT_BT_STATUS_ACL_BUSY == coex_dm->bt_status) {
if (bt_link_info->a2dp_exist && bt_link_info->pan_exist) {
- halbtc8723b1ant_ps_tdma(btcoex, NORMAL_EXEC,
+ halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
true, 22);
- halbtc8723b1ant_coex_table_with_type(btcoex,
+ halbtc8723b1ant_coex_table_with_type(btcoexist,
NORMAL_EXEC, 1);
} else if (bt_link_info->pan_only) {
- halbtc8723b1ant_ps_tdma(btcoex, NORMAL_EXEC,
+ halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
true, 20);
- halbtc8723b1ant_coex_table_with_type(btcoex,
+ halbtc8723b1ant_coex_table_with_type(btcoexist,
NORMAL_EXEC, 2);
} else {
- halbtc8723b1ant_ps_tdma(btcoex, NORMAL_EXEC,
+ halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC,
true, 20);
- halbtc8723b1ant_coex_table_with_type(btcoex,
+ halbtc8723b1ant_coex_table_with_type(btcoexist,
NORMAL_EXEC, 1);
}
} else if ((BT_8723B_1ANT_BT_STATUS_SCO_BUSY == coex_dm->bt_status) ||
(BT_8723B_1ANT_BT_STATUS_ACL_SCO_BUSY ==
coex_dm->bt_status)){
- btc8723b1ant_act_bt_sco_hid_only_busy(btcoex,
+ btc8723b1ant_act_bt_sco_hid_only_busy(btcoexist,
BT_8723B_1ANT_WIFI_STATUS_CONNECTED_SCAN);
} else {
- halbtc8723b1ant_ps_tdma(btcoex, NORMAL_EXEC, false, 8);
- halbtc8723b1ant_coex_table_with_type(btcoex, NORMAL_EXEC, 2);
+ halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 8);
+ halbtc8723b1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
}
}
-static void btc8723b1ant_act_wifi_not_conn_asso_auth(struct btc_coexist *btcoex)
+static void
+btc8723b1ant_act_wifi_not_conn_asso_auth(struct btc_coexist *btcoexist)
{
- struct btc_bt_link_info *bt_link_info = &btcoex->bt_link_info;
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
- halbtc8723b1ant_power_save_state(btcoex, BTC_PS_WIFI_NATIVE,
+ halbtc8723b1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
0x0, 0x0);
if ((BT_8723B_1ANT_BT_STATUS_CONNECTED_IDLE == coex_dm->bt_status) ||
(bt_link_info->sco_exist) || (bt_link_info->hid_only) ||
(bt_link_info->a2dp_only) || (bt_link_info->pan_only)) {
- halbtc8723b1ant_ps_tdma(btcoex, NORMAL_EXEC, false, 8);
- halbtc8723b1ant_coex_table_with_type(btcoex, NORMAL_EXEC, 7);
+ halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 8);
+ halbtc8723b1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
} else {
- halbtc8723b1ant_ps_tdma(btcoex, NORMAL_EXEC, true, 20);
- halbtc8723b1ant_coex_table_with_type(btcoex, NORMAL_EXEC, 1);
+ halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 20);
+ halbtc8723b1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
}
}
@@ -2109,75 +1441,6 @@
}
}
-static void btc8723b1ant_run_sw_coex_mech(struct btc_coexist *btcoexist)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- u8 algorithm = 0;
-
- algorithm = halbtc8723b1ant_action_algorithm(btcoexist);
- coex_dm->cur_algorithm = algorithm;
-
- if (!halbtc8723b1ant_is_common_action(btcoexist)) {
- switch (coex_dm->cur_algorithm) {
- case BT_8723B_1ANT_COEX_ALGO_SCO:
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Action algorithm = SCO\n");
- halbtc8723b1ant_action_sco(btcoexist);
- break;
- case BT_8723B_1ANT_COEX_ALGO_HID:
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Action algorithm = HID\n");
- halbtc8723b1ant_action_hid(btcoexist);
- break;
- case BT_8723B_1ANT_COEX_ALGO_A2DP:
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Action algorithm = A2DP\n");
- halbtc8723b1ant_action_a2dp(btcoexist);
- break;
- case BT_8723B_1ANT_COEX_ALGO_A2DP_PANHS:
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Action algorithm = A2DP+PAN(HS)\n");
- halbtc8723b1ant_action_a2dp_pan_hs(btcoexist);
- break;
- case BT_8723B_1ANT_COEX_ALGO_PANEDR:
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Action algorithm = PAN(EDR)\n");
- halbtc8723b1ant_action_pan_edr(btcoexist);
- break;
- case BT_8723B_1ANT_COEX_ALGO_PANHS:
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Action algorithm = HS mode\n");
- halbtc8723b1ant_action_pan_hs(btcoexist);
- break;
- case BT_8723B_1ANT_COEX_ALGO_PANEDR_A2DP:
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Action algorithm = PAN+A2DP\n");
- halbtc8723b1ant_action_pan_edr_a2dp(btcoexist);
- break;
- case BT_8723B_1ANT_COEX_ALGO_PANEDR_HID:
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Action algorithm = PAN(EDR)+HID\n");
- halbtc8723b1ant_action_pan_edr_hid(btcoexist);
- break;
- case BT_8723B_1ANT_COEX_ALGO_HID_A2DP_PANEDR:
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Action algorithm = HID+A2DP+PAN\n");
- btc8723b1ant_action_hid_a2dp_pan_edr(btcoexist);
- break;
- case BT_8723B_1ANT_COEX_ALGO_HID_A2DP:
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Action algorithm = HID+A2DP\n");
- halbtc8723b1ant_action_hid_a2dp(btcoexist);
- break;
- default:
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Action algorithm = coexist All Off!!\n");
- break;
- }
- coex_dm->pre_algorithm = coex_dm->cur_algorithm;
- }
-}
-
static void halbtc8723b1ant_run_coexist_mechanism(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -2186,7 +1449,6 @@
bool increase_scan_dev_num = false;
bool bt_ctrl_agg_buf_size = false;
u8 agg_buf_size = 5;
- u8 wifi_rssi_state = BTC_RSSI_STATE_HIGH;
u32 wifi_link_status = 0;
u32 num_of_wifi_link = 0;
@@ -2238,16 +1500,12 @@
if (!bt_link_info->sco_exist && !bt_link_info->hid_exist) {
halbtc8723b1ant_limited_tx(btcoexist, NORMAL_EXEC, 0, 0, 0, 0);
} else {
- if (wifi_connected) {
- wifi_rssi_state =
- halbtc8723b1ant_wifi_rssi_state(btcoexist,
- 1, 2, 30, 0);
+ if (wifi_connected)
halbtc8723b1ant_limited_tx(btcoexist,
NORMAL_EXEC, 1, 1, 1, 1);
- } else {
+ else
halbtc8723b1ant_limited_tx(btcoexist, NORMAL_EXEC,
0, 0, 0, 0);
- }
}
if (bt_link_info->sco_exist) {
@@ -2263,8 +1521,6 @@
halbtc8723b1ant_limited_rx(btcoexist, NORMAL_EXEC, false,
bt_ctrl_agg_buf_size, agg_buf_size);
- btc8723b1ant_run_sw_coex_mech(btcoexist);
-
btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
if (coex_sta->c2h_bt_inquiry_page) {
@@ -2364,28 +1620,19 @@
btcoexist->btc_write_1byte(btcoexist, 0x790, u8tmp);
/* Enable counter statistics */
- /*0x76e[3] =1, WLAN_Act control by PTA */
+ /*0x76e[3] = 1, WLAN_Act control by PTA */
btcoexist->btc_write_1byte(btcoexist, 0x76e, 0xc);
btcoexist->btc_write_1byte(btcoexist, 0x778, 0x1);
btcoexist->btc_write_1byte_bitmask(btcoexist, 0x40, 0x20, 0x1);
- /*Antenna config */
- halbtc8723b1ant_SetAntPath(btcoexist, BTC_ANT_PATH_PTA, true, false);
+ /* Antenna config */
+ halbtc8723b1ant_set_ant_path(btcoexist, BTC_ANT_PATH_PTA, true, false);
/* PTA parameter */
halbtc8723b1ant_coex_table_with_type(btcoexist, FORCE_EXEC, 0);
}
-static void halbtc8723b1ant_wifi_off_hw_cfg(struct btc_coexist *btcoexist)
-{
- /* set wlan_act to low */
- btcoexist->btc_write_1byte(btcoexist, 0x76e, 0x4);
-}
-
/**************************************************************
- * work around function start with wa_halbtc8723b1ant_
- **************************************************************/
-/**************************************************************
- * extern function start with EXhalbtc8723b1ant_
+ * extern function start with ex_halbtc8723b1ant_
**************************************************************/
void ex_halbtc8723b1ant_init_hwconfig(struct btc_coexist *btcoexist)
@@ -2539,7 +1786,7 @@
if (coex_sta->bt_info_c2h_cnt[i]) {
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG,
"\r\n %-35s = %7ph(%d)",
- GLBtInfoSrc8723b1Ant[i],
+ glbt_info_src_8723b_1ant[i],
coex_sta->bt_info_c2h[i],
coex_sta->bt_info_c2h_cnt[i]);
}
@@ -2697,13 +1944,12 @@
"[BTCoex], IPS ENTER notify\n");
coex_sta->under_ips = true;
- halbtc8723b1ant_SetAntPath(btcoexist, BTC_ANT_PATH_BT,
- false, true);
+ halbtc8723b1ant_set_ant_path(btcoexist, BTC_ANT_PATH_BT,
+ false, true);
/* set PTA control */
halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 0);
halbtc8723b1ant_coex_table_with_type(btcoexist,
NORMAL_EXEC, 0);
- halbtc8723b1ant_wifi_off_hw_cfg(btcoexist);
} else if (BTC_IPS_LEAVE == type) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], IPS LEAVE notify\n");
@@ -2774,14 +2020,17 @@
if (BTC_SCAN_START == type) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], SCAN START notify\n");
- if (!wifi_connected) /* non-connected scan */
+ if (!wifi_connected)
+ /* non-connected scan */
btc8723b1ant_action_wifi_not_conn_scan(btcoexist);
- else /* wifi is connected */
+ else
+ /* wifi is connected */
btc8723b1ant_action_wifi_conn_scan(btcoexist);
} else if (BTC_SCAN_FINISH == type) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], SCAN FINISH notify\n");
- if (!wifi_connected) /* non-connected scan */
+ if (!wifi_connected)
+ /* non-connected scan */
btc8723b1ant_action_wifi_not_conn(btcoexist);
else
halbtc8723b1ant_action_wifi_connected(btcoexist);
@@ -2831,7 +2080,8 @@
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
&wifi_connected);
- if (!wifi_connected) /* non-connected scan */
+ if (!wifi_connected)
+ /* non-connected scan */
btc8723b1ant_action_wifi_not_conn(btcoexist);
else
halbtc8723b1ant_action_wifi_connected(btcoexist);
@@ -3020,7 +2270,8 @@
coex_sta->a2dp_exist = false;
coex_sta->hid_exist = false;
coex_sta->sco_exist = false;
- } else { /* connection exists */
+ } else {
+ /* connection exists */
coex_sta->bt_link_exist = true;
if (bt_info & BT_INFO_8723B_1ANT_B_FTP)
coex_sta->pan_exist = true;
@@ -3089,9 +2340,8 @@
btcoexist->stop_coex_dm = true;
- halbtc8723b1ant_SetAntPath(btcoexist, BTC_ANT_PATH_BT, false, true);
+ halbtc8723b1ant_set_ant_path(btcoexist, BTC_ANT_PATH_BT, false, true);
- halbtc8723b1ant_wifi_off_hw_cfg(btcoexist);
halbtc8723b1ant_ignore_wlan_act(btcoexist, FORCE_EXEC, true);
halbtc8723b1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
@@ -3111,13 +2361,12 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Pnp notify to SLEEP\n");
btcoexist->stop_coex_dm = true;
- halbtc8723b1ant_SetAntPath(btcoexist, BTC_ANT_PATH_BT, false,
- true);
+ halbtc8723b1ant_set_ant_path(btcoexist, BTC_ANT_PATH_BT, false,
+ true);
halbtc8723b1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
0x0, 0x0);
halbtc8723b1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 0);
halbtc8723b1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
- halbtc8723b1ant_wifi_off_hw_cfg(btcoexist);
} else if (BTC_WIFI_PNP_WAKE_UP == pnp_state) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Pnp notify to WAKE UP\n");
diff --git a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8723b2ant.c b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8723b2ant.c
index 1212596..2f3946b 100644
--- a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8723b2ant.c
+++ b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8723b2ant.c
@@ -240,9 +240,33 @@
return wifi_rssi_state;
}
+static
+void btc8723b2ant_limited_rx(struct btc_coexist *btcoexist, bool force_exec,
+ bool rej_ap_agg_pkt, bool bt_ctrl_agg_buf_size,
+ u8 agg_buf_size)
+{
+ bool reject_rx_agg = rej_ap_agg_pkt;
+ bool bt_ctrl_rx_agg_size = bt_ctrl_agg_buf_size;
+ u8 rx_agg_size = agg_buf_size;
+
+ /* ============================================ */
+ /* Rx Aggregation related setting */
+ /* ============================================ */
+ btcoexist->btc_set(btcoexist, BTC_SET_BL_TO_REJ_AP_AGG_PKT,
+ &reject_rx_agg);
+ /* decide BT control aggregation buf size or not */
+ btcoexist->btc_set(btcoexist, BTC_SET_BL_BT_CTRL_AGG_SIZE,
+ &bt_ctrl_rx_agg_size);
+ /* aggregate buf size, only work when BT control Rx aggregate size */
+ btcoexist->btc_set(btcoexist, BTC_SET_U1_AGG_BUF_SIZE, &rx_agg_size);
+ /* real update aggregation setting */
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_AGGREGATE_CTRL, NULL);
+}
+
static void btc8723b2ant_monitor_bt_ctr(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
u32 reg_hp_txrx, reg_lp_txrx, u32tmp;
u32 reg_hp_tx = 0, reg_hp_rx = 0;
u32 reg_lp_tx = 0, reg_lp_rx = 0;
@@ -263,6 +287,17 @@
coex_sta->low_priority_tx = reg_lp_tx;
coex_sta->low_priority_rx = reg_lp_rx;
+ if ((coex_sta->low_priority_tx > 1050) &&
+ (!coex_sta->c2h_bt_inquiry_page))
+ coex_sta->pop_event_cnt++;
+
+ if ((coex_sta->low_priority_rx >= 950) &&
+ (coex_sta->low_priority_rx >= coex_sta->low_priority_tx) &&
+ (!coex_sta->under_ips))
+ bt_link_info->slave_role = true;
+ else
+ bt_link_info->slave_role = false;
+
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], High Priority Tx/Rx(reg 0x%x)=0x%x(%d)/0x%x(%d)\n",
reg_hp_txrx, reg_hp_tx, reg_hp_tx, reg_hp_rx, reg_hp_rx);
@@ -274,6 +309,43 @@
btcoexist->btc_write_1byte(btcoexist, 0x76e, 0xc);
}
+static void btc8723b2ant_monitor_wifi_ctr(struct btc_coexist *btcoexist)
+{
+ if (coex_sta->under_ips) {
+ coex_sta->crc_ok_cck = 0;
+ coex_sta->crc_ok_11g = 0;
+ coex_sta->crc_ok_11n = 0;
+ coex_sta->crc_ok_11n_agg = 0;
+
+ coex_sta->crc_err_cck = 0;
+ coex_sta->crc_err_11g = 0;
+ coex_sta->crc_err_11n = 0;
+ coex_sta->crc_err_11n_agg = 0;
+ } else {
+ coex_sta->crc_ok_cck =
+ btcoexist->btc_read_4byte(btcoexist, 0xf88);
+ coex_sta->crc_ok_11g =
+ btcoexist->btc_read_2byte(btcoexist, 0xf94);
+ coex_sta->crc_ok_11n =
+ btcoexist->btc_read_2byte(btcoexist, 0xf90);
+ coex_sta->crc_ok_11n_agg =
+ btcoexist->btc_read_2byte(btcoexist, 0xfb8);
+
+ coex_sta->crc_err_cck =
+ btcoexist->btc_read_4byte(btcoexist, 0xf84);
+ coex_sta->crc_err_11g =
+ btcoexist->btc_read_2byte(btcoexist, 0xf96);
+ coex_sta->crc_err_11n =
+ btcoexist->btc_read_2byte(btcoexist, 0xf92);
+ coex_sta->crc_err_11n_agg =
+ btcoexist->btc_read_2byte(btcoexist, 0xfba);
+ }
+
+ /* reset counter */
+ btcoexist->btc_write_1byte_bitmask(btcoexist, 0xf16, 0x1, 0x1);
+ btcoexist->btc_write_1byte_bitmask(btcoexist, 0xf16, 0x1, 0x0);
+}
+
static void btc8723b2ant_query_bt_info(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -297,6 +369,8 @@
static bool pre_bt_hs_on;
bool wifi_busy = false, under_4way = false, bt_hs_on = false;
bool wifi_connected = false;
+ u8 wifi_rssi_state = BTC_RSSI_STATE_HIGH;
+ u8 tmp;
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
&wifi_connected);
@@ -320,6 +394,15 @@
pre_bt_hs_on = bt_hs_on;
return true;
}
+
+ tmp = BT_8723B_2ANT_WIFI_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
+ wifi_rssi_state =
+ btc8723b2ant_wifi_rssi_state(btcoexist, 0, 2, tmp, 0);
+
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_LOW))
+ return true;
}
return false;
@@ -327,11 +410,9 @@
static void btc8723b2ant_update_bt_link_info(struct btc_coexist *btcoexist)
{
- /*struct btc_stack_info *stack_info = &btcoexist->stack_info;*/
struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
bool bt_hs_on = false;
-#if (BT_AUTO_REPORT_ONLY_8723B_2ANT == 1) /* profile from bt patch */
btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
bt_link_info->bt_link_exist = coex_sta->bt_link_exist;
@@ -345,21 +426,7 @@
bt_link_info->pan_exist = true;
bt_link_info->bt_link_exist = true;
}
-#else /* profile from bt stack */
- bt_link_info->bt_link_exist = stack_info->bt_link_exist;
- bt_link_info->sco_exist = stack_info->sco_exist;
- bt_link_info->a2dp_exist = stack_info->a2dp_exist;
- bt_link_info->pan_exist = stack_info->pan_exist;
- bt_link_info->hid_exist = stack_info->hid_exist;
- /*for win-8 stack HID report error*/
- if (!stack_info->hid_exist)
- stack_info->hid_exist = coex_sta->hid_exist;
- /*sync BTInfo with BT firmware and stack*/
- /* when stack HID report error, here we use the info from bt fw.*/
- if (!stack_info->bt_link_exist)
- stack_info->bt_link_exist = coex_sta->bt_link_exist;
-#endif
/* check if Sco only */
if (bt_link_info->sco_exist && !bt_link_info->a2dp_exist &&
!bt_link_info->pan_exist && !bt_link_info->hid_exist)
@@ -584,44 +651,6 @@
return algorithm;
}
-static bool btc8723b_need_dec_pwr(struct btc_coexist *btcoexist)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- bool ret = false;
- bool bt_hs_on = false, wifi_connected = false;
- s32 bt_hs_rssi = 0;
- u8 bt_rssi_state;
-
- if (!btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on))
- return false;
- if (!btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
- &wifi_connected))
- return false;
- if (!btcoexist->btc_get(btcoexist, BTC_GET_S4_HS_RSSI, &bt_hs_rssi))
- return false;
-
- bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, 29, 0);
-
- if (wifi_connected) {
- if (bt_hs_on) {
- if (bt_hs_rssi > 37) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Need to decrease bt power for HS mode!!\n");
- ret = true;
- }
- } else {
- if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Need to decrease bt power for Wifi is connected!!\n");
- ret = true;
- }
- }
- }
-
- return ret;
-}
-
static void btc8723b2ant_set_fw_dac_swing_level(struct btc_coexist *btcoexist,
u8 dac_swing_lvl)
{
@@ -642,44 +671,40 @@
}
static void btc8723b2ant_set_fw_dec_bt_pwr(struct btc_coexist *btcoexist,
- bool dec_bt_pwr)
+ u8 dec_bt_pwr_lvl)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[1] = {0};
- h2c_parameter[0] = 0;
-
- if (dec_bt_pwr)
- h2c_parameter[0] |= BIT1;
+ h2c_parameter[0] = dec_bt_pwr_lvl;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], decrease Bt Power : %s, FW write 0x62=0x%x\n",
- (dec_bt_pwr ? "Yes!!" : "No!!"), h2c_parameter[0]);
+ "[BTCoex], decrease Bt Power Level : %u\n", dec_bt_pwr_lvl);
btcoexist->btc_fill_h2c(btcoexist, 0x62, 1, h2c_parameter);
}
static void btc8723b2ant_dec_bt_pwr(struct btc_coexist *btcoexist,
- bool force_exec, bool dec_bt_pwr)
+ bool force_exec, u8 dec_bt_pwr_lvl)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], %s Dec BT power = %s\n",
- force_exec ? "force to" : "", dec_bt_pwr ? "ON" : "OFF");
- coex_dm->cur_dec_bt_pwr = dec_bt_pwr;
+ "[BTCoex], Dec BT power level = %u\n", dec_bt_pwr_lvl);
+ coex_dm->cur_dec_bt_pwr_lvl = dec_bt_pwr_lvl;
if (!force_exec) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], bPreDecBtPwr=%d, bCurDecBtPwr=%d\n",
- coex_dm->pre_dec_bt_pwr, coex_dm->cur_dec_bt_pwr);
+ "[BTCoex], PreDecBtPwrLvl=%d, CurDecBtPwrLvl=%d\n",
+ coex_dm->pre_dec_bt_pwr_lvl,
+ coex_dm->cur_dec_bt_pwr_lvl);
- if (coex_dm->pre_dec_bt_pwr == coex_dm->cur_dec_bt_pwr)
+ if (coex_dm->pre_dec_bt_pwr_lvl == coex_dm->cur_dec_bt_pwr_lvl)
return;
}
- btc8723b2ant_set_fw_dec_bt_pwr(btcoexist, coex_dm->cur_dec_bt_pwr);
+ btc8723b2ant_set_fw_dec_bt_pwr(btcoexist, coex_dm->cur_dec_bt_pwr_lvl);
- coex_dm->pre_dec_bt_pwr = coex_dm->cur_dec_bt_pwr;
+ coex_dm->pre_dec_bt_pwr_lvl = coex_dm->cur_dec_bt_pwr_lvl;
}
static void btc8723b2ant_fw_dac_swing_lvl(struct btc_coexist *btcoexist,
@@ -708,72 +733,21 @@
coex_dm->pre_fw_dac_swing_lvl = coex_dm->cur_fw_dac_swing_lvl;
}
-static void btc8723b2ant_set_sw_rf_rx_lpf_corner(struct btc_coexist *btcoexist,
- bool rx_rf_shrink_on)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- if (rx_rf_shrink_on) {
- /* Shrink RF Rx LPF corner */
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Shrink RF Rx LPF corner!!\n");
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1e,
- 0xfffff, 0xffffc);
- } else {
- /* Resume RF Rx LPF corner */
- /* After initialized, we can use coex_dm->btRf0x1eBackup */
- if (btcoexist->initilized) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Resume RF Rx LPF corner!!\n");
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1e,
- 0xfffff,
- coex_dm->bt_rf0x1e_backup);
- }
- }
-}
-
-static void btc8723b2ant_rf_shrink(struct btc_coexist *btcoexist,
- bool force_exec, bool rx_rf_shrink_on)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], %s turn Rx RF Shrink = %s\n",
- (force_exec ? "force to" : ""), (rx_rf_shrink_on ?
- "ON" : "OFF"));
- coex_dm->cur_rf_rx_lpf_shrink = rx_rf_shrink_on;
-
- if (!force_exec) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], bPreRfRxLpfShrink=%d, bCurRfRxLpfShrink=%d\n",
- coex_dm->pre_rf_rx_lpf_shrink,
- coex_dm->cur_rf_rx_lpf_shrink);
-
- if (coex_dm->pre_rf_rx_lpf_shrink ==
- coex_dm->cur_rf_rx_lpf_shrink)
- return;
- }
- btc8723b2ant_set_sw_rf_rx_lpf_corner(btcoexist,
- coex_dm->cur_rf_rx_lpf_shrink);
-
- coex_dm->pre_rf_rx_lpf_shrink = coex_dm->cur_rf_rx_lpf_shrink;
-}
-
static void btc8723b_set_penalty_txrate(struct btc_coexist *btcoexist,
bool low_penalty_ra)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[6] = {0};
- h2c_parameter[0] = 0x6; /* opCode, 0x6= Retry_Penalty*/
+ h2c_parameter[0] = 0x6; /* op_code, 0x6 = Retry_Penalty */
if (low_penalty_ra) {
h2c_parameter[1] |= BIT0;
- /*normal rate except MCS7/6/5, OFDM54/48/36*/
+ /* normal rate except MCS7/6/5, OFDM54/48/36 */
h2c_parameter[2] = 0x00;
- h2c_parameter[3] = 0xf7; /*MCS7 or OFDM54*/
- h2c_parameter[4] = 0xf8; /*MCS6 or OFDM48*/
- h2c_parameter[5] = 0xf9; /*MCS5 or OFDM36*/
+ h2c_parameter[3] = 0xf4; /* MCS7 or OFDM54 */
+ h2c_parameter[4] = 0xf5; /* MCS6 or OFDM48 */
+ h2c_parameter[5] = 0xf6; /* MCS5 or OFDM36 */
}
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -788,7 +762,6 @@
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- /*return; */
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], %s turn LowPenaltyRA = %s\n",
(force_exec ? "force to" : ""), (low_penalty_ra ?
@@ -830,9 +803,9 @@
btc8723b2ant_set_dac_swing_reg(btcoex, 0x18);
}
-static void btc8723b2ant_dac_swing(struct btc_coexist *btcoexist,
- bool force_exec, bool dac_swing_on,
- u32 dac_swing_lvl)
+void btc8723b2ant_dac_swing(struct btc_coexist *btcoexist,
+ bool force_exec, bool dac_swing_on,
+ u32 dac_swing_lvl)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -863,105 +836,6 @@
coex_dm->pre_dac_swing_lvl = coex_dm->cur_dac_swing_lvl;
}
-static void btc8723b2ant_set_agc_table(struct btc_coexist *btcoexist,
- bool agc_table_en)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- u8 rssi_adjust_val = 0;
-
- /* BB AGC Gain Table */
- if (agc_table_en) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BB Agc Table On!\n");
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0x6e1A0001);
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0x6d1B0001);
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0x6c1C0001);
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0x6b1D0001);
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0x6a1E0001);
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0x691F0001);
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0x68200001);
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BB Agc Table Off!\n");
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0xaa1A0001);
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0xa91B0001);
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0xa81C0001);
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0xa71D0001);
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0xa61E0001);
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0xa51F0001);
- btcoexist->btc_write_4byte(btcoexist, 0xc78, 0xa4200001);
- }
-
- /* RF Gain */
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0xef, 0xfffff, 0x02000);
- if (agc_table_en) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Agc Table On!\n");
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x3b,
- 0xfffff, 0x38fff);
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x3b,
- 0xfffff, 0x38ffe);
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Agc Table Off!\n");
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x3b,
- 0xfffff, 0x380c3);
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x3b,
- 0xfffff, 0x28ce6);
- }
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0xef, 0xfffff, 0x0);
-
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0xed, 0xfffff, 0x1);
-
- if (agc_table_en) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Agc Table On!\n");
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x40,
- 0xfffff, 0x38fff);
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x40,
- 0xfffff, 0x38ffe);
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Agc Table Off!\n");
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x40,
- 0xfffff, 0x380c3);
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x40,
- 0xfffff, 0x28ce6);
- }
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0xed, 0xfffff, 0x0);
-
- /* set rssiAdjustVal for wifi module. */
- if (agc_table_en)
- rssi_adjust_val = 8;
- btcoexist->btc_set(btcoexist, BTC_SET_U1_RSSI_ADJ_VAL_FOR_AGC_TABLE_ON,
- &rssi_adjust_val);
-}
-
-static void btc8723b2ant_agc_table(struct btc_coexist *btcoexist,
- bool force_exec, bool agc_table_en)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], %s %s Agc Table\n",
- (force_exec ? "force to" : ""),
- (agc_table_en ? "Enable" : "Disable"));
- coex_dm->cur_agc_table_en = agc_table_en;
-
- if (!force_exec) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], bPreAgcTableEn=%d, bCurAgcTableEn=%d\n",
- coex_dm->pre_agc_table_en,
- coex_dm->cur_agc_table_en);
-
- if (coex_dm->pre_agc_table_en == coex_dm->cur_agc_table_en)
- return;
- }
- btc8723b2ant_set_agc_table(btcoexist, agc_table_en);
-
- coex_dm->pre_agc_table_en = coex_dm->cur_agc_table_en;
-}
-
static void btc8723b2ant_set_coex_table(struct btc_coexist *btcoexist,
u32 val0x6c0, u32 val0x6c4,
u32 val0x6c8, u8 val0x6cc)
@@ -1026,61 +900,73 @@
coex_dm->pre_val0x6cc = coex_dm->cur_val0x6cc;
}
-static void btc8723b_coex_tbl_type(struct btc_coexist *btcoexist,
- bool force_exec, u8 type)
+static void btc8723b2ant_coex_table_with_type(struct btc_coexist *btcoexist,
+ bool force_exec, u8 type)
{
switch (type) {
case 0:
btc8723b2ant_coex_table(btcoexist, force_exec, 0x55555555,
- 0x55555555, 0xffff, 0x3);
+ 0x55555555, 0xffffff, 0x3);
break;
case 1:
btc8723b2ant_coex_table(btcoexist, force_exec, 0x55555555,
- 0x5afa5afa, 0xffff, 0x3);
+ 0x5afa5afa, 0xffffff, 0x3);
break;
case 2:
- btc8723b2ant_coex_table(btcoexist, force_exec, 0x5a5a5a5a,
- 0x5a5a5a5a, 0xffff, 0x3);
+ btc8723b2ant_coex_table(btcoexist, force_exec, 0x5ada5ada,
+ 0x5ada5ada, 0xffffff, 0x3);
break;
case 3:
btc8723b2ant_coex_table(btcoexist, force_exec, 0xaaaaaaaa,
- 0xaaaaaaaa, 0xffff, 0x3);
+ 0xaaaaaaaa, 0xffffff, 0x3);
break;
case 4:
btc8723b2ant_coex_table(btcoexist, force_exec, 0xffffffff,
- 0xffffffff, 0xffff, 0x3);
+ 0xffffffff, 0xffffff, 0x3);
break;
case 5:
btc8723b2ant_coex_table(btcoexist, force_exec, 0x5fff5fff,
- 0x5fff5fff, 0xffff, 0x3);
+ 0x5fff5fff, 0xffffff, 0x3);
break;
case 6:
btc8723b2ant_coex_table(btcoexist, force_exec, 0x55ff55ff,
- 0x5a5a5a5a, 0xffff, 0x3);
+ 0x5a5a5a5a, 0xffffff, 0x3);
break;
case 7:
- btc8723b2ant_coex_table(btcoexist, force_exec, 0x55ff55ff,
- 0x5afa5afa, 0xffff, 0x3);
+ btc8723b2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0x5ada5ada, 0xffffff, 0x3);
break;
case 8:
- btc8723b2ant_coex_table(btcoexist, force_exec, 0x5aea5aea,
- 0x5aea5aea, 0xffff, 0x3);
+ btc8723b2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0x5ada5ada, 0xffffff, 0x3);
break;
case 9:
- btc8723b2ant_coex_table(btcoexist, force_exec, 0x55ff55ff,
- 0x5aea5aea, 0xffff, 0x3);
+ btc8723b2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0x5ada5ada, 0xffffff, 0x3);
break;
case 10:
- btc8723b2ant_coex_table(btcoexist, force_exec, 0x55ff55ff,
- 0x5aff5aff, 0xffff, 0x3);
+ btc8723b2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0x5ada5ada, 0xffffff, 0x3);
break;
case 11:
- btc8723b2ant_coex_table(btcoexist, force_exec, 0x55ff55ff,
- 0x5a5f5a5f, 0xffff, 0x3);
+ btc8723b2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0x5ada5ada, 0xffffff, 0x3);
break;
case 12:
- btc8723b2ant_coex_table(btcoexist, force_exec, 0x55ff55ff,
- 0x5f5f5f5f, 0xffff, 0x3);
+ btc8723b2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0x5ada5ada, 0xffffff, 0x3);
+ break;
+ case 13:
+ btc8723b2ant_coex_table(btcoexist, force_exec, 0x5fff5fff,
+ 0xaaaaaaaa, 0xffffff, 0x3);
+ break;
+ case 14:
+ btc8723b2ant_coex_table(btcoexist, force_exec, 0x5fff5fff,
+ 0x5ada5ada, 0xffffff, 0x3);
+ break;
+ case 15:
+ btc8723b2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0xaaaaaaaa, 0xffffff, 0x3);
break;
default:
break;
@@ -1094,7 +980,7 @@
u8 h2c_parameter[1] = {0};
if (enable)
- h2c_parameter[0] |= BIT0;/* function enable*/
+ h2c_parameter[0] |= BIT0; /* function enable */
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], set FW for BT Ignore Wlan_Act, FW write 0x63=0x%x\n",
@@ -1103,6 +989,33 @@
btcoexist->btc_fill_h2c(btcoexist, 0x63, 1, h2c_parameter);
}
+static void btc8723b2ant_set_lps_rpwm(struct btc_coexist *btcoexist,
+ u8 lps_val, u8 rpwm_val)
+{
+ u8 lps = lps_val;
+ u8 rpwm = rpwm_val;
+
+ btcoexist->btc_set(btcoexist, BTC_SET_U1_LPS_VAL, &lps);
+ btcoexist->btc_set(btcoexist, BTC_SET_U1_RPWM_VAL, &rpwm);
+}
+
+static void btc8723b2ant_lps_rpwm(struct btc_coexist *btcoexist,
+ bool force_exec, u8 lps_val, u8 rpwm_val)
+{
+ coex_dm->cur_lps = lps_val;
+ coex_dm->cur_rpwm = rpwm_val;
+
+ if (!force_exec) {
+ if ((coex_dm->pre_lps == coex_dm->cur_lps) &&
+ (coex_dm->pre_rpwm == coex_dm->cur_rpwm))
+ return;
+ }
+ btc8723b2ant_set_lps_rpwm(btcoexist, lps_val, rpwm_val);
+
+ coex_dm->pre_lps = coex_dm->cur_lps;
+ coex_dm->pre_rpwm = coex_dm->cur_rpwm;
+}
+
static void btc8723b2ant_ignore_wlan_act(struct btc_coexist *btcoexist,
bool force_exec, bool enable)
{
@@ -1133,6 +1046,8 @@
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[5];
+ if ((coex_sta->a2dp_exist) && (coex_sta->hid_exist))
+ byte5 = byte5 | 0x1;
h2c_parameter[0] = byte1;
h2c_parameter[1] = byte2;
@@ -1155,23 +1070,13 @@
btcoexist->btc_fill_h2c(btcoexist, 0x60, 5, h2c_parameter);
}
-static void btc8723b2ant_sw_mechanism1(struct btc_coexist *btcoexist,
- bool shrink_rx_lpf, bool low_penalty_ra,
- bool limited_dig, bool bt_lna_constrain)
+static void btc8723b2ant_sw_mechanism(struct btc_coexist *btcoexist,
+ bool shrink_rx_lpf, bool low_penalty_ra,
+ bool limited_dig, bool bt_lna_constrain)
{
- btc8723b2ant_rf_shrink(btcoexist, NORMAL_EXEC, shrink_rx_lpf);
btc8723b2ant_low_penalty_ra(btcoexist, NORMAL_EXEC, low_penalty_ra);
}
-static void btc8723b2ant_sw_mechanism2(struct btc_coexist *btcoexist,
- bool agc_table_shift, bool adc_backoff,
- bool sw_dac_swing, u32 dac_swing_lvl)
-{
- btc8723b2ant_agc_table(btcoexist, NORMAL_EXEC, agc_table_shift);
- btc8723b2ant_dac_swing(btcoexist, NORMAL_EXEC, sw_dac_swing,
- dac_swing_lvl);
-}
-
static void btc8723b2ant_set_ant_path(struct btc_coexist *btcoexist,
u8 antpos_type, bool init_hwcfg,
bool wifi_off)
@@ -1189,44 +1094,66 @@
use_ext_switch = true;
if (init_hwcfg) {
- /* 0x4c[23] = 0, 0x4c[24] = 1 Antenna control by WL/BT */
- u32tmp = btcoexist->btc_read_4byte(btcoexist, 0x4c);
- u32tmp &= ~BIT23;
- u32tmp |= BIT24;
- btcoexist->btc_write_4byte(btcoexist, 0x4c, u32tmp);
-
+ btcoexist->btc_write_1byte_bitmask(btcoexist, 0x39, 0x8, 0x1);
btcoexist->btc_write_1byte(btcoexist, 0x974, 0xff);
btcoexist->btc_write_1byte_bitmask(btcoexist, 0x944, 0x3, 0x3);
btcoexist->btc_write_1byte(btcoexist, 0x930, 0x77);
btcoexist->btc_write_1byte_bitmask(btcoexist, 0x67, 0x20, 0x1);
- /* Force GNT_BT to low */
- btcoexist->btc_write_1byte_bitmask(btcoexist, 0x765, 0x18, 0x0);
+ if (fw_ver >= 0x180000) {
+ /* Use H2C to set GNT_BT to High to avoid A2DP click */
+ h2c_parameter[0] = 1;
+ btcoexist->btc_fill_h2c(btcoexist, 0x6E, 1,
+ h2c_parameter);
+ } else {
+ btcoexist->btc_write_1byte(btcoexist, 0x765, 0x18);
+ }
+
+ btcoexist->btc_write_4byte(btcoexist, 0x948, 0x0);
+
+ /* WiFi TRx Mask off */
+ btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A,
+ 0x1, 0xfffff, 0x0);
if (board_info->btdm_ant_pos == BTC_ANTENNA_AT_MAIN_PORT) {
/* tell firmware "no antenna inverse" */
h2c_parameter[0] = 0;
- h2c_parameter[1] = 1; /* ext switch type */
- btcoexist->btc_fill_h2c(btcoexist, 0x65, 2,
- h2c_parameter);
- btcoexist->btc_write_2byte(btcoexist, 0x948, 0x0);
} else {
/* tell firmware "antenna inverse" */
h2c_parameter[0] = 1;
- h2c_parameter[1] = 1; /* ext switch type */
- btcoexist->btc_fill_h2c(btcoexist, 0x65, 2,
+ }
+
+ if (use_ext_switch) {
+ /* ext switch type */
+ h2c_parameter[1] = 1;
+ } else {
+ /* int switch type */
+ h2c_parameter[1] = 0;
+ }
+ btcoexist->btc_fill_h2c(btcoexist, 0x65, 2, h2c_parameter);
+ } else {
+ if (fw_ver >= 0x180000) {
+ /* Use H2C to set GNT_BT to "Control by PTA"*/
+ h2c_parameter[0] = 0;
+ btcoexist->btc_fill_h2c(btcoexist, 0x6E, 1,
h2c_parameter);
- btcoexist->btc_write_2byte(btcoexist, 0x948, 0x280);
+ } else {
+ btcoexist->btc_write_1byte(btcoexist, 0x765, 0x0);
}
}
/* ext switch setting */
if (use_ext_switch) {
+ if (init_hwcfg) {
+ /* 0x4c[23] = 0, 0x4c[24] = 1 Ant controlled by WL/BT */
+ u32tmp = btcoexist->btc_read_4byte(btcoexist, 0x4c);
+ u32tmp &= ~BIT23;
+ u32tmp |= BIT24;
+ btcoexist->btc_write_4byte(btcoexist, 0x4c, u32tmp);
+ }
+
/* fixed internal switch S1->WiFi, S0->BT */
- if (board_info->btdm_ant_pos == BTC_ANTENNA_AT_MAIN_PORT)
- btcoexist->btc_write_2byte(btcoexist, 0x948, 0x0);
- else
- btcoexist->btc_write_2byte(btcoexist, 0x948, 0x280);
+ btcoexist->btc_write_4byte(btcoexist, 0x948, 0x0);
switch (antpos_type) {
case BTC_ANT_WIFI_AT_MAIN:
@@ -1240,9 +1167,18 @@
0x92c, 0x3, 0x2);
break;
}
- } else { /* internal switch */
- /* fixed ext switch */
- btcoexist->btc_write_1byte_bitmask(btcoexist, 0x92c, 0x3, 0x1);
+ } else {
+ /* internal switch */
+ if (init_hwcfg) {
+ /* 0x4c[23] = 0, 0x4c[24] = 1 Ant controlled by WL/BT */
+ u32tmp = btcoexist->btc_read_4byte(btcoexist, 0x4c);
+ u32tmp |= BIT23;
+ u32tmp &= ~BIT24;
+ btcoexist->btc_write_4byte(btcoexist, 0x4c, u32tmp);
+ }
+
+ /* fixed ext switch, S1->Main, S0->Aux */
+ btcoexist->btc_write_1byte_bitmask(btcoexist, 0x64, 0x1, 0x0);
switch (antpos_type) {
case BTC_ANT_WIFI_AT_MAIN:
/* fixed internal switch S1->WiFi, S0->BT */
@@ -1260,6 +1196,17 @@
bool turn_on, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
+ u8 wifi_rssi_state, bt_rssi_state;
+ s8 wifi_duration_adjust = 0x0;
+ u8 tdma_byte4_modify = 0x0;
+ u8 tmp = BT_8723B_2ANT_WIFI_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
+
+ wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist, 0, 2, tmp, 0);
+ tmp = BT_8723B_2ANT_BT_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
+ bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, tmp, 0);
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], %s turn %s PS TDMA, type=%d\n",
@@ -1268,6 +1215,15 @@
coex_dm->cur_ps_tdma_on = turn_on;
coex_dm->cur_ps_tdma = type;
+ if (!(BTC_RSSI_HIGH(wifi_rssi_state) &&
+ BTC_RSSI_HIGH(bt_rssi_state)) && turn_on) {
+ /* for WiFi RSSI low or BT RSSI low */
+ type = type + 100;
+ coex_dm->is_switch_to_1dot5_ant = true;
+ } else {
+ coex_dm->is_switch_to_1dot5_ant = false;
+ }
+
if (!force_exec) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], bPrePsTdmaOn = %d, bCurPsTdmaOn = %d!!\n",
@@ -1280,83 +1236,131 @@
(coex_dm->pre_ps_tdma == coex_dm->cur_ps_tdma))
return;
}
+
+ if (coex_sta->scan_ap_num <= 5) {
+ if (coex_sta->a2dp_bit_pool >= 45)
+ wifi_duration_adjust = -15;
+ else if (coex_sta->a2dp_bit_pool >= 35)
+ wifi_duration_adjust = -10;
+ else
+ wifi_duration_adjust = 5;
+ } else if (coex_sta->scan_ap_num <= 20) {
+ if (coex_sta->a2dp_bit_pool >= 45)
+ wifi_duration_adjust = -15;
+ else if (coex_sta->a2dp_bit_pool >= 35)
+ wifi_duration_adjust = -10;
+ else
+ wifi_duration_adjust = 0;
+ } else if (coex_sta->scan_ap_num <= 40) {
+ if (coex_sta->a2dp_bit_pool >= 45)
+ wifi_duration_adjust = -15;
+ else if (coex_sta->a2dp_bit_pool >= 35)
+ wifi_duration_adjust = -10;
+ else
+ wifi_duration_adjust = -5;
+ } else {
+ if (coex_sta->a2dp_bit_pool >= 45)
+ wifi_duration_adjust = -15;
+ else if (coex_sta->a2dp_bit_pool >= 35)
+ wifi_duration_adjust = -10;
+ else
+ wifi_duration_adjust = -10;
+ }
+
+ if ((bt_link_info->slave_role) && (bt_link_info->a2dp_exist))
+ /* 0x778 = 0x1 at wifi slot (no blocking BT Low-Pri pkts) */
+ tdma_byte4_modify = 0x1;
+
if (turn_on) {
switch (type) {
case 1:
default:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1a,
- 0x1a, 0xe1, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(
+ btcoexist, 0xe3, 0x3c,
+ 0x03, 0xf1, 0x90 | tdma_byte4_modify);
break;
case 2:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x12,
- 0x12, 0xe1, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(
+ btcoexist, 0xe3, 0x2d,
+ 0x03, 0xf1, 0x90 | tdma_byte4_modify);
break;
case 3:
- /* This call breaks BT when wireless is active -
- * comment it out for now until a better fix is found:
- * btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1c,
- * 0x3, 0xf1, 0x90);
- */
+ btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1c,
+ 0x3, 0xf1,
+ 0x90 | tdma_byte4_modify);
break;
case 4:
btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x10,
- 0x03, 0xf1, 0x90);
+ 0x03, 0xf1,
+ 0x90 | tdma_byte4_modify);
break;
case 5:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1a,
- 0x1a, 0x60, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(
+ btcoexist, 0xe3, 0x3c,
+ 0x3, 0x70, 0x90 | tdma_byte4_modify);
break;
case 6:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x12,
- 0x12, 0x60, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(
+ btcoexist, 0xe3, 0x2d,
+ 0x3, 0x70, 0x90 | tdma_byte4_modify);
break;
case 7:
btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1c,
- 0x3, 0x70, 0x90);
+ 0x3, 0x70,
+ 0x90 | tdma_byte4_modify);
break;
case 8:
btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xa3, 0x10,
- 0x3, 0x70, 0x90);
+ 0x3, 0x70,
+ 0x90 | tdma_byte4_modify);
break;
case 9:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1a,
- 0x1a, 0xe1, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(
+ btcoexist, 0xe3, 0x3c + wifi_duration_adjust,
+ 0x03, 0xf1, 0x90 | tdma_byte4_modify);
break;
case 10:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x12,
- 0x12, 0xe1, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(
+ btcoexist, 0xe3, 0x2d,
+ 0x03, 0xf1, 0x90 | tdma_byte4_modify);
break;
case 11:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0xa,
- 0xa, 0xe1, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1c,
+ 0x3, 0xf1,
+ 0x90 | tdma_byte4_modify);
break;
case 12:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x5,
- 0x5, 0xe1, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x10,
+ 0x3, 0xf1,
+ 0x90 | tdma_byte4_modify);
break;
case 13:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1a,
- 0x1a, 0x60, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(
+ btcoexist, 0xe3, 0x3c,
+ 0x3, 0x70, 0x90 | tdma_byte4_modify);
break;
case 14:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x12,
- 0x12, 0x60, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(
+ btcoexist, 0xe3, 0x2d,
+ 0x3, 0x70, 0x90 | tdma_byte4_modify);
break;
case 15:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0xa,
- 0xa, 0x60, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1c,
+ 0x3, 0x70,
+ 0x90 | tdma_byte4_modify);
break;
case 16:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x5,
- 0x5, 0x60, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x10,
+ 0x3, 0x70,
+ 0x90 | tdma_byte4_modify);
break;
case 17:
btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xa3, 0x2f,
0x2f, 0x60, 0x90);
break;
case 18:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x5,
- 0x5, 0xe1, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x5, 0x5,
+ 0xe1, 0x90);
break;
case 19:
btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x25,
@@ -1370,9 +1374,63 @@
btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x15,
0x03, 0x70, 0x90);
break;
+
+ case 23:
+ case 123:
+ btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x35,
+ 0x03, 0x71, 0x10);
+ break;
case 71:
- btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1a,
- 0x1a, 0xe1, 0x90);
+ btc8723b2ant_set_fw_ps_tdma(
+ btcoexist, 0xe3, 0x3c + wifi_duration_adjust,
+ 0x03, 0xf1, 0x90);
+ break;
+ case 101:
+ case 105:
+ case 113:
+ case 171:
+ btc8723b2ant_set_fw_ps_tdma(
+ btcoexist, 0xd3, 0x3a + wifi_duration_adjust,
+ 0x03, 0x70, 0x50 | tdma_byte4_modify);
+ break;
+ case 102:
+ case 106:
+ case 110:
+ case 114:
+ btc8723b2ant_set_fw_ps_tdma(
+ btcoexist, 0xd3, 0x2d + wifi_duration_adjust,
+ 0x03, 0x70, 0x50 | tdma_byte4_modify);
+ break;
+ case 103:
+ case 107:
+ case 111:
+ case 115:
+ btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xd3, 0x1c,
+ 0x03, 0x70,
+ 0x50 | tdma_byte4_modify);
+ break;
+ case 104:
+ case 108:
+ case 112:
+ case 116:
+ btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xd3, 0x10,
+ 0x03, 0x70,
+ 0x50 | tdma_byte4_modify);
+ break;
+ case 109:
+ btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x3c,
+ 0x03, 0xf1,
+ 0x90 | tdma_byte4_modify);
+ break;
+ case 121:
+ btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x15,
+ 0x03, 0x70,
+ 0x90 | tdma_byte4_modify);
+ break;
+ case 22:
+ case 122:
+ btc8723b2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x35,
+ 0x03, 0x71, 0x11);
break;
}
} else {
@@ -1398,62 +1456,202 @@
coex_dm->pre_ps_tdma = coex_dm->cur_ps_tdma;
}
+static void btc8723b2ant_ps_tdma_check_for_power_save_state(
+ struct btc_coexist *btcoexist, bool new_ps_state)
+{
+ u8 lps_mode = 0x0;
+
+ btcoexist->btc_get(btcoexist, BTC_GET_U1_LPS_MODE, &lps_mode);
+
+ if (lps_mode) {
+ /* already under LPS state */
+ if (new_ps_state) {
+ /* keep state under LPS, do nothing. */
+ } else {
+ /* will leave LPS state, turn off psTdma first */
+ btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+ }
+ } else {
+ /* NO PS state */
+ if (new_ps_state) {
+ /* will enter LPS state, turn off psTdma first */
+ btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+ } else {
+ /* keep state under NO PS state, do nothing. */
+ }
+ }
+}
+
+static void btc8723b2ant_power_save_state(struct btc_coexist *btcoexist,
+ u8 ps_type, u8 lps_val, u8 rpwm_val)
+{
+ bool low_pwr_disable = false;
+
+ switch (ps_type) {
+ case BTC_PS_WIFI_NATIVE:
+ /* recover to original 32k low power setting */
+ low_pwr_disable = false;
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
+ &low_pwr_disable);
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_NORMAL_LPS, NULL);
+ coex_sta->force_lps_on = false;
+ break;
+ case BTC_PS_LPS_ON:
+ btc8723b2ant_ps_tdma_check_for_power_save_state(btcoexist,
+ true);
+ btc8723b2ant_lps_rpwm(btcoexist, NORMAL_EXEC, lps_val,
+ rpwm_val);
+ /* when coex force to enter LPS, do not enter 32k low power */
+ low_pwr_disable = true;
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
+ &low_pwr_disable);
+ /* power save must executed before psTdma */
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_ENTER_LPS, NULL);
+ coex_sta->force_lps_on = true;
+ break;
+ case BTC_PS_LPS_OFF:
+ btc8723b2ant_ps_tdma_check_for_power_save_state(btcoexist,
+ false);
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_LEAVE_LPS, NULL);
+ coex_sta->force_lps_on = false;
+ break;
+ default:
+ break;
+ }
+}
+
static void btc8723b2ant_coex_alloff(struct btc_coexist *btcoexist)
{
/* fw all off */
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
/* sw all off */
- btc8723b2ant_sw_mechanism1(btcoexist, false, false, false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false, false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false, false, false);
/* hw all off */
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 0);
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
}
static void btc8723b2ant_init_coex_dm(struct btc_coexist *btcoexist)
{
/* force to reset coex mechanism*/
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
btc8723b2ant_ps_tdma(btcoexist, FORCE_EXEC, false, 1);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, FORCE_EXEC, 6);
- btc8723b2ant_dec_bt_pwr(btcoexist, FORCE_EXEC, false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, FORCE_EXEC, 0);
- btc8723b2ant_sw_mechanism1(btcoexist, false, false, false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false, false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false, false, false);
+
+ coex_sta->pop_event_cnt = 0;
}
static void btc8723b2ant_action_bt_inquiry(struct btc_coexist *btcoexist)
{
+ struct rtl_priv *rtlpriv = btcoexist->adapter;
bool wifi_connected = false;
bool low_pwr_disable = true;
+ bool scan = false, link = false, roam = false;
btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
&low_pwr_disable);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
&wifi_connected);
- if (wifi_connected) {
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 7);
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 3);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_SCAN, &scan);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_LINK, &link);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_ROAM, &roam);
+
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+
+ if (coex_sta->bt_abnormal_scan) {
+ btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 23);
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 3);
+ } else if (scan || link || roam) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Wifi link process + BT Inq/Page!!\n");
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 15);
+ btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 22);
+ } else if (wifi_connected) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Wifi connected + BT Inq/Page!!\n");
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 15);
+ btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 22);
} else {
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 0);
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
}
btc8723b2ant_fw_dac_swing_lvl(btcoexist, FORCE_EXEC, 6);
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
- btc8723b2ant_sw_mechanism1(btcoexist, false, false, false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false, false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false, false, false);
+}
- coex_dm->need_recover_0x948 = true;
- coex_dm->backup_0x948 = btcoexist->btc_read_2byte(btcoexist, 0x948);
+static void btc8723b2ant_action_wifi_link_process(struct btc_coexist
+ *btcoexist)
+{
+ struct rtl_priv *rtlpriv = btcoexist->adapter;
+ u32 u32tmp;
+ u8 u8tmpa, u8tmpb;
- btc8723b2ant_set_ant_path(btcoexist, BTC_ANT_WIFI_AT_AUX,
- false, false);
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 15);
+ btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 22);
+
+ btc8723b2ant_sw_mechanism(btcoexist, false, false, false, false);
+
+ u32tmp = btcoexist->btc_read_4byte(btcoexist, 0x948);
+ u8tmpa = btcoexist->btc_read_1byte(btcoexist, 0x765);
+ u8tmpb = btcoexist->btc_read_1byte(btcoexist, 0x76e);
+
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], 0x948 = 0x%x, 0x765 = 0x%x, 0x76e = 0x%x\n",
+ u32tmp, u8tmpa, u8tmpb);
+}
+
+static bool btc8723b2ant_action_wifi_idle_process(struct btc_coexist *btcoexist)
+{
+ struct rtl_priv *rtlpriv = btcoexist->adapter;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
+ u8 ap_num = 0;
+ u8 tmp = BT_8723B_2ANT_WIFI_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset - coex_dm->switch_thres_offset;
+
+ wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8723b2ant_wifi_rssi_state(btcoexist, 1, 2,
+ tmp, 0);
+ tmp = BT_8723B_2ANT_BT_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset - coex_dm->switch_thres_offset;
+ bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, tmp, 0);
+
+ btcoexist->btc_get(btcoexist, BTC_GET_U1_AP_NUM, &ap_num);
+
+ /* office environment */
+ if (BTC_RSSI_HIGH(wifi_rssi_state1) && (coex_sta->hid_exist) &&
+ (coex_sta->a2dp_exist)) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Wifi idle process for BT HID+A2DP exist!!\n");
+
+ btc8723b2ant_dac_swing(btcoexist, NORMAL_EXEC, true, 0x6);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+
+ /* sw all off */
+ btc8723b2ant_sw_mechanism(btcoexist, false, false, false,
+ false);
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+
+ return true;
+ }
+
+ btc8723b2ant_dac_swing(btcoexist, NORMAL_EXEC, true, 0x18);
+ return false;
}
static bool btc8723b2ant_is_common_action(struct btc_coexist *btcoexist)
@@ -1472,21 +1670,21 @@
low_pwr_disable = false;
btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
&low_pwr_disable);
+ btc8723b2ant_limited_rx(btcoexist, NORMAL_EXEC,
+ false, false, 0x8);
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Wifi non-connected idle!!\n");
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff,
0x0);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 0);
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
- btc8723b2ant_sw_mechanism1(btcoexist, false, false, false,
- false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false, false,
- 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false, false,
+ false);
common = true;
} else {
@@ -1496,23 +1694,23 @@
btcoexist->btc_set(btcoexist,
BTC_SET_ACT_DISABLE_LOW_POWER,
&low_pwr_disable);
+ btc8723b2ant_limited_rx(btcoexist, NORMAL_EXEC,
+ false, false, 0x8);
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Wifi connected + BT non connected-idle!!\n");
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1,
0xfffff, 0x0);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 0);
+ btc8723b2ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 0);
btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC,
0xb);
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC,
- false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
- btc8723b2ant_sw_mechanism1(btcoexist, false, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false,
+ false, false);
common = true;
} else if (BT_8723B_2ANT_BT_STATUS_CONNECTED_IDLE ==
@@ -1526,20 +1724,20 @@
return false;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Wifi connected + BT connected-idle!!\n");
+ btc8723b2ant_limited_rx(btcoexist, NORMAL_EXEC,
+ false, false, 0x8);
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1,
0xfffff, 0x0);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 0);
+ btc8723b2ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 0);
btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC,
0xb);
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC,
- false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
- btc8723b2ant_sw_mechanism1(btcoexist, true, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, false,
+ false, false);
common = true;
} else {
@@ -1553,36 +1751,12 @@
"[BTCoex], Wifi Connected-Busy + BT Busy!!\n");
common = false;
} else {
- if (bt_hs_on)
- return false;
-
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Wifi Connected-Idle + BT Busy!!\n");
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A,
- 0x1, 0xfffff, 0x0);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC,
- 7);
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 21);
- btc8723b2ant_fw_dac_swing_lvl(btcoexist,
- NORMAL_EXEC,
- 0xb);
- if (btc8723b_need_dec_pwr(btcoexist))
- btc8723b2ant_dec_bt_pwr(btcoexist,
- NORMAL_EXEC,
- true);
- else
- btc8723b2ant_dec_bt_pwr(btcoexist,
- NORMAL_EXEC,
- false);
- btc8723b2ant_sw_mechanism1(btcoexist, false,
- false, false,
- false);
- btc8723b2ant_sw_mechanism2(btcoexist, false,
- false, false,
- 0x18);
- common = true;
+ common =
+ btc8723b2ant_action_wifi_idle_process(
+ btcoexist);
}
}
}
@@ -1590,550 +1764,6 @@
return common;
}
-static void set_tdma_int1(struct btc_coexist *btcoexist, bool tx_pause,
- s32 result)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- /* Set PS TDMA for max interval == 1 */
- if (tx_pause) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 1\n");
-
- if (coex_dm->cur_ps_tdma == 71) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 5);
- coex_dm->tdma_adj_type = 5;
- } else if (coex_dm->cur_ps_tdma == 1) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 5);
- coex_dm->tdma_adj_type = 5;
- } else if (coex_dm->cur_ps_tdma == 2) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 3) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 4) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- }
-
- if (coex_dm->cur_ps_tdma == 9) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 13);
- coex_dm->tdma_adj_type = 13;
- } else if (coex_dm->cur_ps_tdma == 10) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 11) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 12) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
-
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 5) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 6) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- } else if (coex_dm->cur_ps_tdma == 13) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 14) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 8) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 6) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 5);
- coex_dm->tdma_adj_type = 5;
- } else if (coex_dm->cur_ps_tdma == 16) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 14) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 13);
- coex_dm->tdma_adj_type = 13;
- }
- }
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 0\n");
- if (coex_dm->cur_ps_tdma == 5) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 71);
- coex_dm->tdma_adj_type = 71;
- } else if (coex_dm->cur_ps_tdma == 6) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 7) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 8) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 4);
- coex_dm->tdma_adj_type = 4;
- }
-
- if (coex_dm->cur_ps_tdma == 13) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 9);
- coex_dm->tdma_adj_type = 9;
- } else if (coex_dm->cur_ps_tdma == 14) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 15) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 16) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 12);
- coex_dm->tdma_adj_type = 12;
- }
-
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 71) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 1);
- coex_dm->tdma_adj_type = 1;
- } else if (coex_dm->cur_ps_tdma == 1) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- } else if (coex_dm->cur_ps_tdma == 9) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 10) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 4) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 1);
- coex_dm->tdma_adj_type = 1;
- } else if (coex_dm->cur_ps_tdma == 1) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 71);
- coex_dm->tdma_adj_type = 71;
- } else if (coex_dm->cur_ps_tdma == 12) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 10) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
- }
- }
- }
-}
-
-static void set_tdma_int2(struct btc_coexist *btcoexist, bool tx_pause,
- s32 result)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- /* Set PS TDMA for max interval == 2 */
- if (tx_pause) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 1\n");
- if (coex_dm->cur_ps_tdma == 1) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 2) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 3) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 4) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 8);
- coex_dm->tdma_adj_type = 8;
- }
- if (coex_dm->cur_ps_tdma == 9) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 10) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 11) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 12) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 5) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 6) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- } else if (coex_dm->cur_ps_tdma == 13) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 14) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 8) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 6) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 16) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 14) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- }
- }
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 0\n");
- if (coex_dm->cur_ps_tdma == 5) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 6) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 7) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 8) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 4);
- coex_dm->tdma_adj_type = 4;
- }
- if (coex_dm->cur_ps_tdma == 13) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 14) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 15) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 16) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 1) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- } else if (coex_dm->cur_ps_tdma == 9) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 10) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 4) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 12) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 10) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- }
- }
- }
-}
-
-static void set_tdma_int3(struct btc_coexist *btcoexist, bool tx_pause,
- s32 result)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- /* Set PS TDMA for max interval == 3 */
- if (tx_pause) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 1\n");
- if (coex_dm->cur_ps_tdma == 1) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 2) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 3) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 4) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 8);
- coex_dm->tdma_adj_type = 8;
- }
- if (coex_dm->cur_ps_tdma == 9) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 10) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 11) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 12) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 5) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 6) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- } else if (coex_dm->cur_ps_tdma == 13) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 14) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 8) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 6) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 16) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 14) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- }
- }
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 0\n");
- if (coex_dm->cur_ps_tdma == 5) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 6) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 7) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 8) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 4);
- coex_dm->tdma_adj_type = 4;
- }
- if (coex_dm->cur_ps_tdma == 13) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 14) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 15) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 16) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 1) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 2) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- } else if (coex_dm->cur_ps_tdma == 9) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 10) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 4) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 2) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 12) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 10) {
- btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- }
- }
- }
-}
-
static void btc8723b2ant_tdma_duration_adjust(struct btc_coexist *btcoexist,
bool sco_hid, bool tx_pause,
u8 max_interval)
@@ -2157,34 +1787,44 @@
btc8723b2ant_ps_tdma(btcoexist,
NORMAL_EXEC,
true, 13);
- coex_dm->tdma_adj_type = 13;
+ coex_dm->ps_tdma_du_adj_type = 13;
} else if (max_interval == 2) {
btc8723b2ant_ps_tdma(btcoexist,
NORMAL_EXEC,
true, 14);
- coex_dm->tdma_adj_type = 14;
+ coex_dm->ps_tdma_du_adj_type = 14;
+ } else if (max_interval == 3) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type = 15;
} else {
btc8723b2ant_ps_tdma(btcoexist,
NORMAL_EXEC,
true, 15);
- coex_dm->tdma_adj_type = 15;
+ coex_dm->ps_tdma_du_adj_type = 15;
}
} else {
if (max_interval == 1) {
btc8723b2ant_ps_tdma(btcoexist,
NORMAL_EXEC,
true, 9);
- coex_dm->tdma_adj_type = 9;
+ coex_dm->ps_tdma_du_adj_type = 9;
} else if (max_interval == 2) {
btc8723b2ant_ps_tdma(btcoexist,
NORMAL_EXEC,
true, 10);
- coex_dm->tdma_adj_type = 10;
+ coex_dm->ps_tdma_du_adj_type = 10;
+ } else if (max_interval == 3) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type = 11;
} else {
btc8723b2ant_ps_tdma(btcoexist,
NORMAL_EXEC,
true, 11);
- coex_dm->tdma_adj_type = 11;
+ coex_dm->ps_tdma_du_adj_type = 11;
}
}
} else {
@@ -2193,34 +1833,44 @@
btc8723b2ant_ps_tdma(btcoexist,
NORMAL_EXEC,
true, 5);
- coex_dm->tdma_adj_type = 5;
+ coex_dm->ps_tdma_du_adj_type = 5;
} else if (max_interval == 2) {
btc8723b2ant_ps_tdma(btcoexist,
NORMAL_EXEC,
true, 6);
- coex_dm->tdma_adj_type = 6;
+ coex_dm->ps_tdma_du_adj_type = 6;
+ } else if (max_interval == 3) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type = 7;
} else {
btc8723b2ant_ps_tdma(btcoexist,
NORMAL_EXEC,
true, 7);
- coex_dm->tdma_adj_type = 7;
+ coex_dm->ps_tdma_du_adj_type = 7;
}
} else {
if (max_interval == 1) {
btc8723b2ant_ps_tdma(btcoexist,
NORMAL_EXEC,
true, 1);
- coex_dm->tdma_adj_type = 1;
+ coex_dm->ps_tdma_du_adj_type = 1;
} else if (max_interval == 2) {
btc8723b2ant_ps_tdma(btcoexist,
NORMAL_EXEC,
true, 2);
- coex_dm->tdma_adj_type = 2;
+ coex_dm->ps_tdma_du_adj_type = 2;
+ } else if (max_interval == 3) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type = 3;
} else {
btc8723b2ant_ps_tdma(btcoexist,
NORMAL_EXEC,
true, 3);
- coex_dm->tdma_adj_type = 3;
+ coex_dm->ps_tdma_du_adj_type = 3;
}
}
}
@@ -2234,6 +1884,11 @@
} else {
/*accquire the BT TRx retry count from BT_Info byte2*/
retry_count = coex_sta->bt_retry_cnt;
+
+ if ((coex_sta->low_priority_tx) > 1050 ||
+ (coex_sta->low_priority_rx) > 1250)
+ retry_count++;
+
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], retry_count = %d\n", retry_count);
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -2250,6 +1905,9 @@
dn = 0;
if (up >= n) {
+ /* if retry count during continuous n*2
+ * seconds is 0, enlarge WiFi duration
+ */
wait_count = 0;
n = 3;
up = 0;
@@ -2266,12 +1924,20 @@
up = 0;
if (dn == 2) {
+ /* if continuous 2 retry count(every 2
+ * seconds) >0 and < 3, reduce WiFi duration
+ */
if (wait_count <= 2)
+ /* avoid loop between the two levels */
m++;
else
m = 1;
if (m >= 20)
+ /* maximum of m = 20 ' will recheck if
+ * need to adjust wifi duration in
+ * maximum time interval 120 seconds
+ */
m = 20;
n = 3 * m;
@@ -2283,12 +1949,20 @@
"[BTCoex], Decrease wifi duration for retry_counter<3!!\n");
}
} else {
+ /* retry count > 3, once retry count > 3, to reduce
+ * WiFi duration
+ */
if (wait_count == 1)
+ /* to avoid loop between the two levels */
m++;
else
m = 1;
if (m >= 20)
+ /* maximum of m = 20 ' will recheck if need to
+ * adjust wifi duration in maximum time interval
+ * 120 seconds
+ */
m = 20;
n = 3 * m;
@@ -2302,22 +1976,765 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], max Interval = %d\n", max_interval);
- if (max_interval == 1)
- set_tdma_int1(btcoexist, tx_pause, result);
- else if (max_interval == 2)
- set_tdma_int2(btcoexist, tx_pause, result);
- else if (max_interval == 3)
- set_tdma_int3(btcoexist, tx_pause, result);
+ if (max_interval == 1) {
+ if (tx_pause) {
+ if (coex_dm->cur_ps_tdma == 71) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 5);
+ coex_dm->ps_tdma_du_adj_type = 5;
+ } else if (coex_dm->cur_ps_tdma == 1) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 5);
+ coex_dm->ps_tdma_du_adj_type = 5;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 6);
+ coex_dm->ps_tdma_du_adj_type = 6;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type = 7;
+ } else if (coex_dm->cur_ps_tdma == 4) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 8);
+ coex_dm->ps_tdma_du_adj_type = 8;
+ }
+ if (coex_dm->cur_ps_tdma == 9) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 13);
+ coex_dm->ps_tdma_du_adj_type = 13;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 14);
+ coex_dm->ps_tdma_du_adj_type = 14;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type = 15;
+ } else if (coex_dm->cur_ps_tdma == 12) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 16);
+ coex_dm->ps_tdma_du_adj_type = 16;
+ }
+
+ if (result == -1) {
+ if (coex_dm->cur_ps_tdma == 5) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 6);
+ coex_dm->ps_tdma_du_adj_type =
+ 6;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 8);
+ coex_dm->ps_tdma_du_adj_type =
+ 8;
+ } else if (coex_dm->cur_ps_tdma == 13) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 14);
+ coex_dm->ps_tdma_du_adj_type =
+ 14;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 16);
+ coex_dm->ps_tdma_du_adj_type =
+ 16;
+ }
+ } else if (result == 1) {
+ if (coex_dm->cur_ps_tdma == 8) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 6);
+ coex_dm->ps_tdma_du_adj_type =
+ 6;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 5);
+ coex_dm->ps_tdma_du_adj_type =
+ 5;
+ } else if (coex_dm->cur_ps_tdma == 16) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 14);
+ coex_dm->ps_tdma_du_adj_type =
+ 14;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 13);
+ coex_dm->ps_tdma_du_adj_type =
+ 13;
+ }
+ }
+ } else {
+ if (coex_dm->cur_ps_tdma == 5) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 71);
+ coex_dm->ps_tdma_du_adj_type = 71;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 2);
+ coex_dm->ps_tdma_du_adj_type = 2;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type = 3;
+ } else if (coex_dm->cur_ps_tdma == 8) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 4);
+ coex_dm->ps_tdma_du_adj_type = 4;
+ }
+ if (coex_dm->cur_ps_tdma == 13) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 9);
+ coex_dm->ps_tdma_du_adj_type = 9;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 10);
+ coex_dm->ps_tdma_du_adj_type = 10;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type = 11;
+ } else if (coex_dm->cur_ps_tdma == 16) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 12);
+ coex_dm->ps_tdma_du_adj_type = 12;
+ }
+
+ if (result == -1) {
+ if (coex_dm->cur_ps_tdma == 71) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 1);
+ coex_dm->ps_tdma_du_adj_type =
+ 1;
+ } else if (coex_dm->cur_ps_tdma == 1) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 2);
+ coex_dm->ps_tdma_du_adj_type =
+ 2;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 4);
+ coex_dm->ps_tdma_du_adj_type =
+ 4;
+ } else if (coex_dm->cur_ps_tdma == 9) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 10);
+ coex_dm->ps_tdma_du_adj_type =
+ 10;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 12);
+ coex_dm->ps_tdma_du_adj_type =
+ 12;
+ }
+ } else if (result == 1) {
+ if (coex_dm->cur_ps_tdma == 4) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 2);
+ coex_dm->ps_tdma_du_adj_type =
+ 2;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 1);
+ coex_dm->ps_tdma_du_adj_type =
+ 1;
+ } else if (coex_dm->cur_ps_tdma == 1) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 71);
+ coex_dm->ps_tdma_du_adj_type =
+ 71;
+ } else if (coex_dm->cur_ps_tdma == 12) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 10);
+ coex_dm->ps_tdma_du_adj_type =
+ 10;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 9);
+ coex_dm->ps_tdma_du_adj_type =
+ 9;
+ }
+ }
+ }
+ } else if (max_interval == 2) {
+ if (tx_pause) {
+ if (coex_dm->cur_ps_tdma == 1) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 6);
+ coex_dm->ps_tdma_du_adj_type = 6;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 6);
+ coex_dm->ps_tdma_du_adj_type = 6;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type = 7;
+ } else if (coex_dm->cur_ps_tdma == 4) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 8);
+ coex_dm->ps_tdma_du_adj_type = 8;
+ }
+ if (coex_dm->cur_ps_tdma == 9) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 14);
+ coex_dm->ps_tdma_du_adj_type = 14;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 14);
+ coex_dm->ps_tdma_du_adj_type = 14;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type = 15;
+ } else if (coex_dm->cur_ps_tdma == 12) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 16);
+ coex_dm->ps_tdma_du_adj_type = 16;
+ }
+ if (result == -1) {
+ if (coex_dm->cur_ps_tdma == 5) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 6);
+ coex_dm->ps_tdma_du_adj_type =
+ 6;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 8);
+ coex_dm->ps_tdma_du_adj_type =
+ 8;
+ } else if (coex_dm->cur_ps_tdma == 13) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 14);
+ coex_dm->ps_tdma_du_adj_type =
+ 14;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 16);
+ coex_dm->ps_tdma_du_adj_type =
+ 16;
+ }
+ } else if (result == 1) {
+ if (coex_dm->cur_ps_tdma == 8) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 6);
+ coex_dm->ps_tdma_du_adj_type =
+ 6;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 6);
+ coex_dm->ps_tdma_du_adj_type =
+ 6;
+ } else if (coex_dm->cur_ps_tdma == 16) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 14);
+ coex_dm->ps_tdma_du_adj_type =
+ 14;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 14);
+ coex_dm->ps_tdma_du_adj_type =
+ 14;
+ }
+ }
+ } else {
+ if (coex_dm->cur_ps_tdma == 5) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 2);
+ coex_dm->ps_tdma_du_adj_type = 2;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 2);
+ coex_dm->ps_tdma_du_adj_type = 2;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type = 3;
+ } else if (coex_dm->cur_ps_tdma == 8) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 4);
+ coex_dm->ps_tdma_du_adj_type = 4;
+ }
+ if (coex_dm->cur_ps_tdma == 13) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 10);
+ coex_dm->ps_tdma_du_adj_type = 10;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 10);
+ coex_dm->ps_tdma_du_adj_type = 10;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type = 11;
+ } else if (coex_dm->cur_ps_tdma == 16) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 12);
+ coex_dm->ps_tdma_du_adj_type = 12;
+ }
+ if (result == -1) {
+ if (coex_dm->cur_ps_tdma == 1) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 2);
+ coex_dm->ps_tdma_du_adj_type =
+ 2;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 4);
+ coex_dm->ps_tdma_du_adj_type =
+ 4;
+ } else if (coex_dm->cur_ps_tdma == 9) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 10);
+ coex_dm->ps_tdma_du_adj_type =
+ 10;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 12);
+ coex_dm->ps_tdma_du_adj_type =
+ 12;
+ }
+ } else if (result == 1) {
+ if (coex_dm->cur_ps_tdma == 4) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 2);
+ coex_dm->ps_tdma_du_adj_type =
+ 2;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 2);
+ coex_dm->ps_tdma_du_adj_type =
+ 2;
+ } else if (coex_dm->cur_ps_tdma == 12) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 10);
+ coex_dm->ps_tdma_du_adj_type =
+ 10;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 10);
+ coex_dm->ps_tdma_du_adj_type =
+ 10;
+ }
+ }
+ }
+ } else if (max_interval == 3) {
+ if (tx_pause) {
+ if (coex_dm->cur_ps_tdma == 1) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type = 7;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type = 7;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type = 7;
+ } else if (coex_dm->cur_ps_tdma == 4) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 8);
+ coex_dm->ps_tdma_du_adj_type = 8;
+ }
+ if (coex_dm->cur_ps_tdma == 9) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type = 15;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type = 15;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type = 15;
+ } else if (coex_dm->cur_ps_tdma == 12) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 16);
+ coex_dm->ps_tdma_du_adj_type = 16;
+ }
+ if (result == -1) {
+ if (coex_dm->cur_ps_tdma == 5) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 8);
+ coex_dm->ps_tdma_du_adj_type =
+ 8;
+ } else if (coex_dm->cur_ps_tdma == 13) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 16);
+ coex_dm->ps_tdma_du_adj_type =
+ 16;
+ }
+ } else if (result == 1) {
+ if (coex_dm->cur_ps_tdma == 8) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 16) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ }
+ }
+ } else {
+ if (coex_dm->cur_ps_tdma == 5) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type = 3;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type = 3;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type = 3;
+ } else if (coex_dm->cur_ps_tdma == 8) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 4);
+ coex_dm->ps_tdma_du_adj_type = 4;
+ }
+ if (coex_dm->cur_ps_tdma == 13) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type = 11;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type = 11;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type = 11;
+ } else if (coex_dm->cur_ps_tdma == 16) {
+ btc8723b2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC,
+ true, 12);
+ coex_dm->ps_tdma_du_adj_type = 12;
+ }
+ if (result == -1) {
+ if (coex_dm->cur_ps_tdma == 1) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 4);
+ coex_dm->ps_tdma_du_adj_type =
+ 4;
+ } else if (coex_dm->cur_ps_tdma == 9) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 12);
+ coex_dm->ps_tdma_du_adj_type =
+ 12;
+ }
+ } else if (result == 1) {
+ if (coex_dm->cur_ps_tdma == 4) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 12) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8723b2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ }
+ }
+ }
+ }
}
- /*if current PsTdma not match with the recorded one (when scan, dhcp..),
- *then we have to adjust it back to the previous recorded one.
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], max Interval = %d\n", max_interval);
+
+ /* if current PsTdma not match with the recorded one (scan, dhcp, ...),
+ * then we have to adjust it back to the previous recorded one.
*/
- if (coex_dm->cur_ps_tdma != coex_dm->tdma_adj_type) {
+ if (coex_dm->cur_ps_tdma != coex_dm->ps_tdma_du_adj_type) {
bool scan = false, link = false, roam = false;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], PsTdma type dismatch!!!, curPsTdma=%d, recordPsTdma=%d\n",
- coex_dm->cur_ps_tdma, coex_dm->tdma_adj_type);
+ coex_dm->cur_ps_tdma, coex_dm->ps_tdma_du_adj_type);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_SCAN, &scan);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_LINK, &link);
@@ -2325,7 +2742,7 @@
if (!scan && !link && !roam)
btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true,
- coex_dm->tdma_adj_type);
+ coex_dm->ps_tdma_du_adj_type);
else
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], roaming/link/scan is under progress, will adjust next time!!!\n");
@@ -2335,58 +2752,55 @@
/* SCO only or SCO+PAN(HS) */
static void btc8723b2ant_action_sco(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state;
+ u8 wifi_rssi_state, bt_rssi_state;
u32 wifi_bw;
- wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist,
- 0, 2, 15, 0);
+ wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8723b2ant_bt_rssi_state(
+ btcoexist, 2, BT_8723B_2ANT_BT_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset,
+ 0);
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
+ btc8723b2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 4);
- if (btc8723b_need_dec_pwr(btcoexist))
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
else
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- /*for SCO quality at 11b/g mode*/
if (BTC_WIFI_BW_LEGACY == wifi_bw)
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 2);
- else /*for SCO quality & wifi performance balance at 11n mode*/
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 8);
+ /* for SCO quality at 11b/g mode */
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
+ else
+ /* for SCO quality & wifi performance balance at 11n mode */
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 8);
- /*for voice quality */
+ /* for voice quality */
btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 0);
/* sw mechanism */
if (BTC_WIFI_BW_HT40 == wifi_bw) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, true, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- true, 0x4);
+ btc8723b2ant_sw_mechanism(btcoexist, true, true,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, true, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- true, 0x4);
+ btc8723b2ant_sw_mechanism(btcoexist, true, true,
+ false, false);
}
} else {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, false, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- true, 0x4);
+ btc8723b2ant_sw_mechanism(btcoexist, false, true,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, false, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- true, 0x4);
+ btc8723b2ant_sw_mechanism(btcoexist, false, true,
+ false, false);
}
}
}
@@ -2395,26 +2809,32 @@
{
u8 wifi_rssi_state, bt_rssi_state;
u32 wifi_bw;
+ u8 tmp = BT_8723B_2ANT_BT_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
- wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist,
- 0, 2, 15, 0);
- bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, 29, 0);
+ wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, tmp, 0);
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
+ btc8723b2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- if (btc8723b_need_dec_pwr(btcoexist))
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
else
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- if (BTC_WIFI_BW_LEGACY == wifi_bw) /*/for HID at 11b/g mode*/
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 7);
- else /*for HID quality & wifi performance balance at 11n mode*/
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 9);
+ if (wifi_bw == BTC_WIFI_BW_LEGACY)
+ /* for HID at 11b/g mode */
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
+ else
+ /* for HID quality & wifi performance balance at 11n mode */
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 9);
+
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
(bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH))
@@ -2426,44 +2846,36 @@
if (BTC_WIFI_BW_HT40 == wifi_bw) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, true, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, true,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, true, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, true,
+ false, false);
}
} else {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, false, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, true,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, false, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, true,
+ false, false);
}
}
}
-/*A2DP only / PAN(EDR) only/ A2DP+PAN(HS)*/
+/* A2DP only / PAN(EDR) only/ A2DP+PAN(HS) */
static void btc8723b2ant_action_a2dp(struct btc_coexist *btcoexist)
{
u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
u32 wifi_bw;
u8 ap_num = 0;
+ u8 tmp = BT_8723B_2ANT_BT_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
- wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist,
- 0, 2, 15, 0);
- wifi_rssi_state1 = btc8723b2ant_wifi_rssi_state(btcoexist,
- 1, 2, 40, 0);
- bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, 29, 0);
+ wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8723b2ant_wifi_rssi_state(btcoexist, 1, 2, 40, 0);
+ bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, tmp, 0);
btcoexist->btc_get(btcoexist, BTC_GET_U1_AP_NUM, &ap_num);
@@ -2474,35 +2886,40 @@
0x0);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 0);
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
/* sw mechanism */
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
if (BTC_WIFI_BW_HT40 == wifi_bw) {
- btc8723b2ant_sw_mechanism1(btcoexist, true, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- true, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, false,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, false, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- true, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false,
+ false, false);
}
return;
}
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
+ btc8723b2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- if (btc8723b_need_dec_pwr(btcoexist))
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
else
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 7);
+ if (BTC_RSSI_HIGH(wifi_rssi_state1) && BTC_RSSI_HIGH(bt_rssi_state)) {
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ } else {
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 13);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_LPS_ON, 0x50,
+ 0x4);
+ }
if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
(bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH))
@@ -2516,104 +2933,116 @@
if (BTC_WIFI_BW_HT40 == wifi_bw) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, true, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, false,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, true, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, false,
+ false, false);
}
} else {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, false, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, false, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false,
+ false, false);
}
}
}
static void btc8723b2ant_action_a2dp_pan_hs(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
u32 wifi_bw;
+ u8 tmp = BT_8723B_2ANT_WIFI_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
- wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist,
- 0, 2, 15, 0);
+ wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8723b2ant_wifi_rssi_state(btcoexist, 1, 2,
+ tmp, 0);
+ tmp = BT_8723B_2ANT_BT_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
+ bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, tmp, 0);
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
+ btc8723b2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- if (btc8723b_need_dec_pwr(btcoexist))
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
else
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 7);
+ if (BTC_RSSI_HIGH(wifi_rssi_state1) && BTC_RSSI_HIGH(bt_rssi_state)) {
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ } else {
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 13);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_LPS_ON, 0x50,
+ 0x4);
+ }
btc8723b2ant_tdma_duration_adjust(btcoexist, false, true, 2);
/* sw mechanism */
- btcoexist->btc_get(btcoexist,
- BTC_GET_U4_WIFI_BW, &wifi_bw);
+ btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
if (BTC_WIFI_BW_HT40 == wifi_bw) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, true, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, false,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, true, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, false,
+ false, false);
}
} else {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, false, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, false, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false,
+ false, false);
}
}
}
static void btc8723b2ant_action_pan_edr(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
u32 wifi_bw;
+ u8 tmp = BT_8723B_2ANT_WIFI_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
- wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist,
- 0, 2, 15, 0);
- bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, 29, 0);
+ wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8723b2ant_wifi_rssi_state(btcoexist, 1, 2,
+ tmp, 0);
+ tmp = BT_8723B_2ANT_BT_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
+ bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, tmp, 0);
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
+ btc8723b2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- if (btc8723b_need_dec_pwr(btcoexist))
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
else
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 10);
+ if (BTC_RSSI_HIGH(wifi_rssi_state1) && BTC_RSSI_HIGH(bt_rssi_state)) {
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 10);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ } else {
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 13);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_LPS_ON, 0x50,
+ 0x4);
+ }
if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
(bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH))
@@ -2626,109 +3055,109 @@
if (BTC_WIFI_BW_HT40 == wifi_bw) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, true, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, false,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, true, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, false,
+ false, false);
}
} else {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, false, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, false, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false,
+ false, false);
}
}
}
-/*PAN(HS) only*/
+/* PAN(HS) only */
static void btc8723b2ant_action_pan_hs(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
u32 wifi_bw;
+ u8 tmp = BT_8723B_2ANT_WIFI_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
- wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist,
- 0, 2, 15, 0);
+ wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8723b2ant_wifi_rssi_state(btcoexist, 1, 2,
+ tmp, 0);
+ tmp = BT_8723B_2ANT_BT_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
+ bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, tmp, 0);
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
+ btc8723b2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH))
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
else
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 7);
-
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
if (BTC_WIFI_BW_HT40 == wifi_bw) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, true, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, false,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, true, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, false,
+ false, false);
}
} else {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, false, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, false, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false,
+ false, false);
}
}
}
-/*PAN(EDR)+A2DP*/
+/* PAN(EDR) + A2DP */
static void btc8723b2ant_action_pan_edr_a2dp(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
u32 wifi_bw;
+ u8 tmp = BT_8723B_2ANT_WIFI_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
- wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist,
- 0, 2, 15, 0);
- bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, 29, 0);
+ wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8723b2ant_wifi_rssi_state(btcoexist, 1, 2,
+ tmp, 0);
+ tmp = BT_8723B_2ANT_BT_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
+ bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, tmp, 0);
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- if (btc8723b_need_dec_pwr(btcoexist))
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
else
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ if (BTC_RSSI_HIGH(wifi_rssi_state1) && BTC_RSSI_HIGH(bt_rssi_state))
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ else
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_LPS_ON, 0x50,
+ 0x4);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
(bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 12);
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 12);
if (BTC_WIFI_BW_HT40 == wifi_bw)
btc8723b2ant_tdma_duration_adjust(btcoexist, false,
true, 3);
@@ -2736,74 +3165,80 @@
btc8723b2ant_tdma_duration_adjust(btcoexist, false,
false, 3);
} else {
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 7);
- btc8723b2ant_tdma_duration_adjust(btcoexist, false, true, 3);
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
+ btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 3);
}
/* sw mechanism */
if (BTC_WIFI_BW_HT40 == wifi_bw) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, true, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, false,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, true, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, false,
+ false, false);
}
} else {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, false, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, false, false,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, false,
+ false, false);
}
}
}
static void btc8723b2ant_action_pan_edr_hid(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
u32 wifi_bw;
+ u8 tmp = BT_8723B_2ANT_WIFI_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
- wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist,
- 0, 2, 15, 0);
- bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, 29, 0);
+ wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8723b2ant_wifi_rssi_state(btcoexist, 1, 2,
+ tmp, 0);
+ tmp = BT_8723B_2ANT_BT_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
+ bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, tmp, 0);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- if (btc8723b_need_dec_pwr(btcoexist))
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ btc8723b2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
else
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+
+ if (BTC_RSSI_HIGH(wifi_rssi_state1) && BTC_RSSI_HIGH(bt_rssi_state)) {
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ } else {
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 14);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_LPS_ON, 0x50,
+ 0x4);
+ }
if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
(bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
if (BTC_WIFI_BW_HT40 == wifi_bw) {
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC,
3);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 11);
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1,
0xfffff, 0x780);
} else {
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC,
6);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 7);
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1,
0xfffff, 0x0);
}
btc8723b2ant_tdma_duration_adjust(btcoexist, true, false, 2);
} else {
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 11);
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff,
0x0);
btc8723b2ant_tdma_duration_adjust(btcoexist, true, true, 2);
@@ -2813,54 +3248,61 @@
if (BTC_WIFI_BW_HT40 == wifi_bw) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, true, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, true,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, true, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, true,
+ false, false);
}
} else {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, false, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, true,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, false, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, true,
+ false, false);
}
}
}
-/* HID+A2DP+PAN(EDR) */
+/* HID + A2DP + PAN(EDR) */
static void btc8723b2ant_action_hid_a2dp_pan_edr(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
u32 wifi_bw;
+ u8 tmp = BT_8723B_2ANT_WIFI_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
- wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist,
- 0, 2, 15, 0);
- bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, 29, 0);
+ wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8723b2ant_wifi_rssi_state(btcoexist, 1, 2,
+ tmp, 0);
+ tmp = BT_8723B_2ANT_BT_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
+ bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, tmp, 0);
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
+ btc8723b2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- if (btc8723b_need_dec_pwr(btcoexist))
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
else
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+
+ if (BTC_RSSI_HIGH(wifi_rssi_state1) && BTC_RSSI_HIGH(bt_rssi_state)) {
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ } else {
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 14);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_LPS_ON, 0x50,
+ 0x4);
+ }
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 7);
if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
(bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
@@ -2878,94 +3320,148 @@
if (BTC_WIFI_BW_HT40 == wifi_bw) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, true, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, true,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, true, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, true,
+ false, false);
}
} else {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, false, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, true,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, false, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, true,
+ false, false);
}
}
}
static void btc8723b2ant_action_hid_a2dp(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
u32 wifi_bw;
+ u8 ap_num = 0;
+ u8 tmp = BT_8723B_2ANT_WIFI_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
- wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist,
- 0, 2, 15, 0);
- bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 2, 29, 0);
+ wifi_rssi_state = btc8723b2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8723b2ant_wifi_rssi_state(btcoexist, 1, 2,
+ tmp, 0);
+ tmp = BT_8723B_2ANT_BT_RSSI_COEXSWITCH_THRES -
+ coex_dm->switch_thres_offset;
+ bt_rssi_state = btc8723b2ant_bt_rssi_state(btcoexist, 3, tmp, 37);
btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
+ btc8723b2ant_limited_rx(btcoexist, NORMAL_EXEC, false, true, 0x5);
btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- if (btc8723b_need_dec_pwr(btcoexist))
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
- else
- btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
-
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- btc8723b_coex_tbl_type(btcoexist, NORMAL_EXEC, 7);
+ if (wifi_bw == BTC_WIFI_BW_LEGACY) {
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ else if (BTC_RSSI_MEDIUM(bt_rssi_state))
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ else
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ } else {
+ /* only 802.11N mode we have to dec bt power to 4 degree */
+ if (BTC_RSSI_HIGH(bt_rssi_state)) {
+ /* need to check ap Number of Not */
+ if (ap_num < 10)
+ btc8723b2ant_dec_bt_pwr(btcoexist,
+ NORMAL_EXEC, 4);
+ else
+ btc8723b2ant_dec_bt_pwr(btcoexist,
+ NORMAL_EXEC, 2);
+ } else if (BTC_RSSI_MEDIUM(bt_rssi_state)) {
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ } else {
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ }
+ }
- if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH))
- btc8723b2ant_tdma_duration_adjust(btcoexist, true, false, 2);
- else
- btc8723b2ant_tdma_duration_adjust(btcoexist, true, true, 2);
+ if (BTC_RSSI_HIGH(wifi_rssi_state1) && BTC_RSSI_HIGH(bt_rssi_state)) {
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ } else {
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 14);
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_LPS_ON, 0x50,
+ 0x4);
+ }
+
+ if (BTC_RSSI_HIGH(bt_rssi_state)) {
+ if (ap_num < 10)
+ btc8723b2ant_tdma_duration_adjust(btcoexist, true,
+ false, 1);
+ else
+ btc8723b2ant_tdma_duration_adjust(btcoexist, true,
+ false, 3);
+ } else {
+ btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 18);
+ btcoexist->btc_write_1byte(btcoexist, 0x456, 0x38);
+ btcoexist->btc_write_2byte(btcoexist, 0x42a, 0x0808);
+ btcoexist->btc_write_4byte(btcoexist, 0x430, 0x0);
+ btcoexist->btc_write_4byte(btcoexist, 0x434, 0x01010000);
+
+ if (ap_num < 10)
+ btc8723b2ant_tdma_duration_adjust(btcoexist, true,
+ true, 1);
+ else
+ btc8723b2ant_tdma_duration_adjust(btcoexist, true,
+ true, 3);
+ }
/* sw mechanism */
if (BTC_WIFI_BW_HT40 == wifi_bw) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, true, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, true,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, true, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, true, true,
+ false, false);
}
} else {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8723b2ant_sw_mechanism1(btcoexist, false, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, true, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, true,
+ false, false);
} else {
- btc8723b2ant_sw_mechanism1(btcoexist, false, true,
- false, false);
- btc8723b2ant_sw_mechanism2(btcoexist, false, false,
- false, 0x18);
+ btc8723b2ant_sw_mechanism(btcoexist, false, true,
+ false, false);
}
}
}
+static void btc8723b2ant_action_wifi_multi_port(struct btc_coexist *btcoexist)
+{
+ btc8723b2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btc8723b2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+
+ /* sw all off */
+ btc8723b2ant_sw_mechanism(btcoexist, false, false, false, false);
+
+ /* hw all off */
+ btc8723b2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
+
+ btc8723b2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ btc8723b2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+}
+
static void btc8723b2ant_run_coexist_mechanism(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 algorithm = 0;
+ u32 num_of_wifi_link = 0;
+ u32 wifi_link_status = 0;
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
+ bool miracast_plus_bt = false;
+ bool scan = false, link = false, roam = false;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], RunCoexistMechanism()===>\n");
@@ -2989,14 +3485,46 @@
"[BTCoex], BT is under inquiry/page scan !!\n");
btc8723b2ant_action_bt_inquiry(btcoexist);
return;
- } else {
- if (coex_dm->need_recover_0x948) {
- coex_dm->need_recover_0x948 = false;
- btcoexist->btc_write_2byte(btcoexist, 0x948,
- coex_dm->backup_0x948);
- }
}
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_SCAN, &scan);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_LINK, &link);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_ROAM, &roam);
+
+ if (scan || link || roam) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], WiFi is under Link Process !!\n");
+ btc8723b2ant_action_wifi_link_process(btcoexist);
+ return;
+ }
+
+ /* for P2P */
+ btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_LINK_STATUS,
+ &wifi_link_status);
+ num_of_wifi_link = wifi_link_status >> 16;
+
+ if ((num_of_wifi_link >= 2) ||
+ (wifi_link_status & WIFI_P2P_GO_CONNECTED)) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "############# [BTCoex], Multi-Port num_of_wifi_link = %d, wifi_link_status = 0x%x\n",
+ num_of_wifi_link, wifi_link_status);
+
+ if (bt_link_info->bt_link_exist)
+ miracast_plus_bt = true;
+ else
+ miracast_plus_bt = false;
+
+ btcoexist->btc_set(btcoexist, BTC_SET_BL_MIRACAST_PLUS_BT,
+ &miracast_plus_bt);
+ btc8723b2ant_action_wifi_multi_port(btcoexist);
+
+ return;
+ }
+
+ miracast_plus_bt = false;
+ btcoexist->btc_set(btcoexist, BTC_SET_BL_MIRACAST_PLUS_BT,
+ &miracast_plus_bt);
+
coex_dm->cur_algorithm = algorithm;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Algorithm = %d\n",
@@ -3077,19 +3605,37 @@
static void btc8723b2ant_wifioff_hwcfg(struct btc_coexist *btcoexist)
{
+ bool is_in_mp_mode = false;
+ u8 h2c_parameter[2] = {0};
+ u32 fw_ver = 0;
+
/* set wlan_act to low */
btcoexist->btc_write_1byte(btcoexist, 0x76e, 0x4);
- /* Force GNT_BT to High */
- btcoexist->btc_write_1byte_bitmask(btcoexist, 0x765, 0x18, 0x3);
- /* BT select s0/s1 is controlled by BT */
- btcoexist->btc_write_1byte_bitmask(btcoexist, 0x67, 0x20, 0x0);
+
+ /* WiFi standby while GNT_BT 0 -> 1 */
+ btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x780);
+
+ btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_FW_VER, &fw_ver);
+ if (fw_ver >= 0x180000) {
+ /* Use H2C to set GNT_BT to HIGH */
+ h2c_parameter[0] = 1;
+ btcoexist->btc_fill_h2c(btcoexist, 0x6E, 1, h2c_parameter);
+ } else {
+ btcoexist->btc_write_1byte(btcoexist, 0x765, 0x18);
+ }
+
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_IS_IN_MP_MODE,
+ &is_in_mp_mode);
+ if (!is_in_mp_mode)
+ /* BT select s0/s1 is controlled by BT */
+ btcoexist->btc_write_1byte_bitmask(btcoexist, 0x67, 0x20, 0x0);
+ else
+ /* BT select s0/s1 is controlled by WiFi */
+ btcoexist->btc_write_1byte_bitmask(btcoexist, 0x67, 0x20, 0x1);
}
/*********************************************************************
- * work around function start with wa_btc8723b2ant_
- *********************************************************************/
-/*********************************************************************
- * extern function start with EXbtc8723b2ant_
+ * extern function start with ex_btc8723b2ant_
*********************************************************************/
void ex_btc8723b2ant_init_hwconfig(struct btc_coexist *btcoexist)
{
@@ -3107,19 +3653,90 @@
u8tmp |= 0x5;
btcoexist->btc_write_1byte(btcoexist, 0x790, u8tmp);
- /*Antenna config */
+ /* Antenna config */
btc8723b2ant_set_ant_path(btcoexist, BTC_ANT_WIFI_AT_MAIN,
true, false);
+ coex_sta->dis_ver_info_cnt = 0;
+
/* PTA parameter */
- btc8723b_coex_tbl_type(btcoexist, FORCE_EXEC, 0);
+ btc8723b2ant_coex_table_with_type(btcoexist, FORCE_EXEC, 0);
/* Enable counter statistics */
- /*0x76e[3] =1, WLAN_Act control by PTA*/
- btcoexist->btc_write_1byte(btcoexist, 0x76e, 0xc);
+ /* 0x76e[3] = 1, WLAN_ACT controlled by PTA */
+ btcoexist->btc_write_1byte(btcoexist, 0x76e, 0x4);
btcoexist->btc_write_1byte(btcoexist, 0x778, 0x3);
btcoexist->btc_write_1byte_bitmask(btcoexist, 0x40, 0x20, 0x1);
}
+void ex_btc8723b2ant_power_on_setting(struct btc_coexist *btcoexist)
+{
+ struct btc_board_info *board_info = &btcoexist->board_info;
+ u16 u16tmp = 0x0;
+ u32 value = 0;
+
+ btcoexist->btc_write_1byte(btcoexist, 0x67, 0x20);
+
+ /* enable BB, REG_SYS_FUNC_EN such that we can write 0x948 correctly */
+ u16tmp = btcoexist->btc_read_2byte(btcoexist, 0x2);
+ btcoexist->btc_write_2byte(btcoexist, 0x2, u16tmp | BIT0 | BIT1);
+
+ btcoexist->btc_write_4byte(btcoexist, 0x948, 0x0);
+
+ if (btcoexist->chip_interface == BTC_INTF_USB) {
+ /* fixed at S0 for USB interface */
+ board_info->btdm_ant_pos = BTC_ANTENNA_AT_AUX_PORT;
+ } else {
+ /* for PCIE and SDIO interface, we check efuse 0xc3[6] */
+ if (board_info->single_ant_path == 0) {
+ /* set to S1 */
+ board_info->btdm_ant_pos = BTC_ANTENNA_AT_MAIN_PORT;
+ } else if (board_info->single_ant_path == 1) {
+ /* set to S0 */
+ board_info->btdm_ant_pos = BTC_ANTENNA_AT_AUX_PORT;
+ }
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_ANTPOSREGRISTRY_CTRL,
+ &value);
+ }
+}
+
+void ex_btc8723b2ant_pre_load_firmware(struct btc_coexist *btcoexist)
+{
+ struct btc_board_info *board_info = &btcoexist->board_info;
+ u8 u8tmp = 0x4; /* Set BIT2 by default since it's 2ant case */
+
+ /**
+ * S0 or S1 setting and Local register setting(By this fw can get
+ * ant number, S0/S1, ... info)
+ *
+ * Local setting bit define
+ * BIT0: "0" : no antenna inverse; "1" : antenna inverse
+ * BIT1: "0" : internal switch; "1" : external switch
+ * BIT2: "0" : one antenna; "1" : two antennas
+ *
+ * NOTE: here default all internal switch and 1-antenna ==> BIT1=0 and
+ * BIT2 = 0
+ */
+ if (btcoexist->chip_interface == BTC_INTF_USB) {
+ /* fixed at S0 for USB interface */
+ u8tmp |= 0x1; /* antenna inverse */
+ btcoexist->btc_write_local_reg_1byte(btcoexist, 0xfe08, u8tmp);
+ } else {
+ /* for PCIE and SDIO interface, we check efuse 0xc3[6] */
+ if (board_info->single_ant_path == 0) {
+ } else if (board_info->single_ant_path == 1) {
+ /* set to S0 */
+ u8tmp |= 0x1; /* antenna inverse */
+ }
+
+ if (btcoexist->chip_interface == BTC_INTF_PCI)
+ btcoexist->btc_write_local_reg_1byte(btcoexist, 0x384,
+ u8tmp);
+ else if (btcoexist->chip_interface == BTC_INTF_SDIO)
+ btcoexist->btc_write_local_reg_1byte(btcoexist, 0x60,
+ u8tmp);
+ }
+}
+
void ex_btc8723b2ant_init_coex_dm(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -3215,7 +3832,6 @@
((wifi_traffic_dir == BTC_WIFI_TRAFFIC_TX) ?
"uplink" : "downlink")));
-
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG, "\r\n %-35s = %d / %d / %d / %d",
"SCO/HID/PAN/A2DP",
bt_link_info->sco_exist, bt_link_info->hid_exist,
@@ -3265,7 +3881,7 @@
ps_tdma_case, coex_dm->auto_tdma_adjust);
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG, "\r\n %-35s = %d/ %d ",
- "DecBtPwr/ IgnWlanAct", coex_dm->cur_dec_bt_pwr,
+ "DecBtPwr/ IgnWlanAct", coex_dm->cur_dec_bt_pwr_lvl,
coex_dm->cur_ignore_wlan_act);
/* Hw setting */
@@ -3396,6 +4012,12 @@
void ex_btc8723b2ant_scan_notify(struct btc_coexist *btcoexist, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
+ u32 u32tmp;
+ u8 u8tmpa, u8tmpb;
+
+ u32tmp = btcoexist->btc_read_4byte(btcoexist, 0x948);
+ u8tmpa = btcoexist->btc_read_1byte(btcoexist, 0x765);
+ u8tmpb = btcoexist->btc_read_1byte(btcoexist, 0x76e);
if (BTC_SCAN_START == type)
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -3403,6 +4025,12 @@
else if (BTC_SCAN_FINISH == type)
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], SCAN FINISH notify\n");
+ btcoexist->btc_get(btcoexist, BTC_GET_U1_AP_NUM,
+ &coex_sta->scan_ap_num);
+
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "############# [BTCoex], 0x948=0x%x, 0x765=0x%x, 0x76e=0x%x\n",
+ u32tmp, u8tmpa, u8tmpb);
}
void ex_btc8723b2ant_connect_notify(struct btc_coexist *btcoexist, u8 type)
@@ -3424,6 +4052,7 @@
u8 h2c_parameter[3] = {0};
u32 wifi_bw;
u8 wifi_central_chnl;
+ u8 ap_num = 0;
if (BTC_MEDIA_CONNECT == type)
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -3441,10 +4070,16 @@
h2c_parameter[1] = wifi_central_chnl;
btcoexist->btc_get(btcoexist,
BTC_GET_U4_WIFI_BW, &wifi_bw);
- if (BTC_WIFI_BW_HT40 == wifi_bw)
+ if (wifi_bw == BTC_WIFI_BW_HT40) {
h2c_parameter[2] = 0x30;
- else
- h2c_parameter[2] = 0x20;
+ } else {
+ btcoexist->btc_get(btcoexist, BTC_GET_U1_AP_NUM,
+ &ap_num);
+ if (ap_num < 10)
+ h2c_parameter[2] = 0x30;
+ else
+ h2c_parameter[2] = 0x20;
+ }
}
coex_dm->wifi_chnl_info[0] = h2c_parameter[0];
@@ -3492,7 +4127,7 @@
coex_sta->bt_info_c2h[rsp_source][i] = tmpbuf[i];
if (i == 1)
bt_info = tmpbuf[i];
- if (i == length-1)
+ if (i == length - 1)
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"0x%02x]\n", tmpbuf[i]);
else
@@ -3507,17 +4142,30 @@
}
if (BT_INFO_SRC_8723B_2ANT_WIFI_FW != rsp_source) {
- coex_sta->bt_retry_cnt = /* [3:0]*/
+ coex_sta->bt_retry_cnt =
coex_sta->bt_info_c2h[rsp_source][2] & 0xf;
+ if (coex_sta->bt_retry_cnt >= 1)
+ coex_sta->pop_event_cnt++;
+
coex_sta->bt_rssi =
coex_sta->bt_info_c2h[rsp_source][3] * 2 + 10;
- coex_sta->bt_info_ext =
- coex_sta->bt_info_c2h[rsp_source][4];
+ coex_sta->bt_info_ext = coex_sta->bt_info_c2h[rsp_source][4];
+
+ if (coex_sta->bt_info_c2h[rsp_source][2] & 0x20)
+ coex_sta->c2h_bt_remote_name_req = true;
+ else
+ coex_sta->c2h_bt_remote_name_req = false;
+
+ if (coex_sta->bt_info_c2h[rsp_source][1] == 0x49)
+ coex_sta->a2dp_bit_pool =
+ coex_sta->bt_info_c2h[rsp_source][6];
+ else
+ coex_sta->a2dp_bit_pool = 0;
/* Here we need to resend some wifi info to BT
- because bt is reset and loss of the info.
+ * because BT is reset and loss of the info.
*/
if ((coex_sta->bt_info_ext & BIT1)) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -3552,20 +4200,21 @@
#endif
}
- /* check BIT2 first ==> check if bt is under inquiry or page scan*/
+ /* check BIT2 first ==> check if bt is under inquiry or page scan */
if (bt_info & BT_INFO_8723B_2ANT_B_INQ_PAGE)
coex_sta->c2h_bt_inquiry_page = true;
else
coex_sta->c2h_bt_inquiry_page = false;
- /* set link exist status*/
if (!(bt_info & BT_INFO_8723B_2ANT_B_CONNECTION)) {
+ /* set link exist status */
coex_sta->bt_link_exist = false;
coex_sta->pan_exist = false;
coex_sta->a2dp_exist = false;
coex_sta->hid_exist = false;
coex_sta->sco_exist = false;
- } else { /* connection exists */
+ } else {
+ /* connection exists */
coex_sta->bt_link_exist = true;
if (bt_info & BT_INFO_8723B_2ANT_B_FTP)
coex_sta->pan_exist = true;
@@ -3583,6 +4232,16 @@
coex_sta->sco_exist = true;
else
coex_sta->sco_exist = false;
+
+ if ((!coex_sta->hid_exist) &&
+ (!coex_sta->c2h_bt_inquiry_page) &&
+ (!coex_sta->sco_exist)) {
+ if (coex_sta->high_priority_tx +
+ coex_sta->high_priority_rx >= 160) {
+ coex_sta->hid_exist = true;
+ bt_info = bt_info | 0x28;
+ }
+ }
}
btc8723b2ant_update_bt_link_info(btcoexist);
@@ -3640,46 +4299,67 @@
ex_btc8723b2ant_media_status_notify(btcoexist, BTC_MEDIA_DISCONNECT);
}
+void ex_btc8723b2ant_pnp_notify(struct btc_coexist *btcoexist, u8 pnp_state)
+{
+ struct rtl_priv *rtlpriv = btcoexist->adapter;
+
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD, "[BTCoex], Pnp notify\n");
+
+ if (pnp_state == BTC_WIFI_PNP_SLEEP) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Pnp notify to SLEEP\n");
+
+ /* Driver do not leave IPS/LPS when driver is going to sleep, so
+ * BTCoexistence think wifi is still under IPS/LPS
+ *
+ * BT should clear UnderIPS/UnderLPS state to avoid mismatch
+ * state after wakeup.
+ */
+ coex_sta->under_ips = false;
+ coex_sta->under_lps = false;
+ } else if (pnp_state == BTC_WIFI_PNP_WAKE_UP) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Pnp notify to WAKE UP\n");
+ ex_btc8723b2ant_init_hwconfig(btcoexist);
+ btc8723b2ant_init_coex_dm(btcoexist);
+ btc8723b2ant_query_bt_info(btcoexist);
+ }
+}
+
void ex_btc8723b2ant_periodical(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- struct btc_board_info *board_info = &btcoexist->board_info;
- struct btc_stack_info *stack_info = &btcoexist->stack_info;
- static u8 dis_ver_info_cnt;
- u32 fw_ver = 0, bt_patch_ver = 0;
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], ==========================Periodical===========================\n");
- if (dis_ver_info_cnt <= 5) {
- dis_ver_info_cnt += 1;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], ****************************************************************\n");
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Ant PG Num/ Ant Mech/ Ant Pos = %d/ %d/ %d\n",
- board_info->pg_ant_num,
- board_info->btdm_ant_num,
- board_info->btdm_ant_pos);
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT stack/ hci ext ver = %s / %d\n",
- stack_info->profile_notified ? "Yes" : "No",
- stack_info->hci_version);
- btcoexist->btc_get(btcoexist, BTC_GET_U4_BT_PATCH_VER,
- &bt_patch_ver);
- btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_FW_VER, &fw_ver);
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], CoexVer/ fw_ver/ PatchVer = %d_%x/ 0x%x/ 0x%x(%d)\n",
- glcoex_ver_date_8723b_2ant, glcoex_ver_8723b_2ant,
- fw_ver, bt_patch_ver, bt_patch_ver);
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], ****************************************************************\n");
+ if (coex_sta->dis_ver_info_cnt <= 5) {
+ coex_sta->dis_ver_info_cnt += 1;
+ if (coex_sta->dis_ver_info_cnt == 3) {
+ /* Antenna config to set 0x765 = 0x0 (GNT_BT control by
+ * PTA) after initial
+ */
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Set GNT_BT control by PTA\n");
+ btc8723b2ant_set_ant_path(
+ btcoexist, BTC_ANT_WIFI_AT_MAIN, false, false);
+ }
}
#if (BT_AUTO_REPORT_ONLY_8723B_2ANT == 0)
btc8723b2ant_query_bt_info(btcoexist);
- btc8723b2ant_monitor_bt_ctr(btcoexist);
- btc8723b2ant_monitor_bt_enable_disable(btcoexist);
#else
+ btc8723b2ant_monitor_bt_ctr(btcoexist);
+ btc8723b2ant_monitor_wifi_ctr(btcoexist);
+
+ /* for some BT speakers that High-Priority pkts appear before
+ * playing, this will cause HID exist
+ */
+ if ((coex_sta->high_priority_tx + coex_sta->high_priority_rx < 50) &&
+ (bt_link_info->hid_exist))
+ bt_link_info->hid_exist = false;
+
if (btc8723b2ant_is_wifi_status_changed(btcoexist) ||
coex_dm->auto_tdma_adjust)
btc8723b2ant_run_coexist_mechanism(btcoexist);
diff --git a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8723b2ant.h b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8723b2ant.h
index 567f354..18a35c7 100644
--- a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8723b2ant.h
+++ b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8723b2ant.h
@@ -41,6 +41,11 @@
#define BTC_RSSI_COEX_THRESH_TOL_8723B_2ANT 2
+/* WiFi RSSI Threshold for 2-Ant TDMA/1-Ant PS-TDMA translation */
+#define BT_8723B_2ANT_WIFI_RSSI_COEXSWITCH_THRES 42
+/* BT RSSI Threshold for 2-Ant TDMA/1-Ant PS-TDMA translation */
+#define BT_8723B_2ANT_BT_RSSI_COEXSWITCH_THRES 46
+
enum BT_INFO_SRC_8723B_2ANT {
BT_INFO_SRC_8723B_2ANT_WIFI_FW = 0x0,
BT_INFO_SRC_8723B_2ANT_BT_RSP = 0x1,
@@ -75,8 +80,8 @@
struct coex_dm_8723b_2ant {
/* fw mechanism */
- bool pre_dec_bt_pwr;
- bool cur_dec_bt_pwr;
+ bool pre_dec_bt_pwr_lvl;
+ bool cur_dec_bt_pwr_lvl;
u8 pre_fw_dac_swing_lvl;
u8 cur_fw_dac_swing_lvl;
bool cur_ignore_wlan_act;
@@ -84,7 +89,7 @@
u8 pre_ps_tdma;
u8 cur_ps_tdma;
u8 ps_tdma_para[5];
- u8 tdma_adj_type;
+ u8 ps_tdma_du_adj_type;
bool reset_tdma_adjust;
bool auto_tdma_adjust;
bool pre_ps_tdma_on;
@@ -122,8 +127,13 @@
u8 bt_status;
u8 wifi_chnl_info[3];
- bool need_recover_0x948;
- u16 backup_0x948;
+ u8 pre_lps;
+ u8 cur_lps;
+ u8 pre_rpwm;
+ u8 cur_rpwm;
+
+ bool is_switch_to_1dot5_ant;
+ u8 switch_thres_offset;
};
struct coex_sta_8723b_2ant {
@@ -132,6 +142,7 @@
bool a2dp_exist;
bool hid_exist;
bool pan_exist;
+ bool bt_abnormal_scan;
bool under_lps;
bool under_ips;
@@ -140,14 +151,33 @@
u32 low_priority_tx;
u32 low_priority_rx;
u8 bt_rssi;
+ bool bt_tx_rx_mask;
u8 pre_bt_rssi_state;
u8 pre_wifi_rssi_state[4];
bool c2h_bt_info_req_sent;
u8 bt_info_c2h[BT_INFO_SRC_8723B_2ANT_MAX][10];
u32 bt_info_c2h_cnt[BT_INFO_SRC_8723B_2ANT_MAX];
bool c2h_bt_inquiry_page;
+ bool c2h_bt_remote_name_req;
u8 bt_retry_cnt;
u8 bt_info_ext;
+ u32 pop_event_cnt;
+ u8 scan_ap_num;
+
+ u32 crc_ok_cck;
+ u32 crc_ok_11g;
+ u32 crc_ok_11n;
+ u32 crc_ok_11n_agg;
+
+ u32 crc_err_cck;
+ u32 crc_err_11g;
+ u32 crc_err_11n;
+ u32 crc_err_11n_agg;
+ bool force_lps_on;
+
+ u8 dis_ver_info_cnt;
+
+ u8 a2dp_bit_pool;
};
/*********************************************************************
diff --git a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a1ant.c b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a1ant.c
index 8b689ed..5e9f3b0 100644
--- a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a1ant.c
+++ b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a1ant.c
@@ -23,7 +23,7 @@
*
*****************************************************************************/
-/*============================================================
+/**************************************************************
* Description:
*
* This file is for RTL8821A Co-exist mechanism
@@ -31,21 +31,21 @@
* History
* 2012/11/15 Cosa first check in.
*
- *============================================================
-*/
-/*============================================================
+ **************************************************************/
+
+/**************************************************************
* include files
- *============================================================
- */
+ **************************************************************/
#include "halbt_precomp.h"
-/*============================================================
+/**************************************************************
* Global variables, these are static variables
- *============================================================
- */
+ **************************************************************/
static struct coex_dm_8821a_1ant glcoex_dm_8821a_1ant;
static struct coex_dm_8821a_1ant *coex_dm = &glcoex_dm_8821a_1ant;
static struct coex_sta_8821a_1ant glcoex_sta_8821a_1ant;
static struct coex_sta_8821a_1ant *coex_sta = &glcoex_sta_8821a_1ant;
+static void btc8821a1ant_act_bt_sco_hid_only_busy(struct btc_coexist *btcoexist,
+ u8 wifi_status);
static const char *const glbt_info_src_8821a_1ant[] = {
"BT Info[wifi fw]",
@@ -53,22 +53,21 @@
"BT Info[bt auto report]",
};
-static u32 glcoex_ver_date_8821a_1ant = 20130816;
-static u32 glcoex_ver_8821a_1ant = 0x41;
+static u32 glcoex_ver_date_8821a_1ant = 20130816;
+static u32 glcoex_ver_8821a_1ant = 0x41;
-/*============================================================
+/**************************************************************
* local function proto type if needed
*
- * local function start with halbtc8821a1ant_
- *============================================================
- */
-static u8 halbtc8821a1ant_bt_rssi_state(struct btc_coexist *btcoexist,
- u8 level_num, u8 rssi_thresh,
- u8 rssi_thresh1)
+ * local function start with btc8821a1ant_
+ **************************************************************/
+static u8 btc8821a1ant_bt_rssi_state(struct btc_coexist *btcoexist,
+ u8 level_num, u8 rssi_thresh,
+ u8 rssi_thresh1)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- long bt_rssi = 0;
- u8 bt_rssi_state = coex_sta->pre_bt_rssi_state;
+ long bt_rssi = 0;
+ u8 bt_rssi_state = coex_sta->pre_bt_rssi_state;
bt_rssi = coex_sta->bt_rssi;
@@ -150,9 +149,9 @@
return bt_rssi_state;
}
-static u8 halbtc8821a1ant_WifiRssiState(struct btc_coexist *btcoexist,
- u8 index, u8 level_num, u8 rssi_thresh,
- u8 rssi_thresh1)
+static u8 btc8821a1ant_wifi_rssi_state(struct btc_coexist *btcoexist,
+ u8 index, u8 level_num, u8 rssi_thresh,
+ u8 rssi_thresh1)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
long wifi_rssi = 0;
@@ -165,8 +164,8 @@
BTC_RSSI_STATE_LOW) ||
(coex_sta->pre_wifi_rssi_state[index] ==
BTC_RSSI_STATE_STAY_LOW)) {
- if (wifi_rssi >=
- (rssi_thresh+BTC_RSSI_COEX_THRESH_TOL_8821A_1ANT)) {
+ if (wifi_rssi >= (rssi_thresh +
+ BTC_RSSI_COEX_THRESH_TOL_8821A_1ANT)) {
wifi_rssi_state = BTC_RSSI_STATE_HIGH;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], wifi RSSI state switch to High\n");
@@ -197,8 +196,8 @@
BTC_RSSI_STATE_LOW) ||
(coex_sta->pre_wifi_rssi_state[index] ==
BTC_RSSI_STATE_STAY_LOW)) {
- if (wifi_rssi >=
- (rssi_thresh+BTC_RSSI_COEX_THRESH_TOL_8821A_1ANT)) {
+ if (wifi_rssi >= (rssi_thresh +
+ BTC_RSSI_COEX_THRESH_TOL_8821A_1ANT)) {
wifi_rssi_state = BTC_RSSI_STATE_MEDIUM;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], wifi RSSI state switch to Medium\n");
@@ -211,9 +210,8 @@
BTC_RSSI_STATE_MEDIUM) ||
(coex_sta->pre_wifi_rssi_state[index] ==
BTC_RSSI_STATE_STAY_MEDIUM)) {
- if (wifi_rssi >=
- (rssi_thresh1 +
- BTC_RSSI_COEX_THRESH_TOL_8821A_1ANT)) {
+ if (wifi_rssi >= (rssi_thresh1 +
+ BTC_RSSI_COEX_THRESH_TOL_8821A_1ANT)) {
wifi_rssi_state = BTC_RSSI_STATE_HIGH;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], wifi RSSI state switch to High\n");
@@ -243,14 +241,14 @@
return wifi_rssi_state;
}
-static void halbtc8821a1ant_update_ra_mask(struct btc_coexist *btcoexist,
- bool force_exec, u32 dis_rate_mask)
+static void btc8821a1ant_update_ra_mask(struct btc_coexist *btcoexist,
+ bool force_exec, u32 dis_rate_mask)
{
coex_dm->cur_ra_mask = dis_rate_mask;
if (force_exec ||
(coex_dm->pre_ra_mask != coex_dm->cur_ra_mask)) {
- btcoexist->btc_set(btcoexist, BTC_SET_ACT_UPDATE_ra_mask,
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_UPDATE_RAMASK,
&coex_dm->cur_ra_mask);
}
coex_dm->pre_ra_mask = coex_dm->cur_ra_mask;
@@ -259,14 +257,14 @@
static void btc8821a1ant_auto_rate_fb_retry(struct btc_coexist *btcoexist,
bool force_exec, u8 type)
{
- bool wifi_under_b_mode = false;
+ bool wifi_under_b_mode = false;
coex_dm->cur_arfr_type = type;
if (force_exec ||
(coex_dm->pre_arfr_type != coex_dm->cur_arfr_type)) {
switch (coex_dm->cur_arfr_type) {
- case 0: /* normal mode*/
+ case 0: /* normal mode */
btcoexist->btc_write_4byte(btcoexist, 0x430,
coex_dm->backup_arfr_cnt1);
btcoexist->btc_write_4byte(btcoexist, 0x434,
@@ -296,19 +294,19 @@
coex_dm->pre_arfr_type = coex_dm->cur_arfr_type;
}
-static void halbtc8821a1ant_retry_limit(struct btc_coexist *btcoexist,
- bool force_exec, u8 type)
+static void btc8821a1ant_retry_limit(struct btc_coexist *btcoexist,
+ bool force_exec, u8 type)
{
coex_dm->cur_retry_limit_type = type;
if (force_exec ||
(coex_dm->pre_retry_limit_type != coex_dm->cur_retry_limit_type)) {
switch (coex_dm->cur_retry_limit_type) {
- case 0: /* normal mode*/
+ case 0: /* normal mode */
btcoexist->btc_write_2byte(btcoexist, 0x42a,
coex_dm->backup_retry_limit);
break;
- case 1: /* retry limit = 8*/
+ case 1: /* retry limit = 8 */
btcoexist->btc_write_2byte(btcoexist, 0x42a, 0x0808);
break;
default:
@@ -318,19 +316,19 @@
coex_dm->pre_retry_limit_type = coex_dm->cur_retry_limit_type;
}
-static void halbtc8821a1ant_ampdu_max_time(struct btc_coexist *btcoexist,
- bool force_exec, u8 type)
+static void btc8821a1ant_ampdu_max_time(struct btc_coexist *btcoexist,
+ bool force_exec, u8 type)
{
coex_dm->cur_ampdu_time_type = type;
if (force_exec ||
(coex_dm->pre_ampdu_time_type != coex_dm->cur_ampdu_time_type)) {
switch (coex_dm->cur_ampdu_time_type) {
- case 0: /* normal mode*/
+ case 0: /* normal mode */
btcoexist->btc_write_1byte(btcoexist, 0x456,
coex_dm->backup_ampdu_max_time);
break;
- case 1: /* AMPDU timw = 0x38 * 32us*/
+ case 1: /* AMPDU time = 0x38 * 32us */
btcoexist->btc_write_1byte(btcoexist, 0x456, 0x38);
break;
default:
@@ -341,88 +339,85 @@
coex_dm->pre_ampdu_time_type = coex_dm->cur_ampdu_time_type;
}
-static void halbtc8821a1ant_limited_tx(struct btc_coexist *btcoexist,
- bool force_exec, u8 ra_mask_type,
- u8 arfr_type, u8 retry_limit_type,
- u8 ampdu_time_type)
+static void btc8821a1ant_limited_tx(struct btc_coexist *btcoexist,
+ bool force_exec, u8 ra_mask_type,
+ u8 arfr_type, u8 retry_limit_type,
+ u8 ampdu_time_type)
{
switch (ra_mask_type) {
- case 0: /* normal mode*/
- halbtc8821a1ant_update_ra_mask(btcoexist, force_exec, 0x0);
+ case 0: /* normal mode */
+ btc8821a1ant_update_ra_mask(btcoexist, force_exec, 0x0);
break;
- case 1: /* disable cck 1/2*/
- halbtc8821a1ant_update_ra_mask(btcoexist, force_exec,
- 0x00000003);
+ case 1: /* disable cck 1/2 */
+ btc8821a1ant_update_ra_mask(btcoexist, force_exec,
+ 0x00000003);
break;
- case 2: /* disable cck 1/2/5.5, ofdm 6/9/12/18/24, mcs 0/1/2/3/4*/
- halbtc8821a1ant_update_ra_mask(btcoexist, force_exec,
- 0x0001f1f7);
+ case 2: /* disable cck 1/2/5.5, ofdm 6/9/12/18/24, mcs 0/1/2/3/4 */
+ btc8821a1ant_update_ra_mask(btcoexist, force_exec,
+ 0x0001f1f7);
break;
default:
break;
}
btc8821a1ant_auto_rate_fb_retry(btcoexist, force_exec, arfr_type);
- halbtc8821a1ant_retry_limit(btcoexist, force_exec, retry_limit_type);
- halbtc8821a1ant_ampdu_max_time(btcoexist, force_exec, ampdu_time_type);
+ btc8821a1ant_retry_limit(btcoexist, force_exec, retry_limit_type);
+ btc8821a1ant_ampdu_max_time(btcoexist, force_exec, ampdu_time_type);
}
-static void halbtc8821a1ant_limited_rx(struct btc_coexist *btcoexist,
- bool force_exec, bool rej_ap_agg_pkt,
- bool bt_ctrl_agg_buf_size,
- u8 agg_buf_size)
+static void btc8821a1ant_limited_rx(struct btc_coexist *btcoexist,
+ bool force_exec, bool rej_ap_agg_pkt,
+ bool bt_ctrl_agg_buf_size, u8 agg_buf_size)
{
bool reject_rx_agg = rej_ap_agg_pkt;
bool bt_ctrl_rx_agg_size = bt_ctrl_agg_buf_size;
u8 rx_agg_size = agg_buf_size;
- /*============================================*/
- /* Rx Aggregation related setting*/
- /*============================================*/
+ /* Rx Aggregation related setting */
btcoexist->btc_set(btcoexist,
BTC_SET_BL_TO_REJ_AP_AGG_PKT, &reject_rx_agg);
- /* decide BT control aggregation buf size or not*/
+ /* decide BT control aggregation buf size or not */
btcoexist->btc_set(btcoexist, BTC_SET_BL_BT_CTRL_AGG_SIZE,
&bt_ctrl_rx_agg_size);
- /* aggregation buf size, only work when BT control Rx agg size.*/
+ /* aggregation buf size, only work when BT control Rx agg size */
btcoexist->btc_set(btcoexist, BTC_SET_U1_AGG_BUF_SIZE, &rx_agg_size);
- /* real update aggregation setting*/
+ /* real update aggregation setting */
btcoexist->btc_set(btcoexist, BTC_SET_ACT_AGGREGATE_CTRL, NULL);
}
-static void halbtc8821a1ant_monitor_bt_ctr(struct btc_coexist *btcoexist)
+static void btc8821a1ant_monitor_bt_ctr(struct btc_coexist *btcoexist)
{
- u32 reg_hp_tx_rx, reg_lp_tx_rx, u4_tmp;
- u32 reg_hp_tx = 0, reg_hp_rx = 0, reg_lp_tx = 0, reg_lp_rx = 0;
+ u32 reg_hp_tx_rx, reg_lp_tx_rx, u4_tmp;
+ u32 reg_hp_tx = 0, reg_hp_rx = 0, reg_lp_tx = 0, reg_lp_rx = 0;
reg_hp_tx_rx = 0x770;
reg_lp_tx_rx = 0x774;
u4_tmp = btcoexist->btc_read_4byte(btcoexist, reg_hp_tx_rx);
reg_hp_tx = u4_tmp & MASKLWORD;
- reg_hp_rx = (u4_tmp & MASKHWORD)>>16;
+ reg_hp_rx = (u4_tmp & MASKHWORD) >> 16;
u4_tmp = btcoexist->btc_read_4byte(btcoexist, reg_lp_tx_rx);
reg_lp_tx = u4_tmp & MASKLWORD;
- reg_lp_rx = (u4_tmp & MASKHWORD)>>16;
+ reg_lp_rx = (u4_tmp & MASKHWORD) >> 16;
coex_sta->high_priority_tx = reg_hp_tx;
coex_sta->high_priority_rx = reg_hp_rx;
coex_sta->low_priority_tx = reg_lp_tx;
coex_sta->low_priority_rx = reg_lp_rx;
- /* reset counter*/
+ /* reset counter */
btcoexist->btc_write_1byte(btcoexist, 0x76e, 0xc);
}
-static void halbtc8821a1ant_query_bt_info(struct btc_coexist *btcoexist)
+static void btc8821a1ant_query_bt_info(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[1] = {0};
coex_sta->c2h_bt_info_req_sent = true;
- h2c_parameter[0] |= BIT0; /* trigger*/
+ h2c_parameter[0] |= BIT0; /* trigger */
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Query Bt Info, FW write 0x61 = 0x%x\n",
@@ -431,10 +426,43 @@
btcoexist->btc_fill_h2c(btcoexist, 0x61, 1, h2c_parameter);
}
-static void halbtc8821a1ant_update_bt_link_info(struct btc_coexist *btcoexist)
+bool btc8821a1ant_is_wifi_status_changed(struct btc_coexist *btcoexist)
+{
+ static bool pre_wifi_busy = true;
+ static bool pre_under_4way = true;
+ static bool pre_bt_hs_on = true;
+ bool wifi_busy = false, under_4way = false, bt_hs_on = false;
+ bool wifi_connected = false;
+
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
+ &wifi_connected);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_BUSY, &wifi_busy);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_4_WAY_PROGRESS,
+ &under_4way);
+
+ if (wifi_connected) {
+ if (wifi_busy != pre_wifi_busy) {
+ pre_wifi_busy = wifi_busy;
+ return true;
+ }
+ if (under_4way != pre_under_4way) {
+ pre_under_4way = under_4way;
+ return true;
+ }
+ if (bt_hs_on != pre_bt_hs_on) {
+ pre_bt_hs_on = bt_hs_on;
+ return true;
+ }
+ }
+
+ return false;
+}
+
+static void btc8821a1ant_update_bt_link_info(struct btc_coexist *btcoexist)
{
struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
- bool bt_hs_on = false;
+ bool bt_hs_on = false;
btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
@@ -444,13 +472,13 @@
bt_link_info->pan_exist = coex_sta->pan_exist;
bt_link_info->hid_exist = coex_sta->hid_exist;
- /* work around for HS mode.*/
+ /* work around for HS mode */
if (bt_hs_on) {
bt_link_info->pan_exist = true;
bt_link_info->bt_link_exist = true;
}
- /* check if Sco only*/
+ /* check if Sco only */
if (bt_link_info->sco_exist &&
!bt_link_info->a2dp_exist &&
!bt_link_info->pan_exist &&
@@ -459,7 +487,7 @@
else
bt_link_info->sco_only = false;
- /* check if A2dp only*/
+ /* check if A2dp only */
if (!bt_link_info->sco_exist &&
bt_link_info->a2dp_exist &&
!bt_link_info->pan_exist &&
@@ -468,7 +496,7 @@
else
bt_link_info->a2dp_only = false;
- /* check if Pan only*/
+ /* check if Pan only */
if (!bt_link_info->sco_exist &&
!bt_link_info->a2dp_exist &&
bt_link_info->pan_exist &&
@@ -477,7 +505,7 @@
else
bt_link_info->pan_only = false;
- /* check if Hid only*/
+ /* check if Hid only */
if (!bt_link_info->sco_exist &&
!bt_link_info->a2dp_exist &&
!bt_link_info->pan_exist &&
@@ -487,13 +515,13 @@
bt_link_info->hid_only = false;
}
-static u8 halbtc8821a1ant_action_algorithm(struct btc_coexist *btcoexist)
+static u8 btc8821a1ant_action_algorithm(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
- bool bt_hs_on = false;
- u8 algorithm = BT_8821A_1ANT_COEX_ALGO_UNDEFINED;
- u8 num_of_diff_profile = 0;
+ bool bt_hs_on = false;
+ u8 algorithm = BT_8821A_1ANT_COEX_ALGO_UNDEFINED;
+ u8 num_of_diff_profile = 0;
btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
@@ -605,7 +633,7 @@
"[BTCoex], BT Profile = SCO + HID + A2DP ==> HID\n");
algorithm = BT_8821A_1ANT_COEX_ALGO_HID;
} else if (bt_link_info->hid_exist &&
- bt_link_info->pan_exist) {
+ bt_link_info->pan_exist) {
if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
@@ -618,7 +646,7 @@
algorithm = BT_8821A_1ANT_COEX_ALGO_PANEDR_HID;
}
} else if (bt_link_info->pan_exist &&
- bt_link_info->a2dp_exist) {
+ bt_link_info->a2dp_exist) {
if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
@@ -670,53 +698,8 @@
return algorithm;
}
-static void halbtc8821a1ant_set_bt_auto_report(struct btc_coexist *btcoexist,
- bool enable_auto_report)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- u8 h2c_parameter[1] = {0};
-
- h2c_parameter[0] = 0;
-
- if (enable_auto_report)
- h2c_parameter[0] |= BIT0;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT FW auto report : %s, FW write 0x68 = 0x%x\n",
- (enable_auto_report ? "Enabled!!" : "Disabled!!"),
- h2c_parameter[0]);
-
- btcoexist->btc_fill_h2c(btcoexist, 0x68, 1, h2c_parameter);
-}
-
-static void halbtc8821a1ant_bt_auto_report(struct btc_coexist *btcoexist,
- bool force_exec,
- bool enable_auto_report)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], %s BT Auto report = %s\n",
- (force_exec ? "force to" : ""), ((enable_auto_report) ?
- "Enabled" : "Disabled"));
- coex_dm->cur_bt_auto_report = enable_auto_report;
-
- if (!force_exec) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], pre_bt_auto_report = %d, cur_bt_auto_report = %d\n",
- coex_dm->pre_bt_auto_report,
- coex_dm->cur_bt_auto_report);
-
- if (coex_dm->pre_bt_auto_report == coex_dm->cur_bt_auto_report)
- return;
- }
- halbtc8821a1ant_set_bt_auto_report(btcoexist, coex_dm->cur_bt_auto_report);
-
- coex_dm->pre_bt_auto_report = coex_dm->cur_bt_auto_report;
-}
-
-static void btc8821a1ant_set_sw_pen_tx_rate(struct btc_coexist *btcoexist,
- bool low_penalty_ra)
+static void btc8821a1ant_set_sw_penalty_tx_rate(struct btc_coexist *btcoexist,
+ bool low_penalty_ra)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[6] = {0};
@@ -725,11 +708,11 @@
if (low_penalty_ra) {
h2c_parameter[1] |= BIT0;
- /*normal rate except MCS7/6/5, OFDM54/48/36*/
+ /* normal rate except MCS7/6/5, OFDM54/48/36 */
h2c_parameter[2] = 0x00;
- h2c_parameter[3] = 0xf7; /*MCS7 or OFDM54*/
- h2c_parameter[4] = 0xf8; /*MCS6 or OFDM48*/
- h2c_parameter[5] = 0xf9; /*MCS5 or OFDM36*/
+ h2c_parameter[3] = 0xf7; /* MCS7 or OFDM54 */
+ h2c_parameter[4] = 0xf8; /* MCS6 or OFDM48 */
+ h2c_parameter[5] = 0xf9; /* MCS5 or OFDM36 */
}
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -739,8 +722,8 @@
btcoexist->btc_fill_h2c(btcoexist, 0x69, 6, h2c_parameter);
}
-static void halbtc8821a1ant_low_penalty_ra(struct btc_coexist *btcoexist,
- bool force_exec, bool low_penalty_ra)
+static void btc8821a1ant_low_penalty_ra(struct btc_coexist *btcoexist,
+ bool force_exec, bool low_penalty_ra)
{
coex_dm->cur_low_penalty_ra = low_penalty_ra;
@@ -748,14 +731,15 @@
if (coex_dm->pre_low_penalty_ra == coex_dm->cur_low_penalty_ra)
return;
}
- btc8821a1ant_set_sw_pen_tx_rate(btcoexist, coex_dm->cur_low_penalty_ra);
+ btc8821a1ant_set_sw_penalty_tx_rate(btcoexist,
+ coex_dm->cur_low_penalty_ra);
coex_dm->pre_low_penalty_ra = coex_dm->cur_low_penalty_ra;
}
-static void halbtc8821a1ant_set_coex_table(struct btc_coexist *btcoexist,
- u32 val0x6c0, u32 val0x6c4,
- u32 val0x6c8, u8 val0x6cc)
+static void btc8821a1ant_set_coex_table(struct btc_coexist *btcoexist,
+ u32 val0x6c0, u32 val0x6c4,
+ u32 val0x6c8, u8 val0x6cc)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -776,9 +760,9 @@
btcoexist->btc_write_1byte(btcoexist, 0x6cc, val0x6cc);
}
-static void halbtc8821a1ant_coex_table(struct btc_coexist *btcoexist,
- bool force_exec, u32 val0x6c0,
- u32 val0x6c4, u32 val0x6c8, u8 val0x6cc)
+static void btc8821a1ant_coex_table(struct btc_coexist *btcoexist,
+ bool force_exec, u32 val0x6c0, u32 val0x6c4,
+ u32 val0x6c8, u8 val0x6cc)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -798,8 +782,8 @@
(coex_dm->pre_val_0x6cc == coex_dm->cur_val_0x6cc))
return;
}
- halbtc8821a1ant_set_coex_table(btcoexist, val0x6c0, val0x6c4,
- val0x6c8, val0x6cc);
+ btc8821a1ant_set_coex_table(btcoexist, val0x6c0, val0x6c4,
+ val0x6c8, val0x6cc);
coex_dm->pre_val_0x6c0 = coex_dm->cur_val_0x6c0;
coex_dm->pre_val_0x6c4 = coex_dm->cur_val_0x6c4;
@@ -807,42 +791,41 @@
coex_dm->pre_val_0x6cc = coex_dm->cur_val_0x6cc;
}
-static void halbtc8821a1ant_coex_table_with_type(struct btc_coexist *btcoexist,
- bool force_exec, u8 type)
+static void btc8821a1ant_coex_table_with_type(struct btc_coexist *btcoexist,
+ bool force_exec, u8 type)
{
switch (type) {
case 0:
- halbtc8821a1ant_coex_table(btcoexist, force_exec, 0x55555555,
- 0x55555555, 0xffffff, 0x3);
+ btc8821a1ant_coex_table(btcoexist, force_exec, 0x55555555,
+ 0x55555555, 0xffffff, 0x3);
break;
case 1:
- halbtc8821a1ant_coex_table(btcoexist, force_exec,
- 0x55555555, 0x5a5a5a5a,
- 0xffffff, 0x3);
- break;
+ btc8821a1ant_coex_table(btcoexist, force_exec, 0x55555555,
+ 0x5a5a5a5a, 0xffffff, 0x3);
+ break;
case 2:
- halbtc8821a1ant_coex_table(btcoexist, force_exec, 0x5a5a5a5a,
- 0x5a5a5a5a, 0xffffff, 0x3);
+ btc8821a1ant_coex_table(btcoexist, force_exec, 0x5a5a5a5a,
+ 0x5a5a5a5a, 0xffffff, 0x3);
break;
case 3:
- halbtc8821a1ant_coex_table(btcoexist, force_exec, 0x55555555,
- 0xaaaaaaaa, 0xffffff, 0x3);
+ btc8821a1ant_coex_table(btcoexist, force_exec, 0x5a5a5a5a,
+ 0xaaaaaaaa, 0xffffff, 0x3);
break;
case 4:
- halbtc8821a1ant_coex_table(btcoexist, force_exec, 0xffffffff,
- 0xffffffff, 0xffffff, 0x3);
+ btc8821a1ant_coex_table(btcoexist, force_exec, 0x55555555,
+ 0x5a5a5a5a, 0xffffff, 0x3);
break;
case 5:
- halbtc8821a1ant_coex_table(btcoexist, force_exec, 0x5fff5fff,
- 0x5fff5fff, 0xffffff, 0x3);
+ btc8821a1ant_coex_table(btcoexist, force_exec, 0x5a5a5a5a,
+ 0xaaaa5a5a, 0xffffff, 0x3);
break;
case 6:
- halbtc8821a1ant_coex_table(btcoexist, force_exec, 0x55ff55ff,
- 0x5a5a5a5a, 0xffffff, 0x3);
+ btc8821a1ant_coex_table(btcoexist, force_exec, 0x55555555,
+ 0xaaaa5a5a, 0xffffff, 0x3);
break;
case 7:
- halbtc8821a1ant_coex_table(btcoexist, force_exec, 0x5afa5afa,
- 0x5afa5afa, 0xffffff, 0x3);
+ btc8821a1ant_coex_table(btcoexist, force_exec, 0xaaaaaaaa,
+ 0xaaaaaaaa, 0xffffff, 0x3);
break;
default:
break;
@@ -853,10 +836,10 @@
bool enable)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- u8 h2c_parameter[1] = {0};
+ u8 h2c_parameter[1] = {0};
if (enable)
- h2c_parameter[0] |= BIT0; /* function enable*/
+ h2c_parameter[0] |= BIT0; /* function enable */
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], set FW for BT Ignore Wlan_Act, FW write 0x63 = 0x%x\n",
@@ -865,8 +848,8 @@
btcoexist->btc_fill_h2c(btcoexist, 0x63, 1, h2c_parameter);
}
-static void halbtc8821a1ant_ignore_wlan_act(struct btc_coexist *btcoexist,
- bool force_exec, bool enable)
+static void btc8821a1ant_ignore_wlan_act(struct btc_coexist *btcoexist,
+ bool force_exec, bool enable)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -890,24 +873,40 @@
coex_dm->pre_ignore_wlan_act = coex_dm->cur_ignore_wlan_act;
}
-static void halbtc8821a1ant_set_fw_pstdma(struct btc_coexist *btcoexist,
- u8 byte1, u8 byte2, u8 byte3,
- u8 byte4, u8 byte5)
+static void btc8821a1ant_set_fw_ps_tdma(struct btc_coexist *btcoexist, u8 byte1,
+ u8 byte2, u8 byte3, u8 byte4, u8 byte5)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[5] = {0};
+ u8 real_byte1 = byte1, real_byte5 = byte5;
+ bool ap_enable = false;
- h2c_parameter[0] = byte1;
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_AP_MODE_ENABLE,
+ &ap_enable);
+
+ if (ap_enable) {
+ if (byte1 & BIT4 && !(byte1 & BIT5)) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], FW for 1Ant AP mode\n");
+ real_byte1 &= ~BIT4;
+ real_byte1 |= BIT5;
+
+ real_byte5 |= BIT5;
+ real_byte5 &= ~BIT6;
+ }
+ }
+
+ h2c_parameter[0] = real_byte1;
h2c_parameter[1] = byte2;
h2c_parameter[2] = byte3;
h2c_parameter[3] = byte4;
- h2c_parameter[4] = byte5;
+ h2c_parameter[4] = real_byte5;
- coex_dm->ps_tdma_para[0] = byte1;
+ coex_dm->ps_tdma_para[0] = real_byte1;
coex_dm->ps_tdma_para[1] = byte2;
coex_dm->ps_tdma_para[2] = byte3;
coex_dm->ps_tdma_para[3] = byte4;
- coex_dm->ps_tdma_para[4] = byte5;
+ coex_dm->ps_tdma_para[4] = real_byte5;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], PS-TDMA H2C cmd =0x%x%08x\n",
@@ -919,18 +918,18 @@
btcoexist->btc_fill_h2c(btcoexist, 0x60, 5, h2c_parameter);
}
-static void halbtc8821a1ant_set_lps_rpwm(struct btc_coexist *btcoexist,
- u8 lps_val, u8 rpwm_val)
+static void btc8821a1ant_set_lps_rpwm(struct btc_coexist *btcoexist,
+ u8 lps_val, u8 rpwm_val)
{
- u8 lps = lps_val;
- u8 rpwm = rpwm_val;
+ u8 lps = lps_val;
+ u8 rpwm = rpwm_val;
btcoexist->btc_set(btcoexist, BTC_SET_U1_LPS_VAL, &lps);
btcoexist->btc_set(btcoexist, BTC_SET_U1_RPWM_VAL, &rpwm);
}
-static void halbtc8821a1ant_lps_rpwm(struct btc_coexist *btcoexist,
- bool force_exec, u8 lps_val, u8 rpwm_val)
+static void btc8821a1ant_lps_rpwm(struct btc_coexist *btcoexist,
+ bool force_exec, u8 lps_val, u8 rpwm_val)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -954,33 +953,33 @@
return;
}
}
- halbtc8821a1ant_set_lps_rpwm(btcoexist, lps_val, rpwm_val);
+ btc8821a1ant_set_lps_rpwm(btcoexist, lps_val, rpwm_val);
coex_dm->pre_lps = coex_dm->cur_lps;
coex_dm->pre_rpwm = coex_dm->cur_rpwm;
}
-static void halbtc8821a1ant_sw_mechanism(struct btc_coexist *btcoexist,
- bool low_penalty_ra)
+static void btc8821a1ant_sw_mechanism(struct btc_coexist *btcoexist,
+ bool low_penalty_ra)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], SM[LpRA] = %d\n", low_penalty_ra);
- halbtc8821a1ant_low_penalty_ra(btcoexist, NORMAL_EXEC, low_penalty_ra);
+ btc8821a1ant_low_penalty_ra(btcoexist, NORMAL_EXEC, low_penalty_ra);
}
-static void halbtc8821a1ant_set_ant_path(struct btc_coexist *btcoexist,
- u8 ant_pos_type, bool init_hw_cfg,
- bool wifi_off)
+static void btc8821a1ant_set_ant_path(struct btc_coexist *btcoexist,
+ u8 ant_pos_type, bool init_hw_cfg,
+ bool wifi_off)
{
struct btc_board_info *board_info = &btcoexist->board_info;
u32 u4_tmp = 0;
u8 h2c_parameter[2] = {0};
if (init_hw_cfg) {
- /* 0x4c[23] = 0, 0x4c[24] = 1 Antenna control by WL/BT*/
+ /* 0x4c[23] = 0, 0x4c[24] = 1 Antenna control by WL/BT */
u4_tmp = btcoexist->btc_read_4byte(btcoexist, 0x4c);
u4_tmp &= ~BIT23;
u4_tmp |= BIT24;
@@ -990,41 +989,42 @@
btcoexist->btc_write_1byte(btcoexist, 0xcb4, 0x77);
if (board_info->btdm_ant_pos == BTC_ANTENNA_AT_MAIN_PORT) {
- /*tell firmware "antenna inverse" ==>
- * WRONG firmware antenna control code.==>need fw to fix
+ /* tell firmware "antenna inverse"
+ * WRONG firmware antenna control code, need fw to fix
*/
h2c_parameter[0] = 1;
h2c_parameter[1] = 1;
btcoexist->btc_fill_h2c(btcoexist, 0x65, 2,
h2c_parameter);
- /*Main Ant to BT for IPS case 0x4c[23] = 1*/
- btcoexist->btc_write_1byte_bitmask(btcoexist, 0x64,
- 0x1, 0x1);
} else {
- /*tell firmware "no antenna inverse" ==>
- * WRONG firmware antenna control code.==>need fw to fix
+ /* tell firmware "no antenna inverse"
+ * WRONG firmware antenna control code, need fw to fix
*/
h2c_parameter[0] = 0;
h2c_parameter[1] = 1;
btcoexist->btc_fill_h2c(btcoexist, 0x65, 2,
h2c_parameter);
- /*Aux Ant to BT for IPS case 0x4c[23] = 1*/
- btcoexist->btc_write_1byte_bitmask(btcoexist, 0x64,
- 0x1, 0x0);
}
} else if (wifi_off) {
/* 0x4c[24:23] = 00, Set Antenna control
- * by BT_RFE_CTRL BT Vendor 0xac = 0xf002
+ * by BT_RFE_CTRL BT Vendor 0xac = 0xf002
*/
u4_tmp = btcoexist->btc_read_4byte(btcoexist, 0x4c);
u4_tmp &= ~BIT23;
u4_tmp &= ~BIT24;
btcoexist->btc_write_4byte(btcoexist, 0x4c, u4_tmp);
+
+ /* 0x765 = 0x18 */
+ btcoexist->btc_write_1byte_bitmask(btcoexist, 0x765, 0x18, 0x3);
+ } else {
+ /* 0x765 = 0x0 */
+ btcoexist->btc_write_1byte_bitmask(btcoexist, 0x765, 0x18, 0x0);
}
- /* ext switch setting*/
+ /* ext switch setting */
switch (ant_pos_type) {
case BTC_ANT_PATH_WIFI:
+ btcoexist->btc_write_1byte(btcoexist, 0xcb4, 0x77);
if (board_info->btdm_ant_pos == BTC_ANTENNA_AT_MAIN_PORT)
btcoexist->btc_write_1byte_bitmask(btcoexist, 0xcb7,
0x30, 0x1);
@@ -1033,6 +1033,7 @@
0x30, 0x2);
break;
case BTC_ANT_PATH_BT:
+ btcoexist->btc_write_1byte(btcoexist, 0xcb4, 0x77);
if (board_info->btdm_ant_pos == BTC_ANTENNA_AT_MAIN_PORT)
btcoexist->btc_write_1byte_bitmask(btcoexist, 0xcb7,
0x30, 0x2);
@@ -1042,6 +1043,7 @@
break;
default:
case BTC_ANT_PATH_PTA:
+ btcoexist->btc_write_1byte(btcoexist, 0xcb4, 0x66);
if (board_info->btdm_ant_pos == BTC_ANTENNA_AT_MAIN_PORT)
btcoexist->btc_write_1byte_bitmask(btcoexist, 0xcb7,
0x30, 0x1);
@@ -1052,8 +1054,8 @@
}
}
-static void halbtc8821a1ant_ps_tdma(struct btc_coexist *btcoexist,
- bool force_exec, bool turn_on, u8 type)
+static void btc8821a1ant_ps_tdma(struct btc_coexist *btcoexist,
+ bool force_exec, bool turn_on, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 rssi_adjust_val = 0;
@@ -1078,185 +1080,189 @@
if (turn_on) {
switch (type) {
default:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x51, 0x1a,
- 0x1a, 0x0, 0x50);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x51, 0x1a,
+ 0x1a, 0x0, 0x50);
break;
case 1:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x51, 0x3a,
- 0x03, 0x10, 0x50);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x51, 0x3a,
+ 0x03, 0x10, 0x50);
rssi_adjust_val = 11;
break;
case 2:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x51, 0x2b,
- 0x03, 0x10, 0x50);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x51, 0x2b,
+ 0x03, 0x10, 0x50);
rssi_adjust_val = 14;
break;
case 3:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x51, 0x1d,
- 0x1d, 0x0, 0x10);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x51, 0x1d,
+ 0x1d, 0x0, 0x10);
break;
case 4:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x93, 0x15,
- 0x3, 0x14, 0x0);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x93, 0x15,
+ 0x3, 0x14, 0x0);
rssi_adjust_val = 17;
break;
case 5:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x61, 0x15,
- 0x3, 0x11, 0x10);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x61, 0x15,
+ 0x3, 0x11, 0x10);
break;
case 6:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x13, 0xa,
- 0x3, 0x0, 0x0);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x13, 0xa,
+ 0x3, 0x0, 0x0);
break;
case 7:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x13, 0xc,
- 0x5, 0x0, 0x0);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x13, 0xc,
+ 0x5, 0x0, 0x0);
break;
case 8:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x93, 0x25,
- 0x3, 0x10, 0x0);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x93, 0x25,
+ 0x3, 0x10, 0x0);
break;
case 9:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x51, 0x21,
- 0x3, 0x10, 0x50);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x51, 0x21,
+ 0x3, 0x10, 0x50);
rssi_adjust_val = 18;
break;
case 10:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x13, 0xa,
- 0xa, 0x0, 0x40);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x13, 0xa,
+ 0xa, 0x0, 0x40);
break;
case 11:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x51, 0x14,
- 0x03, 0x10, 0x10);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x51, 0x14,
+ 0x03, 0x10, 0x10);
rssi_adjust_val = 20;
break;
case 12:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x51, 0x0a,
- 0x0a, 0x0, 0x50);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x51, 0x0a,
+ 0x0a, 0x0, 0x50);
break;
case 13:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x51, 0x18,
- 0x18, 0x0, 0x10);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x51, 0x18,
+ 0x18, 0x0, 0x10);
break;
case 14:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x51, 0x21,
- 0x3, 0x10, 0x10);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x51, 0x1e,
+ 0x3, 0x10, 0x14);
break;
case 15:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x13, 0xa,
- 0x3, 0x8, 0x0);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x13, 0xa,
+ 0x3, 0x8, 0x0);
break;
case 16:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x93, 0x15,
- 0x3, 0x10, 0x0);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x93, 0x15,
+ 0x3, 0x10, 0x0);
rssi_adjust_val = 18;
break;
case 18:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x93, 0x25,
- 0x3, 0x10, 0x0);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x93, 0x25,
+ 0x3, 0x10, 0x0);
rssi_adjust_val = 14;
break;
case 20:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x61, 0x35,
- 0x03, 0x11, 0x10);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x61, 0x35,
+ 0x03, 0x11, 0x10);
break;
case 21:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x61, 0x15,
- 0x03, 0x11, 0x10);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x61, 0x15,
+ 0x03, 0x11, 0x10);
break;
case 22:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x61, 0x25,
- 0x03, 0x11, 0x10);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x61, 0x25,
+ 0x03, 0x11, 0x10);
break;
case 23:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0xe3, 0x25,
- 0x3, 0x31, 0x18);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x25,
+ 0x3, 0x31, 0x18);
rssi_adjust_val = 22;
break;
case 24:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0xe3, 0x15,
- 0x3, 0x31, 0x18);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x15,
+ 0x3, 0x31, 0x18);
rssi_adjust_val = 22;
break;
case 25:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0xe3, 0xa,
- 0x3, 0x31, 0x18);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0xe3, 0xa,
+ 0x3, 0x31, 0x18);
rssi_adjust_val = 22;
break;
case 26:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0xe3, 0xa,
- 0x3, 0x31, 0x18);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0xe3, 0xa,
+ 0x3, 0x31, 0x18);
rssi_adjust_val = 22;
break;
case 27:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0xe3, 0x25,
- 0x3, 0x31, 0x98);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x25,
+ 0x3, 0x31, 0x98);
rssi_adjust_val = 22;
break;
case 28:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x69, 0x25,
- 0x3, 0x31, 0x0);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x69, 0x25,
+ 0x3, 0x31, 0x0);
break;
case 29:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0xab, 0x1a,
- 0x1a, 0x1, 0x10);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0xab, 0x1a,
+ 0x1a, 0x1, 0x10);
break;
case 30:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x51, 0x14,
- 0x3, 0x10, 0x50);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x51, 0x14,
+ 0x3, 0x10, 0x50);
break;
case 31:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0xd3, 0x1a,
- 0x1a, 0, 0x58);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0xd3, 0x1a,
+ 0x1a, 0, 0x58);
break;
case 32:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x61, 0xa,
- 0x3, 0x10, 0x0);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x61, 0xa,
+ 0x3, 0x10, 0x0);
break;
case 33:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0xa3, 0x25,
- 0x3, 0x30, 0x90);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0xa3, 0x25,
+ 0x3, 0x30, 0x90);
break;
case 34:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x53, 0x1a,
- 0x1a, 0x0, 0x10);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x53, 0x1a,
+ 0x1a, 0x0, 0x10);
break;
case 35:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x63, 0x1a,
- 0x1a, 0x0, 0x10);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x63, 0x1a,
+ 0x1a, 0x0, 0x10);
break;
case 36:
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0xd3, 0x12,
- 0x3, 0x14, 0x50);
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0xd3, 0x12,
+ 0x3, 0x14, 0x50);
break;
}
} else {
- /* disable PS tdma*/
+ /* disable PS tdma */
switch (type) {
- case 8: /*PTA Control*/
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x8, 0x0, 0x0,
- 0x0, 0x0);
- halbtc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_PTA,
- false, false);
+ case 8:
+ /* PTA Control */
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x8, 0x0, 0x0,
+ 0x0, 0x0);
+ btc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_PTA,
+ false, false);
break;
case 0:
- default: /*Software control, Antenna at BT side*/
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x0, 0x0, 0x0,
- 0x0, 0x0);
- halbtc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_BT,
- false, false);
+ default:
+ /* Software control, Antenna at BT side */
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x0, 0x0, 0x0,
+ 0x0, 0x0);
+ btc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_BT,
+ false, false);
break;
- case 9: /*Software control, Antenna at WiFi side*/
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x0, 0x0, 0x0,
- 0x0, 0x0);
- halbtc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_WIFI,
- false, false);
+ case 9:
+ /* Software control, Antenna at WiFi side */
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x0, 0x0, 0x0,
+ 0x0, 0x0);
+ btc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_WIFI,
+ false, false);
break;
- case 10: /* under 5G*/
- halbtc8821a1ant_set_fw_pstdma(btcoexist, 0x0, 0x0, 0x0,
- 0x8, 0x0);
- halbtc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_BT,
- false, false);
+ case 10:
+ /* under 5G */
+ btc8821a1ant_set_fw_ps_tdma(btcoexist, 0x0, 0x0, 0x0,
+ 0x8, 0x0);
+ btc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_BT,
+ false, false);
break;
}
}
@@ -1264,15 +1270,15 @@
btcoexist->btc_set(btcoexist,
BTC_SET_U1_RSSI_ADJ_VAL_FOR_1ANT_COEX_TYPE, &rssi_adjust_val);
- /* update pre state*/
+ /* update pre state */
coex_dm->pre_ps_tdma_on = coex_dm->cur_ps_tdma_on;
coex_dm->pre_ps_tdma = coex_dm->cur_ps_tdma;
}
-static bool halbtc8821a1ant_is_common_action(struct btc_coexist *btcoexist)
+static bool btc8821a1ant_is_common_action(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- bool common = false, wifi_connected = false, wifi_busy = false;
+ bool common = false, wifi_connected = false, wifi_busy = false;
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
&wifi_connected);
@@ -1283,7 +1289,7 @@
coex_dm->bt_status) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Wifi non connected-idle + BT non connected-idle!!\n");
- halbtc8821a1ant_sw_mechanism(btcoexist, false);
+ btc8821a1ant_sw_mechanism(btcoexist, false);
common = true;
} else if (wifi_connected &&
@@ -1291,7 +1297,7 @@
coex_dm->bt_status)) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Wifi connected + BT non connected-idle!!\n");
- halbtc8821a1ant_sw_mechanism(btcoexist, false);
+ btc8821a1ant_sw_mechanism(btcoexist, false);
common = true;
} else if (!wifi_connected &&
@@ -1299,15 +1305,15 @@
coex_dm->bt_status)) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Wifi non connected-idle + BT connected-idle!!\n");
- halbtc8821a1ant_sw_mechanism(btcoexist, false);
+ btc8821a1ant_sw_mechanism(btcoexist, false);
common = true;
} else if (wifi_connected &&
(BT_8821A_1ANT_BT_STATUS_CONNECTED_IDLE ==
- coex_dm->bt_status)) {
+ coex_dm->bt_status)) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Wifi connected + BT connected-idle!!\n");
- halbtc8821a1ant_sw_mechanism(btcoexist, false);
+ btc8821a1ant_sw_mechanism(btcoexist, false);
common = true;
} else if (!wifi_connected &&
@@ -1315,7 +1321,7 @@
coex_dm->bt_status)) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Wifi non connected-idle + BT Busy!!\n");
- halbtc8821a1ant_sw_mechanism(btcoexist, false);
+ btc8821a1ant_sw_mechanism(btcoexist, false);
common = true;
} else {
@@ -1333,231 +1339,40 @@
return common;
}
-static void btc8821a1ant_tdma_dur_adj(struct btc_coexist *btcoexist,
- u8 wifi_status)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- static long up, dn, m, n, wait_count;
- /*0: no change, +1: increase WiFi duration, -1: decrease WiFi duration*/
- long result;
- u8 retry_count = 0, bt_info_ext;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TdmaDurationAdjustForAcl()\n");
-
- if ((BT_8821A_1ANT_WIFI_STATUS_NON_CONNECTED_ASSO_AUTH_SCAN ==
- wifi_status) ||
- (BT_8821A_1ANT_WIFI_STATUS_CONNECTED_SCAN ==
- wifi_status) ||
- (BT_8821A_1ANT_WIFI_STATUS_CONNECTED_SPECIAL_PKT ==
- wifi_status)) {
- if (coex_dm->cur_ps_tdma != 1 &&
- coex_dm->cur_ps_tdma != 2 &&
- coex_dm->cur_ps_tdma != 3 &&
- coex_dm->cur_ps_tdma != 9) {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
-
- up = 0;
- dn = 0;
- m = 1;
- n = 3;
- result = 0;
- wait_count = 0;
- }
- return;
- }
-
- if (!coex_dm->auto_tdma_adjust) {
- coex_dm->auto_tdma_adjust = true;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], first run TdmaDurationAdjust()!!\n");
-
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 2);
- coex_dm->tdma_adj_type = 2;
- /*============*/
- up = 0;
- dn = 0;
- m = 1;
- n = 3;
- result = 0;
- wait_count = 0;
- } else {
- /*accquire the BT TRx retry count from BT_Info byte2*/
- retry_count = coex_sta->bt_retry_cnt;
- bt_info_ext = coex_sta->bt_info_ext;
- result = 0;
- wait_count++;
-
- if (retry_count == 0) {
- /* no retry in the last 2-second duration*/
- up++;
- dn--;
-
- if (dn <= 0)
- dn = 0;
-
- if (up >= n) {
- /* if (retry count == 0) for 2*n seconds ,
- * make WiFi duration wider
- */
- wait_count = 0;
- n = 3;
- up = 0;
- dn = 0;
- result = 1;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Increase wifi duration!!\n");
- }
- } else if (retry_count <= 3) {
- /* <=3 retry in the last 2-second duration*/
- up--;
- dn++;
-
- if (up <= 0)
- up = 0;
-
- if (dn == 2) {
- /* if retry count< 3 for 2*2 seconds,
- * shrink wifi duration
- */
- if (wait_count <= 2)
- m++; /* avoid bounce in two levels */
- else
- m = 1;
-
- if (m >= 20) {
- /* m max value is 20, max time is 120 s,
- * recheck if adjust WiFi duration.
- */
- m = 20;
- }
- n = 3*m;
- up = 0;
- dn = 0;
- wait_count = 0;
- result = -1;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Decrease wifi duration for retryCounter<3!!\n");
- }
- } else {
- /* retry count > 3, if retry count > 3 happens once,
- * shrink WiFi duration
- */
- if (wait_count == 1)
- m++; /* avoid bounce in two levels */
- else
- m = 1;
- /* m max value is 20, max time is 120 second,
- * recheck if adjust WiFi duration.
- */
- if (m >= 20)
- m = 20;
-
- n = 3*m;
- up = 0;
- dn = 0;
- wait_count = 0;
- result = -1;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Decrease wifi duration for retryCounter>3!!\n");
- }
-
- if (result == -1) {
- if ((BT_INFO_8821A_1ANT_A2DP_BASIC_RATE(bt_info_ext)) &&
- ((coex_dm->cur_ps_tdma == 1) ||
- (coex_dm->cur_ps_tdma == 2))) {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
- } else if (coex_dm->cur_ps_tdma == 1) {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
- } else if (coex_dm->cur_ps_tdma == 9) {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- }
- } else if (result == 1) {
- if ((BT_INFO_8821A_1ANT_A2DP_BASIC_RATE(bt_info_ext)) &&
- ((coex_dm->cur_ps_tdma == 1) ||
- (coex_dm->cur_ps_tdma == 2))) {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
- } else if (coex_dm->cur_ps_tdma == 9) {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 1);
- coex_dm->tdma_adj_type = 1;
- }
- } else {
- /*no change*/
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], ********** TDMA(on, %d) **********\n",
- coex_dm->cur_ps_tdma);
- }
-
- if (coex_dm->cur_ps_tdma != 1 &&
- coex_dm->cur_ps_tdma != 2 &&
- coex_dm->cur_ps_tdma != 9 &&
- coex_dm->cur_ps_tdma != 11) {
- /* recover to previous adjust type*/
- halbtc8821a1ant_ps_tdma(btcoexist,
- NORMAL_EXEC, true,
- coex_dm->tdma_adj_type);
- }
- }
-}
-
static void btc8821a1ant_ps_tdma_check_for_pwr_save(struct btc_coexist *btcoex,
bool new_ps_state)
{
- u8 lps_mode = 0x0;
+ u8 lps_mode = 0x0;
btcoex->btc_get(btcoex, BTC_GET_U1_LPS_MODE, &lps_mode);
if (lps_mode) {
- /* already under LPS state*/
+ /* already under LPS state */
if (new_ps_state) {
- /* keep state under LPS, do nothing.*/
+ /* keep state under LPS, do nothing */
} else {
- /* will leave LPS state, turn off psTdma first*/
- halbtc8821a1ant_ps_tdma(btcoex, NORMAL_EXEC, false, 0);
+ /* will leave LPS state, turn off psTdma first */
+ btc8821a1ant_ps_tdma(btcoex, NORMAL_EXEC, false, 0);
}
} else {
/* NO PS state*/
if (new_ps_state) {
- /* will enter LPS state, turn off psTdma first*/
- halbtc8821a1ant_ps_tdma(btcoex, NORMAL_EXEC, false, 0);
+ /* will enter LPS state, turn off psTdma first */
+ btc8821a1ant_ps_tdma(btcoex, NORMAL_EXEC, false, 0);
} else {
- /* keep state under NO PS state, do nothing.*/
+ /* keep state under NO PS state, do nothing */
}
}
}
-static void halbtc8821a1ant_power_save_state(struct btc_coexist *btcoexist,
- u8 ps_type, u8 lps_val,
- u8 rpwm_val)
+static void btc8821a1ant_power_save_state(struct btc_coexist *btcoexist,
+ u8 ps_type, u8 lps_val, u8 rpwm_val)
{
bool low_pwr_disable = false;
switch (ps_type) {
case BTC_PS_WIFI_NATIVE:
- /* recover to original 32k low power setting*/
+ /* recover to original 32k low power setting */
low_pwr_disable = false;
btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
&low_pwr_disable);
@@ -1566,13 +1381,13 @@
case BTC_PS_LPS_ON:
btc8821a1ant_ps_tdma_check_for_pwr_save(btcoexist,
true);
- halbtc8821a1ant_lps_rpwm(btcoexist,
- NORMAL_EXEC, lps_val, rpwm_val);
- /* when coex force to enter LPS, do not enter 32k low power.*/
+ btc8821a1ant_lps_rpwm(btcoexist, NORMAL_EXEC, lps_val,
+ rpwm_val);
+ /* when coex force to enter LPS, do not enter 32k low power */
low_pwr_disable = true;
btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
&low_pwr_disable);
- /* power save must executed before psTdma.*/
+ /* power save must executed before psTdma */
btcoexist->btc_set(btcoexist, BTC_SET_ACT_ENTER_LPS, NULL);
break;
case BTC_PS_LPS_OFF:
@@ -1584,295 +1399,332 @@
}
}
-static void halbtc8821a1ant_coex_under_5g(struct btc_coexist *btcoexist)
+static void btc8821a1ant_coex_under_5g(struct btc_coexist *btcoexist)
{
- halbtc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
- 0x0, 0x0);
- halbtc8821a1ant_ignore_wlan_act(btcoexist, NORMAL_EXEC, true);
+ btc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ btc8821a1ant_ignore_wlan_act(btcoexist, NORMAL_EXEC, true);
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 10);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 10);
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
- halbtc8821a1ant_limited_tx(btcoexist, NORMAL_EXEC, 0, 0, 0, 0);
+ btc8821a1ant_limited_tx(btcoexist, NORMAL_EXEC, 0, 0, 0, 0);
- halbtc8821a1ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 5);
+ btc8821a1ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 5);
}
-static void halbtc8821a1ant_action_wifi_only(struct btc_coexist *btcoexist)
+/***********************************************
+ *
+ * Software Coex Mechanism start
+ *
+ ***********************************************/
+
+/* SCO only or SCO+PAN(HS) */
+static void btc8821a1ant_action_sco(struct btc_coexist *btcoexist)
{
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 9);
+ btc8821a1ant_sw_mechanism(btcoexist, true);
}
-static void btc8821a1ant_mon_bt_en_dis(struct btc_coexist *btcoexist)
+static void btc8821a1ant_action_hid(struct btc_coexist *btcoexist)
{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- static bool pre_bt_disabled;
- static u32 bt_disable_cnt;
- bool bt_active = true, bt_disabled = false;
-
- /* This function check if bt is disabled*/
-
- if (coex_sta->high_priority_tx == 0 &&
- coex_sta->high_priority_rx == 0 &&
- coex_sta->low_priority_tx == 0 &&
- coex_sta->low_priority_rx == 0) {
- bt_active = false;
- }
- if (coex_sta->high_priority_tx == 0xffff &&
- coex_sta->high_priority_rx == 0xffff &&
- coex_sta->low_priority_tx == 0xffff &&
- coex_sta->low_priority_rx == 0xffff) {
- bt_active = false;
- }
- if (bt_active) {
- bt_disable_cnt = 0;
- bt_disabled = false;
- btcoexist->btc_set(btcoexist, BTC_SET_BL_BT_DISABLE,
- &bt_disabled);
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT is enabled !!\n");
- } else {
- bt_disable_cnt++;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], bt all counters = 0, %d times!!\n",
- bt_disable_cnt);
- if (bt_disable_cnt >= 2) {
- bt_disabled = true;
- btcoexist->btc_set(btcoexist, BTC_SET_BL_BT_DISABLE,
- &bt_disabled);
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT is disabled !!\n");
- halbtc8821a1ant_action_wifi_only(btcoexist);
- }
- }
- if (pre_bt_disabled != bt_disabled) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT is from %s to %s!!\n",
- (pre_bt_disabled ? "disabled" : "enabled"),
- (bt_disabled ? "disabled" : "enabled"));
- pre_bt_disabled = bt_disabled;
- if (bt_disabled) {
- btcoexist->btc_set(btcoexist, BTC_SET_ACT_LEAVE_LPS,
- NULL);
- btcoexist->btc_set(btcoexist, BTC_SET_ACT_NORMAL_LPS,
- NULL);
- }
- }
+ btc8821a1ant_sw_mechanism(btcoexist, true);
}
-/*=============================================*/
-/**/
-/* Software Coex Mechanism start*/
-/**/
-/*=============================================*/
-
-/* SCO only or SCO+PAN(HS)*/
-static void halbtc8821a1ant_action_sco(struct btc_coexist *btcoexist)
+/* A2DP only / PAN(EDR) only/ A2DP+PAN(HS) */
+static void btc8821a1ant_action_a2dp(struct btc_coexist *btcoexist)
{
- halbtc8821a1ant_sw_mechanism(btcoexist, true);
+ btc8821a1ant_sw_mechanism(btcoexist, false);
}
-static void halbtc8821a1ant_action_hid(struct btc_coexist *btcoexist)
+static void btc8821a1ant_action_a2dp_pan_hs(struct btc_coexist *btcoexist)
{
- halbtc8821a1ant_sw_mechanism(btcoexist, true);
+ btc8821a1ant_sw_mechanism(btcoexist, false);
}
-/*A2DP only / PAN(EDR) only/ A2DP+PAN(HS)*/
-static void halbtc8821a1ant_action_a2dp(struct btc_coexist *btcoexist)
+static void btc8821a1ant_action_pan_edr(struct btc_coexist *btcoexist)
{
- halbtc8821a1ant_sw_mechanism(btcoexist, false);
+ btc8821a1ant_sw_mechanism(btcoexist, false);
}
-static void halbtc8821a1ant_action_a2dp_pan_hs(struct btc_coexist *btcoexist)
+/* PAN(HS) only */
+static void btc8821a1ant_action_pan_hs(struct btc_coexist *btcoexist)
{
- halbtc8821a1ant_sw_mechanism(btcoexist, false);
+ btc8821a1ant_sw_mechanism(btcoexist, false);
}
-static void halbtc8821a1ant_action_pan_edr(struct btc_coexist *btcoexist)
+/* PAN(EDR)+A2DP */
+static void btc8821a1ant_action_pan_edr_a2dp(struct btc_coexist *btcoexist)
{
- halbtc8821a1ant_sw_mechanism(btcoexist, false);
+ btc8821a1ant_sw_mechanism(btcoexist, false);
}
-/*PAN(HS) only*/
-static void halbtc8821a1ant_action_pan_hs(struct btc_coexist *btcoexist)
+static void btc8821a1ant_action_pan_edr_hid(struct btc_coexist *btcoexist)
{
- halbtc8821a1ant_sw_mechanism(btcoexist, false);
+ btc8821a1ant_sw_mechanism(btcoexist, true);
}
-/*PAN(EDR)+A2DP*/
-static void halbtc8821a1ant_action_pan_edr_a2dp(struct btc_coexist *btcoexist)
-{
- halbtc8821a1ant_sw_mechanism(btcoexist, false);
-}
-
-static void halbtc8821a1ant_action_pan_edr_hid(struct btc_coexist *btcoexist)
-{
- halbtc8821a1ant_sw_mechanism(btcoexist, true);
-}
-
-/* HID+A2DP+PAN(EDR)*/
+/* HID+A2DP+PAN(EDR) */
static void btc8821a1ant_action_hid_a2dp_pan_edr(struct btc_coexist *btcoexist)
{
- halbtc8821a1ant_sw_mechanism(btcoexist, true);
+ btc8821a1ant_sw_mechanism(btcoexist, true);
}
-static void halbtc8821a1ant_action_hid_a2dp(struct btc_coexist *btcoexist)
+static void btc8821a1ant_action_hid_a2dp(struct btc_coexist *btcoexist)
{
- halbtc8821a1ant_sw_mechanism(btcoexist, true);
+ btc8821a1ant_sw_mechanism(btcoexist, true);
}
-/*=============================================*/
-/**/
-/* Non-Software Coex Mechanism start*/
-/**/
-/*=============================================*/
-
-static void halbtc8821a1ant_action_hs(struct btc_coexist *btcoexist)
+/***********************************************
+ *
+ * Non-Software Coex Mechanism start
+ *
+ ***********************************************/
+static
+void btc8821a1ant_action_wifi_multi_port(struct btc_coexist *btcoexist)
{
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 5);
- halbtc8821a1ant_coex_table_with_type(btcoexist, FORCE_EXEC, 2);
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
+
+ btc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ /* tdma and coex table */
+ if (coex_dm->bt_status == BT_8821A_1ANT_BT_STATUS_ACL_BUSY) {
+ if (bt_link_info->a2dp_exist) {
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
+ btc8821a1ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 1);
+ } else if (bt_link_info->a2dp_exist &&
+ bt_link_info->pan_exist) {
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 8);
+ btc8821a1ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 4);
+ } else {
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 20);
+ btc8821a1ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 4);
+ }
+ } else if ((coex_dm->bt_status == BT_8821A_1ANT_BT_STATUS_SCO_BUSY) ||
+ (BT_8821A_1ANT_BT_STATUS_ACL_SCO_BUSY ==
+ coex_dm->bt_status)) {
+ btc8821a1ant_act_bt_sco_hid_only_busy(btcoexist,
+ BT_8821A_1ANT_WIFI_STATUS_CONNECTED_SCAN);
+ } else {
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 8);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
+ }
}
-static void halbtc8821a1ant_action_bt_inquiry(struct btc_coexist *btcoexist)
+static
+void btc8821a1ant_action_wifi_not_connected_asso_auth(
+ struct btc_coexist *btcoexist)
+{
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
+
+ btc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0,
+ 0x0);
+
+ /* tdma and coex table */
+ if ((bt_link_info->sco_exist) || (bt_link_info->hid_exist)) {
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ } else if ((bt_link_info->a2dp_exist) || (bt_link_info->pan_exist)) {
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 20);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 4);
+ } else {
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 8);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
+ }
+}
+
+
+static void btc8821a1ant_action_hs(struct btc_coexist *btcoexist)
+{
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 5);
+ btc8821a1ant_coex_table_with_type(btcoexist, FORCE_EXEC, 2);
+}
+
+static void btc8821a1ant_action_bt_inquiry(struct btc_coexist *btcoexist)
{
struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
bool wifi_connected = false;
+ bool ap_enable = false;
+ bool wifi_busy = false, bt_busy = false;
- btcoexist->btc_get(btcoexist,
- BTC_GET_BL_WIFI_CONNECTED, &wifi_connected);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
+ &wifi_connected);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_AP_MODE_ENABLE,
+ &ap_enable);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_BUSY, &wifi_busy);
+ btcoexist->btc_set(btcoexist, BTC_SET_BL_BT_TRAFFIC_BUSY, &bt_busy);
- if (!wifi_connected) {
- halbtc8821a1ant_power_save_state(btcoexist,
- BTC_PS_WIFI_NATIVE, 0x0, 0x0);
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 5);
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
- } else if ((bt_link_info->sco_exist) ||
+ if (!wifi_connected && !coex_sta->wifi_is_high_pri_task) {
+ btc8821a1ant_power_save_state(btcoexist,
+ BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 8);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
+ } else if ((bt_link_info->sco_exist) || (bt_link_info->a2dp_exist) ||
(bt_link_info->hid_only)) {
- /* SCO/HID-only busy*/
- halbtc8821a1ant_power_save_state(btcoexist,
- BTC_PS_WIFI_NATIVE, 0x0, 0x0);
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 32);
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ /* SCO/HID-only busy */
+ btc8821a1ant_power_save_state(btcoexist,
+ BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 32);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 4);
+ } else if ((bt_link_info->a2dp_exist) && (bt_link_info->hid_exist)) {
+ /* A2DP+HID busy */
+ btc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
+
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ } else if ((bt_link_info->pan_exist) || (wifi_busy)) {
+ btc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 20);
+
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 4);
} else {
- halbtc8821a1ant_power_save_state(btcoexist, BTC_PS_LPS_ON,
- 0x50, 0x4);
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 30);
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ btc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 8);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
}
}
static void btc8821a1ant_act_bt_sco_hid_only_busy(struct btc_coexist *btcoexist,
- u8 wifi_status) {
- /* tdma and coex table*/
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 5);
+ u8 wifi_status)
+{
+ /* tdma and coex table */
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 5);
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ if (BT_8821A_1ANT_WIFI_STATUS_NON_CONNECTED_ASSO_AUTH_SCAN ==
+ wifi_status)
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ else
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
}
static void btc8821a1ant_act_wifi_con_bt_acl_busy(struct btc_coexist *btcoexist,
u8 wifi_status)
{
- u8 bt_rssi_state;
+ u8 bt_rssi_state;
struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
- bt_rssi_state = halbtc8821a1ant_bt_rssi_state(btcoexist, 2, 28, 0);
+ bt_rssi_state = btc8821a1ant_bt_rssi_state(btcoexist, 2, 28, 0);
if (bt_link_info->hid_only) {
- /*HID*/
+ /* HID */
btc8821a1ant_act_bt_sco_hid_only_busy(btcoexist,
wifi_status);
coex_dm->auto_tdma_adjust = false;
return;
} else if (bt_link_info->a2dp_only) {
- /*A2DP*/
- if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a1ant_tdma_dur_adj(btcoexist, wifi_status);
- } else {
- /*for low BT RSSI*/
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
+ /* A2DP */
+ if ((bt_rssi_state != BTC_RSSI_STATE_HIGH) &&
+ (bt_rssi_state != BTC_RSSI_STATE_STAY_HIGH)) {
+ /* for low BT RSSI */
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
+ true, 11);
coex_dm->auto_tdma_adjust = false;
}
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
} else if (bt_link_info->hid_exist && bt_link_info->a2dp_exist) {
- /*HID+A2DP*/
+ /* HID+A2DP */
if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
(bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
+ true, 14);
coex_dm->auto_tdma_adjust = false;
} else {
/*for low BT RSSI*/
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
+ true, 11);
coex_dm->auto_tdma_adjust = false;
}
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
} else if ((bt_link_info->pan_only) ||
(bt_link_info->hid_exist && bt_link_info->pan_exist)) {
- /*PAN(OPP, FTP), HID+PAN(OPP, FTP)*/
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 3);
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ /* PAN(OPP, FTP), HID+PAN(OPP, FTP) */
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 3);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
coex_dm->auto_tdma_adjust = false;
} else if (((bt_link_info->a2dp_exist) && (bt_link_info->pan_exist)) ||
(bt_link_info->hid_exist && bt_link_info->a2dp_exist &&
bt_link_info->pan_exist)) {
- /*A2DP+PAN(OPP, FTP), HID+A2DP+PAN(OPP, FTP)*/
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 13);
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ /* A2DP+PAN(OPP, FTP), HID+A2DP+PAN(OPP, FTP) */
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 13);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
coex_dm->auto_tdma_adjust = false;
} else {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 11);
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 11);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
coex_dm->auto_tdma_adjust = false;
}
}
-static void halbtc8821a1ant_action_wifi_not_connected(
- struct btc_coexist *btcoexist)
+static
+void btc8821a1ant_action_wifi_not_connected(struct btc_coexist *btcoexist)
{
- /* power save state*/
- halbtc8821a1ant_power_save_state(btcoexist,
- BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ /* power save state */
+ btc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
- /* tdma and coex table*/
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 8);
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
+ /* tdma and coex table */
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 8);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
}
static void btc8821a1ant_act_wifi_not_conn_scan(struct btc_coexist *btcoexist)
{
- halbtc8821a1ant_power_save_state(btcoexist,
- BTC_PS_WIFI_NATIVE, 0x0, 0x0);
-
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 22);
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
-}
-
-static void halbtc8821a1ant_action_wifi_connected_scan(
- struct btc_coexist *btcoexist) {
struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
- /* power save state*/
- halbtc8821a1ant_power_save_state(btcoexist,
- BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ btc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
- /* tdma and coex table*/
+ /* tdma and coex table */
+ if (coex_dm->bt_status == BT_8821A_1ANT_BT_STATUS_ACL_BUSY) {
+ if (bt_link_info->a2dp_exist) {
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
+ btc8821a1ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 1);
+ } else if (bt_link_info->a2dp_exist &&
+ bt_link_info->pan_exist) {
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 22);
+ btc8821a1ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 4);
+ } else {
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 20);
+ btc8821a1ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 4);
+ }
+ } else if ((coex_dm->bt_status == BT_8821A_1ANT_BT_STATUS_SCO_BUSY) ||
+ (BT_8821A_1ANT_BT_STATUS_ACL_SCO_BUSY ==
+ coex_dm->bt_status)) {
+ btc8821a1ant_act_bt_sco_hid_only_busy(btcoexist,
+ BT_8821A_1ANT_WIFI_STATUS_CONNECTED_SCAN);
+ } else {
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 8);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
+ }
+}
+
+static
+void btc8821a1ant_action_wifi_connected_scan(struct btc_coexist *btcoexist)
+{
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
+
+ /* power save state */
+ btc8821a1ant_power_save_state(btcoexist,
+ BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+
+ /* tdma and coex table */
if (BT_8821A_1ANT_BT_STATUS_ACL_BUSY == coex_dm->bt_status) {
if (bt_link_info->a2dp_exist && bt_link_info->pan_exist) {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 22);
- halbtc8821a1ant_coex_table_with_type(btcoexist,
- NORMAL_EXEC, 1);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 22);
+ btc8821a1ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 1);
} else {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 20);
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 20);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
}
} else if ((BT_8821A_1ANT_BT_STATUS_SCO_BUSY ==
coex_dm->bt_status) ||
@@ -1881,52 +1733,52 @@
btc8821a1ant_act_bt_sco_hid_only_busy(btcoexist,
BT_8821A_1ANT_WIFI_STATUS_CONNECTED_SCAN);
} else {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 20);
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 20);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
}
}
static void btc8821a1ant_act_wifi_conn_sp_pkt(struct btc_coexist *btcoexist)
{
struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
- bool hs_connecting = false;
+ bool hs_connecting = false;
btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_CONNECTING, &hs_connecting);
- halbtc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
- 0x0, 0x0);
+ btc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
- /* tdma and coex table*/
- if (BT_8821A_1ANT_BT_STATUS_ACL_BUSY == coex_dm->bt_status) {
+ /* tdma and coex table */
+ if (coex_dm->bt_status == BT_8821A_1ANT_BT_STATUS_ACL_BUSY) {
if (bt_link_info->a2dp_exist && bt_link_info->pan_exist) {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 22);
- halbtc8821a1ant_coex_table_with_type(btcoexist,
- NORMAL_EXEC, 1);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
+ true, 22);
+ btc8821a1ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 1);
} else {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 20);
- halbtc8821a1ant_coex_table_with_type(btcoexist,
- NORMAL_EXEC, 1);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
+ true, 20);
+ btc8821a1ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 1);
}
} else {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 20);
- halbtc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 20);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 1);
}
}
-static void halbtc8821a1ant_action_wifi_connected(struct btc_coexist *btcoexist)
+static void btc8821a1ant_action_wifi_connected(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- bool wifi_busy = false;
- bool scan = false, link = false, roam = false;
- bool under_4way = false;
+ bool wifi_busy = false;
+ bool scan = false, link = false, roam = false;
+ bool under_4way = false;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], CoexForWifiConnect()===>\n");
- btcoexist->btc_get(btcoexist,
- BTC_GET_BL_WIFI_4_WAY_PROGRESS, &under_4way);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_4_WAY_PROGRESS,
+ &under_4way);
if (under_4way) {
btc8821a1ant_act_wifi_conn_sp_pkt(btcoexist);
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -1938,7 +1790,7 @@
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_LINK, &link);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_ROAM, &roam);
if (scan || link || roam) {
- halbtc8821a1ant_action_wifi_connected_scan(btcoexist);
+ btc8821a1ant_action_wifi_connected_scan(btcoexist);
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], CoexForWifiConnect(), return for wifi is under scan<===\n");
return;
@@ -1947,14 +1799,14 @@
/* power save state*/
if (BT_8821A_1ANT_BT_STATUS_ACL_BUSY ==
coex_dm->bt_status && !btcoexist->bt_link_info.hid_only)
- halbtc8821a1ant_power_save_state(btcoexist,
- BTC_PS_LPS_ON, 0x50, 0x4);
+ btc8821a1ant_power_save_state(btcoexist,
+ BTC_PS_LPS_ON, 0x50, 0x4);
else
- halbtc8821a1ant_power_save_state(btcoexist,
- BTC_PS_WIFI_NATIVE,
- 0x0, 0x0);
+ btc8821a1ant_power_save_state(btcoexist,
+ BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
- /* tdma and coex table*/
+ /* tdma and coex table */
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_BUSY, &wifi_busy);
if (!wifi_busy) {
if (BT_8821A_1ANT_BT_STATUS_ACL_BUSY == coex_dm->bt_status) {
@@ -1967,10 +1819,10 @@
btc8821a1ant_act_bt_sco_hid_only_busy(btcoexist,
BT_8821A_1ANT_WIFI_STATUS_CONNECTED_IDLE);
} else {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 5);
- halbtc8821a1ant_coex_table_with_type(btcoexist,
- NORMAL_EXEC, 2);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
+ true, 5);
+ btc8821a1ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 2);
}
} else {
if (BT_8821A_1ANT_BT_STATUS_ACL_BUSY == coex_dm->bt_status) {
@@ -1983,10 +1835,9 @@
btc8821a1ant_act_bt_sco_hid_only_busy(btcoexist,
BT_8821A_1ANT_WIFI_STATUS_CONNECTED_BUSY);
} else {
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 5);
- halbtc8821a1ant_coex_table_with_type(btcoexist,
- NORMAL_EXEC, 2);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 5);
+ btc8821a1ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 2);
}
}
}
@@ -1994,52 +1845,52 @@
static void btc8821a1ant_run_sw_coex_mech(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- u8 algorithm = 0;
+ u8 algorithm = 0;
- algorithm = halbtc8821a1ant_action_algorithm(btcoexist);
+ algorithm = btc8821a1ant_action_algorithm(btcoexist);
coex_dm->cur_algorithm = algorithm;
- if (!halbtc8821a1ant_is_common_action(btcoexist)) {
+ if (!btc8821a1ant_is_common_action(btcoexist)) {
switch (coex_dm->cur_algorithm) {
case BT_8821A_1ANT_COEX_ALGO_SCO:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action algorithm = SCO\n");
- halbtc8821a1ant_action_sco(btcoexist);
+ btc8821a1ant_action_sco(btcoexist);
break;
case BT_8821A_1ANT_COEX_ALGO_HID:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action algorithm = HID\n");
- halbtc8821a1ant_action_hid(btcoexist);
+ btc8821a1ant_action_hid(btcoexist);
break;
case BT_8821A_1ANT_COEX_ALGO_A2DP:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action algorithm = A2DP\n");
- halbtc8821a1ant_action_a2dp(btcoexist);
+ btc8821a1ant_action_a2dp(btcoexist);
break;
case BT_8821A_1ANT_COEX_ALGO_A2DP_PANHS:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action algorithm = A2DP+PAN(HS)\n");
- halbtc8821a1ant_action_a2dp_pan_hs(btcoexist);
+ btc8821a1ant_action_a2dp_pan_hs(btcoexist);
break;
case BT_8821A_1ANT_COEX_ALGO_PANEDR:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action algorithm = PAN(EDR)\n");
- halbtc8821a1ant_action_pan_edr(btcoexist);
+ btc8821a1ant_action_pan_edr(btcoexist);
break;
case BT_8821A_1ANT_COEX_ALGO_PANHS:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action algorithm = HS mode\n");
- halbtc8821a1ant_action_pan_hs(btcoexist);
+ btc8821a1ant_action_pan_hs(btcoexist);
break;
case BT_8821A_1ANT_COEX_ALGO_PANEDR_A2DP:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action algorithm = PAN+A2DP\n");
- halbtc8821a1ant_action_pan_edr_a2dp(btcoexist);
+ btc8821a1ant_action_pan_edr_a2dp(btcoexist);
break;
case BT_8821A_1ANT_COEX_ALGO_PANEDR_HID:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action algorithm = PAN(EDR)+HID\n");
- halbtc8821a1ant_action_pan_edr_hid(btcoexist);
+ btc8821a1ant_action_pan_edr_hid(btcoexist);
break;
case BT_8821A_1ANT_COEX_ALGO_HID_A2DP_PANEDR:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -2049,28 +1900,30 @@
case BT_8821A_1ANT_COEX_ALGO_HID_A2DP:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action algorithm = HID+A2DP\n");
- halbtc8821a1ant_action_hid_a2dp(btcoexist);
+ btc8821a1ant_action_hid_a2dp(btcoexist);
break;
default:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action algorithm = coexist All Off!!\n");
- /*halbtc8821a1ant_coex_all_off(btcoexist);*/
+ /*btc8821a1ant_coex_all_off(btcoexist);*/
break;
}
coex_dm->pre_algorithm = coex_dm->cur_algorithm;
}
}
-static void halbtc8821a1ant_run_coexist_mechanism(struct btc_coexist *btcoexist)
+static void btc8821a1ant_run_coexist_mechanism(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
- bool wifi_connected = false, bt_hs_on = false;
- bool increase_scan_dev_num = false;
- bool bt_ctrl_agg_buf_size = false;
- u8 agg_buf_size = 5;
- u8 wifi_rssi_state = BTC_RSSI_STATE_HIGH;
- bool wifi_under_5g = false;
+ bool wifi_connected = false, bt_hs_on = false;
+ bool increase_scan_dev_num = false;
+ bool bt_ctrl_agg_buf_size = false;
+ u8 agg_buf_size = 5;
+ u8 wifi_rssi_state = BTC_RSSI_STATE_HIGH;
+ u32 wifi_link_status = 0;
+ u32 num_of_wifi_link = 0;
+ bool wifi_under_5g = false;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], RunCoexistMechanism()===>\n");
@@ -2097,7 +1950,7 @@
if (wifi_under_5g) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], RunCoexistMechanism(), return for 5G <===\n");
- halbtc8821a1ant_coex_under_5g(btcoexist);
+ btc8821a1ant_coex_under_5g(btcoexist);
return;
}
@@ -2109,21 +1962,41 @@
btcoexist->btc_set(btcoexist, BTC_SET_BL_INC_SCAN_DEV_NUM,
&increase_scan_dev_num);
- btcoexist->btc_get(btcoexist,
- BTC_GET_BL_WIFI_CONNECTED, &wifi_connected);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
+ &wifi_connected);
+
+ btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_LINK_STATUS,
+ &wifi_link_status);
+ num_of_wifi_link = wifi_link_status >> 16;
+ if ((num_of_wifi_link >= 2) ||
+ (wifi_link_status & WIFI_P2P_GO_CONNECTED)) {
+ btc8821a1ant_limited_tx(btcoexist, NORMAL_EXEC, 0, 0, 0, 0);
+ btc8821a1ant_limited_rx(btcoexist, NORMAL_EXEC, false,
+ bt_ctrl_agg_buf_size, agg_buf_size);
+ btc8821a1ant_action_wifi_multi_port(btcoexist);
+ return;
+ }
if (!bt_link_info->sco_exist && !bt_link_info->hid_exist) {
- halbtc8821a1ant_limited_tx(btcoexist, NORMAL_EXEC, 0, 0, 0, 0);
+ btc8821a1ant_limited_tx(btcoexist, NORMAL_EXEC, 0, 0, 0, 0);
} else {
if (wifi_connected) {
wifi_rssi_state =
- halbtc8821a1ant_WifiRssiState(btcoexist, 1, 2,
- 30, 0);
- halbtc8821a1ant_limited_tx(btcoexist,
- NORMAL_EXEC, 1, 1, 1, 1);
+ btc8821a1ant_wifi_rssi_state(btcoexist, 1, 2,
+ 30, 0);
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8821a1ant_limited_tx(btcoexist,
+ NORMAL_EXEC, 1, 1,
+ 1, 1);
+ } else {
+ btc8821a1ant_limited_tx(btcoexist,
+ NORMAL_EXEC, 1, 1,
+ 1, 1);
+ }
} else {
- halbtc8821a1ant_limited_tx(btcoexist, NORMAL_EXEC,
- 0, 0, 0, 0);
+ btc8821a1ant_limited_tx(btcoexist, NORMAL_EXEC,
+ 0, 0, 0, 0);
}
}
@@ -2137,22 +2010,22 @@
bt_ctrl_agg_buf_size = true;
agg_buf_size = 0x8;
}
- halbtc8821a1ant_limited_rx(btcoexist, NORMAL_EXEC, false,
- bt_ctrl_agg_buf_size, agg_buf_size);
+ btc8821a1ant_limited_rx(btcoexist, NORMAL_EXEC, false,
+ bt_ctrl_agg_buf_size, agg_buf_size);
btc8821a1ant_run_sw_coex_mech(btcoexist);
btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
if (coex_sta->c2h_bt_inquiry_page) {
- halbtc8821a1ant_action_bt_inquiry(btcoexist);
+ btc8821a1ant_action_bt_inquiry(btcoexist);
return;
} else if (bt_hs_on) {
- halbtc8821a1ant_action_hs(btcoexist);
+ btc8821a1ant_action_hs(btcoexist);
return;
}
if (!wifi_connected) {
- bool scan = false, link = false, roam = false;
+ bool scan = false, link = false, roam = false;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], wifi is non connected-idle !!!\n");
@@ -2161,48 +2034,57 @@
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_LINK, &link);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_ROAM, &roam);
- if (scan || link || roam)
- btc8821a1ant_act_wifi_not_conn_scan(btcoexist);
- else
- halbtc8821a1ant_action_wifi_not_connected(btcoexist);
+ if (scan || link || roam) {
+ if (scan)
+ btc8821a1ant_act_wifi_not_conn_scan(btcoexist);
+ else
+ btc8821a1ant_action_wifi_not_connected_asso_auth(
+ btcoexist);
+ } else {
+ btc8821a1ant_action_wifi_not_connected(btcoexist);
+ }
} else {
- /* wifi LPS/Busy*/
- halbtc8821a1ant_action_wifi_connected(btcoexist);
+ /* wifi LPS/Busy */
+ btc8821a1ant_action_wifi_connected(btcoexist);
}
}
-static void halbtc8821a1ant_init_coex_dm(struct btc_coexist *btcoexist)
+static void btc8821a1ant_init_coex_dm(struct btc_coexist *btcoexist)
{
- /* force to reset coex mechanism*/
- /* sw all off*/
- halbtc8821a1ant_sw_mechanism(btcoexist, false);
+ /* force to reset coex mechanism
+ * sw all off
+ */
+ btc8821a1ant_sw_mechanism(btcoexist, false);
- halbtc8821a1ant_ps_tdma(btcoexist, FORCE_EXEC, false, 8);
- halbtc8821a1ant_coex_table_with_type(btcoexist, FORCE_EXEC, 0);
+ btc8821a1ant_ps_tdma(btcoexist, FORCE_EXEC, false, 8);
+ btc8821a1ant_coex_table_with_type(btcoexist, FORCE_EXEC, 0);
}
-static void halbtc8821a1ant_init_hw_config(struct btc_coexist *btcoexist,
- bool back_up)
+static void btc8821a1ant_init_hw_config(struct btc_coexist *btcoexist,
+ bool back_up, bool wifi_only)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- u8 u1_tmp = 0;
- bool wifi_under_5g = false;
+ u8 u1_tmp = 0;
+ bool wifi_under_5g = false;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], 1Ant Init HW Config!!\n");
+ if (wifi_only)
+ return;
+
if (back_up) {
coex_dm->backup_arfr_cnt1 = btcoexist->btc_read_4byte(btcoexist,
0x430);
coex_dm->backup_arfr_cnt2 = btcoexist->btc_read_4byte(btcoexist,
0x434);
coex_dm->backup_retry_limit =
- btcoexist->btc_read_2byte(btcoexist, 0x42a);
+ btcoexist->btc_read_2byte(btcoexist, 0x42a);
coex_dm->backup_ampdu_max_time =
- btcoexist->btc_read_1byte(btcoexist, 0x456);
+ btcoexist->btc_read_1byte(btcoexist, 0x456);
}
- /* 0x790[5:0] = 0x5*/
+ /* 0x790[5:0] = 0x5 */
u1_tmp = btcoexist->btc_read_1byte(btcoexist, 0x790);
u1_tmp &= 0xc0;
u1_tmp |= 0x5;
@@ -2210,35 +2092,33 @@
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_UNDER_5G, &wifi_under_5g);
- /*Antenna config*/
+ /* Antenna config */
if (wifi_under_5g)
- halbtc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_BT,
- true, false);
+ btc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_BT,
+ true, false);
else
- halbtc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_PTA,
- true, false);
- /* PTA parameter*/
- halbtc8821a1ant_coex_table_with_type(btcoexist, FORCE_EXEC, 0);
+ btc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_PTA,
+ true, false);
+ /* PTA parameter */
+ btc8821a1ant_coex_table_with_type(btcoexist, FORCE_EXEC, 0);
- /* Enable counter statistics*/
- /*0x76e[3] =1, WLAN_Act control by PTA*/
+ /* Enable counter statistics
+ * 0x76e[3] =1, WLAN_Act control by PTA
+ */
btcoexist->btc_write_1byte(btcoexist, 0x76e, 0xc);
btcoexist->btc_write_1byte(btcoexist, 0x778, 0x3);
btcoexist->btc_write_1byte_bitmask(btcoexist, 0x40, 0x20, 0x1);
}
-/*============================================================*/
-/* work around function start with wa_halbtc8821a1ant_*/
-/*============================================================*/
-/*============================================================*/
-/* extern function start with EXhalbtc8821a1ant_*/
-/*============================================================*/
-void ex_halbtc8821a1ant_init_hwconfig(struct btc_coexist *btcoexist)
+/**************************************************************
+ * extern function start with ex_btc8821a1ant_
+ **************************************************************/
+void ex_btc8821a1ant_init_hwconfig(struct btc_coexist *btcoexist, bool wifionly)
{
- halbtc8821a1ant_init_hw_config(btcoexist, true);
+ btc8821a1ant_init_hw_config(btcoexist, true, wifionly);
}
-void ex_halbtc8821a1ant_init_coex_dm(struct btc_coexist *btcoexist)
+void ex_btc8821a1ant_init_coex_dm(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -2247,12 +2127,12 @@
btcoexist->stop_coex_dm = false;
- halbtc8821a1ant_init_coex_dm(btcoexist);
+ btc8821a1ant_init_coex_dm(btcoexist);
- halbtc8821a1ant_query_bt_info(btcoexist);
+ btc8821a1ant_query_bt_info(btcoexist);
}
-void ex_halbtc8821a1ant_display_coex_info(struct btc_coexist *btcoexist)
+void ex_btc8821a1ant_display_coex_info(struct btc_coexist *btcoexist)
{
struct btc_board_info *board_info = &btcoexist->board_info;
struct btc_stack_info *stack_info = &btcoexist->stack_info;
@@ -2359,7 +2239,7 @@
"uplink" : "downlink")));
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG,
"\r\n %-35s = [%s/ %d/ %d] ", "BT [status/ rssi/ retryCnt]",
- ((btcoexist->bt_info.bt_disabled) ? ("disabled") :
+ ((coex_sta->bt_disabled) ? ("disabled") :
((coex_sta->c2h_bt_inquiry_page) ? ("inquiry/page scan") :
((BT_8821A_1ANT_BT_STATUS_NON_CONNECTED_IDLE ==
coex_dm->bt_status) ?
@@ -2397,7 +2277,7 @@
"\r\n %-35s = %s/%s, (0x%x/0x%x)",
"PS state, IPS/LPS, (lps/rpwm)",
((coex_sta->under_ips ? "IPS ON" : "IPS OFF")),
- ((coex_sta->under_Lps ? "LPS ON" : "LPS OFF")),
+ ((coex_sta->under_lps ? "LPS ON" : "LPS OFF")),
btcoexist->bt_info.lps_val,
btcoexist->bt_info.rpwm_val);
btcoexist->btc_disp_dbg_msg(btcoexist, BTC_DBG_DISP_FW_PWR_MODE_CMD);
@@ -2422,7 +2302,7 @@
"\r\n %-35s = 0x%x ", "Rate Mask",
btcoexist->bt_info.ra_mask);
- /* Fw mechanism*/
+ /* Fw mechanism */
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG, "\r\n %-35s",
"============[Fw mechanism]============");
@@ -2444,7 +2324,7 @@
coex_dm->cur_ignore_wlan_act);
}
- /* Hw setting*/
+ /* Hw setting */
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG,
"\r\n %-35s", "============[Hw setting]============");
@@ -2527,38 +2407,46 @@
"\r\n %-35s = %d/ %d", "0x774(low-pri rx/tx)",
coex_sta->low_priority_rx, coex_sta->low_priority_tx);
#if (BT_AUTO_REPORT_ONLY_8821A_1ANT == 1)
- halbtc8821a1ant_monitor_bt_ctr(btcoexist);
+ btc8821a1ant_monitor_bt_ctr(btcoexist);
#endif
btcoexist->btc_disp_dbg_msg(btcoexist, BTC_DBG_DISP_COEX_STATISTICS);
}
-void ex_halbtc8821a1ant_ips_notify(struct btc_coexist *btcoexist, u8 type)
+void ex_btc8821a1ant_ips_notify(struct btc_coexist *btcoexist, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
+ bool wifi_under_5g = false;
if (btcoexist->manual_control || btcoexist->stop_coex_dm)
return;
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_UNDER_5G, &wifi_under_5g);
+ if (wifi_under_5g) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], RunCoexistMechanism(), return for 5G <===\n");
+ btc8821a1ant_coex_under_5g(btcoexist);
+ return;
+ }
if (BTC_IPS_ENTER == type) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], IPS ENTER notify\n");
coex_sta->under_ips = true;
- halbtc8821a1ant_set_ant_path(btcoexist,
- BTC_ANT_PATH_BT, false, true);
- /*set PTA control*/
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 8);
- halbtc8821a1ant_coex_table_with_type(btcoexist,
- NORMAL_EXEC, 0);
+ btc8821a1ant_set_ant_path(btcoexist,
+ BTC_ANT_PATH_BT, false, true);
+ /* set PTA control */
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 8);
+ btc8821a1ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 0);
} else if (BTC_IPS_LEAVE == type) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], IPS LEAVE notify\n");
coex_sta->under_ips = false;
- halbtc8821a1ant_run_coexist_mechanism(btcoexist);
+ btc8821a1ant_run_coexist_mechanism(btcoexist);
}
}
-void ex_halbtc8821a1ant_lps_notify(struct btc_coexist *btcoexist, u8 type)
+void ex_btc8821a1ant_lps_notify(struct btc_coexist *btcoexist, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -2568,22 +2456,35 @@
if (BTC_LPS_ENABLE == type) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], LPS ENABLE notify\n");
- coex_sta->under_Lps = true;
+ coex_sta->under_lps = true;
} else if (BTC_LPS_DISABLE == type) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], LPS DISABLE notify\n");
- coex_sta->under_Lps = false;
+ coex_sta->under_lps = false;
}
}
-void ex_halbtc8821a1ant_scan_notify(struct btc_coexist *btcoexist, u8 type)
+void ex_btc8821a1ant_scan_notify(struct btc_coexist *btcoexist, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
bool wifi_connected = false, bt_hs_on = false;
+ bool bt_ctrl_agg_buf_size = false;
+ bool wifi_under_5g = false;
+ u32 wifi_link_status = 0;
+ u32 num_of_wifi_link = 0;
+ u8 agg_buf_size = 5;
- if (btcoexist->manual_control ||
- btcoexist->stop_coex_dm ||
- btcoexist->bt_info.bt_disabled)
+ if (btcoexist->manual_control || btcoexist->stop_coex_dm)
+ return;
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_UNDER_5G, &wifi_under_5g);
+ if (wifi_under_5g) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], RunCoexistMechanism(), return for 5G <===\n");
+ btc8821a1ant_coex_under_5g(btcoexist);
+ return;
+ }
+
+ if (coex_sta->bt_disabled)
return;
btcoexist->btc_get(btcoexist,
@@ -2591,13 +2492,24 @@
btcoexist->btc_get(btcoexist,
BTC_GET_BL_WIFI_CONNECTED, &wifi_connected);
- halbtc8821a1ant_query_bt_info(btcoexist);
+ btc8821a1ant_query_bt_info(btcoexist);
+
+ btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_LINK_STATUS,
+ &wifi_link_status);
+ num_of_wifi_link = wifi_link_status >> 16;
+ if (num_of_wifi_link >= 2) {
+ btc8821a1ant_limited_tx(btcoexist, NORMAL_EXEC, 0, 0, 0, 0);
+ btc8821a1ant_limited_rx(btcoexist, NORMAL_EXEC, false,
+ bt_ctrl_agg_buf_size, agg_buf_size);
+ btc8821a1ant_action_wifi_multi_port(btcoexist);
+ return;
+ }
if (coex_sta->c2h_bt_inquiry_page) {
- halbtc8821a1ant_action_bt_inquiry(btcoexist);
+ btc8821a1ant_action_bt_inquiry(btcoexist);
return;
} else if (bt_hs_on) {
- halbtc8821a1ant_action_hs(btcoexist);
+ btc8821a1ant_action_hs(btcoexist);
return;
}
@@ -2605,40 +2517,62 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], SCAN START notify\n");
if (!wifi_connected) {
- /* non-connected scan*/
+ /* non-connected scan */
btc8821a1ant_act_wifi_not_conn_scan(btcoexist);
} else {
- /* wifi is connected*/
- halbtc8821a1ant_action_wifi_connected_scan(btcoexist);
+ /* wifi is connected */
+ btc8821a1ant_action_wifi_connected_scan(btcoexist);
}
} else if (BTC_SCAN_FINISH == type) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], SCAN FINISH notify\n");
if (!wifi_connected) {
- /* non-connected scan*/
- halbtc8821a1ant_action_wifi_not_connected(btcoexist);
+ /* non-connected scan */
+ btc8821a1ant_action_wifi_not_connected(btcoexist);
} else {
- halbtc8821a1ant_action_wifi_connected(btcoexist);
+ btc8821a1ant_action_wifi_connected(btcoexist);
}
}
}
-void ex_halbtc8821a1ant_connect_notify(struct btc_coexist *btcoexist, u8 type)
+void ex_btc8821a1ant_connect_notify(struct btc_coexist *btcoexist, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
bool wifi_connected = false, bt_hs_on = false;
+ u32 wifi_link_status = 0;
+ u32 num_of_wifi_link = 0;
+ bool bt_ctrl_agg_buf_size = false;
+ bool wifi_under_5g = false;
+ u8 agg_buf_size = 5;
- if (btcoexist->manual_control ||
- btcoexist->stop_coex_dm ||
- btcoexist->bt_info.bt_disabled)
+ if (btcoexist->manual_control || btcoexist->stop_coex_dm ||
+ coex_sta->bt_disabled)
return;
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_UNDER_5G, &wifi_under_5g);
+ if (wifi_under_5g) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], RunCoexistMechanism(), return for 5G <===\n");
+ btc8821a1ant_coex_under_5g(btcoexist);
+ return;
+ }
+
+ btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_LINK_STATUS,
+ &wifi_link_status);
+ num_of_wifi_link = wifi_link_status >> 16;
+ if (num_of_wifi_link >= 2) {
+ btc8821a1ant_limited_tx(btcoexist, NORMAL_EXEC, 0, 0, 0, 0);
+ btc8821a1ant_limited_rx(btcoexist, NORMAL_EXEC, false,
+ bt_ctrl_agg_buf_size, agg_buf_size);
+ btc8821a1ant_action_wifi_multi_port(btcoexist);
+ return;
+ }
btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
if (coex_sta->c2h_bt_inquiry_page) {
- halbtc8821a1ant_action_bt_inquiry(btcoexist);
+ btc8821a1ant_action_bt_inquiry(btcoexist);
return;
} else if (bt_hs_on) {
- halbtc8821a1ant_action_hs(btcoexist);
+ btc8821a1ant_action_hs(btcoexist);
return;
}
@@ -2653,26 +2587,33 @@
btcoexist->btc_get(btcoexist,
BTC_GET_BL_WIFI_CONNECTED, &wifi_connected);
if (!wifi_connected) {
- /* non-connected scan*/
- halbtc8821a1ant_action_wifi_not_connected(btcoexist);
+ /* non-connected scan */
+ btc8821a1ant_action_wifi_not_connected(btcoexist);
} else {
- halbtc8821a1ant_action_wifi_connected(btcoexist);
+ btc8821a1ant_action_wifi_connected(btcoexist);
}
}
}
-void ex_halbtc8821a1ant_media_status_notify(struct btc_coexist *btcoexist,
- u8 type)
+void ex_btc8821a1ant_media_status_notify(struct btc_coexist *btcoexist,
+ u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[3] = {0};
u32 wifi_bw;
u8 wifi_central_chnl;
+ bool wifi_under_5g = false;
- if (btcoexist->manual_control ||
- btcoexist->stop_coex_dm ||
- btcoexist->bt_info.bt_disabled)
+ if (btcoexist->manual_control || btcoexist->stop_coex_dm ||
+ coex_sta->bt_disabled)
return;
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_UNDER_5G, &wifi_under_5g);
+ if (wifi_under_5g) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], RunCoexistMechanism(), return for 5G <===\n");
+ btc8821a1ant_coex_under_5g(btcoexist);
+ return;
+ }
if (BTC_MEDIA_CONNECT == type) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -2682,17 +2623,16 @@
"[BTCoex], MEDIA disconnect notify\n");
}
- /* only 2.4G we need to inform bt the chnl mask*/
+ /* only 2.4G we need to inform bt the chnl mask */
btcoexist->btc_get(btcoexist,
BTC_GET_U1_WIFI_CENTRAL_CHNL,
&wifi_central_chnl);
- if ((BTC_MEDIA_CONNECT == type) &&
+ if ((type == BTC_MEDIA_CONNECT) &&
(wifi_central_chnl <= 14)) {
- /*h2c_parameter[0] = 0x1;*/
h2c_parameter[0] = 0x0;
h2c_parameter[1] = wifi_central_chnl;
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- if (BTC_WIFI_BW_HT40 == wifi_bw)
+ if (wifi_bw == BTC_WIFI_BW_HT40)
h2c_parameter[2] = 0x30;
else
h2c_parameter[2] = 0x20;
@@ -2711,25 +2651,48 @@
btcoexist->btc_fill_h2c(btcoexist, 0x66, 3, h2c_parameter);
}
-void ex_halbtc8821a1ant_special_packet_notify(struct btc_coexist *btcoexist,
- u8 type)
+void ex_btc8821a1ant_special_packet_notify(struct btc_coexist *btcoexist,
+ u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
bool bt_hs_on = false;
+ bool bt_ctrl_agg_buf_size = false;
+ bool wifi_under_5g = false;
+ u32 wifi_link_status = 0;
+ u32 num_of_wifi_link = 0;
+ u8 agg_buf_size = 5;
- if (btcoexist->manual_control ||
- btcoexist->stop_coex_dm ||
- btcoexist->bt_info.bt_disabled)
+ if (btcoexist->manual_control || btcoexist->stop_coex_dm ||
+ coex_sta->bt_disabled)
return;
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_UNDER_5G, &wifi_under_5g);
+ if (wifi_under_5g) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], RunCoexistMechanism(), return for 5G <===\n");
+ btc8821a1ant_coex_under_5g(btcoexist);
+ return;
+ }
+
coex_sta->special_pkt_period_cnt = 0;
+ btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_LINK_STATUS,
+ &wifi_link_status);
+ num_of_wifi_link = wifi_link_status >> 16;
+ if (num_of_wifi_link >= 2) {
+ btc8821a1ant_limited_tx(btcoexist, NORMAL_EXEC, 0, 0, 0, 0);
+ btc8821a1ant_limited_rx(btcoexist, NORMAL_EXEC, false,
+ bt_ctrl_agg_buf_size, agg_buf_size);
+ btc8821a1ant_action_wifi_multi_port(btcoexist);
+ return;
+ }
+
btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
if (coex_sta->c2h_bt_inquiry_page) {
- halbtc8821a1ant_action_bt_inquiry(btcoexist);
+ btc8821a1ant_action_bt_inquiry(btcoexist);
return;
} else if (bt_hs_on) {
- halbtc8821a1ant_action_hs(btcoexist);
+ btc8821a1ant_action_hs(btcoexist);
return;
}
@@ -2741,12 +2704,13 @@
}
}
-void ex_halbtc8821a1ant_bt_info_notify(struct btc_coexist *btcoexist,
- u8 *tmp_buf, u8 length)
+void ex_btc8821a1ant_bt_info_notify(struct btc_coexist *btcoexist,
+ u8 *tmp_buf, u8 length)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
+ u8 i;
u8 bt_info = 0;
- u8 i, rsp_source = 0;
+ u8 rsp_source = 0;
bool wifi_connected = false;
bool bt_busy = false;
bool wifi_under_5g = false;
@@ -2756,7 +2720,7 @@
btcoexist->btc_get(btcoexist,
BTC_GET_BL_WIFI_UNDER_5G, &wifi_under_5g);
- rsp_source = tmp_buf[0]&0xf;
+ rsp_source = tmp_buf[0] & 0xf;
if (rsp_source >= BT_INFO_SRC_8821A_1ANT_MAX)
rsp_source = BT_INFO_SRC_8821A_1ANT_WIFI_FW;
coex_sta->bt_info_c2h_cnt[rsp_source]++;
@@ -2768,7 +2732,7 @@
coex_sta->bt_info_c2h[rsp_source][i] = tmp_buf[i];
if (i == 1)
bt_info = tmp_buf[i];
- if (i == length-1) {
+ if (i == length - 1) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"0x%02x]\n", tmp_buf[i]);
} else {
@@ -2787,19 +2751,19 @@
coex_sta->bt_info_ext =
coex_sta->bt_info_c2h[rsp_source][4];
- /* Here we need to resend some wifi info to BT*/
- /* because bt is reset and loss of the info.*/
+ /* Here we need to resend some wifi info to BT
+ * because bt is reset and lost the info
+ */
if (coex_sta->bt_info_ext & BIT1) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], BT ext info bit1 check, send wifi BW&Chnl to BT!!\n");
- btcoexist->btc_get(btcoexist,
- BTC_GET_BL_WIFI_CONNECTED,
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
&wifi_connected);
if (wifi_connected) {
- ex_halbtc8821a1ant_media_status_notify(btcoexist,
+ ex_btc8821a1ant_media_status_notify(btcoexist,
BTC_MEDIA_CONNECT);
} else {
- ex_halbtc8821a1ant_media_status_notify(btcoexist,
+ ex_btc8821a1ant_media_status_notify(btcoexist,
BTC_MEDIA_DISCONNECT);
}
}
@@ -2809,36 +2773,28 @@
!btcoexist->stop_coex_dm) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], BT ext info bit3 check, set BT NOT to ignore Wlan active!!\n");
- halbtc8821a1ant_ignore_wlan_act(btcoexist,
- FORCE_EXEC,
- false);
+ btc8821a1ant_ignore_wlan_act(btcoexist,
+ FORCE_EXEC,
+ false);
}
}
-#if (BT_AUTO_REPORT_ONLY_8821A_1ANT == 0)
- if (!(coex_sta->bt_info_ext & BIT4)) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT ext info bit4 check, set BT to enable Auto Report!!\n");
- halbtc8821a1ant_bt_auto_report(btcoexist,
- FORCE_EXEC, true);
- }
-#endif
}
- /* check BIT2 first ==> check if bt is under inquiry or page scan*/
+ /* check BIT2 first ==> check if bt is under inquiry or page scan */
if (bt_info & BT_INFO_8821A_1ANT_B_INQ_PAGE)
coex_sta->c2h_bt_inquiry_page = true;
else
coex_sta->c2h_bt_inquiry_page = false;
- /* set link exist status*/
- if (!(bt_info&BT_INFO_8821A_1ANT_B_CONNECTION)) {
+ /* set link exist status */
+ if (!(bt_info & BT_INFO_8821A_1ANT_B_CONNECTION)) {
coex_sta->bt_link_exist = false;
coex_sta->pan_exist = false;
coex_sta->a2dp_exist = false;
coex_sta->hid_exist = false;
coex_sta->sco_exist = false;
} else {
- /* connection exists*/
+ /* connection exists */
coex_sta->bt_link_exist = true;
if (bt_info & BT_INFO_8821A_1ANT_B_FTP)
coex_sta->pan_exist = true;
@@ -2858,14 +2814,19 @@
coex_sta->sco_exist = false;
}
- halbtc8821a1ant_update_bt_link_info(btcoexist);
+ btc8821a1ant_update_bt_link_info(btcoexist);
- if (!(bt_info&BT_INFO_8821A_1ANT_B_CONNECTION)) {
+ /* mask profile bit for connect-ilde identification
+ * (for CSR case: A2DP idle --> 0x41)
+ */
+ bt_info = bt_info & 0x1f;
+
+ if (!(bt_info & BT_INFO_8821A_1ANT_B_CONNECTION)) {
coex_dm->bt_status = BT_8821A_1ANT_BT_STATUS_NON_CONNECTED_IDLE;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], BtInfoNotify(), BT Non-Connected idle!!!\n");
} else if (bt_info == BT_INFO_8821A_1ANT_B_CONNECTION) {
- /* connection exists but no busy*/
+ /* connection exists but no busy */
coex_dm->bt_status = BT_8821A_1ANT_BT_STATUS_CONNECTED_IDLE;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], BtInfoNotify(), BT Connected-idle!!!\n");
@@ -2895,33 +2856,48 @@
btcoexist->btc_set(btcoexist,
BTC_SET_BL_BT_TRAFFIC_BUSY, &bt_busy);
- halbtc8821a1ant_run_coexist_mechanism(btcoexist);
+ btc8821a1ant_run_coexist_mechanism(btcoexist);
}
-void ex_halbtc8821a1ant_halt_notify(struct btc_coexist *btcoexist)
+void ex_btc8821a1ant_halt_notify(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
+ bool wifi_under_5g = false;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Halt notify\n");
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_UNDER_5G, &wifi_under_5g);
+ if (wifi_under_5g) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], RunCoexistMechanism(), return for 5G <===\n");
+ btc8821a1ant_coex_under_5g(btcoexist);
+ return;
+ }
+
btcoexist->stop_coex_dm = true;
- halbtc8821a1ant_set_ant_path(btcoexist,
- BTC_ANT_PATH_BT, false, true);
- halbtc8821a1ant_ignore_wlan_act(btcoexist, FORCE_EXEC, true);
+ btc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_BT, false, true);
+ btc8821a1ant_ignore_wlan_act(btcoexist, FORCE_EXEC, true);
- halbtc8821a1ant_power_save_state(btcoexist,
- BTC_PS_WIFI_NATIVE, 0x0, 0x0);
- halbtc8821a1ant_ps_tdma(btcoexist, FORCE_EXEC, false, 0);
+ btc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ btc8821a1ant_ps_tdma(btcoexist, FORCE_EXEC, false, 0);
- ex_halbtc8821a1ant_media_status_notify(btcoexist,
- BTC_MEDIA_DISCONNECT);
+ ex_btc8821a1ant_media_status_notify(btcoexist, BTC_MEDIA_DISCONNECT);
}
-void ex_halbtc8821a1ant_pnp_notify(struct btc_coexist *btcoexist, u8 pnp_state)
+void ex_btc8821a1ant_pnp_notify(struct btc_coexist *btcoexist, u8 pnp_state)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
+ bool wifi_under_5g = false;
+
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_UNDER_5G, &wifi_under_5g);
+ if (wifi_under_5g) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], RunCoexistMechanism(), return for 5G <===\n");
+ btc8821a1ant_coex_under_5g(btcoexist);
+ return;
+ }
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Pnp notify\n");
@@ -2929,26 +2905,33 @@
if (BTC_WIFI_PNP_SLEEP == pnp_state) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Pnp notify to SLEEP\n");
+ /* BT should clear UnderIPS/UnderLPS state to avoid mismatch
+ * state after wakeup.
+ */
+ coex_sta->under_ips = false;
+ coex_sta->under_lps = false;
btcoexist->stop_coex_dm = true;
- halbtc8821a1ant_ignore_wlan_act(btcoexist, FORCE_EXEC, true);
- halbtc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
- 0x0, 0x0);
- halbtc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 9);
+ btc8821a1ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ btc8821a1ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 0);
+ btc8821a1ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
+ btc8821a1ant_set_ant_path(btcoexist, BTC_ANT_PATH_BT, false,
+ true);
} else if (BTC_WIFI_PNP_WAKE_UP == pnp_state) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Pnp notify to WAKE UP\n");
btcoexist->stop_coex_dm = false;
- halbtc8821a1ant_init_hw_config(btcoexist, false);
- halbtc8821a1ant_init_coex_dm(btcoexist);
- halbtc8821a1ant_query_bt_info(btcoexist);
+ btc8821a1ant_init_hw_config(btcoexist, false, false);
+ btc8821a1ant_init_coex_dm(btcoexist);
+ btc8821a1ant_query_bt_info(btcoexist);
}
}
-void ex_halbtc8821a1ant_periodical(struct btc_coexist *btcoexist)
+void ex_btc8821a1ant_periodical(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- static u8 dis_ver_info_cnt;
- u32 fw_ver = 0, bt_patch_ver = 0;
+ static u8 dis_ver_info_cnt;
+ u32 fw_ver = 0, bt_patch_ver = 0;
struct btc_board_info *board_info = &btcoexist->board_info;
struct btc_stack_info *stack_info = &btcoexist->stack_info;
@@ -2982,16 +2965,9 @@
}
#if (BT_AUTO_REPORT_ONLY_8821A_1ANT == 0)
- halbtc8821a1ant_query_bt_info(btcoexist);
- halbtc8821a1ant_monitor_bt_ctr(btcoexist);
- btc8821a1ant_mon_bt_en_dis(btcoexist);
+ btc8821a1ant_query_bt_info(btcoexist);
+ btc8821a1ant_monitor_bt_ctr(btcoexist);
#else
- if (halbtc8821a1ant_Is_wifi_status_changed(btcoexist) ||
- coex_dm->auto_tdma_adjust) {
- if (coex_sta->special_pkt_period_cnt > 2)
- halbtc8821a1ant_run_coexist_mechanism(btcoexist);
- }
-
coex_sta->special_pkt_period_cnt++;
#endif
}
diff --git a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a1ant.h b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a1ant.h
index 20e9048..1bd1ebe 100644
--- a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a1ant.h
+++ b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a1ant.h
@@ -140,13 +140,14 @@
};
struct coex_sta_8821a_1ant {
+ bool bt_disabled;
bool bt_link_exist;
bool sco_exist;
bool a2dp_exist;
bool hid_exist;
bool pan_exist;
- bool under_Lps;
+ bool under_lps;
bool under_ips;
u32 special_pkt_period_cnt;
u32 high_priority_tx;
@@ -160,6 +161,7 @@
u8 bt_info_c2h[BT_INFO_SRC_8821A_1ANT_MAX][10];
u32 bt_info_c2h_cnt[BT_INFO_SRC_8821A_1ANT_MAX];
bool c2h_bt_inquiry_page;
+ bool wifi_is_high_pri_task;
u8 bt_retry_cnt;
u8 bt_info_ext;
};
diff --git a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a2ant.c b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a2ant.c
index 1717e9c..841b4a8 100644
--- a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a2ant.c
+++ b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a2ant.c
@@ -23,7 +23,7 @@
*
*****************************************************************************/
-/*============================================================
+/************************************************************
* Description:
*
* This file is for RTL8821A Co-exist mechanism
@@ -32,22 +32,19 @@
* 2012/08/22 Cosa first check in.
* 2012/11/14 Cosa Revise for 8821A 2Ant out sourcing.
*
- *============================================================
- */
+ ************************************************************/
-/*============================================================
+/************************************************************
* include files
- *============================================================
-*/
+ ************************************************************/
#include "halbt_precomp.h"
-/*============================================================
+/************************************************************
* Global variables, these are static variables
- *============================================================
- */
-static struct coex_dm_8821a_2ant glcoex_dm_8821a_2ant;
-static struct coex_dm_8821a_2ant *coex_dm = &glcoex_dm_8821a_2ant;
-static struct coex_sta_8821a_2ant glcoex_sta_8821a_2ant;
-static struct coex_sta_8821a_2ant *coex_sta = &glcoex_sta_8821a_2ant;
+ ************************************************************/
+static struct coex_dm_8821a_2ant glcoex_dm_8821a_2ant;
+static struct coex_dm_8821a_2ant *coex_dm = &glcoex_dm_8821a_2ant;
+static struct coex_sta_8821a_2ant glcoex_sta_8821a_2ant;
+static struct coex_sta_8821a_2ant *coex_sta = &glcoex_sta_8821a_2ant;
static const char *const glbt_info_src_8821a_2ant[] = {
"BT Info[wifi fw]",
@@ -55,32 +52,29 @@
"BT Info[bt auto report]",
};
-static u32 glcoex_ver_date_8821a_2ant = 20130618;
-static u32 glcoex_ver_8821a_2ant = 0x5050;
+static u32 glcoex_ver_date_8821a_2ant = 20130618;
+static u32 glcoex_ver_8821a_2ant = 0x5050;
-/*============================================================
+/************************************************************
* local function proto type if needed
- *============================================================
- *============================================================
- * local function start with halbtc8821a2ant_
- *============================================================
- */
-static u8 halbtc8821a2ant_bt_rssi_state(struct btc_coexist *btcoexist,
- u8 level_num, u8 rssi_thresh,
- u8 rssi_thresh1)
+ *
+ * local function start with btc8821a2ant_
+ ************************************************************/
+static u8 btc8821a2ant_bt_rssi_state(struct btc_coexist *btcoexist,
+ u8 level_num, u8 rssi_thresh,
+ u8 rssi_thresh1)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- long bt_rssi = 0;
- u8 bt_rssi_state = coex_sta->pre_bt_rssi_state;
+ long bt_rssi = 0;
+ u8 bt_rssi_state = coex_sta->pre_bt_rssi_state;
bt_rssi = coex_sta->bt_rssi;
if (level_num == 2) {
if ((coex_sta->pre_bt_rssi_state == BTC_RSSI_STATE_LOW) ||
(coex_sta->pre_bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
- long tmp = rssi_thresh +
- BTC_RSSI_COEX_THRESH_TOL_8821A_2ANT;
- if (bt_rssi >= tmp) {
+ if (bt_rssi >=
+ rssi_thresh + BTC_RSSI_COEX_THRESH_TOL_8821A_2ANT) {
bt_rssi_state = BTC_RSSI_STATE_HIGH;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], BT Rssi state switch to High\n");
@@ -110,7 +104,8 @@
if ((coex_sta->pre_bt_rssi_state == BTC_RSSI_STATE_LOW) ||
(coex_sta->pre_bt_rssi_state == BTC_RSSI_STATE_STAY_LOW)) {
if (bt_rssi >=
- (rssi_thresh+BTC_RSSI_COEX_THRESH_TOL_8821A_2ANT)) {
+ (rssi_thresh +
+ BTC_RSSI_COEX_THRESH_TOL_8821A_2ANT)) {
bt_rssi_state = BTC_RSSI_STATE_MEDIUM;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], BT Rssi state switch to Medium\n");
@@ -156,13 +151,13 @@
return bt_rssi_state;
}
-static u8 halbtc8821a2ant_wifi_rssi_state(struct btc_coexist *btcoexist,
- u8 index, u8 level_num,
- u8 rssi_thresh, u8 rssi_thresh1)
+static u8 btc8821a2ant_wifi_rssi_state(struct btc_coexist *btcoexist,
+ u8 index, u8 level_num,
+ u8 rssi_thresh, u8 rssi_thresh1)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- long wifi_rssi = 0;
- u8 wifi_rssi_state = coex_sta->pre_wifi_rssi_state[index];
+ long wifi_rssi = 0;
+ u8 wifi_rssi_state = coex_sta->pre_wifi_rssi_state[index];
btcoexist->btc_get(btcoexist, BTC_GET_S4_WIFI_RSSI, &wifi_rssi);
@@ -204,7 +199,8 @@
(coex_sta->pre_wifi_rssi_state[index] ==
BTC_RSSI_STATE_STAY_LOW)) {
if (wifi_rssi >=
- (rssi_thresh+BTC_RSSI_COEX_THRESH_TOL_8821A_2ANT)) {
+ (rssi_thresh +
+ BTC_RSSI_COEX_THRESH_TOL_8821A_2ANT)) {
wifi_rssi_state = BTC_RSSI_STATE_MEDIUM;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], wifi RSSI state switch to Medium\n");
@@ -248,76 +244,57 @@
return wifi_rssi_state;
}
-static void btc8821a2ant_mon_bt_en_dis(struct btc_coexist *btcoexist)
+static
+void btc8821a2ant_limited_rx(struct btc_coexist *btcoexist, bool force_exec,
+ bool rej_ap_agg_pkt, bool bt_ctrl_agg_buf_size,
+ u8 agg_buf_size)
{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- static bool pre_bt_disabled;
- static u32 bt_disable_cnt;
- bool bt_active = true, bt_disabled = false;
+ bool reject_rx_agg = rej_ap_agg_pkt;
+ bool bt_ctrl_rx_agg_size = bt_ctrl_agg_buf_size;
+ u8 rx_agg_size = agg_buf_size;
- /* This function check if bt is disabled*/
-
- if (coex_sta->high_priority_tx == 0 &&
- coex_sta->high_priority_rx == 0 &&
- coex_sta->low_priority_tx == 0 &&
- coex_sta->low_priority_rx == 0)
- bt_active = false;
- if (coex_sta->high_priority_tx == 0xffff &&
- coex_sta->high_priority_rx == 0xffff &&
- coex_sta->low_priority_tx == 0xffff &&
- coex_sta->low_priority_rx == 0xffff)
- bt_active = false;
- if (bt_active) {
- bt_disable_cnt = 0;
- bt_disabled = false;
- btcoexist->btc_set(btcoexist, BTC_SET_BL_BT_DISABLE,
- &bt_disabled);
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT is enabled !!\n");
- } else {
- bt_disable_cnt++;
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], bt all counters = 0, %d times!!\n",
- bt_disable_cnt);
- if (bt_disable_cnt >= 2) {
- bt_disabled = true;
- btcoexist->btc_set(btcoexist, BTC_SET_BL_BT_DISABLE,
- &bt_disabled);
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT is disabled !!\n");
- }
- }
- if (pre_bt_disabled != bt_disabled) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT is from %s to %s!!\n",
- (pre_bt_disabled ? "disabled" : "enabled"),
- (bt_disabled ? "disabled" : "enabled"));
- pre_bt_disabled = bt_disabled;
- }
+ /* Rx Aggregation related setting */
+ btcoexist->btc_set(btcoexist, BTC_SET_BL_TO_REJ_AP_AGG_PKT,
+ &reject_rx_agg);
+ /* decide BT control aggregation buf size or not */
+ btcoexist->btc_set(btcoexist, BTC_SET_BL_BT_CTRL_AGG_SIZE,
+ &bt_ctrl_rx_agg_size);
+ /* aggregation buf size, works when BT control Rx aggregation size */
+ btcoexist->btc_set(btcoexist, BTC_SET_U1_AGG_BUF_SIZE, &rx_agg_size);
+ /* real update aggregation setting */
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_AGGREGATE_CTRL, NULL);
}
-static void halbtc8821a2ant_monitor_bt_ctr(struct btc_coexist *btcoexist)
+static void btc8821a2ant_monitor_bt_ctr(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- u32 reg_hp_txrx, reg_lp_txrx, u4tmp;
- u32 reg_hp_tx = 0, reg_hp_rx = 0, reg_lp_tx = 0, reg_lp_rx = 0;
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
+ u32 reg_hp_txrx, reg_lp_txrx, u4tmp;
+ u32 reg_hp_tx = 0, reg_hp_rx = 0, reg_lp_tx = 0, reg_lp_rx = 0;
reg_hp_txrx = 0x770;
reg_lp_txrx = 0x774;
u4tmp = btcoexist->btc_read_4byte(btcoexist, reg_hp_txrx);
reg_hp_tx = u4tmp & MASKLWORD;
- reg_hp_rx = (u4tmp & MASKHWORD)>>16;
+ reg_hp_rx = (u4tmp & MASKHWORD) >> 16;
u4tmp = btcoexist->btc_read_4byte(btcoexist, reg_lp_txrx);
reg_lp_tx = u4tmp & MASKLWORD;
- reg_lp_rx = (u4tmp & MASKHWORD)>>16;
+ reg_lp_rx = (u4tmp & MASKHWORD) >> 16;
coex_sta->high_priority_tx = reg_hp_tx;
coex_sta->high_priority_rx = reg_hp_rx;
coex_sta->low_priority_tx = reg_lp_tx;
coex_sta->low_priority_rx = reg_lp_rx;
+ if ((coex_sta->low_priority_rx >= 950) &&
+ (coex_sta->low_priority_rx >= coex_sta->low_priority_tx) &&
+ (!coex_sta->under_ips))
+ bt_link_info->slave_role = true;
+ else
+ bt_link_info->slave_role = false;
+
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], High Priority Tx/Rx (reg 0x%x) = 0x%x(%d)/0x%x(%d)\n",
reg_hp_txrx, reg_hp_tx, reg_hp_tx, reg_hp_rx, reg_hp_rx);
@@ -329,14 +306,51 @@
btcoexist->btc_write_1byte(btcoexist, 0x76e, 0xc);
}
-static void halbtc8821a2ant_query_bt_info(struct btc_coexist *btcoexist)
+static void btc8821a2ant_monitor_wifi_ctr(struct btc_coexist *btcoexist)
+{
+ if (coex_sta->under_ips) {
+ coex_sta->crc_ok_cck = 0;
+ coex_sta->crc_ok_11g = 0;
+ coex_sta->crc_ok_11n = 0;
+ coex_sta->crc_ok_11n_agg = 0;
+
+ coex_sta->crc_err_cck = 0;
+ coex_sta->crc_err_11g = 0;
+ coex_sta->crc_err_11n = 0;
+ coex_sta->crc_err_11n_agg = 0;
+ } else {
+ coex_sta->crc_ok_cck =
+ btcoexist->btc_read_4byte(btcoexist, 0xf88);
+ coex_sta->crc_ok_11g =
+ btcoexist->btc_read_2byte(btcoexist, 0xf94);
+ coex_sta->crc_ok_11n =
+ btcoexist->btc_read_2byte(btcoexist, 0xf90);
+ coex_sta->crc_ok_11n_agg =
+ btcoexist->btc_read_2byte(btcoexist, 0xfb8);
+
+ coex_sta->crc_err_cck =
+ btcoexist->btc_read_4byte(btcoexist, 0xf84);
+ coex_sta->crc_err_11g =
+ btcoexist->btc_read_2byte(btcoexist, 0xf96);
+ coex_sta->crc_err_11n =
+ btcoexist->btc_read_2byte(btcoexist, 0xf92);
+ coex_sta->crc_err_11n_agg =
+ btcoexist->btc_read_2byte(btcoexist, 0xfba);
+ }
+
+ /* reset counter */
+ btcoexist->btc_write_1byte_bitmask(btcoexist, 0xf16, 0x1, 0x1);
+ btcoexist->btc_write_1byte_bitmask(btcoexist, 0xf16, 0x1, 0x0);
+}
+
+static void btc8821a2ant_query_bt_info(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[1] = {0};
coex_sta->c2h_bt_info_req_sent = true;
- h2c_parameter[0] |= BIT0; /* trigger */
+ h2c_parameter[0] |= BIT0; /* trigger */
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Query Bt Info, FW write 0x61 = 0x%x\n",
@@ -345,54 +359,135 @@
btcoexist->btc_fill_h2c(btcoexist, 0x61, 1, h2c_parameter);
}
-static u8 halbtc8821a2ant_action_algorithm(struct btc_coexist *btcoexist)
+bool btc8821a2ant_is_wifi_status_changed(struct btc_coexist *btcoexist)
+{
+ static bool pre_wifi_busy = true;
+ static bool pre_under_4way = true;
+ static bool pre_bt_hs_on = true;
+ bool wifi_busy = false, under_4way = false, bt_hs_on = false;
+ bool wifi_connected = false;
+ u8 wifi_rssi_state = BTC_RSSI_STATE_HIGH;
+
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
+ &wifi_connected);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_BUSY, &wifi_busy);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_4_WAY_PROGRESS,
+ &under_4way);
+
+ if (wifi_connected) {
+ if (wifi_busy != pre_wifi_busy) {
+ pre_wifi_busy = wifi_busy;
+ return true;
+ }
+ if (under_4way != pre_under_4way) {
+ pre_under_4way = under_4way;
+ return true;
+ }
+ if (bt_hs_on != pre_bt_hs_on) {
+ pre_bt_hs_on = bt_hs_on;
+ return true;
+ }
+
+ wifi_rssi_state = btc8821a2ant_wifi_rssi_state(btcoexist, 3, 2,
+ BT_8821A_2ANT_WIFI_RSSI_COEXSWITCH_THRES, 0);
+
+ if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (wifi_rssi_state == BTC_RSSI_STATE_LOW))
+ return true;
+ }
+
+ return false;
+}
+
+static void btc8821a2ant_update_bt_link_info(struct btc_coexist *btcoexist)
+{
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
+ bool bt_hs_on = false;
+
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
+
+ bt_link_info->bt_link_exist = coex_sta->bt_link_exist;
+ bt_link_info->sco_exist = coex_sta->sco_exist;
+ bt_link_info->a2dp_exist = coex_sta->a2dp_exist;
+ bt_link_info->pan_exist = coex_sta->pan_exist;
+ bt_link_info->hid_exist = coex_sta->hid_exist;
+
+ /* work around for HS mode. */
+ if (bt_hs_on) {
+ bt_link_info->pan_exist = true;
+ bt_link_info->bt_link_exist = true;
+ }
+
+ /* check if Sco only */
+ if (bt_link_info->sco_exist && !bt_link_info->a2dp_exist &&
+ !bt_link_info->pan_exist && !bt_link_info->hid_exist)
+ bt_link_info->sco_only = true;
+ else
+ bt_link_info->sco_only = false;
+
+ /* check if A2dp only */
+ if (!bt_link_info->sco_exist && bt_link_info->a2dp_exist &&
+ !bt_link_info->pan_exist && !bt_link_info->hid_exist)
+ bt_link_info->a2dp_only = true;
+ else
+ bt_link_info->a2dp_only = false;
+
+ /* check if Pan only */
+ if (!bt_link_info->sco_exist && !bt_link_info->a2dp_exist &&
+ bt_link_info->pan_exist && !bt_link_info->hid_exist)
+ bt_link_info->pan_only = true;
+ else
+ bt_link_info->pan_only = false;
+
+ /* check if Hid only */
+ if (!bt_link_info->sco_exist && !bt_link_info->a2dp_exist &&
+ !bt_link_info->pan_exist && bt_link_info->hid_exist)
+ bt_link_info->hid_only = true;
+ else
+ bt_link_info->hid_only = false;
+}
+
+static u8 btc8821a2ant_action_algorithm(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- struct btc_stack_info *stack_info = &btcoexist->stack_info;
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
bool bt_hs_on = false;
u8 algorithm = BT_8821A_2ANT_COEX_ALGO_UNDEFINED;
u8 num_of_diff_profile = 0;
btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
- /*for win-8 stack HID report error*/
- /* sync BTInfo with BT firmware and stack */
- if (!stack_info->hid_exist)
- stack_info->hid_exist = coex_sta->hid_exist;
- /* when stack HID report error, here we use the info from bt fw. */
- if (!stack_info->bt_link_exist)
- stack_info->bt_link_exist = coex_sta->bt_link_exist;
-
- if (!coex_sta->bt_link_exist) {
+ if (!bt_link_info->bt_link_exist) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], No profile exists!!!\n");
+ "[BTCoex], No BT link exists!!!\n");
return algorithm;
}
- if (coex_sta->sco_exist)
+ if (bt_link_info->sco_exist)
num_of_diff_profile++;
- if (coex_sta->hid_exist)
+ if (bt_link_info->hid_exist)
num_of_diff_profile++;
- if (coex_sta->pan_exist)
+ if (bt_link_info->pan_exist)
num_of_diff_profile++;
- if (coex_sta->a2dp_exist)
+ if (bt_link_info->a2dp_exist)
num_of_diff_profile++;
if (num_of_diff_profile == 1) {
- if (coex_sta->sco_exist) {
+ if (bt_link_info->sco_exist) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], SCO only\n");
algorithm = BT_8821A_2ANT_COEX_ALGO_SCO;
} else {
- if (coex_sta->hid_exist) {
+ if (bt_link_info->hid_exist) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], HID only\n");
algorithm = BT_8821A_2ANT_COEX_ALGO_HID;
- } else if (coex_sta->a2dp_exist) {
+ } else if (bt_link_info->a2dp_exist) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], A2DP only\n");
algorithm = BT_8821A_2ANT_COEX_ALGO_A2DP;
- } else if (coex_sta->pan_exist) {
+ } else if (bt_link_info->pan_exist) {
if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
@@ -407,16 +502,16 @@
}
}
} else if (num_of_diff_profile == 2) {
- if (coex_sta->sco_exist) {
- if (coex_sta->hid_exist) {
+ if (bt_link_info->sco_exist) {
+ if (bt_link_info->hid_exist) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], SCO + HID\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_PANEDR_HID;
- } else if (coex_sta->a2dp_exist) {
+ algorithm = BT_8821A_2ANT_COEX_ALGO_SCO;
+ } else if (bt_link_info->a2dp_exist) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], SCO + A2DP ==> SCO\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_PANEDR_HID;
- } else if (coex_sta->pan_exist) {
+ algorithm = BT_8821A_2ANT_COEX_ALGO_SCO;
+ } else if (bt_link_info->pan_exist) {
if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
@@ -426,99 +521,104 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"[BTCoex], SCO + PAN(EDR)\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_PANEDR_HID;
+ algorithm = BT_8821A_2ANT_COEX_ALGO_SCO;
}
}
} else {
- if (coex_sta->hid_exist &&
- coex_sta->a2dp_exist) {
+ if (bt_link_info->hid_exist &&
+ bt_link_info->a2dp_exist) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], HID + A2DP\n");
algorithm = BT_8821A_2ANT_COEX_ALGO_HID_A2DP;
- } else if (coex_sta->hid_exist &&
- coex_sta->pan_exist) {
+ } else if (bt_link_info->hid_exist &&
+ bt_link_info->pan_exist) {
if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"[BTCoex], HID + PAN(HS)\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_HID;
+ algorithm = BT_8821A_2ANT_COEX_ALGO_HID;
} else {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"[BTCoex], HID + PAN(EDR)\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_PANEDR_HID;
+ algorithm =
+ BT_8821A_2ANT_COEX_ALGO_PANEDR_HID;
}
- } else if (coex_sta->pan_exist &&
- coex_sta->a2dp_exist) {
+ } else if (bt_link_info->pan_exist &&
+ bt_link_info->a2dp_exist) {
if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"[BTCoex], A2DP + PAN(HS)\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_A2DP_PANHS;
+ algorithm =
+ BT_8821A_2ANT_COEX_ALGO_A2DP_PANHS;
} else {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"[BTCoex], A2DP + PAN(EDR)\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_PANEDR_A2DP;
+ algorithm =
+ BT_8821A_2ANT_COEX_ALGO_PANEDR_A2DP;
}
}
}
} else if (num_of_diff_profile == 3) {
- if (coex_sta->sco_exist) {
- if (coex_sta->hid_exist &&
- coex_sta->a2dp_exist) {
+ if (bt_link_info->sco_exist) {
+ if (bt_link_info->hid_exist &&
+ bt_link_info->a2dp_exist) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], SCO + HID + A2DP ==> HID\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_PANEDR_HID;
- } else if (coex_sta->hid_exist &&
- coex_sta->pan_exist) {
+ algorithm = BT_8821A_2ANT_COEX_ALGO_SCO;
+ } else if (bt_link_info->hid_exist &&
+ bt_link_info->pan_exist) {
if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"[BTCoex], SCO + HID + PAN(HS)\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_PANEDR_HID;
+ algorithm = BT_8821A_2ANT_COEX_ALGO_SCO;
} else {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"[BTCoex], SCO + HID + PAN(EDR)\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_PANEDR_HID;
+ algorithm = BT_8821A_2ANT_COEX_ALGO_SCO;
}
- } else if (coex_sta->pan_exist &&
- coex_sta->a2dp_exist) {
+ } else if (bt_link_info->pan_exist &&
+ bt_link_info->a2dp_exist) {
if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"[BTCoex], SCO + A2DP + PAN(HS)\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_PANEDR_HID;
+ algorithm = BT_8821A_2ANT_COEX_ALGO_SCO;
} else {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"[BTCoex], SCO + A2DP + PAN(EDR) ==> HID\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_PANEDR_HID;
+ algorithm = BT_8821A_2ANT_COEX_ALGO_SCO;
}
}
} else {
- if (coex_sta->hid_exist &&
- coex_sta->pan_exist &&
- coex_sta->a2dp_exist) {
+ if (bt_link_info->hid_exist &&
+ bt_link_info->pan_exist &&
+ bt_link_info->a2dp_exist) {
if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"[BTCoex], HID + A2DP + PAN(HS)\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_HID_A2DP;
+ algorithm =
+ BT_8821A_2ANT_COEX_ALGO_HID_A2DP;
} else {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"[BTCoex], HID + A2DP + PAN(EDR)\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_HID_A2DP_PANEDR;
+ algorithm =
+ BT_8821A_2ANT_COEX_ALGO_HID_A2DP_PANEDR;
}
}
}
} else if (num_of_diff_profile >= 3) {
- if (coex_sta->sco_exist) {
- if (coex_sta->hid_exist &&
- coex_sta->pan_exist &&
- coex_sta->a2dp_exist) {
+ if (bt_link_info->sco_exist) {
+ if (bt_link_info->hid_exist &&
+ bt_link_info->pan_exist &&
+ bt_link_info->a2dp_exist) {
if (bt_hs_on) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
@@ -528,7 +628,7 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST,
DBG_LOUD,
"[BTCoex], SCO + HID + A2DP + PAN(EDR)==>PAN(EDR)+HID\n");
- algorithm = BT_8821A_2ANT_COEX_ALGO_PANEDR_HID;
+ algorithm = BT_8821A_2ANT_COEX_ALGO_SCO;
}
}
}
@@ -536,44 +636,7 @@
return algorithm;
}
-static bool halbtc8821a2ant_need_to_dec_bt_pwr(struct btc_coexist *btcoexist)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- bool ret = false;
- bool bt_hs_on = false, wifi_connected = false;
- long bt_hs_rssi = 0;
- u8 bt_rssi_state;
-
- if (!btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on))
- return false;
- if (!btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
- &wifi_connected))
- return false;
- if (!btcoexist->btc_get(btcoexist, BTC_GET_S4_HS_RSSI, &bt_hs_rssi))
- return false;
-
- bt_rssi_state = halbtc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
-
- if (wifi_connected) {
- if (bt_hs_on) {
- if (bt_hs_rssi > 37) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Need to decrease bt power for HS mode!!\n");
- ret = true;
- }
- } else {
- if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Need to decrease bt power for Wifi is connected!!\n");
- ret = true;
- }
- }
- }
- return ret;
-}
-
-static void btc8821a2ant_set_fw_dac_swing_lev(struct btc_coexist *btcoexist,
+static void btc8821a2ant_set_fw_dac_swing_lvl(struct btc_coexist *btcoexist,
u8 dac_swing_lvl)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -592,185 +655,47 @@
btcoexist->btc_fill_h2c(btcoexist, 0x64, 1, h2c_parameter);
}
-static void halbtc8821a2ant_set_fw_dec_bt_pwr(struct btc_coexist *btcoexist,
- bool dec_bt_pwr)
+static void btc8821a2ant_set_fw_dec_bt_pwr(struct btc_coexist *btcoexist,
+ u8 dec_bt_pwr_lvl)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[1] = {0};
- h2c_parameter[0] = 0;
-
- if (dec_bt_pwr)
- h2c_parameter[0] |= BIT1;
+ h2c_parameter[0] = dec_bt_pwr_lvl;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], decrease Bt Power : %s, FW write 0x62 = 0x%x\n",
- (dec_bt_pwr ? "Yes!!" : "No!!"), h2c_parameter[0]);
+ "[BTCoex], decrease Bt Power Level : %u, FW write 0x62 = 0x%x\n",
+ dec_bt_pwr_lvl, h2c_parameter[0]);
btcoexist->btc_fill_h2c(btcoexist, 0x62, 1, h2c_parameter);
}
-static void halbtc8821a2ant_dec_bt_pwr(struct btc_coexist *btcoexist,
- bool force_exec, bool dec_bt_pwr)
+static void btc8821a2ant_dec_bt_pwr(struct btc_coexist *btcoexist,
+ bool force_exec, u8 dec_bt_pwr_lvl)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], %s Dec BT power = %s\n",
- (force_exec ? "force to" : ""),
- ((dec_bt_pwr) ? "ON" : "OFF"));
- coex_dm->cur_dec_bt_pwr = dec_bt_pwr;
+ "[BTCoex], %s Dec BT power level = %u\n",
+ (force_exec ? "force to" : ""), dec_bt_pwr_lvl);
+ coex_dm->cur_dec_bt_pwr_lvl = dec_bt_pwr_lvl;
if (!force_exec) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], pre_dec_bt_pwr = %d, cur_dec_bt_pwr = %d\n",
- coex_dm->pre_dec_bt_pwr, coex_dm->cur_dec_bt_pwr);
+ "[BTCoex], pre_dec_bt_pwr_lvl = %d, cur_dec_bt_pwr_lvl = %d\n",
+ coex_dm->pre_dec_bt_pwr_lvl,
+ coex_dm->cur_dec_bt_pwr_lvl);
- if (coex_dm->pre_dec_bt_pwr == coex_dm->cur_dec_bt_pwr)
+ if (coex_dm->pre_dec_bt_pwr_lvl == coex_dm->cur_dec_bt_pwr_lvl)
return;
}
- halbtc8821a2ant_set_fw_dec_bt_pwr(btcoexist, coex_dm->cur_dec_bt_pwr);
+ btc8821a2ant_set_fw_dec_bt_pwr(btcoexist, coex_dm->cur_dec_bt_pwr_lvl);
- coex_dm->pre_dec_bt_pwr = coex_dm->cur_dec_bt_pwr;
+ coex_dm->pre_dec_bt_pwr_lvl = coex_dm->cur_dec_bt_pwr_lvl;
}
-static void btc8821a2ant_set_fw_bt_lna_constr(struct btc_coexist *btcoexist,
- bool bt_lna_cons_on)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- u8 h2c_parameter[2] = {0};
-
- h2c_parameter[0] = 0x3; /* opCode, 0x3 = BT_SET_LNA_CONSTRAIN */
-
- if (bt_lna_cons_on)
- h2c_parameter[1] |= BIT0;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], set BT LNA Constrain: %s, FW write 0x69 = 0x%x\n",
- bt_lna_cons_on ? "ON!!" : "OFF!!",
- h2c_parameter[0] << 8 | h2c_parameter[1]);
-
- btcoexist->btc_fill_h2c(btcoexist, 0x69, 2, h2c_parameter);
-}
-
-static void btc8821a2_set_bt_lna_const(struct btc_coexist *btcoexist,
- bool force_exec, bool bt_lna_cons_on)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], %s BT Constrain = %s\n",
- (force_exec ? "force" : ""),
- ((bt_lna_cons_on) ? "ON" : "OFF"));
- coex_dm->cur_bt_lna_constrain = bt_lna_cons_on;
-
- if (!force_exec) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], pre_bt_lna_constrain = %d,cur_bt_lna_constrain = %d\n",
- coex_dm->pre_bt_lna_constrain,
- coex_dm->cur_bt_lna_constrain);
-
- if (coex_dm->pre_bt_lna_constrain ==
- coex_dm->cur_bt_lna_constrain)
- return;
- }
- btc8821a2ant_set_fw_bt_lna_constr(btcoexist,
- coex_dm->cur_bt_lna_constrain);
-
- coex_dm->pre_bt_lna_constrain = coex_dm->cur_bt_lna_constrain;
-}
-
-static void halbtc8821a2ant_set_fw_bt_psd_mode(struct btc_coexist *btcoexist,
- u8 bt_psd_mode)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- u8 h2c_parameter[2] = {0};
-
- h2c_parameter[0] = 0x2; /* opCode, 0x2 = BT_SET_PSD_MODE */
-
- h2c_parameter[1] = bt_psd_mode;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], set BT PSD mode = 0x%x, FW write 0x69 = 0x%x\n",
- h2c_parameter[1],
- h2c_parameter[0] << 8 | h2c_parameter[1]);
-
- btcoexist->btc_fill_h2c(btcoexist, 0x69, 2, h2c_parameter);
-}
-
-static void halbtc8821a2ant_set_bt_psd_mode(struct btc_coexist *btcoexist,
- bool force_exec, u8 bt_psd_mode)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], %s BT PSD mode = 0x%x\n",
- (force_exec ? "force" : ""), bt_psd_mode);
- coex_dm->cur_bt_psd_mode = bt_psd_mode;
-
- if (!force_exec) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], pre_bt_psd_mode = 0x%x, cur_bt_psd_mode = 0x%x\n",
- coex_dm->pre_bt_psd_mode, coex_dm->cur_bt_psd_mode);
-
- if (coex_dm->pre_bt_psd_mode == coex_dm->cur_bt_psd_mode)
- return;
- }
- halbtc8821a2ant_set_fw_bt_psd_mode(btcoexist,
- coex_dm->cur_bt_psd_mode);
-
- coex_dm->pre_bt_psd_mode = coex_dm->cur_bt_psd_mode;
-}
-
-static void halbtc8821a2ant_set_bt_auto_report(struct btc_coexist *btcoexist,
- bool enable_auto_report)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- u8 h2c_parameter[1] = {0};
-
- h2c_parameter[0] = 0;
-
- if (enable_auto_report)
- h2c_parameter[0] |= BIT0;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BT FW auto report : %s, FW write 0x68 = 0x%x\n",
- (enable_auto_report ? "Enabled!!" : "Disabled!!"),
- h2c_parameter[0]);
-
- btcoexist->btc_fill_h2c(btcoexist, 0x68, 1, h2c_parameter);
-}
-
-static void halbtc8821a2ant_bt_auto_report(struct btc_coexist *btcoexist,
- bool force_exec,
- bool enable_auto_report)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], %s BT Auto report = %s\n",
- (force_exec ? "force to" : ""),
- ((enable_auto_report) ? "Enabled" : "Disabled"));
- coex_dm->cur_bt_auto_report = enable_auto_report;
-
- if (!force_exec) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], pre_bt_auto_report = %d, cur_bt_auto_report = %d\n",
- coex_dm->pre_bt_auto_report,
- coex_dm->cur_bt_auto_report);
-
- if (coex_dm->pre_bt_auto_report == coex_dm->cur_bt_auto_report)
- return;
- }
- halbtc8821a2ant_set_bt_auto_report(btcoexist,
- coex_dm->cur_bt_auto_report);
-
- coex_dm->pre_bt_auto_report = coex_dm->cur_bt_auto_report;
-}
-
-static void halbtc8821a2ant_fw_dac_swing_lvl(struct btc_coexist *btcoexist,
- bool force_exec,
- u8 fw_dac_swing_lvl)
+static void btc8821a2ant_fw_dac_swing_lvl(struct btc_coexist *btcoexist,
+ bool force_exec, u8 fw_dac_swing_lvl)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -790,66 +715,14 @@
return;
}
- btc8821a2ant_set_fw_dac_swing_lev(btcoexist,
+ btc8821a2ant_set_fw_dac_swing_lvl(btcoexist,
coex_dm->cur_fw_dac_swing_lvl);
coex_dm->pre_fw_dac_swing_lvl = coex_dm->cur_fw_dac_swing_lvl;
}
-static void btc8821a2ant_set_sw_rf_rx_lpf_corner(struct btc_coexist *btcoexist,
- bool rx_rf_shrink_on)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- if (rx_rf_shrink_on) {
- /* Shrink RF Rx LPF corner */
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Shrink RF Rx LPF corner!!\n");
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1e,
- 0xfffff, 0xffffc);
- } else {
- /* Resume RF Rx LPF corner
- * After initialized, we can use coex_dm->bt_rf0x1e_backup
- */
- if (btcoexist->initilized) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Resume RF Rx LPF corner!!\n");
- btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A,
- 0x1e, 0xfffff,
- coex_dm->bt_rf0x1e_backup);
- }
- }
-}
-
-static void halbtc8821a2ant_RfShrink(struct btc_coexist *btcoexist,
- bool force_exec, bool rx_rf_shrink_on)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], %s turn Rx RF Shrink = %s\n",
- (force_exec ? "force to" : ""),
- ((rx_rf_shrink_on) ? "ON" : "OFF"));
- coex_dm->cur_rf_rx_lpf_shrink = rx_rf_shrink_on;
-
- if (!force_exec) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], pre_rf_rx_lpf_shrink = %d, cur_rf_rx_lpf_shrink = %d\n",
- coex_dm->pre_rf_rx_lpf_shrink,
- coex_dm->cur_rf_rx_lpf_shrink);
-
- if (coex_dm->pre_rf_rx_lpf_shrink ==
- coex_dm->cur_rf_rx_lpf_shrink)
- return;
- }
- btc8821a2ant_set_sw_rf_rx_lpf_corner(btcoexist,
- coex_dm->cur_rf_rx_lpf_shrink);
-
- coex_dm->pre_rf_rx_lpf_shrink = coex_dm->cur_rf_rx_lpf_shrink;
-}
-
-static void btc8821a2ant_SetSwPenTxRateAdapt(struct btc_coexist *btcoexist,
- bool low_penalty_ra)
+static void btc8821a2ant_set_sw_penalty_tx_rate_adaptive(
+ struct btc_coexist *btcoexist, bool low_penalty_ra)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[6] = {0};
@@ -858,14 +731,14 @@
if (low_penalty_ra) {
h2c_parameter[1] |= BIT0;
- /*normal rate except MCS7/6/5, OFDM54/48/36 */
+ /* normal rate except MCS7/6/5, OFDM54/48/36 */
h2c_parameter[2] = 0x00;
- /*MCS7 or OFDM54 */
- h2c_parameter[3] = 0xf7;
- /*MCS6 or OFDM48 */
- h2c_parameter[4] = 0xf8;
- /*MCS5 or OFDM36 */
- h2c_parameter[5] = 0xf9;
+ /* MCS7 or OFDM54 */
+ h2c_parameter[3] = 0xf5;
+ /* MCS6 or OFDM48 */
+ h2c_parameter[4] = 0xa0;
+ /* MCS5 or OFDM36 */
+ h2c_parameter[5] = 0xa0;
}
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -875,12 +748,11 @@
btcoexist->btc_fill_h2c(btcoexist, 0x69, 6, h2c_parameter);
}
-static void halbtc8821a2ant_low_penalty_ra(struct btc_coexist *btcoexist,
- bool force_exec, bool low_penalty_ra)
+static void btc8821a2ant_low_penalty_ra(struct btc_coexist *btcoexist,
+ bool force_exec, bool low_penalty_ra)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- /*return;*/
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], %s turn LowPenaltyRA = %s\n",
(force_exec ? "force to" : ""),
@@ -891,19 +763,19 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], pre_low_penalty_ra = %d, cur_low_penalty_ra = %d\n",
coex_dm->pre_low_penalty_ra,
- coex_dm->cur_low_penalty_ra);
+ coex_dm->cur_low_penalty_ra);
if (coex_dm->pre_low_penalty_ra == coex_dm->cur_low_penalty_ra)
return;
}
- btc8821a2ant_SetSwPenTxRateAdapt(btcoexist,
+ btc8821a2ant_set_sw_penalty_tx_rate_adaptive(btcoexist,
coex_dm->cur_low_penalty_ra);
coex_dm->pre_low_penalty_ra = coex_dm->cur_low_penalty_ra;
}
-static void halbtc8821a2ant_set_dac_swing_reg(struct btc_coexist *btcoexist,
- u32 level)
+static void btc8821a2ant_set_dac_swing_reg(struct btc_coexist *btcoexist,
+ u32 level)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 val = (u8)level;
@@ -918,14 +790,14 @@
u32 sw_dac_swing_lvl)
{
if (sw_dac_swing_on)
- halbtc8821a2ant_set_dac_swing_reg(btcoexist, sw_dac_swing_lvl);
+ btc8821a2ant_set_dac_swing_reg(btcoexist, sw_dac_swing_lvl);
else
- halbtc8821a2ant_set_dac_swing_reg(btcoexist, 0x18);
+ btc8821a2ant_set_dac_swing_reg(btcoexist, 0x18);
}
-static void halbtc8821a2ant_dac_swing(struct btc_coexist *btcoexist,
- bool force_exec, bool dac_swing_on,
- u32 dac_swing_lvl)
+static void btc8821a2ant_dac_swing(struct btc_coexist *btcoexist,
+ bool force_exec, bool dac_swing_on,
+ u32 dac_swing_lvl)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -958,50 +830,9 @@
coex_dm->pre_dac_swing_lvl = coex_dm->cur_dac_swing_lvl;
}
-static void halbtc8821a2ant_set_adc_back_off(struct btc_coexist *btcoexist,
- bool adc_back_off)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- if (adc_back_off) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BB BackOff Level On!\n");
- btcoexist->btc_write_1byte_bitmask(btcoexist, 0x8db, 0x60, 0x3);
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], BB BackOff Level Off!\n");
- btcoexist->btc_write_1byte_bitmask(btcoexist, 0x8db, 0x60, 0x1);
- }
-}
-
-static void halbtc8821a2ant_adc_back_off(struct btc_coexist *btcoexist,
- bool force_exec, bool adc_back_off)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], %s turn AdcBackOff = %s\n",
- (force_exec ? "force to" : ""),
- ((adc_back_off) ? "ON" : "OFF"));
- coex_dm->cur_adc_back_off = adc_back_off;
-
- if (!force_exec) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], pre_adc_back_off = %d, cur_adc_back_off = %d\n",
- coex_dm->pre_adc_back_off,
- coex_dm->cur_adc_back_off);
-
- if (coex_dm->pre_adc_back_off == coex_dm->cur_adc_back_off)
- return;
- }
- halbtc8821a2ant_set_adc_back_off(btcoexist, coex_dm->cur_adc_back_off);
-
- coex_dm->pre_adc_back_off = coex_dm->cur_adc_back_off;
-}
-
-static void halbtc8821a2ant_set_coex_table(struct btc_coexist *btcoexist,
- u32 val0x6c0, u32 val0x6c4,
- u32 val0x6c8, u8 val0x6cc)
+static void btc8821a2ant_set_coex_table(struct btc_coexist *btcoexist,
+ u32 val0x6c0, u32 val0x6c4,
+ u32 val0x6c8, u8 val0x6cc)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -1022,9 +853,9 @@
btcoexist->btc_write_1byte(btcoexist, 0x6cc, val0x6cc);
}
-static void halbtc8821a2ant_coex_table(struct btc_coexist *btcoexist,
- bool force_exec, u32 val0x6c0,
- u32 val0x6c4, u32 val0x6c8, u8 val0x6cc)
+static void btc8821a2ant_coex_table(struct btc_coexist *btcoexist,
+ bool force_exec, u32 val0x6c0,
+ u32 val0x6c4, u32 val0x6c8, u8 val0x6cc)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -1057,8 +888,8 @@
(coex_dm->pre_val0x6cc == coex_dm->cur_val0x6cc))
return;
}
- halbtc8821a2ant_set_coex_table(btcoexist, val0x6c0, val0x6c4, val0x6c8,
- val0x6cc);
+ btc8821a2ant_set_coex_table(btcoexist, val0x6c0, val0x6c4, val0x6c8,
+ val0x6cc);
coex_dm->pre_val0x6c0 = coex_dm->cur_val0x6c0;
coex_dm->pre_val0x6c4 = coex_dm->cur_val0x6c4;
@@ -1066,14 +897,97 @@
coex_dm->pre_val0x6cc = coex_dm->cur_val0x6cc;
}
-static void halbtc8821a2ant_set_fw_ignore_wlan_act(struct btc_coexist *btcoex,
- bool enable)
+static void btc8821a2ant_coex_table_with_type(struct btc_coexist *btcoexist,
+ bool force_exec, u8 type)
+{
+ coex_sta->coex_table_type = type;
+
+ switch (type) {
+ case 0:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x55555555,
+ 0x55555555, 0xffffff, 0x3);
+ break;
+ case 1:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x55555555,
+ 0x5afa5afa, 0xffffff, 0x3);
+ break;
+ case 2:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x5ada5ada,
+ 0x5ada5ada, 0xffffff, 0x3);
+ break;
+ case 3:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0xaaaaaaaa,
+ 0xaaaaaaaa, 0xffffff, 0x3);
+ break;
+ case 4:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0xffffffff,
+ 0xffffffff, 0xffffff, 0x3);
+ break;
+ case 5:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x5fff5fff,
+ 0x5fff5fff, 0xffffff, 0x3);
+ break;
+ case 6:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x55ff55ff,
+ 0x5a5a5a5a, 0xffffff, 0x3);
+ break;
+ case 7:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0x5ada5ada, 0xffffff, 0x3);
+ break;
+ case 8:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0x5ada5ada, 0xffffff, 0x3);
+ break;
+ case 9:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0x5ada5ada, 0xffffff, 0x3);
+ break;
+ case 10:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0x5ada5ada, 0xffffff, 0x3);
+ break;
+ case 11:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0x5ada5ada, 0xffffff, 0x3);
+ break;
+ case 12:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0x5ada5ada, 0xffffff, 0x3);
+ break;
+ case 13:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x5fff5fff,
+ 0xaaaaaaaa, 0xffffff, 0x3);
+ break;
+ case 14:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x5fff5fff,
+ 0x5ada5ada, 0xffffff, 0x3);
+ break;
+ case 15:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x55dd55dd,
+ 0xaaaaaaaa, 0xffffff, 0x3);
+ break;
+ case 16:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0x5fdf5fdf,
+ 0x5fdb5fdb, 0xffffff, 0x3);
+ break;
+ case 17:
+ btc8821a2ant_coex_table(btcoexist, force_exec, 0xfafafafa,
+ 0xfafafafa, 0xffffff, 0x3);
+ break;
+ default:
+ break;
+ }
+}
+
+static void btc8821a2ant_set_fw_ignore_wlan_act(struct btc_coexist *btcoex,
+ bool enable)
{
struct rtl_priv *rtlpriv = btcoex->adapter;
u8 h2c_parameter[1] = {0};
if (enable)
- h2c_parameter[0] |= BIT0;/* function enable */
+ h2c_parameter[0] |= BIT0; /* function enable */
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], set FW for BT Ignore Wlan_Act, FW write 0x63 = 0x%x\n",
@@ -1082,8 +996,35 @@
btcoex->btc_fill_h2c(btcoex, 0x63, 1, h2c_parameter);
}
-static void halbtc8821a2ant_ignore_wlan_act(struct btc_coexist *btcoexist,
- bool force_exec, bool enable)
+static void btc8821a2ant_set_lps_rpwm(struct btc_coexist *btcoexist, u8 lps_val,
+ u8 rpwm_val)
+{
+ u8 lps = lps_val;
+ u8 rpwm = rpwm_val;
+
+ btcoexist->btc_set(btcoexist, BTC_SET_U1_LPS_VAL, &lps);
+ btcoexist->btc_set(btcoexist, BTC_SET_U1_RPWM_VAL, &rpwm);
+}
+
+static void btc8821a2ant_lps_rpwm(struct btc_coexist *btcoexist,
+ bool force_exec, u8 lps_val, u8 rpwm_val)
+{
+ coex_dm->cur_lps = lps_val;
+ coex_dm->cur_rpwm = rpwm_val;
+
+ if (!force_exec) {
+ if ((coex_dm->pre_lps == coex_dm->cur_lps) &&
+ (coex_dm->pre_rpwm == coex_dm->cur_rpwm))
+ return;
+ }
+ btc8821a2ant_set_lps_rpwm(btcoexist, lps_val, rpwm_val);
+
+ coex_dm->pre_lps = coex_dm->cur_lps;
+ coex_dm->pre_rpwm = coex_dm->cur_rpwm;
+}
+
+static void btc8821a2ant_ignore_wlan_act(struct btc_coexist *btcoexist,
+ bool force_exec, bool enable)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -1102,14 +1043,14 @@
coex_dm->cur_ignore_wlan_act)
return;
}
- halbtc8821a2ant_set_fw_ignore_wlan_act(btcoexist, enable);
+ btc8821a2ant_set_fw_ignore_wlan_act(btcoexist, enable);
coex_dm->pre_ignore_wlan_act = coex_dm->cur_ignore_wlan_act;
}
-static void halbtc8821a2ant_set_fw_pstdma(struct btc_coexist *btcoexist,
- u8 byte1, u8 byte2, u8 byte3,
- u8 byte4, u8 byte5)
+static void btc8821a2ant_set_fw_ps_tdma(struct btc_coexist *btcoexist,
+ u8 byte1, u8 byte2, u8 byte3,
+ u8 byte4, u8 byte5)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 h2c_parameter[5];
@@ -1137,45 +1078,24 @@
btcoexist->btc_fill_h2c(btcoexist, 0x60, 5, h2c_parameter);
}
-static void btc8821a2ant_sw_mech1(struct btc_coexist *btcoexist,
- bool shrink_rx_lpf,
- bool low_penalty_ra, bool limited_dig,
- bool bt_lna_constrain)
+static void btc8821a2ant_sw_mechanism1(struct btc_coexist *btcoexist,
+ bool shrink_rx_lpf, bool low_penalty_ra,
+ bool limited_dig, bool bt_lna_constrain)
{
- u32 wifi_bw;
-
- btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
-
- if (BTC_WIFI_BW_HT40 != wifi_bw) {
- /*only shrink RF Rx LPF for HT40*/
- if (shrink_rx_lpf)
- shrink_rx_lpf = false;
- }
-
- halbtc8821a2ant_RfShrink(btcoexist, NORMAL_EXEC, shrink_rx_lpf);
- halbtc8821a2ant_low_penalty_ra(btcoexist,
- NORMAL_EXEC, low_penalty_ra);
-
- /* no limited DIG
- * btc8821a2_set_bt_lna_const(btcoexist,
- NORMAL_EXEC, bBTLNAConstrain);
- */
+ btc8821a2ant_low_penalty_ra(btcoexist, NORMAL_EXEC, low_penalty_ra);
}
-static void btc8821a2ant_sw_mech2(struct btc_coexist *btcoexist,
- bool agc_table_shift,
- bool adc_back_off, bool sw_dac_swing,
- u32 dac_swing_lvl)
+static void btc8821a2ant_sw_mechanism2(struct btc_coexist *btcoexist,
+ bool agc_table_shift, bool adc_back_off,
+ bool sw_dac_swing, u32 dac_swing_lvl)
{
- /* halbtc8821a2ant_AgcTable(btcoexist, NORMAL_EXEC, bAGCTableShift); */
- halbtc8821a2ant_adc_back_off(btcoexist, NORMAL_EXEC, adc_back_off);
- halbtc8821a2ant_dac_swing(btcoexist, NORMAL_EXEC, sw_dac_swing,
- sw_dac_swing);
+ btc8821a2ant_dac_swing(btcoexist, NORMAL_EXEC, sw_dac_swing,
+ dac_swing_lvl);
}
-static void halbtc8821a2ant_set_ant_path(struct btc_coexist *btcoexist,
- u8 ant_pos_type, bool init_hw_cfg,
- bool wifi_off)
+static void btc8821a2ant_set_ant_path(struct btc_coexist *btcoexist,
+ u8 ant_pos_type, bool init_hw_cfg,
+ bool wifi_off)
{
struct btc_board_info *board_info = &btcoexist->board_info;
u32 u4tmp = 0;
@@ -1189,21 +1109,18 @@
btcoexist->btc_write_4byte(btcoexist, 0x4c, u4tmp);
btcoexist->btc_write_4byte(btcoexist, 0x974, 0x3ff);
- btcoexist->btc_write_1byte(btcoexist, 0xcb4, 0x77);
if (board_info->btdm_ant_pos == BTC_ANTENNA_AT_MAIN_PORT) {
- /* tell firmware "antenna inverse" ==>
- * WRONG firmware antenna control code.
- * ==>need fw to fix
+ /* tell firmware "antenna inverse" ==> WRONG firmware
+ * antenna control code ==>need fw to fix
*/
h2c_parameter[0] = 1;
h2c_parameter[1] = 1;
btcoexist->btc_fill_h2c(btcoexist, 0x65, 2,
h2c_parameter);
} else {
- /* tell firmware "no antenna inverse"
- * ==> WRONG firmware antenna control code.
- * ==>need fw to fix
+ /* tell firmware "no antenna inverse" ==> WRONG firmware
+ * antenna control code ==>need fw to fix
*/
h2c_parameter[0] = 0;
h2c_parameter[1] = 1;
@@ -1223,11 +1140,25 @@
}
}
-static void halbtc8821a2ant_ps_tdma(struct btc_coexist *btcoexist,
- bool force_exec, bool turn_on, u8 type)
+static void btc8821a2ant_ps_tdma(struct btc_coexist *btcoexist,
+ bool force_exec, bool turn_on, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
+ u8 wifi_rssi_state, bt_rssi_state;
+
+ wifi_rssi_state = btc8821a2ant_wifi_rssi_state(btcoexist, 1, 2,
+ BT_8821A_2ANT_WIFI_RSSI_COEXSWITCH_THRES, 0);
+ bt_rssi_state = btc8821a2ant_bt_rssi_state(btcoexist, 2,
+ BT_8821A_2ANT_BT_RSSI_COEXSWITCH_THRES, 0);
+
+ if (!(BTC_RSSI_HIGH(wifi_rssi_state) &&
+ BTC_RSSI_HIGH(bt_rssi_state)) &&
+ turn_on) {
+ /* for WiFi RSSI low or BT RSSI low */
+ type = type + 100;
+ }
+
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], %s turn %s PS TDMA, type = %d\n",
(force_exec ? "force to" : ""), (turn_on ? "ON" : "OFF"),
@@ -1251,108 +1182,181 @@
switch (type) {
case 1:
default:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x1a,
- 0x1a, 0xe1, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x3c,
+ 0x03, 0xf1, 0x90);
break;
case 2:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x12,
- 0x12, 0xe1, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x2d,
+ 0x03, 0xf1, 0x90);
break;
case 3:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x1c,
- 0x3, 0xf1, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1c,
+ 0x3, 0xf1, 0x90);
break;
case 4:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x10,
- 0x03, 0xf1, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x10,
+ 0x03, 0xf1, 0x90);
break;
case 5:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x1a,
- 0x1a, 0x60, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x3c,
+ 0x3, 0x70, 0x90);
break;
case 6:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x12,
- 0x12, 0x60, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x2d,
+ 0x3, 0x70, 0x90);
break;
case 7:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x1c,
- 0x3, 0x70, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1c,
+ 0x3, 0x70, 0x90);
break;
case 8:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xa3, 0x10,
- 0x3, 0x70, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xa3, 0x10,
+ 0x3, 0x70, 0x90);
break;
case 9:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x1a,
- 0x1a, 0xe1, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x3c,
+ 0x03, 0xf1, 0x90);
break;
case 10:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x12,
- 0x12, 0xe1, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x2d,
+ 0x03, 0xf1, 0x90);
break;
case 11:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0xa,
- 0xa, 0xe1, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1c,
+ 0x3, 0xf1, 0x90);
break;
case 12:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x5,
- 0x5, 0xe1, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x10,
+ 0x3, 0xf1, 0x90);
break;
case 13:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x1a,
- 0x1a, 0x60, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x3c,
+ 0x3, 0x70, 0x90);
break;
case 14:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3,
- 0x12, 0x12, 0x60, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x2d,
+ 0x3, 0x70, 0x90);
break;
case 15:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0xa,
- 0xa, 0x60, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1c,
+ 0x3, 0x70, 0x90);
break;
case 16:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x5,
- 0x5, 0x60, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x10,
+ 0x3, 0x70, 0x90);
break;
case 17:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xa3, 0x2f,
- 0x2f, 0x60, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xa3, 0x2f,
+ 0x2f, 0x60, 0x90);
break;
case 18:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x5,
- 0x5, 0xe1, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x5, 0x5,
+ 0xe1, 0x90);
break;
case 19:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x25,
- 0x25, 0xe1, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x25,
+ 0x25, 0xe1, 0x90);
break;
case 20:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x25,
- 0x25, 0x60, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x25,
+ 0x25, 0x60, 0x90);
break;
case 21:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x15,
- 0x03, 0x70, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x15,
+ 0x03, 0x70, 0x90);
+ break;
+ case 23:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x1e,
+ 0x03, 0xf0, 0x14);
+ break;
+ case 24:
+ case 124:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xd3, 0x3c,
+ 0x03, 0x70, 0x50);
+ break;
+ case 25:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x14,
+ 0x03, 0xf1, 0x90);
+ break;
+ case 26:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x30,
+ 0x03, 0xf1, 0x90);
break;
case 71:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0xe3, 0x1a,
- 0x1a, 0xe1, 0x90);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x3c,
+ 0x03, 0xf1, 0x90);
+ break;
+ case 101:
+ case 105:
+ case 171:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xd3, 0x3a,
+ 0x03, 0x70, 0x50);
+ break;
+ case 102:
+ case 106:
+ case 110:
+ case 114:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xd3, 0x2d,
+ 0x03, 0x70, 0x50);
+ break;
+ case 103:
+ case 107:
+ case 111:
+ case 115:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xd3, 0x1c,
+ 0x03, 0x70, 0x50);
+ break;
+ case 104:
+ case 108:
+ case 112:
+ case 116:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xd3, 0x10,
+ 0x03, 0x70, 0x50);
+ break;
+ case 109:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x3c,
+ 0x03, 0xf1, 0x90);
+ break;
+ case 113:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x3c,
+ 0x03, 0x70, 0x90);
+ break;
+ case 121:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x15,
+ 0x03, 0x70, 0x90);
+ break;
+ case 22:
+ case 122:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xe3, 0x35,
+ 0x03, 0x71, 0x11);
+ break;
+ case 123:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xd3, 0x1c,
+ 0x03, 0x70, 0x54);
+ break;
+ case 125:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xd3, 0x14,
+ 0x03, 0x70, 0x50);
+ break;
+ case 126:
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0xd3, 0x30,
+ 0x03, 0x70, 0x50);
break;
}
} else {
/* disable PS tdma */
switch (type) {
case 0:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0x0, 0x0, 0x0,
- 0x40, 0x0);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0x0, 0x0, 0x0,
+ 0x40, 0x0);
break;
case 1:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0x0, 0x0, 0x0,
- 0x48, 0x0);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0x0, 0x0, 0x0,
+ 0x48, 0x0);
break;
default:
- halbtc8821a2ant_set_fw_pstdma(btcoexist, 0x0, 0x0, 0x0,
- 0x40, 0x0);
+ btc8821a2ant_set_fw_ps_tdma(btcoexist, 0x0, 0x0, 0x0,
+ 0x40, 0x0);
break;
}
}
@@ -1362,867 +1366,450 @@
coex_dm->pre_ps_tdma = coex_dm->cur_ps_tdma;
}
-static void halbtc8821a2ant_coex_all_off(struct btc_coexist *btcoexist)
+static void
+btc8821a2ant_ps_tdma_check_for_power_save_state(struct btc_coexist *btcoexist,
+ bool new_ps_state)
+{
+ u8 lps_mode = 0x0;
+
+ btcoexist->btc_get(btcoexist, BTC_GET_U1_LPS_MODE, &lps_mode);
+
+ if (lps_mode) {
+ /* already under LPS state */
+ if (new_ps_state) {
+ /* keep state under LPS, do nothing */
+ } else {
+ /* will leave LPS state, turn off psTdma first */
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+ }
+ } else {
+ /* NO PS state */
+ if (new_ps_state) {
+ /* will enter LPS state, turn off psTdma first */
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+ } else {
+ /* keep state under NO PS state, do nothing */
+ }
+ }
+}
+
+static void btc8821a2ant_power_save_state(struct btc_coexist *btcoexist,
+ u8 ps_type, u8 lps_val, u8 rpwm_val)
+{
+ bool low_pwr_disable = false;
+
+ switch (ps_type) {
+ case BTC_PS_WIFI_NATIVE:
+ /* recover to original 32k low power setting */
+ low_pwr_disable = false;
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
+ &low_pwr_disable);
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_NORMAL_LPS, NULL);
+ coex_sta->force_lps_on = false;
+ break;
+ case BTC_PS_LPS_ON:
+ btc8821a2ant_ps_tdma_check_for_power_save_state(btcoexist,
+ true);
+ btc8821a2ant_lps_rpwm(btcoexist, NORMAL_EXEC, lps_val,
+ rpwm_val);
+ /* when coex force to enter LPS, do not enter 32k low power */
+ low_pwr_disable = true;
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
+ &low_pwr_disable);
+ /* power save must executed before psTdma */
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_ENTER_LPS, NULL);
+ coex_sta->force_lps_on = true;
+ break;
+ case BTC_PS_LPS_OFF:
+ btc8821a2ant_ps_tdma_check_for_power_save_state(btcoexist,
+ false);
+ btcoexist->btc_set(btcoexist, BTC_SET_ACT_LEAVE_LPS, NULL);
+ coex_sta->force_lps_on = false;
+ break;
+ default:
+ break;
+ }
+}
+
+static void btc8821a2ant_coex_all_off(struct btc_coexist *btcoexist)
{
/* fw all off */
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
/* sw all off */
- btc8821a2ant_sw_mech1(btcoexist, false, false, false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false, false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false, false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false, false, 0x18);
/* hw all off */
- halbtc8821a2ant_coex_table(btcoexist, NORMAL_EXEC,
- 0x55555555, 0x55555555, 0xffff, 0x3);
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
}
-static void halbtc8821a2ant_coex_under_5g(struct btc_coexist *btcoexist)
+static void btc8821a2ant_coex_under_5g(struct btc_coexist *btcoexist)
{
- halbtc8821a2ant_coex_all_off(btcoexist);
+ btc8821a2ant_coex_all_off(btcoexist);
+ btc8821a2ant_ignore_wlan_act(btcoexist, NORMAL_EXEC, true);
}
-static void halbtc8821a2ant_init_coex_dm(struct btc_coexist *btcoexist)
+static void btc8821a2ant_init_coex_dm(struct btc_coexist *btcoexist)
{
/* force to reset coex mechanism */
- halbtc8821a2ant_coex_table(btcoexist, FORCE_EXEC, 0x55555555,
- 0x55555555, 0xffff, 0x3);
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
- halbtc8821a2ant_ps_tdma(btcoexist, FORCE_EXEC, false, 1);
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, FORCE_EXEC, 6);
- halbtc8821a2ant_dec_bt_pwr(btcoexist, FORCE_EXEC, false);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ btc8821a2ant_ps_tdma(btcoexist, FORCE_EXEC, false, 1);
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, FORCE_EXEC, 6);
+ btc8821a2ant_dec_bt_pwr(btcoexist, FORCE_EXEC, 0);
- btc8821a2ant_sw_mech1(btcoexist, false, false, false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false, false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false, false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false, false, 0x18);
}
-static void halbtc8821a2ant_bt_inquiry_page(struct btc_coexist *btcoexist)
+static void btc8821a2ant_action_bt_inquiry(struct btc_coexist *btcoexist)
{
+ struct rtl_priv *rtlpriv = btcoexist->adapter;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
+ bool wifi_connected = false;
bool low_pwr_disable = true;
+ bool scan = false, link = false, roam = false;
+
+ wifi_rssi_state =
+ btc8821a2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8821a2ant_wifi_rssi_state(btcoexist, 1, 2,
+ BT_8821A_2ANT_WIFI_RSSI_COEXSWITCH_THRES, 0);
+ bt_rssi_state = btc8821a2ant_bt_rssi_state(btcoexist,
+ 2, BT_8821A_2ANT_BT_RSSI_COEXSWITCH_THRES, 0);
btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
&low_pwr_disable);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
+ &wifi_connected);
- halbtc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
- 0x5afa5afa, 0xffff, 0x3);
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 3);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_SCAN, &scan);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_LINK, &link);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_ROAM, &roam);
+
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+
+ if (scan || link || roam) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Wifi link process + BT Inq/Page!!\n");
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 15);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 22);
+ } else if (wifi_connected) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Wifi connected + BT Inq/Page!!\n");
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 15);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 22);
+ } else {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Wifi no-link + BT Inq/Page!!\n");
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+ }
+
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, FORCE_EXEC, 6);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false, false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false, false, 0x18);
}
-static bool halbtc8821a2ant_is_common_action(struct btc_coexist *btcoexist)
+void btc8821a2ant_action_wifi_link_process(struct btc_coexist *btcoexist)
+{
+ struct rtl_priv *rtlpriv = btcoexist->adapter;
+ u8 u8tmpa, u8tmpb;
+
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 15);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 22);
+
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false, false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false, false, 0x18);
+
+ u8tmpa = btcoexist->btc_read_1byte(btcoexist, 0x765);
+ u8tmpb = btcoexist->btc_read_1byte(btcoexist, 0x76e);
+
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], 0x765=0x%x, 0x76e=0x%x\n", u8tmpa, u8tmpb);
+}
+
+static bool btc8821a2ant_action_wifi_idle_process(struct btc_coexist *btcoexist)
+{
+ struct rtl_priv *rtlpriv = btcoexist->adapter;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
+ u8 ap_num = 0;
+
+ wifi_rssi_state =
+ btc8821a2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8821a2ant_wifi_rssi_state(btcoexist, 1, 2,
+ BT_8821A_2ANT_WIFI_RSSI_COEXSWITCH_THRES - 20, 0);
+ bt_rssi_state = btc8821a2ant_bt_rssi_state(btcoexist,
+ 2, BT_8821A_2ANT_BT_RSSI_COEXSWITCH_THRES, 0);
+
+ btcoexist->btc_get(btcoexist, BTC_GET_U1_AP_NUM, &ap_num);
+
+ /* define the office environment */
+ if (BTC_RSSI_HIGH(wifi_rssi_state1) && (coex_sta->hid_exist) &&
+ (coex_sta->a2dp_exist)) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Wifi idle process for BT HID+A2DP exist!!\n");
+
+ btc8821a2ant_dac_swing(btcoexist, NORMAL_EXEC, true, 0x6);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+
+ /* sw all off */
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false, false,
+ false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false, false,
+ 0x18);
+
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+
+ return true;
+ } else if (coex_sta->pan_exist) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Wifi idle process for BT PAN exist!!\n");
+
+ btc8821a2ant_dac_swing(btcoexist, NORMAL_EXEC, true, 0x6);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+
+ /* sw all off */
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false, false,
+ false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false, false,
+ 0x18);
+
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+
+ return true;
+ }
+ btc8821a2ant_dac_swing(btcoexist, NORMAL_EXEC, true, 0x18);
+ return false;
+}
+
+static bool btc8821a2ant_is_common_action(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
bool common = false, wifi_connected = false, wifi_busy = false;
bool low_pwr_disable = false;
+ bool bt_hs_on = false;
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_CONNECTED,
&wifi_connected);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_BUSY, &wifi_busy);
- halbtc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
- 0x5afa5afa, 0xffff, 0x3);
-
- if (!wifi_connected &&
- BT_8821A_2ANT_BT_STATUS_IDLE == coex_dm->bt_status) {
+ if (!wifi_connected) {
low_pwr_disable = false;
btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
&low_pwr_disable);
+ btc8821a2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false,
+ 0x8);
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi IPS + BT IPS!!\n");
+ "[BTCoex], Wifi non-connected idle!!\n");
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff,
+ 0x0);
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
- btc8821a2ant_sw_mech1(btcoexist, false, false, false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false, false, 0x18);
-
- common = true;
- } else if (wifi_connected &&
- (BT_8821A_2ANT_BT_STATUS_IDLE == coex_dm->bt_status)) {
- low_pwr_disable = false;
- btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
- &low_pwr_disable);
-
- if (wifi_busy) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi Busy + BT IPS!!\n");
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- false, 1);
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi LPS + BT IPS!!\n");
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- false, 1);
- }
-
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
-
- btc8821a2ant_sw_mech1(btcoexist, false, false, false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false, false, 0x18);
-
- common = true;
- } else if (!wifi_connected &&
- (BT_8821A_2ANT_BT_STATUS_CON_IDLE == coex_dm->bt_status)) {
- low_pwr_disable = true;
- btcoexist->btc_set(btcoexist, BTC_SET_ACT_DISABLE_LOW_POWER,
- &low_pwr_disable);
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi IPS + BT LPS!!\n");
-
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
-
- btc8821a2ant_sw_mech1(btcoexist, false, false, false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false, false, 0x18);
- common = true;
- } else if (wifi_connected &&
- (BT_8821A_2ANT_BT_STATUS_CON_IDLE == coex_dm->bt_status)) {
- low_pwr_disable = true;
- btcoexist->btc_set(btcoexist,
- BTC_SET_ACT_DISABLE_LOW_POWER, &low_pwr_disable);
-
- if (wifi_busy) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi Busy + BT LPS!!\n");
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- false, 1);
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi LPS + BT LPS!!\n");
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- false, 1);
- }
-
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
-
- btc8821a2ant_sw_mech1(btcoexist, true, true, true, true);
- btc8821a2ant_sw_mech2(btcoexist, false, false, false, 0x18);
-
- common = true;
- } else if (!wifi_connected &&
- (BT_8821A_2ANT_BT_STATUS_NON_IDLE ==
- coex_dm->bt_status)) {
- low_pwr_disable = false;
- btcoexist->btc_set(btcoexist,
- BTC_SET_ACT_DISABLE_LOW_POWER, &low_pwr_disable);
-
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi IPS + BT Busy!!\n");
-
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
-
- btc8821a2ant_sw_mech1(btcoexist, false, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false, false,
+ false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false, false,
+ 0x18);
common = true;
} else {
- low_pwr_disable = true;
- btcoexist->btc_set(btcoexist,
- BTC_SET_ACT_DISABLE_LOW_POWER,
- &low_pwr_disable);
+ if (BT_8821A_2ANT_BT_STATUS_IDLE ==
+ coex_dm->bt_status) {
+ low_pwr_disable = false;
+ btcoexist->btc_set(btcoexist,
+ BTC_SET_ACT_DISABLE_LOW_POWER,
+ &low_pwr_disable);
+ btc8821a2ant_limited_rx(btcoexist, NORMAL_EXEC,
+ false, false, 0x8);
- if (wifi_busy) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi Busy + BT Busy!!\n");
- common = false;
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], Wifi LPS + BT Busy!!\n");
- halbtc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC, true, 21);
+ "[BTCoex], Wifi connected + BT non connected-idle!!\n");
- if (halbtc8821a2ant_need_to_dec_bt_pwr(btcoexist))
- halbtc8821a2ant_dec_bt_pwr(btcoexist,
- NORMAL_EXEC, true);
- else
- halbtc8821a2ant_dec_bt_pwr(btcoexist,
- NORMAL_EXEC, false);
+ btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1,
+ 0xfffff, 0x0);
+ btc8821a2ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 0);
+
+ btc8821a2ant_power_save_state(
+ btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC,
+ 0xb);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
common = true;
+ } else if (BT_8821A_2ANT_BT_STATUS_CON_IDLE ==
+ coex_dm->bt_status) {
+ low_pwr_disable = true;
+ btcoexist->btc_set(btcoexist,
+ BTC_SET_ACT_DISABLE_LOW_POWER,
+ &low_pwr_disable);
+
+ if (bt_hs_on)
+ return false;
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Wifi connected + BT connected-idle!!\n");
+ btc8821a2ant_limited_rx(btcoexist, NORMAL_EXEC,
+ false, false, 0x8);
+
+ btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1,
+ 0xfffff, 0x0);
+ btc8821a2ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 0);
+
+ btc8821a2ant_power_save_state(
+ btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC,
+ 0xb);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+
+ btc8821a2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
+ common = true;
+ } else {
+ low_pwr_disable = true;
+ btcoexist->btc_set(btcoexist,
+ BTC_SET_ACT_DISABLE_LOW_POWER,
+ &low_pwr_disable);
+
+ if (wifi_busy) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Wifi Connected-Busy + BT Busy!!\n");
+ common = false;
+ } else {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], Wifi Connected-Idle + BT Busy!!\n");
+ common =
+ btc8821a2ant_action_wifi_idle_process(
+ btcoexist);
+ }
}
- btc8821a2ant_sw_mech1(btcoexist, true, true, true, true);
}
return common;
}
-static void btc8821a2_int1(struct btc_coexist *btcoexist, bool tx_pause,
- int result)
+static void btc8821a2ant_tdma_duration_adjust(struct btc_coexist *btcoexist,
+ bool sco_hid, bool tx_pause,
+ u8 max_interval)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- if (tx_pause) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 1\n");
-
- if (coex_dm->cur_ps_tdma == 71) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 5);
- coex_dm->tdma_adj_type = 5;
- } else if (coex_dm->cur_ps_tdma == 1) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 5);
- coex_dm->tdma_adj_type = 5;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 4) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- }
- if (coex_dm->cur_ps_tdma == 9) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 13);
- coex_dm->tdma_adj_type = 13;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 12) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
-
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 5) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- } else if (coex_dm->cur_ps_tdma == 13) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 8) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 5);
- coex_dm->tdma_adj_type = 5;
- } else if (coex_dm->cur_ps_tdma == 16) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 13);
- coex_dm->tdma_adj_type = 13;
- }
- }
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 0\n");
- if (coex_dm->cur_ps_tdma == 5) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 71);
- coex_dm->tdma_adj_type = 71;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 8) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- }
- if (coex_dm->cur_ps_tdma == 13) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 16) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
-
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 71) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 1);
- coex_dm->tdma_adj_type = 1;
- } else if (coex_dm->cur_ps_tdma == 1) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- } else if (coex_dm->cur_ps_tdma == 9) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 4) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 1);
- coex_dm->tdma_adj_type = 1;
- } else if (coex_dm->cur_ps_tdma == 1) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 71);
- coex_dm->tdma_adj_type = 71;
- } else if (coex_dm->cur_ps_tdma == 12) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
- }
- }
- }
-}
-
-static void btc8821a2_int2(struct btc_coexist *btcoexist, bool tx_pause,
- int result)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- if (tx_pause) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 1\n");
- if (coex_dm->cur_ps_tdma == 1) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 4) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- }
- if (coex_dm->cur_ps_tdma == 9) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 12) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 5) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- } else if (coex_dm->cur_ps_tdma == 13) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 8) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
- } else if (coex_dm->cur_ps_tdma == 16) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
- }
- }
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 0\n");
- if (coex_dm->cur_ps_tdma == 5) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 8) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- }
- if (coex_dm->cur_ps_tdma == 13) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 16) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 1) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- } else if (coex_dm->cur_ps_tdma == 9) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 4) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
- } else if (coex_dm->cur_ps_tdma == 12) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
- }
- }
- }
-}
-
-static void btc8821a2_int3(struct btc_coexist *btcoexist, bool tx_pause,
- int result)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
-
- if (tx_pause) {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 1\n");
- if (coex_dm->cur_ps_tdma == 1) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 4) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- }
- if (coex_dm->cur_ps_tdma == 9) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 12) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 5) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 8);
- coex_dm->tdma_adj_type = 8;
- } else if (coex_dm->cur_ps_tdma == 13) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 16);
- coex_dm->tdma_adj_type = 16;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 8) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
- } else if (coex_dm->cur_ps_tdma == 16) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
- }
- }
- } else {
- RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
- "[BTCoex], TxPause = 0\n");
- if (coex_dm->cur_ps_tdma == 5) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 6) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 7) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 8) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- }
- if (coex_dm->cur_ps_tdma == 13) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 14) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 15) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 16) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- if (result == -1) {
- if (coex_dm->cur_ps_tdma == 1) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 4);
- coex_dm->tdma_adj_type = 4;
- } else if (coex_dm->cur_ps_tdma == 9) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 12);
- coex_dm->tdma_adj_type = 12;
- }
- } else if (result == 1) {
- if (coex_dm->cur_ps_tdma == 4) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 3) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 2) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
- } else if (coex_dm->cur_ps_tdma == 12) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 11) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- } else if (coex_dm->cur_ps_tdma == 10) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
- }
- }
- }
-}
-
-static void btc8821a2ant_tdma_dur_adj(struct btc_coexist *btcoexist,
- bool sco_hid, bool tx_pause,
- u8 max_interval)
-{
- struct rtl_priv *rtlpriv = btcoexist->adapter;
- static long up, dn, m, n, wait_count;
- /* 0: no change, +1: increase WiFi duration,
+ static long up, dn, m, n, wait_count;
+ /* 0 : no change
+ * +1: increase WiFi duration
* -1: decrease WiFi duration
*/
- int result;
- u8 retry_count = 0;
+ int result;
+ u8 retry_count = 0;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], TdmaDurationAdjust()\n");
- if (coex_dm->reset_tdma_adjust) {
- coex_dm->reset_tdma_adjust = false;
+ if (coex_dm->auto_tdma_adjust) {
+ coex_dm->auto_tdma_adjust = false;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], first run TdmaDurationAdjust()!!\n");
if (sco_hid) {
if (tx_pause) {
if (max_interval == 1) {
- halbtc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 13);
- coex_dm->tdma_adj_type = 13;
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 13);
+ coex_dm->ps_tdma_du_adj_type = 13;
} else if (max_interval == 2) {
- halbtc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 14);
- coex_dm->tdma_adj_type = 14;
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 14);
+ coex_dm->ps_tdma_du_adj_type = 14;
+ } else if (max_interval == 3) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 15);
+ coex_dm->ps_tdma_du_adj_type = 15;
} else {
- halbtc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 15);
- coex_dm->tdma_adj_type = 15;
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 15);
+ coex_dm->ps_tdma_du_adj_type = 15;
}
} else {
if (max_interval == 1) {
- halbtc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 9);
- coex_dm->tdma_adj_type = 9;
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 9);
+ coex_dm->ps_tdma_du_adj_type = 9;
} else if (max_interval == 2) {
- halbtc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 10);
- coex_dm->tdma_adj_type = 10;
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 10);
+ coex_dm->ps_tdma_du_adj_type = 10;
+ } else if (max_interval == 3) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 11);
+ coex_dm->ps_tdma_du_adj_type = 11;
} else {
- halbtc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 11);
- coex_dm->tdma_adj_type = 11;
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 11);
+ coex_dm->ps_tdma_du_adj_type = 11;
}
}
} else {
if (tx_pause) {
if (max_interval == 1) {
- halbtc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 5);
- coex_dm->tdma_adj_type = 5;
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 5);
+ coex_dm->ps_tdma_du_adj_type = 5;
} else if (max_interval == 2) {
- halbtc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 6);
- coex_dm->tdma_adj_type = 6;
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 6);
+ coex_dm->ps_tdma_du_adj_type = 6;
+ } else if (max_interval == 3) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 7);
+ coex_dm->ps_tdma_du_adj_type = 7;
} else {
- halbtc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 7);
- coex_dm->tdma_adj_type = 7;
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 7);
+ coex_dm->ps_tdma_du_adj_type = 7;
}
} else {
if (max_interval == 1) {
- halbtc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 1);
- coex_dm->tdma_adj_type = 1;
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 1);
+ coex_dm->ps_tdma_du_adj_type = 1;
} else if (max_interval == 2) {
- halbtc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 2);
- coex_dm->tdma_adj_type = 2;
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 2);
+ coex_dm->ps_tdma_du_adj_type = 2;
+ } else if (max_interval == 3) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 3);
+ coex_dm->ps_tdma_du_adj_type = 3;
} else {
- halbtc8821a2ant_ps_tdma(btcoexist,
- NORMAL_EXEC,
- true, 3);
- coex_dm->tdma_adj_type = 3;
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 3);
+ coex_dm->ps_tdma_du_adj_type = 3;
}
}
}
@@ -2273,7 +1860,7 @@
up = 0;
if (dn == 2) {
- /* if retry count< 3 for 2*2 seconds,
+ /* if retry count < 3 for 2*2 seconds,
* shrink wifi duration
*/
if (wait_count <= 2)
@@ -2286,7 +1873,7 @@
if (m >= 20)
m = 20;
- n = 3*m;
+ n = 3 * m;
up = 0;
dn = 0;
wait_count = 0;
@@ -2308,7 +1895,7 @@
if (m >= 20)
m = 20;
- n = 3*m;
+ n = 3 * m;
up = 0;
dn = 0;
wait_count = 0;
@@ -2319,624 +1906,1313 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], max Interval = %d\n", max_interval);
- if (max_interval == 1)
- btc8821a2_int1(btcoexist, tx_pause, result);
- else if (max_interval == 2)
- btc8821a2_int2(btcoexist, tx_pause, result);
- else if (max_interval == 3)
- btc8821a2_int3(btcoexist, tx_pause, result);
+
+ if (max_interval == 1) {
+ if (tx_pause) {
+ if (coex_dm->cur_ps_tdma == 71) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 5);
+ coex_dm->ps_tdma_du_adj_type = 5;
+ } else if (coex_dm->cur_ps_tdma == 1) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 5);
+ coex_dm->ps_tdma_du_adj_type = 5;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 6);
+ coex_dm->ps_tdma_du_adj_type = 6;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 7);
+ coex_dm->ps_tdma_du_adj_type = 7;
+ } else if (coex_dm->cur_ps_tdma == 4) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 8);
+ coex_dm->ps_tdma_du_adj_type = 8;
+ }
+ if (coex_dm->cur_ps_tdma == 9) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 13);
+ coex_dm->ps_tdma_du_adj_type = 13;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 14);
+ coex_dm->ps_tdma_du_adj_type = 14;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 15);
+ coex_dm->ps_tdma_du_adj_type = 15;
+ } else if (coex_dm->cur_ps_tdma == 12) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 16);
+ coex_dm->ps_tdma_du_adj_type = 16;
+ }
+
+ if (result == -1) {
+ if (coex_dm->cur_ps_tdma == 5) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 6);
+ coex_dm->ps_tdma_du_adj_type =
+ 6;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 8);
+ coex_dm->ps_tdma_du_adj_type =
+ 8;
+ } else if (coex_dm->cur_ps_tdma == 13) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 14);
+ coex_dm->ps_tdma_du_adj_type =
+ 14;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 16);
+ coex_dm->ps_tdma_du_adj_type =
+ 16;
+ }
+ } else if (result == 1) {
+ if (coex_dm->cur_ps_tdma == 8) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 6);
+ coex_dm->ps_tdma_du_adj_type =
+ 6;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 5);
+ coex_dm->ps_tdma_du_adj_type =
+ 5;
+ } else if (coex_dm->cur_ps_tdma == 16) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 14);
+ coex_dm->ps_tdma_du_adj_type =
+ 14;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 13);
+ coex_dm->ps_tdma_du_adj_type =
+ 13;
+ }
+ }
+ } else {
+ if (coex_dm->cur_ps_tdma == 5) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 71);
+ coex_dm->ps_tdma_du_adj_type = 71;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 2);
+ coex_dm->ps_tdma_du_adj_type = 2;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 3);
+ coex_dm->ps_tdma_du_adj_type = 3;
+ } else if (coex_dm->cur_ps_tdma == 8) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 4);
+ coex_dm->ps_tdma_du_adj_type = 4;
+ }
+ if (coex_dm->cur_ps_tdma == 13) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 9);
+ coex_dm->ps_tdma_du_adj_type = 9;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 10);
+ coex_dm->ps_tdma_du_adj_type = 10;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 11);
+ coex_dm->ps_tdma_du_adj_type = 11;
+ } else if (coex_dm->cur_ps_tdma == 16) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 12);
+ coex_dm->ps_tdma_du_adj_type = 12;
+ }
+
+ if (result == -1) {
+ if (coex_dm->cur_ps_tdma == 71) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 1);
+ coex_dm->ps_tdma_du_adj_type =
+ 1;
+ } else if (coex_dm->cur_ps_tdma == 1) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 2);
+ coex_dm->ps_tdma_du_adj_type =
+ 2;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 4);
+ coex_dm->ps_tdma_du_adj_type =
+ 4;
+ } else if (coex_dm->cur_ps_tdma == 9) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 10);
+ coex_dm->ps_tdma_du_adj_type =
+ 10;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 12);
+ coex_dm->ps_tdma_du_adj_type =
+ 12;
+ }
+ } else if (result == 1) {
+ if (coex_dm->cur_ps_tdma == 4) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 2);
+ coex_dm->ps_tdma_du_adj_type =
+ 2;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 1);
+ coex_dm->ps_tdma_du_adj_type =
+ 1;
+ } else if (coex_dm->cur_ps_tdma == 1) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 71);
+ coex_dm->ps_tdma_du_adj_type =
+ 71;
+ } else if (coex_dm->cur_ps_tdma == 12) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 10);
+ coex_dm->ps_tdma_du_adj_type =
+ 10;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 9);
+ coex_dm->ps_tdma_du_adj_type =
+ 9;
+ }
+ }
+ }
+ } else if (max_interval == 2) {
+ if (tx_pause) {
+ if (coex_dm->cur_ps_tdma == 1) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 6);
+ coex_dm->ps_tdma_du_adj_type = 6;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 6);
+ coex_dm->ps_tdma_du_adj_type = 6;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 7);
+ coex_dm->ps_tdma_du_adj_type = 7;
+ } else if (coex_dm->cur_ps_tdma == 4) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 8);
+ coex_dm->ps_tdma_du_adj_type = 8;
+ }
+ if (coex_dm->cur_ps_tdma == 9) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 14);
+ coex_dm->ps_tdma_du_adj_type = 14;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 14);
+ coex_dm->ps_tdma_du_adj_type = 14;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 15);
+ coex_dm->ps_tdma_du_adj_type = 15;
+ } else if (coex_dm->cur_ps_tdma == 12) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 16);
+ coex_dm->ps_tdma_du_adj_type = 16;
+ }
+ if (result == -1) {
+ if (coex_dm->cur_ps_tdma == 5) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 6);
+ coex_dm->ps_tdma_du_adj_type =
+ 6;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 8);
+ coex_dm->ps_tdma_du_adj_type =
+ 8;
+ } else if (coex_dm->cur_ps_tdma == 13) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 14);
+ coex_dm->ps_tdma_du_adj_type =
+ 14;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 16);
+ coex_dm->ps_tdma_du_adj_type =
+ 16;
+ }
+ } else if (result == 1) {
+ if (coex_dm->cur_ps_tdma == 8) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 6);
+ coex_dm->ps_tdma_du_adj_type =
+ 6;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 6);
+ coex_dm->ps_tdma_du_adj_type =
+ 6;
+ } else if (coex_dm->cur_ps_tdma == 16) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 14);
+ coex_dm->ps_tdma_du_adj_type =
+ 14;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 14);
+ coex_dm->ps_tdma_du_adj_type =
+ 14;
+ }
+ }
+ } else {
+ if (coex_dm->cur_ps_tdma == 5) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 2);
+ coex_dm->ps_tdma_du_adj_type = 2;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 2);
+ coex_dm->ps_tdma_du_adj_type = 2;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 3);
+ coex_dm->ps_tdma_du_adj_type = 3;
+ } else if (coex_dm->cur_ps_tdma == 8) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 4);
+ coex_dm->ps_tdma_du_adj_type = 4;
+ }
+ if (coex_dm->cur_ps_tdma == 13) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 10);
+ coex_dm->ps_tdma_du_adj_type = 10;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 10);
+ coex_dm->ps_tdma_du_adj_type = 10;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 11);
+ coex_dm->ps_tdma_du_adj_type = 11;
+ } else if (coex_dm->cur_ps_tdma == 16) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 12);
+ coex_dm->ps_tdma_du_adj_type = 12;
+ }
+ if (result == -1) {
+ if (coex_dm->cur_ps_tdma == 1) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 2);
+ coex_dm->ps_tdma_du_adj_type =
+ 2;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 4);
+ coex_dm->ps_tdma_du_adj_type =
+ 4;
+ } else if (coex_dm->cur_ps_tdma == 9) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 10);
+ coex_dm->ps_tdma_du_adj_type =
+ 10;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 12);
+ coex_dm->ps_tdma_du_adj_type =
+ 12;
+ }
+ } else if (result == 1) {
+ if (coex_dm->cur_ps_tdma == 4) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 2);
+ coex_dm->ps_tdma_du_adj_type =
+ 2;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 2);
+ coex_dm->ps_tdma_du_adj_type =
+ 2;
+ } else if (coex_dm->cur_ps_tdma == 12) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 10);
+ coex_dm->ps_tdma_du_adj_type =
+ 10;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 10);
+ coex_dm->ps_tdma_du_adj_type =
+ 10;
+ }
+ }
+ }
+ } else if (max_interval == 3) {
+ if (tx_pause) {
+ if (coex_dm->cur_ps_tdma == 1) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 7);
+ coex_dm->ps_tdma_du_adj_type = 7;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 7);
+ coex_dm->ps_tdma_du_adj_type = 7;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 7);
+ coex_dm->ps_tdma_du_adj_type = 7;
+ } else if (coex_dm->cur_ps_tdma == 4) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 8);
+ coex_dm->ps_tdma_du_adj_type = 8;
+ }
+ if (coex_dm->cur_ps_tdma == 9) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 15);
+ coex_dm->ps_tdma_du_adj_type = 15;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 15);
+ coex_dm->ps_tdma_du_adj_type = 15;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 15);
+ coex_dm->ps_tdma_du_adj_type = 15;
+ } else if (coex_dm->cur_ps_tdma == 12) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 16);
+ coex_dm->ps_tdma_du_adj_type = 16;
+ }
+ if (result == -1) {
+ if (coex_dm->cur_ps_tdma == 5) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 8);
+ coex_dm->ps_tdma_du_adj_type =
+ 8;
+ } else if (coex_dm->cur_ps_tdma == 13) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 16);
+ coex_dm->ps_tdma_du_adj_type =
+ 16;
+ }
+ } else if (result == 1) {
+ if (coex_dm->cur_ps_tdma == 8) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 7);
+ coex_dm->ps_tdma_du_adj_type =
+ 7;
+ } else if (coex_dm->cur_ps_tdma == 16) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 15);
+ coex_dm->ps_tdma_du_adj_type =
+ 15;
+ }
+ }
+ } else {
+ if (coex_dm->cur_ps_tdma == 5) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 3);
+ coex_dm->ps_tdma_du_adj_type = 3;
+ } else if (coex_dm->cur_ps_tdma == 6) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 3);
+ coex_dm->ps_tdma_du_adj_type = 3;
+ } else if (coex_dm->cur_ps_tdma == 7) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 3);
+ coex_dm->ps_tdma_du_adj_type = 3;
+ } else if (coex_dm->cur_ps_tdma == 8) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 4);
+ coex_dm->ps_tdma_du_adj_type = 4;
+ }
+ if (coex_dm->cur_ps_tdma == 13) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 11);
+ coex_dm->ps_tdma_du_adj_type = 11;
+ } else if (coex_dm->cur_ps_tdma == 14) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 11);
+ coex_dm->ps_tdma_du_adj_type = 11;
+ } else if (coex_dm->cur_ps_tdma == 15) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 11);
+ coex_dm->ps_tdma_du_adj_type = 11;
+ } else if (coex_dm->cur_ps_tdma == 16) {
+ btc8821a2ant_ps_tdma(btcoexist,
+ NORMAL_EXEC, true, 12);
+ coex_dm->ps_tdma_du_adj_type = 12;
+ }
+ if (result == -1) {
+ if (coex_dm->cur_ps_tdma == 1) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 4);
+ coex_dm->ps_tdma_du_adj_type =
+ 4;
+ } else if (coex_dm->cur_ps_tdma == 9) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 12);
+ coex_dm->ps_tdma_du_adj_type =
+ 12;
+ }
+ } else if (result == 1) {
+ if (coex_dm->cur_ps_tdma == 4) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 3) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 2) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 3);
+ coex_dm->ps_tdma_du_adj_type =
+ 3;
+ } else if (coex_dm->cur_ps_tdma == 12) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 11) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ } else if (coex_dm->cur_ps_tdma == 10) {
+ btc8821a2ant_ps_tdma(
+ btcoexist, NORMAL_EXEC,
+ true, 11);
+ coex_dm->ps_tdma_du_adj_type =
+ 11;
+ }
+ }
+ }
+ }
}
/* if current PsTdma not match with the recorded one
* (when scan, dhcp...), then we have to adjust it back to
* the previous recorded one.
*/
- if (coex_dm->cur_ps_tdma != coex_dm->tdma_adj_type) {
- bool scan = false, link = false, roam = false;
+ if (coex_dm->cur_ps_tdma != coex_dm->ps_tdma_du_adj_type) {
+ bool scan = false, link = false, roam = false;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], PsTdma type dismatch!!!, cur_ps_tdma = %d, recordPsTdma = %d\n",
- coex_dm->cur_ps_tdma, coex_dm->tdma_adj_type);
+ coex_dm->cur_ps_tdma, coex_dm->ps_tdma_du_adj_type);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_SCAN, &scan);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_LINK, &link);
btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_ROAM, &roam);
if (!scan && !link && !roam) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true,
- coex_dm->tdma_adj_type);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true,
+ coex_dm->ps_tdma_du_adj_type);
} else {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], roaming/link/scan is under progress, will adjust next time!!!\n");
}
}
-
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 0x6);
}
/* SCO only or SCO+PAN(HS)*/
-static void halbtc8821a2ant_action_sco(struct btc_coexist *btcoexist)
+static void btc8821a2ant_action_sco(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state;
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
+ u8 wifi_rssi_state, bt_rssi_state;
u32 wifi_bw;
- wifi_rssi_state = halbtc8821a2ant_wifi_rssi_state(btcoexist, 0, 2,
- 15, 0);
- bt_rssi_state = halbtc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
+ wifi_rssi_state = btc8821a2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 4);
+ btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
- if (halbtc8821a2ant_need_to_dec_bt_pwr(btcoexist))
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ btc8821a2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 4);
+
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
else
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- if (BTC_WIFI_BW_LEGACY == wifi_bw) {
+ if (wifi_bw == BTC_WIFI_BW_LEGACY) {
/* for SCO quality at 11b/g mode */
- halbtc8821a2ant_coex_table(btcoexist, NORMAL_EXEC,
- 0x5a5a5a5a, 0x5a5a5a5a, 0xffff, 0x3);
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
} else {
/* for SCO quality & wifi performance balance at 11n mode */
- halbtc8821a2ant_coex_table(btcoexist, NORMAL_EXEC,
- 0x5aea5aea, 0x5aea5aea, 0xffff, 0x3);
+ if (wifi_bw == BTC_WIFI_BW_HT40) {
+ btc8821a2ant_coex_table_with_type(btcoexist,
+ NORMAL_EXEC, 8);
+ } else {
+ if (bt_link_info->sco_only)
+ btc8821a2ant_coex_table_with_type(
+ btcoexist, NORMAL_EXEC, 17);
+ else
+ btc8821a2ant_coex_table_with_type(
+ btcoexist, NORMAL_EXEC, 12);
+ }
}
- if (BTC_WIFI_BW_HT40 == wifi_bw) {
- /* fw mechanism
- * halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 5);
- */
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ /* for voice quality */
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 0);
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- false, 0); /*for voice quality*/
-
- /* sw mechanism */
+ /* sw mechanism */
+ if (wifi_bw == BTC_WIFI_BW_HT40) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, true, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ true, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, true, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ true, 0x18);
}
} else {
- /* fw mechanism
- * halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 5);
- */
- if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- false, 0); /*for voice quality*/
- } else {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- false, 0); /*for voice quality*/
- }
-
- /* sw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, false, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ true, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, false, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ true, 0x18);
}
}
}
-static void halbtc8821a2ant_action_hid(struct btc_coexist *btcoexist)
+static void btc8821a2ant_action_hid(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state;
- u32 wifi_bw;
+ u8 wifi_rssi_state, bt_rssi_state;
+ u32 wifi_bw;
- wifi_rssi_state = halbtc8821a2ant_wifi_rssi_state(btcoexist,
- 0, 2, 15, 0);
- bt_rssi_state = halbtc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
+ wifi_rssi_state = btc8821a2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8821a2ant_bt_rssi_state(btcoexist,
+ 2, BT_8821A_2ANT_BT_RSSI_COEXSWITCH_THRES, 0);
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
- if (halbtc8821a2ant_need_to_dec_bt_pwr(btcoexist))
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ btc8821a2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
else
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- if (BTC_WIFI_BW_LEGACY == wifi_bw) {
+ if (wifi_bw == BTC_WIFI_BW_LEGACY) {
/* for HID at 11b/g mode */
- halbtc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
- 0x5a5a5a5a, 0xffff, 0x3);
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
} else {
/* for HID quality & wifi performance balance at 11n mode */
- halbtc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
- 0x5aea5aea, 0xffff, 0x3);
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 2);
}
- if (BTC_WIFI_BW_HT40 == wifi_bw) {
- /* fw mechanism */
- if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- } else {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 13);
- }
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 24);
+ if (wifi_bw == BTC_WIFI_BW_HT40) {
/* sw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, true, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, true, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- /* fw mechanism */
- if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 9);
- } else {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 13);
- }
-
/* sw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, false, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, false, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
/* A2DP only / PAN(EDR) only/ A2DP+PAN(HS) */
-static void halbtc8821a2ant_action_a2dp(struct btc_coexist *btcoexist)
+static void btc8821a2ant_action_a2dp(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state;
- u32 wifi_bw;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
+ u8 ap_num = 0;
+ u32 wifi_bw;
- wifi_rssi_state = halbtc8821a2ant_wifi_rssi_state(btcoexist, 0, 2,
- 15, 0);
- bt_rssi_state = halbtc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
+ wifi_rssi_state = btc8821a2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8821a2ant_wifi_rssi_state(btcoexist, 1, 2,
+ BT_8821A_2ANT_WIFI_RSSI_COEXSWITCH_THRES, 0);
+ bt_rssi_state = btc8821a2ant_bt_rssi_state(btcoexist,
+ 2, BT_8821A_2ANT_BT_RSSI_COEXSWITCH_THRES, 0);
- /* fw dac swing is called in btc8821a2ant_tdma_dur_adj()
- * halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- */
+ if ((ap_num >= 10) && BTC_RSSI_HIGH(wifi_rssi_state1) &&
+ BTC_RSSI_HIGH(bt_rssi_state)) {
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
- if (halbtc8821a2ant_need_to_dec_bt_pwr(btcoexist))
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
- else
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff,
+ 0x0);
+ btc8821a2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false,
+ 0x8);
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
- btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
- if (BTC_WIFI_BW_HT40 == wifi_bw) {
- /* fw mechanism */
- if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_tdma_dur_adj(btcoexist, false, false, 1);
- } else {
- btc8821a2ant_tdma_dur_adj(btcoexist, false, true, 1);
- }
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 23);
/* sw mechanism */
+ btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
+ if (wifi_bw == BTC_WIFI_BW_HT40) {
+ btc8821a2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ true, 0x6);
+ } else {
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ true, 0x6);
+ }
+ return;
+ }
+
+ btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
+ btc8821a2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ else
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+
+ if (BTC_RSSI_HIGH(wifi_rssi_state1) && BTC_RSSI_HIGH(bt_rssi_state)) {
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ } else {
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 13);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_LPS_ON, 0x50,
+ 0x4);
+ }
+
+ if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 23);
+ } else {
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 23);
+ }
+
+ /* sw mechanism */
+ btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
+ if (wifi_bw == BTC_WIFI_BW_HT40) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, true, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, true, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- /* fw mechanism */
- if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_tdma_dur_adj(btcoexist, false, false, 1);
- } else {
- btc8821a2ant_tdma_dur_adj(btcoexist, false, true, 1);
- }
-
- /* sw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, false, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, false, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
-static void halbtc8821a2ant_action_a2dp_pan_hs(struct btc_coexist *btcoexist)
+static void btc8821a2ant_action_a2dp_pan_hs(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state, bt_info_ext;
- u32 wifi_bw;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
+ u32 wifi_bw;
- bt_info_ext = coex_sta->bt_info_ext;
- wifi_rssi_state = halbtc8821a2ant_wifi_rssi_state(btcoexist, 0, 2,
- 15, 0);
- bt_rssi_state = halbtc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
+ wifi_rssi_state = btc8821a2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8821a2ant_wifi_rssi_state(btcoexist, 1, 2,
+ BT_8821A_2ANT_WIFI_RSSI_COEXSWITCH_THRES, 0);
+ bt_rssi_state = btc8821a2ant_bt_rssi_state(btcoexist,
+ 2, BT_8821A_2ANT_BT_RSSI_COEXSWITCH_THRES, 0);
- /*fw dac swing is called in btc8821a2ant_tdma_dur_adj()
- *halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- */
+ btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
- if (halbtc8821a2ant_need_to_dec_bt_pwr(btcoexist))
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ btc8821a2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
else
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+ if (BTC_RSSI_HIGH(wifi_rssi_state1) && BTC_RSSI_HIGH(bt_rssi_state)) {
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ } else {
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 13);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_LPS_ON, 0x50,
+ 0x4);
+ }
+
+ btc8821a2ant_tdma_duration_adjust(btcoexist, false, true, 2);
+
+ /* sw mechanism */
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
-
- if (BTC_WIFI_BW_HT40 == wifi_bw) {
- /* fw mechanism */
- if (bt_info_ext&BIT0) {
- /*a2dp basic rate*/
- btc8821a2ant_tdma_dur_adj(btcoexist, false, true, 2);
- } else {
- /*a2dp edr rate*/
- btc8821a2ant_tdma_dur_adj(btcoexist, false, true, 1);
- }
-
- /* sw mechanism */
+ if (wifi_bw == BTC_WIFI_BW_HT40) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, true, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, true, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- /* fw mechanism */
- if (bt_info_ext&BIT0) {
- /* a2dp basic rate */
- btc8821a2ant_tdma_dur_adj(btcoexist, false, true, 2);
- } else {
- /* a2dp edr rate */
- btc8821a2ant_tdma_dur_adj(btcoexist, false, true, 1);
- }
-
- /* sw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, false, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, false, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
-static void halbtc8821a2ant_action_pan_edr(struct btc_coexist *btcoexist)
+static void btc8821a2ant_action_pan_edr(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state;
- u32 wifi_bw;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
+ u32 wifi_bw;
- wifi_rssi_state = halbtc8821a2ant_wifi_rssi_state(btcoexist, 0, 2,
- 15, 0);
- bt_rssi_state = halbtc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
+ wifi_rssi_state = btc8821a2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8821a2ant_wifi_rssi_state(btcoexist, 1, 2,
+ BT_8821A_2ANT_WIFI_RSSI_COEXSWITCH_THRES, 0);
+ bt_rssi_state = btc8821a2ant_bt_rssi_state(btcoexist,
+ 2, BT_8821A_2ANT_BT_RSSI_COEXSWITCH_THRES, 0);
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
- if (halbtc8821a2ant_need_to_dec_bt_pwr(btcoexist))
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ btc8821a2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
else
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
- btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
-
- if (BTC_WIFI_BW_LEGACY == wifi_bw) {
- /* for HID at 11b/g mode */
- halbtc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
- 0x5aff5aff, 0xffff, 0x3);
+ if (BTC_RSSI_HIGH(wifi_rssi_state1) && BTC_RSSI_HIGH(bt_rssi_state)) {
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 10);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
} else {
- /* for HID quality & wifi performance balance at 11n mode */
- halbtc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
- 0x5aff5aff, 0xffff, 0x3);
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 13);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_LPS_ON, 0x50,
+ 0x4);
}
- if (BTC_WIFI_BW_HT40 == wifi_bw) {
- /* fw mechanism */
- if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 1);
- } else {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 5);
- }
+ if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH))
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 26);
+ else
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 26);
- /* sw mechanism */
+ /* sw mechanism */
+ btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
+ if (wifi_bw == BTC_WIFI_BW_HT40) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, true, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, true, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- /* fw mechanism */
- if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 1);
- } else {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 5);
- }
-
- /* sw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, false, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, false, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
/* PAN(HS) only */
-static void halbtc8821a2ant_action_pan_hs(struct btc_coexist *btcoexist)
+static void btc8821a2ant_action_pan_hs(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state;
- u32 wifi_bw;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
+ u32 wifi_bw;
- wifi_rssi_state = halbtc8821a2ant_wifi_rssi_state(btcoexist,
- 0, 2, 15, 0);
- bt_rssi_state = halbtc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
+ wifi_rssi_state = btc8821a2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8821a2ant_wifi_rssi_state(btcoexist, 1, 2,
+ BT_8821A_2ANT_WIFI_RSSI_COEXSWITCH_THRES, 0);
+ bt_rssi_state = btc8821a2ant_bt_rssi_state(btcoexist,
+ 2, BT_8821A_2ANT_BT_RSSI_COEXSWITCH_THRES, 0);
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
+
+ btc8821a2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
+ else
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
-
- if (BTC_WIFI_BW_HT40 == wifi_bw) {
- /* fw mechanism */
+ if (wifi_bw == BTC_WIFI_BW_HT40) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC,
- true);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC,
- false);
- }
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
-
- /* sw mechanism */
- if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, true, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
- } else {
- btc8821a2ant_sw_mech1(btcoexist, true, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- /* fw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8821a2ant_dec_bt_pwr(btcoexist,
- NORMAL_EXEC, true);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- halbtc8821a2ant_dec_bt_pwr(btcoexist,
- NORMAL_EXEC, false);
- }
-
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
-
- /* sw mechanism */
- if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, false, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
- } else {
- btc8821a2ant_sw_mech1(btcoexist, false, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
/* PAN(EDR)+A2DP */
-static void halbtc8821a2ant_action_pan_edr_a2dp(struct btc_coexist *btcoexist)
+static void btc8821a2ant_action_pan_edr_a2dp(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state, bt_info_ext;
- u32 wifi_bw;
+ u8 wifi_rssi_state, wifi_rssi_state1, bt_rssi_state;
+ u32 wifi_bw;
- bt_info_ext = coex_sta->bt_info_ext;
- wifi_rssi_state = halbtc8821a2ant_wifi_rssi_state(btcoexist, 0, 2,
- 15, 0);
- bt_rssi_state = halbtc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
+ wifi_rssi_state = btc8821a2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ wifi_rssi_state1 = btc8821a2ant_wifi_rssi_state(btcoexist, 1, 2,
+ BT_8821A_2ANT_WIFI_RSSI_COEXSWITCH_THRES, 0);
+ bt_rssi_state = btc8821a2ant_bt_rssi_state(btcoexist,
+ 2, BT_8821A_2ANT_BT_RSSI_COEXSWITCH_THRES, 0);
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btcoexist->btc_set_rf_reg(btcoexist, BTC_RF_A, 0x1, 0xfffff, 0x0);
- if (halbtc8821a2ant_need_to_dec_bt_pwr(btcoexist))
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ btc8821a2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
else
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+
+ if (BTC_RSSI_HIGH(wifi_rssi_state1) && BTC_RSSI_HIGH(bt_rssi_state))
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ else
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_LPS_ON, 0x50,
+ 0x4);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- halbtc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
- 0x5afa5afa, 0xffff, 0x3);
+ if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 12);
- if (BTC_WIFI_BW_HT40 == wifi_bw) {
- /* fw mechanism */
- if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH))
- btc8821a2ant_tdma_dur_adj(btcoexist, false,
- false, 3);
+ if (wifi_bw == BTC_WIFI_BW_HT40)
+ btc8821a2ant_tdma_duration_adjust(btcoexist, false,
+ true, 3);
else
- btc8821a2ant_tdma_dur_adj(btcoexist, false,
- true, 3);
+ btc8821a2ant_tdma_duration_adjust(btcoexist, false,
+ false, 3);
+ } else {
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 13);
+ btc8821a2ant_tdma_duration_adjust(btcoexist, false, true, 3);
+ }
- /* sw mechanism */
+ /* sw mechanism */
+ if (wifi_bw == BTC_WIFI_BW_HT40) {
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, true, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, true, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- /* fw mechanism */
- if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
- (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH))
- btc8821a2ant_tdma_dur_adj(btcoexist, false, false, 3);
- else
- btc8821a2ant_tdma_dur_adj(btcoexist, false, true, 3);
-
- /* sw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, false, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, false, false,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
-static void halbtc8821a2ant_action_pan_edr_hid(struct btc_coexist *btcoexist)
+static void btc8821a2ant_action_pan_edr_hid(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state;
- u32 wifi_bw;
+ u8 wifi_rssi_state, bt_rssi_state;
+ u32 wifi_bw;
- wifi_rssi_state = halbtc8821a2ant_wifi_rssi_state(btcoexist, 0, 2,
- 15, 0);
- bt_rssi_state = halbtc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
+ wifi_rssi_state = btc8821a2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8821a2ant_bt_rssi_state(btcoexist,
+ 2, BT_8821A_2ANT_BT_RSSI_COEXSWITCH_THRES, 0);
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- if (halbtc8821a2ant_need_to_dec_bt_pwr(btcoexist))
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
else
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- halbtc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
- 0x5a5f5a5f, 0xffff, 0x3);
+ if (wifi_bw == BTC_WIFI_BW_LEGACY) {
+ /* for HID at 11b/g mode */
+ btc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
+ 0x5a5f5a5f, 0xffff, 0x3);
+ } else {
+ /* for HID quality & wifi performance balance at 11n mode */
+ btc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
+ 0x5a5f5a5f, 0xffff, 0x3);
+ }
- if (BTC_WIFI_BW_HT40 == wifi_bw) {
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 3);
+ if (wifi_bw == BTC_WIFI_BW_HT40) {
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 3);
/* fw mechanism */
if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
(bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
+ true, 10);
} else {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
}
/* sw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, true, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, true, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
/* fw mechanism */
if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
(bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 10);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 10);
} else {
- halbtc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC,
- true, 14);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, true, 14);
}
/* sw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, false, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, false, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
@@ -2944,42 +3220,70 @@
/* HID+A2DP+PAN(EDR) */
static void btc8821a2ant_act_hid_a2dp_pan_edr(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state, bt_info_ext;
- u32 wifi_bw;
+ u8 wifi_rssi_state, bt_rssi_state, bt_info_ext;
+ u32 wifi_bw;
bt_info_ext = coex_sta->bt_info_ext;
- wifi_rssi_state = halbtc8821a2ant_wifi_rssi_state(btcoexist,
- 0, 2, 15, 0);
- bt_rssi_state = halbtc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
+ wifi_rssi_state = btc8821a2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
- halbtc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btc8821a2ant_limited_rx(btcoexist, NORMAL_EXEC, false, false, 0x8);
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
- if (halbtc8821a2ant_need_to_dec_bt_pwr(btcoexist))
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 2);
else
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- halbtc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
- 0x5a5a5a5a, 0xffff, 0x3);
+ if (wifi_bw == BTC_WIFI_BW_LEGACY) {
+ /* for HID at 11b/g mode */
+ btc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
+ 0x5a5a5a5a, 0xffff, 0x3);
+ } else {
+ /* for HID quality & wifi performance balance at 11n mode */
+ btc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
+ 0x5a5a5a5a, 0xffff, 0x3);
+ }
if (BTC_WIFI_BW_HT40 == wifi_bw) {
/* fw mechanism */
- btc8821a2ant_tdma_dur_adj(btcoexist, true, true, 3);
+ if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ if (bt_info_ext&BIT0) {
+ /* a2dp basic rate */
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true, 3);
+ } else {
+ /* a2dp edr rate */
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true, 3);
+ }
+ } else {
+ if (bt_info_ext&BIT0) {
+ /* a2dp basic rate */
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true, 3);
+ } else {
+ /* a2dp edr rate */
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true, 3);
+ }
+ }
/* sw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, true, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, true, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
/* fw mechanism */
@@ -2987,103 +3291,183 @@
(bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
if (bt_info_ext&BIT0) {
/* a2dp basic rate */
- btc8821a2ant_tdma_dur_adj(btcoexist, true,
- false, 3);
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, false, 3);
} else {
/* a2dp edr rate */
- btc8821a2ant_tdma_dur_adj(btcoexist, true,
- false, 3);
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, false, 3);
}
} else {
if (bt_info_ext&BIT0) {
/* a2dp basic rate */
- btc8821a2ant_tdma_dur_adj(btcoexist, true,
- true, 3);
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true,
+ 3);
} else {
/* a2dp edr rate */
- btc8821a2ant_tdma_dur_adj(btcoexist, true,
- true, 3);
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true,
+ 3);
}
}
/* sw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, false, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, false, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
-static void halbtc8821a2ant_action_hid_a2dp(struct btc_coexist *btcoexist)
+static void btc8821a2ant_action_hid_a2dp(struct btc_coexist *btcoexist)
{
- u8 wifi_rssi_state, bt_rssi_state, bt_info_ext;
- u32 wifi_bw;
+ u8 wifi_rssi_state, bt_rssi_state, bt_info_ext;
+ u32 wifi_bw;
bt_info_ext = coex_sta->bt_info_ext;
- wifi_rssi_state = halbtc8821a2ant_wifi_rssi_state(btcoexist, 0, 2,
- 15, 0);
- bt_rssi_state = halbtc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
+ wifi_rssi_state = btc8821a2ant_wifi_rssi_state(btcoexist, 0, 2, 15, 0);
+ bt_rssi_state = btc8821a2ant_bt_rssi_state(btcoexist, 2, 35, 0);
- if (halbtc8821a2ant_need_to_dec_bt_pwr(btcoexist))
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
+ if (BTC_RSSI_HIGH(bt_rssi_state))
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, true);
else
- halbtc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, false);
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- halbtc8821a2ant_coex_table(btcoexist, NORMAL_EXEC, 0x55ff55ff,
- 0x5f5b5f5b, 0xffffff, 0x3);
+ if (wifi_bw == BTC_WIFI_BW_LEGACY) {
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 7);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE,
+ 0x0, 0x0);
+ } else {
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 14);
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_LPS_ON, 0x50,
+ 0x4);
+ }
if (BTC_WIFI_BW_HT40 == wifi_bw) {
/* fw mechanism */
- btc8821a2ant_tdma_dur_adj(btcoexist, true, true, 2);
+ if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ if (bt_info_ext & BIT0) {
+ /* a2dp basic rate */
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true,
+ 2);
+ } else {
+ /* a2dp edr rate */
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true,
+ 2);
+ }
+ } else {
+ if (bt_info_ext & BIT0) {
+ /* a2dp basic rate */
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true,
+ 2);
+ } else {
+ /* a2dp edr rate */
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true,
+ 2);
+ }
+ }
/* sw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, true, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, true, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, true, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
} else {
/* fw mechanism */
- btc8821a2ant_tdma_dur_adj(btcoexist, true, true, 2);
+ if ((bt_rssi_state == BTC_RSSI_STATE_HIGH) ||
+ (bt_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
+ if (bt_info_ext & BIT0) {
+ /* a2dp basic rate */
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true,
+ 2);
+
+ } else {
+ /* a2dp edr rate */
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true,
+ 2);
+ }
+ } else {
+ if (bt_info_ext & BIT0) {
+ /*a2dp basic rate*/
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true,
+ 2);
+ } else {
+ /*a2dp edr rate*/
+ btc8821a2ant_tdma_duration_adjust(btcoexist,
+ true, true,
+ 2);
+ }
+ }
/* sw mechanism */
if ((wifi_rssi_state == BTC_RSSI_STATE_HIGH) ||
(wifi_rssi_state == BTC_RSSI_STATE_STAY_HIGH)) {
- btc8821a2ant_sw_mech1(btcoexist, false, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, true, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, true, false,
+ false, 0x18);
} else {
- btc8821a2ant_sw_mech1(btcoexist, false, true,
- false, false);
- btc8821a2ant_sw_mech2(btcoexist, false, false,
- false, 0x18);
+ btc8821a2ant_sw_mechanism1(btcoexist, false, true,
+ false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false,
+ false, 0x18);
}
}
}
-static void halbtc8821a2ant_run_coexist_mechanism(struct btc_coexist *btcoexist)
+static void btc8821a2ant_action_wifi_multi_port(struct btc_coexist *btcoexist)
+{
+ btc8821a2ant_fw_dac_swing_lvl(btcoexist, NORMAL_EXEC, 6);
+ btc8821a2ant_dec_bt_pwr(btcoexist, NORMAL_EXEC, 0);
+
+ /* sw all off */
+ btc8821a2ant_sw_mechanism1(btcoexist, false, false, false, false);
+ btc8821a2ant_sw_mechanism2(btcoexist, false, false, false, 0x18);
+
+ /* hw all off */
+ btc8821a2ant_coex_table_with_type(btcoexist, NORMAL_EXEC, 0);
+
+ btc8821a2ant_power_save_state(btcoexist, BTC_PS_WIFI_NATIVE, 0x0, 0x0);
+ btc8821a2ant_ps_tdma(btcoexist, NORMAL_EXEC, false, 1);
+}
+
+static void btc8821a2ant_run_coexist_mechanism(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- bool wifi_under_5g = false;
- u8 algorithm = 0;
+ struct btc_bt_link_info *bt_link_info = &btcoexist->bt_link_info;
+ bool wifi_under_5g = false;
+ u8 algorithm = 0;
+ u32 num_of_wifi_link = 0;
+ u32 wifi_link_status = 0;
+ bool miracast_plus_bt = false;
+ bool scan = false, link = false, roam = false;
if (btcoexist->manual_control) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -3091,30 +3475,73 @@
return;
}
- btcoexist->btc_get(btcoexist,
- BTC_GET_BL_WIFI_UNDER_5G, &wifi_under_5g);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_UNDER_5G, &wifi_under_5g);
if (wifi_under_5g) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], RunCoexistMechanism(), run 5G coex setting!!<===\n");
- halbtc8821a2ant_coex_under_5g(btcoexist);
+ btc8821a2ant_coex_under_5g(btcoexist);
return;
}
- algorithm = halbtc8821a2ant_action_algorithm(btcoexist);
+ if (coex_sta->under_ips) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], wifi is under IPS !!!\n");
+ return;
+ }
+
+ algorithm = btc8821a2ant_action_algorithm(btcoexist);
if (coex_sta->c2h_bt_inquiry_page &&
(BT_8821A_2ANT_COEX_ALGO_PANHS != algorithm)) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], BT is under inquiry/page scan !!\n");
- halbtc8821a2ant_bt_inquiry_page(btcoexist);
+ btc8821a2ant_action_bt_inquiry(btcoexist);
return;
}
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_SCAN, &scan);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_LINK, &link);
+ btcoexist->btc_get(btcoexist, BTC_GET_BL_WIFI_ROAM, &roam);
+
+ if (scan || link || roam) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "[BTCoex], WiFi is under Link Process !!\n");
+ btc8821a2ant_action_wifi_link_process(btcoexist);
+ return;
+ }
+
+ /* for P2P */
+ btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_LINK_STATUS,
+ &wifi_link_status);
+ num_of_wifi_link = wifi_link_status >> 16;
+
+ if ((num_of_wifi_link >= 2) ||
+ (wifi_link_status & WIFI_P2P_GO_CONNECTED)) {
+ RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
+ "############# [BTCoex], Multi-Port num_of_wifi_link = %d, wifi_link_status = 0x%x\n",
+ num_of_wifi_link, wifi_link_status);
+
+ if (bt_link_info->bt_link_exist)
+ miracast_plus_bt = true;
+ else
+ miracast_plus_bt = false;
+
+ btcoexist->btc_set(btcoexist, BTC_SET_BL_MIRACAST_PLUS_BT,
+ &miracast_plus_bt);
+ btc8821a2ant_action_wifi_multi_port(btcoexist);
+
+ return;
+ }
+
+ miracast_plus_bt = false;
+ btcoexist->btc_set(btcoexist, BTC_SET_BL_MIRACAST_PLUS_BT,
+ &miracast_plus_bt);
+
coex_dm->cur_algorithm = algorithm;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Algorithm = %d\n", coex_dm->cur_algorithm);
- if (halbtc8821a2ant_is_common_action(btcoexist)) {
+ if (btc8821a2ant_is_common_action(btcoexist)) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action 2-Ant common\n");
coex_dm->reset_tdma_adjust = true;
@@ -3130,42 +3557,42 @@
case BT_8821A_2ANT_COEX_ALGO_SCO:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action 2-Ant, algorithm = SCO\n");
- halbtc8821a2ant_action_sco(btcoexist);
+ btc8821a2ant_action_sco(btcoexist);
break;
case BT_8821A_2ANT_COEX_ALGO_HID:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action 2-Ant, algorithm = HID\n");
- halbtc8821a2ant_action_hid(btcoexist);
+ btc8821a2ant_action_hid(btcoexist);
break;
case BT_8821A_2ANT_COEX_ALGO_A2DP:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action 2-Ant, algorithm = A2DP\n");
- halbtc8821a2ant_action_a2dp(btcoexist);
+ btc8821a2ant_action_a2dp(btcoexist);
break;
case BT_8821A_2ANT_COEX_ALGO_A2DP_PANHS:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action 2-Ant, algorithm = A2DP+PAN(HS)\n");
- halbtc8821a2ant_action_a2dp_pan_hs(btcoexist);
+ btc8821a2ant_action_a2dp_pan_hs(btcoexist);
break;
case BT_8821A_2ANT_COEX_ALGO_PANEDR:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action 2-Ant, algorithm = PAN(EDR)\n");
- halbtc8821a2ant_action_pan_edr(btcoexist);
+ btc8821a2ant_action_pan_edr(btcoexist);
break;
case BT_8821A_2ANT_COEX_ALGO_PANHS:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action 2-Ant, algorithm = HS mode\n");
- halbtc8821a2ant_action_pan_hs(btcoexist);
+ btc8821a2ant_action_pan_hs(btcoexist);
break;
case BT_8821A_2ANT_COEX_ALGO_PANEDR_A2DP:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action 2-Ant, algorithm = PAN+A2DP\n");
- halbtc8821a2ant_action_pan_edr_a2dp(btcoexist);
+ btc8821a2ant_action_pan_edr_a2dp(btcoexist);
break;
case BT_8821A_2ANT_COEX_ALGO_PANEDR_HID:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action 2-Ant, algorithm = PAN(EDR)+HID\n");
- halbtc8821a2ant_action_pan_edr_hid(btcoexist);
+ btc8821a2ant_action_pan_edr_hid(btcoexist);
break;
case BT_8821A_2ANT_COEX_ALGO_HID_A2DP_PANEDR:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -3175,26 +3602,22 @@
case BT_8821A_2ANT_COEX_ALGO_HID_A2DP:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action 2-Ant, algorithm = HID+A2DP\n");
- halbtc8821a2ant_action_hid_a2dp(btcoexist);
+ btc8821a2ant_action_hid_a2dp(btcoexist);
break;
default:
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Action 2-Ant, algorithm = coexist All Off!!\n");
- halbtc8821a2ant_coex_all_off(btcoexist);
+ btc8821a2ant_coex_all_off(btcoexist);
break;
}
coex_dm->pre_algorithm = coex_dm->cur_algorithm;
}
}
-/*============================================================
- *work around function start with wa_halbtc8821a2ant_
- *============================================================
- *============================================================
- * extern function start with EXhalbtc8821a2ant_
- *============================================================
- */
-void ex_halbtc8821a2ant_init_hwconfig(struct btc_coexist *btcoexist)
+/**************************************************************
+ * extern function start with ex_btc8821a2ant_
+ **************************************************************/
+void ex_btc8821a2ant_init_hwconfig(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
u8 u1tmp = 0;
@@ -3212,36 +3635,30 @@
u1tmp |= 0x5;
btcoexist->btc_write_1byte(btcoexist, 0x790, u1tmp);
- /*Antenna config */
- halbtc8821a2ant_set_ant_path(btcoexist,
- BTC_ANT_WIFI_AT_MAIN, true, false);
+ /* Antenna config */
+ btc8821a2ant_set_ant_path(btcoexist, BTC_ANT_WIFI_AT_MAIN, true, false);
/* PTA parameter */
- halbtc8821a2ant_coex_table(btcoexist,
- FORCE_EXEC, 0x55555555, 0x55555555,
- 0xffff, 0x3);
+ btc8821a2ant_coex_table_with_type(btcoexist, FORCE_EXEC, 0);
/* Enable counter statistics */
- /*0x76e[3] = 1, WLAN_Act control by PTA*/
- btcoexist->btc_write_1byte(btcoexist, 0x76e, 0xc);
+ /* 0x76e[3] = 1, WLAN_Act control by PTA */
+ btcoexist->btc_write_1byte(btcoexist, 0x76e, 0x4);
btcoexist->btc_write_1byte(btcoexist, 0x778, 0x3);
btcoexist->btc_write_1byte_bitmask(btcoexist, 0x40, 0x20, 0x1);
}
-void ex_halbtc8821a2ant_init_coex_dm(struct btc_coexist *btcoexist)
+void ex_btc8821a2ant_init_coex_dm(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Coex Mechanism Init!!\n");
- halbtc8821a2ant_init_coex_dm(btcoexist);
+ btc8821a2ant_init_coex_dm(btcoexist);
}
-void
-ex_halbtc8821a2ant_display_coex_info(
- struct btc_coexist *btcoexist
- )
+void ex_btc8821a2ant_display_coex_info(struct btc_coexist *btcoexist)
{
struct btc_board_info *board_info = &btcoexist->board_info;
struct btc_stack_info *stack_info = &btcoexist->stack_info;
@@ -3397,7 +3814,7 @@
RT_TRACE(rtlpriv, COMP_INIT, DBG_DMESG,
"\r\n %-35s = %d/ %d ", "DecBtPwr/ IgnWlanAct",
- coex_dm->cur_dec_bt_pwr,
+ coex_dm->cur_dec_bt_pwr_lvl,
coex_dm->cur_ignore_wlan_act);
}
@@ -3475,7 +3892,7 @@
btcoexist->btc_disp_dbg_msg(btcoexist, BTC_DBG_DISP_COEX_STATISTICS);
}
-void ex_halbtc8821a2ant_ips_notify(struct btc_coexist *btcoexist, u8 type)
+void ex_btc8821a2ant_ips_notify(struct btc_coexist *btcoexist, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -3483,16 +3900,15 @@
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], IPS ENTER notify\n");
coex_sta->under_ips = true;
- halbtc8821a2ant_coex_all_off(btcoexist);
+ btc8821a2ant_coex_all_off(btcoexist);
} else if (BTC_IPS_LEAVE == type) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], IPS LEAVE notify\n");
coex_sta->under_ips = false;
- /*halbtc8821a2ant_init_coex_dm(btcoexist);*/
}
}
-void ex_halbtc8821a2ant_lps_notify(struct btc_coexist *btcoexist, u8 type)
+void ex_btc8821a2ant_lps_notify(struct btc_coexist *btcoexist, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -3507,7 +3923,7 @@
}
}
-void ex_halbtc8821a2ant_scan_notify(struct btc_coexist *btcoexist, u8 type)
+void ex_btc8821a2ant_scan_notify(struct btc_coexist *btcoexist, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -3520,7 +3936,7 @@
}
}
-void ex_halbtc8821a2ant_connect_notify(struct btc_coexist *btcoexist, u8 type)
+void ex_btc8821a2ant_connect_notify(struct btc_coexist *btcoexist, u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
@@ -3533,13 +3949,14 @@
}
}
-void ex_halbtc8821a2ant_media_status_notify(struct btc_coexist *btcoexist,
- u8 type)
+void ex_btc8821a2ant_media_status_notify(struct btc_coexist *btcoexist,
+ u8 type)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- u8 h2c_parameter[3] = {0};
- u32 wifi_bw;
- u8 wifi_central_chnl;
+ u8 h2c_parameter[3] = {0};
+ u32 wifi_bw;
+ u8 wifi_central_chnl;
+ u8 ap_num = 0;
if (BTC_MEDIA_CONNECT == type) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
@@ -3549,7 +3966,7 @@
"[BTCoex], MEDIA disconnect notify\n");
}
- /* only 2.4G we need to inform bt the chnl mask*/
+ /* only 2.4G we need to inform bt the chnl mask */
btcoexist->btc_get(btcoexist, BTC_GET_U1_WIFI_CENTRAL_CHNL,
&wifi_central_chnl);
if ((BTC_MEDIA_CONNECT == type) &&
@@ -3557,10 +3974,15 @@
h2c_parameter[0] = 0x1;
h2c_parameter[1] = wifi_central_chnl;
btcoexist->btc_get(btcoexist, BTC_GET_U4_WIFI_BW, &wifi_bw);
- if (BTC_WIFI_BW_HT40 == wifi_bw)
+ if (wifi_bw == BTC_WIFI_BW_HT40) {
h2c_parameter[2] = 0x30;
- else
+ } else {
h2c_parameter[2] = 0x20;
+ if (ap_num < 10)
+ h2c_parameter[2] = 0x30;
+ else
+ h2c_parameter[2] = 0x20;
+ }
}
coex_dm->wifi_chnl_info[0] = h2c_parameter[0];
@@ -3576,8 +3998,9 @@
btcoexist->btc_fill_h2c(btcoexist, 0x66, 3, h2c_parameter);
}
-void ex_halbtc8821a2ant_special_packet_notify(struct btc_coexist *btcoexist,
- u8 type) {
+void ex_btc8821a2ant_special_packet_notify(struct btc_coexist *btcoexist,
+ u8 type)
+{
struct rtl_priv *rtlpriv = btcoexist->adapter;
if (type == BTC_PACKET_DHCP) {
@@ -3586,19 +4009,18 @@
}
}
-void ex_halbtc8821a2ant_bt_info_notify(struct btc_coexist *btcoexist,
- u8 *tmp_buf, u8 length)
+void ex_btc8821a2ant_bt_info_notify(struct btc_coexist *btcoexist,
+ u8 *tmp_buf, u8 length)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- u8 bt_info = 0;
- u8 i, rsp_source = 0;
- static u32 set_bt_lna_cnt, set_bt_psd_mode;
- bool bt_busy = false, limited_dig = false;
- bool wifi_connected = false, bt_hs_on = false;
+ u8 bt_info = 0;
+ u8 i, rsp_source = 0;
+ bool bt_busy = false, limited_dig = false;
+ bool wifi_connected = false, bt_hs_on = false;
coex_sta->c2h_bt_info_req_sent = false;
- rsp_source = tmp_buf[0]&0xf;
+ rsp_source = tmp_buf[0] & 0xf;
if (rsp_source >= BT_INFO_SRC_8821A_2ANT_MAX)
rsp_source = BT_INFO_SRC_8821A_2ANT_WIFI_FW;
coex_sta->bt_info_c2h_cnt[rsp_source]++;
@@ -3610,7 +4032,7 @@
coex_sta->bt_info_c2h[rsp_source][i] = tmp_buf[i];
if (i == 1)
bt_info = tmp_buf[i];
- if (i == length-1) {
+ if (i == length - 1) {
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"0x%02x]\n", tmp_buf[i]);
} else {
@@ -3620,7 +4042,8 @@
}
if (BT_INFO_SRC_8821A_2ANT_WIFI_FW != rsp_source) {
- coex_sta->bt_retry_cnt = /* [3:0]*/
+ /* [3:0] */
+ coex_sta->bt_retry_cnt =
coex_sta->bt_info_c2h[rsp_source][2]&0xf;
coex_sta->bt_rssi =
@@ -3629,53 +4052,28 @@
coex_sta->bt_info_ext =
coex_sta->bt_info_c2h[rsp_source][4];
- /* Here we need to resend some wifi info to BT*/
- /* because bt is reset and loss of the info.*/
+ /* Here we need to resend some wifi info to BT
+ * because bt is reset and loss of the info
+ */
if ((coex_sta->bt_info_ext & BIT1)) {
btcoexist->btc_get(btcoexist,
BTC_GET_BL_WIFI_CONNECTED, &wifi_connected);
if (wifi_connected) {
- ex_halbtc8821a2ant_media_status_notify(btcoexist,
+ ex_btc8821a2ant_media_status_notify(btcoexist,
BTC_MEDIA_CONNECT);
} else {
- ex_halbtc8821a2ant_media_status_notify(btcoexist,
+ ex_btc8821a2ant_media_status_notify(btcoexist,
BTC_MEDIA_DISCONNECT);
}
- set_bt_psd_mode = 0;
- }
- if (set_bt_psd_mode <= 3) {
- halbtc8821a2ant_set_bt_psd_mode(btcoexist, FORCE_EXEC,
- 0x0); /*fix CH-BW mode*/
- set_bt_psd_mode++;
- }
-
- if (coex_dm->cur_bt_lna_constrain) {
- if (!(coex_sta->bt_info_ext & BIT2)) {
- if (set_bt_lna_cnt <= 3) {
- btc8821a2_set_bt_lna_const(btcoexist,
- FORCE_EXEC,
- true);
- set_bt_lna_cnt++;
- }
- }
- } else {
- set_bt_lna_cnt = 0;
}
if ((coex_sta->bt_info_ext & BIT3)) {
- halbtc8821a2ant_ignore_wlan_act(btcoexist,
- FORCE_EXEC, false);
+ btc8821a2ant_ignore_wlan_act(btcoexist,
+ FORCE_EXEC, false);
} else {
/* BT already NOT ignore Wlan active, do nothing here.*/
}
-
- if ((coex_sta->bt_info_ext & BIT4)) {
- /* BT auto report already enabled, do nothing*/
- } else {
- halbtc8821a2ant_bt_auto_report(btcoexist,
- FORCE_EXEC, true);
- }
}
btcoexist->btc_get(btcoexist, BTC_GET_BL_HS_OPERATION, &bt_hs_on);
@@ -3718,8 +4116,7 @@
coex_dm->bt_status = BT_8821A_2ANT_BT_STATUS_IDLE;
}
- if (bt_hs_on)
- coex_dm->bt_status = BT_8821A_2ANT_BT_STATUS_NON_IDLE;
+ btc8821a2ant_update_bt_link_info(btcoexist);
}
if (BT_8821A_2ANT_BT_STATUS_NON_IDLE == coex_dm->bt_status)
@@ -3736,27 +4133,27 @@
btcoexist->btc_set(btcoexist,
BTC_SET_BL_BT_LIMITED_DIG, &limited_dig);
- halbtc8821a2ant_run_coexist_mechanism(btcoexist);
+ btc8821a2ant_run_coexist_mechanism(btcoexist);
}
-void ex_halbtc8821a2ant_halt_notify(struct btc_coexist *btcoexist)
+void ex_btc8821a2ant_halt_notify(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], Halt notify\n");
- halbtc8821a2ant_ignore_wlan_act(btcoexist, FORCE_EXEC, true);
- ex_halbtc8821a2ant_media_status_notify(btcoexist, BTC_MEDIA_DISCONNECT);
+ btc8821a2ant_ignore_wlan_act(btcoexist, FORCE_EXEC, true);
+ ex_btc8821a2ant_media_status_notify(btcoexist, BTC_MEDIA_DISCONNECT);
}
-void ex_halbtc8821a2ant_periodical(struct btc_coexist *btcoexist)
+void ex_btc8821a2ant_periodical(struct btc_coexist *btcoexist)
{
struct rtl_priv *rtlpriv = btcoexist->adapter;
- static u8 dis_ver_info_cnt;
- u32 fw_ver = 0, bt_patch_ver = 0;
+ static u8 dis_ver_info_cnt;
struct btc_board_info *board_info = &btcoexist->board_info;
struct btc_stack_info *stack_info = &btcoexist->stack_info;
+ u32 fw_ver = 0, bt_patch_ver = 0;
RT_TRACE(rtlpriv, COMP_BT_COEXIST, DBG_LOUD,
"[BTCoex], ==========================Periodical===========================\n");
@@ -3785,7 +4182,7 @@
"[BTCoex], ****************************************************************\n");
}
- halbtc8821a2ant_query_bt_info(btcoexist);
- halbtc8821a2ant_monitor_bt_ctr(btcoexist);
- btc8821a2ant_mon_bt_en_dis(btcoexist);
+ btc8821a2ant_query_bt_info(btcoexist);
+ btc8821a2ant_monitor_bt_ctr(btcoexist);
+ btc8821a2ant_monitor_wifi_ctr(btcoexist);
}
diff --git a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a2ant.h b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a2ant.h
index b4cf1f5..535ca10 100644
--- a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a2ant.h
+++ b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtc8821a2ant.h
@@ -38,6 +38,11 @@
#define BTC_RSSI_COEX_THRESH_TOL_8821A_2ANT 2
+/* WiFi RSSI Threshold for 2-Ant TDMA/1-Ant PS-TDMA translation */
+#define BT_8821A_2ANT_WIFI_RSSI_COEXSWITCH_THRES 42
+/* BT RSSI Threshold for 2-Ant TDMA/1-Ant PS-TDMA translation */
+#define BT_8821A_2ANT_BT_RSSI_COEXSWITCH_THRES 46
+
enum _BT_INFO_SRC_8821A_2ANT {
BT_INFO_SRC_8821A_2ANT_WIFI_FW = 0x0,
BT_INFO_SRC_8821A_2ANT_BT_RSP = 0x1,
@@ -69,8 +74,8 @@
struct coex_dm_8821a_2ant {
/* fw mechanism */
- bool pre_dec_bt_pwr;
- bool cur_dec_bt_pwr;
+ bool pre_dec_bt_pwr_lvl;
+ bool cur_dec_bt_pwr_lvl;
bool pre_bt_lna_constrain;
bool cur_bt_lna_constrain;
u8 pre_bt_psd_mode;
@@ -82,8 +87,9 @@
u8 pre_ps_tdma;
u8 cur_ps_tdma;
u8 ps_tdma_para[5];
- u8 tdma_adj_type;
+ u8 ps_tdma_du_adj_type;
bool reset_tdma_adjust;
+ bool auto_tdma_adjust;
bool pre_ps_tdma_on;
bool cur_ps_tdma_on;
bool pre_bt_auto_report;
@@ -118,6 +124,10 @@
u8 cur_algorithm;
u8 bt_status;
u8 wifi_chnl_info[3];
+ u8 pre_lps;
+ u8 cur_lps;
+ u8 pre_rpwm;
+ u8 cur_rpwm;
};
struct coex_sta_8821a_2ant {
@@ -141,6 +151,19 @@
bool c2h_bt_inquiry_page;
u8 bt_retry_cnt;
u8 bt_info_ext;
+
+ u32 crc_ok_cck;
+ u32 crc_ok_11g;
+ u32 crc_ok_11n;
+ u32 crc_ok_11n_agg;
+
+ u32 crc_err_cck;
+ u32 crc_err_11g;
+ u32 crc_err_11n;
+ u32 crc_err_11n_agg;
+
+ u8 coex_table_type;
+ bool force_lps_on;
};
/*===========================================
diff --git a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtcoutsrc.c b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtcoutsrc.c
index 150aeb8..f130006 100644
--- a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtcoutsrc.c
+++ b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtcoutsrc.c
@@ -466,7 +466,7 @@
case BTC_SET_ACT_DISABLE_LOW_POWER:
halbtc_disable_low_power();
break;
- case BTC_SET_ACT_UPDATE_ra_mask:
+ case BTC_SET_ACT_UPDATE_RAMASK:
btcoexist->bt_info.ra_mask = *u32_tmp;
break;
case BTC_SET_ACT_SEND_MIMO_PS:
diff --git a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtcoutsrc.h b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtcoutsrc.h
index 601bbe1..c827113 100644
--- a/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtcoutsrc.h
+++ b/drivers/net/wireless/realtek/rtlwifi/btcoexist/halbtcoutsrc.h
@@ -66,6 +66,15 @@
#define BTC_ANT_WIFI_AT_CPL_MAIN 0
#define BTC_ANT_WIFI_AT_CPL_AUX 1
+enum btc_bt_reg_type {
+ BTC_BT_REG_RF = 0,
+ BTC_BT_REG_MODEM = 1,
+ BTC_BT_REG_BLUEWIZE = 2,
+ BTC_BT_REG_VENDOR = 3,
+ BTC_BT_REG_LE = 4,
+ BTC_BT_REG_MAX
+};
+
enum btc_chip_interface {
BTC_INTF_UNKNOWN = 0,
BTC_INTF_PCI = 1,
@@ -139,6 +148,7 @@
u8 pg_ant_num; /* pg ant number */
u8 btdm_ant_num; /* ant number for btdm */
u8 btdm_ant_pos;
+ u8 single_ant_path; /* current used for 8723b only, 1=>s0, 0=>s1 */
bool bt_exist;
};
@@ -205,6 +215,7 @@
BTC_GET_BL_WIFI_ENABLE_ENCRYPTION,
BTC_GET_BL_WIFI_UNDER_B_MODE,
BTC_GET_BL_EXT_SWITCH,
+ BTC_GET_BL_WIFI_IS_IN_MP_MODE,
/* type s4Byte */
BTC_GET_S4_WIFI_RSSI,
@@ -249,6 +260,8 @@
BTC_SET_BL_TO_REJ_AP_AGG_PKT,
BTC_SET_BL_BT_CTRL_AGG_SIZE,
BTC_SET_BL_INC_SCAN_DEV_NUM,
+ BTC_SET_BL_BT_TX_RX_MASK,
+ BTC_SET_BL_MIRACAST_PLUS_BT,
/* type u1Byte */
BTC_SET_U1_RSSI_ADJ_VAL_FOR_AGC_TABLE_ON,
@@ -275,7 +288,7 @@
BTC_SET_ACT_NORMAL_LPS,
BTC_SET_ACT_INC_FORCE_EXEC_PWR_CMD_CNT,
BTC_SET_ACT_DISABLE_LOW_POWER,
- BTC_SET_ACT_UPDATE_ra_mask,
+ BTC_SET_ACT_UPDATE_RAMASK,
BTC_SET_ACT_SEND_MIMO_PS,
/* BT Coex related */
BTC_SET_ACT_CTRL_BT_INFO,
@@ -366,6 +379,7 @@
typedef void (*bfp_btc_w4)(void *btc_context, u32 reg_addr, u32 data);
+typedef void (*bfp_btc_local_reg_w1)(void *btc_context, u32 reg_addr, u8 data);
typedef void (*bfp_btc_wr_1byte_bit_mask)(void *btc_context, u32 reg_addr,
u8 bit_mask, u8 data);
@@ -388,6 +402,9 @@
typedef bool (*bfp_btc_set)(void *btcoexist, u8 set_type, void *in_buf);
+typedef void (*bfp_btc_set_bt_reg)(void *btc_context, u8 reg_type, u32 offset,
+ u32 value);
+
typedef void (*bfp_btc_disp_dbg_msg)(void *btcoexist, u8 disp_type);
struct btc_bt_info {
@@ -459,6 +476,7 @@
bool hid_only;
bool pan_exist;
bool pan_only;
+ bool slave_role;
};
enum btc_antenna_pos {
@@ -492,6 +510,7 @@
bfp_btc_w2 btc_write_2byte;
bfp_btc_r4 btc_read_4byte;
bfp_btc_w4 btc_write_4byte;
+ bfp_btc_local_reg_w1 btc_write_local_reg_1byte;
bfp_btc_set_bb_reg btc_set_bb_reg;
bfp_btc_get_bb_reg btc_get_bb_reg;
@@ -505,6 +524,8 @@
bfp_btc_get btc_get;
bfp_btc_set btc_set;
+
+ bfp_btc_set_bt_reg btc_set_bt_reg;
};
bool halbtc_is_wifi_uplink(struct rtl_priv *adapter);
diff --git a/drivers/net/wireless/realtek/rtlwifi/regd.c b/drivers/net/wireless/realtek/rtlwifi/regd.c
index 558c31b..1bf3eb2 100644
--- a/drivers/net/wireless/realtek/rtlwifi/regd.c
+++ b/drivers/net/wireless/realtek/rtlwifi/regd.c
@@ -435,7 +435,7 @@
channel_plan_to_country_code(rtlpriv->efuse.channel_plan);
RT_TRACE(rtlpriv, COMP_REGD, DBG_DMESG,
- "rtl: EEPROM regdomain: 0x%0x conuntry code: %d\n",
+ "rtl: EEPROM regdomain: 0x%0x country code: %d\n",
rtlpriv->efuse.channel_plan, rtlpriv->regd.country_code);
if (rtlpriv->regd.country_code >= COUNTRY_CODE_MAX) {
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8188ee/trx.c b/drivers/net/wireless/realtek/rtlwifi/rtl8188ee/trx.c
index 09c908d..dd3e12b 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8188ee/trx.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8188ee/trx.c
@@ -444,10 +444,10 @@
rx_status->flag |= RX_FLAG_FAILED_FCS_CRC;
if (status->rx_is40Mhzpacket)
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
if (status->is_ht)
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
rx_status->flag |= RX_FLAG_MACTIME_START;
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8192ce/trx.c b/drivers/net/wireless/realtek/rtlwifi/rtl8192ce/trx.c
index 3616ba2..94a4b39 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8192ce/trx.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8192ce/trx.c
@@ -369,10 +369,10 @@
rx_status->flag |= RX_FLAG_FAILED_FCS_CRC;
if (stats->rx_is40Mhzpacket)
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
if (stats->is_ht)
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
rx_status->flag |= RX_FLAG_MACTIME_START;
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8192cu/trx.c b/drivers/net/wireless/realtek/rtlwifi/rtl8192cu/trx.c
index 1611e42..41422e4 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8192cu/trx.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8192cu/trx.c
@@ -329,9 +329,9 @@
if (!GET_RX_DESC_SWDEC(pdesc))
rx_status->flag |= RX_FLAG_DECRYPTED;
if (GET_RX_DESC_BW(pdesc))
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
if (GET_RX_DESC_RX_HT(pdesc))
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
rx_status->flag |= RX_FLAG_MACTIME_START;
if (stats->decrypted)
rx_status->flag |= RX_FLAG_DECRYPTED;
@@ -398,9 +398,9 @@
if (!GET_RX_DESC_SWDEC(rxdesc))
rx_status->flag |= RX_FLAG_DECRYPTED;
if (GET_RX_DESC_BW(rxdesc))
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
if (GET_RX_DESC_RX_HT(rxdesc))
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
/* Data rate */
rx_status->rate_idx = rtlwifi_rate_mapping(hw, stats.is_ht,
false, stats.rate);
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8192de/trx.c b/drivers/net/wireless/realtek/rtlwifi/rtl8192de/trx.c
index 5c9c874..86019f6 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8192de/trx.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8192de/trx.c
@@ -503,9 +503,9 @@
if (!GET_RX_DESC_SWDEC(pdesc))
rx_status->flag |= RX_FLAG_DECRYPTED;
if (GET_RX_DESC_BW(pdesc))
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
if (GET_RX_DESC_RXHT(pdesc))
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
rx_status->flag |= RX_FLAG_MACTIME_START;
if (stats->decrypted)
rx_status->flag |= RX_FLAG_DECRYPTED;
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/fw.c b/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/fw.c
index 9fec345..1f42ce5 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/fw.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/fw.c
@@ -468,8 +468,10 @@
#define PSPOLL_PG 2
#define NULL_PG 3
#define PROBERSP_PG 4 /* ->5 */
+#define QOS_NULL_PG 6
+#define BT_QOS_NULL_PG 7
-#define TOTAL_RESERVED_PKT_LEN 768
+#define TOTAL_RESERVED_PKT_LEN 1024
static u8 reserved_page_packet[TOTAL_RESERVED_PKT_LEN] = {
/* page 0 beacon */
@@ -570,6 +572,42 @@
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x1A, 0x00, 0x28, 0x8C, 0x00, 0x12, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+
+ /* page 6 qos null data */
+ 0xC8, 0x01, 0x00, 0x00, 0x84, 0xC9, 0xB2, 0xA7,
+ 0xB3, 0x6E, 0x00, 0xE0, 0x4C, 0x02, 0x51, 0x02,
+ 0x84, 0xC9, 0xB2, 0xA7, 0xB3, 0x6E, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x1A, 0x00, 0x28, 0x8C, 0x00, 0x12, 0x00, 0x00,
+ 0x00, 0x00, 0x80, 0x00, 0x00, 0x01, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+
+ /* page 7 BT-qos null data */
+ 0xC8, 0x01, 0x00, 0x00, 0x84, 0xC9, 0xB2, 0xA7,
+ 0xB3, 0x6E, 0x00, 0xE0, 0x4C, 0x02, 0x51, 0x02,
+ 0x84, 0xC9, 0xB2, 0xA7, 0xB3, 0x6E, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
@@ -595,6 +633,8 @@
u8 *p_pspoll;
u8 *nullfunc;
u8 *p_probersp;
+ u8 *qosnull;
+ u8 *btqosnull;
/*---------------------------------------------------------
* (1) beacon
*---------------------------------------------------------
@@ -636,6 +676,28 @@
SET_H2CCMD_RSVDPAGE_LOC_PROBE_RSP(u1rsvdpageloc, PROBERSP_PG);
+ /*---------------------------------------------------------
+ * (5) QoS null data
+ *----------------------------------------------------------
+ */
+ qosnull = &reserved_page_packet[QOS_NULL_PG * 128];
+ SET_80211_HDR_ADDRESS1(qosnull, mac->bssid);
+ SET_80211_HDR_ADDRESS2(qosnull, mac->mac_addr);
+ SET_80211_HDR_ADDRESS3(qosnull, mac->bssid);
+
+ SET_H2CCMD_RSVDPAGE_LOC_QOS_NULL_DATA(u1rsvdpageloc, QOS_NULL_PG);
+
+ /*---------------------------------------------------------
+ * (6) BT QoS null data
+ *----------------------------------------------------------
+ */
+ btqosnull = &reserved_page_packet[BT_QOS_NULL_PG * 128];
+ SET_80211_HDR_ADDRESS1(btqosnull, mac->bssid);
+ SET_80211_HDR_ADDRESS2(btqosnull, mac->mac_addr);
+ SET_80211_HDR_ADDRESS3(btqosnull, mac->bssid);
+
+ SET_H2CCMD_RSVDPAGE_LOC_BT_QOS_NULL_DATA(u1rsvdpageloc, BT_QOS_NULL_PG);
+
totalpacketlen = TOTAL_RESERVED_PKT_LEN;
RT_PRINT_DATA(rtlpriv, COMP_CMD, DBG_LOUD ,
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/fw.h b/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/fw.h
index 72da3f9..af82719 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/fw.h
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/fw.h
@@ -165,6 +165,10 @@
SET_BITS_TO_LE_1BYTE((__ph2ccmd)+1, 0, 8, __val)
#define SET_H2CCMD_RSVDPAGE_LOC_NULL_DATA(__ph2ccmd, __val) \
SET_BITS_TO_LE_1BYTE((__ph2ccmd)+2, 0, 8, __val)
+#define SET_H2CCMD_RSVDPAGE_LOC_QOS_NULL_DATA(__ph2ccmd, __val) \
+ SET_BITS_TO_LE_1BYTE((__ph2ccmd) + 3, 0, 8, __val)
+#define SET_H2CCMD_RSVDPAGE_LOC_BT_QOS_NULL_DATA(__ph2ccmd, __val) \
+ SET_BITS_TO_LE_1BYTE((__ph2ccmd) + 4, 0, 8, __val)
/* _MEDIA_STATUS_RPT_PARM_CMD1 */
#define SET_H2CCMD_MSRRPT_PARM_OPMODE(__cmd, __val) \
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/hw.c b/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/hw.c
index 56ca7f5..6f5098a 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/hw.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/hw.c
@@ -699,9 +699,9 @@
u8 txpktbuf_bndy;
u8 u8tmp, testcnt = 0;
- txpktbuf_bndy = 0xFA;
+ txpktbuf_bndy = 0xF7;
- rtl_write_dword(rtlpriv, REG_RQPN, 0x80E90808);
+ rtl_write_dword(rtlpriv, REG_RQPN, 0x80E60808);
rtl_write_byte(rtlpriv, REG_TRXFF_BNDY, txpktbuf_bndy);
rtl_write_word(rtlpriv, REG_TRXFF_BNDY + 2, 0x3d00 - 1);
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/trx.c b/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/trx.c
index 07440e9..b1864bb 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/trx.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8192ee/trx.c
@@ -394,10 +394,10 @@
rx_status->flag |= RX_FLAG_FAILED_FCS_CRC;
if (status->rx_is40Mhzpacket)
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
if (status->is_ht)
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
rx_status->flag |= RX_FLAG_MACTIME_START;
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8192se/trx.c b/drivers/net/wireless/realtek/rtlwifi/rtl8192se/trx.c
index 12cef01..a01dbd3 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8192se/trx.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8192se/trx.c
@@ -289,10 +289,10 @@
rx_status->flag |= RX_FLAG_FAILED_FCS_CRC;
if (stats->rx_is40Mhzpacket)
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
if (stats->is_ht)
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
rx_status->flag |= RX_FLAG_MACTIME_START;
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/trx.c b/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/trx.c
index c9838f5..f713c72 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/trx.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8723ae/trx.c
@@ -317,10 +317,10 @@
rx_status->flag |= RX_FLAG_FAILED_FCS_CRC;
if (status->rx_is40Mhzpacket)
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
if (status->is_ht)
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
rx_status->flag |= RX_FLAG_MACTIME_START;
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8723be/fw.c b/drivers/net/wireless/realtek/rtlwifi/rtl8723be/fw.c
index c7ee9ba..4fc839b 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8723be/fw.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8723be/fw.c
@@ -284,8 +284,10 @@
#define PSPOLL_PG 2
#define NULL_PG 3
#define PROBERSP_PG 4 /* ->5 */
+#define QOS_NULL_PG 6
+#define BT_QOS_NULL_PG 7
-#define TOTAL_RESERVED_PKT_LEN 768
+#define TOTAL_RESERVED_PKT_LEN 1024 /* can be up to 1280 (tx_bndy=245) */
static u8 reserved_page_packet[TOTAL_RESERVED_PKT_LEN] = {
/* page 0 beacon */
@@ -390,11 +392,48 @@
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x1A, 0x00, 0x28, 0x8C, 0x00, 0x12, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+
+ /* page 6 qos null data */
+ 0xC8, 0x01, 0x00, 0x00, 0x84, 0xC9, 0xB2, 0xA7,
+ 0xB3, 0x6E, 0x00, 0xE0, 0x4C, 0x02, 0x51, 0x02,
+ 0x84, 0xC9, 0xB2, 0xA7, 0xB3, 0x6E, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x1A, 0x00, 0x28, 0x8C, 0x00, 0x12, 0x00, 0x00,
+ 0x00, 0x00, 0x80, 0x00, 0x00, 0x01, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+
+ /* page 7 BT-qos null data */
+ 0xC8, 0x01, 0x00, 0x00, 0x84, 0xC9, 0xB2, 0xA7,
+ 0xB3, 0x6E, 0x00, 0xE0, 0x4C, 0x02, 0x51, 0x02,
+ 0x84, 0xC9, 0xB2, 0xA7, 0xB3, 0x6E, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+
};
void rtl8723be_set_fw_rsvdpagepkt(struct ieee80211_hw *hw,
@@ -413,6 +452,8 @@
u8 *p_pspoll;
u8 *nullfunc;
u8 *p_probersp;
+ u8 *qosnull;
+ u8 *btqosnull;
/*---------------------------------------------------------
* (1) beacon
*---------------------------------------------------------
@@ -454,6 +495,28 @@
SET_H2CCMD_RSVDPAGE_LOC_PROBE_RSP(u1rsvdpageloc, PROBERSP_PG);
+ /*---------------------------------------------------------
+ * (5) QoS Null
+ *---------------------------------------------------------
+ */
+ qosnull = &reserved_page_packet[QOS_NULL_PG * 128];
+ SET_80211_HDR_ADDRESS1(qosnull, mac->bssid);
+ SET_80211_HDR_ADDRESS2(qosnull, mac->mac_addr);
+ SET_80211_HDR_ADDRESS3(qosnull, mac->bssid);
+
+ SET_H2CCMD_RSVDPAGE_LOC_QOS_NULL_DATA(u1rsvdpageloc, QOS_NULL_PG);
+
+ /*---------------------------------------------------------
+ * (5) QoS Null
+ *---------------------------------------------------------
+ */
+ btqosnull = &reserved_page_packet[BT_QOS_NULL_PG * 128];
+ SET_80211_HDR_ADDRESS1(btqosnull, mac->bssid);
+ SET_80211_HDR_ADDRESS2(btqosnull, mac->mac_addr);
+ SET_80211_HDR_ADDRESS3(btqosnull, mac->bssid);
+
+ SET_H2CCMD_RSVDPAGE_LOC_BT_QOS_NULL_DATA(u1rsvdpageloc, BT_QOS_NULL_PG);
+
totalpacketlen = TOTAL_RESERVED_PKT_LEN;
RT_PRINT_DATA(rtlpriv, COMP_CMD, DBG_LOUD,
@@ -461,7 +524,7 @@
&reserved_page_packet[0], totalpacketlen);
RT_PRINT_DATA(rtlpriv, COMP_CMD, DBG_DMESG,
"rtl8723be_set_fw_rsvdpagepkt(): HW_VAR_SET_TX_CMD: ALL\n",
- u1rsvdpageloc, 3);
+ u1rsvdpageloc, sizeof(u1rsvdpageloc));
skb = dev_alloc_skb(totalpacketlen);
memcpy((u8 *)skb_put(skb, totalpacketlen),
@@ -476,7 +539,7 @@
RT_TRACE(rtlpriv, COMP_POWER, DBG_LOUD,
"Set RSVD page location to Fw.\n");
RT_PRINT_DATA(rtlpriv, COMP_CMD, DBG_DMESG, "H2C_RSVDPAGE:\n",
- u1rsvdpageloc, 3);
+ u1rsvdpageloc, sizeof(u1rsvdpageloc));
rtl8723be_fill_h2c_cmd(hw, H2C_8723B_RSVDPAGE,
sizeof(u1rsvdpageloc), u1rsvdpageloc);
} else
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8723be/fw.h b/drivers/net/wireless/realtek/rtlwifi/rtl8723be/fw.h
index c652fa1..2482b3b 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8723be/fw.h
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8723be/fw.h
@@ -139,6 +139,10 @@
SET_BITS_TO_LE_1BYTE((__ph2ccmd)+1, 0, 8, __val)
#define SET_H2CCMD_RSVDPAGE_LOC_NULL_DATA(__ph2ccmd, __val) \
SET_BITS_TO_LE_1BYTE((__ph2ccmd)+2, 0, 8, __val)
+#define SET_H2CCMD_RSVDPAGE_LOC_QOS_NULL_DATA(__ph2ccmd, __val) \
+ SET_BITS_TO_LE_1BYTE((__ph2ccmd) + 3, 0, 8, __val)
+#define SET_H2CCMD_RSVDPAGE_LOC_BT_QOS_NULL_DATA(__ph2ccmd, __val) \
+ SET_BITS_TO_LE_1BYTE((__ph2ccmd) + 4, 0, 8, __val)
void rtl8723be_fill_h2c_cmd(struct ieee80211_hw *hw, u8 element_id,
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8723be/sw.c b/drivers/net/wireless/realtek/rtlwifi/rtl8723be/sw.c
index 92dbfa8..8c0ac96 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8723be/sw.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8723be/sw.c
@@ -91,7 +91,7 @@
struct rtl_priv *rtlpriv = rtl_priv(hw);
struct rtl_pci *rtlpci = rtl_pcidev(rtl_pcipriv(hw));
struct rtl_mac *mac = rtl_mac(rtl_priv(hw));
- char *fw_name = "rtlwifi/rtl8723befw.bin";
+ char *fw_name = "rtlwifi/rtl8723befw_36.bin";
rtl8723be_bt_reg_init(hw);
rtlpriv->btcoexist.btc_ops = rtl_btc_get_ops_pointer();
@@ -187,8 +187,16 @@
rtlpriv->io.dev, GFP_KERNEL, hw,
rtl_fw_cb);
if (err) {
- pr_err("Failed to request firmware!\n");
- return 1;
+ /* Failed to get firmware. Check if old version available */
+ fw_name = "rtlwifi/rtl8723befw.bin";
+ pr_info("Using firmware %s\n", fw_name);
+ err = request_firmware_nowait(THIS_MODULE, 1, fw_name,
+ rtlpriv->io.dev, GFP_KERNEL, hw,
+ rtl_fw_cb);
+ if (err) {
+ pr_err("Failed to request firmware!\n");
+ return 1;
+ }
}
return 0;
}
@@ -384,6 +392,7 @@
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Realtek 8723BE 802.11n PCI wireless");
MODULE_FIRMWARE("rtlwifi/rtl8723befw.bin");
+MODULE_FIRMWARE("rtlwifi/rtl8723befw_36.bin");
module_param_named(swenc, rtl8723be_mod_params.sw_crypto, bool, 0444);
module_param_named(debug_level, rtl8723be_mod_params.debug_level, int, 0644);
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8723be/trx.c b/drivers/net/wireless/realtek/rtlwifi/rtl8723be/trx.c
index 6f65003..3c6ce99 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8723be/trx.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8723be/trx.c
@@ -373,10 +373,10 @@
rx_status->flag |= RX_FLAG_FAILED_FCS_CRC;
if (status->rx_is40Mhzpacket)
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
if (status->is_ht)
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
rx_status->flag |= RX_FLAG_MACTIME_START;
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/fw.c b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/fw.c
index a504dfa..7335010 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/fw.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/fw.c
@@ -678,12 +678,13 @@
#define PSPOLL_PG 1
#define NULL_PG 2
#define QOSNULL_PG 3
-#define ARPRESP_PG 4
-#define REMOTE_PG 5
-#define GTKEXT_PG 6
+#define BT_QOSNULL_PG 4
+#define ARPRESP_PG 5
+#define REMOTE_PG 6
+#define GTKEXT_PG 7
-#define TOTAL_RESERVED_PKT_LEN_8812 3584
-#define TOTAL_RESERVED_PKT_LEN_8821 1792
+#define TOTAL_RESERVED_PKT_LEN_8812 4096
+#define TOTAL_RESERVED_PKT_LEN_8821 2048
static u8 reserved_page_packet_8821[TOTAL_RESERVED_PKT_LEN_8821] = {
/* page 0: beacon */
@@ -813,13 +814,46 @@
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x1A, 0x00, 0x28, 0x8C, 0x00, 0x12, 0x00, 0x00,
+ 0x00, 0x00, 0x80, 0x00, 0x00, 0x01, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ /* page 4: BT qos null data */
+ 0xC8, 0x01, 0x00, 0x00, 0x84, 0xC9, 0xB2, 0xA7,
+ 0xB3, 0x6E, 0x00, 0xE0, 0x4C, 0x02, 0x51, 0x02,
+ 0x84, 0xC9, 0xB2, 0xA7, 0xB3, 0x6E, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x3C, 0x00, 0x28, 0x8C, 0x00, 0x12, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
- /* page 4~6 is for wowlan */
- /* page 4: ARP resp */
+ /* page 5~7 is for wowlan */
+ /* page 5: ARP resp */
0x08, 0x01, 0x00, 0x00, 0x84, 0xC9, 0xB2, 0xA7,
0xB3, 0x6E, 0x00, 0xE0, 0x4C, 0x02, 0x51, 0x02,
0x84, 0xC9, 0xB2, 0xA7, 0xB3, 0x6E, 0x00, 0x00,
@@ -852,7 +886,7 @@
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
- /* page 5: H2C_REMOTE_WAKE_CTRL_INFO */
+ /* page 6: H2C_REMOTE_WAKE_CTRL_INFO */
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
@@ -885,7 +919,7 @@
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
- /* page 6: Rsvd GTK extend memory (zero memory) */
+ /* page 7: Rsvd GTK extend memory (zero memory) */
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
@@ -1176,13 +1210,78 @@
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x1A, 0x00, 0x28, 0x8C, 0x00, 0x12, 0x00, 0x00,
+ 0x00, 0x00, 0x80, 0x00, 0x00, 0x01, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ /* page 4: BT Qos null data */
+ 0xC8, 0x01, 0x00, 0x00, 0x84, 0xC9, 0xB2, 0xA7,
+ 0xB3, 0x6E, 0x00, 0xE0, 0x4C, 0x02, 0x51, 0x02,
+ 0x84, 0xC9, 0xB2, 0xA7, 0xB3, 0x6E, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x3C, 0x00, 0x28, 0x8C, 0x00, 0x12, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
- /* page 4~6 is for wowlan */
- /* page 4: ARP resp */
+ /* page 5~7 is for wowlan */
+ /* page 5: ARP resp */
0x08, 0x01, 0x00, 0x00, 0x84, 0xC9, 0xB2, 0xA7,
0xB3, 0x6E, 0x00, 0xE0, 0x4C, 0x02, 0x51, 0x02,
0x84, 0xC9, 0xB2, 0xA7, 0xB3, 0x6E, 0x00, 0x00,
@@ -1247,7 +1346,7 @@
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
- /* page 5: H2C_REMOTE_WAKE_CTRL_INFO */
+ /* page 6: H2C_REMOTE_WAKE_CTRL_INFO */
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
@@ -1312,7 +1411,7 @@
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
- /* page 6: Rsvd GTK extend memory (zero memory) */
+ /* page 7: Rsvd GTK extend memory (zero memory) */
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
@@ -1394,6 +1493,7 @@
u8 *p_pspoll;
u8 *nullfunc;
u8 *qosnull;
+ u8 *btqosnull;
u8 *arpresp;
/*---------------------------------------------------------
@@ -1441,12 +1541,23 @@
SET_H2CCMD_RSVDPAGE_LOC_QOS_NULL_DATA(u1RsvdPageLoc, QOSNULL_PG);
+ /*---------------------------------------------------------
+ * (5) BT Qos null data
+ *----------------------------------------------------------
+ */
+ btqosnull = &reserved_page_packet_8812[BT_QOSNULL_PG * 512];
+ SET_80211_HDR_ADDRESS1(btqosnull, mac->bssid);
+ SET_80211_HDR_ADDRESS2(btqosnull, mac->mac_addr);
+ SET_80211_HDR_ADDRESS3(btqosnull, mac->bssid);
+
+ SET_H2CCMD_RSVDPAGE_LOC_BT_QOS_NULL_DATA(u1RsvdPageLoc, BT_QOSNULL_PG);
+
if (!dl_whole_packets) {
- totalpacketlen = 512 * (QOSNULL_PG + 1) - 40;
+ totalpacketlen = 512 * (BT_QOSNULL_PG + 1) - 40;
goto out;
}
/*---------------------------------------------------------
- * (5) ARP Resp
+ * (6) ARP Resp
*----------------------------------------------------------
*/
arpresp = &reserved_page_packet_8812[ARPRESP_PG * 512];
@@ -1457,14 +1568,14 @@
SET_8821AE_H2CCMD_AOAC_RSVDPAGE_LOC_ARP_RSP(u1RsvdPageLoc2, ARPRESP_PG);
/*---------------------------------------------------------
- * (6) Remote Wake Ctrl
+ * (7) Remote Wake Ctrl
*----------------------------------------------------------
*/
SET_8821AE_H2CCMD_AOAC_RSVDPAGE_LOC_REMOTE_WAKE_CTRL_INFO(u1RsvdPageLoc2,
REMOTE_PG);
/*---------------------------------------------------------
- * (7) GTK Ext Memory
+ * (8) GTK Ext Memory
*----------------------------------------------------------
*/
SET_8821AE_H2CCMD_AOAC_RSVDPAGE_LOC_GTK_EXT_MEM(u1RsvdPageLoc2, GTKEXT_PG);
@@ -1518,6 +1629,7 @@
u8 *p_pspoll;
u8 *nullfunc;
u8 *qosnull;
+ u8 *btqosnull;
u8 *arpresp;
/*---------------------------------------------------------
@@ -1565,12 +1677,23 @@
SET_H2CCMD_RSVDPAGE_LOC_QOS_NULL_DATA(u1RsvdPageLoc, QOSNULL_PG);
+ /*---------------------------------------------------------
+ * (5) Qos null data
+ *----------------------------------------------------------
+ */
+ btqosnull = &reserved_page_packet_8821[BT_QOSNULL_PG * 256];
+ SET_80211_HDR_ADDRESS1(btqosnull, mac->bssid);
+ SET_80211_HDR_ADDRESS2(btqosnull, mac->mac_addr);
+ SET_80211_HDR_ADDRESS3(btqosnull, mac->bssid);
+
+ SET_H2CCMD_RSVDPAGE_LOC_BT_QOS_NULL_DATA(u1RsvdPageLoc, BT_QOSNULL_PG);
+
if (!dl_whole_packets) {
- totalpacketlen = 256 * (QOSNULL_PG + 1) - 40;
+ totalpacketlen = 256 * (BT_QOSNULL_PG + 1) - 40;
goto out;
}
/*---------------------------------------------------------
- * (5) ARP Resp
+ * (6) ARP Resp
*----------------------------------------------------------
*/
arpresp = &reserved_page_packet_8821[ARPRESP_PG * 256];
@@ -1581,14 +1704,14 @@
SET_8821AE_H2CCMD_AOAC_RSVDPAGE_LOC_ARP_RSP(u1RsvdPageLoc2, ARPRESP_PG);
/*---------------------------------------------------------
- * (6) Remote Wake Ctrl
+ * (7) Remote Wake Ctrl
*----------------------------------------------------------
*/
SET_8821AE_H2CCMD_AOAC_RSVDPAGE_LOC_REMOTE_WAKE_CTRL_INFO(u1RsvdPageLoc2,
REMOTE_PG);
/*---------------------------------------------------------
- * (7) GTK Ext Memory
+ * (8) GTK Ext Memory
*----------------------------------------------------------
*/
SET_8821AE_H2CCMD_AOAC_RSVDPAGE_LOC_GTK_EXT_MEM(u1RsvdPageLoc2, GTKEXT_PG);
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/fw.h b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/fw.h
index 90a98ed..98d871a 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/fw.h
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/fw.h
@@ -229,6 +229,8 @@
SET_BITS_TO_LE_1BYTE((__ph2ccmd)+2, 0, 8, __val)
#define SET_H2CCMD_RSVDPAGE_LOC_QOS_NULL_DATA(__ph2ccmd, __val) \
SET_BITS_TO_LE_1BYTE((__ph2ccmd)+3, 0, 8, __val)
+#define SET_H2CCMD_RSVDPAGE_LOC_BT_QOS_NULL_DATA(__ph2ccmd, __val) \
+ SET_BITS_TO_LE_1BYTE((__ph2ccmd) + 4, 0, 8, __val)
/* _MEDIA_STATUS_RPT_PARM_CMD1 */
#define SET_H2CCMD_MSRRPT_PARM_OPMODE(__cmd, __value) \
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/hw.c b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/hw.c
index 363d2f2..2bc6bac 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/hw.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/hw.c
@@ -842,12 +842,8 @@
bool status;
maxpage = 255;
- txpktbuf_bndy = 0xF8;
- rqpn = 0x80e70808;
- if (rtlpriv->rtlhal.hw_type == HARDWARE_TYPE_RTL8812AE) {
- txpktbuf_bndy = 0xFA;
- rqpn = 0x80e90808;
- }
+ txpktbuf_bndy = 0xF7;
+ rqpn = 0x80e60808;
rtl_write_byte(rtlpriv, REG_TRXFF_BNDY, txpktbuf_bndy);
rtl_write_word(rtlpriv, REG_TRXFF_BNDY + 2, MAX_RX_DMA_BUFFER_SIZE - 1);
@@ -2115,7 +2111,7 @@
break;
default:
RT_TRACE(rtlpriv, COMP_INIT, DBG_LOUD,
- "Chip Version ID: Unknow (0x%X)\n", version);
+ "Chip Version ID: Unknown (0x%X)\n", version);
break;
}
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/phy.c b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/phy.c
index 8da874c..aa3ccc7 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/phy.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/phy.c
@@ -358,6 +358,107 @@
return rtl8821ae_phy_rf6052_config(hw);
}
+static void _rtl8812ae_phy_set_rfe_reg_24g(struct ieee80211_hw *hw)
+{
+ struct rtl_priv *rtlpriv = rtl_priv(hw);
+ struct rtl_hal *rtlhal = rtl_hal(rtl_priv(hw));
+ u8 tmp;
+
+ switch (rtlhal->rfe_type) {
+ case 3:
+ rtl_set_bbreg(hw, RA_RFE_PINMUX, BMASKDWORD, 0x54337770);
+ rtl_set_bbreg(hw, RB_RFE_PINMUX, BMASKDWORD, 0x54337770);
+ rtl_set_bbreg(hw, RA_RFE_INV, BMASKRFEINV, 0x010);
+ rtl_set_bbreg(hw, RB_RFE_INV, BMASKRFEINV, 0x010);
+ rtl_set_bbreg(hw, 0x900, 0x00000303, 0x1);
+ break;
+ case 4:
+ rtl_set_bbreg(hw, RA_RFE_PINMUX, BMASKDWORD, 0x77777777);
+ rtl_set_bbreg(hw, RB_RFE_PINMUX, BMASKDWORD, 0x77777777);
+ rtl_set_bbreg(hw, RA_RFE_INV, BMASKRFEINV, 0x001);
+ rtl_set_bbreg(hw, RB_RFE_INV, BMASKRFEINV, 0x001);
+ break;
+ case 5:
+ rtl_write_byte(rtlpriv, RA_RFE_PINMUX + 2, 0x77);
+ rtl_set_bbreg(hw, RB_RFE_PINMUX, BMASKDWORD, 0x77777777);
+ tmp = rtl_read_byte(rtlpriv, RA_RFE_INV + 3);
+ rtl_write_byte(rtlpriv, RA_RFE_INV + 3, tmp & ~0x1);
+ rtl_set_bbreg(hw, RB_RFE_INV, BMASKRFEINV, 0x000);
+ break;
+ case 1:
+ if (rtlpriv->btcoexist.bt_coexistence) {
+ rtl_set_bbreg(hw, RA_RFE_PINMUX, 0xffffff, 0x777777);
+ rtl_set_bbreg(hw, RB_RFE_PINMUX, BMASKDWORD,
+ 0x77777777);
+ rtl_set_bbreg(hw, RA_RFE_INV, 0x33f00000, 0x000);
+ rtl_set_bbreg(hw, RB_RFE_INV, BMASKRFEINV, 0x000);
+ break;
+ }
+ case 0:
+ case 2:
+ default:
+ rtl_set_bbreg(hw, RA_RFE_PINMUX, BMASKDWORD, 0x77777777);
+ rtl_set_bbreg(hw, RB_RFE_PINMUX, BMASKDWORD, 0x77777777);
+ rtl_set_bbreg(hw, RA_RFE_INV, BMASKRFEINV, 0x000);
+ rtl_set_bbreg(hw, RB_RFE_INV, BMASKRFEINV, 0x000);
+ break;
+ }
+}
+
+static void _rtl8812ae_phy_set_rfe_reg_5g(struct ieee80211_hw *hw)
+{
+ struct rtl_priv *rtlpriv = rtl_priv(hw);
+ struct rtl_hal *rtlhal = rtl_hal(rtl_priv(hw));
+ u8 tmp;
+
+ switch (rtlhal->rfe_type) {
+ case 0:
+ rtl_set_bbreg(hw, RA_RFE_PINMUX, BMASKDWORD, 0x77337717);
+ rtl_set_bbreg(hw, RB_RFE_PINMUX, BMASKDWORD, 0x77337717);
+ rtl_set_bbreg(hw, RA_RFE_INV, BMASKRFEINV, 0x010);
+ rtl_set_bbreg(hw, RB_RFE_INV, BMASKRFEINV, 0x010);
+ break;
+ case 1:
+ if (rtlpriv->btcoexist.bt_coexistence) {
+ rtl_set_bbreg(hw, RA_RFE_PINMUX, 0xffffff, 0x337717);
+ rtl_set_bbreg(hw, RB_RFE_PINMUX, BMASKDWORD,
+ 0x77337717);
+ rtl_set_bbreg(hw, RA_RFE_INV, 0x33f00000, 0x000);
+ rtl_set_bbreg(hw, RB_RFE_INV, BMASKRFEINV, 0x000);
+ } else {
+ rtl_set_bbreg(hw, RA_RFE_PINMUX, BMASKDWORD,
+ 0x77337717);
+ rtl_set_bbreg(hw, RB_RFE_PINMUX, BMASKDWORD,
+ 0x77337717);
+ rtl_set_bbreg(hw, RA_RFE_INV, BMASKRFEINV, 0x000);
+ rtl_set_bbreg(hw, RB_RFE_INV, BMASKRFEINV, 0x000);
+ }
+ break;
+ case 3:
+ rtl_set_bbreg(hw, RA_RFE_PINMUX, BMASKDWORD, 0x54337717);
+ rtl_set_bbreg(hw, RB_RFE_PINMUX, BMASKDWORD, 0x54337717);
+ rtl_set_bbreg(hw, RA_RFE_INV, BMASKRFEINV, 0x010);
+ rtl_set_bbreg(hw, RB_RFE_INV, BMASKRFEINV, 0x010);
+ rtl_set_bbreg(hw, 0x900, 0x00000303, 0x1);
+ break;
+ case 5:
+ rtl_write_byte(rtlpriv, RA_RFE_PINMUX + 2, 0x33);
+ rtl_set_bbreg(hw, RB_RFE_PINMUX, BMASKDWORD, 0x77337777);
+ tmp = rtl_read_byte(rtlpriv, RA_RFE_INV + 3);
+ rtl_write_byte(rtlpriv, RA_RFE_INV + 3, tmp | 0x1);
+ rtl_set_bbreg(hw, RB_RFE_INV, BMASKRFEINV, 0x010);
+ break;
+ case 2:
+ case 4:
+ default:
+ rtl_set_bbreg(hw, RA_RFE_PINMUX, BMASKDWORD, 0x77337777);
+ rtl_set_bbreg(hw, RB_RFE_PINMUX, BMASKDWORD, 0x77337777);
+ rtl_set_bbreg(hw, RA_RFE_INV, BMASKRFEINV, 0x010);
+ rtl_set_bbreg(hw, RB_RFE_INV, BMASKRFEINV, 0x010);
+ break;
+ }
+}
+
u32 phy_get_tx_swing_8812A(struct ieee80211_hw *hw, u8 band,
u8 rf_path)
{
@@ -552,14 +653,9 @@
/* 0x82C[1:0] = 2b'00 */
rtl_set_bbreg(hw, 0x82c, 0x3, 0);
}
- if (rtlhal->hw_type == HARDWARE_TYPE_RTL8812AE) {
- rtl_set_bbreg(hw, RA_RFE_PINMUX, BMASKDWORD,
- 0x77777777);
- rtl_set_bbreg(hw, RB_RFE_PINMUX, BMASKDWORD,
- 0x77777777);
- rtl_set_bbreg(hw, RA_RFE_INV, 0x3ff00000, 0x000);
- rtl_set_bbreg(hw, RB_RFE_INV, 0x3ff00000, 0x000);
- }
+
+ if (rtlhal->hw_type == HARDWARE_TYPE_RTL8812AE)
+ _rtl8812ae_phy_set_rfe_reg_24g(hw);
rtl_set_bbreg(hw, RTXPATH, 0xf0, 0x1);
rtl_set_bbreg(hw, RCCK_RX, 0x0f000000, 0x1);
@@ -614,14 +710,8 @@
/* 0x82C[1:0] = 2'b00 */
rtl_set_bbreg(hw, 0x82c, 0x3, 1);
- if (rtlhal->hw_type == HARDWARE_TYPE_RTL8812AE) {
- rtl_set_bbreg(hw, RA_RFE_PINMUX, BMASKDWORD,
- 0x77337777);
- rtl_set_bbreg(hw, RB_RFE_PINMUX, BMASKDWORD,
- 0x77337777);
- rtl_set_bbreg(hw, RA_RFE_INV, 0x3ff00000, 0x010);
- rtl_set_bbreg(hw, RB_RFE_INV, 0x3ff00000, 0x010);
- }
+ if (rtlhal->hw_type == HARDWARE_TYPE_RTL8812AE)
+ _rtl8812ae_phy_set_rfe_reg_5g(hw);
rtl_set_bbreg(hw, RTXPATH, 0xf0, 0);
rtl_set_bbreg(hw, RCCK_RX, 0x0f000000, 0xf);
@@ -660,6 +750,88 @@
return;
}
+static bool _rtl8821ae_check_positive(struct ieee80211_hw *hw,
+ const u32 condition1,
+ const u32 condition2)
+{
+ struct rtl_priv *rtlpriv = rtl_priv(hw);
+ struct rtl_hal *rtlhal = rtl_hal(rtlpriv);
+ u32 cut_ver = ((rtlhal->version & CHIP_VER_RTL_MASK)
+ >> CHIP_VER_RTL_SHIFT);
+ u32 intf = (rtlhal->interface == INTF_USB ? BIT(1) : BIT(0));
+
+ u8 board_type = ((rtlhal->board_type & BIT(4)) >> 4) << 0 | /* _GLNA */
+ ((rtlhal->board_type & BIT(3)) >> 3) << 1 | /* _GPA */
+ ((rtlhal->board_type & BIT(7)) >> 7) << 2 | /* _ALNA */
+ ((rtlhal->board_type & BIT(6)) >> 6) << 3 | /* _APA */
+ ((rtlhal->board_type & BIT(2)) >> 2) << 4; /* _BT */
+
+ u32 cond1 = condition1, cond2 = condition2;
+ u32 driver1 = cut_ver << 24 | /* CUT ver */
+ 0 << 20 | /* interface 2/2 */
+ 0x04 << 16 | /* platform */
+ rtlhal->package_type << 12 |
+ intf << 8 | /* interface 1/2 */
+ board_type;
+
+ u32 driver2 = rtlhal->type_glna << 0 |
+ rtlhal->type_gpa << 8 |
+ rtlhal->type_alna << 16 |
+ rtlhal->type_apa << 24;
+
+ RT_TRACE(rtlpriv, COMP_INIT, DBG_TRACE,
+ "===> [8812A] CheckPositive (cond1, cond2) = (0x%X 0x%X)\n",
+ cond1, cond2);
+ RT_TRACE(rtlpriv, COMP_INIT, DBG_TRACE,
+ "===> [8812A] CheckPositive (driver1, driver2) = (0x%X 0x%X)\n",
+ driver1, driver2);
+
+ RT_TRACE(rtlpriv, COMP_INIT, DBG_TRACE,
+ " (Platform, Interface) = (0x%X, 0x%X)\n", 0x04, intf);
+ RT_TRACE(rtlpriv, COMP_INIT, DBG_TRACE,
+ " (Board, Package) = (0x%X, 0x%X)\n",
+ rtlhal->board_type, rtlhal->package_type);
+
+ /*============== Value Defined Check ===============*/
+ /*QFN Type [15:12] and Cut Version [27:24] need to do value check*/
+
+ if (((cond1 & 0x0000F000) != 0) && ((cond1 & 0x0000F000) !=
+ (driver1 & 0x0000F000)))
+ return false;
+ if (((cond1 & 0x0F000000) != 0) && ((cond1 & 0x0F000000) !=
+ (driver1 & 0x0F000000)))
+ return false;
+
+ /*=============== Bit Defined Check ================*/
+ /* We don't care [31:28] */
+
+ cond1 &= 0x00FF0FFF;
+ driver1 &= 0x00FF0FFF;
+
+ if ((cond1 & driver1) == cond1) {
+ u32 mask = 0;
+
+ if ((cond1 & 0x0F) == 0) /* BoardType is DONTCARE*/
+ return true;
+
+ if ((cond1 & BIT(0)) != 0) /*GLNA*/
+ mask |= 0x000000FF;
+ if ((cond1 & BIT(1)) != 0) /*GPA*/
+ mask |= 0x0000FF00;
+ if ((cond1 & BIT(2)) != 0) /*ALNA*/
+ mask |= 0x00FF0000;
+ if ((cond1 & BIT(3)) != 0) /*APA*/
+ mask |= 0xFF000000;
+
+ /* BoardType of each RF path is matched*/
+ if ((cond2 & mask) == (driver2 & mask))
+ return true;
+ else
+ return false;
+ } else
+ return false;
+}
+
static bool _rtl8821ae_check_condition(struct ieee80211_hw *hw,
const u32 condition)
{
@@ -1695,55 +1867,78 @@
return true;
}
+static bool
+__rtl8821ae_phy_config_with_headerfile(struct ieee80211_hw *hw,
+ u32 *array_table, u16 arraylen,
+ void (*set_reg)(struct ieee80211_hw *hw,
+ u32 regaddr, u32 data))
+{
+ #define COND_ELSE 2
+ #define COND_ENDIF 3
+
+ int i = 0;
+ u8 cond;
+ bool matched = true, skipped = false;
+
+ while ((i + 1) < arraylen) {
+ u32 v1 = array_table[i];
+ u32 v2 = array_table[i + 1];
+
+ if (v1 & (BIT(31) | BIT(30))) {/*positive & negative condition*/
+ if (v1 & BIT(31)) {/* positive condition*/
+ cond = (u8)((v1 & (BIT(29) | BIT(28))) >> 28);
+ if (cond == COND_ENDIF) {/*end*/
+ matched = true;
+ skipped = false;
+ } else if (cond == COND_ELSE) /*else*/
+ matched = skipped ? false : true;
+ else {/*if , else if*/
+ if (skipped) {
+ matched = false;
+ } else {
+ if (_rtl8821ae_check_positive(
+ hw, v1, v2)) {
+ matched = true;
+ skipped = true;
+ } else {
+ matched = false;
+ skipped = false;
+ }
+ }
+ }
+ } else if (v1 & BIT(30)) { /*negative condition*/
+ /*do nothing*/
+ }
+ } else {
+ if (matched)
+ set_reg(hw, v1, v2);
+ }
+ i = i + 2;
+ }
+
+ return true;
+}
+
static bool _rtl8821ae_phy_config_mac_with_headerfile(struct ieee80211_hw *hw)
{
struct rtl_priv *rtlpriv = rtl_priv(hw);
struct rtl_hal *rtlhal = rtl_hal(rtlpriv);
- u32 i, v1, v2;
u32 arraylength;
u32 *ptrarray;
RT_TRACE(rtlpriv, COMP_INIT, DBG_TRACE, "Read MAC_REG_Array\n");
if (rtlhal->hw_type == HARDWARE_TYPE_RTL8821AE) {
- arraylength = RTL8821AEMAC_1T_ARRAYLEN;
+ arraylength = RTL8821AE_MAC_1T_ARRAYLEN;
ptrarray = RTL8821AE_MAC_REG_ARRAY;
} else {
- arraylength = RTL8812AEMAC_1T_ARRAYLEN;
+ arraylength = RTL8812AE_MAC_1T_ARRAYLEN;
ptrarray = RTL8812AE_MAC_REG_ARRAY;
}
RT_TRACE(rtlpriv, COMP_INIT, DBG_LOUD,
"Img: MAC_REG_ARRAY LEN %d\n", arraylength);
- for (i = 0; i < arraylength; i += 2) {
- v1 = ptrarray[i];
- v2 = (u8)ptrarray[i + 1];
- if (v1 < 0xCDCDCDCD) {
- rtl_write_byte(rtlpriv, v1, (u8)v2);
- continue;
- } else {
- if (!_rtl8821ae_check_condition(hw, v1)) {
- /*Discard the following (offset, data) pairs*/
- READ_NEXT_PAIR(ptrarray, v1, v2, i);
- while (v2 != 0xDEAD &&
- v2 != 0xCDEF &&
- v2 != 0xCDCD && i < arraylength - 2) {
- READ_NEXT_PAIR(ptrarray, v1, v2, i);
- }
- i -= 2; /* prevent from for-loop += 2*/
- } else {/*Configure matched pairs and skip to end of if-else.*/
- READ_NEXT_PAIR(ptrarray, v1, v2, i);
- while (v2 != 0xDEAD &&
- v2 != 0xCDEF &&
- v2 != 0xCDCD && i < arraylength - 2) {
- rtl_write_byte(rtlpriv, v1, v2);
- READ_NEXT_PAIR(ptrarray, v1, v2, i);
- }
- while (v2 != 0xDEAD && i < arraylength - 2)
- READ_NEXT_PAIR(ptrarray, v1, v2, i);
- }
- }
- }
- return true;
+ return __rtl8821ae_phy_config_with_headerfile(hw,
+ ptrarray, arraylength, rtl_write_byte_with_val32);
}
static bool _rtl8821ae_phy_config_bb_with_headerfile(struct ieee80211_hw *hw,
@@ -1751,111 +1946,33 @@
{
struct rtl_priv *rtlpriv = rtl_priv(hw);
struct rtl_hal *rtlhal = rtl_hal(rtlpriv);
- int i;
u32 *array_table;
u16 arraylen;
- u32 v1 = 0, v2 = 0;
if (configtype == BASEBAND_CONFIG_PHY_REG) {
if (rtlhal->hw_type == HARDWARE_TYPE_RTL8812AE) {
- arraylen = RTL8812AEPHY_REG_1TARRAYLEN;
+ arraylen = RTL8812AE_PHY_REG_1TARRAYLEN;
array_table = RTL8812AE_PHY_REG_ARRAY;
} else {
- arraylen = RTL8821AEPHY_REG_1TARRAYLEN;
+ arraylen = RTL8821AE_PHY_REG_1TARRAYLEN;
array_table = RTL8821AE_PHY_REG_ARRAY;
}
- for (i = 0; i < arraylen; i += 2) {
- v1 = array_table[i];
- v2 = array_table[i + 1];
- if (v1 < 0xCDCDCDCD) {
- _rtl8821ae_config_bb_reg(hw, v1, v2);
- continue;
- } else {/*This line is the start line of branch.*/
- if (!_rtl8821ae_check_condition(hw, v1)) {
- /*Discard the following (offset, data) pairs*/
- READ_NEXT_PAIR(array_table, v1, v2, i);
- while (v2 != 0xDEAD &&
- v2 != 0xCDEF &&
- v2 != 0xCDCD &&
- i < arraylen - 2) {
- READ_NEXT_PAIR(array_table, v1,
- v2, i);
- }
-
- i -= 2; /* prevent from for-loop += 2*/
- } else {/*Configure matched pairs and skip to end of if-else.*/
- READ_NEXT_PAIR(array_table, v1, v2, i);
- while (v2 != 0xDEAD &&
- v2 != 0xCDEF &&
- v2 != 0xCDCD &&
- i < arraylen - 2) {
- _rtl8821ae_config_bb_reg(hw, v1,
- v2);
- READ_NEXT_PAIR(array_table, v1,
- v2, i);
- }
-
- while (v2 != 0xDEAD &&
- i < arraylen - 2) {
- READ_NEXT_PAIR(array_table, v1,
- v2, i);
- }
- }
- }
- }
+ return __rtl8821ae_phy_config_with_headerfile(hw,
+ array_table, arraylen,
+ _rtl8821ae_config_bb_reg);
} else if (configtype == BASEBAND_CONFIG_AGC_TAB) {
if (rtlhal->hw_type == HARDWARE_TYPE_RTL8812AE) {
- arraylen = RTL8812AEAGCTAB_1TARRAYLEN;
+ arraylen = RTL8812AE_AGC_TAB_1TARRAYLEN;
array_table = RTL8812AE_AGC_TAB_ARRAY;
} else {
- arraylen = RTL8821AEAGCTAB_1TARRAYLEN;
+ arraylen = RTL8821AE_AGC_TAB_1TARRAYLEN;
array_table = RTL8821AE_AGC_TAB_ARRAY;
}
- for (i = 0; i < arraylen; i = i + 2) {
- v1 = array_table[i];
- v2 = array_table[i+1];
- if (v1 < 0xCDCDCDCD) {
- rtl_set_bbreg(hw, v1, MASKDWORD, v2);
- udelay(1);
- continue;
- } else {/*This line is the start line of branch.*/
- if (!_rtl8821ae_check_condition(hw, v1)) {
- /*Discard the following (offset, data) pairs*/
- READ_NEXT_PAIR(array_table, v1, v2, i);
- while (v2 != 0xDEAD &&
- v2 != 0xCDEF &&
- v2 != 0xCDCD &&
- i < arraylen - 2) {
- READ_NEXT_PAIR(array_table, v1,
- v2, i);
- }
- i -= 2; /* prevent from for-loop += 2*/
- } else {/*Configure matched pairs and skip to end of if-else.*/
- READ_NEXT_PAIR(array_table, v1, v2, i);
- while (v2 != 0xDEAD &&
- v2 != 0xCDEF &&
- v2 != 0xCDCD &&
- i < arraylen - 2) {
- rtl_set_bbreg(hw, v1, MASKDWORD,
- v2);
- udelay(1);
- READ_NEXT_PAIR(array_table, v1,
- v2, i);
- }
-
- while (v2 != 0xDEAD &&
- i < arraylen - 2) {
- READ_NEXT_PAIR(array_table, v1,
- v2, i);
- }
- }
- RT_TRACE(rtlpriv, COMP_INIT, DBG_TRACE,
- "The agctab_array_table[0] is %x Rtl818EEPHY_REGArray[1] is %x\n",
- array_table[i], array_table[i + 1]);
- }
- }
+ return __rtl8821ae_phy_config_with_headerfile(hw,
+ array_table, arraylen,
+ rtl_set_bbreg_with_dwmask);
}
return true;
}
@@ -1913,10 +2030,10 @@
u32 v1, v2, v3, v4, v5, v6;
if (rtlhal->hw_type == HARDWARE_TYPE_RTL8812AE) {
- arraylen = RTL8812AEPHY_REG_ARRAY_PGLEN;
+ arraylen = RTL8812AE_PHY_REG_ARRAY_PGLEN;
array = RTL8812AE_PHY_REG_ARRAY_PG;
} else {
- arraylen = RTL8821AEPHY_REG_ARRAY_PGLEN;
+ arraylen = RTL8821AE_PHY_REG_ARRAY_PGLEN;
array = RTL8821AE_PHY_REG_ARRAY_PG;
}
@@ -1980,12 +2097,10 @@
bool rtl8812ae_phy_config_rf_with_headerfile(struct ieee80211_hw *hw,
enum radio_path rfpath)
{
- int i;
bool rtstatus = true;
u32 *radioa_array_table_a, *radioa_array_table_b;
u16 radioa_arraylen_a, radioa_arraylen_b;
struct rtl_priv *rtlpriv = rtl_priv(hw);
- u32 v1 = 0, v2 = 0;
radioa_arraylen_a = RTL8812AE_RADIOA_1TARRAYLEN;
radioa_array_table_a = RTL8812AE_RADIOA_ARRAY;
@@ -1997,69 +2112,14 @@
rtstatus = true;
switch (rfpath) {
case RF90_PATH_A:
- for (i = 0; i < radioa_arraylen_a; i = i + 2) {
- v1 = radioa_array_table_a[i];
- v2 = radioa_array_table_a[i+1];
- if (v1 < 0xcdcdcdcd) {
- _rtl8821ae_config_rf_radio_a(hw, v1, v2);
- continue;
- } else{/*This line is the start line of branch.*/
- if (!_rtl8821ae_check_condition(hw, v1)) {
- /*Discard the following (offset, data) pairs*/
- READ_NEXT_PAIR(radioa_array_table_a, v1, v2, i);
- while (v2 != 0xDEAD &&
- v2 != 0xCDEF &&
- v2 != 0xCDCD && i < radioa_arraylen_a-2)
- READ_NEXT_PAIR(radioa_array_table_a, v1, v2, i);
-
- i -= 2; /* prevent from for-loop += 2*/
- } else {/*Configure matched pairs and skip to end of if-else.*/
- READ_NEXT_PAIR(radioa_array_table_a, v1, v2, i);
- while (v2 != 0xDEAD &&
- v2 != 0xCDEF &&
- v2 != 0xCDCD && i < radioa_arraylen_a - 2) {
- _rtl8821ae_config_rf_radio_a(hw, v1, v2);
- READ_NEXT_PAIR(radioa_array_table_a, v1, v2, i);
- }
-
- while (v2 != 0xDEAD && i < radioa_arraylen_a-2)
- READ_NEXT_PAIR(radioa_array_table_a, v1, v2, i);
-
- }
- }
- }
+ return __rtl8821ae_phy_config_with_headerfile(hw,
+ radioa_array_table_a, radioa_arraylen_a,
+ _rtl8821ae_config_rf_radio_a);
break;
case RF90_PATH_B:
- for (i = 0; i < radioa_arraylen_b; i = i + 2) {
- v1 = radioa_array_table_b[i];
- v2 = radioa_array_table_b[i+1];
- if (v1 < 0xcdcdcdcd) {
- _rtl8821ae_config_rf_radio_b(hw, v1, v2);
- continue;
- } else{/*This line is the start line of branch.*/
- if (!_rtl8821ae_check_condition(hw, v1)) {
- /*Discard the following (offset, data) pairs*/
- READ_NEXT_PAIR(radioa_array_table_b, v1, v2, i);
- while (v2 != 0xDEAD &&
- v2 != 0xCDEF &&
- v2 != 0xCDCD && i < radioa_arraylen_b-2)
- READ_NEXT_PAIR(radioa_array_table_b, v1, v2, i);
-
- i -= 2; /* prevent from for-loop += 2*/
- } else {/*Configure matched pairs and skip to end of if-else.*/
- READ_NEXT_PAIR(radioa_array_table_b, v1, v2, i);
- while (v2 != 0xDEAD &&
- v2 != 0xCDEF &&
- v2 != 0xCDCD && i < radioa_arraylen_b-2) {
- _rtl8821ae_config_rf_radio_b(hw, v1, v2);
- READ_NEXT_PAIR(radioa_array_table_b, v1, v2, i);
- }
-
- while (v2 != 0xDEAD && i < radioa_arraylen_b-2)
- READ_NEXT_PAIR(radioa_array_table_b, v1, v2, i);
- }
- }
- }
+ return __rtl8821ae_phy_config_with_headerfile(hw,
+ radioa_array_table_b, radioa_arraylen_b,
+ _rtl8821ae_config_rf_radio_b);
break;
case RF90_PATH_C:
case RF90_PATH_D:
@@ -2072,21 +2132,10 @@
bool rtl8821ae_phy_config_rf_with_headerfile(struct ieee80211_hw *hw,
enum radio_path rfpath)
{
- #define READ_NEXT_RF_PAIR(v1, v2, i) \
- do { \
- i += 2; \
- v1 = radioa_array_table[i]; \
- v2 = radioa_array_table[i+1]; \
- } \
- while (0)
-
- int i;
bool rtstatus = true;
u32 *radioa_array_table;
u16 radioa_arraylen;
struct rtl_priv *rtlpriv = rtl_priv(hw);
- /* struct rtl_hal *rtlhal = rtl_hal(rtl_priv(hw)); */
- u32 v1 = 0, v2 = 0;
radioa_arraylen = RTL8821AE_RADIOA_1TARRAYLEN;
radioa_array_table = RTL8821AE_RADIOA_ARRAY;
@@ -2096,35 +2145,9 @@
rtstatus = true;
switch (rfpath) {
case RF90_PATH_A:
- for (i = 0; i < radioa_arraylen; i = i + 2) {
- v1 = radioa_array_table[i];
- v2 = radioa_array_table[i+1];
- if (v1 < 0xcdcdcdcd)
- _rtl8821ae_config_rf_radio_a(hw, v1, v2);
- else{/*This line is the start line of branch.*/
- if (!_rtl8821ae_check_condition(hw, v1)) {
- /*Discard the following (offset, data) pairs*/
- READ_NEXT_RF_PAIR(v1, v2, i);
- while (v2 != 0xDEAD &&
- v2 != 0xCDEF &&
- v2 != 0xCDCD && i < radioa_arraylen - 2)
- READ_NEXT_RF_PAIR(v1, v2, i);
-
- i -= 2; /* prevent from for-loop += 2*/
- } else {/*Configure matched pairs and skip to end of if-else.*/
- READ_NEXT_RF_PAIR(v1, v2, i);
- while (v2 != 0xDEAD &&
- v2 != 0xCDEF &&
- v2 != 0xCDCD && i < radioa_arraylen - 2) {
- _rtl8821ae_config_rf_radio_a(hw, v1, v2);
- READ_NEXT_RF_PAIR(v1, v2, i);
- }
-
- while (v2 != 0xDEAD && i < radioa_arraylen - 2)
- READ_NEXT_RF_PAIR(v1, v2, i);
- }
- }
- }
+ return __rtl8821ae_phy_config_with_headerfile(hw,
+ radioa_array_table, radioa_arraylen,
+ _rtl8821ae_config_rf_radio_a);
break;
case RF90_PATH_B:
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/reg.h b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/reg.h
index 1d6110f..ed69dbe 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/reg.h
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/reg.h
@@ -2424,6 +2424,7 @@
#define BMASKH4BITS 0xf0000000
#define BMASKOFDM_D 0xffc00000
#define BMASKCCK 0x3f3f3f3f
+#define BMASKRFEINV 0x3ff00000
#define BRFREGOFFSETMASK 0xfffff
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/sw.c b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/sw.c
index 77cf3b2..abaf34c 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/sw.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/sw.c
@@ -203,7 +203,7 @@
fw_name = "rtlwifi/rtl8812aefw.bin";
wowlan_fw_name = "rtlwifi/rtl8812aefw_wowlan.bin";
} else {
- fw_name = "rtlwifi/rtl8821aefw.bin";
+ fw_name = "rtlwifi/rtl8821aefw_29.bin";
wowlan_fw_name = "rtlwifi/rtl8821aefw_wowlan.bin";
}
@@ -214,8 +214,16 @@
rtlpriv->io.dev, GFP_KERNEL, hw,
rtl_fw_cb);
if (err) {
- pr_err("Failed to request normal firmware!\n");
- return 1;
+ /* Failed to get firmware. Check if old version available */
+ fw_name = "rtlwifi/rtl8821aefw.bin";
+ pr_info("Using firmware %s\n", fw_name);
+ err = request_firmware_nowait(THIS_MODULE, 1, fw_name,
+ rtlpriv->io.dev, GFP_KERNEL, hw,
+ rtl_fw_cb);
+ if (err) {
+ pr_err("Failed to request normal firmware!\n");
+ return 1;
+ }
}
/*load wowlan firmware*/
pr_info("Using firmware %s\n", wowlan_fw_name);
@@ -428,6 +436,7 @@
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Realtek 8821ae 802.11ac PCI wireless");
MODULE_FIRMWARE("rtlwifi/rtl8821aefw.bin");
+MODULE_FIRMWARE("rtlwifi/rtl8821aefw_29.bin");
module_param_named(swenc, rtl8821ae_mod_params.sw_crypto, bool, 0444);
module_param_named(debug_level, rtl8821ae_mod_params.debug_level, int, 0644);
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/table.c b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/table.c
index 62a0fb7..408c461 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/table.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/table.c
@@ -38,7 +38,7 @@
0x824, 0x00030FE0,
0x828, 0x00000000,
0x82C, 0x002083DD,
- 0x830, 0x2AAA6C86,
+ 0x830, 0x2EAAEEB8,
0x834, 0x0037A706,
0x838, 0x06C89B44,
0x83C, 0x0000095B,
@@ -68,7 +68,7 @@
0x8BC, 0x4CA520A3,
0x8C0, 0x27F00020,
0x8C4, 0x00000000,
- 0x8C8, 0x00013169,
+ 0x8C8, 0x00012D69,
0x8CC, 0x08248492,
0x8D0, 0x0000B800,
0x8DC, 0x00000000,
@@ -76,13 +76,7 @@
0x8D8, 0x290B5612,
0x8F8, 0x400002C0,
0x8FC, 0x00000000,
- 0xFF0F07D8, 0xABCD,
0x900, 0x00000701,
- 0xFF0F07D0, 0xCDEF,
- 0x900, 0x00000701,
- 0xCDCDCDCD, 0xCDCD,
- 0x900, 0x00000700,
- 0xFF0F07D8, 0xDEAD,
0x90C, 0x00000000,
0x910, 0x0000FC00,
0x914, 0x00000404,
@@ -120,7 +114,7 @@
0x9D4, 0x00000000,
0x9D8, 0x00000000,
0x9DC, 0x00000000,
- 0x9E4, 0x00000002,
+ 0x9E4, 0x00000003,
0x9E8, 0x000002D5,
0xA00, 0x00D047C8,
0xA04, 0x01FF000C,
@@ -189,7 +183,21 @@
0xC5C, 0x00000058,
0xC60, 0x34344443,
0xC64, 0x07003333,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
0xC68, 0x59791979,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
+ 0xC68, 0x59791979,
+ 0x90000002, 0x00000000, 0x40000000, 0x00000000,
+ 0xC68, 0x59791979,
+ 0x90000004, 0x00000000, 0x40000000, 0x00000000,
+ 0xC68, 0x59791979,
+ 0x90000001, 0x00000000, 0x40000000, 0x00000000,
+ 0xC68, 0x59791979,
+ 0x90000001, 0x00000005, 0x40000000, 0x00000000,
+ 0xC68, 0x59791979,
+ 0xA0000000, 0x00000000,
+ 0xC68, 0x59799979,
+ 0xB0000000, 0x00000000,
0xC6C, 0x59795979,
0xC70, 0x19795979,
0xC74, 0x19795979,
@@ -203,19 +211,7 @@
0xCA0, 0x00000029,
0xCA4, 0x08040201,
0xCA8, 0x80402010,
- 0xFF0F0740, 0xABCD,
- 0xCB0, 0x77547717,
- 0xFF0F01C0, 0xCDEF,
- 0xCB0, 0x77547717,
- 0xFF0F02C0, 0xCDEF,
- 0xCB0, 0x77547717,
- 0xFF0F07D8, 0xCDEF,
- 0xCB0, 0x54547710,
- 0xFF0F07D0, 0xCDEF,
- 0xCB0, 0x54547710,
- 0xCDCDCDCD, 0xCDCD,
0xCB0, 0x77547777,
- 0xFF0F0740, 0xDEAD,
0xCB4, 0x00000077,
0xCB8, 0x00508242,
0xE00, 0x00000007,
@@ -257,23 +253,14 @@
0xEA0, 0x00000029,
0xEA4, 0x08040201,
0xEA8, 0x80402010,
- 0xFF0F0740, 0xABCD,
- 0xEB0, 0x77547717,
- 0xFF0F01C0, 0xCDEF,
- 0xEB0, 0x77547717,
- 0xFF0F02C0, 0xCDEF,
- 0xEB0, 0x77547717,
- 0xFF0F07D8, 0xCDEF,
- 0xEB0, 0x54547710,
- 0xFF0F07D0, 0xCDEF,
- 0xEB0, 0x54547710,
- 0xCDCDCDCD, 0xCDCD,
0xEB0, 0x77547777,
- 0xFF0F0740, 0xDEAD,
0xEB4, 0x00000077,
0xEB8, 0x00508242,
};
+u32 RTL8812AE_PHY_REG_1TARRAYLEN =
+ sizeof(RTL8812AE_PHY_REG_ARRAY) / sizeof(u32);
+
u32 RTL8821AE_PHY_REG_ARRAY[] = {
0x800, 0x0020D090,
0x804, 0x080112E0,
@@ -449,6 +436,9 @@
0xCB8, 0x00508240,
};
+u32 RTL8821AE_PHY_REG_1TARRAYLEN =
+ sizeof(RTL8821AE_PHY_REG_ARRAY) / sizeof(u32);
+
u32 RTL8812AE_PHY_REG_ARRAY_PG[] = {
0, 0, 0, 0x00000c20, 0xffffffff, 0x34363840,
0, 0, 0, 0x00000c24, 0xffffffff, 0x42424444,
@@ -498,6 +488,9 @@
1, 1, 1, 0x00000e4c, 0xffffffff, 0x22242628
};
+u32 RTL8812AE_PHY_REG_ARRAY_PGLEN =
+ sizeof(RTL8812AE_PHY_REG_ARRAY_PG) / sizeof(u32);
+
u32 RTL8821AE_PHY_REG_ARRAY_PG[] = {
0, 0, 0, 0x00000c20, 0xffffffff, 0x32343638,
0, 0, 0, 0x00000c24, 0xffffffff, 0x36363838,
@@ -516,6 +509,9 @@
1, 0, 0, 0x00000c44, 0x0000ffff, 0x00002022
};
+u32 RTL8821AE_PHY_REG_ARRAY_PGLEN =
+ sizeof(RTL8821AE_PHY_REG_ARRAY_PG) / sizeof(u32);
+
u32 RTL8812AE_RADIOA_ARRAY[] = {
0x000, 0x00010000,
0x018, 0x0001712A,
@@ -523,26 +519,25 @@
0x066, 0x00040000,
0x01E, 0x00080000,
0x089, 0x00000080,
- 0xFF0F0740, 0xABCD,
- 0x086, 0x00014B38,
- 0xFF0F02C0, 0xCDEF,
- 0x086, 0x00014B38,
- 0xFF0F01C0, 0xCDEF,
- 0x086, 0x00014B38,
- 0xFF0F07D8, 0xCDEF,
+ 0x80000001, 0x00000000, 0x40000000, 0x00000000,
0x086, 0x00014B3A,
- 0xFF0F07D0, 0xCDEF,
+ 0x90000001, 0x00000005, 0x40000000, 0x00000000,
0x086, 0x00014B3A,
- 0xCDCDCDCD, 0xCDCD,
+ 0xA0000000, 0x00000000,
0x086, 0x00014B38,
- 0xFF0F0740, 0xDEAD,
+ 0xB0000000, 0x00000000,
+ 0x80000004, 0x00000000, 0x40000000, 0x00000000,
+ 0x08B, 0x00080180,
+ 0xA0000000, 0x00000000,
+ 0x08B, 0x00087180,
+ 0xB0000000, 0x00000000,
0x0B1, 0x0001FC1A,
0x0B3, 0x000F0810,
0x0B4, 0x0001A78D,
0x0BA, 0x00086180,
0x018, 0x00000006,
0x0EF, 0x00002000,
- 0xFF0F07D8, 0xABCD,
+ 0x80000001, 0x00000000, 0x40000000, 0x00000000,
0x03B, 0x0003F218,
0x03B, 0x00030A58,
0x03B, 0x0002FA58,
@@ -550,7 +545,7 @@
0x03B, 0x0001FA50,
0x03B, 0x00010248,
0x03B, 0x00008240,
- 0xFF0F07D0, 0xCDEF,
+ 0x90000001, 0x00000005, 0x40000000, 0x00000000,
0x03B, 0x0003F218,
0x03B, 0x00030A58,
0x03B, 0x0002FA58,
@@ -558,7 +553,7 @@
0x03B, 0x0001FA50,
0x03B, 0x00010248,
0x03B, 0x00008240,
- 0xCDCDCDCD, 0xCDCD,
+ 0xA0000000, 0x00000000,
0x03B, 0x00038A58,
0x03B, 0x00037A58,
0x03B, 0x0002A590,
@@ -566,9 +561,9 @@
0x03B, 0x00018248,
0x03B, 0x00010240,
0x03B, 0x00008240,
- 0xFF0F07D8, 0xDEAD,
+ 0xB0000000, 0x00000000,
0x0EF, 0x00000100,
- 0xFF0F07D8, 0xABCD,
+ 0x80000002, 0x00000000, 0x40000000, 0x00000000,
0x034, 0x0000A4EE,
0x034, 0x00009076,
0x034, 0x00008073,
@@ -580,7 +575,7 @@
0x034, 0x00002028,
0x034, 0x00001025,
0x034, 0x00000022,
- 0xCDCDCDCD, 0xCDCD,
+ 0xA0000000, 0x00000000,
0x034, 0x0000ADF4,
0x034, 0x00009DF1,
0x034, 0x00008DEE,
@@ -592,636 +587,7 @@
0x034, 0x000024E7,
0x034, 0x0000146B,
0x034, 0x0000006D,
- 0xFF0F07D8, 0xDEAD,
- 0x0EF, 0x00000000,
- 0x0EF, 0x000020A2,
- 0x0DF, 0x00000080,
- 0x035, 0x00000192,
- 0x035, 0x00008192,
- 0x035, 0x00010192,
- 0x036, 0x00000024,
- 0x036, 0x00008024,
- 0x036, 0x00010024,
- 0x036, 0x00018024,
- 0x0EF, 0x00000000,
- 0x051, 0x00000C21,
- 0x052, 0x000006D9,
- 0x053, 0x000FC649,
- 0x054, 0x0000017E,
- 0x0EF, 0x00000002,
- 0x008, 0x00008400,
- 0x018, 0x0001712A,
- 0x0EF, 0x00001000,
- 0x03A, 0x00000080,
- 0x03B, 0x0003A02C,
- 0x03C, 0x00004000,
- 0x03A, 0x00000400,
- 0x03B, 0x0003202C,
- 0x03C, 0x00010000,
- 0x03A, 0x000000A0,
- 0x03B, 0x0002B064,
- 0x03C, 0x00004000,
- 0x03A, 0x000000D8,
- 0x03B, 0x00023070,
- 0x03C, 0x00004000,
- 0x03A, 0x00000468,
- 0x03B, 0x0001B870,
- 0x03C, 0x00010000,
- 0x03A, 0x00000098,
- 0x03B, 0x00012085,
- 0x03C, 0x000E4000,
- 0x03A, 0x00000418,
- 0x03B, 0x0000A080,
- 0x03C, 0x000F0000,
- 0x03A, 0x00000418,
- 0x03B, 0x00002080,
- 0x03C, 0x00010000,
- 0x03A, 0x00000080,
- 0x03B, 0x0007A02C,
- 0x03C, 0x00004000,
- 0x03A, 0x00000400,
- 0x03B, 0x0007202C,
- 0x03C, 0x00010000,
- 0x03A, 0x000000A0,
- 0x03B, 0x0006B064,
- 0x03C, 0x00004000,
- 0x03A, 0x000000D8,
- 0x03B, 0x00023070,
- 0x03C, 0x00004000,
- 0x03A, 0x00000468,
- 0x03B, 0x0005B870,
- 0x03C, 0x00010000,
- 0x03A, 0x00000098,
- 0x03B, 0x00052085,
- 0x03C, 0x000E4000,
- 0x03A, 0x00000418,
- 0x03B, 0x0004A080,
- 0x03C, 0x000F0000,
- 0x03A, 0x00000418,
- 0x03B, 0x00042080,
- 0x03C, 0x00010000,
- 0x03A, 0x00000080,
- 0x03B, 0x000BA02C,
- 0x03C, 0x00004000,
- 0x03A, 0x00000400,
- 0x03B, 0x000B202C,
- 0x03C, 0x00010000,
- 0x03A, 0x000000A0,
- 0x03B, 0x000AB064,
- 0x03C, 0x00004000,
- 0x03A, 0x000000D8,
- 0x03B, 0x000A3070,
- 0x03C, 0x00004000,
- 0x03A, 0x00000468,
- 0x03B, 0x0009B870,
- 0x03C, 0x00010000,
- 0x03A, 0x00000098,
- 0x03B, 0x00092085,
- 0x03C, 0x000E4000,
- 0x03A, 0x00000418,
- 0x03B, 0x0008A080,
- 0x03C, 0x000F0000,
- 0x03A, 0x00000418,
- 0x03B, 0x00082080,
- 0x03C, 0x00010000,
- 0x0EF, 0x00001100,
- 0xFF0F0740, 0xABCD,
- 0x034, 0x0004A0B2,
- 0x034, 0x000490AF,
- 0x034, 0x00048070,
- 0x034, 0x0004706D,
- 0x034, 0x00046050,
- 0x034, 0x0004504D,
- 0x034, 0x0004404A,
- 0x034, 0x00043047,
- 0x034, 0x0004200A,
- 0x034, 0x00041007,
- 0x034, 0x00040004,
- 0xFF0F02C0, 0xCDEF,
- 0x034, 0x0004A0B2,
- 0x034, 0x000490AF,
- 0x034, 0x00048070,
- 0x034, 0x0004706D,
- 0x034, 0x00046050,
- 0x034, 0x0004504D,
- 0x034, 0x0004404A,
- 0x034, 0x00043047,
- 0x034, 0x0004200A,
- 0x034, 0x00041007,
- 0x034, 0x00040004,
- 0xFF0F01C0, 0xCDEF,
- 0x034, 0x0004A0B2,
- 0x034, 0x000490AF,
- 0x034, 0x00048070,
- 0x034, 0x0004706D,
- 0x034, 0x00046050,
- 0x034, 0x0004504D,
- 0x034, 0x0004404A,
- 0x034, 0x00043047,
- 0x034, 0x0004200A,
- 0x034, 0x00041007,
- 0x034, 0x00040004,
- 0xFF0F07D8, 0xCDEF,
- 0x034, 0x0004A0B2,
- 0x034, 0x000490AF,
- 0x034, 0x00048070,
- 0x034, 0x0004706D,
- 0x034, 0x00046050,
- 0x034, 0x0004504D,
- 0x034, 0x0004404A,
- 0x034, 0x00043047,
- 0x034, 0x0004200A,
- 0x034, 0x00041007,
- 0x034, 0x00040004,
- 0xFF0F07D0, 0xCDEF,
- 0x034, 0x0004A0B2,
- 0x034, 0x000490AF,
- 0x034, 0x00048070,
- 0x034, 0x0004706D,
- 0x034, 0x00046050,
- 0x034, 0x0004504D,
- 0x034, 0x0004404A,
- 0x034, 0x00043047,
- 0x034, 0x0004200A,
- 0x034, 0x00041007,
- 0x034, 0x00040004,
- 0xCDCDCDCD, 0xCDCD,
- 0x034, 0x0004ADF5,
- 0x034, 0x00049DF2,
- 0x034, 0x00048DEF,
- 0x034, 0x00047DEC,
- 0x034, 0x00046DE9,
- 0x034, 0x00045DC9,
- 0x034, 0x00044CE8,
- 0x034, 0x000438CA,
- 0x034, 0x00042889,
- 0x034, 0x0004184A,
- 0x034, 0x0004044A,
- 0xFF0F0740, 0xDEAD,
- 0xFF0F0740, 0xABCD,
- 0x034, 0x0002A0B2,
- 0x034, 0x000290AF,
- 0x034, 0x00028070,
- 0x034, 0x0002706D,
- 0x034, 0x00026050,
- 0x034, 0x0002504D,
- 0x034, 0x0002404A,
- 0x034, 0x00023047,
- 0x034, 0x0002200A,
- 0x034, 0x00021007,
- 0x034, 0x00020004,
- 0xFF0F02C0, 0xCDEF,
- 0x034, 0x0002A0B2,
- 0x034, 0x000290AF,
- 0x034, 0x00028070,
- 0x034, 0x0002706D,
- 0x034, 0x00026050,
- 0x034, 0x0002504D,
- 0x034, 0x0002404A,
- 0x034, 0x00023047,
- 0x034, 0x0002200A,
- 0x034, 0x00021007,
- 0x034, 0x00020004,
- 0xFF0F01C0, 0xCDEF,
- 0x034, 0x0002A0B2,
- 0x034, 0x000290AF,
- 0x034, 0x00028070,
- 0x034, 0x0002706D,
- 0x034, 0x00026050,
- 0x034, 0x0002504D,
- 0x034, 0x0002404A,
- 0x034, 0x00023047,
- 0x034, 0x0002200A,
- 0x034, 0x00021007,
- 0x034, 0x00020004,
- 0xFF0F07D8, 0xCDEF,
- 0x034, 0x0002A0B2,
- 0x034, 0x000290AF,
- 0x034, 0x00028070,
- 0x034, 0x0002706D,
- 0x034, 0x00026050,
- 0x034, 0x0002504D,
- 0x034, 0x0002404A,
- 0x034, 0x00023047,
- 0x034, 0x0002200A,
- 0x034, 0x00021007,
- 0x034, 0x00020004,
- 0xFF0F07D0, 0xCDEF,
- 0x034, 0x0002A0B2,
- 0x034, 0x000290AF,
- 0x034, 0x00028070,
- 0x034, 0x0002706D,
- 0x034, 0x00026050,
- 0x034, 0x0002504D,
- 0x034, 0x0002404A,
- 0x034, 0x00023047,
- 0x034, 0x0002200A,
- 0x034, 0x00021007,
- 0x034, 0x00020004,
- 0xCDCDCDCD, 0xCDCD,
- 0x034, 0x0002ADF5,
- 0x034, 0x00029DF2,
- 0x034, 0x00028DEF,
- 0x034, 0x00027DEC,
- 0x034, 0x00026DE9,
- 0x034, 0x00025DC9,
- 0x034, 0x00024CE8,
- 0x034, 0x000238CA,
- 0x034, 0x00022889,
- 0x034, 0x0002184A,
- 0x034, 0x0002044A,
- 0xFF0F0740, 0xDEAD,
- 0xFF0F0740, 0xABCD,
- 0x034, 0x0000A0B2,
- 0x034, 0x000090AF,
- 0x034, 0x00008070,
- 0x034, 0x0000706D,
- 0x034, 0x00006050,
- 0x034, 0x0000504D,
- 0x034, 0x0000404A,
- 0x034, 0x00003047,
- 0x034, 0x0000200A,
- 0x034, 0x00001007,
- 0x034, 0x00000004,
- 0xFF0F02C0, 0xCDEF,
- 0x034, 0x0000A0B2,
- 0x034, 0x000090AF,
- 0x034, 0x00008070,
- 0x034, 0x0000706D,
- 0x034, 0x00006050,
- 0x034, 0x0000504D,
- 0x034, 0x0000404A,
- 0x034, 0x00003047,
- 0x034, 0x0000200A,
- 0x034, 0x00001007,
- 0x034, 0x00000004,
- 0xFF0F01C0, 0xCDEF,
- 0x034, 0x0000A0B2,
- 0x034, 0x000090AF,
- 0x034, 0x00008070,
- 0x034, 0x0000706D,
- 0x034, 0x00006050,
- 0x034, 0x0000504D,
- 0x034, 0x0000404A,
- 0x034, 0x00003047,
- 0x034, 0x0000200A,
- 0x034, 0x00001007,
- 0x034, 0x00000004,
- 0xFF0F07D8, 0xCDEF,
- 0x034, 0x0000A0B2,
- 0x034, 0x000090AF,
- 0x034, 0x00008070,
- 0x034, 0x0000706D,
- 0x034, 0x00006050,
- 0x034, 0x0000504D,
- 0x034, 0x0000404A,
- 0x034, 0x00003047,
- 0x034, 0x0000200A,
- 0x034, 0x00001007,
- 0x034, 0x00000004,
- 0xFF0F07D0, 0xCDEF,
- 0x034, 0x0000A0B2,
- 0x034, 0x000090AF,
- 0x034, 0x00008070,
- 0x034, 0x0000706D,
- 0x034, 0x00006050,
- 0x034, 0x0000504D,
- 0x034, 0x0000404A,
- 0x034, 0x00003047,
- 0x034, 0x0000200A,
- 0x034, 0x00001007,
- 0x034, 0x00000004,
- 0xCDCDCDCD, 0xCDCD,
- 0x034, 0x0000AFF7,
- 0x034, 0x00009DF7,
- 0x034, 0x00008DF4,
- 0x034, 0x00007DF1,
- 0x034, 0x00006DEE,
- 0x034, 0x00005DCD,
- 0x034, 0x00004CEB,
- 0x034, 0x000038CC,
- 0x034, 0x0000288B,
- 0x034, 0x0000184C,
- 0x034, 0x0000044C,
- 0xFF0F0740, 0xDEAD,
- 0x0EF, 0x00000000,
- 0xFF0F0740, 0xABCD,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000040,
- 0x035, 0x000001D4,
- 0x035, 0x000081D4,
- 0x035, 0x000101D4,
- 0x035, 0x000201B4,
- 0x035, 0x000281B4,
- 0x035, 0x000301B4,
- 0x035, 0x000401B4,
- 0x035, 0x000481B4,
- 0x035, 0x000501B4,
- 0xFF0F02C0, 0xCDEF,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000040,
- 0x035, 0x000001D4,
- 0x035, 0x000081D4,
- 0x035, 0x000101D4,
- 0x035, 0x000201B4,
- 0x035, 0x000281B4,
- 0x035, 0x000301B4,
- 0x035, 0x000401B4,
- 0x035, 0x000481B4,
- 0x035, 0x000501B4,
- 0xFF0F01C0, 0xCDEF,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000040,
- 0x035, 0x000001D4,
- 0x035, 0x000081D4,
- 0x035, 0x000101D4,
- 0x035, 0x000201B4,
- 0x035, 0x000281B4,
- 0x035, 0x000301B4,
- 0x035, 0x000401B4,
- 0x035, 0x000481B4,
- 0x035, 0x000501B4,
- 0xFF0F07D8, 0xCDEF,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000040,
- 0x035, 0x000001D4,
- 0x035, 0x000081D4,
- 0x035, 0x000101D4,
- 0x035, 0x000201B4,
- 0x035, 0x000281B4,
- 0x035, 0x000301B4,
- 0x035, 0x000401B4,
- 0x035, 0x000481B4,
- 0x035, 0x000501B4,
- 0xFF0F07D0, 0xCDEF,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000040,
- 0x035, 0x000001D4,
- 0x035, 0x000081D4,
- 0x035, 0x000101D4,
- 0x035, 0x000201B4,
- 0x035, 0x000281B4,
- 0x035, 0x000301B4,
- 0x035, 0x000401B4,
- 0x035, 0x000481B4,
- 0x035, 0x000501B4,
- 0xCDCDCDCD, 0xCDCD,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000040,
- 0x035, 0x00000188,
- 0x035, 0x00008147,
- 0x035, 0x00010147,
- 0x035, 0x000201D7,
- 0x035, 0x000281D7,
- 0x035, 0x000301D7,
- 0x035, 0x000401D8,
- 0x035, 0x000481D8,
- 0x035, 0x000501D8,
- 0xFF0F0740, 0xDEAD,
- 0x0EF, 0x00000000,
- 0xFF0F0740, 0xABCD,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000010,
- 0x036, 0x00004BFB,
- 0x036, 0x0000CBFB,
- 0x036, 0x00014BFB,
- 0x036, 0x0001CBFB,
- 0x036, 0x00024F4B,
- 0x036, 0x0002CF4B,
- 0x036, 0x00034F4B,
- 0x036, 0x0003CF4B,
- 0x036, 0x00044F4B,
- 0x036, 0x0004CF4B,
- 0x036, 0x00054F4B,
- 0x036, 0x0005CF4B,
- 0xFF0F02C0, 0xCDEF,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000010,
- 0x036, 0x00004BFB,
- 0x036, 0x0000CBFB,
- 0x036, 0x00014BFB,
- 0x036, 0x0001CBFB,
- 0x036, 0x00024F4B,
- 0x036, 0x0002CF4B,
- 0x036, 0x00034F4B,
- 0x036, 0x0003CF4B,
- 0x036, 0x00044F4B,
- 0x036, 0x0004CF4B,
- 0x036, 0x00054F4B,
- 0x036, 0x0005CF4B,
- 0xFF0F01C0, 0xCDEF,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000010,
- 0x036, 0x00004BFB,
- 0x036, 0x0000CBFB,
- 0x036, 0x00014BFB,
- 0x036, 0x0001CBFB,
- 0x036, 0x00024F4B,
- 0x036, 0x0002CF4B,
- 0x036, 0x00034F4B,
- 0x036, 0x0003CF4B,
- 0x036, 0x00044F4B,
- 0x036, 0x0004CF4B,
- 0x036, 0x00054F4B,
- 0x036, 0x0005CF4B,
- 0xFF0F07D8, 0xCDEF,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000010,
- 0x036, 0x00004BFB,
- 0x036, 0x0000CBFB,
- 0x036, 0x00014BFB,
- 0x036, 0x0001CBFB,
- 0x036, 0x00024F4B,
- 0x036, 0x0002CF4B,
- 0x036, 0x00034F4B,
- 0x036, 0x0003CF4B,
- 0x036, 0x00044F4B,
- 0x036, 0x0004CF4B,
- 0x036, 0x00054F4B,
- 0x036, 0x0005CF4B,
- 0xFF0F07D0, 0xCDEF,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000010,
- 0x036, 0x00004BFB,
- 0x036, 0x0000CBFB,
- 0x036, 0x00014BFB,
- 0x036, 0x0001CBFB,
- 0x036, 0x00024F4B,
- 0x036, 0x0002CF4B,
- 0x036, 0x00034F4B,
- 0x036, 0x0003CF4B,
- 0x036, 0x00044F4B,
- 0x036, 0x0004CF4B,
- 0x036, 0x00054F4B,
- 0x036, 0x0005CF4B,
- 0xCDCDCDCD, 0xCDCD,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000010,
- 0x036, 0x00084EB4,
- 0x036, 0x0008CC35,
- 0x036, 0x00094C35,
- 0x036, 0x0009CC35,
- 0x036, 0x000A4935,
- 0x036, 0x000ACC35,
- 0x036, 0x000B4C35,
- 0x036, 0x000BCC35,
- 0x036, 0x000C4EB4,
- 0x036, 0x000CCEB5,
- 0x036, 0x000D4EB5,
- 0x036, 0x000DCEB5,
- 0xFF0F0740, 0xDEAD,
- 0x0EF, 0x00000000,
- 0x0EF, 0x00000008,
- 0xFF0F0740, 0xABCD,
- 0x03C, 0x000002CC,
- 0x03C, 0x00000522,
- 0x03C, 0x00000902,
- 0xFF0F02C0, 0xCDEF,
- 0x03C, 0x000002CC,
- 0x03C, 0x00000522,
- 0x03C, 0x00000902,
- 0xFF0F01C0, 0xCDEF,
- 0x03C, 0x000002CC,
- 0x03C, 0x00000522,
- 0x03C, 0x00000902,
- 0xFF0F07D8, 0xCDEF,
- 0x03C, 0x000002CC,
- 0x03C, 0x00000522,
- 0x03C, 0x00000902,
- 0xFF0F07D0, 0xCDEF,
- 0x03C, 0x000002CC,
- 0x03C, 0x00000522,
- 0x03C, 0x00000902,
- 0xCDCDCDCD, 0xCDCD,
- 0x03C, 0x000002A8,
- 0x03C, 0x000005A2,
- 0x03C, 0x00000880,
- 0xFF0F0740, 0xDEAD,
- 0x0EF, 0x00000000,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000002,
- 0x0DF, 0x00000080,
- 0x01F, 0x00040064,
- 0xFF0F0740, 0xABCD,
- 0x061, 0x000FDD43,
- 0x062, 0x00038F4B,
- 0x063, 0x00032117,
- 0x064, 0x000194AC,
- 0x065, 0x000931D1,
- 0xFF0F02C0, 0xCDEF,
- 0x061, 0x000FDD43,
- 0x062, 0x00038F4B,
- 0x063, 0x00032117,
- 0x064, 0x000194AC,
- 0x065, 0x000931D1,
- 0xFF0F01C0, 0xCDEF,
- 0x061, 0x000FDD43,
- 0x062, 0x00038F4B,
- 0x063, 0x00032117,
- 0x064, 0x000194AC,
- 0x065, 0x000931D1,
- 0xFF0F07D8, 0xCDEF,
- 0x061, 0x000FDD43,
- 0x062, 0x00038F4B,
- 0x063, 0x00032117,
- 0x064, 0x000194AC,
- 0x065, 0x000931D1,
- 0xFF0F07D0, 0xCDEF,
- 0x061, 0x000FDD43,
- 0x062, 0x00038F4B,
- 0x063, 0x00032117,
- 0x064, 0x000194AC,
- 0x065, 0x000931D1,
- 0xCDCDCDCD, 0xCDCD,
- 0x061, 0x000E5D53,
- 0x062, 0x00038FCD,
- 0x063, 0x000314EB,
- 0x064, 0x000196AC,
- 0x065, 0x000911D7,
- 0xFF0F0740, 0xDEAD,
- 0x008, 0x00008400,
- 0x01C, 0x000739D2,
- 0x0B4, 0x0001E78D,
- 0x018, 0x0001F12A,
- 0x0FE, 0x00000000,
- 0x0FE, 0x00000000,
- 0x0FE, 0x00000000,
- 0x0FE, 0x00000000,
- 0x0B4, 0x0001A78D,
- 0x018, 0x0001712A,
-
-};
-
-u32 RTL8812AE_RADIOB_ARRAY[] = {
- 0x056, 0x00051CF2,
- 0x066, 0x00040000,
- 0x089, 0x00000080,
- 0xFF0F0740, 0xABCD,
- 0x086, 0x00014B38,
- 0xFF0F01C0, 0xCDEF,
- 0x086, 0x00014B38,
- 0xFF0F02C0, 0xCDEF,
- 0x086, 0x00014B38,
- 0xFF0F07D8, 0xCDEF,
- 0x086, 0x00014B3A,
- 0xFF0F07D0, 0xCDEF,
- 0x086, 0x00014B3A,
- 0xCDCDCDCD, 0xCDCD,
- 0x086, 0x00014B38,
- 0xFF0F0740, 0xDEAD,
- 0x018, 0x00000006,
- 0x0EF, 0x00002000,
- 0xFF0F07D8, 0xABCD,
- 0x03B, 0x0003F218,
- 0x03B, 0x00030A58,
- 0x03B, 0x0002FA58,
- 0x03B, 0x00022590,
- 0x03B, 0x0001FA50,
- 0x03B, 0x00010248,
- 0x03B, 0x00008240,
- 0xFF0F07D0, 0xCDEF,
- 0x03B, 0x0003F218,
- 0x03B, 0x00030A58,
- 0x03B, 0x0002FA58,
- 0x03B, 0x00022590,
- 0x03B, 0x0001FA50,
- 0x03B, 0x00010248,
- 0x03B, 0x00008240,
- 0xCDCDCDCD, 0xCDCD,
- 0x03B, 0x00038A58,
- 0x03B, 0x00037A58,
- 0x03B, 0x0002A590,
- 0x03B, 0x00027A50,
- 0x03B, 0x00018248,
- 0x03B, 0x00010240,
- 0x03B, 0x00008240,
- 0xFF0F07D8, 0xDEAD,
- 0x0EF, 0x00000100,
- 0xFF0F07D8, 0xABCD,
- 0x034, 0x0000A4EE,
- 0x034, 0x00009076,
- 0x034, 0x00008073,
- 0x034, 0x00007070,
- 0x034, 0x0000606D,
- 0x034, 0x0000506A,
- 0x034, 0x00004049,
- 0x034, 0x00003046,
- 0x034, 0x00002028,
- 0x034, 0x00001025,
- 0x034, 0x00000022,
- 0xCDCDCDCD, 0xCDCD,
- 0x034, 0x0000ADF4,
- 0x034, 0x00009DF1,
- 0x034, 0x00008DEE,
- 0x034, 0x00007DEB,
- 0x034, 0x00006DE8,
- 0x034, 0x00005CEC,
- 0x034, 0x00004CE9,
- 0x034, 0x000034EA,
- 0x034, 0x000024E7,
- 0x034, 0x0000146B,
- 0x034, 0x0000006D,
- 0xFF0F07D8, 0xDEAD,
+ 0xB0000000, 0x00000000,
0x0EF, 0x00000000,
0x0EF, 0x000020A2,
0x0DF, 0x00000080,
@@ -1314,7 +680,7 @@
0x03B, 0x00082080,
0x03C, 0x00010000,
0x0EF, 0x00001100,
- 0xFF0F0740, 0xABCD,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
0x034, 0x0004A0B2,
0x034, 0x000490AF,
0x034, 0x00048070,
@@ -1326,68 +692,32 @@
0x034, 0x0004200A,
0x034, 0x00041007,
0x034, 0x00040004,
- 0xFF0F01C0, 0xCDEF,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
0x034, 0x0004A0B2,
0x034, 0x000490AF,
0x034, 0x00048070,
0x034, 0x0004706D,
- 0x034, 0x00046050,
- 0x034, 0x0004504D,
- 0x034, 0x0004404A,
- 0x034, 0x00043047,
- 0x034, 0x0004200A,
- 0x034, 0x00041007,
- 0x034, 0x00040004,
- 0xFF0F02C0, 0xCDEF,
- 0x034, 0x0004A0B2,
- 0x034, 0x000490AF,
- 0x034, 0x00048070,
- 0x034, 0x0004706D,
- 0x034, 0x00046050,
- 0x034, 0x0004504D,
- 0x034, 0x0004404A,
- 0x034, 0x00043047,
- 0x034, 0x0004200A,
- 0x034, 0x00041007,
- 0x034, 0x00040004,
- 0xFF0F07D8, 0xCDEF,
- 0x034, 0x0004A0B2,
- 0x034, 0x000490AF,
- 0x034, 0x00048070,
- 0x034, 0x0004706D,
- 0x034, 0x00046050,
- 0x034, 0x0004504D,
- 0x034, 0x0004404A,
- 0x034, 0x00043047,
- 0x034, 0x0004200A,
- 0x034, 0x00041007,
- 0x034, 0x00040004,
- 0xFF0F07D0, 0xCDEF,
- 0x034, 0x0004A0B2,
- 0x034, 0x000490AF,
- 0x034, 0x00048070,
- 0x034, 0x0004706D,
- 0x034, 0x00046050,
- 0x034, 0x0004504D,
- 0x034, 0x0004404A,
- 0x034, 0x00043047,
- 0x034, 0x0004200A,
- 0x034, 0x00041007,
- 0x034, 0x00040004,
- 0xCDCDCDCD, 0xCDCD,
+ 0x034, 0x0004604D,
+ 0x034, 0x0004504A,
+ 0x034, 0x00044047,
+ 0x034, 0x00043044,
+ 0x034, 0x00042007,
+ 0x034, 0x00041004,
+ 0x034, 0x00040001,
+ 0xA0000000, 0x00000000,
0x034, 0x0004ADF5,
0x034, 0x00049DF2,
0x034, 0x00048DEF,
0x034, 0x00047DEC,
0x034, 0x00046DE9,
- 0x034, 0x00045DC9,
- 0x034, 0x00044CE8,
- 0x034, 0x000438CA,
- 0x034, 0x00042889,
- 0x034, 0x0004184A,
- 0x034, 0x0004044A,
- 0xFF0F0740, 0xDEAD,
- 0xFF0F0740, 0xABCD,
+ 0x034, 0x00045DE6,
+ 0x034, 0x00044DE3,
+ 0x034, 0x000438C8,
+ 0x034, 0x000428C5,
+ 0x034, 0x000418C2,
+ 0x034, 0x000408C0,
+ 0xB0000000, 0x00000000,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
0x034, 0x0002A0B2,
0x034, 0x000290AF,
0x034, 0x00028070,
@@ -1399,68 +729,32 @@
0x034, 0x0002200A,
0x034, 0x00021007,
0x034, 0x00020004,
- 0xFF0F01C0, 0xCDEF,
- 0x034, 0x0002A0B2,
- 0x034, 0x000290AF,
- 0x034, 0x00028070,
- 0x034, 0x0002706D,
- 0x034, 0x00026050,
- 0x034, 0x0002504D,
- 0x034, 0x0002404A,
- 0x034, 0x00023047,
- 0x034, 0x0002200A,
- 0x034, 0x00021007,
- 0x034, 0x00020004,
- 0xFF0F02C0, 0xCDEF,
- 0x034, 0x0002A0B2,
- 0x034, 0x000290AF,
- 0x034, 0x00028070,
- 0x034, 0x0002706D,
- 0x034, 0x00026050,
- 0x034, 0x0002504D,
- 0x034, 0x0002404A,
- 0x034, 0x00023047,
- 0x034, 0x0002200A,
- 0x034, 0x00021007,
- 0x034, 0x00020004,
- 0xFF0F07D8, 0xCDEF,
- 0x034, 0x0002A0B2,
- 0x034, 0x000290AF,
- 0x034, 0x00028070,
- 0x034, 0x0002706D,
- 0x034, 0x00026050,
- 0x034, 0x0002504D,
- 0x034, 0x0002404A,
- 0x034, 0x00023047,
- 0x034, 0x0002200A,
- 0x034, 0x00021007,
- 0x034, 0x00020004,
- 0xFF0F07D0, 0xCDEF,
- 0x034, 0x0002A0B2,
- 0x034, 0x000290AF,
- 0x034, 0x00028070,
- 0x034, 0x0002706D,
- 0x034, 0x00026050,
- 0x034, 0x0002504D,
- 0x034, 0x0002404A,
- 0x034, 0x00023047,
- 0x034, 0x0002200A,
- 0x034, 0x00021007,
- 0x034, 0x00020004,
- 0xCDCDCDCD, 0xCDCD,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
+ 0x034, 0x0002A0B4,
+ 0x034, 0x000290B1,
+ 0x034, 0x00028072,
+ 0x034, 0x0002706F,
+ 0x034, 0x0002604F,
+ 0x034, 0x0002504C,
+ 0x034, 0x00024049,
+ 0x034, 0x00023046,
+ 0x034, 0x00022009,
+ 0x034, 0x00021006,
+ 0x034, 0x00020003,
+ 0xA0000000, 0x00000000,
0x034, 0x0002ADF5,
0x034, 0x00029DF2,
0x034, 0x00028DEF,
0x034, 0x00027DEC,
0x034, 0x00026DE9,
- 0x034, 0x00025DC9,
- 0x034, 0x00024CE8,
- 0x034, 0x000238CA,
- 0x034, 0x00022889,
- 0x034, 0x0002184A,
- 0x034, 0x0002044A,
- 0xFF0F0740, 0xDEAD,
- 0xFF0F0740, 0xABCD,
+ 0x034, 0x00025DE6,
+ 0x034, 0x00024DE3,
+ 0x034, 0x000238C8,
+ 0x034, 0x000228C5,
+ 0x034, 0x000218C2,
+ 0x034, 0x000208C0,
+ 0xB0000000, 0x00000000,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
0x034, 0x0000A0B2,
0x034, 0x000090AF,
0x034, 0x00008070,
@@ -1472,310 +766,577 @@
0x034, 0x0000200A,
0x034, 0x00001007,
0x034, 0x00000004,
- 0xFF0F01C0, 0xCDEF,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
0x034, 0x0000A0B2,
0x034, 0x000090AF,
0x034, 0x00008070,
0x034, 0x0000706D,
- 0x034, 0x00006050,
- 0x034, 0x0000504D,
- 0x034, 0x0000404A,
- 0x034, 0x00003047,
- 0x034, 0x0000200A,
- 0x034, 0x00001007,
- 0x034, 0x00000004,
- 0xFF0F02C0, 0xCDEF,
- 0x034, 0x0000A0B2,
- 0x034, 0x000090AF,
- 0x034, 0x00008070,
- 0x034, 0x0000706D,
- 0x034, 0x00006050,
- 0x034, 0x0000504D,
- 0x034, 0x0000404A,
- 0x034, 0x00003047,
- 0x034, 0x0000200A,
- 0x034, 0x00001007,
- 0x034, 0x00000004,
- 0xFF0F07D8, 0xCDEF,
- 0x034, 0x0000A0B2,
- 0x034, 0x000090AF,
- 0x034, 0x00008070,
- 0x034, 0x0000706D,
- 0x034, 0x00006050,
- 0x034, 0x0000504D,
- 0x034, 0x0000404A,
- 0x034, 0x00003047,
- 0x034, 0x0000200A,
- 0x034, 0x00001007,
- 0x034, 0x00000004,
- 0xFF0F07D0, 0xCDEF,
- 0x034, 0x0000A0B2,
- 0x034, 0x000090AF,
- 0x034, 0x00008070,
- 0x034, 0x0000706D,
- 0x034, 0x00006050,
- 0x034, 0x0000504D,
- 0x034, 0x0000404A,
- 0x034, 0x00003047,
- 0x034, 0x0000200A,
- 0x034, 0x00001007,
- 0x034, 0x00000004,
- 0xCDCDCDCD, 0xCDCD,
+ 0x034, 0x0000604D,
+ 0x034, 0x0000504A,
+ 0x034, 0x00004047,
+ 0x034, 0x00003044,
+ 0x034, 0x00002007,
+ 0x034, 0x00001004,
+ 0x034, 0x00000001,
+ 0xA0000000, 0x00000000,
0x034, 0x0000AFF7,
0x034, 0x00009DF7,
0x034, 0x00008DF4,
0x034, 0x00007DF1,
0x034, 0x00006DEE,
- 0x034, 0x00005DCD,
- 0x034, 0x00004CEB,
+ 0x034, 0x00005DEB,
+ 0x034, 0x00004DE8,
0x034, 0x000038CC,
- 0x034, 0x0000288B,
- 0x034, 0x0000184C,
- 0x034, 0x0000044C,
- 0xFF0F0740, 0xDEAD,
+ 0x034, 0x000028C9,
+ 0x034, 0x000018C6,
+ 0x034, 0x000008C3,
+ 0xB0000000, 0x00000000,
0x0EF, 0x00000000,
- 0xFF0F0740, 0xABCD,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
0x018, 0x0001712A,
0x0EF, 0x00000040,
- 0x035, 0x000001C5,
- 0x035, 0x000081C5,
- 0x035, 0x000101C5,
- 0x035, 0x00020174,
- 0x035, 0x00028174,
- 0x035, 0x00030174,
- 0x035, 0x00040185,
- 0x035, 0x00048185,
- 0x035, 0x00050185,
- 0x0EF, 0x00000000,
- 0xFF0F01C0, 0xCDEF,
+ 0x035, 0x000001D4,
+ 0x035, 0x000081D4,
+ 0x035, 0x000101D4,
+ 0x035, 0x000201B4,
+ 0x035, 0x000281B4,
+ 0x035, 0x000301B4,
+ 0x035, 0x000401B4,
+ 0x035, 0x000481B4,
+ 0x035, 0x000501B4,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
0x018, 0x0001712A,
0x0EF, 0x00000040,
- 0x035, 0x000001C5,
- 0x035, 0x000081C5,
- 0x035, 0x000101C5,
- 0x035, 0x00020174,
- 0x035, 0x00028174,
- 0x035, 0x00030174,
- 0x035, 0x00040185,
- 0x035, 0x00048185,
- 0x035, 0x00050185,
- 0x0EF, 0x00000000,
- 0xFF0F02C0, 0xCDEF,
+ 0x035, 0x000001D4,
+ 0x035, 0x000081D4,
+ 0x035, 0x000101D4,
+ 0x035, 0x000201B4,
+ 0x035, 0x000281B4,
+ 0x035, 0x000301B4,
+ 0x035, 0x000401B4,
+ 0x035, 0x000481B4,
+ 0x035, 0x000501B4,
+ 0xA0000000, 0x00000000,
0x018, 0x0001712A,
0x0EF, 0x00000040,
- 0x035, 0x000001C5,
- 0x035, 0x000081C5,
- 0x035, 0x000101C5,
- 0x035, 0x00020174,
- 0x035, 0x00028174,
- 0x035, 0x00030174,
- 0x035, 0x00040185,
- 0x035, 0x00048185,
- 0x035, 0x00050185,
+ 0x035, 0x00000188,
+ 0x035, 0x00008147,
+ 0x035, 0x00010147,
+ 0x035, 0x000201D7,
+ 0x035, 0x000281D7,
+ 0x035, 0x000301D7,
+ 0x035, 0x000401D8,
+ 0x035, 0x000481D8,
+ 0x035, 0x000501D8,
+ 0xB0000000, 0x00000000,
0x0EF, 0x00000000,
- 0xFF0F07D8, 0xCDEF,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000040,
- 0x035, 0x000001C5,
- 0x035, 0x000081C5,
- 0x035, 0x000101C5,
- 0x035, 0x00020174,
- 0x035, 0x00028174,
- 0x035, 0x00030174,
- 0x035, 0x00040185,
- 0x035, 0x00048185,
- 0x035, 0x00050185,
- 0x0EF, 0x00000000,
- 0xFF0F07D0, 0xCDEF,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000040,
- 0x035, 0x000001C5,
- 0x035, 0x000081C5,
- 0x035, 0x000101C5,
- 0x035, 0x00020174,
- 0x035, 0x00028174,
- 0x035, 0x00030174,
- 0x035, 0x00040185,
- 0x035, 0x00048185,
- 0x035, 0x00050185,
- 0x0EF, 0x00000000,
- 0xCDCDCDCD, 0xCDCD,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000040,
- 0x035, 0x00000186,
- 0x035, 0x00008186,
- 0x035, 0x00010185,
- 0x035, 0x000201D5,
- 0x035, 0x000281D5,
- 0x035, 0x000301D5,
- 0x035, 0x000401D5,
- 0x035, 0x000481D5,
- 0x035, 0x000501D5,
- 0x0EF, 0x00000000,
- 0xFF0F0740, 0xDEAD,
- 0xFF0F0740, 0xABCD,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
0x018, 0x0001712A,
0x0EF, 0x00000010,
- 0x036, 0x00005B8B,
- 0x036, 0x0000DB8B,
- 0x036, 0x00015B8B,
- 0x036, 0x0001DB8B,
- 0x036, 0x000262DB,
- 0x036, 0x0002E2DB,
- 0x036, 0x000362DB,
- 0x036, 0x0003E2DB,
- 0x036, 0x0004553B,
- 0x036, 0x0004D53B,
- 0x036, 0x0005553B,
- 0x036, 0x0005D53B,
- 0xFF0F01C0, 0xCDEF,
+ 0x036, 0x00004BFB,
+ 0x036, 0x0000CBFB,
+ 0x036, 0x00014BFB,
+ 0x036, 0x0001CBFB,
+ 0x036, 0x00024F4B,
+ 0x036, 0x0002CF4B,
+ 0x036, 0x00034F4B,
+ 0x036, 0x0003CF4B,
+ 0x036, 0x00044F4B,
+ 0x036, 0x0004CF4B,
+ 0x036, 0x00054F4B,
+ 0x036, 0x0005CF4B,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
0x018, 0x0001712A,
0x0EF, 0x00000010,
- 0x036, 0x00005B8B,
- 0x036, 0x0000DB8B,
- 0x036, 0x00015B8B,
- 0x036, 0x0001DB8B,
- 0x036, 0x000262DB,
- 0x036, 0x0002E2DB,
- 0x036, 0x000362DB,
- 0x036, 0x0003E2DB,
- 0x036, 0x0004553B,
- 0x036, 0x0004D53B,
- 0x036, 0x0005553B,
- 0x036, 0x0005D53B,
- 0xFF0F02C0, 0xCDEF,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000010,
- 0x036, 0x00005B8B,
- 0x036, 0x0000DB8B,
- 0x036, 0x00015B8B,
- 0x036, 0x0001DB8B,
- 0x036, 0x000262DB,
- 0x036, 0x0002E2DB,
- 0x036, 0x000362DB,
- 0x036, 0x0003E2DB,
- 0x036, 0x0004553B,
- 0x036, 0x0004D53B,
- 0x036, 0x0005553B,
- 0x036, 0x0005D53B,
- 0xFF0F07D8, 0xCDEF,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000010,
- 0x036, 0x00005B8B,
- 0x036, 0x0000DB8B,
- 0x036, 0x00015B8B,
- 0x036, 0x0001DB8B,
- 0x036, 0x000262DB,
- 0x036, 0x0002E2DB,
- 0x036, 0x000362DB,
- 0x036, 0x0003E2DB,
- 0x036, 0x0004553B,
- 0x036, 0x0004D53B,
- 0x036, 0x0005553B,
- 0x036, 0x0005D53B,
- 0xFF0F07D0, 0xCDEF,
- 0x018, 0x0001712A,
- 0x0EF, 0x00000010,
- 0x036, 0x00005B8B,
- 0x036, 0x0000DB8B,
- 0x036, 0x00015B8B,
- 0x036, 0x0001DB8B,
- 0x036, 0x000262DB,
- 0x036, 0x0002E2DB,
- 0x036, 0x000362DB,
- 0x036, 0x0003E2DB,
- 0x036, 0x0004553B,
- 0x036, 0x0004D53B,
- 0x036, 0x0005553B,
- 0x036, 0x0005D53B,
- 0xCDCDCDCD, 0xCDCD,
+ 0x036, 0x00004BFB,
+ 0x036, 0x0000CBFB,
+ 0x036, 0x00014BFB,
+ 0x036, 0x0001CBFB,
+ 0x036, 0x00024F4B,
+ 0x036, 0x0002CF4B,
+ 0x036, 0x00034F4B,
+ 0x036, 0x0003CF4B,
+ 0x036, 0x00044F4B,
+ 0x036, 0x0004CF4B,
+ 0x036, 0x00054F4B,
+ 0x036, 0x0005CF4B,
+ 0xA0000000, 0x00000000,
0x018, 0x0001712A,
0x0EF, 0x00000010,
0x036, 0x00084EB4,
- 0x036, 0x0008C9B4,
- 0x036, 0x000949B4,
- 0x036, 0x0009C9B4,
- 0x036, 0x000A4935,
- 0x036, 0x000AC935,
- 0x036, 0x000B4935,
- 0x036, 0x000BC935,
- 0x036, 0x000C4EB4,
- 0x036, 0x000CCEB4,
- 0x036, 0x000D4EB4,
- 0x036, 0x000DCEB4,
- 0xFF0F0740, 0xDEAD,
+ 0x036, 0x0008CC35,
+ 0x036, 0x00094C35,
+ 0x036, 0x0009CC35,
+ 0x036, 0x000A4C35,
+ 0x036, 0x000ACC35,
+ 0x036, 0x000B4C35,
+ 0x036, 0x000BCC35,
+ 0x036, 0x000C4C34,
+ 0x036, 0x000CCC35,
+ 0x036, 0x000D4C35,
+ 0x036, 0x000DCC35,
+ 0xB0000000, 0x00000000,
0x0EF, 0x00000000,
0x0EF, 0x00000008,
- 0xFF0F0740, 0xABCD,
- 0x03C, 0x000002DC,
- 0x03C, 0x00000524,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
+ 0x03C, 0x000002CC,
+ 0x03C, 0x00000522,
0x03C, 0x00000902,
- 0xFF0F01C0, 0xCDEF,
- 0x03C, 0x000002DC,
- 0x03C, 0x00000524,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
+ 0x03C, 0x000002CC,
+ 0x03C, 0x00000522,
0x03C, 0x00000902,
- 0xFF0F02C0, 0xCDEF,
- 0x03C, 0x000002DC,
- 0x03C, 0x00000524,
- 0x03C, 0x00000902,
- 0xFF0F07D8, 0xCDEF,
- 0x03C, 0x000002DC,
- 0x03C, 0x00000524,
- 0x03C, 0x00000902,
- 0xFF0F07D0, 0xCDEF,
- 0x03C, 0x000002DC,
- 0x03C, 0x00000524,
- 0x03C, 0x00000902,
- 0xCDCDCDCD, 0xCDCD,
- 0x03C, 0x000002AA,
+ 0xA0000000, 0x00000000,
+ 0x03C, 0x000002A8,
0x03C, 0x000005A2,
0x03C, 0x00000880,
- 0xFF0F0740, 0xDEAD,
+ 0xB0000000, 0x00000000,
0x0EF, 0x00000000,
0x018, 0x0001712A,
0x0EF, 0x00000002,
0x0DF, 0x00000080,
- 0xFF0F0740, 0xABCD,
- 0x061, 0x000EAC43,
- 0x062, 0x00038F47,
- 0x063, 0x00031157,
- 0x064, 0x0001C4AC,
+ 0x01F, 0x00000064,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
+ 0x061, 0x000FDD43,
+ 0x062, 0x00038F4B,
+ 0x063, 0x00032117,
+ 0x064, 0x000194AC,
0x065, 0x000931D1,
- 0xFF0F01C0, 0xCDEF,
- 0x061, 0x000EAC43,
- 0x062, 0x00038F47,
- 0x063, 0x00031157,
- 0x064, 0x0001C4AC,
- 0x065, 0x000931D1,
- 0xFF0F02C0, 0xCDEF,
- 0x061, 0x000EAC43,
- 0x062, 0x00038F47,
- 0x063, 0x00031157,
- 0x064, 0x0001C4AC,
- 0x065, 0x000931D1,
- 0xFF0F07D8, 0xCDEF,
- 0x061, 0x000EAC43,
- 0x062, 0x00038F47,
- 0x063, 0x00031157,
- 0x064, 0x0001C4AC,
- 0x065, 0x000931D1,
- 0xFF0F07D0, 0xCDEF,
- 0x061, 0x000EAC43,
- 0x062, 0x00038F47,
- 0x063, 0x00031157,
- 0x064, 0x0001C4AC,
- 0x065, 0x000931D1,
- 0xCDCDCDCD, 0xCDCD,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
+ 0x061, 0x000FDD43,
+ 0x062, 0x00038F4B,
+ 0x063, 0x00032117,
+ 0x064, 0x000194AC,
+ 0x065, 0x000931D2,
+ 0xA0000000, 0x00000000,
0x061, 0x000E5D53,
0x062, 0x00038FCD,
- 0x063, 0x000314EB,
+ 0x063, 0x000114EB,
0x064, 0x000196AC,
- 0x065, 0x000931D7,
- 0xFF0F0740, 0xDEAD,
+ 0x065, 0x000911D7,
+ 0xB0000000, 0x00000000,
0x008, 0x00008400,
-
+ 0x01C, 0x000739D2,
+ 0x0B4, 0x0001E78D,
+ 0x018, 0x0001F12A,
+ 0x0FE, 0x00000000,
+ 0x0FE, 0x00000000,
+ 0x0FE, 0x00000000,
+ 0x0FE, 0x00000000,
+ 0x0B4, 0x0001A78D,
+ 0x018, 0x0001712A,
};
+u32 RTL8812AE_RADIOA_1TARRAYLEN = sizeof(RTL8812AE_RADIOA_ARRAY) / sizeof(u32);
+
+u32 RTL8812AE_RADIOB_ARRAY[] = {
+ 0x056, 0x00051CF2,
+ 0x066, 0x00040000,
+ 0x089, 0x00000080,
+ 0x80000001, 0x00000000, 0x40000000, 0x00000000,
+ 0x086, 0x00014B3A,
+ 0x90000001, 0x00000005, 0x40000000, 0x00000000,
+ 0x086, 0x00014B3A,
+ 0xA0000000, 0x00000000,
+ 0x086, 0x00014B38,
+ 0xB0000000, 0x00000000,
+ 0x80000004, 0x00000000, 0x40000000, 0x00000000,
+ 0x08B, 0x00080180,
+ 0xA0000000, 0x00000000,
+ 0x08B, 0x00087180,
+ 0xB0000000, 0x00000000,
+ 0x018, 0x00000006,
+ 0x0EF, 0x00002000,
+ 0x80000001, 0x00000000, 0x40000000, 0x00000000,
+ 0x03B, 0x0003F218,
+ 0x03B, 0x00030A58,
+ 0x03B, 0x0002FA58,
+ 0x03B, 0x00022590,
+ 0x03B, 0x0001FA50,
+ 0x03B, 0x00010248,
+ 0x03B, 0x00008240,
+ 0x90000001, 0x00000005, 0x40000000, 0x00000000,
+ 0x03B, 0x0003F218,
+ 0x03B, 0x00030A58,
+ 0x03B, 0x0002FA58,
+ 0x03B, 0x00022590,
+ 0x03B, 0x0001FA50,
+ 0x03B, 0x00010248,
+ 0x03B, 0x00008240,
+ 0xA0000000, 0x00000000,
+ 0x03B, 0x00038A58,
+ 0x03B, 0x00037A58,
+ 0x03B, 0x0002A590,
+ 0x03B, 0x00027A50,
+ 0x03B, 0x00018248,
+ 0x03B, 0x00010240,
+ 0x03B, 0x00008240,
+ 0xB0000000, 0x00000000,
+ 0x0EF, 0x00000100,
+ 0x80000002, 0x00000000, 0x40000000, 0x00000000,
+ 0x034, 0x0000A4EE,
+ 0x034, 0x00009076,
+ 0x034, 0x00008073,
+ 0x034, 0x00007070,
+ 0x034, 0x0000606D,
+ 0x034, 0x0000506A,
+ 0x034, 0x00004049,
+ 0x034, 0x00003046,
+ 0x034, 0x00002028,
+ 0x034, 0x00001025,
+ 0x034, 0x00000022,
+ 0xA0000000, 0x00000000,
+ 0x034, 0x0000ADF4,
+ 0x034, 0x00009DF1,
+ 0x034, 0x00008DEE,
+ 0x034, 0x00007DEB,
+ 0x034, 0x00006DE8,
+ 0x034, 0x00005CEC,
+ 0x034, 0x00004CE9,
+ 0x034, 0x000034EA,
+ 0x034, 0x000024E7,
+ 0x034, 0x0000146B,
+ 0x034, 0x0000006D,
+ 0xB0000000, 0x00000000,
+ 0x0EF, 0x00000000,
+ 0x0EF, 0x000020A2,
+ 0x0DF, 0x00000080,
+ 0x035, 0x00000192,
+ 0x035, 0x00008192,
+ 0x035, 0x00010192,
+ 0x036, 0x00000024,
+ 0x036, 0x00008024,
+ 0x036, 0x00010024,
+ 0x036, 0x00018024,
+ 0x0EF, 0x00000000,
+ 0x051, 0x00000C21,
+ 0x052, 0x000006D9,
+ 0x053, 0x000FC649,
+ 0x054, 0x0000017E,
+ 0x0EF, 0x00000002,
+ 0x008, 0x00008400,
+ 0x018, 0x0001712A,
+ 0x0EF, 0x00001000,
+ 0x03A, 0x00000080,
+ 0x03B, 0x0003A02C,
+ 0x03C, 0x00004000,
+ 0x03A, 0x00000400,
+ 0x03B, 0x0003202C,
+ 0x03C, 0x00010000,
+ 0x03A, 0x000000A0,
+ 0x03B, 0x0002B064,
+ 0x03C, 0x00004000,
+ 0x03A, 0x000000D8,
+ 0x03B, 0x00023070,
+ 0x03C, 0x00004000,
+ 0x03A, 0x00000468,
+ 0x03B, 0x0001B870,
+ 0x03C, 0x00010000,
+ 0x03A, 0x00000098,
+ 0x03B, 0x00012085,
+ 0x03C, 0x000E4000,
+ 0x03A, 0x00000418,
+ 0x03B, 0x0000A080,
+ 0x03C, 0x000F0000,
+ 0x03A, 0x00000418,
+ 0x03B, 0x00002080,
+ 0x03C, 0x00010000,
+ 0x03A, 0x00000080,
+ 0x03B, 0x0007A02C,
+ 0x03C, 0x00004000,
+ 0x03A, 0x00000400,
+ 0x03B, 0x0007202C,
+ 0x03C, 0x00010000,
+ 0x03A, 0x000000A0,
+ 0x03B, 0x0006B064,
+ 0x03C, 0x00004000,
+ 0x03A, 0x000000D8,
+ 0x03B, 0x00063070,
+ 0x03C, 0x00004000,
+ 0x03A, 0x00000468,
+ 0x03B, 0x0005B870,
+ 0x03C, 0x00010000,
+ 0x03A, 0x00000098,
+ 0x03B, 0x00052085,
+ 0x03C, 0x000E4000,
+ 0x03A, 0x00000418,
+ 0x03B, 0x0004A080,
+ 0x03C, 0x000F0000,
+ 0x03A, 0x00000418,
+ 0x03B, 0x00042080,
+ 0x03C, 0x00010000,
+ 0x03A, 0x00000080,
+ 0x03B, 0x000BA02C,
+ 0x03C, 0x00004000,
+ 0x03A, 0x00000400,
+ 0x03B, 0x000B202C,
+ 0x03C, 0x00010000,
+ 0x03A, 0x000000A0,
+ 0x03B, 0x000AB064,
+ 0x03C, 0x00004000,
+ 0x03A, 0x000000D8,
+ 0x03B, 0x000A3070,
+ 0x03C, 0x00004000,
+ 0x03A, 0x00000468,
+ 0x03B, 0x0009B870,
+ 0x03C, 0x00010000,
+ 0x03A, 0x00000098,
+ 0x03B, 0x00092085,
+ 0x03C, 0x000E4000,
+ 0x03A, 0x00000418,
+ 0x03B, 0x0008A080,
+ 0x03C, 0x000F0000,
+ 0x03A, 0x00000418,
+ 0x03B, 0x00082080,
+ 0x03C, 0x00010000,
+ 0x0EF, 0x00001100,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
+ 0x034, 0x0004A0B2,
+ 0x034, 0x000490AF,
+ 0x034, 0x00048070,
+ 0x034, 0x0004706D,
+ 0x034, 0x00046050,
+ 0x034, 0x0004504D,
+ 0x034, 0x0004404A,
+ 0x034, 0x00043047,
+ 0x034, 0x0004200A,
+ 0x034, 0x00041007,
+ 0x034, 0x00040004,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
+ 0x034, 0x0004A0B1,
+ 0x034, 0x000490AE,
+ 0x034, 0x0004806F,
+ 0x034, 0x0004706C,
+ 0x034, 0x0004604C,
+ 0x034, 0x00045049,
+ 0x034, 0x00044046,
+ 0x034, 0x00043043,
+ 0x034, 0x00042006,
+ 0x034, 0x00041003,
+ 0x034, 0x00040000,
+ 0xA0000000, 0x00000000,
+ 0x034, 0x0004ADF5,
+ 0x034, 0x00049DF2,
+ 0x034, 0x00048DEF,
+ 0x034, 0x00047DEC,
+ 0x034, 0x00046DE9,
+ 0x034, 0x00045DE6,
+ 0x034, 0x00044DE3,
+ 0x034, 0x000438C8,
+ 0x034, 0x000428C5,
+ 0x034, 0x000418C2,
+ 0x034, 0x000408C0,
+ 0xB0000000, 0x00000000,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
+ 0x034, 0x0002A0B2,
+ 0x034, 0x000290AF,
+ 0x034, 0x00028070,
+ 0x034, 0x0002706D,
+ 0x034, 0x00026050,
+ 0x034, 0x0002504D,
+ 0x034, 0x0002404A,
+ 0x034, 0x00023047,
+ 0x034, 0x0002200A,
+ 0x034, 0x00021007,
+ 0x034, 0x00020004,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
+ 0x034, 0x0002A0B3,
+ 0x034, 0x000290B0,
+ 0x034, 0x00028071,
+ 0x034, 0x0002706E,
+ 0x034, 0x0002604E,
+ 0x034, 0x0002504B,
+ 0x034, 0x00024048,
+ 0x034, 0x00023045,
+ 0x034, 0x00022008,
+ 0x034, 0x00021005,
+ 0x034, 0x00020002,
+ 0xA0000000, 0x00000000,
+ 0x034, 0x0002ADF5,
+ 0x034, 0x00029DF2,
+ 0x034, 0x00028DEF,
+ 0x034, 0x00027DEC,
+ 0x034, 0x00026DE9,
+ 0x034, 0x00025DE6,
+ 0x034, 0x00024DE3,
+ 0x034, 0x000238C8,
+ 0x034, 0x000228C5,
+ 0x034, 0x000218C2,
+ 0x034, 0x000208C0,
+ 0xB0000000, 0x00000000,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
+ 0x034, 0x0000A0B2,
+ 0x034, 0x000090AF,
+ 0x034, 0x00008070,
+ 0x034, 0x0000706D,
+ 0x034, 0x00006050,
+ 0x034, 0x0000504D,
+ 0x034, 0x0000404A,
+ 0x034, 0x00003047,
+ 0x034, 0x0000200A,
+ 0x034, 0x00001007,
+ 0x034, 0x00000004,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
+ 0x034, 0x0000A0B3,
+ 0x034, 0x000090B0,
+ 0x034, 0x00008070,
+ 0x034, 0x0000706D,
+ 0x034, 0x0000604D,
+ 0x034, 0x0000504A,
+ 0x034, 0x00004047,
+ 0x034, 0x00003044,
+ 0x034, 0x00002007,
+ 0x034, 0x00001004,
+ 0x034, 0x00000001,
+ 0xA0000000, 0x00000000,
+ 0x034, 0x0000AFF7,
+ 0x034, 0x00009DF7,
+ 0x034, 0x00008DF4,
+ 0x034, 0x00007DF1,
+ 0x034, 0x00006DEE,
+ 0x034, 0x00005DEB,
+ 0x034, 0x00004DE8,
+ 0x034, 0x000038CC,
+ 0x034, 0x000028C9,
+ 0x034, 0x000018C6,
+ 0x034, 0x000008C3,
+ 0xB0000000, 0x00000000,
+ 0x0EF, 0x00000000,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
+ 0x018, 0x0001712A,
+ 0x0EF, 0x00000040,
+ 0x035, 0x000001C5,
+ 0x035, 0x000081C5,
+ 0x035, 0x000101C5,
+ 0x035, 0x00020174,
+ 0x035, 0x00028174,
+ 0x035, 0x00030174,
+ 0x035, 0x00040185,
+ 0x035, 0x00048185,
+ 0x035, 0x00050185,
+ 0x0EF, 0x00000000,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
+ 0x018, 0x0001712A,
+ 0x0EF, 0x00000040,
+ 0x035, 0x000001C5,
+ 0x035, 0x000081C5,
+ 0x035, 0x000101C5,
+ 0x035, 0x00020174,
+ 0x035, 0x00028174,
+ 0x035, 0x00030174,
+ 0x035, 0x00040185,
+ 0x035, 0x00048185,
+ 0x035, 0x00050185,
+ 0x0EF, 0x00000000,
+ 0xA0000000, 0x00000000,
+ 0x018, 0x0001712A,
+ 0x0EF, 0x00000040,
+ 0x035, 0x00000188,
+ 0x035, 0x00008147,
+ 0x035, 0x00010147,
+ 0x035, 0x000201D7,
+ 0x035, 0x000281D7,
+ 0x035, 0x000301D7,
+ 0x035, 0x000401D8,
+ 0x035, 0x000481D8,
+ 0x035, 0x000501D8,
+ 0x0EF, 0x00000000,
+ 0xB0000000, 0x00000000,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
+ 0x018, 0x0001712A,
+ 0x0EF, 0x00000010,
+ 0x036, 0x00005B8B,
+ 0x036, 0x0000DB8B,
+ 0x036, 0x00015B8B,
+ 0x036, 0x0001DB8B,
+ 0x036, 0x000262DB,
+ 0x036, 0x0002E2DB,
+ 0x036, 0x000362DB,
+ 0x036, 0x0003E2DB,
+ 0x036, 0x0004553B,
+ 0x036, 0x0004D53B,
+ 0x036, 0x0005553B,
+ 0x036, 0x0005D53B,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
+ 0x018, 0x0001712A,
+ 0x0EF, 0x00000010,
+ 0x036, 0x00005B8B,
+ 0x036, 0x0000DB8B,
+ 0x036, 0x00015B8B,
+ 0x036, 0x0001DB8B,
+ 0x036, 0x000262DB,
+ 0x036, 0x0002E2DB,
+ 0x036, 0x000362DB,
+ 0x036, 0x0003E2DB,
+ 0x036, 0x0004553B,
+ 0x036, 0x0004D53B,
+ 0x036, 0x0005553B,
+ 0x036, 0x0005D53B,
+ 0xA0000000, 0x00000000,
+ 0x018, 0x0001712A,
+ 0x0EF, 0x00000010,
+ 0x036, 0x00084EB4,
+ 0x036, 0x0008CC35,
+ 0x036, 0x00094C35,
+ 0x036, 0x0009CC35,
+ 0x036, 0x000A4C35,
+ 0x036, 0x000ACC35,
+ 0x036, 0x000B4C35,
+ 0x036, 0x000BCC35,
+ 0x036, 0x000C4C34,
+ 0x036, 0x000CCC35,
+ 0x036, 0x000D4C35,
+ 0x036, 0x000DCC35,
+ 0xB0000000, 0x00000000,
+ 0x0EF, 0x00000000,
+ 0x0EF, 0x00000008,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
+ 0x03C, 0x000002DC,
+ 0x03C, 0x00000524,
+ 0x03C, 0x00000902,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
+ 0x03C, 0x000002DC,
+ 0x03C, 0x00000524,
+ 0x03C, 0x00000902,
+ 0xA0000000, 0x00000000,
+ 0x03C, 0x000002A8,
+ 0x03C, 0x000005A2,
+ 0x03C, 0x00000880,
+ 0xB0000000, 0x00000000,
+ 0x0EF, 0x00000000,
+ 0x018, 0x0001712A,
+ 0x0EF, 0x00000002,
+ 0x0DF, 0x00000080,
+ 0x80000008, 0x00000000, 0x40000000, 0x00000000,
+ 0x061, 0x000EAC43,
+ 0x062, 0x00038F47,
+ 0x063, 0x00031157,
+ 0x064, 0x0001C4AC,
+ 0x065, 0x000931D1,
+ 0x90000008, 0x05000000, 0x40000000, 0x00000000,
+ 0x061, 0x000EAC43,
+ 0x062, 0x00038F47,
+ 0x063, 0x00031157,
+ 0x064, 0x0001C4AC,
+ 0x065, 0x000931D2,
+ 0x90000002, 0x00000000, 0x40000000, 0x00000000,
+ 0x061, 0x000EAC43,
+ 0x062, 0x00038F47,
+ 0x063, 0x00031157,
+ 0x064, 0x0001C4AC,
+ 0x065, 0x000931D1,
+ 0xA0000000, 0x00000000,
+ 0x061, 0x000E5D53,
+ 0x062, 0x00038FCD,
+ 0x063, 0x000114EB,
+ 0x064, 0x000196AC,
+ 0x065, 0x000911D7,
+ 0xB0000000, 0x00000000,
+ 0x008, 0x00008400,
+};
+
+u32 RTL8812AE_RADIOB_1TARRAYLEN = sizeof(RTL8812AE_RADIOB_ARRAY) / sizeof(u32);
+
u32 RTL8821AE_RADIOA_ARRAY[] = {
0x018, 0x0001712A,
0x056, 0x00051CF2,
@@ -2285,16 +1846,16 @@
0x0EF, 0x00000000,
0x0EF, 0x00000100,
0x034, 0x0000ADF3,
- 0x034, 0x00009DEF,
- 0x034, 0x00008DEC,
- 0x034, 0x00007DE9,
- 0x034, 0x00006CED,
- 0x034, 0x00005CE9,
- 0x034, 0x000044E9,
- 0x034, 0x000034E6,
- 0x034, 0x0000246A,
- 0x034, 0x00001467,
- 0x034, 0x00000068,
+ 0x034, 0x00009DF0,
+ 0x034, 0x00008D70,
+ 0x034, 0x00007D6D,
+ 0x034, 0x00006CEE,
+ 0x034, 0x00005CCC,
+ 0x034, 0x000044EC,
+ 0x034, 0x000034AC,
+ 0x034, 0x0000246D,
+ 0x034, 0x0000106F,
+ 0x034, 0x0000006C,
0x0EF, 0x00000000,
0x0ED, 0x00000010,
0x044, 0x0000ADF2,
@@ -2365,18 +1926,21 @@
0x0FE, 0x00000000,
0x0FE, 0x00000000,
0x018, 0x0001712A,
+
};
+u32 RTL8821AE_RADIOA_1TARRAYLEN = sizeof(RTL8821AE_RADIOA_ARRAY) / sizeof(u32);
+
u32 RTL8812AE_MAC_REG_ARRAY[] = {
0x010, 0x0000000C,
- 0xFF0F0180, 0xABCD,
+ 0x80000200, 0x00000000, 0x40000000, 0x00000000,
+ 0x011, 0x00000066,
+ 0xA0000000, 0x00000000,
+ 0x011, 0x0000005A,
+ 0xB0000000, 0x00000000,
0x025, 0x0000000F,
- 0xFF0F01C0, 0xCDEF,
- 0x025, 0x0000000F,
- 0xCDCDCDCD, 0xCDCD,
- 0x025, 0x0000006F,
- 0xFF0F0180, 0xDEAD,
0x072, 0x00000000,
+ 0x420, 0x00000080,
0x428, 0x0000000A,
0x429, 0x00000010,
0x430, 0x00000000,
@@ -2443,7 +2007,7 @@
0x559, 0x00000002,
0x55C, 0x00000050,
0x55D, 0x000000FF,
- 0x604, 0x00000001,
+ 0x604, 0x00000009,
0x605, 0x00000030,
0x607, 0x00000003,
0x608, 0x0000000E,
@@ -2475,9 +2039,10 @@
0x70A, 0x00000065,
0x70B, 0x00000087,
0x718, 0x00000040,
-
};
+u32 RTL8812AE_MAC_1T_ARRAYLEN = sizeof(RTL8812AE_MAC_REG_ARRAY) / sizeof(u32);
+
u32 RTL8821AE_MAC_REG_ARRAY[] = {
0x428, 0x0000000A,
0x429, 0x00000010,
@@ -2578,8 +2143,10 @@
0x718, 0x00000040,
};
+u32 RTL8821AE_MAC_1T_ARRAYLEN = sizeof(RTL8821AE_MAC_REG_ARRAY) / sizeof(u32);
+
u32 RTL8812AE_AGC_TAB_ARRAY[] = {
- 0xFF0F07D8, 0xABCD,
+ 0x80000001, 0x00000000, 0x40000000, 0x00000000,
0x81C, 0xFC000001,
0x81C, 0xFB020001,
0x81C, 0xFA040001,
@@ -2644,7 +2211,7 @@
0x81C, 0x217A0001,
0x81C, 0x217C0001,
0x81C, 0x217E0001,
- 0xFF0F07D0, 0xCDEF,
+ 0x90000001, 0x00000005, 0x40000000, 0x00000000,
0x81C, 0xF9000001,
0x81C, 0xF8020001,
0x81C, 0xF7040001,
@@ -2709,7 +2276,7 @@
0x81C, 0x217A0001,
0x81C, 0x217C0001,
0x81C, 0x217E0001,
- 0xCDCDCDCD, 0xCDCD,
+ 0xA0000000, 0x00000000,
0x81C, 0xFF000001,
0x81C, 0xFF020001,
0x81C, 0xFF040001,
@@ -2774,8 +2341,8 @@
0x81C, 0x417A0001,
0x81C, 0x417C0001,
0x81C, 0x417E0001,
- 0xFF0F07D8, 0xDEAD,
- 0xFF0F0180, 0xABCD,
+ 0xB0000000, 0x00000000,
+ 0x80000004, 0x00000000, 0x40000000, 0x00000000,
0x81C, 0xFC800001,
0x81C, 0xFB820001,
0x81C, 0xFA840001,
@@ -2840,332 +2407,7 @@
0x81C, 0x01FA0001,
0x81C, 0x01FC0001,
0x81C, 0x01FE0001,
- 0xFF0F0280, 0xCDEF,
- 0x81C, 0xFC800001,
- 0x81C, 0xFB820001,
- 0x81C, 0xFA840001,
- 0x81C, 0xF9860001,
- 0x81C, 0xF8880001,
- 0x81C, 0xF78A0001,
- 0x81C, 0xF68C0001,
- 0x81C, 0xF58E0001,
- 0x81C, 0xF4900001,
- 0x81C, 0xF3920001,
- 0x81C, 0xF2940001,
- 0x81C, 0xF1960001,
- 0x81C, 0xF0980001,
- 0x81C, 0xEF9A0001,
- 0x81C, 0xEE9C0001,
- 0x81C, 0xED9E0001,
- 0x81C, 0xECA00001,
- 0x81C, 0xEBA20001,
- 0x81C, 0xEAA40001,
- 0x81C, 0xE9A60001,
- 0x81C, 0xE8A80001,
- 0x81C, 0xE7AA0001,
- 0x81C, 0xE6AC0001,
- 0x81C, 0xE5AE0001,
- 0x81C, 0xE4B00001,
- 0x81C, 0xE3B20001,
- 0x81C, 0xA8B40001,
- 0x81C, 0xA7B60001,
- 0x81C, 0xA6B80001,
- 0x81C, 0xA5BA0001,
- 0x81C, 0xA4BC0001,
- 0x81C, 0xA3BE0001,
- 0x81C, 0xA2C00001,
- 0x81C, 0xA1C20001,
- 0x81C, 0x68C40001,
- 0x81C, 0x67C60001,
- 0x81C, 0x66C80001,
- 0x81C, 0x65CA0001,
- 0x81C, 0x64CC0001,
- 0x81C, 0x47CE0001,
- 0x81C, 0x46D00001,
- 0x81C, 0x45D20001,
- 0x81C, 0x44D40001,
- 0x81C, 0x43D60001,
- 0x81C, 0x42D80001,
- 0x81C, 0x08DA0001,
- 0x81C, 0x07DC0001,
- 0x81C, 0x06DE0001,
- 0x81C, 0x05E00001,
- 0x81C, 0x04E20001,
- 0x81C, 0x03E40001,
- 0x81C, 0x02E60001,
- 0x81C, 0x01E80001,
- 0x81C, 0x01EA0001,
- 0x81C, 0x01EC0001,
- 0x81C, 0x01EE0001,
- 0x81C, 0x01F00001,
- 0x81C, 0x01F20001,
- 0x81C, 0x01F40001,
- 0x81C, 0x01F60001,
- 0x81C, 0x01F80001,
- 0x81C, 0x01FA0001,
- 0x81C, 0x01FC0001,
- 0x81C, 0x01FE0001,
- 0xFF0F01C0, 0xCDEF,
- 0x81C, 0xFC800001,
- 0x81C, 0xFB820001,
- 0x81C, 0xFA840001,
- 0x81C, 0xF9860001,
- 0x81C, 0xF8880001,
- 0x81C, 0xF78A0001,
- 0x81C, 0xF68C0001,
- 0x81C, 0xF58E0001,
- 0x81C, 0xF4900001,
- 0x81C, 0xF3920001,
- 0x81C, 0xF2940001,
- 0x81C, 0xF1960001,
- 0x81C, 0xF0980001,
- 0x81C, 0xEF9A0001,
- 0x81C, 0xEE9C0001,
- 0x81C, 0xED9E0001,
- 0x81C, 0xECA00001,
- 0x81C, 0xEBA20001,
- 0x81C, 0xEAA40001,
- 0x81C, 0xE9A60001,
- 0x81C, 0xE8A80001,
- 0x81C, 0xE7AA0001,
- 0x81C, 0xE6AC0001,
- 0x81C, 0xE5AE0001,
- 0x81C, 0xE4B00001,
- 0x81C, 0xE3B20001,
- 0x81C, 0xA8B40001,
- 0x81C, 0xA7B60001,
- 0x81C, 0xA6B80001,
- 0x81C, 0xA5BA0001,
- 0x81C, 0xA4BC0001,
- 0x81C, 0xA3BE0001,
- 0x81C, 0xA2C00001,
- 0x81C, 0xA1C20001,
- 0x81C, 0x68C40001,
- 0x81C, 0x67C60001,
- 0x81C, 0x66C80001,
- 0x81C, 0x65CA0001,
- 0x81C, 0x64CC0001,
- 0x81C, 0x47CE0001,
- 0x81C, 0x46D00001,
- 0x81C, 0x45D20001,
- 0x81C, 0x44D40001,
- 0x81C, 0x43D60001,
- 0x81C, 0x42D80001,
- 0x81C, 0x08DA0001,
- 0x81C, 0x07DC0001,
- 0x81C, 0x06DE0001,
- 0x81C, 0x05E00001,
- 0x81C, 0x04E20001,
- 0x81C, 0x03E40001,
- 0x81C, 0x02E60001,
- 0x81C, 0x01E80001,
- 0x81C, 0x01EA0001,
- 0x81C, 0x01EC0001,
- 0x81C, 0x01EE0001,
- 0x81C, 0x01F00001,
- 0x81C, 0x01F20001,
- 0x81C, 0x01F40001,
- 0x81C, 0x01F60001,
- 0x81C, 0x01F80001,
- 0x81C, 0x01FA0001,
- 0x81C, 0x01FC0001,
- 0x81C, 0x01FE0001,
- 0xFF0F02C0, 0xCDEF,
- 0x81C, 0xFC800001,
- 0x81C, 0xFB820001,
- 0x81C, 0xFA840001,
- 0x81C, 0xF9860001,
- 0x81C, 0xF8880001,
- 0x81C, 0xF78A0001,
- 0x81C, 0xF68C0001,
- 0x81C, 0xF58E0001,
- 0x81C, 0xF4900001,
- 0x81C, 0xF3920001,
- 0x81C, 0xF2940001,
- 0x81C, 0xF1960001,
- 0x81C, 0xF0980001,
- 0x81C, 0xEF9A0001,
- 0x81C, 0xEE9C0001,
- 0x81C, 0xED9E0001,
- 0x81C, 0xECA00001,
- 0x81C, 0xEBA20001,
- 0x81C, 0xEAA40001,
- 0x81C, 0xE9A60001,
- 0x81C, 0xE8A80001,
- 0x81C, 0xE7AA0001,
- 0x81C, 0xE6AC0001,
- 0x81C, 0xE5AE0001,
- 0x81C, 0xE4B00001,
- 0x81C, 0xE3B20001,
- 0x81C, 0xA8B40001,
- 0x81C, 0xA7B60001,
- 0x81C, 0xA6B80001,
- 0x81C, 0xA5BA0001,
- 0x81C, 0xA4BC0001,
- 0x81C, 0xA3BE0001,
- 0x81C, 0xA2C00001,
- 0x81C, 0xA1C20001,
- 0x81C, 0x68C40001,
- 0x81C, 0x67C60001,
- 0x81C, 0x66C80001,
- 0x81C, 0x65CA0001,
- 0x81C, 0x64CC0001,
- 0x81C, 0x47CE0001,
- 0x81C, 0x46D00001,
- 0x81C, 0x45D20001,
- 0x81C, 0x44D40001,
- 0x81C, 0x43D60001,
- 0x81C, 0x42D80001,
- 0x81C, 0x08DA0001,
- 0x81C, 0x07DC0001,
- 0x81C, 0x06DE0001,
- 0x81C, 0x05E00001,
- 0x81C, 0x04E20001,
- 0x81C, 0x03E40001,
- 0x81C, 0x02E60001,
- 0x81C, 0x01E80001,
- 0x81C, 0x01EA0001,
- 0x81C, 0x01EC0001,
- 0x81C, 0x01EE0001,
- 0x81C, 0x01F00001,
- 0x81C, 0x01F20001,
- 0x81C, 0x01F40001,
- 0x81C, 0x01F60001,
- 0x81C, 0x01F80001,
- 0x81C, 0x01FA0001,
- 0x81C, 0x01FC0001,
- 0x81C, 0x01FE0001,
- 0xFF0F07D8, 0xCDEF,
- 0x81C, 0xFC800001,
- 0x81C, 0xFB820001,
- 0x81C, 0xFA840001,
- 0x81C, 0xF9860001,
- 0x81C, 0xF8880001,
- 0x81C, 0xF78A0001,
- 0x81C, 0xF68C0001,
- 0x81C, 0xF58E0001,
- 0x81C, 0xF4900001,
- 0x81C, 0xF3920001,
- 0x81C, 0xF2940001,
- 0x81C, 0xF1960001,
- 0x81C, 0xF0980001,
- 0x81C, 0xEF9A0001,
- 0x81C, 0xEE9C0001,
- 0x81C, 0xED9E0001,
- 0x81C, 0xECA00001,
- 0x81C, 0xEBA20001,
- 0x81C, 0xEAA40001,
- 0x81C, 0xE9A60001,
- 0x81C, 0xE8A80001,
- 0x81C, 0xE7AA0001,
- 0x81C, 0xE6AC0001,
- 0x81C, 0xE5AE0001,
- 0x81C, 0xE4B00001,
- 0x81C, 0xE3B20001,
- 0x81C, 0xA8B40001,
- 0x81C, 0xA7B60001,
- 0x81C, 0xA6B80001,
- 0x81C, 0xA5BA0001,
- 0x81C, 0xA4BC0001,
- 0x81C, 0xA3BE0001,
- 0x81C, 0xA2C00001,
- 0x81C, 0xA1C20001,
- 0x81C, 0x68C40001,
- 0x81C, 0x67C60001,
- 0x81C, 0x66C80001,
- 0x81C, 0x65CA0001,
- 0x81C, 0x64CC0001,
- 0x81C, 0x47CE0001,
- 0x81C, 0x46D00001,
- 0x81C, 0x45D20001,
- 0x81C, 0x44D40001,
- 0x81C, 0x43D60001,
- 0x81C, 0x42D80001,
- 0x81C, 0x08DA0001,
- 0x81C, 0x07DC0001,
- 0x81C, 0x06DE0001,
- 0x81C, 0x05E00001,
- 0x81C, 0x04E20001,
- 0x81C, 0x03E40001,
- 0x81C, 0x02E60001,
- 0x81C, 0x01E80001,
- 0x81C, 0x01EA0001,
- 0x81C, 0x01EC0001,
- 0x81C, 0x01EE0001,
- 0x81C, 0x01F00001,
- 0x81C, 0x01F20001,
- 0x81C, 0x01F40001,
- 0x81C, 0x01F60001,
- 0x81C, 0x01F80001,
- 0x81C, 0x01FA0001,
- 0x81C, 0x01FC0001,
- 0x81C, 0x01FE0001,
- 0xFF0F07D0, 0xCDEF,
- 0x81C, 0xFC800001,
- 0x81C, 0xFB820001,
- 0x81C, 0xFA840001,
- 0x81C, 0xF9860001,
- 0x81C, 0xF8880001,
- 0x81C, 0xF78A0001,
- 0x81C, 0xF68C0001,
- 0x81C, 0xF58E0001,
- 0x81C, 0xF4900001,
- 0x81C, 0xF3920001,
- 0x81C, 0xF2940001,
- 0x81C, 0xF1960001,
- 0x81C, 0xF0980001,
- 0x81C, 0xEF9A0001,
- 0x81C, 0xEE9C0001,
- 0x81C, 0xED9E0001,
- 0x81C, 0xECA00001,
- 0x81C, 0xEBA20001,
- 0x81C, 0xEAA40001,
- 0x81C, 0xE9A60001,
- 0x81C, 0xE8A80001,
- 0x81C, 0xE7AA0001,
- 0x81C, 0xE6AC0001,
- 0x81C, 0xE5AE0001,
- 0x81C, 0xE4B00001,
- 0x81C, 0xE3B20001,
- 0x81C, 0xA8B40001,
- 0x81C, 0xA7B60001,
- 0x81C, 0xA6B80001,
- 0x81C, 0xA5BA0001,
- 0x81C, 0xA4BC0001,
- 0x81C, 0xA3BE0001,
- 0x81C, 0xA2C00001,
- 0x81C, 0xA1C20001,
- 0x81C, 0x68C40001,
- 0x81C, 0x67C60001,
- 0x81C, 0x66C80001,
- 0x81C, 0x65CA0001,
- 0x81C, 0x64CC0001,
- 0x81C, 0x47CE0001,
- 0x81C, 0x46D00001,
- 0x81C, 0x45D20001,
- 0x81C, 0x44D40001,
- 0x81C, 0x43D60001,
- 0x81C, 0x42D80001,
- 0x81C, 0x08DA0001,
- 0x81C, 0x07DC0001,
- 0x81C, 0x06DE0001,
- 0x81C, 0x05E00001,
- 0x81C, 0x04E20001,
- 0x81C, 0x03E40001,
- 0x81C, 0x02E60001,
- 0x81C, 0x01E80001,
- 0x81C, 0x01EA0001,
- 0x81C, 0x01EC0001,
- 0x81C, 0x01EE0001,
- 0x81C, 0x01F00001,
- 0x81C, 0x01F20001,
- 0x81C, 0x01F40001,
- 0x81C, 0x01F60001,
- 0x81C, 0x01F80001,
- 0x81C, 0x01FA0001,
- 0x81C, 0x01FC0001,
- 0x81C, 0x01FE0001,
- 0xCDCDCDCD, 0xCDCD,
+ 0xA0000000, 0x00000000,
0x81C, 0xFF800001,
0x81C, 0xFF820001,
0x81C, 0xFF840001,
@@ -3230,14 +2472,16 @@
0x81C, 0x01FA0001,
0x81C, 0x01FC0001,
0x81C, 0x01FE0001,
- 0xFF0F0180, 0xDEAD,
+ 0xB0000000, 0x00000000,
0xC50, 0x00000022,
0xC50, 0x00000020,
0xE50, 0x00000022,
0xE50, 0x00000020,
-
};
+u32 RTL8812AE_AGC_TAB_1TARRAYLEN =
+ sizeof(RTL8812AE_AGC_TAB_ARRAY) / sizeof(u32);
+
u32 RTL8821AE_AGC_TAB_ARRAY[] = {
0x81C, 0xBF000001,
0x81C, 0xBF020001,
@@ -3430,9 +2674,11 @@
0x81C, 0x017E0101,
0xC50, 0x00000022,
0xC50, 0x00000020,
-
};
+u32 RTL8821AE_AGC_TAB_1TARRAYLEN =
+ sizeof(RTL8821AE_AGC_TAB_ARRAY) / sizeof(u32);
+
/******************************************************************************
* TXPWR_LMT.TXT
******************************************************************************/
@@ -3717,9 +2963,9 @@
"FCC", "5G", "20M", "OFDM", "1T", "100", "30",
"ETSI", "5G", "20M", "OFDM", "1T", "100", "32",
"MKK", "5G", "20M", "OFDM", "1T", "100", "32",
- "FCC", "5G", "20M", "OFDM", "1T", "114", "30",
- "ETSI", "5G", "20M", "OFDM", "1T", "114", "32",
- "MKK", "5G", "20M", "OFDM", "1T", "114", "32",
+ "FCC", "5G", "20M", "OFDM", "1T", "104", "30",
+ "ETSI", "5G", "20M", "OFDM", "1T", "104", "32",
+ "MKK", "5G", "20M", "OFDM", "1T", "104", "32",
"FCC", "5G", "20M", "OFDM", "1T", "108", "32",
"ETSI", "5G", "20M", "OFDM", "1T", "108", "32",
"MKK", "5G", "20M", "OFDM", "1T", "108", "32",
@@ -3789,9 +3035,9 @@
"FCC", "5G", "20M", "HT", "1T", "100", "30",
"ETSI", "5G", "20M", "HT", "1T", "100", "32",
"MKK", "5G", "20M", "HT", "1T", "100", "32",
- "FCC", "5G", "20M", "HT", "1T", "114", "30",
- "ETSI", "5G", "20M", "HT", "1T", "114", "32",
- "MKK", "5G", "20M", "HT", "1T", "114", "32",
+ "FCC", "5G", "20M", "HT", "1T", "104", "30",
+ "ETSI", "5G", "20M", "HT", "1T", "104", "32",
+ "MKK", "5G", "20M", "HT", "1T", "104", "32",
"FCC", "5G", "20M", "HT", "1T", "108", "32",
"ETSI", "5G", "20M", "HT", "1T", "108", "32",
"MKK", "5G", "20M", "HT", "1T", "108", "32",
@@ -3861,9 +3107,9 @@
"FCC", "5G", "20M", "HT", "2T", "100", "28",
"ETSI", "5G", "20M", "HT", "2T", "100", "30",
"MKK", "5G", "20M", "HT", "2T", "100", "30",
- "FCC", "5G", "20M", "HT", "2T", "114", "28",
- "ETSI", "5G", "20M", "HT", "2T", "114", "30",
- "MKK", "5G", "20M", "HT", "2T", "114", "30",
+ "FCC", "5G", "20M", "HT", "2T", "104", "28",
+ "ETSI", "5G", "20M", "HT", "2T", "104", "30",
+ "MKK", "5G", "20M", "HT", "2T", "104", "30",
"FCC", "5G", "20M", "HT", "2T", "108", "30",
"ETSI", "5G", "20M", "HT", "2T", "108", "30",
"MKK", "5G", "20M", "HT", "2T", "108", "30",
@@ -4004,6 +3250,8 @@
"MKK", "5G", "80M", "VHT", "2T", "155", "63"
};
+u32 RTL8812AE_TXPWR_LMT_ARRAY_LEN = sizeof(RTL8812AE_TXPWR_LMT) / sizeof(u8 *);
+
u8 *RTL8821AE_TXPWR_LMT[] = {
"FCC", "2.4G", "20M", "CCK", "1T", "01", "32",
"ETSI", "2.4G", "20M", "CCK", "1T", "01", "32",
@@ -4284,9 +3532,9 @@
"FCC", "5G", "20M", "OFDM", "1T", "100", "32",
"ETSI", "5G", "20M", "OFDM", "1T", "100", "30",
"MKK", "5G", "20M", "OFDM", "1T", "100", "30",
- "FCC", "5G", "20M", "OFDM", "1T", "114", "32",
- "ETSI", "5G", "20M", "OFDM", "1T", "114", "30",
- "MKK", "5G", "20M", "OFDM", "1T", "114", "30",
+ "FCC", "5G", "20M", "OFDM", "1T", "104", "32",
+ "ETSI", "5G", "20M", "OFDM", "1T", "104", "30",
+ "MKK", "5G", "20M", "OFDM", "1T", "104", "30",
"FCC", "5G", "20M", "OFDM", "1T", "108", "32",
"ETSI", "5G", "20M", "OFDM", "1T", "108", "30",
"MKK", "5G", "20M", "OFDM", "1T", "108", "30",
@@ -4356,9 +3604,9 @@
"FCC", "5G", "20M", "HT", "1T", "100", "32",
"ETSI", "5G", "20M", "HT", "1T", "100", "30",
"MKK", "5G", "20M", "HT", "1T", "100", "30",
- "FCC", "5G", "20M", "HT", "1T", "114", "32",
- "ETSI", "5G", "20M", "HT", "1T", "114", "30",
- "MKK", "5G", "20M", "HT", "1T", "114", "30",
+ "FCC", "5G", "20M", "HT", "1T", "104", "32",
+ "ETSI", "5G", "20M", "HT", "1T", "104", "30",
+ "MKK", "5G", "20M", "HT", "1T", "104", "30",
"FCC", "5G", "20M", "HT", "1T", "108", "32",
"ETSI", "5G", "20M", "HT", "1T", "108", "30",
"MKK", "5G", "20M", "HT", "1T", "108", "30",
@@ -4428,9 +3676,9 @@
"FCC", "5G", "20M", "HT", "2T", "100", "28",
"ETSI", "5G", "20M", "HT", "2T", "100", "30",
"MKK", "5G", "20M", "HT", "2T", "100", "30",
- "FCC", "5G", "20M", "HT", "2T", "114", "28",
- "ETSI", "5G", "20M", "HT", "2T", "114", "30",
- "MKK", "5G", "20M", "HT", "2T", "114", "30",
+ "FCC", "5G", "20M", "HT", "2T", "104", "28",
+ "ETSI", "5G", "20M", "HT", "2T", "104", "30",
+ "MKK", "5G", "20M", "HT", "2T", "104", "30",
"FCC", "5G", "20M", "HT", "2T", "108", "30",
"ETSI", "5G", "20M", "HT", "2T", "108", "30",
"MKK", "5G", "20M", "HT", "2T", "108", "30",
@@ -4570,3 +3818,5 @@
"ETSI", "5G", "80M", "VHT", "2T", "155", "30",
"MKK", "5G", "80M", "VHT", "2T", "155", "63"
};
+
+u32 RTL8821AE_TXPWR_LMT_ARRAY_LEN = sizeof(RTL8821AE_TXPWR_LMT) / sizeof(u8 *);
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/table.h b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/table.h
index 24bcff6..36c2388 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/table.h
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/table.h
@@ -29,32 +29,30 @@
#define __RTL8821AE_TABLE__H_
#include <linux/types.h>
-#define RTL8821AEPHY_REG_1TARRAYLEN 344
+extern u32 RTL8821AE_PHY_REG_1TARRAYLEN;
extern u32 RTL8821AE_PHY_REG_ARRAY[];
-#define RTL8812AEPHY_REG_1TARRAYLEN 490
+extern u32 RTL8812AE_PHY_REG_1TARRAYLEN;
extern u32 RTL8812AE_PHY_REG_ARRAY[];
-#define RTL8821AEPHY_REG_ARRAY_PGLEN 90
+extern u32 RTL8821AE_PHY_REG_ARRAY_PGLEN;
extern u32 RTL8821AE_PHY_REG_ARRAY_PG[];
-#define RTL8812AEPHY_REG_ARRAY_PGLEN 276
+extern u32 RTL8812AE_PHY_REG_ARRAY_PGLEN;
extern u32 RTL8812AE_PHY_REG_ARRAY_PG[];
-/* #define RTL8723BE_RADIOA_1TARRAYLEN 206 */
-/* extern u8 *RTL8821AE_TXPWR_LMT_ARRAY[]; */
-#define RTL8812AE_RADIOA_1TARRAYLEN 1264
+extern u32 RTL8812AE_RADIOA_1TARRAYLEN;
extern u32 RTL8812AE_RADIOA_ARRAY[];
-#define RTL8812AE_RADIOB_1TARRAYLEN 1240
+extern u32 RTL8812AE_RADIOB_1TARRAYLEN;
extern u32 RTL8812AE_RADIOB_ARRAY[];
-#define RTL8821AE_RADIOA_1TARRAYLEN 1176
+extern u32 RTL8821AE_RADIOA_1TARRAYLEN;
extern u32 RTL8821AE_RADIOA_ARRAY[];
-#define RTL8821AEMAC_1T_ARRAYLEN 194
+extern u32 RTL8821AE_MAC_1T_ARRAYLEN;
extern u32 RTL8821AE_MAC_REG_ARRAY[];
-#define RTL8812AEMAC_1T_ARRAYLEN 214
+extern u32 RTL8812AE_MAC_1T_ARRAYLEN;
extern u32 RTL8812AE_MAC_REG_ARRAY[];
-#define RTL8821AEAGCTAB_1TARRAYLEN 382
+extern u32 RTL8821AE_AGC_TAB_1TARRAYLEN;
extern u32 RTL8821AE_AGC_TAB_ARRAY[];
-#define RTL8812AEAGCTAB_1TARRAYLEN 1312
+extern u32 RTL8812AE_AGC_TAB_1TARRAYLEN;
extern u32 RTL8812AE_AGC_TAB_ARRAY[];
-#define RTL8812AE_TXPWR_LMT_ARRAY_LEN 3948
+extern u32 RTL8812AE_TXPWR_LMT_ARRAY_LEN;
extern u8 *RTL8812AE_TXPWR_LMT[];
-#define RTL8821AE_TXPWR_LMT_ARRAY_LEN 3948
+extern u32 RTL8821AE_TXPWR_LMT_ARRAY_LEN;
extern u8 *RTL8821AE_TXPWR_LMT[];
#endif
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/trx.c b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/trx.c
index 1080981..03665e8 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/trx.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/trx.c
@@ -520,18 +520,18 @@
rx_status->flag |= RX_FLAG_FAILED_FCS_CRC;
if (status->rx_packet_bw == HT_CHANNEL_WIDTH_20_40)
- rx_status->flag |= RX_FLAG_40MHZ;
+ rx_status->bw = RATE_INFO_BW_40;
else if (status->rx_packet_bw == HT_CHANNEL_WIDTH_80)
- rx_status->vht_flag |= RX_VHT_FLAG_80MHZ;
+ rx_status->bw = RATE_INFO_BW_80;
if (status->is_ht)
- rx_status->flag |= RX_FLAG_HT;
+ rx_status->encoding = RX_ENC_HT;
if (status->is_vht)
- rx_status->flag |= RX_FLAG_VHT;
+ rx_status->encoding = RX_ENC_VHT;
if (status->is_short_gi)
- rx_status->flag |= RX_FLAG_SHORT_GI;
+ rx_status->enc_flags |= RX_ENC_FLAG_SHORT_GI;
- rx_status->vht_nss = status->vht_nss;
+ rx_status->nss = status->vht_nss;
rx_status->flag |= RX_FLAG_MACTIME_START;
/* hw will set status->decrypted true, if it finds the
diff --git a/drivers/net/wireless/realtek/rtlwifi/wifi.h b/drivers/net/wireless/realtek/rtlwifi/wifi.h
index 65ef42b..c0d2601 100644
--- a/drivers/net/wireless/realtek/rtlwifi/wifi.h
+++ b/drivers/net/wireless/realtek/rtlwifi/wifi.h
@@ -1529,6 +1529,10 @@
u8 external_lna_2g;
u8 external_pa_5g;
u8 external_lna_5g;
+ u8 type_glna;
+ u8 type_gpa;
+ u8 type_alna;
+ u8 type_apa;
u8 rfe_type;
/*firmware */
@@ -2933,6 +2937,14 @@
rtlpriv->io.read8_sync(rtlpriv, addr);
}
+static inline void rtl_write_byte_with_val32(struct ieee80211_hw *hw,
+ u32 addr, u32 val8)
+{
+ struct rtl_priv *rtlpriv = rtl_priv(hw);
+
+ rtl_write_byte(rtlpriv, addr, (u8)val8);
+}
+
static inline void rtl_write_word(struct rtl_priv *rtlpriv, u32 addr, u16 val16)
{
rtlpriv->io.write16_async(rtlpriv, addr, val16);
@@ -2966,6 +2978,12 @@
rtlpriv->cfg->ops->set_bbreg(hw, regaddr, bitmask, data);
}
+static inline void rtl_set_bbreg_with_dwmask(struct ieee80211_hw *hw,
+ u32 regaddr, u32 data)
+{
+ rtl_set_bbreg(hw, regaddr, 0xffffffff, data);
+}
+
static inline u32 rtl_get_rfreg(struct ieee80211_hw *hw,
enum radio_path rfpath, u32 regaddr,
u32 bitmask)
diff --git a/drivers/net/wireless/rndis_wlan.c b/drivers/net/wireless/rndis_wlan.c
index 785334f..9935bd0 100644
--- a/drivers/net/wireless/rndis_wlan.c
+++ b/drivers/net/wireless/rndis_wlan.c
@@ -479,7 +479,7 @@
*/
static int rndis_change_virtual_intf(struct wiphy *wiphy,
struct net_device *dev,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params);
static int rndis_scan(struct wiphy *wiphy,
@@ -1857,7 +1857,7 @@
*/
static int rndis_change_virtual_intf(struct wiphy *wiphy,
struct net_device *dev,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct rndis_wlan_private *priv = wiphy_priv(wiphy);
@@ -2830,15 +2830,22 @@
}
if (priv->infra_mode == NDIS_80211_INFRA_INFRA) {
- if (!roamed)
+ if (!roamed) {
cfg80211_connect_result(usbdev->net, bssid, req_ie,
req_ie_len, resp_ie,
resp_ie_len, 0, GFP_KERNEL);
- else
- cfg80211_roamed(usbdev->net,
- get_current_channel(usbdev, NULL),
- bssid, req_ie, req_ie_len,
- resp_ie, resp_ie_len, GFP_KERNEL);
+ } else {
+ struct cfg80211_roam_info roam_info = {
+ .channel = get_current_channel(usbdev, NULL),
+ .bssid = bssid,
+ .req_ie = req_ie,
+ .req_ie_len = req_ie_len,
+ .resp_ie = resp_ie,
+ .resp_ie_len = resp_ie_len,
+ };
+
+ cfg80211_roamed(usbdev->net, &roam_info, GFP_KERNEL);
+ }
} else if (priv->infra_mode == NDIS_80211_INFRA_ADHOC)
cfg80211_ibss_joined(usbdev->net, bssid,
get_current_channel(usbdev, NULL),
@@ -3392,6 +3399,7 @@
.ndo_stop = usbnet_stop,
.ndo_start_xmit = usbnet_start_xmit,
.ndo_tx_timeout = usbnet_tx_timeout,
+ .ndo_get_stats64 = usbnet_get_stats64,
.ndo_set_mac_address = eth_mac_addr,
.ndo_validate_addr = eth_validate_addr,
.ndo_set_rx_mode = rndis_wlan_set_multicast_list,
@@ -3427,6 +3435,10 @@
/* because rndis_command() sleeps we need to use workqueue */
priv->workqueue = create_singlethread_workqueue("rndis_wlan");
+ if (!priv->workqueue) {
+ wiphy_free(wiphy);
+ return -ENOMEM;
+ }
INIT_WORK(&priv->work, rndis_wlan_worker);
INIT_DELAYED_WORK(&priv->dev_poller_work, rndis_device_poller);
INIT_DELAYED_WORK(&priv->scan_work, rndis_get_scan_results);
diff --git a/drivers/net/wireless/rsi/rsi_91x_mac80211.c b/drivers/net/wireless/rsi/rsi_91x_mac80211.c
index e321647..021e5ac 100644
--- a/drivers/net/wireless/rsi/rsi_91x_mac80211.c
+++ b/drivers/net/wireless/rsi/rsi_91x_mac80211.c
@@ -1261,6 +1261,8 @@
wiphy->reg_notifier = rsi_reg_notify;
+ wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
status = ieee80211_register_hw(hw);
if (status)
return status;
diff --git a/drivers/net/wireless/st/cw1200/cw1200_sdio.c b/drivers/net/wireless/st/cw1200/cw1200_sdio.c
index d3acc85..709f56e 100644
--- a/drivers/net/wireless/st/cw1200/cw1200_sdio.c
+++ b/drivers/net/wireless/st/cw1200/cw1200_sdio.c
@@ -10,6 +10,7 @@
*/
#include <linux/module.h>
+#include <linux/interrupt.h>
#include <linux/gpio.h>
#include <linux/delay.h>
#include <linux/mmc/host.h>
diff --git a/drivers/net/wireless/st/cw1200/txrx.c b/drivers/net/wireless/st/cw1200/txrx.c
index 3d17028..cd63ffe 100644
--- a/drivers/net/wireless/st/cw1200/txrx.c
+++ b/drivers/net/wireless/st/cw1200/txrx.c
@@ -1085,7 +1085,7 @@
hdr->band);
if (arg->rx_rate >= 14) {
- hdr->flag |= RX_FLAG_HT;
+ hdr->encoding = RX_ENC_HT;
hdr->rate_idx = arg->rx_rate - 14;
} else if (arg->rx_rate >= 4) {
hdr->rate_idx = arg->rx_rate - 2;
diff --git a/drivers/net/wireless/ti/wl1251/rx.c b/drivers/net/wireless/ti/wl1251/rx.c
index a27d4c2..50fb2a4 100644
--- a/drivers/net/wireless/ti/wl1251/rx.c
+++ b/drivers/net/wireless/ti/wl1251/rx.c
@@ -141,7 +141,7 @@
}
if (desc->mod_pre & SHORT_PREAMBLE_BIT)
- status->flag |= RX_FLAG_SHORTPRE;
+ status->enc_flags |= RX_ENC_FLAG_SHORTPRE;
}
static void wl1251_rx_body(struct wl1251 *wl,
diff --git a/drivers/net/wireless/ti/wlcore/debugfs.c b/drivers/net/wireless/ti/wlcore/debugfs.c
index 58e148d..de7e2a5 100644
--- a/drivers/net/wireless/ti/wlcore/debugfs.c
+++ b/drivers/net/wireless/ti/wlcore/debugfs.c
@@ -1249,7 +1249,7 @@
}
if (wl->conf.fwlog.output == 0) {
- wl1271_warning("iligal opperation - fw logger disabled by default, please change mode via wlconf");
+ wl1271_warning("invalid operation - fw logger disabled by default, please change mode via wlconf");
return -EINVAL;
}
diff --git a/drivers/net/wireless/ti/wlcore/main.c b/drivers/net/wireless/ti/wlcore/main.c
index a21fda9..382ec15 100644
--- a/drivers/net/wireless/ti/wlcore/main.c
+++ b/drivers/net/wireless/ti/wlcore/main.c
@@ -6128,6 +6128,7 @@
wl->hw->wiphy->max_scan_ie_len = WL1271_CMD_TEMPL_MAX_SIZE -
sizeof(struct ieee80211_header);
+ wl->hw->wiphy->max_sched_scan_reqs = 1;
wl->hw->wiphy->max_sched_scan_ie_len = WL1271_CMD_TEMPL_MAX_SIZE -
sizeof(struct ieee80211_header);
@@ -6135,7 +6136,6 @@
wl->hw->wiphy->flags |= WIPHY_FLAG_AP_UAPSD |
WIPHY_FLAG_HAS_REMAIN_ON_CHANNEL |
- WIPHY_FLAG_SUPPORTS_SCHED_SCAN |
WIPHY_FLAG_HAS_CHANNEL_SWITCH;
wl->hw->wiphy->features |= NL80211_FEATURE_AP_SCAN;
diff --git a/drivers/net/wireless/ti/wlcore/rx.c b/drivers/net/wireless/ti/wlcore/rx.c
index b9e1404..52a55f9 100644
--- a/drivers/net/wireless/ti/wlcore/rx.c
+++ b/drivers/net/wireless/ti/wlcore/rx.c
@@ -72,7 +72,7 @@
/* 11n support */
if (desc->rate <= wl->hw_min_ht_rate)
- status->flag |= RX_FLAG_HT;
+ status->encoding = RX_ENC_HT;
/*
* Read the signal level and antenna diversity indication.
diff --git a/drivers/net/wireless/ti/wlcore/testmode.c b/drivers/net/wireless/ti/wlcore/testmode.c
index ddad58f..009ec07 100644
--- a/drivers/net/wireless/ti/wlcore/testmode.c
+++ b/drivers/net/wireless/ti/wlcore/testmode.c
@@ -366,7 +366,8 @@
u32 nla_cmd;
int err;
- err = nla_parse(tb, WL1271_TM_ATTR_MAX, data, len, wl1271_tm_policy);
+ err = nla_parse(tb, WL1271_TM_ATTR_MAX, data, len, wl1271_tm_policy,
+ NULL);
if (err)
return err;
diff --git a/drivers/net/wireless/ti/wlcore/vendor_cmd.c b/drivers/net/wireless/ti/wlcore/vendor_cmd.c
index fd4e9ba..5c0bcb1 100644
--- a/drivers/net/wireless/ti/wlcore/vendor_cmd.c
+++ b/drivers/net/wireless/ti/wlcore/vendor_cmd.c
@@ -41,7 +41,7 @@
return -EINVAL;
ret = nla_parse(tb, MAX_WLCORE_VENDOR_ATTR, data, data_len,
- wlcore_vendor_attr_policy);
+ wlcore_vendor_attr_policy, NULL);
if (ret)
return ret;
@@ -116,7 +116,7 @@
return -EINVAL;
ret = nla_parse(tb, MAX_WLCORE_VENDOR_ATTR, data, data_len,
- wlcore_vendor_attr_policy);
+ wlcore_vendor_attr_policy, NULL);
if (ret)
return ret;
diff --git a/drivers/net/wireless/zydas/zd1211rw/zd_mac.c b/drivers/net/wireless/zydas/zd1211rw/zd_mac.c
index 3e37a04..fe6517a6 100644
--- a/drivers/net/wireless/zydas/zd1211rw/zd_mac.c
+++ b/drivers/net/wireless/zydas/zd1211rw/zd_mac.c
@@ -1408,6 +1408,8 @@
BIT(NL80211_IFTYPE_ADHOC) |
BIT(NL80211_IFTYPE_AP);
+ wiphy_ext_feature_set(hw->wiphy, NL80211_EXT_FEATURE_CQM_RSSI_LIST);
+
hw->max_signal = 100;
hw->queues = 1;
hw->extra_tx_headroom = sizeof(struct zd_ctrlset);
diff --git a/drivers/net/wireless/zydas/zd1211rw/zd_usb.c b/drivers/net/wireless/zydas/zd1211rw/zd_usb.c
index c5effd6c..01ca1d5 100644
--- a/drivers/net/wireless/zydas/zd1211rw/zd_usb.c
+++ b/drivers/net/wireless/zydas/zd1211rw/zd_usb.c
@@ -1278,6 +1278,9 @@
u8 bulk_out_ep;
int r;
+ if (iface_desc->desc.bNumEndpoints < 2)
+ return -ENODEV;
+
/* Find bulk out endpoint */
for (r = 1; r >= 0; r--) {
endpoint = &iface_desc->endpoint[r].desc;
diff --git a/drivers/nfc/Kconfig b/drivers/nfc/Kconfig
index 9d23692..c420848 100644
--- a/drivers/nfc/Kconfig
+++ b/drivers/nfc/Kconfig
@@ -5,17 +5,6 @@
menu "Near Field Communication (NFC) devices"
depends on NFC
-config NFC_WILINK
- tristate "Texas Instruments NFC WiLink driver"
- depends on TI_ST && NFC_NCI
- help
- This enables the NFC driver for Texas Instrument's BT/FM/GPS/NFC
- combo devices. This makes use of shared transport line discipline
- core driver to communicate with the NFC core of the combo chip.
-
- Say Y here to compile support for Texas Instrument's NFC WiLink driver
- into the kernel or say M to compile it as module.
-
config NFC_TRF7970A
tristate "Texas Instruments TRF7970a NFC driver"
depends on SPI && NFC_DIGITAL
diff --git a/drivers/nfc/Makefile b/drivers/nfc/Makefile
index bab8ef0..640b727 100644
--- a/drivers/nfc/Makefile
+++ b/drivers/nfc/Makefile
@@ -6,7 +6,6 @@
obj-$(CONFIG_NFC_PN544) += pn544/
obj-$(CONFIG_NFC_MICROREAD) += microread/
obj-$(CONFIG_NFC_PN533) += pn533/
-obj-$(CONFIG_NFC_WILINK) += nfcwilink.o
obj-$(CONFIG_NFC_MEI_PHY) += mei_phy.o
obj-$(CONFIG_NFC_SIM) += nfcsim.o
obj-$(CONFIG_NFC_PORT100) += port100.o
diff --git a/drivers/nfc/fdp/i2c.c b/drivers/nfc/fdp/i2c.c
index 5e797d5..712936f 100644
--- a/drivers/nfc/fdp/i2c.c
+++ b/drivers/nfc/fdp/i2c.c
@@ -210,14 +210,14 @@
struct sk_buff *skb;
int r;
- client = phy->i2c_dev;
- dev_dbg(&client->dev, "%s\n", __func__);
-
if (!phy || irq != phy->i2c_dev->irq) {
WARN_ON_ONCE(1);
return IRQ_NONE;
}
+ client = phy->i2c_dev;
+ dev_dbg(&client->dev, "%s\n", __func__);
+
r = fdp_nci_i2c_read(phy, &skb);
if (r == -EREMOTEIO)
diff --git a/drivers/nfc/nfcmrvl/fw_dnld.c b/drivers/nfc/nfcmrvl/fw_dnld.c
index f8dcdf4..c38bdd6 100644
--- a/drivers/nfc/nfcmrvl/fw_dnld.c
+++ b/drivers/nfc/nfcmrvl/fw_dnld.c
@@ -17,7 +17,7 @@
*/
#include <linux/module.h>
-#include <linux/unaligned/access_ok.h>
+#include <asm/unaligned.h>
#include <linux/firmware.h>
#include <linux/nfc.h>
#include <net/nfc/nci.h>
@@ -281,12 +281,11 @@
return -EINVAL;
}
skb_pull(skb, 1);
- memcpy(&len, skb->data, 2);
+ len = get_unaligned_le16(skb->data);
skb_pull(skb, 2);
+ comp_len = get_unaligned_le16(skb->data);
memcpy(&comp_len, skb->data, 2);
skb_pull(skb, 2);
- len = get_unaligned_le16(&len);
- comp_len = get_unaligned_le16(&comp_len);
if (((~len) & 0xFFFF) != comp_len) {
nfc_err(priv->dev, "bad len complement: %x %x %x",
len, comp_len, (~len & 0xFFFF));
diff --git a/drivers/nfc/nfcmrvl/spi.c b/drivers/nfc/nfcmrvl/spi.c
index a7faa0b..8e0ddb4 100644
--- a/drivers/nfc/nfcmrvl/spi.c
+++ b/drivers/nfc/nfcmrvl/spi.c
@@ -26,7 +26,6 @@
#include <net/nfc/nci.h>
#include <net/nfc/nci_core.h>
#include <linux/spi/spi.h>
-#include <linux/gpio.h>
#include "nfcmrvl.h"
#define SPI_WAIT_HANDSHAKE 1
@@ -96,10 +95,9 @@
/* Send the SPI packet */
err = nci_spi_send(drv_data->nci_spi, &drv_data->handshake_completion,
skb);
- if (err != 0) {
+ if (err)
nfc_err(priv->dev, "spi_send failed %d", err);
- kfree_skb(skb);
- }
+
return err;
}
diff --git a/drivers/nfc/nfcwilink.c b/drivers/nfc/nfcwilink.c
deleted file mode 100644
index 3fbd18b..0000000
--- a/drivers/nfc/nfcwilink.c
+++ /dev/null
@@ -1,578 +0,0 @@
-/*
- * Texas Instrument's NFC Driver For Shared Transport.
- *
- * NFC Driver acts as interface between NCI core and
- * TI Shared Transport Layer.
- *
- * Copyright (C) 2011 Texas Instruments, Inc.
- *
- * Written by Ilan Elias <ilane@ti.com>
- *
- * Acknowledgements:
- * This file is based on btwilink.c, which was written
- * by Raja Mani and Pavan Savoy.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- *
- */
-#include <linux/platform_device.h>
-#include <linux/module.h>
-#include <linux/types.h>
-#include <linux/firmware.h>
-#include <linux/nfc.h>
-#include <net/nfc/nci.h>
-#include <net/nfc/nci_core.h>
-#include <linux/ti_wilink_st.h>
-
-#define NFCWILINK_CHNL 12
-#define NFCWILINK_OPCODE 7
-#define NFCWILINK_MAX_FRAME_SIZE 300
-#define NFCWILINK_HDR_LEN 4
-#define NFCWILINK_OFFSET_LEN_IN_HDR 1
-#define NFCWILINK_LEN_SIZE 2
-#define NFCWILINK_REGISTER_TIMEOUT 8000 /* 8 sec */
-#define NFCWILINK_CMD_TIMEOUT 5000 /* 5 sec */
-
-#define BTS_FILE_NAME_MAX_SIZE 40
-#define BTS_FILE_HDR_MAGIC 0x42535442
-#define BTS_FILE_CMD_MAX_LEN 0xff
-#define BTS_FILE_ACTION_TYPE_SEND_CMD 1
-
-#define NCI_VS_NFCC_INFO_CMD_GID 0x2f
-#define NCI_VS_NFCC_INFO_CMD_OID 0x12
-#define NCI_VS_NFCC_INFO_RSP_GID 0x4f
-#define NCI_VS_NFCC_INFO_RSP_OID 0x12
-
-struct nfcwilink_hdr {
- __u8 chnl;
- __u8 opcode;
- __le16 len;
-} __packed;
-
-struct nci_vs_nfcc_info_cmd {
- __u8 gid;
- __u8 oid;
- __u8 plen;
-} __packed;
-
-struct nci_vs_nfcc_info_rsp {
- __u8 gid;
- __u8 oid;
- __u8 plen;
- __u8 status;
- __u8 hw_id;
- __u8 sw_ver_x;
- __u8 sw_ver_z;
- __u8 patch_id;
-} __packed;
-
-struct bts_file_hdr {
- __le32 magic;
- __le32 ver;
- __u8 rfu[24];
- __u8 actions[0];
-} __packed;
-
-struct bts_file_action {
- __le16 type;
- __le16 len;
- __u8 data[0];
-} __packed;
-
-struct nfcwilink {
- struct platform_device *pdev;
- struct nci_dev *ndev;
- unsigned long flags;
-
- int st_register_cb_status;
- long (*st_write) (struct sk_buff *);
-
- struct completion completed;
-
- struct nci_vs_nfcc_info_rsp nfcc_info;
-};
-
-/* NFCWILINK driver flags */
-enum {
- NFCWILINK_RUNNING,
- NFCWILINK_FW_DOWNLOAD,
-};
-
-static int nfcwilink_send(struct nci_dev *ndev, struct sk_buff *skb);
-
-static inline struct sk_buff *nfcwilink_skb_alloc(unsigned int len, gfp_t how)
-{
- struct sk_buff *skb;
-
- skb = alloc_skb(len + NFCWILINK_HDR_LEN, how);
- if (skb)
- skb_reserve(skb, NFCWILINK_HDR_LEN);
-
- return skb;
-}
-
-static void nfcwilink_fw_download_receive(struct nfcwilink *drv,
- struct sk_buff *skb)
-{
- struct nci_vs_nfcc_info_rsp *rsp = (void *)skb->data;
-
- /* Detect NCI_VS_NFCC_INFO_RSP and store the result */
- if ((skb->len > 3) && (rsp->gid == NCI_VS_NFCC_INFO_RSP_GID) &&
- (rsp->oid == NCI_VS_NFCC_INFO_RSP_OID)) {
- memcpy(&drv->nfcc_info, rsp,
- sizeof(struct nci_vs_nfcc_info_rsp));
- }
-
- kfree_skb(skb);
-
- complete(&drv->completed);
-}
-
-static int nfcwilink_get_bts_file_name(struct nfcwilink *drv, char *file_name)
-{
- struct nci_vs_nfcc_info_cmd *cmd;
- struct sk_buff *skb;
- unsigned long comp_ret;
- int rc;
-
- skb = nfcwilink_skb_alloc(sizeof(struct nci_vs_nfcc_info_cmd),
- GFP_KERNEL);
- if (!skb) {
- nfc_err(&drv->pdev->dev,
- "no memory for nci_vs_nfcc_info_cmd\n");
- return -ENOMEM;
- }
-
- cmd = (struct nci_vs_nfcc_info_cmd *)
- skb_put(skb, sizeof(struct nci_vs_nfcc_info_cmd));
- cmd->gid = NCI_VS_NFCC_INFO_CMD_GID;
- cmd->oid = NCI_VS_NFCC_INFO_CMD_OID;
- cmd->plen = 0;
-
- drv->nfcc_info.plen = 0;
-
- rc = nfcwilink_send(drv->ndev, skb);
- if (rc)
- return rc;
-
- comp_ret = wait_for_completion_timeout(&drv->completed,
- msecs_to_jiffies(NFCWILINK_CMD_TIMEOUT));
- dev_dbg(&drv->pdev->dev, "wait_for_completion_timeout returned %ld\n",
- comp_ret);
- if (comp_ret == 0) {
- nfc_err(&drv->pdev->dev,
- "timeout on wait_for_completion_timeout\n");
- return -ETIMEDOUT;
- }
-
- dev_dbg(&drv->pdev->dev, "nci_vs_nfcc_info_rsp: plen %d, status %d\n",
- drv->nfcc_info.plen, drv->nfcc_info.status);
-
- if ((drv->nfcc_info.plen != 5) || (drv->nfcc_info.status != 0)) {
- nfc_err(&drv->pdev->dev, "invalid nci_vs_nfcc_info_rsp\n");
- return -EINVAL;
- }
-
- snprintf(file_name, BTS_FILE_NAME_MAX_SIZE,
- "TINfcInit_%d.%d.%d.%d.bts",
- drv->nfcc_info.hw_id,
- drv->nfcc_info.sw_ver_x,
- drv->nfcc_info.sw_ver_z,
- drv->nfcc_info.patch_id);
-
- nfc_info(&drv->pdev->dev, "nfcwilink FW file name: %s\n", file_name);
-
- return 0;
-}
-
-static int nfcwilink_send_bts_cmd(struct nfcwilink *drv, __u8 *data, int len)
-{
- struct nfcwilink_hdr *hdr = (struct nfcwilink_hdr *)data;
- struct sk_buff *skb;
- unsigned long comp_ret;
- int rc;
-
- /* verify valid cmd for the NFC channel */
- if ((len <= sizeof(struct nfcwilink_hdr)) ||
- (len > BTS_FILE_CMD_MAX_LEN) ||
- (hdr->chnl != NFCWILINK_CHNL) ||
- (hdr->opcode != NFCWILINK_OPCODE)) {
- nfc_err(&drv->pdev->dev,
- "ignoring invalid bts cmd, len %d, chnl %d, opcode %d\n",
- len, hdr->chnl, hdr->opcode);
- return 0;
- }
-
- /* remove the ST header */
- len -= sizeof(struct nfcwilink_hdr);
- data += sizeof(struct nfcwilink_hdr);
-
- skb = nfcwilink_skb_alloc(len, GFP_KERNEL);
- if (!skb) {
- nfc_err(&drv->pdev->dev, "no memory for bts cmd\n");
- return -ENOMEM;
- }
-
- memcpy(skb_put(skb, len), data, len);
-
- rc = nfcwilink_send(drv->ndev, skb);
- if (rc)
- return rc;
-
- comp_ret = wait_for_completion_timeout(&drv->completed,
- msecs_to_jiffies(NFCWILINK_CMD_TIMEOUT));
- dev_dbg(&drv->pdev->dev, "wait_for_completion_timeout returned %ld\n",
- comp_ret);
- if (comp_ret == 0) {
- nfc_err(&drv->pdev->dev,
- "timeout on wait_for_completion_timeout\n");
- return -ETIMEDOUT;
- }
-
- return 0;
-}
-
-static int nfcwilink_download_fw(struct nfcwilink *drv)
-{
- unsigned char file_name[BTS_FILE_NAME_MAX_SIZE];
- const struct firmware *fw;
- __u16 action_type, action_len;
- __u8 *ptr;
- int len, rc;
-
- set_bit(NFCWILINK_FW_DOWNLOAD, &drv->flags);
-
- rc = nfcwilink_get_bts_file_name(drv, file_name);
- if (rc)
- goto exit;
-
- rc = request_firmware(&fw, file_name, &drv->pdev->dev);
- if (rc) {
- nfc_err(&drv->pdev->dev, "request_firmware failed %d\n", rc);
-
- /* if the file is not found, don't exit with failure */
- if (rc == -ENOENT)
- rc = 0;
-
- goto exit;
- }
-
- len = fw->size;
- ptr = (__u8 *)fw->data;
-
- if ((len == 0) || (ptr == NULL)) {
- dev_dbg(&drv->pdev->dev,
- "request_firmware returned size %d\n", len);
- goto release_fw;
- }
-
- if (__le32_to_cpu(((struct bts_file_hdr *)ptr)->magic) !=
- BTS_FILE_HDR_MAGIC) {
- nfc_err(&drv->pdev->dev, "wrong bts magic number\n");
- rc = -EINVAL;
- goto release_fw;
- }
-
- /* remove the BTS header */
- len -= sizeof(struct bts_file_hdr);
- ptr += sizeof(struct bts_file_hdr);
-
- while (len > 0) {
- action_type =
- __le16_to_cpu(((struct bts_file_action *)ptr)->type);
- action_len =
- __le16_to_cpu(((struct bts_file_action *)ptr)->len);
-
- dev_dbg(&drv->pdev->dev, "bts_file_action type %d, len %d\n",
- action_type, action_len);
-
- switch (action_type) {
- case BTS_FILE_ACTION_TYPE_SEND_CMD:
- rc = nfcwilink_send_bts_cmd(drv,
- ((struct bts_file_action *)ptr)->data,
- action_len);
- if (rc)
- goto release_fw;
- break;
- }
-
- /* advance to the next action */
- len -= (sizeof(struct bts_file_action) + action_len);
- ptr += (sizeof(struct bts_file_action) + action_len);
- }
-
-release_fw:
- release_firmware(fw);
-
-exit:
- clear_bit(NFCWILINK_FW_DOWNLOAD, &drv->flags);
- return rc;
-}
-
-/* Called by ST when registration is complete */
-static void nfcwilink_register_complete(void *priv_data, int data)
-{
- struct nfcwilink *drv = priv_data;
-
- /* store ST registration status */
- drv->st_register_cb_status = data;
-
- /* complete the wait in nfc_st_open() */
- complete(&drv->completed);
-}
-
-/* Called by ST when receive data is available */
-static long nfcwilink_receive(void *priv_data, struct sk_buff *skb)
-{
- struct nfcwilink *drv = priv_data;
- int rc;
-
- if (!skb)
- return -EFAULT;
-
- if (!drv) {
- kfree_skb(skb);
- return -EFAULT;
- }
-
- dev_dbg(&drv->pdev->dev, "receive entry, len %d\n", skb->len);
-
- /* strip the ST header
- (apart for the chnl byte, which is not received in the hdr) */
- skb_pull(skb, (NFCWILINK_HDR_LEN-1));
-
- if (test_bit(NFCWILINK_FW_DOWNLOAD, &drv->flags)) {
- nfcwilink_fw_download_receive(drv, skb);
- return 0;
- }
-
- /* Forward skb to NCI core layer */
- rc = nci_recv_frame(drv->ndev, skb);
- if (rc < 0) {
- nfc_err(&drv->pdev->dev, "nci_recv_frame failed %d\n", rc);
- return rc;
- }
-
- return 0;
-}
-
-/* protocol structure registered with ST */
-static struct st_proto_s nfcwilink_proto = {
- .chnl_id = NFCWILINK_CHNL,
- .max_frame_size = NFCWILINK_MAX_FRAME_SIZE,
- .hdr_len = (NFCWILINK_HDR_LEN-1), /* not including chnl byte */
- .offset_len_in_hdr = NFCWILINK_OFFSET_LEN_IN_HDR,
- .len_size = NFCWILINK_LEN_SIZE,
- .reserve = 0,
- .recv = nfcwilink_receive,
- .reg_complete_cb = nfcwilink_register_complete,
- .write = NULL,
-};
-
-static int nfcwilink_open(struct nci_dev *ndev)
-{
- struct nfcwilink *drv = nci_get_drvdata(ndev);
- unsigned long comp_ret;
- int rc;
-
- if (test_and_set_bit(NFCWILINK_RUNNING, &drv->flags)) {
- rc = -EBUSY;
- goto exit;
- }
-
- nfcwilink_proto.priv_data = drv;
-
- init_completion(&drv->completed);
- drv->st_register_cb_status = -EINPROGRESS;
-
- rc = st_register(&nfcwilink_proto);
- if (rc < 0) {
- if (rc == -EINPROGRESS) {
- comp_ret = wait_for_completion_timeout(
- &drv->completed,
- msecs_to_jiffies(NFCWILINK_REGISTER_TIMEOUT));
-
- dev_dbg(&drv->pdev->dev,
- "wait_for_completion_timeout returned %ld\n",
- comp_ret);
-
- if (comp_ret == 0) {
- /* timeout */
- rc = -ETIMEDOUT;
- goto clear_exit;
- } else if (drv->st_register_cb_status != 0) {
- rc = drv->st_register_cb_status;
- nfc_err(&drv->pdev->dev,
- "st_register_cb failed %d\n", rc);
- goto clear_exit;
- }
- } else {
- nfc_err(&drv->pdev->dev, "st_register failed %d\n", rc);
- goto clear_exit;
- }
- }
-
- /* st_register MUST fill the write callback */
- BUG_ON(nfcwilink_proto.write == NULL);
- drv->st_write = nfcwilink_proto.write;
-
- if (nfcwilink_download_fw(drv)) {
- nfc_err(&drv->pdev->dev, "nfcwilink_download_fw failed %d\n",
- rc);
- /* open should succeed, even if the FW download failed */
- }
-
- goto exit;
-
-clear_exit:
- clear_bit(NFCWILINK_RUNNING, &drv->flags);
-
-exit:
- return rc;
-}
-
-static int nfcwilink_close(struct nci_dev *ndev)
-{
- struct nfcwilink *drv = nci_get_drvdata(ndev);
- int rc;
-
- if (!test_and_clear_bit(NFCWILINK_RUNNING, &drv->flags))
- return 0;
-
- rc = st_unregister(&nfcwilink_proto);
- if (rc)
- nfc_err(&drv->pdev->dev, "st_unregister failed %d\n", rc);
-
- drv->st_write = NULL;
-
- return rc;
-}
-
-static int nfcwilink_send(struct nci_dev *ndev, struct sk_buff *skb)
-{
- struct nfcwilink *drv = nci_get_drvdata(ndev);
- struct nfcwilink_hdr hdr = {NFCWILINK_CHNL, NFCWILINK_OPCODE, 0x0000};
- long len;
-
- dev_dbg(&drv->pdev->dev, "send entry, len %d\n", skb->len);
-
- if (!test_bit(NFCWILINK_RUNNING, &drv->flags)) {
- kfree_skb(skb);
- return -EINVAL;
- }
-
- /* add the ST hdr to the start of the buffer */
- hdr.len = cpu_to_le16(skb->len);
- memcpy(skb_push(skb, NFCWILINK_HDR_LEN), &hdr, NFCWILINK_HDR_LEN);
-
- /* Insert skb to shared transport layer's transmit queue.
- * Freeing skb memory is taken care in shared transport layer,
- * so don't free skb memory here.
- */
- len = drv->st_write(skb);
- if (len < 0) {
- kfree_skb(skb);
- nfc_err(&drv->pdev->dev, "st_write failed %ld\n", len);
- return -EFAULT;
- }
-
- return 0;
-}
-
-static struct nci_ops nfcwilink_ops = {
- .open = nfcwilink_open,
- .close = nfcwilink_close,
- .send = nfcwilink_send,
-};
-
-static int nfcwilink_probe(struct platform_device *pdev)
-{
- struct nfcwilink *drv;
- int rc;
- __u32 protocols;
-
- drv = devm_kzalloc(&pdev->dev, sizeof(struct nfcwilink), GFP_KERNEL);
- if (!drv) {
- rc = -ENOMEM;
- goto exit;
- }
-
- drv->pdev = pdev;
-
- protocols = NFC_PROTO_JEWEL_MASK
- | NFC_PROTO_MIFARE_MASK | NFC_PROTO_FELICA_MASK
- | NFC_PROTO_ISO14443_MASK
- | NFC_PROTO_ISO14443_B_MASK
- | NFC_PROTO_NFC_DEP_MASK;
-
- drv->ndev = nci_allocate_device(&nfcwilink_ops,
- protocols,
- NFCWILINK_HDR_LEN,
- 0);
- if (!drv->ndev) {
- nfc_err(&pdev->dev, "nci_allocate_device failed\n");
- rc = -ENOMEM;
- goto exit;
- }
-
- nci_set_parent_dev(drv->ndev, &pdev->dev);
- nci_set_drvdata(drv->ndev, drv);
-
- rc = nci_register_device(drv->ndev);
- if (rc < 0) {
- nfc_err(&pdev->dev, "nci_register_device failed %d\n", rc);
- goto free_dev_exit;
- }
-
- dev_set_drvdata(&pdev->dev, drv);
-
- goto exit;
-
-free_dev_exit:
- nci_free_device(drv->ndev);
-
-exit:
- return rc;
-}
-
-static int nfcwilink_remove(struct platform_device *pdev)
-{
- struct nfcwilink *drv = dev_get_drvdata(&pdev->dev);
- struct nci_dev *ndev;
-
- if (!drv)
- return -EFAULT;
-
- ndev = drv->ndev;
-
- nci_unregister_device(ndev);
- nci_free_device(ndev);
-
- return 0;
-}
-
-static struct platform_driver nfcwilink_driver = {
- .probe = nfcwilink_probe,
- .remove = nfcwilink_remove,
- .driver = {
- .name = "nfcwilink",
- },
-};
-
-module_platform_driver(nfcwilink_driver);
-
-/* ------ Module Info ------ */
-
-MODULE_AUTHOR("Ilan Elias <ilane@ti.com>");
-MODULE_DESCRIPTION("NFC Driver for TI Shared Transport");
-MODULE_LICENSE("GPL");
diff --git a/drivers/nfc/nxp-nci/firmware.c b/drivers/nfc/nxp-nci/firmware.c
index 5291797..553011f 100644
--- a/drivers/nfc/nxp-nci/firmware.c
+++ b/drivers/nfc/nxp-nci/firmware.c
@@ -24,7 +24,7 @@
#include <linux/completion.h>
#include <linux/firmware.h>
#include <linux/nfc.h>
-#include <linux/unaligned/access_ok.h>
+#include <asm/unaligned.h>
#include "nxp-nci.h"
diff --git a/drivers/nfc/nxp-nci/i2c.c b/drivers/nfc/nxp-nci/i2c.c
index 36099e5..ff22d76 100644
--- a/drivers/nfc/nxp-nci/i2c.c
+++ b/drivers/nfc/nxp-nci/i2c.c
@@ -29,14 +29,13 @@
#include <linux/delay.h>
#include <linux/i2c.h>
#include <linux/interrupt.h>
-#include <linux/miscdevice.h>
#include <linux/module.h>
#include <linux/nfc.h>
#include <linux/gpio/consumer.h>
#include <linux/of_gpio.h>
#include <linux/of_irq.h>
#include <linux/platform_data/nxp-nci.h>
-#include <linux/unaligned/access_ok.h>
+#include <asm/unaligned.h>
#include <net/nfc/nfc.h>
@@ -86,7 +85,7 @@
r = i2c_master_send(client, skb->data, skb->len);
if (r < 0) {
/* Retry, chip was in standby */
- usleep_range(110000, 120000);
+ msleep(110);
r = i2c_master_send(client, skb->data, skb->len);
}
@@ -127,7 +126,7 @@
goto fw_read_exit;
}
- frame_len = (get_unaligned_be16(&header) & NXP_NCI_FW_FRAME_LEN_MASK) +
+ frame_len = (be16_to_cpu(header) & NXP_NCI_FW_FRAME_LEN_MASK) +
NXP_NCI_FW_CRC_LEN;
*skb = alloc_skb(NXP_NCI_FW_HDR_LEN + frame_len, GFP_KERNEL);
diff --git a/drivers/nfc/pn533/i2c.c b/drivers/nfc/pn533/i2c.c
index 1dc8924..8f60ce0 100644
--- a/drivers/nfc/pn533/i2c.c
+++ b/drivers/nfc/pn533/i2c.c
@@ -51,7 +51,7 @@
{
struct pn533_i2c_phy *phy = dev->phy;
struct i2c_client *client = phy->i2c_dev;
- u8 ack[6] = {0x00, 0x00, 0xff, 0x00, 0xff, 0x00};
+ static const u8 ack[6] = {0x00, 0x00, 0xff, 0x00, 0xff, 0x00};
/* spec 6.2.1.3: Preamble, SoPC (2), ACK Code (2), Postamble */
int rc;
@@ -206,14 +206,6 @@
phy->i2c_dev = client;
i2c_set_clientdata(client, phy);
- r = request_threaded_irq(client->irq, NULL, pn533_i2c_irq_thread_fn,
- IRQF_TRIGGER_FALLING |
- IRQF_SHARED | IRQF_ONESHOT,
- PN533_I2C_DRIVER_NAME, phy);
-
- if (r < 0)
- nfc_err(&client->dev, "Unable to register IRQ handler\n");
-
priv = pn533_register_device(PN533_DEVICE_PN532,
PN533_NO_TYPE_B_PROTOCOLS,
PN533_PROTO_REQ_ACK_RESP,
@@ -223,16 +215,32 @@
if (IS_ERR(priv)) {
r = PTR_ERR(priv);
- goto err_register;
+ return r;
}
phy->priv = priv;
+ r = request_threaded_irq(client->irq, NULL, pn533_i2c_irq_thread_fn,
+ IRQF_TRIGGER_FALLING |
+ IRQF_SHARED | IRQF_ONESHOT,
+ PN533_I2C_DRIVER_NAME, phy);
+ if (r < 0) {
+ nfc_err(&client->dev, "Unable to register IRQ handler\n");
+ goto irq_rqst_err;
+ }
+
+ r = pn533_finalize_setup(priv);
+ if (r)
+ goto fn_setup_err;
+
return 0;
-err_register:
+fn_setup_err:
free_irq(client->irq, phy);
+irq_rqst_err:
+ pn533_unregister_device(phy->priv);
+
return r;
}
@@ -242,10 +250,10 @@
dev_dbg(&client->dev, "%s\n", __func__);
- pn533_unregister_device(phy->priv);
-
free_irq(client->irq, phy);
+ pn533_unregister_device(phy->priv);
+
return 0;
}
diff --git a/drivers/nfc/pn533/pn533.c b/drivers/nfc/pn533/pn533.c
index a966c6a..65bbaa5 100644
--- a/drivers/nfc/pn533/pn533.c
+++ b/drivers/nfc/pn533/pn533.c
@@ -383,14 +383,18 @@
static int pn533_send_async_complete(struct pn533 *dev)
{
struct pn533_cmd *cmd = dev->cmd;
- int status = cmd->status;
+ struct sk_buff *resp;
+ int status, rc = 0;
- struct sk_buff *req = cmd->req;
- struct sk_buff *resp = cmd->resp;
+ if (!cmd) {
+ dev_dbg(dev->dev, "%s: cmd not set\n", __func__);
+ goto done;
+ }
- int rc;
+ dev_kfree_skb(cmd->req);
- dev_kfree_skb(req);
+ status = cmd->status;
+ resp = cmd->resp;
if (status < 0) {
rc = cmd->complete_cb(dev, cmd->complete_cb_context,
@@ -399,8 +403,14 @@
goto done;
}
- skb_pull(resp, dev->ops->rx_header_len);
- skb_trim(resp, resp->len - dev->ops->rx_tail_len);
+ /* when no response is set we got interrupted */
+ if (!resp)
+ resp = ERR_PTR(-EINTR);
+
+ if (!IS_ERR(resp)) {
+ skb_pull(resp, dev->ops->rx_header_len);
+ skb_trim(resp, resp->len - dev->ops->rx_tail_len);
+ }
rc = cmd->complete_cb(dev, cmd->complete_cb_context, resp);
@@ -434,12 +444,14 @@
mutex_lock(&dev->cmd_lock);
if (!dev->cmd_pending) {
+ dev->cmd = cmd;
rc = dev->phy_ops->send_frame(dev, req);
- if (rc)
+ if (rc) {
+ dev->cmd = NULL;
goto error;
+ }
dev->cmd_pending = 1;
- dev->cmd = cmd;
goto unlock;
}
@@ -511,11 +523,12 @@
pn533_build_cmd_frame(dev, cmd_code, req);
+ dev->cmd = cmd;
rc = dev->phy_ops->send_frame(dev, req);
- if (rc < 0)
+ if (rc < 0) {
+ dev->cmd = NULL;
kfree(cmd);
- else
- dev->cmd = cmd;
+ }
return rc;
}
@@ -550,14 +563,15 @@
mutex_unlock(&dev->cmd_lock);
+ dev->cmd = cmd;
rc = dev->phy_ops->send_frame(dev, cmd->req);
if (rc < 0) {
+ dev->cmd = NULL;
dev_kfree_skb(cmd->req);
kfree(cmd);
return;
}
- dev->cmd = cmd;
}
struct pn533_sync_cmd_response {
@@ -2556,6 +2570,31 @@
return 0;
}
+int pn533_finalize_setup(struct pn533 *dev)
+{
+
+ struct pn533_fw_version fw_ver;
+ int rc;
+
+ memset(&fw_ver, 0, sizeof(fw_ver));
+
+ rc = pn533_get_firmware_version(dev, &fw_ver);
+ if (rc) {
+ nfc_err(dev->dev, "Unable to get FW version\n");
+ return rc;
+ }
+
+ nfc_info(dev->dev, "NXP PN5%02X firmware ver %d.%d now attached\n",
+ fw_ver.ic, fw_ver.ver, fw_ver.rev);
+
+ rc = pn533_setup(dev);
+ if (rc)
+ return rc;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(pn533_finalize_setup);
+
struct pn533 *pn533_register_device(u32 device_type,
u32 protocols,
enum pn533_protocol_type protocol_type,
@@ -2565,7 +2604,6 @@
struct device *dev,
struct device *parent)
{
- struct pn533_fw_version fw_ver;
struct pn533 *priv;
int rc = -ENOMEM;
@@ -2608,15 +2646,6 @@
INIT_LIST_HEAD(&priv->cmd_queue);
- memset(&fw_ver, 0, sizeof(fw_ver));
- rc = pn533_get_firmware_version(priv, &fw_ver);
- if (rc < 0)
- goto destroy_wq;
-
- nfc_info(dev, "NXP PN5%02X firmware ver %d.%d now attached\n",
- fw_ver.ic, fw_ver.ver, fw_ver.rev);
-
-
priv->nfc_dev = nfc_allocate_device(&pn533_nfc_ops, protocols,
priv->ops->tx_header_len +
PN533_CMD_DATAEXCH_HEAD_LEN,
@@ -2633,15 +2662,8 @@
if (rc)
goto free_nfc_dev;
- rc = pn533_setup(priv);
- if (rc)
- goto unregister_nfc_dev;
-
return priv;
-unregister_nfc_dev:
- nfc_unregister_device(priv->nfc_dev);
-
free_nfc_dev:
nfc_free_device(priv->nfc_dev);
diff --git a/drivers/nfc/pn533/pn533.h b/drivers/nfc/pn533/pn533.h
index 553c7d1..88d5696 100644
--- a/drivers/nfc/pn533/pn533.h
+++ b/drivers/nfc/pn533/pn533.h
@@ -231,6 +231,7 @@
struct device *dev,
struct device *parent);
+int pn533_finalize_setup(struct pn533 *dev);
void pn533_unregister_device(struct pn533 *priv);
void pn533_recv_frame(struct pn533 *dev, struct sk_buff *skb, int status);
diff --git a/drivers/nfc/pn533/usb.c b/drivers/nfc/pn533/usb.c
index 33ed78b..8ed203e 100644
--- a/drivers/nfc/pn533/usb.c
+++ b/drivers/nfc/pn533/usb.c
@@ -148,11 +148,11 @@
static int pn533_usb_send_ack(struct pn533 *dev, gfp_t flags)
{
struct pn533_usb_phy *phy = dev->phy;
- u8 ack[6] = {0x00, 0x00, 0xff, 0x00, 0xff, 0x00};
+ static const u8 ack[6] = {0x00, 0x00, 0xff, 0x00, 0xff, 0x00};
/* spec 7.1.1.3: Preamble, SoPC (2), ACK Code (2), Postamble */
int rc;
- phy->out_urb->transfer_buffer = ack;
+ phy->out_urb->transfer_buffer = (u8 *)ack;
phy->out_urb->transfer_buffer_length = sizeof(ack);
rc = usb_submit_urb(phy->out_urb, flags);
@@ -543,6 +543,10 @@
phy->priv = priv;
+ rc = pn533_finalize_setup(priv);
+ if (rc)
+ goto error;
+
usb_set_intfdata(interface, phy);
return 0;
diff --git a/drivers/nfc/pn544/i2c.c b/drivers/nfc/pn544/i2c.c
index f837c39..71ac083 100644
--- a/drivers/nfc/pn544/i2c.c
+++ b/drivers/nfc/pn544/i2c.c
@@ -21,17 +21,13 @@
#include <linux/crc-ccitt.h>
#include <linux/module.h>
#include <linux/i2c.h>
-#include <linux/gpio.h>
-#include <linux/of_gpio.h>
-#include <linux/of_irq.h>
#include <linux/acpi.h>
-#include <linux/miscdevice.h>
#include <linux/interrupt.h>
#include <linux/delay.h>
#include <linux/nfc.h>
#include <linux/firmware.h>
#include <linux/gpio/consumer.h>
-#include <linux/platform_data/pn544.h>
+
#include <asm/unaligned.h>
#include <net/nfc/hci.h>
@@ -165,8 +161,9 @@
struct i2c_client *i2c_dev;
struct nfc_hci_dev *hdev;
- unsigned int gpio_en;
- unsigned int gpio_fw;
+ struct gpio_desc *gpiod_en;
+ struct gpio_desc *gpiod_fw;
+
unsigned int en_polarity;
u8 hw_variant;
@@ -208,19 +205,18 @@
nfc_info(&phy->i2c_dev->dev, "Detecting nfc_en polarity\n");
/* Disable fw download */
- gpio_set_value_cansleep(phy->gpio_fw, 0);
+ gpiod_set_value_cansleep(phy->gpiod_fw, 0);
for (polarity = 0; polarity < 2; polarity++) {
phy->en_polarity = polarity;
retry = 3;
while (retry--) {
/* power off */
- gpio_set_value_cansleep(phy->gpio_en,
- !phy->en_polarity);
+ gpiod_set_value_cansleep(phy->gpiod_en, !phy->en_polarity);
usleep_range(10000, 15000);
/* power on */
- gpio_set_value_cansleep(phy->gpio_en, phy->en_polarity);
+ gpiod_set_value_cansleep(phy->gpiod_en, phy->en_polarity);
usleep_range(10000, 15000);
/* send reset */
@@ -239,14 +235,13 @@
"Could not detect nfc_en polarity, fallback to active high\n");
out:
- gpio_set_value_cansleep(phy->gpio_en, !phy->en_polarity);
+ gpiod_set_value_cansleep(phy->gpiod_en, !phy->en_polarity);
}
static void pn544_hci_i2c_enable_mode(struct pn544_i2c_phy *phy, int run_mode)
{
- gpio_set_value_cansleep(phy->gpio_fw,
- run_mode == PN544_FW_MODE ? 1 : 0);
- gpio_set_value_cansleep(phy->gpio_en, phy->en_polarity);
+ gpiod_set_value_cansleep(phy->gpiod_fw, run_mode == PN544_FW_MODE ? 1 : 0);
+ gpiod_set_value_cansleep(phy->gpiod_en, phy->en_polarity);
usleep_range(10000, 15000);
phy->run_mode = run_mode;
@@ -269,14 +264,14 @@
{
struct pn544_i2c_phy *phy = phy_id;
- gpio_set_value_cansleep(phy->gpio_fw, 0);
- gpio_set_value_cansleep(phy->gpio_en, !phy->en_polarity);
+ gpiod_set_value_cansleep(phy->gpiod_fw, 0);
+ gpiod_set_value_cansleep(phy->gpiod_en, !phy->en_polarity);
usleep_range(10000, 15000);
- gpio_set_value_cansleep(phy->gpio_en, phy->en_polarity);
+ gpiod_set_value_cansleep(phy->gpiod_en, phy->en_polarity);
usleep_range(10000, 15000);
- gpio_set_value_cansleep(phy->gpio_en, !phy->en_polarity);
+ gpiod_set_value_cansleep(phy->gpiod_en, !phy->en_polarity);
usleep_range(10000, 15000);
phy->powered = 0;
@@ -874,106 +869,20 @@
}
}
-static int pn544_hci_i2c_acpi_request_resources(struct i2c_client *client)
-{
- struct pn544_i2c_phy *phy = i2c_get_clientdata(client);
- struct gpio_desc *gpiod_en, *gpiod_fw;
- struct device *dev = &client->dev;
+static const struct acpi_gpio_params enable_gpios = { 1, 0, false };
+static const struct acpi_gpio_params firmware_gpios = { 2, 0, false };
- /* Get EN GPIO from ACPI */
- gpiod_en = devm_gpiod_get_index(dev, PN544_GPIO_NAME_EN, 1,
- GPIOD_OUT_LOW);
- if (IS_ERR(gpiod_en)) {
- nfc_err(dev, "Unable to get EN GPIO\n");
- return -ENODEV;
- }
-
- phy->gpio_en = desc_to_gpio(gpiod_en);
-
- /* Get FW GPIO from ACPI */
- gpiod_fw = devm_gpiod_get_index(dev, PN544_GPIO_NAME_FW, 2,
- GPIOD_OUT_LOW);
- if (IS_ERR(gpiod_fw)) {
- nfc_err(dev, "Unable to get FW GPIO\n");
- return -ENODEV;
- }
-
- phy->gpio_fw = desc_to_gpio(gpiod_fw);
-
- return 0;
-}
-
-static int pn544_hci_i2c_of_request_resources(struct i2c_client *client)
-{
- struct pn544_i2c_phy *phy = i2c_get_clientdata(client);
- struct device_node *pp;
- int ret;
-
- pp = client->dev.of_node;
- if (!pp) {
- ret = -ENODEV;
- goto err_dt;
- }
-
- /* Obtention of EN GPIO from device tree */
- ret = of_get_named_gpio(pp, "enable-gpios", 0);
- if (ret < 0) {
- if (ret != -EPROBE_DEFER)
- nfc_err(&client->dev,
- "Failed to get EN gpio, error: %d\n", ret);
- goto err_dt;
- }
- phy->gpio_en = ret;
-
- /* Configuration of EN GPIO */
- ret = gpio_request(phy->gpio_en, PN544_GPIO_NAME_EN);
- if (ret) {
- nfc_err(&client->dev, "Fail EN pin\n");
- goto err_dt;
- }
- ret = gpio_direction_output(phy->gpio_en, 0);
- if (ret) {
- nfc_err(&client->dev, "Fail EN pin direction\n");
- goto err_gpio_en;
- }
-
- /* Obtention of FW GPIO from device tree */
- ret = of_get_named_gpio(pp, "firmware-gpios", 0);
- if (ret < 0) {
- if (ret != -EPROBE_DEFER)
- nfc_err(&client->dev,
- "Failed to get FW gpio, error: %d\n", ret);
- goto err_gpio_en;
- }
- phy->gpio_fw = ret;
-
- /* Configuration of FW GPIO */
- ret = gpio_request(phy->gpio_fw, PN544_GPIO_NAME_FW);
- if (ret) {
- nfc_err(&client->dev, "Fail FW pin\n");
- goto err_gpio_en;
- }
- ret = gpio_direction_output(phy->gpio_fw, 0);
- if (ret) {
- nfc_err(&client->dev, "Fail FW pin direction\n");
- goto err_gpio_fw;
- }
-
- return 0;
-
-err_gpio_fw:
- gpio_free(phy->gpio_fw);
-err_gpio_en:
- gpio_free(phy->gpio_en);
-err_dt:
- return ret;
-}
+static const struct acpi_gpio_mapping acpi_pn544_gpios[] = {
+ { "enable-gpios", &enable_gpios, 1 },
+ { "firmware-gpios", &firmware_gpios, 1 },
+ { },
+};
static int pn544_hci_i2c_probe(struct i2c_client *client,
const struct i2c_device_id *id)
{
+ struct device *dev = &client->dev;
struct pn544_i2c_phy *phy;
- struct pn544_nfc_platform_data *pdata;
int r = 0;
dev_dbg(&client->dev, "%s\n", __func__);
@@ -995,53 +904,33 @@
phy->i2c_dev = client;
i2c_set_clientdata(client, phy);
- pdata = client->dev.platform_data;
+ r = acpi_dev_add_driver_gpios(ACPI_COMPANION(dev), acpi_pn544_gpios);
+ if (r)
+ dev_dbg(dev, "Unable to add GPIO mapping table\n");
- /* No platform data, using device tree. */
- if (!pdata && client->dev.of_node) {
- r = pn544_hci_i2c_of_request_resources(client);
- if (r) {
- nfc_err(&client->dev, "No DT data\n");
- return r;
- }
- /* Using platform data. */
- } else if (pdata) {
+ /* Get EN GPIO */
+ phy->gpiod_en = devm_gpiod_get(dev, "enable", GPIOD_OUT_LOW);
+ if (IS_ERR(phy->gpiod_en)) {
+ nfc_err(dev, "Unable to get EN GPIO\n");
+ return PTR_ERR(phy->gpiod_en);
+ }
- if (pdata->request_resources == NULL) {
- nfc_err(&client->dev, "request_resources() missing\n");
- return -EINVAL;
- }
-
- r = pdata->request_resources(client);
- if (r) {
- nfc_err(&client->dev,
- "Cannot get platform resources\n");
- return r;
- }
-
- phy->gpio_en = pdata->get_gpio(NFC_GPIO_ENABLE);
- phy->gpio_fw = pdata->get_gpio(NFC_GPIO_FW_RESET);
- /* Using ACPI */
- } else if (ACPI_HANDLE(&client->dev)) {
- r = pn544_hci_i2c_acpi_request_resources(client);
- if (r) {
- nfc_err(&client->dev,
- "Cannot get ACPI data\n");
- return r;
- }
- } else {
- nfc_err(&client->dev, "No platform data\n");
- return -EINVAL;
+ /* Get FW GPIO */
+ phy->gpiod_fw = devm_gpiod_get(dev, "firmware", GPIOD_OUT_LOW);
+ if (IS_ERR(phy->gpiod_fw)) {
+ nfc_err(dev, "Unable to get FW GPIO\n");
+ return PTR_ERR(phy->gpiod_fw);
}
pn544_hci_i2c_platform_init(phy);
- r = request_threaded_irq(client->irq, NULL, pn544_hci_i2c_irq_thread_fn,
- IRQF_TRIGGER_RISING | IRQF_ONESHOT,
- PN544_HCI_I2C_DRIVER_NAME, phy);
+ r = devm_request_threaded_irq(&client->dev, client->irq, NULL,
+ pn544_hci_i2c_irq_thread_fn,
+ IRQF_TRIGGER_RISING | IRQF_ONESHOT,
+ PN544_HCI_I2C_DRIVER_NAME, phy);
if (r < 0) {
nfc_err(&client->dev, "Unable to register IRQ handler\n");
- goto err_rti;
+ return r;
}
r = pn544_hci_probe(phy, &i2c_phy_ops, LLC_SHDLC_NAME,
@@ -1049,28 +938,14 @@
PN544_HCI_I2C_LLC_MAX_PAYLOAD,
pn544_hci_i2c_fw_download, &phy->hdev);
if (r < 0)
- goto err_hci;
+ return r;
return 0;
-
-err_hci:
- free_irq(client->irq, phy);
-
-err_rti:
- if (!pdata) {
- gpio_free(phy->gpio_en);
- gpio_free(phy->gpio_fw);
- } else if (pdata->free_resources) {
- pdata->free_resources();
- }
-
- return r;
}
static int pn544_hci_i2c_remove(struct i2c_client *client)
{
struct pn544_i2c_phy *phy = i2c_get_clientdata(client);
- struct pn544_nfc_platform_data *pdata = client->dev.platform_data;
dev_dbg(&client->dev, "%s\n", __func__);
@@ -1083,17 +958,7 @@
if (phy->powered)
pn544_hci_i2c_disable(phy);
- free_irq(client->irq, phy);
-
- /* No platform data, GPIOs have been requested by this driver */
- if (!pdata) {
- gpio_free(phy->gpio_en);
- gpio_free(phy->gpio_fw);
- /* Using platform data */
- } else if (pdata->free_resources) {
- pdata->free_resources();
- }
-
+ acpi_dev_remove_driver_gpios(ACPI_COMPANION(&client->dev));
return 0;
}
diff --git a/drivers/nfc/port100.c b/drivers/nfc/port100.c
index 2b2330b..19be93e 100644
--- a/drivers/nfc/port100.c
+++ b/drivers/nfc/port100.c
@@ -21,8 +21,9 @@
#define VERSION "0.1"
-#define SONY_VENDOR_ID 0x054c
-#define RCS380_PRODUCT_ID 0x06c1
+#define SONY_VENDOR_ID 0x054c
+#define RCS380S_PRODUCT_ID 0x06c1
+#define RCS380P_PRODUCT_ID 0x06c3
#define PORT100_PROTOCOLS (NFC_PROTO_JEWEL_MASK | \
NFC_PROTO_MIFARE_MASK | \
@@ -725,23 +726,33 @@
static int port100_send_ack(struct port100 *dev)
{
- int rc;
+ int rc = 0;
mutex_lock(&dev->out_urb_lock);
- init_completion(&dev->cmd_cancel_done);
-
- usb_kill_urb(dev->out_urb);
-
- dev->out_urb->transfer_buffer = ack_frame;
- dev->out_urb->transfer_buffer_length = sizeof(ack_frame);
- rc = usb_submit_urb(dev->out_urb, GFP_KERNEL);
-
- /* Set the cmd_cancel flag only if the URB has been successfully
- * submitted. It will be reset by the out URB completion callback
- * port100_send_complete().
+ /*
+ * If prior cancel is in-flight (dev->cmd_cancel == true), we
+ * can skip to send cancel. Then this will wait the prior
+ * cancel, or merged into the next cancel rarely if next
+ * cancel was started before waiting done. In any case, this
+ * will be waked up soon or later.
*/
- dev->cmd_cancel = !rc;
+ if (!dev->cmd_cancel) {
+ reinit_completion(&dev->cmd_cancel_done);
+
+ usb_kill_urb(dev->out_urb);
+
+ dev->out_urb->transfer_buffer = ack_frame;
+ dev->out_urb->transfer_buffer_length = sizeof(ack_frame);
+ rc = usb_submit_urb(dev->out_urb, GFP_KERNEL);
+
+ /*
+ * Set the cmd_cancel flag only if the URB has been
+ * successfully submitted. It will be reset by the out
+ * URB completion callback port100_send_complete().
+ */
+ dev->cmd_cancel = !rc;
+ }
mutex_unlock(&dev->out_urb_lock);
@@ -928,8 +939,8 @@
struct port100 *dev = urb->context;
if (dev->cmd_cancel) {
+ complete_all(&dev->cmd_cancel_done);
dev->cmd_cancel = false;
- complete(&dev->cmd_cancel_done);
}
switch (urb->status) {
@@ -1477,7 +1488,8 @@
};
static const struct usb_device_id port100_table[] = {
- { USB_DEVICE(SONY_VENDOR_ID, RCS380_PRODUCT_ID), },
+ { USB_DEVICE(SONY_VENDOR_ID, RCS380S_PRODUCT_ID), },
+ { USB_DEVICE(SONY_VENDOR_ID, RCS380P_PRODUCT_ID), },
{ }
};
MODULE_DEVICE_TABLE(usb, port100_table);
@@ -1538,11 +1550,13 @@
usb_fill_bulk_urb(dev->out_urb, dev->udev,
usb_sndbulkpipe(dev->udev, out_endpoint),
NULL, 0, port100_send_complete, dev);
+ dev->out_urb->transfer_flags = URB_ZERO_PACKET;
dev->skb_headroom = PORT100_FRAME_HEADER_LEN +
PORT100_COMM_RF_HEAD_MAX_LEN;
dev->skb_tailroom = PORT100_FRAME_TAIL_LEN;
+ init_completion(&dev->cmd_cancel_done);
INIT_WORK(&dev->cmd_complete_work, port100_wq_cmd_complete);
/* The first thing to do with the Port-100 is to set the command type
diff --git a/drivers/nfc/st21nfca/core.c b/drivers/nfc/st21nfca/core.c
index dacb916..50be3b7 100644
--- a/drivers/nfc/st21nfca/core.c
+++ b/drivers/nfc/st21nfca/core.c
@@ -959,10 +959,8 @@
unsigned long quirks = 0;
info = kzalloc(sizeof(struct st21nfca_hci_info), GFP_KERNEL);
- if (!info) {
- r = -ENOMEM;
- goto err_alloc_hdev;
- }
+ if (!info)
+ return -ENOMEM;
info->phy_ops = phy_ops;
info->phy_id = phy_id;
@@ -978,8 +976,10 @@
* persistent info to discriminate 2 identical chips
*/
dev_num = find_first_zero_bit(dev_mask, ST21NFCA_NUM_DEVICES);
- if (dev_num >= ST21NFCA_NUM_DEVICES)
- return -ENODEV;
+ if (dev_num >= ST21NFCA_NUM_DEVICES) {
+ r = -ENODEV;
+ goto err_alloc_hdev;
+ }
set_bit(dev_num, dev_mask);
diff --git a/drivers/nfc/st21nfca/i2c.c b/drivers/nfc/st21nfca/i2c.c
index 5a82f55..02a920c 100644
--- a/drivers/nfc/st21nfca/i2c.c
+++ b/drivers/nfc/st21nfca/i2c.c
@@ -20,17 +20,15 @@
#include <linux/crc-ccitt.h>
#include <linux/module.h>
#include <linux/i2c.h>
-#include <linux/gpio.h>
#include <linux/gpio/consumer.h>
#include <linux/of_irq.h>
#include <linux/of_gpio.h>
#include <linux/acpi.h>
-#include <linux/miscdevice.h>
#include <linux/interrupt.h>
#include <linux/delay.h>
#include <linux/nfc.h>
#include <linux/firmware.h>
-#include <linux/platform_data/st21nfca.h>
+
#include <asm/unaligned.h>
#include <net/nfc/hci.h>
@@ -60,6 +58,7 @@
#define IS_START_OF_FRAME(buf) (buf[0] == ST21NFCA_SOF_EOF && \
buf[1] == 0)
+#define ST21NFCA_HCI_DRIVER_NAME "st21nfca_hci"
#define ST21NFCA_HCI_I2C_DRIVER_NAME "st21nfca_hci_i2c"
#define ST21NFCA_GPIO_NAME_EN "enable"
@@ -68,9 +67,7 @@
struct i2c_client *i2c_dev;
struct nfc_hci_dev *hdev;
- unsigned int gpio_ena;
- unsigned int irq_polarity;
-
+ struct gpio_desc *gpiod_ena;
struct st21nfca_se_status se_status;
struct sk_buff *pending_skb;
@@ -151,7 +148,7 @@
{
struct st21nfca_i2c_phy *phy = phy_id;
- gpio_set_value(phy->gpio_ena, 1);
+ gpiod_set_value(phy->gpiod_ena, 1);
phy->powered = 1;
phy->run_mode = ST21NFCA_HCI_MODE;
@@ -164,7 +161,7 @@
{
struct st21nfca_i2c_phy *phy = phy_id;
- gpio_set_value(phy->gpio_ena, 0);
+ gpiod_set_value(phy->gpiod_ena, 0);
phy->powered = 0;
}
@@ -507,33 +504,14 @@
static int st21nfca_hci_i2c_acpi_request_resources(struct i2c_client *client)
{
struct st21nfca_i2c_phy *phy = i2c_get_clientdata(client);
- struct gpio_desc *gpiod_ena;
struct device *dev = &client->dev;
- u8 tmp;
/* Get EN GPIO from ACPI */
- gpiod_ena = devm_gpiod_get_index(dev, ST21NFCA_GPIO_NAME_EN, 1,
- GPIOD_OUT_LOW);
- if (!IS_ERR(gpiod_ena)) {
+ phy->gpiod_ena = devm_gpiod_get_index(dev, ST21NFCA_GPIO_NAME_EN, 1,
+ GPIOD_OUT_LOW);
+ if (IS_ERR(phy->gpiod_ena)) {
nfc_err(dev, "Unable to get ENABLE GPIO\n");
- return -ENODEV;
- }
-
- phy->gpio_ena = desc_to_gpio(gpiod_ena);
-
- phy->irq_polarity = irq_get_trigger_type(client->irq);
-
- phy->se_status.is_ese_present = false;
- phy->se_status.is_uicc_present = false;
-
- if (device_property_present(dev, "ese-present")) {
- device_property_read_u8(dev, "ese-present", &tmp);
- phy->se_status.is_ese_present = tmp;
- }
-
- if (device_property_present(dev, "uicc-present")) {
- device_property_read_u8(dev, "uicc-present", &tmp);
- phy->se_status.is_uicc_present = tmp;
+ return PTR_ERR(phy->gpiod_ena);
}
return 0;
@@ -542,70 +520,16 @@
static int st21nfca_hci_i2c_of_request_resources(struct i2c_client *client)
{
struct st21nfca_i2c_phy *phy = i2c_get_clientdata(client);
- struct device_node *pp;
- int gpio;
- int r;
-
- pp = client->dev.of_node;
- if (!pp)
- return -ENODEV;
+ struct device *dev = &client->dev;
/* Get GPIO from device tree */
- gpio = of_get_named_gpio(pp, "enable-gpios", 0);
- if (gpio < 0) {
- nfc_err(&client->dev, "Failed to retrieve enable-gpios from device tree\n");
- return gpio;
+ phy->gpiod_ena = devm_gpiod_get_index(dev, ST21NFCA_GPIO_NAME_EN, 0,
+ GPIOD_OUT_HIGH);
+ if (IS_ERR(phy->gpiod_ena)) {
+ nfc_err(dev, "Failed to request enable pin\n");
+ return PTR_ERR(phy->gpiod_ena);
}
- /* GPIO request and configuration */
- r = devm_gpio_request_one(&client->dev, gpio, GPIOF_OUT_INIT_HIGH,
- ST21NFCA_GPIO_NAME_EN);
- if (r) {
- nfc_err(&client->dev, "Failed to request enable pin\n");
- return r;
- }
-
- phy->gpio_ena = gpio;
-
- phy->irq_polarity = irq_get_trigger_type(client->irq);
-
- phy->se_status.is_ese_present =
- of_property_read_bool(pp, "ese-present");
- phy->se_status.is_uicc_present =
- of_property_read_bool(pp, "uicc-present");
-
- return 0;
-}
-
-static int st21nfca_hci_i2c_request_resources(struct i2c_client *client)
-{
- struct st21nfca_nfc_platform_data *pdata;
- struct st21nfca_i2c_phy *phy = i2c_get_clientdata(client);
- int r;
-
- pdata = client->dev.platform_data;
- if (pdata == NULL) {
- nfc_err(&client->dev, "No platform data\n");
- return -EINVAL;
- }
-
- /* store for later use */
- phy->gpio_ena = pdata->gpio_ena;
- phy->irq_polarity = pdata->irq_polarity;
-
- if (phy->gpio_ena > 0) {
- r = devm_gpio_request_one(&client->dev, phy->gpio_ena,
- GPIOF_OUT_INIT_HIGH,
- ST21NFCA_GPIO_NAME_EN);
- if (r) {
- pr_err("%s : ena gpio_request failed\n", __FILE__);
- return r;
- }
- }
-
- phy->se_status.is_ese_present = pdata->is_ese_present;
- phy->se_status.is_uicc_present = pdata->is_uicc_present;
-
return 0;
}
@@ -613,7 +537,6 @@
const struct i2c_device_id *id)
{
struct st21nfca_i2c_phy *phy;
- struct st21nfca_nfc_platform_data *pdata;
int r;
dev_dbg(&client->dev, "%s\n", __func__);
@@ -639,19 +562,12 @@
mutex_init(&phy->phy_lock);
i2c_set_clientdata(client, phy);
- pdata = client->dev.platform_data;
- if (!pdata && client->dev.of_node) {
+ if (client->dev.of_node) {
r = st21nfca_hci_i2c_of_request_resources(client);
if (r) {
nfc_err(&client->dev, "No platform data\n");
return r;
}
- } else if (pdata) {
- r = st21nfca_hci_i2c_request_resources(client);
- if (r) {
- nfc_err(&client->dev, "Cannot get platform resources\n");
- return r;
- }
} else if (ACPI_HANDLE(&client->dev)) {
r = st21nfca_hci_i2c_acpi_request_resources(client);
if (r) {
@@ -663,6 +579,11 @@
return -ENODEV;
}
+ phy->se_status.is_ese_present =
+ device_property_read_bool(&client->dev, "ese-present");
+ phy->se_status.is_uicc_present =
+ device_property_read_bool(&client->dev, "uicc-present");
+
r = st21nfca_hci_platform_init(phy);
if (r < 0) {
nfc_err(&client->dev, "Unable to reboot st21nfca\n");
@@ -671,7 +592,7 @@
r = devm_request_threaded_irq(&client->dev, client->irq, NULL,
st21nfca_hci_irq_thread_fn,
- phy->irq_polarity | IRQF_ONESHOT,
+ IRQF_ONESHOT,
ST21NFCA_HCI_DRIVER_NAME, phy);
if (r < 0) {
nfc_err(&client->dev, "Unable to register IRQ handler\n");
diff --git a/drivers/nfc/trf7970a.c b/drivers/nfc/trf7970a.c
index 26c9dbb..2d1c8ca 100644
--- a/drivers/nfc/trf7970a.c
+++ b/drivers/nfc/trf7970a.c
@@ -124,6 +124,9 @@
NFC_PROTO_ISO15693_MASK | NFC_PROTO_NFC_DEP_MASK)
#define TRF7970A_AUTOSUSPEND_DELAY 30000 /* 30 seconds */
+#define TRF7970A_13MHZ_CLOCK_FREQUENCY 13560000
+#define TRF7970A_27MHZ_CLOCK_FREQUENCY 27120000
+
#define TRF7970A_RX_SKB_ALLOC_SIZE 256
@@ -441,6 +444,7 @@
u8 iso_ctrl_tech;
u8 modulator_sys_clk_ctrl;
u8 special_fcn_reg1;
+ u8 io_ctrl;
unsigned int guard_time;
int technology;
int framing;
@@ -1048,6 +1052,11 @@
if (ret)
goto err_out;
+ ret = trf7970a_write(trf, TRF7970A_REG_IO_CTRL,
+ trf->io_ctrl | TRF7970A_REG_IO_CTRL_VRS(0x1));
+ if (ret)
+ goto err_out;
+
ret = trf7970a_write(trf, TRF7970A_NFC_TARGET_LEVEL, 0);
if (ret)
goto err_out;
@@ -1056,12 +1065,11 @@
trf->chip_status_ctrl &= ~TRF7970A_CHIP_STATUS_RF_ON;
- ret = trf7970a_write(trf, TRF7970A_MODULATOR_SYS_CLK_CTRL, 0);
+ ret = trf7970a_write(trf, TRF7970A_MODULATOR_SYS_CLK_CTRL,
+ trf->modulator_sys_clk_ctrl);
if (ret)
goto err_out;
- trf->modulator_sys_clk_ctrl = 0;
-
ret = trf7970a_write(trf, TRF7970A_ADJUTABLE_FIFO_IRQ_LEVELS,
TRF7970A_ADJUTABLE_FIFO_IRQ_LEVELS_WLH_96 |
TRF7970A_ADJUTABLE_FIFO_IRQ_LEVELS_WLL_32);
@@ -1181,27 +1189,37 @@
switch (tech) {
case NFC_DIGITAL_RF_TECH_106A:
trf->iso_ctrl_tech = TRF7970A_ISO_CTRL_14443A_106;
- trf->modulator_sys_clk_ctrl = TRF7970A_MODULATOR_DEPTH_OOK;
+ trf->modulator_sys_clk_ctrl =
+ (trf->modulator_sys_clk_ctrl & 0xf8) |
+ TRF7970A_MODULATOR_DEPTH_OOK;
trf->guard_time = TRF7970A_GUARD_TIME_NFCA;
break;
case NFC_DIGITAL_RF_TECH_106B:
trf->iso_ctrl_tech = TRF7970A_ISO_CTRL_14443B_106;
- trf->modulator_sys_clk_ctrl = TRF7970A_MODULATOR_DEPTH_ASK10;
+ trf->modulator_sys_clk_ctrl =
+ (trf->modulator_sys_clk_ctrl & 0xf8) |
+ TRF7970A_MODULATOR_DEPTH_ASK10;
trf->guard_time = TRF7970A_GUARD_TIME_NFCB;
break;
case NFC_DIGITAL_RF_TECH_212F:
trf->iso_ctrl_tech = TRF7970A_ISO_CTRL_FELICA_212;
- trf->modulator_sys_clk_ctrl = TRF7970A_MODULATOR_DEPTH_ASK10;
+ trf->modulator_sys_clk_ctrl =
+ (trf->modulator_sys_clk_ctrl & 0xf8) |
+ TRF7970A_MODULATOR_DEPTH_ASK10;
trf->guard_time = TRF7970A_GUARD_TIME_NFCF;
break;
case NFC_DIGITAL_RF_TECH_424F:
trf->iso_ctrl_tech = TRF7970A_ISO_CTRL_FELICA_424;
- trf->modulator_sys_clk_ctrl = TRF7970A_MODULATOR_DEPTH_ASK10;
+ trf->modulator_sys_clk_ctrl =
+ (trf->modulator_sys_clk_ctrl & 0xf8) |
+ TRF7970A_MODULATOR_DEPTH_ASK10;
trf->guard_time = TRF7970A_GUARD_TIME_NFCF;
break;
case NFC_DIGITAL_RF_TECH_ISO15693:
trf->iso_ctrl_tech = TRF7970A_ISO_CTRL_15693_SGL_1OF4_2648;
- trf->modulator_sys_clk_ctrl = TRF7970A_MODULATOR_DEPTH_OOK;
+ trf->modulator_sys_clk_ctrl =
+ (trf->modulator_sys_clk_ctrl & 0xf8) |
+ TRF7970A_MODULATOR_DEPTH_OOK;
trf->guard_time = TRF7970A_GUARD_TIME_15693;
break;
default:
@@ -1571,17 +1589,23 @@
trf->iso_ctrl_tech = TRF7970A_ISO_CTRL_NFC_NFC_CE_MODE |
TRF7970A_ISO_CTRL_NFC_CE |
TRF7970A_ISO_CTRL_NFC_CE_14443A;
- trf->modulator_sys_clk_ctrl = TRF7970A_MODULATOR_DEPTH_OOK;
+ trf->modulator_sys_clk_ctrl =
+ (trf->modulator_sys_clk_ctrl & 0xf8) |
+ TRF7970A_MODULATOR_DEPTH_OOK;
break;
case NFC_DIGITAL_RF_TECH_212F:
trf->iso_ctrl_tech = TRF7970A_ISO_CTRL_NFC_NFC_CE_MODE |
TRF7970A_ISO_CTRL_NFC_NFCF_212;
- trf->modulator_sys_clk_ctrl = TRF7970A_MODULATOR_DEPTH_ASK10;
+ trf->modulator_sys_clk_ctrl =
+ (trf->modulator_sys_clk_ctrl & 0xf8) |
+ TRF7970A_MODULATOR_DEPTH_ASK10;
break;
case NFC_DIGITAL_RF_TECH_424F:
trf->iso_ctrl_tech = TRF7970A_ISO_CTRL_NFC_NFC_CE_MODE |
TRF7970A_ISO_CTRL_NFC_NFCF_424;
- trf->modulator_sys_clk_ctrl = TRF7970A_MODULATOR_DEPTH_ASK10;
+ trf->modulator_sys_clk_ctrl =
+ (trf->modulator_sys_clk_ctrl & 0xf8) |
+ TRF7970A_MODULATOR_DEPTH_ASK10;
break;
default:
dev_dbg(trf->dev, "Unsupported rf technology: %d\n", tech);
@@ -1749,7 +1773,7 @@
goto out_err;
ret = trf7970a_write(trf, TRF7970A_REG_IO_CTRL,
- TRF7970A_REG_IO_CTRL_VRS(0x1));
+ trf->io_ctrl | TRF7970A_REG_IO_CTRL_VRS(0x1));
if (ret)
goto out_err;
@@ -1885,8 +1909,10 @@
usleep_range(5000, 6000);
if (!(trf->quirks & TRF7970A_QUIRK_EN2_MUST_STAY_LOW)) {
- gpio_set_value(trf->en2_gpio, 1);
- usleep_range(1000, 2000);
+ if (gpio_is_valid(trf->en2_gpio)) {
+ gpio_set_value(trf->en2_gpio, 1);
+ usleep_range(1000, 2000);
+ }
}
gpio_set_value(trf->en_gpio, 1);
@@ -1914,7 +1940,8 @@
}
gpio_set_value(trf->en_gpio, 0);
- gpio_set_value(trf->en2_gpio, 0);
+ if (gpio_is_valid(trf->en2_gpio))
+ gpio_set_value(trf->en2_gpio, 0);
ret = regulator_disable(trf->regulator);
if (ret)
@@ -1987,6 +2014,7 @@
struct device_node *np = spi->dev.of_node;
struct trf7970a *trf;
int uvolts, autosuspend_delay, ret;
+ u32 clk_freq = TRF7970A_13MHZ_CLOCK_FREQUENCY;
if (!np) {
dev_err(&spi->dev, "No Device Tree entry\n");
@@ -2032,15 +2060,23 @@
trf->en2_gpio = of_get_named_gpio(np, "ti,enable-gpios", 1);
if (!gpio_is_valid(trf->en2_gpio)) {
- dev_err(trf->dev, "No EN2 GPIO property\n");
- return trf->en2_gpio;
+ dev_info(trf->dev, "No EN2 GPIO property\n");
+ } else {
+ ret = devm_gpio_request_one(trf->dev, trf->en2_gpio,
+ GPIOF_DIR_OUT | GPIOF_INIT_LOW, "trf7970a EN2");
+ if (ret) {
+ dev_err(trf->dev, "Can't request EN2 GPIO: %d\n", ret);
+ return ret;
+ }
}
- ret = devm_gpio_request_one(trf->dev, trf->en2_gpio,
- GPIOF_DIR_OUT | GPIOF_INIT_LOW, "trf7970a EN2");
- if (ret) {
- dev_err(trf->dev, "Can't request EN2 GPIO: %d\n", ret);
- return ret;
+ of_property_read_u32(np, "clock-frequency", &clk_freq);
+ if ((clk_freq != TRF7970A_27MHZ_CLOCK_FREQUENCY) ||
+ (clk_freq != TRF7970A_13MHZ_CLOCK_FREQUENCY)) {
+ dev_err(trf->dev,
+ "clock-frequency (%u Hz) unsupported\n",
+ clk_freq);
+ return -EINVAL;
}
if (of_property_read_bool(np, "en2-rf-quirk"))
@@ -2077,6 +2113,24 @@
if (uvolts > 4000000)
trf->chip_status_ctrl = TRF7970A_CHIP_STATUS_VRS5_3;
+ trf->regulator = devm_regulator_get(&spi->dev, "vdd-io");
+ if (IS_ERR(trf->regulator)) {
+ ret = PTR_ERR(trf->regulator);
+ dev_err(trf->dev, "Can't get VDD_IO regulator: %d\n", ret);
+ goto err_destroy_lock;
+ }
+
+ ret = regulator_enable(trf->regulator);
+ if (ret) {
+ dev_err(trf->dev, "Can't enable VDD_IO: %d\n", ret);
+ goto err_destroy_lock;
+ }
+
+ if (regulator_get_voltage(trf->regulator) == 1800000) {
+ trf->io_ctrl = TRF7970A_REG_IO_CTRL_IO_LOW;
+ dev_dbg(trf->dev, "trf7970a config vdd_io to 1.8V\n");
+ }
+
trf->ddev = nfc_digital_allocate_device(&trf7970a_nfc_ops,
TRF7970A_SUPPORTED_PROTOCOLS,
NFC_DIGITAL_DRV_CAPS_IN_CRC |
diff --git a/drivers/nubus/nubus.c b/drivers/nubus/nubus.c
index 3319cf1..77a48a5 100644
--- a/drivers/nubus/nubus.c
+++ b/drivers/nubus/nubus.c
@@ -44,8 +44,8 @@
/* Globals */
-struct nubus_dev* nubus_devices;
-struct nubus_board* nubus_boards;
+struct nubus_dev *nubus_devices;
+struct nubus_board *nubus_boards;
/* Meaning of "bytelanes":
@@ -69,26 +69,26 @@
Etcetera, etcetera. Hopefully this clears up some confusion over
what the following code actually does. */
-
+
static inline int not_useful(void *p, int map)
{
- unsigned long pv=(unsigned long)p;
+ unsigned long pv = (unsigned long)p;
+
pv &= 3;
- if(map & (1<<pv))
+ if (map & (1 << pv))
return 0;
return 1;
}
-
+
static unsigned long nubus_get_rom(unsigned char **ptr, int len, int map)
{
/* This will hold the result */
unsigned long v = 0;
unsigned char *p = *ptr;
- while(len)
- {
+ while (len) {
v <<= 8;
- while(not_useful(p,map))
+ while (not_useful(p, map))
p++;
v |= *p++;
len--;
@@ -99,31 +99,28 @@
static void nubus_rewind(unsigned char **ptr, int len, int map)
{
- unsigned char *p=*ptr;
+ unsigned char *p = *ptr;
/* Sanity check */
- if(len > 65536)
- printk(KERN_ERR "rewind of 0x%08x!\n", len);
- while(len)
- {
- do
- {
+ if (len > 65536)
+ pr_err("rewind of 0x%08x!\n", len);
+ while (len) {
+ do {
p--;
- }
- while(not_useful(p, map));
+ } while (not_useful(p, map));
len--;
}
- *ptr=p;
+ *ptr = p;
}
static void nubus_advance(unsigned char **ptr, int len, int map)
{
unsigned char *p = *ptr;
- if(len>65536)
- printk(KERN_ERR "advance of 0x%08x!\n", len);
- while(len)
- {
- while(not_useful(p,map))
+
+ if (len > 65536)
+ pr_err("advance of 0x%08x!\n", len);
+ while (len) {
+ while (not_useful(p, map))
p++;
p++;
len--;
@@ -133,9 +130,9 @@
static void nubus_move(unsigned char **ptr, int len, int map)
{
- if(len > 0)
+ if (len > 0)
nubus_advance(ptr, len, map);
- else if(len < 0)
+ else if (len < 0)
nubus_rewind(ptr, -len, map);
}
@@ -148,23 +145,24 @@
static inline long nubus_expand32(long foo)
{
- if(foo & 0x00800000) /* 24bit negative */
+ if (foo & 0x00800000) /* 24bit negative */
foo |= 0xFF000000;
return foo;
}
static inline void *nubus_rom_addr(int slot)
-{
+{
/*
* Returns the first byte after the card. We then walk
* backwards to get the lane register and the config
*/
- return (void *)(0xF1000000+(slot<<24));
+ return (void *)(0xF1000000 + (slot << 24));
}
static unsigned char *nubus_dirptr(const struct nubus_dirent *nd)
{
unsigned char *p = nd->base;
+
/* Essentially, just step over the bytelanes using whatever
offset we might have found */
nubus_move(&p, nubus_expand32(nd->data), nd->mask);
@@ -175,36 +173,36 @@
/* These two are for pulling resource data blocks (i.e. stuff that's
pointed to with offsets) out of the card ROM. */
-void nubus_get_rsrc_mem(void *dest, const struct nubus_dirent* dirent,
+void nubus_get_rsrc_mem(void *dest, const struct nubus_dirent *dirent,
int len)
{
unsigned char *t = (unsigned char *)dest;
unsigned char *p = nubus_dirptr(dirent);
- while(len)
- {
+
+ while (len) {
*t++ = nubus_get_rom(&p, 1, dirent->mask);
len--;
}
}
EXPORT_SYMBOL(nubus_get_rsrc_mem);
-void nubus_get_rsrc_str(void *dest, const struct nubus_dirent* dirent,
+void nubus_get_rsrc_str(void *dest, const struct nubus_dirent *dirent,
int len)
{
- unsigned char *t=(unsigned char *)dest;
+ unsigned char *t = (unsigned char *)dest;
unsigned char *p = nubus_dirptr(dirent);
- while(len)
- {
+
+ while (len) {
*t = nubus_get_rom(&p, 1, dirent->mask);
- if(!*t++)
+ if (!*t++)
break;
len--;
}
}
EXPORT_SYMBOL(nubus_get_rsrc_str);
-int nubus_get_root_dir(const struct nubus_board* board,
- struct nubus_dir* dir)
+int nubus_get_root_dir(const struct nubus_board *board,
+ struct nubus_dir *dir)
{
dir->ptr = dir->base = board->directory;
dir->done = 0;
@@ -214,8 +212,8 @@
EXPORT_SYMBOL(nubus_get_root_dir);
/* This is a slyly renamed version of the above */
-int nubus_get_func_dir(const struct nubus_dev* dev,
- struct nubus_dir* dir)
+int nubus_get_func_dir(const struct nubus_dev *dev,
+ struct nubus_dir *dir)
{
dir->ptr = dir->base = dev->directory;
dir->done = 0;
@@ -224,11 +222,11 @@
}
EXPORT_SYMBOL(nubus_get_func_dir);
-int nubus_get_board_dir(const struct nubus_board* board,
- struct nubus_dir* dir)
+int nubus_get_board_dir(const struct nubus_board *board,
+ struct nubus_dir *dir)
{
struct nubus_dirent ent;
-
+
dir->ptr = dir->base = board->directory;
dir->done = 0;
dir->mask = board->lanes;
@@ -256,6 +254,7 @@
int nubus_readdir(struct nubus_dir *nd, struct nubus_dirent *ent)
{
u32 resid;
+
if (nd->done)
return -1;
@@ -266,25 +265,25 @@
resid = nubus_get_rom(&nd->ptr, 4, nd->mask);
/* EOL marker, as per the Apple docs */
- if((resid&0xff000000) == 0xff000000)
- {
+ if ((resid & 0xff000000) == 0xff000000) {
/* Mark it as done */
nd->done = 1;
return -1;
}
/* First byte is the resource ID */
- ent->type = resid >> 24;
+ ent->type = resid >> 24;
/* Low 3 bytes might contain data (or might not) */
ent->data = resid & 0xffffff;
- ent->mask = nd->mask;
+ ent->mask = nd->mask;
return 0;
}
EXPORT_SYMBOL(nubus_readdir);
-int nubus_rewinddir(struct nubus_dir* dir)
+int nubus_rewinddir(struct nubus_dir *dir)
{
dir->ptr = dir->base;
+ dir->done = 0;
return 0;
}
EXPORT_SYMBOL(nubus_rewinddir);
@@ -292,20 +291,15 @@
/* Driver interface functions, more or less like in pci.c */
struct nubus_dev*
-nubus_find_device(unsigned short category,
- unsigned short type,
- unsigned short dr_hw,
- unsigned short dr_sw,
- const struct nubus_dev* from)
+nubus_find_device(unsigned short category, unsigned short type,
+ unsigned short dr_hw, unsigned short dr_sw,
+ const struct nubus_dev *from)
{
- struct nubus_dev* itor =
- from ? from->next : nubus_devices;
+ struct nubus_dev *itor = from ? from->next : nubus_devices;
while (itor) {
- if (itor->category == category
- && itor->type == type
- && itor->dr_hw == dr_hw
- && itor->dr_sw == dr_sw)
+ if (itor->category == category && itor->type == type &&
+ itor->dr_hw == dr_hw && itor->dr_sw == dr_sw)
return itor;
itor = itor->next;
}
@@ -314,16 +308,13 @@
EXPORT_SYMBOL(nubus_find_device);
struct nubus_dev*
-nubus_find_type(unsigned short category,
- unsigned short type,
- const struct nubus_dev* from)
+nubus_find_type(unsigned short category, unsigned short type,
+ const struct nubus_dev *from)
{
- struct nubus_dev* itor =
- from ? from->next : nubus_devices;
+ struct nubus_dev *itor = from ? from->next : nubus_devices;
while (itor) {
- if (itor->category == category
- && itor->type == type)
+ if (itor->category == category && itor->type == type)
return itor;
itor = itor->next;
}
@@ -332,12 +323,10 @@
EXPORT_SYMBOL(nubus_find_type);
struct nubus_dev*
-nubus_find_slot(unsigned int slot,
- const struct nubus_dev* from)
+nubus_find_slot(unsigned int slot, const struct nubus_dev *from)
{
- struct nubus_dev* itor =
- from ? from->next : nubus_devices;
-
+ struct nubus_dev *itor = from ? from->next : nubus_devices;
+
while (itor) {
if (itor->board->slot == slot)
return itor;
@@ -348,13 +337,13 @@
EXPORT_SYMBOL(nubus_find_slot);
int
-nubus_find_rsrc(struct nubus_dir* dir, unsigned char rsrc_type,
- struct nubus_dirent* ent)
+nubus_find_rsrc(struct nubus_dir *dir, unsigned char rsrc_type,
+ struct nubus_dirent *ent)
{
while (nubus_readdir(dir, ent) != -1) {
if (ent->type == rsrc_type)
return 0;
- }
+ }
return -1;
}
EXPORT_SYMBOL(nubus_find_rsrc);
@@ -368,77 +357,74 @@
among other things. The rest of it should go in the /proc code.
For now, we just use it to give verbose boot logs. */
-static int __init nubus_show_display_resource(struct nubus_dev* dev,
- const struct nubus_dirent* ent)
+static int __init nubus_show_display_resource(struct nubus_dev *dev,
+ const struct nubus_dirent *ent)
{
switch (ent->type) {
case NUBUS_RESID_GAMMADIR:
- printk(KERN_INFO " gamma directory offset: 0x%06x\n", ent->data);
+ pr_info(" gamma directory offset: 0x%06x\n", ent->data);
break;
case 0x0080 ... 0x0085:
- printk(KERN_INFO " mode %02X info offset: 0x%06x\n",
+ pr_info(" mode %02X info offset: 0x%06x\n",
ent->type, ent->data);
break;
default:
- printk(KERN_INFO " unknown resource %02X, data 0x%06x\n",
+ pr_info(" unknown resource %02X, data 0x%06x\n",
ent->type, ent->data);
}
return 0;
}
-static int __init nubus_show_network_resource(struct nubus_dev* dev,
- const struct nubus_dirent* ent)
+static int __init nubus_show_network_resource(struct nubus_dev *dev,
+ const struct nubus_dirent *ent)
{
switch (ent->type) {
case NUBUS_RESID_MAC_ADDRESS:
{
char addr[6];
- int i;
-
+
nubus_get_rsrc_mem(addr, ent, 6);
- printk(KERN_INFO " MAC address: ");
- for (i = 0; i < 6; i++)
- printk("%02x%s", addr[i] & 0xff,
- i == 5 ? "" : ":");
- printk("\n");
+ pr_info(" MAC address: %pM\n", addr);
break;
}
default:
- printk(KERN_INFO " unknown resource %02X, data 0x%06x\n",
+ pr_info(" unknown resource %02X, data 0x%06x\n",
ent->type, ent->data);
}
return 0;
}
-static int __init nubus_show_cpu_resource(struct nubus_dev* dev,
- const struct nubus_dirent* ent)
+static int __init nubus_show_cpu_resource(struct nubus_dev *dev,
+ const struct nubus_dirent *ent)
{
switch (ent->type) {
case NUBUS_RESID_MEMINFO:
{
unsigned long meminfo[2];
+
nubus_get_rsrc_mem(&meminfo, ent, 8);
- printk(KERN_INFO " memory: [ 0x%08lx 0x%08lx ]\n",
+ pr_info(" memory: [ 0x%08lx 0x%08lx ]\n",
meminfo[0], meminfo[1]);
break;
}
case NUBUS_RESID_ROMINFO:
{
unsigned long rominfo[2];
+
nubus_get_rsrc_mem(&rominfo, ent, 8);
- printk(KERN_INFO " ROM: [ 0x%08lx 0x%08lx ]\n",
+ pr_info(" ROM: [ 0x%08lx 0x%08lx ]\n",
rominfo[0], rominfo[1]);
break;
}
default:
- printk(KERN_INFO " unknown resource %02X, data 0x%06x\n",
+ pr_info(" unknown resource %02X, data 0x%06x\n",
ent->type, ent->data);
}
return 0;
}
-static int __init nubus_show_private_resource(struct nubus_dev* dev,
- const struct nubus_dirent* ent)
+static int __init nubus_show_private_resource(struct nubus_dev *dev,
+ const struct nubus_dirent *ent)
{
switch (dev->category) {
case NUBUS_CAT_DISPLAY:
@@ -451,59 +437,56 @@
nubus_show_cpu_resource(dev, ent);
break;
default:
- printk(KERN_INFO " unknown resource %02X, data 0x%06x\n",
+ pr_info(" unknown resource %02X, data 0x%06x\n",
ent->type, ent->data);
}
return 0;
}
-static struct nubus_dev* __init
- nubus_get_functional_resource(struct nubus_board* board,
- int slot,
- const struct nubus_dirent* parent)
+static struct nubus_dev * __init
+nubus_get_functional_resource(struct nubus_board *board, int slot,
+ const struct nubus_dirent *parent)
{
- struct nubus_dir dir;
+ struct nubus_dir dir;
struct nubus_dirent ent;
- struct nubus_dev* dev;
-
- printk(KERN_INFO " Function 0x%02x:\n", parent->type);
+ struct nubus_dev *dev;
+
+ pr_info(" Function 0x%02x:\n", parent->type);
nubus_get_subdir(parent, &dir);
/* Apple seems to have botched the ROM on the IIx */
if (slot == 0 && (unsigned long)dir.base % 2)
dir.base += 1;
-
- if (console_loglevel >= CONSOLE_LOGLEVEL_DEBUG)
- printk(KERN_DEBUG "nubus_get_functional_resource: parent is 0x%p, dir is 0x%p\n",
- parent->base, dir.base);
+
+ pr_debug("%s: parent is 0x%p, dir is 0x%p\n",
+ __func__, parent->base, dir.base);
/* Actually we should probably panic if this fails */
if ((dev = kzalloc(sizeof(*dev), GFP_ATOMIC)) == NULL)
- return NULL;
+ return NULL;
dev->resid = parent->type;
dev->directory = dir.base;
dev->board = board;
-
- while (nubus_readdir(&dir, &ent) != -1)
- {
- switch(ent.type)
- {
+
+ while (nubus_readdir(&dir, &ent) != -1) {
+ switch (ent.type) {
case NUBUS_RESID_TYPE:
{
unsigned short nbtdata[4];
+
nubus_get_rsrc_mem(nbtdata, &ent, 8);
dev->category = nbtdata[0];
dev->type = nbtdata[1];
dev->dr_sw = nbtdata[2];
dev->dr_hw = nbtdata[3];
- printk(KERN_INFO " type: [cat 0x%x type 0x%x hw 0x%x sw 0x%x]\n",
- nbtdata[0], nbtdata[1], nbtdata[2], nbtdata[3]);
+ pr_info(" type: [cat 0x%x type 0x%x sw 0x%x hw 0x%x]\n",
+ nbtdata[0], nbtdata[1], nbtdata[2], nbtdata[3]);
break;
}
case NUBUS_RESID_NAME:
{
nubus_get_rsrc_str(dev->name, &ent, 64);
- printk(KERN_INFO " name: %s\n", dev->name);
+ pr_info(" name: %s\n", dev->name);
break;
}
case NUBUS_RESID_DRVRDIR:
@@ -512,11 +495,11 @@
use this :-) */
struct nubus_dir drvr_dir;
struct nubus_dirent drvr_ent;
+
nubus_get_subdir(&ent, &drvr_dir);
nubus_readdir(&drvr_dir, &drvr_ent);
dev->driver = nubus_dirptr(&drvr_ent);
- printk(KERN_INFO " driver at: 0x%p\n",
- dev->driver);
+ pr_info(" driver at: 0x%p\n", dev->driver);
break;
}
case NUBUS_RESID_MINOR_BASEOS:
@@ -524,22 +507,20 @@
multiple framebuffers. It might be handy
for Ethernet as well */
nubus_get_rsrc_mem(&dev->iobase, &ent, 4);
- printk(KERN_INFO " memory offset: 0x%08lx\n",
- dev->iobase);
+ pr_info(" memory offset: 0x%08lx\n", dev->iobase);
break;
case NUBUS_RESID_MINOR_LENGTH:
/* Ditto */
nubus_get_rsrc_mem(&dev->iosize, &ent, 4);
- printk(KERN_INFO " memory length: 0x%08lx\n",
- dev->iosize);
- break;
+ pr_info(" memory length: 0x%08lx\n", dev->iosize);
+ break;
case NUBUS_RESID_FLAGS:
dev->flags = ent.data;
- printk(KERN_INFO " flags: 0x%06x\n", dev->flags);
+ pr_info(" flags: 0x%06x\n", dev->flags);
break;
case NUBUS_RESID_HWDEVID:
dev->hwdevid = ent.data;
- printk(KERN_INFO " hwdevid: 0x%06x\n", dev->hwdevid);
+ pr_info(" hwdevid: 0x%06x\n", dev->hwdevid);
break;
default:
/* Local/Private resources have their own
@@ -547,16 +528,17 @@
nubus_show_private_resource(dev, &ent);
}
}
-
+
return dev;
}
/* This is cool. */
-static int __init nubus_get_vidnames(struct nubus_board* board,
- const struct nubus_dirent* parent)
+static int __init nubus_get_vidnames(struct nubus_board *board,
+ const struct nubus_dirent *parent)
{
- struct nubus_dir dir;
+ struct nubus_dir dir;
struct nubus_dirent ent;
+
/* FIXME: obviously we want to put this in a header file soon */
struct vidmode {
u32 size;
@@ -566,100 +548,92 @@
char name[32];
};
- printk(KERN_INFO " video modes supported:\n");
+ pr_info(" video modes supported:\n");
nubus_get_subdir(parent, &dir);
- if (console_loglevel >= CONSOLE_LOGLEVEL_DEBUG)
- printk(KERN_DEBUG "nubus_get_vidnames: parent is 0x%p, dir is 0x%p\n",
- parent->base, dir.base);
+ pr_debug("%s: parent is 0x%p, dir is 0x%p\n",
+ __func__, parent->base, dir.base);
- while(nubus_readdir(&dir, &ent) != -1)
- {
+ while (nubus_readdir(&dir, &ent) != -1) {
struct vidmode mode;
u32 size;
/* First get the length */
nubus_get_rsrc_mem(&size, &ent, 4);
-
+
/* Now clobber the whole thing */
if (size > sizeof(mode) - 1)
size = sizeof(mode) - 1;
memset(&mode, 0, sizeof(mode));
nubus_get_rsrc_mem(&mode, &ent, size);
- printk (KERN_INFO " %02X: (%02X) %s\n", ent.type,
+ pr_info(" %02X: (%02X) %s\n", ent.type,
mode.id, mode.name);
}
return 0;
}
/* This is *really* cool. */
-static int __init nubus_get_icon(struct nubus_board* board,
- const struct nubus_dirent* ent)
+static int __init nubus_get_icon(struct nubus_board *board,
+ const struct nubus_dirent *ent)
{
/* Should be 32x32 if my memory serves me correctly */
unsigned char icon[128];
int x, y;
-
+
nubus_get_rsrc_mem(&icon, ent, 128);
- printk(KERN_INFO " icon:\n");
+ pr_info(" icon:\n");
/* We should actually plot these somewhere in the framebuffer
init. This is just to demonstrate that they do, in fact,
exist */
for (y = 0; y < 32; y++) {
- printk(KERN_INFO " ");
+ pr_info(" ");
for (x = 0; x < 32; x++) {
- if (icon[y*4 + x/8]
- & (0x80 >> (x%8)))
- printk("*");
+ if (icon[y * 4 + x / 8] & (0x80 >> (x % 8)))
+ pr_cont("*");
else
- printk(" ");
+ pr_cont(" ");
}
- printk("\n");
+ pr_cont("\n");
}
return 0;
}
-static int __init nubus_get_vendorinfo(struct nubus_board* board,
- const struct nubus_dirent* parent)
+static int __init nubus_get_vendorinfo(struct nubus_board *board,
+ const struct nubus_dirent *parent)
{
- struct nubus_dir dir;
+ struct nubus_dir dir;
struct nubus_dirent ent;
- static char* vendor_fields[6] = {"ID", "serial", "revision",
- "part", "date", "unknown field"};
+ static char *vendor_fields[6] = { "ID", "serial", "revision",
+ "part", "date", "unknown field" };
- printk(KERN_INFO " vendor info:\n");
+ pr_info(" vendor info:\n");
nubus_get_subdir(parent, &dir);
- if (console_loglevel >= CONSOLE_LOGLEVEL_DEBUG)
- printk(KERN_DEBUG "nubus_get_vendorinfo: parent is 0x%p, dir is 0x%p\n",
- parent->base, dir.base);
+ pr_debug("%s: parent is 0x%p, dir is 0x%p\n",
+ __func__, parent->base, dir.base);
- while(nubus_readdir(&dir, &ent) != -1)
- {
+ while (nubus_readdir(&dir, &ent) != -1) {
char name[64];
-
+
/* These are all strings, we think */
nubus_get_rsrc_str(name, &ent, 64);
if (ent.type > 5)
ent.type = 5;
- printk(KERN_INFO " %s: %s\n",
- vendor_fields[ent.type-1], name);
+ pr_info(" %s: %s\n", vendor_fields[ent.type - 1], name);
}
return 0;
}
-static int __init nubus_get_board_resource(struct nubus_board* board, int slot,
- const struct nubus_dirent* parent)
+static int __init nubus_get_board_resource(struct nubus_board *board, int slot,
+ const struct nubus_dirent *parent)
{
- struct nubus_dir dir;
+ struct nubus_dir dir;
struct nubus_dirent ent;
-
- nubus_get_subdir(parent, &dir);
- if (console_loglevel >= CONSOLE_LOGLEVEL_DEBUG)
- printk(KERN_DEBUG "nubus_get_board_resource: parent is 0x%p, dir is 0x%p\n",
- parent->base, dir.base);
- while(nubus_readdir(&dir, &ent) != -1)
- {
+ nubus_get_subdir(parent, &dir);
+ pr_debug("%s: parent is 0x%p, dir is 0x%p\n",
+ __func__, parent->base, dir.base);
+
+ while (nubus_readdir(&dir, &ent) != -1) {
switch (ent.type) {
case NUBUS_RESID_TYPE:
{
@@ -668,50 +642,49 @@
useful except insofar as it tells us that
we really are looking at a board resource. */
nubus_get_rsrc_mem(nbtdata, &ent, 8);
- printk(KERN_INFO " type: [cat 0x%x type 0x%x hw 0x%x sw 0x%x]\n",
- nbtdata[0], nbtdata[1], nbtdata[2],
- nbtdata[3]);
+ pr_info(" type: [cat 0x%x type 0x%x sw 0x%x hw 0x%x]\n",
+ nbtdata[0], nbtdata[1], nbtdata[2], nbtdata[3]);
if (nbtdata[0] != 1 || nbtdata[1] != 0 ||
nbtdata[2] != 0 || nbtdata[3] != 0)
- printk(KERN_ERR "this sResource is not a board resource!\n");
+ pr_err("this sResource is not a board resource!\n");
break;
}
case NUBUS_RESID_NAME:
nubus_get_rsrc_str(board->name, &ent, 64);
- printk(KERN_INFO " name: %s\n", board->name);
+ pr_info(" name: %s\n", board->name);
break;
case NUBUS_RESID_ICON:
nubus_get_icon(board, &ent);
break;
case NUBUS_RESID_BOARDID:
- printk(KERN_INFO " board id: 0x%x\n", ent.data);
+ pr_info(" board id: 0x%x\n", ent.data);
break;
case NUBUS_RESID_PRIMARYINIT:
- printk(KERN_INFO " primary init offset: 0x%06x\n", ent.data);
+ pr_info(" primary init offset: 0x%06x\n", ent.data);
break;
case NUBUS_RESID_VENDORINFO:
nubus_get_vendorinfo(board, &ent);
break;
case NUBUS_RESID_FLAGS:
- printk(KERN_INFO " flags: 0x%06x\n", ent.data);
+ pr_info(" flags: 0x%06x\n", ent.data);
break;
case NUBUS_RESID_HWDEVID:
- printk(KERN_INFO " hwdevid: 0x%06x\n", ent.data);
+ pr_info(" hwdevid: 0x%06x\n", ent.data);
break;
case NUBUS_RESID_SECONDINIT:
- printk(KERN_INFO " secondary init offset: 0x%06x\n", ent.data);
+ pr_info(" secondary init offset: 0x%06x\n", ent.data);
break;
- /* WTF isn't this in the functional resources? */
+ /* WTF isn't this in the functional resources? */
case NUBUS_RESID_VIDNAMES:
nubus_get_vidnames(board, &ent);
break;
/* Same goes for this */
case NUBUS_RESID_VIDMODES:
- printk(KERN_INFO " video mode parameter directory offset: 0x%06x\n",
+ pr_info(" video mode parameter directory offset: 0x%06x\n",
ent.data);
- break;
+ break;
default:
- printk(KERN_INFO " unknown resource %02X, data 0x%06x\n",
+ pr_info(" unknown resource %02X, data 0x%06x\n",
ent.type, ent.data);
}
}
@@ -722,8 +695,8 @@
sResources in the motherboard ROM */
static void __init nubus_find_rom_dir(struct nubus_board* board)
{
- unsigned char* rp;
- unsigned char* romdir;
+ unsigned char *rp;
+ unsigned char *romdir;
struct nubus_dir dir;
struct nubus_dirent ent;
@@ -756,14 +729,14 @@
if (console_loglevel >= CONSOLE_LOGLEVEL_DEBUG)
printk(KERN_INFO "nubus_get_rom_dir: entry %02x %06x\n", ent.type, ent.data);
/* This one takes us to where we want to go. */
- if (nubus_readdir(&dir, &ent) == -1)
+ if (nubus_readdir(&dir, &ent) == -1)
goto badrom;
if (console_loglevel >= CONSOLE_LOGLEVEL_DEBUG)
printk(KERN_DEBUG "nubus_get_rom_dir: entry %02x %06x\n", ent.type, ent.data);
nubus_get_subdir(&ent, &dir);
/* Resource ID 01, also an "Unknown Macintosh" */
- if (nubus_readdir(&dir, &ent) == -1)
+ if (nubus_readdir(&dir, &ent) == -1)
goto badrom;
if (console_loglevel >= CONSOLE_LOGLEVEL_DEBUG)
printk(KERN_DEBUG "nubus_get_rom_dir: entry %02x %06x\n", ent.type, ent.data);
@@ -782,12 +755,12 @@
goto badrom;
if (console_loglevel >= CONSOLE_LOGLEVEL_DEBUG)
printk(KERN_DEBUG "nubus_get_rom_dir: entry %02x %06x\n", ent.type, ent.data);
-
+
/* Bwahahahaha... */
nubus_get_subdir(&ent, &dir);
board->directory = dir.base;
return;
-
+
/* Even more evil laughter... */
badrom:
board->directory = board->fblock;
@@ -796,43 +769,38 @@
}
/* Add a board (might be many devices) to the list */
-static struct nubus_board* __init nubus_add_board(int slot, int bytelanes)
+static struct nubus_board * __init nubus_add_board(int slot, int bytelanes)
{
- struct nubus_board* board;
- struct nubus_board** boardp;
-
+ struct nubus_board *board;
+ struct nubus_board **boardp;
unsigned char *rp;
unsigned long dpat;
struct nubus_dir dir;
struct nubus_dirent ent;
/* Move to the start of the format block */
- rp = nubus_rom_addr(slot);
+ rp = nubus_rom_addr(slot);
nubus_rewind(&rp, FORMAT_BLOCK_SIZE, bytelanes);
/* Actually we should probably panic if this fails */
if ((board = kzalloc(sizeof(*board), GFP_ATOMIC)) == NULL)
- return NULL;
+ return NULL;
board->fblock = rp;
/* Dump the format block for debugging purposes */
- if (console_loglevel >= CONSOLE_LOGLEVEL_DEBUG) {
- int i;
- printk(KERN_DEBUG "Slot %X, format block at 0x%p\n",
- slot, rp);
- printk(KERN_DEBUG "Format block: ");
- for (i = 0; i < FORMAT_BLOCK_SIZE; i += 4) {
- unsigned short foo, bar;
- foo = nubus_get_rom(&rp, 2, bytelanes);
- bar = nubus_get_rom(&rp, 2, bytelanes);
- printk("%04x %04x ", foo, bar);
- }
- printk("\n");
- rp = board->fblock;
- }
-
+ pr_debug("Slot %X, format block at 0x%p:\n", slot, rp);
+ pr_debug("%02lx\n", nubus_get_rom(&rp, 1, bytelanes));
+ pr_debug("%02lx\n", nubus_get_rom(&rp, 1, bytelanes));
+ pr_debug("%08lx\n", nubus_get_rom(&rp, 4, bytelanes));
+ pr_debug("%02lx\n", nubus_get_rom(&rp, 1, bytelanes));
+ pr_debug("%02lx\n", nubus_get_rom(&rp, 1, bytelanes));
+ pr_debug("%08lx\n", nubus_get_rom(&rp, 4, bytelanes));
+ pr_debug("%08lx\n", nubus_get_rom(&rp, 4, bytelanes));
+ pr_debug("%08lx\n", nubus_get_rom(&rp, 4, bytelanes));
+ rp = board->fblock;
+
board->slot = slot;
- board->slot_addr = (unsigned long) nubus_slot_addr(slot);
+ board->slot_addr = (unsigned long)nubus_slot_addr(slot);
board->doffset = nubus_get_rom(&rp, 4, bytelanes);
/* rom_length is *supposed* to be the total length of the
* ROM. In practice it is the "amount of ROM used to compute
@@ -843,16 +811,16 @@
board->rom_length = nubus_get_rom(&rp, 4, bytelanes);
board->crc = nubus_get_rom(&rp, 4, bytelanes);
board->rev = nubus_get_rom(&rp, 1, bytelanes);
- board->format = nubus_get_rom(&rp,1, bytelanes);
+ board->format = nubus_get_rom(&rp, 1, bytelanes);
board->lanes = bytelanes;
/* Directory offset should be small and negative... */
- if(!(board->doffset & 0x00FF0000))
- printk(KERN_WARNING "Dodgy doffset!\n");
+ if (!(board->doffset & 0x00FF0000))
+ pr_warn("Dodgy doffset!\n");
dpat = nubus_get_rom(&rp, 4, bytelanes);
- if(dpat != NUBUS_TEST_PATTERN)
- printk(KERN_WARNING "Wrong test pattern %08lx!\n", dpat);
-
+ if (dpat != NUBUS_TEST_PATTERN)
+ pr_warn("Wrong test pattern %08lx!\n", dpat);
+
/*
* I wonder how the CRC is meant to work -
* any takers ?
@@ -862,10 +830,10 @@
/* Attempt to work around slot zero weirdness */
nubus_find_rom_dir(board);
- nubus_get_root_dir(board, &dir);
+ nubus_get_root_dir(board, &dir);
/* We're ready to rock */
- printk(KERN_INFO "Slot %X:\n", slot);
+ pr_info("Slot %X:\n", slot);
/* Each slot should have one board resource and any number of
functional resources. So we'll fill in some fields in the
@@ -874,10 +842,10 @@
for each of them. */
if (nubus_readdir(&dir, &ent) == -1) {
/* We can't have this! */
- printk(KERN_ERR "Board resource not found!\n");
+ pr_err("Board resource not found!\n");
return NULL;
} else {
- printk(KERN_INFO " Board resource:\n");
+ pr_info(" Board resource:\n");
nubus_get_board_resource(board, slot, &ent);
}
@@ -885,8 +853,9 @@
resources. I have no idea WTF to do about this. */
while (nubus_readdir(&dir, &ent) != -1) {
- struct nubus_dev* dev;
- struct nubus_dev** devp;
+ struct nubus_dev *dev;
+ struct nubus_dev **devp;
+
dev = nubus_get_functional_resource(board, slot, &ent);
if (dev == NULL)
continue;
@@ -894,32 +863,33 @@
/* We zeroed this out above */
if (board->first_dev == NULL)
board->first_dev = dev;
-
+
/* Put it on the global NuBus device chain. Keep entries in order. */
- for (devp=&nubus_devices; *devp!=NULL; devp=&((*devp)->next))
+ for (devp = &nubus_devices; *devp != NULL;
+ devp = &((*devp)->next))
/* spin */;
*devp = dev;
- dev->next = NULL;
+ dev->next = NULL;
}
/* Put it on the global NuBus board chain. Keep entries in order. */
- for (boardp=&nubus_boards; *boardp!=NULL; boardp=&((*boardp)->next))
+ for (boardp = &nubus_boards; *boardp != NULL;
+ boardp = &((*boardp)->next))
/* spin */;
*boardp = board;
board->next = NULL;
-
+
return board;
}
void __init nubus_probe_slot(int slot)
{
unsigned char dp;
- unsigned char* rp;
+ unsigned char *rp;
int i;
- rp = nubus_rom_addr(slot);
- for(i = 4; i; i--)
- {
+ rp = nubus_rom_addr(slot);
+ for (i = 4; i; i--) {
int card_present;
rp--;
@@ -927,7 +897,6 @@
if (!card_present)
continue;
- printk(KERN_DEBUG "Now probing slot %X at %p\n", slot, rp);
dp = *rp;
if(dp == 0)
continue;
@@ -935,11 +904,11 @@
/* The last byte of the format block consists of two
nybbles which are "mirror images" of each other.
These show us the valid bytelanes */
- if ((((dp>>4) ^ dp) & 0x0F) != 0x0F)
+ if ((((dp >> 4) ^ dp) & 0x0F) != 0x0F)
continue;
/* Check that this value is actually *on* one of the
bytelanes it claims are valid! */
- if ((dp & 0x0F) >= (1<<i))
+ if ((dp & 0x0F) >= (1 << i))
continue;
/* Looks promising. Let's put it on the list. */
@@ -952,19 +921,19 @@
void __init nubus_scan_bus(void)
{
int slot;
+
/* This might not work on your machine */
#ifdef I_WANT_TO_PROBE_SLOT_ZERO
nubus_probe_slot(0);
#endif
- for(slot = 9; slot < 15; slot++)
- {
+ for (slot = 9; slot < 15; slot++) {
nubus_probe_slot(slot);
}
}
static int __init nubus_init(void)
{
- if (!MACH_IS_MAC)
+ if (!MACH_IS_MAC)
return 0;
/* Initialize the NuBus interrupts */
@@ -980,11 +949,11 @@
gurus can fix the real cause of the problem. */
mdelay(1000);
#endif
-
+
/* And probe */
- printk("NuBus: Scanning NuBus slots.\n");
+ pr_info("NuBus: Scanning NuBus slots.\n");
nubus_devices = NULL;
- nubus_boards = NULL;
+ nubus_boards = NULL;
nubus_scan_bus();
nubus_proc_init();
return 0;
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index 5b536be..fbc640b 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -25,6 +25,7 @@
#include <linux/badblocks.h>
#include <linux/memremap.h>
#include <linux/vmalloc.h>
+#include <linux/blk-mq.h>
#include <linux/pfn_t.h>
#include <linux/slab.h>
#include <linux/pmem.h>
@@ -231,6 +232,11 @@
blk_cleanup_queue(q);
}
+static void pmem_freeze_queue(void *q)
+{
+ blk_freeze_queue_start(q);
+}
+
static void pmem_release_disk(void *disk)
{
del_gendisk(disk);
@@ -284,6 +290,9 @@
if (!q)
return -ENOMEM;
+ if (devm_add_action_or_reset(dev, pmem_release_queue, q))
+ return -ENOMEM;
+
pmem->pfn_flags = PFN_DEV;
if (is_nd_pfn(dev)) {
addr = devm_memremap_pages(dev, &pfn_res, &q->q_usage_counter,
@@ -303,10 +312,10 @@
pmem->size, ARCH_MEMREMAP_PMEM);
/*
- * At release time the queue must be dead before
+ * At release time the queue must be frozen before
* devm_memremap_pages is unwound
*/
- if (devm_add_action_or_reset(dev, pmem_release_queue, q))
+ if (devm_add_action_or_reset(dev, pmem_freeze_queue, q))
return -ENOMEM;
if (IS_ERR(addr))
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index eeb409c..d5e0906 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -49,10 +49,9 @@
module_param(shutdown_timeout, byte, 0644);
MODULE_PARM_DESC(shutdown_timeout, "timeout in seconds for controller shutdown");
-unsigned int nvme_max_retries = 5;
-module_param_named(max_retries, nvme_max_retries, uint, 0644);
+static u8 nvme_max_retries = 5;
+module_param_named(max_retries, nvme_max_retries, byte, 0644);
MODULE_PARM_DESC(max_retries, "max number of retries a command may have");
-EXPORT_SYMBOL_GPL(nvme_max_retries);
static int nvme_char_major;
module_param(nvme_char_major, int, 0);
@@ -62,11 +61,66 @@
MODULE_PARM_DESC(default_ps_max_latency_us,
"max power saving latency for new devices; use PM QOS to change per device");
+static bool force_apst;
+module_param(force_apst, bool, 0644);
+MODULE_PARM_DESC(force_apst, "allow APST for newly enumerated devices even if quirked off");
+
static LIST_HEAD(nvme_ctrl_list);
static DEFINE_SPINLOCK(dev_list_lock);
static struct class *nvme_class;
+static int nvme_error_status(struct request *req)
+{
+ switch (nvme_req(req)->status & 0x7ff) {
+ case NVME_SC_SUCCESS:
+ return 0;
+ case NVME_SC_CAP_EXCEEDED:
+ return -ENOSPC;
+ default:
+ return -EIO;
+
+ /*
+ * XXX: these errors are a nasty side-band protocol to
+ * drivers/md/dm-mpath.c:noretry_error() that aren't documented
+ * anywhere..
+ */
+ case NVME_SC_CMD_SEQ_ERROR:
+ return -EILSEQ;
+ case NVME_SC_ONCS_NOT_SUPPORTED:
+ return -EOPNOTSUPP;
+ case NVME_SC_WRITE_FAULT:
+ case NVME_SC_READ_ERROR:
+ case NVME_SC_UNWRITTEN_BLOCK:
+ return -ENODATA;
+ }
+}
+
+static inline bool nvme_req_needs_retry(struct request *req)
+{
+ if (blk_noretry_request(req))
+ return false;
+ if (nvme_req(req)->status & NVME_SC_DNR)
+ return false;
+ if (jiffies - req->start_time >= req->timeout)
+ return false;
+ if (nvme_req(req)->retries >= nvme_max_retries)
+ return false;
+ return true;
+}
+
+void nvme_complete_rq(struct request *req)
+{
+ if (unlikely(nvme_req(req)->status && nvme_req_needs_retry(req))) {
+ nvme_req(req)->retries++;
+ blk_mq_requeue_request(req, !blk_mq_queue_stopped(req->q));
+ return;
+ }
+
+ blk_mq_end_request(req, nvme_error_status(req));
+}
+EXPORT_SYMBOL_GPL(nvme_complete_rq);
+
void nvme_cancel_request(struct request *req, void *data, bool reserved)
{
int status;
@@ -80,7 +134,9 @@
status = NVME_SC_ABORT_REQ;
if (blk_queue_dying(req->q))
status |= NVME_SC_DNR;
- blk_mq_complete_request(req, status);
+ nvme_req(req)->status = status;
+ blk_mq_complete_request(req);
+
}
EXPORT_SYMBOL_GPL(nvme_cancel_request);
@@ -205,12 +261,6 @@
return NULL;
}
-void nvme_requeue_req(struct request *req)
-{
- blk_mq_requeue_request(req, !blk_mq_queue_stopped(req->q));
-}
-EXPORT_SYMBOL_GPL(nvme_requeue_req);
-
struct request *nvme_alloc_request(struct request_queue *q,
struct nvme_command *cmd, unsigned int flags, int qid)
{
@@ -327,6 +377,12 @@
{
int ret = BLK_MQ_RQ_QUEUE_OK;
+ if (!(req->rq_flags & RQF_DONTPREP)) {
+ nvme_req(req)->retries = 0;
+ nvme_req(req)->flags = 0;
+ req->rq_flags |= RQF_DONTPREP;
+ }
+
switch (req_op(req)) {
case REQ_OP_DRV_IN:
case REQ_OP_DRV_OUT:
@@ -335,6 +391,8 @@
case REQ_OP_FLUSH:
nvme_setup_flush(ns, cmd);
break;
+ case REQ_OP_WRITE_ZEROES:
+ /* currently only aliased to deallocate for a few ctrls: */
case REQ_OP_DISCARD:
ret = nvme_setup_discard(ns, req, cmd);
break;
@@ -378,7 +436,10 @@
blk_execute_rq(req->q, NULL, req, at_head);
if (result)
*result = nvme_req(req)->result;
- ret = req->errors;
+ if (nvme_req(req)->flags & NVME_REQ_CANCELLED)
+ ret = -EINTR;
+ else
+ ret = nvme_req(req)->status;
out:
blk_mq_free_request(req);
return ret;
@@ -463,7 +524,10 @@
}
submit:
blk_execute_rq(req->q, disk, req, 0);
- ret = req->errors;
+ if (nvme_req(req)->flags & NVME_REQ_CANCELLED)
+ ret = -EINTR;
+ else
+ ret = nvme_req(req)->status;
if (result)
*result = le32_to_cpu(nvme_req(req)->result.u32);
if (meta && !ret && !write) {
@@ -900,16 +964,14 @@
BUILD_BUG_ON(PAGE_SIZE / sizeof(struct nvme_dsm_range) <
NVME_DSM_MAX_RANGES);
- if (ctrl->quirks & NVME_QUIRK_DISCARD_ZEROES)
- ns->queue->limits.discard_zeroes_data = 1;
- else
- ns->queue->limits.discard_zeroes_data = 0;
-
ns->queue->limits.discard_alignment = logical_block_size;
ns->queue->limits.discard_granularity = logical_block_size;
blk_queue_max_discard_sectors(ns->queue, UINT_MAX);
blk_queue_max_discard_segments(ns->queue, NVME_DSM_MAX_RANGES);
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, ns->queue);
+
+ if (ctrl->quirks & NVME_QUIRK_DEALLOCATE_ZEROES)
+ blk_queue_max_write_zeroes_sectors(ns->queue, UINT_MAX);
}
static int nvme_revalidate_ns(struct nvme_ns *ns, struct nvme_id_ns **id)
@@ -1267,7 +1329,7 @@
* heuristic: we are willing to spend at most 2% of the time
* transitioning between power states. Therefore, when running
* in any given state, we will enter the next lower-power
- * non-operational state after waiting 100 * (enlat + exlat)
+ * non-operational state after waiting 50 * (enlat + exlat)
* microseconds, as long as that state's total latency is under
* the requested maximum latency.
*
@@ -1278,6 +1340,8 @@
unsigned apste;
struct nvme_feat_auto_pst *table;
+ u64 max_lat_us = 0;
+ int max_ps = -1;
int ret;
/*
@@ -1299,6 +1363,7 @@
if (ctrl->ps_max_latency_us == 0) {
/* Turn off APST. */
apste = 0;
+ dev_dbg(ctrl->device, "APST disabled\n");
} else {
__le64 target = cpu_to_le64(0);
int state;
@@ -1348,9 +1413,22 @@
target = cpu_to_le64((state << 3) |
(transition_ms << 8));
+
+ if (max_ps == -1)
+ max_ps = state;
+
+ if (total_latency_us > max_lat_us)
+ max_lat_us = total_latency_us;
}
apste = 1;
+
+ if (max_ps == -1) {
+ dev_dbg(ctrl->device, "APST enabled but no non-operational states are available\n");
+ } else {
+ dev_dbg(ctrl->device, "APST enabled: max PS = %d, max round-trip latency = %lluus, table = %*phN\n",
+ max_ps, max_lat_us, (int)sizeof(*table), table);
+ }
}
ret = nvme_set_features(ctrl, NVME_FEAT_AUTO_PST, apste,
@@ -1488,6 +1566,11 @@
}
}
+ if (force_apst && (ctrl->quirks & NVME_QUIRK_NO_DEEPEST_PS)) {
+ dev_warn(ctrl->dev, "forcibly allowing all power states due to nvme_core.force_apst -- use at your own risk\n");
+ ctrl->quirks &= ~NVME_QUIRK_NO_DEEPEST_PS;
+ }
+
ctrl->oacs = le16_to_cpu(id->oacs);
ctrl->vid = le16_to_cpu(id->vid);
ctrl->oncs = le16_to_cpup(&id->oncs);
@@ -1510,7 +1593,16 @@
ctrl->npss = id->npss;
prev_apsta = ctrl->apsta;
- ctrl->apsta = (ctrl->quirks & NVME_QUIRK_NO_APST) ? 0 : id->apsta;
+ if (ctrl->quirks & NVME_QUIRK_NO_APST) {
+ if (force_apst && id->apsta) {
+ dev_warn(ctrl->dev, "forcibly allowing APST due to nvme_core.force_apst -- use at your own risk\n");
+ ctrl->apsta = 1;
+ } else {
+ ctrl->apsta = 0;
+ }
+ } else {
+ ctrl->apsta = id->apsta;
+ }
memcpy(ctrl->psd, id->psd, sizeof(ctrl->psd));
if (ctrl->ops->is_fabrics) {
@@ -2393,7 +2485,7 @@
mutex_lock(&ctrl->namespaces_mutex);
list_for_each_entry(ns, &ctrl->namespaces, list)
- blk_mq_freeze_queue_start(ns->queue);
+ blk_freeze_queue_start(ns->queue);
mutex_unlock(&ctrl->namespaces_mutex);
}
EXPORT_SYMBOL_GPL(nvme_start_freeze);
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 5b7386f..990e6fb 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -471,6 +471,16 @@
}
EXPORT_SYMBOL_GPL(nvmf_connect_io_queue);
+bool nvmf_should_reconnect(struct nvme_ctrl *ctrl)
+{
+ if (ctrl->opts->max_reconnects != -1 &&
+ ctrl->opts->nr_reconnects < ctrl->opts->max_reconnects)
+ return true;
+
+ return false;
+}
+EXPORT_SYMBOL_GPL(nvmf_should_reconnect);
+
/**
* nvmf_register_transport() - NVMe Fabrics Library registration function.
* @ops: Transport ops instance to be registered to the
@@ -533,6 +543,7 @@
{ NVMF_OPT_QUEUE_SIZE, "queue_size=%d" },
{ NVMF_OPT_NR_IO_QUEUES, "nr_io_queues=%d" },
{ NVMF_OPT_RECONNECT_DELAY, "reconnect_delay=%d" },
+ { NVMF_OPT_CTRL_LOSS_TMO, "ctrl_loss_tmo=%d" },
{ NVMF_OPT_KATO, "keep_alive_tmo=%d" },
{ NVMF_OPT_HOSTNQN, "hostnqn=%s" },
{ NVMF_OPT_HOST_TRADDR, "host_traddr=%s" },
@@ -546,6 +557,7 @@
char *options, *o, *p;
int token, ret = 0;
size_t nqnlen = 0;
+ int ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO;
/* Set defaults */
opts->queue_size = NVMF_DEF_QUEUE_SIZE;
@@ -655,6 +667,16 @@
}
opts->kato = token;
break;
+ case NVMF_OPT_CTRL_LOSS_TMO:
+ if (match_int(args, &token)) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ if (token < 0)
+ pr_warn("ctrl_loss_tmo < 0 will reconnect forever\n");
+ ctrl_loss_tmo = token;
+ break;
case NVMF_OPT_HOSTNQN:
if (opts->host) {
pr_err("hostnqn already user-assigned: %s\n",
@@ -710,6 +732,12 @@
}
}
+ if (ctrl_loss_tmo < 0)
+ opts->max_reconnects = -1;
+ else
+ opts->max_reconnects = DIV_ROUND_UP(ctrl_loss_tmo,
+ opts->reconnect_delay);
+
if (!opts->host) {
kref_get(&nvmf_default_host->ref);
opts->host = nvmf_default_host;
diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h
index 1560181..f5a9c1f 100644
--- a/drivers/nvme/host/fabrics.h
+++ b/drivers/nvme/host/fabrics.h
@@ -21,6 +21,8 @@
#define NVMF_MAX_QUEUE_SIZE 1024
#define NVMF_DEF_QUEUE_SIZE 128
#define NVMF_DEF_RECONNECT_DELAY 10
+/* default to 600 seconds of reconnect attempts before giving up */
+#define NVMF_DEF_CTRL_LOSS_TMO 600
/*
* Define a host as seen by the target. We allocate one at boot, but also
@@ -53,6 +55,7 @@
NVMF_OPT_HOSTNQN = 1 << 8,
NVMF_OPT_RECONNECT_DELAY = 1 << 9,
NVMF_OPT_HOST_TRADDR = 1 << 10,
+ NVMF_OPT_CTRL_LOSS_TMO = 1 << 11,
};
/**
@@ -77,6 +80,10 @@
* @discovery_nqn: indicates if the subsysnqn is the well-known discovery NQN.
* @kato: Keep-alive timeout.
* @host: Virtual NVMe host, contains the NQN and Host ID.
+ * @nr_reconnects: number of reconnect attempted since the last ctrl failure
+ * @max_reconnects: maximum number of allowed reconnect attempts before removing
+ * the controller, (-1) means reconnect forever, zero means remove
+ * immediately;
*/
struct nvmf_ctrl_options {
unsigned mask;
@@ -91,6 +98,8 @@
bool discovery_nqn;
unsigned int kato;
struct nvmf_host *host;
+ int nr_reconnects;
+ int max_reconnects;
};
/*
@@ -133,5 +142,6 @@
void nvmf_free_options(struct nvmf_ctrl_options *opts);
const char *nvmf_get_subsysnqn(struct nvme_ctrl *ctrl);
int nvmf_get_address(struct nvme_ctrl *ctrl, char *buf, int size);
+bool nvmf_should_reconnect(struct nvme_ctrl *ctrl);
#endif /* _NVME_FABRICS_H */
diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index d996ca7..4976db5 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -19,6 +19,7 @@
#include <linux/parser.h>
#include <uapi/scsi/fc/fc_fs.h>
#include <uapi/scsi/fc/fc_els.h>
+#include <linux/delay.h>
#include "nvme.h"
#include "fabrics.h"
@@ -44,6 +45,8 @@
#define NVMEFC_QUEUE_DELAY 3 /* ms units */
+#define NVME_FC_MAX_CONNECT_ATTEMPTS 1
+
struct nvme_fc_queue {
struct nvme_fc_ctrl *ctrl;
struct device *dev;
@@ -61,16 +64,24 @@
unsigned long flags;
} __aligned(sizeof(u64)); /* alignment for other things alloc'd with */
+enum nvme_fcop_flags {
+ FCOP_FLAGS_TERMIO = (1 << 0),
+ FCOP_FLAGS_RELEASED = (1 << 1),
+ FCOP_FLAGS_COMPLETE = (1 << 2),
+ FCOP_FLAGS_AEN = (1 << 3),
+};
+
struct nvmefc_ls_req_op {
struct nvmefc_ls_req ls_req;
- struct nvme_fc_ctrl *ctrl;
+ struct nvme_fc_rport *rport;
struct nvme_fc_queue *queue;
struct request *rq;
+ u32 flags;
int ls_error;
struct completion ls_done;
- struct list_head lsreq_list; /* ctrl->ls_req_list */
+ struct list_head lsreq_list; /* rport->ls_req_list */
bool req_queued;
};
@@ -79,6 +90,7 @@
FCPOP_STATE_IDLE = 1,
FCPOP_STATE_ACTIVE = 2,
FCPOP_STATE_ABORTED = 3,
+ FCPOP_STATE_COMPLETE = 4,
};
struct nvme_fc_fcp_op {
@@ -97,6 +109,7 @@
struct request *rq;
atomic_t state;
+ u32 flags;
u32 rqno;
u32 nents;
@@ -120,23 +133,24 @@
struct list_head endp_list; /* for lport->endp_list */
struct list_head ctrl_list;
+ struct list_head ls_req_list;
+ struct device *dev; /* physical device for dma */
+ struct nvme_fc_lport *lport;
spinlock_t lock;
struct kref ref;
} __aligned(sizeof(u64)); /* alignment for other things alloc'd with */
-enum nvme_fcctrl_state {
- FCCTRL_INIT = 0,
- FCCTRL_ACTIVE = 1,
+enum nvme_fcctrl_flags {
+ FCCTRL_TERMIO = (1 << 0),
};
struct nvme_fc_ctrl {
spinlock_t lock;
struct nvme_fc_queue *queues;
- u32 queue_count;
-
struct device *dev;
struct nvme_fc_lport *lport;
struct nvme_fc_rport *rport;
+ u32 queue_count;
u32 cnum;
u64 association_id;
@@ -144,14 +158,19 @@
u64 cap;
struct list_head ctrl_list; /* rport->ctrl_list */
- struct list_head ls_req_list;
struct blk_mq_tag_set admin_tag_set;
struct blk_mq_tag_set tag_set;
struct work_struct delete_work;
+ struct work_struct reset_work;
+ struct delayed_work connect_work;
+ int reconnect_delay;
+ int connect_attempts;
+
struct kref ref;
- int state;
+ u32 flags;
+ u32 iocnt;
struct nvme_fc_fcp_op aen_ops[NVME_FC_NR_AEN_COMMANDS];
@@ -419,9 +438,12 @@
INIT_LIST_HEAD(&newrec->endp_list);
INIT_LIST_HEAD(&newrec->ctrl_list);
+ INIT_LIST_HEAD(&newrec->ls_req_list);
kref_init(&newrec->ref);
spin_lock_init(&newrec->lock);
newrec->remoteport.localport = &lport->localport;
+ newrec->dev = lport->dev;
+ newrec->lport = lport;
newrec->remoteport.private = &newrec[1];
newrec->remoteport.port_role = pinfo->port_role;
newrec->remoteport.node_name = pinfo->node_name;
@@ -444,7 +466,6 @@
out_reghost_failed:
*portptr = NULL;
return ret;
-
}
EXPORT_SYMBOL_GPL(nvme_fc_register_remoteport);
@@ -487,6 +508,30 @@
return kref_get_unless_zero(&rport->ref);
}
+static int
+nvme_fc_abort_lsops(struct nvme_fc_rport *rport)
+{
+ struct nvmefc_ls_req_op *lsop;
+ unsigned long flags;
+
+restart:
+ spin_lock_irqsave(&rport->lock, flags);
+
+ list_for_each_entry(lsop, &rport->ls_req_list, lsreq_list) {
+ if (!(lsop->flags & FCOP_FLAGS_TERMIO)) {
+ lsop->flags |= FCOP_FLAGS_TERMIO;
+ spin_unlock_irqrestore(&rport->lock, flags);
+ rport->lport->ops->ls_abort(&rport->lport->localport,
+ &rport->remoteport,
+ &lsop->ls_req);
+ goto restart;
+ }
+ }
+ spin_unlock_irqrestore(&rport->lock, flags);
+
+ return 0;
+}
+
/**
* nvme_fc_unregister_remoteport - transport entry point called by an
* LLDD to deregister/remove a previously
@@ -522,6 +567,8 @@
spin_unlock_irqrestore(&rport->lock, flags);
+ nvme_fc_abort_lsops(rport);
+
nvme_fc_rport_put(rport);
return 0;
}
@@ -624,16 +671,16 @@
static void
-__nvme_fc_finish_ls_req(struct nvme_fc_ctrl *ctrl,
- struct nvmefc_ls_req_op *lsop)
+__nvme_fc_finish_ls_req(struct nvmefc_ls_req_op *lsop)
{
+ struct nvme_fc_rport *rport = lsop->rport;
struct nvmefc_ls_req *lsreq = &lsop->ls_req;
unsigned long flags;
- spin_lock_irqsave(&ctrl->lock, flags);
+ spin_lock_irqsave(&rport->lock, flags);
if (!lsop->req_queued) {
- spin_unlock_irqrestore(&ctrl->lock, flags);
+ spin_unlock_irqrestore(&rport->lock, flags);
return;
}
@@ -641,56 +688,71 @@
lsop->req_queued = false;
- spin_unlock_irqrestore(&ctrl->lock, flags);
+ spin_unlock_irqrestore(&rport->lock, flags);
- fc_dma_unmap_single(ctrl->dev, lsreq->rqstdma,
+ fc_dma_unmap_single(rport->dev, lsreq->rqstdma,
(lsreq->rqstlen + lsreq->rsplen),
DMA_BIDIRECTIONAL);
- nvme_fc_ctrl_put(ctrl);
+ nvme_fc_rport_put(rport);
}
static int
-__nvme_fc_send_ls_req(struct nvme_fc_ctrl *ctrl,
+__nvme_fc_send_ls_req(struct nvme_fc_rport *rport,
struct nvmefc_ls_req_op *lsop,
void (*done)(struct nvmefc_ls_req *req, int status))
{
struct nvmefc_ls_req *lsreq = &lsop->ls_req;
unsigned long flags;
- int ret;
+ int ret = 0;
- if (!nvme_fc_ctrl_get(ctrl))
+ if (rport->remoteport.port_state != FC_OBJSTATE_ONLINE)
+ return -ECONNREFUSED;
+
+ if (!nvme_fc_rport_get(rport))
return -ESHUTDOWN;
lsreq->done = done;
- lsop->ctrl = ctrl;
+ lsop->rport = rport;
lsop->req_queued = false;
INIT_LIST_HEAD(&lsop->lsreq_list);
init_completion(&lsop->ls_done);
- lsreq->rqstdma = fc_dma_map_single(ctrl->dev, lsreq->rqstaddr,
+ lsreq->rqstdma = fc_dma_map_single(rport->dev, lsreq->rqstaddr,
lsreq->rqstlen + lsreq->rsplen,
DMA_BIDIRECTIONAL);
- if (fc_dma_mapping_error(ctrl->dev, lsreq->rqstdma)) {
- nvme_fc_ctrl_put(ctrl);
- dev_err(ctrl->dev,
- "els request command failed EFAULT.\n");
- return -EFAULT;
+ if (fc_dma_mapping_error(rport->dev, lsreq->rqstdma)) {
+ ret = -EFAULT;
+ goto out_putrport;
}
lsreq->rspdma = lsreq->rqstdma + lsreq->rqstlen;
- spin_lock_irqsave(&ctrl->lock, flags);
+ spin_lock_irqsave(&rport->lock, flags);
- list_add_tail(&lsop->lsreq_list, &ctrl->ls_req_list);
+ list_add_tail(&lsop->lsreq_list, &rport->ls_req_list);
lsop->req_queued = true;
- spin_unlock_irqrestore(&ctrl->lock, flags);
+ spin_unlock_irqrestore(&rport->lock, flags);
- ret = ctrl->lport->ops->ls_req(&ctrl->lport->localport,
- &ctrl->rport->remoteport, lsreq);
+ ret = rport->lport->ops->ls_req(&rport->lport->localport,
+ &rport->remoteport, lsreq);
if (ret)
- lsop->ls_error = ret;
+ goto out_unlink;
+
+ return 0;
+
+out_unlink:
+ lsop->ls_error = ret;
+ spin_lock_irqsave(&rport->lock, flags);
+ lsop->req_queued = false;
+ list_del(&lsop->lsreq_list);
+ spin_unlock_irqrestore(&rport->lock, flags);
+ fc_dma_unmap_single(rport->dev, lsreq->rqstdma,
+ (lsreq->rqstlen + lsreq->rsplen),
+ DMA_BIDIRECTIONAL);
+out_putrport:
+ nvme_fc_rport_put(rport);
return ret;
}
@@ -705,15 +767,15 @@
}
static int
-nvme_fc_send_ls_req(struct nvme_fc_ctrl *ctrl, struct nvmefc_ls_req_op *lsop)
+nvme_fc_send_ls_req(struct nvme_fc_rport *rport, struct nvmefc_ls_req_op *lsop)
{
struct nvmefc_ls_req *lsreq = &lsop->ls_req;
struct fcnvme_ls_rjt *rjt = lsreq->rspaddr;
int ret;
- ret = __nvme_fc_send_ls_req(ctrl, lsop, nvme_fc_send_ls_req_done);
+ ret = __nvme_fc_send_ls_req(rport, lsop, nvme_fc_send_ls_req_done);
- if (!ret)
+ if (!ret) {
/*
* No timeout/not interruptible as we need the struct
* to exist until the lldd calls us back. Thus mandate
@@ -722,14 +784,14 @@
*/
wait_for_completion(&lsop->ls_done);
- __nvme_fc_finish_ls_req(ctrl, lsop);
+ __nvme_fc_finish_ls_req(lsop);
- if (ret) {
- dev_err(ctrl->dev,
- "ls request command failed (%d).\n", ret);
- return ret;
+ ret = lsop->ls_error;
}
+ if (ret)
+ return ret;
+
/* ACC or RJT payload ? */
if (rjt->w0.ls_cmd == FCNVME_LS_RJT)
return -ENXIO;
@@ -737,19 +799,14 @@
return 0;
}
-static void
-nvme_fc_send_ls_req_async(struct nvme_fc_ctrl *ctrl,
+static int
+nvme_fc_send_ls_req_async(struct nvme_fc_rport *rport,
struct nvmefc_ls_req_op *lsop,
void (*done)(struct nvmefc_ls_req *req, int status))
{
- int ret;
-
- ret = __nvme_fc_send_ls_req(ctrl, lsop, done);
-
/* don't wait for completion */
- if (ret)
- done(&lsop->ls_req, ret);
+ return __nvme_fc_send_ls_req(rport, lsop, done);
}
/* Validation Error indexes into the string table below */
@@ -839,7 +896,7 @@
lsreq->rsplen = sizeof(*assoc_acc);
lsreq->timeout = NVME_FC_CONNECT_TIMEOUT_SEC;
- ret = nvme_fc_send_ls_req(ctrl, lsop);
+ ret = nvme_fc_send_ls_req(ctrl->rport, lsop);
if (ret)
goto out_free_buffer;
@@ -848,11 +905,12 @@
/* validate the ACC response */
if (assoc_acc->hdr.w0.ls_cmd != FCNVME_LS_ACC)
fcret = VERR_LSACC;
- if (assoc_acc->hdr.desc_list_len !=
+ else if (assoc_acc->hdr.desc_list_len !=
fcnvme_lsdesc_len(
sizeof(struct fcnvme_ls_cr_assoc_acc)))
fcret = VERR_CR_ASSOC_ACC_LEN;
- if (assoc_acc->hdr.rqst.desc_tag != cpu_to_be32(FCNVME_LSDESC_RQST))
+ else if (assoc_acc->hdr.rqst.desc_tag !=
+ cpu_to_be32(FCNVME_LSDESC_RQST))
fcret = VERR_LSDESC_RQST;
else if (assoc_acc->hdr.rqst.desc_len !=
fcnvme_lsdesc_len(sizeof(struct fcnvme_lsdesc_rqst)))
@@ -946,7 +1004,7 @@
lsreq->rsplen = sizeof(*conn_acc);
lsreq->timeout = NVME_FC_CONNECT_TIMEOUT_SEC;
- ret = nvme_fc_send_ls_req(ctrl, lsop);
+ ret = nvme_fc_send_ls_req(ctrl->rport, lsop);
if (ret)
goto out_free_buffer;
@@ -955,10 +1013,10 @@
/* validate the ACC response */
if (conn_acc->hdr.w0.ls_cmd != FCNVME_LS_ACC)
fcret = VERR_LSACC;
- if (conn_acc->hdr.desc_list_len !=
+ else if (conn_acc->hdr.desc_list_len !=
fcnvme_lsdesc_len(sizeof(struct fcnvme_ls_cr_conn_acc)))
fcret = VERR_CR_CONN_ACC_LEN;
- if (conn_acc->hdr.rqst.desc_tag != cpu_to_be32(FCNVME_LSDESC_RQST))
+ else if (conn_acc->hdr.rqst.desc_tag != cpu_to_be32(FCNVME_LSDESC_RQST))
fcret = VERR_LSDESC_RQST;
else if (conn_acc->hdr.rqst.desc_len !=
fcnvme_lsdesc_len(sizeof(struct fcnvme_lsdesc_rqst)))
@@ -997,14 +1055,8 @@
nvme_fc_disconnect_assoc_done(struct nvmefc_ls_req *lsreq, int status)
{
struct nvmefc_ls_req_op *lsop = ls_req_to_lsop(lsreq);
- struct nvme_fc_ctrl *ctrl = lsop->ctrl;
- __nvme_fc_finish_ls_req(ctrl, lsop);
-
- if (status)
- dev_err(ctrl->dev,
- "disconnect assoc ls request command failed (%d).\n",
- status);
+ __nvme_fc_finish_ls_req(lsop);
/* fc-nvme iniator doesn't care about success or failure of cmd */
@@ -1035,6 +1087,7 @@
struct fcnvme_ls_disconnect_acc *discon_acc;
struct nvmefc_ls_req_op *lsop;
struct nvmefc_ls_req *lsreq;
+ int ret;
lsop = kzalloc((sizeof(*lsop) +
ctrl->lport->ops->lsrqst_priv_sz +
@@ -1077,7 +1130,10 @@
lsreq->rsplen = sizeof(*discon_acc);
lsreq->timeout = NVME_FC_CONNECT_TIMEOUT_SEC;
- nvme_fc_send_ls_req_async(ctrl, lsop, nvme_fc_disconnect_assoc_done);
+ ret = nvme_fc_send_ls_req_async(ctrl->rport, lsop,
+ nvme_fc_disconnect_assoc_done);
+ if (ret)
+ kfree(lsop);
/* only meaningful part to terminating the association */
ctrl->association_id = 0;
@@ -1086,6 +1142,7 @@
/* *********************** NVME Ctrl Routines **************************** */
+static void __nvme_fc_final_op_cleanup(struct request *rq);
static int
nvme_fc_reinit_request(void *data, struct request *rq)
@@ -1123,21 +1180,84 @@
return __nvme_fc_exit_request(data, op);
}
+static int
+__nvme_fc_abort_op(struct nvme_fc_ctrl *ctrl, struct nvme_fc_fcp_op *op)
+{
+ int state;
+
+ state = atomic_xchg(&op->state, FCPOP_STATE_ABORTED);
+ if (state != FCPOP_STATE_ACTIVE) {
+ atomic_set(&op->state, state);
+ return -ECANCELED;
+ }
+
+ ctrl->lport->ops->fcp_abort(&ctrl->lport->localport,
+ &ctrl->rport->remoteport,
+ op->queue->lldd_handle,
+ &op->fcp_req);
+
+ return 0;
+}
+
static void
-nvme_fc_exit_aen_ops(struct nvme_fc_ctrl *ctrl)
+nvme_fc_abort_aen_ops(struct nvme_fc_ctrl *ctrl)
{
struct nvme_fc_fcp_op *aen_op = ctrl->aen_ops;
- int i;
+ unsigned long flags;
+ int i, ret;
for (i = 0; i < NVME_FC_NR_AEN_COMMANDS; i++, aen_op++) {
- if (atomic_read(&aen_op->state) == FCPOP_STATE_UNINIT)
+ if (atomic_read(&aen_op->state) != FCPOP_STATE_ACTIVE)
continue;
- __nvme_fc_exit_request(ctrl, aen_op);
- nvme_fc_ctrl_put(ctrl);
+
+ spin_lock_irqsave(&ctrl->lock, flags);
+ if (ctrl->flags & FCCTRL_TERMIO) {
+ ctrl->iocnt++;
+ aen_op->flags |= FCOP_FLAGS_TERMIO;
+ }
+ spin_unlock_irqrestore(&ctrl->lock, flags);
+
+ ret = __nvme_fc_abort_op(ctrl, aen_op);
+ if (ret) {
+ /*
+ * if __nvme_fc_abort_op failed the io wasn't
+ * active. Thus this call path is running in
+ * parallel to the io complete. Treat as non-error.
+ */
+
+ /* back out the flags/counters */
+ spin_lock_irqsave(&ctrl->lock, flags);
+ if (ctrl->flags & FCCTRL_TERMIO)
+ ctrl->iocnt--;
+ aen_op->flags &= ~FCOP_FLAGS_TERMIO;
+ spin_unlock_irqrestore(&ctrl->lock, flags);
+ return;
+ }
}
}
-void
+static inline int
+__nvme_fc_fcpop_chk_teardowns(struct nvme_fc_ctrl *ctrl,
+ struct nvme_fc_fcp_op *op)
+{
+ unsigned long flags;
+ bool complete_rq = false;
+
+ spin_lock_irqsave(&ctrl->lock, flags);
+ if (unlikely(op->flags & FCOP_FLAGS_TERMIO)) {
+ if (ctrl->flags & FCCTRL_TERMIO)
+ ctrl->iocnt--;
+ }
+ if (op->flags & FCOP_FLAGS_RELEASED)
+ complete_rq = true;
+ else
+ op->flags |= FCOP_FLAGS_COMPLETE;
+ spin_unlock_irqrestore(&ctrl->lock, flags);
+
+ return complete_rq;
+}
+
+static void
nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)
{
struct nvme_fc_fcp_op *op = fcp_req_to_fcp_op(req);
@@ -1146,7 +1266,10 @@
struct nvme_fc_ctrl *ctrl = op->ctrl;
struct nvme_fc_queue *queue = op->queue;
struct nvme_completion *cqe = &op->rsp_iu.cqe;
- u16 status;
+ struct nvme_command *sqe = &op->cmd_iu.sqe;
+ __le16 status = cpu_to_le16(NVME_SC_SUCCESS << 1);
+ union nvme_result result;
+ bool complete_rq;
/*
* WARNING:
@@ -1181,9 +1304,9 @@
sizeof(op->rsp_iu), DMA_FROM_DEVICE);
if (atomic_read(&op->state) == FCPOP_STATE_ABORTED)
- status = NVME_SC_ABORT_REQ | NVME_SC_DNR;
- else
- status = freq->status;
+ status = cpu_to_le16((NVME_SC_ABORT_REQ | NVME_SC_DNR) << 1);
+ else if (freq->status)
+ status = cpu_to_le16(NVME_SC_FC_TRANSPORT_ERROR << 1);
/*
* For the linux implementation, if we have an unsuccesful
@@ -1211,10 +1334,10 @@
*/
if (freq->transferred_length !=
be32_to_cpu(op->cmd_iu.data_len)) {
- status = -EIO;
+ status = cpu_to_le16(NVME_SC_FC_TRANSPORT_ERROR << 1);
goto done;
}
- op->nreq.result.u64 = 0;
+ result.u64 = 0;
break;
case sizeof(struct nvme_fc_ersp_iu):
@@ -1226,28 +1349,40 @@
(freq->rcv_rsplen / 4) ||
be32_to_cpu(op->rsp_iu.xfrd_len) !=
freq->transferred_length ||
- op->rqno != le16_to_cpu(cqe->command_id))) {
- status = -EIO;
+ op->rsp_iu.status_code ||
+ sqe->common.command_id != cqe->command_id)) {
+ status = cpu_to_le16(NVME_SC_FC_TRANSPORT_ERROR << 1);
goto done;
}
- op->nreq.result = cqe->result;
- status = le16_to_cpu(cqe->status) >> 1;
+ result = cqe->result;
+ status = cqe->status;
break;
default:
- status = -EIO;
+ status = cpu_to_le16(NVME_SC_FC_TRANSPORT_ERROR << 1);
goto done;
}
done:
- if (!queue->qnum && op->rqno >= AEN_CMDID_BASE) {
- nvme_complete_async_event(&queue->ctrl->ctrl, status,
- &op->nreq.result);
+ if (op->flags & FCOP_FLAGS_AEN) {
+ nvme_complete_async_event(&queue->ctrl->ctrl, status, &result);
+ complete_rq = __nvme_fc_fcpop_chk_teardowns(ctrl, op);
+ atomic_set(&op->state, FCPOP_STATE_IDLE);
+ op->flags = FCOP_FLAGS_AEN; /* clear other flags */
nvme_fc_ctrl_put(ctrl);
return;
}
- blk_mq_complete_request(rq, status);
+ complete_rq = __nvme_fc_fcpop_chk_teardowns(ctrl, op);
+ if (!complete_rq) {
+ if (unlikely(op->flags & FCOP_FLAGS_TERMIO)) {
+ status = cpu_to_le16(NVME_SC_ABORT_REQ);
+ if (blk_queue_dying(rq->q))
+ status |= cpu_to_le16(NVME_SC_DNR);
+ }
+ nvme_end_request(rq, status, result);
+ } else
+ __nvme_fc_final_op_cleanup(rq);
}
static int
@@ -1328,25 +1463,55 @@
struct nvme_fc_fcp_op *aen_op;
struct nvme_fc_cmd_iu *cmdiu;
struct nvme_command *sqe;
+ void *private;
int i, ret;
aen_op = ctrl->aen_ops;
for (i = 0; i < NVME_FC_NR_AEN_COMMANDS; i++, aen_op++) {
+ private = kzalloc(ctrl->lport->ops->fcprqst_priv_sz,
+ GFP_KERNEL);
+ if (!private)
+ return -ENOMEM;
+
cmdiu = &aen_op->cmd_iu;
sqe = &cmdiu->sqe;
ret = __nvme_fc_init_request(ctrl, &ctrl->queues[0],
aen_op, (struct request *)NULL,
(AEN_CMDID_BASE + i));
- if (ret)
+ if (ret) {
+ kfree(private);
return ret;
+ }
+
+ aen_op->flags = FCOP_FLAGS_AEN;
+ aen_op->fcp_req.first_sgl = NULL; /* no sg list */
+ aen_op->fcp_req.private = private;
memset(sqe, 0, sizeof(*sqe));
sqe->common.opcode = nvme_admin_async_event;
+ /* Note: core layer may overwrite the sqe.command_id value */
sqe->common.command_id = AEN_CMDID_BASE + i;
}
return 0;
}
+static void
+nvme_fc_term_aen_ops(struct nvme_fc_ctrl *ctrl)
+{
+ struct nvme_fc_fcp_op *aen_op;
+ int i;
+
+ aen_op = ctrl->aen_ops;
+ for (i = 0; i < NVME_FC_NR_AEN_COMMANDS; i++, aen_op++) {
+ if (!aen_op->fcp_req.private)
+ continue;
+
+ __nvme_fc_exit_request(ctrl, aen_op);
+
+ kfree(aen_op->fcp_req.private);
+ aen_op->fcp_req.private = NULL;
+ }
+}
static inline void
__nvme_fc_init_hctx(struct blk_mq_hw_ctx *hctx, struct nvme_fc_ctrl *ctrl,
@@ -1446,15 +1611,6 @@
}
static void
-nvme_fc_destroy_admin_queue(struct nvme_fc_ctrl *ctrl)
-{
- __nvme_fc_delete_hw_queue(ctrl, &ctrl->queues[0], 0);
- blk_cleanup_queue(ctrl->ctrl.admin_q);
- blk_mq_free_tag_set(&ctrl->admin_tag_set);
- nvme_fc_free_queue(&ctrl->queues[0]);
-}
-
-static void
nvme_fc_free_io_queues(struct nvme_fc_ctrl *ctrl)
{
int i;
@@ -1541,19 +1697,27 @@
container_of(ref, struct nvme_fc_ctrl, ref);
unsigned long flags;
- if (ctrl->state != FCCTRL_INIT) {
- /* remove from rport list */
- spin_lock_irqsave(&ctrl->rport->lock, flags);
- list_del(&ctrl->ctrl_list);
- spin_unlock_irqrestore(&ctrl->rport->lock, flags);
+ if (ctrl->ctrl.tagset) {
+ blk_cleanup_queue(ctrl->ctrl.connect_q);
+ blk_mq_free_tag_set(&ctrl->tag_set);
}
+ /* remove from rport list */
+ spin_lock_irqsave(&ctrl->rport->lock, flags);
+ list_del(&ctrl->ctrl_list);
+ spin_unlock_irqrestore(&ctrl->rport->lock, flags);
+
+ blk_cleanup_queue(ctrl->ctrl.admin_q);
+ blk_mq_free_tag_set(&ctrl->admin_tag_set);
+
+ kfree(ctrl->queues);
+
put_device(ctrl->dev);
nvme_fc_rport_put(ctrl->rport);
- kfree(ctrl->queues);
ida_simple_remove(&nvme_fc_ctrl_cnt, ctrl->cnum);
- nvmf_free_options(ctrl->ctrl.opts);
+ if (ctrl->ctrl.opts)
+ nvmf_free_options(ctrl->ctrl.opts);
kfree(ctrl);
}
@@ -1574,57 +1738,38 @@
* controller. Called after last nvme_put_ctrl() call
*/
static void
-nvme_fc_free_nvme_ctrl(struct nvme_ctrl *nctrl)
+nvme_fc_nvme_ctrl_freed(struct nvme_ctrl *nctrl)
{
struct nvme_fc_ctrl *ctrl = to_fc_ctrl(nctrl);
WARN_ON(nctrl != &ctrl->ctrl);
- /*
- * Tear down the association, which will generate link
- * traffic to terminate connections
- */
-
- if (ctrl->state != FCCTRL_INIT) {
- /* send a Disconnect(association) LS to fc-nvme target */
- nvme_fc_xmt_disconnect_assoc(ctrl);
-
- if (ctrl->ctrl.tagset) {
- blk_cleanup_queue(ctrl->ctrl.connect_q);
- blk_mq_free_tag_set(&ctrl->tag_set);
- nvme_fc_delete_hw_io_queues(ctrl);
- nvme_fc_free_io_queues(ctrl);
- }
-
- nvme_fc_exit_aen_ops(ctrl);
-
- nvme_fc_destroy_admin_queue(ctrl);
- }
-
nvme_fc_ctrl_put(ctrl);
}
-
-static int
-__nvme_fc_abort_op(struct nvme_fc_ctrl *ctrl, struct nvme_fc_fcp_op *op)
+static void
+nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl, char *errmsg)
{
- int state;
+ dev_warn(ctrl->ctrl.device,
+ "NVME-FC{%d}: transport association error detected: %s\n",
+ ctrl->cnum, errmsg);
+ dev_info(ctrl->ctrl.device,
+ "NVME-FC{%d}: resetting controller\n", ctrl->cnum);
- state = atomic_xchg(&op->state, FCPOP_STATE_ABORTED);
- if (state != FCPOP_STATE_ACTIVE) {
- atomic_set(&op->state, state);
- return -ECANCELED; /* fail */
+ if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RECONNECTING)) {
+ dev_err(ctrl->ctrl.device,
+ "NVME-FC{%d}: error_recovery: Couldn't change state "
+ "to RECONNECTING\n", ctrl->cnum);
+ return;
}
- ctrl->lport->ops->fcp_abort(&ctrl->lport->localport,
- &ctrl->rport->remoteport,
- op->queue->lldd_handle,
- &op->fcp_req);
-
- return 0;
+ if (!queue_work(nvme_fc_wq, &ctrl->reset_work))
+ dev_err(ctrl->ctrl.device,
+ "NVME-FC{%d}: error_recovery: Failed to schedule "
+ "reset work\n", ctrl->cnum);
}
-enum blk_eh_timer_return
+static enum blk_eh_timer_return
nvme_fc_timeout(struct request *rq, bool reserved)
{
struct nvme_fc_fcp_op *op = blk_mq_rq_to_pdu(rq);
@@ -1640,11 +1785,13 @@
return BLK_EH_HANDLED;
/*
- * TODO: force a controller reset
- * when that happens, queues will be torn down and outstanding
- * ios will be terminated, and the above abort, on a single io
- * will no longer be needed.
+ * we can't individually ABTS an io without affecting the queue,
+ * thus killing the queue, adn thus the association.
+ * So resolve by performing a controller reset, which will stop
+ * the host/io stack, terminate the association on the link,
+ * and recreate an association on the link.
*/
+ nvme_fc_error_recovery(ctrl, "io timeout error");
return BLK_EH_HANDLED;
}
@@ -1738,6 +1885,13 @@
u32 csn;
int ret;
+ /*
+ * before attempting to send the io, check to see if we believe
+ * the target device is present
+ */
+ if (ctrl->rport->remoteport.port_state != FC_OBJSTATE_ONLINE)
+ return BLK_MQ_RQ_QUEUE_ERROR;
+
if (!nvme_fc_ctrl_get(ctrl))
return BLK_MQ_RQ_QUEUE_ERROR;
@@ -1761,7 +1915,7 @@
op->fcp_req.io_dir = io_dir;
op->fcp_req.transferred_length = 0;
op->fcp_req.rcv_rsplen = 0;
- op->fcp_req.status = 0;
+ op->fcp_req.status = NVME_SC_SUCCESS;
op->fcp_req.sqid = cpu_to_le16(queue->qnum);
/*
@@ -1782,14 +1936,9 @@
sqe->rw.dptr.sgl.length = cpu_to_le32(data_len);
sqe->rw.dptr.sgl.addr = 0;
- /* odd that we set the command_id - should come from nvme-fabrics */
- WARN_ON_ONCE(sqe->common.command_id != cpu_to_le16(op->rqno));
-
- if (op->rq) { /* skipped on aens */
+ if (!(op->flags & FCOP_FLAGS_AEN)) {
ret = nvme_fc_map_data(ctrl, op->rq, op);
if (ret < 0) {
- dev_err(queue->ctrl->ctrl.device,
- "Failed to map data (%d)\n", ret);
nvme_cleanup_cmd(op->rq);
nvme_fc_ctrl_put(ctrl);
return (ret == -ENOMEM || ret == -EAGAIN) ?
@@ -1802,7 +1951,7 @@
atomic_set(&op->state, FCPOP_STATE_ACTIVE);
- if (op->rq)
+ if (!(op->flags & FCOP_FLAGS_AEN))
blk_mq_start_request(op->rq);
ret = ctrl->lport->ops->fcp_io(&ctrl->lport->localport,
@@ -1810,9 +1959,6 @@
queue->lldd_handle, &op->fcp_req);
if (ret) {
- dev_err(ctrl->dev,
- "Send nvme command failed - lldd returned %d.\n", ret);
-
if (op->rq) { /* normal request */
nvme_fc_unmap_data(ctrl, op->rq, op);
nvme_cleanup_cmd(op->rq);
@@ -1882,12 +2028,8 @@
struct nvme_fc_fcp_op *op;
req = blk_mq_tag_to_rq(nvme_fc_tagset(queue), tag);
- if (!req) {
- dev_err(queue->ctrl->ctrl.device,
- "tag 0x%x on QNum %#x not found\n",
- tag, queue->qnum);
+ if (!req)
return 0;
- }
op = blk_mq_rq_to_pdu(req);
@@ -1904,11 +2046,21 @@
{
struct nvme_fc_ctrl *ctrl = to_fc_ctrl(arg);
struct nvme_fc_fcp_op *aen_op;
+ unsigned long flags;
+ bool terminating = false;
int ret;
if (aer_idx > NVME_FC_NR_AEN_COMMANDS)
return;
+ spin_lock_irqsave(&ctrl->lock, flags);
+ if (ctrl->flags & FCCTRL_TERMIO)
+ terminating = true;
+ spin_unlock_irqrestore(&ctrl->lock, flags);
+
+ if (terminating)
+ return;
+
aen_op = &ctrl->aen_ops[aer_idx];
ret = nvme_fc_start_fcp_op(ctrl, aen_op->queue, aen_op, 0,
@@ -1919,137 +2071,47 @@
}
static void
+__nvme_fc_final_op_cleanup(struct request *rq)
+{
+ struct nvme_fc_fcp_op *op = blk_mq_rq_to_pdu(rq);
+ struct nvme_fc_ctrl *ctrl = op->ctrl;
+
+ atomic_set(&op->state, FCPOP_STATE_IDLE);
+ op->flags &= ~(FCOP_FLAGS_TERMIO | FCOP_FLAGS_RELEASED |
+ FCOP_FLAGS_COMPLETE);
+
+ nvme_cleanup_cmd(rq);
+ nvme_fc_unmap_data(ctrl, rq, op);
+ nvme_complete_rq(rq);
+ nvme_fc_ctrl_put(ctrl);
+
+}
+
+static void
nvme_fc_complete_rq(struct request *rq)
{
struct nvme_fc_fcp_op *op = blk_mq_rq_to_pdu(rq);
struct nvme_fc_ctrl *ctrl = op->ctrl;
- int error = 0, state;
+ unsigned long flags;
+ bool completed = false;
- state = atomic_xchg(&op->state, FCPOP_STATE_IDLE);
+ /*
+ * the core layer, on controller resets after calling
+ * nvme_shutdown_ctrl(), calls complete_rq without our
+ * calling blk_mq_complete_request(), thus there may still
+ * be live i/o outstanding with the LLDD. Means transport has
+ * to track complete calls vs fcpio_done calls to know what
+ * path to take on completes and dones.
+ */
+ spin_lock_irqsave(&ctrl->lock, flags);
+ if (op->flags & FCOP_FLAGS_COMPLETE)
+ completed = true;
+ else
+ op->flags |= FCOP_FLAGS_RELEASED;
+ spin_unlock_irqrestore(&ctrl->lock, flags);
- nvme_cleanup_cmd(rq);
-
- nvme_fc_unmap_data(ctrl, rq, op);
-
- if (unlikely(rq->errors)) {
- if (nvme_req_needs_retry(rq, rq->errors)) {
- nvme_requeue_req(rq);
- return;
- }
-
- if (blk_rq_is_passthrough(rq))
- error = rq->errors;
- else
- error = nvme_error_status(rq->errors);
- }
-
- nvme_fc_ctrl_put(ctrl);
-
- blk_mq_end_request(rq, error);
-}
-
-static struct blk_mq_ops nvme_fc_mq_ops = {
- .queue_rq = nvme_fc_queue_rq,
- .complete = nvme_fc_complete_rq,
- .init_request = nvme_fc_init_request,
- .exit_request = nvme_fc_exit_request,
- .reinit_request = nvme_fc_reinit_request,
- .init_hctx = nvme_fc_init_hctx,
- .poll = nvme_fc_poll,
- .timeout = nvme_fc_timeout,
-};
-
-static struct blk_mq_ops nvme_fc_admin_mq_ops = {
- .queue_rq = nvme_fc_queue_rq,
- .complete = nvme_fc_complete_rq,
- .init_request = nvme_fc_init_admin_request,
- .exit_request = nvme_fc_exit_request,
- .reinit_request = nvme_fc_reinit_request,
- .init_hctx = nvme_fc_init_admin_hctx,
- .timeout = nvme_fc_timeout,
-};
-
-static int
-nvme_fc_configure_admin_queue(struct nvme_fc_ctrl *ctrl)
-{
- u32 segs;
- int error;
-
- nvme_fc_init_queue(ctrl, 0, NVME_FC_AQ_BLKMQ_DEPTH);
-
- error = nvme_fc_connect_admin_queue(ctrl, &ctrl->queues[0],
- NVME_FC_AQ_BLKMQ_DEPTH,
- (NVME_FC_AQ_BLKMQ_DEPTH / 4));
- if (error)
- return error;
-
- memset(&ctrl->admin_tag_set, 0, sizeof(ctrl->admin_tag_set));
- ctrl->admin_tag_set.ops = &nvme_fc_admin_mq_ops;
- ctrl->admin_tag_set.queue_depth = NVME_FC_AQ_BLKMQ_DEPTH;
- ctrl->admin_tag_set.reserved_tags = 2; /* fabric connect + Keep-Alive */
- ctrl->admin_tag_set.numa_node = NUMA_NO_NODE;
- ctrl->admin_tag_set.cmd_size = sizeof(struct nvme_fc_fcp_op) +
- (SG_CHUNK_SIZE *
- sizeof(struct scatterlist)) +
- ctrl->lport->ops->fcprqst_priv_sz;
- ctrl->admin_tag_set.driver_data = ctrl;
- ctrl->admin_tag_set.nr_hw_queues = 1;
- ctrl->admin_tag_set.timeout = ADMIN_TIMEOUT;
-
- error = blk_mq_alloc_tag_set(&ctrl->admin_tag_set);
- if (error)
- goto out_free_queue;
-
- ctrl->ctrl.admin_q = blk_mq_init_queue(&ctrl->admin_tag_set);
- if (IS_ERR(ctrl->ctrl.admin_q)) {
- error = PTR_ERR(ctrl->ctrl.admin_q);
- goto out_free_tagset;
- }
-
- error = __nvme_fc_create_hw_queue(ctrl, &ctrl->queues[0], 0,
- NVME_FC_AQ_BLKMQ_DEPTH);
- if (error)
- goto out_cleanup_queue;
-
- error = nvmf_connect_admin_queue(&ctrl->ctrl);
- if (error)
- goto out_delete_hw_queue;
-
- error = nvmf_reg_read64(&ctrl->ctrl, NVME_REG_CAP, &ctrl->cap);
- if (error) {
- dev_err(ctrl->ctrl.device,
- "prop_get NVME_REG_CAP failed\n");
- goto out_delete_hw_queue;
- }
-
- ctrl->ctrl.sqsize =
- min_t(int, NVME_CAP_MQES(ctrl->cap), ctrl->ctrl.sqsize);
-
- error = nvme_enable_ctrl(&ctrl->ctrl, ctrl->cap);
- if (error)
- goto out_delete_hw_queue;
-
- segs = min_t(u32, NVME_FC_MAX_SEGMENTS,
- ctrl->lport->ops->max_sgl_segments);
- ctrl->ctrl.max_hw_sectors = (segs - 1) << (PAGE_SHIFT - 9);
-
- error = nvme_init_identify(&ctrl->ctrl);
- if (error)
- goto out_delete_hw_queue;
-
- nvme_start_keep_alive(&ctrl->ctrl);
-
- return 0;
-
-out_delete_hw_queue:
- __nvme_fc_delete_hw_queue(ctrl, &ctrl->queues[0], 0);
-out_cleanup_queue:
- blk_cleanup_queue(ctrl->ctrl.admin_q);
-out_free_tagset:
- blk_mq_free_tag_set(&ctrl->admin_tag_set);
-out_free_queue:
- nvme_fc_free_queue(&ctrl->queues[0]);
- return error;
+ if (completed)
+ __nvme_fc_final_op_cleanup(rq);
}
/*
@@ -2071,150 +2133,47 @@
struct nvme_ctrl *nctrl = data;
struct nvme_fc_ctrl *ctrl = to_fc_ctrl(nctrl);
struct nvme_fc_fcp_op *op = blk_mq_rq_to_pdu(req);
-int status;
+ unsigned long flags;
+ int status;
if (!blk_mq_request_started(req))
return;
- /* this performs an ABTS-LS on the FC exchange for the io */
- status = __nvme_fc_abort_op(ctrl, op);
- /*
- * if __nvme_fc_abort_op failed: io wasn't active to abort
- * consider it done. Assume completion path already completing
- * in parallel
- */
- if (status)
- /* io wasn't active to abort consider it done */
- /* assume completion path already completing in parallel */
- return;
-}
-
-
-/*
- * This routine stops operation of the controller. Admin and IO queues
- * are stopped, outstanding ios on them terminated, and the nvme ctrl
- * is shutdown.
- */
-static void
-nvme_fc_shutdown_ctrl(struct nvme_fc_ctrl *ctrl)
-{
- /*
- * If io queues are present, stop them and terminate all outstanding
- * ios on them. As FC allocates FC exchange for each io, the
- * transport must contact the LLDD to terminate the exchange,
- * thus releasing the FC exchange. We use blk_mq_tagset_busy_itr()
- * to tell us what io's are busy and invoke a transport routine
- * to kill them with the LLDD. After terminating the exchange
- * the LLDD will call the transport's normal io done path, but it
- * will have an aborted status. The done path will return the
- * io requests back to the block layer as part of normal completions
- * (but with error status).
- */
- if (ctrl->queue_count > 1) {
- nvme_stop_queues(&ctrl->ctrl);
- blk_mq_tagset_busy_iter(&ctrl->tag_set,
- nvme_fc_terminate_exchange, &ctrl->ctrl);
+ spin_lock_irqsave(&ctrl->lock, flags);
+ if (ctrl->flags & FCCTRL_TERMIO) {
+ ctrl->iocnt++;
+ op->flags |= FCOP_FLAGS_TERMIO;
}
+ spin_unlock_irqrestore(&ctrl->lock, flags);
- if (ctrl->ctrl.state == NVME_CTRL_LIVE)
- nvme_shutdown_ctrl(&ctrl->ctrl);
+ status = __nvme_fc_abort_op(ctrl, op);
+ if (status) {
+ /*
+ * if __nvme_fc_abort_op failed the io wasn't
+ * active. Thus this call path is running in
+ * parallel to the io complete. Treat as non-error.
+ */
- /*
- * now clean up the admin queue. Same thing as above.
- * use blk_mq_tagset_busy_itr() and the transport routine to
- * terminate the exchanges.
- */
- blk_mq_stop_hw_queues(ctrl->ctrl.admin_q);
- blk_mq_tagset_busy_iter(&ctrl->admin_tag_set,
- nvme_fc_terminate_exchange, &ctrl->ctrl);
+ /* back out the flags/counters */
+ spin_lock_irqsave(&ctrl->lock, flags);
+ if (ctrl->flags & FCCTRL_TERMIO)
+ ctrl->iocnt--;
+ op->flags &= ~FCOP_FLAGS_TERMIO;
+ spin_unlock_irqrestore(&ctrl->lock, flags);
+ return;
+ }
}
-/*
- * Called to teardown an association.
- * May be called with association fully in place or partially in place.
- */
-static void
-__nvme_fc_remove_ctrl(struct nvme_fc_ctrl *ctrl)
-{
- nvme_stop_keep_alive(&ctrl->ctrl);
- /* stop and terminate ios on admin and io queues */
- nvme_fc_shutdown_ctrl(ctrl);
-
- /*
- * tear down the controller
- * This will result in the last reference on the nvme ctrl to
- * expire, calling the transport nvme_fc_free_nvme_ctrl() callback.
- * From there, the transport will tear down it's logical queues and
- * association.
- */
- nvme_uninit_ctrl(&ctrl->ctrl);
-
- nvme_put_ctrl(&ctrl->ctrl);
-}
-
-static void
-nvme_fc_del_ctrl_work(struct work_struct *work)
-{
- struct nvme_fc_ctrl *ctrl =
- container_of(work, struct nvme_fc_ctrl, delete_work);
-
- __nvme_fc_remove_ctrl(ctrl);
-}
-
-static int
-__nvme_fc_del_ctrl(struct nvme_fc_ctrl *ctrl)
-{
- if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_DELETING))
- return -EBUSY;
-
- if (!queue_work(nvme_fc_wq, &ctrl->delete_work))
- return -EBUSY;
-
- return 0;
-}
-
-/*
- * Request from nvme core layer to delete the controller
- */
-static int
-nvme_fc_del_nvme_ctrl(struct nvme_ctrl *nctrl)
-{
- struct nvme_fc_ctrl *ctrl = to_fc_ctrl(nctrl);
- struct nvme_fc_rport *rport = ctrl->rport;
- unsigned long flags;
- int ret;
-
- spin_lock_irqsave(&rport->lock, flags);
- ret = __nvme_fc_del_ctrl(ctrl);
- spin_unlock_irqrestore(&rport->lock, flags);
- if (ret)
- return ret;
-
- flush_work(&ctrl->delete_work);
-
- return 0;
-}
-
-static int
-nvme_fc_reset_nvme_ctrl(struct nvme_ctrl *nctrl)
-{
- return -EIO;
-}
-
-static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = {
- .name = "fc",
- .module = THIS_MODULE,
- .is_fabrics = true,
- .reg_read32 = nvmf_reg_read32,
- .reg_read64 = nvmf_reg_read64,
- .reg_write32 = nvmf_reg_write32,
- .reset_ctrl = nvme_fc_reset_nvme_ctrl,
- .free_ctrl = nvme_fc_free_nvme_ctrl,
- .submit_async_event = nvme_fc_submit_async_event,
- .delete_ctrl = nvme_fc_del_nvme_ctrl,
- .get_subsysnqn = nvmf_get_subsysnqn,
- .get_address = nvmf_get_address,
+static const struct blk_mq_ops nvme_fc_mq_ops = {
+ .queue_rq = nvme_fc_queue_rq,
+ .complete = nvme_fc_complete_rq,
+ .init_request = nvme_fc_init_request,
+ .exit_request = nvme_fc_exit_request,
+ .reinit_request = nvme_fc_reinit_request,
+ .init_hctx = nvme_fc_init_hctx,
+ .poll = nvme_fc_poll,
+ .timeout = nvme_fc_timeout,
};
static int
@@ -2290,15 +2249,478 @@
return ret;
}
+static int
+nvme_fc_reinit_io_queues(struct nvme_fc_ctrl *ctrl)
+{
+ struct nvmf_ctrl_options *opts = ctrl->ctrl.opts;
+ int ret;
+
+ ret = nvme_set_queue_count(&ctrl->ctrl, &opts->nr_io_queues);
+ if (ret) {
+ dev_info(ctrl->ctrl.device,
+ "set_queue_count failed: %d\n", ret);
+ return ret;
+ }
+
+ /* check for io queues existing */
+ if (ctrl->queue_count == 1)
+ return 0;
+
+ dev_info(ctrl->ctrl.device, "Recreating %d I/O queues.\n",
+ opts->nr_io_queues);
+
+ nvme_fc_init_io_queues(ctrl);
+
+ ret = blk_mq_reinit_tagset(&ctrl->tag_set);
+ if (ret)
+ goto out_free_io_queues;
+
+ ret = nvme_fc_create_hw_io_queues(ctrl, ctrl->ctrl.opts->queue_size);
+ if (ret)
+ goto out_free_io_queues;
+
+ ret = nvme_fc_connect_io_queues(ctrl, ctrl->ctrl.opts->queue_size);
+ if (ret)
+ goto out_delete_hw_queues;
+
+ return 0;
+
+out_delete_hw_queues:
+ nvme_fc_delete_hw_io_queues(ctrl);
+out_free_io_queues:
+ nvme_fc_free_io_queues(ctrl);
+ return ret;
+}
+
+/*
+ * This routine restarts the controller on the host side, and
+ * on the link side, recreates the controller association.
+ */
+static int
+nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
+{
+ struct nvmf_ctrl_options *opts = ctrl->ctrl.opts;
+ u32 segs;
+ int ret;
+ bool changed;
+
+ ctrl->connect_attempts++;
+
+ /*
+ * Create the admin queue
+ */
+
+ nvme_fc_init_queue(ctrl, 0, NVME_FC_AQ_BLKMQ_DEPTH);
+
+ ret = __nvme_fc_create_hw_queue(ctrl, &ctrl->queues[0], 0,
+ NVME_FC_AQ_BLKMQ_DEPTH);
+ if (ret)
+ goto out_free_queue;
+
+ ret = nvme_fc_connect_admin_queue(ctrl, &ctrl->queues[0],
+ NVME_FC_AQ_BLKMQ_DEPTH,
+ (NVME_FC_AQ_BLKMQ_DEPTH / 4));
+ if (ret)
+ goto out_delete_hw_queue;
+
+ if (ctrl->ctrl.state != NVME_CTRL_NEW)
+ blk_mq_start_stopped_hw_queues(ctrl->ctrl.admin_q, true);
+
+ ret = nvmf_connect_admin_queue(&ctrl->ctrl);
+ if (ret)
+ goto out_disconnect_admin_queue;
+
+ /*
+ * Check controller capabilities
+ *
+ * todo:- add code to check if ctrl attributes changed from
+ * prior connection values
+ */
+
+ ret = nvmf_reg_read64(&ctrl->ctrl, NVME_REG_CAP, &ctrl->cap);
+ if (ret) {
+ dev_err(ctrl->ctrl.device,
+ "prop_get NVME_REG_CAP failed\n");
+ goto out_disconnect_admin_queue;
+ }
+
+ ctrl->ctrl.sqsize =
+ min_t(int, NVME_CAP_MQES(ctrl->cap) + 1, ctrl->ctrl.sqsize);
+
+ ret = nvme_enable_ctrl(&ctrl->ctrl, ctrl->cap);
+ if (ret)
+ goto out_disconnect_admin_queue;
+
+ segs = min_t(u32, NVME_FC_MAX_SEGMENTS,
+ ctrl->lport->ops->max_sgl_segments);
+ ctrl->ctrl.max_hw_sectors = (segs - 1) << (PAGE_SHIFT - 9);
+
+ ret = nvme_init_identify(&ctrl->ctrl);
+ if (ret)
+ goto out_disconnect_admin_queue;
+
+ /* sanity checks */
+
+ /* FC-NVME does not have other data in the capsule */
+ if (ctrl->ctrl.icdoff) {
+ dev_err(ctrl->ctrl.device, "icdoff %d is not supported!\n",
+ ctrl->ctrl.icdoff);
+ goto out_disconnect_admin_queue;
+ }
+
+ nvme_start_keep_alive(&ctrl->ctrl);
+
+ /* FC-NVME supports normal SGL Data Block Descriptors */
+
+ if (opts->queue_size > ctrl->ctrl.maxcmd) {
+ /* warn if maxcmd is lower than queue_size */
+ dev_warn(ctrl->ctrl.device,
+ "queue_size %zu > ctrl maxcmd %u, reducing "
+ "to queue_size\n",
+ opts->queue_size, ctrl->ctrl.maxcmd);
+ opts->queue_size = ctrl->ctrl.maxcmd;
+ }
+
+ ret = nvme_fc_init_aen_ops(ctrl);
+ if (ret)
+ goto out_term_aen_ops;
+
+ /*
+ * Create the io queues
+ */
+
+ if (ctrl->queue_count > 1) {
+ if (ctrl->ctrl.state == NVME_CTRL_NEW)
+ ret = nvme_fc_create_io_queues(ctrl);
+ else
+ ret = nvme_fc_reinit_io_queues(ctrl);
+ if (ret)
+ goto out_term_aen_ops;
+ }
+
+ changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
+ WARN_ON_ONCE(!changed);
+
+ ctrl->connect_attempts = 0;
+
+ kref_get(&ctrl->ctrl.kref);
+
+ if (ctrl->queue_count > 1) {
+ nvme_start_queues(&ctrl->ctrl);
+ nvme_queue_scan(&ctrl->ctrl);
+ nvme_queue_async_events(&ctrl->ctrl);
+ }
+
+ return 0; /* Success */
+
+out_term_aen_ops:
+ nvme_fc_term_aen_ops(ctrl);
+ nvme_stop_keep_alive(&ctrl->ctrl);
+out_disconnect_admin_queue:
+ /* send a Disconnect(association) LS to fc-nvme target */
+ nvme_fc_xmt_disconnect_assoc(ctrl);
+out_delete_hw_queue:
+ __nvme_fc_delete_hw_queue(ctrl, &ctrl->queues[0], 0);
+out_free_queue:
+ nvme_fc_free_queue(&ctrl->queues[0]);
+
+ return ret;
+}
+
+/*
+ * This routine stops operation of the controller on the host side.
+ * On the host os stack side: Admin and IO queues are stopped,
+ * outstanding ios on them terminated via FC ABTS.
+ * On the link side: the association is terminated.
+ */
+static void
+nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)
+{
+ unsigned long flags;
+
+ nvme_stop_keep_alive(&ctrl->ctrl);
+
+ spin_lock_irqsave(&ctrl->lock, flags);
+ ctrl->flags |= FCCTRL_TERMIO;
+ ctrl->iocnt = 0;
+ spin_unlock_irqrestore(&ctrl->lock, flags);
+
+ /*
+ * If io queues are present, stop them and terminate all outstanding
+ * ios on them. As FC allocates FC exchange for each io, the
+ * transport must contact the LLDD to terminate the exchange,
+ * thus releasing the FC exchange. We use blk_mq_tagset_busy_itr()
+ * to tell us what io's are busy and invoke a transport routine
+ * to kill them with the LLDD. After terminating the exchange
+ * the LLDD will call the transport's normal io done path, but it
+ * will have an aborted status. The done path will return the
+ * io requests back to the block layer as part of normal completions
+ * (but with error status).
+ */
+ if (ctrl->queue_count > 1) {
+ nvme_stop_queues(&ctrl->ctrl);
+ blk_mq_tagset_busy_iter(&ctrl->tag_set,
+ nvme_fc_terminate_exchange, &ctrl->ctrl);
+ }
+
+ /*
+ * Other transports, which don't have link-level contexts bound
+ * to sqe's, would try to gracefully shutdown the controller by
+ * writing the registers for shutdown and polling (call
+ * nvme_shutdown_ctrl()). Given a bunch of i/o was potentially
+ * just aborted and we will wait on those contexts, and given
+ * there was no indication of how live the controlelr is on the
+ * link, don't send more io to create more contexts for the
+ * shutdown. Let the controller fail via keepalive failure if
+ * its still present.
+ */
+
+ /*
+ * clean up the admin queue. Same thing as above.
+ * use blk_mq_tagset_busy_itr() and the transport routine to
+ * terminate the exchanges.
+ */
+ blk_mq_stop_hw_queues(ctrl->ctrl.admin_q);
+ blk_mq_tagset_busy_iter(&ctrl->admin_tag_set,
+ nvme_fc_terminate_exchange, &ctrl->ctrl);
+
+ /* kill the aens as they are a separate path */
+ nvme_fc_abort_aen_ops(ctrl);
+
+ /* wait for all io that had to be aborted */
+ spin_lock_irqsave(&ctrl->lock, flags);
+ while (ctrl->iocnt) {
+ spin_unlock_irqrestore(&ctrl->lock, flags);
+ msleep(1000);
+ spin_lock_irqsave(&ctrl->lock, flags);
+ }
+ ctrl->flags &= ~FCCTRL_TERMIO;
+ spin_unlock_irqrestore(&ctrl->lock, flags);
+
+ nvme_fc_term_aen_ops(ctrl);
+
+ /*
+ * send a Disconnect(association) LS to fc-nvme target
+ * Note: could have been sent at top of process, but
+ * cleaner on link traffic if after the aborts complete.
+ * Note: if association doesn't exist, association_id will be 0
+ */
+ if (ctrl->association_id)
+ nvme_fc_xmt_disconnect_assoc(ctrl);
+
+ if (ctrl->ctrl.tagset) {
+ nvme_fc_delete_hw_io_queues(ctrl);
+ nvme_fc_free_io_queues(ctrl);
+ }
+
+ __nvme_fc_delete_hw_queue(ctrl, &ctrl->queues[0], 0);
+ nvme_fc_free_queue(&ctrl->queues[0]);
+}
+
+static void
+nvme_fc_delete_ctrl_work(struct work_struct *work)
+{
+ struct nvme_fc_ctrl *ctrl =
+ container_of(work, struct nvme_fc_ctrl, delete_work);
+
+ cancel_work_sync(&ctrl->reset_work);
+ cancel_delayed_work_sync(&ctrl->connect_work);
+
+ /*
+ * kill the association on the link side. this will block
+ * waiting for io to terminate
+ */
+ nvme_fc_delete_association(ctrl);
+
+ /*
+ * tear down the controller
+ * This will result in the last reference on the nvme ctrl to
+ * expire, calling the transport nvme_fc_nvme_ctrl_freed() callback.
+ * From there, the transport will tear down it's logical queues and
+ * association.
+ */
+ nvme_uninit_ctrl(&ctrl->ctrl);
+
+ nvme_put_ctrl(&ctrl->ctrl);
+}
+
+static int
+__nvme_fc_del_ctrl(struct nvme_fc_ctrl *ctrl)
+{
+ if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_DELETING))
+ return -EBUSY;
+
+ if (!queue_work(nvme_fc_wq, &ctrl->delete_work))
+ return -EBUSY;
+
+ return 0;
+}
+
+/*
+ * Request from nvme core layer to delete the controller
+ */
+static int
+nvme_fc_del_nvme_ctrl(struct nvme_ctrl *nctrl)
+{
+ struct nvme_fc_ctrl *ctrl = to_fc_ctrl(nctrl);
+ int ret;
+
+ if (!kref_get_unless_zero(&ctrl->ctrl.kref))
+ return -EBUSY;
+
+ ret = __nvme_fc_del_ctrl(ctrl);
+
+ if (!ret)
+ flush_workqueue(nvme_fc_wq);
+
+ nvme_put_ctrl(&ctrl->ctrl);
+
+ return ret;
+}
+
+static void
+nvme_fc_reset_ctrl_work(struct work_struct *work)
+{
+ struct nvme_fc_ctrl *ctrl =
+ container_of(work, struct nvme_fc_ctrl, reset_work);
+ int ret;
+
+ /* will block will waiting for io to terminate */
+ nvme_fc_delete_association(ctrl);
+
+ ret = nvme_fc_create_association(ctrl);
+ if (ret) {
+ dev_warn(ctrl->ctrl.device,
+ "NVME-FC{%d}: reset: Reconnect attempt failed (%d)\n",
+ ctrl->cnum, ret);
+ if (ctrl->connect_attempts >= NVME_FC_MAX_CONNECT_ATTEMPTS) {
+ dev_warn(ctrl->ctrl.device,
+ "NVME-FC{%d}: Max reconnect attempts (%d) "
+ "reached. Removing controller\n",
+ ctrl->cnum, ctrl->connect_attempts);
+
+ if (!nvme_change_ctrl_state(&ctrl->ctrl,
+ NVME_CTRL_DELETING)) {
+ dev_err(ctrl->ctrl.device,
+ "NVME-FC{%d}: failed to change state "
+ "to DELETING\n", ctrl->cnum);
+ return;
+ }
+
+ WARN_ON(!queue_work(nvme_fc_wq, &ctrl->delete_work));
+ return;
+ }
+
+ dev_warn(ctrl->ctrl.device,
+ "NVME-FC{%d}: Reconnect attempt in %d seconds.\n",
+ ctrl->cnum, ctrl->reconnect_delay);
+ queue_delayed_work(nvme_fc_wq, &ctrl->connect_work,
+ ctrl->reconnect_delay * HZ);
+ } else
+ dev_info(ctrl->ctrl.device,
+ "NVME-FC{%d}: controller reset complete\n", ctrl->cnum);
+}
+
+/*
+ * called by the nvme core layer, for sysfs interface that requests
+ * a reset of the nvme controller
+ */
+static int
+nvme_fc_reset_nvme_ctrl(struct nvme_ctrl *nctrl)
+{
+ struct nvme_fc_ctrl *ctrl = to_fc_ctrl(nctrl);
+
+ dev_warn(ctrl->ctrl.device,
+ "NVME-FC{%d}: admin requested controller reset\n", ctrl->cnum);
+
+ if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RESETTING))
+ return -EBUSY;
+
+ if (!queue_work(nvme_fc_wq, &ctrl->reset_work))
+ return -EBUSY;
+
+ flush_work(&ctrl->reset_work);
+
+ return 0;
+}
+
+static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = {
+ .name = "fc",
+ .module = THIS_MODULE,
+ .is_fabrics = true,
+ .reg_read32 = nvmf_reg_read32,
+ .reg_read64 = nvmf_reg_read64,
+ .reg_write32 = nvmf_reg_write32,
+ .reset_ctrl = nvme_fc_reset_nvme_ctrl,
+ .free_ctrl = nvme_fc_nvme_ctrl_freed,
+ .submit_async_event = nvme_fc_submit_async_event,
+ .delete_ctrl = nvme_fc_del_nvme_ctrl,
+ .get_subsysnqn = nvmf_get_subsysnqn,
+ .get_address = nvmf_get_address,
+};
+
+static void
+nvme_fc_connect_ctrl_work(struct work_struct *work)
+{
+ int ret;
+
+ struct nvme_fc_ctrl *ctrl =
+ container_of(to_delayed_work(work),
+ struct nvme_fc_ctrl, connect_work);
+
+ ret = nvme_fc_create_association(ctrl);
+ if (ret) {
+ dev_warn(ctrl->ctrl.device,
+ "NVME-FC{%d}: Reconnect attempt failed (%d)\n",
+ ctrl->cnum, ret);
+ if (ctrl->connect_attempts >= NVME_FC_MAX_CONNECT_ATTEMPTS) {
+ dev_warn(ctrl->ctrl.device,
+ "NVME-FC{%d}: Max reconnect attempts (%d) "
+ "reached. Removing controller\n",
+ ctrl->cnum, ctrl->connect_attempts);
+
+ if (!nvme_change_ctrl_state(&ctrl->ctrl,
+ NVME_CTRL_DELETING)) {
+ dev_err(ctrl->ctrl.device,
+ "NVME-FC{%d}: failed to change state "
+ "to DELETING\n", ctrl->cnum);
+ return;
+ }
+
+ WARN_ON(!queue_work(nvme_fc_wq, &ctrl->delete_work));
+ return;
+ }
+
+ dev_warn(ctrl->ctrl.device,
+ "NVME-FC{%d}: Reconnect attempt in %d seconds.\n",
+ ctrl->cnum, ctrl->reconnect_delay);
+ queue_delayed_work(nvme_fc_wq, &ctrl->connect_work,
+ ctrl->reconnect_delay * HZ);
+ } else
+ dev_info(ctrl->ctrl.device,
+ "NVME-FC{%d}: controller reconnect complete\n",
+ ctrl->cnum);
+}
+
+
+static const struct blk_mq_ops nvme_fc_admin_mq_ops = {
+ .queue_rq = nvme_fc_queue_rq,
+ .complete = nvme_fc_complete_rq,
+ .init_request = nvme_fc_init_admin_request,
+ .exit_request = nvme_fc_exit_request,
+ .reinit_request = nvme_fc_reinit_request,
+ .init_hctx = nvme_fc_init_admin_hctx,
+ .timeout = nvme_fc_timeout,
+};
+
static struct nvme_ctrl *
-__nvme_fc_create_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
+nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,
struct nvme_fc_lport *lport, struct nvme_fc_rport *rport)
{
struct nvme_fc_ctrl *ctrl;
unsigned long flags;
int ret, idx;
- bool changed;
ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
if (!ctrl) {
@@ -2314,21 +2736,18 @@
ctrl->ctrl.opts = opts;
INIT_LIST_HEAD(&ctrl->ctrl_list);
- INIT_LIST_HEAD(&ctrl->ls_req_list);
ctrl->lport = lport;
ctrl->rport = rport;
ctrl->dev = lport->dev;
- ctrl->state = FCCTRL_INIT;
ctrl->cnum = idx;
- ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_fc_ctrl_ops, 0);
- if (ret)
- goto out_free_ida;
-
get_device(ctrl->dev);
kref_init(&ctrl->ref);
- INIT_WORK(&ctrl->delete_work, nvme_fc_del_ctrl_work);
+ INIT_WORK(&ctrl->delete_work, nvme_fc_delete_ctrl_work);
+ INIT_WORK(&ctrl->reset_work, nvme_fc_reset_ctrl_work);
+ INIT_DELAYED_WORK(&ctrl->connect_work, nvme_fc_connect_ctrl_work);
+ ctrl->reconnect_delay = opts->reconnect_delay;
spin_lock_init(&ctrl->lock);
/* io queue count */
@@ -2345,87 +2764,87 @@
ctrl->queues = kcalloc(ctrl->queue_count, sizeof(struct nvme_fc_queue),
GFP_KERNEL);
if (!ctrl->queues)
- goto out_uninit_ctrl;
+ goto out_free_ida;
- ret = nvme_fc_configure_admin_queue(ctrl);
+ memset(&ctrl->admin_tag_set, 0, sizeof(ctrl->admin_tag_set));
+ ctrl->admin_tag_set.ops = &nvme_fc_admin_mq_ops;
+ ctrl->admin_tag_set.queue_depth = NVME_FC_AQ_BLKMQ_DEPTH;
+ ctrl->admin_tag_set.reserved_tags = 2; /* fabric connect + Keep-Alive */
+ ctrl->admin_tag_set.numa_node = NUMA_NO_NODE;
+ ctrl->admin_tag_set.cmd_size = sizeof(struct nvme_fc_fcp_op) +
+ (SG_CHUNK_SIZE *
+ sizeof(struct scatterlist)) +
+ ctrl->lport->ops->fcprqst_priv_sz;
+ ctrl->admin_tag_set.driver_data = ctrl;
+ ctrl->admin_tag_set.nr_hw_queues = 1;
+ ctrl->admin_tag_set.timeout = ADMIN_TIMEOUT;
+
+ ret = blk_mq_alloc_tag_set(&ctrl->admin_tag_set);
if (ret)
- goto out_uninit_ctrl;
+ goto out_free_queues;
- /* sanity checks */
-
- /* FC-NVME does not have other data in the capsule */
- if (ctrl->ctrl.icdoff) {
- dev_err(ctrl->ctrl.device, "icdoff %d is not supported!\n",
- ctrl->ctrl.icdoff);
- goto out_remove_admin_queue;
+ ctrl->ctrl.admin_q = blk_mq_init_queue(&ctrl->admin_tag_set);
+ if (IS_ERR(ctrl->ctrl.admin_q)) {
+ ret = PTR_ERR(ctrl->ctrl.admin_q);
+ goto out_free_admin_tag_set;
}
- /* FC-NVME supports normal SGL Data Block Descriptors */
+ /*
+ * Would have been nice to init io queues tag set as well.
+ * However, we require interaction from the controller
+ * for max io queue count before we can do so.
+ * Defer this to the connect path.
+ */
- if (opts->queue_size > ctrl->ctrl.maxcmd) {
- /* warn if maxcmd is lower than queue_size */
- dev_warn(ctrl->ctrl.device,
- "queue_size %zu > ctrl maxcmd %u, reducing "
- "to queue_size\n",
- opts->queue_size, ctrl->ctrl.maxcmd);
- opts->queue_size = ctrl->ctrl.maxcmd;
- }
-
- ret = nvme_fc_init_aen_ops(ctrl);
+ ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_fc_ctrl_ops, 0);
if (ret)
- goto out_exit_aen_ops;
+ goto out_cleanup_admin_q;
- if (ctrl->queue_count > 1) {
- ret = nvme_fc_create_io_queues(ctrl);
- if (ret)
- goto out_exit_aen_ops;
- }
-
- spin_lock_irqsave(&ctrl->lock, flags);
- ctrl->state = FCCTRL_ACTIVE;
- spin_unlock_irqrestore(&ctrl->lock, flags);
-
- changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
- WARN_ON_ONCE(!changed);
-
- dev_info(ctrl->ctrl.device,
- "NVME-FC{%d}: new ctrl: NQN \"%s\"\n",
- ctrl->cnum, ctrl->ctrl.opts->subsysnqn);
-
- kref_get(&ctrl->ctrl.kref);
+ /* at this point, teardown path changes to ref counting on nvme ctrl */
spin_lock_irqsave(&rport->lock, flags);
list_add_tail(&ctrl->ctrl_list, &rport->ctrl_list);
spin_unlock_irqrestore(&rport->lock, flags);
- if (opts->nr_io_queues) {
- nvme_queue_scan(&ctrl->ctrl);
- nvme_queue_async_events(&ctrl->ctrl);
+ ret = nvme_fc_create_association(ctrl);
+ if (ret) {
+ ctrl->ctrl.opts = NULL;
+ /* initiate nvme ctrl ref counting teardown */
+ nvme_uninit_ctrl(&ctrl->ctrl);
+ nvme_put_ctrl(&ctrl->ctrl);
+
+ /* as we're past the point where we transition to the ref
+ * counting teardown path, if we return a bad pointer here,
+ * the calling routine, thinking it's prior to the
+ * transition, will do an rport put. Since the teardown
+ * path also does a rport put, we do an extra get here to
+ * so proper order/teardown happens.
+ */
+ nvme_fc_rport_get(rport);
+
+ if (ret > 0)
+ ret = -EIO;
+ return ERR_PTR(ret);
}
+ dev_info(ctrl->ctrl.device,
+ "NVME-FC{%d}: new ctrl: NQN \"%s\"\n",
+ ctrl->cnum, ctrl->ctrl.opts->subsysnqn);
+
return &ctrl->ctrl;
-out_exit_aen_ops:
- nvme_fc_exit_aen_ops(ctrl);
-out_remove_admin_queue:
- /* send a Disconnect(association) LS to fc-nvme target */
- nvme_fc_xmt_disconnect_assoc(ctrl);
- nvme_stop_keep_alive(&ctrl->ctrl);
- nvme_fc_destroy_admin_queue(ctrl);
-out_uninit_ctrl:
- nvme_uninit_ctrl(&ctrl->ctrl);
- nvme_put_ctrl(&ctrl->ctrl);
- if (ret > 0)
- ret = -EIO;
- /* exit via here will follow ctlr ref point callbacks to free */
- return ERR_PTR(ret);
-
+out_cleanup_admin_q:
+ blk_cleanup_queue(ctrl->ctrl.admin_q);
+out_free_admin_tag_set:
+ blk_mq_free_tag_set(&ctrl->admin_tag_set);
+out_free_queues:
+ kfree(ctrl->queues);
out_free_ida:
+ put_device(ctrl->dev);
ida_simple_remove(&nvme_fc_ctrl_cnt, ctrl->cnum);
out_free_ctrl:
kfree(ctrl);
out_fail:
- nvme_fc_rport_put(rport);
/* exit via here doesn't follow ctlr ref points */
return ERR_PTR(ret);
}
@@ -2497,6 +2916,7 @@
{
struct nvme_fc_lport *lport;
struct nvme_fc_rport *rport;
+ struct nvme_ctrl *ctrl;
struct nvmet_fc_traddr laddr = { 0L, 0L };
struct nvmet_fc_traddr raddr = { 0L, 0L };
unsigned long flags;
@@ -2528,7 +2948,10 @@
spin_unlock_irqrestore(&nvme_fc_lock, flags);
- return __nvme_fc_create_ctrl(dev, opts, lport, rport);
+ ctrl = nvme_fc_init_ctrl(dev, opts, lport, rport);
+ if (IS_ERR(ctrl))
+ nvme_fc_rport_put(rport);
+ return ctrl;
}
}
spin_unlock_irqrestore(&nvme_fc_lock, flags);
@@ -2546,11 +2969,20 @@
static int __init nvme_fc_init_module(void)
{
+ int ret;
+
nvme_fc_wq = create_workqueue("nvme_fc_wq");
if (!nvme_fc_wq)
return -ENOMEM;
- return nvmf_register_transport(&nvme_fc_transport);
+ ret = nvmf_register_transport(&nvme_fc_transport);
+ if (ret)
+ goto err;
+
+ return 0;
+err:
+ destroy_workqueue(nvme_fc_wq);
+ return ret;
}
static void __exit nvme_fc_exit_module(void)
diff --git a/drivers/nvme/host/lightnvm.c b/drivers/nvme/host/lightnvm.c
index 21cac85..e4e4e60 100644
--- a/drivers/nvme/host/lightnvm.c
+++ b/drivers/nvme/host/lightnvm.c
@@ -241,9 +241,9 @@
BUILD_BUG_ON(sizeof(struct nvme_nvm_l2ptbl) != 64);
BUILD_BUG_ON(sizeof(struct nvme_nvm_erase_blk) != 64);
BUILD_BUG_ON(sizeof(struct nvme_nvm_id_group) != 960);
- BUILD_BUG_ON(sizeof(struct nvme_nvm_addr_format) != 128);
+ BUILD_BUG_ON(sizeof(struct nvme_nvm_addr_format) != 16);
BUILD_BUG_ON(sizeof(struct nvme_nvm_id) != 4096);
- BUILD_BUG_ON(sizeof(struct nvme_nvm_bb_tbl) != 512);
+ BUILD_BUG_ON(sizeof(struct nvme_nvm_bb_tbl) != 64);
}
static int init_grps(struct nvm_id *nvm_id, struct nvme_nvm_id *nvme_nvm_id)
@@ -324,7 +324,7 @@
nvm_id->cap = le32_to_cpu(nvme_nvm_id->cap);
nvm_id->dom = le32_to_cpu(nvme_nvm_id->dom);
memcpy(&nvm_id->ppaf, &nvme_nvm_id->ppaf,
- sizeof(struct nvme_nvm_addr_format));
+ sizeof(struct nvm_addr_format));
ret = init_grps(nvm_id, nvme_nvm_id);
out:
@@ -483,8 +483,8 @@
{
struct nvm_rq *rqd = rq->end_io_data;
- rqd->ppa_status = nvme_req(rq)->result.u64;
- rqd->error = error;
+ rqd->ppa_status = le64_to_cpu(nvme_req(rq)->result.u64);
+ rqd->error = nvme_req(rq)->status;
nvm_end_io(rqd);
kfree(nvme_req(rq)->cmd);
@@ -510,12 +510,12 @@
}
rq->cmd_flags &= ~REQ_FAILFAST_DRIVER;
- rq->ioprio = bio_prio(bio);
- if (bio_has_data(bio))
- rq->nr_phys_segments = bio_phys_segments(q, bio);
-
- rq->__data_len = bio->bi_iter.bi_size;
- rq->bio = rq->biotail = bio;
+ if (bio) {
+ blk_init_request_from_bio(rq, bio);
+ } else {
+ rq->ioprio = IOPRIO_PRIO_VALUE(IOPRIO_CLASS_BE, IOPRIO_NORM);
+ rq->__data_len = 0;
+ }
nvme_nvm_rqtocmd(rq, rqd, ns, cmd);
@@ -526,21 +526,6 @@
return 0;
}
-static int nvme_nvm_erase_block(struct nvm_dev *dev, struct nvm_rq *rqd)
-{
- struct request_queue *q = dev->q;
- struct nvme_ns *ns = q->queuedata;
- struct nvme_nvm_command c = {};
-
- c.erase.opcode = NVM_OP_ERASE;
- c.erase.nsid = cpu_to_le32(ns->ns_id);
- c.erase.spba = cpu_to_le64(rqd->ppa_addr.ppa);
- c.erase.length = cpu_to_le16(rqd->nr_ppas - 1);
- c.erase.control = cpu_to_le16(rqd->flags);
-
- return nvme_submit_sync_cmd(q, (struct nvme_command *)&c, NULL, 0);
-}
-
static void *nvme_nvm_create_dma_pool(struct nvm_dev *nvmdev, char *name)
{
struct nvme_ns *ns = nvmdev->q->queuedata;
@@ -576,7 +561,6 @@
.set_bb_tbl = nvme_nvm_set_bb_tbl,
.submit_io = nvme_nvm_submit_io,
- .erase_block = nvme_nvm_erase_block,
.create_dma_pool = nvme_nvm_create_dma_pool,
.destroy_dma_pool = nvme_nvm_destroy_dma_pool,
@@ -611,7 +595,7 @@
__le64 *metadata = NULL;
dma_addr_t metadata_dma;
DECLARE_COMPLETION_ONSTACK(wait);
- int ret;
+ int ret = 0;
rq = nvme_alloc_request(q, (struct nvme_command *)vcmd, 0,
NVME_QID_ANY);
@@ -681,9 +665,12 @@
wait_for_completion_io(&wait);
- ret = nvme_error_status(rq->errors);
+ if (nvme_req(rq)->flags & NVME_REQ_CANCELLED)
+ ret = -EINTR;
+ else if (nvme_req(rq)->status & 0x7ff)
+ ret = -EIO;
if (result)
- *result = rq->errors & 0x7ff;
+ *result = nvme_req(rq)->status & 0x7ff;
if (status)
*status = le64_to_cpu(nvme_req(rq)->result.u64);
@@ -766,7 +753,7 @@
c.common.cdw2[1] = cpu_to_le32(vcmd.cdw3);
/* cdw11-12 */
c.ph_rw.length = cpu_to_le16(vcmd.nppas);
- c.ph_rw.control = cpu_to_le32(vcmd.control);
+ c.ph_rw.control = cpu_to_le16(vcmd.control);
c.common.cdw10[3] = cpu_to_le32(vcmd.cdw13);
c.common.cdw10[4] = cpu_to_le32(vcmd.cdw14);
c.common.cdw10[5] = cpu_to_le32(vcmd.cdw15);
@@ -809,6 +796,8 @@
struct request_queue *q = ns->queue;
struct nvm_dev *dev;
+ _nvme_nvm_check_size();
+
dev = nvm_alloc_dev(node);
if (!dev)
return -ENOMEM;
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index ab2d6ec..29c708c 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -21,16 +21,6 @@
#include <linux/lightnvm.h>
#include <linux/sed-opal.h>
-enum {
- /*
- * Driver internal status code for commands that were cancelled due
- * to timeouts or controller shutdown. The value is negative so
- * that it a) doesn't overlap with the unsigned hardware error codes,
- * and b) can easily be tested for.
- */
- NVME_SC_CANCELLED = -EINTR,
-};
-
extern unsigned char nvme_io_timeout;
#define NVME_IO_TIMEOUT (nvme_io_timeout * HZ)
@@ -43,8 +33,6 @@
#define NVME_DEFAULT_KATO 5
#define NVME_KATO_GRACE 10
-extern unsigned int nvme_max_retries;
-
enum {
NVME_NS_LBA = 0,
NVME_NS_LIGHTNVM = 1,
@@ -68,10 +56,10 @@
NVME_QUIRK_IDENTIFY_CNS = (1 << 1),
/*
- * The controller deterministically returns O's on reads to discarded
- * logical blocks.
+ * The controller deterministically returns O's on reads to
+ * logical blocks that deallocate was called on.
*/
- NVME_QUIRK_DISCARD_ZEROES = (1 << 2),
+ NVME_QUIRK_DEALLOCATE_ZEROES = (1 << 2),
/*
* The controller needs a delay before starts checking the device
@@ -97,6 +85,13 @@
struct nvme_request {
struct nvme_command *cmd;
union nvme_result result;
+ u8 retries;
+ u8 flags;
+ u16 status;
+};
+
+enum {
+ NVME_REQ_CANCELLED = (1 << 0),
};
static inline struct nvme_request *nvme_req(struct request *req)
@@ -254,25 +249,17 @@
}
}
-static inline int nvme_error_status(u16 status)
+static inline void nvme_end_request(struct request *req, __le16 status,
+ union nvme_result result)
{
- switch (status & 0x7ff) {
- case NVME_SC_SUCCESS:
- return 0;
- case NVME_SC_CAP_EXCEEDED:
- return -ENOSPC;
- default:
- return -EIO;
- }
+ struct nvme_request *rq = nvme_req(req);
+
+ rq->status = le16_to_cpu(status) >> 1;
+ rq->result = result;
+ blk_mq_complete_request(req);
}
-static inline bool nvme_req_needs_retry(struct request *req, u16 status)
-{
- return !(status & NVME_SC_DNR || blk_noretry_request(req)) &&
- (jiffies - req->start_time) < req->timeout &&
- req->retries < nvme_max_retries;
-}
-
+void nvme_complete_rq(struct request *req);
void nvme_cancel_request(struct request *req, void *data, bool reserved);
bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,
enum nvme_ctrl_state new_state);
@@ -307,7 +294,6 @@
#define NVME_QID_ANY -1
struct request *nvme_alloc_request(struct request_queue *q,
struct nvme_command *cmd, unsigned int flags, int qid);
-void nvme_requeue_req(struct request *req);
int nvme_setup_cmd(struct nvme_ns *ns, struct request *req,
struct nvme_command *cmd);
int nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 5d30953..c8541c3 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -104,8 +104,22 @@
u32 cmbloc;
struct nvme_ctrl ctrl;
struct completion ioq_wait;
+ u32 *dbbuf_dbs;
+ dma_addr_t dbbuf_dbs_dma_addr;
+ u32 *dbbuf_eis;
+ dma_addr_t dbbuf_eis_dma_addr;
};
+static inline unsigned int sq_idx(unsigned int qid, u32 stride)
+{
+ return qid * 2 * stride;
+}
+
+static inline unsigned int cq_idx(unsigned int qid, u32 stride)
+{
+ return (qid * 2 + 1) * stride;
+}
+
static inline struct nvme_dev *to_nvme_dev(struct nvme_ctrl *ctrl)
{
return container_of(ctrl, struct nvme_dev, ctrl);
@@ -134,6 +148,10 @@
u16 qid;
u8 cq_phase;
u8 cqe_seen;
+ u32 *dbbuf_sq_db;
+ u32 *dbbuf_cq_db;
+ u32 *dbbuf_sq_ei;
+ u32 *dbbuf_cq_ei;
};
/*
@@ -172,6 +190,112 @@
BUILD_BUG_ON(sizeof(struct nvme_id_ns) != 4096);
BUILD_BUG_ON(sizeof(struct nvme_lba_range_type) != 64);
BUILD_BUG_ON(sizeof(struct nvme_smart_log) != 512);
+ BUILD_BUG_ON(sizeof(struct nvme_dbbuf) != 64);
+}
+
+static inline unsigned int nvme_dbbuf_size(u32 stride)
+{
+ return ((num_possible_cpus() + 1) * 8 * stride);
+}
+
+static int nvme_dbbuf_dma_alloc(struct nvme_dev *dev)
+{
+ unsigned int mem_size = nvme_dbbuf_size(dev->db_stride);
+
+ if (dev->dbbuf_dbs)
+ return 0;
+
+ dev->dbbuf_dbs = dma_alloc_coherent(dev->dev, mem_size,
+ &dev->dbbuf_dbs_dma_addr,
+ GFP_KERNEL);
+ if (!dev->dbbuf_dbs)
+ return -ENOMEM;
+ dev->dbbuf_eis = dma_alloc_coherent(dev->dev, mem_size,
+ &dev->dbbuf_eis_dma_addr,
+ GFP_KERNEL);
+ if (!dev->dbbuf_eis) {
+ dma_free_coherent(dev->dev, mem_size,
+ dev->dbbuf_dbs, dev->dbbuf_dbs_dma_addr);
+ dev->dbbuf_dbs = NULL;
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+static void nvme_dbbuf_dma_free(struct nvme_dev *dev)
+{
+ unsigned int mem_size = nvme_dbbuf_size(dev->db_stride);
+
+ if (dev->dbbuf_dbs) {
+ dma_free_coherent(dev->dev, mem_size,
+ dev->dbbuf_dbs, dev->dbbuf_dbs_dma_addr);
+ dev->dbbuf_dbs = NULL;
+ }
+ if (dev->dbbuf_eis) {
+ dma_free_coherent(dev->dev, mem_size,
+ dev->dbbuf_eis, dev->dbbuf_eis_dma_addr);
+ dev->dbbuf_eis = NULL;
+ }
+}
+
+static void nvme_dbbuf_init(struct nvme_dev *dev,
+ struct nvme_queue *nvmeq, int qid)
+{
+ if (!dev->dbbuf_dbs || !qid)
+ return;
+
+ nvmeq->dbbuf_sq_db = &dev->dbbuf_dbs[sq_idx(qid, dev->db_stride)];
+ nvmeq->dbbuf_cq_db = &dev->dbbuf_dbs[cq_idx(qid, dev->db_stride)];
+ nvmeq->dbbuf_sq_ei = &dev->dbbuf_eis[sq_idx(qid, dev->db_stride)];
+ nvmeq->dbbuf_cq_ei = &dev->dbbuf_eis[cq_idx(qid, dev->db_stride)];
+}
+
+static void nvme_dbbuf_set(struct nvme_dev *dev)
+{
+ struct nvme_command c;
+
+ if (!dev->dbbuf_dbs)
+ return;
+
+ memset(&c, 0, sizeof(c));
+ c.dbbuf.opcode = nvme_admin_dbbuf;
+ c.dbbuf.prp1 = cpu_to_le64(dev->dbbuf_dbs_dma_addr);
+ c.dbbuf.prp2 = cpu_to_le64(dev->dbbuf_eis_dma_addr);
+
+ if (nvme_submit_sync_cmd(dev->ctrl.admin_q, &c, NULL, 0)) {
+ dev_warn(dev->dev, "unable to set dbbuf\n");
+ /* Free memory and continue on */
+ nvme_dbbuf_dma_free(dev);
+ }
+}
+
+static inline int nvme_dbbuf_need_event(u16 event_idx, u16 new_idx, u16 old)
+{
+ return (u16)(new_idx - event_idx - 1) < (u16)(new_idx - old);
+}
+
+/* Update dbbuf and return true if an MMIO is required */
+static bool nvme_dbbuf_update_and_check_event(u16 value, u32 *dbbuf_db,
+ volatile u32 *dbbuf_ei)
+{
+ if (dbbuf_db) {
+ u16 old_value;
+
+ /*
+ * Ensure that the queue is written before updating
+ * the doorbell in memory
+ */
+ wmb();
+
+ old_value = *dbbuf_db;
+ *dbbuf_db = value;
+
+ if (!nvme_dbbuf_need_event(*dbbuf_ei, value, old_value))
+ return false;
+ }
+
+ return true;
}
/*
@@ -298,7 +422,9 @@
if (++tail == nvmeq->q_depth)
tail = 0;
- writel(tail, nvmeq->q_db);
+ if (nvme_dbbuf_update_and_check_event(tail, nvmeq->dbbuf_sq_db,
+ nvmeq->dbbuf_sq_ei))
+ writel(tail, nvmeq->q_db);
nvmeq->sq_tail = tail;
}
@@ -327,10 +453,6 @@
iod->nents = 0;
iod->length = size;
- if (!(rq->rq_flags & RQF_DONTPREP)) {
- rq->retries = 0;
- rq->rq_flags |= RQF_DONTPREP;
- }
return BLK_MQ_RQ_QUEUE_OK;
}
@@ -629,34 +751,12 @@
return ret;
}
-static void nvme_complete_rq(struct request *req)
+static void nvme_pci_complete_rq(struct request *req)
{
struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
- struct nvme_dev *dev = iod->nvmeq->dev;
- int error = 0;
- nvme_unmap_data(dev, req);
-
- if (unlikely(req->errors)) {
- if (nvme_req_needs_retry(req, req->errors)) {
- req->retries++;
- nvme_requeue_req(req);
- return;
- }
-
- if (blk_rq_is_passthrough(req))
- error = req->errors;
- else
- error = nvme_error_status(req->errors);
- }
-
- if (unlikely(iod->aborted)) {
- dev_warn(dev->ctrl.device,
- "completing aborted command with status: %04x\n",
- req->errors);
- }
-
- blk_mq_end_request(req, error);
+ nvme_unmap_data(iod->nvmeq->dev, req);
+ nvme_complete_rq(req);
}
/* We read the CQE phase first to check if the rest of the entry is valid */
@@ -706,15 +806,16 @@
}
req = blk_mq_tag_to_rq(*nvmeq->tags, cqe.command_id);
- nvme_req(req)->result = cqe.result;
- blk_mq_complete_request(req, le16_to_cpu(cqe.status) >> 1);
+ nvme_end_request(req, cqe.status, cqe.result);
}
if (head == nvmeq->cq_head && phase == nvmeq->cq_phase)
return;
if (likely(nvmeq->cq_vector >= 0))
- writel(head, nvmeq->q_db + nvmeq->dev->db_stride);
+ if (nvme_dbbuf_update_and_check_event(head, nvmeq->dbbuf_cq_db,
+ nvmeq->dbbuf_cq_ei))
+ writel(head, nvmeq->q_db + nvmeq->dev->db_stride);
nvmeq->cq_head = head;
nvmeq->cq_phase = phase;
@@ -746,10 +847,8 @@
return IRQ_NONE;
}
-static int nvme_poll(struct blk_mq_hw_ctx *hctx, unsigned int tag)
+static int __nvme_poll(struct nvme_queue *nvmeq, unsigned int tag)
{
- struct nvme_queue *nvmeq = hctx->driver_data;
-
if (nvme_cqe_valid(nvmeq, nvmeq->cq_head, nvmeq->cq_phase)) {
spin_lock_irq(&nvmeq->q_lock);
__nvme_process_cq(nvmeq, &tag);
@@ -762,6 +861,13 @@
return 0;
}
+static int nvme_poll(struct blk_mq_hw_ctx *hctx, unsigned int tag)
+{
+ struct nvme_queue *nvmeq = hctx->driver_data;
+
+ return __nvme_poll(nvmeq, tag);
+}
+
static void nvme_pci_submit_async_event(struct nvme_ctrl *ctrl, int aer_idx)
{
struct nvme_dev *dev = to_nvme_dev(ctrl);
@@ -813,7 +919,7 @@
struct nvme_queue *nvmeq)
{
struct nvme_command c;
- int flags = NVME_QUEUE_PHYS_CONTIG | NVME_SQ_PRIO_MEDIUM;
+ int flags = NVME_QUEUE_PHYS_CONTIG;
/*
* Note: we (ab)use the fact the the prp fields survive if no data
@@ -844,9 +950,9 @@
{
struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
struct nvme_queue *nvmeq = iod->nvmeq;
- u16 status = req->errors;
- dev_warn(nvmeq->dev->ctrl.device, "Abort status: 0x%x", status);
+ dev_warn(nvmeq->dev->ctrl.device,
+ "Abort status: 0x%x", nvme_req(req)->status);
atomic_inc(&nvmeq->dev->ctrl.abort_limit);
blk_mq_free_request(req);
}
@@ -860,6 +966,16 @@
struct nvme_command cmd;
/*
+ * Did we miss an interrupt?
+ */
+ if (__nvme_poll(nvmeq, req->tag)) {
+ dev_warn(dev->ctrl.device,
+ "I/O %d QID %d timeout, completion polled\n",
+ req->tag, nvmeq->qid);
+ return BLK_EH_HANDLED;
+ }
+
+ /*
* Shutdown immediately if controller times out while starting. The
* reset work will see the pci device disabled when it gets the forced
* cancellation error. All outstanding requests are completed on
@@ -870,7 +986,7 @@
"I/O %d QID %d timeout, disable controller\n",
req->tag, nvmeq->qid);
nvme_dev_disable(dev, false);
- req->errors = NVME_SC_CANCELLED;
+ nvme_req(req)->flags |= NVME_REQ_CANCELLED;
return BLK_EH_HANDLED;
}
@@ -890,7 +1006,7 @@
* Mark the request as handled, since the inline shutdown
* forces all outstanding requests to complete.
*/
- req->errors = NVME_SC_CANCELLED;
+ nvme_req(req)->flags |= NVME_REQ_CANCELLED;
return BLK_EH_HANDLED;
}
@@ -1098,6 +1214,7 @@
nvmeq->cq_phase = 1;
nvmeq->q_db = &dev->dbs[qid * 2 * dev->db_stride];
memset((void *)nvmeq->cqes, 0, CQ_SIZE(nvmeq->q_depth));
+ nvme_dbbuf_init(dev, nvmeq, qid);
dev->online_queues++;
spin_unlock_irq(&nvmeq->q_lock);
}
@@ -1130,18 +1247,18 @@
return result;
}
-static struct blk_mq_ops nvme_mq_admin_ops = {
+static const struct blk_mq_ops nvme_mq_admin_ops = {
.queue_rq = nvme_queue_rq,
- .complete = nvme_complete_rq,
+ .complete = nvme_pci_complete_rq,
.init_hctx = nvme_admin_init_hctx,
.exit_hctx = nvme_admin_exit_hctx,
.init_request = nvme_admin_init_request,
.timeout = nvme_timeout,
};
-static struct blk_mq_ops nvme_mq_ops = {
+static const struct blk_mq_ops nvme_mq_ops = {
.queue_rq = nvme_queue_rq,
- .complete = nvme_complete_rq,
+ .complete = nvme_pci_complete_rq,
.init_hctx = nvme_init_hctx,
.init_request = nvme_init_request,
.map_queues = nvme_pci_map_queues,
@@ -1570,6 +1687,8 @@
if (blk_mq_alloc_tag_set(&dev->tagset))
return 0;
dev->ctrl.tagset = &dev->tagset;
+
+ nvme_dbbuf_set(dev);
} else {
blk_mq_update_nr_hw_queues(&dev->tagset, dev->online_queues - 1);
@@ -1756,6 +1875,7 @@
{
struct nvme_dev *dev = to_nvme_dev(ctrl);
+ nvme_dbbuf_dma_free(dev);
put_device(dev->dev);
if (dev->tagset.tags)
blk_mq_free_tag_set(&dev->tagset);
@@ -1823,6 +1943,13 @@
dev->ctrl.opal_dev = NULL;
}
+ if (dev->ctrl.oacs & NVME_CTRL_OACS_DBBUF_SUPP) {
+ result = nvme_dbbuf_dma_alloc(dev);
+ if (result)
+ dev_warn(dev->dev,
+ "unable to allocate dma for dbbuf\n");
+ }
+
result = nvme_setup_io_queues(dev);
if (result)
goto out;
@@ -2159,13 +2286,13 @@
static const struct pci_device_id nvme_id_table[] = {
{ PCI_VDEVICE(INTEL, 0x0953),
.driver_data = NVME_QUIRK_STRIPE_SIZE |
- NVME_QUIRK_DISCARD_ZEROES, },
+ NVME_QUIRK_DEALLOCATE_ZEROES, },
{ PCI_VDEVICE(INTEL, 0x0a53),
.driver_data = NVME_QUIRK_STRIPE_SIZE |
- NVME_QUIRK_DISCARD_ZEROES, },
+ NVME_QUIRK_DEALLOCATE_ZEROES, },
{ PCI_VDEVICE(INTEL, 0x0a54),
.driver_data = NVME_QUIRK_STRIPE_SIZE |
- NVME_QUIRK_DISCARD_ZEROES, },
+ NVME_QUIRK_DEALLOCATE_ZEROES, },
{ PCI_VDEVICE(INTEL, 0x5845), /* Qemu emulated controller */
.driver_data = NVME_QUIRK_IDENTIFY_CNS, },
{ PCI_DEVICE(0x1c58, 0x0003), /* HGST adapter */
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 16f84eb0..29cf88a 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -34,7 +34,7 @@
#include "fabrics.h"
-#define NVME_RDMA_CONNECT_TIMEOUT_MS 1000 /* 1 second */
+#define NVME_RDMA_CONNECT_TIMEOUT_MS 3000 /* 3 second */
#define NVME_RDMA_MAX_SEGMENT_SIZE 0xffffff /* 24-bit SGL field */
@@ -118,7 +118,6 @@
struct nvme_rdma_qe async_event_sqe;
- int reconnect_delay;
struct delayed_work reconnect_work;
struct list_head list;
@@ -129,14 +128,8 @@
u64 cap;
u32 max_fr_pages;
- union {
- struct sockaddr addr;
- struct sockaddr_in addr_in;
- };
- union {
- struct sockaddr src_addr;
- struct sockaddr_in src_addr_in;
- };
+ struct sockaddr_storage addr;
+ struct sockaddr_storage src_addr;
struct nvme_ctrl ctrl;
};
@@ -569,11 +562,12 @@
return PTR_ERR(queue->cm_id);
}
- queue->cm_error = -ETIMEDOUT;
if (ctrl->ctrl.opts->mask & NVMF_OPT_HOST_TRADDR)
- src_addr = &ctrl->src_addr;
+ src_addr = (struct sockaddr *)&ctrl->src_addr;
- ret = rdma_resolve_addr(queue->cm_id, src_addr, &ctrl->addr,
+ queue->cm_error = -ETIMEDOUT;
+ ret = rdma_resolve_addr(queue->cm_id, src_addr,
+ (struct sockaddr *)&ctrl->addr,
NVME_RDMA_CONNECT_TIMEOUT_MS);
if (ret) {
dev_info(ctrl->ctrl.device,
@@ -712,6 +706,26 @@
kfree(ctrl);
}
+static void nvme_rdma_reconnect_or_remove(struct nvme_rdma_ctrl *ctrl)
+{
+ /* If we are resetting/deleting then do nothing */
+ if (ctrl->ctrl.state != NVME_CTRL_RECONNECTING) {
+ WARN_ON_ONCE(ctrl->ctrl.state == NVME_CTRL_NEW ||
+ ctrl->ctrl.state == NVME_CTRL_LIVE);
+ return;
+ }
+
+ if (nvmf_should_reconnect(&ctrl->ctrl)) {
+ dev_info(ctrl->ctrl.device, "Reconnecting in %d seconds...\n",
+ ctrl->ctrl.opts->reconnect_delay);
+ queue_delayed_work(nvme_rdma_wq, &ctrl->reconnect_work,
+ ctrl->ctrl.opts->reconnect_delay * HZ);
+ } else {
+ dev_info(ctrl->ctrl.device, "Removing controller...\n");
+ queue_work(nvme_rdma_wq, &ctrl->delete_work);
+ }
+}
+
static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
{
struct nvme_rdma_ctrl *ctrl = container_of(to_delayed_work(work),
@@ -719,6 +733,8 @@
bool changed;
int ret;
+ ++ctrl->ctrl.opts->nr_reconnects;
+
if (ctrl->queue_count > 1) {
nvme_rdma_free_io_queues(ctrl);
@@ -763,6 +779,7 @@
changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
WARN_ON_ONCE(!changed);
+ ctrl->ctrl.opts->nr_reconnects = 0;
if (ctrl->queue_count > 1) {
nvme_start_queues(&ctrl->ctrl);
@@ -777,13 +794,9 @@
stop_admin_q:
blk_mq_stop_hw_queues(ctrl->ctrl.admin_q);
requeue:
- /* Make sure we are not resetting/deleting */
- if (ctrl->ctrl.state == NVME_CTRL_RECONNECTING) {
- dev_info(ctrl->ctrl.device,
- "Failed reconnect attempt, requeueing...\n");
- queue_delayed_work(nvme_rdma_wq, &ctrl->reconnect_work,
- ctrl->reconnect_delay * HZ);
- }
+ dev_info(ctrl->ctrl.device, "Failed reconnect attempt %d\n",
+ ctrl->ctrl.opts->nr_reconnects);
+ nvme_rdma_reconnect_or_remove(ctrl);
}
static void nvme_rdma_error_recovery_work(struct work_struct *work)
@@ -810,11 +823,7 @@
blk_mq_tagset_busy_iter(&ctrl->admin_tag_set,
nvme_cancel_request, &ctrl->ctrl);
- dev_info(ctrl->ctrl.device, "reconnecting in %d seconds\n",
- ctrl->reconnect_delay);
-
- queue_delayed_work(nvme_rdma_wq, &ctrl->reconnect_work,
- ctrl->reconnect_delay * HZ);
+ nvme_rdma_reconnect_or_remove(ctrl);
}
static void nvme_rdma_error_recovery(struct nvme_rdma_ctrl *ctrl)
@@ -1169,8 +1178,7 @@
wc->ex.invalidate_rkey == req->mr->rkey)
req->mr->need_inval = false;
- req->req.result = cqe->result;
- blk_mq_complete_request(rq, le16_to_cpu(cqe->status) >> 1);
+ nvme_end_request(rq, cqe->status, cqe->result);
return ret;
}
@@ -1407,7 +1415,7 @@
nvme_rdma_error_recovery(req->queue->ctrl);
/* fail with DNR on cmd timeout */
- rq->errors = NVME_SC_ABORT_REQ | NVME_SC_DNR;
+ nvme_req(rq)->status = NVME_SC_ABORT_REQ | NVME_SC_DNR;
return BLK_EH_HANDLED;
}
@@ -1509,27 +1517,12 @@
static void nvme_rdma_complete_rq(struct request *rq)
{
struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq);
- struct nvme_rdma_queue *queue = req->queue;
- int error = 0;
- nvme_rdma_unmap_data(queue, rq);
-
- if (unlikely(rq->errors)) {
- if (nvme_req_needs_retry(rq, rq->errors)) {
- nvme_requeue_req(rq);
- return;
- }
-
- if (blk_rq_is_passthrough(rq))
- error = rq->errors;
- else
- error = nvme_error_status(rq->errors);
- }
-
- blk_mq_end_request(rq, error);
+ nvme_rdma_unmap_data(req->queue, rq);
+ nvme_complete_rq(rq);
}
-static struct blk_mq_ops nvme_rdma_mq_ops = {
+static const struct blk_mq_ops nvme_rdma_mq_ops = {
.queue_rq = nvme_rdma_queue_rq,
.complete = nvme_rdma_complete_rq,
.init_request = nvme_rdma_init_request,
@@ -1540,7 +1533,7 @@
.timeout = nvme_rdma_timeout,
};
-static struct blk_mq_ops nvme_rdma_admin_mq_ops = {
+static const struct blk_mq_ops nvme_rdma_admin_mq_ops = {
.queue_rq = nvme_rdma_queue_rq,
.complete = nvme_rdma_complete_rq,
.init_request = nvme_rdma_init_admin_request,
@@ -1857,27 +1850,13 @@
return ret;
}
-static int nvme_rdma_parse_ipaddr(struct sockaddr_in *in_addr, char *p)
-{
- u8 *addr = (u8 *)&in_addr->sin_addr.s_addr;
- size_t buflen = strlen(p);
-
- /* XXX: handle IPv6 addresses */
-
- if (buflen > INET_ADDRSTRLEN)
- return -EINVAL;
- if (in4_pton(p, buflen, addr, '\0', NULL) == 0)
- return -EINVAL;
- in_addr->sin_family = AF_INET;
- return 0;
-}
-
static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
struct nvmf_ctrl_options *opts)
{
struct nvme_rdma_ctrl *ctrl;
int ret;
bool changed;
+ char *port;
ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
if (!ctrl)
@@ -1885,40 +1864,33 @@
ctrl->ctrl.opts = opts;
INIT_LIST_HEAD(&ctrl->list);
- ret = nvme_rdma_parse_ipaddr(&ctrl->addr_in, opts->traddr);
+ if (opts->mask & NVMF_OPT_TRSVCID)
+ port = opts->trsvcid;
+ else
+ port = __stringify(NVME_RDMA_IP_PORT);
+
+ ret = inet_pton_with_scope(&init_net, AF_UNSPEC,
+ opts->traddr, port, &ctrl->addr);
if (ret) {
- pr_err("malformed IP address passed: %s\n", opts->traddr);
+ pr_err("malformed address passed: %s:%s\n", opts->traddr, port);
goto out_free_ctrl;
}
if (opts->mask & NVMF_OPT_HOST_TRADDR) {
- ret = nvme_rdma_parse_ipaddr(&ctrl->src_addr_in,
- opts->host_traddr);
+ ret = inet_pton_with_scope(&init_net, AF_UNSPEC,
+ opts->host_traddr, NULL, &ctrl->src_addr);
if (ret) {
- pr_err("malformed src IP address passed: %s\n",
+ pr_err("malformed src address passed: %s\n",
opts->host_traddr);
goto out_free_ctrl;
}
}
- if (opts->mask & NVMF_OPT_TRSVCID) {
- u16 port;
-
- ret = kstrtou16(opts->trsvcid, 0, &port);
- if (ret)
- goto out_free_ctrl;
-
- ctrl->addr_in.sin_port = cpu_to_be16(port);
- } else {
- ctrl->addr_in.sin_port = cpu_to_be16(NVME_RDMA_IP_PORT);
- }
-
ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_rdma_ctrl_ops,
0 /* no quirks, we're perfect! */);
if (ret)
goto out_free_ctrl;
- ctrl->reconnect_delay = opts->reconnect_delay;
INIT_DELAYED_WORK(&ctrl->reconnect_work,
nvme_rdma_reconnect_ctrl_work);
INIT_WORK(&ctrl->err_work, nvme_rdma_error_recovery_work);
@@ -1977,7 +1949,7 @@
changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
WARN_ON_ONCE(!changed);
- dev_info(ctrl->ctrl.device, "new ctrl: NQN \"%s\", addr %pISp\n",
+ dev_info(ctrl->ctrl.device, "new ctrl: NQN \"%s\", addr %pISpcs\n",
ctrl->ctrl.opts->subsysnqn, &ctrl->addr);
kref_get(&ctrl->ctrl.kref);
@@ -2013,7 +1985,7 @@
.name = "rdma",
.required_opts = NVMF_OPT_TRADDR,
.allowed_opts = NVMF_OPT_TRSVCID | NVMF_OPT_RECONNECT_DELAY |
- NVMF_OPT_HOST_TRADDR,
+ NVMF_OPT_HOST_TRADDR | NVMF_OPT_CTRL_LOSS_TMO,
.create_ctrl = nvme_rdma_create_ctrl,
};
@@ -2055,12 +2027,20 @@
return -ENOMEM;
ret = ib_register_client(&nvme_rdma_ib_client);
- if (ret) {
- destroy_workqueue(nvme_rdma_wq);
- return ret;
- }
+ if (ret)
+ goto err_destroy_wq;
- return nvmf_register_transport(&nvme_rdma_transport);
+ ret = nvmf_register_transport(&nvme_rdma_transport);
+ if (ret)
+ goto err_unreg_client;
+
+ return 0;
+
+err_unreg_client:
+ ib_unregister_client(&nvme_rdma_ib_client);
+err_destroy_wq:
+ destroy_workqueue(nvme_rdma_wq);
+ return ret;
}
static void __exit nvme_rdma_cleanup_module(void)
diff --git a/drivers/nvme/host/scsi.c b/drivers/nvme/host/scsi.c
index f49ae27..1f7671e 100644
--- a/drivers/nvme/host/scsi.c
+++ b/drivers/nvme/host/scsi.c
@@ -1609,7 +1609,7 @@
struct nvme_command c;
u8 opcode = (is_write ? nvme_cmd_write : nvme_cmd_read);
u16 control;
- u32 max_blocks = queue_max_hw_sectors(ns->queue);
+ u32 max_blocks = queue_max_hw_sectors(ns->queue) >> (ns->lba_shift - 9);
num_cmds = nvme_trans_io_get_num_cmds(hdr, cdb_info, max_blocks);
@@ -2138,15 +2138,6 @@
return res;
}
-static int nvme_trans_security_protocol(struct nvme_ns *ns,
- struct sg_io_hdr *hdr,
- u8 *cmd)
-{
- return nvme_trans_completion(hdr, SAM_STAT_CHECK_CONDITION,
- ILLEGAL_REQUEST, SCSI_ASC_ILLEGAL_COMMAND,
- SCSI_ASCQ_CAUSE_NOT_REPORTABLE);
-}
-
static int nvme_trans_synchronize_cache(struct nvme_ns *ns,
struct sg_io_hdr *hdr)
{
@@ -2414,10 +2405,6 @@
case REQUEST_SENSE:
retcode = nvme_trans_request_sense(ns, hdr, cmd);
break;
- case SECURITY_PROTOCOL_IN:
- case SECURITY_PROTOCOL_OUT:
- retcode = nvme_trans_security_protocol(ns, hdr, cmd);
- break;
case SYNCHRONIZE_CACHE:
retcode = nvme_trans_synchronize_cache(ns, hdr);
break;
diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 76450b0..ff1f970 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -121,7 +121,7 @@
}
switch (req->cmd->get_log_page.lid) {
- case 0x01:
+ case NVME_LOG_ERROR:
/*
* We currently never set the More bit in the status field,
* so all error log entries are invalid and can be zeroed out.
@@ -129,7 +129,7 @@
* mandatory log page.
*/
break;
- case 0x02:
+ case NVME_LOG_SMART:
/*
* XXX: fill out actual smart log
*
@@ -149,7 +149,7 @@
goto err;
}
break;
- case 0x03:
+ case NVME_LOG_FW_SLOT:
/*
* We only support a single firmware slot which always is
* active, so we can zero out the whole firmware slot log and
@@ -480,31 +480,25 @@
nvmet_req_complete(req, 0);
}
-int nvmet_parse_admin_cmd(struct nvmet_req *req)
+u16 nvmet_parse_admin_cmd(struct nvmet_req *req)
{
struct nvme_command *cmd = req->cmd;
+ u16 ret;
req->ns = NULL;
- if (unlikely(!(req->sq->ctrl->cc & NVME_CC_ENABLE))) {
- pr_err("nvmet: got admin cmd %d while CC.EN == 0\n",
- cmd->common.opcode);
- return NVME_SC_CMD_SEQ_ERROR | NVME_SC_DNR;
- }
- if (unlikely(!(req->sq->ctrl->csts & NVME_CSTS_RDY))) {
- pr_err("nvmet: got admin cmd %d while CSTS.RDY == 0\n",
- cmd->common.opcode);
- return NVME_SC_CMD_SEQ_ERROR | NVME_SC_DNR;
- }
+ ret = nvmet_check_ctrl_status(req, cmd);
+ if (unlikely(ret))
+ return ret;
switch (cmd->common.opcode) {
case nvme_admin_get_log_page:
req->data_len = nvmet_get_log_page_len(cmd);
switch (cmd->get_log_page.lid) {
- case 0x01:
- case 0x02:
- case 0x03:
+ case NVME_LOG_ERROR:
+ case NVME_LOG_SMART:
+ case NVME_LOG_FW_SLOT:
req->execute = nvmet_execute_get_log_page;
return 0;
}
@@ -545,6 +539,7 @@
return 0;
}
- pr_err("nvmet: unhandled cmd %d\n", cmd->common.opcode);
+ pr_err("unhandled cmd %d on qid %d\n", cmd->common.opcode,
+ req->sq->qid);
return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
}
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 798653b..cf90713 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -273,8 +273,8 @@
ns->bdev = blkdev_get_by_path(ns->device_path, FMODE_READ | FMODE_WRITE,
NULL);
if (IS_ERR(ns->bdev)) {
- pr_err("nvmet: failed to open block device %s: (%ld)\n",
- ns->device_path, PTR_ERR(ns->bdev));
+ pr_err("failed to open block device %s: (%ld)\n",
+ ns->device_path, PTR_ERR(ns->bdev));
ret = PTR_ERR(ns->bdev);
ns->bdev = NULL;
goto out_unlock;
@@ -661,6 +661,23 @@
return status;
}
+u16 nvmet_check_ctrl_status(struct nvmet_req *req, struct nvme_command *cmd)
+{
+ if (unlikely(!(req->sq->ctrl->cc & NVME_CC_ENABLE))) {
+ pr_err("got io cmd %d while CC.EN == 0 on qid = %d\n",
+ cmd->common.opcode, req->sq->qid);
+ return NVME_SC_CMD_SEQ_ERROR | NVME_SC_DNR;
+ }
+
+ if (unlikely(!(req->sq->ctrl->csts & NVME_CSTS_RDY))) {
+ pr_err("got io cmd %d while CSTS.RDY == 0 on qid = %d\n",
+ cmd->common.opcode, req->sq->qid);
+ req->ns = NULL;
+ return NVME_SC_CMD_SEQ_ERROR | NVME_SC_DNR;
+ }
+ return 0;
+}
+
static bool __nvmet_host_allowed(struct nvmet_subsys *subsys,
const char *hostnqn)
{
diff --git a/drivers/nvme/target/discovery.c b/drivers/nvme/target/discovery.c
index af8aabf..1aaf597 100644
--- a/drivers/nvme/target/discovery.c
+++ b/drivers/nvme/target/discovery.c
@@ -159,15 +159,15 @@
nvmet_req_complete(req, status);
}
-int nvmet_parse_discovery_cmd(struct nvmet_req *req)
+u16 nvmet_parse_discovery_cmd(struct nvmet_req *req)
{
struct nvme_command *cmd = req->cmd;
req->ns = NULL;
if (unlikely(!(req->sq->ctrl->csts & NVME_CSTS_RDY))) {
- pr_err("nvmet: got cmd %d while not ready\n",
- cmd->common.opcode);
+ pr_err("got cmd %d while not ready\n",
+ cmd->common.opcode);
return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
}
@@ -180,8 +180,8 @@
req->execute = nvmet_execute_get_disc_log_page;
return 0;
default:
- pr_err("nvmet: unsupported get_log_page lid %d\n",
- cmd->get_log_page.lid);
+ pr_err("unsupported get_log_page lid %d\n",
+ cmd->get_log_page.lid);
return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
}
case nvme_admin_identify:
@@ -192,17 +192,16 @@
nvmet_execute_identify_disc_ctrl;
return 0;
default:
- pr_err("nvmet: unsupported identify cns %d\n",
- cmd->identify.cns);
+ pr_err("unsupported identify cns %d\n",
+ cmd->identify.cns);
return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
}
default:
- pr_err("nvmet: unsupported cmd %d\n",
- cmd->common.opcode);
+ pr_err("unsupported cmd %d\n", cmd->common.opcode);
return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
}
- pr_err("nvmet: unhandled cmd %d\n", cmd->common.opcode);
+ pr_err("unhandled cmd %d\n", cmd->common.opcode);
return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
}
diff --git a/drivers/nvme/target/fabrics-cmd.c b/drivers/nvme/target/fabrics-cmd.c
index 8bd022af..3cc1726 100644
--- a/drivers/nvme/target/fabrics-cmd.c
+++ b/drivers/nvme/target/fabrics-cmd.c
@@ -73,7 +73,7 @@
nvmet_req_complete(req, status);
}
-int nvmet_parse_fabrics_cmd(struct nvmet_req *req)
+u16 nvmet_parse_fabrics_cmd(struct nvmet_req *req)
{
struct nvme_command *cmd = req->cmd;
@@ -122,7 +122,15 @@
struct nvmet_ctrl *ctrl = NULL;
u16 status = 0;
- d = kmap(sg_page(req->sg)) + req->sg->offset;
+ d = kmalloc(sizeof(*d), GFP_KERNEL);
+ if (!d) {
+ status = NVME_SC_INTERNAL;
+ goto complete;
+ }
+
+ status = nvmet_copy_from_sgl(req, 0, d, sizeof(*d));
+ if (status)
+ goto out;
/* zero out initial completion result, assign values as needed */
req->rsp->result.u32 = 0;
@@ -143,7 +151,7 @@
}
status = nvmet_alloc_ctrl(d->subsysnqn, d->hostnqn, req,
- le32_to_cpu(c->kato), &ctrl);
+ le32_to_cpu(c->kato), &ctrl);
if (status)
goto out;
@@ -158,7 +166,8 @@
req->rsp->result.u16 = cpu_to_le16(ctrl->cntlid);
out:
- kunmap(sg_page(req->sg));
+ kfree(d);
+complete:
nvmet_req_complete(req, status);
}
@@ -170,7 +179,15 @@
u16 qid = le16_to_cpu(c->qid);
u16 status = 0;
- d = kmap(sg_page(req->sg)) + req->sg->offset;
+ d = kmalloc(sizeof(*d), GFP_KERNEL);
+ if (!d) {
+ status = NVME_SC_INTERNAL;
+ goto complete;
+ }
+
+ status = nvmet_copy_from_sgl(req, 0, d, sizeof(*d));
+ if (status)
+ goto out;
/* zero out initial completion result, assign values as needed */
req->rsp->result.u32 = 0;
@@ -183,8 +200,8 @@
}
status = nvmet_ctrl_find_get(d->subsysnqn, d->hostnqn,
- le16_to_cpu(d->cntlid),
- req, &ctrl);
+ le16_to_cpu(d->cntlid),
+ req, &ctrl);
if (status)
goto out;
@@ -205,7 +222,8 @@
pr_info("adding queue %d to ctrl %d.\n", qid, ctrl->cntlid);
out:
- kunmap(sg_page(req->sg));
+ kfree(d);
+complete:
nvmet_req_complete(req, status);
return;
@@ -214,7 +232,7 @@
goto out;
}
-int nvmet_parse_connect_cmd(struct nvmet_req *req)
+u16 nvmet_parse_connect_cmd(struct nvmet_req *req)
{
struct nvme_command *cmd = req->cmd;
diff --git a/drivers/nvme/target/fc.c b/drivers/nvme/target/fc.c
index 8f483ee..62eba29 100644
--- a/drivers/nvme/target/fc.c
+++ b/drivers/nvme/target/fc.c
@@ -82,10 +82,13 @@
enum nvmet_fcp_datadir io_dir;
bool active;
bool abort;
+ bool aborted;
+ bool writedataactive;
spinlock_t flock;
struct nvmet_req req;
struct work_struct work;
+ struct work_struct done_work;
struct nvmet_fc_tgtport *tgtport;
struct nvmet_fc_tgt_queue *queue;
@@ -116,7 +119,7 @@
u16 qid;
u16 sqsize;
u16 ersp_ratio;
- u16 sqhd;
+ __le16 sqhd;
int cpu;
atomic_t connected;
atomic_t sqtail;
@@ -213,6 +216,7 @@
static void nvmet_fc_handle_ls_rqst_work(struct work_struct *work);
static void nvmet_fc_handle_fcp_rqst_work(struct work_struct *work);
+static void nvmet_fc_fcp_rqst_op_done_work(struct work_struct *work);
static void nvmet_fc_tgt_a_put(struct nvmet_fc_tgt_assoc *assoc);
static int nvmet_fc_tgt_a_get(struct nvmet_fc_tgt_assoc *assoc);
static void nvmet_fc_tgt_q_put(struct nvmet_fc_tgt_queue *queue);
@@ -414,9 +418,13 @@
for (i = 0; i < queue->sqsize; fod++, i++) {
INIT_WORK(&fod->work, nvmet_fc_handle_fcp_rqst_work);
+ INIT_WORK(&fod->done_work, nvmet_fc_fcp_rqst_op_done_work);
fod->tgtport = tgtport;
fod->queue = queue;
fod->active = false;
+ fod->abort = false;
+ fod->aborted = false;
+ fod->fcpreq = NULL;
list_add_tail(&fod->fcp_list, &queue->fod_list);
spin_lock_init(&fod->flock);
@@ -463,7 +471,6 @@
if (fod) {
list_del(&fod->fcp_list);
fod->active = true;
- fod->abort = false;
/*
* no queue reference is taken, as it was taken by the
* queue lookup just prior to the allocation. The iod
@@ -479,17 +486,30 @@
nvmet_fc_free_fcp_iod(struct nvmet_fc_tgt_queue *queue,
struct nvmet_fc_fcp_iod *fod)
{
+ struct nvmefc_tgt_fcp_req *fcpreq = fod->fcpreq;
+ struct nvmet_fc_tgtport *tgtport = fod->tgtport;
unsigned long flags;
+ fc_dma_sync_single_for_cpu(tgtport->dev, fod->rspdma,
+ sizeof(fod->rspiubuf), DMA_TO_DEVICE);
+
+ fcpreq->nvmet_fc_private = NULL;
+
spin_lock_irqsave(&queue->qlock, flags);
list_add_tail(&fod->fcp_list, &fod->queue->fod_list);
fod->active = false;
+ fod->abort = false;
+ fod->aborted = false;
+ fod->writedataactive = false;
+ fod->fcpreq = NULL;
spin_unlock_irqrestore(&queue->qlock, flags);
/*
* release the reference taken at queue lookup and fod allocation
*/
nvmet_fc_tgt_q_put(queue);
+
+ tgtport->ops->fcp_req_release(&tgtport->fc_target_port, fcpreq);
}
static int
@@ -616,32 +636,12 @@
static void
-nvmet_fc_abort_op(struct nvmet_fc_tgtport *tgtport,
- struct nvmefc_tgt_fcp_req *fcpreq)
-{
- int ret;
-
- fcpreq->op = NVMET_FCOP_ABORT;
- fcpreq->offset = 0;
- fcpreq->timeout = 0;
- fcpreq->transfer_length = 0;
- fcpreq->transferred_length = 0;
- fcpreq->fcp_error = 0;
- fcpreq->sg_cnt = 0;
-
- ret = tgtport->ops->fcp_op(&tgtport->fc_target_port, fcpreq);
- if (ret)
- /* should never reach here !! */
- WARN_ON(1);
-}
-
-
-static void
nvmet_fc_delete_target_queue(struct nvmet_fc_tgt_queue *queue)
{
+ struct nvmet_fc_tgtport *tgtport = queue->assoc->tgtport;
struct nvmet_fc_fcp_iod *fod = queue->fod;
unsigned long flags;
- int i;
+ int i, writedataactive;
bool disconnect;
disconnect = atomic_xchg(&queue->connected, 0);
@@ -652,7 +652,20 @@
if (fod->active) {
spin_lock(&fod->flock);
fod->abort = true;
+ writedataactive = fod->writedataactive;
spin_unlock(&fod->flock);
+ /*
+ * only call lldd abort routine if waiting for
+ * writedata. other outstanding ops should finish
+ * on their own.
+ */
+ if (writedataactive) {
+ spin_lock(&fod->flock);
+ fod->aborted = true;
+ spin_unlock(&fod->flock);
+ tgtport->ops->fcp_abort(
+ &tgtport->fc_target_port, fod->fcpreq);
+ }
}
}
spin_unlock_irqrestore(&queue->qlock, flags);
@@ -846,7 +859,8 @@
int ret, idx;
if (!template->xmt_ls_rsp || !template->fcp_op ||
- !template->targetport_delete ||
+ !template->fcp_abort ||
+ !template->fcp_req_release || !template->targetport_delete ||
!template->max_hw_queues || !template->max_sgl_segments ||
!template->max_dif_sgl_segments || !template->dma_boundary) {
ret = -EINVAL;
@@ -1044,7 +1058,7 @@
static void
-nvmet_fc_format_rsp_hdr(void *buf, u8 ls_cmd, u32 desc_len, u8 rqst_ls_cmd)
+nvmet_fc_format_rsp_hdr(void *buf, u8 ls_cmd, __be32 desc_len, u8 rqst_ls_cmd)
{
struct fcnvme_ls_acc_hdr *acc = buf;
@@ -1189,8 +1203,8 @@
validation_errors[ret]);
iod->lsreq->rsplen = nvmet_fc_format_rjt(acc,
NVME_FC_MAX_LS_BUFFER_SIZE, rqst->w0.ls_cmd,
- ELS_RJT_LOGIC,
- ELS_EXPL_NONE, 0);
+ FCNVME_RJT_RC_LOGIC,
+ FCNVME_RJT_EXP_NONE, 0);
return;
}
@@ -1281,8 +1295,9 @@
iod->lsreq->rsplen = nvmet_fc_format_rjt(acc,
NVME_FC_MAX_LS_BUFFER_SIZE, rqst->w0.ls_cmd,
(ret == VERR_NO_ASSOC) ?
- ELS_RJT_PROT : ELS_RJT_LOGIC,
- ELS_EXPL_NONE, 0);
+ FCNVME_RJT_RC_INV_ASSOC :
+ FCNVME_RJT_RC_LOGIC,
+ FCNVME_RJT_EXP_NONE, 0);
return;
}
@@ -1369,8 +1384,12 @@
validation_errors[ret]);
iod->lsreq->rsplen = nvmet_fc_format_rjt(acc,
NVME_FC_MAX_LS_BUFFER_SIZE, rqst->w0.ls_cmd,
- (ret == 8) ? ELS_RJT_PROT : ELS_RJT_LOGIC,
- ELS_EXPL_NONE, 0);
+ (ret == VERR_NO_ASSOC) ?
+ FCNVME_RJT_RC_INV_ASSOC :
+ (ret == VERR_NO_CONN) ?
+ FCNVME_RJT_RC_INV_CONN :
+ FCNVME_RJT_RC_LOGIC,
+ FCNVME_RJT_EXP_NONE, 0);
return;
}
@@ -1479,7 +1498,7 @@
default:
iod->lsreq->rsplen = nvmet_fc_format_rjt(iod->rspbuf,
NVME_FC_MAX_LS_BUFFER_SIZE, w0->ls_cmd,
- ELS_RJT_INVAL, ELS_EXPL_NONE, 0);
+ FCNVME_RJT_RC_INVAL, FCNVME_RJT_EXP_NONE, 0);
}
nvmet_fc_xmt_ls_rsp(tgtport, iod);
@@ -1619,6 +1638,8 @@
for_each_sg(fod->data_sg, sg, fod->data_sg_cnt, count)
__free_page(sg_page(sg));
kfree(fod->data_sg);
+ fod->data_sg = NULL;
+ fod->data_sg_cnt = 0;
}
@@ -1679,7 +1700,7 @@
xfr_length != fod->total_length ||
(le16_to_cpu(cqe->status) & 0xFFFE) || cqewd[0] || cqewd[1] ||
(sqe->flags & (NVME_CMD_FUSE_FIRST | NVME_CMD_FUSE_SECOND)) ||
- queue_90percent_full(fod->queue, cqe->sq_head))
+ queue_90percent_full(fod->queue, le16_to_cpu(cqe->sq_head)))
send_ersp = true;
/* re-set the fields */
@@ -1704,6 +1725,26 @@
static void nvmet_fc_xmt_fcp_op_done(struct nvmefc_tgt_fcp_req *fcpreq);
static void
+nvmet_fc_abort_op(struct nvmet_fc_tgtport *tgtport,
+ struct nvmet_fc_fcp_iod *fod)
+{
+ struct nvmefc_tgt_fcp_req *fcpreq = fod->fcpreq;
+
+ /* data no longer needed */
+ nvmet_fc_free_tgt_pgs(fod);
+
+ /*
+ * if an ABTS was received or we issued the fcp_abort early
+ * don't call abort routine again.
+ */
+ /* no need to take lock - lock was taken earlier to get here */
+ if (!fod->aborted)
+ tgtport->ops->fcp_abort(&tgtport->fc_target_port, fcpreq);
+
+ nvmet_fc_free_fcp_iod(fod->queue, fod);
+}
+
+static void
nvmet_fc_xmt_fcp_rsp(struct nvmet_fc_tgtport *tgtport,
struct nvmet_fc_fcp_iod *fod)
{
@@ -1716,7 +1757,7 @@
ret = tgtport->ops->fcp_op(&tgtport->fc_target_port, fod->fcpreq);
if (ret)
- nvmet_fc_abort_op(tgtport, fod->fcpreq);
+ nvmet_fc_abort_op(tgtport, fod);
}
static void
@@ -1725,6 +1766,7 @@
{
struct nvmefc_tgt_fcp_req *fcpreq = fod->fcpreq;
struct scatterlist *sg, *datasg;
+ unsigned long flags;
u32 tlen, sg_off;
int ret;
@@ -1789,10 +1831,13 @@
*/
fod->abort = true;
- if (op == NVMET_FCOP_WRITEDATA)
+ if (op == NVMET_FCOP_WRITEDATA) {
+ spin_lock_irqsave(&fod->flock, flags);
+ fod->writedataactive = false;
+ spin_unlock_irqrestore(&fod->flock, flags);
nvmet_req_complete(&fod->req,
NVME_SC_FC_TRANSPORT_ERROR);
- else /* NVMET_FCOP_READDATA or NVMET_FCOP_READDATA_RSP */ {
+ } else /* NVMET_FCOP_READDATA or NVMET_FCOP_READDATA_RSP */ {
fcpreq->fcp_error = ret;
fcpreq->transferred_length = 0;
nvmet_fc_xmt_fcp_op_done(fod->fcpreq);
@@ -1800,32 +1845,54 @@
}
}
-static void
-nvmet_fc_xmt_fcp_op_done(struct nvmefc_tgt_fcp_req *fcpreq)
+static inline bool
+__nvmet_fc_fod_op_abort(struct nvmet_fc_fcp_iod *fod, bool abort)
{
- struct nvmet_fc_fcp_iod *fod = fcpreq->nvmet_fc_private;
+ struct nvmefc_tgt_fcp_req *fcpreq = fod->fcpreq;
+ struct nvmet_fc_tgtport *tgtport = fod->tgtport;
+
+ /* if in the middle of an io and we need to tear down */
+ if (abort) {
+ if (fcpreq->op == NVMET_FCOP_WRITEDATA) {
+ nvmet_req_complete(&fod->req,
+ NVME_SC_FC_TRANSPORT_ERROR);
+ return true;
+ }
+
+ nvmet_fc_abort_op(tgtport, fod);
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * actual done handler for FCP operations when completed by the lldd
+ */
+static void
+nvmet_fc_fod_op_done(struct nvmet_fc_fcp_iod *fod)
+{
+ struct nvmefc_tgt_fcp_req *fcpreq = fod->fcpreq;
struct nvmet_fc_tgtport *tgtport = fod->tgtport;
unsigned long flags;
bool abort;
spin_lock_irqsave(&fod->flock, flags);
abort = fod->abort;
+ fod->writedataactive = false;
spin_unlock_irqrestore(&fod->flock, flags);
- /* if in the middle of an io and we need to tear down */
- if (abort && fcpreq->op != NVMET_FCOP_ABORT) {
- /* data no longer needed */
- nvmet_fc_free_tgt_pgs(fod);
-
- nvmet_req_complete(&fod->req, fcpreq->fcp_error);
- return;
- }
-
switch (fcpreq->op) {
case NVMET_FCOP_WRITEDATA:
+ if (__nvmet_fc_fod_op_abort(fod, abort))
+ return;
if (fcpreq->fcp_error ||
fcpreq->transferred_length != fcpreq->transfer_length) {
+ spin_lock(&fod->flock);
+ fod->abort = true;
+ spin_unlock(&fod->flock);
+
nvmet_req_complete(&fod->req,
NVME_SC_FC_TRANSPORT_ERROR);
return;
@@ -1833,6 +1900,10 @@
fod->offset += fcpreq->transferred_length;
if (fod->offset != fod->total_length) {
+ spin_lock_irqsave(&fod->flock, flags);
+ fod->writedataactive = true;
+ spin_unlock_irqrestore(&fod->flock, flags);
+
/* transfer the next chunk */
nvmet_fc_transfer_fcp_data(tgtport, fod,
NVMET_FCOP_WRITEDATA);
@@ -1847,12 +1918,11 @@
case NVMET_FCOP_READDATA:
case NVMET_FCOP_READDATA_RSP:
+ if (__nvmet_fc_fod_op_abort(fod, abort))
+ return;
if (fcpreq->fcp_error ||
fcpreq->transferred_length != fcpreq->transfer_length) {
- /* data no longer needed */
- nvmet_fc_free_tgt_pgs(fod);
-
- nvmet_fc_abort_op(tgtport, fod->fcpreq);
+ nvmet_fc_abort_op(tgtport, fod);
return;
}
@@ -1861,8 +1931,6 @@
if (fcpreq->op == NVMET_FCOP_READDATA_RSP) {
/* data no longer needed */
nvmet_fc_free_tgt_pgs(fod);
- fc_dma_sync_single_for_cpu(tgtport->dev, fod->rspdma,
- sizeof(fod->rspiubuf), DMA_TO_DEVICE);
nvmet_fc_free_fcp_iod(fod->queue, fod);
return;
}
@@ -1885,19 +1953,38 @@
break;
case NVMET_FCOP_RSP:
- case NVMET_FCOP_ABORT:
- fc_dma_sync_single_for_cpu(tgtport->dev, fod->rspdma,
- sizeof(fod->rspiubuf), DMA_TO_DEVICE);
+ if (__nvmet_fc_fod_op_abort(fod, abort))
+ return;
nvmet_fc_free_fcp_iod(fod->queue, fod);
break;
default:
- nvmet_fc_free_tgt_pgs(fod);
- nvmet_fc_abort_op(tgtport, fod->fcpreq);
break;
}
}
+static void
+nvmet_fc_fcp_rqst_op_done_work(struct work_struct *work)
+{
+ struct nvmet_fc_fcp_iod *fod =
+ container_of(work, struct nvmet_fc_fcp_iod, done_work);
+
+ nvmet_fc_fod_op_done(fod);
+}
+
+static void
+nvmet_fc_xmt_fcp_op_done(struct nvmefc_tgt_fcp_req *fcpreq)
+{
+ struct nvmet_fc_fcp_iod *fod = fcpreq->nvmet_fc_private;
+ struct nvmet_fc_tgt_queue *queue = fod->queue;
+
+ if (fod->tgtport->ops->target_features & NVMET_FCTGTFEAT_OPDONE_IN_ISR)
+ /* context switch so completion is not in ISR context */
+ queue_work_on(queue->cpu, queue->work_q, &fod->done_work);
+ else
+ nvmet_fc_fod_op_done(fod);
+}
+
/*
* actual completion handler after execution by the nvmet layer
*/
@@ -1919,10 +2006,7 @@
fod->queue->sqhd = cqe->sq_head;
if (abort) {
- /* data no longer needed */
- nvmet_fc_free_tgt_pgs(fod);
-
- nvmet_fc_abort_op(tgtport, fod->fcpreq);
+ nvmet_fc_abort_op(tgtport, fod);
return;
}
@@ -1971,7 +2055,7 @@
/*
* Actual processing routine for received FC-NVME LS Requests from the LLD
*/
-void
+static void
nvmet_fc_handle_fcp_rqst(struct nvmet_fc_tgtport *tgtport,
struct nvmet_fc_fcp_iod *fod)
{
@@ -2018,8 +2102,8 @@
&fod->queue->nvme_cq,
&fod->queue->nvme_sq,
&nvmet_fc_tgt_fcp_ops);
- if (!ret) { /* bad SQE content */
- nvmet_fc_abort_op(tgtport, fod->fcpreq);
+ if (!ret) { /* bad SQE content or invalid ctrl state */
+ nvmet_fc_abort_op(tgtport, fod);
return;
}
@@ -2059,7 +2143,7 @@
return;
transport_error:
- nvmet_fc_abort_op(tgtport, fod->fcpreq);
+ nvmet_fc_abort_op(tgtport, fod);
}
/*
@@ -2089,7 +2173,7 @@
* If this routine returns error, the lldd should abort the exchange.
*
* @target_port: pointer to the (registered) target port the FCP CMD IU
- * was receive on.
+ * was received on.
* @fcpreq: pointer to a fcpreq request structure to be used to reference
* the exchange corresponding to the FCP Exchange.
* @cmdiubuf: pointer to the buffer containing the FCP CMD IU
@@ -2112,7 +2196,6 @@
(be16_to_cpu(cmdiu->iu_len) != (sizeof(*cmdiu)/4)))
return -EIO;
-
queue = nvmet_fc_find_target_queue(tgtport,
be64_to_cpu(cmdiu->connection_id));
if (!queue)
@@ -2142,12 +2225,68 @@
((queue->qid - 1) % tgtport->ops->max_hw_queues) : 0;
memcpy(&fod->cmdiubuf, cmdiubuf, cmdiubuf_len);
- queue_work_on(queue->cpu, queue->work_q, &fod->work);
+ if (tgtport->ops->target_features & NVMET_FCTGTFEAT_CMD_IN_ISR)
+ queue_work_on(queue->cpu, queue->work_q, &fod->work);
+ else
+ nvmet_fc_handle_fcp_rqst(tgtport, fod);
return 0;
}
EXPORT_SYMBOL_GPL(nvmet_fc_rcv_fcp_req);
+/**
+ * nvmet_fc_rcv_fcp_abort - transport entry point called by an LLDD
+ * upon the reception of an ABTS for a FCP command
+ *
+ * Notify the transport that an ABTS has been received for a FCP command
+ * that had been given to the transport via nvmet_fc_rcv_fcp_req(). The
+ * LLDD believes the command is still being worked on
+ * (template_ops->fcp_req_release() has not been called).
+ *
+ * The transport will wait for any outstanding work (an op to the LLDD,
+ * which the lldd should complete with error due to the ABTS; or the
+ * completion from the nvmet layer of the nvme command), then will
+ * stop processing and call the nvmet_fc_rcv_fcp_req() callback to
+ * return the i/o context to the LLDD. The LLDD may send the BA_ACC
+ * to the ABTS either after return from this function (assuming any
+ * outstanding op work has been terminated) or upon the callback being
+ * called.
+ *
+ * @target_port: pointer to the (registered) target port the FCP CMD IU
+ * was received on.
+ * @fcpreq: pointer to the fcpreq request structure that corresponds
+ * to the exchange that received the ABTS.
+ */
+void
+nvmet_fc_rcv_fcp_abort(struct nvmet_fc_target_port *target_port,
+ struct nvmefc_tgt_fcp_req *fcpreq)
+{
+ struct nvmet_fc_fcp_iod *fod = fcpreq->nvmet_fc_private;
+ struct nvmet_fc_tgt_queue *queue;
+ unsigned long flags;
+
+ if (!fod || fod->fcpreq != fcpreq)
+ /* job appears to have already completed, ignore abort */
+ return;
+
+ queue = fod->queue;
+
+ spin_lock_irqsave(&queue->qlock, flags);
+ if (fod->active) {
+ /*
+ * mark as abort. The abort handler, invoked upon completion
+ * of any work, will detect the aborted status and do the
+ * callback.
+ */
+ spin_lock(&fod->flock);
+ fod->abort = true;
+ fod->aborted = true;
+ spin_unlock(&fod->flock);
+ }
+ spin_unlock_irqrestore(&queue->qlock, flags);
+}
+EXPORT_SYMBOL_GPL(nvmet_fc_rcv_fcp_abort);
+
enum {
FCT_TRADDR_ERR = 0,
FCT_TRADDR_WWNN = 1 << 0,
@@ -2177,7 +2316,7 @@
if (!options)
return -ENOMEM;
- while ((p = strsep(&o, ",\n")) != NULL) {
+ while ((p = strsep(&o, ":\n")) != NULL) {
if (!*p)
continue;
@@ -2238,6 +2377,7 @@
if (!tgtport->port) {
tgtport->port = port;
port->priv = tgtport;
+ nvmet_fc_tgtport_get(tgtport);
ret = 0;
} else
ret = -EALREADY;
diff --git a/drivers/nvme/target/fcloop.c b/drivers/nvme/target/fcloop.c
index 4e8e6a2..15551ef 100644
--- a/drivers/nvme/target/fcloop.c
+++ b/drivers/nvme/target/fcloop.c
@@ -246,11 +246,19 @@
struct fcloop_fcpreq {
struct fcloop_tport *tport;
struct nvmefc_fcp_req *fcpreq;
+ spinlock_t reqlock;
u16 status;
+ bool active;
+ bool aborted;
struct work_struct work;
struct nvmefc_tgt_fcp_req tgt_fcp_req;
};
+struct fcloop_ini_fcpreq {
+ struct nvmefc_fcp_req *fcpreq;
+ struct fcloop_fcpreq *tfcp_req;
+ struct work_struct iniwork;
+};
static inline struct fcloop_lsreq *
tgt_ls_req_to_lsreq(struct nvmefc_tgt_ls_req *tgt_lsreq)
@@ -341,7 +349,21 @@
}
/*
- * FCP IO operation done. call back up initiator "done" flows.
+ * FCP IO operation done by initiator abort.
+ * call back up initiator "done" flows.
+ */
+static void
+fcloop_tgt_fcprqst_ini_done_work(struct work_struct *work)
+{
+ struct fcloop_ini_fcpreq *inireq =
+ container_of(work, struct fcloop_ini_fcpreq, iniwork);
+
+ inireq->fcpreq->done(inireq->fcpreq);
+}
+
+/*
+ * FCP IO operation done by target completion.
+ * call back up initiator "done" flows.
*/
static void
fcloop_tgt_fcprqst_done_work(struct work_struct *work)
@@ -349,12 +371,18 @@
struct fcloop_fcpreq *tfcp_req =
container_of(work, struct fcloop_fcpreq, work);
struct fcloop_tport *tport = tfcp_req->tport;
- struct nvmefc_fcp_req *fcpreq = tfcp_req->fcpreq;
+ struct nvmefc_fcp_req *fcpreq;
- if (tport->remoteport) {
+ spin_lock(&tfcp_req->reqlock);
+ fcpreq = tfcp_req->fcpreq;
+ spin_unlock(&tfcp_req->reqlock);
+
+ if (tport->remoteport && fcpreq) {
fcpreq->status = tfcp_req->status;
fcpreq->done(fcpreq);
}
+
+ kfree(tfcp_req);
}
@@ -364,20 +392,25 @@
void *hw_queue_handle,
struct nvmefc_fcp_req *fcpreq)
{
- struct fcloop_fcpreq *tfcp_req = fcpreq->private;
struct fcloop_rport *rport = remoteport->private;
+ struct fcloop_ini_fcpreq *inireq = fcpreq->private;
+ struct fcloop_fcpreq *tfcp_req;
int ret = 0;
- INIT_WORK(&tfcp_req->work, fcloop_tgt_fcprqst_done_work);
+ if (!rport->targetport)
+ return -ECONNREFUSED;
- if (!rport->targetport) {
- tfcp_req->status = NVME_SC_FC_TRANSPORT_ERROR;
- schedule_work(&tfcp_req->work);
- return ret;
- }
+ tfcp_req = kzalloc(sizeof(*tfcp_req), GFP_KERNEL);
+ if (!tfcp_req)
+ return -ENOMEM;
+ inireq->fcpreq = fcpreq;
+ inireq->tfcp_req = tfcp_req;
+ INIT_WORK(&inireq->iniwork, fcloop_tgt_fcprqst_ini_done_work);
tfcp_req->fcpreq = fcpreq;
tfcp_req->tport = rport->targetport->private;
+ spin_lock_init(&tfcp_req->reqlock);
+ INIT_WORK(&tfcp_req->work, fcloop_tgt_fcprqst_done_work);
ret = nvmet_fc_rcv_fcp_req(rport->targetport, &tfcp_req->tgt_fcp_req,
fcpreq->cmdaddr, fcpreq->cmdlen);
@@ -444,63 +477,129 @@
struct nvmefc_tgt_fcp_req *tgt_fcpreq)
{
struct fcloop_fcpreq *tfcp_req = tgt_fcp_req_to_fcpreq(tgt_fcpreq);
- struct nvmefc_fcp_req *fcpreq = tfcp_req->fcpreq;
+ struct nvmefc_fcp_req *fcpreq;
u32 rsplen = 0, xfrlen = 0;
- int fcp_err = 0;
+ int fcp_err = 0, active, aborted;
u8 op = tgt_fcpreq->op;
+ spin_lock(&tfcp_req->reqlock);
+ fcpreq = tfcp_req->fcpreq;
+ active = tfcp_req->active;
+ aborted = tfcp_req->aborted;
+ tfcp_req->active = true;
+ spin_unlock(&tfcp_req->reqlock);
+
+ if (unlikely(active))
+ /* illegal - call while i/o active */
+ return -EALREADY;
+
+ if (unlikely(aborted)) {
+ /* target transport has aborted i/o prior */
+ spin_lock(&tfcp_req->reqlock);
+ tfcp_req->active = false;
+ spin_unlock(&tfcp_req->reqlock);
+ tgt_fcpreq->transferred_length = 0;
+ tgt_fcpreq->fcp_error = -ECANCELED;
+ tgt_fcpreq->done(tgt_fcpreq);
+ return 0;
+ }
+
+ /*
+ * if fcpreq is NULL, the I/O has been aborted (from
+ * initiator side). For the target side, act as if all is well
+ * but don't actually move data.
+ */
+
switch (op) {
case NVMET_FCOP_WRITEDATA:
xfrlen = tgt_fcpreq->transfer_length;
- fcloop_fcp_copy_data(op, tgt_fcpreq->sg, fcpreq->first_sgl,
- tgt_fcpreq->offset, xfrlen);
- fcpreq->transferred_length += xfrlen;
+ if (fcpreq) {
+ fcloop_fcp_copy_data(op, tgt_fcpreq->sg,
+ fcpreq->first_sgl, tgt_fcpreq->offset,
+ xfrlen);
+ fcpreq->transferred_length += xfrlen;
+ }
break;
case NVMET_FCOP_READDATA:
case NVMET_FCOP_READDATA_RSP:
xfrlen = tgt_fcpreq->transfer_length;
- fcloop_fcp_copy_data(op, tgt_fcpreq->sg, fcpreq->first_sgl,
- tgt_fcpreq->offset, xfrlen);
- fcpreq->transferred_length += xfrlen;
+ if (fcpreq) {
+ fcloop_fcp_copy_data(op, tgt_fcpreq->sg,
+ fcpreq->first_sgl, tgt_fcpreq->offset,
+ xfrlen);
+ fcpreq->transferred_length += xfrlen;
+ }
if (op == NVMET_FCOP_READDATA)
break;
/* Fall-Thru to RSP handling */
case NVMET_FCOP_RSP:
- rsplen = ((fcpreq->rsplen < tgt_fcpreq->rsplen) ?
- fcpreq->rsplen : tgt_fcpreq->rsplen);
- memcpy(fcpreq->rspaddr, tgt_fcpreq->rspaddr, rsplen);
- if (rsplen < tgt_fcpreq->rsplen)
- fcp_err = -E2BIG;
- fcpreq->rcv_rsplen = rsplen;
- fcpreq->status = 0;
+ if (fcpreq) {
+ rsplen = ((fcpreq->rsplen < tgt_fcpreq->rsplen) ?
+ fcpreq->rsplen : tgt_fcpreq->rsplen);
+ memcpy(fcpreq->rspaddr, tgt_fcpreq->rspaddr, rsplen);
+ if (rsplen < tgt_fcpreq->rsplen)
+ fcp_err = -E2BIG;
+ fcpreq->rcv_rsplen = rsplen;
+ fcpreq->status = 0;
+ }
tfcp_req->status = 0;
break;
- case NVMET_FCOP_ABORT:
- tfcp_req->status = NVME_SC_FC_TRANSPORT_ABORTED;
- break;
-
default:
fcp_err = -EINVAL;
break;
}
+ spin_lock(&tfcp_req->reqlock);
+ tfcp_req->active = false;
+ spin_unlock(&tfcp_req->reqlock);
+
tgt_fcpreq->transferred_length = xfrlen;
tgt_fcpreq->fcp_error = fcp_err;
tgt_fcpreq->done(tgt_fcpreq);
- if ((!fcp_err) && (op == NVMET_FCOP_RSP ||
- op == NVMET_FCOP_READDATA_RSP ||
- op == NVMET_FCOP_ABORT))
- schedule_work(&tfcp_req->work);
-
return 0;
}
static void
+fcloop_tgt_fcp_abort(struct nvmet_fc_target_port *tgtport,
+ struct nvmefc_tgt_fcp_req *tgt_fcpreq)
+{
+ struct fcloop_fcpreq *tfcp_req = tgt_fcp_req_to_fcpreq(tgt_fcpreq);
+ int active;
+
+ /*
+ * mark aborted only in case there were 2 threads in transport
+ * (one doing io, other doing abort) and only kills ops posted
+ * after the abort request
+ */
+ spin_lock(&tfcp_req->reqlock);
+ active = tfcp_req->active;
+ tfcp_req->aborted = true;
+ spin_unlock(&tfcp_req->reqlock);
+
+ tfcp_req->status = NVME_SC_FC_TRANSPORT_ABORTED;
+
+ /*
+ * nothing more to do. If io wasn't active, the transport should
+ * immediately call the req_release. If it was active, the op
+ * will complete, and the lldd should call req_release.
+ */
+}
+
+static void
+fcloop_fcp_req_release(struct nvmet_fc_target_port *tgtport,
+ struct nvmefc_tgt_fcp_req *tgt_fcpreq)
+{
+ struct fcloop_fcpreq *tfcp_req = tgt_fcp_req_to_fcpreq(tgt_fcpreq);
+
+ schedule_work(&tfcp_req->work);
+}
+
+static void
fcloop_ls_abort(struct nvme_fc_local_port *localport,
struct nvme_fc_remote_port *remoteport,
struct nvmefc_ls_req *lsreq)
@@ -513,6 +612,27 @@
void *hw_queue_handle,
struct nvmefc_fcp_req *fcpreq)
{
+ struct fcloop_rport *rport = remoteport->private;
+ struct fcloop_ini_fcpreq *inireq = fcpreq->private;
+ struct fcloop_fcpreq *tfcp_req = inireq->tfcp_req;
+
+ if (!tfcp_req)
+ /* abort has already been called */
+ return;
+
+ if (rport->targetport)
+ nvmet_fc_rcv_fcp_abort(rport->targetport,
+ &tfcp_req->tgt_fcp_req);
+
+ /* break initiator/target relationship for io */
+ spin_lock(&tfcp_req->reqlock);
+ inireq->tfcp_req = NULL;
+ tfcp_req->fcpreq = NULL;
+ spin_unlock(&tfcp_req->reqlock);
+
+ /* post the aborted io completion */
+ fcpreq->status = -ECANCELED;
+ schedule_work(&inireq->iniwork);
}
static void
@@ -546,7 +666,7 @@
#define FCLOOP_SGL_SEGS 256
#define FCLOOP_DMABOUND_4G 0xFFFFFFFF
-struct nvme_fc_port_template fctemplate = {
+static struct nvme_fc_port_template fctemplate = {
.localport_delete = fcloop_localport_delete,
.remoteport_delete = fcloop_remoteport_delete,
.create_queue = fcloop_create_queue,
@@ -563,20 +683,23 @@
.local_priv_sz = sizeof(struct fcloop_lport),
.remote_priv_sz = sizeof(struct fcloop_rport),
.lsrqst_priv_sz = sizeof(struct fcloop_lsreq),
- .fcprqst_priv_sz = sizeof(struct fcloop_fcpreq),
+ .fcprqst_priv_sz = sizeof(struct fcloop_ini_fcpreq),
};
-struct nvmet_fc_target_template tgttemplate = {
+static struct nvmet_fc_target_template tgttemplate = {
.targetport_delete = fcloop_targetport_delete,
.xmt_ls_rsp = fcloop_xmt_ls_rsp,
.fcp_op = fcloop_fcp_op,
+ .fcp_abort = fcloop_tgt_fcp_abort,
+ .fcp_req_release = fcloop_fcp_req_release,
.max_hw_queues = FCLOOP_HW_QUEUES,
.max_sgl_segments = FCLOOP_SGL_SEGS,
.max_dif_sgl_segments = FCLOOP_SGL_SEGS,
.dma_boundary = FCLOOP_DMABOUND_4G,
/* optional features */
- .target_features = NVMET_FCTGTFEAT_READDATA_RSP |
- NVMET_FCTGTFEAT_NEEDS_CMD_CPUSCHED,
+ .target_features = NVMET_FCTGTFEAT_CMD_IN_ISR |
+ NVMET_FCTGTFEAT_NEEDS_CMD_CPUSCHED |
+ NVMET_FCTGTFEAT_OPDONE_IN_ISR,
/* sizes of additional private data for data structures */
.target_priv_sz = sizeof(struct fcloop_tport),
};
diff --git a/drivers/nvme/target/io-cmd.c b/drivers/nvme/target/io-cmd.c
index 6b0baa9..c77940d 100644
--- a/drivers/nvme/target/io-cmd.c
+++ b/drivers/nvme/target/io-cmd.c
@@ -184,7 +184,7 @@
(req->ns->blksize_shift - 9)) + 1;
if (__blkdev_issue_zeroout(req->ns->bdev, sector, nr_sector,
- GFP_KERNEL, &bio, true))
+ GFP_KERNEL, &bio, 0))
status = NVME_SC_INTERNAL | NVME_SC_DNR;
if (bio) {
@@ -196,26 +196,19 @@
}
}
-int nvmet_parse_io_cmd(struct nvmet_req *req)
+u16 nvmet_parse_io_cmd(struct nvmet_req *req)
{
struct nvme_command *cmd = req->cmd;
+ u16 ret;
- if (unlikely(!(req->sq->ctrl->cc & NVME_CC_ENABLE))) {
- pr_err("nvmet: got io cmd %d while CC.EN == 0\n",
- cmd->common.opcode);
+ ret = nvmet_check_ctrl_status(req, cmd);
+ if (unlikely(ret)) {
req->ns = NULL;
- return NVME_SC_CMD_SEQ_ERROR | NVME_SC_DNR;
- }
-
- if (unlikely(!(req->sq->ctrl->csts & NVME_CSTS_RDY))) {
- pr_err("nvmet: got io cmd %d while CSTS.RDY == 0\n",
- cmd->common.opcode);
- req->ns = NULL;
- return NVME_SC_CMD_SEQ_ERROR | NVME_SC_DNR;
+ return ret;
}
req->ns = nvmet_find_namespace(req->sq->ctrl, cmd->rw.nsid);
- if (!req->ns)
+ if (unlikely(!req->ns))
return NVME_SC_INVALID_NS | NVME_SC_DNR;
switch (cmd->common.opcode) {
@@ -237,7 +230,8 @@
req->execute = nvmet_execute_write_zeroes;
return 0;
default:
- pr_err("nvmet: unhandled cmd %d\n", cmd->common.opcode);
+ pr_err("unhandled cmd %d on qid %d\n", cmd->common.opcode,
+ req->sq->qid);
return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
}
}
diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index c7b0b6a..304f1c87 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -13,12 +13,10 @@
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/scatterlist.h>
-#include <linux/delay.h>
#include <linux/blk-mq.h>
#include <linux/nvme.h>
#include <linux/module.h>
#include <linux/parser.h>
-#include <linux/t10-pi.h>
#include "nvmet.h"
#include "../host/nvme.h"
#include "../host/fabrics.h"
@@ -93,31 +91,26 @@
static void nvme_loop_complete_rq(struct request *req)
{
struct nvme_loop_iod *iod = blk_mq_rq_to_pdu(req);
- int error = 0;
nvme_cleanup_cmd(req);
sg_free_table_chained(&iod->sg_table, true);
+ nvme_complete_rq(req);
+}
- if (unlikely(req->errors)) {
- if (nvme_req_needs_retry(req, req->errors)) {
- nvme_requeue_req(req);
- return;
- }
+static struct blk_mq_tags *nvme_loop_tagset(struct nvme_loop_queue *queue)
+{
+ u32 queue_idx = nvme_loop_queue_idx(queue);
- if (blk_rq_is_passthrough(req))
- error = req->errors;
- else
- error = nvme_error_status(req->errors);
- }
-
- blk_mq_end_request(req, error);
+ if (queue_idx == 0)
+ return queue->ctrl->admin_tag_set.tags[queue_idx];
+ return queue->ctrl->tag_set.tags[queue_idx - 1];
}
static void nvme_loop_queue_response(struct nvmet_req *req)
{
- struct nvme_loop_iod *iod =
- container_of(req, struct nvme_loop_iod, req);
- struct nvme_completion *cqe = &iod->rsp;
+ struct nvme_loop_queue *queue =
+ container_of(req->sq, struct nvme_loop_queue, nvme_sq);
+ struct nvme_completion *cqe = req->rsp;
/*
* AEN requests are special as they don't time out and can
@@ -125,15 +118,22 @@
* aborts. We don't even bother to allocate a struct request
* for them but rather special case them here.
*/
- if (unlikely(nvme_loop_queue_idx(iod->queue) == 0 &&
+ if (unlikely(nvme_loop_queue_idx(queue) == 0 &&
cqe->command_id >= NVME_LOOP_AQ_BLKMQ_DEPTH)) {
- nvme_complete_async_event(&iod->queue->ctrl->ctrl, cqe->status,
+ nvme_complete_async_event(&queue->ctrl->ctrl, cqe->status,
&cqe->result);
} else {
- struct request *rq = blk_mq_rq_from_pdu(iod);
+ struct request *rq;
- iod->nvme_req.result = cqe->result;
- blk_mq_complete_request(rq, le16_to_cpu(cqe->status) >> 1);
+ rq = blk_mq_tag_to_rq(nvme_loop_tagset(queue), cqe->command_id);
+ if (!rq) {
+ dev_err(queue->ctrl->ctrl.device,
+ "tag 0x%x on queue %d not found\n",
+ cqe->command_id, nvme_loop_queue_idx(queue));
+ return;
+ }
+
+ nvme_end_request(rq, cqe->status, cqe->result);
}
}
@@ -154,7 +154,7 @@
schedule_work(&iod->queue->ctrl->reset_work);
/* fail with DNR on admin cmd timeout */
- rq->errors = NVME_SC_ABORT_REQ | NVME_SC_DNR;
+ nvme_req(rq)->status = NVME_SC_ABORT_REQ | NVME_SC_DNR;
return BLK_EH_HANDLED;
}
@@ -268,7 +268,7 @@
return 0;
}
-static struct blk_mq_ops nvme_loop_mq_ops = {
+static const struct blk_mq_ops nvme_loop_mq_ops = {
.queue_rq = nvme_loop_queue_rq,
.complete = nvme_loop_complete_rq,
.init_request = nvme_loop_init_request,
@@ -276,7 +276,7 @@
.timeout = nvme_loop_timeout,
};
-static struct blk_mq_ops nvme_loop_admin_mq_ops = {
+static const struct blk_mq_ops nvme_loop_admin_mq_ops = {
.queue_rq = nvme_loop_queue_rq,
.complete = nvme_loop_complete_rq,
.init_request = nvme_loop_init_admin_request,
@@ -349,6 +349,19 @@
return ret;
}
+static int nvme_loop_connect_io_queues(struct nvme_loop_ctrl *ctrl)
+{
+ int i, ret;
+
+ for (i = 1; i < ctrl->queue_count; i++) {
+ ret = nvmf_connect_io_queue(&ctrl->ctrl, i);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl)
{
int error;
@@ -490,7 +503,7 @@
struct nvme_loop_ctrl *ctrl = container_of(work,
struct nvme_loop_ctrl, reset_work);
bool changed;
- int i, ret;
+ int ret;
nvme_loop_shutdown_ctrl(ctrl);
@@ -502,11 +515,9 @@
if (ret)
goto out_destroy_admin;
- for (i = 1; i < ctrl->queue_count; i++) {
- ret = nvmf_connect_io_queue(&ctrl->ctrl, i);
- if (ret)
- goto out_destroy_io;
- }
+ ret = nvme_loop_connect_io_queues(ctrl);
+ if (ret)
+ goto out_destroy_io;
changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
WARN_ON_ONCE(!changed);
@@ -559,7 +570,7 @@
static int nvme_loop_create_io_queues(struct nvme_loop_ctrl *ctrl)
{
- int ret, i;
+ int ret;
ret = nvme_loop_init_io_queues(ctrl);
if (ret)
@@ -588,11 +599,9 @@
goto out_free_tagset;
}
- for (i = 1; i < ctrl->queue_count; i++) {
- ret = nvmf_connect_io_queue(&ctrl->ctrl, i);
- if (ret)
- goto out_cleanup_connect_q;
- }
+ ret = nvme_loop_connect_io_queues(ctrl);
+ if (ret)
+ goto out_cleanup_connect_q;
return 0;
@@ -736,7 +745,12 @@
ret = nvmet_register_transport(&nvme_loop_ops);
if (ret)
return ret;
- return nvmf_register_transport(&nvme_loop_transport);
+
+ ret = nvmf_register_transport(&nvme_loop_transport);
+ if (ret)
+ nvmet_unregister_transport(&nvme_loop_ops);
+
+ return ret;
}
static void __exit nvme_loop_cleanup_module(void)
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index f7ff15f..7cb77ba 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -253,11 +253,11 @@
u8 log_page;
};
-int nvmet_parse_connect_cmd(struct nvmet_req *req);
-int nvmet_parse_io_cmd(struct nvmet_req *req);
-int nvmet_parse_admin_cmd(struct nvmet_req *req);
-int nvmet_parse_discovery_cmd(struct nvmet_req *req);
-int nvmet_parse_fabrics_cmd(struct nvmet_req *req);
+u16 nvmet_parse_connect_cmd(struct nvmet_req *req);
+u16 nvmet_parse_io_cmd(struct nvmet_req *req);
+u16 nvmet_parse_admin_cmd(struct nvmet_req *req);
+u16 nvmet_parse_discovery_cmd(struct nvmet_req *req);
+u16 nvmet_parse_fabrics_cmd(struct nvmet_req *req);
bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,
struct nvmet_sq *sq, struct nvmet_fabrics_ops *ops);
@@ -278,6 +278,7 @@
u16 nvmet_ctrl_find_get(const char *subsysnqn, const char *hostnqn, u16 cntlid,
struct nvmet_req *req, struct nvmet_ctrl **ret);
void nvmet_ctrl_put(struct nvmet_ctrl *ctrl);
+u16 nvmet_check_ctrl_status(struct nvmet_req *req, struct nvme_command *cmd);
struct nvmet_subsys *nvmet_subsys_alloc(const char *subsysnqn,
enum nvme_subsys_type type);
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index ecc4fe8..99c6901 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -1199,6 +1199,11 @@
}
queue->port = cm_id->context;
+ if (queue->host_qid == 0) {
+ /* Let inflight controller teardown complete */
+ flush_scheduled_work();
+ }
+
ret = nvmet_rdma_cm_accept(cm_id, queue, &event->param.conn);
if (ret)
goto release_queue;
@@ -1427,12 +1432,16 @@
static int nvmet_rdma_add_port(struct nvmet_port *port)
{
struct rdma_cm_id *cm_id;
- struct sockaddr_in addr_in;
- u16 port_in;
+ struct sockaddr_storage addr = { };
+ __kernel_sa_family_t af;
int ret;
switch (port->disc_addr.adrfam) {
case NVMF_ADDR_FAMILY_IP4:
+ af = AF_INET;
+ break;
+ case NVMF_ADDR_FAMILY_IP6:
+ af = AF_INET6;
break;
default:
pr_err("address family %d not supported\n",
@@ -1440,13 +1449,13 @@
return -EINVAL;
}
- ret = kstrtou16(port->disc_addr.trsvcid, 0, &port_in);
- if (ret)
+ ret = inet_pton_with_scope(&init_net, af, port->disc_addr.traddr,
+ port->disc_addr.trsvcid, &addr);
+ if (ret) {
+ pr_err("malformed ip/port passed: %s:%s\n",
+ port->disc_addr.traddr, port->disc_addr.trsvcid);
return ret;
-
- addr_in.sin_family = AF_INET;
- addr_in.sin_addr.s_addr = in_aton(port->disc_addr.traddr);
- addr_in.sin_port = htons(port_in);
+ }
cm_id = rdma_create_id(&init_net, nvmet_rdma_cm_handler, port,
RDMA_PS_TCP, IB_QPT_RC);
@@ -1455,20 +1464,32 @@
return PTR_ERR(cm_id);
}
- ret = rdma_bind_addr(cm_id, (struct sockaddr *)&addr_in);
+ /*
+ * Allow both IPv4 and IPv6 sockets to bind a single port
+ * at the same time.
+ */
+ ret = rdma_set_afonly(cm_id, 1);
if (ret) {
- pr_err("binding CM ID to %pISpc failed (%d)\n", &addr_in, ret);
+ pr_err("rdma_set_afonly failed (%d)\n", ret);
+ goto out_destroy_id;
+ }
+
+ ret = rdma_bind_addr(cm_id, (struct sockaddr *)&addr);
+ if (ret) {
+ pr_err("binding CM ID to %pISpcs failed (%d)\n",
+ (struct sockaddr *)&addr, ret);
goto out_destroy_id;
}
ret = rdma_listen(cm_id, 128);
if (ret) {
- pr_err("listening to %pISpc failed (%d)\n", &addr_in, ret);
+ pr_err("listening to %pISpcs failed (%d)\n",
+ (struct sockaddr *)&addr, ret);
goto out_destroy_id;
}
- pr_info("enabling port %d (%pISpc)\n",
- le16_to_cpu(port->disc_addr.portid), &addr_in);
+ pr_info("enabling port %d (%pISpcs)\n",
+ le16_to_cpu(port->disc_addr.portid), (struct sockaddr *)&addr);
port->priv = cm_id;
return 0;
diff --git a/drivers/of/of_mdio.c b/drivers/of/of_mdio.c
index 0b29798..7e4c80f 100644
--- a/drivers/of/of_mdio.c
+++ b/drivers/of/of_mdio.c
@@ -22,6 +22,8 @@
#include <linux/of_net.h>
#include <linux/module.h>
+#define DEFAULT_GPIO_RESET_DELAY 10 /* in microseconds */
+
MODULE_AUTHOR("Grant Likely <grant.likely@secretlab.ca>");
MODULE_LICENSE("GPL");
@@ -221,6 +223,11 @@
mdio->dev.of_node = np;
+ /* Get bus level PHY reset GPIO details */
+ mdio->reset_delay_us = DEFAULT_GPIO_RESET_DELAY;
+ of_property_read_u32(np, "reset-delay-us", &mdio->reset_delay_us);
+ mdio->num_reset_gpios = of_gpio_named_count(np, "reset-gpios");
+
/* Register the MDIO bus */
rc = mdiobus_register(mdio);
if (rc)
diff --git a/drivers/pci/irq.c b/drivers/pci/irq.c
index 6684f15..f9f2a03 100644
--- a/drivers/pci/irq.c
+++ b/drivers/pci/irq.c
@@ -31,7 +31,7 @@
* driver).
*
* Returns:
- * a suggestion for fixing it (although the driver is not required to
+ * a suggestion for fixing it (although the driver is not required to
* act on this).
*/
enum pci_lost_interrupt_reason pci_lost_interrupt(struct pci_dev *pdev)
diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index d571bc3..0042c36 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -973,27 +973,6 @@
return msix_capability_init(dev, entries, nvec, affd);
}
-/**
- * pci_enable_msix - configure device's MSI-X capability structure
- * @dev: pointer to the pci_dev data structure of MSI-X device function
- * @entries: pointer to an array of MSI-X entries (optional)
- * @nvec: number of MSI-X irqs requested for allocation by device driver
- *
- * Setup the MSI-X capability structure of device function with the number
- * of requested irqs upon its software driver call to request for
- * MSI-X mode enabled on its hardware device function. A return of zero
- * indicates the successful configuration of MSI-X capability structure
- * with new allocated MSI-X irqs. A return of < 0 indicates a failure.
- * Or a return of > 0 indicates that driver request is exceeding the number
- * of irqs or MSI-X vectors available. Driver should use the returned value to
- * re-send its request.
- **/
-int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec)
-{
- return __pci_enable_msix(dev, entries, nvec, NULL);
-}
-EXPORT_SYMBOL(pci_enable_msix);
-
void pci_msix_shutdown(struct pci_dev *dev)
{
struct msi_desc *entry;
diff --git a/drivers/pinctrl/Kconfig b/drivers/pinctrl/Kconfig
index 8f8c2af..37af5e3 100644
--- a/drivers/pinctrl/Kconfig
+++ b/drivers/pinctrl/Kconfig
@@ -41,6 +41,17 @@
future processors. This option is selected automatically when specific
machine and arch are selected to build.
+config PINCTRL_ARTPEC6
+ bool "Axis ARTPEC-6 pin controller driver"
+ depends on MACH_ARTPEC6
+ select PINMUX
+ select GENERIC_PINCONF
+ help
+ This is the driver for the Axis ARTPEC-6 pin controller. This driver
+ supports pin function multiplexing as well as pin bias and drive
+ strength configuration. Device tree integration instructions can be
+ found in Documentation/devicetree/bindings/pinctrl/axis,artpec6-pinctrl.txt
+
config PINCTRL_AS3722
tristate "Pinctrl and GPIO driver for ams AS3722 PMIC"
depends on MFD_AS3722 && GPIOLIB
diff --git a/drivers/pinctrl/Makefile b/drivers/pinctrl/Makefile
index a251f43..0e9b2226 100644
--- a/drivers/pinctrl/Makefile
+++ b/drivers/pinctrl/Makefile
@@ -8,6 +8,7 @@
obj-$(CONFIG_OF) += devicetree.o
obj-$(CONFIG_GENERIC_PINCONF) += pinconf-generic.o
obj-$(CONFIG_PINCTRL_ADI2) += pinctrl-adi2.o
+obj-$(CONFIG_PINCTRL_ARTPEC6) += pinctrl-artpec6.o
obj-$(CONFIG_PINCTRL_AS3722) += pinctrl-as3722.o
obj-$(CONFIG_PINCTRL_BF54x) += pinctrl-adi2-bf54x.o
obj-$(CONFIG_PINCTRL_BF60x) += pinctrl-adi2-bf60x.o
@@ -44,7 +45,7 @@
obj-$(CONFIG_PINCTRL_BERLIN) += berlin/
obj-y += freescale/
obj-$(CONFIG_X86) += intel/
-obj-$(CONFIG_PINCTRL_MVEBU) += mvebu/
+obj-y += mvebu/
obj-y += nomadik/
obj-$(CONFIG_PINCTRL_PXA) += pxa/
obj-$(CONFIG_ARCH_QCOM) += qcom/
diff --git a/drivers/pinctrl/aspeed/pinctrl-aspeed-g4.c b/drivers/pinctrl/aspeed/pinctrl-aspeed-g4.c
index 7de596e..cf3106c 100644
--- a/drivers/pinctrl/aspeed/pinctrl-aspeed-g4.c
+++ b/drivers/pinctrl/aspeed/pinctrl-aspeed-g4.c
@@ -856,8 +856,8 @@
SIG_EXPR_DECL(VPIG3, VPI24, VPI24_DESC, AB1_DESC);
SIG_EXPR_DECL(VPIG3, VPI30, VPI30_DESC, AB1_DESC);
SIG_EXPR_LIST_DECL(VPIG3, SIG_EXPR_PTR(VPIG3, VPI18),
- SIG_EXPR_PTR(VPIG2, VPI24),
- SIG_EXPR_PTR(VPIG2, VPI30));
+ SIG_EXPR_PTR(VPIG3, VPI24),
+ SIG_EXPR_PTR(VPIG3, VPI30));
SIG_EXPR_LIST_DECL_SINGLE(PWM3, PWM3, AB1_DESC);
MS_PIN_DECL(AB1, GPION3, VPIG3, PWM3);
FUNC_GROUP_DECL(PWM3, AB1);
@@ -868,8 +868,8 @@
SIG_EXPR_DECL(VPIG4, VPI24, VPI24_DESC, W5_DESC);
SIG_EXPR_DECL(VPIG4, VPI30, VPI30_DESC, W5_DESC);
SIG_EXPR_LIST_DECL(VPIG4, SIG_EXPR_PTR(VPIG4, VPI18),
- SIG_EXPR_PTR(VPIG2, VPI24),
- SIG_EXPR_PTR(VPIG2, VPI30));
+ SIG_EXPR_PTR(VPIG4, VPI24),
+ SIG_EXPR_PTR(VPIG4, VPI30));
SIG_EXPR_LIST_DECL_SINGLE(PWM4, PWM4, W5_DESC);
MS_PIN_DECL(W5, GPION4, VPIG4, PWM4);
FUNC_GROUP_DECL(PWM4, W5);
@@ -880,8 +880,8 @@
SIG_EXPR_DECL(VPIG5, VPI24, VPI24_DESC, Y4_DESC);
SIG_EXPR_DECL(VPIG5, VPI30, VPI30_DESC, Y4_DESC);
SIG_EXPR_LIST_DECL(VPIG5, SIG_EXPR_PTR(VPIG5, VPI18),
- SIG_EXPR_PTR(VPIG2, VPI24),
- SIG_EXPR_PTR(VPIG2, VPI30));
+ SIG_EXPR_PTR(VPIG5, VPI24),
+ SIG_EXPR_PTR(VPIG5, VPI30));
SIG_EXPR_LIST_DECL_SINGLE(PWM5, PWM5, Y4_DESC);
MS_PIN_DECL(Y4, GPION5, VPIG5, PWM5);
FUNC_GROUP_DECL(PWM5, Y4);
@@ -2234,6 +2234,110 @@
ASPEED_PINCTRL_FUNC(WDTRST2),
};
+static const struct aspeed_pin_config aspeed_g4_configs[] = {
+ /* GPIO banks ranges [A, B], [D, J], [M, R] */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { D6, D5 }, SCU8C, 16 },
+ { PIN_CONFIG_BIAS_DISABLE, { D6, D5 }, SCU8C, 16 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { J21, E18 }, SCU8C, 17 },
+ { PIN_CONFIG_BIAS_DISABLE, { J21, E18 }, SCU8C, 17 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { A18, E15 }, SCU8C, 19 },
+ { PIN_CONFIG_BIAS_DISABLE, { A18, E15 }, SCU8C, 19 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { D15, B14 }, SCU8C, 20 },
+ { PIN_CONFIG_BIAS_DISABLE, { D15, B14 }, SCU8C, 20 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { D18, C17 }, SCU8C, 21 },
+ { PIN_CONFIG_BIAS_DISABLE, { D18, C17 }, SCU8C, 21 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { A14, U18 }, SCU8C, 22 },
+ { PIN_CONFIG_BIAS_DISABLE, { A14, U18 }, SCU8C, 22 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { A8, E7 }, SCU8C, 23 },
+ { PIN_CONFIG_BIAS_DISABLE, { A8, E7 }, SCU8C, 23 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { C22, E20 }, SCU8C, 24 },
+ { PIN_CONFIG_BIAS_DISABLE, { C22, E20 }, SCU8C, 24 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { J5, T1 }, SCU8C, 25 },
+ { PIN_CONFIG_BIAS_DISABLE, { J5, T1 }, SCU8C, 25 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { U1, U5 }, SCU8C, 26 },
+ { PIN_CONFIG_BIAS_DISABLE, { U1, U5 }, SCU8C, 26 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { V3, V5 }, SCU8C, 27 },
+ { PIN_CONFIG_BIAS_DISABLE, { V3, V5 }, SCU8C, 27 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { W4, AB2 }, SCU8C, 28 },
+ { PIN_CONFIG_BIAS_DISABLE, { W4, AB2 }, SCU8C, 28 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { V6, V7 }, SCU8C, 29 },
+ { PIN_CONFIG_BIAS_DISABLE, { V6, V7 }, SCU8C, 29 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { Y6, AB7 }, SCU8C, 30 },
+ { PIN_CONFIG_BIAS_DISABLE, { Y6, AB7 }, SCU8C, 30 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { V20, A5 }, SCU8C, 31 },
+ { PIN_CONFIG_BIAS_DISABLE, { V20, A5 }, SCU8C, 31 },
+
+ /* GPIOs T[0-5] (RGMII1 Tx pins) */
+ { PIN_CONFIG_DRIVE_STRENGTH, { A12, A13 }, SCU90, 9 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { A12, A13 }, SCU90, 12 },
+ { PIN_CONFIG_BIAS_DISABLE, { A12, A13 }, SCU90, 12 },
+
+ /* GPIOs T[6-7], U[0-3] (RGMII2 TX pins) */
+ { PIN_CONFIG_DRIVE_STRENGTH, { D9, D10 }, SCU90, 11 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { D9, D10 }, SCU90, 14 },
+ { PIN_CONFIG_BIAS_DISABLE, { D9, D10 }, SCU90, 14 },
+
+ /* GPIOs U[4-7], V[0-1] (RGMII1 Rx pins) */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { E11, E10 }, SCU90, 13 },
+ { PIN_CONFIG_BIAS_DISABLE, { E11, E10 }, SCU90, 13 },
+
+ /* GPIOs V[2-7] (RGMII2 Rx pins) */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { C9, C8 }, SCU90, 15 },
+ { PIN_CONFIG_BIAS_DISABLE, { C9, C8 }, SCU90, 15 },
+
+ /* ADC pull-downs (SCUA8[19:4]) */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { L5, L5 }, SCUA8, 4 },
+ { PIN_CONFIG_BIAS_DISABLE, { L5, L5 }, SCUA8, 4 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { L4, L4 }, SCUA8, 5 },
+ { PIN_CONFIG_BIAS_DISABLE, { L4, L4 }, SCUA8, 5 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { L3, L3 }, SCUA8, 6 },
+ { PIN_CONFIG_BIAS_DISABLE, { L3, L3 }, SCUA8, 6 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { L2, L2 }, SCUA8, 7 },
+ { PIN_CONFIG_BIAS_DISABLE, { L2, L2 }, SCUA8, 7 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { L1, L1 }, SCUA8, 8 },
+ { PIN_CONFIG_BIAS_DISABLE, { L1, L1 }, SCUA8, 8 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { M5, M5 }, SCUA8, 9 },
+ { PIN_CONFIG_BIAS_DISABLE, { M5, M5 }, SCUA8, 9 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { M4, M4 }, SCUA8, 10 },
+ { PIN_CONFIG_BIAS_DISABLE, { M4, M4 }, SCUA8, 10 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { M3, M3 }, SCUA8, 11 },
+ { PIN_CONFIG_BIAS_DISABLE, { M3, M3 }, SCUA8, 11 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { M2, M2 }, SCUA8, 12 },
+ { PIN_CONFIG_BIAS_DISABLE, { M2, M2 }, SCUA8, 12 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { M1, M1 }, SCUA8, 13 },
+ { PIN_CONFIG_BIAS_DISABLE, { M1, M1 }, SCUA8, 13 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { N5, N5 }, SCUA8, 14 },
+ { PIN_CONFIG_BIAS_DISABLE, { N5, N5 }, SCUA8, 14 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { N4, N4 }, SCUA8, 15 },
+ { PIN_CONFIG_BIAS_DISABLE, { N4, N4 }, SCUA8, 15 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { N3, N3 }, SCUA8, 16 },
+ { PIN_CONFIG_BIAS_DISABLE, { N3, N3 }, SCUA8, 16 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { N2, N2 }, SCUA8, 17 },
+ { PIN_CONFIG_BIAS_DISABLE, { N2, N2 }, SCUA8, 17 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { N1, N1 }, SCUA8, 18 },
+ { PIN_CONFIG_BIAS_DISABLE, { N1, N1 }, SCUA8, 18 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { P5, P5 }, SCUA8, 19 },
+ { PIN_CONFIG_BIAS_DISABLE, { P5, P5 }, SCUA8, 19 },
+
+ /*
+ * Debounce settings for GPIOs D and E passthrough mode are in
+ * SCUA8[27:20] and so are managed by pinctrl. Normal GPIO debounce for
+ * banks D and E is handled by the GPIO driver - GPIO passthrough is
+ * treated like any other non-GPIO mux function. There is a catch
+ * however, in that the debounce period is configured in the GPIO
+ * controller. Due to this tangle between GPIO and pinctrl we don't yet
+ * fully support pass-through debounce.
+ */
+ { PIN_CONFIG_INPUT_DEBOUNCE, { A18, D16 }, SCUA8, 20 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { B17, A17 }, SCUA8, 21 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { C16, B16 }, SCUA8, 22 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { A16, E15 }, SCUA8, 23 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { D15, C15 }, SCUA8, 24 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { B15, A15 }, SCUA8, 25 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { E14, D14 }, SCUA8, 26 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { C14, B14 }, SCUA8, 27 },
+};
+
static struct aspeed_pinctrl_data aspeed_g4_pinctrl_data = {
.pins = aspeed_g4_pins,
.npins = ARRAY_SIZE(aspeed_g4_pins),
@@ -2241,6 +2345,8 @@
.ngroups = ARRAY_SIZE(aspeed_g4_groups),
.functions = aspeed_g4_functions,
.nfunctions = ARRAY_SIZE(aspeed_g4_functions),
+ .configs = aspeed_g4_configs,
+ .nconfigs = ARRAY_SIZE(aspeed_g4_configs),
};
static struct pinmux_ops aspeed_g4_pinmux_ops = {
@@ -2257,16 +2363,25 @@
.get_group_name = aspeed_pinctrl_get_group_name,
.get_group_pins = aspeed_pinctrl_get_group_pins,
.pin_dbg_show = aspeed_pinctrl_pin_dbg_show,
- .dt_node_to_map = pinconf_generic_dt_node_to_map_pin,
+ .dt_node_to_map = pinconf_generic_dt_node_to_map_all,
.dt_free_map = pinctrl_utils_free_map,
};
+static const struct pinconf_ops aspeed_g4_conf_ops = {
+ .is_generic = true,
+ .pin_config_get = aspeed_pin_config_get,
+ .pin_config_set = aspeed_pin_config_set,
+ .pin_config_group_get = aspeed_pin_config_group_get,
+ .pin_config_group_set = aspeed_pin_config_group_set,
+};
+
static struct pinctrl_desc aspeed_g4_pinctrl_desc = {
.name = "aspeed-g4-pinctrl",
.pins = aspeed_g4_pins,
.npins = ARRAY_SIZE(aspeed_g4_pins),
.pctlops = &aspeed_g4_pinctrl_ops,
.pmxops = &aspeed_g4_pinmux_ops,
+ .confops = &aspeed_g4_conf_ops,
};
static int aspeed_g4_pinctrl_probe(struct platform_device *pdev)
diff --git a/drivers/pinctrl/aspeed/pinctrl-aspeed-g5.c b/drivers/pinctrl/aspeed/pinctrl-aspeed-g5.c
index 43221a3..68aa046 100644
--- a/drivers/pinctrl/aspeed/pinctrl-aspeed-g5.c
+++ b/drivers/pinctrl/aspeed/pinctrl-aspeed-g5.c
@@ -2285,6 +2285,146 @@
ASPEED_PINCTRL_FUNC(WDTRST2),
};
+static struct aspeed_pin_config aspeed_g5_configs[] = {
+ /* GPIOA, GPIOQ */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { B14, B13 }, SCU8C, 16 },
+ { PIN_CONFIG_BIAS_DISABLE, { B14, B13 }, SCU8C, 16 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { A11, N20 }, SCU8C, 16 },
+ { PIN_CONFIG_BIAS_DISABLE, { A11, N20 }, SCU8C, 16 },
+
+ /* GPIOB, GPIOR */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { K19, H20 }, SCU8C, 17 },
+ { PIN_CONFIG_BIAS_DISABLE, { K19, H20 }, SCU8C, 17 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { AA19, E10 }, SCU8C, 17 },
+ { PIN_CONFIG_BIAS_DISABLE, { AA19, E10 }, SCU8C, 17 },
+
+ /* GPIOC, GPIOS*/
+ { PIN_CONFIG_BIAS_PULL_DOWN, { C12, B11 }, SCU8C, 18 },
+ { PIN_CONFIG_BIAS_DISABLE, { C12, B11 }, SCU8C, 18 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { V20, AA20 }, SCU8C, 18 },
+ { PIN_CONFIG_BIAS_DISABLE, { V20, AA20 }, SCU8C, 18 },
+
+ /* GPIOD, GPIOY */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { F19, C21 }, SCU8C, 19 },
+ { PIN_CONFIG_BIAS_DISABLE, { F19, C21 }, SCU8C, 19 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { R22, P20 }, SCU8C, 19 },
+ { PIN_CONFIG_BIAS_DISABLE, { R22, P20 }, SCU8C, 19 },
+
+ /* GPIOE, GPIOZ */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { B20, B19 }, SCU8C, 20 },
+ { PIN_CONFIG_BIAS_DISABLE, { B20, B19 }, SCU8C, 20 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { Y20, W21 }, SCU8C, 20 },
+ { PIN_CONFIG_BIAS_DISABLE, { Y20, W21 }, SCU8C, 20 },
+
+ /* GPIOF, GPIOAA */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { J19, H18 }, SCU8C, 21 },
+ { PIN_CONFIG_BIAS_DISABLE, { J19, H18 }, SCU8C, 21 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { Y21, P19 }, SCU8C, 21 },
+ { PIN_CONFIG_BIAS_DISABLE, { Y21, P19 }, SCU8C, 21 },
+
+ /* GPIOG, GPIOAB */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { A19, E14 }, SCU8C, 22 },
+ { PIN_CONFIG_BIAS_DISABLE, { A19, E14 }, SCU8C, 22 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { N19, R20 }, SCU8C, 22 },
+ { PIN_CONFIG_BIAS_DISABLE, { N19, R20 }, SCU8C, 22 },
+
+ /* GPIOH, GPIOAC */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { A18, D18 }, SCU8C, 23 },
+ { PIN_CONFIG_BIAS_DISABLE, { A18, D18 }, SCU8C, 23 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { G21, G22 }, SCU8C, 23 },
+ { PIN_CONFIG_BIAS_DISABLE, { G21, G22 }, SCU8C, 23 },
+
+ /* GPIOs [I, P] */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { C18, A15 }, SCU8C, 24 },
+ { PIN_CONFIG_BIAS_DISABLE, { C18, A15 }, SCU8C, 24 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { R2, T3 }, SCU8C, 25 },
+ { PIN_CONFIG_BIAS_DISABLE, { R2, T3 }, SCU8C, 25 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { L3, R1 }, SCU8C, 26 },
+ { PIN_CONFIG_BIAS_DISABLE, { L3, R1 }, SCU8C, 26 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { T2, W1 }, SCU8C, 27 },
+ { PIN_CONFIG_BIAS_DISABLE, { T2, W1 }, SCU8C, 27 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { Y1, T5 }, SCU8C, 28 },
+ { PIN_CONFIG_BIAS_DISABLE, { Y1, T5 }, SCU8C, 28 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { V2, T4 }, SCU8C, 29 },
+ { PIN_CONFIG_BIAS_DISABLE, { V2, T4 }, SCU8C, 29 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { U5, W4 }, SCU8C, 30 },
+ { PIN_CONFIG_BIAS_DISABLE, { U5, W4 }, SCU8C, 30 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { V4, V6 }, SCU8C, 31 },
+ { PIN_CONFIG_BIAS_DISABLE, { V4, V6 }, SCU8C, 31 },
+
+ /* GPIOs T[0-5] (RGMII1 Tx pins) */
+ { PIN_CONFIG_DRIVE_STRENGTH, { B5, B5 }, SCU90, 8 },
+ { PIN_CONFIG_DRIVE_STRENGTH, { E9, A5 }, SCU90, 9 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { B5, D7 }, SCU90, 12 },
+ { PIN_CONFIG_BIAS_DISABLE, { B5, D7 }, SCU90, 12 },
+
+ /* GPIOs T[6-7], U[0-3] (RGMII2 TX pins) */
+ { PIN_CONFIG_DRIVE_STRENGTH, { B2, B2 }, SCU90, 10 },
+ { PIN_CONFIG_DRIVE_STRENGTH, { B1, B3 }, SCU90, 11 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { B2, D4 }, SCU90, 14 },
+ { PIN_CONFIG_BIAS_DISABLE, { B2, D4 }, SCU90, 14 },
+
+ /* GPIOs U[4-7], V[0-1] (RGMII1 Rx pins) */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { B4, C4 }, SCU90, 13 },
+ { PIN_CONFIG_BIAS_DISABLE, { B4, C4 }, SCU90, 13 },
+
+ /* GPIOs V[2-7] (RGMII2 Rx pins) */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { C2, E6 }, SCU90, 15 },
+ { PIN_CONFIG_BIAS_DISABLE, { C2, E6 }, SCU90, 15 },
+
+ /* ADC pull-downs (SCUA8[19:4]) */
+ { PIN_CONFIG_BIAS_PULL_DOWN, { F4, F4 }, SCUA8, 4 },
+ { PIN_CONFIG_BIAS_DISABLE, { F4, F4 }, SCUA8, 4 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { F5, F5 }, SCUA8, 5 },
+ { PIN_CONFIG_BIAS_DISABLE, { F5, F5 }, SCUA8, 5 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { E2, E2 }, SCUA8, 6 },
+ { PIN_CONFIG_BIAS_DISABLE, { E2, E2 }, SCUA8, 6 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { E1, E1 }, SCUA8, 7 },
+ { PIN_CONFIG_BIAS_DISABLE, { E1, E1 }, SCUA8, 7 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { F3, F3 }, SCUA8, 8 },
+ { PIN_CONFIG_BIAS_DISABLE, { F3, F3 }, SCUA8, 8 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { E3, E3 }, SCUA8, 9 },
+ { PIN_CONFIG_BIAS_DISABLE, { E3, E3 }, SCUA8, 9 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { G5, G5 }, SCUA8, 10 },
+ { PIN_CONFIG_BIAS_DISABLE, { G5, G5 }, SCUA8, 10 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { G4, G4 }, SCUA8, 11 },
+ { PIN_CONFIG_BIAS_DISABLE, { G4, G4 }, SCUA8, 11 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { F2, F2 }, SCUA8, 12 },
+ { PIN_CONFIG_BIAS_DISABLE, { F2, F2 }, SCUA8, 12 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { G3, G3 }, SCUA8, 13 },
+ { PIN_CONFIG_BIAS_DISABLE, { G3, G3 }, SCUA8, 13 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { G2, G2 }, SCUA8, 14 },
+ { PIN_CONFIG_BIAS_DISABLE, { G2, G2 }, SCUA8, 14 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { F1, F1 }, SCUA8, 15 },
+ { PIN_CONFIG_BIAS_DISABLE, { F1, F1 }, SCUA8, 15 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { H5, H5 }, SCUA8, 16 },
+ { PIN_CONFIG_BIAS_DISABLE, { H5, H5 }, SCUA8, 16 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { G1, G1 }, SCUA8, 17 },
+ { PIN_CONFIG_BIAS_DISABLE, { G1, G1 }, SCUA8, 17 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { H3, H3 }, SCUA8, 18 },
+ { PIN_CONFIG_BIAS_DISABLE, { H3, H3 }, SCUA8, 18 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, { H4, H4 }, SCUA8, 19 },
+ { PIN_CONFIG_BIAS_DISABLE, { H4, H4 }, SCUA8, 19 },
+
+ /*
+ * Debounce settings for GPIOs D and E passthrough mode are in
+ * SCUA8[27:20] and so are managed by pinctrl. Normal GPIO debounce for
+ * banks D and E is handled by the GPIO driver - GPIO passthrough is
+ * treated like any other non-GPIO mux function. There is a catch
+ * however, in that the debounce period is configured in the GPIO
+ * controller. Due to this tangle between GPIO and pinctrl we don't yet
+ * fully support pass-through debounce.
+ */
+ { PIN_CONFIG_INPUT_DEBOUNCE, { F19, E21 }, SCUA8, 20 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { F20, D20 }, SCUA8, 21 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { D21, E20 }, SCUA8, 22 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { G18, C21 }, SCUA8, 23 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { B20, C20 }, SCUA8, 24 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { F18, F17 }, SCUA8, 25 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { E18, D19 }, SCUA8, 26 },
+ { PIN_CONFIG_INPUT_DEBOUNCE, { A20, B19 }, SCUA8, 27 },
+};
+
static struct aspeed_pinctrl_data aspeed_g5_pinctrl_data = {
.pins = aspeed_g5_pins,
.npins = ARRAY_SIZE(aspeed_g5_pins),
@@ -2292,6 +2432,8 @@
.ngroups = ARRAY_SIZE(aspeed_g5_groups),
.functions = aspeed_g5_functions,
.nfunctions = ARRAY_SIZE(aspeed_g5_functions),
+ .configs = aspeed_g5_configs,
+ .nconfigs = ARRAY_SIZE(aspeed_g5_configs),
};
static struct pinmux_ops aspeed_g5_pinmux_ops = {
@@ -2308,16 +2450,25 @@
.get_group_name = aspeed_pinctrl_get_group_name,
.get_group_pins = aspeed_pinctrl_get_group_pins,
.pin_dbg_show = aspeed_pinctrl_pin_dbg_show,
- .dt_node_to_map = pinconf_generic_dt_node_to_map_pin,
+ .dt_node_to_map = pinconf_generic_dt_node_to_map_all,
.dt_free_map = pinctrl_utils_free_map,
};
+static struct pinconf_ops aspeed_g5_conf_ops = {
+ .is_generic = true,
+ .pin_config_get = aspeed_pin_config_get,
+ .pin_config_set = aspeed_pin_config_set,
+ .pin_config_group_get = aspeed_pin_config_group_get,
+ .pin_config_group_set = aspeed_pin_config_group_set,
+};
+
static struct pinctrl_desc aspeed_g5_pinctrl_desc = {
.name = "aspeed-g5-pinctrl",
.pins = aspeed_g5_pins,
.npins = ARRAY_SIZE(aspeed_g5_pins),
.pctlops = &aspeed_g5_pinctrl_ops,
.pmxops = &aspeed_g5_pinmux_ops,
+ .confops = &aspeed_g5_conf_ops,
};
static int aspeed_g5_pinctrl_probe(struct platform_device *pdev)
diff --git a/drivers/pinctrl/aspeed/pinctrl-aspeed.c b/drivers/pinctrl/aspeed/pinctrl-aspeed.c
index 76f62bd..a86a4d6 100644
--- a/drivers/pinctrl/aspeed/pinctrl-aspeed.c
+++ b/drivers/pinctrl/aspeed/pinctrl-aspeed.c
@@ -198,9 +198,19 @@
* them. This may mean that certain functions cannot be
* deconfigured and is the reason we re-evaluate after writing
* all descriptor bits.
+ *
+ * Port D and port E GPIO loopback modes are the only exception
+ * as those are commonly used with front-panel buttons to allow
+ * normal operation of the host when the BMC is powered off or
+ * fails to boot. Once the BMC has booted, the loopback mode
+ * must be disabled for the BMC to control host power-on and
+ * reset.
*/
- if ((desc->reg == HW_STRAP1 || desc->reg == HW_STRAP2) &&
- desc->ip == ASPEED_IP_SCU)
+ if (desc->ip == ASPEED_IP_SCU && desc->reg == HW_STRAP1 &&
+ !(desc->mask & (BIT(21) | BIT(22))))
+ continue;
+
+ if (desc->ip == ASPEED_IP_SCU && desc->reg == HW_STRAP2)
continue;
ret = regmap_update_bits(maps[desc->ip], desc->reg,
@@ -537,3 +547,214 @@
return 0;
}
+
+static inline bool pin_in_config_range(unsigned int offset,
+ const struct aspeed_pin_config *config)
+{
+ return offset >= config->pins[0] && offset <= config->pins[1];
+}
+
+static inline const struct aspeed_pin_config *find_pinconf_config(
+ const struct aspeed_pinctrl_data *pdata,
+ unsigned int offset,
+ enum pin_config_param param)
+{
+ unsigned int i;
+
+ for (i = 0; i < pdata->nconfigs; i++) {
+ if (param == pdata->configs[i].param &&
+ pin_in_config_range(offset, &pdata->configs[i]))
+ return &pdata->configs[i];
+ }
+
+ return NULL;
+}
+
+/**
+ * @param: pinconf configuration parameter
+ * @arg: The supported argument for @param, or -1 if any value is supported
+ * @value: The register value to write to configure @arg for @param
+ *
+ * The map is to be used in conjunction with the configuration array supplied
+ * by the driver implementation.
+ */
+struct aspeed_pin_config_map {
+ enum pin_config_param param;
+ s32 arg;
+ u32 val;
+};
+
+enum aspeed_pin_config_map_type { MAP_TYPE_ARG, MAP_TYPE_VAL };
+
+/* Aspeed consistently both:
+ *
+ * 1. Defines "disable bits" for internal pull-downs
+ * 2. Uses 8mA or 16mA drive strengths
+ */
+static const struct aspeed_pin_config_map pin_config_map[] = {
+ { PIN_CONFIG_BIAS_PULL_DOWN, 0, 1 },
+ { PIN_CONFIG_BIAS_PULL_DOWN, -1, 0 },
+ { PIN_CONFIG_BIAS_DISABLE, -1, 1 },
+ { PIN_CONFIG_DRIVE_STRENGTH, 8, 0 },
+ { PIN_CONFIG_DRIVE_STRENGTH, 16, 1 },
+};
+
+static const struct aspeed_pin_config_map *find_pinconf_map(
+ enum pin_config_param param,
+ enum aspeed_pin_config_map_type type,
+ s64 value)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(pin_config_map); i++) {
+ const struct aspeed_pin_config_map *elem;
+ bool match;
+
+ elem = &pin_config_map[i];
+
+ switch (type) {
+ case MAP_TYPE_ARG:
+ match = (elem->arg == -1 || elem->arg == value);
+ break;
+ case MAP_TYPE_VAL:
+ match = (elem->val == value);
+ break;
+ }
+
+ if (param == elem->param && match)
+ return elem;
+ }
+
+ return NULL;
+}
+
+int aspeed_pin_config_get(struct pinctrl_dev *pctldev, unsigned int offset,
+ unsigned long *config)
+{
+ const enum pin_config_param param = pinconf_to_config_param(*config);
+ const struct aspeed_pin_config_map *pmap;
+ const struct aspeed_pinctrl_data *pdata;
+ const struct aspeed_pin_config *pconf;
+ unsigned int val;
+ int rc = 0;
+ u32 arg;
+
+ pdata = pinctrl_dev_get_drvdata(pctldev);
+ pconf = find_pinconf_config(pdata, offset, param);
+ if (!pconf)
+ return -ENOTSUPP;
+
+ rc = regmap_read(pdata->maps[ASPEED_IP_SCU], pconf->reg, &val);
+ if (rc < 0)
+ return rc;
+
+ pmap = find_pinconf_map(param, MAP_TYPE_VAL,
+ (val & BIT(pconf->bit)) >> pconf->bit);
+
+ if (!pmap)
+ return -EINVAL;
+
+ if (param == PIN_CONFIG_DRIVE_STRENGTH)
+ arg = (u32) pmap->arg;
+ else if (param == PIN_CONFIG_BIAS_PULL_DOWN)
+ arg = !!pmap->arg;
+ else
+ arg = 1;
+
+ if (!arg)
+ return -EINVAL;
+
+ *config = pinconf_to_config_packed(param, arg);
+
+ return 0;
+}
+
+int aspeed_pin_config_set(struct pinctrl_dev *pctldev, unsigned int offset,
+ unsigned long *configs, unsigned int num_configs)
+{
+ const struct aspeed_pinctrl_data *pdata;
+ unsigned int i;
+ int rc = 0;
+
+ pdata = pinctrl_dev_get_drvdata(pctldev);
+
+ for (i = 0; i < num_configs; i++) {
+ const struct aspeed_pin_config_map *pmap;
+ const struct aspeed_pin_config *pconf;
+ enum pin_config_param param;
+ unsigned int val;
+ u32 arg;
+
+ param = pinconf_to_config_param(configs[i]);
+ arg = pinconf_to_config_argument(configs[i]);
+
+ pconf = find_pinconf_config(pdata, offset, param);
+ if (!pconf)
+ return -ENOTSUPP;
+
+ pmap = find_pinconf_map(param, MAP_TYPE_ARG, arg);
+
+ if (unlikely(WARN_ON(!pmap)))
+ return -EINVAL;
+
+ val = pmap->val << pconf->bit;
+
+ rc = regmap_update_bits(pdata->maps[ASPEED_IP_SCU], pconf->reg,
+ BIT(pconf->bit), val);
+
+ if (rc < 0)
+ return rc;
+
+ pr_debug("%s: Set SCU%02X[%d]=%d for param %d(=%d) on pin %d\n",
+ __func__, pconf->reg, pconf->bit, pmap->val,
+ param, arg, offset);
+ }
+
+ return 0;
+}
+
+int aspeed_pin_config_group_get(struct pinctrl_dev *pctldev,
+ unsigned int selector,
+ unsigned long *config)
+{
+ const unsigned int *pins;
+ unsigned int npins;
+ int rc;
+
+ rc = aspeed_pinctrl_get_group_pins(pctldev, selector, &pins, &npins);
+ if (rc < 0)
+ return rc;
+
+ if (!npins)
+ return -ENODEV;
+
+ rc = aspeed_pin_config_get(pctldev, pins[0], config);
+
+ return rc;
+}
+
+int aspeed_pin_config_group_set(struct pinctrl_dev *pctldev,
+ unsigned int selector,
+ unsigned long *configs,
+ unsigned int num_configs)
+{
+ const unsigned int *pins;
+ unsigned int npins;
+ int rc;
+ int i;
+
+ pr_debug("%s: Fetching pins for group selector %d\n",
+ __func__, selector);
+ rc = aspeed_pinctrl_get_group_pins(pctldev, selector, &pins, &npins);
+ if (rc < 0)
+ return rc;
+
+ for (i = 0; i < npins; i++) {
+ rc = aspeed_pin_config_set(pctldev, pins[i], configs,
+ num_configs);
+ if (rc < 0)
+ return rc;
+ }
+
+ return 0;
+}
diff --git a/drivers/pinctrl/aspeed/pinctrl-aspeed.h b/drivers/pinctrl/aspeed/pinctrl-aspeed.h
index 08a10d4..fa125db 100644
--- a/drivers/pinctrl/aspeed/pinctrl-aspeed.h
+++ b/drivers/pinctrl/aspeed/pinctrl-aspeed.h
@@ -514,6 +514,20 @@
SIG_EXPR_LIST_DECL_SINGLE(gpio, gpio); \
MS_PIN_DECL_(pin, SIG_EXPR_LIST_PTR(gpio))
+/**
+ * @param The pinconf parameter type
+ * @pins The pin range this config struct covers, [low, high]
+ * @reg The register housing the configuration bits
+ * @mask The mask to select the bits of interest in @reg
+ */
+struct aspeed_pin_config {
+ enum pin_config_param param;
+ unsigned int pins[2];
+ unsigned int reg;
+ u8 bit;
+ u8 value;
+};
+
struct aspeed_pinctrl_data {
struct regmap *maps[ASPEED_NR_PINMUX_IPS];
@@ -525,6 +539,9 @@
const struct aspeed_pin_function *functions;
const unsigned int nfunctions;
+
+ const struct aspeed_pin_config *configs;
+ const unsigned int nconfigs;
};
#define ASPEED_PINCTRL_PIN(name_) \
@@ -580,5 +597,16 @@
int aspeed_pinctrl_probe(struct platform_device *pdev,
struct pinctrl_desc *pdesc,
struct aspeed_pinctrl_data *pdata);
+int aspeed_pin_config_get(struct pinctrl_dev *pctldev, unsigned int offset,
+ unsigned long *config);
+int aspeed_pin_config_set(struct pinctrl_dev *pctldev, unsigned int offset,
+ unsigned long *configs, unsigned int num_configs);
+int aspeed_pin_config_group_get(struct pinctrl_dev *pctldev,
+ unsigned int selector,
+ unsigned long *config);
+int aspeed_pin_config_group_set(struct pinctrl_dev *pctldev,
+ unsigned int selector,
+ unsigned long *configs,
+ unsigned int num_configs);
#endif /* PINCTRL_ASPEED */
diff --git a/drivers/pinctrl/bcm/pinctrl-iproc-gpio.c b/drivers/pinctrl/bcm/pinctrl-iproc-gpio.c
index 3ca925d..af5e904 100644
--- a/drivers/pinctrl/bcm/pinctrl-iproc-gpio.c
+++ b/drivers/pinctrl/bcm/pinctrl-iproc-gpio.c
@@ -99,7 +99,7 @@
void __iomem *base;
void __iomem *io_ctrl;
- spinlock_t lock;
+ raw_spinlock_t lock;
struct gpio_chip gc;
unsigned num_banks;
@@ -221,9 +221,9 @@
struct iproc_gpio *chip = gpiochip_get_data(gc);
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
iproc_gpio_irq_set_mask(d, false);
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
}
static void iproc_gpio_irq_unmask(struct irq_data *d)
@@ -232,9 +232,9 @@
struct iproc_gpio *chip = gpiochip_get_data(gc);
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
iproc_gpio_irq_set_mask(d, true);
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
}
static int iproc_gpio_irq_set_type(struct irq_data *d, unsigned int type)
@@ -274,13 +274,13 @@
return -EINVAL;
}
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
iproc_set_bit(chip, IPROC_GPIO_INT_TYPE_OFFSET, gpio,
level_triggered);
iproc_set_bit(chip, IPROC_GPIO_INT_DE_OFFSET, gpio, dual_edge);
iproc_set_bit(chip, IPROC_GPIO_INT_EDGE_OFFSET, gpio,
rising_or_high);
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
dev_dbg(chip->dev,
"gpio:%u level_triggered:%d dual_edge:%d rising_or_high:%d\n",
@@ -328,9 +328,9 @@
struct iproc_gpio *chip = gpiochip_get_data(gc);
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
iproc_set_bit(chip, IPROC_GPIO_OUT_EN_OFFSET, gpio, false);
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
dev_dbg(chip->dev, "gpio:%u set input\n", gpio);
@@ -343,10 +343,10 @@
struct iproc_gpio *chip = gpiochip_get_data(gc);
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
iproc_set_bit(chip, IPROC_GPIO_OUT_EN_OFFSET, gpio, true);
iproc_set_bit(chip, IPROC_GPIO_DATA_OUT_OFFSET, gpio, !!(val));
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
dev_dbg(chip->dev, "gpio:%u set output, value:%d\n", gpio, val);
@@ -358,9 +358,9 @@
struct iproc_gpio *chip = gpiochip_get_data(gc);
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
iproc_set_bit(chip, IPROC_GPIO_DATA_OUT_OFFSET, gpio, !!(val));
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
dev_dbg(chip->dev, "gpio:%u set, value:%d\n", gpio, val);
}
@@ -461,7 +461,7 @@
{
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
if (disable) {
iproc_set_bit(chip, IPROC_GPIO_RES_EN_OFFSET, gpio, false);
@@ -471,7 +471,7 @@
iproc_set_bit(chip, IPROC_GPIO_RES_EN_OFFSET, gpio, true);
}
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
dev_dbg(chip->dev, "gpio:%u set pullup:%d\n", gpio, pull_up);
@@ -483,10 +483,10 @@
{
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
*disable = !iproc_get_bit(chip, IPROC_GPIO_RES_EN_OFFSET, gpio);
*pull_up = iproc_get_bit(chip, IPROC_GPIO_PAD_RES_OFFSET, gpio);
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
}
static int iproc_gpio_set_strength(struct iproc_gpio *chip, unsigned gpio,
@@ -515,7 +515,7 @@
dev_dbg(chip->dev, "gpio:%u set drive strength:%d mA\n", gpio,
strength);
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
strength = (strength / 2) - 1;
for (i = 0; i < GPIO_DRV_STRENGTH_BITS; i++) {
val = readl(base + offset);
@@ -524,7 +524,7 @@
writel(val, base + offset);
offset += 4;
}
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
return 0;
}
@@ -548,7 +548,7 @@
shift = IPROC_GPIO_SHIFT(gpio);
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
*strength = 0;
for (i = 0; i < GPIO_DRV_STRENGTH_BITS; i++) {
val = readl(base + offset) & BIT(shift);
@@ -559,7 +559,7 @@
/* convert to mA */
*strength = (*strength + 1) * 2;
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
return 0;
}
@@ -769,7 +769,7 @@
return -ENODEV;
}
- spin_lock_init(&chip->lock);
+ raw_spin_lock_init(&chip->lock);
gc = &chip->gc;
gc->base = -1;
diff --git a/drivers/pinctrl/bcm/pinctrl-nsp-gpio.c b/drivers/pinctrl/bcm/pinctrl-nsp-gpio.c
index 91ea32d..2244243 100644
--- a/drivers/pinctrl/bcm/pinctrl-nsp-gpio.c
+++ b/drivers/pinctrl/bcm/pinctrl-nsp-gpio.c
@@ -73,7 +73,7 @@
struct pinctrl_dev *pctl;
struct pinctrl_desc pctldesc;
struct irq_domain *irq_domain;
- spinlock_t lock;
+ raw_spinlock_t lock;
};
enum base_type {
@@ -203,9 +203,9 @@
struct nsp_gpio *chip = irq_data_get_irq_chip_data(d);
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
nsp_gpio_irq_set_mask(d, false);
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
}
static void nsp_gpio_irq_unmask(struct irq_data *d)
@@ -213,9 +213,9 @@
struct nsp_gpio *chip = irq_data_get_irq_chip_data(d);
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
nsp_gpio_irq_set_mask(d, true);
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
}
static int nsp_gpio_irq_set_type(struct irq_data *d, unsigned int type)
@@ -226,7 +226,7 @@
bool falling;
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
falling = nsp_get_bit(chip, REG, NSP_GPIO_EVENT_INT_POLARITY, gpio);
level_low = nsp_get_bit(chip, REG, NSP_GPIO_INT_POLARITY, gpio);
@@ -250,13 +250,13 @@
default:
dev_err(chip->dev, "invalid GPIO IRQ type 0x%x\n",
type);
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
return -EINVAL;
}
nsp_set_bit(chip, REG, NSP_GPIO_EVENT_INT_POLARITY, gpio, falling);
nsp_set_bit(chip, REG, NSP_GPIO_INT_POLARITY, gpio, level_low);
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
dev_dbg(chip->dev, "gpio:%u level_low:%s falling:%s\n", gpio,
level_low ? "true" : "false", falling ? "true" : "false");
@@ -295,9 +295,9 @@
struct nsp_gpio *chip = gpiochip_get_data(gc);
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
nsp_set_bit(chip, REG, NSP_GPIO_OUT_EN, gpio, false);
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
dev_dbg(chip->dev, "gpio:%u set input\n", gpio);
return 0;
@@ -309,10 +309,10 @@
struct nsp_gpio *chip = gpiochip_get_data(gc);
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
nsp_set_bit(chip, REG, NSP_GPIO_OUT_EN, gpio, true);
nsp_set_bit(chip, REG, NSP_GPIO_DATA_OUT, gpio, !!(val));
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
dev_dbg(chip->dev, "gpio:%u set output, value:%d\n", gpio, val);
return 0;
@@ -323,9 +323,9 @@
struct nsp_gpio *chip = gpiochip_get_data(gc);
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
nsp_set_bit(chip, REG, NSP_GPIO_DATA_OUT, gpio, !!(val));
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
dev_dbg(chip->dev, "gpio:%u set, value:%d\n", gpio, val);
}
@@ -381,10 +381,10 @@
{
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
nsp_set_bit(chip, IO_CTRL, NSP_PULL_DOWN_EN, gpio, pull_down);
nsp_set_bit(chip, IO_CTRL, NSP_PULL_UP_EN, gpio, pull_up);
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
dev_dbg(chip->dev, "gpio:%u set pullup:%d pulldown: %d\n",
gpio, pull_up, pull_down);
@@ -396,10 +396,10 @@
{
unsigned long flags;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
*pull_up = nsp_get_bit(chip, IO_CTRL, NSP_PULL_UP_EN, gpio);
*pull_down = nsp_get_bit(chip, IO_CTRL, NSP_PULL_DOWN_EN, gpio);
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
}
static int nsp_gpio_set_strength(struct nsp_gpio *chip, unsigned gpio,
@@ -417,7 +417,7 @@
offset = NSP_GPIO_DRV_CTRL;
dev_dbg(chip->dev, "gpio:%u set drive strength:%d mA\n", gpio,
strength);
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
strength = (strength / 2) - 1;
for (i = GPIO_DRV_STRENGTH_BITS; i > 0; i--) {
val = readl(chip->io_ctrl + offset);
@@ -426,7 +426,7 @@
writel(val, chip->io_ctrl + offset);
offset += 4;
}
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
return 0;
}
@@ -442,7 +442,7 @@
offset = NSP_GPIO_DRV_CTRL;
shift = gpio;
- spin_lock_irqsave(&chip->lock, flags);
+ raw_spin_lock_irqsave(&chip->lock, flags);
*strength = 0;
for (i = (GPIO_DRV_STRENGTH_BITS - 1); i >= 0; i--) {
val = readl(chip->io_ctrl + offset) & BIT(shift);
@@ -453,7 +453,7 @@
/* convert to mA */
*strength = (*strength + 1) * 2;
- spin_unlock_irqrestore(&chip->lock, flags);
+ raw_spin_unlock_irqrestore(&chip->lock, flags);
return 0;
}
@@ -660,7 +660,7 @@
return PTR_ERR(chip->io_ctrl);
}
- spin_lock_init(&chip->lock);
+ raw_spin_lock_init(&chip->lock);
gc = &chip->gc;
gc->base = -1;
gc->can_sleep = false;
diff --git a/drivers/pinctrl/core.c b/drivers/pinctrl/core.c
index 32822b0..1653cbd 100644
--- a/drivers/pinctrl/core.c
+++ b/drivers/pinctrl/core.c
@@ -525,7 +525,7 @@
EXPORT_SYMBOL_GPL(pinctrl_find_gpio_range_from_pin);
/**
- * pinctrl_remove_gpio_range() - remove a range of GPIOs fro a pin controller
+ * pinctrl_remove_gpio_range() - remove a range of GPIOs from a pin controller
* @pctldev: pin controller device to remove the range from
* @range: the GPIO range to remove
*/
@@ -1062,7 +1062,7 @@
mutex_unlock(&pinctrl_maps_mutex);
if (ret < 0) {
- /* If some other error than deferral occured, return here */
+ /* If some other error than deferral occurred, return here */
pinctrl_free(p, false);
return ERR_PTR(ret);
}
@@ -1939,9 +1939,9 @@
* @dev: parent device for this pin controller
* @driver_data: private pin controller data for this pin controller
*/
-struct pinctrl_dev *pinctrl_init_controller(struct pinctrl_desc *pctldesc,
- struct device *dev,
- void *driver_data)
+static struct pinctrl_dev *
+pinctrl_init_controller(struct pinctrl_desc *pctldesc, struct device *dev,
+ void *driver_data)
{
struct pinctrl_dev *pctldev;
int ret;
diff --git a/drivers/pinctrl/intel/pinctrl-cherryview.c b/drivers/pinctrl/intel/pinctrl-cherryview.c
index 9ff7901..2debba6 100644
--- a/drivers/pinctrl/intel/pinctrl-cherryview.c
+++ b/drivers/pinctrl/intel/pinctrl-cherryview.c
@@ -149,6 +149,7 @@
size_t ngpio_ranges;
size_t ngpios;
size_t nirqs;
+ acpi_adr_space_type acpi_space_id;
};
struct chv_pin_context {
@@ -405,6 +406,7 @@
* trigger GPEs.
*/
.nirqs = 8,
+ .acpi_space_id = 0x91,
};
static const struct pinctrl_pin_desc north_pins[] = {
@@ -494,6 +496,7 @@
* GPEs.
*/
.nirqs = 8,
+ .acpi_space_id = 0x92,
};
static const struct pinctrl_pin_desc east_pins[] = {
@@ -537,6 +540,7 @@
.ngpio_ranges = ARRAY_SIZE(east_gpio_ranges),
.ngpios = ARRAY_SIZE(east_pins),
.nirqs = 16,
+ .acpi_space_id = 0x93,
};
static const struct pinctrl_pin_desc southeast_pins[] = {
@@ -663,6 +667,7 @@
.ngpio_ranges = ARRAY_SIZE(southeast_gpio_ranges),
.ngpios = ARRAY_SIZE(southeast_pins),
.nirqs = 16,
+ .acpi_space_id = 0x94,
};
static const struct chv_community *chv_communities[] = {
@@ -1608,11 +1613,34 @@
return 0;
}
+static acpi_status chv_pinctrl_mmio_access_handler(u32 function,
+ acpi_physical_address address, u32 bits, u64 *value,
+ void *handler_context, void *region_context)
+{
+ struct chv_pinctrl *pctrl = region_context;
+ unsigned long flags;
+ acpi_status ret = AE_OK;
+
+ raw_spin_lock_irqsave(&chv_lock, flags);
+
+ if (function == ACPI_WRITE)
+ chv_writel((u32)(*value), pctrl->regs + (u32)address);
+ else if (function == ACPI_READ)
+ *value = readl(pctrl->regs + (u32)address);
+ else
+ ret = AE_BAD_PARAMETER;
+
+ raw_spin_unlock_irqrestore(&chv_lock, flags);
+
+ return ret;
+}
+
static int chv_pinctrl_probe(struct platform_device *pdev)
{
struct chv_pinctrl *pctrl;
struct acpi_device *adev;
struct resource *res;
+ acpi_status status;
int ret, irq, i;
adev = ACPI_COMPANION(&pdev->dev);
@@ -1668,11 +1696,29 @@
if (ret)
return ret;
+ status = acpi_install_address_space_handler(adev->handle,
+ pctrl->community->acpi_space_id,
+ chv_pinctrl_mmio_access_handler,
+ NULL, pctrl);
+ if (ACPI_FAILURE(status))
+ dev_err(&pdev->dev, "failed to install ACPI addr space handler\n");
+
platform_set_drvdata(pdev, pctrl);
return 0;
}
+static int chv_pinctrl_remove(struct platform_device *pdev)
+{
+ struct chv_pinctrl *pctrl = platform_get_drvdata(pdev);
+
+ acpi_remove_address_space_handler(ACPI_COMPANION(&pdev->dev),
+ pctrl->community->acpi_space_id,
+ chv_pinctrl_mmio_access_handler);
+
+ return 0;
+}
+
#ifdef CONFIG_PM_SLEEP
static int chv_pinctrl_suspend_noirq(struct device *dev)
{
@@ -1780,6 +1826,7 @@
static struct platform_driver chv_pinctrl_driver = {
.probe = chv_pinctrl_probe,
+ .remove = chv_pinctrl_remove,
.driver = {
.name = "cherryview-pinctrl",
.pm = &chv_pinctrl_pm_ops,
diff --git a/drivers/pinctrl/meson/pinctrl-meson-gxbb.c b/drivers/pinctrl/meson/pinctrl-meson-gxbb.c
index 31a3a98..9b00be1 100644
--- a/drivers/pinctrl/meson/pinctrl-meson-gxbb.c
+++ b/drivers/pinctrl/meson/pinctrl-meson-gxbb.c
@@ -236,6 +236,12 @@
static const unsigned int hdmi_sda_pins[] = { PIN(GPIOH_1, EE_OFF) };
static const unsigned int hdmi_scl_pins[] = { PIN(GPIOH_2, EE_OFF) };
+static const unsigned int i2s_out_ch23_y_pins[] = { PIN(GPIOY_8, EE_OFF) };
+static const unsigned int i2s_out_ch45_y_pins[] = { PIN(GPIOY_9, EE_OFF) };
+static const unsigned int i2s_out_ch67_y_pins[] = { PIN(GPIOY_10, EE_OFF) };
+
+static const unsigned int spdif_out_y_pins[] = { PIN(GPIOY_12, EE_OFF) };
+
static const struct pinctrl_pin_desc meson_gxbb_aobus_pins[] = {
MESON_PIN(GPIOAO_0, 0),
MESON_PIN(GPIOAO_1, 0),
@@ -274,6 +280,16 @@
static const unsigned int pwm_ao_a_12_pins[] = { PIN(GPIOAO_12, 0) };
static const unsigned int pwm_ao_b_pins[] = { PIN(GPIOAO_13, 0) };
+static const unsigned int i2s_am_clk_pins[] = { PIN(GPIOAO_8, 0) };
+static const unsigned int i2s_out_ao_clk_pins[] = { PIN(GPIOAO_9, 0) };
+static const unsigned int i2s_out_lr_clk_pins[] = { PIN(GPIOAO_10, 0) };
+static const unsigned int i2s_out_ch01_ao_pins[] = { PIN(GPIOAO_11, 0) };
+static const unsigned int i2s_out_ch23_ao_pins[] = { PIN(GPIOAO_12, 0) };
+static const unsigned int i2s_out_ch45_ao_pins[] = { PIN(GPIOAO_13, 0) };
+
+static const unsigned int spdif_out_ao_6_pins[] = { PIN(GPIOAO_6, 0) };
+static const unsigned int spdif_out_ao_13_pins[] = { PIN(GPIOAO_13, 0) };
+
static struct meson_pmx_group meson_gxbb_periphs_groups[] = {
GPIO_GROUP(GPIOZ_0, EE_OFF),
GPIO_GROUP(GPIOZ_1, EE_OFF),
@@ -426,6 +442,10 @@
GROUP(uart_rx_c, 1, 16),
GROUP(pwm_a_y, 1, 21),
GROUP(pwm_f_y, 1, 20),
+ GROUP(i2s_out_ch23_y, 1, 5),
+ GROUP(i2s_out_ch45_y, 1, 6),
+ GROUP(i2s_out_ch67_y, 1, 7),
+ GROUP(spdif_out_y, 1, 9),
/* Bank Z */
GROUP(eth_mdio, 6, 1),
@@ -523,6 +543,14 @@
GROUP(pwm_ao_a_6, 0, 18),
GROUP(pwm_ao_a_12, 0, 17),
GROUP(pwm_ao_b, 0, 3),
+ GROUP(i2s_am_clk, 0, 30),
+ GROUP(i2s_out_ao_clk, 0, 29),
+ GROUP(i2s_out_lr_clk, 0, 28),
+ GROUP(i2s_out_ch01_ao, 0, 27),
+ GROUP(i2s_out_ch23_ao, 1, 0),
+ GROUP(i2s_out_ch45_ao, 1, 1),
+ GROUP(spdif_out_ao_6, 0, 16),
+ GROUP(spdif_out_ao_13, 0, 4),
};
static const char * const gpio_periphs_groups[] = {
@@ -652,6 +680,14 @@
"hdmi_sda", "hdmi_scl",
};
+static const char * const i2s_out_groups[] = {
+ "i2s_out_ch23_y", "i2s_out_ch45_y", "i2s_out_ch67_y",
+};
+
+static const char * const spdif_out_groups[] = {
+ "spdif_out_y",
+};
+
static const char * const gpio_aobus_groups[] = {
"GPIOAO_0", "GPIOAO_1", "GPIOAO_2", "GPIOAO_3", "GPIOAO_4",
"GPIOAO_5", "GPIOAO_6", "GPIOAO_7", "GPIOAO_8", "GPIOAO_9",
@@ -694,6 +730,15 @@
"pwm_ao_b",
};
+static const char * const i2s_out_ao_groups[] = {
+ "i2s_am_clk", "i2s_out_ao_clk", "i2s_out_lr_clk",
+ "i2s_out_ch01_ao", "i2s_out_ch23_ao", "i2s_out_ch45_ao",
+};
+
+static const char * const spdif_out_ao_groups[] = {
+ "spdif_out_ao_6", "spdif_out_ao_13",
+};
+
static struct meson_pmx_func meson_gxbb_periphs_functions[] = {
FUNCTION(gpio_periphs),
FUNCTION(emmc),
@@ -717,6 +762,8 @@
FUNCTION(pwm_f_y),
FUNCTION(hdmi_hpd),
FUNCTION(hdmi_i2c),
+ FUNCTION(i2s_out),
+ FUNCTION(spdif_out),
};
static struct meson_pmx_func meson_gxbb_aobus_functions[] = {
@@ -730,6 +777,8 @@
FUNCTION(pwm_ao_a_6),
FUNCTION(pwm_ao_a_12),
FUNCTION(pwm_ao_b),
+ FUNCTION(i2s_out_ao),
+ FUNCTION(spdif_out_ao),
};
static struct meson_bank meson_gxbb_periphs_banks[] = {
diff --git a/drivers/pinctrl/meson/pinctrl-meson-gxl.c b/drivers/pinctrl/meson/pinctrl-meson-gxl.c
index 4ab94a8..998210e 100644
--- a/drivers/pinctrl/meson/pinctrl-meson-gxl.c
+++ b/drivers/pinctrl/meson/pinctrl-meson-gxl.c
@@ -136,6 +136,11 @@
static const unsigned int emmc_cmd_pins[] = { PIN(BOOT_10, EE_OFF) };
static const unsigned int emmc_ds_pins[] = { PIN(BOOT_15, EE_OFF) };
+static const unsigned int nor_d_pins[] = { PIN(BOOT_11, EE_OFF) };
+static const unsigned int nor_q_pins[] = { PIN(BOOT_12, EE_OFF) };
+static const unsigned int nor_c_pins[] = { PIN(BOOT_13, EE_OFF) };
+static const unsigned int nor_cs_pins[] = { PIN(BOOT_15, EE_OFF) };
+
static const unsigned int sdcard_d0_pins[] = { PIN(CARD_1, EE_OFF) };
static const unsigned int sdcard_d1_pins[] = { PIN(CARD_0, EE_OFF) };
static const unsigned int sdcard_d2_pins[] = { PIN(CARD_5, EE_OFF) };
@@ -167,9 +172,13 @@
static const unsigned int uart_tx_b_pins[] = { PIN(GPIODV_24, EE_OFF) };
static const unsigned int uart_rx_b_pins[] = { PIN(GPIODV_25, EE_OFF) };
+static const unsigned int uart_cts_b_pins[] = { PIN(GPIODV_26, EE_OFF) };
+static const unsigned int uart_rts_b_pins[] = { PIN(GPIODV_27, EE_OFF) };
static const unsigned int uart_tx_c_pins[] = { PIN(GPIOX_8, EE_OFF) };
static const unsigned int uart_rx_c_pins[] = { PIN(GPIOX_9, EE_OFF) };
+static const unsigned int uart_cts_c_pins[] = { PIN(GPIOX_10, EE_OFF) };
+static const unsigned int uart_rts_c_pins[] = { PIN(GPIOX_11, EE_OFF) };
static const unsigned int i2c_sck_a_pins[] = { PIN(GPIODV_25, EE_OFF) };
static const unsigned int i2c_sda_a_pins[] = { PIN(GPIODV_24, EE_OFF) };
@@ -180,6 +189,9 @@
static const unsigned int i2c_sck_c_pins[] = { PIN(GPIODV_29, EE_OFF) };
static const unsigned int i2c_sda_c_pins[] = { PIN(GPIODV_28, EE_OFF) };
+static const unsigned int i2c_sck_c_dv19_pins[] = { PIN(GPIODV_19, EE_OFF) };
+static const unsigned int i2c_sda_c_dv18_pins[] = { PIN(GPIODV_18, EE_OFF) };
+
static const unsigned int eth_mdio_pins[] = { PIN(GPIOZ_0, EE_OFF) };
static const unsigned int eth_mdc_pins[] = { PIN(GPIOZ_1, EE_OFF) };
static const unsigned int eth_clk_rx_clk_pins[] = { PIN(GPIOZ_2, EE_OFF) };
@@ -195,12 +207,33 @@
static const unsigned int eth_txd2_pins[] = { PIN(GPIOZ_12, EE_OFF) };
static const unsigned int eth_txd3_pins[] = { PIN(GPIOZ_13, EE_OFF) };
+static const unsigned int pwm_a_pins[] = { PIN(GPIOX_6, EE_OFF) };
+
+static const unsigned int pwm_b_pins[] = { PIN(GPIODV_29, EE_OFF) };
+
+static const unsigned int pwm_c_pins[] = { PIN(GPIOZ_15, EE_OFF) };
+
+static const unsigned int pwm_d_pins[] = { PIN(GPIODV_28, EE_OFF) };
+
static const unsigned int pwm_e_pins[] = { PIN(GPIOX_16, EE_OFF) };
+static const unsigned int pwm_f_clk_pins[] = { PIN(GPIOCLK_1, EE_OFF) };
+static const unsigned int pwm_f_x_pins[] = { PIN(GPIOX_7, EE_OFF) };
+
static const unsigned int hdmi_hpd_pins[] = { PIN(GPIOH_0, EE_OFF) };
static const unsigned int hdmi_sda_pins[] = { PIN(GPIOH_1, EE_OFF) };
static const unsigned int hdmi_scl_pins[] = { PIN(GPIOH_2, EE_OFF) };
+static const unsigned int i2s_am_clk_pins[] = { PIN(GPIOH_6, EE_OFF) };
+static const unsigned int i2s_out_ao_clk_pins[] = { PIN(GPIOH_7, EE_OFF) };
+static const unsigned int i2s_out_lr_clk_pins[] = { PIN(GPIOH_8, EE_OFF) };
+static const unsigned int i2s_out_ch01_pins[] = { PIN(GPIOH_9, EE_OFF) };
+static const unsigned int i2s_out_ch23_z_pins[] = { PIN(GPIOZ_5, EE_OFF) };
+static const unsigned int i2s_out_ch45_z_pins[] = { PIN(GPIOZ_6, EE_OFF) };
+static const unsigned int i2s_out_ch67_z_pins[] = { PIN(GPIOZ_7, EE_OFF) };
+
+static const unsigned int spdif_out_h_pins[] = { PIN(GPIOH_4, EE_OFF) };
+
static const struct pinctrl_pin_desc meson_gxl_aobus_pins[] = {
MESON_PIN(GPIOAO_0, 0),
MESON_PIN(GPIOAO_1, 0),
@@ -216,6 +249,8 @@
static const unsigned int uart_tx_ao_a_pins[] = { PIN(GPIOAO_0, 0) };
static const unsigned int uart_rx_ao_a_pins[] = { PIN(GPIOAO_1, 0) };
+static const unsigned int uart_tx_ao_b_0_pins[] = { PIN(GPIOAO_0, 0) };
+static const unsigned int uart_rx_ao_b_1_pins[] = { PIN(GPIOAO_1, 0) };
static const unsigned int uart_cts_ao_a_pins[] = { PIN(GPIOAO_2, 0) };
static const unsigned int uart_rts_ao_a_pins[] = { PIN(GPIOAO_3, 0) };
static const unsigned int uart_tx_ao_b_pins[] = { PIN(GPIOAO_4, 0) };
@@ -223,9 +258,24 @@
static const unsigned int uart_cts_ao_b_pins[] = { PIN(GPIOAO_2, 0) };
static const unsigned int uart_rts_ao_b_pins[] = { PIN(GPIOAO_3, 0) };
+static const unsigned int i2c_sck_ao_pins[] = {PIN(GPIOAO_4, 0) };
+static const unsigned int i2c_sda_ao_pins[] = {PIN(GPIOAO_5, 0) };
+static const unsigned int i2c_slave_sck_ao_pins[] = {PIN(GPIOAO_4, 0) };
+static const unsigned int i2c_slave_sda_ao_pins[] = {PIN(GPIOAO_5, 0) };
+
static const unsigned int remote_input_ao_pins[] = {PIN(GPIOAO_7, 0) };
+static const unsigned int pwm_ao_a_3_pins[] = { PIN(GPIOAO_3, 0) };
+static const unsigned int pwm_ao_a_8_pins[] = { PIN(GPIOAO_8, 0) };
+
static const unsigned int pwm_ao_b_pins[] = { PIN(GPIOAO_9, 0) };
+static const unsigned int pwm_ao_b_6_pins[] = { PIN(GPIOAO_6, 0) };
+
+static const unsigned int i2s_out_ch23_ao_pins[] = { PIN(GPIOAO_8, EE_OFF) };
+static const unsigned int i2s_out_ch45_ao_pins[] = { PIN(GPIOAO_9, EE_OFF) };
+
+static const unsigned int spdif_out_ao_6_pins[] = { PIN(GPIOAO_6, EE_OFF) };
+static const unsigned int spdif_out_ao_9_pins[] = { PIN(GPIOAO_9, EE_OFF) };
static struct meson_pmx_group meson_gxl_periphs_groups[] = {
GPIO_GROUP(GPIOZ_0, EE_OFF),
@@ -341,8 +391,8 @@
GROUP(sdio_d1, 5, 30),
GROUP(sdio_d2, 5, 29),
GROUP(sdio_d3, 5, 28),
- GROUP(sdio_cmd, 5, 27),
- GROUP(sdio_clk, 5, 26),
+ GROUP(sdio_clk, 5, 27),
+ GROUP(sdio_cmd, 5, 26),
GROUP(sdio_irq, 5, 24),
GROUP(uart_tx_a, 5, 19),
GROUP(uart_rx_a, 5, 18),
@@ -350,11 +400,15 @@
GROUP(uart_rts_a, 5, 16),
GROUP(uart_tx_c, 5, 13),
GROUP(uart_rx_c, 5, 12),
+ GROUP(uart_cts_c, 5, 11),
+ GROUP(uart_rts_c, 5, 10),
+ GROUP(pwm_a, 5, 25),
GROUP(pwm_e, 5, 15),
+ GROUP(pwm_f_x, 5, 14),
/* Bank Z */
- GROUP(eth_mdio, 4, 22),
- GROUP(eth_mdc, 4, 23),
+ GROUP(eth_mdio, 4, 23),
+ GROUP(eth_mdc, 4, 22),
GROUP(eth_clk_rx_clk, 4, 21),
GROUP(eth_rx_dv, 4, 20),
GROUP(eth_rxd0, 4, 19),
@@ -367,27 +421,46 @@
GROUP(eth_txd1, 4, 12),
GROUP(eth_txd2, 4, 11),
GROUP(eth_txd3, 4, 10),
+ GROUP(pwm_c, 3, 20),
+ GROUP(i2s_out_ch23_z, 3, 26),
+ GROUP(i2s_out_ch45_z, 3, 25),
+ GROUP(i2s_out_ch67_z, 3, 24),
/* Bank H */
GROUP(hdmi_hpd, 6, 31),
GROUP(hdmi_sda, 6, 30),
GROUP(hdmi_scl, 6, 29),
+ GROUP(i2s_am_clk, 6, 26),
+ GROUP(i2s_out_ao_clk, 6, 25),
+ GROUP(i2s_out_lr_clk, 6, 24),
+ GROUP(i2s_out_ch01, 6, 23),
+ GROUP(spdif_out_h, 6, 28),
/* Bank DV */
GROUP(uart_tx_b, 2, 16),
GROUP(uart_rx_b, 2, 15),
- GROUP(i2c_sck_a, 1, 15),
- GROUP(i2c_sda_a, 1, 14),
- GROUP(i2c_sck_b, 1, 13),
- GROUP(i2c_sda_b, 1, 12),
- GROUP(i2c_sck_c, 1, 11),
- GROUP(i2c_sda_c, 1, 10),
+ GROUP(uart_cts_b, 2, 14),
+ GROUP(uart_rts_b, 2, 13),
+ GROUP(i2c_sda_c_dv18, 1, 17),
+ GROUP(i2c_sck_c_dv19, 1, 16),
+ GROUP(i2c_sda_a, 1, 15),
+ GROUP(i2c_sck_a, 1, 14),
+ GROUP(i2c_sda_b, 1, 13),
+ GROUP(i2c_sck_b, 1, 12),
+ GROUP(i2c_sda_c, 1, 11),
+ GROUP(i2c_sck_c, 1, 10),
+ GROUP(pwm_b, 2, 11),
+ GROUP(pwm_d, 2, 12),
/* Bank BOOT */
GROUP(emmc_nand_d07, 7, 31),
GROUP(emmc_clk, 7, 30),
GROUP(emmc_cmd, 7, 29),
GROUP(emmc_ds, 7, 28),
+ GROUP(nor_d, 7, 13),
+ GROUP(nor_q, 7, 12),
+ GROUP(nor_c, 7, 11),
+ GROUP(nor_cs, 7, 10),
GROUP(nand_ce0, 7, 7),
GROUP(nand_ce1, 7, 6),
GROUP(nand_rb0, 7, 5),
@@ -404,6 +477,9 @@
GROUP(sdcard_d2, 6, 0),
GROUP(sdcard_cmd, 6, 2),
GROUP(sdcard_clk, 6, 3),
+
+ /* Bank CLK */
+ GROUP(pwm_f_clk, 8, 30),
};
static struct meson_pmx_group meson_gxl_aobus_groups[] = {
@@ -419,16 +495,29 @@
GPIO_GROUP(GPIOAO_9, 0),
/* bank AO */
+ GROUP(uart_tx_ao_b_0, 0, 26),
+ GROUP(uart_rx_ao_b_1, 0, 25),
GROUP(uart_tx_ao_b, 0, 24),
- GROUP(uart_rx_ao_b, 0, 25),
+ GROUP(uart_rx_ao_b, 0, 23),
GROUP(uart_tx_ao_a, 0, 12),
GROUP(uart_rx_ao_a, 0, 11),
GROUP(uart_cts_ao_a, 0, 10),
GROUP(uart_rts_ao_a, 0, 9),
GROUP(uart_cts_ao_b, 0, 8),
GROUP(uart_rts_ao_b, 0, 7),
+ GROUP(i2c_sck_ao, 0, 6),
+ GROUP(i2c_sda_ao, 0, 5),
+ GROUP(i2c_slave_sck_ao, 0, 2),
+ GROUP(i2c_slave_sda_ao, 0, 1),
GROUP(remote_input_ao, 0, 0),
+ GROUP(pwm_ao_a_3, 0, 22),
+ GROUP(pwm_ao_b_6, 0, 18),
+ GROUP(pwm_ao_a_8, 0, 17),
GROUP(pwm_ao_b, 0, 3),
+ GROUP(i2s_out_ch23_ao, 1, 0),
+ GROUP(i2s_out_ch45_ao, 1, 1),
+ GROUP(spdif_out_ao_6, 0, 16),
+ GROUP(spdif_out_ao_9, 0, 4),
};
static const char * const gpio_periphs_groups[] = {
@@ -467,6 +556,10 @@
"emmc_nand_d07", "emmc_clk", "emmc_cmd", "emmc_ds",
};
+static const char * const nor_groups[] = {
+ "nor_d", "nor_q", "nor_c", "nor_cs",
+};
+
static const char * const sdcard_groups[] = {
"sdcard_d0", "sdcard_d1", "sdcard_d2", "sdcard_d3",
"sdcard_cmd", "sdcard_clk",
@@ -487,11 +580,11 @@
};
static const char * const uart_b_groups[] = {
- "uart_tx_b", "uart_rx_b",
+ "uart_tx_b", "uart_rx_b", "uart_cts_b", "uart_rts_b",
};
static const char * const uart_c_groups[] = {
- "uart_tx_c", "uart_rx_c",
+ "uart_tx_c", "uart_rx_c", "uart_cts_c", "uart_rts_c",
};
static const char * const i2c_a_groups[] = {
@@ -503,7 +596,7 @@
};
static const char * const i2c_c_groups[] = {
- "i2c_sck_c", "i2c_sda_c",
+ "i2c_sck_c", "i2c_sda_c", "i2c_sda_c_dv18", "i2c_sck_c_dv19",
};
static const char * const eth_groups[] = {
@@ -513,10 +606,30 @@
"eth_txd0", "eth_txd1", "eth_txd2", "eth_txd3",
};
+static const char * const pwm_a_groups[] = {
+ "pwm_a",
+};
+
+static const char * const pwm_b_groups[] = {
+ "pwm_b",
+};
+
+static const char * const pwm_c_groups[] = {
+ "pwm_c",
+};
+
+static const char * const pwm_d_groups[] = {
+ "pwm_d",
+};
+
static const char * const pwm_e_groups[] = {
"pwm_e",
};
+static const char * const pwm_f_groups[] = {
+ "pwm_f_clk", "pwm_f_x",
+};
+
static const char * const hdmi_hpd_groups[] = {
"hdmi_hpd",
};
@@ -525,6 +638,15 @@
"hdmi_sda", "hdmi_scl",
};
+static const char * const i2s_out_groups[] = {
+ "i2s_am_clk", "i2s_out_ao_clk", "i2s_out_lr_clk",
+ "i2s_out_ch01", "i2s_out_ch23_z", "i2s_out_ch45_z", "i2s_out_ch67_z",
+};
+
+static const char * const spdif_out_groups[] = {
+ "spdif_out_h",
+};
+
static const char * const gpio_aobus_groups[] = {
"GPIOAO_0", "GPIOAO_1", "GPIOAO_2", "GPIOAO_3", "GPIOAO_4",
"GPIOAO_5", "GPIOAO_6", "GPIOAO_7", "GPIOAO_8", "GPIOAO_9",
@@ -536,19 +658,41 @@
static const char * const uart_ao_b_groups[] = {
"uart_tx_ao_b", "uart_rx_ao_b", "uart_cts_ao_b", "uart_rts_ao_b",
+ "uart_tx_ao_b_0", "uart_rx_ao_b_1",
+};
+
+static const char * const i2c_ao_groups[] = {
+ "i2c_sck_ao", "i2c_sda_ao",
+};
+
+static const char * const i2c_slave_ao_groups[] = {
+ "i2c_slave_sck_ao", "i2c_slave_sda_ao",
};
static const char * const remote_input_ao_groups[] = {
"remote_input_ao",
};
+static const char * const pwm_ao_a_groups[] = {
+ "pwm_ao_a_3", "pwm_ao_a_8",
+};
+
static const char * const pwm_ao_b_groups[] = {
- "pwm_ao_b",
+ "pwm_ao_b", "pwm_ao_b_6",
+};
+
+static const char * const i2s_out_ao_groups[] = {
+ "i2s_out_ch23_ao", "i2s_out_ch45_ao",
+};
+
+static const char * const spdif_out_ao_groups[] = {
+ "spdif_out_ao_6", "spdif_out_ao_9",
};
static struct meson_pmx_func meson_gxl_periphs_functions[] = {
FUNCTION(gpio_periphs),
FUNCTION(emmc),
+ FUNCTION(nor),
FUNCTION(sdcard),
FUNCTION(sdio),
FUNCTION(nand),
@@ -559,17 +703,29 @@
FUNCTION(i2c_b),
FUNCTION(i2c_c),
FUNCTION(eth),
+ FUNCTION(pwm_a),
+ FUNCTION(pwm_b),
+ FUNCTION(pwm_c),
+ FUNCTION(pwm_d),
FUNCTION(pwm_e),
+ FUNCTION(pwm_f),
FUNCTION(hdmi_hpd),
FUNCTION(hdmi_i2c),
+ FUNCTION(i2s_out),
+ FUNCTION(spdif_out),
};
static struct meson_pmx_func meson_gxl_aobus_functions[] = {
FUNCTION(gpio_aobus),
FUNCTION(uart_ao),
FUNCTION(uart_ao_b),
+ FUNCTION(i2c_ao),
+ FUNCTION(i2c_slave_ao),
FUNCTION(remote_input_ao),
+ FUNCTION(pwm_ao_a),
FUNCTION(pwm_ao_b),
+ FUNCTION(i2s_out_ao),
+ FUNCTION(spdif_out_ao),
};
static struct meson_bank meson_gxl_periphs_banks[] = {
diff --git a/drivers/pinctrl/meson/pinctrl-meson.c b/drivers/pinctrl/meson/pinctrl-meson.c
index cf1686e..66ed70c 100644
--- a/drivers/pinctrl/meson/pinctrl-meson.c
+++ b/drivers/pinctrl/meson/pinctrl-meson.c
@@ -555,22 +555,10 @@
if (ret) {
dev_err(pc->dev, "can't add gpio chip %s\n",
pc->data->name);
- goto fail;
- }
-
- ret = gpiochip_add_pin_range(&pc->chip, dev_name(pc->dev),
- 0, pc->data->pin_base,
- pc->chip.ngpio);
- if (ret) {
- dev_err(pc->dev, "can't add pin range\n");
- goto fail;
+ return ret;
}
return 0;
-fail:
- gpiochip_remove(&pc->chip);
-
- return ret;
}
static struct regmap_config meson_regmap_config = {
diff --git a/drivers/pinctrl/meson/pinctrl-meson8b.c b/drivers/pinctrl/meson/pinctrl-meson8b.c
index 76f077f..bf747eb 100644
--- a/drivers/pinctrl/meson/pinctrl-meson8b.c
+++ b/drivers/pinctrl/meson/pinctrl-meson8b.c
@@ -267,8 +267,8 @@
static const unsigned int nand_cle_pins[] = { PIN(BOOT_12, 0) };
static const unsigned int nand_wen_clk_pins[] = { PIN(BOOT_13, 0) };
static const unsigned int nand_ren_clk_pins[] = { PIN(BOOT_14, 0) };
-static const unsigned int nand_dqs_0_pins[] = { PIN(BOOT_15, 0) };
-static const unsigned int nand_dqs_1_pins[] = { PIN(BOOT_18, 0) };
+static const unsigned int nand_dqs_15_pins[] = { PIN(BOOT_15, 0) };
+static const unsigned int nand_dqs_18_pins[] = { PIN(BOOT_18, 0) };
static const unsigned int sdxc_d0_c_pins[] = { PIN(BOOT_0, 0)};
static const unsigned int sdxc_d13_c_pins[] = { PIN(BOOT_1, 0), PIN(BOOT_2, 0),
@@ -527,8 +527,8 @@
GROUP(nand_cle, 2, 20),
GROUP(nand_wen_clk, 2, 19),
GROUP(nand_ren_clk, 2, 18),
- GROUP(nand_dqs_0, 2, 27),
- GROUP(nand_dqs_1, 2, 28),
+ GROUP(nand_dqs_15, 2, 27),
+ GROUP(nand_dqs_18, 2, 28),
GROUP(sdxc_d0_c, 4, 30),
GROUP(sdxc_d13_c, 4, 29),
GROUP(sdxc_d47_c, 4, 28),
@@ -739,8 +739,8 @@
static const char * const nand_groups[] = {
"nand_io", "nand_io_ce0", "nand_io_ce1",
"nand_io_rb0", "nand_ale", "nand_cle",
- "nand_wen_clk", "nand_ren_clk", "nand_dqs0",
- "nand_dqs1"
+ "nand_wen_clk", "nand_ren_clk", "nand_dqs_15",
+ "nand_dqs_18"
};
static const char * const nor_groups[] = {
diff --git a/drivers/pinctrl/mvebu/Kconfig b/drivers/pinctrl/mvebu/Kconfig
index 1706024..5bade32 100644
--- a/drivers/pinctrl/mvebu/Kconfig
+++ b/drivers/pinctrl/mvebu/Kconfig
@@ -39,3 +39,10 @@
select PINCTRL_MVEBU
endif
+
+config PINCTRL_ARMADA_37XX
+ bool
+ select GENERIC_PINCONF
+ select MFD_SYSCON
+ select PINCONF
+ select PINMUX
diff --git a/drivers/pinctrl/mvebu/Makefile b/drivers/pinctrl/mvebu/Makefile
index 18270cd..60c245a 100644
--- a/drivers/pinctrl/mvebu/Makefile
+++ b/drivers/pinctrl/mvebu/Makefile
@@ -1,4 +1,4 @@
-obj-y += pinctrl-mvebu.o
+obj-$(CONFIG_PINCTRL_MVEBU) += pinctrl-mvebu.o
obj-$(CONFIG_PINCTRL_DOVE) += pinctrl-dove.o
obj-$(CONFIG_PINCTRL_KIRKWOOD) += pinctrl-kirkwood.o
obj-$(CONFIG_PINCTRL_ARMADA_370) += pinctrl-armada-370.o
@@ -6,4 +6,5 @@
obj-$(CONFIG_PINCTRL_ARMADA_38X) += pinctrl-armada-38x.o
obj-$(CONFIG_PINCTRL_ARMADA_39X) += pinctrl-armada-39x.o
obj-$(CONFIG_PINCTRL_ARMADA_XP) += pinctrl-armada-xp.o
+obj-$(CONFIG_PINCTRL_ARMADA_37XX) += pinctrl-armada-37xx.o
obj-$(CONFIG_PINCTRL_ORION) += pinctrl-orion.o
diff --git a/drivers/pinctrl/mvebu/pinctrl-armada-37xx.c b/drivers/pinctrl/mvebu/pinctrl-armada-37xx.c
new file mode 100644
index 0000000..5c96f55
--- /dev/null
+++ b/drivers/pinctrl/mvebu/pinctrl-armada-37xx.c
@@ -0,0 +1,748 @@
+/*
+ * Marvell 37xx SoC pinctrl driver
+ *
+ * Copyright (C) 2017 Marvell
+ *
+ * Gregory CLEMENT <gregory.clement@free-electrons.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2 or later. This program is licensed "as is"
+ * without any warranty of any kind, whether express or implied.
+ */
+
+#include <linux/gpio/driver.h>
+#include <linux/mfd/syscon.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/pinctrl/pinconf-generic.h>
+#include <linux/pinctrl/pinconf.h>
+#include <linux/pinctrl/pinctrl.h>
+#include <linux/pinctrl/pinmux.h>
+#include <linux/platform_device.h>
+#include <linux/regmap.h>
+#include <linux/slab.h>
+
+#include "../pinctrl-utils.h"
+
+#define OUTPUT_EN 0x0
+#define INPUT_VAL 0x10
+#define OUTPUT_VAL 0x18
+#define OUTPUT_CTL 0x20
+#define SELECTION 0x30
+
+#define NB_FUNCS 2
+#define GPIO_PER_REG 32
+
+/**
+ * struct armada_37xx_pin_group: represents group of pins of a pinmux function.
+ * The pins of a pinmux groups are composed of one or two groups of contiguous
+ * pins.
+ * @name: Name of the pin group, used to lookup the group.
+ * @start_pins: Index of the first pin of the main range of pins belonging to
+ * the group
+ * @npins: Number of pins included in the first range
+ * @reg_mask: Bit mask matching the group in the selection register
+ * @extra_pins: Index of the first pin of the optional second range of pins
+ * belonging to the group
+ * @npins: Number of pins included in the second optional range
+ * @funcs: A list of pinmux functions that can be selected for this group.
+ * @pins: List of the pins included in the group
+ */
+struct armada_37xx_pin_group {
+ const char *name;
+ unsigned int start_pin;
+ unsigned int npins;
+ u32 reg_mask;
+ u32 val[NB_FUNCS];
+ unsigned int extra_pin;
+ unsigned int extra_npins;
+ const char *funcs[NB_FUNCS];
+ unsigned int *pins;
+};
+
+struct armada_37xx_pin_data {
+ u8 nr_pins;
+ char *name;
+ struct armada_37xx_pin_group *groups;
+ int ngroups;
+};
+
+struct armada_37xx_pmx_func {
+ const char *name;
+ const char **groups;
+ unsigned int ngroups;
+};
+
+struct armada_37xx_pinctrl {
+ struct regmap *regmap;
+ const struct armada_37xx_pin_data *data;
+ struct device *dev;
+ struct gpio_chip gpio_chip;
+ struct pinctrl_desc pctl;
+ struct pinctrl_dev *pctl_dev;
+ struct armada_37xx_pin_group *groups;
+ unsigned int ngroups;
+ struct armada_37xx_pmx_func *funcs;
+ unsigned int nfuncs;
+};
+
+#define PIN_GRP(_name, _start, _nr, _mask, _func1, _func2) \
+ { \
+ .name = _name, \
+ .start_pin = _start, \
+ .npins = _nr, \
+ .reg_mask = _mask, \
+ .val = {0, _mask}, \
+ .funcs = {_func1, _func2} \
+ }
+
+#define PIN_GRP_GPIO(_name, _start, _nr, _mask, _func1) \
+ { \
+ .name = _name, \
+ .start_pin = _start, \
+ .npins = _nr, \
+ .reg_mask = _mask, \
+ .val = {0, _mask}, \
+ .funcs = {_func1, "gpio"} \
+ }
+
+#define PIN_GRP_GPIO_2(_name, _start, _nr, _mask, _val1, _val2, _func1) \
+ { \
+ .name = _name, \
+ .start_pin = _start, \
+ .npins = _nr, \
+ .reg_mask = _mask, \
+ .val = {_val1, _val2}, \
+ .funcs = {_func1, "gpio"} \
+ }
+
+#define PIN_GRP_EXTRA(_name, _start, _nr, _mask, _v1, _v2, _start2, _nr2, \
+ _f1, _f2) \
+ { \
+ .name = _name, \
+ .start_pin = _start, \
+ .npins = _nr, \
+ .reg_mask = _mask, \
+ .val = {_v1, _v2}, \
+ .extra_pin = _start2, \
+ .extra_npins = _nr2, \
+ .funcs = {_f1, _f2} \
+ }
+
+static struct armada_37xx_pin_group armada_37xx_nb_groups[] = {
+ PIN_GRP_GPIO("jtag", 20, 5, BIT(0), "jtag"),
+ PIN_GRP_GPIO("sdio0", 8, 3, BIT(1), "sdio"),
+ PIN_GRP_GPIO("emmc_nb", 27, 9, BIT(2), "emmc"),
+ PIN_GRP_GPIO("pwm0", 11, 1, BIT(3), "pwm"),
+ PIN_GRP_GPIO("pwm1", 12, 1, BIT(4), "pwm"),
+ PIN_GRP_GPIO("pwm2", 13, 1, BIT(5), "pwm"),
+ PIN_GRP_GPIO("pwm3", 14, 1, BIT(6), "pwm"),
+ PIN_GRP_GPIO("pmic1", 17, 1, BIT(7), "pmic"),
+ PIN_GRP_GPIO("pmic0", 16, 1, BIT(8), "pmic"),
+ PIN_GRP_GPIO("i2c2", 2, 2, BIT(9), "i2c"),
+ PIN_GRP_GPIO("i2c1", 0, 2, BIT(10), "i2c"),
+ PIN_GRP_GPIO("spi_cs1", 17, 1, BIT(12), "spi"),
+ PIN_GRP_GPIO_2("spi_cs2", 18, 1, BIT(13) | BIT(19), 0, BIT(13), "spi"),
+ PIN_GRP_GPIO_2("spi_cs3", 19, 1, BIT(14) | BIT(19), 0, BIT(14), "spi"),
+ PIN_GRP_GPIO("onewire", 4, 1, BIT(16), "onewire"),
+ PIN_GRP_GPIO("uart1", 25, 2, BIT(17), "uart"),
+ PIN_GRP_GPIO("spi_quad", 15, 2, BIT(18), "spi"),
+ PIN_GRP_EXTRA("uart2", 9, 2, BIT(13) | BIT(14) | BIT(19),
+ BIT(13) | BIT(14), BIT(19), 18, 2, "gpio", "uart"),
+ PIN_GRP_GPIO("led0_od", 11, 1, BIT(20), "led"),
+ PIN_GRP_GPIO("led1_od", 12, 1, BIT(21), "led"),
+ PIN_GRP_GPIO("led2_od", 13, 1, BIT(22), "led"),
+ PIN_GRP_GPIO("led3_od", 14, 1, BIT(23), "led"),
+
+};
+
+static struct armada_37xx_pin_group armada_37xx_sb_groups[] = {
+ PIN_GRP_GPIO("usb32_drvvbus0", 0, 1, BIT(0), "drvbus"),
+ PIN_GRP_GPIO("usb2_drvvbus1", 1, 1, BIT(1), "drvbus"),
+ PIN_GRP_GPIO("sdio_sb", 24, 5, BIT(2), "sdio"),
+ PIN_GRP_EXTRA("rgmii", 6, 14, BIT(3), 0, BIT(3), 23, 1, "mii", "gpio"),
+ PIN_GRP_GPIO("pcie1", 3, 2, BIT(4), "pcie"),
+ PIN_GRP_GPIO("ptp", 20, 3, BIT(5), "ptp"),
+ PIN_GRP("ptp_clk", 21, 1, BIT(6), "ptp", "mii"),
+ PIN_GRP("ptp_trig", 22, 1, BIT(7), "ptp", "mii"),
+ PIN_GRP("mii_col", 23, 1, BIT(8), "mii", "mii_err"),
+};
+
+const struct armada_37xx_pin_data armada_37xx_pin_nb = {
+ .nr_pins = 36,
+ .name = "GPIO1",
+ .groups = armada_37xx_nb_groups,
+ .ngroups = ARRAY_SIZE(armada_37xx_nb_groups),
+};
+
+const struct armada_37xx_pin_data armada_37xx_pin_sb = {
+ .nr_pins = 29,
+ .name = "GPIO2",
+ .groups = armada_37xx_sb_groups,
+ .ngroups = ARRAY_SIZE(armada_37xx_sb_groups),
+};
+
+static inline void armada_37xx_update_reg(unsigned int *reg,
+ unsigned int offset)
+{
+ /* We never have more than 2 registers */
+ if (offset >= GPIO_PER_REG) {
+ offset -= GPIO_PER_REG;
+ *reg += sizeof(u32);
+ }
+}
+
+static int armada_37xx_get_func_reg(struct armada_37xx_pin_group *grp,
+ const char *func)
+{
+ int f;
+
+ for (f = 0; f < NB_FUNCS; f++)
+ if (!strcmp(grp->funcs[f], func))
+ return f;
+
+ return -ENOTSUPP;
+}
+
+static struct armada_37xx_pin_group *armada_37xx_find_next_grp_by_pin(
+ struct armada_37xx_pinctrl *info, int pin, int *grp)
+{
+ while (*grp < info->ngroups) {
+ struct armada_37xx_pin_group *group = &info->groups[*grp];
+ int j;
+
+ *grp = *grp + 1;
+ for (j = 0; j < (group->npins + group->extra_npins); j++)
+ if (group->pins[j] == pin)
+ return group;
+ }
+ return NULL;
+}
+
+static int armada_37xx_pin_config_group_get(struct pinctrl_dev *pctldev,
+ unsigned int selector, unsigned long *config)
+{
+ return -ENOTSUPP;
+}
+
+static int armada_37xx_pin_config_group_set(struct pinctrl_dev *pctldev,
+ unsigned int selector, unsigned long *configs,
+ unsigned int num_configs)
+{
+ return -ENOTSUPP;
+}
+
+static struct pinconf_ops armada_37xx_pinconf_ops = {
+ .is_generic = true,
+ .pin_config_group_get = armada_37xx_pin_config_group_get,
+ .pin_config_group_set = armada_37xx_pin_config_group_set,
+};
+
+static int armada_37xx_get_groups_count(struct pinctrl_dev *pctldev)
+{
+ struct armada_37xx_pinctrl *info = pinctrl_dev_get_drvdata(pctldev);
+
+ return info->ngroups;
+}
+
+static const char *armada_37xx_get_group_name(struct pinctrl_dev *pctldev,
+ unsigned int group)
+{
+ struct armada_37xx_pinctrl *info = pinctrl_dev_get_drvdata(pctldev);
+
+ return info->groups[group].name;
+}
+
+static int armada_37xx_get_group_pins(struct pinctrl_dev *pctldev,
+ unsigned int selector,
+ const unsigned int **pins,
+ unsigned int *npins)
+{
+ struct armada_37xx_pinctrl *info = pinctrl_dev_get_drvdata(pctldev);
+
+ if (selector >= info->ngroups)
+ return -EINVAL;
+
+ *pins = info->groups[selector].pins;
+ *npins = info->groups[selector].npins +
+ info->groups[selector].extra_npins;
+
+ return 0;
+}
+
+static const struct pinctrl_ops armada_37xx_pctrl_ops = {
+ .get_groups_count = armada_37xx_get_groups_count,
+ .get_group_name = armada_37xx_get_group_name,
+ .get_group_pins = armada_37xx_get_group_pins,
+ .dt_node_to_map = pinconf_generic_dt_node_to_map_group,
+ .dt_free_map = pinctrl_utils_free_map,
+};
+
+/*
+ * Pinmux_ops handling
+ */
+
+static int armada_37xx_pmx_get_funcs_count(struct pinctrl_dev *pctldev)
+{
+ struct armada_37xx_pinctrl *info = pinctrl_dev_get_drvdata(pctldev);
+
+ return info->nfuncs;
+}
+
+static const char *armada_37xx_pmx_get_func_name(struct pinctrl_dev *pctldev,
+ unsigned int selector)
+{
+ struct armada_37xx_pinctrl *info = pinctrl_dev_get_drvdata(pctldev);
+
+ return info->funcs[selector].name;
+}
+
+static int armada_37xx_pmx_get_groups(struct pinctrl_dev *pctldev,
+ unsigned int selector,
+ const char * const **groups,
+ unsigned int * const num_groups)
+{
+ struct armada_37xx_pinctrl *info = pinctrl_dev_get_drvdata(pctldev);
+
+ *groups = info->funcs[selector].groups;
+ *num_groups = info->funcs[selector].ngroups;
+
+ return 0;
+}
+
+static int armada_37xx_pmx_set_by_name(struct pinctrl_dev *pctldev,
+ const char *name,
+ struct armada_37xx_pin_group *grp)
+{
+ struct armada_37xx_pinctrl *info = pinctrl_dev_get_drvdata(pctldev);
+ unsigned int reg = SELECTION;
+ unsigned int mask = grp->reg_mask;
+ int func, val;
+
+ dev_dbg(info->dev, "enable function %s group %s\n",
+ name, grp->name);
+
+ func = armada_37xx_get_func_reg(grp, name);
+
+ if (func < 0)
+ return func;
+
+ val = grp->val[func];
+
+ regmap_update_bits(info->regmap, reg, mask, val);
+
+ return 0;
+}
+
+static int armada_37xx_pmx_set(struct pinctrl_dev *pctldev,
+ unsigned int selector,
+ unsigned int group)
+{
+
+ struct armada_37xx_pinctrl *info = pinctrl_dev_get_drvdata(pctldev);
+ struct armada_37xx_pin_group *grp = &info->groups[group];
+ const char *name = info->funcs[selector].name;
+
+ return armada_37xx_pmx_set_by_name(pctldev, name, grp);
+}
+
+static int armada_37xx_gpio_direction_input(struct gpio_chip *chip,
+ unsigned int offset)
+{
+ struct armada_37xx_pinctrl *info = gpiochip_get_data(chip);
+ unsigned int reg = OUTPUT_EN;
+ unsigned int mask;
+
+ armada_37xx_update_reg(®, offset);
+ mask = BIT(offset);
+
+ return regmap_update_bits(info->regmap, reg, mask, 0);
+}
+
+static int armada_37xx_gpio_get_direction(struct gpio_chip *chip,
+ unsigned int offset)
+{
+ struct armada_37xx_pinctrl *info = gpiochip_get_data(chip);
+ unsigned int reg = OUTPUT_EN;
+ unsigned int val, mask;
+
+ armada_37xx_update_reg(®, offset);
+ mask = BIT(offset);
+ regmap_read(info->regmap, reg, &val);
+
+ return !(val & mask);
+}
+
+static int armada_37xx_gpio_direction_output(struct gpio_chip *chip,
+ unsigned int offset, int value)
+{
+ struct armada_37xx_pinctrl *info = gpiochip_get_data(chip);
+ unsigned int reg = OUTPUT_EN;
+ unsigned int mask;
+
+ armada_37xx_update_reg(®, offset);
+ mask = BIT(offset);
+
+ return regmap_update_bits(info->regmap, reg, mask, mask);
+}
+
+static int armada_37xx_gpio_get(struct gpio_chip *chip, unsigned int offset)
+{
+ struct armada_37xx_pinctrl *info = gpiochip_get_data(chip);
+ unsigned int reg = INPUT_VAL;
+ unsigned int val, mask;
+
+ armada_37xx_update_reg(®, offset);
+ mask = BIT(offset);
+
+ regmap_read(info->regmap, reg, &val);
+
+ return (val & mask) != 0;
+}
+
+static void armada_37xx_gpio_set(struct gpio_chip *chip, unsigned int offset,
+ int value)
+{
+ struct armada_37xx_pinctrl *info = gpiochip_get_data(chip);
+ unsigned int reg = OUTPUT_VAL;
+ unsigned int mask, val;
+
+ armada_37xx_update_reg(®, offset);
+ mask = BIT(offset);
+ val = value ? mask : 0;
+
+ regmap_update_bits(info->regmap, reg, mask, val);
+}
+
+static int armada_37xx_pmx_gpio_set_direction(struct pinctrl_dev *pctldev,
+ struct pinctrl_gpio_range *range,
+ unsigned int offset, bool input)
+{
+ struct armada_37xx_pinctrl *info = pinctrl_dev_get_drvdata(pctldev);
+ struct gpio_chip *chip = range->gc;
+
+ dev_dbg(info->dev, "gpio_direction for pin %u as %s-%d to %s\n",
+ offset, range->name, offset, input ? "input" : "output");
+
+ if (input)
+ armada_37xx_gpio_direction_input(chip, offset);
+ else
+ armada_37xx_gpio_direction_output(chip, offset, 0);
+
+ return 0;
+}
+
+static int armada_37xx_gpio_request_enable(struct pinctrl_dev *pctldev,
+ struct pinctrl_gpio_range *range,
+ unsigned int offset)
+{
+ struct armada_37xx_pinctrl *info = pinctrl_dev_get_drvdata(pctldev);
+ struct armada_37xx_pin_group *group;
+ int grp = 0;
+
+ dev_dbg(info->dev, "requesting gpio %d\n", offset);
+
+ while ((group = armada_37xx_find_next_grp_by_pin(info, offset, &grp)))
+ armada_37xx_pmx_set_by_name(pctldev, "gpio", group);
+
+ return 0;
+}
+
+static const struct pinmux_ops armada_37xx_pmx_ops = {
+ .get_functions_count = armada_37xx_pmx_get_funcs_count,
+ .get_function_name = armada_37xx_pmx_get_func_name,
+ .get_function_groups = armada_37xx_pmx_get_groups,
+ .set_mux = armada_37xx_pmx_set,
+ .gpio_request_enable = armada_37xx_gpio_request_enable,
+ .gpio_set_direction = armada_37xx_pmx_gpio_set_direction,
+};
+
+static const struct gpio_chip armada_37xx_gpiolib_chip = {
+ .request = gpiochip_generic_request,
+ .free = gpiochip_generic_free,
+ .set = armada_37xx_gpio_set,
+ .get = armada_37xx_gpio_get,
+ .get_direction = armada_37xx_gpio_get_direction,
+ .direction_input = armada_37xx_gpio_direction_input,
+ .direction_output = armada_37xx_gpio_direction_output,
+ .owner = THIS_MODULE,
+};
+
+static int armada_37xx_gpiochip_register(struct platform_device *pdev,
+ struct armada_37xx_pinctrl *info)
+{
+ struct device_node *np;
+ struct gpio_chip *gc;
+ int ret = -ENODEV;
+
+ for_each_child_of_node(info->dev->of_node, np) {
+ if (of_find_property(np, "gpio-controller", NULL)) {
+ ret = 0;
+ break;
+ }
+ };
+ if (ret)
+ return ret;
+
+ info->gpio_chip = armada_37xx_gpiolib_chip;
+
+ gc = &info->gpio_chip;
+ gc->ngpio = info->data->nr_pins;
+ gc->parent = &pdev->dev;
+ gc->base = -1;
+ gc->of_node = np;
+ gc->label = info->data->name;
+
+ ret = devm_gpiochip_add_data(&pdev->dev, gc, info);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+/**
+ * armada_37xx_add_function() - Add a new function to the list
+ * @funcs: array of function to add the new one
+ * @funcsize: size of the remaining space for the function
+ * @name: name of the function to add
+ *
+ * If it is a new function then create it by adding its name else
+ * increment the number of group associated to this function.
+ */
+static int armada_37xx_add_function(struct armada_37xx_pmx_func *funcs,
+ int *funcsize, const char *name)
+{
+ int i = 0;
+
+ if (*funcsize <= 0)
+ return -EOVERFLOW;
+
+ while (funcs->ngroups) {
+ /* function already there */
+ if (strcmp(funcs->name, name) == 0) {
+ funcs->ngroups++;
+
+ return -EEXIST;
+ }
+ funcs++;
+ i++;
+ }
+
+ /* append new unique function */
+ funcs->name = name;
+ funcs->ngroups = 1;
+ (*funcsize)--;
+
+ return 0;
+}
+
+/**
+ * armada_37xx_fill_group() - complete the group array
+ * @info: info driver instance
+ *
+ * Based on the data available from the armada_37xx_pin_group array
+ * completes the last member of the struct for each function: the list
+ * of the groups associated to this function.
+ *
+ */
+static int armada_37xx_fill_group(struct armada_37xx_pinctrl *info)
+{
+ int n, num = 0, funcsize = info->data->nr_pins;
+
+ for (n = 0; n < info->ngroups; n++) {
+ struct armada_37xx_pin_group *grp = &info->groups[n];
+ int i, j, f;
+
+ grp->pins = devm_kzalloc(info->dev,
+ (grp->npins + grp->extra_npins) *
+ sizeof(*grp->pins), GFP_KERNEL);
+ if (!grp->pins)
+ return -ENOMEM;
+
+ for (i = 0; i < grp->npins; i++)
+ grp->pins[i] = grp->start_pin + i;
+
+ for (j = 0; j < grp->extra_npins; j++)
+ grp->pins[i+j] = grp->extra_pin + j;
+
+ for (f = 0; f < NB_FUNCS; f++) {
+ int ret;
+ /* check for unique functions and count groups */
+ ret = armada_37xx_add_function(info->funcs, &funcsize,
+ grp->funcs[f]);
+ if (ret == -EOVERFLOW)
+ dev_err(info->dev,
+ "More functions than pins(%d)\n",
+ info->data->nr_pins);
+ if (ret < 0)
+ continue;
+ num++;
+ }
+ }
+
+ info->nfuncs = num;
+
+ return 0;
+}
+
+/**
+ * armada_37xx_fill_funcs() - complete the funcs array
+ * @info: info driver instance
+ *
+ * Based on the data available from the armada_37xx_pin_group array
+ * completes the last two member of the struct for each group:
+ * - the list of the pins included in the group
+ * - the list of pinmux functions that can be selected for this group
+ *
+ */
+static int armada_37xx_fill_func(struct armada_37xx_pinctrl *info)
+{
+ struct armada_37xx_pmx_func *funcs = info->funcs;
+ int n;
+
+ for (n = 0; n < info->nfuncs; n++) {
+ const char *name = funcs[n].name;
+ const char **groups;
+ int g;
+
+ funcs[n].groups = devm_kzalloc(info->dev, funcs[n].ngroups *
+ sizeof(*(funcs[n].groups)),
+ GFP_KERNEL);
+ if (!funcs[n].groups)
+ return -ENOMEM;
+
+ groups = funcs[n].groups;
+
+ for (g = 0; g < info->ngroups; g++) {
+ struct armada_37xx_pin_group *gp = &info->groups[g];
+ int f;
+
+ for (f = 0; f < NB_FUNCS; f++) {
+ if (strcmp(gp->funcs[f], name) == 0) {
+ *groups = gp->name;
+ groups++;
+ }
+ }
+ }
+ }
+ return 0;
+}
+
+static int armada_37xx_pinctrl_register(struct platform_device *pdev,
+ struct armada_37xx_pinctrl *info)
+{
+ const struct armada_37xx_pin_data *pin_data = info->data;
+ struct pinctrl_desc *ctrldesc = &info->pctl;
+ struct pinctrl_pin_desc *pindesc, *pdesc;
+ int pin, ret;
+
+ info->groups = pin_data->groups;
+ info->ngroups = pin_data->ngroups;
+
+ ctrldesc->name = "armada_37xx-pinctrl";
+ ctrldesc->owner = THIS_MODULE;
+ ctrldesc->pctlops = &armada_37xx_pctrl_ops;
+ ctrldesc->pmxops = &armada_37xx_pmx_ops;
+ ctrldesc->confops = &armada_37xx_pinconf_ops;
+
+ pindesc = devm_kzalloc(&pdev->dev, sizeof(*pindesc) *
+ pin_data->nr_pins, GFP_KERNEL);
+ if (!pindesc)
+ return -ENOMEM;
+
+ ctrldesc->pins = pindesc;
+ ctrldesc->npins = pin_data->nr_pins;
+
+ pdesc = pindesc;
+ for (pin = 0; pin < pin_data->nr_pins; pin++) {
+ pdesc->number = pin;
+ pdesc->name = kasprintf(GFP_KERNEL, "%s-%d",
+ pin_data->name, pin);
+ pdesc++;
+ }
+
+ /*
+ * we allocate functions for number of pins and hope there are
+ * fewer unique functions than pins available
+ */
+ info->funcs = devm_kzalloc(&pdev->dev, pin_data->nr_pins *
+ sizeof(struct armada_37xx_pmx_func), GFP_KERNEL);
+ if (!info->funcs)
+ return -ENOMEM;
+
+
+ ret = armada_37xx_fill_group(info);
+ if (ret)
+ return ret;
+
+ ret = armada_37xx_fill_func(info);
+ if (ret)
+ return ret;
+
+ info->pctl_dev = devm_pinctrl_register(&pdev->dev, ctrldesc, info);
+ if (IS_ERR(info->pctl_dev)) {
+ dev_err(&pdev->dev, "could not register pinctrl driver\n");
+ return PTR_ERR(info->pctl_dev);
+ }
+
+ return 0;
+}
+
+static const struct of_device_id armada_37xx_pinctrl_of_match[] = {
+ {
+ .compatible = "marvell,armada3710-sb-pinctrl",
+ .data = (void *)&armada_37xx_pin_sb,
+ },
+ {
+ .compatible = "marvell,armada3710-nb-pinctrl",
+ .data = (void *)&armada_37xx_pin_nb,
+ },
+ { },
+};
+
+static int __init armada_37xx_pinctrl_probe(struct platform_device *pdev)
+{
+ struct armada_37xx_pinctrl *info;
+ struct device *dev = &pdev->dev;
+ struct device_node *np = dev->of_node;
+ struct regmap *regmap;
+ int ret;
+
+ info = devm_kzalloc(dev, sizeof(struct armada_37xx_pinctrl),
+ GFP_KERNEL);
+ if (!info)
+ return -ENOMEM;
+
+ info->dev = dev;
+
+ regmap = syscon_node_to_regmap(np);
+ if (IS_ERR(regmap)) {
+ dev_err(&pdev->dev, "cannot get regmap\n");
+ return PTR_ERR(regmap);
+ }
+ info->regmap = regmap;
+
+ info->data = of_device_get_match_data(dev);
+
+ ret = armada_37xx_pinctrl_register(pdev, info);
+ if (ret)
+ return ret;
+
+ ret = armada_37xx_gpiochip_register(pdev, info);
+ if (ret)
+ return ret;
+
+ platform_set_drvdata(pdev, info);
+
+ return 0;
+}
+
+static struct platform_driver armada_37xx_pinctrl_driver = {
+ .driver = {
+ .name = "armada-37xx-pinctrl",
+ .of_match_table = armada_37xx_pinctrl_of_match,
+ },
+};
+
+builtin_platform_driver_probe(armada_37xx_pinctrl_driver,
+ armada_37xx_pinctrl_probe);
diff --git a/drivers/pinctrl/pinconf-generic.c b/drivers/pinctrl/pinconf-generic.c
index ce3335a..0d6b7f4 100644
--- a/drivers/pinctrl/pinconf-generic.c
+++ b/drivers/pinctrl/pinconf-generic.c
@@ -35,6 +35,7 @@
PCONFDUMP(PIN_CONFIG_BIAS_PULL_PIN_DEFAULT,
"input bias pull to pin specific state", NULL, false),
PCONFDUMP(PIN_CONFIG_BIAS_PULL_UP, "input bias pull up", NULL, false),
+ PCONFDUMP(PIN_CONFIG_BIDIRECTIONAL, "bi-directional pin operations", NULL, false),
PCONFDUMP(PIN_CONFIG_DRIVE_OPEN_DRAIN, "output drive open drain", NULL, false),
PCONFDUMP(PIN_CONFIG_DRIVE_OPEN_SOURCE, "output drive open source", NULL, false),
PCONFDUMP(PIN_CONFIG_DRIVE_PUSH_PULL, "output drive push pull", NULL, false),
@@ -160,6 +161,7 @@
{ "bias-pull-up", PIN_CONFIG_BIAS_PULL_UP, 1 },
{ "bias-pull-pin-default", PIN_CONFIG_BIAS_PULL_PIN_DEFAULT, 1 },
{ "bias-pull-down", PIN_CONFIG_BIAS_PULL_DOWN, 1 },
+ { "bi-directional", PIN_CONFIG_BIDIRECTIONAL, 1 },
{ "drive-open-drain", PIN_CONFIG_DRIVE_OPEN_DRAIN, 0 },
{ "drive-open-source", PIN_CONFIG_DRIVE_OPEN_SOURCE, 0 },
{ "drive-push-pull", PIN_CONFIG_DRIVE_PUSH_PULL, 0 },
@@ -172,6 +174,7 @@
{ "input-schmitt-enable", PIN_CONFIG_INPUT_SCHMITT_ENABLE, 1 },
{ "low-power-disable", PIN_CONFIG_LOW_POWER_MODE, 0 },
{ "low-power-enable", PIN_CONFIG_LOW_POWER_MODE, 1 },
+ { "output-enable", PIN_CONFIG_OUTPUT, 1, },
{ "output-high", PIN_CONFIG_OUTPUT, 1, },
{ "output-low", PIN_CONFIG_OUTPUT, 0, },
{ "power-source", PIN_CONFIG_POWER_SOURCE, 0 },
@@ -187,7 +190,7 @@
* @ncfg: Number of entries in @cfg
*
* Parse the config options described in @params from @np and puts the result
- * in @cfg. @cfg does not need to be empty, entries are added beggining at
+ * in @cfg. @cfg does not need to be empty, entries are added beginning at
* @ncfg. @ncfg is updated to reflect the number of entries after parsing. @cfg
* needs to have enough memory allocated to hold all possible entries.
*/
diff --git a/drivers/pinctrl/pinconf.c b/drivers/pinctrl/pinconf.c
index c1c1ccc..a02dba3 100644
--- a/drivers/pinctrl/pinconf.c
+++ b/drivers/pinctrl/pinconf.c
@@ -284,7 +284,7 @@
}
/*
- * FIXME: We should really get the pin controler to dump the config
+ * FIXME: We should really get the pin controller to dump the config
* values, so they can be decoded to something meaningful.
*/
pinconf_show_config(s, pctldev, setting->data.configs.configs,
@@ -473,7 +473,7 @@
* "config_pin" or "config_group", alternatives like config_mux are not
* supported yet.
* <devicename> <state> <name> are values that should match the pinctrl-maps
- * <newvalue> reflects the new config and is driver dependant
+ * <newvalue> reflects the new config and is driver dependent
*/
static ssize_t pinconf_dbg_config_write(struct file *file,
const char __user *user_buf, size_t count, loff_t *ppos)
diff --git a/drivers/pinctrl/pinctrl-amd.c b/drivers/pinctrl/pinctrl-amd.c
index d69e357..1482d13 100644
--- a/drivers/pinctrl/pinctrl-amd.c
+++ b/drivers/pinctrl/pinctrl-amd.c
@@ -41,11 +41,11 @@
u32 pin_reg;
struct amd_gpio *gpio_dev = gpiochip_get_data(gc);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
pin_reg = readl(gpio_dev->base + offset * 4);
pin_reg &= ~BIT(OUTPUT_ENABLE_OFF);
writel(pin_reg, gpio_dev->base + offset * 4);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
return 0;
}
@@ -57,7 +57,7 @@
unsigned long flags;
struct amd_gpio *gpio_dev = gpiochip_get_data(gc);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
pin_reg = readl(gpio_dev->base + offset * 4);
pin_reg |= BIT(OUTPUT_ENABLE_OFF);
if (value)
@@ -65,7 +65,7 @@
else
pin_reg &= ~BIT(OUTPUT_VALUE_OFF);
writel(pin_reg, gpio_dev->base + offset * 4);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
return 0;
}
@@ -76,9 +76,9 @@
unsigned long flags;
struct amd_gpio *gpio_dev = gpiochip_get_data(gc);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
pin_reg = readl(gpio_dev->base + offset * 4);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
return !!(pin_reg & BIT(PIN_STS_OFF));
}
@@ -89,14 +89,14 @@
unsigned long flags;
struct amd_gpio *gpio_dev = gpiochip_get_data(gc);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
pin_reg = readl(gpio_dev->base + offset * 4);
if (value)
pin_reg |= BIT(OUTPUT_VALUE_OFF);
else
pin_reg &= ~BIT(OUTPUT_VALUE_OFF);
writel(pin_reg, gpio_dev->base + offset * 4);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
}
static int amd_gpio_set_debounce(struct gpio_chip *gc, unsigned offset,
@@ -108,7 +108,7 @@
unsigned long flags;
struct amd_gpio *gpio_dev = gpiochip_get_data(gc);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
pin_reg = readl(gpio_dev->base + offset * 4);
if (debounce) {
@@ -159,7 +159,7 @@
pin_reg &= ~DB_CNTRl_MASK;
}
writel(pin_reg, gpio_dev->base + offset * 4);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
return ret;
}
@@ -224,9 +224,9 @@
}
for (; i < pin_num; i++) {
seq_printf(s, "pin%d\t", i);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
pin_reg = readl(gpio_dev->base + i * 4);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
if (pin_reg & BIT(INTERRUPT_ENABLE_OFF)) {
interrupt_enable = "interrupt is enabled|";
@@ -241,7 +241,7 @@
pin_reg & BIT(ACTIVE_LEVEL_OFF + 1))
active_level = "Active on both|";
else
- active_level = "Unknow Active level|";
+ active_level = "Unknown Active level|";
if (pin_reg & BIT(LEVEL_TRIG_OFF))
level_trig = "Level trigger|";
@@ -331,12 +331,12 @@
struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
struct amd_gpio *gpio_dev = gpiochip_get_data(gc);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
pin_reg = readl(gpio_dev->base + (d->hwirq)*4);
pin_reg |= BIT(INTERRUPT_ENABLE_OFF);
pin_reg |= BIT(INTERRUPT_MASK_OFF);
writel(pin_reg, gpio_dev->base + (d->hwirq)*4);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
}
static void amd_gpio_irq_disable(struct irq_data *d)
@@ -346,12 +346,12 @@
struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
struct amd_gpio *gpio_dev = gpiochip_get_data(gc);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
pin_reg = readl(gpio_dev->base + (d->hwirq)*4);
pin_reg &= ~BIT(INTERRUPT_ENABLE_OFF);
pin_reg &= ~BIT(INTERRUPT_MASK_OFF);
writel(pin_reg, gpio_dev->base + (d->hwirq)*4);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
}
static void amd_gpio_irq_mask(struct irq_data *d)
@@ -361,11 +361,11 @@
struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
struct amd_gpio *gpio_dev = gpiochip_get_data(gc);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
pin_reg = readl(gpio_dev->base + (d->hwirq)*4);
pin_reg &= ~BIT(INTERRUPT_MASK_OFF);
writel(pin_reg, gpio_dev->base + (d->hwirq)*4);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
}
static void amd_gpio_irq_unmask(struct irq_data *d)
@@ -375,11 +375,11 @@
struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
struct amd_gpio *gpio_dev = gpiochip_get_data(gc);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
pin_reg = readl(gpio_dev->base + (d->hwirq)*4);
pin_reg |= BIT(INTERRUPT_MASK_OFF);
writel(pin_reg, gpio_dev->base + (d->hwirq)*4);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
}
static void amd_gpio_irq_eoi(struct irq_data *d)
@@ -389,11 +389,11 @@
struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
struct amd_gpio *gpio_dev = gpiochip_get_data(gc);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
reg = readl(gpio_dev->base + WAKE_INT_MASTER_REG);
reg |= EOI_MASK;
writel(reg, gpio_dev->base + WAKE_INT_MASTER_REG);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
}
static int amd_gpio_irq_set_type(struct irq_data *d, unsigned int type)
@@ -404,7 +404,7 @@
struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
struct amd_gpio *gpio_dev = gpiochip_get_data(gc);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
pin_reg = readl(gpio_dev->base + (d->hwirq)*4);
/* Ignore the settings coming from the client and
@@ -469,7 +469,7 @@
pin_reg |= CLR_INTR_STAT << INTERRUPT_STS_OFF;
writel(pin_reg, gpio_dev->base + (d->hwirq)*4);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
return ret;
}
@@ -511,14 +511,14 @@
chained_irq_enter(chip, desc);
/*enable GPIO interrupt again*/
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
reg = readl(gpio_dev->base + WAKE_INT_STATUS_REG1);
reg64 = reg;
reg64 = reg64 << 32;
reg = readl(gpio_dev->base + WAKE_INT_STATUS_REG0);
reg64 |= reg;
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
/*
* first 46 bits indicates interrupt status.
@@ -546,11 +546,11 @@
if (handled == 0)
handle_bad_irq(desc);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
reg = readl(gpio_dev->base + WAKE_INT_MASTER_REG);
reg |= EOI_MASK;
writel(reg, gpio_dev->base + WAKE_INT_MASTER_REG);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
chained_irq_exit(chip, desc);
}
@@ -602,9 +602,9 @@
struct amd_gpio *gpio_dev = pinctrl_dev_get_drvdata(pctldev);
enum pin_config_param param = pinconf_to_config_param(*config);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
pin_reg = readl(gpio_dev->base + pin*4);
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
switch (param) {
case PIN_CONFIG_INPUT_DEBOUNCE:
arg = pin_reg & DB_TMR_OUT_MASK;
@@ -644,7 +644,7 @@
enum pin_config_param param;
struct amd_gpio *gpio_dev = pinctrl_dev_get_drvdata(pctldev);
- spin_lock_irqsave(&gpio_dev->lock, flags);
+ raw_spin_lock_irqsave(&gpio_dev->lock, flags);
for (i = 0; i < num_configs; i++) {
param = pinconf_to_config_param(configs[i]);
arg = pinconf_to_config_argument(configs[i]);
@@ -683,7 +683,7 @@
writel(pin_reg, gpio_dev->base + pin*4);
}
- spin_unlock_irqrestore(&gpio_dev->lock, flags);
+ raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
return ret;
}
@@ -751,7 +751,7 @@
if (!gpio_dev)
return -ENOMEM;
- spin_lock_init(&gpio_dev->lock);
+ raw_spin_lock_init(&gpio_dev->lock);
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
if (!res) {
diff --git a/drivers/pinctrl/pinctrl-amd.h b/drivers/pinctrl/pinctrl-amd.h
index c03f778..5b1cb96 100644
--- a/drivers/pinctrl/pinctrl-amd.h
+++ b/drivers/pinctrl/pinctrl-amd.h
@@ -87,7 +87,7 @@
};
struct amd_gpio {
- spinlock_t lock;
+ raw_spinlock_t lock;
void __iomem *base;
const struct amd_pingroup *groups;
diff --git a/drivers/pinctrl/pinctrl-artpec6.c b/drivers/pinctrl/pinctrl-artpec6.c
new file mode 100644
index 0000000..357516d
--- /dev/null
+++ b/drivers/pinctrl/pinctrl-artpec6.c
@@ -0,0 +1,979 @@
+/*
+ * Driver for the Axis ARTPEC-6 pin controller
+ *
+ * Author: Chris Paterson <chris.paterson@linux.pieboy.co.uk>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/io.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/pinctrl/pinctrl.h>
+#include <linux/pinctrl/pinconf-generic.h>
+#include <linux/pinctrl/pinconf.h>
+#include <linux/pinctrl/pinmux.h>
+#include <linux/slab.h>
+#include "core.h"
+#include "pinconf.h"
+#include "pinctrl-utils.h"
+
+#define ARTPEC6_LAST_PIN 97 /* 97 pins in pinmux */
+#define ARTPEC6_MAX_MUXABLE 35 /* Last pin with muxable function */
+
+/* Pinmux control register bit definitions */
+#define ARTPEC6_PINMUX_UDC0_MASK 0x00000001
+#define ARTPEC6_PINMUX_UDC0_SHIFT 0
+#define ARTPEC6_PINMUX_UDC1_MASK 0x00000002
+#define ARTPEC6_PINMUX_UDC1_SHIFT 1
+#define ARTPEC6_PINMUX_DRV_MASK 0x00000060
+#define ARTPEC6_PINMUX_DRV_SHIFT 5
+#define ARTPEC6_PINMUX_SEL_MASK 0x00003000
+#define ARTPEC6_PINMUX_SEL_SHIFT 12
+
+/* Pinmux configurations */
+#define ARTPEC6_CONFIG_0 0
+#define ARTPEC6_CONFIG_1 1
+#define ARTPEC6_CONFIG_2 2
+#define ARTPEC6_CONFIG_3 3
+
+/* Pin drive strength options */
+#define ARTPEC6_DRIVE_4mA 4
+#define ARTPEC6_DRIVE_4mA_SET 0
+#define ARTPEC6_DRIVE_6mA 6
+#define ARTPEC6_DRIVE_6mA_SET 1
+#define ARTPEC6_DRIVE_8mA 8
+#define ARTPEC6_DRIVE_8mA_SET 2
+#define ARTPEC6_DRIVE_9mA 9
+#define ARTPEC6_DRIVE_9mA_SET 3
+
+struct artpec6_pmx {
+ struct device *dev;
+ struct pinctrl_dev *pctl;
+ void __iomem *base;
+ struct pinctrl_pin_desc *pins;
+ unsigned int num_pins;
+ const struct artpec6_pin_group *pin_groups;
+ unsigned int num_pin_groups;
+ const struct artpec6_pmx_func *functions;
+ unsigned int num_functions;
+};
+
+struct artpec6_pin_group {
+ const char *name;
+ const unsigned int *pins;
+ const unsigned int num_pins;
+ unsigned char config;
+};
+
+struct artpec6_pmx_func {
+ const char *name;
+ const char * const *groups;
+ const unsigned int num_groups;
+};
+
+/* pins */
+static struct pinctrl_pin_desc artpec6_pins[] = {
+ PINCTRL_PIN(0, "GPIO0"),
+ PINCTRL_PIN(1, "GPIO1"),
+ PINCTRL_PIN(2, "GPIO2"),
+ PINCTRL_PIN(3, "GPIO3"),
+ PINCTRL_PIN(4, "GPIO4"),
+ PINCTRL_PIN(5, "GPIO5"),
+ PINCTRL_PIN(6, "GPIO6"),
+ PINCTRL_PIN(7, "GPIO7"),
+ PINCTRL_PIN(8, "GPIO8"),
+ PINCTRL_PIN(9, "GPIO9"),
+ PINCTRL_PIN(10, "GPIO10"),
+ PINCTRL_PIN(11, "GPIO11"),
+ PINCTRL_PIN(12, "GPIO12"),
+ PINCTRL_PIN(13, "GPIO13"),
+ PINCTRL_PIN(14, "GPIO14"),
+ PINCTRL_PIN(15, "GPIO15"),
+ PINCTRL_PIN(16, "GPIO16"),
+ PINCTRL_PIN(17, "GPIO17"),
+ PINCTRL_PIN(18, "GPIO18"),
+ PINCTRL_PIN(19, "GPIO19"),
+ PINCTRL_PIN(20, "GPIO20"),
+ PINCTRL_PIN(21, "GPIO21"),
+ PINCTRL_PIN(22, "GPIO22"),
+ PINCTRL_PIN(23, "GPIO23"),
+ PINCTRL_PIN(24, "GPIO24"),
+ PINCTRL_PIN(25, "GPIO25"),
+ PINCTRL_PIN(26, "GPIO26"),
+ PINCTRL_PIN(27, "GPIO27"),
+ PINCTRL_PIN(28, "GPIO28"),
+ PINCTRL_PIN(29, "GPIO29"),
+ PINCTRL_PIN(30, "GPIO30"),
+ PINCTRL_PIN(31, "GPIO31"),
+ PINCTRL_PIN(32, "UART3_TXD"),
+ PINCTRL_PIN(33, "UART3_RXD"),
+ PINCTRL_PIN(34, "UART3_RTS"),
+ PINCTRL_PIN(35, "UART3_CTS"),
+ PINCTRL_PIN(36, "NF_ALE"),
+ PINCTRL_PIN(37, "NF_CE0_N"),
+ PINCTRL_PIN(38, "NF_CE1_N"),
+ PINCTRL_PIN(39, "NF_CLE"),
+ PINCTRL_PIN(40, "NF_RE_N"),
+ PINCTRL_PIN(41, "NF_WE_N"),
+ PINCTRL_PIN(42, "NF_WP0_N"),
+ PINCTRL_PIN(43, "NF_WP1_N"),
+ PINCTRL_PIN(44, "NF_IO0"),
+ PINCTRL_PIN(45, "NF_IO1"),
+ PINCTRL_PIN(46, "NF_IO2"),
+ PINCTRL_PIN(47, "NF_IO3"),
+ PINCTRL_PIN(48, "NF_IO4"),
+ PINCTRL_PIN(49, "NF_IO5"),
+ PINCTRL_PIN(50, "NF_IO6"),
+ PINCTRL_PIN(51, "NF_IO7"),
+ PINCTRL_PIN(52, "NF_RB0_N"),
+ PINCTRL_PIN(53, "SDIO0_CLK"),
+ PINCTRL_PIN(54, "SDIO0_CMD"),
+ PINCTRL_PIN(55, "SDIO0_DAT0"),
+ PINCTRL_PIN(56, "SDIO0_DAT1"),
+ PINCTRL_PIN(57, "SDIO0_DAT2"),
+ PINCTRL_PIN(58, "SDIO0_DAT3"),
+ PINCTRL_PIN(59, "SDI0_CD"),
+ PINCTRL_PIN(60, "SDI0_WP"),
+ PINCTRL_PIN(61, "SDIO1_CLK"),
+ PINCTRL_PIN(62, "SDIO1_CMD"),
+ PINCTRL_PIN(63, "SDIO1_DAT0"),
+ PINCTRL_PIN(64, "SDIO1_DAT1"),
+ PINCTRL_PIN(65, "SDIO1_DAT2"),
+ PINCTRL_PIN(66, "SDIO1_DAT3"),
+ PINCTRL_PIN(67, "SDIO1_CD"),
+ PINCTRL_PIN(68, "SDIO1_WP"),
+ PINCTRL_PIN(69, "GBE_REFCLk"),
+ PINCTRL_PIN(70, "GBE_GTX_CLK"),
+ PINCTRL_PIN(71, "GBE_TX_CLK"),
+ PINCTRL_PIN(72, "GBE_TX_EN"),
+ PINCTRL_PIN(73, "GBE_TX_ER"),
+ PINCTRL_PIN(74, "GBE_TXD0"),
+ PINCTRL_PIN(75, "GBE_TXD1"),
+ PINCTRL_PIN(76, "GBE_TXD2"),
+ PINCTRL_PIN(77, "GBE_TXD3"),
+ PINCTRL_PIN(78, "GBE_TXD4"),
+ PINCTRL_PIN(79, "GBE_TXD5"),
+ PINCTRL_PIN(80, "GBE_TXD6"),
+ PINCTRL_PIN(81, "GBE_TXD7"),
+ PINCTRL_PIN(82, "GBE_RX_CLK"),
+ PINCTRL_PIN(83, "GBE_RX_DV"),
+ PINCTRL_PIN(84, "GBE_RX_ER"),
+ PINCTRL_PIN(85, "GBE_RXD0"),
+ PINCTRL_PIN(86, "GBE_RXD1"),
+ PINCTRL_PIN(87, "GBE_RXD2"),
+ PINCTRL_PIN(88, "GBE_RXD3"),
+ PINCTRL_PIN(89, "GBE_RXD4"),
+ PINCTRL_PIN(90, "GBE_RXD5"),
+ PINCTRL_PIN(91, "GBE_RXD6"),
+ PINCTRL_PIN(92, "GBE_RXD7"),
+ PINCTRL_PIN(93, "GBE_CRS"),
+ PINCTRL_PIN(94, "GBE_COL"),
+ PINCTRL_PIN(95, "GBE_MDC"),
+ PINCTRL_PIN(96, "GBE_MDIO"),
+};
+
+static const unsigned int cpuclkout_pins0[] = { 0 };
+static const unsigned int udlclkout_pins0[] = { 1 };
+static const unsigned int i2c1_pins0[] = { 2, 3 };
+static const unsigned int i2c2_pins0[] = { 4, 5 };
+static const unsigned int i2c3_pins0[] = { 6, 7 };
+static const unsigned int i2s0_pins0[] = { 8, 9, 10, 11 };
+static const unsigned int i2s1_pins0[] = { 12, 13, 14, 15 };
+static const unsigned int i2srefclk_pins0[] = { 19 };
+static const unsigned int spi0_pins0[] = { 12, 13, 14, 15 };
+static const unsigned int spi1_pins0[] = { 16, 17, 18, 19 };
+static const unsigned int pciedebug_pins0[] = { 12, 13, 14, 15 };
+static const unsigned int uart0_pins0[] = { 16, 17, 18, 19, 20,
+ 21, 22, 23, 24, 25 };
+static const unsigned int uart0_pins1[] = { 20, 21, 22, 23 };
+static const unsigned int uart1_pins0[] = { 24, 25, 26, 27 };
+static const unsigned int uart2_pins0[] = { 26, 27, 28, 29, 30,
+ 31, 32, 33, 34, 35 };
+static const unsigned int uart2_pins1[] = { 28, 29, 30, 31 };
+static const unsigned int uart3_pins0[] = { 32, 33, 34, 35 };
+static const unsigned int uart4_pins0[] = { 20, 21, 22, 23 };
+static const unsigned int uart5_pins0[] = { 28, 29, 30, 31 };
+static const unsigned int nand_pins0[] = { 36, 37, 38, 39, 40, 41,
+ 42, 43, 44, 45, 46, 47,
+ 48, 49, 50, 51, 52 };
+static const unsigned int sdio0_pins0[] = { 53, 54, 55, 56, 57, 58, 59, 60 };
+static const unsigned int sdio1_pins0[] = { 61, 62, 63, 64, 65, 66, 67, 68 };
+static const unsigned int ethernet_pins0[] = { 69, 70, 71, 72, 73, 74, 75,
+ 76, 77, 78, 79, 80, 81, 82,
+ 83, 84, 85, 86, 87, 88, 89,
+ 90, 91, 92, 93, 94, 95, 96 };
+
+static const struct artpec6_pin_group artpec6_pin_groups[] = {
+ {
+ .name = "cpuclkoutgrp0",
+ .pins = cpuclkout_pins0,
+ .num_pins = ARRAY_SIZE(cpuclkout_pins0),
+ .config = ARTPEC6_CONFIG_1,
+ },
+ {
+ .name = "udlclkoutgrp0",
+ .pins = udlclkout_pins0,
+ .num_pins = ARRAY_SIZE(udlclkout_pins0),
+ .config = ARTPEC6_CONFIG_1,
+ },
+ {
+ .name = "i2c1grp0",
+ .pins = i2c1_pins0,
+ .num_pins = ARRAY_SIZE(i2c1_pins0),
+ .config = ARTPEC6_CONFIG_1,
+ },
+ {
+ .name = "i2c2grp0",
+ .pins = i2c2_pins0,
+ .num_pins = ARRAY_SIZE(i2c2_pins0),
+ .config = ARTPEC6_CONFIG_1,
+ },
+ {
+ .name = "i2c3grp0",
+ .pins = i2c3_pins0,
+ .num_pins = ARRAY_SIZE(i2c3_pins0),
+ .config = ARTPEC6_CONFIG_1,
+ },
+ {
+ .name = "i2s0grp0",
+ .pins = i2s0_pins0,
+ .num_pins = ARRAY_SIZE(i2s0_pins0),
+ .config = ARTPEC6_CONFIG_1,
+ },
+ {
+ .name = "i2s1grp0",
+ .pins = i2s1_pins0,
+ .num_pins = ARRAY_SIZE(i2s1_pins0),
+ .config = ARTPEC6_CONFIG_1,
+ },
+ {
+ .name = "i2srefclkgrp0",
+ .pins = i2srefclk_pins0,
+ .num_pins = ARRAY_SIZE(i2srefclk_pins0),
+ .config = ARTPEC6_CONFIG_3,
+ },
+ {
+ .name = "spi0grp0",
+ .pins = spi0_pins0,
+ .num_pins = ARRAY_SIZE(spi0_pins0),
+ .config = ARTPEC6_CONFIG_2,
+ },
+ {
+ .name = "spi1grp0",
+ .pins = spi1_pins0,
+ .num_pins = ARRAY_SIZE(spi1_pins0),
+ .config = ARTPEC6_CONFIG_2,
+ },
+ {
+ .name = "pciedebuggrp0",
+ .pins = pciedebug_pins0,
+ .num_pins = ARRAY_SIZE(pciedebug_pins0),
+ .config = ARTPEC6_CONFIG_3,
+ },
+ {
+ .name = "uart0grp0",
+ .pins = uart0_pins0,
+ .num_pins = ARRAY_SIZE(uart0_pins0),
+ .config = ARTPEC6_CONFIG_1,
+ },
+ {
+ .name = "uart0grp1",
+ .pins = uart0_pins1,
+ .num_pins = ARRAY_SIZE(uart0_pins1),
+ .config = ARTPEC6_CONFIG_1,
+ },
+ {
+ .name = "uart1grp0",
+ .pins = uart1_pins0,
+ .num_pins = ARRAY_SIZE(uart1_pins0),
+ .config = ARTPEC6_CONFIG_2,
+ },
+ {
+ .name = "uart2grp0",
+ .pins = uart2_pins0,
+ .num_pins = ARRAY_SIZE(uart2_pins0),
+ .config = ARTPEC6_CONFIG_1,
+ },
+ {
+ .name = "uart2grp1",
+ .pins = uart2_pins1,
+ .num_pins = ARRAY_SIZE(uart2_pins1),
+ .config = ARTPEC6_CONFIG_1,
+ },
+ {
+ .name = "uart3grp0",
+ .pins = uart3_pins0,
+ .num_pins = ARRAY_SIZE(uart3_pins0),
+ .config = ARTPEC6_CONFIG_0,
+ },
+ {
+ .name = "uart4grp0",
+ .pins = uart4_pins0,
+ .num_pins = ARRAY_SIZE(uart4_pins0),
+ .config = ARTPEC6_CONFIG_2,
+ },
+ {
+ .name = "uart5grp0",
+ .pins = uart5_pins0,
+ .num_pins = ARRAY_SIZE(uart5_pins0),
+ .config = ARTPEC6_CONFIG_2,
+ },
+ {
+ .name = "uart5nocts",
+ .pins = uart5_pins0,
+ .num_pins = ARRAY_SIZE(uart5_pins0) - 1,
+ .config = ARTPEC6_CONFIG_2,
+ },
+ {
+ .name = "nandgrp0",
+ .pins = nand_pins0,
+ .num_pins = ARRAY_SIZE(nand_pins0),
+ .config = ARTPEC6_CONFIG_0,
+ },
+ {
+ .name = "sdio0grp0",
+ .pins = sdio0_pins0,
+ .num_pins = ARRAY_SIZE(sdio0_pins0),
+ .config = ARTPEC6_CONFIG_0,
+ },
+ {
+ .name = "sdio1grp0",
+ .pins = sdio1_pins0,
+ .num_pins = ARRAY_SIZE(sdio1_pins0),
+ .config = ARTPEC6_CONFIG_0,
+ },
+ {
+ .name = "ethernetgrp0",
+ .pins = ethernet_pins0,
+ .num_pins = ARRAY_SIZE(ethernet_pins0),
+ .config = ARTPEC6_CONFIG_0,
+ },
+};
+
+struct pin_register {
+ unsigned int start;
+ unsigned int end;
+ unsigned int reg_base;
+};
+
+/*
+ * The register map has two holes where the pin number
+ * no longer fits directly with the register offset.
+ * This table allows us to map this easily.
+ */
+static const struct pin_register pin_register[] = {
+ { 0, 35, 0x0 }, /* 0x0 - 0x8c */
+ { 36, 52, 0x100 }, /* 0x100 - 0x140 */
+ { 53, 96, 0x180 }, /* 0x180 - 0x22c */
+};
+
+static unsigned int artpec6_pmx_reg_offset(unsigned int pin)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(pin_register); i++) {
+ if (pin <= pin_register[i].end) {
+ return (pin - pin_register[i].start) * 4 +
+ pin_register[i].reg_base;
+ }
+ }
+ /*
+ * Anything we return here is wrong, but we can only
+ * get here if pin is outside registered range.
+ */
+ pr_err("%s: Impossible pin %d\n", __func__, pin);
+ return 0;
+}
+
+static int artpec6_get_groups_count(struct pinctrl_dev *pctldev)
+{
+ return ARRAY_SIZE(artpec6_pin_groups);
+}
+
+static const char *artpec6_get_group_name(struct pinctrl_dev *pctldev,
+ unsigned int group)
+{
+ return artpec6_pin_groups[group].name;
+}
+
+static int artpec6_get_group_pins(struct pinctrl_dev *pctldev,
+ unsigned int group,
+ const unsigned int **pins,
+ unsigned int *num_pins)
+{
+ *pins = (unsigned int *)artpec6_pin_groups[group].pins;
+ *num_pins = artpec6_pin_groups[group].num_pins;
+ return 0;
+}
+
+static int artpec6_pconf_drive_mA_to_field(unsigned int mA)
+{
+ switch (mA) {
+ case ARTPEC6_DRIVE_4mA:
+ return ARTPEC6_DRIVE_4mA_SET;
+ case ARTPEC6_DRIVE_6mA:
+ return ARTPEC6_DRIVE_6mA_SET;
+ case ARTPEC6_DRIVE_8mA:
+ return ARTPEC6_DRIVE_8mA_SET;
+ case ARTPEC6_DRIVE_9mA:
+ return ARTPEC6_DRIVE_9mA_SET;
+ default:
+ return -EINVAL;
+ }
+}
+
+static unsigned int artpec6_pconf_drive_field_to_mA(int field)
+{
+ switch (field) {
+ case ARTPEC6_DRIVE_4mA_SET:
+ return ARTPEC6_DRIVE_4mA;
+ case ARTPEC6_DRIVE_6mA_SET:
+ return ARTPEC6_DRIVE_6mA;
+ case ARTPEC6_DRIVE_8mA_SET:
+ return ARTPEC6_DRIVE_8mA;
+ case ARTPEC6_DRIVE_9mA_SET:
+ return ARTPEC6_DRIVE_9mA;
+ default:
+ /* Shouldn't happen */
+ return 0;
+ }
+}
+
+static struct pinctrl_ops artpec6_pctrl_ops = {
+ .get_group_pins = artpec6_get_group_pins,
+ .get_groups_count = artpec6_get_groups_count,
+ .get_group_name = artpec6_get_group_name,
+ .dt_node_to_map = pinconf_generic_dt_node_to_map_all,
+ .dt_free_map = pinctrl_utils_free_map,
+};
+
+static const char * const gpiogrps[] = {
+ "cpuclkoutgrp0", "udlclkoutgrp0", "i2c1grp0", "i2c2grp0",
+ "i2c3grp0", "i2s0grp0", "i2s1grp0", "i2srefclkgrp0",
+ "spi0grp0", "spi1grp0", "pciedebuggrp0", "uart0grp0",
+ "uart0grp1", "uart1grp0", "uart2grp0", "uart2grp1",
+ "uart4grp0", "uart5grp0",
+};
+static const char * const cpuclkoutgrps[] = { "cpuclkoutgrp0" };
+static const char * const udlclkoutgrps[] = { "udlclkoutgrp0" };
+static const char * const i2c1grps[] = { "i2c1grp0" };
+static const char * const i2c2grps[] = { "i2c2grp0" };
+static const char * const i2c3grps[] = { "i2c3grp0" };
+static const char * const i2s0grps[] = { "i2s0grp0" };
+static const char * const i2s1grps[] = { "i2s1grp0" };
+static const char * const i2srefclkgrps[] = { "i2srefclkgrp0" };
+static const char * const spi0grps[] = { "spi0grp0" };
+static const char * const spi1grps[] = { "spi1grp0" };
+static const char * const pciedebuggrps[] = { "pciedebuggrp0" };
+static const char * const uart0grps[] = { "uart0grp0", "uart0grp1" };
+static const char * const uart1grps[] = { "uart1grp0" };
+static const char * const uart2grps[] = { "uart2grp0", "uart2grp1" };
+static const char * const uart3grps[] = { "uart3grp0" };
+static const char * const uart4grps[] = { "uart4grp0" };
+static const char * const uart5grps[] = { "uart5grp0", "uart5nocts" };
+static const char * const nandgrps[] = { "nandgrp0" };
+static const char * const sdio0grps[] = { "sdio0grp0" };
+static const char * const sdio1grps[] = { "sdio1grp0" };
+static const char * const ethernetgrps[] = { "ethernetgrp0" };
+
+static const struct artpec6_pmx_func artpec6_pmx_functions[] = {
+ {
+ .name = "gpio",
+ .groups = gpiogrps,
+ .num_groups = ARRAY_SIZE(gpiogrps),
+ },
+ {
+ .name = "cpuclkout",
+ .groups = cpuclkoutgrps,
+ .num_groups = ARRAY_SIZE(cpuclkoutgrps),
+ },
+ {
+ .name = "udlclkout",
+ .groups = udlclkoutgrps,
+ .num_groups = ARRAY_SIZE(udlclkoutgrps),
+ },
+ {
+ .name = "i2c1",
+ .groups = i2c1grps,
+ .num_groups = ARRAY_SIZE(i2c1grps),
+ },
+ {
+ .name = "i2c2",
+ .groups = i2c2grps,
+ .num_groups = ARRAY_SIZE(i2c2grps),
+ },
+ {
+ .name = "i2c3",
+ .groups = i2c3grps,
+ .num_groups = ARRAY_SIZE(i2c3grps),
+ },
+ {
+ .name = "i2s0",
+ .groups = i2s0grps,
+ .num_groups = ARRAY_SIZE(i2s0grps),
+ },
+ {
+ .name = "i2s1",
+ .groups = i2s1grps,
+ .num_groups = ARRAY_SIZE(i2s1grps),
+ },
+ {
+ .name = "i2srefclk",
+ .groups = i2srefclkgrps,
+ .num_groups = ARRAY_SIZE(i2srefclkgrps),
+ },
+ {
+ .name = "spi0",
+ .groups = spi0grps,
+ .num_groups = ARRAY_SIZE(spi0grps),
+ },
+ {
+ .name = "spi1",
+ .groups = spi1grps,
+ .num_groups = ARRAY_SIZE(spi1grps),
+ },
+ {
+ .name = "pciedebug",
+ .groups = pciedebuggrps,
+ .num_groups = ARRAY_SIZE(pciedebuggrps),
+ },
+ {
+ .name = "uart0",
+ .groups = uart0grps,
+ .num_groups = ARRAY_SIZE(uart0grps),
+ },
+ {
+ .name = "uart1",
+ .groups = uart1grps,
+ .num_groups = ARRAY_SIZE(uart1grps),
+ },
+ {
+ .name = "uart2",
+ .groups = uart2grps,
+ .num_groups = ARRAY_SIZE(uart2grps),
+ },
+ {
+ .name = "uart3",
+ .groups = uart3grps,
+ .num_groups = ARRAY_SIZE(uart3grps),
+ },
+ {
+ .name = "uart4",
+ .groups = uart4grps,
+ .num_groups = ARRAY_SIZE(uart4grps),
+ },
+ {
+ .name = "uart5",
+ .groups = uart5grps,
+ .num_groups = ARRAY_SIZE(uart5grps),
+ },
+ {
+ .name = "nand",
+ .groups = nandgrps,
+ .num_groups = ARRAY_SIZE(nandgrps),
+ },
+ {
+ .name = "sdio0",
+ .groups = sdio0grps,
+ .num_groups = ARRAY_SIZE(sdio0grps),
+ },
+ {
+ .name = "sdio1",
+ .groups = sdio1grps,
+ .num_groups = ARRAY_SIZE(sdio1grps),
+ },
+ {
+ .name = "ethernet",
+ .groups = ethernetgrps,
+ .num_groups = ARRAY_SIZE(ethernetgrps),
+ },
+};
+
+static int artpec6_pmx_get_functions_count(struct pinctrl_dev *pctldev)
+{
+ return ARRAY_SIZE(artpec6_pmx_functions);
+}
+
+static const char *artpec6_pmx_get_fname(struct pinctrl_dev *pctldev,
+ unsigned int function)
+{
+ return artpec6_pmx_functions[function].name;
+}
+
+static int artpec6_pmx_get_fgroups(struct pinctrl_dev *pctldev,
+ unsigned int function,
+ const char * const **groups,
+ unsigned int * const num_groups)
+{
+ *groups = artpec6_pmx_functions[function].groups;
+ *num_groups = artpec6_pmx_functions[function].num_groups;
+ return 0;
+}
+
+static void artpec6_pmx_select_func(struct pinctrl_dev *pctldev,
+ unsigned int function, unsigned int group,
+ bool enable)
+{
+ unsigned int regval, val;
+ unsigned int reg;
+ int i;
+ struct artpec6_pmx *pmx = pinctrl_dev_get_drvdata(pctldev);
+
+ for (i = 0; i < artpec6_pin_groups[group].num_pins; i++) {
+ /*
+ * Registers for pins above a ARTPEC6_MAX_MUXABLE
+ * do not have a SEL field and are always selected.
+ */
+ if (artpec6_pin_groups[group].pins[i] > ARTPEC6_MAX_MUXABLE)
+ continue;
+
+ if (!strcmp(artpec6_pmx_get_fname(pctldev, function), "gpio")) {
+ /* GPIO is always config 0 */
+ val = ARTPEC6_CONFIG_0 << ARTPEC6_PINMUX_SEL_SHIFT;
+ } else {
+ if (enable)
+ val = artpec6_pin_groups[group].config
+ << ARTPEC6_PINMUX_SEL_SHIFT;
+ else
+ val = ARTPEC6_CONFIG_0
+ << ARTPEC6_PINMUX_SEL_SHIFT;
+ }
+
+ reg = artpec6_pmx_reg_offset(artpec6_pin_groups[group].pins[i]);
+
+ regval = readl(pmx->base + reg);
+ regval &= ~ARTPEC6_PINMUX_SEL_MASK;
+ regval |= val;
+ writel(regval, pmx->base + reg);
+ }
+}
+
+int artpec6_pmx_enable(struct pinctrl_dev *pctldev, unsigned int function,
+ unsigned int group)
+{
+ struct artpec6_pmx *pmx = pinctrl_dev_get_drvdata(pctldev);
+
+ dev_dbg(pmx->dev, "enabling %s function for pin group %s\n",
+ artpec6_pmx_get_fname(pctldev, function),
+ artpec6_get_group_name(pctldev, group));
+
+ artpec6_pmx_select_func(pctldev, function, group, true);
+
+ return 0;
+}
+
+void artpec6_pmx_disable(struct pinctrl_dev *pctldev, unsigned int function,
+ unsigned int group)
+{
+ struct artpec6_pmx *pmx = pinctrl_dev_get_drvdata(pctldev);
+
+ dev_dbg(pmx->dev, "disabling %s function for pin group %s\n",
+ artpec6_pmx_get_fname(pctldev, function),
+ artpec6_get_group_name(pctldev, group));
+
+ artpec6_pmx_select_func(pctldev, function, group, false);
+}
+
+static int artpec6_pmx_request_gpio(struct pinctrl_dev *pctldev,
+ struct pinctrl_gpio_range *range,
+ unsigned int pin)
+{
+ struct artpec6_pmx *pmx = pinctrl_dev_get_drvdata(pctldev);
+ unsigned int reg = artpec6_pmx_reg_offset(pin);
+ u32 val;
+
+ if (pin >= 32)
+ return -EINVAL;
+
+ val = readl_relaxed(pmx->base + reg);
+ val &= ~ARTPEC6_PINMUX_SEL_MASK;
+ val |= ARTPEC6_CONFIG_0 << ARTPEC6_PINMUX_SEL_SHIFT;
+ writel_relaxed(val, pmx->base + reg);
+
+ return 0;
+}
+
+static const struct pinmux_ops artpec6_pmx_ops = {
+ .get_functions_count = artpec6_pmx_get_functions_count,
+ .get_function_name = artpec6_pmx_get_fname,
+ .get_function_groups = artpec6_pmx_get_fgroups,
+ .set_mux = artpec6_pmx_enable,
+ .gpio_request_enable = artpec6_pmx_request_gpio,
+};
+
+static int artpec6_pconf_get(struct pinctrl_dev *pctldev, unsigned int pin,
+ unsigned long *config)
+{
+ struct artpec6_pmx *pmx = pinctrl_dev_get_drvdata(pctldev);
+ enum pin_config_param param = pinconf_to_config_param(*config);
+ unsigned int regval;
+
+ /* Check for valid pin */
+ if (pin >= pmx->num_pins) {
+ dev_dbg(pmx->dev, "pinconf is not supported for pin %s\n",
+ pmx->pins[pin].name);
+ return -ENOTSUPP;
+ }
+
+ dev_dbg(pmx->dev, "getting configuration for pin %s\n",
+ pmx->pins[pin].name);
+
+ /* Read pin register values */
+ regval = readl(pmx->base + artpec6_pmx_reg_offset(pin));
+
+ /* If valid, get configuration for parameter */
+ switch (param) {
+ case PIN_CONFIG_BIAS_DISABLE:
+ if (!(regval & ARTPEC6_PINMUX_UDC1_MASK))
+ return -EINVAL;
+ break;
+
+ case PIN_CONFIG_BIAS_PULL_UP:
+ case PIN_CONFIG_BIAS_PULL_DOWN:
+ if (regval & ARTPEC6_PINMUX_UDC1_MASK)
+ return -EINVAL;
+
+ regval = regval & ARTPEC6_PINMUX_UDC0_MASK;
+ if ((param == PIN_CONFIG_BIAS_PULL_UP && !regval) ||
+ (param == PIN_CONFIG_BIAS_PULL_DOWN && regval))
+ return -EINVAL;
+ break;
+ case PIN_CONFIG_DRIVE_STRENGTH:
+ regval = (regval & ARTPEC6_PINMUX_DRV_MASK)
+ >> ARTPEC6_PINMUX_DRV_SHIFT;
+ regval = artpec6_pconf_drive_field_to_mA(regval);
+ *config = pinconf_to_config_packed(param, regval);
+ break;
+ default:
+ return -ENOTSUPP;
+ }
+
+ return 0;
+}
+
+/*
+ * Valid combinations of param and arg:
+ *
+ * param arg
+ * PIN_CONFIG_BIAS_DISABLE: x (disable bias)
+ * PIN_CONFIG_BIAS_PULL_UP: 1 (pull up bias + enable)
+ * PIN_CONFIG_BIAS_PULL_DOWN: 1 (pull down bias + enable)
+ * PIN_CONFIG_DRIVE_STRENGTH: x (4mA, 6mA, 8mA, 9mA)
+ *
+ * All other args are invalid. All other params are not supported.
+ */
+static int artpec6_pconf_set(struct pinctrl_dev *pctldev, unsigned int pin,
+ unsigned long *configs, unsigned int num_configs)
+{
+ struct artpec6_pmx *pmx = pinctrl_dev_get_drvdata(pctldev);
+ enum pin_config_param param;
+ unsigned int arg;
+ unsigned int regval;
+ unsigned int *reg;
+ int i;
+
+ /* Check for valid pin */
+ if (pin >= pmx->num_pins) {
+ dev_dbg(pmx->dev, "pinconf is not supported for pin %s\n",
+ pmx->pins[pin].name);
+ return -ENOTSUPP;
+ }
+
+ dev_dbg(pmx->dev, "setting configuration for pin %s\n",
+ pmx->pins[pin].name);
+
+ reg = pmx->base + artpec6_pmx_reg_offset(pin);
+
+ /* For each config */
+ for (i = 0; i < num_configs; i++) {
+ int drive;
+
+ param = pinconf_to_config_param(configs[i]);
+ arg = pinconf_to_config_argument(configs[i]);
+
+ switch (param) {
+ case PIN_CONFIG_BIAS_DISABLE:
+ regval = readl(reg);
+ regval |= (1 << ARTPEC6_PINMUX_UDC1_SHIFT);
+ writel(regval, reg);
+ break;
+
+ case PIN_CONFIG_BIAS_PULL_UP:
+ if (arg != 1) {
+ dev_dbg(pctldev->dev, "%s: arg %u out of range\n",
+ __func__, arg);
+ return -EINVAL;
+ }
+
+ regval = readl(reg);
+ regval |= (arg << ARTPEC6_PINMUX_UDC0_SHIFT);
+ regval &= ~ARTPEC6_PINMUX_UDC1_MASK; /* Enable */
+ writel(regval, reg);
+ break;
+
+ case PIN_CONFIG_BIAS_PULL_DOWN:
+ if (arg != 1) {
+ dev_dbg(pctldev->dev, "%s: arg %u out of range\n",
+ __func__, arg);
+ return -EINVAL;
+ }
+
+ regval = readl(reg);
+ regval &= ~(arg << ARTPEC6_PINMUX_UDC0_SHIFT);
+ regval &= ~ARTPEC6_PINMUX_UDC1_MASK; /* Enable */
+ writel(regval, reg);
+ break;
+
+ case PIN_CONFIG_DRIVE_STRENGTH:
+ drive = artpec6_pconf_drive_mA_to_field(arg);
+ if (drive < 0) {
+ dev_dbg(pctldev->dev, "%s: arg %u out of range\n",
+ __func__, arg);
+ return -EINVAL;
+ }
+
+ regval = readl(reg);
+ regval &= ~ARTPEC6_PINMUX_DRV_MASK;
+ regval |= (drive << ARTPEC6_PINMUX_DRV_SHIFT);
+ writel(regval, reg);
+ break;
+
+ default:
+ dev_dbg(pmx->dev, "parameter not supported\n");
+ return -ENOTSUPP;
+ }
+ }
+
+ return 0;
+}
+
+static int artpec6_pconf_group_set(struct pinctrl_dev *pctldev,
+ unsigned int group, unsigned long *configs,
+ unsigned int num_configs)
+{
+ unsigned int num_pins, current_pin;
+ int ret;
+
+ dev_dbg(pctldev->dev, "setting group %s configuration\n",
+ artpec6_get_group_name(pctldev, group));
+
+ num_pins = artpec6_pin_groups[group].num_pins;
+
+ for (current_pin = 0; current_pin < num_pins; current_pin++) {
+ ret = artpec6_pconf_set(pctldev,
+ artpec6_pin_groups[group].pins[current_pin],
+ configs, num_configs);
+
+ if (ret < 0)
+ return ret;
+ }
+
+ return 0;
+}
+
+static const struct pinconf_ops artpec6_pconf_ops = {
+ .is_generic = true,
+ .pin_config_get = artpec6_pconf_get,
+ .pin_config_set = artpec6_pconf_set,
+ .pin_config_group_set = artpec6_pconf_group_set,
+};
+
+static struct pinctrl_desc artpec6_desc = {
+ .name = "artpec6-pinctrl",
+ .owner = THIS_MODULE,
+ .pins = artpec6_pins,
+ .npins = ARRAY_SIZE(artpec6_pins),
+ .pctlops = &artpec6_pctrl_ops,
+ .pmxops = &artpec6_pmx_ops,
+ .confops = &artpec6_pconf_ops,
+};
+
+/* The reset values say 4mA, but we want 8mA as default. */
+static void artpec6_pmx_reset(struct artpec6_pmx *pmx)
+{
+ void __iomem *base = pmx->base;
+ int i;
+
+ for (i = 0; i < ARTPEC6_LAST_PIN; i++) {
+ u32 val;
+
+ val = readl_relaxed(base + artpec6_pmx_reg_offset(i));
+ val &= ~ARTPEC6_PINMUX_DRV_MASK;
+ val |= ARTPEC6_DRIVE_8mA_SET << ARTPEC6_PINMUX_DRV_SHIFT;
+ writel_relaxed(val, base + artpec6_pmx_reg_offset(i));
+ }
+}
+
+static int artpec6_pmx_probe(struct platform_device *pdev)
+{
+ struct artpec6_pmx *pmx;
+ struct resource *res;
+
+ pmx = devm_kzalloc(&pdev->dev, sizeof(*pmx), GFP_KERNEL);
+ if (!pmx)
+ return -ENOMEM;
+
+ pmx->dev = &pdev->dev;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ pmx->base = devm_ioremap_resource(&pdev->dev, res);
+
+ if (IS_ERR(pmx->base))
+ return PTR_ERR(pmx->base);
+
+ artpec6_pmx_reset(pmx);
+
+ pmx->pins = artpec6_pins;
+ pmx->num_pins = ARRAY_SIZE(artpec6_pins);
+ pmx->functions = artpec6_pmx_functions;
+ pmx->num_functions = ARRAY_SIZE(artpec6_pmx_functions);
+ pmx->pin_groups = artpec6_pin_groups;
+ pmx->num_pin_groups = ARRAY_SIZE(artpec6_pin_groups);
+ pmx->pctl = pinctrl_register(&artpec6_desc, &pdev->dev, pmx);
+
+ if (IS_ERR(pmx->pctl)) {
+ dev_err(&pdev->dev, "could not register pinctrl driver\n");
+ return PTR_ERR(pmx->pctl);
+ }
+
+ platform_set_drvdata(pdev, pmx);
+
+ dev_info(&pdev->dev, "initialised Axis ARTPEC-6 pinctrl driver\n");
+
+ return 0;
+}
+
+static int artpec6_pmx_remove(struct platform_device *pdev)
+{
+ struct artpec6_pmx *pmx = platform_get_drvdata(pdev);
+
+ pinctrl_unregister(pmx->pctl);
+
+ return 0;
+}
+
+static const struct of_device_id artpec6_pinctrl_match[] = {
+ { .compatible = "axis,artpec6-pinctrl" },
+ {},
+};
+
+static struct platform_driver artpec6_pmx_driver = {
+ .driver = {
+ .name = "artpec6-pinctrl",
+ .of_match_table = artpec6_pinctrl_match,
+ },
+ .probe = artpec6_pmx_probe,
+ .remove = artpec6_pmx_remove,
+};
+
+static int __init artpec6_pmx_init(void)
+{
+ return platform_driver_register(&artpec6_pmx_driver);
+}
+arch_initcall(artpec6_pmx_init);
diff --git a/drivers/pinctrl/pinctrl-at91-pio4.c b/drivers/pinctrl/pinctrl-at91-pio4.c
index 28bbc1b..dc85915 100644
--- a/drivers/pinctrl/pinctrl-at91-pio4.c
+++ b/drivers/pinctrl/pinctrl-at91-pio4.c
@@ -126,7 +126,11 @@
struct irq_domain *irq_domain;
int *irqs;
unsigned *pm_wakeup_sources;
- unsigned *pm_suspend_backup;
+ struct {
+ u32 imr;
+ u32 odsr;
+ u32 cfgr[ATMEL_PIO_NPINS_PER_BANK];
+ } *pm_suspend_backup;
struct device *dev;
struct device_node *node;
};
@@ -830,17 +834,26 @@
{
struct platform_device *pdev = to_platform_device(dev);
struct atmel_pioctrl *atmel_pioctrl = platform_get_drvdata(pdev);
- int i;
+ int i, j;
/*
* For each bank, save IMR to restore it later and disable all GPIO
* interrupts excepting the ones marked as wakeup sources.
*/
for (i = 0; i < atmel_pioctrl->nbanks; i++) {
- atmel_pioctrl->pm_suspend_backup[i] =
+ atmel_pioctrl->pm_suspend_backup[i].imr =
atmel_gpio_read(atmel_pioctrl, i, ATMEL_PIO_IMR);
atmel_gpio_write(atmel_pioctrl, i, ATMEL_PIO_IDR,
~atmel_pioctrl->pm_wakeup_sources[i]);
+ atmel_pioctrl->pm_suspend_backup[i].odsr =
+ atmel_gpio_read(atmel_pioctrl, i, ATMEL_PIO_ODSR);
+ for (j = 0; j < ATMEL_PIO_NPINS_PER_BANK; j++) {
+ atmel_gpio_write(atmel_pioctrl, i,
+ ATMEL_PIO_MSKR, BIT(j));
+ atmel_pioctrl->pm_suspend_backup[i].cfgr[j] =
+ atmel_gpio_read(atmel_pioctrl, i,
+ ATMEL_PIO_CFGR);
+ }
}
return 0;
@@ -850,11 +863,20 @@
{
struct platform_device *pdev = to_platform_device(dev);
struct atmel_pioctrl *atmel_pioctrl = platform_get_drvdata(pdev);
- int i;
+ int i, j;
- for (i = 0; i < atmel_pioctrl->nbanks; i++)
+ for (i = 0; i < atmel_pioctrl->nbanks; i++) {
atmel_gpio_write(atmel_pioctrl, i, ATMEL_PIO_IER,
- atmel_pioctrl->pm_suspend_backup[i]);
+ atmel_pioctrl->pm_suspend_backup[i].imr);
+ atmel_gpio_write(atmel_pioctrl, i, ATMEL_PIO_SODR,
+ atmel_pioctrl->pm_suspend_backup[i].odsr);
+ for (j = 0; j < ATMEL_PIO_NPINS_PER_BANK; j++) {
+ atmel_gpio_write(atmel_pioctrl, i,
+ ATMEL_PIO_MSKR, BIT(j));
+ atmel_gpio_write(atmel_pioctrl, i, ATMEL_PIO_CFGR,
+ atmel_pioctrl->pm_suspend_backup[i].cfgr[j]);
+ }
+ }
return 0;
}
diff --git a/drivers/pinctrl/pinctrl-rockchip.c b/drivers/pinctrl/pinctrl-rockchip.c
index 7813599..f141aa0 100644
--- a/drivers/pinctrl/pinctrl-rockchip.c
+++ b/drivers/pinctrl/pinctrl-rockchip.c
@@ -59,7 +59,7 @@
#define GPIO_LS_SYNC 0x60
enum rockchip_pinctrl_type {
- RK1108,
+ RV1108,
RK2928,
RK3066B,
RK3188,
@@ -75,6 +75,8 @@
#define IOMUX_WIDTH_4BIT BIT(1)
#define IOMUX_SOURCE_PMU BIT(2)
#define IOMUX_UNROUTED BIT(3)
+#define IOMUX_WIDTH_3BIT BIT(4)
+#define IOMUX_RECALCED BIT(5)
/**
* @type: iomux variant using IOMUX_* constants
@@ -141,6 +143,9 @@
* @gpio_chip: gpiolib chip
* @grange: gpio range
* @slock: spinlock for the gpio bank
+ * @irq_lock: bus lock for irq chip
+ * @new_irqs: newly configured irqs which must be muxed as GPIOs in
+ * irq_bus_sync_unlock()
*/
struct rockchip_pin_bank {
void __iomem *reg_base;
@@ -161,8 +166,10 @@
struct irq_domain *domain;
struct gpio_chip gpio_chip;
struct pinctrl_gpio_range grange;
- spinlock_t slock;
+ raw_spinlock_t slock;
u32 toggle_edge_mode;
+ struct mutex irq_lock;
+ u32 new_irqs;
};
#define PIN_BANK(id, pins, label) \
@@ -304,6 +311,11 @@
void (*drv_calc_reg)(struct rockchip_pin_bank *bank,
int pin_num, struct regmap **regmap,
int *reg, u8 *bit);
+ void (*iomux_recalc)(u8 bank_num, int pin, int *reg,
+ u8 *bit, int *mask);
+ int (*schmitt_calc_reg)(struct rockchip_pin_bank *bank,
+ int pin_num, struct regmap **regmap,
+ int *reg, u8 *bit);
};
struct rockchip_pin_config {
@@ -355,6 +367,22 @@
unsigned int nfunctions;
};
+/**
+ * struct rockchip_mux_recalced_data: represent a pin iomux data.
+ * @num: bank number.
+ * @pin: pin number.
+ * @bit: index at register.
+ * @reg: register offset.
+ * @mask: mask bit
+ */
+struct rockchip_mux_recalced_data {
+ u8 num;
+ u8 pin;
+ u8 reg;
+ u8 bit;
+ u8 mask;
+};
+
static struct regmap_config rockchip_regmap_config = {
.reg_bits = 32,
.val_bits = 32,
@@ -514,13 +542,57 @@
* Hardware access
*/
+static const struct rockchip_mux_recalced_data rk3328_mux_recalced_data[] = {
+ {
+ .num = 2,
+ .pin = 12,
+ .reg = 0x24,
+ .bit = 8,
+ .mask = 0x3
+ }, {
+ .num = 2,
+ .pin = 15,
+ .reg = 0x28,
+ .bit = 0,
+ .mask = 0x7
+ }, {
+ .num = 2,
+ .pin = 23,
+ .reg = 0x30,
+ .bit = 14,
+ .mask = 0x3
+ },
+};
+
+static void rk3328_recalc_mux(u8 bank_num, int pin, int *reg,
+ u8 *bit, int *mask)
+{
+ const struct rockchip_mux_recalced_data *data = NULL;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(rk3328_mux_recalced_data); i++)
+ if (rk3328_mux_recalced_data[i].num == bank_num &&
+ rk3328_mux_recalced_data[i].pin == pin) {
+ data = &rk3328_mux_recalced_data[i];
+ break;
+ }
+
+ if (!data)
+ return;
+
+ *reg = data->reg;
+ *mask = data->mask;
+ *bit = data->bit;
+}
+
static int rockchip_get_mux(struct rockchip_pin_bank *bank, int pin)
{
struct rockchip_pinctrl *info = bank->drvdata;
+ struct rockchip_pin_ctrl *ctrl = info->ctrl;
int iomux_num = (pin / 8);
struct regmap *regmap;
unsigned int val;
- int reg, ret, mask;
+ int reg, ret, mask, mux_type;
u8 bit;
if (iomux_num > 3)
@@ -538,16 +610,26 @@
? info->regmap_pmu : info->regmap_base;
/* get basic quadrupel of mux registers and the correct reg inside */
- mask = (bank->iomux[iomux_num].type & IOMUX_WIDTH_4BIT) ? 0xf : 0x3;
+ mux_type = bank->iomux[iomux_num].type;
reg = bank->iomux[iomux_num].offset;
- if (bank->iomux[iomux_num].type & IOMUX_WIDTH_4BIT) {
+ if (mux_type & IOMUX_WIDTH_4BIT) {
if ((pin % 8) >= 4)
reg += 0x4;
bit = (pin % 4) * 4;
+ mask = 0xf;
+ } else if (mux_type & IOMUX_WIDTH_3BIT) {
+ if ((pin % 8) >= 5)
+ reg += 0x4;
+ bit = (pin % 8 % 5) * 3;
+ mask = 0x7;
} else {
bit = (pin % 8) * 2;
+ mask = 0x3;
}
+ if (ctrl->iomux_recalc && (mux_type & IOMUX_RECALCED))
+ ctrl->iomux_recalc(bank->bank_num, pin, ®, &bit, &mask);
+
ret = regmap_read(regmap, reg, &val);
if (ret)
return ret;
@@ -555,6 +637,31 @@
return ((val >> bit) & mask);
}
+static int rockchip_verify_mux(struct rockchip_pin_bank *bank,
+ int pin, int mux)
+{
+ struct rockchip_pinctrl *info = bank->drvdata;
+ int iomux_num = (pin / 8);
+
+ if (iomux_num > 3)
+ return -EINVAL;
+
+ if (bank->iomux[iomux_num].type & IOMUX_UNROUTED) {
+ dev_err(info->dev, "pin %d is unrouted\n", pin);
+ return -EINVAL;
+ }
+
+ if (bank->iomux[iomux_num].type & IOMUX_GPIO_ONLY) {
+ if (mux != RK_FUNC_GPIO) {
+ dev_err(info->dev,
+ "pin %d only supports a gpio mux\n", pin);
+ return -ENOTSUPP;
+ }
+ }
+
+ return 0;
+}
+
/*
* Set a new mux function for a pin.
*
@@ -571,30 +678,19 @@
static int rockchip_set_mux(struct rockchip_pin_bank *bank, int pin, int mux)
{
struct rockchip_pinctrl *info = bank->drvdata;
+ struct rockchip_pin_ctrl *ctrl = info->ctrl;
int iomux_num = (pin / 8);
struct regmap *regmap;
- int reg, ret, mask;
- unsigned long flags;
+ int reg, ret, mask, mux_type;
u8 bit;
u32 data, rmask;
- if (iomux_num > 3)
- return -EINVAL;
+ ret = rockchip_verify_mux(bank, pin, mux);
+ if (ret < 0)
+ return ret;
- if (bank->iomux[iomux_num].type & IOMUX_UNROUTED) {
- dev_err(info->dev, "pin %d is unrouted\n", pin);
- return -EINVAL;
- }
-
- if (bank->iomux[iomux_num].type & IOMUX_GPIO_ONLY) {
- if (mux != RK_FUNC_GPIO) {
- dev_err(info->dev,
- "pin %d only supports a gpio mux\n", pin);
- return -ENOTSUPP;
- } else {
- return 0;
- }
- }
+ if (bank->iomux[iomux_num].type & IOMUX_GPIO_ONLY)
+ return 0;
dev_dbg(info->dev, "setting mux of GPIO%d-%d to %d\n",
bank->bank_num, pin, mux);
@@ -603,35 +699,41 @@
? info->regmap_pmu : info->regmap_base;
/* get basic quadrupel of mux registers and the correct reg inside */
- mask = (bank->iomux[iomux_num].type & IOMUX_WIDTH_4BIT) ? 0xf : 0x3;
+ mux_type = bank->iomux[iomux_num].type;
reg = bank->iomux[iomux_num].offset;
- if (bank->iomux[iomux_num].type & IOMUX_WIDTH_4BIT) {
+ if (mux_type & IOMUX_WIDTH_4BIT) {
if ((pin % 8) >= 4)
reg += 0x4;
bit = (pin % 4) * 4;
+ mask = 0xf;
+ } else if (mux_type & IOMUX_WIDTH_3BIT) {
+ if ((pin % 8) >= 5)
+ reg += 0x4;
+ bit = (pin % 8 % 5) * 3;
+ mask = 0x7;
} else {
bit = (pin % 8) * 2;
+ mask = 0x3;
}
- spin_lock_irqsave(&bank->slock, flags);
+ if (ctrl->iomux_recalc && (mux_type & IOMUX_RECALCED))
+ ctrl->iomux_recalc(bank->bank_num, pin, ®, &bit, &mask);
data = (mask << (bit + 16));
rmask = data | (data >> 16);
data |= (mux & mask) << bit;
ret = regmap_update_bits(regmap, reg, rmask, data);
- spin_unlock_irqrestore(&bank->slock, flags);
-
return ret;
}
-#define RK1108_PULL_PMU_OFFSET 0x10
-#define RK1108_PULL_OFFSET 0x110
-#define RK1108_PULL_PINS_PER_REG 8
-#define RK1108_PULL_BITS_PER_PIN 2
-#define RK1108_PULL_BANK_STRIDE 16
+#define RV1108_PULL_PMU_OFFSET 0x10
+#define RV1108_PULL_OFFSET 0x110
+#define RV1108_PULL_PINS_PER_REG 8
+#define RV1108_PULL_BITS_PER_PIN 2
+#define RV1108_PULL_BANK_STRIDE 16
-static void rk1108_calc_pull_reg_and_bit(struct rockchip_pin_bank *bank,
+static void rv1108_calc_pull_reg_and_bit(struct rockchip_pin_bank *bank,
int pin_num, struct regmap **regmap,
int *reg, u8 *bit)
{
@@ -640,27 +742,27 @@
/* The first 24 pins of the first bank are located in PMU */
if (bank->bank_num == 0) {
*regmap = info->regmap_pmu;
- *reg = RK1108_PULL_PMU_OFFSET;
+ *reg = RV1108_PULL_PMU_OFFSET;
} else {
- *reg = RK1108_PULL_OFFSET;
+ *reg = RV1108_PULL_OFFSET;
*regmap = info->regmap_base;
/* correct the offset, as we're starting with the 2nd bank */
*reg -= 0x10;
- *reg += bank->bank_num * RK1108_PULL_BANK_STRIDE;
+ *reg += bank->bank_num * RV1108_PULL_BANK_STRIDE;
}
- *reg += ((pin_num / RK1108_PULL_PINS_PER_REG) * 4);
- *bit = (pin_num % RK1108_PULL_PINS_PER_REG);
- *bit *= RK1108_PULL_BITS_PER_PIN;
+ *reg += ((pin_num / RV1108_PULL_PINS_PER_REG) * 4);
+ *bit = (pin_num % RV1108_PULL_PINS_PER_REG);
+ *bit *= RV1108_PULL_BITS_PER_PIN;
}
-#define RK1108_DRV_PMU_OFFSET 0x20
-#define RK1108_DRV_GRF_OFFSET 0x210
-#define RK1108_DRV_BITS_PER_PIN 2
-#define RK1108_DRV_PINS_PER_REG 8
-#define RK1108_DRV_BANK_STRIDE 16
+#define RV1108_DRV_PMU_OFFSET 0x20
+#define RV1108_DRV_GRF_OFFSET 0x210
+#define RV1108_DRV_BITS_PER_PIN 2
+#define RV1108_DRV_PINS_PER_REG 8
+#define RV1108_DRV_BANK_STRIDE 16
-static void rk1108_calc_drv_reg_and_bit(struct rockchip_pin_bank *bank,
+static void rv1108_calc_drv_reg_and_bit(struct rockchip_pin_bank *bank,
int pin_num, struct regmap **regmap,
int *reg, u8 *bit)
{
@@ -669,19 +771,19 @@
/* The first 24 pins of the first bank are located in PMU */
if (bank->bank_num == 0) {
*regmap = info->regmap_pmu;
- *reg = RK1108_DRV_PMU_OFFSET;
+ *reg = RV1108_DRV_PMU_OFFSET;
} else {
*regmap = info->regmap_base;
- *reg = RK1108_DRV_GRF_OFFSET;
+ *reg = RV1108_DRV_GRF_OFFSET;
/* correct the offset, as we're starting with the 2nd bank */
*reg -= 0x10;
- *reg += bank->bank_num * RK1108_DRV_BANK_STRIDE;
+ *reg += bank->bank_num * RV1108_DRV_BANK_STRIDE;
}
- *reg += ((pin_num / RK1108_DRV_PINS_PER_REG) * 4);
- *bit = pin_num % RK1108_DRV_PINS_PER_REG;
- *bit *= RK1108_DRV_BITS_PER_PIN;
+ *reg += ((pin_num / RV1108_DRV_PINS_PER_REG) * 4);
+ *bit = pin_num % RV1108_DRV_PINS_PER_REG;
+ *bit *= RV1108_DRV_BITS_PER_PIN;
}
#define RK2928_PULL_OFFSET 0x118
@@ -1047,7 +1149,6 @@
struct rockchip_pinctrl *info = bank->drvdata;
struct rockchip_pin_ctrl *ctrl = info->ctrl;
struct regmap *regmap;
- unsigned long flags;
int reg, ret, i;
u32 data, rmask, rmask_bits, temp;
u8 bit;
@@ -1075,8 +1176,6 @@
return ret;
}
- spin_lock_irqsave(&bank->slock, flags);
-
switch (drv_type) {
case DRV_TYPE_IO_1V8_3V0_AUTO:
case DRV_TYPE_IO_3V3_ONLY:
@@ -1097,17 +1196,14 @@
rmask = BIT(15) | BIT(31);
data |= BIT(31);
ret = regmap_update_bits(regmap, reg, rmask, data);
- if (ret) {
- spin_unlock_irqrestore(&bank->slock, flags);
+ if (ret)
return ret;
- }
rmask = 0x3 | (0x3 << 16);
temp |= (0x3 << 16);
reg += 0x4;
ret = regmap_update_bits(regmap, reg, rmask, temp);
- spin_unlock_irqrestore(&bank->slock, flags);
return ret;
case 18 ... 21:
/* setting fully enclosed in the second register */
@@ -1115,7 +1211,6 @@
bit -= 16;
break;
default:
- spin_unlock_irqrestore(&bank->slock, flags);
dev_err(info->dev, "unsupported bit: %d for pinctrl drive type: %d\n",
bit, drv_type);
return -EINVAL;
@@ -1127,7 +1222,6 @@
rmask_bits = RK3288_DRV_BITS_PER_PIN;
break;
default:
- spin_unlock_irqrestore(&bank->slock, flags);
dev_err(info->dev, "unsupported pinctrl drive type: %d\n",
drv_type);
return -EINVAL;
@@ -1139,7 +1233,6 @@
data |= (ret << bit);
ret = regmap_update_bits(regmap, reg, rmask, data);
- spin_unlock_irqrestore(&bank->slock, flags);
return ret;
}
@@ -1183,7 +1276,7 @@
return !(data & BIT(bit))
? PIN_CONFIG_BIAS_PULL_PIN_DEFAULT
: PIN_CONFIG_BIAS_DISABLE;
- case RK1108:
+ case RV1108:
case RK3188:
case RK3288:
case RK3368:
@@ -1206,7 +1299,6 @@
struct rockchip_pin_ctrl *ctrl = info->ctrl;
struct regmap *regmap;
int reg, ret, i, pull_type;
- unsigned long flags;
u8 bit;
u32 data, rmask;
@@ -1221,16 +1313,12 @@
switch (ctrl->type) {
case RK2928:
- spin_lock_irqsave(&bank->slock, flags);
-
data = BIT(bit + 16);
if (pull == PIN_CONFIG_BIAS_DISABLE)
data |= BIT(bit);
ret = regmap_write(regmap, reg, data);
-
- spin_unlock_irqrestore(&bank->slock, flags);
break;
- case RK1108:
+ case RV1108:
case RK3188:
case RK3288:
case RK3368:
@@ -1251,16 +1339,12 @@
return ret;
}
- spin_lock_irqsave(&bank->slock, flags);
-
/* enable the write to the equivalent lower bits */
data = ((1 << RK3188_PULL_BITS_PER_PIN) - 1) << (bit + 16);
rmask = data | (data >> 16);
data |= (ret << bit);
ret = regmap_update_bits(regmap, reg, rmask, data);
-
- spin_unlock_irqrestore(&bank->slock, flags);
break;
default:
dev_err(info->dev, "unsupported pinctrl type\n");
@@ -1270,6 +1354,73 @@
return ret;
}
+#define RK3328_SCHMITT_BITS_PER_PIN 1
+#define RK3328_SCHMITT_PINS_PER_REG 16
+#define RK3328_SCHMITT_BANK_STRIDE 8
+#define RK3328_SCHMITT_GRF_OFFSET 0x380
+
+static int rk3328_calc_schmitt_reg_and_bit(struct rockchip_pin_bank *bank,
+ int pin_num,
+ struct regmap **regmap,
+ int *reg, u8 *bit)
+{
+ struct rockchip_pinctrl *info = bank->drvdata;
+
+ *regmap = info->regmap_base;
+ *reg = RK3328_SCHMITT_GRF_OFFSET;
+
+ *reg += bank->bank_num * RK3328_SCHMITT_BANK_STRIDE;
+ *reg += ((pin_num / RK3328_SCHMITT_PINS_PER_REG) * 4);
+ *bit = pin_num % RK3328_SCHMITT_PINS_PER_REG;
+
+ return 0;
+}
+
+static int rockchip_get_schmitt(struct rockchip_pin_bank *bank, int pin_num)
+{
+ struct rockchip_pinctrl *info = bank->drvdata;
+ struct rockchip_pin_ctrl *ctrl = info->ctrl;
+ struct regmap *regmap;
+ int reg, ret;
+ u8 bit;
+ u32 data;
+
+ ret = ctrl->schmitt_calc_reg(bank, pin_num, ®map, ®, &bit);
+ if (ret)
+ return ret;
+
+ ret = regmap_read(regmap, reg, &data);
+ if (ret)
+ return ret;
+
+ data >>= bit;
+ return data & 0x1;
+}
+
+static int rockchip_set_schmitt(struct rockchip_pin_bank *bank,
+ int pin_num, int enable)
+{
+ struct rockchip_pinctrl *info = bank->drvdata;
+ struct rockchip_pin_ctrl *ctrl = info->ctrl;
+ struct regmap *regmap;
+ int reg, ret;
+ u8 bit;
+ u32 data, rmask;
+
+ dev_dbg(info->dev, "setting input schmitt of GPIO%d-%d to %d\n",
+ bank->bank_num, pin_num, enable);
+
+ ret = ctrl->schmitt_calc_reg(bank, pin_num, ®map, ®, &bit);
+ if (ret)
+ return ret;
+
+ /* enable the write to the equivalent lower bits */
+ data = BIT(bit + 16) | (enable << bit);
+ rmask = BIT(bit + 16) | BIT(bit);
+
+ return regmap_update_bits(regmap, reg, rmask, data);
+}
+
/*
* Pinmux_ops handling
*/
@@ -1366,7 +1517,7 @@
return ret;
clk_enable(bank->clk);
- spin_lock_irqsave(&bank->slock, flags);
+ raw_spin_lock_irqsave(&bank->slock, flags);
data = readl_relaxed(bank->reg_base + GPIO_SWPORT_DDR);
/* set bit to 1 for output, 0 for input */
@@ -1376,7 +1527,7 @@
data &= ~BIT(pin);
writel_relaxed(data, bank->reg_base + GPIO_SWPORT_DDR);
- spin_unlock_irqrestore(&bank->slock, flags);
+ raw_spin_unlock_irqrestore(&bank->slock, flags);
clk_disable(bank->clk);
return 0;
@@ -1420,7 +1571,7 @@
pull == PIN_CONFIG_BIAS_DISABLE);
case RK3066B:
return pull ? false : true;
- case RK1108:
+ case RV1108:
case RK3188:
case RK3288:
case RK3368:
@@ -1489,6 +1640,15 @@
if (rc < 0)
return rc;
break;
+ case PIN_CONFIG_INPUT_SCHMITT_ENABLE:
+ if (!info->ctrl->schmitt_calc_reg)
+ return -ENOTSUPP;
+
+ rc = rockchip_set_schmitt(bank,
+ pin - bank->pin_base, arg);
+ if (rc < 0)
+ return rc;
+ break;
default:
return -ENOTSUPP;
break;
@@ -1549,6 +1709,16 @@
arg = rc;
break;
+ case PIN_CONFIG_INPUT_SCHMITT_ENABLE:
+ if (!info->ctrl->schmitt_calc_reg)
+ return -ENOTSUPP;
+
+ rc = rockchip_get_schmitt(bank, pin - bank->pin_base);
+ if (rc < 0)
+ return rc;
+
+ arg = rc;
+ break;
default:
return -ENOTSUPP;
break;
@@ -1807,7 +1977,7 @@
u32 data;
clk_enable(bank->clk);
- spin_lock_irqsave(&bank->slock, flags);
+ raw_spin_lock_irqsave(&bank->slock, flags);
data = readl(reg);
data &= ~BIT(offset);
@@ -1815,7 +1985,7 @@
data |= BIT(offset);
writel(data, reg);
- spin_unlock_irqrestore(&bank->slock, flags);
+ raw_spin_unlock_irqrestore(&bank->slock, flags);
clk_disable(bank->clk);
}
@@ -1927,7 +2097,7 @@
data = readl_relaxed(bank->reg_base + GPIO_EXT_PORT);
do {
- spin_lock_irqsave(&bank->slock, flags);
+ raw_spin_lock_irqsave(&bank->slock, flags);
polarity = readl_relaxed(bank->reg_base +
GPIO_INT_POLARITY);
@@ -1938,7 +2108,7 @@
writel(polarity,
bank->reg_base + GPIO_INT_POLARITY);
- spin_unlock_irqrestore(&bank->slock, flags);
+ raw_spin_unlock_irqrestore(&bank->slock, flags);
data_old = data;
data = readl_relaxed(bank->reg_base +
@@ -1964,25 +2134,26 @@
int ret;
/* make sure the pin is configured as gpio input */
- ret = rockchip_set_mux(bank, d->hwirq, RK_FUNC_GPIO);
+ ret = rockchip_verify_mux(bank, d->hwirq, RK_FUNC_GPIO);
if (ret < 0)
return ret;
- clk_enable(bank->clk);
- spin_lock_irqsave(&bank->slock, flags);
+ bank->new_irqs |= mask;
+
+ raw_spin_lock_irqsave(&bank->slock, flags);
data = readl_relaxed(bank->reg_base + GPIO_SWPORT_DDR);
data &= ~mask;
writel_relaxed(data, bank->reg_base + GPIO_SWPORT_DDR);
- spin_unlock_irqrestore(&bank->slock, flags);
+ raw_spin_unlock_irqrestore(&bank->slock, flags);
if (type & IRQ_TYPE_EDGE_BOTH)
irq_set_handler_locked(d, handle_edge_irq);
else
irq_set_handler_locked(d, handle_level_irq);
- spin_lock_irqsave(&bank->slock, flags);
+ raw_spin_lock_irqsave(&bank->slock, flags);
irq_gc_lock(gc);
level = readl_relaxed(gc->reg_base + GPIO_INTTYPE_LEVEL);
@@ -2025,8 +2196,7 @@
break;
default:
irq_gc_unlock(gc);
- spin_unlock_irqrestore(&bank->slock, flags);
- clk_disable(bank->clk);
+ raw_spin_unlock_irqrestore(&bank->slock, flags);
return -EINVAL;
}
@@ -2034,8 +2204,7 @@
writel_relaxed(polarity, gc->reg_base + GPIO_INT_POLARITY);
irq_gc_unlock(gc);
- spin_unlock_irqrestore(&bank->slock, flags);
- clk_disable(bank->clk);
+ raw_spin_unlock_irqrestore(&bank->slock, flags);
return 0;
}
@@ -2061,7 +2230,7 @@
clk_disable(bank->clk);
}
-static void rockchip_irq_gc_mask_clr_bit(struct irq_data *d)
+static void rockchip_irq_enable(struct irq_data *d)
{
struct irq_chip_generic *gc = irq_data_get_irq_chip_data(d);
struct rockchip_pin_bank *bank = gc->private;
@@ -2070,7 +2239,7 @@
irq_gc_mask_clr_bit(d);
}
-static void rockchip_irq_gc_mask_set_bit(struct irq_data *d)
+static void rockchip_irq_disable(struct irq_data *d)
{
struct irq_chip_generic *gc = irq_data_get_irq_chip_data(d);
struct rockchip_pin_bank *bank = gc->private;
@@ -2079,6 +2248,34 @@
clk_disable(bank->clk);
}
+static void rockchip_irq_bus_lock(struct irq_data *d)
+{
+ struct irq_chip_generic *gc = irq_data_get_irq_chip_data(d);
+ struct rockchip_pin_bank *bank = gc->private;
+
+ clk_enable(bank->clk);
+ mutex_lock(&bank->irq_lock);
+}
+
+static void rockchip_irq_bus_sync_unlock(struct irq_data *d)
+{
+ struct irq_chip_generic *gc = irq_data_get_irq_chip_data(d);
+ struct rockchip_pin_bank *bank = gc->private;
+
+ while (bank->new_irqs) {
+ unsigned int irq = __ffs(bank->new_irqs);
+ int ret;
+
+ ret = rockchip_set_mux(bank, irq, RK_FUNC_GPIO);
+ WARN_ON(ret < 0);
+
+ bank->new_irqs &= ~BIT(irq);
+ }
+
+ mutex_unlock(&bank->irq_lock);
+ clk_disable(bank->clk);
+}
+
static int rockchip_interrupts_register(struct platform_device *pdev,
struct rockchip_pinctrl *info)
{
@@ -2137,13 +2334,17 @@
gc->chip_types[0].regs.mask = GPIO_INTMASK;
gc->chip_types[0].regs.ack = GPIO_PORTS_EOI;
gc->chip_types[0].chip.irq_ack = irq_gc_ack_set_bit;
- gc->chip_types[0].chip.irq_mask = rockchip_irq_gc_mask_set_bit;
- gc->chip_types[0].chip.irq_unmask =
- rockchip_irq_gc_mask_clr_bit;
+ gc->chip_types[0].chip.irq_mask = irq_gc_mask_set_bit;
+ gc->chip_types[0].chip.irq_unmask = irq_gc_mask_clr_bit;
+ gc->chip_types[0].chip.irq_enable = rockchip_irq_enable;
+ gc->chip_types[0].chip.irq_disable = rockchip_irq_disable;
gc->chip_types[0].chip.irq_set_wake = irq_gc_set_wake;
gc->chip_types[0].chip.irq_suspend = rockchip_irq_suspend;
gc->chip_types[0].chip.irq_resume = rockchip_irq_resume;
gc->chip_types[0].chip.irq_set_type = rockchip_irq_set_type;
+ gc->chip_types[0].chip.irq_bus_lock = rockchip_irq_bus_lock;
+ gc->chip_types[0].chip.irq_bus_sync_unlock =
+ rockchip_irq_bus_sync_unlock;
gc->wake_enabled = IRQ_MSK(bank->nr_pins);
irq_set_chained_handler_and_data(bank->irq,
@@ -2316,7 +2517,8 @@
for (i = 0; i < ctrl->nr_banks; ++i, ++bank) {
int bank_pins = 0;
- spin_lock_init(&bank->slock);
+ raw_spin_lock_init(&bank->slock);
+ mutex_init(&bank->irq_lock);
bank->drvdata = d;
bank->pin_base = ctrl->nr_pins;
ctrl->nr_pins += bank->nr_pins;
@@ -2359,7 +2561,8 @@
* Increase offset according to iomux width.
* 4bit iomux'es are spread over two registers.
*/
- inc = (iom->type & IOMUX_WIDTH_4BIT) ? 8 : 4;
+ inc = (iom->type & (IOMUX_WIDTH_4BIT |
+ IOMUX_WIDTH_3BIT)) ? 8 : 4;
if (iom->type & IOMUX_SOURCE_PMU)
pmu_offs += inc;
else
@@ -2518,7 +2721,7 @@
return 0;
}
-static struct rockchip_pin_bank rk1108_pin_banks[] = {
+static struct rockchip_pin_bank rv1108_pin_banks[] = {
PIN_BANK_IOMUX_FLAGS(0, 32, "gpio0", IOMUX_SOURCE_PMU,
IOMUX_SOURCE_PMU,
IOMUX_SOURCE_PMU,
@@ -2528,15 +2731,15 @@
PIN_BANK_IOMUX_FLAGS(3, 32, "gpio3", 0, 0, 0, 0),
};
-static struct rockchip_pin_ctrl rk1108_pin_ctrl = {
- .pin_banks = rk1108_pin_banks,
- .nr_banks = ARRAY_SIZE(rk1108_pin_banks),
- .label = "RK1108-GPIO",
- .type = RK1108,
+static struct rockchip_pin_ctrl rv1108_pin_ctrl = {
+ .pin_banks = rv1108_pin_banks,
+ .nr_banks = ARRAY_SIZE(rv1108_pin_banks),
+ .label = "RV1108-GPIO",
+ .type = RV1108,
.grf_mux_offset = 0x10,
.pmu_mux_offset = 0x0,
- .pull_calc_reg = rk1108_calc_pull_reg_and_bit,
- .drv_calc_reg = rk1108_calc_drv_reg_and_bit,
+ .pull_calc_reg = rv1108_calc_pull_reg_and_bit,
+ .drv_calc_reg = rv1108_calc_drv_reg_and_bit,
};
static struct rockchip_pin_bank rk2928_pin_banks[] = {
@@ -2679,6 +2882,32 @@
.drv_calc_reg = rk3288_calc_drv_reg_and_bit,
};
+static struct rockchip_pin_bank rk3328_pin_banks[] = {
+ PIN_BANK_IOMUX_FLAGS(0, 32, "gpio0", 0, 0, 0, 0),
+ PIN_BANK_IOMUX_FLAGS(1, 32, "gpio1", 0, 0, 0, 0),
+ PIN_BANK_IOMUX_FLAGS(2, 32, "gpio2", 0,
+ IOMUX_WIDTH_3BIT | IOMUX_RECALCED,
+ IOMUX_WIDTH_3BIT | IOMUX_RECALCED,
+ 0),
+ PIN_BANK_IOMUX_FLAGS(3, 32, "gpio3",
+ IOMUX_WIDTH_3BIT,
+ IOMUX_WIDTH_3BIT | IOMUX_RECALCED,
+ 0,
+ 0),
+};
+
+static struct rockchip_pin_ctrl rk3328_pin_ctrl = {
+ .pin_banks = rk3328_pin_banks,
+ .nr_banks = ARRAY_SIZE(rk3328_pin_banks),
+ .label = "RK3328-GPIO",
+ .type = RK3288,
+ .grf_mux_offset = 0x0,
+ .pull_calc_reg = rk3228_calc_pull_reg_and_bit,
+ .drv_calc_reg = rk3228_calc_drv_reg_and_bit,
+ .iomux_recalc = rk3328_recalc_mux,
+ .schmitt_calc_reg = rk3328_calc_schmitt_reg_and_bit,
+};
+
static struct rockchip_pin_bank rk3368_pin_banks[] = {
PIN_BANK_IOMUX_FLAGS(0, 32, "gpio0", IOMUX_SOURCE_PMU,
IOMUX_SOURCE_PMU,
@@ -2768,8 +2997,8 @@
};
static const struct of_device_id rockchip_pinctrl_dt_match[] = {
- { .compatible = "rockchip,rk1108-pinctrl",
- .data = (void *)&rk1108_pin_ctrl },
+ { .compatible = "rockchip,rv1108-pinctrl",
+ .data = (void *)&rv1108_pin_ctrl },
{ .compatible = "rockchip,rk2928-pinctrl",
.data = (void *)&rk2928_pin_ctrl },
{ .compatible = "rockchip,rk3036-pinctrl",
@@ -2784,6 +3013,8 @@
.data = (void *)&rk3228_pin_ctrl },
{ .compatible = "rockchip,rk3288-pinctrl",
.data = (void *)&rk3288_pin_ctrl },
+ { .compatible = "rockchip,rk3328-pinctrl",
+ .data = (void *)&rk3328_pin_ctrl },
{ .compatible = "rockchip,rk3368-pinctrl",
.data = (void *)&rk3368_pin_ctrl },
{ .compatible = "rockchip,rk3399-pinctrl",
diff --git a/drivers/pinctrl/pinmux.c b/drivers/pinctrl/pinmux.c
index 29ad315..9fd6d90 100644
--- a/drivers/pinctrl/pinmux.c
+++ b/drivers/pinctrl/pinmux.c
@@ -763,7 +763,7 @@
EXPORT_SYMBOL_GPL(pinmux_generic_get_function);
/**
- * pinmux_generic_get_function_groups() - gets the function groups
+ * pinmux_generic_add_function() - adds a function group
* @pctldev: pin controller device
* @name: name of the function
* @groups: array of pin groups
diff --git a/drivers/pinctrl/qcom/pinctrl-qdf2xxx.c b/drivers/pinctrl/qcom/pinctrl-qdf2xxx.c
index f448534..bb3ce5c 100644
--- a/drivers/pinctrl/qcom/pinctrl-qdf2xxx.c
+++ b/drivers/pinctrl/qcom/pinctrl-qdf2xxx.c
@@ -35,10 +35,14 @@
/* A reasonable limit to the number of GPIOS */
#define MAX_GPIOS 256
+/* maximum size of each gpio name (enough room for "gpioXXX" + null) */
+#define NAME_SIZE 8
+
static int qdf2xxx_pinctrl_probe(struct platform_device *pdev)
{
struct pinctrl_pin_desc *pins;
struct msm_pingroup *groups;
+ char (*names)[NAME_SIZE];
unsigned int i;
u32 num_gpios;
int ret;
@@ -59,15 +63,21 @@
sizeof(struct pinctrl_pin_desc), GFP_KERNEL);
groups = devm_kcalloc(&pdev->dev, num_gpios,
sizeof(struct msm_pingroup), GFP_KERNEL);
+ names = devm_kcalloc(&pdev->dev, num_gpios, NAME_SIZE, GFP_KERNEL);
- if (!pins || !groups)
+ if (!pins || !groups || !names)
return -ENOMEM;
for (i = 0; i < num_gpios; i++) {
- pins[i].number = i;
+ snprintf(names[i], NAME_SIZE, "gpio%u", i);
- groups[i].npins = 1,
+ pins[i].number = i;
+ pins[i].name = names[i];
+
+ groups[i].npins = 1;
+ groups[i].name = names[i];
groups[i].pins = &pins[i].number;
+
groups[i].ctl_reg = 0x10000 * i;
groups[i].io_reg = 0x04 + 0x10000 * i;
groups[i].intr_cfg_reg = 0x08 + 0x10000 * i;
diff --git a/drivers/pinctrl/samsung/pinctrl-exynos.c b/drivers/pinctrl/samsung/pinctrl-exynos.c
index 63e51b5..7b0e6cc 100644
--- a/drivers/pinctrl/samsung/pinctrl-exynos.c
+++ b/drivers/pinctrl/samsung/pinctrl-exynos.c
@@ -777,6 +777,7 @@
{
struct samsung_retention_ctrl *ctrl;
struct regmap *pmu_regs;
+ int i;
ctrl = devm_kzalloc(drvdata->dev, sizeof(*ctrl), GFP_KERNEL);
if (!ctrl)
@@ -794,6 +795,10 @@
ctrl->enable = exynos_retention_enable;
ctrl->disable = exynos_retention_disable;
+ /* Ensure that retention is disabled on driver init */
+ for (i = 0; i < ctrl->nr_regs; i++)
+ regmap_write(pmu_regs, ctrl->regs[i], ctrl->value);
+
return ctrl;
}
@@ -1546,6 +1551,54 @@
EXYNOS5433_PIN_BANK_EINTG(3, 0x000, "gpj1", 0x00),
};
+/* PMU pin retention groups registers for Exynos5433 (without audio & fsys) */
+static const u32 exynos5433_retention_regs[] = {
+ EXYNOS5433_PAD_RETENTION_TOP_OPTION,
+ EXYNOS5433_PAD_RETENTION_UART_OPTION,
+ EXYNOS5433_PAD_RETENTION_EBIA_OPTION,
+ EXYNOS5433_PAD_RETENTION_EBIB_OPTION,
+ EXYNOS5433_PAD_RETENTION_SPI_OPTION,
+ EXYNOS5433_PAD_RETENTION_MIF_OPTION,
+ EXYNOS5433_PAD_RETENTION_USBXTI_OPTION,
+ EXYNOS5433_PAD_RETENTION_BOOTLDO_OPTION,
+ EXYNOS5433_PAD_RETENTION_UFS_OPTION,
+ EXYNOS5433_PAD_RETENTION_FSYSGENIO_OPTION,
+};
+
+static const struct samsung_retention_data exynos5433_retention_data __initconst = {
+ .regs = exynos5433_retention_regs,
+ .nr_regs = ARRAY_SIZE(exynos5433_retention_regs),
+ .value = EXYNOS_WAKEUP_FROM_LOWPWR,
+ .refcnt = &exynos_shared_retention_refcnt,
+ .init = exynos_retention_init,
+};
+
+/* PMU retention control for audio pins can be tied to audio pin bank */
+static const u32 exynos5433_audio_retention_regs[] = {
+ EXYNOS5433_PAD_RETENTION_AUD_OPTION,
+};
+
+static const struct samsung_retention_data exynos5433_audio_retention_data __initconst = {
+ .regs = exynos5433_audio_retention_regs,
+ .nr_regs = ARRAY_SIZE(exynos5433_audio_retention_regs),
+ .value = EXYNOS_WAKEUP_FROM_LOWPWR,
+ .init = exynos_retention_init,
+};
+
+/* PMU retention control for mmc pins can be tied to fsys pin bank */
+static const u32 exynos5433_fsys_retention_regs[] = {
+ EXYNOS5433_PAD_RETENTION_MMC0_OPTION,
+ EXYNOS5433_PAD_RETENTION_MMC1_OPTION,
+ EXYNOS5433_PAD_RETENTION_MMC2_OPTION,
+};
+
+static const struct samsung_retention_data exynos5433_fsys_retention_data __initconst = {
+ .regs = exynos5433_fsys_retention_regs,
+ .nr_regs = ARRAY_SIZE(exynos5433_fsys_retention_regs),
+ .value = EXYNOS_WAKEUP_FROM_LOWPWR,
+ .init = exynos_retention_init,
+};
+
/*
* Samsung pinctrl driver data for Exynos5433 SoC. Exynos5433 SoC includes
* ten gpio/pin-mux/pinconfig controllers.
@@ -1559,6 +1612,7 @@
.suspend = exynos_pinctrl_suspend,
.resume = exynos_pinctrl_resume,
.nr_ext_resources = 1,
+ .retention_data = &exynos5433_retention_data,
}, {
/* pin-controller instance 1 data */
.pin_banks = exynos5433_pin_banks1,
@@ -1566,6 +1620,7 @@
.eint_gpio_init = exynos_eint_gpio_init,
.suspend = exynos_pinctrl_suspend,
.resume = exynos_pinctrl_resume,
+ .retention_data = &exynos5433_audio_retention_data,
}, {
/* pin-controller instance 2 data */
.pin_banks = exynos5433_pin_banks2,
@@ -1573,6 +1628,7 @@
.eint_gpio_init = exynos_eint_gpio_init,
.suspend = exynos_pinctrl_suspend,
.resume = exynos_pinctrl_resume,
+ .retention_data = &exynos5433_retention_data,
}, {
/* pin-controller instance 3 data */
.pin_banks = exynos5433_pin_banks3,
@@ -1580,6 +1636,7 @@
.eint_gpio_init = exynos_eint_gpio_init,
.suspend = exynos_pinctrl_suspend,
.resume = exynos_pinctrl_resume,
+ .retention_data = &exynos5433_retention_data,
}, {
/* pin-controller instance 4 data */
.pin_banks = exynos5433_pin_banks4,
@@ -1587,6 +1644,7 @@
.eint_gpio_init = exynos_eint_gpio_init,
.suspend = exynos_pinctrl_suspend,
.resume = exynos_pinctrl_resume,
+ .retention_data = &exynos5433_retention_data,
}, {
/* pin-controller instance 5 data */
.pin_banks = exynos5433_pin_banks5,
@@ -1594,6 +1652,7 @@
.eint_gpio_init = exynos_eint_gpio_init,
.suspend = exynos_pinctrl_suspend,
.resume = exynos_pinctrl_resume,
+ .retention_data = &exynos5433_fsys_retention_data,
}, {
/* pin-controller instance 6 data */
.pin_banks = exynos5433_pin_banks6,
@@ -1601,6 +1660,7 @@
.eint_gpio_init = exynos_eint_gpio_init,
.suspend = exynos_pinctrl_suspend,
.resume = exynos_pinctrl_resume,
+ .retention_data = &exynos5433_retention_data,
}, {
/* pin-controller instance 7 data */
.pin_banks = exynos5433_pin_banks7,
@@ -1608,6 +1668,7 @@
.eint_gpio_init = exynos_eint_gpio_init,
.suspend = exynos_pinctrl_suspend,
.resume = exynos_pinctrl_resume,
+ .retention_data = &exynos5433_retention_data,
}, {
/* pin-controller instance 8 data */
.pin_banks = exynos5433_pin_banks8,
@@ -1615,6 +1676,7 @@
.eint_gpio_init = exynos_eint_gpio_init,
.suspend = exynos_pinctrl_suspend,
.resume = exynos_pinctrl_resume,
+ .retention_data = &exynos5433_retention_data,
}, {
/* pin-controller instance 9 data */
.pin_banks = exynos5433_pin_banks9,
@@ -1622,6 +1684,7 @@
.eint_gpio_init = exynos_eint_gpio_init,
.suspend = exynos_pinctrl_suspend,
.resume = exynos_pinctrl_resume,
+ .retention_data = &exynos5433_retention_data,
},
};
diff --git a/drivers/pinctrl/samsung/pinctrl-samsung.c b/drivers/pinctrl/samsung/pinctrl-samsung.c
index d7aa22c..a4a0da5 100644
--- a/drivers/pinctrl/samsung/pinctrl-samsung.c
+++ b/drivers/pinctrl/samsung/pinctrl-samsung.c
@@ -566,13 +566,11 @@
{
const struct samsung_pin_bank_type *type;
struct samsung_pin_bank *bank;
- struct samsung_pinctrl_drv_data *drvdata;
void __iomem *reg;
u32 data, mask, shift;
bank = gpiochip_get_data(gc);
type = bank->type;
- drvdata = bank->drvdata;
reg = bank->pctl_base + bank->pctl_offset
+ type->reg_offset[PINCFG_TYPE_FUNC];
@@ -884,7 +882,7 @@
pin_bank->grange.id = bank;
pin_bank->grange.pin_base = drvdata->pin_base
+ pin_bank->pin_base;
- pin_bank->grange.base = pin_bank->gpio_chip.base;
+ pin_bank->grange.base = pin_bank->grange.pin_base;
pin_bank->grange.npins = pin_bank->gpio_chip.ngpio;
pin_bank->grange.gc = &pin_bank->gpio_chip;
pinctrl_add_gpio_range(drvdata->pctl_dev, &pin_bank->grange);
@@ -893,6 +891,19 @@
return 0;
}
+/* unregister the pinctrl interface with the pinctrl subsystem */
+static int samsung_pinctrl_unregister(struct platform_device *pdev,
+ struct samsung_pinctrl_drv_data *drvdata)
+{
+ struct samsung_pin_bank *bank = drvdata->pin_banks;
+ int i;
+
+ for (i = 0; i < drvdata->nr_banks; ++i, ++bank)
+ pinctrl_remove_gpio_range(drvdata->pctl_dev, &bank->grange);
+
+ return 0;
+}
+
static const struct gpio_chip samsung_gpiolib_chip = {
.request = gpiochip_generic_request,
.free = gpiochip_generic_free,
@@ -917,39 +928,21 @@
bank->gpio_chip = samsung_gpiolib_chip;
gc = &bank->gpio_chip;
- gc->base = drvdata->pin_base + bank->pin_base;
+ gc->base = bank->grange.base;
gc->ngpio = bank->nr_pins;
gc->parent = &pdev->dev;
gc->of_node = bank->of_node;
gc->label = bank->name;
- ret = gpiochip_add_data(gc, bank);
+ ret = devm_gpiochip_add_data(&pdev->dev, gc, bank);
if (ret) {
dev_err(&pdev->dev, "failed to register gpio_chip %s, error code: %d\n",
gc->label, ret);
- goto fail;
+ return ret;
}
}
return 0;
-
-fail:
- for (--i, --bank; i >= 0; --i, --bank)
- gpiochip_remove(&bank->gpio_chip);
- return ret;
-}
-
-/* unregister the gpiolib interface with the gpiolib subsystem */
-static int samsung_gpiolib_unregister(struct platform_device *pdev,
- struct samsung_pinctrl_drv_data *drvdata)
-{
- struct samsung_pin_bank *bank = drvdata->pin_banks;
- int i;
-
- for (i = 0; i < drvdata->nr_banks; ++i, ++bank)
- gpiochip_remove(&bank->gpio_chip);
-
- return 0;
}
/* retrieve the soc specific data */
@@ -1069,13 +1062,13 @@
return PTR_ERR(drvdata->retention_ctrl);
}
- ret = samsung_gpiolib_register(pdev, drvdata);
+ ret = samsung_pinctrl_register(pdev, drvdata);
if (ret)
return ret;
- ret = samsung_pinctrl_register(pdev, drvdata);
+ ret = samsung_gpiolib_register(pdev, drvdata);
if (ret) {
- samsung_gpiolib_unregister(pdev, drvdata);
+ samsung_pinctrl_unregister(pdev, drvdata);
return ret;
}
diff --git a/drivers/pinctrl/sh-pfc/Makefile b/drivers/pinctrl/sh-pfc/Makefile
index 2dda8c6..8e08684 100644
--- a/drivers/pinctrl/sh-pfc/Makefile
+++ b/drivers/pinctrl/sh-pfc/Makefile
@@ -11,6 +11,7 @@
obj-$(CONFIG_PINCTRL_PFC_R8A7793) += pfc-r8a7791.o
obj-$(CONFIG_PINCTRL_PFC_R8A7794) += pfc-r8a7794.o
obj-$(CONFIG_PINCTRL_PFC_R8A7795) += pfc-r8a7795.o
+obj-$(CONFIG_PINCTRL_PFC_R8A7795) += pfc-r8a7795-es1.o
obj-$(CONFIG_PINCTRL_PFC_R8A7796) += pfc-r8a7796.o
obj-$(CONFIG_PINCTRL_PFC_SH7203) += pfc-sh7203.o
obj-$(CONFIG_PINCTRL_PFC_SH7264) += pfc-sh7264.o
diff --git a/drivers/pinctrl/sh-pfc/core.c b/drivers/pinctrl/sh-pfc/core.c
index cf80ce1..4a5a0fe 100644
--- a/drivers/pinctrl/sh-pfc/core.c
+++ b/drivers/pinctrl/sh-pfc/core.c
@@ -586,6 +586,9 @@
ret = info->ops->init(pfc);
if (ret < 0)
return ret;
+
+ /* .init() may have overridden pfc->info */
+ info = pfc->info;
}
/* Enable dummy states for those platforms without pinctrl support */
diff --git a/drivers/pinctrl/sh-pfc/pfc-r8a7791.c b/drivers/pinctrl/sh-pfc/pfc-r8a7791.c
index 841cecd..2ed7eeb 100644
--- a/drivers/pinctrl/sh-pfc/pfc-r8a7791.c
+++ b/drivers/pinctrl/sh-pfc/pfc-r8a7791.c
@@ -203,7 +203,7 @@
/* IPSR6 */
FN_AUDIO_CLKB, FN_STP_OPWM_0_B, FN_MSIOF1_SCK_B,
- FN_SCIF_CLK, FN_BPFCLK_E,
+ FN_SCIF_CLK, FN_DVC_MUTE, FN_BPFCLK_E,
FN_AUDIO_CLKC, FN_SCIFB0_SCK_C, FN_MSIOF1_SYNC_B, FN_RX2,
FN_SCIFA2_RXD, FN_FMIN_E,
FN_AUDIO_CLKOUT, FN_MSIOF1_SS1_B, FN_TX2, FN_SCIFA2_TXD,
@@ -573,7 +573,7 @@
/* IPSR6 */
AUDIO_CLKB_MARK, STP_OPWM_0_B_MARK, MSIOF1_SCK_B_MARK,
- SCIF_CLK_MARK, BPFCLK_E_MARK,
+ SCIF_CLK_MARK, DVC_MUTE_MARK, BPFCLK_E_MARK,
AUDIO_CLKC_MARK, SCIFB0_SCK_C_MARK, MSIOF1_SYNC_B_MARK, RX2_MARK,
SCIFA2_RXD_MARK, FMIN_E_MARK,
AUDIO_CLKOUT_MARK, MSIOF1_SS1_B_MARK, TX2_MARK, SCIFA2_TXD_MARK,
@@ -1010,14 +1010,17 @@
PINMUX_IPSR_MSEL(IP4_12_10, SCL2, SEL_IIC2_0),
PINMUX_IPSR_MSEL(IP4_12_10, GPS_CLK_B, SEL_GPS_1),
PINMUX_IPSR_MSEL(IP4_12_10, GLO_Q0_D, SEL_GPS_3),
+ PINMUX_IPSR_MSEL(IP4_12_10, HSCK1_E, SEL_HSCIF1_4),
PINMUX_IPSR_GPSR(IP4_15_13, SSI_WS2),
PINMUX_IPSR_MSEL(IP4_15_13, SDA2, SEL_IIC2_0),
PINMUX_IPSR_MSEL(IP4_15_13, GPS_SIGN_B, SEL_GPS_1),
PINMUX_IPSR_MSEL(IP4_15_13, RX2_E, SEL_SCIF2_4),
PINMUX_IPSR_MSEL(IP4_15_13, GLO_Q1_D, SEL_GPS_3),
+ PINMUX_IPSR_MSEL(IP4_15_13, HCTS1_N_E, SEL_HSCIF1_4),
PINMUX_IPSR_GPSR(IP4_18_16, SSI_SDATA2),
PINMUX_IPSR_MSEL(IP4_18_16, GPS_MAG_B, SEL_GPS_1),
PINMUX_IPSR_MSEL(IP4_18_16, TX2_E, SEL_SCIF2_4),
+ PINMUX_IPSR_MSEL(IP4_18_16, HRTS1_N_E, SEL_HSCIF1_4),
PINMUX_IPSR_GPSR(IP4_19, SSI_SCK34),
PINMUX_IPSR_GPSR(IP4_20, SSI_WS34),
PINMUX_IPSR_GPSR(IP4_21, SSI_SDATA3),
@@ -1090,6 +1093,7 @@
PINMUX_IPSR_MSEL(IP6_2_0, STP_OPWM_0_B, SEL_SSP_1),
PINMUX_IPSR_MSEL(IP6_2_0, MSIOF1_SCK_B, SEL_SOF1_1),
PINMUX_IPSR_MSEL(IP6_2_0, SCIF_CLK, SEL_SCIF_0),
+ PINMUX_IPSR_GPSR(IP6_2_0, DVC_MUTE),
PINMUX_IPSR_MSEL(IP6_2_0, BPFCLK_E, SEL_FM_4),
PINMUX_IPSR_GPSR(IP6_5_3, AUDIO_CLKC),
PINMUX_IPSR_MSEL(IP6_5_3, SCIFB0_SCK_C, SEL_SCIFB_2),
@@ -1099,7 +1103,7 @@
PINMUX_IPSR_MSEL(IP6_5_3, FMIN_E, SEL_FM_4),
PINMUX_IPSR_GPSR(IP6_7_6, AUDIO_CLKOUT),
PINMUX_IPSR_MSEL(IP6_7_6, MSIOF1_SS1_B, SEL_SOF1_1),
- PINMUX_IPSR_MSEL(IP6_5_3, TX2, SEL_SCIF2_0),
+ PINMUX_IPSR_MSEL(IP6_7_6, TX2, SEL_SCIF2_0),
PINMUX_IPSR_MSEL(IP6_7_6, SCIFA2_TXD, SEL_SCIFA2_0),
PINMUX_IPSR_GPSR(IP6_9_8, IRQ0),
PINMUX_IPSR_MSEL(IP6_9_8, SCIFB1_RXD_D, SEL_SCIFB1_3),
@@ -5707,7 +5711,7 @@
},
{ PINMUX_CFG_REG_VAR("IPSR2", 0xE6060028, 32,
2, 3, 2, 2, 2, 2, 3, 3, 3, 3, 2, 2, 3) {
- /* IP2_31_20 [2] */
+ /* IP2_31_30 [2] */
0, 0, 0, 0,
/* IP2_29_27 [3] */
FN_EX_CS3_N, FN_ATADIR0_N, FN_MSIOF2_TXD,
@@ -5727,7 +5731,7 @@
/* IP2_15_13 [3] */
FN_A24, FN_DREQ2, FN_IO3, FN_TX1, FN_SCIFA1_TXD,
0, 0, 0,
- /* IP2_12_0 [3] */
+ /* IP2_12_10 [3] */
FN_A23, FN_IO2, FN_BPFCLK_B, FN_RX0, FN_SCIFA0_RXD,
0, 0, 0,
/* IP2_9_7 [3] */
@@ -5896,7 +5900,7 @@
0, 0,
/* IP6_2_0 [3] */
FN_AUDIO_CLKB, FN_STP_OPWM_0_B, FN_MSIOF1_SCK_B,
- FN_SCIF_CLK, 0, FN_BPFCLK_E,
+ FN_SCIF_CLK, FN_DVC_MUTE, FN_BPFCLK_E,
0, 0, }
},
{ PINMUX_CFG_REG_VAR("IPSR7", 0xE606003C, 32,
@@ -6038,7 +6042,7 @@
/* IP10_24_22 [3] */
FN_VI0_R1, FN_VI2_DATA2, FN_GLO_I1_B, FN_TS_SCK0_C, FN_ATAG1_N,
0, 0, 0,
- /* IP10_21_29 [3] */
+ /* IP10_21_19 [3] */
FN_VI0_R0, FN_VI2_DATA1, FN_GLO_I0_B,
FN_TS_SDATA0_C, FN_ATACS11_N,
0, 0, 0,
diff --git a/drivers/pinctrl/sh-pfc/pfc-r8a7794.c b/drivers/pinctrl/sh-pfc/pfc-r8a7794.c
index ed734f56..ef093ac 100644
--- a/drivers/pinctrl/sh-pfc/pfc-r8a7794.c
+++ b/drivers/pinctrl/sh-pfc/pfc-r8a7794.c
@@ -281,8 +281,8 @@
FN_AVB_AVTP_CAPTURE, FN_ETH_CRS_DV_B, FN_SSI_WS1, FN_SCIF1_TXD_B,
FN_IIC1_SDA_C, FN_VI1_DATA0, FN_CAN0_TX_D, FN_AVB_AVTP_MATCH,
FN_ETH_RX_ER_B, FN_SSI_SDATA1, FN_HSCIF1_HRX_B, FN_SDATA, FN_VI1_DATA1,
- FN_ATAG0_N, FN_ETH_RXD0_B, FN_SSI_SCK2, FN_HSCIF1_HTX_B, FN_VI1_DATA2,
- FN_MDATA, FN_ATAWR0_N, FN_ETH_RXD1_B,
+ FN_ATAWR0_N, FN_ETH_RXD0_B, FN_SSI_SCK2, FN_HSCIF1_HTX_B, FN_VI1_DATA2,
+ FN_MDATA, FN_ATAG0_N, FN_ETH_RXD1_B,
/* IPSR13 */
FN_SSI_WS2, FN_HSCIF1_HCTS_N_B, FN_SCIFA0_RXD_D, FN_VI1_DATA3, FN_SCKZ,
@@ -575,8 +575,8 @@
ETH_CRS_DV_B_MARK, SSI_WS1_MARK, SCIF1_TXD_B_MARK, IIC1_SDA_C_MARK,
VI1_DATA0_MARK, CAN0_TX_D_MARK, AVB_AVTP_MATCH_MARK, ETH_RX_ER_B_MARK,
SSI_SDATA1_MARK, HSCIF1_HRX_B_MARK, VI1_DATA1_MARK, SDATA_MARK,
- ATAG0_N_MARK, ETH_RXD0_B_MARK, SSI_SCK2_MARK, HSCIF1_HTX_B_MARK,
- VI1_DATA2_MARK, MDATA_MARK, ATAWR0_N_MARK, ETH_RXD1_B_MARK,
+ ATAWR0_N_MARK, ETH_RXD0_B_MARK, SSI_SCK2_MARK, HSCIF1_HTX_B_MARK,
+ VI1_DATA2_MARK, MDATA_MARK, ATAG0_N_MARK, ETH_RXD1_B_MARK,
/* IPSR13 */
SSI_WS2_MARK, HSCIF1_HCTS_N_B_MARK, SCIFA0_RXD_D_MARK, VI1_DATA3_MARK,
@@ -1413,13 +1413,13 @@
PINMUX_IPSR_MSEL(IP12_26_24, HSCIF1_HRX_B, SEL_HSCIF1_1),
PINMUX_IPSR_GPSR(IP12_26_24, VI1_DATA1),
PINMUX_IPSR_MSEL(IP12_26_24, SDATA, SEL_FSN_0),
- PINMUX_IPSR_GPSR(IP12_26_24, ATAG0_N),
+ PINMUX_IPSR_GPSR(IP12_26_24, ATAWR0_N),
PINMUX_IPSR_MSEL(IP12_26_24, ETH_RXD0_B, SEL_ETH_1),
PINMUX_IPSR_MSEL(IP12_29_27, SSI_SCK2, SEL_SSI2_0),
PINMUX_IPSR_MSEL(IP12_29_27, HSCIF1_HTX_B, SEL_HSCIF1_1),
PINMUX_IPSR_GPSR(IP12_29_27, VI1_DATA2),
PINMUX_IPSR_MSEL(IP12_29_27, MDATA, SEL_FSN_0),
- PINMUX_IPSR_GPSR(IP12_29_27, ATAWR0_N),
+ PINMUX_IPSR_GPSR(IP12_29_27, ATAG0_N),
PINMUX_IPSR_MSEL(IP12_29_27, ETH_RXD1_B, SEL_ETH_1),
/* IPSR13 */
@@ -4938,10 +4938,10 @@
0, 0, 0, 0,
/* IP12_29_27 [3] */
FN_SSI_SCK2, FN_HSCIF1_HTX_B, FN_VI1_DATA2, FN_MDATA,
- FN_ATAWR0_N, FN_ETH_RXD1_B, 0, 0,
+ FN_ATAG0_N, FN_ETH_RXD1_B, 0, 0,
/* IP12_26_24 [3] */
FN_SSI_SDATA1, FN_HSCIF1_HRX_B, FN_VI1_DATA1, FN_SDATA,
- FN_ATAG0_N, FN_ETH_RXD0_B, 0, 0,
+ FN_ATAWR0_N, FN_ETH_RXD0_B, 0, 0,
/* IP12_23_21 [3] */
FN_SSI_WS1, FN_SCIF1_TXD_B, FN_IIC1_SDA_C, FN_VI1_DATA0,
FN_CAN0_TX_D, FN_AVB_AVTP_MATCH, FN_ETH_RX_ER_B, 0,
diff --git a/drivers/pinctrl/sh-pfc/pfc-r8a7795-es1.c b/drivers/pinctrl/sh-pfc/pfc-r8a7795-es1.c
new file mode 100644
index 0000000..081efda
--- /dev/null
+++ b/drivers/pinctrl/sh-pfc/pfc-r8a7795-es1.c
@@ -0,0 +1,5705 @@
+/*
+ * R8A7795 ES1.x processor support - PFC hardware block.
+ *
+ * Copyright (C) 2015 Renesas Electronics Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; version 2 of the License.
+ */
+
+#include <linux/kernel.h>
+
+#include "core.h"
+#include "sh_pfc.h"
+
+#define CFG_FLAGS (SH_PFC_PIN_CFG_DRIVE_STRENGTH | \
+ SH_PFC_PIN_CFG_PULL_UP | \
+ SH_PFC_PIN_CFG_PULL_DOWN)
+
+#define CPU_ALL_PORT(fn, sfx) \
+ PORT_GP_CFG_16(0, fn, sfx, CFG_FLAGS), \
+ PORT_GP_CFG_28(1, fn, sfx, CFG_FLAGS), \
+ PORT_GP_CFG_15(2, fn, sfx, CFG_FLAGS), \
+ PORT_GP_CFG_12(3, fn, sfx, CFG_FLAGS | SH_PFC_PIN_CFG_IO_VOLTAGE), \
+ PORT_GP_CFG_1(3, 12, fn, sfx, CFG_FLAGS), \
+ PORT_GP_CFG_1(3, 13, fn, sfx, CFG_FLAGS), \
+ PORT_GP_CFG_1(3, 14, fn, sfx, CFG_FLAGS), \
+ PORT_GP_CFG_1(3, 15, fn, sfx, CFG_FLAGS), \
+ PORT_GP_CFG_18(4, fn, sfx, CFG_FLAGS | SH_PFC_PIN_CFG_IO_VOLTAGE), \
+ PORT_GP_CFG_26(5, fn, sfx, CFG_FLAGS), \
+ PORT_GP_CFG_32(6, fn, sfx, CFG_FLAGS), \
+ PORT_GP_CFG_4(7, fn, sfx, CFG_FLAGS)
+/*
+ * F_() : just information
+ * FM() : macro for FN_xxx / xxx_MARK
+ */
+
+/* GPSR0 */
+#define GPSR0_15 F_(D15, IP7_11_8)
+#define GPSR0_14 F_(D14, IP7_7_4)
+#define GPSR0_13 F_(D13, IP7_3_0)
+#define GPSR0_12 F_(D12, IP6_31_28)
+#define GPSR0_11 F_(D11, IP6_27_24)
+#define GPSR0_10 F_(D10, IP6_23_20)
+#define GPSR0_9 F_(D9, IP6_19_16)
+#define GPSR0_8 F_(D8, IP6_15_12)
+#define GPSR0_7 F_(D7, IP6_11_8)
+#define GPSR0_6 F_(D6, IP6_7_4)
+#define GPSR0_5 F_(D5, IP6_3_0)
+#define GPSR0_4 F_(D4, IP5_31_28)
+#define GPSR0_3 F_(D3, IP5_27_24)
+#define GPSR0_2 F_(D2, IP5_23_20)
+#define GPSR0_1 F_(D1, IP5_19_16)
+#define GPSR0_0 F_(D0, IP5_15_12)
+
+/* GPSR1 */
+#define GPSR1_27 F_(EX_WAIT0_A, IP5_11_8)
+#define GPSR1_26 F_(WE1_N, IP5_7_4)
+#define GPSR1_25 F_(WE0_N, IP5_3_0)
+#define GPSR1_24 F_(RD_WR_N, IP4_31_28)
+#define GPSR1_23 F_(RD_N, IP4_27_24)
+#define GPSR1_22 F_(BS_N, IP4_23_20)
+#define GPSR1_21 F_(CS1_N_A26, IP4_19_16)
+#define GPSR1_20 F_(CS0_N, IP4_15_12)
+#define GPSR1_19 F_(A19, IP4_11_8)
+#define GPSR1_18 F_(A18, IP4_7_4)
+#define GPSR1_17 F_(A17, IP4_3_0)
+#define GPSR1_16 F_(A16, IP3_31_28)
+#define GPSR1_15 F_(A15, IP3_27_24)
+#define GPSR1_14 F_(A14, IP3_23_20)
+#define GPSR1_13 F_(A13, IP3_19_16)
+#define GPSR1_12 F_(A12, IP3_15_12)
+#define GPSR1_11 F_(A11, IP3_11_8)
+#define GPSR1_10 F_(A10, IP3_7_4)
+#define GPSR1_9 F_(A9, IP3_3_0)
+#define GPSR1_8 F_(A8, IP2_31_28)
+#define GPSR1_7 F_(A7, IP2_27_24)
+#define GPSR1_6 F_(A6, IP2_23_20)
+#define GPSR1_5 F_(A5, IP2_19_16)
+#define GPSR1_4 F_(A4, IP2_15_12)
+#define GPSR1_3 F_(A3, IP2_11_8)
+#define GPSR1_2 F_(A2, IP2_7_4)
+#define GPSR1_1 F_(A1, IP2_3_0)
+#define GPSR1_0 F_(A0, IP1_31_28)
+
+/* GPSR2 */
+#define GPSR2_14 F_(AVB_AVTP_CAPTURE_A, IP0_23_20)
+#define GPSR2_13 F_(AVB_AVTP_MATCH_A, IP0_19_16)
+#define GPSR2_12 F_(AVB_LINK, IP0_15_12)
+#define GPSR2_11 F_(AVB_PHY_INT, IP0_11_8)
+#define GPSR2_10 F_(AVB_MAGIC, IP0_7_4)
+#define GPSR2_9 F_(AVB_MDC, IP0_3_0)
+#define GPSR2_8 F_(PWM2_A, IP1_27_24)
+#define GPSR2_7 F_(PWM1_A, IP1_23_20)
+#define GPSR2_6 F_(PWM0, IP1_19_16)
+#define GPSR2_5 F_(IRQ5, IP1_15_12)
+#define GPSR2_4 F_(IRQ4, IP1_11_8)
+#define GPSR2_3 F_(IRQ3, IP1_7_4)
+#define GPSR2_2 F_(IRQ2, IP1_3_0)
+#define GPSR2_1 F_(IRQ1, IP0_31_28)
+#define GPSR2_0 F_(IRQ0, IP0_27_24)
+
+/* GPSR3 */
+#define GPSR3_15 F_(SD1_WP, IP10_23_20)
+#define GPSR3_14 F_(SD1_CD, IP10_19_16)
+#define GPSR3_13 F_(SD0_WP, IP10_15_12)
+#define GPSR3_12 F_(SD0_CD, IP10_11_8)
+#define GPSR3_11 F_(SD1_DAT3, IP8_31_28)
+#define GPSR3_10 F_(SD1_DAT2, IP8_27_24)
+#define GPSR3_9 F_(SD1_DAT1, IP8_23_20)
+#define GPSR3_8 F_(SD1_DAT0, IP8_19_16)
+#define GPSR3_7 F_(SD1_CMD, IP8_15_12)
+#define GPSR3_6 F_(SD1_CLK, IP8_11_8)
+#define GPSR3_5 F_(SD0_DAT3, IP8_7_4)
+#define GPSR3_4 F_(SD0_DAT2, IP8_3_0)
+#define GPSR3_3 F_(SD0_DAT1, IP7_31_28)
+#define GPSR3_2 F_(SD0_DAT0, IP7_27_24)
+#define GPSR3_1 F_(SD0_CMD, IP7_23_20)
+#define GPSR3_0 F_(SD0_CLK, IP7_19_16)
+
+/* GPSR4 */
+#define GPSR4_17 FM(SD3_DS)
+#define GPSR4_16 F_(SD3_DAT7, IP10_7_4)
+#define GPSR4_15 F_(SD3_DAT6, IP10_3_0)
+#define GPSR4_14 F_(SD3_DAT5, IP9_31_28)
+#define GPSR4_13 F_(SD3_DAT4, IP9_27_24)
+#define GPSR4_12 FM(SD3_DAT3)
+#define GPSR4_11 FM(SD3_DAT2)
+#define GPSR4_10 FM(SD3_DAT1)
+#define GPSR4_9 FM(SD3_DAT0)
+#define GPSR4_8 FM(SD3_CMD)
+#define GPSR4_7 FM(SD3_CLK)
+#define GPSR4_6 F_(SD2_DS, IP9_23_20)
+#define GPSR4_5 F_(SD2_DAT3, IP9_19_16)
+#define GPSR4_4 F_(SD2_DAT2, IP9_15_12)
+#define GPSR4_3 F_(SD2_DAT1, IP9_11_8)
+#define GPSR4_2 F_(SD2_DAT0, IP9_7_4)
+#define GPSR4_1 FM(SD2_CMD)
+#define GPSR4_0 F_(SD2_CLK, IP9_3_0)
+
+/* GPSR5 */
+#define GPSR5_25 F_(MLB_DAT, IP13_19_16)
+#define GPSR5_24 F_(MLB_SIG, IP13_15_12)
+#define GPSR5_23 F_(MLB_CLK, IP13_11_8)
+#define GPSR5_22 FM(MSIOF0_RXD)
+#define GPSR5_21 F_(MSIOF0_SS2, IP13_7_4)
+#define GPSR5_20 FM(MSIOF0_TXD)
+#define GPSR5_19 F_(MSIOF0_SS1, IP13_3_0)
+#define GPSR5_18 F_(MSIOF0_SYNC, IP12_31_28)
+#define GPSR5_17 FM(MSIOF0_SCK)
+#define GPSR5_16 F_(HRTS0_N, IP12_27_24)
+#define GPSR5_15 F_(HCTS0_N, IP12_23_20)
+#define GPSR5_14 F_(HTX0, IP12_19_16)
+#define GPSR5_13 F_(HRX0, IP12_15_12)
+#define GPSR5_12 F_(HSCK0, IP12_11_8)
+#define GPSR5_11 F_(RX2_A, IP12_7_4)
+#define GPSR5_10 F_(TX2_A, IP12_3_0)
+#define GPSR5_9 F_(SCK2, IP11_31_28)
+#define GPSR5_8 F_(RTS1_N_TANS, IP11_27_24)
+#define GPSR5_7 F_(CTS1_N, IP11_23_20)
+#define GPSR5_6 F_(TX1_A, IP11_19_16)
+#define GPSR5_5 F_(RX1_A, IP11_15_12)
+#define GPSR5_4 F_(RTS0_N_TANS, IP11_11_8)
+#define GPSR5_3 F_(CTS0_N, IP11_7_4)
+#define GPSR5_2 F_(TX0, IP11_3_0)
+#define GPSR5_1 F_(RX0, IP10_31_28)
+#define GPSR5_0 F_(SCK0, IP10_27_24)
+
+/* GPSR6 */
+#define GPSR6_31 F_(USB31_OVC, IP17_7_4)
+#define GPSR6_30 F_(USB31_PWEN, IP17_3_0)
+#define GPSR6_29 F_(USB30_OVC, IP16_31_28)
+#define GPSR6_28 F_(USB30_PWEN, IP16_27_24)
+#define GPSR6_27 F_(USB1_OVC, IP16_23_20)
+#define GPSR6_26 F_(USB1_PWEN, IP16_19_16)
+#define GPSR6_25 F_(USB0_OVC, IP16_15_12)
+#define GPSR6_24 F_(USB0_PWEN, IP16_11_8)
+#define GPSR6_23 F_(AUDIO_CLKB_B, IP16_7_4)
+#define GPSR6_22 F_(AUDIO_CLKA_A, IP16_3_0)
+#define GPSR6_21 F_(SSI_SDATA9_A, IP15_31_28)
+#define GPSR6_20 F_(SSI_SDATA8, IP15_27_24)
+#define GPSR6_19 F_(SSI_SDATA7, IP15_23_20)
+#define GPSR6_18 F_(SSI_WS78, IP15_19_16)
+#define GPSR6_17 F_(SSI_SCK78, IP15_15_12)
+#define GPSR6_16 F_(SSI_SDATA6, IP15_11_8)
+#define GPSR6_15 F_(SSI_WS6, IP15_7_4)
+#define GPSR6_14 F_(SSI_SCK6, IP15_3_0)
+#define GPSR6_13 FM(SSI_SDATA5)
+#define GPSR6_12 FM(SSI_WS5)
+#define GPSR6_11 FM(SSI_SCK5)
+#define GPSR6_10 F_(SSI_SDATA4, IP14_31_28)
+#define GPSR6_9 F_(SSI_WS4, IP14_27_24)
+#define GPSR6_8 F_(SSI_SCK4, IP14_23_20)
+#define GPSR6_7 F_(SSI_SDATA3, IP14_19_16)
+#define GPSR6_6 F_(SSI_WS34, IP14_15_12)
+#define GPSR6_5 F_(SSI_SCK34, IP14_11_8)
+#define GPSR6_4 F_(SSI_SDATA2_A, IP14_7_4)
+#define GPSR6_3 F_(SSI_SDATA1_A, IP14_3_0)
+#define GPSR6_2 F_(SSI_SDATA0, IP13_31_28)
+#define GPSR6_1 F_(SSI_WS01239, IP13_27_24)
+#define GPSR6_0 F_(SSI_SCK01239, IP13_23_20)
+
+/* GPSR7 */
+#define GPSR7_3 FM(HDMI1_CEC)
+#define GPSR7_2 FM(HDMI0_CEC)
+#define GPSR7_1 FM(AVS2)
+#define GPSR7_0 FM(AVS1)
+
+
+/* IPSRx */ /* 0 */ /* 1 */ /* 2 */ /* 3 */ /* 4 */ /* 5 */ /* 6 */ /* 7 */ /* 8 */ /* 9 */ /* A */ /* B */ /* C - F */
+#define IP0_3_0 FM(AVB_MDC) F_(0, 0) FM(MSIOF2_SS2_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP0_7_4 FM(AVB_MAGIC) F_(0, 0) FM(MSIOF2_SS1_C) FM(SCK4_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP0_11_8 FM(AVB_PHY_INT) F_(0, 0) FM(MSIOF2_SYNC_C) FM(RX4_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP0_15_12 FM(AVB_LINK) F_(0, 0) FM(MSIOF2_SCK_C) FM(TX4_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP0_19_16 FM(AVB_AVTP_MATCH_A) F_(0, 0) FM(MSIOF2_RXD_C) FM(CTS4_N_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP0_23_20 FM(AVB_AVTP_CAPTURE_A) F_(0, 0) FM(MSIOF2_TXD_C) FM(RTS4_N_TANS_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP0_27_24 FM(IRQ0) FM(QPOLB) F_(0, 0) FM(DU_CDE) FM(VI4_DATA0_B) FM(CAN0_TX_B) FM(CANFD0_TX_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP0_31_28 FM(IRQ1) FM(QPOLA) F_(0, 0) FM(DU_DISP) FM(VI4_DATA1_B) FM(CAN0_RX_B) FM(CANFD0_RX_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP1_3_0 FM(IRQ2) FM(QCPV_QDE) F_(0, 0) FM(DU_EXODDF_DU_ODDF_DISP_CDE) FM(VI4_DATA2_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(PWM3_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP1_7_4 FM(IRQ3) FM(QSTVB_QVE) FM(A25) FM(DU_DOTCLKOUT1) FM(VI4_DATA3_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(PWM4_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP1_11_8 FM(IRQ4) FM(QSTH_QHS) FM(A24) FM(DU_EXHSYNC_DU_HSYNC) FM(VI4_DATA4_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(PWM5_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP1_15_12 FM(IRQ5) FM(QSTB_QHE) FM(A23) FM(DU_EXVSYNC_DU_VSYNC) FM(VI4_DATA5_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(PWM6_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP1_19_16 FM(PWM0) FM(AVB_AVTP_PPS)FM(A22) F_(0, 0) FM(VI4_DATA6_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(IECLK_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP1_23_20 FM(PWM1_A) F_(0, 0) FM(A21) FM(HRX3_D) FM(VI4_DATA7_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(IERX_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP1_27_24 FM(PWM2_A) F_(0, 0) FM(A20) FM(HTX3_D) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(IETX_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP1_31_28 FM(A0) FM(LCDOUT16) FM(MSIOF3_SYNC_B) F_(0, 0) FM(VI4_DATA8) F_(0, 0) FM(DU_DB0) F_(0, 0) F_(0, 0) FM(PWM3_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP2_3_0 FM(A1) FM(LCDOUT17) FM(MSIOF3_TXD_B) F_(0, 0) FM(VI4_DATA9) F_(0, 0) FM(DU_DB1) F_(0, 0) F_(0, 0) FM(PWM4_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP2_7_4 FM(A2) FM(LCDOUT18) FM(MSIOF3_SCK_B) F_(0, 0) FM(VI4_DATA10) F_(0, 0) FM(DU_DB2) F_(0, 0) F_(0, 0) FM(PWM5_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP2_11_8 FM(A3) FM(LCDOUT19) FM(MSIOF3_RXD_B) F_(0, 0) FM(VI4_DATA11) F_(0, 0) FM(DU_DB3) F_(0, 0) F_(0, 0) FM(PWM6_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+
+/* IPSRx */ /* 0 */ /* 1 */ /* 2 */ /* 3 */ /* 4 */ /* 5 */ /* 6 */ /* 7 */ /* 8 */ /* 9 */ /* A */ /* B */ /* C - F */
+#define IP2_15_12 FM(A4) FM(LCDOUT20) FM(MSIOF3_SS1_B) F_(0, 0) FM(VI4_DATA12) FM(VI5_DATA12) FM(DU_DB4) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP2_19_16 FM(A5) FM(LCDOUT21) FM(MSIOF3_SS2_B) FM(SCK4_B) FM(VI4_DATA13) FM(VI5_DATA13) FM(DU_DB5) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP2_23_20 FM(A6) FM(LCDOUT22) FM(MSIOF2_SS1_A) FM(RX4_B) FM(VI4_DATA14) FM(VI5_DATA14) FM(DU_DB6) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP2_27_24 FM(A7) FM(LCDOUT23) FM(MSIOF2_SS2_A) FM(TX4_B) FM(VI4_DATA15) FM(VI5_DATA15) FM(DU_DB7) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP2_31_28 FM(A8) FM(RX3_B) FM(MSIOF2_SYNC_A) FM(HRX4_B) F_(0, 0) F_(0, 0) F_(0, 0) FM(SDA6_A) FM(AVB_AVTP_MATCH_B) FM(PWM1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP3_3_0 FM(A9) F_(0, 0) FM(MSIOF2_SCK_A) FM(CTS4_N_B) F_(0, 0) FM(VI5_VSYNC_N) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP3_7_4 FM(A10) F_(0, 0) FM(MSIOF2_RXD_A) FM(RTS4_N_TANS_B) F_(0, 0) FM(VI5_HSYNC_N) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP3_11_8 FM(A11) FM(TX3_B) FM(MSIOF2_TXD_A) FM(HTX4_B) FM(HSCK4) FM(VI5_FIELD) F_(0, 0) FM(SCL6_A) FM(AVB_AVTP_CAPTURE_B) FM(PWM2_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP3_15_12 FM(A12) FM(LCDOUT12) FM(MSIOF3_SCK_C) F_(0, 0) FM(HRX4_A) FM(VI5_DATA8) FM(DU_DG4) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP3_19_16 FM(A13) FM(LCDOUT13) FM(MSIOF3_SYNC_C) F_(0, 0) FM(HTX4_A) FM(VI5_DATA9) FM(DU_DG5) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP3_23_20 FM(A14) FM(LCDOUT14) FM(MSIOF3_RXD_C) F_(0, 0) FM(HCTS4_N) FM(VI5_DATA10) FM(DU_DG6) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP3_27_24 FM(A15) FM(LCDOUT15) FM(MSIOF3_TXD_C) F_(0, 0) FM(HRTS4_N) FM(VI5_DATA11) FM(DU_DG7) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP3_31_28 FM(A16) FM(LCDOUT8) F_(0, 0) F_(0, 0) FM(VI4_FIELD) F_(0, 0) FM(DU_DG0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP4_3_0 FM(A17) FM(LCDOUT9) F_(0, 0) F_(0, 0) FM(VI4_VSYNC_N) F_(0, 0) FM(DU_DG1) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP4_7_4 FM(A18) FM(LCDOUT10) F_(0, 0) F_(0, 0) FM(VI4_HSYNC_N) F_(0, 0) FM(DU_DG2) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP4_11_8 FM(A19) FM(LCDOUT11) F_(0, 0) F_(0, 0) FM(VI4_CLKENB) F_(0, 0) FM(DU_DG3) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP4_15_12 FM(CS0_N) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(VI5_CLKENB) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP4_19_16 FM(CS1_N_A26) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(VI5_CLK) F_(0, 0) FM(EX_WAIT0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP4_23_20 FM(BS_N) FM(QSTVA_QVS) FM(MSIOF3_SCK_D) FM(SCK3) FM(HSCK3) F_(0, 0) F_(0, 0) F_(0, 0) FM(CAN1_TX) FM(CANFD1_TX) FM(IETX_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP4_27_24 FM(RD_N) F_(0, 0) FM(MSIOF3_SYNC_D) FM(RX3_A) FM(HRX3_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(CAN0_TX_A) FM(CANFD0_TX_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP4_31_28 FM(RD_WR_N) F_(0, 0) FM(MSIOF3_RXD_D) FM(TX3_A) FM(HTX3_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(CAN0_RX_A) FM(CANFD0_RX_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP5_3_0 FM(WE0_N) F_(0, 0) FM(MSIOF3_TXD_D) FM(CTS3_N) FM(HCTS3_N) F_(0, 0) F_(0, 0) FM(SCL6_B) FM(CAN_CLK) F_(0, 0) FM(IECLK_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP5_7_4 FM(WE1_N) F_(0, 0) FM(MSIOF3_SS1_D) FM(RTS3_N_TANS) FM(HRTS3_N) F_(0, 0) F_(0, 0) FM(SDA6_B) FM(CAN1_RX) FM(CANFD1_RX) FM(IERX_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP5_11_8 FM(EX_WAIT0_A) FM(QCLK) F_(0, 0) F_(0, 0) FM(VI4_CLK) F_(0, 0) FM(DU_DOTCLKOUT0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP5_15_12 FM(D0) FM(MSIOF2_SS1_B)FM(MSIOF3_SCK_A) F_(0, 0) FM(VI4_DATA16) FM(VI5_DATA0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP5_19_16 FM(D1) FM(MSIOF2_SS2_B)FM(MSIOF3_SYNC_A) F_(0, 0) FM(VI4_DATA17) FM(VI5_DATA1) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP5_23_20 FM(D2) F_(0, 0) FM(MSIOF3_RXD_A) F_(0, 0) FM(VI4_DATA18) FM(VI5_DATA2) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP5_27_24 FM(D3) F_(0, 0) FM(MSIOF3_TXD_A) F_(0, 0) FM(VI4_DATA19) FM(VI5_DATA3) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP5_31_28 FM(D4) FM(MSIOF2_SCK_B)F_(0, 0) F_(0, 0) FM(VI4_DATA20) FM(VI5_DATA4) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP6_3_0 FM(D5) FM(MSIOF2_SYNC_B)F_(0, 0) F_(0, 0) FM(VI4_DATA21) FM(VI5_DATA5) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP6_7_4 FM(D6) FM(MSIOF2_RXD_B)F_(0, 0) F_(0, 0) FM(VI4_DATA22) FM(VI5_DATA6) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP6_11_8 FM(D7) FM(MSIOF2_TXD_B)F_(0, 0) F_(0, 0) FM(VI4_DATA23) FM(VI5_DATA7) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP6_15_12 FM(D8) FM(LCDOUT0) FM(MSIOF2_SCK_D) FM(SCK4_C) FM(VI4_DATA0_A) F_(0, 0) FM(DU_DR0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP6_19_16 FM(D9) FM(LCDOUT1) FM(MSIOF2_SYNC_D) F_(0, 0) FM(VI4_DATA1_A) F_(0, 0) FM(DU_DR1) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP6_23_20 FM(D10) FM(LCDOUT2) FM(MSIOF2_RXD_D) FM(HRX3_B) FM(VI4_DATA2_A) FM(CTS4_N_C) FM(DU_DR2) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP6_27_24 FM(D11) FM(LCDOUT3) FM(MSIOF2_TXD_D) FM(HTX3_B) FM(VI4_DATA3_A) FM(RTS4_N_TANS_C)FM(DU_DR3) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP6_31_28 FM(D12) FM(LCDOUT4) FM(MSIOF2_SS1_D) FM(RX4_C) FM(VI4_DATA4_A) F_(0, 0) FM(DU_DR4) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP7_3_0 FM(D13) FM(LCDOUT5) FM(MSIOF2_SS2_D) FM(TX4_C) FM(VI4_DATA5_A) F_(0, 0) FM(DU_DR5) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP7_7_4 FM(D14) FM(LCDOUT6) FM(MSIOF3_SS1_A) FM(HRX3_C) FM(VI4_DATA6_A) F_(0, 0) FM(DU_DR6) FM(SCL6_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP7_11_8 FM(D15) FM(LCDOUT7) FM(MSIOF3_SS2_A) FM(HTX3_C) FM(VI4_DATA7_A) F_(0, 0) FM(DU_DR7) FM(SDA6_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP7_15_12 FM(FSCLKST) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP7_19_16 FM(SD0_CLK) F_(0, 0) FM(MSIOF1_SCK_E) F_(0, 0) F_(0, 0) F_(0, 0) FM(STP_OPWM_0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+
+/* IPSRx */ /* 0 */ /* 1 */ /* 2 */ /* 3 */ /* 4 */ /* 5 */ /* 6 */ /* 7 */ /* 8 */ /* 9 */ /* A */ /* B */ /* C - F */
+#define IP7_23_20 FM(SD0_CMD) F_(0, 0) FM(MSIOF1_SYNC_E) F_(0, 0) F_(0, 0) F_(0, 0) FM(STP_IVCXO27_0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP7_27_24 FM(SD0_DAT0) F_(0, 0) FM(MSIOF1_RXD_E) F_(0, 0) F_(0, 0) FM(TS_SCK0_B) FM(STP_ISCLK_0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP7_31_28 FM(SD0_DAT1) F_(0, 0) FM(MSIOF1_TXD_E) F_(0, 0) F_(0, 0) FM(TS_SPSYNC0_B)FM(STP_ISSYNC_0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP8_3_0 FM(SD0_DAT2) F_(0, 0) FM(MSIOF1_SS1_E) F_(0, 0) F_(0, 0) FM(TS_SDAT0_B) FM(STP_ISD_0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP8_7_4 FM(SD0_DAT3) F_(0, 0) FM(MSIOF1_SS2_E) F_(0, 0) F_(0, 0) FM(TS_SDEN0_B) FM(STP_ISEN_0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP8_11_8 FM(SD1_CLK) F_(0, 0) FM(MSIOF1_SCK_G) F_(0, 0) F_(0, 0) FM(SIM0_CLK_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP8_15_12 FM(SD1_CMD) F_(0, 0) FM(MSIOF1_SYNC_G) F_(0, 0) F_(0, 0) FM(SIM0_D_A) FM(STP_IVCXO27_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP8_19_16 FM(SD1_DAT0) FM(SD2_DAT4) FM(MSIOF1_RXD_G) F_(0, 0) F_(0, 0) FM(TS_SCK1_B) FM(STP_ISCLK_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP8_23_20 FM(SD1_DAT1) FM(SD2_DAT5) FM(MSIOF1_TXD_G) F_(0, 0) F_(0, 0) FM(TS_SPSYNC1_B)FM(STP_ISSYNC_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP8_27_24 FM(SD1_DAT2) FM(SD2_DAT6) FM(MSIOF1_SS1_G) F_(0, 0) F_(0, 0) FM(TS_SDAT1_B) FM(STP_ISD_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP8_31_28 FM(SD1_DAT3) FM(SD2_DAT7) FM(MSIOF1_SS2_G) F_(0, 0) F_(0, 0) FM(TS_SDEN1_B) FM(STP_ISEN_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_3_0 FM(SD2_CLK) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_7_4 FM(SD2_DAT0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_11_8 FM(SD2_DAT1) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_15_12 FM(SD2_DAT2) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_19_16 FM(SD2_DAT3) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_23_20 FM(SD2_DS) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(SATA_DEVSLP_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_27_24 FM(SD3_DAT4) FM(SD2_CD_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_31_28 FM(SD3_DAT5) FM(SD2_WP_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_3_0 FM(SD3_DAT6) FM(SD3_CD) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_7_4 FM(SD3_DAT7) FM(SD3_WP) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_11_8 FM(SD0_CD) F_(0, 0) F_(0, 0) F_(0, 0) FM(SCL2_B) FM(SIM0_RST_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_15_12 FM(SD0_WP) F_(0, 0) F_(0, 0) F_(0, 0) FM(SDA2_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_19_16 FM(SD1_CD) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(SIM0_CLK_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_23_20 FM(SD1_WP) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(SIM0_D_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_27_24 FM(SCK0) FM(HSCK1_B) FM(MSIOF1_SS2_B) FM(AUDIO_CLKC_B) FM(SDA2_A) FM(SIM0_RST_B) FM(STP_OPWM_0_C) FM(RIF0_CLK_B) F_(0, 0) FM(ADICHS2) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_31_28 FM(RX0) FM(HRX1_B) F_(0, 0) F_(0, 0) F_(0, 0) FM(TS_SCK0_C) FM(STP_ISCLK_0_C) FM(RIF0_D0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_3_0 FM(TX0) FM(HTX1_B) F_(0, 0) F_(0, 0) F_(0, 0) FM(TS_SPSYNC0_C)FM(STP_ISSYNC_0_C) FM(RIF0_D1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_7_4 FM(CTS0_N) FM(HCTS1_N_B) FM(MSIOF1_SYNC_B) F_(0, 0) F_(0, 0) FM(TS_SPSYNC1_C)FM(STP_ISSYNC_1_C) FM(RIF1_SYNC_B) FM(AUDIO_CLKOUT_C) FM(ADICS_SAMP) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_11_8 FM(RTS0_N_TANS) FM(HRTS1_N_B) FM(MSIOF1_SS1_B) FM(AUDIO_CLKA_B) FM(SCL2_A) F_(0, 0) FM(STP_IVCXO27_1_C) FM(RIF0_SYNC_B) F_(0, 0) FM(ADICHS1) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_15_12 FM(RX1_A) FM(HRX1_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(TS_SDAT0_C) FM(STP_ISD_0_C) FM(RIF1_CLK_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_19_16 FM(TX1_A) FM(HTX1_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(TS_SDEN0_C) FM(STP_ISEN_0_C) FM(RIF1_D0_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_23_20 FM(CTS1_N) FM(HCTS1_N_A) FM(MSIOF1_RXD_B) F_(0, 0) F_(0, 0) FM(TS_SDEN1_C) FM(STP_ISEN_1_C) FM(RIF1_D0_B) F_(0, 0) FM(ADIDATA) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_27_24 FM(RTS1_N_TANS) FM(HRTS1_N_A) FM(MSIOF1_TXD_B) F_(0, 0) F_(0, 0) FM(TS_SDAT1_C) FM(STP_ISD_1_C) FM(RIF1_D1_B) F_(0, 0) FM(ADICHS0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_31_28 FM(SCK2) FM(SCIF_CLK_B) FM(MSIOF1_SCK_B) F_(0, 0) F_(0, 0) FM(TS_SCK1_C) FM(STP_ISCLK_1_C) FM(RIF1_CLK_B) F_(0, 0) FM(ADICLK) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_3_0 FM(TX2_A) F_(0, 0) F_(0, 0) FM(SD2_CD_B) FM(SCL1_A) F_(0, 0) FM(FMCLK_A) FM(RIF1_D1_C) F_(0, 0) FM(FSO_CFE_0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_7_4 FM(RX2_A) F_(0, 0) F_(0, 0) FM(SD2_WP_B) FM(SDA1_A) F_(0, 0) FM(FMIN_A) FM(RIF1_SYNC_C) F_(0, 0) FM(FSO_CFE_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_11_8 FM(HSCK0) F_(0, 0) FM(MSIOF1_SCK_D) FM(AUDIO_CLKB_A) FM(SSI_SDATA1_B)FM(TS_SCK0_D) FM(STP_ISCLK_0_D) FM(RIF0_CLK_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_15_12 FM(HRX0) F_(0, 0) FM(MSIOF1_RXD_D) F_(0, 0) FM(SSI_SDATA2_B)FM(TS_SDEN0_D) FM(STP_ISEN_0_D) FM(RIF0_D0_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_19_16 FM(HTX0) F_(0, 0) FM(MSIOF1_TXD_D) F_(0, 0) FM(SSI_SDATA9_B)FM(TS_SDAT0_D) FM(STP_ISD_0_D) FM(RIF0_D1_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_23_20 FM(HCTS0_N) FM(RX2_B) FM(MSIOF1_SYNC_D) F_(0, 0) FM(SSI_SCK9_A) FM(TS_SPSYNC0_D)FM(STP_ISSYNC_0_D) FM(RIF0_SYNC_C) FM(AUDIO_CLKOUT1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_27_24 FM(HRTS0_N) FM(TX2_B) FM(MSIOF1_SS1_D) F_(0, 0) FM(SSI_WS9_A) F_(0, 0) FM(STP_IVCXO27_0_D) FM(BPFCLK_A) FM(AUDIO_CLKOUT2_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+
+/* IPSRx */ /* 0 */ /* 1 */ /* 2 */ /* 3 */ /* 4 */ /* 5 */ /* 6 */ /* 7 */ /* 8 */ /* 9 */ /* A */ /* B */ /* C - F */
+#define IP12_31_28 FM(MSIOF0_SYNC) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_3_0 FM(MSIOF0_SS1) FM(RX5) F_(0, 0) FM(AUDIO_CLKA_C) FM(SSI_SCK2_A) F_(0, 0) FM(STP_IVCXO27_0_C) F_(0, 0) FM(AUDIO_CLKOUT3_A) F_(0, 0) FM(TCLK1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_7_4 FM(MSIOF0_SS2) FM(TX5) FM(MSIOF1_SS2_D) FM(AUDIO_CLKC_A) FM(SSI_WS2_A) F_(0, 0) FM(STP_OPWM_0_D) F_(0, 0) FM(AUDIO_CLKOUT_D) F_(0, 0) FM(SPEEDIN_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_11_8 FM(MLB_CLK) F_(0, 0) FM(MSIOF1_SCK_F) F_(0, 0) FM(SCL1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_15_12 FM(MLB_SIG) FM(RX1_B) FM(MSIOF1_SYNC_F) F_(0, 0) FM(SDA1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_19_16 FM(MLB_DAT) FM(TX1_B) FM(MSIOF1_RXD_F) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_23_20 FM(SSI_SCK01239) F_(0, 0) FM(MSIOF1_TXD_F) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_27_24 FM(SSI_WS01239) F_(0, 0) FM(MSIOF1_SS1_F) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_31_28 FM(SSI_SDATA0) F_(0, 0) FM(MSIOF1_SS2_F) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_3_0 FM(SSI_SDATA1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_7_4 FM(SSI_SDATA2_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(SSI_SCK1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_11_8 FM(SSI_SCK34) F_(0, 0) FM(MSIOF1_SS1_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(STP_OPWM_0_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_15_12 FM(SSI_WS34) FM(HCTS2_N_A) FM(MSIOF1_SS2_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(STP_IVCXO27_0_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_19_16 FM(SSI_SDATA3) FM(HRTS2_N_A) FM(MSIOF1_TXD_A) F_(0, 0) F_(0, 0) FM(TS_SCK0_A) FM(STP_ISCLK_0_A) FM(RIF0_D1_A) FM(RIF2_D0_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_23_20 FM(SSI_SCK4) FM(HRX2_A) FM(MSIOF1_SCK_A) F_(0, 0) F_(0, 0) FM(TS_SDAT0_A) FM(STP_ISD_0_A) FM(RIF0_CLK_A) FM(RIF2_CLK_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_27_24 FM(SSI_WS4) FM(HTX2_A) FM(MSIOF1_SYNC_A) F_(0, 0) F_(0, 0) FM(TS_SDEN0_A) FM(STP_ISEN_0_A) FM(RIF0_SYNC_A) FM(RIF2_SYNC_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_31_28 FM(SSI_SDATA4) FM(HSCK2_A) FM(MSIOF1_RXD_A) F_(0, 0) F_(0, 0) FM(TS_SPSYNC0_A)FM(STP_ISSYNC_0_A) FM(RIF0_D0_A) FM(RIF2_D1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_3_0 FM(SSI_SCK6) FM(USB2_PWEN) F_(0, 0) FM(SIM0_RST_D) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_7_4 FM(SSI_WS6) FM(USB2_OVC) F_(0, 0) FM(SIM0_D_D) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_11_8 FM(SSI_SDATA6) F_(0, 0) F_(0, 0) FM(SIM0_CLK_D) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(SATA_DEVSLP_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_15_12 FM(SSI_SCK78) FM(HRX2_B) FM(MSIOF1_SCK_C) F_(0, 0) F_(0, 0) FM(TS_SCK1_A) FM(STP_ISCLK_1_A) FM(RIF1_CLK_A) FM(RIF3_CLK_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_19_16 FM(SSI_WS78) FM(HTX2_B) FM(MSIOF1_SYNC_C) F_(0, 0) F_(0, 0) FM(TS_SDAT1_A) FM(STP_ISD_1_A) FM(RIF1_SYNC_A) FM(RIF3_SYNC_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_23_20 FM(SSI_SDATA7) FM(HCTS2_N_B) FM(MSIOF1_RXD_C) F_(0, 0) F_(0, 0) FM(TS_SDEN1_A) FM(STP_ISEN_1_A) FM(RIF1_D0_A) FM(RIF3_D0_A) F_(0, 0) FM(TCLK2_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_27_24 FM(SSI_SDATA8) FM(HRTS2_N_B) FM(MSIOF1_TXD_C) F_(0, 0) F_(0, 0) FM(TS_SPSYNC1_A)FM(STP_ISSYNC_1_A) FM(RIF1_D1_A) FM(RIF3_D1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_31_28 FM(SSI_SDATA9_A) FM(HSCK2_B) FM(MSIOF1_SS1_C) FM(HSCK1_A) FM(SSI_WS1_B) FM(SCK1) FM(STP_IVCXO27_1_A) FM(SCK5) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_3_0 FM(AUDIO_CLKA_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(CC5_OSCOUT) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_7_4 FM(AUDIO_CLKB_B) FM(SCIF_CLK_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(STP_IVCXO27_1_D) FM(REMOCON_A) F_(0, 0) F_(0, 0) FM(TCLK1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_11_8 FM(USB0_PWEN) F_(0, 0) F_(0, 0) FM(SIM0_RST_C) F_(0, 0) FM(TS_SCK1_D) FM(STP_ISCLK_1_D) FM(BPFCLK_B) FM(RIF3_CLK_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_15_12 FM(USB0_OVC) F_(0, 0) F_(0, 0) FM(SIM0_D_C) F_(0, 0) FM(TS_SDAT1_D) FM(STP_ISD_1_D) F_(0, 0) FM(RIF3_SYNC_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_19_16 FM(USB1_PWEN) F_(0, 0) F_(0, 0) FM(SIM0_CLK_C) FM(SSI_SCK1_A) FM(TS_SCK0_E) FM(STP_ISCLK_0_E) FM(FMCLK_B) FM(RIF2_CLK_B) F_(0, 0) FM(SPEEDIN_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_23_20 FM(USB1_OVC) F_(0, 0) FM(MSIOF1_SS2_C) F_(0, 0) FM(SSI_WS1_A) FM(TS_SDAT0_E) FM(STP_ISD_0_E) FM(FMIN_B) FM(RIF2_SYNC_B) F_(0, 0) FM(REMOCON_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_27_24 FM(USB30_PWEN) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT_B) FM(SSI_SCK2_B) FM(TS_SDEN1_D) FM(STP_ISEN_1_D) FM(STP_OPWM_0_E)FM(RIF3_D0_B) F_(0, 0) FM(TCLK2_B) FM(TPU0TO0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_31_28 FM(USB30_OVC) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT1_B) FM(SSI_WS2_B) FM(TS_SPSYNC1_D)FM(STP_ISSYNC_1_D) FM(STP_IVCXO27_0_E)FM(RIF3_D1_B) F_(0, 0) FM(FSO_TOE_B) FM(TPU0TO1) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP17_3_0 FM(USB31_PWEN) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT2_B) FM(SSI_SCK9_B) FM(TS_SDEN0_E) FM(STP_ISEN_0_E) F_(0, 0) FM(RIF2_D0_B) F_(0, 0) F_(0, 0) FM(TPU0TO2) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP17_7_4 FM(USB31_OVC) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT3_B) FM(SSI_WS9_B) FM(TS_SPSYNC0_E)FM(STP_ISSYNC_0_E) F_(0, 0) FM(RIF2_D1_B) F_(0, 0) F_(0, 0) FM(TPU0TO3) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+
+#define PINMUX_GPSR \
+\
+ GPSR6_31 \
+ GPSR6_30 \
+ GPSR6_29 \
+ GPSR6_28 \
+ GPSR1_27 GPSR6_27 \
+ GPSR1_26 GPSR6_26 \
+ GPSR1_25 GPSR5_25 GPSR6_25 \
+ GPSR1_24 GPSR5_24 GPSR6_24 \
+ GPSR1_23 GPSR5_23 GPSR6_23 \
+ GPSR1_22 GPSR5_22 GPSR6_22 \
+ GPSR1_21 GPSR5_21 GPSR6_21 \
+ GPSR1_20 GPSR5_20 GPSR6_20 \
+ GPSR1_19 GPSR5_19 GPSR6_19 \
+ GPSR1_18 GPSR5_18 GPSR6_18 \
+ GPSR1_17 GPSR4_17 GPSR5_17 GPSR6_17 \
+ GPSR1_16 GPSR4_16 GPSR5_16 GPSR6_16 \
+GPSR0_15 GPSR1_15 GPSR3_15 GPSR4_15 GPSR5_15 GPSR6_15 \
+GPSR0_14 GPSR1_14 GPSR2_14 GPSR3_14 GPSR4_14 GPSR5_14 GPSR6_14 \
+GPSR0_13 GPSR1_13 GPSR2_13 GPSR3_13 GPSR4_13 GPSR5_13 GPSR6_13 \
+GPSR0_12 GPSR1_12 GPSR2_12 GPSR3_12 GPSR4_12 GPSR5_12 GPSR6_12 \
+GPSR0_11 GPSR1_11 GPSR2_11 GPSR3_11 GPSR4_11 GPSR5_11 GPSR6_11 \
+GPSR0_10 GPSR1_10 GPSR2_10 GPSR3_10 GPSR4_10 GPSR5_10 GPSR6_10 \
+GPSR0_9 GPSR1_9 GPSR2_9 GPSR3_9 GPSR4_9 GPSR5_9 GPSR6_9 \
+GPSR0_8 GPSR1_8 GPSR2_8 GPSR3_8 GPSR4_8 GPSR5_8 GPSR6_8 \
+GPSR0_7 GPSR1_7 GPSR2_7 GPSR3_7 GPSR4_7 GPSR5_7 GPSR6_7 \
+GPSR0_6 GPSR1_6 GPSR2_6 GPSR3_6 GPSR4_6 GPSR5_6 GPSR6_6 \
+GPSR0_5 GPSR1_5 GPSR2_5 GPSR3_5 GPSR4_5 GPSR5_5 GPSR6_5 \
+GPSR0_4 GPSR1_4 GPSR2_4 GPSR3_4 GPSR4_4 GPSR5_4 GPSR6_4 \
+GPSR0_3 GPSR1_3 GPSR2_3 GPSR3_3 GPSR4_3 GPSR5_3 GPSR6_3 GPSR7_3 \
+GPSR0_2 GPSR1_2 GPSR2_2 GPSR3_2 GPSR4_2 GPSR5_2 GPSR6_2 GPSR7_2 \
+GPSR0_1 GPSR1_1 GPSR2_1 GPSR3_1 GPSR4_1 GPSR5_1 GPSR6_1 GPSR7_1 \
+GPSR0_0 GPSR1_0 GPSR2_0 GPSR3_0 GPSR4_0 GPSR5_0 GPSR6_0 GPSR7_0
+
+#define PINMUX_IPSR \
+\
+FM(IP0_3_0) IP0_3_0 FM(IP1_3_0) IP1_3_0 FM(IP2_3_0) IP2_3_0 FM(IP3_3_0) IP3_3_0 \
+FM(IP0_7_4) IP0_7_4 FM(IP1_7_4) IP1_7_4 FM(IP2_7_4) IP2_7_4 FM(IP3_7_4) IP3_7_4 \
+FM(IP0_11_8) IP0_11_8 FM(IP1_11_8) IP1_11_8 FM(IP2_11_8) IP2_11_8 FM(IP3_11_8) IP3_11_8 \
+FM(IP0_15_12) IP0_15_12 FM(IP1_15_12) IP1_15_12 FM(IP2_15_12) IP2_15_12 FM(IP3_15_12) IP3_15_12 \
+FM(IP0_19_16) IP0_19_16 FM(IP1_19_16) IP1_19_16 FM(IP2_19_16) IP2_19_16 FM(IP3_19_16) IP3_19_16 \
+FM(IP0_23_20) IP0_23_20 FM(IP1_23_20) IP1_23_20 FM(IP2_23_20) IP2_23_20 FM(IP3_23_20) IP3_23_20 \
+FM(IP0_27_24) IP0_27_24 FM(IP1_27_24) IP1_27_24 FM(IP2_27_24) IP2_27_24 FM(IP3_27_24) IP3_27_24 \
+FM(IP0_31_28) IP0_31_28 FM(IP1_31_28) IP1_31_28 FM(IP2_31_28) IP2_31_28 FM(IP3_31_28) IP3_31_28 \
+\
+FM(IP4_3_0) IP4_3_0 FM(IP5_3_0) IP5_3_0 FM(IP6_3_0) IP6_3_0 FM(IP7_3_0) IP7_3_0 \
+FM(IP4_7_4) IP4_7_4 FM(IP5_7_4) IP5_7_4 FM(IP6_7_4) IP6_7_4 FM(IP7_7_4) IP7_7_4 \
+FM(IP4_11_8) IP4_11_8 FM(IP5_11_8) IP5_11_8 FM(IP6_11_8) IP6_11_8 FM(IP7_11_8) IP7_11_8 \
+FM(IP4_15_12) IP4_15_12 FM(IP5_15_12) IP5_15_12 FM(IP6_15_12) IP6_15_12 FM(IP7_15_12) IP7_15_12 \
+FM(IP4_19_16) IP4_19_16 FM(IP5_19_16) IP5_19_16 FM(IP6_19_16) IP6_19_16 FM(IP7_19_16) IP7_19_16 \
+FM(IP4_23_20) IP4_23_20 FM(IP5_23_20) IP5_23_20 FM(IP6_23_20) IP6_23_20 FM(IP7_23_20) IP7_23_20 \
+FM(IP4_27_24) IP4_27_24 FM(IP5_27_24) IP5_27_24 FM(IP6_27_24) IP6_27_24 FM(IP7_27_24) IP7_27_24 \
+FM(IP4_31_28) IP4_31_28 FM(IP5_31_28) IP5_31_28 FM(IP6_31_28) IP6_31_28 FM(IP7_31_28) IP7_31_28 \
+\
+FM(IP8_3_0) IP8_3_0 FM(IP9_3_0) IP9_3_0 FM(IP10_3_0) IP10_3_0 FM(IP11_3_0) IP11_3_0 \
+FM(IP8_7_4) IP8_7_4 FM(IP9_7_4) IP9_7_4 FM(IP10_7_4) IP10_7_4 FM(IP11_7_4) IP11_7_4 \
+FM(IP8_11_8) IP8_11_8 FM(IP9_11_8) IP9_11_8 FM(IP10_11_8) IP10_11_8 FM(IP11_11_8) IP11_11_8 \
+FM(IP8_15_12) IP8_15_12 FM(IP9_15_12) IP9_15_12 FM(IP10_15_12) IP10_15_12 FM(IP11_15_12) IP11_15_12 \
+FM(IP8_19_16) IP8_19_16 FM(IP9_19_16) IP9_19_16 FM(IP10_19_16) IP10_19_16 FM(IP11_19_16) IP11_19_16 \
+FM(IP8_23_20) IP8_23_20 FM(IP9_23_20) IP9_23_20 FM(IP10_23_20) IP10_23_20 FM(IP11_23_20) IP11_23_20 \
+FM(IP8_27_24) IP8_27_24 FM(IP9_27_24) IP9_27_24 FM(IP10_27_24) IP10_27_24 FM(IP11_27_24) IP11_27_24 \
+FM(IP8_31_28) IP8_31_28 FM(IP9_31_28) IP9_31_28 FM(IP10_31_28) IP10_31_28 FM(IP11_31_28) IP11_31_28 \
+\
+FM(IP12_3_0) IP12_3_0 FM(IP13_3_0) IP13_3_0 FM(IP14_3_0) IP14_3_0 FM(IP15_3_0) IP15_3_0 \
+FM(IP12_7_4) IP12_7_4 FM(IP13_7_4) IP13_7_4 FM(IP14_7_4) IP14_7_4 FM(IP15_7_4) IP15_7_4 \
+FM(IP12_11_8) IP12_11_8 FM(IP13_11_8) IP13_11_8 FM(IP14_11_8) IP14_11_8 FM(IP15_11_8) IP15_11_8 \
+FM(IP12_15_12) IP12_15_12 FM(IP13_15_12) IP13_15_12 FM(IP14_15_12) IP14_15_12 FM(IP15_15_12) IP15_15_12 \
+FM(IP12_19_16) IP12_19_16 FM(IP13_19_16) IP13_19_16 FM(IP14_19_16) IP14_19_16 FM(IP15_19_16) IP15_19_16 \
+FM(IP12_23_20) IP12_23_20 FM(IP13_23_20) IP13_23_20 FM(IP14_23_20) IP14_23_20 FM(IP15_23_20) IP15_23_20 \
+FM(IP12_27_24) IP12_27_24 FM(IP13_27_24) IP13_27_24 FM(IP14_27_24) IP14_27_24 FM(IP15_27_24) IP15_27_24 \
+FM(IP12_31_28) IP12_31_28 FM(IP13_31_28) IP13_31_28 FM(IP14_31_28) IP14_31_28 FM(IP15_31_28) IP15_31_28 \
+\
+FM(IP16_3_0) IP16_3_0 FM(IP17_3_0) IP17_3_0 \
+FM(IP16_7_4) IP16_7_4 FM(IP17_7_4) IP17_7_4 \
+FM(IP16_11_8) IP16_11_8 \
+FM(IP16_15_12) IP16_15_12 \
+FM(IP16_19_16) IP16_19_16 \
+FM(IP16_23_20) IP16_23_20 \
+FM(IP16_27_24) IP16_27_24 \
+FM(IP16_31_28) IP16_31_28
+
+/* MOD_SEL0 */ /* 0 */ /* 1 */ /* 2 */ /* 3 */ /* 4 */ /* 5 */ /* 6 */ /* 7 */
+#define MOD_SEL0_30_29 FM(SEL_MSIOF3_0) FM(SEL_MSIOF3_1) FM(SEL_MSIOF3_2) FM(SEL_MSIOF3_3)
+#define MOD_SEL0_28_27 FM(SEL_MSIOF2_0) FM(SEL_MSIOF2_1) FM(SEL_MSIOF2_2) FM(SEL_MSIOF2_3)
+#define MOD_SEL0_26_25_24 FM(SEL_MSIOF1_0) FM(SEL_MSIOF1_1) FM(SEL_MSIOF1_2) FM(SEL_MSIOF1_3) FM(SEL_MSIOF1_4) FM(SEL_MSIOF1_5) FM(SEL_MSIOF1_6) F_(0, 0)
+#define MOD_SEL0_23 FM(SEL_LBSC_0) FM(SEL_LBSC_1)
+#define MOD_SEL0_22 FM(SEL_IEBUS_0) FM(SEL_IEBUS_1)
+#define MOD_SEL0_21_20 FM(SEL_I2C6_0) FM(SEL_I2C6_1) FM(SEL_I2C6_2) F_(0, 0)
+#define MOD_SEL0_19 FM(SEL_I2C2_0) FM(SEL_I2C2_1)
+#define MOD_SEL0_18 FM(SEL_I2C1_0) FM(SEL_I2C1_1)
+#define MOD_SEL0_17 FM(SEL_HSCIF4_0) FM(SEL_HSCIF4_1)
+#define MOD_SEL0_16_15 FM(SEL_HSCIF3_0) FM(SEL_HSCIF3_1) FM(SEL_HSCIF3_2) FM(SEL_HSCIF3_3)
+#define MOD_SEL0_14 FM(SEL_HSCIF2_0) FM(SEL_HSCIF2_1)
+#define MOD_SEL0_13 FM(SEL_HSCIF1_0) FM(SEL_HSCIF1_1)
+#define MOD_SEL0_12 FM(SEL_FSO_0) FM(SEL_FSO_1)
+#define MOD_SEL0_11 FM(SEL_FM_0) FM(SEL_FM_1)
+#define MOD_SEL0_10 FM(SEL_ETHERAVB_0) FM(SEL_ETHERAVB_1)
+#define MOD_SEL0_9 FM(SEL_DRIF3_0) FM(SEL_DRIF3_1)
+#define MOD_SEL0_8 FM(SEL_DRIF2_0) FM(SEL_DRIF2_1)
+#define MOD_SEL0_7_6 FM(SEL_DRIF1_0) FM(SEL_DRIF1_1) FM(SEL_DRIF1_2) F_(0, 0)
+#define MOD_SEL0_5_4 FM(SEL_DRIF0_0) FM(SEL_DRIF0_1) FM(SEL_DRIF0_2) F_(0, 0)
+#define MOD_SEL0_3 FM(SEL_CANFD0_0) FM(SEL_CANFD0_1)
+#define MOD_SEL0_2_1 FM(SEL_ADG_0) FM(SEL_ADG_1) FM(SEL_ADG_2) FM(SEL_ADG_3)
+
+/* MOD_SEL1 */ /* 0 */ /* 1 */ /* 2 */ /* 3 */ /* 4 */ /* 5 */ /* 6 */ /* 7 */
+#define MOD_SEL1_31_30 FM(SEL_TSIF1_0) FM(SEL_TSIF1_1) FM(SEL_TSIF1_2) FM(SEL_TSIF1_3)
+#define MOD_SEL1_29_28_27 FM(SEL_TSIF0_0) FM(SEL_TSIF0_1) FM(SEL_TSIF0_2) FM(SEL_TSIF0_3) FM(SEL_TSIF0_4) F_(0, 0) F_(0, 0) F_(0, 0)
+#define MOD_SEL1_26 FM(SEL_TIMER_TMU_0) FM(SEL_TIMER_TMU_1)
+#define MOD_SEL1_25_24 FM(SEL_SSP1_1_0) FM(SEL_SSP1_1_1) FM(SEL_SSP1_1_2) FM(SEL_SSP1_1_3)
+#define MOD_SEL1_23_22_21 FM(SEL_SSP1_0_0) FM(SEL_SSP1_0_1) FM(SEL_SSP1_0_2) FM(SEL_SSP1_0_3) FM(SEL_SSP1_0_4) F_(0, 0) F_(0, 0) F_(0, 0)
+#define MOD_SEL1_20 FM(SEL_SSI_0) FM(SEL_SSI_1)
+#define MOD_SEL1_19 FM(SEL_SPEED_PULSE_0) FM(SEL_SPEED_PULSE_1)
+#define MOD_SEL1_18_17 FM(SEL_SIMCARD_0) FM(SEL_SIMCARD_1) FM(SEL_SIMCARD_2) FM(SEL_SIMCARD_3)
+#define MOD_SEL1_16 FM(SEL_SDHI2_0) FM(SEL_SDHI2_1)
+#define MOD_SEL1_15_14 FM(SEL_SCIF4_0) FM(SEL_SCIF4_1) FM(SEL_SCIF4_2) F_(0, 0)
+#define MOD_SEL1_13 FM(SEL_SCIF3_0) FM(SEL_SCIF3_1)
+#define MOD_SEL1_12 FM(SEL_SCIF2_0) FM(SEL_SCIF2_1)
+#define MOD_SEL1_11 FM(SEL_SCIF1_0) FM(SEL_SCIF1_1)
+#define MOD_SEL1_10 FM(SEL_SATA_0) FM(SEL_SATA_1)
+#define MOD_SEL1_9 FM(SEL_REMOCON_0) FM(SEL_REMOCON_1)
+#define MOD_SEL1_6 FM(SEL_RCAN0_0) FM(SEL_RCAN0_1)
+#define MOD_SEL1_5 FM(SEL_PWM6_0) FM(SEL_PWM6_1)
+#define MOD_SEL1_4 FM(SEL_PWM5_0) FM(SEL_PWM5_1)
+#define MOD_SEL1_3 FM(SEL_PWM4_0) FM(SEL_PWM4_1)
+#define MOD_SEL1_2 FM(SEL_PWM3_0) FM(SEL_PWM3_1)
+#define MOD_SEL1_1 FM(SEL_PWM2_0) FM(SEL_PWM2_1)
+#define MOD_SEL1_0 FM(SEL_PWM1_0) FM(SEL_PWM1_1)
+
+/* MOD_SEL2 */ /* 0 */ /* 1 */ /* 2 */ /* 3 */
+#define MOD_SEL2_31 FM(I2C_SEL_5_0) FM(I2C_SEL_5_1)
+#define MOD_SEL2_30 FM(I2C_SEL_3_0) FM(I2C_SEL_3_1)
+#define MOD_SEL2_29 FM(I2C_SEL_0_0) FM(I2C_SEL_0_1)
+#define MOD_SEL2_0 FM(SEL_VIN4_0) FM(SEL_VIN4_1)
+
+#define PINMUX_MOD_SELS\
+\
+ MOD_SEL1_31_30 MOD_SEL2_31 \
+MOD_SEL0_30_29 MOD_SEL2_30 \
+ MOD_SEL1_29_28_27 MOD_SEL2_29 \
+MOD_SEL0_28_27 \
+\
+MOD_SEL0_26_25_24 MOD_SEL1_26 \
+ MOD_SEL1_25_24 \
+\
+MOD_SEL0_23 MOD_SEL1_23_22_21 \
+MOD_SEL0_22 \
+MOD_SEL0_21_20 \
+ MOD_SEL1_20 \
+MOD_SEL0_19 MOD_SEL1_19 \
+MOD_SEL0_18 MOD_SEL1_18_17 \
+MOD_SEL0_17 \
+MOD_SEL0_16_15 MOD_SEL1_16 \
+ MOD_SEL1_15_14 \
+MOD_SEL0_14 \
+MOD_SEL0_13 MOD_SEL1_13 \
+MOD_SEL0_12 MOD_SEL1_12 \
+MOD_SEL0_11 MOD_SEL1_11 \
+MOD_SEL0_10 MOD_SEL1_10 \
+MOD_SEL0_9 MOD_SEL1_9 \
+MOD_SEL0_8 \
+MOD_SEL0_7_6 \
+ MOD_SEL1_6 \
+MOD_SEL0_5_4 MOD_SEL1_5 \
+ MOD_SEL1_4 \
+MOD_SEL0_3 MOD_SEL1_3 \
+MOD_SEL0_2_1 MOD_SEL1_2 \
+ MOD_SEL1_1 \
+ MOD_SEL1_0 MOD_SEL2_0
+
+/*
+ * These pins are not able to be muxed but have other properties
+ * that can be set, such as drive-strength or pull-up/pull-down enable.
+ */
+#define PINMUX_STATIC \
+ FM(QSPI0_SPCLK) FM(QSPI0_SSL) FM(QSPI0_MOSI_IO0) FM(QSPI0_MISO_IO1) \
+ FM(QSPI0_IO2) FM(QSPI0_IO3) \
+ FM(QSPI1_SPCLK) FM(QSPI1_SSL) FM(QSPI1_MOSI_IO0) FM(QSPI1_MISO_IO1) \
+ FM(QSPI1_IO2) FM(QSPI1_IO3) \
+ FM(RPC_INT) FM(RPC_WP) FM(RPC_RESET) \
+ FM(AVB_TX_CTL) FM(AVB_TXC) FM(AVB_TD0) FM(AVB_TD1) FM(AVB_TD2) FM(AVB_TD3) \
+ FM(AVB_RX_CTL) FM(AVB_RXC) FM(AVB_RD0) FM(AVB_RD1) FM(AVB_RD2) FM(AVB_RD3) \
+ FM(AVB_TXCREFCLK) FM(AVB_MDIO) \
+ FM(CLKOUT) FM(PRESETOUT) \
+ FM(DU_DOTCLKIN0) FM(DU_DOTCLKIN1) FM(DU_DOTCLKIN2) FM(DU_DOTCLKIN3) \
+ FM(TMS) FM(TDO) FM(ASEBRK) FM(MLB_REF) FM(TDI) FM(TCK) FM(TRST) FM(EXTALR)
+
+enum {
+ PINMUX_RESERVED = 0,
+
+ PINMUX_DATA_BEGIN,
+ GP_ALL(DATA),
+ PINMUX_DATA_END,
+
+#define F_(x, y)
+#define FM(x) FN_##x,
+ PINMUX_FUNCTION_BEGIN,
+ GP_ALL(FN),
+ PINMUX_GPSR
+ PINMUX_IPSR
+ PINMUX_MOD_SELS
+ PINMUX_FUNCTION_END,
+#undef F_
+#undef FM
+
+#define F_(x, y)
+#define FM(x) x##_MARK,
+ PINMUX_MARK_BEGIN,
+ PINMUX_GPSR
+ PINMUX_IPSR
+ PINMUX_MOD_SELS
+ PINMUX_STATIC
+ PINMUX_MARK_END,
+#undef F_
+#undef FM
+};
+
+static const u16 pinmux_data[] = {
+ PINMUX_DATA_GP_ALL(),
+
+ PINMUX_SINGLE(AVS1),
+ PINMUX_SINGLE(AVS2),
+ PINMUX_SINGLE(HDMI0_CEC),
+ PINMUX_SINGLE(HDMI1_CEC),
+ PINMUX_SINGLE(I2C_SEL_0_1),
+ PINMUX_SINGLE(I2C_SEL_3_1),
+ PINMUX_SINGLE(I2C_SEL_5_1),
+ PINMUX_SINGLE(MSIOF0_RXD),
+ PINMUX_SINGLE(MSIOF0_SCK),
+ PINMUX_SINGLE(MSIOF0_TXD),
+ PINMUX_SINGLE(SD2_CMD),
+ PINMUX_SINGLE(SD3_CLK),
+ PINMUX_SINGLE(SD3_CMD),
+ PINMUX_SINGLE(SD3_DAT0),
+ PINMUX_SINGLE(SD3_DAT1),
+ PINMUX_SINGLE(SD3_DAT2),
+ PINMUX_SINGLE(SD3_DAT3),
+ PINMUX_SINGLE(SD3_DS),
+ PINMUX_SINGLE(SSI_SCK5),
+ PINMUX_SINGLE(SSI_SDATA5),
+ PINMUX_SINGLE(SSI_WS5),
+
+ /* IPSR0 */
+ PINMUX_IPSR_GPSR(IP0_3_0, AVB_MDC),
+ PINMUX_IPSR_MSEL(IP0_3_0, MSIOF2_SS2_C, SEL_MSIOF2_2),
+
+ PINMUX_IPSR_GPSR(IP0_7_4, AVB_MAGIC),
+ PINMUX_IPSR_MSEL(IP0_7_4, MSIOF2_SS1_C, SEL_MSIOF2_2),
+ PINMUX_IPSR_MSEL(IP0_7_4, SCK4_A, SEL_SCIF4_0),
+
+ PINMUX_IPSR_GPSR(IP0_11_8, AVB_PHY_INT),
+ PINMUX_IPSR_MSEL(IP0_11_8, MSIOF2_SYNC_C, SEL_MSIOF2_2),
+ PINMUX_IPSR_MSEL(IP0_11_8, RX4_A, SEL_SCIF4_0),
+
+ PINMUX_IPSR_GPSR(IP0_15_12, AVB_LINK),
+ PINMUX_IPSR_MSEL(IP0_15_12, MSIOF2_SCK_C, SEL_MSIOF2_2),
+ PINMUX_IPSR_MSEL(IP0_15_12, TX4_A, SEL_SCIF4_0),
+
+ PINMUX_IPSR_MSEL(IP0_19_16, AVB_AVTP_MATCH_A, SEL_ETHERAVB_0),
+ PINMUX_IPSR_MSEL(IP0_19_16, MSIOF2_RXD_C, SEL_MSIOF2_2),
+ PINMUX_IPSR_MSEL(IP0_19_16, CTS4_N_A, SEL_SCIF4_0),
+
+ PINMUX_IPSR_MSEL(IP0_23_20, AVB_AVTP_CAPTURE_A, SEL_ETHERAVB_0),
+ PINMUX_IPSR_MSEL(IP0_23_20, MSIOF2_TXD_C, SEL_MSIOF2_2),
+ PINMUX_IPSR_MSEL(IP0_23_20, RTS4_N_TANS_A, SEL_SCIF4_0),
+
+ PINMUX_IPSR_GPSR(IP0_27_24, IRQ0),
+ PINMUX_IPSR_GPSR(IP0_27_24, QPOLB),
+ PINMUX_IPSR_GPSR(IP0_27_24, DU_CDE),
+ PINMUX_IPSR_MSEL(IP0_27_24, VI4_DATA0_B, SEL_VIN4_1),
+ PINMUX_IPSR_MSEL(IP0_27_24, CAN0_TX_B, SEL_RCAN0_1),
+ PINMUX_IPSR_MSEL(IP0_27_24, CANFD0_TX_B, SEL_CANFD0_1),
+
+ PINMUX_IPSR_GPSR(IP0_31_28, IRQ1),
+ PINMUX_IPSR_GPSR(IP0_31_28, QPOLA),
+ PINMUX_IPSR_GPSR(IP0_31_28, DU_DISP),
+ PINMUX_IPSR_MSEL(IP0_31_28, VI4_DATA1_B, SEL_VIN4_1),
+ PINMUX_IPSR_MSEL(IP0_31_28, CAN0_RX_B, SEL_RCAN0_1),
+ PINMUX_IPSR_MSEL(IP0_31_28, CANFD0_RX_B, SEL_CANFD0_1),
+
+ /* IPSR1 */
+ PINMUX_IPSR_GPSR(IP1_3_0, IRQ2),
+ PINMUX_IPSR_GPSR(IP1_3_0, QCPV_QDE),
+ PINMUX_IPSR_GPSR(IP1_3_0, DU_EXODDF_DU_ODDF_DISP_CDE),
+ PINMUX_IPSR_MSEL(IP1_3_0, VI4_DATA2_B, SEL_VIN4_1),
+ PINMUX_IPSR_MSEL(IP1_3_0, PWM3_B, SEL_PWM3_1),
+
+ PINMUX_IPSR_GPSR(IP1_7_4, IRQ3),
+ PINMUX_IPSR_GPSR(IP1_7_4, QSTVB_QVE),
+ PINMUX_IPSR_GPSR(IP1_7_4, A25),
+ PINMUX_IPSR_GPSR(IP1_7_4, DU_DOTCLKOUT1),
+ PINMUX_IPSR_MSEL(IP1_7_4, VI4_DATA3_B, SEL_VIN4_1),
+ PINMUX_IPSR_MSEL(IP1_7_4, PWM4_B, SEL_PWM4_1),
+
+ PINMUX_IPSR_GPSR(IP1_11_8, IRQ4),
+ PINMUX_IPSR_GPSR(IP1_11_8, QSTH_QHS),
+ PINMUX_IPSR_GPSR(IP1_11_8, A24),
+ PINMUX_IPSR_GPSR(IP1_11_8, DU_EXHSYNC_DU_HSYNC),
+ PINMUX_IPSR_MSEL(IP1_11_8, VI4_DATA4_B, SEL_VIN4_1),
+ PINMUX_IPSR_MSEL(IP1_11_8, PWM5_B, SEL_PWM5_1),
+
+ PINMUX_IPSR_GPSR(IP1_15_12, IRQ5),
+ PINMUX_IPSR_GPSR(IP1_15_12, QSTB_QHE),
+ PINMUX_IPSR_GPSR(IP1_15_12, A23),
+ PINMUX_IPSR_GPSR(IP1_15_12, DU_EXVSYNC_DU_VSYNC),
+ PINMUX_IPSR_MSEL(IP1_15_12, VI4_DATA5_B, SEL_VIN4_1),
+ PINMUX_IPSR_MSEL(IP1_15_12, PWM6_B, SEL_PWM6_1),
+
+ PINMUX_IPSR_GPSR(IP1_19_16, PWM0),
+ PINMUX_IPSR_GPSR(IP1_19_16, AVB_AVTP_PPS),
+ PINMUX_IPSR_GPSR(IP1_19_16, A22),
+ PINMUX_IPSR_MSEL(IP1_19_16, VI4_DATA6_B, SEL_VIN4_1),
+ PINMUX_IPSR_MSEL(IP1_19_16, IECLK_B, SEL_IEBUS_1),
+
+ PINMUX_IPSR_MSEL(IP1_23_20, PWM1_A, SEL_PWM1_0),
+ PINMUX_IPSR_GPSR(IP1_23_20, A21),
+ PINMUX_IPSR_MSEL(IP1_23_20, HRX3_D, SEL_HSCIF3_3),
+ PINMUX_IPSR_MSEL(IP1_23_20, VI4_DATA7_B, SEL_VIN4_1),
+ PINMUX_IPSR_MSEL(IP1_23_20, IERX_B, SEL_IEBUS_1),
+
+ PINMUX_IPSR_MSEL(IP1_27_24, PWM2_A, SEL_PWM2_0),
+ PINMUX_IPSR_GPSR(IP1_27_24, A20),
+ PINMUX_IPSR_MSEL(IP1_27_24, HTX3_D, SEL_HSCIF3_3),
+ PINMUX_IPSR_MSEL(IP1_27_24, IETX_B, SEL_IEBUS_1),
+
+ PINMUX_IPSR_GPSR(IP1_31_28, A0),
+ PINMUX_IPSR_GPSR(IP1_31_28, LCDOUT16),
+ PINMUX_IPSR_MSEL(IP1_31_28, MSIOF3_SYNC_B, SEL_MSIOF3_1),
+ PINMUX_IPSR_GPSR(IP1_31_28, VI4_DATA8),
+ PINMUX_IPSR_GPSR(IP1_31_28, DU_DB0),
+ PINMUX_IPSR_MSEL(IP1_31_28, PWM3_A, SEL_PWM3_0),
+
+ /* IPSR2 */
+ PINMUX_IPSR_GPSR(IP2_3_0, A1),
+ PINMUX_IPSR_GPSR(IP2_3_0, LCDOUT17),
+ PINMUX_IPSR_MSEL(IP2_3_0, MSIOF3_TXD_B, SEL_MSIOF3_1),
+ PINMUX_IPSR_GPSR(IP2_3_0, VI4_DATA9),
+ PINMUX_IPSR_GPSR(IP2_3_0, DU_DB1),
+ PINMUX_IPSR_MSEL(IP2_3_0, PWM4_A, SEL_PWM4_0),
+
+ PINMUX_IPSR_GPSR(IP2_7_4, A2),
+ PINMUX_IPSR_GPSR(IP2_7_4, LCDOUT18),
+ PINMUX_IPSR_MSEL(IP2_7_4, MSIOF3_SCK_B, SEL_MSIOF3_1),
+ PINMUX_IPSR_GPSR(IP2_7_4, VI4_DATA10),
+ PINMUX_IPSR_GPSR(IP2_7_4, DU_DB2),
+ PINMUX_IPSR_MSEL(IP2_7_4, PWM5_A, SEL_PWM5_0),
+
+ PINMUX_IPSR_GPSR(IP2_11_8, A3),
+ PINMUX_IPSR_GPSR(IP2_11_8, LCDOUT19),
+ PINMUX_IPSR_MSEL(IP2_11_8, MSIOF3_RXD_B, SEL_MSIOF3_1),
+ PINMUX_IPSR_GPSR(IP2_11_8, VI4_DATA11),
+ PINMUX_IPSR_GPSR(IP2_11_8, DU_DB3),
+ PINMUX_IPSR_MSEL(IP2_11_8, PWM6_A, SEL_PWM6_0),
+
+ PINMUX_IPSR_GPSR(IP2_15_12, A4),
+ PINMUX_IPSR_GPSR(IP2_15_12, LCDOUT20),
+ PINMUX_IPSR_MSEL(IP2_15_12, MSIOF3_SS1_B, SEL_MSIOF3_1),
+ PINMUX_IPSR_GPSR(IP2_15_12, VI4_DATA12),
+ PINMUX_IPSR_GPSR(IP2_15_12, VI5_DATA12),
+ PINMUX_IPSR_GPSR(IP2_15_12, DU_DB4),
+
+ PINMUX_IPSR_GPSR(IP2_19_16, A5),
+ PINMUX_IPSR_GPSR(IP2_19_16, LCDOUT21),
+ PINMUX_IPSR_MSEL(IP2_19_16, MSIOF3_SS2_B, SEL_MSIOF3_1),
+ PINMUX_IPSR_MSEL(IP2_19_16, SCK4_B, SEL_SCIF4_1),
+ PINMUX_IPSR_GPSR(IP2_19_16, VI4_DATA13),
+ PINMUX_IPSR_GPSR(IP2_19_16, VI5_DATA13),
+ PINMUX_IPSR_GPSR(IP2_19_16, DU_DB5),
+
+ PINMUX_IPSR_GPSR(IP2_23_20, A6),
+ PINMUX_IPSR_GPSR(IP2_23_20, LCDOUT22),
+ PINMUX_IPSR_MSEL(IP2_23_20, MSIOF2_SS1_A, SEL_MSIOF2_0),
+ PINMUX_IPSR_MSEL(IP2_23_20, RX4_B, SEL_SCIF4_1),
+ PINMUX_IPSR_GPSR(IP2_23_20, VI4_DATA14),
+ PINMUX_IPSR_GPSR(IP2_23_20, VI5_DATA14),
+ PINMUX_IPSR_GPSR(IP2_23_20, DU_DB6),
+
+ PINMUX_IPSR_GPSR(IP2_27_24, A7),
+ PINMUX_IPSR_GPSR(IP2_27_24, LCDOUT23),
+ PINMUX_IPSR_MSEL(IP2_27_24, MSIOF2_SS2_A, SEL_MSIOF2_0),
+ PINMUX_IPSR_MSEL(IP2_27_24, TX4_B, SEL_SCIF4_1),
+ PINMUX_IPSR_GPSR(IP2_27_24, VI4_DATA15),
+ PINMUX_IPSR_GPSR(IP2_27_24, VI5_DATA15),
+ PINMUX_IPSR_GPSR(IP2_27_24, DU_DB7),
+
+ PINMUX_IPSR_GPSR(IP2_31_28, A8),
+ PINMUX_IPSR_MSEL(IP2_31_28, RX3_B, SEL_SCIF3_1),
+ PINMUX_IPSR_MSEL(IP2_31_28, MSIOF2_SYNC_A, SEL_MSIOF2_0),
+ PINMUX_IPSR_MSEL(IP2_31_28, HRX4_B, SEL_HSCIF4_1),
+ PINMUX_IPSR_MSEL(IP2_31_28, SDA6_A, SEL_I2C6_0),
+ PINMUX_IPSR_MSEL(IP2_31_28, AVB_AVTP_MATCH_B, SEL_ETHERAVB_1),
+ PINMUX_IPSR_MSEL(IP2_31_28, PWM1_B, SEL_PWM1_1),
+
+ /* IPSR3 */
+ PINMUX_IPSR_GPSR(IP3_3_0, A9),
+ PINMUX_IPSR_MSEL(IP3_3_0, MSIOF2_SCK_A, SEL_MSIOF2_0),
+ PINMUX_IPSR_MSEL(IP3_3_0, CTS4_N_B, SEL_SCIF4_1),
+ PINMUX_IPSR_GPSR(IP3_3_0, VI5_VSYNC_N),
+
+ PINMUX_IPSR_GPSR(IP3_7_4, A10),
+ PINMUX_IPSR_MSEL(IP3_7_4, MSIOF2_RXD_A, SEL_MSIOF2_0),
+ PINMUX_IPSR_MSEL(IP3_7_4, RTS4_N_TANS_B, SEL_SCIF4_1),
+ PINMUX_IPSR_GPSR(IP3_7_4, VI5_HSYNC_N),
+
+ PINMUX_IPSR_GPSR(IP3_11_8, A11),
+ PINMUX_IPSR_MSEL(IP3_11_8, TX3_B, SEL_SCIF3_1),
+ PINMUX_IPSR_MSEL(IP3_11_8, MSIOF2_TXD_A, SEL_MSIOF2_0),
+ PINMUX_IPSR_MSEL(IP3_11_8, HTX4_B, SEL_HSCIF4_1),
+ PINMUX_IPSR_GPSR(IP3_11_8, HSCK4),
+ PINMUX_IPSR_GPSR(IP3_11_8, VI5_FIELD),
+ PINMUX_IPSR_MSEL(IP3_11_8, SCL6_A, SEL_I2C6_0),
+ PINMUX_IPSR_MSEL(IP3_11_8, AVB_AVTP_CAPTURE_B, SEL_ETHERAVB_1),
+ PINMUX_IPSR_MSEL(IP3_11_8, PWM2_B, SEL_PWM2_1),
+
+ PINMUX_IPSR_GPSR(IP3_15_12, A12),
+ PINMUX_IPSR_GPSR(IP3_15_12, LCDOUT12),
+ PINMUX_IPSR_MSEL(IP3_15_12, MSIOF3_SCK_C, SEL_MSIOF3_2),
+ PINMUX_IPSR_MSEL(IP3_15_12, HRX4_A, SEL_HSCIF4_0),
+ PINMUX_IPSR_GPSR(IP3_15_12, VI5_DATA8),
+ PINMUX_IPSR_GPSR(IP3_15_12, DU_DG4),
+
+ PINMUX_IPSR_GPSR(IP3_19_16, A13),
+ PINMUX_IPSR_GPSR(IP3_19_16, LCDOUT13),
+ PINMUX_IPSR_MSEL(IP3_19_16, MSIOF3_SYNC_C, SEL_MSIOF3_2),
+ PINMUX_IPSR_MSEL(IP3_19_16, HTX4_A, SEL_HSCIF4_0),
+ PINMUX_IPSR_GPSR(IP3_19_16, VI5_DATA9),
+ PINMUX_IPSR_GPSR(IP3_19_16, DU_DG5),
+
+ PINMUX_IPSR_GPSR(IP3_23_20, A14),
+ PINMUX_IPSR_GPSR(IP3_23_20, LCDOUT14),
+ PINMUX_IPSR_MSEL(IP3_23_20, MSIOF3_RXD_C, SEL_MSIOF3_2),
+ PINMUX_IPSR_GPSR(IP3_23_20, HCTS4_N),
+ PINMUX_IPSR_GPSR(IP3_23_20, VI5_DATA10),
+ PINMUX_IPSR_GPSR(IP3_23_20, DU_DG6),
+
+ PINMUX_IPSR_GPSR(IP3_27_24, A15),
+ PINMUX_IPSR_GPSR(IP3_27_24, LCDOUT15),
+ PINMUX_IPSR_MSEL(IP3_27_24, MSIOF3_TXD_C, SEL_MSIOF3_2),
+ PINMUX_IPSR_GPSR(IP3_27_24, HRTS4_N),
+ PINMUX_IPSR_GPSR(IP3_27_24, VI5_DATA11),
+ PINMUX_IPSR_GPSR(IP3_27_24, DU_DG7),
+
+ PINMUX_IPSR_GPSR(IP3_31_28, A16),
+ PINMUX_IPSR_GPSR(IP3_31_28, LCDOUT8),
+ PINMUX_IPSR_GPSR(IP3_31_28, VI4_FIELD),
+ PINMUX_IPSR_GPSR(IP3_31_28, DU_DG0),
+
+ /* IPSR4 */
+ PINMUX_IPSR_GPSR(IP4_3_0, A17),
+ PINMUX_IPSR_GPSR(IP4_3_0, LCDOUT9),
+ PINMUX_IPSR_GPSR(IP4_3_0, VI4_VSYNC_N),
+ PINMUX_IPSR_GPSR(IP4_3_0, DU_DG1),
+
+ PINMUX_IPSR_GPSR(IP4_7_4, A18),
+ PINMUX_IPSR_GPSR(IP4_7_4, LCDOUT10),
+ PINMUX_IPSR_GPSR(IP4_7_4, VI4_HSYNC_N),
+ PINMUX_IPSR_GPSR(IP4_7_4, DU_DG2),
+
+ PINMUX_IPSR_GPSR(IP4_11_8, A19),
+ PINMUX_IPSR_GPSR(IP4_11_8, LCDOUT11),
+ PINMUX_IPSR_GPSR(IP4_11_8, VI4_CLKENB),
+ PINMUX_IPSR_GPSR(IP4_11_8, DU_DG3),
+
+ PINMUX_IPSR_GPSR(IP4_15_12, CS0_N),
+ PINMUX_IPSR_GPSR(IP4_15_12, VI5_CLKENB),
+
+ PINMUX_IPSR_GPSR(IP4_19_16, CS1_N_A26),
+ PINMUX_IPSR_GPSR(IP4_19_16, VI5_CLK),
+ PINMUX_IPSR_MSEL(IP4_19_16, EX_WAIT0_B, SEL_LBSC_1),
+
+ PINMUX_IPSR_GPSR(IP4_23_20, BS_N),
+ PINMUX_IPSR_GPSR(IP4_23_20, QSTVA_QVS),
+ PINMUX_IPSR_MSEL(IP4_23_20, MSIOF3_SCK_D, SEL_MSIOF3_3),
+ PINMUX_IPSR_GPSR(IP4_23_20, SCK3),
+ PINMUX_IPSR_GPSR(IP4_23_20, HSCK3),
+ PINMUX_IPSR_GPSR(IP4_23_20, CAN1_TX),
+ PINMUX_IPSR_GPSR(IP4_23_20, CANFD1_TX),
+ PINMUX_IPSR_MSEL(IP4_23_20, IETX_A, SEL_IEBUS_0),
+
+ PINMUX_IPSR_GPSR(IP4_27_24, RD_N),
+ PINMUX_IPSR_MSEL(IP4_27_24, MSIOF3_SYNC_D, SEL_MSIOF3_3),
+ PINMUX_IPSR_MSEL(IP4_27_24, RX3_A, SEL_SCIF3_0),
+ PINMUX_IPSR_MSEL(IP4_27_24, HRX3_A, SEL_HSCIF3_0),
+ PINMUX_IPSR_MSEL(IP4_27_24, CAN0_TX_A, SEL_RCAN0_0),
+ PINMUX_IPSR_MSEL(IP4_27_24, CANFD0_TX_A, SEL_CANFD0_0),
+
+ PINMUX_IPSR_GPSR(IP4_31_28, RD_WR_N),
+ PINMUX_IPSR_MSEL(IP4_31_28, MSIOF3_RXD_D, SEL_MSIOF3_3),
+ PINMUX_IPSR_MSEL(IP4_31_28, TX3_A, SEL_SCIF3_0),
+ PINMUX_IPSR_MSEL(IP4_31_28, HTX3_A, SEL_HSCIF3_0),
+ PINMUX_IPSR_MSEL(IP4_31_28, CAN0_RX_A, SEL_RCAN0_0),
+ PINMUX_IPSR_MSEL(IP4_31_28, CANFD0_RX_A, SEL_CANFD0_0),
+
+ /* IPSR5 */
+ PINMUX_IPSR_GPSR(IP5_3_0, WE0_N),
+ PINMUX_IPSR_MSEL(IP5_3_0, MSIOF3_TXD_D, SEL_MSIOF3_3),
+ PINMUX_IPSR_GPSR(IP5_3_0, CTS3_N),
+ PINMUX_IPSR_GPSR(IP5_3_0, HCTS3_N),
+ PINMUX_IPSR_MSEL(IP5_3_0, SCL6_B, SEL_I2C6_1),
+ PINMUX_IPSR_GPSR(IP5_3_0, CAN_CLK),
+ PINMUX_IPSR_MSEL(IP5_3_0, IECLK_A, SEL_IEBUS_0),
+
+ PINMUX_IPSR_GPSR(IP5_7_4, WE1_N),
+ PINMUX_IPSR_MSEL(IP5_7_4, MSIOF3_SS1_D, SEL_MSIOF3_3),
+ PINMUX_IPSR_GPSR(IP5_7_4, RTS3_N_TANS),
+ PINMUX_IPSR_GPSR(IP5_7_4, HRTS3_N),
+ PINMUX_IPSR_MSEL(IP5_7_4, SDA6_B, SEL_I2C6_1),
+ PINMUX_IPSR_GPSR(IP5_7_4, CAN1_RX),
+ PINMUX_IPSR_GPSR(IP5_7_4, CANFD1_RX),
+ PINMUX_IPSR_MSEL(IP5_7_4, IERX_A, SEL_IEBUS_0),
+
+ PINMUX_IPSR_MSEL(IP5_11_8, EX_WAIT0_A, SEL_LBSC_0),
+ PINMUX_IPSR_GPSR(IP5_11_8, QCLK),
+ PINMUX_IPSR_GPSR(IP5_11_8, VI4_CLK),
+ PINMUX_IPSR_GPSR(IP5_11_8, DU_DOTCLKOUT0),
+
+ PINMUX_IPSR_GPSR(IP5_15_12, D0),
+ PINMUX_IPSR_MSEL(IP5_15_12, MSIOF2_SS1_B, SEL_MSIOF2_1),
+ PINMUX_IPSR_MSEL(IP5_15_12, MSIOF3_SCK_A, SEL_MSIOF3_0),
+ PINMUX_IPSR_GPSR(IP5_15_12, VI4_DATA16),
+ PINMUX_IPSR_GPSR(IP5_15_12, VI5_DATA0),
+
+ PINMUX_IPSR_GPSR(IP5_19_16, D1),
+ PINMUX_IPSR_MSEL(IP5_19_16, MSIOF2_SS2_B, SEL_MSIOF2_1),
+ PINMUX_IPSR_MSEL(IP5_19_16, MSIOF3_SYNC_A, SEL_MSIOF3_0),
+ PINMUX_IPSR_GPSR(IP5_19_16, VI4_DATA17),
+ PINMUX_IPSR_GPSR(IP5_19_16, VI5_DATA1),
+
+ PINMUX_IPSR_GPSR(IP5_23_20, D2),
+ PINMUX_IPSR_MSEL(IP5_23_20, MSIOF3_RXD_A, SEL_MSIOF3_0),
+ PINMUX_IPSR_GPSR(IP5_23_20, VI4_DATA18),
+ PINMUX_IPSR_GPSR(IP5_23_20, VI5_DATA2),
+
+ PINMUX_IPSR_GPSR(IP5_27_24, D3),
+ PINMUX_IPSR_MSEL(IP5_27_24, MSIOF3_TXD_A, SEL_MSIOF3_0),
+ PINMUX_IPSR_GPSR(IP5_27_24, VI4_DATA19),
+ PINMUX_IPSR_GPSR(IP5_27_24, VI5_DATA3),
+
+ PINMUX_IPSR_GPSR(IP5_31_28, D4),
+ PINMUX_IPSR_MSEL(IP5_31_28, MSIOF2_SCK_B, SEL_MSIOF2_1),
+ PINMUX_IPSR_GPSR(IP5_31_28, VI4_DATA20),
+ PINMUX_IPSR_GPSR(IP5_31_28, VI5_DATA4),
+
+ /* IPSR6 */
+ PINMUX_IPSR_GPSR(IP6_3_0, D5),
+ PINMUX_IPSR_MSEL(IP6_3_0, MSIOF2_SYNC_B, SEL_MSIOF2_1),
+ PINMUX_IPSR_GPSR(IP6_3_0, VI4_DATA21),
+ PINMUX_IPSR_GPSR(IP6_3_0, VI5_DATA5),
+
+ PINMUX_IPSR_GPSR(IP6_7_4, D6),
+ PINMUX_IPSR_MSEL(IP6_7_4, MSIOF2_RXD_B, SEL_MSIOF2_1),
+ PINMUX_IPSR_GPSR(IP6_7_4, VI4_DATA22),
+ PINMUX_IPSR_GPSR(IP6_7_4, VI5_DATA6),
+
+ PINMUX_IPSR_GPSR(IP6_11_8, D7),
+ PINMUX_IPSR_MSEL(IP6_11_8, MSIOF2_TXD_B, SEL_MSIOF2_1),
+ PINMUX_IPSR_GPSR(IP6_11_8, VI4_DATA23),
+ PINMUX_IPSR_GPSR(IP6_11_8, VI5_DATA7),
+
+ PINMUX_IPSR_GPSR(IP6_15_12, D8),
+ PINMUX_IPSR_GPSR(IP6_15_12, LCDOUT0),
+ PINMUX_IPSR_MSEL(IP6_15_12, MSIOF2_SCK_D, SEL_MSIOF2_3),
+ PINMUX_IPSR_MSEL(IP6_15_12, SCK4_C, SEL_SCIF4_2),
+ PINMUX_IPSR_MSEL(IP6_15_12, VI4_DATA0_A, SEL_VIN4_0),
+ PINMUX_IPSR_GPSR(IP6_15_12, DU_DR0),
+
+ PINMUX_IPSR_GPSR(IP6_19_16, D9),
+ PINMUX_IPSR_GPSR(IP6_19_16, LCDOUT1),
+ PINMUX_IPSR_MSEL(IP6_19_16, MSIOF2_SYNC_D, SEL_MSIOF2_3),
+ PINMUX_IPSR_MSEL(IP6_19_16, VI4_DATA1_A, SEL_VIN4_0),
+ PINMUX_IPSR_GPSR(IP6_19_16, DU_DR1),
+
+ PINMUX_IPSR_GPSR(IP6_23_20, D10),
+ PINMUX_IPSR_GPSR(IP6_23_20, LCDOUT2),
+ PINMUX_IPSR_MSEL(IP6_23_20, MSIOF2_RXD_D, SEL_MSIOF2_3),
+ PINMUX_IPSR_MSEL(IP6_23_20, HRX3_B, SEL_HSCIF3_1),
+ PINMUX_IPSR_MSEL(IP6_23_20, VI4_DATA2_A, SEL_VIN4_0),
+ PINMUX_IPSR_MSEL(IP6_23_20, CTS4_N_C, SEL_SCIF4_2),
+ PINMUX_IPSR_GPSR(IP6_23_20, DU_DR2),
+
+ PINMUX_IPSR_GPSR(IP6_27_24, D11),
+ PINMUX_IPSR_GPSR(IP6_27_24, LCDOUT3),
+ PINMUX_IPSR_MSEL(IP6_27_24, MSIOF2_TXD_D, SEL_MSIOF2_3),
+ PINMUX_IPSR_MSEL(IP6_27_24, HTX3_B, SEL_HSCIF3_1),
+ PINMUX_IPSR_MSEL(IP6_27_24, VI4_DATA3_A, SEL_VIN4_0),
+ PINMUX_IPSR_MSEL(IP6_27_24, RTS4_N_TANS_C, SEL_SCIF4_2),
+ PINMUX_IPSR_GPSR(IP6_27_24, DU_DR3),
+
+ PINMUX_IPSR_GPSR(IP6_31_28, D12),
+ PINMUX_IPSR_GPSR(IP6_31_28, LCDOUT4),
+ PINMUX_IPSR_MSEL(IP6_31_28, MSIOF2_SS1_D, SEL_MSIOF2_3),
+ PINMUX_IPSR_MSEL(IP6_31_28, RX4_C, SEL_SCIF4_2),
+ PINMUX_IPSR_MSEL(IP6_31_28, VI4_DATA4_A, SEL_VIN4_0),
+ PINMUX_IPSR_GPSR(IP6_31_28, DU_DR4),
+
+ /* IPSR7 */
+ PINMUX_IPSR_GPSR(IP7_3_0, D13),
+ PINMUX_IPSR_GPSR(IP7_3_0, LCDOUT5),
+ PINMUX_IPSR_MSEL(IP7_3_0, MSIOF2_SS2_D, SEL_MSIOF2_3),
+ PINMUX_IPSR_MSEL(IP7_3_0, TX4_C, SEL_SCIF4_2),
+ PINMUX_IPSR_MSEL(IP7_3_0, VI4_DATA5_A, SEL_VIN4_0),
+ PINMUX_IPSR_GPSR(IP7_3_0, DU_DR5),
+
+ PINMUX_IPSR_GPSR(IP7_7_4, D14),
+ PINMUX_IPSR_GPSR(IP7_7_4, LCDOUT6),
+ PINMUX_IPSR_MSEL(IP7_7_4, MSIOF3_SS1_A, SEL_MSIOF3_0),
+ PINMUX_IPSR_MSEL(IP7_7_4, HRX3_C, SEL_HSCIF3_2),
+ PINMUX_IPSR_MSEL(IP7_7_4, VI4_DATA6_A, SEL_VIN4_0),
+ PINMUX_IPSR_GPSR(IP7_7_4, DU_DR6),
+ PINMUX_IPSR_MSEL(IP7_7_4, SCL6_C, SEL_I2C6_2),
+
+ PINMUX_IPSR_GPSR(IP7_11_8, D15),
+ PINMUX_IPSR_GPSR(IP7_11_8, LCDOUT7),
+ PINMUX_IPSR_MSEL(IP7_11_8, MSIOF3_SS2_A, SEL_MSIOF3_0),
+ PINMUX_IPSR_MSEL(IP7_11_8, HTX3_C, SEL_HSCIF3_2),
+ PINMUX_IPSR_MSEL(IP7_11_8, VI4_DATA7_A, SEL_VIN4_0),
+ PINMUX_IPSR_GPSR(IP7_11_8, DU_DR7),
+ PINMUX_IPSR_MSEL(IP7_11_8, SDA6_C, SEL_I2C6_2),
+
+ PINMUX_IPSR_GPSR(IP7_15_12, FSCLKST),
+
+ PINMUX_IPSR_GPSR(IP7_19_16, SD0_CLK),
+ PINMUX_IPSR_MSEL(IP7_19_16, MSIOF1_SCK_E, SEL_MSIOF1_4),
+ PINMUX_IPSR_MSEL(IP7_19_16, STP_OPWM_0_B, SEL_SSP1_0_1),
+
+ PINMUX_IPSR_GPSR(IP7_23_20, SD0_CMD),
+ PINMUX_IPSR_MSEL(IP7_23_20, MSIOF1_SYNC_E, SEL_MSIOF1_4),
+ PINMUX_IPSR_MSEL(IP7_23_20, STP_IVCXO27_0_B, SEL_SSP1_0_1),
+
+ PINMUX_IPSR_GPSR(IP7_27_24, SD0_DAT0),
+ PINMUX_IPSR_MSEL(IP7_27_24, MSIOF1_RXD_E, SEL_MSIOF1_4),
+ PINMUX_IPSR_MSEL(IP7_27_24, TS_SCK0_B, SEL_TSIF0_1),
+ PINMUX_IPSR_MSEL(IP7_27_24, STP_ISCLK_0_B, SEL_SSP1_0_1),
+
+ PINMUX_IPSR_GPSR(IP7_31_28, SD0_DAT1),
+ PINMUX_IPSR_MSEL(IP7_31_28, MSIOF1_TXD_E, SEL_MSIOF1_4),
+ PINMUX_IPSR_MSEL(IP7_31_28, TS_SPSYNC0_B, SEL_TSIF0_1),
+ PINMUX_IPSR_MSEL(IP7_31_28, STP_ISSYNC_0_B, SEL_SSP1_0_1),
+
+ /* IPSR8 */
+ PINMUX_IPSR_GPSR(IP8_3_0, SD0_DAT2),
+ PINMUX_IPSR_MSEL(IP8_3_0, MSIOF1_SS1_E, SEL_MSIOF1_4),
+ PINMUX_IPSR_MSEL(IP8_3_0, TS_SDAT0_B, SEL_TSIF0_1),
+ PINMUX_IPSR_MSEL(IP8_3_0, STP_ISD_0_B, SEL_SSP1_0_1),
+
+ PINMUX_IPSR_GPSR(IP8_7_4, SD0_DAT3),
+ PINMUX_IPSR_MSEL(IP8_7_4, MSIOF1_SS2_E, SEL_MSIOF1_4),
+ PINMUX_IPSR_MSEL(IP8_7_4, TS_SDEN0_B, SEL_TSIF0_1),
+ PINMUX_IPSR_MSEL(IP8_7_4, STP_ISEN_0_B, SEL_SSP1_0_1),
+
+ PINMUX_IPSR_GPSR(IP8_11_8, SD1_CLK),
+ PINMUX_IPSR_MSEL(IP8_11_8, MSIOF1_SCK_G, SEL_MSIOF1_6),
+ PINMUX_IPSR_MSEL(IP8_11_8, SIM0_CLK_A, SEL_SIMCARD_0),
+
+ PINMUX_IPSR_GPSR(IP8_15_12, SD1_CMD),
+ PINMUX_IPSR_MSEL(IP8_15_12, MSIOF1_SYNC_G, SEL_MSIOF1_6),
+ PINMUX_IPSR_MSEL(IP8_15_12, SIM0_D_A, SEL_SIMCARD_0),
+ PINMUX_IPSR_MSEL(IP8_15_12, STP_IVCXO27_1_B, SEL_SSP1_1_1),
+
+ PINMUX_IPSR_GPSR(IP8_19_16, SD1_DAT0),
+ PINMUX_IPSR_GPSR(IP8_19_16, SD2_DAT4),
+ PINMUX_IPSR_MSEL(IP8_19_16, MSIOF1_RXD_G, SEL_MSIOF1_6),
+ PINMUX_IPSR_MSEL(IP8_19_16, TS_SCK1_B, SEL_TSIF1_1),
+ PINMUX_IPSR_MSEL(IP8_19_16, STP_ISCLK_1_B, SEL_SSP1_1_1),
+
+ PINMUX_IPSR_GPSR(IP8_23_20, SD1_DAT1),
+ PINMUX_IPSR_GPSR(IP8_23_20, SD2_DAT5),
+ PINMUX_IPSR_MSEL(IP8_23_20, MSIOF1_TXD_G, SEL_MSIOF1_6),
+ PINMUX_IPSR_MSEL(IP8_23_20, TS_SPSYNC1_B, SEL_TSIF1_1),
+ PINMUX_IPSR_MSEL(IP8_23_20, STP_ISSYNC_1_B, SEL_SSP1_1_1),
+
+ PINMUX_IPSR_GPSR(IP8_27_24, SD1_DAT2),
+ PINMUX_IPSR_GPSR(IP8_27_24, SD2_DAT6),
+ PINMUX_IPSR_MSEL(IP8_27_24, MSIOF1_SS1_G, SEL_MSIOF1_6),
+ PINMUX_IPSR_MSEL(IP8_27_24, TS_SDAT1_B, SEL_TSIF1_1),
+ PINMUX_IPSR_MSEL(IP8_27_24, STP_ISD_1_B, SEL_SSP1_1_1),
+
+ PINMUX_IPSR_GPSR(IP8_31_28, SD1_DAT3),
+ PINMUX_IPSR_GPSR(IP8_31_28, SD2_DAT7),
+ PINMUX_IPSR_MSEL(IP8_31_28, MSIOF1_SS2_G, SEL_MSIOF1_6),
+ PINMUX_IPSR_MSEL(IP8_31_28, TS_SDEN1_B, SEL_TSIF1_1),
+ PINMUX_IPSR_MSEL(IP8_31_28, STP_ISEN_1_B, SEL_SSP1_1_1),
+
+ /* IPSR9 */
+ PINMUX_IPSR_GPSR(IP9_3_0, SD2_CLK),
+
+ PINMUX_IPSR_GPSR(IP9_7_4, SD2_DAT0),
+
+ PINMUX_IPSR_GPSR(IP9_11_8, SD2_DAT1),
+
+ PINMUX_IPSR_GPSR(IP9_15_12, SD2_DAT2),
+
+ PINMUX_IPSR_GPSR(IP9_19_16, SD2_DAT3),
+
+ PINMUX_IPSR_GPSR(IP9_23_20, SD2_DS),
+ PINMUX_IPSR_MSEL(IP9_23_20, SATA_DEVSLP_B, SEL_SATA_1),
+
+ PINMUX_IPSR_GPSR(IP9_27_24, SD3_DAT4),
+ PINMUX_IPSR_MSEL(IP9_27_24, SD2_CD_A, SEL_SDHI2_0),
+
+ PINMUX_IPSR_GPSR(IP9_31_28, SD3_DAT5),
+ PINMUX_IPSR_MSEL(IP9_31_28, SD2_WP_A, SEL_SDHI2_0),
+
+ /* IPSR10 */
+ PINMUX_IPSR_GPSR(IP10_3_0, SD3_DAT6),
+ PINMUX_IPSR_GPSR(IP10_3_0, SD3_CD),
+
+ PINMUX_IPSR_GPSR(IP10_7_4, SD3_DAT7),
+ PINMUX_IPSR_GPSR(IP10_7_4, SD3_WP),
+
+ PINMUX_IPSR_GPSR(IP10_11_8, SD0_CD),
+ PINMUX_IPSR_MSEL(IP10_11_8, SCL2_B, SEL_I2C2_1),
+ PINMUX_IPSR_MSEL(IP10_11_8, SIM0_RST_A, SEL_SIMCARD_0),
+
+ PINMUX_IPSR_GPSR(IP10_15_12, SD0_WP),
+ PINMUX_IPSR_MSEL(IP10_15_12, SDA2_B, SEL_I2C2_1),
+
+ PINMUX_IPSR_GPSR(IP10_19_16, SD1_CD),
+ PINMUX_IPSR_MSEL(IP10_19_16, SIM0_CLK_B, SEL_SIMCARD_1),
+
+ PINMUX_IPSR_GPSR(IP10_23_20, SD1_WP),
+ PINMUX_IPSR_MSEL(IP10_23_20, SIM0_D_B, SEL_SIMCARD_1),
+
+ PINMUX_IPSR_GPSR(IP10_27_24, SCK0),
+ PINMUX_IPSR_MSEL(IP10_27_24, HSCK1_B, SEL_HSCIF1_1),
+ PINMUX_IPSR_MSEL(IP10_27_24, MSIOF1_SS2_B, SEL_MSIOF1_1),
+ PINMUX_IPSR_MSEL(IP10_27_24, AUDIO_CLKC_B, SEL_ADG_1),
+ PINMUX_IPSR_MSEL(IP10_27_24, SDA2_A, SEL_I2C2_0),
+ PINMUX_IPSR_MSEL(IP10_27_24, SIM0_RST_B, SEL_SIMCARD_1),
+ PINMUX_IPSR_MSEL(IP10_27_24, STP_OPWM_0_C, SEL_SSP1_0_2),
+ PINMUX_IPSR_MSEL(IP10_27_24, RIF0_CLK_B, SEL_DRIF0_1),
+ PINMUX_IPSR_GPSR(IP10_27_24, ADICHS2),
+
+ PINMUX_IPSR_GPSR(IP10_31_28, RX0),
+ PINMUX_IPSR_MSEL(IP10_31_28, HRX1_B, SEL_HSCIF1_1),
+ PINMUX_IPSR_MSEL(IP10_31_28, TS_SCK0_C, SEL_TSIF0_2),
+ PINMUX_IPSR_MSEL(IP10_31_28, STP_ISCLK_0_C, SEL_SSP1_0_2),
+ PINMUX_IPSR_MSEL(IP10_31_28, RIF0_D0_B, SEL_DRIF0_1),
+
+ /* IPSR11 */
+ PINMUX_IPSR_GPSR(IP11_3_0, TX0),
+ PINMUX_IPSR_MSEL(IP11_3_0, HTX1_B, SEL_HSCIF1_1),
+ PINMUX_IPSR_MSEL(IP11_3_0, TS_SPSYNC0_C, SEL_TSIF0_2),
+ PINMUX_IPSR_MSEL(IP11_3_0, STP_ISSYNC_0_C, SEL_SSP1_0_2),
+ PINMUX_IPSR_MSEL(IP11_3_0, RIF0_D1_B, SEL_DRIF0_1),
+
+ PINMUX_IPSR_GPSR(IP11_7_4, CTS0_N),
+ PINMUX_IPSR_MSEL(IP11_7_4, HCTS1_N_B, SEL_HSCIF1_1),
+ PINMUX_IPSR_MSEL(IP11_7_4, MSIOF1_SYNC_B, SEL_MSIOF1_1),
+ PINMUX_IPSR_MSEL(IP11_7_4, TS_SPSYNC1_C, SEL_TSIF1_2),
+ PINMUX_IPSR_MSEL(IP11_7_4, STP_ISSYNC_1_C, SEL_SSP1_1_2),
+ PINMUX_IPSR_MSEL(IP11_7_4, RIF1_SYNC_B, SEL_DRIF1_1),
+ PINMUX_IPSR_MSEL(IP11_7_4, AUDIO_CLKOUT_C, SEL_ADG_2),
+ PINMUX_IPSR_GPSR(IP11_7_4, ADICS_SAMP),
+
+ PINMUX_IPSR_GPSR(IP11_11_8, RTS0_N_TANS),
+ PINMUX_IPSR_MSEL(IP11_11_8, HRTS1_N_B, SEL_HSCIF1_1),
+ PINMUX_IPSR_MSEL(IP11_11_8, MSIOF1_SS1_B, SEL_MSIOF1_1),
+ PINMUX_IPSR_MSEL(IP11_11_8, AUDIO_CLKA_B, SEL_ADG_1),
+ PINMUX_IPSR_MSEL(IP11_11_8, SCL2_A, SEL_I2C2_0),
+ PINMUX_IPSR_MSEL(IP11_11_8, STP_IVCXO27_1_C, SEL_SSP1_1_2),
+ PINMUX_IPSR_MSEL(IP11_11_8, RIF0_SYNC_B, SEL_DRIF0_1),
+ PINMUX_IPSR_GPSR(IP11_11_8, ADICHS1),
+
+ PINMUX_IPSR_MSEL(IP11_15_12, RX1_A, SEL_SCIF1_0),
+ PINMUX_IPSR_MSEL(IP11_15_12, HRX1_A, SEL_HSCIF1_0),
+ PINMUX_IPSR_MSEL(IP11_15_12, TS_SDAT0_C, SEL_TSIF0_2),
+ PINMUX_IPSR_MSEL(IP11_15_12, STP_ISD_0_C, SEL_SSP1_0_2),
+ PINMUX_IPSR_MSEL(IP11_15_12, RIF1_CLK_C, SEL_DRIF1_2),
+
+ PINMUX_IPSR_MSEL(IP11_19_16, TX1_A, SEL_SCIF1_0),
+ PINMUX_IPSR_MSEL(IP11_19_16, HTX1_A, SEL_HSCIF1_0),
+ PINMUX_IPSR_MSEL(IP11_19_16, TS_SDEN0_C, SEL_TSIF0_2),
+ PINMUX_IPSR_MSEL(IP11_19_16, STP_ISEN_0_C, SEL_SSP1_0_2),
+ PINMUX_IPSR_MSEL(IP11_19_16, RIF1_D0_C, SEL_DRIF1_2),
+
+ PINMUX_IPSR_GPSR(IP11_23_20, CTS1_N),
+ PINMUX_IPSR_MSEL(IP11_23_20, HCTS1_N_A, SEL_HSCIF1_0),
+ PINMUX_IPSR_MSEL(IP11_23_20, MSIOF1_RXD_B, SEL_MSIOF1_1),
+ PINMUX_IPSR_MSEL(IP11_23_20, TS_SDEN1_C, SEL_TSIF1_2),
+ PINMUX_IPSR_MSEL(IP11_23_20, STP_ISEN_1_C, SEL_SSP1_1_2),
+ PINMUX_IPSR_MSEL(IP11_23_20, RIF1_D0_B, SEL_DRIF1_1),
+ PINMUX_IPSR_GPSR(IP11_23_20, ADIDATA),
+
+ PINMUX_IPSR_GPSR(IP11_27_24, RTS1_N_TANS),
+ PINMUX_IPSR_MSEL(IP11_27_24, HRTS1_N_A, SEL_HSCIF1_0),
+ PINMUX_IPSR_MSEL(IP11_27_24, MSIOF1_TXD_B, SEL_MSIOF1_1),
+ PINMUX_IPSR_MSEL(IP11_27_24, TS_SDAT1_C, SEL_TSIF1_2),
+ PINMUX_IPSR_MSEL(IP11_27_24, STP_ISD_1_C, SEL_SSP1_1_2),
+ PINMUX_IPSR_MSEL(IP11_27_24, RIF1_D1_B, SEL_DRIF1_1),
+ PINMUX_IPSR_GPSR(IP11_27_24, ADICHS0),
+
+ PINMUX_IPSR_GPSR(IP11_31_28, SCK2),
+ PINMUX_IPSR_MSEL(IP11_31_28, SCIF_CLK_B, SEL_SCIF1_1),
+ PINMUX_IPSR_MSEL(IP11_31_28, MSIOF1_SCK_B, SEL_MSIOF1_1),
+ PINMUX_IPSR_MSEL(IP11_31_28, TS_SCK1_C, SEL_TSIF1_2),
+ PINMUX_IPSR_MSEL(IP11_31_28, STP_ISCLK_1_C, SEL_SSP1_1_2),
+ PINMUX_IPSR_MSEL(IP11_31_28, RIF1_CLK_B, SEL_DRIF1_1),
+ PINMUX_IPSR_GPSR(IP11_31_28, ADICLK),
+
+ /* IPSR12 */
+ PINMUX_IPSR_MSEL(IP12_3_0, TX2_A, SEL_SCIF2_0),
+ PINMUX_IPSR_MSEL(IP12_3_0, SD2_CD_B, SEL_SDHI2_1),
+ PINMUX_IPSR_MSEL(IP12_3_0, SCL1_A, SEL_I2C1_0),
+ PINMUX_IPSR_MSEL(IP12_3_0, FMCLK_A, SEL_FM_0),
+ PINMUX_IPSR_MSEL(IP12_3_0, RIF1_D1_C, SEL_DRIF1_2),
+ PINMUX_IPSR_MSEL(IP12_3_0, FSO_CFE_0_B, SEL_FSO_1),
+
+ PINMUX_IPSR_MSEL(IP12_7_4, RX2_A, SEL_SCIF2_0),
+ PINMUX_IPSR_MSEL(IP12_7_4, SD2_WP_B, SEL_SDHI2_1),
+ PINMUX_IPSR_MSEL(IP12_7_4, SDA1_A, SEL_I2C1_0),
+ PINMUX_IPSR_MSEL(IP12_7_4, FMIN_A, SEL_FM_0),
+ PINMUX_IPSR_MSEL(IP12_7_4, RIF1_SYNC_C, SEL_DRIF1_2),
+ PINMUX_IPSR_MSEL(IP12_7_4, FSO_CFE_1_B, SEL_FSO_1),
+
+ PINMUX_IPSR_GPSR(IP12_11_8, HSCK0),
+ PINMUX_IPSR_MSEL(IP12_11_8, MSIOF1_SCK_D, SEL_MSIOF1_3),
+ PINMUX_IPSR_MSEL(IP12_11_8, AUDIO_CLKB_A, SEL_ADG_0),
+ PINMUX_IPSR_MSEL(IP12_11_8, SSI_SDATA1_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP12_11_8, TS_SCK0_D, SEL_TSIF0_3),
+ PINMUX_IPSR_MSEL(IP12_11_8, STP_ISCLK_0_D, SEL_SSP1_0_3),
+ PINMUX_IPSR_MSEL(IP12_11_8, RIF0_CLK_C, SEL_DRIF0_2),
+
+ PINMUX_IPSR_GPSR(IP12_15_12, HRX0),
+ PINMUX_IPSR_MSEL(IP12_15_12, MSIOF1_RXD_D, SEL_MSIOF1_3),
+ PINMUX_IPSR_MSEL(IP12_15_12, SSI_SDATA2_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP12_15_12, TS_SDEN0_D, SEL_TSIF0_3),
+ PINMUX_IPSR_MSEL(IP12_15_12, STP_ISEN_0_D, SEL_SSP1_0_3),
+ PINMUX_IPSR_MSEL(IP12_15_12, RIF0_D0_C, SEL_DRIF0_2),
+
+ PINMUX_IPSR_GPSR(IP12_19_16, HTX0),
+ PINMUX_IPSR_MSEL(IP12_19_16, MSIOF1_TXD_D, SEL_MSIOF1_3),
+ PINMUX_IPSR_MSEL(IP12_19_16, SSI_SDATA9_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP12_19_16, TS_SDAT0_D, SEL_TSIF0_3),
+ PINMUX_IPSR_MSEL(IP12_19_16, STP_ISD_0_D, SEL_SSP1_0_3),
+ PINMUX_IPSR_MSEL(IP12_19_16, RIF0_D1_C, SEL_DRIF0_2),
+
+ PINMUX_IPSR_GPSR(IP12_23_20, HCTS0_N),
+ PINMUX_IPSR_MSEL(IP12_23_20, RX2_B, SEL_SCIF2_1),
+ PINMUX_IPSR_MSEL(IP12_23_20, MSIOF1_SYNC_D, SEL_MSIOF1_3),
+ PINMUX_IPSR_MSEL(IP12_23_20, SSI_SCK9_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP12_23_20, TS_SPSYNC0_D, SEL_TSIF0_3),
+ PINMUX_IPSR_MSEL(IP12_23_20, STP_ISSYNC_0_D, SEL_SSP1_0_3),
+ PINMUX_IPSR_MSEL(IP12_23_20, RIF0_SYNC_C, SEL_DRIF0_2),
+ PINMUX_IPSR_MSEL(IP12_23_20, AUDIO_CLKOUT1_A, SEL_ADG_0),
+
+ PINMUX_IPSR_GPSR(IP12_27_24, HRTS0_N),
+ PINMUX_IPSR_MSEL(IP12_27_24, TX2_B, SEL_SCIF2_1),
+ PINMUX_IPSR_MSEL(IP12_27_24, MSIOF1_SS1_D, SEL_MSIOF1_3),
+ PINMUX_IPSR_MSEL(IP12_27_24, SSI_WS9_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP12_27_24, STP_IVCXO27_0_D, SEL_SSP1_0_3),
+ PINMUX_IPSR_MSEL(IP12_27_24, BPFCLK_A, SEL_FM_0),
+ PINMUX_IPSR_MSEL(IP12_27_24, AUDIO_CLKOUT2_A, SEL_ADG_0),
+
+ PINMUX_IPSR_GPSR(IP12_31_28, MSIOF0_SYNC),
+ PINMUX_IPSR_MSEL(IP12_31_28, AUDIO_CLKOUT_A, SEL_ADG_0),
+
+ /* IPSR13 */
+ PINMUX_IPSR_GPSR(IP13_3_0, MSIOF0_SS1),
+ PINMUX_IPSR_GPSR(IP13_3_0, RX5),
+ PINMUX_IPSR_MSEL(IP13_3_0, AUDIO_CLKA_C, SEL_ADG_2),
+ PINMUX_IPSR_MSEL(IP13_3_0, SSI_SCK2_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP13_3_0, STP_IVCXO27_0_C, SEL_SSP1_0_2),
+ PINMUX_IPSR_MSEL(IP13_3_0, AUDIO_CLKOUT3_A, SEL_ADG_0),
+ PINMUX_IPSR_MSEL(IP13_3_0, TCLK1_B, SEL_TIMER_TMU_1),
+
+ PINMUX_IPSR_GPSR(IP13_7_4, MSIOF0_SS2),
+ PINMUX_IPSR_GPSR(IP13_7_4, TX5),
+ PINMUX_IPSR_MSEL(IP13_7_4, MSIOF1_SS2_D, SEL_MSIOF1_3),
+ PINMUX_IPSR_MSEL(IP13_7_4, AUDIO_CLKC_A, SEL_ADG_0),
+ PINMUX_IPSR_MSEL(IP13_7_4, SSI_WS2_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP13_7_4, STP_OPWM_0_D, SEL_SSP1_0_3),
+ PINMUX_IPSR_MSEL(IP13_7_4, AUDIO_CLKOUT_D, SEL_ADG_3),
+ PINMUX_IPSR_MSEL(IP13_7_4, SPEEDIN_B, SEL_SPEED_PULSE_1),
+
+ PINMUX_IPSR_GPSR(IP13_11_8, MLB_CLK),
+ PINMUX_IPSR_MSEL(IP13_11_8, MSIOF1_SCK_F, SEL_MSIOF1_5),
+ PINMUX_IPSR_MSEL(IP13_11_8, SCL1_B, SEL_I2C1_1),
+
+ PINMUX_IPSR_GPSR(IP13_15_12, MLB_SIG),
+ PINMUX_IPSR_MSEL(IP13_15_12, RX1_B, SEL_SCIF1_1),
+ PINMUX_IPSR_MSEL(IP13_15_12, MSIOF1_SYNC_F, SEL_MSIOF1_5),
+ PINMUX_IPSR_MSEL(IP13_15_12, SDA1_B, SEL_I2C1_1),
+
+ PINMUX_IPSR_GPSR(IP13_19_16, MLB_DAT),
+ PINMUX_IPSR_MSEL(IP13_19_16, TX1_B, SEL_SCIF1_1),
+ PINMUX_IPSR_MSEL(IP13_19_16, MSIOF1_RXD_F, SEL_MSIOF1_5),
+
+ PINMUX_IPSR_GPSR(IP13_23_20, SSI_SCK01239),
+ PINMUX_IPSR_MSEL(IP13_23_20, MSIOF1_TXD_F, SEL_MSIOF1_5),
+
+ PINMUX_IPSR_GPSR(IP13_27_24, SSI_WS01239),
+ PINMUX_IPSR_MSEL(IP13_27_24, MSIOF1_SS1_F, SEL_MSIOF1_5),
+
+ PINMUX_IPSR_GPSR(IP13_31_28, SSI_SDATA0),
+ PINMUX_IPSR_MSEL(IP13_31_28, MSIOF1_SS2_F, SEL_MSIOF1_5),
+
+ /* IPSR14 */
+ PINMUX_IPSR_MSEL(IP14_3_0, SSI_SDATA1_A, SEL_SSI_0),
+
+ PINMUX_IPSR_MSEL(IP14_7_4, SSI_SDATA2_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP14_7_4, SSI_SCK1_B, SEL_SSI_1),
+
+ PINMUX_IPSR_GPSR(IP14_11_8, SSI_SCK34),
+ PINMUX_IPSR_MSEL(IP14_11_8, MSIOF1_SS1_A, SEL_MSIOF1_0),
+ PINMUX_IPSR_MSEL(IP14_11_8, STP_OPWM_0_A, SEL_SSP1_0_0),
+
+ PINMUX_IPSR_GPSR(IP14_15_12, SSI_WS34),
+ PINMUX_IPSR_MSEL(IP14_15_12, HCTS2_N_A, SEL_HSCIF2_0),
+ PINMUX_IPSR_MSEL(IP14_15_12, MSIOF1_SS2_A, SEL_MSIOF1_0),
+ PINMUX_IPSR_MSEL(IP14_15_12, STP_IVCXO27_0_A, SEL_SSP1_0_0),
+
+ PINMUX_IPSR_GPSR(IP14_19_16, SSI_SDATA3),
+ PINMUX_IPSR_MSEL(IP14_19_16, HRTS2_N_A, SEL_HSCIF2_0),
+ PINMUX_IPSR_MSEL(IP14_19_16, MSIOF1_TXD_A, SEL_MSIOF1_0),
+ PINMUX_IPSR_MSEL(IP14_19_16, TS_SCK0_A, SEL_TSIF0_0),
+ PINMUX_IPSR_MSEL(IP14_19_16, STP_ISCLK_0_A, SEL_SSP1_0_0),
+ PINMUX_IPSR_MSEL(IP14_19_16, RIF0_D1_A, SEL_DRIF0_0),
+ PINMUX_IPSR_MSEL(IP14_19_16, RIF2_D0_A, SEL_DRIF2_0),
+
+ PINMUX_IPSR_GPSR(IP14_23_20, SSI_SCK4),
+ PINMUX_IPSR_MSEL(IP14_23_20, HRX2_A, SEL_HSCIF2_0),
+ PINMUX_IPSR_MSEL(IP14_23_20, MSIOF1_SCK_A, SEL_MSIOF1_0),
+ PINMUX_IPSR_MSEL(IP14_23_20, TS_SDAT0_A, SEL_TSIF0_0),
+ PINMUX_IPSR_MSEL(IP14_23_20, STP_ISD_0_A, SEL_SSP1_0_0),
+ PINMUX_IPSR_MSEL(IP14_23_20, RIF0_CLK_A, SEL_DRIF0_0),
+ PINMUX_IPSR_MSEL(IP14_23_20, RIF2_CLK_A, SEL_DRIF2_0),
+
+ PINMUX_IPSR_GPSR(IP14_27_24, SSI_WS4),
+ PINMUX_IPSR_MSEL(IP14_27_24, HTX2_A, SEL_HSCIF2_0),
+ PINMUX_IPSR_MSEL(IP14_27_24, MSIOF1_SYNC_A, SEL_MSIOF1_0),
+ PINMUX_IPSR_MSEL(IP14_27_24, TS_SDEN0_A, SEL_TSIF0_0),
+ PINMUX_IPSR_MSEL(IP14_27_24, STP_ISEN_0_A, SEL_SSP1_0_0),
+ PINMUX_IPSR_MSEL(IP14_27_24, RIF0_SYNC_A, SEL_DRIF0_0),
+ PINMUX_IPSR_MSEL(IP14_27_24, RIF2_SYNC_A, SEL_DRIF2_0),
+
+ PINMUX_IPSR_GPSR(IP14_31_28, SSI_SDATA4),
+ PINMUX_IPSR_MSEL(IP14_31_28, HSCK2_A, SEL_HSCIF2_0),
+ PINMUX_IPSR_MSEL(IP14_31_28, MSIOF1_RXD_A, SEL_MSIOF1_0),
+ PINMUX_IPSR_MSEL(IP14_31_28, TS_SPSYNC0_A, SEL_TSIF0_0),
+ PINMUX_IPSR_MSEL(IP14_31_28, STP_ISSYNC_0_A, SEL_SSP1_0_0),
+ PINMUX_IPSR_MSEL(IP14_31_28, RIF0_D0_A, SEL_DRIF0_0),
+ PINMUX_IPSR_MSEL(IP14_31_28, RIF2_D1_A, SEL_DRIF2_0),
+
+ /* IPSR15 */
+ PINMUX_IPSR_GPSR(IP15_3_0, SSI_SCK6),
+ PINMUX_IPSR_GPSR(IP15_3_0, USB2_PWEN),
+ PINMUX_IPSR_MSEL(IP15_3_0, SIM0_RST_D, SEL_SIMCARD_3),
+
+ PINMUX_IPSR_GPSR(IP15_7_4, SSI_WS6),
+ PINMUX_IPSR_GPSR(IP15_7_4, USB2_OVC),
+ PINMUX_IPSR_MSEL(IP15_7_4, SIM0_D_D, SEL_SIMCARD_3),
+
+ PINMUX_IPSR_GPSR(IP15_11_8, SSI_SDATA6),
+ PINMUX_IPSR_MSEL(IP15_11_8, SIM0_CLK_D, SEL_SIMCARD_3),
+ PINMUX_IPSR_MSEL(IP15_11_8, SATA_DEVSLP_A, SEL_SATA_0),
+
+ PINMUX_IPSR_GPSR(IP15_15_12, SSI_SCK78),
+ PINMUX_IPSR_MSEL(IP15_15_12, HRX2_B, SEL_HSCIF2_1),
+ PINMUX_IPSR_MSEL(IP15_15_12, MSIOF1_SCK_C, SEL_MSIOF1_2),
+ PINMUX_IPSR_MSEL(IP15_15_12, TS_SCK1_A, SEL_TSIF1_0),
+ PINMUX_IPSR_MSEL(IP15_15_12, STP_ISCLK_1_A, SEL_SSP1_1_0),
+ PINMUX_IPSR_MSEL(IP15_15_12, RIF1_CLK_A, SEL_DRIF1_0),
+ PINMUX_IPSR_MSEL(IP15_15_12, RIF3_CLK_A, SEL_DRIF3_0),
+
+ PINMUX_IPSR_GPSR(IP15_19_16, SSI_WS78),
+ PINMUX_IPSR_MSEL(IP15_19_16, HTX2_B, SEL_HSCIF2_1),
+ PINMUX_IPSR_MSEL(IP15_19_16, MSIOF1_SYNC_C, SEL_MSIOF1_2),
+ PINMUX_IPSR_MSEL(IP15_19_16, TS_SDAT1_A, SEL_TSIF1_0),
+ PINMUX_IPSR_MSEL(IP15_19_16, STP_ISD_1_A, SEL_SSP1_1_0),
+ PINMUX_IPSR_MSEL(IP15_19_16, RIF1_SYNC_A, SEL_DRIF1_0),
+ PINMUX_IPSR_MSEL(IP15_19_16, RIF3_SYNC_A, SEL_DRIF3_0),
+
+ PINMUX_IPSR_GPSR(IP15_23_20, SSI_SDATA7),
+ PINMUX_IPSR_MSEL(IP15_23_20, HCTS2_N_B, SEL_HSCIF2_1),
+ PINMUX_IPSR_MSEL(IP15_23_20, MSIOF1_RXD_C, SEL_MSIOF1_2),
+ PINMUX_IPSR_MSEL(IP15_23_20, TS_SDEN1_A, SEL_TSIF1_0),
+ PINMUX_IPSR_MSEL(IP15_23_20, STP_ISEN_1_A, SEL_SSP1_1_0),
+ PINMUX_IPSR_MSEL(IP15_23_20, RIF1_D0_A, SEL_DRIF1_0),
+ PINMUX_IPSR_MSEL(IP15_23_20, RIF3_D0_A, SEL_DRIF3_0),
+ PINMUX_IPSR_MSEL(IP15_23_20, TCLK2_A, SEL_TIMER_TMU_0),
+
+ PINMUX_IPSR_GPSR(IP15_27_24, SSI_SDATA8),
+ PINMUX_IPSR_MSEL(IP15_27_24, HRTS2_N_B, SEL_HSCIF2_1),
+ PINMUX_IPSR_MSEL(IP15_27_24, MSIOF1_TXD_C, SEL_MSIOF1_2),
+ PINMUX_IPSR_MSEL(IP15_27_24, TS_SPSYNC1_A, SEL_TSIF1_0),
+ PINMUX_IPSR_MSEL(IP15_27_24, STP_ISSYNC_1_A, SEL_SSP1_1_0),
+ PINMUX_IPSR_MSEL(IP15_27_24, RIF1_D1_A, SEL_DRIF1_0),
+ PINMUX_IPSR_MSEL(IP15_27_24, RIF3_D1_A, SEL_DRIF3_0),
+
+ PINMUX_IPSR_MSEL(IP15_31_28, SSI_SDATA9_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP15_31_28, HSCK2_B, SEL_HSCIF2_1),
+ PINMUX_IPSR_MSEL(IP15_31_28, MSIOF1_SS1_C, SEL_MSIOF1_2),
+ PINMUX_IPSR_MSEL(IP15_31_28, HSCK1_A, SEL_HSCIF1_0),
+ PINMUX_IPSR_MSEL(IP15_31_28, SSI_WS1_B, SEL_SSI_1),
+ PINMUX_IPSR_GPSR(IP15_31_28, SCK1),
+ PINMUX_IPSR_MSEL(IP15_31_28, STP_IVCXO27_1_A, SEL_SSP1_1_0),
+ PINMUX_IPSR_GPSR(IP15_31_28, SCK5),
+
+ /* IPSR16 */
+ PINMUX_IPSR_MSEL(IP16_3_0, AUDIO_CLKA_A, SEL_ADG_0),
+ PINMUX_IPSR_GPSR(IP16_3_0, CC5_OSCOUT),
+
+ PINMUX_IPSR_MSEL(IP16_7_4, AUDIO_CLKB_B, SEL_ADG_1),
+ PINMUX_IPSR_MSEL(IP16_7_4, SCIF_CLK_A, SEL_SCIF1_0),
+ PINMUX_IPSR_MSEL(IP16_7_4, STP_IVCXO27_1_D, SEL_SSP1_1_3),
+ PINMUX_IPSR_MSEL(IP16_7_4, REMOCON_A, SEL_REMOCON_0),
+ PINMUX_IPSR_MSEL(IP16_7_4, TCLK1_A, SEL_TIMER_TMU_0),
+
+ PINMUX_IPSR_GPSR(IP16_11_8, USB0_PWEN),
+ PINMUX_IPSR_MSEL(IP16_11_8, SIM0_RST_C, SEL_SIMCARD_2),
+ PINMUX_IPSR_MSEL(IP16_11_8, TS_SCK1_D, SEL_TSIF1_3),
+ PINMUX_IPSR_MSEL(IP16_11_8, STP_ISCLK_1_D, SEL_SSP1_1_3),
+ PINMUX_IPSR_MSEL(IP16_11_8, BPFCLK_B, SEL_FM_1),
+ PINMUX_IPSR_MSEL(IP16_11_8, RIF3_CLK_B, SEL_DRIF3_1),
+
+ PINMUX_IPSR_GPSR(IP16_15_12, USB0_OVC),
+ PINMUX_IPSR_MSEL(IP16_11_8, SIM0_D_C, SEL_SIMCARD_2),
+ PINMUX_IPSR_MSEL(IP16_11_8, TS_SDAT1_D, SEL_TSIF1_3),
+ PINMUX_IPSR_MSEL(IP16_11_8, STP_ISD_1_D, SEL_SSP1_1_3),
+ PINMUX_IPSR_MSEL(IP16_11_8, RIF3_SYNC_B, SEL_DRIF3_1),
+
+ PINMUX_IPSR_GPSR(IP16_19_16, USB1_PWEN),
+ PINMUX_IPSR_MSEL(IP16_19_16, SIM0_CLK_C, SEL_SIMCARD_2),
+ PINMUX_IPSR_MSEL(IP16_19_16, SSI_SCK1_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP16_19_16, TS_SCK0_E, SEL_TSIF0_4),
+ PINMUX_IPSR_MSEL(IP16_19_16, STP_ISCLK_0_E, SEL_SSP1_0_4),
+ PINMUX_IPSR_MSEL(IP16_19_16, FMCLK_B, SEL_FM_1),
+ PINMUX_IPSR_MSEL(IP16_19_16, RIF2_CLK_B, SEL_DRIF2_1),
+ PINMUX_IPSR_MSEL(IP16_19_16, SPEEDIN_A, SEL_SPEED_PULSE_0),
+
+ PINMUX_IPSR_GPSR(IP16_23_20, USB1_OVC),
+ PINMUX_IPSR_MSEL(IP16_23_20, MSIOF1_SS2_C, SEL_MSIOF1_2),
+ PINMUX_IPSR_MSEL(IP16_23_20, SSI_WS1_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP16_23_20, TS_SDAT0_E, SEL_TSIF0_4),
+ PINMUX_IPSR_MSEL(IP16_23_20, STP_ISD_0_E, SEL_SSP1_0_4),
+ PINMUX_IPSR_MSEL(IP16_23_20, FMIN_B, SEL_FM_1),
+ PINMUX_IPSR_MSEL(IP16_23_20, RIF2_SYNC_B, SEL_DRIF2_1),
+ PINMUX_IPSR_MSEL(IP16_23_20, REMOCON_B, SEL_REMOCON_1),
+
+ PINMUX_IPSR_GPSR(IP16_27_24, USB30_PWEN),
+ PINMUX_IPSR_MSEL(IP16_27_24, AUDIO_CLKOUT_B, SEL_ADG_1),
+ PINMUX_IPSR_MSEL(IP16_27_24, SSI_SCK2_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP16_27_24, TS_SDEN1_D, SEL_TSIF1_3),
+ PINMUX_IPSR_MSEL(IP16_27_24, STP_ISEN_1_D, SEL_SSP1_1_2),
+ PINMUX_IPSR_MSEL(IP16_27_24, STP_OPWM_0_E, SEL_SSP1_0_4),
+ PINMUX_IPSR_MSEL(IP16_27_24, RIF3_D0_B, SEL_DRIF3_1),
+ PINMUX_IPSR_MSEL(IP16_27_24, TCLK2_B, SEL_TIMER_TMU_1),
+ PINMUX_IPSR_GPSR(IP16_27_24, TPU0TO0),
+
+ PINMUX_IPSR_GPSR(IP16_31_28, USB30_OVC),
+ PINMUX_IPSR_MSEL(IP16_31_28, AUDIO_CLKOUT1_B, SEL_ADG_1),
+ PINMUX_IPSR_MSEL(IP16_31_28, SSI_WS2_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP16_31_28, TS_SPSYNC1_D, SEL_TSIF1_3),
+ PINMUX_IPSR_MSEL(IP16_31_28, STP_ISSYNC_1_D, SEL_SSP1_1_3),
+ PINMUX_IPSR_MSEL(IP16_31_28, STP_IVCXO27_0_E, SEL_SSP1_0_4),
+ PINMUX_IPSR_MSEL(IP16_31_28, RIF3_D1_B, SEL_DRIF3_1),
+ PINMUX_IPSR_MSEL(IP16_31_28, FSO_TOE_B, SEL_FSO_1),
+ PINMUX_IPSR_GPSR(IP16_31_28, TPU0TO1),
+
+ /* IPSR17 */
+ PINMUX_IPSR_GPSR(IP17_3_0, USB31_PWEN),
+ PINMUX_IPSR_MSEL(IP17_3_0, AUDIO_CLKOUT2_B, SEL_ADG_1),
+ PINMUX_IPSR_MSEL(IP17_3_0, SSI_SCK9_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP17_3_0, TS_SDEN0_E, SEL_TSIF0_4),
+ PINMUX_IPSR_MSEL(IP17_3_0, STP_ISEN_0_E, SEL_SSP1_0_4),
+ PINMUX_IPSR_MSEL(IP17_3_0, RIF2_D0_B, SEL_DRIF2_1),
+ PINMUX_IPSR_GPSR(IP17_3_0, TPU0TO2),
+
+ PINMUX_IPSR_GPSR(IP17_7_4, USB31_OVC),
+ PINMUX_IPSR_MSEL(IP17_7_4, AUDIO_CLKOUT3_B, SEL_ADG_1),
+ PINMUX_IPSR_MSEL(IP17_7_4, SSI_WS9_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP17_7_4, TS_SPSYNC0_E, SEL_TSIF0_4),
+ PINMUX_IPSR_MSEL(IP17_7_4, STP_ISSYNC_0_E, SEL_SSP1_0_4),
+ PINMUX_IPSR_MSEL(IP17_7_4, RIF2_D1_B, SEL_DRIF2_1),
+ PINMUX_IPSR_GPSR(IP17_7_4, TPU0TO3),
+
+/*
+ * Static pins can not be muxed between different functions but
+ * still needs a mark entry in the pinmux list. Add each static
+ * pin to the list without an associated function. The sh-pfc
+ * core will do the right thing and skip trying to mux then pin
+ * while still applying configuration to it
+ */
+#define FM(x) PINMUX_DATA(x##_MARK, 0),
+ PINMUX_STATIC
+#undef FM
+};
+
+/*
+ * R8A7795 has 8 banks with 32 PGIOS in each => 256 GPIOs.
+ * Physical layout rows: A - AW, cols: 1 - 39.
+ */
+#define ROW_GROUP_A(r) ('Z' - 'A' + 1 + (r))
+#define PIN_NUMBER(r, c) (((r) - 'A') * 39 + (c) + 300)
+#define PIN_A_NUMBER(r, c) PIN_NUMBER(ROW_GROUP_A(r), c)
+
+static const struct sh_pfc_pin pinmux_pins[] = {
+ PINMUX_GPIO_GP_ALL(),
+
+ /*
+ * Pins not associated with a GPIO port.
+ *
+ * The pin positions are different between different r8a7795
+ * packages, all that is needed for the pfc driver is a unique
+ * number for each pin. To this end use the pin layout from
+ * R-Car H3SiP to calculate a unique number for each pin.
+ */
+ SH_PFC_PIN_NAMED_CFG('A', 8, AVB_TX_CTL, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('A', 9, AVB_MDIO, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('A', 12, AVB_TXCREFCLK, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('A', 13, AVB_RD0, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('A', 14, AVB_RD2, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('A', 16, AVB_RX_CTL, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('A', 17, AVB_TD2, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('A', 18, AVB_TD0, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('A', 19, AVB_TXC, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('B', 13, AVB_RD1, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('B', 14, AVB_RD3, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('B', 17, AVB_TD3, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('B', 18, AVB_TD1, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('B', 19, AVB_RXC, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('C', 1, PRESETOUT#, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('F', 1, CLKOUT, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('H', 37, MLB_REF, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('V', 3, QSPI1_SPCLK, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('V', 5, QSPI1_SSL, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('V', 6, RPC_WP#, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('V', 7, RPC_RESET#, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('W', 3, QSPI0_SPCLK, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('Y', 3, QSPI0_SSL, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('Y', 6, QSPI0_IO2, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG('Y', 7, RPC_INT#, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('B'), 4, QSPI0_MISO_IO1, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('B'), 6, QSPI0_IO3, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('C'), 3, QSPI1_IO3, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('C'), 5, QSPI0_MOSI_IO0, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('C'), 7, QSPI1_MOSI_IO0, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('D'), 38, FSCLKST#, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('D'), 39, EXTALR, SH_PFC_PIN_CFG_PULL_UP | SH_PFC_PIN_CFG_PULL_DOWN),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('E'), 4, QSPI1_IO2, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('E'), 5, QSPI1_MISO_IO1, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('P'), 7, DU_DOTCLKIN0, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('P'), 8, DU_DOTCLKIN1, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('R'), 7, DU_DOTCLKIN2, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('R'), 8, DU_DOTCLKIN3, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('R'), 26, TRST#, SH_PFC_PIN_CFG_PULL_UP | SH_PFC_PIN_CFG_PULL_DOWN),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('R'), 29, TDI, SH_PFC_PIN_CFG_PULL_UP | SH_PFC_PIN_CFG_PULL_DOWN),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('R'), 30, TMS, CFG_FLAGS),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('T'), 27, TCK, SH_PFC_PIN_CFG_PULL_UP | SH_PFC_PIN_CFG_PULL_DOWN),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('T'), 28, TDO, SH_PFC_PIN_CFG_DRIVE_STRENGTH),
+ SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('T'), 30, ASEBRK, CFG_FLAGS),
+};
+
+/* - AUDIO CLOCK ------------------------------------------------------------ */
+static const unsigned int audio_clk_a_a_pins[] = {
+ /* CLK A */
+ RCAR_GP_PIN(6, 22),
+};
+static const unsigned int audio_clk_a_a_mux[] = {
+ AUDIO_CLKA_A_MARK,
+};
+static const unsigned int audio_clk_a_b_pins[] = {
+ /* CLK A */
+ RCAR_GP_PIN(5, 4),
+};
+static const unsigned int audio_clk_a_b_mux[] = {
+ AUDIO_CLKA_B_MARK,
+};
+static const unsigned int audio_clk_a_c_pins[] = {
+ /* CLK A */
+ RCAR_GP_PIN(5, 19),
+};
+static const unsigned int audio_clk_a_c_mux[] = {
+ AUDIO_CLKA_C_MARK,
+};
+static const unsigned int audio_clk_b_a_pins[] = {
+ /* CLK B */
+ RCAR_GP_PIN(5, 12),
+};
+static const unsigned int audio_clk_b_a_mux[] = {
+ AUDIO_CLKB_A_MARK,
+};
+static const unsigned int audio_clk_b_b_pins[] = {
+ /* CLK B */
+ RCAR_GP_PIN(6, 23),
+};
+static const unsigned int audio_clk_b_b_mux[] = {
+ AUDIO_CLKB_B_MARK,
+};
+static const unsigned int audio_clk_c_a_pins[] = {
+ /* CLK C */
+ RCAR_GP_PIN(5, 21),
+};
+static const unsigned int audio_clk_c_a_mux[] = {
+ AUDIO_CLKC_A_MARK,
+};
+static const unsigned int audio_clk_c_b_pins[] = {
+ /* CLK C */
+ RCAR_GP_PIN(5, 0),
+};
+static const unsigned int audio_clk_c_b_mux[] = {
+ AUDIO_CLKC_B_MARK,
+};
+static const unsigned int audio_clkout_a_pins[] = {
+ /* CLKOUT */
+ RCAR_GP_PIN(5, 18),
+};
+static const unsigned int audio_clkout_a_mux[] = {
+ AUDIO_CLKOUT_A_MARK,
+};
+static const unsigned int audio_clkout_b_pins[] = {
+ /* CLKOUT */
+ RCAR_GP_PIN(6, 28),
+};
+static const unsigned int audio_clkout_b_mux[] = {
+ AUDIO_CLKOUT_B_MARK,
+};
+static const unsigned int audio_clkout_c_pins[] = {
+ /* CLKOUT */
+ RCAR_GP_PIN(5, 3),
+};
+static const unsigned int audio_clkout_c_mux[] = {
+ AUDIO_CLKOUT_C_MARK,
+};
+static const unsigned int audio_clkout_d_pins[] = {
+ /* CLKOUT */
+ RCAR_GP_PIN(5, 21),
+};
+static const unsigned int audio_clkout_d_mux[] = {
+ AUDIO_CLKOUT_D_MARK,
+};
+static const unsigned int audio_clkout1_a_pins[] = {
+ /* CLKOUT1 */
+ RCAR_GP_PIN(5, 15),
+};
+static const unsigned int audio_clkout1_a_mux[] = {
+ AUDIO_CLKOUT1_A_MARK,
+};
+static const unsigned int audio_clkout1_b_pins[] = {
+ /* CLKOUT1 */
+ RCAR_GP_PIN(6, 29),
+};
+static const unsigned int audio_clkout1_b_mux[] = {
+ AUDIO_CLKOUT1_B_MARK,
+};
+static const unsigned int audio_clkout2_a_pins[] = {
+ /* CLKOUT2 */
+ RCAR_GP_PIN(5, 16),
+};
+static const unsigned int audio_clkout2_a_mux[] = {
+ AUDIO_CLKOUT2_A_MARK,
+};
+static const unsigned int audio_clkout2_b_pins[] = {
+ /* CLKOUT2 */
+ RCAR_GP_PIN(6, 30),
+};
+static const unsigned int audio_clkout2_b_mux[] = {
+ AUDIO_CLKOUT2_B_MARK,
+};
+
+static const unsigned int audio_clkout3_a_pins[] = {
+ /* CLKOUT3 */
+ RCAR_GP_PIN(5, 19),
+};
+static const unsigned int audio_clkout3_a_mux[] = {
+ AUDIO_CLKOUT3_A_MARK,
+};
+static const unsigned int audio_clkout3_b_pins[] = {
+ /* CLKOUT3 */
+ RCAR_GP_PIN(6, 31),
+};
+static const unsigned int audio_clkout3_b_mux[] = {
+ AUDIO_CLKOUT3_B_MARK,
+};
+
+/* - EtherAVB --------------------------------------------------------------- */
+static const unsigned int avb_link_pins[] = {
+ /* AVB_LINK */
+ RCAR_GP_PIN(2, 12),
+};
+static const unsigned int avb_link_mux[] = {
+ AVB_LINK_MARK,
+};
+static const unsigned int avb_magic_pins[] = {
+ /* AVB_MAGIC_ */
+ RCAR_GP_PIN(2, 10),
+};
+static const unsigned int avb_magic_mux[] = {
+ AVB_MAGIC_MARK,
+};
+static const unsigned int avb_phy_int_pins[] = {
+ /* AVB_PHY_INT */
+ RCAR_GP_PIN(2, 11),
+};
+static const unsigned int avb_phy_int_mux[] = {
+ AVB_PHY_INT_MARK,
+};
+static const unsigned int avb_mdc_pins[] = {
+ /* AVB_MDC, AVB_MDIO */
+ RCAR_GP_PIN(2, 9), PIN_NUMBER('A', 9),
+};
+static const unsigned int avb_mdc_mux[] = {
+ AVB_MDC_MARK, AVB_MDIO_MARK,
+};
+static const unsigned int avb_mii_pins[] = {
+ /*
+ * AVB_TX_CTL, AVB_TXC, AVB_TD0,
+ * AVB_TD1, AVB_TD2, AVB_TD3,
+ * AVB_RX_CTL, AVB_RXC, AVB_RD0,
+ * AVB_RD1, AVB_RD2, AVB_RD3,
+ * AVB_TXCREFCLK
+ */
+ PIN_NUMBER('A', 8), PIN_NUMBER('A', 19), PIN_NUMBER('A', 18),
+ PIN_NUMBER('B', 18), PIN_NUMBER('A', 17), PIN_NUMBER('B', 17),
+ PIN_NUMBER('A', 16), PIN_NUMBER('B', 19), PIN_NUMBER('A', 13),
+ PIN_NUMBER('B', 13), PIN_NUMBER('A', 14), PIN_NUMBER('B', 14),
+ PIN_NUMBER('A', 12),
+
+};
+static const unsigned int avb_mii_mux[] = {
+ AVB_TX_CTL_MARK, AVB_TXC_MARK, AVB_TD0_MARK,
+ AVB_TD1_MARK, AVB_TD2_MARK, AVB_TD3_MARK,
+ AVB_RX_CTL_MARK, AVB_RXC_MARK, AVB_RD0_MARK,
+ AVB_RD1_MARK, AVB_RD2_MARK, AVB_RD3_MARK,
+ AVB_TXCREFCLK_MARK,
+};
+static const unsigned int avb_avtp_pps_pins[] = {
+ /* AVB_AVTP_PPS */
+ RCAR_GP_PIN(2, 6),
+};
+static const unsigned int avb_avtp_pps_mux[] = {
+ AVB_AVTP_PPS_MARK,
+};
+static const unsigned int avb_avtp_match_a_pins[] = {
+ /* AVB_AVTP_MATCH_A */
+ RCAR_GP_PIN(2, 13),
+};
+static const unsigned int avb_avtp_match_a_mux[] = {
+ AVB_AVTP_MATCH_A_MARK,
+};
+static const unsigned int avb_avtp_capture_a_pins[] = {
+ /* AVB_AVTP_CAPTURE_A */
+ RCAR_GP_PIN(2, 14),
+};
+static const unsigned int avb_avtp_capture_a_mux[] = {
+ AVB_AVTP_CAPTURE_A_MARK,
+};
+static const unsigned int avb_avtp_match_b_pins[] = {
+ /* AVB_AVTP_MATCH_B */
+ RCAR_GP_PIN(1, 8),
+};
+static const unsigned int avb_avtp_match_b_mux[] = {
+ AVB_AVTP_MATCH_B_MARK,
+};
+static const unsigned int avb_avtp_capture_b_pins[] = {
+ /* AVB_AVTP_CAPTURE_B */
+ RCAR_GP_PIN(1, 11),
+};
+static const unsigned int avb_avtp_capture_b_mux[] = {
+ AVB_AVTP_CAPTURE_B_MARK,
+};
+
+/* - CAN ------------------------------------------------------------------ */
+static const unsigned int can0_data_a_pins[] = {
+ /* TX, RX */
+ RCAR_GP_PIN(1, 23), RCAR_GP_PIN(1, 24),
+};
+static const unsigned int can0_data_a_mux[] = {
+ CAN0_TX_A_MARK, CAN0_RX_A_MARK,
+};
+static const unsigned int can0_data_b_pins[] = {
+ /* TX, RX */
+ RCAR_GP_PIN(2, 0), RCAR_GP_PIN(2, 1),
+};
+static const unsigned int can0_data_b_mux[] = {
+ CAN0_TX_B_MARK, CAN0_RX_B_MARK,
+};
+static const unsigned int can1_data_pins[] = {
+ /* TX, RX */
+ RCAR_GP_PIN(1, 22), RCAR_GP_PIN(1, 26),
+};
+static const unsigned int can1_data_mux[] = {
+ CAN1_TX_MARK, CAN1_RX_MARK,
+};
+
+/* - CAN Clock -------------------------------------------------------------- */
+static const unsigned int can_clk_pins[] = {
+ /* CLK */
+ RCAR_GP_PIN(1, 25),
+};
+static const unsigned int can_clk_mux[] = {
+ CAN_CLK_MARK,
+};
+
+/* - CAN FD --------------------------------------------------------------- */
+static const unsigned int canfd0_data_a_pins[] = {
+ /* TX, RX */
+ RCAR_GP_PIN(1, 23), RCAR_GP_PIN(1, 24),
+};
+static const unsigned int canfd0_data_a_mux[] = {
+ CANFD0_TX_A_MARK, CANFD0_RX_A_MARK,
+};
+static const unsigned int canfd0_data_b_pins[] = {
+ /* TX, RX */
+ RCAR_GP_PIN(2, 0), RCAR_GP_PIN(2, 1),
+};
+static const unsigned int canfd0_data_b_mux[] = {
+ CANFD0_TX_B_MARK, CANFD0_RX_B_MARK,
+};
+static const unsigned int canfd1_data_pins[] = {
+ /* TX, RX */
+ RCAR_GP_PIN(1, 22), RCAR_GP_PIN(1, 26),
+};
+static const unsigned int canfd1_data_mux[] = {
+ CANFD1_TX_MARK, CANFD1_RX_MARK,
+};
+
+/* - DRIF0 --------------------------------------------------------------- */
+static const unsigned int drif0_ctrl_a_pins[] = {
+ /* CLK, SYNC */
+ RCAR_GP_PIN(6, 8), RCAR_GP_PIN(6, 9),
+};
+static const unsigned int drif0_ctrl_a_mux[] = {
+ RIF0_CLK_A_MARK, RIF0_SYNC_A_MARK,
+};
+static const unsigned int drif0_data0_a_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(6, 10),
+};
+static const unsigned int drif0_data0_a_mux[] = {
+ RIF0_D0_A_MARK,
+};
+static const unsigned int drif0_data1_a_pins[] = {
+ /* D1 */
+ RCAR_GP_PIN(6, 7),
+};
+static const unsigned int drif0_data1_a_mux[] = {
+ RIF0_D1_A_MARK,
+};
+static const unsigned int drif0_ctrl_b_pins[] = {
+ /* CLK, SYNC */
+ RCAR_GP_PIN(5, 0), RCAR_GP_PIN(5, 4),
+};
+static const unsigned int drif0_ctrl_b_mux[] = {
+ RIF0_CLK_B_MARK, RIF0_SYNC_B_MARK,
+};
+static const unsigned int drif0_data0_b_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(5, 1),
+};
+static const unsigned int drif0_data0_b_mux[] = {
+ RIF0_D0_B_MARK,
+};
+static const unsigned int drif0_data1_b_pins[] = {
+ /* D1 */
+ RCAR_GP_PIN(5, 2),
+};
+static const unsigned int drif0_data1_b_mux[] = {
+ RIF0_D1_B_MARK,
+};
+static const unsigned int drif0_ctrl_c_pins[] = {
+ /* CLK, SYNC */
+ RCAR_GP_PIN(5, 12), RCAR_GP_PIN(5, 15),
+};
+static const unsigned int drif0_ctrl_c_mux[] = {
+ RIF0_CLK_C_MARK, RIF0_SYNC_C_MARK,
+};
+static const unsigned int drif0_data0_c_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(5, 13),
+};
+static const unsigned int drif0_data0_c_mux[] = {
+ RIF0_D0_C_MARK,
+};
+static const unsigned int drif0_data1_c_pins[] = {
+ /* D1 */
+ RCAR_GP_PIN(5, 14),
+};
+static const unsigned int drif0_data1_c_mux[] = {
+ RIF0_D1_C_MARK,
+};
+/* - DRIF1 --------------------------------------------------------------- */
+static const unsigned int drif1_ctrl_a_pins[] = {
+ /* CLK, SYNC */
+ RCAR_GP_PIN(6, 17), RCAR_GP_PIN(6, 18),
+};
+static const unsigned int drif1_ctrl_a_mux[] = {
+ RIF1_CLK_A_MARK, RIF1_SYNC_A_MARK,
+};
+static const unsigned int drif1_data0_a_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(6, 19),
+};
+static const unsigned int drif1_data0_a_mux[] = {
+ RIF1_D0_A_MARK,
+};
+static const unsigned int drif1_data1_a_pins[] = {
+ /* D1 */
+ RCAR_GP_PIN(6, 20),
+};
+static const unsigned int drif1_data1_a_mux[] = {
+ RIF1_D1_A_MARK,
+};
+static const unsigned int drif1_ctrl_b_pins[] = {
+ /* CLK, SYNC */
+ RCAR_GP_PIN(5, 9), RCAR_GP_PIN(5, 3),
+};
+static const unsigned int drif1_ctrl_b_mux[] = {
+ RIF1_CLK_B_MARK, RIF1_SYNC_B_MARK,
+};
+static const unsigned int drif1_data0_b_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(5, 7),
+};
+static const unsigned int drif1_data0_b_mux[] = {
+ RIF1_D0_B_MARK,
+};
+static const unsigned int drif1_data1_b_pins[] = {
+ /* D1 */
+ RCAR_GP_PIN(5, 8),
+};
+static const unsigned int drif1_data1_b_mux[] = {
+ RIF1_D1_B_MARK,
+};
+static const unsigned int drif1_ctrl_c_pins[] = {
+ /* CLK, SYNC */
+ RCAR_GP_PIN(5, 5), RCAR_GP_PIN(5, 11),
+};
+static const unsigned int drif1_ctrl_c_mux[] = {
+ RIF1_CLK_C_MARK, RIF1_SYNC_C_MARK,
+};
+static const unsigned int drif1_data0_c_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(5, 6),
+};
+static const unsigned int drif1_data0_c_mux[] = {
+ RIF1_D0_C_MARK,
+};
+static const unsigned int drif1_data1_c_pins[] = {
+ /* D1 */
+ RCAR_GP_PIN(5, 10),
+};
+static const unsigned int drif1_data1_c_mux[] = {
+ RIF1_D1_C_MARK,
+};
+/* - DRIF2 --------------------------------------------------------------- */
+static const unsigned int drif2_ctrl_a_pins[] = {
+ /* CLK, SYNC */
+ RCAR_GP_PIN(6, 8), RCAR_GP_PIN(6, 9),
+};
+static const unsigned int drif2_ctrl_a_mux[] = {
+ RIF2_CLK_A_MARK, RIF2_SYNC_A_MARK,
+};
+static const unsigned int drif2_data0_a_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(6, 7),
+};
+static const unsigned int drif2_data0_a_mux[] = {
+ RIF2_D0_A_MARK,
+};
+static const unsigned int drif2_data1_a_pins[] = {
+ /* D1 */
+ RCAR_GP_PIN(6, 10),
+};
+static const unsigned int drif2_data1_a_mux[] = {
+ RIF2_D1_A_MARK,
+};
+static const unsigned int drif2_ctrl_b_pins[] = {
+ /* CLK, SYNC */
+ RCAR_GP_PIN(6, 26), RCAR_GP_PIN(6, 27),
+};
+static const unsigned int drif2_ctrl_b_mux[] = {
+ RIF2_CLK_B_MARK, RIF2_SYNC_B_MARK,
+};
+static const unsigned int drif2_data0_b_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(6, 30),
+};
+static const unsigned int drif2_data0_b_mux[] = {
+ RIF2_D0_B_MARK,
+};
+static const unsigned int drif2_data1_b_pins[] = {
+ /* D1 */
+ RCAR_GP_PIN(6, 31),
+};
+static const unsigned int drif2_data1_b_mux[] = {
+ RIF2_D1_B_MARK,
+};
+/* - DRIF3 --------------------------------------------------------------- */
+static const unsigned int drif3_ctrl_a_pins[] = {
+ /* CLK, SYNC */
+ RCAR_GP_PIN(6, 17), RCAR_GP_PIN(6, 18),
+};
+static const unsigned int drif3_ctrl_a_mux[] = {
+ RIF3_CLK_A_MARK, RIF3_SYNC_A_MARK,
+};
+static const unsigned int drif3_data0_a_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(6, 19),
+};
+static const unsigned int drif3_data0_a_mux[] = {
+ RIF3_D0_A_MARK,
+};
+static const unsigned int drif3_data1_a_pins[] = {
+ /* D1 */
+ RCAR_GP_PIN(6, 20),
+};
+static const unsigned int drif3_data1_a_mux[] = {
+ RIF3_D1_A_MARK,
+};
+static const unsigned int drif3_ctrl_b_pins[] = {
+ /* CLK, SYNC */
+ RCAR_GP_PIN(6, 24), RCAR_GP_PIN(6, 25),
+};
+static const unsigned int drif3_ctrl_b_mux[] = {
+ RIF3_CLK_B_MARK, RIF3_SYNC_B_MARK,
+};
+static const unsigned int drif3_data0_b_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(6, 28),
+};
+static const unsigned int drif3_data0_b_mux[] = {
+ RIF3_D0_B_MARK,
+};
+static const unsigned int drif3_data1_b_pins[] = {
+ /* D1 */
+ RCAR_GP_PIN(6, 29),
+};
+static const unsigned int drif3_data1_b_mux[] = {
+ RIF3_D1_B_MARK,
+};
+
+/* - DU --------------------------------------------------------------------- */
+static const unsigned int du_rgb666_pins[] = {
+ /* R[7:2], G[7:2], B[7:2] */
+ RCAR_GP_PIN(0, 15), RCAR_GP_PIN(0, 14), RCAR_GP_PIN(0, 13),
+ RCAR_GP_PIN(0, 12), RCAR_GP_PIN(0, 11), RCAR_GP_PIN(0, 10),
+ RCAR_GP_PIN(1, 15), RCAR_GP_PIN(1, 14), RCAR_GP_PIN(1, 13),
+ RCAR_GP_PIN(1, 12), RCAR_GP_PIN(1, 19), RCAR_GP_PIN(1, 18),
+ RCAR_GP_PIN(1, 7), RCAR_GP_PIN(1, 6), RCAR_GP_PIN(1, 5),
+ RCAR_GP_PIN(1, 4), RCAR_GP_PIN(1, 3), RCAR_GP_PIN(1, 2),
+};
+static const unsigned int du_rgb666_mux[] = {
+ DU_DR7_MARK, DU_DR6_MARK, DU_DR5_MARK, DU_DR4_MARK,
+ DU_DR3_MARK, DU_DR2_MARK,
+ DU_DG7_MARK, DU_DG6_MARK, DU_DG5_MARK, DU_DG4_MARK,
+ DU_DG3_MARK, DU_DG2_MARK,
+ DU_DB7_MARK, DU_DB6_MARK, DU_DB5_MARK, DU_DB4_MARK,
+ DU_DB3_MARK, DU_DB2_MARK,
+};
+static const unsigned int du_rgb888_pins[] = {
+ /* R[7:0], G[7:0], B[7:0] */
+ RCAR_GP_PIN(0, 15), RCAR_GP_PIN(0, 14), RCAR_GP_PIN(0, 13),
+ RCAR_GP_PIN(0, 12), RCAR_GP_PIN(0, 11), RCAR_GP_PIN(0, 10),
+ RCAR_GP_PIN(0, 9), RCAR_GP_PIN(0, 8),
+ RCAR_GP_PIN(1, 15), RCAR_GP_PIN(1, 14), RCAR_GP_PIN(1, 13),
+ RCAR_GP_PIN(1, 12), RCAR_GP_PIN(1, 19), RCAR_GP_PIN(1, 18),
+ RCAR_GP_PIN(1, 17), RCAR_GP_PIN(1, 16),
+ RCAR_GP_PIN(1, 7), RCAR_GP_PIN(1, 6), RCAR_GP_PIN(1, 5),
+ RCAR_GP_PIN(1, 4), RCAR_GP_PIN(1, 3), RCAR_GP_PIN(1, 2),
+ RCAR_GP_PIN(1, 1), RCAR_GP_PIN(1, 0),
+};
+static const unsigned int du_rgb888_mux[] = {
+ DU_DR7_MARK, DU_DR6_MARK, DU_DR5_MARK, DU_DR4_MARK,
+ DU_DR3_MARK, DU_DR2_MARK, DU_DR1_MARK, DU_DR0_MARK,
+ DU_DG7_MARK, DU_DG6_MARK, DU_DG5_MARK, DU_DG4_MARK,
+ DU_DG3_MARK, DU_DG2_MARK, DU_DG1_MARK, DU_DG0_MARK,
+ DU_DB7_MARK, DU_DB6_MARK, DU_DB5_MARK, DU_DB4_MARK,
+ DU_DB3_MARK, DU_DB2_MARK, DU_DB1_MARK, DU_DB0_MARK,
+};
+static const unsigned int du_clk_out_0_pins[] = {
+ /* CLKOUT */
+ RCAR_GP_PIN(1, 27),
+};
+static const unsigned int du_clk_out_0_mux[] = {
+ DU_DOTCLKOUT0_MARK
+};
+static const unsigned int du_clk_out_1_pins[] = {
+ /* CLKOUT */
+ RCAR_GP_PIN(2, 3),
+};
+static const unsigned int du_clk_out_1_mux[] = {
+ DU_DOTCLKOUT1_MARK
+};
+static const unsigned int du_sync_pins[] = {
+ /* EXVSYNC/VSYNC, EXHSYNC/HSYNC */
+ RCAR_GP_PIN(2, 5), RCAR_GP_PIN(2, 4),
+};
+static const unsigned int du_sync_mux[] = {
+ DU_EXVSYNC_DU_VSYNC_MARK, DU_EXHSYNC_DU_HSYNC_MARK
+};
+static const unsigned int du_oddf_pins[] = {
+ /* EXDISP/EXODDF/EXCDE */
+ RCAR_GP_PIN(2, 2),
+};
+static const unsigned int du_oddf_mux[] = {
+ DU_EXODDF_DU_ODDF_DISP_CDE_MARK,
+};
+static const unsigned int du_cde_pins[] = {
+ /* CDE */
+ RCAR_GP_PIN(2, 0),
+};
+static const unsigned int du_cde_mux[] = {
+ DU_CDE_MARK,
+};
+static const unsigned int du_disp_pins[] = {
+ /* DISP */
+ RCAR_GP_PIN(2, 1),
+};
+static const unsigned int du_disp_mux[] = {
+ DU_DISP_MARK,
+};
+/* - HSCIF0 ----------------------------------------------------------------- */
+static const unsigned int hscif0_data_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(5, 13), RCAR_GP_PIN(5, 14),
+};
+static const unsigned int hscif0_data_mux[] = {
+ HRX0_MARK, HTX0_MARK,
+};
+static const unsigned int hscif0_clk_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(5, 12),
+};
+static const unsigned int hscif0_clk_mux[] = {
+ HSCK0_MARK,
+};
+static const unsigned int hscif0_ctrl_pins[] = {
+ /* RTS, CTS */
+ RCAR_GP_PIN(5, 16), RCAR_GP_PIN(5, 15),
+};
+static const unsigned int hscif0_ctrl_mux[] = {
+ HRTS0_N_MARK, HCTS0_N_MARK,
+};
+/* - HSCIF1 ----------------------------------------------------------------- */
+static const unsigned int hscif1_data_a_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(5, 5), RCAR_GP_PIN(5, 6),
+};
+static const unsigned int hscif1_data_a_mux[] = {
+ HRX1_A_MARK, HTX1_A_MARK,
+};
+static const unsigned int hscif1_clk_a_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(6, 21),
+};
+static const unsigned int hscif1_clk_a_mux[] = {
+ HSCK1_A_MARK,
+};
+static const unsigned int hscif1_ctrl_a_pins[] = {
+ /* RTS, CTS */
+ RCAR_GP_PIN(5, 8), RCAR_GP_PIN(5, 7),
+};
+static const unsigned int hscif1_ctrl_a_mux[] = {
+ HRTS1_N_A_MARK, HCTS1_N_A_MARK,
+};
+
+static const unsigned int hscif1_data_b_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(5, 1), RCAR_GP_PIN(5, 2),
+};
+static const unsigned int hscif1_data_b_mux[] = {
+ HRX1_B_MARK, HTX1_B_MARK,
+};
+static const unsigned int hscif1_clk_b_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(5, 0),
+};
+static const unsigned int hscif1_clk_b_mux[] = {
+ HSCK1_B_MARK,
+};
+static const unsigned int hscif1_ctrl_b_pins[] = {
+ /* RTS, CTS */
+ RCAR_GP_PIN(5, 4), RCAR_GP_PIN(5, 3),
+};
+static const unsigned int hscif1_ctrl_b_mux[] = {
+ HRTS1_N_B_MARK, HCTS1_N_B_MARK,
+};
+/* - HSCIF2 ----------------------------------------------------------------- */
+static const unsigned int hscif2_data_a_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(6, 8), RCAR_GP_PIN(6, 9),
+};
+static const unsigned int hscif2_data_a_mux[] = {
+ HRX2_A_MARK, HTX2_A_MARK,
+};
+static const unsigned int hscif2_clk_a_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(6, 10),
+};
+static const unsigned int hscif2_clk_a_mux[] = {
+ HSCK2_A_MARK,
+};
+static const unsigned int hscif2_ctrl_a_pins[] = {
+ /* RTS, CTS */
+ RCAR_GP_PIN(6, 7), RCAR_GP_PIN(6, 6),
+};
+static const unsigned int hscif2_ctrl_a_mux[] = {
+ HRTS2_N_A_MARK, HCTS2_N_A_MARK,
+};
+
+static const unsigned int hscif2_data_b_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(6, 17), RCAR_GP_PIN(6, 18),
+};
+static const unsigned int hscif2_data_b_mux[] = {
+ HRX2_B_MARK, HTX2_B_MARK,
+};
+static const unsigned int hscif2_clk_b_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(6, 21),
+};
+static const unsigned int hscif2_clk_b_mux[] = {
+ HSCK2_B_MARK,
+};
+static const unsigned int hscif2_ctrl_b_pins[] = {
+ /* RTS, CTS */
+ RCAR_GP_PIN(6, 20), RCAR_GP_PIN(6, 19),
+};
+static const unsigned int hscif2_ctrl_b_mux[] = {
+ HRTS2_N_B_MARK, HCTS2_N_B_MARK,
+};
+/* - HSCIF3 ----------------------------------------------------------------- */
+static const unsigned int hscif3_data_a_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(1, 23), RCAR_GP_PIN(1, 24),
+};
+static const unsigned int hscif3_data_a_mux[] = {
+ HRX3_A_MARK, HTX3_A_MARK,
+};
+static const unsigned int hscif3_clk_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(1, 22),
+};
+static const unsigned int hscif3_clk_mux[] = {
+ HSCK3_MARK,
+};
+static const unsigned int hscif3_ctrl_pins[] = {
+ /* RTS, CTS */
+ RCAR_GP_PIN(1, 26), RCAR_GP_PIN(1, 25),
+};
+static const unsigned int hscif3_ctrl_mux[] = {
+ HRTS3_N_MARK, HCTS3_N_MARK,
+};
+
+static const unsigned int hscif3_data_b_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(0, 10), RCAR_GP_PIN(0, 11),
+};
+static const unsigned int hscif3_data_b_mux[] = {
+ HRX3_B_MARK, HTX3_B_MARK,
+};
+static const unsigned int hscif3_data_c_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(0, 14), RCAR_GP_PIN(0, 15),
+};
+static const unsigned int hscif3_data_c_mux[] = {
+ HRX3_C_MARK, HTX3_C_MARK,
+};
+static const unsigned int hscif3_data_d_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(2, 7), RCAR_GP_PIN(2, 8),
+};
+static const unsigned int hscif3_data_d_mux[] = {
+ HRX3_D_MARK, HTX3_D_MARK,
+};
+/* - HSCIF4 ----------------------------------------------------------------- */
+static const unsigned int hscif4_data_a_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(1, 12), RCAR_GP_PIN(1, 13),
+};
+static const unsigned int hscif4_data_a_mux[] = {
+ HRX4_A_MARK, HTX4_A_MARK,
+};
+static const unsigned int hscif4_clk_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(1, 11),
+};
+static const unsigned int hscif4_clk_mux[] = {
+ HSCK4_MARK,
+};
+static const unsigned int hscif4_ctrl_pins[] = {
+ /* RTS, CTS */
+ RCAR_GP_PIN(1, 15), RCAR_GP_PIN(1, 14),
+};
+static const unsigned int hscif4_ctrl_mux[] = {
+ HRTS4_N_MARK, HCTS4_N_MARK,
+};
+
+static const unsigned int hscif4_data_b_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(1, 8), RCAR_GP_PIN(1, 11),
+};
+static const unsigned int hscif4_data_b_mux[] = {
+ HRX4_B_MARK, HTX4_B_MARK,
+};
+
+/* - I2C -------------------------------------------------------------------- */
+static const unsigned int i2c1_a_pins[] = {
+ /* SDA, SCL */
+ RCAR_GP_PIN(5, 11), RCAR_GP_PIN(5, 10),
+};
+static const unsigned int i2c1_a_mux[] = {
+ SDA1_A_MARK, SCL1_A_MARK,
+};
+static const unsigned int i2c1_b_pins[] = {
+ /* SDA, SCL */
+ RCAR_GP_PIN(5, 24), RCAR_GP_PIN(5, 23),
+};
+static const unsigned int i2c1_b_mux[] = {
+ SDA1_B_MARK, SCL1_B_MARK,
+};
+static const unsigned int i2c2_a_pins[] = {
+ /* SDA, SCL */
+ RCAR_GP_PIN(5, 0), RCAR_GP_PIN(5, 4),
+};
+static const unsigned int i2c2_a_mux[] = {
+ SDA2_A_MARK, SCL2_A_MARK,
+};
+static const unsigned int i2c2_b_pins[] = {
+ /* SDA, SCL */
+ RCAR_GP_PIN(3, 13), RCAR_GP_PIN(3, 12),
+};
+static const unsigned int i2c2_b_mux[] = {
+ SDA2_B_MARK, SCL2_B_MARK,
+};
+static const unsigned int i2c6_a_pins[] = {
+ /* SDA, SCL */
+ RCAR_GP_PIN(1, 8), RCAR_GP_PIN(1, 11),
+};
+static const unsigned int i2c6_a_mux[] = {
+ SDA6_A_MARK, SCL6_A_MARK,
+};
+static const unsigned int i2c6_b_pins[] = {
+ /* SDA, SCL */
+ RCAR_GP_PIN(1, 26), RCAR_GP_PIN(1, 25),
+};
+static const unsigned int i2c6_b_mux[] = {
+ SDA6_B_MARK, SCL6_B_MARK,
+};
+static const unsigned int i2c6_c_pins[] = {
+ /* SDA, SCL */
+ RCAR_GP_PIN(0, 15), RCAR_GP_PIN(0, 14),
+};
+static const unsigned int i2c6_c_mux[] = {
+ SDA6_C_MARK, SCL6_C_MARK,
+};
+
+/* - INTC-EX ---------------------------------------------------------------- */
+static const unsigned int intc_ex_irq0_pins[] = {
+ /* IRQ0 */
+ RCAR_GP_PIN(2, 0),
+};
+static const unsigned int intc_ex_irq0_mux[] = {
+ IRQ0_MARK,
+};
+static const unsigned int intc_ex_irq1_pins[] = {
+ /* IRQ1 */
+ RCAR_GP_PIN(2, 1),
+};
+static const unsigned int intc_ex_irq1_mux[] = {
+ IRQ1_MARK,
+};
+static const unsigned int intc_ex_irq2_pins[] = {
+ /* IRQ2 */
+ RCAR_GP_PIN(2, 2),
+};
+static const unsigned int intc_ex_irq2_mux[] = {
+ IRQ2_MARK,
+};
+static const unsigned int intc_ex_irq3_pins[] = {
+ /* IRQ3 */
+ RCAR_GP_PIN(2, 3),
+};
+static const unsigned int intc_ex_irq3_mux[] = {
+ IRQ3_MARK,
+};
+static const unsigned int intc_ex_irq4_pins[] = {
+ /* IRQ4 */
+ RCAR_GP_PIN(2, 4),
+};
+static const unsigned int intc_ex_irq4_mux[] = {
+ IRQ4_MARK,
+};
+static const unsigned int intc_ex_irq5_pins[] = {
+ /* IRQ5 */
+ RCAR_GP_PIN(2, 5),
+};
+static const unsigned int intc_ex_irq5_mux[] = {
+ IRQ5_MARK,
+};
+
+/* - MSIOF0 ----------------------------------------------------------------- */
+static const unsigned int msiof0_clk_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(5, 17),
+};
+static const unsigned int msiof0_clk_mux[] = {
+ MSIOF0_SCK_MARK,
+};
+static const unsigned int msiof0_sync_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(5, 18),
+};
+static const unsigned int msiof0_sync_mux[] = {
+ MSIOF0_SYNC_MARK,
+};
+static const unsigned int msiof0_ss1_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(5, 19),
+};
+static const unsigned int msiof0_ss1_mux[] = {
+ MSIOF0_SS1_MARK,
+};
+static const unsigned int msiof0_ss2_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(5, 21),
+};
+static const unsigned int msiof0_ss2_mux[] = {
+ MSIOF0_SS2_MARK,
+};
+static const unsigned int msiof0_txd_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(5, 20),
+};
+static const unsigned int msiof0_txd_mux[] = {
+ MSIOF0_TXD_MARK,
+};
+static const unsigned int msiof0_rxd_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(5, 22),
+};
+static const unsigned int msiof0_rxd_mux[] = {
+ MSIOF0_RXD_MARK,
+};
+/* - MSIOF1 ----------------------------------------------------------------- */
+static const unsigned int msiof1_clk_a_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(6, 8),
+};
+static const unsigned int msiof1_clk_a_mux[] = {
+ MSIOF1_SCK_A_MARK,
+};
+static const unsigned int msiof1_sync_a_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(6, 9),
+};
+static const unsigned int msiof1_sync_a_mux[] = {
+ MSIOF1_SYNC_A_MARK,
+};
+static const unsigned int msiof1_ss1_a_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(6, 5),
+};
+static const unsigned int msiof1_ss1_a_mux[] = {
+ MSIOF1_SS1_A_MARK,
+};
+static const unsigned int msiof1_ss2_a_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(6, 6),
+};
+static const unsigned int msiof1_ss2_a_mux[] = {
+ MSIOF1_SS2_A_MARK,
+};
+static const unsigned int msiof1_txd_a_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(6, 7),
+};
+static const unsigned int msiof1_txd_a_mux[] = {
+ MSIOF1_TXD_A_MARK,
+};
+static const unsigned int msiof1_rxd_a_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(6, 10),
+};
+static const unsigned int msiof1_rxd_a_mux[] = {
+ MSIOF1_RXD_A_MARK,
+};
+static const unsigned int msiof1_clk_b_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(5, 9),
+};
+static const unsigned int msiof1_clk_b_mux[] = {
+ MSIOF1_SCK_B_MARK,
+};
+static const unsigned int msiof1_sync_b_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(5, 3),
+};
+static const unsigned int msiof1_sync_b_mux[] = {
+ MSIOF1_SYNC_B_MARK,
+};
+static const unsigned int msiof1_ss1_b_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(5, 4),
+};
+static const unsigned int msiof1_ss1_b_mux[] = {
+ MSIOF1_SS1_B_MARK,
+};
+static const unsigned int msiof1_ss2_b_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(5, 0),
+};
+static const unsigned int msiof1_ss2_b_mux[] = {
+ MSIOF1_SS2_B_MARK,
+};
+static const unsigned int msiof1_txd_b_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(5, 8),
+};
+static const unsigned int msiof1_txd_b_mux[] = {
+ MSIOF1_TXD_B_MARK,
+};
+static const unsigned int msiof1_rxd_b_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(5, 7),
+};
+static const unsigned int msiof1_rxd_b_mux[] = {
+ MSIOF1_RXD_B_MARK,
+};
+static const unsigned int msiof1_clk_c_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(6, 17),
+};
+static const unsigned int msiof1_clk_c_mux[] = {
+ MSIOF1_SCK_C_MARK,
+};
+static const unsigned int msiof1_sync_c_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(6, 18),
+};
+static const unsigned int msiof1_sync_c_mux[] = {
+ MSIOF1_SYNC_C_MARK,
+};
+static const unsigned int msiof1_ss1_c_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(6, 21),
+};
+static const unsigned int msiof1_ss1_c_mux[] = {
+ MSIOF1_SS1_C_MARK,
+};
+static const unsigned int msiof1_ss2_c_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(6, 27),
+};
+static const unsigned int msiof1_ss2_c_mux[] = {
+ MSIOF1_SS2_C_MARK,
+};
+static const unsigned int msiof1_txd_c_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(6, 20),
+};
+static const unsigned int msiof1_txd_c_mux[] = {
+ MSIOF1_TXD_C_MARK,
+};
+static const unsigned int msiof1_rxd_c_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(6, 19),
+};
+static const unsigned int msiof1_rxd_c_mux[] = {
+ MSIOF1_RXD_C_MARK,
+};
+static const unsigned int msiof1_clk_d_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(5, 12),
+};
+static const unsigned int msiof1_clk_d_mux[] = {
+ MSIOF1_SCK_D_MARK,
+};
+static const unsigned int msiof1_sync_d_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(5, 15),
+};
+static const unsigned int msiof1_sync_d_mux[] = {
+ MSIOF1_SYNC_D_MARK,
+};
+static const unsigned int msiof1_ss1_d_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(5, 16),
+};
+static const unsigned int msiof1_ss1_d_mux[] = {
+ MSIOF1_SS1_D_MARK,
+};
+static const unsigned int msiof1_ss2_d_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(5, 21),
+};
+static const unsigned int msiof1_ss2_d_mux[] = {
+ MSIOF1_SS2_D_MARK,
+};
+static const unsigned int msiof1_txd_d_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(5, 14),
+};
+static const unsigned int msiof1_txd_d_mux[] = {
+ MSIOF1_TXD_D_MARK,
+};
+static const unsigned int msiof1_rxd_d_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(5, 13),
+};
+static const unsigned int msiof1_rxd_d_mux[] = {
+ MSIOF1_RXD_D_MARK,
+};
+static const unsigned int msiof1_clk_e_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(3, 0),
+};
+static const unsigned int msiof1_clk_e_mux[] = {
+ MSIOF1_SCK_E_MARK,
+};
+static const unsigned int msiof1_sync_e_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(3, 1),
+};
+static const unsigned int msiof1_sync_e_mux[] = {
+ MSIOF1_SYNC_E_MARK,
+};
+static const unsigned int msiof1_ss1_e_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(3, 4),
+};
+static const unsigned int msiof1_ss1_e_mux[] = {
+ MSIOF1_SS1_E_MARK,
+};
+static const unsigned int msiof1_ss2_e_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(3, 5),
+};
+static const unsigned int msiof1_ss2_e_mux[] = {
+ MSIOF1_SS2_E_MARK,
+};
+static const unsigned int msiof1_txd_e_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(3, 3),
+};
+static const unsigned int msiof1_txd_e_mux[] = {
+ MSIOF1_TXD_E_MARK,
+};
+static const unsigned int msiof1_rxd_e_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(3, 2),
+};
+static const unsigned int msiof1_rxd_e_mux[] = {
+ MSIOF1_RXD_E_MARK,
+};
+static const unsigned int msiof1_clk_f_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(5, 23),
+};
+static const unsigned int msiof1_clk_f_mux[] = {
+ MSIOF1_SCK_F_MARK,
+};
+static const unsigned int msiof1_sync_f_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(5, 24),
+};
+static const unsigned int msiof1_sync_f_mux[] = {
+ MSIOF1_SYNC_F_MARK,
+};
+static const unsigned int msiof1_ss1_f_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(6, 1),
+};
+static const unsigned int msiof1_ss1_f_mux[] = {
+ MSIOF1_SS1_F_MARK,
+};
+static const unsigned int msiof1_ss2_f_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(6, 2),
+};
+static const unsigned int msiof1_ss2_f_mux[] = {
+ MSIOF1_SS2_F_MARK,
+};
+static const unsigned int msiof1_txd_f_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(6, 0),
+};
+static const unsigned int msiof1_txd_f_mux[] = {
+ MSIOF1_TXD_F_MARK,
+};
+static const unsigned int msiof1_rxd_f_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(5, 25),
+};
+static const unsigned int msiof1_rxd_f_mux[] = {
+ MSIOF1_RXD_F_MARK,
+};
+static const unsigned int msiof1_clk_g_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(3, 6),
+};
+static const unsigned int msiof1_clk_g_mux[] = {
+ MSIOF1_SCK_G_MARK,
+};
+static const unsigned int msiof1_sync_g_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(3, 7),
+};
+static const unsigned int msiof1_sync_g_mux[] = {
+ MSIOF1_SYNC_G_MARK,
+};
+static const unsigned int msiof1_ss1_g_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(3, 10),
+};
+static const unsigned int msiof1_ss1_g_mux[] = {
+ MSIOF1_SS1_G_MARK,
+};
+static const unsigned int msiof1_ss2_g_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(3, 11),
+};
+static const unsigned int msiof1_ss2_g_mux[] = {
+ MSIOF1_SS2_G_MARK,
+};
+static const unsigned int msiof1_txd_g_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(3, 9),
+};
+static const unsigned int msiof1_txd_g_mux[] = {
+ MSIOF1_TXD_G_MARK,
+};
+static const unsigned int msiof1_rxd_g_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(3, 8),
+};
+static const unsigned int msiof1_rxd_g_mux[] = {
+ MSIOF1_RXD_G_MARK,
+};
+/* - MSIOF2 ----------------------------------------------------------------- */
+static const unsigned int msiof2_clk_a_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(1, 9),
+};
+static const unsigned int msiof2_clk_a_mux[] = {
+ MSIOF2_SCK_A_MARK,
+};
+static const unsigned int msiof2_sync_a_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(1, 8),
+};
+static const unsigned int msiof2_sync_a_mux[] = {
+ MSIOF2_SYNC_A_MARK,
+};
+static const unsigned int msiof2_ss1_a_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(1, 6),
+};
+static const unsigned int msiof2_ss1_a_mux[] = {
+ MSIOF2_SS1_A_MARK,
+};
+static const unsigned int msiof2_ss2_a_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(1, 7),
+};
+static const unsigned int msiof2_ss2_a_mux[] = {
+ MSIOF2_SS2_A_MARK,
+};
+static const unsigned int msiof2_txd_a_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(1, 11),
+};
+static const unsigned int msiof2_txd_a_mux[] = {
+ MSIOF2_TXD_A_MARK,
+};
+static const unsigned int msiof2_rxd_a_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(1, 10),
+};
+static const unsigned int msiof2_rxd_a_mux[] = {
+ MSIOF2_RXD_A_MARK,
+};
+static const unsigned int msiof2_clk_b_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(0, 4),
+};
+static const unsigned int msiof2_clk_b_mux[] = {
+ MSIOF2_SCK_B_MARK,
+};
+static const unsigned int msiof2_sync_b_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(0, 5),
+};
+static const unsigned int msiof2_sync_b_mux[] = {
+ MSIOF2_SYNC_B_MARK,
+};
+static const unsigned int msiof2_ss1_b_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(0, 0),
+};
+static const unsigned int msiof2_ss1_b_mux[] = {
+ MSIOF2_SS1_B_MARK,
+};
+static const unsigned int msiof2_ss2_b_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(0, 1),
+};
+static const unsigned int msiof2_ss2_b_mux[] = {
+ MSIOF2_SS2_B_MARK,
+};
+static const unsigned int msiof2_txd_b_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(0, 7),
+};
+static const unsigned int msiof2_txd_b_mux[] = {
+ MSIOF2_TXD_B_MARK,
+};
+static const unsigned int msiof2_rxd_b_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(0, 6),
+};
+static const unsigned int msiof2_rxd_b_mux[] = {
+ MSIOF2_RXD_B_MARK,
+};
+static const unsigned int msiof2_clk_c_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(2, 12),
+};
+static const unsigned int msiof2_clk_c_mux[] = {
+ MSIOF2_SCK_C_MARK,
+};
+static const unsigned int msiof2_sync_c_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(2, 11),
+};
+static const unsigned int msiof2_sync_c_mux[] = {
+ MSIOF2_SYNC_C_MARK,
+};
+static const unsigned int msiof2_ss1_c_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(2, 10),
+};
+static const unsigned int msiof2_ss1_c_mux[] = {
+ MSIOF2_SS1_C_MARK,
+};
+static const unsigned int msiof2_ss2_c_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(2, 9),
+};
+static const unsigned int msiof2_ss2_c_mux[] = {
+ MSIOF2_SS2_C_MARK,
+};
+static const unsigned int msiof2_txd_c_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(2, 14),
+};
+static const unsigned int msiof2_txd_c_mux[] = {
+ MSIOF2_TXD_C_MARK,
+};
+static const unsigned int msiof2_rxd_c_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(2, 13),
+};
+static const unsigned int msiof2_rxd_c_mux[] = {
+ MSIOF2_RXD_C_MARK,
+};
+static const unsigned int msiof2_clk_d_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(0, 8),
+};
+static const unsigned int msiof2_clk_d_mux[] = {
+ MSIOF2_SCK_D_MARK,
+};
+static const unsigned int msiof2_sync_d_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(0, 9),
+};
+static const unsigned int msiof2_sync_d_mux[] = {
+ MSIOF2_SYNC_D_MARK,
+};
+static const unsigned int msiof2_ss1_d_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(0, 12),
+};
+static const unsigned int msiof2_ss1_d_mux[] = {
+ MSIOF2_SS1_D_MARK,
+};
+static const unsigned int msiof2_ss2_d_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(0, 13),
+};
+static const unsigned int msiof2_ss2_d_mux[] = {
+ MSIOF2_SS2_D_MARK,
+};
+static const unsigned int msiof2_txd_d_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(0, 11),
+};
+static const unsigned int msiof2_txd_d_mux[] = {
+ MSIOF2_TXD_D_MARK,
+};
+static const unsigned int msiof2_rxd_d_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(0, 10),
+};
+static const unsigned int msiof2_rxd_d_mux[] = {
+ MSIOF2_RXD_D_MARK,
+};
+/* - MSIOF3 ----------------------------------------------------------------- */
+static const unsigned int msiof3_clk_a_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(0, 0),
+};
+static const unsigned int msiof3_clk_a_mux[] = {
+ MSIOF3_SCK_A_MARK,
+};
+static const unsigned int msiof3_sync_a_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(0, 1),
+};
+static const unsigned int msiof3_sync_a_mux[] = {
+ MSIOF3_SYNC_A_MARK,
+};
+static const unsigned int msiof3_ss1_a_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(0, 14),
+};
+static const unsigned int msiof3_ss1_a_mux[] = {
+ MSIOF3_SS1_A_MARK,
+};
+static const unsigned int msiof3_ss2_a_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(0, 15),
+};
+static const unsigned int msiof3_ss2_a_mux[] = {
+ MSIOF3_SS2_A_MARK,
+};
+static const unsigned int msiof3_txd_a_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(0, 3),
+};
+static const unsigned int msiof3_txd_a_mux[] = {
+ MSIOF3_TXD_A_MARK,
+};
+static const unsigned int msiof3_rxd_a_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(0, 2),
+};
+static const unsigned int msiof3_rxd_a_mux[] = {
+ MSIOF3_RXD_A_MARK,
+};
+static const unsigned int msiof3_clk_b_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(1, 2),
+};
+static const unsigned int msiof3_clk_b_mux[] = {
+ MSIOF3_SCK_B_MARK,
+};
+static const unsigned int msiof3_sync_b_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(1, 0),
+};
+static const unsigned int msiof3_sync_b_mux[] = {
+ MSIOF3_SYNC_B_MARK,
+};
+static const unsigned int msiof3_ss1_b_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(1, 4),
+};
+static const unsigned int msiof3_ss1_b_mux[] = {
+ MSIOF3_SS1_B_MARK,
+};
+static const unsigned int msiof3_ss2_b_pins[] = {
+ /* SS2 */
+ RCAR_GP_PIN(1, 5),
+};
+static const unsigned int msiof3_ss2_b_mux[] = {
+ MSIOF3_SS2_B_MARK,
+};
+static const unsigned int msiof3_txd_b_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(1, 1),
+};
+static const unsigned int msiof3_txd_b_mux[] = {
+ MSIOF3_TXD_B_MARK,
+};
+static const unsigned int msiof3_rxd_b_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(1, 3),
+};
+static const unsigned int msiof3_rxd_b_mux[] = {
+ MSIOF3_RXD_B_MARK,
+};
+static const unsigned int msiof3_clk_c_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(1, 12),
+};
+static const unsigned int msiof3_clk_c_mux[] = {
+ MSIOF3_SCK_C_MARK,
+};
+static const unsigned int msiof3_sync_c_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(1, 13),
+};
+static const unsigned int msiof3_sync_c_mux[] = {
+ MSIOF3_SYNC_C_MARK,
+};
+static const unsigned int msiof3_txd_c_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(1, 15),
+};
+static const unsigned int msiof3_txd_c_mux[] = {
+ MSIOF3_TXD_C_MARK,
+};
+static const unsigned int msiof3_rxd_c_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(1, 14),
+};
+static const unsigned int msiof3_rxd_c_mux[] = {
+ MSIOF3_RXD_C_MARK,
+};
+static const unsigned int msiof3_clk_d_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(1, 22),
+};
+static const unsigned int msiof3_clk_d_mux[] = {
+ MSIOF3_SCK_D_MARK,
+};
+static const unsigned int msiof3_sync_d_pins[] = {
+ /* SYNC */
+ RCAR_GP_PIN(1, 23),
+};
+static const unsigned int msiof3_sync_d_mux[] = {
+ MSIOF3_SYNC_D_MARK,
+};
+static const unsigned int msiof3_ss1_d_pins[] = {
+ /* SS1 */
+ RCAR_GP_PIN(1, 26),
+};
+static const unsigned int msiof3_ss1_d_mux[] = {
+ MSIOF3_SS1_D_MARK,
+};
+static const unsigned int msiof3_txd_d_pins[] = {
+ /* TXD */
+ RCAR_GP_PIN(1, 25),
+};
+static const unsigned int msiof3_txd_d_mux[] = {
+ MSIOF3_TXD_D_MARK,
+};
+static const unsigned int msiof3_rxd_d_pins[] = {
+ /* RXD */
+ RCAR_GP_PIN(1, 24),
+};
+static const unsigned int msiof3_rxd_d_mux[] = {
+ MSIOF3_RXD_D_MARK,
+};
+
+/* - PWM0 --------------------------------------------------------------------*/
+static const unsigned int pwm0_pins[] = {
+ /* PWM */
+ RCAR_GP_PIN(2, 6),
+};
+static const unsigned int pwm0_mux[] = {
+ PWM0_MARK,
+};
+/* - PWM1 --------------------------------------------------------------------*/
+static const unsigned int pwm1_a_pins[] = {
+ /* PWM */
+ RCAR_GP_PIN(2, 7),
+};
+static const unsigned int pwm1_a_mux[] = {
+ PWM1_A_MARK,
+};
+static const unsigned int pwm1_b_pins[] = {
+ /* PWM */
+ RCAR_GP_PIN(1, 8),
+};
+static const unsigned int pwm1_b_mux[] = {
+ PWM1_B_MARK,
+};
+/* - PWM2 --------------------------------------------------------------------*/
+static const unsigned int pwm2_a_pins[] = {
+ /* PWM */
+ RCAR_GP_PIN(2, 8),
+};
+static const unsigned int pwm2_a_mux[] = {
+ PWM2_A_MARK,
+};
+static const unsigned int pwm2_b_pins[] = {
+ /* PWM */
+ RCAR_GP_PIN(1, 11),
+};
+static const unsigned int pwm2_b_mux[] = {
+ PWM2_B_MARK,
+};
+/* - PWM3 --------------------------------------------------------------------*/
+static const unsigned int pwm3_a_pins[] = {
+ /* PWM */
+ RCAR_GP_PIN(1, 0),
+};
+static const unsigned int pwm3_a_mux[] = {
+ PWM3_A_MARK,
+};
+static const unsigned int pwm3_b_pins[] = {
+ /* PWM */
+ RCAR_GP_PIN(2, 2),
+};
+static const unsigned int pwm3_b_mux[] = {
+ PWM3_B_MARK,
+};
+/* - PWM4 --------------------------------------------------------------------*/
+static const unsigned int pwm4_a_pins[] = {
+ /* PWM */
+ RCAR_GP_PIN(1, 1),
+};
+static const unsigned int pwm4_a_mux[] = {
+ PWM4_A_MARK,
+};
+static const unsigned int pwm4_b_pins[] = {
+ /* PWM */
+ RCAR_GP_PIN(2, 3),
+};
+static const unsigned int pwm4_b_mux[] = {
+ PWM4_B_MARK,
+};
+/* - PWM5 --------------------------------------------------------------------*/
+static const unsigned int pwm5_a_pins[] = {
+ /* PWM */
+ RCAR_GP_PIN(1, 2),
+};
+static const unsigned int pwm5_a_mux[] = {
+ PWM5_A_MARK,
+};
+static const unsigned int pwm5_b_pins[] = {
+ /* PWM */
+ RCAR_GP_PIN(2, 4),
+};
+static const unsigned int pwm5_b_mux[] = {
+ PWM5_B_MARK,
+};
+/* - PWM6 --------------------------------------------------------------------*/
+static const unsigned int pwm6_a_pins[] = {
+ /* PWM */
+ RCAR_GP_PIN(1, 3),
+};
+static const unsigned int pwm6_a_mux[] = {
+ PWM6_A_MARK,
+};
+static const unsigned int pwm6_b_pins[] = {
+ /* PWM */
+ RCAR_GP_PIN(2, 5),
+};
+static const unsigned int pwm6_b_mux[] = {
+ PWM6_B_MARK,
+};
+
+/* - QSPI0 ------------------------------------------------------------------ */
+static const unsigned int qspi0_ctrl_pins[] = {
+ /* QSPI0_SPCLK, QSPI0_SSL */
+ PIN_NUMBER('W', 3), PIN_NUMBER('Y', 3),
+};
+static const unsigned int qspi0_ctrl_mux[] = {
+ QSPI0_SPCLK_MARK, QSPI0_SSL_MARK,
+};
+static const unsigned int qspi0_data2_pins[] = {
+ /* QSPI0_MOSI_IO0, QSPI0_MISO_IO1 */
+ PIN_A_NUMBER('C', 5), PIN_A_NUMBER('B', 4),
+};
+static const unsigned int qspi0_data2_mux[] = {
+ QSPI0_MOSI_IO0_MARK, QSPI0_MISO_IO1_MARK,
+};
+static const unsigned int qspi0_data4_pins[] = {
+ /* QSPI0_MOSI_IO0, QSPI0_MISO_IO1, QSPI0_IO2, QSPI0_IO3 */
+ PIN_A_NUMBER('C', 5), PIN_A_NUMBER('B', 4),
+ PIN_NUMBER('Y', 6), PIN_A_NUMBER('B', 6),
+};
+static const unsigned int qspi0_data4_mux[] = {
+ QSPI0_MOSI_IO0_MARK, QSPI0_MISO_IO1_MARK,
+ QSPI0_IO2_MARK, QSPI0_IO3_MARK,
+};
+/* - QSPI1 ------------------------------------------------------------------ */
+static const unsigned int qspi1_ctrl_pins[] = {
+ /* QSPI1_SPCLK, QSPI1_SSL */
+ PIN_NUMBER('V', 3), PIN_NUMBER('V', 5),
+};
+static const unsigned int qspi1_ctrl_mux[] = {
+ QSPI1_SPCLK_MARK, QSPI1_SSL_MARK,
+};
+static const unsigned int qspi1_data2_pins[] = {
+ /* QSPI1_MOSI_IO0, QSPI1_MISO_IO1 */
+ PIN_A_NUMBER('C', 7), PIN_A_NUMBER('E', 5),
+};
+static const unsigned int qspi1_data2_mux[] = {
+ QSPI1_MOSI_IO0_MARK, QSPI1_MISO_IO1_MARK,
+};
+static const unsigned int qspi1_data4_pins[] = {
+ /* QSPI1_MOSI_IO0, QSPI1_MISO_IO1, QSPI1_IO2, QSPI1_IO3 */
+ PIN_A_NUMBER('C', 7), PIN_A_NUMBER('E', 5),
+ PIN_A_NUMBER('E', 4), PIN_A_NUMBER('C', 3),
+};
+static const unsigned int qspi1_data4_mux[] = {
+ QSPI1_MOSI_IO0_MARK, QSPI1_MISO_IO1_MARK,
+ QSPI1_IO2_MARK, QSPI1_IO3_MARK,
+};
+
+/* - SATA --------------------------------------------------------------------*/
+static const unsigned int sata0_devslp_a_pins[] = {
+ /* DEVSLP */
+ RCAR_GP_PIN(6, 16),
+};
+static const unsigned int sata0_devslp_a_mux[] = {
+ SATA_DEVSLP_A_MARK,
+};
+static const unsigned int sata0_devslp_b_pins[] = {
+ /* DEVSLP */
+ RCAR_GP_PIN(4, 6),
+};
+static const unsigned int sata0_devslp_b_mux[] = {
+ SATA_DEVSLP_B_MARK,
+};
+
+/* - SCIF0 ------------------------------------------------------------------ */
+static const unsigned int scif0_data_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(5, 1), RCAR_GP_PIN(5, 2),
+};
+static const unsigned int scif0_data_mux[] = {
+ RX0_MARK, TX0_MARK,
+};
+static const unsigned int scif0_clk_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(5, 0),
+};
+static const unsigned int scif0_clk_mux[] = {
+ SCK0_MARK,
+};
+static const unsigned int scif0_ctrl_pins[] = {
+ /* RTS, CTS */
+ RCAR_GP_PIN(5, 4), RCAR_GP_PIN(5, 3),
+};
+static const unsigned int scif0_ctrl_mux[] = {
+ RTS0_N_TANS_MARK, CTS0_N_MARK,
+};
+/* - SCIF1 ------------------------------------------------------------------ */
+static const unsigned int scif1_data_a_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(5, 5), RCAR_GP_PIN(5, 6),
+};
+static const unsigned int scif1_data_a_mux[] = {
+ RX1_A_MARK, TX1_A_MARK,
+};
+static const unsigned int scif1_clk_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(6, 21),
+};
+static const unsigned int scif1_clk_mux[] = {
+ SCK1_MARK,
+};
+static const unsigned int scif1_ctrl_pins[] = {
+ /* RTS, CTS */
+ RCAR_GP_PIN(5, 8), RCAR_GP_PIN(5, 7),
+};
+static const unsigned int scif1_ctrl_mux[] = {
+ RTS1_N_TANS_MARK, CTS1_N_MARK,
+};
+
+static const unsigned int scif1_data_b_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(5, 24), RCAR_GP_PIN(5, 25),
+};
+static const unsigned int scif1_data_b_mux[] = {
+ RX1_B_MARK, TX1_B_MARK,
+};
+/* - SCIF2 ------------------------------------------------------------------ */
+static const unsigned int scif2_data_a_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(5, 11), RCAR_GP_PIN(5, 10),
+};
+static const unsigned int scif2_data_a_mux[] = {
+ RX2_A_MARK, TX2_A_MARK,
+};
+static const unsigned int scif2_clk_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(5, 9),
+};
+static const unsigned int scif2_clk_mux[] = {
+ SCK2_MARK,
+};
+static const unsigned int scif2_data_b_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(5, 15), RCAR_GP_PIN(5, 16),
+};
+static const unsigned int scif2_data_b_mux[] = {
+ RX2_B_MARK, TX2_B_MARK,
+};
+/* - SCIF3 ------------------------------------------------------------------ */
+static const unsigned int scif3_data_a_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(1, 23), RCAR_GP_PIN(1, 24),
+};
+static const unsigned int scif3_data_a_mux[] = {
+ RX3_A_MARK, TX3_A_MARK,
+};
+static const unsigned int scif3_clk_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(1, 22),
+};
+static const unsigned int scif3_clk_mux[] = {
+ SCK3_MARK,
+};
+static const unsigned int scif3_ctrl_pins[] = {
+ /* RTS, CTS */
+ RCAR_GP_PIN(1, 26), RCAR_GP_PIN(1, 25),
+};
+static const unsigned int scif3_ctrl_mux[] = {
+ RTS3_N_TANS_MARK, CTS3_N_MARK,
+};
+static const unsigned int scif3_data_b_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(1, 8), RCAR_GP_PIN(1, 11),
+};
+static const unsigned int scif3_data_b_mux[] = {
+ RX3_B_MARK, TX3_B_MARK,
+};
+/* - SCIF4 ------------------------------------------------------------------ */
+static const unsigned int scif4_data_a_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(2, 11), RCAR_GP_PIN(2, 12),
+};
+static const unsigned int scif4_data_a_mux[] = {
+ RX4_A_MARK, TX4_A_MARK,
+};
+static const unsigned int scif4_clk_a_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(2, 10),
+};
+static const unsigned int scif4_clk_a_mux[] = {
+ SCK4_A_MARK,
+};
+static const unsigned int scif4_ctrl_a_pins[] = {
+ /* RTS, CTS */
+ RCAR_GP_PIN(2, 14), RCAR_GP_PIN(2, 13),
+};
+static const unsigned int scif4_ctrl_a_mux[] = {
+ RTS4_N_TANS_A_MARK, CTS4_N_A_MARK,
+};
+static const unsigned int scif4_data_b_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(1, 6), RCAR_GP_PIN(1, 7),
+};
+static const unsigned int scif4_data_b_mux[] = {
+ RX4_B_MARK, TX4_B_MARK,
+};
+static const unsigned int scif4_clk_b_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(1, 5),
+};
+static const unsigned int scif4_clk_b_mux[] = {
+ SCK4_B_MARK,
+};
+static const unsigned int scif4_ctrl_b_pins[] = {
+ /* RTS, CTS */
+ RCAR_GP_PIN(1, 10), RCAR_GP_PIN(1, 9),
+};
+static const unsigned int scif4_ctrl_b_mux[] = {
+ RTS4_N_TANS_B_MARK, CTS4_N_B_MARK,
+};
+static const unsigned int scif4_data_c_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(0, 12), RCAR_GP_PIN(0, 13),
+};
+static const unsigned int scif4_data_c_mux[] = {
+ RX4_C_MARK, TX4_C_MARK,
+};
+static const unsigned int scif4_clk_c_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(0, 8),
+};
+static const unsigned int scif4_clk_c_mux[] = {
+ SCK4_C_MARK,
+};
+static const unsigned int scif4_ctrl_c_pins[] = {
+ /* RTS, CTS */
+ RCAR_GP_PIN(0, 11), RCAR_GP_PIN(0, 10),
+};
+static const unsigned int scif4_ctrl_c_mux[] = {
+ RTS4_N_TANS_C_MARK, CTS4_N_C_MARK,
+};
+/* - SCIF5 ------------------------------------------------------------------ */
+static const unsigned int scif5_data_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(5, 19), RCAR_GP_PIN(5, 21),
+};
+static const unsigned int scif5_data_mux[] = {
+ RX5_MARK, TX5_MARK,
+};
+static const unsigned int scif5_clk_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(6, 21),
+};
+static const unsigned int scif5_clk_mux[] = {
+ SCK5_MARK,
+};
+
+/* - SCIF Clock ------------------------------------------------------------- */
+static const unsigned int scif_clk_a_pins[] = {
+ /* SCIF_CLK */
+ RCAR_GP_PIN(6, 23),
+};
+static const unsigned int scif_clk_a_mux[] = {
+ SCIF_CLK_A_MARK,
+};
+static const unsigned int scif_clk_b_pins[] = {
+ /* SCIF_CLK */
+ RCAR_GP_PIN(5, 9),
+};
+static const unsigned int scif_clk_b_mux[] = {
+ SCIF_CLK_B_MARK,
+};
+
+/* - SDHI0 ------------------------------------------------------------------ */
+static const unsigned int sdhi0_data1_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(3, 2),
+};
+static const unsigned int sdhi0_data1_mux[] = {
+ SD0_DAT0_MARK,
+};
+static const unsigned int sdhi0_data4_pins[] = {
+ /* D[0:3] */
+ RCAR_GP_PIN(3, 2), RCAR_GP_PIN(3, 3),
+ RCAR_GP_PIN(3, 4), RCAR_GP_PIN(3, 5),
+};
+static const unsigned int sdhi0_data4_mux[] = {
+ SD0_DAT0_MARK, SD0_DAT1_MARK,
+ SD0_DAT2_MARK, SD0_DAT3_MARK,
+};
+static const unsigned int sdhi0_ctrl_pins[] = {
+ /* CLK, CMD */
+ RCAR_GP_PIN(3, 0), RCAR_GP_PIN(3, 1),
+};
+static const unsigned int sdhi0_ctrl_mux[] = {
+ SD0_CLK_MARK, SD0_CMD_MARK,
+};
+static const unsigned int sdhi0_cd_pins[] = {
+ /* CD */
+ RCAR_GP_PIN(3, 12),
+};
+static const unsigned int sdhi0_cd_mux[] = {
+ SD0_CD_MARK,
+};
+static const unsigned int sdhi0_wp_pins[] = {
+ /* WP */
+ RCAR_GP_PIN(3, 13),
+};
+static const unsigned int sdhi0_wp_mux[] = {
+ SD0_WP_MARK,
+};
+/* - SDHI1 ------------------------------------------------------------------ */
+static const unsigned int sdhi1_data1_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(3, 8),
+};
+static const unsigned int sdhi1_data1_mux[] = {
+ SD1_DAT0_MARK,
+};
+static const unsigned int sdhi1_data4_pins[] = {
+ /* D[0:3] */
+ RCAR_GP_PIN(3, 8), RCAR_GP_PIN(3, 9),
+ RCAR_GP_PIN(3, 10), RCAR_GP_PIN(3, 11),
+};
+static const unsigned int sdhi1_data4_mux[] = {
+ SD1_DAT0_MARK, SD1_DAT1_MARK,
+ SD1_DAT2_MARK, SD1_DAT3_MARK,
+};
+static const unsigned int sdhi1_ctrl_pins[] = {
+ /* CLK, CMD */
+ RCAR_GP_PIN(3, 6), RCAR_GP_PIN(3, 7),
+};
+static const unsigned int sdhi1_ctrl_mux[] = {
+ SD1_CLK_MARK, SD1_CMD_MARK,
+};
+static const unsigned int sdhi1_cd_pins[] = {
+ /* CD */
+ RCAR_GP_PIN(3, 14),
+};
+static const unsigned int sdhi1_cd_mux[] = {
+ SD1_CD_MARK,
+};
+static const unsigned int sdhi1_wp_pins[] = {
+ /* WP */
+ RCAR_GP_PIN(3, 15),
+};
+static const unsigned int sdhi1_wp_mux[] = {
+ SD1_WP_MARK,
+};
+/* - SDHI2 ------------------------------------------------------------------ */
+static const unsigned int sdhi2_data1_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(4, 2),
+};
+static const unsigned int sdhi2_data1_mux[] = {
+ SD2_DAT0_MARK,
+};
+static const unsigned int sdhi2_data4_pins[] = {
+ /* D[0:3] */
+ RCAR_GP_PIN(4, 2), RCAR_GP_PIN(4, 3),
+ RCAR_GP_PIN(4, 4), RCAR_GP_PIN(4, 5),
+};
+static const unsigned int sdhi2_data4_mux[] = {
+ SD2_DAT0_MARK, SD2_DAT1_MARK,
+ SD2_DAT2_MARK, SD2_DAT3_MARK,
+};
+static const unsigned int sdhi2_data8_pins[] = {
+ /* D[0:7] */
+ RCAR_GP_PIN(4, 2), RCAR_GP_PIN(4, 3),
+ RCAR_GP_PIN(4, 4), RCAR_GP_PIN(4, 5),
+ RCAR_GP_PIN(3, 8), RCAR_GP_PIN(3, 9),
+ RCAR_GP_PIN(3, 10), RCAR_GP_PIN(3, 11),
+};
+static const unsigned int sdhi2_data8_mux[] = {
+ SD2_DAT0_MARK, SD2_DAT1_MARK,
+ SD2_DAT2_MARK, SD2_DAT3_MARK,
+ SD2_DAT4_MARK, SD2_DAT5_MARK,
+ SD2_DAT6_MARK, SD2_DAT7_MARK,
+};
+static const unsigned int sdhi2_ctrl_pins[] = {
+ /* CLK, CMD */
+ RCAR_GP_PIN(4, 0), RCAR_GP_PIN(4, 1),
+};
+static const unsigned int sdhi2_ctrl_mux[] = {
+ SD2_CLK_MARK, SD2_CMD_MARK,
+};
+static const unsigned int sdhi2_cd_a_pins[] = {
+ /* CD */
+ RCAR_GP_PIN(4, 13),
+};
+static const unsigned int sdhi2_cd_a_mux[] = {
+ SD2_CD_A_MARK,
+};
+static const unsigned int sdhi2_cd_b_pins[] = {
+ /* CD */
+ RCAR_GP_PIN(5, 10),
+};
+static const unsigned int sdhi2_cd_b_mux[] = {
+ SD2_CD_B_MARK,
+};
+static const unsigned int sdhi2_wp_a_pins[] = {
+ /* WP */
+ RCAR_GP_PIN(4, 14),
+};
+static const unsigned int sdhi2_wp_a_mux[] = {
+ SD2_WP_A_MARK,
+};
+static const unsigned int sdhi2_wp_b_pins[] = {
+ /* WP */
+ RCAR_GP_PIN(5, 11),
+};
+static const unsigned int sdhi2_wp_b_mux[] = {
+ SD2_WP_B_MARK,
+};
+static const unsigned int sdhi2_ds_pins[] = {
+ /* DS */
+ RCAR_GP_PIN(4, 6),
+};
+static const unsigned int sdhi2_ds_mux[] = {
+ SD2_DS_MARK,
+};
+/* - SDHI3 ------------------------------------------------------------------ */
+static const unsigned int sdhi3_data1_pins[] = {
+ /* D0 */
+ RCAR_GP_PIN(4, 9),
+};
+static const unsigned int sdhi3_data1_mux[] = {
+ SD3_DAT0_MARK,
+};
+static const unsigned int sdhi3_data4_pins[] = {
+ /* D[0:3] */
+ RCAR_GP_PIN(4, 9), RCAR_GP_PIN(4, 10),
+ RCAR_GP_PIN(4, 11), RCAR_GP_PIN(4, 12),
+};
+static const unsigned int sdhi3_data4_mux[] = {
+ SD3_DAT0_MARK, SD3_DAT1_MARK,
+ SD3_DAT2_MARK, SD3_DAT3_MARK,
+};
+static const unsigned int sdhi3_data8_pins[] = {
+ /* D[0:7] */
+ RCAR_GP_PIN(4, 9), RCAR_GP_PIN(4, 10),
+ RCAR_GP_PIN(4, 11), RCAR_GP_PIN(4, 12),
+ RCAR_GP_PIN(4, 13), RCAR_GP_PIN(4, 14),
+ RCAR_GP_PIN(4, 15), RCAR_GP_PIN(4, 16),
+};
+static const unsigned int sdhi3_data8_mux[] = {
+ SD3_DAT0_MARK, SD3_DAT1_MARK,
+ SD3_DAT2_MARK, SD3_DAT3_MARK,
+ SD3_DAT4_MARK, SD3_DAT5_MARK,
+ SD3_DAT6_MARK, SD3_DAT7_MARK,
+};
+static const unsigned int sdhi3_ctrl_pins[] = {
+ /* CLK, CMD */
+ RCAR_GP_PIN(4, 7), RCAR_GP_PIN(4, 8),
+};
+static const unsigned int sdhi3_ctrl_mux[] = {
+ SD3_CLK_MARK, SD3_CMD_MARK,
+};
+static const unsigned int sdhi3_cd_pins[] = {
+ /* CD */
+ RCAR_GP_PIN(4, 15),
+};
+static const unsigned int sdhi3_cd_mux[] = {
+ SD3_CD_MARK,
+};
+static const unsigned int sdhi3_wp_pins[] = {
+ /* WP */
+ RCAR_GP_PIN(4, 16),
+};
+static const unsigned int sdhi3_wp_mux[] = {
+ SD3_WP_MARK,
+};
+static const unsigned int sdhi3_ds_pins[] = {
+ /* DS */
+ RCAR_GP_PIN(4, 17),
+};
+static const unsigned int sdhi3_ds_mux[] = {
+ SD3_DS_MARK,
+};
+
+/* - SSI -------------------------------------------------------------------- */
+static const unsigned int ssi0_data_pins[] = {
+ /* SDATA */
+ RCAR_GP_PIN(6, 2),
+};
+static const unsigned int ssi0_data_mux[] = {
+ SSI_SDATA0_MARK,
+};
+static const unsigned int ssi01239_ctrl_pins[] = {
+ /* SCK, WS */
+ RCAR_GP_PIN(6, 0), RCAR_GP_PIN(6, 1),
+};
+static const unsigned int ssi01239_ctrl_mux[] = {
+ SSI_SCK01239_MARK, SSI_WS01239_MARK,
+};
+static const unsigned int ssi1_data_a_pins[] = {
+ /* SDATA */
+ RCAR_GP_PIN(6, 3),
+};
+static const unsigned int ssi1_data_a_mux[] = {
+ SSI_SDATA1_A_MARK,
+};
+static const unsigned int ssi1_data_b_pins[] = {
+ /* SDATA */
+ RCAR_GP_PIN(5, 12),
+};
+static const unsigned int ssi1_data_b_mux[] = {
+ SSI_SDATA1_B_MARK,
+};
+static const unsigned int ssi1_ctrl_a_pins[] = {
+ /* SCK, WS */
+ RCAR_GP_PIN(6, 26), RCAR_GP_PIN(6, 27),
+};
+static const unsigned int ssi1_ctrl_a_mux[] = {
+ SSI_SCK1_A_MARK, SSI_WS1_A_MARK,
+};
+static const unsigned int ssi1_ctrl_b_pins[] = {
+ /* SCK, WS */
+ RCAR_GP_PIN(6, 4), RCAR_GP_PIN(6, 21),
+};
+static const unsigned int ssi1_ctrl_b_mux[] = {
+ SSI_SCK1_B_MARK, SSI_WS1_B_MARK,
+};
+static const unsigned int ssi2_data_a_pins[] = {
+ /* SDATA */
+ RCAR_GP_PIN(6, 4),
+};
+static const unsigned int ssi2_data_a_mux[] = {
+ SSI_SDATA2_A_MARK,
+};
+static const unsigned int ssi2_data_b_pins[] = {
+ /* SDATA */
+ RCAR_GP_PIN(5, 13),
+};
+static const unsigned int ssi2_data_b_mux[] = {
+ SSI_SDATA2_B_MARK,
+};
+static const unsigned int ssi2_ctrl_a_pins[] = {
+ /* SCK, WS */
+ RCAR_GP_PIN(5, 19), RCAR_GP_PIN(5, 21),
+};
+static const unsigned int ssi2_ctrl_a_mux[] = {
+ SSI_SCK2_A_MARK, SSI_WS2_A_MARK,
+};
+static const unsigned int ssi2_ctrl_b_pins[] = {
+ /* SCK, WS */
+ RCAR_GP_PIN(6, 28), RCAR_GP_PIN(6, 29),
+};
+static const unsigned int ssi2_ctrl_b_mux[] = {
+ SSI_SCK2_B_MARK, SSI_WS2_B_MARK,
+};
+static const unsigned int ssi3_data_pins[] = {
+ /* SDATA */
+ RCAR_GP_PIN(6, 7),
+};
+static const unsigned int ssi3_data_mux[] = {
+ SSI_SDATA3_MARK,
+};
+static const unsigned int ssi34_ctrl_pins[] = {
+ /* SCK, WS */
+ RCAR_GP_PIN(6, 5), RCAR_GP_PIN(6, 6),
+};
+static const unsigned int ssi34_ctrl_mux[] = {
+ SSI_SCK34_MARK, SSI_WS34_MARK,
+};
+static const unsigned int ssi4_data_pins[] = {
+ /* SDATA */
+ RCAR_GP_PIN(6, 10),
+};
+static const unsigned int ssi4_data_mux[] = {
+ SSI_SDATA4_MARK,
+};
+static const unsigned int ssi4_ctrl_pins[] = {
+ /* SCK, WS */
+ RCAR_GP_PIN(6, 8), RCAR_GP_PIN(6, 9),
+};
+static const unsigned int ssi4_ctrl_mux[] = {
+ SSI_SCK4_MARK, SSI_WS4_MARK,
+};
+static const unsigned int ssi5_data_pins[] = {
+ /* SDATA */
+ RCAR_GP_PIN(6, 13),
+};
+static const unsigned int ssi5_data_mux[] = {
+ SSI_SDATA5_MARK,
+};
+static const unsigned int ssi5_ctrl_pins[] = {
+ /* SCK, WS */
+ RCAR_GP_PIN(6, 11), RCAR_GP_PIN(6, 12),
+};
+static const unsigned int ssi5_ctrl_mux[] = {
+ SSI_SCK5_MARK, SSI_WS5_MARK,
+};
+static const unsigned int ssi6_data_pins[] = {
+ /* SDATA */
+ RCAR_GP_PIN(6, 16),
+};
+static const unsigned int ssi6_data_mux[] = {
+ SSI_SDATA6_MARK,
+};
+static const unsigned int ssi6_ctrl_pins[] = {
+ /* SCK, WS */
+ RCAR_GP_PIN(6, 14), RCAR_GP_PIN(6, 15),
+};
+static const unsigned int ssi6_ctrl_mux[] = {
+ SSI_SCK6_MARK, SSI_WS6_MARK,
+};
+static const unsigned int ssi7_data_pins[] = {
+ /* SDATA */
+ RCAR_GP_PIN(6, 19),
+};
+static const unsigned int ssi7_data_mux[] = {
+ SSI_SDATA7_MARK,
+};
+static const unsigned int ssi78_ctrl_pins[] = {
+ /* SCK, WS */
+ RCAR_GP_PIN(6, 17), RCAR_GP_PIN(6, 18),
+};
+static const unsigned int ssi78_ctrl_mux[] = {
+ SSI_SCK78_MARK, SSI_WS78_MARK,
+};
+static const unsigned int ssi8_data_pins[] = {
+ /* SDATA */
+ RCAR_GP_PIN(6, 20),
+};
+static const unsigned int ssi8_data_mux[] = {
+ SSI_SDATA8_MARK,
+};
+static const unsigned int ssi9_data_a_pins[] = {
+ /* SDATA */
+ RCAR_GP_PIN(6, 21),
+};
+static const unsigned int ssi9_data_a_mux[] = {
+ SSI_SDATA9_A_MARK,
+};
+static const unsigned int ssi9_data_b_pins[] = {
+ /* SDATA */
+ RCAR_GP_PIN(5, 14),
+};
+static const unsigned int ssi9_data_b_mux[] = {
+ SSI_SDATA9_B_MARK,
+};
+static const unsigned int ssi9_ctrl_a_pins[] = {
+ /* SCK, WS */
+ RCAR_GP_PIN(5, 15), RCAR_GP_PIN(5, 16),
+};
+static const unsigned int ssi9_ctrl_a_mux[] = {
+ SSI_SCK9_A_MARK, SSI_WS9_A_MARK,
+};
+static const unsigned int ssi9_ctrl_b_pins[] = {
+ /* SCK, WS */
+ RCAR_GP_PIN(6, 30), RCAR_GP_PIN(6, 31),
+};
+static const unsigned int ssi9_ctrl_b_mux[] = {
+ SSI_SCK9_B_MARK, SSI_WS9_B_MARK,
+};
+
+/* - USB0 ------------------------------------------------------------------- */
+static const unsigned int usb0_pins[] = {
+ /* PWEN, OVC */
+ RCAR_GP_PIN(6, 24), RCAR_GP_PIN(6, 25),
+};
+static const unsigned int usb0_mux[] = {
+ USB0_PWEN_MARK, USB0_OVC_MARK,
+};
+/* - USB1 ------------------------------------------------------------------- */
+static const unsigned int usb1_pins[] = {
+ /* PWEN, OVC */
+ RCAR_GP_PIN(6, 26), RCAR_GP_PIN(6, 27),
+};
+static const unsigned int usb1_mux[] = {
+ USB1_PWEN_MARK, USB1_OVC_MARK,
+};
+/* - USB2 ------------------------------------------------------------------- */
+static const unsigned int usb2_pins[] = {
+ /* PWEN, OVC */
+ RCAR_GP_PIN(6, 14), RCAR_GP_PIN(6, 15),
+};
+static const unsigned int usb2_mux[] = {
+ USB2_PWEN_MARK, USB2_OVC_MARK,
+};
+
+static const struct sh_pfc_pin_group pinmux_groups[] = {
+ SH_PFC_PIN_GROUP(audio_clk_a_a),
+ SH_PFC_PIN_GROUP(audio_clk_a_b),
+ SH_PFC_PIN_GROUP(audio_clk_a_c),
+ SH_PFC_PIN_GROUP(audio_clk_b_a),
+ SH_PFC_PIN_GROUP(audio_clk_b_b),
+ SH_PFC_PIN_GROUP(audio_clk_c_a),
+ SH_PFC_PIN_GROUP(audio_clk_c_b),
+ SH_PFC_PIN_GROUP(audio_clkout_a),
+ SH_PFC_PIN_GROUP(audio_clkout_b),
+ SH_PFC_PIN_GROUP(audio_clkout_c),
+ SH_PFC_PIN_GROUP(audio_clkout_d),
+ SH_PFC_PIN_GROUP(audio_clkout1_a),
+ SH_PFC_PIN_GROUP(audio_clkout1_b),
+ SH_PFC_PIN_GROUP(audio_clkout2_a),
+ SH_PFC_PIN_GROUP(audio_clkout2_b),
+ SH_PFC_PIN_GROUP(audio_clkout3_a),
+ SH_PFC_PIN_GROUP(audio_clkout3_b),
+ SH_PFC_PIN_GROUP(avb_link),
+ SH_PFC_PIN_GROUP(avb_magic),
+ SH_PFC_PIN_GROUP(avb_phy_int),
+ SH_PFC_PIN_GROUP(avb_mdc),
+ SH_PFC_PIN_GROUP(avb_mii),
+ SH_PFC_PIN_GROUP(avb_avtp_pps),
+ SH_PFC_PIN_GROUP(avb_avtp_match_a),
+ SH_PFC_PIN_GROUP(avb_avtp_capture_a),
+ SH_PFC_PIN_GROUP(avb_avtp_match_b),
+ SH_PFC_PIN_GROUP(avb_avtp_capture_b),
+ SH_PFC_PIN_GROUP(can0_data_a),
+ SH_PFC_PIN_GROUP(can0_data_b),
+ SH_PFC_PIN_GROUP(can1_data),
+ SH_PFC_PIN_GROUP(can_clk),
+ SH_PFC_PIN_GROUP(canfd0_data_a),
+ SH_PFC_PIN_GROUP(canfd0_data_b),
+ SH_PFC_PIN_GROUP(canfd1_data),
+ SH_PFC_PIN_GROUP(drif0_ctrl_a),
+ SH_PFC_PIN_GROUP(drif0_data0_a),
+ SH_PFC_PIN_GROUP(drif0_data1_a),
+ SH_PFC_PIN_GROUP(drif0_ctrl_b),
+ SH_PFC_PIN_GROUP(drif0_data0_b),
+ SH_PFC_PIN_GROUP(drif0_data1_b),
+ SH_PFC_PIN_GROUP(drif0_ctrl_c),
+ SH_PFC_PIN_GROUP(drif0_data0_c),
+ SH_PFC_PIN_GROUP(drif0_data1_c),
+ SH_PFC_PIN_GROUP(drif1_ctrl_a),
+ SH_PFC_PIN_GROUP(drif1_data0_a),
+ SH_PFC_PIN_GROUP(drif1_data1_a),
+ SH_PFC_PIN_GROUP(drif1_ctrl_b),
+ SH_PFC_PIN_GROUP(drif1_data0_b),
+ SH_PFC_PIN_GROUP(drif1_data1_b),
+ SH_PFC_PIN_GROUP(drif1_ctrl_c),
+ SH_PFC_PIN_GROUP(drif1_data0_c),
+ SH_PFC_PIN_GROUP(drif1_data1_c),
+ SH_PFC_PIN_GROUP(drif2_ctrl_a),
+ SH_PFC_PIN_GROUP(drif2_data0_a),
+ SH_PFC_PIN_GROUP(drif2_data1_a),
+ SH_PFC_PIN_GROUP(drif2_ctrl_b),
+ SH_PFC_PIN_GROUP(drif2_data0_b),
+ SH_PFC_PIN_GROUP(drif2_data1_b),
+ SH_PFC_PIN_GROUP(drif3_ctrl_a),
+ SH_PFC_PIN_GROUP(drif3_data0_a),
+ SH_PFC_PIN_GROUP(drif3_data1_a),
+ SH_PFC_PIN_GROUP(drif3_ctrl_b),
+ SH_PFC_PIN_GROUP(drif3_data0_b),
+ SH_PFC_PIN_GROUP(drif3_data1_b),
+ SH_PFC_PIN_GROUP(du_rgb666),
+ SH_PFC_PIN_GROUP(du_rgb888),
+ SH_PFC_PIN_GROUP(du_clk_out_0),
+ SH_PFC_PIN_GROUP(du_clk_out_1),
+ SH_PFC_PIN_GROUP(du_sync),
+ SH_PFC_PIN_GROUP(du_oddf),
+ SH_PFC_PIN_GROUP(du_cde),
+ SH_PFC_PIN_GROUP(du_disp),
+ SH_PFC_PIN_GROUP(hscif0_data),
+ SH_PFC_PIN_GROUP(hscif0_clk),
+ SH_PFC_PIN_GROUP(hscif0_ctrl),
+ SH_PFC_PIN_GROUP(hscif1_data_a),
+ SH_PFC_PIN_GROUP(hscif1_clk_a),
+ SH_PFC_PIN_GROUP(hscif1_ctrl_a),
+ SH_PFC_PIN_GROUP(hscif1_data_b),
+ SH_PFC_PIN_GROUP(hscif1_clk_b),
+ SH_PFC_PIN_GROUP(hscif1_ctrl_b),
+ SH_PFC_PIN_GROUP(hscif2_data_a),
+ SH_PFC_PIN_GROUP(hscif2_clk_a),
+ SH_PFC_PIN_GROUP(hscif2_ctrl_a),
+ SH_PFC_PIN_GROUP(hscif2_data_b),
+ SH_PFC_PIN_GROUP(hscif2_clk_b),
+ SH_PFC_PIN_GROUP(hscif2_ctrl_b),
+ SH_PFC_PIN_GROUP(hscif3_data_a),
+ SH_PFC_PIN_GROUP(hscif3_clk),
+ SH_PFC_PIN_GROUP(hscif3_ctrl),
+ SH_PFC_PIN_GROUP(hscif3_data_b),
+ SH_PFC_PIN_GROUP(hscif3_data_c),
+ SH_PFC_PIN_GROUP(hscif3_data_d),
+ SH_PFC_PIN_GROUP(hscif4_data_a),
+ SH_PFC_PIN_GROUP(hscif4_clk),
+ SH_PFC_PIN_GROUP(hscif4_ctrl),
+ SH_PFC_PIN_GROUP(hscif4_data_b),
+ SH_PFC_PIN_GROUP(i2c1_a),
+ SH_PFC_PIN_GROUP(i2c1_b),
+ SH_PFC_PIN_GROUP(i2c2_a),
+ SH_PFC_PIN_GROUP(i2c2_b),
+ SH_PFC_PIN_GROUP(i2c6_a),
+ SH_PFC_PIN_GROUP(i2c6_b),
+ SH_PFC_PIN_GROUP(i2c6_c),
+ SH_PFC_PIN_GROUP(intc_ex_irq0),
+ SH_PFC_PIN_GROUP(intc_ex_irq1),
+ SH_PFC_PIN_GROUP(intc_ex_irq2),
+ SH_PFC_PIN_GROUP(intc_ex_irq3),
+ SH_PFC_PIN_GROUP(intc_ex_irq4),
+ SH_PFC_PIN_GROUP(intc_ex_irq5),
+ SH_PFC_PIN_GROUP(msiof0_clk),
+ SH_PFC_PIN_GROUP(msiof0_sync),
+ SH_PFC_PIN_GROUP(msiof0_ss1),
+ SH_PFC_PIN_GROUP(msiof0_ss2),
+ SH_PFC_PIN_GROUP(msiof0_txd),
+ SH_PFC_PIN_GROUP(msiof0_rxd),
+ SH_PFC_PIN_GROUP(msiof1_clk_a),
+ SH_PFC_PIN_GROUP(msiof1_sync_a),
+ SH_PFC_PIN_GROUP(msiof1_ss1_a),
+ SH_PFC_PIN_GROUP(msiof1_ss2_a),
+ SH_PFC_PIN_GROUP(msiof1_txd_a),
+ SH_PFC_PIN_GROUP(msiof1_rxd_a),
+ SH_PFC_PIN_GROUP(msiof1_clk_b),
+ SH_PFC_PIN_GROUP(msiof1_sync_b),
+ SH_PFC_PIN_GROUP(msiof1_ss1_b),
+ SH_PFC_PIN_GROUP(msiof1_ss2_b),
+ SH_PFC_PIN_GROUP(msiof1_txd_b),
+ SH_PFC_PIN_GROUP(msiof1_rxd_b),
+ SH_PFC_PIN_GROUP(msiof1_clk_c),
+ SH_PFC_PIN_GROUP(msiof1_sync_c),
+ SH_PFC_PIN_GROUP(msiof1_ss1_c),
+ SH_PFC_PIN_GROUP(msiof1_ss2_c),
+ SH_PFC_PIN_GROUP(msiof1_txd_c),
+ SH_PFC_PIN_GROUP(msiof1_rxd_c),
+ SH_PFC_PIN_GROUP(msiof1_clk_d),
+ SH_PFC_PIN_GROUP(msiof1_sync_d),
+ SH_PFC_PIN_GROUP(msiof1_ss1_d),
+ SH_PFC_PIN_GROUP(msiof1_ss2_d),
+ SH_PFC_PIN_GROUP(msiof1_txd_d),
+ SH_PFC_PIN_GROUP(msiof1_rxd_d),
+ SH_PFC_PIN_GROUP(msiof1_clk_e),
+ SH_PFC_PIN_GROUP(msiof1_sync_e),
+ SH_PFC_PIN_GROUP(msiof1_ss1_e),
+ SH_PFC_PIN_GROUP(msiof1_ss2_e),
+ SH_PFC_PIN_GROUP(msiof1_txd_e),
+ SH_PFC_PIN_GROUP(msiof1_rxd_e),
+ SH_PFC_PIN_GROUP(msiof1_clk_f),
+ SH_PFC_PIN_GROUP(msiof1_sync_f),
+ SH_PFC_PIN_GROUP(msiof1_ss1_f),
+ SH_PFC_PIN_GROUP(msiof1_ss2_f),
+ SH_PFC_PIN_GROUP(msiof1_txd_f),
+ SH_PFC_PIN_GROUP(msiof1_rxd_f),
+ SH_PFC_PIN_GROUP(msiof1_clk_g),
+ SH_PFC_PIN_GROUP(msiof1_sync_g),
+ SH_PFC_PIN_GROUP(msiof1_ss1_g),
+ SH_PFC_PIN_GROUP(msiof1_ss2_g),
+ SH_PFC_PIN_GROUP(msiof1_txd_g),
+ SH_PFC_PIN_GROUP(msiof1_rxd_g),
+ SH_PFC_PIN_GROUP(msiof2_clk_a),
+ SH_PFC_PIN_GROUP(msiof2_sync_a),
+ SH_PFC_PIN_GROUP(msiof2_ss1_a),
+ SH_PFC_PIN_GROUP(msiof2_ss2_a),
+ SH_PFC_PIN_GROUP(msiof2_txd_a),
+ SH_PFC_PIN_GROUP(msiof2_rxd_a),
+ SH_PFC_PIN_GROUP(msiof2_clk_b),
+ SH_PFC_PIN_GROUP(msiof2_sync_b),
+ SH_PFC_PIN_GROUP(msiof2_ss1_b),
+ SH_PFC_PIN_GROUP(msiof2_ss2_b),
+ SH_PFC_PIN_GROUP(msiof2_txd_b),
+ SH_PFC_PIN_GROUP(msiof2_rxd_b),
+ SH_PFC_PIN_GROUP(msiof2_clk_c),
+ SH_PFC_PIN_GROUP(msiof2_sync_c),
+ SH_PFC_PIN_GROUP(msiof2_ss1_c),
+ SH_PFC_PIN_GROUP(msiof2_ss2_c),
+ SH_PFC_PIN_GROUP(msiof2_txd_c),
+ SH_PFC_PIN_GROUP(msiof2_rxd_c),
+ SH_PFC_PIN_GROUP(msiof2_clk_d),
+ SH_PFC_PIN_GROUP(msiof2_sync_d),
+ SH_PFC_PIN_GROUP(msiof2_ss1_d),
+ SH_PFC_PIN_GROUP(msiof2_ss2_d),
+ SH_PFC_PIN_GROUP(msiof2_txd_d),
+ SH_PFC_PIN_GROUP(msiof2_rxd_d),
+ SH_PFC_PIN_GROUP(msiof3_clk_a),
+ SH_PFC_PIN_GROUP(msiof3_sync_a),
+ SH_PFC_PIN_GROUP(msiof3_ss1_a),
+ SH_PFC_PIN_GROUP(msiof3_ss2_a),
+ SH_PFC_PIN_GROUP(msiof3_txd_a),
+ SH_PFC_PIN_GROUP(msiof3_rxd_a),
+ SH_PFC_PIN_GROUP(msiof3_clk_b),
+ SH_PFC_PIN_GROUP(msiof3_sync_b),
+ SH_PFC_PIN_GROUP(msiof3_ss1_b),
+ SH_PFC_PIN_GROUP(msiof3_ss2_b),
+ SH_PFC_PIN_GROUP(msiof3_txd_b),
+ SH_PFC_PIN_GROUP(msiof3_rxd_b),
+ SH_PFC_PIN_GROUP(msiof3_clk_c),
+ SH_PFC_PIN_GROUP(msiof3_sync_c),
+ SH_PFC_PIN_GROUP(msiof3_txd_c),
+ SH_PFC_PIN_GROUP(msiof3_rxd_c),
+ SH_PFC_PIN_GROUP(msiof3_clk_d),
+ SH_PFC_PIN_GROUP(msiof3_sync_d),
+ SH_PFC_PIN_GROUP(msiof3_ss1_d),
+ SH_PFC_PIN_GROUP(msiof3_txd_d),
+ SH_PFC_PIN_GROUP(msiof3_rxd_d),
+ SH_PFC_PIN_GROUP(pwm0),
+ SH_PFC_PIN_GROUP(pwm1_a),
+ SH_PFC_PIN_GROUP(pwm1_b),
+ SH_PFC_PIN_GROUP(pwm2_a),
+ SH_PFC_PIN_GROUP(pwm2_b),
+ SH_PFC_PIN_GROUP(pwm3_a),
+ SH_PFC_PIN_GROUP(pwm3_b),
+ SH_PFC_PIN_GROUP(pwm4_a),
+ SH_PFC_PIN_GROUP(pwm4_b),
+ SH_PFC_PIN_GROUP(pwm5_a),
+ SH_PFC_PIN_GROUP(pwm5_b),
+ SH_PFC_PIN_GROUP(pwm6_a),
+ SH_PFC_PIN_GROUP(pwm6_b),
+ SH_PFC_PIN_GROUP(qspi0_ctrl),
+ SH_PFC_PIN_GROUP(qspi0_data2),
+ SH_PFC_PIN_GROUP(qspi0_data4),
+ SH_PFC_PIN_GROUP(qspi1_ctrl),
+ SH_PFC_PIN_GROUP(qspi1_data2),
+ SH_PFC_PIN_GROUP(qspi1_data4),
+ SH_PFC_PIN_GROUP(sata0_devslp_a),
+ SH_PFC_PIN_GROUP(sata0_devslp_b),
+ SH_PFC_PIN_GROUP(scif0_data),
+ SH_PFC_PIN_GROUP(scif0_clk),
+ SH_PFC_PIN_GROUP(scif0_ctrl),
+ SH_PFC_PIN_GROUP(scif1_data_a),
+ SH_PFC_PIN_GROUP(scif1_clk),
+ SH_PFC_PIN_GROUP(scif1_ctrl),
+ SH_PFC_PIN_GROUP(scif1_data_b),
+ SH_PFC_PIN_GROUP(scif2_data_a),
+ SH_PFC_PIN_GROUP(scif2_clk),
+ SH_PFC_PIN_GROUP(scif2_data_b),
+ SH_PFC_PIN_GROUP(scif3_data_a),
+ SH_PFC_PIN_GROUP(scif3_clk),
+ SH_PFC_PIN_GROUP(scif3_ctrl),
+ SH_PFC_PIN_GROUP(scif3_data_b),
+ SH_PFC_PIN_GROUP(scif4_data_a),
+ SH_PFC_PIN_GROUP(scif4_clk_a),
+ SH_PFC_PIN_GROUP(scif4_ctrl_a),
+ SH_PFC_PIN_GROUP(scif4_data_b),
+ SH_PFC_PIN_GROUP(scif4_clk_b),
+ SH_PFC_PIN_GROUP(scif4_ctrl_b),
+ SH_PFC_PIN_GROUP(scif4_data_c),
+ SH_PFC_PIN_GROUP(scif4_clk_c),
+ SH_PFC_PIN_GROUP(scif4_ctrl_c),
+ SH_PFC_PIN_GROUP(scif5_data),
+ SH_PFC_PIN_GROUP(scif5_clk),
+ SH_PFC_PIN_GROUP(scif_clk_a),
+ SH_PFC_PIN_GROUP(scif_clk_b),
+ SH_PFC_PIN_GROUP(sdhi0_data1),
+ SH_PFC_PIN_GROUP(sdhi0_data4),
+ SH_PFC_PIN_GROUP(sdhi0_ctrl),
+ SH_PFC_PIN_GROUP(sdhi0_cd),
+ SH_PFC_PIN_GROUP(sdhi0_wp),
+ SH_PFC_PIN_GROUP(sdhi1_data1),
+ SH_PFC_PIN_GROUP(sdhi1_data4),
+ SH_PFC_PIN_GROUP(sdhi1_ctrl),
+ SH_PFC_PIN_GROUP(sdhi1_cd),
+ SH_PFC_PIN_GROUP(sdhi1_wp),
+ SH_PFC_PIN_GROUP(sdhi2_data1),
+ SH_PFC_PIN_GROUP(sdhi2_data4),
+ SH_PFC_PIN_GROUP(sdhi2_data8),
+ SH_PFC_PIN_GROUP(sdhi2_ctrl),
+ SH_PFC_PIN_GROUP(sdhi2_cd_a),
+ SH_PFC_PIN_GROUP(sdhi2_wp_a),
+ SH_PFC_PIN_GROUP(sdhi2_cd_b),
+ SH_PFC_PIN_GROUP(sdhi2_wp_b),
+ SH_PFC_PIN_GROUP(sdhi2_ds),
+ SH_PFC_PIN_GROUP(sdhi3_data1),
+ SH_PFC_PIN_GROUP(sdhi3_data4),
+ SH_PFC_PIN_GROUP(sdhi3_data8),
+ SH_PFC_PIN_GROUP(sdhi3_ctrl),
+ SH_PFC_PIN_GROUP(sdhi3_cd),
+ SH_PFC_PIN_GROUP(sdhi3_wp),
+ SH_PFC_PIN_GROUP(sdhi3_ds),
+ SH_PFC_PIN_GROUP(ssi0_data),
+ SH_PFC_PIN_GROUP(ssi01239_ctrl),
+ SH_PFC_PIN_GROUP(ssi1_data_a),
+ SH_PFC_PIN_GROUP(ssi1_data_b),
+ SH_PFC_PIN_GROUP(ssi1_ctrl_a),
+ SH_PFC_PIN_GROUP(ssi1_ctrl_b),
+ SH_PFC_PIN_GROUP(ssi2_data_a),
+ SH_PFC_PIN_GROUP(ssi2_data_b),
+ SH_PFC_PIN_GROUP(ssi2_ctrl_a),
+ SH_PFC_PIN_GROUP(ssi2_ctrl_b),
+ SH_PFC_PIN_GROUP(ssi3_data),
+ SH_PFC_PIN_GROUP(ssi34_ctrl),
+ SH_PFC_PIN_GROUP(ssi4_data),
+ SH_PFC_PIN_GROUP(ssi4_ctrl),
+ SH_PFC_PIN_GROUP(ssi5_data),
+ SH_PFC_PIN_GROUP(ssi5_ctrl),
+ SH_PFC_PIN_GROUP(ssi6_data),
+ SH_PFC_PIN_GROUP(ssi6_ctrl),
+ SH_PFC_PIN_GROUP(ssi7_data),
+ SH_PFC_PIN_GROUP(ssi78_ctrl),
+ SH_PFC_PIN_GROUP(ssi8_data),
+ SH_PFC_PIN_GROUP(ssi9_data_a),
+ SH_PFC_PIN_GROUP(ssi9_data_b),
+ SH_PFC_PIN_GROUP(ssi9_ctrl_a),
+ SH_PFC_PIN_GROUP(ssi9_ctrl_b),
+ SH_PFC_PIN_GROUP(usb0),
+ SH_PFC_PIN_GROUP(usb1),
+ SH_PFC_PIN_GROUP(usb2),
+};
+
+static const char * const audio_clk_groups[] = {
+ "audio_clk_a_a",
+ "audio_clk_a_b",
+ "audio_clk_a_c",
+ "audio_clk_b_a",
+ "audio_clk_b_b",
+ "audio_clk_c_a",
+ "audio_clk_c_b",
+ "audio_clkout_a",
+ "audio_clkout_b",
+ "audio_clkout_c",
+ "audio_clkout_d",
+ "audio_clkout1_a",
+ "audio_clkout1_b",
+ "audio_clkout2_a",
+ "audio_clkout2_b",
+ "audio_clkout3_a",
+ "audio_clkout3_b",
+};
+
+static const char * const avb_groups[] = {
+ "avb_link",
+ "avb_magic",
+ "avb_phy_int",
+ "avb_mdc",
+ "avb_mii",
+ "avb_avtp_pps",
+ "avb_avtp_match_a",
+ "avb_avtp_capture_a",
+ "avb_avtp_match_b",
+ "avb_avtp_capture_b",
+};
+
+static const char * const can0_groups[] = {
+ "can0_data_a",
+ "can0_data_b",
+};
+
+static const char * const can1_groups[] = {
+ "can1_data",
+};
+
+static const char * const can_clk_groups[] = {
+ "can_clk",
+};
+
+static const char * const canfd0_groups[] = {
+ "canfd0_data_a",
+ "canfd0_data_b",
+};
+
+static const char * const canfd1_groups[] = {
+ "canfd1_data",
+};
+
+static const char * const drif0_groups[] = {
+ "drif0_ctrl_a",
+ "drif0_data0_a",
+ "drif0_data1_a",
+ "drif0_ctrl_b",
+ "drif0_data0_b",
+ "drif0_data1_b",
+ "drif0_ctrl_c",
+ "drif0_data0_c",
+ "drif0_data1_c",
+};
+
+static const char * const drif1_groups[] = {
+ "drif1_ctrl_a",
+ "drif1_data0_a",
+ "drif1_data1_a",
+ "drif1_ctrl_b",
+ "drif1_data0_b",
+ "drif1_data1_b",
+ "drif1_ctrl_c",
+ "drif1_data0_c",
+ "drif1_data1_c",
+};
+
+static const char * const drif2_groups[] = {
+ "drif2_ctrl_a",
+ "drif2_data0_a",
+ "drif2_data1_a",
+ "drif2_ctrl_b",
+ "drif2_data0_b",
+ "drif2_data1_b",
+};
+
+static const char * const drif3_groups[] = {
+ "drif3_ctrl_a",
+ "drif3_data0_a",
+ "drif3_data1_a",
+ "drif3_ctrl_b",
+ "drif3_data0_b",
+ "drif3_data1_b",
+};
+
+static const char * const du_groups[] = {
+ "du_rgb666",
+ "du_rgb888",
+ "du_clk_out_0",
+ "du_clk_out_1",
+ "du_sync",
+ "du_oddf",
+ "du_cde",
+ "du_disp",
+};
+
+static const char * const hscif0_groups[] = {
+ "hscif0_data",
+ "hscif0_clk",
+ "hscif0_ctrl",
+};
+
+static const char * const hscif1_groups[] = {
+ "hscif1_data_a",
+ "hscif1_clk_a",
+ "hscif1_ctrl_a",
+ "hscif1_data_b",
+ "hscif1_clk_b",
+ "hscif1_ctrl_b",
+};
+
+static const char * const hscif2_groups[] = {
+ "hscif2_data_a",
+ "hscif2_clk_a",
+ "hscif2_ctrl_a",
+ "hscif2_data_b",
+ "hscif2_clk_b",
+ "hscif2_ctrl_b",
+};
+
+static const char * const hscif3_groups[] = {
+ "hscif3_data_a",
+ "hscif3_clk",
+ "hscif3_ctrl",
+ "hscif3_data_b",
+ "hscif3_data_c",
+ "hscif3_data_d",
+};
+
+static const char * const hscif4_groups[] = {
+ "hscif4_data_a",
+ "hscif4_clk",
+ "hscif4_ctrl",
+ "hscif4_data_b",
+};
+
+static const char * const i2c1_groups[] = {
+ "i2c1_a",
+ "i2c1_b",
+};
+
+static const char * const i2c2_groups[] = {
+ "i2c2_a",
+ "i2c2_b",
+};
+
+static const char * const i2c6_groups[] = {
+ "i2c6_a",
+ "i2c6_b",
+ "i2c6_c",
+};
+
+static const char * const intc_ex_groups[] = {
+ "intc_ex_irq0",
+ "intc_ex_irq1",
+ "intc_ex_irq2",
+ "intc_ex_irq3",
+ "intc_ex_irq4",
+ "intc_ex_irq5",
+};
+
+static const char * const msiof0_groups[] = {
+ "msiof0_clk",
+ "msiof0_sync",
+ "msiof0_ss1",
+ "msiof0_ss2",
+ "msiof0_txd",
+ "msiof0_rxd",
+};
+
+static const char * const msiof1_groups[] = {
+ "msiof1_clk_a",
+ "msiof1_sync_a",
+ "msiof1_ss1_a",
+ "msiof1_ss2_a",
+ "msiof1_txd_a",
+ "msiof1_rxd_a",
+ "msiof1_clk_b",
+ "msiof1_sync_b",
+ "msiof1_ss1_b",
+ "msiof1_ss2_b",
+ "msiof1_txd_b",
+ "msiof1_rxd_b",
+ "msiof1_clk_c",
+ "msiof1_sync_c",
+ "msiof1_ss1_c",
+ "msiof1_ss2_c",
+ "msiof1_txd_c",
+ "msiof1_rxd_c",
+ "msiof1_clk_d",
+ "msiof1_sync_d",
+ "msiof1_ss1_d",
+ "msiof1_ss2_d",
+ "msiof1_txd_d",
+ "msiof1_rxd_d",
+ "msiof1_clk_e",
+ "msiof1_sync_e",
+ "msiof1_ss1_e",
+ "msiof1_ss2_e",
+ "msiof1_txd_e",
+ "msiof1_rxd_e",
+ "msiof1_clk_f",
+ "msiof1_sync_f",
+ "msiof1_ss1_f",
+ "msiof1_ss2_f",
+ "msiof1_txd_f",
+ "msiof1_rxd_f",
+ "msiof1_clk_g",
+ "msiof1_sync_g",
+ "msiof1_ss1_g",
+ "msiof1_ss2_g",
+ "msiof1_txd_g",
+ "msiof1_rxd_g",
+};
+
+static const char * const msiof2_groups[] = {
+ "msiof2_clk_a",
+ "msiof2_sync_a",
+ "msiof2_ss1_a",
+ "msiof2_ss2_a",
+ "msiof2_txd_a",
+ "msiof2_rxd_a",
+ "msiof2_clk_b",
+ "msiof2_sync_b",
+ "msiof2_ss1_b",
+ "msiof2_ss2_b",
+ "msiof2_txd_b",
+ "msiof2_rxd_b",
+ "msiof2_clk_c",
+ "msiof2_sync_c",
+ "msiof2_ss1_c",
+ "msiof2_ss2_c",
+ "msiof2_txd_c",
+ "msiof2_rxd_c",
+ "msiof2_clk_d",
+ "msiof2_sync_d",
+ "msiof2_ss1_d",
+ "msiof2_ss2_d",
+ "msiof2_txd_d",
+ "msiof2_rxd_d",
+};
+
+static const char * const msiof3_groups[] = {
+ "msiof3_clk_a",
+ "msiof3_sync_a",
+ "msiof3_ss1_a",
+ "msiof3_ss2_a",
+ "msiof3_txd_a",
+ "msiof3_rxd_a",
+ "msiof3_clk_b",
+ "msiof3_sync_b",
+ "msiof3_ss1_b",
+ "msiof3_ss2_b",
+ "msiof3_txd_b",
+ "msiof3_rxd_b",
+ "msiof3_clk_c",
+ "msiof3_sync_c",
+ "msiof3_txd_c",
+ "msiof3_rxd_c",
+ "msiof3_clk_d",
+ "msiof3_sync_d",
+ "msiof3_ss1_d",
+ "msiof3_txd_d",
+ "msiof3_rxd_d",
+};
+
+static const char * const pwm0_groups[] = {
+ "pwm0",
+};
+
+static const char * const pwm1_groups[] = {
+ "pwm1_a",
+ "pwm1_b",
+};
+
+static const char * const pwm2_groups[] = {
+ "pwm2_a",
+ "pwm2_b",
+};
+
+static const char * const pwm3_groups[] = {
+ "pwm3_a",
+ "pwm3_b",
+};
+
+static const char * const pwm4_groups[] = {
+ "pwm4_a",
+ "pwm4_b",
+};
+
+static const char * const pwm5_groups[] = {
+ "pwm5_a",
+ "pwm5_b",
+};
+
+static const char * const pwm6_groups[] = {
+ "pwm6_a",
+ "pwm6_b",
+};
+
+static const char * const qspi0_groups[] = {
+ "qspi0_ctrl",
+ "qspi0_data2",
+ "qspi0_data4",
+};
+
+static const char * const qspi1_groups[] = {
+ "qspi1_ctrl",
+ "qspi1_data2",
+ "qspi1_data4",
+};
+
+static const char * const sata0_groups[] = {
+ "sata0_devslp_a",
+ "sata0_devslp_b",
+};
+
+static const char * const scif0_groups[] = {
+ "scif0_data",
+ "scif0_clk",
+ "scif0_ctrl",
+};
+
+static const char * const scif1_groups[] = {
+ "scif1_data_a",
+ "scif1_clk",
+ "scif1_ctrl",
+ "scif1_data_b",
+};
+
+static const char * const scif2_groups[] = {
+ "scif2_data_a",
+ "scif2_clk",
+ "scif2_data_b",
+};
+
+static const char * const scif3_groups[] = {
+ "scif3_data_a",
+ "scif3_clk",
+ "scif3_ctrl",
+ "scif3_data_b",
+};
+
+static const char * const scif4_groups[] = {
+ "scif4_data_a",
+ "scif4_clk_a",
+ "scif4_ctrl_a",
+ "scif4_data_b",
+ "scif4_clk_b",
+ "scif4_ctrl_b",
+ "scif4_data_c",
+ "scif4_clk_c",
+ "scif4_ctrl_c",
+};
+
+static const char * const scif5_groups[] = {
+ "scif5_data",
+ "scif5_clk",
+};
+
+static const char * const scif_clk_groups[] = {
+ "scif_clk_a",
+ "scif_clk_b",
+};
+
+static const char * const sdhi0_groups[] = {
+ "sdhi0_data1",
+ "sdhi0_data4",
+ "sdhi0_ctrl",
+ "sdhi0_cd",
+ "sdhi0_wp",
+};
+
+static const char * const sdhi1_groups[] = {
+ "sdhi1_data1",
+ "sdhi1_data4",
+ "sdhi1_ctrl",
+ "sdhi1_cd",
+ "sdhi1_wp",
+};
+
+static const char * const sdhi2_groups[] = {
+ "sdhi2_data1",
+ "sdhi2_data4",
+ "sdhi2_data8",
+ "sdhi2_ctrl",
+ "sdhi2_cd_a",
+ "sdhi2_wp_a",
+ "sdhi2_cd_b",
+ "sdhi2_wp_b",
+ "sdhi2_ds",
+};
+
+static const char * const sdhi3_groups[] = {
+ "sdhi3_data1",
+ "sdhi3_data4",
+ "sdhi3_data8",
+ "sdhi3_ctrl",
+ "sdhi3_cd",
+ "sdhi3_wp",
+ "sdhi3_ds",
+};
+
+static const char * const ssi_groups[] = {
+ "ssi0_data",
+ "ssi01239_ctrl",
+ "ssi1_data_a",
+ "ssi1_data_b",
+ "ssi1_ctrl_a",
+ "ssi1_ctrl_b",
+ "ssi2_data_a",
+ "ssi2_data_b",
+ "ssi2_ctrl_a",
+ "ssi2_ctrl_b",
+ "ssi3_data",
+ "ssi34_ctrl",
+ "ssi4_data",
+ "ssi4_ctrl",
+ "ssi5_data",
+ "ssi5_ctrl",
+ "ssi6_data",
+ "ssi6_ctrl",
+ "ssi7_data",
+ "ssi78_ctrl",
+ "ssi8_data",
+ "ssi9_data_a",
+ "ssi9_data_b",
+ "ssi9_ctrl_a",
+ "ssi9_ctrl_b",
+};
+
+static const char * const usb0_groups[] = {
+ "usb0",
+};
+
+static const char * const usb1_groups[] = {
+ "usb1",
+};
+
+static const char * const usb2_groups[] = {
+ "usb2",
+};
+
+static const struct sh_pfc_function pinmux_functions[] = {
+ SH_PFC_FUNCTION(audio_clk),
+ SH_PFC_FUNCTION(avb),
+ SH_PFC_FUNCTION(can0),
+ SH_PFC_FUNCTION(can1),
+ SH_PFC_FUNCTION(can_clk),
+ SH_PFC_FUNCTION(canfd0),
+ SH_PFC_FUNCTION(canfd1),
+ SH_PFC_FUNCTION(drif0),
+ SH_PFC_FUNCTION(drif1),
+ SH_PFC_FUNCTION(drif2),
+ SH_PFC_FUNCTION(drif3),
+ SH_PFC_FUNCTION(du),
+ SH_PFC_FUNCTION(hscif0),
+ SH_PFC_FUNCTION(hscif1),
+ SH_PFC_FUNCTION(hscif2),
+ SH_PFC_FUNCTION(hscif3),
+ SH_PFC_FUNCTION(hscif4),
+ SH_PFC_FUNCTION(i2c1),
+ SH_PFC_FUNCTION(i2c2),
+ SH_PFC_FUNCTION(i2c6),
+ SH_PFC_FUNCTION(intc_ex),
+ SH_PFC_FUNCTION(msiof0),
+ SH_PFC_FUNCTION(msiof1),
+ SH_PFC_FUNCTION(msiof2),
+ SH_PFC_FUNCTION(msiof3),
+ SH_PFC_FUNCTION(pwm0),
+ SH_PFC_FUNCTION(pwm1),
+ SH_PFC_FUNCTION(pwm2),
+ SH_PFC_FUNCTION(pwm3),
+ SH_PFC_FUNCTION(pwm4),
+ SH_PFC_FUNCTION(pwm5),
+ SH_PFC_FUNCTION(pwm6),
+ SH_PFC_FUNCTION(qspi0),
+ SH_PFC_FUNCTION(qspi1),
+ SH_PFC_FUNCTION(sata0),
+ SH_PFC_FUNCTION(scif0),
+ SH_PFC_FUNCTION(scif1),
+ SH_PFC_FUNCTION(scif2),
+ SH_PFC_FUNCTION(scif3),
+ SH_PFC_FUNCTION(scif4),
+ SH_PFC_FUNCTION(scif5),
+ SH_PFC_FUNCTION(scif_clk),
+ SH_PFC_FUNCTION(sdhi0),
+ SH_PFC_FUNCTION(sdhi1),
+ SH_PFC_FUNCTION(sdhi2),
+ SH_PFC_FUNCTION(sdhi3),
+ SH_PFC_FUNCTION(ssi),
+ SH_PFC_FUNCTION(usb0),
+ SH_PFC_FUNCTION(usb1),
+ SH_PFC_FUNCTION(usb2),
+};
+
+static const struct pinmux_cfg_reg pinmux_config_regs[] = {
+#define F_(x, y) FN_##y
+#define FM(x) FN_##x
+ { PINMUX_CFG_REG("GPSR0", 0xe6060100, 32, 1) {
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ GP_0_15_FN, GPSR0_15,
+ GP_0_14_FN, GPSR0_14,
+ GP_0_13_FN, GPSR0_13,
+ GP_0_12_FN, GPSR0_12,
+ GP_0_11_FN, GPSR0_11,
+ GP_0_10_FN, GPSR0_10,
+ GP_0_9_FN, GPSR0_9,
+ GP_0_8_FN, GPSR0_8,
+ GP_0_7_FN, GPSR0_7,
+ GP_0_6_FN, GPSR0_6,
+ GP_0_5_FN, GPSR0_5,
+ GP_0_4_FN, GPSR0_4,
+ GP_0_3_FN, GPSR0_3,
+ GP_0_2_FN, GPSR0_2,
+ GP_0_1_FN, GPSR0_1,
+ GP_0_0_FN, GPSR0_0, }
+ },
+ { PINMUX_CFG_REG("GPSR1", 0xe6060104, 32, 1) {
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ GP_1_27_FN, GPSR1_27,
+ GP_1_26_FN, GPSR1_26,
+ GP_1_25_FN, GPSR1_25,
+ GP_1_24_FN, GPSR1_24,
+ GP_1_23_FN, GPSR1_23,
+ GP_1_22_FN, GPSR1_22,
+ GP_1_21_FN, GPSR1_21,
+ GP_1_20_FN, GPSR1_20,
+ GP_1_19_FN, GPSR1_19,
+ GP_1_18_FN, GPSR1_18,
+ GP_1_17_FN, GPSR1_17,
+ GP_1_16_FN, GPSR1_16,
+ GP_1_15_FN, GPSR1_15,
+ GP_1_14_FN, GPSR1_14,
+ GP_1_13_FN, GPSR1_13,
+ GP_1_12_FN, GPSR1_12,
+ GP_1_11_FN, GPSR1_11,
+ GP_1_10_FN, GPSR1_10,
+ GP_1_9_FN, GPSR1_9,
+ GP_1_8_FN, GPSR1_8,
+ GP_1_7_FN, GPSR1_7,
+ GP_1_6_FN, GPSR1_6,
+ GP_1_5_FN, GPSR1_5,
+ GP_1_4_FN, GPSR1_4,
+ GP_1_3_FN, GPSR1_3,
+ GP_1_2_FN, GPSR1_2,
+ GP_1_1_FN, GPSR1_1,
+ GP_1_0_FN, GPSR1_0, }
+ },
+ { PINMUX_CFG_REG("GPSR2", 0xe6060108, 32, 1) {
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ GP_2_14_FN, GPSR2_14,
+ GP_2_13_FN, GPSR2_13,
+ GP_2_12_FN, GPSR2_12,
+ GP_2_11_FN, GPSR2_11,
+ GP_2_10_FN, GPSR2_10,
+ GP_2_9_FN, GPSR2_9,
+ GP_2_8_FN, GPSR2_8,
+ GP_2_7_FN, GPSR2_7,
+ GP_2_6_FN, GPSR2_6,
+ GP_2_5_FN, GPSR2_5,
+ GP_2_4_FN, GPSR2_4,
+ GP_2_3_FN, GPSR2_3,
+ GP_2_2_FN, GPSR2_2,
+ GP_2_1_FN, GPSR2_1,
+ GP_2_0_FN, GPSR2_0, }
+ },
+ { PINMUX_CFG_REG("GPSR3", 0xe606010c, 32, 1) {
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ GP_3_15_FN, GPSR3_15,
+ GP_3_14_FN, GPSR3_14,
+ GP_3_13_FN, GPSR3_13,
+ GP_3_12_FN, GPSR3_12,
+ GP_3_11_FN, GPSR3_11,
+ GP_3_10_FN, GPSR3_10,
+ GP_3_9_FN, GPSR3_9,
+ GP_3_8_FN, GPSR3_8,
+ GP_3_7_FN, GPSR3_7,
+ GP_3_6_FN, GPSR3_6,
+ GP_3_5_FN, GPSR3_5,
+ GP_3_4_FN, GPSR3_4,
+ GP_3_3_FN, GPSR3_3,
+ GP_3_2_FN, GPSR3_2,
+ GP_3_1_FN, GPSR3_1,
+ GP_3_0_FN, GPSR3_0, }
+ },
+ { PINMUX_CFG_REG("GPSR4", 0xe6060110, 32, 1) {
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ GP_4_17_FN, GPSR4_17,
+ GP_4_16_FN, GPSR4_16,
+ GP_4_15_FN, GPSR4_15,
+ GP_4_14_FN, GPSR4_14,
+ GP_4_13_FN, GPSR4_13,
+ GP_4_12_FN, GPSR4_12,
+ GP_4_11_FN, GPSR4_11,
+ GP_4_10_FN, GPSR4_10,
+ GP_4_9_FN, GPSR4_9,
+ GP_4_8_FN, GPSR4_8,
+ GP_4_7_FN, GPSR4_7,
+ GP_4_6_FN, GPSR4_6,
+ GP_4_5_FN, GPSR4_5,
+ GP_4_4_FN, GPSR4_4,
+ GP_4_3_FN, GPSR4_3,
+ GP_4_2_FN, GPSR4_2,
+ GP_4_1_FN, GPSR4_1,
+ GP_4_0_FN, GPSR4_0, }
+ },
+ { PINMUX_CFG_REG("GPSR5", 0xe6060114, 32, 1) {
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ GP_5_25_FN, GPSR5_25,
+ GP_5_24_FN, GPSR5_24,
+ GP_5_23_FN, GPSR5_23,
+ GP_5_22_FN, GPSR5_22,
+ GP_5_21_FN, GPSR5_21,
+ GP_5_20_FN, GPSR5_20,
+ GP_5_19_FN, GPSR5_19,
+ GP_5_18_FN, GPSR5_18,
+ GP_5_17_FN, GPSR5_17,
+ GP_5_16_FN, GPSR5_16,
+ GP_5_15_FN, GPSR5_15,
+ GP_5_14_FN, GPSR5_14,
+ GP_5_13_FN, GPSR5_13,
+ GP_5_12_FN, GPSR5_12,
+ GP_5_11_FN, GPSR5_11,
+ GP_5_10_FN, GPSR5_10,
+ GP_5_9_FN, GPSR5_9,
+ GP_5_8_FN, GPSR5_8,
+ GP_5_7_FN, GPSR5_7,
+ GP_5_6_FN, GPSR5_6,
+ GP_5_5_FN, GPSR5_5,
+ GP_5_4_FN, GPSR5_4,
+ GP_5_3_FN, GPSR5_3,
+ GP_5_2_FN, GPSR5_2,
+ GP_5_1_FN, GPSR5_1,
+ GP_5_0_FN, GPSR5_0, }
+ },
+ { PINMUX_CFG_REG("GPSR6", 0xe6060118, 32, 1) {
+ GP_6_31_FN, GPSR6_31,
+ GP_6_30_FN, GPSR6_30,
+ GP_6_29_FN, GPSR6_29,
+ GP_6_28_FN, GPSR6_28,
+ GP_6_27_FN, GPSR6_27,
+ GP_6_26_FN, GPSR6_26,
+ GP_6_25_FN, GPSR6_25,
+ GP_6_24_FN, GPSR6_24,
+ GP_6_23_FN, GPSR6_23,
+ GP_6_22_FN, GPSR6_22,
+ GP_6_21_FN, GPSR6_21,
+ GP_6_20_FN, GPSR6_20,
+ GP_6_19_FN, GPSR6_19,
+ GP_6_18_FN, GPSR6_18,
+ GP_6_17_FN, GPSR6_17,
+ GP_6_16_FN, GPSR6_16,
+ GP_6_15_FN, GPSR6_15,
+ GP_6_14_FN, GPSR6_14,
+ GP_6_13_FN, GPSR6_13,
+ GP_6_12_FN, GPSR6_12,
+ GP_6_11_FN, GPSR6_11,
+ GP_6_10_FN, GPSR6_10,
+ GP_6_9_FN, GPSR6_9,
+ GP_6_8_FN, GPSR6_8,
+ GP_6_7_FN, GPSR6_7,
+ GP_6_6_FN, GPSR6_6,
+ GP_6_5_FN, GPSR6_5,
+ GP_6_4_FN, GPSR6_4,
+ GP_6_3_FN, GPSR6_3,
+ GP_6_2_FN, GPSR6_2,
+ GP_6_1_FN, GPSR6_1,
+ GP_6_0_FN, GPSR6_0, }
+ },
+ { PINMUX_CFG_REG("GPSR7", 0xe606011c, 32, 1) {
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ 0, 0,
+ GP_7_3_FN, GPSR7_3,
+ GP_7_2_FN, GPSR7_2,
+ GP_7_1_FN, GPSR7_1,
+ GP_7_0_FN, GPSR7_0, }
+ },
+#undef F_
+#undef FM
+
+#define F_(x, y) x,
+#define FM(x) FN_##x,
+ { PINMUX_CFG_REG("IPSR0", 0xe6060200, 32, 4) {
+ IP0_31_28
+ IP0_27_24
+ IP0_23_20
+ IP0_19_16
+ IP0_15_12
+ IP0_11_8
+ IP0_7_4
+ IP0_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR1", 0xe6060204, 32, 4) {
+ IP1_31_28
+ IP1_27_24
+ IP1_23_20
+ IP1_19_16
+ IP1_15_12
+ IP1_11_8
+ IP1_7_4
+ IP1_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR2", 0xe6060208, 32, 4) {
+ IP2_31_28
+ IP2_27_24
+ IP2_23_20
+ IP2_19_16
+ IP2_15_12
+ IP2_11_8
+ IP2_7_4
+ IP2_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR3", 0xe606020c, 32, 4) {
+ IP3_31_28
+ IP3_27_24
+ IP3_23_20
+ IP3_19_16
+ IP3_15_12
+ IP3_11_8
+ IP3_7_4
+ IP3_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR4", 0xe6060210, 32, 4) {
+ IP4_31_28
+ IP4_27_24
+ IP4_23_20
+ IP4_19_16
+ IP4_15_12
+ IP4_11_8
+ IP4_7_4
+ IP4_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR5", 0xe6060214, 32, 4) {
+ IP5_31_28
+ IP5_27_24
+ IP5_23_20
+ IP5_19_16
+ IP5_15_12
+ IP5_11_8
+ IP5_7_4
+ IP5_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR6", 0xe6060218, 32, 4) {
+ IP6_31_28
+ IP6_27_24
+ IP6_23_20
+ IP6_19_16
+ IP6_15_12
+ IP6_11_8
+ IP6_7_4
+ IP6_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR7", 0xe606021c, 32, 4) {
+ IP7_31_28
+ IP7_27_24
+ IP7_23_20
+ IP7_19_16
+ IP7_15_12
+ IP7_11_8
+ IP7_7_4
+ IP7_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR8", 0xe6060220, 32, 4) {
+ IP8_31_28
+ IP8_27_24
+ IP8_23_20
+ IP8_19_16
+ IP8_15_12
+ IP8_11_8
+ IP8_7_4
+ IP8_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR9", 0xe6060224, 32, 4) {
+ IP9_31_28
+ IP9_27_24
+ IP9_23_20
+ IP9_19_16
+ IP9_15_12
+ IP9_11_8
+ IP9_7_4
+ IP9_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR10", 0xe6060228, 32, 4) {
+ IP10_31_28
+ IP10_27_24
+ IP10_23_20
+ IP10_19_16
+ IP10_15_12
+ IP10_11_8
+ IP10_7_4
+ IP10_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR11", 0xe606022c, 32, 4) {
+ IP11_31_28
+ IP11_27_24
+ IP11_23_20
+ IP11_19_16
+ IP11_15_12
+ IP11_11_8
+ IP11_7_4
+ IP11_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR12", 0xe6060230, 32, 4) {
+ IP12_31_28
+ IP12_27_24
+ IP12_23_20
+ IP12_19_16
+ IP12_15_12
+ IP12_11_8
+ IP12_7_4
+ IP12_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR13", 0xe6060234, 32, 4) {
+ IP13_31_28
+ IP13_27_24
+ IP13_23_20
+ IP13_19_16
+ IP13_15_12
+ IP13_11_8
+ IP13_7_4
+ IP13_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR14", 0xe6060238, 32, 4) {
+ IP14_31_28
+ IP14_27_24
+ IP14_23_20
+ IP14_19_16
+ IP14_15_12
+ IP14_11_8
+ IP14_7_4
+ IP14_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR15", 0xe606023c, 32, 4) {
+ IP15_31_28
+ IP15_27_24
+ IP15_23_20
+ IP15_19_16
+ IP15_15_12
+ IP15_11_8
+ IP15_7_4
+ IP15_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR16", 0xe6060240, 32, 4) {
+ IP16_31_28
+ IP16_27_24
+ IP16_23_20
+ IP16_19_16
+ IP16_15_12
+ IP16_11_8
+ IP16_7_4
+ IP16_3_0 }
+ },
+ { PINMUX_CFG_REG("IPSR17", 0xe6060244, 32, 4) {
+ /* IP17_31_28 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ /* IP17_27_24 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ /* IP17_23_20 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ /* IP17_19_16 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ /* IP17_15_12 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ /* IP17_11_8 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ IP17_7_4
+ IP17_3_0 }
+ },
+#undef F_
+#undef FM
+
+#define F_(x, y) x,
+#define FM(x) FN_##x,
+ { PINMUX_CFG_REG_VAR("MOD_SEL0", 0xe6060500, 32,
+ 1, 2, 2, 3, 1, 1, 2, 1, 1, 1,
+ 2, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 2, 1) {
+ 0, 0, /* RESERVED 31 */
+ MOD_SEL0_30_29
+ MOD_SEL0_28_27
+ MOD_SEL0_26_25_24
+ MOD_SEL0_23
+ MOD_SEL0_22
+ MOD_SEL0_21_20
+ MOD_SEL0_19
+ MOD_SEL0_18
+ MOD_SEL0_17
+ MOD_SEL0_16_15
+ MOD_SEL0_14
+ MOD_SEL0_13
+ MOD_SEL0_12
+ MOD_SEL0_11
+ MOD_SEL0_10
+ MOD_SEL0_9
+ MOD_SEL0_8
+ MOD_SEL0_7_6
+ MOD_SEL0_5_4
+ MOD_SEL0_3
+ MOD_SEL0_2_1
+ 0, 0, /* RESERVED 0 */ }
+ },
+ { PINMUX_CFG_REG_VAR("MOD_SEL1", 0xe6060504, 32,
+ 2, 3, 1, 2, 3, 1, 1, 2, 1,
+ 2, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1) {
+ MOD_SEL1_31_30
+ MOD_SEL1_29_28_27
+ MOD_SEL1_26
+ MOD_SEL1_25_24
+ MOD_SEL1_23_22_21
+ MOD_SEL1_20
+ MOD_SEL1_19
+ MOD_SEL1_18_17
+ MOD_SEL1_16
+ MOD_SEL1_15_14
+ MOD_SEL1_13
+ MOD_SEL1_12
+ MOD_SEL1_11
+ MOD_SEL1_10
+ MOD_SEL1_9
+ 0, 0, 0, 0, /* RESERVED 8, 7 */
+ MOD_SEL1_6
+ MOD_SEL1_5
+ MOD_SEL1_4
+ MOD_SEL1_3
+ MOD_SEL1_2
+ MOD_SEL1_1
+ MOD_SEL1_0 }
+ },
+ { PINMUX_CFG_REG_VAR("MOD_SEL2", 0xe6060508, 32,
+ 1, 1, 1, 1, 4, 4, 4,
+ 4, 4, 4, 1, 2, 1) {
+ MOD_SEL2_31
+ MOD_SEL2_30
+ MOD_SEL2_29
+ /* RESERVED 28 */
+ 0, 0,
+ /* RESERVED 27, 26, 25, 24 */
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ /* RESERVED 23, 22, 21, 20 */
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ /* RESERVED 19, 18, 17, 16 */
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ /* RESERVED 15, 14, 13, 12 */
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ /* RESERVED 11, 10, 9, 8 */
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ /* RESERVED 7, 6, 5, 4 */
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ /* RESERVED 3 */
+ 0, 0,
+ /* RESERVED 2, 1 */
+ 0, 0, 0, 0,
+ MOD_SEL2_0 }
+ },
+ { },
+};
+
+static const struct pinmux_drive_reg pinmux_drive_regs[] = {
+ { PINMUX_DRIVE_REG("DRVCTRL0", 0xe6060300) {
+ { PIN_NUMBER('W', 3), 28, 2 }, /* QSPI0_SPCLK */
+ { PIN_A_NUMBER('C', 5), 24, 2 }, /* QSPI0_MOSI_IO0 */
+ { PIN_A_NUMBER('B', 4), 20, 2 }, /* QSPI0_MISO_IO1 */
+ { PIN_NUMBER('Y', 6), 16, 2 }, /* QSPI0_IO2 */
+ { PIN_A_NUMBER('B', 6), 12, 2 }, /* QSPI0_IO3 */
+ { PIN_NUMBER('Y', 3), 8, 2 }, /* QSPI0_SSL */
+ { PIN_NUMBER('V', 3), 4, 2 }, /* QSPI1_SPCLK */
+ { PIN_A_NUMBER('C', 7), 0, 2 }, /* QSPI1_MOSI_IO0 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL1", 0xe6060304) {
+ { PIN_A_NUMBER('E', 5), 28, 2 }, /* QSPI1_MISO_IO1 */
+ { PIN_A_NUMBER('E', 4), 24, 2 }, /* QSPI1_IO2 */
+ { PIN_A_NUMBER('C', 3), 20, 2 }, /* QSPI1_IO3 */
+ { PIN_NUMBER('V', 5), 16, 2 }, /* QSPI1_SSL */
+ { PIN_NUMBER('Y', 7), 12, 2 }, /* RPC_INT# */
+ { PIN_NUMBER('V', 6), 8, 2 }, /* RPC_WP# */
+ { PIN_NUMBER('V', 7), 4, 2 }, /* RPC_RESET# */
+ { PIN_NUMBER('A', 16), 0, 3 }, /* AVB_RX_CTL */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL2", 0xe6060308) {
+ { PIN_NUMBER('B', 19), 28, 3 }, /* AVB_RXC */
+ { PIN_NUMBER('A', 13), 24, 3 }, /* AVB_RD0 */
+ { PIN_NUMBER('B', 13), 20, 3 }, /* AVB_RD1 */
+ { PIN_NUMBER('A', 14), 16, 3 }, /* AVB_RD2 */
+ { PIN_NUMBER('B', 14), 12, 3 }, /* AVB_RD3 */
+ { PIN_NUMBER('A', 8), 8, 3 }, /* AVB_TX_CTL */
+ { PIN_NUMBER('A', 19), 4, 3 }, /* AVB_TXC */
+ { PIN_NUMBER('A', 18), 0, 3 }, /* AVB_TD0 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL3", 0xe606030c) {
+ { PIN_NUMBER('B', 18), 28, 3 }, /* AVB_TD1 */
+ { PIN_NUMBER('A', 17), 24, 3 }, /* AVB_TD2 */
+ { PIN_NUMBER('B', 17), 20, 3 }, /* AVB_TD3 */
+ { PIN_NUMBER('A', 12), 16, 3 }, /* AVB_TXCREFCLK */
+ { PIN_NUMBER('A', 9), 12, 3 }, /* AVB_MDIO */
+ { RCAR_GP_PIN(2, 9), 8, 3 }, /* AVB_MDC */
+ { RCAR_GP_PIN(2, 10), 4, 3 }, /* AVB_MAGIC */
+ { RCAR_GP_PIN(2, 11), 0, 3 }, /* AVB_PHY_INT */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL4", 0xe6060310) {
+ { RCAR_GP_PIN(2, 12), 28, 3 }, /* AVB_LINK */
+ { RCAR_GP_PIN(2, 13), 24, 3 }, /* AVB_AVTP_MATCH */
+ { RCAR_GP_PIN(2, 14), 20, 3 }, /* AVB_AVTP_CAPTURE */
+ { RCAR_GP_PIN(2, 0), 16, 3 }, /* IRQ0 */
+ { RCAR_GP_PIN(2, 1), 12, 3 }, /* IRQ1 */
+ { RCAR_GP_PIN(2, 2), 8, 3 }, /* IRQ2 */
+ { RCAR_GP_PIN(2, 3), 4, 3 }, /* IRQ3 */
+ { RCAR_GP_PIN(2, 4), 0, 3 }, /* IRQ4 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL5", 0xe6060314) {
+ { RCAR_GP_PIN(2, 5), 28, 3 }, /* IRQ5 */
+ { RCAR_GP_PIN(2, 6), 24, 3 }, /* PWM0 */
+ { RCAR_GP_PIN(2, 7), 20, 3 }, /* PWM1 */
+ { RCAR_GP_PIN(2, 8), 16, 3 }, /* PWM2 */
+ { RCAR_GP_PIN(1, 0), 12, 3 }, /* A0 */
+ { RCAR_GP_PIN(1, 1), 8, 3 }, /* A1 */
+ { RCAR_GP_PIN(1, 2), 4, 3 }, /* A2 */
+ { RCAR_GP_PIN(1, 3), 0, 3 }, /* A3 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL6", 0xe6060318) {
+ { RCAR_GP_PIN(1, 4), 28, 3 }, /* A4 */
+ { RCAR_GP_PIN(1, 5), 24, 3 }, /* A5 */
+ { RCAR_GP_PIN(1, 6), 20, 3 }, /* A6 */
+ { RCAR_GP_PIN(1, 7), 16, 3 }, /* A7 */
+ { RCAR_GP_PIN(1, 8), 12, 3 }, /* A8 */
+ { RCAR_GP_PIN(1, 9), 8, 3 }, /* A9 */
+ { RCAR_GP_PIN(1, 10), 4, 3 }, /* A10 */
+ { RCAR_GP_PIN(1, 11), 0, 3 }, /* A11 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL7", 0xe606031c) {
+ { RCAR_GP_PIN(1, 12), 28, 3 }, /* A12 */
+ { RCAR_GP_PIN(1, 13), 24, 3 }, /* A13 */
+ { RCAR_GP_PIN(1, 14), 20, 3 }, /* A14 */
+ { RCAR_GP_PIN(1, 15), 16, 3 }, /* A15 */
+ { RCAR_GP_PIN(1, 16), 12, 3 }, /* A16 */
+ { RCAR_GP_PIN(1, 17), 8, 3 }, /* A17 */
+ { RCAR_GP_PIN(1, 18), 4, 3 }, /* A18 */
+ { RCAR_GP_PIN(1, 19), 0, 3 }, /* A19 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL8", 0xe6060320) {
+ { PIN_NUMBER('F', 1), 28, 3 }, /* CLKOUT */
+ { RCAR_GP_PIN(1, 20), 24, 3 }, /* CS0 */
+ { RCAR_GP_PIN(1, 21), 20, 3 }, /* CS1_A26 */
+ { RCAR_GP_PIN(1, 22), 16, 3 }, /* BS */
+ { RCAR_GP_PIN(1, 23), 12, 3 }, /* RD */
+ { RCAR_GP_PIN(1, 24), 8, 3 }, /* RD_WR */
+ { RCAR_GP_PIN(1, 25), 4, 3 }, /* WE0 */
+ { RCAR_GP_PIN(1, 26), 0, 3 }, /* WE1 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL9", 0xe6060324) {
+ { RCAR_GP_PIN(1, 27), 28, 3 }, /* EX_WAIT0 */
+ { PIN_NUMBER('C', 1), 24, 3 }, /* PRESETOUT# */
+ { RCAR_GP_PIN(0, 0), 20, 3 }, /* D0 */
+ { RCAR_GP_PIN(0, 1), 16, 3 }, /* D1 */
+ { RCAR_GP_PIN(0, 2), 12, 3 }, /* D2 */
+ { RCAR_GP_PIN(0, 3), 8, 3 }, /* D3 */
+ { RCAR_GP_PIN(0, 4), 4, 3 }, /* D4 */
+ { RCAR_GP_PIN(0, 5), 0, 3 }, /* D5 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL10", 0xe6060328) {
+ { RCAR_GP_PIN(0, 6), 28, 3 }, /* D6 */
+ { RCAR_GP_PIN(0, 7), 24, 3 }, /* D7 */
+ { RCAR_GP_PIN(0, 8), 20, 3 }, /* D8 */
+ { RCAR_GP_PIN(0, 9), 16, 3 }, /* D9 */
+ { RCAR_GP_PIN(0, 10), 12, 3 }, /* D10 */
+ { RCAR_GP_PIN(0, 11), 8, 3 }, /* D11 */
+ { RCAR_GP_PIN(0, 12), 4, 3 }, /* D12 */
+ { RCAR_GP_PIN(0, 13), 0, 3 }, /* D13 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL11", 0xe606032c) {
+ { RCAR_GP_PIN(0, 14), 28, 3 }, /* D14 */
+ { RCAR_GP_PIN(0, 15), 24, 3 }, /* D15 */
+ { RCAR_GP_PIN(7, 0), 20, 3 }, /* AVS1 */
+ { RCAR_GP_PIN(7, 1), 16, 3 }, /* AVS2 */
+ { RCAR_GP_PIN(7, 2), 12, 3 }, /* HDMI0_CEC */
+ { RCAR_GP_PIN(7, 3), 8, 3 }, /* HDMI1_CEC */
+ { PIN_A_NUMBER('P', 7), 4, 2 }, /* DU_DOTCLKIN0 */
+ { PIN_A_NUMBER('P', 8), 0, 2 }, /* DU_DOTCLKIN1 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL12", 0xe6060330) {
+ { PIN_A_NUMBER('R', 7), 28, 2 }, /* DU_DOTCLKIN2 */
+ { PIN_A_NUMBER('R', 8), 24, 2 }, /* DU_DOTCLKIN3 */
+ { PIN_A_NUMBER('D', 38), 20, 2 }, /* FSCLKST# */
+ { PIN_A_NUMBER('R', 30), 4, 2 }, /* TMS */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL13", 0xe6060334) {
+ { PIN_A_NUMBER('T', 28), 28, 2 }, /* TDO */
+ { PIN_A_NUMBER('T', 30), 24, 2 }, /* ASEBRK */
+ { RCAR_GP_PIN(3, 0), 20, 3 }, /* SD0_CLK */
+ { RCAR_GP_PIN(3, 1), 16, 3 }, /* SD0_CMD */
+ { RCAR_GP_PIN(3, 2), 12, 3 }, /* SD0_DAT0 */
+ { RCAR_GP_PIN(3, 3), 8, 3 }, /* SD0_DAT1 */
+ { RCAR_GP_PIN(3, 4), 4, 3 }, /* SD0_DAT2 */
+ { RCAR_GP_PIN(3, 5), 0, 3 }, /* SD0_DAT3 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL14", 0xe6060338) {
+ { RCAR_GP_PIN(3, 6), 28, 3 }, /* SD1_CLK */
+ { RCAR_GP_PIN(3, 7), 24, 3 }, /* SD1_CMD */
+ { RCAR_GP_PIN(3, 8), 20, 3 }, /* SD1_DAT0 */
+ { RCAR_GP_PIN(3, 9), 16, 3 }, /* SD1_DAT1 */
+ { RCAR_GP_PIN(3, 10), 12, 3 }, /* SD1_DAT2 */
+ { RCAR_GP_PIN(3, 11), 8, 3 }, /* SD1_DAT3 */
+ { RCAR_GP_PIN(4, 0), 4, 3 }, /* SD2_CLK */
+ { RCAR_GP_PIN(4, 1), 0, 3 }, /* SD2_CMD */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL15", 0xe606033c) {
+ { RCAR_GP_PIN(4, 2), 28, 3 }, /* SD2_DAT0 */
+ { RCAR_GP_PIN(4, 3), 24, 3 }, /* SD2_DAT1 */
+ { RCAR_GP_PIN(4, 4), 20, 3 }, /* SD2_DAT2 */
+ { RCAR_GP_PIN(4, 5), 16, 3 }, /* SD2_DAT3 */
+ { RCAR_GP_PIN(4, 6), 12, 3 }, /* SD2_DS */
+ { RCAR_GP_PIN(4, 7), 8, 3 }, /* SD3_CLK */
+ { RCAR_GP_PIN(4, 8), 4, 3 }, /* SD3_CMD */
+ { RCAR_GP_PIN(4, 9), 0, 3 }, /* SD3_DAT0 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL16", 0xe6060340) {
+ { RCAR_GP_PIN(4, 10), 28, 3 }, /* SD3_DAT1 */
+ { RCAR_GP_PIN(4, 11), 24, 3 }, /* SD3_DAT2 */
+ { RCAR_GP_PIN(4, 12), 20, 3 }, /* SD3_DAT3 */
+ { RCAR_GP_PIN(4, 13), 16, 3 }, /* SD3_DAT4 */
+ { RCAR_GP_PIN(4, 14), 12, 3 }, /* SD3_DAT5 */
+ { RCAR_GP_PIN(4, 15), 8, 3 }, /* SD3_DAT6 */
+ { RCAR_GP_PIN(4, 16), 4, 3 }, /* SD3_DAT7 */
+ { RCAR_GP_PIN(4, 17), 0, 3 }, /* SD3_DS */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL17", 0xe6060344) {
+ { RCAR_GP_PIN(3, 12), 28, 3 }, /* SD0_CD */
+ { RCAR_GP_PIN(3, 13), 24, 3 }, /* SD0_WP */
+ { RCAR_GP_PIN(3, 14), 20, 3 }, /* SD1_CD */
+ { RCAR_GP_PIN(3, 15), 16, 3 }, /* SD1_WP */
+ { RCAR_GP_PIN(5, 0), 12, 3 }, /* SCK0 */
+ { RCAR_GP_PIN(5, 1), 8, 3 }, /* RX0 */
+ { RCAR_GP_PIN(5, 2), 4, 3 }, /* TX0 */
+ { RCAR_GP_PIN(5, 3), 0, 3 }, /* CTS0 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL18", 0xe6060348) {
+ { RCAR_GP_PIN(5, 4), 28, 3 }, /* RTS0_TANS */
+ { RCAR_GP_PIN(5, 5), 24, 3 }, /* RX1 */
+ { RCAR_GP_PIN(5, 6), 20, 3 }, /* TX1 */
+ { RCAR_GP_PIN(5, 7), 16, 3 }, /* CTS1 */
+ { RCAR_GP_PIN(5, 8), 12, 3 }, /* RTS1_TANS */
+ { RCAR_GP_PIN(5, 9), 8, 3 }, /* SCK2 */
+ { RCAR_GP_PIN(5, 10), 4, 3 }, /* TX2 */
+ { RCAR_GP_PIN(5, 11), 0, 3 }, /* RX2 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL19", 0xe606034c) {
+ { RCAR_GP_PIN(5, 12), 28, 3 }, /* HSCK0 */
+ { RCAR_GP_PIN(5, 13), 24, 3 }, /* HRX0 */
+ { RCAR_GP_PIN(5, 14), 20, 3 }, /* HTX0 */
+ { RCAR_GP_PIN(5, 15), 16, 3 }, /* HCTS0 */
+ { RCAR_GP_PIN(5, 16), 12, 3 }, /* HRTS0 */
+ { RCAR_GP_PIN(5, 17), 8, 3 }, /* MSIOF0_SCK */
+ { RCAR_GP_PIN(5, 18), 4, 3 }, /* MSIOF0_SYNC */
+ { RCAR_GP_PIN(5, 19), 0, 3 }, /* MSIOF0_SS1 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL20", 0xe6060350) {
+ { RCAR_GP_PIN(5, 20), 28, 3 }, /* MSIOF0_TXD */
+ { RCAR_GP_PIN(5, 21), 24, 3 }, /* MSIOF0_SS2 */
+ { RCAR_GP_PIN(5, 22), 20, 3 }, /* MSIOF0_RXD */
+ { RCAR_GP_PIN(5, 23), 16, 3 }, /* MLB_CLK */
+ { RCAR_GP_PIN(5, 24), 12, 3 }, /* MLB_SIG */
+ { RCAR_GP_PIN(5, 25), 8, 3 }, /* MLB_DAT */
+ { PIN_NUMBER('H', 37), 4, 3 }, /* MLB_REF */
+ { RCAR_GP_PIN(6, 0), 0, 3 }, /* SSI_SCK01239 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL21", 0xe6060354) {
+ { RCAR_GP_PIN(6, 1), 28, 3 }, /* SSI_WS01239 */
+ { RCAR_GP_PIN(6, 2), 24, 3 }, /* SSI_SDATA0 */
+ { RCAR_GP_PIN(6, 3), 20, 3 }, /* SSI_SDATA1 */
+ { RCAR_GP_PIN(6, 4), 16, 3 }, /* SSI_SDATA2 */
+ { RCAR_GP_PIN(6, 5), 12, 3 }, /* SSI_SCK34 */
+ { RCAR_GP_PIN(6, 6), 8, 3 }, /* SSI_WS34 */
+ { RCAR_GP_PIN(6, 7), 4, 3 }, /* SSI_SDATA3 */
+ { RCAR_GP_PIN(6, 8), 0, 3 }, /* SSI_SCK4 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL22", 0xe6060358) {
+ { RCAR_GP_PIN(6, 9), 28, 3 }, /* SSI_WS4 */
+ { RCAR_GP_PIN(6, 10), 24, 3 }, /* SSI_SDATA4 */
+ { RCAR_GP_PIN(6, 11), 20, 3 }, /* SSI_SCK5 */
+ { RCAR_GP_PIN(6, 12), 16, 3 }, /* SSI_WS5 */
+ { RCAR_GP_PIN(6, 13), 12, 3 }, /* SSI_SDATA5 */
+ { RCAR_GP_PIN(6, 14), 8, 3 }, /* SSI_SCK6 */
+ { RCAR_GP_PIN(6, 15), 4, 3 }, /* SSI_WS6 */
+ { RCAR_GP_PIN(6, 16), 0, 3 }, /* SSI_SDATA6 */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL23", 0xe606035c) {
+ { RCAR_GP_PIN(6, 17), 28, 3 }, /* SSI_SCK78 */
+ { RCAR_GP_PIN(6, 18), 24, 3 }, /* SSI_WS78 */
+ { RCAR_GP_PIN(6, 19), 20, 3 }, /* SSI_SDATA7 */
+ { RCAR_GP_PIN(6, 20), 16, 3 }, /* SSI_SDATA8 */
+ { RCAR_GP_PIN(6, 21), 12, 3 }, /* SSI_SDATA9 */
+ { RCAR_GP_PIN(6, 22), 8, 3 }, /* AUDIO_CLKA */
+ { RCAR_GP_PIN(6, 23), 4, 3 }, /* AUDIO_CLKB */
+ { RCAR_GP_PIN(6, 24), 0, 3 }, /* USB0_PWEN */
+ } },
+ { PINMUX_DRIVE_REG("DRVCTRL24", 0xe6060360) {
+ { RCAR_GP_PIN(6, 25), 28, 3 }, /* USB0_OVC */
+ { RCAR_GP_PIN(6, 26), 24, 3 }, /* USB1_PWEN */
+ { RCAR_GP_PIN(6, 27), 20, 3 }, /* USB1_OVC */
+ { RCAR_GP_PIN(6, 28), 16, 3 }, /* USB30_PWEN */
+ { RCAR_GP_PIN(6, 29), 12, 3 }, /* USB30_OVC */
+ { RCAR_GP_PIN(6, 30), 8, 3 }, /* USB31_PWEN */
+ { RCAR_GP_PIN(6, 31), 4, 3 }, /* USB31_OVC */
+ } },
+ { },
+};
+
+static int r8a7795es1_pin_to_pocctrl(struct sh_pfc *pfc, unsigned int pin,
+ u32 *pocctrl)
+{
+ int bit = -EINVAL;
+
+ *pocctrl = 0xe6060380;
+
+ if (pin >= RCAR_GP_PIN(3, 0) && pin <= RCAR_GP_PIN(3, 11))
+ bit = pin & 0x1f;
+
+ if (pin >= RCAR_GP_PIN(4, 0) && pin <= RCAR_GP_PIN(4, 17))
+ bit = (pin & 0x1f) + 12;
+
+ return bit;
+}
+
+#define PUEN 0xe6060400
+#define PUD 0xe6060440
+
+#define PU0 0x00
+#define PU1 0x04
+#define PU2 0x08
+#define PU3 0x0c
+#define PU4 0x10
+#define PU5 0x14
+#define PU6 0x18
+
+static const struct sh_pfc_bias_info bias_info[] = {
+ { RCAR_GP_PIN(2, 11), PU0, 31 }, /* AVB_PHY_INT */
+ { RCAR_GP_PIN(2, 10), PU0, 30 }, /* AVB_MAGIC */
+ { RCAR_GP_PIN(2, 9), PU0, 29 }, /* AVB_MDC */
+ { PIN_NUMBER('A', 9), PU0, 28 }, /* AVB_MDIO */
+ { PIN_NUMBER('A', 12), PU0, 27 }, /* AVB_TXCREFCLK */
+ { PIN_NUMBER('B', 17), PU0, 26 }, /* AVB_TD3 */
+ { PIN_NUMBER('A', 17), PU0, 25 }, /* AVB_TD2 */
+ { PIN_NUMBER('B', 18), PU0, 24 }, /* AVB_TD1 */
+ { PIN_NUMBER('A', 18), PU0, 23 }, /* AVB_TD0 */
+ { PIN_NUMBER('A', 19), PU0, 22 }, /* AVB_TXC */
+ { PIN_NUMBER('A', 8), PU0, 21 }, /* AVB_TX_CTL */
+ { PIN_NUMBER('B', 14), PU0, 20 }, /* AVB_RD3 */
+ { PIN_NUMBER('A', 14), PU0, 19 }, /* AVB_RD2 */
+ { PIN_NUMBER('B', 13), PU0, 18 }, /* AVB_RD1 */
+ { PIN_NUMBER('A', 13), PU0, 17 }, /* AVB_RD0 */
+ { PIN_NUMBER('B', 19), PU0, 16 }, /* AVB_RXC */
+ { PIN_NUMBER('A', 16), PU0, 15 }, /* AVB_RX_CTL */
+ { PIN_NUMBER('V', 7), PU0, 14 }, /* RPC_RESET# */
+ { PIN_NUMBER('V', 6), PU0, 13 }, /* RPC_WP# */
+ { PIN_NUMBER('Y', 7), PU0, 12 }, /* RPC_INT# */
+ { PIN_NUMBER('V', 5), PU0, 11 }, /* QSPI1_SSL */
+ { PIN_A_NUMBER('C', 3), PU0, 10 }, /* QSPI1_IO3 */
+ { PIN_A_NUMBER('E', 4), PU0, 9 }, /* QSPI1_IO2 */
+ { PIN_A_NUMBER('E', 5), PU0, 8 }, /* QSPI1_MISO_IO1 */
+ { PIN_A_NUMBER('C', 7), PU0, 7 }, /* QSPI1_MOSI_IO0 */
+ { PIN_NUMBER('V', 3), PU0, 6 }, /* QSPI1_SPCLK */
+ { PIN_NUMBER('Y', 3), PU0, 5 }, /* QSPI0_SSL */
+ { PIN_A_NUMBER('B', 6), PU0, 4 }, /* QSPI0_IO3 */
+ { PIN_NUMBER('Y', 6), PU0, 3 }, /* QSPI0_IO2 */
+ { PIN_A_NUMBER('B', 4), PU0, 2 }, /* QSPI0_MISO_IO1 */
+ { PIN_A_NUMBER('C', 5), PU0, 1 }, /* QSPI0_MOSI_IO0 */
+ { PIN_NUMBER('W', 3), PU0, 0 }, /* QSPI0_SPCLK */
+
+ { RCAR_GP_PIN(1, 19), PU1, 31 }, /* A19 */
+ { RCAR_GP_PIN(1, 18), PU1, 30 }, /* A18 */
+ { RCAR_GP_PIN(1, 17), PU1, 29 }, /* A17 */
+ { RCAR_GP_PIN(1, 16), PU1, 28 }, /* A16 */
+ { RCAR_GP_PIN(1, 15), PU1, 27 }, /* A15 */
+ { RCAR_GP_PIN(1, 14), PU1, 26 }, /* A14 */
+ { RCAR_GP_PIN(1, 13), PU1, 25 }, /* A13 */
+ { RCAR_GP_PIN(1, 12), PU1, 24 }, /* A12 */
+ { RCAR_GP_PIN(1, 11), PU1, 23 }, /* A11 */
+ { RCAR_GP_PIN(1, 10), PU1, 22 }, /* A10 */
+ { RCAR_GP_PIN(1, 9), PU1, 21 }, /* A9 */
+ { RCAR_GP_PIN(1, 8), PU1, 20 }, /* A8 */
+ { RCAR_GP_PIN(1, 7), PU1, 19 }, /* A7 */
+ { RCAR_GP_PIN(1, 6), PU1, 18 }, /* A6 */
+ { RCAR_GP_PIN(1, 5), PU1, 17 }, /* A5 */
+ { RCAR_GP_PIN(1, 4), PU1, 16 }, /* A4 */
+ { RCAR_GP_PIN(1, 3), PU1, 15 }, /* A3 */
+ { RCAR_GP_PIN(1, 2), PU1, 14 }, /* A2 */
+ { RCAR_GP_PIN(1, 1), PU1, 13 }, /* A1 */
+ { RCAR_GP_PIN(1, 0), PU1, 12 }, /* A0 */
+ { RCAR_GP_PIN(2, 8), PU1, 11 }, /* PWM2_A */
+ { RCAR_GP_PIN(2, 7), PU1, 10 }, /* PWM1_A */
+ { RCAR_GP_PIN(2, 6), PU1, 9 }, /* PWM0 */
+ { RCAR_GP_PIN(2, 5), PU1, 8 }, /* IRQ5 */
+ { RCAR_GP_PIN(2, 4), PU1, 7 }, /* IRQ4 */
+ { RCAR_GP_PIN(2, 3), PU1, 6 }, /* IRQ3 */
+ { RCAR_GP_PIN(2, 2), PU1, 5 }, /* IRQ2 */
+ { RCAR_GP_PIN(2, 1), PU1, 4 }, /* IRQ1 */
+ { RCAR_GP_PIN(2, 0), PU1, 3 }, /* IRQ0 */
+ { RCAR_GP_PIN(2, 14), PU1, 2 }, /* AVB_AVTP_CAPTURE_A */
+ { RCAR_GP_PIN(2, 13), PU1, 1 }, /* AVB_AVTP_MATCH_A */
+ { RCAR_GP_PIN(2, 12), PU1, 0 }, /* AVB_LINK */
+
+ { PIN_A_NUMBER('P', 8), PU2, 31 }, /* DU_DOTCLKIN1 */
+ { PIN_A_NUMBER('P', 7), PU2, 30 }, /* DU_DOTCLKIN0 */
+ { RCAR_GP_PIN(7, 3), PU2, 29 }, /* HDMI1_CEC */
+ { RCAR_GP_PIN(7, 2), PU2, 28 }, /* HDMI0_CEC */
+ { RCAR_GP_PIN(7, 1), PU2, 27 }, /* AVS2 */
+ { RCAR_GP_PIN(7, 0), PU2, 26 }, /* AVS1 */
+ { RCAR_GP_PIN(0, 15), PU2, 25 }, /* D15 */
+ { RCAR_GP_PIN(0, 14), PU2, 24 }, /* D14 */
+ { RCAR_GP_PIN(0, 13), PU2, 23 }, /* D13 */
+ { RCAR_GP_PIN(0, 12), PU2, 22 }, /* D12 */
+ { RCAR_GP_PIN(0, 11), PU2, 21 }, /* D11 */
+ { RCAR_GP_PIN(0, 10), PU2, 20 }, /* D10 */
+ { RCAR_GP_PIN(0, 9), PU2, 19 }, /* D9 */
+ { RCAR_GP_PIN(0, 8), PU2, 18 }, /* D8 */
+ { RCAR_GP_PIN(0, 7), PU2, 17 }, /* D7 */
+ { RCAR_GP_PIN(0, 6), PU2, 16 }, /* D6 */
+ { RCAR_GP_PIN(0, 5), PU2, 15 }, /* D5 */
+ { RCAR_GP_PIN(0, 4), PU2, 14 }, /* D4 */
+ { RCAR_GP_PIN(0, 3), PU2, 13 }, /* D3 */
+ { RCAR_GP_PIN(0, 2), PU2, 12 }, /* D2 */
+ { RCAR_GP_PIN(0, 1), PU2, 11 }, /* D1 */
+ { RCAR_GP_PIN(0, 0), PU2, 10 }, /* D0 */
+ { PIN_NUMBER('C', 1), PU2, 9 }, /* PRESETOUT# */
+ { RCAR_GP_PIN(1, 27), PU2, 8 }, /* EX_WAIT0_A */
+ { RCAR_GP_PIN(1, 26), PU2, 7 }, /* WE1_N */
+ { RCAR_GP_PIN(1, 25), PU2, 6 }, /* WE0_N */
+ { RCAR_GP_PIN(1, 24), PU2, 5 }, /* RD_WR_N */
+ { RCAR_GP_PIN(1, 23), PU2, 4 }, /* RD_N */
+ { RCAR_GP_PIN(1, 22), PU2, 3 }, /* BS_N */
+ { RCAR_GP_PIN(1, 21), PU2, 2 }, /* CS1_N_A26 */
+ { RCAR_GP_PIN(1, 20), PU2, 1 }, /* CS0_N */
+ { PIN_NUMBER('F', 1), PU2, 0 }, /* CLKOUT */
+
+ { RCAR_GP_PIN(4, 9), PU3, 31 }, /* SD3_DAT0 */
+ { RCAR_GP_PIN(4, 8), PU3, 30 }, /* SD3_CMD */
+ { RCAR_GP_PIN(4, 7), PU3, 29 }, /* SD3_CLK */
+ { RCAR_GP_PIN(4, 6), PU3, 28 }, /* SD2_DS */
+ { RCAR_GP_PIN(4, 5), PU3, 27 }, /* SD2_DAT3 */
+ { RCAR_GP_PIN(4, 4), PU3, 26 }, /* SD2_DAT2 */
+ { RCAR_GP_PIN(4, 3), PU3, 25 }, /* SD2_DAT1 */
+ { RCAR_GP_PIN(4, 2), PU3, 24 }, /* SD2_DAT0 */
+ { RCAR_GP_PIN(4, 1), PU3, 23 }, /* SD2_CMD */
+ { RCAR_GP_PIN(4, 0), PU3, 22 }, /* SD2_CLK */
+ { RCAR_GP_PIN(3, 11), PU3, 21 }, /* SD1_DAT3 */
+ { RCAR_GP_PIN(3, 10), PU3, 20 }, /* SD1_DAT2 */
+ { RCAR_GP_PIN(3, 9), PU3, 19 }, /* SD1_DAT1 */
+ { RCAR_GP_PIN(3, 8), PU3, 18 }, /* SD1_DAT0 */
+ { RCAR_GP_PIN(3, 7), PU3, 17 }, /* SD1_CMD */
+ { RCAR_GP_PIN(3, 6), PU3, 16 }, /* SD1_CLK */
+ { RCAR_GP_PIN(3, 5), PU3, 15 }, /* SD0_DAT3 */
+ { RCAR_GP_PIN(3, 4), PU3, 14 }, /* SD0_DAT2 */
+ { RCAR_GP_PIN(3, 3), PU3, 13 }, /* SD0_DAT1 */
+ { RCAR_GP_PIN(3, 2), PU3, 12 }, /* SD0_DAT0 */
+ { RCAR_GP_PIN(3, 1), PU3, 11 }, /* SD0_CMD */
+ { RCAR_GP_PIN(3, 0), PU3, 10 }, /* SD0_CLK */
+ { PIN_A_NUMBER('T', 30), PU3, 9 }, /* ASEBRK */
+ /* bit 8 n/a */
+ { PIN_A_NUMBER('R', 29), PU3, 7 }, /* TDI */
+ { PIN_A_NUMBER('R', 30), PU3, 6 }, /* TMS */
+ { PIN_A_NUMBER('T', 27), PU3, 5 }, /* TCK */
+ { PIN_A_NUMBER('R', 26), PU3, 4 }, /* TRST# */
+ { PIN_A_NUMBER('D', 39), PU3, 3 }, /* EXTALR*/
+ { PIN_A_NUMBER('D', 38), PU3, 2 }, /* FSCLKST# */
+ { PIN_A_NUMBER('R', 8), PU3, 1 }, /* DU_DOTCLKIN3 */
+ { PIN_A_NUMBER('R', 7), PU3, 0 }, /* DU_DOTCLKIN2 */
+
+ { RCAR_GP_PIN(5, 19), PU4, 31 }, /* MSIOF0_SS1 */
+ { RCAR_GP_PIN(5, 18), PU4, 30 }, /* MSIOF0_SYNC */
+ { RCAR_GP_PIN(5, 17), PU4, 29 }, /* MSIOF0_SCK */
+ { RCAR_GP_PIN(5, 16), PU4, 28 }, /* HRTS0_N */
+ { RCAR_GP_PIN(5, 15), PU4, 27 }, /* HCTS0_N */
+ { RCAR_GP_PIN(5, 14), PU4, 26 }, /* HTX0 */
+ { RCAR_GP_PIN(5, 13), PU4, 25 }, /* HRX0 */
+ { RCAR_GP_PIN(5, 12), PU4, 24 }, /* HSCK0 */
+ { RCAR_GP_PIN(5, 11), PU4, 23 }, /* RX2_A */
+ { RCAR_GP_PIN(5, 10), PU4, 22 }, /* TX2_A */
+ { RCAR_GP_PIN(5, 9), PU4, 21 }, /* SCK2 */
+ { RCAR_GP_PIN(5, 8), PU4, 20 }, /* RTS1_N_TANS */
+ { RCAR_GP_PIN(5, 7), PU4, 19 }, /* CTS1_N */
+ { RCAR_GP_PIN(5, 6), PU4, 18 }, /* TX1_A */
+ { RCAR_GP_PIN(5, 5), PU4, 17 }, /* RX1_A */
+ { RCAR_GP_PIN(5, 4), PU4, 16 }, /* RTS0_N_TANS */
+ { RCAR_GP_PIN(5, 3), PU4, 15 }, /* CTS0_N */
+ { RCAR_GP_PIN(5, 2), PU4, 14 }, /* TX0 */
+ { RCAR_GP_PIN(5, 1), PU4, 13 }, /* RX0 */
+ { RCAR_GP_PIN(5, 0), PU4, 12 }, /* SCK0 */
+ { RCAR_GP_PIN(3, 15), PU4, 11 }, /* SD1_WP */
+ { RCAR_GP_PIN(3, 14), PU4, 10 }, /* SD1_CD */
+ { RCAR_GP_PIN(3, 13), PU4, 9 }, /* SD0_WP */
+ { RCAR_GP_PIN(3, 12), PU4, 8 }, /* SD0_CD */
+ { RCAR_GP_PIN(4, 17), PU4, 7 }, /* SD3_DS */
+ { RCAR_GP_PIN(4, 16), PU4, 6 }, /* SD3_DAT7 */
+ { RCAR_GP_PIN(4, 15), PU4, 5 }, /* SD3_DAT6 */
+ { RCAR_GP_PIN(4, 14), PU4, 4 }, /* SD3_DAT5 */
+ { RCAR_GP_PIN(4, 13), PU4, 3 }, /* SD3_DAT4 */
+ { RCAR_GP_PIN(4, 12), PU4, 2 }, /* SD3_DAT3 */
+ { RCAR_GP_PIN(4, 11), PU4, 1 }, /* SD3_DAT2 */
+ { RCAR_GP_PIN(4, 10), PU4, 0 }, /* SD3_DAT1 */
+
+ { RCAR_GP_PIN(6, 24), PU5, 31 }, /* USB0_PWEN */
+ { RCAR_GP_PIN(6, 23), PU5, 30 }, /* AUDIO_CLKB_B */
+ { RCAR_GP_PIN(6, 22), PU5, 29 }, /* AUDIO_CLKA_A */
+ { RCAR_GP_PIN(6, 21), PU5, 28 }, /* SSI_SDATA9_A */
+ { RCAR_GP_PIN(6, 20), PU5, 27 }, /* SSI_SDATA8 */
+ { RCAR_GP_PIN(6, 19), PU5, 26 }, /* SSI_SDATA7 */
+ { RCAR_GP_PIN(6, 18), PU5, 25 }, /* SSI_WS78 */
+ { RCAR_GP_PIN(6, 17), PU5, 24 }, /* SSI_SCK78 */
+ { RCAR_GP_PIN(6, 16), PU5, 23 }, /* SSI_SDATA6 */
+ { RCAR_GP_PIN(6, 15), PU5, 22 }, /* SSI_WS6 */
+ { RCAR_GP_PIN(6, 14), PU5, 21 }, /* SSI_SCK6 */
+ { RCAR_GP_PIN(6, 13), PU5, 20 }, /* SSI_SDATA5 */
+ { RCAR_GP_PIN(6, 12), PU5, 19 }, /* SSI_WS5 */
+ { RCAR_GP_PIN(6, 11), PU5, 18 }, /* SSI_SCK5 */
+ { RCAR_GP_PIN(6, 10), PU5, 17 }, /* SSI_SDATA4 */
+ { RCAR_GP_PIN(6, 9), PU5, 16 }, /* SSI_WS4 */
+ { RCAR_GP_PIN(6, 8), PU5, 15 }, /* SSI_SCK4 */
+ { RCAR_GP_PIN(6, 7), PU5, 14 }, /* SSI_SDATA3 */
+ { RCAR_GP_PIN(6, 6), PU5, 13 }, /* SSI_WS34 */
+ { RCAR_GP_PIN(6, 5), PU5, 12 }, /* SSI_SCK34 */
+ { RCAR_GP_PIN(6, 4), PU5, 11 }, /* SSI_SDATA2_A */
+ { RCAR_GP_PIN(6, 3), PU5, 10 }, /* SSI_SDATA1_A */
+ { RCAR_GP_PIN(6, 2), PU5, 9 }, /* SSI_SDATA0 */
+ { RCAR_GP_PIN(6, 1), PU5, 8 }, /* SSI_WS01239 */
+ { RCAR_GP_PIN(6, 0), PU5, 7 }, /* SSI_SCK01239 */
+ { PIN_NUMBER('H', 37), PU5, 6 }, /* MLB_REF */
+ { RCAR_GP_PIN(5, 25), PU5, 5 }, /* MLB_DAT */
+ { RCAR_GP_PIN(5, 24), PU5, 4 }, /* MLB_SIG */
+ { RCAR_GP_PIN(5, 23), PU5, 3 }, /* MLB_CLK */
+ { RCAR_GP_PIN(5, 22), PU5, 2 }, /* MSIOF0_RXD */
+ { RCAR_GP_PIN(5, 21), PU5, 1 }, /* MSIOF0_SS2 */
+ { RCAR_GP_PIN(5, 20), PU5, 0 }, /* MSIOF0_TXD */
+
+ { RCAR_GP_PIN(6, 31), PU6, 6 }, /* USB31_OVC */
+ { RCAR_GP_PIN(6, 30), PU6, 5 }, /* USB31_PWEN */
+ { RCAR_GP_PIN(6, 29), PU6, 4 }, /* USB30_OVC */
+ { RCAR_GP_PIN(6, 28), PU6, 3 }, /* USB30_PWEN */
+ { RCAR_GP_PIN(6, 27), PU6, 2 }, /* USB1_OVC */
+ { RCAR_GP_PIN(6, 26), PU6, 1 }, /* USB1_PWEN */
+ { RCAR_GP_PIN(6, 25), PU6, 0 }, /* USB0_OVC */
+};
+
+static unsigned int r8a7795es1_pinmux_get_bias(struct sh_pfc *pfc,
+ unsigned int pin)
+{
+ const struct sh_pfc_bias_info *info;
+ u32 reg;
+ u32 bit;
+
+ info = sh_pfc_pin_to_bias_info(bias_info, ARRAY_SIZE(bias_info), pin);
+ if (!info)
+ return PIN_CONFIG_BIAS_DISABLE;
+
+ reg = info->reg;
+ bit = BIT(info->bit);
+
+ if (!(sh_pfc_read_reg(pfc, PUEN + reg, 32) & bit))
+ return PIN_CONFIG_BIAS_DISABLE;
+ else if (sh_pfc_read_reg(pfc, PUD + reg, 32) & bit)
+ return PIN_CONFIG_BIAS_PULL_UP;
+ else
+ return PIN_CONFIG_BIAS_PULL_DOWN;
+}
+
+static void r8a7795es1_pinmux_set_bias(struct sh_pfc *pfc, unsigned int pin,
+ unsigned int bias)
+{
+ const struct sh_pfc_bias_info *info;
+ u32 enable, updown;
+ u32 reg;
+ u32 bit;
+
+ info = sh_pfc_pin_to_bias_info(bias_info, ARRAY_SIZE(bias_info), pin);
+ if (!info)
+ return;
+
+ reg = info->reg;
+ bit = BIT(info->bit);
+
+ enable = sh_pfc_read_reg(pfc, PUEN + reg, 32) & ~bit;
+ if (bias != PIN_CONFIG_BIAS_DISABLE)
+ enable |= bit;
+
+ updown = sh_pfc_read_reg(pfc, PUD + reg, 32) & ~bit;
+ if (bias == PIN_CONFIG_BIAS_PULL_UP)
+ updown |= bit;
+
+ sh_pfc_write_reg(pfc, PUD + reg, 32, updown);
+ sh_pfc_write_reg(pfc, PUEN + reg, 32, enable);
+}
+
+static const struct sh_pfc_soc_operations r8a7795es1_pinmux_ops = {
+ .pin_to_pocctrl = r8a7795es1_pin_to_pocctrl,
+ .get_bias = r8a7795es1_pinmux_get_bias,
+ .set_bias = r8a7795es1_pinmux_set_bias,
+};
+
+const struct sh_pfc_soc_info r8a7795es1_pinmux_info = {
+ .name = "r8a77950_pfc",
+ .ops = &r8a7795es1_pinmux_ops,
+ .unlock_reg = 0xe6060000, /* PMMR */
+
+ .function = { PINMUX_FUNCTION_BEGIN, PINMUX_FUNCTION_END },
+
+ .pins = pinmux_pins,
+ .nr_pins = ARRAY_SIZE(pinmux_pins),
+ .groups = pinmux_groups,
+ .nr_groups = ARRAY_SIZE(pinmux_groups),
+ .functions = pinmux_functions,
+ .nr_functions = ARRAY_SIZE(pinmux_functions),
+
+ .cfg_regs = pinmux_config_regs,
+ .drive_regs = pinmux_drive_regs,
+
+ .pinmux_data = pinmux_data,
+ .pinmux_data_size = ARRAY_SIZE(pinmux_data),
+};
diff --git a/drivers/pinctrl/sh-pfc/pfc-r8a7795.c b/drivers/pinctrl/sh-pfc/pfc-r8a7795.c
index 504d0c3..0454f31 100644
--- a/drivers/pinctrl/sh-pfc/pfc-r8a7795.c
+++ b/drivers/pinctrl/sh-pfc/pfc-r8a7795.c
@@ -1,7 +1,7 @@
/*
- * R8A7795 processor support - PFC hardware block.
+ * R8A7795 ES2.0+ processor support - PFC hardware block.
*
- * Copyright (C) 2015 Renesas Electronics Corporation
+ * Copyright (C) 2015-2016 Renesas Electronics Corporation
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -9,6 +9,7 @@
*/
#include <linux/kernel.h>
+#include <linux/sys_soc.h>
#include "core.h"
#include "sh_pfc.h"
@@ -101,10 +102,10 @@
#define GPSR2_0 F_(IRQ0, IP0_27_24)
/* GPSR3 */
-#define GPSR3_15 F_(SD1_WP, IP10_23_20)
-#define GPSR3_14 F_(SD1_CD, IP10_19_16)
-#define GPSR3_13 F_(SD0_WP, IP10_15_12)
-#define GPSR3_12 F_(SD0_CD, IP10_11_8)
+#define GPSR3_15 F_(SD1_WP, IP11_23_20)
+#define GPSR3_14 F_(SD1_CD, IP11_19_16)
+#define GPSR3_13 F_(SD0_WP, IP11_15_12)
+#define GPSR3_12 F_(SD0_CD, IP11_11_8)
#define GPSR3_11 F_(SD1_DAT3, IP8_31_28)
#define GPSR3_10 F_(SD1_DAT2, IP8_27_24)
#define GPSR3_9 F_(SD1_DAT1, IP8_23_20)
@@ -119,86 +120,86 @@
#define GPSR3_0 F_(SD0_CLK, IP7_19_16)
/* GPSR4 */
-#define GPSR4_17 FM(SD3_DS)
-#define GPSR4_16 F_(SD3_DAT7, IP10_7_4)
-#define GPSR4_15 F_(SD3_DAT6, IP10_3_0)
-#define GPSR4_14 F_(SD3_DAT5, IP9_31_28)
-#define GPSR4_13 F_(SD3_DAT4, IP9_27_24)
-#define GPSR4_12 FM(SD3_DAT3)
-#define GPSR4_11 FM(SD3_DAT2)
-#define GPSR4_10 FM(SD3_DAT1)
-#define GPSR4_9 FM(SD3_DAT0)
-#define GPSR4_8 FM(SD3_CMD)
-#define GPSR4_7 FM(SD3_CLK)
-#define GPSR4_6 F_(SD2_DS, IP9_23_20)
-#define GPSR4_5 F_(SD2_DAT3, IP9_19_16)
-#define GPSR4_4 F_(SD2_DAT2, IP9_15_12)
-#define GPSR4_3 F_(SD2_DAT1, IP9_11_8)
-#define GPSR4_2 F_(SD2_DAT0, IP9_7_4)
-#define GPSR4_1 FM(SD2_CMD)
+#define GPSR4_17 F_(SD3_DS, IP11_7_4)
+#define GPSR4_16 F_(SD3_DAT7, IP11_3_0)
+#define GPSR4_15 F_(SD3_DAT6, IP10_31_28)
+#define GPSR4_14 F_(SD3_DAT5, IP10_27_24)
+#define GPSR4_13 F_(SD3_DAT4, IP10_23_20)
+#define GPSR4_12 F_(SD3_DAT3, IP10_19_16)
+#define GPSR4_11 F_(SD3_DAT2, IP10_15_12)
+#define GPSR4_10 F_(SD3_DAT1, IP10_11_8)
+#define GPSR4_9 F_(SD3_DAT0, IP10_7_4)
+#define GPSR4_8 F_(SD3_CMD, IP10_3_0)
+#define GPSR4_7 F_(SD3_CLK, IP9_31_28)
+#define GPSR4_6 F_(SD2_DS, IP9_27_24)
+#define GPSR4_5 F_(SD2_DAT3, IP9_23_20)
+#define GPSR4_4 F_(SD2_DAT2, IP9_19_16)
+#define GPSR4_3 F_(SD2_DAT1, IP9_15_12)
+#define GPSR4_2 F_(SD2_DAT0, IP9_11_8)
+#define GPSR4_1 F_(SD2_CMD, IP9_7_4)
#define GPSR4_0 F_(SD2_CLK, IP9_3_0)
/* GPSR5 */
-#define GPSR5_25 F_(MLB_DAT, IP13_19_16)
-#define GPSR5_24 F_(MLB_SIG, IP13_15_12)
-#define GPSR5_23 F_(MLB_CLK, IP13_11_8)
+#define GPSR5_25 F_(MLB_DAT, IP14_19_16)
+#define GPSR5_24 F_(MLB_SIG, IP14_15_12)
+#define GPSR5_23 F_(MLB_CLK, IP14_11_8)
#define GPSR5_22 FM(MSIOF0_RXD)
-#define GPSR5_21 F_(MSIOF0_SS2, IP13_7_4)
+#define GPSR5_21 F_(MSIOF0_SS2, IP14_7_4)
#define GPSR5_20 FM(MSIOF0_TXD)
-#define GPSR5_19 F_(MSIOF0_SS1, IP13_3_0)
-#define GPSR5_18 F_(MSIOF0_SYNC, IP12_31_28)
+#define GPSR5_19 F_(MSIOF0_SS1, IP14_3_0)
+#define GPSR5_18 F_(MSIOF0_SYNC, IP13_31_28)
#define GPSR5_17 FM(MSIOF0_SCK)
-#define GPSR5_16 F_(HRTS0_N, IP12_27_24)
-#define GPSR5_15 F_(HCTS0_N, IP12_23_20)
-#define GPSR5_14 F_(HTX0, IP12_19_16)
-#define GPSR5_13 F_(HRX0, IP12_15_12)
-#define GPSR5_12 F_(HSCK0, IP12_11_8)
-#define GPSR5_11 F_(RX2_A, IP12_7_4)
-#define GPSR5_10 F_(TX2_A, IP12_3_0)
-#define GPSR5_9 F_(SCK2, IP11_31_28)
-#define GPSR5_8 F_(RTS1_N_TANS, IP11_27_24)
-#define GPSR5_7 F_(CTS1_N, IP11_23_20)
-#define GPSR5_6 F_(TX1_A, IP11_19_16)
-#define GPSR5_5 F_(RX1_A, IP11_15_12)
-#define GPSR5_4 F_(RTS0_N_TANS, IP11_11_8)
-#define GPSR5_3 F_(CTS0_N, IP11_7_4)
-#define GPSR5_2 F_(TX0, IP11_3_0)
-#define GPSR5_1 F_(RX0, IP10_31_28)
-#define GPSR5_0 F_(SCK0, IP10_27_24)
+#define GPSR5_16 F_(HRTS0_N, IP13_27_24)
+#define GPSR5_15 F_(HCTS0_N, IP13_23_20)
+#define GPSR5_14 F_(HTX0, IP13_19_16)
+#define GPSR5_13 F_(HRX0, IP13_15_12)
+#define GPSR5_12 F_(HSCK0, IP13_11_8)
+#define GPSR5_11 F_(RX2_A, IP13_7_4)
+#define GPSR5_10 F_(TX2_A, IP13_3_0)
+#define GPSR5_9 F_(SCK2, IP12_31_28)
+#define GPSR5_8 F_(RTS1_N_TANS, IP12_27_24)
+#define GPSR5_7 F_(CTS1_N, IP12_23_20)
+#define GPSR5_6 F_(TX1_A, IP12_19_16)
+#define GPSR5_5 F_(RX1_A, IP12_15_12)
+#define GPSR5_4 F_(RTS0_N_TANS, IP12_11_8)
+#define GPSR5_3 F_(CTS0_N, IP12_7_4)
+#define GPSR5_2 F_(TX0, IP12_3_0)
+#define GPSR5_1 F_(RX0, IP11_31_28)
+#define GPSR5_0 F_(SCK0, IP11_27_24)
/* GPSR6 */
-#define GPSR6_31 F_(USB31_OVC, IP17_7_4)
-#define GPSR6_30 F_(USB31_PWEN, IP17_3_0)
-#define GPSR6_29 F_(USB30_OVC, IP16_31_28)
-#define GPSR6_28 F_(USB30_PWEN, IP16_27_24)
-#define GPSR6_27 F_(USB1_OVC, IP16_23_20)
-#define GPSR6_26 F_(USB1_PWEN, IP16_19_16)
-#define GPSR6_25 F_(USB0_OVC, IP16_15_12)
-#define GPSR6_24 F_(USB0_PWEN, IP16_11_8)
-#define GPSR6_23 F_(AUDIO_CLKB_B, IP16_7_4)
-#define GPSR6_22 F_(AUDIO_CLKA_A, IP16_3_0)
-#define GPSR6_21 F_(SSI_SDATA9_A, IP15_31_28)
-#define GPSR6_20 F_(SSI_SDATA8, IP15_27_24)
-#define GPSR6_19 F_(SSI_SDATA7, IP15_23_20)
-#define GPSR6_18 F_(SSI_WS78, IP15_19_16)
-#define GPSR6_17 F_(SSI_SCK78, IP15_15_12)
-#define GPSR6_16 F_(SSI_SDATA6, IP15_11_8)
-#define GPSR6_15 F_(SSI_WS6, IP15_7_4)
-#define GPSR6_14 F_(SSI_SCK6, IP15_3_0)
+#define GPSR6_31 F_(USB3_OVC, IP18_7_4)
+#define GPSR6_30 F_(USB3_PWEN, IP18_3_0)
+#define GPSR6_29 F_(USB30_OVC, IP17_31_28)
+#define GPSR6_28 F_(USB30_PWEN, IP17_27_24)
+#define GPSR6_27 F_(USB1_OVC, IP17_23_20)
+#define GPSR6_26 F_(USB1_PWEN, IP17_19_16)
+#define GPSR6_25 F_(USB0_OVC, IP17_15_12)
+#define GPSR6_24 F_(USB0_PWEN, IP17_11_8)
+#define GPSR6_23 F_(AUDIO_CLKB_B, IP17_7_4)
+#define GPSR6_22 F_(AUDIO_CLKA_A, IP17_3_0)
+#define GPSR6_21 F_(SSI_SDATA9_A, IP16_31_28)
+#define GPSR6_20 F_(SSI_SDATA8, IP16_27_24)
+#define GPSR6_19 F_(SSI_SDATA7, IP16_23_20)
+#define GPSR6_18 F_(SSI_WS78, IP16_19_16)
+#define GPSR6_17 F_(SSI_SCK78, IP16_15_12)
+#define GPSR6_16 F_(SSI_SDATA6, IP16_11_8)
+#define GPSR6_15 F_(SSI_WS6, IP16_7_4)
+#define GPSR6_14 F_(SSI_SCK6, IP16_3_0)
#define GPSR6_13 FM(SSI_SDATA5)
#define GPSR6_12 FM(SSI_WS5)
#define GPSR6_11 FM(SSI_SCK5)
-#define GPSR6_10 F_(SSI_SDATA4, IP14_31_28)
-#define GPSR6_9 F_(SSI_WS4, IP14_27_24)
-#define GPSR6_8 F_(SSI_SCK4, IP14_23_20)
-#define GPSR6_7 F_(SSI_SDATA3, IP14_19_16)
-#define GPSR6_6 F_(SSI_WS34, IP14_15_12)
-#define GPSR6_5 F_(SSI_SCK34, IP14_11_8)
-#define GPSR6_4 F_(SSI_SDATA2_A, IP14_7_4)
-#define GPSR6_3 F_(SSI_SDATA1_A, IP14_3_0)
-#define GPSR6_2 F_(SSI_SDATA0, IP13_31_28)
-#define GPSR6_1 F_(SSI_WS01239, IP13_27_24)
-#define GPSR6_0 F_(SSI_SCK01239, IP13_23_20)
+#define GPSR6_10 F_(SSI_SDATA4, IP15_31_28)
+#define GPSR6_9 F_(SSI_WS4, IP15_27_24)
+#define GPSR6_8 F_(SSI_SCK4, IP15_23_20)
+#define GPSR6_7 F_(SSI_SDATA3, IP15_19_16)
+#define GPSR6_6 F_(SSI_WS34, IP15_15_12)
+#define GPSR6_5 F_(SSI_SCK34, IP15_11_8)
+#define GPSR6_4 F_(SSI_SDATA2_A, IP15_7_4)
+#define GPSR6_3 F_(SSI_SDATA1_A, IP15_3_0)
+#define GPSR6_2 F_(SSI_SDATA0, IP14_31_28)
+#define GPSR6_1 F_(SSI_WS01239, IP14_27_24)
+#define GPSR6_0 F_(SSI_SCK01239, IP14_23_20)
/* GPSR7 */
#define GPSR7_3 FM(HDMI1_CEC)
@@ -212,14 +213,14 @@
#define IP0_7_4 FM(AVB_MAGIC) F_(0, 0) FM(MSIOF2_SS1_C) FM(SCK4_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
#define IP0_11_8 FM(AVB_PHY_INT) F_(0, 0) FM(MSIOF2_SYNC_C) FM(RX4_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
#define IP0_15_12 FM(AVB_LINK) F_(0, 0) FM(MSIOF2_SCK_C) FM(TX4_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP0_19_16 FM(AVB_AVTP_MATCH_A) F_(0, 0) FM(MSIOF2_RXD_C) FM(CTS4_N_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP0_19_16 FM(AVB_AVTP_MATCH_A) F_(0, 0) FM(MSIOF2_RXD_C) FM(CTS4_N_A) F_(0, 0) FM(FSCLKST2_N_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
#define IP0_23_20 FM(AVB_AVTP_CAPTURE_A) F_(0, 0) FM(MSIOF2_TXD_C) FM(RTS4_N_TANS_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP0_27_24 FM(IRQ0) FM(QPOLB) F_(0, 0) FM(DU_CDE) FM(VI4_DATA0_B) FM(CAN0_TX_B) FM(CANFD0_TX_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP0_31_28 FM(IRQ1) FM(QPOLA) F_(0, 0) FM(DU_DISP) FM(VI4_DATA1_B) FM(CAN0_RX_B) FM(CANFD0_RX_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP1_3_0 FM(IRQ2) FM(QCPV_QDE) F_(0, 0) FM(DU_EXODDF_DU_ODDF_DISP_CDE) FM(VI4_DATA2_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(PWM3_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP1_7_4 FM(IRQ3) FM(QSTVB_QVE) FM(A25) FM(DU_DOTCLKOUT1) FM(VI4_DATA3_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(PWM4_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP1_11_8 FM(IRQ4) FM(QSTH_QHS) FM(A24) FM(DU_EXHSYNC_DU_HSYNC) FM(VI4_DATA4_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(PWM5_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP1_15_12 FM(IRQ5) FM(QSTB_QHE) FM(A23) FM(DU_EXVSYNC_DU_VSYNC) FM(VI4_DATA5_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(PWM6_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP0_27_24 FM(IRQ0) FM(QPOLB) F_(0, 0) FM(DU_CDE) FM(VI4_DATA0_B) FM(CAN0_TX_B) FM(CANFD0_TX_B) FM(MSIOF3_SS1_E) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP0_31_28 FM(IRQ1) FM(QPOLA) F_(0, 0) FM(DU_DISP) FM(VI4_DATA1_B) FM(CAN0_RX_B) FM(CANFD0_RX_B) FM(MSIOF3_SS2_E) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP1_3_0 FM(IRQ2) FM(QCPV_QDE) F_(0, 0) FM(DU_EXODDF_DU_ODDF_DISP_CDE) FM(VI4_DATA2_B) F_(0, 0) F_(0, 0) FM(MSIOF3_SYNC_E) F_(0, 0) FM(PWM3_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP1_7_4 FM(IRQ3) FM(QSTVB_QVE) FM(A25) FM(DU_DOTCLKOUT1) FM(VI4_DATA3_B) F_(0, 0) F_(0, 0) FM(MSIOF3_SCK_E) F_(0, 0) FM(PWM4_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP1_11_8 FM(IRQ4) FM(QSTH_QHS) FM(A24) FM(DU_EXHSYNC_DU_HSYNC) FM(VI4_DATA4_B) F_(0, 0) F_(0, 0) FM(MSIOF3_RXD_E) F_(0, 0) FM(PWM5_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP1_15_12 FM(IRQ5) FM(QSTB_QHE) FM(A23) FM(DU_EXVSYNC_DU_VSYNC) FM(VI4_DATA5_B) FM(FSCLKST2_N_B) F_(0, 0) FM(MSIOF3_TXD_E) F_(0, 0) FM(PWM6_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
#define IP1_19_16 FM(PWM0) FM(AVB_AVTP_PPS)FM(A22) F_(0, 0) FM(VI4_DATA6_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(IECLK_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
#define IP1_23_20 FM(PWM1_A) F_(0, 0) FM(A21) FM(HRX3_D) FM(VI4_DATA7_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(IERX_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
#define IP1_27_24 FM(PWM2_A) F_(0, 0) FM(A20) FM(HTX3_D) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(IETX_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
@@ -279,79 +280,89 @@
#define IP8_3_0 FM(SD0_DAT2) F_(0, 0) FM(MSIOF1_SS1_E) F_(0, 0) F_(0, 0) FM(TS_SDAT0_B) FM(STP_ISD_0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
#define IP8_7_4 FM(SD0_DAT3) F_(0, 0) FM(MSIOF1_SS2_E) F_(0, 0) F_(0, 0) FM(TS_SDEN0_B) FM(STP_ISEN_0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
#define IP8_11_8 FM(SD1_CLK) F_(0, 0) FM(MSIOF1_SCK_G) F_(0, 0) F_(0, 0) FM(SIM0_CLK_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP8_15_12 FM(SD1_CMD) F_(0, 0) FM(MSIOF1_SYNC_G) F_(0, 0) F_(0, 0) FM(SIM0_D_A) FM(STP_IVCXO27_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP8_19_16 FM(SD1_DAT0) FM(SD2_DAT4) FM(MSIOF1_RXD_G) F_(0, 0) F_(0, 0) FM(TS_SCK1_B) FM(STP_ISCLK_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP8_23_20 FM(SD1_DAT1) FM(SD2_DAT5) FM(MSIOF1_TXD_G) F_(0, 0) F_(0, 0) FM(TS_SPSYNC1_B)FM(STP_ISSYNC_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP8_27_24 FM(SD1_DAT2) FM(SD2_DAT6) FM(MSIOF1_SS1_G) F_(0, 0) F_(0, 0) FM(TS_SDAT1_B) FM(STP_ISD_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP8_31_28 FM(SD1_DAT3) FM(SD2_DAT7) FM(MSIOF1_SS2_G) F_(0, 0) F_(0, 0) FM(TS_SDEN1_B) FM(STP_ISEN_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP9_3_0 FM(SD2_CLK) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP9_7_4 FM(SD2_DAT0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP9_11_8 FM(SD2_DAT1) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP9_15_12 FM(SD2_DAT2) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP9_19_16 FM(SD2_DAT3) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP9_23_20 FM(SD2_DS) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(SATA_DEVSLP_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP9_27_24 FM(SD3_DAT4) FM(SD2_CD_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP9_31_28 FM(SD3_DAT5) FM(SD2_WP_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP10_3_0 FM(SD3_DAT6) FM(SD3_CD) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP10_7_4 FM(SD3_DAT7) FM(SD3_WP) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP10_11_8 FM(SD0_CD) F_(0, 0) F_(0, 0) F_(0, 0) FM(SCL2_B) FM(SIM0_RST_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP10_15_12 FM(SD0_WP) F_(0, 0) F_(0, 0) F_(0, 0) FM(SDA2_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP10_19_16 FM(SD1_CD) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(SIM0_CLK_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP10_23_20 FM(SD1_WP) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(SIM0_D_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP10_27_24 FM(SCK0) FM(HSCK1_B) FM(MSIOF1_SS2_B) FM(AUDIO_CLKC_B) FM(SDA2_A) FM(SIM0_RST_B) FM(STP_OPWM_0_C) FM(RIF0_CLK_B) F_(0, 0) FM(ADICHS2) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP10_31_28 FM(RX0) FM(HRX1_B) F_(0, 0) F_(0, 0) F_(0, 0) FM(TS_SCK0_C) FM(STP_ISCLK_0_C) FM(RIF0_D0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP11_3_0 FM(TX0) FM(HTX1_B) F_(0, 0) F_(0, 0) F_(0, 0) FM(TS_SPSYNC0_C)FM(STP_ISSYNC_0_C) FM(RIF0_D1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP11_7_4 FM(CTS0_N) FM(HCTS1_N_B) FM(MSIOF1_SYNC_B) F_(0, 0) F_(0, 0) FM(TS_SPSYNC1_C)FM(STP_ISSYNC_1_C) FM(RIF1_SYNC_B) FM(AUDIO_CLKOUT_C) FM(ADICS_SAMP) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP11_11_8 FM(RTS0_N_TANS) FM(HRTS1_N_B) FM(MSIOF1_SS1_B) FM(AUDIO_CLKA_B) FM(SCL2_A) F_(0, 0) FM(STP_IVCXO27_1_C) FM(RIF0_SYNC_B) F_(0, 0) FM(ADICHS1) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP11_15_12 FM(RX1_A) FM(HRX1_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(TS_SDAT0_C) FM(STP_ISD_0_C) FM(RIF1_CLK_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP11_19_16 FM(TX1_A) FM(HTX1_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(TS_SDEN0_C) FM(STP_ISEN_0_C) FM(RIF1_D0_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP11_23_20 FM(CTS1_N) FM(HCTS1_N_A) FM(MSIOF1_RXD_B) F_(0, 0) F_(0, 0) FM(TS_SDEN1_C) FM(STP_ISEN_1_C) FM(RIF1_D0_B) F_(0, 0) FM(ADIDATA) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP11_27_24 FM(RTS1_N_TANS) FM(HRTS1_N_A) FM(MSIOF1_TXD_B) F_(0, 0) F_(0, 0) FM(TS_SDAT1_C) FM(STP_ISD_1_C) FM(RIF1_D1_B) F_(0, 0) FM(ADICHS0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP11_31_28 FM(SCK2) FM(SCIF_CLK_B) FM(MSIOF1_SCK_B) F_(0, 0) F_(0, 0) FM(TS_SCK1_C) FM(STP_ISCLK_1_C) FM(RIF1_CLK_B) F_(0, 0) FM(ADICLK) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP12_3_0 FM(TX2_A) F_(0, 0) F_(0, 0) FM(SD2_CD_B) FM(SCL1_A) F_(0, 0) FM(FMCLK_A) FM(RIF1_D1_C) F_(0, 0) FM(FSO_CFE_0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP12_7_4 FM(RX2_A) F_(0, 0) F_(0, 0) FM(SD2_WP_B) FM(SDA1_A) F_(0, 0) FM(FMIN_A) FM(RIF1_SYNC_C) F_(0, 0) FM(FSO_CFE_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP12_11_8 FM(HSCK0) F_(0, 0) FM(MSIOF1_SCK_D) FM(AUDIO_CLKB_A) FM(SSI_SDATA1_B)FM(TS_SCK0_D) FM(STP_ISCLK_0_D) FM(RIF0_CLK_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP12_15_12 FM(HRX0) F_(0, 0) FM(MSIOF1_RXD_D) F_(0, 0) FM(SSI_SDATA2_B)FM(TS_SDEN0_D) FM(STP_ISEN_0_D) FM(RIF0_D0_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP12_19_16 FM(HTX0) F_(0, 0) FM(MSIOF1_TXD_D) F_(0, 0) FM(SSI_SDATA9_B)FM(TS_SDAT0_D) FM(STP_ISD_0_D) FM(RIF0_D1_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP12_23_20 FM(HCTS0_N) FM(RX2_B) FM(MSIOF1_SYNC_D) F_(0, 0) FM(SSI_SCK9_A) FM(TS_SPSYNC0_D)FM(STP_ISSYNC_0_D) FM(RIF0_SYNC_C) FM(AUDIO_CLKOUT1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP12_27_24 FM(HRTS0_N) FM(TX2_B) FM(MSIOF1_SS1_D) F_(0, 0) FM(SSI_WS9_A) F_(0, 0) FM(STP_IVCXO27_0_D) FM(BPFCLK_A) FM(AUDIO_CLKOUT2_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP8_15_12 FM(SD1_CMD) F_(0, 0) FM(MSIOF1_SYNC_G) FM(NFCE_N_B) F_(0, 0) FM(SIM0_D_A) FM(STP_IVCXO27_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP8_19_16 FM(SD1_DAT0) FM(SD2_DAT4) FM(MSIOF1_RXD_G) FM(NFWP_N_B) F_(0, 0) FM(TS_SCK1_B) FM(STP_ISCLK_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP8_23_20 FM(SD1_DAT1) FM(SD2_DAT5) FM(MSIOF1_TXD_G) FM(NFDATA14_B) F_(0, 0) FM(TS_SPSYNC1_B)FM(STP_ISSYNC_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP8_27_24 FM(SD1_DAT2) FM(SD2_DAT6) FM(MSIOF1_SS1_G) FM(NFDATA15_B) F_(0, 0) FM(TS_SDAT1_B) FM(STP_ISD_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP8_31_28 FM(SD1_DAT3) FM(SD2_DAT7) FM(MSIOF1_SS2_G) FM(NFRB_N_B) F_(0, 0) FM(TS_SDEN1_B) FM(STP_ISEN_1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_3_0 FM(SD2_CLK) F_(0, 0) F_(0, 0) FM(NFDATA8) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_7_4 FM(SD2_CMD) F_(0, 0) F_(0, 0) FM(NFDATA9) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_11_8 FM(SD2_DAT0) F_(0, 0) F_(0, 0) FM(NFDATA10) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_15_12 FM(SD2_DAT1) F_(0, 0) F_(0, 0) FM(NFDATA11) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_19_16 FM(SD2_DAT2) F_(0, 0) F_(0, 0) FM(NFDATA12) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_23_20 FM(SD2_DAT3) F_(0, 0) F_(0, 0) FM(NFDATA13) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_27_24 FM(SD2_DS) F_(0, 0) F_(0, 0) FM(NFALE) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(SATA_DEVSLP_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP9_31_28 FM(SD3_CLK) F_(0, 0) F_(0, 0) FM(NFWE_N) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_3_0 FM(SD3_CMD) F_(0, 0) F_(0, 0) FM(NFRE_N) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_7_4 FM(SD3_DAT0) F_(0, 0) F_(0, 0) FM(NFDATA0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_11_8 FM(SD3_DAT1) F_(0, 0) F_(0, 0) FM(NFDATA1) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_15_12 FM(SD3_DAT2) F_(0, 0) F_(0, 0) FM(NFDATA2) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_19_16 FM(SD3_DAT3) F_(0, 0) F_(0, 0) FM(NFDATA3) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_23_20 FM(SD3_DAT4) FM(SD2_CD_A) F_(0, 0) FM(NFDATA4) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_27_24 FM(SD3_DAT5) FM(SD2_WP_A) F_(0, 0) FM(NFDATA5) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP10_31_28 FM(SD3_DAT6) FM(SD3_CD) F_(0, 0) FM(NFDATA6) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_3_0 FM(SD3_DAT7) FM(SD3_WP) F_(0, 0) FM(NFDATA7) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_7_4 FM(SD3_DS) F_(0, 0) F_(0, 0) FM(NFCLE) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_11_8 FM(SD0_CD) F_(0, 0) FM(NFDATA14_A) F_(0, 0) FM(SCL2_B) FM(SIM0_RST_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
/* IPSRx */ /* 0 */ /* 1 */ /* 2 */ /* 3 */ /* 4 */ /* 5 */ /* 6 */ /* 7 */ /* 8 */ /* 9 */ /* A */ /* B */ /* C - F */
-#define IP12_31_28 FM(MSIOF0_SYNC) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP13_3_0 FM(MSIOF0_SS1) FM(RX5) F_(0, 0) FM(AUDIO_CLKA_C) FM(SSI_SCK2_A) F_(0, 0) FM(STP_IVCXO27_0_C) F_(0, 0) FM(AUDIO_CLKOUT3_A) F_(0, 0) FM(TCLK1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP13_7_4 FM(MSIOF0_SS2) FM(TX5) FM(MSIOF1_SS2_D) FM(AUDIO_CLKC_A) FM(SSI_WS2_A) F_(0, 0) FM(STP_OPWM_0_D) F_(0, 0) FM(AUDIO_CLKOUT_D) F_(0, 0) FM(SPEEDIN_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP13_11_8 FM(MLB_CLK) F_(0, 0) FM(MSIOF1_SCK_F) F_(0, 0) FM(SCL1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP13_15_12 FM(MLB_SIG) FM(RX1_B) FM(MSIOF1_SYNC_F) F_(0, 0) FM(SDA1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP13_19_16 FM(MLB_DAT) FM(TX1_B) FM(MSIOF1_RXD_F) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP13_23_20 FM(SSI_SCK01239) F_(0, 0) FM(MSIOF1_TXD_F) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP13_27_24 FM(SSI_WS01239) F_(0, 0) FM(MSIOF1_SS1_F) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP13_31_28 FM(SSI_SDATA0) F_(0, 0) FM(MSIOF1_SS2_F) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP14_3_0 FM(SSI_SDATA1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP14_7_4 FM(SSI_SDATA2_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(SSI_SCK1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP14_11_8 FM(SSI_SCK34) F_(0, 0) FM(MSIOF1_SS1_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(STP_OPWM_0_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP14_15_12 FM(SSI_WS34) FM(HCTS2_N_A) FM(MSIOF1_SS2_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(STP_IVCXO27_0_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP14_19_16 FM(SSI_SDATA3) FM(HRTS2_N_A) FM(MSIOF1_TXD_A) F_(0, 0) F_(0, 0) FM(TS_SCK0_A) FM(STP_ISCLK_0_A) FM(RIF0_D1_A) FM(RIF2_D0_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP14_23_20 FM(SSI_SCK4) FM(HRX2_A) FM(MSIOF1_SCK_A) F_(0, 0) F_(0, 0) FM(TS_SDAT0_A) FM(STP_ISD_0_A) FM(RIF0_CLK_A) FM(RIF2_CLK_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP14_27_24 FM(SSI_WS4) FM(HTX2_A) FM(MSIOF1_SYNC_A) F_(0, 0) F_(0, 0) FM(TS_SDEN0_A) FM(STP_ISEN_0_A) FM(RIF0_SYNC_A) FM(RIF2_SYNC_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP14_31_28 FM(SSI_SDATA4) FM(HSCK2_A) FM(MSIOF1_RXD_A) F_(0, 0) F_(0, 0) FM(TS_SPSYNC0_A)FM(STP_ISSYNC_0_A) FM(RIF0_D0_A) FM(RIF2_D1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP15_3_0 FM(SSI_SCK6) FM(USB2_PWEN) F_(0, 0) FM(SIM0_RST_D) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP15_7_4 FM(SSI_WS6) FM(USB2_OVC) F_(0, 0) FM(SIM0_D_D) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP15_11_8 FM(SSI_SDATA6) F_(0, 0) F_(0, 0) FM(SIM0_CLK_D) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(SATA_DEVSLP_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP15_15_12 FM(SSI_SCK78) FM(HRX2_B) FM(MSIOF1_SCK_C) F_(0, 0) F_(0, 0) FM(TS_SCK1_A) FM(STP_ISCLK_1_A) FM(RIF1_CLK_A) FM(RIF3_CLK_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP15_19_16 FM(SSI_WS78) FM(HTX2_B) FM(MSIOF1_SYNC_C) F_(0, 0) F_(0, 0) FM(TS_SDAT1_A) FM(STP_ISD_1_A) FM(RIF1_SYNC_A) FM(RIF3_SYNC_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP15_23_20 FM(SSI_SDATA7) FM(HCTS2_N_B) FM(MSIOF1_RXD_C) F_(0, 0) F_(0, 0) FM(TS_SDEN1_A) FM(STP_ISEN_1_A) FM(RIF1_D0_A) FM(RIF3_D0_A) F_(0, 0) FM(TCLK2_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP15_27_24 FM(SSI_SDATA8) FM(HRTS2_N_B) FM(MSIOF1_TXD_C) F_(0, 0) F_(0, 0) FM(TS_SPSYNC1_A)FM(STP_ISSYNC_1_A) FM(RIF1_D1_A) FM(RIF3_D1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP15_31_28 FM(SSI_SDATA9_A) FM(HSCK2_B) FM(MSIOF1_SS1_C) FM(HSCK1_A) FM(SSI_WS1_B) FM(SCK1) FM(STP_IVCXO27_1_A) FM(SCK5) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP16_3_0 FM(AUDIO_CLKA_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(CC5_OSCOUT) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP16_7_4 FM(AUDIO_CLKB_B) FM(SCIF_CLK_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(STP_IVCXO27_1_D) FM(REMOCON_A) F_(0, 0) F_(0, 0) FM(TCLK1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP16_11_8 FM(USB0_PWEN) F_(0, 0) F_(0, 0) FM(SIM0_RST_C) F_(0, 0) FM(TS_SCK1_D) FM(STP_ISCLK_1_D) FM(BPFCLK_B) FM(RIF3_CLK_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP16_15_12 FM(USB0_OVC) F_(0, 0) F_(0, 0) FM(SIM0_D_C) F_(0, 0) FM(TS_SDAT1_D) FM(STP_ISD_1_D) F_(0, 0) FM(RIF3_SYNC_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP16_19_16 FM(USB1_PWEN) F_(0, 0) F_(0, 0) FM(SIM0_CLK_C) FM(SSI_SCK1_A) FM(TS_SCK0_E) FM(STP_ISCLK_0_E) FM(FMCLK_B) FM(RIF2_CLK_B) F_(0, 0) FM(SPEEDIN_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP16_23_20 FM(USB1_OVC) F_(0, 0) FM(MSIOF1_SS2_C) F_(0, 0) FM(SSI_WS1_A) FM(TS_SDAT0_E) FM(STP_ISD_0_E) FM(FMIN_B) FM(RIF2_SYNC_B) F_(0, 0) FM(REMOCON_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP16_27_24 FM(USB30_PWEN) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT_B) FM(SSI_SCK2_B) FM(TS_SDEN1_D) FM(STP_ISEN_1_D) FM(STP_OPWM_0_E)FM(RIF3_D0_B) F_(0, 0) FM(TCLK2_B) FM(TPU0TO0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP16_31_28 FM(USB30_OVC) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT1_B) FM(SSI_WS2_B) FM(TS_SPSYNC1_D)FM(STP_ISSYNC_1_D) FM(STP_IVCXO27_0_E)FM(RIF3_D1_B) F_(0, 0) FM(FSO_TOE_B) FM(TPU0TO1) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP17_3_0 FM(USB31_PWEN) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT2_B) FM(SSI_SCK9_B) FM(TS_SDEN0_E) FM(STP_ISEN_0_E) F_(0, 0) FM(RIF2_D0_B) F_(0, 0) F_(0, 0) FM(TPU0TO2) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
-#define IP17_7_4 FM(USB31_OVC) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT3_B) FM(SSI_WS9_B) FM(TS_SPSYNC0_E)FM(STP_ISSYNC_0_E) F_(0, 0) FM(RIF2_D1_B) F_(0, 0) F_(0, 0) FM(TPU0TO3) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_15_12 FM(SD0_WP) F_(0, 0) FM(NFDATA15_A) F_(0, 0) FM(SDA2_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_19_16 FM(SD1_CD) F_(0, 0) FM(NFRB_N_A) F_(0, 0) F_(0, 0) FM(SIM0_CLK_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_23_20 FM(SD1_WP) F_(0, 0) FM(NFCE_N_A) F_(0, 0) F_(0, 0) FM(SIM0_D_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_27_24 FM(SCK0) FM(HSCK1_B) FM(MSIOF1_SS2_B) FM(AUDIO_CLKC_B) FM(SDA2_A) FM(SIM0_RST_B) FM(STP_OPWM_0_C) FM(RIF0_CLK_B) F_(0, 0) FM(ADICHS2) FM(SCK5_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP11_31_28 FM(RX0) FM(HRX1_B) F_(0, 0) F_(0, 0) F_(0, 0) FM(TS_SCK0_C) FM(STP_ISCLK_0_C) FM(RIF0_D0_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_3_0 FM(TX0) FM(HTX1_B) F_(0, 0) F_(0, 0) F_(0, 0) FM(TS_SPSYNC0_C)FM(STP_ISSYNC_0_C) FM(RIF0_D1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_7_4 FM(CTS0_N) FM(HCTS1_N_B) FM(MSIOF1_SYNC_B) F_(0, 0) F_(0, 0) FM(TS_SPSYNC1_C)FM(STP_ISSYNC_1_C) FM(RIF1_SYNC_B) FM(AUDIO_CLKOUT_C) FM(ADICS_SAMP) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_11_8 FM(RTS0_N_TANS) FM(HRTS1_N_B) FM(MSIOF1_SS1_B) FM(AUDIO_CLKA_B) FM(SCL2_A) F_(0, 0) FM(STP_IVCXO27_1_C) FM(RIF0_SYNC_B) F_(0, 0) FM(ADICHS1) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_15_12 FM(RX1_A) FM(HRX1_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(TS_SDAT0_C) FM(STP_ISD_0_C) FM(RIF1_CLK_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_19_16 FM(TX1_A) FM(HTX1_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(TS_SDEN0_C) FM(STP_ISEN_0_C) FM(RIF1_D0_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_23_20 FM(CTS1_N) FM(HCTS1_N_A) FM(MSIOF1_RXD_B) F_(0, 0) F_(0, 0) FM(TS_SDEN1_C) FM(STP_ISEN_1_C) FM(RIF1_D0_B) F_(0, 0) FM(ADIDATA) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_27_24 FM(RTS1_N_TANS) FM(HRTS1_N_A) FM(MSIOF1_TXD_B) F_(0, 0) F_(0, 0) FM(TS_SDAT1_C) FM(STP_ISD_1_C) FM(RIF1_D1_B) F_(0, 0) FM(ADICHS0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP12_31_28 FM(SCK2) FM(SCIF_CLK_B) FM(MSIOF1_SCK_B) F_(0, 0) F_(0, 0) FM(TS_SCK1_C) FM(STP_ISCLK_1_C) FM(RIF1_CLK_B) F_(0, 0) FM(ADICLK) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_3_0 FM(TX2_A) F_(0, 0) F_(0, 0) FM(SD2_CD_B) FM(SCL1_A) F_(0, 0) FM(FMCLK_A) FM(RIF1_D1_C) F_(0, 0) FM(FSO_CFE_0_N) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_7_4 FM(RX2_A) F_(0, 0) F_(0, 0) FM(SD2_WP_B) FM(SDA1_A) F_(0, 0) FM(FMIN_A) FM(RIF1_SYNC_C) F_(0, 0) FM(FSO_CFE_1_N) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_11_8 FM(HSCK0) F_(0, 0) FM(MSIOF1_SCK_D) FM(AUDIO_CLKB_A) FM(SSI_SDATA1_B)FM(TS_SCK0_D) FM(STP_ISCLK_0_D) FM(RIF0_CLK_C) F_(0, 0) F_(0, 0) FM(RX5_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_15_12 FM(HRX0) F_(0, 0) FM(MSIOF1_RXD_D) F_(0, 0) FM(SSI_SDATA2_B)FM(TS_SDEN0_D) FM(STP_ISEN_0_D) FM(RIF0_D0_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_19_16 FM(HTX0) F_(0, 0) FM(MSIOF1_TXD_D) F_(0, 0) FM(SSI_SDATA9_B)FM(TS_SDAT0_D) FM(STP_ISD_0_D) FM(RIF0_D1_C) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_23_20 FM(HCTS0_N) FM(RX2_B) FM(MSIOF1_SYNC_D) F_(0, 0) FM(SSI_SCK9_A) FM(TS_SPSYNC0_D)FM(STP_ISSYNC_0_D) FM(RIF0_SYNC_C) FM(AUDIO_CLKOUT1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_27_24 FM(HRTS0_N) FM(TX2_B) FM(MSIOF1_SS1_D) F_(0, 0) FM(SSI_WS9_A) F_(0, 0) FM(STP_IVCXO27_0_D) FM(BPFCLK_A) FM(AUDIO_CLKOUT2_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP13_31_28 FM(MSIOF0_SYNC) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT_A) F_(0, 0) FM(TX5_B) F_(0, 0) F_(0, 0) FM(BPFCLK_D) F_(0, 0) F_(0, 0)
+#define IP14_3_0 FM(MSIOF0_SS1) FM(RX5_A) FM(NFWP_N_A) FM(AUDIO_CLKA_C) FM(SSI_SCK2_A) F_(0, 0) FM(STP_IVCXO27_0_C) F_(0, 0) FM(AUDIO_CLKOUT3_A) F_(0, 0) FM(TCLK1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_7_4 FM(MSIOF0_SS2) FM(TX5_A) FM(MSIOF1_SS2_D) FM(AUDIO_CLKC_A) FM(SSI_WS2_A) F_(0, 0) FM(STP_OPWM_0_D) F_(0, 0) FM(AUDIO_CLKOUT_D) F_(0, 0) FM(SPEEDIN_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_11_8 FM(MLB_CLK) F_(0, 0) FM(MSIOF1_SCK_F) F_(0, 0) FM(SCL1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_15_12 FM(MLB_SIG) FM(RX1_B) FM(MSIOF1_SYNC_F) F_(0, 0) FM(SDA1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_19_16 FM(MLB_DAT) FM(TX1_B) FM(MSIOF1_RXD_F) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_23_20 FM(SSI_SCK01239) F_(0, 0) FM(MSIOF1_TXD_F) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP14_27_24 FM(SSI_WS01239) F_(0, 0) FM(MSIOF1_SS1_F) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+
+/* IPSRx */ /* 0 */ /* 1 */ /* 2 */ /* 3 */ /* 4 */ /* 5 */ /* 6 */ /* 7 */ /* 8 */ /* 9 */ /* A */ /* B */ /* C - F */
+#define IP14_31_28 FM(SSI_SDATA0) F_(0, 0) FM(MSIOF1_SS2_F) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_3_0 FM(SSI_SDATA1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_7_4 FM(SSI_SDATA2_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(SSI_SCK1_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_11_8 FM(SSI_SCK34) F_(0, 0) FM(MSIOF1_SS1_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(STP_OPWM_0_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_15_12 FM(SSI_WS34) FM(HCTS2_N_A) FM(MSIOF1_SS2_A) F_(0, 0) F_(0, 0) F_(0, 0) FM(STP_IVCXO27_0_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_19_16 FM(SSI_SDATA3) FM(HRTS2_N_A) FM(MSIOF1_TXD_A) F_(0, 0) F_(0, 0) FM(TS_SCK0_A) FM(STP_ISCLK_0_A) FM(RIF0_D1_A) FM(RIF2_D0_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_23_20 FM(SSI_SCK4) FM(HRX2_A) FM(MSIOF1_SCK_A) F_(0, 0) F_(0, 0) FM(TS_SDAT0_A) FM(STP_ISD_0_A) FM(RIF0_CLK_A) FM(RIF2_CLK_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_27_24 FM(SSI_WS4) FM(HTX2_A) FM(MSIOF1_SYNC_A) F_(0, 0) F_(0, 0) FM(TS_SDEN0_A) FM(STP_ISEN_0_A) FM(RIF0_SYNC_A) FM(RIF2_SYNC_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP15_31_28 FM(SSI_SDATA4) FM(HSCK2_A) FM(MSIOF1_RXD_A) F_(0, 0) F_(0, 0) FM(TS_SPSYNC0_A)FM(STP_ISSYNC_0_A) FM(RIF0_D0_A) FM(RIF2_D1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_3_0 FM(SSI_SCK6) FM(USB2_PWEN) F_(0, 0) FM(SIM0_RST_D) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_7_4 FM(SSI_WS6) FM(USB2_OVC) F_(0, 0) FM(SIM0_D_D) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_11_8 FM(SSI_SDATA6) F_(0, 0) F_(0, 0) FM(SIM0_CLK_D) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(SATA_DEVSLP_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_15_12 FM(SSI_SCK78) FM(HRX2_B) FM(MSIOF1_SCK_C) F_(0, 0) F_(0, 0) FM(TS_SCK1_A) FM(STP_ISCLK_1_A) FM(RIF1_CLK_A) FM(RIF3_CLK_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_19_16 FM(SSI_WS78) FM(HTX2_B) FM(MSIOF1_SYNC_C) F_(0, 0) F_(0, 0) FM(TS_SDAT1_A) FM(STP_ISD_1_A) FM(RIF1_SYNC_A) FM(RIF3_SYNC_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_23_20 FM(SSI_SDATA7) FM(HCTS2_N_B) FM(MSIOF1_RXD_C) F_(0, 0) F_(0, 0) FM(TS_SDEN1_A) FM(STP_ISEN_1_A) FM(RIF1_D0_A) FM(RIF3_D0_A) F_(0, 0) FM(TCLK2_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_27_24 FM(SSI_SDATA8) FM(HRTS2_N_B) FM(MSIOF1_TXD_C) F_(0, 0) F_(0, 0) FM(TS_SPSYNC1_A)FM(STP_ISSYNC_1_A) FM(RIF1_D1_A) FM(RIF3_D1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP16_31_28 FM(SSI_SDATA9_A) FM(HSCK2_B) FM(MSIOF1_SS1_C) FM(HSCK1_A) FM(SSI_WS1_B) FM(SCK1) FM(STP_IVCXO27_1_A) FM(SCK5_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP17_3_0 FM(AUDIO_CLKA_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(CC5_OSCOUT) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP17_7_4 FM(AUDIO_CLKB_B) FM(SCIF_CLK_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(STP_IVCXO27_1_D) FM(REMOCON_A) F_(0, 0) F_(0, 0) FM(TCLK1_A) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP17_11_8 FM(USB0_PWEN) F_(0, 0) F_(0, 0) FM(SIM0_RST_C) F_(0, 0) FM(TS_SCK1_D) FM(STP_ISCLK_1_D) FM(BPFCLK_B) FM(RIF3_CLK_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(HSCK2_C) F_(0, 0) F_(0, 0)
+#define IP17_15_12 FM(USB0_OVC) F_(0, 0) F_(0, 0) FM(SIM0_D_C) F_(0, 0) FM(TS_SDAT1_D) FM(STP_ISD_1_D) F_(0, 0) FM(RIF3_SYNC_B) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) FM(HRX2_C) F_(0, 0) F_(0, 0)
+#define IP17_19_16 FM(USB1_PWEN) F_(0, 0) F_(0, 0) FM(SIM0_CLK_C) FM(SSI_SCK1_A) FM(TS_SCK0_E) FM(STP_ISCLK_0_E) FM(FMCLK_B) FM(RIF2_CLK_B) F_(0, 0) FM(SPEEDIN_A) F_(0, 0) F_(0, 0) FM(HTX2_C) F_(0, 0) F_(0, 0)
+#define IP17_23_20 FM(USB1_OVC) F_(0, 0) FM(MSIOF1_SS2_C) F_(0, 0) FM(SSI_WS1_A) FM(TS_SDAT0_E) FM(STP_ISD_0_E) FM(FMIN_B) FM(RIF2_SYNC_B) F_(0, 0) FM(REMOCON_B) F_(0, 0) F_(0, 0) FM(HCTS2_N_C) F_(0, 0) F_(0, 0)
+#define IP17_27_24 FM(USB30_PWEN) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT_B) FM(SSI_SCK2_B) FM(TS_SDEN1_D) FM(STP_ISEN_1_D) FM(STP_OPWM_0_E)FM(RIF3_D0_B) F_(0, 0) FM(TCLK2_B) FM(TPU0TO0) FM(BPFCLK_C) FM(HRTS2_N_C) F_(0, 0) F_(0, 0)
+#define IP17_31_28 FM(USB30_OVC) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT1_B) FM(SSI_WS2_B) FM(TS_SPSYNC1_D)FM(STP_ISSYNC_1_D) FM(STP_IVCXO27_0_E)FM(RIF3_D1_B) F_(0, 0) FM(FSO_TOE_N) FM(TPU0TO1) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define IP18_3_0 FM(USB3_PWEN) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT2_B) FM(SSI_SCK9_B) FM(TS_SDEN0_E) FM(STP_ISEN_0_E) F_(0, 0) FM(RIF2_D0_B) F_(0, 0) F_(0, 0) FM(TPU0TO2) F_(0, 0) FM(FMCLK_C) FM(FMCLK_D) F_(0, 0)
+#define IP18_7_4 FM(USB3_OVC) F_(0, 0) F_(0, 0) FM(AUDIO_CLKOUT3_B) FM(SSI_WS9_B) FM(TS_SPSYNC0_E)FM(STP_ISSYNC_0_E) F_(0, 0) FM(RIF2_D1_B) F_(0, 0) F_(0, 0) FM(TPU0TO3) F_(0, 0) FM(FMIN_C) FM(FMIN_D) F_(0, 0)
#define PINMUX_GPSR \
\
@@ -426,37 +437,34 @@
FM(IP12_27_24) IP12_27_24 FM(IP13_27_24) IP13_27_24 FM(IP14_27_24) IP14_27_24 FM(IP15_27_24) IP15_27_24 \
FM(IP12_31_28) IP12_31_28 FM(IP13_31_28) IP13_31_28 FM(IP14_31_28) IP14_31_28 FM(IP15_31_28) IP15_31_28 \
\
-FM(IP16_3_0) IP16_3_0 FM(IP17_3_0) IP17_3_0 \
-FM(IP16_7_4) IP16_7_4 FM(IP17_7_4) IP17_7_4 \
-FM(IP16_11_8) IP16_11_8 \
-FM(IP16_15_12) IP16_15_12 \
-FM(IP16_19_16) IP16_19_16 \
-FM(IP16_23_20) IP16_23_20 \
-FM(IP16_27_24) IP16_27_24 \
-FM(IP16_31_28) IP16_31_28
+FM(IP16_3_0) IP16_3_0 FM(IP17_3_0) IP17_3_0 FM(IP18_3_0) IP18_3_0 \
+FM(IP16_7_4) IP16_7_4 FM(IP17_7_4) IP17_7_4 FM(IP18_7_4) IP18_7_4 \
+FM(IP16_11_8) IP16_11_8 FM(IP17_11_8) IP17_11_8 \
+FM(IP16_15_12) IP16_15_12 FM(IP17_15_12) IP17_15_12 \
+FM(IP16_19_16) IP16_19_16 FM(IP17_19_16) IP17_19_16 \
+FM(IP16_23_20) IP16_23_20 FM(IP17_23_20) IP17_23_20 \
+FM(IP16_27_24) IP16_27_24 FM(IP17_27_24) IP17_27_24 \
+FM(IP16_31_28) IP16_31_28 FM(IP17_31_28) IP17_31_28
/* MOD_SEL0 */ /* 0 */ /* 1 */ /* 2 */ /* 3 */ /* 4 */ /* 5 */ /* 6 */ /* 7 */
-#define MOD_SEL0_30_29 FM(SEL_MSIOF3_0) FM(SEL_MSIOF3_1) FM(SEL_MSIOF3_2) FM(SEL_MSIOF3_3)
+#define MOD_SEL0_31_30_29 FM(SEL_MSIOF3_0) FM(SEL_MSIOF3_1) FM(SEL_MSIOF3_2) FM(SEL_MSIOF3_3) FM(SEL_MSIOF3_4) F_(0, 0) F_(0, 0) F_(0, 0)
#define MOD_SEL0_28_27 FM(SEL_MSIOF2_0) FM(SEL_MSIOF2_1) FM(SEL_MSIOF2_2) FM(SEL_MSIOF2_3)
#define MOD_SEL0_26_25_24 FM(SEL_MSIOF1_0) FM(SEL_MSIOF1_1) FM(SEL_MSIOF1_2) FM(SEL_MSIOF1_3) FM(SEL_MSIOF1_4) FM(SEL_MSIOF1_5) FM(SEL_MSIOF1_6) F_(0, 0)
#define MOD_SEL0_23 FM(SEL_LBSC_0) FM(SEL_LBSC_1)
#define MOD_SEL0_22 FM(SEL_IEBUS_0) FM(SEL_IEBUS_1)
-#define MOD_SEL0_21_20 FM(SEL_I2C6_0) FM(SEL_I2C6_1) FM(SEL_I2C6_2) F_(0, 0)
-#define MOD_SEL0_19 FM(SEL_I2C2_0) FM(SEL_I2C2_1)
-#define MOD_SEL0_18 FM(SEL_I2C1_0) FM(SEL_I2C1_1)
-#define MOD_SEL0_17 FM(SEL_HSCIF4_0) FM(SEL_HSCIF4_1)
-#define MOD_SEL0_16_15 FM(SEL_HSCIF3_0) FM(SEL_HSCIF3_1) FM(SEL_HSCIF3_2) FM(SEL_HSCIF3_3)
-#define MOD_SEL0_14 FM(SEL_HSCIF2_0) FM(SEL_HSCIF2_1)
-#define MOD_SEL0_13 FM(SEL_HSCIF1_0) FM(SEL_HSCIF1_1)
-#define MOD_SEL0_12 FM(SEL_FSO_0) FM(SEL_FSO_1)
-#define MOD_SEL0_11 FM(SEL_FM_0) FM(SEL_FM_1)
-#define MOD_SEL0_10 FM(SEL_ETHERAVB_0) FM(SEL_ETHERAVB_1)
-#define MOD_SEL0_9 FM(SEL_DRIF3_0) FM(SEL_DRIF3_1)
-#define MOD_SEL0_8 FM(SEL_DRIF2_0) FM(SEL_DRIF2_1)
-#define MOD_SEL0_7_6 FM(SEL_DRIF1_0) FM(SEL_DRIF1_1) FM(SEL_DRIF1_2) F_(0, 0)
-#define MOD_SEL0_5_4 FM(SEL_DRIF0_0) FM(SEL_DRIF0_1) FM(SEL_DRIF0_2) F_(0, 0)
-#define MOD_SEL0_3 FM(SEL_CANFD0_0) FM(SEL_CANFD0_1)
-#define MOD_SEL0_2_1 FM(SEL_ADG_0) FM(SEL_ADG_1) FM(SEL_ADG_2) FM(SEL_ADG_3)
+#define MOD_SEL0_21 FM(SEL_I2C2_0) FM(SEL_I2C2_1)
+#define MOD_SEL0_20 FM(SEL_I2C1_0) FM(SEL_I2C1_1)
+#define MOD_SEL0_19 FM(SEL_HSCIF4_0) FM(SEL_HSCIF4_1)
+#define MOD_SEL0_18_17 FM(SEL_HSCIF3_0) FM(SEL_HSCIF3_1) FM(SEL_HSCIF3_2) FM(SEL_HSCIF3_3)
+#define MOD_SEL0_16 FM(SEL_HSCIF1_0) FM(SEL_HSCIF1_1)
+#define MOD_SEL0_14_13 FM(SEL_HSCIF2_0) FM(SEL_HSCIF2_1) FM(SEL_HSCIF2_2) F_(0, 0)
+#define MOD_SEL0_12 FM(SEL_ETHERAVB_0) FM(SEL_ETHERAVB_1)
+#define MOD_SEL0_11 FM(SEL_DRIF3_0) FM(SEL_DRIF3_1)
+#define MOD_SEL0_10 FM(SEL_DRIF2_0) FM(SEL_DRIF2_1)
+#define MOD_SEL0_9_8 FM(SEL_DRIF1_0) FM(SEL_DRIF1_1) FM(SEL_DRIF1_2) F_(0, 0)
+#define MOD_SEL0_7_6 FM(SEL_DRIF0_0) FM(SEL_DRIF0_1) FM(SEL_DRIF0_2) F_(0, 0)
+#define MOD_SEL0_5 FM(SEL_CANFD0_0) FM(SEL_CANFD0_1)
+#define MOD_SEL0_4_3 FM(SEL_ADG_A_0) FM(SEL_ADG_A_1) FM(SEL_ADG_A_2) FM(SEL_ADG_A_3)
/* MOD_SEL1 */ /* 0 */ /* 1 */ /* 2 */ /* 3 */ /* 4 */ /* 5 */ /* 6 */ /* 7 */
#define MOD_SEL1_31_30 FM(SEL_TSIF1_0) FM(SEL_TSIF1_1) FM(SEL_TSIF1_2) FM(SEL_TSIF1_3)
@@ -486,40 +494,46 @@
#define MOD_SEL2_31 FM(I2C_SEL_5_0) FM(I2C_SEL_5_1)
#define MOD_SEL2_30 FM(I2C_SEL_3_0) FM(I2C_SEL_3_1)
#define MOD_SEL2_29 FM(I2C_SEL_0_0) FM(I2C_SEL_0_1)
+#define MOD_SEL2_28_27 FM(SEL_FM_0) FM(SEL_FM_1) FM(SEL_FM_2) FM(SEL_FM_3)
+#define MOD_SEL2_26 FM(SEL_SCIF5_0) FM(SEL_SCIF5_1)
+#define MOD_SEL2_25_24_23 FM(SEL_I2C6_0) FM(SEL_I2C6_1) FM(SEL_I2C6_2) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0) F_(0, 0)
+#define MOD_SEL2_22 FM(SEL_NDF_0) FM(SEL_NDF_1)
+#define MOD_SEL2_21 FM(SEL_SSI2_0) FM(SEL_SSI2_1)
+#define MOD_SEL2_20 FM(SEL_SSI9_0) FM(SEL_SSI9_1)
+#define MOD_SEL2_19 FM(SEL_TIMER_TMU2_0) FM(SEL_TIMER_TMU2_1)
+#define MOD_SEL2_18 FM(SEL_ADG_B_0) FM(SEL_ADG_B_1)
+#define MOD_SEL2_17 FM(SEL_ADG_C_0) FM(SEL_ADG_C_1)
#define MOD_SEL2_0 FM(SEL_VIN4_0) FM(SEL_VIN4_1)
-#define PINMUX_MOD_SELS\
+#define PINMUX_MOD_SELS \
\
- MOD_SEL1_31_30 MOD_SEL2_31 \
-MOD_SEL0_30_29 MOD_SEL2_30 \
+MOD_SEL0_31_30_29 MOD_SEL1_31_30 MOD_SEL2_31 \
+ MOD_SEL2_30 \
MOD_SEL1_29_28_27 MOD_SEL2_29 \
-MOD_SEL0_28_27 \
-\
-MOD_SEL0_26_25_24 MOD_SEL1_26 \
- MOD_SEL1_25_24 \
-\
+MOD_SEL0_28_27 MOD_SEL2_28_27 \
+MOD_SEL0_26_25_24 MOD_SEL1_26 MOD_SEL2_26 \
+ MOD_SEL1_25_24 MOD_SEL2_25_24_23 \
MOD_SEL0_23 MOD_SEL1_23_22_21 \
-MOD_SEL0_22 \
-MOD_SEL0_21_20 \
- MOD_SEL1_20 \
-MOD_SEL0_19 MOD_SEL1_19 \
-MOD_SEL0_18 MOD_SEL1_18_17 \
-MOD_SEL0_17 \
-MOD_SEL0_16_15 MOD_SEL1_16 \
+MOD_SEL0_22 MOD_SEL2_22 \
+MOD_SEL0_21 MOD_SEL2_21 \
+MOD_SEL0_20 MOD_SEL1_20 MOD_SEL2_20 \
+MOD_SEL0_19 MOD_SEL1_19 MOD_SEL2_19 \
+MOD_SEL0_18_17 MOD_SEL1_18_17 MOD_SEL2_18 \
+ MOD_SEL2_17 \
+MOD_SEL0_16 MOD_SEL1_16 \
MOD_SEL1_15_14 \
-MOD_SEL0_14 \
-MOD_SEL0_13 MOD_SEL1_13 \
+MOD_SEL0_14_13 \
+ MOD_SEL1_13 \
MOD_SEL0_12 MOD_SEL1_12 \
MOD_SEL0_11 MOD_SEL1_11 \
MOD_SEL0_10 MOD_SEL1_10 \
-MOD_SEL0_9 MOD_SEL1_9 \
-MOD_SEL0_8 \
+MOD_SEL0_9_8 MOD_SEL1_9 \
MOD_SEL0_7_6 \
MOD_SEL1_6 \
-MOD_SEL0_5_4 MOD_SEL1_5 \
- MOD_SEL1_4 \
-MOD_SEL0_3 MOD_SEL1_3 \
-MOD_SEL0_2_1 MOD_SEL1_2 \
+MOD_SEL0_5 MOD_SEL1_5 \
+MOD_SEL0_4_3 MOD_SEL1_4 \
+ MOD_SEL1_3 \
+ MOD_SEL1_2 \
MOD_SEL1_1 \
MOD_SEL1_0 MOD_SEL2_0
@@ -583,14 +597,6 @@
PINMUX_SINGLE(MSIOF0_RXD),
PINMUX_SINGLE(MSIOF0_SCK),
PINMUX_SINGLE(MSIOF0_TXD),
- PINMUX_SINGLE(SD2_CMD),
- PINMUX_SINGLE(SD3_CLK),
- PINMUX_SINGLE(SD3_CMD),
- PINMUX_SINGLE(SD3_DAT0),
- PINMUX_SINGLE(SD3_DAT1),
- PINMUX_SINGLE(SD3_DAT2),
- PINMUX_SINGLE(SD3_DAT3),
- PINMUX_SINGLE(SD3_DS),
PINMUX_SINGLE(SSI_SCK5),
PINMUX_SINGLE(SSI_SDATA5),
PINMUX_SINGLE(SSI_WS5),
@@ -614,6 +620,7 @@
PINMUX_IPSR_MSEL(IP0_19_16, AVB_AVTP_MATCH_A, SEL_ETHERAVB_0),
PINMUX_IPSR_MSEL(IP0_19_16, MSIOF2_RXD_C, SEL_MSIOF2_2),
PINMUX_IPSR_MSEL(IP0_19_16, CTS4_N_A, SEL_SCIF4_0),
+ PINMUX_IPSR_GPSR(IP0_19_16, FSCLKST2_N_A),
PINMUX_IPSR_MSEL(IP0_23_20, AVB_AVTP_CAPTURE_A, SEL_ETHERAVB_0),
PINMUX_IPSR_MSEL(IP0_23_20, MSIOF2_TXD_C, SEL_MSIOF2_2),
@@ -625,6 +632,7 @@
PINMUX_IPSR_MSEL(IP0_27_24, VI4_DATA0_B, SEL_VIN4_1),
PINMUX_IPSR_MSEL(IP0_27_24, CAN0_TX_B, SEL_RCAN0_1),
PINMUX_IPSR_MSEL(IP0_27_24, CANFD0_TX_B, SEL_CANFD0_1),
+ PINMUX_IPSR_MSEL(IP0_27_24, MSIOF3_SS2_E, SEL_MSIOF3_4),
PINMUX_IPSR_GPSR(IP0_31_28, IRQ1),
PINMUX_IPSR_GPSR(IP0_31_28, QPOLA),
@@ -632,6 +640,7 @@
PINMUX_IPSR_MSEL(IP0_31_28, VI4_DATA1_B, SEL_VIN4_1),
PINMUX_IPSR_MSEL(IP0_31_28, CAN0_RX_B, SEL_RCAN0_1),
PINMUX_IPSR_MSEL(IP0_31_28, CANFD0_RX_B, SEL_CANFD0_1),
+ PINMUX_IPSR_MSEL(IP0_31_28, MSIOF3_SS1_E, SEL_MSIOF3_4),
/* IPSR1 */
PINMUX_IPSR_GPSR(IP1_3_0, IRQ2),
@@ -639,6 +648,7 @@
PINMUX_IPSR_GPSR(IP1_3_0, DU_EXODDF_DU_ODDF_DISP_CDE),
PINMUX_IPSR_MSEL(IP1_3_0, VI4_DATA2_B, SEL_VIN4_1),
PINMUX_IPSR_MSEL(IP1_3_0, PWM3_B, SEL_PWM3_1),
+ PINMUX_IPSR_MSEL(IP1_3_0, MSIOF3_SYNC_E, SEL_MSIOF3_4),
PINMUX_IPSR_GPSR(IP1_7_4, IRQ3),
PINMUX_IPSR_GPSR(IP1_7_4, QSTVB_QVE),
@@ -646,6 +656,7 @@
PINMUX_IPSR_GPSR(IP1_7_4, DU_DOTCLKOUT1),
PINMUX_IPSR_MSEL(IP1_7_4, VI4_DATA3_B, SEL_VIN4_1),
PINMUX_IPSR_MSEL(IP1_7_4, PWM4_B, SEL_PWM4_1),
+ PINMUX_IPSR_MSEL(IP1_7_4, MSIOF3_SCK_E, SEL_MSIOF3_4),
PINMUX_IPSR_GPSR(IP1_11_8, IRQ4),
PINMUX_IPSR_GPSR(IP1_11_8, QSTH_QHS),
@@ -653,6 +664,7 @@
PINMUX_IPSR_GPSR(IP1_11_8, DU_EXHSYNC_DU_HSYNC),
PINMUX_IPSR_MSEL(IP1_11_8, VI4_DATA4_B, SEL_VIN4_1),
PINMUX_IPSR_MSEL(IP1_11_8, PWM5_B, SEL_PWM5_1),
+ PINMUX_IPSR_MSEL(IP1_11_8, MSIOF3_RXD_E, SEL_MSIOF3_4),
PINMUX_IPSR_GPSR(IP1_15_12, IRQ5),
PINMUX_IPSR_GPSR(IP1_15_12, QSTB_QHE),
@@ -660,6 +672,8 @@
PINMUX_IPSR_GPSR(IP1_15_12, DU_EXVSYNC_DU_VSYNC),
PINMUX_IPSR_MSEL(IP1_15_12, VI4_DATA5_B, SEL_VIN4_1),
PINMUX_IPSR_MSEL(IP1_15_12, PWM6_B, SEL_PWM6_1),
+ PINMUX_IPSR_GPSR(IP1_15_12, FSCLKST2_N_B),
+ PINMUX_IPSR_MSEL(IP1_15_12, MSIOF3_TXD_E, SEL_MSIOF3_4),
PINMUX_IPSR_GPSR(IP1_19_16, PWM0),
PINMUX_IPSR_GPSR(IP1_19_16, AVB_AVTP_PPS),
@@ -1009,426 +1023,481 @@
PINMUX_IPSR_GPSR(IP8_15_12, SD1_CMD),
PINMUX_IPSR_MSEL(IP8_15_12, MSIOF1_SYNC_G, SEL_MSIOF1_6),
+ PINMUX_IPSR_MSEL(IP8_15_12, NFCE_N_B, SEL_NDF_1),
PINMUX_IPSR_MSEL(IP8_15_12, SIM0_D_A, SEL_SIMCARD_0),
PINMUX_IPSR_MSEL(IP8_15_12, STP_IVCXO27_1_B, SEL_SSP1_1_1),
PINMUX_IPSR_GPSR(IP8_19_16, SD1_DAT0),
PINMUX_IPSR_GPSR(IP8_19_16, SD2_DAT4),
PINMUX_IPSR_MSEL(IP8_19_16, MSIOF1_RXD_G, SEL_MSIOF1_6),
+ PINMUX_IPSR_MSEL(IP8_19_16, NFWP_N_B, SEL_NDF_1),
PINMUX_IPSR_MSEL(IP8_19_16, TS_SCK1_B, SEL_TSIF1_1),
PINMUX_IPSR_MSEL(IP8_19_16, STP_ISCLK_1_B, SEL_SSP1_1_1),
PINMUX_IPSR_GPSR(IP8_23_20, SD1_DAT1),
PINMUX_IPSR_GPSR(IP8_23_20, SD2_DAT5),
PINMUX_IPSR_MSEL(IP8_23_20, MSIOF1_TXD_G, SEL_MSIOF1_6),
+ PINMUX_IPSR_MSEL(IP8_23_20, NFDATA14_B, SEL_NDF_1),
PINMUX_IPSR_MSEL(IP8_23_20, TS_SPSYNC1_B, SEL_TSIF1_1),
PINMUX_IPSR_MSEL(IP8_23_20, STP_ISSYNC_1_B, SEL_SSP1_1_1),
PINMUX_IPSR_GPSR(IP8_27_24, SD1_DAT2),
PINMUX_IPSR_GPSR(IP8_27_24, SD2_DAT6),
PINMUX_IPSR_MSEL(IP8_27_24, MSIOF1_SS1_G, SEL_MSIOF1_6),
+ PINMUX_IPSR_MSEL(IP8_27_24, NFDATA15_B, SEL_NDF_1),
PINMUX_IPSR_MSEL(IP8_27_24, TS_SDAT1_B, SEL_TSIF1_1),
PINMUX_IPSR_MSEL(IP8_27_24, STP_ISD_1_B, SEL_SSP1_1_1),
PINMUX_IPSR_GPSR(IP8_31_28, SD1_DAT3),
PINMUX_IPSR_GPSR(IP8_31_28, SD2_DAT7),
PINMUX_IPSR_MSEL(IP8_31_28, MSIOF1_SS2_G, SEL_MSIOF1_6),
+ PINMUX_IPSR_MSEL(IP8_31_28, NFRB_N_B, SEL_NDF_1),
PINMUX_IPSR_MSEL(IP8_31_28, TS_SDEN1_B, SEL_TSIF1_1),
PINMUX_IPSR_MSEL(IP8_31_28, STP_ISEN_1_B, SEL_SSP1_1_1),
/* IPSR9 */
PINMUX_IPSR_GPSR(IP9_3_0, SD2_CLK),
+ PINMUX_IPSR_GPSR(IP9_3_0, NFDATA8),
- PINMUX_IPSR_GPSR(IP9_7_4, SD2_DAT0),
+ PINMUX_IPSR_GPSR(IP9_7_4, SD2_CMD),
+ PINMUX_IPSR_GPSR(IP9_7_4, NFDATA9),
- PINMUX_IPSR_GPSR(IP9_11_8, SD2_DAT1),
+ PINMUX_IPSR_GPSR(IP9_11_8, SD2_DAT0),
+ PINMUX_IPSR_GPSR(IP9_11_8, NFDATA10),
- PINMUX_IPSR_GPSR(IP9_15_12, SD2_DAT2),
+ PINMUX_IPSR_GPSR(IP9_15_12, SD2_DAT1),
+ PINMUX_IPSR_GPSR(IP9_15_12, NFDATA11),
- PINMUX_IPSR_GPSR(IP9_19_16, SD2_DAT3),
+ PINMUX_IPSR_GPSR(IP9_19_16, SD2_DAT2),
+ PINMUX_IPSR_GPSR(IP9_19_16, NFDATA12),
- PINMUX_IPSR_GPSR(IP9_23_20, SD2_DS),
- PINMUX_IPSR_MSEL(IP9_23_20, SATA_DEVSLP_B, SEL_SATA_1),
+ PINMUX_IPSR_GPSR(IP9_23_20, SD2_DAT3),
+ PINMUX_IPSR_GPSR(IP9_23_20, NFDATA13),
- PINMUX_IPSR_GPSR(IP9_27_24, SD3_DAT4),
- PINMUX_IPSR_MSEL(IP9_27_24, SD2_CD_A, SEL_SDHI2_0),
+ PINMUX_IPSR_GPSR(IP9_27_24, SD2_DS),
+ PINMUX_IPSR_GPSR(IP9_27_24, NFALE),
+ PINMUX_IPSR_GPSR(IP9_27_24, SATA_DEVSLP_B),
- PINMUX_IPSR_GPSR(IP9_31_28, SD3_DAT5),
- PINMUX_IPSR_MSEL(IP9_31_28, SD2_WP_A, SEL_SDHI2_0),
+ PINMUX_IPSR_GPSR(IP9_31_28, SD3_CLK),
+ PINMUX_IPSR_GPSR(IP9_31_28, NFWE_N),
/* IPSR10 */
- PINMUX_IPSR_GPSR(IP10_3_0, SD3_DAT6),
- PINMUX_IPSR_GPSR(IP10_3_0, SD3_CD),
+ PINMUX_IPSR_GPSR(IP10_3_0, SD3_CMD),
+ PINMUX_IPSR_GPSR(IP10_3_0, NFRE_N),
- PINMUX_IPSR_GPSR(IP10_7_4, SD3_DAT7),
- PINMUX_IPSR_GPSR(IP10_7_4, SD3_WP),
+ PINMUX_IPSR_GPSR(IP10_7_4, SD3_DAT0),
+ PINMUX_IPSR_GPSR(IP10_7_4, NFDATA0),
- PINMUX_IPSR_GPSR(IP10_11_8, SD0_CD),
- PINMUX_IPSR_MSEL(IP10_11_8, SCL2_B, SEL_I2C2_1),
- PINMUX_IPSR_MSEL(IP10_11_8, SIM0_RST_A, SEL_SIMCARD_0),
+ PINMUX_IPSR_GPSR(IP10_11_8, SD3_DAT1),
+ PINMUX_IPSR_GPSR(IP10_11_8, NFDATA1),
- PINMUX_IPSR_GPSR(IP10_15_12, SD0_WP),
- PINMUX_IPSR_MSEL(IP10_15_12, SDA2_B, SEL_I2C2_1),
+ PINMUX_IPSR_GPSR(IP10_15_12, SD3_DAT2),
+ PINMUX_IPSR_GPSR(IP10_15_12, NFDATA2),
- PINMUX_IPSR_GPSR(IP10_19_16, SD1_CD),
- PINMUX_IPSR_MSEL(IP10_19_16, SIM0_CLK_B, SEL_SIMCARD_1),
+ PINMUX_IPSR_GPSR(IP10_19_16, SD3_DAT3),
+ PINMUX_IPSR_GPSR(IP10_19_16, NFDATA3),
- PINMUX_IPSR_GPSR(IP10_23_20, SD1_WP),
- PINMUX_IPSR_MSEL(IP10_23_20, SIM0_D_B, SEL_SIMCARD_1),
+ PINMUX_IPSR_GPSR(IP10_23_20, SD3_DAT4),
+ PINMUX_IPSR_MSEL(IP10_23_20, SD2_CD_A, SEL_SDHI2_0),
+ PINMUX_IPSR_GPSR(IP10_23_20, NFDATA4),
- PINMUX_IPSR_GPSR(IP10_27_24, SCK0),
- PINMUX_IPSR_MSEL(IP10_27_24, HSCK1_B, SEL_HSCIF1_1),
- PINMUX_IPSR_MSEL(IP10_27_24, MSIOF1_SS2_B, SEL_MSIOF1_1),
- PINMUX_IPSR_MSEL(IP10_27_24, AUDIO_CLKC_B, SEL_ADG_1),
- PINMUX_IPSR_MSEL(IP10_27_24, SDA2_A, SEL_I2C2_0),
- PINMUX_IPSR_MSEL(IP10_27_24, SIM0_RST_B, SEL_SIMCARD_1),
- PINMUX_IPSR_MSEL(IP10_27_24, STP_OPWM_0_C, SEL_SSP1_0_2),
- PINMUX_IPSR_MSEL(IP10_27_24, RIF0_CLK_B, SEL_DRIF0_1),
- PINMUX_IPSR_GPSR(IP10_27_24, ADICHS2),
+ PINMUX_IPSR_GPSR(IP10_27_24, SD3_DAT5),
+ PINMUX_IPSR_MSEL(IP10_27_24, SD2_WP_A, SEL_SDHI2_0),
+ PINMUX_IPSR_GPSR(IP10_27_24, NFDATA5),
- PINMUX_IPSR_GPSR(IP10_31_28, RX0),
- PINMUX_IPSR_MSEL(IP10_31_28, HRX1_B, SEL_HSCIF1_1),
- PINMUX_IPSR_MSEL(IP10_31_28, TS_SCK0_C, SEL_TSIF0_2),
- PINMUX_IPSR_MSEL(IP10_31_28, STP_ISCLK_0_C, SEL_SSP1_0_2),
- PINMUX_IPSR_MSEL(IP10_31_28, RIF0_D0_B, SEL_DRIF0_1),
+ PINMUX_IPSR_GPSR(IP10_31_28, SD3_DAT6),
+ PINMUX_IPSR_GPSR(IP10_31_28, SD3_CD),
+ PINMUX_IPSR_GPSR(IP10_31_28, NFDATA6),
/* IPSR11 */
- PINMUX_IPSR_GPSR(IP11_3_0, TX0),
- PINMUX_IPSR_MSEL(IP11_3_0, HTX1_B, SEL_HSCIF1_1),
- PINMUX_IPSR_MSEL(IP11_3_0, TS_SPSYNC0_C, SEL_TSIF0_2),
- PINMUX_IPSR_MSEL(IP11_3_0, STP_ISSYNC_0_C, SEL_SSP1_0_2),
- PINMUX_IPSR_MSEL(IP11_3_0, RIF0_D1_B, SEL_DRIF0_1),
+ PINMUX_IPSR_GPSR(IP11_3_0, SD3_DAT7),
+ PINMUX_IPSR_GPSR(IP11_3_0, SD3_WP),
+ PINMUX_IPSR_GPSR(IP11_3_0, NFDATA7),
- PINMUX_IPSR_GPSR(IP11_7_4, CTS0_N),
- PINMUX_IPSR_MSEL(IP11_7_4, HCTS1_N_B, SEL_HSCIF1_1),
- PINMUX_IPSR_MSEL(IP11_7_4, MSIOF1_SYNC_B, SEL_MSIOF1_1),
- PINMUX_IPSR_MSEL(IP11_7_4, TS_SPSYNC1_C, SEL_TSIF1_2),
- PINMUX_IPSR_MSEL(IP11_7_4, STP_ISSYNC_1_C, SEL_SSP1_1_2),
- PINMUX_IPSR_MSEL(IP11_7_4, RIF1_SYNC_B, SEL_DRIF1_1),
- PINMUX_IPSR_MSEL(IP11_7_4, AUDIO_CLKOUT_C, SEL_ADG_2),
- PINMUX_IPSR_GPSR(IP11_7_4, ADICS_SAMP),
+ PINMUX_IPSR_GPSR(IP11_7_4, SD3_DS),
+ PINMUX_IPSR_GPSR(IP11_7_4, NFCLE),
- PINMUX_IPSR_GPSR(IP11_11_8, RTS0_N_TANS),
- PINMUX_IPSR_MSEL(IP11_11_8, HRTS1_N_B, SEL_HSCIF1_1),
- PINMUX_IPSR_MSEL(IP11_11_8, MSIOF1_SS1_B, SEL_MSIOF1_1),
- PINMUX_IPSR_MSEL(IP11_11_8, AUDIO_CLKA_B, SEL_ADG_1),
- PINMUX_IPSR_MSEL(IP11_11_8, SCL2_A, SEL_I2C2_0),
- PINMUX_IPSR_MSEL(IP11_11_8, STP_IVCXO27_1_C, SEL_SSP1_1_2),
- PINMUX_IPSR_MSEL(IP11_11_8, RIF0_SYNC_B, SEL_DRIF0_1),
- PINMUX_IPSR_GPSR(IP11_11_8, ADICHS1),
+ PINMUX_IPSR_GPSR(IP11_11_8, SD0_CD),
+ PINMUX_IPSR_MSEL(IP11_11_8, SCL2_B, SEL_I2C2_1),
+ PINMUX_IPSR_MSEL(IP11_11_8, SIM0_RST_A, SEL_SIMCARD_0),
- PINMUX_IPSR_MSEL(IP11_15_12, RX1_A, SEL_SCIF1_0),
- PINMUX_IPSR_MSEL(IP11_15_12, HRX1_A, SEL_HSCIF1_0),
- PINMUX_IPSR_MSEL(IP11_15_12, TS_SDAT0_C, SEL_TSIF0_2),
- PINMUX_IPSR_MSEL(IP11_15_12, STP_ISD_0_C, SEL_SSP1_0_2),
- PINMUX_IPSR_MSEL(IP11_15_12, RIF1_CLK_C, SEL_DRIF1_2),
+ PINMUX_IPSR_GPSR(IP11_15_12, SD0_WP),
+ PINMUX_IPSR_MSEL(IP11_15_12, SDA2_B, SEL_I2C2_1),
- PINMUX_IPSR_MSEL(IP11_19_16, TX1_A, SEL_SCIF1_0),
- PINMUX_IPSR_MSEL(IP11_19_16, HTX1_A, SEL_HSCIF1_0),
- PINMUX_IPSR_MSEL(IP11_19_16, TS_SDEN0_C, SEL_TSIF0_2),
- PINMUX_IPSR_MSEL(IP11_19_16, STP_ISEN_0_C, SEL_SSP1_0_2),
- PINMUX_IPSR_MSEL(IP11_19_16, RIF1_D0_C, SEL_DRIF1_2),
+ PINMUX_IPSR_GPSR(IP11_19_16, SD1_CD),
+ PINMUX_IPSR_MSEL(IP11_19_16, SIM0_CLK_B, SEL_SIMCARD_1),
- PINMUX_IPSR_GPSR(IP11_23_20, CTS1_N),
- PINMUX_IPSR_MSEL(IP11_23_20, HCTS1_N_A, SEL_HSCIF1_0),
- PINMUX_IPSR_MSEL(IP11_23_20, MSIOF1_RXD_B, SEL_MSIOF1_1),
- PINMUX_IPSR_MSEL(IP11_23_20, TS_SDEN1_C, SEL_TSIF1_2),
- PINMUX_IPSR_MSEL(IP11_23_20, STP_ISEN_1_C, SEL_SSP1_1_2),
- PINMUX_IPSR_MSEL(IP11_23_20, RIF1_D0_B, SEL_DRIF1_1),
- PINMUX_IPSR_GPSR(IP11_23_20, ADIDATA),
+ PINMUX_IPSR_GPSR(IP11_23_20, SD1_WP),
+ PINMUX_IPSR_MSEL(IP11_23_20, SIM0_D_B, SEL_SIMCARD_1),
- PINMUX_IPSR_GPSR(IP11_27_24, RTS1_N_TANS),
- PINMUX_IPSR_MSEL(IP11_27_24, HRTS1_N_A, SEL_HSCIF1_0),
- PINMUX_IPSR_MSEL(IP11_27_24, MSIOF1_TXD_B, SEL_MSIOF1_1),
- PINMUX_IPSR_MSEL(IP11_27_24, TS_SDAT1_C, SEL_TSIF1_2),
- PINMUX_IPSR_MSEL(IP11_27_24, STP_ISD_1_C, SEL_SSP1_1_2),
- PINMUX_IPSR_MSEL(IP11_27_24, RIF1_D1_B, SEL_DRIF1_1),
- PINMUX_IPSR_GPSR(IP11_27_24, ADICHS0),
+ PINMUX_IPSR_GPSR(IP11_27_24, SCK0),
+ PINMUX_IPSR_MSEL(IP11_27_24, HSCK1_B, SEL_HSCIF1_1),
+ PINMUX_IPSR_MSEL(IP11_27_24, MSIOF1_SS2_B, SEL_MSIOF1_1),
+ PINMUX_IPSR_MSEL(IP11_27_24, AUDIO_CLKC_B, SEL_ADG_C_1),
+ PINMUX_IPSR_MSEL(IP11_27_24, SDA2_A, SEL_I2C2_0),
+ PINMUX_IPSR_MSEL(IP11_27_24, SIM0_RST_B, SEL_SIMCARD_1),
+ PINMUX_IPSR_MSEL(IP11_27_24, STP_OPWM_0_C, SEL_SSP1_0_2),
+ PINMUX_IPSR_MSEL(IP11_27_24, RIF0_CLK_B, SEL_DRIF0_1),
+ PINMUX_IPSR_GPSR(IP11_27_24, ADICHS2),
+ PINMUX_IPSR_MSEL(IP11_27_24, SCK5_B, SEL_SCIF5_1),
- PINMUX_IPSR_GPSR(IP11_31_28, SCK2),
- PINMUX_IPSR_MSEL(IP11_31_28, SCIF_CLK_B, SEL_SCIF1_1),
- PINMUX_IPSR_MSEL(IP11_31_28, MSIOF1_SCK_B, SEL_MSIOF1_1),
- PINMUX_IPSR_MSEL(IP11_31_28, TS_SCK1_C, SEL_TSIF1_2),
- PINMUX_IPSR_MSEL(IP11_31_28, STP_ISCLK_1_C, SEL_SSP1_1_2),
- PINMUX_IPSR_MSEL(IP11_31_28, RIF1_CLK_B, SEL_DRIF1_1),
- PINMUX_IPSR_GPSR(IP11_31_28, ADICLK),
+ PINMUX_IPSR_GPSR(IP11_31_28, RX0),
+ PINMUX_IPSR_MSEL(IP11_31_28, HRX1_B, SEL_HSCIF1_1),
+ PINMUX_IPSR_MSEL(IP11_31_28, TS_SCK0_C, SEL_TSIF0_2),
+ PINMUX_IPSR_MSEL(IP11_31_28, STP_ISCLK_0_C, SEL_SSP1_0_2),
+ PINMUX_IPSR_MSEL(IP11_31_28, RIF0_D0_B, SEL_DRIF0_1),
/* IPSR12 */
- PINMUX_IPSR_MSEL(IP12_3_0, TX2_A, SEL_SCIF2_0),
- PINMUX_IPSR_MSEL(IP12_3_0, SD2_CD_B, SEL_SDHI2_1),
- PINMUX_IPSR_MSEL(IP12_3_0, SCL1_A, SEL_I2C1_0),
- PINMUX_IPSR_MSEL(IP12_3_0, FMCLK_A, SEL_FM_0),
- PINMUX_IPSR_MSEL(IP12_3_0, RIF1_D1_C, SEL_DRIF1_2),
- PINMUX_IPSR_MSEL(IP12_3_0, FSO_CFE_0_B, SEL_FSO_1),
+ PINMUX_IPSR_GPSR(IP12_3_0, TX0),
+ PINMUX_IPSR_MSEL(IP12_3_0, HTX1_B, SEL_HSCIF1_1),
+ PINMUX_IPSR_MSEL(IP12_3_0, TS_SPSYNC0_C, SEL_TSIF0_2),
+ PINMUX_IPSR_MSEL(IP12_3_0, STP_ISSYNC_0_C, SEL_SSP1_0_2),
+ PINMUX_IPSR_MSEL(IP12_3_0, RIF0_D1_B, SEL_DRIF0_1),
- PINMUX_IPSR_MSEL(IP12_7_4, RX2_A, SEL_SCIF2_0),
- PINMUX_IPSR_MSEL(IP12_7_4, SD2_WP_B, SEL_SDHI2_1),
- PINMUX_IPSR_MSEL(IP12_7_4, SDA1_A, SEL_I2C1_0),
- PINMUX_IPSR_MSEL(IP12_7_4, FMIN_A, SEL_FM_0),
- PINMUX_IPSR_MSEL(IP12_7_4, RIF1_SYNC_C, SEL_DRIF1_2),
- PINMUX_IPSR_MSEL(IP12_7_4, FSO_CFE_1_B, SEL_FSO_1),
+ PINMUX_IPSR_GPSR(IP12_7_4, CTS0_N),
+ PINMUX_IPSR_MSEL(IP12_7_4, HCTS1_N_B, SEL_HSCIF1_1),
+ PINMUX_IPSR_MSEL(IP12_7_4, MSIOF1_SYNC_B, SEL_MSIOF1_1),
+ PINMUX_IPSR_MSEL(IP12_7_4, TS_SPSYNC1_C, SEL_TSIF1_2),
+ PINMUX_IPSR_MSEL(IP12_7_4, STP_ISSYNC_1_C, SEL_SSP1_1_2),
+ PINMUX_IPSR_MSEL(IP12_7_4, RIF1_SYNC_B, SEL_DRIF1_1),
+ PINMUX_IPSR_GPSR(IP12_7_4, AUDIO_CLKOUT_C),
+ PINMUX_IPSR_GPSR(IP12_7_4, ADICS_SAMP),
- PINMUX_IPSR_GPSR(IP12_11_8, HSCK0),
- PINMUX_IPSR_MSEL(IP12_11_8, MSIOF1_SCK_D, SEL_MSIOF1_3),
- PINMUX_IPSR_MSEL(IP12_11_8, AUDIO_CLKB_A, SEL_ADG_0),
- PINMUX_IPSR_MSEL(IP12_11_8, SSI_SDATA1_B, SEL_SSI_1),
- PINMUX_IPSR_MSEL(IP12_11_8, TS_SCK0_D, SEL_TSIF0_3),
- PINMUX_IPSR_MSEL(IP12_11_8, STP_ISCLK_0_D, SEL_SSP1_0_3),
- PINMUX_IPSR_MSEL(IP12_11_8, RIF0_CLK_C, SEL_DRIF0_2),
+ PINMUX_IPSR_GPSR(IP12_11_8, RTS0_N_TANS),
+ PINMUX_IPSR_MSEL(IP12_11_8, HRTS1_N_B, SEL_HSCIF1_1),
+ PINMUX_IPSR_MSEL(IP12_11_8, MSIOF1_SS1_B, SEL_MSIOF1_1),
+ PINMUX_IPSR_MSEL(IP12_11_8, AUDIO_CLKA_B, SEL_ADG_A_1),
+ PINMUX_IPSR_MSEL(IP12_11_8, SCL2_A, SEL_I2C2_0),
+ PINMUX_IPSR_MSEL(IP12_11_8, STP_IVCXO27_1_C, SEL_SSP1_1_2),
+ PINMUX_IPSR_MSEL(IP12_11_8, RIF0_SYNC_B, SEL_DRIF0_1),
+ PINMUX_IPSR_GPSR(IP12_11_8, ADICHS1),
- PINMUX_IPSR_GPSR(IP12_15_12, HRX0),
- PINMUX_IPSR_MSEL(IP12_15_12, MSIOF1_RXD_D, SEL_MSIOF1_3),
- PINMUX_IPSR_MSEL(IP12_15_12, SSI_SDATA2_B, SEL_SSI_1),
- PINMUX_IPSR_MSEL(IP12_15_12, TS_SDEN0_D, SEL_TSIF0_3),
- PINMUX_IPSR_MSEL(IP12_15_12, STP_ISEN_0_D, SEL_SSP1_0_3),
- PINMUX_IPSR_MSEL(IP12_15_12, RIF0_D0_C, SEL_DRIF0_2),
+ PINMUX_IPSR_MSEL(IP12_15_12, RX1_A, SEL_SCIF1_0),
+ PINMUX_IPSR_MSEL(IP12_15_12, HRX1_A, SEL_HSCIF1_0),
+ PINMUX_IPSR_MSEL(IP12_15_12, TS_SDAT0_C, SEL_TSIF0_2),
+ PINMUX_IPSR_MSEL(IP12_15_12, STP_ISD_0_C, SEL_SSP1_0_2),
+ PINMUX_IPSR_MSEL(IP12_15_12, RIF1_CLK_C, SEL_DRIF1_2),
- PINMUX_IPSR_GPSR(IP12_19_16, HTX0),
- PINMUX_IPSR_MSEL(IP12_19_16, MSIOF1_TXD_D, SEL_MSIOF1_3),
- PINMUX_IPSR_MSEL(IP12_19_16, SSI_SDATA9_B, SEL_SSI_1),
- PINMUX_IPSR_MSEL(IP12_19_16, TS_SDAT0_D, SEL_TSIF0_3),
- PINMUX_IPSR_MSEL(IP12_19_16, STP_ISD_0_D, SEL_SSP1_0_3),
- PINMUX_IPSR_MSEL(IP12_19_16, RIF0_D1_C, SEL_DRIF0_2),
+ PINMUX_IPSR_MSEL(IP12_19_16, TX1_A, SEL_SCIF1_0),
+ PINMUX_IPSR_MSEL(IP12_19_16, HTX1_A, SEL_HSCIF1_0),
+ PINMUX_IPSR_MSEL(IP12_19_16, TS_SDEN0_C, SEL_TSIF0_2),
+ PINMUX_IPSR_MSEL(IP12_19_16, STP_ISEN_0_C, SEL_SSP1_0_2),
+ PINMUX_IPSR_MSEL(IP12_19_16, RIF1_D0_C, SEL_DRIF1_2),
- PINMUX_IPSR_GPSR(IP12_23_20, HCTS0_N),
- PINMUX_IPSR_MSEL(IP12_23_20, RX2_B, SEL_SCIF2_1),
- PINMUX_IPSR_MSEL(IP12_23_20, MSIOF1_SYNC_D, SEL_MSIOF1_3),
- PINMUX_IPSR_MSEL(IP12_23_20, SSI_SCK9_A, SEL_SSI_0),
- PINMUX_IPSR_MSEL(IP12_23_20, TS_SPSYNC0_D, SEL_TSIF0_3),
- PINMUX_IPSR_MSEL(IP12_23_20, STP_ISSYNC_0_D, SEL_SSP1_0_3),
- PINMUX_IPSR_MSEL(IP12_23_20, RIF0_SYNC_C, SEL_DRIF0_2),
- PINMUX_IPSR_MSEL(IP12_23_20, AUDIO_CLKOUT1_A, SEL_ADG_0),
+ PINMUX_IPSR_GPSR(IP12_23_20, CTS1_N),
+ PINMUX_IPSR_MSEL(IP12_23_20, HCTS1_N_A, SEL_HSCIF1_0),
+ PINMUX_IPSR_MSEL(IP12_23_20, MSIOF1_RXD_B, SEL_MSIOF1_1),
+ PINMUX_IPSR_MSEL(IP12_23_20, TS_SDEN1_C, SEL_TSIF1_2),
+ PINMUX_IPSR_MSEL(IP12_23_20, STP_ISEN_1_C, SEL_SSP1_1_2),
+ PINMUX_IPSR_MSEL(IP12_23_20, RIF1_D0_B, SEL_DRIF1_1),
+ PINMUX_IPSR_GPSR(IP12_23_20, ADIDATA),
- PINMUX_IPSR_GPSR(IP12_27_24, HRTS0_N),
- PINMUX_IPSR_MSEL(IP12_27_24, TX2_B, SEL_SCIF2_1),
- PINMUX_IPSR_MSEL(IP12_27_24, MSIOF1_SS1_D, SEL_MSIOF1_3),
- PINMUX_IPSR_MSEL(IP12_27_24, SSI_WS9_A, SEL_SSI_0),
- PINMUX_IPSR_MSEL(IP12_27_24, STP_IVCXO27_0_D, SEL_SSP1_0_3),
- PINMUX_IPSR_MSEL(IP12_27_24, BPFCLK_A, SEL_FM_0),
- PINMUX_IPSR_MSEL(IP12_27_24, AUDIO_CLKOUT2_A, SEL_ADG_0),
+ PINMUX_IPSR_GPSR(IP12_27_24, RTS1_N_TANS),
+ PINMUX_IPSR_MSEL(IP12_27_24, HRTS1_N_A, SEL_HSCIF1_0),
+ PINMUX_IPSR_MSEL(IP12_27_24, MSIOF1_TXD_B, SEL_MSIOF1_1),
+ PINMUX_IPSR_MSEL(IP12_27_24, TS_SDAT1_C, SEL_TSIF1_2),
+ PINMUX_IPSR_MSEL(IP12_27_24, STP_ISD_1_C, SEL_SSP1_1_2),
+ PINMUX_IPSR_MSEL(IP12_27_24, RIF1_D1_B, SEL_DRIF1_1),
+ PINMUX_IPSR_GPSR(IP12_27_24, ADICHS0),
- PINMUX_IPSR_GPSR(IP12_31_28, MSIOF0_SYNC),
- PINMUX_IPSR_MSEL(IP12_31_28, AUDIO_CLKOUT_A, SEL_ADG_0),
+ PINMUX_IPSR_GPSR(IP12_31_28, SCK2),
+ PINMUX_IPSR_MSEL(IP12_31_28, SCIF_CLK_B, SEL_SCIF1_1),
+ PINMUX_IPSR_MSEL(IP12_31_28, MSIOF1_SCK_B, SEL_MSIOF1_1),
+ PINMUX_IPSR_MSEL(IP12_31_28, TS_SCK1_C, SEL_TSIF1_2),
+ PINMUX_IPSR_MSEL(IP12_31_28, STP_ISCLK_1_C, SEL_SSP1_1_2),
+ PINMUX_IPSR_MSEL(IP12_31_28, RIF1_CLK_B, SEL_DRIF1_1),
+ PINMUX_IPSR_GPSR(IP12_31_28, ADICLK),
/* IPSR13 */
- PINMUX_IPSR_GPSR(IP13_3_0, MSIOF0_SS1),
- PINMUX_IPSR_GPSR(IP13_3_0, RX5),
- PINMUX_IPSR_MSEL(IP13_3_0, AUDIO_CLKA_C, SEL_ADG_2),
- PINMUX_IPSR_MSEL(IP13_3_0, SSI_SCK2_A, SEL_SSI_0),
- PINMUX_IPSR_MSEL(IP13_3_0, STP_IVCXO27_0_C, SEL_SSP1_0_2),
- PINMUX_IPSR_MSEL(IP13_3_0, AUDIO_CLKOUT3_A, SEL_ADG_0),
- PINMUX_IPSR_MSEL(IP13_3_0, TCLK1_B, SEL_TIMER_TMU_1),
+ PINMUX_IPSR_MSEL(IP13_3_0, TX2_A, SEL_SCIF2_0),
+ PINMUX_IPSR_MSEL(IP13_3_0, SD2_CD_B, SEL_SDHI2_1),
+ PINMUX_IPSR_MSEL(IP13_3_0, SCL1_A, SEL_I2C1_0),
+ PINMUX_IPSR_MSEL(IP13_3_0, FMCLK_A, SEL_FM_0),
+ PINMUX_IPSR_MSEL(IP13_3_0, RIF1_D1_C, SEL_DRIF1_2),
+ PINMUX_IPSR_GPSR(IP13_3_0, FSO_CFE_0_N),
- PINMUX_IPSR_GPSR(IP13_7_4, MSIOF0_SS2),
- PINMUX_IPSR_GPSR(IP13_7_4, TX5),
- PINMUX_IPSR_MSEL(IP13_7_4, MSIOF1_SS2_D, SEL_MSIOF1_3),
- PINMUX_IPSR_MSEL(IP13_7_4, AUDIO_CLKC_A, SEL_ADG_0),
- PINMUX_IPSR_MSEL(IP13_7_4, SSI_WS2_A, SEL_SSI_0),
- PINMUX_IPSR_MSEL(IP13_7_4, STP_OPWM_0_D, SEL_SSP1_0_3),
- PINMUX_IPSR_MSEL(IP13_7_4, AUDIO_CLKOUT_D, SEL_ADG_3),
- PINMUX_IPSR_MSEL(IP13_7_4, SPEEDIN_B, SEL_SPEED_PULSE_1),
+ PINMUX_IPSR_MSEL(IP13_7_4, RX2_A, SEL_SCIF2_0),
+ PINMUX_IPSR_MSEL(IP13_7_4, SD2_WP_B, SEL_SDHI2_1),
+ PINMUX_IPSR_MSEL(IP13_7_4, SDA1_A, SEL_I2C1_0),
+ PINMUX_IPSR_MSEL(IP13_7_4, FMIN_A, SEL_FM_0),
+ PINMUX_IPSR_MSEL(IP13_7_4, RIF1_SYNC_C, SEL_DRIF1_2),
+ PINMUX_IPSR_GPSR(IP13_7_4, FSO_CFE_1_N),
- PINMUX_IPSR_GPSR(IP13_11_8, MLB_CLK),
- PINMUX_IPSR_MSEL(IP13_11_8, MSIOF1_SCK_F, SEL_MSIOF1_5),
- PINMUX_IPSR_MSEL(IP13_11_8, SCL1_B, SEL_I2C1_1),
+ PINMUX_IPSR_GPSR(IP13_11_8, HSCK0),
+ PINMUX_IPSR_MSEL(IP13_11_8, MSIOF1_SCK_D, SEL_MSIOF1_3),
+ PINMUX_IPSR_MSEL(IP13_11_8, AUDIO_CLKB_A, SEL_ADG_B_0),
+ PINMUX_IPSR_MSEL(IP13_11_8, SSI_SDATA1_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP13_11_8, TS_SCK0_D, SEL_TSIF0_3),
+ PINMUX_IPSR_MSEL(IP13_11_8, STP_ISCLK_0_D, SEL_SSP1_0_3),
+ PINMUX_IPSR_MSEL(IP13_11_8, RIF0_CLK_C, SEL_DRIF0_2),
+ PINMUX_IPSR_MSEL(IP13_11_8, RX5_B, SEL_SCIF5_1),
- PINMUX_IPSR_GPSR(IP13_15_12, MLB_SIG),
- PINMUX_IPSR_MSEL(IP13_15_12, RX1_B, SEL_SCIF1_1),
- PINMUX_IPSR_MSEL(IP13_15_12, MSIOF1_SYNC_F, SEL_MSIOF1_5),
- PINMUX_IPSR_MSEL(IP13_15_12, SDA1_B, SEL_I2C1_1),
+ PINMUX_IPSR_GPSR(IP13_15_12, HRX0),
+ PINMUX_IPSR_MSEL(IP13_15_12, MSIOF1_RXD_D, SEL_MSIOF1_3),
+ PINMUX_IPSR_MSEL(IP13_15_12, SSI_SDATA2_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP13_15_12, TS_SDEN0_D, SEL_TSIF0_3),
+ PINMUX_IPSR_MSEL(IP13_15_12, STP_ISEN_0_D, SEL_SSP1_0_3),
+ PINMUX_IPSR_MSEL(IP13_15_12, RIF0_D0_C, SEL_DRIF0_2),
- PINMUX_IPSR_GPSR(IP13_19_16, MLB_DAT),
- PINMUX_IPSR_MSEL(IP13_19_16, TX1_B, SEL_SCIF1_1),
- PINMUX_IPSR_MSEL(IP13_19_16, MSIOF1_RXD_F, SEL_MSIOF1_5),
+ PINMUX_IPSR_GPSR(IP13_19_16, HTX0),
+ PINMUX_IPSR_MSEL(IP13_19_16, MSIOF1_TXD_D, SEL_MSIOF1_3),
+ PINMUX_IPSR_MSEL(IP13_19_16, SSI_SDATA9_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP13_19_16, TS_SDAT0_D, SEL_TSIF0_3),
+ PINMUX_IPSR_MSEL(IP13_19_16, STP_ISD_0_D, SEL_SSP1_0_3),
+ PINMUX_IPSR_MSEL(IP13_19_16, RIF0_D1_C, SEL_DRIF0_2),
- PINMUX_IPSR_GPSR(IP13_23_20, SSI_SCK01239),
- PINMUX_IPSR_MSEL(IP13_23_20, MSIOF1_TXD_F, SEL_MSIOF1_5),
+ PINMUX_IPSR_GPSR(IP13_23_20, HCTS0_N),
+ PINMUX_IPSR_MSEL(IP13_23_20, RX2_B, SEL_SCIF2_1),
+ PINMUX_IPSR_MSEL(IP13_23_20, MSIOF1_SYNC_D, SEL_MSIOF1_3),
+ PINMUX_IPSR_MSEL(IP13_23_20, SSI_SCK9_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP13_23_20, TS_SPSYNC0_D, SEL_TSIF0_3),
+ PINMUX_IPSR_MSEL(IP13_23_20, STP_ISSYNC_0_D, SEL_SSP1_0_3),
+ PINMUX_IPSR_MSEL(IP13_23_20, RIF0_SYNC_C, SEL_DRIF0_2),
+ PINMUX_IPSR_GPSR(IP13_23_20, AUDIO_CLKOUT1_A),
- PINMUX_IPSR_GPSR(IP13_27_24, SSI_WS01239),
- PINMUX_IPSR_MSEL(IP13_27_24, MSIOF1_SS1_F, SEL_MSIOF1_5),
+ PINMUX_IPSR_GPSR(IP13_27_24, HRTS0_N),
+ PINMUX_IPSR_MSEL(IP13_27_24, TX2_B, SEL_SCIF2_1),
+ PINMUX_IPSR_MSEL(IP13_27_24, MSIOF1_SS1_D, SEL_MSIOF1_3),
+ PINMUX_IPSR_MSEL(IP13_27_24, SSI_WS9_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP13_27_24, STP_IVCXO27_0_D, SEL_SSP1_0_3),
+ PINMUX_IPSR_MSEL(IP13_27_24, BPFCLK_A, SEL_FM_0),
+ PINMUX_IPSR_GPSR(IP13_27_24, AUDIO_CLKOUT2_A),
- PINMUX_IPSR_GPSR(IP13_31_28, SSI_SDATA0),
- PINMUX_IPSR_MSEL(IP13_31_28, MSIOF1_SS2_F, SEL_MSIOF1_5),
+ PINMUX_IPSR_GPSR(IP13_31_28, MSIOF0_SYNC),
+ PINMUX_IPSR_GPSR(IP13_31_28, AUDIO_CLKOUT_A),
+ PINMUX_IPSR_MSEL(IP13_31_28, TX5_B, SEL_SCIF5_1),
+ PINMUX_IPSR_MSEL(IP13_31_28, BPFCLK_D, SEL_FM_3),
/* IPSR14 */
- PINMUX_IPSR_MSEL(IP14_3_0, SSI_SDATA1_A, SEL_SSI_0),
+ PINMUX_IPSR_GPSR(IP14_3_0, MSIOF0_SS1),
+ PINMUX_IPSR_MSEL(IP14_3_0, RX5_A, SEL_SCIF5_0),
+ PINMUX_IPSR_MSEL(IP14_3_0, NFWP_N_A, SEL_NDF_0),
+ PINMUX_IPSR_MSEL(IP14_3_0, AUDIO_CLKA_C, SEL_ADG_A_2),
+ PINMUX_IPSR_MSEL(IP14_3_0, SSI_SCK2_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP14_3_0, STP_IVCXO27_0_C, SEL_SSP1_0_2),
+ PINMUX_IPSR_GPSR(IP14_3_0, AUDIO_CLKOUT3_A),
+ PINMUX_IPSR_MSEL(IP14_3_0, TCLK1_B, SEL_TIMER_TMU_1),
- PINMUX_IPSR_MSEL(IP14_7_4, SSI_SDATA2_A, SEL_SSI_0),
- PINMUX_IPSR_MSEL(IP14_7_4, SSI_SCK1_B, SEL_SSI_1),
+ PINMUX_IPSR_GPSR(IP14_7_4, MSIOF0_SS2),
+ PINMUX_IPSR_MSEL(IP14_7_4, TX5_A, SEL_SCIF5_0),
+ PINMUX_IPSR_MSEL(IP14_7_4, MSIOF1_SS2_D, SEL_MSIOF1_3),
+ PINMUX_IPSR_MSEL(IP14_7_4, AUDIO_CLKC_A, SEL_ADG_C_0),
+ PINMUX_IPSR_MSEL(IP14_7_4, SSI_WS2_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP14_7_4, STP_OPWM_0_D, SEL_SSP1_0_3),
+ PINMUX_IPSR_GPSR(IP14_7_4, AUDIO_CLKOUT_D),
+ PINMUX_IPSR_MSEL(IP14_7_4, SPEEDIN_B, SEL_SPEED_PULSE_1),
- PINMUX_IPSR_GPSR(IP14_11_8, SSI_SCK34),
- PINMUX_IPSR_MSEL(IP14_11_8, MSIOF1_SS1_A, SEL_MSIOF1_0),
- PINMUX_IPSR_MSEL(IP14_11_8, STP_OPWM_0_A, SEL_SSP1_0_0),
+ PINMUX_IPSR_GPSR(IP14_11_8, MLB_CLK),
+ PINMUX_IPSR_MSEL(IP14_11_8, MSIOF1_SCK_F, SEL_MSIOF1_5),
+ PINMUX_IPSR_MSEL(IP14_11_8, SCL1_B, SEL_I2C1_1),
- PINMUX_IPSR_GPSR(IP14_15_12, SSI_WS34),
- PINMUX_IPSR_MSEL(IP14_15_12, HCTS2_N_A, SEL_HSCIF2_0),
- PINMUX_IPSR_MSEL(IP14_15_12, MSIOF1_SS2_A, SEL_MSIOF1_0),
- PINMUX_IPSR_MSEL(IP14_15_12, STP_IVCXO27_0_A, SEL_SSP1_0_0),
+ PINMUX_IPSR_GPSR(IP14_15_12, MLB_SIG),
+ PINMUX_IPSR_MSEL(IP14_15_12, RX1_B, SEL_SCIF1_1),
+ PINMUX_IPSR_MSEL(IP14_15_12, MSIOF1_SYNC_F, SEL_MSIOF1_5),
+ PINMUX_IPSR_MSEL(IP14_15_12, SDA1_B, SEL_I2C1_1),
- PINMUX_IPSR_GPSR(IP14_19_16, SSI_SDATA3),
- PINMUX_IPSR_MSEL(IP14_19_16, HRTS2_N_A, SEL_HSCIF2_0),
- PINMUX_IPSR_MSEL(IP14_19_16, MSIOF1_TXD_A, SEL_MSIOF1_0),
- PINMUX_IPSR_MSEL(IP14_19_16, TS_SCK0_A, SEL_TSIF0_0),
- PINMUX_IPSR_MSEL(IP14_19_16, STP_ISCLK_0_A, SEL_SSP1_0_0),
- PINMUX_IPSR_MSEL(IP14_19_16, RIF0_D1_A, SEL_DRIF0_0),
- PINMUX_IPSR_MSEL(IP14_19_16, RIF2_D0_A, SEL_DRIF2_0),
+ PINMUX_IPSR_GPSR(IP14_19_16, MLB_DAT),
+ PINMUX_IPSR_MSEL(IP14_19_16, TX1_B, SEL_SCIF1_1),
+ PINMUX_IPSR_MSEL(IP14_19_16, MSIOF1_RXD_F, SEL_MSIOF1_5),
- PINMUX_IPSR_GPSR(IP14_23_20, SSI_SCK4),
- PINMUX_IPSR_MSEL(IP14_23_20, HRX2_A, SEL_HSCIF2_0),
- PINMUX_IPSR_MSEL(IP14_23_20, MSIOF1_SCK_A, SEL_MSIOF1_0),
- PINMUX_IPSR_MSEL(IP14_23_20, TS_SDAT0_A, SEL_TSIF0_0),
- PINMUX_IPSR_MSEL(IP14_23_20, STP_ISD_0_A, SEL_SSP1_0_0),
- PINMUX_IPSR_MSEL(IP14_23_20, RIF0_CLK_A, SEL_DRIF0_0),
- PINMUX_IPSR_MSEL(IP14_23_20, RIF2_CLK_A, SEL_DRIF2_0),
+ PINMUX_IPSR_GPSR(IP14_23_20, SSI_SCK01239),
+ PINMUX_IPSR_MSEL(IP14_23_20, MSIOF1_TXD_F, SEL_MSIOF1_5),
- PINMUX_IPSR_GPSR(IP14_27_24, SSI_WS4),
- PINMUX_IPSR_MSEL(IP14_27_24, HTX2_A, SEL_HSCIF2_0),
- PINMUX_IPSR_MSEL(IP14_27_24, MSIOF1_SYNC_A, SEL_MSIOF1_0),
- PINMUX_IPSR_MSEL(IP14_27_24, TS_SDEN0_A, SEL_TSIF0_0),
- PINMUX_IPSR_MSEL(IP14_27_24, STP_ISEN_0_A, SEL_SSP1_0_0),
- PINMUX_IPSR_MSEL(IP14_27_24, RIF0_SYNC_A, SEL_DRIF0_0),
- PINMUX_IPSR_MSEL(IP14_27_24, RIF2_SYNC_A, SEL_DRIF2_0),
+ PINMUX_IPSR_GPSR(IP14_27_24, SSI_WS01239),
+ PINMUX_IPSR_MSEL(IP14_27_24, MSIOF1_SS1_F, SEL_MSIOF1_5),
- PINMUX_IPSR_GPSR(IP14_31_28, SSI_SDATA4),
- PINMUX_IPSR_MSEL(IP14_31_28, HSCK2_A, SEL_HSCIF2_0),
- PINMUX_IPSR_MSEL(IP14_31_28, MSIOF1_RXD_A, SEL_MSIOF1_0),
- PINMUX_IPSR_MSEL(IP14_31_28, TS_SPSYNC0_A, SEL_TSIF0_0),
- PINMUX_IPSR_MSEL(IP14_31_28, STP_ISSYNC_0_A, SEL_SSP1_0_0),
- PINMUX_IPSR_MSEL(IP14_31_28, RIF0_D0_A, SEL_DRIF0_0),
- PINMUX_IPSR_MSEL(IP14_31_28, RIF2_D1_A, SEL_DRIF2_0),
+ PINMUX_IPSR_GPSR(IP14_31_28, SSI_SDATA0),
+ PINMUX_IPSR_MSEL(IP14_31_28, MSIOF1_SS2_F, SEL_MSIOF1_5),
/* IPSR15 */
- PINMUX_IPSR_GPSR(IP15_3_0, SSI_SCK6),
- PINMUX_IPSR_GPSR(IP15_3_0, USB2_PWEN),
- PINMUX_IPSR_MSEL(IP15_3_0, SIM0_RST_D, SEL_SIMCARD_3),
+ PINMUX_IPSR_MSEL(IP15_3_0, SSI_SDATA1_A, SEL_SSI_0),
- PINMUX_IPSR_GPSR(IP15_7_4, SSI_WS6),
- PINMUX_IPSR_GPSR(IP15_7_4, USB2_OVC),
- PINMUX_IPSR_MSEL(IP15_7_4, SIM0_D_D, SEL_SIMCARD_3),
+ PINMUX_IPSR_MSEL(IP15_7_4, SSI_SDATA2_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP15_7_4, SSI_SCK1_B, SEL_SSI_1),
- PINMUX_IPSR_GPSR(IP15_11_8, SSI_SDATA6),
- PINMUX_IPSR_MSEL(IP15_11_8, SIM0_CLK_D, SEL_SIMCARD_3),
- PINMUX_IPSR_MSEL(IP15_11_8, SATA_DEVSLP_A, SEL_SATA_0),
+ PINMUX_IPSR_GPSR(IP15_11_8, SSI_SCK34),
+ PINMUX_IPSR_MSEL(IP15_11_8, MSIOF1_SS1_A, SEL_MSIOF1_0),
+ PINMUX_IPSR_MSEL(IP15_11_8, STP_OPWM_0_A, SEL_SSP1_0_0),
- PINMUX_IPSR_GPSR(IP15_15_12, SSI_SCK78),
- PINMUX_IPSR_MSEL(IP15_15_12, HRX2_B, SEL_HSCIF2_1),
- PINMUX_IPSR_MSEL(IP15_15_12, MSIOF1_SCK_C, SEL_MSIOF1_2),
- PINMUX_IPSR_MSEL(IP15_15_12, TS_SCK1_A, SEL_TSIF1_0),
- PINMUX_IPSR_MSEL(IP15_15_12, STP_ISCLK_1_A, SEL_SSP1_1_0),
- PINMUX_IPSR_MSEL(IP15_15_12, RIF1_CLK_A, SEL_DRIF1_0),
- PINMUX_IPSR_MSEL(IP15_15_12, RIF3_CLK_A, SEL_DRIF3_0),
+ PINMUX_IPSR_GPSR(IP15_15_12, SSI_WS34),
+ PINMUX_IPSR_MSEL(IP15_15_12, HCTS2_N_A, SEL_HSCIF2_0),
+ PINMUX_IPSR_MSEL(IP15_15_12, MSIOF1_SS2_A, SEL_MSIOF1_0),
+ PINMUX_IPSR_MSEL(IP15_15_12, STP_IVCXO27_0_A, SEL_SSP1_0_0),
- PINMUX_IPSR_GPSR(IP15_19_16, SSI_WS78),
- PINMUX_IPSR_MSEL(IP15_19_16, HTX2_B, SEL_HSCIF2_1),
- PINMUX_IPSR_MSEL(IP15_19_16, MSIOF1_SYNC_C, SEL_MSIOF1_2),
- PINMUX_IPSR_MSEL(IP15_19_16, TS_SDAT1_A, SEL_TSIF1_0),
- PINMUX_IPSR_MSEL(IP15_19_16, STP_ISD_1_A, SEL_SSP1_1_0),
- PINMUX_IPSR_MSEL(IP15_19_16, RIF1_SYNC_A, SEL_DRIF1_0),
- PINMUX_IPSR_MSEL(IP15_19_16, RIF3_SYNC_A, SEL_DRIF3_0),
+ PINMUX_IPSR_GPSR(IP15_19_16, SSI_SDATA3),
+ PINMUX_IPSR_MSEL(IP15_19_16, HRTS2_N_A, SEL_HSCIF2_0),
+ PINMUX_IPSR_MSEL(IP15_19_16, MSIOF1_TXD_A, SEL_MSIOF1_0),
+ PINMUX_IPSR_MSEL(IP15_19_16, TS_SCK0_A, SEL_TSIF0_0),
+ PINMUX_IPSR_MSEL(IP15_19_16, STP_ISCLK_0_A, SEL_SSP1_0_0),
+ PINMUX_IPSR_MSEL(IP15_19_16, RIF0_D1_A, SEL_DRIF0_0),
+ PINMUX_IPSR_MSEL(IP15_19_16, RIF2_D0_A, SEL_DRIF2_0),
- PINMUX_IPSR_GPSR(IP15_23_20, SSI_SDATA7),
- PINMUX_IPSR_MSEL(IP15_23_20, HCTS2_N_B, SEL_HSCIF2_1),
- PINMUX_IPSR_MSEL(IP15_23_20, MSIOF1_RXD_C, SEL_MSIOF1_2),
- PINMUX_IPSR_MSEL(IP15_23_20, TS_SDEN1_A, SEL_TSIF1_0),
- PINMUX_IPSR_MSEL(IP15_23_20, STP_ISEN_1_A, SEL_SSP1_1_0),
- PINMUX_IPSR_MSEL(IP15_23_20, RIF1_D0_A, SEL_DRIF1_0),
- PINMUX_IPSR_MSEL(IP15_23_20, RIF3_D0_A, SEL_DRIF3_0),
- PINMUX_IPSR_MSEL(IP15_23_20, TCLK2_A, SEL_TIMER_TMU_0),
+ PINMUX_IPSR_GPSR(IP15_23_20, SSI_SCK4),
+ PINMUX_IPSR_MSEL(IP15_23_20, HRX2_A, SEL_HSCIF2_0),
+ PINMUX_IPSR_MSEL(IP15_23_20, MSIOF1_SCK_A, SEL_MSIOF1_0),
+ PINMUX_IPSR_MSEL(IP15_23_20, TS_SDAT0_A, SEL_TSIF0_0),
+ PINMUX_IPSR_MSEL(IP15_23_20, STP_ISD_0_A, SEL_SSP1_0_0),
+ PINMUX_IPSR_MSEL(IP15_23_20, RIF0_CLK_A, SEL_DRIF0_0),
+ PINMUX_IPSR_MSEL(IP15_23_20, RIF2_CLK_A, SEL_DRIF2_0),
- PINMUX_IPSR_GPSR(IP15_27_24, SSI_SDATA8),
- PINMUX_IPSR_MSEL(IP15_27_24, HRTS2_N_B, SEL_HSCIF2_1),
- PINMUX_IPSR_MSEL(IP15_27_24, MSIOF1_TXD_C, SEL_MSIOF1_2),
- PINMUX_IPSR_MSEL(IP15_27_24, TS_SPSYNC1_A, SEL_TSIF1_0),
- PINMUX_IPSR_MSEL(IP15_27_24, STP_ISSYNC_1_A, SEL_SSP1_1_0),
- PINMUX_IPSR_MSEL(IP15_27_24, RIF1_D1_A, SEL_DRIF1_0),
- PINMUX_IPSR_MSEL(IP15_27_24, RIF3_D1_A, SEL_DRIF3_0),
+ PINMUX_IPSR_GPSR(IP15_27_24, SSI_WS4),
+ PINMUX_IPSR_MSEL(IP15_27_24, HTX2_A, SEL_HSCIF2_0),
+ PINMUX_IPSR_MSEL(IP15_27_24, MSIOF1_SYNC_A, SEL_MSIOF1_0),
+ PINMUX_IPSR_MSEL(IP15_27_24, TS_SDEN0_A, SEL_TSIF0_0),
+ PINMUX_IPSR_MSEL(IP15_27_24, STP_ISEN_0_A, SEL_SSP1_0_0),
+ PINMUX_IPSR_MSEL(IP15_27_24, RIF0_SYNC_A, SEL_DRIF0_0),
+ PINMUX_IPSR_MSEL(IP15_27_24, RIF2_SYNC_A, SEL_DRIF2_0),
- PINMUX_IPSR_MSEL(IP15_31_28, SSI_SDATA9_A, SEL_SSI_0),
- PINMUX_IPSR_MSEL(IP15_31_28, HSCK2_B, SEL_HSCIF2_1),
- PINMUX_IPSR_MSEL(IP15_31_28, MSIOF1_SS1_C, SEL_MSIOF1_2),
- PINMUX_IPSR_MSEL(IP15_31_28, HSCK1_A, SEL_HSCIF1_0),
- PINMUX_IPSR_MSEL(IP15_31_28, SSI_WS1_B, SEL_SSI_1),
- PINMUX_IPSR_GPSR(IP15_31_28, SCK1),
- PINMUX_IPSR_MSEL(IP15_31_28, STP_IVCXO27_1_A, SEL_SSP1_1_0),
- PINMUX_IPSR_GPSR(IP15_31_28, SCK5),
+ PINMUX_IPSR_GPSR(IP15_31_28, SSI_SDATA4),
+ PINMUX_IPSR_MSEL(IP15_31_28, HSCK2_A, SEL_HSCIF2_0),
+ PINMUX_IPSR_MSEL(IP15_31_28, MSIOF1_RXD_A, SEL_MSIOF1_0),
+ PINMUX_IPSR_MSEL(IP15_31_28, TS_SPSYNC0_A, SEL_TSIF0_0),
+ PINMUX_IPSR_MSEL(IP15_31_28, STP_ISSYNC_0_A, SEL_SSP1_0_0),
+ PINMUX_IPSR_MSEL(IP15_31_28, RIF0_D0_A, SEL_DRIF0_0),
+ PINMUX_IPSR_MSEL(IP15_31_28, RIF2_D1_A, SEL_DRIF2_0),
/* IPSR16 */
- PINMUX_IPSR_MSEL(IP16_3_0, AUDIO_CLKA_A, SEL_ADG_0),
- PINMUX_IPSR_GPSR(IP16_3_0, CC5_OSCOUT),
+ PINMUX_IPSR_GPSR(IP16_3_0, SSI_SCK6),
+ PINMUX_IPSR_GPSR(IP16_3_0, USB2_PWEN),
+ PINMUX_IPSR_MSEL(IP16_3_0, SIM0_RST_D, SEL_SIMCARD_3),
- PINMUX_IPSR_MSEL(IP16_7_4, AUDIO_CLKB_B, SEL_ADG_1),
- PINMUX_IPSR_MSEL(IP16_7_4, SCIF_CLK_A, SEL_SCIF1_0),
- PINMUX_IPSR_MSEL(IP16_7_4, STP_IVCXO27_1_D, SEL_SSP1_1_3),
- PINMUX_IPSR_MSEL(IP16_7_4, REMOCON_A, SEL_REMOCON_0),
- PINMUX_IPSR_MSEL(IP16_7_4, TCLK1_A, SEL_TIMER_TMU_0),
+ PINMUX_IPSR_GPSR(IP16_7_4, SSI_WS6),
+ PINMUX_IPSR_GPSR(IP16_7_4, USB2_OVC),
+ PINMUX_IPSR_MSEL(IP16_7_4, SIM0_D_D, SEL_SIMCARD_3),
- PINMUX_IPSR_GPSR(IP16_11_8, USB0_PWEN),
- PINMUX_IPSR_MSEL(IP16_11_8, SIM0_RST_C, SEL_SIMCARD_2),
- PINMUX_IPSR_MSEL(IP16_11_8, TS_SCK1_D, SEL_TSIF1_3),
- PINMUX_IPSR_MSEL(IP16_11_8, STP_ISCLK_1_D, SEL_SSP1_1_3),
- PINMUX_IPSR_MSEL(IP16_11_8, BPFCLK_B, SEL_FM_1),
- PINMUX_IPSR_MSEL(IP16_11_8, RIF3_CLK_B, SEL_DRIF3_1),
+ PINMUX_IPSR_GPSR(IP16_11_8, SSI_SDATA6),
+ PINMUX_IPSR_MSEL(IP16_11_8, SIM0_CLK_D, SEL_SIMCARD_3),
+ PINMUX_IPSR_GPSR(IP16_11_8, SATA_DEVSLP_A),
- PINMUX_IPSR_GPSR(IP16_15_12, USB0_OVC),
- PINMUX_IPSR_MSEL(IP16_11_8, SIM0_D_C, SEL_SIMCARD_2),
- PINMUX_IPSR_MSEL(IP16_11_8, TS_SDAT1_D, SEL_TSIF1_3),
- PINMUX_IPSR_MSEL(IP16_11_8, STP_ISD_1_D, SEL_SSP1_1_3),
- PINMUX_IPSR_MSEL(IP16_11_8, RIF3_SYNC_B, SEL_DRIF3_1),
+ PINMUX_IPSR_GPSR(IP16_15_12, SSI_SCK78),
+ PINMUX_IPSR_MSEL(IP16_15_12, HRX2_B, SEL_HSCIF2_1),
+ PINMUX_IPSR_MSEL(IP16_15_12, MSIOF1_SCK_C, SEL_MSIOF1_2),
+ PINMUX_IPSR_MSEL(IP16_15_12, TS_SCK1_A, SEL_TSIF1_0),
+ PINMUX_IPSR_MSEL(IP16_15_12, STP_ISCLK_1_A, SEL_SSP1_1_0),
+ PINMUX_IPSR_MSEL(IP16_15_12, RIF1_CLK_A, SEL_DRIF1_0),
+ PINMUX_IPSR_MSEL(IP16_15_12, RIF3_CLK_A, SEL_DRIF3_0),
- PINMUX_IPSR_GPSR(IP16_19_16, USB1_PWEN),
- PINMUX_IPSR_MSEL(IP16_19_16, SIM0_CLK_C, SEL_SIMCARD_2),
- PINMUX_IPSR_MSEL(IP16_19_16, SSI_SCK1_A, SEL_SSI_0),
- PINMUX_IPSR_MSEL(IP16_19_16, TS_SCK0_E, SEL_TSIF0_4),
- PINMUX_IPSR_MSEL(IP16_19_16, STP_ISCLK_0_E, SEL_SSP1_0_4),
- PINMUX_IPSR_MSEL(IP16_19_16, FMCLK_B, SEL_FM_1),
- PINMUX_IPSR_MSEL(IP16_19_16, RIF2_CLK_B, SEL_DRIF2_1),
- PINMUX_IPSR_MSEL(IP16_19_16, SPEEDIN_A, SEL_SPEED_PULSE_0),
+ PINMUX_IPSR_GPSR(IP16_19_16, SSI_WS78),
+ PINMUX_IPSR_MSEL(IP16_19_16, HTX2_B, SEL_HSCIF2_1),
+ PINMUX_IPSR_MSEL(IP16_19_16, MSIOF1_SYNC_C, SEL_MSIOF1_2),
+ PINMUX_IPSR_MSEL(IP16_19_16, TS_SDAT1_A, SEL_TSIF1_0),
+ PINMUX_IPSR_MSEL(IP16_19_16, STP_ISD_1_A, SEL_SSP1_1_0),
+ PINMUX_IPSR_MSEL(IP16_19_16, RIF1_SYNC_A, SEL_DRIF1_0),
+ PINMUX_IPSR_MSEL(IP16_19_16, RIF3_SYNC_A, SEL_DRIF3_0),
- PINMUX_IPSR_GPSR(IP16_23_20, USB1_OVC),
- PINMUX_IPSR_MSEL(IP16_23_20, MSIOF1_SS2_C, SEL_MSIOF1_2),
- PINMUX_IPSR_MSEL(IP16_23_20, SSI_WS1_A, SEL_SSI_0),
- PINMUX_IPSR_MSEL(IP16_23_20, TS_SDAT0_E, SEL_TSIF0_4),
- PINMUX_IPSR_MSEL(IP16_23_20, STP_ISD_0_E, SEL_SSP1_0_4),
- PINMUX_IPSR_MSEL(IP16_23_20, FMIN_B, SEL_FM_1),
- PINMUX_IPSR_MSEL(IP16_23_20, RIF2_SYNC_B, SEL_DRIF2_1),
- PINMUX_IPSR_MSEL(IP16_23_20, REMOCON_B, SEL_REMOCON_1),
+ PINMUX_IPSR_GPSR(IP16_23_20, SSI_SDATA7),
+ PINMUX_IPSR_MSEL(IP16_23_20, HCTS2_N_B, SEL_HSCIF2_1),
+ PINMUX_IPSR_MSEL(IP16_23_20, MSIOF1_RXD_C, SEL_MSIOF1_2),
+ PINMUX_IPSR_MSEL(IP16_23_20, TS_SDEN1_A, SEL_TSIF1_0),
+ PINMUX_IPSR_MSEL(IP16_23_20, STP_ISEN_1_A, SEL_SSP1_1_0),
+ PINMUX_IPSR_MSEL(IP16_23_20, RIF1_D0_A, SEL_DRIF1_0),
+ PINMUX_IPSR_MSEL(IP16_23_20, RIF3_D0_A, SEL_DRIF3_0),
+ PINMUX_IPSR_MSEL(IP16_23_20, TCLK2_A, SEL_TIMER_TMU_0),
- PINMUX_IPSR_GPSR(IP16_27_24, USB30_PWEN),
- PINMUX_IPSR_MSEL(IP16_27_24, AUDIO_CLKOUT_B, SEL_ADG_1),
- PINMUX_IPSR_MSEL(IP16_27_24, SSI_SCK2_B, SEL_SSI_1),
- PINMUX_IPSR_MSEL(IP16_27_24, TS_SDEN1_D, SEL_TSIF1_3),
- PINMUX_IPSR_MSEL(IP16_27_24, STP_ISEN_1_D, SEL_SSP1_1_2),
- PINMUX_IPSR_MSEL(IP16_27_24, STP_OPWM_0_E, SEL_SSP1_0_4),
- PINMUX_IPSR_MSEL(IP16_27_24, RIF3_D0_B, SEL_DRIF3_1),
- PINMUX_IPSR_MSEL(IP16_27_24, TCLK2_B, SEL_TIMER_TMU_1),
- PINMUX_IPSR_GPSR(IP16_27_24, TPU0TO0),
+ PINMUX_IPSR_GPSR(IP16_27_24, SSI_SDATA8),
+ PINMUX_IPSR_MSEL(IP16_27_24, HRTS2_N_B, SEL_HSCIF2_1),
+ PINMUX_IPSR_MSEL(IP16_27_24, MSIOF1_TXD_C, SEL_MSIOF1_2),
+ PINMUX_IPSR_MSEL(IP16_27_24, TS_SPSYNC1_A, SEL_TSIF1_0),
+ PINMUX_IPSR_MSEL(IP16_27_24, STP_ISSYNC_1_A, SEL_SSP1_1_0),
+ PINMUX_IPSR_MSEL(IP16_27_24, RIF1_D1_A, SEL_DRIF1_0),
+ PINMUX_IPSR_MSEL(IP16_27_24, RIF3_D1_A, SEL_DRIF3_0),
- PINMUX_IPSR_GPSR(IP16_31_28, USB30_OVC),
- PINMUX_IPSR_MSEL(IP16_31_28, AUDIO_CLKOUT1_B, SEL_ADG_1),
- PINMUX_IPSR_MSEL(IP16_31_28, SSI_WS2_B, SEL_SSI_1),
- PINMUX_IPSR_MSEL(IP16_31_28, TS_SPSYNC1_D, SEL_TSIF1_3),
- PINMUX_IPSR_MSEL(IP16_31_28, STP_ISSYNC_1_D, SEL_SSP1_1_3),
- PINMUX_IPSR_MSEL(IP16_31_28, STP_IVCXO27_0_E, SEL_SSP1_0_4),
- PINMUX_IPSR_MSEL(IP16_31_28, RIF3_D1_B, SEL_DRIF3_1),
- PINMUX_IPSR_MSEL(IP16_31_28, FSO_TOE_B, SEL_FSO_1),
- PINMUX_IPSR_GPSR(IP16_31_28, TPU0TO1),
+ PINMUX_IPSR_MSEL(IP16_31_28, SSI_SDATA9_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP16_31_28, HSCK2_B, SEL_HSCIF2_1),
+ PINMUX_IPSR_MSEL(IP16_31_28, MSIOF1_SS1_C, SEL_MSIOF1_2),
+ PINMUX_IPSR_MSEL(IP16_31_28, HSCK1_A, SEL_HSCIF1_0),
+ PINMUX_IPSR_MSEL(IP16_31_28, SSI_WS1_B, SEL_SSI_1),
+ PINMUX_IPSR_GPSR(IP16_31_28, SCK1),
+ PINMUX_IPSR_MSEL(IP16_31_28, STP_IVCXO27_1_A, SEL_SSP1_1_0),
+ PINMUX_IPSR_GPSR(IP16_31_28, SCK5_A),
/* IPSR17 */
- PINMUX_IPSR_GPSR(IP17_3_0, USB31_PWEN),
- PINMUX_IPSR_MSEL(IP17_3_0, AUDIO_CLKOUT2_B, SEL_ADG_1),
- PINMUX_IPSR_MSEL(IP17_3_0, SSI_SCK9_B, SEL_SSI_1),
- PINMUX_IPSR_MSEL(IP17_3_0, TS_SDEN0_E, SEL_TSIF0_4),
- PINMUX_IPSR_MSEL(IP17_3_0, STP_ISEN_0_E, SEL_SSP1_0_4),
- PINMUX_IPSR_MSEL(IP17_3_0, RIF2_D0_B, SEL_DRIF2_1),
- PINMUX_IPSR_GPSR(IP17_3_0, TPU0TO2),
+ PINMUX_IPSR_MSEL(IP17_3_0, AUDIO_CLKA_A, SEL_ADG_A_0),
+ PINMUX_IPSR_GPSR(IP17_3_0, CC5_OSCOUT),
- PINMUX_IPSR_GPSR(IP17_7_4, USB31_OVC),
- PINMUX_IPSR_MSEL(IP17_7_4, AUDIO_CLKOUT3_B, SEL_ADG_1),
- PINMUX_IPSR_MSEL(IP17_7_4, SSI_WS9_B, SEL_SSI_1),
- PINMUX_IPSR_MSEL(IP17_7_4, TS_SPSYNC0_E, SEL_TSIF0_4),
- PINMUX_IPSR_MSEL(IP17_7_4, STP_ISSYNC_0_E, SEL_SSP1_0_4),
- PINMUX_IPSR_MSEL(IP17_7_4, RIF2_D1_B, SEL_DRIF2_1),
- PINMUX_IPSR_GPSR(IP17_7_4, TPU0TO3),
+ PINMUX_IPSR_MSEL(IP17_7_4, AUDIO_CLKB_B, SEL_ADG_B_1),
+ PINMUX_IPSR_MSEL(IP17_7_4, SCIF_CLK_A, SEL_SCIF1_0),
+ PINMUX_IPSR_MSEL(IP17_7_4, STP_IVCXO27_1_D, SEL_SSP1_1_3),
+ PINMUX_IPSR_MSEL(IP17_7_4, REMOCON_A, SEL_REMOCON_0),
+ PINMUX_IPSR_MSEL(IP17_7_4, TCLK1_A, SEL_TIMER_TMU_0),
+
+ PINMUX_IPSR_GPSR(IP17_11_8, USB0_PWEN),
+ PINMUX_IPSR_MSEL(IP17_11_8, SIM0_RST_C, SEL_SIMCARD_2),
+ PINMUX_IPSR_MSEL(IP17_11_8, TS_SCK1_D, SEL_TSIF1_3),
+ PINMUX_IPSR_MSEL(IP17_11_8, STP_ISCLK_1_D, SEL_SSP1_1_3),
+ PINMUX_IPSR_MSEL(IP17_11_8, BPFCLK_B, SEL_FM_1),
+ PINMUX_IPSR_MSEL(IP17_11_8, RIF3_CLK_B, SEL_DRIF3_1),
+ PINMUX_IPSR_MSEL(IP17_11_8, HSCK2_C, SEL_HSCIF2_2),
+
+ PINMUX_IPSR_GPSR(IP17_15_12, USB0_OVC),
+ PINMUX_IPSR_MSEL(IP17_15_12, SIM0_D_C, SEL_SIMCARD_2),
+ PINMUX_IPSR_MSEL(IP17_15_12, TS_SDAT1_D, SEL_TSIF1_3),
+ PINMUX_IPSR_MSEL(IP17_15_12, STP_ISD_1_D, SEL_SSP1_1_3),
+ PINMUX_IPSR_MSEL(IP17_15_12, RIF3_SYNC_B, SEL_DRIF3_1),
+ PINMUX_IPSR_MSEL(IP17_15_12, HRX2_C, SEL_HSCIF2_2),
+
+ PINMUX_IPSR_GPSR(IP17_19_16, USB1_PWEN),
+ PINMUX_IPSR_MSEL(IP17_19_16, SIM0_CLK_C, SEL_SIMCARD_2),
+ PINMUX_IPSR_MSEL(IP17_19_16, SSI_SCK1_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP17_19_16, TS_SCK0_E, SEL_TSIF0_4),
+ PINMUX_IPSR_MSEL(IP17_19_16, STP_ISCLK_0_E, SEL_SSP1_0_4),
+ PINMUX_IPSR_MSEL(IP17_19_16, FMCLK_B, SEL_FM_1),
+ PINMUX_IPSR_MSEL(IP17_19_16, RIF2_CLK_B, SEL_DRIF2_1),
+ PINMUX_IPSR_MSEL(IP17_19_16, SPEEDIN_A, SEL_SPEED_PULSE_0),
+ PINMUX_IPSR_MSEL(IP17_19_16, HTX2_C, SEL_HSCIF2_2),
+
+ PINMUX_IPSR_GPSR(IP17_23_20, USB1_OVC),
+ PINMUX_IPSR_MSEL(IP17_23_20, MSIOF1_SS2_C, SEL_MSIOF1_2),
+ PINMUX_IPSR_MSEL(IP17_23_20, SSI_WS1_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP17_23_20, TS_SDAT0_E, SEL_TSIF0_4),
+ PINMUX_IPSR_MSEL(IP17_23_20, STP_ISD_0_E, SEL_SSP1_0_4),
+ PINMUX_IPSR_MSEL(IP17_23_20, FMIN_B, SEL_FM_1),
+ PINMUX_IPSR_MSEL(IP17_23_20, RIF2_SYNC_B, SEL_DRIF2_1),
+ PINMUX_IPSR_MSEL(IP17_23_20, REMOCON_B, SEL_REMOCON_1),
+ PINMUX_IPSR_MSEL(IP17_23_20, HCTS2_N_C, SEL_HSCIF2_2),
+
+ PINMUX_IPSR_GPSR(IP17_27_24, USB30_PWEN),
+ PINMUX_IPSR_GPSR(IP17_27_24, AUDIO_CLKOUT_B),
+ PINMUX_IPSR_MSEL(IP17_27_24, SSI_SCK2_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP17_27_24, TS_SDEN1_D, SEL_TSIF1_3),
+ PINMUX_IPSR_MSEL(IP17_27_24, STP_ISEN_1_D, SEL_SSP1_1_2),
+ PINMUX_IPSR_MSEL(IP17_27_24, STP_OPWM_0_E, SEL_SSP1_0_4),
+ PINMUX_IPSR_MSEL(IP17_27_24, RIF3_D0_B, SEL_DRIF3_1),
+ PINMUX_IPSR_MSEL(IP17_27_24, TCLK2_B, SEL_TIMER_TMU_1),
+ PINMUX_IPSR_GPSR(IP17_27_24, TPU0TO0),
+ PINMUX_IPSR_MSEL(IP17_27_24, BPFCLK_C, SEL_FM_2),
+ PINMUX_IPSR_MSEL(IP17_27_24, HRTS2_N_C, SEL_HSCIF2_2),
+
+ PINMUX_IPSR_GPSR(IP17_31_28, USB30_OVC),
+ PINMUX_IPSR_GPSR(IP17_31_28, AUDIO_CLKOUT1_B),
+ PINMUX_IPSR_MSEL(IP17_31_28, SSI_WS2_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP17_31_28, TS_SPSYNC1_D, SEL_TSIF1_3),
+ PINMUX_IPSR_MSEL(IP17_31_28, STP_ISSYNC_1_D, SEL_SSP1_1_3),
+ PINMUX_IPSR_MSEL(IP17_31_28, STP_IVCXO27_0_E, SEL_SSP1_0_4),
+ PINMUX_IPSR_MSEL(IP17_31_28, RIF3_D1_B, SEL_DRIF3_1),
+ PINMUX_IPSR_GPSR(IP17_31_28, FSO_TOE_N),
+ PINMUX_IPSR_GPSR(IP17_31_28, TPU0TO1),
+
+ /* IPSR18 */
+ PINMUX_IPSR_GPSR(IP18_3_0, USB3_PWEN),
+ PINMUX_IPSR_GPSR(IP18_3_0, AUDIO_CLKOUT2_B),
+ PINMUX_IPSR_MSEL(IP18_3_0, SSI_SCK9_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP18_3_0, TS_SDEN0_E, SEL_TSIF0_4),
+ PINMUX_IPSR_MSEL(IP18_3_0, STP_ISEN_0_E, SEL_SSP1_0_4),
+ PINMUX_IPSR_MSEL(IP18_3_0, RIF2_D0_B, SEL_DRIF2_1),
+ PINMUX_IPSR_GPSR(IP18_3_0, TPU0TO2),
+ PINMUX_IPSR_MSEL(IP18_3_0, FMCLK_C, SEL_FM_2),
+ PINMUX_IPSR_MSEL(IP18_3_0, FMCLK_D, SEL_FM_3),
+
+ PINMUX_IPSR_GPSR(IP18_7_4, USB3_OVC),
+ PINMUX_IPSR_GPSR(IP18_7_4, AUDIO_CLKOUT3_B),
+ PINMUX_IPSR_MSEL(IP18_7_4, SSI_WS9_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP18_7_4, TS_SPSYNC0_E, SEL_TSIF0_4),
+ PINMUX_IPSR_MSEL(IP18_7_4, STP_ISSYNC_0_E, SEL_SSP1_0_4),
+ PINMUX_IPSR_MSEL(IP18_7_4, RIF2_D1_B, SEL_DRIF2_1),
+ PINMUX_IPSR_GPSR(IP18_7_4, TPU0TO3),
+ PINMUX_IPSR_MSEL(IP18_7_4, FMIN_C, SEL_FM_2),
+ PINMUX_IPSR_MSEL(IP18_7_4, FMIN_D, SEL_FM_3),
/*
* Static pins can not be muxed between different functions but
@@ -1507,1616 +1576,6 @@
SH_PFC_PIN_NAMED_CFG(ROW_GROUP_A('T'), 30, ASEBRK, CFG_FLAGS),
};
-/* - AUDIO CLOCK ------------------------------------------------------------ */
-static const unsigned int audio_clk_a_a_pins[] = {
- /* CLK A */
- RCAR_GP_PIN(6, 22),
-};
-static const unsigned int audio_clk_a_a_mux[] = {
- AUDIO_CLKA_A_MARK,
-};
-static const unsigned int audio_clk_a_b_pins[] = {
- /* CLK A */
- RCAR_GP_PIN(5, 4),
-};
-static const unsigned int audio_clk_a_b_mux[] = {
- AUDIO_CLKA_B_MARK,
-};
-static const unsigned int audio_clk_a_c_pins[] = {
- /* CLK A */
- RCAR_GP_PIN(5, 19),
-};
-static const unsigned int audio_clk_a_c_mux[] = {
- AUDIO_CLKA_C_MARK,
-};
-static const unsigned int audio_clk_b_a_pins[] = {
- /* CLK B */
- RCAR_GP_PIN(5, 12),
-};
-static const unsigned int audio_clk_b_a_mux[] = {
- AUDIO_CLKB_A_MARK,
-};
-static const unsigned int audio_clk_b_b_pins[] = {
- /* CLK B */
- RCAR_GP_PIN(6, 23),
-};
-static const unsigned int audio_clk_b_b_mux[] = {
- AUDIO_CLKB_B_MARK,
-};
-static const unsigned int audio_clk_c_a_pins[] = {
- /* CLK C */
- RCAR_GP_PIN(5, 21),
-};
-static const unsigned int audio_clk_c_a_mux[] = {
- AUDIO_CLKC_A_MARK,
-};
-static const unsigned int audio_clk_c_b_pins[] = {
- /* CLK C */
- RCAR_GP_PIN(5, 0),
-};
-static const unsigned int audio_clk_c_b_mux[] = {
- AUDIO_CLKC_B_MARK,
-};
-static const unsigned int audio_clkout_a_pins[] = {
- /* CLKOUT */
- RCAR_GP_PIN(5, 18),
-};
-static const unsigned int audio_clkout_a_mux[] = {
- AUDIO_CLKOUT_A_MARK,
-};
-static const unsigned int audio_clkout_b_pins[] = {
- /* CLKOUT */
- RCAR_GP_PIN(6, 28),
-};
-static const unsigned int audio_clkout_b_mux[] = {
- AUDIO_CLKOUT_B_MARK,
-};
-static const unsigned int audio_clkout_c_pins[] = {
- /* CLKOUT */
- RCAR_GP_PIN(5, 3),
-};
-static const unsigned int audio_clkout_c_mux[] = {
- AUDIO_CLKOUT_C_MARK,
-};
-static const unsigned int audio_clkout_d_pins[] = {
- /* CLKOUT */
- RCAR_GP_PIN(5, 21),
-};
-static const unsigned int audio_clkout_d_mux[] = {
- AUDIO_CLKOUT_D_MARK,
-};
-static const unsigned int audio_clkout1_a_pins[] = {
- /* CLKOUT1 */
- RCAR_GP_PIN(5, 15),
-};
-static const unsigned int audio_clkout1_a_mux[] = {
- AUDIO_CLKOUT1_A_MARK,
-};
-static const unsigned int audio_clkout1_b_pins[] = {
- /* CLKOUT1 */
- RCAR_GP_PIN(6, 29),
-};
-static const unsigned int audio_clkout1_b_mux[] = {
- AUDIO_CLKOUT1_B_MARK,
-};
-static const unsigned int audio_clkout2_a_pins[] = {
- /* CLKOUT2 */
- RCAR_GP_PIN(5, 16),
-};
-static const unsigned int audio_clkout2_a_mux[] = {
- AUDIO_CLKOUT2_A_MARK,
-};
-static const unsigned int audio_clkout2_b_pins[] = {
- /* CLKOUT2 */
- RCAR_GP_PIN(6, 30),
-};
-static const unsigned int audio_clkout2_b_mux[] = {
- AUDIO_CLKOUT2_B_MARK,
-};
-
-static const unsigned int audio_clkout3_a_pins[] = {
- /* CLKOUT3 */
- RCAR_GP_PIN(5, 19),
-};
-static const unsigned int audio_clkout3_a_mux[] = {
- AUDIO_CLKOUT3_A_MARK,
-};
-static const unsigned int audio_clkout3_b_pins[] = {
- /* CLKOUT3 */
- RCAR_GP_PIN(6, 31),
-};
-static const unsigned int audio_clkout3_b_mux[] = {
- AUDIO_CLKOUT3_B_MARK,
-};
-
-/* - EtherAVB --------------------------------------------------------------- */
-static const unsigned int avb_link_pins[] = {
- /* AVB_LINK */
- RCAR_GP_PIN(2, 12),
-};
-static const unsigned int avb_link_mux[] = {
- AVB_LINK_MARK,
-};
-static const unsigned int avb_magic_pins[] = {
- /* AVB_MAGIC_ */
- RCAR_GP_PIN(2, 10),
-};
-static const unsigned int avb_magic_mux[] = {
- AVB_MAGIC_MARK,
-};
-static const unsigned int avb_phy_int_pins[] = {
- /* AVB_PHY_INT */
- RCAR_GP_PIN(2, 11),
-};
-static const unsigned int avb_phy_int_mux[] = {
- AVB_PHY_INT_MARK,
-};
-static const unsigned int avb_mdc_pins[] = {
- /* AVB_MDC, AVB_MDIO */
- RCAR_GP_PIN(2, 9), PIN_NUMBER('A', 9),
-};
-static const unsigned int avb_mdc_mux[] = {
- AVB_MDC_MARK, AVB_MDIO_MARK,
-};
-static const unsigned int avb_mii_pins[] = {
- /*
- * AVB_TX_CTL, AVB_TXC, AVB_TD0,
- * AVB_TD1, AVB_TD2, AVB_TD3,
- * AVB_RX_CTL, AVB_RXC, AVB_RD0,
- * AVB_RD1, AVB_RD2, AVB_RD3,
- * AVB_TXCREFCLK
- */
- PIN_NUMBER('A', 8), PIN_NUMBER('A', 19), PIN_NUMBER('A', 18),
- PIN_NUMBER('B', 18), PIN_NUMBER('A', 17), PIN_NUMBER('B', 17),
- PIN_NUMBER('A', 16), PIN_NUMBER('B', 19), PIN_NUMBER('A', 13),
- PIN_NUMBER('B', 13), PIN_NUMBER('A', 14), PIN_NUMBER('B', 14),
- PIN_NUMBER('A', 12),
-
-};
-static const unsigned int avb_mii_mux[] = {
- AVB_TX_CTL_MARK, AVB_TXC_MARK, AVB_TD0_MARK,
- AVB_TD1_MARK, AVB_TD2_MARK, AVB_TD3_MARK,
- AVB_RX_CTL_MARK, AVB_RXC_MARK, AVB_RD0_MARK,
- AVB_RD1_MARK, AVB_RD2_MARK, AVB_RD3_MARK,
- AVB_TXCREFCLK_MARK,
-};
-static const unsigned int avb_avtp_pps_pins[] = {
- /* AVB_AVTP_PPS */
- RCAR_GP_PIN(2, 6),
-};
-static const unsigned int avb_avtp_pps_mux[] = {
- AVB_AVTP_PPS_MARK,
-};
-static const unsigned int avb_avtp_match_a_pins[] = {
- /* AVB_AVTP_MATCH_A */
- RCAR_GP_PIN(2, 13),
-};
-static const unsigned int avb_avtp_match_a_mux[] = {
- AVB_AVTP_MATCH_A_MARK,
-};
-static const unsigned int avb_avtp_capture_a_pins[] = {
- /* AVB_AVTP_CAPTURE_A */
- RCAR_GP_PIN(2, 14),
-};
-static const unsigned int avb_avtp_capture_a_mux[] = {
- AVB_AVTP_CAPTURE_A_MARK,
-};
-static const unsigned int avb_avtp_match_b_pins[] = {
- /* AVB_AVTP_MATCH_B */
- RCAR_GP_PIN(1, 8),
-};
-static const unsigned int avb_avtp_match_b_mux[] = {
- AVB_AVTP_MATCH_B_MARK,
-};
-static const unsigned int avb_avtp_capture_b_pins[] = {
- /* AVB_AVTP_CAPTURE_B */
- RCAR_GP_PIN(1, 11),
-};
-static const unsigned int avb_avtp_capture_b_mux[] = {
- AVB_AVTP_CAPTURE_B_MARK,
-};
-
-/* - CAN ------------------------------------------------------------------ */
-static const unsigned int can0_data_a_pins[] = {
- /* TX, RX */
- RCAR_GP_PIN(1, 23), RCAR_GP_PIN(1, 24),
-};
-static const unsigned int can0_data_a_mux[] = {
- CAN0_TX_A_MARK, CAN0_RX_A_MARK,
-};
-static const unsigned int can0_data_b_pins[] = {
- /* TX, RX */
- RCAR_GP_PIN(2, 0), RCAR_GP_PIN(2, 1),
-};
-static const unsigned int can0_data_b_mux[] = {
- CAN0_TX_B_MARK, CAN0_RX_B_MARK,
-};
-static const unsigned int can1_data_pins[] = {
- /* TX, RX */
- RCAR_GP_PIN(1, 22), RCAR_GP_PIN(1, 26),
-};
-static const unsigned int can1_data_mux[] = {
- CAN1_TX_MARK, CAN1_RX_MARK,
-};
-
-/* - CAN Clock -------------------------------------------------------------- */
-static const unsigned int can_clk_pins[] = {
- /* CLK */
- RCAR_GP_PIN(1, 25),
-};
-static const unsigned int can_clk_mux[] = {
- CAN_CLK_MARK,
-};
-
-/* - CAN FD --------------------------------------------------------------- */
-static const unsigned int canfd0_data_a_pins[] = {
- /* TX, RX */
- RCAR_GP_PIN(1, 23), RCAR_GP_PIN(1, 24),
-};
-static const unsigned int canfd0_data_a_mux[] = {
- CANFD0_TX_A_MARK, CANFD0_RX_A_MARK,
-};
-static const unsigned int canfd0_data_b_pins[] = {
- /* TX, RX */
- RCAR_GP_PIN(2, 0), RCAR_GP_PIN(2, 1),
-};
-static const unsigned int canfd0_data_b_mux[] = {
- CANFD0_TX_B_MARK, CANFD0_RX_B_MARK,
-};
-static const unsigned int canfd1_data_pins[] = {
- /* TX, RX */
- RCAR_GP_PIN(1, 22), RCAR_GP_PIN(1, 26),
-};
-static const unsigned int canfd1_data_mux[] = {
- CANFD1_TX_MARK, CANFD1_RX_MARK,
-};
-
-/* - DRIF0 --------------------------------------------------------------- */
-static const unsigned int drif0_ctrl_a_pins[] = {
- /* CLK, SYNC */
- RCAR_GP_PIN(6, 8), RCAR_GP_PIN(6, 9),
-};
-static const unsigned int drif0_ctrl_a_mux[] = {
- RIF0_CLK_A_MARK, RIF0_SYNC_A_MARK,
-};
-static const unsigned int drif0_data0_a_pins[] = {
- /* D0 */
- RCAR_GP_PIN(6, 10),
-};
-static const unsigned int drif0_data0_a_mux[] = {
- RIF0_D0_A_MARK,
-};
-static const unsigned int drif0_data1_a_pins[] = {
- /* D1 */
- RCAR_GP_PIN(6, 7),
-};
-static const unsigned int drif0_data1_a_mux[] = {
- RIF0_D1_A_MARK,
-};
-static const unsigned int drif0_ctrl_b_pins[] = {
- /* CLK, SYNC */
- RCAR_GP_PIN(5, 0), RCAR_GP_PIN(5, 4),
-};
-static const unsigned int drif0_ctrl_b_mux[] = {
- RIF0_CLK_B_MARK, RIF0_SYNC_B_MARK,
-};
-static const unsigned int drif0_data0_b_pins[] = {
- /* D0 */
- RCAR_GP_PIN(5, 1),
-};
-static const unsigned int drif0_data0_b_mux[] = {
- RIF0_D0_B_MARK,
-};
-static const unsigned int drif0_data1_b_pins[] = {
- /* D1 */
- RCAR_GP_PIN(5, 2),
-};
-static const unsigned int drif0_data1_b_mux[] = {
- RIF0_D1_B_MARK,
-};
-static const unsigned int drif0_ctrl_c_pins[] = {
- /* CLK, SYNC */
- RCAR_GP_PIN(5, 12), RCAR_GP_PIN(5, 15),
-};
-static const unsigned int drif0_ctrl_c_mux[] = {
- RIF0_CLK_C_MARK, RIF0_SYNC_C_MARK,
-};
-static const unsigned int drif0_data0_c_pins[] = {
- /* D0 */
- RCAR_GP_PIN(5, 13),
-};
-static const unsigned int drif0_data0_c_mux[] = {
- RIF0_D0_C_MARK,
-};
-static const unsigned int drif0_data1_c_pins[] = {
- /* D1 */
- RCAR_GP_PIN(5, 14),
-};
-static const unsigned int drif0_data1_c_mux[] = {
- RIF0_D1_C_MARK,
-};
-/* - DRIF1 --------------------------------------------------------------- */
-static const unsigned int drif1_ctrl_a_pins[] = {
- /* CLK, SYNC */
- RCAR_GP_PIN(6, 17), RCAR_GP_PIN(6, 18),
-};
-static const unsigned int drif1_ctrl_a_mux[] = {
- RIF1_CLK_A_MARK, RIF1_SYNC_A_MARK,
-};
-static const unsigned int drif1_data0_a_pins[] = {
- /* D0 */
- RCAR_GP_PIN(6, 19),
-};
-static const unsigned int drif1_data0_a_mux[] = {
- RIF1_D0_A_MARK,
-};
-static const unsigned int drif1_data1_a_pins[] = {
- /* D1 */
- RCAR_GP_PIN(6, 20),
-};
-static const unsigned int drif1_data1_a_mux[] = {
- RIF1_D1_A_MARK,
-};
-static const unsigned int drif1_ctrl_b_pins[] = {
- /* CLK, SYNC */
- RCAR_GP_PIN(5, 9), RCAR_GP_PIN(5, 3),
-};
-static const unsigned int drif1_ctrl_b_mux[] = {
- RIF1_CLK_B_MARK, RIF1_SYNC_B_MARK,
-};
-static const unsigned int drif1_data0_b_pins[] = {
- /* D0 */
- RCAR_GP_PIN(5, 7),
-};
-static const unsigned int drif1_data0_b_mux[] = {
- RIF1_D0_B_MARK,
-};
-static const unsigned int drif1_data1_b_pins[] = {
- /* D1 */
- RCAR_GP_PIN(5, 8),
-};
-static const unsigned int drif1_data1_b_mux[] = {
- RIF1_D1_B_MARK,
-};
-static const unsigned int drif1_ctrl_c_pins[] = {
- /* CLK, SYNC */
- RCAR_GP_PIN(5, 5), RCAR_GP_PIN(5, 11),
-};
-static const unsigned int drif1_ctrl_c_mux[] = {
- RIF1_CLK_C_MARK, RIF1_SYNC_C_MARK,
-};
-static const unsigned int drif1_data0_c_pins[] = {
- /* D0 */
- RCAR_GP_PIN(5, 6),
-};
-static const unsigned int drif1_data0_c_mux[] = {
- RIF1_D0_C_MARK,
-};
-static const unsigned int drif1_data1_c_pins[] = {
- /* D1 */
- RCAR_GP_PIN(5, 10),
-};
-static const unsigned int drif1_data1_c_mux[] = {
- RIF1_D1_C_MARK,
-};
-/* - DRIF2 --------------------------------------------------------------- */
-static const unsigned int drif2_ctrl_a_pins[] = {
- /* CLK, SYNC */
- RCAR_GP_PIN(6, 8), RCAR_GP_PIN(6, 9),
-};
-static const unsigned int drif2_ctrl_a_mux[] = {
- RIF2_CLK_A_MARK, RIF2_SYNC_A_MARK,
-};
-static const unsigned int drif2_data0_a_pins[] = {
- /* D0 */
- RCAR_GP_PIN(6, 7),
-};
-static const unsigned int drif2_data0_a_mux[] = {
- RIF2_D0_A_MARK,
-};
-static const unsigned int drif2_data1_a_pins[] = {
- /* D1 */
- RCAR_GP_PIN(6, 10),
-};
-static const unsigned int drif2_data1_a_mux[] = {
- RIF2_D1_A_MARK,
-};
-static const unsigned int drif2_ctrl_b_pins[] = {
- /* CLK, SYNC */
- RCAR_GP_PIN(6, 26), RCAR_GP_PIN(6, 27),
-};
-static const unsigned int drif2_ctrl_b_mux[] = {
- RIF2_CLK_B_MARK, RIF2_SYNC_B_MARK,
-};
-static const unsigned int drif2_data0_b_pins[] = {
- /* D0 */
- RCAR_GP_PIN(6, 30),
-};
-static const unsigned int drif2_data0_b_mux[] = {
- RIF2_D0_B_MARK,
-};
-static const unsigned int drif2_data1_b_pins[] = {
- /* D1 */
- RCAR_GP_PIN(6, 31),
-};
-static const unsigned int drif2_data1_b_mux[] = {
- RIF2_D1_B_MARK,
-};
-/* - DRIF3 --------------------------------------------------------------- */
-static const unsigned int drif3_ctrl_a_pins[] = {
- /* CLK, SYNC */
- RCAR_GP_PIN(6, 17), RCAR_GP_PIN(6, 18),
-};
-static const unsigned int drif3_ctrl_a_mux[] = {
- RIF3_CLK_A_MARK, RIF3_SYNC_A_MARK,
-};
-static const unsigned int drif3_data0_a_pins[] = {
- /* D0 */
- RCAR_GP_PIN(6, 19),
-};
-static const unsigned int drif3_data0_a_mux[] = {
- RIF3_D0_A_MARK,
-};
-static const unsigned int drif3_data1_a_pins[] = {
- /* D1 */
- RCAR_GP_PIN(6, 20),
-};
-static const unsigned int drif3_data1_a_mux[] = {
- RIF3_D1_A_MARK,
-};
-static const unsigned int drif3_ctrl_b_pins[] = {
- /* CLK, SYNC */
- RCAR_GP_PIN(6, 24), RCAR_GP_PIN(6, 25),
-};
-static const unsigned int drif3_ctrl_b_mux[] = {
- RIF3_CLK_B_MARK, RIF3_SYNC_B_MARK,
-};
-static const unsigned int drif3_data0_b_pins[] = {
- /* D0 */
- RCAR_GP_PIN(6, 28),
-};
-static const unsigned int drif3_data0_b_mux[] = {
- RIF3_D0_B_MARK,
-};
-static const unsigned int drif3_data1_b_pins[] = {
- /* D1 */
- RCAR_GP_PIN(6, 29),
-};
-static const unsigned int drif3_data1_b_mux[] = {
- RIF3_D1_B_MARK,
-};
-
-/* - DU --------------------------------------------------------------------- */
-static const unsigned int du_rgb666_pins[] = {
- /* R[7:2], G[7:2], B[7:2] */
- RCAR_GP_PIN(0, 15), RCAR_GP_PIN(0, 14), RCAR_GP_PIN(0, 13),
- RCAR_GP_PIN(0, 12), RCAR_GP_PIN(0, 11), RCAR_GP_PIN(0, 10),
- RCAR_GP_PIN(1, 15), RCAR_GP_PIN(1, 14), RCAR_GP_PIN(1, 13),
- RCAR_GP_PIN(1, 12), RCAR_GP_PIN(1, 19), RCAR_GP_PIN(1, 18),
- RCAR_GP_PIN(1, 7), RCAR_GP_PIN(1, 6), RCAR_GP_PIN(1, 5),
- RCAR_GP_PIN(1, 4), RCAR_GP_PIN(1, 3), RCAR_GP_PIN(1, 2),
-};
-static const unsigned int du_rgb666_mux[] = {
- DU_DR7_MARK, DU_DR6_MARK, DU_DR5_MARK, DU_DR4_MARK,
- DU_DR3_MARK, DU_DR2_MARK,
- DU_DG7_MARK, DU_DG6_MARK, DU_DG5_MARK, DU_DG4_MARK,
- DU_DG3_MARK, DU_DG2_MARK,
- DU_DB7_MARK, DU_DB6_MARK, DU_DB5_MARK, DU_DB4_MARK,
- DU_DB3_MARK, DU_DB2_MARK,
-};
-static const unsigned int du_rgb888_pins[] = {
- /* R[7:0], G[7:0], B[7:0] */
- RCAR_GP_PIN(0, 15), RCAR_GP_PIN(0, 14), RCAR_GP_PIN(0, 13),
- RCAR_GP_PIN(0, 12), RCAR_GP_PIN(0, 11), RCAR_GP_PIN(0, 10),
- RCAR_GP_PIN(0, 9), RCAR_GP_PIN(0, 8),
- RCAR_GP_PIN(1, 15), RCAR_GP_PIN(1, 14), RCAR_GP_PIN(1, 13),
- RCAR_GP_PIN(1, 12), RCAR_GP_PIN(1, 19), RCAR_GP_PIN(1, 18),
- RCAR_GP_PIN(1, 17), RCAR_GP_PIN(1, 16),
- RCAR_GP_PIN(1, 7), RCAR_GP_PIN(1, 6), RCAR_GP_PIN(1, 5),
- RCAR_GP_PIN(1, 4), RCAR_GP_PIN(1, 3), RCAR_GP_PIN(1, 2),
- RCAR_GP_PIN(1, 1), RCAR_GP_PIN(1, 0),
-};
-static const unsigned int du_rgb888_mux[] = {
- DU_DR7_MARK, DU_DR6_MARK, DU_DR5_MARK, DU_DR4_MARK,
- DU_DR3_MARK, DU_DR2_MARK, DU_DR1_MARK, DU_DR0_MARK,
- DU_DG7_MARK, DU_DG6_MARK, DU_DG5_MARK, DU_DG4_MARK,
- DU_DG3_MARK, DU_DG2_MARK, DU_DG1_MARK, DU_DG0_MARK,
- DU_DB7_MARK, DU_DB6_MARK, DU_DB5_MARK, DU_DB4_MARK,
- DU_DB3_MARK, DU_DB2_MARK, DU_DB1_MARK, DU_DB0_MARK,
-};
-static const unsigned int du_clk_out_0_pins[] = {
- /* CLKOUT */
- RCAR_GP_PIN(1, 27),
-};
-static const unsigned int du_clk_out_0_mux[] = {
- DU_DOTCLKOUT0_MARK
-};
-static const unsigned int du_clk_out_1_pins[] = {
- /* CLKOUT */
- RCAR_GP_PIN(2, 3),
-};
-static const unsigned int du_clk_out_1_mux[] = {
- DU_DOTCLKOUT1_MARK
-};
-static const unsigned int du_sync_pins[] = {
- /* EXVSYNC/VSYNC, EXHSYNC/HSYNC */
- RCAR_GP_PIN(2, 5), RCAR_GP_PIN(2, 4),
-};
-static const unsigned int du_sync_mux[] = {
- DU_EXVSYNC_DU_VSYNC_MARK, DU_EXHSYNC_DU_HSYNC_MARK
-};
-static const unsigned int du_oddf_pins[] = {
- /* EXDISP/EXODDF/EXCDE */
- RCAR_GP_PIN(2, 2),
-};
-static const unsigned int du_oddf_mux[] = {
- DU_EXODDF_DU_ODDF_DISP_CDE_MARK,
-};
-static const unsigned int du_cde_pins[] = {
- /* CDE */
- RCAR_GP_PIN(2, 0),
-};
-static const unsigned int du_cde_mux[] = {
- DU_CDE_MARK,
-};
-static const unsigned int du_disp_pins[] = {
- /* DISP */
- RCAR_GP_PIN(2, 1),
-};
-static const unsigned int du_disp_mux[] = {
- DU_DISP_MARK,
-};
-/* - HSCIF0 ----------------------------------------------------------------- */
-static const unsigned int hscif0_data_pins[] = {
- /* RX, TX */
- RCAR_GP_PIN(5, 13), RCAR_GP_PIN(5, 14),
-};
-static const unsigned int hscif0_data_mux[] = {
- HRX0_MARK, HTX0_MARK,
-};
-static const unsigned int hscif0_clk_pins[] = {
- /* SCK */
- RCAR_GP_PIN(5, 12),
-};
-static const unsigned int hscif0_clk_mux[] = {
- HSCK0_MARK,
-};
-static const unsigned int hscif0_ctrl_pins[] = {
- /* RTS, CTS */
- RCAR_GP_PIN(5, 16), RCAR_GP_PIN(5, 15),
-};
-static const unsigned int hscif0_ctrl_mux[] = {
- HRTS0_N_MARK, HCTS0_N_MARK,
-};
-/* - HSCIF1 ----------------------------------------------------------------- */
-static const unsigned int hscif1_data_a_pins[] = {
- /* RX, TX */
- RCAR_GP_PIN(5, 5), RCAR_GP_PIN(5, 6),
-};
-static const unsigned int hscif1_data_a_mux[] = {
- HRX1_A_MARK, HTX1_A_MARK,
-};
-static const unsigned int hscif1_clk_a_pins[] = {
- /* SCK */
- RCAR_GP_PIN(6, 21),
-};
-static const unsigned int hscif1_clk_a_mux[] = {
- HSCK1_A_MARK,
-};
-static const unsigned int hscif1_ctrl_a_pins[] = {
- /* RTS, CTS */
- RCAR_GP_PIN(5, 8), RCAR_GP_PIN(5, 7),
-};
-static const unsigned int hscif1_ctrl_a_mux[] = {
- HRTS1_N_A_MARK, HCTS1_N_A_MARK,
-};
-
-static const unsigned int hscif1_data_b_pins[] = {
- /* RX, TX */
- RCAR_GP_PIN(5, 1), RCAR_GP_PIN(5, 2),
-};
-static const unsigned int hscif1_data_b_mux[] = {
- HRX1_B_MARK, HTX1_B_MARK,
-};
-static const unsigned int hscif1_clk_b_pins[] = {
- /* SCK */
- RCAR_GP_PIN(5, 0),
-};
-static const unsigned int hscif1_clk_b_mux[] = {
- HSCK1_B_MARK,
-};
-static const unsigned int hscif1_ctrl_b_pins[] = {
- /* RTS, CTS */
- RCAR_GP_PIN(5, 4), RCAR_GP_PIN(5, 3),
-};
-static const unsigned int hscif1_ctrl_b_mux[] = {
- HRTS1_N_B_MARK, HCTS1_N_B_MARK,
-};
-/* - HSCIF2 ----------------------------------------------------------------- */
-static const unsigned int hscif2_data_a_pins[] = {
- /* RX, TX */
- RCAR_GP_PIN(6, 8), RCAR_GP_PIN(6, 9),
-};
-static const unsigned int hscif2_data_a_mux[] = {
- HRX2_A_MARK, HTX2_A_MARK,
-};
-static const unsigned int hscif2_clk_a_pins[] = {
- /* SCK */
- RCAR_GP_PIN(6, 10),
-};
-static const unsigned int hscif2_clk_a_mux[] = {
- HSCK2_A_MARK,
-};
-static const unsigned int hscif2_ctrl_a_pins[] = {
- /* RTS, CTS */
- RCAR_GP_PIN(6, 7), RCAR_GP_PIN(6, 6),
-};
-static const unsigned int hscif2_ctrl_a_mux[] = {
- HRTS2_N_A_MARK, HCTS2_N_A_MARK,
-};
-
-static const unsigned int hscif2_data_b_pins[] = {
- /* RX, TX */
- RCAR_GP_PIN(6, 17), RCAR_GP_PIN(6, 18),
-};
-static const unsigned int hscif2_data_b_mux[] = {
- HRX2_B_MARK, HTX2_B_MARK,
-};
-static const unsigned int hscif2_clk_b_pins[] = {
- /* SCK */
- RCAR_GP_PIN(6, 21),
-};
-static const unsigned int hscif2_clk_b_mux[] = {
- HSCK1_B_MARK,
-};
-static const unsigned int hscif2_ctrl_b_pins[] = {
- /* RTS, CTS */
- RCAR_GP_PIN(6, 20), RCAR_GP_PIN(6, 19),
-};
-static const unsigned int hscif2_ctrl_b_mux[] = {
- HRTS2_N_B_MARK, HCTS2_N_B_MARK,
-};
-/* - HSCIF3 ----------------------------------------------------------------- */
-static const unsigned int hscif3_data_a_pins[] = {
- /* RX, TX */
- RCAR_GP_PIN(1, 23), RCAR_GP_PIN(1, 24),
-};
-static const unsigned int hscif3_data_a_mux[] = {
- HRX3_A_MARK, HTX3_A_MARK,
-};
-static const unsigned int hscif3_clk_pins[] = {
- /* SCK */
- RCAR_GP_PIN(1, 22),
-};
-static const unsigned int hscif3_clk_mux[] = {
- HSCK3_MARK,
-};
-static const unsigned int hscif3_ctrl_pins[] = {
- /* RTS, CTS */
- RCAR_GP_PIN(1, 26), RCAR_GP_PIN(1, 25),
-};
-static const unsigned int hscif3_ctrl_mux[] = {
- HRTS3_N_MARK, HCTS3_N_MARK,
-};
-
-static const unsigned int hscif3_data_b_pins[] = {
- /* RX, TX */
- RCAR_GP_PIN(0, 10), RCAR_GP_PIN(0, 11),
-};
-static const unsigned int hscif3_data_b_mux[] = {
- HRX3_B_MARK, HTX3_B_MARK,
-};
-static const unsigned int hscif3_data_c_pins[] = {
- /* RX, TX */
- RCAR_GP_PIN(0, 14), RCAR_GP_PIN(0, 15),
-};
-static const unsigned int hscif3_data_c_mux[] = {
- HRX3_C_MARK, HTX3_C_MARK,
-};
-static const unsigned int hscif3_data_d_pins[] = {
- /* RX, TX */
- RCAR_GP_PIN(2, 7), RCAR_GP_PIN(2, 8),
-};
-static const unsigned int hscif3_data_d_mux[] = {
- HRX3_D_MARK, HTX3_D_MARK,
-};
-/* - HSCIF4 ----------------------------------------------------------------- */
-static const unsigned int hscif4_data_a_pins[] = {
- /* RX, TX */
- RCAR_GP_PIN(1, 12), RCAR_GP_PIN(1, 13),
-};
-static const unsigned int hscif4_data_a_mux[] = {
- HRX4_A_MARK, HTX4_A_MARK,
-};
-static const unsigned int hscif4_clk_pins[] = {
- /* SCK */
- RCAR_GP_PIN(1, 11),
-};
-static const unsigned int hscif4_clk_mux[] = {
- HSCK4_MARK,
-};
-static const unsigned int hscif4_ctrl_pins[] = {
- /* RTS, CTS */
- RCAR_GP_PIN(1, 15), RCAR_GP_PIN(1, 14),
-};
-static const unsigned int hscif4_ctrl_mux[] = {
- HRTS4_N_MARK, HCTS3_N_MARK,
-};
-
-static const unsigned int hscif4_data_b_pins[] = {
- /* RX, TX */
- RCAR_GP_PIN(1, 8), RCAR_GP_PIN(1, 11),
-};
-static const unsigned int hscif4_data_b_mux[] = {
- HRX4_B_MARK, HTX4_B_MARK,
-};
-
-/* - I2C -------------------------------------------------------------------- */
-static const unsigned int i2c1_a_pins[] = {
- /* SDA, SCL */
- RCAR_GP_PIN(5, 11), RCAR_GP_PIN(5, 10),
-};
-static const unsigned int i2c1_a_mux[] = {
- SDA1_A_MARK, SCL1_A_MARK,
-};
-static const unsigned int i2c1_b_pins[] = {
- /* SDA, SCL */
- RCAR_GP_PIN(5, 24), RCAR_GP_PIN(5, 23),
-};
-static const unsigned int i2c1_b_mux[] = {
- SDA1_B_MARK, SCL1_B_MARK,
-};
-static const unsigned int i2c2_a_pins[] = {
- /* SDA, SCL */
- RCAR_GP_PIN(5, 0), RCAR_GP_PIN(5, 4),
-};
-static const unsigned int i2c2_a_mux[] = {
- SDA2_A_MARK, SCL2_A_MARK,
-};
-static const unsigned int i2c2_b_pins[] = {
- /* SDA, SCL */
- RCAR_GP_PIN(3, 13), RCAR_GP_PIN(3, 12),
-};
-static const unsigned int i2c2_b_mux[] = {
- SDA2_B_MARK, SCL2_B_MARK,
-};
-static const unsigned int i2c6_a_pins[] = {
- /* SDA, SCL */
- RCAR_GP_PIN(1, 8), RCAR_GP_PIN(1, 11),
-};
-static const unsigned int i2c6_a_mux[] = {
- SDA6_A_MARK, SCL6_A_MARK,
-};
-static const unsigned int i2c6_b_pins[] = {
- /* SDA, SCL */
- RCAR_GP_PIN(1, 26), RCAR_GP_PIN(1, 25),
-};
-static const unsigned int i2c6_b_mux[] = {
- SDA6_B_MARK, SCL6_B_MARK,
-};
-static const unsigned int i2c6_c_pins[] = {
- /* SDA, SCL */
- RCAR_GP_PIN(0, 15), RCAR_GP_PIN(0, 14),
-};
-static const unsigned int i2c6_c_mux[] = {
- SDA6_C_MARK, SCL6_C_MARK,
-};
-
-/* - INTC-EX ---------------------------------------------------------------- */
-static const unsigned int intc_ex_irq0_pins[] = {
- /* IRQ0 */
- RCAR_GP_PIN(2, 0),
-};
-static const unsigned int intc_ex_irq0_mux[] = {
- IRQ0_MARK,
-};
-static const unsigned int intc_ex_irq1_pins[] = {
- /* IRQ1 */
- RCAR_GP_PIN(2, 1),
-};
-static const unsigned int intc_ex_irq1_mux[] = {
- IRQ1_MARK,
-};
-static const unsigned int intc_ex_irq2_pins[] = {
- /* IRQ2 */
- RCAR_GP_PIN(2, 2),
-};
-static const unsigned int intc_ex_irq2_mux[] = {
- IRQ2_MARK,
-};
-static const unsigned int intc_ex_irq3_pins[] = {
- /* IRQ3 */
- RCAR_GP_PIN(2, 3),
-};
-static const unsigned int intc_ex_irq3_mux[] = {
- IRQ3_MARK,
-};
-static const unsigned int intc_ex_irq4_pins[] = {
- /* IRQ4 */
- RCAR_GP_PIN(2, 4),
-};
-static const unsigned int intc_ex_irq4_mux[] = {
- IRQ4_MARK,
-};
-static const unsigned int intc_ex_irq5_pins[] = {
- /* IRQ5 */
- RCAR_GP_PIN(2, 5),
-};
-static const unsigned int intc_ex_irq5_mux[] = {
- IRQ5_MARK,
-};
-
-/* - MSIOF0 ----------------------------------------------------------------- */
-static const unsigned int msiof0_clk_pins[] = {
- /* SCK */
- RCAR_GP_PIN(5, 17),
-};
-static const unsigned int msiof0_clk_mux[] = {
- MSIOF0_SCK_MARK,
-};
-static const unsigned int msiof0_sync_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(5, 18),
-};
-static const unsigned int msiof0_sync_mux[] = {
- MSIOF0_SYNC_MARK,
-};
-static const unsigned int msiof0_ss1_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(5, 19),
-};
-static const unsigned int msiof0_ss1_mux[] = {
- MSIOF0_SS1_MARK,
-};
-static const unsigned int msiof0_ss2_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(5, 21),
-};
-static const unsigned int msiof0_ss2_mux[] = {
- MSIOF0_SS2_MARK,
-};
-static const unsigned int msiof0_txd_pins[] = {
- /* TXD */
- RCAR_GP_PIN(5, 20),
-};
-static const unsigned int msiof0_txd_mux[] = {
- MSIOF0_TXD_MARK,
-};
-static const unsigned int msiof0_rxd_pins[] = {
- /* RXD */
- RCAR_GP_PIN(5, 22),
-};
-static const unsigned int msiof0_rxd_mux[] = {
- MSIOF0_RXD_MARK,
-};
-/* - MSIOF1 ----------------------------------------------------------------- */
-static const unsigned int msiof1_clk_a_pins[] = {
- /* SCK */
- RCAR_GP_PIN(6, 8),
-};
-static const unsigned int msiof1_clk_a_mux[] = {
- MSIOF1_SCK_A_MARK,
-};
-static const unsigned int msiof1_sync_a_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(6, 9),
-};
-static const unsigned int msiof1_sync_a_mux[] = {
- MSIOF1_SYNC_A_MARK,
-};
-static const unsigned int msiof1_ss1_a_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(6, 5),
-};
-static const unsigned int msiof1_ss1_a_mux[] = {
- MSIOF1_SS1_A_MARK,
-};
-static const unsigned int msiof1_ss2_a_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(6, 6),
-};
-static const unsigned int msiof1_ss2_a_mux[] = {
- MSIOF1_SS2_A_MARK,
-};
-static const unsigned int msiof1_txd_a_pins[] = {
- /* TXD */
- RCAR_GP_PIN(6, 7),
-};
-static const unsigned int msiof1_txd_a_mux[] = {
- MSIOF1_TXD_A_MARK,
-};
-static const unsigned int msiof1_rxd_a_pins[] = {
- /* RXD */
- RCAR_GP_PIN(6, 10),
-};
-static const unsigned int msiof1_rxd_a_mux[] = {
- MSIOF1_RXD_A_MARK,
-};
-static const unsigned int msiof1_clk_b_pins[] = {
- /* SCK */
- RCAR_GP_PIN(5, 9),
-};
-static const unsigned int msiof1_clk_b_mux[] = {
- MSIOF1_SCK_B_MARK,
-};
-static const unsigned int msiof1_sync_b_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(5, 3),
-};
-static const unsigned int msiof1_sync_b_mux[] = {
- MSIOF1_SYNC_B_MARK,
-};
-static const unsigned int msiof1_ss1_b_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(5, 4),
-};
-static const unsigned int msiof1_ss1_b_mux[] = {
- MSIOF1_SS1_B_MARK,
-};
-static const unsigned int msiof1_ss2_b_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(5, 0),
-};
-static const unsigned int msiof1_ss2_b_mux[] = {
- MSIOF1_SS2_B_MARK,
-};
-static const unsigned int msiof1_txd_b_pins[] = {
- /* TXD */
- RCAR_GP_PIN(5, 8),
-};
-static const unsigned int msiof1_txd_b_mux[] = {
- MSIOF1_TXD_B_MARK,
-};
-static const unsigned int msiof1_rxd_b_pins[] = {
- /* RXD */
- RCAR_GP_PIN(5, 7),
-};
-static const unsigned int msiof1_rxd_b_mux[] = {
- MSIOF1_RXD_B_MARK,
-};
-static const unsigned int msiof1_clk_c_pins[] = {
- /* SCK */
- RCAR_GP_PIN(6, 17),
-};
-static const unsigned int msiof1_clk_c_mux[] = {
- MSIOF1_SCK_C_MARK,
-};
-static const unsigned int msiof1_sync_c_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(6, 18),
-};
-static const unsigned int msiof1_sync_c_mux[] = {
- MSIOF1_SYNC_C_MARK,
-};
-static const unsigned int msiof1_ss1_c_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(6, 21),
-};
-static const unsigned int msiof1_ss1_c_mux[] = {
- MSIOF1_SS1_C_MARK,
-};
-static const unsigned int msiof1_ss2_c_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(6, 27),
-};
-static const unsigned int msiof1_ss2_c_mux[] = {
- MSIOF1_SS2_C_MARK,
-};
-static const unsigned int msiof1_txd_c_pins[] = {
- /* TXD */
- RCAR_GP_PIN(6, 20),
-};
-static const unsigned int msiof1_txd_c_mux[] = {
- MSIOF1_TXD_C_MARK,
-};
-static const unsigned int msiof1_rxd_c_pins[] = {
- /* RXD */
- RCAR_GP_PIN(6, 19),
-};
-static const unsigned int msiof1_rxd_c_mux[] = {
- MSIOF1_RXD_C_MARK,
-};
-static const unsigned int msiof1_clk_d_pins[] = {
- /* SCK */
- RCAR_GP_PIN(5, 12),
-};
-static const unsigned int msiof1_clk_d_mux[] = {
- MSIOF1_SCK_D_MARK,
-};
-static const unsigned int msiof1_sync_d_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(5, 15),
-};
-static const unsigned int msiof1_sync_d_mux[] = {
- MSIOF1_SYNC_D_MARK,
-};
-static const unsigned int msiof1_ss1_d_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(5, 16),
-};
-static const unsigned int msiof1_ss1_d_mux[] = {
- MSIOF1_SS1_D_MARK,
-};
-static const unsigned int msiof1_ss2_d_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(5, 21),
-};
-static const unsigned int msiof1_ss2_d_mux[] = {
- MSIOF1_SS2_D_MARK,
-};
-static const unsigned int msiof1_txd_d_pins[] = {
- /* TXD */
- RCAR_GP_PIN(5, 14),
-};
-static const unsigned int msiof1_txd_d_mux[] = {
- MSIOF1_TXD_D_MARK,
-};
-static const unsigned int msiof1_rxd_d_pins[] = {
- /* RXD */
- RCAR_GP_PIN(5, 13),
-};
-static const unsigned int msiof1_rxd_d_mux[] = {
- MSIOF1_RXD_D_MARK,
-};
-static const unsigned int msiof1_clk_e_pins[] = {
- /* SCK */
- RCAR_GP_PIN(3, 0),
-};
-static const unsigned int msiof1_clk_e_mux[] = {
- MSIOF1_SCK_E_MARK,
-};
-static const unsigned int msiof1_sync_e_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(3, 1),
-};
-static const unsigned int msiof1_sync_e_mux[] = {
- MSIOF1_SYNC_E_MARK,
-};
-static const unsigned int msiof1_ss1_e_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(3, 4),
-};
-static const unsigned int msiof1_ss1_e_mux[] = {
- MSIOF1_SS1_E_MARK,
-};
-static const unsigned int msiof1_ss2_e_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(3, 5),
-};
-static const unsigned int msiof1_ss2_e_mux[] = {
- MSIOF1_SS2_E_MARK,
-};
-static const unsigned int msiof1_txd_e_pins[] = {
- /* TXD */
- RCAR_GP_PIN(3, 3),
-};
-static const unsigned int msiof1_txd_e_mux[] = {
- MSIOF1_TXD_E_MARK,
-};
-static const unsigned int msiof1_rxd_e_pins[] = {
- /* RXD */
- RCAR_GP_PIN(3, 2),
-};
-static const unsigned int msiof1_rxd_e_mux[] = {
- MSIOF1_RXD_E_MARK,
-};
-static const unsigned int msiof1_clk_f_pins[] = {
- /* SCK */
- RCAR_GP_PIN(5, 23),
-};
-static const unsigned int msiof1_clk_f_mux[] = {
- MSIOF1_SCK_F_MARK,
-};
-static const unsigned int msiof1_sync_f_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(5, 24),
-};
-static const unsigned int msiof1_sync_f_mux[] = {
- MSIOF1_SYNC_F_MARK,
-};
-static const unsigned int msiof1_ss1_f_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(6, 1),
-};
-static const unsigned int msiof1_ss1_f_mux[] = {
- MSIOF1_SS1_F_MARK,
-};
-static const unsigned int msiof1_ss2_f_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(6, 2),
-};
-static const unsigned int msiof1_ss2_f_mux[] = {
- MSIOF1_SS2_F_MARK,
-};
-static const unsigned int msiof1_txd_f_pins[] = {
- /* TXD */
- RCAR_GP_PIN(6, 0),
-};
-static const unsigned int msiof1_txd_f_mux[] = {
- MSIOF1_TXD_F_MARK,
-};
-static const unsigned int msiof1_rxd_f_pins[] = {
- /* RXD */
- RCAR_GP_PIN(5, 25),
-};
-static const unsigned int msiof1_rxd_f_mux[] = {
- MSIOF1_RXD_F_MARK,
-};
-static const unsigned int msiof1_clk_g_pins[] = {
- /* SCK */
- RCAR_GP_PIN(3, 6),
-};
-static const unsigned int msiof1_clk_g_mux[] = {
- MSIOF1_SCK_G_MARK,
-};
-static const unsigned int msiof1_sync_g_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(3, 7),
-};
-static const unsigned int msiof1_sync_g_mux[] = {
- MSIOF1_SYNC_G_MARK,
-};
-static const unsigned int msiof1_ss1_g_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(3, 10),
-};
-static const unsigned int msiof1_ss1_g_mux[] = {
- MSIOF1_SS1_G_MARK,
-};
-static const unsigned int msiof1_ss2_g_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(3, 11),
-};
-static const unsigned int msiof1_ss2_g_mux[] = {
- MSIOF1_SS2_G_MARK,
-};
-static const unsigned int msiof1_txd_g_pins[] = {
- /* TXD */
- RCAR_GP_PIN(3, 9),
-};
-static const unsigned int msiof1_txd_g_mux[] = {
- MSIOF1_TXD_G_MARK,
-};
-static const unsigned int msiof1_rxd_g_pins[] = {
- /* RXD */
- RCAR_GP_PIN(3, 8),
-};
-static const unsigned int msiof1_rxd_g_mux[] = {
- MSIOF1_RXD_G_MARK,
-};
-/* - MSIOF2 ----------------------------------------------------------------- */
-static const unsigned int msiof2_clk_a_pins[] = {
- /* SCK */
- RCAR_GP_PIN(1, 9),
-};
-static const unsigned int msiof2_clk_a_mux[] = {
- MSIOF2_SCK_A_MARK,
-};
-static const unsigned int msiof2_sync_a_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(1, 8),
-};
-static const unsigned int msiof2_sync_a_mux[] = {
- MSIOF2_SYNC_A_MARK,
-};
-static const unsigned int msiof2_ss1_a_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(1, 6),
-};
-static const unsigned int msiof2_ss1_a_mux[] = {
- MSIOF2_SS1_A_MARK,
-};
-static const unsigned int msiof2_ss2_a_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(1, 7),
-};
-static const unsigned int msiof2_ss2_a_mux[] = {
- MSIOF2_SS2_A_MARK,
-};
-static const unsigned int msiof2_txd_a_pins[] = {
- /* TXD */
- RCAR_GP_PIN(1, 11),
-};
-static const unsigned int msiof2_txd_a_mux[] = {
- MSIOF2_TXD_A_MARK,
-};
-static const unsigned int msiof2_rxd_a_pins[] = {
- /* RXD */
- RCAR_GP_PIN(1, 10),
-};
-static const unsigned int msiof2_rxd_a_mux[] = {
- MSIOF2_RXD_A_MARK,
-};
-static const unsigned int msiof2_clk_b_pins[] = {
- /* SCK */
- RCAR_GP_PIN(0, 4),
-};
-static const unsigned int msiof2_clk_b_mux[] = {
- MSIOF2_SCK_B_MARK,
-};
-static const unsigned int msiof2_sync_b_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(0, 5),
-};
-static const unsigned int msiof2_sync_b_mux[] = {
- MSIOF2_SYNC_B_MARK,
-};
-static const unsigned int msiof2_ss1_b_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(0, 0),
-};
-static const unsigned int msiof2_ss1_b_mux[] = {
- MSIOF2_SS1_B_MARK,
-};
-static const unsigned int msiof2_ss2_b_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(0, 1),
-};
-static const unsigned int msiof2_ss2_b_mux[] = {
- MSIOF2_SS2_B_MARK,
-};
-static const unsigned int msiof2_txd_b_pins[] = {
- /* TXD */
- RCAR_GP_PIN(0, 7),
-};
-static const unsigned int msiof2_txd_b_mux[] = {
- MSIOF2_TXD_B_MARK,
-};
-static const unsigned int msiof2_rxd_b_pins[] = {
- /* RXD */
- RCAR_GP_PIN(0, 6),
-};
-static const unsigned int msiof2_rxd_b_mux[] = {
- MSIOF2_RXD_B_MARK,
-};
-static const unsigned int msiof2_clk_c_pins[] = {
- /* SCK */
- RCAR_GP_PIN(2, 12),
-};
-static const unsigned int msiof2_clk_c_mux[] = {
- MSIOF2_SCK_C_MARK,
-};
-static const unsigned int msiof2_sync_c_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(2, 11),
-};
-static const unsigned int msiof2_sync_c_mux[] = {
- MSIOF2_SYNC_C_MARK,
-};
-static const unsigned int msiof2_ss1_c_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(2, 10),
-};
-static const unsigned int msiof2_ss1_c_mux[] = {
- MSIOF2_SS1_C_MARK,
-};
-static const unsigned int msiof2_ss2_c_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(2, 9),
-};
-static const unsigned int msiof2_ss2_c_mux[] = {
- MSIOF2_SS2_C_MARK,
-};
-static const unsigned int msiof2_txd_c_pins[] = {
- /* TXD */
- RCAR_GP_PIN(2, 14),
-};
-static const unsigned int msiof2_txd_c_mux[] = {
- MSIOF2_TXD_C_MARK,
-};
-static const unsigned int msiof2_rxd_c_pins[] = {
- /* RXD */
- RCAR_GP_PIN(2, 13),
-};
-static const unsigned int msiof2_rxd_c_mux[] = {
- MSIOF2_RXD_C_MARK,
-};
-static const unsigned int msiof2_clk_d_pins[] = {
- /* SCK */
- RCAR_GP_PIN(0, 8),
-};
-static const unsigned int msiof2_clk_d_mux[] = {
- MSIOF2_SCK_D_MARK,
-};
-static const unsigned int msiof2_sync_d_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(0, 9),
-};
-static const unsigned int msiof2_sync_d_mux[] = {
- MSIOF2_SYNC_D_MARK,
-};
-static const unsigned int msiof2_ss1_d_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(0, 12),
-};
-static const unsigned int msiof2_ss1_d_mux[] = {
- MSIOF2_SS1_D_MARK,
-};
-static const unsigned int msiof2_ss2_d_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(0, 13),
-};
-static const unsigned int msiof2_ss2_d_mux[] = {
- MSIOF2_SS2_D_MARK,
-};
-static const unsigned int msiof2_txd_d_pins[] = {
- /* TXD */
- RCAR_GP_PIN(0, 11),
-};
-static const unsigned int msiof2_txd_d_mux[] = {
- MSIOF2_TXD_D_MARK,
-};
-static const unsigned int msiof2_rxd_d_pins[] = {
- /* RXD */
- RCAR_GP_PIN(0, 10),
-};
-static const unsigned int msiof2_rxd_d_mux[] = {
- MSIOF2_RXD_D_MARK,
-};
-/* - MSIOF3 ----------------------------------------------------------------- */
-static const unsigned int msiof3_clk_a_pins[] = {
- /* SCK */
- RCAR_GP_PIN(0, 0),
-};
-static const unsigned int msiof3_clk_a_mux[] = {
- MSIOF3_SCK_A_MARK,
-};
-static const unsigned int msiof3_sync_a_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(0, 1),
-};
-static const unsigned int msiof3_sync_a_mux[] = {
- MSIOF3_SYNC_A_MARK,
-};
-static const unsigned int msiof3_ss1_a_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(0, 14),
-};
-static const unsigned int msiof3_ss1_a_mux[] = {
- MSIOF3_SS1_A_MARK,
-};
-static const unsigned int msiof3_ss2_a_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(0, 15),
-};
-static const unsigned int msiof3_ss2_a_mux[] = {
- MSIOF3_SS2_A_MARK,
-};
-static const unsigned int msiof3_txd_a_pins[] = {
- /* TXD */
- RCAR_GP_PIN(0, 3),
-};
-static const unsigned int msiof3_txd_a_mux[] = {
- MSIOF3_TXD_A_MARK,
-};
-static const unsigned int msiof3_rxd_a_pins[] = {
- /* RXD */
- RCAR_GP_PIN(0, 2),
-};
-static const unsigned int msiof3_rxd_a_mux[] = {
- MSIOF3_RXD_A_MARK,
-};
-static const unsigned int msiof3_clk_b_pins[] = {
- /* SCK */
- RCAR_GP_PIN(1, 2),
-};
-static const unsigned int msiof3_clk_b_mux[] = {
- MSIOF3_SCK_B_MARK,
-};
-static const unsigned int msiof3_sync_b_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(1, 0),
-};
-static const unsigned int msiof3_sync_b_mux[] = {
- MSIOF3_SYNC_B_MARK,
-};
-static const unsigned int msiof3_ss1_b_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(1, 4),
-};
-static const unsigned int msiof3_ss1_b_mux[] = {
- MSIOF3_SS1_B_MARK,
-};
-static const unsigned int msiof3_ss2_b_pins[] = {
- /* SS2 */
- RCAR_GP_PIN(1, 5),
-};
-static const unsigned int msiof3_ss2_b_mux[] = {
- MSIOF3_SS2_B_MARK,
-};
-static const unsigned int msiof3_txd_b_pins[] = {
- /* TXD */
- RCAR_GP_PIN(1, 1),
-};
-static const unsigned int msiof3_txd_b_mux[] = {
- MSIOF3_TXD_B_MARK,
-};
-static const unsigned int msiof3_rxd_b_pins[] = {
- /* RXD */
- RCAR_GP_PIN(1, 3),
-};
-static const unsigned int msiof3_rxd_b_mux[] = {
- MSIOF3_RXD_B_MARK,
-};
-static const unsigned int msiof3_clk_c_pins[] = {
- /* SCK */
- RCAR_GP_PIN(1, 12),
-};
-static const unsigned int msiof3_clk_c_mux[] = {
- MSIOF3_SCK_C_MARK,
-};
-static const unsigned int msiof3_sync_c_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(1, 13),
-};
-static const unsigned int msiof3_sync_c_mux[] = {
- MSIOF3_SYNC_C_MARK,
-};
-static const unsigned int msiof3_txd_c_pins[] = {
- /* TXD */
- RCAR_GP_PIN(1, 15),
-};
-static const unsigned int msiof3_txd_c_mux[] = {
- MSIOF3_TXD_C_MARK,
-};
-static const unsigned int msiof3_rxd_c_pins[] = {
- /* RXD */
- RCAR_GP_PIN(1, 14),
-};
-static const unsigned int msiof3_rxd_c_mux[] = {
- MSIOF3_RXD_C_MARK,
-};
-static const unsigned int msiof3_clk_d_pins[] = {
- /* SCK */
- RCAR_GP_PIN(1, 22),
-};
-static const unsigned int msiof3_clk_d_mux[] = {
- MSIOF3_SCK_D_MARK,
-};
-static const unsigned int msiof3_sync_d_pins[] = {
- /* SYNC */
- RCAR_GP_PIN(1, 23),
-};
-static const unsigned int msiof3_sync_d_mux[] = {
- MSIOF3_SYNC_D_MARK,
-};
-static const unsigned int msiof3_ss1_d_pins[] = {
- /* SS1 */
- RCAR_GP_PIN(1, 26),
-};
-static const unsigned int msiof3_ss1_d_mux[] = {
- MSIOF3_SS1_D_MARK,
-};
-static const unsigned int msiof3_txd_d_pins[] = {
- /* TXD */
- RCAR_GP_PIN(1, 25),
-};
-static const unsigned int msiof3_txd_d_mux[] = {
- MSIOF3_TXD_D_MARK,
-};
-static const unsigned int msiof3_rxd_d_pins[] = {
- /* RXD */
- RCAR_GP_PIN(1, 24),
-};
-static const unsigned int msiof3_rxd_d_mux[] = {
- MSIOF3_RXD_D_MARK,
-};
-
-/* - PWM0 --------------------------------------------------------------------*/
-static const unsigned int pwm0_pins[] = {
- /* PWM */
- RCAR_GP_PIN(2, 6),
-};
-static const unsigned int pwm0_mux[] = {
- PWM0_MARK,
-};
-/* - PWM1 --------------------------------------------------------------------*/
-static const unsigned int pwm1_a_pins[] = {
- /* PWM */
- RCAR_GP_PIN(2, 7),
-};
-static const unsigned int pwm1_a_mux[] = {
- PWM1_A_MARK,
-};
-static const unsigned int pwm1_b_pins[] = {
- /* PWM */
- RCAR_GP_PIN(1, 8),
-};
-static const unsigned int pwm1_b_mux[] = {
- PWM1_B_MARK,
-};
-/* - PWM2 --------------------------------------------------------------------*/
-static const unsigned int pwm2_a_pins[] = {
- /* PWM */
- RCAR_GP_PIN(2, 8),
-};
-static const unsigned int pwm2_a_mux[] = {
- PWM2_A_MARK,
-};
-static const unsigned int pwm2_b_pins[] = {
- /* PWM */
- RCAR_GP_PIN(1, 11),
-};
-static const unsigned int pwm2_b_mux[] = {
- PWM2_B_MARK,
-};
-/* - PWM3 --------------------------------------------------------------------*/
-static const unsigned int pwm3_a_pins[] = {
- /* PWM */
- RCAR_GP_PIN(1, 0),
-};
-static const unsigned int pwm3_a_mux[] = {
- PWM3_A_MARK,
-};
-static const unsigned int pwm3_b_pins[] = {
- /* PWM */
- RCAR_GP_PIN(2, 2),
-};
-static const unsigned int pwm3_b_mux[] = {
- PWM3_B_MARK,
-};
-/* - PWM4 --------------------------------------------------------------------*/
-static const unsigned int pwm4_a_pins[] = {
- /* PWM */
- RCAR_GP_PIN(1, 1),
-};
-static const unsigned int pwm4_a_mux[] = {
- PWM4_A_MARK,
-};
-static const unsigned int pwm4_b_pins[] = {
- /* PWM */
- RCAR_GP_PIN(2, 3),
-};
-static const unsigned int pwm4_b_mux[] = {
- PWM4_B_MARK,
-};
-/* - PWM5 --------------------------------------------------------------------*/
-static const unsigned int pwm5_a_pins[] = {
- /* PWM */
- RCAR_GP_PIN(1, 2),
-};
-static const unsigned int pwm5_a_mux[] = {
- PWM5_A_MARK,
-};
-static const unsigned int pwm5_b_pins[] = {
- /* PWM */
- RCAR_GP_PIN(2, 4),
-};
-static const unsigned int pwm5_b_mux[] = {
- PWM5_B_MARK,
-};
-/* - PWM6 --------------------------------------------------------------------*/
-static const unsigned int pwm6_a_pins[] = {
- /* PWM */
- RCAR_GP_PIN(1, 3),
-};
-static const unsigned int pwm6_a_mux[] = {
- PWM6_A_MARK,
-};
-static const unsigned int pwm6_b_pins[] = {
- /* PWM */
- RCAR_GP_PIN(2, 5),
-};
-static const unsigned int pwm6_b_mux[] = {
- PWM6_B_MARK,
-};
-
-/* - SATA --------------------------------------------------------------------*/
-static const unsigned int sata0_devslp_a_pins[] = {
- /* DEVSLP */
- RCAR_GP_PIN(6, 16),
-};
-static const unsigned int sata0_devslp_a_mux[] = {
- SATA_DEVSLP_A_MARK,
-};
-static const unsigned int sata0_devslp_b_pins[] = {
- /* DEVSLP */
- RCAR_GP_PIN(4, 6),
-};
-static const unsigned int sata0_devslp_b_mux[] = {
- SATA_DEVSLP_B_MARK,
-};
-
/* - SCIF0 ------------------------------------------------------------------ */
static const unsigned int scif0_data_pins[] = {
/* RX, TX */
@@ -3285,225 +1744,33 @@
RTS4_N_TANS_C_MARK, CTS4_N_C_MARK,
};
/* - SCIF5 ------------------------------------------------------------------ */
-static const unsigned int scif5_data_pins[] = {
+static const unsigned int scif5_data_a_pins[] = {
/* RX, TX */
RCAR_GP_PIN(5, 19), RCAR_GP_PIN(5, 21),
};
-static const unsigned int scif5_data_mux[] = {
- RX5_MARK, TX5_MARK,
+static const unsigned int scif5_data_a_mux[] = {
+ RX5_A_MARK, TX5_A_MARK,
};
-static const unsigned int scif5_clk_pins[] = {
+static const unsigned int scif5_clk_a_pins[] = {
/* SCK */
RCAR_GP_PIN(6, 21),
};
-static const unsigned int scif5_clk_mux[] = {
- SCK5_MARK,
+static const unsigned int scif5_clk_a_mux[] = {
+ SCK5_A_MARK,
};
-/* - SDHI0 ------------------------------------------------------------------ */
-static const unsigned int sdhi0_data1_pins[] = {
- /* D0 */
- RCAR_GP_PIN(3, 2),
+static const unsigned int scif5_data_b_pins[] = {
+ /* RX, TX */
+ RCAR_GP_PIN(5, 12), RCAR_GP_PIN(5, 18),
};
-static const unsigned int sdhi0_data1_mux[] = {
- SD0_DAT0_MARK,
+static const unsigned int scif5_data_b_mux[] = {
+ RX5_B_MARK, TX5_B_MARK,
};
-static const unsigned int sdhi0_data4_pins[] = {
- /* D[0:3] */
- RCAR_GP_PIN(3, 2), RCAR_GP_PIN(3, 3),
- RCAR_GP_PIN(3, 4), RCAR_GP_PIN(3, 5),
+static const unsigned int scif5_clk_b_pins[] = {
+ /* SCK */
+ RCAR_GP_PIN(5, 0),
};
-static const unsigned int sdhi0_data4_mux[] = {
- SD0_DAT0_MARK, SD0_DAT1_MARK,
- SD0_DAT2_MARK, SD0_DAT3_MARK,
-};
-static const unsigned int sdhi0_ctrl_pins[] = {
- /* CLK, CMD */
- RCAR_GP_PIN(3, 0), RCAR_GP_PIN(3, 1),
-};
-static const unsigned int sdhi0_ctrl_mux[] = {
- SD0_CLK_MARK, SD0_CMD_MARK,
-};
-static const unsigned int sdhi0_cd_pins[] = {
- /* CD */
- RCAR_GP_PIN(3, 12),
-};
-static const unsigned int sdhi0_cd_mux[] = {
- SD0_CD_MARK,
-};
-static const unsigned int sdhi0_wp_pins[] = {
- /* WP */
- RCAR_GP_PIN(3, 13),
-};
-static const unsigned int sdhi0_wp_mux[] = {
- SD0_WP_MARK,
-};
-/* - SDHI1 ------------------------------------------------------------------ */
-static const unsigned int sdhi1_data1_pins[] = {
- /* D0 */
- RCAR_GP_PIN(3, 8),
-};
-static const unsigned int sdhi1_data1_mux[] = {
- SD1_DAT0_MARK,
-};
-static const unsigned int sdhi1_data4_pins[] = {
- /* D[0:3] */
- RCAR_GP_PIN(3, 8), RCAR_GP_PIN(3, 9),
- RCAR_GP_PIN(3, 10), RCAR_GP_PIN(3, 11),
-};
-static const unsigned int sdhi1_data4_mux[] = {
- SD1_DAT0_MARK, SD1_DAT1_MARK,
- SD1_DAT2_MARK, SD1_DAT3_MARK,
-};
-static const unsigned int sdhi1_ctrl_pins[] = {
- /* CLK, CMD */
- RCAR_GP_PIN(3, 6), RCAR_GP_PIN(3, 7),
-};
-static const unsigned int sdhi1_ctrl_mux[] = {
- SD1_CLK_MARK, SD1_CMD_MARK,
-};
-static const unsigned int sdhi1_cd_pins[] = {
- /* CD */
- RCAR_GP_PIN(3, 14),
-};
-static const unsigned int sdhi1_cd_mux[] = {
- SD1_CD_MARK,
-};
-static const unsigned int sdhi1_wp_pins[] = {
- /* WP */
- RCAR_GP_PIN(3, 15),
-};
-static const unsigned int sdhi1_wp_mux[] = {
- SD1_WP_MARK,
-};
-/* - SDHI2 ------------------------------------------------------------------ */
-static const unsigned int sdhi2_data1_pins[] = {
- /* D0 */
- RCAR_GP_PIN(4, 2),
-};
-static const unsigned int sdhi2_data1_mux[] = {
- SD2_DAT0_MARK,
-};
-static const unsigned int sdhi2_data4_pins[] = {
- /* D[0:3] */
- RCAR_GP_PIN(4, 2), RCAR_GP_PIN(4, 3),
- RCAR_GP_PIN(4, 4), RCAR_GP_PIN(4, 5),
-};
-static const unsigned int sdhi2_data4_mux[] = {
- SD2_DAT0_MARK, SD2_DAT1_MARK,
- SD2_DAT2_MARK, SD2_DAT3_MARK,
-};
-static const unsigned int sdhi2_data8_pins[] = {
- /* D[0:7] */
- RCAR_GP_PIN(4, 2), RCAR_GP_PIN(4, 3),
- RCAR_GP_PIN(4, 4), RCAR_GP_PIN(4, 5),
- RCAR_GP_PIN(3, 8), RCAR_GP_PIN(3, 9),
- RCAR_GP_PIN(3, 10), RCAR_GP_PIN(3, 11),
-};
-static const unsigned int sdhi2_data8_mux[] = {
- SD2_DAT0_MARK, SD2_DAT1_MARK,
- SD2_DAT2_MARK, SD2_DAT3_MARK,
- SD2_DAT4_MARK, SD2_DAT5_MARK,
- SD2_DAT6_MARK, SD2_DAT7_MARK,
-};
-static const unsigned int sdhi2_ctrl_pins[] = {
- /* CLK, CMD */
- RCAR_GP_PIN(4, 0), RCAR_GP_PIN(4, 1),
-};
-static const unsigned int sdhi2_ctrl_mux[] = {
- SD2_CLK_MARK, SD2_CMD_MARK,
-};
-static const unsigned int sdhi2_cd_a_pins[] = {
- /* CD */
- RCAR_GP_PIN(4, 13),
-};
-static const unsigned int sdhi2_cd_a_mux[] = {
- SD2_CD_A_MARK,
-};
-static const unsigned int sdhi2_cd_b_pins[] = {
- /* CD */
- RCAR_GP_PIN(5, 10),
-};
-static const unsigned int sdhi2_cd_b_mux[] = {
- SD2_CD_B_MARK,
-};
-static const unsigned int sdhi2_wp_a_pins[] = {
- /* WP */
- RCAR_GP_PIN(4, 14),
-};
-static const unsigned int sdhi2_wp_a_mux[] = {
- SD2_WP_A_MARK,
-};
-static const unsigned int sdhi2_wp_b_pins[] = {
- /* WP */
- RCAR_GP_PIN(5, 11),
-};
-static const unsigned int sdhi2_wp_b_mux[] = {
- SD2_WP_B_MARK,
-};
-static const unsigned int sdhi2_ds_pins[] = {
- /* DS */
- RCAR_GP_PIN(4, 6),
-};
-static const unsigned int sdhi2_ds_mux[] = {
- SD2_DS_MARK,
-};
-/* - SDHI3 ------------------------------------------------------------------ */
-static const unsigned int sdhi3_data1_pins[] = {
- /* D0 */
- RCAR_GP_PIN(4, 9),
-};
-static const unsigned int sdhi3_data1_mux[] = {
- SD3_DAT0_MARK,
-};
-static const unsigned int sdhi3_data4_pins[] = {
- /* D[0:3] */
- RCAR_GP_PIN(4, 9), RCAR_GP_PIN(4, 10),
- RCAR_GP_PIN(4, 11), RCAR_GP_PIN(4, 12),
-};
-static const unsigned int sdhi3_data4_mux[] = {
- SD3_DAT0_MARK, SD3_DAT1_MARK,
- SD3_DAT2_MARK, SD3_DAT3_MARK,
-};
-static const unsigned int sdhi3_data8_pins[] = {
- /* D[0:7] */
- RCAR_GP_PIN(4, 9), RCAR_GP_PIN(4, 10),
- RCAR_GP_PIN(4, 11), RCAR_GP_PIN(4, 12),
- RCAR_GP_PIN(4, 13), RCAR_GP_PIN(4, 14),
- RCAR_GP_PIN(4, 15), RCAR_GP_PIN(4, 16),
-};
-static const unsigned int sdhi3_data8_mux[] = {
- SD3_DAT0_MARK, SD3_DAT1_MARK,
- SD3_DAT2_MARK, SD3_DAT3_MARK,
- SD3_DAT4_MARK, SD3_DAT5_MARK,
- SD3_DAT6_MARK, SD3_DAT7_MARK,
-};
-static const unsigned int sdhi3_ctrl_pins[] = {
- /* CLK, CMD */
- RCAR_GP_PIN(4, 7), RCAR_GP_PIN(4, 8),
-};
-static const unsigned int sdhi3_ctrl_mux[] = {
- SD3_CLK_MARK, SD3_CMD_MARK,
-};
-static const unsigned int sdhi3_cd_pins[] = {
- /* CD */
- RCAR_GP_PIN(4, 15),
-};
-static const unsigned int sdhi3_cd_mux[] = {
- SD3_CD_MARK,
-};
-static const unsigned int sdhi3_wp_pins[] = {
- /* WP */
- RCAR_GP_PIN(4, 16),
-};
-static const unsigned int sdhi3_wp_mux[] = {
- SD3_WP_MARK,
-};
-static const unsigned int sdhi3_ds_pins[] = {
- /* DS */
- RCAR_GP_PIN(4, 17),
-};
-static const unsigned int sdhi3_ds_mux[] = {
- SD3_DS_MARK,
+static const unsigned int scif5_clk_b_mux[] = {
+ SCK5_B_MARK,
};
/* - SCIF Clock ------------------------------------------------------------- */
@@ -3522,476 +1789,7 @@
SCIF_CLK_B_MARK,
};
-/* - SSI -------------------------------------------------------------------- */
-static const unsigned int ssi0_data_pins[] = {
- /* SDATA */
- RCAR_GP_PIN(6, 2),
-};
-static const unsigned int ssi0_data_mux[] = {
- SSI_SDATA0_MARK,
-};
-static const unsigned int ssi01239_ctrl_pins[] = {
- /* SCK, WS */
- RCAR_GP_PIN(6, 0), RCAR_GP_PIN(6, 1),
-};
-static const unsigned int ssi01239_ctrl_mux[] = {
- SSI_SCK01239_MARK, SSI_WS01239_MARK,
-};
-static const unsigned int ssi1_data_a_pins[] = {
- /* SDATA */
- RCAR_GP_PIN(6, 3),
-};
-static const unsigned int ssi1_data_a_mux[] = {
- SSI_SDATA1_A_MARK,
-};
-static const unsigned int ssi1_data_b_pins[] = {
- /* SDATA */
- RCAR_GP_PIN(5, 12),
-};
-static const unsigned int ssi1_data_b_mux[] = {
- SSI_SDATA1_B_MARK,
-};
-static const unsigned int ssi1_ctrl_a_pins[] = {
- /* SCK, WS */
- RCAR_GP_PIN(6, 26), RCAR_GP_PIN(6, 27),
-};
-static const unsigned int ssi1_ctrl_a_mux[] = {
- SSI_SCK1_A_MARK, SSI_WS1_A_MARK,
-};
-static const unsigned int ssi1_ctrl_b_pins[] = {
- /* SCK, WS */
- RCAR_GP_PIN(6, 4), RCAR_GP_PIN(6, 21),
-};
-static const unsigned int ssi1_ctrl_b_mux[] = {
- SSI_SCK1_B_MARK, SSI_WS1_B_MARK,
-};
-static const unsigned int ssi2_data_a_pins[] = {
- /* SDATA */
- RCAR_GP_PIN(6, 4),
-};
-static const unsigned int ssi2_data_a_mux[] = {
- SSI_SDATA2_A_MARK,
-};
-static const unsigned int ssi2_data_b_pins[] = {
- /* SDATA */
- RCAR_GP_PIN(5, 13),
-};
-static const unsigned int ssi2_data_b_mux[] = {
- SSI_SDATA2_B_MARK,
-};
-static const unsigned int ssi2_ctrl_a_pins[] = {
- /* SCK, WS */
- RCAR_GP_PIN(5, 19), RCAR_GP_PIN(5, 21),
-};
-static const unsigned int ssi2_ctrl_a_mux[] = {
- SSI_SCK2_A_MARK, SSI_WS2_A_MARK,
-};
-static const unsigned int ssi2_ctrl_b_pins[] = {
- /* SCK, WS */
- RCAR_GP_PIN(6, 28), RCAR_GP_PIN(6, 29),
-};
-static const unsigned int ssi2_ctrl_b_mux[] = {
- SSI_SCK2_B_MARK, SSI_WS2_B_MARK,
-};
-static const unsigned int ssi3_data_pins[] = {
- /* SDATA */
- RCAR_GP_PIN(6, 7),
-};
-static const unsigned int ssi3_data_mux[] = {
- SSI_SDATA3_MARK,
-};
-static const unsigned int ssi34_ctrl_pins[] = {
- /* SCK, WS */
- RCAR_GP_PIN(6, 5), RCAR_GP_PIN(6, 6),
-};
-static const unsigned int ssi34_ctrl_mux[] = {
- SSI_SCK34_MARK, SSI_WS34_MARK,
-};
-static const unsigned int ssi4_data_pins[] = {
- /* SDATA */
- RCAR_GP_PIN(6, 10),
-};
-static const unsigned int ssi4_data_mux[] = {
- SSI_SDATA4_MARK,
-};
-static const unsigned int ssi4_ctrl_pins[] = {
- /* SCK, WS */
- RCAR_GP_PIN(6, 8), RCAR_GP_PIN(6, 9),
-};
-static const unsigned int ssi4_ctrl_mux[] = {
- SSI_SCK4_MARK, SSI_WS4_MARK,
-};
-static const unsigned int ssi5_data_pins[] = {
- /* SDATA */
- RCAR_GP_PIN(6, 13),
-};
-static const unsigned int ssi5_data_mux[] = {
- SSI_SDATA5_MARK,
-};
-static const unsigned int ssi5_ctrl_pins[] = {
- /* SCK, WS */
- RCAR_GP_PIN(6, 11), RCAR_GP_PIN(6, 12),
-};
-static const unsigned int ssi5_ctrl_mux[] = {
- SSI_SCK5_MARK, SSI_WS5_MARK,
-};
-static const unsigned int ssi6_data_pins[] = {
- /* SDATA */
- RCAR_GP_PIN(6, 16),
-};
-static const unsigned int ssi6_data_mux[] = {
- SSI_SDATA6_MARK,
-};
-static const unsigned int ssi6_ctrl_pins[] = {
- /* SCK, WS */
- RCAR_GP_PIN(6, 14), RCAR_GP_PIN(6, 15),
-};
-static const unsigned int ssi6_ctrl_mux[] = {
- SSI_SCK6_MARK, SSI_WS6_MARK,
-};
-static const unsigned int ssi7_data_pins[] = {
- /* SDATA */
- RCAR_GP_PIN(6, 19),
-};
-static const unsigned int ssi7_data_mux[] = {
- SSI_SDATA7_MARK,
-};
-static const unsigned int ssi78_ctrl_pins[] = {
- /* SCK, WS */
- RCAR_GP_PIN(6, 17), RCAR_GP_PIN(6, 18),
-};
-static const unsigned int ssi78_ctrl_mux[] = {
- SSI_SCK78_MARK, SSI_WS78_MARK,
-};
-static const unsigned int ssi8_data_pins[] = {
- /* SDATA */
- RCAR_GP_PIN(6, 20),
-};
-static const unsigned int ssi8_data_mux[] = {
- SSI_SDATA8_MARK,
-};
-static const unsigned int ssi9_data_a_pins[] = {
- /* SDATA */
- RCAR_GP_PIN(6, 21),
-};
-static const unsigned int ssi9_data_a_mux[] = {
- SSI_SDATA9_A_MARK,
-};
-static const unsigned int ssi9_data_b_pins[] = {
- /* SDATA */
- RCAR_GP_PIN(5, 14),
-};
-static const unsigned int ssi9_data_b_mux[] = {
- SSI_SDATA9_B_MARK,
-};
-static const unsigned int ssi9_ctrl_a_pins[] = {
- /* SCK, WS */
- RCAR_GP_PIN(5, 15), RCAR_GP_PIN(5, 16),
-};
-static const unsigned int ssi9_ctrl_a_mux[] = {
- SSI_SCK9_A_MARK, SSI_WS9_A_MARK,
-};
-static const unsigned int ssi9_ctrl_b_pins[] = {
- /* SCK, WS */
- RCAR_GP_PIN(6, 30), RCAR_GP_PIN(6, 31),
-};
-static const unsigned int ssi9_ctrl_b_mux[] = {
- SSI_SCK9_B_MARK, SSI_WS9_B_MARK,
-};
-
-/* - USB0 ------------------------------------------------------------------- */
-static const unsigned int usb0_pins[] = {
- /* PWEN, OVC */
- RCAR_GP_PIN(6, 24), RCAR_GP_PIN(6, 25),
-};
-static const unsigned int usb0_mux[] = {
- USB0_PWEN_MARK, USB0_OVC_MARK,
-};
-/* - USB1 ------------------------------------------------------------------- */
-static const unsigned int usb1_pins[] = {
- /* PWEN, OVC */
- RCAR_GP_PIN(6, 26), RCAR_GP_PIN(6, 27),
-};
-static const unsigned int usb1_mux[] = {
- USB1_PWEN_MARK, USB1_OVC_MARK,
-};
-/* - USB2 ------------------------------------------------------------------- */
-static const unsigned int usb2_pins[] = {
- /* PWEN, OVC */
- RCAR_GP_PIN(6, 14), RCAR_GP_PIN(6, 15),
-};
-static const unsigned int usb2_mux[] = {
- USB2_PWEN_MARK, USB2_OVC_MARK,
-};
-
-/* - QSPI0 ------------------------------------------------------------------ */
-static const unsigned int qspi0_ctrl_pins[] = {
- /* QSPI0_SPCLK, QSPI0_SSL */
- PIN_NUMBER('W', 3), PIN_NUMBER('Y', 3),
-};
-static const unsigned int qspi0_ctrl_mux[] = {
- QSPI0_SPCLK_MARK, QSPI0_SSL_MARK,
-};
-static const unsigned int qspi0_data2_pins[] = {
- /* QSPI0_MOSI_IO0, QSPI0_MISO_IO1 */
- PIN_A_NUMBER('C', 5), PIN_A_NUMBER('B', 4),
-};
-static const unsigned int qspi0_data2_mux[] = {
- QSPI0_MOSI_IO0_MARK, QSPI0_MISO_IO1_MARK,
-};
-static const unsigned int qspi0_data4_pins[] = {
- /* QSPI0_MOSI_IO0, QSPI0_MISO_IO1, QSPI0_IO2, QSPI0_IO3 */
- PIN_A_NUMBER('C', 5), PIN_A_NUMBER('B', 4),
- PIN_NUMBER('Y', 6), PIN_A_NUMBER('B', 6),
-};
-static const unsigned int qspi0_data4_mux[] = {
- QSPI0_MOSI_IO0_MARK, QSPI0_MISO_IO1_MARK,
- QSPI0_IO2_MARK, QSPI0_IO3_MARK,
-};
-/* - QSPI1 ------------------------------------------------------------------ */
-static const unsigned int qspi1_ctrl_pins[] = {
- /* QSPI1_SPCLK, QSPI1_SSL */
- PIN_NUMBER('V', 3), PIN_NUMBER('V', 5),
-};
-static const unsigned int qspi1_ctrl_mux[] = {
- QSPI1_SPCLK_MARK, QSPI1_SSL_MARK,
-};
-static const unsigned int qspi1_data2_pins[] = {
- /* QSPI1_MOSI_IO0, QSPI1_MISO_IO1 */
- PIN_A_NUMBER('C', 7), PIN_A_NUMBER('E', 5),
-};
-static const unsigned int qspi1_data2_mux[] = {
- QSPI1_MOSI_IO0_MARK, QSPI1_MISO_IO1_MARK,
-};
-static const unsigned int qspi1_data4_pins[] = {
- /* QSPI1_MOSI_IO0, QSPI1_MISO_IO1, QSPI1_IO2, QSPI1_IO3 */
- PIN_A_NUMBER('C', 7), PIN_A_NUMBER('E', 5),
- PIN_A_NUMBER('E', 4), PIN_A_NUMBER('C', 3),
-};
-static const unsigned int qspi1_data4_mux[] = {
- QSPI1_MOSI_IO0_MARK, QSPI1_MISO_IO1_MARK,
- QSPI1_IO2_MARK, QSPI1_IO3_MARK,
-};
-
static const struct sh_pfc_pin_group pinmux_groups[] = {
- SH_PFC_PIN_GROUP(audio_clk_a_a),
- SH_PFC_PIN_GROUP(audio_clk_a_b),
- SH_PFC_PIN_GROUP(audio_clk_a_c),
- SH_PFC_PIN_GROUP(audio_clk_b_a),
- SH_PFC_PIN_GROUP(audio_clk_b_b),
- SH_PFC_PIN_GROUP(audio_clk_c_a),
- SH_PFC_PIN_GROUP(audio_clk_c_b),
- SH_PFC_PIN_GROUP(audio_clkout_a),
- SH_PFC_PIN_GROUP(audio_clkout_b),
- SH_PFC_PIN_GROUP(audio_clkout_c),
- SH_PFC_PIN_GROUP(audio_clkout_d),
- SH_PFC_PIN_GROUP(audio_clkout1_a),
- SH_PFC_PIN_GROUP(audio_clkout1_b),
- SH_PFC_PIN_GROUP(audio_clkout2_a),
- SH_PFC_PIN_GROUP(audio_clkout2_b),
- SH_PFC_PIN_GROUP(audio_clkout3_a),
- SH_PFC_PIN_GROUP(audio_clkout3_b),
- SH_PFC_PIN_GROUP(avb_link),
- SH_PFC_PIN_GROUP(avb_magic),
- SH_PFC_PIN_GROUP(avb_phy_int),
- SH_PFC_PIN_GROUP(avb_mdc),
- SH_PFC_PIN_GROUP(avb_mii),
- SH_PFC_PIN_GROUP(avb_avtp_pps),
- SH_PFC_PIN_GROUP(avb_avtp_match_a),
- SH_PFC_PIN_GROUP(avb_avtp_capture_a),
- SH_PFC_PIN_GROUP(avb_avtp_match_b),
- SH_PFC_PIN_GROUP(avb_avtp_capture_b),
- SH_PFC_PIN_GROUP(can0_data_a),
- SH_PFC_PIN_GROUP(can0_data_b),
- SH_PFC_PIN_GROUP(can1_data),
- SH_PFC_PIN_GROUP(can_clk),
- SH_PFC_PIN_GROUP(canfd0_data_a),
- SH_PFC_PIN_GROUP(canfd0_data_b),
- SH_PFC_PIN_GROUP(canfd1_data),
- SH_PFC_PIN_GROUP(drif0_ctrl_a),
- SH_PFC_PIN_GROUP(drif0_data0_a),
- SH_PFC_PIN_GROUP(drif0_data1_a),
- SH_PFC_PIN_GROUP(drif0_ctrl_b),
- SH_PFC_PIN_GROUP(drif0_data0_b),
- SH_PFC_PIN_GROUP(drif0_data1_b),
- SH_PFC_PIN_GROUP(drif0_ctrl_c),
- SH_PFC_PIN_GROUP(drif0_data0_c),
- SH_PFC_PIN_GROUP(drif0_data1_c),
- SH_PFC_PIN_GROUP(drif1_ctrl_a),
- SH_PFC_PIN_GROUP(drif1_data0_a),
- SH_PFC_PIN_GROUP(drif1_data1_a),
- SH_PFC_PIN_GROUP(drif1_ctrl_b),
- SH_PFC_PIN_GROUP(drif1_data0_b),
- SH_PFC_PIN_GROUP(drif1_data1_b),
- SH_PFC_PIN_GROUP(drif1_ctrl_c),
- SH_PFC_PIN_GROUP(drif1_data0_c),
- SH_PFC_PIN_GROUP(drif1_data1_c),
- SH_PFC_PIN_GROUP(drif2_ctrl_a),
- SH_PFC_PIN_GROUP(drif2_data0_a),
- SH_PFC_PIN_GROUP(drif2_data1_a),
- SH_PFC_PIN_GROUP(drif2_ctrl_b),
- SH_PFC_PIN_GROUP(drif2_data0_b),
- SH_PFC_PIN_GROUP(drif2_data1_b),
- SH_PFC_PIN_GROUP(drif3_ctrl_a),
- SH_PFC_PIN_GROUP(drif3_data0_a),
- SH_PFC_PIN_GROUP(drif3_data1_a),
- SH_PFC_PIN_GROUP(drif3_ctrl_b),
- SH_PFC_PIN_GROUP(drif3_data0_b),
- SH_PFC_PIN_GROUP(drif3_data1_b),
- SH_PFC_PIN_GROUP(du_rgb666),
- SH_PFC_PIN_GROUP(du_rgb888),
- SH_PFC_PIN_GROUP(du_clk_out_0),
- SH_PFC_PIN_GROUP(du_clk_out_1),
- SH_PFC_PIN_GROUP(du_sync),
- SH_PFC_PIN_GROUP(du_oddf),
- SH_PFC_PIN_GROUP(du_cde),
- SH_PFC_PIN_GROUP(du_disp),
- SH_PFC_PIN_GROUP(hscif0_data),
- SH_PFC_PIN_GROUP(hscif0_clk),
- SH_PFC_PIN_GROUP(hscif0_ctrl),
- SH_PFC_PIN_GROUP(hscif1_data_a),
- SH_PFC_PIN_GROUP(hscif1_clk_a),
- SH_PFC_PIN_GROUP(hscif1_ctrl_a),
- SH_PFC_PIN_GROUP(hscif1_data_b),
- SH_PFC_PIN_GROUP(hscif1_clk_b),
- SH_PFC_PIN_GROUP(hscif1_ctrl_b),
- SH_PFC_PIN_GROUP(hscif2_data_a),
- SH_PFC_PIN_GROUP(hscif2_clk_a),
- SH_PFC_PIN_GROUP(hscif2_ctrl_a),
- SH_PFC_PIN_GROUP(hscif2_data_b),
- SH_PFC_PIN_GROUP(hscif2_clk_b),
- SH_PFC_PIN_GROUP(hscif2_ctrl_b),
- SH_PFC_PIN_GROUP(hscif3_data_a),
- SH_PFC_PIN_GROUP(hscif3_clk),
- SH_PFC_PIN_GROUP(hscif3_ctrl),
- SH_PFC_PIN_GROUP(hscif3_data_b),
- SH_PFC_PIN_GROUP(hscif3_data_c),
- SH_PFC_PIN_GROUP(hscif3_data_d),
- SH_PFC_PIN_GROUP(hscif4_data_a),
- SH_PFC_PIN_GROUP(hscif4_clk),
- SH_PFC_PIN_GROUP(hscif4_ctrl),
- SH_PFC_PIN_GROUP(hscif4_data_b),
- SH_PFC_PIN_GROUP(i2c1_a),
- SH_PFC_PIN_GROUP(i2c1_b),
- SH_PFC_PIN_GROUP(i2c2_a),
- SH_PFC_PIN_GROUP(i2c2_b),
- SH_PFC_PIN_GROUP(i2c6_a),
- SH_PFC_PIN_GROUP(i2c6_b),
- SH_PFC_PIN_GROUP(i2c6_c),
- SH_PFC_PIN_GROUP(intc_ex_irq0),
- SH_PFC_PIN_GROUP(intc_ex_irq1),
- SH_PFC_PIN_GROUP(intc_ex_irq2),
- SH_PFC_PIN_GROUP(intc_ex_irq3),
- SH_PFC_PIN_GROUP(intc_ex_irq4),
- SH_PFC_PIN_GROUP(intc_ex_irq5),
- SH_PFC_PIN_GROUP(msiof0_clk),
- SH_PFC_PIN_GROUP(msiof0_sync),
- SH_PFC_PIN_GROUP(msiof0_ss1),
- SH_PFC_PIN_GROUP(msiof0_ss2),
- SH_PFC_PIN_GROUP(msiof0_txd),
- SH_PFC_PIN_GROUP(msiof0_rxd),
- SH_PFC_PIN_GROUP(msiof1_clk_a),
- SH_PFC_PIN_GROUP(msiof1_sync_a),
- SH_PFC_PIN_GROUP(msiof1_ss1_a),
- SH_PFC_PIN_GROUP(msiof1_ss2_a),
- SH_PFC_PIN_GROUP(msiof1_txd_a),
- SH_PFC_PIN_GROUP(msiof1_rxd_a),
- SH_PFC_PIN_GROUP(msiof1_clk_b),
- SH_PFC_PIN_GROUP(msiof1_sync_b),
- SH_PFC_PIN_GROUP(msiof1_ss1_b),
- SH_PFC_PIN_GROUP(msiof1_ss2_b),
- SH_PFC_PIN_GROUP(msiof1_txd_b),
- SH_PFC_PIN_GROUP(msiof1_rxd_b),
- SH_PFC_PIN_GROUP(msiof1_clk_c),
- SH_PFC_PIN_GROUP(msiof1_sync_c),
- SH_PFC_PIN_GROUP(msiof1_ss1_c),
- SH_PFC_PIN_GROUP(msiof1_ss2_c),
- SH_PFC_PIN_GROUP(msiof1_txd_c),
- SH_PFC_PIN_GROUP(msiof1_rxd_c),
- SH_PFC_PIN_GROUP(msiof1_clk_d),
- SH_PFC_PIN_GROUP(msiof1_sync_d),
- SH_PFC_PIN_GROUP(msiof1_ss1_d),
- SH_PFC_PIN_GROUP(msiof1_ss2_d),
- SH_PFC_PIN_GROUP(msiof1_txd_d),
- SH_PFC_PIN_GROUP(msiof1_rxd_d),
- SH_PFC_PIN_GROUP(msiof1_clk_e),
- SH_PFC_PIN_GROUP(msiof1_sync_e),
- SH_PFC_PIN_GROUP(msiof1_ss1_e),
- SH_PFC_PIN_GROUP(msiof1_ss2_e),
- SH_PFC_PIN_GROUP(msiof1_txd_e),
- SH_PFC_PIN_GROUP(msiof1_rxd_e),
- SH_PFC_PIN_GROUP(msiof1_clk_f),
- SH_PFC_PIN_GROUP(msiof1_sync_f),
- SH_PFC_PIN_GROUP(msiof1_ss1_f),
- SH_PFC_PIN_GROUP(msiof1_ss2_f),
- SH_PFC_PIN_GROUP(msiof1_txd_f),
- SH_PFC_PIN_GROUP(msiof1_rxd_f),
- SH_PFC_PIN_GROUP(msiof1_clk_g),
- SH_PFC_PIN_GROUP(msiof1_sync_g),
- SH_PFC_PIN_GROUP(msiof1_ss1_g),
- SH_PFC_PIN_GROUP(msiof1_ss2_g),
- SH_PFC_PIN_GROUP(msiof1_txd_g),
- SH_PFC_PIN_GROUP(msiof1_rxd_g),
- SH_PFC_PIN_GROUP(msiof2_clk_a),
- SH_PFC_PIN_GROUP(msiof2_sync_a),
- SH_PFC_PIN_GROUP(msiof2_ss1_a),
- SH_PFC_PIN_GROUP(msiof2_ss2_a),
- SH_PFC_PIN_GROUP(msiof2_txd_a),
- SH_PFC_PIN_GROUP(msiof2_rxd_a),
- SH_PFC_PIN_GROUP(msiof2_clk_b),
- SH_PFC_PIN_GROUP(msiof2_sync_b),
- SH_PFC_PIN_GROUP(msiof2_ss1_b),
- SH_PFC_PIN_GROUP(msiof2_ss2_b),
- SH_PFC_PIN_GROUP(msiof2_txd_b),
- SH_PFC_PIN_GROUP(msiof2_rxd_b),
- SH_PFC_PIN_GROUP(msiof2_clk_c),
- SH_PFC_PIN_GROUP(msiof2_sync_c),
- SH_PFC_PIN_GROUP(msiof2_ss1_c),
- SH_PFC_PIN_GROUP(msiof2_ss2_c),
- SH_PFC_PIN_GROUP(msiof2_txd_c),
- SH_PFC_PIN_GROUP(msiof2_rxd_c),
- SH_PFC_PIN_GROUP(msiof2_clk_d),
- SH_PFC_PIN_GROUP(msiof2_sync_d),
- SH_PFC_PIN_GROUP(msiof2_ss1_d),
- SH_PFC_PIN_GROUP(msiof2_ss2_d),
- SH_PFC_PIN_GROUP(msiof2_txd_d),
- SH_PFC_PIN_GROUP(msiof2_rxd_d),
- SH_PFC_PIN_GROUP(msiof3_clk_a),
- SH_PFC_PIN_GROUP(msiof3_sync_a),
- SH_PFC_PIN_GROUP(msiof3_ss1_a),
- SH_PFC_PIN_GROUP(msiof3_ss2_a),
- SH_PFC_PIN_GROUP(msiof3_txd_a),
- SH_PFC_PIN_GROUP(msiof3_rxd_a),
- SH_PFC_PIN_GROUP(msiof3_clk_b),
- SH_PFC_PIN_GROUP(msiof3_sync_b),
- SH_PFC_PIN_GROUP(msiof3_ss1_b),
- SH_PFC_PIN_GROUP(msiof3_ss2_b),
- SH_PFC_PIN_GROUP(msiof3_txd_b),
- SH_PFC_PIN_GROUP(msiof3_rxd_b),
- SH_PFC_PIN_GROUP(msiof3_clk_c),
- SH_PFC_PIN_GROUP(msiof3_sync_c),
- SH_PFC_PIN_GROUP(msiof3_txd_c),
- SH_PFC_PIN_GROUP(msiof3_rxd_c),
- SH_PFC_PIN_GROUP(msiof3_clk_d),
- SH_PFC_PIN_GROUP(msiof3_sync_d),
- SH_PFC_PIN_GROUP(msiof3_ss1_d),
- SH_PFC_PIN_GROUP(msiof3_txd_d),
- SH_PFC_PIN_GROUP(msiof3_rxd_d),
- SH_PFC_PIN_GROUP(pwm0),
- SH_PFC_PIN_GROUP(pwm1_a),
- SH_PFC_PIN_GROUP(pwm1_b),
- SH_PFC_PIN_GROUP(pwm2_a),
- SH_PFC_PIN_GROUP(pwm2_b),
- SH_PFC_PIN_GROUP(pwm3_a),
- SH_PFC_PIN_GROUP(pwm3_b),
- SH_PFC_PIN_GROUP(pwm4_a),
- SH_PFC_PIN_GROUP(pwm4_b),
- SH_PFC_PIN_GROUP(pwm5_a),
- SH_PFC_PIN_GROUP(pwm5_b),
- SH_PFC_PIN_GROUP(pwm6_a),
- SH_PFC_PIN_GROUP(pwm6_b),
- SH_PFC_PIN_GROUP(sata0_devslp_a),
- SH_PFC_PIN_GROUP(sata0_devslp_b),
SH_PFC_PIN_GROUP(scif0_data),
SH_PFC_PIN_GROUP(scif0_clk),
SH_PFC_PIN_GROUP(scif0_ctrl),
@@ -4015,387 +1813,12 @@
SH_PFC_PIN_GROUP(scif4_data_c),
SH_PFC_PIN_GROUP(scif4_clk_c),
SH_PFC_PIN_GROUP(scif4_ctrl_c),
- SH_PFC_PIN_GROUP(scif5_data),
- SH_PFC_PIN_GROUP(scif5_clk),
+ SH_PFC_PIN_GROUP(scif5_data_a),
+ SH_PFC_PIN_GROUP(scif5_clk_a),
+ SH_PFC_PIN_GROUP(scif5_data_b),
+ SH_PFC_PIN_GROUP(scif5_clk_b),
SH_PFC_PIN_GROUP(scif_clk_a),
SH_PFC_PIN_GROUP(scif_clk_b),
- SH_PFC_PIN_GROUP(sdhi0_data1),
- SH_PFC_PIN_GROUP(sdhi0_data4),
- SH_PFC_PIN_GROUP(sdhi0_ctrl),
- SH_PFC_PIN_GROUP(sdhi0_cd),
- SH_PFC_PIN_GROUP(sdhi0_wp),
- SH_PFC_PIN_GROUP(sdhi1_data1),
- SH_PFC_PIN_GROUP(sdhi1_data4),
- SH_PFC_PIN_GROUP(sdhi1_ctrl),
- SH_PFC_PIN_GROUP(sdhi1_cd),
- SH_PFC_PIN_GROUP(sdhi1_wp),
- SH_PFC_PIN_GROUP(sdhi2_data1),
- SH_PFC_PIN_GROUP(sdhi2_data4),
- SH_PFC_PIN_GROUP(sdhi2_data8),
- SH_PFC_PIN_GROUP(sdhi2_ctrl),
- SH_PFC_PIN_GROUP(sdhi2_cd_a),
- SH_PFC_PIN_GROUP(sdhi2_wp_a),
- SH_PFC_PIN_GROUP(sdhi2_cd_b),
- SH_PFC_PIN_GROUP(sdhi2_wp_b),
- SH_PFC_PIN_GROUP(sdhi2_ds),
- SH_PFC_PIN_GROUP(sdhi3_data1),
- SH_PFC_PIN_GROUP(sdhi3_data4),
- SH_PFC_PIN_GROUP(sdhi3_data8),
- SH_PFC_PIN_GROUP(sdhi3_ctrl),
- SH_PFC_PIN_GROUP(sdhi3_cd),
- SH_PFC_PIN_GROUP(sdhi3_wp),
- SH_PFC_PIN_GROUP(sdhi3_ds),
- SH_PFC_PIN_GROUP(ssi0_data),
- SH_PFC_PIN_GROUP(ssi01239_ctrl),
- SH_PFC_PIN_GROUP(ssi1_data_a),
- SH_PFC_PIN_GROUP(ssi1_data_b),
- SH_PFC_PIN_GROUP(ssi1_ctrl_a),
- SH_PFC_PIN_GROUP(ssi1_ctrl_b),
- SH_PFC_PIN_GROUP(ssi2_data_a),
- SH_PFC_PIN_GROUP(ssi2_data_b),
- SH_PFC_PIN_GROUP(ssi2_ctrl_a),
- SH_PFC_PIN_GROUP(ssi2_ctrl_b),
- SH_PFC_PIN_GROUP(ssi3_data),
- SH_PFC_PIN_GROUP(ssi34_ctrl),
- SH_PFC_PIN_GROUP(ssi4_data),
- SH_PFC_PIN_GROUP(ssi4_ctrl),
- SH_PFC_PIN_GROUP(ssi5_data),
- SH_PFC_PIN_GROUP(ssi5_ctrl),
- SH_PFC_PIN_GROUP(ssi6_data),
- SH_PFC_PIN_GROUP(ssi6_ctrl),
- SH_PFC_PIN_GROUP(ssi7_data),
- SH_PFC_PIN_GROUP(ssi78_ctrl),
- SH_PFC_PIN_GROUP(ssi8_data),
- SH_PFC_PIN_GROUP(ssi9_data_a),
- SH_PFC_PIN_GROUP(ssi9_data_b),
- SH_PFC_PIN_GROUP(ssi9_ctrl_a),
- SH_PFC_PIN_GROUP(ssi9_ctrl_b),
- SH_PFC_PIN_GROUP(usb0),
- SH_PFC_PIN_GROUP(usb1),
- SH_PFC_PIN_GROUP(usb2),
- SH_PFC_PIN_GROUP(qspi0_ctrl),
- SH_PFC_PIN_GROUP(qspi0_data2),
- SH_PFC_PIN_GROUP(qspi0_data4),
- SH_PFC_PIN_GROUP(qspi1_ctrl),
- SH_PFC_PIN_GROUP(qspi1_data2),
- SH_PFC_PIN_GROUP(qspi1_data4),
-};
-
-static const char * const audio_clk_groups[] = {
- "audio_clk_a_a",
- "audio_clk_a_b",
- "audio_clk_a_c",
- "audio_clk_b_a",
- "audio_clk_b_b",
- "audio_clk_c_a",
- "audio_clk_c_b",
- "audio_clkout_a",
- "audio_clkout_b",
- "audio_clkout_c",
- "audio_clkout_d",
- "audio_clkout1_a",
- "audio_clkout1_b",
- "audio_clkout2_a",
- "audio_clkout2_b",
- "audio_clkout3_a",
- "audio_clkout3_b",
-};
-
-static const char * const avb_groups[] = {
- "avb_link",
- "avb_magic",
- "avb_phy_int",
- "avb_mdc",
- "avb_mii",
- "avb_avtp_pps",
- "avb_avtp_match_a",
- "avb_avtp_capture_a",
- "avb_avtp_match_b",
- "avb_avtp_capture_b",
-};
-
-static const char * const can0_groups[] = {
- "can0_data_a",
- "can0_data_b",
-};
-
-static const char * const can1_groups[] = {
- "can1_data",
-};
-
-static const char * const can_clk_groups[] = {
- "can_clk",
-};
-
-static const char * const canfd0_groups[] = {
- "canfd0_data_a",
- "canfd0_data_b",
-};
-
-static const char * const canfd1_groups[] = {
- "canfd1_data",
-};
-
-static const char * const drif0_groups[] = {
- "drif0_ctrl_a",
- "drif0_data0_a",
- "drif0_data1_a",
- "drif0_ctrl_b",
- "drif0_data0_b",
- "drif0_data1_b",
- "drif0_ctrl_c",
- "drif0_data0_c",
- "drif0_data1_c",
-};
-
-static const char * const drif1_groups[] = {
- "drif1_ctrl_a",
- "drif1_data0_a",
- "drif1_data1_a",
- "drif1_ctrl_b",
- "drif1_data0_b",
- "drif1_data1_b",
- "drif1_ctrl_c",
- "drif1_data0_c",
- "drif1_data1_c",
-};
-
-static const char * const drif2_groups[] = {
- "drif2_ctrl_a",
- "drif2_data0_a",
- "drif2_data1_a",
- "drif2_ctrl_b",
- "drif2_data0_b",
- "drif2_data1_b",
-};
-
-static const char * const drif3_groups[] = {
- "drif3_ctrl_a",
- "drif3_data0_a",
- "drif3_data1_a",
- "drif3_ctrl_b",
- "drif3_data0_b",
- "drif3_data1_b",
-};
-
-static const char * const du_groups[] = {
- "du_rgb666",
- "du_rgb888",
- "du_clk_out_0",
- "du_clk_out_1",
- "du_sync",
- "du_oddf",
- "du_cde",
- "du_disp",
-};
-
-static const char * const hscif0_groups[] = {
- "hscif0_data",
- "hscif0_clk",
- "hscif0_ctrl",
-};
-
-static const char * const hscif1_groups[] = {
- "hscif1_data_a",
- "hscif1_clk_a",
- "hscif1_ctrl_a",
- "hscif1_data_b",
- "hscif1_clk_b",
- "hscif1_ctrl_b",
-};
-
-static const char * const hscif2_groups[] = {
- "hscif2_data_a",
- "hscif2_clk_a",
- "hscif2_ctrl_a",
- "hscif2_data_b",
- "hscif2_clk_b",
- "hscif2_ctrl_b",
-};
-
-static const char * const hscif3_groups[] = {
- "hscif3_data_a",
- "hscif3_clk",
- "hscif3_ctrl",
- "hscif3_data_b",
- "hscif3_data_c",
- "hscif3_data_d",
-};
-
-static const char * const hscif4_groups[] = {
- "hscif4_data_a",
- "hscif4_clk",
- "hscif4_ctrl",
- "hscif4_data_b",
-};
-
-static const char * const i2c1_groups[] = {
- "i2c1_a",
- "i2c1_b",
-};
-
-static const char * const i2c2_groups[] = {
- "i2c2_a",
- "i2c2_b",
-};
-
-static const char * const i2c6_groups[] = {
- "i2c6_a",
- "i2c6_b",
- "i2c6_c",
-};
-
-static const char * const intc_ex_groups[] = {
- "intc_ex_irq0",
- "intc_ex_irq1",
- "intc_ex_irq2",
- "intc_ex_irq3",
- "intc_ex_irq4",
- "intc_ex_irq5",
-};
-
-static const char * const msiof0_groups[] = {
- "msiof0_clk",
- "msiof0_sync",
- "msiof0_ss1",
- "msiof0_ss2",
- "msiof0_txd",
- "msiof0_rxd",
-};
-
-static const char * const msiof1_groups[] = {
- "msiof1_clk_a",
- "msiof1_sync_a",
- "msiof1_ss1_a",
- "msiof1_ss2_a",
- "msiof1_txd_a",
- "msiof1_rxd_a",
- "msiof1_clk_b",
- "msiof1_sync_b",
- "msiof1_ss1_b",
- "msiof1_ss2_b",
- "msiof1_txd_b",
- "msiof1_rxd_b",
- "msiof1_clk_c",
- "msiof1_sync_c",
- "msiof1_ss1_c",
- "msiof1_ss2_c",
- "msiof1_txd_c",
- "msiof1_rxd_c",
- "msiof1_clk_d",
- "msiof1_sync_d",
- "msiof1_ss1_d",
- "msiof1_ss2_d",
- "msiof1_txd_d",
- "msiof1_rxd_d",
- "msiof1_clk_e",
- "msiof1_sync_e",
- "msiof1_ss1_e",
- "msiof1_ss2_e",
- "msiof1_txd_e",
- "msiof1_rxd_e",
- "msiof1_clk_f",
- "msiof1_sync_f",
- "msiof1_ss1_f",
- "msiof1_ss2_f",
- "msiof1_txd_f",
- "msiof1_rxd_f",
- "msiof1_clk_g",
- "msiof1_sync_g",
- "msiof1_ss1_g",
- "msiof1_ss2_g",
- "msiof1_txd_g",
- "msiof1_rxd_g",
-};
-
-static const char * const msiof2_groups[] = {
- "msiof2_clk_a",
- "msiof2_sync_a",
- "msiof2_ss1_a",
- "msiof2_ss2_a",
- "msiof2_txd_a",
- "msiof2_rxd_a",
- "msiof2_clk_b",
- "msiof2_sync_b",
- "msiof2_ss1_b",
- "msiof2_ss2_b",
- "msiof2_txd_b",
- "msiof2_rxd_b",
- "msiof2_clk_c",
- "msiof2_sync_c",
- "msiof2_ss1_c",
- "msiof2_ss2_c",
- "msiof2_txd_c",
- "msiof2_rxd_c",
- "msiof2_clk_d",
- "msiof2_sync_d",
- "msiof2_ss1_d",
- "msiof2_ss2_d",
- "msiof2_txd_d",
- "msiof2_rxd_d",
-};
-
-static const char * const msiof3_groups[] = {
- "msiof3_clk_a",
- "msiof3_sync_a",
- "msiof3_ss1_a",
- "msiof3_ss2_a",
- "msiof3_txd_a",
- "msiof3_rxd_a",
- "msiof3_clk_b",
- "msiof3_sync_b",
- "msiof3_ss1_b",
- "msiof3_ss2_b",
- "msiof3_txd_b",
- "msiof3_rxd_b",
- "msiof3_clk_c",
- "msiof3_sync_c",
- "msiof3_txd_c",
- "msiof3_rxd_c",
- "msiof3_clk_d",
- "msiof3_sync_d",
- "msiof3_ss1_d",
- "msiof3_txd_d",
- "msiof3_rxd_d",
-};
-
-static const char * const pwm0_groups[] = {
- "pwm0",
-};
-
-static const char * const pwm1_groups[] = {
- "pwm1_a",
- "pwm1_b",
-};
-
-static const char * const pwm2_groups[] = {
- "pwm2_a",
- "pwm2_b",
-};
-
-static const char * const pwm3_groups[] = {
- "pwm3_a",
- "pwm3_b",
-};
-
-static const char * const pwm4_groups[] = {
- "pwm4_a",
- "pwm4_b",
-};
-
-static const char * const pwm5_groups[] = {
- "pwm5_a",
- "pwm5_b",
-};
-
-static const char * const pwm6_groups[] = {
- "pwm6_a",
- "pwm6_b",
-};
-
-static const char * const sata0_groups[] = {
- "sata0_devslp_a",
- "sata0_devslp_b",
};
static const char * const scif0_groups[] = {
@@ -4437,8 +1860,10 @@
};
static const char * const scif5_groups[] = {
- "scif5_data",
- "scif5_clk",
+ "scif5_data_a",
+ "scif5_clk_a",
+ "scif5_data_b",
+ "scif5_clk_b",
};
static const char * const scif_clk_groups[] = {
@@ -4446,130 +1871,7 @@
"scif_clk_b",
};
-static const char * const sdhi0_groups[] = {
- "sdhi0_data1",
- "sdhi0_data4",
- "sdhi0_ctrl",
- "sdhi0_cd",
- "sdhi0_wp",
-};
-
-static const char * const sdhi1_groups[] = {
- "sdhi1_data1",
- "sdhi1_data4",
- "sdhi1_ctrl",
- "sdhi1_cd",
- "sdhi1_wp",
-};
-
-static const char * const sdhi2_groups[] = {
- "sdhi2_data1",
- "sdhi2_data4",
- "sdhi2_data8",
- "sdhi2_ctrl",
- "sdhi2_cd_a",
- "sdhi2_wp_a",
- "sdhi2_cd_b",
- "sdhi2_wp_b",
- "sdhi2_ds",
-};
-
-static const char * const sdhi3_groups[] = {
- "sdhi3_data1",
- "sdhi3_data4",
- "sdhi3_data8",
- "sdhi3_ctrl",
- "sdhi3_cd",
- "sdhi3_wp",
- "sdhi3_ds",
-};
-
-static const char * const ssi_groups[] = {
- "ssi0_data",
- "ssi01239_ctrl",
- "ssi1_data_a",
- "ssi1_data_b",
- "ssi1_ctrl_a",
- "ssi1_ctrl_b",
- "ssi2_data_a",
- "ssi2_data_b",
- "ssi2_ctrl_a",
- "ssi2_ctrl_b",
- "ssi3_data",
- "ssi34_ctrl",
- "ssi4_data",
- "ssi4_ctrl",
- "ssi5_data",
- "ssi5_ctrl",
- "ssi6_data",
- "ssi6_ctrl",
- "ssi7_data",
- "ssi78_ctrl",
- "ssi8_data",
- "ssi9_data_a",
- "ssi9_data_b",
- "ssi9_ctrl_a",
- "ssi9_ctrl_b",
-};
-
-static const char * const usb0_groups[] = {
- "usb0",
-};
-
-static const char * const usb1_groups[] = {
- "usb1",
-};
-
-static const char * const usb2_groups[] = {
- "usb2",
-};
-
-static const char * const qspi0_groups[] = {
- "qspi0_ctrl",
- "qspi0_data2",
- "qspi0_data4",
-};
-
-static const char * const qspi1_groups[] = {
- "qspi1_ctrl",
- "qspi1_data2",
- "qspi1_data4",
-};
-
static const struct sh_pfc_function pinmux_functions[] = {
- SH_PFC_FUNCTION(audio_clk),
- SH_PFC_FUNCTION(avb),
- SH_PFC_FUNCTION(can0),
- SH_PFC_FUNCTION(can1),
- SH_PFC_FUNCTION(can_clk),
- SH_PFC_FUNCTION(canfd0),
- SH_PFC_FUNCTION(canfd1),
- SH_PFC_FUNCTION(drif0),
- SH_PFC_FUNCTION(drif1),
- SH_PFC_FUNCTION(drif2),
- SH_PFC_FUNCTION(drif3),
- SH_PFC_FUNCTION(du),
- SH_PFC_FUNCTION(hscif0),
- SH_PFC_FUNCTION(hscif1),
- SH_PFC_FUNCTION(hscif2),
- SH_PFC_FUNCTION(hscif3),
- SH_PFC_FUNCTION(hscif4),
- SH_PFC_FUNCTION(i2c1),
- SH_PFC_FUNCTION(i2c2),
- SH_PFC_FUNCTION(i2c6),
- SH_PFC_FUNCTION(intc_ex),
- SH_PFC_FUNCTION(msiof0),
- SH_PFC_FUNCTION(msiof1),
- SH_PFC_FUNCTION(msiof2),
- SH_PFC_FUNCTION(msiof3),
- SH_PFC_FUNCTION(pwm0),
- SH_PFC_FUNCTION(pwm1),
- SH_PFC_FUNCTION(pwm2),
- SH_PFC_FUNCTION(pwm3),
- SH_PFC_FUNCTION(pwm4),
- SH_PFC_FUNCTION(pwm5),
- SH_PFC_FUNCTION(pwm6),
- SH_PFC_FUNCTION(sata0),
SH_PFC_FUNCTION(scif0),
SH_PFC_FUNCTION(scif1),
SH_PFC_FUNCTION(scif2),
@@ -4577,16 +1879,6 @@
SH_PFC_FUNCTION(scif4),
SH_PFC_FUNCTION(scif5),
SH_PFC_FUNCTION(scif_clk),
- SH_PFC_FUNCTION(sdhi0),
- SH_PFC_FUNCTION(sdhi1),
- SH_PFC_FUNCTION(sdhi2),
- SH_PFC_FUNCTION(sdhi3),
- SH_PFC_FUNCTION(ssi),
- SH_PFC_FUNCTION(usb0),
- SH_PFC_FUNCTION(usb1),
- SH_PFC_FUNCTION(usb2),
- SH_PFC_FUNCTION(qspi0),
- SH_PFC_FUNCTION(qspi1),
};
static const struct pinmux_cfg_reg pinmux_config_regs[] = {
@@ -5040,46 +2332,54 @@
IP16_3_0 }
},
{ PINMUX_CFG_REG("IPSR17", 0xe6060244, 32, 4) {
- /* IP17_31_28 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
- /* IP17_27_24 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
- /* IP17_23_20 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
- /* IP17_19_16 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
- /* IP17_15_12 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
- /* IP17_11_8 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ IP17_31_28
+ IP17_27_24
+ IP17_23_20
+ IP17_19_16
+ IP17_15_12
+ IP17_11_8
IP17_7_4
IP17_3_0 }
},
+ { PINMUX_CFG_REG("IPSR18", 0xe6060248, 32, 4) {
+ /* IP18_31_28 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ /* IP18_27_24 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ /* IP18_23_20 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ /* IP18_19_16 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ /* IP18_15_12 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ /* IP18_11_8 */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+ IP18_7_4
+ IP18_3_0 }
+ },
#undef F_
#undef FM
#define F_(x, y) x,
#define FM(x) FN_##x,
{ PINMUX_CFG_REG_VAR("MOD_SEL0", 0xe6060500, 32,
- 1, 2, 2, 3, 1, 1, 2, 1, 1, 1,
- 2, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 2, 1) {
- 0, 0, /* RESERVED 31 */
- MOD_SEL0_30_29
+ 3, 2, 3, 1, 1, 1, 1, 1, 2, 1,
+ 1, 2, 1, 1, 1, 2, 2, 1, 2, 3) {
+ MOD_SEL0_31_30_29
MOD_SEL0_28_27
MOD_SEL0_26_25_24
MOD_SEL0_23
MOD_SEL0_22
- MOD_SEL0_21_20
+ MOD_SEL0_21
+ MOD_SEL0_20
MOD_SEL0_19
- MOD_SEL0_18
- MOD_SEL0_17
- MOD_SEL0_16_15
- MOD_SEL0_14
- MOD_SEL0_13
+ MOD_SEL0_18_17
+ MOD_SEL0_16
+ 0, 0, /* RESERVED 15 */
+ MOD_SEL0_14_13
MOD_SEL0_12
MOD_SEL0_11
MOD_SEL0_10
- MOD_SEL0_9
- MOD_SEL0_8
+ MOD_SEL0_9_8
MOD_SEL0_7_6
- MOD_SEL0_5_4
- MOD_SEL0_3
- MOD_SEL0_2_1
- 0, 0, /* RESERVED 0 */ }
+ MOD_SEL0_5
+ MOD_SEL0_4_3
+ /* RESERVED 2, 1, 0 */
+ 0, 0, 0, 0, 0, 0, 0, 0 }
},
{ PINMUX_CFG_REG_VAR("MOD_SEL1", 0xe6060504, 32,
2, 3, 1, 2, 3, 1, 1, 2, 1,
@@ -5109,22 +2409,22 @@
MOD_SEL1_0 }
},
{ PINMUX_CFG_REG_VAR("MOD_SEL2", 0xe6060508, 32,
- 1, 1, 1, 1, 4, 4, 4,
- 4, 4, 4, 1, 2, 1) {
+ 1, 1, 1, 2, 1, 3, 1, 1, 1, 1, 1, 1, 1,
+ 4, 4, 4, 3, 1) {
MOD_SEL2_31
MOD_SEL2_30
MOD_SEL2_29
- /* RESERVED 28 */
+ MOD_SEL2_28_27
+ MOD_SEL2_26
+ MOD_SEL2_25_24_23
+ MOD_SEL2_22
+ MOD_SEL2_21
+ MOD_SEL2_20
+ MOD_SEL2_19
+ MOD_SEL2_18
+ MOD_SEL2_17
+ /* RESERVED 16 */
0, 0,
- /* RESERVED 27, 26, 25, 24 */
- 0, 0, 0, 0, 0, 0, 0, 0,
- 0, 0, 0, 0, 0, 0, 0, 0,
- /* RESERVED 23, 22, 21, 20 */
- 0, 0, 0, 0, 0, 0, 0, 0,
- 0, 0, 0, 0, 0, 0, 0, 0,
- /* RESERVED 19, 18, 17, 16 */
- 0, 0, 0, 0, 0, 0, 0, 0,
- 0, 0, 0, 0, 0, 0, 0, 0,
/* RESERVED 15, 14, 13, 12 */
0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0,
@@ -5134,10 +2434,8 @@
/* RESERVED 7, 6, 5, 4 */
0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0,
- /* RESERVED 3 */
- 0, 0,
- /* RESERVED 2, 1 */
- 0, 0, 0, 0,
+ /* RESERVED 3, 2, 1 */
+ 0, 0, 0, 0, 0, 0, 0, 0,
MOD_SEL2_0 }
},
{ },
@@ -5386,8 +2684,8 @@
{ RCAR_GP_PIN(6, 27), 20, 3 }, /* USB1_OVC */
{ RCAR_GP_PIN(6, 28), 16, 3 }, /* USB30_PWEN */
{ RCAR_GP_PIN(6, 29), 12, 3 }, /* USB30_OVC */
- { RCAR_GP_PIN(6, 30), 8, 3 }, /* USB31_PWEN */
- { RCAR_GP_PIN(6, 31), 4, 3 }, /* USB31_OVC */
+ { RCAR_GP_PIN(6, 30), 8, 3 }, /* USB3_PWEN */
+ { RCAR_GP_PIN(6, 31), 4, 3 }, /* USB3_OVC */
} },
{ },
};
@@ -5617,8 +2915,8 @@
{ RCAR_GP_PIN(5, 21), PU5, 1 }, /* MSIOF0_SS2 */
{ RCAR_GP_PIN(5, 20), PU5, 0 }, /* MSIOF0_TXD */
- { RCAR_GP_PIN(6, 31), PU6, 6 }, /* USB31_OVC */
- { RCAR_GP_PIN(6, 30), PU6, 5 }, /* USB31_PWEN */
+ { RCAR_GP_PIN(6, 31), PU6, 6 }, /* USB3_OVC */
+ { RCAR_GP_PIN(6, 30), PU6, 5 }, /* USB3_PWEN */
{ RCAR_GP_PIN(6, 29), PU6, 4 }, /* USB30_OVC */
{ RCAR_GP_PIN(6, 28), PU6, 3 }, /* USB30_PWEN */
{ RCAR_GP_PIN(6, 27), PU6, 2 }, /* USB1_OVC */
@@ -5675,14 +2973,28 @@
sh_pfc_write_reg(pfc, PUEN + reg, 32, enable);
}
+static const struct soc_device_attribute r8a7795es1[] = {
+ { .soc_id = "r8a7795", .revision = "ES1.*" },
+ { /* sentinel */ }
+};
+
+static int r8a7795_pinmux_init(struct sh_pfc *pfc)
+{
+ if (soc_device_match(r8a7795es1))
+ pfc->info = &r8a7795es1_pinmux_info;
+
+ return 0;
+}
+
static const struct sh_pfc_soc_operations r8a7795_pinmux_ops = {
+ .init = r8a7795_pinmux_init,
.pin_to_pocctrl = r8a7795_pin_to_pocctrl,
.get_bias = r8a7795_pinmux_get_bias,
.set_bias = r8a7795_pinmux_set_bias,
};
const struct sh_pfc_soc_info r8a7795_pinmux_info = {
- .name = "r8a77950_pfc",
+ .name = "r8a77951_pfc",
.ops = &r8a7795_pinmux_ops,
.unlock_reg = 0xe6060000, /* PMMR */
diff --git a/drivers/pinctrl/sh-pfc/sh_pfc.h b/drivers/pinctrl/sh-pfc/sh_pfc.h
index e42cc7a..f31eb6c 100644
--- a/drivers/pinctrl/sh-pfc/sh_pfc.h
+++ b/drivers/pinctrl/sh-pfc/sh_pfc.h
@@ -267,6 +267,7 @@
extern const struct sh_pfc_soc_info r8a7793_pinmux_info;
extern const struct sh_pfc_soc_info r8a7794_pinmux_info;
extern const struct sh_pfc_soc_info r8a7795_pinmux_info;
+extern const struct sh_pfc_soc_info r8a7795es1_pinmux_info;
extern const struct sh_pfc_soc_info r8a7796_pinmux_info;
extern const struct sh_pfc_soc_info sh7203_pinmux_info;
extern const struct sh_pfc_soc_info sh7264_pinmux_info;
diff --git a/drivers/pinctrl/sirf/pinctrl-atlas7.c b/drivers/pinctrl/sirf/pinctrl-atlas7.c
index 600d642..1efa315 100644
--- a/drivers/pinctrl/sirf/pinctrl-atlas7.c
+++ b/drivers/pinctrl/sirf/pinctrl-atlas7.c
@@ -352,7 +352,7 @@
void __iomem *reg;
struct clk *clk;
int nbank;
- spinlock_t lock;
+ raw_spinlock_t lock;
struct gpio_chip chip;
struct atlas7_gpio_bank banks[0];
};
@@ -5650,13 +5650,13 @@
pin_in_bank = d->hwirq - bank->gpio_offset;
ctrl_reg = ATLAS7_GPIO_CTRL(bank, pin_in_bank);
- spin_lock_irqsave(&a7gc->lock, flags);
+ raw_spin_lock_irqsave(&a7gc->lock, flags);
val = readl(ctrl_reg);
/* clear interrupt status */
writel(val, ctrl_reg);
- spin_unlock_irqrestore(&a7gc->lock, flags);
+ raw_spin_unlock_irqrestore(&a7gc->lock, flags);
}
static void __atlas7_gpio_irq_mask(struct atlas7_gpio_chip *a7gc, int idx)
@@ -5681,11 +5681,11 @@
struct atlas7_gpio_chip *a7gc = gpiochip_get_data(gc);
unsigned long flags;
- spin_lock_irqsave(&a7gc->lock, flags);
+ raw_spin_lock_irqsave(&a7gc->lock, flags);
__atlas7_gpio_irq_mask(a7gc, d->hwirq);
- spin_unlock_irqrestore(&a7gc->lock, flags);
+ raw_spin_unlock_irqrestore(&a7gc->lock, flags);
}
static void atlas7_gpio_irq_unmask(struct irq_data *d)
@@ -5701,14 +5701,14 @@
pin_in_bank = d->hwirq - bank->gpio_offset;
ctrl_reg = ATLAS7_GPIO_CTRL(bank, pin_in_bank);
- spin_lock_irqsave(&a7gc->lock, flags);
+ raw_spin_lock_irqsave(&a7gc->lock, flags);
val = readl(ctrl_reg);
val &= ~ATLAS7_GPIO_CTL_INTR_STATUS_MASK;
val |= ATLAS7_GPIO_CTL_INTR_EN_MASK;
writel(val, ctrl_reg);
- spin_unlock_irqrestore(&a7gc->lock, flags);
+ raw_spin_unlock_irqrestore(&a7gc->lock, flags);
}
static int atlas7_gpio_irq_type(struct irq_data *d,
@@ -5725,7 +5725,7 @@
pin_in_bank = d->hwirq - bank->gpio_offset;
ctrl_reg = ATLAS7_GPIO_CTRL(bank, pin_in_bank);
- spin_lock_irqsave(&a7gc->lock, flags);
+ raw_spin_lock_irqsave(&a7gc->lock, flags);
val = readl(ctrl_reg);
val &= ~(ATLAS7_GPIO_CTL_INTR_STATUS_MASK |
@@ -5768,7 +5768,7 @@
writel(val, ctrl_reg);
- spin_unlock_irqrestore(&a7gc->lock, flags);
+ raw_spin_unlock_irqrestore(&a7gc->lock, flags);
return 0;
}
@@ -5863,7 +5863,7 @@
if (pinctrl_request_gpio(chip->base + gpio))
return -ENODEV;
- spin_lock_irqsave(&a7gc->lock, flags);
+ raw_spin_lock_irqsave(&a7gc->lock, flags);
/*
* default status:
@@ -5872,7 +5872,7 @@
__atlas7_gpio_set_input(a7gc, gpio);
__atlas7_gpio_irq_mask(a7gc, gpio);
- spin_unlock_irqrestore(&a7gc->lock, flags);
+ raw_spin_unlock_irqrestore(&a7gc->lock, flags);
return 0;
}
@@ -5883,12 +5883,12 @@
struct atlas7_gpio_chip *a7gc = gpiochip_get_data(chip);
unsigned long flags;
- spin_lock_irqsave(&a7gc->lock, flags);
+ raw_spin_lock_irqsave(&a7gc->lock, flags);
__atlas7_gpio_irq_mask(a7gc, gpio);
__atlas7_gpio_set_input(a7gc, gpio);
- spin_unlock_irqrestore(&a7gc->lock, flags);
+ raw_spin_unlock_irqrestore(&a7gc->lock, flags);
pinctrl_free_gpio(chip->base + gpio);
}
@@ -5899,11 +5899,11 @@
struct atlas7_gpio_chip *a7gc = gpiochip_get_data(chip);
unsigned long flags;
- spin_lock_irqsave(&a7gc->lock, flags);
+ raw_spin_lock_irqsave(&a7gc->lock, flags);
__atlas7_gpio_set_input(a7gc, gpio);
- spin_unlock_irqrestore(&a7gc->lock, flags);
+ raw_spin_unlock_irqrestore(&a7gc->lock, flags);
return 0;
}
@@ -5936,11 +5936,11 @@
struct atlas7_gpio_chip *a7gc = gpiochip_get_data(chip);
unsigned long flags;
- spin_lock_irqsave(&a7gc->lock, flags);
+ raw_spin_lock_irqsave(&a7gc->lock, flags);
__atlas7_gpio_set_output(a7gc, gpio, value);
- spin_unlock_irqrestore(&a7gc->lock, flags);
+ raw_spin_unlock_irqrestore(&a7gc->lock, flags);
return 0;
}
@@ -5956,11 +5956,11 @@
bank = atlas7_gpio_to_bank(a7gc, gpio);
pin_in_bank = gpio - bank->gpio_offset;
- spin_lock_irqsave(&a7gc->lock, flags);
+ raw_spin_lock_irqsave(&a7gc->lock, flags);
val = readl(ATLAS7_GPIO_CTRL(bank, pin_in_bank));
- spin_unlock_irqrestore(&a7gc->lock, flags);
+ raw_spin_unlock_irqrestore(&a7gc->lock, flags);
return !!(val & ATLAS7_GPIO_CTL_DATAIN_MASK);
}
@@ -5978,7 +5978,7 @@
pin_in_bank = gpio - bank->gpio_offset;
ctrl_reg = ATLAS7_GPIO_CTRL(bank, pin_in_bank);
- spin_lock_irqsave(&a7gc->lock, flags);
+ raw_spin_lock_irqsave(&a7gc->lock, flags);
ctrl = readl(ctrl_reg);
if (value)
@@ -5987,7 +5987,7 @@
ctrl &= ~ATLAS7_GPIO_CTL_DATAOUT_MASK;
writel(ctrl, ctrl_reg);
- spin_unlock_irqrestore(&a7gc->lock, flags);
+ raw_spin_unlock_irqrestore(&a7gc->lock, flags);
}
static const struct of_device_id atlas7_gpio_ids[] = {
@@ -6036,7 +6036,7 @@
}
a7gc->nbank = nbank;
- spin_lock_init(&a7gc->lock);
+ raw_spin_lock_init(&a7gc->lock);
/* Setup GPIO Chip */
chip = &a7gc->chip;
diff --git a/drivers/pinctrl/stm32/Kconfig b/drivers/pinctrl/stm32/Kconfig
index f5ccabd..3b8026f 100644
--- a/drivers/pinctrl/stm32/Kconfig
+++ b/drivers/pinctrl/stm32/Kconfig
@@ -14,6 +14,12 @@
default MACH_STM32F429
select PINCTRL_STM32
+config PINCTRL_STM32F469
+ bool "STMicroelectronics STM32F469 pin control" if COMPILE_TEST && !MACH_STM32F469
+ depends on OF && IRQ_DOMAIN_HIERARCHY
+ default MACH_STM32F469
+ select PINCTRL_STM32
+
config PINCTRL_STM32F746
bool "STMicroelectronics STM32F746 pin control" if COMPILE_TEST && !MACH_STM32F746
depends on OF && IRQ_DOMAIN_HIERARCHY
diff --git a/drivers/pinctrl/stm32/Makefile b/drivers/pinctrl/stm32/Makefile
index cb31b4d..5f379f5 100644
--- a/drivers/pinctrl/stm32/Makefile
+++ b/drivers/pinctrl/stm32/Makefile
@@ -3,5 +3,6 @@
# SoC Drivers
obj-$(CONFIG_PINCTRL_STM32F429) += pinctrl-stm32f429.o
+obj-$(CONFIG_PINCTRL_STM32F469) += pinctrl-stm32f469.o
obj-$(CONFIG_PINCTRL_STM32F746) += pinctrl-stm32f746.o
obj-$(CONFIG_PINCTRL_STM32H743) += pinctrl-stm32h743.o
diff --git a/drivers/pinctrl/stm32/pinctrl-stm32.c b/drivers/pinctrl/stm32/pinctrl-stm32.c
index abc405b..d3c5f5d 100644
--- a/drivers/pinctrl/stm32/pinctrl-stm32.c
+++ b/drivers/pinctrl/stm32/pinctrl-stm32.c
@@ -71,6 +71,7 @@
struct pinctrl_gpio_range range;
struct fwnode_handle *fwnode;
struct irq_domain *domain;
+ u32 bank_nr;
};
struct stm32_pinctrl {
@@ -138,6 +139,17 @@
static int stm32_gpio_request(struct gpio_chip *chip, unsigned offset)
{
+ struct stm32_gpio_bank *bank = gpiochip_get_data(chip);
+ struct stm32_pinctrl *pctl = dev_get_drvdata(bank->gpio_chip.parent);
+ struct pinctrl_gpio_range *range;
+ int pin = offset + (bank->bank_nr * STM32_GPIO_PINS_PER_BANK);
+
+ range = pinctrl_find_gpio_range_from_pin_nolock(pctl->pctl_dev, pin);
+ if (!range) {
+ dev_err(pctl->dev, "pin %d not in range.\n", pin);
+ return -EINVAL;
+ }
+
return pinctrl_request_gpio(chip->base + offset);
}
@@ -235,7 +247,7 @@
struct stm32_gpio_bank *bank = d->host_data;
struct stm32_pinctrl *pctl = dev_get_drvdata(bank->gpio_chip.parent);
- regmap_field_write(pctl->irqmux[irq_data->hwirq], bank->range.id);
+ regmap_field_write(pctl->irqmux[irq_data->hwirq], bank->bank_nr);
gpiochip_lock_as_irq(&bank->gpio_chip, irq_data->hwirq);
}
@@ -589,7 +601,7 @@
}
range = pinctrl_find_gpio_range_from_pin(pctldev, g->pin);
- bank = gpio_range_to_bank(range);
+ bank = gpiochip_get_data(range->gc);
pin = stm32_gpio_pin(g->pin);
mode = stm32_gpio_get_mode(function);
@@ -604,7 +616,7 @@
struct pinctrl_gpio_range *range, unsigned gpio,
bool input)
{
- struct stm32_gpio_bank *bank = gpio_range_to_bank(range);
+ struct stm32_gpio_bank *bank = gpiochip_get_data(range->gc);
int pin = stm32_gpio_pin(gpio);
stm32_pmx_set_mode(bank, pin, !input, 0);
@@ -762,7 +774,7 @@
int offset, ret = 0;
range = pinctrl_find_gpio_range_from_pin(pctldev, pin);
- bank = gpio_range_to_bank(range);
+ bank = gpiochip_get_data(range->gc);
offset = stm32_gpio_pin(pin);
switch (param) {
@@ -843,7 +855,7 @@
bool val;
range = pinctrl_find_gpio_range_from_pin_nolock(pctldev, pin);
- bank = gpio_range_to_bank(range);
+ bank = gpiochip_get_data(range->gc);
offset = stm32_gpio_pin(pin);
stm32_pmx_get_mode(bank, offset, &mode, &alt);
@@ -898,13 +910,14 @@
static int stm32_gpiolib_register_bank(struct stm32_pinctrl *pctl,
struct device_node *np)
{
- int bank_nr = pctl->nbanks;
- struct stm32_gpio_bank *bank = &pctl->banks[bank_nr];
+ struct stm32_gpio_bank *bank = &pctl->banks[pctl->nbanks];
struct pinctrl_gpio_range *range = &bank->range;
+ struct of_phandle_args args;
struct device *dev = pctl->dev;
struct resource res;
struct reset_control *rstc;
- int err, npins;
+ int npins = STM32_GPIO_PINS_PER_BANK;
+ int bank_nr, err;
rstc = of_reset_control_get(np, NULL);
if (!IS_ERR(rstc))
@@ -929,28 +942,33 @@
return err;
}
- npins = pctl->match_data->npins;
- npins -= bank_nr * STM32_GPIO_PINS_PER_BANK;
- if (npins < 0)
- return -EINVAL;
- else if (npins > STM32_GPIO_PINS_PER_BANK)
- npins = STM32_GPIO_PINS_PER_BANK;
-
bank->gpio_chip = stm32_gpio_template;
+
+ of_property_read_string(np, "st,bank-name", &bank->gpio_chip.label);
+
+ if (!of_parse_phandle_with_fixed_args(np, "gpio-ranges", 3, 0, &args)) {
+ bank_nr = args.args[1] / STM32_GPIO_PINS_PER_BANK;
+ bank->gpio_chip.base = args.args[1];
+ } else {
+ bank_nr = pctl->nbanks;
+ bank->gpio_chip.base = bank_nr * STM32_GPIO_PINS_PER_BANK;
+ range->name = bank->gpio_chip.label;
+ range->id = bank_nr;
+ range->pin_base = range->id * STM32_GPIO_PINS_PER_BANK;
+ range->base = range->id * STM32_GPIO_PINS_PER_BANK;
+ range->npins = npins;
+ range->gc = &bank->gpio_chip;
+ pinctrl_add_gpio_range(pctl->pctl_dev,
+ &pctl->banks[bank_nr].range);
+ }
bank->gpio_chip.base = bank_nr * STM32_GPIO_PINS_PER_BANK;
+
bank->gpio_chip.ngpio = npins;
bank->gpio_chip.of_node = np;
bank->gpio_chip.parent = dev;
+ bank->bank_nr = bank_nr;
spin_lock_init(&bank->lock);
- of_property_read_string(np, "st,bank-name", &range->name);
- bank->gpio_chip.label = range->name;
-
- range->id = bank_nr;
- range->pin_base = range->base = range->id * STM32_GPIO_PINS_PER_BANK;
- range->npins = bank->gpio_chip.ngpio;
- range->gc = &bank->gpio_chip;
-
/* create irq hierarchical domain */
bank->fwnode = of_node_to_fwnode(np);
@@ -967,7 +985,7 @@
return err;
}
- dev_info(dev, "%s bank added\n", range->name);
+ dev_info(dev, "%s bank added\n", bank->gpio_chip.label);
return 0;
}
@@ -1086,30 +1104,6 @@
return ret;
}
- for_each_child_of_node(np, child)
- if (of_property_read_bool(child, "gpio-controller"))
- banks++;
-
- if (!banks) {
- dev_err(dev, "at least one GPIO bank is required\n");
- return -EINVAL;
- }
-
- pctl->banks = devm_kcalloc(dev, banks, sizeof(*pctl->banks),
- GFP_KERNEL);
- if (!pctl->banks)
- return -ENOMEM;
-
- for_each_child_of_node(np, child) {
- if (of_property_read_bool(child, "gpio-controller")) {
- ret = stm32_gpiolib_register_bank(pctl, child);
- if (ret)
- return ret;
-
- pctl->nbanks++;
- }
- }
-
pins = devm_kcalloc(&pdev->dev, pctl->match_data->npins, sizeof(*pins),
GFP_KERNEL);
if (!pins)
@@ -1129,13 +1123,34 @@
pctl->pctl_dev = devm_pinctrl_register(&pdev->dev, &pctl->pctl_desc,
pctl);
+
if (IS_ERR(pctl->pctl_dev)) {
dev_err(&pdev->dev, "Failed pinctrl registration\n");
return PTR_ERR(pctl->pctl_dev);
}
- for (i = 0; i < pctl->nbanks; i++)
- pinctrl_add_gpio_range(pctl->pctl_dev, &pctl->banks[i].range);
+ for_each_child_of_node(np, child)
+ if (of_property_read_bool(child, "gpio-controller"))
+ banks++;
+
+ if (!banks) {
+ dev_err(dev, "at least one GPIO bank is required\n");
+ return -EINVAL;
+ }
+ pctl->banks = devm_kcalloc(dev, banks, sizeof(*pctl->banks),
+ GFP_KERNEL);
+ if (!pctl->banks)
+ return -ENOMEM;
+
+ for_each_child_of_node(np, child) {
+ if (of_property_read_bool(child, "gpio-controller")) {
+ ret = stm32_gpiolib_register_bank(pctl, child);
+ if (ret)
+ return ret;
+
+ pctl->nbanks++;
+ }
+ }
dev_info(dev, "Pinctrl STM32 initialized\n");
diff --git a/drivers/pinctrl/stm32/pinctrl-stm32f429.c b/drivers/pinctrl/stm32/pinctrl-stm32f429.c
index 990b867..4bbade2 100644
--- a/drivers/pinctrl/stm32/pinctrl-stm32f429.c
+++ b/drivers/pinctrl/stm32/pinctrl-stm32f429.c
@@ -1584,4 +1584,8 @@
},
};
-builtin_platform_driver(stm32f429_pinctrl_driver);
+static int __init stm32f429_pinctrl_init(void)
+{
+ return platform_driver_register(&stm32f429_pinctrl_driver);
+}
+arch_initcall(stm32f429_pinctrl_init);
diff --git a/drivers/pinctrl/stm32/pinctrl-stm32f469.c b/drivers/pinctrl/stm32/pinctrl-stm32f469.c
new file mode 100644
index 0000000..86c8ceb
--- /dev/null
+++ b/drivers/pinctrl/stm32/pinctrl-stm32f469.c
@@ -0,0 +1,1578 @@
+/*
+ * Copyright (C) Alexandre Torgue 2016
+ * Author: Alexandre Torgue <alexandre.torgue@st.com>
+ * License terms: GNU General Public License (GPL), version 2
+ */
+#include <linux/init.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+
+#include "pinctrl-stm32.h"
+
+static const struct stm32_desc_pin stm32f469_pins[] = {
+ STM32_PIN(
+ PINCTRL_PIN(0, "PA0"),
+ STM32_FUNCTION(0, "GPIOA0"),
+ STM32_FUNCTION(2, "TIM2_CH1 TIM2_ETR"),
+ STM32_FUNCTION(3, "TIM5_CH1"),
+ STM32_FUNCTION(4, "TIM8_ETR"),
+ STM32_FUNCTION(8, "USART2_CTS"),
+ STM32_FUNCTION(9, "UART4_TX"),
+ STM32_FUNCTION(12, "ETH_MII_CRS"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(1, "PA1"),
+ STM32_FUNCTION(0, "GPIOA1"),
+ STM32_FUNCTION(2, "TIM2_CH2"),
+ STM32_FUNCTION(3, "TIM5_CH2"),
+ STM32_FUNCTION(8, "USART2_RTS"),
+ STM32_FUNCTION(9, "UART4_RX"),
+ STM32_FUNCTION(10, "QUADSPI_BK1_IO3"),
+ STM32_FUNCTION(12, "ETH_MII_RX_CLK ETH_RMII_REF_CLK"),
+ STM32_FUNCTION(15, "LCD_R2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(2, "PA2"),
+ STM32_FUNCTION(0, "GPIOA2"),
+ STM32_FUNCTION(2, "TIM2_CH3"),
+ STM32_FUNCTION(3, "TIM5_CH3"),
+ STM32_FUNCTION(4, "TIM9_CH1"),
+ STM32_FUNCTION(8, "USART2_TX"),
+ STM32_FUNCTION(12, "ETH_MDIO"),
+ STM32_FUNCTION(15, "LCD_R1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(3, "PA3"),
+ STM32_FUNCTION(0, "GPIOA3"),
+ STM32_FUNCTION(2, "TIM2_CH4"),
+ STM32_FUNCTION(3, "TIM5_CH4"),
+ STM32_FUNCTION(4, "TIM9_CH2"),
+ STM32_FUNCTION(8, "USART2_RX"),
+ STM32_FUNCTION(10, "LCD_B2"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_D0"),
+ STM32_FUNCTION(12, "ETH_MII_COL"),
+ STM32_FUNCTION(15, "LCD_B5"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(4, "PA4"),
+ STM32_FUNCTION(0, "GPIOA4"),
+ STM32_FUNCTION(6, "SPI1_NSS"),
+ STM32_FUNCTION(7, "SPI3_NSS I2S3_WS"),
+ STM32_FUNCTION(8, "USART2_CK"),
+ STM32_FUNCTION(13, "OTG_HS_SOF"),
+ STM32_FUNCTION(14, "DCMI_HSYNC"),
+ STM32_FUNCTION(15, "LCD_VSYNC"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(5, "PA5"),
+ STM32_FUNCTION(0, "GPIOA5"),
+ STM32_FUNCTION(2, "TIM2_CH1 TIM2_ETR"),
+ STM32_FUNCTION(4, "TIM8_CH1N"),
+ STM32_FUNCTION(6, "SPI1_SCK"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_CK"),
+ STM32_FUNCTION(15, "LCD_R4"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(6, "PA6"),
+ STM32_FUNCTION(0, "GPIOA6"),
+ STM32_FUNCTION(2, "TIM1_BKIN"),
+ STM32_FUNCTION(3, "TIM3_CH1"),
+ STM32_FUNCTION(4, "TIM8_BKIN"),
+ STM32_FUNCTION(6, "SPI1_MISO"),
+ STM32_FUNCTION(10, "TIM13_CH1"),
+ STM32_FUNCTION(14, "DCMI_PIXCLK"),
+ STM32_FUNCTION(15, "LCD_G2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(7, "PA7"),
+ STM32_FUNCTION(0, "GPIOA7"),
+ STM32_FUNCTION(2, "TIM1_CH1N"),
+ STM32_FUNCTION(3, "TIM3_CH2"),
+ STM32_FUNCTION(4, "TIM8_CH1N"),
+ STM32_FUNCTION(6, "SPI1_MOSI"),
+ STM32_FUNCTION(10, "TIM14_CH1"),
+ STM32_FUNCTION(11, "QUADSPI_CLK"),
+ STM32_FUNCTION(12, "ETH_MII_RX_DV ETH_RMII_CRS_DV"),
+ STM32_FUNCTION(13, "FMC_SDNWE"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(8, "PA8"),
+ STM32_FUNCTION(0, "GPIOA8"),
+ STM32_FUNCTION(1, "MCO1"),
+ STM32_FUNCTION(2, "TIM1_CH1"),
+ STM32_FUNCTION(5, "I2C3_SCL"),
+ STM32_FUNCTION(8, "USART1_CK"),
+ STM32_FUNCTION(11, "OTG_FS_SOF"),
+ STM32_FUNCTION(15, "LCD_R6"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(9, "PA9"),
+ STM32_FUNCTION(0, "GPIOA9"),
+ STM32_FUNCTION(2, "TIM1_CH2"),
+ STM32_FUNCTION(5, "I2C3_SMBA"),
+ STM32_FUNCTION(6, "SPI2_SCK I2S2_CK"),
+ STM32_FUNCTION(8, "USART1_TX"),
+ STM32_FUNCTION(14, "DCMI_D0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(10, "PA10"),
+ STM32_FUNCTION(0, "GPIOA10"),
+ STM32_FUNCTION(2, "TIM1_CH3"),
+ STM32_FUNCTION(8, "USART1_RX"),
+ STM32_FUNCTION(11, "OTG_FS_ID"),
+ STM32_FUNCTION(14, "DCMI_D1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(11, "PA11"),
+ STM32_FUNCTION(0, "GPIOA11"),
+ STM32_FUNCTION(2, "TIM1_CH4"),
+ STM32_FUNCTION(8, "USART1_CTS"),
+ STM32_FUNCTION(10, "CAN1_RX"),
+ STM32_FUNCTION(11, "OTG_FS_DM"),
+ STM32_FUNCTION(15, "LCD_R4"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(12, "PA12"),
+ STM32_FUNCTION(0, "GPIOA12"),
+ STM32_FUNCTION(2, "TIM1_ETR"),
+ STM32_FUNCTION(8, "USART1_RTS"),
+ STM32_FUNCTION(10, "CAN1_TX"),
+ STM32_FUNCTION(11, "OTG_FS_DP"),
+ STM32_FUNCTION(15, "LCD_R5"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(13, "PA13"),
+ STM32_FUNCTION(0, "GPIOA13"),
+ STM32_FUNCTION(1, "JTMS SWDIO"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(14, "PA14"),
+ STM32_FUNCTION(0, "GPIOA14"),
+ STM32_FUNCTION(1, "JTCK SWCLK"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(15, "PA15"),
+ STM32_FUNCTION(0, "GPIOA15"),
+ STM32_FUNCTION(1, "JTDI"),
+ STM32_FUNCTION(2, "TIM2_CH1 TIM2_ETR"),
+ STM32_FUNCTION(6, "SPI1_NSS"),
+ STM32_FUNCTION(7, "SPI3_NSS I2S3_WS"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(16, "PB0"),
+ STM32_FUNCTION(0, "GPIOB0"),
+ STM32_FUNCTION(2, "TIM1_CH2N"),
+ STM32_FUNCTION(3, "TIM3_CH3"),
+ STM32_FUNCTION(4, "TIM8_CH2N"),
+ STM32_FUNCTION(10, "LCD_R3"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_D1"),
+ STM32_FUNCTION(12, "ETH_MII_RXD2"),
+ STM32_FUNCTION(15, "LCD_G1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(17, "PB1"),
+ STM32_FUNCTION(0, "GPIOB1"),
+ STM32_FUNCTION(2, "TIM1_CH3N"),
+ STM32_FUNCTION(3, "TIM3_CH4"),
+ STM32_FUNCTION(4, "TIM8_CH3N"),
+ STM32_FUNCTION(10, "LCD_R6"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_D2"),
+ STM32_FUNCTION(12, "ETH_MII_RXD3"),
+ STM32_FUNCTION(15, "LCD_G0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(18, "PB2"),
+ STM32_FUNCTION(0, "GPIOB2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(19, "PB3"),
+ STM32_FUNCTION(0, "GPIOB3"),
+ STM32_FUNCTION(1, "JTDO TRACESWO"),
+ STM32_FUNCTION(2, "TIM2_CH2"),
+ STM32_FUNCTION(6, "SPI1_SCK"),
+ STM32_FUNCTION(7, "SPI3_SCK I2S3_CK"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(20, "PB4"),
+ STM32_FUNCTION(0, "GPIOB4"),
+ STM32_FUNCTION(1, "NJTRST"),
+ STM32_FUNCTION(3, "TIM3_CH1"),
+ STM32_FUNCTION(6, "SPI1_MISO"),
+ STM32_FUNCTION(7, "SPI3_MISO"),
+ STM32_FUNCTION(8, "I2S3EXT_SD"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(21, "PB5"),
+ STM32_FUNCTION(0, "GPIOB5"),
+ STM32_FUNCTION(3, "TIM3_CH2"),
+ STM32_FUNCTION(5, "I2C1_SMBA"),
+ STM32_FUNCTION(6, "SPI1_MOSI"),
+ STM32_FUNCTION(7, "SPI3_MOSI I2S3_SD"),
+ STM32_FUNCTION(10, "CAN2_RX"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_D7"),
+ STM32_FUNCTION(12, "ETH_PPS_OUT"),
+ STM32_FUNCTION(13, "FMC_SDCKE1"),
+ STM32_FUNCTION(14, "DCMI_D10"),
+ STM32_FUNCTION(15, "LCD_G7"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(22, "PB6"),
+ STM32_FUNCTION(0, "GPIOB6"),
+ STM32_FUNCTION(3, "TIM4_CH1"),
+ STM32_FUNCTION(5, "I2C1_SCL"),
+ STM32_FUNCTION(8, "USART1_TX"),
+ STM32_FUNCTION(10, "CAN2_TX"),
+ STM32_FUNCTION(11, "QUADSPI_BK1_NCS"),
+ STM32_FUNCTION(13, "FMC_SDNE1"),
+ STM32_FUNCTION(14, "DCMI_D5"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(23, "PB7"),
+ STM32_FUNCTION(0, "GPIOB7"),
+ STM32_FUNCTION(3, "TIM4_CH2"),
+ STM32_FUNCTION(5, "I2C1_SDA"),
+ STM32_FUNCTION(8, "USART1_RX"),
+ STM32_FUNCTION(13, "FMC_NL"),
+ STM32_FUNCTION(14, "DCMI_VSYNC"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(24, "PB8"),
+ STM32_FUNCTION(0, "GPIOB8"),
+ STM32_FUNCTION(3, "TIM4_CH3"),
+ STM32_FUNCTION(4, "TIM10_CH1"),
+ STM32_FUNCTION(5, "I2C1_SCL"),
+ STM32_FUNCTION(10, "CAN1_RX"),
+ STM32_FUNCTION(12, "ETH_MII_TXD3"),
+ STM32_FUNCTION(13, "SDIO_D4"),
+ STM32_FUNCTION(14, "DCMI_D6"),
+ STM32_FUNCTION(15, "LCD_B6"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(25, "PB9"),
+ STM32_FUNCTION(0, "GPIOB9"),
+ STM32_FUNCTION(3, "TIM4_CH4"),
+ STM32_FUNCTION(4, "TIM11_CH1"),
+ STM32_FUNCTION(5, "I2C1_SDA"),
+ STM32_FUNCTION(6, "SPI2_NSS I2S2_WS"),
+ STM32_FUNCTION(10, "CAN1_TX"),
+ STM32_FUNCTION(13, "SDIO_D5"),
+ STM32_FUNCTION(14, "DCMI_D7"),
+ STM32_FUNCTION(15, "LCD_B7"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(26, "PB10"),
+ STM32_FUNCTION(0, "GPIOB10"),
+ STM32_FUNCTION(2, "TIM2_CH3"),
+ STM32_FUNCTION(5, "I2C2_SCL"),
+ STM32_FUNCTION(6, "SPI2_SCK I2S2_CK"),
+ STM32_FUNCTION(8, "USART3_TX"),
+ STM32_FUNCTION(10, "QUADSPI_BK1_NCS"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_D3"),
+ STM32_FUNCTION(12, "ETH_MII_RX_ER"),
+ STM32_FUNCTION(15, "LCD_G4"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(27, "PB11"),
+ STM32_FUNCTION(0, "GPIOB11"),
+ STM32_FUNCTION(2, "TIM2_CH4"),
+ STM32_FUNCTION(5, "I2C2_SDA"),
+ STM32_FUNCTION(8, "USART3_RX"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_D4"),
+ STM32_FUNCTION(12, "ETH_MII_TX_EN ETH_RMII_TX_EN"),
+ STM32_FUNCTION(14, "DSIHOST_TE"),
+ STM32_FUNCTION(15, "LCD_G5"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(28, "PB12"),
+ STM32_FUNCTION(0, "GPIOB12"),
+ STM32_FUNCTION(2, "TIM1_BKIN"),
+ STM32_FUNCTION(5, "I2C2_SMBA"),
+ STM32_FUNCTION(6, "SPI2_NSS I2S2_WS"),
+ STM32_FUNCTION(8, "USART3_CK"),
+ STM32_FUNCTION(10, "CAN2_RX"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_D5"),
+ STM32_FUNCTION(12, "ETH_MII_TXD0 ETH_RMII_TXD0"),
+ STM32_FUNCTION(13, "OTG_HS_ID"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(29, "PB13"),
+ STM32_FUNCTION(0, "GPIOB13"),
+ STM32_FUNCTION(2, "TIM1_CH1N"),
+ STM32_FUNCTION(6, "SPI2_SCK I2S2_CK"),
+ STM32_FUNCTION(8, "USART3_CTS"),
+ STM32_FUNCTION(10, "CAN2_TX"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_D6"),
+ STM32_FUNCTION(12, "ETH_MII_TXD1 ETH_RMII_TXD1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(30, "PB14"),
+ STM32_FUNCTION(0, "GPIOB14"),
+ STM32_FUNCTION(2, "TIM1_CH2N"),
+ STM32_FUNCTION(4, "TIM8_CH2N"),
+ STM32_FUNCTION(6, "SPI2_MISO"),
+ STM32_FUNCTION(7, "I2S2EXT_SD"),
+ STM32_FUNCTION(8, "USART3_RTS"),
+ STM32_FUNCTION(10, "TIM12_CH1"),
+ STM32_FUNCTION(13, "OTG_HS_DM"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(31, "PB15"),
+ STM32_FUNCTION(0, "GPIOB15"),
+ STM32_FUNCTION(1, "RTC_REFIN"),
+ STM32_FUNCTION(2, "TIM1_CH3N"),
+ STM32_FUNCTION(4, "TIM8_CH3N"),
+ STM32_FUNCTION(6, "SPI2_MOSI I2S2_SD"),
+ STM32_FUNCTION(10, "TIM12_CH2"),
+ STM32_FUNCTION(13, "OTG_HS_DP"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(32, "PC0"),
+ STM32_FUNCTION(0, "GPIOC0"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_STP"),
+ STM32_FUNCTION(13, "FMC_SDNWE"),
+ STM32_FUNCTION(15, "LCD_R5"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(33, "PC1"),
+ STM32_FUNCTION(0, "GPIOC1"),
+ STM32_FUNCTION(1, "TRACED0"),
+ STM32_FUNCTION(6, "SPI2_MOSI I2S2_SD"),
+ STM32_FUNCTION(7, "SAI1_SD_A"),
+ STM32_FUNCTION(12, "ETH_MDC"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(34, "PC2"),
+ STM32_FUNCTION(0, "GPIOC2"),
+ STM32_FUNCTION(6, "SPI2_MISO"),
+ STM32_FUNCTION(7, "I2S2EXT_SD"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_DIR"),
+ STM32_FUNCTION(12, "ETH_MII_TXD2"),
+ STM32_FUNCTION(13, "FMC_SDNE0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(35, "PC3"),
+ STM32_FUNCTION(0, "GPIOC3"),
+ STM32_FUNCTION(6, "SPI2_MOSI I2S2_SD"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_NXT"),
+ STM32_FUNCTION(12, "ETH_MII_TX_CLK"),
+ STM32_FUNCTION(13, "FMC_SDCKE0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(36, "PC4"),
+ STM32_FUNCTION(0, "GPIOC4"),
+ STM32_FUNCTION(12, "ETH_MII_RXD0 ETH_RMII_RXD0"),
+ STM32_FUNCTION(13, "FMC_SDNE0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(37, "PC5"),
+ STM32_FUNCTION(0, "GPIOC5"),
+ STM32_FUNCTION(12, "ETH_MII_RXD1 ETH_RMII_RXD1"),
+ STM32_FUNCTION(13, "FMC_SDCKE0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(38, "PC6"),
+ STM32_FUNCTION(0, "GPIOC6"),
+ STM32_FUNCTION(3, "TIM3_CH1"),
+ STM32_FUNCTION(4, "TIM8_CH1"),
+ STM32_FUNCTION(6, "I2S2_MCK"),
+ STM32_FUNCTION(9, "USART6_TX"),
+ STM32_FUNCTION(13, "SDIO_D6"),
+ STM32_FUNCTION(14, "DCMI_D0"),
+ STM32_FUNCTION(15, "LCD_HSYNC"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(39, "PC7"),
+ STM32_FUNCTION(0, "GPIOC7"),
+ STM32_FUNCTION(3, "TIM3_CH2"),
+ STM32_FUNCTION(4, "TIM8_CH2"),
+ STM32_FUNCTION(7, "I2S3_MCK"),
+ STM32_FUNCTION(9, "USART6_RX"),
+ STM32_FUNCTION(13, "SDIO_D7"),
+ STM32_FUNCTION(14, "DCMI_D1"),
+ STM32_FUNCTION(15, "LCD_G6"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(40, "PC8"),
+ STM32_FUNCTION(0, "GPIOC8"),
+ STM32_FUNCTION(1, "TRACED1"),
+ STM32_FUNCTION(3, "TIM3_CH3"),
+ STM32_FUNCTION(4, "TIM8_CH3"),
+ STM32_FUNCTION(9, "USART6_CK"),
+ STM32_FUNCTION(13, "SDIO_D0"),
+ STM32_FUNCTION(14, "DCMI_D2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(41, "PC9"),
+ STM32_FUNCTION(0, "GPIOC9"),
+ STM32_FUNCTION(1, "MCO2"),
+ STM32_FUNCTION(3, "TIM3_CH4"),
+ STM32_FUNCTION(4, "TIM8_CH4"),
+ STM32_FUNCTION(5, "I2C3_SDA"),
+ STM32_FUNCTION(6, "I2S_CKIN"),
+ STM32_FUNCTION(10, "QUADSPI_BK1_IO0"),
+ STM32_FUNCTION(13, "SDIO_D1"),
+ STM32_FUNCTION(14, "DCMI_D3"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(42, "PC10"),
+ STM32_FUNCTION(0, "GPIOC10"),
+ STM32_FUNCTION(7, "SPI3_SCK I2S3_CK"),
+ STM32_FUNCTION(8, "USART3_TX"),
+ STM32_FUNCTION(9, "UART4_TX"),
+ STM32_FUNCTION(10, "QUADSPI_BK1_IO1"),
+ STM32_FUNCTION(13, "SDIO_D2"),
+ STM32_FUNCTION(14, "DCMI_D8"),
+ STM32_FUNCTION(15, "LCD_R2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(43, "PC11"),
+ STM32_FUNCTION(0, "GPIOC11"),
+ STM32_FUNCTION(6, "I2S3EXT_SD"),
+ STM32_FUNCTION(7, "SPI3_MISO"),
+ STM32_FUNCTION(8, "USART3_RX"),
+ STM32_FUNCTION(9, "UART4_RX"),
+ STM32_FUNCTION(10, "QUADSPI_BK2_NCS"),
+ STM32_FUNCTION(13, "SDIO_D3"),
+ STM32_FUNCTION(14, "DCMI_D4"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(44, "PC12"),
+ STM32_FUNCTION(0, "GPIOC12"),
+ STM32_FUNCTION(1, "TRACED3"),
+ STM32_FUNCTION(7, "SPI3_MOSI I2S3_SD"),
+ STM32_FUNCTION(8, "USART3_CK"),
+ STM32_FUNCTION(9, "UART5_TX"),
+ STM32_FUNCTION(13, "SDIO_CK"),
+ STM32_FUNCTION(14, "DCMI_D9"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(45, "PC13"),
+ STM32_FUNCTION(0, "GPIOC13"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(46, "PC14"),
+ STM32_FUNCTION(0, "GPIOC14"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(47, "PC15"),
+ STM32_FUNCTION(0, "GPIOC15"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(48, "PD0"),
+ STM32_FUNCTION(0, "GPIOD0"),
+ STM32_FUNCTION(10, "CAN1_RX"),
+ STM32_FUNCTION(13, "FMC_D2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(49, "PD1"),
+ STM32_FUNCTION(0, "GPIOD1"),
+ STM32_FUNCTION(10, "CAN1_TX"),
+ STM32_FUNCTION(13, "FMC_D3"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(50, "PD2"),
+ STM32_FUNCTION(0, "GPIOD2"),
+ STM32_FUNCTION(1, "TRACED2"),
+ STM32_FUNCTION(3, "TIM3_ETR"),
+ STM32_FUNCTION(9, "UART5_RX"),
+ STM32_FUNCTION(13, "SDIO_CMD"),
+ STM32_FUNCTION(14, "DCMI_D11"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(51, "PD3"),
+ STM32_FUNCTION(0, "GPIOD3"),
+ STM32_FUNCTION(6, "SPI2_SCK I2S2_CK"),
+ STM32_FUNCTION(8, "USART2_CTS"),
+ STM32_FUNCTION(13, "FMC_CLK"),
+ STM32_FUNCTION(14, "DCMI_D5"),
+ STM32_FUNCTION(15, "LCD_G7"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(52, "PD4"),
+ STM32_FUNCTION(0, "GPIOD4"),
+ STM32_FUNCTION(8, "USART2_RTS"),
+ STM32_FUNCTION(13, "FMC_NOE"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(53, "PD5"),
+ STM32_FUNCTION(0, "GPIOD5"),
+ STM32_FUNCTION(8, "USART2_TX"),
+ STM32_FUNCTION(13, "FMC_NWE"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(54, "PD6"),
+ STM32_FUNCTION(0, "GPIOD6"),
+ STM32_FUNCTION(6, "SPI3_MOSI I2S3_SD"),
+ STM32_FUNCTION(7, "SAI1_SD_A"),
+ STM32_FUNCTION(8, "USART2_RX"),
+ STM32_FUNCTION(13, "FMC_NWAIT"),
+ STM32_FUNCTION(14, "DCMI_D10"),
+ STM32_FUNCTION(15, "LCD_B2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(55, "PD7"),
+ STM32_FUNCTION(0, "GPIOD7"),
+ STM32_FUNCTION(8, "USART2_CK"),
+ STM32_FUNCTION(13, "FMC_NE1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(56, "PD8"),
+ STM32_FUNCTION(0, "GPIOD8"),
+ STM32_FUNCTION(8, "USART3_TX"),
+ STM32_FUNCTION(13, "FMC_D13"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(57, "PD9"),
+ STM32_FUNCTION(0, "GPIOD9"),
+ STM32_FUNCTION(8, "USART3_RX"),
+ STM32_FUNCTION(13, "FMC_D14"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(58, "PD10"),
+ STM32_FUNCTION(0, "GPIOD10"),
+ STM32_FUNCTION(8, "USART3_CK"),
+ STM32_FUNCTION(13, "FMC_D15"),
+ STM32_FUNCTION(15, "LCD_B3"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(59, "PD11"),
+ STM32_FUNCTION(0, "GPIOD11"),
+ STM32_FUNCTION(8, "USART3_CTS"),
+ STM32_FUNCTION(10, "QUADSPI_BK1_IO0"),
+ STM32_FUNCTION(13, "FMC_A16 FMC_CLE"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(60, "PD12"),
+ STM32_FUNCTION(0, "GPIOD12"),
+ STM32_FUNCTION(3, "TIM4_CH1"),
+ STM32_FUNCTION(8, "USART3_RTS"),
+ STM32_FUNCTION(10, "QUADSPI_BK1_IO1"),
+ STM32_FUNCTION(13, "FMC_A17 FMC_ALE"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(61, "PD13"),
+ STM32_FUNCTION(0, "GPIOD13"),
+ STM32_FUNCTION(3, "TIM4_CH2"),
+ STM32_FUNCTION(10, "QUADSPI_BK1_IO3"),
+ STM32_FUNCTION(13, "FMC_A18"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(62, "PD14"),
+ STM32_FUNCTION(0, "GPIOD14"),
+ STM32_FUNCTION(3, "TIM4_CH3"),
+ STM32_FUNCTION(13, "FMC_D0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(63, "PD15"),
+ STM32_FUNCTION(0, "GPIOD15"),
+ STM32_FUNCTION(3, "TIM4_CH4"),
+ STM32_FUNCTION(13, "FMC_D1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(64, "PE0"),
+ STM32_FUNCTION(0, "GPIOE0"),
+ STM32_FUNCTION(3, "TIM4_ETR"),
+ STM32_FUNCTION(9, "UART8_RX"),
+ STM32_FUNCTION(13, "FMC_NBL0"),
+ STM32_FUNCTION(14, "DCMI_D2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(65, "PE1"),
+ STM32_FUNCTION(0, "GPIOE1"),
+ STM32_FUNCTION(9, "UART8_TX"),
+ STM32_FUNCTION(13, "FMC_NBL1"),
+ STM32_FUNCTION(14, "DCMI_D3"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(66, "PE2"),
+ STM32_FUNCTION(0, "GPIOE2"),
+ STM32_FUNCTION(1, "TRACECLK"),
+ STM32_FUNCTION(6, "SPI4_SCK"),
+ STM32_FUNCTION(7, "SAI1_MCLK_A"),
+ STM32_FUNCTION(10, "QUADSPI_BK1_IO2"),
+ STM32_FUNCTION(12, "ETH_MII_TXD3"),
+ STM32_FUNCTION(13, "FMC_A23"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(67, "PE3"),
+ STM32_FUNCTION(0, "GPIOE3"),
+ STM32_FUNCTION(1, "TRACED0"),
+ STM32_FUNCTION(7, "SAI1_SD_B"),
+ STM32_FUNCTION(13, "FMC_A19"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(68, "PE4"),
+ STM32_FUNCTION(0, "GPIOE4"),
+ STM32_FUNCTION(1, "TRACED1"),
+ STM32_FUNCTION(6, "SPI4_NSS"),
+ STM32_FUNCTION(7, "SAI1_FS_A"),
+ STM32_FUNCTION(13, "FMC_A20"),
+ STM32_FUNCTION(14, "DCMI_D4"),
+ STM32_FUNCTION(15, "LCD_B0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(69, "PE5"),
+ STM32_FUNCTION(0, "GPIOE5"),
+ STM32_FUNCTION(1, "TRACED2"),
+ STM32_FUNCTION(4, "TIM9_CH1"),
+ STM32_FUNCTION(6, "SPI4_MISO"),
+ STM32_FUNCTION(7, "SAI1_SCK_A"),
+ STM32_FUNCTION(13, "FMC_A21"),
+ STM32_FUNCTION(14, "DCMI_D6"),
+ STM32_FUNCTION(15, "LCD_G0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(70, "PE6"),
+ STM32_FUNCTION(0, "GPIOE6"),
+ STM32_FUNCTION(1, "TRACED3"),
+ STM32_FUNCTION(4, "TIM9_CH2"),
+ STM32_FUNCTION(6, "SPI4_MOSI"),
+ STM32_FUNCTION(7, "SAI1_SD_A"),
+ STM32_FUNCTION(13, "FMC_A22"),
+ STM32_FUNCTION(14, "DCMI_D7"),
+ STM32_FUNCTION(15, "LCD_G1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(71, "PE7"),
+ STM32_FUNCTION(0, "GPIOE7"),
+ STM32_FUNCTION(2, "TIM1_ETR"),
+ STM32_FUNCTION(9, "UART7_RX"),
+ STM32_FUNCTION(11, "QUADSPI_BK2_IO0"),
+ STM32_FUNCTION(13, "FMC_D4"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(72, "PE8"),
+ STM32_FUNCTION(0, "GPIOE8"),
+ STM32_FUNCTION(2, "TIM1_CH1N"),
+ STM32_FUNCTION(9, "UART7_TX"),
+ STM32_FUNCTION(11, "QUADSPI_BK2_IO1"),
+ STM32_FUNCTION(13, "FMC_D5"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(73, "PE9"),
+ STM32_FUNCTION(0, "GPIOE9"),
+ STM32_FUNCTION(2, "TIM1_CH1"),
+ STM32_FUNCTION(11, "QUADSPI_BK2_IO2"),
+ STM32_FUNCTION(13, "FMC_D6"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(74, "PE10"),
+ STM32_FUNCTION(0, "GPIOE10"),
+ STM32_FUNCTION(2, "TIM1_CH2N"),
+ STM32_FUNCTION(11, "QUADSPI_BK2_IO3"),
+ STM32_FUNCTION(13, "FMC_D7"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(75, "PE11"),
+ STM32_FUNCTION(0, "GPIOE11"),
+ STM32_FUNCTION(2, "TIM1_CH2"),
+ STM32_FUNCTION(6, "SPI4_NSS"),
+ STM32_FUNCTION(13, "FMC_D8"),
+ STM32_FUNCTION(15, "LCD_G3"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(76, "PE12"),
+ STM32_FUNCTION(0, "GPIOE12"),
+ STM32_FUNCTION(2, "TIM1_CH3N"),
+ STM32_FUNCTION(6, "SPI4_SCK"),
+ STM32_FUNCTION(13, "FMC_D9"),
+ STM32_FUNCTION(15, "LCD_B4"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(77, "PE13"),
+ STM32_FUNCTION(0, "GPIOE13"),
+ STM32_FUNCTION(2, "TIM1_CH3"),
+ STM32_FUNCTION(6, "SPI4_MISO"),
+ STM32_FUNCTION(13, "FMC_D10"),
+ STM32_FUNCTION(15, "LCD_DE"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(78, "PE14"),
+ STM32_FUNCTION(0, "GPIOE14"),
+ STM32_FUNCTION(2, "TIM1_CH4"),
+ STM32_FUNCTION(6, "SPI4_MOSI"),
+ STM32_FUNCTION(13, "FMC_D11"),
+ STM32_FUNCTION(15, "LCD_CLK"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(79, "PE15"),
+ STM32_FUNCTION(0, "GPIOE15"),
+ STM32_FUNCTION(2, "TIM1_BKIN"),
+ STM32_FUNCTION(13, "FMC_D12"),
+ STM32_FUNCTION(15, "LCD_R7"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(80, "PF0"),
+ STM32_FUNCTION(0, "GPIOF0"),
+ STM32_FUNCTION(5, "I2C2_SDA"),
+ STM32_FUNCTION(13, "FMC_A0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(81, "PF1"),
+ STM32_FUNCTION(0, "GPIOF1"),
+ STM32_FUNCTION(5, "I2C2_SCL"),
+ STM32_FUNCTION(13, "FMC_A1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(82, "PF2"),
+ STM32_FUNCTION(0, "GPIOF2"),
+ STM32_FUNCTION(5, "I2C2_SMBA"),
+ STM32_FUNCTION(13, "FMC_A2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(83, "PF3"),
+ STM32_FUNCTION(0, "GPIOF3"),
+ STM32_FUNCTION(13, "FMC_A3"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(84, "PF4"),
+ STM32_FUNCTION(0, "GPIOF4"),
+ STM32_FUNCTION(13, "FMC_A4"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(85, "PF5"),
+ STM32_FUNCTION(0, "GPIOF5"),
+ STM32_FUNCTION(13, "FMC_A5"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(86, "PF6"),
+ STM32_FUNCTION(0, "GPIOF6"),
+ STM32_FUNCTION(4, "TIM10_CH1"),
+ STM32_FUNCTION(6, "SPI5_NSS"),
+ STM32_FUNCTION(7, "SAI1_SD_B"),
+ STM32_FUNCTION(9, "UART7_RX"),
+ STM32_FUNCTION(10, "QUADSPI_BK1_IO3"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(87, "PF7"),
+ STM32_FUNCTION(0, "GPIOF7"),
+ STM32_FUNCTION(4, "TIM11_CH1"),
+ STM32_FUNCTION(6, "SPI5_SCK"),
+ STM32_FUNCTION(7, "SAI1_MCLK_B"),
+ STM32_FUNCTION(9, "UART7_TX"),
+ STM32_FUNCTION(10, "QUADSPI_BK1_IO2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(88, "PF8"),
+ STM32_FUNCTION(0, "GPIOF8"),
+ STM32_FUNCTION(6, "SPI5_MISO"),
+ STM32_FUNCTION(7, "SAI1_SCK_B"),
+ STM32_FUNCTION(10, "TIM13_CH1"),
+ STM32_FUNCTION(11, "QUADSPI_BK1_IO0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(89, "PF9"),
+ STM32_FUNCTION(0, "GPIOF9"),
+ STM32_FUNCTION(6, "SPI5_MOSI"),
+ STM32_FUNCTION(7, "SAI1_FS_B"),
+ STM32_FUNCTION(10, "TIM14_CH1"),
+ STM32_FUNCTION(11, "QUADSPI_BK1_IO1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(90, "PF10"),
+ STM32_FUNCTION(0, "GPIOF10"),
+ STM32_FUNCTION(10, "QUADSPI_CLK"),
+ STM32_FUNCTION(14, "DCMI_D11"),
+ STM32_FUNCTION(15, "LCD_DE"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(91, "PF11"),
+ STM32_FUNCTION(0, "GPIOF11"),
+ STM32_FUNCTION(6, "SPI5_MOSI"),
+ STM32_FUNCTION(13, "FMC_SDNRAS"),
+ STM32_FUNCTION(14, "DCMI_D12"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(92, "PF12"),
+ STM32_FUNCTION(0, "GPIOF12"),
+ STM32_FUNCTION(13, "FMC_A6"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(93, "PF13"),
+ STM32_FUNCTION(0, "GPIOF13"),
+ STM32_FUNCTION(13, "FMC_A7"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(94, "PF14"),
+ STM32_FUNCTION(0, "GPIOF14"),
+ STM32_FUNCTION(13, "FMC_A8"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(95, "PF15"),
+ STM32_FUNCTION(0, "GPIOF15"),
+ STM32_FUNCTION(13, "FMC_A9"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(96, "PG0"),
+ STM32_FUNCTION(0, "GPIOG0"),
+ STM32_FUNCTION(13, "FMC_A10"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(97, "PG1"),
+ STM32_FUNCTION(0, "GPIOG1"),
+ STM32_FUNCTION(13, "FMC_A11"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(98, "PG2"),
+ STM32_FUNCTION(0, "GPIOG2"),
+ STM32_FUNCTION(13, "FMC_A12"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(99, "PG3"),
+ STM32_FUNCTION(0, "GPIOG3"),
+ STM32_FUNCTION(13, "FMC_A13"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(100, "PG4"),
+ STM32_FUNCTION(0, "GPIOG4"),
+ STM32_FUNCTION(13, "FMC_A14 FMC_BA0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(101, "PG5"),
+ STM32_FUNCTION(0, "GPIOG5"),
+ STM32_FUNCTION(13, "FMC_A15 FMC_BA1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(102, "PG6"),
+ STM32_FUNCTION(0, "GPIOG6"),
+ STM32_FUNCTION(14, "DCMI_D12"),
+ STM32_FUNCTION(15, "LCD_R7"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(103, "PG7"),
+ STM32_FUNCTION(0, "GPIOG7"),
+ STM32_FUNCTION(7, "SAI1_MCLK_A"),
+ STM32_FUNCTION(9, "USART6_CK"),
+ STM32_FUNCTION(13, "FMC_INT"),
+ STM32_FUNCTION(14, "DCMI_D13"),
+ STM32_FUNCTION(15, "LCD_CLK"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(104, "PG8"),
+ STM32_FUNCTION(0, "GPIOG8"),
+ STM32_FUNCTION(6, "SPI6_NSS"),
+ STM32_FUNCTION(9, "USART6_RTS"),
+ STM32_FUNCTION(12, "ETH_PPS_OUT"),
+ STM32_FUNCTION(13, "FMC_SDCLK"),
+ STM32_FUNCTION(15, "LCD_G7"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(105, "PG9"),
+ STM32_FUNCTION(0, "GPIOG9"),
+ STM32_FUNCTION(9, "USART6_RX"),
+ STM32_FUNCTION(10, "QUADSPI_BK2_IO2"),
+ STM32_FUNCTION(13, "FMC_NE2 FMC_NCE"),
+ STM32_FUNCTION(14, "DCMI_VSYNC"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(106, "PG10"),
+ STM32_FUNCTION(0, "GPIOG10"),
+ STM32_FUNCTION(10, "LCD_G3"),
+ STM32_FUNCTION(13, "FMC_NE3"),
+ STM32_FUNCTION(14, "DCMI_D2"),
+ STM32_FUNCTION(15, "LCD_B2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(107, "PG11"),
+ STM32_FUNCTION(0, "GPIOG11"),
+ STM32_FUNCTION(12, "ETH_MII_TX_EN ETH_RMII_TX_EN"),
+ STM32_FUNCTION(14, "DCMI_D3"),
+ STM32_FUNCTION(15, "LCD_B3"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(108, "PG12"),
+ STM32_FUNCTION(0, "GPIOG12"),
+ STM32_FUNCTION(6, "SPI6_MISO"),
+ STM32_FUNCTION(9, "USART6_RTS"),
+ STM32_FUNCTION(10, "LCD_B4"),
+ STM32_FUNCTION(13, "FMC_NE4"),
+ STM32_FUNCTION(15, "LCD_B1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(109, "PG13"),
+ STM32_FUNCTION(0, "GPIOG13"),
+ STM32_FUNCTION(1, "TRACED0"),
+ STM32_FUNCTION(6, "SPI6_SCK"),
+ STM32_FUNCTION(9, "USART6_CTS"),
+ STM32_FUNCTION(12, "ETH_MII_TXD0 ETH_RMII_TXD0"),
+ STM32_FUNCTION(13, "FMC_A24"),
+ STM32_FUNCTION(15, "LCD_R0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(110, "PG14"),
+ STM32_FUNCTION(0, "GPIOG14"),
+ STM32_FUNCTION(1, "TRACED1"),
+ STM32_FUNCTION(6, "SPI6_MOSI"),
+ STM32_FUNCTION(9, "USART6_TX"),
+ STM32_FUNCTION(10, "QUADSPI_BK2_IO3"),
+ STM32_FUNCTION(12, "ETH_MII_TXD1 ETH_RMII_TXD1"),
+ STM32_FUNCTION(13, "FMC_A25"),
+ STM32_FUNCTION(15, "LCD_B0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(111, "PG15"),
+ STM32_FUNCTION(0, "GPIOG15"),
+ STM32_FUNCTION(9, "USART6_CTS"),
+ STM32_FUNCTION(13, "FMC_SDNCAS"),
+ STM32_FUNCTION(14, "DCMI_D13"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(112, "PH0"),
+ STM32_FUNCTION(0, "GPIOH0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(113, "PH1"),
+ STM32_FUNCTION(0, "GPIOH1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(114, "PH2"),
+ STM32_FUNCTION(0, "GPIOH2"),
+ STM32_FUNCTION(10, "QUADSPI_BK2_IO0"),
+ STM32_FUNCTION(12, "ETH_MII_CRS"),
+ STM32_FUNCTION(13, "FMC_SDCKE0"),
+ STM32_FUNCTION(15, "LCD_R0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(115, "PH3"),
+ STM32_FUNCTION(0, "GPIOH3"),
+ STM32_FUNCTION(10, "QUADSPI_BK2_IO1"),
+ STM32_FUNCTION(12, "ETH_MII_COL"),
+ STM32_FUNCTION(13, "FMC_SDNE0"),
+ STM32_FUNCTION(15, "LCD_R1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(116, "PH4"),
+ STM32_FUNCTION(0, "GPIOH4"),
+ STM32_FUNCTION(5, "I2C2_SCL"),
+ STM32_FUNCTION(10, "LCD_G5"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_NXT"),
+ STM32_FUNCTION(15, "LCD_G4"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(117, "PH5"),
+ STM32_FUNCTION(0, "GPIOH5"),
+ STM32_FUNCTION(5, "I2C2_SDA"),
+ STM32_FUNCTION(6, "SPI5_NSS"),
+ STM32_FUNCTION(13, "FMC_SDNWE"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(118, "PH6"),
+ STM32_FUNCTION(0, "GPIOH6"),
+ STM32_FUNCTION(5, "I2C2_SMBA"),
+ STM32_FUNCTION(6, "SPI5_SCK"),
+ STM32_FUNCTION(10, "TIM12_CH1"),
+ STM32_FUNCTION(12, "ETH_MII_RXD2"),
+ STM32_FUNCTION(13, "FMC_SDNE1"),
+ STM32_FUNCTION(14, "DCMI_D8"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(119, "PH7"),
+ STM32_FUNCTION(0, "GPIOH7"),
+ STM32_FUNCTION(5, "I2C3_SCL"),
+ STM32_FUNCTION(6, "SPI5_MISO"),
+ STM32_FUNCTION(12, "ETH_MII_RXD3"),
+ STM32_FUNCTION(13, "FMC_SDCKE1"),
+ STM32_FUNCTION(14, "DCMI_D9"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(120, "PH8"),
+ STM32_FUNCTION(0, "GPIOH8"),
+ STM32_FUNCTION(5, "I2C3_SDA"),
+ STM32_FUNCTION(13, "FMC_D16"),
+ STM32_FUNCTION(14, "DCMI_HSYNC"),
+ STM32_FUNCTION(15, "LCD_R2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(121, "PH9"),
+ STM32_FUNCTION(0, "GPIOH9"),
+ STM32_FUNCTION(5, "I2C3_SMBA"),
+ STM32_FUNCTION(10, "TIM12_CH2"),
+ STM32_FUNCTION(13, "FMC_D17"),
+ STM32_FUNCTION(14, "DCMI_D0"),
+ STM32_FUNCTION(15, "LCD_R3"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(122, "PH10"),
+ STM32_FUNCTION(0, "GPIOH10"),
+ STM32_FUNCTION(3, "TIM5_CH1"),
+ STM32_FUNCTION(13, "FMC_D18"),
+ STM32_FUNCTION(14, "DCMI_D1"),
+ STM32_FUNCTION(15, "LCD_R4"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(123, "PH11"),
+ STM32_FUNCTION(0, "GPIOH11"),
+ STM32_FUNCTION(3, "TIM5_CH2"),
+ STM32_FUNCTION(13, "FMC_D19"),
+ STM32_FUNCTION(14, "DCMI_D2"),
+ STM32_FUNCTION(15, "LCD_R5"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(124, "PH12"),
+ STM32_FUNCTION(0, "GPIOH12"),
+ STM32_FUNCTION(3, "TIM5_CH3"),
+ STM32_FUNCTION(13, "FMC_D20"),
+ STM32_FUNCTION(14, "DCMI_D3"),
+ STM32_FUNCTION(15, "LCD_R6"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(125, "PH13"),
+ STM32_FUNCTION(0, "GPIOH13"),
+ STM32_FUNCTION(4, "TIM8_CH1N"),
+ STM32_FUNCTION(10, "CAN1_TX"),
+ STM32_FUNCTION(13, "FMC_D21"),
+ STM32_FUNCTION(15, "LCD_G2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(126, "PH14"),
+ STM32_FUNCTION(0, "GPIOH14"),
+ STM32_FUNCTION(4, "TIM8_CH2N"),
+ STM32_FUNCTION(13, "FMC_D22"),
+ STM32_FUNCTION(14, "DCMI_D4"),
+ STM32_FUNCTION(15, "LCD_G3"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(127, "PH15"),
+ STM32_FUNCTION(0, "GPIOH15"),
+ STM32_FUNCTION(4, "TIM8_CH3N"),
+ STM32_FUNCTION(13, "FMC_D23"),
+ STM32_FUNCTION(14, "DCMI_D11"),
+ STM32_FUNCTION(15, "LCD_G4"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(128, "PI0"),
+ STM32_FUNCTION(0, "GPIOI0"),
+ STM32_FUNCTION(3, "TIM5_CH4"),
+ STM32_FUNCTION(6, "SPI2_NSS I2S2_WS"),
+ STM32_FUNCTION(13, "FMC_D24"),
+ STM32_FUNCTION(14, "DCMI_D13"),
+ STM32_FUNCTION(15, "LCD_G5"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(129, "PI1"),
+ STM32_FUNCTION(0, "GPIOI1"),
+ STM32_FUNCTION(6, "SPI2_SCK I2S2_CK"),
+ STM32_FUNCTION(13, "FMC_D25"),
+ STM32_FUNCTION(14, "DCMI_D8"),
+ STM32_FUNCTION(15, "LCD_G6"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(130, "PI2"),
+ STM32_FUNCTION(0, "GPIOI2"),
+ STM32_FUNCTION(4, "TIM8_CH4"),
+ STM32_FUNCTION(6, "SPI2_MISO"),
+ STM32_FUNCTION(7, "I2S2EXT_SD"),
+ STM32_FUNCTION(13, "FMC_D26"),
+ STM32_FUNCTION(14, "DCMI_D9"),
+ STM32_FUNCTION(15, "LCD_G7"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(131, "PI3"),
+ STM32_FUNCTION(0, "GPIOI3"),
+ STM32_FUNCTION(4, "TIM8_ETR"),
+ STM32_FUNCTION(6, "SPI2_MOSI I2S2_SD"),
+ STM32_FUNCTION(13, "FMC_D27"),
+ STM32_FUNCTION(14, "DCMI_D10"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(132, "PI4"),
+ STM32_FUNCTION(0, "GPIOI4"),
+ STM32_FUNCTION(4, "TIM8_BKIN"),
+ STM32_FUNCTION(13, "FMC_NBL2"),
+ STM32_FUNCTION(14, "DCMI_D5"),
+ STM32_FUNCTION(15, "LCD_B4"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(133, "PI5"),
+ STM32_FUNCTION(0, "GPIOI5"),
+ STM32_FUNCTION(4, "TIM8_CH1"),
+ STM32_FUNCTION(13, "FMC_NBL3"),
+ STM32_FUNCTION(14, "DCMI_VSYNC"),
+ STM32_FUNCTION(15, "LCD_B5"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(134, "PI6"),
+ STM32_FUNCTION(0, "GPIOI6"),
+ STM32_FUNCTION(4, "TIM8_CH2"),
+ STM32_FUNCTION(13, "FMC_D28"),
+ STM32_FUNCTION(14, "DCMI_D6"),
+ STM32_FUNCTION(15, "LCD_B6"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(135, "PI7"),
+ STM32_FUNCTION(0, "GPIOI7"),
+ STM32_FUNCTION(4, "TIM8_CH3"),
+ STM32_FUNCTION(13, "FMC_D29"),
+ STM32_FUNCTION(14, "DCMI_D7"),
+ STM32_FUNCTION(15, "LCD_B7"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(136, "PI8"),
+ STM32_FUNCTION(0, "GPIOI8"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(137, "PI9"),
+ STM32_FUNCTION(0, "GPIOI9"),
+ STM32_FUNCTION(10, "CAN1_RX"),
+ STM32_FUNCTION(13, "FMC_D30"),
+ STM32_FUNCTION(15, "LCD_VSYNC"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(138, "PI10"),
+ STM32_FUNCTION(0, "GPIOI10"),
+ STM32_FUNCTION(12, "ETH_MII_RX_ER"),
+ STM32_FUNCTION(13, "FMC_D31"),
+ STM32_FUNCTION(15, "LCD_HSYNC"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(139, "PI11"),
+ STM32_FUNCTION(0, "GPIOI11"),
+ STM32_FUNCTION(10, "LCD_G6"),
+ STM32_FUNCTION(11, "OTG_HS_ULPI_DIR"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(140, "PI12"),
+ STM32_FUNCTION(0, "GPIOI12"),
+ STM32_FUNCTION(15, "LCD_HSYNC"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(141, "PI13"),
+ STM32_FUNCTION(0, "GPIOI13"),
+ STM32_FUNCTION(15, "LCD_VSYNC"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(142, "PI14"),
+ STM32_FUNCTION(0, "GPIOI14"),
+ STM32_FUNCTION(15, "LCD_CLK"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(143, "PI15"),
+ STM32_FUNCTION(0, "GPIOI15"),
+ STM32_FUNCTION(10, "LCD_G2"),
+ STM32_FUNCTION(15, "LCD_R0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(144, "PJ0"),
+ STM32_FUNCTION(0, "GPIOJ0"),
+ STM32_FUNCTION(10, "LCD_R7"),
+ STM32_FUNCTION(15, "LCD_R1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(145, "PJ1"),
+ STM32_FUNCTION(0, "GPIOJ1"),
+ STM32_FUNCTION(15, "LCD_R2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(146, "PJ2"),
+ STM32_FUNCTION(0, "GPIOJ2"),
+ STM32_FUNCTION(14, "DSIHOST_TE"),
+ STM32_FUNCTION(15, "LCD_R3"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(147, "PJ3"),
+ STM32_FUNCTION(0, "GPIOJ3"),
+ STM32_FUNCTION(15, "LCD_R4"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(148, "PJ4"),
+ STM32_FUNCTION(0, "GPIOJ4"),
+ STM32_FUNCTION(15, "LCD_R5"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(149, "PJ5"),
+ STM32_FUNCTION(0, "GPIOJ5"),
+ STM32_FUNCTION(15, "LCD_R6"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(156, "PJ12"),
+ STM32_FUNCTION(0, "GPIOJ12"),
+ STM32_FUNCTION(10, "LCD_G3"),
+ STM32_FUNCTION(15, "LCD_B0"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(157, "PJ13"),
+ STM32_FUNCTION(0, "GPIOJ13"),
+ STM32_FUNCTION(10, "LCD_G4"),
+ STM32_FUNCTION(15, "LCD_B1"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(158, "PJ14"),
+ STM32_FUNCTION(0, "GPIOJ14"),
+ STM32_FUNCTION(15, "LCD_B2"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(159, "PJ15"),
+ STM32_FUNCTION(0, "GPIOJ15"),
+ STM32_FUNCTION(15, "LCD_B3"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(163, "PK3"),
+ STM32_FUNCTION(0, "GPIOK3"),
+ STM32_FUNCTION(15, "LCD_B4"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(164, "PK4"),
+ STM32_FUNCTION(0, "GPIOK4"),
+ STM32_FUNCTION(15, "LCD_B5"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(165, "PK5"),
+ STM32_FUNCTION(0, "GPIOK5"),
+ STM32_FUNCTION(15, "LCD_B6"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(166, "PK6"),
+ STM32_FUNCTION(0, "GPIOK6"),
+ STM32_FUNCTION(15, "LCD_B7"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+ STM32_PIN(
+ PINCTRL_PIN(167, "PK7"),
+ STM32_FUNCTION(0, "GPIOK7"),
+ STM32_FUNCTION(15, "LCD_DE"),
+ STM32_FUNCTION(16, "EVENTOUT"),
+ STM32_FUNCTION(17, "ANALOG")
+ ),
+};
+
+static struct stm32_pinctrl_match_data stm32f469_match_data = {
+ .pins = stm32f469_pins,
+ .npins = ARRAY_SIZE(stm32f469_pins),
+};
+
+static const struct of_device_id stm32f469_pctrl_match[] = {
+ {
+ .compatible = "st,stm32f469-pinctrl",
+ .data = &stm32f469_match_data,
+ },
+ { }
+};
+
+static struct platform_driver stm32f469_pinctrl_driver = {
+ .probe = stm32_pctl_probe,
+ .driver = {
+ .name = "stm32f469-pinctrl",
+ .of_match_table = stm32f469_pctrl_match,
+ },
+};
+
+static int __init stm32f469_pinctrl_init(void)
+{
+ return platform_driver_register(&stm32f469_pinctrl_driver);
+}
+arch_initcall(stm32f469_pinctrl_init);
diff --git a/drivers/pinctrl/stm32/pinctrl-stm32f746.c b/drivers/pinctrl/stm32/pinctrl-stm32f746.c
index c0b4462..a2fae73 100644
--- a/drivers/pinctrl/stm32/pinctrl-stm32f746.c
+++ b/drivers/pinctrl/stm32/pinctrl-stm32f746.c
@@ -1678,4 +1678,9 @@
.of_match_table = stm32f746_pctrl_match,
},
};
-builtin_platform_driver(stm32f746_pinctrl_driver);
+
+static int __init stm32f746_pinctrl_init(void)
+{
+ return platform_driver_register(&stm32f746_pinctrl_driver);
+}
+arch_initcall(stm32f746_pinctrl_init);
diff --git a/drivers/pinctrl/stm32/pinctrl-stm32h743.c b/drivers/pinctrl/stm32/pinctrl-stm32h743.c
index f7f9eac..e34b2b9 100644
--- a/drivers/pinctrl/stm32/pinctrl-stm32h743.c
+++ b/drivers/pinctrl/stm32/pinctrl-stm32h743.c
@@ -1977,4 +1977,8 @@
},
};
-builtin_platform_driver(stm32h743_pinctrl_driver);
+static int __init stm32h743_pinctrl_init(void)
+{
+ return platform_driver_register(&stm32h743_pinctrl_driver);
+}
+arch_initcall(stm32h743_pinctrl_init);
diff --git a/drivers/pinctrl/sunxi/Kconfig b/drivers/pinctrl/sunxi/Kconfig
index 816015c..793e6f9 100644
--- a/drivers/pinctrl/sunxi/Kconfig
+++ b/drivers/pinctrl/sunxi/Kconfig
@@ -4,6 +4,7 @@
bool
select PINMUX
select GENERIC_PINCONF
+ select GPIOLIB
config PINCTRL_SUN4I_A10
def_bool MACH_SUN4I
@@ -48,8 +49,8 @@
select PINCTRL_SUNXI
config PINCTRL_SUN8I_H3_R
- def_bool MACH_SUN8I
- select PINCTRL_SUNXI_COMMON
+ def_bool MACH_SUN8I || (ARM64 && ARCH_SUNXI)
+ select PINCTRL_SUNXI
config PINCTRL_SUN8I_V3S
def_bool MACH_SUN8I
@@ -65,11 +66,15 @@
select PINCTRL_SUNXI
config PINCTRL_SUN50I_A64
- bool
+ def_bool ARM64 && ARCH_SUNXI
+ select PINCTRL_SUNXI
+
+config PINCTRL_SUN50I_A64_R
+ def_bool ARM64 && ARCH_SUNXI
select PINCTRL_SUNXI
config PINCTRL_SUN50I_H5
- bool
+ def_bool ARM64 && ARCH_SUNXI
select PINCTRL_SUNXI
endif
diff --git a/drivers/pinctrl/sunxi/Makefile b/drivers/pinctrl/sunxi/Makefile
index 04ccb88..df4ccd6 100644
--- a/drivers/pinctrl/sunxi/Makefile
+++ b/drivers/pinctrl/sunxi/Makefile
@@ -11,6 +11,7 @@
obj-$(CONFIG_PINCTRL_SUN8I_A23_R) += pinctrl-sun8i-a23-r.o
obj-$(CONFIG_PINCTRL_SUN8I_A33) += pinctrl-sun8i-a33.o
obj-$(CONFIG_PINCTRL_SUN50I_A64) += pinctrl-sun50i-a64.o
+obj-$(CONFIG_PINCTRL_SUN50I_A64_R) += pinctrl-sun50i-a64-r.o
obj-$(CONFIG_PINCTRL_SUN8I_A83T) += pinctrl-sun8i-a83t.o
obj-$(CONFIG_PINCTRL_SUN8I_H3) += pinctrl-sun8i-h3.o
obj-$(CONFIG_PINCTRL_SUN8I_H3_R) += pinctrl-sun8i-h3-r.o
diff --git a/drivers/pinctrl/sunxi/pinctrl-sun50i-a64-r.c b/drivers/pinctrl/sunxi/pinctrl-sun50i-a64-r.c
new file mode 100644
index 0000000..e69c8da
--- /dev/null
+++ b/drivers/pinctrl/sunxi/pinctrl-sun50i-a64-r.c
@@ -0,0 +1,125 @@
+/*
+ * Allwinner A64 SoCs special pins pinctrl driver.
+ *
+ * Based on pinctrl-sun8i-a23-r.c
+ *
+ * Copyright (C) 2016 Icenowy Zheng
+ * Icenowy Zheng <icenowy@aosc.xyz>
+ *
+ * Copyright (C) 2014 Chen-Yu Tsai
+ * Chen-Yu Tsai <wens@csie.org>
+ *
+ * Copyright (C) 2014 Boris Brezillon
+ * Boris Brezillon <boris.brezillon@free-electrons.com>
+ *
+ * Copyright (C) 2014 Maxime Ripard
+ * Maxime Ripard <maxime.ripard@free-electrons.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2. This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/pinctrl/pinctrl.h>
+#include <linux/platform_device.h>
+#include <linux/reset.h>
+
+#include "pinctrl-sunxi.h"
+
+static const struct sunxi_desc_pin sun50i_a64_r_pins[] = {
+ SUNXI_PIN(SUNXI_PINCTRL_PIN(L, 0),
+ SUNXI_FUNCTION(0x0, "gpio_in"),
+ SUNXI_FUNCTION(0x1, "gpio_out"),
+ SUNXI_FUNCTION(0x2, "s_rsb"), /* SCK */
+ SUNXI_FUNCTION(0x3, "s_i2c"), /* SCK */
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 0, 0)), /* PL_EINT0 */
+ SUNXI_PIN(SUNXI_PINCTRL_PIN(L, 1),
+ SUNXI_FUNCTION(0x0, "gpio_in"),
+ SUNXI_FUNCTION(0x1, "gpio_out"),
+ SUNXI_FUNCTION(0x2, "s_rsb"), /* SDA */
+ SUNXI_FUNCTION(0x3, "s_i2c"), /* SDA */
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 0, 1)), /* PL_EINT1 */
+ SUNXI_PIN(SUNXI_PINCTRL_PIN(L, 2),
+ SUNXI_FUNCTION(0x0, "gpio_in"),
+ SUNXI_FUNCTION(0x1, "gpio_out"),
+ SUNXI_FUNCTION(0x2, "s_uart"), /* TX */
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 0, 2)), /* PL_EINT2 */
+ SUNXI_PIN(SUNXI_PINCTRL_PIN(L, 3),
+ SUNXI_FUNCTION(0x0, "gpio_in"),
+ SUNXI_FUNCTION(0x1, "gpio_out"),
+ SUNXI_FUNCTION(0x2, "s_uart"), /* RX */
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 0, 3)), /* PL_EINT3 */
+ SUNXI_PIN(SUNXI_PINCTRL_PIN(L, 4),
+ SUNXI_FUNCTION(0x0, "gpio_in"),
+ SUNXI_FUNCTION(0x1, "gpio_out"),
+ SUNXI_FUNCTION(0x2, "s_jtag"), /* MS */
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 0, 4)), /* PL_EINT4 */
+ SUNXI_PIN(SUNXI_PINCTRL_PIN(L, 5),
+ SUNXI_FUNCTION(0x0, "gpio_in"),
+ SUNXI_FUNCTION(0x1, "gpio_out"),
+ SUNXI_FUNCTION(0x2, "s_jtag"), /* CK */
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 0, 5)), /* PL_EINT5 */
+ SUNXI_PIN(SUNXI_PINCTRL_PIN(L, 6),
+ SUNXI_FUNCTION(0x0, "gpio_in"),
+ SUNXI_FUNCTION(0x1, "gpio_out"),
+ SUNXI_FUNCTION(0x2, "s_jtag"), /* DO */
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 0, 6)), /* PL_EINT6 */
+ SUNXI_PIN(SUNXI_PINCTRL_PIN(L, 7),
+ SUNXI_FUNCTION(0x0, "gpio_in"),
+ SUNXI_FUNCTION(0x1, "gpio_out"),
+ SUNXI_FUNCTION(0x2, "s_jtag"), /* DI */
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 0, 7)), /* PL_EINT7 */
+ SUNXI_PIN(SUNXI_PINCTRL_PIN(L, 8),
+ SUNXI_FUNCTION(0x0, "gpio_in"),
+ SUNXI_FUNCTION(0x1, "gpio_out"),
+ SUNXI_FUNCTION(0x2, "s_i2c"), /* SCK */
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 0, 8)), /* PL_EINT8 */
+ SUNXI_PIN(SUNXI_PINCTRL_PIN(L, 9),
+ SUNXI_FUNCTION(0x0, "gpio_in"),
+ SUNXI_FUNCTION(0x1, "gpio_out"),
+ SUNXI_FUNCTION(0x2, "s_i2c"), /* SDA */
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 0, 9)), /* PL_EINT9 */
+ SUNXI_PIN(SUNXI_PINCTRL_PIN(L, 10),
+ SUNXI_FUNCTION(0x0, "gpio_in"),
+ SUNXI_FUNCTION(0x1, "gpio_out"),
+ SUNXI_FUNCTION(0x2, "s_pwm"),
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 0, 10)), /* PL_EINT10 */
+ SUNXI_PIN(SUNXI_PINCTRL_PIN(L, 11),
+ SUNXI_FUNCTION(0x0, "gpio_in"),
+ SUNXI_FUNCTION(0x1, "gpio_out"),
+ SUNXI_FUNCTION(0x2, "s_cir_rx"),
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 0, 11)), /* PL_EINT11 */
+ SUNXI_PIN(SUNXI_PINCTRL_PIN(L, 12),
+ SUNXI_FUNCTION(0x0, "gpio_in"),
+ SUNXI_FUNCTION(0x1, "gpio_out"),
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 0, 12)), /* PL_EINT12 */
+};
+
+static const struct sunxi_pinctrl_desc sun50i_a64_r_pinctrl_data = {
+ .pins = sun50i_a64_r_pins,
+ .npins = ARRAY_SIZE(sun50i_a64_r_pins),
+ .pin_base = PL_BASE,
+ .irq_banks = 1,
+};
+
+static int sun50i_a64_r_pinctrl_probe(struct platform_device *pdev)
+{
+ return sunxi_pinctrl_init(pdev,
+ &sun50i_a64_r_pinctrl_data);
+}
+
+static const struct of_device_id sun50i_a64_r_pinctrl_match[] = {
+ { .compatible = "allwinner,sun50i-a64-r-pinctrl", },
+ {}
+};
+
+static struct platform_driver sun50i_a64_r_pinctrl_driver = {
+ .probe = sun50i_a64_r_pinctrl_probe,
+ .driver = {
+ .name = "sun50i-a64-r-pinctrl",
+ .of_match_table = sun50i_a64_r_pinctrl_match,
+ },
+};
+builtin_platform_driver(sun50i_a64_r_pinctrl_driver);
diff --git a/drivers/pinctrl/sunxi/pinctrl-sunxi.c b/drivers/pinctrl/sunxi/pinctrl-sunxi.c
index 60e6e36..58774ac 100644
--- a/drivers/pinctrl/sunxi/pinctrl-sunxi.c
+++ b/drivers/pinctrl/sunxi/pinctrl-sunxi.c
@@ -581,11 +581,11 @@
return -ENOTSUPP;
}
- spin_lock_irqsave(&pctl->lock, flags);
+ raw_spin_lock_irqsave(&pctl->lock, flags);
reg = readl(pctl->membase + offset);
reg &= ~(mask << shift);
writel(reg | val << shift, pctl->membase + offset);
- spin_unlock_irqrestore(&pctl->lock, flags);
+ raw_spin_unlock_irqrestore(&pctl->lock, flags);
} /* for each config */
return 0;
@@ -634,7 +634,7 @@
unsigned long flags;
u32 val, mask;
- spin_lock_irqsave(&pctl->lock, flags);
+ raw_spin_lock_irqsave(&pctl->lock, flags);
pin -= pctl->desc->pin_base;
val = readl(pctl->membase + sunxi_mux_reg(pin));
@@ -642,7 +642,7 @@
writel((val & ~mask) | config << sunxi_mux_offset(pin),
pctl->membase + sunxi_mux_reg(pin));
- spin_unlock_irqrestore(&pctl->lock, flags);
+ raw_spin_unlock_irqrestore(&pctl->lock, flags);
}
static int sunxi_pmx_set_mux(struct pinctrl_dev *pctldev,
@@ -733,7 +733,7 @@
unsigned long flags;
u32 regval;
- spin_lock_irqsave(&pctl->lock, flags);
+ raw_spin_lock_irqsave(&pctl->lock, flags);
regval = readl(pctl->membase + reg);
@@ -744,7 +744,7 @@
writel(regval, pctl->membase + reg);
- spin_unlock_irqrestore(&pctl->lock, flags);
+ raw_spin_unlock_irqrestore(&pctl->lock, flags);
}
static int sunxi_pinctrl_gpio_direction_output(struct gpio_chip *chip,
@@ -856,7 +856,7 @@
return -EINVAL;
}
- spin_lock_irqsave(&pctl->lock, flags);
+ raw_spin_lock_irqsave(&pctl->lock, flags);
if (type & IRQ_TYPE_LEVEL_MASK)
irq_set_chip_handler_name_locked(d, &sunxi_pinctrl_level_irq_chip,
@@ -869,7 +869,7 @@
regval &= ~(IRQ_CFG_IRQ_MASK << index);
writel(regval | (mode << index), pctl->membase + reg);
- spin_unlock_irqrestore(&pctl->lock, flags);
+ raw_spin_unlock_irqrestore(&pctl->lock, flags);
return 0;
}
@@ -893,13 +893,13 @@
unsigned long flags;
u32 val;
- spin_lock_irqsave(&pctl->lock, flags);
+ raw_spin_lock_irqsave(&pctl->lock, flags);
/* Mask the IRQ */
val = readl(pctl->membase + reg);
writel(val & ~(1 << idx), pctl->membase + reg);
- spin_unlock_irqrestore(&pctl->lock, flags);
+ raw_spin_unlock_irqrestore(&pctl->lock, flags);
}
static void sunxi_pinctrl_irq_unmask(struct irq_data *d)
@@ -910,13 +910,13 @@
unsigned long flags;
u32 val;
- spin_lock_irqsave(&pctl->lock, flags);
+ raw_spin_lock_irqsave(&pctl->lock, flags);
/* Unmask the IRQ */
val = readl(pctl->membase + reg);
writel(val | (1 << idx), pctl->membase + reg);
- spin_unlock_irqrestore(&pctl->lock, flags);
+ raw_spin_unlock_irqrestore(&pctl->lock, flags);
}
static void sunxi_pinctrl_irq_ack_unmask(struct irq_data *d)
@@ -1253,7 +1253,7 @@
return -ENOMEM;
platform_set_drvdata(pdev, pctl);
- spin_lock_init(&pctl->lock);
+ raw_spin_lock_init(&pctl->lock);
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
pctl->membase = devm_ioremap_resource(&pdev->dev, res);
diff --git a/drivers/pinctrl/sunxi/pinctrl-sunxi.h b/drivers/pinctrl/sunxi/pinctrl-sunxi.h
index e1aedd2..a9d315a 100644
--- a/drivers/pinctrl/sunxi/pinctrl-sunxi.h
+++ b/drivers/pinctrl/sunxi/pinctrl-sunxi.h
@@ -134,7 +134,7 @@
unsigned ngroups;
int *irq;
unsigned *irq_array;
- spinlock_t lock;
+ raw_spinlock_t lock;
struct pinctrl_dev *pctl_dev;
unsigned long variant;
};
diff --git a/drivers/pinctrl/tegra/pinctrl-tegra-xusb.c b/drivers/pinctrl/tegra/pinctrl-tegra-xusb.c
index 6f68a9e..ebedc2d 100644
--- a/drivers/pinctrl/tegra/pinctrl-tegra-xusb.c
+++ b/drivers/pinctrl/tegra/pinctrl-tegra-xusb.c
@@ -873,6 +873,10 @@
};
MODULE_DEVICE_TABLE(of, tegra_xusb_padctl_of_match);
+/* predeclare these in order to silence sparse */
+int tegra_xusb_padctl_legacy_probe(struct platform_device *pdev);
+int tegra_xusb_padctl_legacy_remove(struct platform_device *pdev);
+
int tegra_xusb_padctl_legacy_probe(struct platform_device *pdev)
{
struct tegra_xusb_padctl *padctl;
diff --git a/drivers/pinctrl/uniphier/Kconfig b/drivers/pinctrl/uniphier/Kconfig
index e077a9e..e5826ea 100644
--- a/drivers/pinctrl/uniphier/Kconfig
+++ b/drivers/pinctrl/uniphier/Kconfig
@@ -9,35 +9,35 @@
if PINCTRL_UNIPHIER
config PINCTRL_UNIPHIER_LD4
- tristate "UniPhier PH1-LD4 SoC pinctrl driver"
+ bool "UniPhier LD4 SoC pinctrl driver"
default ARM
config PINCTRL_UNIPHIER_PRO4
- tristate "UniPhier PH1-Pro4 SoC pinctrl driver"
+ bool "UniPhier Pro4 SoC pinctrl driver"
default ARM
config PINCTRL_UNIPHIER_SLD8
- tristate "UniPhier PH1-sLD8 SoC pinctrl driver"
+ bool "UniPhier sLD8 SoC pinctrl driver"
default ARM
config PINCTRL_UNIPHIER_PRO5
- tristate "UniPhier PH1-Pro5 SoC pinctrl driver"
+ bool "UniPhier Pro5 SoC pinctrl driver"
default ARM
config PINCTRL_UNIPHIER_PXS2
- tristate "UniPhier ProXstream2 SoC pinctrl driver"
+ bool "UniPhier PXs2 SoC pinctrl driver"
default ARM
config PINCTRL_UNIPHIER_LD6B
- tristate "UniPhier PH1-LD6b SoC pinctrl driver"
+ bool "UniPhier LD6b SoC pinctrl driver"
default ARM
config PINCTRL_UNIPHIER_LD11
- tristate "UniPhier PH1-LD11 SoC pinctrl driver"
+ bool "UniPhier LD11 SoC pinctrl driver"
default ARM64
config PINCTRL_UNIPHIER_LD20
- tristate "UniPhier PH1-LD20 SoC pinctrl driver"
+ bool "UniPhier LD20 SoC pinctrl driver"
default ARM64
endif
diff --git a/drivers/pinctrl/uniphier/pinctrl-uniphier-core.c b/drivers/pinctrl/uniphier/pinctrl-uniphier-core.c
index 546f23c..30dec0e 100644
--- a/drivers/pinctrl/uniphier/pinctrl-uniphier-core.c
+++ b/drivers/pinctrl/uniphier/pinctrl-uniphier-core.c
@@ -1,5 +1,6 @@
/*
- * Copyright (C) 2015 Masahiro Yamada <yamada.masahiro@socionext.com>
+ * Copyright (C) 2015-2017 Socionext Inc.
+ * Author: Masahiro Yamada <yamada.masahiro@socionext.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -26,11 +27,18 @@
#include "../pinctrl-utils.h"
#include "pinctrl-uniphier.h"
+#define UNIPHIER_PINCTRL_PINMUX_BASE 0x1000
+#define UNIPHIER_PINCTRL_LOAD_PINMUX 0x1700
+#define UNIPHIER_PINCTRL_DRVCTRL_BASE 0x1800
+#define UNIPHIER_PINCTRL_DRV2CTRL_BASE 0x1900
+#define UNIPHIER_PINCTRL_DRV3CTRL_BASE 0x1980
+#define UNIPHIER_PINCTRL_PUPDCTRL_BASE 0x1a00
+#define UNIPHIER_PINCTRL_IECTRL 0x1d00
+
struct uniphier_pinctrl_priv {
struct pinctrl_desc pctldesc;
struct pinctrl_dev *pctldev;
struct regmap *regmap;
- unsigned int regbase;
struct uniphier_pinctrl_socdata *socdata;
};
@@ -171,7 +179,7 @@
reg = UNIPHIER_PINCTRL_PUPDCTRL_BASE + pupdctrl / 32 * 4;
shift = pupdctrl % 32;
- ret = regmap_read(priv->regmap, priv->regbase + reg, &val);
+ ret = regmap_read(priv->regmap, reg, &val);
if (ret)
return ret;
@@ -231,7 +239,7 @@
shift = drvctrl % 32;
mask = (1U << width) - 1;
- ret = regmap_read(priv->regmap, priv->regbase + reg, &val);
+ ret = regmap_read(priv->regmap, reg, &val);
if (ret)
return ret;
@@ -252,8 +260,7 @@
/* This pin is always input-enabled. */
return 0;
- ret = regmap_read(priv->regmap,
- priv->regbase + UNIPHIER_PINCTRL_IECTRL, &val);
+ ret = regmap_read(priv->regmap, UNIPHIER_PINCTRL_IECTRL, &val);
if (ret)
return ret;
@@ -366,8 +373,7 @@
reg = UNIPHIER_PINCTRL_PUPDCTRL_BASE + pupdctrl / 32 * 4;
shift = pupdctrl % 32;
- return regmap_update_bits(priv->regmap, priv->regbase + reg,
- 1 << shift, val << shift);
+ return regmap_update_bits(priv->regmap, reg, 1 << shift, val << shift);
}
static int uniphier_conf_pin_drive_set(struct pinctrl_dev *pctldev,
@@ -427,7 +433,7 @@
shift = drvctrl % 32;
mask = (1U << width) - 1;
- return regmap_update_bits(priv->regmap, priv->regbase + reg,
+ return regmap_update_bits(priv->regmap, reg,
mask << shift, val << shift);
}
@@ -451,7 +457,7 @@
if (iectrl == UNIPHIER_PIN_IECTRL_NONE)
return enable ? 0 : -EINVAL;
- reg = priv->regbase + UNIPHIER_PINCTRL_IECTRL + iectrl / 32 * 4;
+ reg = UNIPHIER_PINCTRL_IECTRL + iectrl / 32 * 4;
mask = BIT(iectrl % 32);
return regmap_update_bits(priv->regmap, reg, mask, enable ? mask : 0);
@@ -601,7 +607,7 @@
* stored in the offset+4.
*/
for (; reg < reg_end; reg += 4) {
- ret = regmap_update_bits(priv->regmap, priv->regbase + reg,
+ ret = regmap_update_bits(priv->regmap, reg,
mask << shift, muxval << shift);
if (ret)
return ret;
@@ -610,8 +616,7 @@
if (load_pinctrl) {
ret = regmap_write(priv->regmap,
- priv->regbase + UNIPHIER_PINCTRL_LOAD_PINMUX,
- 1);
+ UNIPHIER_PINCTRL_LOAD_PINMUX, 1);
if (ret)
return ret;
}
@@ -698,20 +703,9 @@
if (!priv)
return -ENOMEM;
- if (of_device_is_compatible(dev->of_node, "socionext,ph1-ld4-pinctrl") ||
- of_device_is_compatible(dev->of_node, "socionext,ph1-pro4-pinctrl") ||
- of_device_is_compatible(dev->of_node, "socionext,ph1-sld8-pinctrl") ||
- of_device_is_compatible(dev->of_node, "socionext,ph1-pro5-pinctrl") ||
- of_device_is_compatible(dev->of_node, "socionext,proxstream2-pinctrl") ||
- of_device_is_compatible(dev->of_node, "socionext,ph1-ld6b-pinctrl")) {
- /* old binding */
- priv->regmap = syscon_node_to_regmap(dev->of_node);
- } else {
- priv->regbase = 0x1000;
- parent = of_get_parent(dev->of_node);
- priv->regmap = syscon_node_to_regmap(parent);
- of_node_put(parent);
- }
+ parent = of_get_parent(dev->of_node);
+ priv->regmap = syscon_node_to_regmap(parent);
+ of_node_put(parent);
if (IS_ERR(priv->regmap)) {
dev_err(dev, "failed to get regmap\n");
diff --git a/drivers/pinctrl/uniphier/pinctrl-uniphier-ld11.c b/drivers/pinctrl/uniphier/pinctrl-uniphier-ld11.c
index 83f8864..706effe 100644
--- a/drivers/pinctrl/uniphier/pinctrl-uniphier-ld11.c
+++ b/drivers/pinctrl/uniphier/pinctrl-uniphier-ld11.c
@@ -1,5 +1,5 @@
/*
- * Copyright (C) 2016 Socionext Inc.
+ * Copyright (C) 2016-2017 Socionext Inc.
* Author: Masahiro Yamada <yamada.masahiro@socionext.com>
*
* This program is free software; you can redistribute it and/or modify
@@ -14,7 +14,7 @@
*/
#include <linux/kernel.h>
-#include <linux/module.h>
+#include <linux/init.h>
#include <linux/pinctrl/pinctrl.h>
#include <linux/platform_device.h>
@@ -936,7 +936,6 @@
{ .compatible = "socionext,uniphier-ld11-pinctrl" },
{ /* sentinel */ }
};
-MODULE_DEVICE_TABLE(of, uniphier_ld11_pinctrl_match);
static struct platform_driver uniphier_ld11_pinctrl_driver = {
.probe = uniphier_ld11_pinctrl_probe,
@@ -945,8 +944,4 @@
.of_match_table = uniphier_ld11_pinctrl_match,
},
};
-module_platform_driver(uniphier_ld11_pinctrl_driver);
-
-MODULE_AUTHOR("Masahiro Yamada <yamada.masahiro@socionext.com>");
-MODULE_DESCRIPTION("UniPhier PH1-LD11 pinctrl driver");
-MODULE_LICENSE("GPL");
+builtin_platform_driver(uniphier_ld11_pinctrl_driver);
diff --git a/drivers/pinctrl/uniphier/pinctrl-uniphier-ld20.c b/drivers/pinctrl/uniphier/pinctrl-uniphier-ld20.c
index 9668633..c8d18a2 100644
--- a/drivers/pinctrl/uniphier/pinctrl-uniphier-ld20.c
+++ b/drivers/pinctrl/uniphier/pinctrl-uniphier-ld20.c
@@ -1,5 +1,5 @@
/*
- * Copyright (C) 2016 Socionext Inc.
+ * Copyright (C) 2016-2017 Socionext Inc.
* Author: Masahiro Yamada <yamada.masahiro@socionext.com>
*
* This program is free software; you can redistribute it and/or modify
@@ -14,7 +14,7 @@
*/
#include <linux/kernel.h>
-#include <linux/module.h>
+#include <linux/init.h>
#include <linux/pinctrl/pinctrl.h>
#include <linux/platform_device.h>
@@ -1034,7 +1034,6 @@
{ .compatible = "socionext,uniphier-ld20-pinctrl" },
{ /* sentinel */ }
};
-MODULE_DEVICE_TABLE(of, uniphier_ld20_pinctrl_match);
static struct platform_driver uniphier_ld20_pinctrl_driver = {
.probe = uniphier_ld20_pinctrl_probe,
@@ -1043,8 +1042,4 @@
.of_match_table = uniphier_ld20_pinctrl_match,
},
};
-module_platform_driver(uniphier_ld20_pinctrl_driver);
-
-MODULE_AUTHOR("Masahiro Yamada <yamada.masahiro@socionext.com>");
-MODULE_DESCRIPTION("UniPhier PH1-LD20 pinctrl driver");
-MODULE_LICENSE("GPL");
+builtin_platform_driver(uniphier_ld20_pinctrl_driver);
diff --git a/drivers/pinctrl/uniphier/pinctrl-uniphier-ld4.c b/drivers/pinctrl/uniphier/pinctrl-uniphier-ld4.c
index 3edfb6f..8f2ad1c 100644
--- a/drivers/pinctrl/uniphier/pinctrl-uniphier-ld4.c
+++ b/drivers/pinctrl/uniphier/pinctrl-uniphier-ld4.c
@@ -1,5 +1,6 @@
/*
- * Copyright (C) 2015 Masahiro Yamada <yamada.masahiro@socionext.com>
+ * Copyright (C) 2015-2017 Socionext Inc.
+ * Author: Masahiro Yamada <yamada.masahiro@socionext.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -13,7 +14,7 @@
*/
#include <linux/kernel.h>
-#include <linux/module.h>
+#include <linux/init.h>
#include <linux/pinctrl/pinctrl.h>
#include <linux/platform_device.h>
@@ -929,10 +930,8 @@
static const struct of_device_id uniphier_ld4_pinctrl_match[] = {
{ .compatible = "socionext,uniphier-ld4-pinctrl" },
- { .compatible = "socionext,ph1-ld4-pinctrl" },
{ /* sentinel */ }
};
-MODULE_DEVICE_TABLE(of, uniphier_ld4_pinctrl_match);
static struct platform_driver uniphier_ld4_pinctrl_driver = {
.probe = uniphier_ld4_pinctrl_probe,
@@ -941,8 +940,4 @@
.of_match_table = uniphier_ld4_pinctrl_match,
},
};
-module_platform_driver(uniphier_ld4_pinctrl_driver);
-
-MODULE_AUTHOR("Masahiro Yamada <yamada.masahiro@socionext.com>");
-MODULE_DESCRIPTION("UniPhier PH1-LD4 pinctrl driver");
-MODULE_LICENSE("GPL");
+builtin_platform_driver(uniphier_ld4_pinctrl_driver);
diff --git a/drivers/pinctrl/uniphier/pinctrl-uniphier-ld6b.c b/drivers/pinctrl/uniphier/pinctrl-uniphier-ld6b.c
index 708e510..8a0da93 100644
--- a/drivers/pinctrl/uniphier/pinctrl-uniphier-ld6b.c
+++ b/drivers/pinctrl/uniphier/pinctrl-uniphier-ld6b.c
@@ -1,5 +1,6 @@
/*
- * Copyright (C) 2015 Masahiro Yamada <yamada.masahiro@socionext.com>
+ * Copyright (C) 2015-2017 Socionext Inc.
+ * Author: Masahiro Yamada <yamada.masahiro@socionext.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -13,7 +14,7 @@
*/
#include <linux/kernel.h>
-#include <linux/module.h>
+#include <linux/init.h>
#include <linux/pinctrl/pinctrl.h>
#include <linux/platform_device.h>
@@ -1290,10 +1291,8 @@
static const struct of_device_id uniphier_ld6b_pinctrl_match[] = {
{ .compatible = "socionext,uniphier-ld6b-pinctrl" },
- { .compatible = "socionext,ph1-ld6b-pinctrl" },
{ /* sentinel */ }
};
-MODULE_DEVICE_TABLE(of, uniphier_ld6b_pinctrl_match);
static struct platform_driver uniphier_ld6b_pinctrl_driver = {
.probe = uniphier_ld6b_pinctrl_probe,
@@ -1302,8 +1301,4 @@
.of_match_table = uniphier_ld6b_pinctrl_match,
},
};
-module_platform_driver(uniphier_ld6b_pinctrl_driver);
-
-MODULE_AUTHOR("Masahiro Yamada <yamada.masahiro@socionext.com>");
-MODULE_DESCRIPTION("UniPhier PH1-LD6b pinctrl driver");
-MODULE_LICENSE("GPL");
+builtin_platform_driver(uniphier_ld6b_pinctrl_driver);
diff --git a/drivers/pinctrl/uniphier/pinctrl-uniphier-pro4.c b/drivers/pinctrl/uniphier/pinctrl-uniphier-pro4.c
index c306e84..a433a30 100644
--- a/drivers/pinctrl/uniphier/pinctrl-uniphier-pro4.c
+++ b/drivers/pinctrl/uniphier/pinctrl-uniphier-pro4.c
@@ -1,5 +1,6 @@
/*
- * Copyright (C) 2015 Masahiro Yamada <yamada.masahiro@socionext.com>
+ * Copyright (C) 2015-2017 Socionext Inc.
+ * Author: Masahiro Yamada <yamada.masahiro@socionext.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -13,7 +14,7 @@
*/
#include <linux/kernel.h>
-#include <linux/module.h>
+#include <linux/init.h>
#include <linux/pinctrl/pinctrl.h>
#include <linux/platform_device.h>
@@ -1600,10 +1601,8 @@
static const struct of_device_id uniphier_pro4_pinctrl_match[] = {
{ .compatible = "socionext,uniphier-pro4-pinctrl" },
- { .compatible = "socionext,ph1-pro4-pinctrl" },
{ /* sentinel */ }
};
-MODULE_DEVICE_TABLE(of, uniphier_pro4_pinctrl_match);
static struct platform_driver uniphier_pro4_pinctrl_driver = {
.probe = uniphier_pro4_pinctrl_probe,
@@ -1612,8 +1611,4 @@
.of_match_table = uniphier_pro4_pinctrl_match,
},
};
-module_platform_driver(uniphier_pro4_pinctrl_driver);
-
-MODULE_AUTHOR("Masahiro Yamada <yamada.masahiro@socionext.com>");
-MODULE_DESCRIPTION("UniPhier PH1-Pro4 pinctrl driver");
-MODULE_LICENSE("GPL");
+builtin_platform_driver(uniphier_pro4_pinctrl_driver);
diff --git a/drivers/pinctrl/uniphier/pinctrl-uniphier-pro5.c b/drivers/pinctrl/uniphier/pinctrl-uniphier-pro5.c
index 55d4a12..04d00c3 100644
--- a/drivers/pinctrl/uniphier/pinctrl-uniphier-pro5.c
+++ b/drivers/pinctrl/uniphier/pinctrl-uniphier-pro5.c
@@ -1,5 +1,6 @@
/*
- * Copyright (C) 2015 Masahiro Yamada <yamada.masahiro@socionext.com>
+ * Copyright (C) 2015-2017 Socionext Inc.
+ * Author: Masahiro Yamada <yamada.masahiro@socionext.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -13,7 +14,7 @@
*/
#include <linux/kernel.h>
-#include <linux/module.h>
+#include <linux/init.h>
#include <linux/pinctrl/pinctrl.h>
#include <linux/platform_device.h>
@@ -1365,10 +1366,8 @@
static const struct of_device_id uniphier_pro5_pinctrl_match[] = {
{ .compatible = "socionext,uniphier-pro5-pinctrl" },
- { .compatible = "socionext,ph1-pro5-pinctrl" },
{ /* sentinel */ }
};
-MODULE_DEVICE_TABLE(of, uniphier_pro5_pinctrl_match);
static struct platform_driver uniphier_pro5_pinctrl_driver = {
.probe = uniphier_pro5_pinctrl_probe,
@@ -1377,8 +1376,4 @@
.of_match_table = uniphier_pro5_pinctrl_match,
},
};
-module_platform_driver(uniphier_pro5_pinctrl_driver);
-
-MODULE_AUTHOR("Masahiro Yamada <yamada.masahiro@socionext.com>");
-MODULE_DESCRIPTION("UniPhier PH1-Pro5 pinctrl driver");
-MODULE_LICENSE("GPL");
+builtin_platform_driver(uniphier_pro5_pinctrl_driver);
diff --git a/drivers/pinctrl/uniphier/pinctrl-uniphier-pxs2.c b/drivers/pinctrl/uniphier/pinctrl-uniphier-pxs2.c
index 85ca5e2..53b6b77 100644
--- a/drivers/pinctrl/uniphier/pinctrl-uniphier-pxs2.c
+++ b/drivers/pinctrl/uniphier/pinctrl-uniphier-pxs2.c
@@ -1,5 +1,6 @@
/*
- * Copyright (C) 2015 Masahiro Yamada <yamada.masahiro@socionext.com>
+ * Copyright (C) 2015-2017 Socionext Inc.
+ * Author: Masahiro Yamada <yamada.masahiro@socionext.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -13,7 +14,7 @@
*/
#include <linux/kernel.h>
-#include <linux/module.h>
+#include <linux/init.h>
#include <linux/pinctrl/pinctrl.h>
#include <linux/platform_device.h>
@@ -1277,10 +1278,8 @@
static const struct of_device_id uniphier_pxs2_pinctrl_match[] = {
{ .compatible = "socionext,uniphier-pxs2-pinctrl" },
- { .compatible = "socionext,proxstream2-pinctrl" },
{ /* sentinel */ }
};
-MODULE_DEVICE_TABLE(of, uniphier_pxs2_pinctrl_match);
static struct platform_driver uniphier_pxs2_pinctrl_driver = {
.probe = uniphier_pxs2_pinctrl_probe,
@@ -1289,8 +1288,4 @@
.of_match_table = uniphier_pxs2_pinctrl_match,
},
};
-module_platform_driver(uniphier_pxs2_pinctrl_driver);
-
-MODULE_AUTHOR("Masahiro Yamada <yamada.masahiro@socionext.com>");
-MODULE_DESCRIPTION("UniPhier ProXstream2 pinctrl driver");
-MODULE_LICENSE("GPL");
+builtin_platform_driver(uniphier_pxs2_pinctrl_driver);
diff --git a/drivers/pinctrl/uniphier/pinctrl-uniphier-sld8.c b/drivers/pinctrl/uniphier/pinctrl-uniphier-sld8.c
index da689d8..37deaf6 100644
--- a/drivers/pinctrl/uniphier/pinctrl-uniphier-sld8.c
+++ b/drivers/pinctrl/uniphier/pinctrl-uniphier-sld8.c
@@ -1,5 +1,6 @@
/*
- * Copyright (C) 2015 Masahiro Yamada <yamada.masahiro@socionext.com>
+ * Copyright (C) 2015-2017 Socionext Inc.
+ * Author: Masahiro Yamada <yamada.masahiro@socionext.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -13,7 +14,7 @@
*/
#include <linux/kernel.h>
-#include <linux/module.h>
+#include <linux/init.h>
#include <linux/pinctrl/pinctrl.h>
#include <linux/platform_device.h>
@@ -856,10 +857,8 @@
static const struct of_device_id uniphier_sld8_pinctrl_match[] = {
{ .compatible = "socionext,uniphier-sld8-pinctrl" },
- { .compatible = "socionext,ph1-sld8-pinctrl" },
{ /* sentinel */ }
};
-MODULE_DEVICE_TABLE(of, uniphier_sld8_pinctrl_match);
static struct platform_driver uniphier_sld8_pinctrl_driver = {
.probe = uniphier_sld8_pinctrl_probe,
@@ -868,8 +867,4 @@
.of_match_table = uniphier_sld8_pinctrl_match,
},
};
-module_platform_driver(uniphier_sld8_pinctrl_driver);
-
-MODULE_AUTHOR("Masahiro Yamada <yamada.masahiro@socionext.com>");
-MODULE_DESCRIPTION("UniPhier PH1-sLD8 pinctrl driver");
-MODULE_LICENSE("GPL");
+builtin_platform_driver(uniphier_sld8_pinctrl_driver);
diff --git a/drivers/pinctrl/uniphier/pinctrl-uniphier.h b/drivers/pinctrl/uniphier/pinctrl-uniphier.h
index 923f36c..6f2f33b 100644
--- a/drivers/pinctrl/uniphier/pinctrl-uniphier.h
+++ b/drivers/pinctrl/uniphier/pinctrl-uniphier.h
@@ -1,5 +1,6 @@
/*
- * Copyright (C) 2015 Masahiro Yamada <yamada.masahiro@socionext.com>
+ * Copyright (C) 2015-2017 Socionext Inc.
+ * Author: Masahiro Yamada <yamada.masahiro@socionext.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -22,14 +23,6 @@
struct platform_device;
-#define UNIPHIER_PINCTRL_PINMUX_BASE 0x0
-#define UNIPHIER_PINCTRL_LOAD_PINMUX 0x700
-#define UNIPHIER_PINCTRL_DRVCTRL_BASE 0x800
-#define UNIPHIER_PINCTRL_DRV2CTRL_BASE 0x900
-#define UNIPHIER_PINCTRL_DRV3CTRL_BASE 0x980
-#define UNIPHIER_PINCTRL_PUPDCTRL_BASE 0xa00
-#define UNIPHIER_PINCTRL_IECTRL 0xd00
-
/* input enable control register bit */
#define UNIPHIER_PIN_IECTRL_SHIFT 0
#define UNIPHIER_PIN_IECTRL_BITS 8
diff --git a/drivers/platform/x86/Kconfig b/drivers/platform/x86/Kconfig
index 4bc88eb..e1bffc9 100644
--- a/drivers/platform/x86/Kconfig
+++ b/drivers/platform/x86/Kconfig
@@ -141,6 +141,14 @@
To compile this driver as a module, choose M here: the module will
be called dell-wmi-aio.
+config DELL_WMI_LED
+ tristate "External LED on Dell Business Netbooks"
+ depends on LEDS_CLASS
+ depends on ACPI_WMI
+ help
+ This adds support for the Latitude 2100 and similar
+ notebooks that have an external LED.
+
config DELL_SMO8800
tristate "Dell Latitude freefall driver (ACPI SMO88XX)"
depends on ACPI
diff --git a/drivers/platform/x86/Makefile b/drivers/platform/x86/Makefile
index 299d0f9..776b3a7 100644
--- a/drivers/platform/x86/Makefile
+++ b/drivers/platform/x86/Makefile
@@ -15,6 +15,7 @@
obj-$(CONFIG_DELL_LAPTOP) += dell-laptop.o
obj-$(CONFIG_DELL_WMI) += dell-wmi.o
obj-$(CONFIG_DELL_WMI_AIO) += dell-wmi-aio.o
+obj-$(CONFIG_DELL_WMI_LED) += dell-wmi-led.o
obj-$(CONFIG_DELL_SMO8800) += dell-smo8800.o
obj-$(CONFIG_DELL_RBTN) += dell-rbtn.o
obj-$(CONFIG_ACER_WMI) += acer-wmi.o
diff --git a/drivers/platform/x86/dell-laptop.c b/drivers/platform/x86/dell-laptop.c
index f57dd28..2e237ba 100644
--- a/drivers/platform/x86/dell-laptop.c
+++ b/drivers/platform/x86/dell-laptop.c
@@ -29,6 +29,7 @@
#include <linux/mm.h>
#include <linux/i8042.h>
#include <linux/debugfs.h>
+#include <linux/dell-led.h>
#include <linux/seq_file.h>
#include <acpi/video.h>
#include "dell-rbtn.h"
@@ -42,6 +43,8 @@
#define KBD_LED_AUTO_50_TOKEN 0x02EB
#define KBD_LED_AUTO_75_TOKEN 0x02EC
#define KBD_LED_AUTO_100_TOKEN 0x02F6
+#define GLOBAL_MIC_MUTE_ENABLE 0x0364
+#define GLOBAL_MIC_MUTE_DISABLE 0x0365
struct quirk_entry {
u8 touchpad_led;
@@ -1978,6 +1981,31 @@
led_classdev_unregister(&kbd_led);
}
+int dell_micmute_led_set(int state)
+{
+ struct calling_interface_buffer *buffer;
+ struct calling_interface_token *token;
+
+ if (state == 0)
+ token = dell_smbios_find_token(GLOBAL_MIC_MUTE_DISABLE);
+ else if (state == 1)
+ token = dell_smbios_find_token(GLOBAL_MIC_MUTE_ENABLE);
+ else
+ return -EINVAL;
+
+ if (!token)
+ return -ENODEV;
+
+ buffer = dell_smbios_get_buffer();
+ buffer->input[0] = token->location;
+ buffer->input[1] = token->value;
+ dell_smbios_send_request(1, 0);
+ dell_smbios_release_buffer();
+
+ return state;
+}
+EXPORT_SYMBOL_GPL(dell_micmute_led_set);
+
static int __init dell_init(void)
{
struct calling_interface_buffer *buffer;
diff --git a/drivers/platform/x86/dell-wmi-led.c b/drivers/platform/x86/dell-wmi-led.c
new file mode 100644
index 0000000..a0c7e99
--- /dev/null
+++ b/drivers/platform/x86/dell-wmi-led.c
@@ -0,0 +1,186 @@
+/*
+ * Copyright (C) 2010 Dell Inc.
+ * Louis Davis <louis_davis@dell.com>
+ * Jim Dailey <jim_dailey@dell.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ */
+
+#include <linux/acpi.h>
+#include <linux/leds.h>
+#include <linux/slab.h>
+#include <linux/module.h>
+
+MODULE_AUTHOR("Louis Davis/Jim Dailey");
+MODULE_DESCRIPTION("Dell LED Control Driver");
+MODULE_LICENSE("GPL");
+
+#define DELL_LED_BIOS_GUID "F6E4FE6E-909D-47cb-8BAB-C9F6F2F8D396"
+MODULE_ALIAS("wmi:" DELL_LED_BIOS_GUID);
+
+/* Error Result Codes: */
+#define INVALID_DEVICE_ID 250
+#define INVALID_PARAMETER 251
+#define INVALID_BUFFER 252
+#define INTERFACE_ERROR 253
+#define UNSUPPORTED_COMMAND 254
+#define UNSPECIFIED_ERROR 255
+
+/* Device ID */
+#define DEVICE_ID_PANEL_BACK 1
+
+/* LED Commands */
+#define CMD_LED_ON 16
+#define CMD_LED_OFF 17
+#define CMD_LED_BLINK 18
+
+struct bios_args {
+ unsigned char length;
+ unsigned char result_code;
+ unsigned char device_id;
+ unsigned char command;
+ unsigned char on_time;
+ unsigned char off_time;
+};
+
+static int dell_led_perform_fn(u8 length, u8 result_code, u8 device_id,
+ u8 command, u8 on_time, u8 off_time)
+{
+ struct acpi_buffer output = { ACPI_ALLOCATE_BUFFER, NULL };
+ struct bios_args *bios_return;
+ struct acpi_buffer input;
+ union acpi_object *obj;
+ acpi_status status;
+ u8 return_code;
+
+ struct bios_args args = {
+ .length = length,
+ .result_code = result_code,
+ .device_id = device_id,
+ .command = command,
+ .on_time = on_time,
+ .off_time = off_time
+ };
+
+ input.length = sizeof(struct bios_args);
+ input.pointer = &args;
+
+ status = wmi_evaluate_method(DELL_LED_BIOS_GUID, 1, 1, &input, &output);
+ if (ACPI_FAILURE(status))
+ return status;
+
+ obj = output.pointer;
+
+ if (!obj)
+ return -EINVAL;
+ if (obj->type != ACPI_TYPE_BUFFER) {
+ kfree(obj);
+ return -EINVAL;
+ }
+
+ bios_return = ((struct bios_args *)obj->buffer.pointer);
+ return_code = bios_return->result_code;
+
+ kfree(obj);
+
+ return return_code;
+}
+
+static int led_on(void)
+{
+ return dell_led_perform_fn(3, /* Length of command */
+ INTERFACE_ERROR, /* Init to INTERFACE_ERROR */
+ DEVICE_ID_PANEL_BACK, /* Device ID */
+ CMD_LED_ON, /* Command */
+ 0, /* not used */
+ 0); /* not used */
+}
+
+static int led_off(void)
+{
+ return dell_led_perform_fn(3, /* Length of command */
+ INTERFACE_ERROR, /* Init to INTERFACE_ERROR */
+ DEVICE_ID_PANEL_BACK, /* Device ID */
+ CMD_LED_OFF, /* Command */
+ 0, /* not used */
+ 0); /* not used */
+}
+
+static int led_blink(unsigned char on_eighths, unsigned char off_eighths)
+{
+ return dell_led_perform_fn(5, /* Length of command */
+ INTERFACE_ERROR, /* Init to INTERFACE_ERROR */
+ DEVICE_ID_PANEL_BACK, /* Device ID */
+ CMD_LED_BLINK, /* Command */
+ on_eighths, /* blink on in eigths of a second */
+ off_eighths); /* blink off in eights of a second */
+}
+
+static void dell_led_set(struct led_classdev *led_cdev,
+ enum led_brightness value)
+{
+ if (value == LED_OFF)
+ led_off();
+ else
+ led_on();
+}
+
+static int dell_led_blink(struct led_classdev *led_cdev,
+ unsigned long *delay_on, unsigned long *delay_off)
+{
+ unsigned long on_eighths;
+ unsigned long off_eighths;
+
+ /*
+ * The Dell LED delay is based on 125ms intervals.
+ * Need to round up to next interval.
+ */
+
+ on_eighths = DIV_ROUND_UP(*delay_on, 125);
+ on_eighths = clamp_t(unsigned long, on_eighths, 1, 255);
+ *delay_on = on_eighths * 125;
+
+ off_eighths = DIV_ROUND_UP(*delay_off, 125);
+ off_eighths = clamp_t(unsigned long, off_eighths, 1, 255);
+ *delay_off = off_eighths * 125;
+
+ led_blink(on_eighths, off_eighths);
+
+ return 0;
+}
+
+static struct led_classdev dell_led = {
+ .name = "dell::lid",
+ .brightness = LED_OFF,
+ .max_brightness = 1,
+ .brightness_set = dell_led_set,
+ .blink_set = dell_led_blink,
+ .flags = LED_CORE_SUSPENDRESUME,
+};
+
+static int __init dell_led_init(void)
+{
+ int error = 0;
+
+ if (!wmi_has_guid(DELL_LED_BIOS_GUID))
+ return -ENODEV;
+
+ error = led_off();
+ if (error != 0)
+ return -ENODEV;
+
+ return led_classdev_register(NULL, &dell_led);
+}
+
+static void __exit dell_led_exit(void)
+{
+ led_classdev_unregister(&dell_led);
+
+ led_off();
+}
+
+module_init(dell_led_init);
+module_exit(dell_led_exit);
diff --git a/drivers/pnp/pnpbios/bioscalls.c b/drivers/pnp/pnpbios/bioscalls.c
index 438d4c7..ff563db 100644
--- a/drivers/pnp/pnpbios/bioscalls.c
+++ b/drivers/pnp/pnpbios/bioscalls.c
@@ -54,7 +54,7 @@
#define Q2_SET_SEL(cpu, selname, address, size) \
do { \
- struct desc_struct *gdt = get_cpu_gdt_table((cpu)); \
+ struct desc_struct *gdt = get_cpu_gdt_rw((cpu)); \
set_desc_base(&gdt[(selname) >> 3], (u32)(address)); \
set_desc_limit(&gdt[(selname) >> 3], (size) - 1); \
} while(0)
@@ -95,8 +95,8 @@
return PNP_FUNCTION_NOT_SUPPORTED;
cpu = get_cpu();
- save_desc_40 = get_cpu_gdt_table(cpu)[0x40 / 8];
- get_cpu_gdt_table(cpu)[0x40 / 8] = bad_bios_desc;
+ save_desc_40 = get_cpu_gdt_rw(cpu)[0x40 / 8];
+ get_cpu_gdt_rw(cpu)[0x40 / 8] = bad_bios_desc;
/* On some boxes IRQ's during PnP BIOS calls are deadly. */
spin_lock_irqsave(&pnp_bios_lock, flags);
@@ -134,7 +134,7 @@
:"memory");
spin_unlock_irqrestore(&pnp_bios_lock, flags);
- get_cpu_gdt_table(cpu)[0x40 / 8] = save_desc_40;
+ get_cpu_gdt_rw(cpu)[0x40 / 8] = save_desc_40;
put_cpu();
/* If we get here and this is set then the PnP BIOS faulted on us. */
@@ -477,7 +477,7 @@
pnp_bios_callpoint.segment = PNP_CS16;
for_each_possible_cpu(i) {
- struct desc_struct *gdt = get_cpu_gdt_table(i);
+ struct desc_struct *gdt = get_cpu_gdt_rw(i);
if (!gdt)
continue;
set_desc_base(&gdt[GDT_ENTRY_PNPBIOS_CS32],
diff --git a/drivers/power/avs/rockchip-io-domain.c b/drivers/power/avs/rockchip-io-domain.c
index 56bce19..8581252 100644
--- a/drivers/power/avs/rockchip-io-domain.c
+++ b/drivers/power/avs/rockchip-io-domain.c
@@ -43,6 +43,10 @@
#define RK3288_SOC_CON2_FLASH0 BIT(7)
#define RK3288_SOC_FLASH_SUPPLY_NUM 2
+#define RK3328_SOC_CON4 0x410
+#define RK3328_SOC_CON4_VCCIO2 BIT(7)
+#define RK3328_SOC_VCCIO2_SUPPLY_NUM 1
+
#define RK3368_SOC_CON15 0x43c
#define RK3368_SOC_CON15_FLASH0 BIT(14)
#define RK3368_SOC_FLASH_SUPPLY_NUM 2
@@ -166,6 +170,25 @@
dev_warn(iod->dev, "couldn't update flash0 ctrl\n");
}
+static void rk3328_iodomain_init(struct rockchip_iodomain *iod)
+{
+ int ret;
+ u32 val;
+
+ /* if no vccio2 supply we should leave things alone */
+ if (!iod->supplies[RK3328_SOC_VCCIO2_SUPPLY_NUM].reg)
+ return;
+
+ /*
+ * set vccio2 iodomain to also use this framework
+ * instead of a special gpio.
+ */
+ val = RK3328_SOC_CON4_VCCIO2 | (RK3328_SOC_CON4_VCCIO2 << 16);
+ ret = regmap_write(iod->grf, RK3328_SOC_CON4, val);
+ if (ret < 0)
+ dev_warn(iod->dev, "couldn't update vccio2 vsel ctrl\n");
+}
+
static void rk3368_iodomain_init(struct rockchip_iodomain *iod)
{
int ret;
@@ -247,6 +270,20 @@
.init = rk3288_iodomain_init,
};
+static const struct rockchip_iodomain_soc_data soc_data_rk3328 = {
+ .grf_offset = 0x410,
+ .supply_names = {
+ "vccio1",
+ "vccio2",
+ "vccio3",
+ "vccio4",
+ "vccio5",
+ "vccio6",
+ "pmuio",
+ },
+ .init = rk3328_iodomain_init,
+};
+
static const struct rockchip_iodomain_soc_data soc_data_rk3368 = {
.grf_offset = 0x900,
.supply_names = {
@@ -312,6 +349,10 @@
.data = (void *)&soc_data_rk3288
},
{
+ .compatible = "rockchip,rk3328-io-voltage-domain",
+ .data = (void *)&soc_data_rk3328
+ },
+ {
.compatible = "rockchip,rk3368-io-voltage-domain",
.data = (void *)&soc_data_rk3368
},
diff --git a/drivers/power/reset/Kconfig b/drivers/power/reset/Kconfig
index b8caccc..13f1714 100644
--- a/drivers/power/reset/Kconfig
+++ b/drivers/power/reset/Kconfig
@@ -67,6 +67,15 @@
Say Y here if you have a Broadcom STB board and you wish
to have restart support.
+config POWER_RESET_GEMINI_POWEROFF
+ bool "Cortina Gemini power-off driver"
+ depends on ARCH_GEMINI || COMPILE_TEST
+ depends on OF && HAS_IOMEM
+ default ARCH_GEMINI
+ help
+ This driver supports turning off the Cortina Gemini SoC.
+ Select this if you're building a kernel with Gemini SoC support.
+
config POWER_RESET_GPIO
bool "GPIO power-off driver"
depends on OF_GPIO
diff --git a/drivers/power/reset/Makefile b/drivers/power/reset/Makefile
index 11dae3b..58cf5b3 100644
--- a/drivers/power/reset/Makefile
+++ b/drivers/power/reset/Makefile
@@ -5,6 +5,7 @@
obj-$(CONFIG_POWER_RESET_AXXIA) += axxia-reset.o
obj-$(CONFIG_POWER_RESET_BRCMKONA) += brcm-kona-reset.o
obj-$(CONFIG_POWER_RESET_BRCMSTB) += brcmstb-reboot.o
+obj-$(CONFIG_POWER_RESET_GEMINI_POWEROFF) += gemini-poweroff.o
obj-$(CONFIG_POWER_RESET_GPIO) += gpio-poweroff.o
obj-$(CONFIG_POWER_RESET_GPIO_RESTART) += gpio-restart.o
obj-$(CONFIG_POWER_RESET_HISI) += hisi-reboot.o
diff --git a/drivers/power/reset/gemini-poweroff.c b/drivers/power/reset/gemini-poweroff.c
new file mode 100644
index 0000000..de878fd
--- /dev/null
+++ b/drivers/power/reset/gemini-poweroff.c
@@ -0,0 +1,160 @@
+/*
+ * Gemini power management controller
+ * Copyright (C) 2017 Linus Walleij <linus.walleij@linaro.org>
+ *
+ * Inspired by code from the SL3516 board support by Jason Lee
+ * Inspired by code from Janos Laube <janos.dev@gmail.com>
+ */
+#include <linux/of.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+#include <linux/pm.h>
+#include <linux/bitops.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/reboot.h>
+
+#define GEMINI_PWC_ID 0x00010500
+#define GEMINI_PWC_IDREG 0x00
+#define GEMINI_PWC_CTRLREG 0x04
+#define GEMINI_PWC_STATREG 0x08
+
+#define GEMINI_CTRL_SHUTDOWN BIT(0)
+#define GEMINI_CTRL_ENABLE BIT(1)
+#define GEMINI_CTRL_IRQ_CLR BIT(2)
+
+#define GEMINI_STAT_CIR BIT(4)
+#define GEMINI_STAT_RTC BIT(5)
+#define GEMINI_STAT_POWERBUTTON BIT(6)
+
+struct gemini_powercon {
+ struct device *dev;
+ void __iomem *base;
+};
+
+static irqreturn_t gemini_powerbutton_interrupt(int irq, void *data)
+{
+ struct gemini_powercon *gpw = data;
+ u32 val;
+
+ /* ACK the IRQ */
+ val = readl(gpw->base + GEMINI_PWC_CTRLREG);
+ val |= GEMINI_CTRL_IRQ_CLR;
+ writel(val, gpw->base + GEMINI_PWC_CTRLREG);
+
+ val = readl(gpw->base + GEMINI_PWC_STATREG);
+ val &= 0x70U;
+ switch (val) {
+ case GEMINI_STAT_CIR:
+ dev_info(gpw->dev, "infrared poweroff\n");
+ orderly_poweroff(true);
+ break;
+ case GEMINI_STAT_RTC:
+ dev_info(gpw->dev, "RTC poweroff\n");
+ orderly_poweroff(true);
+ break;
+ case GEMINI_STAT_POWERBUTTON:
+ dev_info(gpw->dev, "poweroff button pressed\n");
+ orderly_poweroff(true);
+ break;
+ default:
+ dev_info(gpw->dev, "other power management IRQ\n");
+ break;
+ }
+
+ return IRQ_HANDLED;
+}
+
+/* This callback needs this static local as it has void as argument */
+static struct gemini_powercon *gpw_poweroff;
+
+static void gemini_poweroff(void)
+{
+ struct gemini_powercon *gpw = gpw_poweroff;
+ u32 val;
+
+ dev_crit(gpw->dev, "Gemini power off\n");
+ val = readl(gpw->base + GEMINI_PWC_CTRLREG);
+ val |= GEMINI_CTRL_ENABLE | GEMINI_CTRL_IRQ_CLR;
+ writel(val, gpw->base + GEMINI_PWC_CTRLREG);
+
+ val &= ~GEMINI_CTRL_ENABLE;
+ val |= GEMINI_CTRL_SHUTDOWN;
+ writel(val, gpw->base + GEMINI_PWC_CTRLREG);
+}
+
+static int gemini_poweroff_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct resource *res;
+ struct gemini_powercon *gpw;
+ u32 val;
+ int irq;
+ int ret;
+
+ gpw = devm_kzalloc(dev, sizeof(*gpw), GFP_KERNEL);
+ if (!gpw)
+ return -ENOMEM;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ gpw->base = devm_ioremap_resource(dev, res);
+ if (IS_ERR(gpw->base))
+ return PTR_ERR(gpw->base);
+
+ irq = platform_get_irq(pdev, 0);
+ if (!irq)
+ return -EINVAL;
+
+ gpw->dev = dev;
+
+ val = readl(gpw->base + GEMINI_PWC_IDREG);
+ val &= 0xFFFFFF00U;
+ if (val != GEMINI_PWC_ID) {
+ dev_err(dev, "wrong power controller ID: %08x\n",
+ val);
+ return -ENODEV;
+ }
+
+ /* Clear the power management IRQ */
+ val = readl(gpw->base + GEMINI_PWC_CTRLREG);
+ val |= GEMINI_CTRL_IRQ_CLR;
+ writel(val, gpw->base + GEMINI_PWC_CTRLREG);
+
+ ret = devm_request_irq(dev, irq, gemini_powerbutton_interrupt, 0,
+ "poweroff", gpw);
+ if (ret)
+ return ret;
+
+ pm_power_off = gemini_poweroff;
+ gpw_poweroff = gpw;
+
+ /*
+ * Enable the power controller. This is crucial on Gemini
+ * systems: if this is not done, pressing the power button
+ * will result in unconditional poweroff without any warning.
+ * This makes the kernel handle the poweroff.
+ */
+ val = readl(gpw->base + GEMINI_PWC_CTRLREG);
+ val |= GEMINI_CTRL_ENABLE;
+ writel(val, gpw->base + GEMINI_PWC_CTRLREG);
+
+ dev_info(dev, "Gemini poweroff driver registered\n");
+
+ return 0;
+}
+
+static const struct of_device_id gemini_poweroff_of_match[] = {
+ {
+ .compatible = "cortina,gemini-power-controller",
+ },
+ {}
+};
+
+static struct platform_driver gemini_poweroff_driver = {
+ .probe = gemini_poweroff_probe,
+ .driver = {
+ .name = "gemini-poweroff",
+ .of_match_table = gemini_poweroff_of_match,
+ },
+};
+builtin_platform_driver(gemini_poweroff_driver);
diff --git a/drivers/power/reset/syscon-poweroff.c b/drivers/power/reset/syscon-poweroff.c
index b683383..f9f1cb5 100644
--- a/drivers/power/reset/syscon-poweroff.c
+++ b/drivers/power/reset/syscon-poweroff.c
@@ -28,12 +28,13 @@
static struct regmap *map;
static u32 offset;
+static u32 value;
static u32 mask;
static void syscon_poweroff(void)
{
/* Issue the poweroff */
- regmap_write(map, offset, mask);
+ regmap_update_bits(map, offset, mask, value);
mdelay(1000);
@@ -43,6 +44,7 @@
static int syscon_poweroff_probe(struct platform_device *pdev)
{
char symname[KSYM_NAME_LEN];
+ int mask_err, value_err;
map = syscon_regmap_lookup_by_phandle(pdev->dev.of_node, "regmap");
if (IS_ERR(map)) {
@@ -55,11 +57,22 @@
return -EINVAL;
}
- if (of_property_read_u32(pdev->dev.of_node, "mask", &mask)) {
- dev_err(&pdev->dev, "unable to read 'mask'");
+ value_err = of_property_read_u32(pdev->dev.of_node, "value", &value);
+ mask_err = of_property_read_u32(pdev->dev.of_node, "mask", &mask);
+ if (value_err && mask_err) {
+ dev_err(&pdev->dev, "unable to read 'value' and 'mask'");
return -EINVAL;
}
+ if (value_err) {
+ /* support old binding */
+ value = mask;
+ mask = 0xFFFFFFFF;
+ } else if (mask_err) {
+ /* support value without mask*/
+ mask = 0xFFFFFFFF;
+ }
+
if (pm_power_off) {
lookup_symbol_name((ulong)pm_power_off, symname);
dev_err(&pdev->dev,
diff --git a/drivers/power/supply/Kconfig b/drivers/power/supply/Kconfig
index da54ac8..da92275 100644
--- a/drivers/power/supply/Kconfig
+++ b/drivers/power/supply/Kconfig
@@ -117,6 +117,12 @@
Say Y here to enable support for the DS2782/DS2786 standalone battery
gas-gauge.
+config BATTERY_LEGO_EV3
+ tristate "LEGO MINDSTORMS EV3 battery"
+ depends on OF && IIO && GPIOLIB
+ help
+ Say Y here to enable support for the LEGO MINDSTORMS EV3 battery.
+
config BATTERY_PMU
tristate "Apple PMU battery"
depends on PPC32 && ADB_PMU
@@ -317,6 +323,14 @@
Say Y here to enable support for battery information on Nokia
RX-51, also known as N900 tablet.
+config CHARGER_CPCAP
+ tristate "CPCAP PMIC Charger Driver"
+ depends on MFD_CPCAP && IIO
+ default MFD_CPCAP
+ help
+ Say Y to enable support for CPCAP PMIC charger driver for Motorola
+ mobile devices such as Droid 4.
+
config CHARGER_ISP1704
tristate "ISP1704 USB Charger Detection"
depends on USB_PHY
@@ -438,6 +452,7 @@
config CHARGER_BQ24190
tristate "TI BQ24190 battery charger driver"
depends on I2C
+ depends on EXTCON
depends on GPIOLIB || COMPILE_TEST
help
Say Y to enable support for the TI BQ24190 battery charger.
diff --git a/drivers/power/supply/Makefile b/drivers/power/supply/Makefile
index 3789a2c..39fc733 100644
--- a/drivers/power/supply/Makefile
+++ b/drivers/power/supply/Makefile
@@ -25,6 +25,7 @@
obj-$(CONFIG_BATTERY_DS2782) += ds2782_battery.o
obj-$(CONFIG_BATTERY_GAUGE_LTC2941) += ltc2941-battery-gauge.o
obj-$(CONFIG_BATTERY_GOLDFISH) += goldfish_battery.o
+obj-$(CONFIG_BATTERY_LEGO_EV3) += lego_ev3_battery.o
obj-$(CONFIG_BATTERY_PMU) += pmu_battery.o
obj-$(CONFIG_BATTERY_OLPC) += olpc_battery.o
obj-$(CONFIG_BATTERY_TOSA) += tosa_battery.o
@@ -51,6 +52,7 @@
obj-$(CONFIG_BATTERY_JZ4740) += jz4740-battery.o
obj-$(CONFIG_BATTERY_RX51) += rx51_battery.o
obj-$(CONFIG_AB8500_BM) += ab8500_bmdata.o ab8500_charger.o ab8500_fg.o ab8500_btemp.o abx500_chargalg.o pm2301_charger.o
+obj-$(CONFIG_CHARGER_CPCAP) += cpcap-charger.o
obj-$(CONFIG_CHARGER_ISP1704) += isp1704_charger.o
obj-$(CONFIG_CHARGER_MAX8903) += max8903_charger.o
obj-$(CONFIG_CHARGER_TWL4030) += twl4030_charger.o
diff --git a/drivers/power/supply/ab8500_bmdata.c b/drivers/power/supply/ab8500_bmdata.c
index d298645..8c495860 100644
--- a/drivers/power/supply/ab8500_bmdata.c
+++ b/drivers/power/supply/ab8500_bmdata.c
@@ -430,10 +430,10 @@
};
static const struct abx500_maxim_parameters abx540_maxi_params = {
- .ena_maxi = true,
- .chg_curr = 3000,
- .wait_cycles = 10,
- .charger_curr_step = 200,
+ .ena_maxi = true,
+ .chg_curr = 3000,
+ .wait_cycles = 10,
+ .charger_curr_step = 200,
};
static const struct abx500_bm_charger_parameters chg = {
diff --git a/drivers/power/supply/axp288_charger.c b/drivers/power/supply/axp288_charger.c
index 6be2fe2..d51ebd1 100644
--- a/drivers/power/supply/axp288_charger.c
+++ b/drivers/power/supply/axp288_charger.c
@@ -14,6 +14,7 @@
* GNU General Public License for more details.
*/
+#include <linux/acpi.h>
#include <linux/module.h>
#include <linux/device.h>
#include <linux/regmap.h>
@@ -113,7 +114,8 @@
#define ILIM_3000MA 3000 /* 3000mA */
#define AXP288_EXTCON_DEV_NAME "axp288_extcon"
-#define USB_HOST_EXTCON_DEV_NAME "INT3496:00"
+#define USB_HOST_EXTCON_HID "INT3496"
+#define USB_HOST_EXTCON_NAME "INT3496:00"
static const unsigned int cable_ids[] =
{ EXTCON_CHG_USB_SDP, EXTCON_CHG_USB_CDP, EXTCON_CHG_USB_DCP };
@@ -807,10 +809,14 @@
return -EPROBE_DEFER;
}
- info->otg.cable = extcon_get_extcon_dev(USB_HOST_EXTCON_DEV_NAME);
- if (info->otg.cable == NULL) {
- dev_dbg(dev, "EXTCON_USB_HOST is not ready, probe deferred\n");
- return -EPROBE_DEFER;
+ if (acpi_dev_present(USB_HOST_EXTCON_HID, NULL, -1)) {
+ info->otg.cable = extcon_get_extcon_dev(USB_HOST_EXTCON_NAME);
+ if (info->otg.cable == NULL) {
+ dev_dbg(dev, "EXTCON_USB_HOST is not ready, probe deferred\n");
+ return -EPROBE_DEFER;
+ }
+ dev_info(&pdev->dev,
+ "Using " USB_HOST_EXTCON_HID " extcon for usb-id\n");
}
platform_set_drvdata(pdev, info);
@@ -849,13 +855,15 @@
/* Register for OTG notification */
INIT_WORK(&info->otg.work, axp288_charger_otg_evt_worker);
info->otg.id_nb.notifier_call = axp288_charger_handle_otg_evt;
- ret = devm_extcon_register_notifier(&pdev->dev, info->otg.cable,
+ if (info->otg.cable) {
+ ret = devm_extcon_register_notifier(&pdev->dev, info->otg.cable,
EXTCON_USB_HOST, &info->otg.id_nb);
- if (ret) {
- dev_err(dev, "failed to register EXTCON_USB_HOST notifier\n");
- return ret;
+ if (ret) {
+ dev_err(dev, "failed to register EXTCON_USB_HOST notifier\n");
+ return ret;
+ }
+ schedule_work(&info->otg.work);
}
- schedule_work(&info->otg.work);
/* Register charger interrupts */
for (i = 0; i < CHRG_INTR_END; i++) {
diff --git a/drivers/power/supply/bq24190_charger.c b/drivers/power/supply/bq24190_charger.c
index a4f0849..bd9e5c3 100644
--- a/drivers/power/supply/bq24190_charger.c
+++ b/drivers/power/supply/bq24190_charger.c
@@ -11,16 +11,15 @@
#include <linux/module.h>
#include <linux/interrupt.h>
#include <linux/delay.h>
+#include <linux/extcon.h>
#include <linux/of_irq.h>
#include <linux/of_device.h>
#include <linux/pm_runtime.h>
#include <linux/power_supply.h>
+#include <linux/workqueue.h>
#include <linux/gpio.h>
#include <linux/i2c.h>
-#include <linux/power/bq24190_charger.h>
-
-
#define BQ24190_MANUFACTURER "Texas Instruments"
#define BQ24190_REG_ISC 0x00 /* Input Source Control */
@@ -39,6 +38,9 @@
#define BQ24190_REG_POC_WDT_RESET_SHIFT 6
#define BQ24190_REG_POC_CHG_CONFIG_MASK (BIT(5) | BIT(4))
#define BQ24190_REG_POC_CHG_CONFIG_SHIFT 4
+#define BQ24190_REG_POC_CHG_CONFIG_DISABLE 0x0
+#define BQ24190_REG_POC_CHG_CONFIG_CHARGE 0x1
+#define BQ24190_REG_POC_CHG_CONFIG_OTG 0x2
#define BQ24190_REG_POC_SYS_MIN_MASK (BIT(3) | BIT(2) | BIT(1))
#define BQ24190_REG_POC_SYS_MIN_SHIFT 1
#define BQ24190_REG_POC_BOOST_LIM_MASK BIT(0)
@@ -151,10 +153,12 @@
struct device *dev;
struct power_supply *charger;
struct power_supply *battery;
+ struct extcon_dev *extcon;
+ struct notifier_block extcon_nb;
+ struct delayed_work extcon_work;
char model_name[I2C_NAME_SIZE];
- kernel_ulong_t model;
- unsigned int gpio_int;
- unsigned int irq;
+ bool initialized;
+ bool irq_event;
struct mutex f_reg_lock;
u8 f_reg;
u8 ss_reg;
@@ -168,6 +172,12 @@
* number at that index in the array is the real-world value that it
* represents.
*/
+
+/* REG00[2:0] (IINLIM) in uAh */
+static const int bq24190_isc_iinlim_values[] = {
+ 100000, 150000, 500000, 900000, 1200000, 1500000, 2000000, 3000000
+};
+
/* REG02[7:2] (ICHG) in uAh */
static const int bq24190_ccc_ichg_values[] = {
512000, 576000, 640000, 704000, 768000, 832000, 896000, 960000,
@@ -418,6 +428,7 @@
struct power_supply *psy = dev_get_drvdata(dev);
struct bq24190_dev_info *bdi = power_supply_get_drvdata(psy);
struct bq24190_sysfs_field_info *info;
+ ssize_t count;
int ret;
u8 v;
@@ -425,11 +436,20 @@
if (!info)
return -EINVAL;
- ret = bq24190_read_mask(bdi, info->reg, info->mask, info->shift, &v);
- if (ret)
+ ret = pm_runtime_get_sync(bdi->dev);
+ if (ret < 0)
return ret;
- return scnprintf(buf, PAGE_SIZE, "%hhx\n", v);
+ ret = bq24190_read_mask(bdi, info->reg, info->mask, info->shift, &v);
+ if (ret)
+ count = ret;
+ else
+ count = scnprintf(buf, PAGE_SIZE, "%hhx\n", v);
+
+ pm_runtime_mark_last_busy(bdi->dev);
+ pm_runtime_put_autosuspend(bdi->dev);
+
+ return count;
}
static ssize_t bq24190_sysfs_store(struct device *dev,
@@ -449,9 +469,16 @@
if (ret < 0)
return ret;
+ ret = pm_runtime_get_sync(bdi->dev);
+ if (ret < 0)
+ return ret;
+
ret = bq24190_write_mask(bdi, info->reg, info->mask, info->shift, v);
if (ret)
- return ret;
+ count = ret;
+
+ pm_runtime_mark_last_busy(bdi->dev);
+ pm_runtime_put_autosuspend(bdi->dev);
return count;
}
@@ -523,16 +550,13 @@
if (ret < 0)
return ret;
- if (!v)
- break;
+ if (v == 0)
+ return 0;
- udelay(10);
+ usleep_range(100, 200);
} while (--limit);
- if (!limit)
- return -EIO;
-
- return 0;
+ return -EIO;
}
/* Charger power supply property routines */
@@ -793,7 +817,9 @@
dev_dbg(bdi->dev, "prop: %d\n", psp);
- pm_runtime_get_sync(bdi->dev);
+ ret = pm_runtime_get_sync(bdi->dev);
+ if (ret < 0)
+ return ret;
switch (psp) {
case POWER_SUPPLY_PROP_CHARGE_TYPE:
@@ -833,7 +859,9 @@
ret = -ENODATA;
}
- pm_runtime_put_sync(bdi->dev);
+ pm_runtime_mark_last_busy(bdi->dev);
+ pm_runtime_put_autosuspend(bdi->dev);
+
return ret;
}
@@ -846,7 +874,9 @@
dev_dbg(bdi->dev, "prop: %d\n", psp);
- pm_runtime_get_sync(bdi->dev);
+ ret = pm_runtime_get_sync(bdi->dev);
+ if (ret < 0)
+ return ret;
switch (psp) {
case POWER_SUPPLY_PROP_CHARGE_TYPE:
@@ -862,7 +892,9 @@
ret = -EINVAL;
}
- pm_runtime_put_sync(bdi->dev);
+ pm_runtime_mark_last_busy(bdi->dev);
+ pm_runtime_put_autosuspend(bdi->dev);
+
return ret;
}
@@ -1063,7 +1095,9 @@
dev_dbg(bdi->dev, "prop: %d\n", psp);
- pm_runtime_get_sync(bdi->dev);
+ ret = pm_runtime_get_sync(bdi->dev);
+ if (ret < 0)
+ return ret;
switch (psp) {
case POWER_SUPPLY_PROP_STATUS:
@@ -1091,7 +1125,9 @@
ret = -ENODATA;
}
- pm_runtime_put_sync(bdi->dev);
+ pm_runtime_mark_last_busy(bdi->dev);
+ pm_runtime_put_autosuspend(bdi->dev);
+
return ret;
}
@@ -1104,7 +1140,9 @@
dev_dbg(bdi->dev, "prop: %d\n", psp);
- pm_runtime_get_sync(bdi->dev);
+ ret = pm_runtime_get_sync(bdi->dev);
+ if (ret < 0)
+ return ret;
switch (psp) {
case POWER_SUPPLY_PROP_ONLINE:
@@ -1117,7 +1155,9 @@
ret = -EINVAL;
}
- pm_runtime_put_sync(bdi->dev);
+ pm_runtime_mark_last_busy(bdi->dev);
+ pm_runtime_put_autosuspend(bdi->dev);
+
return ret;
}
@@ -1157,9 +1197,8 @@
.property_is_writeable = bq24190_battery_property_is_writeable,
};
-static irqreturn_t bq24190_irq_handler_thread(int irq, void *data)
+static void bq24190_check_status(struct bq24190_dev_info *bdi)
{
- struct bq24190_dev_info *bdi = data;
const u8 battery_mask_ss = BQ24190_REG_SS_CHRG_STAT_MASK;
const u8 battery_mask_f = BQ24190_REG_F_BAT_FAULT_MASK
| BQ24190_REG_F_NTC_FAULT_MASK;
@@ -1167,12 +1206,10 @@
u8 ss_reg = 0, f_reg = 0;
int i, ret;
- pm_runtime_get_sync(bdi->dev);
-
ret = bq24190_read(bdi, BQ24190_REG_SS, &ss_reg);
if (ret < 0) {
dev_err(bdi->dev, "Can't read SS reg: %d\n", ret);
- goto out;
+ return;
}
i = 0;
@@ -1180,12 +1217,17 @@
ret = bq24190_read(bdi, BQ24190_REG_F, &f_reg);
if (ret < 0) {
dev_err(bdi->dev, "Can't read F reg: %d\n", ret);
- goto out;
+ return;
}
} while (f_reg && ++i < 2);
+ /* ignore over/under voltage fault after disconnect */
+ if (f_reg == (1 << BQ24190_REG_F_CHRG_FAULT_SHIFT) &&
+ !(ss_reg & BQ24190_REG_SS_PG_STAT_MASK))
+ f_reg = 0;
+
if (f_reg != bdi->f_reg) {
- dev_info(bdi->dev,
+ dev_warn(bdi->dev,
"Fault: boost %d, charge %d, battery %d, ntc %d\n",
!!(f_reg & BQ24190_REG_F_BOOST_FAULT_MASK),
!!(f_reg & BQ24190_REG_F_CHRG_FAULT_MASK),
@@ -1229,90 +1271,126 @@
if (alert_battery)
power_supply_changed(bdi->battery);
-out:
- pm_runtime_put_sync(bdi->dev);
-
dev_dbg(bdi->dev, "ss_reg: 0x%02x, f_reg: 0x%02x\n", ss_reg, f_reg);
+}
+
+static irqreturn_t bq24190_irq_handler_thread(int irq, void *data)
+{
+ struct bq24190_dev_info *bdi = data;
+ int error;
+
+ bdi->irq_event = true;
+ error = pm_runtime_get_sync(bdi->dev);
+ if (error < 0) {
+ dev_warn(bdi->dev, "pm_runtime_get failed: %i\n", error);
+ pm_runtime_put_noidle(bdi->dev);
+ return IRQ_NONE;
+ }
+ bq24190_check_status(bdi);
+ pm_runtime_mark_last_busy(bdi->dev);
+ pm_runtime_put_autosuspend(bdi->dev);
+ bdi->irq_event = false;
return IRQ_HANDLED;
}
+static void bq24190_extcon_work(struct work_struct *work)
+{
+ struct bq24190_dev_info *bdi =
+ container_of(work, struct bq24190_dev_info, extcon_work.work);
+ int error, iinlim = 0;
+ u8 v;
+
+ error = pm_runtime_get_sync(bdi->dev);
+ if (error < 0) {
+ dev_warn(bdi->dev, "pm_runtime_get failed: %i\n", error);
+ pm_runtime_put_noidle(bdi->dev);
+ return;
+ }
+
+ if (extcon_get_state(bdi->extcon, EXTCON_CHG_USB_SDP) == 1)
+ iinlim = 500000;
+ else if (extcon_get_state(bdi->extcon, EXTCON_CHG_USB_CDP) == 1 ||
+ extcon_get_state(bdi->extcon, EXTCON_CHG_USB_ACA) == 1)
+ iinlim = 1500000;
+ else if (extcon_get_state(bdi->extcon, EXTCON_CHG_USB_DCP) == 1)
+ iinlim = 2000000;
+
+ if (iinlim) {
+ error = bq24190_set_field_val(bdi, BQ24190_REG_ISC,
+ BQ24190_REG_ISC_IINLIM_MASK,
+ BQ24190_REG_ISC_IINLIM_SHIFT,
+ bq24190_isc_iinlim_values,
+ ARRAY_SIZE(bq24190_isc_iinlim_values),
+ iinlim);
+ if (error < 0)
+ dev_err(bdi->dev, "Can't set IINLIM: %d\n", error);
+ }
+
+ /* if no charger found and in USB host mode, set OTG 5V boost, else normal */
+ if (!iinlim && extcon_get_state(bdi->extcon, EXTCON_USB_HOST) == 1)
+ v = BQ24190_REG_POC_CHG_CONFIG_OTG;
+ else
+ v = BQ24190_REG_POC_CHG_CONFIG_CHARGE;
+
+ error = bq24190_write_mask(bdi, BQ24190_REG_POC,
+ BQ24190_REG_POC_CHG_CONFIG_MASK,
+ BQ24190_REG_POC_CHG_CONFIG_SHIFT,
+ v);
+ if (error < 0)
+ dev_err(bdi->dev, "Can't set CHG_CONFIG: %d\n", error);
+
+ pm_runtime_mark_last_busy(bdi->dev);
+ pm_runtime_put_autosuspend(bdi->dev);
+}
+
+static int bq24190_extcon_event(struct notifier_block *nb, unsigned long event,
+ void *param)
+{
+ struct bq24190_dev_info *bdi =
+ container_of(nb, struct bq24190_dev_info, extcon_nb);
+
+ /*
+ * The Power-Good detection may take up to 220ms, sometimes
+ * the external charger detection is quicker, and the bq24190 will
+ * reset to iinlim based on its own charger detection (which is not
+ * hooked up when using external charger detection) resulting in
+ * a too low default 500mA iinlim. Delay applying the extcon value
+ * for 300ms to avoid this.
+ */
+ queue_delayed_work(system_wq, &bdi->extcon_work, msecs_to_jiffies(300));
+
+ return NOTIFY_OK;
+}
+
static int bq24190_hw_init(struct bq24190_dev_info *bdi)
{
u8 v;
int ret;
- pm_runtime_get_sync(bdi->dev);
-
/* First check that the device really is what its supposed to be */
ret = bq24190_read_mask(bdi, BQ24190_REG_VPRS,
BQ24190_REG_VPRS_PN_MASK,
BQ24190_REG_VPRS_PN_SHIFT,
&v);
if (ret < 0)
- goto out;
+ return ret;
- if (v != bdi->model) {
- ret = -ENODEV;
- goto out;
+ if (v != BQ24190_REG_VPRS_PN_24190 &&
+ v != BQ24190_REG_VPRS_PN_24192I) {
+ dev_err(bdi->dev, "Error unknown model: 0x%02x\n", v);
+ return -ENODEV;
}
ret = bq24190_register_reset(bdi);
if (ret < 0)
- goto out;
+ return ret;
ret = bq24190_set_mode_host(bdi);
if (ret < 0)
- goto out;
+ return ret;
- ret = bq24190_read(bdi, BQ24190_REG_SS, &bdi->ss_reg);
-out:
- pm_runtime_put_sync(bdi->dev);
- return ret;
-}
-
-#ifdef CONFIG_OF
-static int bq24190_setup_dt(struct bq24190_dev_info *bdi)
-{
- bdi->irq = irq_of_parse_and_map(bdi->dev->of_node, 0);
- if (bdi->irq <= 0)
- return -1;
-
- return 0;
-}
-#else
-static int bq24190_setup_dt(struct bq24190_dev_info *bdi)
-{
- return -1;
-}
-#endif
-
-static int bq24190_setup_pdata(struct bq24190_dev_info *bdi,
- struct bq24190_platform_data *pdata)
-{
- int ret;
-
- if (!gpio_is_valid(pdata->gpio_int))
- return -1;
-
- ret = gpio_request(pdata->gpio_int, dev_name(bdi->dev));
- if (ret < 0)
- return -1;
-
- ret = gpio_direction_input(pdata->gpio_int);
- if (ret < 0)
- goto out;
-
- bdi->irq = gpio_to_irq(pdata->gpio_int);
- if (!bdi->irq)
- goto out;
-
- bdi->gpio_int = pdata->gpio_int;
- return 0;
-
-out:
- gpio_free(pdata->gpio_int);
- return -1;
+ return bq24190_read(bdi, BQ24190_REG_SS, &bdi->ss_reg);
}
static int bq24190_probe(struct i2c_client *client,
@@ -1320,9 +1398,9 @@
{
struct i2c_adapter *adapter = to_i2c_adapter(client->dev.parent);
struct device *dev = &client->dev;
- struct bq24190_platform_data *pdata = client->dev.platform_data;
struct power_supply_config charger_cfg = {}, battery_cfg = {};
struct bq24190_dev_info *bdi;
+ const char *name;
int ret;
if (!i2c_check_functionality(adapter, I2C_FUNC_SMBUS_BYTE_DATA)) {
@@ -1338,7 +1416,6 @@
bdi->client = client;
bdi->dev = dev;
- bdi->model = id->driver_data;
strncpy(bdi->model_name, id->name, I2C_NAME_SIZE);
mutex_init(&bdi->f_reg_lock);
bdi->f_reg = 0;
@@ -1346,23 +1423,43 @@
i2c_set_clientdata(client, bdi);
- if (dev->of_node)
- ret = bq24190_setup_dt(bdi);
- else
- ret = bq24190_setup_pdata(bdi, pdata);
-
- if (ret) {
+ if (!client->irq) {
dev_err(dev, "Can't get irq info\n");
return -EINVAL;
}
+ /*
+ * Devicetree platforms should get extcon via phandle (not yet supported).
+ * On ACPI platforms, extcon clients may invoke us with:
+ * struct property_entry pe[] =
+ * { PROPERTY_ENTRY_STRING("extcon-name", client_name), ... };
+ * struct i2c_board_info bi =
+ * { .type = "bq24190", .addr = 0x6b, .properties = pe, .irq = irq };
+ * struct i2c_adapter ad = { ... };
+ * i2c_add_adapter(&ad);
+ * i2c_new_device(&ad, &bi);
+ */
+ if (device_property_read_string(dev, "extcon-name", &name) == 0) {
+ bdi->extcon = extcon_get_extcon_dev(name);
+ if (!bdi->extcon)
+ return -EPROBE_DEFER;
+
+ dev_info(bdi->dev, "using extcon device %s\n", name);
+ }
+
pm_runtime_enable(dev);
- pm_runtime_resume(dev);
+ pm_runtime_use_autosuspend(dev);
+ pm_runtime_set_autosuspend_delay(dev, 600);
+ ret = pm_runtime_get_sync(dev);
+ if (ret < 0) {
+ dev_err(dev, "pm_runtime_get failed: %i\n", ret);
+ goto out_pmrt;
+ }
ret = bq24190_hw_init(bdi);
if (ret < 0) {
dev_err(dev, "Hardware init failed\n");
- goto out1;
+ goto out_pmrt;
}
charger_cfg.drv_data = bdi;
@@ -1373,7 +1470,7 @@
if (IS_ERR(bdi->charger)) {
dev_err(dev, "Can't register charger\n");
ret = PTR_ERR(bdi->charger);
- goto out1;
+ goto out_pmrt;
}
battery_cfg.drv_data = bdi;
@@ -1382,87 +1479,160 @@
if (IS_ERR(bdi->battery)) {
dev_err(dev, "Can't register battery\n");
ret = PTR_ERR(bdi->battery);
- goto out2;
+ goto out_charger;
}
ret = bq24190_sysfs_create_group(bdi);
if (ret) {
dev_err(dev, "Can't create sysfs entries\n");
- goto out3;
+ goto out_battery;
}
- ret = devm_request_threaded_irq(dev, bdi->irq, NULL,
+ bdi->initialized = true;
+
+ ret = devm_request_threaded_irq(dev, client->irq, NULL,
bq24190_irq_handler_thread,
IRQF_TRIGGER_FALLING | IRQF_ONESHOT,
"bq24190-charger", bdi);
if (ret < 0) {
dev_err(dev, "Can't set up irq handler\n");
- goto out4;
+ goto out_sysfs;
}
+ if (bdi->extcon) {
+ INIT_DELAYED_WORK(&bdi->extcon_work, bq24190_extcon_work);
+ bdi->extcon_nb.notifier_call = bq24190_extcon_event;
+ ret = devm_extcon_register_notifier_all(dev, bdi->extcon,
+ &bdi->extcon_nb);
+ if (ret) {
+ dev_err(dev, "Can't register extcon\n");
+ goto out_sysfs;
+ }
+
+ /* Sync initial cable state */
+ queue_delayed_work(system_wq, &bdi->extcon_work, 0);
+ }
+
+ enable_irq_wake(client->irq);
+
+ pm_runtime_mark_last_busy(dev);
+ pm_runtime_put_autosuspend(dev);
+
return 0;
-out4:
+out_sysfs:
bq24190_sysfs_remove_group(bdi);
-out3:
+out_battery:
power_supply_unregister(bdi->battery);
-out2:
+out_charger:
power_supply_unregister(bdi->charger);
-out1:
+out_pmrt:
+ pm_runtime_put_sync(dev);
+ pm_runtime_dont_use_autosuspend(dev);
pm_runtime_disable(dev);
- if (bdi->gpio_int)
- gpio_free(bdi->gpio_int);
return ret;
}
static int bq24190_remove(struct i2c_client *client)
{
struct bq24190_dev_info *bdi = i2c_get_clientdata(client);
+ int error;
- pm_runtime_get_sync(bdi->dev);
+ error = pm_runtime_get_sync(bdi->dev);
+ if (error < 0) {
+ dev_warn(bdi->dev, "pm_runtime_get failed: %i\n", error);
+ pm_runtime_put_noidle(bdi->dev);
+ }
+
bq24190_register_reset(bdi);
- pm_runtime_put_sync(bdi->dev);
-
bq24190_sysfs_remove_group(bdi);
power_supply_unregister(bdi->battery);
power_supply_unregister(bdi->charger);
+ if (error >= 0)
+ pm_runtime_put_sync(bdi->dev);
+ pm_runtime_dont_use_autosuspend(bdi->dev);
pm_runtime_disable(bdi->dev);
- if (bdi->gpio_int)
- gpio_free(bdi->gpio_int);
-
return 0;
}
-#ifdef CONFIG_PM_SLEEP
-static int bq24190_pm_suspend(struct device *dev)
+static __maybe_unused int bq24190_runtime_suspend(struct device *dev)
{
struct i2c_client *client = to_i2c_client(dev);
struct bq24190_dev_info *bdi = i2c_get_clientdata(client);
- pm_runtime_get_sync(bdi->dev);
+ if (!bdi->initialized)
+ return 0;
+
+ dev_dbg(bdi->dev, "%s\n", __func__);
+
+ return 0;
+}
+
+static __maybe_unused int bq24190_runtime_resume(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct bq24190_dev_info *bdi = i2c_get_clientdata(client);
+
+ if (!bdi->initialized)
+ return 0;
+
+ if (!bdi->irq_event) {
+ dev_dbg(bdi->dev, "checking events on possible wakeirq\n");
+ bq24190_check_status(bdi);
+ }
+
+ return 0;
+}
+
+static __maybe_unused int bq24190_pm_suspend(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct bq24190_dev_info *bdi = i2c_get_clientdata(client);
+ int error;
+
+ error = pm_runtime_get_sync(bdi->dev);
+ if (error < 0) {
+ dev_warn(bdi->dev, "pm_runtime_get failed: %i\n", error);
+ pm_runtime_put_noidle(bdi->dev);
+ }
+
bq24190_register_reset(bdi);
- pm_runtime_put_sync(bdi->dev);
+
+ if (error >= 0) {
+ pm_runtime_mark_last_busy(bdi->dev);
+ pm_runtime_put_autosuspend(bdi->dev);
+ }
return 0;
}
-static int bq24190_pm_resume(struct device *dev)
+static __maybe_unused int bq24190_pm_resume(struct device *dev)
{
struct i2c_client *client = to_i2c_client(dev);
struct bq24190_dev_info *bdi = i2c_get_clientdata(client);
+ int error;
bdi->f_reg = 0;
bdi->ss_reg = BQ24190_REG_SS_VBUS_STAT_MASK; /* impossible state */
- pm_runtime_get_sync(bdi->dev);
+ error = pm_runtime_get_sync(bdi->dev);
+ if (error < 0) {
+ dev_warn(bdi->dev, "pm_runtime_get failed: %i\n", error);
+ pm_runtime_put_noidle(bdi->dev);
+ }
+
bq24190_register_reset(bdi);
bq24190_set_mode_host(bdi);
bq24190_read(bdi, BQ24190_REG_SS, &bdi->ss_reg);
- pm_runtime_put_sync(bdi->dev);
+
+ if (error >= 0) {
+ pm_runtime_mark_last_busy(bdi->dev);
+ pm_runtime_put_autosuspend(bdi->dev);
+ }
/* Things may have changed while suspended so alert upper layer */
power_supply_changed(bdi->charger);
@@ -1470,17 +1640,16 @@
return 0;
}
-#endif
-static SIMPLE_DEV_PM_OPS(bq24190_pm_ops, bq24190_pm_suspend, bq24190_pm_resume);
+static const struct dev_pm_ops bq24190_pm_ops = {
+ SET_RUNTIME_PM_OPS(bq24190_runtime_suspend, bq24190_runtime_resume,
+ NULL)
+ SET_SYSTEM_SLEEP_PM_OPS(bq24190_pm_suspend, bq24190_pm_resume)
+};
-/*
- * Only support the bq24190 right now. The bq24192, bq24192i, and bq24193
- * are similar but not identical so the driver needs to be extended to
- * support them.
- */
static const struct i2c_device_id bq24190_i2c_ids[] = {
- { "bq24190", BQ24190_REG_VPRS_PN_24190 },
+ { "bq24190" },
+ { "bq24192i" },
{ },
};
MODULE_DEVICE_TABLE(i2c, bq24190_i2c_ids);
diff --git a/drivers/power/supply/bq25890_charger.c b/drivers/power/supply/bq25890_charger.c
index f993a55..8e2c41d 100644
--- a/drivers/power/supply/bq25890_charger.c
+++ b/drivers/power/supply/bq25890_charger.c
@@ -723,7 +723,7 @@
{
struct gpio_desc *irq;
- irq = devm_gpiod_get_index(bq->dev, BQ25890_IRQ_PIN, 0, GPIOD_IN);
+ irq = devm_gpiod_get(bq->dev, BQ25890_IRQ_PIN, GPIOD_IN);
if (IS_ERR(irq)) {
dev_err(bq->dev, "Could not probe irq pin.\n");
return PTR_ERR(irq);
diff --git a/drivers/power/supply/charger-manager.c b/drivers/power/supply/charger-manager.c
index e664ca7..adc3761 100644
--- a/drivers/power/supply/charger-manager.c
+++ b/drivers/power/supply/charger-manager.c
@@ -1198,7 +1198,7 @@
static int charger_extcon_init(struct charger_manager *cm,
struct charger_cable *cable)
{
- int ret = 0;
+ int ret;
/*
* Charger manager use Extcon framework to identify
@@ -1232,7 +1232,7 @@
{
struct charger_desc *desc = cm->desc;
struct charger_regulator *charger;
- int ret = 0;
+ int ret;
int i;
int j;
@@ -1255,15 +1255,14 @@
if (ret < 0) {
dev_err(cm->dev, "Cannot initialize charger(%s)\n",
charger->regulator_name);
- goto err;
+ return ret;
}
cable->charger = charger;
cable->cm = cm;
}
}
-err:
- return ret;
+ return 0;
}
/* help function of sysfs node to control charger(regulator) */
@@ -1372,7 +1371,7 @@
int chargers_externally_control = 1;
char buf[11];
char *str;
- int ret = 0;
+ int ret;
int i;
/* Create sysfs entry to control charger(regulator) */
@@ -1382,10 +1381,9 @@
snprintf(buf, 10, "charger.%d", i);
str = devm_kzalloc(cm->dev,
sizeof(char) * (strlen(buf) + 1), GFP_KERNEL);
- if (!str) {
- ret = -ENOMEM;
- goto err;
- }
+ if (!str)
+ return -ENOMEM;
+
strcpy(str, buf);
charger->attrs[0] = &charger->attr_name.attr;
@@ -1426,19 +1424,16 @@
if (ret < 0) {
dev_err(cm->dev, "Cannot create sysfs entry of %s regulator\n",
charger->regulator_name);
- ret = -EINVAL;
- goto err;
+ return ret;
}
}
if (chargers_externally_control) {
dev_err(cm->dev, "Cannot register regulator because charger-manager must need at least one charger for charging battery\n");
- ret = -EINVAL;
- goto err;
+ return -EINVAL;
}
-err:
- return ret;
+ return 0;
}
static int cm_init_thermal_data(struct charger_manager *cm,
@@ -1626,7 +1621,7 @@
{
struct charger_desc *desc = cm_get_drv_data(pdev);
struct charger_manager *cm;
- int ret = 0, i = 0;
+ int ret, i = 0;
int j = 0;
union power_supply_propval val;
struct power_supply *fuel_gauge;
@@ -1887,14 +1882,12 @@
static int cm_suspend_noirq(struct device *dev)
{
- int ret = 0;
-
if (device_may_wakeup(dev)) {
device_set_wakeup_capable(dev, false);
- ret = -EAGAIN;
+ return -EAGAIN;
}
- return ret;
+ return 0;
}
static bool cm_need_to_awake(void)
diff --git a/drivers/power/supply/cpcap-charger.c b/drivers/power/supply/cpcap-charger.c
new file mode 100644
index 0000000..543a1bd
--- /dev/null
+++ b/drivers/power/supply/cpcap-charger.c
@@ -0,0 +1,681 @@
+/*
+ * Motorola CPCAP PMIC battery charger driver
+ *
+ * Copyright (C) 2017 Tony Lindgren <tony@atomide.com>
+ *
+ * Rewritten for Linux power framework with some parts based on
+ * on earlier driver found in the Motorola Linux kernel:
+ *
+ * Copyright (C) 2009-2010 Motorola, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/atomic.h>
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/notifier.h>
+#include <linux/of.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+#include <linux/power_supply.h>
+#include <linux/regmap.h>
+
+#include <linux/gpio/consumer.h>
+#include <linux/usb/phy_companion.h>
+#include <linux/phy/omap_usb.h>
+#include <linux/usb/otg.h>
+#include <linux/iio/consumer.h>
+#include <linux/mfd/motorola-cpcap.h>
+
+/* CPCAP_REG_CRM register bits */
+#define CPCAP_REG_CRM_UNUSED_641_15 BIT(15) /* 641 = register number */
+#define CPCAP_REG_CRM_UNUSED_641_14 BIT(14) /* 641 = register number */
+#define CPCAP_REG_CRM_CHRG_LED_EN BIT(13)
+#define CPCAP_REG_CRM_RVRSMODE BIT(12)
+#define CPCAP_REG_CRM_ICHRG_TR1 BIT(11)
+#define CPCAP_REG_CRM_ICHRG_TR0 BIT(10)
+#define CPCAP_REG_CRM_FET_OVRD BIT(9)
+#define CPCAP_REG_CRM_FET_CTRL BIT(8)
+#define CPCAP_REG_CRM_VCHRG3 BIT(7)
+#define CPCAP_REG_CRM_VCHRG2 BIT(6)
+#define CPCAP_REG_CRM_VCHRG1 BIT(5)
+#define CPCAP_REG_CRM_VCHRG0 BIT(4)
+#define CPCAP_REG_CRM_ICHRG3 BIT(3)
+#define CPCAP_REG_CRM_ICHRG2 BIT(2)
+#define CPCAP_REG_CRM_ICHRG1 BIT(1)
+#define CPCAP_REG_CRM_ICHRG0 BIT(0)
+
+/* CPCAP_REG_CRM trickle charge voltages */
+#define CPCAP_REG_CRM_TR(val) (((val) & 0x3) << 10)
+#define CPCAP_REG_CRM_TR_0A00 CPCAP_REG_CRM_TR(0x0)
+#define CPCAP_REG_CRM_TR_0A24 CPCAP_REG_CRM_TR(0x1)
+#define CPCAP_REG_CRM_TR_0A48 CPCAP_REG_CRM_TR(0x2)
+#define CPCAP_REG_CRM_TR_0A72 CPCAP_REG_CRM_TR(0x4)
+
+/* CPCAP_REG_CRM charge voltages */
+#define CPCAP_REG_CRM_VCHRG(val) (((val) & 0xf) << 4)
+#define CPCAP_REG_CRM_VCHRG_3V80 CPCAP_REG_CRM_VCHRG(0x0)
+#define CPCAP_REG_CRM_VCHRG_4V10 CPCAP_REG_CRM_VCHRG(0x1)
+#define CPCAP_REG_CRM_VCHRG_4V15 CPCAP_REG_CRM_VCHRG(0x2)
+#define CPCAP_REG_CRM_VCHRG_4V20 CPCAP_REG_CRM_VCHRG(0x3)
+#define CPCAP_REG_CRM_VCHRG_4V22 CPCAP_REG_CRM_VCHRG(0x4)
+#define CPCAP_REG_CRM_VCHRG_4V24 CPCAP_REG_CRM_VCHRG(0x5)
+#define CPCAP_REG_CRM_VCHRG_4V26 CPCAP_REG_CRM_VCHRG(0x6)
+#define CPCAP_REG_CRM_VCHRG_4V28 CPCAP_REG_CRM_VCHRG(0x7)
+#define CPCAP_REG_CRM_VCHRG_4V30 CPCAP_REG_CRM_VCHRG(0x8)
+#define CPCAP_REG_CRM_VCHRG_4V32 CPCAP_REG_CRM_VCHRG(0x9)
+#define CPCAP_REG_CRM_VCHRG_4V34 CPCAP_REG_CRM_VCHRG(0xa)
+#define CPCAP_REG_CRM_VCHRG_4V36 CPCAP_REG_CRM_VCHRG(0xb)
+#define CPCAP_REG_CRM_VCHRG_4V38 CPCAP_REG_CRM_VCHRG(0xc)
+#define CPCAP_REG_CRM_VCHRG_4V40 CPCAP_REG_CRM_VCHRG(0xd)
+#define CPCAP_REG_CRM_VCHRG_4V42 CPCAP_REG_CRM_VCHRG(0xe)
+#define CPCAP_REG_CRM_VCHRG_4V44 CPCAP_REG_CRM_VCHRG(0xf)
+
+/* CPCAP_REG_CRM charge currents */
+#define CPCAP_REG_CRM_ICHRG(val) (((val) & 0xf) << 0)
+#define CPCAP_REG_CRM_ICHRG_0A000 CPCAP_REG_CRM_ICHRG(0x0)
+#define CPCAP_REG_CRM_ICHRG_0A070 CPCAP_REG_CRM_ICHRG(0x1)
+#define CPCAP_REG_CRM_ICHRG_0A176 CPCAP_REG_CRM_ICHRG(0x2)
+#define CPCAP_REG_CRM_ICHRG_0A264 CPCAP_REG_CRM_ICHRG(0x3)
+#define CPCAP_REG_CRM_ICHRG_0A352 CPCAP_REG_CRM_ICHRG(0x4)
+#define CPCAP_REG_CRM_ICHRG_0A440 CPCAP_REG_CRM_ICHRG(0x5)
+#define CPCAP_REG_CRM_ICHRG_0A528 CPCAP_REG_CRM_ICHRG(0x6)
+#define CPCAP_REG_CRM_ICHRG_0A616 CPCAP_REG_CRM_ICHRG(0x7)
+#define CPCAP_REG_CRM_ICHRG_0A704 CPCAP_REG_CRM_ICHRG(0x8)
+#define CPCAP_REG_CRM_ICHRG_0A792 CPCAP_REG_CRM_ICHRG(0x9)
+#define CPCAP_REG_CRM_ICHRG_0A880 CPCAP_REG_CRM_ICHRG(0xa)
+#define CPCAP_REG_CRM_ICHRG_0A968 CPCAP_REG_CRM_ICHRG(0xb)
+#define CPCAP_REG_CRM_ICHRG_1A056 CPCAP_REG_CRM_ICHRG(0xc)
+#define CPCAP_REG_CRM_ICHRG_1A144 CPCAP_REG_CRM_ICHRG(0xd)
+#define CPCAP_REG_CRM_ICHRG_1A584 CPCAP_REG_CRM_ICHRG(0xe)
+#define CPCAP_REG_CRM_ICHRG_NO_LIMIT CPCAP_REG_CRM_ICHRG(0xf)
+
+enum {
+ CPCAP_CHARGER_IIO_BATTDET,
+ CPCAP_CHARGER_IIO_VOLTAGE,
+ CPCAP_CHARGER_IIO_VBUS,
+ CPCAP_CHARGER_IIO_CHRG_CURRENT,
+ CPCAP_CHARGER_IIO_BATT_CURRENT,
+ CPCAP_CHARGER_IIO_NR,
+};
+
+struct cpcap_charger_ddata {
+ struct device *dev;
+ struct regmap *reg;
+ struct list_head irq_list;
+ struct delayed_work detect_work;
+ struct delayed_work vbus_work;
+ struct gpio_desc *gpio[2]; /* gpio_reven0 & 1 */
+
+ struct iio_channel *channels[CPCAP_CHARGER_IIO_NR];
+
+ struct power_supply *usb;
+
+ struct phy_companion comparator; /* For USB VBUS */
+ bool vbus_enabled;
+ atomic_t active;
+
+ int status;
+};
+
+struct cpcap_interrupt_desc {
+ int irq;
+ struct list_head node;
+ const char *name;
+};
+
+struct cpcap_charger_ints_state {
+ bool chrg_det;
+ bool rvrs_chrg;
+ bool vbusov;
+
+ bool chrg_se1b;
+ bool rvrs_mode;
+ bool chrgcurr1;
+ bool vbusvld;
+
+ bool battdetb;
+};
+
+static enum power_supply_property cpcap_charger_props[] = {
+ POWER_SUPPLY_PROP_STATUS,
+ POWER_SUPPLY_PROP_ONLINE,
+ POWER_SUPPLY_PROP_VOLTAGE_NOW,
+ POWER_SUPPLY_PROP_CURRENT_NOW,
+};
+
+static bool cpcap_charger_battery_found(struct cpcap_charger_ddata *ddata)
+{
+ struct iio_channel *channel;
+ int error, value;
+
+ channel = ddata->channels[CPCAP_CHARGER_IIO_BATTDET];
+ error = iio_read_channel_raw(channel, &value);
+ if (error < 0) {
+ dev_warn(ddata->dev, "%s failed: %i\n", __func__, error);
+
+ return false;
+ }
+
+ return value == 1;
+}
+
+static int cpcap_charger_get_charge_voltage(struct cpcap_charger_ddata *ddata)
+{
+ struct iio_channel *channel;
+ int error, value = 0;
+
+ channel = ddata->channels[CPCAP_CHARGER_IIO_VOLTAGE];
+ error = iio_read_channel_processed(channel, &value);
+ if (error < 0) {
+ dev_warn(ddata->dev, "%s failed: %i\n", __func__, error);
+
+ return 0;
+ }
+
+ return value;
+}
+
+static int cpcap_charger_get_charge_current(struct cpcap_charger_ddata *ddata)
+{
+ struct iio_channel *channel;
+ int error, value = 0;
+
+ channel = ddata->channels[CPCAP_CHARGER_IIO_CHRG_CURRENT];
+ error = iio_read_channel_processed(channel, &value);
+ if (error < 0) {
+ dev_warn(ddata->dev, "%s failed: %i\n", __func__, error);
+
+ return 0;
+ }
+
+ return value;
+}
+
+static int cpcap_charger_get_property(struct power_supply *psy,
+ enum power_supply_property psp,
+ union power_supply_propval *val)
+{
+ struct cpcap_charger_ddata *ddata = dev_get_drvdata(psy->dev.parent);
+
+ switch (psp) {
+ case POWER_SUPPLY_PROP_STATUS:
+ val->intval = ddata->status;
+ break;
+ case POWER_SUPPLY_PROP_VOLTAGE_NOW:
+ if (ddata->status == POWER_SUPPLY_STATUS_CHARGING)
+ val->intval = cpcap_charger_get_charge_voltage(ddata) *
+ 1000;
+ else
+ val->intval = 0;
+ break;
+ case POWER_SUPPLY_PROP_CURRENT_NOW:
+ if (ddata->status == POWER_SUPPLY_STATUS_CHARGING)
+ val->intval = cpcap_charger_get_charge_current(ddata) *
+ 1000;
+ else
+ val->intval = 0;
+ break;
+ case POWER_SUPPLY_PROP_ONLINE:
+ val->intval = ddata->status == POWER_SUPPLY_STATUS_CHARGING;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void cpcap_charger_set_cable_path(struct cpcap_charger_ddata *ddata,
+ bool enabled)
+{
+ if (!ddata->gpio[0])
+ return;
+
+ gpiod_set_value(ddata->gpio[0], enabled);
+}
+
+static void cpcap_charger_set_inductive_path(struct cpcap_charger_ddata *ddata,
+ bool enabled)
+{
+ if (!ddata->gpio[1])
+ return;
+
+ gpiod_set_value(ddata->gpio[1], enabled);
+}
+
+static int cpcap_charger_set_state(struct cpcap_charger_ddata *ddata,
+ int max_voltage, int charge_current,
+ int trickle_current)
+{
+ bool enable;
+ int error;
+
+ enable = max_voltage && (charge_current || trickle_current);
+ dev_dbg(ddata->dev, "%s enable: %i\n", __func__, enable);
+
+ if (!enable) {
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_CRM,
+ 0x3fff,
+ CPCAP_REG_CRM_FET_OVRD |
+ CPCAP_REG_CRM_FET_CTRL);
+ if (error) {
+ ddata->status = POWER_SUPPLY_STATUS_UNKNOWN;
+ goto out_err;
+ }
+
+ ddata->status = POWER_SUPPLY_STATUS_DISCHARGING;
+
+ return 0;
+ }
+
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_CRM, 0x3fff,
+ CPCAP_REG_CRM_CHRG_LED_EN |
+ trickle_current |
+ CPCAP_REG_CRM_FET_OVRD |
+ CPCAP_REG_CRM_FET_CTRL |
+ max_voltage |
+ charge_current);
+ if (error) {
+ ddata->status = POWER_SUPPLY_STATUS_UNKNOWN;
+ goto out_err;
+ }
+
+ ddata->status = POWER_SUPPLY_STATUS_CHARGING;
+
+ return 0;
+
+out_err:
+ dev_err(ddata->dev, "%s failed with %i\n", __func__, error);
+
+ return error;
+}
+
+static bool cpcap_charger_vbus_valid(struct cpcap_charger_ddata *ddata)
+{
+ int error, value = 0;
+ struct iio_channel *channel =
+ ddata->channels[CPCAP_CHARGER_IIO_VBUS];
+
+ error = iio_read_channel_processed(channel, &value);
+ if (error >= 0)
+ return value > 3900 ? true : false;
+
+ dev_err(ddata->dev, "error reading VBUS: %i\n", error);
+
+ return false;
+}
+
+/* VBUS control functions for the USB PHY companion */
+
+static void cpcap_charger_vbus_work(struct work_struct *work)
+{
+ struct cpcap_charger_ddata *ddata;
+ bool vbus = false;
+ int error;
+
+ ddata = container_of(work, struct cpcap_charger_ddata,
+ vbus_work.work);
+
+ if (ddata->vbus_enabled) {
+ vbus = cpcap_charger_vbus_valid(ddata);
+ if (vbus) {
+ dev_info(ddata->dev, "VBUS already provided\n");
+
+ return;
+ }
+
+ cpcap_charger_set_cable_path(ddata, false);
+ cpcap_charger_set_inductive_path(ddata, false);
+
+ error = cpcap_charger_set_state(ddata, 0, 0, 0);
+ if (error)
+ goto out_err;
+
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_CRM,
+ CPCAP_REG_CRM_RVRSMODE,
+ CPCAP_REG_CRM_RVRSMODE);
+ if (error)
+ goto out_err;
+ } else {
+ error = regmap_update_bits(ddata->reg, CPCAP_REG_CRM,
+ CPCAP_REG_CRM_RVRSMODE, 0);
+ if (error)
+ goto out_err;
+
+ cpcap_charger_set_cable_path(ddata, true);
+ cpcap_charger_set_inductive_path(ddata, true);
+ }
+
+ return;
+
+out_err:
+ dev_err(ddata->dev, "%s could not %s vbus: %i\n", __func__,
+ ddata->vbus_enabled ? "enable" : "disable", error);
+}
+
+static int cpcap_charger_set_vbus(struct phy_companion *comparator,
+ bool enabled)
+{
+ struct cpcap_charger_ddata *ddata =
+ container_of(comparator, struct cpcap_charger_ddata,
+ comparator);
+
+ ddata->vbus_enabled = enabled;
+ schedule_delayed_work(&ddata->vbus_work, 0);
+
+ return 0;
+}
+
+/* Charger interrupt handling functions */
+
+static int cpcap_charger_get_ints_state(struct cpcap_charger_ddata *ddata,
+ struct cpcap_charger_ints_state *s)
+{
+ int val, error;
+
+ error = regmap_read(ddata->reg, CPCAP_REG_INTS1, &val);
+ if (error)
+ return error;
+
+ s->chrg_det = val & BIT(13);
+ s->rvrs_chrg = val & BIT(12);
+ s->vbusov = val & BIT(11);
+
+ error = regmap_read(ddata->reg, CPCAP_REG_INTS2, &val);
+ if (error)
+ return error;
+
+ s->chrg_se1b = val & BIT(13);
+ s->rvrs_mode = val & BIT(6);
+ s->chrgcurr1 = val & BIT(4);
+ s->vbusvld = val & BIT(3);
+
+ error = regmap_read(ddata->reg, CPCAP_REG_INTS4, &val);
+ if (error)
+ return error;
+
+ s->battdetb = val & BIT(6);
+
+ return 0;
+}
+
+static void cpcap_usb_detect(struct work_struct *work)
+{
+ struct cpcap_charger_ddata *ddata;
+ struct cpcap_charger_ints_state s;
+ int error;
+
+ ddata = container_of(work, struct cpcap_charger_ddata,
+ detect_work.work);
+
+ error = cpcap_charger_get_ints_state(ddata, &s);
+ if (error)
+ return;
+
+ if (cpcap_charger_vbus_valid(ddata) && s.chrgcurr1) {
+ int max_current;
+
+ if (cpcap_charger_battery_found(ddata))
+ max_current = CPCAP_REG_CRM_ICHRG_1A584;
+ else
+ max_current = CPCAP_REG_CRM_ICHRG_0A528;
+
+ error = cpcap_charger_set_state(ddata,
+ CPCAP_REG_CRM_VCHRG_4V20,
+ max_current,
+ CPCAP_REG_CRM_TR_0A72);
+ if (error)
+ goto out_err;
+ } else {
+ error = cpcap_charger_set_state(ddata, 0, 0, 0);
+ if (error)
+ goto out_err;
+ }
+
+ return;
+
+out_err:
+ dev_err(ddata->dev, "%s failed with %i\n", __func__, error);
+}
+
+static irqreturn_t cpcap_charger_irq_thread(int irq, void *data)
+{
+ struct cpcap_charger_ddata *ddata = data;
+
+ if (!atomic_read(&ddata->active))
+ return IRQ_NONE;
+
+ schedule_delayed_work(&ddata->detect_work, 0);
+
+ return IRQ_HANDLED;
+}
+
+static int cpcap_usb_init_irq(struct platform_device *pdev,
+ struct cpcap_charger_ddata *ddata,
+ const char *name)
+{
+ struct cpcap_interrupt_desc *d;
+ int irq, error;
+
+ irq = platform_get_irq_byname(pdev, name);
+ if (!irq)
+ return -ENODEV;
+
+ error = devm_request_threaded_irq(ddata->dev, irq, NULL,
+ cpcap_charger_irq_thread,
+ IRQF_SHARED,
+ name, ddata);
+ if (error) {
+ dev_err(ddata->dev, "could not get irq %s: %i\n",
+ name, error);
+
+ return error;
+ }
+
+ d = devm_kzalloc(ddata->dev, sizeof(*d), GFP_KERNEL);
+ if (!d)
+ return -ENOMEM;
+
+ d->name = name;
+ d->irq = irq;
+ list_add(&d->node, &ddata->irq_list);
+
+ return 0;
+}
+
+static const char * const cpcap_charger_irqs[] = {
+ /* REG_INT_0 */
+ "chrg_det", "rvrs_chrg",
+
+ /* REG_INT1 */
+ "chrg_se1b", "se0conn", "rvrs_mode", "chrgcurr1", "vbusvld",
+
+ /* REG_INT_3 */
+ "battdetb",
+};
+
+static int cpcap_usb_init_interrupts(struct platform_device *pdev,
+ struct cpcap_charger_ddata *ddata)
+{
+ int i, error;
+
+ for (i = 0; i < ARRAY_SIZE(cpcap_charger_irqs); i++) {
+ error = cpcap_usb_init_irq(pdev, ddata, cpcap_charger_irqs[i]);
+ if (error)
+ return error;
+ }
+
+ return 0;
+}
+
+static void cpcap_charger_init_optional_gpios(struct cpcap_charger_ddata *ddata)
+{
+ int i;
+
+ for (i = 0; i < 2; i++) {
+ ddata->gpio[i] = devm_gpiod_get_index(ddata->dev, "mode",
+ i, GPIOD_OUT_HIGH);
+ if (IS_ERR(ddata->gpio[i])) {
+ dev_info(ddata->dev, "no mode change GPIO%i: %li\n",
+ i, PTR_ERR(ddata->gpio[i]));
+ ddata->gpio[i] = NULL;
+ }
+ }
+}
+
+static int cpcap_charger_init_iio(struct cpcap_charger_ddata *ddata)
+{
+ const char * const names[CPCAP_CHARGER_IIO_NR] = {
+ "battdetb", "battp", "vbus", "chg_isense", "batti",
+ };
+ int error, i;
+
+ for (i = 0; i < CPCAP_CHARGER_IIO_NR; i++) {
+ ddata->channels[i] = devm_iio_channel_get(ddata->dev,
+ names[i]);
+ if (IS_ERR(ddata->channels[i])) {
+ error = PTR_ERR(ddata->channels[i]);
+ goto out_err;
+ }
+
+ if (!ddata->channels[i]->indio_dev) {
+ error = -ENXIO;
+ goto out_err;
+ }
+ }
+
+ return 0;
+
+out_err:
+ dev_err(ddata->dev, "could not initialize VBUS or ID IIO: %i\n",
+ error);
+
+ return error;
+}
+
+static const struct power_supply_desc cpcap_charger_usb_desc = {
+ .name = "cpcap_usb",
+ .type = POWER_SUPPLY_TYPE_USB,
+ .properties = cpcap_charger_props,
+ .num_properties = ARRAY_SIZE(cpcap_charger_props),
+ .get_property = cpcap_charger_get_property,
+};
+
+#ifdef CONFIG_OF
+static const struct of_device_id cpcap_charger_id_table[] = {
+ {
+ .compatible = "motorola,mapphone-cpcap-charger",
+ },
+ {},
+};
+MODULE_DEVICE_TABLE(of, cpcap_charger_id_table);
+#endif
+
+static int cpcap_charger_probe(struct platform_device *pdev)
+{
+ struct cpcap_charger_ddata *ddata;
+ const struct of_device_id *of_id;
+ int error;
+
+ of_id = of_match_device(of_match_ptr(cpcap_charger_id_table),
+ &pdev->dev);
+ if (!of_id)
+ return -EINVAL;
+
+ ddata = devm_kzalloc(&pdev->dev, sizeof(*ddata), GFP_KERNEL);
+ if (!ddata)
+ return -ENOMEM;
+
+ ddata->dev = &pdev->dev;
+
+ ddata->reg = dev_get_regmap(ddata->dev->parent, NULL);
+ if (!ddata->reg)
+ return -ENODEV;
+
+ INIT_LIST_HEAD(&ddata->irq_list);
+ INIT_DELAYED_WORK(&ddata->detect_work, cpcap_usb_detect);
+ INIT_DELAYED_WORK(&ddata->vbus_work, cpcap_charger_vbus_work);
+ platform_set_drvdata(pdev, ddata);
+
+ error = cpcap_charger_init_iio(ddata);
+ if (error)
+ return error;
+
+ atomic_set(&ddata->active, 1);
+
+ ddata->usb = devm_power_supply_register(ddata->dev,
+ &cpcap_charger_usb_desc,
+ NULL);
+ if (IS_ERR(ddata->usb)) {
+ error = PTR_ERR(ddata->usb);
+ dev_err(ddata->dev, "failed to register USB charger: %i\n",
+ error);
+
+ return error;
+ }
+
+ error = cpcap_usb_init_interrupts(pdev, ddata);
+ if (error)
+ return error;
+
+ ddata->comparator.set_vbus = cpcap_charger_set_vbus;
+ error = omap_usb2_set_comparator(&ddata->comparator);
+ if (error == -ENODEV) {
+ dev_info(ddata->dev, "charger needs phy, deferring probe\n");
+ return -EPROBE_DEFER;
+ }
+
+ cpcap_charger_init_optional_gpios(ddata);
+
+ schedule_delayed_work(&ddata->detect_work, 0);
+
+ return 0;
+}
+
+static int cpcap_charger_remove(struct platform_device *pdev)
+{
+ struct cpcap_charger_ddata *ddata = platform_get_drvdata(pdev);
+ int error;
+
+ atomic_set(&ddata->active, 0);
+ error = omap_usb2_set_comparator(NULL);
+ if (error)
+ dev_warn(ddata->dev, "could not clear USB comparator: %i\n",
+ error);
+
+ error = cpcap_charger_set_state(ddata, 0, 0, 0);
+ if (error)
+ dev_warn(ddata->dev, "could not clear charger: %i\n",
+ error);
+ cancel_delayed_work_sync(&ddata->vbus_work);
+ cancel_delayed_work_sync(&ddata->detect_work);
+
+ return 0;
+}
+
+static struct platform_driver cpcap_charger_driver = {
+ .probe = cpcap_charger_probe,
+ .driver = {
+ .name = "cpcap-charger",
+ .of_match_table = of_match_ptr(cpcap_charger_id_table),
+ },
+ .remove = cpcap_charger_remove,
+};
+module_platform_driver(cpcap_charger_driver);
+
+MODULE_AUTHOR("Tony Lindgren <tony@atomide.com>");
+MODULE_DESCRIPTION("CPCAP Battery Charger Interface driver");
+MODULE_LICENSE("GPL v2");
+MODULE_ALIAS("platform:cpcap-charger");
diff --git a/drivers/power/supply/lego_ev3_battery.c b/drivers/power/supply/lego_ev3_battery.c
new file mode 100644
index 0000000..7b993d6
--- /dev/null
+++ b/drivers/power/supply/lego_ev3_battery.c
@@ -0,0 +1,228 @@
+/*
+ * Battery driver for LEGO MINDSTORMS EV3
+ *
+ * Copyright (C) 2017 David Lechner <david@lechnology.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+
+ * This program is distributed "as is" WITHOUT ANY WARRANTY of any
+ * kind, whether express or implied; without even the implied warranty
+ * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/delay.h>
+#include <linux/err.h>
+#include <linux/gpio/consumer.h>
+#include <linux/iio/consumer.h>
+#include <linux/iio/types.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of_device.h>
+#include <linux/platform_device.h>
+#include <linux/power_supply.h>
+
+struct lego_ev3_battery {
+ struct iio_channel *iio_v;
+ struct iio_channel *iio_i;
+ struct gpio_desc *rechargeable_gpio;
+ struct power_supply *psy;
+ int technology;
+ int v_max;
+ int v_min;
+};
+
+static int lego_ev3_battery_get_property(struct power_supply *psy,
+ enum power_supply_property psp,
+ union power_supply_propval *val)
+{
+ struct lego_ev3_battery *batt = power_supply_get_drvdata(psy);
+ int val2;
+
+ switch (psp) {
+ case POWER_SUPPLY_PROP_TECHNOLOGY:
+ val->intval = batt->technology;
+ break;
+ case POWER_SUPPLY_PROP_VOLTAGE_NOW:
+ /* battery voltage is iio channel * 2 + Vce of transistor */
+ iio_read_channel_processed(batt->iio_v, &val->intval);
+ val->intval *= 2000;
+ val->intval += 200000;
+ /* plus adjust for shunt resistor drop */
+ iio_read_channel_processed(batt->iio_i, &val2);
+ val2 *= 1000;
+ val2 /= 15;
+ val->intval += val2;
+ break;
+ case POWER_SUPPLY_PROP_VOLTAGE_MAX_DESIGN:
+ val->intval = batt->v_max;
+ break;
+ case POWER_SUPPLY_PROP_VOLTAGE_MIN_DESIGN:
+ val->intval = batt->v_min;
+ break;
+ case POWER_SUPPLY_PROP_CURRENT_NOW:
+ /* battery current is iio channel / 15 / 0.05 ohms */
+ iio_read_channel_processed(batt->iio_i, &val->intval);
+ val->intval *= 20000;
+ val->intval /= 15;
+ break;
+ case POWER_SUPPLY_PROP_SCOPE:
+ val->intval = POWER_SUPPLY_SCOPE_SYSTEM;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int lego_ev3_battery_set_property(struct power_supply *psy,
+ enum power_supply_property psp,
+ const union power_supply_propval *val)
+{
+ struct lego_ev3_battery *batt = power_supply_get_drvdata(psy);
+
+ switch (psp) {
+ case POWER_SUPPLY_PROP_TECHNOLOGY:
+ /*
+ * Only allow changing technology from Unknown to NiMH. Li-ion
+ * batteries are automatically detected and should not be
+ * overridden. Rechargeable AA batteries, on the other hand,
+ * cannot be automatically detected, and so must be manually
+ * specified. This should only be set once during system init,
+ * so there is no mechanism to go back to Unknown.
+ */
+ if (batt->technology != POWER_SUPPLY_TECHNOLOGY_UNKNOWN)
+ return -EINVAL;
+ switch (val->intval) {
+ case POWER_SUPPLY_TECHNOLOGY_NiMH:
+ batt->technology = POWER_SUPPLY_TECHNOLOGY_NiMH;
+ batt->v_max = 7800000;
+ batt->v_min = 5400000;
+ break;
+ default:
+ return -EINVAL;
+ }
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int lego_ev3_battery_property_is_writeable(struct power_supply *psy,
+ enum power_supply_property psp)
+{
+ struct lego_ev3_battery *batt = power_supply_get_drvdata(psy);
+
+ return psp == POWER_SUPPLY_PROP_TECHNOLOGY &&
+ batt->technology == POWER_SUPPLY_TECHNOLOGY_UNKNOWN;
+}
+
+static enum power_supply_property lego_ev3_battery_props[] = {
+ POWER_SUPPLY_PROP_TECHNOLOGY,
+ POWER_SUPPLY_PROP_VOLTAGE_NOW,
+ POWER_SUPPLY_PROP_VOLTAGE_MAX_DESIGN,
+ POWER_SUPPLY_PROP_VOLTAGE_MIN_DESIGN,
+ POWER_SUPPLY_PROP_CURRENT_NOW,
+ POWER_SUPPLY_PROP_SCOPE,
+};
+
+static const struct power_supply_desc lego_ev3_battery_desc = {
+ .name = "lego-ev3-battery",
+ .type = POWER_SUPPLY_TYPE_BATTERY,
+ .properties = lego_ev3_battery_props,
+ .num_properties = ARRAY_SIZE(lego_ev3_battery_props),
+ .get_property = lego_ev3_battery_get_property,
+ .set_property = lego_ev3_battery_set_property,
+ .property_is_writeable = lego_ev3_battery_property_is_writeable,
+};
+
+static int lego_ev3_battery_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct lego_ev3_battery *batt;
+ struct power_supply_config psy_cfg = {};
+ int err;
+
+ batt = devm_kzalloc(dev, sizeof(*batt), GFP_KERNEL);
+ if (!batt)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, batt);
+
+ batt->iio_v = devm_iio_channel_get(dev, "voltage");
+ err = PTR_ERR_OR_ZERO(batt->iio_v);
+ if (err) {
+ if (err != -EPROBE_DEFER)
+ dev_err(dev, "Failed to get voltage iio channel\n");
+ return err;
+ }
+
+ batt->iio_i = devm_iio_channel_get(dev, "current");
+ err = PTR_ERR_OR_ZERO(batt->iio_i);
+ if (err) {
+ if (err != -EPROBE_DEFER)
+ dev_err(dev, "Failed to get current iio channel\n");
+ return err;
+ }
+
+ batt->rechargeable_gpio = devm_gpiod_get(dev, "rechargeable", GPIOD_IN);
+ err = PTR_ERR_OR_ZERO(batt->rechargeable_gpio);
+ if (err) {
+ if (err != -EPROBE_DEFER)
+ dev_err(dev, "Failed to get rechargeable gpio\n");
+ return err;
+ }
+
+ /*
+ * The rechargeable battery indication switch cannot be changed without
+ * removing the battery, so we only need to read it once.
+ */
+ if (gpiod_get_value(batt->rechargeable_gpio)) {
+ /* 2-cell Li-ion, 7.4V nominal */
+ batt->technology = POWER_SUPPLY_TECHNOLOGY_LION;
+ batt->v_max = 84000000;
+ batt->v_min = 60000000;
+ } else {
+ /* 6x AA Alkaline, 9V nominal */
+ batt->technology = POWER_SUPPLY_TECHNOLOGY_UNKNOWN;
+ batt->v_max = 90000000;
+ batt->v_min = 48000000;
+ }
+
+ psy_cfg.of_node = pdev->dev.of_node;
+ psy_cfg.drv_data = batt;
+
+ batt->psy = devm_power_supply_register(dev, &lego_ev3_battery_desc,
+ &psy_cfg);
+ err = PTR_ERR_OR_ZERO(batt->psy);
+ if (err) {
+ dev_err(dev, "failed to register power supply\n");
+ return err;
+ }
+
+ return 0;
+}
+
+static const struct of_device_id of_lego_ev3_battery_match[] = {
+ { .compatible = "lego,ev3-battery", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, of_lego_ev3_battery_match);
+
+static struct platform_driver lego_ev3_battery_driver = {
+ .driver = {
+ .name = "lego-ev3-battery",
+ .of_match_table = of_lego_ev3_battery_match,
+ },
+ .probe = lego_ev3_battery_probe,
+};
+module_platform_driver(lego_ev3_battery_driver);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("David Lechner <david@lechnology.com>");
+MODULE_DESCRIPTION("LEGO MINDSTORMS EV3 Battery Driver");
diff --git a/drivers/power/supply/lp8788-charger.c b/drivers/power/supply/lp8788-charger.c
index 509e2b3..677f7c4 100644
--- a/drivers/power/supply/lp8788-charger.c
+++ b/drivers/power/supply/lp8788-charger.c
@@ -651,7 +651,7 @@
{
struct lp8788_charger *pchg = dev_get_drvdata(dev);
char *stime[] = { "400ms", "5min", "10min", "15min",
- "20min", "25min", "30min" "No timeout" };
+ "20min", "25min", "30min", "No timeout" };
u8 val;
lp8788_read_byte(pchg->lp, LP8788_CHG_EOC, &val);
diff --git a/drivers/power/supply/ltc2941-battery-gauge.c b/drivers/power/supply/ltc2941-battery-gauge.c
index 4adf2ba..7efb908 100644
--- a/drivers/power/supply/ltc2941-battery-gauge.c
+++ b/drivers/power/supply/ltc2941-battery-gauge.c
@@ -9,6 +9,7 @@
*/
#include <linux/kernel.h>
#include <linux/module.h>
+#include <linux/of_device.h>
#include <linux/types.h>
#include <linux/errno.h>
#include <linux/swab.h>
@@ -61,7 +62,7 @@
struct power_supply *supply; /* Supply pointer */
struct power_supply_desc supply_desc; /* Supply description */
struct delayed_work work; /* Work scheduler */
- int num_regs; /* Number of registers (chip type) */
+ unsigned long num_regs; /* Number of registers (chip type) */
int charge; /* Last charge register content */
int r_sense; /* mOhm */
int Qlsb; /* nAh */
@@ -387,7 +388,7 @@
np = of_node_get(client->dev.of_node);
- info->num_regs = id->driver_data;
+ info->num_regs = (unsigned long)of_device_get_match_data(&client->dev);
info->supply_desc.name = np->name;
/* r_sense can be negative, when sense+ is connected to the battery
@@ -497,9 +498,23 @@
};
MODULE_DEVICE_TABLE(i2c, ltc294x_i2c_id);
+static const struct of_device_id ltc294x_i2c_of_match[] = {
+ {
+ .compatible = "lltc,ltc2941",
+ .data = (void *)LTC2941_NUM_REGS
+ },
+ {
+ .compatible = "lltc,ltc2943",
+ .data = (void *)LTC2943_NUM_REGS
+ },
+ { },
+};
+MODULE_DEVICE_TABLE(of, ltc294x_i2c_of_match);
+
static struct i2c_driver ltc294x_driver = {
.driver = {
.name = "LTC2941",
+ .of_match_table = ltc294x_i2c_of_match,
.pm = LTC294X_PM_OPS,
},
.probe = ltc294x_i2c_probe,
diff --git a/drivers/power/supply/max17040_battery.c b/drivers/power/supply/max17040_battery.c
index e7c3649..33c40f7 100644
--- a/drivers/power/supply/max17040_battery.c
+++ b/drivers/power/supply/max17040_battery.c
@@ -277,9 +277,17 @@
};
MODULE_DEVICE_TABLE(i2c, max17040_id);
+static const struct of_device_id max17040_of_match[] = {
+ { .compatible = "maxim,max17040" },
+ { .compatible = "maxim,max77836-battery" },
+ { },
+};
+MODULE_DEVICE_TABLE(of, max17040_of_match);
+
static struct i2c_driver max17040_i2c_driver = {
.driver = {
.name = "max17040",
+ .of_match_table = max17040_of_match,
.pm = MAX17040_PM_OPS,
},
.probe = max17040_probe,
diff --git a/drivers/power/supply/sbs-charger.c b/drivers/power/supply/sbs-charger.c
index 353765a..15947db 100644
--- a/drivers/power/supply/sbs-charger.c
+++ b/drivers/power/supply/sbs-charger.c
@@ -137,10 +137,7 @@
static bool sbs_readable_reg(struct device *dev, unsigned int reg)
{
- if (reg < SBS_CHARGER_REG_SPEC_INFO)
- return false;
- else
- return true;
+ return reg >= SBS_CHARGER_REG_SPEC_INFO;
}
static bool sbs_volatile_reg(struct device *dev, unsigned int reg)
diff --git a/drivers/power/supply/tps65217_charger.c b/drivers/power/supply/tps65217_charger.c
index 29b61e8..1f52340 100644
--- a/drivers/power/supply/tps65217_charger.c
+++ b/drivers/power/supply/tps65217_charger.c
@@ -58,8 +58,6 @@
{
int ret;
- dev_dbg(charger->dev, "%s\n", __func__);
-
/*
* tps65217 rev. G, p. 31 (see p. 32 for NTC schematic)
*
@@ -205,8 +203,6 @@
int ret;
int i;
- dev_dbg(&pdev->dev, "%s\n", __func__);
-
charger = devm_kzalloc(&pdev->dev, sizeof(*charger), GFP_KERNEL);
if (!charger)
return -ENOMEM;
diff --git a/drivers/power/supply/twl4030_charger.c b/drivers/power/supply/twl4030_charger.c
index bcd4dc3..990ff3d 100644
--- a/drivers/power/supply/twl4030_charger.c
+++ b/drivers/power/supply/twl4030_charger.c
@@ -1117,7 +1117,7 @@
return ret;
}
-static int __exit twl4030_bci_remove(struct platform_device *pdev)
+static int twl4030_bci_remove(struct platform_device *pdev)
{
struct twl4030_bci *bci = platform_get_drvdata(pdev);
@@ -1148,11 +1148,11 @@
static struct platform_driver twl4030_bci_driver = {
.probe = twl4030_bci_probe,
+ .remove = twl4030_bci_remove,
.driver = {
.name = "twl4030_bci",
.of_match_table = of_match_ptr(twl_bci_of_match),
},
- .remove = __exit_p(twl4030_bci_remove),
};
module_platform_driver(twl4030_bci_driver);
diff --git a/drivers/ptp/ptp_clock.c b/drivers/ptp/ptp_clock.c
index e814280..b774357 100644
--- a/drivers/ptp/ptp_clock.c
+++ b/drivers/ptp/ptp_clock.c
@@ -97,30 +97,26 @@
/* posix clock implementation */
-static int ptp_clock_getres(struct posix_clock *pc, struct timespec *tp)
+static int ptp_clock_getres(struct posix_clock *pc, struct timespec64 *tp)
{
tp->tv_sec = 0;
tp->tv_nsec = 1;
return 0;
}
-static int ptp_clock_settime(struct posix_clock *pc, const struct timespec *tp)
+static int ptp_clock_settime(struct posix_clock *pc, const struct timespec64 *tp)
{
struct ptp_clock *ptp = container_of(pc, struct ptp_clock, clock);
- struct timespec64 ts = timespec_to_timespec64(*tp);
- return ptp->info->settime64(ptp->info, &ts);
+ return ptp->info->settime64(ptp->info, tp);
}
-static int ptp_clock_gettime(struct posix_clock *pc, struct timespec *tp)
+static int ptp_clock_gettime(struct posix_clock *pc, struct timespec64 *tp)
{
struct ptp_clock *ptp = container_of(pc, struct ptp_clock, clock);
- struct timespec64 ts;
int err;
- err = ptp->info->gettime64(ptp->info, &ts);
- if (!err)
- *tp = timespec64_to_timespec(ts);
+ err = ptp->info->gettime64(ptp->info, tp);
return err;
}
@@ -133,7 +129,7 @@
ops = ptp->info;
if (tx->modes & ADJ_SETOFFSET) {
- struct timespec ts;
+ struct timespec64 ts;
ktime_t kt;
s64 delta;
@@ -146,7 +142,7 @@
if ((unsigned long) ts.tv_nsec >= NSEC_PER_SEC)
return -EINVAL;
- kt = timespec_to_ktime(ts);
+ kt = timespec64_to_ktime(ts);
delta = ktime_to_ns(kt);
err = ops->adjtime(ops, delta);
} else if (tx->modes & ADJ_FREQUENCY) {
diff --git a/drivers/ras/Makefile b/drivers/ras/Makefile
index d7f7334..7b26dd3 100644
--- a/drivers/ras/Makefile
+++ b/drivers/ras/Makefile
@@ -1 +1,2 @@
-obj-$(CONFIG_RAS) += ras.o debugfs.o
+obj-$(CONFIG_RAS) += ras.o debugfs.o
+obj-$(CONFIG_RAS_CEC) += cec.o
diff --git a/drivers/ras/cec.c b/drivers/ras/cec.c
new file mode 100644
index 0000000..6aab46d
--- /dev/null
+++ b/drivers/ras/cec.c
@@ -0,0 +1,532 @@
+#include <linux/mm.h>
+#include <linux/gfp.h>
+#include <linux/kernel.h>
+
+#include <asm/mce.h>
+
+#include "debugfs.h"
+
+/*
+ * RAS Correctable Errors Collector
+ *
+ * This is a simple gadget which collects correctable errors and counts their
+ * occurrence per physical page address.
+ *
+ * We've opted for possibly the simplest data structure to collect those - an
+ * array of the size of a memory page. It stores 512 u64's with the following
+ * structure:
+ *
+ * [63 ... PFN ... 12 | 11 ... generation ... 10 | 9 ... count ... 0]
+ *
+ * The generation in the two highest order bits is two bits which are set to 11b
+ * on every insertion. During the course of each entry's existence, the
+ * generation field gets decremented during spring cleaning to 10b, then 01b and
+ * then 00b.
+ *
+ * This way we're employing the natural numeric ordering to make sure that newly
+ * inserted/touched elements have higher 12-bit counts (which we've manufactured)
+ * and thus iterating over the array initially won't kick out those elements
+ * which were inserted last.
+ *
+ * Spring cleaning is what we do when we reach a certain number CLEAN_ELEMS of
+ * elements entered into the array, during which, we're decaying all elements.
+ * If, after decay, an element gets inserted again, its generation is set to 11b
+ * to make sure it has higher numerical count than other, older elements and
+ * thus emulate an an LRU-like behavior when deleting elements to free up space
+ * in the page.
+ *
+ * When an element reaches it's max count of count_threshold, we try to poison
+ * it by assuming that errors triggered count_threshold times in a single page
+ * are excessive and that page shouldn't be used anymore. count_threshold is
+ * initialized to COUNT_MASK which is the maximum.
+ *
+ * That error event entry causes cec_add_elem() to return !0 value and thus
+ * signal to its callers to log the error.
+ *
+ * To the question why we've chosen a page and moving elements around with
+ * memmove(), it is because it is a very simple structure to handle and max data
+ * movement is 4K which on highly optimized modern CPUs is almost unnoticeable.
+ * We wanted to avoid the pointer traversal of more complex structures like a
+ * linked list or some sort of a balancing search tree.
+ *
+ * Deleting an element takes O(n) but since it is only a single page, it should
+ * be fast enough and it shouldn't happen all too often depending on error
+ * patterns.
+ */
+
+#undef pr_fmt
+#define pr_fmt(fmt) "RAS: " fmt
+
+/*
+ * We use DECAY_BITS bits of PAGE_SHIFT bits for counting decay, i.e., how long
+ * elements have stayed in the array without having been accessed again.
+ */
+#define DECAY_BITS 2
+#define DECAY_MASK ((1ULL << DECAY_BITS) - 1)
+#define MAX_ELEMS (PAGE_SIZE / sizeof(u64))
+
+/*
+ * Threshold amount of inserted elements after which we start spring
+ * cleaning.
+ */
+#define CLEAN_ELEMS (MAX_ELEMS >> DECAY_BITS)
+
+/* Bits which count the number of errors happened in this 4K page. */
+#define COUNT_BITS (PAGE_SHIFT - DECAY_BITS)
+#define COUNT_MASK ((1ULL << COUNT_BITS) - 1)
+#define FULL_COUNT_MASK (PAGE_SIZE - 1)
+
+/*
+ * u64: [ 63 ... 12 | DECAY_BITS | COUNT_BITS ]
+ */
+
+#define PFN(e) ((e) >> PAGE_SHIFT)
+#define DECAY(e) (((e) >> COUNT_BITS) & DECAY_MASK)
+#define COUNT(e) ((unsigned int)(e) & COUNT_MASK)
+#define FULL_COUNT(e) ((e) & (PAGE_SIZE - 1))
+
+static struct ce_array {
+ u64 *array; /* container page */
+ unsigned int n; /* number of elements in the array */
+
+ unsigned int decay_count; /*
+ * number of element insertions/increments
+ * since the last spring cleaning.
+ */
+
+ u64 pfns_poisoned; /*
+ * number of PFNs which got poisoned.
+ */
+
+ u64 ces_entered; /*
+ * The number of correctable errors
+ * entered into the collector.
+ */
+
+ u64 decays_done; /*
+ * Times we did spring cleaning.
+ */
+
+ union {
+ struct {
+ __u32 disabled : 1, /* cmdline disabled */
+ __resv : 31;
+ };
+ __u32 flags;
+ };
+} ce_arr;
+
+static DEFINE_MUTEX(ce_mutex);
+static u64 dfs_pfn;
+
+/* Amount of errors after which we offline */
+static unsigned int count_threshold = COUNT_MASK;
+
+/*
+ * The timer "decays" element count each timer_interval which is 24hrs by
+ * default.
+ */
+
+#define CEC_TIMER_DEFAULT_INTERVAL 24 * 60 * 60 /* 24 hrs */
+#define CEC_TIMER_MIN_INTERVAL 1 * 60 * 60 /* 1h */
+#define CEC_TIMER_MAX_INTERVAL 30 * 24 * 60 * 60 /* one month */
+static struct timer_list cec_timer;
+static u64 timer_interval = CEC_TIMER_DEFAULT_INTERVAL;
+
+/*
+ * Decrement decay value. We're using DECAY_BITS bits to denote decay of an
+ * element in the array. On insertion and any access, it gets reset to max.
+ */
+static void do_spring_cleaning(struct ce_array *ca)
+{
+ int i;
+
+ for (i = 0; i < ca->n; i++) {
+ u8 decay = DECAY(ca->array[i]);
+
+ if (!decay)
+ continue;
+
+ decay--;
+
+ ca->array[i] &= ~(DECAY_MASK << COUNT_BITS);
+ ca->array[i] |= (decay << COUNT_BITS);
+ }
+ ca->decay_count = 0;
+ ca->decays_done++;
+}
+
+/*
+ * @interval in seconds
+ */
+static void cec_mod_timer(struct timer_list *t, unsigned long interval)
+{
+ unsigned long iv;
+
+ iv = interval * HZ + jiffies;
+
+ mod_timer(t, round_jiffies(iv));
+}
+
+static void cec_timer_fn(unsigned long data)
+{
+ struct ce_array *ca = (struct ce_array *)data;
+
+ do_spring_cleaning(ca);
+
+ cec_mod_timer(&cec_timer, timer_interval);
+}
+
+/*
+ * @to: index of the smallest element which is >= then @pfn.
+ *
+ * Return the index of the pfn if found, otherwise negative value.
+ */
+static int __find_elem(struct ce_array *ca, u64 pfn, unsigned int *to)
+{
+ u64 this_pfn;
+ int min = 0, max = ca->n;
+
+ while (min < max) {
+ int tmp = (max + min) >> 1;
+
+ this_pfn = PFN(ca->array[tmp]);
+
+ if (this_pfn < pfn)
+ min = tmp + 1;
+ else if (this_pfn > pfn)
+ max = tmp;
+ else {
+ min = tmp;
+ break;
+ }
+ }
+
+ if (to)
+ *to = min;
+
+ this_pfn = PFN(ca->array[min]);
+
+ if (this_pfn == pfn)
+ return min;
+
+ return -ENOKEY;
+}
+
+static int find_elem(struct ce_array *ca, u64 pfn, unsigned int *to)
+{
+ WARN_ON(!to);
+
+ if (!ca->n) {
+ *to = 0;
+ return -ENOKEY;
+ }
+ return __find_elem(ca, pfn, to);
+}
+
+static void del_elem(struct ce_array *ca, int idx)
+{
+ /* Save us a function call when deleting the last element. */
+ if (ca->n - (idx + 1))
+ memmove((void *)&ca->array[idx],
+ (void *)&ca->array[idx + 1],
+ (ca->n - (idx + 1)) * sizeof(u64));
+
+ ca->n--;
+}
+
+static u64 del_lru_elem_unlocked(struct ce_array *ca)
+{
+ unsigned int min = FULL_COUNT_MASK;
+ int i, min_idx = 0;
+
+ for (i = 0; i < ca->n; i++) {
+ unsigned int this = FULL_COUNT(ca->array[i]);
+
+ if (min > this) {
+ min = this;
+ min_idx = i;
+ }
+ }
+
+ del_elem(ca, min_idx);
+
+ return PFN(ca->array[min_idx]);
+}
+
+/*
+ * We return the 0th pfn in the error case under the assumption that it cannot
+ * be poisoned and excessive CEs in there are a serious deal anyway.
+ */
+static u64 __maybe_unused del_lru_elem(void)
+{
+ struct ce_array *ca = &ce_arr;
+ u64 pfn;
+
+ if (!ca->n)
+ return 0;
+
+ mutex_lock(&ce_mutex);
+ pfn = del_lru_elem_unlocked(ca);
+ mutex_unlock(&ce_mutex);
+
+ return pfn;
+}
+
+
+int cec_add_elem(u64 pfn)
+{
+ struct ce_array *ca = &ce_arr;
+ unsigned int to;
+ int count, ret = 0;
+
+ /*
+ * We can be called very early on the identify_cpu() path where we are
+ * not initialized yet. We ignore the error for simplicity.
+ */
+ if (!ce_arr.array || ce_arr.disabled)
+ return -ENODEV;
+
+ ca->ces_entered++;
+
+ mutex_lock(&ce_mutex);
+
+ if (ca->n == MAX_ELEMS)
+ WARN_ON(!del_lru_elem_unlocked(ca));
+
+ ret = find_elem(ca, pfn, &to);
+ if (ret < 0) {
+ /*
+ * Shift range [to-end] to make room for one more element.
+ */
+ memmove((void *)&ca->array[to + 1],
+ (void *)&ca->array[to],
+ (ca->n - to) * sizeof(u64));
+
+ ca->array[to] = (pfn << PAGE_SHIFT) |
+ (DECAY_MASK << COUNT_BITS) | 1;
+
+ ca->n++;
+
+ ret = 0;
+
+ goto decay;
+ }
+
+ count = COUNT(ca->array[to]);
+
+ if (count < count_threshold) {
+ ca->array[to] |= (DECAY_MASK << COUNT_BITS);
+ ca->array[to]++;
+
+ ret = 0;
+ } else {
+ u64 pfn = ca->array[to] >> PAGE_SHIFT;
+
+ if (!pfn_valid(pfn)) {
+ pr_warn("CEC: Invalid pfn: 0x%llx\n", pfn);
+ } else {
+ /* We have reached max count for this page, soft-offline it. */
+ pr_err("Soft-offlining pfn: 0x%llx\n", pfn);
+ memory_failure_queue(pfn, 0, MF_SOFT_OFFLINE);
+ ca->pfns_poisoned++;
+ }
+
+ del_elem(ca, to);
+
+ /*
+ * Return a >0 value to denote that we've reached the offlining
+ * threshold.
+ */
+ ret = 1;
+
+ goto unlock;
+ }
+
+decay:
+ ca->decay_count++;
+
+ if (ca->decay_count >= CLEAN_ELEMS)
+ do_spring_cleaning(ca);
+
+unlock:
+ mutex_unlock(&ce_mutex);
+
+ return ret;
+}
+
+static int u64_get(void *data, u64 *val)
+{
+ *val = *(u64 *)data;
+
+ return 0;
+}
+
+static int pfn_set(void *data, u64 val)
+{
+ *(u64 *)data = val;
+
+ return cec_add_elem(val);
+}
+
+DEFINE_DEBUGFS_ATTRIBUTE(pfn_ops, u64_get, pfn_set, "0x%llx\n");
+
+static int decay_interval_set(void *data, u64 val)
+{
+ *(u64 *)data = val;
+
+ if (val < CEC_TIMER_MIN_INTERVAL)
+ return -EINVAL;
+
+ if (val > CEC_TIMER_MAX_INTERVAL)
+ return -EINVAL;
+
+ timer_interval = val;
+
+ cec_mod_timer(&cec_timer, timer_interval);
+ return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(decay_interval_ops, u64_get, decay_interval_set, "%lld\n");
+
+static int count_threshold_set(void *data, u64 val)
+{
+ *(u64 *)data = val;
+
+ if (val > COUNT_MASK)
+ val = COUNT_MASK;
+
+ count_threshold = val;
+
+ return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(count_threshold_ops, u64_get, count_threshold_set, "%lld\n");
+
+static int array_dump(struct seq_file *m, void *v)
+{
+ struct ce_array *ca = &ce_arr;
+ u64 prev = 0;
+ int i;
+
+ mutex_lock(&ce_mutex);
+
+ seq_printf(m, "{ n: %d\n", ca->n);
+ for (i = 0; i < ca->n; i++) {
+ u64 this = PFN(ca->array[i]);
+
+ seq_printf(m, " %03d: [%016llx|%03llx]\n", i, this, FULL_COUNT(ca->array[i]));
+
+ WARN_ON(prev > this);
+
+ prev = this;
+ }
+
+ seq_printf(m, "}\n");
+
+ seq_printf(m, "Stats:\nCEs: %llu\nofflined pages: %llu\n",
+ ca->ces_entered, ca->pfns_poisoned);
+
+ seq_printf(m, "Flags: 0x%x\n", ca->flags);
+
+ seq_printf(m, "Timer interval: %lld seconds\n", timer_interval);
+ seq_printf(m, "Decays: %lld\n", ca->decays_done);
+
+ seq_printf(m, "Action threshold: %d\n", count_threshold);
+
+ mutex_unlock(&ce_mutex);
+
+ return 0;
+}
+
+static int array_open(struct inode *inode, struct file *filp)
+{
+ return single_open(filp, array_dump, NULL);
+}
+
+static const struct file_operations array_ops = {
+ .owner = THIS_MODULE,
+ .open = array_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static int __init create_debugfs_nodes(void)
+{
+ struct dentry *d, *pfn, *decay, *count, *array;
+
+ d = debugfs_create_dir("cec", ras_debugfs_dir);
+ if (!d) {
+ pr_warn("Error creating cec debugfs node!\n");
+ return -1;
+ }
+
+ pfn = debugfs_create_file("pfn", S_IRUSR | S_IWUSR, d, &dfs_pfn, &pfn_ops);
+ if (!pfn) {
+ pr_warn("Error creating pfn debugfs node!\n");
+ goto err;
+ }
+
+ array = debugfs_create_file("array", S_IRUSR, d, NULL, &array_ops);
+ if (!array) {
+ pr_warn("Error creating array debugfs node!\n");
+ goto err;
+ }
+
+ decay = debugfs_create_file("decay_interval", S_IRUSR | S_IWUSR, d,
+ &timer_interval, &decay_interval_ops);
+ if (!decay) {
+ pr_warn("Error creating decay_interval debugfs node!\n");
+ goto err;
+ }
+
+ count = debugfs_create_file("count_threshold", S_IRUSR | S_IWUSR, d,
+ &count_threshold, &count_threshold_ops);
+ if (!decay) {
+ pr_warn("Error creating count_threshold debugfs node!\n");
+ goto err;
+ }
+
+
+ return 0;
+
+err:
+ debugfs_remove_recursive(d);
+
+ return 1;
+}
+
+void __init cec_init(void)
+{
+ if (ce_arr.disabled)
+ return;
+
+ ce_arr.array = (void *)get_zeroed_page(GFP_KERNEL);
+ if (!ce_arr.array) {
+ pr_err("Error allocating CE array page!\n");
+ return;
+ }
+
+ if (create_debugfs_nodes())
+ return;
+
+ setup_timer(&cec_timer, cec_timer_fn, (unsigned long)&ce_arr);
+ cec_mod_timer(&cec_timer, CEC_TIMER_DEFAULT_INTERVAL);
+
+ pr_info("Correctable Errors collector initialized.\n");
+}
+
+int __init parse_cec_param(char *str)
+{
+ if (!str)
+ return 0;
+
+ if (*str == '=')
+ str++;
+
+ if (!strncmp(str, "cec_disable", 7))
+ ce_arr.disabled = 1;
+ else
+ return 0;
+
+ return 1;
+}
diff --git a/drivers/ras/debugfs.c b/drivers/ras/debugfs.c
index 0322acf..5016030 100644
--- a/drivers/ras/debugfs.c
+++ b/drivers/ras/debugfs.c
@@ -1,6 +1,6 @@
#include <linux/debugfs.h>
-static struct dentry *ras_debugfs_dir;
+struct dentry *ras_debugfs_dir;
static atomic_t trace_count = ATOMIC_INIT(0);
diff --git a/drivers/ras/debugfs.h b/drivers/ras/debugfs.h
new file mode 100644
index 0000000..db72e45
--- /dev/null
+++ b/drivers/ras/debugfs.h
@@ -0,0 +1,8 @@
+#ifndef __RAS_DEBUGFS_H__
+#define __RAS_DEBUGFS_H__
+
+#include <linux/debugfs.h>
+
+extern struct dentry *ras_debugfs_dir;
+
+#endif /* __RAS_DEBUGFS_H__ */
diff --git a/drivers/ras/ras.c b/drivers/ras/ras.c
index b67dd36..94f8038 100644
--- a/drivers/ras/ras.c
+++ b/drivers/ras/ras.c
@@ -27,3 +27,14 @@
EXPORT_TRACEPOINT_SYMBOL_GPL(extlog_mem_event);
#endif
EXPORT_TRACEPOINT_SYMBOL_GPL(mc_event);
+
+
+int __init parse_ras_param(char *str)
+{
+#ifdef CONFIG_RAS_CEC
+ parse_cec_param(str);
+#endif
+
+ return 1;
+}
+__setup("ras", parse_ras_param);
diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig
index 1dc43fc..faad69a 100644
--- a/drivers/remoteproc/Kconfig
+++ b/drivers/remoteproc/Kconfig
@@ -76,7 +76,7 @@
depends on OF && ARCH_QCOM
depends on REMOTEPROC
depends on QCOM_SMEM
- depends on RPMSG_QCOM_SMD || QCOM_SMD || (COMPILE_TEST && QCOM_SMD=n && RPMSG_QCOM_SMD=n)
+ depends on RPMSG_QCOM_SMD || (COMPILE_TEST && RPMSG_QCOM_SMD=n)
select MFD_SYSCON
select QCOM_MDT_LOADER
select QCOM_RPROC_COMMON
@@ -93,7 +93,7 @@
depends on OF && ARCH_QCOM
depends on QCOM_SMEM
depends on REMOTEPROC
- depends on RPMSG_QCOM_SMD || QCOM_SMD || (COMPILE_TEST && QCOM_SMD=n && RPMSG_QCOM_SMD=n)
+ depends on RPMSG_QCOM_SMD || (COMPILE_TEST && RPMSG_QCOM_SMD=n)
select MFD_SYSCON
select QCOM_RPROC_COMMON
select QCOM_SCM
@@ -104,7 +104,7 @@
config QCOM_WCNSS_PIL
tristate "Qualcomm WCNSS Peripheral Image Loader"
depends on OF && ARCH_QCOM
- depends on RPMSG_QCOM_SMD || QCOM_SMD || (COMPILE_TEST && QCOM_SMD=n && RPMSG_QCOM_SMD=n)
+ depends on RPMSG_QCOM_SMD || (COMPILE_TEST && RPMSG_QCOM_SMD=n)
depends on QCOM_SMEM
depends on REMOTEPROC
select QCOM_MDT_LOADER
diff --git a/drivers/rpmsg/Kconfig b/drivers/rpmsg/Kconfig
index f12ac0b..edc008f 100644
--- a/drivers/rpmsg/Kconfig
+++ b/drivers/rpmsg/Kconfig
@@ -16,7 +16,6 @@
config RPMSG_QCOM_SMD
tristate "Qualcomm Shared Memory Driver (SMD)"
depends on QCOM_SMEM
- depends on QCOM_SMD=n
select RPMSG
help
Say y here to enable support for the Qualcomm Shared Memory Driver
diff --git a/drivers/s390/block/dasd_3990_erp.c b/drivers/s390/block/dasd_3990_erp.c
index 774da20..107cd33 100644
--- a/drivers/s390/block/dasd_3990_erp.c
+++ b/drivers/s390/block/dasd_3990_erp.c
@@ -1052,8 +1052,9 @@
} else {
/* fatal error - set status to FAILED
internal error 09 - Command Reject */
- dev_err(&device->cdev->dev, "An error occurred in the DASD "
- "device driver, reason=%s\n", "09");
+ if (!test_bit(DASD_CQR_SUPPRESS_CR, &erp->flags))
+ dev_err(&device->cdev->dev,
+ "An error occurred in the DASD device driver, reason=09\n");
erp = dasd_3990_erp_cleanup(erp, DASD_CQR_FAILED);
}
diff --git a/drivers/s390/block/dasd_eckd.c b/drivers/s390/block/dasd_eckd.c
index 0b38217..122456e 100644
--- a/drivers/s390/block/dasd_eckd.c
+++ b/drivers/s390/block/dasd_eckd.c
@@ -4927,10 +4927,14 @@
dasd_eckd_dump_sense_tcw(device, req, irb);
} else {
/*
- * In some cases the 'No Record Found' error might be expected
- * and log messages shouldn't be written then. Check if the
- * according suppress bit is set.
+ * In some cases the 'Command Reject' or 'No Record Found'
+ * error might be expected and log messages shouldn't be
+ * written then. Check if the according suppress bit is set.
*/
+ if (sense && sense[0] & SNS0_CMD_REJECT &&
+ test_bit(DASD_CQR_SUPPRESS_CR, &req->flags))
+ return;
+
if (sense && sense[1] & SNS1_NO_REC_FOUND &&
test_bit(DASD_CQR_SUPPRESS_NRF, &req->flags))
return;
@@ -5172,6 +5176,10 @@
if (!device->block && private->lcu->pav == HYPER_PAV)
return -EOPNOTSUPP;
+ /* may not be supported by the storage server */
+ if (!(private->features.feature[14] & 0x80))
+ return -EOPNOTSUPP;
+
cqr = dasd_smalloc_request(DASD_ECKD_MAGIC, 1 /* PSF */ + 1 /* RSSD */,
sizeof(struct dasd_psf_prssd_data) + 1,
device);
@@ -5219,6 +5227,8 @@
cqr->buildclk = get_tod_clock();
cqr->status = DASD_CQR_FILLED;
+ /* the command might not be supported, suppress error message */
+ __set_bit(DASD_CQR_SUPPRESS_CR, &cqr->flags);
rc = dasd_sleep_on_interruptible(cqr);
if (rc == 0) {
*data = *host_access;
diff --git a/drivers/s390/block/dasd_int.h b/drivers/s390/block/dasd_int.h
index 518dba2..dca7cb1 100644
--- a/drivers/s390/block/dasd_int.h
+++ b/drivers/s390/block/dasd_int.h
@@ -239,11 +239,11 @@
*/
/*
* The following flags are used to suppress output of certain errors.
- * These flags should only be used for format checks!
*/
#define DASD_CQR_SUPPRESS_NRF 4 /* Suppress 'No Record Found' error */
#define DASD_CQR_SUPPRESS_FP 5 /* Suppress 'File Protected' error*/
#define DASD_CQR_SUPPRESS_IL 6 /* Suppress 'Incorrect Length' error */
+#define DASD_CQR_SUPPRESS_CR 7 /* Suppress 'Command Reject' error */
/* Signature for error recovery functions. */
typedef struct dasd_ccw_req *(*dasd_erp_fn_t) (struct dasd_ccw_req *);
diff --git a/drivers/s390/cio/Makefile b/drivers/s390/cio/Makefile
index 3ab9aed..bdf4752 100644
--- a/drivers/s390/cio/Makefile
+++ b/drivers/s390/cio/Makefile
@@ -17,3 +17,6 @@
qdio-objs := qdio_main.o qdio_thinint.o qdio_debug.o qdio_setup.o
obj-$(CONFIG_QDIO) += qdio.o
+
+vfio_ccw-objs += vfio_ccw_drv.o vfio_ccw_cp.o vfio_ccw_ops.o vfio_ccw_fsm.o
+obj-$(CONFIG_VFIO_CCW) += vfio_ccw.o
diff --git a/drivers/s390/cio/cio.c b/drivers/s390/cio/cio.c
index 1b35066..8921617 100644
--- a/drivers/s390/cio/cio.c
+++ b/drivers/s390/cio/cio.c
@@ -170,12 +170,14 @@
return ccode;
}
}
+EXPORT_SYMBOL_GPL(cio_start_key);
int
cio_start (struct subchannel *sch, struct ccw1 *cpa, __u8 lpm)
{
return cio_start_key(sch, cpa, lpm, PAGE_DEFAULT_KEY);
}
+EXPORT_SYMBOL_GPL(cio_start);
/*
* resume suspended I/O operation
@@ -208,6 +210,7 @@
return -ENODEV;
}
}
+EXPORT_SYMBOL_GPL(cio_resume);
/*
* halt I/O operation
@@ -241,6 +244,7 @@
return -ENODEV;
}
}
+EXPORT_SYMBOL_GPL(cio_halt);
/*
* Clear I/O operation
@@ -271,6 +275,7 @@
return -ENODEV;
}
}
+EXPORT_SYMBOL_GPL(cio_clear);
/*
* Function: cio_cancel
@@ -308,7 +313,68 @@
return -ENODEV;
}
}
+EXPORT_SYMBOL_GPL(cio_cancel);
+/**
+ * cio_cancel_halt_clear - Cancel running I/O by performing cancel, halt
+ * and clear ordinally if subchannel is valid.
+ * @sch: subchannel on which to perform the cancel_halt_clear operation
+ * @iretry: the number of the times remained to retry the next operation
+ *
+ * This should be called repeatedly since halt/clear are asynchronous
+ * operations. We do one try with cio_cancel, three tries with cio_halt,
+ * 255 tries with cio_clear. The caller should initialize @iretry with
+ * the value 255 for its first call to this, and keep using the same
+ * @iretry in the subsequent calls until it gets a non -EBUSY return.
+ *
+ * Returns 0 if device now idle, -ENODEV for device not operational,
+ * -EBUSY if an interrupt is expected (either from halt/clear or from a
+ * status pending), and -EIO if out of retries.
+ */
+int cio_cancel_halt_clear(struct subchannel *sch, int *iretry)
+{
+ int ret;
+
+ if (cio_update_schib(sch))
+ return -ENODEV;
+ if (!sch->schib.pmcw.ena)
+ /* Not operational -> done. */
+ return 0;
+ /* Stage 1: cancel io. */
+ if (!(scsw_actl(&sch->schib.scsw) & SCSW_ACTL_HALT_PEND) &&
+ !(scsw_actl(&sch->schib.scsw) & SCSW_ACTL_CLEAR_PEND)) {
+ if (!scsw_is_tm(&sch->schib.scsw)) {
+ ret = cio_cancel(sch);
+ if (ret != -EINVAL)
+ return ret;
+ }
+ /*
+ * Cancel io unsuccessful or not applicable (transport mode).
+ * Continue with asynchronous instructions.
+ */
+ *iretry = 3; /* 3 halt retries. */
+ }
+ /* Stage 2: halt io. */
+ if (!(scsw_actl(&sch->schib.scsw) & SCSW_ACTL_CLEAR_PEND)) {
+ if (*iretry) {
+ *iretry -= 1;
+ ret = cio_halt(sch);
+ if (ret != -EBUSY)
+ return (ret == 0) ? -EBUSY : ret;
+ }
+ /* Halt io unsuccessful. */
+ *iretry = 255; /* 255 clear retries. */
+ }
+ /* Stage 3: clear io. */
+ if (*iretry) {
+ *iretry -= 1;
+ ret = cio_clear(sch);
+ return (ret == 0) ? -EBUSY : ret;
+ }
+ /* Function was unsuccessful */
+ return -EIO;
+}
+EXPORT_SYMBOL_GPL(cio_cancel_halt_clear);
static void cio_apply_config(struct subchannel *sch, struct schib *schib)
{
@@ -382,6 +448,7 @@
}
return ret;
}
+EXPORT_SYMBOL_GPL(cio_commit_config);
/**
* cio_update_schib - Perform stsch and update schib if subchannel is valid.
@@ -987,6 +1054,7 @@
return cio_start_handle_notoper(sch, lpm);
}
}
+EXPORT_SYMBOL_GPL(cio_tm_start_key);
/**
* cio_tm_intrg - perform interrogate function
@@ -1012,3 +1080,4 @@
return -ENODEV;
}
}
+EXPORT_SYMBOL_GPL(cio_tm_intrg);
diff --git a/drivers/s390/cio/cio.h b/drivers/s390/cio/cio.h
index f0e57ae..939596d 100644
--- a/drivers/s390/cio/cio.h
+++ b/drivers/s390/cio/cio.h
@@ -123,6 +123,7 @@
extern int cio_disable_subchannel (struct subchannel *);
extern int cio_cancel (struct subchannel *);
extern int cio_clear (struct subchannel *);
+extern int cio_cancel_halt_clear(struct subchannel *, int *);
extern int cio_resume (struct subchannel *);
extern int cio_halt (struct subchannel *);
extern int cio_start (struct subchannel *, struct ccw1 *, __u8);
diff --git a/drivers/s390/cio/device_fsm.c b/drivers/s390/cio/device_fsm.c
index 9afb5ce..12016e3 100644
--- a/drivers/s390/cio/device_fsm.c
+++ b/drivers/s390/cio/device_fsm.c
@@ -124,14 +124,6 @@
add_timer(&cdev->private->timer);
}
-/*
- * Cancel running i/o. This is called repeatedly since halt/clear are
- * asynchronous operations. We do one try with cio_cancel, two tries
- * with cio_halt, 255 tries with cio_clear. If everythings fails panic.
- * Returns 0 if device now idle, -ENODEV for device not operational and
- * -EBUSY if an interrupt is expected (either from halt/clear or from a
- * status pending).
- */
int
ccw_device_cancel_halt_clear(struct ccw_device *cdev)
{
@@ -139,44 +131,14 @@
int ret;
sch = to_subchannel(cdev->dev.parent);
- if (cio_update_schib(sch))
- return -ENODEV;
- if (!sch->schib.pmcw.ena)
- /* Not operational -> done. */
- return 0;
- /* Stage 1: cancel io. */
- if (!(scsw_actl(&sch->schib.scsw) & SCSW_ACTL_HALT_PEND) &&
- !(scsw_actl(&sch->schib.scsw) & SCSW_ACTL_CLEAR_PEND)) {
- if (!scsw_is_tm(&sch->schib.scsw)) {
- ret = cio_cancel(sch);
- if (ret != -EINVAL)
- return ret;
- }
- /* cancel io unsuccessful or not applicable (transport mode).
- * Continue with asynchronous instructions. */
- cdev->private->iretry = 3; /* 3 halt retries. */
- }
- if (!(scsw_actl(&sch->schib.scsw) & SCSW_ACTL_CLEAR_PEND)) {
- /* Stage 2: halt io. */
- if (cdev->private->iretry) {
- cdev->private->iretry--;
- ret = cio_halt(sch);
- if (ret != -EBUSY)
- return (ret == 0) ? -EBUSY : ret;
- }
- /* halt io unsuccessful. */
- cdev->private->iretry = 255; /* 255 clear retries. */
- }
- /* Stage 3: clear io. */
- if (cdev->private->iretry) {
- cdev->private->iretry--;
- ret = cio_clear (sch);
- return (ret == 0) ? -EBUSY : ret;
- }
- /* Function was unsuccessful */
- CIO_MSG_EVENT(0, "0.%x.%04x: could not stop I/O\n",
- cdev->private->dev_id.ssid, cdev->private->dev_id.devno);
- return -EIO;
+ ret = cio_cancel_halt_clear(sch, &cdev->private->iretry);
+
+ if (ret == -EIO)
+ CIO_MSG_EVENT(0, "0.%x.%04x: could not stop I/O\n",
+ cdev->private->dev_id.ssid,
+ cdev->private->dev_id.devno);
+
+ return ret;
}
void ccw_device_update_sense_data(struct ccw_device *cdev)
diff --git a/drivers/s390/cio/vfio_ccw_cp.c b/drivers/s390/cio/vfio_ccw_cp.c
new file mode 100644
index 0000000..ba6ac83
--- /dev/null
+++ b/drivers/s390/cio/vfio_ccw_cp.c
@@ -0,0 +1,842 @@
+/*
+ * channel program interfaces
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author(s): Dong Jia Shi <bjsdjshi@linux.vnet.ibm.com>
+ * Xiao Feng Ren <renxiaof@linux.vnet.ibm.com>
+ */
+
+#include <linux/mm.h>
+#include <linux/slab.h>
+#include <linux/iommu.h>
+#include <linux/vfio.h>
+#include <asm/idals.h>
+
+#include "vfio_ccw_cp.h"
+
+/*
+ * Max length for ccw chain.
+ * XXX: Limit to 256, need to check more?
+ */
+#define CCWCHAIN_LEN_MAX 256
+
+struct pfn_array {
+ unsigned long pa_iova;
+ unsigned long *pa_iova_pfn;
+ unsigned long *pa_pfn;
+ int pa_nr;
+};
+
+struct pfn_array_table {
+ struct pfn_array *pat_pa;
+ int pat_nr;
+};
+
+struct ccwchain {
+ struct list_head next;
+ struct ccw1 *ch_ccw;
+ /* Guest physical address of the current chain. */
+ u64 ch_iova;
+ /* Count of the valid ccws in chain. */
+ int ch_len;
+ /* Pinned PAGEs for the original data. */
+ struct pfn_array_table *ch_pat;
+};
+
+/*
+ * pfn_array_pin() - pin user pages in memory
+ * @pa: pfn_array on which to perform the operation
+ * @mdev: the mediated device to perform pin/unpin operations
+ *
+ * Attempt to pin user pages in memory.
+ *
+ * Usage of pfn_array:
+ * @pa->pa_iova starting guest physical I/O address. Assigned by caller.
+ * @pa->pa_iova_pfn array that stores PFNs of the pages need to pin. Allocated
+ * by caller.
+ * @pa->pa_pfn array that receives PFNs of the pages pinned. Allocated by
+ * caller.
+ * @pa->pa_nr number of pages from @pa->pa_iova to pin. Assigned by
+ * caller.
+ * number of pages pinned. Assigned by callee.
+ *
+ * Returns:
+ * Number of pages pinned on success.
+ * If @pa->pa_nr is 0 or negative, returns 0.
+ * If no pages were pinned, returns -errno.
+ */
+static int pfn_array_pin(struct pfn_array *pa, struct device *mdev)
+{
+ int i, ret;
+
+ if (pa->pa_nr <= 0) {
+ pa->pa_nr = 0;
+ return 0;
+ }
+
+ pa->pa_iova_pfn[0] = pa->pa_iova >> PAGE_SHIFT;
+ for (i = 1; i < pa->pa_nr; i++)
+ pa->pa_iova_pfn[i] = pa->pa_iova_pfn[i - 1] + 1;
+
+ ret = vfio_pin_pages(mdev, pa->pa_iova_pfn, pa->pa_nr,
+ IOMMU_READ | IOMMU_WRITE, pa->pa_pfn);
+
+ if (ret > 0 && ret != pa->pa_nr) {
+ vfio_unpin_pages(mdev, pa->pa_iova_pfn, ret);
+ pa->pa_nr = 0;
+ return 0;
+ }
+
+ return ret;
+}
+
+/* Unpin the pages before releasing the memory. */
+static void pfn_array_unpin_free(struct pfn_array *pa, struct device *mdev)
+{
+ vfio_unpin_pages(mdev, pa->pa_iova_pfn, pa->pa_nr);
+ pa->pa_nr = 0;
+ kfree(pa->pa_iova_pfn);
+}
+
+/* Alloc memory for PFNs, then pin pages with them. */
+static int pfn_array_alloc_pin(struct pfn_array *pa, struct device *mdev,
+ u64 iova, unsigned int len)
+{
+ int ret = 0;
+
+ if (!len || pa->pa_nr)
+ return -EINVAL;
+
+ pa->pa_iova = iova;
+
+ pa->pa_nr = ((iova & ~PAGE_MASK) + len + (PAGE_SIZE - 1)) >> PAGE_SHIFT;
+ if (!pa->pa_nr)
+ return -EINVAL;
+
+ pa->pa_iova_pfn = kcalloc(pa->pa_nr,
+ sizeof(*pa->pa_iova_pfn) +
+ sizeof(*pa->pa_pfn),
+ GFP_KERNEL);
+ if (unlikely(!pa->pa_iova_pfn))
+ return -ENOMEM;
+ pa->pa_pfn = pa->pa_iova_pfn + pa->pa_nr;
+
+ ret = pfn_array_pin(pa, mdev);
+
+ if (ret > 0)
+ return ret;
+ else if (!ret)
+ ret = -EINVAL;
+
+ kfree(pa->pa_iova_pfn);
+
+ return ret;
+}
+
+static int pfn_array_table_init(struct pfn_array_table *pat, int nr)
+{
+ pat->pat_pa = kcalloc(nr, sizeof(*pat->pat_pa), GFP_KERNEL);
+ if (unlikely(ZERO_OR_NULL_PTR(pat->pat_pa))) {
+ pat->pat_nr = 0;
+ return -ENOMEM;
+ }
+
+ pat->pat_nr = nr;
+
+ return 0;
+}
+
+static void pfn_array_table_unpin_free(struct pfn_array_table *pat,
+ struct device *mdev)
+{
+ int i;
+
+ for (i = 0; i < pat->pat_nr; i++)
+ pfn_array_unpin_free(pat->pat_pa + i, mdev);
+
+ if (pat->pat_nr) {
+ kfree(pat->pat_pa);
+ pat->pat_pa = NULL;
+ pat->pat_nr = 0;
+ }
+}
+
+static bool pfn_array_table_iova_pinned(struct pfn_array_table *pat,
+ unsigned long iova)
+{
+ struct pfn_array *pa = pat->pat_pa;
+ unsigned long iova_pfn = iova >> PAGE_SHIFT;
+ int i, j;
+
+ for (i = 0; i < pat->pat_nr; i++, pa++)
+ for (j = 0; j < pa->pa_nr; j++)
+ if (pa->pa_iova_pfn[i] == iova_pfn)
+ return true;
+
+ return false;
+}
+/* Create the list idal words for a pfn_array_table. */
+static inline void pfn_array_table_idal_create_words(
+ struct pfn_array_table *pat,
+ unsigned long *idaws)
+{
+ struct pfn_array *pa;
+ int i, j, k;
+
+ /*
+ * Idal words (execept the first one) rely on the memory being 4k
+ * aligned. If a user virtual address is 4K aligned, then it's
+ * corresponding kernel physical address will also be 4K aligned. Thus
+ * there will be no problem here to simply use the phys to create an
+ * idaw.
+ */
+ k = 0;
+ for (i = 0; i < pat->pat_nr; i++) {
+ pa = pat->pat_pa + i;
+ for (j = 0; j < pa->pa_nr; j++) {
+ idaws[k] = pa->pa_pfn[j] << PAGE_SHIFT;
+ if (k == 0)
+ idaws[k] += pa->pa_iova & (PAGE_SIZE - 1);
+ k++;
+ }
+ }
+}
+
+
+/*
+ * Within the domain (@mdev), copy @n bytes from a guest physical
+ * address (@iova) to a host physical address (@to).
+ */
+static long copy_from_iova(struct device *mdev,
+ void *to, u64 iova,
+ unsigned long n)
+{
+ struct pfn_array pa = {0};
+ u64 from;
+ int i, ret;
+ unsigned long l, m;
+
+ ret = pfn_array_alloc_pin(&pa, mdev, iova, n);
+ if (ret <= 0)
+ return ret;
+
+ l = n;
+ for (i = 0; i < pa.pa_nr; i++) {
+ from = pa.pa_pfn[i] << PAGE_SHIFT;
+ m = PAGE_SIZE;
+ if (i == 0) {
+ from += iova & (PAGE_SIZE - 1);
+ m -= iova & (PAGE_SIZE - 1);
+ }
+
+ m = min(l, m);
+ memcpy(to + (n - l), (void *)from, m);
+
+ l -= m;
+ if (l == 0)
+ break;
+ }
+
+ pfn_array_unpin_free(&pa, mdev);
+
+ return l;
+}
+
+static long copy_ccw_from_iova(struct channel_program *cp,
+ struct ccw1 *to, u64 iova,
+ unsigned long len)
+{
+ struct ccw0 ccw0;
+ struct ccw1 *pccw1;
+ int ret;
+ int i;
+
+ ret = copy_from_iova(cp->mdev, to, iova, len * sizeof(struct ccw1));
+ if (ret)
+ return ret;
+
+ if (!cp->orb.cmd.fmt) {
+ pccw1 = to;
+ for (i = 0; i < len; i++) {
+ ccw0 = *(struct ccw0 *)pccw1;
+ if ((pccw1->cmd_code & 0x0f) == CCW_CMD_TIC) {
+ pccw1->cmd_code = CCW_CMD_TIC;
+ pccw1->flags = 0;
+ pccw1->count = 0;
+ } else {
+ pccw1->cmd_code = ccw0.cmd_code;
+ pccw1->flags = ccw0.flags;
+ pccw1->count = ccw0.count;
+ }
+ pccw1->cda = ccw0.cda;
+ pccw1++;
+ }
+ }
+
+ return ret;
+}
+
+/*
+ * Helpers to operate ccwchain.
+ */
+#define ccw_is_test(_ccw) (((_ccw)->cmd_code & 0x0F) == 0)
+
+#define ccw_is_noop(_ccw) ((_ccw)->cmd_code == CCW_CMD_NOOP)
+
+#define ccw_is_tic(_ccw) ((_ccw)->cmd_code == CCW_CMD_TIC)
+
+#define ccw_is_idal(_ccw) ((_ccw)->flags & CCW_FLAG_IDA)
+
+
+#define ccw_is_chain(_ccw) ((_ccw)->flags & (CCW_FLAG_CC | CCW_FLAG_DC))
+
+static struct ccwchain *ccwchain_alloc(struct channel_program *cp, int len)
+{
+ struct ccwchain *chain;
+ void *data;
+ size_t size;
+
+ /* Make ccw address aligned to 8. */
+ size = ((sizeof(*chain) + 7L) & -8L) +
+ sizeof(*chain->ch_ccw) * len +
+ sizeof(*chain->ch_pat) * len;
+ chain = kzalloc(size, GFP_DMA | GFP_KERNEL);
+ if (!chain)
+ return NULL;
+
+ data = (u8 *)chain + ((sizeof(*chain) + 7L) & -8L);
+ chain->ch_ccw = (struct ccw1 *)data;
+
+ data = (u8 *)(chain->ch_ccw) + sizeof(*chain->ch_ccw) * len;
+ chain->ch_pat = (struct pfn_array_table *)data;
+
+ chain->ch_len = len;
+
+ list_add_tail(&chain->next, &cp->ccwchain_list);
+
+ return chain;
+}
+
+static void ccwchain_free(struct ccwchain *chain)
+{
+ list_del(&chain->next);
+ kfree(chain);
+}
+
+/* Free resource for a ccw that allocated memory for its cda. */
+static void ccwchain_cda_free(struct ccwchain *chain, int idx)
+{
+ struct ccw1 *ccw = chain->ch_ccw + idx;
+
+ if (!ccw->count)
+ return;
+
+ kfree((void *)(u64)ccw->cda);
+}
+
+/* Unpin the pages then free the memory resources. */
+static void cp_unpin_free(struct channel_program *cp)
+{
+ struct ccwchain *chain, *temp;
+ int i;
+
+ list_for_each_entry_safe(chain, temp, &cp->ccwchain_list, next) {
+ for (i = 0; i < chain->ch_len; i++) {
+ pfn_array_table_unpin_free(chain->ch_pat + i,
+ cp->mdev);
+ ccwchain_cda_free(chain, i);
+ }
+ ccwchain_free(chain);
+ }
+}
+
+/**
+ * ccwchain_calc_length - calculate the length of the ccw chain.
+ * @iova: guest physical address of the target ccw chain
+ * @cp: channel_program on which to perform the operation
+ *
+ * This is the chain length not considering any TICs.
+ * You need to do a new round for each TIC target.
+ *
+ * Returns: the length of the ccw chain or -errno.
+ */
+static int ccwchain_calc_length(u64 iova, struct channel_program *cp)
+{
+ struct ccw1 *ccw, *p;
+ int cnt;
+
+ /*
+ * Copy current chain from guest to host kernel.
+ * Currently the chain length is limited to CCWCHAIN_LEN_MAX (256).
+ * So copying 2K is enough (safe).
+ */
+ p = ccw = kcalloc(CCWCHAIN_LEN_MAX, sizeof(*ccw), GFP_KERNEL);
+ if (!ccw)
+ return -ENOMEM;
+
+ cnt = copy_ccw_from_iova(cp, ccw, iova, CCWCHAIN_LEN_MAX);
+ if (cnt) {
+ kfree(ccw);
+ return cnt;
+ }
+
+ cnt = 0;
+ do {
+ cnt++;
+
+ if ((!ccw_is_chain(ccw)) && (!ccw_is_tic(ccw)))
+ break;
+
+ ccw++;
+ } while (cnt < CCWCHAIN_LEN_MAX + 1);
+
+ if (cnt == CCWCHAIN_LEN_MAX + 1)
+ cnt = -EINVAL;
+
+ kfree(p);
+ return cnt;
+}
+
+static int tic_target_chain_exists(struct ccw1 *tic, struct channel_program *cp)
+{
+ struct ccwchain *chain;
+ u32 ccw_head, ccw_tail;
+
+ list_for_each_entry(chain, &cp->ccwchain_list, next) {
+ ccw_head = chain->ch_iova;
+ ccw_tail = ccw_head + (chain->ch_len - 1) * sizeof(struct ccw1);
+
+ if ((ccw_head <= tic->cda) && (tic->cda <= ccw_tail))
+ return 1;
+ }
+
+ return 0;
+}
+
+static int ccwchain_loop_tic(struct ccwchain *chain,
+ struct channel_program *cp);
+
+static int ccwchain_handle_tic(struct ccw1 *tic, struct channel_program *cp)
+{
+ struct ccwchain *chain;
+ int len, ret;
+
+ /* May transfer to an existing chain. */
+ if (tic_target_chain_exists(tic, cp))
+ return 0;
+
+ /* Get chain length. */
+ len = ccwchain_calc_length(tic->cda, cp);
+ if (len < 0)
+ return len;
+
+ /* Need alloc a new chain for this one. */
+ chain = ccwchain_alloc(cp, len);
+ if (!chain)
+ return -ENOMEM;
+ chain->ch_iova = tic->cda;
+
+ /* Copy the new chain from user. */
+ ret = copy_ccw_from_iova(cp, chain->ch_ccw, tic->cda, len);
+ if (ret) {
+ ccwchain_free(chain);
+ return ret;
+ }
+
+ /* Loop for tics on this new chain. */
+ return ccwchain_loop_tic(chain, cp);
+}
+
+/* Loop for TICs. */
+static int ccwchain_loop_tic(struct ccwchain *chain, struct channel_program *cp)
+{
+ struct ccw1 *tic;
+ int i, ret;
+
+ for (i = 0; i < chain->ch_len; i++) {
+ tic = chain->ch_ccw + i;
+
+ if (!ccw_is_tic(tic))
+ continue;
+
+ ret = ccwchain_handle_tic(tic, cp);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int ccwchain_fetch_tic(struct ccwchain *chain,
+ int idx,
+ struct channel_program *cp)
+{
+ struct ccw1 *ccw = chain->ch_ccw + idx;
+ struct ccwchain *iter;
+ u32 ccw_head, ccw_tail;
+
+ list_for_each_entry(iter, &cp->ccwchain_list, next) {
+ ccw_head = iter->ch_iova;
+ ccw_tail = ccw_head + (iter->ch_len - 1) * sizeof(struct ccw1);
+
+ if ((ccw_head <= ccw->cda) && (ccw->cda <= ccw_tail)) {
+ ccw->cda = (__u32) (addr_t) (iter->ch_ccw +
+ (ccw->cda - ccw_head));
+ return 0;
+ }
+ }
+
+ return -EFAULT;
+}
+
+static int ccwchain_fetch_direct(struct ccwchain *chain,
+ int idx,
+ struct channel_program *cp)
+{
+ struct ccw1 *ccw;
+ struct pfn_array_table *pat;
+ unsigned long *idaws;
+ int idaw_nr;
+
+ ccw = chain->ch_ccw + idx;
+
+ /*
+ * Pin data page(s) in memory.
+ * The number of pages actually is the count of the idaws which will be
+ * needed when translating a direct ccw to a idal ccw.
+ */
+ pat = chain->ch_pat + idx;
+ if (pfn_array_table_init(pat, 1))
+ return -ENOMEM;
+ idaw_nr = pfn_array_alloc_pin(pat->pat_pa, cp->mdev,
+ ccw->cda, ccw->count);
+ if (idaw_nr < 0)
+ return idaw_nr;
+
+ /* Translate this direct ccw to a idal ccw. */
+ idaws = kcalloc(idaw_nr, sizeof(*idaws), GFP_DMA | GFP_KERNEL);
+ if (!idaws) {
+ pfn_array_table_unpin_free(pat, cp->mdev);
+ return -ENOMEM;
+ }
+ ccw->cda = (__u32) virt_to_phys(idaws);
+ ccw->flags |= CCW_FLAG_IDA;
+
+ pfn_array_table_idal_create_words(pat, idaws);
+
+ return 0;
+}
+
+static int ccwchain_fetch_idal(struct ccwchain *chain,
+ int idx,
+ struct channel_program *cp)
+{
+ struct ccw1 *ccw;
+ struct pfn_array_table *pat;
+ unsigned long *idaws;
+ u64 idaw_iova;
+ unsigned int idaw_nr, idaw_len;
+ int i, ret;
+
+ ccw = chain->ch_ccw + idx;
+
+ /* Calculate size of idaws. */
+ ret = copy_from_iova(cp->mdev, &idaw_iova, ccw->cda, sizeof(idaw_iova));
+ if (ret)
+ return ret;
+ idaw_nr = idal_nr_words((void *)(idaw_iova), ccw->count);
+ idaw_len = idaw_nr * sizeof(*idaws);
+
+ /* Pin data page(s) in memory. */
+ pat = chain->ch_pat + idx;
+ ret = pfn_array_table_init(pat, idaw_nr);
+ if (ret)
+ return ret;
+
+ /* Translate idal ccw to use new allocated idaws. */
+ idaws = kzalloc(idaw_len, GFP_DMA | GFP_KERNEL);
+ if (!idaws) {
+ ret = -ENOMEM;
+ goto out_unpin;
+ }
+
+ ret = copy_from_iova(cp->mdev, idaws, ccw->cda, idaw_len);
+ if (ret)
+ goto out_free_idaws;
+
+ ccw->cda = virt_to_phys(idaws);
+
+ for (i = 0; i < idaw_nr; i++) {
+ idaw_iova = *(idaws + i);
+ if (IS_ERR_VALUE(idaw_iova)) {
+ ret = -EFAULT;
+ goto out_free_idaws;
+ }
+
+ ret = pfn_array_alloc_pin(pat->pat_pa + i, cp->mdev,
+ idaw_iova, 1);
+ if (ret < 0)
+ goto out_free_idaws;
+ }
+
+ pfn_array_table_idal_create_words(pat, idaws);
+
+ return 0;
+
+out_free_idaws:
+ kfree(idaws);
+out_unpin:
+ pfn_array_table_unpin_free(pat, cp->mdev);
+ return ret;
+}
+
+/*
+ * Fetch one ccw.
+ * To reduce memory copy, we'll pin the cda page in memory,
+ * and to get rid of the cda 2G limitiaion of ccw1, we'll translate
+ * direct ccws to idal ccws.
+ */
+static int ccwchain_fetch_one(struct ccwchain *chain,
+ int idx,
+ struct channel_program *cp)
+{
+ struct ccw1 *ccw = chain->ch_ccw + idx;
+
+ if (ccw_is_test(ccw) || ccw_is_noop(ccw))
+ return 0;
+
+ if (ccw_is_tic(ccw))
+ return ccwchain_fetch_tic(chain, idx, cp);
+
+ if (ccw_is_idal(ccw))
+ return ccwchain_fetch_idal(chain, idx, cp);
+
+ return ccwchain_fetch_direct(chain, idx, cp);
+}
+
+/**
+ * cp_init() - allocate ccwchains for a channel program.
+ * @cp: channel_program on which to perform the operation
+ * @mdev: the mediated device to perform pin/unpin operations
+ * @orb: control block for the channel program from the guest
+ *
+ * This creates one or more ccwchain(s), and copies the raw data of
+ * the target channel program from @orb->cmd.iova to the new ccwchain(s).
+ *
+ * Limitations:
+ * 1. Supports only prefetch enabled mode.
+ * 2. Supports idal(c64) ccw chaining.
+ * 3. Supports 4k idaw.
+ *
+ * Returns:
+ * %0 on success and a negative error value on failure.
+ */
+int cp_init(struct channel_program *cp, struct device *mdev, union orb *orb)
+{
+ u64 iova = orb->cmd.cpa;
+ struct ccwchain *chain;
+ int len, ret;
+
+ /*
+ * XXX:
+ * Only support prefetch enable mode now.
+ * Only support 64bit addressing idal.
+ * Only support 4k IDAW.
+ */
+ if (!orb->cmd.pfch || !orb->cmd.c64 || orb->cmd.i2k)
+ return -EOPNOTSUPP;
+
+ INIT_LIST_HEAD(&cp->ccwchain_list);
+ memcpy(&cp->orb, orb, sizeof(*orb));
+ cp->mdev = mdev;
+
+ /* Get chain length. */
+ len = ccwchain_calc_length(iova, cp);
+ if (len < 0)
+ return len;
+
+ /* Alloc mem for the head chain. */
+ chain = ccwchain_alloc(cp, len);
+ if (!chain)
+ return -ENOMEM;
+ chain->ch_iova = iova;
+
+ /* Copy the head chain from guest. */
+ ret = copy_ccw_from_iova(cp, chain->ch_ccw, iova, len);
+ if (ret) {
+ ccwchain_free(chain);
+ return ret;
+ }
+
+ /* Now loop for its TICs. */
+ ret = ccwchain_loop_tic(chain, cp);
+ if (ret)
+ cp_unpin_free(cp);
+
+ return ret;
+}
+
+
+/**
+ * cp_free() - free resources for channel program.
+ * @cp: channel_program on which to perform the operation
+ *
+ * This unpins the memory pages and frees the memory space occupied by
+ * @cp, which must have been returned by a previous call to cp_init().
+ * Otherwise, undefined behavior occurs.
+ */
+void cp_free(struct channel_program *cp)
+{
+ cp_unpin_free(cp);
+}
+
+/**
+ * cp_prefetch() - translate a guest physical address channel program to
+ * a real-device runnable channel program.
+ * @cp: channel_program on which to perform the operation
+ *
+ * This function translates the guest-physical-address channel program
+ * and stores the result to ccwchain list. @cp must have been
+ * initialized by a previous call with cp_init(). Otherwise, undefined
+ * behavior occurs.
+ *
+ * The S/390 CCW Translation APIS (prefixed by 'cp_') are introduced
+ * as helpers to do ccw chain translation inside the kernel. Basically
+ * they accept a channel program issued by a virtual machine, and
+ * translate the channel program to a real-device runnable channel
+ * program.
+ *
+ * These APIs will copy the ccws into kernel-space buffers, and update
+ * the guest phsical addresses with their corresponding host physical
+ * addresses. Then channel I/O device drivers could issue the
+ * translated channel program to real devices to perform an I/O
+ * operation.
+ *
+ * These interfaces are designed to support translation only for
+ * channel programs, which are generated and formatted by a
+ * guest. Thus this will make it possible for things like VFIO to
+ * leverage the interfaces to passthrough a channel I/O mediated
+ * device in QEMU.
+ *
+ * We support direct ccw chaining by translating them to idal ccws.
+ *
+ * Returns:
+ * %0 on success and a negative error value on failure.
+ */
+int cp_prefetch(struct channel_program *cp)
+{
+ struct ccwchain *chain;
+ int len, idx, ret;
+
+ list_for_each_entry(chain, &cp->ccwchain_list, next) {
+ len = chain->ch_len;
+ for (idx = 0; idx < len; idx++) {
+ ret = ccwchain_fetch_one(chain, idx, cp);
+ if (ret)
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+/**
+ * cp_get_orb() - get the orb of the channel program
+ * @cp: channel_program on which to perform the operation
+ * @intparm: new intparm for the returned orb
+ * @lpm: candidate value of the logical-path mask for the returned orb
+ *
+ * This function returns the address of the updated orb of the channel
+ * program. Channel I/O device drivers could use this orb to issue a
+ * ssch.
+ */
+union orb *cp_get_orb(struct channel_program *cp, u32 intparm, u8 lpm)
+{
+ union orb *orb;
+ struct ccwchain *chain;
+ struct ccw1 *cpa;
+
+ orb = &cp->orb;
+
+ orb->cmd.intparm = intparm;
+ orb->cmd.fmt = 1;
+ orb->cmd.key = PAGE_DEFAULT_KEY >> 4;
+
+ if (orb->cmd.lpm == 0)
+ orb->cmd.lpm = lpm;
+
+ chain = list_first_entry(&cp->ccwchain_list, struct ccwchain, next);
+ cpa = chain->ch_ccw;
+ orb->cmd.cpa = (__u32) __pa(cpa);
+
+ return orb;
+}
+
+/**
+ * cp_update_scsw() - update scsw for a channel program.
+ * @cp: channel_program on which to perform the operation
+ * @scsw: I/O results of the channel program and also the target to be
+ * updated
+ *
+ * @scsw contains the I/O results of the channel program that pointed
+ * to by @cp. However what @scsw->cpa stores is a host physical
+ * address, which is meaningless for the guest, which is waiting for
+ * the I/O results.
+ *
+ * This function updates @scsw->cpa to its coressponding guest physical
+ * address.
+ */
+void cp_update_scsw(struct channel_program *cp, union scsw *scsw)
+{
+ struct ccwchain *chain;
+ u32 cpa = scsw->cmd.cpa;
+ u32 ccw_head, ccw_tail;
+
+ /*
+ * LATER:
+ * For now, only update the cmd.cpa part. We may need to deal with
+ * other portions of the schib as well, even if we don't return them
+ * in the ioctl directly. Path status changes etc.
+ */
+ list_for_each_entry(chain, &cp->ccwchain_list, next) {
+ ccw_head = (u32)(u64)chain->ch_ccw;
+ ccw_tail = (u32)(u64)(chain->ch_ccw + chain->ch_len - 1);
+
+ if ((ccw_head <= cpa) && (cpa <= ccw_tail)) {
+ /*
+ * (cpa - ccw_head) is the offset value of the host
+ * physical ccw to its chain head.
+ * Adding this value to the guest physical ccw chain
+ * head gets us the guest cpa.
+ */
+ cpa = chain->ch_iova + (cpa - ccw_head);
+ break;
+ }
+ }
+
+ scsw->cmd.cpa = cpa;
+}
+
+/**
+ * cp_iova_pinned() - check if an iova is pinned for a ccw chain.
+ * @cmd: ccwchain command on which to perform the operation
+ * @iova: the iova to check
+ *
+ * If the @iova is currently pinned for the ccw chain, return true;
+ * else return false.
+ */
+bool cp_iova_pinned(struct channel_program *cp, u64 iova)
+{
+ struct ccwchain *chain;
+ int i;
+
+ list_for_each_entry(chain, &cp->ccwchain_list, next) {
+ for (i = 0; i < chain->ch_len; i++)
+ if (pfn_array_table_iova_pinned(chain->ch_pat + i,
+ iova))
+ return true;
+ }
+
+ return false;
+}
diff --git a/drivers/s390/cio/vfio_ccw_cp.h b/drivers/s390/cio/vfio_ccw_cp.h
new file mode 100644
index 0000000..7a1996b
--- /dev/null
+++ b/drivers/s390/cio/vfio_ccw_cp.h
@@ -0,0 +1,42 @@
+/*
+ * channel program interfaces
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author(s): Dong Jia Shi <bjsdjshi@linux.vnet.ibm.com>
+ * Xiao Feng Ren <renxiaof@linux.vnet.ibm.com>
+ */
+
+#ifndef _VFIO_CCW_CP_H_
+#define _VFIO_CCW_CP_H_
+
+#include <asm/cio.h>
+#include <asm/scsw.h>
+
+#include "orb.h"
+
+/**
+ * struct channel_program - manage information for channel program
+ * @ccwchain_list: list head of ccwchains
+ * @orb: orb for the currently processed ssch request
+ * @mdev: the mediated device to perform page pinning/unpinning
+ *
+ * @ccwchain_list is the head of a ccwchain list, that contents the
+ * translated result of the guest channel program that pointed out by
+ * the iova parameter when calling cp_init.
+ */
+struct channel_program {
+ struct list_head ccwchain_list;
+ union orb orb;
+ struct device *mdev;
+};
+
+extern int cp_init(struct channel_program *cp, struct device *mdev,
+ union orb *orb);
+extern void cp_free(struct channel_program *cp);
+extern int cp_prefetch(struct channel_program *cp);
+extern union orb *cp_get_orb(struct channel_program *cp, u32 intparm, u8 lpm);
+extern void cp_update_scsw(struct channel_program *cp, union scsw *scsw);
+extern bool cp_iova_pinned(struct channel_program *cp, u64 iova);
+
+#endif
diff --git a/drivers/s390/cio/vfio_ccw_drv.c b/drivers/s390/cio/vfio_ccw_drv.c
new file mode 100644
index 0000000..e90dd43
--- /dev/null
+++ b/drivers/s390/cio/vfio_ccw_drv.c
@@ -0,0 +1,308 @@
+/*
+ * VFIO based Physical Subchannel device driver
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author(s): Dong Jia Shi <bjsdjshi@linux.vnet.ibm.com>
+ * Xiao Feng Ren <renxiaof@linux.vnet.ibm.com>
+ */
+
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/device.h>
+#include <linux/slab.h>
+#include <linux/uuid.h>
+#include <linux/mdev.h>
+
+#include <asm/isc.h>
+
+#include "ioasm.h"
+#include "css.h"
+#include "vfio_ccw_private.h"
+
+struct workqueue_struct *vfio_ccw_work_q;
+
+/*
+ * Helpers
+ */
+int vfio_ccw_sch_quiesce(struct subchannel *sch)
+{
+ struct vfio_ccw_private *private = dev_get_drvdata(&sch->dev);
+ DECLARE_COMPLETION_ONSTACK(completion);
+ int iretry, ret = 0;
+
+ spin_lock_irq(sch->lock);
+ if (!sch->schib.pmcw.ena)
+ goto out_unlock;
+ ret = cio_disable_subchannel(sch);
+ if (ret != -EBUSY)
+ goto out_unlock;
+
+ do {
+ iretry = 255;
+
+ ret = cio_cancel_halt_clear(sch, &iretry);
+ while (ret == -EBUSY) {
+ /*
+ * Flush all I/O and wait for
+ * cancel/halt/clear completion.
+ */
+ private->completion = &completion;
+ spin_unlock_irq(sch->lock);
+
+ wait_for_completion_timeout(&completion, 3*HZ);
+
+ spin_lock_irq(sch->lock);
+ private->completion = NULL;
+ flush_workqueue(vfio_ccw_work_q);
+ ret = cio_cancel_halt_clear(sch, &iretry);
+ };
+
+ ret = cio_disable_subchannel(sch);
+ } while (ret == -EBUSY);
+out_unlock:
+ private->state = VFIO_CCW_STATE_NOT_OPER;
+ spin_unlock_irq(sch->lock);
+ return ret;
+}
+
+static void vfio_ccw_sch_io_todo(struct work_struct *work)
+{
+ struct vfio_ccw_private *private;
+ struct subchannel *sch;
+ struct irb *irb;
+
+ private = container_of(work, struct vfio_ccw_private, io_work);
+ irb = &private->irb;
+ sch = private->sch;
+
+ if (scsw_is_solicited(&irb->scsw)) {
+ cp_update_scsw(&private->cp, &irb->scsw);
+ cp_free(&private->cp);
+ }
+ memcpy(private->io_region.irb_area, irb, sizeof(*irb));
+
+ if (private->io_trigger)
+ eventfd_signal(private->io_trigger, 1);
+
+ if (private->mdev)
+ private->state = VFIO_CCW_STATE_IDLE;
+}
+
+/*
+ * Sysfs interfaces
+ */
+static ssize_t chpids_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct subchannel *sch = to_subchannel(dev);
+ struct chsc_ssd_info *ssd = &sch->ssd_info;
+ ssize_t ret = 0;
+ int chp;
+ int mask;
+
+ for (chp = 0; chp < 8; chp++) {
+ mask = 0x80 >> chp;
+ if (ssd->path_mask & mask)
+ ret += sprintf(buf + ret, "%02x ", ssd->chpid[chp].id);
+ else
+ ret += sprintf(buf + ret, "00 ");
+ }
+ ret += sprintf(buf+ret, "\n");
+ return ret;
+}
+
+static ssize_t pimpampom_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct subchannel *sch = to_subchannel(dev);
+ struct pmcw *pmcw = &sch->schib.pmcw;
+
+ return sprintf(buf, "%02x %02x %02x\n",
+ pmcw->pim, pmcw->pam, pmcw->pom);
+}
+
+static DEVICE_ATTR(chpids, 0444, chpids_show, NULL);
+static DEVICE_ATTR(pimpampom, 0444, pimpampom_show, NULL);
+
+static struct attribute *vfio_subchannel_attrs[] = {
+ &dev_attr_chpids.attr,
+ &dev_attr_pimpampom.attr,
+ NULL,
+};
+
+static struct attribute_group vfio_subchannel_attr_group = {
+ .attrs = vfio_subchannel_attrs,
+};
+
+/*
+ * Css driver callbacks
+ */
+static void vfio_ccw_sch_irq(struct subchannel *sch)
+{
+ struct vfio_ccw_private *private = dev_get_drvdata(&sch->dev);
+
+ inc_irq_stat(IRQIO_CIO);
+ vfio_ccw_fsm_event(private, VFIO_CCW_EVENT_INTERRUPT);
+}
+
+static int vfio_ccw_sch_probe(struct subchannel *sch)
+{
+ struct pmcw *pmcw = &sch->schib.pmcw;
+ struct vfio_ccw_private *private;
+ int ret;
+
+ if (pmcw->qf) {
+ dev_warn(&sch->dev, "vfio: ccw: does not support QDIO: %s\n",
+ dev_name(&sch->dev));
+ return -ENODEV;
+ }
+
+ private = kzalloc(sizeof(*private), GFP_KERNEL | GFP_DMA);
+ if (!private)
+ return -ENOMEM;
+ private->sch = sch;
+ dev_set_drvdata(&sch->dev, private);
+
+ spin_lock_irq(sch->lock);
+ private->state = VFIO_CCW_STATE_NOT_OPER;
+ sch->isc = VFIO_CCW_ISC;
+ ret = cio_enable_subchannel(sch, (u32)(unsigned long)sch);
+ spin_unlock_irq(sch->lock);
+ if (ret)
+ goto out_free;
+
+ ret = sysfs_create_group(&sch->dev.kobj, &vfio_subchannel_attr_group);
+ if (ret)
+ goto out_disable;
+
+ ret = vfio_ccw_mdev_reg(sch);
+ if (ret)
+ goto out_rm_group;
+
+ INIT_WORK(&private->io_work, vfio_ccw_sch_io_todo);
+ atomic_set(&private->avail, 1);
+ private->state = VFIO_CCW_STATE_STANDBY;
+
+ return 0;
+
+out_rm_group:
+ sysfs_remove_group(&sch->dev.kobj, &vfio_subchannel_attr_group);
+out_disable:
+ cio_disable_subchannel(sch);
+out_free:
+ dev_set_drvdata(&sch->dev, NULL);
+ kfree(private);
+ return ret;
+}
+
+static int vfio_ccw_sch_remove(struct subchannel *sch)
+{
+ struct vfio_ccw_private *private = dev_get_drvdata(&sch->dev);
+
+ vfio_ccw_sch_quiesce(sch);
+
+ vfio_ccw_mdev_unreg(sch);
+
+ sysfs_remove_group(&sch->dev.kobj, &vfio_subchannel_attr_group);
+
+ dev_set_drvdata(&sch->dev, NULL);
+
+ kfree(private);
+
+ return 0;
+}
+
+static void vfio_ccw_sch_shutdown(struct subchannel *sch)
+{
+ vfio_ccw_sch_quiesce(sch);
+}
+
+/**
+ * vfio_ccw_sch_event - process subchannel event
+ * @sch: subchannel
+ * @process: non-zero if function is called in process context
+ *
+ * An unspecified event occurred for this subchannel. Adjust data according
+ * to the current operational state of the subchannel. Return zero when the
+ * event has been handled sufficiently or -EAGAIN when this function should
+ * be called again in process context.
+ */
+static int vfio_ccw_sch_event(struct subchannel *sch, int process)
+{
+ struct vfio_ccw_private *private = dev_get_drvdata(&sch->dev);
+ unsigned long flags;
+
+ spin_lock_irqsave(sch->lock, flags);
+ if (!device_is_registered(&sch->dev))
+ goto out_unlock;
+
+ if (work_pending(&sch->todo_work))
+ goto out_unlock;
+
+ if (cio_update_schib(sch)) {
+ vfio_ccw_fsm_event(private, VFIO_CCW_EVENT_NOT_OPER);
+ goto out_unlock;
+ }
+
+ private = dev_get_drvdata(&sch->dev);
+ if (private->state == VFIO_CCW_STATE_NOT_OPER) {
+ private->state = private->mdev ? VFIO_CCW_STATE_IDLE :
+ VFIO_CCW_STATE_STANDBY;
+ }
+
+out_unlock:
+ spin_unlock_irqrestore(sch->lock, flags);
+
+ return 0;
+}
+
+static struct css_device_id vfio_ccw_sch_ids[] = {
+ { .match_flags = 0x1, .type = SUBCHANNEL_TYPE_IO, },
+ { /* end of list */ },
+};
+MODULE_DEVICE_TABLE(css, vfio_ccw_sch_ids);
+
+static struct css_driver vfio_ccw_sch_driver = {
+ .drv = {
+ .name = "vfio_ccw",
+ .owner = THIS_MODULE,
+ },
+ .subchannel_type = vfio_ccw_sch_ids,
+ .irq = vfio_ccw_sch_irq,
+ .probe = vfio_ccw_sch_probe,
+ .remove = vfio_ccw_sch_remove,
+ .shutdown = vfio_ccw_sch_shutdown,
+ .sch_event = vfio_ccw_sch_event,
+};
+
+static int __init vfio_ccw_sch_init(void)
+{
+ int ret;
+
+ vfio_ccw_work_q = create_singlethread_workqueue("vfio-ccw");
+ if (!vfio_ccw_work_q)
+ return -ENOMEM;
+
+ isc_register(VFIO_CCW_ISC);
+ ret = css_driver_register(&vfio_ccw_sch_driver);
+ if (ret) {
+ isc_unregister(VFIO_CCW_ISC);
+ destroy_workqueue(vfio_ccw_work_q);
+ }
+
+ return ret;
+}
+
+static void __exit vfio_ccw_sch_exit(void)
+{
+ css_driver_unregister(&vfio_ccw_sch_driver);
+ isc_unregister(VFIO_CCW_ISC);
+ destroy_workqueue(vfio_ccw_work_q);
+}
+module_init(vfio_ccw_sch_init);
+module_exit(vfio_ccw_sch_exit);
+
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/s390/cio/vfio_ccw_fsm.c b/drivers/s390/cio/vfio_ccw_fsm.c
new file mode 100644
index 0000000..80a0559c
--- /dev/null
+++ b/drivers/s390/cio/vfio_ccw_fsm.c
@@ -0,0 +1,203 @@
+/*
+ * Finite state machine for vfio-ccw device handling
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author(s): Dong Jia Shi <bjsdjshi@linux.vnet.ibm.com>
+ */
+
+#include <linux/vfio.h>
+#include <linux/mdev.h>
+
+#include "ioasm.h"
+#include "vfio_ccw_private.h"
+
+static int fsm_io_helper(struct vfio_ccw_private *private)
+{
+ struct subchannel *sch;
+ union orb *orb;
+ int ccode;
+ __u8 lpm;
+ unsigned long flags;
+
+ sch = private->sch;
+
+ spin_lock_irqsave(sch->lock, flags);
+ private->state = VFIO_CCW_STATE_BUSY;
+ spin_unlock_irqrestore(sch->lock, flags);
+
+ orb = cp_get_orb(&private->cp, (u32)(addr_t)sch, sch->lpm);
+
+ /* Issue "Start Subchannel" */
+ ccode = ssch(sch->schid, orb);
+
+ switch (ccode) {
+ case 0:
+ /*
+ * Initialize device status information
+ */
+ sch->schib.scsw.cmd.actl |= SCSW_ACTL_START_PEND;
+ return 0;
+ case 1: /* Status pending */
+ case 2: /* Busy */
+ return -EBUSY;
+ case 3: /* Device/path not operational */
+ {
+ lpm = orb->cmd.lpm;
+ if (lpm != 0)
+ sch->lpm &= ~lpm;
+ else
+ sch->lpm = 0;
+
+ if (cio_update_schib(sch))
+ return -ENODEV;
+
+ return sch->lpm ? -EACCES : -ENODEV;
+ }
+ default:
+ return ccode;
+ }
+}
+
+static void fsm_notoper(struct vfio_ccw_private *private,
+ enum vfio_ccw_event event)
+{
+ struct subchannel *sch = private->sch;
+
+ /*
+ * TODO:
+ * Probably we should send the machine check to the guest.
+ */
+ css_sched_sch_todo(sch, SCH_TODO_UNREG);
+ private->state = VFIO_CCW_STATE_NOT_OPER;
+}
+
+/*
+ * No operation action.
+ */
+static void fsm_nop(struct vfio_ccw_private *private,
+ enum vfio_ccw_event event)
+{
+}
+
+static void fsm_io_error(struct vfio_ccw_private *private,
+ enum vfio_ccw_event event)
+{
+ pr_err("vfio-ccw: FSM: I/O request from state:%d\n", private->state);
+ private->io_region.ret_code = -EIO;
+}
+
+static void fsm_io_busy(struct vfio_ccw_private *private,
+ enum vfio_ccw_event event)
+{
+ private->io_region.ret_code = -EBUSY;
+}
+
+static void fsm_disabled_irq(struct vfio_ccw_private *private,
+ enum vfio_ccw_event event)
+{
+ struct subchannel *sch = private->sch;
+
+ /*
+ * An interrupt in a disabled state means a previous disable was not
+ * successful - should not happen, but we try to disable again.
+ */
+ cio_disable_subchannel(sch);
+}
+
+/*
+ * Deal with the ccw command request from the userspace.
+ */
+static void fsm_io_request(struct vfio_ccw_private *private,
+ enum vfio_ccw_event event)
+{
+ union orb *orb;
+ union scsw *scsw = &private->scsw;
+ struct ccw_io_region *io_region = &private->io_region;
+ struct mdev_device *mdev = private->mdev;
+
+ private->state = VFIO_CCW_STATE_BOXED;
+
+ memcpy(scsw, io_region->scsw_area, sizeof(*scsw));
+
+ if (scsw->cmd.fctl & SCSW_FCTL_START_FUNC) {
+ orb = (union orb *)io_region->orb_area;
+
+ io_region->ret_code = cp_init(&private->cp, mdev_dev(mdev),
+ orb);
+ if (io_region->ret_code)
+ goto err_out;
+
+ io_region->ret_code = cp_prefetch(&private->cp);
+ if (io_region->ret_code) {
+ cp_free(&private->cp);
+ goto err_out;
+ }
+
+ /* Start channel program and wait for I/O interrupt. */
+ io_region->ret_code = fsm_io_helper(private);
+ if (io_region->ret_code) {
+ cp_free(&private->cp);
+ goto err_out;
+ }
+ return;
+ } else if (scsw->cmd.fctl & SCSW_FCTL_HALT_FUNC) {
+ /* XXX: Handle halt. */
+ io_region->ret_code = -EOPNOTSUPP;
+ goto err_out;
+ } else if (scsw->cmd.fctl & SCSW_FCTL_CLEAR_FUNC) {
+ /* XXX: Handle clear. */
+ io_region->ret_code = -EOPNOTSUPP;
+ goto err_out;
+ }
+
+err_out:
+ private->state = VFIO_CCW_STATE_IDLE;
+}
+
+/*
+ * Got an interrupt for a normal io (state busy).
+ */
+static void fsm_irq(struct vfio_ccw_private *private,
+ enum vfio_ccw_event event)
+{
+ struct irb *irb = this_cpu_ptr(&cio_irb);
+
+ memcpy(&private->irb, irb, sizeof(*irb));
+
+ queue_work(vfio_ccw_work_q, &private->io_work);
+
+ if (private->completion)
+ complete(private->completion);
+}
+
+/*
+ * Device statemachine
+ */
+fsm_func_t *vfio_ccw_jumptable[NR_VFIO_CCW_STATES][NR_VFIO_CCW_EVENTS] = {
+ [VFIO_CCW_STATE_NOT_OPER] = {
+ [VFIO_CCW_EVENT_NOT_OPER] = fsm_nop,
+ [VFIO_CCW_EVENT_IO_REQ] = fsm_io_error,
+ [VFIO_CCW_EVENT_INTERRUPT] = fsm_disabled_irq,
+ },
+ [VFIO_CCW_STATE_STANDBY] = {
+ [VFIO_CCW_EVENT_NOT_OPER] = fsm_notoper,
+ [VFIO_CCW_EVENT_IO_REQ] = fsm_io_error,
+ [VFIO_CCW_EVENT_INTERRUPT] = fsm_irq,
+ },
+ [VFIO_CCW_STATE_IDLE] = {
+ [VFIO_CCW_EVENT_NOT_OPER] = fsm_notoper,
+ [VFIO_CCW_EVENT_IO_REQ] = fsm_io_request,
+ [VFIO_CCW_EVENT_INTERRUPT] = fsm_irq,
+ },
+ [VFIO_CCW_STATE_BOXED] = {
+ [VFIO_CCW_EVENT_NOT_OPER] = fsm_notoper,
+ [VFIO_CCW_EVENT_IO_REQ] = fsm_io_busy,
+ [VFIO_CCW_EVENT_INTERRUPT] = fsm_irq,
+ },
+ [VFIO_CCW_STATE_BUSY] = {
+ [VFIO_CCW_EVENT_NOT_OPER] = fsm_notoper,
+ [VFIO_CCW_EVENT_IO_REQ] = fsm_io_busy,
+ [VFIO_CCW_EVENT_INTERRUPT] = fsm_irq,
+ },
+};
diff --git a/drivers/s390/cio/vfio_ccw_ops.c b/drivers/s390/cio/vfio_ccw_ops.c
new file mode 100644
index 0000000..e72abbc
--- /dev/null
+++ b/drivers/s390/cio/vfio_ccw_ops.c
@@ -0,0 +1,425 @@
+/*
+ * Physical device callbacks for vfio_ccw
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author(s): Dong Jia Shi <bjsdjshi@linux.vnet.ibm.com>
+ * Xiao Feng Ren <renxiaof@linux.vnet.ibm.com>
+ */
+
+#include <linux/vfio.h>
+#include <linux/mdev.h>
+
+#include "vfio_ccw_private.h"
+
+static int vfio_ccw_mdev_reset(struct mdev_device *mdev)
+{
+ struct vfio_ccw_private *private;
+ struct subchannel *sch;
+ int ret;
+
+ private = dev_get_drvdata(mdev_parent_dev(mdev));
+ sch = private->sch;
+ /*
+ * TODO:
+ * In the cureent stage, some things like "no I/O running" and "no
+ * interrupt pending" are clear, but we are not sure what other state
+ * we need to care about.
+ * There are still a lot more instructions need to be handled. We
+ * should come back here later.
+ */
+ ret = vfio_ccw_sch_quiesce(sch);
+ if (ret)
+ return ret;
+
+ ret = cio_enable_subchannel(sch, (u32)(unsigned long)sch);
+ if (!ret)
+ private->state = VFIO_CCW_STATE_IDLE;
+
+ return ret;
+}
+
+static int vfio_ccw_mdev_notifier(struct notifier_block *nb,
+ unsigned long action,
+ void *data)
+{
+ struct vfio_ccw_private *private =
+ container_of(nb, struct vfio_ccw_private, nb);
+
+ /*
+ * Vendor drivers MUST unpin pages in response to an
+ * invalidation.
+ */
+ if (action == VFIO_IOMMU_NOTIFY_DMA_UNMAP) {
+ struct vfio_iommu_type1_dma_unmap *unmap = data;
+
+ if (!cp_iova_pinned(&private->cp, unmap->iova))
+ return NOTIFY_OK;
+
+ if (vfio_ccw_mdev_reset(private->mdev))
+ return NOTIFY_BAD;
+
+ cp_free(&private->cp);
+ return NOTIFY_OK;
+ }
+
+ return NOTIFY_DONE;
+}
+
+static ssize_t name_show(struct kobject *kobj, struct device *dev, char *buf)
+{
+ return sprintf(buf, "I/O subchannel (Non-QDIO)\n");
+}
+MDEV_TYPE_ATTR_RO(name);
+
+static ssize_t device_api_show(struct kobject *kobj, struct device *dev,
+ char *buf)
+{
+ return sprintf(buf, "%s\n", VFIO_DEVICE_API_CCW_STRING);
+}
+MDEV_TYPE_ATTR_RO(device_api);
+
+static ssize_t available_instances_show(struct kobject *kobj,
+ struct device *dev, char *buf)
+{
+ struct vfio_ccw_private *private = dev_get_drvdata(dev);
+
+ return sprintf(buf, "%d\n", atomic_read(&private->avail));
+}
+MDEV_TYPE_ATTR_RO(available_instances);
+
+static struct attribute *mdev_types_attrs[] = {
+ &mdev_type_attr_name.attr,
+ &mdev_type_attr_device_api.attr,
+ &mdev_type_attr_available_instances.attr,
+ NULL,
+};
+
+static struct attribute_group mdev_type_group = {
+ .name = "io",
+ .attrs = mdev_types_attrs,
+};
+
+struct attribute_group *mdev_type_groups[] = {
+ &mdev_type_group,
+ NULL,
+};
+
+static int vfio_ccw_mdev_create(struct kobject *kobj, struct mdev_device *mdev)
+{
+ struct vfio_ccw_private *private =
+ dev_get_drvdata(mdev_parent_dev(mdev));
+
+ if (private->state == VFIO_CCW_STATE_NOT_OPER)
+ return -ENODEV;
+
+ if (atomic_dec_if_positive(&private->avail) < 0)
+ return -EPERM;
+
+ private->mdev = mdev;
+ private->state = VFIO_CCW_STATE_IDLE;
+
+ return 0;
+}
+
+static int vfio_ccw_mdev_remove(struct mdev_device *mdev)
+{
+ struct vfio_ccw_private *private =
+ dev_get_drvdata(mdev_parent_dev(mdev));
+
+ if ((private->state != VFIO_CCW_STATE_NOT_OPER) &&
+ (private->state != VFIO_CCW_STATE_STANDBY)) {
+ if (!vfio_ccw_mdev_reset(mdev))
+ private->state = VFIO_CCW_STATE_STANDBY;
+ /* The state will be NOT_OPER on error. */
+ }
+
+ private->mdev = NULL;
+ atomic_inc(&private->avail);
+
+ return 0;
+}
+
+static int vfio_ccw_mdev_open(struct mdev_device *mdev)
+{
+ struct vfio_ccw_private *private =
+ dev_get_drvdata(mdev_parent_dev(mdev));
+ unsigned long events = VFIO_IOMMU_NOTIFY_DMA_UNMAP;
+
+ private->nb.notifier_call = vfio_ccw_mdev_notifier;
+
+ return vfio_register_notifier(mdev_dev(mdev), VFIO_IOMMU_NOTIFY,
+ &events, &private->nb);
+}
+
+void vfio_ccw_mdev_release(struct mdev_device *mdev)
+{
+ struct vfio_ccw_private *private =
+ dev_get_drvdata(mdev_parent_dev(mdev));
+
+ vfio_unregister_notifier(mdev_dev(mdev), VFIO_IOMMU_NOTIFY,
+ &private->nb);
+}
+
+static ssize_t vfio_ccw_mdev_read(struct mdev_device *mdev,
+ char __user *buf,
+ size_t count,
+ loff_t *ppos)
+{
+ struct vfio_ccw_private *private;
+ struct ccw_io_region *region;
+
+ if (*ppos + count > sizeof(*region))
+ return -EINVAL;
+
+ private = dev_get_drvdata(mdev_parent_dev(mdev));
+ region = &private->io_region;
+ if (copy_to_user(buf, (void *)region + *ppos, count))
+ return -EFAULT;
+
+ return count;
+}
+
+static ssize_t vfio_ccw_mdev_write(struct mdev_device *mdev,
+ const char __user *buf,
+ size_t count,
+ loff_t *ppos)
+{
+ struct vfio_ccw_private *private;
+ struct ccw_io_region *region;
+
+ if (*ppos + count > sizeof(*region))
+ return -EINVAL;
+
+ private = dev_get_drvdata(mdev_parent_dev(mdev));
+ if (private->state != VFIO_CCW_STATE_IDLE)
+ return -EACCES;
+
+ region = &private->io_region;
+ if (copy_from_user((void *)region + *ppos, buf, count))
+ return -EFAULT;
+
+ vfio_ccw_fsm_event(private, VFIO_CCW_EVENT_IO_REQ);
+ if (region->ret_code != 0) {
+ private->state = VFIO_CCW_STATE_IDLE;
+ return region->ret_code;
+ }
+
+ return count;
+}
+
+static int vfio_ccw_mdev_get_device_info(struct vfio_device_info *info)
+{
+ info->flags = VFIO_DEVICE_FLAGS_CCW | VFIO_DEVICE_FLAGS_RESET;
+ info->num_regions = VFIO_CCW_NUM_REGIONS;
+ info->num_irqs = VFIO_CCW_NUM_IRQS;
+
+ return 0;
+}
+
+static int vfio_ccw_mdev_get_region_info(struct vfio_region_info *info,
+ u16 *cap_type_id,
+ void **cap_type)
+{
+ switch (info->index) {
+ case VFIO_CCW_CONFIG_REGION_INDEX:
+ info->offset = 0;
+ info->size = sizeof(struct ccw_io_region);
+ info->flags = VFIO_REGION_INFO_FLAG_READ
+ | VFIO_REGION_INFO_FLAG_WRITE;
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
+int vfio_ccw_mdev_get_irq_info(struct vfio_irq_info *info)
+{
+ if (info->index != VFIO_CCW_IO_IRQ_INDEX)
+ return -EINVAL;
+
+ info->count = 1;
+ info->flags = VFIO_IRQ_INFO_EVENTFD;
+
+ return 0;
+}
+
+static int vfio_ccw_mdev_set_irqs(struct mdev_device *mdev,
+ uint32_t flags,
+ void __user *data)
+{
+ struct vfio_ccw_private *private;
+ struct eventfd_ctx **ctx;
+
+ if (!(flags & VFIO_IRQ_SET_ACTION_TRIGGER))
+ return -EINVAL;
+
+ private = dev_get_drvdata(mdev_parent_dev(mdev));
+ ctx = &private->io_trigger;
+
+ switch (flags & VFIO_IRQ_SET_DATA_TYPE_MASK) {
+ case VFIO_IRQ_SET_DATA_NONE:
+ {
+ if (*ctx)
+ eventfd_signal(*ctx, 1);
+ return 0;
+ }
+ case VFIO_IRQ_SET_DATA_BOOL:
+ {
+ uint8_t trigger;
+
+ if (get_user(trigger, (uint8_t __user *)data))
+ return -EFAULT;
+
+ if (trigger && *ctx)
+ eventfd_signal(*ctx, 1);
+ return 0;
+ }
+ case VFIO_IRQ_SET_DATA_EVENTFD:
+ {
+ int32_t fd;
+
+ if (get_user(fd, (int32_t __user *)data))
+ return -EFAULT;
+
+ if (fd == -1) {
+ if (*ctx)
+ eventfd_ctx_put(*ctx);
+ *ctx = NULL;
+ } else if (fd >= 0) {
+ struct eventfd_ctx *efdctx;
+
+ efdctx = eventfd_ctx_fdget(fd);
+ if (IS_ERR(efdctx))
+ return PTR_ERR(efdctx);
+
+ if (*ctx)
+ eventfd_ctx_put(*ctx);
+
+ *ctx = efdctx;
+ } else
+ return -EINVAL;
+
+ return 0;
+ }
+ default:
+ return -EINVAL;
+ }
+}
+
+static ssize_t vfio_ccw_mdev_ioctl(struct mdev_device *mdev,
+ unsigned int cmd,
+ unsigned long arg)
+{
+ int ret = 0;
+ unsigned long minsz;
+
+ switch (cmd) {
+ case VFIO_DEVICE_GET_INFO:
+ {
+ struct vfio_device_info info;
+
+ minsz = offsetofend(struct vfio_device_info, num_irqs);
+
+ if (copy_from_user(&info, (void __user *)arg, minsz))
+ return -EFAULT;
+
+ if (info.argsz < minsz)
+ return -EINVAL;
+
+ ret = vfio_ccw_mdev_get_device_info(&info);
+ if (ret)
+ return ret;
+
+ return copy_to_user((void __user *)arg, &info, minsz);
+ }
+ case VFIO_DEVICE_GET_REGION_INFO:
+ {
+ struct vfio_region_info info;
+ u16 cap_type_id = 0;
+ void *cap_type = NULL;
+
+ minsz = offsetofend(struct vfio_region_info, offset);
+
+ if (copy_from_user(&info, (void __user *)arg, minsz))
+ return -EFAULT;
+
+ if (info.argsz < minsz)
+ return -EINVAL;
+
+ ret = vfio_ccw_mdev_get_region_info(&info, &cap_type_id,
+ &cap_type);
+ if (ret)
+ return ret;
+
+ return copy_to_user((void __user *)arg, &info, minsz);
+ }
+ case VFIO_DEVICE_GET_IRQ_INFO:
+ {
+ struct vfio_irq_info info;
+
+ minsz = offsetofend(struct vfio_irq_info, count);
+
+ if (copy_from_user(&info, (void __user *)arg, minsz))
+ return -EFAULT;
+
+ if (info.argsz < minsz || info.index >= VFIO_CCW_NUM_IRQS)
+ return -EINVAL;
+
+ ret = vfio_ccw_mdev_get_irq_info(&info);
+ if (ret)
+ return ret;
+
+ if (info.count == -1)
+ return -EINVAL;
+
+ return copy_to_user((void __user *)arg, &info, minsz);
+ }
+ case VFIO_DEVICE_SET_IRQS:
+ {
+ struct vfio_irq_set hdr;
+ size_t data_size;
+ void __user *data;
+
+ minsz = offsetofend(struct vfio_irq_set, count);
+
+ if (copy_from_user(&hdr, (void __user *)arg, minsz))
+ return -EFAULT;
+
+ ret = vfio_set_irqs_validate_and_prepare(&hdr, 1,
+ VFIO_CCW_NUM_IRQS,
+ &data_size);
+ if (ret)
+ return ret;
+
+ data = (void __user *)(arg + minsz);
+ return vfio_ccw_mdev_set_irqs(mdev, hdr.flags, data);
+ }
+ case VFIO_DEVICE_RESET:
+ return vfio_ccw_mdev_reset(mdev);
+ default:
+ return -ENOTTY;
+ }
+}
+
+static const struct mdev_parent_ops vfio_ccw_mdev_ops = {
+ .owner = THIS_MODULE,
+ .supported_type_groups = mdev_type_groups,
+ .create = vfio_ccw_mdev_create,
+ .remove = vfio_ccw_mdev_remove,
+ .open = vfio_ccw_mdev_open,
+ .release = vfio_ccw_mdev_release,
+ .read = vfio_ccw_mdev_read,
+ .write = vfio_ccw_mdev_write,
+ .ioctl = vfio_ccw_mdev_ioctl,
+};
+
+int vfio_ccw_mdev_reg(struct subchannel *sch)
+{
+ return mdev_register_device(&sch->dev, &vfio_ccw_mdev_ops);
+}
+
+void vfio_ccw_mdev_unreg(struct subchannel *sch)
+{
+ mdev_unregister_device(&sch->dev);
+}
diff --git a/drivers/s390/cio/vfio_ccw_private.h b/drivers/s390/cio/vfio_ccw_private.h
new file mode 100644
index 0000000..fc0f01c
--- /dev/null
+++ b/drivers/s390/cio/vfio_ccw_private.h
@@ -0,0 +1,96 @@
+/*
+ * Private stuff for vfio_ccw driver
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author(s): Dong Jia Shi <bjsdjshi@linux.vnet.ibm.com>
+ * Xiao Feng Ren <renxiaof@linux.vnet.ibm.com>
+ */
+
+#ifndef _VFIO_CCW_PRIVATE_H_
+#define _VFIO_CCW_PRIVATE_H_
+
+#include <linux/completion.h>
+#include <linux/eventfd.h>
+#include <linux/workqueue.h>
+#include <linux/vfio_ccw.h>
+
+#include "css.h"
+#include "vfio_ccw_cp.h"
+
+/**
+ * struct vfio_ccw_private
+ * @sch: pointer to the subchannel
+ * @state: internal state of the device
+ * @completion: synchronization helper of the I/O completion
+ * @avail: available for creating a mediated device
+ * @mdev: pointer to the mediated device
+ * @nb: notifier for vfio events
+ * @io_region: MMIO region to input/output I/O arguments/results
+ * @cp: channel program for the current I/O operation
+ * @irb: irb info received from interrupt
+ * @scsw: scsw info
+ * @io_trigger: eventfd ctx for signaling userspace I/O results
+ * @io_work: work for deferral process of I/O handling
+ */
+struct vfio_ccw_private {
+ struct subchannel *sch;
+ int state;
+ struct completion *completion;
+ atomic_t avail;
+ struct mdev_device *mdev;
+ struct notifier_block nb;
+ struct ccw_io_region io_region;
+
+ struct channel_program cp;
+ struct irb irb;
+ union scsw scsw;
+
+ struct eventfd_ctx *io_trigger;
+ struct work_struct io_work;
+} __aligned(8);
+
+extern int vfio_ccw_mdev_reg(struct subchannel *sch);
+extern void vfio_ccw_mdev_unreg(struct subchannel *sch);
+
+extern int vfio_ccw_sch_quiesce(struct subchannel *sch);
+
+/*
+ * States of the device statemachine.
+ */
+enum vfio_ccw_state {
+ VFIO_CCW_STATE_NOT_OPER,
+ VFIO_CCW_STATE_STANDBY,
+ VFIO_CCW_STATE_IDLE,
+ VFIO_CCW_STATE_BOXED,
+ VFIO_CCW_STATE_BUSY,
+ /* last element! */
+ NR_VFIO_CCW_STATES
+};
+
+/*
+ * Asynchronous events of the device statemachine.
+ */
+enum vfio_ccw_event {
+ VFIO_CCW_EVENT_NOT_OPER,
+ VFIO_CCW_EVENT_IO_REQ,
+ VFIO_CCW_EVENT_INTERRUPT,
+ /* last element! */
+ NR_VFIO_CCW_EVENTS
+};
+
+/*
+ * Action called through jumptable.
+ */
+typedef void (fsm_func_t)(struct vfio_ccw_private *, enum vfio_ccw_event);
+extern fsm_func_t *vfio_ccw_jumptable[NR_VFIO_CCW_STATES][NR_VFIO_CCW_EVENTS];
+
+static inline void vfio_ccw_fsm_event(struct vfio_ccw_private *private,
+ int event)
+{
+ vfio_ccw_jumptable[private->state][event](private, event);
+}
+
+extern struct workqueue_struct *vfio_ccw_work_q;
+
+#endif
diff --git a/drivers/s390/crypto/pkey_api.c b/drivers/s390/crypto/pkey_api.c
index 058db72..ea86da8 100644
--- a/drivers/s390/crypto/pkey_api.c
+++ b/drivers/s390/crypto/pkey_api.c
@@ -80,7 +80,7 @@
* token. If keybitsize is given, the bitsize of the key is
* also checked. Returns 0 on success or errno value on failure.
*/
-static int check_secaeskeytoken(u8 *token, int keybitsize)
+static int check_secaeskeytoken(const u8 *token, int keybitsize)
{
struct secaeskeytoken *t = (struct secaeskeytoken *) token;
@@ -1004,6 +1004,53 @@
EXPORT_SYMBOL(pkey_skey2pkey);
/*
+ * Verify key and give back some info about the key.
+ */
+int pkey_verifykey(const struct pkey_seckey *seckey,
+ u16 *pcardnr, u16 *pdomain,
+ u16 *pkeysize, u32 *pattributes)
+{
+ struct secaeskeytoken *t = (struct secaeskeytoken *) seckey;
+ u16 cardnr, domain;
+ u64 mkvp[2];
+ int rc;
+
+ /* check the secure key for valid AES secure key */
+ rc = check_secaeskeytoken((u8 *) seckey, 0);
+ if (rc)
+ goto out;
+ if (pattributes)
+ *pattributes = PKEY_VERIFY_ATTR_AES;
+ if (pkeysize)
+ *pkeysize = t->bitsize;
+
+ /* try to find a card which can handle this key */
+ rc = pkey_findcard(seckey, &cardnr, &domain, 1);
+ if (rc)
+ goto out;
+
+ /* check mkvp for old mkvp match */
+ rc = mkvp_cache_fetch(cardnr, domain, mkvp);
+ if (rc)
+ goto out;
+ if (t->mkvp == mkvp[1]) {
+ DEBUG_DBG("pkey_verifykey secure key has old mkvp\n");
+ if (pattributes)
+ *pattributes |= PKEY_VERIFY_ATTR_OLD_MKVP;
+ }
+
+ if (pcardnr)
+ *pcardnr = cardnr;
+ if (pdomain)
+ *pdomain = domain;
+
+out:
+ DEBUG_DBG("pkey_verifykey rc=%d\n", rc);
+ return rc;
+}
+EXPORT_SYMBOL(pkey_verifykey);
+
+/*
* File io functions
*/
@@ -1104,6 +1151,21 @@
return -EFAULT;
break;
}
+ case PKEY_VERIFYKEY: {
+ struct pkey_verifykey __user *uvk = (void __user *) arg;
+ struct pkey_verifykey kvk;
+
+ if (copy_from_user(&kvk, uvk, sizeof(kvk)))
+ return -EFAULT;
+ rc = pkey_verifykey(&kvk.seckey, &kvk.cardnr, &kvk.domain,
+ &kvk.keysize, &kvk.attributes);
+ DEBUG_DBG("pkey_ioctl pkey_verifykey()=%d\n", rc);
+ if (rc)
+ break;
+ if (copy_to_user(uvk, &kvk, sizeof(kvk)))
+ return -EFAULT;
+ break;
+ }
default:
/* unknown/unsupported ioctl cmd */
return -ENOTTY;
diff --git a/drivers/s390/net/ctcm_fsms.c b/drivers/s390/net/ctcm_fsms.c
index fd5944b..730d961 100644
--- a/drivers/s390/net/ctcm_fsms.c
+++ b/drivers/s390/net/ctcm_fsms.c
@@ -1283,7 +1283,7 @@
p_header = (struct pdu *)
(skb_tail_pointer(ch->trans_skb) - skb->len);
p_header->pdu_flag = 0x00;
- if (skb->protocol == ntohs(ETH_P_SNAP))
+ if (be16_to_cpu(skb->protocol) == ETH_P_SNAP)
p_header->pdu_flag |= 0x60;
else
p_header->pdu_flag |= 0x20;
diff --git a/drivers/s390/net/ctcm_main.c b/drivers/s390/net/ctcm_main.c
index ac65f12..198842c 100644
--- a/drivers/s390/net/ctcm_main.c
+++ b/drivers/s390/net/ctcm_main.c
@@ -106,7 +106,7 @@
priv->stats.rx_frame_errors++;
return;
}
- pskb->protocol = ntohs(header->type);
+ pskb->protocol = cpu_to_be16(header->type);
if ((header->length <= LL_HEADER_LENGTH) ||
(len <= LL_HEADER_LENGTH)) {
if (!(ch->logflags & LOG_FLAG_ILLEGALSIZE)) {
@@ -125,7 +125,7 @@
header->length -= LL_HEADER_LENGTH;
len -= LL_HEADER_LENGTH;
if ((header->length > skb_tailroom(pskb)) ||
- (header->length > len)) {
+ (header->length > len)) {
if (!(ch->logflags & LOG_FLAG_OVERRUN)) {
CTCM_DBF_TEXT_(ERROR, CTC_DBF_ERROR,
"%s(%s): Packet size %d (overrun)"
@@ -485,7 +485,7 @@
} else {
atomic_inc(&skb->users);
header.length = l;
- header.type = skb->protocol;
+ header.type = be16_to_cpu(skb->protocol);
header.unused = 0;
memcpy(skb_push(skb, LL_HEADER_LENGTH), &header,
LL_HEADER_LENGTH);
@@ -503,7 +503,7 @@
atomic_inc(&skb->users);
ch->prof.txlen += skb->len;
header.length = skb->len + LL_HEADER_LENGTH;
- header.type = skb->protocol;
+ header.type = be16_to_cpu(skb->protocol);
header.unused = 0;
memcpy(skb_push(skb, LL_HEADER_LENGTH), &header, LL_HEADER_LENGTH);
block_len = skb->len + 2;
@@ -690,7 +690,7 @@
p_header->pdu_offset = skb->len;
p_header->pdu_proto = 0x01;
p_header->pdu_flag = 0x00;
- if (skb->protocol == ntohs(ETH_P_SNAP)) {
+ if (be16_to_cpu(skb->protocol) == ETH_P_SNAP) {
p_header->pdu_flag |= PDU_FIRST | PDU_CNTL;
} else {
p_header->pdu_flag |= PDU_FIRST;
@@ -745,7 +745,7 @@
p_header->pdu_proto = 0x01;
p_header->pdu_flag = 0x00;
p_header->pdu_seq = 0;
- if (skb->protocol == ntohs(ETH_P_SNAP)) {
+ if (be16_to_cpu(skb->protocol) == ETH_P_SNAP) {
p_header->pdu_flag |= PDU_FIRST | PDU_CNTL;
} else {
p_header->pdu_flag |= PDU_FIRST;
diff --git a/drivers/s390/net/netiucv.c b/drivers/s390/net/netiucv.c
index 3f85b97..dba94b4 100644
--- a/drivers/s390/net/netiucv.c
+++ b/drivers/s390/net/netiucv.c
@@ -635,7 +635,7 @@
skb_put(pskb, NETIUCV_HDRLEN);
pskb->dev = dev;
pskb->ip_summed = CHECKSUM_NONE;
- pskb->protocol = ntohs(ETH_P_IP);
+ pskb->protocol = cpu_to_be16(ETH_P_IP);
while (1) {
struct sk_buff *skb;
diff --git a/drivers/s390/net/qeth_core.h b/drivers/s390/net/qeth_core.h
index d9561e3..f6aa211 100644
--- a/drivers/s390/net/qeth_core.h
+++ b/drivers/s390/net/qeth_core.h
@@ -240,7 +240,6 @@
#define QETH_TX_TIMEOUT 100 * HZ
#define QETH_RCD_TIMEOUT 60 * HZ
#define QETH_RECLAIM_WORK_TIME HZ
-#define QETH_HEADER_SIZE 32
#define QETH_MAX_PORTNO 15
/*IPv6 address autoconfiguration stuff*/
@@ -447,7 +446,7 @@
atomic_t state;
int next_element_to_fill;
struct sk_buff_head skb_list;
- int is_header[16];
+ int is_header[QDIO_MAX_ELEMENTS_PER_BUFFER];
struct qaob *aob;
struct qeth_qdio_out_q *q;
@@ -503,22 +502,12 @@
int default_out_queue;
};
-enum qeth_send_errors {
- QETH_SEND_ERROR_NONE,
- QETH_SEND_ERROR_LINK_FAILURE,
- QETH_SEND_ERROR_RETRY,
- QETH_SEND_ERROR_KICK_IT,
-};
-
#define QETH_ETH_MAC_V4 0x0100 /* like v4 */
#define QETH_ETH_MAC_V6 0x3333 /* like v6 */
/* tr mc mac is longer, but that will be enough to detect mc frames */
#define QETH_TR_MAC_NC 0xc000 /* non-canonical */
#define QETH_TR_MAC_C 0x0300 /* canonical */
-#define DEFAULT_ADD_HHLEN 0
-#define MAX_ADD_HHLEN 1024
-
/**
* buffer stuff for read channel
*/
@@ -644,7 +633,6 @@
atomic_t refcnt;
};
-
struct qeth_card_blkt {
int time_total;
int inter_packet;
@@ -685,7 +673,6 @@
struct qeth_ipa_info ipa6;
struct qeth_sbp_info sbp; /* SETBRIDGEPORT options */
int fake_broadcast;
- int add_hhlen;
int layer2;
int performance_stats;
int rx_sg_cb;
@@ -717,17 +704,16 @@
void (*start_poll)(struct ccw_device *, int, unsigned long);
qdio_handler_t *input_handler;
qdio_handler_t *output_handler;
+ int (*process_rx_buffer)(struct qeth_card *card, int budget, int *done);
int (*recover)(void *ptr);
int (*setup) (struct ccwgroup_device *);
void (*remove) (struct ccwgroup_device *);
int (*set_online) (struct ccwgroup_device *);
int (*set_offline) (struct ccwgroup_device *);
- void (*shutdown)(struct ccwgroup_device *);
- int (*prepare) (struct ccwgroup_device *);
- void (*complete) (struct ccwgroup_device *);
int (*freeze)(struct ccwgroup_device *);
int (*thaw) (struct ccwgroup_device *);
int (*restore)(struct ccwgroup_device *);
+ int (*do_ioctl)(struct net_device *dev, struct ifreq *rq, int cmd);
int (*control_event_handler)(struct qeth_card *card,
struct qeth_ipa_cmd *cmd);
};
@@ -856,9 +842,9 @@
{
__be16 *p = &((struct ethhdr *)skb->data)->h_proto;
- if (*p == ETH_P_8021Q)
+ if (be16_to_cpu(*p) == ETH_P_8021Q)
p += 2;
- switch (*p) {
+ switch (be16_to_cpu(*p)) {
case ETH_P_IPV6:
return 6;
case ETH_P_IP:
@@ -920,14 +906,12 @@
struct qeth_cmd_buffer *qeth_get_ipacmd_buffer(struct qeth_card *,
enum qeth_ipa_cmds, enum qeth_prot_versions);
int qeth_query_setadapterparms(struct qeth_card *);
-int qeth_check_qdio_errors(struct qeth_card *, struct qdio_buffer *,
- unsigned int, const char *);
-void qeth_queue_input_buffer(struct qeth_card *, int);
struct sk_buff *qeth_core_get_next_skb(struct qeth_card *,
struct qeth_qdio_buffer *, struct qdio_buffer_element **, int *,
struct qeth_hdr **);
void qeth_schedule_recovery(struct qeth_card *);
void qeth_qdio_start_poll(struct ccw_device *, int, unsigned long);
+int qeth_poll(struct napi_struct *napi, int budget);
void qeth_qdio_input_handler(struct ccw_device *,
unsigned int, unsigned int, int,
int, unsigned long);
@@ -948,9 +932,6 @@
void qeth_release_buffer(struct qeth_channel *, struct qeth_cmd_buffer *);
void qeth_prepare_ipa_cmd(struct qeth_card *, struct qeth_cmd_buffer *, char);
struct qeth_cmd_buffer *qeth_wait_for_buffer(struct qeth_channel *);
-int qeth_mdio_read(struct net_device *, int, int);
-int qeth_snmp_command(struct qeth_card *, char __user *);
-int qeth_query_oat_command(struct qeth_card *, char __user *);
int qeth_query_switch_attributes(struct qeth_card *card,
struct qeth_switch_info *sw_info);
int qeth_send_control_data(struct qeth_card *, int, struct qeth_cmd_buffer *,
@@ -965,16 +946,18 @@
int extra_elems, int data_offset);
int qeth_get_elements_for_frags(struct sk_buff *);
int qeth_do_send_packet_fast(struct qeth_card *, struct qeth_qdio_out_q *,
- struct sk_buff *, struct qeth_hdr *, int, int, int);
+ struct sk_buff *, struct qeth_hdr *, int, int);
int qeth_do_send_packet(struct qeth_card *, struct qeth_qdio_out_q *,
struct sk_buff *, struct qeth_hdr *, int);
+int qeth_do_ioctl(struct net_device *dev, struct ifreq *rq, int cmd);
int qeth_core_get_sset_count(struct net_device *, int);
void qeth_core_get_ethtool_stats(struct net_device *,
struct ethtool_stats *, u64 *);
void qeth_core_get_strings(struct net_device *, u32, u8 *);
void qeth_core_get_drvinfo(struct net_device *, struct ethtool_drvinfo *);
void qeth_dbf_longtext(debug_info_t *id, int level, char *text, ...);
-int qeth_core_ethtool_get_settings(struct net_device *, struct ethtool_cmd *);
+int qeth_core_ethtool_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd);
int qeth_set_access_ctrl_online(struct qeth_card *card, int fallback);
int qeth_hdr_chk_and_bounce(struct sk_buff *, struct qeth_hdr **, int);
int qeth_configure_cq(struct qeth_card *, enum qeth_cq);
diff --git a/drivers/s390/net/qeth_core_main.c b/drivers/s390/net/qeth_core_main.c
index 9a5f99c..38114a8 100644
--- a/drivers/s390/net/qeth_core_main.c
+++ b/drivers/s390/net/qeth_core_main.c
@@ -55,7 +55,6 @@
static void qeth_send_control_data_cb(struct qeth_channel *,
struct qeth_cmd_buffer *);
-static int qeth_issue_next_read(struct qeth_card *);
static struct qeth_cmd_buffer *qeth_get_buffer(struct qeth_channel *);
static void qeth_setup_ccw(struct qeth_channel *, unsigned char *, __u32);
static void qeth_free_buffer_pool(struct qeth_card *);
@@ -1202,7 +1201,7 @@
while (skb) {
QETH_CARD_TEXT_(q->card, 5, "skbn%d", notification);
QETH_CARD_TEXT_(q->card, 5, "%lx", (long) skb);
- if (skb->protocol == ETH_P_AF_IUCV) {
+ if (be16_to_cpu(skb->protocol) == ETH_P_AF_IUCV) {
if (skb->sk) {
struct iucv_sock *iucv = iucv_sk(skb->sk);
iucv->sk_txnotify(skb, notification);
@@ -1233,7 +1232,8 @@
while (skb) {
QETH_CARD_TEXT(buf->q->card, 5, "skbr");
QETH_CARD_TEXT_(buf->q->card, 5, "%lx", (long) skb);
- if (notify_general_error && skb->protocol == ETH_P_AF_IUCV) {
+ if (notify_general_error &&
+ be16_to_cpu(skb->protocol) == ETH_P_AF_IUCV) {
if (skb->sk) {
iucv = iucv_sk(skb->sk);
iucv->sk_txnotify(skb, TX_NOTIFY_GENERALERROR);
@@ -1396,7 +1396,6 @@
card->options.route4.type = NO_ROUTER;
card->options.route6.type = NO_ROUTER;
card->options.fake_broadcast = 0;
- card->options.add_hhlen = DEFAULT_ADD_HHLEN;
card->options.performance_stats = 0;
card->options.rx_sg_cb = QETH_RX_SG_CB;
card->options.isolation = ISOLATION_MODE_NONE;
@@ -3217,8 +3216,10 @@
}
EXPORT_SYMBOL_GPL(qeth_hw_trap);
-int qeth_check_qdio_errors(struct qeth_card *card, struct qdio_buffer *buf,
- unsigned int qdio_error, const char *dbftext)
+static int qeth_check_qdio_errors(struct qeth_card *card,
+ struct qdio_buffer *buf,
+ unsigned int qdio_error,
+ const char *dbftext)
{
if (qdio_error) {
QETH_CARD_TEXT(card, 2, dbftext);
@@ -3235,18 +3236,8 @@
}
return 0;
}
-EXPORT_SYMBOL_GPL(qeth_check_qdio_errors);
-static void qeth_buffer_reclaim_work(struct work_struct *work)
-{
- struct qeth_card *card = container_of(work, struct qeth_card,
- buffer_reclaim_work.work);
-
- QETH_CARD_TEXT_(card, 2, "brw:%x", card->reclaim_index);
- qeth_queue_input_buffer(card, card->reclaim_index);
-}
-
-void qeth_queue_input_buffer(struct qeth_card *card, int index)
+static void qeth_queue_input_buffer(struct qeth_card *card, int index)
{
struct qeth_qdio_q *queue = card->qdio.in_q;
struct list_head *lh;
@@ -3320,9 +3311,17 @@
QDIO_MAX_BUFFERS_PER_Q;
}
}
-EXPORT_SYMBOL_GPL(qeth_queue_input_buffer);
-static int qeth_handle_send_error(struct qeth_card *card,
+static void qeth_buffer_reclaim_work(struct work_struct *work)
+{
+ struct qeth_card *card = container_of(work, struct qeth_card,
+ buffer_reclaim_work.work);
+
+ QETH_CARD_TEXT_(card, 2, "brw:%x", card->reclaim_index);
+ qeth_queue_input_buffer(card, card->reclaim_index);
+}
+
+static void qeth_handle_send_error(struct qeth_card *card,
struct qeth_qdio_out_buffer *buffer, unsigned int qdio_err)
{
int sbalf15 = buffer->buffer->element[15].sflags;
@@ -3338,15 +3337,14 @@
qeth_check_qdio_errors(card, buffer->buffer, qdio_err, "qouterr");
if (!qdio_err)
- return QETH_SEND_ERROR_NONE;
+ return;
if ((sbalf15 >= 15) && (sbalf15 <= 31))
- return QETH_SEND_ERROR_RETRY;
+ return;
QETH_CARD_TEXT(card, 1, "lnkfail");
QETH_CARD_TEXT_(card, 1, "%04x %02x",
(u16)qdio_err, (u8)sbalf15);
- return QETH_SEND_ERROR_LINK_FAILURE;
}
/*
@@ -3799,9 +3797,9 @@
return qeth_cut_iqd_prio(card, ~skb->priority >> 1 & 3);
case QETH_PRIO_Q_ING_VLAN:
tci = &((struct ethhdr *)skb->data)->h_proto;
- if (*tci == ETH_P_8021Q)
- return qeth_cut_iqd_prio(card, ~*(tci + 1) >>
- (VLAN_PRIO_SHIFT + 1) & 3);
+ if (be16_to_cpu(*tci) == ETH_P_8021Q)
+ return qeth_cut_iqd_prio(card,
+ ~be16_to_cpu(*(tci + 1)) >> (VLAN_PRIO_SHIFT + 1) & 3);
break;
default:
break;
@@ -4026,8 +4024,7 @@
int qeth_do_send_packet_fast(struct qeth_card *card,
struct qeth_qdio_out_q *queue, struct sk_buff *skb,
- struct qeth_hdr *hdr, int elements_needed,
- int offset, int hd_len)
+ struct qeth_hdr *hdr, int offset, int hd_len)
{
struct qeth_qdio_out_buffer *buffer;
int index;
@@ -4419,7 +4416,7 @@
}
EXPORT_SYMBOL_GPL(qeth_tx_timeout);
-int qeth_mdio_read(struct net_device *dev, int phy_id, int regnum)
+static int qeth_mdio_read(struct net_device *dev, int phy_id, int regnum)
{
struct qeth_card *card = dev->ml_priv;
int rc = 0;
@@ -4482,7 +4479,6 @@
}
return rc;
}
-EXPORT_SYMBOL_GPL(qeth_mdio_read);
static int qeth_send_ipa_snmp_cmd(struct qeth_card *card,
struct qeth_cmd_buffer *iob, int len,
@@ -4572,7 +4568,7 @@
return 0;
}
-int qeth_snmp_command(struct qeth_card *card, char __user *udata)
+static int qeth_snmp_command(struct qeth_card *card, char __user *udata)
{
struct qeth_cmd_buffer *iob;
struct qeth_ipa_cmd *cmd;
@@ -4632,7 +4628,6 @@
kfree(qinfo.udata);
return rc;
}
-EXPORT_SYMBOL_GPL(qeth_snmp_command);
static int qeth_setadpparms_query_oat_cb(struct qeth_card *card,
struct qeth_reply *reply, unsigned long data)
@@ -4664,7 +4659,7 @@
return 0;
}
-int qeth_query_oat_command(struct qeth_card *card, char __user *udata)
+static int qeth_query_oat_command(struct qeth_card *card, char __user *udata)
{
int rc = 0;
struct qeth_cmd_buffer *iob;
@@ -4734,7 +4729,6 @@
out:
return rc;
}
-EXPORT_SYMBOL_GPL(qeth_query_oat_command);
static int qeth_query_card_info_cb(struct qeth_card *card,
struct qeth_reply *reply, unsigned long data)
@@ -4775,12 +4769,10 @@
static inline int qeth_get_qdio_q_format(struct qeth_card *card)
{
- switch (card->info.type) {
- case QETH_CARD_TYPE_IQD:
- return 2;
- default:
- return 0;
- }
+ if (card->info.type == QETH_CARD_TYPE_IQD)
+ return QDIO_IQDIO_QFMT;
+ else
+ return QDIO_QETH_QFMT;
}
static void qeth_determine_capabilities(struct qeth_card *card)
@@ -4819,8 +4811,9 @@
QETH_DBF_TEXT_(SETUP, 2, "6err%d", rc);
QETH_DBF_TEXT_(SETUP, 2, "qfmt%d", card->ssqd.qfmt);
- QETH_DBF_TEXT_(SETUP, 2, "%d", card->ssqd.qdioac1);
- QETH_DBF_TEXT_(SETUP, 2, "%d", card->ssqd.qdioac3);
+ QETH_DBF_TEXT_(SETUP, 2, "ac1:%02x", card->ssqd.qdioac1);
+ QETH_DBF_TEXT_(SETUP, 2, "ac2:%04x", card->ssqd.qdioac2);
+ QETH_DBF_TEXT_(SETUP, 2, "ac3:%04x", card->ssqd.qdioac3);
QETH_DBF_TEXT_(SETUP, 2, "icnt%d", card->ssqd.icnt);
if (!((card->ssqd.qfmt != QDIO_IQDIO_QFMT) ||
((card->ssqd.qdioac1 & CHSC_AC1_INITIATE_INPUTQ) == 0) ||
@@ -5288,6 +5281,83 @@
}
EXPORT_SYMBOL_GPL(qeth_core_get_next_skb);
+int qeth_poll(struct napi_struct *napi, int budget)
+{
+ struct qeth_card *card = container_of(napi, struct qeth_card, napi);
+ int work_done = 0;
+ struct qeth_qdio_buffer *buffer;
+ int done;
+ int new_budget = budget;
+
+ if (card->options.performance_stats) {
+ card->perf_stats.inbound_cnt++;
+ card->perf_stats.inbound_start_time = qeth_get_micros();
+ }
+
+ while (1) {
+ if (!card->rx.b_count) {
+ card->rx.qdio_err = 0;
+ card->rx.b_count = qdio_get_next_buffers(
+ card->data.ccwdev, 0, &card->rx.b_index,
+ &card->rx.qdio_err);
+ if (card->rx.b_count <= 0) {
+ card->rx.b_count = 0;
+ break;
+ }
+ card->rx.b_element =
+ &card->qdio.in_q->bufs[card->rx.b_index]
+ .buffer->element[0];
+ card->rx.e_offset = 0;
+ }
+
+ while (card->rx.b_count) {
+ buffer = &card->qdio.in_q->bufs[card->rx.b_index];
+ if (!(card->rx.qdio_err &&
+ qeth_check_qdio_errors(card, buffer->buffer,
+ card->rx.qdio_err, "qinerr")))
+ work_done +=
+ card->discipline->process_rx_buffer(
+ card, new_budget, &done);
+ else
+ done = 1;
+
+ if (done) {
+ if (card->options.performance_stats)
+ card->perf_stats.bufs_rec++;
+ qeth_put_buffer_pool_entry(card,
+ buffer->pool_entry);
+ qeth_queue_input_buffer(card, card->rx.b_index);
+ card->rx.b_count--;
+ if (card->rx.b_count) {
+ card->rx.b_index =
+ (card->rx.b_index + 1) %
+ QDIO_MAX_BUFFERS_PER_Q;
+ card->rx.b_element =
+ &card->qdio.in_q
+ ->bufs[card->rx.b_index]
+ .buffer->element[0];
+ card->rx.e_offset = 0;
+ }
+ }
+
+ if (work_done >= budget)
+ goto out;
+ else
+ new_budget = budget - work_done;
+ }
+ }
+
+ napi_complete(napi);
+ if (qdio_start_irq(card->data.ccwdev, 0))
+ napi_schedule(&card->napi);
+out:
+ if (card->options.performance_stats)
+ card->perf_stats.inbound_time += qeth_get_micros() -
+ card->perf_stats.inbound_start_time;
+ return work_done;
+}
+EXPORT_SYMBOL_GPL(qeth_poll);
+
int qeth_setassparms_cb(struct qeth_card *card,
struct qeth_reply *reply, unsigned long data)
{
@@ -5678,23 +5748,12 @@
static void qeth_core_shutdown(struct ccwgroup_device *gdev)
{
struct qeth_card *card = dev_get_drvdata(&gdev->dev);
- if (card->discipline && card->discipline->shutdown)
- card->discipline->shutdown(gdev);
-}
-
-static int qeth_core_prepare(struct ccwgroup_device *gdev)
-{
- struct qeth_card *card = dev_get_drvdata(&gdev->dev);
- if (card->discipline && card->discipline->prepare)
- return card->discipline->prepare(gdev);
- return 0;
-}
-
-static void qeth_core_complete(struct ccwgroup_device *gdev)
-{
- struct qeth_card *card = dev_get_drvdata(&gdev->dev);
- if (card->discipline && card->discipline->complete)
- card->discipline->complete(gdev);
+ qeth_set_allowed_threads(card, 0, 1);
+ if ((gdev->state == CCWGROUP_ONLINE) && card->info.hwtrap)
+ qeth_hw_trap(card, QETH_DIAGS_TRAP_DISARM);
+ qeth_qdio_clear_card(card, 0);
+ qeth_clear_qdio_buffers(card);
+ qdio_free(CARD_DDEV(card));
}
static int qeth_core_freeze(struct ccwgroup_device *gdev)
@@ -5731,8 +5790,8 @@
.set_online = qeth_core_set_online,
.set_offline = qeth_core_set_offline,
.shutdown = qeth_core_shutdown,
- .prepare = qeth_core_prepare,
- .complete = qeth_core_complete,
+ .prepare = NULL,
+ .complete = NULL,
.freeze = qeth_core_freeze,
.thaw = qeth_core_thaw,
.restore = qeth_core_restore,
@@ -5762,6 +5821,60 @@
NULL,
};
+int qeth_do_ioctl(struct net_device *dev, struct ifreq *rq, int cmd)
+{
+ struct qeth_card *card = dev->ml_priv;
+ struct mii_ioctl_data *mii_data;
+ int rc = 0;
+
+ if (!card)
+ return -ENODEV;
+
+ if (!qeth_card_hw_is_reachable(card))
+ return -ENODEV;
+
+ if (card->info.type == QETH_CARD_TYPE_OSN)
+ return -EPERM;
+
+ switch (cmd) {
+ case SIOC_QETH_ADP_SET_SNMP_CONTROL:
+ rc = qeth_snmp_command(card, rq->ifr_ifru.ifru_data);
+ break;
+ case SIOC_QETH_GET_CARD_TYPE:
+ if ((card->info.type == QETH_CARD_TYPE_OSD ||
+ card->info.type == QETH_CARD_TYPE_OSM ||
+ card->info.type == QETH_CARD_TYPE_OSX) &&
+ !card->info.guestlan)
+ return 1;
+ else
+ return 0;
+ case SIOCGMIIPHY:
+ mii_data = if_mii(rq);
+ mii_data->phy_id = 0;
+ break;
+ case SIOCGMIIREG:
+ mii_data = if_mii(rq);
+ if (mii_data->phy_id != 0)
+ rc = -EINVAL;
+ else
+ mii_data->val_out = qeth_mdio_read(dev,
+ mii_data->phy_id, mii_data->reg_num);
+ break;
+ case SIOC_QETH_QUERY_OAT:
+ rc = qeth_query_oat_command(card, rq->ifr_ifru.ifru_data);
+ break;
+ default:
+ if (card->discipline->do_ioctl)
+ rc = card->discipline->do_ioctl(dev, rq, cmd);
+ else
+ rc = -EOPNOTSUPP;
+ }
+ if (rc)
+ QETH_CARD_TEXT_(card, 2, "ioce%x", rc);
+ return rc;
+}
+EXPORT_SYMBOL_GPL(qeth_do_ioctl);
+
static struct {
const char str[ETH_GSTRING_LEN];
} qeth_ethtool_stats_keys[] = {
@@ -5896,104 +6009,124 @@
}
EXPORT_SYMBOL_GPL(qeth_core_get_drvinfo);
-/* Helper function to fill 'advertizing' and 'supported' which are the same. */
-/* Autoneg and full-duplex are supported and advertized uncondionally. */
-/* Always advertize and support all speeds up to specified, and only one */
+/* Helper function to fill 'advertising' and 'supported' which are the same. */
+/* Autoneg and full-duplex are supported and advertised unconditionally. */
+/* Always advertise and support all speeds up to specified, and only one */
/* specified port type. */
-static void qeth_set_ecmd_adv_sup(struct ethtool_cmd *ecmd,
+static void qeth_set_cmd_adv_sup(struct ethtool_link_ksettings *cmd,
int maxspeed, int porttype)
{
- int port_sup, port_adv, spd_sup, spd_adv;
+ ethtool_link_ksettings_zero_link_mode(cmd, supported);
+ ethtool_link_ksettings_zero_link_mode(cmd, advertising);
+ ethtool_link_ksettings_zero_link_mode(cmd, lp_advertising);
+
+ ethtool_link_ksettings_add_link_mode(cmd, supported, Autoneg);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, Autoneg);
switch (porttype) {
case PORT_TP:
- port_sup = SUPPORTED_TP;
- port_adv = ADVERTISED_TP;
+ ethtool_link_ksettings_add_link_mode(cmd, supported, TP);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, TP);
break;
case PORT_FIBRE:
- port_sup = SUPPORTED_FIBRE;
- port_adv = ADVERTISED_FIBRE;
+ ethtool_link_ksettings_add_link_mode(cmd, supported, FIBRE);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, FIBRE);
break;
default:
- port_sup = SUPPORTED_TP;
- port_adv = ADVERTISED_TP;
+ ethtool_link_ksettings_add_link_mode(cmd, supported, TP);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising, TP);
WARN_ON_ONCE(1);
}
- /* "Fallthrough" case'es ordered from high to low result in setting */
- /* flags cumulatively, starting from the specified speed and down to */
- /* the lowest possible. */
- spd_sup = 0;
- spd_adv = 0;
+ /* fallthrough from high to low, to select all legal speeds: */
switch (maxspeed) {
case SPEED_10000:
- spd_sup |= SUPPORTED_10000baseT_Full;
- spd_adv |= ADVERTISED_10000baseT_Full;
+ ethtool_link_ksettings_add_link_mode(cmd, supported,
+ 10000baseT_Full);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ 10000baseT_Full);
case SPEED_1000:
- spd_sup |= SUPPORTED_1000baseT_Half | SUPPORTED_1000baseT_Full;
- spd_adv |= ADVERTISED_1000baseT_Half |
- ADVERTISED_1000baseT_Full;
+ ethtool_link_ksettings_add_link_mode(cmd, supported,
+ 1000baseT_Full);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ 1000baseT_Full);
+ ethtool_link_ksettings_add_link_mode(cmd, supported,
+ 1000baseT_Half);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ 1000baseT_Half);
case SPEED_100:
- spd_sup |= SUPPORTED_100baseT_Half | SUPPORTED_100baseT_Full;
- spd_adv |= ADVERTISED_100baseT_Half | ADVERTISED_100baseT_Full;
+ ethtool_link_ksettings_add_link_mode(cmd, supported,
+ 100baseT_Full);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ 100baseT_Full);
+ ethtool_link_ksettings_add_link_mode(cmd, supported,
+ 100baseT_Half);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ 100baseT_Half);
case SPEED_10:
- spd_sup |= SUPPORTED_10baseT_Half | SUPPORTED_10baseT_Full;
- spd_adv |= ADVERTISED_10baseT_Half | ADVERTISED_10baseT_Full;
- break;
+ ethtool_link_ksettings_add_link_mode(cmd, supported,
+ 10baseT_Full);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ 10baseT_Full);
+ ethtool_link_ksettings_add_link_mode(cmd, supported,
+ 10baseT_Half);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ 10baseT_Half);
+ /* end fallthrough */
+ break;
default:
- spd_sup = SUPPORTED_10baseT_Half | SUPPORTED_10baseT_Full;
- spd_adv = ADVERTISED_10baseT_Half | ADVERTISED_10baseT_Full;
+ ethtool_link_ksettings_add_link_mode(cmd, supported,
+ 10baseT_Full);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ 10baseT_Full);
+ ethtool_link_ksettings_add_link_mode(cmd, supported,
+ 10baseT_Half);
+ ethtool_link_ksettings_add_link_mode(cmd, advertising,
+ 10baseT_Half);
WARN_ON_ONCE(1);
}
- ecmd->advertising = ADVERTISED_Autoneg | port_adv | spd_adv;
- ecmd->supported = SUPPORTED_Autoneg | port_sup | spd_sup;
}
-int qeth_core_ethtool_get_settings(struct net_device *netdev,
- struct ethtool_cmd *ecmd)
+int qeth_core_ethtool_get_link_ksettings(struct net_device *netdev,
+ struct ethtool_link_ksettings *cmd)
{
struct qeth_card *card = netdev->ml_priv;
enum qeth_link_types link_type;
struct carrier_info carrier_info;
int rc;
- u32 speed;
if ((card->info.type == QETH_CARD_TYPE_IQD) || (card->info.guestlan))
link_type = QETH_LINK_TYPE_10GBIT_ETH;
else
link_type = card->info.link_type;
- ecmd->transceiver = XCVR_INTERNAL;
- ecmd->duplex = DUPLEX_FULL;
- ecmd->autoneg = AUTONEG_ENABLE;
+ cmd->base.duplex = DUPLEX_FULL;
+ cmd->base.autoneg = AUTONEG_ENABLE;
+ cmd->base.phy_address = 0;
+ cmd->base.mdio_support = 0;
+ cmd->base.eth_tp_mdix = ETH_TP_MDI_INVALID;
+ cmd->base.eth_tp_mdix_ctrl = ETH_TP_MDI_INVALID;
switch (link_type) {
case QETH_LINK_TYPE_FAST_ETH:
case QETH_LINK_TYPE_LANE_ETH100:
- qeth_set_ecmd_adv_sup(ecmd, SPEED_100, PORT_TP);
- speed = SPEED_100;
- ecmd->port = PORT_TP;
+ cmd->base.speed = SPEED_100;
+ cmd->base.port = PORT_TP;
break;
-
case QETH_LINK_TYPE_GBIT_ETH:
case QETH_LINK_TYPE_LANE_ETH1000:
- qeth_set_ecmd_adv_sup(ecmd, SPEED_1000, PORT_FIBRE);
- speed = SPEED_1000;
- ecmd->port = PORT_FIBRE;
+ cmd->base.speed = SPEED_1000;
+ cmd->base.port = PORT_FIBRE;
break;
-
case QETH_LINK_TYPE_10GBIT_ETH:
- qeth_set_ecmd_adv_sup(ecmd, SPEED_10000, PORT_FIBRE);
- speed = SPEED_10000;
- ecmd->port = PORT_FIBRE;
+ cmd->base.speed = SPEED_10000;
+ cmd->base.port = PORT_FIBRE;
break;
-
default:
- qeth_set_ecmd_adv_sup(ecmd, SPEED_10, PORT_TP);
- speed = SPEED_10;
- ecmd->port = PORT_TP;
+ cmd->base.speed = SPEED_10;
+ cmd->base.port = PORT_TP;
}
- ethtool_cmd_speed_set(ecmd, speed);
+ qeth_set_cmd_adv_sup(cmd, cmd->base.speed, cmd->base.port);
/* Check if we can obtain more accurate information. */
/* If QUERY_CARD_INFO command is not supported or fails, */
@@ -6018,49 +6151,48 @@
switch (carrier_info.card_type) {
case CARD_INFO_TYPE_1G_COPPER_A:
case CARD_INFO_TYPE_1G_COPPER_B:
- qeth_set_ecmd_adv_sup(ecmd, SPEED_1000, PORT_TP);
- ecmd->port = PORT_TP;
+ cmd->base.port = PORT_TP;
+ qeth_set_cmd_adv_sup(cmd, SPEED_1000, cmd->base.port);
break;
case CARD_INFO_TYPE_1G_FIBRE_A:
case CARD_INFO_TYPE_1G_FIBRE_B:
- qeth_set_ecmd_adv_sup(ecmd, SPEED_1000, PORT_FIBRE);
- ecmd->port = PORT_FIBRE;
+ cmd->base.port = PORT_FIBRE;
+ qeth_set_cmd_adv_sup(cmd, SPEED_1000, cmd->base.port);
break;
case CARD_INFO_TYPE_10G_FIBRE_A:
case CARD_INFO_TYPE_10G_FIBRE_B:
- qeth_set_ecmd_adv_sup(ecmd, SPEED_10000, PORT_FIBRE);
- ecmd->port = PORT_FIBRE;
+ cmd->base.port = PORT_FIBRE;
+ qeth_set_cmd_adv_sup(cmd, SPEED_10000, cmd->base.port);
break;
}
switch (carrier_info.port_mode) {
case CARD_INFO_PORTM_FULLDUPLEX:
- ecmd->duplex = DUPLEX_FULL;
+ cmd->base.duplex = DUPLEX_FULL;
break;
case CARD_INFO_PORTM_HALFDUPLEX:
- ecmd->duplex = DUPLEX_HALF;
+ cmd->base.duplex = DUPLEX_HALF;
break;
}
switch (carrier_info.port_speed) {
case CARD_INFO_PORTS_10M:
- speed = SPEED_10;
+ cmd->base.speed = SPEED_10;
break;
case CARD_INFO_PORTS_100M:
- speed = SPEED_100;
+ cmd->base.speed = SPEED_100;
break;
case CARD_INFO_PORTS_1G:
- speed = SPEED_1000;
+ cmd->base.speed = SPEED_1000;
break;
case CARD_INFO_PORTS_10G:
- speed = SPEED_10000;
+ cmd->base.speed = SPEED_10000;
break;
}
- ethtool_cmd_speed_set(ecmd, speed);
return 0;
}
-EXPORT_SYMBOL_GPL(qeth_core_ethtool_get_settings);
+EXPORT_SYMBOL_GPL(qeth_core_ethtool_get_link_ksettings);
/* Callback to handle checksum offload command reply from OSA card.
* Verify that required features have been enabled on the card.
diff --git a/drivers/s390/net/qeth_core_mpc.h b/drivers/s390/net/qeth_core_mpc.h
index bc69d0a..4accb0a 100644
--- a/drivers/s390/net/qeth_core_mpc.h
+++ b/drivers/s390/net/qeth_core_mpc.h
@@ -29,7 +29,6 @@
#define QETH_TIMEOUT (10 * HZ)
#define QETH_IPA_TIMEOUT (45 * HZ)
#define QETH_IDX_COMMAND_SEQNO 0xffff0000
-#define SR_INFO_LEN 16
#define QETH_CLEAR_CHANNEL_PARM -10
#define QETH_HALT_CHANNEL_PARM -11
@@ -65,7 +64,6 @@
QETH_LINK_TYPE_LANE_TR = 0x82,
QETH_LINK_TYPE_LANE_ETH1000 = 0x83,
QETH_LINK_TYPE_LANE = 0x88,
- QETH_LINK_TYPE_ATM_NATIVE = 0x90,
};
/*
@@ -185,8 +183,6 @@
IPA_RC_ENOMEM = 0xfffe,
IPA_RC_FFFF = 0xffff
};
-/* for DELIP */
-#define IPA_RC_IP_ADDRESS_NOT_DEFINED IPA_RC_PRIMARY_ALREADY_DEFINED
/* for SET_DIAGNOSTIC_ASSIST */
#define IPA_RC_INVALID_SUBCMD IPA_RC_IP_TABLE_FULL
#define IPA_RC_HARDWARE_AUTH_ERROR IPA_RC_UNKNOWN_ERROR
@@ -631,14 +627,6 @@
IPA_ADDR_CHANGE_CODE_MACADDR = 0x02,
IPA_ADDR_CHANGE_CODE_REMOVAL = 0x80, /* else addition */
};
-enum qeth_ipa_addr_change_retcode {
- IPA_ADDR_CHANGE_RETCODE_OK = 0x0000,
- IPA_ADDR_CHANGE_RETCODE_LOSTEVENTS = 0x0010,
-};
-enum qeth_ipa_addr_change_lostmask {
- IPA_ADDR_CHANGE_MASK_OVERFLOW = 0x01,
- IPA_ADDR_CHANGE_MASK_STATECHANGE = 0x02,
-};
struct qeth_ipacmd_addr_change_entry {
struct net_if_token token;
@@ -817,9 +805,4 @@
((buffer) && \
(*(buffer + ((*(buffer + 0x0b)) + 4)) == 0xc1))
-#define ADDR_FRAME_TYPE_DIX 1
-#define ADDR_FRAME_TYPE_802_3 2
-#define ADDR_FRAME_TYPE_TR_WITHOUT_SR 0x10
-#define ADDR_FRAME_TYPE_TR_WITH_SR 0x20
-
#endif
diff --git a/drivers/s390/net/qeth_l2_main.c b/drivers/s390/net/qeth_l2_main.c
index af4e6a6..1b07f38 100644
--- a/drivers/s390/net/qeth_l2_main.c
+++ b/drivers/s390/net/qeth_l2_main.c
@@ -16,7 +16,6 @@
#include <linux/kernel.h>
#include <linux/slab.h>
#include <linux/etherdevice.h>
-#include <linux/mii.h>
#include <linux/ip.h>
#include <linux/list.h>
#include <linux/hash.h>
@@ -28,63 +27,12 @@
static int qeth_l2_set_offline(struct ccwgroup_device *);
static int qeth_l2_stop(struct net_device *);
static void qeth_l2_set_rx_mode(struct net_device *);
-static int qeth_l2_recover(void *);
static void qeth_bridgeport_query_support(struct qeth_card *card);
static void qeth_bridge_state_change(struct qeth_card *card,
struct qeth_ipa_cmd *cmd);
static void qeth_bridge_host_event(struct qeth_card *card,
struct qeth_ipa_cmd *cmd);
-static int qeth_l2_do_ioctl(struct net_device *dev, struct ifreq *rq, int cmd)
-{
- struct qeth_card *card = dev->ml_priv;
- struct mii_ioctl_data *mii_data;
- int rc = 0;
-
- if (!card)
- return -ENODEV;
-
- if (!qeth_card_hw_is_reachable(card))
- return -ENODEV;
-
- if (card->info.type == QETH_CARD_TYPE_OSN)
- return -EPERM;
-
- switch (cmd) {
- case SIOC_QETH_ADP_SET_SNMP_CONTROL:
- rc = qeth_snmp_command(card, rq->ifr_ifru.ifru_data);
- break;
- case SIOC_QETH_GET_CARD_TYPE:
- if ((card->info.type == QETH_CARD_TYPE_OSD ||
- card->info.type == QETH_CARD_TYPE_OSM ||
- card->info.type == QETH_CARD_TYPE_OSX) &&
- !card->info.guestlan)
- return 1;
- return 0;
- break;
- case SIOCGMIIPHY:
- mii_data = if_mii(rq);
- mii_data->phy_id = 0;
- break;
- case SIOCGMIIREG:
- mii_data = if_mii(rq);
- if (mii_data->phy_id != 0)
- rc = -EINVAL;
- else
- mii_data->val_out = qeth_mdio_read(dev,
- mii_data->phy_id, mii_data->reg_num);
- break;
- case SIOC_QETH_QUERY_OAT:
- rc = qeth_query_oat_command(card, rq->ifr_ifru.ifru_data);
- break;
- default:
- rc = -EOPNOTSUPP;
- }
- if (rc)
- QETH_CARD_TEXT_(card, 2, "ioce%d", rc);
- return rc;
-}
-
static int qeth_l2_verify_dev(struct net_device *dev)
{
struct qeth_card *card;
@@ -332,7 +280,7 @@
else
hdr->hdr.l2.flags[2] |= QETH_LAYER2_FLAG_UNICAST;
- hdr->hdr.l2.pkt_length = skb->len-QETH_HEADER_SIZE;
+ hdr->hdr.l2.pkt_length = skb->len - sizeof(struct qeth_hdr);
/* VSWITCH relies on the VLAN
* information to be present in
* the QDIO header */
@@ -552,81 +500,6 @@
return work_done;
}
-static int qeth_l2_poll(struct napi_struct *napi, int budget)
-{
- struct qeth_card *card = container_of(napi, struct qeth_card, napi);
- int work_done = 0;
- struct qeth_qdio_buffer *buffer;
- int done;
- int new_budget = budget;
-
- if (card->options.performance_stats) {
- card->perf_stats.inbound_cnt++;
- card->perf_stats.inbound_start_time = qeth_get_micros();
- }
-
- while (1) {
- if (!card->rx.b_count) {
- card->rx.qdio_err = 0;
- card->rx.b_count = qdio_get_next_buffers(
- card->data.ccwdev, 0, &card->rx.b_index,
- &card->rx.qdio_err);
- if (card->rx.b_count <= 0) {
- card->rx.b_count = 0;
- break;
- }
- card->rx.b_element =
- &card->qdio.in_q->bufs[card->rx.b_index]
- .buffer->element[0];
- card->rx.e_offset = 0;
- }
-
- while (card->rx.b_count) {
- buffer = &card->qdio.in_q->bufs[card->rx.b_index];
- if (!(card->rx.qdio_err &&
- qeth_check_qdio_errors(card, buffer->buffer,
- card->rx.qdio_err, "qinerr")))
- work_done += qeth_l2_process_inbound_buffer(
- card, new_budget, &done);
- else
- done = 1;
-
- if (done) {
- if (card->options.performance_stats)
- card->perf_stats.bufs_rec++;
- qeth_put_buffer_pool_entry(card,
- buffer->pool_entry);
- qeth_queue_input_buffer(card, card->rx.b_index);
- card->rx.b_count--;
- if (card->rx.b_count) {
- card->rx.b_index =
- (card->rx.b_index + 1) %
- QDIO_MAX_BUFFERS_PER_Q;
- card->rx.b_element =
- &card->qdio.in_q
- ->bufs[card->rx.b_index]
- .buffer->element[0];
- card->rx.e_offset = 0;
- }
- }
-
- if (work_done >= budget)
- goto out;
- else
- new_budget = budget - work_done;
- }
- }
-
- napi_complete(napi);
- if (qdio_start_irq(card->data.ccwdev, 0))
- napi_schedule(&card->napi);
-out:
- if (card->options.performance_stats)
- card->perf_stats.inbound_time += qeth_get_micros() -
- card->perf_stats.inbound_start_time;
- return work_done;
-}
-
static int qeth_l2_request_initial_mac(struct qeth_card *card)
{
int rc = 0;
@@ -808,7 +681,8 @@
qeth_promisc_to_bridge(card);
}
-static int qeth_l2_hard_start_xmit(struct sk_buff *skb, struct net_device *dev)
+static netdev_tx_t qeth_l2_hard_start_xmit(struct sk_buff *skb,
+ struct net_device *dev)
{
int rc;
struct qeth_hdr *hdr = NULL;
@@ -910,7 +784,7 @@
elements);
} else
rc = qeth_do_send_packet_fast(card, queue, new_skb, hdr,
- elements, data_offset, hd_len);
+ data_offset, hd_len);
if (!rc) {
card->stats.tx_packets++;
card->stats.tx_bytes += tx_bytes;
@@ -1043,7 +917,7 @@
.get_ethtool_stats = qeth_core_get_ethtool_stats,
.get_sset_count = qeth_core_get_sset_count,
.get_drvinfo = qeth_core_get_drvinfo,
- .get_settings = qeth_core_ethtool_get_settings,
+ .get_link_ksettings = qeth_core_ethtool_get_link_ksettings,
};
static const struct ethtool_ops qeth_l2_osn_ops = {
@@ -1060,7 +934,7 @@
.ndo_start_xmit = qeth_l2_hard_start_xmit,
.ndo_validate_addr = eth_validate_addr,
.ndo_set_rx_mode = qeth_l2_set_rx_mode,
- .ndo_do_ioctl = qeth_l2_do_ioctl,
+ .ndo_do_ioctl = qeth_do_ioctl,
.ndo_set_mac_address = qeth_l2_set_mac_address,
.ndo_change_mtu = qeth_change_mtu,
.ndo_vlan_rx_add_vid = qeth_l2_vlan_rx_add_vid,
@@ -1117,7 +991,7 @@
card->dev->gso_max_size = (QETH_MAX_BUFFER_ELEMENTS(card) - 1) *
PAGE_SIZE;
SET_NETDEV_DEV(card->dev, &card->gdev->dev);
- netif_napi_add(card->dev, &card->napi, qeth_l2_poll, QETH_NAPI_WEIGHT);
+ netif_napi_add(card->dev, &card->napi, qeth_poll, QETH_NAPI_WEIGHT);
netif_carrier_off(card->dev);
return register_netdev(card->dev);
}
@@ -1327,17 +1201,6 @@
pr_info("unregister layer 2 discipline\n");
}
-static void qeth_l2_shutdown(struct ccwgroup_device *gdev)
-{
- struct qeth_card *card = dev_get_drvdata(&gdev->dev);
- qeth_set_allowed_threads(card, 0, 1);
- if ((gdev->state == CCWGROUP_ONLINE) && card->info.hwtrap)
- qeth_hw_trap(card, QETH_DIAGS_TRAP_DISARM);
- qeth_qdio_clear_card(card, 0);
- qeth_clear_qdio_buffers(card);
- qdio_free(CARD_DDEV(card));
-}
-
static int qeth_l2_pm_suspend(struct ccwgroup_device *gdev)
{
struct qeth_card *card = dev_get_drvdata(&gdev->dev);
@@ -1409,15 +1272,16 @@
.start_poll = qeth_qdio_start_poll,
.input_handler = (qdio_handler_t *) qeth_qdio_input_handler,
.output_handler = (qdio_handler_t *) qeth_qdio_output_handler,
+ .process_rx_buffer = qeth_l2_process_inbound_buffer,
.recover = qeth_l2_recover,
.setup = qeth_l2_probe_device,
.remove = qeth_l2_remove_device,
.set_online = qeth_l2_set_online,
.set_offline = qeth_l2_set_offline,
- .shutdown = qeth_l2_shutdown,
.freeze = qeth_l2_pm_suspend,
.thaw = qeth_l2_pm_resume,
.restore = qeth_l2_pm_resume,
+ .do_ioctl = NULL,
.control_event_handler = qeth_l2_control_event,
};
EXPORT_SYMBOL_GPL(qeth_l2_discipline);
diff --git a/drivers/s390/net/qeth_l2_sys.c b/drivers/s390/net/qeth_l2_sys.c
index 692db49..6879723 100644
--- a/drivers/s390/net/qeth_l2_sys.c
+++ b/drivers/s390/net/qeth_l2_sys.c
@@ -8,9 +8,6 @@
#include "qeth_core.h"
#include "qeth_l2.h"
-#define QETH_DEVICE_ATTR(_id, _name, _mode, _show, _store) \
-struct device_attribute dev_attr_##_id = __ATTR(_name, _mode, _show, _store)
-
static ssize_t qeth_bridge_port_role_state_show(struct device *dev,
struct device_attribute *attr, char *buf,
int show_state)
diff --git a/drivers/s390/net/qeth_l3_main.c b/drivers/s390/net/qeth_l3_main.c
index 653f0fb..6e0354e 100644
--- a/drivers/s390/net/qeth_l3_main.c
+++ b/drivers/s390/net/qeth_l3_main.c
@@ -16,7 +16,6 @@
#include <linux/errno.h>
#include <linux/kernel.h>
#include <linux/etherdevice.h>
-#include <linux/mii.h>
#include <linux/ip.h>
#include <linux/ipv6.h>
#include <linux/inetdevice.h>
@@ -36,16 +35,12 @@
static int qeth_l3_set_offline(struct ccwgroup_device *);
-static int qeth_l3_recover(void *);
static int qeth_l3_stop(struct net_device *);
static void qeth_l3_set_multicast_list(struct net_device *);
-static int qeth_l3_neigh_setup(struct net_device *, struct neigh_parms *);
static int qeth_l3_register_addr_entry(struct qeth_card *,
struct qeth_ipaddr *);
static int qeth_l3_deregister_addr_entry(struct qeth_card *,
struct qeth_ipaddr *);
-static int __qeth_l3_set_online(struct ccwgroup_device *, int);
-static int __qeth_l3_set_offline(struct ccwgroup_device *, int);
static int qeth_l3_isxdigit(char *buf)
{
@@ -1341,7 +1336,7 @@
return qeth_send_ipa_cmd(card, iob, qeth_diags_trace_cb, NULL);
}
-static void qeth_l3_get_mac_for_ipm(__u32 ipm, char *mac)
+static void qeth_l3_get_mac_for_ipm(__be32 ipm, char *mac)
{
ip_eth_mc_map(ipm, mac);
}
@@ -1414,7 +1409,7 @@
im4 = rcu_dereference(im4->next_rcu)) {
qeth_l3_get_mac_for_ipm(im4->multiaddr, buf);
- tmp->u.a4.addr = im4->multiaddr;
+ tmp->u.a4.addr = be32_to_cpu(im4->multiaddr);
memcpy(tmp->mac, buf, sizeof(tmp->mac));
ipm = qeth_l3_ip_from_hash(card, tmp);
@@ -1425,7 +1420,7 @@
if (!ipm)
continue;
memcpy(ipm->mac, buf, sizeof(tmp->mac));
- ipm->u.a4.addr = im4->multiaddr;
+ ipm->u.a4.addr = be32_to_cpu(im4->multiaddr);
ipm->is_multicast = 1;
ipm->disp_flag = QETH_DISP_ADDR_ADD;
hash_add(card->ip_mc_htable,
@@ -1598,8 +1593,8 @@
spin_lock_bh(&card->ip_lock);
for (ifa = in_dev->ifa_list; ifa; ifa = ifa->ifa_next) {
- addr->u.a4.addr = ifa->ifa_address;
- addr->u.a4.mask = ifa->ifa_mask;
+ addr->u.a4.addr = be32_to_cpu(ifa->ifa_address);
+ addr->u.a4.mask = be32_to_cpu(ifa->ifa_mask);
addr->type = QETH_IP_TYPE_NORMAL;
qeth_l3_delete_ip(card, addr);
}
@@ -1690,25 +1685,25 @@
struct sk_buff *skb, struct qeth_hdr *hdr,
unsigned short *vlan_id)
{
- __be16 prot;
+ __u16 prot;
struct iphdr *ip_hdr;
unsigned char tg_addr[MAX_ADDR_LEN];
int is_vlan = 0;
if (!(hdr->hdr.l3.flags & QETH_HDR_PASSTHRU)) {
- prot = htons((hdr->hdr.l3.flags & QETH_HDR_IPV6)? ETH_P_IPV6 :
- ETH_P_IP);
+ prot = (hdr->hdr.l3.flags & QETH_HDR_IPV6) ? ETH_P_IPV6 :
+ ETH_P_IP;
switch (hdr->hdr.l3.flags & QETH_HDR_CAST_MASK) {
case QETH_CAST_MULTICAST:
switch (prot) {
#ifdef CONFIG_QETH_IPV6
- case __constant_htons(ETH_P_IPV6):
+ case ETH_P_IPV6:
ndisc_mc_map((struct in6_addr *)
skb->data + 24,
tg_addr, card->dev, 0);
break;
#endif
- case __constant_htons(ETH_P_IP):
+ case ETH_P_IP:
ip_hdr = (struct iphdr *)skb->data;
ip_eth_mc_map(ip_hdr->daddr, tg_addr);
break;
@@ -1795,7 +1790,7 @@
magic = *(__u16 *)skb->data;
if ((card->info.type == QETH_CARD_TYPE_IQD) &&
(magic == ETH_P_AF_IUCV)) {
- skb->protocol = ETH_P_AF_IUCV;
+ skb->protocol = cpu_to_be16(ETH_P_AF_IUCV);
skb->pkt_type = PACKET_HOST;
skb->mac_header = NET_SKB_PAD;
skb->dev = card->dev;
@@ -1834,81 +1829,6 @@
return work_done;
}
-static int qeth_l3_poll(struct napi_struct *napi, int budget)
-{
- struct qeth_card *card = container_of(napi, struct qeth_card, napi);
- int work_done = 0;
- struct qeth_qdio_buffer *buffer;
- int done;
- int new_budget = budget;
-
- if (card->options.performance_stats) {
- card->perf_stats.inbound_cnt++;
- card->perf_stats.inbound_start_time = qeth_get_micros();
- }
-
- while (1) {
- if (!card->rx.b_count) {
- card->rx.qdio_err = 0;
- card->rx.b_count = qdio_get_next_buffers(
- card->data.ccwdev, 0, &card->rx.b_index,
- &card->rx.qdio_err);
- if (card->rx.b_count <= 0) {
- card->rx.b_count = 0;
- break;
- }
- card->rx.b_element =
- &card->qdio.in_q->bufs[card->rx.b_index]
- .buffer->element[0];
- card->rx.e_offset = 0;
- }
-
- while (card->rx.b_count) {
- buffer = &card->qdio.in_q->bufs[card->rx.b_index];
- if (!(card->rx.qdio_err &&
- qeth_check_qdio_errors(card, buffer->buffer,
- card->rx.qdio_err, "qinerr")))
- work_done += qeth_l3_process_inbound_buffer(
- card, new_budget, &done);
- else
- done = 1;
-
- if (done) {
- if (card->options.performance_stats)
- card->perf_stats.bufs_rec++;
- qeth_put_buffer_pool_entry(card,
- buffer->pool_entry);
- qeth_queue_input_buffer(card, card->rx.b_index);
- card->rx.b_count--;
- if (card->rx.b_count) {
- card->rx.b_index =
- (card->rx.b_index + 1) %
- QDIO_MAX_BUFFERS_PER_Q;
- card->rx.b_element =
- &card->qdio.in_q
- ->bufs[card->rx.b_index]
- .buffer->element[0];
- card->rx.e_offset = 0;
- }
- }
-
- if (work_done >= budget)
- goto out;
- else
- new_budget = budget - work_done;
- }
- }
-
- napi_complete(napi);
- if (qdio_start_irq(card->data.ccwdev, 0))
- napi_schedule(&card->napi);
-out:
- if (card->options.performance_stats)
- card->perf_stats.inbound_time += qeth_get_micros() -
- card->perf_stats.inbound_start_time;
- return work_done;
-}
-
static int qeth_l3_verify_vlan_dev(struct net_device *dev,
struct qeth_card *card)
{
@@ -2461,15 +2381,8 @@
{
struct qeth_card *card = dev->ml_priv;
struct qeth_arp_cache_entry arp_entry;
- struct mii_ioctl_data *mii_data;
int rc = 0;
- if (!card)
- return -ENODEV;
-
- if (!qeth_card_hw_is_reachable(card))
- return -ENODEV;
-
switch (cmd) {
case SIOC_QETH_ARP_SET_NO_ENTRIES:
if (!capable(CAP_NET_ADMIN)) {
@@ -2514,37 +2427,9 @@
}
rc = qeth_l3_arp_flush_cache(card);
break;
- case SIOC_QETH_ADP_SET_SNMP_CONTROL:
- rc = qeth_snmp_command(card, rq->ifr_ifru.ifru_data);
- break;
- case SIOC_QETH_GET_CARD_TYPE:
- if ((card->info.type == QETH_CARD_TYPE_OSD ||
- card->info.type == QETH_CARD_TYPE_OSX) &&
- !card->info.guestlan)
- return 1;
- return 0;
- break;
- case SIOCGMIIPHY:
- mii_data = if_mii(rq);
- mii_data->phy_id = 0;
- break;
- case SIOCGMIIREG:
- mii_data = if_mii(rq);
- if (mii_data->phy_id != 0)
- rc = -EINVAL;
- else
- mii_data->val_out = qeth_mdio_read(dev,
- mii_data->phy_id,
- mii_data->reg_num);
- break;
- case SIOC_QETH_QUERY_OAT:
- rc = qeth_query_oat_command(card, rq->ifr_ifru.ifru_data);
- break;
default:
rc = -EOPNOTSUPP;
}
- if (rc)
- QETH_CARD_TEXT_(card, 2, "ioce%d", rc);
return rc;
}
@@ -2572,10 +2457,10 @@
rcu_read_unlock();
/* try something else */
- if (skb->protocol == ETH_P_IPV6)
+ if (be16_to_cpu(skb->protocol) == ETH_P_IPV6)
return (skb_network_header(skb)[24] == 0xff) ?
RTN_MULTICAST : 0;
- else if (skb->protocol == ETH_P_IP)
+ else if (be16_to_cpu(skb->protocol) == ETH_P_IP)
return ((skb_network_header(skb)[16] & 0xf0) == 0xe0) ?
RTN_MULTICAST : 0;
/* ... */
@@ -2726,7 +2611,7 @@
hdr->ext.payload_len = (__u16)(skb->len - hdr->ext.dg_hdr_len -
sizeof(struct qeth_hdr_tso));
tcph->check = 0;
- if (skb->protocol == ETH_P_IPV6) {
+ if (be16_to_cpu(skb->protocol) == ETH_P_IPV6) {
ip6h->payload_len = 0;
tcph->check = ~csum_ipv6_magic(&ip6h->saddr, &ip6h->daddr,
0, IPPROTO_TCP, 0);
@@ -2770,10 +2655,11 @@
return elements;
}
-static int qeth_l3_hard_start_xmit(struct sk_buff *skb, struct net_device *dev)
+static netdev_tx_t qeth_l3_hard_start_xmit(struct sk_buff *skb,
+ struct net_device *dev)
{
int rc;
- u16 *tag;
+ __be16 *tag;
struct qeth_hdr *hdr = NULL;
int hdr_elements = 0;
int elements;
@@ -2794,7 +2680,7 @@
if (((card->info.type == QETH_CARD_TYPE_IQD) &&
(((card->options.cq != QETH_CQ_ENABLED) && !ipv) ||
((card->options.cq == QETH_CQ_ENABLED) &&
- (skb->protocol != ETH_P_AF_IUCV)))) ||
+ (be16_to_cpu(skb->protocol) != ETH_P_AF_IUCV)))) ||
card->options.sniffer)
goto tx_drop;
@@ -2847,9 +2733,9 @@
new_skb->data + 8, 4);
skb_copy_to_linear_data_offset(new_skb, 8,
new_skb->data + 12, 4);
- tag = (u16 *)(new_skb->data + 12);
- *tag = __constant_htons(ETH_P_8021Q);
- *(tag + 1) = htons(skb_vlan_tag_get(new_skb));
+ tag = (__be16 *)(new_skb->data + 12);
+ *tag = cpu_to_be16(ETH_P_8021Q);
+ *(tag + 1) = cpu_to_be16(skb_vlan_tag_get(new_skb));
}
}
@@ -2887,7 +2773,7 @@
qeth_l3_fill_header(card, hdr, new_skb, ipv,
cast_type);
} else {
- if (new_skb->protocol == ETH_P_AF_IUCV)
+ if (be16_to_cpu(new_skb->protocol) == ETH_P_AF_IUCV)
qeth_l3_fill_af_iucv_hdr(card, hdr, new_skb);
else {
qeth_l3_fill_header(card, hdr, new_skb, ipv,
@@ -2925,7 +2811,7 @@
rc = qeth_do_send_packet(card, queue, new_skb, hdr, elements);
} else
rc = qeth_do_send_packet_fast(card, queue, new_skb, hdr,
- elements, data_offset, 0);
+ data_offset, 0);
if (!rc) {
card->stats.tx_packets++;
@@ -3026,7 +2912,7 @@
.get_ethtool_stats = qeth_core_get_ethtool_stats,
.get_sset_count = qeth_core_get_sset_count,
.get_drvinfo = qeth_core_get_drvinfo,
- .get_settings = qeth_core_ethtool_get_settings,
+ .get_link_ksettings = qeth_core_ethtool_get_link_ksettings,
};
/*
@@ -3060,7 +2946,7 @@
.ndo_start_xmit = qeth_l3_hard_start_xmit,
.ndo_validate_addr = eth_validate_addr,
.ndo_set_rx_mode = qeth_l3_set_multicast_list,
- .ndo_do_ioctl = qeth_l3_do_ioctl,
+ .ndo_do_ioctl = qeth_do_ioctl,
.ndo_change_mtu = qeth_change_mtu,
.ndo_fix_features = qeth_fix_features,
.ndo_set_features = qeth_set_features,
@@ -3076,7 +2962,7 @@
.ndo_start_xmit = qeth_l3_hard_start_xmit,
.ndo_validate_addr = eth_validate_addr,
.ndo_set_rx_mode = qeth_l3_set_multicast_list,
- .ndo_do_ioctl = qeth_l3_do_ioctl,
+ .ndo_do_ioctl = qeth_do_ioctl,
.ndo_change_mtu = qeth_change_mtu,
.ndo_fix_features = qeth_fix_features,
.ndo_set_features = qeth_set_features,
@@ -3145,7 +3031,7 @@
PAGE_SIZE;
SET_NETDEV_DEV(card->dev, &card->gdev->dev);
- netif_napi_add(card->dev, &card->napi, qeth_l3_poll, QETH_NAPI_WEIGHT);
+ netif_napi_add(card->dev, &card->napi, qeth_poll, QETH_NAPI_WEIGHT);
netif_carrier_off(card->dev);
return register_netdev(card->dev);
}
@@ -3366,17 +3252,6 @@
return 0;
}
-static void qeth_l3_shutdown(struct ccwgroup_device *gdev)
-{
- struct qeth_card *card = dev_get_drvdata(&gdev->dev);
- qeth_set_allowed_threads(card, 0, 1);
- if ((gdev->state == CCWGROUP_ONLINE) && card->info.hwtrap)
- qeth_hw_trap(card, QETH_DIAGS_TRAP_DISARM);
- qeth_qdio_clear_card(card, 0);
- qeth_clear_qdio_buffers(card);
- qdio_free(CARD_DDEV(card));
-}
-
static int qeth_l3_pm_suspend(struct ccwgroup_device *gdev)
{
struct qeth_card *card = dev_get_drvdata(&gdev->dev);
@@ -3434,15 +3309,16 @@
.start_poll = qeth_qdio_start_poll,
.input_handler = (qdio_handler_t *) qeth_qdio_input_handler,
.output_handler = (qdio_handler_t *) qeth_qdio_output_handler,
+ .process_rx_buffer = qeth_l3_process_inbound_buffer,
.recover = qeth_l3_recover,
.setup = qeth_l3_probe_device,
.remove = qeth_l3_remove_device,
.set_online = qeth_l3_set_online,
.set_offline = qeth_l3_set_offline,
- .shutdown = qeth_l3_shutdown,
.freeze = qeth_l3_pm_suspend,
.thaw = qeth_l3_pm_resume,
.restore = qeth_l3_pm_resume,
+ .do_ioctl = qeth_l3_do_ioctl,
.control_event_handler = qeth_l3_control_event,
};
EXPORT_SYMBOL_GPL(qeth_l3_discipline);
@@ -3466,8 +3342,8 @@
addr = qeth_l3_get_addr_buffer(QETH_PROT_IPV4);
if (addr) {
- addr->u.a4.addr = ifa->ifa_address;
- addr->u.a4.mask = ifa->ifa_mask;
+ addr->u.a4.addr = be32_to_cpu(ifa->ifa_address);
+ addr->u.a4.mask = be32_to_cpu(ifa->ifa_mask);
addr->type = QETH_IP_TYPE_NORMAL;
} else
return NOTIFY_DONE;
diff --git a/drivers/s390/net/qeth_l3_sys.c b/drivers/s390/net/qeth_l3_sys.c
index 05e9471..ff29a4b 100644
--- a/drivers/s390/net/qeth_l3_sys.c
+++ b/drivers/s390/net/qeth_l3_sys.c
@@ -286,7 +286,7 @@
if (!addr)
return -ENOMEM;
- addr->u.a6.addr.s6_addr32[0] = 0xfe800000;
+ addr->u.a6.addr.s6_addr32[0] = cpu_to_be32(0xfe800000);
addr->u.a6.addr.s6_addr32[1] = 0x00000000;
for (i = 8; i < 16; i++)
addr->u.a6.addr.s6_addr[i] =
@@ -320,7 +320,7 @@
addr = qeth_l3_get_addr_buffer(QETH_PROT_IPV6);
if (addr != NULL) {
- addr->u.a6.addr.s6_addr32[0] = 0xfe800000;
+ addr->u.a6.addr.s6_addr32[0] = cpu_to_be32(0xfe800000);
addr->u.a6.addr.s6_addr32[1] = 0x00000000;
for (i = 8; i < 16; i++)
addr->u.a6.addr.s6_addr[i] = card->options.hsuid[i - 8];
diff --git a/drivers/sbus/char/jsflash.c b/drivers/sbus/char/jsflash.c
index 6ff61da..62fed9d 100644
--- a/drivers/sbus/char/jsflash.c
+++ b/drivers/sbus/char/jsflash.c
@@ -183,11 +183,33 @@
}
}
-static void jsfd_do_request(struct request_queue *q)
+static int jsfd_queue;
+
+static struct request *jsfd_next_request(void)
+{
+ struct request_queue *q;
+ struct request *rq;
+ int old_pos = jsfd_queue;
+
+ do {
+ q = jsfd_disk[jsfd_queue]->queue;
+ if (++jsfd_queue == JSF_MAX)
+ jsfd_queue = 0;
+ if (q) {
+ rq = blk_fetch_request(q);
+ if (rq)
+ return rq;
+ }
+ } while (jsfd_queue != old_pos);
+
+ return NULL;
+}
+
+static void jsfd_request(void)
{
struct request *req;
- req = blk_fetch_request(q);
+ req = jsfd_next_request();
while (req) {
struct jsfd_part *jdp = req->rq_disk->private_data;
unsigned long offset = blk_rq_pos(req) << 9;
@@ -211,10 +233,15 @@
err = 0;
end:
if (!__blk_end_request_cur(req, err))
- req = blk_fetch_request(q);
+ req = jsfd_next_request();
}
}
+static void jsfd_do_request(struct request_queue *q)
+{
+ jsfd_request();
+}
+
/*
* The memory devices use the full 32/64 bits of the offset, and so we cannot
* check against negative addresses: they are ok. The return value is weird,
@@ -544,8 +571,6 @@
return 0;
}
-static struct request_queue *jsf_queue;
-
static int jsfd_init(void)
{
static DEFINE_SPINLOCK(lock);
@@ -562,6 +587,11 @@
struct gendisk *disk = alloc_disk(1);
if (!disk)
goto out;
+ disk->queue = blk_init_queue(jsfd_do_request, &lock);
+ if (!disk->queue) {
+ put_disk(disk);
+ goto out;
+ }
jsfd_disk[i] = disk;
}
@@ -570,13 +600,6 @@
goto out;
}
- jsf_queue = blk_init_queue(jsfd_do_request, &lock);
- if (!jsf_queue) {
- err = -ENOMEM;
- unregister_blkdev(JSFD_MAJOR, "jsfd");
- goto out;
- }
-
for (i = 0; i < JSF_MAX; i++) {
struct gendisk *disk = jsfd_disk[i];
if ((i & JSF_PART_MASK) >= JSF_NPART) continue;
@@ -589,7 +612,6 @@
disk->fops = &jsfd_fops;
set_capacity(disk, jdp->dsize >> 9);
disk->private_data = jdp;
- disk->queue = jsf_queue;
add_disk(disk);
set_disk_ro(disk, 1);
}
@@ -619,6 +641,7 @@
for (i = 0; i < JSF_MAX; i++) {
if ((i & JSF_PART_MASK) >= JSF_NPART) continue;
del_gendisk(jsfd_disk[i]);
+ blk_cleanup_queue(jsfd_disk[i]->queue);
put_disk(jsfd_disk[i]);
}
if (jsf0.busy)
@@ -628,7 +651,6 @@
misc_deregister(&jsf_dev);
unregister_blkdev(JSFD_MAJOR, "jsfd");
- blk_cleanup_queue(jsf_queue);
}
module_init(jsflash_init_module);
diff --git a/drivers/scsi/Makefile b/drivers/scsi/Makefile
index fc28555..93dbe58 100644
--- a/drivers/scsi/Makefile
+++ b/drivers/scsi/Makefile
@@ -166,6 +166,7 @@
scsi_mod-$(CONFIG_SCSI_NETLINK) += scsi_netlink.o
scsi_mod-$(CONFIG_SYSCTL) += scsi_sysctl.o
scsi_mod-$(CONFIG_SCSI_PROC_FS) += scsi_proc.o
+scsi_mod-$(CONFIG_BLK_DEBUG_FS) += scsi_debugfs.o
scsi_mod-y += scsi_trace.o scsi_logging.o
scsi_mod-$(CONFIG_PM) += scsi_pm.o
scsi_mod-$(CONFIG_SCSI_DH) += scsi_dh.o
diff --git a/drivers/scsi/esas2r/esas2r_ioctl.c b/drivers/scsi/esas2r/esas2r_ioctl.c
index b35ed38..2d4b7f0 100644
--- a/drivers/scsi/esas2r/esas2r_ioctl.c
+++ b/drivers/scsi/esas2r/esas2r_ioctl.c
@@ -1289,32 +1289,13 @@
|| (cmd > EXPRESS_IOCTL_MAX))
return -ENOTSUPP;
- if (!access_ok(VERIFY_WRITE, arg, sizeof(struct atto_express_ioctl))) {
+ ioctl = memdup_user(arg, sizeof(struct atto_express_ioctl));
+ if (IS_ERR(ioctl)) {
esas2r_log(ESAS2R_LOG_WARN,
"ioctl_handler access_ok failed for cmd %d, "
"address %p", cmd,
arg);
- return -EFAULT;
- }
-
- /* allocate a kernel memory buffer for the IOCTL data */
- ioctl = kzalloc(sizeof(struct atto_express_ioctl), GFP_KERNEL);
- if (ioctl == NULL) {
- esas2r_log(ESAS2R_LOG_WARN,
- "ioctl_handler kzalloc failed for %zu bytes",
- sizeof(struct atto_express_ioctl));
- return -ENOMEM;
- }
-
- err = __copy_from_user(ioctl, arg, sizeof(struct atto_express_ioctl));
- if (err != 0) {
- esas2r_log(ESAS2R_LOG_WARN,
- "copy_from_user didn't copy everything (err %d, cmd %d)",
- err,
- cmd);
- kfree(ioctl);
-
- return -EFAULT;
+ return PTR_ERR(ioctl);
}
/* verify the signature */
diff --git a/drivers/scsi/iscsi_tcp.c b/drivers/scsi/iscsi_tcp.c
index 4228aba..bbea8ea 100644
--- a/drivers/scsi/iscsi_tcp.c
+++ b/drivers/scsi/iscsi_tcp.c
@@ -387,7 +387,7 @@
rc = 0;
}
- tsk_restore_flags(current, pflags, PF_MEMALLOC);
+ current_restore_flags(pflags, PF_MEMALLOC);
return rc;
}
diff --git a/drivers/scsi/lpfc/lpfc.h b/drivers/scsi/lpfc/lpfc.h
index 257bbdd..6d7840b 100644
--- a/drivers/scsi/lpfc/lpfc.h
+++ b/drivers/scsi/lpfc/lpfc.h
@@ -56,7 +56,7 @@
#define LPFC_MAX_SG_SEG_CNT 4096 /* sg element count per scsi cmnd */
#define LPFC_MAX_SGL_SEG_CNT 512 /* SGL element count per scsi cmnd */
#define LPFC_MAX_BPL_SEG_CNT 4096 /* BPL element count per scsi cmnd */
-#define LPFC_MIN_NVME_SEG_CNT 254
+#define LPFC_MAX_NVME_SEG_CNT 128 /* max SGL element cnt per NVME cmnd */
#define LPFC_MAX_SGE_SIZE 0x80000000 /* Maximum data allowed in a SGE */
#define LPFC_IOCB_LIST_CNT 2250 /* list of IOCBs for fast-path usage. */
@@ -474,6 +474,8 @@
unsigned long rcv_buffer_time_stamp;
uint32_t vport_flag;
#define STATIC_VPORT 1
+#define FAWWPN_SET 2
+#define FAWWPN_PARAM_CHG 4
uint16_t fdmi_num_disc;
uint32_t fdmi_hba_mask;
@@ -781,6 +783,7 @@
uint32_t cfg_nvmet_fb_size;
uint32_t cfg_total_seg_cnt;
uint32_t cfg_sg_seg_cnt;
+ uint32_t cfg_nvme_seg_cnt;
uint32_t cfg_sg_dma_buf_size;
uint64_t cfg_soft_wwnn;
uint64_t cfg_soft_wwpn;
diff --git a/drivers/scsi/lpfc/lpfc_attr.c b/drivers/scsi/lpfc/lpfc_attr.c
index 22819af..513fd07 100644
--- a/drivers/scsi/lpfc/lpfc_attr.c
+++ b/drivers/scsi/lpfc/lpfc_attr.c
@@ -2292,6 +2292,8 @@
struct lpfc_vport *vport = (struct lpfc_vport *) shost->hostdata;
struct lpfc_hba *phba = vport->phba;
unsigned int cnt = count;
+ uint8_t vvvl = vport->fc_sparam.cmn.valid_vendor_ver_level;
+ u32 *fawwpn_key = (uint32_t *)&vport->fc_sparam.un.vendorVersion[0];
/*
* We're doing a simple sanity check for soft_wwpn setting.
@@ -2305,6 +2307,12 @@
* here. The intent is to protect against the random user or
* application that is just writing attributes.
*/
+ if (vvvl == 1 && cpu_to_be32(*fawwpn_key) == FAPWWN_KEY_VENDOR) {
+ lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
+ "0051 "LPFC_DRIVER_NAME" soft wwpn can not"
+ " be enabled: fawwpn is enabled\n");
+ return -EINVAL;
+ }
/* count may include a LF at end of string */
if (buf[cnt-1] == '\n')
@@ -3335,7 +3343,7 @@
* percentage will go to NVME.
*/
LPFC_ATTR_R(xri_split, 50, 10, 90,
- "Division of XRI resources between SCSI and NVME");
+ "Division of XRI resources between SCSI and NVME");
/*
# lpfc_log_verbose: Only turn this flag on if you are willing to risk being
diff --git a/drivers/scsi/lpfc/lpfc_bsg.c b/drivers/scsi/lpfc/lpfc_bsg.c
index 18157d2..a1686c2 100644
--- a/drivers/scsi/lpfc/lpfc_bsg.c
+++ b/drivers/scsi/lpfc/lpfc_bsg.c
@@ -2486,6 +2486,10 @@
mbox, *rpi);
else {
*rpi = lpfc_sli4_alloc_rpi(phba);
+ if (*rpi == LPFC_RPI_ALLOC_ERROR) {
+ mempool_free(mbox, phba->mbox_mem_pool);
+ return -EBUSY;
+ }
status = lpfc_reg_rpi(phba, phba->pport->vpi,
phba->pport->fc_myDID,
(uint8_t *)&phba->pport->fc_sparam,
diff --git a/drivers/scsi/lpfc/lpfc_crtn.h b/drivers/scsi/lpfc/lpfc_crtn.h
index 54e6ac4..944b32c 100644
--- a/drivers/scsi/lpfc/lpfc_crtn.h
+++ b/drivers/scsi/lpfc/lpfc_crtn.h
@@ -24,6 +24,7 @@
struct fc_rport;
struct fc_frame_header;
+struct lpfc_nvmet_rcv_ctx;
void lpfc_down_link(struct lpfc_hba *, LPFC_MBOXQ_t *);
void lpfc_sli_read_link_ste(struct lpfc_hba *);
void lpfc_dump_mem(struct lpfc_hba *, LPFC_MBOXQ_t *, uint16_t, uint16_t);
@@ -99,7 +100,7 @@
int lpfc_check_sli_ndlp(struct lpfc_hba *, struct lpfc_sli_ring *,
struct lpfc_iocbq *, struct lpfc_nodelist *);
-void lpfc_nlp_init(struct lpfc_vport *, struct lpfc_nodelist *, uint32_t);
+struct lpfc_nodelist *lpfc_nlp_init(struct lpfc_vport *vport, uint32_t did);
struct lpfc_nodelist *lpfc_nlp_get(struct lpfc_nodelist *);
int lpfc_nlp_put(struct lpfc_nodelist *);
int lpfc_nlp_not_used(struct lpfc_nodelist *ndlp);
@@ -245,6 +246,10 @@
void lpfc_sli4_rb_free(struct lpfc_hba *, struct hbq_dmabuf *);
struct rqb_dmabuf *lpfc_sli4_nvmet_alloc(struct lpfc_hba *phba);
void lpfc_sli4_nvmet_free(struct lpfc_hba *phba, struct rqb_dmabuf *dmab);
+void lpfc_nvmet_rq_post(struct lpfc_hba *phba, struct lpfc_nvmet_rcv_ctx *ctxp,
+ struct lpfc_dmabuf *mp);
+int lpfc_nvmet_rcv_unsol_abort(struct lpfc_vport *vport,
+ struct fc_frame_header *fc_hdr);
void lpfc_sli4_build_dflt_fcf_record(struct lpfc_hba *, struct fcf_record *,
uint16_t);
int lpfc_sli4_rq_put(struct lpfc_queue *hq, struct lpfc_queue *dq,
@@ -302,6 +307,8 @@
void lpfc_sli_handle_slow_ring_event(struct lpfc_hba *,
struct lpfc_sli_ring *, uint32_t);
void lpfc_sli4_handle_received_buffer(struct lpfc_hba *, struct hbq_dmabuf *);
+void lpfc_sli4_seq_abort_rsp(struct lpfc_vport *vport,
+ struct fc_frame_header *fc_hdr, bool aborted);
void lpfc_sli_def_mbox_cmpl(struct lpfc_hba *, LPFC_MBOXQ_t *);
void lpfc_sli4_unreg_rpi_cmpl_clr(struct lpfc_hba *, LPFC_MBOXQ_t *);
int lpfc_sli_issue_iocb(struct lpfc_hba *, uint32_t,
diff --git a/drivers/scsi/lpfc/lpfc_ct.c b/drivers/scsi/lpfc/lpfc_ct.c
index d3e9af9..1487406 100644
--- a/drivers/scsi/lpfc/lpfc_ct.c
+++ b/drivers/scsi/lpfc/lpfc_ct.c
@@ -537,19 +537,53 @@
}
}
+static void
+lpfc_ns_rsp_audit_did(struct lpfc_vport *vport, uint32_t Did, uint8_t fc4_type)
+{
+ struct lpfc_hba *phba = vport->phba;
+ struct lpfc_nodelist *ndlp = NULL;
+ struct Scsi_Host *shost = lpfc_shost_from_vport(vport);
+
+ /*
+ * To conserve rpi's, filter out addresses for other
+ * vports on the same physical HBAs.
+ */
+ if (Did != vport->fc_myDID &&
+ (!lpfc_find_vport_by_did(phba, Did) ||
+ vport->cfg_peer_port_login)) {
+ if (!phba->nvmet_support) {
+ /* FCPI/NVMEI path. Process Did */
+ lpfc_prep_node_fc4type(vport, Did, fc4_type);
+ return;
+ }
+ /* NVMET path. NVMET only cares about NVMEI nodes. */
+ list_for_each_entry(ndlp, &vport->fc_nodes, nlp_listp) {
+ if (ndlp->nlp_type != NLP_NVME_INITIATOR ||
+ ndlp->nlp_state != NLP_STE_UNMAPPED_NODE)
+ continue;
+ spin_lock_irq(shost->host_lock);
+ if (ndlp->nlp_DID == Did)
+ ndlp->nlp_flag &= ~NLP_NVMET_RECOV;
+ else
+ ndlp->nlp_flag |= NLP_NVMET_RECOV;
+ spin_unlock_irq(shost->host_lock);
+ }
+ }
+}
+
static int
lpfc_ns_rsp(struct lpfc_vport *vport, struct lpfc_dmabuf *mp, uint8_t fc4_type,
uint32_t Size)
{
- struct lpfc_hba *phba = vport->phba;
struct lpfc_sli_ct_request *Response =
(struct lpfc_sli_ct_request *) mp->virt;
- struct lpfc_nodelist *ndlp = NULL;
struct lpfc_dmabuf *mlast, *next_mp;
uint32_t *ctptr = (uint32_t *) & Response->un.gid.PortType;
uint32_t Did, CTentry;
int Cnt;
struct list_head head;
+ struct Scsi_Host *shost = lpfc_shost_from_vport(vport);
+ struct lpfc_nodelist *ndlp = NULL;
lpfc_set_disctmo(vport);
vport->num_disc_nodes = 0;
@@ -574,19 +608,7 @@
/* Get next DID from NameServer List */
CTentry = *ctptr++;
Did = ((be32_to_cpu(CTentry)) & Mask_DID);
-
- ndlp = NULL;
-
- /*
- * Check for rscn processing or not
- * To conserve rpi's, filter out addresses for other
- * vports on the same physical HBAs.
- */
- if ((Did != vport->fc_myDID) &&
- ((lpfc_find_vport_by_did(phba, Did) == NULL) ||
- vport->cfg_peer_port_login))
- lpfc_prep_node_fc4type(vport, Did, fc4_type);
-
+ lpfc_ns_rsp_audit_did(vport, Did, fc4_type);
if (CTentry & (cpu_to_be32(SLI_CT_LAST_ENTRY)))
goto nsout1;
@@ -596,6 +618,22 @@
}
+ /* All GID_FT entries processed. If the driver is running in
+ * in target mode, put impacted nodes into recovery and drop
+ * the RPI to flush outstanding IO.
+ */
+ if (vport->phba->nvmet_support) {
+ list_for_each_entry(ndlp, &vport->fc_nodes, nlp_listp) {
+ if (!(ndlp->nlp_flag & NLP_NVMET_RECOV))
+ continue;
+ lpfc_disc_state_machine(vport, ndlp, NULL,
+ NLP_EVT_DEVICE_RECOVERY);
+ spin_lock_irq(shost->host_lock);
+ ndlp->nlp_flag &= ~NLP_NVMET_RECOV;
+ spin_lock_irq(shost->host_lock);
+ }
+ }
+
nsout1:
list_del(&head);
return 0;
diff --git a/drivers/scsi/lpfc/lpfc_debugfs.c b/drivers/scsi/lpfc/lpfc_debugfs.c
index 913eed82..fce549a 100644
--- a/drivers/scsi/lpfc/lpfc_debugfs.c
+++ b/drivers/scsi/lpfc/lpfc_debugfs.c
@@ -745,73 +745,102 @@
{
struct lpfc_hba *phba = vport->phba;
struct lpfc_nvmet_tgtport *tgtp;
+ struct lpfc_nvmet_rcv_ctx *ctxp, *next_ctxp;
int len = 0;
+ int cnt;
if (phba->nvmet_support) {
if (!phba->targetport)
return len;
tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
- len += snprintf(buf+len, size-len,
+ len += snprintf(buf + len, size - len,
"\nNVME Targetport Statistics\n");
- len += snprintf(buf+len, size-len,
+ len += snprintf(buf + len, size - len,
"LS: Rcv %08x Drop %08x Abort %08x\n",
atomic_read(&tgtp->rcv_ls_req_in),
atomic_read(&tgtp->rcv_ls_req_drop),
atomic_read(&tgtp->xmt_ls_abort));
if (atomic_read(&tgtp->rcv_ls_req_in) !=
atomic_read(&tgtp->rcv_ls_req_out)) {
- len += snprintf(buf+len, size-len,
+ len += snprintf(buf + len, size - len,
"Rcv LS: in %08x != out %08x\n",
atomic_read(&tgtp->rcv_ls_req_in),
atomic_read(&tgtp->rcv_ls_req_out));
}
- len += snprintf(buf+len, size-len,
+ len += snprintf(buf + len, size - len,
"LS: Xmt %08x Drop %08x Cmpl %08x Err %08x\n",
atomic_read(&tgtp->xmt_ls_rsp),
atomic_read(&tgtp->xmt_ls_drop),
atomic_read(&tgtp->xmt_ls_rsp_cmpl),
atomic_read(&tgtp->xmt_ls_rsp_error));
- len += snprintf(buf+len, size-len,
+ len += snprintf(buf + len, size - len,
"FCP: Rcv %08x Drop %08x\n",
atomic_read(&tgtp->rcv_fcp_cmd_in),
atomic_read(&tgtp->rcv_fcp_cmd_drop));
if (atomic_read(&tgtp->rcv_fcp_cmd_in) !=
atomic_read(&tgtp->rcv_fcp_cmd_out)) {
- len += snprintf(buf+len, size-len,
+ len += snprintf(buf + len, size - len,
"Rcv FCP: in %08x != out %08x\n",
atomic_read(&tgtp->rcv_fcp_cmd_in),
atomic_read(&tgtp->rcv_fcp_cmd_out));
}
- len += snprintf(buf+len, size-len,
- "FCP Rsp: read %08x readrsp %08x write %08x rsp %08x\n",
+ len += snprintf(buf + len, size - len,
+ "FCP Rsp: read %08x readrsp %08x "
+ "write %08x rsp %08x\n",
atomic_read(&tgtp->xmt_fcp_read),
atomic_read(&tgtp->xmt_fcp_read_rsp),
atomic_read(&tgtp->xmt_fcp_write),
atomic_read(&tgtp->xmt_fcp_rsp));
- len += snprintf(buf+len, size-len,
+ len += snprintf(buf + len, size - len,
"FCP Rsp: abort %08x drop %08x\n",
atomic_read(&tgtp->xmt_fcp_abort),
atomic_read(&tgtp->xmt_fcp_drop));
- len += snprintf(buf+len, size-len,
+ len += snprintf(buf + len, size - len,
"FCP Rsp Cmpl: %08x err %08x drop %08x\n",
atomic_read(&tgtp->xmt_fcp_rsp_cmpl),
atomic_read(&tgtp->xmt_fcp_rsp_error),
atomic_read(&tgtp->xmt_fcp_rsp_drop));
- len += snprintf(buf+len, size-len,
+ len += snprintf(buf + len, size - len,
"ABORT: Xmt %08x Err %08x Cmpl %08x",
atomic_read(&tgtp->xmt_abort_rsp),
atomic_read(&tgtp->xmt_abort_rsp_error),
atomic_read(&tgtp->xmt_abort_cmpl));
- len += snprintf(buf+len, size-len, "\n");
+ len += snprintf(buf + len, size - len, "\n");
+
+ cnt = 0;
+ spin_lock(&phba->sli4_hba.abts_nvme_buf_list_lock);
+ list_for_each_entry_safe(ctxp, next_ctxp,
+ &phba->sli4_hba.lpfc_abts_nvmet_ctx_list,
+ list) {
+ cnt++;
+ }
+ spin_unlock(&phba->sli4_hba.abts_nvme_buf_list_lock);
+ if (cnt) {
+ len += snprintf(buf + len, size - len,
+ "ABORT: %d ctx entries\n", cnt);
+ spin_lock(&phba->sli4_hba.abts_nvme_buf_list_lock);
+ list_for_each_entry_safe(ctxp, next_ctxp,
+ &phba->sli4_hba.lpfc_abts_nvmet_ctx_list,
+ list) {
+ if (len >= (size - LPFC_DEBUG_OUT_LINE_SZ))
+ break;
+ len += snprintf(buf + len, size - len,
+ "Entry: oxid %x state %x "
+ "flag %x\n",
+ ctxp->oxid, ctxp->state,
+ ctxp->flag);
+ }
+ spin_unlock(&phba->sli4_hba.abts_nvme_buf_list_lock);
+ }
} else {
if (!(phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME))
return len;
@@ -3128,8 +3157,6 @@
datqp->queue_id, datqp->entry_count,
datqp->entry_size, datqp->host_index,
datqp->hba_index);
- len += snprintf(pbuffer + len, LPFC_QUE_INFO_GET_BUF_SIZE - len, "\n");
-
return len;
}
@@ -5700,10 +5727,8 @@
#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
struct lpfc_hba *phba = vport->phba;
- if (vport->disc_trc) {
- kfree(vport->disc_trc);
- vport->disc_trc = NULL;
- }
+ kfree(vport->disc_trc);
+ vport->disc_trc = NULL;
debugfs_remove(vport->debug_disc_trc); /* discovery_trace */
vport->debug_disc_trc = NULL;
@@ -5770,10 +5795,8 @@
debugfs_remove(phba->debug_readRef); /* readRef */
phba->debug_readRef = NULL;
- if (phba->slow_ring_trc) {
- kfree(phba->slow_ring_trc);
- phba->slow_ring_trc = NULL;
- }
+ kfree(phba->slow_ring_trc);
+ phba->slow_ring_trc = NULL;
/* slow_ring_trace */
debugfs_remove(phba->debug_slow_ring_trc);
diff --git a/drivers/scsi/lpfc/lpfc_disc.h b/drivers/scsi/lpfc/lpfc_disc.h
index f4ff99d9..9d5a379 100644
--- a/drivers/scsi/lpfc/lpfc_disc.h
+++ b/drivers/scsi/lpfc/lpfc_disc.h
@@ -157,6 +157,7 @@
#define NLP_LOGO_SND 0x00000100 /* sent LOGO request for this entry */
#define NLP_RNID_SND 0x00000400 /* sent RNID request for this entry */
#define NLP_ELS_SND_MASK 0x000007e0 /* sent ELS request for this entry */
+#define NLP_NVMET_RECOV 0x00001000 /* NVMET auditing node for recovery. */
#define NLP_DEFER_RM 0x00010000 /* Remove this ndlp if no longer used */
#define NLP_DELAY_TMO 0x00020000 /* delay timeout is running for node */
#define NLP_NPR_2B_DISC 0x00040000 /* node is included in num_disc_nodes */
diff --git a/drivers/scsi/lpfc/lpfc_els.c b/drivers/scsi/lpfc/lpfc_els.c
index a5ca37e..67827e3 100644
--- a/drivers/scsi/lpfc/lpfc_els.c
+++ b/drivers/scsi/lpfc/lpfc_els.c
@@ -603,9 +603,11 @@
memcmp(&vport->fabric_portname, &sp->portName,
sizeof(struct lpfc_name)) ||
memcmp(&vport->fabric_nodename, &sp->nodeName,
- sizeof(struct lpfc_name)))
+ sizeof(struct lpfc_name)) ||
+ (vport->vport_flag & FAWWPN_PARAM_CHG)) {
fabric_param_changed = 1;
-
+ vport->vport_flag &= ~FAWWPN_PARAM_CHG;
+ }
/*
* Word 1 Bit 31 in common service parameter is overloaded.
* Word 1 Bit 31 in FLOGI request is multiple NPort request
@@ -895,10 +897,9 @@
* Cannot find existing Fabric ndlp, so allocate a
* new one
*/
- ndlp = mempool_alloc(phba->nlp_mem_pool, GFP_KERNEL);
+ ndlp = lpfc_nlp_init(vport, PT2PT_RemoteID);
if (!ndlp)
goto fail;
- lpfc_nlp_init(vport, ndlp, PT2PT_RemoteID);
} else if (!NLP_CHK_NODE_ACT(ndlp)) {
ndlp = lpfc_enable_node(vport, ndlp,
NLP_STE_UNUSED_NODE);
@@ -1364,7 +1365,6 @@
int
lpfc_initial_flogi(struct lpfc_vport *vport)
{
- struct lpfc_hba *phba = vport->phba;
struct lpfc_nodelist *ndlp;
vport->port_state = LPFC_FLOGI;
@@ -1374,10 +1374,9 @@
ndlp = lpfc_findnode_did(vport, Fabric_DID);
if (!ndlp) {
/* Cannot find existing Fabric ndlp, so allocate a new one */
- ndlp = mempool_alloc(phba->nlp_mem_pool, GFP_KERNEL);
+ ndlp = lpfc_nlp_init(vport, Fabric_DID);
if (!ndlp)
return 0;
- lpfc_nlp_init(vport, ndlp, Fabric_DID);
/* Set the node type */
ndlp->nlp_type |= NLP_FABRIC;
/* Put ndlp onto node list */
@@ -1418,17 +1417,15 @@
int
lpfc_initial_fdisc(struct lpfc_vport *vport)
{
- struct lpfc_hba *phba = vport->phba;
struct lpfc_nodelist *ndlp;
/* First look for the Fabric ndlp */
ndlp = lpfc_findnode_did(vport, Fabric_DID);
if (!ndlp) {
/* Cannot find existing Fabric ndlp, so allocate a new one */
- ndlp = mempool_alloc(phba->nlp_mem_pool, GFP_KERNEL);
+ ndlp = lpfc_nlp_init(vport, Fabric_DID);
if (!ndlp)
return 0;
- lpfc_nlp_init(vport, ndlp, Fabric_DID);
/* Put ndlp onto node list */
lpfc_enqueue_node(vport, ndlp);
} else if (!NLP_CHK_NODE_ACT(ndlp)) {
@@ -1564,14 +1561,13 @@
phba->active_rrq_pool);
return ndlp;
}
- new_ndlp = mempool_alloc(phba->nlp_mem_pool, GFP_ATOMIC);
+ new_ndlp = lpfc_nlp_init(vport, ndlp->nlp_DID);
if (!new_ndlp) {
if (active_rrqs_xri_bitmap)
mempool_free(active_rrqs_xri_bitmap,
phba->active_rrq_pool);
return ndlp;
}
- lpfc_nlp_init(vport, new_ndlp, ndlp->nlp_DID);
} else if (!NLP_CHK_NODE_ACT(new_ndlp)) {
rc = memcmp(&ndlp->nlp_portname, name,
sizeof(struct lpfc_name));
@@ -2845,10 +2841,9 @@
ndlp = lpfc_findnode_did(vport, nportid);
if (!ndlp) {
- ndlp = mempool_alloc(phba->nlp_mem_pool, GFP_KERNEL);
+ ndlp = lpfc_nlp_init(vport, nportid);
if (!ndlp)
return 1;
- lpfc_nlp_init(vport, ndlp, nportid);
lpfc_enqueue_node(vport, ndlp);
} else if (!NLP_CHK_NODE_ACT(ndlp)) {
ndlp = lpfc_enable_node(vport, ndlp, NLP_STE_UNUSED_NODE);
@@ -2938,10 +2933,9 @@
ndlp = lpfc_findnode_did(vport, nportid);
if (!ndlp) {
- ndlp = mempool_alloc(phba->nlp_mem_pool, GFP_KERNEL);
+ ndlp = lpfc_nlp_init(vport, nportid);
if (!ndlp)
return 1;
- lpfc_nlp_init(vport, ndlp, nportid);
lpfc_enqueue_node(vport, ndlp);
} else if (!NLP_CHK_NODE_ACT(ndlp)) {
ndlp = lpfc_enable_node(vport, ndlp, NLP_STE_UNUSED_NODE);
@@ -4403,7 +4397,7 @@
pcmd = (uint8_t *) (((struct lpfc_dmabuf *) elsiocb->context2)->virt);
memset(pcmd, 0, cmdsize);
- *((uint32_t *) (pcmd)) = (ELS_CMD_ACC | (ELS_CMD_PRLI & ~ELS_RSP_MASK));
+ *((uint32_t *)(pcmd)) = elsrspcmd;
pcmd += sizeof(uint32_t);
/* For PRLI, remainder of payload is PRLI parameter page */
@@ -5867,8 +5861,11 @@
(ndlp->nlp_state == NLP_STE_UNUSED_NODE) ||
!lpfc_rscn_payload_check(vport, ndlp->nlp_DID))
continue;
+
+ /* NVME Target mode does not do RSCN Recovery. */
if (vport->phba->nvmet_support)
continue;
+
lpfc_disc_state_machine(vport, ndlp, NULL,
NLP_EVT_DEVICE_RECOVERY);
lpfc_cancel_retry_delay_tmo(vport, ndlp);
@@ -6133,7 +6130,6 @@
lpfc_els_handle_rscn(struct lpfc_vport *vport)
{
struct lpfc_nodelist *ndlp;
- struct lpfc_hba *phba = vport->phba;
/* Ignore RSCN if the port is being torn down. */
if (vport->load_flag & FC_UNLOADING) {
@@ -6157,22 +6153,16 @@
ndlp = lpfc_findnode_did(vport, NameServer_DID);
if (ndlp && NLP_CHK_NODE_ACT(ndlp)
&& ndlp->nlp_state == NLP_STE_UNMAPPED_NODE) {
- /* Good ndlp, issue CT Request to NameServer */
+ /* Good ndlp, issue CT Request to NameServer. Need to
+ * know how many gidfts were issued. If none, then just
+ * flush the RSCN. Otherwise, the outstanding requests
+ * need to complete.
+ */
vport->gidft_inp = 0;
- if (lpfc_issue_gidft(vport) == 0)
- /* Wait for NameServer query cmpl before we can
- * continue
- */
+ if (lpfc_issue_gidft(vport) > 0)
return 1;
} else {
- /* If login to NameServer does not exist, issue one */
- /* Good status, issue PLOGI to NameServer */
- ndlp = lpfc_findnode_did(vport, NameServer_DID);
- if (ndlp && NLP_CHK_NODE_ACT(ndlp))
- /* Wait for NameServer login cmpl before we can
- continue */
- return 1;
-
+ /* Nameserver login in question. Revalidate. */
if (ndlp) {
ndlp = lpfc_enable_node(vport, ndlp,
NLP_STE_PLOGI_ISSUE);
@@ -6182,12 +6172,11 @@
}
ndlp->nlp_prev_state = NLP_STE_UNUSED_NODE;
} else {
- ndlp = mempool_alloc(phba->nlp_mem_pool, GFP_KERNEL);
+ ndlp = lpfc_nlp_init(vport, NameServer_DID);
if (!ndlp) {
lpfc_els_flush_rscn(vport);
return 0;
}
- lpfc_nlp_init(vport, ndlp, NameServer_DID);
ndlp->nlp_prev_state = ndlp->nlp_state;
lpfc_nlp_set_state(vport, ndlp, NLP_STE_PLOGI_ISSUE);
}
@@ -7746,11 +7735,9 @@
ndlp = lpfc_findnode_did(vport, did);
if (!ndlp) {
/* Cannot find existing Fabric ndlp, so allocate a new one */
- ndlp = mempool_alloc(phba->nlp_mem_pool, GFP_KERNEL);
+ ndlp = lpfc_nlp_init(vport, did);
if (!ndlp)
goto dropit;
-
- lpfc_nlp_init(vport, ndlp, did);
lpfc_nlp_set_state(vport, ndlp, NLP_STE_NPR_NODE);
newnode = 1;
if ((did & Fabric_DID_MASK) == Fabric_DID_MASK)
@@ -8193,7 +8180,6 @@
static void
lpfc_start_fdmi(struct lpfc_vport *vport)
{
- struct lpfc_hba *phba = vport->phba;
struct lpfc_nodelist *ndlp;
/* If this is the first time, allocate an ndlp and initialize
@@ -8202,9 +8188,8 @@
*/
ndlp = lpfc_findnode_did(vport, FDMI_DID);
if (!ndlp) {
- ndlp = mempool_alloc(phba->nlp_mem_pool, GFP_KERNEL);
+ ndlp = lpfc_nlp_init(vport, FDMI_DID);
if (ndlp) {
- lpfc_nlp_init(vport, ndlp, FDMI_DID);
ndlp->nlp_type |= NLP_FABRIC;
} else {
return;
@@ -8257,7 +8242,7 @@
ndlp = lpfc_findnode_did(vport, NameServer_DID);
if (!ndlp) {
- ndlp = mempool_alloc(phba->nlp_mem_pool, GFP_KERNEL);
+ ndlp = lpfc_nlp_init(vport, NameServer_DID);
if (!ndlp) {
if (phba->fc_topology == LPFC_TOPOLOGY_LOOP) {
lpfc_disc_start(vport);
@@ -8268,7 +8253,6 @@
"0251 NameServer login: no memory\n");
return;
}
- lpfc_nlp_init(vport, ndlp, NameServer_DID);
} else if (!NLP_CHK_NODE_ACT(ndlp)) {
ndlp = lpfc_enable_node(vport, ndlp, NLP_STE_UNUSED_NODE);
if (!ndlp) {
@@ -8771,7 +8755,7 @@
pcmd += sizeof(uint32_t); /* Node Name */
pcmd += sizeof(uint32_t); /* Node Name */
memcpy(pcmd, &vport->fc_nodename, 8);
-
+ memset(sp->un.vendorVersion, 0, sizeof(sp->un.vendorVersion));
lpfc_set_disctmo(vport);
phba->fc_stat.elsXmitFDISC++;
diff --git a/drivers/scsi/lpfc/lpfc_hbadisc.c b/drivers/scsi/lpfc/lpfc_hbadisc.c
index 180b072b..0482c55 100644
--- a/drivers/scsi/lpfc/lpfc_hbadisc.c
+++ b/drivers/scsi/lpfc/lpfc_hbadisc.c
@@ -3002,6 +3002,7 @@
MAILBOX_t *mb = &pmb->u.mb;
struct lpfc_dmabuf *mp = (struct lpfc_dmabuf *) pmb->context1;
struct lpfc_vport *vport = pmb->vport;
+ struct Scsi_Host *shost = lpfc_shost_from_vport(vport);
struct serv_parm *sp = &vport->fc_sparam;
uint32_t ed_tov;
@@ -3031,6 +3032,7 @@
}
lpfc_update_vport_wwn(vport);
+ fc_host_port_name(shost) = wwn_to_u64(vport->fc_portname.u.wwn);
if (vport->port_type == LPFC_PHYSICAL_PORT) {
memcpy(&phba->wwnn, &vport->fc_nodename, sizeof(phba->wwnn));
memcpy(&phba->wwpn, &vport->fc_portname, sizeof(phba->wwnn));
@@ -3309,6 +3311,7 @@
struct lpfc_sli_ring *pring;
MAILBOX_t *mb = &pmb->u.mb;
struct lpfc_dmabuf *mp = (struct lpfc_dmabuf *) (pmb->context1);
+ uint8_t attn_type;
/* Unblock ELS traffic */
pring = lpfc_phba_elsring(phba);
@@ -3325,6 +3328,7 @@
}
la = (struct lpfc_mbx_read_top *) &pmb->u.mb.un.varReadTop;
+ attn_type = bf_get(lpfc_mbx_read_top_att_type, la);
memcpy(&phba->alpa_map[0], mp->virt, 128);
@@ -3337,7 +3341,7 @@
if (phba->fc_eventTag <= la->eventTag) {
phba->fc_stat.LinkMultiEvent++;
- if (bf_get(lpfc_mbx_read_top_att_type, la) == LPFC_ATT_LINK_UP)
+ if (attn_type == LPFC_ATT_LINK_UP)
if (phba->fc_eventTag != 0)
lpfc_linkdown(phba);
}
@@ -3353,7 +3357,7 @@
}
phba->link_events++;
- if ((bf_get(lpfc_mbx_read_top_att_type, la) == LPFC_ATT_LINK_UP) &&
+ if ((attn_type == LPFC_ATT_LINK_UP) &&
!(phba->sli.sli_flag & LPFC_MENLO_MAINT)) {
phba->fc_stat.LinkUp++;
if (phba->link_flag & LS_LOOPBACK_MODE) {
@@ -3379,8 +3383,8 @@
phba->wait_4_mlo_maint_flg);
}
lpfc_mbx_process_link_up(phba, la);
- } else if (bf_get(lpfc_mbx_read_top_att_type, la) ==
- LPFC_ATT_LINK_DOWN) {
+ } else if (attn_type == LPFC_ATT_LINK_DOWN ||
+ attn_type == LPFC_ATT_UNEXP_WWPN) {
phba->fc_stat.LinkDown++;
if (phba->link_flag & LS_LOOPBACK_MODE)
lpfc_printf_log(phba, KERN_ERR, LOG_LINK_EVENT,
@@ -3389,6 +3393,14 @@
"Data: x%x x%x x%x\n",
la->eventTag, phba->fc_eventTag,
phba->pport->port_state, vport->fc_flag);
+ else if (attn_type == LPFC_ATT_UNEXP_WWPN)
+ lpfc_printf_log(phba, KERN_ERR, LOG_LINK_EVENT,
+ "1313 Link Down UNEXP WWPN Event x%x received "
+ "Data: x%x x%x x%x x%x x%x\n",
+ la->eventTag, phba->fc_eventTag,
+ phba->pport->port_state, vport->fc_flag,
+ bf_get(lpfc_mbx_read_top_mm, la),
+ bf_get(lpfc_mbx_read_top_fa, la));
else
lpfc_printf_log(phba, KERN_ERR, LOG_LINK_EVENT,
"1305 Link Down Event x%x received "
@@ -3399,8 +3411,8 @@
bf_get(lpfc_mbx_read_top_fa, la));
lpfc_mbx_issue_link_down(phba);
}
- if ((phba->sli.sli_flag & LPFC_MENLO_MAINT) &&
- ((bf_get(lpfc_mbx_read_top_att_type, la) == LPFC_ATT_LINK_UP))) {
+ if (phba->sli.sli_flag & LPFC_MENLO_MAINT &&
+ attn_type == LPFC_ATT_LINK_UP) {
if (phba->link_state != LPFC_LINK_DOWN) {
phba->fc_stat.LinkDown++;
lpfc_printf_log(phba, KERN_ERR, LOG_LINK_EVENT,
@@ -4136,7 +4148,6 @@
int old_state, int new_state)
{
struct Scsi_Host *shost = lpfc_shost_from_vport(vport);
- struct lpfc_hba *phba = vport->phba;
if (new_state == NLP_STE_UNMAPPED_NODE) {
ndlp->nlp_flag &= ~NLP_NODEV_REMOVE;
@@ -4155,14 +4166,14 @@
lpfc_unregister_remote_port(ndlp);
}
- /* Notify the NVME transport of this rport's loss */
- if (((phba->cfg_enable_fc4_type == LPFC_ENABLE_BOTH) ||
- (phba->cfg_enable_fc4_type == LPFC_ENABLE_NVME)) &&
- (vport->phba->nvmet_support == 0) &&
- ((ndlp->nlp_fc4_type & NLP_FC4_NVME) ||
- (ndlp->nlp_DID == Fabric_DID))) {
+ /* Notify the NVME transport of this rport's loss on the
+ * Initiator. For NVME Target, should upcall transport
+ * in the else clause when API available.
+ */
+ if (ndlp->nlp_fc4_type & NLP_FC4_NVME) {
vport->phba->nport_event_cnt++;
- lpfc_nvme_unregister_port(vport, ndlp);
+ if (vport->phba->nvmet_support == 0)
+ lpfc_nvme_unregister_port(vport, ndlp);
}
}
@@ -4368,10 +4379,17 @@
uint32_t did;
unsigned long flags;
unsigned long *active_rrqs_xri_bitmap = NULL;
+ int rpi = LPFC_RPI_ALLOC_ERROR;
if (!ndlp)
return NULL;
+ if (phba->sli_rev == LPFC_SLI_REV4) {
+ rpi = lpfc_sli4_alloc_rpi(vport->phba);
+ if (rpi == LPFC_RPI_ALLOC_ERROR)
+ return NULL;
+ }
+
spin_lock_irqsave(&phba->ndlp_lock, flags);
/* The ndlp should not be in memory free mode */
if (NLP_CHK_FREE_REQ(ndlp)) {
@@ -4381,7 +4399,7 @@
"usgmap:x%x refcnt:%d\n",
(void *)ndlp, ndlp->nlp_usg_map,
kref_read(&ndlp->kref));
- return NULL;
+ goto free_rpi;
}
/* The ndlp should not already be in active mode */
if (NLP_CHK_NODE_ACT(ndlp)) {
@@ -4391,7 +4409,7 @@
"usgmap:x%x refcnt:%d\n",
(void *)ndlp, ndlp->nlp_usg_map,
kref_read(&ndlp->kref));
- return NULL;
+ goto free_rpi;
}
/* Keep the original DID */
@@ -4409,7 +4427,7 @@
spin_unlock_irqrestore(&phba->ndlp_lock, flags);
if (vport->phba->sli_rev == LPFC_SLI_REV4) {
- ndlp->nlp_rpi = lpfc_sli4_alloc_rpi(vport->phba);
+ ndlp->nlp_rpi = rpi;
lpfc_printf_vlog(vport, KERN_INFO, LOG_NODE,
"0008 rpi:%x DID:%x flg:%x refcnt:%d "
"map:%x %p\n", ndlp->nlp_rpi, ndlp->nlp_DID,
@@ -4426,6 +4444,11 @@
"node enable: did:x%x",
ndlp->nlp_DID, 0, 0);
return ndlp;
+
+free_rpi:
+ if (phba->sli_rev == LPFC_SLI_REV4)
+ lpfc_sli4_free_rpi(vport->phba, rpi);
+ return NULL;
}
void
@@ -5104,65 +5127,82 @@
ndlp = lpfc_findnode_did(vport, did);
if (!ndlp) {
+ if (vport->phba->nvmet_support)
+ return NULL;
if ((vport->fc_flag & FC_RSCN_MODE) != 0 &&
lpfc_rscn_payload_check(vport, did) == 0)
return NULL;
- ndlp = (struct lpfc_nodelist *)
- mempool_alloc(vport->phba->nlp_mem_pool, GFP_KERNEL);
+ ndlp = lpfc_nlp_init(vport, did);
if (!ndlp)
return NULL;
- lpfc_nlp_init(vport, ndlp, did);
lpfc_nlp_set_state(vport, ndlp, NLP_STE_NPR_NODE);
- if (vport->phba->nvmet_support)
- return ndlp;
spin_lock_irq(shost->host_lock);
ndlp->nlp_flag |= NLP_NPR_2B_DISC;
spin_unlock_irq(shost->host_lock);
return ndlp;
} else if (!NLP_CHK_NODE_ACT(ndlp)) {
+ if (vport->phba->nvmet_support)
+ return NULL;
ndlp = lpfc_enable_node(vport, ndlp, NLP_STE_NPR_NODE);
if (!ndlp)
return NULL;
- if (vport->phba->nvmet_support)
- return ndlp;
spin_lock_irq(shost->host_lock);
ndlp->nlp_flag |= NLP_NPR_2B_DISC;
spin_unlock_irq(shost->host_lock);
return ndlp;
}
+ /* The NVME Target does not want to actively manage an rport.
+ * The goal is to allow the target to reset its state and clear
+ * pending IO in preparation for the initiator to recover.
+ */
if ((vport->fc_flag & FC_RSCN_MODE) &&
!(vport->fc_flag & FC_NDISC_ACTIVE)) {
if (lpfc_rscn_payload_check(vport, did)) {
+
+ /* Since this node is marked for discovery,
+ * delay timeout is not needed.
+ */
+ lpfc_cancel_retry_delay_tmo(vport, ndlp);
+
+ /* NVME Target mode waits until rport is known to be
+ * impacted by the RSCN before it transitions. No
+ * active management - just go to NPR provided the
+ * node had a valid login.
+ */
+ if (vport->phba->nvmet_support)
+ return ndlp;
+
/* If we've already received a PLOGI from this NPort
* we don't need to try to discover it again.
*/
if (ndlp->nlp_flag & NLP_RCV_PLOGI)
return NULL;
- /* Since this node is marked for discovery,
- * delay timeout is not needed.
- */
- lpfc_cancel_retry_delay_tmo(vport, ndlp);
- if (vport->phba->nvmet_support)
- return ndlp;
spin_lock_irq(shost->host_lock);
ndlp->nlp_flag |= NLP_NPR_2B_DISC;
spin_unlock_irq(shost->host_lock);
} else
ndlp = NULL;
} else {
- /* If we've already received a PLOGI from this NPort,
- * or we are already in the process of discovery on it,
- * we don't need to try to discover it again.
+ /* If the initiator received a PLOGI from this NPort or if the
+ * initiator is already in the process of discovery on it,
+ * there's no need to try to discover it again.
*/
if (ndlp->nlp_state == NLP_STE_ADISC_ISSUE ||
ndlp->nlp_state == NLP_STE_PLOGI_ISSUE ||
- ndlp->nlp_flag & NLP_RCV_PLOGI)
+ (!vport->phba->nvmet_support &&
+ ndlp->nlp_flag & NLP_RCV_PLOGI))
return NULL;
- lpfc_nlp_set_state(vport, ndlp, NLP_STE_NPR_NODE);
+
if (vport->phba->nvmet_support)
return ndlp;
+
+ /* Moving to NPR state clears unsolicited flags and
+ * allows for rediscovery
+ */
+ lpfc_nlp_set_state(vport, ndlp, NLP_STE_NPR_NODE);
+
spin_lock_irq(shost->host_lock);
ndlp->nlp_flag |= NLP_NPR_2B_DISC;
spin_unlock_irq(shost->host_lock);
@@ -5887,16 +5927,31 @@
return NULL;
}
-void
-lpfc_nlp_init(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp,
- uint32_t did)
+struct lpfc_nodelist *
+lpfc_nlp_init(struct lpfc_vport *vport, uint32_t did)
{
+ struct lpfc_nodelist *ndlp;
+ int rpi = LPFC_RPI_ALLOC_ERROR;
+
+ if (vport->phba->sli_rev == LPFC_SLI_REV4) {
+ rpi = lpfc_sli4_alloc_rpi(vport->phba);
+ if (rpi == LPFC_RPI_ALLOC_ERROR)
+ return NULL;
+ }
+
+ ndlp = mempool_alloc(vport->phba->nlp_mem_pool, GFP_KERNEL);
+ if (!ndlp) {
+ if (vport->phba->sli_rev == LPFC_SLI_REV4)
+ lpfc_sli4_free_rpi(vport->phba, rpi);
+ return NULL;
+ }
+
memset(ndlp, 0, sizeof (struct lpfc_nodelist));
lpfc_initialize_node(vport, ndlp, did);
INIT_LIST_HEAD(&ndlp->nlp_listp);
if (vport->phba->sli_rev == LPFC_SLI_REV4) {
- ndlp->nlp_rpi = lpfc_sli4_alloc_rpi(vport->phba);
+ ndlp->nlp_rpi = rpi;
lpfc_printf_vlog(vport, KERN_INFO, LOG_NODE,
"0007 rpi:%x DID:%x flg:%x refcnt:%d "
"map:%x %p\n", ndlp->nlp_rpi, ndlp->nlp_DID,
@@ -5918,7 +5973,7 @@
"node init: did:x%x",
ndlp->nlp_DID, 0, 0);
- return;
+ return ndlp;
}
/* This routine releases all resources associated with a specifc NPort's ndlp
diff --git a/drivers/scsi/lpfc/lpfc_hw.h b/drivers/scsi/lpfc/lpfc_hw.h
index 15ca214..26a5647 100644
--- a/drivers/scsi/lpfc/lpfc_hw.h
+++ b/drivers/scsi/lpfc/lpfc_hw.h
@@ -509,6 +509,8 @@
uint8_t word3Reserved2; /* Fc Word 3, bit 0: 7 */
};
+#define FAPWWN_KEY_VENDOR 0x42524344 /*valid vendor version fawwpn key*/
+
struct serv_parm { /* Structure is in Big Endian format */
struct csp cmn;
struct lpfc_name portName;
@@ -2885,6 +2887,7 @@
#define LPFC_ATT_RESERVED 0x00 /* Reserved - attType */
#define LPFC_ATT_LINK_UP 0x01 /* Link is up */
#define LPFC_ATT_LINK_DOWN 0x02 /* Link is down */
+#define LPFC_ATT_UNEXP_WWPN 0x06 /* Link is down Unexpected WWWPN */
uint32_t word3;
#define lpfc_mbx_read_top_alpa_granted_SHIFT 24
#define lpfc_mbx_read_top_alpa_granted_MASK 0x000000FF
diff --git a/drivers/scsi/lpfc/lpfc_hw4.h b/drivers/scsi/lpfc/lpfc_hw4.h
index 1527770..1d12f2b 100644
--- a/drivers/scsi/lpfc/lpfc_hw4.h
+++ b/drivers/scsi/lpfc/lpfc_hw4.h
@@ -2720,6 +2720,9 @@
#define lpfc_mbx_rq_ftr_rq_ifip_SHIFT 7
#define lpfc_mbx_rq_ftr_rq_ifip_MASK 0x00000001
#define lpfc_mbx_rq_ftr_rq_ifip_WORD word2
+#define lpfc_mbx_rq_ftr_rq_iaar_SHIFT 9
+#define lpfc_mbx_rq_ftr_rq_iaar_MASK 0x00000001
+#define lpfc_mbx_rq_ftr_rq_iaar_WORD word2
#define lpfc_mbx_rq_ftr_rq_perfh_SHIFT 11
#define lpfc_mbx_rq_ftr_rq_perfh_MASK 0x00000001
#define lpfc_mbx_rq_ftr_rq_perfh_WORD word2
@@ -3853,6 +3856,7 @@
#define LPFC_FC_LA_TYPE_NO_HARD_ALPA 0x3
#define LPFC_FC_LA_TYPE_MDS_LINK_DOWN 0x4
#define LPFC_FC_LA_TYPE_MDS_LOOPBACK 0x5
+#define LPFC_FC_LA_TYPE_UNEXP_WWPN 0x6
#define lpfc_acqe_fc_la_port_type_SHIFT 6
#define lpfc_acqe_fc_la_port_type_MASK 0x00000003
#define lpfc_acqe_fc_la_port_type_WORD word0
diff --git a/drivers/scsi/lpfc/lpfc_init.c b/drivers/scsi/lpfc/lpfc_init.c
index 6cc561b..90ae354 100644
--- a/drivers/scsi/lpfc/lpfc_init.c
+++ b/drivers/scsi/lpfc/lpfc_init.c
@@ -42,6 +42,10 @@
#include <scsi/scsi_device.h>
#include <scsi/scsi_host.h>
#include <scsi/scsi_transport_fc.h>
+#include <scsi/scsi_tcq.h>
+#include <scsi/fc/fc_fs.h>
+
+#include <linux/nvme-fc-driver.h>
#include "lpfc_hw4.h"
#include "lpfc_hw.h"
@@ -52,6 +56,7 @@
#include "lpfc.h"
#include "lpfc_scsi.h"
#include "lpfc_nvme.h"
+#include "lpfc_nvmet.h"
#include "lpfc_logmsg.h"
#include "lpfc_crtn.h"
#include "lpfc_vport.h"
@@ -335,6 +340,9 @@
void
lpfc_update_vport_wwn(struct lpfc_vport *vport)
{
+ uint8_t vvvl = vport->fc_sparam.cmn.valid_vendor_ver_level;
+ u32 *fawwpn_key = (u32 *)&vport->fc_sparam.un.vendorVersion[0];
+
/* If the soft name exists then update it using the service params */
if (vport->phba->cfg_soft_wwnn)
u64_to_wwn(vport->phba->cfg_soft_wwnn,
@@ -354,9 +362,25 @@
memcpy(&vport->fc_sparam.nodeName, &vport->fc_nodename,
sizeof(struct lpfc_name));
- if (vport->fc_portname.u.wwn[0] == 0 || vport->phba->cfg_soft_wwpn)
+ /*
+ * If the port name has changed, then set the Param changes flag
+ * to unreg the login
+ */
+ if (vport->fc_portname.u.wwn[0] != 0 &&
+ memcmp(&vport->fc_portname, &vport->fc_sparam.portName,
+ sizeof(struct lpfc_name)))
+ vport->vport_flag |= FAWWPN_PARAM_CHG;
+
+ if (vport->fc_portname.u.wwn[0] == 0 ||
+ vport->phba->cfg_soft_wwpn ||
+ (vvvl == 1 && cpu_to_be32(*fawwpn_key) == FAPWWN_KEY_VENDOR) ||
+ vport->vport_flag & FAWWPN_SET) {
memcpy(&vport->fc_portname, &vport->fc_sparam.portName,
sizeof(struct lpfc_name));
+ vport->vport_flag &= ~FAWWPN_SET;
+ if (vvvl == 1 && cpu_to_be32(*fawwpn_key) == FAPWWN_KEY_VENDOR)
+ vport->vport_flag |= FAWWPN_SET;
+ }
else
memcpy(&vport->fc_sparam.portName, &vport->fc_portname,
sizeof(struct lpfc_name));
@@ -1003,8 +1027,10 @@
lpfc_hba_down_post_s4(struct lpfc_hba *phba)
{
struct lpfc_scsi_buf *psb, *psb_next;
+ struct lpfc_nvmet_rcv_ctx *ctxp, *ctxp_next;
LIST_HEAD(aborts);
LIST_HEAD(nvme_aborts);
+ LIST_HEAD(nvmet_aborts);
unsigned long iflag = 0;
struct lpfc_sglq *sglq_entry = NULL;
@@ -1027,16 +1053,10 @@
list_for_each_entry(sglq_entry,
&phba->sli4_hba.lpfc_abts_els_sgl_list, list)
sglq_entry->state = SGL_FREED;
- list_for_each_entry(sglq_entry,
- &phba->sli4_hba.lpfc_abts_nvmet_sgl_list, list)
- sglq_entry->state = SGL_FREED;
list_splice_init(&phba->sli4_hba.lpfc_abts_els_sgl_list,
&phba->sli4_hba.lpfc_els_sgl_list);
- if (phba->sli4_hba.nvme_wq)
- list_splice_init(&phba->sli4_hba.lpfc_abts_nvmet_sgl_list,
- &phba->sli4_hba.lpfc_nvmet_sgl_list);
spin_unlock(&phba->sli4_hba.sgl_list_lock);
/* abts_scsi_buf_list_lock required because worker thread uses this
@@ -1053,6 +1073,8 @@
spin_lock(&phba->sli4_hba.abts_nvme_buf_list_lock);
list_splice_init(&phba->sli4_hba.lpfc_abts_nvme_buf_list,
&nvme_aborts);
+ list_splice_init(&phba->sli4_hba.lpfc_abts_nvmet_ctx_list,
+ &nvmet_aborts);
spin_unlock(&phba->sli4_hba.abts_nvme_buf_list_lock);
}
@@ -1066,13 +1088,20 @@
list_splice(&aborts, &phba->lpfc_scsi_buf_list_put);
spin_unlock_irqrestore(&phba->scsi_buf_list_put_lock, iflag);
- list_for_each_entry_safe(psb, psb_next, &nvme_aborts, list) {
- psb->pCmd = NULL;
- psb->status = IOSTAT_SUCCESS;
+ if (phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME) {
+ list_for_each_entry_safe(psb, psb_next, &nvme_aborts, list) {
+ psb->pCmd = NULL;
+ psb->status = IOSTAT_SUCCESS;
+ }
+ spin_lock_irqsave(&phba->nvme_buf_list_put_lock, iflag);
+ list_splice(&nvme_aborts, &phba->lpfc_nvme_buf_list_put);
+ spin_unlock_irqrestore(&phba->nvme_buf_list_put_lock, iflag);
+
+ list_for_each_entry_safe(ctxp, ctxp_next, &nvmet_aborts, list) {
+ ctxp->flag &= ~(LPFC_NVMET_XBUSY | LPFC_NVMET_ABORT_OP);
+ lpfc_nvmet_rq_post(phba, ctxp, &ctxp->rqb_buffer->hbuf);
+ }
}
- spin_lock_irqsave(&phba->nvme_buf_list_put_lock, iflag);
- list_splice(&nvme_aborts, &phba->lpfc_nvme_buf_list_put);
- spin_unlock_irqrestore(&phba->nvme_buf_list_put_lock, iflag);
lpfc_sli4_free_sp_events(phba);
return 0;
@@ -2874,34 +2903,38 @@
{
struct lpfc_nodelist *ndlp, *next_ndlp;
struct lpfc_vport **vports;
- int i;
+ int i, rpi;
+ unsigned long flags;
if (phba->sli_rev != LPFC_SLI_REV4)
return;
vports = lpfc_create_vport_work_array(phba);
- if (vports != NULL) {
- for (i = 0; i <= phba->max_vports && vports[i] != NULL; i++) {
- if (vports[i]->load_flag & FC_UNLOADING)
- continue;
+ if (vports == NULL)
+ return;
- list_for_each_entry_safe(ndlp, next_ndlp,
- &vports[i]->fc_nodes,
- nlp_listp) {
- if (NLP_CHK_NODE_ACT(ndlp)) {
- ndlp->nlp_rpi =
- lpfc_sli4_alloc_rpi(phba);
- lpfc_printf_vlog(ndlp->vport, KERN_INFO,
- LOG_NODE,
- "0009 rpi:%x DID:%x "
- "flg:%x map:%x %p\n",
- ndlp->nlp_rpi,
- ndlp->nlp_DID,
- ndlp->nlp_flag,
- ndlp->nlp_usg_map,
- ndlp);
- }
+ for (i = 0; i <= phba->max_vports && vports[i] != NULL; i++) {
+ if (vports[i]->load_flag & FC_UNLOADING)
+ continue;
+
+ list_for_each_entry_safe(ndlp, next_ndlp,
+ &vports[i]->fc_nodes,
+ nlp_listp) {
+ if (!NLP_CHK_NODE_ACT(ndlp))
+ continue;
+ rpi = lpfc_sli4_alloc_rpi(phba);
+ if (rpi == LPFC_RPI_ALLOC_ERROR) {
+ spin_lock_irqsave(&phba->ndlp_lock, flags);
+ NLP_CLR_NODE_ACT(ndlp);
+ spin_unlock_irqrestore(&phba->ndlp_lock, flags);
+ continue;
}
+ ndlp->nlp_rpi = rpi;
+ lpfc_printf_vlog(ndlp->vport, KERN_INFO, LOG_NODE,
+ "0009 rpi:%x DID:%x "
+ "flg:%x map:%x %p\n", ndlp->nlp_rpi,
+ ndlp->nlp_DID, ndlp->nlp_flag,
+ ndlp->nlp_usg_map, ndlp);
}
}
lpfc_destroy_vport_work_array(phba, vports);
@@ -3508,6 +3541,12 @@
spin_unlock(&phba->scsi_buf_list_put_lock);
spin_unlock_irq(&phba->scsi_buf_list_get_lock);
+ lpfc_printf_log(phba, KERN_INFO, LOG_SLI,
+ "6060 Current allocated SCSI xri-sgl count:%d, "
+ "maximum SCSI xri count:%d (split:%d)\n",
+ phba->sli4_hba.scsi_xri_cnt,
+ phba->sli4_hba.scsi_xri_max, phba->cfg_xri_split);
+
if (phba->sli4_hba.scsi_xri_cnt > phba->sli4_hba.scsi_xri_max) {
/* max scsi xri shrinked below the allocated scsi buffers */
scsi_xri_cnt = phba->sli4_hba.scsi_xri_cnt -
@@ -4508,9 +4547,15 @@
/* Parse and translate link attention fields */
la = (struct lpfc_mbx_read_top *)&pmb->u.mb.un.varReadTop;
la->eventTag = acqe_fc->event_tag;
- bf_set(lpfc_mbx_read_top_att_type, la,
- LPFC_FC_LA_TYPE_LINK_DOWN);
+ if (phba->sli4_hba.link_state.status ==
+ LPFC_FC_LA_TYPE_UNEXP_WWPN) {
+ bf_set(lpfc_mbx_read_top_att_type, la,
+ LPFC_FC_LA_TYPE_UNEXP_WWPN);
+ } else {
+ bf_set(lpfc_mbx_read_top_att_type, la,
+ LPFC_FC_LA_TYPE_LINK_DOWN);
+ }
/* Invoke the mailbox command callback function */
lpfc_mbx_cmpl_read_topology(phba, pmb);
@@ -4716,10 +4761,9 @@
ndlp = lpfc_findnode_did(vport, Fabric_DID);
if (!ndlp) {
/* Cannot find existing Fabric ndlp, so allocate a new one */
- ndlp = mempool_alloc(phba->nlp_mem_pool, GFP_KERNEL);
+ ndlp = lpfc_nlp_init(vport, Fabric_DID);
if (!ndlp)
return 0;
- lpfc_nlp_init(vport, ndlp, Fabric_DID);
/* Set the node type */
ndlp->nlp_type |= NLP_FABRIC;
/* Put ndlp onto node list */
@@ -5778,6 +5822,7 @@
/* Initialize the Abort nvme buffer list used by driver */
spin_lock_init(&phba->sli4_hba.abts_nvme_buf_list_lock);
INIT_LIST_HEAD(&phba->sli4_hba.lpfc_abts_nvme_buf_list);
+ INIT_LIST_HEAD(&phba->sli4_hba.lpfc_abts_nvmet_ctx_list);
/* Fast-path XRI aborted CQ Event work queue list */
INIT_LIST_HEAD(&phba->sli4_hba.sp_nvme_xri_aborted_work_queue);
}
@@ -5809,6 +5854,12 @@
INIT_LIST_HEAD(&phba->sli4_hba.lpfc_vfi_blk_list);
INIT_LIST_HEAD(&phba->lpfc_vpi_blk_list);
+ /* Initialize mboxq lists. If the early init routines fail
+ * these lists need to be correctly initialized.
+ */
+ INIT_LIST_HEAD(&phba->sli.mboxq);
+ INIT_LIST_HEAD(&phba->sli.mboxq_cmpl);
+
/* initialize optic_state to 0xFF */
phba->sli4_hba.lnk_info.optic_state = 0xff;
@@ -5874,6 +5925,7 @@
"READ_NV, mbxStatus x%x\n",
bf_get(lpfc_mqe_command, &mboxq->u.mqe),
bf_get(lpfc_mqe_status, &mboxq->u.mqe));
+ mempool_free(mboxq, phba->mbox_mem_pool);
rc = -EIO;
goto out_free_bsmbx;
}
@@ -6398,7 +6450,7 @@
INIT_LIST_HEAD(&phba->sli4_hba.lpfc_els_sgl_list);
INIT_LIST_HEAD(&phba->sli4_hba.lpfc_abts_els_sgl_list);
INIT_LIST_HEAD(&phba->sli4_hba.lpfc_nvmet_sgl_list);
- INIT_LIST_HEAD(&phba->sli4_hba.lpfc_abts_nvmet_sgl_list);
+ INIT_LIST_HEAD(&phba->sli4_hba.lpfc_abts_nvmet_ctx_list);
/* els xri-sgl book keeping */
phba->sli4_hba.els_xri_cnt = 0;
@@ -7799,7 +7851,7 @@
/* Create Fast Path FCP WQs */
wqesize = (phba->fcp_embed_io) ?
- LPFC_WQE128_SIZE : phba->sli4_hba.wq_esize;
+ LPFC_WQE128_SIZE : phba->sli4_hba.wq_esize;
qdesc = lpfc_sli4_queue_alloc(phba, wqesize, phba->sli4_hba.wq_ecount);
if (!qdesc) {
lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
@@ -7830,7 +7882,7 @@
lpfc_sli4_queue_create(struct lpfc_hba *phba)
{
struct lpfc_queue *qdesc;
- int idx, io_channel, max;
+ int idx, io_channel;
/*
* Create HBA Record arrays.
@@ -7991,15 +8043,6 @@
if (lpfc_alloc_nvme_wq_cq(phba, idx))
goto out_error;
- /* allocate MRQ CQs */
- max = phba->cfg_nvme_io_channel;
- if (max < phba->cfg_nvmet_mrq)
- max = phba->cfg_nvmet_mrq;
-
- for (idx = 0; idx < max; idx++)
- if (lpfc_alloc_nvme_wq_cq(phba, idx))
- goto out_error;
-
if (phba->nvmet_support) {
for (idx = 0; idx < phba->cfg_nvmet_mrq; idx++) {
qdesc = lpfc_sli4_queue_alloc(phba,
@@ -8221,11 +8264,11 @@
/* Release FCP cqs */
lpfc_sli4_release_queues(&phba->sli4_hba.fcp_cq,
- phba->cfg_fcp_io_channel);
+ phba->cfg_fcp_io_channel);
/* Release FCP wqs */
lpfc_sli4_release_queues(&phba->sli4_hba.fcp_wq,
- phba->cfg_fcp_io_channel);
+ phba->cfg_fcp_io_channel);
/* Release FCP CQ mapping array */
lpfc_sli4_release_queue_map(&phba->sli4_hba.fcp_cq_map);
@@ -8571,15 +8614,15 @@
lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
"0528 %s not allocated\n",
phba->sli4_hba.mbx_cq ?
- "Mailbox WQ" : "Mailbox CQ");
+ "Mailbox WQ" : "Mailbox CQ");
rc = -ENOMEM;
goto out_destroy;
}
rc = lpfc_create_wq_cq(phba, phba->sli4_hba.hba_eq[0],
- phba->sli4_hba.mbx_cq,
- phba->sli4_hba.mbx_wq,
- NULL, 0, LPFC_MBOX);
+ phba->sli4_hba.mbx_cq,
+ phba->sli4_hba.mbx_wq,
+ NULL, 0, LPFC_MBOX);
if (rc) {
lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
"0529 Failed setup of mailbox WQ/CQ: rc = 0x%x\n",
@@ -9934,17 +9977,19 @@
{
int wait_time = 0;
int nvme_xri_cmpl = 1;
+ int nvmet_xri_cmpl = 1;
int fcp_xri_cmpl = 1;
int els_xri_cmpl = list_empty(&phba->sli4_hba.lpfc_abts_els_sgl_list);
- int nvmet_xri_cmpl =
- list_empty(&phba->sli4_hba.lpfc_abts_nvmet_sgl_list);
if (phba->cfg_enable_fc4_type & LPFC_ENABLE_FCP)
fcp_xri_cmpl =
list_empty(&phba->sli4_hba.lpfc_abts_scsi_buf_list);
- if (phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME)
+ if (phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME) {
nvme_xri_cmpl =
list_empty(&phba->sli4_hba.lpfc_abts_nvme_buf_list);
+ nvmet_xri_cmpl =
+ list_empty(&phba->sli4_hba.lpfc_abts_nvmet_ctx_list);
+ }
while (!fcp_xri_cmpl || !els_xri_cmpl || !nvme_xri_cmpl ||
!nvmet_xri_cmpl) {
@@ -9970,9 +10015,12 @@
msleep(LPFC_XRI_EXCH_BUSY_WAIT_T1);
wait_time += LPFC_XRI_EXCH_BUSY_WAIT_T1;
}
- if (phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME)
+ if (phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME) {
nvme_xri_cmpl = list_empty(
&phba->sli4_hba.lpfc_abts_nvme_buf_list);
+ nvmet_xri_cmpl = list_empty(
+ &phba->sli4_hba.lpfc_abts_nvmet_ctx_list);
+ }
if (phba->cfg_enable_fc4_type & LPFC_ENABLE_FCP)
fcp_xri_cmpl = list_empty(
@@ -9981,8 +10029,6 @@
els_xri_cmpl =
list_empty(&phba->sli4_hba.lpfc_abts_els_sgl_list);
- nvmet_xri_cmpl =
- list_empty(&phba->sli4_hba.lpfc_abts_nvmet_sgl_list);
}
}
@@ -10048,9 +10094,14 @@
/* Stop kthread signal shall trigger work_done one more time */
kthread_stop(phba->worker_thread);
+ /* Unset the queues shared with the hardware then release all
+ * allocated resources.
+ */
+ lpfc_sli4_queue_unset(phba);
+ lpfc_sli4_queue_destroy(phba);
+
/* Reset SLI4 HBA FCoE function */
lpfc_pci_function_reset(phba);
- lpfc_sli4_queue_destroy(phba);
/* Stop the SLI4 device port */
phba->pport->work_port_events = 0;
@@ -10306,6 +10357,7 @@
}
/* Initialize and populate the iocb list per host */
+
error = lpfc_init_iocb_list(phba, LPFC_IOCB_LIST_CNT);
if (error) {
lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
@@ -11051,7 +11103,7 @@
struct lpfc_hba *phba;
struct lpfc_vport *vport = NULL;
struct Scsi_Host *shost = NULL;
- int error;
+ int error, cnt;
uint32_t cfg_mode, intr_mode;
/* Allocate memory for HBA structure */
@@ -11085,12 +11137,15 @@
goto out_unset_pci_mem_s4;
}
- /* Initialize and populate the iocb list per host */
+ cnt = phba->cfg_iocb_cnt * 1024;
+ if (phba->nvmet_support)
+ cnt += phba->cfg_nvmet_mrq_post * phba->cfg_nvmet_mrq;
+ /* Initialize and populate the iocb list per host */
lpfc_printf_log(phba, KERN_INFO, LOG_INIT,
- "2821 initialize iocb list %d.\n",
- phba->cfg_iocb_cnt*1024);
- error = lpfc_init_iocb_list(phba, phba->cfg_iocb_cnt*1024);
+ "2821 initialize iocb list %d total %d\n",
+ phba->cfg_iocb_cnt, cnt);
+ error = lpfc_init_iocb_list(phba, cnt);
if (error) {
lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
@@ -11177,7 +11232,9 @@
if ((phba->nvmet_support == 0) &&
(phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME)) {
/* Create NVME binding with nvme_fc_transport. This
- * ensures the vport is initialized.
+ * ensures the vport is initialized. If the localport
+ * create fails, it should not unload the driver to
+ * support field issues.
*/
error = lpfc_nvme_create_localport(vport);
if (error) {
@@ -11185,7 +11242,6 @@
"6004 NVME registration failed, "
"error x%x\n",
error);
- goto out_disable_intr;
}
}
@@ -11984,6 +12040,7 @@
lpfc_fof_queue_create(struct lpfc_hba *phba)
{
struct lpfc_queue *qdesc;
+ uint32_t wqesize;
/* Create FOF EQ */
qdesc = lpfc_sli4_queue_alloc(phba, phba->sli4_hba.eq_esize,
@@ -12004,8 +12061,11 @@
phba->sli4_hba.oas_cq = qdesc;
/* Create OAS WQ */
- qdesc = lpfc_sli4_queue_alloc(phba, phba->sli4_hba.wq_esize,
+ wqesize = (phba->fcp_embed_io) ?
+ LPFC_WQE128_SIZE : phba->sli4_hba.wq_esize;
+ qdesc = lpfc_sli4_queue_alloc(phba, wqesize,
phba->sli4_hba.wq_ecount);
+
if (!qdesc)
goto out_error;
diff --git a/drivers/scsi/lpfc/lpfc_mbox.c b/drivers/scsi/lpfc/lpfc_mbox.c
index a928f518..ce25a18 100644
--- a/drivers/scsi/lpfc/lpfc_mbox.c
+++ b/drivers/scsi/lpfc/lpfc_mbox.c
@@ -2083,9 +2083,12 @@
if (phba->max_vpi && phba->cfg_enable_npiv)
bf_set(lpfc_mbx_rq_ftr_rq_npiv, &mboxq->u.mqe.un.req_ftrs, 1);
- if (phba->nvmet_support)
+ if (phba->nvmet_support) {
bf_set(lpfc_mbx_rq_ftr_rq_mrqp, &mboxq->u.mqe.un.req_ftrs, 1);
-
+ /* iaab/iaar NOT set for now */
+ bf_set(lpfc_mbx_rq_ftr_rq_iaab, &mboxq->u.mqe.un.req_ftrs, 0);
+ bf_set(lpfc_mbx_rq_ftr_rq_iaar, &mboxq->u.mqe.un.req_ftrs, 0);
+ }
return;
}
diff --git a/drivers/scsi/lpfc/lpfc_nportdisc.c b/drivers/scsi/lpfc/lpfc_nportdisc.c
index 061626b..8777c2d 100644
--- a/drivers/scsi/lpfc/lpfc_nportdisc.c
+++ b/drivers/scsi/lpfc/lpfc_nportdisc.c
@@ -361,8 +361,12 @@
case NLP_STE_PRLI_ISSUE:
case NLP_STE_UNMAPPED_NODE:
case NLP_STE_MAPPED_NODE:
- /* lpfc_plogi_confirm_nport skips fabric did, handle it here */
- if (!(ndlp->nlp_type & NLP_FABRIC)) {
+ /* For initiators, lpfc_plogi_confirm_nport skips fabric did.
+ * For target mode, execute implicit logo.
+ * Fabric nodes go into NPR.
+ */
+ if (!(ndlp->nlp_type & NLP_FABRIC) &&
+ !(phba->nvmet_support)) {
lpfc_els_rsp_acc(vport, ELS_CMD_PLOGI, cmdiocb,
ndlp, NULL);
return 1;
diff --git a/drivers/scsi/lpfc/lpfc_nvme.c b/drivers/scsi/lpfc/lpfc_nvme.c
index 0024de1..8008c82 100644
--- a/drivers/scsi/lpfc/lpfc_nvme.c
+++ b/drivers/scsi/lpfc/lpfc_nvme.c
@@ -401,6 +401,7 @@
struct lpfc_nodelist *ndlp;
struct ulp_bde64 *bpl;
struct lpfc_dmabuf *bmp;
+ uint16_t ntype, nstate;
/* there are two dma buf in the request, actually there is one and
* the second one is just the start address + cmd size.
@@ -417,11 +418,26 @@
vport = lport->vport;
ndlp = lpfc_findnode_did(vport, pnvme_rport->port_id);
- if (!ndlp) {
- lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_DISC,
- "6043 Could not find node for DID %x\n",
+ if (!ndlp || !NLP_CHK_NODE_ACT(ndlp)) {
+ lpfc_printf_vlog(vport, KERN_ERR, LOG_NODE | LOG_NVME_IOERR,
+ "6051 DID x%06x not an active rport.\n",
pnvme_rport->port_id);
- return 1;
+ return -ENODEV;
+ }
+
+ /* The remote node has to be a mapped nvme target or an
+ * unmapped nvme initiator or it's an error.
+ */
+ ntype = ndlp->nlp_type;
+ nstate = ndlp->nlp_state;
+ if ((ntype & NLP_NVME_TARGET && nstate != NLP_STE_MAPPED_NODE) ||
+ (ntype & NLP_NVME_INITIATOR && nstate != NLP_STE_UNMAPPED_NODE)) {
+ lpfc_printf_vlog(vport, KERN_ERR, LOG_NODE | LOG_NVME_IOERR,
+ "6088 DID x%06x not ready for "
+ "IO. State x%x, Type x%x\n",
+ pnvme_rport->port_id,
+ ndlp->nlp_state, ndlp->nlp_type);
+ return -ENODEV;
}
bmp = kmalloc(sizeof(struct lpfc_dmabuf), GFP_KERNEL);
if (!bmp) {
@@ -456,7 +472,7 @@
/* Expand print to include key fields. */
lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME_DISC,
- "6051 ENTER. lport %p, rport %p lsreq%p rqstlen:%d "
+ "6149 ENTER. lport %p, rport %p lsreq%p rqstlen:%d "
"rsplen:%d %pad %pad\n",
pnvme_lport, pnvme_rport,
pnvme_lsreq, pnvme_lsreq->rqstlen,
@@ -745,6 +761,7 @@
struct nvme_fc_cmd_iu *cp;
struct lpfc_nvme_rport *rport;
struct lpfc_nodelist *ndlp;
+ struct lpfc_nvme_fcpreq_priv *freqpriv;
unsigned long flags;
uint32_t code;
uint16_t cid, sqhd, data;
@@ -772,9 +789,8 @@
ndlp = rport->ndlp;
if (!ndlp || !NLP_CHK_NODE_ACT(ndlp)) {
lpfc_printf_vlog(vport, KERN_ERR, LOG_NODE | LOG_NVME_IOERR,
- "6061 rport %p, ndlp %p, DID x%06x ndlp "
- "not ready.\n",
- rport, ndlp, rport->remoteport->port_id);
+ "6061 rport %p, DID x%06x node not ready.\n",
+ rport, rport->remoteport->port_id);
ndlp = lpfc_findnode_did(vport, rport->remoteport->port_id);
if (!ndlp) {
@@ -853,15 +869,18 @@
break;
lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_IOERR,
"6081 NVME Completion Protocol Error: "
- "status x%x result x%x placed x%x\n",
+ "xri %x status x%x result x%x "
+ "placed x%x\n",
+ lpfc_ncmd->cur_iocbq.sli4_xritag,
lpfc_ncmd->status, lpfc_ncmd->result,
wcqe->total_data_placed);
break;
default:
out_err:
lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_IOERR,
- "6072 NVME Completion Error: "
+ "6072 NVME Completion Error: xri %x "
"status x%x result x%x placed x%x\n",
+ lpfc_ncmd->cur_iocbq.sli4_xritag,
lpfc_ncmd->status, lpfc_ncmd->result,
wcqe->total_data_placed);
nCmd->transferred_length = 0;
@@ -900,6 +919,8 @@
phba->cpucheck_cmpl_io[lpfc_ncmd->cpu]++;
}
#endif
+ freqpriv = nCmd->private;
+ freqpriv->nvme_buf = NULL;
nCmd->done(nCmd);
spin_lock_irqsave(&phba->hbalock, flags);
@@ -1099,12 +1120,12 @@
first_data_sgl = sgl;
lpfc_ncmd->seg_cnt = nCmd->sg_cnt;
- if (lpfc_ncmd->seg_cnt > phba->cfg_sg_seg_cnt) {
+ if (lpfc_ncmd->seg_cnt > phba->cfg_nvme_seg_cnt) {
lpfc_printf_log(phba, KERN_ERR, LOG_NVME_IOERR,
"6058 Too many sg segments from "
"NVME Transport. Max %d, "
"nvmeIO sg_cnt %d\n",
- phba->cfg_sg_seg_cnt,
+ phba->cfg_nvme_seg_cnt,
lpfc_ncmd->seg_cnt);
lpfc_ncmd->seg_cnt = 0;
return 1;
@@ -1196,6 +1217,7 @@
struct lpfc_nvme_buf *lpfc_ncmd;
struct lpfc_nvme_rport *rport;
struct lpfc_nvme_qhandle *lpfc_queue_info;
+ struct lpfc_nvme_fcpreq_priv *freqpriv = pnvme_fcreq->private;
#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
uint64_t start = 0;
#endif
@@ -1274,7 +1296,7 @@
* Do not let the IO hang out forever. There is no midlayer issuing
* an abort so inform the FW of the maximum IO pending time.
*/
- pnvme_fcreq->private = (void *)lpfc_ncmd;
+ freqpriv->nvme_buf = lpfc_ncmd;
lpfc_ncmd->nvmeCmd = pnvme_fcreq;
lpfc_ncmd->nrport = rport;
lpfc_ncmd->ndlp = ndlp;
@@ -1404,6 +1426,7 @@
struct lpfc_nvme_buf *lpfc_nbuf;
struct lpfc_iocbq *abts_buf;
struct lpfc_iocbq *nvmereq_wqe;
+ struct lpfc_nvme_fcpreq_priv *freqpriv = pnvme_fcreq->private;
union lpfc_wqe *abts_wqe;
unsigned long flags;
int ret_val;
@@ -1414,7 +1437,7 @@
phba = vport->phba;
/* Announce entry to new IO submit field. */
- lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_ABTS,
+ lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME_ABTS,
"6002 Abort Request to rport DID x%06x "
"for nvme_fc_req %p\n",
pnvme_rport->port_id,
@@ -1444,7 +1467,7 @@
/* The remote node has to be ready to send an abort. */
if ((ndlp->nlp_state != NLP_STE_MAPPED_NODE) &&
!(ndlp->nlp_type & NLP_NVME_TARGET)) {
- lpfc_printf_vlog(vport, KERN_ERR, LOG_NODE | LOG_NVME_ABTS,
+ lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_ABTS,
"6048 rport %p, DID x%06x not ready for "
"IO. State x%x, Type x%x\n",
rport, pnvme_rport->port_id,
@@ -1459,27 +1482,28 @@
/* driver queued commands are in process of being flushed */
if (phba->hba_flag & HBA_NVME_IOQ_FLUSH) {
spin_unlock_irqrestore(&phba->hbalock, flags);
- lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME,
+ lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_ABTS,
"6139 Driver in reset cleanup - flushing "
"NVME Req now. hba_flag x%x\n",
phba->hba_flag);
return;
}
- lpfc_nbuf = (struct lpfc_nvme_buf *)pnvme_fcreq->private;
+ lpfc_nbuf = freqpriv->nvme_buf;
if (!lpfc_nbuf) {
spin_unlock_irqrestore(&phba->hbalock, flags);
- lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME,
+ lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_ABTS,
"6140 NVME IO req has no matching lpfc nvme "
"io buffer. Skipping abort req.\n");
return;
} else if (!lpfc_nbuf->nvmeCmd) {
spin_unlock_irqrestore(&phba->hbalock, flags);
- lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME,
+ lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_ABTS,
"6141 lpfc NVME IO req has no nvme_fcreq "
"io buffer. Skipping abort req.\n");
return;
}
+ nvmereq_wqe = &lpfc_nbuf->cur_iocbq;
/*
* The lpfc_nbuf and the mapped nvme_fcreq in the driver's
@@ -1490,23 +1514,22 @@
*/
if (lpfc_nbuf->nvmeCmd != pnvme_fcreq) {
spin_unlock_irqrestore(&phba->hbalock, flags);
- lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME,
+ lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_ABTS,
"6143 NVME req mismatch: "
"lpfc_nbuf %p nvmeCmd %p, "
- "pnvme_fcreq %p. Skipping Abort\n",
+ "pnvme_fcreq %p. Skipping Abort xri x%x\n",
lpfc_nbuf, lpfc_nbuf->nvmeCmd,
- pnvme_fcreq);
+ pnvme_fcreq, nvmereq_wqe->sli4_xritag);
return;
}
/* Don't abort IOs no longer on the pending queue. */
- nvmereq_wqe = &lpfc_nbuf->cur_iocbq;
if (!(nvmereq_wqe->iocb_flag & LPFC_IO_ON_TXCMPLQ)) {
spin_unlock_irqrestore(&phba->hbalock, flags);
- lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME,
+ lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_ABTS,
"6142 NVME IO req %p not queued - skipping "
- "abort req\n",
- pnvme_fcreq);
+ "abort req xri x%x\n",
+ pnvme_fcreq, nvmereq_wqe->sli4_xritag);
return;
}
@@ -1517,21 +1540,22 @@
/* Outstanding abort is in progress */
if (nvmereq_wqe->iocb_flag & LPFC_DRIVER_ABORTED) {
spin_unlock_irqrestore(&phba->hbalock, flags);
- lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME,
+ lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_ABTS,
"6144 Outstanding NVME I/O Abort Request "
"still pending on nvme_fcreq %p, "
- "lpfc_ncmd %p\n",
- pnvme_fcreq, lpfc_nbuf);
+ "lpfc_ncmd %p xri x%x\n",
+ pnvme_fcreq, lpfc_nbuf,
+ nvmereq_wqe->sli4_xritag);
return;
}
abts_buf = __lpfc_sli_get_iocbq(phba);
if (!abts_buf) {
spin_unlock_irqrestore(&phba->hbalock, flags);
- lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME,
+ lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_ABTS,
"6136 No available abort wqes. Skipping "
- "Abts req for nvme_fcreq %p.\n",
- pnvme_fcreq);
+ "Abts req for nvme_fcreq %p xri x%x\n",
+ pnvme_fcreq, nvmereq_wqe->sli4_xritag);
return;
}
@@ -1580,7 +1604,7 @@
ret_val = lpfc_sli4_issue_wqe(phba, LPFC_FCP_RING, abts_buf);
spin_unlock_irqrestore(&phba->hbalock, flags);
if (ret_val == IOCB_ERROR) {
- lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME,
+ lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_ABTS,
"6137 Failed abts issue_wqe with status x%x "
"for nvme_fcreq %p.\n",
ret_val, pnvme_fcreq);
@@ -1588,8 +1612,8 @@
return;
}
- lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME,
- "6138 Transport Abort NVME Request Issued for\n"
+ lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME_ABTS,
+ "6138 Transport Abort NVME Request Issued for "
"ox_id x%x on reqtag x%x\n",
nvmereq_wqe->sli4_xritag,
abts_buf->iotag);
@@ -1618,7 +1642,7 @@
.local_priv_sz = sizeof(struct lpfc_nvme_lport),
.remote_priv_sz = sizeof(struct lpfc_nvme_rport),
.lsrqst_priv_sz = 0,
- .fcprqst_priv_sz = 0,
+ .fcprqst_priv_sz = sizeof(struct lpfc_nvme_fcpreq_priv),
};
/**
@@ -2049,7 +2073,7 @@
if (lpfc_test_rrq_active(phba, ndlp,
lpfc_ncmd->cur_iocbq.sli4_lxritag))
continue;
- list_del(&lpfc_ncmd->list);
+ list_del_init(&lpfc_ncmd->list);
found = 1;
break;
}
@@ -2064,7 +2088,7 @@
if (lpfc_test_rrq_active(
phba, ndlp, lpfc_ncmd->cur_iocbq.sli4_lxritag))
continue;
- list_del(&lpfc_ncmd->list);
+ list_del_init(&lpfc_ncmd->list);
found = 1;
break;
}
@@ -2092,6 +2116,12 @@
lpfc_ncmd->nonsg_phys = 0;
if (lpfc_ncmd->flags & LPFC_SBUF_XBUSY) {
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6310 XB release deferred for "
+ "ox_id x%x on reqtag x%x\n",
+ lpfc_ncmd->cur_iocbq.sli4_xritag,
+ lpfc_ncmd->cur_iocbq.iotag);
+
spin_lock_irqsave(&phba->sli4_hba.abts_nvme_buf_list_lock,
iflag);
lpfc_ncmd->nvmeCmd = NULL;
@@ -2142,8 +2172,18 @@
nfcp_info.node_name = wwn_to_u64(vport->fc_nodename.u.wwn);
nfcp_info.port_name = wwn_to_u64(vport->fc_portname.u.wwn);
- /* For now need + 1 to get around NVME transport logic */
- lpfc_nvme_template.max_sgl_segments = phba->cfg_sg_seg_cnt + 1;
+ /* Limit to LPFC_MAX_NVME_SEG_CNT.
+ * For now need + 1 to get around NVME transport logic.
+ */
+ if (phba->cfg_sg_seg_cnt > LPFC_MAX_NVME_SEG_CNT) {
+ lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME | LOG_INIT,
+ "6300 Reducing sg segment cnt to %d\n",
+ LPFC_MAX_NVME_SEG_CNT);
+ phba->cfg_nvme_seg_cnt = LPFC_MAX_NVME_SEG_CNT;
+ } else {
+ phba->cfg_nvme_seg_cnt = phba->cfg_sg_seg_cnt;
+ }
+ lpfc_nvme_template.max_sgl_segments = phba->cfg_nvme_seg_cnt + 1;
lpfc_nvme_template.max_hw_queues = phba->cfg_nvme_io_channel;
/* localport is allocated from the stack, but the registration
@@ -2249,12 +2289,23 @@
void
lpfc_nvme_update_localport(struct lpfc_vport *vport)
{
+#if (IS_ENABLED(CONFIG_NVME_FC))
struct nvme_fc_local_port *localport;
struct lpfc_nvme_lport *lport;
localport = vport->localport;
+ if (!localport) {
+ lpfc_printf_vlog(vport, KERN_WARNING, LOG_NVME,
+ "6710 Update NVME fail. No localport\n");
+ return;
+ }
lport = (struct lpfc_nvme_lport *)localport->private;
-
+ if (!lport) {
+ lpfc_printf_vlog(vport, KERN_WARNING, LOG_NVME,
+ "6171 Update NVME fail. localP %p, No lport\n",
+ localport);
+ return;
+ }
lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME,
"6012 Update NVME lport %p did x%x\n",
localport, vport->fc_myDID);
@@ -2268,7 +2319,7 @@
lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME_DISC,
"6030 bound lport %p to DID x%06x\n",
lport, localport->port_id);
-
+#endif
}
int
@@ -2409,6 +2460,7 @@
struct lpfc_nvme_lport *lport;
struct lpfc_nvme_rport *rport;
struct nvme_fc_remote_port *remoteport;
+ unsigned long wait_tmo;
localport = vport->localport;
@@ -2451,11 +2503,12 @@
* before proceeding. This guarantees the transport and driver
* have completed the unreg process.
*/
- ret = wait_for_completion_timeout(&rport->rport_unreg_done, 5);
+ wait_tmo = msecs_to_jiffies(5000);
+ ret = wait_for_completion_timeout(&rport->rport_unreg_done,
+ wait_tmo);
if (ret == 0) {
lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_DISC,
- "6169 Unreg nvme wait failed %d\n",
- ret);
+ "6169 Unreg nvme wait timeout\n");
}
}
return;
@@ -2463,7 +2516,7 @@
input_err:
#endif
lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_DISC,
- "6168: State error: lport %p, rport%p FCID x%06x\n",
+ "6168 State error: lport %p, rport%p FCID x%06x\n",
vport->localport, ndlp->rport, ndlp->nlp_DID);
}
@@ -2494,7 +2547,7 @@
&phba->sli4_hba.lpfc_abts_nvme_buf_list,
list) {
if (lpfc_ncmd->cur_iocbq.sli4_xritag == xri) {
- list_del(&lpfc_ncmd->list);
+ list_del_init(&lpfc_ncmd->list);
lpfc_ncmd->flags &= ~LPFC_SBUF_XBUSY;
lpfc_ncmd->status = IOSTAT_SUCCESS;
spin_unlock(
@@ -2510,6 +2563,12 @@
rxid, 1);
lpfc_sli4_abts_err_handler(phba, ndlp, axri);
}
+
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6311 XRI Aborted xri x%x tag x%x "
+ "released\n",
+ xri, lpfc_ncmd->cur_iocbq.iotag);
+
lpfc_release_nvme_buf(phba, lpfc_ncmd);
if (rrq_empty)
lpfc_worker_wake_up(phba);
@@ -2518,4 +2577,8 @@
}
spin_unlock(&phba->sli4_hba.abts_nvme_buf_list_lock);
spin_unlock_irqrestore(&phba->hbalock, iflag);
+
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6312 XRI Aborted xri x%x not found\n", xri);
+
}
diff --git a/drivers/scsi/lpfc/lpfc_nvme.h b/drivers/scsi/lpfc/lpfc_nvme.h
index 1347deb..ec32f45 100644
--- a/drivers/scsi/lpfc/lpfc_nvme.h
+++ b/drivers/scsi/lpfc/lpfc_nvme.h
@@ -21,12 +21,7 @@
* included with this package. *
********************************************************************/
-#define LPFC_NVME_MIN_SEGS 16
-#define LPFC_NVME_DEFAULT_SEGS 66 /* 256K IOs - 64 + 2 */
-#define LPFC_NVME_MAX_SEGS 510
-#define LPFC_NVMET_MIN_POSTBUF 16
-#define LPFC_NVMET_DEFAULT_POSTBUF 1024
-#define LPFC_NVMET_MAX_POSTBUF 4096
+#define LPFC_NVME_DEFAULT_SEGS (64 + 1) /* 256K IOs */
#define LPFC_NVME_WQSIZE 256
#define LPFC_NVME_ERSP_LEN 0x20
@@ -102,3 +97,7 @@
uint64_t ts_data_nvme;
#endif
};
+
+struct lpfc_nvme_fcpreq_priv {
+ struct lpfc_nvme_buf *nvme_buf;
+};
diff --git a/drivers/scsi/lpfc/lpfc_nvmet.c b/drivers/scsi/lpfc/lpfc_nvmet.c
index acba1b6..94434e6 100644
--- a/drivers/scsi/lpfc/lpfc_nvmet.c
+++ b/drivers/scsi/lpfc/lpfc_nvmet.c
@@ -71,6 +71,26 @@
struct lpfc_nvmet_rcv_ctx *,
uint32_t, uint16_t);
+void
+lpfc_nvmet_defer_release(struct lpfc_hba *phba, struct lpfc_nvmet_rcv_ctx *ctxp)
+{
+ unsigned long iflag;
+
+ lpfc_printf_log(phba, KERN_ERR, LOG_NVME_ABTS,
+ "6313 NVMET Defer ctx release xri x%x flg x%x\n",
+ ctxp->oxid, ctxp->flag);
+
+ spin_lock_irqsave(&phba->sli4_hba.abts_nvme_buf_list_lock, iflag);
+ if (ctxp->flag & LPFC_NVMET_CTX_RLS) {
+ spin_unlock_irqrestore(&phba->sli4_hba.abts_nvme_buf_list_lock,
+ iflag);
+ return;
+ }
+ ctxp->flag |= LPFC_NVMET_CTX_RLS;
+ list_add_tail(&ctxp->list, &phba->sli4_hba.lpfc_abts_nvmet_ctx_list);
+ spin_unlock_irqrestore(&phba->sli4_hba.abts_nvme_buf_list_lock, iflag);
+}
+
/**
* lpfc_nvmet_xmt_ls_rsp_cmp - Completion handler for LS Response
* @phba: Pointer to HBA context object.
@@ -139,6 +159,11 @@
struct lpfc_dmabuf *mp)
{
if (ctxp) {
+ if (ctxp->flag)
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6314 rq_post ctx xri x%x flag x%x\n",
+ ctxp->oxid, ctxp->flag);
+
if (ctxp->txrdy) {
pci_pool_free(phba->txrdy_payload_pool, ctxp->txrdy,
ctxp->txrdy_phys);
@@ -337,39 +362,55 @@
#endif
ctxp = cmdwqe->context2;
+ ctxp->flag &= ~LPFC_NVMET_IO_INP;
+
rsp = &ctxp->ctx.fcp_req;
op = rsp->op;
- ctxp->flag &= ~LPFC_NVMET_IO_INP;
status = bf_get(lpfc_wcqe_c_status, wcqe);
result = wcqe->parameter;
- if (!phba->targetport)
- goto out;
+ if (phba->targetport)
+ tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
+ else
+ tgtp = NULL;
lpfc_nvmeio_data(phba, "NVMET FCP CMPL: xri x%x op x%x status x%x\n",
ctxp->oxid, op, status);
- tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
if (status) {
rsp->fcp_error = NVME_SC_DATA_XFER_ERROR;
rsp->transferred_length = 0;
- atomic_inc(&tgtp->xmt_fcp_rsp_error);
+ if (tgtp)
+ atomic_inc(&tgtp->xmt_fcp_rsp_error);
+
+ /* pick up SLI4 exhange busy condition */
+ if (bf_get(lpfc_wcqe_c_xb, wcqe)) {
+ ctxp->flag |= LPFC_NVMET_XBUSY;
+
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6315 IO Cmpl XBUSY: xri x%x: %x/%x\n",
+ ctxp->oxid, status, result);
+ } else {
+ ctxp->flag &= ~LPFC_NVMET_XBUSY;
+ }
+
} else {
rsp->fcp_error = NVME_SC_SUCCESS;
if (op == NVMET_FCOP_RSP)
rsp->transferred_length = rsp->rsplen;
else
rsp->transferred_length = rsp->transfer_length;
- atomic_inc(&tgtp->xmt_fcp_rsp_cmpl);
+ if (tgtp)
+ atomic_inc(&tgtp->xmt_fcp_rsp_cmpl);
}
-out:
if ((op == NVMET_FCOP_READDATA_RSP) ||
(op == NVMET_FCOP_RSP)) {
/* Sanity check */
ctxp->state = LPFC_NVMET_STE_DONE;
ctxp->entry_cnt++;
+
#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
if (phba->ktime_on) {
if (rsp->op == NVMET_FCOP_READDATA_RSP) {
@@ -408,9 +449,7 @@
if (phba->ktime_on)
lpfc_nvmet_ktime(phba, ctxp);
#endif
- /* Let Abort cmpl repost the context */
- if (!(ctxp->flag & LPFC_NVMET_ABORT_OP))
- lpfc_nvmet_rq_post(phba, ctxp, &ctxp->rqb_buffer->hbuf);
+ /* lpfc_nvmet_xmt_fcp_release() will recycle the context */
} else {
ctxp->entry_cnt++;
start_clean = offsetof(struct lpfc_iocbq, wqe);
@@ -519,7 +558,6 @@
container_of(rsp, struct lpfc_nvmet_rcv_ctx, ctx.fcp_req);
struct lpfc_hba *phba = ctxp->phba;
struct lpfc_iocbq *nvmewqeq;
- unsigned long iflags;
int rc;
#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
@@ -544,32 +582,12 @@
}
#endif
- if (rsp->op == NVMET_FCOP_ABORT) {
- lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
- "6103 Abort op: oxri x%x %d cnt %d\n",
- ctxp->oxid, ctxp->state, ctxp->entry_cnt);
-
- lpfc_nvmeio_data(phba, "NVMET FCP ABRT: "
- "xri x%x state x%x cnt x%x\n",
- ctxp->oxid, ctxp->state, ctxp->entry_cnt);
-
- atomic_inc(&lpfc_nvmep->xmt_fcp_abort);
- ctxp->entry_cnt++;
- ctxp->flag |= LPFC_NVMET_ABORT_OP;
- if (ctxp->flag & LPFC_NVMET_IO_INP)
- lpfc_nvmet_sol_fcp_issue_abort(phba, ctxp, ctxp->sid,
- ctxp->oxid);
- else
- lpfc_nvmet_unsol_fcp_issue_abort(phba, ctxp, ctxp->sid,
- ctxp->oxid);
- return 0;
- }
-
/* Sanity check */
- if (ctxp->state == LPFC_NVMET_STE_ABORT) {
+ if ((ctxp->flag & LPFC_NVMET_ABTS_RCV) ||
+ (ctxp->state == LPFC_NVMET_STE_ABORT)) {
atomic_inc(&lpfc_nvmep->xmt_fcp_drop);
lpfc_printf_log(phba, KERN_ERR, LOG_NVME_IOERR,
- "6102 Bad state IO x%x aborted\n",
+ "6102 IO xri x%x aborted\n",
ctxp->oxid);
rc = -ENXIO;
goto aerr;
@@ -594,10 +612,7 @@
lpfc_nvmeio_data(phba, "NVMET FCP CMND: xri x%x op x%x len x%x\n",
ctxp->oxid, rsp->op, rsp->rsplen);
- /* For now we take hbalock */
- spin_lock_irqsave(&phba->hbalock, iflags);
rc = lpfc_sli4_issue_wqe(phba, LPFC_FCP_RING, nvmewqeq);
- spin_unlock_irqrestore(&phba->hbalock, iflags);
if (rc == WQE_SUCCESS) {
ctxp->flag |= LPFC_NVMET_IO_INP;
#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
@@ -634,10 +649,79 @@
complete(&tport->tport_unreg_done);
}
+static void
+lpfc_nvmet_xmt_fcp_abort(struct nvmet_fc_target_port *tgtport,
+ struct nvmefc_tgt_fcp_req *req)
+{
+ struct lpfc_nvmet_tgtport *lpfc_nvmep = tgtport->private;
+ struct lpfc_nvmet_rcv_ctx *ctxp =
+ container_of(req, struct lpfc_nvmet_rcv_ctx, ctx.fcp_req);
+ struct lpfc_hba *phba = ctxp->phba;
+ unsigned long flags;
+
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6103 Abort op: oxri x%x flg x%x cnt %d\n",
+ ctxp->oxid, ctxp->flag, ctxp->entry_cnt);
+
+ lpfc_nvmeio_data(phba, "NVMET FCP ABRT: "
+ "xri x%x flg x%x cnt x%x\n",
+ ctxp->oxid, ctxp->flag, ctxp->entry_cnt);
+
+ atomic_inc(&lpfc_nvmep->xmt_fcp_abort);
+ ctxp->entry_cnt++;
+ spin_lock_irqsave(&ctxp->ctxlock, flags);
+
+ /* Since iaab/iaar are NOT set, we need to check
+ * if the firmware is in process of aborting IO
+ */
+ if (ctxp->flag & LPFC_NVMET_XBUSY) {
+ spin_unlock_irqrestore(&ctxp->ctxlock, flags);
+ return;
+ }
+ ctxp->flag |= LPFC_NVMET_ABORT_OP;
+ if (ctxp->flag & LPFC_NVMET_IO_INP)
+ lpfc_nvmet_sol_fcp_issue_abort(phba, ctxp, ctxp->sid,
+ ctxp->oxid);
+ else
+ lpfc_nvmet_unsol_fcp_issue_abort(phba, ctxp, ctxp->sid,
+ ctxp->oxid);
+ spin_unlock_irqrestore(&ctxp->ctxlock, flags);
+}
+
+static void
+lpfc_nvmet_xmt_fcp_release(struct nvmet_fc_target_port *tgtport,
+ struct nvmefc_tgt_fcp_req *rsp)
+{
+ struct lpfc_nvmet_rcv_ctx *ctxp =
+ container_of(rsp, struct lpfc_nvmet_rcv_ctx, ctx.fcp_req);
+ struct lpfc_hba *phba = ctxp->phba;
+ unsigned long flags;
+ bool aborting = false;
+
+ spin_lock_irqsave(&ctxp->ctxlock, flags);
+ if ((ctxp->flag & LPFC_NVMET_ABORT_OP) ||
+ (ctxp->flag & LPFC_NVMET_XBUSY)) {
+ aborting = true;
+ /* let the abort path do the real release */
+ lpfc_nvmet_defer_release(phba, ctxp);
+ }
+ spin_unlock_irqrestore(&ctxp->ctxlock, flags);
+
+ lpfc_nvmeio_data(phba, "NVMET FCP FREE: xri x%x ste %d\n", ctxp->oxid,
+ ctxp->state, 0);
+
+ if (aborting)
+ return;
+
+ lpfc_nvmet_rq_post(phba, ctxp, &ctxp->rqb_buffer->hbuf);
+}
+
static struct nvmet_fc_target_template lpfc_tgttemplate = {
.targetport_delete = lpfc_nvmet_targetport_delete,
.xmt_ls_rsp = lpfc_nvmet_xmt_ls_rsp,
.fcp_op = lpfc_nvmet_xmt_fcp_op,
+ .fcp_abort = lpfc_nvmet_xmt_fcp_abort,
+ .fcp_req_release = lpfc_nvmet_xmt_fcp_release,
.max_hw_queues = 1,
.max_sgl_segments = LPFC_NVMET_DEFAULT_SEGS,
@@ -666,10 +750,23 @@
pinfo.port_name = wwn_to_u64(vport->fc_portname.u.wwn);
pinfo.port_id = vport->fc_myDID;
+ /* Limit to LPFC_MAX_NVME_SEG_CNT.
+ * For now need + 1 to get around NVME transport logic.
+ */
+ if (phba->cfg_sg_seg_cnt > LPFC_MAX_NVME_SEG_CNT) {
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME | LOG_INIT,
+ "6400 Reducing sg segment cnt to %d\n",
+ LPFC_MAX_NVME_SEG_CNT);
+ phba->cfg_nvme_seg_cnt = LPFC_MAX_NVME_SEG_CNT;
+ } else {
+ phba->cfg_nvme_seg_cnt = phba->cfg_sg_seg_cnt;
+ }
+ lpfc_tgttemplate.max_sgl_segments = phba->cfg_nvme_seg_cnt + 1;
lpfc_tgttemplate.max_hw_queues = phba->cfg_nvme_io_channel;
- lpfc_tgttemplate.max_sgl_segments = phba->cfg_sg_seg_cnt;
lpfc_tgttemplate.target_features = NVMET_FCTGTFEAT_READDATA_RSP |
- NVMET_FCTGTFEAT_NEEDS_CMD_CPUSCHED;
+ NVMET_FCTGTFEAT_NEEDS_CMD_CPUSCHED |
+ NVMET_FCTGTFEAT_CMD_IN_ISR |
+ NVMET_FCTGTFEAT_OPDONE_IN_ISR;
#if (IS_ENABLED(CONFIG_NVME_TARGET_FC))
error = nvmet_fc_register_targetport(&pinfo, &lpfc_tgttemplate,
@@ -750,7 +847,120 @@
lpfc_sli4_nvmet_xri_aborted(struct lpfc_hba *phba,
struct sli4_wcqe_xri_aborted *axri)
{
- /* TODO: work in progress */
+ uint16_t xri = bf_get(lpfc_wcqe_xa_xri, axri);
+ uint16_t rxid = bf_get(lpfc_wcqe_xa_remote_xid, axri);
+ struct lpfc_nvmet_rcv_ctx *ctxp, *next_ctxp;
+ struct lpfc_nodelist *ndlp;
+ unsigned long iflag = 0;
+ int rrq_empty = 0;
+ bool released = false;
+
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6317 XB aborted xri x%x rxid x%x\n", xri, rxid);
+
+ if (!(phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME))
+ return;
+ spin_lock_irqsave(&phba->hbalock, iflag);
+ spin_lock(&phba->sli4_hba.abts_nvme_buf_list_lock);
+ list_for_each_entry_safe(ctxp, next_ctxp,
+ &phba->sli4_hba.lpfc_abts_nvmet_ctx_list,
+ list) {
+ if (ctxp->rqb_buffer->sglq->sli4_xritag != xri)
+ continue;
+
+ /* Check if we already received a free context call
+ * and we have completed processing an abort situation.
+ */
+ if (ctxp->flag & LPFC_NVMET_CTX_RLS &&
+ !(ctxp->flag & LPFC_NVMET_ABORT_OP)) {
+ list_del(&ctxp->list);
+ released = true;
+ }
+ ctxp->flag &= ~LPFC_NVMET_XBUSY;
+ spin_unlock(&phba->sli4_hba.abts_nvme_buf_list_lock);
+
+ rrq_empty = list_empty(&phba->active_rrq_list);
+ spin_unlock_irqrestore(&phba->hbalock, iflag);
+ ndlp = lpfc_findnode_did(phba->pport, ctxp->sid);
+ if (ndlp && NLP_CHK_NODE_ACT(ndlp) &&
+ (ndlp->nlp_state == NLP_STE_UNMAPPED_NODE ||
+ ndlp->nlp_state == NLP_STE_MAPPED_NODE)) {
+ lpfc_set_rrq_active(phba, ndlp,
+ ctxp->rqb_buffer->sglq->sli4_lxritag,
+ rxid, 1);
+ lpfc_sli4_abts_err_handler(phba, ndlp, axri);
+ }
+
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6318 XB aborted %x flg x%x (%x)\n",
+ ctxp->oxid, ctxp->flag, released);
+ if (released)
+ lpfc_nvmet_rq_post(phba, ctxp,
+ &ctxp->rqb_buffer->hbuf);
+ if (rrq_empty)
+ lpfc_worker_wake_up(phba);
+ return;
+ }
+ spin_unlock(&phba->sli4_hba.abts_nvme_buf_list_lock);
+ spin_unlock_irqrestore(&phba->hbalock, iflag);
+}
+
+int
+lpfc_nvmet_rcv_unsol_abort(struct lpfc_vport *vport,
+ struct fc_frame_header *fc_hdr)
+
+{
+#if (IS_ENABLED(CONFIG_NVME_TARGET_FC))
+ struct lpfc_hba *phba = vport->phba;
+ struct lpfc_nvmet_rcv_ctx *ctxp, *next_ctxp;
+ struct nvmefc_tgt_fcp_req *rsp;
+ uint16_t xri;
+ unsigned long iflag = 0;
+
+ xri = be16_to_cpu(fc_hdr->fh_ox_id);
+
+ spin_lock_irqsave(&phba->hbalock, iflag);
+ spin_lock(&phba->sli4_hba.abts_nvme_buf_list_lock);
+ list_for_each_entry_safe(ctxp, next_ctxp,
+ &phba->sli4_hba.lpfc_abts_nvmet_ctx_list,
+ list) {
+ if (ctxp->rqb_buffer->sglq->sli4_xritag != xri)
+ continue;
+
+ spin_unlock(&phba->sli4_hba.abts_nvme_buf_list_lock);
+ spin_unlock_irqrestore(&phba->hbalock, iflag);
+
+ spin_lock_irqsave(&ctxp->ctxlock, iflag);
+ ctxp->flag |= LPFC_NVMET_ABTS_RCV;
+ spin_unlock_irqrestore(&ctxp->ctxlock, iflag);
+
+ lpfc_nvmeio_data(phba,
+ "NVMET ABTS RCV: xri x%x CPU %02x rjt %d\n",
+ xri, smp_processor_id(), 0);
+
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6319 NVMET Rcv ABTS:acc xri x%x\n", xri);
+
+ rsp = &ctxp->ctx.fcp_req;
+ nvmet_fc_rcv_fcp_abort(phba->targetport, rsp);
+
+ /* Respond with BA_ACC accordingly */
+ lpfc_sli4_seq_abort_rsp(vport, fc_hdr, 1);
+ return 0;
+ }
+ spin_unlock(&phba->sli4_hba.abts_nvme_buf_list_lock);
+ spin_unlock_irqrestore(&phba->hbalock, iflag);
+
+ lpfc_nvmeio_data(phba, "NVMET ABTS RCV: xri x%x CPU %02x rjt %d\n",
+ xri, smp_processor_id(), 1);
+
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6320 NVMET Rcv ABTS:rjt xri x%x\n", xri);
+
+ /* Respond with BA_RJT accordingly */
+ lpfc_sli4_seq_abort_rsp(vport, fc_hdr, 0);
+#endif
+ return 0;
}
void
@@ -940,6 +1150,7 @@
ctxp->rqb_buffer = nvmebuf;
ctxp->entry_cnt = 1;
ctxp->flag = 0;
+ spin_lock_init(&ctxp->ctxlock);
#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
if (phba->ktime_on) {
@@ -962,8 +1173,8 @@
}
#endif
- lpfc_nvmeio_data(phba, "NVMET FCP RCV: xri x%x sz %d from %06x\n",
- oxid, size, sid);
+ lpfc_nvmeio_data(phba, "NVMET FCP RCV: xri x%x sz %d CPU %02x\n",
+ oxid, size, smp_processor_id());
atomic_inc(&tgtp->rcv_fcp_cmd_in);
/*
@@ -1237,11 +1448,11 @@
return NULL;
}
- if (rsp->sg_cnt > phba->cfg_sg_seg_cnt) {
+ if (rsp->sg_cnt > phba->cfg_nvme_seg_cnt) {
lpfc_printf_log(phba, KERN_ERR, LOG_NVME_IOERR,
"6109 lpfc_nvmet_prep_fcp_wqe: seg cnt err: "
- "NPORT x%x oxid:x%x\n",
- ctxp->sid, ctxp->oxid);
+ "NPORT x%x oxid:x%x cnt %d\n",
+ ctxp->sid, ctxp->oxid, phba->cfg_nvme_seg_cnt);
return NULL;
}
@@ -1593,6 +1804,8 @@
struct lpfc_nvmet_rcv_ctx *ctxp;
struct lpfc_nvmet_tgtport *tgtp;
uint32_t status, result;
+ unsigned long flags;
+ bool released = false;
ctxp = cmdwqe->context2;
status = bf_get(lpfc_wcqe_c_status, wcqe);
@@ -1601,21 +1814,46 @@
tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
atomic_inc(&tgtp->xmt_abort_cmpl);
+ ctxp->state = LPFC_NVMET_STE_DONE;
+
+ /* Check if we already received a free context call
+ * and we have completed processing an abort situation.
+ */
+ spin_lock_irqsave(&ctxp->ctxlock, flags);
+ if ((ctxp->flag & LPFC_NVMET_CTX_RLS) &&
+ !(ctxp->flag & LPFC_NVMET_XBUSY)) {
+ list_del(&ctxp->list);
+ released = true;
+ }
+ ctxp->flag &= ~LPFC_NVMET_ABORT_OP;
+ spin_unlock_irqrestore(&ctxp->ctxlock, flags);
+
lpfc_printf_log(phba, KERN_ERR, LOG_NVME_ABTS,
- "6165 Abort cmpl: xri x%x WCQE: %08x %08x %08x %08x\n",
- ctxp->oxid, wcqe->word0, wcqe->total_data_placed,
+ "6165 ABORT cmpl: xri x%x flg x%x (%d) "
+ "WCQE: %08x %08x %08x %08x\n",
+ ctxp->oxid, ctxp->flag, released,
+ wcqe->word0, wcqe->total_data_placed,
result, wcqe->word3);
- ctxp->state = LPFC_NVMET_STE_DONE;
- lpfc_nvmet_rq_post(phba, ctxp, &ctxp->rqb_buffer->hbuf);
+ /*
+ * if transport has released ctx, then can reuse it. Otherwise,
+ * will be recycled by transport release call.
+ */
+ if (released)
+ lpfc_nvmet_rq_post(phba, ctxp, &ctxp->rqb_buffer->hbuf);
cmdwqe->context2 = NULL;
cmdwqe->context3 = NULL;
lpfc_sli_release_iocbq(phba, cmdwqe);
+
+ /* Since iaab/iaar are NOT set, there is no work left.
+ * For LPFC_NVMET_XBUSY, lpfc_sli4_nvmet_xri_aborted
+ * should have been called already.
+ */
}
/**
- * lpfc_nvmet_xmt_fcp_abort_cmp - Completion handler for ABTS
+ * lpfc_nvmet_unsol_fcp_abort_cmp - Completion handler for ABTS
* @phba: Pointer to HBA context object.
* @cmdwqe: Pointer to driver command WQE object.
* @wcqe: Pointer to driver response CQE object.
@@ -1625,12 +1863,14 @@
* The function frees memory resources used for the NVME commands.
**/
static void
-lpfc_nvmet_xmt_fcp_abort_cmp(struct lpfc_hba *phba, struct lpfc_iocbq *cmdwqe,
- struct lpfc_wcqe_complete *wcqe)
+lpfc_nvmet_unsol_fcp_abort_cmp(struct lpfc_hba *phba, struct lpfc_iocbq *cmdwqe,
+ struct lpfc_wcqe_complete *wcqe)
{
struct lpfc_nvmet_rcv_ctx *ctxp;
struct lpfc_nvmet_tgtport *tgtp;
+ unsigned long flags;
uint32_t status, result;
+ bool released = false;
ctxp = cmdwqe->context2;
status = bf_get(lpfc_wcqe_c_status, wcqe);
@@ -1639,23 +1879,55 @@
tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
atomic_inc(&tgtp->xmt_abort_cmpl);
- lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
- "6070 Abort cmpl: ctx %p WCQE: %08x %08x %08x %08x\n",
- ctxp, wcqe->word0, wcqe->total_data_placed,
- result, wcqe->word3);
-
- if (ctxp) {
- /* Sanity check */
- if (ctxp->state != LPFC_NVMET_STE_ABORT) {
- lpfc_printf_log(phba, KERN_ERR, LOG_NVME_ABTS,
- "6112 ABORT Wrong state:%d oxid x%x\n",
- ctxp->state, ctxp->oxid);
- }
- ctxp->state = LPFC_NVMET_STE_DONE;
- lpfc_nvmet_rq_post(phba, ctxp, &ctxp->rqb_buffer->hbuf);
- cmdwqe->context2 = NULL;
- cmdwqe->context3 = NULL;
+ if (!ctxp) {
+ /* if context is clear, related io alrady complete */
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6070 ABTS cmpl: WCQE: %08x %08x %08x %08x\n",
+ wcqe->word0, wcqe->total_data_placed,
+ result, wcqe->word3);
+ return;
}
+
+ /* Sanity check */
+ if (ctxp->state != LPFC_NVMET_STE_ABORT) {
+ lpfc_printf_log(phba, KERN_ERR, LOG_NVME_ABTS,
+ "6112 ABTS Wrong state:%d oxid x%x\n",
+ ctxp->state, ctxp->oxid);
+ }
+
+ /* Check if we already received a free context call
+ * and we have completed processing an abort situation.
+ */
+ ctxp->state = LPFC_NVMET_STE_DONE;
+ spin_lock_irqsave(&ctxp->ctxlock, flags);
+ if ((ctxp->flag & LPFC_NVMET_CTX_RLS) &&
+ !(ctxp->flag & LPFC_NVMET_XBUSY)) {
+ list_del(&ctxp->list);
+ released = true;
+ }
+ ctxp->flag &= ~LPFC_NVMET_ABORT_OP;
+ spin_unlock_irqrestore(&ctxp->ctxlock, flags);
+
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6316 ABTS cmpl xri x%x flg x%x (%x) "
+ "WCQE: %08x %08x %08x %08x\n",
+ ctxp->oxid, ctxp->flag, released,
+ wcqe->word0, wcqe->total_data_placed,
+ result, wcqe->word3);
+ /*
+ * if transport has released ctx, then can reuse it. Otherwise,
+ * will be recycled by transport release call.
+ */
+ if (released)
+ lpfc_nvmet_rq_post(phba, ctxp, &ctxp->rqb_buffer->hbuf);
+
+ cmdwqe->context2 = NULL;
+ cmdwqe->context3 = NULL;
+
+ /* Since iaab/iaar are NOT set, there is no work left.
+ * For LPFC_NVMET_XBUSY, lpfc_sli4_nvmet_xri_aborted
+ * should have been called already.
+ */
}
/**
@@ -1708,10 +1980,14 @@
struct lpfc_nodelist *ndlp;
lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
- "6067 Abort: sid %x xri x%x/x%x\n",
+ "6067 ABTS: sid %x xri x%x/x%x\n",
sid, xri, ctxp->wqeq->sli4_xritag);
tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
+ if (!ctxp->wqeq) {
+ ctxp->wqeq = ctxp->rqb_buffer->iocbq;
+ ctxp->wqeq->hba_wqidx = 0;
+ }
ndlp = lpfc_findnode_did(phba->pport, sid);
if (!ndlp || !NLP_CHK_NODE_ACT(ndlp) ||
@@ -1817,10 +2093,11 @@
(ndlp->nlp_state != NLP_STE_MAPPED_NODE))) {
atomic_inc(&tgtp->xmt_abort_rsp_error);
lpfc_printf_log(phba, KERN_WARNING, LOG_NVME_ABTS,
- "6160 Drop ABTS - wrong NDLP state x%x.\n",
+ "6160 Drop ABORT - wrong NDLP state x%x.\n",
(ndlp) ? ndlp->nlp_state : NLP_STE_MAX_STATE);
/* No failure to an ABTS request. */
+ ctxp->flag &= ~LPFC_NVMET_ABORT_OP;
return 0;
}
@@ -1828,9 +2105,10 @@
ctxp->abort_wqeq = lpfc_sli_get_iocbq(phba);
if (!ctxp->abort_wqeq) {
lpfc_printf_log(phba, KERN_WARNING, LOG_NVME_ABTS,
- "6161 Abort failed: No wqeqs: "
+ "6161 ABORT failed: No wqeqs: "
"xri: x%x\n", ctxp->oxid);
/* No failure to an ABTS request. */
+ ctxp->flag &= ~LPFC_NVMET_ABORT_OP;
return 0;
}
abts_wqeq = ctxp->abort_wqeq;
@@ -1838,8 +2116,8 @@
ctxp->state = LPFC_NVMET_STE_ABORT;
/* Announce entry to new IO submit field. */
- lpfc_printf_log(phba, KERN_ERR, LOG_NVME_ABTS,
- "6162 Abort Request to rport DID x%06x "
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6162 ABORT Request to rport DID x%06x "
"for xri x%x x%x\n",
ctxp->sid, ctxp->oxid, ctxp->wqeq->sli4_xritag);
@@ -1855,6 +2133,7 @@
"NVME Req now. hba_flag x%x oxid x%x\n",
phba->hba_flag, ctxp->oxid);
lpfc_sli_release_iocbq(phba, abts_wqeq);
+ ctxp->flag &= ~LPFC_NVMET_ABORT_OP;
return 0;
}
@@ -1866,6 +2145,7 @@
"still pending on oxid x%x\n",
ctxp->oxid);
lpfc_sli_release_iocbq(phba, abts_wqeq);
+ ctxp->flag &= ~LPFC_NVMET_ABORT_OP;
return 0;
}
@@ -1913,9 +2193,10 @@
if (rc == WQE_SUCCESS)
return 0;
+ ctxp->flag &= ~LPFC_NVMET_ABORT_OP;
lpfc_sli_release_iocbq(phba, abts_wqeq);
- lpfc_printf_log(phba, KERN_ERR, LOG_NVME,
- "6166 Failed abts issue_wqe with status x%x "
+ lpfc_printf_log(phba, KERN_ERR, LOG_NVME_ABTS,
+ "6166 Failed ABORT issue_wqe with status x%x "
"for oxid x%x.\n",
rc, ctxp->oxid);
return 1;
@@ -1944,8 +2225,8 @@
spin_lock_irqsave(&phba->hbalock, flags);
abts_wqeq = ctxp->wqeq;
- abts_wqeq->wqe_cmpl = lpfc_nvmet_xmt_fcp_abort_cmp;
- abts_wqeq->iocb_cmpl = 0;
+ abts_wqeq->wqe_cmpl = lpfc_nvmet_unsol_fcp_abort_cmp;
+ abts_wqeq->iocb_cmpl = NULL;
abts_wqeq->iocb_flag |= LPFC_IO_NVMET;
rc = lpfc_sli4_issue_wqe(phba, LPFC_FCP_RING, abts_wqeq);
spin_unlock_irqrestore(&phba->hbalock, flags);
@@ -1955,7 +2236,7 @@
}
aerr:
- lpfc_nvmet_rq_post(phba, ctxp, &ctxp->rqb_buffer->hbuf);
+ ctxp->flag &= ~LPFC_NVMET_ABORT_OP;
atomic_inc(&tgtp->xmt_abort_rsp_error);
lpfc_printf_log(phba, KERN_WARNING, LOG_NVME_ABTS,
"6135 Failed to Issue ABTS for oxid x%x. Status x%x\n",
diff --git a/drivers/scsi/lpfc/lpfc_nvmet.h b/drivers/scsi/lpfc/lpfc_nvmet.h
index ca96f05..128759f 100644
--- a/drivers/scsi/lpfc/lpfc_nvmet.h
+++ b/drivers/scsi/lpfc/lpfc_nvmet.h
@@ -21,9 +21,7 @@
* included with this package. *
********************************************************************/
-#define LPFC_NVMET_MIN_SEGS 16
-#define LPFC_NVMET_DEFAULT_SEGS 64 /* 256K IOs */
-#define LPFC_NVMET_MAX_SEGS 510
+#define LPFC_NVMET_DEFAULT_SEGS (64 + 1) /* 256K IOs */
#define LPFC_NVMET_SUCCESS_LEN 12
/* Used for NVME Target */
@@ -77,10 +75,12 @@
struct nvmefc_tgt_ls_req ls_req;
struct nvmefc_tgt_fcp_req fcp_req;
} ctx;
+ struct list_head list;
struct lpfc_hba *phba;
struct lpfc_iocbq *wqeq;
struct lpfc_iocbq *abort_wqeq;
dma_addr_t txrdy_phys;
+ spinlock_t ctxlock; /* protect flag access */
uint32_t *txrdy;
uint32_t sid;
uint32_t offset;
@@ -97,8 +97,11 @@
#define LPFC_NVMET_STE_RSP 4
#define LPFC_NVMET_STE_DONE 5
uint16_t flag;
-#define LPFC_NVMET_IO_INP 1
-#define LPFC_NVMET_ABORT_OP 2
+#define LPFC_NVMET_IO_INP 0x1 /* IO is in progress on exchange */
+#define LPFC_NVMET_ABORT_OP 0x2 /* Abort WQE issued on exchange */
+#define LPFC_NVMET_XBUSY 0x4 /* XB bit set on IO cmpl */
+#define LPFC_NVMET_CTX_RLS 0x8 /* ctx free requested */
+#define LPFC_NVMET_ABTS_RCV 0x10 /* ABTS received on exchange */
struct rqb_dmabuf *rqb_buffer;
#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
diff --git a/drivers/scsi/lpfc/lpfc_sli.c b/drivers/scsi/lpfc/lpfc_sli.c
index 1c9fa45..cf19f49 100644
--- a/drivers/scsi/lpfc/lpfc_sli.c
+++ b/drivers/scsi/lpfc/lpfc_sli.c
@@ -6338,7 +6338,7 @@
}
/**
- * lpfc_sli4_repost_sgl_list - Repsot the buffers sgl pages as block
+ * lpfc_sli4_repost_sgl_list - Repost the buffers sgl pages as block
* @phba: pointer to lpfc hba data structure.
* @pring: Pointer to driver SLI ring object.
* @sgl_list: linked link of sgl buffers to post
@@ -13758,7 +13758,10 @@
lpfc_free_rq_buffer(queue->phba, queue);
kfree(queue->rqbp);
}
- kfree(queue->pring);
+
+ if (!list_empty(&queue->wq_list))
+ list_del(&queue->wq_list);
+
kfree(queue);
return;
}
@@ -14738,6 +14741,9 @@
case LPFC_Q_CREATE_VERSION_1:
bf_set(lpfc_mbx_wq_create_wqe_count, &wq_create->u.request_1,
wq->entry_count);
+ bf_set(lpfc_mbox_hdr_version, &shdr->request,
+ LPFC_Q_CREATE_VERSION_1);
+
switch (wq->entry_size) {
default:
case 64:
@@ -15561,6 +15567,8 @@
}
/* Remove wq from any list */
list_del_init(&wq->list);
+ kfree(wq->pring);
+ wq->pring = NULL;
mempool_free(mbox, wq->phba->mbox_mem_pool);
return status;
}
@@ -16513,7 +16521,7 @@
* This function sends a basic response to a previous unsol sequence abort
* event after aborting the sequence handling.
**/
-static void
+void
lpfc_sli4_seq_abort_rsp(struct lpfc_vport *vport,
struct fc_frame_header *fc_hdr, bool aborted)
{
@@ -16534,14 +16542,13 @@
ndlp = lpfc_findnode_did(vport, sid);
if (!ndlp) {
- ndlp = mempool_alloc(phba->nlp_mem_pool, GFP_KERNEL);
+ ndlp = lpfc_nlp_init(vport, sid);
if (!ndlp) {
lpfc_printf_vlog(vport, KERN_WARNING, LOG_ELS,
"1268 Failed to allocate ndlp for "
"oxid:x%x SID:x%x\n", oxid, sid);
return;
}
- lpfc_nlp_init(vport, ndlp, sid);
/* Put ndlp onto pport node list */
lpfc_enqueue_node(vport, ndlp);
} else if (!NLP_CHK_NODE_ACT(ndlp)) {
@@ -16690,6 +16697,11 @@
}
lpfc_in_buf_free(phba, &dmabuf->dbuf);
+ if (phba->nvmet_support) {
+ lpfc_nvmet_rcv_unsol_abort(vport, &fc_hdr);
+ return;
+ }
+
/* Respond with BA_ACC or BA_RJT accordingly */
lpfc_sli4_seq_abort_rsp(vport, &fc_hdr, aborted);
}
diff --git a/drivers/scsi/lpfc/lpfc_sli4.h b/drivers/scsi/lpfc/lpfc_sli4.h
index 710458c..da46471 100644
--- a/drivers/scsi/lpfc/lpfc_sli4.h
+++ b/drivers/scsi/lpfc/lpfc_sli4.h
@@ -620,7 +620,7 @@
struct list_head lpfc_els_sgl_list;
struct list_head lpfc_abts_els_sgl_list;
struct list_head lpfc_nvmet_sgl_list;
- struct list_head lpfc_abts_nvmet_sgl_list;
+ struct list_head lpfc_abts_nvmet_ctx_list;
struct list_head lpfc_abts_scsi_buf_list;
struct list_head lpfc_abts_nvme_buf_list;
struct lpfc_sglq **lpfc_sglq_active_list;
diff --git a/drivers/scsi/lpfc/lpfc_version.h b/drivers/scsi/lpfc/lpfc_version.h
index d4e95e2..1c26dc6 100644
--- a/drivers/scsi/lpfc/lpfc_version.h
+++ b/drivers/scsi/lpfc/lpfc_version.h
@@ -20,7 +20,7 @@
* included with this package. *
*******************************************************************/
-#define LPFC_DRIVER_VERSION "11.2.0.10"
+#define LPFC_DRIVER_VERSION "11.2.0.12"
#define LPFC_DRIVER_NAME "lpfc"
/* Used for SLI 2/3 */
diff --git a/drivers/scsi/lpfc/lpfc_vport.c b/drivers/scsi/lpfc/lpfc_vport.c
index 9a0339d..c714482 100644
--- a/drivers/scsi/lpfc/lpfc_vport.c
+++ b/drivers/scsi/lpfc/lpfc_vport.c
@@ -738,10 +738,9 @@
ndlp = lpfc_findnode_did(vport, Fabric_DID);
if (!ndlp) {
/* Cannot find existing Fabric ndlp, allocate one */
- ndlp = mempool_alloc(phba->nlp_mem_pool, GFP_KERNEL);
+ ndlp = lpfc_nlp_init(vport, Fabric_DID);
if (!ndlp)
goto skip_logo;
- lpfc_nlp_init(vport, ndlp, Fabric_DID);
/* Indicate free memory when release */
NLP_SET_FREE_REQ(ndlp);
} else {
diff --git a/drivers/scsi/mac_esp.c b/drivers/scsi/mac_esp.c
index 14c0334..bb567d3 100644
--- a/drivers/scsi/mac_esp.c
+++ b/drivers/scsi/mac_esp.c
@@ -631,7 +631,7 @@
}
MODULE_DESCRIPTION("Mac ESP SCSI driver");
-MODULE_AUTHOR("Finn Thain <fthain@telegraphics.com.au>");
+MODULE_AUTHOR("Finn Thain");
MODULE_LICENSE("GPL v2");
MODULE_VERSION(DRV_VERSION);
MODULE_ALIAS("platform:" DRV_MODULE_NAME);
diff --git a/drivers/scsi/osd/osd_initiator.c b/drivers/scsi/osd/osd_initiator.c
index 6903f03..8a1b948 100644
--- a/drivers/scsi/osd/osd_initiator.c
+++ b/drivers/scsi/osd/osd_initiator.c
@@ -477,7 +477,7 @@
int error)
{
or->async_error = error;
- or->req_errors = req->errors ? : error;
+ or->req_errors = scsi_req(req)->result ? : error;
or->sense_len = scsi_req(req)->sense_len;
if (or->sense_len)
memcpy(or->sense, scsi_req(req)->sense, or->sense_len);
@@ -489,7 +489,10 @@
int osd_execute_request(struct osd_request *or)
{
- int error = blk_execute_rq(or->request->q, NULL, or->request, 0);
+ int error;
+
+ blk_execute_rq(or->request->q, NULL, or->request, 0);
+ error = scsi_req(or->request)->result ? -EIO : 0;
_set_error_resid(or, or->request, error);
return error;
@@ -1602,7 +1605,7 @@
req->rq_flags |= RQF_QUIET;
req->timeout = or->timeout;
- req->retries = or->retries;
+ scsi_req(req)->retries = or->retries;
if (has_out) {
or->out.req = req;
diff --git a/drivers/scsi/osst.c b/drivers/scsi/osst.c
index c47f4b3..67cbed9 100644
--- a/drivers/scsi/osst.c
+++ b/drivers/scsi/osst.c
@@ -327,7 +327,7 @@
struct osst_tape *STp = SRpnt->stp;
struct rq_map_data *mdata = &SRpnt->stp->buffer->map_data;
- STp->buffer->cmdstat.midlevel_result = SRpnt->result = req->errors;
+ STp->buffer->cmdstat.midlevel_result = SRpnt->result = rq->result;
#if DEBUG
STp->write_pending = 0;
#endif
@@ -414,7 +414,7 @@
memset(rq->cmd, 0, BLK_MAX_CDB); /* ATAPI hates garbage after CDB */
memcpy(rq->cmd, cmd, rq->cmd_len);
req->timeout = timeout;
- req->retries = retries;
+ rq->retries = retries;
req->end_io_data = SRpnt;
blk_execute_rq_nowait(req->q, NULL, req, 1, osst_end_async);
diff --git a/drivers/scsi/qedf/Makefile b/drivers/scsi/qedf/Makefile
index 64e9f50..414f2a7 100644
--- a/drivers/scsi/qedf/Makefile
+++ b/drivers/scsi/qedf/Makefile
@@ -1,5 +1,5 @@
obj-$(CONFIG_QEDF) := qedf.o
qedf-y = qedf_dbg.o qedf_main.o qedf_io.o qedf_fip.o \
- qedf_attr.o qedf_els.o
+ qedf_attr.o qedf_els.o drv_scsi_fw_funcs.o drv_fcoe_fw_funcs.o
qedf-$(CONFIG_DEBUG_FS) += qedf_debugfs.o
diff --git a/drivers/scsi/qedf/drv_fcoe_fw_funcs.c b/drivers/scsi/qedf/drv_fcoe_fw_funcs.c
new file mode 100644
index 0000000..8c65e3b
--- /dev/null
+++ b/drivers/scsi/qedf/drv_fcoe_fw_funcs.c
@@ -0,0 +1,190 @@
+/* QLogic FCoE Offload Driver
+ * Copyright (c) 2016 Cavium Inc.
+ *
+ * This software is available under the terms of the GNU General Public License
+ * (GPL) Version 2, available from the file COPYING in the main directory of
+ * this source tree.
+ */
+#include "drv_fcoe_fw_funcs.h"
+#include "drv_scsi_fw_funcs.h"
+
+#define FCOE_RX_ID (0xFFFFu)
+
+static inline void init_common_sqe(struct fcoe_task_params *task_params,
+ enum fcoe_sqe_request_type request_type)
+{
+ memset(task_params->sqe, 0, sizeof(*(task_params->sqe)));
+ SET_FIELD(task_params->sqe->flags, FCOE_WQE_REQ_TYPE,
+ request_type);
+ task_params->sqe->task_id = task_params->itid;
+}
+
+int init_initiator_rw_fcoe_task(struct fcoe_task_params *task_params,
+ struct scsi_sgl_task_params *sgl_task_params,
+ struct regpair sense_data_buffer_phys_addr,
+ u32 task_retry_id,
+ u8 fcp_cmd_payload[32])
+{
+ struct fcoe_task_context *ctx = task_params->context;
+ struct ystorm_fcoe_task_st_ctx *y_st_ctx;
+ struct tstorm_fcoe_task_st_ctx *t_st_ctx;
+ struct ustorm_fcoe_task_ag_ctx *u_ag_ctx;
+ struct mstorm_fcoe_task_st_ctx *m_st_ctx;
+ u32 io_size, val;
+ bool slow_sgl;
+
+ memset(ctx, 0, sizeof(*(ctx)));
+ slow_sgl = scsi_is_slow_sgl(sgl_task_params->num_sges,
+ sgl_task_params->small_mid_sge);
+ io_size = (task_params->task_type == FCOE_TASK_TYPE_WRITE_INITIATOR ?
+ task_params->tx_io_size : task_params->rx_io_size);
+
+ /* Ystorm ctx */
+ y_st_ctx = &ctx->ystorm_st_context;
+ y_st_ctx->data_2_trns_rem = cpu_to_le32(io_size);
+ y_st_ctx->task_rety_identifier = cpu_to_le32(task_retry_id);
+ y_st_ctx->task_type = task_params->task_type;
+ memcpy(&y_st_ctx->tx_info_union.fcp_cmd_payload,
+ fcp_cmd_payload, sizeof(struct fcoe_fcp_cmd_payload));
+
+ /* Tstorm ctx */
+ t_st_ctx = &ctx->tstorm_st_context;
+ t_st_ctx->read_only.dev_type = (task_params->is_tape_device == 1 ?
+ FCOE_TASK_DEV_TYPE_TAPE :
+ FCOE_TASK_DEV_TYPE_DISK);
+ t_st_ctx->read_only.cid = cpu_to_le32(task_params->conn_cid);
+ val = cpu_to_le32(task_params->cq_rss_number);
+ t_st_ctx->read_only.glbl_q_num = val;
+ t_st_ctx->read_only.fcp_cmd_trns_size = cpu_to_le32(io_size);
+ t_st_ctx->read_only.task_type = task_params->task_type;
+ SET_FIELD(t_st_ctx->read_write.flags,
+ FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_EXP_FIRST_FRAME, 1);
+ t_st_ctx->read_write.rx_id = cpu_to_le16(FCOE_RX_ID);
+
+ /* Ustorm ctx */
+ u_ag_ctx = &ctx->ustorm_ag_context;
+ u_ag_ctx->global_cq_num = cpu_to_le32(task_params->cq_rss_number);
+
+ /* Mstorm buffer for sense/rsp data placement */
+ m_st_ctx = &ctx->mstorm_st_context;
+ val = cpu_to_le32(sense_data_buffer_phys_addr.hi);
+ m_st_ctx->rsp_buf_addr.hi = val;
+ val = cpu_to_le32(sense_data_buffer_phys_addr.lo);
+ m_st_ctx->rsp_buf_addr.lo = val;
+
+ if (task_params->task_type == FCOE_TASK_TYPE_WRITE_INITIATOR) {
+ /* Ystorm ctx */
+ y_st_ctx->expect_first_xfer = 1;
+
+ /* Set the amount of super SGEs. Can be up to 4. */
+ SET_FIELD(y_st_ctx->sgl_mode,
+ YSTORM_FCOE_TASK_ST_CTX_TX_SGL_MODE,
+ (slow_sgl ? SCSI_TX_SLOW_SGL : SCSI_FAST_SGL));
+ init_scsi_sgl_context(&y_st_ctx->sgl_params,
+ &y_st_ctx->data_desc,
+ sgl_task_params);
+
+ /* Mstorm ctx */
+ SET_FIELD(m_st_ctx->flags,
+ MSTORM_FCOE_TASK_ST_CTX_TX_SGL_MODE,
+ (slow_sgl ? SCSI_TX_SLOW_SGL : SCSI_FAST_SGL));
+ } else {
+ /* Tstorm ctx */
+ SET_FIELD(t_st_ctx->read_write.flags,
+ FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_RX_SGL_MODE,
+ (slow_sgl ? SCSI_TX_SLOW_SGL : SCSI_FAST_SGL));
+
+ /* Mstorm ctx */
+ m_st_ctx->data_2_trns_rem = cpu_to_le32(io_size);
+ init_scsi_sgl_context(&m_st_ctx->sgl_params,
+ &m_st_ctx->data_desc,
+ sgl_task_params);
+ }
+
+ init_common_sqe(task_params, SEND_FCOE_CMD);
+ return 0;
+}
+
+int init_initiator_midpath_unsolicited_fcoe_task(
+ struct fcoe_task_params *task_params,
+ struct fcoe_tx_mid_path_params *mid_path_fc_header,
+ struct scsi_sgl_task_params *tx_sgl_task_params,
+ struct scsi_sgl_task_params *rx_sgl_task_params,
+ u8 fw_to_place_fc_header)
+{
+ struct fcoe_task_context *ctx = task_params->context;
+ struct ystorm_fcoe_task_st_ctx *y_st_ctx;
+ struct tstorm_fcoe_task_st_ctx *t_st_ctx;
+ struct ustorm_fcoe_task_ag_ctx *u_ag_ctx;
+ struct mstorm_fcoe_task_st_ctx *m_st_ctx;
+ u32 val;
+
+ memset(ctx, 0, sizeof(*(ctx)));
+
+ /* Init Ystorm */
+ y_st_ctx = &ctx->ystorm_st_context;
+ init_scsi_sgl_context(&y_st_ctx->sgl_params,
+ &y_st_ctx->data_desc,
+ tx_sgl_task_params);
+ SET_FIELD(y_st_ctx->sgl_mode,
+ YSTORM_FCOE_TASK_ST_CTX_TX_SGL_MODE, SCSI_FAST_SGL);
+ y_st_ctx->data_2_trns_rem = cpu_to_le32(task_params->tx_io_size);
+ y_st_ctx->task_type = task_params->task_type;
+ memcpy(&y_st_ctx->tx_info_union.tx_params.mid_path,
+ mid_path_fc_header, sizeof(struct fcoe_tx_mid_path_params));
+
+ /* Init Mstorm */
+ m_st_ctx = &ctx->mstorm_st_context;
+ init_scsi_sgl_context(&m_st_ctx->sgl_params,
+ &m_st_ctx->data_desc,
+ rx_sgl_task_params);
+ SET_FIELD(m_st_ctx->flags,
+ MSTORM_FCOE_TASK_ST_CTX_MP_INCLUDE_FC_HEADER,
+ fw_to_place_fc_header);
+ m_st_ctx->data_2_trns_rem = cpu_to_le32(task_params->rx_io_size);
+
+ /* Init Tstorm */
+ t_st_ctx = &ctx->tstorm_st_context;
+ t_st_ctx->read_only.cid = cpu_to_le32(task_params->conn_cid);
+ val = cpu_to_le32(task_params->cq_rss_number);
+ t_st_ctx->read_only.glbl_q_num = val;
+ t_st_ctx->read_only.task_type = task_params->task_type;
+ SET_FIELD(t_st_ctx->read_write.flags,
+ FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_EXP_FIRST_FRAME, 1);
+ t_st_ctx->read_write.rx_id = cpu_to_le16(FCOE_RX_ID);
+
+ /* Init Ustorm */
+ u_ag_ctx = &ctx->ustorm_ag_context;
+ u_ag_ctx->global_cq_num = cpu_to_le32(task_params->cq_rss_number);
+
+ /* Init SQE */
+ init_common_sqe(task_params, SEND_FCOE_MIDPATH);
+ task_params->sqe->additional_info_union.burst_length =
+ tx_sgl_task_params->total_buffer_size;
+ SET_FIELD(task_params->sqe->flags,
+ FCOE_WQE_NUM_SGES, tx_sgl_task_params->num_sges);
+ SET_FIELD(task_params->sqe->flags, FCOE_WQE_SGL_MODE,
+ SCSI_FAST_SGL);
+
+ return 0;
+}
+
+int init_initiator_abort_fcoe_task(struct fcoe_task_params *task_params)
+{
+ init_common_sqe(task_params, SEND_FCOE_ABTS_REQUEST);
+ return 0;
+}
+
+int init_initiator_cleanup_fcoe_task(struct fcoe_task_params *task_params)
+{
+ init_common_sqe(task_params, FCOE_EXCHANGE_CLEANUP);
+ return 0;
+}
+
+int init_initiator_sequence_recovery_fcoe_task(
+ struct fcoe_task_params *task_params, u32 off)
+{
+ init_common_sqe(task_params, FCOE_SEQUENCE_RECOVERY);
+ task_params->sqe->additional_info_union.seq_rec_updated_offset = off;
+ return 0;
+}
diff --git a/drivers/scsi/qedf/drv_fcoe_fw_funcs.h b/drivers/scsi/qedf/drv_fcoe_fw_funcs.h
new file mode 100644
index 0000000..617529b
--- /dev/null
+++ b/drivers/scsi/qedf/drv_fcoe_fw_funcs.h
@@ -0,0 +1,93 @@
+/* QLogic FCoE Offload Driver
+ * Copyright (c) 2016 Cavium Inc.
+ *
+ * This software is available under the terms of the GNU General Public License
+ * (GPL) Version 2, available from the file COPYING in the main directory of
+ * this source tree.
+ */
+#ifndef _FCOE_FW_FUNCS_H
+#define _FCOE_FW_FUNCS_H
+#include "drv_scsi_fw_funcs.h"
+#include "qedf_hsi.h"
+#include <linux/qed/qed_if.h>
+
+struct fcoe_task_params {
+ /* Output parameter [set/filled by the HSI function] */
+ struct fcoe_task_context *context;
+
+ /* Output parameter [set/filled by the HSI function] */
+ struct fcoe_wqe *sqe;
+ enum fcoe_task_type task_type;
+ u32 tx_io_size; /* in bytes */
+ u32 rx_io_size; /* in bytes */
+ u32 conn_cid;
+ u16 itid;
+ u8 cq_rss_number;
+
+ /* Whether it's Tape device or not (0=Disk, 1=Tape) */
+ u8 is_tape_device;
+};
+
+/**
+ * @brief init_initiator_rw_fcoe_task - Initializes FCoE task context for
+ * read/write task types and init fcoe_sqe
+ *
+ * @param task_params - Pointer to task parameters struct
+ * @param sgl_task_params - Pointer to SGL task params
+ * @param sense_data_buffer_phys_addr - Pointer to sense data buffer
+ * @param task_retry_id - retry identification - Used only for Tape device
+ * @param fcp_cmnd_payload - FCP CMD Payload
+ */
+int init_initiator_rw_fcoe_task(struct fcoe_task_params *task_params,
+ struct scsi_sgl_task_params *sgl_task_params,
+ struct regpair sense_data_buffer_phys_addr,
+ u32 task_retry_id,
+ u8 fcp_cmd_payload[32]);
+
+/**
+ * @brief init_initiator_midpath_fcoe_task - Initializes FCoE task context for
+ * midpath/unsolicited task types and init fcoe_sqe
+ *
+ * @param task_params - Pointer to task parameters struct
+ * @param mid_path_fc_header - FC header
+ * @param tx_sgl_task_params - Pointer to Tx SGL task params
+ * @param rx_sgl_task_params - Pointer to Rx SGL task params
+ * @param fw_to_place_fc_header - Indication if the FW will place the FC header
+ * in addition to the data arrives.
+ */
+int init_initiator_midpath_unsolicited_fcoe_task(
+ struct fcoe_task_params *task_params,
+ struct fcoe_tx_mid_path_params *mid_path_fc_header,
+ struct scsi_sgl_task_params *tx_sgl_task_params,
+ struct scsi_sgl_task_params *rx_sgl_task_params,
+ u8 fw_to_place_fc_header);
+
+/**
+ * @brief init_initiator_abort_fcoe_task - Initializes FCoE task context for
+ * abort task types and init fcoe_sqe
+ *
+ * @param task_params - Pointer to task parameters struct
+ */
+int init_initiator_abort_fcoe_task(struct fcoe_task_params *task_params);
+
+/**
+ * @brief init_initiator_cleanup_fcoe_task - Initializes FCoE task context for
+ * cleanup task types and init fcoe_sqe
+ *
+ *
+ * @param task_params - Pointer to task parameters struct
+ */
+int init_initiator_cleanup_fcoe_task(struct fcoe_task_params *task_params);
+
+/**
+ * @brief init_initiator_cleanup_fcoe_task - Initializes FCoE task context for
+ * sequence recovery task types and init fcoe_sqe
+ *
+ *
+ * @param task_params - Pointer to task parameters struct
+ * @param desired_offset - The desired offest the task will be re-sent from
+ */
+int init_initiator_sequence_recovery_fcoe_task(
+ struct fcoe_task_params *task_params,
+ u32 desired_offset);
+#endif
diff --git a/drivers/scsi/qedf/drv_scsi_fw_funcs.c b/drivers/scsi/qedf/drv_scsi_fw_funcs.c
new file mode 100644
index 0000000..11e0cc0
--- /dev/null
+++ b/drivers/scsi/qedf/drv_scsi_fw_funcs.c
@@ -0,0 +1,44 @@
+/* QLogic FCoE Offload Driver
+ * Copyright (c) 2016 Cavium Inc.
+ *
+ * This software is available under the terms of the GNU General Public License
+ * (GPL) Version 2, available from the file COPYING in the main directory of
+ * this source tree.
+ */
+#include "drv_scsi_fw_funcs.h"
+
+#define SCSI_NUM_SGES_IN_CACHE 0x4
+
+bool scsi_is_slow_sgl(u16 num_sges, bool small_mid_sge)
+{
+ return (num_sges > SCSI_NUM_SGES_SLOW_SGL_THR && small_mid_sge);
+}
+
+void init_scsi_sgl_context(struct scsi_sgl_params *ctx_sgl_params,
+ struct scsi_cached_sges *ctx_data_desc,
+ struct scsi_sgl_task_params *sgl_task_params)
+{
+ /* no need to check for sgl_task_params->sgl validity */
+ u8 num_sges_to_init = sgl_task_params->num_sges >
+ SCSI_NUM_SGES_IN_CACHE ? SCSI_NUM_SGES_IN_CACHE :
+ sgl_task_params->num_sges;
+ u8 sge_index;
+ u32 val;
+
+ val = cpu_to_le32(sgl_task_params->sgl_phys_addr.lo);
+ ctx_sgl_params->sgl_addr.lo = val;
+ val = cpu_to_le32(sgl_task_params->sgl_phys_addr.hi);
+ ctx_sgl_params->sgl_addr.hi = val;
+ val = cpu_to_le32(sgl_task_params->total_buffer_size);
+ ctx_sgl_params->sgl_total_length = val;
+ ctx_sgl_params->sgl_num_sges = cpu_to_le16(sgl_task_params->num_sges);
+
+ for (sge_index = 0; sge_index < num_sges_to_init; sge_index++) {
+ val = cpu_to_le32(sgl_task_params->sgl[sge_index].sge_addr.lo);
+ ctx_data_desc->sge[sge_index].sge_addr.lo = val;
+ val = cpu_to_le32(sgl_task_params->sgl[sge_index].sge_addr.hi);
+ ctx_data_desc->sge[sge_index].sge_addr.hi = val;
+ val = cpu_to_le32(sgl_task_params->sgl[sge_index].sge_len);
+ ctx_data_desc->sge[sge_index].sge_len = val;
+ }
+}
diff --git a/drivers/scsi/qedf/drv_scsi_fw_funcs.h b/drivers/scsi/qedf/drv_scsi_fw_funcs.h
new file mode 100644
index 0000000..9cb4541
--- /dev/null
+++ b/drivers/scsi/qedf/drv_scsi_fw_funcs.h
@@ -0,0 +1,85 @@
+/* QLogic FCoE Offload Driver
+ * Copyright (c) 2016 Cavium Inc.
+ *
+ * This software is available under the terms of the GNU General Public License
+ * (GPL) Version 2, available from the file COPYING in the main directory of
+ * this source tree.
+ */
+#ifndef _SCSI_FW_FUNCS_H
+#define _SCSI_FW_FUNCS_H
+#include <linux/qed/common_hsi.h>
+#include <linux/qed/storage_common.h>
+#include <linux/qed/fcoe_common.h>
+
+struct scsi_sgl_task_params {
+ struct scsi_sge *sgl;
+ struct regpair sgl_phys_addr;
+ u32 total_buffer_size;
+ u16 num_sges;
+
+ /* true if SGL contains a small (< 4KB) SGE in middle(not 1st or last)
+ * -> relevant for tx only
+ */
+ bool small_mid_sge;
+};
+
+struct scsi_dif_task_params {
+ u32 initial_ref_tag;
+ bool initial_ref_tag_is_valid;
+ u16 application_tag;
+ u16 application_tag_mask;
+ u16 dif_block_size_log;
+ bool dif_on_network;
+ bool dif_on_host;
+ u8 host_guard_type;
+ u8 protection_type;
+ u8 ref_tag_mask;
+ bool crc_seed;
+
+ /* Enable Connection error upon DIF error (segments with DIF errors are
+ * dropped)
+ */
+ bool tx_dif_conn_err_en;
+ bool ignore_app_tag;
+ bool keep_ref_tag_const;
+ bool validate_guard;
+ bool validate_app_tag;
+ bool validate_ref_tag;
+ bool forward_guard;
+ bool forward_app_tag;
+ bool forward_ref_tag;
+ bool forward_app_tag_with_mask;
+ bool forward_ref_tag_with_mask;
+};
+
+struct scsi_initiator_cmd_params {
+ /* for cdb_size > default CDB size (extended CDB > 16 bytes) ->
+ * pointer to the CDB buffer SGE
+ */
+ struct scsi_sge extended_cdb_sge;
+
+ /* Physical address of sense data buffer for sense data - 256B buffer */
+ struct regpair sense_data_buffer_phys_addr;
+};
+
+/**
+ * @brief scsi_is_slow_sgl - checks for slow SGL
+ *
+ * @param num_sges - number of sges in SGL
+ * @param small_mid_sge - True is the SGL contains an SGE which is smaller than
+ * 4KB and its not the 1st or last SGE in the SGL
+ */
+bool scsi_is_slow_sgl(u16 num_sges, bool small_mid_sge);
+
+/**
+ * @brief init_scsi_sgl_context - initializes SGL task context
+ *
+ * @param sgl_params - SGL context parameters to initialize (output parameter)
+ * @param data_desc - context struct containing SGEs array to set (output
+ * parameter)
+ * @param sgl_task_params - SGL parameters (input)
+ */
+void init_scsi_sgl_context(struct scsi_sgl_params *sgl_params,
+ struct scsi_cached_sges *ctx_data_desc,
+ struct scsi_sgl_task_params *sgl_task_params);
+#endif
diff --git a/drivers/scsi/qedf/qedf.h b/drivers/scsi/qedf/qedf.h
index 96346a1..40aeb6b 100644
--- a/drivers/scsi/qedf/qedf.h
+++ b/drivers/scsi/qedf/qedf.h
@@ -26,6 +26,7 @@
#include <linux/qed/qed_ll2_if.h>
#include "qedf_version.h"
#include "qedf_dbg.h"
+#include "drv_fcoe_fw_funcs.h"
/* Helpers to extract upper and lower 32-bits of pointer */
#define U64_HI(val) ((u32)(((u64)(val)) >> 32))
@@ -59,19 +60,17 @@
#define UPSTREAM_KEEP 1
struct qedf_mp_req {
- uint8_t tm_flags;
-
uint32_t req_len;
void *req_buf;
dma_addr_t req_buf_dma;
- struct fcoe_sge *mp_req_bd;
+ struct scsi_sge *mp_req_bd;
dma_addr_t mp_req_bd_dma;
struct fc_frame_header req_fc_hdr;
uint32_t resp_len;
void *resp_buf;
dma_addr_t resp_buf_dma;
- struct fcoe_sge *mp_resp_bd;
+ struct scsi_sge *mp_resp_bd;
dma_addr_t mp_resp_bd_dma;
struct fc_frame_header resp_fc_hdr;
};
@@ -119,6 +118,7 @@
#define QEDF_CMD_IN_CLEANUP 0x2
#define QEDF_CMD_SRR_SENT 0x3
u8 io_req_flags;
+ uint8_t tm_flags;
struct qedf_rport *fcport;
unsigned long flags;
enum qedf_ioreq_event event;
@@ -130,6 +130,8 @@
struct completion tm_done;
struct completion abts_done;
struct fcoe_task_context *task;
+ struct fcoe_task_params *task_params;
+ struct scsi_sgl_task_params *sgl_task_params;
int idx;
/*
* Need to allocate enough room for both sense data and FCP response data
@@ -199,8 +201,8 @@
dma_addr_t sq_pbl_dma;
u32 sq_pbl_size;
u32 sid;
-#define QEDF_RPORT_TYPE_DISK 1
-#define QEDF_RPORT_TYPE_TAPE 2
+#define QEDF_RPORT_TYPE_DISK 0
+#define QEDF_RPORT_TYPE_TAPE 1
uint dev_type; /* Disk or tape */
struct list_head peers;
};
@@ -391,7 +393,7 @@
struct io_bdt {
struct qedf_ioreq *io_req;
- struct fcoe_sge *bd_tbl;
+ struct scsi_sge *bd_tbl;
dma_addr_t bd_tbl_dma;
u16 bd_valid;
};
@@ -400,7 +402,7 @@
struct qedf_ctx *qedf;
u16 idx;
struct io_bdt **io_bdt_pool;
-#define FCOE_PARAMS_NUM_TASKS 4096
+#define FCOE_PARAMS_NUM_TASKS 2048
struct qedf_ioreq cmds[FCOE_PARAMS_NUM_TASKS];
spinlock_t lock;
atomic_t free_list_cnt;
@@ -465,9 +467,8 @@
unsigned int timer_msec);
extern int qedf_init_mp_req(struct qedf_ioreq *io_req);
extern void qedf_init_mp_task(struct qedf_ioreq *io_req,
- struct fcoe_task_context *task_ctx);
-extern void qedf_add_to_sq(struct qedf_rport *fcport, u16 xid,
- u32 ptu_invalidate, enum fcoe_task_type req_type, u32 offset);
+ struct fcoe_task_context *task_ctx, struct fcoe_wqe *wqe);
+extern u16 qedf_get_sqe_idx(struct qedf_rport *fcport);
extern void qedf_ring_doorbell(struct qedf_rport *fcport);
extern void qedf_process_els_compl(struct qedf_ctx *qedf, struct fcoe_cqe *cqe,
struct qedf_ioreq *els_req);
diff --git a/drivers/scsi/qedf/qedf_els.c b/drivers/scsi/qedf/qedf_els.c
index 59f3e5c..c505d41 100644
--- a/drivers/scsi/qedf/qedf_els.c
+++ b/drivers/scsi/qedf/qedf_els.c
@@ -25,6 +25,9 @@
uint16_t xid;
uint32_t start_time = jiffies / HZ;
uint32_t current_time;
+ struct fcoe_wqe *sqe;
+ unsigned long flags;
+ u16 sqe_idx;
QEDF_INFO(&(qedf->dbg_ctx), QEDF_LOG_ELS, "Sending ELS\n");
@@ -113,20 +116,25 @@
/* Obtain exchange id */
xid = els_req->xid;
+ spin_lock_irqsave(&fcport->rport_lock, flags);
+
+ sqe_idx = qedf_get_sqe_idx(fcport);
+ sqe = &fcport->sq[sqe_idx];
+ memset(sqe, 0, sizeof(struct fcoe_wqe));
+
/* Initialize task context for this IO request */
task = qedf_get_task_mem(&qedf->tasks, xid);
- qedf_init_mp_task(els_req, task);
+ qedf_init_mp_task(els_req, task, sqe);
/* Put timer on original I/O request */
if (timer_msec)
qedf_cmd_timer_set(qedf, els_req, timer_msec);
- qedf_add_to_sq(fcport, xid, 0, FCOE_TASK_TYPE_MIDPATH, 0);
-
/* Ring doorbell */
QEDF_INFO(&(qedf->dbg_ctx), QEDF_LOG_ELS, "Ringing doorbell for ELS "
"req\n");
qedf_ring_doorbell(fcport);
+ spin_unlock_irqrestore(&fcport->rport_lock, flags);
els_err:
return rc;
}
@@ -604,6 +612,8 @@
struct qedf_rport *fcport;
unsigned long flags;
struct qedf_els_cb_arg *cb_arg;
+ struct fcoe_wqe *sqe;
+ u16 sqe_idx;
fcport = orig_io_req->fcport;
@@ -631,8 +641,13 @@
spin_lock_irqsave(&fcport->rport_lock, flags);
- qedf_add_to_sq(fcport, orig_io_req->xid, 0,
- FCOE_TASK_TYPE_SEQUENCE_CLEANUP, offset);
+ sqe_idx = qedf_get_sqe_idx(fcport);
+ sqe = &fcport->sq[sqe_idx];
+ memset(sqe, 0, sizeof(struct fcoe_wqe));
+ orig_io_req->task_params->sqe = sqe;
+
+ init_initiator_sequence_recovery_fcoe_task(orig_io_req->task_params,
+ offset);
qedf_ring_doorbell(fcport);
spin_unlock_irqrestore(&fcport->rport_lock, flags);
diff --git a/drivers/scsi/qedf/qedf_io.c b/drivers/scsi/qedf/qedf_io.c
index 46debe5..1d7f90d 100644
--- a/drivers/scsi/qedf/qedf_io.c
+++ b/drivers/scsi/qedf/qedf_io.c
@@ -96,7 +96,7 @@
if (!cmgr->io_bdt_pool)
goto free_cmd_pool;
- bd_tbl_sz = QEDF_MAX_BDS_PER_CMD * sizeof(struct fcoe_sge);
+ bd_tbl_sz = QEDF_MAX_BDS_PER_CMD * sizeof(struct scsi_sge);
for (i = 0; i < num_ios; i++) {
bdt_info = cmgr->io_bdt_pool[i];
if (bdt_info->bd_tbl) {
@@ -119,6 +119,8 @@
for (i = 0; i < num_ios; i++) {
io_req = &cmgr->cmds[i];
+ kfree(io_req->sgl_task_params);
+ kfree(io_req->task_params);
/* Make sure we free per command sense buffer */
if (io_req->sense_buffer)
dma_free_coherent(&qedf->pdev->dev,
@@ -178,7 +180,7 @@
spin_lock_init(&cmgr->lock);
/*
- * Initialize list of qedf_ioreq.
+ * Initialize I/O request fields.
*/
xid = QEDF_MIN_XID;
@@ -196,6 +198,29 @@
GFP_KERNEL);
if (!io_req->sense_buffer)
goto mem_err;
+
+ /* Allocate task parameters to pass to f/w init funcions */
+ io_req->task_params = kzalloc(sizeof(*io_req->task_params),
+ GFP_KERNEL);
+ if (!io_req->task_params) {
+ QEDF_ERR(&(qedf->dbg_ctx),
+ "Failed to allocate task_params for xid=0x%x\n",
+ i);
+ goto mem_err;
+ }
+
+ /*
+ * Allocate scatter/gather list info to pass to f/w init
+ * functions.
+ */
+ io_req->sgl_task_params = kzalloc(
+ sizeof(struct scsi_sgl_task_params), GFP_KERNEL);
+ if (!io_req->sgl_task_params) {
+ QEDF_ERR(&(qedf->dbg_ctx),
+ "Failed to allocate sgl_task_params for xid=0x%x\n",
+ i);
+ goto mem_err;
+ }
}
/* Allocate pool of io_bdts - one for each qedf_ioreq */
@@ -211,8 +236,8 @@
cmgr->io_bdt_pool[i] = kmalloc(sizeof(struct io_bdt),
GFP_KERNEL);
if (!cmgr->io_bdt_pool[i]) {
- QEDF_WARN(&(qedf->dbg_ctx), "Failed to alloc "
- "io_bdt_pool[%d].\n", i);
+ QEDF_WARN(&(qedf->dbg_ctx),
+ "Failed to alloc io_bdt_pool[%d].\n", i);
goto mem_err;
}
}
@@ -220,11 +245,11 @@
for (i = 0; i < num_ios; i++) {
bdt_info = cmgr->io_bdt_pool[i];
bdt_info->bd_tbl = dma_alloc_coherent(&qedf->pdev->dev,
- QEDF_MAX_BDS_PER_CMD * sizeof(struct fcoe_sge),
+ QEDF_MAX_BDS_PER_CMD * sizeof(struct scsi_sge),
&bdt_info->bd_tbl_dma, GFP_KERNEL);
if (!bdt_info->bd_tbl) {
- QEDF_WARN(&(qedf->dbg_ctx), "Failed to alloc "
- "bdt_tbl[%d].\n", i);
+ QEDF_WARN(&(qedf->dbg_ctx),
+ "Failed to alloc bdt_tbl[%d].\n", i);
goto mem_err;
}
}
@@ -318,6 +343,7 @@
}
bd_tbl->io_req = io_req;
io_req->cmd_type = cmd_type;
+ io_req->tm_flags = 0;
/* Reset sequence offset data */
io_req->rx_buf_off = 0;
@@ -336,10 +362,9 @@
{
struct qedf_mp_req *mp_req = &(io_req->mp_req);
struct qedf_ctx *qedf = io_req->fcport->qedf;
- uint64_t sz = sizeof(struct fcoe_sge);
+ uint64_t sz = sizeof(struct scsi_sge);
/* clear tm flags */
- mp_req->tm_flags = 0;
if (mp_req->mp_req_bd) {
dma_free_coherent(&qedf->pdev->dev, sz,
mp_req->mp_req_bd, mp_req->mp_req_bd_dma);
@@ -387,7 +412,7 @@
static int qedf_split_bd(struct qedf_ioreq *io_req, u64 addr, int sg_len,
int bd_index)
{
- struct fcoe_sge *bd = io_req->bd_tbl->bd_tbl;
+ struct scsi_sge *bd = io_req->bd_tbl->bd_tbl;
int frag_size, sg_frags;
sg_frags = 0;
@@ -398,7 +423,7 @@
frag_size = sg_len;
bd[bd_index + sg_frags].sge_addr.lo = U64_LO(addr);
bd[bd_index + sg_frags].sge_addr.hi = U64_HI(addr);
- bd[bd_index + sg_frags].size = (uint16_t)frag_size;
+ bd[bd_index + sg_frags].sge_len = (uint16_t)frag_size;
addr += (u64)frag_size;
sg_frags++;
@@ -413,7 +438,7 @@
struct Scsi_Host *host = sc->device->host;
struct fc_lport *lport = shost_priv(host);
struct qedf_ctx *qedf = lport_priv(lport);
- struct fcoe_sge *bd = io_req->bd_tbl->bd_tbl;
+ struct scsi_sge *bd = io_req->bd_tbl->bd_tbl;
struct scatterlist *sg;
int byte_count = 0;
int sg_count = 0;
@@ -439,7 +464,7 @@
bd[bd_count].sge_addr.lo = (addr & 0xffffffff);
bd[bd_count].sge_addr.hi = (addr >> 32);
- bd[bd_count].size = (u16)sg_len;
+ bd[bd_count].sge_len = (u16)sg_len;
return ++bd_count;
}
@@ -480,7 +505,7 @@
sg_frags = 1;
bd[bd_count].sge_addr.lo = U64_LO(addr);
bd[bd_count].sge_addr.hi = U64_HI(addr);
- bd[bd_count].size = (uint16_t)sg_len;
+ bd[bd_count].sge_len = (uint16_t)sg_len;
}
bd_count += sg_frags;
@@ -498,7 +523,7 @@
static int qedf_build_bd_list_from_sg(struct qedf_ioreq *io_req)
{
struct scsi_cmnd *sc = io_req->sc_cmd;
- struct fcoe_sge *bd = io_req->bd_tbl->bd_tbl;
+ struct scsi_sge *bd = io_req->bd_tbl->bd_tbl;
int bd_count;
if (scsi_sg_count(sc)) {
@@ -508,7 +533,7 @@
} else {
bd_count = 0;
bd[0].sge_addr.lo = bd[0].sge_addr.hi = 0;
- bd[0].size = 0;
+ bd[0].sge_len = 0;
}
io_req->bd_tbl->bd_valid = bd_count;
@@ -529,430 +554,223 @@
/* 4 bytes: flag info */
fcp_cmnd->fc_pri_ta = 0;
- fcp_cmnd->fc_tm_flags = io_req->mp_req.tm_flags;
+ fcp_cmnd->fc_tm_flags = io_req->tm_flags;
fcp_cmnd->fc_flags = io_req->io_req_flags;
fcp_cmnd->fc_cmdref = 0;
/* Populate data direction */
- if (sc_cmd->sc_data_direction == DMA_TO_DEVICE)
- fcp_cmnd->fc_flags |= FCP_CFL_WRDATA;
- else if (sc_cmd->sc_data_direction == DMA_FROM_DEVICE)
+ if (io_req->cmd_type == QEDF_TASK_MGMT_CMD) {
fcp_cmnd->fc_flags |= FCP_CFL_RDDATA;
+ } else {
+ if (sc_cmd->sc_data_direction == DMA_TO_DEVICE)
+ fcp_cmnd->fc_flags |= FCP_CFL_WRDATA;
+ else if (sc_cmd->sc_data_direction == DMA_FROM_DEVICE)
+ fcp_cmnd->fc_flags |= FCP_CFL_RDDATA;
+ }
fcp_cmnd->fc_pri_ta = FCP_PTA_SIMPLE;
/* 16 bytes: CDB information */
- memcpy(fcp_cmnd->fc_cdb, sc_cmd->cmnd, sc_cmd->cmd_len);
+ if (io_req->cmd_type != QEDF_TASK_MGMT_CMD)
+ memcpy(fcp_cmnd->fc_cdb, sc_cmd->cmnd, sc_cmd->cmd_len);
/* 4 bytes: FCP data length */
fcp_cmnd->fc_dl = htonl(io_req->data_xfer_len);
-
}
static void qedf_init_task(struct qedf_rport *fcport, struct fc_lport *lport,
- struct qedf_ioreq *io_req, u32 *ptu_invalidate,
- struct fcoe_task_context *task_ctx)
+ struct qedf_ioreq *io_req, struct fcoe_task_context *task_ctx,
+ struct fcoe_wqe *sqe)
{
enum fcoe_task_type task_type;
struct scsi_cmnd *sc_cmd = io_req->sc_cmd;
struct io_bdt *bd_tbl = io_req->bd_tbl;
- union fcoe_data_desc_ctx *data_desc;
- u32 *fcp_cmnd;
+ u8 fcp_cmnd[32];
u32 tmp_fcp_cmnd[8];
- int cnt, i;
- int bd_count;
+ int bd_count = 0;
struct qedf_ctx *qedf = fcport->qedf;
uint16_t cq_idx = smp_processor_id() % qedf->num_queues;
- u8 tmp_sgl_mode = 0;
- u8 mst_sgl_mode = 0;
+ struct regpair sense_data_buffer_phys_addr;
+ u32 tx_io_size = 0;
+ u32 rx_io_size = 0;
+ int i, cnt;
- memset(task_ctx, 0, sizeof(struct fcoe_task_context));
+ /* Note init_initiator_rw_fcoe_task memsets the task context */
io_req->task = task_ctx;
+ memset(task_ctx, 0, sizeof(struct fcoe_task_context));
+ memset(io_req->task_params, 0, sizeof(struct fcoe_task_params));
+ memset(io_req->sgl_task_params, 0, sizeof(struct scsi_sgl_task_params));
- if (sc_cmd->sc_data_direction == DMA_TO_DEVICE)
- task_type = FCOE_TASK_TYPE_WRITE_INITIATOR;
- else
+ /* Set task type bassed on DMA directio of command */
+ if (io_req->cmd_type == QEDF_TASK_MGMT_CMD) {
task_type = FCOE_TASK_TYPE_READ_INITIATOR;
-
- /* Y Storm context */
- task_ctx->ystorm_st_context.expect_first_xfer = 1;
- task_ctx->ystorm_st_context.data_2_trns_rem = io_req->data_xfer_len;
- /* Check if this is required */
- task_ctx->ystorm_st_context.ox_id = io_req->xid;
- task_ctx->ystorm_st_context.task_rety_identifier =
- io_req->task_retry_identifier;
-
- /* T Storm ag context */
- SET_FIELD(task_ctx->tstorm_ag_context.flags0,
- TSTORM_FCOE_TASK_AG_CTX_CONNECTION_TYPE, PROTOCOLID_FCOE);
- task_ctx->tstorm_ag_context.icid = (u16)fcport->fw_cid;
-
- /* T Storm st context */
- SET_FIELD(task_ctx->tstorm_st_context.read_write.flags,
- FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_EXP_FIRST_FRAME,
- 1);
- task_ctx->tstorm_st_context.read_write.rx_id = 0xffff;
-
- task_ctx->tstorm_st_context.read_only.dev_type =
- FCOE_TASK_DEV_TYPE_DISK;
- task_ctx->tstorm_st_context.read_only.conf_supported = 0;
- task_ctx->tstorm_st_context.read_only.cid = fcport->fw_cid;
-
- /* Completion queue for response. */
- task_ctx->tstorm_st_context.read_only.glbl_q_num = cq_idx;
- task_ctx->tstorm_st_context.read_only.fcp_cmd_trns_size =
- io_req->data_xfer_len;
- task_ctx->tstorm_st_context.read_write.e_d_tov_exp_timeout_val =
- lport->e_d_tov;
-
- task_ctx->ustorm_ag_context.global_cq_num = cq_idx;
- io_req->fp_idx = cq_idx;
-
- bd_count = bd_tbl->bd_valid;
- if (task_type == FCOE_TASK_TYPE_WRITE_INITIATOR) {
- /* Setup WRITE task */
- struct fcoe_sge *fcoe_bd_tbl = bd_tbl->bd_tbl;
-
- task_ctx->ystorm_st_context.task_type =
- FCOE_TASK_TYPE_WRITE_INITIATOR;
- data_desc = &task_ctx->ystorm_st_context.data_desc;
-
- if (io_req->use_slowpath) {
- SET_FIELD(task_ctx->ystorm_st_context.sgl_mode,
- YSTORM_FCOE_TASK_ST_CTX_TX_SGL_MODE,
- FCOE_SLOW_SGL);
- data_desc->slow.base_sgl_addr.lo =
- U64_LO(bd_tbl->bd_tbl_dma);
- data_desc->slow.base_sgl_addr.hi =
- U64_HI(bd_tbl->bd_tbl_dma);
- data_desc->slow.remainder_num_sges = bd_count;
- data_desc->slow.curr_sge_off = 0;
- data_desc->slow.curr_sgl_index = 0;
- qedf->slow_sge_ios++;
- io_req->sge_type = QEDF_IOREQ_SLOW_SGE;
- } else {
- SET_FIELD(task_ctx->ystorm_st_context.sgl_mode,
- YSTORM_FCOE_TASK_ST_CTX_TX_SGL_MODE,
- (bd_count <= 4) ? (enum fcoe_sgl_mode)bd_count :
- FCOE_MUL_FAST_SGES);
-
- if (bd_count == 1) {
- data_desc->single_sge.sge_addr.lo =
- fcoe_bd_tbl->sge_addr.lo;
- data_desc->single_sge.sge_addr.hi =
- fcoe_bd_tbl->sge_addr.hi;
- data_desc->single_sge.size =
- fcoe_bd_tbl->size;
- data_desc->single_sge.is_valid_sge = 0;
- qedf->single_sge_ios++;
- io_req->sge_type = QEDF_IOREQ_SINGLE_SGE;
- } else {
- data_desc->fast.sgl_start_addr.lo =
- U64_LO(bd_tbl->bd_tbl_dma);
- data_desc->fast.sgl_start_addr.hi =
- U64_HI(bd_tbl->bd_tbl_dma);
- data_desc->fast.sgl_byte_offset =
- data_desc->fast.sgl_start_addr.lo &
- (QEDF_PAGE_SIZE - 1);
- if (data_desc->fast.sgl_byte_offset > 0)
- QEDF_ERR(&(qedf->dbg_ctx),
- "byte_offset=%u for xid=0x%x.\n",
- io_req->xid,
- data_desc->fast.sgl_byte_offset);
- data_desc->fast.task_reuse_cnt =
- io_req->reuse_count;
- io_req->reuse_count++;
- if (io_req->reuse_count == QEDF_MAX_REUSE) {
- *ptu_invalidate = 1;
- io_req->reuse_count = 0;
- }
- qedf->fast_sge_ios++;
- io_req->sge_type = QEDF_IOREQ_FAST_SGE;
- }
- }
-
- /* T Storm context */
- task_ctx->tstorm_st_context.read_only.task_type =
- FCOE_TASK_TYPE_WRITE_INITIATOR;
-
- /* M Storm context */
- tmp_sgl_mode = GET_FIELD(task_ctx->ystorm_st_context.sgl_mode,
- YSTORM_FCOE_TASK_ST_CTX_TX_SGL_MODE);
- SET_FIELD(task_ctx->mstorm_st_context.non_fp.tx_rx_sgl_mode,
- FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_TX_SGL_MODE,
- tmp_sgl_mode);
-
} else {
- /* Setup READ task */
-
- /* M Storm context */
- struct fcoe_sge *fcoe_bd_tbl = bd_tbl->bd_tbl;
-
- data_desc = &task_ctx->mstorm_st_context.fp.data_desc;
- task_ctx->mstorm_st_context.fp.data_2_trns_rem =
- io_req->data_xfer_len;
-
- if (io_req->use_slowpath) {
- SET_FIELD(
- task_ctx->mstorm_st_context.non_fp.tx_rx_sgl_mode,
- FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_RX_SGL_MODE,
- FCOE_SLOW_SGL);
- data_desc->slow.base_sgl_addr.lo =
- U64_LO(bd_tbl->bd_tbl_dma);
- data_desc->slow.base_sgl_addr.hi =
- U64_HI(bd_tbl->bd_tbl_dma);
- data_desc->slow.remainder_num_sges =
- bd_count;
- data_desc->slow.curr_sge_off = 0;
- data_desc->slow.curr_sgl_index = 0;
- qedf->slow_sge_ios++;
- io_req->sge_type = QEDF_IOREQ_SLOW_SGE;
+ if (sc_cmd->sc_data_direction == DMA_TO_DEVICE) {
+ task_type = FCOE_TASK_TYPE_WRITE_INITIATOR;
+ tx_io_size = io_req->data_xfer_len;
} else {
- SET_FIELD(
- task_ctx->mstorm_st_context.non_fp.tx_rx_sgl_mode,
- FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_RX_SGL_MODE,
- (bd_count <= 4) ? (enum fcoe_sgl_mode)bd_count :
- FCOE_MUL_FAST_SGES);
-
- if (bd_count == 1) {
- data_desc->single_sge.sge_addr.lo =
- fcoe_bd_tbl->sge_addr.lo;
- data_desc->single_sge.sge_addr.hi =
- fcoe_bd_tbl->sge_addr.hi;
- data_desc->single_sge.size =
- fcoe_bd_tbl->size;
- data_desc->single_sge.is_valid_sge = 0;
- qedf->single_sge_ios++;
- io_req->sge_type = QEDF_IOREQ_SINGLE_SGE;
- } else {
- data_desc->fast.sgl_start_addr.lo =
- U64_LO(bd_tbl->bd_tbl_dma);
- data_desc->fast.sgl_start_addr.hi =
- U64_HI(bd_tbl->bd_tbl_dma);
- data_desc->fast.sgl_byte_offset = 0;
- data_desc->fast.task_reuse_cnt =
- io_req->reuse_count;
- io_req->reuse_count++;
- if (io_req->reuse_count == QEDF_MAX_REUSE) {
- *ptu_invalidate = 1;
- io_req->reuse_count = 0;
- }
- qedf->fast_sge_ios++;
- io_req->sge_type = QEDF_IOREQ_FAST_SGE;
- }
+ task_type = FCOE_TASK_TYPE_READ_INITIATOR;
+ rx_io_size = io_req->data_xfer_len;
}
-
- /* Y Storm context */
- task_ctx->ystorm_st_context.expect_first_xfer = 0;
- task_ctx->ystorm_st_context.task_type =
- FCOE_TASK_TYPE_READ_INITIATOR;
-
- /* T Storm context */
- task_ctx->tstorm_st_context.read_only.task_type =
- FCOE_TASK_TYPE_READ_INITIATOR;
- mst_sgl_mode = GET_FIELD(
- task_ctx->mstorm_st_context.non_fp.tx_rx_sgl_mode,
- FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_RX_SGL_MODE);
- SET_FIELD(task_ctx->tstorm_st_context.read_write.flags,
- FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_RX_SGL_MODE,
- mst_sgl_mode);
}
+ /* Setup the fields for fcoe_task_params */
+ io_req->task_params->context = task_ctx;
+ io_req->task_params->sqe = sqe;
+ io_req->task_params->task_type = task_type;
+ io_req->task_params->tx_io_size = tx_io_size;
+ io_req->task_params->rx_io_size = rx_io_size;
+ io_req->task_params->conn_cid = fcport->fw_cid;
+ io_req->task_params->itid = io_req->xid;
+ io_req->task_params->cq_rss_number = cq_idx;
+ io_req->task_params->is_tape_device = fcport->dev_type;
+
+ /* Fill in information for scatter/gather list */
+ if (io_req->cmd_type != QEDF_TASK_MGMT_CMD) {
+ bd_count = bd_tbl->bd_valid;
+ io_req->sgl_task_params->sgl = bd_tbl->bd_tbl;
+ io_req->sgl_task_params->sgl_phys_addr.lo =
+ U64_LO(bd_tbl->bd_tbl_dma);
+ io_req->sgl_task_params->sgl_phys_addr.hi =
+ U64_HI(bd_tbl->bd_tbl_dma);
+ io_req->sgl_task_params->num_sges = bd_count;
+ io_req->sgl_task_params->total_buffer_size =
+ scsi_bufflen(io_req->sc_cmd);
+ io_req->sgl_task_params->small_mid_sge =
+ io_req->use_slowpath;
+ }
+
+ /* Fill in physical address of sense buffer */
+ sense_data_buffer_phys_addr.lo = U64_LO(io_req->sense_buffer_dma);
+ sense_data_buffer_phys_addr.hi = U64_HI(io_req->sense_buffer_dma);
+
/* fill FCP_CMND IU */
- fcp_cmnd = (u32 *)task_ctx->ystorm_st_context.tx_info_union.fcp_cmd_payload.opaque;
- qedf_build_fcp_cmnd(io_req, (struct fcp_cmnd *)&tmp_fcp_cmnd);
+ qedf_build_fcp_cmnd(io_req, (struct fcp_cmnd *)tmp_fcp_cmnd);
/* Swap fcp_cmnd since FC is big endian */
cnt = sizeof(struct fcp_cmnd) / sizeof(u32);
-
for (i = 0; i < cnt; i++) {
- *fcp_cmnd = cpu_to_be32(tmp_fcp_cmnd[i]);
- fcp_cmnd++;
+ tmp_fcp_cmnd[i] = cpu_to_be32(tmp_fcp_cmnd[i]);
}
+ memcpy(fcp_cmnd, tmp_fcp_cmnd, sizeof(struct fcp_cmnd));
- /* M Storm context - Sense buffer */
- task_ctx->mstorm_st_context.non_fp.rsp_buf_addr.lo =
- U64_LO(io_req->sense_buffer_dma);
- task_ctx->mstorm_st_context.non_fp.rsp_buf_addr.hi =
- U64_HI(io_req->sense_buffer_dma);
+ init_initiator_rw_fcoe_task(io_req->task_params,
+ io_req->sgl_task_params,
+ sense_data_buffer_phys_addr,
+ io_req->task_retry_identifier, fcp_cmnd);
+
+ /* Increment SGL type counters */
+ if (bd_count == 1) {
+ qedf->single_sge_ios++;
+ io_req->sge_type = QEDF_IOREQ_SINGLE_SGE;
+ } else if (io_req->use_slowpath) {
+ qedf->slow_sge_ios++;
+ io_req->sge_type = QEDF_IOREQ_SLOW_SGE;
+ } else {
+ qedf->fast_sge_ios++;
+ io_req->sge_type = QEDF_IOREQ_FAST_SGE;
+ }
}
void qedf_init_mp_task(struct qedf_ioreq *io_req,
- struct fcoe_task_context *task_ctx)
+ struct fcoe_task_context *task_ctx, struct fcoe_wqe *sqe)
{
struct qedf_mp_req *mp_req = &(io_req->mp_req);
struct qedf_rport *fcport = io_req->fcport;
struct qedf_ctx *qedf = io_req->fcport->qedf;
struct fc_frame_header *fc_hdr;
- enum fcoe_task_type task_type = 0;
- union fcoe_data_desc_ctx *data_desc;
+ struct fcoe_tx_mid_path_params task_fc_hdr;
+ struct scsi_sgl_task_params tx_sgl_task_params;
+ struct scsi_sgl_task_params rx_sgl_task_params;
- QEDF_INFO(&(qedf->dbg_ctx), QEDF_LOG_DISC, "Initializing MP task "
- "for cmd_type = %d\n", io_req->cmd_type);
+ QEDF_INFO(&(qedf->dbg_ctx), QEDF_LOG_DISC,
+ "Initializing MP task for cmd_type=%d\n",
+ io_req->cmd_type);
qedf->control_requests++;
- /* Obtain task_type */
- if ((io_req->cmd_type == QEDF_TASK_MGMT_CMD) ||
- (io_req->cmd_type == QEDF_ELS)) {
- task_type = FCOE_TASK_TYPE_MIDPATH;
- } else if (io_req->cmd_type == QEDF_ABTS) {
- task_type = FCOE_TASK_TYPE_ABTS;
- }
-
+ memset(&tx_sgl_task_params, 0, sizeof(struct scsi_sgl_task_params));
+ memset(&rx_sgl_task_params, 0, sizeof(struct scsi_sgl_task_params));
memset(task_ctx, 0, sizeof(struct fcoe_task_context));
+ memset(&task_fc_hdr, 0, sizeof(struct fcoe_tx_mid_path_params));
/* Setup the task from io_req for easy reference */
io_req->task = task_ctx;
- QEDF_INFO(&(qedf->dbg_ctx), QEDF_LOG_DISC, "task type = %d\n",
- task_type);
+ /* Setup the fields for fcoe_task_params */
+ io_req->task_params->context = task_ctx;
+ io_req->task_params->sqe = sqe;
+ io_req->task_params->task_type = FCOE_TASK_TYPE_MIDPATH;
+ io_req->task_params->tx_io_size = io_req->data_xfer_len;
+ /* rx_io_size tells the f/w how large a response buffer we have */
+ io_req->task_params->rx_io_size = PAGE_SIZE;
+ io_req->task_params->conn_cid = fcport->fw_cid;
+ io_req->task_params->itid = io_req->xid;
+ /* Return middle path commands on CQ 0 */
+ io_req->task_params->cq_rss_number = 0;
+ io_req->task_params->is_tape_device = fcport->dev_type;
- /* YSTORM only */
- {
- /* Initialize YSTORM task context */
- struct fcoe_tx_mid_path_params *task_fc_hdr =
- &task_ctx->ystorm_st_context.tx_info_union.tx_params.mid_path;
- memset(task_fc_hdr, 0, sizeof(struct fcoe_tx_mid_path_params));
- task_ctx->ystorm_st_context.task_rety_identifier =
- io_req->task_retry_identifier;
+ fc_hdr = &(mp_req->req_fc_hdr);
+ /* Set OX_ID and RX_ID based on driver task id */
+ fc_hdr->fh_ox_id = io_req->xid;
+ fc_hdr->fh_rx_id = htons(0xffff);
- /* Init SGL parameters */
- if ((task_type == FCOE_TASK_TYPE_MIDPATH) ||
- (task_type == FCOE_TASK_TYPE_UNSOLICITED)) {
- data_desc = &task_ctx->ystorm_st_context.data_desc;
- data_desc->slow.base_sgl_addr.lo =
- U64_LO(mp_req->mp_req_bd_dma);
- data_desc->slow.base_sgl_addr.hi =
- U64_HI(mp_req->mp_req_bd_dma);
- data_desc->slow.remainder_num_sges = 1;
- data_desc->slow.curr_sge_off = 0;
- data_desc->slow.curr_sgl_index = 0;
- }
+ /* Set up FC header information */
+ task_fc_hdr.parameter = fc_hdr->fh_parm_offset;
+ task_fc_hdr.r_ctl = fc_hdr->fh_r_ctl;
+ task_fc_hdr.type = fc_hdr->fh_type;
+ task_fc_hdr.cs_ctl = fc_hdr->fh_cs_ctl;
+ task_fc_hdr.df_ctl = fc_hdr->fh_df_ctl;
+ task_fc_hdr.rx_id = fc_hdr->fh_rx_id;
+ task_fc_hdr.ox_id = fc_hdr->fh_ox_id;
- fc_hdr = &(mp_req->req_fc_hdr);
- if (task_type == FCOE_TASK_TYPE_MIDPATH) {
- fc_hdr->fh_ox_id = io_req->xid;
- fc_hdr->fh_rx_id = htons(0xffff);
- } else if (task_type == FCOE_TASK_TYPE_UNSOLICITED) {
- fc_hdr->fh_rx_id = io_req->xid;
- }
+ /* Set up s/g list parameters for request buffer */
+ tx_sgl_task_params.sgl = mp_req->mp_req_bd;
+ tx_sgl_task_params.sgl_phys_addr.lo = U64_LO(mp_req->mp_req_bd_dma);
+ tx_sgl_task_params.sgl_phys_addr.hi = U64_HI(mp_req->mp_req_bd_dma);
+ tx_sgl_task_params.num_sges = 1;
+ /* Set PAGE_SIZE for now since sg element is that size ??? */
+ tx_sgl_task_params.total_buffer_size = io_req->data_xfer_len;
+ tx_sgl_task_params.small_mid_sge = 0;
- /* Fill FC Header into middle path buffer */
- task_fc_hdr->parameter = fc_hdr->fh_parm_offset;
- task_fc_hdr->r_ctl = fc_hdr->fh_r_ctl;
- task_fc_hdr->type = fc_hdr->fh_type;
- task_fc_hdr->cs_ctl = fc_hdr->fh_cs_ctl;
- task_fc_hdr->df_ctl = fc_hdr->fh_df_ctl;
- task_fc_hdr->rx_id = fc_hdr->fh_rx_id;
- task_fc_hdr->ox_id = fc_hdr->fh_ox_id;
+ /* Set up s/g list parameters for request buffer */
+ rx_sgl_task_params.sgl = mp_req->mp_resp_bd;
+ rx_sgl_task_params.sgl_phys_addr.lo = U64_LO(mp_req->mp_resp_bd_dma);
+ rx_sgl_task_params.sgl_phys_addr.hi = U64_HI(mp_req->mp_resp_bd_dma);
+ rx_sgl_task_params.num_sges = 1;
+ /* Set PAGE_SIZE for now since sg element is that size ??? */
+ rx_sgl_task_params.total_buffer_size = PAGE_SIZE;
+ rx_sgl_task_params.small_mid_sge = 0;
- task_ctx->ystorm_st_context.data_2_trns_rem =
- io_req->data_xfer_len;
- task_ctx->ystorm_st_context.task_type = task_type;
- }
- /* TSTORM ONLY */
- {
- task_ctx->tstorm_ag_context.icid = (u16)fcport->fw_cid;
- task_ctx->tstorm_st_context.read_only.cid = fcport->fw_cid;
- /* Always send middle-path repsonses on CQ #0 */
- task_ctx->tstorm_st_context.read_only.glbl_q_num = 0;
- io_req->fp_idx = 0;
- SET_FIELD(task_ctx->tstorm_ag_context.flags0,
- TSTORM_FCOE_TASK_AG_CTX_CONNECTION_TYPE,
- PROTOCOLID_FCOE);
- task_ctx->tstorm_st_context.read_only.task_type = task_type;
- SET_FIELD(task_ctx->tstorm_st_context.read_write.flags,
- FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_EXP_FIRST_FRAME,
- 1);
- task_ctx->tstorm_st_context.read_write.rx_id = 0xffff;
- }
+ /*
+ * Last arg is 0 as previous code did not set that we wanted the
+ * fc header information.
+ */
+ init_initiator_midpath_unsolicited_fcoe_task(io_req->task_params,
+ &task_fc_hdr,
+ &tx_sgl_task_params,
+ &rx_sgl_task_params, 0);
- /* MSTORM only */
- {
- if (task_type == FCOE_TASK_TYPE_MIDPATH) {
- /* Initialize task context */
- data_desc = &task_ctx->mstorm_st_context.fp.data_desc;
-
- /* Set cache sges address and length */
- data_desc->slow.base_sgl_addr.lo =
- U64_LO(mp_req->mp_resp_bd_dma);
- data_desc->slow.base_sgl_addr.hi =
- U64_HI(mp_req->mp_resp_bd_dma);
- data_desc->slow.remainder_num_sges = 1;
- data_desc->slow.curr_sge_off = 0;
- data_desc->slow.curr_sgl_index = 0;
-
- /*
- * Also need to fil in non-fastpath response address
- * for middle path commands.
- */
- task_ctx->mstorm_st_context.non_fp.rsp_buf_addr.lo =
- U64_LO(mp_req->mp_resp_bd_dma);
- task_ctx->mstorm_st_context.non_fp.rsp_buf_addr.hi =
- U64_HI(mp_req->mp_resp_bd_dma);
- }
- }
-
- /* USTORM ONLY */
- {
- task_ctx->ustorm_ag_context.global_cq_num = 0;
- }
-
- /* I/O stats. Middle path commands always use slow SGEs */
- qedf->slow_sge_ios++;
- io_req->sge_type = QEDF_IOREQ_SLOW_SGE;
+ /* Midpath requests always consume 1 SGE */
+ qedf->single_sge_ios++;
}
-void qedf_add_to_sq(struct qedf_rport *fcport, u16 xid, u32 ptu_invalidate,
- enum fcoe_task_type req_type, u32 offset)
+/* Presumed that fcport->rport_lock is held */
+u16 qedf_get_sqe_idx(struct qedf_rport *fcport)
{
- struct fcoe_wqe *sqe;
uint16_t total_sqe = (fcport->sq_mem_size)/(sizeof(struct fcoe_wqe));
+ u16 rval;
- sqe = &fcport->sq[fcport->sq_prod_idx];
+ rval = fcport->sq_prod_idx;
+ /* Adjust ring index */
fcport->sq_prod_idx++;
fcport->fw_sq_prod_idx++;
if (fcport->sq_prod_idx == total_sqe)
fcport->sq_prod_idx = 0;
- switch (req_type) {
- case FCOE_TASK_TYPE_WRITE_INITIATOR:
- case FCOE_TASK_TYPE_READ_INITIATOR:
- SET_FIELD(sqe->flags, FCOE_WQE_REQ_TYPE, SEND_FCOE_CMD);
- if (ptu_invalidate)
- SET_FIELD(sqe->flags, FCOE_WQE_INVALIDATE_PTU, 1);
- break;
- case FCOE_TASK_TYPE_MIDPATH:
- SET_FIELD(sqe->flags, FCOE_WQE_REQ_TYPE, SEND_FCOE_MIDPATH);
- break;
- case FCOE_TASK_TYPE_ABTS:
- SET_FIELD(sqe->flags, FCOE_WQE_REQ_TYPE,
- SEND_FCOE_ABTS_REQUEST);
- break;
- case FCOE_TASK_TYPE_EXCHANGE_CLEANUP:
- SET_FIELD(sqe->flags, FCOE_WQE_REQ_TYPE,
- FCOE_EXCHANGE_CLEANUP);
- break;
- case FCOE_TASK_TYPE_SEQUENCE_CLEANUP:
- SET_FIELD(sqe->flags, FCOE_WQE_REQ_TYPE,
- FCOE_SEQUENCE_RECOVERY);
- /* NOTE: offset param only used for sequence recovery */
- sqe->additional_info_union.seq_rec_updated_offset = offset;
- break;
- case FCOE_TASK_TYPE_UNSOLICITED:
- break;
- default:
- break;
- }
-
- sqe->task_id = xid;
-
- /* Make sure SQ data is coherent */
- wmb();
-
+ return rval;
}
void qedf_ring_doorbell(struct qedf_rport *fcport)
@@ -1029,7 +847,8 @@
struct fcoe_task_context *task_ctx;
u16 xid;
enum fcoe_task_type req_type = 0;
- u32 ptu_invalidate = 0;
+ struct fcoe_wqe *sqe;
+ u16 sqe_idx;
/* Initialize rest of io_req fileds */
io_req->data_xfer_len = scsi_bufflen(sc_cmd);
@@ -1061,6 +880,16 @@
return -EAGAIN;
}
+ if (!test_bit(QEDF_RPORT_SESSION_READY, &fcport->flags)) {
+ QEDF_ERR(&(qedf->dbg_ctx), "Session not offloaded yet.\n");
+ kref_put(&io_req->refcount, qedf_release_cmd);
+ }
+
+ /* Obtain free SQE */
+ sqe_idx = qedf_get_sqe_idx(fcport);
+ sqe = &fcport->sq[sqe_idx];
+ memset(sqe, 0, sizeof(struct fcoe_wqe));
+
/* Get the task context */
task_ctx = qedf_get_task_mem(&qedf->tasks, xid);
if (!task_ctx) {
@@ -1070,15 +899,7 @@
return -EINVAL;
}
- qedf_init_task(fcport, lport, io_req, &ptu_invalidate, task_ctx);
-
- if (!test_bit(QEDF_RPORT_SESSION_READY, &fcport->flags)) {
- QEDF_ERR(&(qedf->dbg_ctx), "Session not offloaded yet.\n");
- kref_put(&io_req->refcount, qedf_release_cmd);
- }
-
- /* Obtain free SQ entry */
- qedf_add_to_sq(fcport, xid, ptu_invalidate, req_type, 0);
+ qedf_init_task(fcport, lport, io_req, task_ctx, sqe);
/* Ring doorbell */
qedf_ring_doorbell(fcport);
@@ -1661,6 +1482,8 @@
u32 r_a_tov = 0;
int rc = 0;
unsigned long flags;
+ struct fcoe_wqe *sqe;
+ u16 sqe_idx;
r_a_tov = rdata->r_a_tov;
lport = qedf->lport;
@@ -1712,10 +1535,12 @@
spin_lock_irqsave(&fcport->rport_lock, flags);
- /* Add ABTS to send queue */
- qedf_add_to_sq(fcport, xid, 0, FCOE_TASK_TYPE_ABTS, 0);
+ sqe_idx = qedf_get_sqe_idx(fcport);
+ sqe = &fcport->sq[sqe_idx];
+ memset(sqe, 0, sizeof(struct fcoe_wqe));
+ io_req->task_params->sqe = sqe;
- /* Ring doorbell */
+ init_initiator_abort_fcoe_task(io_req->task_params);
qedf_ring_doorbell(fcport);
spin_unlock_irqrestore(&fcport->rport_lock, flags);
@@ -1784,8 +1609,8 @@
int qedf_init_mp_req(struct qedf_ioreq *io_req)
{
struct qedf_mp_req *mp_req;
- struct fcoe_sge *mp_req_bd;
- struct fcoe_sge *mp_resp_bd;
+ struct scsi_sge *mp_req_bd;
+ struct scsi_sge *mp_resp_bd;
struct qedf_ctx *qedf = io_req->fcport->qedf;
dma_addr_t addr;
uint64_t sz;
@@ -1819,7 +1644,7 @@
}
/* Allocate and map mp_req_bd and mp_resp_bd */
- sz = sizeof(struct fcoe_sge);
+ sz = sizeof(struct scsi_sge);
mp_req->mp_req_bd = dma_alloc_coherent(&qedf->pdev->dev, sz,
&mp_req->mp_req_bd_dma, GFP_KERNEL);
if (!mp_req->mp_req_bd) {
@@ -1841,7 +1666,7 @@
mp_req_bd = mp_req->mp_req_bd;
mp_req_bd->sge_addr.lo = U64_LO(addr);
mp_req_bd->sge_addr.hi = U64_HI(addr);
- mp_req_bd->size = QEDF_PAGE_SIZE;
+ mp_req_bd->sge_len = QEDF_PAGE_SIZE;
/*
* MP buffer is either a task mgmt command or an ELS.
@@ -1852,7 +1677,7 @@
addr = mp_req->resp_buf_dma;
mp_resp_bd->sge_addr.lo = U64_LO(addr);
mp_resp_bd->sge_addr.hi = U64_HI(addr);
- mp_resp_bd->size = QEDF_PAGE_SIZE;
+ mp_resp_bd->sge_len = QEDF_PAGE_SIZE;
return 0;
}
@@ -1895,6 +1720,8 @@
int tmo = 0;
int rc = SUCCESS;
unsigned long flags;
+ struct fcoe_wqe *sqe;
+ u16 sqe_idx;
fcport = io_req->fcport;
if (!fcport) {
@@ -1940,12 +1767,16 @@
init_completion(&io_req->tm_done);
- /* Obtain free SQ entry */
spin_lock_irqsave(&fcport->rport_lock, flags);
- qedf_add_to_sq(fcport, xid, 0, FCOE_TASK_TYPE_EXCHANGE_CLEANUP, 0);
- /* Ring doorbell */
+ sqe_idx = qedf_get_sqe_idx(fcport);
+ sqe = &fcport->sq[sqe_idx];
+ memset(sqe, 0, sizeof(struct fcoe_wqe));
+ io_req->task_params->sqe = sqe;
+
+ init_initiator_cleanup_fcoe_task(io_req->task_params);
qedf_ring_doorbell(fcport);
+
spin_unlock_irqrestore(&fcport->rport_lock, flags);
tmo = wait_for_completion_timeout(&io_req->tm_done,
@@ -1991,16 +1822,15 @@
uint8_t tm_flags)
{
struct qedf_ioreq *io_req;
- struct qedf_mp_req *tm_req;
struct fcoe_task_context *task;
- struct fc_frame_header *fc_hdr;
- struct fcp_cmnd *fcp_cmnd;
struct qedf_ctx *qedf = fcport->qedf;
+ struct fc_lport *lport = qedf->lport;
int rc = 0;
uint16_t xid;
- uint32_t sid, did;
int tmo = 0;
unsigned long flags;
+ struct fcoe_wqe *sqe;
+ u16 sqe_idx;
if (!sc_cmd) {
QEDF_ERR(&(qedf->dbg_ctx), "invalid arg\n");
@@ -2031,36 +1861,14 @@
/* Set the return CPU to be the same as the request one */
io_req->cpu = smp_processor_id();
- tm_req = (struct qedf_mp_req *)&(io_req->mp_req);
-
- rc = qedf_init_mp_req(io_req);
- if (rc == FAILED) {
- QEDF_ERR(&(qedf->dbg_ctx), "Task mgmt MP request init "
- "failed\n");
- kref_put(&io_req->refcount, qedf_release_cmd);
- goto reset_tmf_err;
- }
-
/* Set TM flags */
- io_req->io_req_flags = 0;
- tm_req->tm_flags = tm_flags;
+ io_req->io_req_flags = QEDF_READ;
+ io_req->data_xfer_len = 0;
+ io_req->tm_flags = tm_flags;
/* Default is to return a SCSI command when an error occurs */
io_req->return_scsi_cmd_on_abts = true;
- /* Fill FCP_CMND */
- qedf_build_fcp_cmnd(io_req, (struct fcp_cmnd *)tm_req->req_buf);
- fcp_cmnd = (struct fcp_cmnd *)tm_req->req_buf;
- memset(fcp_cmnd->fc_cdb, 0, FCP_CMND_LEN);
- fcp_cmnd->fc_dl = 0;
-
- /* Fill FC header */
- fc_hdr = &(tm_req->req_fc_hdr);
- sid = fcport->sid;
- did = fcport->rdata->ids.port_id;
- __fc_fill_fc_hdr(fc_hdr, FC_RCTL_DD_UNSOL_CMD, sid, did,
- FC_TYPE_FCP, FC_FC_FIRST_SEQ | FC_FC_END_SEQ |
- FC_FC_SEQ_INIT, 0);
/* Obtain exchange id */
xid = io_req->xid;
@@ -2069,16 +1877,18 @@
/* Initialize task context for this IO request */
task = qedf_get_task_mem(&qedf->tasks, xid);
- qedf_init_mp_task(io_req, task);
init_completion(&io_req->tm_done);
- /* Obtain free SQ entry */
spin_lock_irqsave(&fcport->rport_lock, flags);
- qedf_add_to_sq(fcport, xid, 0, FCOE_TASK_TYPE_MIDPATH, 0);
- /* Ring doorbell */
+ sqe_idx = qedf_get_sqe_idx(fcport);
+ sqe = &fcport->sq[sqe_idx];
+ memset(sqe, 0, sizeof(struct fcoe_wqe));
+
+ qedf_init_task(fcport, lport, io_req, task, sqe);
qedf_ring_doorbell(fcport);
+
spin_unlock_irqrestore(&fcport->rport_lock, flags);
tmo = wait_for_completion_timeout(&io_req->tm_done,
@@ -2162,14 +1972,6 @@
struct qedf_ioreq *io_req)
{
struct fcoe_cqe_rsp_info *fcp_rsp;
- struct fcoe_cqe_midpath_info *mp_info;
-
-
- /* Get TMF response length from CQE */
- mp_info = &cqe->cqe_info.midpath_info;
- io_req->mp_req.resp_len = mp_info->data_placement_size;
- QEDF_INFO(&(qedf->dbg_ctx), QEDF_LOG_SCSI_TM,
- "Response len is %d.\n", io_req->mp_req.resp_len);
fcp_rsp = &cqe->cqe_info.rsp_info;
qedf_parse_fcp_rsp(io_req, fcp_rsp);
diff --git a/drivers/scsi/qedi/Makefile b/drivers/scsi/qedi/Makefile
index 2b3e16b..90a6925 100644
--- a/drivers/scsi/qedi/Makefile
+++ b/drivers/scsi/qedi/Makefile
@@ -1,5 +1,5 @@
obj-$(CONFIG_QEDI) := qedi.o
qedi-y := qedi_main.o qedi_iscsi.o qedi_fw.o qedi_sysfs.o \
- qedi_dbg.o
+ qedi_dbg.o qedi_fw_api.o
qedi-$(CONFIG_DEBUG_FS) += qedi_debugfs.o
diff --git a/drivers/scsi/qedi/qedi_fw.c b/drivers/scsi/qedi/qedi_fw.c
index 2bce3ef..d6978cb 100644
--- a/drivers/scsi/qedi/qedi_fw.c
+++ b/drivers/scsi/qedi/qedi_fw.c
@@ -14,6 +14,8 @@
#include "qedi.h"
#include "qedi_iscsi.h"
#include "qedi_gbl.h"
+#include "qedi_fw_iscsi.h"
+#include "qedi_fw_scsi.h"
static int qedi_send_iscsi_tmf(struct qedi_conn *qedi_conn,
struct iscsi_task *mtask);
@@ -53,8 +55,8 @@
resp_hdr->exp_cmdsn = cpu_to_be32(cqe_logout_response->exp_cmd_sn);
resp_hdr->max_cmdsn = cpu_to_be32(cqe_logout_response->max_cmd_sn);
- resp_hdr->t2wait = cpu_to_be32(cqe_logout_response->time2wait);
- resp_hdr->t2retain = cpu_to_be32(cqe_logout_response->time2retain);
+ resp_hdr->t2wait = cpu_to_be32(cqe_logout_response->time_2_wait);
+ resp_hdr->t2retain = cpu_to_be32(cqe_logout_response->time_2_retain);
QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_TID,
"Freeing tid=0x%x for cid=0x%x\n",
@@ -975,81 +977,6 @@
return;
}
-static void qedi_add_to_sq(struct qedi_conn *qedi_conn, struct iscsi_task *task,
- u16 tid, uint16_t ptu_invalidate, int is_cleanup)
-{
- struct iscsi_wqe *wqe;
- struct iscsi_wqe_field *cont_field;
- struct qedi_endpoint *ep;
- struct scsi_cmnd *sc = task->sc;
- struct iscsi_login_req *login_hdr;
- struct qedi_cmd *cmd = task->dd_data;
-
- login_hdr = (struct iscsi_login_req *)task->hdr;
- ep = qedi_conn->ep;
- wqe = &ep->sq[ep->sq_prod_idx];
-
- memset(wqe, 0, sizeof(*wqe));
-
- ep->sq_prod_idx++;
- ep->fw_sq_prod_idx++;
- if (ep->sq_prod_idx == QEDI_SQ_SIZE)
- ep->sq_prod_idx = 0;
-
- if (is_cleanup) {
- SET_FIELD(wqe->flags, ISCSI_WQE_WQE_TYPE,
- ISCSI_WQE_TYPE_TASK_CLEANUP);
- wqe->task_id = tid;
- return;
- }
-
- if (ptu_invalidate) {
- SET_FIELD(wqe->flags, ISCSI_WQE_PTU_INVALIDATE,
- ISCSI_WQE_SET_PTU_INVALIDATE);
- }
-
- cont_field = &wqe->cont_prevtid_union.cont_field;
-
- switch (task->hdr->opcode & ISCSI_OPCODE_MASK) {
- case ISCSI_OP_LOGIN:
- case ISCSI_OP_TEXT:
- SET_FIELD(wqe->flags, ISCSI_WQE_WQE_TYPE,
- ISCSI_WQE_TYPE_MIDDLE_PATH);
- SET_FIELD(wqe->flags, ISCSI_WQE_NUM_FAST_SGES,
- 1);
- cont_field->contlen_cdbsize_field = ntoh24(login_hdr->dlength);
- break;
- case ISCSI_OP_LOGOUT:
- case ISCSI_OP_NOOP_OUT:
- case ISCSI_OP_SCSI_TMFUNC:
- SET_FIELD(wqe->flags, ISCSI_WQE_WQE_TYPE,
- ISCSI_WQE_TYPE_NORMAL);
- break;
- default:
- if (!sc)
- break;
-
- SET_FIELD(wqe->flags, ISCSI_WQE_WQE_TYPE,
- ISCSI_WQE_TYPE_NORMAL);
- cont_field->contlen_cdbsize_field =
- (sc->sc_data_direction == DMA_TO_DEVICE) ?
- scsi_bufflen(sc) : 0;
- if (cmd->use_slowpath)
- SET_FIELD(wqe->flags, ISCSI_WQE_NUM_FAST_SGES, 0);
- else
- SET_FIELD(wqe->flags, ISCSI_WQE_NUM_FAST_SGES,
- (sc->sc_data_direction ==
- DMA_TO_DEVICE) ?
- min((u16)QEDI_FAST_SGE_COUNT,
- (u16)cmd->io_tbl.sge_valid) : 0);
- break;
- }
-
- wqe->task_id = tid;
- /* Make sure SQ data is coherent */
- wmb();
-}
-
static void qedi_ring_doorbell(struct qedi_conn *qedi_conn)
{
struct iscsi_db_data dbell = { 0 };
@@ -1076,96 +1003,116 @@
qedi_conn->iscsi_conn_id);
}
+static u16 qedi_get_wqe_idx(struct qedi_conn *qedi_conn)
+{
+ struct qedi_endpoint *ep;
+ u16 rval;
+
+ ep = qedi_conn->ep;
+ rval = ep->sq_prod_idx;
+
+ /* Increament SQ index */
+ ep->sq_prod_idx++;
+ ep->fw_sq_prod_idx++;
+ if (ep->sq_prod_idx == QEDI_SQ_SIZE)
+ ep->sq_prod_idx = 0;
+
+ return rval;
+}
+
int qedi_send_iscsi_login(struct qedi_conn *qedi_conn,
struct iscsi_task *task)
{
- struct qedi_ctx *qedi = qedi_conn->qedi;
+ struct iscsi_login_req_hdr login_req_pdu_header;
+ struct scsi_sgl_task_params tx_sgl_task_params;
+ struct scsi_sgl_task_params rx_sgl_task_params;
+ struct iscsi_task_params task_params;
struct iscsi_task_context *fw_task_ctx;
+ struct qedi_ctx *qedi = qedi_conn->qedi;
struct iscsi_login_req *login_hdr;
- struct iscsi_login_req_hdr *fw_login_req = NULL;
- struct iscsi_cached_sge_ctx *cached_sge = NULL;
- struct iscsi_sge *single_sge = NULL;
- struct iscsi_sge *req_sge = NULL;
- struct iscsi_sge *resp_sge = NULL;
+ struct scsi_sge *req_sge = NULL;
+ struct scsi_sge *resp_sge = NULL;
struct qedi_cmd *qedi_cmd;
- s16 ptu_invalidate = 0;
+ struct qedi_endpoint *ep;
s16 tid = 0;
+ u16 sq_idx = 0;
+ int rval = 0;
- req_sge = (struct iscsi_sge *)qedi_conn->gen_pdu.req_bd_tbl;
- resp_sge = (struct iscsi_sge *)qedi_conn->gen_pdu.resp_bd_tbl;
+ req_sge = (struct scsi_sge *)qedi_conn->gen_pdu.req_bd_tbl;
+ resp_sge = (struct scsi_sge *)qedi_conn->gen_pdu.resp_bd_tbl;
qedi_cmd = (struct qedi_cmd *)task->dd_data;
+ ep = qedi_conn->ep;
login_hdr = (struct iscsi_login_req *)task->hdr;
tid = qedi_get_task_idx(qedi);
if (tid == -1)
return -ENOMEM;
- fw_task_ctx = qedi_get_task_mem(&qedi->tasks, tid);
+ fw_task_ctx =
+ (struct iscsi_task_context *)qedi_get_task_mem(&qedi->tasks, tid);
memset(fw_task_ctx, 0, sizeof(struct iscsi_task_context));
qedi_cmd->task_id = tid;
- /* Ystorm context */
- fw_login_req = &fw_task_ctx->ystorm_st_context.pdu_hdr.login_req;
- fw_login_req->opcode = login_hdr->opcode;
- fw_login_req->version_min = login_hdr->min_version;
- fw_login_req->version_max = login_hdr->max_version;
- fw_login_req->flags_attr = login_hdr->flags;
- fw_login_req->isid_tabc = *((u16 *)login_hdr->isid + 2);
- fw_login_req->isid_d = *((u32 *)login_hdr->isid);
- fw_login_req->tsih = login_hdr->tsih;
+ memset(&task_params, 0, sizeof(task_params));
+ memset(&login_req_pdu_header, 0, sizeof(login_req_pdu_header));
+ memset(&tx_sgl_task_params, 0, sizeof(tx_sgl_task_params));
+ memset(&rx_sgl_task_params, 0, sizeof(rx_sgl_task_params));
+ /* Update header info */
+ login_req_pdu_header.opcode = login_hdr->opcode;
+ login_req_pdu_header.version_min = login_hdr->min_version;
+ login_req_pdu_header.version_max = login_hdr->max_version;
+ login_req_pdu_header.flags_attr = login_hdr->flags;
+ login_req_pdu_header.isid_tabc = swab32p((u32 *)login_hdr->isid);
+ login_req_pdu_header.isid_d = swab16p((u16 *)&login_hdr->isid[4]);
+
+ login_req_pdu_header.tsih = login_hdr->tsih;
+ login_req_pdu_header.hdr_second_dword = ntoh24(login_hdr->dlength);
+
qedi_update_itt_map(qedi, tid, task->itt, qedi_cmd);
- fw_login_req->itt = qedi_set_itt(tid, get_itt(task->itt));
- fw_login_req->cid = qedi_conn->iscsi_conn_id;
- fw_login_req->cmd_sn = be32_to_cpu(login_hdr->cmdsn);
- fw_login_req->exp_stat_sn = be32_to_cpu(login_hdr->exp_statsn);
- fw_login_req->exp_stat_sn = 0;
+ login_req_pdu_header.itt = qedi_set_itt(tid, get_itt(task->itt));
+ login_req_pdu_header.cid = qedi_conn->iscsi_conn_id;
+ login_req_pdu_header.cmd_sn = be32_to_cpu(login_hdr->cmdsn);
+ login_req_pdu_header.exp_stat_sn = be32_to_cpu(login_hdr->exp_statsn);
+ login_req_pdu_header.exp_stat_sn = 0;
- if (qedi->tid_reuse_count[tid] == QEDI_MAX_TASK_NUM) {
- ptu_invalidate = 1;
- qedi->tid_reuse_count[tid] = 0;
- }
+ /* Fill tx AHS and rx buffer */
+ tx_sgl_task_params.sgl =
+ (struct scsi_sge *)qedi_conn->gen_pdu.req_bd_tbl;
+ tx_sgl_task_params.sgl_phys_addr.lo =
+ (u32)(qedi_conn->gen_pdu.req_dma_addr);
+ tx_sgl_task_params.sgl_phys_addr.hi =
+ (u32)((u64)qedi_conn->gen_pdu.req_dma_addr >> 32);
+ tx_sgl_task_params.total_buffer_size = ntoh24(login_hdr->dlength);
+ tx_sgl_task_params.num_sges = 1;
- fw_task_ctx->ystorm_st_context.state.reuse_count =
- qedi->tid_reuse_count[tid];
- fw_task_ctx->mstorm_st_context.reuse_count =
- qedi->tid_reuse_count[tid]++;
- cached_sge =
- &fw_task_ctx->ystorm_st_context.state.sgl_ctx_union.cached_sge;
- cached_sge->sge.sge_len = req_sge->sge_len;
- cached_sge->sge.sge_addr.lo = (u32)(qedi_conn->gen_pdu.req_dma_addr);
- cached_sge->sge.sge_addr.hi =
- (u32)((u64)qedi_conn->gen_pdu.req_dma_addr >> 32);
+ rx_sgl_task_params.sgl =
+ (struct scsi_sge *)qedi_conn->gen_pdu.resp_bd_tbl;
+ rx_sgl_task_params.sgl_phys_addr.lo =
+ (u32)(qedi_conn->gen_pdu.resp_dma_addr);
+ rx_sgl_task_params.sgl_phys_addr.hi =
+ (u32)((u64)qedi_conn->gen_pdu.resp_dma_addr >> 32);
+ rx_sgl_task_params.total_buffer_size = resp_sge->sge_len;
+ rx_sgl_task_params.num_sges = 1;
- /* Mstorm context */
- single_sge = &fw_task_ctx->mstorm_st_context.sgl_union.single_sge;
- fw_task_ctx->mstorm_st_context.task_type = 0x2;
- fw_task_ctx->mstorm_ag_context.task_cid = (u16)qedi_conn->iscsi_conn_id;
- single_sge->sge_addr.lo = resp_sge->sge_addr.lo;
- single_sge->sge_addr.hi = resp_sge->sge_addr.hi;
- single_sge->sge_len = resp_sge->sge_len;
+ /* Fill fw input params */
+ task_params.context = fw_task_ctx;
+ task_params.conn_icid = (u16)qedi_conn->iscsi_conn_id;
+ task_params.itid = tid;
+ task_params.cq_rss_number = 0;
+ task_params.tx_io_size = ntoh24(login_hdr->dlength);
+ task_params.rx_io_size = resp_sge->sge_len;
- SET_FIELD(fw_task_ctx->mstorm_st_context.flags.mflags,
- ISCSI_MFLAGS_SINGLE_SGE, 1);
- SET_FIELD(fw_task_ctx->mstorm_st_context.flags.mflags,
- ISCSI_MFLAGS_SLOW_IO, 0);
- fw_task_ctx->mstorm_st_context.sgl_size = 1;
- fw_task_ctx->mstorm_st_context.rem_task_size = resp_sge->sge_len;
+ sq_idx = qedi_get_wqe_idx(qedi_conn);
+ task_params.sqe = &ep->sq[sq_idx];
- /* Ustorm context */
- fw_task_ctx->ustorm_st_context.rem_rcv_len = resp_sge->sge_len;
- fw_task_ctx->ustorm_st_context.exp_data_transfer_len =
- ntoh24(login_hdr->dlength);
- fw_task_ctx->ustorm_st_context.exp_data_sn = 0;
- fw_task_ctx->ustorm_st_context.cq_rss_number = 0;
- fw_task_ctx->ustorm_st_context.task_type = 0x2;
- fw_task_ctx->ustorm_ag_context.icid = (u16)qedi_conn->iscsi_conn_id;
- fw_task_ctx->ustorm_ag_context.exp_data_acked =
- ntoh24(login_hdr->dlength);
- SET_FIELD(fw_task_ctx->ustorm_ag_context.flags1,
- USTORM_ISCSI_TASK_AG_CTX_R2T2RECV, 1);
- SET_FIELD(fw_task_ctx->ustorm_st_context.flags,
- USTORM_ISCSI_TASK_ST_CTX_LOCAL_COMP, 0);
+ memset(task_params.sqe, 0, sizeof(struct iscsi_wqe));
+ rval = init_initiator_login_request_task(&task_params,
+ &login_req_pdu_header,
+ &tx_sgl_task_params,
+ &rx_sgl_task_params);
+ if (rval)
+ return -1;
spin_lock(&qedi_conn->list_lock);
list_add_tail(&qedi_cmd->io_cmd, &qedi_conn->active_cmd_list);
@@ -1173,7 +1120,6 @@
qedi_conn->active_cmd_count++;
spin_unlock(&qedi_conn->list_lock);
- qedi_add_to_sq(qedi_conn, task, tid, ptu_invalidate, false);
qedi_ring_doorbell(qedi_conn);
return 0;
}
@@ -1181,65 +1127,64 @@
int qedi_send_iscsi_logout(struct qedi_conn *qedi_conn,
struct iscsi_task *task)
{
- struct qedi_ctx *qedi = qedi_conn->qedi;
- struct iscsi_logout_req_hdr *fw_logout_req = NULL;
- struct iscsi_task_context *fw_task_ctx = NULL;
+ struct iscsi_logout_req_hdr logout_pdu_header;
+ struct scsi_sgl_task_params tx_sgl_task_params;
+ struct scsi_sgl_task_params rx_sgl_task_params;
+ struct iscsi_task_params task_params;
+ struct iscsi_task_context *fw_task_ctx;
struct iscsi_logout *logout_hdr = NULL;
- struct qedi_cmd *qedi_cmd = NULL;
- s16 tid = 0;
- s16 ptu_invalidate = 0;
+ struct qedi_ctx *qedi = qedi_conn->qedi;
+ struct qedi_cmd *qedi_cmd;
+ struct qedi_endpoint *ep;
+ s16 tid = 0;
+ u16 sq_idx = 0;
+ int rval = 0;
qedi_cmd = (struct qedi_cmd *)task->dd_data;
logout_hdr = (struct iscsi_logout *)task->hdr;
+ ep = qedi_conn->ep;
tid = qedi_get_task_idx(qedi);
if (tid == -1)
return -ENOMEM;
- fw_task_ctx = qedi_get_task_mem(&qedi->tasks, tid);
-
+ fw_task_ctx =
+ (struct iscsi_task_context *)qedi_get_task_mem(&qedi->tasks, tid);
memset(fw_task_ctx, 0, sizeof(struct iscsi_task_context));
+
qedi_cmd->task_id = tid;
- /* Ystorm context */
- fw_logout_req = &fw_task_ctx->ystorm_st_context.pdu_hdr.logout_req;
- fw_logout_req->opcode = ISCSI_OPCODE_LOGOUT_REQUEST;
- fw_logout_req->reason_code = 0x80 | logout_hdr->flags;
+ memset(&task_params, 0, sizeof(task_params));
+ memset(&logout_pdu_header, 0, sizeof(logout_pdu_header));
+ memset(&tx_sgl_task_params, 0, sizeof(tx_sgl_task_params));
+ memset(&rx_sgl_task_params, 0, sizeof(rx_sgl_task_params));
+
+ /* Update header info */
+ logout_pdu_header.opcode = logout_hdr->opcode;
+ logout_pdu_header.reason_code = 0x80 | logout_hdr->flags;
qedi_update_itt_map(qedi, tid, task->itt, qedi_cmd);
- fw_logout_req->itt = qedi_set_itt(tid, get_itt(task->itt));
- fw_logout_req->exp_stat_sn = be32_to_cpu(logout_hdr->exp_statsn);
- fw_logout_req->cmd_sn = be32_to_cpu(logout_hdr->cmdsn);
+ logout_pdu_header.itt = qedi_set_itt(tid, get_itt(task->itt));
+ logout_pdu_header.exp_stat_sn = be32_to_cpu(logout_hdr->exp_statsn);
+ logout_pdu_header.cmd_sn = be32_to_cpu(logout_hdr->cmdsn);
+ logout_pdu_header.cid = qedi_conn->iscsi_conn_id;
- if (qedi->tid_reuse_count[tid] == QEDI_MAX_TASK_NUM) {
- ptu_invalidate = 1;
- qedi->tid_reuse_count[tid] = 0;
- }
- fw_task_ctx->ystorm_st_context.state.reuse_count =
- qedi->tid_reuse_count[tid];
- fw_task_ctx->mstorm_st_context.reuse_count =
- qedi->tid_reuse_count[tid]++;
- fw_logout_req->cid = qedi_conn->iscsi_conn_id;
- fw_task_ctx->ystorm_st_context.state.buffer_offset[0] = 0;
+ /* Fill fw input params */
+ task_params.context = fw_task_ctx;
+ task_params.conn_icid = (u16)qedi_conn->iscsi_conn_id;
+ task_params.itid = tid;
+ task_params.cq_rss_number = 0;
+ task_params.tx_io_size = 0;
+ task_params.rx_io_size = 0;
- /* Mstorm context */
- fw_task_ctx->mstorm_st_context.task_type = ISCSI_TASK_TYPE_MIDPATH;
- fw_task_ctx->mstorm_ag_context.task_cid = (u16)qedi_conn->iscsi_conn_id;
+ sq_idx = qedi_get_wqe_idx(qedi_conn);
+ task_params.sqe = &ep->sq[sq_idx];
+ memset(task_params.sqe, 0, sizeof(struct iscsi_wqe));
- /* Ustorm context */
- fw_task_ctx->ustorm_st_context.rem_rcv_len = 0;
- fw_task_ctx->ustorm_st_context.exp_data_transfer_len = 0;
- fw_task_ctx->ustorm_st_context.exp_data_sn = 0;
- fw_task_ctx->ustorm_st_context.task_type = ISCSI_TASK_TYPE_MIDPATH;
- fw_task_ctx->ustorm_st_context.cq_rss_number = 0;
-
- SET_FIELD(fw_task_ctx->ustorm_st_context.flags,
- USTORM_ISCSI_TASK_ST_CTX_LOCAL_COMP, 0);
- SET_FIELD(fw_task_ctx->ustorm_st_context.reg1.reg1_map,
- ISCSI_REG1_NUM_FAST_SGES, 0);
-
- fw_task_ctx->ustorm_ag_context.icid = (u16)qedi_conn->iscsi_conn_id;
- SET_FIELD(fw_task_ctx->ustorm_ag_context.flags1,
- USTORM_ISCSI_TASK_AG_CTX_R2T2RECV, 1);
+ rval = init_initiator_logout_request_task(&task_params,
+ &logout_pdu_header,
+ NULL, NULL);
+ if (rval)
+ return -1;
spin_lock(&qedi_conn->list_lock);
list_add_tail(&qedi_cmd->io_cmd, &qedi_conn->active_cmd_list);
@@ -1247,9 +1192,7 @@
qedi_conn->active_cmd_count++;
spin_unlock(&qedi_conn->list_lock);
- qedi_add_to_sq(qedi_conn, task, tid, ptu_invalidate, false);
qedi_ring_doorbell(qedi_conn);
-
return 0;
}
@@ -1533,47 +1476,46 @@
static int qedi_send_iscsi_tmf(struct qedi_conn *qedi_conn,
struct iscsi_task *mtask)
{
- struct iscsi_conn *conn = qedi_conn->cls_conn->dd_data;
+ struct iscsi_tmf_request_hdr tmf_pdu_header;
+ struct iscsi_task_params task_params;
struct qedi_ctx *qedi = qedi_conn->qedi;
struct iscsi_task_context *fw_task_ctx;
- struct iscsi_tmf_request_hdr *fw_tmf_request;
- struct iscsi_sge *single_sge;
- struct qedi_cmd *qedi_cmd;
- struct qedi_cmd *cmd;
+ struct iscsi_conn *conn = qedi_conn->cls_conn->dd_data;
struct iscsi_task *ctask;
struct iscsi_tm *tmf_hdr;
- struct iscsi_sge *req_sge;
- struct iscsi_sge *resp_sge;
- u32 lun[2];
- s16 tid = 0, ptu_invalidate = 0;
+ struct qedi_cmd *qedi_cmd;
+ struct qedi_cmd *cmd;
+ struct qedi_endpoint *ep;
+ u32 scsi_lun[2];
+ s16 tid = 0;
+ u16 sq_idx = 0;
+ int rval = 0;
- req_sge = (struct iscsi_sge *)qedi_conn->gen_pdu.req_bd_tbl;
- resp_sge = (struct iscsi_sge *)qedi_conn->gen_pdu.resp_bd_tbl;
- qedi_cmd = (struct qedi_cmd *)mtask->dd_data;
tmf_hdr = (struct iscsi_tm *)mtask->hdr;
+ qedi_cmd = (struct qedi_cmd *)mtask->dd_data;
+ ep = qedi_conn->ep;
- tid = qedi_cmd->task_id;
- qedi_update_itt_map(qedi, tid, mtask->itt, qedi_cmd);
+ tid = qedi_get_task_idx(qedi);
+ if (tid == -1)
+ return -ENOMEM;
- fw_task_ctx = qedi_get_task_mem(&qedi->tasks, tid);
+ fw_task_ctx =
+ (struct iscsi_task_context *)qedi_get_task_mem(&qedi->tasks, tid);
memset(fw_task_ctx, 0, sizeof(struct iscsi_task_context));
- fw_tmf_request = &fw_task_ctx->ystorm_st_context.pdu_hdr.tmf_request;
- fw_tmf_request->itt = qedi_set_itt(tid, get_itt(mtask->itt));
- fw_tmf_request->cmd_sn = be32_to_cpu(tmf_hdr->cmdsn);
+ qedi_cmd->task_id = tid;
- memcpy(lun, &tmf_hdr->lun, sizeof(struct scsi_lun));
- fw_tmf_request->lun.lo = be32_to_cpu(lun[0]);
- fw_tmf_request->lun.hi = be32_to_cpu(lun[1]);
+ memset(&task_params, 0, sizeof(task_params));
+ memset(&tmf_pdu_header, 0, sizeof(tmf_pdu_header));
- if (qedi->tid_reuse_count[tid] == QEDI_MAX_TASK_NUM) {
- ptu_invalidate = 1;
- qedi->tid_reuse_count[tid] = 0;
- }
- fw_task_ctx->ystorm_st_context.state.reuse_count =
- qedi->tid_reuse_count[tid];
- fw_task_ctx->mstorm_st_context.reuse_count =
- qedi->tid_reuse_count[tid]++;
+ /* Update header info */
+ qedi_update_itt_map(qedi, tid, mtask->itt, qedi_cmd);
+ tmf_pdu_header.itt = qedi_set_itt(tid, get_itt(mtask->itt));
+ tmf_pdu_header.cmd_sn = be32_to_cpu(tmf_hdr->cmdsn);
+
+ memcpy(scsi_lun, &tmf_hdr->lun, sizeof(struct scsi_lun));
+ tmf_pdu_header.lun.lo = be32_to_cpu(scsi_lun[0]);
+ tmf_pdu_header.lun.hi = be32_to_cpu(scsi_lun[1]);
if ((tmf_hdr->flags & ISCSI_FLAG_TM_FUNC_MASK) ==
ISCSI_TM_FUNC_ABORT_TASK) {
@@ -1584,53 +1526,34 @@
return 0;
}
cmd = (struct qedi_cmd *)ctask->dd_data;
- fw_tmf_request->rtt =
+ tmf_pdu_header.rtt =
qedi_set_itt(cmd->task_id,
get_itt(tmf_hdr->rtt));
} else {
- fw_tmf_request->rtt = ISCSI_RESERVED_TAG;
+ tmf_pdu_header.rtt = ISCSI_RESERVED_TAG;
}
- fw_tmf_request->opcode = tmf_hdr->opcode;
- fw_tmf_request->function = tmf_hdr->flags;
- fw_tmf_request->hdr_second_dword = ntoh24(tmf_hdr->dlength);
- fw_tmf_request->ref_cmd_sn = be32_to_cpu(tmf_hdr->refcmdsn);
+ tmf_pdu_header.opcode = tmf_hdr->opcode;
+ tmf_pdu_header.function = tmf_hdr->flags;
+ tmf_pdu_header.hdr_second_dword = ntoh24(tmf_hdr->dlength);
+ tmf_pdu_header.ref_cmd_sn = be32_to_cpu(tmf_hdr->refcmdsn);
- single_sge = &fw_task_ctx->mstorm_st_context.sgl_union.single_sge;
- fw_task_ctx->mstorm_st_context.task_type = ISCSI_TASK_TYPE_MIDPATH;
- fw_task_ctx->mstorm_ag_context.task_cid = (u16)qedi_conn->iscsi_conn_id;
- single_sge->sge_addr.lo = resp_sge->sge_addr.lo;
- single_sge->sge_addr.hi = resp_sge->sge_addr.hi;
- single_sge->sge_len = resp_sge->sge_len;
+ /* Fill fw input params */
+ task_params.context = fw_task_ctx;
+ task_params.conn_icid = (u16)qedi_conn->iscsi_conn_id;
+ task_params.itid = tid;
+ task_params.cq_rss_number = 0;
+ task_params.tx_io_size = 0;
+ task_params.rx_io_size = 0;
- SET_FIELD(fw_task_ctx->mstorm_st_context.flags.mflags,
- ISCSI_MFLAGS_SINGLE_SGE, 1);
- SET_FIELD(fw_task_ctx->mstorm_st_context.flags.mflags,
- ISCSI_MFLAGS_SLOW_IO, 0);
- fw_task_ctx->mstorm_st_context.sgl_size = 1;
- fw_task_ctx->mstorm_st_context.rem_task_size = resp_sge->sge_len;
+ sq_idx = qedi_get_wqe_idx(qedi_conn);
+ task_params.sqe = &ep->sq[sq_idx];
- /* Ustorm context */
- fw_task_ctx->ustorm_st_context.rem_rcv_len = 0;
- fw_task_ctx->ustorm_st_context.exp_data_transfer_len = 0;
- fw_task_ctx->ustorm_st_context.exp_data_sn = 0;
- fw_task_ctx->ustorm_st_context.task_type = ISCSI_TASK_TYPE_MIDPATH;
- fw_task_ctx->ustorm_st_context.cq_rss_number = 0;
-
- SET_FIELD(fw_task_ctx->ustorm_st_context.flags,
- USTORM_ISCSI_TASK_ST_CTX_LOCAL_COMP, 0);
- SET_FIELD(fw_task_ctx->ustorm_st_context.reg1.reg1_map,
- ISCSI_REG1_NUM_FAST_SGES, 0);
-
- fw_task_ctx->ustorm_ag_context.icid = (u16)qedi_conn->iscsi_conn_id;
- SET_FIELD(fw_task_ctx->ustorm_ag_context.flags1,
- USTORM_ISCSI_TASK_AG_CTX_R2T2RECV, 1);
- fw_task_ctx->ustorm_st_context.lun.lo = be32_to_cpu(lun[0]);
- fw_task_ctx->ustorm_st_context.lun.hi = be32_to_cpu(lun[1]);
-
- QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_SCSI_TM,
- "Add TMF to SQ, tmf tid=0x%x, itt=0x%x, cid=0x%x\n",
- tid, mtask->itt, qedi_conn->iscsi_conn_id);
+ memset(task_params.sqe, 0, sizeof(struct iscsi_wqe));
+ rval = init_initiator_tmf_request_task(&task_params,
+ &tmf_pdu_header);
+ if (rval)
+ return -1;
spin_lock(&qedi_conn->list_lock);
list_add_tail(&qedi_cmd->io_cmd, &qedi_conn->active_cmd_list);
@@ -1638,7 +1561,6 @@
qedi_conn->active_cmd_count++;
spin_unlock(&qedi_conn->list_lock);
- qedi_add_to_sq(qedi_conn, mtask, tid, ptu_invalidate, false);
qedi_ring_doorbell(qedi_conn);
return 0;
}
@@ -1689,101 +1611,98 @@
int qedi_send_iscsi_text(struct qedi_conn *qedi_conn,
struct iscsi_task *task)
{
- struct qedi_ctx *qedi = qedi_conn->qedi;
+ struct iscsi_text_request_hdr text_request_pdu_header;
+ struct scsi_sgl_task_params tx_sgl_task_params;
+ struct scsi_sgl_task_params rx_sgl_task_params;
+ struct iscsi_task_params task_params;
struct iscsi_task_context *fw_task_ctx;
- struct iscsi_text_request_hdr *fw_text_request;
- struct iscsi_cached_sge_ctx *cached_sge;
- struct iscsi_sge *single_sge;
- struct qedi_cmd *qedi_cmd;
- /* For 6.5 hdr iscsi_hdr */
+ struct qedi_ctx *qedi = qedi_conn->qedi;
struct iscsi_text *text_hdr;
- struct iscsi_sge *req_sge;
- struct iscsi_sge *resp_sge;
- s16 ptu_invalidate = 0;
+ struct scsi_sge *req_sge = NULL;
+ struct scsi_sge *resp_sge = NULL;
+ struct qedi_cmd *qedi_cmd;
+ struct qedi_endpoint *ep;
s16 tid = 0;
+ u16 sq_idx = 0;
+ int rval = 0;
- req_sge = (struct iscsi_sge *)qedi_conn->gen_pdu.req_bd_tbl;
- resp_sge = (struct iscsi_sge *)qedi_conn->gen_pdu.resp_bd_tbl;
+ req_sge = (struct scsi_sge *)qedi_conn->gen_pdu.req_bd_tbl;
+ resp_sge = (struct scsi_sge *)qedi_conn->gen_pdu.resp_bd_tbl;
qedi_cmd = (struct qedi_cmd *)task->dd_data;
text_hdr = (struct iscsi_text *)task->hdr;
+ ep = qedi_conn->ep;
tid = qedi_get_task_idx(qedi);
if (tid == -1)
return -ENOMEM;
- fw_task_ctx = qedi_get_task_mem(&qedi->tasks, tid);
+ fw_task_ctx =
+ (struct iscsi_task_context *)qedi_get_task_mem(&qedi->tasks, tid);
memset(fw_task_ctx, 0, sizeof(struct iscsi_task_context));
qedi_cmd->task_id = tid;
- /* Ystorm context */
- fw_text_request =
- &fw_task_ctx->ystorm_st_context.pdu_hdr.text_request;
- fw_text_request->opcode = text_hdr->opcode;
- fw_text_request->flags_attr = text_hdr->flags;
+ memset(&task_params, 0, sizeof(task_params));
+ memset(&text_request_pdu_header, 0, sizeof(text_request_pdu_header));
+ memset(&tx_sgl_task_params, 0, sizeof(tx_sgl_task_params));
+ memset(&rx_sgl_task_params, 0, sizeof(rx_sgl_task_params));
+
+ /* Update header info */
+ text_request_pdu_header.opcode = text_hdr->opcode;
+ text_request_pdu_header.flags_attr = text_hdr->flags;
qedi_update_itt_map(qedi, tid, task->itt, qedi_cmd);
- fw_text_request->itt = qedi_set_itt(tid, get_itt(task->itt));
- fw_text_request->ttt = text_hdr->ttt;
- fw_text_request->cmd_sn = be32_to_cpu(text_hdr->cmdsn);
- fw_text_request->exp_stat_sn = be32_to_cpu(text_hdr->exp_statsn);
- fw_text_request->hdr_second_dword = ntoh24(text_hdr->dlength);
+ text_request_pdu_header.itt = qedi_set_itt(tid, get_itt(task->itt));
+ text_request_pdu_header.ttt = text_hdr->ttt;
+ text_request_pdu_header.cmd_sn = be32_to_cpu(text_hdr->cmdsn);
+ text_request_pdu_header.exp_stat_sn = be32_to_cpu(text_hdr->exp_statsn);
+ text_request_pdu_header.hdr_second_dword = ntoh24(text_hdr->dlength);
- if (qedi->tid_reuse_count[tid] == QEDI_MAX_TASK_NUM) {
- ptu_invalidate = 1;
- qedi->tid_reuse_count[tid] = 0;
- }
- fw_task_ctx->ystorm_st_context.state.reuse_count =
- qedi->tid_reuse_count[tid];
- fw_task_ctx->mstorm_st_context.reuse_count =
- qedi->tid_reuse_count[tid]++;
-
- cached_sge =
- &fw_task_ctx->ystorm_st_context.state.sgl_ctx_union.cached_sge;
- cached_sge->sge.sge_len = req_sge->sge_len;
- cached_sge->sge.sge_addr.lo = (u32)(qedi_conn->gen_pdu.req_dma_addr);
- cached_sge->sge.sge_addr.hi =
+ /* Fill tx AHS and rx buffer */
+ tx_sgl_task_params.sgl =
+ (struct scsi_sge *)qedi_conn->gen_pdu.req_bd_tbl;
+ tx_sgl_task_params.sgl_phys_addr.lo =
+ (u32)(qedi_conn->gen_pdu.req_dma_addr);
+ tx_sgl_task_params.sgl_phys_addr.hi =
(u32)((u64)qedi_conn->gen_pdu.req_dma_addr >> 32);
+ tx_sgl_task_params.total_buffer_size = req_sge->sge_len;
+ tx_sgl_task_params.num_sges = 1;
- /* Mstorm context */
- single_sge = &fw_task_ctx->mstorm_st_context.sgl_union.single_sge;
- fw_task_ctx->mstorm_st_context.task_type = 0x2;
- fw_task_ctx->mstorm_ag_context.task_cid = (u16)qedi_conn->iscsi_conn_id;
- single_sge->sge_addr.lo = resp_sge->sge_addr.lo;
- single_sge->sge_addr.hi = resp_sge->sge_addr.hi;
- single_sge->sge_len = resp_sge->sge_len;
+ rx_sgl_task_params.sgl =
+ (struct scsi_sge *)qedi_conn->gen_pdu.resp_bd_tbl;
+ rx_sgl_task_params.sgl_phys_addr.lo =
+ (u32)(qedi_conn->gen_pdu.resp_dma_addr);
+ rx_sgl_task_params.sgl_phys_addr.hi =
+ (u32)((u64)qedi_conn->gen_pdu.resp_dma_addr >> 32);
+ rx_sgl_task_params.total_buffer_size = resp_sge->sge_len;
+ rx_sgl_task_params.num_sges = 1;
- SET_FIELD(fw_task_ctx->mstorm_st_context.flags.mflags,
- ISCSI_MFLAGS_SINGLE_SGE, 1);
- SET_FIELD(fw_task_ctx->mstorm_st_context.flags.mflags,
- ISCSI_MFLAGS_SLOW_IO, 0);
- fw_task_ctx->mstorm_st_context.sgl_size = 1;
- fw_task_ctx->mstorm_st_context.rem_task_size = resp_sge->sge_len;
+ /* Fill fw input params */
+ task_params.context = fw_task_ctx;
+ task_params.conn_icid = (u16)qedi_conn->iscsi_conn_id;
+ task_params.itid = tid;
+ task_params.cq_rss_number = 0;
+ task_params.tx_io_size = ntoh24(text_hdr->dlength);
+ task_params.rx_io_size = resp_sge->sge_len;
- /* Ustorm context */
- fw_task_ctx->ustorm_ag_context.exp_data_acked =
- ntoh24(text_hdr->dlength);
- fw_task_ctx->ustorm_st_context.rem_rcv_len = resp_sge->sge_len;
- fw_task_ctx->ustorm_st_context.exp_data_transfer_len =
- ntoh24(text_hdr->dlength);
- fw_task_ctx->ustorm_st_context.exp_data_sn =
- be32_to_cpu(text_hdr->exp_statsn);
- fw_task_ctx->ustorm_st_context.cq_rss_number = 0;
- fw_task_ctx->ustorm_st_context.task_type = 0x2;
- fw_task_ctx->ustorm_ag_context.icid = (u16)qedi_conn->iscsi_conn_id;
- SET_FIELD(fw_task_ctx->ustorm_ag_context.flags1,
- USTORM_ISCSI_TASK_AG_CTX_R2T2RECV, 1);
+ sq_idx = qedi_get_wqe_idx(qedi_conn);
+ task_params.sqe = &ep->sq[sq_idx];
- /* Add command in active command list */
+ memset(task_params.sqe, 0, sizeof(struct iscsi_wqe));
+ rval = init_initiator_text_request_task(&task_params,
+ &text_request_pdu_header,
+ &tx_sgl_task_params,
+ &rx_sgl_task_params);
+ if (rval)
+ return -1;
+
spin_lock(&qedi_conn->list_lock);
list_add_tail(&qedi_cmd->io_cmd, &qedi_conn->active_cmd_list);
qedi_cmd->io_cmd_in_list = true;
qedi_conn->active_cmd_count++;
spin_unlock(&qedi_conn->list_lock);
- qedi_add_to_sq(qedi_conn, task, tid, ptu_invalidate, false);
qedi_ring_doorbell(qedi_conn);
-
return 0;
}
@@ -1791,58 +1710,62 @@
struct iscsi_task *task,
char *datap, int data_len, int unsol)
{
+ struct iscsi_nop_out_hdr nop_out_pdu_header;
+ struct scsi_sgl_task_params tx_sgl_task_params;
+ struct scsi_sgl_task_params rx_sgl_task_params;
+ struct iscsi_task_params task_params;
struct qedi_ctx *qedi = qedi_conn->qedi;
struct iscsi_task_context *fw_task_ctx;
- struct iscsi_nop_out_hdr *fw_nop_out;
- struct qedi_cmd *qedi_cmd;
- /* For 6.5 hdr iscsi_hdr */
struct iscsi_nopout *nopout_hdr;
- struct iscsi_cached_sge_ctx *cached_sge;
- struct iscsi_sge *single_sge;
- struct iscsi_sge *req_sge;
- struct iscsi_sge *resp_sge;
- u32 lun[2];
- s16 ptu_invalidate = 0;
+ struct scsi_sge *req_sge = NULL;
+ struct scsi_sge *resp_sge = NULL;
+ struct qedi_cmd *qedi_cmd;
+ struct qedi_endpoint *ep;
+ u32 scsi_lun[2];
s16 tid = 0;
+ u16 sq_idx = 0;
+ int rval = 0;
- req_sge = (struct iscsi_sge *)qedi_conn->gen_pdu.req_bd_tbl;
- resp_sge = (struct iscsi_sge *)qedi_conn->gen_pdu.resp_bd_tbl;
+ req_sge = (struct scsi_sge *)qedi_conn->gen_pdu.req_bd_tbl;
+ resp_sge = (struct scsi_sge *)qedi_conn->gen_pdu.resp_bd_tbl;
qedi_cmd = (struct qedi_cmd *)task->dd_data;
nopout_hdr = (struct iscsi_nopout *)task->hdr;
+ ep = qedi_conn->ep;
tid = qedi_get_task_idx(qedi);
- if (tid == -1) {
- QEDI_WARN(&qedi->dbg_ctx, "Invalid tid\n");
+ if (tid == -1)
return -ENOMEM;
- }
- fw_task_ctx = qedi_get_task_mem(&qedi->tasks, tid);
-
+ fw_task_ctx =
+ (struct iscsi_task_context *)qedi_get_task_mem(&qedi->tasks, tid);
memset(fw_task_ctx, 0, sizeof(struct iscsi_task_context));
+
qedi_cmd->task_id = tid;
- /* Ystorm context */
- fw_nop_out = &fw_task_ctx->ystorm_st_context.pdu_hdr.nop_out;
- SET_FIELD(fw_nop_out->flags_attr, ISCSI_NOP_OUT_HDR_CONST1, 1);
- SET_FIELD(fw_nop_out->flags_attr, ISCSI_NOP_OUT_HDR_RSRV, 0);
+ memset(&task_params, 0, sizeof(task_params));
+ memset(&nop_out_pdu_header, 0, sizeof(nop_out_pdu_header));
+ memset(&tx_sgl_task_params, 0, sizeof(tx_sgl_task_params));
+ memset(&rx_sgl_task_params, 0, sizeof(rx_sgl_task_params));
- memcpy(lun, &nopout_hdr->lun, sizeof(struct scsi_lun));
- fw_nop_out->lun.lo = be32_to_cpu(lun[0]);
- fw_nop_out->lun.hi = be32_to_cpu(lun[1]);
+ /* Update header info */
+ nop_out_pdu_header.opcode = nopout_hdr->opcode;
+ SET_FIELD(nop_out_pdu_header.flags_attr, ISCSI_NOP_OUT_HDR_CONST1, 1);
+ SET_FIELD(nop_out_pdu_header.flags_attr, ISCSI_NOP_OUT_HDR_RSRV, 0);
+
+ memcpy(scsi_lun, &nopout_hdr->lun, sizeof(struct scsi_lun));
+ nop_out_pdu_header.lun.lo = be32_to_cpu(scsi_lun[0]);
+ nop_out_pdu_header.lun.hi = be32_to_cpu(scsi_lun[1]);
+ nop_out_pdu_header.cmd_sn = be32_to_cpu(nopout_hdr->cmdsn);
+ nop_out_pdu_header.exp_stat_sn = be32_to_cpu(nopout_hdr->exp_statsn);
qedi_update_itt_map(qedi, tid, task->itt, qedi_cmd);
if (nopout_hdr->ttt != ISCSI_TTT_ALL_ONES) {
- fw_nop_out->itt = be32_to_cpu(nopout_hdr->itt);
- fw_nop_out->ttt = be32_to_cpu(nopout_hdr->ttt);
- fw_task_ctx->ystorm_st_context.state.buffer_offset[0] = 0;
- fw_task_ctx->ystorm_st_context.state.local_comp = 1;
- SET_FIELD(fw_task_ctx->ustorm_st_context.flags,
- USTORM_ISCSI_TASK_ST_CTX_LOCAL_COMP, 1);
+ nop_out_pdu_header.itt = be32_to_cpu(nopout_hdr->itt);
+ nop_out_pdu_header.ttt = be32_to_cpu(nopout_hdr->ttt);
} else {
- fw_nop_out->itt = qedi_set_itt(tid, get_itt(task->itt));
- fw_nop_out->ttt = ISCSI_TTT_ALL_ONES;
- fw_task_ctx->ystorm_st_context.state.buffer_offset[0] = 0;
+ nop_out_pdu_header.itt = qedi_set_itt(tid, get_itt(task->itt));
+ nop_out_pdu_header.ttt = ISCSI_TTT_ALL_ONES;
spin_lock(&qedi_conn->list_lock);
list_add_tail(&qedi_cmd->io_cmd, &qedi_conn->active_cmd_list);
@@ -1851,53 +1774,46 @@
spin_unlock(&qedi_conn->list_lock);
}
- fw_nop_out->opcode = ISCSI_OPCODE_NOP_OUT;
- fw_nop_out->cmd_sn = be32_to_cpu(nopout_hdr->cmdsn);
- fw_nop_out->exp_stat_sn = be32_to_cpu(nopout_hdr->exp_statsn);
+ /* Fill tx AHS and rx buffer */
+ if (data_len) {
+ tx_sgl_task_params.sgl =
+ (struct scsi_sge *)qedi_conn->gen_pdu.req_bd_tbl;
+ tx_sgl_task_params.sgl_phys_addr.lo =
+ (u32)(qedi_conn->gen_pdu.req_dma_addr);
+ tx_sgl_task_params.sgl_phys_addr.hi =
+ (u32)((u64)qedi_conn->gen_pdu.req_dma_addr >> 32);
+ tx_sgl_task_params.total_buffer_size = data_len;
+ tx_sgl_task_params.num_sges = 1;
- cached_sge =
- &fw_task_ctx->ystorm_st_context.state.sgl_ctx_union.cached_sge;
- cached_sge->sge.sge_len = req_sge->sge_len;
- cached_sge->sge.sge_addr.lo = (u32)(qedi_conn->gen_pdu.req_dma_addr);
- cached_sge->sge.sge_addr.hi =
- (u32)((u64)qedi_conn->gen_pdu.req_dma_addr >> 32);
-
- /* Mstorm context */
- fw_task_ctx->mstorm_st_context.task_type = ISCSI_TASK_TYPE_MIDPATH;
- fw_task_ctx->mstorm_ag_context.task_cid = (u16)qedi_conn->iscsi_conn_id;
-
- single_sge = &fw_task_ctx->mstorm_st_context.sgl_union.single_sge;
- single_sge->sge_addr.lo = resp_sge->sge_addr.lo;
- single_sge->sge_addr.hi = resp_sge->sge_addr.hi;
- single_sge->sge_len = resp_sge->sge_len;
- fw_task_ctx->mstorm_st_context.rem_task_size = resp_sge->sge_len;
-
- if (qedi->tid_reuse_count[tid] == QEDI_MAX_TASK_NUM) {
- ptu_invalidate = 1;
- qedi->tid_reuse_count[tid] = 0;
+ rx_sgl_task_params.sgl =
+ (struct scsi_sge *)qedi_conn->gen_pdu.resp_bd_tbl;
+ rx_sgl_task_params.sgl_phys_addr.lo =
+ (u32)(qedi_conn->gen_pdu.resp_dma_addr);
+ rx_sgl_task_params.sgl_phys_addr.hi =
+ (u32)((u64)qedi_conn->gen_pdu.resp_dma_addr >> 32);
+ rx_sgl_task_params.total_buffer_size = resp_sge->sge_len;
+ rx_sgl_task_params.num_sges = 1;
}
- fw_task_ctx->ystorm_st_context.state.reuse_count =
- qedi->tid_reuse_count[tid];
- fw_task_ctx->mstorm_st_context.reuse_count =
- qedi->tid_reuse_count[tid]++;
- /* Ustorm context */
- fw_task_ctx->ustorm_st_context.rem_rcv_len = resp_sge->sge_len;
- fw_task_ctx->ustorm_st_context.exp_data_transfer_len = data_len;
- fw_task_ctx->ustorm_st_context.exp_data_sn = 0;
- fw_task_ctx->ustorm_st_context.task_type = ISCSI_TASK_TYPE_MIDPATH;
- fw_task_ctx->ustorm_st_context.cq_rss_number = 0;
- SET_FIELD(fw_task_ctx->ustorm_st_context.reg1.reg1_map,
- ISCSI_REG1_NUM_FAST_SGES, 0);
+ /* Fill fw input params */
+ task_params.context = fw_task_ctx;
+ task_params.conn_icid = (u16)qedi_conn->iscsi_conn_id;
+ task_params.itid = tid;
+ task_params.cq_rss_number = 0;
+ task_params.tx_io_size = data_len;
+ task_params.rx_io_size = resp_sge->sge_len;
- fw_task_ctx->ustorm_ag_context.icid = (u16)qedi_conn->iscsi_conn_id;
- SET_FIELD(fw_task_ctx->ustorm_ag_context.flags1,
- USTORM_ISCSI_TASK_AG_CTX_R2T2RECV, 1);
+ sq_idx = qedi_get_wqe_idx(qedi_conn);
+ task_params.sqe = &ep->sq[sq_idx];
- fw_task_ctx->ustorm_st_context.lun.lo = be32_to_cpu(lun[0]);
- fw_task_ctx->ustorm_st_context.lun.hi = be32_to_cpu(lun[1]);
+ memset(task_params.sqe, 0, sizeof(struct iscsi_wqe));
+ rval = init_initiator_nop_out_task(&task_params,
+ &nop_out_pdu_header,
+ &tx_sgl_task_params,
+ &rx_sgl_task_params);
+ if (rval)
+ return -1;
- qedi_add_to_sq(qedi_conn, task, tid, ptu_invalidate, false);
qedi_ring_doorbell(qedi_conn);
return 0;
}
@@ -1905,7 +1821,7 @@
static int qedi_split_bd(struct qedi_cmd *cmd, u64 addr, int sg_len,
int bd_index)
{
- struct iscsi_sge *bd = cmd->io_tbl.sge_tbl;
+ struct scsi_sge *bd = cmd->io_tbl.sge_tbl;
int frag_size, sg_frags;
sg_frags = 0;
@@ -1938,7 +1854,7 @@
static int qedi_map_scsi_sg(struct qedi_ctx *qedi, struct qedi_cmd *cmd)
{
struct scsi_cmnd *sc = cmd->scsi_cmd;
- struct iscsi_sge *bd = cmd->io_tbl.sge_tbl;
+ struct scsi_sge *bd = cmd->io_tbl.sge_tbl;
struct scatterlist *sg;
int byte_count = 0;
int bd_count = 0;
@@ -2040,7 +1956,7 @@
if (bd_count == 0)
return;
} else {
- struct iscsi_sge *bd = cmd->io_tbl.sge_tbl;
+ struct scsi_sge *bd = cmd->io_tbl.sge_tbl;
bd[0].sge_addr.lo = 0;
bd[0].sge_addr.hi = 0;
@@ -2136,244 +2052,182 @@
struct qedi_conn *qedi_conn = conn->dd_data;
struct qedi_cmd *cmd = task->dd_data;
struct scsi_cmnd *sc = task->sc;
+ struct iscsi_cmd_hdr cmd_pdu_header;
+ struct scsi_sgl_task_params tx_sgl_task_params;
+ struct scsi_sgl_task_params rx_sgl_task_params;
+ struct scsi_sgl_task_params *prx_sgl = NULL;
+ struct scsi_sgl_task_params *ptx_sgl = NULL;
+ struct iscsi_task_params task_params;
+ struct iscsi_conn_params conn_params;
+ struct scsi_initiator_cmd_params cmd_params;
struct iscsi_task_context *fw_task_ctx;
- struct iscsi_cached_sge_ctx *cached_sge;
- struct iscsi_phys_sgl_ctx *phys_sgl;
- struct iscsi_virt_sgl_ctx *virt_sgl;
- struct ystorm_iscsi_task_st_ctx *yst_cxt;
- struct mstorm_iscsi_task_st_ctx *mst_cxt;
- struct iscsi_sgl *sgl_struct;
- struct iscsi_sge *single_sge;
+ struct iscsi_cls_conn *cls_conn;
struct iscsi_scsi_req *hdr = (struct iscsi_scsi_req *)task->hdr;
- struct iscsi_sge *bd = cmd->io_tbl.sge_tbl;
- enum iscsi_task_type task_type;
- struct iscsi_cmd_hdr *fw_cmd;
- u32 lun[2];
- u32 exp_data;
- u16 cq_idx = smp_processor_id() % qedi->num_queues;
- s16 ptu_invalidate = 0;
+ enum iscsi_task_type task_type = MAX_ISCSI_TASK_TYPE;
+ struct qedi_endpoint *ep;
+ u32 scsi_lun[2];
s16 tid = 0;
- u8 num_fast_sgs;
+ u16 sq_idx = 0;
+ u16 cq_idx;
+ int rval = 0;
+
+ ep = qedi_conn->ep;
+ cls_conn = qedi_conn->cls_conn;
+ conn = cls_conn->dd_data;
+
+ qedi_iscsi_map_sg_list(cmd);
+ int_to_scsilun(sc->device->lun, (struct scsi_lun *)scsi_lun);
tid = qedi_get_task_idx(qedi);
if (tid == -1)
return -ENOMEM;
- qedi_iscsi_map_sg_list(cmd);
-
- int_to_scsilun(sc->device->lun, (struct scsi_lun *)lun);
- fw_task_ctx = qedi_get_task_mem(&qedi->tasks, tid);
-
+ fw_task_ctx =
+ (struct iscsi_task_context *)qedi_get_task_mem(&qedi->tasks, tid);
memset(fw_task_ctx, 0, sizeof(struct iscsi_task_context));
+
cmd->task_id = tid;
- /* Ystorm context */
- fw_cmd = &fw_task_ctx->ystorm_st_context.pdu_hdr.cmd;
- SET_FIELD(fw_cmd->flags_attr, ISCSI_CMD_HDR_ATTR, ISCSI_ATTR_SIMPLE);
+ memset(&task_params, 0, sizeof(task_params));
+ memset(&cmd_pdu_header, 0, sizeof(cmd_pdu_header));
+ memset(&tx_sgl_task_params, 0, sizeof(tx_sgl_task_params));
+ memset(&rx_sgl_task_params, 0, sizeof(rx_sgl_task_params));
+ memset(&conn_params, 0, sizeof(conn_params));
+ memset(&cmd_params, 0, sizeof(cmd_params));
+ cq_idx = smp_processor_id() % qedi->num_queues;
+ /* Update header info */
+ SET_FIELD(cmd_pdu_header.flags_attr, ISCSI_CMD_HDR_ATTR,
+ ISCSI_ATTR_SIMPLE);
if (sc->sc_data_direction == DMA_TO_DEVICE) {
- if (conn->session->initial_r2t_en) {
- exp_data = min((conn->session->imm_data_en *
- conn->max_xmit_dlength),
- conn->session->first_burst);
- exp_data = min(exp_data, scsi_bufflen(sc));
- fw_task_ctx->ustorm_ag_context.exp_data_acked =
- cpu_to_le32(exp_data);
- } else {
- fw_task_ctx->ustorm_ag_context.exp_data_acked =
- min(conn->session->first_burst, scsi_bufflen(sc));
- }
-
- SET_FIELD(fw_cmd->flags_attr, ISCSI_CMD_HDR_WRITE, 1);
+ SET_FIELD(cmd_pdu_header.flags_attr,
+ ISCSI_CMD_HDR_WRITE, 1);
task_type = ISCSI_TASK_TYPE_INITIATOR_WRITE;
} else {
- if (scsi_bufflen(sc))
- SET_FIELD(fw_cmd->flags_attr, ISCSI_CMD_HDR_READ, 1);
+ SET_FIELD(cmd_pdu_header.flags_attr,
+ ISCSI_CMD_HDR_READ, 1);
task_type = ISCSI_TASK_TYPE_INITIATOR_READ;
}
- fw_cmd->lun.lo = be32_to_cpu(lun[0]);
- fw_cmd->lun.hi = be32_to_cpu(lun[1]);
+ cmd_pdu_header.lun.lo = be32_to_cpu(scsi_lun[0]);
+ cmd_pdu_header.lun.hi = be32_to_cpu(scsi_lun[1]);
qedi_update_itt_map(qedi, tid, task->itt, cmd);
- fw_cmd->itt = qedi_set_itt(tid, get_itt(task->itt));
- fw_cmd->expected_transfer_length = scsi_bufflen(sc);
- fw_cmd->cmd_sn = be32_to_cpu(hdr->cmdsn);
- fw_cmd->opcode = hdr->opcode;
- qedi_cpy_scsi_cdb(sc, (u32 *)fw_cmd->cdb);
+ cmd_pdu_header.itt = qedi_set_itt(tid, get_itt(task->itt));
+ cmd_pdu_header.expected_transfer_length = cpu_to_be32(hdr->data_length);
+ cmd_pdu_header.hdr_second_dword = ntoh24(hdr->dlength);
+ cmd_pdu_header.cmd_sn = be32_to_cpu(hdr->cmdsn);
+ cmd_pdu_header.opcode = hdr->opcode;
+ qedi_cpy_scsi_cdb(sc, (u32 *)cmd_pdu_header.cdb);
- /* Mstorm context */
- fw_task_ctx->mstorm_st_context.sense_db.lo = (u32)cmd->sense_buffer_dma;
- fw_task_ctx->mstorm_st_context.sense_db.hi =
- (u32)((u64)cmd->sense_buffer_dma >> 32);
- fw_task_ctx->mstorm_ag_context.task_cid = qedi_conn->iscsi_conn_id;
- fw_task_ctx->mstorm_st_context.task_type = task_type;
-
- if (qedi->tid_reuse_count[tid] == QEDI_MAX_TASK_NUM) {
- ptu_invalidate = 1;
- qedi->tid_reuse_count[tid] = 0;
- }
- fw_task_ctx->ystorm_st_context.state.reuse_count =
- qedi->tid_reuse_count[tid];
- fw_task_ctx->mstorm_st_context.reuse_count =
- qedi->tid_reuse_count[tid]++;
-
- /* Ustorm context */
- fw_task_ctx->ustorm_st_context.rem_rcv_len = scsi_bufflen(sc);
- fw_task_ctx->ustorm_st_context.exp_data_transfer_len = scsi_bufflen(sc);
- fw_task_ctx->ustorm_st_context.exp_data_sn =
- be32_to_cpu(hdr->exp_statsn);
- fw_task_ctx->ustorm_st_context.task_type = task_type;
- fw_task_ctx->ustorm_st_context.cq_rss_number = cq_idx;
- fw_task_ctx->ustorm_ag_context.icid = (u16)qedi_conn->iscsi_conn_id;
-
- SET_FIELD(fw_task_ctx->ustorm_ag_context.flags1,
- USTORM_ISCSI_TASK_AG_CTX_R2T2RECV, 1);
- SET_FIELD(fw_task_ctx->ustorm_st_context.flags,
- USTORM_ISCSI_TASK_ST_CTX_LOCAL_COMP, 0);
-
- num_fast_sgs = (cmd->io_tbl.sge_valid ?
- min((u16)QEDI_FAST_SGE_COUNT,
- (u16)cmd->io_tbl.sge_valid) : 0);
- SET_FIELD(fw_task_ctx->ustorm_st_context.reg1.reg1_map,
- ISCSI_REG1_NUM_FAST_SGES, num_fast_sgs);
-
- fw_task_ctx->ustorm_st_context.lun.lo = be32_to_cpu(lun[0]);
- fw_task_ctx->ustorm_st_context.lun.hi = be32_to_cpu(lun[1]);
-
- QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_IO, "Total sge count [%d]\n",
- cmd->io_tbl.sge_valid);
-
- yst_cxt = &fw_task_ctx->ystorm_st_context;
- mst_cxt = &fw_task_ctx->mstorm_st_context;
- /* Tx path */
+ /* Fill tx AHS and rx buffer */
if (task_type == ISCSI_TASK_TYPE_INITIATOR_WRITE) {
- /* not considering superIO or FastIO */
- if (cmd->io_tbl.sge_valid == 1) {
- cached_sge = &yst_cxt->state.sgl_ctx_union.cached_sge;
- cached_sge->sge.sge_addr.lo = bd[0].sge_addr.lo;
- cached_sge->sge.sge_addr.hi = bd[0].sge_addr.hi;
- cached_sge->sge.sge_len = bd[0].sge_len;
- qedi->cached_sgls++;
- } else if ((cmd->io_tbl.sge_valid != 1) && cmd->use_slowpath) {
- SET_FIELD(fw_task_ctx->mstorm_st_context.flags.mflags,
- ISCSI_MFLAGS_SLOW_IO, 1);
- SET_FIELD(fw_task_ctx->ustorm_st_context.reg1.reg1_map,
- ISCSI_REG1_NUM_FAST_SGES, 0);
- phys_sgl = &yst_cxt->state.sgl_ctx_union.phys_sgl;
- phys_sgl->sgl_base.lo = (u32)(cmd->io_tbl.sge_tbl_dma);
- phys_sgl->sgl_base.hi =
- (u32)((u64)cmd->io_tbl.sge_tbl_dma >> 32);
- phys_sgl->sgl_size = cmd->io_tbl.sge_valid;
- qedi->slow_sgls++;
- } else if ((cmd->io_tbl.sge_valid != 1) && !cmd->use_slowpath) {
- SET_FIELD(fw_task_ctx->mstorm_st_context.flags.mflags,
- ISCSI_MFLAGS_SLOW_IO, 0);
- SET_FIELD(fw_task_ctx->ustorm_st_context.reg1.reg1_map,
- ISCSI_REG1_NUM_FAST_SGES,
- min((u16)QEDI_FAST_SGE_COUNT,
- (u16)cmd->io_tbl.sge_valid));
- virt_sgl = &yst_cxt->state.sgl_ctx_union.virt_sgl;
- virt_sgl->sgl_base.lo = (u32)(cmd->io_tbl.sge_tbl_dma);
- virt_sgl->sgl_base.hi =
+ tx_sgl_task_params.sgl = cmd->io_tbl.sge_tbl;
+ tx_sgl_task_params.sgl_phys_addr.lo =
+ (u32)(cmd->io_tbl.sge_tbl_dma);
+ tx_sgl_task_params.sgl_phys_addr.hi =
(u32)((u64)cmd->io_tbl.sge_tbl_dma >> 32);
- virt_sgl->sgl_initial_offset =
- (u32)bd[0].sge_addr.lo & (QEDI_PAGE_SIZE - 1);
- qedi->fast_sgls++;
- }
- fw_task_ctx->mstorm_st_context.sgl_size = cmd->io_tbl.sge_valid;
- fw_task_ctx->mstorm_st_context.rem_task_size = scsi_bufflen(sc);
- } else {
- /* Rx path */
- if (cmd->io_tbl.sge_valid == 1) {
- SET_FIELD(fw_task_ctx->mstorm_st_context.flags.mflags,
- ISCSI_MFLAGS_SLOW_IO, 0);
- SET_FIELD(fw_task_ctx->mstorm_st_context.flags.mflags,
- ISCSI_MFLAGS_SINGLE_SGE, 1);
- single_sge = &mst_cxt->sgl_union.single_sge;
- single_sge->sge_addr.lo = bd[0].sge_addr.lo;
- single_sge->sge_addr.hi = bd[0].sge_addr.hi;
- single_sge->sge_len = bd[0].sge_len;
- qedi->cached_sgls++;
- } else if ((cmd->io_tbl.sge_valid != 1) && cmd->use_slowpath) {
- sgl_struct = &mst_cxt->sgl_union.sgl_struct;
- sgl_struct->sgl_addr.lo =
- (u32)(cmd->io_tbl.sge_tbl_dma);
- sgl_struct->sgl_addr.hi =
- (u32)((u64)cmd->io_tbl.sge_tbl_dma >> 32);
- SET_FIELD(fw_task_ctx->mstorm_st_context.flags.mflags,
- ISCSI_MFLAGS_SLOW_IO, 1);
- SET_FIELD(fw_task_ctx->ustorm_st_context.reg1.reg1_map,
- ISCSI_REG1_NUM_FAST_SGES, 0);
- sgl_struct->updated_sge_size = 0;
- sgl_struct->updated_sge_offset = 0;
- qedi->slow_sgls++;
- } else if ((cmd->io_tbl.sge_valid != 1) && !cmd->use_slowpath) {
- sgl_struct = &mst_cxt->sgl_union.sgl_struct;
- sgl_struct->sgl_addr.lo =
- (u32)(cmd->io_tbl.sge_tbl_dma);
- sgl_struct->sgl_addr.hi =
- (u32)((u64)cmd->io_tbl.sge_tbl_dma >> 32);
- sgl_struct->byte_offset =
- (u32)bd[0].sge_addr.lo & (QEDI_PAGE_SIZE - 1);
- SET_FIELD(fw_task_ctx->mstorm_st_context.flags.mflags,
- ISCSI_MFLAGS_SLOW_IO, 0);
- SET_FIELD(fw_task_ctx->ustorm_st_context.reg1.reg1_map,
- ISCSI_REG1_NUM_FAST_SGES, 0);
- sgl_struct->updated_sge_size = 0;
- sgl_struct->updated_sge_offset = 0;
- qedi->fast_sgls++;
- }
- fw_task_ctx->mstorm_st_context.sgl_size = cmd->io_tbl.sge_valid;
- fw_task_ctx->mstorm_st_context.rem_task_size = scsi_bufflen(sc);
+ tx_sgl_task_params.total_buffer_size = scsi_bufflen(sc);
+ tx_sgl_task_params.num_sges = cmd->io_tbl.sge_valid;
+ if (cmd->use_slowpath)
+ tx_sgl_task_params.small_mid_sge = true;
+ } else if (task_type == ISCSI_TASK_TYPE_INITIATOR_READ) {
+ rx_sgl_task_params.sgl = cmd->io_tbl.sge_tbl;
+ rx_sgl_task_params.sgl_phys_addr.lo =
+ (u32)(cmd->io_tbl.sge_tbl_dma);
+ rx_sgl_task_params.sgl_phys_addr.hi =
+ (u32)((u64)cmd->io_tbl.sge_tbl_dma >> 32);
+ rx_sgl_task_params.total_buffer_size = scsi_bufflen(sc);
+ rx_sgl_task_params.num_sges = cmd->io_tbl.sge_valid;
}
- if (cmd->io_tbl.sge_valid == 1)
- /* Singel-SGL */
- qedi->use_cached_sge = true;
- else {
- if (cmd->use_slowpath)
- qedi->use_slow_sge = true;
- else
- qedi->use_fast_sge = true;
- }
+ /* Add conn param */
+ conn_params.first_burst_length = conn->session->first_burst;
+ conn_params.max_send_pdu_length = conn->max_xmit_dlength;
+ conn_params.max_burst_length = conn->session->max_burst;
+ if (conn->session->initial_r2t_en)
+ conn_params.initial_r2t = true;
+ if (conn->session->imm_data_en)
+ conn_params.immediate_data = true;
+
+ /* Add cmd params */
+ cmd_params.sense_data_buffer_phys_addr.lo = (u32)cmd->sense_buffer_dma;
+ cmd_params.sense_data_buffer_phys_addr.hi =
+ (u32)((u64)cmd->sense_buffer_dma >> 32);
+ /* Fill fw input params */
+ task_params.context = fw_task_ctx;
+ task_params.conn_icid = (u16)qedi_conn->iscsi_conn_id;
+ task_params.itid = tid;
+ task_params.cq_rss_number = cq_idx;
+ if (task_type == ISCSI_TASK_TYPE_INITIATOR_WRITE)
+ task_params.tx_io_size = scsi_bufflen(sc);
+ else if (task_type == ISCSI_TASK_TYPE_INITIATOR_READ)
+ task_params.rx_io_size = scsi_bufflen(sc);
+
+ sq_idx = qedi_get_wqe_idx(qedi_conn);
+ task_params.sqe = &ep->sq[sq_idx];
+
QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_IO,
- "%s: %s-SGL: num_sges=0x%x first-sge-lo=0x%x first-sge-hi=0x%x",
+ "%s: %s-SGL: sg_len=0x%x num_sges=0x%x first-sge-lo=0x%x first-sge-hi=0x%x\n",
(task_type == ISCSI_TASK_TYPE_INITIATOR_WRITE) ?
"Write " : "Read ", (cmd->io_tbl.sge_valid == 1) ?
"Single" : (cmd->use_slowpath ? "SLOW" : "FAST"),
- (u16)cmd->io_tbl.sge_valid, (u32)(cmd->io_tbl.sge_tbl_dma),
+ (u16)cmd->io_tbl.sge_valid, scsi_bufflen(sc),
+ (u32)(cmd->io_tbl.sge_tbl_dma),
(u32)((u64)cmd->io_tbl.sge_tbl_dma >> 32));
- /* Add command in active command list */
+ memset(task_params.sqe, 0, sizeof(struct iscsi_wqe));
+
+ if (task_params.tx_io_size != 0)
+ ptx_sgl = &tx_sgl_task_params;
+ if (task_params.rx_io_size != 0)
+ prx_sgl = &rx_sgl_task_params;
+
+ rval = init_initiator_rw_iscsi_task(&task_params, &conn_params,
+ &cmd_params, &cmd_pdu_header,
+ ptx_sgl, prx_sgl,
+ NULL);
+ if (rval)
+ return -1;
+
spin_lock(&qedi_conn->list_lock);
list_add_tail(&cmd->io_cmd, &qedi_conn->active_cmd_list);
cmd->io_cmd_in_list = true;
qedi_conn->active_cmd_count++;
spin_unlock(&qedi_conn->list_lock);
- qedi_add_to_sq(qedi_conn, task, tid, ptu_invalidate, false);
qedi_ring_doorbell(qedi_conn);
- if (qedi_io_tracing)
- qedi_trace_io(qedi, task, tid, QEDI_IO_TRACE_REQ);
-
return 0;
}
int qedi_iscsi_cleanup_task(struct iscsi_task *task, bool mark_cmd_node_deleted)
{
+ struct iscsi_task_params task_params;
+ struct qedi_endpoint *ep;
struct iscsi_conn *conn = task->conn;
struct qedi_conn *qedi_conn = conn->dd_data;
struct qedi_cmd *cmd = task->dd_data;
- s16 ptu_invalidate = 0;
+ u16 sq_idx = 0;
+ int rval = 0;
QEDI_INFO(&qedi_conn->qedi->dbg_ctx, QEDI_LOG_SCSI_TM,
"issue cleanup tid=0x%x itt=0x%x task_state=%d cmd_state=0%x cid=0x%x\n",
cmd->task_id, get_itt(task->itt), task->state,
cmd->state, qedi_conn->iscsi_conn_id);
- qedi_add_to_sq(qedi_conn, task, cmd->task_id, ptu_invalidate, true);
- qedi_ring_doorbell(qedi_conn);
+ memset(&task_params, 0, sizeof(task_params));
+ ep = qedi_conn->ep;
+ sq_idx = qedi_get_wqe_idx(qedi_conn);
+
+ task_params.sqe = &ep->sq[sq_idx];
+ memset(task_params.sqe, 0, sizeof(struct iscsi_wqe));
+ task_params.itid = cmd->task_id;
+
+ rval = init_cleanup_task(&task_params);
+ if (rval)
+ return rval;
+
+ qedi_ring_doorbell(qedi_conn);
return 0;
}
diff --git a/drivers/scsi/qedi/qedi_fw_api.c b/drivers/scsi/qedi/qedi_fw_api.c
new file mode 100644
index 0000000..fd354d4
--- /dev/null
+++ b/drivers/scsi/qedi/qedi_fw_api.c
@@ -0,0 +1,781 @@
+/* QLogic iSCSI Offload Driver
+ * Copyright (c) 2016 Cavium Inc.
+ *
+ * This software is available under the terms of the GNU General Public License
+ * (GPL) Version 2, available from the file COPYING in the main directory of
+ * this source tree.
+ */
+
+#include <linux/types.h>
+#include <asm/byteorder.h>
+#include "qedi_hsi.h"
+#include <linux/qed/qed_if.h>
+
+#include "qedi_fw_iscsi.h"
+#include "qedi_fw_scsi.h"
+
+#define SCSI_NUM_SGES_IN_CACHE 0x4
+
+static bool scsi_is_slow_sgl(u16 num_sges, bool small_mid_sge)
+{
+ return (num_sges > SCSI_NUM_SGES_SLOW_SGL_THR && small_mid_sge);
+}
+
+static
+void init_scsi_sgl_context(struct scsi_sgl_params *ctx_sgl_params,
+ struct scsi_cached_sges *ctx_data_desc,
+ struct scsi_sgl_task_params *sgl_task_params)
+{
+ u8 sge_index;
+ u8 num_sges;
+ u32 val;
+
+ num_sges = (sgl_task_params->num_sges > SCSI_NUM_SGES_IN_CACHE) ?
+ SCSI_NUM_SGES_IN_CACHE : sgl_task_params->num_sges;
+
+ /* sgl params */
+ val = cpu_to_le32(sgl_task_params->sgl_phys_addr.lo);
+ ctx_sgl_params->sgl_addr.lo = val;
+ val = cpu_to_le32(sgl_task_params->sgl_phys_addr.hi);
+ ctx_sgl_params->sgl_addr.hi = val;
+ val = cpu_to_le32(sgl_task_params->total_buffer_size);
+ ctx_sgl_params->sgl_total_length = val;
+ ctx_sgl_params->sgl_num_sges = cpu_to_le16(sgl_task_params->num_sges);
+
+ for (sge_index = 0; sge_index < num_sges; sge_index++) {
+ val = cpu_to_le32(sgl_task_params->sgl[sge_index].sge_addr.lo);
+ ctx_data_desc->sge[sge_index].sge_addr.lo = val;
+ val = cpu_to_le32(sgl_task_params->sgl[sge_index].sge_addr.hi);
+ ctx_data_desc->sge[sge_index].sge_addr.hi = val;
+ val = cpu_to_le32(sgl_task_params->sgl[sge_index].sge_len);
+ ctx_data_desc->sge[sge_index].sge_len = val;
+ }
+}
+
+static u32 calc_rw_task_size(struct iscsi_task_params *task_params,
+ enum iscsi_task_type task_type,
+ struct scsi_sgl_task_params *sgl_task_params,
+ struct scsi_dif_task_params *dif_task_params)
+{
+ u32 io_size;
+
+ if (task_type == ISCSI_TASK_TYPE_INITIATOR_WRITE ||
+ task_type == ISCSI_TASK_TYPE_TARGET_READ)
+ io_size = task_params->tx_io_size;
+ else
+ io_size = task_params->rx_io_size;
+
+ if (!io_size)
+ return 0;
+
+ if (!dif_task_params)
+ return io_size;
+
+ return !dif_task_params->dif_on_network ?
+ io_size : sgl_task_params->total_buffer_size;
+}
+
+static void
+init_dif_context_flags(struct iscsi_dif_flags *ctx_dif_flags,
+ struct scsi_dif_task_params *dif_task_params)
+{
+ if (!dif_task_params)
+ return;
+
+ SET_FIELD(ctx_dif_flags->flags, ISCSI_DIF_FLAGS_PROT_INTERVAL_SIZE_LOG,
+ dif_task_params->dif_block_size_log);
+ SET_FIELD(ctx_dif_flags->flags, ISCSI_DIF_FLAGS_DIF_TO_PEER,
+ dif_task_params->dif_on_network ? 1 : 0);
+ SET_FIELD(ctx_dif_flags->flags, ISCSI_DIF_FLAGS_HOST_INTERFACE,
+ dif_task_params->dif_on_host ? 1 : 0);
+}
+
+static void init_sqe(struct iscsi_task_params *task_params,
+ struct scsi_sgl_task_params *sgl_task_params,
+ struct scsi_dif_task_params *dif_task_params,
+ struct iscsi_common_hdr *pdu_header,
+ struct scsi_initiator_cmd_params *cmd_params,
+ enum iscsi_task_type task_type,
+ bool is_cleanup)
+{
+ if (!task_params->sqe)
+ return;
+
+ memset(task_params->sqe, 0, sizeof(*task_params->sqe));
+ task_params->sqe->task_id = cpu_to_le16(task_params->itid);
+ if (is_cleanup) {
+ SET_FIELD(task_params->sqe->flags, ISCSI_WQE_WQE_TYPE,
+ ISCSI_WQE_TYPE_TASK_CLEANUP);
+ return;
+ }
+
+ switch (task_type) {
+ case ISCSI_TASK_TYPE_INITIATOR_WRITE:
+ {
+ u32 buf_size = 0;
+ u32 num_sges = 0;
+
+ init_dif_context_flags(&task_params->sqe->prot_flags,
+ dif_task_params);
+
+ SET_FIELD(task_params->sqe->flags, ISCSI_WQE_WQE_TYPE,
+ ISCSI_WQE_TYPE_NORMAL);
+
+ if (task_params->tx_io_size) {
+ buf_size = calc_rw_task_size(task_params, task_type,
+ sgl_task_params,
+ dif_task_params);
+
+ if (scsi_is_slow_sgl(sgl_task_params->num_sges,
+ sgl_task_params->small_mid_sge))
+ num_sges = ISCSI_WQE_NUM_SGES_SLOWIO;
+ else
+ num_sges = min(sgl_task_params->num_sges,
+ (u16)SCSI_NUM_SGES_SLOW_SGL_THR);
+ }
+
+ SET_FIELD(task_params->sqe->flags, ISCSI_WQE_NUM_SGES, num_sges);
+ SET_FIELD(task_params->sqe->contlen_cdbsize, ISCSI_WQE_CONT_LEN,
+ buf_size);
+
+ if (GET_FIELD(pdu_header->hdr_second_dword,
+ ISCSI_CMD_HDR_TOTAL_AHS_LEN))
+ SET_FIELD(task_params->sqe->contlen_cdbsize, ISCSI_WQE_CDB_SIZE,
+ cmd_params->extended_cdb_sge.sge_len);
+ }
+ break;
+ case ISCSI_TASK_TYPE_INITIATOR_READ:
+ SET_FIELD(task_params->sqe->flags, ISCSI_WQE_WQE_TYPE,
+ ISCSI_WQE_TYPE_NORMAL);
+
+ if (GET_FIELD(pdu_header->hdr_second_dword,
+ ISCSI_CMD_HDR_TOTAL_AHS_LEN))
+ SET_FIELD(task_params->sqe->contlen_cdbsize,
+ ISCSI_WQE_CDB_SIZE,
+ cmd_params->extended_cdb_sge.sge_len);
+ break;
+ case ISCSI_TASK_TYPE_LOGIN_RESPONSE:
+ case ISCSI_TASK_TYPE_MIDPATH:
+ {
+ bool advance_statsn = true;
+
+ if (task_type == ISCSI_TASK_TYPE_LOGIN_RESPONSE)
+ SET_FIELD(task_params->sqe->flags, ISCSI_WQE_WQE_TYPE,
+ ISCSI_WQE_TYPE_LOGIN);
+ else
+ SET_FIELD(task_params->sqe->flags, ISCSI_WQE_WQE_TYPE,
+ ISCSI_WQE_TYPE_MIDDLE_PATH);
+
+ if (task_type == ISCSI_TASK_TYPE_MIDPATH) {
+ u8 opcode = GET_FIELD(pdu_header->hdr_first_byte,
+ ISCSI_COMMON_HDR_OPCODE);
+
+ if (opcode != ISCSI_OPCODE_TEXT_RESPONSE &&
+ (opcode != ISCSI_OPCODE_NOP_IN ||
+ pdu_header->itt == ISCSI_TTT_ALL_ONES))
+ advance_statsn = false;
+ }
+
+ SET_FIELD(task_params->sqe->flags, ISCSI_WQE_RESPONSE,
+ advance_statsn ? 1 : 0);
+
+ if (task_params->tx_io_size) {
+ SET_FIELD(task_params->sqe->contlen_cdbsize,
+ ISCSI_WQE_CONT_LEN, task_params->tx_io_size);
+
+ if (scsi_is_slow_sgl(sgl_task_params->num_sges,
+ sgl_task_params->small_mid_sge))
+ SET_FIELD(task_params->sqe->flags, ISCSI_WQE_NUM_SGES,
+ ISCSI_WQE_NUM_SGES_SLOWIO);
+ else
+ SET_FIELD(task_params->sqe->flags, ISCSI_WQE_NUM_SGES,
+ min(sgl_task_params->num_sges,
+ (u16)SCSI_NUM_SGES_SLOW_SGL_THR));
+ }
+ }
+ break;
+ default:
+ break;
+ }
+}
+
+static void init_default_iscsi_task(struct iscsi_task_params *task_params,
+ struct data_hdr *pdu_header,
+ enum iscsi_task_type task_type)
+{
+ struct iscsi_task_context *context;
+ u16 index;
+ u32 val;
+
+ context = task_params->context;
+ memset(context, 0, sizeof(*context));
+
+ for (index = 0; index <
+ ARRAY_SIZE(context->ystorm_st_context.pdu_hdr.data.data);
+ index++) {
+ val = cpu_to_le32(pdu_header->data[index]);
+ context->ystorm_st_context.pdu_hdr.data.data[index] = val;
+ }
+
+ context->mstorm_st_context.task_type = task_type;
+ context->mstorm_ag_context.task_cid =
+ cpu_to_le16(task_params->conn_icid);
+
+ SET_FIELD(context->ustorm_ag_context.flags1,
+ USTORM_ISCSI_TASK_AG_CTX_R2T2RECV, 1);
+
+ context->ustorm_st_context.task_type = task_type;
+ context->ustorm_st_context.cq_rss_number = task_params->cq_rss_number;
+ context->ustorm_ag_context.icid = cpu_to_le16(task_params->conn_icid);
+}
+
+static
+void init_initiator_rw_cdb_ystorm_context(struct ystorm_iscsi_task_st_ctx *ystc,
+ struct scsi_initiator_cmd_params *cmd)
+{
+ union iscsi_task_hdr *ctx_pdu_hdr = &ystc->pdu_hdr;
+ u32 val;
+
+ if (!cmd->extended_cdb_sge.sge_len)
+ return;
+
+ SET_FIELD(ctx_pdu_hdr->ext_cdb_cmd.hdr_second_dword,
+ ISCSI_EXT_CDB_CMD_HDR_CDB_SIZE,
+ cmd->extended_cdb_sge.sge_len);
+ val = cpu_to_le32(cmd->extended_cdb_sge.sge_addr.lo);
+ ctx_pdu_hdr->ext_cdb_cmd.cdb_sge.sge_addr.lo = val;
+ val = cpu_to_le32(cmd->extended_cdb_sge.sge_addr.hi);
+ ctx_pdu_hdr->ext_cdb_cmd.cdb_sge.sge_addr.hi = val;
+ val = cpu_to_le32(cmd->extended_cdb_sge.sge_len);
+ ctx_pdu_hdr->ext_cdb_cmd.cdb_sge.sge_len = val;
+}
+
+static
+void init_ustorm_task_contexts(struct ustorm_iscsi_task_st_ctx *ustorm_st_cxt,
+ struct ustorm_iscsi_task_ag_ctx *ustorm_ag_cxt,
+ u32 remaining_recv_len,
+ u32 expected_data_transfer_len,
+ u8 num_sges, bool tx_dif_conn_err_en)
+{
+ u32 val;
+
+ ustorm_st_cxt->rem_rcv_len = cpu_to_le32(remaining_recv_len);
+ ustorm_ag_cxt->exp_data_acked = cpu_to_le32(expected_data_transfer_len);
+ val = cpu_to_le32(expected_data_transfer_len);
+ ustorm_st_cxt->exp_data_transfer_len = val;
+ SET_FIELD(ustorm_st_cxt->reg1.reg1_map, ISCSI_REG1_NUM_SGES, num_sges);
+ SET_FIELD(ustorm_ag_cxt->flags2,
+ USTORM_ISCSI_TASK_AG_CTX_DIF_ERROR_CF_EN,
+ tx_dif_conn_err_en ? 1 : 0);
+}
+
+static
+void set_rw_exp_data_acked_and_cont_len(struct iscsi_task_context *context,
+ struct iscsi_conn_params *conn_params,
+ enum iscsi_task_type task_type,
+ u32 task_size,
+ u32 exp_data_transfer_len,
+ u8 total_ahs_length)
+{
+ u32 max_unsolicited_data = 0, val;
+
+ if (total_ahs_length &&
+ (task_type == ISCSI_TASK_TYPE_INITIATOR_WRITE ||
+ task_type == ISCSI_TASK_TYPE_INITIATOR_READ))
+ SET_FIELD(context->ustorm_st_context.flags2,
+ USTORM_ISCSI_TASK_ST_CTX_AHS_EXIST, 1);
+
+ switch (task_type) {
+ case ISCSI_TASK_TYPE_INITIATOR_WRITE:
+ if (!conn_params->initial_r2t)
+ max_unsolicited_data = conn_params->first_burst_length;
+ else if (conn_params->immediate_data)
+ max_unsolicited_data =
+ min(conn_params->first_burst_length,
+ conn_params->max_send_pdu_length);
+
+ context->ustorm_ag_context.exp_data_acked =
+ cpu_to_le32(total_ahs_length == 0 ?
+ min(exp_data_transfer_len,
+ max_unsolicited_data) :
+ ((u32)(total_ahs_length +
+ ISCSI_AHS_CNTL_SIZE)));
+ break;
+ case ISCSI_TASK_TYPE_TARGET_READ:
+ val = cpu_to_le32(exp_data_transfer_len);
+ context->ustorm_ag_context.exp_data_acked = val;
+ break;
+ case ISCSI_TASK_TYPE_INITIATOR_READ:
+ context->ustorm_ag_context.exp_data_acked =
+ cpu_to_le32((total_ahs_length == 0 ? 0 :
+ total_ahs_length +
+ ISCSI_AHS_CNTL_SIZE));
+ break;
+ case ISCSI_TASK_TYPE_TARGET_WRITE:
+ val = cpu_to_le32(task_size);
+ context->ustorm_ag_context.exp_cont_len = val;
+ break;
+ default:
+ break;
+ }
+}
+
+static
+void init_rtdif_task_context(struct rdif_task_context *rdif_context,
+ struct tdif_task_context *tdif_context,
+ struct scsi_dif_task_params *dif_task_params,
+ enum iscsi_task_type task_type)
+{
+ u32 val;
+
+ if (!dif_task_params->dif_on_network || !dif_task_params->dif_on_host)
+ return;
+
+ if (task_type == ISCSI_TASK_TYPE_TARGET_WRITE ||
+ task_type == ISCSI_TASK_TYPE_INITIATOR_READ) {
+ rdif_context->app_tag_value =
+ cpu_to_le16(dif_task_params->application_tag);
+ rdif_context->partial_crc_value = cpu_to_le16(0xffff);
+ val = cpu_to_le32(dif_task_params->initial_ref_tag);
+ rdif_context->initial_ref_tag = val;
+ rdif_context->app_tag_mask =
+ cpu_to_le16(dif_task_params->application_tag_mask);
+ SET_FIELD(rdif_context->flags0, RDIF_TASK_CONTEXT_CRC_SEED,
+ dif_task_params->crc_seed ? 1 : 0);
+ SET_FIELD(rdif_context->flags0, RDIF_TASK_CONTEXT_HOSTGUARDTYPE,
+ dif_task_params->host_guard_type);
+ SET_FIELD(rdif_context->flags0,
+ RDIF_TASK_CONTEXT_PROTECTIONTYPE,
+ dif_task_params->protection_type);
+ SET_FIELD(rdif_context->flags0,
+ RDIF_TASK_CONTEXT_INITIALREFTAGVALID, 1);
+ SET_FIELD(rdif_context->flags0,
+ RDIF_TASK_CONTEXT_KEEPREFTAGCONST,
+ dif_task_params->keep_ref_tag_const ? 1 : 0);
+ SET_FIELD(rdif_context->flags1,
+ RDIF_TASK_CONTEXT_VALIDATEAPPTAG,
+ (dif_task_params->validate_app_tag &&
+ dif_task_params->dif_on_network) ? 1 : 0);
+ SET_FIELD(rdif_context->flags1,
+ RDIF_TASK_CONTEXT_VALIDATEGUARD,
+ (dif_task_params->validate_guard &&
+ dif_task_params->dif_on_network) ? 1 : 0);
+ SET_FIELD(rdif_context->flags1,
+ RDIF_TASK_CONTEXT_VALIDATEREFTAG,
+ (dif_task_params->validate_ref_tag &&
+ dif_task_params->dif_on_network) ? 1 : 0);
+ SET_FIELD(rdif_context->flags1,
+ RDIF_TASK_CONTEXT_HOSTINTERFACE,
+ dif_task_params->dif_on_host ? 1 : 0);
+ SET_FIELD(rdif_context->flags1,
+ RDIF_TASK_CONTEXT_NETWORKINTERFACE,
+ dif_task_params->dif_on_network ? 1 : 0);
+ SET_FIELD(rdif_context->flags1,
+ RDIF_TASK_CONTEXT_FORWARDGUARD,
+ dif_task_params->forward_guard ? 1 : 0);
+ SET_FIELD(rdif_context->flags1,
+ RDIF_TASK_CONTEXT_FORWARDAPPTAG,
+ dif_task_params->forward_app_tag ? 1 : 0);
+ SET_FIELD(rdif_context->flags1,
+ RDIF_TASK_CONTEXT_FORWARDREFTAG,
+ dif_task_params->forward_ref_tag ? 1 : 0);
+ SET_FIELD(rdif_context->flags1,
+ RDIF_TASK_CONTEXT_FORWARDAPPTAGWITHMASK,
+ dif_task_params->forward_app_tag_with_mask ? 1 : 0);
+ SET_FIELD(rdif_context->flags1,
+ RDIF_TASK_CONTEXT_FORWARDREFTAGWITHMASK,
+ dif_task_params->forward_ref_tag_with_mask ? 1 : 0);
+ SET_FIELD(rdif_context->flags1,
+ RDIF_TASK_CONTEXT_INTERVALSIZE,
+ dif_task_params->dif_block_size_log - 9);
+ SET_FIELD(rdif_context->state,
+ RDIF_TASK_CONTEXT_REFTAGMASK,
+ dif_task_params->ref_tag_mask);
+ SET_FIELD(rdif_context->state, RDIF_TASK_CONTEXT_IGNOREAPPTAG,
+ dif_task_params->ignore_app_tag);
+ }
+
+ if (task_type == ISCSI_TASK_TYPE_TARGET_READ ||
+ task_type == ISCSI_TASK_TYPE_INITIATOR_WRITE) {
+ tdif_context->app_tag_value =
+ cpu_to_le16(dif_task_params->application_tag);
+ tdif_context->partial_crc_valueB =
+ cpu_to_le16(dif_task_params->crc_seed ? 0xffff : 0x0000);
+ tdif_context->partial_crc_value_a =
+ cpu_to_le16(dif_task_params->crc_seed ? 0xffff : 0x0000);
+ SET_FIELD(tdif_context->flags0, TDIF_TASK_CONTEXT_CRC_SEED,
+ dif_task_params->crc_seed ? 1 : 0);
+
+ SET_FIELD(tdif_context->flags0,
+ TDIF_TASK_CONTEXT_SETERRORWITHEOP,
+ dif_task_params->tx_dif_conn_err_en ? 1 : 0);
+ SET_FIELD(tdif_context->flags1, TDIF_TASK_CONTEXT_FORWARDGUARD,
+ dif_task_params->forward_guard ? 1 : 0);
+ SET_FIELD(tdif_context->flags1, TDIF_TASK_CONTEXT_FORWARDAPPTAG,
+ dif_task_params->forward_app_tag ? 1 : 0);
+ SET_FIELD(tdif_context->flags1, TDIF_TASK_CONTEXT_FORWARDREFTAG,
+ dif_task_params->forward_ref_tag ? 1 : 0);
+ SET_FIELD(tdif_context->flags1, TDIF_TASK_CONTEXT_INTERVALSIZE,
+ dif_task_params->dif_block_size_log - 9);
+ SET_FIELD(tdif_context->flags1, TDIF_TASK_CONTEXT_HOSTINTERFACE,
+ dif_task_params->dif_on_host ? 1 : 0);
+ SET_FIELD(tdif_context->flags1,
+ TDIF_TASK_CONTEXT_NETWORKINTERFACE,
+ dif_task_params->dif_on_network ? 1 : 0);
+ val = cpu_to_le32(dif_task_params->initial_ref_tag);
+ tdif_context->initial_ref_tag = val;
+ tdif_context->app_tag_mask =
+ cpu_to_le16(dif_task_params->application_tag_mask);
+ SET_FIELD(tdif_context->flags0,
+ TDIF_TASK_CONTEXT_HOSTGUARDTYPE,
+ dif_task_params->host_guard_type);
+ SET_FIELD(tdif_context->flags0,
+ TDIF_TASK_CONTEXT_PROTECTIONTYPE,
+ dif_task_params->protection_type);
+ SET_FIELD(tdif_context->flags0,
+ TDIF_TASK_CONTEXT_INITIALREFTAGVALID,
+ dif_task_params->initial_ref_tag_is_valid ? 1 : 0);
+ SET_FIELD(tdif_context->flags0,
+ TDIF_TASK_CONTEXT_KEEPREFTAGCONST,
+ dif_task_params->keep_ref_tag_const ? 1 : 0);
+ SET_FIELD(tdif_context->flags1, TDIF_TASK_CONTEXT_VALIDATEGUARD,
+ (dif_task_params->validate_guard &&
+ dif_task_params->dif_on_host) ? 1 : 0);
+ SET_FIELD(tdif_context->flags1,
+ TDIF_TASK_CONTEXT_VALIDATEAPPTAG,
+ (dif_task_params->validate_app_tag &&
+ dif_task_params->dif_on_host) ? 1 : 0);
+ SET_FIELD(tdif_context->flags1,
+ TDIF_TASK_CONTEXT_VALIDATEREFTAG,
+ (dif_task_params->validate_ref_tag &&
+ dif_task_params->dif_on_host) ? 1 : 0);
+ SET_FIELD(tdif_context->flags1,
+ TDIF_TASK_CONTEXT_FORWARDAPPTAGWITHMASK,
+ dif_task_params->forward_app_tag_with_mask ? 1 : 0);
+ SET_FIELD(tdif_context->flags1,
+ TDIF_TASK_CONTEXT_FORWARDREFTAGWITHMASK,
+ dif_task_params->forward_ref_tag_with_mask ? 1 : 0);
+ SET_FIELD(tdif_context->flags1,
+ TDIF_TASK_CONTEXT_REFTAGMASK,
+ dif_task_params->ref_tag_mask);
+ SET_FIELD(tdif_context->flags0,
+ TDIF_TASK_CONTEXT_IGNOREAPPTAG,
+ dif_task_params->ignore_app_tag ? 1 : 0);
+ }
+}
+
+static void set_local_completion_context(struct iscsi_task_context *context)
+{
+ SET_FIELD(context->ystorm_st_context.state.flags,
+ YSTORM_ISCSI_TASK_STATE_LOCAL_COMP, 1);
+ SET_FIELD(context->ustorm_st_context.flags,
+ USTORM_ISCSI_TASK_ST_CTX_LOCAL_COMP, 1);
+}
+
+static int init_rw_iscsi_task(struct iscsi_task_params *task_params,
+ enum iscsi_task_type task_type,
+ struct iscsi_conn_params *conn_params,
+ struct iscsi_common_hdr *pdu_header,
+ struct scsi_sgl_task_params *sgl_task_params,
+ struct scsi_initiator_cmd_params *cmd_params,
+ struct scsi_dif_task_params *dif_task_params)
+{
+ u32 exp_data_transfer_len = conn_params->max_burst_length;
+ struct iscsi_task_context *cxt;
+ bool slow_io = false;
+ u32 task_size, val;
+ u8 num_sges = 0;
+
+ task_size = calc_rw_task_size(task_params, task_type, sgl_task_params,
+ dif_task_params);
+
+ init_default_iscsi_task(task_params, (struct data_hdr *)pdu_header,
+ task_type);
+
+ cxt = task_params->context;
+
+ val = cpu_to_le32(task_size);
+ cxt->ystorm_st_context.pdu_hdr.cmd.expected_transfer_length = val;
+ init_initiator_rw_cdb_ystorm_context(&cxt->ystorm_st_context,
+ cmd_params);
+ val = cpu_to_le32(cmd_params->sense_data_buffer_phys_addr.lo);
+ cxt->mstorm_st_context.sense_db.lo = val;
+
+ val = cpu_to_le32(cmd_params->sense_data_buffer_phys_addr.hi);
+ cxt->mstorm_st_context.sense_db.hi = val;
+
+ if (task_params->tx_io_size) {
+ init_dif_context_flags(&cxt->ystorm_st_context.state.dif_flags,
+ dif_task_params);
+ init_scsi_sgl_context(&cxt->ystorm_st_context.state.sgl_params,
+ &cxt->ystorm_st_context.state.data_desc,
+ sgl_task_params);
+
+ slow_io = scsi_is_slow_sgl(sgl_task_params->num_sges,
+ sgl_task_params->small_mid_sge);
+
+ num_sges = !slow_io ? min_t(u16, sgl_task_params->num_sges,
+ (u16)SCSI_NUM_SGES_SLOW_SGL_THR) :
+ ISCSI_WQE_NUM_SGES_SLOWIO;
+
+ if (slow_io) {
+ SET_FIELD(cxt->ystorm_st_context.state.flags,
+ YSTORM_ISCSI_TASK_STATE_SLOW_IO, 1);
+ }
+ } else if (task_params->rx_io_size) {
+ init_dif_context_flags(&cxt->mstorm_st_context.dif_flags,
+ dif_task_params);
+ init_scsi_sgl_context(&cxt->mstorm_st_context.sgl_params,
+ &cxt->mstorm_st_context.data_desc,
+ sgl_task_params);
+ num_sges = !scsi_is_slow_sgl(sgl_task_params->num_sges,
+ sgl_task_params->small_mid_sge) ?
+ min_t(u16, sgl_task_params->num_sges,
+ (u16)SCSI_NUM_SGES_SLOW_SGL_THR) :
+ ISCSI_WQE_NUM_SGES_SLOWIO;
+ cxt->mstorm_st_context.rem_task_size = cpu_to_le32(task_size);
+ }
+
+ if (exp_data_transfer_len > task_size ||
+ task_type != ISCSI_TASK_TYPE_TARGET_WRITE)
+ exp_data_transfer_len = task_size;
+
+ init_ustorm_task_contexts(&task_params->context->ustorm_st_context,
+ &task_params->context->ustorm_ag_context,
+ task_size, exp_data_transfer_len, num_sges,
+ dif_task_params ?
+ dif_task_params->tx_dif_conn_err_en : false);
+
+ set_rw_exp_data_acked_and_cont_len(task_params->context, conn_params,
+ task_type, task_size,
+ exp_data_transfer_len,
+ GET_FIELD(pdu_header->hdr_second_dword,
+ ISCSI_CMD_HDR_TOTAL_AHS_LEN));
+
+ if (dif_task_params)
+ init_rtdif_task_context(&task_params->context->rdif_context,
+ &task_params->context->tdif_context,
+ dif_task_params, task_type);
+
+ init_sqe(task_params, sgl_task_params, dif_task_params, pdu_header,
+ cmd_params, task_type, false);
+
+ return 0;
+}
+
+int init_initiator_rw_iscsi_task(struct iscsi_task_params *task_params,
+ struct iscsi_conn_params *conn_params,
+ struct scsi_initiator_cmd_params *cmd_params,
+ struct iscsi_cmd_hdr *cmd_header,
+ struct scsi_sgl_task_params *tx_sgl_params,
+ struct scsi_sgl_task_params *rx_sgl_params,
+ struct scsi_dif_task_params *dif_task_params)
+{
+ if (GET_FIELD(cmd_header->flags_attr, ISCSI_CMD_HDR_WRITE))
+ return init_rw_iscsi_task(task_params,
+ ISCSI_TASK_TYPE_INITIATOR_WRITE,
+ conn_params,
+ (struct iscsi_common_hdr *)cmd_header,
+ tx_sgl_params, cmd_params,
+ dif_task_params);
+ else if (GET_FIELD(cmd_header->flags_attr, ISCSI_CMD_HDR_READ))
+ return init_rw_iscsi_task(task_params,
+ ISCSI_TASK_TYPE_INITIATOR_READ,
+ conn_params,
+ (struct iscsi_common_hdr *)cmd_header,
+ rx_sgl_params, cmd_params,
+ dif_task_params);
+ else
+ return -1;
+}
+
+int init_initiator_login_request_task(struct iscsi_task_params *task_params,
+ struct iscsi_login_req_hdr *login_header,
+ struct scsi_sgl_task_params *tx_params,
+ struct scsi_sgl_task_params *rx_params)
+{
+ struct iscsi_task_context *cxt;
+
+ cxt = task_params->context;
+
+ init_default_iscsi_task(task_params,
+ (struct data_hdr *)login_header,
+ ISCSI_TASK_TYPE_MIDPATH);
+
+ init_ustorm_task_contexts(&cxt->ustorm_st_context,
+ &cxt->ustorm_ag_context,
+ task_params->rx_io_size ?
+ rx_params->total_buffer_size : 0,
+ task_params->tx_io_size ?
+ tx_params->total_buffer_size : 0, 0,
+ 0);
+
+ if (task_params->tx_io_size)
+ init_scsi_sgl_context(&cxt->ystorm_st_context.state.sgl_params,
+ &cxt->ystorm_st_context.state.data_desc,
+ tx_params);
+
+ if (task_params->rx_io_size)
+ init_scsi_sgl_context(&cxt->mstorm_st_context.sgl_params,
+ &cxt->mstorm_st_context.data_desc,
+ rx_params);
+
+ cxt->mstorm_st_context.rem_task_size =
+ cpu_to_le32(task_params->rx_io_size ?
+ rx_params->total_buffer_size : 0);
+
+ init_sqe(task_params, tx_params, NULL,
+ (struct iscsi_common_hdr *)login_header, NULL,
+ ISCSI_TASK_TYPE_MIDPATH, false);
+
+ return 0;
+}
+
+int init_initiator_nop_out_task(struct iscsi_task_params *task_params,
+ struct iscsi_nop_out_hdr *nop_out_pdu_header,
+ struct scsi_sgl_task_params *tx_sgl_task_params,
+ struct scsi_sgl_task_params *rx_sgl_task_params)
+{
+ struct iscsi_task_context *cxt;
+
+ cxt = task_params->context;
+
+ init_default_iscsi_task(task_params,
+ (struct data_hdr *)nop_out_pdu_header,
+ ISCSI_TASK_TYPE_MIDPATH);
+
+ if (nop_out_pdu_header->itt == ISCSI_ITT_ALL_ONES)
+ set_local_completion_context(task_params->context);
+
+ if (task_params->tx_io_size)
+ init_scsi_sgl_context(&cxt->ystorm_st_context.state.sgl_params,
+ &cxt->ystorm_st_context.state.data_desc,
+ tx_sgl_task_params);
+
+ if (task_params->rx_io_size)
+ init_scsi_sgl_context(&cxt->mstorm_st_context.sgl_params,
+ &cxt->mstorm_st_context.data_desc,
+ rx_sgl_task_params);
+
+ init_ustorm_task_contexts(&cxt->ustorm_st_context,
+ &cxt->ustorm_ag_context,
+ task_params->rx_io_size ?
+ rx_sgl_task_params->total_buffer_size : 0,
+ task_params->tx_io_size ?
+ tx_sgl_task_params->total_buffer_size : 0,
+ 0, 0);
+
+ cxt->mstorm_st_context.rem_task_size =
+ cpu_to_le32(task_params->rx_io_size ?
+ rx_sgl_task_params->total_buffer_size :
+ 0);
+
+ init_sqe(task_params, tx_sgl_task_params, NULL,
+ (struct iscsi_common_hdr *)nop_out_pdu_header, NULL,
+ ISCSI_TASK_TYPE_MIDPATH, false);
+
+ return 0;
+}
+
+int init_initiator_logout_request_task(struct iscsi_task_params *task_params,
+ struct iscsi_logout_req_hdr *logout_hdr,
+ struct scsi_sgl_task_params *tx_params,
+ struct scsi_sgl_task_params *rx_params)
+{
+ struct iscsi_task_context *cxt;
+
+ cxt = task_params->context;
+
+ init_default_iscsi_task(task_params,
+ (struct data_hdr *)logout_hdr,
+ ISCSI_TASK_TYPE_MIDPATH);
+
+ if (task_params->tx_io_size)
+ init_scsi_sgl_context(&cxt->ystorm_st_context.state.sgl_params,
+ &cxt->ystorm_st_context.state.data_desc,
+ tx_params);
+
+ if (task_params->rx_io_size)
+ init_scsi_sgl_context(&cxt->mstorm_st_context.sgl_params,
+ &cxt->mstorm_st_context.data_desc,
+ rx_params);
+
+ init_ustorm_task_contexts(&cxt->ustorm_st_context,
+ &cxt->ustorm_ag_context,
+ task_params->rx_io_size ?
+ rx_params->total_buffer_size : 0,
+ task_params->tx_io_size ?
+ tx_params->total_buffer_size : 0,
+ 0, 0);
+
+ cxt->mstorm_st_context.rem_task_size =
+ cpu_to_le32(task_params->rx_io_size ?
+ rx_params->total_buffer_size : 0);
+
+ init_sqe(task_params, tx_params, NULL,
+ (struct iscsi_common_hdr *)logout_hdr, NULL,
+ ISCSI_TASK_TYPE_MIDPATH, false);
+
+ return 0;
+}
+
+int init_initiator_tmf_request_task(struct iscsi_task_params *task_params,
+ struct iscsi_tmf_request_hdr *tmf_header)
+{
+ init_default_iscsi_task(task_params, (struct data_hdr *)tmf_header,
+ ISCSI_TASK_TYPE_MIDPATH);
+
+ init_sqe(task_params, NULL, NULL,
+ (struct iscsi_common_hdr *)tmf_header, NULL,
+ ISCSI_TASK_TYPE_MIDPATH, false);
+
+ return 0;
+}
+
+int init_initiator_text_request_task(struct iscsi_task_params *task_params,
+ struct iscsi_text_request_hdr *text_header,
+ struct scsi_sgl_task_params *tx_params,
+ struct scsi_sgl_task_params *rx_params)
+{
+ struct iscsi_task_context *cxt;
+
+ cxt = task_params->context;
+
+ init_default_iscsi_task(task_params,
+ (struct data_hdr *)text_header,
+ ISCSI_TASK_TYPE_MIDPATH);
+
+ if (task_params->tx_io_size)
+ init_scsi_sgl_context(&cxt->ystorm_st_context.state.sgl_params,
+ &cxt->ystorm_st_context.state.data_desc,
+ tx_params);
+
+ if (task_params->rx_io_size)
+ init_scsi_sgl_context(&cxt->mstorm_st_context.sgl_params,
+ &cxt->mstorm_st_context.data_desc,
+ rx_params);
+
+ cxt->mstorm_st_context.rem_task_size =
+ cpu_to_le32(task_params->rx_io_size ?
+ rx_params->total_buffer_size : 0);
+
+ init_ustorm_task_contexts(&cxt->ustorm_st_context,
+ &cxt->ustorm_ag_context,
+ task_params->rx_io_size ?
+ rx_params->total_buffer_size : 0,
+ task_params->tx_io_size ?
+ tx_params->total_buffer_size : 0, 0, 0);
+
+ init_sqe(task_params, tx_params, NULL,
+ (struct iscsi_common_hdr *)text_header, NULL,
+ ISCSI_TASK_TYPE_MIDPATH, false);
+
+ return 0;
+}
+
+int init_cleanup_task(struct iscsi_task_params *task_params)
+{
+ init_sqe(task_params, NULL, NULL, NULL, NULL, ISCSI_TASK_TYPE_MIDPATH,
+ true);
+ return 0;
+}
diff --git a/drivers/scsi/qedi/qedi_fw_iscsi.h b/drivers/scsi/qedi/qedi_fw_iscsi.h
new file mode 100644
index 0000000..b6f24f9
--- /dev/null
+++ b/drivers/scsi/qedi/qedi_fw_iscsi.h
@@ -0,0 +1,117 @@
+/*
+ * QLogic iSCSI Offload Driver
+ * Copyright (c) 2016 Cavium Inc.
+ *
+ * This software is available under the terms of the GNU General Public License
+ * (GPL) Version 2, available from the file COPYING in the main directory of
+ * this source tree.
+ */
+
+#ifndef _QEDI_FW_ISCSI_H_
+#define _QEDI_FW_ISCSI_H_
+
+#include "qedi_fw_scsi.h"
+
+struct iscsi_task_params {
+ struct iscsi_task_context *context;
+ struct iscsi_wqe *sqe;
+ u32 tx_io_size;
+ u32 rx_io_size;
+ u16 conn_icid;
+ u16 itid;
+ u8 cq_rss_number;
+};
+
+struct iscsi_conn_params {
+ u32 first_burst_length;
+ u32 max_send_pdu_length;
+ u32 max_burst_length;
+ bool initial_r2t;
+ bool immediate_data;
+};
+
+/* @brief init_initiator_read_iscsi_task - initializes iSCSI Initiator Read
+ * task context.
+ *
+ * @param task_params - Pointer to task parameters struct
+ * @param conn_params - Connection Parameters
+ * @param cmd_params - command specific parameters
+ * @param cmd_pdu_header - PDU Header Parameters
+ * @param sgl_task_params - Pointer to SGL task params
+ * @param dif_task_params - Pointer to DIF parameters struct
+ */
+int init_initiator_rw_iscsi_task(struct iscsi_task_params *task_params,
+ struct iscsi_conn_params *conn_params,
+ struct scsi_initiator_cmd_params *cmd_params,
+ struct iscsi_cmd_hdr *cmd_pdu_header,
+ struct scsi_sgl_task_params *tx_sgl_params,
+ struct scsi_sgl_task_params *rx_sgl_params,
+ struct scsi_dif_task_params *dif_task_params);
+
+/* @brief init_initiator_login_request_task - initializes iSCSI Initiator Login
+ * Request task context.
+ *
+ * @param task_params - Pointer to task parameters struct
+ * @param login_req_pdu_header - PDU Header Parameters
+ * @param tx_sgl_task_params - Pointer to SGL task params
+ * @param rx_sgl_task_params - Pointer to SGL task params
+ */
+int init_initiator_login_request_task(struct iscsi_task_params *task_params,
+ struct iscsi_login_req_hdr *login_header,
+ struct scsi_sgl_task_params *tx_params,
+ struct scsi_sgl_task_params *rx_params);
+
+/* @brief init_initiator_nop_out_task - initializes iSCSI Initiator NOP Out
+ * task context.
+ *
+ * @param task_params - Pointer to task parameters struct
+ * @param nop_out_pdu_header - PDU Header Parameters
+ * @param tx_sgl_task_params - Pointer to SGL task params
+ * @param rx_sgl_task_params - Pointer to SGL task params
+ */
+int init_initiator_nop_out_task(struct iscsi_task_params *task_params,
+ struct iscsi_nop_out_hdr *nop_out_pdu_header,
+ struct scsi_sgl_task_params *tx_sgl_params,
+ struct scsi_sgl_task_params *rx_sgl_params);
+
+/* @brief init_initiator_logout_request_task - initializes iSCSI Initiator
+ * Logout Request task context.
+ *
+ * @param task_params - Pointer to task parameters struct
+ * @param logout_pdu_header - PDU Header Parameters
+ * @param tx_sgl_task_params - Pointer to SGL task params
+ * @param rx_sgl_task_params - Pointer to SGL task params
+ */
+int init_initiator_logout_request_task(struct iscsi_task_params *task_params,
+ struct iscsi_logout_req_hdr *logout_hdr,
+ struct scsi_sgl_task_params *tx_params,
+ struct scsi_sgl_task_params *rx_params);
+
+/* @brief init_initiator_tmf_request_task - initializes iSCSI Initiator TMF
+ * task context.
+ *
+ * @param task_params - Pointer to task parameters struct
+ * @param tmf_pdu_header - PDU Header Parameters
+ */
+int init_initiator_tmf_request_task(struct iscsi_task_params *task_params,
+ struct iscsi_tmf_request_hdr *tmf_header);
+
+/* @brief init_initiator_text_request_task - initializes iSCSI Initiator Text
+ * Request task context.
+ *
+ * @param task_params - Pointer to task parameters struct
+ * @param text_request_pdu_header - PDU Header Parameters
+ * @param tx_sgl_task_params - Pointer to Tx SGL task params
+ * @param rx_sgl_task_params - Pointer to Rx SGL task params
+ */
+int init_initiator_text_request_task(struct iscsi_task_params *task_params,
+ struct iscsi_text_request_hdr *text_header,
+ struct scsi_sgl_task_params *tx_params,
+ struct scsi_sgl_task_params *rx_params);
+
+/* @brief init_cleanup_task - initializes Clean task (SQE)
+ *
+ * @param task_params - Pointer to task parameters struct
+ */
+int init_cleanup_task(struct iscsi_task_params *task_params);
+#endif
diff --git a/drivers/scsi/qedi/qedi_fw_scsi.h b/drivers/scsi/qedi/qedi_fw_scsi.h
new file mode 100644
index 0000000..cdaf918
--- /dev/null
+++ b/drivers/scsi/qedi/qedi_fw_scsi.h
@@ -0,0 +1,55 @@
+/*
+ * QLogic iSCSI Offload Driver
+ * Copyright (c) 2016 Cavium Inc.
+ *
+ * This software is available under the terms of the GNU General Public License
+ * (GPL) Version 2, available from the file COPYING in the main directory of
+ * this source tree.
+ */
+
+#ifndef _QEDI_FW_SCSI_H_
+#define _QEDI_FW_SCSI_H_
+
+#include <linux/types.h>
+#include <asm/byteorder.h>
+#include "qedi_hsi.h"
+#include <linux/qed/qed_if.h>
+
+struct scsi_sgl_task_params {
+ struct scsi_sge *sgl;
+ struct regpair sgl_phys_addr;
+ u32 total_buffer_size;
+ u16 num_sges;
+ bool small_mid_sge;
+};
+
+struct scsi_dif_task_params {
+ u32 initial_ref_tag;
+ bool initial_ref_tag_is_valid;
+ u16 application_tag;
+ u16 application_tag_mask;
+ u16 dif_block_size_log;
+ bool dif_on_network;
+ bool dif_on_host;
+ u8 host_guard_type;
+ u8 protection_type;
+ u8 ref_tag_mask;
+ bool crc_seed;
+ bool tx_dif_conn_err_en;
+ bool ignore_app_tag;
+ bool keep_ref_tag_const;
+ bool validate_guard;
+ bool validate_app_tag;
+ bool validate_ref_tag;
+ bool forward_guard;
+ bool forward_app_tag;
+ bool forward_ref_tag;
+ bool forward_app_tag_with_mask;
+ bool forward_ref_tag_with_mask;
+};
+
+struct scsi_initiator_cmd_params {
+ struct scsi_sge extended_cdb_sge;
+ struct regpair sense_data_buffer_phys_addr;
+};
+#endif
diff --git a/drivers/scsi/qedi/qedi_iscsi.c b/drivers/scsi/qedi/qedi_iscsi.c
index 4cc4743..d1de172 100644
--- a/drivers/scsi/qedi/qedi_iscsi.c
+++ b/drivers/scsi/qedi/qedi_iscsi.c
@@ -175,7 +175,7 @@
if (cmd->io_tbl.sge_tbl)
dma_free_coherent(&qedi->pdev->dev,
QEDI_ISCSI_MAX_BDS_PER_CMD *
- sizeof(struct iscsi_sge),
+ sizeof(struct scsi_sge),
cmd->io_tbl.sge_tbl,
cmd->io_tbl.sge_tbl_dma);
@@ -191,7 +191,7 @@
struct qedi_cmd *cmd)
{
struct qedi_io_bdt *io = &cmd->io_tbl;
- struct iscsi_sge *sge;
+ struct scsi_sge *sge;
io->sge_tbl = dma_alloc_coherent(&qedi->pdev->dev,
QEDI_ISCSI_MAX_BDS_PER_CMD *
@@ -708,22 +708,20 @@
static void qedi_iscsi_prep_generic_pdu_bd(struct qedi_conn *qedi_conn)
{
- struct iscsi_sge *bd_tbl;
+ struct scsi_sge *bd_tbl;
- bd_tbl = (struct iscsi_sge *)qedi_conn->gen_pdu.req_bd_tbl;
+ bd_tbl = (struct scsi_sge *)qedi_conn->gen_pdu.req_bd_tbl;
bd_tbl->sge_addr.hi =
(u32)((u64)qedi_conn->gen_pdu.req_dma_addr >> 32);
bd_tbl->sge_addr.lo = (u32)qedi_conn->gen_pdu.req_dma_addr;
bd_tbl->sge_len = qedi_conn->gen_pdu.req_wr_ptr -
qedi_conn->gen_pdu.req_buf;
- bd_tbl->reserved0 = 0;
- bd_tbl = (struct iscsi_sge *)qedi_conn->gen_pdu.resp_bd_tbl;
+ bd_tbl = (struct scsi_sge *)qedi_conn->gen_pdu.resp_bd_tbl;
bd_tbl->sge_addr.hi =
(u32)((u64)qedi_conn->gen_pdu.resp_dma_addr >> 32);
bd_tbl->sge_addr.lo = (u32)qedi_conn->gen_pdu.resp_dma_addr;
bd_tbl->sge_len = ISCSI_DEF_MAX_RECV_SEG_LEN;
- bd_tbl->reserved0 = 0;
}
static int qedi_iscsi_send_generic_request(struct iscsi_task *task)
diff --git a/drivers/scsi/qedi/qedi_iscsi.h b/drivers/scsi/qedi/qedi_iscsi.h
index d3c06bb..3247287 100644
--- a/drivers/scsi/qedi/qedi_iscsi.h
+++ b/drivers/scsi/qedi/qedi_iscsi.h
@@ -102,7 +102,7 @@
#define QEDI_SQ_WQES_MIN 16
struct qedi_io_bdt {
- struct iscsi_sge *sge_tbl;
+ struct scsi_sge *sge_tbl;
dma_addr_t sge_tbl_dma;
u16 sge_valid;
};
diff --git a/drivers/scsi/qedi/qedi_version.h b/drivers/scsi/qedi/qedi_version.h
index 9543a1b..d61e3ac 100644
--- a/drivers/scsi/qedi/qedi_version.h
+++ b/drivers/scsi/qedi/qedi_version.h
@@ -7,8 +7,8 @@
* this source tree.
*/
-#define QEDI_MODULE_VERSION "8.10.3.0"
+#define QEDI_MODULE_VERSION "8.10.4.0"
#define QEDI_DRIVER_MAJOR_VER 8
#define QEDI_DRIVER_MINOR_VER 10
-#define QEDI_DRIVER_REV_VER 3
+#define QEDI_DRIVER_REV_VER 4
#define QEDI_DRIVER_ENG_VER 0
diff --git a/drivers/scsi/qla2xxx/qla_bsg.c b/drivers/scsi/qla2xxx/qla_bsg.c
index 84c9098..b6e40fd 100644
--- a/drivers/scsi/qla2xxx/qla_bsg.c
+++ b/drivers/scsi/qla2xxx/qla_bsg.c
@@ -2553,13 +2553,13 @@
ql_log(ql_log_warn, vha, 0x7089,
"mbx abort_command "
"failed.\n");
- bsg_job->req->errors =
+ scsi_req(bsg_job->req)->result =
bsg_reply->result = -EIO;
} else {
ql_dbg(ql_dbg_user, vha, 0x708a,
"mbx abort_command "
"success.\n");
- bsg_job->req->errors =
+ scsi_req(bsg_job->req)->result =
bsg_reply->result = 0;
}
spin_lock_irqsave(&ha->hardware_lock, flags);
@@ -2570,7 +2570,7 @@
}
spin_unlock_irqrestore(&ha->hardware_lock, flags);
ql_log(ql_log_info, vha, 0x708b, "SRB not found to abort.\n");
- bsg_job->req->errors = bsg_reply->result = -ENXIO;
+ scsi_req(bsg_job->req)->result = bsg_reply->result = -ENXIO;
return 0;
done:
diff --git a/drivers/scsi/scsi_debugfs.c b/drivers/scsi/scsi_debugfs.c
new file mode 100644
index 0000000..a97c950
--- /dev/null
+++ b/drivers/scsi/scsi_debugfs.c
@@ -0,0 +1,13 @@
+#include <linux/seq_file.h>
+#include <scsi/scsi_cmnd.h>
+#include <scsi/scsi_dbg.h>
+#include "scsi_debugfs.h"
+
+void scsi_show_rq(struct seq_file *m, struct request *rq)
+{
+ struct scsi_cmnd *cmd = container_of(scsi_req(rq), typeof(*cmd), req);
+ char buf[80];
+
+ __scsi_format_command(buf, sizeof(buf), cmd->cmnd, cmd->cmd_len);
+ seq_printf(m, ", .cmd=%s", buf);
+}
diff --git a/drivers/scsi/scsi_debugfs.h b/drivers/scsi/scsi_debugfs.h
new file mode 100644
index 0000000..951b043
--- /dev/null
+++ b/drivers/scsi/scsi_debugfs.h
@@ -0,0 +1,4 @@
+struct request;
+struct seq_file;
+
+void scsi_show_rq(struct seq_file *m, struct request *rq);
diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
index f2cafae..2db412d 100644
--- a/drivers/scsi/scsi_error.c
+++ b/drivers/scsi/scsi_error.c
@@ -1988,7 +1988,7 @@
req->rq_flags |= RQF_QUIET;
req->timeout = 10 * HZ;
- req->retries = 5;
+ rq->retries = 5;
blk_execute_rq_nowait(req->q, NULL, req, 1, eh_lock_door_done);
}
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index e5a2d59..1c3e87d 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -34,6 +34,7 @@
#include <trace/events/scsi.h>
+#include "scsi_debugfs.h"
#include "scsi_priv.h"
#include "scsi_logging.h"
@@ -229,8 +230,8 @@
* @rq_flags: flags for ->rq_flags
* @resid: optional residual length
*
- * returns the req->errors value which is the scsi_cmnd result
- * field.
+ * Returns the scsi_cmnd result field if a command was executed, or a negative
+ * Linux error code if we didn't get that far.
*/
int scsi_execute(struct scsi_device *sdev, const unsigned char *cmd,
int data_direction, void *buffer, unsigned bufflen,
@@ -256,7 +257,7 @@
rq->cmd_len = COMMAND_SIZE(cmd[0]);
memcpy(rq->cmd, cmd, rq->cmd_len);
- req->retries = retries;
+ rq->retries = retries;
req->timeout = timeout;
req->cmd_flags |= flags;
req->rq_flags |= rq_flags | RQF_QUIET | RQF_PREEMPT;
@@ -281,7 +282,7 @@
memcpy(sense, rq->sense, SCSI_SENSE_BUFFERSIZE);
if (sshdr)
scsi_normalize_sense(rq->sense, rq->sense_len, sshdr);
- ret = req->errors;
+ ret = rq->result;
out:
blk_put_request(req);
@@ -797,8 +798,7 @@
/*
* __scsi_error_from_host_byte may have reset the host_byte
*/
- req->errors = cmd->result;
-
+ scsi_req(req)->result = cmd->result;
scsi_req(req)->resid_len = scsi_get_resid(cmd);
if (scsi_bidi_cmnd(cmd)) {
@@ -835,7 +835,7 @@
/*
* Recovered errors need reporting, but they're always treated as
* success, so fiddle the result code here. For passthrough requests
- * we already took a copy of the original into rq->errors which
+ * we already took a copy of the original into sreq->result which
* is what gets returned to the user
*/
if (sense_valid && (sshdr.sense_key == RECOVERED_ERROR)) {
@@ -1061,10 +1061,10 @@
struct scsi_device *sdev = cmd->device;
struct request *rq = cmd->request;
bool is_mq = (rq->mq_ctx != NULL);
- int error;
+ int error = BLKPREP_KILL;
if (WARN_ON_ONCE(!blk_rq_nr_phys_segments(rq)))
- return -EINVAL;
+ goto err_exit;
error = scsi_init_sgtable(rq, &cmd->sdb);
if (error)
@@ -1177,7 +1177,7 @@
cmd->cmd_len = scsi_req(req)->cmd_len;
cmd->cmnd = scsi_req(req)->cmd;
cmd->transfersize = blk_rq_bytes(req);
- cmd->allowed = req->retries;
+ cmd->allowed = scsi_req(req)->retries;
return BLKPREP_OK;
}
@@ -1281,7 +1281,7 @@
switch (ret) {
case BLKPREP_KILL:
case BLKPREP_INVALID:
- req->errors = DID_NO_CONNECT << 16;
+ scsi_req(req)->result = DID_NO_CONNECT << 16;
/* release the command and kill it */
if (req->special) {
struct scsi_cmnd *cmd = req->special;
@@ -1905,7 +1905,7 @@
static void scsi_mq_done(struct scsi_cmnd *cmd)
{
trace_scsi_dispatch_cmd_done(cmd);
- blk_mq_complete_request(cmd->request, cmd->request->errors);
+ blk_mq_complete_request(cmd->request);
}
static int scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
@@ -2154,10 +2154,13 @@
return q;
}
-static struct blk_mq_ops scsi_mq_ops = {
+static const struct blk_mq_ops scsi_mq_ops = {
.queue_rq = scsi_queue_rq,
.complete = scsi_softirq_done,
.timeout = scsi_timeout,
+#ifdef CONFIG_BLK_DEBUG_FS
+ .show_rq = scsi_show_rq,
+#endif
.init_request = scsi_init_request,
.exit_request = scsi_exit_request,
.map_queues = scsi_map_queues,
diff --git a/drivers/scsi/scsi_netlink.c b/drivers/scsi/scsi_netlink.c
index 109802f..50e624f 100644
--- a/drivers/scsi/scsi_netlink.c
+++ b/drivers/scsi/scsi_netlink.c
@@ -111,7 +111,7 @@
next_msg:
if ((err) || (nlh->nlmsg_flags & NLM_F_ACK))
- netlink_ack(skb, nlh, err);
+ netlink_ack(skb, nlh, err, NULL);
skb_pull(skb, rlen);
}
diff --git a/drivers/scsi/scsi_transport_sas.c b/drivers/scsi/scsi_transport_sas.c
index cdbb293..9fdbd50 100644
--- a/drivers/scsi/scsi_transport_sas.c
+++ b/drivers/scsi/scsi_transport_sas.c
@@ -184,9 +184,9 @@
blk_rq_bytes(req->next_rq);
handler = to_sas_internal(shost->transportt)->f->smp_handler;
ret = handler(shost, rphy, req);
- req->errors = ret;
+ scsi_req(req)->result = ret;
- blk_end_request_all(req, ret);
+ blk_end_request_all(req, 0);
spin_lock_irq(q->queue_lock);
}
diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c
index 35ad5e8..0dc95e1 100644
--- a/drivers/scsi/sd.c
+++ b/drivers/scsi/sd.c
@@ -418,6 +418,46 @@
}
static DEVICE_ATTR_RW(provisioning_mode);
+static const char *zeroing_mode[] = {
+ [SD_ZERO_WRITE] = "write",
+ [SD_ZERO_WS] = "writesame",
+ [SD_ZERO_WS16_UNMAP] = "writesame_16_unmap",
+ [SD_ZERO_WS10_UNMAP] = "writesame_10_unmap",
+};
+
+static ssize_t
+zeroing_mode_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct scsi_disk *sdkp = to_scsi_disk(dev);
+
+ return snprintf(buf, 20, "%s\n", zeroing_mode[sdkp->zeroing_mode]);
+}
+
+static ssize_t
+zeroing_mode_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct scsi_disk *sdkp = to_scsi_disk(dev);
+
+ if (!capable(CAP_SYS_ADMIN))
+ return -EACCES;
+
+ if (!strncmp(buf, zeroing_mode[SD_ZERO_WRITE], 20))
+ sdkp->zeroing_mode = SD_ZERO_WRITE;
+ else if (!strncmp(buf, zeroing_mode[SD_ZERO_WS], 20))
+ sdkp->zeroing_mode = SD_ZERO_WS;
+ else if (!strncmp(buf, zeroing_mode[SD_ZERO_WS16_UNMAP], 20))
+ sdkp->zeroing_mode = SD_ZERO_WS16_UNMAP;
+ else if (!strncmp(buf, zeroing_mode[SD_ZERO_WS10_UNMAP], 20))
+ sdkp->zeroing_mode = SD_ZERO_WS10_UNMAP;
+ else
+ return -EINVAL;
+
+ return count;
+}
+static DEVICE_ATTR_RW(zeroing_mode);
+
static ssize_t
max_medium_access_timeouts_show(struct device *dev,
struct device_attribute *attr, char *buf)
@@ -496,6 +536,7 @@
&dev_attr_app_tag_own.attr,
&dev_attr_thin_provisioning.attr,
&dev_attr_provisioning_mode.attr,
+ &dev_attr_zeroing_mode.attr,
&dev_attr_max_write_same_blocks.attr,
&dev_attr_max_medium_access_timeouts.attr,
NULL,
@@ -644,26 +685,11 @@
unsigned int logical_block_size = sdkp->device->sector_size;
unsigned int max_blocks = 0;
- q->limits.discard_zeroes_data = 0;
-
- /*
- * When LBPRZ is reported, discard alignment and granularity
- * must be fixed to the logical block size. Otherwise the block
- * layer will drop misaligned portions of the request which can
- * lead to data corruption. If LBPRZ is not set, we honor the
- * device preference.
- */
- if (sdkp->lbprz) {
- q->limits.discard_alignment = 0;
- q->limits.discard_granularity = logical_block_size;
- } else {
- q->limits.discard_alignment = sdkp->unmap_alignment *
- logical_block_size;
- q->limits.discard_granularity =
- max(sdkp->physical_block_size,
- sdkp->unmap_granularity * logical_block_size);
- }
-
+ q->limits.discard_alignment =
+ sdkp->unmap_alignment * logical_block_size;
+ q->limits.discard_granularity =
+ max(sdkp->physical_block_size,
+ sdkp->unmap_granularity * logical_block_size);
sdkp->provisioning_mode = mode;
switch (mode) {
@@ -681,19 +707,16 @@
case SD_LBP_WS16:
max_blocks = min_not_zero(sdkp->max_ws_blocks,
(u32)SD_MAX_WS16_BLOCKS);
- q->limits.discard_zeroes_data = sdkp->lbprz;
break;
case SD_LBP_WS10:
max_blocks = min_not_zero(sdkp->max_ws_blocks,
(u32)SD_MAX_WS10_BLOCKS);
- q->limits.discard_zeroes_data = sdkp->lbprz;
break;
case SD_LBP_ZERO:
max_blocks = min_not_zero(sdkp->max_ws_blocks,
(u32)SD_MAX_WS10_BLOCKS);
- q->limits.discard_zeroes_data = 1;
break;
}
@@ -701,93 +724,122 @@
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, q);
}
-/**
- * sd_setup_discard_cmnd - unmap blocks on thinly provisioned device
- * @sdp: scsi device to operate on
- * @rq: Request to prepare
- *
- * Will issue either UNMAP or WRITE SAME(16) depending on preference
- * indicated by target device.
- **/
-static int sd_setup_discard_cmnd(struct scsi_cmnd *cmd)
+static int sd_setup_unmap_cmnd(struct scsi_cmnd *cmd)
+{
+ struct scsi_device *sdp = cmd->device;
+ struct request *rq = cmd->request;
+ u64 sector = blk_rq_pos(rq) >> (ilog2(sdp->sector_size) - 9);
+ u32 nr_sectors = blk_rq_sectors(rq) >> (ilog2(sdp->sector_size) - 9);
+ unsigned int data_len = 24;
+ char *buf;
+
+ rq->special_vec.bv_page = alloc_page(GFP_ATOMIC | __GFP_ZERO);
+ if (!rq->special_vec.bv_page)
+ return BLKPREP_DEFER;
+ rq->special_vec.bv_offset = 0;
+ rq->special_vec.bv_len = data_len;
+ rq->rq_flags |= RQF_SPECIAL_PAYLOAD;
+
+ cmd->cmd_len = 10;
+ cmd->cmnd[0] = UNMAP;
+ cmd->cmnd[8] = 24;
+
+ buf = page_address(rq->special_vec.bv_page);
+ put_unaligned_be16(6 + 16, &buf[0]);
+ put_unaligned_be16(16, &buf[2]);
+ put_unaligned_be64(sector, &buf[8]);
+ put_unaligned_be32(nr_sectors, &buf[16]);
+
+ cmd->allowed = SD_MAX_RETRIES;
+ cmd->transfersize = data_len;
+ rq->timeout = SD_TIMEOUT;
+ scsi_req(rq)->resid_len = data_len;
+
+ return scsi_init_io(cmd);
+}
+
+static int sd_setup_write_same16_cmnd(struct scsi_cmnd *cmd, bool unmap)
+{
+ struct scsi_device *sdp = cmd->device;
+ struct request *rq = cmd->request;
+ u64 sector = blk_rq_pos(rq) >> (ilog2(sdp->sector_size) - 9);
+ u32 nr_sectors = blk_rq_sectors(rq) >> (ilog2(sdp->sector_size) - 9);
+ u32 data_len = sdp->sector_size;
+
+ rq->special_vec.bv_page = alloc_page(GFP_ATOMIC | __GFP_ZERO);
+ if (!rq->special_vec.bv_page)
+ return BLKPREP_DEFER;
+ rq->special_vec.bv_offset = 0;
+ rq->special_vec.bv_len = data_len;
+ rq->rq_flags |= RQF_SPECIAL_PAYLOAD;
+
+ cmd->cmd_len = 16;
+ cmd->cmnd[0] = WRITE_SAME_16;
+ if (unmap)
+ cmd->cmnd[1] = 0x8; /* UNMAP */
+ put_unaligned_be64(sector, &cmd->cmnd[2]);
+ put_unaligned_be32(nr_sectors, &cmd->cmnd[10]);
+
+ cmd->allowed = SD_MAX_RETRIES;
+ cmd->transfersize = data_len;
+ rq->timeout = unmap ? SD_TIMEOUT : SD_WRITE_SAME_TIMEOUT;
+ scsi_req(rq)->resid_len = data_len;
+
+ return scsi_init_io(cmd);
+}
+
+static int sd_setup_write_same10_cmnd(struct scsi_cmnd *cmd, bool unmap)
+{
+ struct scsi_device *sdp = cmd->device;
+ struct request *rq = cmd->request;
+ u64 sector = blk_rq_pos(rq) >> (ilog2(sdp->sector_size) - 9);
+ u32 nr_sectors = blk_rq_sectors(rq) >> (ilog2(sdp->sector_size) - 9);
+ u32 data_len = sdp->sector_size;
+
+ rq->special_vec.bv_page = alloc_page(GFP_ATOMIC | __GFP_ZERO);
+ if (!rq->special_vec.bv_page)
+ return BLKPREP_DEFER;
+ rq->special_vec.bv_offset = 0;
+ rq->special_vec.bv_len = data_len;
+ rq->rq_flags |= RQF_SPECIAL_PAYLOAD;
+
+ cmd->cmd_len = 10;
+ cmd->cmnd[0] = WRITE_SAME;
+ if (unmap)
+ cmd->cmnd[1] = 0x8; /* UNMAP */
+ put_unaligned_be32(sector, &cmd->cmnd[2]);
+ put_unaligned_be16(nr_sectors, &cmd->cmnd[7]);
+
+ cmd->allowed = SD_MAX_RETRIES;
+ cmd->transfersize = data_len;
+ rq->timeout = unmap ? SD_TIMEOUT : SD_WRITE_SAME_TIMEOUT;
+ scsi_req(rq)->resid_len = data_len;
+
+ return scsi_init_io(cmd);
+}
+
+static int sd_setup_write_zeroes_cmnd(struct scsi_cmnd *cmd)
{
struct request *rq = cmd->request;
struct scsi_device *sdp = cmd->device;
struct scsi_disk *sdkp = scsi_disk(rq->rq_disk);
- sector_t sector = blk_rq_pos(rq);
- unsigned int nr_sectors = blk_rq_sectors(rq);
- unsigned int len;
- int ret;
- char *buf;
- struct page *page;
+ u64 sector = blk_rq_pos(rq) >> (ilog2(sdp->sector_size) - 9);
+ u32 nr_sectors = blk_rq_sectors(rq) >> (ilog2(sdp->sector_size) - 9);
- sector >>= ilog2(sdp->sector_size) - 9;
- nr_sectors >>= ilog2(sdp->sector_size) - 9;
-
- page = alloc_page(GFP_ATOMIC | __GFP_ZERO);
- if (!page)
- return BLKPREP_DEFER;
-
- switch (sdkp->provisioning_mode) {
- case SD_LBP_UNMAP:
- buf = page_address(page);
-
- cmd->cmd_len = 10;
- cmd->cmnd[0] = UNMAP;
- cmd->cmnd[8] = 24;
-
- put_unaligned_be16(6 + 16, &buf[0]);
- put_unaligned_be16(16, &buf[2]);
- put_unaligned_be64(sector, &buf[8]);
- put_unaligned_be32(nr_sectors, &buf[16]);
-
- len = 24;
- break;
-
- case SD_LBP_WS16:
- cmd->cmd_len = 16;
- cmd->cmnd[0] = WRITE_SAME_16;
- cmd->cmnd[1] = 0x8; /* UNMAP */
- put_unaligned_be64(sector, &cmd->cmnd[2]);
- put_unaligned_be32(nr_sectors, &cmd->cmnd[10]);
-
- len = sdkp->device->sector_size;
- break;
-
- case SD_LBP_WS10:
- case SD_LBP_ZERO:
- cmd->cmd_len = 10;
- cmd->cmnd[0] = WRITE_SAME;
- if (sdkp->provisioning_mode == SD_LBP_WS10)
- cmd->cmnd[1] = 0x8; /* UNMAP */
- put_unaligned_be32(sector, &cmd->cmnd[2]);
- put_unaligned_be16(nr_sectors, &cmd->cmnd[7]);
-
- len = sdkp->device->sector_size;
- break;
-
- default:
- ret = BLKPREP_INVALID;
- goto out;
+ if (!(rq->cmd_flags & REQ_NOUNMAP)) {
+ switch (sdkp->zeroing_mode) {
+ case SD_ZERO_WS16_UNMAP:
+ return sd_setup_write_same16_cmnd(cmd, true);
+ case SD_ZERO_WS10_UNMAP:
+ return sd_setup_write_same10_cmnd(cmd, true);
+ }
}
- rq->timeout = SD_TIMEOUT;
-
- cmd->transfersize = len;
- cmd->allowed = SD_MAX_RETRIES;
-
- rq->special_vec.bv_page = page;
- rq->special_vec.bv_offset = 0;
- rq->special_vec.bv_len = len;
-
- rq->rq_flags |= RQF_SPECIAL_PAYLOAD;
- scsi_req(rq)->resid_len = len;
-
- ret = scsi_init_io(cmd);
-out:
- if (ret != BLKPREP_OK)
- __free_page(page);
- return ret;
+ if (sdp->no_write_same)
+ return BLKPREP_INVALID;
+ if (sdkp->ws16 || sector > 0xffffffff || nr_sectors > 0xffff)
+ return sd_setup_write_same16_cmnd(cmd, false);
+ return sd_setup_write_same10_cmnd(cmd, false);
}
static void sd_config_write_same(struct scsi_disk *sdkp)
@@ -816,9 +868,20 @@
sdkp->max_ws_blocks = 0;
}
+ if (sdkp->lbprz && sdkp->lbpws)
+ sdkp->zeroing_mode = SD_ZERO_WS16_UNMAP;
+ else if (sdkp->lbprz && sdkp->lbpws10)
+ sdkp->zeroing_mode = SD_ZERO_WS10_UNMAP;
+ else if (sdkp->max_ws_blocks)
+ sdkp->zeroing_mode = SD_ZERO_WS;
+ else
+ sdkp->zeroing_mode = SD_ZERO_WRITE;
+
out:
blk_queue_max_write_same_sectors(q, sdkp->max_ws_blocks *
(logical_block_size >> 9));
+ blk_queue_max_write_zeroes_sectors(q, sdkp->max_ws_blocks *
+ (logical_block_size >> 9));
}
/**
@@ -1155,7 +1218,20 @@
switch (req_op(rq)) {
case REQ_OP_DISCARD:
- return sd_setup_discard_cmnd(cmd);
+ switch (scsi_disk(rq->rq_disk)->provisioning_mode) {
+ case SD_LBP_UNMAP:
+ return sd_setup_unmap_cmnd(cmd);
+ case SD_LBP_WS16:
+ return sd_setup_write_same16_cmnd(cmd, true);
+ case SD_LBP_WS10:
+ return sd_setup_write_same10_cmnd(cmd, true);
+ case SD_LBP_ZERO:
+ return sd_setup_write_same10_cmnd(cmd, false);
+ default:
+ return BLKPREP_INVALID;
+ }
+ case REQ_OP_WRITE_ZEROES:
+ return sd_setup_write_zeroes_cmnd(cmd);
case REQ_OP_WRITE_SAME:
return sd_setup_write_same_cmnd(cmd);
case REQ_OP_FLUSH:
@@ -1795,6 +1871,7 @@
switch (req_op(req)) {
case REQ_OP_DISCARD:
+ case REQ_OP_WRITE_ZEROES:
case REQ_OP_WRITE_SAME:
case REQ_OP_ZONE_RESET:
if (!result) {
@@ -2768,7 +2845,7 @@
sd_config_discard(sdkp, SD_LBP_WS16);
} else { /* LBP VPD page tells us what to use */
- if (sdkp->lbpu && sdkp->max_unmap_blocks && !sdkp->lbprz)
+ if (sdkp->lbpu && sdkp->max_unmap_blocks)
sd_config_discard(sdkp, SD_LBP_UNMAP);
else if (sdkp->lbpws)
sd_config_discard(sdkp, SD_LBP_WS16);
diff --git a/drivers/scsi/sd.h b/drivers/scsi/sd.h
index 4dac35e..a2c4b5c 100644
--- a/drivers/scsi/sd.h
+++ b/drivers/scsi/sd.h
@@ -59,6 +59,13 @@
SD_LBP_DISABLE, /* Discard disabled due to failed cmd */
};
+enum {
+ SD_ZERO_WRITE = 0, /* Use WRITE(10/16) command */
+ SD_ZERO_WS, /* Use WRITE SAME(10/16) command */
+ SD_ZERO_WS16_UNMAP, /* Use WRITE SAME(16) with UNMAP */
+ SD_ZERO_WS10_UNMAP, /* Use WRITE SAME(10) with UNMAP */
+};
+
struct scsi_disk {
struct scsi_driver *driver; /* always &sd_template */
struct scsi_device *device;
@@ -89,6 +96,7 @@
u8 write_prot;
u8 protection_type;/* Data Integrity Field */
u8 provisioning_mode;
+ u8 zeroing_mode;
unsigned ATO : 1; /* state of disk ATO bit */
unsigned cache_override : 1; /* temp override of WCE,RCD */
unsigned WCE : 1; /* state of disk WCE bit */
diff --git a/drivers/scsi/sd_zbc.c b/drivers/scsi/sd_zbc.c
index 92620c8..1994f77 100644
--- a/drivers/scsi/sd_zbc.c
+++ b/drivers/scsi/sd_zbc.c
@@ -329,6 +329,7 @@
switch (req_op(rq)) {
case REQ_OP_WRITE:
+ case REQ_OP_WRITE_ZEROES:
case REQ_OP_WRITE_SAME:
case REQ_OP_ZONE_RESET:
diff --git a/drivers/scsi/sg.c b/drivers/scsi/sg.c
index 225abaa..0b60245 100644
--- a/drivers/scsi/sg.c
+++ b/drivers/scsi/sg.c
@@ -581,7 +581,7 @@
sg_io_hdr_t *hp;
unsigned char cmnd[SG_MAX_CDB_SIZE];
- if (unlikely(segment_eq(get_fs(), KERNEL_DS)))
+ if (unlikely(uaccess_kernel()))
return -EINVAL;
if ((!(sfp = (Sg_fd *) filp->private_data)) || (!(sdp = sfp->parentdp)))
@@ -1300,7 +1300,7 @@
pr_info("%s: device detaching\n", __func__);
sense = req->sense;
- result = rq->errors;
+ result = req->result;
resid = req->resid_len;
SCSI_LOG_TIMEOUT(4, sg_printk(KERN_INFO, sdp,
@@ -1718,7 +1718,7 @@
srp->rq = rq;
rq->end_io_data = srp;
- rq->retries = SG_DEFAULT_RETRIES;
+ req->retries = SG_DEFAULT_RETRIES;
if ((dxfer_len <= 0) || (dxfer_dir == SG_DXFER_NONE))
return 0;
diff --git a/drivers/scsi/snic/snic_scsi.c b/drivers/scsi/snic/snic_scsi.c
index abada16..da979a7 100644
--- a/drivers/scsi/snic/snic_scsi.c
+++ b/drivers/scsi/snic/snic_scsi.c
@@ -1066,7 +1066,7 @@
if (!snic->remove_wait) {
spin_unlock_irqrestore(io_lock, flags);
SNIC_HOST_ERR(snic->shost,
- "reset_cmpl:host reset completed after timout\n");
+ "reset_cmpl:host reset completed after timeout\n");
ret = 1;
return ret;
diff --git a/drivers/scsi/st.c b/drivers/scsi/st.c
index e5ef78a..1ea34d6 100644
--- a/drivers/scsi/st.c
+++ b/drivers/scsi/st.c
@@ -480,7 +480,7 @@
atomic64_add(ktime_to_ns(now), &STp->stats->tot_write_time);
atomic64_add(ktime_to_ns(now), &STp->stats->tot_io_time);
atomic64_inc(&STp->stats->write_cnt);
- if (req->errors) {
+ if (scsi_req(req)->result) {
atomic64_add(atomic_read(&STp->stats->last_write_size)
- STp->buffer->cmdstat.residual,
&STp->stats->write_byte_cnt);
@@ -494,7 +494,7 @@
atomic64_add(ktime_to_ns(now), &STp->stats->tot_read_time);
atomic64_add(ktime_to_ns(now), &STp->stats->tot_io_time);
atomic64_inc(&STp->stats->read_cnt);
- if (req->errors) {
+ if (scsi_req(req)->result) {
atomic64_add(atomic_read(&STp->stats->last_read_size)
- STp->buffer->cmdstat.residual,
&STp->stats->read_byte_cnt);
@@ -518,7 +518,7 @@
struct scsi_tape *STp = SRpnt->stp;
struct bio *tmp;
- STp->buffer->cmdstat.midlevel_result = SRpnt->result = req->errors;
+ STp->buffer->cmdstat.midlevel_result = SRpnt->result = rq->result;
STp->buffer->cmdstat.residual = rq->resid_len;
st_do_stats(STp, req);
@@ -579,7 +579,7 @@
memset(rq->cmd, 0, BLK_MAX_CDB);
memcpy(rq->cmd, cmd, rq->cmd_len);
req->timeout = timeout;
- req->retries = retries;
+ rq->retries = retries;
req->end_io_data = SRpnt;
blk_execute_rq_nowait(req->q, NULL, req, 1, st_scsi_execute_end);
diff --git a/drivers/soc/fsl/qbman/qman.c b/drivers/soc/fsl/qbman/qman.c
index 6f509f6..3d891db 100644
--- a/drivers/soc/fsl/qbman/qman.c
+++ b/drivers/soc/fsl/qbman/qman.c
@@ -2019,8 +2019,7 @@
return ret;
}
-static int qman_query_fq_np(struct qman_fq *fq,
- struct qm_mcr_queryfq_np *np)
+int qman_query_fq_np(struct qman_fq *fq, struct qm_mcr_queryfq_np *np)
{
union qm_mc_command *mcc;
union qm_mc_result *mcr;
@@ -2046,6 +2045,7 @@
put_affine_portal();
return ret;
}
+EXPORT_SYMBOL(qman_query_fq_np);
static int qman_query_cgr(struct qman_cgr *cgr,
struct qm_mcr_querycgr *cgrd)
diff --git a/drivers/soc/fsl/qbman/qman_ccsr.c b/drivers/soc/fsl/qbman/qman_ccsr.c
index f4e6e70..90bc40c 100644
--- a/drivers/soc/fsl/qbman/qman_ccsr.c
+++ b/drivers/soc/fsl/qbman/qman_ccsr.c
@@ -34,6 +34,8 @@
EXPORT_SYMBOL(qman_ip_rev);
u16 qm_channel_pool1 = QMAN_CHANNEL_POOL1;
EXPORT_SYMBOL(qm_channel_pool1);
+u16 qm_channel_caam = QMAN_CHANNEL_CAAM;
+EXPORT_SYMBOL(qm_channel_caam);
/* Register offsets */
#define REG_QCSP_LIO_CFG(n) (0x0000 + ((n) * 0x10))
@@ -720,8 +722,10 @@
return -ENODEV;
}
- if ((qman_ip_rev & 0xff00) >= QMAN_REV30)
+ if ((qman_ip_rev & 0xff00) >= QMAN_REV30) {
qm_channel_pool1 = QMAN_CHANNEL_POOL1_REV3;
+ qm_channel_caam = QMAN_CHANNEL_CAAM_REV3;
+ }
ret = zero_priv_mem(dev, node, fqd_a, fqd_sz);
WARN_ON(ret);
diff --git a/drivers/soc/fsl/qbman/qman_priv.h b/drivers/soc/fsl/qbman/qman_priv.h
index 53685b5..22725bd 100644
--- a/drivers/soc/fsl/qbman/qman_priv.h
+++ b/drivers/soc/fsl/qbman/qman_priv.h
@@ -89,67 +89,6 @@
return ((u64)q->a_bcnt_hi << 32) | be32_to_cpu(q->a_bcnt_lo);
}
-/* "Query FQ Non-Programmable Fields" */
-
-struct qm_mcr_queryfq_np {
- u8 verb;
- u8 result;
- u8 __reserved1;
- u8 state; /* QM_MCR_NP_STATE_*** */
- u32 fqd_link; /* 24-bit, _res2[24-31] */
- u16 odp_seq; /* 14-bit, _res3[14-15] */
- u16 orp_nesn; /* 14-bit, _res4[14-15] */
- u16 orp_ea_hseq; /* 15-bit, _res5[15] */
- u16 orp_ea_tseq; /* 15-bit, _res6[15] */
- u32 orp_ea_hptr; /* 24-bit, _res7[24-31] */
- u32 orp_ea_tptr; /* 24-bit, _res8[24-31] */
- u32 pfdr_hptr; /* 24-bit, _res9[24-31] */
- u32 pfdr_tptr; /* 24-bit, _res10[24-31] */
- u8 __reserved2[5];
- u8 is; /* 1-bit, _res12[1-7] */
- u16 ics_surp;
- u32 byte_cnt;
- u32 frm_cnt; /* 24-bit, _res13[24-31] */
- u32 __reserved3;
- u16 ra1_sfdr; /* QM_MCR_NP_RA1_*** */
- u16 ra2_sfdr; /* QM_MCR_NP_RA2_*** */
- u16 __reserved4;
- u16 od1_sfdr; /* QM_MCR_NP_OD1_*** */
- u16 od2_sfdr; /* QM_MCR_NP_OD2_*** */
- u16 od3_sfdr; /* QM_MCR_NP_OD3_*** */
-} __packed;
-
-#define QM_MCR_NP_STATE_FE 0x10
-#define QM_MCR_NP_STATE_R 0x08
-#define QM_MCR_NP_STATE_MASK 0x07 /* Reads FQD::STATE; */
-#define QM_MCR_NP_STATE_OOS 0x00
-#define QM_MCR_NP_STATE_RETIRED 0x01
-#define QM_MCR_NP_STATE_TEN_SCHED 0x02
-#define QM_MCR_NP_STATE_TRU_SCHED 0x03
-#define QM_MCR_NP_STATE_PARKED 0x04
-#define QM_MCR_NP_STATE_ACTIVE 0x05
-#define QM_MCR_NP_PTR_MASK 0x07ff /* for RA[12] & OD[123] */
-#define QM_MCR_NP_RA1_NRA(v) (((v) >> 14) & 0x3) /* FQD::NRA */
-#define QM_MCR_NP_RA2_IT(v) (((v) >> 14) & 0x1) /* FQD::IT */
-#define QM_MCR_NP_OD1_NOD(v) (((v) >> 14) & 0x3) /* FQD::NOD */
-#define QM_MCR_NP_OD3_NPC(v) (((v) >> 14) & 0x3) /* FQD::NPC */
-
-enum qm_mcr_queryfq_np_masks {
- qm_mcr_fqd_link_mask = BIT(24)-1,
- qm_mcr_odp_seq_mask = BIT(14)-1,
- qm_mcr_orp_nesn_mask = BIT(14)-1,
- qm_mcr_orp_ea_hseq_mask = BIT(15)-1,
- qm_mcr_orp_ea_tseq_mask = BIT(15)-1,
- qm_mcr_orp_ea_hptr_mask = BIT(24)-1,
- qm_mcr_orp_ea_tptr_mask = BIT(24)-1,
- qm_mcr_pfdr_hptr_mask = BIT(24)-1,
- qm_mcr_pfdr_tptr_mask = BIT(24)-1,
- qm_mcr_is_mask = BIT(1)-1,
- qm_mcr_frm_cnt_mask = BIT(24)-1,
-};
-#define qm_mcr_np_get(np, field) \
- ((np)->field & (qm_mcr_##field##_mask))
-
/* Congestion Groups */
/*
@@ -271,42 +210,6 @@
*/
int qman_query_fq(struct qman_fq *fq, struct qm_fqd *fqd);
-/*
- * For qman_volatile_dequeue(); Choose one PRECEDENCE. EXACT is optional. Use
- * NUMFRAMES(n) (6-bit) or NUMFRAMES_TILLEMPTY to fill in the frame-count. Use
- * FQID(n) to fill in the frame queue ID.
- */
-#define QM_VDQCR_PRECEDENCE_VDQCR 0x0
-#define QM_VDQCR_PRECEDENCE_SDQCR 0x80000000
-#define QM_VDQCR_EXACT 0x40000000
-#define QM_VDQCR_NUMFRAMES_MASK 0x3f000000
-#define QM_VDQCR_NUMFRAMES_SET(n) (((n) & 0x3f) << 24)
-#define QM_VDQCR_NUMFRAMES_GET(n) (((n) >> 24) & 0x3f)
-#define QM_VDQCR_NUMFRAMES_TILLEMPTY QM_VDQCR_NUMFRAMES_SET(0)
-
-#define QMAN_VOLATILE_FLAG_WAIT 0x00000001 /* wait if VDQCR is in use */
-#define QMAN_VOLATILE_FLAG_WAIT_INT 0x00000002 /* if wait, interruptible? */
-#define QMAN_VOLATILE_FLAG_FINISH 0x00000004 /* wait till VDQCR completes */
-
-/*
- * qman_volatile_dequeue - Issue a volatile dequeue command
- * @fq: the frame queue object to dequeue from
- * @flags: a bit-mask of QMAN_VOLATILE_FLAG_*** options
- * @vdqcr: bit mask of QM_VDQCR_*** options, as per qm_dqrr_vdqcr_set()
- *
- * Attempts to lock access to the portal's VDQCR volatile dequeue functionality.
- * The function will block and sleep if QMAN_VOLATILE_FLAG_WAIT is specified and
- * the VDQCR is already in use, otherwise returns non-zero for failure. If
- * QMAN_VOLATILE_FLAG_FINISH is specified, the function will only return once
- * the VDQCR command has finished executing (ie. once the callback for the last
- * DQRR entry resulting from the VDQCR command has been called). If not using
- * the FINISH flag, completion can be determined either by detecting the
- * presence of the QM_DQRR_STAT_UNSCHEDULED and QM_DQRR_STAT_DQCR_EXPIRED bits
- * in the "stat" parameter passed to the FQ's dequeue callback, or by waiting
- * for the QMAN_FQ_STATE_VDQCR bit to disappear.
- */
-int qman_volatile_dequeue(struct qman_fq *fq, u32 flags, u32 vdqcr);
-
int qman_alloc_fq_table(u32 num_fqids);
/* QMan s/w corenet portal, low-level i/face */
diff --git a/drivers/soc/qcom/Kconfig b/drivers/soc/qcom/Kconfig
index 78b1bb7..9fca977 100644
--- a/drivers/soc/qcom/Kconfig
+++ b/drivers/soc/qcom/Kconfig
@@ -33,17 +33,10 @@
The driver provides an interface to items in a heap shared among all
processors in a Qualcomm platform.
-config QCOM_SMD
- tristate "Qualcomm Shared Memory Driver (SMD)"
- depends on QCOM_SMEM
- help
- Say y here to enable support for the Qualcomm Shared Memory Driver
- providing communication channels to remote processors in Qualcomm
- platforms.
-
config QCOM_SMD_RPM
tristate "Qualcomm Resource Power Manager (RPM) over SMD"
- depends on QCOM_SMD && OF
+ depends on ARCH_QCOM
+ depends on RPMSG && OF
help
If you say yes to this option, support will be included for the
Resource Power Manager system found in the Qualcomm 8974 based
@@ -76,7 +69,8 @@
config QCOM_WCNSS_CTRL
tristate "Qualcomm WCNSS control driver"
- depends on QCOM_SMD
+ depends on ARCH_QCOM
+ depends on RPMSG
help
Client driver for the WCNSS_CTRL SMD channel, used to download nv
firmware to a newly booted WCNSS chip.
diff --git a/drivers/soc/qcom/Makefile b/drivers/soc/qcom/Makefile
index 1f30260..414f0de 100644
--- a/drivers/soc/qcom/Makefile
+++ b/drivers/soc/qcom/Makefile
@@ -1,7 +1,6 @@
obj-$(CONFIG_QCOM_GSBI) += qcom_gsbi.o
obj-$(CONFIG_QCOM_MDT_LOADER) += mdt_loader.o
obj-$(CONFIG_QCOM_PM) += spm.o
-obj-$(CONFIG_QCOM_SMD) += smd.o
obj-$(CONFIG_QCOM_SMD_RPM) += smd-rpm.o
obj-$(CONFIG_QCOM_SMEM) += smem.o
obj-$(CONFIG_QCOM_SMEM_STATE) += smem_state.o
diff --git a/drivers/soc/qcom/smd-rpm.c b/drivers/soc/qcom/smd-rpm.c
index 6609d7e..c234675 100644
--- a/drivers/soc/qcom/smd-rpm.c
+++ b/drivers/soc/qcom/smd-rpm.c
@@ -19,7 +19,7 @@
#include <linux/interrupt.h>
#include <linux/slab.h>
-#include <linux/soc/qcom/smd.h>
+#include <linux/rpmsg.h>
#include <linux/soc/qcom/smd-rpm.h>
#define RPM_REQUEST_TIMEOUT (5 * HZ)
@@ -32,7 +32,7 @@
* @ack_status: result of the rpm request
*/
struct qcom_smd_rpm {
- struct qcom_smd_channel *rpm_channel;
+ struct rpmsg_endpoint *rpm_channel;
struct device *dev;
struct completion ack;
@@ -133,7 +133,7 @@
pkt->req.data_len = cpu_to_le32(count);
memcpy(pkt->payload, buf, count);
- ret = qcom_smd_send(rpm->rpm_channel, pkt, size);
+ ret = rpmsg_send(rpm->rpm_channel, pkt, size);
if (ret)
goto out;
@@ -150,14 +150,16 @@
}
EXPORT_SYMBOL(qcom_rpm_smd_write);
-static int qcom_smd_rpm_callback(struct qcom_smd_channel *channel,
- const void *data,
- size_t count)
+static int qcom_smd_rpm_callback(struct rpmsg_device *rpdev,
+ void *data,
+ int count,
+ void *priv,
+ u32 addr)
{
const struct qcom_rpm_header *hdr = data;
size_t hdr_length = le32_to_cpu(hdr->length);
const struct qcom_rpm_message *msg;
- struct qcom_smd_rpm *rpm = qcom_smd_get_drvdata(channel);
+ struct qcom_smd_rpm *rpm = dev_get_drvdata(&rpdev->dev);
const u8 *buf = data + sizeof(struct qcom_rpm_header);
const u8 *end = buf + hdr_length;
char msgbuf[32];
@@ -196,59 +198,57 @@
return 0;
}
-static int qcom_smd_rpm_probe(struct qcom_smd_device *sdev)
+static int qcom_smd_rpm_probe(struct rpmsg_device *rpdev)
{
struct qcom_smd_rpm *rpm;
- rpm = devm_kzalloc(&sdev->dev, sizeof(*rpm), GFP_KERNEL);
+ rpm = devm_kzalloc(&rpdev->dev, sizeof(*rpm), GFP_KERNEL);
if (!rpm)
return -ENOMEM;
mutex_init(&rpm->lock);
init_completion(&rpm->ack);
- rpm->dev = &sdev->dev;
- rpm->rpm_channel = sdev->channel;
- qcom_smd_set_drvdata(sdev->channel, rpm);
+ rpm->dev = &rpdev->dev;
+ rpm->rpm_channel = rpdev->ept;
+ dev_set_drvdata(&rpdev->dev, rpm);
- dev_set_drvdata(&sdev->dev, rpm);
-
- return of_platform_populate(sdev->dev.of_node, NULL, NULL, &sdev->dev);
+ return of_platform_populate(rpdev->dev.of_node, NULL, NULL, &rpdev->dev);
}
-static void qcom_smd_rpm_remove(struct qcom_smd_device *sdev)
+static void qcom_smd_rpm_remove(struct rpmsg_device *rpdev)
{
- of_platform_depopulate(&sdev->dev);
+ of_platform_depopulate(&rpdev->dev);
}
static const struct of_device_id qcom_smd_rpm_of_match[] = {
{ .compatible = "qcom,rpm-apq8084" },
{ .compatible = "qcom,rpm-msm8916" },
{ .compatible = "qcom,rpm-msm8974" },
+ { .compatible = "qcom,rpm-msm8996" },
{}
};
MODULE_DEVICE_TABLE(of, qcom_smd_rpm_of_match);
-static struct qcom_smd_driver qcom_smd_rpm_driver = {
+static struct rpmsg_driver qcom_smd_rpm_driver = {
.probe = qcom_smd_rpm_probe,
.remove = qcom_smd_rpm_remove,
.callback = qcom_smd_rpm_callback,
- .driver = {
+ .drv = {
.name = "qcom_smd_rpm",
- .owner = THIS_MODULE,
.of_match_table = qcom_smd_rpm_of_match,
},
};
static int __init qcom_smd_rpm_init(void)
{
- return qcom_smd_driver_register(&qcom_smd_rpm_driver);
+ return register_rpmsg_driver(&qcom_smd_rpm_driver);
}
arch_initcall(qcom_smd_rpm_init);
static void __exit qcom_smd_rpm_exit(void)
{
- qcom_smd_driver_unregister(&qcom_smd_rpm_driver);
+ unregister_rpmsg_driver(&qcom_smd_rpm_driver);
}
module_exit(qcom_smd_rpm_exit);
diff --git a/drivers/soc/qcom/smd.c b/drivers/soc/qcom/smd.c
deleted file mode 100644
index 322034a..0000000
--- a/drivers/soc/qcom/smd.c
+++ /dev/null
@@ -1,1560 +0,0 @@
-/*
- * Copyright (c) 2015, Sony Mobile Communications AB.
- * Copyright (c) 2012-2013, The Linux Foundation. All rights reserved.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 and
- * only version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- */
-
-#include <linux/interrupt.h>
-#include <linux/io.h>
-#include <linux/mfd/syscon.h>
-#include <linux/module.h>
-#include <linux/of_irq.h>
-#include <linux/of_platform.h>
-#include <linux/platform_device.h>
-#include <linux/regmap.h>
-#include <linux/sched.h>
-#include <linux/slab.h>
-#include <linux/soc/qcom/smd.h>
-#include <linux/soc/qcom/smem.h>
-#include <linux/wait.h>
-
-/*
- * The Qualcomm Shared Memory communication solution provides point-to-point
- * channels for clients to send and receive streaming or packet based data.
- *
- * Each channel consists of a control item (channel info) and a ring buffer
- * pair. The channel info carry information related to channel state, flow
- * control and the offsets within the ring buffer.
- *
- * All allocated channels are listed in an allocation table, identifying the
- * pair of items by name, type and remote processor.
- *
- * Upon creating a new channel the remote processor allocates channel info and
- * ring buffer items from the smem heap and populate the allocation table. An
- * interrupt is sent to the other end of the channel and a scan for new
- * channels should be done. A channel never goes away, it will only change
- * state.
- *
- * The remote processor signals it intent for bring up the communication
- * channel by setting the state of its end of the channel to "opening" and
- * sends out an interrupt. We detect this change and register a smd device to
- * consume the channel. Upon finding a consumer we finish the handshake and the
- * channel is up.
- *
- * Upon closing a channel, the remote processor will update the state of its
- * end of the channel and signal us, we will then unregister any attached
- * device and close our end of the channel.
- *
- * Devices attached to a channel can use the qcom_smd_send function to push
- * data to the channel, this is done by copying the data into the tx ring
- * buffer, updating the pointers in the channel info and signaling the remote
- * processor.
- *
- * The remote processor does the equivalent when it transfer data and upon
- * receiving the interrupt we check the channel info for new data and delivers
- * this to the attached device. If the device is not ready to receive the data
- * we leave it in the ring buffer for now.
- */
-
-struct smd_channel_info;
-struct smd_channel_info_pair;
-struct smd_channel_info_word;
-struct smd_channel_info_word_pair;
-
-#define SMD_ALLOC_TBL_COUNT 2
-#define SMD_ALLOC_TBL_SIZE 64
-
-/*
- * This lists the various smem heap items relevant for the allocation table and
- * smd channel entries.
- */
-static const struct {
- unsigned alloc_tbl_id;
- unsigned info_base_id;
- unsigned fifo_base_id;
-} smem_items[SMD_ALLOC_TBL_COUNT] = {
- {
- .alloc_tbl_id = 13,
- .info_base_id = 14,
- .fifo_base_id = 338
- },
- {
- .alloc_tbl_id = 266,
- .info_base_id = 138,
- .fifo_base_id = 202,
- },
-};
-
-/**
- * struct qcom_smd_edge - representing a remote processor
- * @dev: device for this edge
- * @of_node: of_node handle for information related to this edge
- * @edge_id: identifier of this edge
- * @remote_pid: identifier of remote processor
- * @irq: interrupt for signals on this edge
- * @ipc_regmap: regmap handle holding the outgoing ipc register
- * @ipc_offset: offset within @ipc_regmap of the register for ipc
- * @ipc_bit: bit in the register at @ipc_offset of @ipc_regmap
- * @channels: list of all channels detected on this edge
- * @channels_lock: guard for modifications of @channels
- * @allocated: array of bitmaps representing already allocated channels
- * @smem_available: last available amount of smem triggering a channel scan
- * @scan_work: work item for discovering new channels
- * @state_work: work item for edge state changes
- */
-struct qcom_smd_edge {
- struct device dev;
-
- struct device_node *of_node;
- unsigned edge_id;
- unsigned remote_pid;
-
- int irq;
-
- struct regmap *ipc_regmap;
- int ipc_offset;
- int ipc_bit;
-
- struct list_head channels;
- spinlock_t channels_lock;
-
- DECLARE_BITMAP(allocated[SMD_ALLOC_TBL_COUNT], SMD_ALLOC_TBL_SIZE);
-
- unsigned smem_available;
-
- wait_queue_head_t new_channel_event;
-
- struct work_struct scan_work;
- struct work_struct state_work;
-};
-
-#define to_smd_edge(d) container_of(d, struct qcom_smd_edge, dev)
-
-/*
- * SMD channel states.
- */
-enum smd_channel_state {
- SMD_CHANNEL_CLOSED,
- SMD_CHANNEL_OPENING,
- SMD_CHANNEL_OPENED,
- SMD_CHANNEL_FLUSHING,
- SMD_CHANNEL_CLOSING,
- SMD_CHANNEL_RESET,
- SMD_CHANNEL_RESET_OPENING
-};
-
-/**
- * struct qcom_smd_channel - smd channel struct
- * @edge: qcom_smd_edge this channel is living on
- * @qsdev: reference to a associated smd client device
- * @name: name of the channel
- * @state: local state of the channel
- * @remote_state: remote state of the channel
- * @info: byte aligned outgoing/incoming channel info
- * @info_word: word aligned outgoing/incoming channel info
- * @tx_lock: lock to make writes to the channel mutually exclusive
- * @fblockread_event: wakeup event tied to tx fBLOCKREADINTR
- * @tx_fifo: pointer to the outgoing ring buffer
- * @rx_fifo: pointer to the incoming ring buffer
- * @fifo_size: size of each ring buffer
- * @bounce_buffer: bounce buffer for reading wrapped packets
- * @cb: callback function registered for this channel
- * @recv_lock: guard for rx info modifications and cb pointer
- * @pkt_size: size of the currently handled packet
- * @list: lite entry for @channels in qcom_smd_edge
- */
-struct qcom_smd_channel {
- struct qcom_smd_edge *edge;
-
- struct qcom_smd_device *qsdev;
-
- char *name;
- enum smd_channel_state state;
- enum smd_channel_state remote_state;
-
- struct smd_channel_info_pair *info;
- struct smd_channel_info_word_pair *info_word;
-
- struct mutex tx_lock;
- wait_queue_head_t fblockread_event;
-
- void *tx_fifo;
- void *rx_fifo;
- int fifo_size;
-
- void *bounce_buffer;
- qcom_smd_cb_t cb;
-
- spinlock_t recv_lock;
-
- int pkt_size;
-
- void *drvdata;
-
- struct list_head list;
-};
-
-/*
- * Format of the smd_info smem items, for byte aligned channels.
- */
-struct smd_channel_info {
- __le32 state;
- u8 fDSR;
- u8 fCTS;
- u8 fCD;
- u8 fRI;
- u8 fHEAD;
- u8 fTAIL;
- u8 fSTATE;
- u8 fBLOCKREADINTR;
- __le32 tail;
- __le32 head;
-};
-
-struct smd_channel_info_pair {
- struct smd_channel_info tx;
- struct smd_channel_info rx;
-};
-
-/*
- * Format of the smd_info smem items, for word aligned channels.
- */
-struct smd_channel_info_word {
- __le32 state;
- __le32 fDSR;
- __le32 fCTS;
- __le32 fCD;
- __le32 fRI;
- __le32 fHEAD;
- __le32 fTAIL;
- __le32 fSTATE;
- __le32 fBLOCKREADINTR;
- __le32 tail;
- __le32 head;
-};
-
-struct smd_channel_info_word_pair {
- struct smd_channel_info_word tx;
- struct smd_channel_info_word rx;
-};
-
-#define GET_RX_CHANNEL_FLAG(channel, param) \
- ({ \
- BUILD_BUG_ON(sizeof(channel->info->rx.param) != sizeof(u8)); \
- channel->info_word ? \
- le32_to_cpu(channel->info_word->rx.param) : \
- channel->info->rx.param; \
- })
-
-#define GET_RX_CHANNEL_INFO(channel, param) \
- ({ \
- BUILD_BUG_ON(sizeof(channel->info->rx.param) != sizeof(u32)); \
- le32_to_cpu(channel->info_word ? \
- channel->info_word->rx.param : \
- channel->info->rx.param); \
- })
-
-#define SET_RX_CHANNEL_FLAG(channel, param, value) \
- ({ \
- BUILD_BUG_ON(sizeof(channel->info->rx.param) != sizeof(u8)); \
- if (channel->info_word) \
- channel->info_word->rx.param = cpu_to_le32(value); \
- else \
- channel->info->rx.param = value; \
- })
-
-#define SET_RX_CHANNEL_INFO(channel, param, value) \
- ({ \
- BUILD_BUG_ON(sizeof(channel->info->rx.param) != sizeof(u32)); \
- if (channel->info_word) \
- channel->info_word->rx.param = cpu_to_le32(value); \
- else \
- channel->info->rx.param = cpu_to_le32(value); \
- })
-
-#define GET_TX_CHANNEL_FLAG(channel, param) \
- ({ \
- BUILD_BUG_ON(sizeof(channel->info->tx.param) != sizeof(u8)); \
- channel->info_word ? \
- le32_to_cpu(channel->info_word->tx.param) : \
- channel->info->tx.param; \
- })
-
-#define GET_TX_CHANNEL_INFO(channel, param) \
- ({ \
- BUILD_BUG_ON(sizeof(channel->info->tx.param) != sizeof(u32)); \
- le32_to_cpu(channel->info_word ? \
- channel->info_word->tx.param : \
- channel->info->tx.param); \
- })
-
-#define SET_TX_CHANNEL_FLAG(channel, param, value) \
- ({ \
- BUILD_BUG_ON(sizeof(channel->info->tx.param) != sizeof(u8)); \
- if (channel->info_word) \
- channel->info_word->tx.param = cpu_to_le32(value); \
- else \
- channel->info->tx.param = value; \
- })
-
-#define SET_TX_CHANNEL_INFO(channel, param, value) \
- ({ \
- BUILD_BUG_ON(sizeof(channel->info->tx.param) != sizeof(u32)); \
- if (channel->info_word) \
- channel->info_word->tx.param = cpu_to_le32(value); \
- else \
- channel->info->tx.param = cpu_to_le32(value); \
- })
-
-/**
- * struct qcom_smd_alloc_entry - channel allocation entry
- * @name: channel name
- * @cid: channel index
- * @flags: channel flags and edge id
- * @ref_count: reference count of the channel
- */
-struct qcom_smd_alloc_entry {
- u8 name[20];
- __le32 cid;
- __le32 flags;
- __le32 ref_count;
-} __packed;
-
-#define SMD_CHANNEL_FLAGS_EDGE_MASK 0xff
-#define SMD_CHANNEL_FLAGS_STREAM BIT(8)
-#define SMD_CHANNEL_FLAGS_PACKET BIT(9)
-
-/*
- * Each smd packet contains a 20 byte header, with the first 4 being the length
- * of the packet.
- */
-#define SMD_PACKET_HEADER_LEN 20
-
-/*
- * Signal the remote processor associated with 'channel'.
- */
-static void qcom_smd_signal_channel(struct qcom_smd_channel *channel)
-{
- struct qcom_smd_edge *edge = channel->edge;
-
- regmap_write(edge->ipc_regmap, edge->ipc_offset, BIT(edge->ipc_bit));
-}
-
-/*
- * Initialize the tx channel info
- */
-static void qcom_smd_channel_reset(struct qcom_smd_channel *channel)
-{
- SET_TX_CHANNEL_INFO(channel, state, SMD_CHANNEL_CLOSED);
- SET_TX_CHANNEL_FLAG(channel, fDSR, 0);
- SET_TX_CHANNEL_FLAG(channel, fCTS, 0);
- SET_TX_CHANNEL_FLAG(channel, fCD, 0);
- SET_TX_CHANNEL_FLAG(channel, fRI, 0);
- SET_TX_CHANNEL_FLAG(channel, fHEAD, 0);
- SET_TX_CHANNEL_FLAG(channel, fTAIL, 0);
- SET_TX_CHANNEL_FLAG(channel, fSTATE, 1);
- SET_TX_CHANNEL_FLAG(channel, fBLOCKREADINTR, 1);
- SET_TX_CHANNEL_INFO(channel, head, 0);
- SET_RX_CHANNEL_INFO(channel, tail, 0);
-
- qcom_smd_signal_channel(channel);
-
- channel->state = SMD_CHANNEL_CLOSED;
- channel->pkt_size = 0;
-}
-
-/*
- * Set the callback for a channel, with appropriate locking
- */
-static void qcom_smd_channel_set_callback(struct qcom_smd_channel *channel,
- qcom_smd_cb_t cb)
-{
- unsigned long flags;
-
- spin_lock_irqsave(&channel->recv_lock, flags);
- channel->cb = cb;
- spin_unlock_irqrestore(&channel->recv_lock, flags);
-};
-
-/*
- * Calculate the amount of data available in the rx fifo
- */
-static size_t qcom_smd_channel_get_rx_avail(struct qcom_smd_channel *channel)
-{
- unsigned head;
- unsigned tail;
-
- head = GET_RX_CHANNEL_INFO(channel, head);
- tail = GET_RX_CHANNEL_INFO(channel, tail);
-
- return (head - tail) & (channel->fifo_size - 1);
-}
-
-/*
- * Set tx channel state and inform the remote processor
- */
-static void qcom_smd_channel_set_state(struct qcom_smd_channel *channel,
- int state)
-{
- struct qcom_smd_edge *edge = channel->edge;
- bool is_open = state == SMD_CHANNEL_OPENED;
-
- if (channel->state == state)
- return;
-
- dev_dbg(&edge->dev, "set_state(%s, %d)\n", channel->name, state);
-
- SET_TX_CHANNEL_FLAG(channel, fDSR, is_open);
- SET_TX_CHANNEL_FLAG(channel, fCTS, is_open);
- SET_TX_CHANNEL_FLAG(channel, fCD, is_open);
-
- SET_TX_CHANNEL_INFO(channel, state, state);
- SET_TX_CHANNEL_FLAG(channel, fSTATE, 1);
-
- channel->state = state;
- qcom_smd_signal_channel(channel);
-}
-
-/*
- * Copy count bytes of data using 32bit accesses, if that's required.
- */
-static void smd_copy_to_fifo(void __iomem *dst,
- const void *src,
- size_t count,
- bool word_aligned)
-{
- if (word_aligned) {
- __iowrite32_copy(dst, src, count / sizeof(u32));
- } else {
- memcpy_toio(dst, src, count);
- }
-}
-
-/*
- * Copy count bytes of data using 32bit accesses, if that is required.
- */
-static void smd_copy_from_fifo(void *dst,
- const void __iomem *src,
- size_t count,
- bool word_aligned)
-{
- if (word_aligned) {
- __ioread32_copy(dst, src, count / sizeof(u32));
- } else {
- memcpy_fromio(dst, src, count);
- }
-}
-
-/*
- * Read count bytes of data from the rx fifo into buf, but don't advance the
- * tail.
- */
-static size_t qcom_smd_channel_peek(struct qcom_smd_channel *channel,
- void *buf, size_t count)
-{
- bool word_aligned;
- unsigned tail;
- size_t len;
-
- word_aligned = channel->info_word;
- tail = GET_RX_CHANNEL_INFO(channel, tail);
-
- len = min_t(size_t, count, channel->fifo_size - tail);
- if (len) {
- smd_copy_from_fifo(buf,
- channel->rx_fifo + tail,
- len,
- word_aligned);
- }
-
- if (len != count) {
- smd_copy_from_fifo(buf + len,
- channel->rx_fifo,
- count - len,
- word_aligned);
- }
-
- return count;
-}
-
-/*
- * Advance the rx tail by count bytes.
- */
-static void qcom_smd_channel_advance(struct qcom_smd_channel *channel,
- size_t count)
-{
- unsigned tail;
-
- tail = GET_RX_CHANNEL_INFO(channel, tail);
- tail += count;
- tail &= (channel->fifo_size - 1);
- SET_RX_CHANNEL_INFO(channel, tail, tail);
-}
-
-/*
- * Read out a single packet from the rx fifo and deliver it to the device
- */
-static int qcom_smd_channel_recv_single(struct qcom_smd_channel *channel)
-{
- unsigned tail;
- size_t len;
- void *ptr;
- int ret;
-
- if (!channel->cb)
- return 0;
-
- tail = GET_RX_CHANNEL_INFO(channel, tail);
-
- /* Use bounce buffer if the data wraps */
- if (tail + channel->pkt_size >= channel->fifo_size) {
- ptr = channel->bounce_buffer;
- len = qcom_smd_channel_peek(channel, ptr, channel->pkt_size);
- } else {
- ptr = channel->rx_fifo + tail;
- len = channel->pkt_size;
- }
-
- ret = channel->cb(channel, ptr, len);
- if (ret < 0)
- return ret;
-
- /* Only forward the tail if the client consumed the data */
- qcom_smd_channel_advance(channel, len);
-
- channel->pkt_size = 0;
-
- return 0;
-}
-
-/*
- * Per channel interrupt handling
- */
-static bool qcom_smd_channel_intr(struct qcom_smd_channel *channel)
-{
- bool need_state_scan = false;
- int remote_state;
- __le32 pktlen;
- int avail;
- int ret;
-
- /* Handle state changes */
- remote_state = GET_RX_CHANNEL_INFO(channel, state);
- if (remote_state != channel->remote_state) {
- channel->remote_state = remote_state;
- need_state_scan = true;
- }
- /* Indicate that we have seen any state change */
- SET_RX_CHANNEL_FLAG(channel, fSTATE, 0);
-
- /* Signal waiting qcom_smd_send() about the interrupt */
- if (!GET_TX_CHANNEL_FLAG(channel, fBLOCKREADINTR))
- wake_up_interruptible(&channel->fblockread_event);
-
- /* Don't consume any data until we've opened the channel */
- if (channel->state != SMD_CHANNEL_OPENED)
- goto out;
-
- /* Indicate that we've seen the new data */
- SET_RX_CHANNEL_FLAG(channel, fHEAD, 0);
-
- /* Consume data */
- for (;;) {
- avail = qcom_smd_channel_get_rx_avail(channel);
-
- if (!channel->pkt_size && avail >= SMD_PACKET_HEADER_LEN) {
- qcom_smd_channel_peek(channel, &pktlen, sizeof(pktlen));
- qcom_smd_channel_advance(channel, SMD_PACKET_HEADER_LEN);
- channel->pkt_size = le32_to_cpu(pktlen);
- } else if (channel->pkt_size && avail >= channel->pkt_size) {
- ret = qcom_smd_channel_recv_single(channel);
- if (ret)
- break;
- } else {
- break;
- }
- }
-
- /* Indicate that we have seen and updated tail */
- SET_RX_CHANNEL_FLAG(channel, fTAIL, 1);
-
- /* Signal the remote that we've consumed the data (if requested) */
- if (!GET_RX_CHANNEL_FLAG(channel, fBLOCKREADINTR)) {
- /* Ensure ordering of channel info updates */
- wmb();
-
- qcom_smd_signal_channel(channel);
- }
-
-out:
- return need_state_scan;
-}
-
-/*
- * The edge interrupts are triggered by the remote processor on state changes,
- * channel info updates or when new channels are created.
- */
-static irqreturn_t qcom_smd_edge_intr(int irq, void *data)
-{
- struct qcom_smd_edge *edge = data;
- struct qcom_smd_channel *channel;
- unsigned available;
- bool kick_scanner = false;
- bool kick_state = false;
-
- /*
- * Handle state changes or data on each of the channels on this edge
- */
- spin_lock(&edge->channels_lock);
- list_for_each_entry(channel, &edge->channels, list) {
- spin_lock(&channel->recv_lock);
- kick_state |= qcom_smd_channel_intr(channel);
- spin_unlock(&channel->recv_lock);
- }
- spin_unlock(&edge->channels_lock);
-
- /*
- * Creating a new channel requires allocating an smem entry, so we only
- * have to scan if the amount of available space in smem have changed
- * since last scan.
- */
- available = qcom_smem_get_free_space(edge->remote_pid);
- if (available != edge->smem_available) {
- edge->smem_available = available;
- kick_scanner = true;
- }
-
- if (kick_scanner)
- schedule_work(&edge->scan_work);
- if (kick_state)
- schedule_work(&edge->state_work);
-
- return IRQ_HANDLED;
-}
-
-/*
- * Delivers any outstanding packets in the rx fifo, can be used after probe of
- * the clients to deliver any packets that wasn't delivered before the client
- * was setup.
- */
-static void qcom_smd_channel_resume(struct qcom_smd_channel *channel)
-{
- unsigned long flags;
-
- spin_lock_irqsave(&channel->recv_lock, flags);
- qcom_smd_channel_intr(channel);
- spin_unlock_irqrestore(&channel->recv_lock, flags);
-}
-
-/*
- * Calculate how much space is available in the tx fifo.
- */
-static size_t qcom_smd_get_tx_avail(struct qcom_smd_channel *channel)
-{
- unsigned head;
- unsigned tail;
- unsigned mask = channel->fifo_size - 1;
-
- head = GET_TX_CHANNEL_INFO(channel, head);
- tail = GET_TX_CHANNEL_INFO(channel, tail);
-
- return mask - ((head - tail) & mask);
-}
-
-/*
- * Write count bytes of data into channel, possibly wrapping in the ring buffer
- */
-static int qcom_smd_write_fifo(struct qcom_smd_channel *channel,
- const void *data,
- size_t count)
-{
- bool word_aligned;
- unsigned head;
- size_t len;
-
- word_aligned = channel->info_word;
- head = GET_TX_CHANNEL_INFO(channel, head);
-
- len = min_t(size_t, count, channel->fifo_size - head);
- if (len) {
- smd_copy_to_fifo(channel->tx_fifo + head,
- data,
- len,
- word_aligned);
- }
-
- if (len != count) {
- smd_copy_to_fifo(channel->tx_fifo,
- data + len,
- count - len,
- word_aligned);
- }
-
- head += count;
- head &= (channel->fifo_size - 1);
- SET_TX_CHANNEL_INFO(channel, head, head);
-
- return count;
-}
-
-/**
- * qcom_smd_send - write data to smd channel
- * @channel: channel handle
- * @data: buffer of data to write
- * @len: number of bytes to write
- *
- * This is a blocking write of len bytes into the channel's tx ring buffer and
- * signal the remote end. It will sleep until there is enough space available
- * in the tx buffer, utilizing the fBLOCKREADINTR signaling mechanism to avoid
- * polling.
- */
-int qcom_smd_send(struct qcom_smd_channel *channel, const void *data, int len)
-{
- __le32 hdr[5] = { cpu_to_le32(len), };
- int tlen = sizeof(hdr) + len;
- int ret;
-
- /* Word aligned channels only accept word size aligned data */
- if (channel->info_word && len % 4)
- return -EINVAL;
-
- /* Reject packets that are too big */
- if (tlen >= channel->fifo_size)
- return -EINVAL;
-
- ret = mutex_lock_interruptible(&channel->tx_lock);
- if (ret)
- return ret;
-
- while (qcom_smd_get_tx_avail(channel) < tlen) {
- if (channel->state != SMD_CHANNEL_OPENED) {
- ret = -EPIPE;
- goto out;
- }
-
- SET_TX_CHANNEL_FLAG(channel, fBLOCKREADINTR, 0);
-
- ret = wait_event_interruptible(channel->fblockread_event,
- qcom_smd_get_tx_avail(channel) >= tlen ||
- channel->state != SMD_CHANNEL_OPENED);
- if (ret)
- goto out;
-
- SET_TX_CHANNEL_FLAG(channel, fBLOCKREADINTR, 1);
- }
-
- SET_TX_CHANNEL_FLAG(channel, fTAIL, 0);
-
- qcom_smd_write_fifo(channel, hdr, sizeof(hdr));
- qcom_smd_write_fifo(channel, data, len);
-
- SET_TX_CHANNEL_FLAG(channel, fHEAD, 1);
-
- /* Ensure ordering of channel info updates */
- wmb();
-
- qcom_smd_signal_channel(channel);
-
-out:
- mutex_unlock(&channel->tx_lock);
-
- return ret;
-}
-EXPORT_SYMBOL(qcom_smd_send);
-
-static struct qcom_smd_device *to_smd_device(struct device *dev)
-{
- return container_of(dev, struct qcom_smd_device, dev);
-}
-
-static struct qcom_smd_driver *to_smd_driver(struct device *dev)
-{
- struct qcom_smd_device *qsdev = to_smd_device(dev);
-
- return container_of(qsdev->dev.driver, struct qcom_smd_driver, driver);
-}
-
-static int qcom_smd_dev_match(struct device *dev, struct device_driver *drv)
-{
- struct qcom_smd_device *qsdev = to_smd_device(dev);
- struct qcom_smd_driver *qsdrv = container_of(drv, struct qcom_smd_driver, driver);
- const struct qcom_smd_id *match = qsdrv->smd_match_table;
- const char *name = qsdev->channel->name;
-
- if (match) {
- while (match->name[0]) {
- if (!strcmp(match->name, name))
- return 1;
- match++;
- }
- }
-
- return of_driver_match_device(dev, drv);
-}
-
-/*
- * Helper for opening a channel
- */
-static int qcom_smd_channel_open(struct qcom_smd_channel *channel,
- qcom_smd_cb_t cb)
-{
- size_t bb_size;
-
- /*
- * Packets are maximum 4k, but reduce if the fifo is smaller
- */
- bb_size = min(channel->fifo_size, SZ_4K);
- channel->bounce_buffer = kmalloc(bb_size, GFP_KERNEL);
- if (!channel->bounce_buffer)
- return -ENOMEM;
-
- qcom_smd_channel_set_callback(channel, cb);
- qcom_smd_channel_set_state(channel, SMD_CHANNEL_OPENING);
- qcom_smd_channel_set_state(channel, SMD_CHANNEL_OPENED);
-
- return 0;
-}
-
-/*
- * Helper for closing and resetting a channel
- */
-static void qcom_smd_channel_close(struct qcom_smd_channel *channel)
-{
- qcom_smd_channel_set_callback(channel, NULL);
-
- kfree(channel->bounce_buffer);
- channel->bounce_buffer = NULL;
-
- qcom_smd_channel_set_state(channel, SMD_CHANNEL_CLOSED);
- qcom_smd_channel_reset(channel);
-}
-
-/*
- * Probe the smd client.
- *
- * The remote side have indicated that it want the channel to be opened, so
- * complete the state handshake and probe our client driver.
- */
-static int qcom_smd_dev_probe(struct device *dev)
-{
- struct qcom_smd_device *qsdev = to_smd_device(dev);
- struct qcom_smd_driver *qsdrv = to_smd_driver(dev);
- struct qcom_smd_channel *channel = qsdev->channel;
- int ret;
-
- ret = qcom_smd_channel_open(channel, qsdrv->callback);
- if (ret)
- return ret;
-
- ret = qsdrv->probe(qsdev);
- if (ret)
- goto err;
-
- qcom_smd_channel_resume(channel);
-
- return 0;
-
-err:
- dev_err(&qsdev->dev, "probe failed\n");
-
- qcom_smd_channel_close(channel);
- return ret;
-}
-
-/*
- * Remove the smd client.
- *
- * The channel is going away, for some reason, so remove the smd client and
- * reset the channel state.
- */
-static int qcom_smd_dev_remove(struct device *dev)
-{
- struct qcom_smd_device *qsdev = to_smd_device(dev);
- struct qcom_smd_driver *qsdrv = to_smd_driver(dev);
- struct qcom_smd_channel *channel = qsdev->channel;
-
- qcom_smd_channel_set_state(channel, SMD_CHANNEL_CLOSING);
-
- /*
- * Make sure we don't race with the code receiving data.
- */
- qcom_smd_channel_set_callback(channel, NULL);
-
- /* Wake up any sleepers in qcom_smd_send() */
- wake_up_interruptible(&channel->fblockread_event);
-
- /*
- * We expect that the client might block in remove() waiting for any
- * outstanding calls to qcom_smd_send() to wake up and finish.
- */
- if (qsdrv->remove)
- qsdrv->remove(qsdev);
-
- /* The client is now gone, close the primary channel */
- qcom_smd_channel_close(channel);
- channel->qsdev = NULL;
-
- return 0;
-}
-
-static struct bus_type qcom_smd_bus = {
- .name = "qcom_smd",
- .match = qcom_smd_dev_match,
- .probe = qcom_smd_dev_probe,
- .remove = qcom_smd_dev_remove,
-};
-
-/*
- * Release function for the qcom_smd_device object.
- */
-static void qcom_smd_release_device(struct device *dev)
-{
- struct qcom_smd_device *qsdev = to_smd_device(dev);
-
- kfree(qsdev);
-}
-
-/*
- * Finds the device_node for the smd child interested in this channel.
- */
-static struct device_node *qcom_smd_match_channel(struct device_node *edge_node,
- const char *channel)
-{
- struct device_node *child;
- const char *name;
- const char *key;
- int ret;
-
- for_each_available_child_of_node(edge_node, child) {
- key = "qcom,smd-channels";
- ret = of_property_read_string(child, key, &name);
- if (ret)
- continue;
-
- if (strcmp(name, channel) == 0)
- return child;
- }
-
- return NULL;
-}
-
-/*
- * Create a smd client device for channel that is being opened.
- */
-static int qcom_smd_create_device(struct qcom_smd_channel *channel)
-{
- struct qcom_smd_device *qsdev;
- struct qcom_smd_edge *edge = channel->edge;
- struct device_node *node;
- int ret;
-
- if (channel->qsdev)
- return -EEXIST;
-
- dev_dbg(&edge->dev, "registering '%s'\n", channel->name);
-
- qsdev = kzalloc(sizeof(*qsdev), GFP_KERNEL);
- if (!qsdev)
- return -ENOMEM;
-
- node = qcom_smd_match_channel(edge->of_node, channel->name);
- dev_set_name(&qsdev->dev, "%s.%s",
- edge->of_node->name,
- node ? node->name : channel->name);
-
- qsdev->dev.parent = &edge->dev;
- qsdev->dev.bus = &qcom_smd_bus;
- qsdev->dev.release = qcom_smd_release_device;
- qsdev->dev.of_node = node;
-
- qsdev->channel = channel;
-
- channel->qsdev = qsdev;
-
- ret = device_register(&qsdev->dev);
- if (ret) {
- dev_err(&edge->dev, "device_register failed: %d\n", ret);
- put_device(&qsdev->dev);
- }
-
- return ret;
-}
-
-/*
- * Destroy a smd client device for a channel that's going away.
- */
-static void qcom_smd_destroy_device(struct qcom_smd_channel *channel)
-{
- struct device *dev;
-
- BUG_ON(!channel->qsdev);
-
- dev = &channel->qsdev->dev;
-
- device_unregister(dev);
- of_node_put(dev->of_node);
- put_device(dev);
-}
-
-/**
- * qcom_smd_driver_register - register a smd driver
- * @qsdrv: qcom_smd_driver struct
- */
-int qcom_smd_driver_register(struct qcom_smd_driver *qsdrv)
-{
- qsdrv->driver.bus = &qcom_smd_bus;
- return driver_register(&qsdrv->driver);
-}
-EXPORT_SYMBOL(qcom_smd_driver_register);
-
-void *qcom_smd_get_drvdata(struct qcom_smd_channel *channel)
-{
- return channel->drvdata;
-}
-EXPORT_SYMBOL(qcom_smd_get_drvdata);
-
-void qcom_smd_set_drvdata(struct qcom_smd_channel *channel, void *data)
-{
- channel->drvdata = data;
-}
-EXPORT_SYMBOL(qcom_smd_set_drvdata);
-
-/**
- * qcom_smd_driver_unregister - unregister a smd driver
- * @qsdrv: qcom_smd_driver struct
- */
-void qcom_smd_driver_unregister(struct qcom_smd_driver *qsdrv)
-{
- driver_unregister(&qsdrv->driver);
-}
-EXPORT_SYMBOL(qcom_smd_driver_unregister);
-
-static struct qcom_smd_channel *
-qcom_smd_find_channel(struct qcom_smd_edge *edge, const char *name)
-{
- struct qcom_smd_channel *channel;
- struct qcom_smd_channel *ret = NULL;
- unsigned long flags;
- unsigned state;
-
- spin_lock_irqsave(&edge->channels_lock, flags);
- list_for_each_entry(channel, &edge->channels, list) {
- if (strcmp(channel->name, name))
- continue;
-
- state = GET_RX_CHANNEL_INFO(channel, state);
- if (state != SMD_CHANNEL_OPENING &&
- state != SMD_CHANNEL_OPENED)
- continue;
-
- ret = channel;
- break;
- }
- spin_unlock_irqrestore(&edge->channels_lock, flags);
-
- return ret;
-}
-
-/**
- * qcom_smd_open_channel() - claim additional channels on the same edge
- * @sdev: smd_device handle
- * @name: channel name
- * @cb: callback method to use for incoming data
- *
- * Returns a channel handle on success, or -EPROBE_DEFER if the channel isn't
- * ready.
- *
- * Any channels returned must be closed with a call to qcom_smd_close_channel()
- */
-struct qcom_smd_channel *qcom_smd_open_channel(struct qcom_smd_channel *parent,
- const char *name,
- qcom_smd_cb_t cb)
-{
- struct qcom_smd_channel *channel;
- struct qcom_smd_device *sdev = parent->qsdev;
- struct qcom_smd_edge *edge = parent->edge;
- int ret;
-
- /* Wait up to HZ for the channel to appear */
- ret = wait_event_interruptible_timeout(edge->new_channel_event,
- (channel = qcom_smd_find_channel(edge, name)) != NULL,
- HZ);
- if (!ret)
- return ERR_PTR(-ETIMEDOUT);
-
- if (channel->state != SMD_CHANNEL_CLOSED) {
- dev_err(&sdev->dev, "channel %s is busy\n", channel->name);
- return ERR_PTR(-EBUSY);
- }
-
- channel->qsdev = sdev;
- ret = qcom_smd_channel_open(channel, cb);
- if (ret) {
- channel->qsdev = NULL;
- return ERR_PTR(ret);
- }
-
- return channel;
-}
-EXPORT_SYMBOL(qcom_smd_open_channel);
-
-/**
- * qcom_smd_close_channel() - close an additionally opened channel
- * @channel: channel handle, returned by qcom_smd_open_channel()
- */
-void qcom_smd_close_channel(struct qcom_smd_channel *channel)
-{
- qcom_smd_channel_close(channel);
- channel->qsdev = NULL;
-}
-EXPORT_SYMBOL(qcom_smd_close_channel);
-
-/*
- * Allocate the qcom_smd_channel object for a newly found smd channel,
- * retrieving and validating the smem items involved.
- */
-static struct qcom_smd_channel *qcom_smd_create_channel(struct qcom_smd_edge *edge,
- unsigned smem_info_item,
- unsigned smem_fifo_item,
- char *name)
-{
- struct qcom_smd_channel *channel;
- size_t fifo_size;
- size_t info_size;
- void *fifo_base;
- void *info;
- int ret;
-
- channel = devm_kzalloc(&edge->dev, sizeof(*channel), GFP_KERNEL);
- if (!channel)
- return ERR_PTR(-ENOMEM);
-
- channel->edge = edge;
- channel->name = devm_kstrdup(&edge->dev, name, GFP_KERNEL);
- if (!channel->name)
- return ERR_PTR(-ENOMEM);
-
- mutex_init(&channel->tx_lock);
- spin_lock_init(&channel->recv_lock);
- init_waitqueue_head(&channel->fblockread_event);
-
- info = qcom_smem_get(edge->remote_pid, smem_info_item, &info_size);
- if (IS_ERR(info)) {
- ret = PTR_ERR(info);
- goto free_name_and_channel;
- }
-
- /*
- * Use the size of the item to figure out which channel info struct to
- * use.
- */
- if (info_size == 2 * sizeof(struct smd_channel_info_word)) {
- channel->info_word = info;
- } else if (info_size == 2 * sizeof(struct smd_channel_info)) {
- channel->info = info;
- } else {
- dev_err(&edge->dev,
- "channel info of size %zu not supported\n", info_size);
- ret = -EINVAL;
- goto free_name_and_channel;
- }
-
- fifo_base = qcom_smem_get(edge->remote_pid, smem_fifo_item, &fifo_size);
- if (IS_ERR(fifo_base)) {
- ret = PTR_ERR(fifo_base);
- goto free_name_and_channel;
- }
-
- /* The channel consist of a rx and tx fifo of equal size */
- fifo_size /= 2;
-
- dev_dbg(&edge->dev, "new channel '%s' info-size: %zu fifo-size: %zu\n",
- name, info_size, fifo_size);
-
- channel->tx_fifo = fifo_base;
- channel->rx_fifo = fifo_base + fifo_size;
- channel->fifo_size = fifo_size;
-
- qcom_smd_channel_reset(channel);
-
- return channel;
-
-free_name_and_channel:
- devm_kfree(&edge->dev, channel->name);
- devm_kfree(&edge->dev, channel);
-
- return ERR_PTR(ret);
-}
-
-/*
- * Scans the allocation table for any newly allocated channels, calls
- * qcom_smd_create_channel() to create representations of these and add
- * them to the edge's list of channels.
- */
-static void qcom_channel_scan_worker(struct work_struct *work)
-{
- struct qcom_smd_edge *edge = container_of(work, struct qcom_smd_edge, scan_work);
- struct qcom_smd_alloc_entry *alloc_tbl;
- struct qcom_smd_alloc_entry *entry;
- struct qcom_smd_channel *channel;
- unsigned long flags;
- unsigned fifo_id;
- unsigned info_id;
- int tbl;
- int i;
- u32 eflags, cid;
-
- for (tbl = 0; tbl < SMD_ALLOC_TBL_COUNT; tbl++) {
- alloc_tbl = qcom_smem_get(edge->remote_pid,
- smem_items[tbl].alloc_tbl_id, NULL);
- if (IS_ERR(alloc_tbl))
- continue;
-
- for (i = 0; i < SMD_ALLOC_TBL_SIZE; i++) {
- entry = &alloc_tbl[i];
- eflags = le32_to_cpu(entry->flags);
- if (test_bit(i, edge->allocated[tbl]))
- continue;
-
- if (entry->ref_count == 0)
- continue;
-
- if (!entry->name[0])
- continue;
-
- if (!(eflags & SMD_CHANNEL_FLAGS_PACKET))
- continue;
-
- if ((eflags & SMD_CHANNEL_FLAGS_EDGE_MASK) != edge->edge_id)
- continue;
-
- cid = le32_to_cpu(entry->cid);
- info_id = smem_items[tbl].info_base_id + cid;
- fifo_id = smem_items[tbl].fifo_base_id + cid;
-
- channel = qcom_smd_create_channel(edge, info_id, fifo_id, entry->name);
- if (IS_ERR(channel))
- continue;
-
- spin_lock_irqsave(&edge->channels_lock, flags);
- list_add(&channel->list, &edge->channels);
- spin_unlock_irqrestore(&edge->channels_lock, flags);
-
- dev_dbg(&edge->dev, "new channel found: '%s'\n", channel->name);
- set_bit(i, edge->allocated[tbl]);
-
- wake_up_interruptible(&edge->new_channel_event);
- }
- }
-
- schedule_work(&edge->state_work);
-}
-
-/*
- * This per edge worker scans smem for any new channels and register these. It
- * then scans all registered channels for state changes that should be handled
- * by creating or destroying smd client devices for the registered channels.
- *
- * LOCKING: edge->channels_lock only needs to cover the list operations, as the
- * worker is killed before any channels are deallocated
- */
-static void qcom_channel_state_worker(struct work_struct *work)
-{
- struct qcom_smd_channel *channel;
- struct qcom_smd_edge *edge = container_of(work,
- struct qcom_smd_edge,
- state_work);
- unsigned remote_state;
- unsigned long flags;
-
- /*
- * Register a device for any closed channel where the remote processor
- * is showing interest in opening the channel.
- */
- spin_lock_irqsave(&edge->channels_lock, flags);
- list_for_each_entry(channel, &edge->channels, list) {
- if (channel->state != SMD_CHANNEL_CLOSED)
- continue;
-
- remote_state = GET_RX_CHANNEL_INFO(channel, state);
- if (remote_state != SMD_CHANNEL_OPENING &&
- remote_state != SMD_CHANNEL_OPENED)
- continue;
-
- spin_unlock_irqrestore(&edge->channels_lock, flags);
- qcom_smd_create_device(channel);
- spin_lock_irqsave(&edge->channels_lock, flags);
- }
-
- /*
- * Unregister the device for any channel that is opened where the
- * remote processor is closing the channel.
- */
- list_for_each_entry(channel, &edge->channels, list) {
- if (channel->state != SMD_CHANNEL_OPENING &&
- channel->state != SMD_CHANNEL_OPENED)
- continue;
-
- remote_state = GET_RX_CHANNEL_INFO(channel, state);
- if (remote_state == SMD_CHANNEL_OPENING ||
- remote_state == SMD_CHANNEL_OPENED)
- continue;
-
- spin_unlock_irqrestore(&edge->channels_lock, flags);
- qcom_smd_destroy_device(channel);
- spin_lock_irqsave(&edge->channels_lock, flags);
- }
- spin_unlock_irqrestore(&edge->channels_lock, flags);
-}
-
-/*
- * Parses an of_node describing an edge.
- */
-static int qcom_smd_parse_edge(struct device *dev,
- struct device_node *node,
- struct qcom_smd_edge *edge)
-{
- struct device_node *syscon_np;
- const char *key;
- int irq;
- int ret;
-
- INIT_LIST_HEAD(&edge->channels);
- spin_lock_init(&edge->channels_lock);
-
- INIT_WORK(&edge->scan_work, qcom_channel_scan_worker);
- INIT_WORK(&edge->state_work, qcom_channel_state_worker);
-
- edge->of_node = of_node_get(node);
-
- key = "qcom,smd-edge";
- ret = of_property_read_u32(node, key, &edge->edge_id);
- if (ret) {
- dev_err(dev, "edge missing %s property\n", key);
- return -EINVAL;
- }
-
- edge->remote_pid = QCOM_SMEM_HOST_ANY;
- key = "qcom,remote-pid";
- of_property_read_u32(node, key, &edge->remote_pid);
-
- syscon_np = of_parse_phandle(node, "qcom,ipc", 0);
- if (!syscon_np) {
- dev_err(dev, "no qcom,ipc node\n");
- return -ENODEV;
- }
-
- edge->ipc_regmap = syscon_node_to_regmap(syscon_np);
- if (IS_ERR(edge->ipc_regmap))
- return PTR_ERR(edge->ipc_regmap);
-
- key = "qcom,ipc";
- ret = of_property_read_u32_index(node, key, 1, &edge->ipc_offset);
- if (ret < 0) {
- dev_err(dev, "no offset in %s\n", key);
- return -EINVAL;
- }
-
- ret = of_property_read_u32_index(node, key, 2, &edge->ipc_bit);
- if (ret < 0) {
- dev_err(dev, "no bit in %s\n", key);
- return -EINVAL;
- }
-
- irq = irq_of_parse_and_map(node, 0);
- if (irq < 0) {
- dev_err(dev, "required smd interrupt missing\n");
- return -EINVAL;
- }
-
- ret = devm_request_irq(dev, irq,
- qcom_smd_edge_intr, IRQF_TRIGGER_RISING,
- node->name, edge);
- if (ret) {
- dev_err(dev, "failed to request smd irq\n");
- return ret;
- }
-
- edge->irq = irq;
-
- return 0;
-}
-
-/*
- * Release function for an edge.
- * Reset the state of each associated channel and free the edge context.
- */
-static void qcom_smd_edge_release(struct device *dev)
-{
- struct qcom_smd_channel *channel;
- struct qcom_smd_edge *edge = to_smd_edge(dev);
-
- list_for_each_entry(channel, &edge->channels, list) {
- SET_RX_CHANNEL_INFO(channel, state, SMD_CHANNEL_CLOSED);
- SET_RX_CHANNEL_INFO(channel, head, 0);
- SET_RX_CHANNEL_INFO(channel, tail, 0);
- }
-
- kfree(edge);
-}
-
-/**
- * qcom_smd_register_edge() - register an edge based on an device_node
- * @parent: parent device for the edge
- * @node: device_node describing the edge
- *
- * Returns an edge reference, or negative ERR_PTR() on failure.
- */
-struct qcom_smd_edge *qcom_smd_register_edge(struct device *parent,
- struct device_node *node)
-{
- struct qcom_smd_edge *edge;
- int ret;
-
- edge = kzalloc(sizeof(*edge), GFP_KERNEL);
- if (!edge)
- return ERR_PTR(-ENOMEM);
-
- init_waitqueue_head(&edge->new_channel_event);
-
- edge->dev.parent = parent;
- edge->dev.release = qcom_smd_edge_release;
- dev_set_name(&edge->dev, "%s:%s", dev_name(parent), node->name);
- ret = device_register(&edge->dev);
- if (ret) {
- pr_err("failed to register smd edge\n");
- return ERR_PTR(ret);
- }
-
- ret = qcom_smd_parse_edge(&edge->dev, node, edge);
- if (ret) {
- dev_err(&edge->dev, "failed to parse smd edge\n");
- goto unregister_dev;
- }
-
- schedule_work(&edge->scan_work);
-
- return edge;
-
-unregister_dev:
- put_device(&edge->dev);
- return ERR_PTR(ret);
-}
-EXPORT_SYMBOL(qcom_smd_register_edge);
-
-static int qcom_smd_remove_device(struct device *dev, void *data)
-{
- device_unregister(dev);
- of_node_put(dev->of_node);
- put_device(dev);
-
- return 0;
-}
-
-/**
- * qcom_smd_unregister_edge() - release an edge and its children
- * @edge: edge reference acquired from qcom_smd_register_edge
- */
-int qcom_smd_unregister_edge(struct qcom_smd_edge *edge)
-{
- int ret;
-
- disable_irq(edge->irq);
- cancel_work_sync(&edge->scan_work);
- cancel_work_sync(&edge->state_work);
-
- ret = device_for_each_child(&edge->dev, NULL, qcom_smd_remove_device);
- if (ret)
- dev_warn(&edge->dev, "can't remove smd device: %d\n", ret);
-
- device_unregister(&edge->dev);
-
- return 0;
-}
-EXPORT_SYMBOL(qcom_smd_unregister_edge);
-
-static int qcom_smd_probe(struct platform_device *pdev)
-{
- struct device_node *node;
- void *p;
-
- /* Wait for smem */
- p = qcom_smem_get(QCOM_SMEM_HOST_ANY, smem_items[0].alloc_tbl_id, NULL);
- if (PTR_ERR(p) == -EPROBE_DEFER)
- return PTR_ERR(p);
-
- for_each_available_child_of_node(pdev->dev.of_node, node)
- qcom_smd_register_edge(&pdev->dev, node);
-
- return 0;
-}
-
-static int qcom_smd_remove_edge(struct device *dev, void *data)
-{
- struct qcom_smd_edge *edge = to_smd_edge(dev);
-
- return qcom_smd_unregister_edge(edge);
-}
-
-/*
- * Shut down all smd clients by making sure that each edge stops processing
- * events and scanning for new channels, then call destroy on the devices.
- */
-static int qcom_smd_remove(struct platform_device *pdev)
-{
- int ret;
-
- ret = device_for_each_child(&pdev->dev, NULL, qcom_smd_remove_edge);
- if (ret)
- dev_warn(&pdev->dev, "can't remove smd device: %d\n", ret);
-
- return ret;
-}
-
-static const struct of_device_id qcom_smd_of_match[] = {
- { .compatible = "qcom,smd" },
- {}
-};
-MODULE_DEVICE_TABLE(of, qcom_smd_of_match);
-
-static struct platform_driver qcom_smd_driver = {
- .probe = qcom_smd_probe,
- .remove = qcom_smd_remove,
- .driver = {
- .name = "qcom-smd",
- .of_match_table = qcom_smd_of_match,
- },
-};
-
-static int __init qcom_smd_init(void)
-{
- int ret;
-
- ret = bus_register(&qcom_smd_bus);
- if (ret) {
- pr_err("failed to register smd bus: %d\n", ret);
- return ret;
- }
-
- return platform_driver_register(&qcom_smd_driver);
-}
-postcore_initcall(qcom_smd_init);
-
-static void __exit qcom_smd_exit(void)
-{
- platform_driver_unregister(&qcom_smd_driver);
- bus_unregister(&qcom_smd_bus);
-}
-module_exit(qcom_smd_exit);
-
-MODULE_AUTHOR("Bjorn Andersson <bjorn.andersson@sonymobile.com>");
-MODULE_DESCRIPTION("Qualcomm Shared Memory Driver");
-MODULE_LICENSE("GPL v2");
diff --git a/drivers/soc/qcom/wcnss_ctrl.c b/drivers/soc/qcom/wcnss_ctrl.c
index 520aedd..b906918 100644
--- a/drivers/soc/qcom/wcnss_ctrl.c
+++ b/drivers/soc/qcom/wcnss_ctrl.c
@@ -14,10 +14,10 @@
#include <linux/firmware.h>
#include <linux/module.h>
#include <linux/slab.h>
-#include <linux/soc/qcom/smd.h>
#include <linux/io.h>
#include <linux/of_platform.h>
#include <linux/platform_device.h>
+#include <linux/rpmsg.h>
#include <linux/soc/qcom/wcnss_ctrl.h>
#define WCNSS_REQUEST_TIMEOUT (5 * HZ)
@@ -40,7 +40,7 @@
*/
struct wcnss_ctrl {
struct device *dev;
- struct qcom_smd_channel *channel;
+ struct rpmsg_endpoint *channel;
struct completion ack;
struct completion cbc;
@@ -122,11 +122,13 @@
*
* Handles any incoming packets from the remote WCNSS_CTRL service.
*/
-static int wcnss_ctrl_smd_callback(struct qcom_smd_channel *channel,
- const void *data,
- size_t count)
+static int wcnss_ctrl_smd_callback(struct rpmsg_device *rpdev,
+ void *data,
+ int count,
+ void *priv,
+ u32 addr)
{
- struct wcnss_ctrl *wcnss = qcom_smd_get_drvdata(channel);
+ struct wcnss_ctrl *wcnss = dev_get_drvdata(&rpdev->dev);
const struct wcnss_download_nv_resp *nvresp;
const struct wcnss_version_resp *version;
const struct wcnss_msg_hdr *hdr = data;
@@ -180,7 +182,7 @@
msg.type = WCNSS_VERSION_REQ;
msg.len = sizeof(msg);
- ret = qcom_smd_send(wcnss->channel, &msg, sizeof(msg));
+ ret = rpmsg_send(wcnss->channel, &msg, sizeof(msg));
if (ret < 0)
return ret;
@@ -238,7 +240,7 @@
memcpy(req->fragment, data, req->frag_size);
- ret = qcom_smd_send(wcnss->channel, req, req->hdr.len);
+ ret = rpmsg_send(wcnss->channel, req, req->hdr.len);
if (ret < 0) {
dev_err(wcnss->dev, "failed to send smd packet\n");
goto release_fw;
@@ -274,11 +276,16 @@
* @name: SMD channel name
* @cb: callback to handle incoming data on the channel
*/
-struct qcom_smd_channel *qcom_wcnss_open_channel(void *wcnss, const char *name, qcom_smd_cb_t cb)
+struct rpmsg_endpoint *qcom_wcnss_open_channel(void *wcnss, const char *name, rpmsg_rx_cb_t cb, void *priv)
{
+ struct rpmsg_channel_info chinfo;
struct wcnss_ctrl *_wcnss = wcnss;
- return qcom_smd_open_channel(_wcnss->channel, name, cb);
+ strncpy(chinfo.name, name, sizeof(chinfo.name));
+ chinfo.src = RPMSG_ADDR_ANY;
+ chinfo.dst = RPMSG_ADDR_ANY;
+
+ return rpmsg_create_ept(_wcnss->channel->rpdev, cb, priv, chinfo);
}
EXPORT_SYMBOL(qcom_wcnss_open_channel);
@@ -306,35 +313,34 @@
of_platform_populate(wcnss->dev->of_node, NULL, NULL, wcnss->dev);
}
-static int wcnss_ctrl_probe(struct qcom_smd_device *sdev)
+static int wcnss_ctrl_probe(struct rpmsg_device *rpdev)
{
struct wcnss_ctrl *wcnss;
- wcnss = devm_kzalloc(&sdev->dev, sizeof(*wcnss), GFP_KERNEL);
+ wcnss = devm_kzalloc(&rpdev->dev, sizeof(*wcnss), GFP_KERNEL);
if (!wcnss)
return -ENOMEM;
- wcnss->dev = &sdev->dev;
- wcnss->channel = sdev->channel;
+ wcnss->dev = &rpdev->dev;
+ wcnss->channel = rpdev->ept;
init_completion(&wcnss->ack);
init_completion(&wcnss->cbc);
INIT_WORK(&wcnss->probe_work, wcnss_async_probe);
- qcom_smd_set_drvdata(sdev->channel, wcnss);
- dev_set_drvdata(&sdev->dev, wcnss);
+ dev_set_drvdata(&rpdev->dev, wcnss);
schedule_work(&wcnss->probe_work);
return 0;
}
-static void wcnss_ctrl_remove(struct qcom_smd_device *sdev)
+static void wcnss_ctrl_remove(struct rpmsg_device *rpdev)
{
- struct wcnss_ctrl *wcnss = qcom_smd_get_drvdata(sdev->channel);
+ struct wcnss_ctrl *wcnss = dev_get_drvdata(&rpdev->dev);
cancel_work_sync(&wcnss->probe_work);
- of_platform_depopulate(&sdev->dev);
+ of_platform_depopulate(&rpdev->dev);
}
static const struct of_device_id wcnss_ctrl_of_match[] = {
@@ -342,18 +348,18 @@
{}
};
-static struct qcom_smd_driver wcnss_ctrl_driver = {
+static struct rpmsg_driver wcnss_ctrl_driver = {
.probe = wcnss_ctrl_probe,
.remove = wcnss_ctrl_remove,
.callback = wcnss_ctrl_smd_callback,
- .driver = {
+ .drv = {
.name = "qcom_wcnss_ctrl",
.owner = THIS_MODULE,
.of_match_table = wcnss_ctrl_of_match,
},
};
-module_qcom_smd_driver(wcnss_ctrl_driver);
+module_rpmsg_driver(wcnss_ctrl_driver);
MODULE_DESCRIPTION("Qualcomm WCNSS control driver");
MODULE_LICENSE("GPL v2");
diff --git a/drivers/staging/lustre/lustre/include/lustre_disk.h b/drivers/staging/lustre/lustre/include/lustre_disk.h
index 8886458..a676bcc 100644
--- a/drivers/staging/lustre/lustre/include/lustre_disk.h
+++ b/drivers/staging/lustre/lustre/include/lustre_disk.h
@@ -133,13 +133,9 @@
struct obd_export *lsi_osd_exp;
char lsi_osd_type[16];
char lsi_fstype[16];
- struct backing_dev_info lsi_bdi; /* each client mountpoint needs
- * own backing_dev_info
- */
};
#define LSI_UMOUNT_FAILOVER 0x00200000
-#define LSI_BDI_INITIALIZED 0x00400000
#define s2lsi(sb) ((struct lustre_sb_info *)((sb)->s_fs_info))
#define s2lsi_nocast(sb) ((sb)->s_fs_info)
diff --git a/drivers/staging/lustre/lustre/llite/llite_lib.c b/drivers/staging/lustre/lustre/llite/llite_lib.c
index b229cbc..d483c44 100644
--- a/drivers/staging/lustre/lustre/llite/llite_lib.c
+++ b/drivers/staging/lustre/lustre/llite/llite_lib.c
@@ -863,15 +863,6 @@
mutex_init(&lli->lli_layout_mutex);
}
-static inline int ll_bdi_register(struct backing_dev_info *bdi)
-{
- static atomic_t ll_bdi_num = ATOMIC_INIT(0);
-
- bdi->name = "lustre";
- return bdi_register(bdi, NULL, "lustre-%d",
- atomic_inc_return(&ll_bdi_num));
-}
-
int ll_fill_super(struct super_block *sb, struct vfsmount *mnt)
{
struct lustre_profile *lprof = NULL;
@@ -881,6 +872,7 @@
char *profilenm = get_profile_name(sb);
struct config_llog_instance *cfg;
int err;
+ static atomic_t ll_bdi_num = ATOMIC_INIT(0);
CDEBUG(D_VFSTRACE, "VFS Op: sb %p\n", sb);
@@ -903,16 +895,11 @@
if (err)
goto out_free;
- err = bdi_init(&lsi->lsi_bdi);
- if (err)
- goto out_free;
- lsi->lsi_flags |= LSI_BDI_INITIALIZED;
- lsi->lsi_bdi.capabilities = 0;
- err = ll_bdi_register(&lsi->lsi_bdi);
+ err = super_setup_bdi_name(sb, "lustre-%d",
+ atomic_inc_return(&ll_bdi_num));
if (err)
goto out_free;
- sb->s_bdi = &lsi->lsi_bdi;
/* kernel >= 2.6.38 store dentry operations in sb->s_d_op. */
sb->s_d_op = &ll_d_ops;
@@ -1033,11 +1020,6 @@
if (profilenm)
class_del_profile(profilenm);
- if (lsi->lsi_flags & LSI_BDI_INITIALIZED) {
- bdi_destroy(&lsi->lsi_bdi);
- lsi->lsi_flags &= ~LSI_BDI_INITIALIZED;
- }
-
ll_free_sbi(sb);
lsi->lsi_llsbi = NULL;
diff --git a/drivers/staging/most/hdm-usb/hdm_usb.c b/drivers/staging/most/hdm-usb/hdm_usb.c
index 65211d1..2bfea9b 100644
--- a/drivers/staging/most/hdm-usb/hdm_usb.c
+++ b/drivers/staging/most/hdm-usb/hdm_usb.c
@@ -490,7 +490,7 @@
* disconnect. In the interval before the hub driver starts disconnect
* processing, devices may receive such fault reports for every request.
*
- * See <https://www.kernel.org/doc/Documentation/usb/error-codes.txt>
+ * See <https://www.kernel.org/doc/Documentation/driver-api/usb/error-codes.rst>
*/
static void hdm_read_completion(struct urb *urb)
{
diff --git a/drivers/staging/wilc1000/wilc_wfi_cfgoperations.c b/drivers/staging/wilc1000/wilc_wfi_cfgoperations.c
index 7961d1c..2b45363 100644
--- a/drivers/staging/wilc1000/wilc_wfi_cfgoperations.c
+++ b/drivers/staging/wilc1000/wilc_wfi_cfgoperations.c
@@ -1837,7 +1837,7 @@
}
static int change_virtual_intf(struct wiphy *wiphy, struct net_device *dev,
- enum nl80211_iftype type, u32 *flags, struct vif_params *params)
+ enum nl80211_iftype type, struct vif_params *params)
{
struct wilc_priv *priv;
struct wilc_vif *vif;
@@ -2099,7 +2099,6 @@
const char *name,
unsigned char name_assign_type,
enum nl80211_iftype type,
- u32 *flags,
struct vif_params *params)
{
struct wilc_vif *vif;
diff --git a/drivers/staging/wlan-ng/cfg80211.c b/drivers/staging/wlan-ng/cfg80211.c
index 11870cb..178f6f5 100644
--- a/drivers/staging/wlan-ng/cfg80211.c
+++ b/drivers/staging/wlan-ng/cfg80211.c
@@ -100,7 +100,7 @@
/* The interface functions, called by the cfg80211 layer */
static int prism2_change_virtual_intf(struct wiphy *wiphy,
struct net_device *dev,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct wlandevice *wlandev = dev->ml_priv;
@@ -666,8 +666,11 @@
void prism2_roamed(struct wlandevice *wlandev)
{
- cfg80211_roamed(wlandev->netdev, NULL, wlandev->bssid,
- NULL, 0, NULL, 0, GFP_KERNEL);
+ struct cfg80211_roam_info roam_info = {
+ .bssid = wlandev->bssid,
+ };
+
+ cfg80211_roamed(wlandev->netdev, &roam_info, GFP_KERNEL);
}
/* Structures for declaring wiphy interface */
diff --git a/drivers/target/iscsi/iscsi_target_configfs.c b/drivers/target/iscsi/iscsi_target_configfs.c
index 344e844..5798810 100644
--- a/drivers/target/iscsi/iscsi_target_configfs.c
+++ b/drivers/target/iscsi/iscsi_target_configfs.c
@@ -167,10 +167,7 @@
struct iscsi_portal_group *tpg;
struct iscsi_tpg_np *tpg_np;
char *str, *str2, *ip_str, *port_str;
- struct sockaddr_storage sockaddr;
- struct sockaddr_in *sock_in;
- struct sockaddr_in6 *sock_in6;
- unsigned long port;
+ struct sockaddr_storage sockaddr = { };
int ret;
char buf[MAX_PORTAL_LEN + 1];
@@ -182,21 +179,19 @@
memset(buf, 0, MAX_PORTAL_LEN + 1);
snprintf(buf, MAX_PORTAL_LEN + 1, "%s", name);
- memset(&sockaddr, 0, sizeof(struct sockaddr_storage));
-
str = strstr(buf, "[");
if (str) {
- const char *end;
-
str2 = strstr(str, "]");
if (!str2) {
pr_err("Unable to locate trailing \"]\""
" in IPv6 iSCSI network portal address\n");
return ERR_PTR(-EINVAL);
}
- str++; /* Skip over leading "[" */
+
+ ip_str = str + 1; /* Skip over leading "[" */
*str2 = '\0'; /* Terminate the unbracketed IPv6 address */
str2++; /* Skip over the \0 */
+
port_str = strstr(str2, ":");
if (!port_str) {
pr_err("Unable to locate \":port\""
@@ -205,23 +200,8 @@
}
*port_str = '\0'; /* Terminate string for IP */
port_str++; /* Skip over ":" */
-
- ret = kstrtoul(port_str, 0, &port);
- if (ret < 0) {
- pr_err("kstrtoul() failed for port_str: %d\n", ret);
- return ERR_PTR(ret);
- }
- sock_in6 = (struct sockaddr_in6 *)&sockaddr;
- sock_in6->sin6_family = AF_INET6;
- sock_in6->sin6_port = htons((unsigned short)port);
- ret = in6_pton(str, -1,
- (void *)&sock_in6->sin6_addr.in6_u, -1, &end);
- if (ret <= 0) {
- pr_err("in6_pton returned: %d\n", ret);
- return ERR_PTR(-EINVAL);
- }
} else {
- str = ip_str = &buf[0];
+ ip_str = &buf[0];
port_str = strstr(ip_str, ":");
if (!port_str) {
pr_err("Unable to locate \":port\""
@@ -230,17 +210,15 @@
}
*port_str = '\0'; /* Terminate string for IP */
port_str++; /* Skip over ":" */
-
- ret = kstrtoul(port_str, 0, &port);
- if (ret < 0) {
- pr_err("kstrtoul() failed for port_str: %d\n", ret);
- return ERR_PTR(ret);
- }
- sock_in = (struct sockaddr_in *)&sockaddr;
- sock_in->sin_family = AF_INET;
- sock_in->sin_port = htons((unsigned short)port);
- sock_in->sin_addr.s_addr = in_aton(ip_str);
}
+
+ ret = inet_pton_with_scope(&init_net, AF_UNSPEC, ip_str,
+ port_str, &sockaddr);
+ if (ret) {
+ pr_err("malformed ip/port passed: %s\n", name);
+ return ERR_PTR(ret);
+ }
+
tpg = container_of(se_tpg, struct iscsi_portal_group, tpg_se_tpg);
ret = iscsit_get_tpg(tpg);
if (ret < 0)
diff --git a/drivers/target/target_core_device.c b/drivers/target/target_core_device.c
index c754ae3..d2f089c 100644
--- a/drivers/target/target_core_device.c
+++ b/drivers/target/target_core_device.c
@@ -851,7 +851,7 @@
attrib->unmap_granularity = q->limits.discard_granularity / block_size;
attrib->unmap_granularity_alignment = q->limits.discard_alignment /
block_size;
- attrib->unmap_zeroes_data = q->limits.discard_zeroes_data;
+ attrib->unmap_zeroes_data = 0;
return true;
}
EXPORT_SYMBOL(target_configure_unmap_from_queue);
diff --git a/drivers/target/target_core_pscsi.c b/drivers/target/target_core_pscsi.c
index 94cda79..a93d94e6 100644
--- a/drivers/target/target_core_pscsi.c
+++ b/drivers/target/target_core_pscsi.c
@@ -1008,7 +1008,7 @@
req->timeout = PS_TIMEOUT_DISK;
else
req->timeout = PS_TIMEOUT_OTHER;
- req->retries = PS_RETRY;
+ scsi_req(req)->retries = PS_RETRY;
blk_execute_rq_nowait(pdv->pdv_sd->request_queue, NULL, req,
(cmd->sam_task_attr == TCM_HEAD_TAG),
@@ -1050,7 +1050,7 @@
struct se_cmd *cmd = req->end_io_data;
struct pscsi_plugin_task *pt = cmd->priv;
- pt->pscsi_result = req->errors;
+ pt->pscsi_result = scsi_req(req)->result;
pt->pscsi_resid = scsi_req(req)->resid_len;
cmd->scsi_status = status_byte(pt->pscsi_result) << 1;
diff --git a/drivers/thermal/Kconfig b/drivers/thermal/Kconfig
index 776b343..0a16cf4 100644
--- a/drivers/thermal/Kconfig
+++ b/drivers/thermal/Kconfig
@@ -291,18 +291,6 @@
Enable this option if you want to have support for thermal management
controller present in Armada 370 and Armada XP SoC.
-config DB8500_CPUFREQ_COOLING
- tristate "DB8500 cpufreq cooling"
- depends on ARCH_U8500 || COMPILE_TEST
- depends on HAS_IOMEM
- depends on CPU_THERMAL
- default y
- help
- Adds DB8500 cpufreq cooling devices, and these cooling devices can be
- bound to thermal zone trip points. When a trip point reached, the
- bound cpufreq cooling device turns active to set CPU frequency low to
- cool down the CPU.
-
config INTEL_POWERCLAMP
tristate "Intel PowerClamp idle injection driver"
depends on THERMAL
diff --git a/drivers/thermal/Makefile b/drivers/thermal/Makefile
index 7adae20..c2372f1 100644
--- a/drivers/thermal/Makefile
+++ b/drivers/thermal/Makefile
@@ -41,7 +41,6 @@
obj-$(CONFIG_IMX_THERMAL) += imx_thermal.o
obj-$(CONFIG_MAX77620_THERMAL) += max77620_thermal.o
obj-$(CONFIG_QORIQ_THERMAL) += qoriq_thermal.o
-obj-$(CONFIG_DB8500_CPUFREQ_COOLING) += db8500_cpufreq_cooling.o
obj-$(CONFIG_INTEL_POWERCLAMP) += intel_powerclamp.o
obj-$(CONFIG_X86_PKG_TEMP_THERMAL) += x86_pkg_temp_thermal.o
obj-$(CONFIG_INTEL_SOC_DTS_IOSF_CORE) += intel_soc_dts_iosf.o
diff --git a/drivers/thermal/db8500_cpufreq_cooling.c b/drivers/thermal/db8500_cpufreq_cooling.c
deleted file mode 100644
index e58bd0b..0000000
--- a/drivers/thermal/db8500_cpufreq_cooling.c
+++ /dev/null
@@ -1,105 +0,0 @@
-/*
- * db8500_cpufreq_cooling.c - DB8500 cpufreq works as cooling device.
- *
- * Copyright (C) 2012 ST-Ericsson
- * Copyright (C) 2012 Linaro Ltd.
- *
- * Author: Hongbo Zhang <hongbo.zhang@linaro.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- */
-
-#include <linux/cpu_cooling.h>
-#include <linux/err.h>
-#include <linux/module.h>
-#include <linux/of.h>
-#include <linux/platform_device.h>
-#include <linux/slab.h>
-
-static int db8500_cpufreq_cooling_probe(struct platform_device *pdev)
-{
- struct thermal_cooling_device *cdev;
-
- cdev = cpufreq_cooling_register(cpu_present_mask);
- if (IS_ERR(cdev)) {
- int ret = PTR_ERR(cdev);
-
- if (ret != -EPROBE_DEFER)
- dev_err(&pdev->dev,
- "Failed to register cooling device %d\n",
- ret);
-
- return ret;
- }
-
- platform_set_drvdata(pdev, cdev);
-
- dev_info(&pdev->dev, "Cooling device registered: %s\n", cdev->type);
-
- return 0;
-}
-
-static int db8500_cpufreq_cooling_remove(struct platform_device *pdev)
-{
- struct thermal_cooling_device *cdev = platform_get_drvdata(pdev);
-
- cpufreq_cooling_unregister(cdev);
-
- return 0;
-}
-
-static int db8500_cpufreq_cooling_suspend(struct platform_device *pdev,
- pm_message_t state)
-{
- return -ENOSYS;
-}
-
-static int db8500_cpufreq_cooling_resume(struct platform_device *pdev)
-{
- return -ENOSYS;
-}
-
-#ifdef CONFIG_OF
-static const struct of_device_id db8500_cpufreq_cooling_match[] = {
- { .compatible = "stericsson,db8500-cpufreq-cooling" },
- {},
-};
-MODULE_DEVICE_TABLE(of, db8500_cpufreq_cooling_match);
-#endif
-
-static struct platform_driver db8500_cpufreq_cooling_driver = {
- .driver = {
- .name = "db8500-cpufreq-cooling",
- .of_match_table = of_match_ptr(db8500_cpufreq_cooling_match),
- },
- .probe = db8500_cpufreq_cooling_probe,
- .suspend = db8500_cpufreq_cooling_suspend,
- .resume = db8500_cpufreq_cooling_resume,
- .remove = db8500_cpufreq_cooling_remove,
-};
-
-static int __init db8500_cpufreq_cooling_init(void)
-{
- return platform_driver_register(&db8500_cpufreq_cooling_driver);
-}
-
-static void __exit db8500_cpufreq_cooling_exit(void)
-{
- platform_driver_unregister(&db8500_cpufreq_cooling_driver);
-}
-
-/* Should be later than db8500_cpufreq_register */
-late_initcall(db8500_cpufreq_cooling_init);
-module_exit(db8500_cpufreq_cooling_exit);
-
-MODULE_AUTHOR("Hongbo Zhang <hongbo.zhang@stericsson.com>");
-MODULE_DESCRIPTION("DB8500 cpufreq cooling driver");
-MODULE_LICENSE("GPL");
diff --git a/drivers/tty/serdev/core.c b/drivers/tty/serdev/core.c
index f4c6c90..1e1cbae 100644
--- a/drivers/tty/serdev/core.c
+++ b/drivers/tty/serdev/core.c
@@ -173,6 +173,39 @@
}
EXPORT_SYMBOL_GPL(serdev_device_set_flow_control);
+void serdev_device_wait_until_sent(struct serdev_device *serdev, long timeout)
+{
+ struct serdev_controller *ctrl = serdev->ctrl;
+
+ if (!ctrl || !ctrl->ops->wait_until_sent)
+ return;
+
+ ctrl->ops->wait_until_sent(ctrl, timeout);
+}
+EXPORT_SYMBOL_GPL(serdev_device_wait_until_sent);
+
+int serdev_device_get_tiocm(struct serdev_device *serdev)
+{
+ struct serdev_controller *ctrl = serdev->ctrl;
+
+ if (!ctrl || !ctrl->ops->get_tiocm)
+ return -ENOTSUPP;
+
+ return ctrl->ops->get_tiocm(ctrl);
+}
+EXPORT_SYMBOL_GPL(serdev_device_get_tiocm);
+
+int serdev_device_set_tiocm(struct serdev_device *serdev, int set, int clear)
+{
+ struct serdev_controller *ctrl = serdev->ctrl;
+
+ if (!ctrl || !ctrl->ops->set_tiocm)
+ return -ENOTSUPP;
+
+ return ctrl->ops->set_tiocm(ctrl, set, clear);
+}
+EXPORT_SYMBOL_GPL(serdev_device_set_tiocm);
+
static int serdev_drv_probe(struct device *dev)
{
const struct serdev_device_driver *sdrv = to_serdev_device_driver(dev->driver);
diff --git a/drivers/tty/serdev/serdev-ttyport.c b/drivers/tty/serdev/serdev-ttyport.c
index d053935..487c88f 100644
--- a/drivers/tty/serdev/serdev-ttyport.c
+++ b/drivers/tty/serdev/serdev-ttyport.c
@@ -14,6 +14,7 @@
#include <linux/serdev.h>
#include <linux/tty.h>
#include <linux/tty_driver.h>
+#include <linux/poll.h>
#define SERPORT_ACTIVE 1
@@ -46,11 +47,11 @@
struct serdev_controller *ctrl = port->client_data;
struct serport *serport = serdev_controller_get_drvdata(ctrl);
- if (!test_and_clear_bit(TTY_DO_WRITE_WAKEUP, &port->tty->flags))
- return;
-
- if (test_bit(SERPORT_ACTIVE, &serport->flags))
+ if (test_and_clear_bit(TTY_DO_WRITE_WAKEUP, &port->tty->flags) &&
+ test_bit(SERPORT_ACTIVE, &serport->flags))
serdev_controller_write_wakeup(ctrl);
+
+ wake_up_interruptible_poll(&port->tty->write_wait, POLLOUT);
}
static const struct tty_port_client_operations client_ops = {
@@ -167,6 +168,36 @@
tty_set_termios(tty, &ktermios);
}
+static void ttyport_wait_until_sent(struct serdev_controller *ctrl, long timeout)
+{
+ struct serport *serport = serdev_controller_get_drvdata(ctrl);
+ struct tty_struct *tty = serport->tty;
+
+ tty_wait_until_sent(tty, timeout);
+}
+
+static int ttyport_get_tiocm(struct serdev_controller *ctrl)
+{
+ struct serport *serport = serdev_controller_get_drvdata(ctrl);
+ struct tty_struct *tty = serport->tty;
+
+ if (!tty->ops->tiocmget)
+ return -ENOTSUPP;
+
+ return tty->driver->ops->tiocmget(tty);
+}
+
+static int ttyport_set_tiocm(struct serdev_controller *ctrl, unsigned int set, unsigned int clear)
+{
+ struct serport *serport = serdev_controller_get_drvdata(ctrl);
+ struct tty_struct *tty = serport->tty;
+
+ if (!tty->ops->tiocmset)
+ return -ENOTSUPP;
+
+ return tty->driver->ops->tiocmset(tty, set, clear);
+}
+
static const struct serdev_controller_ops ctrl_ops = {
.write_buf = ttyport_write_buf,
.write_flush = ttyport_write_flush,
@@ -175,6 +206,9 @@
.close = ttyport_close,
.set_flow_control = ttyport_set_flow_control,
.set_baudrate = ttyport_set_baudrate,
+ .wait_until_sent = ttyport_wait_until_sent,
+ .get_tiocm = ttyport_get_tiocm,
+ .set_tiocm = ttyport_set_tiocm,
};
struct device *serdev_tty_port_register(struct tty_port *port,
diff --git a/drivers/tty/serial/omap-serial.c b/drivers/tty/serial/omap-serial.c
index 6c6f82a..a473464 100644
--- a/drivers/tty/serial/omap-serial.c
+++ b/drivers/tty/serial/omap-serial.c
@@ -1597,6 +1597,9 @@
of_property_read_u32(dev->of_node, "clock-frequency",
&omap_up_info->uartclk);
+
+ omap_up_info->flags = UPF_BOOT_AUTOCONF;
+
return omap_up_info;
}
diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c
index c6fc714..677f0dd 100644
--- a/drivers/tty/sysrq.c
+++ b/drivers/tty/sysrq.c
@@ -446,7 +446,7 @@
*/
NULL, /* a */
&sysrq_reboot_op, /* b */
- &sysrq_crash_op, /* c & ibm_emac driver debug */
+ &sysrq_crash_op, /* c */
&sysrq_showlocks_op, /* d */
&sysrq_term_op, /* e */
&sysrq_moom_op, /* f */
diff --git a/drivers/tty/tty_io.c b/drivers/tty/tty_io.c
index e6d1a65..309d250 100644
--- a/drivers/tty/tty_io.c
+++ b/drivers/tty/tty_io.c
@@ -3370,7 +3370,7 @@
/**
* __tty_alloc_driver -- allocate tty driver
* @lines: count of lines this driver can handle at most
- * @owner: module which is repsonsible for this driver
+ * @owner: module which is responsible for this driver
* @flags: some of TTY_DRIVER_* flags, will be set in driver->flags
*
* This should not be called directly, some of the provided macros should be
diff --git a/drivers/usb/Makefile b/drivers/usb/Makefile
index 7791af6..53d1356 100644
--- a/drivers/usb/Makefile
+++ b/drivers/usb/Makefile
@@ -49,7 +49,7 @@
obj-$(CONFIG_USB_SERIAL) += serial/
obj-$(CONFIG_USB) += misc/
-obj-$(CONFIG_EARLY_PRINTK_DBGP) += early/
+obj-$(CONFIG_EARLY_PRINTK_USB) += early/
obj-$(CONFIG_USB_ATM) += atm/
obj-$(CONFIG_USB_SPEEDTOUCH) += atm/
diff --git a/drivers/usb/core/Kconfig b/drivers/usb/core/Kconfig
index 0e5a889..4d75d9a 100644
--- a/drivers/usb/core/Kconfig
+++ b/drivers/usb/core/Kconfig
@@ -26,7 +26,7 @@
unplugged, causing any mounted filesystems to be lost. The
persist feature can still be enabled for individual devices
through the power/persist sysfs node. See
- Documentation/usb/persist.txt for more info.
+ Documentation/driver-api/usb/persist.rst for more info.
If you have any questions about this, say Y here, only say N
if you know exactly what you are doing.
diff --git a/drivers/usb/core/message.c b/drivers/usb/core/message.c
index 2184ef4..4c38ea4 100644
--- a/drivers/usb/core/message.c
+++ b/drivers/usb/core/message.c
@@ -474,6 +474,7 @@
* significantly improve USB throughput.
*
* There are three kinds of completion for this function.
+ *
* (1) success, where io->status is zero. The number of io->bytes
* transferred is as requested.
* (2) error, where io->status is a negative errno value. The number
diff --git a/drivers/usb/early/Makefile b/drivers/usb/early/Makefile
index 24bbe519..fcde228 100644
--- a/drivers/usb/early/Makefile
+++ b/drivers/usb/early/Makefile
@@ -3,3 +3,4 @@
#
obj-$(CONFIG_EARLY_PRINTK_DBGP) += ehci-dbgp.o
+obj-$(CONFIG_EARLY_PRINTK_USB_XDBC) += xhci-dbc.o
diff --git a/drivers/usb/early/xhci-dbc.c b/drivers/usb/early/xhci-dbc.c
new file mode 100644
index 0000000..1268818
--- /dev/null
+++ b/drivers/usb/early/xhci-dbc.c
@@ -0,0 +1,1014 @@
+/**
+ * xhci-dbc.c - xHCI debug capability early driver
+ *
+ * Copyright (C) 2016 Intel Corporation
+ *
+ * Author: Lu Baolu <baolu.lu@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ":%s: " fmt, __func__
+
+#include <linux/console.h>
+#include <linux/pci_regs.h>
+#include <linux/pci_ids.h>
+#include <linux/bootmem.h>
+#include <linux/io.h>
+#include <asm/pci-direct.h>
+#include <asm/fixmap.h>
+#include <linux/bcd.h>
+#include <linux/export.h>
+#include <linux/version.h>
+#include <linux/module.h>
+#include <linux/delay.h>
+#include <linux/kthread.h>
+
+#include "../host/xhci.h"
+#include "xhci-dbc.h"
+
+static struct xdbc_state xdbc;
+static bool early_console_keep;
+
+#define XDBC_TRACE
+#ifdef XDBC_TRACE
+#define xdbc_trace trace_printk
+#else
+static inline void xdbc_trace(const char *fmt, ...) { }
+#endif /* XDBC_TRACE */
+
+static void __iomem * __init xdbc_map_pci_mmio(u32 bus, u32 dev, u32 func)
+{
+ u64 val64, sz64, mask64;
+ void __iomem *base;
+ u32 val, sz;
+ u8 byte;
+
+ val = read_pci_config(bus, dev, func, PCI_BASE_ADDRESS_0);
+ write_pci_config(bus, dev, func, PCI_BASE_ADDRESS_0, ~0);
+ sz = read_pci_config(bus, dev, func, PCI_BASE_ADDRESS_0);
+ write_pci_config(bus, dev, func, PCI_BASE_ADDRESS_0, val);
+
+ if (val == 0xffffffff || sz == 0xffffffff) {
+ pr_notice("invalid mmio bar\n");
+ return NULL;
+ }
+
+ val64 = val & PCI_BASE_ADDRESS_MEM_MASK;
+ sz64 = sz & PCI_BASE_ADDRESS_MEM_MASK;
+ mask64 = PCI_BASE_ADDRESS_MEM_MASK;
+
+ if ((val & PCI_BASE_ADDRESS_MEM_TYPE_MASK) == PCI_BASE_ADDRESS_MEM_TYPE_64) {
+ val = read_pci_config(bus, dev, func, PCI_BASE_ADDRESS_0 + 4);
+ write_pci_config(bus, dev, func, PCI_BASE_ADDRESS_0 + 4, ~0);
+ sz = read_pci_config(bus, dev, func, PCI_BASE_ADDRESS_0 + 4);
+ write_pci_config(bus, dev, func, PCI_BASE_ADDRESS_0 + 4, val);
+
+ val64 |= (u64)val << 32;
+ sz64 |= (u64)sz << 32;
+ mask64 |= ~0ULL << 32;
+ }
+
+ sz64 &= mask64;
+
+ if (!sz64) {
+ pr_notice("invalid mmio address\n");
+ return NULL;
+ }
+
+ sz64 = 1ULL << __ffs64(sz64);
+
+ /* Check if the mem space is enabled: */
+ byte = read_pci_config_byte(bus, dev, func, PCI_COMMAND);
+ if (!(byte & PCI_COMMAND_MEMORY)) {
+ byte |= PCI_COMMAND_MEMORY;
+ write_pci_config_byte(bus, dev, func, PCI_COMMAND, byte);
+ }
+
+ xdbc.xhci_start = val64;
+ xdbc.xhci_length = sz64;
+ base = early_ioremap(val64, sz64);
+
+ return base;
+}
+
+static void * __init xdbc_get_page(dma_addr_t *dma_addr)
+{
+ void *virt;
+
+ virt = alloc_bootmem_pages_nopanic(PAGE_SIZE);
+ if (!virt)
+ return NULL;
+
+ if (dma_addr)
+ *dma_addr = (dma_addr_t)__pa(virt);
+
+ return virt;
+}
+
+static u32 __init xdbc_find_dbgp(int xdbc_num, u32 *b, u32 *d, u32 *f)
+{
+ u32 bus, dev, func, class;
+
+ for (bus = 0; bus < XDBC_PCI_MAX_BUSES; bus++) {
+ for (dev = 0; dev < XDBC_PCI_MAX_DEVICES; dev++) {
+ for (func = 0; func < XDBC_PCI_MAX_FUNCTION; func++) {
+
+ class = read_pci_config(bus, dev, func, PCI_CLASS_REVISION);
+ if ((class >> 8) != PCI_CLASS_SERIAL_USB_XHCI)
+ continue;
+
+ if (xdbc_num-- != 0)
+ continue;
+
+ *b = bus;
+ *d = dev;
+ *f = func;
+
+ return 0;
+ }
+ }
+ }
+
+ return -1;
+}
+
+static int handshake(void __iomem *ptr, u32 mask, u32 done, int wait, int delay)
+{
+ u32 result;
+
+ do {
+ result = readl(ptr);
+ result &= mask;
+ if (result == done)
+ return 0;
+ udelay(delay);
+ wait -= delay;
+ } while (wait > 0);
+
+ return -ETIMEDOUT;
+}
+
+static void __init xdbc_bios_handoff(void)
+{
+ int offset, timeout;
+ u32 val;
+
+ offset = xhci_find_next_ext_cap(xdbc.xhci_base, 0, XHCI_EXT_CAPS_LEGACY);
+ val = readl(xdbc.xhci_base + offset);
+
+ if (val & XHCI_HC_BIOS_OWNED) {
+ writel(val | XHCI_HC_OS_OWNED, xdbc.xhci_base + offset);
+ timeout = handshake(xdbc.xhci_base + offset, XHCI_HC_BIOS_OWNED, 0, 5000, 10);
+
+ if (timeout) {
+ pr_notice("failed to hand over xHCI control from BIOS\n");
+ writel(val & ~XHCI_HC_BIOS_OWNED, xdbc.xhci_base + offset);
+ }
+ }
+
+ /* Disable BIOS SMIs and clear all SMI events: */
+ val = readl(xdbc.xhci_base + offset + XHCI_LEGACY_CONTROL_OFFSET);
+ val &= XHCI_LEGACY_DISABLE_SMI;
+ val |= XHCI_LEGACY_SMI_EVENTS;
+ writel(val, xdbc.xhci_base + offset + XHCI_LEGACY_CONTROL_OFFSET);
+}
+
+static int __init
+xdbc_alloc_ring(struct xdbc_segment *seg, struct xdbc_ring *ring)
+{
+ seg->trbs = xdbc_get_page(&seg->dma);
+ if (!seg->trbs)
+ return -ENOMEM;
+
+ ring->segment = seg;
+
+ return 0;
+}
+
+static void __init xdbc_free_ring(struct xdbc_ring *ring)
+{
+ struct xdbc_segment *seg = ring->segment;
+
+ if (!seg)
+ return;
+
+ free_bootmem(seg->dma, PAGE_SIZE);
+ ring->segment = NULL;
+}
+
+static void xdbc_reset_ring(struct xdbc_ring *ring)
+{
+ struct xdbc_segment *seg = ring->segment;
+ struct xdbc_trb *link_trb;
+
+ memset(seg->trbs, 0, PAGE_SIZE);
+
+ ring->enqueue = seg->trbs;
+ ring->dequeue = seg->trbs;
+ ring->cycle_state = 1;
+
+ if (ring != &xdbc.evt_ring) {
+ link_trb = &seg->trbs[XDBC_TRBS_PER_SEGMENT - 1];
+ link_trb->field[0] = cpu_to_le32(lower_32_bits(seg->dma));
+ link_trb->field[1] = cpu_to_le32(upper_32_bits(seg->dma));
+ link_trb->field[3] = cpu_to_le32(TRB_TYPE(TRB_LINK)) | cpu_to_le32(LINK_TOGGLE);
+ }
+}
+
+static inline void xdbc_put_utf16(u16 *s, const char *c, size_t size)
+{
+ int i;
+
+ for (i = 0; i < size; i++)
+ s[i] = cpu_to_le16(c[i]);
+}
+
+static void xdbc_mem_init(void)
+{
+ struct xdbc_ep_context *ep_in, *ep_out;
+ struct usb_string_descriptor *s_desc;
+ struct xdbc_erst_entry *entry;
+ struct xdbc_strings *strings;
+ struct xdbc_context *ctx;
+ unsigned int max_burst;
+ u32 string_length;
+ int index = 0;
+ u32 dev_info;
+
+ xdbc_reset_ring(&xdbc.evt_ring);
+ xdbc_reset_ring(&xdbc.in_ring);
+ xdbc_reset_ring(&xdbc.out_ring);
+ memset(xdbc.table_base, 0, PAGE_SIZE);
+ memset(xdbc.out_buf, 0, PAGE_SIZE);
+
+ /* Initialize event ring segment table: */
+ xdbc.erst_size = 16;
+ xdbc.erst_base = xdbc.table_base + index * XDBC_TABLE_ENTRY_SIZE;
+ xdbc.erst_dma = xdbc.table_dma + index * XDBC_TABLE_ENTRY_SIZE;
+
+ index += XDBC_ERST_ENTRY_NUM;
+ entry = (struct xdbc_erst_entry *)xdbc.erst_base;
+
+ entry->seg_addr = cpu_to_le64(xdbc.evt_seg.dma);
+ entry->seg_size = cpu_to_le32(XDBC_TRBS_PER_SEGMENT);
+ entry->__reserved_0 = 0;
+
+ /* Initialize ERST registers: */
+ writel(1, &xdbc.xdbc_reg->ersts);
+ xdbc_write64(xdbc.erst_dma, &xdbc.xdbc_reg->erstba);
+ xdbc_write64(xdbc.evt_seg.dma, &xdbc.xdbc_reg->erdp);
+
+ /* Debug capability contexts: */
+ xdbc.dbcc_size = 64 * 3;
+ xdbc.dbcc_base = xdbc.table_base + index * XDBC_TABLE_ENTRY_SIZE;
+ xdbc.dbcc_dma = xdbc.table_dma + index * XDBC_TABLE_ENTRY_SIZE;
+
+ index += XDBC_DBCC_ENTRY_NUM;
+
+ /* Popluate the strings: */
+ xdbc.string_size = sizeof(struct xdbc_strings);
+ xdbc.string_base = xdbc.table_base + index * XDBC_TABLE_ENTRY_SIZE;
+ xdbc.string_dma = xdbc.table_dma + index * XDBC_TABLE_ENTRY_SIZE;
+ strings = (struct xdbc_strings *)xdbc.string_base;
+
+ index += XDBC_STRING_ENTRY_NUM;
+
+ /* Serial string: */
+ s_desc = (struct usb_string_descriptor *)strings->serial;
+ s_desc->bLength = (strlen(XDBC_STRING_SERIAL) + 1) * 2;
+ s_desc->bDescriptorType = USB_DT_STRING;
+
+ xdbc_put_utf16(s_desc->wData, XDBC_STRING_SERIAL, strlen(XDBC_STRING_SERIAL));
+ string_length = s_desc->bLength;
+ string_length <<= 8;
+
+ /* Product string: */
+ s_desc = (struct usb_string_descriptor *)strings->product;
+ s_desc->bLength = (strlen(XDBC_STRING_PRODUCT) + 1) * 2;
+ s_desc->bDescriptorType = USB_DT_STRING;
+
+ xdbc_put_utf16(s_desc->wData, XDBC_STRING_PRODUCT, strlen(XDBC_STRING_PRODUCT));
+ string_length += s_desc->bLength;
+ string_length <<= 8;
+
+ /* Manufacture string: */
+ s_desc = (struct usb_string_descriptor *)strings->manufacturer;
+ s_desc->bLength = (strlen(XDBC_STRING_MANUFACTURER) + 1) * 2;
+ s_desc->bDescriptorType = USB_DT_STRING;
+
+ xdbc_put_utf16(s_desc->wData, XDBC_STRING_MANUFACTURER, strlen(XDBC_STRING_MANUFACTURER));
+ string_length += s_desc->bLength;
+ string_length <<= 8;
+
+ /* String0: */
+ strings->string0[0] = 4;
+ strings->string0[1] = USB_DT_STRING;
+ strings->string0[2] = 0x09;
+ strings->string0[3] = 0x04;
+
+ string_length += 4;
+
+ /* Populate info Context: */
+ ctx = (struct xdbc_context *)xdbc.dbcc_base;
+
+ ctx->info.string0 = cpu_to_le64(xdbc.string_dma);
+ ctx->info.manufacturer = cpu_to_le64(xdbc.string_dma + XDBC_MAX_STRING_LENGTH);
+ ctx->info.product = cpu_to_le64(xdbc.string_dma + XDBC_MAX_STRING_LENGTH * 2);
+ ctx->info.serial = cpu_to_le64(xdbc.string_dma + XDBC_MAX_STRING_LENGTH * 3);
+ ctx->info.length = cpu_to_le32(string_length);
+
+ /* Populate bulk out endpoint context: */
+ max_burst = DEBUG_MAX_BURST(readl(&xdbc.xdbc_reg->control));
+ ep_out = (struct xdbc_ep_context *)&ctx->out;
+
+ ep_out->ep_info1 = 0;
+ ep_out->ep_info2 = cpu_to_le32(EP_TYPE(BULK_OUT_EP) | MAX_PACKET(1024) | MAX_BURST(max_burst));
+ ep_out->deq = cpu_to_le64(xdbc.out_seg.dma | xdbc.out_ring.cycle_state);
+
+ /* Populate bulk in endpoint context: */
+ ep_in = (struct xdbc_ep_context *)&ctx->in;
+
+ ep_in->ep_info1 = 0;
+ ep_in->ep_info2 = cpu_to_le32(EP_TYPE(BULK_OUT_EP) | MAX_PACKET(1024) | MAX_BURST(max_burst));
+ ep_in->deq = cpu_to_le64(xdbc.in_seg.dma | xdbc.in_ring.cycle_state);
+
+ /* Set DbC context and info registers: */
+ xdbc_write64(xdbc.dbcc_dma, &xdbc.xdbc_reg->dccp);
+
+ dev_info = cpu_to_le32((XDBC_VENDOR_ID << 16) | XDBC_PROTOCOL);
+ writel(dev_info, &xdbc.xdbc_reg->devinfo1);
+
+ dev_info = cpu_to_le32((XDBC_DEVICE_REV << 16) | XDBC_PRODUCT_ID);
+ writel(dev_info, &xdbc.xdbc_reg->devinfo2);
+
+ xdbc.in_buf = xdbc.out_buf + XDBC_MAX_PACKET;
+ xdbc.in_dma = xdbc.out_dma + XDBC_MAX_PACKET;
+}
+
+static void xdbc_do_reset_debug_port(u32 id, u32 count)
+{
+ void __iomem *ops_reg;
+ void __iomem *portsc;
+ u32 val, cap_length;
+ int i;
+
+ cap_length = readl(xdbc.xhci_base) & 0xff;
+ ops_reg = xdbc.xhci_base + cap_length;
+
+ id--;
+ for (i = id; i < (id + count); i++) {
+ portsc = ops_reg + 0x400 + i * 0x10;
+ val = readl(portsc);
+ if (!(val & PORT_CONNECT))
+ writel(val | PORT_RESET, portsc);
+ }
+}
+
+static void xdbc_reset_debug_port(void)
+{
+ u32 val, port_offset, port_count;
+ int offset = 0;
+
+ do {
+ offset = xhci_find_next_ext_cap(xdbc.xhci_base, offset, XHCI_EXT_CAPS_PROTOCOL);
+ if (!offset)
+ break;
+
+ val = readl(xdbc.xhci_base + offset);
+ if (XHCI_EXT_PORT_MAJOR(val) != 0x3)
+ continue;
+
+ val = readl(xdbc.xhci_base + offset + 8);
+ port_offset = XHCI_EXT_PORT_OFF(val);
+ port_count = XHCI_EXT_PORT_COUNT(val);
+
+ xdbc_do_reset_debug_port(port_offset, port_count);
+ } while (1);
+}
+
+static void
+xdbc_queue_trb(struct xdbc_ring *ring, u32 field1, u32 field2, u32 field3, u32 field4)
+{
+ struct xdbc_trb *trb, *link_trb;
+
+ trb = ring->enqueue;
+ trb->field[0] = cpu_to_le32(field1);
+ trb->field[1] = cpu_to_le32(field2);
+ trb->field[2] = cpu_to_le32(field3);
+ trb->field[3] = cpu_to_le32(field4);
+
+ ++(ring->enqueue);
+ if (ring->enqueue >= &ring->segment->trbs[TRBS_PER_SEGMENT - 1]) {
+ link_trb = ring->enqueue;
+ if (ring->cycle_state)
+ link_trb->field[3] |= cpu_to_le32(TRB_CYCLE);
+ else
+ link_trb->field[3] &= cpu_to_le32(~TRB_CYCLE);
+
+ ring->enqueue = ring->segment->trbs;
+ ring->cycle_state ^= 1;
+ }
+}
+
+static void xdbc_ring_doorbell(int target)
+{
+ writel(DOOR_BELL_TARGET(target), &xdbc.xdbc_reg->doorbell);
+}
+
+static int xdbc_start(void)
+{
+ u32 ctrl, status;
+ int ret;
+
+ ctrl = readl(&xdbc.xdbc_reg->control);
+ writel(ctrl | CTRL_DBC_ENABLE | CTRL_PORT_ENABLE, &xdbc.xdbc_reg->control);
+ ret = handshake(&xdbc.xdbc_reg->control, CTRL_DBC_ENABLE, CTRL_DBC_ENABLE, 100000, 100);
+ if (ret) {
+ xdbc_trace("failed to initialize hardware\n");
+ return ret;
+ }
+
+ /* Reset port to avoid bus hang: */
+ if (xdbc.vendor == PCI_VENDOR_ID_INTEL)
+ xdbc_reset_debug_port();
+
+ /* Wait for port connection: */
+ ret = handshake(&xdbc.xdbc_reg->portsc, PORTSC_CONN_STATUS, PORTSC_CONN_STATUS, 5000000, 100);
+ if (ret) {
+ xdbc_trace("waiting for connection timed out\n");
+ return ret;
+ }
+
+ /* Wait for debug device to be configured: */
+ ret = handshake(&xdbc.xdbc_reg->control, CTRL_DBC_RUN, CTRL_DBC_RUN, 5000000, 100);
+ if (ret) {
+ xdbc_trace("waiting for device configuration timed out\n");
+ return ret;
+ }
+
+ /* Check port number: */
+ status = readl(&xdbc.xdbc_reg->status);
+ if (!DCST_DEBUG_PORT(status)) {
+ xdbc_trace("invalid root hub port number\n");
+ return -ENODEV;
+ }
+
+ xdbc.port_number = DCST_DEBUG_PORT(status);
+
+ xdbc_trace("DbC is running now, control 0x%08x port ID %d\n",
+ readl(&xdbc.xdbc_reg->control), xdbc.port_number);
+
+ return 0;
+}
+
+static int xdbc_bulk_transfer(void *data, int size, bool read)
+{
+ struct xdbc_ring *ring;
+ struct xdbc_trb *trb;
+ u32 length, control;
+ u32 cycle;
+ u64 addr;
+
+ if (size > XDBC_MAX_PACKET) {
+ xdbc_trace("bad parameter, size %d\n", size);
+ return -EINVAL;
+ }
+
+ if (!(xdbc.flags & XDBC_FLAGS_INITIALIZED) ||
+ !(xdbc.flags & XDBC_FLAGS_CONFIGURED) ||
+ (!read && (xdbc.flags & XDBC_FLAGS_OUT_STALL)) ||
+ (read && (xdbc.flags & XDBC_FLAGS_IN_STALL))) {
+
+ xdbc_trace("connection not ready, flags %08x\n", xdbc.flags);
+ return -EIO;
+ }
+
+ ring = (read ? &xdbc.in_ring : &xdbc.out_ring);
+ trb = ring->enqueue;
+ cycle = ring->cycle_state;
+ length = TRB_LEN(size);
+ control = TRB_TYPE(TRB_NORMAL) | TRB_IOC;
+
+ if (cycle)
+ control &= cpu_to_le32(~TRB_CYCLE);
+ else
+ control |= cpu_to_le32(TRB_CYCLE);
+
+ if (read) {
+ memset(xdbc.in_buf, 0, XDBC_MAX_PACKET);
+ addr = xdbc.in_dma;
+ xdbc.flags |= XDBC_FLAGS_IN_PROCESS;
+ } else {
+ memset(xdbc.out_buf, 0, XDBC_MAX_PACKET);
+ memcpy(xdbc.out_buf, data, size);
+ addr = xdbc.out_dma;
+ xdbc.flags |= XDBC_FLAGS_OUT_PROCESS;
+ }
+
+ xdbc_queue_trb(ring, lower_32_bits(addr), upper_32_bits(addr), length, control);
+
+ /*
+ * Add a barrier between writes of trb fields and flipping
+ * the cycle bit:
+ */
+ wmb();
+ if (cycle)
+ trb->field[3] |= cpu_to_le32(cycle);
+ else
+ trb->field[3] &= cpu_to_le32(~TRB_CYCLE);
+
+ xdbc_ring_doorbell(read ? IN_EP_DOORBELL : OUT_EP_DOORBELL);
+
+ return size;
+}
+
+static int xdbc_handle_external_reset(void)
+{
+ int ret = 0;
+
+ xdbc.flags = 0;
+ writel(0, &xdbc.xdbc_reg->control);
+ ret = handshake(&xdbc.xdbc_reg->control, CTRL_DBC_ENABLE, 0, 100000, 10);
+ if (ret)
+ goto reset_out;
+
+ xdbc_mem_init();
+
+ mmiowb();
+
+ ret = xdbc_start();
+ if (ret < 0)
+ goto reset_out;
+
+ xdbc_trace("dbc recovered\n");
+
+ xdbc.flags |= XDBC_FLAGS_INITIALIZED | XDBC_FLAGS_CONFIGURED;
+
+ xdbc_bulk_transfer(NULL, XDBC_MAX_PACKET, true);
+
+ return 0;
+
+reset_out:
+ xdbc_trace("failed to recover from external reset\n");
+ return ret;
+}
+
+static int __init xdbc_early_setup(void)
+{
+ int ret;
+
+ writel(0, &xdbc.xdbc_reg->control);
+ ret = handshake(&xdbc.xdbc_reg->control, CTRL_DBC_ENABLE, 0, 100000, 100);
+ if (ret)
+ return ret;
+
+ /* Allocate the table page: */
+ xdbc.table_base = xdbc_get_page(&xdbc.table_dma);
+ if (!xdbc.table_base)
+ return -ENOMEM;
+
+ /* Get and store the transfer buffer: */
+ xdbc.out_buf = xdbc_get_page(&xdbc.out_dma);
+ if (!xdbc.out_buf)
+ return -ENOMEM;
+
+ /* Allocate the event ring: */
+ ret = xdbc_alloc_ring(&xdbc.evt_seg, &xdbc.evt_ring);
+ if (ret < 0)
+ return ret;
+
+ /* Allocate IN/OUT endpoint transfer rings: */
+ ret = xdbc_alloc_ring(&xdbc.in_seg, &xdbc.in_ring);
+ if (ret < 0)
+ return ret;
+
+ ret = xdbc_alloc_ring(&xdbc.out_seg, &xdbc.out_ring);
+ if (ret < 0)
+ return ret;
+
+ xdbc_mem_init();
+
+ mmiowb();
+
+ ret = xdbc_start();
+ if (ret < 0) {
+ writel(0, &xdbc.xdbc_reg->control);
+ return ret;
+ }
+
+ xdbc.flags |= XDBC_FLAGS_INITIALIZED | XDBC_FLAGS_CONFIGURED;
+
+ xdbc_bulk_transfer(NULL, XDBC_MAX_PACKET, true);
+
+ return 0;
+}
+
+int __init early_xdbc_parse_parameter(char *s)
+{
+ unsigned long dbgp_num = 0;
+ u32 bus, dev, func, offset;
+ int ret;
+
+ if (!early_pci_allowed())
+ return -EPERM;
+
+ if (strstr(s, "keep"))
+ early_console_keep = true;
+
+ if (xdbc.xdbc_reg)
+ return 0;
+
+ if (*s && kstrtoul(s, 0, &dbgp_num))
+ dbgp_num = 0;
+
+ pr_notice("dbgp_num: %lu\n", dbgp_num);
+
+ /* Locate the host controller: */
+ ret = xdbc_find_dbgp(dbgp_num, &bus, &dev, &func);
+ if (ret) {
+ pr_notice("failed to locate xhci host\n");
+ return -ENODEV;
+ }
+
+ xdbc.vendor = read_pci_config_16(bus, dev, func, PCI_VENDOR_ID);
+ xdbc.device = read_pci_config_16(bus, dev, func, PCI_DEVICE_ID);
+ xdbc.bus = bus;
+ xdbc.dev = dev;
+ xdbc.func = func;
+
+ /* Map the IO memory: */
+ xdbc.xhci_base = xdbc_map_pci_mmio(bus, dev, func);
+ if (!xdbc.xhci_base)
+ return -EINVAL;
+
+ /* Locate DbC registers: */
+ offset = xhci_find_next_ext_cap(xdbc.xhci_base, 0, XHCI_EXT_CAPS_DEBUG);
+ if (!offset) {
+ pr_notice("xhci host doesn't support debug capability\n");
+ early_iounmap(xdbc.xhci_base, xdbc.xhci_length);
+ xdbc.xhci_base = NULL;
+ xdbc.xhci_length = 0;
+
+ return -ENODEV;
+ }
+ xdbc.xdbc_reg = (struct xdbc_regs __iomem *)(xdbc.xhci_base + offset);
+
+ return 0;
+}
+
+int __init early_xdbc_setup_hardware(void)
+{
+ int ret;
+
+ if (!xdbc.xdbc_reg)
+ return -ENODEV;
+
+ xdbc_bios_handoff();
+
+ raw_spin_lock_init(&xdbc.lock);
+
+ ret = xdbc_early_setup();
+ if (ret) {
+ pr_notice("failed to setup the connection to host\n");
+
+ xdbc_free_ring(&xdbc.evt_ring);
+ xdbc_free_ring(&xdbc.out_ring);
+ xdbc_free_ring(&xdbc.in_ring);
+
+ if (xdbc.table_dma)
+ free_bootmem(xdbc.table_dma, PAGE_SIZE);
+
+ if (xdbc.out_dma)
+ free_bootmem(xdbc.out_dma, PAGE_SIZE);
+
+ xdbc.table_base = NULL;
+ xdbc.out_buf = NULL;
+ }
+
+ return ret;
+}
+
+static void xdbc_handle_port_status(struct xdbc_trb *evt_trb)
+{
+ u32 port_reg;
+
+ port_reg = readl(&xdbc.xdbc_reg->portsc);
+ if (port_reg & PORTSC_CONN_CHANGE) {
+ xdbc_trace("connect status change event\n");
+
+ /* Check whether cable unplugged: */
+ if (!(port_reg & PORTSC_CONN_STATUS)) {
+ xdbc.flags = 0;
+ xdbc_trace("cable unplugged\n");
+ }
+ }
+
+ if (port_reg & PORTSC_RESET_CHANGE)
+ xdbc_trace("port reset change event\n");
+
+ if (port_reg & PORTSC_LINK_CHANGE)
+ xdbc_trace("port link status change event\n");
+
+ if (port_reg & PORTSC_CONFIG_CHANGE)
+ xdbc_trace("config error change\n");
+
+ /* Write back the value to clear RW1C bits: */
+ writel(port_reg, &xdbc.xdbc_reg->portsc);
+}
+
+static void xdbc_handle_tx_event(struct xdbc_trb *evt_trb)
+{
+ size_t remain_length;
+ u32 comp_code;
+ int ep_id;
+
+ comp_code = GET_COMP_CODE(le32_to_cpu(evt_trb->field[2]));
+ remain_length = EVENT_TRB_LEN(le32_to_cpu(evt_trb->field[2]));
+ ep_id = TRB_TO_EP_ID(le32_to_cpu(evt_trb->field[3]));
+
+ switch (comp_code) {
+ case COMP_SUCCESS:
+ remain_length = 0;
+ case COMP_SHORT_PACKET:
+ break;
+ case COMP_TRB_ERROR:
+ case COMP_BABBLE_DETECTED_ERROR:
+ case COMP_USB_TRANSACTION_ERROR:
+ case COMP_STALL_ERROR:
+ default:
+ if (ep_id == XDBC_EPID_OUT)
+ xdbc.flags |= XDBC_FLAGS_OUT_STALL;
+ if (ep_id == XDBC_EPID_IN)
+ xdbc.flags |= XDBC_FLAGS_IN_STALL;
+
+ xdbc_trace("endpoint %d stalled\n", ep_id);
+ break;
+ }
+
+ if (ep_id == XDBC_EPID_IN) {
+ xdbc.flags &= ~XDBC_FLAGS_IN_PROCESS;
+ xdbc_bulk_transfer(NULL, XDBC_MAX_PACKET, true);
+ } else if (ep_id == XDBC_EPID_OUT) {
+ xdbc.flags &= ~XDBC_FLAGS_OUT_PROCESS;
+ } else {
+ xdbc_trace("invalid endpoint id %d\n", ep_id);
+ }
+}
+
+static void xdbc_handle_events(void)
+{
+ struct xdbc_trb *evt_trb;
+ bool update_erdp = false;
+ u32 reg;
+ u8 cmd;
+
+ cmd = read_pci_config_byte(xdbc.bus, xdbc.dev, xdbc.func, PCI_COMMAND);
+ if (!(cmd & PCI_COMMAND_MASTER)) {
+ cmd |= PCI_COMMAND_MASTER | PCI_COMMAND_MEMORY;
+ write_pci_config_byte(xdbc.bus, xdbc.dev, xdbc.func, PCI_COMMAND, cmd);
+ }
+
+ if (!(xdbc.flags & XDBC_FLAGS_INITIALIZED))
+ return;
+
+ /* Handle external reset events: */
+ reg = readl(&xdbc.xdbc_reg->control);
+ if (!(reg & CTRL_DBC_ENABLE)) {
+ if (xdbc_handle_external_reset()) {
+ xdbc_trace("failed to recover connection\n");
+ return;
+ }
+ }
+
+ /* Handle configure-exit event: */
+ reg = readl(&xdbc.xdbc_reg->control);
+ if (reg & CTRL_DBC_RUN_CHANGE) {
+ writel(reg, &xdbc.xdbc_reg->control);
+ if (reg & CTRL_DBC_RUN)
+ xdbc.flags |= XDBC_FLAGS_CONFIGURED;
+ else
+ xdbc.flags &= ~XDBC_FLAGS_CONFIGURED;
+ }
+
+ /* Handle endpoint stall event: */
+ reg = readl(&xdbc.xdbc_reg->control);
+ if (reg & CTRL_HALT_IN_TR) {
+ xdbc.flags |= XDBC_FLAGS_IN_STALL;
+ } else {
+ xdbc.flags &= ~XDBC_FLAGS_IN_STALL;
+ if (!(xdbc.flags & XDBC_FLAGS_IN_PROCESS))
+ xdbc_bulk_transfer(NULL, XDBC_MAX_PACKET, true);
+ }
+
+ if (reg & CTRL_HALT_OUT_TR)
+ xdbc.flags |= XDBC_FLAGS_OUT_STALL;
+ else
+ xdbc.flags &= ~XDBC_FLAGS_OUT_STALL;
+
+ /* Handle the events in the event ring: */
+ evt_trb = xdbc.evt_ring.dequeue;
+ while ((le32_to_cpu(evt_trb->field[3]) & TRB_CYCLE) == xdbc.evt_ring.cycle_state) {
+ /*
+ * Add a barrier between reading the cycle flag and any
+ * reads of the event's flags/data below:
+ */
+ rmb();
+
+ switch ((le32_to_cpu(evt_trb->field[3]) & TRB_TYPE_BITMASK)) {
+ case TRB_TYPE(TRB_PORT_STATUS):
+ xdbc_handle_port_status(evt_trb);
+ break;
+ case TRB_TYPE(TRB_TRANSFER):
+ xdbc_handle_tx_event(evt_trb);
+ break;
+ default:
+ break;
+ }
+
+ ++(xdbc.evt_ring.dequeue);
+ if (xdbc.evt_ring.dequeue == &xdbc.evt_seg.trbs[TRBS_PER_SEGMENT]) {
+ xdbc.evt_ring.dequeue = xdbc.evt_seg.trbs;
+ xdbc.evt_ring.cycle_state ^= 1;
+ }
+
+ evt_trb = xdbc.evt_ring.dequeue;
+ update_erdp = true;
+ }
+
+ /* Update event ring dequeue pointer: */
+ if (update_erdp)
+ xdbc_write64(__pa(xdbc.evt_ring.dequeue), &xdbc.xdbc_reg->erdp);
+}
+
+static int xdbc_bulk_write(const char *bytes, int size)
+{
+ int ret, timeout = 0;
+ unsigned long flags;
+
+retry:
+ if (in_nmi()) {
+ if (!raw_spin_trylock_irqsave(&xdbc.lock, flags))
+ return -EAGAIN;
+ } else {
+ raw_spin_lock_irqsave(&xdbc.lock, flags);
+ }
+
+ xdbc_handle_events();
+
+ /* Check completion of the previous request: */
+ if ((xdbc.flags & XDBC_FLAGS_OUT_PROCESS) && (timeout < 2000000)) {
+ raw_spin_unlock_irqrestore(&xdbc.lock, flags);
+ udelay(100);
+ timeout += 100;
+ goto retry;
+ }
+
+ if (xdbc.flags & XDBC_FLAGS_OUT_PROCESS) {
+ raw_spin_unlock_irqrestore(&xdbc.lock, flags);
+ xdbc_trace("previous transfer not completed yet\n");
+
+ return -ETIMEDOUT;
+ }
+
+ ret = xdbc_bulk_transfer((void *)bytes, size, false);
+ raw_spin_unlock_irqrestore(&xdbc.lock, flags);
+
+ return ret;
+}
+
+static void early_xdbc_write(struct console *con, const char *str, u32 n)
+{
+ static char buf[XDBC_MAX_PACKET];
+ int chunk, ret;
+ int use_cr = 0;
+
+ if (!xdbc.xdbc_reg)
+ return;
+ memset(buf, 0, XDBC_MAX_PACKET);
+ while (n > 0) {
+ for (chunk = 0; chunk < XDBC_MAX_PACKET && n > 0; str++, chunk++, n--) {
+
+ if (!use_cr && *str == '\n') {
+ use_cr = 1;
+ buf[chunk] = '\r';
+ str--;
+ n++;
+ continue;
+ }
+
+ if (use_cr)
+ use_cr = 0;
+ buf[chunk] = *str;
+ }
+
+ if (chunk > 0) {
+ ret = xdbc_bulk_write(buf, chunk);
+ if (ret < 0)
+ xdbc_trace("missed message {%s}\n", buf);
+ }
+ }
+}
+
+static struct console early_xdbc_console = {
+ .name = "earlyxdbc",
+ .write = early_xdbc_write,
+ .flags = CON_PRINTBUFFER,
+ .index = -1,
+};
+
+void __init early_xdbc_register_console(void)
+{
+ if (early_console)
+ return;
+
+ early_console = &early_xdbc_console;
+ if (early_console_keep)
+ early_console->flags &= ~CON_BOOT;
+ else
+ early_console->flags |= CON_BOOT;
+ register_console(early_console);
+}
+
+static void xdbc_unregister_console(void)
+{
+ if (early_xdbc_console.flags & CON_ENABLED)
+ unregister_console(&early_xdbc_console);
+}
+
+static int xdbc_scrub_function(void *ptr)
+{
+ unsigned long flags;
+
+ while (true) {
+ raw_spin_lock_irqsave(&xdbc.lock, flags);
+ xdbc_handle_events();
+
+ if (!(xdbc.flags & XDBC_FLAGS_INITIALIZED)) {
+ raw_spin_unlock_irqrestore(&xdbc.lock, flags);
+ break;
+ }
+
+ raw_spin_unlock_irqrestore(&xdbc.lock, flags);
+ schedule_timeout_interruptible(1);
+ }
+
+ xdbc_unregister_console();
+ writel(0, &xdbc.xdbc_reg->control);
+ xdbc_trace("dbc scrub function exits\n");
+
+ return 0;
+}
+
+static int __init xdbc_init(void)
+{
+ unsigned long flags;
+ void __iomem *base;
+ int ret = 0;
+ u32 offset;
+
+ if (!(xdbc.flags & XDBC_FLAGS_INITIALIZED))
+ return 0;
+
+ /*
+ * It's time to shut down the DbC, so that the debug
+ * port can be reused by the host controller:
+ */
+ if (early_xdbc_console.index == -1 ||
+ (early_xdbc_console.flags & CON_BOOT)) {
+ xdbc_trace("hardware not used anymore\n");
+ goto free_and_quit;
+ }
+
+ base = ioremap_nocache(xdbc.xhci_start, xdbc.xhci_length);
+ if (!base) {
+ xdbc_trace("failed to remap the io address\n");
+ ret = -ENOMEM;
+ goto free_and_quit;
+ }
+
+ raw_spin_lock_irqsave(&xdbc.lock, flags);
+ early_iounmap(xdbc.xhci_base, xdbc.xhci_length);
+ xdbc.xhci_base = base;
+ offset = xhci_find_next_ext_cap(xdbc.xhci_base, 0, XHCI_EXT_CAPS_DEBUG);
+ xdbc.xdbc_reg = (struct xdbc_regs __iomem *)(xdbc.xhci_base + offset);
+ raw_spin_unlock_irqrestore(&xdbc.lock, flags);
+
+ kthread_run(xdbc_scrub_function, NULL, "%s", "xdbc");
+
+ return 0;
+
+free_and_quit:
+ xdbc_free_ring(&xdbc.evt_ring);
+ xdbc_free_ring(&xdbc.out_ring);
+ xdbc_free_ring(&xdbc.in_ring);
+ free_bootmem(xdbc.table_dma, PAGE_SIZE);
+ free_bootmem(xdbc.out_dma, PAGE_SIZE);
+ writel(0, &xdbc.xdbc_reg->control);
+ early_iounmap(xdbc.xhci_base, xdbc.xhci_length);
+
+ return ret;
+}
+subsys_initcall(xdbc_init);
diff --git a/drivers/usb/early/xhci-dbc.h b/drivers/usb/early/xhci-dbc.h
new file mode 100644
index 0000000..2df0f6e
--- /dev/null
+++ b/drivers/usb/early/xhci-dbc.h
@@ -0,0 +1,211 @@
+/*
+ * xhci-dbc.h - xHCI debug capability early driver
+ *
+ * Copyright (C) 2016 Intel Corporation
+ *
+ * Author: Lu Baolu <baolu.lu@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __LINUX_XHCI_DBC_H
+#define __LINUX_XHCI_DBC_H
+
+#include <linux/types.h>
+#include <linux/usb/ch9.h>
+
+/*
+ * xHCI Debug Capability Register interfaces:
+ */
+struct xdbc_regs {
+ __le32 capability;
+ __le32 doorbell;
+ __le32 ersts; /* Event Ring Segment Table Size*/
+ __le32 __reserved_0; /* 0c~0f reserved bits */
+ __le64 erstba; /* Event Ring Segment Table Base Address */
+ __le64 erdp; /* Event Ring Dequeue Pointer */
+ __le32 control;
+ __le32 status;
+ __le32 portsc; /* Port status and control */
+ __le32 __reserved_1; /* 2b~28 reserved bits */
+ __le64 dccp; /* Debug Capability Context Pointer */
+ __le32 devinfo1; /* Device Descriptor Info Register 1 */
+ __le32 devinfo2; /* Device Descriptor Info Register 2 */
+};
+
+#define DEBUG_MAX_BURST(p) (((p) >> 16) & 0xff)
+
+#define CTRL_DBC_RUN BIT(0)
+#define CTRL_PORT_ENABLE BIT(1)
+#define CTRL_HALT_OUT_TR BIT(2)
+#define CTRL_HALT_IN_TR BIT(3)
+#define CTRL_DBC_RUN_CHANGE BIT(4)
+#define CTRL_DBC_ENABLE BIT(31)
+
+#define DCST_DEBUG_PORT(p) (((p) >> 24) & 0xff)
+
+#define PORTSC_CONN_STATUS BIT(0)
+#define PORTSC_CONN_CHANGE BIT(17)
+#define PORTSC_RESET_CHANGE BIT(21)
+#define PORTSC_LINK_CHANGE BIT(22)
+#define PORTSC_CONFIG_CHANGE BIT(23)
+
+/*
+ * xHCI Debug Capability data structures:
+ */
+struct xdbc_trb {
+ __le32 field[4];
+};
+
+struct xdbc_erst_entry {
+ __le64 seg_addr;
+ __le32 seg_size;
+ __le32 __reserved_0;
+};
+
+struct xdbc_info_context {
+ __le64 string0;
+ __le64 manufacturer;
+ __le64 product;
+ __le64 serial;
+ __le32 length;
+ __le32 __reserved_0[7];
+};
+
+struct xdbc_ep_context {
+ __le32 ep_info1;
+ __le32 ep_info2;
+ __le64 deq;
+ __le32 tx_info;
+ __le32 __reserved_0[11];
+};
+
+struct xdbc_context {
+ struct xdbc_info_context info;
+ struct xdbc_ep_context out;
+ struct xdbc_ep_context in;
+};
+
+#define XDBC_INFO_CONTEXT_SIZE 48
+#define XDBC_MAX_STRING_LENGTH 64
+#define XDBC_STRING_MANUFACTURER "Linux"
+#define XDBC_STRING_PRODUCT "Remote GDB"
+#define XDBC_STRING_SERIAL "0001"
+
+struct xdbc_strings {
+ char string0[XDBC_MAX_STRING_LENGTH];
+ char manufacturer[XDBC_MAX_STRING_LENGTH];
+ char product[XDBC_MAX_STRING_LENGTH];
+ char serial[XDBC_MAX_STRING_LENGTH];
+};
+
+#define XDBC_PROTOCOL 1 /* GNU Remote Debug Command Set */
+#define XDBC_VENDOR_ID 0x1d6b /* Linux Foundation 0x1d6b */
+#define XDBC_PRODUCT_ID 0x0004 /* __le16 idProduct; device 0004 */
+#define XDBC_DEVICE_REV 0x0010 /* 0.10 */
+
+/*
+ * xHCI Debug Capability software state structures:
+ */
+struct xdbc_segment {
+ struct xdbc_trb *trbs;
+ dma_addr_t dma;
+};
+
+#define XDBC_TRBS_PER_SEGMENT 256
+
+struct xdbc_ring {
+ struct xdbc_segment *segment;
+ struct xdbc_trb *enqueue;
+ struct xdbc_trb *dequeue;
+ u32 cycle_state;
+};
+
+#define XDBC_EPID_OUT 2
+#define XDBC_EPID_IN 3
+
+struct xdbc_state {
+ u16 vendor;
+ u16 device;
+ u32 bus;
+ u32 dev;
+ u32 func;
+ void __iomem *xhci_base;
+ u64 xhci_start;
+ size_t xhci_length;
+ int port_number;
+
+ /* DbC register base */
+ struct xdbc_regs __iomem *xdbc_reg;
+
+ /* DbC table page */
+ dma_addr_t table_dma;
+ void *table_base;
+
+ /* event ring segment table */
+ dma_addr_t erst_dma;
+ size_t erst_size;
+ void *erst_base;
+
+ /* event ring segments */
+ struct xdbc_ring evt_ring;
+ struct xdbc_segment evt_seg;
+
+ /* debug capability contexts */
+ dma_addr_t dbcc_dma;
+ size_t dbcc_size;
+ void *dbcc_base;
+
+ /* descriptor strings */
+ dma_addr_t string_dma;
+ size_t string_size;
+ void *string_base;
+
+ /* bulk OUT endpoint */
+ struct xdbc_ring out_ring;
+ struct xdbc_segment out_seg;
+ void *out_buf;
+ dma_addr_t out_dma;
+
+ /* bulk IN endpoint */
+ struct xdbc_ring in_ring;
+ struct xdbc_segment in_seg;
+ void *in_buf;
+ dma_addr_t in_dma;
+
+ u32 flags;
+
+ /* spinlock for early_xdbc_write() reentrancy */
+ raw_spinlock_t lock;
+};
+
+#define XDBC_PCI_MAX_BUSES 256
+#define XDBC_PCI_MAX_DEVICES 32
+#define XDBC_PCI_MAX_FUNCTION 8
+
+#define XDBC_TABLE_ENTRY_SIZE 64
+#define XDBC_ERST_ENTRY_NUM 1
+#define XDBC_DBCC_ENTRY_NUM 3
+#define XDBC_STRING_ENTRY_NUM 4
+
+/* Bits definitions for xdbc_state.flags: */
+#define XDBC_FLAGS_INITIALIZED BIT(0)
+#define XDBC_FLAGS_IN_STALL BIT(1)
+#define XDBC_FLAGS_OUT_STALL BIT(2)
+#define XDBC_FLAGS_IN_PROCESS BIT(3)
+#define XDBC_FLAGS_OUT_PROCESS BIT(4)
+#define XDBC_FLAGS_CONFIGURED BIT(5)
+
+#define XDBC_MAX_PACKET 1024
+
+/* Door bell target: */
+#define OUT_EP_DOORBELL 0
+#define IN_EP_DOORBELL 1
+#define DOOR_BELL_TARGET(p) (((p) & 0xff) << 8)
+
+#define xdbc_read64(regs) xhci_read_64(NULL, (regs))
+#define xdbc_write64(val, regs) xhci_write_64(NULL, (val), (regs))
+
+#endif /* __LINUX_XHCI_DBC_H */
diff --git a/drivers/usb/gadget/function/f_ncm.c b/drivers/usb/gadget/function/f_ncm.c
index 224717e6..864819f 100644
--- a/drivers/usb/gadget/function/f_ncm.c
+++ b/drivers/usb/gadget/function/f_ncm.c
@@ -16,6 +16,7 @@
*/
#include <linux/kernel.h>
+#include <linux/interrupt.h>
#include <linux/module.h>
#include <linux/device.h>
#include <linux/etherdevice.h>
diff --git a/drivers/usb/serial/usb_debug.c b/drivers/usb/serial/usb_debug.c
index ca2fa5b..92f7e5c 100644
--- a/drivers/usb/serial/usb_debug.c
+++ b/drivers/usb/serial/usb_debug.c
@@ -32,7 +32,18 @@
{ USB_DEVICE(0x0525, 0x127a) },
{ },
};
-MODULE_DEVICE_TABLE(usb, id_table);
+
+static const struct usb_device_id dbc_id_table[] = {
+ { USB_DEVICE(0x1d6b, 0x0004) },
+ { },
+};
+
+static const struct usb_device_id id_table_combined[] = {
+ { USB_DEVICE(0x0525, 0x127a) },
+ { USB_DEVICE(0x1d6b, 0x0004) },
+ { },
+};
+MODULE_DEVICE_TABLE(usb, id_table_combined);
/* This HW really does not support a serial break, so one will be
* emulated when ever the break state is set to true.
@@ -71,9 +82,20 @@
.process_read_urb = usb_debug_process_read_urb,
};
-static struct usb_serial_driver * const serial_drivers[] = {
- &debug_device, NULL
+static struct usb_serial_driver dbc_device = {
+ .driver = {
+ .owner = THIS_MODULE,
+ .name = "xhci_dbc",
+ },
+ .id_table = dbc_id_table,
+ .num_ports = 1,
+ .break_ctl = usb_debug_break_ctl,
+ .process_read_urb = usb_debug_process_read_urb,
};
-module_usb_serial_driver(serial_drivers, id_table);
+static struct usb_serial_driver * const serial_drivers[] = {
+ &debug_device, &dbc_device, NULL
+};
+
+module_usb_serial_driver(serial_drivers, id_table_combined);
MODULE_LICENSE("GPL");
diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
index 44eed8e..d939ac1 100644
--- a/drivers/vhost/vsock.c
+++ b/drivers/vhost/vsock.c
@@ -176,6 +176,11 @@
restart_tx = true;
}
+ /* Deliver to monitoring devices all correctly transmitted
+ * packets.
+ */
+ virtio_transport_deliver_tap_pkt(pkt);
+
virtio_transport_free_pkt(pkt);
}
if (added)
@@ -383,6 +388,9 @@
len = pkt->len;
+ /* Deliver to monitoring devices all received packets */
+ virtio_transport_deliver_tap_pkt(pkt);
+
/* Only accept correctly addressed packets */
if (le64_to_cpu(pkt->hdr.src_cid) == vsock->guest_cid)
virtio_transport_recv_pkt(pkt);
diff --git a/fs/9p/v9fs.c b/fs/9p/v9fs.c
index a89f3cf..c202930 100644
--- a/fs/9p/v9fs.c
+++ b/fs/9p/v9fs.c
@@ -333,10 +333,6 @@
goto err_names;
init_rwsem(&v9ses->rename_sem);
- rc = bdi_setup_and_register(&v9ses->bdi, "9p");
- if (rc)
- goto err_names;
-
v9ses->uid = INVALID_UID;
v9ses->dfltuid = V9FS_DEFUID;
v9ses->dfltgid = V9FS_DEFGID;
@@ -345,7 +341,7 @@
if (IS_ERR(v9ses->clnt)) {
rc = PTR_ERR(v9ses->clnt);
p9_debug(P9_DEBUG_ERROR, "problem initializing 9p client\n");
- goto err_bdi;
+ goto err_names;
}
v9ses->flags = V9FS_ACCESS_USER;
@@ -415,8 +411,6 @@
err_clnt:
p9_client_destroy(v9ses->clnt);
-err_bdi:
- bdi_destroy(&v9ses->bdi);
err_names:
kfree(v9ses->uname);
kfree(v9ses->aname);
@@ -445,8 +439,6 @@
kfree(v9ses->uname);
kfree(v9ses->aname);
- bdi_destroy(&v9ses->bdi);
-
spin_lock(&v9fs_sessionlist_lock);
list_del(&v9ses->slist);
spin_unlock(&v9fs_sessionlist_lock);
diff --git a/fs/9p/v9fs.h b/fs/9p/v9fs.h
index 443d12e..76eaf49 100644
--- a/fs/9p/v9fs.h
+++ b/fs/9p/v9fs.h
@@ -114,7 +114,6 @@
kuid_t uid; /* if ACCESS_SINGLE, the uid that has access */
struct p9_client *clnt; /* 9p client */
struct list_head slist; /* list of sessions registered with v9fs */
- struct backing_dev_info bdi;
struct rw_semaphore rename_sem;
};
diff --git a/fs/9p/vfs_super.c b/fs/9p/vfs_super.c
index de3ed86..a0965fb 100644
--- a/fs/9p/vfs_super.c
+++ b/fs/9p/vfs_super.c
@@ -72,10 +72,12 @@
*
*/
-static void
+static int
v9fs_fill_super(struct super_block *sb, struct v9fs_session_info *v9ses,
int flags, void *data)
{
+ int ret;
+
sb->s_maxbytes = MAX_LFS_FILESIZE;
sb->s_blocksize_bits = fls(v9ses->maxdata - 1);
sb->s_blocksize = 1 << sb->s_blocksize_bits;
@@ -85,7 +87,11 @@
sb->s_xattr = v9fs_xattr_handlers;
} else
sb->s_op = &v9fs_super_ops;
- sb->s_bdi = &v9ses->bdi;
+
+ ret = super_setup_bdi(sb);
+ if (ret)
+ return ret;
+
if (v9ses->cache)
sb->s_bdi->ra_pages = (VM_MAX_READAHEAD * 1024)/PAGE_SIZE;
@@ -99,6 +105,7 @@
#endif
save_mount_options(sb, data);
+ return 0;
}
/**
@@ -138,7 +145,9 @@
retval = PTR_ERR(sb);
goto clunk_fid;
}
- v9fs_fill_super(sb, v9ses, flags, data);
+ retval = v9fs_fill_super(sb, v9ses, flags, data);
+ if (retval)
+ goto release_sb;
if (v9ses->cache == CACHE_LOOSE || v9ses->cache == CACHE_FSCACHE)
sb->s_d_op = &v9fs_cached_dentry_operations;
diff --git a/fs/afs/internal.h b/fs/afs/internal.h
index a690136..3936729 100644
--- a/fs/afs/internal.h
+++ b/fs/afs/internal.h
@@ -318,7 +318,6 @@
unsigned short rjservers; /* number of servers discarded due to -ENOMEDIUM */
struct afs_server *servers[8]; /* servers on which volume resides (ordered) */
struct rw_semaphore server_sem; /* lock for accessing current server */
- struct backing_dev_info bdi;
};
/*
diff --git a/fs/afs/rxrpc.c b/fs/afs/rxrpc.c
index 8f76b13..d5990eb 100644
--- a/fs/afs/rxrpc.c
+++ b/fs/afs/rxrpc.c
@@ -419,7 +419,7 @@
call->state = AFS_CALL_COMPLETE;
if (ret != -ECONNABORTED) {
rxrpc_kernel_abort_call(afs_socket, rxcall, RX_USER_ABORT,
- -ret, "KSD");
+ ret, "KSD");
} else {
abort_code = 0;
offset = 0;
@@ -478,12 +478,12 @@
case -ENOTCONN:
abort_code = RX_CALL_DEAD;
rxrpc_kernel_abort_call(afs_socket, call->rxcall,
- abort_code, -ret, "KNC");
+ abort_code, ret, "KNC");
goto save_error;
case -ENOTSUPP:
abort_code = RXGEN_OPCODE;
rxrpc_kernel_abort_call(afs_socket, call->rxcall,
- abort_code, -ret, "KIV");
+ abort_code, ret, "KIV");
goto save_error;
case -ENODATA:
case -EBADMSG:
@@ -493,7 +493,7 @@
if (call->state != AFS_CALL_AWAIT_REPLY)
abort_code = RXGEN_SS_UNMARSHAL;
rxrpc_kernel_abort_call(afs_socket, call->rxcall,
- abort_code, EBADMSG, "KUM");
+ abort_code, -EBADMSG, "KUM");
goto save_error;
}
}
@@ -754,7 +754,7 @@
case -ENOMEM:
_debug("oom");
rxrpc_kernel_abort_call(afs_socket, call->rxcall,
- RX_USER_ABORT, ENOMEM, "KOO");
+ RX_USER_ABORT, -ENOMEM, "KOO");
default:
_leave(" [error]");
return;
@@ -792,7 +792,7 @@
if (n == -ENOMEM) {
_debug("oom");
rxrpc_kernel_abort_call(afs_socket, call->rxcall,
- RX_USER_ABORT, ENOMEM, "KOO");
+ RX_USER_ABORT, -ENOMEM, "KOO");
}
_leave(" [error]");
}
diff --git a/fs/afs/super.c b/fs/afs/super.c
index fbdb022..c79633e5 100644
--- a/fs/afs/super.c
+++ b/fs/afs/super.c
@@ -319,7 +319,10 @@
sb->s_blocksize_bits = PAGE_SHIFT;
sb->s_magic = AFS_FS_MAGIC;
sb->s_op = &afs_super_ops;
- sb->s_bdi = &as->volume->bdi;
+ ret = super_setup_bdi(sb);
+ if (ret)
+ return ret;
+ sb->s_bdi->ra_pages = VM_MAX_READAHEAD * 1024 / PAGE_SIZE;
strlcpy(sb->s_id, as->volume->vlocation->vldb.name, sizeof(sb->s_id));
/* allocate the root inode and dentry */
diff --git a/fs/afs/volume.c b/fs/afs/volume.c
index 546f9d0..db73d6d 100644
--- a/fs/afs/volume.c
+++ b/fs/afs/volume.c
@@ -106,11 +106,6 @@
volume->cell = params->cell;
volume->vid = vlocation->vldb.vid[params->type];
- volume->bdi.ra_pages = VM_MAX_READAHEAD*1024/PAGE_SIZE;
- ret = bdi_setup_and_register(&volume->bdi, "afs");
- if (ret)
- goto error_bdi;
-
init_rwsem(&volume->server_sem);
/* look up all the applicable server records */
@@ -156,8 +151,6 @@
return ERR_PTR(ret);
error_discard:
- bdi_destroy(&volume->bdi);
-error_bdi:
up_write(¶ms->cell->vl_sem);
for (loop = volume->nservers - 1; loop >= 0; loop--)
@@ -207,7 +200,6 @@
for (loop = volume->nservers - 1; loop >= 0; loop--)
afs_put_server(volume->servers[loop]);
- bdi_destroy(&volume->bdi);
kfree(volume);
_leave(" [destroyed]");
diff --git a/fs/autofs4/Kconfig b/fs/autofs4/Kconfig
index 1204d63..44727bf 100644
--- a/fs/autofs4/Kconfig
+++ b/fs/autofs4/Kconfig
@@ -7,7 +7,7 @@
automounter (amd), which is a pure user space daemon.
To use the automounter you need the user-space tools from
- <ftp://ftp.kernel.org/pub/linux/daemons/autofs/v4/>; you also
+ <https://www.kernel.org/pub/linux/daemons/autofs/v4/>; you also
want to answer Y to "NFS file system support", below.
To compile this support as a module, choose M here: the module will be
diff --git a/fs/block_dev.c b/fs/block_dev.c
index 2eca00e..9ccabe3 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -885,6 +885,8 @@
spin_lock(&bdev_lock);
list_del_init(&bdev->bd_list);
spin_unlock(&bdev_lock);
+ /* Detach inode from wb early as bdi_put() may free bdi->wb */
+ inode_detach_wb(inode);
if (bdev->bd_bdi != &noop_backing_dev_info) {
bdi_put(bdev->bd_bdi);
bdev->bd_bdi = &noop_backing_dev_info;
@@ -1451,7 +1453,6 @@
if (disk->fops->revalidate_disk)
ret = disk->fops->revalidate_disk(disk);
- blk_integrity_revalidate(disk);
bdev = bdget_disk(disk, 0);
if (!bdev)
return ret;
@@ -1556,8 +1557,6 @@
bdev->bd_disk = disk;
bdev->bd_queue = disk->queue;
bdev->bd_contains = bdev;
- if (bdev->bd_bdi == &noop_backing_dev_info)
- bdev->bd_bdi = bdi_get(disk->queue->backing_dev_info);
if (!partno) {
ret = -ENXIO;
@@ -1622,6 +1621,9 @@
}
bd_set_size(bdev, (loff_t)bdev->bd_part->nr_sects << 9);
}
+
+ if (bdev->bd_bdi == &noop_backing_dev_info)
+ bdev->bd_bdi = bdi_get(disk->queue->backing_dev_info);
} else {
if (bdev->bd_contains == bdev) {
ret = 0;
@@ -1653,8 +1655,6 @@
bdev->bd_disk = NULL;
bdev->bd_part = NULL;
bdev->bd_queue = NULL;
- bdi_put(bdev->bd_bdi);
- bdev->bd_bdi = &noop_backing_dev_info;
if (bdev != bdev->bd_contains)
__blkdev_put(bdev->bd_contains, mode, 1);
bdev->bd_contains = NULL;
@@ -1876,12 +1876,6 @@
kill_bdev(bdev);
bdev_write_inode(bdev);
- /*
- * Detaching bdev inode from its wb in __destroy_inode()
- * is too late: the queue which embeds its bdi (along with
- * root wb) can be gone as soon as we put_disk() below.
- */
- inode_detach_wb(bdev->bd_inode);
}
if (bdev->bd_contains == bdev) {
if (disk->fops->release)
@@ -2074,7 +2068,6 @@
loff_t len)
{
struct block_device *bdev = I_BDEV(bdev_file_inode(file));
- struct request_queue *q = bdev_get_queue(bdev);
struct address_space *mapping;
loff_t end = start + len - 1;
loff_t isize;
@@ -2110,18 +2103,13 @@
case FALLOC_FL_ZERO_RANGE:
case FALLOC_FL_ZERO_RANGE | FALLOC_FL_KEEP_SIZE:
error = blkdev_issue_zeroout(bdev, start >> 9, len >> 9,
- GFP_KERNEL, false);
+ GFP_KERNEL, BLKDEV_ZERO_NOUNMAP);
break;
case FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE:
- /* Only punch if the device can do zeroing discard. */
- if (!blk_queue_discard(q) || !q->limits.discard_zeroes_data)
- return -EOPNOTSUPP;
- error = blkdev_issue_discard(bdev, start >> 9, len >> 9,
- GFP_KERNEL, 0);
+ error = blkdev_issue_zeroout(bdev, start >> 9, len >> 9,
+ GFP_KERNEL, BLKDEV_ZERO_NOFALLBACK);
break;
case FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE | FALLOC_FL_NO_HIDE_STALE:
- if (!blk_queue_discard(q))
- return -EOPNOTSUPP;
error = blkdev_issue_discard(bdev, start >> 9, len >> 9,
GFP_KERNEL, 0);
break;
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index c411590..3e21211 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -810,7 +810,6 @@
struct btrfs_super_block *super_for_commit;
struct super_block *sb;
struct inode *btree_inode;
- struct backing_dev_info bdi;
struct mutex tree_log_mutex;
struct mutex transaction_kthread_mutex;
struct mutex cleaner_mutex;
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index eb1ee7b..061c1d1 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -1808,21 +1808,6 @@
return ret;
}
-static int setup_bdi(struct btrfs_fs_info *info, struct backing_dev_info *bdi)
-{
- int err;
-
- err = bdi_setup_and_register(bdi, "btrfs");
- if (err)
- return err;
-
- bdi->ra_pages = VM_MAX_READAHEAD * 1024 / PAGE_SIZE;
- bdi->congested_fn = btrfs_congested_fn;
- bdi->congested_data = info;
- bdi->capabilities |= BDI_CAP_CGROUP_WRITEBACK;
- return 0;
-}
-
/*
* called by the kthread helper functions to finally call the bio end_io
* functions. This is where read checksum verification actually happens
@@ -2601,16 +2586,10 @@
goto fail;
}
- ret = setup_bdi(fs_info, &fs_info->bdi);
- if (ret) {
- err = ret;
- goto fail_srcu;
- }
-
ret = percpu_counter_init(&fs_info->dirty_metadata_bytes, 0, GFP_KERNEL);
if (ret) {
err = ret;
- goto fail_bdi;
+ goto fail_srcu;
}
fs_info->dirty_metadata_batch = PAGE_SIZE *
(1 + ilog2(nr_cpu_ids));
@@ -2718,7 +2697,6 @@
sb->s_blocksize = 4096;
sb->s_blocksize_bits = blksize_bits(4096);
- sb->s_bdi = &fs_info->bdi;
btrfs_init_btree_inode(fs_info);
@@ -2915,9 +2893,12 @@
goto fail_sb_buffer;
}
- fs_info->bdi.ra_pages *= btrfs_super_num_devices(disk_super);
- fs_info->bdi.ra_pages = max(fs_info->bdi.ra_pages,
- SZ_4M / PAGE_SIZE);
+ sb->s_bdi->congested_fn = btrfs_congested_fn;
+ sb->s_bdi->congested_data = fs_info;
+ sb->s_bdi->capabilities |= BDI_CAP_CGROUP_WRITEBACK;
+ sb->s_bdi->ra_pages = VM_MAX_READAHEAD * 1024 / PAGE_SIZE;
+ sb->s_bdi->ra_pages *= btrfs_super_num_devices(disk_super);
+ sb->s_bdi->ra_pages = max(sb->s_bdi->ra_pages, SZ_4M / PAGE_SIZE);
sb->s_blocksize = sectorsize;
sb->s_blocksize_bits = blksize_bits(sectorsize);
@@ -3285,8 +3266,6 @@
percpu_counter_destroy(&fs_info->delalloc_bytes);
fail_dirty_metadata_bytes:
percpu_counter_destroy(&fs_info->dirty_metadata_bytes);
-fail_bdi:
- bdi_destroy(&fs_info->bdi);
fail_srcu:
cleanup_srcu_struct(&fs_info->subvol_srcu);
fail:
@@ -4007,7 +3986,6 @@
percpu_counter_destroy(&fs_info->dirty_metadata_bytes);
percpu_counter_destroy(&fs_info->delalloc_bytes);
percpu_counter_destroy(&fs_info->bio_counter);
- bdi_destroy(&fs_info->bdi);
cleanup_srcu_struct(&fs_info->subvol_srcu);
btrfs_free_stripe_hash_table(fs_info);
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index a59801d..afbea61 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -1042,9 +1042,12 @@
struct btrfs_qgroup *qgroup,
u64 num_bytes)
{
- btrfs_warn(fs_info,
+#ifdef CONFIG_BTRFS_DEBUG
+ WARN_ON(qgroup->reserved < num_bytes);
+ btrfs_debug(fs_info,
"qgroup %llu reserved space underflow, have: %llu, to free: %llu",
qgroup->qgroupid, qgroup->reserved, num_bytes);
+#endif
qgroup->reserved = 0;
}
/*
@@ -1075,7 +1078,7 @@
qgroup->excl += sign * num_bytes;
qgroup->excl_cmpr += sign * num_bytes;
if (sign > 0) {
- if (WARN_ON(qgroup->reserved < num_bytes))
+ if (qgroup->reserved < num_bytes)
report_reserved_underflow(fs_info, qgroup, num_bytes);
else
qgroup->reserved -= num_bytes;
@@ -1100,7 +1103,7 @@
WARN_ON(sign < 0 && qgroup->excl < num_bytes);
qgroup->excl += sign * num_bytes;
if (sign > 0) {
- if (WARN_ON(qgroup->reserved < num_bytes))
+ if (qgroup->reserved < num_bytes)
report_reserved_underflow(fs_info, qgroup,
num_bytes);
else
@@ -2469,7 +2472,7 @@
qg = unode_aux_to_qgroup(unode);
- if (WARN_ON(qg->reserved < num_bytes))
+ if (qg->reserved < num_bytes)
report_reserved_underflow(fs_info, qg, num_bytes);
else
qg->reserved -= num_bytes;
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index 9530a33..72a053c 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -1136,6 +1136,13 @@
#endif
sb->s_flags |= MS_I_VERSION;
sb->s_iflags |= SB_I_CGROUPWB;
+
+ err = super_setup_bdi(sb);
+ if (err) {
+ btrfs_err(fs_info, "super_setup_bdi failed");
+ return err;
+ }
+
err = open_ctree(sb, fs_devices, (char *)data);
if (err) {
btrfs_err(fs_info, "open_ctree failed");
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index 1a3e1b4..9ecb2fd 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -578,7 +578,7 @@
writeback_stat = atomic_long_inc_return(&fsc->writeback_count);
if (writeback_stat >
CONGESTION_ON_THRESH(fsc->mount_options->congestion_kb))
- set_bdi_congested(&fsc->backing_dev_info, BLK_RW_ASYNC);
+ set_bdi_congested(inode_to_bdi(inode), BLK_RW_ASYNC);
set_page_writeback(page);
err = ceph_osdc_writepages(osdc, ceph_vino(inode),
@@ -700,7 +700,7 @@
if (atomic_long_dec_return(&fsc->writeback_count) <
CONGESTION_OFF_THRESH(
fsc->mount_options->congestion_kb))
- clear_bdi_congested(&fsc->backing_dev_info,
+ clear_bdi_congested(inode_to_bdi(inode),
BLK_RW_ASYNC);
if (rc < 0)
@@ -979,7 +979,7 @@
if (atomic_long_inc_return(&fsc->writeback_count) >
CONGESTION_ON_THRESH(
fsc->mount_options->congestion_kb)) {
- set_bdi_congested(&fsc->backing_dev_info,
+ set_bdi_congested(inode_to_bdi(inode),
BLK_RW_ASYNC);
}
diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c
index f2ae393..3ef11bc 100644
--- a/fs/ceph/debugfs.c
+++ b/fs/ceph/debugfs.c
@@ -251,7 +251,7 @@
goto out;
snprintf(name, sizeof(name), "../../bdi/%s",
- dev_name(fsc->backing_dev_info.dev));
+ dev_name(fsc->sb->s_bdi->dev));
fsc->debugfs_bdi =
debugfs_create_symlink("bdi",
fsc->client->debugfs_dir,
diff --git a/fs/ceph/inode.c b/fs/ceph/inode.c
index d449e1c..d3119fe 100644
--- a/fs/ceph/inode.c
+++ b/fs/ceph/inode.c
@@ -2071,11 +2071,6 @@
if (inode_dirty_flags)
__mark_inode_dirty(inode, inode_dirty_flags);
- if (ia_valid & ATTR_MODE) {
- err = posix_acl_chmod(inode, attr->ia_mode);
- if (err)
- goto out_put;
- }
if (mask) {
req->r_inode = inode;
@@ -2089,13 +2084,11 @@
ceph_cap_string(dirtied), mask);
ceph_mdsc_put_request(req);
- if (mask & CEPH_SETATTR_SIZE)
+ ceph_free_cap_flush(prealloc_cf);
+
+ if (err >= 0 && (mask & CEPH_SETATTR_SIZE))
__ceph_do_pending_vmtruncate(inode);
- ceph_free_cap_flush(prealloc_cf);
- return err;
-out_put:
- ceph_mdsc_put_request(req);
- ceph_free_cap_flush(prealloc_cf);
+
return err;
}
@@ -2114,7 +2107,12 @@
if (err != 0)
return err;
- return __ceph_setattr(inode, attr);
+ err = __ceph_setattr(inode, attr);
+
+ if (err >= 0 && (attr->ia_valid & ATTR_MODE))
+ err = posix_acl_chmod(inode, attr->ia_mode);
+
+ return err;
}
/*
diff --git a/fs/ceph/super.c b/fs/ceph/super.c
index 0ec8d01..a8c81b2 100644
--- a/fs/ceph/super.c
+++ b/fs/ceph/super.c
@@ -579,10 +579,6 @@
atomic_long_set(&fsc->writeback_count, 0);
- err = bdi_init(&fsc->backing_dev_info);
- if (err < 0)
- goto fail_client;
-
err = -ENOMEM;
/*
* The number of concurrent works can be high but they don't need
@@ -590,7 +586,7 @@
*/
fsc->wb_wq = alloc_workqueue("ceph-writeback", 0, 1);
if (fsc->wb_wq == NULL)
- goto fail_bdi;
+ goto fail_client;
fsc->pg_inv_wq = alloc_workqueue("ceph-pg-invalid", 0, 1);
if (fsc->pg_inv_wq == NULL)
goto fail_wb_wq;
@@ -624,8 +620,6 @@
destroy_workqueue(fsc->pg_inv_wq);
fail_wb_wq:
destroy_workqueue(fsc->wb_wq);
-fail_bdi:
- bdi_destroy(&fsc->backing_dev_info);
fail_client:
ceph_destroy_client(fsc->client);
fail:
@@ -643,8 +637,6 @@
destroy_workqueue(fsc->pg_inv_wq);
destroy_workqueue(fsc->trunc_wq);
- bdi_destroy(&fsc->backing_dev_info);
-
mempool_destroy(fsc->wb_pagevec_pool);
destroy_mount_options(fsc->mount_options);
@@ -937,33 +929,32 @@
*/
static atomic_long_t bdi_seq = ATOMIC_LONG_INIT(0);
-static int ceph_register_bdi(struct super_block *sb,
- struct ceph_fs_client *fsc)
+static int ceph_setup_bdi(struct super_block *sb, struct ceph_fs_client *fsc)
{
int err;
+ err = super_setup_bdi_name(sb, "ceph-%ld",
+ atomic_long_inc_return(&bdi_seq));
+ if (err)
+ return err;
+
/* set ra_pages based on rasize mount option? */
if (fsc->mount_options->rasize >= PAGE_SIZE)
- fsc->backing_dev_info.ra_pages =
+ sb->s_bdi->ra_pages =
(fsc->mount_options->rasize + PAGE_SIZE - 1)
>> PAGE_SHIFT;
else
- fsc->backing_dev_info.ra_pages =
- VM_MAX_READAHEAD * 1024 / PAGE_SIZE;
+ sb->s_bdi->ra_pages = VM_MAX_READAHEAD * 1024 / PAGE_SIZE;
if (fsc->mount_options->rsize > fsc->mount_options->rasize &&
fsc->mount_options->rsize >= PAGE_SIZE)
- fsc->backing_dev_info.io_pages =
+ sb->s_bdi->io_pages =
(fsc->mount_options->rsize + PAGE_SIZE - 1)
>> PAGE_SHIFT;
else if (fsc->mount_options->rsize == 0)
- fsc->backing_dev_info.io_pages = ULONG_MAX;
+ sb->s_bdi->io_pages = ULONG_MAX;
- err = bdi_register(&fsc->backing_dev_info, NULL, "ceph-%ld",
- atomic_long_inc_return(&bdi_seq));
- if (!err)
- sb->s_bdi = &fsc->backing_dev_info;
- return err;
+ return 0;
}
static struct dentry *ceph_mount(struct file_system_type *fs_type,
@@ -1018,7 +1009,7 @@
dout("get_sb got existing client %p\n", fsc);
} else {
dout("get_sb using new client %p\n", fsc);
- err = ceph_register_bdi(sb, fsc);
+ err = ceph_setup_bdi(sb, fsc);
if (err < 0) {
res = ERR_PTR(err);
goto out_splat;
diff --git a/fs/ceph/super.h b/fs/ceph/super.h
index fe6b9cf..176186b 100644
--- a/fs/ceph/super.h
+++ b/fs/ceph/super.h
@@ -92,8 +92,6 @@
struct workqueue_struct *trunc_wq;
atomic_long_t writeback_count;
- struct backing_dev_info backing_dev_info;
-
#ifdef CONFIG_DEBUG_FS
struct dentry *debugfs_dentry_lru, *debugfs_caps;
struct dentry *debugfs_congestion_kb;
diff --git a/fs/cifs/cifs_fs_sb.h b/fs/cifs/cifs_fs_sb.h
index 07ed81c..cbd216b 100644
--- a/fs/cifs/cifs_fs_sb.h
+++ b/fs/cifs/cifs_fs_sb.h
@@ -68,7 +68,6 @@
umode_t mnt_dir_mode;
unsigned int mnt_cifs_flags;
char *mountdata; /* options received at mount time or via DFS refs */
- struct backing_dev_info bdi;
struct delayed_work prune_tlinks;
struct rcu_head rcu;
char *prepath;
diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c
index dd3f5fa..d0d11b7 100644
--- a/fs/cifs/cifsfs.c
+++ b/fs/cifs/cifsfs.c
@@ -37,6 +37,7 @@
#include <linux/freezer.h>
#include <linux/namei.h>
#include <linux/random.h>
+#include <linux/uuid.h>
#include <linux/xattr.h>
#include <net/ipv6.h>
#include "cifsfs.h"
@@ -138,7 +139,12 @@
sb->s_magic = CIFS_MAGIC_NUMBER;
sb->s_op = &cifs_super_ops;
sb->s_xattr = cifs_xattr_handlers;
- sb->s_bdi = &cifs_sb->bdi;
+ rc = super_setup_bdi(sb);
+ if (rc)
+ goto out_no_root;
+ /* tune readahead according to rsize */
+ sb->s_bdi->ra_pages = cifs_sb->rsize / PAGE_SIZE;
+
sb->s_blocksize = CIFS_MAX_MSGSIZE;
sb->s_blocksize_bits = 14; /* default 2**14 = CIFS_MAX_MSGSIZE */
inode = cifs_root_iget(sb);
diff --git a/fs/cifs/connect.c b/fs/cifs/connect.c
index d82467c..9bc0b4d 100644
--- a/fs/cifs/connect.c
+++ b/fs/cifs/connect.c
@@ -35,6 +35,7 @@
#include <linux/pagevec.h>
#include <linux/freezer.h>
#include <linux/namei.h>
+#include <linux/uuid.h>
#include <linux/uaccess.h>
#include <asm/processor.h>
#include <linux/inet.h>
@@ -2912,16 +2913,14 @@
{
struct cifs_sb_info *old = CIFS_SB(sb);
struct cifs_sb_info *new = mnt_data->cifs_sb;
+ bool old_set = old->mnt_cifs_flags & CIFS_MOUNT_USE_PREFIX_PATH;
+ bool new_set = new->mnt_cifs_flags & CIFS_MOUNT_USE_PREFIX_PATH;
- if (old->mnt_cifs_flags & CIFS_MOUNT_USE_PREFIX_PATH) {
- if (!(new->mnt_cifs_flags & CIFS_MOUNT_USE_PREFIX_PATH))
- return 0;
- /* The prepath should be null terminated strings */
- if (strcmp(new->prepath, old->prepath))
- return 0;
-
+ if (old_set && new_set && !strcmp(new->prepath, old->prepath))
return 1;
- }
+ else if (!old_set && !new_set)
+ return 1;
+
return 0;
}
@@ -3692,10 +3691,6 @@
int referral_walks_count = 0;
#endif
- rc = bdi_setup_and_register(&cifs_sb->bdi, "cifs");
- if (rc)
- return rc;
-
#ifdef CONFIG_CIFS_DFS_UPCALL
try_mount_again:
/* cleanup activities if we're chasing a referral */
@@ -3723,7 +3718,6 @@
server = cifs_get_tcp_session(volume_info);
if (IS_ERR(server)) {
rc = PTR_ERR(server);
- bdi_destroy(&cifs_sb->bdi);
goto out;
}
if ((volume_info->max_credits < 20) ||
@@ -3780,9 +3774,6 @@
cifs_sb->wsize = server->ops->negotiate_wsize(tcon, volume_info);
cifs_sb->rsize = server->ops->negotiate_rsize(tcon, volume_info);
- /* tune readahead according to rsize */
- cifs_sb->bdi.ra_pages = cifs_sb->rsize / PAGE_SIZE;
-
remote_path_check:
#ifdef CONFIG_CIFS_DFS_UPCALL
/*
@@ -3899,7 +3890,6 @@
cifs_put_smb_ses(ses);
else
cifs_put_tcp_session(server, 0);
- bdi_destroy(&cifs_sb->bdi);
}
out:
@@ -4102,7 +4092,6 @@
}
spin_unlock(&cifs_sb->tlink_tree_lock);
- bdi_destroy(&cifs_sb->bdi);
kfree(cifs_sb->mountdata);
kfree(cifs_sb->prepath);
call_rcu(&cifs_sb->rcu, delayed_free);
diff --git a/fs/cifs/ioctl.c b/fs/cifs/ioctl.c
index 265c45f..7f4bba5 100644
--- a/fs/cifs/ioctl.c
+++ b/fs/cifs/ioctl.c
@@ -74,7 +74,8 @@
rc = cifs_file_copychunk_range(xid, src_file.file, 0, dst_file, 0,
src_inode->i_size, 0);
-
+ if (rc > 0)
+ rc = 0;
out_fput:
fdput(src_file);
out_drop_write:
diff --git a/fs/cifs/misc.c b/fs/cifs/misc.c
index d3fb115..8437878 100644
--- a/fs/cifs/misc.c
+++ b/fs/cifs/misc.c
@@ -167,13 +167,11 @@
/* clear the first few header bytes */
/* for most paths, more is cleared in header_assemble */
- if (ret_buf) {
- memset(ret_buf, 0, buf_size + 3);
- atomic_inc(&bufAllocCount);
+ memset(ret_buf, 0, buf_size + 3);
+ atomic_inc(&bufAllocCount);
#ifdef CONFIG_CIFS_STATS2
- atomic_inc(&totBufAllocCount);
+ atomic_inc(&totBufAllocCount);
#endif /* CONFIG_CIFS_STATS2 */
- }
return ret_buf;
}
@@ -201,15 +199,13 @@
albeit slightly larger than necessary and maxbuffersize
defaults to this and can not be bigger */
ret_buf = mempool_alloc(cifs_sm_req_poolp, GFP_NOFS);
- if (ret_buf) {
/* No need to clear memory here, cleared in header assemble */
/* memset(ret_buf, 0, sizeof(struct smb_hdr) + 27);*/
- atomic_inc(&smBufAllocCount);
+ atomic_inc(&smBufAllocCount);
#ifdef CONFIG_CIFS_STATS2
- atomic_inc(&totSmBufAllocCount);
+ atomic_inc(&totSmBufAllocCount);
#endif /* CONFIG_CIFS_STATS2 */
- }
return ret_buf;
}
diff --git a/fs/cifs/smb2pdu.c b/fs/cifs/smb2pdu.c
index 02da648..fb0da09 100644
--- a/fs/cifs/smb2pdu.c
+++ b/fs/cifs/smb2pdu.c
@@ -33,6 +33,7 @@
#include <linux/vfs.h>
#include <linux/task_io_accounting_ops.h>
#include <linux/uaccess.h>
+#include <linux/uuid.h>
#include <linux/pagemap.h>
#include <linux/xattr.h>
#include "smb2pdu.h"
diff --git a/fs/cifs/smb2transport.c b/fs/cifs/smb2transport.c
index 506b67f..c69ec96 100644
--- a/fs/cifs/smb2transport.c
+++ b/fs/cifs/smb2transport.c
@@ -538,23 +538,19 @@
}
temp = mempool_alloc(cifs_mid_poolp, GFP_NOFS);
- if (temp == NULL)
- return temp;
- else {
- memset(temp, 0, sizeof(struct mid_q_entry));
- temp->mid = le64_to_cpu(shdr->MessageId);
- temp->pid = current->pid;
- temp->command = shdr->Command; /* Always LE */
- temp->when_alloc = jiffies;
- temp->server = server;
+ memset(temp, 0, sizeof(struct mid_q_entry));
+ temp->mid = le64_to_cpu(shdr->MessageId);
+ temp->pid = current->pid;
+ temp->command = shdr->Command; /* Always LE */
+ temp->when_alloc = jiffies;
+ temp->server = server;
- /*
- * The default is for the mid to be synchronous, so the
- * default callback just wakes up the current task.
- */
- temp->callback = cifs_wake_up_task;
- temp->callback_data = current;
- }
+ /*
+ * The default is for the mid to be synchronous, so the
+ * default callback just wakes up the current task.
+ */
+ temp->callback = cifs_wake_up_task;
+ temp->callback_data = current;
atomic_inc(&midCount);
temp->mid_state = MID_REQUEST_ALLOCATED;
diff --git a/fs/cifs/transport.c b/fs/cifs/transport.c
index f6e13a9..4d64b5b 100644
--- a/fs/cifs/transport.c
+++ b/fs/cifs/transport.c
@@ -55,26 +55,22 @@
}
temp = mempool_alloc(cifs_mid_poolp, GFP_NOFS);
- if (temp == NULL)
- return temp;
- else {
- memset(temp, 0, sizeof(struct mid_q_entry));
- temp->mid = get_mid(smb_buffer);
- temp->pid = current->pid;
- temp->command = cpu_to_le16(smb_buffer->Command);
- cifs_dbg(FYI, "For smb_command %d\n", smb_buffer->Command);
+ memset(temp, 0, sizeof(struct mid_q_entry));
+ temp->mid = get_mid(smb_buffer);
+ temp->pid = current->pid;
+ temp->command = cpu_to_le16(smb_buffer->Command);
+ cifs_dbg(FYI, "For smb_command %d\n", smb_buffer->Command);
/* do_gettimeofday(&temp->when_sent);*/ /* easier to use jiffies */
- /* when mid allocated can be before when sent */
- temp->when_alloc = jiffies;
- temp->server = server;
+ /* when mid allocated can be before when sent */
+ temp->when_alloc = jiffies;
+ temp->server = server;
- /*
- * The default is for the mid to be synchronous, so the
- * default callback just wakes up the current task.
- */
- temp->callback = cifs_wake_up_task;
- temp->callback_data = current;
- }
+ /*
+ * The default is for the mid to be synchronous, so the
+ * default callback just wakes up the current task.
+ */
+ temp->callback = cifs_wake_up_task;
+ temp->callback_data = current;
atomic_inc(&midCount);
temp->mid_state = MID_REQUEST_ALLOCATED;
diff --git a/fs/coda/inode.c b/fs/coda/inode.c
index 2dea594..6058df3 100644
--- a/fs/coda/inode.c
+++ b/fs/coda/inode.c
@@ -183,10 +183,6 @@
goto unlock_out;
}
- error = bdi_setup_and_register(&vc->bdi, "coda");
- if (error)
- goto unlock_out;
-
vc->vc_sb = sb;
mutex_unlock(&vc->vc_mutex);
@@ -197,7 +193,10 @@
sb->s_magic = CODA_SUPER_MAGIC;
sb->s_op = &coda_super_operations;
sb->s_d_op = &coda_dentry_operations;
- sb->s_bdi = &vc->bdi;
+
+ error = super_setup_bdi(sb);
+ if (error)
+ goto error;
/* get root fid from Venus: this needs the root inode */
error = venus_rootfid(sb, &fid);
@@ -228,7 +227,6 @@
error:
mutex_lock(&vc->vc_mutex);
- bdi_destroy(&vc->bdi);
vc->vc_sb = NULL;
sb->s_fs_info = NULL;
unlock_out:
@@ -240,7 +238,6 @@
{
struct venus_comm *vcp = coda_vcp(sb);
mutex_lock(&vcp->vc_mutex);
- bdi_destroy(&vcp->bdi);
vcp->vc_sb = NULL;
sb->s_fs_info = NULL;
mutex_unlock(&vcp->vc_mutex);
diff --git a/fs/compat.c b/fs/compat.c
index c61b506..190b38b 100644
--- a/fs/compat.c
+++ b/fs/compat.c
@@ -15,555 +15,14 @@
* published by the Free Software Foundation.
*/
-#include <linux/stddef.h>
-#include <linux/kernel.h>
-#include <linux/linkage.h>
#include <linux/compat.h>
-#include <linux/errno.h>
-#include <linux/time.h>
-#include <linux/cred.h>
-#include <linux/fs.h>
-#include <linux/fcntl.h>
-#include <linux/namei.h>
-#include <linux/file.h>
-#include <linux/fdtable.h>
-#include <linux/vfs.h>
-#include <linux/ioctl.h>
-#include <linux/init.h>
#include <linux/ncp_mount.h>
#include <linux/nfs4_mount.h>
#include <linux/syscalls.h>
-#include <linux/ctype.h>
-#include <linux/dirent.h>
-#include <linux/fsnotify.h>
-#include <linux/highuid.h>
-#include <linux/personality.h>
-#include <linux/rwsem.h>
-#include <linux/tsacct_kern.h>
-#include <linux/security.h>
-#include <linux/highmem.h>
-#include <linux/signal.h>
-#include <linux/poll.h>
-#include <linux/mm.h>
-#include <linux/fs_struct.h>
#include <linux/slab.h>
-#include <linux/pagemap.h>
-#include <linux/aio.h>
-
#include <linux/uaccess.h>
-#include <asm/mmu_context.h>
-#include <asm/ioctls.h>
#include "internal.h"
-/*
- * Not all architectures have sys_utime, so implement this in terms
- * of sys_utimes.
- */
-COMPAT_SYSCALL_DEFINE2(utime, const char __user *, filename,
- struct compat_utimbuf __user *, t)
-{
- struct timespec tv[2];
-
- if (t) {
- if (get_user(tv[0].tv_sec, &t->actime) ||
- get_user(tv[1].tv_sec, &t->modtime))
- return -EFAULT;
- tv[0].tv_nsec = 0;
- tv[1].tv_nsec = 0;
- }
- return do_utimes(AT_FDCWD, filename, t ? tv : NULL, 0);
-}
-
-COMPAT_SYSCALL_DEFINE4(utimensat, unsigned int, dfd, const char __user *, filename, struct compat_timespec __user *, t, int, flags)
-{
- struct timespec tv[2];
-
- if (t) {
- if (compat_get_timespec(&tv[0], &t[0]) ||
- compat_get_timespec(&tv[1], &t[1]))
- return -EFAULT;
-
- if (tv[0].tv_nsec == UTIME_OMIT && tv[1].tv_nsec == UTIME_OMIT)
- return 0;
- }
- return do_utimes(dfd, filename, t ? tv : NULL, flags);
-}
-
-COMPAT_SYSCALL_DEFINE3(futimesat, unsigned int, dfd, const char __user *, filename, struct compat_timeval __user *, t)
-{
- struct timespec tv[2];
-
- if (t) {
- if (get_user(tv[0].tv_sec, &t[0].tv_sec) ||
- get_user(tv[0].tv_nsec, &t[0].tv_usec) ||
- get_user(tv[1].tv_sec, &t[1].tv_sec) ||
- get_user(tv[1].tv_nsec, &t[1].tv_usec))
- return -EFAULT;
- if (tv[0].tv_nsec >= 1000000 || tv[0].tv_nsec < 0 ||
- tv[1].tv_nsec >= 1000000 || tv[1].tv_nsec < 0)
- return -EINVAL;
- tv[0].tv_nsec *= 1000;
- tv[1].tv_nsec *= 1000;
- }
- return do_utimes(dfd, filename, t ? tv : NULL, 0);
-}
-
-COMPAT_SYSCALL_DEFINE2(utimes, const char __user *, filename, struct compat_timeval __user *, t)
-{
- return compat_sys_futimesat(AT_FDCWD, filename, t);
-}
-
-static int cp_compat_stat(struct kstat *stat, struct compat_stat __user *ubuf)
-{
- struct compat_stat tmp;
-
- if (!old_valid_dev(stat->dev) || !old_valid_dev(stat->rdev))
- return -EOVERFLOW;
-
- memset(&tmp, 0, sizeof(tmp));
- tmp.st_dev = old_encode_dev(stat->dev);
- tmp.st_ino = stat->ino;
- if (sizeof(tmp.st_ino) < sizeof(stat->ino) && tmp.st_ino != stat->ino)
- return -EOVERFLOW;
- tmp.st_mode = stat->mode;
- tmp.st_nlink = stat->nlink;
- if (tmp.st_nlink != stat->nlink)
- return -EOVERFLOW;
- SET_UID(tmp.st_uid, from_kuid_munged(current_user_ns(), stat->uid));
- SET_GID(tmp.st_gid, from_kgid_munged(current_user_ns(), stat->gid));
- tmp.st_rdev = old_encode_dev(stat->rdev);
- if ((u64) stat->size > MAX_NON_LFS)
- return -EOVERFLOW;
- tmp.st_size = stat->size;
- tmp.st_atime = stat->atime.tv_sec;
- tmp.st_atime_nsec = stat->atime.tv_nsec;
- tmp.st_mtime = stat->mtime.tv_sec;
- tmp.st_mtime_nsec = stat->mtime.tv_nsec;
- tmp.st_ctime = stat->ctime.tv_sec;
- tmp.st_ctime_nsec = stat->ctime.tv_nsec;
- tmp.st_blocks = stat->blocks;
- tmp.st_blksize = stat->blksize;
- return copy_to_user(ubuf, &tmp, sizeof(tmp)) ? -EFAULT : 0;
-}
-
-COMPAT_SYSCALL_DEFINE2(newstat, const char __user *, filename,
- struct compat_stat __user *, statbuf)
-{
- struct kstat stat;
- int error;
-
- error = vfs_stat(filename, &stat);
- if (error)
- return error;
- return cp_compat_stat(&stat, statbuf);
-}
-
-COMPAT_SYSCALL_DEFINE2(newlstat, const char __user *, filename,
- struct compat_stat __user *, statbuf)
-{
- struct kstat stat;
- int error;
-
- error = vfs_lstat(filename, &stat);
- if (error)
- return error;
- return cp_compat_stat(&stat, statbuf);
-}
-
-#ifndef __ARCH_WANT_STAT64
-COMPAT_SYSCALL_DEFINE4(newfstatat, unsigned int, dfd,
- const char __user *, filename,
- struct compat_stat __user *, statbuf, int, flag)
-{
- struct kstat stat;
- int error;
-
- error = vfs_fstatat(dfd, filename, &stat, flag);
- if (error)
- return error;
- return cp_compat_stat(&stat, statbuf);
-}
-#endif
-
-COMPAT_SYSCALL_DEFINE2(newfstat, unsigned int, fd,
- struct compat_stat __user *, statbuf)
-{
- struct kstat stat;
- int error = vfs_fstat(fd, &stat);
-
- if (!error)
- error = cp_compat_stat(&stat, statbuf);
- return error;
-}
-
-static int put_compat_statfs(struct compat_statfs __user *ubuf, struct kstatfs *kbuf)
-{
-
- if (sizeof ubuf->f_blocks == 4) {
- if ((kbuf->f_blocks | kbuf->f_bfree | kbuf->f_bavail |
- kbuf->f_bsize | kbuf->f_frsize) & 0xffffffff00000000ULL)
- return -EOVERFLOW;
- /* f_files and f_ffree may be -1; it's okay
- * to stuff that into 32 bits */
- if (kbuf->f_files != 0xffffffffffffffffULL
- && (kbuf->f_files & 0xffffffff00000000ULL))
- return -EOVERFLOW;
- if (kbuf->f_ffree != 0xffffffffffffffffULL
- && (kbuf->f_ffree & 0xffffffff00000000ULL))
- return -EOVERFLOW;
- }
- if (!access_ok(VERIFY_WRITE, ubuf, sizeof(*ubuf)) ||
- __put_user(kbuf->f_type, &ubuf->f_type) ||
- __put_user(kbuf->f_bsize, &ubuf->f_bsize) ||
- __put_user(kbuf->f_blocks, &ubuf->f_blocks) ||
- __put_user(kbuf->f_bfree, &ubuf->f_bfree) ||
- __put_user(kbuf->f_bavail, &ubuf->f_bavail) ||
- __put_user(kbuf->f_files, &ubuf->f_files) ||
- __put_user(kbuf->f_ffree, &ubuf->f_ffree) ||
- __put_user(kbuf->f_namelen, &ubuf->f_namelen) ||
- __put_user(kbuf->f_fsid.val[0], &ubuf->f_fsid.val[0]) ||
- __put_user(kbuf->f_fsid.val[1], &ubuf->f_fsid.val[1]) ||
- __put_user(kbuf->f_frsize, &ubuf->f_frsize) ||
- __put_user(kbuf->f_flags, &ubuf->f_flags) ||
- __clear_user(ubuf->f_spare, sizeof(ubuf->f_spare)))
- return -EFAULT;
- return 0;
-}
-
-/*
- * The following statfs calls are copies of code from fs/statfs.c and
- * should be checked against those from time to time
- */
-COMPAT_SYSCALL_DEFINE2(statfs, const char __user *, pathname, struct compat_statfs __user *, buf)
-{
- struct kstatfs tmp;
- int error = user_statfs(pathname, &tmp);
- if (!error)
- error = put_compat_statfs(buf, &tmp);
- return error;
-}
-
-COMPAT_SYSCALL_DEFINE2(fstatfs, unsigned int, fd, struct compat_statfs __user *, buf)
-{
- struct kstatfs tmp;
- int error = fd_statfs(fd, &tmp);
- if (!error)
- error = put_compat_statfs(buf, &tmp);
- return error;
-}
-
-static int put_compat_statfs64(struct compat_statfs64 __user *ubuf, struct kstatfs *kbuf)
-{
- if (sizeof(ubuf->f_bsize) == 4) {
- if ((kbuf->f_type | kbuf->f_bsize | kbuf->f_namelen |
- kbuf->f_frsize | kbuf->f_flags) & 0xffffffff00000000ULL)
- return -EOVERFLOW;
- /* f_files and f_ffree may be -1; it's okay
- * to stuff that into 32 bits */
- if (kbuf->f_files != 0xffffffffffffffffULL
- && (kbuf->f_files & 0xffffffff00000000ULL))
- return -EOVERFLOW;
- if (kbuf->f_ffree != 0xffffffffffffffffULL
- && (kbuf->f_ffree & 0xffffffff00000000ULL))
- return -EOVERFLOW;
- }
- if (!access_ok(VERIFY_WRITE, ubuf, sizeof(*ubuf)) ||
- __put_user(kbuf->f_type, &ubuf->f_type) ||
- __put_user(kbuf->f_bsize, &ubuf->f_bsize) ||
- __put_user(kbuf->f_blocks, &ubuf->f_blocks) ||
- __put_user(kbuf->f_bfree, &ubuf->f_bfree) ||
- __put_user(kbuf->f_bavail, &ubuf->f_bavail) ||
- __put_user(kbuf->f_files, &ubuf->f_files) ||
- __put_user(kbuf->f_ffree, &ubuf->f_ffree) ||
- __put_user(kbuf->f_namelen, &ubuf->f_namelen) ||
- __put_user(kbuf->f_fsid.val[0], &ubuf->f_fsid.val[0]) ||
- __put_user(kbuf->f_fsid.val[1], &ubuf->f_fsid.val[1]) ||
- __put_user(kbuf->f_frsize, &ubuf->f_frsize) ||
- __put_user(kbuf->f_flags, &ubuf->f_flags) ||
- __clear_user(ubuf->f_spare, sizeof(ubuf->f_spare)))
- return -EFAULT;
- return 0;
-}
-
-COMPAT_SYSCALL_DEFINE3(statfs64, const char __user *, pathname, compat_size_t, sz, struct compat_statfs64 __user *, buf)
-{
- struct kstatfs tmp;
- int error;
-
- if (sz != sizeof(*buf))
- return -EINVAL;
-
- error = user_statfs(pathname, &tmp);
- if (!error)
- error = put_compat_statfs64(buf, &tmp);
- return error;
-}
-
-COMPAT_SYSCALL_DEFINE3(fstatfs64, unsigned int, fd, compat_size_t, sz, struct compat_statfs64 __user *, buf)
-{
- struct kstatfs tmp;
- int error;
-
- if (sz != sizeof(*buf))
- return -EINVAL;
-
- error = fd_statfs(fd, &tmp);
- if (!error)
- error = put_compat_statfs64(buf, &tmp);
- return error;
-}
-
-/*
- * This is a copy of sys_ustat, just dealing with a structure layout.
- * Given how simple this syscall is that apporach is more maintainable
- * than the various conversion hacks.
- */
-COMPAT_SYSCALL_DEFINE2(ustat, unsigned, dev, struct compat_ustat __user *, u)
-{
- struct compat_ustat tmp;
- struct kstatfs sbuf;
- int err = vfs_ustat(new_decode_dev(dev), &sbuf);
- if (err)
- return err;
-
- memset(&tmp, 0, sizeof(struct compat_ustat));
- tmp.f_tfree = sbuf.f_bfree;
- tmp.f_tinode = sbuf.f_ffree;
- if (copy_to_user(u, &tmp, sizeof(struct compat_ustat)))
- return -EFAULT;
- return 0;
-}
-
-static int get_compat_flock(struct flock *kfl, struct compat_flock __user *ufl)
-{
- if (!access_ok(VERIFY_READ, ufl, sizeof(*ufl)) ||
- __get_user(kfl->l_type, &ufl->l_type) ||
- __get_user(kfl->l_whence, &ufl->l_whence) ||
- __get_user(kfl->l_start, &ufl->l_start) ||
- __get_user(kfl->l_len, &ufl->l_len) ||
- __get_user(kfl->l_pid, &ufl->l_pid))
- return -EFAULT;
- return 0;
-}
-
-static int put_compat_flock(struct flock *kfl, struct compat_flock __user *ufl)
-{
- if (!access_ok(VERIFY_WRITE, ufl, sizeof(*ufl)) ||
- __put_user(kfl->l_type, &ufl->l_type) ||
- __put_user(kfl->l_whence, &ufl->l_whence) ||
- __put_user(kfl->l_start, &ufl->l_start) ||
- __put_user(kfl->l_len, &ufl->l_len) ||
- __put_user(kfl->l_pid, &ufl->l_pid))
- return -EFAULT;
- return 0;
-}
-
-#ifndef HAVE_ARCH_GET_COMPAT_FLOCK64
-static int get_compat_flock64(struct flock *kfl, struct compat_flock64 __user *ufl)
-{
- if (!access_ok(VERIFY_READ, ufl, sizeof(*ufl)) ||
- __get_user(kfl->l_type, &ufl->l_type) ||
- __get_user(kfl->l_whence, &ufl->l_whence) ||
- __get_user(kfl->l_start, &ufl->l_start) ||
- __get_user(kfl->l_len, &ufl->l_len) ||
- __get_user(kfl->l_pid, &ufl->l_pid))
- return -EFAULT;
- return 0;
-}
-#endif
-
-#ifndef HAVE_ARCH_PUT_COMPAT_FLOCK64
-static int put_compat_flock64(struct flock *kfl, struct compat_flock64 __user *ufl)
-{
- if (!access_ok(VERIFY_WRITE, ufl, sizeof(*ufl)) ||
- __put_user(kfl->l_type, &ufl->l_type) ||
- __put_user(kfl->l_whence, &ufl->l_whence) ||
- __put_user(kfl->l_start, &ufl->l_start) ||
- __put_user(kfl->l_len, &ufl->l_len) ||
- __put_user(kfl->l_pid, &ufl->l_pid))
- return -EFAULT;
- return 0;
-}
-#endif
-
-static unsigned int
-convert_fcntl_cmd(unsigned int cmd)
-{
- switch (cmd) {
- case F_GETLK64:
- return F_GETLK;
- case F_SETLK64:
- return F_SETLK;
- case F_SETLKW64:
- return F_SETLKW;
- }
-
- return cmd;
-}
-
-COMPAT_SYSCALL_DEFINE3(fcntl64, unsigned int, fd, unsigned int, cmd,
- compat_ulong_t, arg)
-{
- mm_segment_t old_fs;
- struct flock f;
- long ret;
- unsigned int conv_cmd;
-
- switch (cmd) {
- case F_GETLK:
- case F_SETLK:
- case F_SETLKW:
- ret = get_compat_flock(&f, compat_ptr(arg));
- if (ret != 0)
- break;
- old_fs = get_fs();
- set_fs(KERNEL_DS);
- ret = sys_fcntl(fd, cmd, (unsigned long)&f);
- set_fs(old_fs);
- if (cmd == F_GETLK && ret == 0) {
- /* GETLK was successful and we need to return the data...
- * but it needs to fit in the compat structure.
- * l_start shouldn't be too big, unless the original
- * start + end is greater than COMPAT_OFF_T_MAX, in which
- * case the app was asking for trouble, so we return
- * -EOVERFLOW in that case.
- * l_len could be too big, in which case we just truncate it,
- * and only allow the app to see that part of the conflicting
- * lock that might make sense to it anyway
- */
-
- if (f.l_start > COMPAT_OFF_T_MAX)
- ret = -EOVERFLOW;
- if (f.l_len > COMPAT_OFF_T_MAX)
- f.l_len = COMPAT_OFF_T_MAX;
- if (ret == 0)
- ret = put_compat_flock(&f, compat_ptr(arg));
- }
- break;
-
- case F_GETLK64:
- case F_SETLK64:
- case F_SETLKW64:
- case F_OFD_GETLK:
- case F_OFD_SETLK:
- case F_OFD_SETLKW:
- ret = get_compat_flock64(&f, compat_ptr(arg));
- if (ret != 0)
- break;
- old_fs = get_fs();
- set_fs(KERNEL_DS);
- conv_cmd = convert_fcntl_cmd(cmd);
- ret = sys_fcntl(fd, conv_cmd, (unsigned long)&f);
- set_fs(old_fs);
- if ((conv_cmd == F_GETLK || conv_cmd == F_OFD_GETLK) && ret == 0) {
- /* need to return lock information - see above for commentary */
- if (f.l_start > COMPAT_LOFF_T_MAX)
- ret = -EOVERFLOW;
- if (f.l_len > COMPAT_LOFF_T_MAX)
- f.l_len = COMPAT_LOFF_T_MAX;
- if (ret == 0)
- ret = put_compat_flock64(&f, compat_ptr(arg));
- }
- break;
-
- default:
- ret = sys_fcntl(fd, cmd, arg);
- break;
- }
- return ret;
-}
-
-COMPAT_SYSCALL_DEFINE3(fcntl, unsigned int, fd, unsigned int, cmd,
- compat_ulong_t, arg)
-{
- switch (cmd) {
- case F_GETLK64:
- case F_SETLK64:
- case F_SETLKW64:
- case F_OFD_GETLK:
- case F_OFD_SETLK:
- case F_OFD_SETLKW:
- return -EINVAL;
- }
- return compat_sys_fcntl64(fd, cmd, arg);
-}
-
-/* A write operation does a read from user space and vice versa */
-#define vrfy_dir(type) ((type) == READ ? VERIFY_WRITE : VERIFY_READ)
-
-ssize_t compat_rw_copy_check_uvector(int type,
- const struct compat_iovec __user *uvector, unsigned long nr_segs,
- unsigned long fast_segs, struct iovec *fast_pointer,
- struct iovec **ret_pointer)
-{
- compat_ssize_t tot_len;
- struct iovec *iov = *ret_pointer = fast_pointer;
- ssize_t ret = 0;
- int seg;
-
- /*
- * SuS says "The readv() function *may* fail if the iovcnt argument
- * was less than or equal to 0, or greater than {IOV_MAX}. Linux has
- * traditionally returned zero for zero segments, so...
- */
- if (nr_segs == 0)
- goto out;
-
- ret = -EINVAL;
- if (nr_segs > UIO_MAXIOV)
- goto out;
- if (nr_segs > fast_segs) {
- ret = -ENOMEM;
- iov = kmalloc(nr_segs*sizeof(struct iovec), GFP_KERNEL);
- if (iov == NULL)
- goto out;
- }
- *ret_pointer = iov;
-
- ret = -EFAULT;
- if (!access_ok(VERIFY_READ, uvector, nr_segs*sizeof(*uvector)))
- goto out;
-
- /*
- * Single unix specification:
- * We should -EINVAL if an element length is not >= 0 and fitting an
- * ssize_t.
- *
- * In Linux, the total length is limited to MAX_RW_COUNT, there is
- * no overflow possibility.
- */
- tot_len = 0;
- ret = -EINVAL;
- for (seg = 0; seg < nr_segs; seg++) {
- compat_uptr_t buf;
- compat_ssize_t len;
-
- if (__get_user(len, &uvector->iov_len) ||
- __get_user(buf, &uvector->iov_base)) {
- ret = -EFAULT;
- goto out;
- }
- if (len < 0) /* size_t not fitting in compat_ssize_t .. */
- goto out;
- if (type >= 0 &&
- !access_ok(vrfy_dir(type), compat_ptr(buf), len)) {
- ret = -EFAULT;
- goto out;
- }
- if (len > MAX_RW_COUNT - tot_len)
- len = MAX_RW_COUNT - tot_len;
- tot_len += len;
- iov->iov_base = compat_ptr(buf);
- iov->iov_len = (compat_size_t) len;
- uvector++;
- iov++;
- }
- ret = tot_len;
-
-out:
- return ret;
-}
-
struct compat_ncp_mount_data {
compat_int_t version;
compat_uint_t ncp_fd;
@@ -744,653 +203,3 @@
out:
return retval;
}
-
-struct compat_old_linux_dirent {
- compat_ulong_t d_ino;
- compat_ulong_t d_offset;
- unsigned short d_namlen;
- char d_name[1];
-};
-
-struct compat_readdir_callback {
- struct dir_context ctx;
- struct compat_old_linux_dirent __user *dirent;
- int result;
-};
-
-static int compat_fillonedir(struct dir_context *ctx, const char *name,
- int namlen, loff_t offset, u64 ino,
- unsigned int d_type)
-{
- struct compat_readdir_callback *buf =
- container_of(ctx, struct compat_readdir_callback, ctx);
- struct compat_old_linux_dirent __user *dirent;
- compat_ulong_t d_ino;
-
- if (buf->result)
- return -EINVAL;
- d_ino = ino;
- if (sizeof(d_ino) < sizeof(ino) && d_ino != ino) {
- buf->result = -EOVERFLOW;
- return -EOVERFLOW;
- }
- buf->result++;
- dirent = buf->dirent;
- if (!access_ok(VERIFY_WRITE, dirent,
- (unsigned long)(dirent->d_name + namlen + 1) -
- (unsigned long)dirent))
- goto efault;
- if ( __put_user(d_ino, &dirent->d_ino) ||
- __put_user(offset, &dirent->d_offset) ||
- __put_user(namlen, &dirent->d_namlen) ||
- __copy_to_user(dirent->d_name, name, namlen) ||
- __put_user(0, dirent->d_name + namlen))
- goto efault;
- return 0;
-efault:
- buf->result = -EFAULT;
- return -EFAULT;
-}
-
-COMPAT_SYSCALL_DEFINE3(old_readdir, unsigned int, fd,
- struct compat_old_linux_dirent __user *, dirent, unsigned int, count)
-{
- int error;
- struct fd f = fdget_pos(fd);
- struct compat_readdir_callback buf = {
- .ctx.actor = compat_fillonedir,
- .dirent = dirent
- };
-
- if (!f.file)
- return -EBADF;
-
- error = iterate_dir(f.file, &buf.ctx);
- if (buf.result)
- error = buf.result;
-
- fdput_pos(f);
- return error;
-}
-
-struct compat_linux_dirent {
- compat_ulong_t d_ino;
- compat_ulong_t d_off;
- unsigned short d_reclen;
- char d_name[1];
-};
-
-struct compat_getdents_callback {
- struct dir_context ctx;
- struct compat_linux_dirent __user *current_dir;
- struct compat_linux_dirent __user *previous;
- int count;
- int error;
-};
-
-static int compat_filldir(struct dir_context *ctx, const char *name, int namlen,
- loff_t offset, u64 ino, unsigned int d_type)
-{
- struct compat_linux_dirent __user * dirent;
- struct compat_getdents_callback *buf =
- container_of(ctx, struct compat_getdents_callback, ctx);
- compat_ulong_t d_ino;
- int reclen = ALIGN(offsetof(struct compat_linux_dirent, d_name) +
- namlen + 2, sizeof(compat_long_t));
-
- buf->error = -EINVAL; /* only used if we fail.. */
- if (reclen > buf->count)
- return -EINVAL;
- d_ino = ino;
- if (sizeof(d_ino) < sizeof(ino) && d_ino != ino) {
- buf->error = -EOVERFLOW;
- return -EOVERFLOW;
- }
- dirent = buf->previous;
- if (dirent) {
- if (signal_pending(current))
- return -EINTR;
- if (__put_user(offset, &dirent->d_off))
- goto efault;
- }
- dirent = buf->current_dir;
- if (__put_user(d_ino, &dirent->d_ino))
- goto efault;
- if (__put_user(reclen, &dirent->d_reclen))
- goto efault;
- if (copy_to_user(dirent->d_name, name, namlen))
- goto efault;
- if (__put_user(0, dirent->d_name + namlen))
- goto efault;
- if (__put_user(d_type, (char __user *) dirent + reclen - 1))
- goto efault;
- buf->previous = dirent;
- dirent = (void __user *)dirent + reclen;
- buf->current_dir = dirent;
- buf->count -= reclen;
- return 0;
-efault:
- buf->error = -EFAULT;
- return -EFAULT;
-}
-
-COMPAT_SYSCALL_DEFINE3(getdents, unsigned int, fd,
- struct compat_linux_dirent __user *, dirent, unsigned int, count)
-{
- struct fd f;
- struct compat_linux_dirent __user * lastdirent;
- struct compat_getdents_callback buf = {
- .ctx.actor = compat_filldir,
- .current_dir = dirent,
- .count = count
- };
- int error;
-
- if (!access_ok(VERIFY_WRITE, dirent, count))
- return -EFAULT;
-
- f = fdget_pos(fd);
- if (!f.file)
- return -EBADF;
-
- error = iterate_dir(f.file, &buf.ctx);
- if (error >= 0)
- error = buf.error;
- lastdirent = buf.previous;
- if (lastdirent) {
- if (put_user(buf.ctx.pos, &lastdirent->d_off))
- error = -EFAULT;
- else
- error = count - buf.count;
- }
- fdput_pos(f);
- return error;
-}
-
-#ifdef __ARCH_WANT_COMPAT_SYS_GETDENTS64
-
-struct compat_getdents_callback64 {
- struct dir_context ctx;
- struct linux_dirent64 __user *current_dir;
- struct linux_dirent64 __user *previous;
- int count;
- int error;
-};
-
-static int compat_filldir64(struct dir_context *ctx, const char *name,
- int namlen, loff_t offset, u64 ino,
- unsigned int d_type)
-{
- struct linux_dirent64 __user *dirent;
- struct compat_getdents_callback64 *buf =
- container_of(ctx, struct compat_getdents_callback64, ctx);
- int reclen = ALIGN(offsetof(struct linux_dirent64, d_name) + namlen + 1,
- sizeof(u64));
- u64 off;
-
- buf->error = -EINVAL; /* only used if we fail.. */
- if (reclen > buf->count)
- return -EINVAL;
- dirent = buf->previous;
-
- if (dirent) {
- if (signal_pending(current))
- return -EINTR;
- if (__put_user_unaligned(offset, &dirent->d_off))
- goto efault;
- }
- dirent = buf->current_dir;
- if (__put_user_unaligned(ino, &dirent->d_ino))
- goto efault;
- off = 0;
- if (__put_user_unaligned(off, &dirent->d_off))
- goto efault;
- if (__put_user(reclen, &dirent->d_reclen))
- goto efault;
- if (__put_user(d_type, &dirent->d_type))
- goto efault;
- if (copy_to_user(dirent->d_name, name, namlen))
- goto efault;
- if (__put_user(0, dirent->d_name + namlen))
- goto efault;
- buf->previous = dirent;
- dirent = (void __user *)dirent + reclen;
- buf->current_dir = dirent;
- buf->count -= reclen;
- return 0;
-efault:
- buf->error = -EFAULT;
- return -EFAULT;
-}
-
-COMPAT_SYSCALL_DEFINE3(getdents64, unsigned int, fd,
- struct linux_dirent64 __user *, dirent, unsigned int, count)
-{
- struct fd f;
- struct linux_dirent64 __user * lastdirent;
- struct compat_getdents_callback64 buf = {
- .ctx.actor = compat_filldir64,
- .current_dir = dirent,
- .count = count
- };
- int error;
-
- if (!access_ok(VERIFY_WRITE, dirent, count))
- return -EFAULT;
-
- f = fdget_pos(fd);
- if (!f.file)
- return -EBADF;
-
- error = iterate_dir(f.file, &buf.ctx);
- if (error >= 0)
- error = buf.error;
- lastdirent = buf.previous;
- if (lastdirent) {
- typeof(lastdirent->d_off) d_off = buf.ctx.pos;
- if (__put_user_unaligned(d_off, &lastdirent->d_off))
- error = -EFAULT;
- else
- error = count - buf.count;
- }
- fdput_pos(f);
- return error;
-}
-#endif /* __ARCH_WANT_COMPAT_SYS_GETDENTS64 */
-
-/*
- * Exactly like fs/open.c:sys_open(), except that it doesn't set the
- * O_LARGEFILE flag.
- */
-COMPAT_SYSCALL_DEFINE3(open, const char __user *, filename, int, flags, umode_t, mode)
-{
- return do_sys_open(AT_FDCWD, filename, flags, mode);
-}
-
-/*
- * Exactly like fs/open.c:sys_openat(), except that it doesn't set the
- * O_LARGEFILE flag.
- */
-COMPAT_SYSCALL_DEFINE4(openat, int, dfd, const char __user *, filename, int, flags, umode_t, mode)
-{
- return do_sys_open(dfd, filename, flags, mode);
-}
-
-#define __COMPAT_NFDBITS (8 * sizeof(compat_ulong_t))
-
-static int poll_select_copy_remaining(struct timespec *end_time, void __user *p,
- int timeval, int ret)
-{
- struct timespec ts;
-
- if (!p)
- return ret;
-
- if (current->personality & STICKY_TIMEOUTS)
- goto sticky;
-
- /* No update for zero timeout */
- if (!end_time->tv_sec && !end_time->tv_nsec)
- return ret;
-
- ktime_get_ts(&ts);
- ts = timespec_sub(*end_time, ts);
- if (ts.tv_sec < 0)
- ts.tv_sec = ts.tv_nsec = 0;
-
- if (timeval) {
- struct compat_timeval rtv;
-
- rtv.tv_sec = ts.tv_sec;
- rtv.tv_usec = ts.tv_nsec / NSEC_PER_USEC;
-
- if (!copy_to_user(p, &rtv, sizeof(rtv)))
- return ret;
- } else {
- struct compat_timespec rts;
-
- rts.tv_sec = ts.tv_sec;
- rts.tv_nsec = ts.tv_nsec;
-
- if (!copy_to_user(p, &rts, sizeof(rts)))
- return ret;
- }
- /*
- * If an application puts its timeval in read-only memory, we
- * don't want the Linux-specific update to the timeval to
- * cause a fault after the select has completed
- * successfully. However, because we're not updating the
- * timeval, we can't restart the system call.
- */
-
-sticky:
- if (ret == -ERESTARTNOHAND)
- ret = -EINTR;
- return ret;
-}
-
-/*
- * Ooo, nasty. We need here to frob 32-bit unsigned longs to
- * 64-bit unsigned longs.
- */
-static
-int compat_get_fd_set(unsigned long nr, compat_ulong_t __user *ufdset,
- unsigned long *fdset)
-{
- nr = DIV_ROUND_UP(nr, __COMPAT_NFDBITS);
- if (ufdset) {
- unsigned long odd;
-
- if (!access_ok(VERIFY_WRITE, ufdset, nr*sizeof(compat_ulong_t)))
- return -EFAULT;
-
- odd = nr & 1UL;
- nr &= ~1UL;
- while (nr) {
- unsigned long h, l;
- if (__get_user(l, ufdset) || __get_user(h, ufdset+1))
- return -EFAULT;
- ufdset += 2;
- *fdset++ = h << 32 | l;
- nr -= 2;
- }
- if (odd && __get_user(*fdset, ufdset))
- return -EFAULT;
- } else {
- /* Tricky, must clear full unsigned long in the
- * kernel fdset at the end, this makes sure that
- * actually happens.
- */
- memset(fdset, 0, ((nr + 1) & ~1)*sizeof(compat_ulong_t));
- }
- return 0;
-}
-
-static
-int compat_set_fd_set(unsigned long nr, compat_ulong_t __user *ufdset,
- unsigned long *fdset)
-{
- unsigned long odd;
- nr = DIV_ROUND_UP(nr, __COMPAT_NFDBITS);
-
- if (!ufdset)
- return 0;
-
- odd = nr & 1UL;
- nr &= ~1UL;
- while (nr) {
- unsigned long h, l;
- l = *fdset++;
- h = l >> 32;
- if (__put_user(l, ufdset) || __put_user(h, ufdset+1))
- return -EFAULT;
- ufdset += 2;
- nr -= 2;
- }
- if (odd && __put_user(*fdset, ufdset))
- return -EFAULT;
- return 0;
-}
-
-
-/*
- * This is a virtual copy of sys_select from fs/select.c and probably
- * should be compared to it from time to time
- */
-
-/*
- * We can actually return ERESTARTSYS instead of EINTR, but I'd
- * like to be certain this leads to no problems. So I return
- * EINTR just for safety.
- *
- * Update: ERESTARTSYS breaks at least the xview clock binary, so
- * I'm trying ERESTARTNOHAND which restart only when you want to.
- */
-int compat_core_sys_select(int n, compat_ulong_t __user *inp,
- compat_ulong_t __user *outp, compat_ulong_t __user *exp,
- struct timespec *end_time)
-{
- fd_set_bits fds;
- void *bits;
- int size, max_fds, ret = -EINVAL;
- struct fdtable *fdt;
- long stack_fds[SELECT_STACK_ALLOC/sizeof(long)];
-
- if (n < 0)
- goto out_nofds;
-
- /* max_fds can increase, so grab it once to avoid race */
- rcu_read_lock();
- fdt = files_fdtable(current->files);
- max_fds = fdt->max_fds;
- rcu_read_unlock();
- if (n > max_fds)
- n = max_fds;
-
- /*
- * We need 6 bitmaps (in/out/ex for both incoming and outgoing),
- * since we used fdset we need to allocate memory in units of
- * long-words.
- */
- size = FDS_BYTES(n);
- bits = stack_fds;
- if (size > sizeof(stack_fds) / 6) {
- bits = kmalloc(6 * size, GFP_KERNEL);
- ret = -ENOMEM;
- if (!bits)
- goto out_nofds;
- }
- fds.in = (unsigned long *) bits;
- fds.out = (unsigned long *) (bits + size);
- fds.ex = (unsigned long *) (bits + 2*size);
- fds.res_in = (unsigned long *) (bits + 3*size);
- fds.res_out = (unsigned long *) (bits + 4*size);
- fds.res_ex = (unsigned long *) (bits + 5*size);
-
- if ((ret = compat_get_fd_set(n, inp, fds.in)) ||
- (ret = compat_get_fd_set(n, outp, fds.out)) ||
- (ret = compat_get_fd_set(n, exp, fds.ex)))
- goto out;
- zero_fd_set(n, fds.res_in);
- zero_fd_set(n, fds.res_out);
- zero_fd_set(n, fds.res_ex);
-
- ret = do_select(n, &fds, end_time);
-
- if (ret < 0)
- goto out;
- if (!ret) {
- ret = -ERESTARTNOHAND;
- if (signal_pending(current))
- goto out;
- ret = 0;
- }
-
- if (compat_set_fd_set(n, inp, fds.res_in) ||
- compat_set_fd_set(n, outp, fds.res_out) ||
- compat_set_fd_set(n, exp, fds.res_ex))
- ret = -EFAULT;
-out:
- if (bits != stack_fds)
- kfree(bits);
-out_nofds:
- return ret;
-}
-
-COMPAT_SYSCALL_DEFINE5(select, int, n, compat_ulong_t __user *, inp,
- compat_ulong_t __user *, outp, compat_ulong_t __user *, exp,
- struct compat_timeval __user *, tvp)
-{
- struct timespec end_time, *to = NULL;
- struct compat_timeval tv;
- int ret;
-
- if (tvp) {
- if (copy_from_user(&tv, tvp, sizeof(tv)))
- return -EFAULT;
-
- to = &end_time;
- if (poll_select_set_timeout(to,
- tv.tv_sec + (tv.tv_usec / USEC_PER_SEC),
- (tv.tv_usec % USEC_PER_SEC) * NSEC_PER_USEC))
- return -EINVAL;
- }
-
- ret = compat_core_sys_select(n, inp, outp, exp, to);
- ret = poll_select_copy_remaining(&end_time, tvp, 1, ret);
-
- return ret;
-}
-
-struct compat_sel_arg_struct {
- compat_ulong_t n;
- compat_uptr_t inp;
- compat_uptr_t outp;
- compat_uptr_t exp;
- compat_uptr_t tvp;
-};
-
-COMPAT_SYSCALL_DEFINE1(old_select, struct compat_sel_arg_struct __user *, arg)
-{
- struct compat_sel_arg_struct a;
-
- if (copy_from_user(&a, arg, sizeof(a)))
- return -EFAULT;
- return compat_sys_select(a.n, compat_ptr(a.inp), compat_ptr(a.outp),
- compat_ptr(a.exp), compat_ptr(a.tvp));
-}
-
-static long do_compat_pselect(int n, compat_ulong_t __user *inp,
- compat_ulong_t __user *outp, compat_ulong_t __user *exp,
- struct compat_timespec __user *tsp, compat_sigset_t __user *sigmask,
- compat_size_t sigsetsize)
-{
- compat_sigset_t ss32;
- sigset_t ksigmask, sigsaved;
- struct compat_timespec ts;
- struct timespec end_time, *to = NULL;
- int ret;
-
- if (tsp) {
- if (copy_from_user(&ts, tsp, sizeof(ts)))
- return -EFAULT;
-
- to = &end_time;
- if (poll_select_set_timeout(to, ts.tv_sec, ts.tv_nsec))
- return -EINVAL;
- }
-
- if (sigmask) {
- if (sigsetsize != sizeof(compat_sigset_t))
- return -EINVAL;
- if (copy_from_user(&ss32, sigmask, sizeof(ss32)))
- return -EFAULT;
- sigset_from_compat(&ksigmask, &ss32);
-
- sigdelsetmask(&ksigmask, sigmask(SIGKILL)|sigmask(SIGSTOP));
- sigprocmask(SIG_SETMASK, &ksigmask, &sigsaved);
- }
-
- ret = compat_core_sys_select(n, inp, outp, exp, to);
- ret = poll_select_copy_remaining(&end_time, tsp, 0, ret);
-
- if (ret == -ERESTARTNOHAND) {
- /*
- * Don't restore the signal mask yet. Let do_signal() deliver
- * the signal on the way back to userspace, before the signal
- * mask is restored.
- */
- if (sigmask) {
- memcpy(¤t->saved_sigmask, &sigsaved,
- sizeof(sigsaved));
- set_restore_sigmask();
- }
- } else if (sigmask)
- sigprocmask(SIG_SETMASK, &sigsaved, NULL);
-
- return ret;
-}
-
-COMPAT_SYSCALL_DEFINE6(pselect6, int, n, compat_ulong_t __user *, inp,
- compat_ulong_t __user *, outp, compat_ulong_t __user *, exp,
- struct compat_timespec __user *, tsp, void __user *, sig)
-{
- compat_size_t sigsetsize = 0;
- compat_uptr_t up = 0;
-
- if (sig) {
- if (!access_ok(VERIFY_READ, sig,
- sizeof(compat_uptr_t)+sizeof(compat_size_t)) ||
- __get_user(up, (compat_uptr_t __user *)sig) ||
- __get_user(sigsetsize,
- (compat_size_t __user *)(sig+sizeof(up))))
- return -EFAULT;
- }
- return do_compat_pselect(n, inp, outp, exp, tsp, compat_ptr(up),
- sigsetsize);
-}
-
-COMPAT_SYSCALL_DEFINE5(ppoll, struct pollfd __user *, ufds,
- unsigned int, nfds, struct compat_timespec __user *, tsp,
- const compat_sigset_t __user *, sigmask, compat_size_t, sigsetsize)
-{
- compat_sigset_t ss32;
- sigset_t ksigmask, sigsaved;
- struct compat_timespec ts;
- struct timespec end_time, *to = NULL;
- int ret;
-
- if (tsp) {
- if (copy_from_user(&ts, tsp, sizeof(ts)))
- return -EFAULT;
-
- to = &end_time;
- if (poll_select_set_timeout(to, ts.tv_sec, ts.tv_nsec))
- return -EINVAL;
- }
-
- if (sigmask) {
- if (sigsetsize != sizeof(compat_sigset_t))
- return -EINVAL;
- if (copy_from_user(&ss32, sigmask, sizeof(ss32)))
- return -EFAULT;
- sigset_from_compat(&ksigmask, &ss32);
-
- sigdelsetmask(&ksigmask, sigmask(SIGKILL)|sigmask(SIGSTOP));
- sigprocmask(SIG_SETMASK, &ksigmask, &sigsaved);
- }
-
- ret = do_sys_poll(ufds, nfds, to);
-
- /* We can restart this syscall, usually */
- if (ret == -EINTR) {
- /*
- * Don't restore the signal mask yet. Let do_signal() deliver
- * the signal on the way back to userspace, before the signal
- * mask is restored.
- */
- if (sigmask) {
- memcpy(¤t->saved_sigmask, &sigsaved,
- sizeof(sigsaved));
- set_restore_sigmask();
- }
- ret = -ERESTARTNOHAND;
- } else if (sigmask)
- sigprocmask(SIG_SETMASK, &sigsaved, NULL);
-
- ret = poll_select_copy_remaining(&end_time, tsp, 0, ret);
-
- return ret;
-}
-
-#ifdef CONFIG_FHANDLE
-/*
- * Exactly like fs/open.c:sys_open_by_handle_at(), except that it
- * doesn't set the O_LARGEFILE flag.
- */
-COMPAT_SYSCALL_DEFINE3(open_by_handle_at, int, mountdirfd,
- struct file_handle __user *, handle, int, flags)
-{
- return do_handle_open(mountdirfd, handle, flags);
-}
-#endif
diff --git a/fs/dax.c b/fs/dax.c
index 85abd74..6433650 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -991,7 +991,7 @@
sector_t start_sector = dax.sector + (offset >> 9);
return blkdev_issue_zeroout(bdev, start_sector,
- length >> 9, GFP_NOFS, true);
+ length >> 9, GFP_NOFS, 0);
} else {
if (dax_map_atomic(bdev, &dax) < 0)
return PTR_ERR(dax.addr);
diff --git a/fs/ecryptfs/ecryptfs_kernel.h b/fs/ecryptfs/ecryptfs_kernel.h
index 95c1c8d..9c351bf 100644
--- a/fs/ecryptfs/ecryptfs_kernel.h
+++ b/fs/ecryptfs/ecryptfs_kernel.h
@@ -349,7 +349,6 @@
struct ecryptfs_sb_info {
struct super_block *wsi_sb;
struct ecryptfs_mount_crypt_stat mount_crypt_stat;
- struct backing_dev_info bdi;
};
/* file private data. */
diff --git a/fs/ecryptfs/main.c b/fs/ecryptfs/main.c
index 151872d..9014479 100644
--- a/fs/ecryptfs/main.c
+++ b/fs/ecryptfs/main.c
@@ -519,12 +519,11 @@
goto out;
}
- rc = bdi_setup_and_register(&sbi->bdi, "ecryptfs");
+ rc = super_setup_bdi(s);
if (rc)
goto out1;
ecryptfs_set_superblock_private(s, sbi);
- s->s_bdi = &sbi->bdi;
/* ->kill_sb() will take care of sbi after that point */
sbi = NULL;
@@ -633,7 +632,6 @@
if (!sb_info)
return;
ecryptfs_destroy_mount_crypt_stat(&sb_info->mount_crypt_stat);
- bdi_destroy(&sb_info->bdi);
kmem_cache_free(ecryptfs_sb_info_cache, sb_info);
}
diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index 3412514..5420767 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -42,6 +42,7 @@
#include <linux/seq_file.h>
#include <linux/compat.h>
#include <linux/rculist.h>
+#include <net/busy_poll.h>
/*
* LOCKING:
@@ -224,6 +225,11 @@
/* used to optimize loop detection check */
int visited;
struct list_head visited_list_link;
+
+#ifdef CONFIG_NET_RX_BUSY_POLL
+ /* used to track busy poll napi_id */
+ unsigned int napi_id;
+#endif
};
/* Wait structure used by the poll hooks */
@@ -384,6 +390,77 @@
return !list_empty(&ep->rdllist) || ep->ovflist != EP_UNACTIVE_PTR;
}
+#ifdef CONFIG_NET_RX_BUSY_POLL
+static bool ep_busy_loop_end(void *p, unsigned long start_time)
+{
+ struct eventpoll *ep = p;
+
+ return ep_events_available(ep) || busy_loop_timeout(start_time);
+}
+#endif /* CONFIG_NET_RX_BUSY_POLL */
+
+/*
+ * Busy poll if globally on and supporting sockets found && no events,
+ * busy loop will return if need_resched or ep_events_available.
+ *
+ * we must do our busy polling with irqs enabled
+ */
+static void ep_busy_loop(struct eventpoll *ep, int nonblock)
+{
+#ifdef CONFIG_NET_RX_BUSY_POLL
+ unsigned int napi_id = READ_ONCE(ep->napi_id);
+
+ if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on())
+ napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep);
+#endif
+}
+
+static inline void ep_reset_busy_poll_napi_id(struct eventpoll *ep)
+{
+#ifdef CONFIG_NET_RX_BUSY_POLL
+ if (ep->napi_id)
+ ep->napi_id = 0;
+#endif
+}
+
+/*
+ * Set epoll busy poll NAPI ID from sk.
+ */
+static inline void ep_set_busy_poll_napi_id(struct epitem *epi)
+{
+#ifdef CONFIG_NET_RX_BUSY_POLL
+ struct eventpoll *ep;
+ unsigned int napi_id;
+ struct socket *sock;
+ struct sock *sk;
+ int err;
+
+ if (!net_busy_loop_on())
+ return;
+
+ sock = sock_from_file(epi->ffd.file, &err);
+ if (!sock)
+ return;
+
+ sk = sock->sk;
+ if (!sk)
+ return;
+
+ napi_id = READ_ONCE(sk->sk_napi_id);
+ ep = epi->ep;
+
+ /* Non-NAPI IDs can be rejected
+ * or
+ * Nothing to do if we already have this ID
+ */
+ if (napi_id < MIN_NAPI_ID || napi_id == ep->napi_id)
+ return;
+
+ /* record NAPI ID for use in next busy poll */
+ ep->napi_id = napi_id;
+#endif
+}
+
/**
* ep_call_nested - Perform a bound (possibly) nested call, by checking
* that the recursion limit is not exceeded, and that
@@ -1022,6 +1099,8 @@
spin_lock_irqsave(&ep->lock, flags);
+ ep_set_busy_poll_napi_id(epi);
+
/*
* If the event mask does not contain any poll(2) event, we consider the
* descriptor to be disabled. This condition is likely the effect of the
@@ -1363,6 +1442,9 @@
/* We have to drop the new item inside our item list to keep track of it */
spin_lock_irqsave(&ep->lock, flags);
+ /* record NAPI ID of new item if present */
+ ep_set_busy_poll_napi_id(epi);
+
/* If the file is already "ready" we drop it inside the ready list */
if ((revents & event->events) && !ep_is_linked(&epi->rdllink)) {
list_add_tail(&epi->rdllink, &ep->rdllist);
@@ -1637,10 +1719,21 @@
}
fetch_events:
+
+ if (!ep_events_available(ep))
+ ep_busy_loop(ep, timed_out);
+
spin_lock_irqsave(&ep->lock, flags);
if (!ep_events_available(ep)) {
/*
+ * Busy poll timed out. Drop NAPI ID for now, we can add
+ * it back in when we have moved a socket with a valid NAPI
+ * ID onto the ready list.
+ */
+ ep_reset_busy_poll_napi_id(ep);
+
+ /*
* We don't have any available event to return to the caller.
* We need to sleep here, and we will be wake up by
* ep_poll_callback() when events will become available.
diff --git a/fs/exec.c b/fs/exec.c
index 65145a3..72934df 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -1320,6 +1320,7 @@
else
set_dumpable(current->mm, suid_dumpable);
+ arch_setup_new_exec();
perf_event_exec();
__set_task_comm(current, kbasename(bprm->filename), true);
diff --git a/fs/exofs/exofs.h b/fs/exofs/exofs.h
index 2e86086..5dc3924 100644
--- a/fs/exofs/exofs.h
+++ b/fs/exofs/exofs.h
@@ -64,7 +64,6 @@
* our extension to the in-memory superblock
*/
struct exofs_sb_info {
- struct backing_dev_info bdi; /* register our bdi with VFS */
struct exofs_sb_stats s_ess; /* Written often, pre-allocate*/
int s_timeout; /* timeout for OSD operations */
uint64_t s_nextid; /* highest object ID used */
diff --git a/fs/exofs/super.c b/fs/exofs/super.c
index 1076a42..819624c 100644
--- a/fs/exofs/super.c
+++ b/fs/exofs/super.c
@@ -464,7 +464,6 @@
sbi->one_comp.obj.partition);
exofs_sysfs_sb_del(sbi);
- bdi_destroy(&sbi->bdi);
exofs_free_sbi(sbi);
sb->s_fs_info = NULL;
}
@@ -809,8 +808,12 @@
__sbi_read_stats(sbi);
/* set up operation vectors */
- sbi->bdi.ra_pages = __ra_pages(&sbi->layout);
- sb->s_bdi = &sbi->bdi;
+ ret = super_setup_bdi(sb);
+ if (ret) {
+ EXOFS_DBGMSG("Failed to super_setup_bdi\n");
+ goto free_sbi;
+ }
+ sb->s_bdi->ra_pages = __ra_pages(&sbi->layout);
sb->s_fs_info = sbi;
sb->s_op = &exofs_sops;
sb->s_export_op = &exofs_export_ops;
@@ -836,14 +839,6 @@
goto free_sbi;
}
- ret = bdi_setup_and_register(&sbi->bdi, "exofs");
- if (ret) {
- EXOFS_DBGMSG("Failed to bdi_setup_and_register\n");
- dput(sb->s_root);
- sb->s_root = NULL;
- goto free_sbi;
- }
-
exofs_sysfs_dbg_print();
_exofs_print_device("Mounting", opts->dev_name,
ore_comp_dev(&sbi->oc, 0),
diff --git a/fs/fcntl.c b/fs/fcntl.c
index be8fbe2..8bd81c2 100644
--- a/fs/fcntl.c
+++ b/fs/fcntl.c
@@ -23,6 +23,7 @@
#include <linux/pid_namespace.h>
#include <linux/user_namespace.h>
#include <linux/shmem_fs.h>
+#include <linux/compat.h>
#include <asm/poll.h>
#include <asm/siginfo.h>
@@ -420,6 +421,162 @@
}
#endif
+#ifdef CONFIG_COMPAT
+static int get_compat_flock(struct flock *kfl, struct compat_flock __user *ufl)
+{
+ if (!access_ok(VERIFY_READ, ufl, sizeof(*ufl)) ||
+ __get_user(kfl->l_type, &ufl->l_type) ||
+ __get_user(kfl->l_whence, &ufl->l_whence) ||
+ __get_user(kfl->l_start, &ufl->l_start) ||
+ __get_user(kfl->l_len, &ufl->l_len) ||
+ __get_user(kfl->l_pid, &ufl->l_pid))
+ return -EFAULT;
+ return 0;
+}
+
+static int put_compat_flock(struct flock *kfl, struct compat_flock __user *ufl)
+{
+ if (!access_ok(VERIFY_WRITE, ufl, sizeof(*ufl)) ||
+ __put_user(kfl->l_type, &ufl->l_type) ||
+ __put_user(kfl->l_whence, &ufl->l_whence) ||
+ __put_user(kfl->l_start, &ufl->l_start) ||
+ __put_user(kfl->l_len, &ufl->l_len) ||
+ __put_user(kfl->l_pid, &ufl->l_pid))
+ return -EFAULT;
+ return 0;
+}
+
+#ifndef HAVE_ARCH_GET_COMPAT_FLOCK64
+static int get_compat_flock64(struct flock *kfl, struct compat_flock64 __user *ufl)
+{
+ if (!access_ok(VERIFY_READ, ufl, sizeof(*ufl)) ||
+ __get_user(kfl->l_type, &ufl->l_type) ||
+ __get_user(kfl->l_whence, &ufl->l_whence) ||
+ __get_user(kfl->l_start, &ufl->l_start) ||
+ __get_user(kfl->l_len, &ufl->l_len) ||
+ __get_user(kfl->l_pid, &ufl->l_pid))
+ return -EFAULT;
+ return 0;
+}
+#endif
+
+#ifndef HAVE_ARCH_PUT_COMPAT_FLOCK64
+static int put_compat_flock64(struct flock *kfl, struct compat_flock64 __user *ufl)
+{
+ if (!access_ok(VERIFY_WRITE, ufl, sizeof(*ufl)) ||
+ __put_user(kfl->l_type, &ufl->l_type) ||
+ __put_user(kfl->l_whence, &ufl->l_whence) ||
+ __put_user(kfl->l_start, &ufl->l_start) ||
+ __put_user(kfl->l_len, &ufl->l_len) ||
+ __put_user(kfl->l_pid, &ufl->l_pid))
+ return -EFAULT;
+ return 0;
+}
+#endif
+
+static unsigned int
+convert_fcntl_cmd(unsigned int cmd)
+{
+ switch (cmd) {
+ case F_GETLK64:
+ return F_GETLK;
+ case F_SETLK64:
+ return F_SETLK;
+ case F_SETLKW64:
+ return F_SETLKW;
+ }
+
+ return cmd;
+}
+
+COMPAT_SYSCALL_DEFINE3(fcntl64, unsigned int, fd, unsigned int, cmd,
+ compat_ulong_t, arg)
+{
+ mm_segment_t old_fs;
+ struct flock f;
+ long ret;
+ unsigned int conv_cmd;
+
+ switch (cmd) {
+ case F_GETLK:
+ case F_SETLK:
+ case F_SETLKW:
+ ret = get_compat_flock(&f, compat_ptr(arg));
+ if (ret != 0)
+ break;
+ old_fs = get_fs();
+ set_fs(KERNEL_DS);
+ ret = sys_fcntl(fd, cmd, (unsigned long)&f);
+ set_fs(old_fs);
+ if (cmd == F_GETLK && ret == 0) {
+ /* GETLK was successful and we need to return the data...
+ * but it needs to fit in the compat structure.
+ * l_start shouldn't be too big, unless the original
+ * start + end is greater than COMPAT_OFF_T_MAX, in which
+ * case the app was asking for trouble, so we return
+ * -EOVERFLOW in that case.
+ * l_len could be too big, in which case we just truncate it,
+ * and only allow the app to see that part of the conflicting
+ * lock that might make sense to it anyway
+ */
+
+ if (f.l_start > COMPAT_OFF_T_MAX)
+ ret = -EOVERFLOW;
+ if (f.l_len > COMPAT_OFF_T_MAX)
+ f.l_len = COMPAT_OFF_T_MAX;
+ if (ret == 0)
+ ret = put_compat_flock(&f, compat_ptr(arg));
+ }
+ break;
+
+ case F_GETLK64:
+ case F_SETLK64:
+ case F_SETLKW64:
+ case F_OFD_GETLK:
+ case F_OFD_SETLK:
+ case F_OFD_SETLKW:
+ ret = get_compat_flock64(&f, compat_ptr(arg));
+ if (ret != 0)
+ break;
+ old_fs = get_fs();
+ set_fs(KERNEL_DS);
+ conv_cmd = convert_fcntl_cmd(cmd);
+ ret = sys_fcntl(fd, conv_cmd, (unsigned long)&f);
+ set_fs(old_fs);
+ if ((conv_cmd == F_GETLK || conv_cmd == F_OFD_GETLK) && ret == 0) {
+ /* need to return lock information - see above for commentary */
+ if (f.l_start > COMPAT_LOFF_T_MAX)
+ ret = -EOVERFLOW;
+ if (f.l_len > COMPAT_LOFF_T_MAX)
+ f.l_len = COMPAT_LOFF_T_MAX;
+ if (ret == 0)
+ ret = put_compat_flock64(&f, compat_ptr(arg));
+ }
+ break;
+
+ default:
+ ret = sys_fcntl(fd, cmd, arg);
+ break;
+ }
+ return ret;
+}
+
+COMPAT_SYSCALL_DEFINE3(fcntl, unsigned int, fd, unsigned int, cmd,
+ compat_ulong_t, arg)
+{
+ switch (cmd) {
+ case F_GETLK64:
+ case F_SETLK64:
+ case F_SETLKW64:
+ case F_OFD_GETLK:
+ case F_OFD_SETLK:
+ case F_OFD_SETLKW:
+ return -EINVAL;
+ }
+ return compat_sys_fcntl64(fd, cmd, arg);
+}
+#endif
+
/* Table to convert sigio signal codes into poll band bitmaps */
static const long band_table[NSIGPOLL] = {
diff --git a/fs/fhandle.c b/fs/fhandle.c
index 5559168..58a61f5 100644
--- a/fs/fhandle.c
+++ b/fs/fhandle.c
@@ -9,6 +9,7 @@
#include <linux/fsnotify.h>
#include <linux/personality.h>
#include <linux/uaccess.h>
+#include <linux/compat.h>
#include "internal.h"
#include "mount.h"
@@ -264,3 +265,15 @@
ret = do_handle_open(mountdirfd, handle, flags);
return ret;
}
+
+#ifdef CONFIG_COMPAT
+/*
+ * Exactly like fs/open.c:sys_open_by_handle_at(), except that it
+ * doesn't set the O_LARGEFILE flag.
+ */
+COMPAT_SYSCALL_DEFINE3(open_by_handle_at, int, mountdirfd,
+ struct file_handle __user *, handle, int, flags)
+{
+ return do_handle_open(mountdirfd, handle, flags);
+}
+#endif
diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index b681b43..c2d7f3a 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -382,9 +382,9 @@
wake_up(&fc->blocked_waitq);
if (fc->num_background == fc->congestion_threshold &&
- fc->connected && fc->bdi_initialized) {
- clear_bdi_congested(&fc->bdi, BLK_RW_SYNC);
- clear_bdi_congested(&fc->bdi, BLK_RW_ASYNC);
+ fc->connected && fc->sb) {
+ clear_bdi_congested(fc->sb->s_bdi, BLK_RW_SYNC);
+ clear_bdi_congested(fc->sb->s_bdi, BLK_RW_ASYNC);
}
fc->num_background--;
fc->active_background--;
@@ -573,10 +573,9 @@
fc->num_background++;
if (fc->num_background == fc->max_background)
fc->blocked = 1;
- if (fc->num_background == fc->congestion_threshold &&
- fc->bdi_initialized) {
- set_bdi_congested(&fc->bdi, BLK_RW_SYNC);
- set_bdi_congested(&fc->bdi, BLK_RW_ASYNC);
+ if (fc->num_background == fc->congestion_threshold && fc->sb) {
+ set_bdi_congested(fc->sb->s_bdi, BLK_RW_SYNC);
+ set_bdi_congested(fc->sb->s_bdi, BLK_RW_ASYNC);
}
list_add_tail(&req->list, &fc->bg_queue);
flush_bg_queue(fc);
diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h
index 32ac2c9..f33341d 100644
--- a/fs/fuse/fuse_i.h
+++ b/fs/fuse/fuse_i.h
@@ -527,9 +527,6 @@
/** Filesystem supports NFS exporting. Only set in INIT */
unsigned export_support:1;
- /** Set if bdi is valid */
- unsigned bdi_initialized:1;
-
/** write-back cache policy (default is write-through) */
unsigned writeback_cache:1;
@@ -631,9 +628,6 @@
/** Negotiated minor version */
unsigned minor;
- /** Backing dev info */
- struct backing_dev_info bdi;
-
/** Entry on the fuse_conn_list */
struct list_head entry;
diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c
index 6fe6a88..73cf051 100644
--- a/fs/fuse/inode.c
+++ b/fs/fuse/inode.c
@@ -386,12 +386,6 @@
}
}
-static void fuse_bdi_destroy(struct fuse_conn *fc)
-{
- if (fc->bdi_initialized)
- bdi_destroy(&fc->bdi);
-}
-
static void fuse_put_super(struct super_block *sb)
{
struct fuse_conn *fc = get_fuse_conn_super(sb);
@@ -403,7 +397,6 @@
list_del(&fc->entry);
fuse_ctl_remove_conn(fc);
mutex_unlock(&fuse_mutex);
- fuse_bdi_destroy(fc);
fuse_conn_put(fc);
}
@@ -928,7 +921,8 @@
fc->no_flock = 1;
}
- fc->bdi.ra_pages = min(fc->bdi.ra_pages, ra_pages);
+ fc->sb->s_bdi->ra_pages =
+ min(fc->sb->s_bdi->ra_pages, ra_pages);
fc->minor = arg->minor;
fc->max_write = arg->minor < 5 ? 4096 : arg->max_write;
fc->max_write = max_t(unsigned, 4096, fc->max_write);
@@ -944,7 +938,7 @@
arg->major = FUSE_KERNEL_VERSION;
arg->minor = FUSE_KERNEL_MINOR_VERSION;
- arg->max_readahead = fc->bdi.ra_pages * PAGE_SIZE;
+ arg->max_readahead = fc->sb->s_bdi->ra_pages * PAGE_SIZE;
arg->flags |= FUSE_ASYNC_READ | FUSE_POSIX_LOCKS | FUSE_ATOMIC_O_TRUNC |
FUSE_EXPORT_SUPPORT | FUSE_BIG_WRITES | FUSE_DONT_MASK |
FUSE_SPLICE_WRITE | FUSE_SPLICE_MOVE | FUSE_SPLICE_READ |
@@ -976,27 +970,18 @@
static int fuse_bdi_init(struct fuse_conn *fc, struct super_block *sb)
{
int err;
+ char *suffix = "";
- fc->bdi.name = "fuse";
- fc->bdi.ra_pages = (VM_MAX_READAHEAD * 1024) / PAGE_SIZE;
+ if (sb->s_bdev)
+ suffix = "-fuseblk";
+ err = super_setup_bdi_name(sb, "%u:%u%s", MAJOR(fc->dev),
+ MINOR(fc->dev), suffix);
+ if (err)
+ return err;
+
+ sb->s_bdi->ra_pages = (VM_MAX_READAHEAD * 1024) / PAGE_SIZE;
/* fuse does it's own writeback accounting */
- fc->bdi.capabilities = BDI_CAP_NO_ACCT_WB | BDI_CAP_STRICTLIMIT;
-
- err = bdi_init(&fc->bdi);
- if (err)
- return err;
-
- fc->bdi_initialized = 1;
-
- if (sb->s_bdev) {
- err = bdi_register(&fc->bdi, NULL, "%u:%u-fuseblk",
- MAJOR(fc->dev), MINOR(fc->dev));
- } else {
- err = bdi_register_dev(&fc->bdi, fc->dev);
- }
-
- if (err)
- return err;
+ sb->s_bdi->capabilities = BDI_CAP_NO_ACCT_WB | BDI_CAP_STRICTLIMIT;
/*
* For a single fuse filesystem use max 1% of dirty +
@@ -1010,7 +995,7 @@
*
* /sys/class/bdi/<bdi>/max_ratio
*/
- bdi_set_max_ratio(&fc->bdi, 1);
+ bdi_set_max_ratio(sb->s_bdi, 1);
return 0;
}
@@ -1113,8 +1098,6 @@
if (err)
goto err_dev_free;
- sb->s_bdi = &fc->bdi;
-
/* Handle umasking inside the fuse code */
if (sb->s_flags & MS_POSIXACL)
fc->dont_mask = 1;
@@ -1182,7 +1165,6 @@
err_dev_free:
fuse_dev_free(fud);
err_put_conn:
- fuse_bdi_destroy(fc);
fuse_conn_put(fc);
err_fput:
fput(file);
diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c
index b108e7b..ed67548 100644
--- a/fs/gfs2/ops_fstype.c
+++ b/fs/gfs2/ops_fstype.c
@@ -23,6 +23,7 @@
#include <linux/quotaops.h>
#include <linux/lockdep.h>
#include <linux/module.h>
+#include <linux/backing-dev.h>
#include "gfs2.h"
#include "incore.h"
@@ -1222,12 +1223,7 @@
{
s->s_bdev = data;
s->s_dev = s->s_bdev->bd_dev;
-
- /*
- * We set the bdi here to the queue backing, file systems can
- * overwrite this in ->fill_super()
- */
- s->s_bdi = bdev_get_queue(s->s_bdev)->backing_dev_info;
+ s->s_bdi = bdi_get(s->s_bdev->bd_bdi);
return 0;
}
diff --git a/fs/internal.h b/fs/internal.h
index 11c6d89..076751d 100644
--- a/fs/internal.h
+++ b/fs/internal.h
@@ -108,8 +108,6 @@
extern struct file *do_file_open_root(struct dentry *, struct vfsmount *,
const char *, const struct open_flags *);
-extern long do_handle_open(int mountdirfd,
- struct file_handle __user *ufh, int open_flag);
extern int open_check_o_direct(struct file *f);
extern int vfs_open(const struct path *, struct file *, const struct cred *);
extern struct file *filp_clone_open(struct file *);
diff --git a/fs/ncpfs/inode.c b/fs/ncpfs/inode.c
index d560609..6d0f14c 100644
--- a/fs/ncpfs/inode.c
+++ b/fs/ncpfs/inode.c
@@ -554,12 +554,11 @@
sb->s_magic = NCP_SUPER_MAGIC;
sb->s_op = &ncp_sops;
sb->s_d_op = &ncp_dentry_operations;
- sb->s_bdi = &server->bdi;
server = NCP_SBP(sb);
memset(server, 0, sizeof(*server));
- error = bdi_setup_and_register(&server->bdi, "ncpfs");
+ error = super_setup_bdi(sb);
if (error)
goto out_fput;
@@ -568,7 +567,7 @@
if (data.info_fd != -1) {
struct socket *info_sock = sockfd_lookup(data.info_fd, &error);
if (!info_sock)
- goto out_bdi;
+ goto out_fput;
server->info_sock = info_sock;
error = -EBADFD;
if (info_sock->type != SOCK_STREAM)
@@ -746,8 +745,6 @@
out_fput2:
if (server->info_sock)
sockfd_put(server->info_sock);
-out_bdi:
- bdi_destroy(&server->bdi);
out_fput:
sockfd_put(sock);
out:
@@ -788,7 +785,6 @@
kill_pid(server->m.wdog_pid, SIGTERM, 1);
put_pid(server->m.wdog_pid);
- bdi_destroy(&server->bdi);
kfree(server->priv.data);
kfree(server->auth.object_name);
vfree(server->rxbuf);
diff --git a/fs/ncpfs/ncp_fs_sb.h b/fs/ncpfs/ncp_fs_sb.h
index 55e26fd..366fd63c 100644
--- a/fs/ncpfs/ncp_fs_sb.h
+++ b/fs/ncpfs/ncp_fs_sb.h
@@ -143,7 +143,6 @@
size_t len;
__u8 data[128];
} unexpected_packet;
- struct backing_dev_info bdi;
};
extern void ncp_tcp_rcv_proc(struct work_struct *work);
diff --git a/fs/nfs/client.c b/fs/nfs/client.c
index 390ada8..04d15a0 100644
--- a/fs/nfs/client.c
+++ b/fs/nfs/client.c
@@ -761,9 +761,6 @@
server->rsize = NFS_MAX_FILE_IO_SIZE;
server->rpages = (server->rsize + PAGE_SIZE - 1) >> PAGE_SHIFT;
- server->backing_dev_info.name = "nfs";
- server->backing_dev_info.ra_pages = server->rpages * NFS_MAX_READAHEAD;
-
if (server->wsize > max_rpc_payload)
server->wsize = max_rpc_payload;
if (server->wsize > NFS_MAX_FILE_IO_SIZE)
@@ -917,12 +914,6 @@
return NULL;
}
- if (bdi_init(&server->backing_dev_info)) {
- nfs_free_iostats(server->io_stats);
- kfree(server);
- return NULL;
- }
-
ida_init(&server->openowner_id);
ida_init(&server->lockowner_id);
pnfs_init_server(server);
@@ -953,7 +944,6 @@
ida_destroy(&server->lockowner_id);
ida_destroy(&server->openowner_id);
nfs_free_iostats(server->io_stats);
- bdi_destroy(&server->backing_dev_info);
kfree(server);
nfs_release_automount_timer();
dprintk("<-- nfs_free_server()\n");
diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c
index aab32fc..c1b5fed 100644
--- a/fs/nfs/direct.c
+++ b/fs/nfs/direct.c
@@ -537,7 +537,7 @@
if (put_dreq(dreq))
nfs_direct_complete(dreq);
- return 0;
+ return requested_bytes;
}
/**
@@ -566,7 +566,7 @@
struct inode *inode = mapping->host;
struct nfs_direct_req *dreq;
struct nfs_lock_context *l_ctx;
- ssize_t result = -EINVAL;
+ ssize_t result = -EINVAL, requested;
size_t count = iov_iter_count(iter);
nfs_add_stats(mapping->host, NFSIOS_DIRECTREADBYTES, count);
@@ -600,14 +600,19 @@
nfs_start_io_direct(inode);
NFS_I(inode)->read_io += count;
- result = nfs_direct_read_schedule_iovec(dreq, iter, iocb->ki_pos);
+ requested = nfs_direct_read_schedule_iovec(dreq, iter, iocb->ki_pos);
nfs_end_io_direct(inode);
- if (!result) {
+ if (requested > 0) {
result = nfs_direct_wait(dreq);
- if (result > 0)
+ if (result > 0) {
+ requested -= result;
iocb->ki_pos += result;
+ }
+ iov_iter_revert(iter, requested);
+ } else {
+ result = requested;
}
out_release:
@@ -954,7 +959,7 @@
if (put_dreq(dreq))
nfs_direct_write_complete(dreq);
- return 0;
+ return requested_bytes;
}
/**
@@ -979,7 +984,7 @@
*/
ssize_t nfs_file_direct_write(struct kiocb *iocb, struct iov_iter *iter)
{
- ssize_t result = -EINVAL;
+ ssize_t result = -EINVAL, requested;
size_t count;
struct file *file = iocb->ki_filp;
struct address_space *mapping = file->f_mapping;
@@ -1022,7 +1027,7 @@
nfs_start_io_direct(inode);
- result = nfs_direct_write_schedule_iovec(dreq, iter, pos);
+ requested = nfs_direct_write_schedule_iovec(dreq, iter, pos);
if (mapping->nrpages) {
invalidate_inode_pages2_range(mapping,
@@ -1031,13 +1036,17 @@
nfs_end_io_direct(inode);
- if (!result) {
+ if (requested > 0) {
result = nfs_direct_wait(dreq);
if (result > 0) {
+ requested -= result;
iocb->ki_pos = pos + result;
/* XXX: should check the generic_write_sync retval */
generic_write_sync(iocb, result);
}
+ iov_iter_revert(iter, requested);
+ } else {
+ result = requested;
}
out_release:
nfs_direct_req_release(dreq);
diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 7b38fed..9dc65d7 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -139,7 +139,7 @@
};
struct nfs_mount_info {
- void (*fill_super)(struct super_block *, struct nfs_mount_info *);
+ int (*fill_super)(struct super_block *, struct nfs_mount_info *);
int (*set_security)(struct super_block *, struct dentry *, struct nfs_mount_info *);
struct nfs_parsed_mount_data *parsed;
struct nfs_clone_mount *cloned;
@@ -407,7 +407,7 @@
struct dentry * nfs_xdev_mount_common(struct file_system_type *, int,
const char *, struct nfs_mount_info *);
void nfs_kill_super(struct super_block *);
-void nfs_fill_super(struct super_block *, struct nfs_mount_info *);
+int nfs_fill_super(struct super_block *, struct nfs_mount_info *);
extern struct rpc_stat nfs_rpcstat;
@@ -458,7 +458,7 @@
extern void nfs_pageio_reset_read_mds(struct nfs_pageio_descriptor *pgio);
/* super.c */
-void nfs_clone_super(struct super_block *, struct nfs_mount_info *);
+int nfs_clone_super(struct super_block *, struct nfs_mount_info *);
void nfs_umount_begin(struct super_block *);
int nfs_statfs(struct dentry *, struct kstatfs *);
int nfs_show_options(struct seq_file *, struct dentry *);
diff --git a/fs/nfs/super.c b/fs/nfs/super.c
index 54e0f9f..dc69314 100644
--- a/fs/nfs/super.c
+++ b/fs/nfs/super.c
@@ -2315,18 +2315,17 @@
sb->s_blocksize = nfs_block_bits(server->wsize,
&sb->s_blocksize_bits);
- sb->s_bdi = &server->backing_dev_info;
-
nfs_super_set_maxbytes(sb, server->maxfilesize);
}
/*
* Finish setting up an NFS2/3 superblock
*/
-void nfs_fill_super(struct super_block *sb, struct nfs_mount_info *mount_info)
+int nfs_fill_super(struct super_block *sb, struct nfs_mount_info *mount_info)
{
struct nfs_parsed_mount_data *data = mount_info->parsed;
struct nfs_server *server = NFS_SB(sb);
+ int ret;
sb->s_blocksize_bits = 0;
sb->s_blocksize = 0;
@@ -2344,13 +2343,21 @@
}
nfs_initialise_sb(sb);
+
+ ret = super_setup_bdi_name(sb, "%u:%u", MAJOR(server->s_dev),
+ MINOR(server->s_dev));
+ if (ret)
+ return ret;
+ sb->s_bdi->ra_pages = server->rpages * NFS_MAX_READAHEAD;
+ return 0;
+
}
EXPORT_SYMBOL_GPL(nfs_fill_super);
/*
* Finish setting up a cloned NFS2/3/4 superblock
*/
-void nfs_clone_super(struct super_block *sb, struct nfs_mount_info *mount_info)
+int nfs_clone_super(struct super_block *sb, struct nfs_mount_info *mount_info)
{
const struct super_block *old_sb = mount_info->cloned->sb;
struct nfs_server *server = NFS_SB(sb);
@@ -2370,6 +2377,10 @@
}
nfs_initialise_sb(sb);
+
+ sb->s_bdi = bdi_get(old_sb->s_bdi);
+
+ return 0;
}
static int nfs_compare_mount_options(const struct super_block *s, const struct nfs_server *b, int flags)
@@ -2522,11 +2533,6 @@
}
#endif
-static int nfs_bdi_register(struct nfs_server *server)
-{
- return bdi_register_dev(&server->backing_dev_info, server->s_dev);
-}
-
int nfs_set_sb_security(struct super_block *s, struct dentry *mntroot,
struct nfs_mount_info *mount_info)
{
@@ -2594,17 +2600,14 @@
nfs_free_server(server);
server = NULL;
} else {
- error = nfs_bdi_register(server);
- if (error) {
- mntroot = ERR_PTR(error);
- goto error_splat_super;
- }
server->super = s;
}
if (!s->s_root) {
/* initial superblock/root creation */
- mount_info->fill_super(s, mount_info);
+ error = mount_info->fill_super(s, mount_info);
+ if (error)
+ goto error_splat_super;
nfs_get_cache_cookie(s, mount_info->parsed, mount_info->cloned);
}
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index abb2c8a..cc341fc 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -263,16 +263,15 @@
static void nfs_set_page_writeback(struct page *page)
{
- struct nfs_server *nfss = NFS_SERVER(page_file_mapping(page)->host);
+ struct inode *inode = page_file_mapping(page)->host;
+ struct nfs_server *nfss = NFS_SERVER(inode);
int ret = test_set_page_writeback(page);
WARN_ON_ONCE(ret != 0);
if (atomic_long_inc_return(&nfss->writeback) >
- NFS_CONGESTION_ON_THRESH) {
- set_bdi_congested(&nfss->backing_dev_info,
- BLK_RW_ASYNC);
- }
+ NFS_CONGESTION_ON_THRESH)
+ set_bdi_congested(inode_to_bdi(inode), BLK_RW_ASYNC);
}
static void nfs_end_page_writeback(struct nfs_page *req)
@@ -285,7 +284,7 @@
end_page_writeback(req->wb_page);
if (atomic_long_dec_return(&nfss->writeback) < NFS_CONGESTION_OFF_THRESH)
- clear_bdi_congested(&nfss->backing_dev_info, BLK_RW_ASYNC);
+ clear_bdi_congested(inode_to_bdi(inode), BLK_RW_ASYNC);
}
@@ -1808,7 +1807,7 @@
}
nfss = NFS_SERVER(data->inode);
if (atomic_long_read(&nfss->writeback) < NFS_CONGESTION_OFF_THRESH)
- clear_bdi_congested(&nfss->backing_dev_info, BLK_RW_ASYNC);
+ clear_bdi_congested(inode_to_bdi(data->inode), BLK_RW_ASYNC);
nfs_init_cinfo(&cinfo, data->inode, data->dreq);
nfs_commit_end(cinfo.mds);
diff --git a/fs/nfsd/blocklayout.c b/fs/nfsd/blocklayout.c
index 92b4b41..fb5213a 100644
--- a/fs/nfsd/blocklayout.c
+++ b/fs/nfsd/blocklayout.c
@@ -242,10 +242,11 @@
req->cmd[4] = bufflen & 0xff;
req->cmd_len = COMMAND_SIZE(INQUIRY);
- error = blk_execute_rq(rq->q, NULL, rq, 1);
- if (error) {
+ blk_execute_rq(rq->q, NULL, rq, 1);
+ if (req->result) {
pr_err("pNFS: INQUIRY 0x83 failed with: %x\n",
- rq->errors);
+ req->result);
+ error = -EIO;
goto out_put_request;
}
diff --git a/fs/nfsd/nfs3xdr.c b/fs/nfsd/nfs3xdr.c
index dba2ff8..4523346 100644
--- a/fs/nfsd/nfs3xdr.c
+++ b/fs/nfsd/nfs3xdr.c
@@ -358,6 +358,8 @@
{
unsigned int len, v, hdr, dlen;
u32 max_blocksize = svc_max_payload(rqstp);
+ struct kvec *head = rqstp->rq_arg.head;
+ struct kvec *tail = rqstp->rq_arg.tail;
p = decode_fh(p, &args->fh);
if (!p)
@@ -367,6 +369,8 @@
args->count = ntohl(*p++);
args->stable = ntohl(*p++);
len = args->len = ntohl(*p++);
+ if ((void *)p > head->iov_base + head->iov_len)
+ return 0;
/*
* The count must equal the amount of data passed.
*/
@@ -377,9 +381,8 @@
* Check to make sure that we got the right number of
* bytes.
*/
- hdr = (void*)p - rqstp->rq_arg.head[0].iov_base;
- dlen = rqstp->rq_arg.head[0].iov_len + rqstp->rq_arg.page_len
- + rqstp->rq_arg.tail[0].iov_len - hdr;
+ hdr = (void*)p - head->iov_base;
+ dlen = head->iov_len + rqstp->rq_arg.page_len + tail->iov_len - hdr;
/*
* Round the length of the data which was specified up to
* the next multiple of XDR units and then compare that
@@ -396,7 +399,7 @@
len = args->len = max_blocksize;
}
rqstp->rq_vec[0].iov_base = (void*)p;
- rqstp->rq_vec[0].iov_len = rqstp->rq_arg.head[0].iov_len - hdr;
+ rqstp->rq_vec[0].iov_len = head->iov_len - hdr;
v = 0;
while (len > rqstp->rq_vec[v].iov_len) {
len -= rqstp->rq_vec[v].iov_len;
@@ -471,6 +474,8 @@
/* first copy and check from the first page */
old = (char*)p;
vec = &rqstp->rq_arg.head[0];
+ if ((void *)old > vec->iov_base + vec->iov_len)
+ return 0;
avail = vec->iov_len - (old - (char*)vec->iov_base);
while (len && avail && *old) {
*new++ = *old++;
diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index 31e1f95..59979f0 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -747,6 +747,37 @@
return nfserr;
}
+/*
+ * A write procedure can have a large argument, and a read procedure can
+ * have a large reply, but no NFSv2 or NFSv3 procedure has argument and
+ * reply that can both be larger than a page. The xdr code has taken
+ * advantage of this assumption to be a sloppy about bounds checking in
+ * some cases. Pending a rewrite of the NFSv2/v3 xdr code to fix that
+ * problem, we enforce these assumptions here:
+ */
+static bool nfs_request_too_big(struct svc_rqst *rqstp,
+ struct svc_procedure *proc)
+{
+ /*
+ * The ACL code has more careful bounds-checking and is not
+ * susceptible to this problem:
+ */
+ if (rqstp->rq_prog != NFS_PROGRAM)
+ return false;
+ /*
+ * Ditto NFSv4 (which can in theory have argument and reply both
+ * more than a page):
+ */
+ if (rqstp->rq_vers >= 4)
+ return false;
+ /* The reply will be small, we're OK: */
+ if (proc->pc_xdrressize > 0 &&
+ proc->pc_xdrressize < XDR_QUADLEN(PAGE_SIZE))
+ return false;
+
+ return rqstp->rq_arg.len > PAGE_SIZE;
+}
+
int
nfsd_dispatch(struct svc_rqst *rqstp, __be32 *statp)
{
@@ -759,6 +790,11 @@
rqstp->rq_vers, rqstp->rq_proc);
proc = rqstp->rq_procinfo;
+ if (nfs_request_too_big(rqstp, proc)) {
+ dprintk("nfsd: NFSv%d argument too large\n", rqstp->rq_vers);
+ *statp = rpc_garbage_args;
+ return 1;
+ }
/*
* Give the xdr decoder a chance to change this if it wants
* (necessary in the NFSv4.0 compound case)
diff --git a/fs/nfsd/nfsxdr.c b/fs/nfsd/nfsxdr.c
index 41b468a..de07ff6 100644
--- a/fs/nfsd/nfsxdr.c
+++ b/fs/nfsd/nfsxdr.c
@@ -280,6 +280,7 @@
struct nfsd_writeargs *args)
{
unsigned int len, hdr, dlen;
+ struct kvec *head = rqstp->rq_arg.head;
int v;
p = decode_fh(p, &args->fh);
@@ -300,9 +301,10 @@
* Check to make sure that we got the right number of
* bytes.
*/
- hdr = (void*)p - rqstp->rq_arg.head[0].iov_base;
- dlen = rqstp->rq_arg.head[0].iov_len + rqstp->rq_arg.page_len
- - hdr;
+ hdr = (void*)p - head->iov_base;
+ if (hdr > head->iov_len)
+ return 0;
+ dlen = head->iov_len + rqstp->rq_arg.page_len - hdr;
/*
* Round the length of the data which was specified up to
@@ -316,7 +318,7 @@
return 0;
rqstp->rq_vec[0].iov_base = (void*)p;
- rqstp->rq_vec[0].iov_len = rqstp->rq_arg.head[0].iov_len - hdr;
+ rqstp->rq_vec[0].iov_len = head->iov_len - hdr;
v = 0;
while (len > rqstp->rq_vec[v].iov_len) {
len -= rqstp->rq_vec[v].iov_len;
diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
index 19d50f6..9aaf6ca 100644
--- a/fs/nfsd/vfs.c
+++ b/fs/nfsd/vfs.c
@@ -1004,7 +1004,7 @@
else
err = nfserrno(host_err);
if (test_bit(RQ_LOCAL, &rqstp->rq_flags))
- tsk_restore_flags(current, pflags, PF_LESS_THROTTLE);
+ current_restore_flags(pflags, PF_LESS_THROTTLE);
return err;
}
diff --git a/fs/nilfs2/super.c b/fs/nilfs2/super.c
index e1872f3..9266829 100644
--- a/fs/nilfs2/super.c
+++ b/fs/nilfs2/super.c
@@ -1068,7 +1068,7 @@
sb->s_time_gran = 1;
sb->s_max_links = NILFS_LINK_MAX;
- sb->s_bdi = bdev_get_queue(sb->s_bdev)->backing_dev_info;
+ sb->s_bdi = bdi_get(sb->s_bdev->bd_bdi);
err = load_nilfs(nilfs, sb);
if (err)
diff --git a/fs/ocfs2/cluster/tcp.c b/fs/ocfs2/cluster/tcp.c
index d0ab7e5..5b51c31 100644
--- a/fs/ocfs2/cluster/tcp.c
+++ b/fs/ocfs2/cluster/tcp.c
@@ -1460,27 +1460,10 @@
static int o2net_set_nodelay(struct socket *sock)
{
- int ret, val = 1;
- mm_segment_t oldfs;
+ int val = 1;
- oldfs = get_fs();
- set_fs(KERNEL_DS);
-
- /*
- * Dear unsuspecting programmer,
- *
- * Don't use sock_setsockopt() for SOL_TCP. It doesn't check its level
- * argument and assumes SOL_SOCKET so, say, your TCP_NODELAY will
- * silently turn into SO_DEBUG.
- *
- * Yours,
- * Keeper of hilariously fragile interfaces.
- */
- ret = sock->ops->setsockopt(sock, SOL_TCP, TCP_NODELAY,
- (char __user *)&val, sizeof(val));
-
- set_fs(oldfs);
- return ret;
+ return kernel_setsockopt(sock, SOL_TCP, TCP_NODELAY,
+ (void *)&val, sizeof(val));
}
static int o2net_set_usertimeout(struct socket *sock)
@@ -1488,7 +1471,7 @@
int user_timeout = O2NET_TCP_USER_TIMEOUT;
return kernel_setsockopt(sock, SOL_TCP, TCP_USER_TIMEOUT,
- (char *)&user_timeout, sizeof(user_timeout));
+ (void *)&user_timeout, sizeof(user_timeout));
}
static void o2net_initialize_handshake(void)
diff --git a/fs/open.c b/fs/open.c
index 949cef2..4d23f72 100644
--- a/fs/open.c
+++ b/fs/open.c
@@ -1078,6 +1078,26 @@
return do_sys_open(dfd, filename, flags, mode);
}
+#ifdef CONFIG_COMPAT
+/*
+ * Exactly like sys_open(), except that it doesn't set the
+ * O_LARGEFILE flag.
+ */
+COMPAT_SYSCALL_DEFINE3(open, const char __user *, filename, int, flags, umode_t, mode)
+{
+ return do_sys_open(AT_FDCWD, filename, flags, mode);
+}
+
+/*
+ * Exactly like sys_openat(), except that it doesn't set the
+ * O_LARGEFILE flag.
+ */
+COMPAT_SYSCALL_DEFINE4(openat, int, dfd, const char __user *, filename, int, flags, umode_t, mode)
+{
+ return do_sys_open(dfd, filename, flags, mode);
+}
+#endif
+
#ifndef __alpha__
/*
diff --git a/fs/orangefs/file.c b/fs/orangefs/file.c
index e6bbc80..1cd37eb 100644
--- a/fs/orangefs/file.c
+++ b/fs/orangefs/file.c
@@ -114,7 +114,6 @@
struct orangefs_inode_s *orangefs_inode = ORANGEFS_I(inode);
struct orangefs_khandle *handle = &orangefs_inode->refn.khandle;
struct orangefs_kernel_op_s *new_op = NULL;
- struct iov_iter saved = *iter;
int buffer_index = -1;
ssize_t ret;
@@ -193,7 +192,7 @@
orangefs_bufmap_put(buffer_index);
buffer_index = -1;
if (type == ORANGEFS_IO_WRITE)
- *iter = saved;
+ iov_iter_revert(iter, total_size);
gossip_debug(GOSSIP_FILE_DEBUG,
"%s:going to repopulate_shared_memory.\n",
__func__);
diff --git a/fs/orangefs/orangefs-bufmap.c b/fs/orangefs/orangefs-bufmap.c
index 6333cbb..83b5060 100644
--- a/fs/orangefs/orangefs-bufmap.c
+++ b/fs/orangefs/orangefs-bufmap.c
@@ -521,13 +521,11 @@
size_t n = size;
if (n > PAGE_SIZE)
n = PAGE_SIZE;
- n = copy_page_from_iter(page, 0, n, iter);
- if (!n)
+ if (copy_page_from_iter(page, 0, n, iter) != n)
return -EFAULT;
size -= n;
}
return 0;
-
}
/*
diff --git a/fs/proc/base.c b/fs/proc/base.c
index c87b6b9..9e3ac5c 100644
--- a/fs/proc/base.c
+++ b/fs/proc/base.c
@@ -2834,6 +2834,15 @@
return err;
}
+#ifdef CONFIG_LIVEPATCH
+static int proc_pid_patch_state(struct seq_file *m, struct pid_namespace *ns,
+ struct pid *pid, struct task_struct *task)
+{
+ seq_printf(m, "%d\n", task->patch_state);
+ return 0;
+}
+#endif /* CONFIG_LIVEPATCH */
+
/*
* Thread groups
*/
@@ -2933,6 +2942,9 @@
REG("timers", S_IRUGO, proc_timers_operations),
#endif
REG("timerslack_ns", S_IRUGO|S_IWUGO, proc_pid_set_timerslack_ns_operations),
+#ifdef CONFIG_LIVEPATCH
+ ONE("patch_state", S_IRUSR, proc_pid_patch_state),
+#endif
};
static int proc_tgid_base_readdir(struct file *file, struct dir_context *ctx)
@@ -3315,6 +3327,9 @@
REG("projid_map", S_IRUGO|S_IWUSR, proc_projid_map_operations),
REG("setgroups", S_IRUGO|S_IWUSR, proc_setgroups_operations),
#endif
+#ifdef CONFIG_LIVEPATCH
+ ONE("patch_state", S_IRUSR, proc_pid_patch_state),
+#endif
};
static int proc_tid_base_readdir(struct file *file, struct dir_context *ctx)
diff --git a/fs/pstore/ftrace.c b/fs/pstore/ftrace.c
index 899d0ba..06aab07 100644
--- a/fs/pstore/ftrace.c
+++ b/fs/pstore/ftrace.c
@@ -37,6 +37,12 @@
{
unsigned long flags;
struct pstore_ftrace_record rec = {};
+ struct pstore_record record = {
+ .type = PSTORE_TYPE_FTRACE,
+ .buf = (char *)&rec,
+ .size = sizeof(rec),
+ .psi = psinfo,
+ };
if (unlikely(oops_in_progress))
return;
@@ -47,8 +53,7 @@
rec.parent_ip = parent_ip;
pstore_ftrace_write_timestamp(&rec, pstore_ftrace_stamp++);
pstore_ftrace_encode_cpu(&rec, raw_smp_processor_id());
- psinfo->write_buf(PSTORE_TYPE_FTRACE, 0, NULL, 0, (void *)&rec,
- 0, sizeof(rec), psinfo);
+ psinfo->write(&record);
local_irq_restore(flags);
}
@@ -117,7 +122,7 @@
{
struct dentry *file;
- if (!psinfo->write_buf)
+ if (!psinfo->write)
return;
pstore_ftrace_dir = debugfs_create_dir("pstore", NULL);
diff --git a/fs/pstore/inode.c b/fs/pstore/inode.c
index 57c0646..792a4e5 100644
--- a/fs/pstore/inode.c
+++ b/fs/pstore/inode.c
@@ -47,12 +47,8 @@
struct pstore_private {
struct list_head list;
- struct pstore_info *psi;
- enum pstore_type_id type;
- u64 id;
- int count;
- ssize_t size;
- char data[];
+ struct pstore_record *record;
+ size_t total_size;
};
struct pstore_ftrace_seq_data {
@@ -63,6 +59,17 @@
#define REC_SIZE sizeof(struct pstore_ftrace_record)
+static void free_pstore_private(struct pstore_private *private)
+{
+ if (!private)
+ return;
+ if (private->record) {
+ kfree(private->record->buf);
+ kfree(private->record);
+ }
+ kfree(private);
+}
+
static void *pstore_ftrace_seq_start(struct seq_file *s, loff_t *pos)
{
struct pstore_private *ps = s->private;
@@ -72,9 +79,9 @@
if (!data)
return NULL;
- data->off = ps->size % REC_SIZE;
+ data->off = ps->total_size % REC_SIZE;
data->off += *pos * REC_SIZE;
- if (data->off + REC_SIZE > ps->size) {
+ if (data->off + REC_SIZE > ps->total_size) {
kfree(data);
return NULL;
}
@@ -94,7 +101,7 @@
struct pstore_ftrace_seq_data *data = v;
data->off += REC_SIZE;
- if (data->off + REC_SIZE > ps->size)
+ if (data->off + REC_SIZE > ps->total_size)
return NULL;
(*pos)++;
@@ -105,7 +112,9 @@
{
struct pstore_private *ps = s->private;
struct pstore_ftrace_seq_data *data = v;
- struct pstore_ftrace_record *rec = (void *)(ps->data + data->off);
+ struct pstore_ftrace_record *rec;
+
+ rec = (struct pstore_ftrace_record *)(ps->record->buf + data->off);
seq_printf(s, "CPU:%d ts:%llu %08lx %08lx %pf <- %pF\n",
pstore_ftrace_decode_cpu(rec),
@@ -125,7 +134,7 @@
static int pstore_check_syslog_permissions(struct pstore_private *ps)
{
- switch (ps->type) {
+ switch (ps->record->type) {
case PSTORE_TYPE_DMESG:
case PSTORE_TYPE_CONSOLE:
return check_syslog_permissions(SYSLOG_ACTION_READ_ALL,
@@ -141,9 +150,10 @@
struct seq_file *sf = file->private_data;
struct pstore_private *ps = sf->private;
- if (ps->type == PSTORE_TYPE_FTRACE)
+ if (ps->record->type == PSTORE_TYPE_FTRACE)
return seq_read(file, userbuf, count, ppos);
- return simple_read_from_buffer(userbuf, count, ppos, ps->data, ps->size);
+ return simple_read_from_buffer(userbuf, count, ppos,
+ ps->record->buf, ps->total_size);
}
static int pstore_file_open(struct inode *inode, struct file *file)
@@ -157,7 +167,7 @@
if (err)
return err;
- if (ps->type == PSTORE_TYPE_FTRACE)
+ if (ps->record->type == PSTORE_TYPE_FTRACE)
sops = &pstore_ftrace_seq_ops;
err = seq_open(file, sops);
@@ -193,20 +203,19 @@
static int pstore_unlink(struct inode *dir, struct dentry *dentry)
{
struct pstore_private *p = d_inode(dentry)->i_private;
+ struct pstore_record *record = p->record;
int err;
err = pstore_check_syslog_permissions(p);
if (err)
return err;
- if (p->psi->erase) {
- mutex_lock(&p->psi->read_mutex);
- p->psi->erase(p->type, p->id, p->count,
- d_inode(dentry)->i_ctime, p->psi);
- mutex_unlock(&p->psi->read_mutex);
- } else {
+ if (!record->psi->erase)
return -EPERM;
- }
+
+ mutex_lock(&record->psi->read_mutex);
+ record->psi->erase(record);
+ mutex_unlock(&record->psi->read_mutex);
return simple_unlink(dir, dentry);
}
@@ -221,7 +230,7 @@
spin_lock_irqsave(&allpstore_lock, flags);
list_del(&p->list);
spin_unlock_irqrestore(&allpstore_lock, flags);
- kfree(p);
+ free_pstore_private(p);
}
}
@@ -302,23 +311,23 @@
* Load it up with "size" bytes of data from "buf".
* Set the mtime & ctime to the date that this record was originally stored.
*/
-int pstore_mkfile(enum pstore_type_id type, char *psname, u64 id, int count,
- char *data, bool compressed, size_t size,
- struct timespec time, struct pstore_info *psi)
+int pstore_mkfile(struct dentry *root, struct pstore_record *record)
{
- struct dentry *root = pstore_sb->s_root;
struct dentry *dentry;
struct inode *inode;
int rc = 0;
char name[PSTORE_NAMELEN];
struct pstore_private *private, *pos;
unsigned long flags;
+ size_t size = record->size + record->ecc_notice_size;
+
+ WARN_ON(!inode_is_locked(d_inode(root)));
spin_lock_irqsave(&allpstore_lock, flags);
list_for_each_entry(pos, &allpstore, list) {
- if (pos->type == type &&
- pos->id == id &&
- pos->psi == psi) {
+ if (pos->record->type == record->type &&
+ pos->record->id == record->id &&
+ pos->record->psi == record->psi) {
rc = -EEXIST;
break;
}
@@ -328,72 +337,74 @@
return rc;
rc = -ENOMEM;
- inode = pstore_get_inode(pstore_sb);
+ inode = pstore_get_inode(root->d_sb);
if (!inode)
goto fail;
inode->i_mode = S_IFREG | 0444;
inode->i_fop = &pstore_file_operations;
- private = kmalloc(sizeof *private + size, GFP_KERNEL);
+ private = kzalloc(sizeof(*private), GFP_KERNEL);
if (!private)
goto fail_alloc;
- private->type = type;
- private->id = id;
- private->count = count;
- private->psi = psi;
+ private->record = record;
- switch (type) {
+ switch (record->type) {
case PSTORE_TYPE_DMESG:
scnprintf(name, sizeof(name), "dmesg-%s-%lld%s",
- psname, id, compressed ? ".enc.z" : "");
+ record->psi->name, record->id,
+ record->compressed ? ".enc.z" : "");
break;
case PSTORE_TYPE_CONSOLE:
- scnprintf(name, sizeof(name), "console-%s-%lld", psname, id);
+ scnprintf(name, sizeof(name), "console-%s-%lld",
+ record->psi->name, record->id);
break;
case PSTORE_TYPE_FTRACE:
- scnprintf(name, sizeof(name), "ftrace-%s-%lld", psname, id);
+ scnprintf(name, sizeof(name), "ftrace-%s-%lld",
+ record->psi->name, record->id);
break;
case PSTORE_TYPE_MCE:
- scnprintf(name, sizeof(name), "mce-%s-%lld", psname, id);
+ scnprintf(name, sizeof(name), "mce-%s-%lld",
+ record->psi->name, record->id);
break;
case PSTORE_TYPE_PPC_RTAS:
- scnprintf(name, sizeof(name), "rtas-%s-%lld", psname, id);
+ scnprintf(name, sizeof(name), "rtas-%s-%lld",
+ record->psi->name, record->id);
break;
case PSTORE_TYPE_PPC_OF:
scnprintf(name, sizeof(name), "powerpc-ofw-%s-%lld",
- psname, id);
+ record->psi->name, record->id);
break;
case PSTORE_TYPE_PPC_COMMON:
scnprintf(name, sizeof(name), "powerpc-common-%s-%lld",
- psname, id);
+ record->psi->name, record->id);
break;
case PSTORE_TYPE_PMSG:
- scnprintf(name, sizeof(name), "pmsg-%s-%lld", psname, id);
+ scnprintf(name, sizeof(name), "pmsg-%s-%lld",
+ record->psi->name, record->id);
break;
case PSTORE_TYPE_PPC_OPAL:
- sprintf(name, "powerpc-opal-%s-%lld", psname, id);
+ scnprintf(name, sizeof(name), "powerpc-opal-%s-%lld",
+ record->psi->name, record->id);
break;
case PSTORE_TYPE_UNKNOWN:
- scnprintf(name, sizeof(name), "unknown-%s-%lld", psname, id);
+ scnprintf(name, sizeof(name), "unknown-%s-%lld",
+ record->psi->name, record->id);
break;
default:
scnprintf(name, sizeof(name), "type%d-%s-%lld",
- type, psname, id);
+ record->type, record->psi->name, record->id);
break;
}
- inode_lock(d_inode(root));
-
dentry = d_alloc_name(root, name);
if (!dentry)
- goto fail_lockedalloc;
+ goto fail_private;
- memcpy(private->data, data, size);
- inode->i_size = private->size = size;
+ inode->i_size = private->total_size = size;
inode->i_private = private;
- if (time.tv_sec)
- inode->i_mtime = inode->i_ctime = time;
+ if (record->time.tv_sec)
+ inode->i_mtime = inode->i_ctime = record->time;
d_add(dentry, inode);
@@ -401,13 +412,10 @@
list_add(&private->list, &allpstore);
spin_unlock_irqrestore(&allpstore_lock, flags);
- inode_unlock(d_inode(root));
-
return 0;
-fail_lockedalloc:
- inode_unlock(d_inode(root));
- kfree(private);
+fail_private:
+ free_pstore_private(private);
fail_alloc:
iput(inode);
@@ -415,6 +423,27 @@
return rc;
}
+/*
+ * Read all the records from the persistent store. Create
+ * files in our filesystem. Don't warn about -EEXIST errors
+ * when we are re-scanning the backing store looking to add new
+ * error records.
+ */
+void pstore_get_records(int quiet)
+{
+ struct pstore_info *psi = psinfo;
+ struct dentry *root;
+
+ if (!psi || !pstore_sb)
+ return;
+
+ root = pstore_sb->s_root;
+
+ inode_lock(d_inode(root));
+ pstore_get_backend_records(psi, root, quiet);
+ inode_unlock(d_inode(root));
+}
+
static int pstore_fill_super(struct super_block *sb, void *data, int silent)
{
struct inode *inode;
diff --git a/fs/pstore/internal.h b/fs/pstore/internal.h
index da416e6..c416e65 100644
--- a/fs/pstore/internal.h
+++ b/fs/pstore/internal.h
@@ -25,10 +25,10 @@
extern void pstore_set_kmsg_bytes(int);
extern void pstore_get_records(int);
-extern int pstore_mkfile(enum pstore_type_id, char *psname, u64 id,
- int count, char *data, bool compressed,
- size_t size, struct timespec time,
- struct pstore_info *psi);
+extern void pstore_get_backend_records(struct pstore_info *psi,
+ struct dentry *root, int quiet);
+extern int pstore_mkfile(struct dentry *root,
+ struct pstore_record *record);
extern bool pstore_is_mounted(void);
#endif
diff --git a/fs/pstore/platform.c b/fs/pstore/platform.c
index efab7b6..d468eec 100644
--- a/fs/pstore/platform.c
+++ b/fs/pstore/platform.c
@@ -267,7 +267,7 @@
big_oops_buf_sz = 0;
}
-static struct pstore_zbackend backend_zlib = {
+static const struct pstore_zbackend backend_zlib = {
.compress = compress_zlib,
.decompress = decompress_zlib,
.allocate = allocate_zlib,
@@ -328,7 +328,7 @@
big_oops_buf_sz = 0;
}
-static struct pstore_zbackend backend_lzo = {
+static const struct pstore_zbackend backend_lzo = {
.compress = compress_lzo,
.decompress = decompress_lzo,
.allocate = allocate_lzo,
@@ -393,7 +393,7 @@
big_oops_buf_sz = 0;
}
-static struct pstore_zbackend backend_lz4 = {
+static const struct pstore_zbackend backend_lz4 = {
.compress = compress_lz4,
.decompress = decompress_lz4,
.allocate = allocate_lz4,
@@ -402,7 +402,7 @@
};
#endif
-static struct pstore_zbackend *zbackend =
+static const struct pstore_zbackend *zbackend =
#if defined(CONFIG_PSTORE_ZLIB_COMPRESS)
&backend_zlib;
#elif defined(CONFIG_PSTORE_LZO_COMPRESS)
@@ -484,7 +484,6 @@
{
unsigned long total = 0;
const char *why;
- u64 id;
unsigned int part = 1;
unsigned long flags = 0;
int is_locked;
@@ -506,48 +505,59 @@
oopscount++;
while (total < kmsg_bytes) {
char *dst;
- unsigned long size;
- int hsize;
+ size_t dst_size;
+ int header_size;
int zipped_len = -1;
- size_t len;
- bool compressed = false;
- size_t total_len;
+ size_t dump_size;
+ struct pstore_record record = {
+ .type = PSTORE_TYPE_DMESG,
+ .count = oopscount,
+ .reason = reason,
+ .part = part,
+ .compressed = false,
+ .buf = psinfo->buf,
+ .psi = psinfo,
+ };
if (big_oops_buf && is_locked) {
dst = big_oops_buf;
- size = big_oops_buf_sz;
+ dst_size = big_oops_buf_sz;
} else {
dst = psinfo->buf;
- size = psinfo->bufsize;
+ dst_size = psinfo->bufsize;
}
- hsize = sprintf(dst, "%s#%d Part%u\n", why, oopscount, part);
- size -= hsize;
+ /* Write dump header. */
+ header_size = snprintf(dst, dst_size, "%s#%d Part%u\n", why,
+ oopscount, part);
+ dst_size -= header_size;
- if (!kmsg_dump_get_buffer(dumper, true, dst + hsize,
- size, &len))
+ /* Write dump contents. */
+ if (!kmsg_dump_get_buffer(dumper, true, dst + header_size,
+ dst_size, &dump_size))
break;
if (big_oops_buf && is_locked) {
zipped_len = pstore_compress(dst, psinfo->buf,
- hsize + len, psinfo->bufsize);
+ header_size + dump_size,
+ psinfo->bufsize);
if (zipped_len > 0) {
- compressed = true;
- total_len = zipped_len;
+ record.compressed = true;
+ record.size = zipped_len;
} else {
- total_len = copy_kmsg_to_buffer(hsize, len);
+ record.size = copy_kmsg_to_buffer(header_size,
+ dump_size);
}
} else {
- total_len = hsize + len;
+ record.size = header_size + dump_size;
}
- ret = psinfo->write(PSTORE_TYPE_DMESG, reason, &id, part,
- oopscount, compressed, total_len, psinfo);
+ ret = psinfo->write(&record);
if (ret == 0 && reason == KMSG_DUMP_OOPS && pstore_is_mounted())
pstore_new_entry = 1;
- total += total_len;
+ total += record.size;
part++;
}
if (is_locked)
@@ -577,8 +587,11 @@
const char *e = s + c;
while (s < e) {
+ struct pstore_record record = {
+ .type = PSTORE_TYPE_CONSOLE,
+ .psi = psinfo,
+ };
unsigned long flags;
- u64 id;
if (c > psinfo->bufsize)
c = psinfo->bufsize;
@@ -589,8 +602,9 @@
} else {
spin_lock_irqsave(&psinfo->buf_lock, flags);
}
- psinfo->write_buf(PSTORE_TYPE_CONSOLE, 0, &id, 0,
- s, 0, c, psinfo);
+ record.buf = (char *)s;
+ record.size = c;
+ psinfo->write(&record);
spin_unlock_irqrestore(&psinfo->buf_lock, flags);
s += c;
c = e - s;
@@ -618,48 +632,30 @@
static void pstore_unregister_console(void) {}
#endif
-static int pstore_write_compat(enum pstore_type_id type,
- enum kmsg_dump_reason reason,
- u64 *id, unsigned int part, int count,
- bool compressed, size_t size,
- struct pstore_info *psi)
+static int pstore_write_user_compat(struct pstore_record *record,
+ const char __user *buf)
{
- return psi->write_buf(type, reason, id, part, psinfo->buf, compressed,
- size, psi);
-}
+ int ret = 0;
-static int pstore_write_buf_user_compat(enum pstore_type_id type,
- enum kmsg_dump_reason reason,
- u64 *id, unsigned int part,
- const char __user *buf,
- bool compressed, size_t size,
- struct pstore_info *psi)
-{
- unsigned long flags = 0;
- size_t i, bufsize = size;
- long ret = 0;
+ if (record->buf)
+ return -EINVAL;
- if (unlikely(!access_ok(VERIFY_READ, buf, size)))
- return -EFAULT;
- if (bufsize > psinfo->bufsize)
- bufsize = psinfo->bufsize;
- spin_lock_irqsave(&psinfo->buf_lock, flags);
- for (i = 0; i < size; ) {
- size_t c = min(size - i, bufsize);
+ record->buf = kmalloc(record->size, GFP_KERNEL);
+ if (!record->buf)
+ return -ENOMEM;
- ret = __copy_from_user(psinfo->buf, buf + i, c);
- if (unlikely(ret != 0)) {
- ret = -EFAULT;
- break;
- }
- ret = psi->write_buf(type, reason, id, part, psinfo->buf,
- compressed, c, psi);
- if (unlikely(ret < 0))
- break;
- i += c;
+ if (unlikely(copy_from_user(record->buf, buf, record->size))) {
+ ret = -EFAULT;
+ goto out;
}
- spin_unlock_irqrestore(&psinfo->buf_lock, flags);
- return unlikely(ret < 0) ? ret : size;
+
+ ret = record->psi->write(record);
+
+out:
+ kfree(record->buf);
+ record->buf = NULL;
+
+ return unlikely(ret < 0) ? ret : record->size;
}
/*
@@ -673,19 +669,35 @@
{
struct module *owner = psi->owner;
- if (backend && strcmp(backend, psi->name))
+ if (backend && strcmp(backend, psi->name)) {
+ pr_warn("ignoring unexpected backend '%s'\n", psi->name);
return -EPERM;
+ }
+
+ /* Sanity check flags. */
+ if (!psi->flags) {
+ pr_warn("backend '%s' must support at least one frontend\n",
+ psi->name);
+ return -EINVAL;
+ }
+
+ /* Check for required functions. */
+ if (!psi->read || !psi->write) {
+ pr_warn("backend '%s' must implement read() and write()\n",
+ psi->name);
+ return -EINVAL;
+ }
spin_lock(&pstore_lock);
if (psinfo) {
+ pr_warn("backend '%s' already loaded: ignoring '%s'\n",
+ psinfo->name, psi->name);
spin_unlock(&pstore_lock);
return -EBUSY;
}
- if (!psi->write)
- psi->write = pstore_write_compat;
- if (!psi->write_buf_user)
- psi->write_buf_user = pstore_write_buf_user_compat;
+ if (!psi->write_user)
+ psi->write_user = pstore_write_user_compat;
psinfo = psi;
mutex_init(&psinfo->read_mutex);
spin_unlock(&pstore_lock);
@@ -709,6 +721,7 @@
if (psi->flags & PSTORE_FLAGS_PMSG)
pstore_register_pmsg();
+ /* Start watching for new records, if desired. */
if (pstore_update_ms >= 0) {
pstore_timer.expires = jiffies +
msecs_to_jiffies(pstore_update_ms);
@@ -721,16 +734,21 @@
*/
backend = psi->name;
- module_put(owner);
-
pr_info("Registered %s as persistent store backend\n", psi->name);
+ module_put(owner);
+
return 0;
}
EXPORT_SYMBOL_GPL(pstore_register);
void pstore_unregister(struct pstore_info *psi)
{
+ /* Stop timer and make sure all work has finished. */
+ pstore_update_ms = -1;
+ del_timer_sync(&pstore_timer);
+ flush_work(&pstore_work);
+
if (psi->flags & PSTORE_FLAGS_PMSG)
pstore_unregister_pmsg();
if (psi->flags & PSTORE_FLAGS_FTRACE)
@@ -747,66 +765,99 @@
}
EXPORT_SYMBOL_GPL(pstore_unregister);
+static void decompress_record(struct pstore_record *record)
+{
+ int unzipped_len;
+ char *decompressed;
+
+ /* Only PSTORE_TYPE_DMESG support compression. */
+ if (!record->compressed || record->type != PSTORE_TYPE_DMESG) {
+ pr_warn("ignored compressed record type %d\n", record->type);
+ return;
+ }
+
+ /* No compression method has created the common buffer. */
+ if (!big_oops_buf) {
+ pr_warn("no decompression buffer allocated\n");
+ return;
+ }
+
+ unzipped_len = pstore_decompress(record->buf, big_oops_buf,
+ record->size, big_oops_buf_sz);
+ if (unzipped_len <= 0) {
+ pr_err("decompression failed: %d\n", unzipped_len);
+ return;
+ }
+
+ /* Build new buffer for decompressed contents. */
+ decompressed = kmalloc(unzipped_len + record->ecc_notice_size,
+ GFP_KERNEL);
+ if (!decompressed) {
+ pr_err("decompression ran out of memory\n");
+ return;
+ }
+ memcpy(decompressed, big_oops_buf, unzipped_len);
+
+ /* Append ECC notice to decompressed buffer. */
+ memcpy(decompressed + unzipped_len, record->buf + record->size,
+ record->ecc_notice_size);
+
+ /* Swap out compresed contents with decompressed contents. */
+ kfree(record->buf);
+ record->buf = decompressed;
+ record->size = unzipped_len;
+ record->compressed = false;
+}
+
/*
- * Read all the records from the persistent store. Create
+ * Read all the records from one persistent store backend. Create
* files in our filesystem. Don't warn about -EEXIST errors
* when we are re-scanning the backing store looking to add new
* error records.
*/
-void pstore_get_records(int quiet)
+void pstore_get_backend_records(struct pstore_info *psi,
+ struct dentry *root, int quiet)
{
- struct pstore_info *psi = psinfo;
- char *buf = NULL;
- ssize_t size;
- u64 id;
- int count;
- enum pstore_type_id type;
- struct timespec time;
- int failed = 0, rc;
- bool compressed;
- int unzipped_len = -1;
- ssize_t ecc_notice_size = 0;
+ int failed = 0;
- if (!psi)
+ if (!psi || !root)
return;
mutex_lock(&psi->read_mutex);
if (psi->open && psi->open(psi))
goto out;
- while ((size = psi->read(&id, &type, &count, &time, &buf, &compressed,
- &ecc_notice_size, psi)) > 0) {
- if (compressed && (type == PSTORE_TYPE_DMESG)) {
- if (big_oops_buf)
- unzipped_len = pstore_decompress(buf,
- big_oops_buf, size,
- big_oops_buf_sz);
+ /*
+ * Backend callback read() allocates record.buf. decompress_record()
+ * may reallocate record.buf. On success, pstore_mkfile() will keep
+ * the record.buf, so free it only on failure.
+ */
+ for (;;) {
+ struct pstore_record *record;
+ int rc;
- if (unzipped_len > 0) {
- if (ecc_notice_size)
- memcpy(big_oops_buf + unzipped_len,
- buf + size, ecc_notice_size);
- kfree(buf);
- buf = big_oops_buf;
- size = unzipped_len;
- compressed = false;
- } else {
- pr_err("decompression failed;returned %d\n",
- unzipped_len);
- compressed = true;
- }
+ record = kzalloc(sizeof(*record), GFP_KERNEL);
+ if (!record) {
+ pr_err("out of memory creating record\n");
+ break;
}
- rc = pstore_mkfile(type, psi->name, id, count, buf,
- compressed, size + ecc_notice_size,
- time, psi);
- if (unzipped_len < 0) {
- /* Free buffer other than big oops */
- kfree(buf);
- buf = NULL;
- } else
- unzipped_len = -1;
- if (rc && (rc != -EEXIST || !quiet))
- failed++;
+ record->psi = psi;
+
+ record->size = psi->read(record);
+
+ /* No more records left in backend? */
+ if (record->size <= 0)
+ break;
+
+ decompress_record(record);
+ rc = pstore_mkfile(root, record);
+ if (rc) {
+ /* pstore_mkfile() did not take record, so free it. */
+ kfree(record->buf);
+ kfree(record);
+ if (rc != -EEXIST || !quiet)
+ failed++;
+ }
}
if (psi->close)
psi->close(psi);
@@ -830,7 +881,9 @@
schedule_work(&pstore_work);
}
- mod_timer(&pstore_timer, jiffies + msecs_to_jiffies(pstore_update_ms));
+ if (pstore_update_ms >= 0)
+ mod_timer(&pstore_timer,
+ jiffies + msecs_to_jiffies(pstore_update_ms));
}
module_param(backend, charp, 0444);
diff --git a/fs/pstore/pmsg.c b/fs/pstore/pmsg.c
index 78f6176..209755e 100644
--- a/fs/pstore/pmsg.c
+++ b/fs/pstore/pmsg.c
@@ -15,7 +15,6 @@
#include <linux/device.h>
#include <linux/fs.h>
#include <linux/uaccess.h>
-#include <linux/vmalloc.h>
#include "internal.h"
static DEFINE_MUTEX(pmsg_lock);
@@ -23,19 +22,22 @@
static ssize_t write_pmsg(struct file *file, const char __user *buf,
size_t count, loff_t *ppos)
{
- u64 id;
+ struct pstore_record record = {
+ .type = PSTORE_TYPE_PMSG,
+ .size = count,
+ .psi = psinfo,
+ };
int ret;
if (!count)
return 0;
- /* check outside lock, page in any data. write_buf_user also checks */
+ /* check outside lock, page in any data. write_user also checks */
if (!access_ok(VERIFY_READ, buf, count))
return -EFAULT;
mutex_lock(&pmsg_lock);
- ret = psinfo->write_buf_user(PSTORE_TYPE_PMSG, 0, &id, 0, buf, 0, count,
- psinfo);
+ ret = psinfo->write_user(&record, buf);
mutex_unlock(&pmsg_lock);
return ret ? ret : count;
}
diff --git a/fs/pstore/ram.c b/fs/pstore/ram.c
index 11f918d34..5523df7 100644
--- a/fs/pstore/ram.c
+++ b/fs/pstore/ram.c
@@ -235,35 +235,34 @@
return 0;
}
-static ssize_t ramoops_pstore_read(u64 *id, enum pstore_type_id *type,
- int *count, struct timespec *time,
- char **buf, bool *compressed,
- ssize_t *ecc_notice_size,
- struct pstore_info *psi)
+static ssize_t ramoops_pstore_read(struct pstore_record *record)
{
ssize_t size = 0;
- struct ramoops_context *cxt = psi->data;
+ struct ramoops_context *cxt = record->psi->data;
struct persistent_ram_zone *prz = NULL;
int header_length = 0;
bool free_prz = false;
- /* Ramoops headers provide time stamps for PSTORE_TYPE_DMESG, but
+ /*
+ * Ramoops headers provide time stamps for PSTORE_TYPE_DMESG, but
* PSTORE_TYPE_CONSOLE and PSTORE_TYPE_FTRACE don't currently have
* valid time stamps, so it is initialized to zero.
*/
- time->tv_sec = 0;
- time->tv_nsec = 0;
- *compressed = false;
+ record->time.tv_sec = 0;
+ record->time.tv_nsec = 0;
+ record->compressed = false;
/* Find the next valid persistent_ram_zone for DMESG */
while (cxt->dump_read_cnt < cxt->max_dump_cnt && !prz) {
prz = ramoops_get_next_prz(cxt->dprzs, &cxt->dump_read_cnt,
- cxt->max_dump_cnt, id, type,
+ cxt->max_dump_cnt, &record->id,
+ &record->type,
PSTORE_TYPE_DMESG, 1);
if (!prz_ok(prz))
continue;
header_length = ramoops_read_kmsg_hdr(persistent_ram_old(prz),
- time, compressed);
+ &record->time,
+ &record->compressed);
/* Clear and skip this DMESG record if it has no valid header */
if (!header_length) {
persistent_ram_free_old(prz);
@@ -274,18 +273,20 @@
if (!prz_ok(prz))
prz = ramoops_get_next_prz(&cxt->cprz, &cxt->console_read_cnt,
- 1, id, type, PSTORE_TYPE_CONSOLE, 0);
+ 1, &record->id, &record->type,
+ PSTORE_TYPE_CONSOLE, 0);
if (!prz_ok(prz))
prz = ramoops_get_next_prz(&cxt->mprz, &cxt->pmsg_read_cnt,
- 1, id, type, PSTORE_TYPE_PMSG, 0);
+ 1, &record->id, &record->type,
+ PSTORE_TYPE_PMSG, 0);
/* ftrace is last since it may want to dynamically allocate memory. */
if (!prz_ok(prz)) {
if (!(cxt->flags & RAMOOPS_FLAG_FTRACE_PER_CPU)) {
prz = ramoops_get_next_prz(cxt->fprzs,
- &cxt->ftrace_read_cnt, 1, id, type,
- PSTORE_TYPE_FTRACE, 0);
+ &cxt->ftrace_read_cnt, 1, &record->id,
+ &record->type, PSTORE_TYPE_FTRACE, 0);
} else {
/*
* Build a new dummy record which combines all the
@@ -302,8 +303,10 @@
while (cxt->ftrace_read_cnt < cxt->max_ftrace_cnt) {
prz_next = ramoops_get_next_prz(cxt->fprzs,
&cxt->ftrace_read_cnt,
- cxt->max_ftrace_cnt, id,
- type, PSTORE_TYPE_FTRACE, 0);
+ cxt->max_ftrace_cnt,
+ &record->id,
+ &record->type,
+ PSTORE_TYPE_FTRACE, 0);
if (!prz_ok(prz_next))
continue;
@@ -316,7 +319,7 @@
if (size)
goto out;
}
- *id = 0;
+ record->id = 0;
prz = tmp_prz;
}
}
@@ -329,17 +332,19 @@
size = persistent_ram_old_size(prz) - header_length;
/* ECC correction notice */
- *ecc_notice_size = persistent_ram_ecc_string(prz, NULL, 0);
+ record->ecc_notice_size = persistent_ram_ecc_string(prz, NULL, 0);
- *buf = kmalloc(size + *ecc_notice_size + 1, GFP_KERNEL);
- if (*buf == NULL) {
+ record->buf = kmalloc(size + record->ecc_notice_size + 1, GFP_KERNEL);
+ if (record->buf == NULL) {
size = -ENOMEM;
goto out;
}
- memcpy(*buf, (char *)persistent_ram_old(prz) + header_length, size);
+ memcpy(record->buf, (char *)persistent_ram_old(prz) + header_length,
+ size);
- persistent_ram_ecc_string(prz, *buf + size, *ecc_notice_size + 1);
+ persistent_ram_ecc_string(prz, record->buf + size,
+ record->ecc_notice_size + 1);
out:
if (free_prz) {
@@ -373,23 +378,18 @@
return len;
}
-static int notrace ramoops_pstore_write_buf(enum pstore_type_id type,
- enum kmsg_dump_reason reason,
- u64 *id, unsigned int part,
- const char *buf,
- bool compressed, size_t size,
- struct pstore_info *psi)
+static int notrace ramoops_pstore_write(struct pstore_record *record)
{
- struct ramoops_context *cxt = psi->data;
+ struct ramoops_context *cxt = record->psi->data;
struct persistent_ram_zone *prz;
- size_t hlen;
+ size_t size, hlen;
- if (type == PSTORE_TYPE_CONSOLE) {
+ if (record->type == PSTORE_TYPE_CONSOLE) {
if (!cxt->cprz)
return -ENOMEM;
- persistent_ram_write(cxt->cprz, buf, size);
+ persistent_ram_write(cxt->cprz, record->buf, record->size);
return 0;
- } else if (type == PSTORE_TYPE_FTRACE) {
+ } else if (record->type == PSTORE_TYPE_FTRACE) {
int zonenum;
if (!cxt->fprzs)
@@ -402,33 +402,36 @@
else
zonenum = 0;
- persistent_ram_write(cxt->fprzs[zonenum], buf, size);
+ persistent_ram_write(cxt->fprzs[zonenum], record->buf,
+ record->size);
return 0;
- } else if (type == PSTORE_TYPE_PMSG) {
+ } else if (record->type == PSTORE_TYPE_PMSG) {
pr_warn_ratelimited("PMSG shouldn't call %s\n", __func__);
return -EINVAL;
}
- if (type != PSTORE_TYPE_DMESG)
+ if (record->type != PSTORE_TYPE_DMESG)
return -EINVAL;
- /* Out of the various dmesg dump types, ramoops is currently designed
+ /*
+ * Out of the various dmesg dump types, ramoops is currently designed
* to only store crash logs, rather than storing general kernel logs.
*/
- if (reason != KMSG_DUMP_OOPS &&
- reason != KMSG_DUMP_PANIC)
+ if (record->reason != KMSG_DUMP_OOPS &&
+ record->reason != KMSG_DUMP_PANIC)
return -EINVAL;
/* Skip Oopes when configured to do so. */
- if (reason == KMSG_DUMP_OOPS && !cxt->dump_oops)
+ if (record->reason == KMSG_DUMP_OOPS && !cxt->dump_oops)
return -EINVAL;
- /* Explicitly only take the first part of any new crash.
+ /*
+ * Explicitly only take the first part of any new crash.
* If our buffer is larger than kmsg_bytes, this can never happen,
* and if our buffer is smaller than kmsg_bytes, we don't want the
* report split across multiple records.
*/
- if (part != 1)
+ if (record->part != 1)
return -ENOSPC;
if (!cxt->dprzs)
@@ -436,53 +439,50 @@
prz = cxt->dprzs[cxt->dump_write_cnt];
- hlen = ramoops_write_kmsg_hdr(prz, compressed);
+ /* Build header and append record contents. */
+ hlen = ramoops_write_kmsg_hdr(prz, record->compressed);
+ size = record->size;
if (size + hlen > prz->buffer_size)
size = prz->buffer_size - hlen;
- persistent_ram_write(prz, buf, size);
+ persistent_ram_write(prz, record->buf, size);
cxt->dump_write_cnt = (cxt->dump_write_cnt + 1) % cxt->max_dump_cnt;
return 0;
}
-static int notrace ramoops_pstore_write_buf_user(enum pstore_type_id type,
- enum kmsg_dump_reason reason,
- u64 *id, unsigned int part,
- const char __user *buf,
- bool compressed, size_t size,
- struct pstore_info *psi)
+static int notrace ramoops_pstore_write_user(struct pstore_record *record,
+ const char __user *buf)
{
- if (type == PSTORE_TYPE_PMSG) {
- struct ramoops_context *cxt = psi->data;
+ if (record->type == PSTORE_TYPE_PMSG) {
+ struct ramoops_context *cxt = record->psi->data;
if (!cxt->mprz)
return -ENOMEM;
- return persistent_ram_write_user(cxt->mprz, buf, size);
+ return persistent_ram_write_user(cxt->mprz, buf, record->size);
}
return -EINVAL;
}
-static int ramoops_pstore_erase(enum pstore_type_id type, u64 id, int count,
- struct timespec time, struct pstore_info *psi)
+static int ramoops_pstore_erase(struct pstore_record *record)
{
- struct ramoops_context *cxt = psi->data;
+ struct ramoops_context *cxt = record->psi->data;
struct persistent_ram_zone *prz;
- switch (type) {
+ switch (record->type) {
case PSTORE_TYPE_DMESG:
- if (id >= cxt->max_dump_cnt)
+ if (record->id >= cxt->max_dump_cnt)
return -EINVAL;
- prz = cxt->dprzs[id];
+ prz = cxt->dprzs[record->id];
break;
case PSTORE_TYPE_CONSOLE:
prz = cxt->cprz;
break;
case PSTORE_TYPE_FTRACE:
- if (id >= cxt->max_ftrace_cnt)
+ if (record->id >= cxt->max_ftrace_cnt)
return -EINVAL;
- prz = cxt->fprzs[id];
+ prz = cxt->fprzs[record->id];
break;
case PSTORE_TYPE_PMSG:
prz = cxt->mprz;
@@ -503,8 +503,8 @@
.name = "ramoops",
.open = ramoops_pstore_open,
.read = ramoops_pstore_read,
- .write_buf = ramoops_pstore_write_buf,
- .write_buf_user = ramoops_pstore_write_buf_user,
+ .write = ramoops_pstore_write,
+ .write_user = ramoops_pstore_write_user,
.erase = ramoops_pstore_erase,
},
};
diff --git a/fs/pstore/ram_core.c b/fs/pstore/ram_core.c
index bc927e3..e11672a 100644
--- a/fs/pstore/ram_core.c
+++ b/fs/pstore/ram_core.c
@@ -532,7 +532,7 @@
}
/* Initialize general buffer state. */
- prz->buffer_lock = __RAW_SPIN_LOCK_UNLOCKED(buffer_lock);
+ raw_spin_lock_init(&prz->buffer_lock);
prz->flags = flags;
ret = persistent_ram_buffer_map(start, size, prz, memtype);
diff --git a/fs/read_write.c b/fs/read_write.c
index c4f88af..47c1d44 100644
--- a/fs/read_write.c
+++ b/fs/read_write.c
@@ -841,6 +841,81 @@
return ret;
}
+#ifdef CONFIG_COMPAT
+ssize_t compat_rw_copy_check_uvector(int type,
+ const struct compat_iovec __user *uvector, unsigned long nr_segs,
+ unsigned long fast_segs, struct iovec *fast_pointer,
+ struct iovec **ret_pointer)
+{
+ compat_ssize_t tot_len;
+ struct iovec *iov = *ret_pointer = fast_pointer;
+ ssize_t ret = 0;
+ int seg;
+
+ /*
+ * SuS says "The readv() function *may* fail if the iovcnt argument
+ * was less than or equal to 0, or greater than {IOV_MAX}. Linux has
+ * traditionally returned zero for zero segments, so...
+ */
+ if (nr_segs == 0)
+ goto out;
+
+ ret = -EINVAL;
+ if (nr_segs > UIO_MAXIOV)
+ goto out;
+ if (nr_segs > fast_segs) {
+ ret = -ENOMEM;
+ iov = kmalloc(nr_segs*sizeof(struct iovec), GFP_KERNEL);
+ if (iov == NULL)
+ goto out;
+ }
+ *ret_pointer = iov;
+
+ ret = -EFAULT;
+ if (!access_ok(VERIFY_READ, uvector, nr_segs*sizeof(*uvector)))
+ goto out;
+
+ /*
+ * Single unix specification:
+ * We should -EINVAL if an element length is not >= 0 and fitting an
+ * ssize_t.
+ *
+ * In Linux, the total length is limited to MAX_RW_COUNT, there is
+ * no overflow possibility.
+ */
+ tot_len = 0;
+ ret = -EINVAL;
+ for (seg = 0; seg < nr_segs; seg++) {
+ compat_uptr_t buf;
+ compat_ssize_t len;
+
+ if (__get_user(len, &uvector->iov_len) ||
+ __get_user(buf, &uvector->iov_base)) {
+ ret = -EFAULT;
+ goto out;
+ }
+ if (len < 0) /* size_t not fitting in compat_ssize_t .. */
+ goto out;
+ if (type >= 0 &&
+ !access_ok(vrfy_dir(type), compat_ptr(buf), len)) {
+ ret = -EFAULT;
+ goto out;
+ }
+ if (len > MAX_RW_COUNT - tot_len)
+ len = MAX_RW_COUNT - tot_len;
+ tot_len += len;
+ iov->iov_base = compat_ptr(buf);
+ iov->iov_len = (compat_size_t) len;
+ uvector++;
+ iov++;
+ }
+ ret = tot_len;
+
+out:
+ return ret;
+}
+#endif
+
static ssize_t __do_readv_writev(int type, struct file *file,
struct iov_iter *iter, loff_t *pos, int flags)
{
diff --git a/fs/readdir.c b/fs/readdir.c
index 0e8a7f3..8965954 100644
--- a/fs/readdir.c
+++ b/fs/readdir.c
@@ -18,6 +18,7 @@
#include <linux/security.h>
#include <linux/syscalls.h>
#include <linux/unistd.h>
+#include <linux/compat.h>
#include <linux/uaccess.h>
@@ -324,3 +325,167 @@
fdput_pos(f);
return error;
}
+
+#ifdef CONFIG_COMPAT
+struct compat_old_linux_dirent {
+ compat_ulong_t d_ino;
+ compat_ulong_t d_offset;
+ unsigned short d_namlen;
+ char d_name[1];
+};
+
+struct compat_readdir_callback {
+ struct dir_context ctx;
+ struct compat_old_linux_dirent __user *dirent;
+ int result;
+};
+
+static int compat_fillonedir(struct dir_context *ctx, const char *name,
+ int namlen, loff_t offset, u64 ino,
+ unsigned int d_type)
+{
+ struct compat_readdir_callback *buf =
+ container_of(ctx, struct compat_readdir_callback, ctx);
+ struct compat_old_linux_dirent __user *dirent;
+ compat_ulong_t d_ino;
+
+ if (buf->result)
+ return -EINVAL;
+ d_ino = ino;
+ if (sizeof(d_ino) < sizeof(ino) && d_ino != ino) {
+ buf->result = -EOVERFLOW;
+ return -EOVERFLOW;
+ }
+ buf->result++;
+ dirent = buf->dirent;
+ if (!access_ok(VERIFY_WRITE, dirent,
+ (unsigned long)(dirent->d_name + namlen + 1) -
+ (unsigned long)dirent))
+ goto efault;
+ if ( __put_user(d_ino, &dirent->d_ino) ||
+ __put_user(offset, &dirent->d_offset) ||
+ __put_user(namlen, &dirent->d_namlen) ||
+ __copy_to_user(dirent->d_name, name, namlen) ||
+ __put_user(0, dirent->d_name + namlen))
+ goto efault;
+ return 0;
+efault:
+ buf->result = -EFAULT;
+ return -EFAULT;
+}
+
+COMPAT_SYSCALL_DEFINE3(old_readdir, unsigned int, fd,
+ struct compat_old_linux_dirent __user *, dirent, unsigned int, count)
+{
+ int error;
+ struct fd f = fdget_pos(fd);
+ struct compat_readdir_callback buf = {
+ .ctx.actor = compat_fillonedir,
+ .dirent = dirent
+ };
+
+ if (!f.file)
+ return -EBADF;
+
+ error = iterate_dir(f.file, &buf.ctx);
+ if (buf.result)
+ error = buf.result;
+
+ fdput_pos(f);
+ return error;
+}
+
+struct compat_linux_dirent {
+ compat_ulong_t d_ino;
+ compat_ulong_t d_off;
+ unsigned short d_reclen;
+ char d_name[1];
+};
+
+struct compat_getdents_callback {
+ struct dir_context ctx;
+ struct compat_linux_dirent __user *current_dir;
+ struct compat_linux_dirent __user *previous;
+ int count;
+ int error;
+};
+
+static int compat_filldir(struct dir_context *ctx, const char *name, int namlen,
+ loff_t offset, u64 ino, unsigned int d_type)
+{
+ struct compat_linux_dirent __user * dirent;
+ struct compat_getdents_callback *buf =
+ container_of(ctx, struct compat_getdents_callback, ctx);
+ compat_ulong_t d_ino;
+ int reclen = ALIGN(offsetof(struct compat_linux_dirent, d_name) +
+ namlen + 2, sizeof(compat_long_t));
+
+ buf->error = -EINVAL; /* only used if we fail.. */
+ if (reclen > buf->count)
+ return -EINVAL;
+ d_ino = ino;
+ if (sizeof(d_ino) < sizeof(ino) && d_ino != ino) {
+ buf->error = -EOVERFLOW;
+ return -EOVERFLOW;
+ }
+ dirent = buf->previous;
+ if (dirent) {
+ if (signal_pending(current))
+ return -EINTR;
+ if (__put_user(offset, &dirent->d_off))
+ goto efault;
+ }
+ dirent = buf->current_dir;
+ if (__put_user(d_ino, &dirent->d_ino))
+ goto efault;
+ if (__put_user(reclen, &dirent->d_reclen))
+ goto efault;
+ if (copy_to_user(dirent->d_name, name, namlen))
+ goto efault;
+ if (__put_user(0, dirent->d_name + namlen))
+ goto efault;
+ if (__put_user(d_type, (char __user *) dirent + reclen - 1))
+ goto efault;
+ buf->previous = dirent;
+ dirent = (void __user *)dirent + reclen;
+ buf->current_dir = dirent;
+ buf->count -= reclen;
+ return 0;
+efault:
+ buf->error = -EFAULT;
+ return -EFAULT;
+}
+
+COMPAT_SYSCALL_DEFINE3(getdents, unsigned int, fd,
+ struct compat_linux_dirent __user *, dirent, unsigned int, count)
+{
+ struct fd f;
+ struct compat_linux_dirent __user * lastdirent;
+ struct compat_getdents_callback buf = {
+ .ctx.actor = compat_filldir,
+ .current_dir = dirent,
+ .count = count
+ };
+ int error;
+
+ if (!access_ok(VERIFY_WRITE, dirent, count))
+ return -EFAULT;
+
+ f = fdget_pos(fd);
+ if (!f.file)
+ return -EBADF;
+
+ error = iterate_dir(f.file, &buf.ctx);
+ if (error >= 0)
+ error = buf.error;
+ lastdirent = buf.previous;
+ if (lastdirent) {
+ if (put_user(buf.ctx.pos, &lastdirent->d_off))
+ error = -EFAULT;
+ else
+ error = count - buf.count;
+ }
+ fdput_pos(f);
+ return error;
+}
+#endif
diff --git a/fs/select.c b/fs/select.c
index e211227..bd4b2cc 100644
--- a/fs/select.c
+++ b/fs/select.c
@@ -338,6 +338,53 @@
return ret;
}
+/*
+ * Scalable version of the fd_set.
+ */
+
+typedef struct {
+ unsigned long *in, *out, *ex;
+ unsigned long *res_in, *res_out, *res_ex;
+} fd_set_bits;
+
+/*
+ * How many longwords for "nr" bits?
+ */
+#define FDS_BITPERLONG (8*sizeof(long))
+#define FDS_LONGS(nr) (((nr)+FDS_BITPERLONG-1)/FDS_BITPERLONG)
+#define FDS_BYTES(nr) (FDS_LONGS(nr)*sizeof(long))
+
+/*
+ * We do a VERIFY_WRITE here even though we are only reading this time:
+ * we'll write to it eventually..
+ *
+ * Use "unsigned long" accesses to let user-mode fd_set's be long-aligned.
+ */
+static inline
+int get_fd_set(unsigned long nr, void __user *ufdset, unsigned long *fdset)
+{
+ nr = FDS_BYTES(nr);
+ if (ufdset)
+ return copy_from_user(fdset, ufdset, nr) ? -EFAULT : 0;
+
+ memset(fdset, 0, nr);
+ return 0;
+}
+
+static inline unsigned long __must_check
+set_fd_set(unsigned long nr, void __user *ufdset, unsigned long *fdset)
+{
+ if (ufdset)
+ return __copy_to_user(ufdset, fdset, FDS_BYTES(nr));
+ return 0;
+}
+
+static inline
+void zero_fd_set(unsigned long nr, unsigned long *fdset)
+{
+ memset(fdset, 0, FDS_BYTES(nr));
+}
+
#define FDS_IN(fds, n) (fds->in + n)
#define FDS_OUT(fds, n) (fds->out + n)
#define FDS_EX(fds, n) (fds->ex + n)
@@ -401,7 +448,7 @@
wait->_key |= POLLOUT_SET;
}
-int do_select(int n, fd_set_bits *fds, struct timespec64 *end_time)
+static int do_select(int n, fd_set_bits *fds, struct timespec64 *end_time)
{
ktime_t expire, *to = NULL;
struct poll_wqueues table;
@@ -409,7 +456,7 @@
int retval, i, timed_out = 0;
u64 slack = 0;
unsigned int busy_flag = net_busy_loop_on() ? POLL_BUSY_LOOP : 0;
- unsigned long busy_end = 0;
+ unsigned long busy_start = 0;
rcu_read_lock();
retval = max_select_fd(n, fds);
@@ -512,11 +559,11 @@
/* only if found POLL_BUSY_LOOP sockets && not out of time */
if (can_busy_loop && !need_resched()) {
- if (!busy_end) {
- busy_end = busy_loop_end_time();
+ if (!busy_start) {
+ busy_start = busy_loop_current_time();
continue;
}
- if (!busy_loop_timeout(busy_end))
+ if (!busy_loop_timeout(busy_start))
continue;
}
busy_flag = 0;
@@ -800,7 +847,7 @@
int timed_out = 0, count = 0;
u64 slack = 0;
unsigned int busy_flag = net_busy_loop_on() ? POLL_BUSY_LOOP : 0;
- unsigned long busy_end = 0;
+ unsigned long busy_start = 0;
/* Optimise the no-wait case */
if (end_time && !end_time->tv_sec && !end_time->tv_nsec) {
@@ -853,11 +900,11 @@
/* only if found POLL_BUSY_LOOP sockets && not out of time */
if (can_busy_loop && !need_resched()) {
- if (!busy_end) {
- busy_end = busy_loop_end_time();
+ if (!busy_start) {
+ busy_start = busy_loop_current_time();
continue;
}
- if (!busy_loop_timeout(busy_end))
+ if (!busy_loop_timeout(busy_start))
continue;
}
busy_flag = 0;
@@ -881,7 +928,7 @@
#define N_STACK_PPS ((sizeof(stack_pps) - sizeof(struct poll_list)) / \
sizeof(struct pollfd))
-int do_sys_poll(struct pollfd __user *ufds, unsigned int nfds,
+static int do_sys_poll(struct pollfd __user *ufds, unsigned int nfds,
struct timespec64 *end_time)
{
struct poll_wqueues table;
@@ -1053,3 +1100,373 @@
return ret;
}
+
+#ifdef CONFIG_COMPAT
+#define __COMPAT_NFDBITS (8 * sizeof(compat_ulong_t))
+
+static
+int compat_poll_select_copy_remaining(struct timespec *end_time, void __user *p,
+ int timeval, int ret)
+{
+ struct timespec ts;
+
+ if (!p)
+ return ret;
+
+ if (current->personality & STICKY_TIMEOUTS)
+ goto sticky;
+
+ /* No update for zero timeout */
+ if (!end_time->tv_sec && !end_time->tv_nsec)
+ return ret;
+
+ ktime_get_ts(&ts);
+ ts = timespec_sub(*end_time, ts);
+ if (ts.tv_sec < 0)
+ ts.tv_sec = ts.tv_nsec = 0;
+
+ if (timeval) {
+ struct compat_timeval rtv;
+
+ rtv.tv_sec = ts.tv_sec;
+ rtv.tv_usec = ts.tv_nsec / NSEC_PER_USEC;
+
+ if (!copy_to_user(p, &rtv, sizeof(rtv)))
+ return ret;
+ } else {
+ struct compat_timespec rts;
+
+ rts.tv_sec = ts.tv_sec;
+ rts.tv_nsec = ts.tv_nsec;
+
+ if (!copy_to_user(p, &rts, sizeof(rts)))
+ return ret;
+ }
+ /*
+ * If an application puts its timeval in read-only memory, we
+ * don't want the Linux-specific update to the timeval to
+ * cause a fault after the select has completed
+ * successfully. However, because we're not updating the
+ * timeval, we can't restart the system call.
+ */
+
+sticky:
+ if (ret == -ERESTARTNOHAND)
+ ret = -EINTR;
+ return ret;
+}
+
+/*
+ * Ooo, nasty. We need here to frob 32-bit unsigned longs to
+ * 64-bit unsigned longs.
+ */
+static
+int compat_get_fd_set(unsigned long nr, compat_ulong_t __user *ufdset,
+ unsigned long *fdset)
+{
+ nr = DIV_ROUND_UP(nr, __COMPAT_NFDBITS);
+ if (ufdset) {
+ unsigned long odd;
+
+ if (!access_ok(VERIFY_WRITE, ufdset, nr*sizeof(compat_ulong_t)))
+ return -EFAULT;
+
+ odd = nr & 1UL;
+ nr &= ~1UL;
+ while (nr) {
+ unsigned long h, l;
+ if (__get_user(l, ufdset) || __get_user(h, ufdset+1))
+ return -EFAULT;
+ ufdset += 2;
+ *fdset++ = h << 32 | l;
+ nr -= 2;
+ }
+ if (odd && __get_user(*fdset, ufdset))
+ return -EFAULT;
+ } else {
+ /* Tricky, must clear full unsigned long in the
+ * kernel fdset at the end, this makes sure that
+ * actually happens.
+ */
+ memset(fdset, 0, ((nr + 1) & ~1)*sizeof(compat_ulong_t));
+ }
+ return 0;
+}
+
+static
+int compat_set_fd_set(unsigned long nr, compat_ulong_t __user *ufdset,
+ unsigned long *fdset)
+{
+ unsigned long odd;
+ nr = DIV_ROUND_UP(nr, __COMPAT_NFDBITS);
+
+ if (!ufdset)
+ return 0;
+
+ odd = nr & 1UL;
+ nr &= ~1UL;
+ while (nr) {
+ unsigned long h, l;
+ l = *fdset++;
+ h = l >> 32;
+ if (__put_user(l, ufdset) || __put_user(h, ufdset+1))
+ return -EFAULT;
+ ufdset += 2;
+ nr -= 2;
+ }
+ if (odd && __put_user(*fdset, ufdset))
+ return -EFAULT;
+ return 0;
+}
+
+
+/*
+ * This is a virtual copy of sys_select from fs/select.c and probably
+ * should be compared to it from time to time
+ */
+
+/*
+ * We can actually return ERESTARTSYS instead of EINTR, but I'd
+ * like to be certain this leads to no problems. So I return
+ * EINTR just for safety.
+ *
+ * Update: ERESTARTSYS breaks at least the xview clock binary, so
+ * I'm trying ERESTARTNOHAND which restart only when you want to.
+ */
+static int compat_core_sys_select(int n, compat_ulong_t __user *inp,
+ compat_ulong_t __user *outp, compat_ulong_t __user *exp,
+ struct timespec *end_time)
+{
+ fd_set_bits fds;
+ void *bits;
+ int size, max_fds, ret = -EINVAL;
+ struct fdtable *fdt;
+ long stack_fds[SELECT_STACK_ALLOC/sizeof(long)];
+
+ if (n < 0)
+ goto out_nofds;
+
+ /* max_fds can increase, so grab it once to avoid race */
+ rcu_read_lock();
+ fdt = files_fdtable(current->files);
+ max_fds = fdt->max_fds;
+ rcu_read_unlock();
+ if (n > max_fds)
+ n = max_fds;
+
+ /*
+ * We need 6 bitmaps (in/out/ex for both incoming and outgoing),
+ * since we used fdset we need to allocate memory in units of
+ * long-words.
+ */
+ size = FDS_BYTES(n);
+ bits = stack_fds;
+ if (size > sizeof(stack_fds) / 6) {
+ bits = kmalloc(6 * size, GFP_KERNEL);
+ ret = -ENOMEM;
+ if (!bits)
+ goto out_nofds;
+ }
+ fds.in = (unsigned long *) bits;
+ fds.out = (unsigned long *) (bits + size);
+ fds.ex = (unsigned long *) (bits + 2*size);
+ fds.res_in = (unsigned long *) (bits + 3*size);
+ fds.res_out = (unsigned long *) (bits + 4*size);
+ fds.res_ex = (unsigned long *) (bits + 5*size);
+
+ if ((ret = compat_get_fd_set(n, inp, fds.in)) ||
+ (ret = compat_get_fd_set(n, outp, fds.out)) ||
+ (ret = compat_get_fd_set(n, exp, fds.ex)))
+ goto out;
+ zero_fd_set(n, fds.res_in);
+ zero_fd_set(n, fds.res_out);
+ zero_fd_set(n, fds.res_ex);
+
+ ret = do_select(n, &fds, end_time);
+
+ if (ret < 0)
+ goto out;
+ if (!ret) {
+ ret = -ERESTARTNOHAND;
+ if (signal_pending(current))
+ goto out;
+ ret = 0;
+ }
+
+ if (compat_set_fd_set(n, inp, fds.res_in) ||
+ compat_set_fd_set(n, outp, fds.res_out) ||
+ compat_set_fd_set(n, exp, fds.res_ex))
+ ret = -EFAULT;
+out:
+ if (bits != stack_fds)
+ kfree(bits);
+out_nofds:
+ return ret;
+}
+
+COMPAT_SYSCALL_DEFINE5(select, int, n, compat_ulong_t __user *, inp,
+ compat_ulong_t __user *, outp, compat_ulong_t __user *, exp,
+ struct compat_timeval __user *, tvp)
+{
+ struct timespec end_time, *to = NULL;
+ struct compat_timeval tv;
+ int ret;
+
+ if (tvp) {
+ if (copy_from_user(&tv, tvp, sizeof(tv)))
+ return -EFAULT;
+
+ to = &end_time;
+ if (poll_select_set_timeout(to,
+ tv.tv_sec + (tv.tv_usec / USEC_PER_SEC),
+ (tv.tv_usec % USEC_PER_SEC) * NSEC_PER_USEC))
+ return -EINVAL;
+ }
+
+ ret = compat_core_sys_select(n, inp, outp, exp, to);
+ ret = compat_poll_select_copy_remaining(&end_time, tvp, 1, ret);
+
+ return ret;
+}
+
+struct compat_sel_arg_struct {
+ compat_ulong_t n;
+ compat_uptr_t inp;
+ compat_uptr_t outp;
+ compat_uptr_t exp;
+ compat_uptr_t tvp;
+};
+
+COMPAT_SYSCALL_DEFINE1(old_select, struct compat_sel_arg_struct __user *, arg)
+{
+ struct compat_sel_arg_struct a;
+
+ if (copy_from_user(&a, arg, sizeof(a)))
+ return -EFAULT;
+ return compat_sys_select(a.n, compat_ptr(a.inp), compat_ptr(a.outp),
+ compat_ptr(a.exp), compat_ptr(a.tvp));
+}
+
+static long do_compat_pselect(int n, compat_ulong_t __user *inp,
+ compat_ulong_t __user *outp, compat_ulong_t __user *exp,
+ struct compat_timespec __user *tsp, compat_sigset_t __user *sigmask,
+ compat_size_t sigsetsize)
+{
+ compat_sigset_t ss32;
+ sigset_t ksigmask, sigsaved;
+ struct compat_timespec ts;
+ struct timespec end_time, *to = NULL;
+ int ret;
+
+ if (tsp) {
+ if (copy_from_user(&ts, tsp, sizeof(ts)))
+ return -EFAULT;
+
+ to = &end_time;
+ if (poll_select_set_timeout(to, ts.tv_sec, ts.tv_nsec))
+ return -EINVAL;
+ }
+
+ if (sigmask) {
+ if (sigsetsize != sizeof(compat_sigset_t))
+ return -EINVAL;
+ if (copy_from_user(&ss32, sigmask, sizeof(ss32)))
+ return -EFAULT;
+ sigset_from_compat(&ksigmask, &ss32);
+
+ sigdelsetmask(&ksigmask, sigmask(SIGKILL)|sigmask(SIGSTOP));
+ sigprocmask(SIG_SETMASK, &ksigmask, &sigsaved);
+ }
+
+ ret = compat_core_sys_select(n, inp, outp, exp, to);
+ ret = compat_poll_select_copy_remaining(&end_time, tsp, 0, ret);
+
+ if (ret == -ERESTARTNOHAND) {
+ /*
+ * Don't restore the signal mask yet. Let do_signal() deliver
+ * the signal on the way back to userspace, before the signal
+ * mask is restored.
+ */
+ if (sigmask) {
+ memcpy(¤t->saved_sigmask, &sigsaved,
+ sizeof(sigsaved));
+ set_restore_sigmask();
+ }
+ } else if (sigmask)
+ sigprocmask(SIG_SETMASK, &sigsaved, NULL);
+
+ return ret;
+}
+
+COMPAT_SYSCALL_DEFINE6(pselect6, int, n, compat_ulong_t __user *, inp,
+ compat_ulong_t __user *, outp, compat_ulong_t __user *, exp,
+ struct compat_timespec __user *, tsp, void __user *, sig)
+{
+ compat_size_t sigsetsize = 0;
+ compat_uptr_t up = 0;
+
+ if (sig) {
+ if (!access_ok(VERIFY_READ, sig,
+ sizeof(compat_uptr_t)+sizeof(compat_size_t)) ||
+ __get_user(up, (compat_uptr_t __user *)sig) ||
+ __get_user(sigsetsize,
+ (compat_size_t __user *)(sig+sizeof(up))))
+ return -EFAULT;
+ }
+ return do_compat_pselect(n, inp, outp, exp, tsp, compat_ptr(up),
+ sigsetsize);
+}
+
+COMPAT_SYSCALL_DEFINE5(ppoll, struct pollfd __user *, ufds,
+ unsigned int, nfds, struct compat_timespec __user *, tsp,
+ const compat_sigset_t __user *, sigmask, compat_size_t, sigsetsize)
+{
+ compat_sigset_t ss32;
+ sigset_t ksigmask, sigsaved;
+ struct compat_timespec ts;
+ struct timespec end_time, *to = NULL;
+ int ret;
+
+ if (tsp) {
+ if (copy_from_user(&ts, tsp, sizeof(ts)))
+ return -EFAULT;
+
+ to = &end_time;
+ if (poll_select_set_timeout(to, ts.tv_sec, ts.tv_nsec))
+ return -EINVAL;
+ }
+
+ if (sigmask) {
+ if (sigsetsize != sizeof(compat_sigset_t))
+ return -EINVAL;
+ if (copy_from_user(&ss32, sigmask, sizeof(ss32)))
+ return -EFAULT;
+ sigset_from_compat(&ksigmask, &ss32);
+
+ sigdelsetmask(&ksigmask, sigmask(SIGKILL)|sigmask(SIGSTOP));
+ sigprocmask(SIG_SETMASK, &ksigmask, &sigsaved);
+ }
+
+ ret = do_sys_poll(ufds, nfds, to);
+
+ /* We can restart this syscall, usually */
+ if (ret == -EINTR) {
+ /*
+ * Don't restore the signal mask yet. Let do_signal() deliver
+ * the signal on the way back to userspace, before the signal
+ * mask is restored.
+ */
+ if (sigmask) {
+ memcpy(¤t->saved_sigmask, &sigsaved,
+ sizeof(sigsaved));
+ set_restore_sigmask();
+ }
+ ret = -ERESTARTNOHAND;
+ } else if (sigmask)
+ sigprocmask(SIG_SETMASK, &sigsaved, NULL);
+
+ ret = compat_poll_select_copy_remaining(&end_time, tsp, 0, ret);
+
+ return ret;
+}
+#endif
diff --git a/fs/splice.c b/fs/splice.c
index 006ba50..540c4a4 100644
--- a/fs/splice.c
+++ b/fs/splice.c
@@ -247,11 +247,6 @@
}
EXPORT_SYMBOL(add_to_pipe);
-void spd_release_page(struct splice_pipe_desc *spd, unsigned int i)
-{
- put_page(spd->pages[i]);
-}
-
/*
* Check if we need to grow the arrays holding pages and partial page
* descriptions.
@@ -393,7 +388,7 @@
struct iov_iter to;
struct page **pages;
unsigned int nr_pages;
- size_t offset, dummy, copied = 0;
+ size_t offset, base, copied = 0;
ssize_t res;
int i;
@@ -408,12 +403,11 @@
iov_iter_pipe(&to, ITER_PIPE | READ, pipe, len + offset);
- res = iov_iter_get_pages_alloc(&to, &pages, len + offset, &dummy);
+ res = iov_iter_get_pages_alloc(&to, &pages, len + offset, &base);
if (res <= 0)
return -ENOMEM;
- BUG_ON(dummy);
- nr_pages = DIV_ROUND_UP(res, PAGE_SIZE);
+ nr_pages = DIV_ROUND_UP(res + base, PAGE_SIZE);
vec = __vec;
if (nr_pages > PIPE_DEF_BUFFERS) {
@@ -1359,6 +1353,8 @@
struct fd f;
long error;
+ if (unlikely(flags & ~SPLICE_F_ALL))
+ return -EINVAL;
if (unlikely(nr_segs > UIO_MAXIOV))
return -EINVAL;
else if (unlikely(!nr_segs))
@@ -1409,6 +1405,9 @@
if (unlikely(!len))
return 0;
+ if (unlikely(flags & ~SPLICE_F_ALL))
+ return -EINVAL;
+
error = -EBADF;
in = fdget(fd_in);
if (in.file) {
@@ -1737,6 +1736,9 @@
struct fd in;
int error;
+ if (unlikely(flags & ~SPLICE_F_ALL))
+ return -EINVAL;
+
if (unlikely(!len))
return 0;
diff --git a/fs/stat.c b/fs/stat.c
index c6c963b..f494b18 100644
--- a/fs/stat.c
+++ b/fs/stat.c
@@ -15,6 +15,7 @@
#include <linux/cred.h>
#include <linux/syscalls.h>
#include <linux/pagemap.h>
+#include <linux/compat.h>
#include <linux/uaccess.h>
#include <asm/unistd.h>
@@ -547,13 +548,13 @@
/**
* sys_statx - System call to get enhanced stats
* @dfd: Base directory to pathwalk from *or* fd to stat.
- * @filename: File to stat *or* NULL.
+ * @filename: File to stat or "" with AT_EMPTY_PATH
* @flags: AT_* flags to control pathwalk.
* @mask: Parts of statx struct actually required.
* @buffer: Result buffer.
*
- * Note that if filename is NULL, then it does the equivalent of fstat() using
- * dfd to indicate the file of interest.
+ * Note that fstat() can be emulated by setting dfd to the fd of interest,
+ * supplying "" as the filename and setting AT_EMPTY_PATH in the flags.
*/
SYSCALL_DEFINE5(statx,
int, dfd, const char __user *, filename, unsigned, flags,
@@ -568,16 +569,98 @@
if ((flags & AT_STATX_SYNC_TYPE) == AT_STATX_SYNC_TYPE)
return -EINVAL;
- if (filename)
- error = vfs_statx(dfd, filename, flags, &stat, mask);
- else
- error = vfs_statx_fd(dfd, &stat, mask, flags);
+ error = vfs_statx(dfd, filename, flags, &stat, mask);
if (error)
return error;
return cp_statx(&stat, buffer);
}
+#ifdef CONFIG_COMPAT
+static int cp_compat_stat(struct kstat *stat, struct compat_stat __user *ubuf)
+{
+ struct compat_stat tmp;
+
+ if (!old_valid_dev(stat->dev) || !old_valid_dev(stat->rdev))
+ return -EOVERFLOW;
+
+ memset(&tmp, 0, sizeof(tmp));
+ tmp.st_dev = old_encode_dev(stat->dev);
+ tmp.st_ino = stat->ino;
+ if (sizeof(tmp.st_ino) < sizeof(stat->ino) && tmp.st_ino != stat->ino)
+ return -EOVERFLOW;
+ tmp.st_mode = stat->mode;
+ tmp.st_nlink = stat->nlink;
+ if (tmp.st_nlink != stat->nlink)
+ return -EOVERFLOW;
+ SET_UID(tmp.st_uid, from_kuid_munged(current_user_ns(), stat->uid));
+ SET_GID(tmp.st_gid, from_kgid_munged(current_user_ns(), stat->gid));
+ tmp.st_rdev = old_encode_dev(stat->rdev);
+ if ((u64) stat->size > MAX_NON_LFS)
+ return -EOVERFLOW;
+ tmp.st_size = stat->size;
+ tmp.st_atime = stat->atime.tv_sec;
+ tmp.st_atime_nsec = stat->atime.tv_nsec;
+ tmp.st_mtime = stat->mtime.tv_sec;
+ tmp.st_mtime_nsec = stat->mtime.tv_nsec;
+ tmp.st_ctime = stat->ctime.tv_sec;
+ tmp.st_ctime_nsec = stat->ctime.tv_nsec;
+ tmp.st_blocks = stat->blocks;
+ tmp.st_blksize = stat->blksize;
+ return copy_to_user(ubuf, &tmp, sizeof(tmp)) ? -EFAULT : 0;
+}
+
+COMPAT_SYSCALL_DEFINE2(newstat, const char __user *, filename,
+ struct compat_stat __user *, statbuf)
+{
+ struct kstat stat;
+ int error;
+
+ error = vfs_stat(filename, &stat);
+ if (error)
+ return error;
+ return cp_compat_stat(&stat, statbuf);
+}
+
+COMPAT_SYSCALL_DEFINE2(newlstat, const char __user *, filename,
+ struct compat_stat __user *, statbuf)
+{
+ struct kstat stat;
+ int error;
+
+ error = vfs_lstat(filename, &stat);
+ if (error)
+ return error;
+ return cp_compat_stat(&stat, statbuf);
+}
+
+#ifndef __ARCH_WANT_STAT64
+COMPAT_SYSCALL_DEFINE4(newfstatat, unsigned int, dfd,
+ const char __user *, filename,
+ struct compat_stat __user *, statbuf, int, flag)
+{
+ struct kstat stat;
+ int error;
+
+ error = vfs_fstatat(dfd, filename, &stat, flag);
+ if (error)
+ return error;
+ return cp_compat_stat(&stat, statbuf);
+}
+#endif
+
+COMPAT_SYSCALL_DEFINE2(newfstat, unsigned int, fd,
+ struct compat_stat __user *, statbuf)
+{
+ struct kstat stat;
+ int error = vfs_fstat(fd, &stat);
+
+ if (!error)
+ error = cp_compat_stat(&stat, statbuf);
+ return error;
+}
+#endif
+
/* Caller is here responsible for sufficient locking (ie. inode->i_lock) */
void __inode_add_bytes(struct inode *inode, loff_t bytes)
{
diff --git a/fs/statfs.c b/fs/statfs.c
index 13ae259..4e4623c 100644
--- a/fs/statfs.c
+++ b/fs/statfs.c
@@ -7,6 +7,7 @@
#include <linux/statfs.h>
#include <linux/security.h>
#include <linux/uaccess.h>
+#include <linux/compat.h>
#include "internal.h"
static int flags_by_mnt(int mnt_flags)
@@ -239,3 +240,142 @@
return copy_to_user(ubuf, &tmp, sizeof(struct ustat)) ? -EFAULT : 0;
}
+
+#ifdef CONFIG_COMPAT
+static int put_compat_statfs(struct compat_statfs __user *ubuf, struct kstatfs *kbuf)
+{
+ if (sizeof ubuf->f_blocks == 4) {
+ if ((kbuf->f_blocks | kbuf->f_bfree | kbuf->f_bavail |
+ kbuf->f_bsize | kbuf->f_frsize) & 0xffffffff00000000ULL)
+ return -EOVERFLOW;
+ /* f_files and f_ffree may be -1; it's okay
+ * to stuff that into 32 bits */
+ if (kbuf->f_files != 0xffffffffffffffffULL
+ && (kbuf->f_files & 0xffffffff00000000ULL))
+ return -EOVERFLOW;
+ if (kbuf->f_ffree != 0xffffffffffffffffULL
+ && (kbuf->f_ffree & 0xffffffff00000000ULL))
+ return -EOVERFLOW;
+ }
+ if (!access_ok(VERIFY_WRITE, ubuf, sizeof(*ubuf)) ||
+ __put_user(kbuf->f_type, &ubuf->f_type) ||
+ __put_user(kbuf->f_bsize, &ubuf->f_bsize) ||
+ __put_user(kbuf->f_blocks, &ubuf->f_blocks) ||
+ __put_user(kbuf->f_bfree, &ubuf->f_bfree) ||
+ __put_user(kbuf->f_bavail, &ubuf->f_bavail) ||
+ __put_user(kbuf->f_files, &ubuf->f_files) ||
+ __put_user(kbuf->f_ffree, &ubuf->f_ffree) ||
+ __put_user(kbuf->f_namelen, &ubuf->f_namelen) ||
+ __put_user(kbuf->f_fsid.val[0], &ubuf->f_fsid.val[0]) ||
+ __put_user(kbuf->f_fsid.val[1], &ubuf->f_fsid.val[1]) ||
+ __put_user(kbuf->f_frsize, &ubuf->f_frsize) ||
+ __put_user(kbuf->f_flags, &ubuf->f_flags) ||
+ __clear_user(ubuf->f_spare, sizeof(ubuf->f_spare)))
+ return -EFAULT;
+ return 0;
+}
+
+/*
+ * The following statfs calls are copies of code from fs/statfs.c and
+ * should be checked against those from time to time
+ */
+COMPAT_SYSCALL_DEFINE2(statfs, const char __user *, pathname, struct compat_statfs __user *, buf)
+{
+ struct kstatfs tmp;
+ int error = user_statfs(pathname, &tmp);
+ if (!error)
+ error = put_compat_statfs(buf, &tmp);
+ return error;
+}
+
+COMPAT_SYSCALL_DEFINE2(fstatfs, unsigned int, fd, struct compat_statfs __user *, buf)
+{
+ struct kstatfs tmp;
+ int error = fd_statfs(fd, &tmp);
+ if (!error)
+ error = put_compat_statfs(buf, &tmp);
+ return error;
+}
+
+static int put_compat_statfs64(struct compat_statfs64 __user *ubuf, struct kstatfs *kbuf)
+{
+ if (sizeof(ubuf->f_bsize) == 4) {
+ if ((kbuf->f_type | kbuf->f_bsize | kbuf->f_namelen |
+ kbuf->f_frsize | kbuf->f_flags) & 0xffffffff00000000ULL)
+ return -EOVERFLOW;
+ /* f_files and f_ffree may be -1; it's okay
+ * to stuff that into 32 bits */
+ if (kbuf->f_files != 0xffffffffffffffffULL
+ && (kbuf->f_files & 0xffffffff00000000ULL))
+ return -EOVERFLOW;
+ if (kbuf->f_ffree != 0xffffffffffffffffULL
+ && (kbuf->f_ffree & 0xffffffff00000000ULL))
+ return -EOVERFLOW;
+ }
+ if (!access_ok(VERIFY_WRITE, ubuf, sizeof(*ubuf)) ||
+ __put_user(kbuf->f_type, &ubuf->f_type) ||
+ __put_user(kbuf->f_bsize, &ubuf->f_bsize) ||
+ __put_user(kbuf->f_blocks, &ubuf->f_blocks) ||
+ __put_user(kbuf->f_bfree, &ubuf->f_bfree) ||
+ __put_user(kbuf->f_bavail, &ubuf->f_bavail) ||
+ __put_user(kbuf->f_files, &ubuf->f_files) ||
+ __put_user(kbuf->f_ffree, &ubuf->f_ffree) ||
+ __put_user(kbuf->f_namelen, &ubuf->f_namelen) ||
+ __put_user(kbuf->f_fsid.val[0], &ubuf->f_fsid.val[0]) ||
+ __put_user(kbuf->f_fsid.val[1], &ubuf->f_fsid.val[1]) ||
+ __put_user(kbuf->f_frsize, &ubuf->f_frsize) ||
+ __put_user(kbuf->f_flags, &ubuf->f_flags) ||
+ __clear_user(ubuf->f_spare, sizeof(ubuf->f_spare)))
+ return -EFAULT;
+ return 0;
+}
+
+COMPAT_SYSCALL_DEFINE3(statfs64, const char __user *, pathname, compat_size_t, sz, struct compat_statfs64 __user *, buf)
+{
+ struct kstatfs tmp;
+ int error;
+
+ if (sz != sizeof(*buf))
+ return -EINVAL;
+
+ error = user_statfs(pathname, &tmp);
+ if (!error)
+ error = put_compat_statfs64(buf, &tmp);
+ return error;
+}
+
+COMPAT_SYSCALL_DEFINE3(fstatfs64, unsigned int, fd, compat_size_t, sz, struct compat_statfs64 __user *, buf)
+{
+ struct kstatfs tmp;
+ int error;
+
+ if (sz != sizeof(*buf))
+ return -EINVAL;
+
+ error = fd_statfs(fd, &tmp);
+ if (!error)
+ error = put_compat_statfs64(buf, &tmp);
+ return error;
+}
+
+/*
+ * This is a copy of sys_ustat, just dealing with a structure layout.
+ * Given how simple this syscall is that apporach is more maintainable
+ * than the various conversion hacks.
+ */
+COMPAT_SYSCALL_DEFINE2(ustat, unsigned, dev, struct compat_ustat __user *, u)
+{
+ struct compat_ustat tmp;
+ struct kstatfs sbuf;
+ int err = vfs_ustat(new_decode_dev(dev), &sbuf);
+ if (err)
+ return err;
+
+ memset(&tmp, 0, sizeof(struct compat_ustat));
+ tmp.f_tfree = sbuf.f_bfree;
+ tmp.f_tinode = sbuf.f_ffree;
+ if (copy_to_user(u, &tmp, sizeof(struct compat_ustat)))
+ return -EFAULT;
+ return 0;
+}
+#endif
diff --git a/fs/super.c b/fs/super.c
index b8b6a08..adb0c0d 100644
--- a/fs/super.c
+++ b/fs/super.c
@@ -446,6 +446,10 @@
hlist_del_init(&sb->s_instances);
spin_unlock(&sb_lock);
up_write(&sb->s_umount);
+ if (sb->s_bdi != &noop_backing_dev_info) {
+ bdi_put(sb->s_bdi);
+ sb->s_bdi = &noop_backing_dev_info;
+ }
}
EXPORT_SYMBOL(generic_shutdown_super);
@@ -1049,12 +1053,8 @@
{
s->s_bdev = data;
s->s_dev = s->s_bdev->bd_dev;
+ s->s_bdi = bdi_get(s->s_bdev->bd_bdi);
- /*
- * We set the bdi here to the queue backing, file systems can
- * overwrite this in ->fill_super()
- */
- s->s_bdi = bdev_get_queue(s->s_bdev)->backing_dev_info;
return 0;
}
@@ -1256,6 +1256,49 @@
}
/*
+ * Setup private BDI for given superblock. It gets automatically cleaned up
+ * in generic_shutdown_super().
+ */
+int super_setup_bdi_name(struct super_block *sb, char *fmt, ...)
+{
+ struct backing_dev_info *bdi;
+ int err;
+ va_list args;
+
+ bdi = bdi_alloc(GFP_KERNEL);
+ if (!bdi)
+ return -ENOMEM;
+
+ bdi->name = sb->s_type->name;
+
+ va_start(args, fmt);
+ err = bdi_register_va(bdi, fmt, args);
+ va_end(args);
+ if (err) {
+ bdi_put(bdi);
+ return err;
+ }
+ WARN_ON(sb->s_bdi != &noop_backing_dev_info);
+ sb->s_bdi = bdi;
+
+ return 0;
+}
+EXPORT_SYMBOL(super_setup_bdi_name);
+
+/*
+ * Setup private BDI for given superblock. I gets automatically cleaned up
+ * in generic_shutdown_super().
+ */
+int super_setup_bdi(struct super_block *sb)
+{
+ static atomic_long_t bdi_seq = ATOMIC_LONG_INIT(0);
+
+ return super_setup_bdi_name(sb, "%.28s-%ld", sb->s_type->name,
+ atomic_long_inc_return(&bdi_seq));
+}
+EXPORT_SYMBOL(super_setup_bdi);
+
+/*
* This is an internal function, please use sb_end_{write,pagefault,intwrite}
* instead.
*/
diff --git a/fs/ubifs/debug.c b/fs/ubifs/debug.c
index 1e712a36..718b749 100644
--- a/fs/ubifs/debug.c
+++ b/fs/ubifs/debug.c
@@ -32,6 +32,7 @@
#include <linux/math64.h>
#include <linux/uaccess.h>
#include <linux/random.h>
+#include <linux/ctype.h>
#include "ubifs.h"
static DEFINE_SPINLOCK(dbg_lock);
@@ -286,8 +287,10 @@
break;
}
- pr_err("\t%d: %s (%s)\n",
- count++, dent->name, get_dent_type(dent->type));
+ pr_err("\t%d: inode %llu, type %s, len %d\n",
+ count++, (unsigned long long) le64_to_cpu(dent->inum),
+ get_dent_type(dent->type),
+ le16_to_cpu(dent->nlen));
fname_name(&nm) = dent->name;
fname_len(&nm) = le16_to_cpu(dent->nlen);
@@ -464,7 +467,8 @@
pr_err("(bad name length, not printing, bad or corrupted node)");
else {
for (i = 0; i < nlen && dent->name[i]; i++)
- pr_cont("%c", dent->name[i]);
+ pr_cont("%c", isprint(dent->name[i]) ?
+ dent->name[i] : '?');
}
pr_cont("\n");
diff --git a/fs/ubifs/dir.c b/fs/ubifs/dir.c
index 30825d88..b777bdd 100644
--- a/fs/ubifs/dir.c
+++ b/fs/ubifs/dir.c
@@ -606,8 +606,8 @@
}
while (1) {
- dbg_gen("feed '%s', ino %llu, new f_pos %#x",
- dent->name, (unsigned long long)le64_to_cpu(dent->inum),
+ dbg_gen("ino %llu, new f_pos %#x",
+ (unsigned long long)le64_to_cpu(dent->inum),
key_hash_flash(c, &dent->key));
ubifs_assert(le64_to_cpu(dent->ch.sqnum) >
ubifs_inode(dir)->creat_sqnum);
@@ -748,6 +748,11 @@
goto out_fname;
lock_2_inodes(dir, inode);
+
+ /* Handle O_TMPFILE corner case, it is allowed to link a O_TMPFILE. */
+ if (inode->i_nlink == 0)
+ ubifs_delete_orphan(c, inode->i_ino);
+
inc_nlink(inode);
ihold(inode);
inode->i_ctime = ubifs_current_time(inode);
@@ -768,6 +773,8 @@
dir->i_size -= sz_change;
dir_ui->ui_size = dir->i_size;
drop_nlink(inode);
+ if (inode->i_nlink == 0)
+ ubifs_add_orphan(c, inode->i_ino);
unlock_2_inodes(dir, inode);
ubifs_release_budget(c, &req);
iput(inode);
@@ -1068,8 +1075,10 @@
}
err = fscrypt_setup_filename(dir, &dentry->d_name, 0, &nm);
- if (err)
+ if (err) {
+ kfree(dev);
goto out_budg;
+ }
sz_change = CALC_DENT_SIZE(fname_len(&nm));
@@ -1316,9 +1325,6 @@
unsigned int uninitialized_var(saved_nlink);
struct fscrypt_name old_nm, new_nm;
- if (flags & ~RENAME_NOREPLACE)
- return -EINVAL;
-
/*
* Budget request settings: deletion direntry, new direntry, removing
* the old inode, and changing old and new parent directory inodes.
diff --git a/fs/ubifs/super.c b/fs/ubifs/super.c
index b73811b..cf4cc99 100644
--- a/fs/ubifs/super.c
+++ b/fs/ubifs/super.c
@@ -1827,7 +1827,6 @@
}
ubifs_umount(c);
- bdi_destroy(&c->bdi);
ubi_close_volume(c->ubi);
mutex_unlock(&c->umount_mutex);
}
@@ -2019,29 +2018,25 @@
goto out;
}
+ err = ubifs_parse_options(c, data, 0);
+ if (err)
+ goto out_close;
+
/*
* UBIFS provides 'backing_dev_info' in order to disable read-ahead. For
* UBIFS, I/O is not deferred, it is done immediately in readpage,
* which means the user would have to wait not just for their own I/O
* but the read-ahead I/O as well i.e. completely pointless.
*
- * Read-ahead will be disabled because @c->bdi.ra_pages is 0.
+ * Read-ahead will be disabled because @sb->s_bdi->ra_pages is 0. Also
+ * @sb->s_bdi->capabilities are initialized to 0 so there won't be any
+ * writeback happening.
*/
- c->bdi.name = "ubifs",
- c->bdi.capabilities = 0;
- err = bdi_init(&c->bdi);
+ err = super_setup_bdi_name(sb, "ubifs_%d_%d", c->vi.ubi_num,
+ c->vi.vol_id);
if (err)
goto out_close;
- err = bdi_register(&c->bdi, NULL, "ubifs_%d_%d",
- c->vi.ubi_num, c->vi.vol_id);
- if (err)
- goto out_bdi;
- err = ubifs_parse_options(c, data, 0);
- if (err)
- goto out_bdi;
-
- sb->s_bdi = &c->bdi;
sb->s_fs_info = c;
sb->s_magic = UBIFS_SUPER_MAGIC;
sb->s_blocksize = UBIFS_BLOCK_SIZE;
@@ -2080,8 +2075,6 @@
ubifs_umount(c);
out_unlock:
mutex_unlock(&c->umount_mutex);
-out_bdi:
- bdi_destroy(&c->bdi);
out_close:
ubi_close_volume(c->ubi);
out:
diff --git a/fs/ubifs/ubifs.h b/fs/ubifs/ubifs.h
index 4d57e48..4da10a6 100644
--- a/fs/ubifs/ubifs.h
+++ b/fs/ubifs/ubifs.h
@@ -972,7 +972,6 @@
* struct ubifs_info - UBIFS file-system description data structure
* (per-superblock).
* @vfs_sb: VFS @struct super_block object
- * @bdi: backing device info object to make VFS happy and disable read-ahead
*
* @highest_inum: highest used inode number
* @max_sqnum: current global sequence number
@@ -1220,7 +1219,6 @@
*/
struct ubifs_info {
struct super_block *vfs_sb;
- struct backing_dev_info bdi;
ino_t highest_inum;
unsigned long long max_sqnum;
@@ -1461,7 +1459,6 @@
extern const struct file_operations ubifs_dir_operations;
extern const struct inode_operations ubifs_dir_inode_operations;
extern const struct inode_operations ubifs_symlink_inode_operations;
-extern struct backing_dev_info ubifs_backing_dev_info;
extern struct ubifs_compressor *ubifs_compressors[UBIFS_COMPR_TYPES_CNT];
/* io.c */
diff --git a/fs/utimes.c b/fs/utimes.c
index 32b15b3..6571d8c 100644
--- a/fs/utimes.c
+++ b/fs/utimes.c
@@ -1,14 +1,10 @@
-#include <linux/compiler.h>
#include <linux/file.h>
-#include <linux/fs.h>
-#include <linux/linkage.h>
#include <linux/mount.h>
#include <linux/namei.h>
-#include <linux/sched.h>
-#include <linux/stat.h>
#include <linux/utime.h>
#include <linux/syscalls.h>
#include <linux/uaccess.h>
+#include <linux/compat.h>
#include <asm/unistd.h>
#ifdef __ARCH_WANT_SYS_UTIME
@@ -219,3 +215,63 @@
{
return sys_futimesat(AT_FDCWD, filename, utimes);
}
+
+#ifdef CONFIG_COMPAT
+/*
+ * Not all architectures have sys_utime, so implement this in terms
+ * of sys_utimes.
+ */
+COMPAT_SYSCALL_DEFINE2(utime, const char __user *, filename,
+ struct compat_utimbuf __user *, t)
+{
+ struct timespec tv[2];
+
+ if (t) {
+ if (get_user(tv[0].tv_sec, &t->actime) ||
+ get_user(tv[1].tv_sec, &t->modtime))
+ return -EFAULT;
+ tv[0].tv_nsec = 0;
+ tv[1].tv_nsec = 0;
+ }
+ return do_utimes(AT_FDCWD, filename, t ? tv : NULL, 0);
+}
+
+COMPAT_SYSCALL_DEFINE4(utimensat, unsigned int, dfd, const char __user *, filename, struct compat_timespec __user *, t, int, flags)
+{
+ struct timespec tv[2];
+
+ if (t) {
+ if (compat_get_timespec(&tv[0], &t[0]) ||
+ compat_get_timespec(&tv[1], &t[1]))
+ return -EFAULT;
+
+ if (tv[0].tv_nsec == UTIME_OMIT && tv[1].tv_nsec == UTIME_OMIT)
+ return 0;
+ }
+ return do_utimes(dfd, filename, t ? tv : NULL, flags);
+}
+
+COMPAT_SYSCALL_DEFINE3(futimesat, unsigned int, dfd, const char __user *, filename, struct compat_timeval __user *, t)
+{
+ struct timespec tv[2];
+
+ if (t) {
+ if (get_user(tv[0].tv_sec, &t[0].tv_sec) ||
+ get_user(tv[0].tv_nsec, &t[0].tv_usec) ||
+ get_user(tv[1].tv_sec, &t[1].tv_sec) ||
+ get_user(tv[1].tv_nsec, &t[1].tv_usec))
+ return -EFAULT;
+ if (tv[0].tv_nsec >= 1000000 || tv[0].tv_nsec < 0 ||
+ tv[1].tv_nsec >= 1000000 || tv[1].tv_nsec < 0)
+ return -EINVAL;
+ tv[0].tv_nsec *= 1000;
+ tv[1].tv_nsec *= 1000;
+ }
+ return do_utimes(dfd, filename, t ? tv : NULL, 0);
+}
+
+COMPAT_SYSCALL_DEFINE2(utimes, const char __user *, filename, struct compat_timeval __user *, t)
+{
+ return compat_sys_futimesat(AT_FDCWD, filename, t);
+}
+#endif
diff --git a/fs/xfs/xfs_bmap_util.c b/fs/xfs/xfs_bmap_util.c
index 828532c..8795e9c 100644
--- a/fs/xfs/xfs_bmap_util.c
+++ b/fs/xfs/xfs_bmap_util.c
@@ -81,7 +81,7 @@
return blkdev_issue_zeroout(xfs_find_bdev_for_inode(VFS_I(ip)),
block << (mp->m_super->s_blocksize_bits - 9),
count_fsb << (mp->m_super->s_blocksize_bits - 9),
- GFP_NOFS, true);
+ GFP_NOFS, 0);
}
int
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index ef0ae8a..2fc678e 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -88,6 +88,7 @@
}
bool acpi_dev_found(const char *hid);
+bool acpi_dev_present(const char *hid, const char *uid, s64 hrv);
#ifdef CONFIG_ACPI
@@ -386,6 +387,7 @@
const char *name;
acpi_handle handle;
struct fwnode_handle fwnode;
+ struct fwnode_handle *parent;
struct acpi_device_data data;
struct list_head sibling;
struct kobject kobj;
diff --git a/include/acpi/cppc_acpi.h b/include/acpi/cppc_acpi.h
index 427a7c3..2010c05 100644
--- a/include/acpi/cppc_acpi.h
+++ b/include/acpi/cppc_acpi.h
@@ -103,6 +103,7 @@
u32 highest_perf;
u32 nominal_perf;
u32 lowest_perf;
+ u32 lowest_nonlinear_perf;
};
struct cppc_perf_ctrls {
@@ -115,7 +116,7 @@
u64 reference;
u64 delivered;
u64 reference_perf;
- u64 ctr_wrap_time;
+ u64 wraparound_time;
};
/* Per CPU container for runtime CPPC management. */
diff --git a/include/asm-generic/bug.h b/include/asm-generic/bug.h
index 6f96247..d6f4aed 100644
--- a/include/asm-generic/bug.h
+++ b/include/asm-generic/bug.h
@@ -5,7 +5,9 @@
#ifdef CONFIG_GENERIC_BUG
#define BUGFLAG_WARNING (1 << 0)
-#define BUGFLAG_TAINT(taint) (BUGFLAG_WARNING | ((taint) << 8))
+#define BUGFLAG_ONCE (1 << 1)
+#define BUGFLAG_DONE (1 << 2)
+#define BUGFLAG_TAINT(taint) ((taint) << 8)
#define BUG_GET_TAINT(bug) ((bug)->flags >> 8)
#endif
@@ -55,6 +57,18 @@
#define BUG_ON(condition) do { if (unlikely(condition)) BUG(); } while (0)
#endif
+#ifdef __WARN_FLAGS
+#define __WARN_TAINT(taint) __WARN_FLAGS(BUGFLAG_TAINT(taint))
+#define __WARN_ONCE_TAINT(taint) __WARN_FLAGS(BUGFLAG_ONCE|BUGFLAG_TAINT(taint))
+
+#define WARN_ON_ONCE(condition) ({ \
+ int __ret_warn_on = !!(condition); \
+ if (unlikely(__ret_warn_on)) \
+ __WARN_ONCE_TAINT(TAINT_WARN); \
+ unlikely(__ret_warn_on); \
+})
+#endif
+
/*
* WARN(), WARN_ON(), WARN_ON_ONCE, and so on can be used to report
* significant issues that need prompt attention if they should ever
@@ -97,7 +111,7 @@
#endif
#ifndef WARN
-#define WARN(condition, format...) ({ \
+#define WARN(condition, format...) ({ \
int __ret_warn_on = !!(condition); \
if (unlikely(__ret_warn_on)) \
__WARN_printf(format); \
@@ -112,6 +126,7 @@
unlikely(__ret_warn_on); \
})
+#ifndef WARN_ON_ONCE
#define WARN_ON_ONCE(condition) ({ \
static bool __section(.data.unlikely) __warned; \
int __ret_warn_once = !!(condition); \
@@ -122,6 +137,7 @@
} \
unlikely(__ret_warn_once); \
})
+#endif
#define WARN_ONCE(condition, format...) ({ \
static bool __section(.data.unlikely) __warned; \
diff --git a/include/asm-generic/extable.h b/include/asm-generic/extable.h
new file mode 100644
index 0000000..ca14c66
--- /dev/null
+++ b/include/asm-generic/extable.h
@@ -0,0 +1,26 @@
+#ifndef __ASM_GENERIC_EXTABLE_H
+#define __ASM_GENERIC_EXTABLE_H
+
+/*
+ * The exception table consists of pairs of addresses: the first is the
+ * address of an instruction that is allowed to fault, and the second is
+ * the address at which the program should continue. No registers are
+ * modified, so it is entirely up to the continuation code to figure out
+ * what to do.
+ *
+ * All the routines below use bits of fixup code that are out of line
+ * with the main instruction path. This means when everything is well,
+ * we don't even have to jump over them. Further, they do not intrude
+ * on our cache or tlb entries.
+ */
+
+struct exception_table_entry
+{
+ unsigned long insn, fixup;
+};
+
+
+struct pt_regs;
+extern int fixup_exception(struct pt_regs *regs);
+
+#endif
diff --git a/include/asm-generic/mm_hooks.h b/include/asm-generic/mm_hooks.h
index cc5d9a1..41e5b67 100644
--- a/include/asm-generic/mm_hooks.h
+++ b/include/asm-generic/mm_hooks.h
@@ -32,10 +32,4 @@
/* by default, allow everything */
return true;
}
-
-static inline bool arch_pte_access_permitted(pte_t pte, bool write)
-{
- /* by default, allow everything */
- return true;
-}
#endif /* _ASM_GENERIC_MM_HOOKS_H */
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index 1fad160..7dfa767 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -341,6 +341,31 @@
}
#endif
+#ifndef pte_access_permitted
+#define pte_access_permitted(pte, write) \
+ (pte_present(pte) && (!(write) || pte_write(pte)))
+#endif
+
+#ifndef pmd_access_permitted
+#define pmd_access_permitted(pmd, write) \
+ (pmd_present(pmd) && (!(write) || pmd_write(pmd)))
+#endif
+
+#ifndef pud_access_permitted
+#define pud_access_permitted(pud, write) \
+ (pud_present(pud) && (!(write) || pud_write(pud)))
+#endif
+
+#ifndef p4d_access_permitted
+#define p4d_access_permitted(p4d, write) \
+ (p4d_present(p4d) && (!(write) || p4d_write(p4d)))
+#endif
+
+#ifndef pgd_access_permitted
+#define pgd_access_permitted(pgd, write) \
+ (pgd_present(pgd) && (!(write) || pgd_write(pgd)))
+#endif
+
#ifndef __HAVE_ARCH_PMD_SAME
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b)
diff --git a/include/asm-generic/uaccess.h b/include/asm-generic/uaccess.h
index cc6bb31..bbe4bb4 100644
--- a/include/asm-generic/uaccess.h
+++ b/include/asm-generic/uaccess.h
@@ -6,7 +6,6 @@
* on any machine that has kernel and user data in the same
* address space, e.g. all NOMMU machines.
*/
-#include <linux/sched.h>
#include <linux/string.h>
#include <asm/segment.h>
@@ -35,9 +34,6 @@
#define segment_eq(a, b) ((a).seg == (b).seg)
#endif
-#define VERIFY_READ 0
-#define VERIFY_WRITE 1
-
#define access_ok(type, addr, size) __access_ok((unsigned long)(addr),(size))
/*
@@ -52,87 +48,6 @@
#endif
/*
- * The exception table consists of pairs of addresses: the first is the
- * address of an instruction that is allowed to fault, and the second is
- * the address at which the program should continue. No registers are
- * modified, so it is entirely up to the continuation code to figure out
- * what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path. This means when everything is well,
- * we don't even have to jump over them. Further, they do not intrude
- * on our cache or tlb entries.
- */
-
-struct exception_table_entry
-{
- unsigned long insn, fixup;
-};
-
-/*
- * architectures with an MMU should override these two
- */
-#ifndef __copy_from_user
-static inline __must_check long __copy_from_user(void *to,
- const void __user * from, unsigned long n)
-{
- if (__builtin_constant_p(n)) {
- switch(n) {
- case 1:
- *(u8 *)to = *(u8 __force *)from;
- return 0;
- case 2:
- *(u16 *)to = *(u16 __force *)from;
- return 0;
- case 4:
- *(u32 *)to = *(u32 __force *)from;
- return 0;
-#ifdef CONFIG_64BIT
- case 8:
- *(u64 *)to = *(u64 __force *)from;
- return 0;
-#endif
- default:
- break;
- }
- }
-
- memcpy(to, (const void __force *)from, n);
- return 0;
-}
-#endif
-
-#ifndef __copy_to_user
-static inline __must_check long __copy_to_user(void __user *to,
- const void *from, unsigned long n)
-{
- if (__builtin_constant_p(n)) {
- switch(n) {
- case 1:
- *(u8 __force *)to = *(u8 *)from;
- return 0;
- case 2:
- *(u16 __force *)to = *(u16 *)from;
- return 0;
- case 4:
- *(u32 __force *)to = *(u32 *)from;
- return 0;
-#ifdef CONFIG_64BIT
- case 8:
- *(u64 __force *)to = *(u64 *)from;
- return 0;
-#endif
- default:
- break;
- }
- }
-
- memcpy((void __force *)to, from, n);
- return 0;
-}
-#endif
-
-/*
* These are the main single-value transfer routines. They automatically
* use the right size if we just have the right pointer type.
* This version just falls back to copy_{from,to}_user, which should
@@ -171,8 +86,7 @@
static inline int __put_user_fn(size_t size, void __user *ptr, void *x)
{
- size = __copy_to_user(ptr, x, size);
- return size ? -EFAULT : size;
+ return unlikely(raw_copy_to_user(ptr, x, size)) ? -EFAULT : 0;
}
#define __put_user_fn(sz, u, k) __put_user_fn(sz, u, k)
@@ -187,28 +101,28 @@
__chk_user_ptr(ptr); \
switch (sizeof(*(ptr))) { \
case 1: { \
- unsigned char __x; \
+ unsigned char __x = 0; \
__gu_err = __get_user_fn(sizeof (*(ptr)), \
ptr, &__x); \
(x) = *(__force __typeof__(*(ptr)) *) &__x; \
break; \
}; \
case 2: { \
- unsigned short __x; \
+ unsigned short __x = 0; \
__gu_err = __get_user_fn(sizeof (*(ptr)), \
ptr, &__x); \
(x) = *(__force __typeof__(*(ptr)) *) &__x; \
break; \
}; \
case 4: { \
- unsigned int __x; \
+ unsigned int __x = 0; \
__gu_err = __get_user_fn(sizeof (*(ptr)), \
ptr, &__x); \
(x) = *(__force __typeof__(*(ptr)) *) &__x; \
break; \
}; \
case 8: { \
- unsigned long long __x; \
+ unsigned long long __x = 0; \
__gu_err = __get_user_fn(sizeof (*(ptr)), \
ptr, &__x); \
(x) = *(__force __typeof__(*(ptr)) *) &__x; \
@@ -233,12 +147,7 @@
#ifndef __get_user_fn
static inline int __get_user_fn(size_t size, const void __user *ptr, void *x)
{
- size_t n = __copy_from_user(x, ptr, size);
- if (unlikely(n)) {
- memset(x + (size - n), 0, n);
- return -EFAULT;
- }
- return 0;
+ return unlikely(raw_copy_from_user(x, ptr, size)) ? -EFAULT : 0;
}
#define __get_user_fn(sz, u, k) __get_user_fn(sz, u, k)
@@ -247,36 +156,6 @@
extern int __get_user_bad(void) __attribute__((noreturn));
-#ifndef __copy_from_user_inatomic
-#define __copy_from_user_inatomic __copy_from_user
-#endif
-
-#ifndef __copy_to_user_inatomic
-#define __copy_to_user_inatomic __copy_to_user
-#endif
-
-static inline long copy_from_user(void *to,
- const void __user * from, unsigned long n)
-{
- unsigned long res = n;
- might_fault();
- if (likely(access_ok(VERIFY_READ, from, n)))
- res = __copy_from_user(to, from, n);
- if (unlikely(res))
- memset(to + (n - res), 0, res);
- return res;
-}
-
-static inline long copy_to_user(void __user *to,
- const void *from, unsigned long n)
-{
- might_fault();
- if (access_ok(VERIFY_WRITE, to, n))
- return __copy_to_user(to, from, n);
- else
- return n;
-}
-
/*
* Copy a null terminated string from userspace.
*/
@@ -348,4 +227,6 @@
return __clear_user(to, n);
}
+#include <asm/extable.h>
+
#endif /* __ASM_GENERIC_UACCESS_H */
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 143db9c..3558f4e 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -287,8 +287,6 @@
*(.rodata1) \
} \
\
- BUG_TABLE \
- \
/* PCI quirks */ \
.pci_fixup : AT(ADDR(.pci_fixup) - LOAD_OFFSET) { \
VMLINUX_SYMBOL(__start_pci_fixups_early) = .; \
@@ -857,7 +855,8 @@
READ_MOSTLY_DATA(cacheline) \
DATA_DATA \
CONSTRUCTORS \
- }
+ } \
+ BUG_TABLE
#define INIT_TEXT_SECTION(inittext_align) \
. = ALIGN(inittext_align); \
diff --git a/include/clocksource/arm_arch_timer.h b/include/clocksource/arm_arch_timer.h
index caedb74..cc805b7 100644
--- a/include/clocksource/arm_arch_timer.h
+++ b/include/clocksource/arm_arch_timer.h
@@ -16,9 +16,13 @@
#ifndef __CLKSOURCE_ARM_ARCH_TIMER_H
#define __CLKSOURCE_ARM_ARCH_TIMER_H
+#include <linux/bitops.h>
#include <linux/timecounter.h>
#include <linux/types.h>
+#define ARCH_TIMER_TYPE_CP15 BIT(0)
+#define ARCH_TIMER_TYPE_MEM BIT(1)
+
#define ARCH_TIMER_CTRL_ENABLE (1 << 0)
#define ARCH_TIMER_CTRL_IT_MASK (1 << 1)
#define ARCH_TIMER_CTRL_IT_STAT (1 << 2)
@@ -34,11 +38,27 @@
ARCH_TIMER_REG_TVAL,
};
+enum arch_timer_ppi_nr {
+ ARCH_TIMER_PHYS_SECURE_PPI,
+ ARCH_TIMER_PHYS_NONSECURE_PPI,
+ ARCH_TIMER_VIRT_PPI,
+ ARCH_TIMER_HYP_PPI,
+ ARCH_TIMER_MAX_TIMER_PPI
+};
+
+enum arch_timer_spi_nr {
+ ARCH_TIMER_PHYS_SPI,
+ ARCH_TIMER_VIRT_SPI,
+ ARCH_TIMER_MAX_TIMER_SPI
+};
+
#define ARCH_TIMER_PHYS_ACCESS 0
#define ARCH_TIMER_VIRT_ACCESS 1
#define ARCH_TIMER_MEM_PHYS_ACCESS 2
#define ARCH_TIMER_MEM_VIRT_ACCESS 3
+#define ARCH_TIMER_MEM_MAX_FRAMES 8
+
#define ARCH_TIMER_USR_PCT_ACCESS_EN (1 << 0) /* physical counter */
#define ARCH_TIMER_USR_VCT_ACCESS_EN (1 << 1) /* virtual counter */
#define ARCH_TIMER_VIRT_EVT_EN (1 << 2)
@@ -54,6 +74,20 @@
int virtual_irq;
};
+struct arch_timer_mem_frame {
+ bool valid;
+ phys_addr_t cntbase;
+ size_t size;
+ int phys_irq;
+ int virt_irq;
+};
+
+struct arch_timer_mem {
+ phys_addr_t cntctlbase;
+ size_t size;
+ struct arch_timer_mem_frame frame[ARCH_TIMER_MEM_MAX_FRAMES];
+};
+
#ifdef CONFIG_ARM_ARCH_TIMER
extern u32 arch_timer_get_rate(void);
diff --git a/include/crypto/gf128mul.h b/include/crypto/gf128mul.h
index 592d47e..0977fb1 100644
--- a/include/crypto/gf128mul.h
+++ b/include/crypto/gf128mul.h
@@ -43,12 +43,13 @@
---------------------------------------------------------------------------
Issue Date: 31/01/2006
- An implementation of field multiplication in Galois Field GF(128)
+ An implementation of field multiplication in Galois Field GF(2^128)
*/
#ifndef _CRYPTO_GF128MUL_H
#define _CRYPTO_GF128MUL_H
+#include <asm/byteorder.h>
#include <crypto/b128ops.h>
#include <linux/slab.h>
@@ -65,7 +66,7 @@
* are left and the lsb's are right. char b[16] is an array and b[0] is
* the first octet.
*
- * 80000000 00000000 00000000 00000000 .... 00000000 00000000 00000000
+ * 10000000 00000000 00000000 00000000 .... 00000000 00000000 00000000
* b[0] b[1] b[2] b[3] b[13] b[14] b[15]
*
* Every bit is a coefficient of some power of X. We can store the bits
@@ -85,15 +86,17 @@
* Both of the above formats are easy to implement on big-endian
* machines.
*
- * EME (which is patent encumbered) uses the ble format (bits are stored
- * in big endian order and the bytes in little endian). The above buffer
- * represents X^7 in this case and the primitive polynomial is b[0] = 0x87.
+ * XTS and EME (the latter of which is patent encumbered) use the ble
+ * format (bits are stored in big endian order and the bytes in little
+ * endian). The above buffer represents X^7 in this case and the
+ * primitive polynomial is b[0] = 0x87.
*
* The common machine word-size is smaller than 128 bits, so to make
* an efficient implementation we must split into machine word sizes.
- * This file uses one 32bit for the moment. Machine endianness comes into
- * play. The lle format in relation to machine endianness is discussed
- * below by the original author of gf128mul Dr Brian Gladman.
+ * This implementation uses 64-bit words for the moment. Machine
+ * endianness comes into play. The lle format in relation to machine
+ * endianness is discussed below by the original author of gf128mul Dr
+ * Brian Gladman.
*
* Let's look at the bbe and ble format on a little endian machine.
*
@@ -127,10 +130,10 @@
* machines this will automatically aligned to wordsize and on a 64-bit
* machine also.
*/
-/* Multiply a GF128 field element by x. Field elements are held in arrays
- of bytes in which field bits 8n..8n + 7 are held in byte[n], with lower
- indexed bits placed in the more numerically significant bit positions
- within bytes.
+/* Multiply a GF(2^128) field element by x. Field elements are
+ held in arrays of bytes in which field bits 8n..8n + 7 are held in
+ byte[n], with lower indexed bits placed in the more numerically
+ significant bit positions within bytes.
On little endian machines the bit indexes translate into the bit
positions within four 32-bit words in the following way
@@ -161,8 +164,58 @@
void gf128mul_bbe(be128 *a, const be128 *b);
-/* multiply by x in ble format, needed by XTS */
-void gf128mul_x_ble(be128 *a, const be128 *b);
+/*
+ * The following functions multiply a field element by x in
+ * the polynomial field representation. They use 64-bit word operations
+ * to gain speed but compensate for machine endianness and hence work
+ * correctly on both styles of machine.
+ *
+ * They are defined here for performance.
+ */
+
+static inline u64 gf128mul_mask_from_bit(u64 x, int which)
+{
+ /* a constant-time version of 'x & ((u64)1 << which) ? (u64)-1 : 0' */
+ return ((s64)(x << (63 - which)) >> 63);
+}
+
+static inline void gf128mul_x_lle(be128 *r, const be128 *x)
+{
+ u64 a = be64_to_cpu(x->a);
+ u64 b = be64_to_cpu(x->b);
+
+ /* equivalent to gf128mul_table_le[(b << 7) & 0xff] << 48
+ * (see crypto/gf128mul.c): */
+ u64 _tt = gf128mul_mask_from_bit(b, 0) & ((u64)0xe1 << 56);
+
+ r->b = cpu_to_be64((b >> 1) | (a << 63));
+ r->a = cpu_to_be64((a >> 1) ^ _tt);
+}
+
+static inline void gf128mul_x_bbe(be128 *r, const be128 *x)
+{
+ u64 a = be64_to_cpu(x->a);
+ u64 b = be64_to_cpu(x->b);
+
+ /* equivalent to gf128mul_table_be[a >> 63] (see crypto/gf128mul.c): */
+ u64 _tt = gf128mul_mask_from_bit(a, 63) & 0x87;
+
+ r->a = cpu_to_be64((a << 1) | (b >> 63));
+ r->b = cpu_to_be64((b << 1) ^ _tt);
+}
+
+/* needed by XTS */
+static inline void gf128mul_x_ble(le128 *r, const le128 *x)
+{
+ u64 a = le64_to_cpu(x->a);
+ u64 b = le64_to_cpu(x->b);
+
+ /* equivalent to gf128mul_table_be[b >> 63] (see crypto/gf128mul.c): */
+ u64 _tt = gf128mul_mask_from_bit(a, 63) & 0x87;
+
+ r->a = cpu_to_le64((a << 1) | (b >> 63));
+ r->b = cpu_to_le64((b << 1) ^ _tt);
+}
/* 4k table optimization */
@@ -172,8 +225,8 @@
struct gf128mul_4k *gf128mul_init_4k_lle(const be128 *g);
struct gf128mul_4k *gf128mul_init_4k_bbe(const be128 *g);
-void gf128mul_4k_lle(be128 *a, struct gf128mul_4k *t);
-void gf128mul_4k_bbe(be128 *a, struct gf128mul_4k *t);
+void gf128mul_4k_lle(be128 *a, const struct gf128mul_4k *t);
+void gf128mul_4k_bbe(be128 *a, const struct gf128mul_4k *t);
static inline void gf128mul_free_4k(struct gf128mul_4k *t)
{
@@ -194,6 +247,6 @@
*/
struct gf128mul_64k *gf128mul_init_64k_bbe(const be128 *g);
void gf128mul_free_64k(struct gf128mul_64k *t);
-void gf128mul_64k_bbe(be128 *a, struct gf128mul_64k *t);
+void gf128mul_64k_bbe(be128 *a, const struct gf128mul_64k *t);
#endif /* _CRYPTO_GF128MUL_H */
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index 1de2b5a..51052f6 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -78,4 +78,7 @@
*/
int crypto_unregister_acomp(struct acomp_alg *alg);
+int crypto_register_acomps(struct acomp_alg *algs, int count);
+void crypto_unregister_acomps(struct acomp_alg *algs, int count);
+
#endif
diff --git a/include/crypto/internal/scompress.h b/include/crypto/internal/scompress.h
index 3fda3c5..ccad9b2 100644
--- a/include/crypto/internal/scompress.h
+++ b/include/crypto/internal/scompress.h
@@ -133,4 +133,7 @@
*/
int crypto_unregister_scomp(struct scomp_alg *alg);
+int crypto_register_scomps(struct scomp_alg *algs, int count);
+void crypto_unregister_scomps(struct scomp_alg *algs, int count);
+
#endif
diff --git a/include/crypto/kpp.h b/include/crypto/kpp.h
index 4307a2f..ce8e1f7 100644
--- a/include/crypto/kpp.h
+++ b/include/crypto/kpp.h
@@ -74,7 +74,7 @@
* @base: Common crypto API algorithm data structure
*/
struct kpp_alg {
- int (*set_secret)(struct crypto_kpp *tfm, void *buffer,
+ int (*set_secret)(struct crypto_kpp *tfm, const void *buffer,
unsigned int len);
int (*generate_public_key)(struct kpp_request *req);
int (*compute_shared_secret)(struct kpp_request *req);
@@ -273,8 +273,8 @@
*
* Return: zero on success; error code in case of error
*/
-static inline int crypto_kpp_set_secret(struct crypto_kpp *tfm, void *buffer,
- unsigned int len)
+static inline int crypto_kpp_set_secret(struct crypto_kpp *tfm,
+ const void *buffer, unsigned int len)
{
struct kpp_alg *alg = crypto_kpp_alg(tfm);
diff --git a/include/crypto/xts.h b/include/crypto/xts.h
index 77b6306..c0bde30 100644
--- a/include/crypto/xts.h
+++ b/include/crypto/xts.h
@@ -11,7 +11,7 @@
#define XTS_BLOCK_SIZE 16
struct xts_crypt_req {
- be128 *tbuf;
+ le128 *tbuf;
unsigned int tbuflen;
void *tweak_ctx;
diff --git a/include/dt-bindings/clock/gxbb-clkc.h b/include/dt-bindings/clock/gxbb-clkc.h
index 692846c..63f4c2c 100644
--- a/include/dt-bindings/clock/gxbb-clkc.h
+++ b/include/dt-bindings/clock/gxbb-clkc.h
@@ -12,9 +12,10 @@
#define CLKID_FCLK_DIV4 6
#define CLKID_CLK81 12
#define CLKID_MPLL2 15
-#define CLKID_SPI 34
#define CLKID_I2C 22
#define CLKID_SAR_ADC 23
+#define CLKID_RNG0 25
+#define CLKID_SPI 34
#define CLKID_ETH 36
#define CLKID_USB0 50
#define CLKID_USB1 51
diff --git a/include/dt-bindings/pinctrl/mt7623-pinfunc.h b/include/dt-bindings/pinctrl/mt7623-pinfunc.h
index 2f00bdc..436a87b 100644
--- a/include/dt-bindings/pinctrl/mt7623-pinfunc.h
+++ b/include/dt-bindings/pinctrl/mt7623-pinfunc.h
@@ -185,6 +185,12 @@
#define MT7623_PIN_56_SPI0_MO_FUNC_SPI0_MO (MTK_PIN_NO(56) | 1)
#define MT7623_PIN_56_SPI0_MO_FUNC_SPI0_MI (MTK_PIN_NO(56) | 2)
+#define MT7623_PIN_57_SDA1_FUNC_GPIO57 (MTK_PIN_NO(57) | 0)
+#define MT7623_PIN_57_SDA1_FUNC_SDA1 (MTK_PIN_NO(57) | 1)
+
+#define MT7623_PIN_58_SCL1_FUNC_GPIO58 (MTK_PIN_NO(58) | 0)
+#define MT7623_PIN_58_SCL1_FUNC_SCL1 (MTK_PIN_NO(58) | 1)
+
#define MT7623_PIN_60_WB_RSTB_FUNC_GPIO60 (MTK_PIN_NO(60) | 0)
#define MT7623_PIN_60_WB_RSTB_FUNC_WB_RSTB (MTK_PIN_NO(60) | 1)
@@ -244,6 +250,22 @@
#define MT7623_PIN_76_SCL0_FUNC_GPIO76 (MTK_PIN_NO(76) | 0)
#define MT7623_PIN_76_SCL0_FUNC_SCL0 (MTK_PIN_NO(76) | 1)
+#define MT7623_PIN_79_URXD0_FUNC_GPIO79 (MTK_PIN_NO(79) | 0)
+#define MT7623_PIN_79_URXD0_FUNC_URXD0 (MTK_PIN_NO(79) | 1)
+#define MT7623_PIN_79_URXD0_FUNC_UTXD0 (MTK_PIN_NO(79) | 2)
+
+#define MT7623_PIN_80_UTXD0_FUNC_GPIO80 (MTK_PIN_NO(80) | 0)
+#define MT7623_PIN_80_UTXD0_FUNC_UTXD0 (MTK_PIN_NO(80) | 1)
+#define MT7623_PIN_80_UTXD0_FUNC_URXD0 (MTK_PIN_NO(80) | 2)
+
+#define MT7623_PIN_81_URXD1_FUNC_GPIO81 (MTK_PIN_NO(81) | 0)
+#define MT7623_PIN_81_URXD1_FUNC_URXD1 (MTK_PIN_NO(81) | 1)
+#define MT7623_PIN_81_URXD1_FUNC_UTXD1 (MTK_PIN_NO(81) | 2)
+
+#define MT7623_PIN_82_UTXD1_FUNC_GPIO82 (MTK_PIN_NO(82) | 0)
+#define MT7623_PIN_82_UTXD1_FUNC_UTXD1 (MTK_PIN_NO(82) | 1)
+#define MT7623_PIN_82_UTXD1_FUNC_URXD1 (MTK_PIN_NO(82) | 2)
+
#define MT7623_PIN_83_LCM_RST_FUNC_GPIO83 (MTK_PIN_NO(83) | 0)
#define MT7623_PIN_83_LCM_RST_FUNC_LCM_RST (MTK_PIN_NO(83) | 1)
@@ -351,10 +373,10 @@
#define MT7623_PIN_122_GPIO122_FUNC_SDA2 (MTK_PIN_NO(122) | 4)
#define MT7623_PIN_122_GPIO122_FUNC_URXD0 (MTK_PIN_NO(122) | 5)
-#define MT7623_PIN_123_GPIO123_FUNC_GPIO123 (MTK_PIN_NO(123) | 0)
-#define MT7623_PIN_123_GPIO123_FUNC_TEST (MTK_PIN_NO(123) | 1)
-#define MT7623_PIN_123_GPIO123_FUNC_SCL2 (MTK_PIN_NO(123) | 4)
-#define MT7623_PIN_123_GPIO123_FUNC_UTXD0 (MTK_PIN_NO(123) | 5)
+#define MT7623_PIN_123_HTPLG_FUNC_GPIO123 (MTK_PIN_NO(123) | 0)
+#define MT7623_PIN_123_HTPLG_FUNC_HTPLG (MTK_PIN_NO(123) | 1)
+#define MT7623_PIN_123_HTPLG_FUNC_SCL2 (MTK_PIN_NO(123) | 4)
+#define MT7623_PIN_123_HTPLG_FUNC_UTXD0 (MTK_PIN_NO(123) | 5)
#define MT7623_PIN_124_GPIO124_FUNC_GPIO124 (MTK_PIN_NO(124) | 0)
#define MT7623_PIN_124_GPIO124_FUNC_TEST (MTK_PIN_NO(124) | 1)
diff --git a/include/linux/acpi.h b/include/linux/acpi.h
index 9b05886..f729ada 100644
--- a/include/linux/acpi.h
+++ b/include/linux/acpi.h
@@ -233,10 +233,6 @@
int acpi_table_init (void);
int acpi_table_parse(char *id, acpi_tbl_table_handler handler);
-int __init acpi_parse_entries(char *id, unsigned long table_size,
- acpi_tbl_entry_handler handler,
- struct acpi_table_header *table_header,
- int entry_id, unsigned int max_entries);
int __init acpi_table_parse_entries(char *id, unsigned long table_size,
int entry_id,
acpi_tbl_entry_handler handler,
@@ -595,6 +591,13 @@
int acpi_reconfig_notifier_register(struct notifier_block *nb);
int acpi_reconfig_notifier_unregister(struct notifier_block *nb);
+#ifdef CONFIG_ACPI_GTDT
+int acpi_gtdt_init(struct acpi_table_header *table, int *platform_timer_count);
+int acpi_gtdt_map_ppi(int type);
+bool acpi_gtdt_c3stop(int type);
+int acpi_arch_timer_mem_init(struct arch_timer_mem *timer_mem, int *timer_count);
+#endif
+
#else /* !CONFIG_ACPI */
#define acpi_disabled 1
@@ -611,6 +614,11 @@
return false;
}
+static inline bool acpi_dev_present(const char *hid, const char *uid, s64 hrv)
+{
+ return false;
+}
+
static inline bool is_acpi_node(struct fwnode_handle *fwnode)
{
return false;
@@ -997,8 +1005,16 @@
int acpi_dev_prop_read(struct acpi_device *adev, const char *propname,
enum dev_prop_type proptype, void *val, size_t nval);
-struct fwnode_handle *acpi_get_next_subnode(struct device *dev,
- struct fwnode_handle *subnode);
+struct fwnode_handle *acpi_get_next_subnode(struct fwnode_handle *fwnode,
+ struct fwnode_handle *child);
+struct fwnode_handle *acpi_node_get_parent(struct fwnode_handle *fwnode);
+
+struct fwnode_handle *acpi_graph_get_next_endpoint(struct fwnode_handle *fwnode,
+ struct fwnode_handle *prev);
+int acpi_graph_get_remote_endpoint(struct fwnode_handle *fwnode,
+ struct fwnode_handle **remote,
+ struct fwnode_handle **port,
+ struct fwnode_handle **endpoint);
struct acpi_probe_entry;
typedef bool (*acpi_probe_entry_validate_subtbl)(struct acpi_subtable_header *,
@@ -1115,12 +1131,34 @@
return -ENXIO;
}
-static inline struct fwnode_handle *acpi_get_next_subnode(struct device *dev,
- struct fwnode_handle *subnode)
+static inline struct fwnode_handle *
+acpi_get_next_subnode(struct fwnode_handle *fwnode, struct fwnode_handle *child)
{
return NULL;
}
+static inline struct fwnode_handle *
+acpi_node_get_parent(struct fwnode_handle *fwnode)
+{
+ return NULL;
+}
+
+static inline struct fwnode_handle *
+acpi_graph_get_next_endpoint(struct fwnode_handle *fwnode,
+ struct fwnode_handle *prev)
+{
+ return ERR_PTR(-ENXIO);
+}
+
+static inline int
+acpi_graph_get_remote_endpoint(struct fwnode_handle *fwnode,
+ struct fwnode_handle **remote,
+ struct fwnode_handle **port,
+ struct fwnode_handle **endpoint)
+{
+ return -ENXIO;
+}
+
#define ACPI_DECLARE_PROBE_ENTRY(table, name, table_id, subtable, valid, data, fn) \
static const void * __acpi_table_##name[] \
__attribute__((unused)) \
diff --git a/include/linux/ata.h b/include/linux/ata.h
index af6859b..ad7d9ee 100644
--- a/include/linux/ata.h
+++ b/include/linux/ata.h
@@ -817,11 +817,6 @@
return id[ATA_ID_SCT_CMD_XPORT] & (1 << 3) ? true : false;
}
-static inline bool ata_id_sct_write_same(const u16 *id)
-{
- return id[ATA_ID_SCT_CMD_XPORT] & (1 << 2) ? true : false;
-}
-
static inline bool ata_id_sct_long_sector_access(const u16 *id)
{
return id[ATA_ID_SCT_CMD_XPORT] & (1 << 1) ? true : false;
diff --git a/include/linux/atomic.h b/include/linux/atomic.h
index e71835b..c56be74 100644
--- a/include/linux/atomic.h
+++ b/include/linux/atomic.h
@@ -423,6 +423,29 @@
#endif
#endif /* atomic_cmpxchg_relaxed */
+#ifndef atomic_try_cmpxchg
+
+#define __atomic_try_cmpxchg(type, _p, _po, _n) \
+({ \
+ typeof(_po) __po = (_po); \
+ typeof(*(_po)) __r, __o = *__po; \
+ __r = atomic_cmpxchg##type((_p), __o, (_n)); \
+ if (unlikely(__r != __o)) \
+ *__po = __r; \
+ likely(__r == __o); \
+})
+
+#define atomic_try_cmpxchg(_p, _po, _n) __atomic_try_cmpxchg(, _p, _po, _n)
+#define atomic_try_cmpxchg_relaxed(_p, _po, _n) __atomic_try_cmpxchg(_relaxed, _p, _po, _n)
+#define atomic_try_cmpxchg_acquire(_p, _po, _n) __atomic_try_cmpxchg(_acquire, _p, _po, _n)
+#define atomic_try_cmpxchg_release(_p, _po, _n) __atomic_try_cmpxchg(_release, _p, _po, _n)
+
+#else /* atomic_try_cmpxchg */
+#define atomic_try_cmpxchg_relaxed atomic_try_cmpxchg
+#define atomic_try_cmpxchg_acquire atomic_try_cmpxchg
+#define atomic_try_cmpxchg_release atomic_try_cmpxchg
+#endif /* atomic_try_cmpxchg */
+
/* cmpxchg_relaxed */
#ifndef cmpxchg_relaxed
#define cmpxchg_relaxed cmpxchg
@@ -996,6 +1019,29 @@
#endif
#endif /* atomic64_cmpxchg_relaxed */
+#ifndef atomic64_try_cmpxchg
+
+#define __atomic64_try_cmpxchg(type, _p, _po, _n) \
+({ \
+ typeof(_po) __po = (_po); \
+ typeof(*(_po)) __r, __o = *__po; \
+ __r = atomic64_cmpxchg##type((_p), __o, (_n)); \
+ if (unlikely(__r != __o)) \
+ *__po = __r; \
+ likely(__r == __o); \
+})
+
+#define atomic64_try_cmpxchg(_p, _po, _n) __atomic64_try_cmpxchg(, _p, _po, _n)
+#define atomic64_try_cmpxchg_relaxed(_p, _po, _n) __atomic64_try_cmpxchg(_relaxed, _p, _po, _n)
+#define atomic64_try_cmpxchg_acquire(_p, _po, _n) __atomic64_try_cmpxchg(_acquire, _p, _po, _n)
+#define atomic64_try_cmpxchg_release(_p, _po, _n) __atomic64_try_cmpxchg(_release, _p, _po, _n)
+
+#else /* atomic64_try_cmpxchg */
+#define atomic64_try_cmpxchg_relaxed atomic64_try_cmpxchg
+#define atomic64_try_cmpxchg_acquire atomic64_try_cmpxchg
+#define atomic64_try_cmpxchg_release atomic64_try_cmpxchg
+#endif /* atomic64_try_cmpxchg */
+
#ifndef atomic64_andnot
static inline void atomic64_andnot(long long i, atomic64_t *v)
{
diff --git a/include/linux/backing-dev-defs.h b/include/linux/backing-dev-defs.h
index ad95581..866c433 100644
--- a/include/linux/backing-dev-defs.h
+++ b/include/linux/backing-dev-defs.h
@@ -21,6 +21,7 @@
*/
enum wb_state {
WB_registered, /* bdi_register() was done */
+ WB_shutting_down, /* wb_shutdown() in progress */
WB_writeback_running, /* Writeback is in progress */
WB_has_dirty_io, /* Dirty inodes on ->b_{dirty|io|more_io} */
};
@@ -54,7 +55,9 @@
atomic_t refcnt; /* nr of attached wb's and blkg */
#ifdef CONFIG_CGROUP_WRITEBACK
- struct backing_dev_info *bdi; /* the associated bdi */
+ struct backing_dev_info *__bdi; /* the associated bdi, set to NULL
+ * on bdi unregistration. For memcg-wb
+ * internal use only! */
int blkcg_id; /* ID of the associated blkcg */
struct rb_node rb_node; /* on bdi->cgwb_congestion_tree */
#endif
@@ -143,7 +146,7 @@
congested_fn *congested_fn; /* Function pointer if device is md/dm */
void *congested_data; /* Pointer to aux data for congested func */
- char *name;
+ const char *name;
struct kref refcnt; /* Reference counter for the structure */
unsigned int capabilities; /* Device capabilities */
@@ -161,7 +164,6 @@
#ifdef CONFIG_CGROUP_WRITEBACK
struct radix_tree_root cgwb_tree; /* radix tree of active cgroup wbs */
struct rb_root cgwb_congested_tree; /* their congested states */
- atomic_t usage_cnt; /* counts both cgwbs and cgwb_contested's */
#else
struct bdi_writeback_congested *wb_congested;
#endif
diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
index c52a48c..557d840 100644
--- a/include/linux/backing-dev.h
+++ b/include/linux/backing-dev.h
@@ -17,8 +17,6 @@
#include <linux/backing-dev-defs.h>
#include <linux/slab.h>
-int __must_check bdi_init(struct backing_dev_info *bdi);
-
static inline struct backing_dev_info *bdi_get(struct backing_dev_info *bdi)
{
kref_get(&bdi->refcnt);
@@ -27,16 +25,18 @@
void bdi_put(struct backing_dev_info *bdi);
-__printf(3, 4)
-int bdi_register(struct backing_dev_info *bdi, struct device *parent,
- const char *fmt, ...);
-int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev);
+__printf(2, 3)
+int bdi_register(struct backing_dev_info *bdi, const char *fmt, ...);
+int bdi_register_va(struct backing_dev_info *bdi, const char *fmt,
+ va_list args);
int bdi_register_owner(struct backing_dev_info *bdi, struct device *owner);
void bdi_unregister(struct backing_dev_info *bdi);
-int __must_check bdi_setup_and_register(struct backing_dev_info *, char *);
-void bdi_destroy(struct backing_dev_info *bdi);
struct backing_dev_info *bdi_alloc_node(gfp_t gfp_mask, int node_id);
+static inline struct backing_dev_info *bdi_alloc(gfp_t gfp_mask)
+{
+ return bdi_alloc_node(gfp_mask, NUMA_NO_NODE);
+}
void wb_start_writeback(struct bdi_writeback *wb, long nr_pages,
bool range_cyclic, enum wb_reason reason);
diff --git a/include/linux/bio.h b/include/linux/bio.h
index 8e52119..4931756 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -383,7 +383,7 @@
extern void bioset_free(struct bio_set *);
extern mempool_t *biovec_create_pool(int pool_entries);
-extern struct bio *bio_alloc_bioset(gfp_t, int, struct bio_set *);
+extern struct bio *bio_alloc_bioset(gfp_t, unsigned int, struct bio_set *);
extern void bio_put(struct bio *);
extern void __bio_clone_fast(struct bio *, struct bio *);
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index 9382c5d..f3e5e1d 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -15,7 +15,7 @@
unsigned long state; /* BLK_MQ_S_* flags */
} ____cacheline_aligned_in_smp;
- struct work_struct run_work;
+ struct delayed_work run_work;
cpumask_var_t cpumask;
int next_cpu;
int next_cpu_batch;
@@ -51,9 +51,6 @@
atomic_t nr_active;
- struct delayed_work delayed_run_work;
- struct delayed_work delay_work;
-
struct hlist_node cpuhp_dead;
struct kobject kobj;
@@ -82,7 +79,6 @@
struct blk_mq_queue_data {
struct request *rq;
- struct list_head *list;
bool last;
};
@@ -143,6 +139,14 @@
reinit_request_fn *reinit_request;
map_queues_fn *map_queues;
+
+#ifdef CONFIG_BLK_DEBUG_FS
+ /*
+ * Used by the debugfs implementation to show driver-specific
+ * information about a request.
+ */
+ void (*show_rq)(struct seq_file *m, struct request *rq);
+#endif
};
enum {
@@ -153,7 +157,6 @@
BLK_MQ_F_SHOULD_MERGE = 1 << 0,
BLK_MQ_F_TAG_SHARED = 1 << 1,
BLK_MQ_F_SG_MERGE = 1 << 2,
- BLK_MQ_F_DEFER_ISSUE = 1 << 4,
BLK_MQ_F_BLOCKING = 1 << 5,
BLK_MQ_F_NO_SCHED = 1 << 6,
BLK_MQ_F_ALLOC_POLICY_START_BIT = 8,
@@ -163,6 +166,7 @@
BLK_MQ_S_TAG_ACTIVE = 1,
BLK_MQ_S_SCHED_RESTART = 2,
BLK_MQ_S_TAG_WAITING = 3,
+ BLK_MQ_S_START_ON_RUN = 4,
BLK_MQ_MAX_DEPTH = 10240,
@@ -230,7 +234,7 @@
void blk_mq_kick_requeue_list(struct request_queue *q);
void blk_mq_delay_kick_requeue_list(struct request_queue *q, unsigned long msecs);
void blk_mq_abort_requeue_list(struct request_queue *q);
-void blk_mq_complete_request(struct request *rq, int error);
+void blk_mq_complete_request(struct request *rq);
bool blk_mq_queue_stopped(struct request_queue *q);
void blk_mq_stop_hw_queue(struct blk_mq_hw_ctx *hctx);
@@ -240,13 +244,14 @@
void blk_mq_start_stopped_hw_queue(struct blk_mq_hw_ctx *hctx, bool async);
void blk_mq_start_stopped_hw_queues(struct request_queue *q, bool async);
void blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs);
+void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async);
void blk_mq_run_hw_queues(struct request_queue *q, bool async);
void blk_mq_delay_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs);
void blk_mq_tagset_busy_iter(struct blk_mq_tag_set *tagset,
busy_tag_iter_fn *fn, void *priv);
void blk_mq_freeze_queue(struct request_queue *q);
void blk_mq_unfreeze_queue(struct request_queue *q);
-void blk_mq_freeze_queue_start(struct request_queue *q);
+void blk_freeze_queue_start(struct request_queue *q);
void blk_mq_freeze_queue_wait(struct request_queue *q);
int blk_mq_freeze_queue_wait_timeout(struct request_queue *q,
unsigned long timeout);
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index d703acb..61339bc 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -17,6 +17,10 @@
struct cgroup_subsys_state;
typedef void (bio_end_io_t) (struct bio *);
+struct blk_issue_stat {
+ u64 stat;
+};
+
/*
* main unit of I/O for the block layer and lower layers (ie drivers and
* stacking drivers)
@@ -29,7 +33,7 @@
* top bits REQ_OP. Use
* accessors.
*/
- unsigned short bi_flags; /* status, command, etc */
+ unsigned short bi_flags; /* status, etc and bvec pool number */
unsigned short bi_ioprio;
struct bvec_iter bi_iter;
@@ -58,6 +62,10 @@
*/
struct io_context *bi_ioc;
struct cgroup_subsys_state *bi_css;
+#ifdef CONFIG_BLK_DEV_THROTTLING_LOW
+ void *bi_cg_private;
+ struct blk_issue_stat bi_issue_stat;
+#endif
#endif
union {
#if defined(CONFIG_BLK_DEV_INTEGRITY)
@@ -102,12 +110,9 @@
#define BIO_REFFED 8 /* bio has elevated ->bi_cnt */
#define BIO_THROTTLED 9 /* This bio has already been subjected to
* throttling rules. Don't do it again. */
-
-/*
- * Flags starting here get preserved by bio_reset() - this includes
- * BVEC_POOL_IDX()
- */
-#define BIO_RESET_BITS 10
+#define BIO_TRACE_COMPLETION 10 /* bio_endio() should trace the final completion
+ * of this bio. */
+/* See BVEC_POOL_OFFSET below before adding new flags */
/*
* We support 6 different bvec pools, the last one is magic in that it
@@ -117,13 +122,22 @@
#define BVEC_POOL_MAX (BVEC_POOL_NR - 1)
/*
- * Top 4 bits of bio flags indicate the pool the bvecs came from. We add
+ * Top 3 bits of bio flags indicate the pool the bvecs came from. We add
* 1 to the actual index so that 0 indicates that there are no bvecs to be
* freed.
*/
-#define BVEC_POOL_BITS (4)
+#define BVEC_POOL_BITS (3)
#define BVEC_POOL_OFFSET (16 - BVEC_POOL_BITS)
#define BVEC_POOL_IDX(bio) ((bio)->bi_flags >> BVEC_POOL_OFFSET)
+#if (1<< BVEC_POOL_BITS) < (BVEC_POOL_NR+1)
+# error "BVEC_POOL_BITS is too small"
+#endif
+
+/*
+ * Flags starting here get preserved by bio_reset() - this includes
+ * only BVEC_POOL_IDX()
+ */
+#define BIO_RESET_BITS BVEC_POOL_OFFSET
/*
* Operations and flags common to the bio and request structures.
@@ -160,7 +174,7 @@
/* write the same sector many times */
REQ_OP_WRITE_SAME = 7,
/* write the zero filled sector many times */
- REQ_OP_WRITE_ZEROES = 8,
+ REQ_OP_WRITE_ZEROES = 9,
/* SCSI passthrough using struct scsi_request */
REQ_OP_SCSI_IN = 32,
@@ -187,6 +201,10 @@
__REQ_PREFLUSH, /* request for cache flush */
__REQ_RAHEAD, /* read ahead, can fail anytime */
__REQ_BACKGROUND, /* background IO */
+
+ /* command specific flags for REQ_OP_WRITE_ZEROES: */
+ __REQ_NOUNMAP, /* do not free blocks when zeroing */
+
__REQ_NR_BITS, /* stops here */
};
@@ -204,6 +222,8 @@
#define REQ_RAHEAD (1ULL << __REQ_RAHEAD)
#define REQ_BACKGROUND (1ULL << __REQ_BACKGROUND)
+#define REQ_NOUNMAP (1ULL << __REQ_NOUNMAP)
+
#define REQ_FAILFAST_MASK \
(REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER)
@@ -283,12 +303,6 @@
return (cookie & BLK_QC_T_INTERNAL) != 0;
}
-struct blk_issue_stat {
- u64 time;
-};
-
-#define BLK_RQ_STAT_BATCH 64
-
struct blk_rq_stat {
s64 mean;
u64 min;
@@ -296,7 +310,6 @@
s32 nr_samples;
s32 nr_batch;
u64 batch;
- s64 time;
};
#endif /* __LINUX_BLK_TYPES_H */
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 01a696b..83d2862 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -40,15 +40,20 @@
struct blk_flush_queue;
struct pr_ops;
struct rq_wb;
+struct blk_queue_stats;
+struct blk_stat_callback;
#define BLKDEV_MIN_RQ 4
#define BLKDEV_MAX_RQ 128 /* Default maximum */
+/* Must be consisitent with blk_mq_poll_stats_bkt() */
+#define BLK_MQ_POLL_STATS_BKTS 16
+
/*
* Maximum number of blkcg policies allowed to be registered concurrently.
* Defined here to simplify include dependency.
*/
-#define BLKCG_MAX_POLS 2
+#define BLKCG_MAX_POLS 3
typedef void (rq_end_io_fn)(struct request *, int);
@@ -173,6 +178,7 @@
struct rb_node rb_node; /* sort/lookup */
struct bio_vec special_vec;
void *completion_data;
+ int error_count; /* for legacy drivers, don't use */
};
/*
@@ -213,16 +219,14 @@
unsigned short ioprio;
- void *special; /* opaque pointer available for LLD use */
+ unsigned int timeout;
- int errors;
+ void *special; /* opaque pointer available for LLD use */
unsigned int extra_len; /* length of alignment and padding */
unsigned long deadline;
struct list_head timeout_list;
- unsigned int timeout;
- int retries;
/*
* completion callback.
@@ -337,7 +341,6 @@
unsigned char misaligned;
unsigned char discard_misaligned;
unsigned char cluster;
- unsigned char discard_zeroes_data;
unsigned char raid_partial_stripes_expensive;
enum blk_zoned_model zoned;
};
@@ -388,6 +391,7 @@
int nr_rqs[2]; /* # allocated [a]sync rqs */
int nr_rqs_elvpriv; /* # allocated rqs w/ elvpriv */
+ struct blk_queue_stats *stats;
struct rq_wb *rq_wb;
/*
@@ -505,8 +509,6 @@
unsigned int nr_sorted;
unsigned int in_flight[2];
- struct blk_rq_stat rq_stats[2];
-
/*
* Number of active block driver functions for which blk_drain_queue()
* must wait. Must be incremented around functions that unlock the
@@ -516,6 +518,10 @@
unsigned int rq_timeout;
int poll_nsec;
+
+ struct blk_stat_callback *poll_cb;
+ struct blk_rq_stat poll_stat[BLK_MQ_POLL_STATS_BKTS];
+
struct timer_list timeout;
struct work_struct timeout_work;
struct list_head timeout_list;
@@ -610,6 +616,8 @@
#define QUEUE_FLAG_FLUSH_NQ 25 /* flush not queueuable */
#define QUEUE_FLAG_DAX 26 /* device supports DAX */
#define QUEUE_FLAG_STATS 27 /* track rq completion times */
+#define QUEUE_FLAG_POLL_STATS 28 /* collecting stats for hybrid polling */
+#define QUEUE_FLAG_REGISTERED 29 /* queue has been registered to a disk */
#define QUEUE_FLAG_DEFAULT ((1 << QUEUE_FLAG_IO_STAT) | \
(1 << QUEUE_FLAG_STACKABLE) | \
@@ -918,6 +926,7 @@
extern void blk_unregister_queue(struct gendisk *disk);
extern blk_qc_t generic_make_request(struct bio *bio);
extern void blk_rq_init(struct request_queue *q, struct request *rq);
+extern void blk_init_request_from_bio(struct request *req, struct bio *bio);
extern void blk_put_request(struct request *);
extern void __blk_put_request(struct request_queue *, struct request *);
extern struct request *blk_get_request(struct request_queue *, int, gfp_t);
@@ -963,7 +972,7 @@
extern int blk_rq_map_user_iov(struct request_queue *, struct request *,
struct rq_map_data *, const struct iov_iter *,
gfp_t);
-extern int blk_execute_rq(struct request_queue *, struct gendisk *,
+extern void blk_execute_rq(struct request_queue *, struct gendisk *,
struct request *, int);
extern void blk_execute_rq_nowait(struct request_queue *, struct gendisk *,
struct request *, int, rq_end_io_fn *);
@@ -1081,20 +1090,6 @@
}
/*
- * blk_rq_set_prio - associate a request with prio from ioc
- * @rq: request of interest
- * @ioc: target iocontext
- *
- * Assocate request prio with ioc prio so request based drivers
- * can leverage priority information.
- */
-static inline void blk_rq_set_prio(struct request *rq, struct io_context *ioc)
-{
- if (ioc)
- rq->ioprio = ioc->ioprio;
-}
-
-/*
* Request issue related functions.
*/
extern struct request *blk_peek_request(struct request_queue *q);
@@ -1120,13 +1115,10 @@
extern bool blk_end_request(struct request *rq, int error,
unsigned int nr_bytes);
extern void blk_end_request_all(struct request *rq, int error);
-extern bool blk_end_request_cur(struct request *rq, int error);
-extern bool blk_end_request_err(struct request *rq, int error);
extern bool __blk_end_request(struct request *rq, int error,
unsigned int nr_bytes);
extern void __blk_end_request_all(struct request *rq, int error);
extern bool __blk_end_request_cur(struct request *rq, int error);
-extern bool __blk_end_request_err(struct request *rq, int error);
extern void blk_complete_request(struct request *);
extern void __blk_complete_request(struct request *);
@@ -1329,23 +1321,27 @@
return bqt->tag_index[tag];
}
+extern int blkdev_issue_flush(struct block_device *, gfp_t, sector_t *);
+extern int blkdev_issue_write_same(struct block_device *bdev, sector_t sector,
+ sector_t nr_sects, gfp_t gfp_mask, struct page *page);
#define BLKDEV_DISCARD_SECURE (1 << 0) /* issue a secure erase */
-#define BLKDEV_DISCARD_ZERO (1 << 1) /* must reliably zero data */
-extern int blkdev_issue_flush(struct block_device *, gfp_t, sector_t *);
extern int blkdev_issue_discard(struct block_device *bdev, sector_t sector,
sector_t nr_sects, gfp_t gfp_mask, unsigned long flags);
extern int __blkdev_issue_discard(struct block_device *bdev, sector_t sector,
sector_t nr_sects, gfp_t gfp_mask, int flags,
struct bio **biop);
-extern int blkdev_issue_write_same(struct block_device *bdev, sector_t sector,
- sector_t nr_sects, gfp_t gfp_mask, struct page *page);
+
+#define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */
+#define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */
+
extern int __blkdev_issue_zeroout(struct block_device *bdev, sector_t sector,
sector_t nr_sects, gfp_t gfp_mask, struct bio **biop,
- bool discard);
+ unsigned flags);
extern int blkdev_issue_zeroout(struct block_device *bdev, sector_t sector,
- sector_t nr_sects, gfp_t gfp_mask, bool discard);
+ sector_t nr_sects, gfp_t gfp_mask, unsigned flags);
+
static inline int sb_issue_discard(struct super_block *sb, sector_t block,
sector_t nr_blocks, gfp_t gfp_mask, unsigned long flags)
{
@@ -1359,7 +1355,7 @@
return blkdev_issue_zeroout(sb->s_bdev,
block << (sb->s_blocksize_bits - 9),
nr_blocks << (sb->s_blocksize_bits - 9),
- gfp_mask, true);
+ gfp_mask, 0);
}
extern int blk_verify_command(unsigned char *cmd, fmode_t has_write_perm);
@@ -1529,19 +1525,6 @@
return q->limits.discard_alignment;
}
-static inline unsigned int queue_discard_zeroes_data(struct request_queue *q)
-{
- if (q->limits.max_discard_sectors && q->limits.discard_zeroes_data == 1)
- return 1;
-
- return 0;
-}
-
-static inline unsigned int bdev_discard_zeroes_data(struct block_device *bdev)
-{
- return queue_discard_zeroes_data(bdev_get_queue(bdev));
-}
-
static inline unsigned int bdev_write_same(struct block_device *bdev)
{
struct request_queue *q = bdev_get_queue(bdev);
@@ -1726,6 +1709,7 @@
int kblockd_schedule_work_on(int cpu, struct work_struct *work);
int kblockd_schedule_delayed_work(struct delayed_work *dwork, unsigned long delay);
int kblockd_schedule_delayed_work_on(int cpu, struct delayed_work *dwork, unsigned long delay);
+int kblockd_mod_delayed_work_on(int cpu, struct delayed_work *dwork, unsigned long delay);
#ifdef CONFIG_BLK_CGROUP
/*
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 909fc03..6bb38d7 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -35,6 +35,7 @@
void *(*map_fd_get_ptr)(struct bpf_map *map, struct file *map_file,
int fd);
void (*map_fd_put_ptr)(void *ptr);
+ u32 (*map_gen_lookup)(struct bpf_map *map, struct bpf_insn *insn_buf);
};
struct bpf_map {
@@ -49,12 +50,7 @@
const struct bpf_map_ops *ops;
struct work_struct work;
atomic_t usercnt;
-};
-
-struct bpf_map_type_list {
- struct list_head list_node;
- const struct bpf_map_ops *ops;
- enum bpf_map_type type;
+ struct bpf_map *inner_map_meta;
};
/* function argument constraints */
@@ -167,12 +163,8 @@
const struct bpf_insn *src,
struct bpf_insn *dst,
struct bpf_prog *prog);
-};
-
-struct bpf_prog_type_list {
- struct list_head list_node;
- const struct bpf_verifier_ops *ops;
- enum bpf_prog_type type;
+ int (*test_run)(struct bpf_prog *prog, const union bpf_attr *kattr,
+ union bpf_attr __user *uattr);
};
struct bpf_prog_aux {
@@ -231,11 +223,21 @@
u64 bpf_event_output(struct bpf_map *map, u64 flags, void *meta, u64 meta_size,
void *ctx, u64 ctx_size, bpf_ctx_copy_t ctx_copy);
+int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,
+ union bpf_attr __user *uattr);
+int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr,
+ union bpf_attr __user *uattr);
+
#ifdef CONFIG_BPF_SYSCALL
DECLARE_PER_CPU(int, bpf_prog_active);
-void bpf_register_prog_type(struct bpf_prog_type_list *tl);
-void bpf_register_map_type(struct bpf_map_type_list *tl);
+#define BPF_PROG_TYPE(_id, _ops) \
+ extern const struct bpf_verifier_ops _ops;
+#define BPF_MAP_TYPE(_id, _ops) \
+ extern const struct bpf_map_ops _ops;
+#include <linux/bpf_types.h>
+#undef BPF_PROG_TYPE
+#undef BPF_MAP_TYPE
struct bpf_prog *bpf_prog_get(u32 ufd);
struct bpf_prog *bpf_prog_get_type(u32 ufd, enum bpf_prog_type type);
@@ -275,6 +277,8 @@
int bpf_fd_array_map_update_elem(struct bpf_map *map, struct file *map_file,
void *key, void *value, u64 map_flags);
void bpf_fd_array_map_clear(struct bpf_map *map);
+int bpf_fd_htab_map_update_elem(struct bpf_map *map, struct file *map_file,
+ void *key, void *value, u64 map_flags);
/* memcpy that is used with 8-byte aligned pointers, power-of-8 size and
* forced to use 'long' read/writes to try to atomically copy long counters.
@@ -295,10 +299,6 @@
/* verify correctness of eBPF program */
int bpf_check(struct bpf_prog **fp, union bpf_attr *attr);
#else
-static inline void bpf_register_prog_type(struct bpf_prog_type_list *tl)
-{
-}
-
static inline struct bpf_prog *bpf_prog_get(u32 ufd)
{
return ERR_PTR(-EOPNOTSUPP);
diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h
new file mode 100644
index 0000000..03bf223
--- /dev/null
+++ b/include/linux/bpf_types.h
@@ -0,0 +1,36 @@
+/* internal file - do not include directly */
+
+#ifdef CONFIG_NET
+BPF_PROG_TYPE(BPF_PROG_TYPE_SOCKET_FILTER, sk_filter_prog_ops)
+BPF_PROG_TYPE(BPF_PROG_TYPE_SCHED_CLS, tc_cls_act_prog_ops)
+BPF_PROG_TYPE(BPF_PROG_TYPE_SCHED_ACT, tc_cls_act_prog_ops)
+BPF_PROG_TYPE(BPF_PROG_TYPE_XDP, xdp_prog_ops)
+BPF_PROG_TYPE(BPF_PROG_TYPE_CGROUP_SKB, cg_skb_prog_ops)
+BPF_PROG_TYPE(BPF_PROG_TYPE_CGROUP_SOCK, cg_sock_prog_ops)
+BPF_PROG_TYPE(BPF_PROG_TYPE_LWT_IN, lwt_inout_prog_ops)
+BPF_PROG_TYPE(BPF_PROG_TYPE_LWT_OUT, lwt_inout_prog_ops)
+BPF_PROG_TYPE(BPF_PROG_TYPE_LWT_XMIT, lwt_xmit_prog_ops)
+#endif
+#ifdef CONFIG_BPF_EVENTS
+BPF_PROG_TYPE(BPF_PROG_TYPE_KPROBE, kprobe_prog_ops)
+BPF_PROG_TYPE(BPF_PROG_TYPE_TRACEPOINT, tracepoint_prog_ops)
+BPF_PROG_TYPE(BPF_PROG_TYPE_PERF_EVENT, perf_event_prog_ops)
+#endif
+
+BPF_MAP_TYPE(BPF_MAP_TYPE_ARRAY, array_map_ops)
+BPF_MAP_TYPE(BPF_MAP_TYPE_PERCPU_ARRAY, percpu_array_map_ops)
+BPF_MAP_TYPE(BPF_MAP_TYPE_PROG_ARRAY, prog_array_map_ops)
+BPF_MAP_TYPE(BPF_MAP_TYPE_PERF_EVENT_ARRAY, perf_event_array_map_ops)
+#ifdef CONFIG_CGROUPS
+BPF_MAP_TYPE(BPF_MAP_TYPE_CGROUP_ARRAY, cgroup_array_map_ops)
+#endif
+BPF_MAP_TYPE(BPF_MAP_TYPE_HASH, htab_map_ops)
+BPF_MAP_TYPE(BPF_MAP_TYPE_PERCPU_HASH, htab_percpu_map_ops)
+BPF_MAP_TYPE(BPF_MAP_TYPE_LRU_HASH, htab_lru_map_ops)
+BPF_MAP_TYPE(BPF_MAP_TYPE_LRU_PERCPU_HASH, htab_lru_percpu_map_ops)
+BPF_MAP_TYPE(BPF_MAP_TYPE_LPM_TRIE, trie_map_ops)
+#ifdef CONFIG_PERF_EVENTS
+BPF_MAP_TYPE(BPF_MAP_TYPE_STACK_TRACE, stack_map_ops)
+#endif
+BPF_MAP_TYPE(BPF_MAP_TYPE_ARRAY_OF_MAPS, array_of_maps_map_ops)
+BPF_MAP_TYPE(BPF_MAP_TYPE_HASH_OF_MAPS, htab_of_maps_map_ops)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index a13b031..5efb4db 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -66,7 +66,10 @@
};
struct bpf_insn_aux_data {
- enum bpf_reg_type ptr_type; /* pointer type for load/store insns */
+ union {
+ enum bpf_reg_type ptr_type; /* pointer type for load/store insns */
+ struct bpf_map *map_ptr; /* pointer for call insn into lookup_elem */
+ };
};
#define MAX_USED_MAPS 64 /* max number of maps accessed by one eBPF program */
diff --git a/include/linux/brcmphy.h b/include/linux/brcmphy.h
index 55e5171..abcda9b 100644
--- a/include/linux/brcmphy.h
+++ b/include/linux/brcmphy.h
@@ -25,6 +25,9 @@
#define PHY_ID_BCM57780 0x03625d90
#define PHY_ID_BCM7250 0xae025280
+#define PHY_ID_BCM7260 0xae025190
+#define PHY_ID_BCM7268 0xae025090
+#define PHY_ID_BCM7271 0xae0253b0
#define PHY_ID_BCM7278 0xae0251a0
#define PHY_ID_BCM7364 0xae025260
#define PHY_ID_BCM7366 0x600d8490
diff --git a/include/linux/bug.h b/include/linux/bug.h
index 5828489..687b557 100644
--- a/include/linux/bug.h
+++ b/include/linux/bug.h
@@ -105,7 +105,7 @@
return bug->flags & BUGFLAG_WARNING;
}
-const struct bug_entry *find_bug(unsigned long bugaddr);
+struct bug_entry *find_bug(unsigned long bugaddr);
enum bug_trap_type report_bug(unsigned long bug_addr, struct pt_regs *regs);
diff --git a/include/linux/can/core.h b/include/linux/can/core.h
index df08a41..c9a17bb 100644
--- a/include/linux/can/core.h
+++ b/include/linux/can/core.h
@@ -5,7 +5,7 @@
*
* Authors: Oliver Hartkopp <oliver.hartkopp@volkswagen.de>
* Urs Thuermann <urs.thuermann@volkswagen.de>
- * Copyright (c) 2002-2007 Volkswagen Group Electronic Research
+ * Copyright (c) 2002-2017 Volkswagen Group Electronic Research
* All rights reserved.
*
*/
@@ -17,7 +17,7 @@
#include <linux/skbuff.h>
#include <linux/netdevice.h>
-#define CAN_VERSION "20120528"
+#define CAN_VERSION "20170425"
/* increment this number each time you change some user-space interface */
#define CAN_ABI_VERSION "9"
@@ -45,12 +45,13 @@
extern int can_proto_register(const struct can_proto *cp);
extern void can_proto_unregister(const struct can_proto *cp);
-int can_rx_register(struct net_device *dev, canid_t can_id, canid_t mask,
+int can_rx_register(struct net *net, struct net_device *dev,
+ canid_t can_id, canid_t mask,
void (*func)(struct sk_buff *, void *),
void *data, char *ident, struct sock *sk);
-extern void can_rx_unregister(struct net_device *dev, canid_t can_id,
- canid_t mask,
+extern void can_rx_unregister(struct net *net, struct net_device *dev,
+ canid_t can_id, canid_t mask,
void (*func)(struct sk_buff *, void *),
void *data);
diff --git a/drivers/net/can/usb/peak_usb/pcan_ucan.h b/include/linux/can/dev/peak_canfd.h
similarity index 73%
rename from drivers/net/can/usb/peak_usb/pcan_ucan.h
rename to include/linux/can/dev/peak_canfd.h
index 2147678..46dceef 100644
--- a/drivers/net/can/usb/peak_usb/pcan_ucan.h
+++ b/include/linux/can/dev/peak_canfd.h
@@ -23,11 +23,14 @@
#define PUCAN_CMD_LISTEN_ONLY_MODE 0x003
#define PUCAN_CMD_TIMING_SLOW 0x004
#define PUCAN_CMD_TIMING_FAST 0x005
+#define PUCAN_CMD_SET_STD_FILTER 0x006
+#define PUCAN_CMD_RESERVED2 0x007
#define PUCAN_CMD_FILTER_STD 0x008
#define PUCAN_CMD_TX_ABORT 0x009
#define PUCAN_CMD_WR_ERR_CNT 0x00a
#define PUCAN_CMD_SET_EN_OPTION 0x00b
#define PUCAN_CMD_CLR_DIS_OPTION 0x00c
+#define PUCAN_CMD_RX_BARRIER 0x010
#define PUCAN_CMD_END_OF_COLLECTION 0x3ff
/* uCAN received messages list */
@@ -35,6 +38,10 @@
#define PUCAN_MSG_ERROR 0x0002
#define PUCAN_MSG_STATUS 0x0003
#define PUCAN_MSG_BUSLOAD 0x0004
+
+#define PUCAN_MSG_CACHE_CRITICAL 0x0102
+
+/* uCAN transmitted messages */
#define PUCAN_MSG_CAN_TX 0x1000
/* uCAN command common header */
@@ -43,6 +50,12 @@
u16 args[3];
};
+/* return the opcode from the opcode_channel field of a command */
+static inline u16 pucan_cmd_get_opcode(struct pucan_command *c)
+{
+ return le16_to_cpu(c->opcode_channel) & 0x3ff;
+}
+
#define PUCAN_TSLOW_BRP_BITS 10
#define PUCAN_TSLOW_TSGEG1_BITS 8
#define PUCAN_TSLOW_TSGEG2_BITS 7
@@ -108,6 +121,27 @@
__le32 mask; /* CAN-ID bitmask in idx range */
};
+#define PUCAN_FLTSTD_ROW_IDX_MAX ((1 << PUCAN_FLTSTD_ROW_IDX_BITS) - 1)
+
+/* uCAN SET_STD_FILTER command fields */
+struct __packed pucan_std_filter {
+ __le16 opcode_channel;
+
+ u8 unused;
+ u8 idx;
+ __le32 mask; /* CAN-ID bitmask in idx range */
+};
+
+/* uCAN TX_ABORT commands fields */
+#define PUCAN_TX_ABORT_FLUSH 0x0001
+
+struct __packed pucan_tx_abort {
+ __le16 opcode_channel;
+
+ __le16 flags;
+ u32 unused;
+};
+
/* uCAN WR_ERR_CNT command fields */
#define PUCAN_WRERRCNT_TE 0x4000 /* Tx error cntr write Enable */
#define PUCAN_WRERRCNT_RE 0x8000 /* Rx error cntr write Enable */
@@ -184,6 +218,12 @@
u8 rx_err_cnt;
};
+static inline int pucan_error_get_channel(const struct pucan_error_msg *msg)
+{
+ return msg->channel_type_d & 0x0f;
+}
+
+#define PUCAN_RX_BARRIER 0x10
#define PUCAN_BUS_PASSIVE 0x20
#define PUCAN_BUS_WARNING 0x40
#define PUCAN_BUS_BUSOFF 0x80
@@ -197,6 +237,31 @@
u8 unused[3];
};
+static inline int pucan_status_get_channel(const struct pucan_status_msg *msg)
+{
+ return msg->channel_p_w_b & 0x0f;
+}
+
+static inline int pucan_status_is_rx_barrier(const struct pucan_status_msg *msg)
+{
+ return msg->channel_p_w_b & PUCAN_RX_BARRIER;
+}
+
+static inline int pucan_status_is_passive(const struct pucan_status_msg *msg)
+{
+ return msg->channel_p_w_b & PUCAN_BUS_PASSIVE;
+}
+
+static inline int pucan_status_is_warning(const struct pucan_status_msg *msg)
+{
+ return msg->channel_p_w_b & PUCAN_BUS_WARNING;
+}
+
+static inline int pucan_status_is_busoff(const struct pucan_status_msg *msg)
+{
+ return msg->channel_p_w_b & PUCAN_BUS_BUSOFF;
+}
+
/* uCAN transmitted message format */
#define PUCAN_MSG_CHANNEL_DLC(c, d) (((c) & 0xf) | ((d) << 4))
@@ -213,32 +278,31 @@
};
/* build the cmd opcode_channel field with respect to the correct endianness */
-static inline __le16 pucan_cmd_opcode_channel(struct peak_usb_device *dev,
- int opcode)
+static inline __le16 pucan_cmd_opcode_channel(int index, int opcode)
{
- return cpu_to_le16(((dev->ctrl_idx) << 12) | ((opcode) & 0x3ff));
+ return cpu_to_le16(((index) << 12) | ((opcode) & 0x3ff));
}
/* return the channel number part from any received message channel_dlc field */
-static inline int pucan_msg_get_channel(struct pucan_rx_msg *rm)
+static inline int pucan_msg_get_channel(const struct pucan_rx_msg *msg)
{
- return rm->channel_dlc & 0xf;
+ return msg->channel_dlc & 0xf;
}
/* return the dlc value from any received message channel_dlc field */
-static inline int pucan_msg_get_dlc(struct pucan_rx_msg *rm)
+static inline int pucan_msg_get_dlc(const struct pucan_rx_msg *msg)
{
- return rm->channel_dlc >> 4;
+ return msg->channel_dlc >> 4;
}
-static inline int pucan_ermsg_get_channel(struct pucan_error_msg *em)
+static inline int pucan_ermsg_get_channel(const struct pucan_error_msg *msg)
{
- return em->channel_type_d & 0x0f;
+ return msg->channel_type_d & 0x0f;
}
-static inline int pucan_stmsg_get_channel(struct pucan_status_msg *sm)
+static inline int pucan_stmsg_get_channel(const struct pucan_status_msg *msg)
{
- return sm->channel_p_w_b & 0x0f;
+ return msg->channel_p_w_b & 0x0f;
}
#endif
diff --git a/include/linux/can/platform/ti_hecc.h b/include/linux/can/platform/ti_hecc.h
deleted file mode 100644
index a52f47c..0000000
--- a/include/linux/can/platform/ti_hecc.h
+++ /dev/null
@@ -1,44 +0,0 @@
-#ifndef _CAN_PLATFORM_TI_HECC_H
-#define _CAN_PLATFORM_TI_HECC_H
-
-/*
- * TI HECC (High End CAN Controller) driver platform header
- *
- * Copyright (C) 2009 Texas Instruments Incorporated - http://www.ti.com/
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation version 2.
- *
- * This program is distributed as is WITHOUT ANY WARRANTY of any
- * kind, whether express or implied; without even the implied warranty
- * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- */
-
-/**
- * struct hecc_platform_data - HECC Platform Data
- *
- * @scc_hecc_offset: mostly 0 - should really never change
- * @scc_ram_offset: SCC RAM offset
- * @hecc_ram_offset: HECC RAM offset
- * @mbx_offset: Mailbox RAM offset
- * @int_line: Interrupt line to use - 0 or 1
- * @version: version for future use
- * @transceiver_switch: platform specific callback fn for transceiver control
- *
- * Platform data structure to get all platform specific settings.
- * this structure also accounts the fact that the IP may have different
- * RAM and mailbox offsets for different SOC's
- */
-struct ti_hecc_platform_data {
- u32 scc_hecc_offset;
- u32 scc_ram_offset;
- u32 hecc_ram_offset;
- u32 mbx_offset;
- u32 int_line;
- u32 version;
- void (*transceiver_switch) (int);
-};
-#endif /* !_CAN_PLATFORM_TI_HECC_H */
diff --git a/include/linux/ccp.h b/include/linux/ccp.h
index c41b8d99..3285c94 100644
--- a/include/linux/ccp.h
+++ b/include/linux/ccp.h
@@ -123,6 +123,10 @@
CCP_AES_MODE_CFB,
CCP_AES_MODE_CTR,
CCP_AES_MODE_CMAC,
+ CCP_AES_MODE_GHASH,
+ CCP_AES_MODE_GCTR,
+ CCP_AES_MODE_GCM,
+ CCP_AES_MODE_GMAC,
CCP_AES_MODE__LAST,
};
@@ -137,6 +141,9 @@
CCP_AES_ACTION_ENCRYPT,
CCP_AES_ACTION__LAST,
};
+/* Overloaded field */
+#define CCP_AES_GHASHAAD CCP_AES_ACTION_DECRYPT
+#define CCP_AES_GHASHFINAL CCP_AES_ACTION_ENCRYPT
/**
* struct ccp_aes_engine - CCP AES operation
@@ -181,6 +188,8 @@
struct scatterlist *cmac_key; /* K1/K2 cmac key required for
* final cmac cmd */
u32 cmac_key_len; /* In bytes */
+
+ u32 aad_len; /* In bytes */
};
/***** XTS-AES engine *****/
@@ -249,6 +258,8 @@
CCP_SHA_TYPE_1 = 1,
CCP_SHA_TYPE_224,
CCP_SHA_TYPE_256,
+ CCP_SHA_TYPE_384,
+ CCP_SHA_TYPE_512,
CCP_SHA_TYPE__LAST,
};
@@ -290,6 +301,60 @@
* final sha cmd */
};
+/***** 3DES engine *****/
+enum ccp_des3_mode {
+ CCP_DES3_MODE_ECB = 0,
+ CCP_DES3_MODE_CBC,
+ CCP_DES3_MODE_CFB,
+ CCP_DES3_MODE__LAST,
+};
+
+enum ccp_des3_type {
+ CCP_DES3_TYPE_168 = 1,
+ CCP_DES3_TYPE__LAST,
+ };
+
+enum ccp_des3_action {
+ CCP_DES3_ACTION_DECRYPT = 0,
+ CCP_DES3_ACTION_ENCRYPT,
+ CCP_DES3_ACTION__LAST,
+};
+
+/**
+ * struct ccp_des3_engine - CCP SHA operation
+ * @type: Type of 3DES operation
+ * @mode: cipher mode
+ * @action: 3DES operation (decrypt/encrypt)
+ * @key: key to be used for this 3DES operation
+ * @key_len: length of key (in bytes)
+ * @iv: IV to be used for this AES operation
+ * @iv_len: length in bytes of iv
+ * @src: input data to be used for this operation
+ * @src_len: length of input data used for this operation (in bytes)
+ * @dst: output data produced by this operation
+ *
+ * Variables required to be set when calling ccp_enqueue_cmd():
+ * - type, mode, action, key, key_len, src, dst, src_len
+ * - iv, iv_len for any mode other than ECB
+ *
+ * The iv variable is used as both input and output. On completion of the
+ * 3DES operation the new IV overwrites the old IV.
+ */
+struct ccp_des3_engine {
+ enum ccp_des3_type type;
+ enum ccp_des3_mode mode;
+ enum ccp_des3_action action;
+
+ struct scatterlist *key;
+ u32 key_len; /* In bytes */
+
+ struct scatterlist *iv;
+ u32 iv_len; /* In bytes */
+
+ struct scatterlist *src, *dst;
+ u64 src_len; /* In bytes */
+};
+
/***** RSA engine *****/
/**
* struct ccp_rsa_engine - CCP RSA operation
@@ -539,7 +604,7 @@
enum ccp_engine {
CCP_ENGINE_AES = 0,
CCP_ENGINE_XTS_AES_128,
- CCP_ENGINE_RSVD1,
+ CCP_ENGINE_DES3,
CCP_ENGINE_SHA,
CCP_ENGINE_RSA,
CCP_ENGINE_PASSTHRU,
@@ -587,6 +652,7 @@
union {
struct ccp_aes_engine aes;
struct ccp_xts_aes_engine xts;
+ struct ccp_des3_engine des3;
struct ccp_sha_engine sha;
struct ccp_rsa_engine rsa;
struct ccp_passthru_engine passthru;
diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h
index 6a3f850..2174594 100644
--- a/include/linux/cgroup-defs.h
+++ b/include/linux/cgroup-defs.h
@@ -13,6 +13,7 @@
#include <linux/wait.h>
#include <linux/mutex.h>
#include <linux/rcupdate.h>
+#include <linux/refcount.h>
#include <linux/percpu-refcount.h>
#include <linux/percpu-rwsem.h>
#include <linux/workqueue.h>
@@ -106,9 +107,6 @@
/* reference count - access via css_[try]get() and css_put() */
struct percpu_ref refcnt;
- /* PI: the parent css */
- struct cgroup_subsys_state *parent;
-
/* siblings list anchored at the parent's ->children */
struct list_head sibling;
struct list_head children;
@@ -138,6 +136,12 @@
/* percpu_ref killing and RCU release */
struct rcu_head rcu_head;
struct work_struct destroy_work;
+
+ /*
+ * PI: the parent css. Placed here for cache proximity to following
+ * fields of the containing structure.
+ */
+ struct cgroup_subsys_state *parent;
};
/*
@@ -156,7 +160,7 @@
struct cgroup_subsys_state *subsys[CGROUP_SUBSYS_COUNT];
/* reference count */
- atomic_t refcount;
+ refcount_t refcount;
/* the default cgroup associated with this css_set */
struct cgroup *dfl_cgrp;
diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index af9c86e..ed2573e 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -17,11 +17,11 @@
#include <linux/seq_file.h>
#include <linux/kernfs.h>
#include <linux/jump_label.h>
-#include <linux/nsproxy.h>
#include <linux/types.h>
#include <linux/ns_common.h>
#include <linux/nsproxy.h>
#include <linux/user_namespace.h>
+#include <linux/refcount.h>
#include <linux/cgroup-defs.h>
@@ -661,7 +661,7 @@
#endif /* CONFIG_CGROUP_DATA */
struct cgroup_namespace {
- atomic_t count;
+ refcount_t count;
struct ns_common ns;
struct user_namespace *user_ns;
struct ucounts *ucounts;
@@ -696,12 +696,12 @@
static inline void get_cgroup_ns(struct cgroup_namespace *ns)
{
if (ns)
- atomic_inc(&ns->count);
+ refcount_inc(&ns->count);
}
static inline void put_cgroup_ns(struct cgroup_namespace *ns)
{
- if (ns && atomic_dec_and_test(&ns->count))
+ if (ns && refcount_dec_and_test(&ns->count))
free_cgroup_ns(ns);
}
diff --git a/include/linux/clk.h b/include/linux/clk.h
index e9d36b3..024cd07 100644
--- a/include/linux/clk.h
+++ b/include/linux/clk.h
@@ -132,8 +132,8 @@
* @q: clk compared against p
*
* Returns true if the two struct clk pointers both point to the same hardware
- * clock node. Put differently, returns true if struct clk *p and struct clk *q
- * share the same struct clk_core object.
+ * clock node. Put differently, returns true if @p and @q
+ * share the same &struct clk_core object.
*
* Returns false otherwise. Note that two NULL clks are treated as matching.
*/
diff --git a/include/linux/clockchips.h b/include/linux/clockchips.h
index 6d7edc3..acc9ce0 100644
--- a/include/linux/clockchips.h
+++ b/include/linux/clockchips.h
@@ -182,7 +182,6 @@
extern void clockevents_register_device(struct clock_event_device *dev);
extern int clockevents_unbind_device(struct clock_event_device *ced, int cpu);
-extern void clockevents_config(struct clock_event_device *dev, u32 freq);
extern void clockevents_config_and_register(struct clock_event_device *dev,
u32 freq, unsigned long min_delta,
unsigned long max_delta);
diff --git a/include/linux/clocksource.h b/include/linux/clocksource.h
index cfc7584..f2b10d9 100644
--- a/include/linux/clocksource.h
+++ b/include/linux/clocksource.h
@@ -120,7 +120,7 @@
#define CLOCK_SOURCE_RESELECT 0x100
/* simplify initialization of mask field */
-#define CLOCKSOURCE_MASK(bits) (u64)((bits) < 64 ? ((1ULL<<(bits))-1) : -1)
+#define CLOCKSOURCE_MASK(bits) GENMASK_ULL((bits) - 1, 0)
static inline u32 clocksource_freq2mult(u32 freq, u32 shift_constant, u64 from)
{
diff --git a/include/linux/coda_psdev.h b/include/linux/coda_psdev.h
index 5b8721e..31e4e1f 100644
--- a/include/linux/coda_psdev.h
+++ b/include/linux/coda_psdev.h
@@ -15,7 +15,6 @@
struct list_head vc_processing;
int vc_inuse;
struct super_block *vc_sb;
- struct backing_dev_info bdi;
struct mutex vc_mutex;
};
diff --git a/include/linux/compat.h b/include/linux/compat.h
index aef47be..8172b03 100644
--- a/include/linux/compat.h
+++ b/include/linux/compat.h
@@ -528,11 +528,6 @@
asmlinkage long compat_sys_getdents(unsigned int fd,
struct compat_linux_dirent __user *dirent,
unsigned int count);
-#ifdef __ARCH_WANT_COMPAT_SYS_GETDENTS64
-asmlinkage long compat_sys_getdents64(unsigned int fd,
- struct linux_dirent64 __user *dirent,
- unsigned int count);
-#endif
asmlinkage long compat_sys_vmsplice(int fd, const struct compat_iovec __user *,
unsigned int nr_segs, unsigned int flags);
asmlinkage long compat_sys_open(const char __user *filename, int flags,
@@ -723,6 +718,8 @@
asmlinkage long compat_sys_fanotify_mark(int, unsigned int, __u32, __u32,
int, const char __user *);
+asmlinkage long compat_sys_arch_prctl(int option, unsigned long arg2);
+
/*
* For most but not all architectures, "am I in a compat syscall?" and
* "am I a compat task?" are the same question. For architectures on which
diff --git a/include/linux/coresight.h b/include/linux/coresight.h
index 2a5982c..035c16c 100644
--- a/include/linux/coresight.h
+++ b/include/linux/coresight.h
@@ -201,7 +201,7 @@
void *sink_config);
unsigned long (*reset_buffer)(struct coresight_device *csdev,
struct perf_output_handle *handle,
- void *sink_config, bool *lost);
+ void *sink_config);
void (*update_buffer)(struct coresight_device *csdev,
struct perf_output_handle *handle,
void *sink_config);
diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h
index 87165f0..a5ce0bbe 100644
--- a/include/linux/cpufreq.h
+++ b/include/linux/cpufreq.h
@@ -120,6 +120,13 @@
bool fast_switch_possible;
bool fast_switch_enabled;
+ /*
+ * Preferred average time interval between consecutive invocations of
+ * the driver to set the frequency for this policy. To be set by the
+ * scaling driver (0, which is the default, means no preference).
+ */
+ unsigned int transition_delay_us;
+
/* Cached frequency lookup from cpufreq_driver_resolve_freq. */
unsigned int cached_target_freq;
int cached_resolved_idx;
diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h
index 96f1e88..1a67560 100644
--- a/include/linux/cpumask.h
+++ b/include/linux/cpumask.h
@@ -667,6 +667,11 @@
void free_cpumask_var(cpumask_var_t mask);
void free_bootmem_cpumask_var(cpumask_var_t mask);
+static inline bool cpumask_available(cpumask_var_t mask)
+{
+ return mask != NULL;
+}
+
#else
typedef struct cpumask cpumask_var_t[1];
@@ -708,6 +713,11 @@
static inline void free_bootmem_cpumask_var(cpumask_var_t mask)
{
}
+
+static inline bool cpumask_available(cpumask_var_t mask)
+{
+ return true;
+}
#endif /* CONFIG_CPUMASK_OFFSTACK */
/* It's common to want to use cpu_all_mask in struct member initializers,
diff --git a/include/linux/cpuset.h b/include/linux/cpuset.h
index 611fce5..119a3f9 100644
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -42,7 +42,7 @@
extern int cpuset_init(void);
extern void cpuset_init_smp(void);
-extern void cpuset_update_active_cpus(bool cpu_online);
+extern void cpuset_update_active_cpus(void);
extern void cpuset_cpus_allowed(struct task_struct *p, struct cpumask *mask);
extern void cpuset_cpus_allowed_fallback(struct task_struct *p);
extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
@@ -155,7 +155,7 @@
static inline int cpuset_init(void) { return 0; }
static inline void cpuset_init_smp(void) {}
-static inline void cpuset_update_active_cpus(bool cpu_online)
+static inline void cpuset_update_active_cpus(void)
{
partition_sched_domains(1, NULL, NULL);
}
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index c0b0cf3..84da997 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -123,7 +123,7 @@
/*
* Miscellaneous stuff.
*/
-#define CRYPTO_MAX_ALG_NAME 64
+#define CRYPTO_MAX_ALG_NAME 128
/*
* The macro CRYPTO_MINALIGN_ATTR (along with the void * type in the actual
diff --git a/include/linux/cryptohash.h b/include/linux/cryptohash.h
index 3252799..df4d3e9 100644
--- a/include/linux/cryptohash.h
+++ b/include/linux/cryptohash.h
@@ -10,9 +10,4 @@
void sha_init(__u32 *buf);
void sha_transform(__u32 *digest, const char *data, __u32 *W);
-#define MD5_DIGEST_WORDS 4
-#define MD5_MESSAGE_BYTES 64
-
-void md5_transform(__u32 *hash, __u32 const *in);
-
#endif
diff --git a/include/linux/dell-led.h b/include/linux/dell-led.h
index 7009b8b..3f033c4 100644
--- a/include/linux/dell-led.h
+++ b/include/linux/dell-led.h
@@ -1,10 +1,6 @@
#ifndef __DELL_LED_H__
#define __DELL_LED_H__
-enum {
- DELL_LED_MICMUTE,
-};
-
-int dell_app_wmi_led_set(int whichled, int on);
+int dell_micmute_led_set(int on);
#endif
diff --git a/include/linux/devfreq.h b/include/linux/devfreq.h
index e0acb0e..6c220e4 100644
--- a/include/linux/devfreq.h
+++ b/include/linux/devfreq.h
@@ -27,6 +27,7 @@
#define DEVFREQ_POSTCHANGE (1)
struct devfreq;
+struct devfreq_governor;
/**
* struct devfreq_dev_status - Data given from devfreq user device to
@@ -101,35 +102,6 @@
};
/**
- * struct devfreq_governor - Devfreq policy governor
- * @node: list node - contains registered devfreq governors
- * @name: Governor's name
- * @immutable: Immutable flag for governor. If the value is 1,
- * this govenror is never changeable to other governor.
- * @get_target_freq: Returns desired operating frequency for the device.
- * Basically, get_target_freq will run
- * devfreq_dev_profile.get_dev_status() to get the
- * status of the device (load = busy_time / total_time).
- * If no_central_polling is set, this callback is called
- * only with update_devfreq() notified by OPP.
- * @event_handler: Callback for devfreq core framework to notify events
- * to governors. Events include per device governor
- * init and exit, opp changes out of devfreq, suspend
- * and resume of per device devfreq during device idle.
- *
- * Note that the callbacks are called with devfreq->lock locked by devfreq.
- */
-struct devfreq_governor {
- struct list_head node;
-
- const char name[DEVFREQ_NAME_LEN];
- const unsigned int immutable;
- int (*get_target_freq)(struct devfreq *this, unsigned long *freq);
- int (*event_handler)(struct devfreq *devfreq,
- unsigned int event, void *data);
-};
-
-/**
* struct devfreq - Device devfreq structure
* @node: list node - contains the devices with devfreq that have been
* registered.
diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h
index a7e6903..c7ea33e 100644
--- a/include/linux/device-mapper.h
+++ b/include/linux/device-mapper.h
@@ -255,6 +255,12 @@
unsigned num_write_same_bios;
/*
+ * The number of WRITE ZEROES bios that will be submitted to the target.
+ * The bio number can be accessed with dm_bio_get_target_bio_nr.
+ */
+ unsigned num_write_zeroes_bios;
+
+ /*
* The minimum number of extra bytes allocated in each io for the
* target to use.
*/
@@ -290,11 +296,6 @@
* on max_io_len boundary.
*/
bool split_discard_bios:1;
-
- /*
- * Set if this target does not return zeroes on discarded blocks.
- */
- bool discard_zeroes_data_unsupported:1;
};
/* Each target can link one of these into the table */
diff --git a/include/linux/edac.h b/include/linux/edac.h
index 5b6adf9..8ae0f45 100644
--- a/include/linux/edac.h
+++ b/include/linux/edac.h
@@ -28,12 +28,10 @@
#define EDAC_OPSTATE_INT 2
extern int edac_op_state;
-extern int edac_err_assert;
-extern atomic_t edac_handlers;
-extern int edac_handler_set(void);
-extern void edac_atomic_assert_error(void);
-extern struct bus_type *edac_get_sysfs_subsys(void);
+struct bus_type *edac_get_sysfs_subsys(void);
+int edac_get_report_status(void);
+void edac_set_report_status(int new);
enum {
EDAC_REPORTING_ENABLED,
@@ -41,28 +39,6 @@
EDAC_REPORTING_FORCE
};
-extern int edac_report_status;
-#ifdef CONFIG_EDAC
-static inline int get_edac_report_status(void)
-{
- return edac_report_status;
-}
-
-static inline void set_edac_report_status(int new)
-{
- edac_report_status = new;
-}
-#else
-static inline int get_edac_report_status(void)
-{
- return EDAC_REPORTING_DISABLED;
-}
-
-static inline void set_edac_report_status(int new)
-{
-}
-#endif
-
static inline void opstate_init(void)
{
switch (edac_op_state) {
diff --git a/include/linux/efi-bgrt.h b/include/linux/efi-bgrt.h
index 2fd3993..e6f624b 100644
--- a/include/linux/efi-bgrt.h
+++ b/include/linux/efi-bgrt.h
@@ -6,6 +6,7 @@
#ifdef CONFIG_ACPI_BGRT
void efi_bgrt_init(struct acpi_table_header *table);
+int __init acpi_parse_bgrt(struct acpi_table_header *table);
/* The BGRT data itself; only valid if bgrt_image != NULL. */
extern size_t bgrt_image_size;
@@ -14,6 +15,10 @@
#else /* !CONFIG_ACPI_BGRT */
static inline void efi_bgrt_init(struct acpi_table_header *table) {}
+static inline int __init acpi_parse_bgrt(struct acpi_table_header *table)
+{
+ return 0;
+}
#endif /* !CONFIG_ACPI_BGRT */
diff --git a/include/linux/efi.h b/include/linux/efi.h
index 94d34e0..ec36f42 100644
--- a/include/linux/efi.h
+++ b/include/linux/efi.h
@@ -1435,9 +1435,6 @@
/* prototypes shared between arch specific and generic stub code */
-#define pr_efi(sys_table, msg) efi_printk(sys_table, "EFI stub: "msg)
-#define pr_efi_err(sys_table, msg) efi_printk(sys_table, "EFI stub: ERROR: "msg)
-
void efi_printk(efi_system_table_t *sys_table_arg, char *str);
void efi_free(efi_system_table_t *sys_table_arg, unsigned long size,
@@ -1471,7 +1468,7 @@
unsigned long *load_addr,
unsigned long *load_size);
-efi_status_t efi_parse_options(char *cmdline);
+efi_status_t efi_parse_options(char const *cmdline);
efi_status_t efi_setup_gop(efi_system_table_t *sys_table_arg,
struct screen_info *si, efi_guid_t *proto,
diff --git a/include/linux/elevator.h b/include/linux/elevator.h
index 22d39e8..3a21631 100644
--- a/include/linux/elevator.h
+++ b/include/linux/elevator.h
@@ -93,6 +93,8 @@
struct elevator_mq_ops {
int (*init_sched)(struct request_queue *, struct elevator_type *);
void (*exit_sched)(struct elevator_queue *);
+ int (*init_hctx)(struct blk_mq_hw_ctx *, unsigned int);
+ void (*exit_hctx)(struct blk_mq_hw_ctx *, unsigned int);
bool (*allow_merge)(struct request_queue *, struct request *, struct bio *);
bool (*bio_merge)(struct blk_mq_hw_ctx *, struct bio *);
@@ -104,7 +106,7 @@
void (*insert_requests)(struct blk_mq_hw_ctx *, struct list_head *, bool);
struct request *(*dispatch_request)(struct blk_mq_hw_ctx *);
bool (*has_work)(struct blk_mq_hw_ctx *);
- void (*completed_request)(struct blk_mq_hw_ctx *, struct request *);
+ void (*completed_request)(struct request *);
void (*started_request)(struct request *);
void (*requeue_request)(struct request *);
struct request *(*former_request)(struct request_queue *, struct request *);
diff --git a/include/linux/etherdevice.h b/include/linux/etherdevice.h
index c62b709..2d9f808 100644
--- a/include/linux/etherdevice.h
+++ b/include/linux/etherdevice.h
@@ -447,21 +447,6 @@
}
/**
- * ether_addr_greater - Compare two Ethernet addresses
- * @addr1: Pointer to a six-byte array containing the Ethernet address
- * @addr2: Pointer other six-byte array containing the Ethernet address
- *
- * Compare two Ethernet addresses, returns true addr1 is greater than addr2
- */
-static inline bool ether_addr_greater(const u8 *addr1, const u8 *addr2)
-{
- u64 u1 = ether_addr_to_u64(addr1);
- u64 u2 = ether_addr_to_u64(addr2);
-
- return u1 > u2;
-}
-
-/**
* is_etherdev_addr - Tell if given Ethernet address belongs to the device.
* @dev: Pointer to a device structure
* @addr: Pointer to a six-byte array containing the Ethernet address
diff --git a/include/linux/ethtool.h b/include/linux/ethtool.h
index 9ded8c6..83cc986 100644
--- a/include/linux/ethtool.h
+++ b/include/linux/ethtool.h
@@ -60,6 +60,7 @@
enum {
ETH_RSS_HASH_TOP_BIT, /* Configurable RSS hash function - Toeplitz */
ETH_RSS_HASH_XOR_BIT, /* Configurable RSS hash function - Xor */
+ ETH_RSS_HASH_CRC32_BIT, /* Configurable RSS hash function - Crc32 */
/*
* Add your fresh new hash function bits above and remember to update
@@ -73,6 +74,7 @@
#define ETH_RSS_HASH_TOP __ETH_RSS_HASH(TOP)
#define ETH_RSS_HASH_XOR __ETH_RSS_HASH(XOR)
+#define ETH_RSS_HASH_CRC32 __ETH_RSS_HASH(CRC32)
#define ETH_RSS_HASH_UNKNOWN 0
#define ETH_RSS_HASH_NO_CHANGE 0
diff --git a/include/linux/extcon.h b/include/linux/extcon.h
index 7010fb0..7e206a9 100644
--- a/include/linux/extcon.h
+++ b/include/linux/extcon.h
@@ -236,11 +236,11 @@
unsigned int id, unsigned int prop);
/*
- * Following APIs are to monitor every action of a notifier.
- * Registrar gets notified for every external port of a connection device.
- * Probably this could be used to debug an action of notifier; however,
- * we do not recommend to use this for normal 'notifiee' device drivers who
- * want to be notified by a specific external port of the notifier.
+ * Following APIs are to monitor the status change of the external connectors.
+ * extcon_register_notifier(*edev, id, *nb) : Register a notifier block
+ * for specific external connector of the extcon.
+ * extcon_register_notifier_all(*edev, *nb) : Register a notifier block
+ * for all supported external connectors of the extcon.
*/
extern int extcon_register_notifier(struct extcon_dev *edev, unsigned int id,
struct notifier_block *nb);
@@ -253,6 +253,17 @@
struct extcon_dev *edev, unsigned int id,
struct notifier_block *nb);
+extern int extcon_register_notifier_all(struct extcon_dev *edev,
+ struct notifier_block *nb);
+extern int extcon_unregister_notifier_all(struct extcon_dev *edev,
+ struct notifier_block *nb);
+extern int devm_extcon_register_notifier_all(struct device *dev,
+ struct extcon_dev *edev,
+ struct notifier_block *nb);
+extern void devm_extcon_unregister_notifier_all(struct device *dev,
+ struct extcon_dev *edev,
+ struct notifier_block *nb);
+
/*
* Following API get the extcon device from devicetree.
* This function use phandle of devicetree to get extcon device directly.
diff --git a/include/linux/filter.h b/include/linux/filter.h
index fbf7b39..9a7786d 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -7,6 +7,7 @@
#include <stdarg.h>
#include <linux/atomic.h>
+#include <linux/refcount.h>
#include <linux/compat.h>
#include <linux/skbuff.h>
#include <linux/linkage.h>
@@ -412,8 +413,7 @@
locked:1, /* Program image locked? */
gpl_compatible:1, /* Is filter GPL compatible? */
cb_access:1, /* Is control block accessed? */
- dst_needed:1, /* Do we need dst entry? */
- xdp_adjust_head:1; /* Adjusting pkt head? */
+ dst_needed:1; /* Do we need dst entry? */
kmemcheck_bitfield_end(meta);
enum bpf_prog_type type; /* Type of BPF program */
u32 len; /* Number of filter blocks */
@@ -430,7 +430,7 @@
};
struct sk_filter {
- atomic_t refcnt;
+ refcount_t refcnt;
struct rcu_head rcu;
struct bpf_prog *prog;
};
@@ -693,6 +693,11 @@
# endif
}
+static inline bool ebpf_jit_enabled(void)
+{
+ return bpf_jit_enable && bpf_jit_is_ebpf();
+}
+
static inline bool bpf_prog_ebpf_jited(const struct bpf_prog *fp)
{
return fp->jited && bpf_jit_is_ebpf();
@@ -753,6 +758,11 @@
#else /* CONFIG_BPF_JIT */
+static inline bool ebpf_jit_enabled(void)
+{
+ return false;
+}
+
static inline bool bpf_prog_ebpf_jited(const struct bpf_prog *fp)
{
return false;
diff --git a/include/linux/flex_array.h b/include/linux/flex_array.h
index b6efb0c..11366b3 100644
--- a/include/linux/flex_array.h
+++ b/include/linux/flex_array.h
@@ -61,16 +61,83 @@
FLEX_ARRAY_ELEMENTS_PER_PART(__element_size)); \
}
+/**
+ * flex_array_alloc() - Creates a flexible array.
+ * @element_size: individual object size.
+ * @total: maximum number of objects which can be stored.
+ * @flags: GFP flags
+ *
+ * Return: Returns an object of structure flex_array.
+ */
struct flex_array *flex_array_alloc(int element_size, unsigned int total,
gfp_t flags);
+
+/**
+ * flex_array_prealloc() - Ensures that memory for the elements indexed in the
+ * range defined by start and nr_elements has been allocated.
+ * @fa: array to allocate memory to.
+ * @start: start address
+ * @nr_elements: number of elements to be allocated.
+ * @flags: GFP flags
+ *
+ */
int flex_array_prealloc(struct flex_array *fa, unsigned int start,
unsigned int nr_elements, gfp_t flags);
+
+/**
+ * flex_array_free() - Removes all elements of a flexible array.
+ * @fa: array to be freed.
+ */
void flex_array_free(struct flex_array *fa);
+
+/**
+ * flex_array_free_parts() - Removes all elements of a flexible array, but
+ * leaves the array itself in place.
+ * @fa: array to be emptied.
+ */
void flex_array_free_parts(struct flex_array *fa);
+
+/**
+ * flex_array_put() - Stores data into a flexible array.
+ * @fa: array where element is to be stored.
+ * @element_nr: position to copy, must be less than the maximum specified when
+ * the array was created.
+ * @src: data source to be copied into the array.
+ * @flags: GFP flags
+ *
+ * Return: Returns zero on success, a negative error code otherwise.
+ */
int flex_array_put(struct flex_array *fa, unsigned int element_nr, void *src,
gfp_t flags);
+
+/**
+ * flex_array_clear() - Clears an individual element in the array, sets the
+ * given element to FLEX_ARRAY_FREE.
+ * @element_nr: element position to clear.
+ * @fa: array to which element to be cleared belongs.
+ *
+ * Return: Returns zero on success, -EINVAL otherwise.
+ */
int flex_array_clear(struct flex_array *fa, unsigned int element_nr);
+
+/**
+ * flex_array_get() - Retrieves data into a flexible array.
+ *
+ * @element_nr: Element position to retrieve data from.
+ * @fa: array from which data is to be retrieved.
+ *
+ * Return: Returns a pointer to the data element, or NULL if that
+ * particular element has never been allocated.
+ */
void *flex_array_get(struct flex_array *fa, unsigned int element_nr);
+
+/**
+ * flex_array_shrink() - Reduces the allocated size of an array.
+ * @fa: array to shrink.
+ *
+ * Return: Returns number of pages of memory actually freed.
+ *
+ */
int flex_array_shrink(struct flex_array *fa);
#define flex_array_put_ptr(fa, nr, src, gfp) \
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 7251f7b..30e5c14 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -2121,6 +2121,9 @@
extern int freeze_super(struct super_block *super);
extern int thaw_super(struct super_block *super);
extern bool our_mnt(struct vfsmount *mnt);
+extern __printf(2, 3)
+int super_setup_bdi_name(struct super_block *sb, char *fmt, ...);
+extern int super_setup_bdi(struct super_block *sb);
extern int current_umask(void);
diff --git a/include/linux/fwnode.h b/include/linux/fwnode.h
index 8bd28ce..3dff239 100644
--- a/include/linux/fwnode.h
+++ b/include/linux/fwnode.h
@@ -27,4 +27,16 @@
struct fwnode_handle *secondary;
};
+/**
+ * struct fwnode_endpoint - Fwnode graph endpoint
+ * @port: Port number
+ * @id: Endpoint id
+ * @local_fwnode: reference to the related fwnode
+ */
+struct fwnode_endpoint {
+ unsigned int port;
+ unsigned int id;
+ const struct fwnode_handle *local_fwnode;
+};
+
#endif
diff --git a/include/linux/genhd.h b/include/linux/genhd.h
index 76f3975..acff943 100644
--- a/include/linux/genhd.h
+++ b/include/linux/genhd.h
@@ -159,11 +159,11 @@
#if defined(CONFIG_BLK_DEV_INTEGRITY)
struct blk_integrity {
- struct blk_integrity_profile *profile;
- unsigned char flags;
- unsigned char tuple_size;
- unsigned char interval_exp;
- unsigned char tag_size;
+ const struct blk_integrity_profile *profile;
+ unsigned char flags;
+ unsigned char tuple_size;
+ unsigned char interval_exp;
+ unsigned char tag_size;
};
#endif /* CONFIG_BLK_DEV_INTEGRITY */
@@ -722,11 +722,9 @@
#if defined(CONFIG_BLK_DEV_INTEGRITY)
extern void blk_integrity_add(struct gendisk *);
extern void blk_integrity_del(struct gendisk *);
-extern void blk_integrity_revalidate(struct gendisk *);
#else /* CONFIG_BLK_DEV_INTEGRITY */
static inline void blk_integrity_add(struct gendisk *disk) { }
static inline void blk_integrity_del(struct gendisk *disk) { }
-static inline void blk_integrity_revalidate(struct gendisk *disk) { }
#endif /* CONFIG_BLK_DEV_INTEGRITY */
#else /* CONFIG_BLOCK */
diff --git a/include/linux/hid.h b/include/linux/hid.h
index 28f38e2b8..5be325d 100644
--- a/include/linux/hid.h
+++ b/include/linux/hid.h
@@ -268,6 +268,8 @@
#define HID_CP_APPLICATIONLAUNCHBUTTONS 0x000c0180
#define HID_CP_GENERICGUIAPPLICATIONCONTROLS 0x000c0200
+#define HID_DG_DEVICECONFIG 0x000d000e
+#define HID_DG_DEVICESETTINGS 0x000d0023
#define HID_DG_CONFIDENCE 0x000d0047
#define HID_DG_WIDTH 0x000d0048
#define HID_DG_HEIGHT 0x000d0049
@@ -322,7 +324,7 @@
#define HID_QUIRK_MULTI_INPUT 0x00000040
#define HID_QUIRK_HIDINPUT_FORCE 0x00000080
#define HID_QUIRK_NO_EMPTY_INPUT 0x00000100
-#define HID_QUIRK_NO_INIT_INPUT_REPORTS 0x00000200
+/* 0x00000200 reserved for backward compatibility, was NO_INIT_INPUT_REPORTS */
#define HID_QUIRK_ALWAYS_POLL 0x00000400
#define HID_QUIRK_SKIP_OUTPUT_REPORTS 0x00010000
#define HID_QUIRK_SKIP_OUTPUT_REPORT_ID 0x00020000
@@ -541,7 +543,6 @@
struct list_head inputs; /* The list of inputs */
void *hiddev; /* The hiddev structure */
void *hidraw;
- int minor; /* Hiddev minor number */
int open; /* is the device open by anyone? */
char name[128]; /* Device name */
diff --git a/include/linux/hiddev.h b/include/linux/hiddev.h
index a5dd814..9216222 100644
--- a/include/linux/hiddev.h
+++ b/include/linux/hiddev.h
@@ -32,6 +32,18 @@
* In-kernel definitions.
*/
+struct hiddev {
+ int minor;
+ int exist;
+ int open;
+ struct mutex existancelock;
+ wait_queue_head_t wait;
+ struct hid_device *hid;
+ struct list_head list;
+ spinlock_t list_lock;
+ bool initialized;
+};
+
struct hid_device;
struct hid_usage;
struct hid_field;
diff --git a/include/linux/hrtimer.h b/include/linux/hrtimer.h
index 249e579..8c5b10e 100644
--- a/include/linux/hrtimer.h
+++ b/include/linux/hrtimer.h
@@ -276,8 +276,6 @@
return timer->base->cpu_base->hres_active;
}
-extern void hrtimer_peek_ahead_timers(void);
-
/*
* The resolution of the clocks. The resolution value is returned in
* the clock_getres() system call to give application programmers an
@@ -300,8 +298,6 @@
#define hrtimer_resolution (unsigned int)LOW_RES_NSEC
-static inline void hrtimer_peek_ahead_timers(void) { }
-
static inline int hrtimer_is_hres_active(struct hrtimer *timer)
{
return 0;
@@ -456,7 +452,7 @@
}
/* Precise sleep: */
-extern long hrtimer_nanosleep(struct timespec *rqtp,
+extern long hrtimer_nanosleep(struct timespec64 *rqtp,
struct timespec __user *rmtp,
const enum hrtimer_mode mode,
const clockid_t clockid);
diff --git a/include/linux/hwmon.h b/include/linux/hwmon.h
index 88b6737..ceb7519 100644
--- a/include/linux/hwmon.h
+++ b/include/linux/hwmon.h
@@ -337,7 +337,7 @@
int (*read)(struct device *dev, enum hwmon_sensor_types type,
u32 attr, int channel, long *val);
int (*read_string)(struct device *dev, enum hwmon_sensor_types type,
- u32 attr, int channel, char **str);
+ u32 attr, int channel, const char **str);
int (*write)(struct device *dev, enum hwmon_sensor_types type,
u32 attr, int channel, long val);
};
diff --git a/include/linux/hyperv.h b/include/linux/hyperv.h
index 970771a..0c170a3 100644
--- a/include/linux/hyperv.h
+++ b/include/linux/hyperv.h
@@ -1508,14 +1508,6 @@
return;
}
-static inline void
-init_cached_read_index(struct vmbus_channel *channel)
-{
- struct hv_ring_buffer_info *rbi = &channel->inbound;
-
- rbi->cached_read_index = rbi->ring_buffer->read_index;
-}
-
/*
* Mask off host interrupt callback notifications
*/
@@ -1549,76 +1541,48 @@
/*
* An API to support in-place processing of incoming VMBUS packets.
*/
-#define VMBUS_PKT_TRAILER 8
+/* Get data payload associated with descriptor */
+static inline void *hv_pkt_data(const struct vmpacket_descriptor *desc)
+{
+ return (void *)((unsigned long)desc + (desc->offset8 << 3));
+}
+
+/* Get data size associated with descriptor */
+static inline u32 hv_pkt_datalen(const struct vmpacket_descriptor *desc)
+{
+ return (desc->len8 << 3) - (desc->offset8 << 3);
+}
+
+
+struct vmpacket_descriptor *
+hv_pkt_iter_first(struct vmbus_channel *channel);
+
+struct vmpacket_descriptor *
+__hv_pkt_iter_next(struct vmbus_channel *channel,
+ const struct vmpacket_descriptor *pkt);
+
+void hv_pkt_iter_close(struct vmbus_channel *channel);
+
+/*
+ * Get next packet descriptor from iterator
+ * If at end of list, return NULL and update host.
+ */
static inline struct vmpacket_descriptor *
-get_next_pkt_raw(struct vmbus_channel *channel)
+hv_pkt_iter_next(struct vmbus_channel *channel,
+ const struct vmpacket_descriptor *pkt)
{
- struct hv_ring_buffer_info *ring_info = &channel->inbound;
- u32 priv_read_loc = ring_info->priv_read_index;
- void *ring_buffer = hv_get_ring_buffer(ring_info);
- u32 dsize = ring_info->ring_datasize;
- /*
- * delta is the difference between what is available to read and
- * what was already consumed in place. We commit read index after
- * the whole batch is processed.
- */
- u32 delta = priv_read_loc >= ring_info->ring_buffer->read_index ?
- priv_read_loc - ring_info->ring_buffer->read_index :
- (dsize - ring_info->ring_buffer->read_index) + priv_read_loc;
- u32 bytes_avail_toread = (hv_get_bytes_to_read(ring_info) - delta);
+ struct vmpacket_descriptor *nxt;
- if (bytes_avail_toread < sizeof(struct vmpacket_descriptor))
- return NULL;
+ nxt = __hv_pkt_iter_next(channel, pkt);
+ if (!nxt)
+ hv_pkt_iter_close(channel);
- return ring_buffer + priv_read_loc;
+ return nxt;
}
-/*
- * A helper function to step through packets "in-place"
- * This API is to be called after each successful call
- * get_next_pkt_raw().
- */
-static inline void put_pkt_raw(struct vmbus_channel *channel,
- struct vmpacket_descriptor *desc)
-{
- struct hv_ring_buffer_info *ring_info = &channel->inbound;
- u32 packetlen = desc->len8 << 3;
- u32 dsize = ring_info->ring_datasize;
-
- /*
- * Include the packet trailer.
- */
- ring_info->priv_read_index += packetlen + VMBUS_PKT_TRAILER;
- ring_info->priv_read_index %= dsize;
-}
-
-/*
- * This call commits the read index and potentially signals the host.
- * Here is the pattern for using the "in-place" consumption APIs:
- *
- * init_cached_read_index();
- *
- * while (get_next_pkt_raw() {
- * process the packet "in-place";
- * put_pkt_raw();
- * }
- * if (packets processed in place)
- * commit_rd_index();
- */
-static inline void commit_rd_index(struct vmbus_channel *channel)
-{
- struct hv_ring_buffer_info *ring_info = &channel->inbound;
- /*
- * Make sure all reads are done before we update the read index since
- * the writer may start writing to the read area once the read index
- * is updated.
- */
- virt_rmb();
- ring_info->ring_buffer->read_index = ring_info->priv_read_index;
-
- hv_signal_on_read(channel);
-}
-
+#define foreach_vmbus_pkt(pkt, channel) \
+ for (pkt = hv_pkt_iter_first(channel); pkt; \
+ pkt = hv_pkt_iter_next(channel, pkt))
#endif /* _HYPERV_H */
diff --git a/include/linux/i2c/i2c-hid.h b/include/linux/i2c/i2c-hid.h
index 7aa901d..1fb0882 100644
--- a/include/linux/i2c/i2c-hid.h
+++ b/include/linux/i2c/i2c-hid.h
@@ -14,9 +14,13 @@
#include <linux/types.h>
+struct regulator;
+
/**
* struct i2chid_platform_data - used by hid over i2c implementation.
* @hid_descriptor_address: i2c register where the HID descriptor is stored.
+ * @supply: regulator for powering on the device.
+ * @post_power_delay_ms: delay after powering on before device is usable.
*
* Note that it is the responsibility of the platform driver (or the acpi 5.0
* driver, or the flattened device tree) to setup the irq related to the gpio in
@@ -31,6 +35,8 @@
*/
struct i2c_hid_platform_data {
u16 hid_descriptor_address;
+ struct regulator *supply;
+ int post_power_delay_ms;
};
#endif /* __LINUX_I2C_HID_H */
diff --git a/include/linux/ide.h b/include/linux/ide.h
index 2f51c17..6980ca3 100644
--- a/include/linux/ide.h
+++ b/include/linux/ide.h
@@ -88,7 +88,7 @@
ide_req(rq)->type == ATA_PRIV_PM_RESUME);
}
-/* Error codes returned in rq->errors to the higher part of the driver. */
+/* Error codes returned in result to the higher part of the driver. */
enum {
IDE_DRV_ERROR_GENERAL = 101,
IDE_DRV_ERROR_FILEMARK = 102,
diff --git a/include/linux/ieee80211.h b/include/linux/ieee80211.h
index 0dd9498..6903335 100644
--- a/include/linux/ieee80211.h
+++ b/include/linux/ieee80211.h
@@ -7,7 +7,7 @@
* Copyright (c) 2005, Devicescape Software, Inc.
* Copyright (c) 2006, Michael Wu <flamingice@sourmilk.net>
* Copyright (c) 2013 - 2014 Intel Mobile Communications GmbH
- * Copyright (c) 2016 Intel Deutschland GmbH
+ * Copyright (c) 2016 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -1411,6 +1411,8 @@
#define IEEE80211_HT_OP_MODE_PROTECTION_NONHT_MIXED 3
#define IEEE80211_HT_OP_MODE_NON_GF_STA_PRSNT 0x0004
#define IEEE80211_HT_OP_MODE_NON_HT_STA_PRSNT 0x0010
+#define IEEE80211_HT_OP_MODE_CCFS2_SHIFT 5
+#define IEEE80211_HT_OP_MODE_CCFS2_MASK 0x1fe0
/* for stbc_param */
#define IEEE80211_HT_STBC_PARAM_DUAL_BEACON 0x0040
@@ -1525,14 +1527,14 @@
* This structure is the "VHT operation element" as
* described in 802.11ac D3.0 8.4.2.161
* @chan_width: Operating channel width
+ * @center_freq_seg0_idx: center freq segment 0 index
* @center_freq_seg1_idx: center freq segment 1 index
- * @center_freq_seg2_idx: center freq segment 2 index
* @basic_mcs_set: VHT Basic MCS rate set
*/
struct ieee80211_vht_operation {
u8 chan_width;
+ u8 center_freq_seg0_idx;
u8 center_freq_seg1_idx;
- u8 center_freq_seg2_idx;
__le16 basic_mcs_set;
} __packed;
@@ -1721,6 +1723,9 @@
WLAN_STATUS_REJECT_DSE_BAND = 96,
WLAN_STATUS_DENIED_WITH_SUGGESTED_BAND_AND_CHANNEL = 99,
WLAN_STATUS_DENIED_DUE_TO_SPECTRUM_MANAGEMENT = 103,
+ /* 802.11ai */
+ WLAN_STATUS_FILS_AUTHENTICATION_FAILURE = 108,
+ WLAN_STATUS_UNKNOWN_AUTHENTICATION_SERVER = 109,
};
@@ -2102,6 +2107,12 @@
#define FILS_NONCE_LEN 16
#define FILS_MAX_KEK_LEN 64
+#define FILS_ERP_MAX_USERNAME_LEN 16
+#define FILS_ERP_MAX_REALM_LEN 253
+#define FILS_ERP_MAX_RRK_LEN 64
+
+#define PMK_MAX_LEN 48
+
/* Public action codes */
enum ieee80211_pub_actioncode {
WLAN_PUB_ACTION_EXT_CHANSW_ANN = 4,
@@ -2166,37 +2177,37 @@
#define WLAN_BSS_COEX_INFORMATION_REQUEST BIT(0)
/**
- * enum - mesh synchronization method identifier
+ * enum ieee80211_mesh_sync_method - mesh synchronization method identifier
*
* @IEEE80211_SYNC_METHOD_NEIGHBOR_OFFSET: the default synchronization method
* @IEEE80211_SYNC_METHOD_VENDOR: a vendor specific synchronization method
* that will be specified in a vendor specific information element
*/
-enum {
+enum ieee80211_mesh_sync_method {
IEEE80211_SYNC_METHOD_NEIGHBOR_OFFSET = 1,
IEEE80211_SYNC_METHOD_VENDOR = 255,
};
/**
- * enum - mesh path selection protocol identifier
+ * enum ieee80211_mesh_path_protocol - mesh path selection protocol identifier
*
* @IEEE80211_PATH_PROTOCOL_HWMP: the default path selection protocol
* @IEEE80211_PATH_PROTOCOL_VENDOR: a vendor specific protocol that will
* be specified in a vendor specific information element
*/
-enum {
+enum ieee80211_mesh_path_protocol {
IEEE80211_PATH_PROTOCOL_HWMP = 1,
IEEE80211_PATH_PROTOCOL_VENDOR = 255,
};
/**
- * enum - mesh path selection metric identifier
+ * enum ieee80211_mesh_path_metric - mesh path selection metric identifier
*
* @IEEE80211_PATH_METRIC_AIRTIME: the default path selection metric
* @IEEE80211_PATH_METRIC_VENDOR: a vendor specific metric that will be
* specified in a vendor specific information element
*/
-enum {
+enum ieee80211_mesh_path_metric {
IEEE80211_PATH_METRIC_AIRTIME = 1,
IEEE80211_PATH_METRIC_VENDOR = 255,
};
@@ -2305,6 +2316,32 @@
__le32 value;
} __packed;
+/**
+ * enum ieee80211_idle_options - BSS idle options
+ * @WLAN_IDLE_OPTIONS_PROTECTED_KEEP_ALIVE: the station should send an RSN
+ * protected frame to the AP to reset the idle timer at the AP for
+ * the station.
+ */
+enum ieee80211_idle_options {
+ WLAN_IDLE_OPTIONS_PROTECTED_KEEP_ALIVE = BIT(0),
+};
+
+/**
+ * struct ieee80211_bss_max_idle_period_ie
+ *
+ * This structure refers to "BSS Max idle period element"
+ *
+ * @max_idle_period: indicates the time period during which a station can
+ * refrain from transmitting frames to its associated AP without being
+ * disassociated. In units of 1000 TUs.
+ * @idle_options: indicates the options associated with the BSS idle capability
+ * as specified in &enum ieee80211_idle_options.
+ */
+struct ieee80211_bss_max_idle_period_ie {
+ __le16 max_idle_period;
+ u8 idle_options;
+} __packed;
+
/* BACK action code */
enum ieee80211_back_actioncode {
WLAN_ACTION_ADDBA_REQ = 0,
@@ -2345,13 +2382,21 @@
#define WLAN_CIPHER_SUITE_SMS4 SUITE(0x001472, 1)
/* AKM suite selectors */
-#define WLAN_AKM_SUITE_8021X SUITE(0x000FAC, 1)
-#define WLAN_AKM_SUITE_PSK SUITE(0x000FAC, 2)
-#define WLAN_AKM_SUITE_8021X_SHA256 SUITE(0x000FAC, 5)
-#define WLAN_AKM_SUITE_PSK_SHA256 SUITE(0x000FAC, 6)
-#define WLAN_AKM_SUITE_TDLS SUITE(0x000FAC, 7)
-#define WLAN_AKM_SUITE_SAE SUITE(0x000FAC, 8)
-#define WLAN_AKM_SUITE_FT_OVER_SAE SUITE(0x000FAC, 9)
+#define WLAN_AKM_SUITE_8021X SUITE(0x000FAC, 1)
+#define WLAN_AKM_SUITE_PSK SUITE(0x000FAC, 2)
+#define WLAN_AKM_SUITE_FT_8021X SUITE(0x000FAC, 3)
+#define WLAN_AKM_SUITE_FT_PSK SUITE(0x000FAC, 4)
+#define WLAN_AKM_SUITE_8021X_SHA256 SUITE(0x000FAC, 5)
+#define WLAN_AKM_SUITE_PSK_SHA256 SUITE(0x000FAC, 6)
+#define WLAN_AKM_SUITE_TDLS SUITE(0x000FAC, 7)
+#define WLAN_AKM_SUITE_SAE SUITE(0x000FAC, 8)
+#define WLAN_AKM_SUITE_FT_OVER_SAE SUITE(0x000FAC, 9)
+#define WLAN_AKM_SUITE_8021X_SUITE_B SUITE(0x000FAC, 11)
+#define WLAN_AKM_SUITE_8021X_SUITE_B_192 SUITE(0x000FAC, 12)
+#define WLAN_AKM_SUITE_FILS_SHA256 SUITE(0x000FAC, 14)
+#define WLAN_AKM_SUITE_FILS_SHA384 SUITE(0x000FAC, 15)
+#define WLAN_AKM_SUITE_FT_FILS_SHA256 SUITE(0x000FAC, 16)
+#define WLAN_AKM_SUITE_FT_FILS_SHA384 SUITE(0x000FAC, 17)
#define WLAN_MAX_KEY_LEN 32
diff --git a/include/linux/if_bridge.h b/include/linux/if_bridge.h
index c5847dc..0c16866 100644
--- a/include/linux/if_bridge.h
+++ b/include/linux/if_bridge.h
@@ -48,6 +48,7 @@
#define BR_MCAST_FLOOD BIT(11)
#define BR_MULTICAST_TO_UNICAST BIT(12)
#define BR_VLAN_TUNNEL BIT(13)
+#define BR_BCAST_FLOOD BIT(14)
#define BR_DEFAULT_AGEING_TIME (300 * HZ)
diff --git a/include/linux/inet.h b/include/linux/inet.h
index 4cca05c..636ebe8 100644
--- a/include/linux/inet.h
+++ b/include/linux/inet.h
@@ -43,6 +43,8 @@
#define _LINUX_INET_H
#include <linux/types.h>
+#include <net/net_namespace.h>
+#include <linux/socket.h>
/*
* These mimic similar macros defined in user-space for inet_ntop(3).
@@ -54,4 +56,8 @@
extern __be32 in_aton(const char *str);
extern int in4_pton(const char *src, int srclen, u8 *dst, int delim, const char **end);
extern int in6_pton(const char *src, int srclen, u8 *dst, int delim, const char **end);
+
+extern int inet_pton_with_scope(struct net *net, unsigned short af,
+ const char *src, const char *port, struct sockaddr_storage *addr);
+
#endif /* _LINUX_INET_H */
diff --git a/include/linux/inetdevice.h b/include/linux/inetdevice.h
index ee971f3..a2e9d6e 100644
--- a/include/linux/inetdevice.h
+++ b/include/linux/inetdevice.h
@@ -153,8 +153,8 @@
int register_inetaddr_notifier(struct notifier_block *nb);
int unregister_inetaddr_notifier(struct notifier_block *nb);
-void inet_netconf_notify_devconf(struct net *net, int type, int ifindex,
- struct ipv4_devconf *devconf);
+void inet_netconf_notify_devconf(struct net *net, int event, int type,
+ int ifindex, struct ipv4_devconf *devconf);
struct net_device *__ip_dev_find(struct net *net, __be32 addr, bool devref);
static inline struct net_device *ip_dev_find(struct net *net, __be32 addr)
diff --git a/include/linux/init_task.h b/include/linux/init_task.h
index 91d9049..82be965 100644
--- a/include/linux/init_task.h
+++ b/include/linux/init_task.h
@@ -15,6 +15,7 @@
#include <linux/sched/autogroup.h>
#include <net/net_namespace.h>
#include <linux/sched/rt.h>
+#include <linux/livepatch.h>
#include <linux/mm_types.h>
#include <asm/thread_info.h>
@@ -181,6 +182,7 @@
#ifdef CONFIG_RT_MUTEXES
# define INIT_RT_MUTEXES(tsk) \
.pi_waiters = RB_ROOT, \
+ .pi_top_task = NULL, \
.pi_waiters_leftmost = NULL,
#else
# define INIT_RT_MUTEXES(tsk)
@@ -202,6 +204,13 @@
# define INIT_KASAN(tsk)
#endif
+#ifdef CONFIG_LIVEPATCH
+# define INIT_LIVEPATCH(tsk) \
+ .patch_state = KLP_UNDEFINED,
+#else
+# define INIT_LIVEPATCH(tsk)
+#endif
+
#ifdef CONFIG_THREAD_INFO_IN_TASK
# define INIT_TASK_TI(tsk) \
.thread_info = INIT_THREAD_INFO(tsk), \
@@ -288,6 +297,7 @@
INIT_VTIME(tsk) \
INIT_NUMA_BALANCING(tsk) \
INIT_KASAN(tsk) \
+ INIT_LIVEPATCH(tsk) \
}
diff --git a/include/linux/ipv6.h b/include/linux/ipv6.h
index 71be5b3..e1b4429 100644
--- a/include/linux/ipv6.h
+++ b/include/linux/ipv6.h
@@ -37,6 +37,7 @@
__s32 accept_ra_rtr_pref;
__s32 rtr_probe_interval;
#ifdef CONFIG_IPV6_ROUTE_INFO
+ __s32 accept_ra_rt_info_min_plen;
__s32 accept_ra_rt_info_max_plen;
#endif
#endif
@@ -70,6 +71,7 @@
#endif
__u32 enhanced_dad;
__u32 addr_gen_mode;
+ __s32 disable_policy;
struct ctl_table_header *sysctl_header;
};
diff --git a/include/linux/irqchip/mips-gic.h b/include/linux/irqchip/mips-gic.h
index 7b49c71..2b0e566 100644
--- a/include/linux/irqchip/mips-gic.h
+++ b/include/linux/irqchip/mips-gic.h
@@ -258,7 +258,6 @@
extern void gic_init(unsigned long gic_base_addr,
unsigned long gic_addrspace_size, unsigned int cpu_vec,
unsigned int irqbase);
-extern void gic_clocksource_init(unsigned int);
extern u64 gic_read_count(void);
extern unsigned int gic_get_count_width(void);
extern u64 gic_read_compare(void);
diff --git a/include/linux/kernel.h b/include/linux/kernel.h
index 4c26dc3..13bc08a 100644
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -47,6 +47,7 @@
/* @a is a power of 2 value */
#define ALIGN(x, a) __ALIGN_KERNEL((x), (a))
+#define ALIGN_DOWN(x, a) __ALIGN_KERNEL((x) - ((a) - 1), (a))
#define __ALIGN_MASK(x, mask) __ALIGN_KERNEL_MASK((x), (mask))
#define PTR_ALIGN(p, a) ((typeof(p))ALIGN((unsigned long)(p), (a)))
#define IS_ALIGNED(x, a) (((x) & ((typeof(x))(a) - 1)) == 0)
@@ -438,6 +439,7 @@
extern char *get_options(const char *str, int nints, int *ints);
extern unsigned long long memparse(const char *ptr, char **retptr);
extern bool parse_option_str(const char *str, const char *option);
+extern char *next_arg(char *args, char **param, char **val);
extern int core_kernel_text(unsigned long addr);
extern int core_kernel_data(unsigned long addr);
diff --git a/include/linux/kobject.h b/include/linux/kobject.h
index e628459..ca85cb8 100644
--- a/include/linux/kobject.h
+++ b/include/linux/kobject.h
@@ -108,6 +108,8 @@
extern int __must_check kobject_move(struct kobject *, struct kobject *);
extern struct kobject *kobject_get(struct kobject *kobj);
+extern struct kobject * __must_check kobject_get_unless_zero(
+ struct kobject *kobj);
extern void kobject_put(struct kobject *kobj);
extern const void *kobject_namespace(struct kobject *kobj);
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index c328e4f..47e4da5 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -267,6 +267,8 @@
extern void show_registers(struct pt_regs *regs);
extern void kprobes_inc_nmissed_count(struct kprobe *p);
extern bool arch_within_kprobe_blacklist(unsigned long addr);
+extern bool arch_function_offset_within_entry(unsigned long offset);
+extern bool function_offset_within_entry(kprobe_opcode_t *addr, const char *sym, unsigned long offset);
extern bool within_kprobe_blacklist(unsigned long addr);
diff --git a/include/linux/leds-pca9532.h b/include/linux/leds-pca9532.h
index d215b45..5e240b2 100644
--- a/include/linux/leds-pca9532.h
+++ b/include/linux/leds-pca9532.h
@@ -22,7 +22,8 @@
PCA9532_OFF = 0x0,
PCA9532_ON = 0x1,
PCA9532_PWM0 = 0x2,
- PCA9532_PWM1 = 0x3
+ PCA9532_PWM1 = 0x3,
+ PCA9532_KEEP = 0xff,
};
struct pca9532_led {
@@ -44,4 +45,3 @@
};
#endif /* __LINUX_PCA9532_H */
-
diff --git a/include/linux/leds.h b/include/linux/leds.h
index 38c0bd7..64c56d4 100644
--- a/include/linux/leds.h
+++ b/include/linux/leds.h
@@ -122,10 +122,16 @@
struct mutex led_access;
};
-extern int led_classdev_register(struct device *parent,
- struct led_classdev *led_cdev);
-extern int devm_led_classdev_register(struct device *parent,
- struct led_classdev *led_cdev);
+extern int of_led_classdev_register(struct device *parent,
+ struct device_node *np,
+ struct led_classdev *led_cdev);
+#define led_classdev_register(parent, led_cdev) \
+ of_led_classdev_register(parent, NULL, led_cdev)
+extern int devm_of_led_classdev_register(struct device *parent,
+ struct device_node *np,
+ struct led_classdev *led_cdev);
+#define devm_led_classdev_register(parent, led_cdev) \
+ devm_of_led_classdev_register(parent, NULL, led_cdev)
extern void led_classdev_unregister(struct led_classdev *led_cdev);
extern void devm_led_classdev_unregister(struct device *parent,
struct led_classdev *led_cdev);
diff --git a/include/linux/lightnvm.h b/include/linux/lightnvm.h
index ca45e4a..7dfa56e 100644
--- a/include/linux/lightnvm.h
+++ b/include/linux/lightnvm.h
@@ -56,7 +56,6 @@
typedef int (nvm_op_bb_tbl_fn)(struct nvm_dev *, struct ppa_addr, u8 *);
typedef int (nvm_op_set_bb_fn)(struct nvm_dev *, struct ppa_addr *, int, int);
typedef int (nvm_submit_io_fn)(struct nvm_dev *, struct nvm_rq *);
-typedef int (nvm_erase_blk_fn)(struct nvm_dev *, struct nvm_rq *);
typedef void *(nvm_create_dma_pool_fn)(struct nvm_dev *, char *);
typedef void (nvm_destroy_dma_pool_fn)(void *);
typedef void *(nvm_dev_dma_alloc_fn)(struct nvm_dev *, void *, gfp_t,
@@ -70,7 +69,6 @@
nvm_op_set_bb_fn *set_bb_tbl;
nvm_submit_io_fn *submit_io;
- nvm_erase_blk_fn *erase_block;
nvm_create_dma_pool_fn *create_dma_pool;
nvm_destroy_dma_pool_fn *destroy_dma_pool;
@@ -125,7 +123,7 @@
/* NAND Access Modes */
NVM_IO_SUSPEND = 0x80,
NVM_IO_SLC_MODE = 0x100,
- NVM_IO_SCRAMBLE_DISABLE = 0x200,
+ NVM_IO_SCRAMBLE_ENABLE = 0x200,
/* Block Types */
NVM_BLK_T_FREE = 0x0,
@@ -438,7 +436,8 @@
typedef blk_qc_t (nvm_tgt_make_rq_fn)(struct request_queue *, struct bio *);
typedef sector_t (nvm_tgt_capacity_fn)(void *);
-typedef void *(nvm_tgt_init_fn)(struct nvm_tgt_dev *, struct gendisk *);
+typedef void *(nvm_tgt_init_fn)(struct nvm_tgt_dev *, struct gendisk *,
+ int flags);
typedef void (nvm_tgt_exit_fn)(void *);
typedef int (nvm_tgt_sysfs_init_fn)(struct gendisk *);
typedef void (nvm_tgt_sysfs_exit_fn)(struct gendisk *);
@@ -479,10 +478,10 @@
int, int);
extern int nvm_max_phys_sects(struct nvm_tgt_dev *);
extern int nvm_submit_io(struct nvm_tgt_dev *, struct nvm_rq *);
-extern int nvm_set_rqd_ppalist(struct nvm_dev *, struct nvm_rq *,
+extern int nvm_erase_sync(struct nvm_tgt_dev *, struct ppa_addr *, int);
+extern int nvm_set_rqd_ppalist(struct nvm_tgt_dev *, struct nvm_rq *,
const struct ppa_addr *, int, int);
-extern void nvm_free_rqd_ppalist(struct nvm_dev *, struct nvm_rq *);
-extern int nvm_erase_blk(struct nvm_tgt_dev *, struct ppa_addr *, int);
+extern void nvm_free_rqd_ppalist(struct nvm_tgt_dev *, struct nvm_rq *);
extern int nvm_get_l2p_tbl(struct nvm_tgt_dev *, u64, u32, nvm_l2p_update_fn *,
void *);
extern int nvm_get_area(struct nvm_tgt_dev *, sector_t *, sector_t);
diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
index 9072f04..194991e 100644
--- a/include/linux/livepatch.h
+++ b/include/linux/livepatch.h
@@ -23,15 +23,16 @@
#include <linux/module.h>
#include <linux/ftrace.h>
+#include <linux/completion.h>
#if IS_ENABLED(CONFIG_LIVEPATCH)
#include <asm/livepatch.h>
-enum klp_state {
- KLP_DISABLED,
- KLP_ENABLED
-};
+/* task patch states */
+#define KLP_UNDEFINED -1
+#define KLP_UNPATCHED 0
+#define KLP_PATCHED 1
/**
* struct klp_func - function structure for live patching
@@ -39,10 +40,29 @@
* @new_func: pointer to the patched function code
* @old_sympos: a hint indicating which symbol position the old function
* can be found (optional)
+ * @immediate: patch the func immediately, bypassing safety mechanisms
* @old_addr: the address of the function being patched
* @kobj: kobject for sysfs resources
- * @state: tracks function-level patch application state
* @stack_node: list node for klp_ops func_stack list
+ * @old_size: size of the old function
+ * @new_size: size of the new function
+ * @patched: the func has been added to the klp_ops list
+ * @transition: the func is currently being applied or reverted
+ *
+ * The patched and transition variables define the func's patching state. When
+ * patching, a func is always in one of the following states:
+ *
+ * patched=0 transition=0: unpatched
+ * patched=0 transition=1: unpatched, temporary starting state
+ * patched=1 transition=1: patched, may be visible to some tasks
+ * patched=1 transition=0: patched, visible to all tasks
+ *
+ * And when unpatching, it goes in the reverse order:
+ *
+ * patched=1 transition=0: patched, visible to all tasks
+ * patched=1 transition=1: patched, may be visible to some tasks
+ * patched=0 transition=1: unpatched, temporary ending state
+ * patched=0 transition=0: unpatched
*/
struct klp_func {
/* external */
@@ -56,12 +76,15 @@
* in kallsyms for the given object is used.
*/
unsigned long old_sympos;
+ bool immediate;
/* internal */
unsigned long old_addr;
struct kobject kobj;
- enum klp_state state;
struct list_head stack_node;
+ unsigned long old_size, new_size;
+ bool patched;
+ bool transition;
};
/**
@@ -70,8 +93,8 @@
* @funcs: function entries for functions to be patched in the object
* @kobj: kobject for sysfs resources
* @mod: kernel module associated with the patched object
- * (NULL for vmlinux)
- * @state: tracks object-level patch application state
+ * (NULL for vmlinux)
+ * @patched: the object's funcs have been added to the klp_ops list
*/
struct klp_object {
/* external */
@@ -81,26 +104,30 @@
/* internal */
struct kobject kobj;
struct module *mod;
- enum klp_state state;
+ bool patched;
};
/**
* struct klp_patch - patch structure for live patching
* @mod: reference to the live patch module
* @objs: object entries for kernel objects to be patched
+ * @immediate: patch all funcs immediately, bypassing safety mechanisms
* @list: list node for global list of registered patches
* @kobj: kobject for sysfs resources
- * @state: tracks patch-level application state
+ * @enabled: the patch is enabled (but operation may be incomplete)
+ * @finish: for waiting till it is safe to remove the patch module
*/
struct klp_patch {
/* external */
struct module *mod;
struct klp_object *objs;
+ bool immediate;
/* internal */
struct list_head list;
struct kobject kobj;
- enum klp_state state;
+ bool enabled;
+ struct completion finish;
};
#define klp_for_each_object(patch, obj) \
@@ -123,10 +150,27 @@
int klp_module_coming(struct module *mod);
void klp_module_going(struct module *mod);
+void klp_copy_process(struct task_struct *child);
+void klp_update_patch_state(struct task_struct *task);
+
+static inline bool klp_patch_pending(struct task_struct *task)
+{
+ return test_tsk_thread_flag(task, TIF_PATCH_PENDING);
+}
+
+static inline bool klp_have_reliable_stack(void)
+{
+ return IS_ENABLED(CONFIG_STACKTRACE) &&
+ IS_ENABLED(CONFIG_HAVE_RELIABLE_STACKTRACE);
+}
+
#else /* !CONFIG_LIVEPATCH */
static inline int klp_module_coming(struct module *mod) { return 0; }
-static inline void klp_module_going(struct module *mod) { }
+static inline void klp_module_going(struct module *mod) {}
+static inline bool klp_patch_pending(struct task_struct *task) { return false; }
+static inline void klp_update_patch_state(struct task_struct *task) {}
+static inline void klp_copy_process(struct task_struct *child) {}
#endif /* CONFIG_LIVEPATCH */
diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h
index 1e327bb..fffe49f 100644
--- a/include/linux/lockdep.h
+++ b/include/linux/lockdep.h
@@ -361,6 +361,8 @@
lock_set_class(lock, lock->name, lock->key, subclass, ip);
}
+extern void lock_downgrade(struct lockdep_map *lock, unsigned long ip);
+
extern void lockdep_set_current_reclaim_state(gfp_t gfp_mask);
extern void lockdep_clear_current_reclaim_state(void);
extern void lockdep_trace_alloc(gfp_t mask);
@@ -411,6 +413,7 @@
# define lock_acquire(l, s, t, r, c, n, i) do { } while (0)
# define lock_release(l, n, i) do { } while (0)
+# define lock_downgrade(l, i) do { } while (0)
# define lock_set_class(l, n, k, s, i) do { } while (0)
# define lock_set_subclass(l, s, i) do { } while (0)
# define lockdep_set_current_reclaim_state(g) do { } while (0)
diff --git a/include/linux/mailbox/brcm-message.h b/include/linux/mailbox/brcm-message.h
index 6b55c93..c20b484 100644
--- a/include/linux/mailbox/brcm-message.h
+++ b/include/linux/mailbox/brcm-message.h
@@ -16,6 +16,7 @@
enum brcm_message_type {
BRCM_MESSAGE_UNKNOWN = 0,
+ BRCM_MESSAGE_BATCH,
BRCM_MESSAGE_SPU,
BRCM_MESSAGE_SBA,
BRCM_MESSAGE_MAX,
@@ -23,24 +24,29 @@
struct brcm_sba_command {
u64 cmd;
+ u64 *cmd_dma;
+ dma_addr_t cmd_dma_addr;
#define BRCM_SBA_CMD_TYPE_A BIT(0)
#define BRCM_SBA_CMD_TYPE_B BIT(1)
#define BRCM_SBA_CMD_TYPE_C BIT(2)
#define BRCM_SBA_CMD_HAS_RESP BIT(3)
#define BRCM_SBA_CMD_HAS_OUTPUT BIT(4)
u64 flags;
- dma_addr_t input;
- size_t input_len;
dma_addr_t resp;
size_t resp_len;
- dma_addr_t output;
- size_t output_len;
+ dma_addr_t data;
+ size_t data_len;
};
struct brcm_message {
enum brcm_message_type type;
union {
struct {
+ struct brcm_message *msgs;
+ unsigned int msgs_queued;
+ unsigned int msgs_count;
+ } batch;
+ struct {
struct scatterlist *src;
struct scatterlist *dst;
} spu;
diff --git a/include/linux/mfd/motorola-cpcap.h b/include/linux/mfd/motorola-cpcap.h
index b4031c2..53758a7 100644
--- a/include/linux/mfd/motorola-cpcap.h
+++ b/include/linux/mfd/motorola-cpcap.h
@@ -14,6 +14,9 @@
* published by the Free Software Foundation.
*/
+#include <linux/device.h>
+#include <linux/regmap.h>
+
#define CPCAP_VENDOR_ST 0
#define CPCAP_VENDOR_TI 1
diff --git a/include/linux/mg_disk.h b/include/linux/mg_disk.h
deleted file mode 100644
index e11f4d9..0000000
--- a/include/linux/mg_disk.h
+++ /dev/null
@@ -1,45 +0,0 @@
-/*
- * include/linux/mg_disk.c
- *
- * Private data for mflash platform driver
- *
- * (c) 2008 mGine Co.,LTD
- * (c) 2008 unsik Kim <donari75@gmail.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#ifndef __MG_DISK_H__
-#define __MG_DISK_H__
-
-/* name for platform device */
-#define MG_DEV_NAME "mg_disk"
-
-/* names of GPIO resource */
-#define MG_RST_PIN "mg_rst"
-/* except MG_BOOT_DEV, reset-out pin should be assigned */
-#define MG_RSTOUT_PIN "mg_rstout"
-
-/* device attribution */
-/* use mflash as boot device */
-#define MG_BOOT_DEV (1 << 0)
-/* use mflash as storage device */
-#define MG_STORAGE_DEV (1 << 1)
-/* same as MG_STORAGE_DEV, but bootloader already done reset sequence */
-#define MG_STORAGE_DEV_SKIP_RST (1 << 2)
-
-/* private driver data */
-struct mg_drv_data {
- /* disk resource */
- u32 use_polling;
-
- /* device attribution */
- u32 dev_attr;
-
- /* internally used */
- void *host;
-};
-
-#endif
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 2fcff6b..3fece51 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -540,6 +540,7 @@
struct workqueue_struct *wq;
struct delayed_work work;
unsigned long next_query;
+ unsigned long sampling_interval; /* jiffies */
};
struct mlx5_eswitch;
@@ -728,6 +729,7 @@
u32 pdn;
struct mlx5_td td;
struct mlx5_core_mkey mkey;
+ struct mlx5_sq_bfreg bfreg;
};
struct mlx5_core_dev {
diff --git a/include/linux/mlx5/fs.h b/include/linux/mlx5/fs.h
index 949b24b..1b166d2 100644
--- a/include/linux/mlx5/fs.h
+++ b/include/linux/mlx5/fs.h
@@ -104,12 +104,18 @@
u32 level,
u32 flags);
+struct mlx5_flow_table_attr {
+ int prio;
+ int max_fte;
+ u32 level;
+ u32 flags;
+ u32 underlay_qpn;
+};
+
struct mlx5_flow_table *
mlx5_create_flow_table(struct mlx5_flow_namespace *ns,
- int prio,
- int num_flow_table_entries,
- u32 level,
- u32 flags);
+ struct mlx5_flow_table_attr *ft_attr);
+
struct mlx5_flow_table *
mlx5_create_vport_flow_table(struct mlx5_flow_namespace *ns,
int prio,
@@ -134,8 +140,13 @@
u32 action;
u32 flow_tag;
u32 encap_id;
+ u32 modify_id;
};
+#define MLX5_DECLARE_FLOW_ACT(name) \
+ struct mlx5_flow_act name = {MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,\
+ MLX5_FS_DEFAULT_FLOW_TAG, 0, 0}
+
/* Single destination per rule.
* Group ID is implied by the match criteria.
*/
@@ -156,5 +167,4 @@
void mlx5_fc_destroy(struct mlx5_core_dev *dev, struct mlx5_fc *counter);
void mlx5_fc_query_cached(struct mlx5_fc *counter,
u64 *bytes, u64 *packets, u64 *lastuse);
-
#endif
diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index 8382426..7c50bd3 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -227,6 +227,8 @@
MLX5_CMD_OP_MODIFY_FLOW_TABLE = 0x93c,
MLX5_CMD_OP_ALLOC_ENCAP_HEADER = 0x93d,
MLX5_CMD_OP_DEALLOC_ENCAP_HEADER = 0x93e,
+ MLX5_CMD_OP_ALLOC_MODIFY_HEADER_CONTEXT = 0x940,
+ MLX5_CMD_OP_DEALLOC_MODIFY_HEADER_CONTEXT = 0x941,
MLX5_CMD_OP_MAX
};
@@ -302,7 +304,8 @@
u8 reserved_at_20[0x2];
u8 log_max_ft_size[0x6];
- u8 reserved_at_28[0x10];
+ u8 log_max_modify_header_context[0x8];
+ u8 max_modify_header_actions[0x8];
u8 max_ft_level[0x8];
u8 reserved_at_40[0x20];
@@ -869,7 +872,8 @@
u8 compact_address_vector[0x1];
u8 striding_rq[0x1];
- u8 reserved_at_202[0x2];
+ u8 reserved_at_202[0x1];
+ u8 ipoib_enhanced_offloads[0x1];
u8 ipoib_basic_offloads[0x1];
u8 reserved_at_205[0xa];
u8 drain_sigerr[0x1];
@@ -2190,6 +2194,7 @@
MLX5_FLOW_CONTEXT_ACTION_COUNT = 0x8,
MLX5_FLOW_CONTEXT_ACTION_ENCAP = 0x10,
MLX5_FLOW_CONTEXT_ACTION_DECAP = 0x20,
+ MLX5_FLOW_CONTEXT_ACTION_MOD_HDR = 0x40,
};
struct mlx5_ifc_flow_context_bits {
@@ -2211,7 +2216,9 @@
u8 encap_id[0x20];
- u8 reserved_at_e0[0x120];
+ u8 modify_header_id[0x20];
+
+ u8 reserved_at_100[0x100];
struct mlx5_ifc_fte_match_param_bits match_value;
@@ -2287,7 +2294,9 @@
u8 reserved_at_120[0x8];
u8 transport_domain[0x18];
- u8 reserved_at_140[0x3c0];
+ u8 reserved_at_140[0x8];
+ u8 underlay_qpn[0x18];
+ u8 reserved_at_160[0x3a0];
};
enum {
@@ -4534,6 +4543,109 @@
u8 reserved_60[0x20];
};
+struct mlx5_ifc_set_action_in_bits {
+ u8 action_type[0x4];
+ u8 field[0xc];
+ u8 reserved_at_10[0x3];
+ u8 offset[0x5];
+ u8 reserved_at_18[0x3];
+ u8 length[0x5];
+
+ u8 data[0x20];
+};
+
+struct mlx5_ifc_add_action_in_bits {
+ u8 action_type[0x4];
+ u8 field[0xc];
+ u8 reserved_at_10[0x10];
+
+ u8 data[0x20];
+};
+
+union mlx5_ifc_set_action_in_add_action_in_auto_bits {
+ struct mlx5_ifc_set_action_in_bits set_action_in;
+ struct mlx5_ifc_add_action_in_bits add_action_in;
+ u8 reserved_at_0[0x40];
+};
+
+enum {
+ MLX5_ACTION_TYPE_SET = 0x1,
+ MLX5_ACTION_TYPE_ADD = 0x2,
+};
+
+enum {
+ MLX5_ACTION_IN_FIELD_OUT_SMAC_47_16 = 0x1,
+ MLX5_ACTION_IN_FIELD_OUT_SMAC_15_0 = 0x2,
+ MLX5_ACTION_IN_FIELD_OUT_ETHERTYPE = 0x3,
+ MLX5_ACTION_IN_FIELD_OUT_DMAC_47_16 = 0x4,
+ MLX5_ACTION_IN_FIELD_OUT_DMAC_15_0 = 0x5,
+ MLX5_ACTION_IN_FIELD_OUT_IP_DSCP = 0x6,
+ MLX5_ACTION_IN_FIELD_OUT_TCP_FLAGS = 0x7,
+ MLX5_ACTION_IN_FIELD_OUT_TCP_SPORT = 0x8,
+ MLX5_ACTION_IN_FIELD_OUT_TCP_DPORT = 0x9,
+ MLX5_ACTION_IN_FIELD_OUT_IP_TTL = 0xa,
+ MLX5_ACTION_IN_FIELD_OUT_UDP_SPORT = 0xb,
+ MLX5_ACTION_IN_FIELD_OUT_UDP_DPORT = 0xc,
+ MLX5_ACTION_IN_FIELD_OUT_SIPV6_127_96 = 0xd,
+ MLX5_ACTION_IN_FIELD_OUT_SIPV6_95_64 = 0xe,
+ MLX5_ACTION_IN_FIELD_OUT_SIPV6_63_32 = 0xf,
+ MLX5_ACTION_IN_FIELD_OUT_SIPV6_31_0 = 0x10,
+ MLX5_ACTION_IN_FIELD_OUT_DIPV6_127_96 = 0x11,
+ MLX5_ACTION_IN_FIELD_OUT_DIPV6_95_64 = 0x12,
+ MLX5_ACTION_IN_FIELD_OUT_DIPV6_63_32 = 0x13,
+ MLX5_ACTION_IN_FIELD_OUT_DIPV6_31_0 = 0x14,
+ MLX5_ACTION_IN_FIELD_OUT_SIPV4 = 0x15,
+ MLX5_ACTION_IN_FIELD_OUT_DIPV4 = 0x16,
+};
+
+struct mlx5_ifc_alloc_modify_header_context_out_bits {
+ u8 status[0x8];
+ u8 reserved_at_8[0x18];
+
+ u8 syndrome[0x20];
+
+ u8 modify_header_id[0x20];
+
+ u8 reserved_at_60[0x20];
+};
+
+struct mlx5_ifc_alloc_modify_header_context_in_bits {
+ u8 opcode[0x10];
+ u8 reserved_at_10[0x10];
+
+ u8 reserved_at_20[0x10];
+ u8 op_mod[0x10];
+
+ u8 reserved_at_40[0x20];
+
+ u8 table_type[0x8];
+ u8 reserved_at_68[0x10];
+ u8 num_of_actions[0x8];
+
+ union mlx5_ifc_set_action_in_add_action_in_auto_bits actions[0];
+};
+
+struct mlx5_ifc_dealloc_modify_header_context_out_bits {
+ u8 status[0x8];
+ u8 reserved_at_8[0x18];
+
+ u8 syndrome[0x20];
+
+ u8 reserved_at_40[0x40];
+};
+
+struct mlx5_ifc_dealloc_modify_header_context_in_bits {
+ u8 opcode[0x10];
+ u8 reserved_at_10[0x10];
+
+ u8 reserved_at_20[0x10];
+ u8 op_mod[0x10];
+
+ u8 modify_header_id[0x20];
+
+ u8 reserved_at_60[0x20];
+};
+
struct mlx5_ifc_query_dct_out_bits {
u8 status[0x8];
u8 reserved_at_8[0x18];
@@ -5013,6 +5125,7 @@
enum {
MLX5_MODIFY_RQ_IN_MODIFY_BITMASK_VSD = 1ULL << 1,
+ MLX5_MODIFY_RQ_IN_MODIFY_BITMASK_SCATTER_FCS = 1ULL << 2,
MLX5_MODIFY_RQ_IN_MODIFY_BITMASK_RQ_COUNTER_SET_ID = 1ULL << 3,
};
@@ -8108,7 +8221,9 @@
u8 reserved_at_a0[0x8];
u8 table_id[0x18];
- u8 reserved_at_c0[0x140];
+ u8 reserved_at_c0[0x8];
+ u8 underlay_qpn[0x18];
+ u8 reserved_at_e0[0x120];
};
enum {
diff --git a/include/linux/mlx5/qp.h b/include/linux/mlx5/qp.h
index 3096370..bef80d0 100644
--- a/include/linux/mlx5/qp.h
+++ b/include/linux/mlx5/qp.h
@@ -295,6 +295,16 @@
u8 rgid[16];
};
+struct mlx5_ib_ah {
+ struct ib_ah ibah;
+ struct mlx5_av av;
+};
+
+static inline struct mlx5_ib_ah *to_mah(struct ib_ah *ibah)
+{
+ return container_of(ibah, struct mlx5_ib_ah, ibah);
+}
+
struct mlx5_wqe_datagram_seg {
struct mlx5_av av;
};
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 00a8fa7..695da2a 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -432,6 +432,10 @@
{
return 0;
}
+static inline int pgd_devmap(pgd_t pgd)
+{
+ return 0;
+}
#endif
/*
@@ -758,19 +762,11 @@
}
#ifdef CONFIG_ZONE_DEVICE
-void get_zone_device_page(struct page *page);
-void put_zone_device_page(struct page *page);
static inline bool is_zone_device_page(const struct page *page)
{
return page_zonenum(page) == ZONE_DEVICE;
}
#else
-static inline void get_zone_device_page(struct page *page)
-{
-}
-static inline void put_zone_device_page(struct page *page)
-{
-}
static inline bool is_zone_device_page(const struct page *page)
{
return false;
@@ -786,9 +782,6 @@
*/
VM_BUG_ON_PAGE(page_ref_count(page) <= 0, page);
page_ref_inc(page);
-
- if (unlikely(is_zone_device_page(page)))
- get_zone_device_page(page);
}
static inline void put_page(struct page *page)
@@ -797,9 +790,6 @@
if (put_page_testzero(page))
__put_page(page);
-
- if (unlikely(is_zone_device_page(page)))
- put_zone_device_page(page);
}
#if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index f60f45f..45cdb27 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -367,6 +367,11 @@
#endif
unsigned long mmap_base; /* base of mmap area */
unsigned long mmap_legacy_base; /* base of mmap area in bottom-up allocations */
+#ifdef CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES
+ /* Base adresses for compatible mmap() */
+ unsigned long mmap_compat_base;
+ unsigned long mmap_compat_legacy_base;
+#endif
unsigned long task_size; /* size of task vm space */
unsigned long highest_vm_end; /* highest vma end address */
pgd_t * pgd;
diff --git a/include/linux/mmc/card.h b/include/linux/mmc/card.h
index 77e61e0..aad015e 100644
--- a/include/linux/mmc/card.h
+++ b/include/linux/mmc/card.h
@@ -89,6 +89,7 @@
unsigned int boot_ro_lock; /* ro lock support */
bool boot_ro_lockable;
bool ffu_capable; /* Firmware upgrade support */
+ bool cmdq_en; /* Command Queue enabled */
bool cmdq_support; /* Command Queue supported */
unsigned int cmdq_depth; /* Command Queue depth */
#define MMC_FIRMWARE_LEN 8
@@ -208,6 +209,7 @@
struct mmc_host;
struct sdio_func;
struct sdio_func_tuple;
+struct mmc_queue_req;
#define SDIO_MAX_FUNCS 7
@@ -267,6 +269,8 @@
#define MMC_QUIRK_TRIM_BROKEN (1<<12) /* Skip trim */
#define MMC_QUIRK_BROKEN_HPI (1<<13) /* Disable broken HPI support */
+ bool reenable_cmdq; /* Re-enable Command Queue */
+
unsigned int erase_size; /* erase size in sectors */
unsigned int erase_shift; /* if erase unit is power 2 */
unsigned int pref_erase; /* in sectors */
@@ -300,6 +304,10 @@
struct dentry *debugfs_root;
struct mmc_part part[MMC_NUM_PHY_PARTITION]; /* physical partitions */
unsigned int nr_parts;
+
+ struct mmc_queue_req *mqrq; /* Shared queue structure */
+ unsigned int bouncesz; /* Bounce buffer size */
+ int qdepth; /* Shared queue depth */
};
static inline bool mmc_large_sector(struct mmc_card *card)
@@ -307,6 +315,8 @@
return card->ext_csd.data_sector_size == 4096;
}
+bool mmc_card_is_blockaddr(struct mmc_card *card);
+
#define mmc_card_mmc(c) ((c)->type == MMC_TYPE_MMC)
#define mmc_card_sd(c) ((c)->type == MMC_TYPE_SD)
#define mmc_card_sdio(c) ((c)->type == MMC_TYPE_SDIO)
diff --git a/include/linux/mmc/host.h b/include/linux/mmc/host.h
index 83f1c4a..21385ac 100644
--- a/include/linux/mmc/host.h
+++ b/include/linux/mmc/host.h
@@ -17,6 +17,7 @@
#include <linux/mmc/core.h>
#include <linux/mmc/card.h>
#include <linux/mmc/pm.h>
+#include <linux/dma-direction.h>
struct mmc_ios {
unsigned int clock; /* clock rate */
@@ -499,6 +500,11 @@
return host->can_retune == 1;
}
+static inline enum dma_data_direction mmc_get_dma_dir(struct mmc_data *data)
+{
+ return data->flags & MMC_DATA_WRITE ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
+}
+
int mmc_send_tuning(struct mmc_host *host, u32 opcode, int *cmd_error);
int mmc_abort_tuning(struct mmc_host *host, u32 opcode);
diff --git a/include/linux/module.h b/include/linux/module.h
index 0297c5c..9ad6856 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -493,6 +493,7 @@
struct module *__module_text_address(unsigned long addr);
struct module *__module_address(unsigned long addr);
bool is_module_address(unsigned long addr);
+bool __is_module_percpu_address(unsigned long addr, unsigned long *can_addr);
bool is_module_percpu_address(unsigned long addr);
bool is_module_text_address(unsigned long addr);
@@ -660,6 +661,11 @@
return false;
}
+static inline bool __is_module_percpu_address(unsigned long addr, unsigned long *can_addr)
+{
+ return false;
+}
+
static inline bool is_module_text_address(unsigned long addr)
{
return false;
diff --git a/include/linux/mpls.h b/include/linux/mpls.h
index 9999145..384fb22 100644
--- a/include/linux/mpls.h
+++ b/include/linux/mpls.h
@@ -3,4 +3,9 @@
#include <uapi/linux/mpls.h>
+#define MPLS_TTL_MASK (MPLS_LS_TTL_MASK >> MPLS_LS_TTL_SHIFT)
+#define MPLS_BOS_MASK (MPLS_LS_S_MASK >> MPLS_LS_S_SHIFT)
+#define MPLS_TC_MASK (MPLS_LS_TC_MASK >> MPLS_LS_TC_SHIFT)
+#define MPLS_LABEL_MASK (MPLS_LS_LABEL_MASK >> MPLS_LS_LABEL_SHIFT)
+
#endif /* _LINUX_MPLS_H */
diff --git a/include/linux/mtd/mtd.h b/include/linux/mtd/mtd.h
index eebdc63..79b176e 100644
--- a/include/linux/mtd/mtd.h
+++ b/include/linux/mtd/mtd.h
@@ -334,11 +334,6 @@
int (*_get_device) (struct mtd_info *mtd);
void (*_put_device) (struct mtd_info *mtd);
- /* Backing device capabilities for this device
- * - provides mmap capabilities
- */
- struct backing_dev_info *backing_dev_info;
-
struct notifier_block reboot_notifier; /* default mode before reboot */
/* ECC status information */
diff --git a/include/linux/net.h b/include/linux/net.h
index 0620f5e..abcfa46 100644
--- a/include/linux/net.h
+++ b/include/linux/net.h
@@ -298,6 +298,9 @@
int kernel_sock_ioctl(struct socket *sock, int cmd, unsigned long arg);
int kernel_sock_shutdown(struct socket *sock, enum sock_shutdown_cmd how);
+/* Routine returns the IP overhead imposed by a (caller-protected) socket. */
+u32 kernel_sock_ip_overhead(struct sock *sk);
+
#define MODULE_ALIAS_NETPROTO(proto) \
MODULE_ALIAS("net-pf-" __stringify(proto))
diff --git a/include/linux/netdev_features.h b/include/linux/netdev_features.h
index 9a04195..1d4737c 100644
--- a/include/linux/netdev_features.h
+++ b/include/linux/netdev_features.h
@@ -54,8 +54,9 @@
*/
NETIF_F_GSO_TUNNEL_REMCSUM_BIT, /* ... TUNNEL with TSO & REMCSUM */
NETIF_F_GSO_SCTP_BIT, /* ... SCTP fragmentation */
+ NETIF_F_GSO_ESP_BIT, /* ... ESP with TSO */
/**/NETIF_F_GSO_LAST = /* last bit, see GSO_MASK */
- NETIF_F_GSO_SCTP_BIT,
+ NETIF_F_GSO_ESP_BIT,
NETIF_F_FCOE_CRC_BIT, /* FCoE CRC32 */
NETIF_F_SCTP_CRC_BIT, /* SCTP checksum offload */
@@ -73,6 +74,8 @@
NETIF_F_HW_L2FW_DOFFLOAD_BIT, /* Allow L2 Forwarding in Hardware */
NETIF_F_HW_TC_BIT, /* Offload TC infrastructure */
+ NETIF_F_HW_ESP_BIT, /* Hardware ESP transformation offload */
+ NETIF_F_HW_ESP_TX_CSUM_BIT, /* ESP with TX checksum offload */
/*
* Add your fresh new feature above and remember to update
@@ -129,11 +132,14 @@
#define NETIF_F_GSO_PARTIAL __NETIF_F(GSO_PARTIAL)
#define NETIF_F_GSO_TUNNEL_REMCSUM __NETIF_F(GSO_TUNNEL_REMCSUM)
#define NETIF_F_GSO_SCTP __NETIF_F(GSO_SCTP)
+#define NETIF_F_GSO_ESP __NETIF_F(GSO_ESP)
#define NETIF_F_HW_VLAN_STAG_FILTER __NETIF_F(HW_VLAN_STAG_FILTER)
#define NETIF_F_HW_VLAN_STAG_RX __NETIF_F(HW_VLAN_STAG_RX)
#define NETIF_F_HW_VLAN_STAG_TX __NETIF_F(HW_VLAN_STAG_TX)
#define NETIF_F_HW_L2FW_DOFFLOAD __NETIF_F(HW_L2FW_DOFFLOAD)
#define NETIF_F_HW_TC __NETIF_F(HW_TC)
+#define NETIF_F_HW_ESP __NETIF_F(HW_ESP)
+#define NETIF_F_HW_ESP_TX_CSUM __NETIF_F(HW_ESP_TX_CSUM)
#define for_each_netdev_feature(mask_addr, bit) \
for_each_set_bit(bit, (unsigned long *)mask_addr, NETDEV_FEATURE_COUNT)
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 97456b25..9c23bd2 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -41,7 +41,6 @@
#include <linux/ethtool.h>
#include <net/net_namespace.h>
-#include <net/dsa.h>
#ifdef CONFIG_DCB
#include <net/dcbnl.h>
#endif
@@ -57,6 +56,8 @@
struct netpoll_info;
struct device;
struct phy_device;
+struct dsa_switch_tree;
+
/* 802.11 specific */
struct wireless_dev;
/* 802.15.4 specific */
@@ -236,8 +237,7 @@
netdev_hw_addr_list_for_each(ha, &(dev)->mc)
struct hh_cache {
- u16 hh_len;
- u16 __pad;
+ unsigned int hh_len;
seqlock_t hh_lock;
/* cached hardware header; allow for machine alignment needs. */
@@ -786,11 +786,11 @@
struct tc_to_netdev {
unsigned int type;
union {
- u8 tc;
struct tc_cls_u32_offload *cls_u32;
struct tc_cls_flower_offload *cls_flower;
struct tc_cls_matchall_offload *cls_mall;
struct tc_cls_bpf_offload *cls_bpf;
+ struct tc_mqprio_qopt *mqprio;
};
bool egress_dev;
};
@@ -813,16 +813,31 @@
XDP_QUERY_PROG,
};
+struct netlink_ext_ack;
+
struct netdev_xdp {
enum xdp_netdev_command command;
union {
/* XDP_SETUP_PROG */
- struct bpf_prog *prog;
+ struct {
+ struct bpf_prog *prog;
+ struct netlink_ext_ack *extack;
+ };
/* XDP_QUERY_PROG */
bool prog_attached;
};
};
+#ifdef CONFIG_XFRM_OFFLOAD
+struct xfrmdev_ops {
+ int (*xdo_dev_state_add) (struct xfrm_state *x);
+ void (*xdo_dev_state_delete) (struct xfrm_state *x);
+ void (*xdo_dev_state_free) (struct xfrm_state *x);
+ bool (*xdo_dev_offload_ok) (struct sk_buff *skb,
+ struct xfrm_state *x);
+};
+#endif
+
/*
* This structure defines the management hooks for network devices.
* The following hooks can be defined; unless noted otherwise, they are
@@ -1696,6 +1711,10 @@
const struct ndisc_ops *ndisc_ops;
#endif
+#ifdef CONFIG_XFRM
+ const struct xfrmdev_ops *xfrmdev_ops;
+#endif
+
const struct header_ops *header_ops;
unsigned int flags;
@@ -1715,7 +1734,7 @@
unsigned int max_mtu;
unsigned short type;
unsigned short hard_header_len;
- unsigned short min_header_len;
+ unsigned char min_header_len;
unsigned short needed_headroom;
unsigned short needed_tailroom;
@@ -1776,6 +1795,7 @@
unsigned int real_num_rx_queues;
#endif
+ struct bpf_prog __rcu *xdp_prog;
unsigned long gro_flush_timeout;
rx_handler_func_t __rcu *rx_handler;
void __rcu *rx_handler_data;
@@ -1894,6 +1914,13 @@
};
#define to_net_dev(d) container_of(d, struct net_device, dev)
+static inline bool netif_elide_gro(const struct net_device *dev)
+{
+ if (!(dev->features & NETIF_F_GRO) || dev->xdp_prog)
+ return true;
+ return false;
+}
+
#define NETDEV_ALIGN 32
static inline
@@ -2004,15 +2031,6 @@
write_pnet(&dev->nd_net, net);
}
-static inline bool netdev_uses_dsa(struct net_device *dev)
-{
-#if IS_ENABLED(CONFIG_NET_DSA)
- if (dev->dsa_ptr != NULL)
- return dsa_uses_tagged_protocol(dev->dsa_ptr);
-#endif
- return false;
-}
-
/**
* netdev_priv - access network device private data
* @dev: network device
@@ -3278,7 +3296,8 @@
int dev_get_phys_port_name(struct net_device *dev,
char *name, size_t len);
int dev_change_proto_down(struct net_device *dev, bool proto_down);
-int dev_change_xdp_fd(struct net_device *dev, int fd, u32 flags);
+int dev_change_xdp_fd(struct net_device *dev, struct netlink_ext_ack *extack,
+ int fd, u32 flags);
struct sk_buff *validate_xmit_skb_list(struct sk_buff *skb, struct net_device *dev);
struct sk_buff *dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
struct netdev_queue *txq, int *ret);
@@ -3305,6 +3324,7 @@
void dev_queue_xmit_nit(struct sk_buff *skb, struct net_device *dev);
extern int netdev_budget;
+extern unsigned int netdev_budget_usecs;
/* Called by rtnetlink.c:rtnl_unlock() */
void netdev_run_todo(void);
@@ -3394,10 +3414,10 @@
}
/**
- * netif_dormant - test if carrier present
+ * netif_dormant - test if device is dormant
* @dev: network device
*
- * Check if carrier is present on device
+ * Check if device is dormant.
*/
static inline bool netif_dormant(const struct net_device *dev)
{
@@ -4078,6 +4098,7 @@
BUILD_BUG_ON(SKB_GSO_PARTIAL != (NETIF_F_GSO_PARTIAL >> NETIF_F_GSO_SHIFT));
BUILD_BUG_ON(SKB_GSO_TUNNEL_REMCSUM != (NETIF_F_GSO_TUNNEL_REMCSUM >> NETIF_F_GSO_SHIFT));
BUILD_BUG_ON(SKB_GSO_SCTP != (NETIF_F_GSO_SCTP >> NETIF_F_GSO_SHIFT));
+ BUILD_BUG_ON(SKB_GSO_ESP != (NETIF_F_GSO_ESP >> NETIF_F_GSO_SHIFT));
return (features & feature) == feature;
}
@@ -4180,6 +4201,11 @@
return dev->priv_flags & IFF_OPENVSWITCH;
}
+static inline bool netif_is_ovs_port(const struct net_device *dev)
+{
+ return dev->priv_flags & IFF_OVS_DATAPATH;
+}
+
static inline bool netif_is_team_master(const struct net_device *dev)
{
return dev->priv_flags & IFF_TEAM;
diff --git a/include/linux/netfilter/nfnetlink.h b/include/linux/netfilter/nfnetlink.h
index 1b49209..996711d 100644
--- a/include/linux/netfilter/nfnetlink.h
+++ b/include/linux/netfilter/nfnetlink.h
@@ -41,6 +41,11 @@
int nfnetlink_unicast(struct sk_buff *skb, struct net *net, u32 portid,
int flags);
+static inline u16 nfnl_msg_type(u8 subsys, u8 msg_type)
+{
+ return subsys << 8 | msg_type;
+}
+
void nfnl_lock(__u8 subsys_id);
void nfnl_unlock(__u8 subsys_id);
#ifdef CONFIG_PROVE_LOCKING
diff --git a/include/linux/netfilter_bridge/ebtables.h b/include/linux/netfilter_bridge/ebtables.h
index 984b211..a30efb4 100644
--- a/include/linux/netfilter_bridge/ebtables.h
+++ b/include/linux/netfilter_bridge/ebtables.h
@@ -109,8 +109,10 @@
#define EBT_ALIGN(s) (((s) + (__alignof__(struct _xt_align)-1)) & \
~(__alignof__(struct _xt_align)-1))
extern struct ebt_table *ebt_register_table(struct net *net,
- const struct ebt_table *table);
-extern void ebt_unregister_table(struct net *net, struct ebt_table *table);
+ const struct ebt_table *table,
+ const struct nf_hook_ops *);
+extern void ebt_unregister_table(struct net *net, struct ebt_table *table,
+ const struct nf_hook_ops *);
extern unsigned int ebt_do_table(struct sk_buff *skb,
const struct nf_hook_state *state,
struct ebt_table *table);
diff --git a/include/linux/netlink.h b/include/linux/netlink.h
index da14ab6..5fff5ba 100644
--- a/include/linux/netlink.h
+++ b/include/linux/netlink.h
@@ -62,11 +62,47 @@
return __netlink_kernel_create(net, unit, THIS_MODULE, cfg);
}
+/* this can be increased when necessary - don't expose to userland */
+#define NETLINK_MAX_COOKIE_LEN 20
+
+/**
+ * struct netlink_ext_ack - netlink extended ACK report struct
+ * @_msg: message string to report - don't access directly, use
+ * %NL_SET_ERR_MSG
+ * @bad_attr: attribute with error
+ * @cookie: cookie data to return to userspace (for success)
+ * @cookie_len: actual cookie data length
+ */
+struct netlink_ext_ack {
+ const char *_msg;
+ const struct nlattr *bad_attr;
+ u8 cookie[NETLINK_MAX_COOKIE_LEN];
+ u8 cookie_len;
+};
+
+/* Always use this macro, this allows later putting the
+ * message into a separate section or such for things
+ * like translation or listing all possible messages.
+ * Currently string formatting is not supported (due
+ * to the lack of an output buffer.)
+ */
+#define NL_SET_ERR_MSG(extack, msg) do { \
+ static const char __msg[] = (msg); \
+ struct netlink_ext_ack *__extack = (extack); \
+ \
+ if (__extack) \
+ __extack->_msg = __msg; \
+} while (0)
+
+#define NL_SET_ERR_MSG_MOD(extack, msg) \
+ NL_SET_ERR_MSG((extack), KBUILD_MODNAME ": " msg)
+
extern void netlink_kernel_release(struct sock *sk);
extern int __netlink_change_ngroups(struct sock *sk, unsigned int groups);
extern int netlink_change_ngroups(struct sock *sk, unsigned int groups);
extern void __netlink_clear_multicast_users(struct sock *sk, unsigned int group);
-extern void netlink_ack(struct sk_buff *in_skb, struct nlmsghdr *nlh, int err);
+extern void netlink_ack(struct sk_buff *in_skb, struct nlmsghdr *nlh, int err,
+ const struct netlink_ext_ack *extack);
extern int netlink_has_listeners(struct sock *sk, unsigned int group);
extern int netlink_unicast(struct sock *ssk, struct sk_buff *skb, __u32 portid, int nonblock);
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index b34097c..e1502c5 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -133,7 +133,6 @@
struct rpc_clnt * client_acl; /* ACL RPC client handle */
struct nlm_host *nlm_host; /* NLM client handle */
struct nfs_iostats __percpu *io_stats; /* I/O statistics */
- struct backing_dev_info backing_dev_info;
atomic_long_t writeback; /* number of writeback pages */
int flags; /* various flags */
unsigned int caps; /* server capabilities */
diff --git a/include/linux/nvme-fc-driver.h b/include/linux/nvme-fc-driver.h
index f21471f..0db3715 100644
--- a/include/linux/nvme-fc-driver.h
+++ b/include/linux/nvme-fc-driver.h
@@ -137,9 +137,9 @@
* transferred. Should equal payload_length on success.
* @rcv_rsplen: length, in bytes, of the FCP RSP IU received.
* @status: Completion status of the FCP operation. must be 0 upon success,
- * NVME_SC_FC_xxx value upon failure. Note: this is NOT a
- * reflection of the NVME CQE completion status. Only the status
- * of the FCP operation at the NVME-FC level.
+ * negative errno value upon failure (ex: -EIO). Note: this is
+ * NOT a reflection of the NVME CQE completion status. Only the
+ * status of the FCP operation at the NVME-FC level.
*/
struct nvmefc_fcp_req {
void *cmdaddr;
@@ -533,9 +533,6 @@
* rsp as well
*/
NVMET_FCOP_RSP = 4, /* send rsp frame */
- NVMET_FCOP_ABORT = 5, /* abort exchange via ABTS */
- NVMET_FCOP_BA_ACC = 6, /* send BA_ACC */
- NVMET_FCOP_BA_RJT = 7, /* send BA_RJT */
};
/**
@@ -572,8 +569,6 @@
* upon compeletion of the operation. The nvmet-fc layer will also set a
* private pointer for its own use in the done routine.
*
- * Note: the LLDD must never fail a NVMET_FCOP_ABORT request !!
- *
* Values set by the NVMET-FC layer prior to calling the LLDD fcp_op
* entrypoint.
* @op: Indicates the FCP IU operation to perform (see NVMET_FCOP_xxx)
@@ -655,6 +650,22 @@
* on. The transport should pick a cpu to schedule the work
* on.
*/
+ NVMET_FCTGTFEAT_CMD_IN_ISR = (1 << 2),
+ /* Bit 2: When 0, the LLDD is calling the cmd rcv handler
+ * in a non-isr context, allowing the transport to finish
+ * op completion in the calling context. When 1, the LLDD
+ * is calling the cmd rcv handler in an ISR context,
+ * requiring the transport to transition to a workqueue
+ * for op completion.
+ */
+ NVMET_FCTGTFEAT_OPDONE_IN_ISR = (1 << 3),
+ /* Bit 3: When 0, the LLDD is calling the op done handler
+ * in a non-isr context, allowing the transport to finish
+ * op completion in the calling context. When 1, the LLDD
+ * is calling the op done handler in an ISR context,
+ * requiring the transport to transition to a workqueue
+ * for op completion.
+ */
};
@@ -725,12 +736,12 @@
* be freed/released.
* Entrypoint is Mandatory.
*
- * @fcp_op: Called to perform a data transfer, transmit a response, or
- * abort an FCP opertion. The nvmefc_tgt_fcp_req structure is the same
- * LLDD-supplied exchange structure specified in the
- * nvmet_fc_rcv_fcp_req() call made when the FCP CMD IU was received.
- * The op field in the structure shall indicate the operation for
- * the LLDD to perform relative to the io.
+ * @fcp_op: Called to perform a data transfer or transmit a response.
+ * The nvmefc_tgt_fcp_req structure is the same LLDD-supplied
+ * exchange structure specified in the nvmet_fc_rcv_fcp_req() call
+ * made when the FCP CMD IU was received. The op field in the
+ * structure shall indicate the operation for the LLDD to perform
+ * relative to the io.
* NVMET_FCOP_READDATA operation: the LLDD is to send the
* payload data (described by sglist) to the host in 1 or
* more FC sequences (preferrably 1). Note: the fc-nvme layer
@@ -752,29 +763,31 @@
* successfully, the LLDD is to update the nvmefc_tgt_fcp_req
* transferred_length field and may subsequently transmit the
* FCP_RSP iu payload (described by rspbuf, rspdma, rsplen).
- * The LLDD is to await FCP_CONF reception to confirm the RSP
- * reception by the host. The LLDD may retramsit the FCP_RSP iu
- * if necessary per FC-NVME. Upon reception of FCP_CONF, or upon
- * FCP_CONF failure, the LLDD is to set the nvmefc_tgt_fcp_req
- * fcp_error field and consider the operation complete..
- * NVMET_FCOP_RSP: the LLDD is to transmit the FCP_RSP iu payload
- * (described by rspbuf, rspdma, rsplen). The LLDD is to await
- * FCP_CONF reception to confirm the RSP reception by the host.
- * The LLDD may retramsit the FCP_RSP iu if necessary per FC-NVME.
- * Upon reception of FCP_CONF, or upon FCP_CONF failure, the
+ * If FCP_CONF is supported, the LLDD is to await FCP_CONF
+ * reception to confirm the RSP reception by the host. The LLDD
+ * may retramsit the FCP_RSP iu if necessary per FC-NVME. Upon
+ * transmission of the FCP_RSP iu if FCP_CONF is not supported,
+ * or upon success/failure of FCP_CONF if it is supported, the
* LLDD is to set the nvmefc_tgt_fcp_req fcp_error field and
- * consider the operation complete..
- * NVMET_FCOP_ABORT: the LLDD is to terminate the exchange
- * corresponding to the fcp operation. The LLDD shall send
- * ABTS and follow FC exchange abort-multi rules, including
- * ABTS retries and possible logout.
+ * consider the operation complete.
+ * NVMET_FCOP_RSP: the LLDD is to transmit the FCP_RSP iu payload
+ * (described by rspbuf, rspdma, rsplen). If FCP_CONF is
+ * supported, the LLDD is to await FCP_CONF reception to confirm
+ * the RSP reception by the host. The LLDD may retramsit the
+ * FCP_RSP iu if FCP_CONF is not received per FC-NVME. Upon
+ * transmission of the FCP_RSP iu if FCP_CONF is not supported,
+ * or upon success/failure of FCP_CONF if it is supported, the
+ * LLDD is to set the nvmefc_tgt_fcp_req fcp_error field and
+ * consider the operation complete.
* Upon completing the indicated operation, the LLDD is to set the
* status fields for the operation (tranferred_length and fcp_error
- * status) in the request, then all the "done" routine
- * indicated in the fcp request. Upon return from the "done"
- * routine for either a NVMET_FCOP_RSP or NVMET_FCOP_ABORT operation
- * the fc-nvme layer will not longer reference the fcp request,
- * allowing the LLDD to free/release the fcp request.
+ * status) in the request, then call the "done" routine
+ * indicated in the fcp request. After the operation completes,
+ * regardless of whether the FCP_RSP iu was successfully transmit,
+ * the LLDD-supplied exchange structure must remain valid until the
+ * transport calls the fcp_req_release() callback to return ownership
+ * of the exchange structure back to the LLDD so that it may be used
+ * for another fcp command.
* Note: when calling the done routine for READDATA or WRITEDATA
* operations, the fc-nvme layer may immediate convert, in the same
* thread and before returning to the LLDD, the fcp operation to
@@ -786,6 +799,22 @@
* Returns 0 on success, -<errno> on failure (Ex: -EIO)
* Entrypoint is Mandatory.
*
+ * @fcp_abort: Called by the transport to abort an active command.
+ * The command may be in-between operations (nothing active in LLDD)
+ * or may have an active WRITEDATA operation pending. The LLDD is to
+ * initiate the ABTS process for the command and return from the
+ * callback. The ABTS does not need to be complete on the command.
+ * The fcp_abort callback inherently cannot fail. After the
+ * fcp_abort() callback completes, the transport will wait for any
+ * outstanding operation (if there was one) to complete, then will
+ * call the fcp_req_release() callback to return the command's
+ * exchange context back to the LLDD.
+ *
+ * @fcp_req_release: Called by the transport to return a nvmefc_tgt_fcp_req
+ * to the LLDD after all operations on the fcp operation are complete.
+ * This may be due to the command completing or upon completion of
+ * abort cleanup.
+ *
* @max_hw_queues: indicates the maximum number of hw queues the LLDD
* supports for cpu affinitization.
* Value is Mandatory. Must be at least 1.
@@ -820,7 +849,11 @@
int (*xmt_ls_rsp)(struct nvmet_fc_target_port *tgtport,
struct nvmefc_tgt_ls_req *tls_req);
int (*fcp_op)(struct nvmet_fc_target_port *tgtport,
- struct nvmefc_tgt_fcp_req *);
+ struct nvmefc_tgt_fcp_req *fcpreq);
+ void (*fcp_abort)(struct nvmet_fc_target_port *tgtport,
+ struct nvmefc_tgt_fcp_req *fcpreq);
+ void (*fcp_req_release)(struct nvmet_fc_target_port *tgtport,
+ struct nvmefc_tgt_fcp_req *fcpreq);
u32 max_hw_queues;
u16 max_sgl_segments;
@@ -848,4 +881,7 @@
struct nvmefc_tgt_fcp_req *fcpreq,
void *cmdiubuf, u32 cmdiubuf_len);
+void nvmet_fc_rcv_fcp_abort(struct nvmet_fc_target_port *tgtport,
+ struct nvmefc_tgt_fcp_req *fcpreq);
+
#endif /* _NVME_FC_DRIVER_H */
diff --git a/include/linux/nvme-fc.h b/include/linux/nvme-fc.h
index 4b45226..e997c4a 100644
--- a/include/linux/nvme-fc.h
+++ b/include/linux/nvme-fc.h
@@ -16,8 +16,7 @@
*/
/*
- * This file contains definitions relative to FC-NVME r1.11 and a few
- * newer items
+ * This file contains definitions relative to FC-NVME r1.14 (16-020vB).
*/
#ifndef _NVME_FC_H
@@ -47,8 +46,15 @@
#define NVME_FC_SIZEOF_ZEROS_RSP 12
+enum {
+ FCNVME_SC_SUCCESS = 0,
+ FCNVME_SC_INVALID_FIELD = 1,
+ FCNVME_SC_INVALID_CONNID = 2,
+};
+
struct nvme_fc_ersp_iu {
- __u8 rsvd0[2];
+ __u8 status_code;
+ __u8 rsvd1;
__be16 iu_len;
__be32 rsn;
__be32 xfrd_len;
@@ -58,7 +64,7 @@
};
-/* FC-NVME r1.03/16-119v0 NVME Link Services */
+/* FC-NVME Link Services */
enum {
FCNVME_LS_RSVD = 0,
FCNVME_LS_RJT = 1,
@@ -68,7 +74,7 @@
FCNVME_LS_DISCONNECT = 5,
};
-/* FC-NVME r1.03/16-119v0 NVME Link Service Descriptors */
+/* FC-NVME Link Service Descriptors */
enum {
FCNVME_LSDESC_RSVD = 0x0,
FCNVME_LSDESC_RQST = 0x1,
@@ -92,7 +98,6 @@
return cpu_to_be32(sz - (2 * sizeof(u32)));
}
-
struct fcnvme_ls_rqst_w0 {
u8 ls_cmd; /* FCNVME_LS_xxx */
u8 zeros[3];
@@ -106,8 +111,53 @@
__be32 rsvd12;
};
+/* FC-NVME LS RJT reason_code values */
+enum fcnvme_ls_rjt_reason {
+ FCNVME_RJT_RC_NONE = 0,
+ /* no reason - not to be sent */
+ FCNVME_RJT_RC_INVAL = 0x01,
+ /* invalid NVMe_LS command code */
+ FCNVME_RJT_RC_LOGIC = 0x03,
+ /* logical error */
+
+ FCNVME_RJT_RC_UNAB = 0x09,
+ /* unable to perform command request */
+
+ FCNVME_RJT_RC_UNSUP = 0x0b,
+ /* command not supported */
+
+ FCNVME_RJT_RC_INPROG = 0x0e,
+ /* command already in progress */
+
+ FCNVME_RJT_RC_INV_ASSOC = 0x40,
+ /* Invalid Association ID*/
+
+ FCNVME_RJT_RC_INV_CONN = 0x41,
+ /* Invalid Connection ID*/
+
+ FCNVME_RJT_RC_VENDOR = 0xff,
+ /* vendor specific error */
+};
+
+/* FC-NVME LS RJT reason_explanation values */
+enum fcnvme_ls_rjt_explan {
+ FCNVME_RJT_EXP_NONE = 0x00,
+ /* No additional explanation */
+
+ FCNVME_RJT_EXP_OXID_RXID = 0x17,
+ /* invalid OX_ID-RX_ID combination */
+
+ FCNVME_RJT_EXP_INSUF_RES = 0x29,
+ /* insufficient resources */
+
+ FCNVME_RJT_EXP_UNAB_DATA = 0x2a,
+ /* unable to supply requested data */
+
+ FCNVME_RJT_EXP_INV_LEN = 0x2d,
+ /* Invalid payload length */
+};
/* FCNVME_LSDESC_RJT */
struct fcnvme_lsdesc_rjt {
@@ -119,15 +169,15 @@
* Reject reason and explanaction codes are generic
* to ELs's from LS-3.
*/
- u8 reason_code;
- u8 reason_explanation;
+ u8 reason_code; /* fcnvme_ls_rjt_reason */
+ u8 reason_explanation; /* fcnvme_ls_rjt_explan */
u8 vendor;
__be32 rsvd12;
};
-#define FCNVME_ASSOC_HOSTID_LEN 64
+#define FCNVME_ASSOC_HOSTID_LEN 16
#define FCNVME_ASSOC_HOSTNQN_LEN 256
#define FCNVME_ASSOC_SUBNQN_LEN 256
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 9061780..b625bac 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -245,6 +245,7 @@
NVME_CTRL_ONCS_WRITE_ZEROES = 1 << 3,
NVME_CTRL_VWC_PRESENT = 1 << 0,
NVME_CTRL_OACS_SEC_SUPP = 1 << 0,
+ NVME_CTRL_OACS_DBBUF_SUPP = 1 << 7,
};
struct nvme_lbaf {
@@ -603,6 +604,7 @@
nvme_admin_download_fw = 0x11,
nvme_admin_ns_attach = 0x15,
nvme_admin_keep_alive = 0x18,
+ nvme_admin_dbbuf = 0x7C,
nvme_admin_format_nvm = 0x80,
nvme_admin_security_send = 0x81,
nvme_admin_security_recv = 0x82,
@@ -874,6 +876,16 @@
__u8 resv4[16];
};
+struct nvme_dbbuf {
+ __u8 opcode;
+ __u8 flags;
+ __u16 command_id;
+ __u32 rsvd1[5];
+ __le64 prp1;
+ __le64 prp2;
+ __u32 rsvd12[6];
+};
+
struct nvme_command {
union {
struct nvme_common_command common;
@@ -893,6 +905,7 @@
struct nvmf_connect_command connect;
struct nvmf_property_set_command prop_set;
struct nvmf_property_get_command prop_get;
+ struct nvme_dbbuf dbbuf;
};
};
diff --git a/include/linux/of.h b/include/linux/of.h
index 21e6323..e5d4225f 100644
--- a/include/linux/of.h
+++ b/include/linux/of.h
@@ -159,6 +159,8 @@
container_of(fwnode, struct device_node, fwnode) : NULL;
}
+#define of_fwnode_handle(node) (&(node)->fwnode)
+
static inline bool of_have_populated_dt(void)
{
return of_root != NULL;
@@ -602,6 +604,8 @@
return NULL;
}
+#define of_fwnode_handle(node) NULL
+
static inline bool of_have_populated_dt(void)
{
return false;
diff --git a/include/linux/of_mdio.h b/include/linux/of_mdio.h
index a58cca8..ba35ba5 100644
--- a/include/linux/of_mdio.h
+++ b/include/linux/of_mdio.h
@@ -12,7 +12,7 @@
#include <linux/phy.h>
#include <linux/of.h>
-#ifdef CONFIG_OF
+#if IS_ENABLED(CONFIG_OF_MDIO)
extern int of_mdiobus_register(struct mii_bus *mdio, struct device_node *np);
extern struct phy_device *of_phy_find_device(struct device_node *phy_np);
extern struct phy_device *of_phy_connect(struct net_device *dev,
@@ -32,7 +32,7 @@
extern void of_phy_deregister_fixed_link(struct device_node *np);
extern bool of_phy_is_fixed_link(struct device_node *np);
-#else /* CONFIG_OF */
+#else /* CONFIG_OF_MDIO */
static inline int of_mdiobus_register(struct mii_bus *mdio, struct device_node *np)
{
/*
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 84943e8..316a19f 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -148,7 +148,7 @@
#ifdef CONFIG_TINY_RCU
# ifdef CONFIG_PREEMPT_COUNT
- VM_BUG_ON(!in_atomic());
+ VM_BUG_ON(!in_atomic() && !irqs_disabled());
# endif
/*
* Preempt must be disabled here - we rely on rcu_read_lock doing
@@ -186,7 +186,7 @@
#if !defined(CONFIG_SMP) && defined(CONFIG_TREE_RCU)
# ifdef CONFIG_PREEMPT_COUNT
- VM_BUG_ON(!in_atomic());
+ VM_BUG_ON(!in_atomic() && !irqs_disabled());
# endif
VM_BUG_ON_PAGE(page_count(page) == 0, page);
page_ref_add(page, count);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index eb3da1a..82dec36 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1300,7 +1300,6 @@
void pci_msi_shutdown(struct pci_dev *dev);
void pci_disable_msi(struct pci_dev *dev);
int pci_msix_vec_count(struct pci_dev *dev);
-int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec);
void pci_msix_shutdown(struct pci_dev *dev);
void pci_disable_msix(struct pci_dev *dev);
void pci_restore_msi_state(struct pci_dev *dev);
@@ -1330,9 +1329,6 @@
static inline void pci_msi_shutdown(struct pci_dev *dev) { }
static inline void pci_disable_msi(struct pci_dev *dev) { }
static inline int pci_msix_vec_count(struct pci_dev *dev) { return -ENOSYS; }
-static inline int pci_enable_msix(struct pci_dev *dev,
- struct msix_entry *entries, int nvec)
-{ return -ENOSYS; }
static inline void pci_msix_shutdown(struct pci_dev *dev) { }
static inline void pci_disable_msix(struct pci_dev *dev) { }
static inline void pci_restore_msi_state(struct pci_dev *dev) { }
diff --git a/include/linux/percpu.h b/include/linux/percpu.h
index 56939d3..491b3f5 100644
--- a/include/linux/percpu.h
+++ b/include/linux/percpu.h
@@ -110,6 +110,7 @@
#endif
extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align);
+extern bool __is_kernel_percpu_address(unsigned long addr, unsigned long *can_addr);
extern bool is_kernel_percpu_address(unsigned long addr);
#if !defined(CONFIG_SMP) || !defined(CONFIG_HAVE_SETUP_PER_CPU_AREA)
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 000fdb2..24a6358 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -165,6 +165,13 @@
struct list_head bp_list;
};
#endif
+ struct { /* amd_iommu */
+ u8 iommu_bank;
+ u8 iommu_cntr;
+ u16 padding;
+ u64 conf;
+ u64 conf1;
+ };
};
/*
* If the event is a per task event, this will point to the task in
@@ -801,6 +808,7 @@
struct ring_buffer *rb;
unsigned long wakeup;
unsigned long size;
+ u64 aux_flags;
union {
void *addr;
unsigned long head;
@@ -849,10 +857,11 @@
extern void *perf_aux_output_begin(struct perf_output_handle *handle,
struct perf_event *event);
extern void perf_aux_output_end(struct perf_output_handle *handle,
- unsigned long size, bool truncated);
+ unsigned long size);
extern int perf_aux_output_skip(struct perf_output_handle *handle,
unsigned long size);
extern void *perf_get_aux(struct perf_output_handle *handle);
+extern void perf_aux_output_flag(struct perf_output_handle *handle, u64 flags);
extern int perf_pmu_register(struct pmu *pmu, const char *name, int type);
extern void perf_pmu_unregister(struct pmu *pmu);
@@ -1112,6 +1121,7 @@
extern void perf_event_exec(void);
extern void perf_event_comm(struct task_struct *tsk, bool exec);
+extern void perf_event_namespaces(struct task_struct *tsk);
extern void perf_event_fork(struct task_struct *tsk);
/* Callchains */
@@ -1267,8 +1277,8 @@
perf_aux_output_begin(struct perf_output_handle *handle,
struct perf_event *event) { return NULL; }
static inline void
-perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
- bool truncated) { }
+perf_aux_output_end(struct perf_output_handle *handle, unsigned long size)
+ { }
static inline int
perf_aux_output_skip(struct perf_output_handle *handle,
unsigned long size) { return -EINVAL; }
@@ -1315,6 +1325,7 @@
static inline void perf_event_mmap(struct vm_area_struct *vma) { }
static inline void perf_event_exec(void) { }
static inline void perf_event_comm(struct task_struct *tsk, bool exec) { }
+static inline void perf_event_namespaces(struct task_struct *tsk) { }
static inline void perf_event_fork(struct task_struct *tsk) { }
static inline void perf_event_init(void) { }
static inline int perf_swevent_get_recursion_context(void) { return -1; }
diff --git a/include/linux/phy.h b/include/linux/phy.h
index 43a7748..e76e4ad 100644
--- a/include/linux/phy.h
+++ b/include/linux/phy.h
@@ -217,6 +217,13 @@
* matching its address
*/
int irq[PHY_MAX_ADDR];
+
+ /* GPIO reset pulse width in microseconds */
+ int reset_delay_us;
+ /* Number of reset GPIOs */
+ int num_reset_gpios;
+ /* Array of RESET GPIO descriptors */
+ struct gpio_desc **reset_gpiod;
};
#define to_mii_bus(d) container_of(d, struct mii_bus, dev)
@@ -587,23 +594,29 @@
*/
void (*link_change_notify)(struct phy_device *dev);
- /* A function provided by a phy specific driver to override the
- * the PHY driver framework support for reading a MMD register
- * from the PHY. If not supported, return -1. This function is
- * optional for PHY specific drivers, if not provided then the
- * default MMD read function is used by the PHY framework.
+ /*
+ * Phy specific driver override for reading a MMD register.
+ * This function is optional for PHY specific drivers. When
+ * not provided, the default MMD read function will be used
+ * by phy_read_mmd(), which will use either a direct read for
+ * Clause 45 PHYs or an indirect read for Clause 22 PHYs.
+ * devnum is the MMD device number within the PHY device,
+ * regnum is the register within the selected MMD device.
*/
- int (*read_mmd_indirect)(struct phy_device *dev, int ptrad,
- int devnum, int regnum);
+ int (*read_mmd)(struct phy_device *dev, int devnum, u16 regnum);
- /* A function provided by a phy specific driver to override the
- * the PHY driver framework support for writing a MMD register
- * from the PHY. This function is optional for PHY specific drivers,
- * if not provided then the default MMD read function is used by
- * the PHY framework.
+ /*
+ * Phy specific driver override for writing a MMD register.
+ * This function is optional for PHY specific drivers. When
+ * not provided, the default MMD write function will be used
+ * by phy_write_mmd(), which will use either a direct write for
+ * Clause 45 PHYs, or an indirect write for Clause 22 PHYs.
+ * devnum is the MMD device number within the PHY device,
+ * regnum is the register within the selected MMD device.
+ * val is the value to be written.
*/
- void (*write_mmd_indirect)(struct phy_device *dev, int ptrad,
- int devnum, int regnum, u32 val);
+ int (*write_mmd)(struct phy_device *dev, int devnum, u16 regnum,
+ u16 val);
/* Get the size and type of the eeprom contained within a plug-in
* module */
@@ -651,25 +664,7 @@
*
* Same rules as for phy_read();
*/
-static inline int phy_read_mmd(struct phy_device *phydev, int devad, u32 regnum)
-{
- if (!phydev->is_c45)
- return -EOPNOTSUPP;
-
- return mdiobus_read(phydev->mdio.bus, phydev->mdio.addr,
- MII_ADDR_C45 | (devad << 16) | (regnum & 0xffff));
-}
-
-/**
- * phy_read_mmd_indirect - reads data from the MMD registers
- * @phydev: The PHY device bus
- * @prtad: MMD Address
- * @addr: PHY address on the MII bus
- *
- * Description: it reads data from the MMD registers (clause 22 to access to
- * clause 45) of the specified phy address.
- */
-int phy_read_mmd_indirect(struct phy_device *phydev, int prtad, int devad);
+int phy_read_mmd(struct phy_device *phydev, int devad, u32 regnum);
/**
* phy_read - Convenience function for reading a given PHY register
@@ -752,35 +747,29 @@
*
* Same rules as for phy_write();
*/
-static inline int phy_write_mmd(struct phy_device *phydev, int devad,
- u32 regnum, u16 val)
-{
- if (!phydev->is_c45)
- return -EOPNOTSUPP;
-
- regnum = MII_ADDR_C45 | ((devad & 0x1f) << 16) | (regnum & 0xffff);
-
- return mdiobus_write(phydev->mdio.bus, phydev->mdio.addr, regnum, val);
-}
-
-/**
- * phy_write_mmd_indirect - writes data to the MMD registers
- * @phydev: The PHY device
- * @prtad: MMD Address
- * @devad: MMD DEVAD
- * @data: data to write in the MMD register
- *
- * Description: Write data from the MMD registers of the specified
- * phy address.
- */
-void phy_write_mmd_indirect(struct phy_device *phydev, int prtad,
- int devad, u32 data);
+int phy_write_mmd(struct phy_device *phydev, int devad, u32 regnum, u16 val);
struct phy_device *phy_device_create(struct mii_bus *bus, int addr, int phy_id,
bool is_c45,
struct phy_c45_device_ids *c45_ids);
+#if IS_ENABLED(CONFIG_PHYLIB)
struct phy_device *get_phy_device(struct mii_bus *bus, int addr, bool is_c45);
int phy_device_register(struct phy_device *phy);
+void phy_device_free(struct phy_device *phydev);
+#else
+static inline
+struct phy_device *get_phy_device(struct mii_bus *bus, int addr, bool is_c45)
+{
+ return NULL;
+}
+
+static inline int phy_device_register(struct phy_device *phy)
+{
+ return 0;
+}
+
+static inline void phy_device_free(struct phy_device *phydev) { }
+#endif /* CONFIG_PHYLIB */
void phy_device_remove(struct phy_device *phydev);
int phy_init_hw(struct phy_device *phydev);
int phy_suspend(struct phy_device *phydev);
@@ -852,6 +841,7 @@
void phy_mac_interrupt(struct phy_device *phydev, int new_link);
void phy_start_machine(struct phy_device *phydev);
void phy_stop_machine(struct phy_device *phydev);
+void phy_trigger_machine(struct phy_device *phydev, bool sync);
int phy_ethtool_sset(struct phy_device *phydev, struct ethtool_cmd *cmd);
int phy_ethtool_gset(struct phy_device *phydev, struct ethtool_cmd *cmd);
int phy_ethtool_ksettings_get(struct phy_device *phydev,
@@ -861,7 +851,6 @@
int phy_mii_ioctl(struct phy_device *phydev, struct ifreq *ifr, int cmd);
int phy_start_interrupts(struct phy_device *phydev);
void phy_print_status(struct phy_device *phydev);
-void phy_device_free(struct phy_device *phydev);
int phy_set_max_speed(struct phy_device *phydev, u32 max_speed);
int phy_register_fixup(const char *bus_id, u32 phy_uid, u32 phy_uid_mask,
@@ -888,8 +877,10 @@
const struct ethtool_link_ksettings *cmd);
int phy_ethtool_nway_reset(struct net_device *ndev);
+#if IS_ENABLED(CONFIG_PHYLIB)
int __init mdio_bus_init(void);
void mdio_bus_exit(void);
+#endif
extern struct bus_type mdio_bus_type;
@@ -900,7 +891,7 @@
const void *platform_data;
};
-#if IS_ENABLED(CONFIG_PHYLIB)
+#if IS_ENABLED(CONFIG_MDIO_DEVICE)
int mdiobus_register_board_info(const struct mdio_board_info *info,
unsigned int n);
#else
diff --git a/include/linux/pinctrl/pinconf-generic.h b/include/linux/pinctrl/pinconf-generic.h
index 7620eb1..279e3c5 100644
--- a/include/linux/pinctrl/pinconf-generic.h
+++ b/include/linux/pinctrl/pinconf-generic.h
@@ -42,6 +42,8 @@
* @PIN_CONFIG_BIAS_PULL_UP: the pin will be pulled up (usually with high
* impedance to VDD). If the argument is != 0 pull-up is enabled,
* if it is 0, pull-up is total, i.e. the pin is connected to VDD.
+ * @PIN_CONFIG_BIDIRECTIONAL: the pin will be configured to allow simultaneous
+ * input and output operations.
* @PIN_CONFIG_DRIVE_OPEN_DRAIN: the pin will be driven with open drain (open
* collector) which means it is usually wired with other output ports
* which are then pulled up with an external resistor. Setting this
@@ -96,6 +98,7 @@
PIN_CONFIG_BIAS_PULL_DOWN,
PIN_CONFIG_BIAS_PULL_PIN_DEFAULT,
PIN_CONFIG_BIAS_PULL_UP,
+ PIN_CONFIG_BIDIRECTIONAL,
PIN_CONFIG_DRIVE_OPEN_DRAIN,
PIN_CONFIG_DRIVE_OPEN_SOURCE,
PIN_CONFIG_DRIVE_PUSH_PULL,
diff --git a/include/linux/platform_data/pn544.h b/include/linux/platform_data/pn544.h
deleted file mode 100644
index 5ce1ab9..0000000
--- a/include/linux/platform_data/pn544.h
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Driver include for the PN544 NFC chip.
- *
- * Copyright (C) Nokia Corporation
- *
- * Author: Jari Vanhala <ext-jari.vanhala@nokia.com>
- * Contact: Matti Aaltoenn <matti.j.aaltonen@nokia.com>
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-#ifndef _PN544_H_
-#define _PN544_H_
-
-#include <linux/i2c.h>
-
-enum {
- NFC_GPIO_ENABLE,
- NFC_GPIO_FW_RESET,
- NFC_GPIO_IRQ
-};
-
-/* board config */
-struct pn544_nfc_platform_data {
- int (*request_resources) (struct i2c_client *client);
- void (*free_resources) (void);
- void (*enable) (int fw);
- int (*test) (void);
- void (*disable) (void);
- int (*get_gpio)(int type);
-};
-
-#endif /* _PN544_H_ */
diff --git a/include/linux/platform_data/st21nfca.h b/include/linux/platform_data/st21nfca.h
deleted file mode 100644
index cc2bdaf..0000000
--- a/include/linux/platform_data/st21nfca.h
+++ /dev/null
@@ -1,33 +0,0 @@
-/*
- * Driver include for the ST21NFCA NFC chip.
- *
- * Copyright (C) 2014 STMicroelectronics SAS. All rights reserved.
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms and conditions of the GNU General Public License,
- * version 2, as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, see <http://www.gnu.org/licenses/>.
- */
-
-#ifndef _ST21NFCA_HCI_H_
-#define _ST21NFCA_HCI_H_
-
-#include <linux/i2c.h>
-
-#define ST21NFCA_HCI_DRIVER_NAME "st21nfca_hci"
-
-struct st21nfca_nfc_platform_data {
- unsigned int gpio_ena;
- unsigned int irq_polarity;
- bool is_ese_present;
- bool is_uicc_present;
-};
-
-#endif /* _ST21NFCA_HCI_H_ */
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index 5339ed5..9b6abe6 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -20,6 +20,7 @@
/* Defines used for the flags field in the struct generic_pm_domain */
#define GENPD_FLAG_PM_CLK (1U << 0) /* PM domain uses PM clk */
#define GENPD_FLAG_IRQ_SAFE (1U << 1) /* PM domain operates in atomic */
+#define GENPD_FLAG_ALWAYS_ON (1U << 2) /* PM domain is always powered on */
enum gpd_status {
GPD_STATE_ACTIVE = 0, /* PM domain is active */
diff --git a/include/linux/poll.h b/include/linux/poll.h
index a46d675..75ffc57 100644
--- a/include/linux/poll.h
+++ b/include/linux/poll.h
@@ -98,64 +98,8 @@
ktime_t *expires, unsigned long slack);
extern u64 select_estimate_accuracy(struct timespec64 *tv);
-
-static inline int poll_schedule(struct poll_wqueues *pwq, int state)
-{
- return poll_schedule_timeout(pwq, state, NULL, 0);
-}
-
-/*
- * Scalable version of the fd_set.
- */
-
-typedef struct {
- unsigned long *in, *out, *ex;
- unsigned long *res_in, *res_out, *res_ex;
-} fd_set_bits;
-
-/*
- * How many longwords for "nr" bits?
- */
-#define FDS_BITPERLONG (8*sizeof(long))
-#define FDS_LONGS(nr) (((nr)+FDS_BITPERLONG-1)/FDS_BITPERLONG)
-#define FDS_BYTES(nr) (FDS_LONGS(nr)*sizeof(long))
-
-/*
- * We do a VERIFY_WRITE here even though we are only reading this time:
- * we'll write to it eventually..
- *
- * Use "unsigned long" accesses to let user-mode fd_set's be long-aligned.
- */
-static inline
-int get_fd_set(unsigned long nr, void __user *ufdset, unsigned long *fdset)
-{
- nr = FDS_BYTES(nr);
- if (ufdset)
- return copy_from_user(fdset, ufdset, nr) ? -EFAULT : 0;
-
- memset(fdset, 0, nr);
- return 0;
-}
-
-static inline unsigned long __must_check
-set_fd_set(unsigned long nr, void __user *ufdset, unsigned long *fdset)
-{
- if (ufdset)
- return __copy_to_user(ufdset, fdset, FDS_BYTES(nr));
- return 0;
-}
-
-static inline
-void zero_fd_set(unsigned long nr, unsigned long *fdset)
-{
- memset(fdset, 0, FDS_BYTES(nr));
-}
-
#define MAX_INT64_SECONDS (((s64)(~((u64)0)>>1)/HZ)-1)
-extern int do_select(int n, fd_set_bits *fds, struct timespec64 *end_time);
-extern int do_sys_poll(struct pollfd __user * ufds, unsigned int nfds,
- struct timespec64 *end_time);
extern int core_sys_select(int n, fd_set __user *inp, fd_set __user *outp,
fd_set __user *exp, struct timespec64 *end_time);
diff --git a/include/linux/posix-clock.h b/include/linux/posix-clock.h
index 34c4498..83b22ae 100644
--- a/include/linux/posix-clock.h
+++ b/include/linux/posix-clock.h
@@ -59,23 +59,23 @@
int (*clock_adjtime)(struct posix_clock *pc, struct timex *tx);
- int (*clock_gettime)(struct posix_clock *pc, struct timespec *ts);
+ int (*clock_gettime)(struct posix_clock *pc, struct timespec64 *ts);
- int (*clock_getres) (struct posix_clock *pc, struct timespec *ts);
+ int (*clock_getres) (struct posix_clock *pc, struct timespec64 *ts);
int (*clock_settime)(struct posix_clock *pc,
- const struct timespec *ts);
+ const struct timespec64 *ts);
int (*timer_create) (struct posix_clock *pc, struct k_itimer *kit);
int (*timer_delete) (struct posix_clock *pc, struct k_itimer *kit);
void (*timer_gettime)(struct posix_clock *pc,
- struct k_itimer *kit, struct itimerspec *tsp);
+ struct k_itimer *kit, struct itimerspec64 *tsp);
int (*timer_settime)(struct posix_clock *pc,
struct k_itimer *kit, int flags,
- struct itimerspec *tsp, struct itimerspec *old);
+ struct itimerspec64 *tsp, struct itimerspec64 *old);
/*
* Optional character device methods:
*/
diff --git a/include/linux/posix-timers.h b/include/linux/posix-timers.h
index 64aa189..8c1e43a 100644
--- a/include/linux/posix-timers.h
+++ b/include/linux/posix-timers.h
@@ -87,22 +87,22 @@
};
struct k_clock {
- int (*clock_getres) (const clockid_t which_clock, struct timespec *tp);
+ int (*clock_getres) (const clockid_t which_clock, struct timespec64 *tp);
int (*clock_set) (const clockid_t which_clock,
- const struct timespec *tp);
- int (*clock_get) (const clockid_t which_clock, struct timespec * tp);
+ const struct timespec64 *tp);
+ int (*clock_get) (const clockid_t which_clock, struct timespec64 *tp);
int (*clock_adj) (const clockid_t which_clock, struct timex *tx);
int (*timer_create) (struct k_itimer *timer);
int (*nsleep) (const clockid_t which_clock, int flags,
- struct timespec *, struct timespec __user *);
+ struct timespec64 *, struct timespec __user *);
long (*nsleep_restart) (struct restart_block *restart_block);
- int (*timer_set) (struct k_itimer * timr, int flags,
- struct itimerspec * new_setting,
- struct itimerspec * old_setting);
- int (*timer_del) (struct k_itimer * timr);
+ int (*timer_set) (struct k_itimer *timr, int flags,
+ struct itimerspec64 *new_setting,
+ struct itimerspec64 *old_setting);
+ int (*timer_del) (struct k_itimer *timr);
#define TIMER_RETRY 1
- void (*timer_get) (struct k_itimer * timr,
- struct itimerspec * cur_setting);
+ void (*timer_get) (struct k_itimer *timr,
+ struct itimerspec64 *cur_setting);
};
extern struct k_clock clock_posix_cpu;
diff --git a/include/linux/power/bq24190_charger.h b/include/linux/power/bq24190_charger.h
deleted file mode 100644
index 9f02837..0000000
--- a/include/linux/power/bq24190_charger.h
+++ /dev/null
@@ -1,16 +0,0 @@
-/*
- * Platform data for the TI bq24190 battery charger driver.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
-
-#ifndef _BQ24190_CHARGER_H_
-#define _BQ24190_CHARGER_H_
-
-struct bq24190_platform_data {
- unsigned int gpio_int; /* GPIO pin that's connected to INT# */
-};
-
-#endif
diff --git a/include/linux/property.h b/include/linux/property.h
index 64e3a9c..2f48261 100644
--- a/include/linux/property.h
+++ b/include/linux/property.h
@@ -33,6 +33,8 @@
DEV_DMA_COHERENT,
};
+struct fwnode_handle *dev_fwnode(struct device *dev);
+
bool device_property_present(struct device *dev, const char *propname);
int device_property_read_u8_array(struct device *dev, const char *propname,
u8 *val, size_t nval);
@@ -70,6 +72,15 @@
int fwnode_property_match_string(struct fwnode_handle *fwnode,
const char *propname, const char *string);
+struct fwnode_handle *fwnode_get_parent(struct fwnode_handle *fwnode);
+struct fwnode_handle *fwnode_get_next_parent(struct fwnode_handle *fwnode);
+struct fwnode_handle *fwnode_get_next_child_node(struct fwnode_handle *fwnode,
+ struct fwnode_handle *child);
+
+#define fwnode_for_each_child_node(fwnode, child) \
+ for (child = fwnode_get_next_child_node(fwnode, NULL); child; \
+ child = fwnode_get_next_child_node(fwnode, child))
+
struct fwnode_handle *device_get_next_child_node(struct device *dev,
struct fwnode_handle *child);
@@ -77,9 +88,12 @@
for (child = device_get_next_child_node(dev, NULL); child; \
child = device_get_next_child_node(dev, child))
+struct fwnode_handle *fwnode_get_named_child_node(struct fwnode_handle *fwnode,
+ const char *childname);
struct fwnode_handle *device_get_named_child_node(struct device *dev,
const char *childname);
+void fwnode_handle_get(struct fwnode_handle *fwnode);
void fwnode_handle_put(struct fwnode_handle *fwnode);
unsigned int device_get_child_node_count(struct device *dev);
@@ -258,4 +272,16 @@
void *device_get_mac_address(struct device *dev, char *addr, int alen);
+struct fwnode_handle *fwnode_graph_get_next_endpoint(
+ struct fwnode_handle *fwnode, struct fwnode_handle *prev);
+struct fwnode_handle *fwnode_graph_get_remote_port_parent(
+ struct fwnode_handle *fwnode);
+struct fwnode_handle *fwnode_graph_get_remote_port(
+ struct fwnode_handle *fwnode);
+struct fwnode_handle *fwnode_graph_get_remote_endpoint(
+ struct fwnode_handle *fwnode);
+
+int fwnode_graph_parse_endpoint(struct fwnode_handle *fwnode,
+ struct fwnode_endpoint *endpoint);
+
#endif /* _LINUX_PROPERTY_H_ */
diff --git a/include/linux/pstore.h b/include/linux/pstore.h
index 0da29ca..e2233f5 100644
--- a/include/linux/pstore.h
+++ b/include/linux/pstore.h
@@ -30,7 +30,9 @@
#include <linux/time.h>
#include <linux/types.h>
-/* types */
+struct module;
+
+/* pstore record types (see fs/pstore/inode.c for filename templates) */
enum pstore_type_id {
PSTORE_TYPE_DMESG = 0,
PSTORE_TYPE_MCE = 1,
@@ -45,42 +47,146 @@
PSTORE_TYPE_UNKNOWN = 255
};
-struct module;
+struct pstore_info;
+/**
+ * struct pstore_record - details of a pstore record entry
+ * @psi: pstore backend driver information
+ * @type: pstore record type
+ * @id: per-type unique identifier for record
+ * @time: timestamp of the record
+ * @buf: pointer to record contents
+ * @size: size of @buf
+ * @ecc_notice_size:
+ * ECC information for @buf
+ *
+ * Valid for PSTORE_TYPE_DMESG @type:
+ *
+ * @count: Oops count since boot
+ * @reason: kdump reason for notification
+ * @part: position in a multipart record
+ * @compressed: whether the buffer is compressed
+ *
+ */
+struct pstore_record {
+ struct pstore_info *psi;
+ enum pstore_type_id type;
+ u64 id;
+ struct timespec time;
+ char *buf;
+ ssize_t size;
+ ssize_t ecc_notice_size;
+ int count;
+ enum kmsg_dump_reason reason;
+ unsigned int part;
+ bool compressed;
+};
+
+/**
+ * struct pstore_info - backend pstore driver structure
+ *
+ * @owner: module which is repsonsible for this backend driver
+ * @name: name of the backend driver
+ *
+ * @buf_lock: spinlock to serialize access to @buf
+ * @buf: preallocated crash dump buffer
+ * @bufsize: size of @buf available for crash dump writes
+ *
+ * @read_mutex: serializes @open, @read, @close, and @erase callbacks
+ * @flags: bitfield of frontends the backend can accept writes for
+ * @data: backend-private pointer passed back during callbacks
+ *
+ * Callbacks:
+ *
+ * @open:
+ * Notify backend that pstore is starting a full read of backend
+ * records. Followed by one or more @read calls, and a final @close.
+ *
+ * @psi: in: pointer to the struct pstore_info for the backend
+ *
+ * Returns 0 on success, and non-zero on error.
+ *
+ * @close:
+ * Notify backend that pstore has finished a full read of backend
+ * records. Always preceded by an @open call and one or more @read
+ * calls.
+ *
+ * @psi: in: pointer to the struct pstore_info for the backend
+ *
+ * Returns 0 on success, and non-zero on error. (Though pstore will
+ * ignore the error.)
+ *
+ * @read:
+ * Read next available backend record. Called after a successful
+ * @open.
+ *
+ * @record:
+ * pointer to record to populate. @buf should be allocated
+ * by the backend and filled. At least @type and @id should
+ * be populated, since these are used when creating pstorefs
+ * file names.
+ *
+ * Returns record size on success, zero when no more records are
+ * available, or negative on error.
+ *
+ * @write:
+ * A newly generated record needs to be written to backend storage.
+ *
+ * @record:
+ * pointer to record metadata. When @type is PSTORE_TYPE_DMESG,
+ * @buf will be pointing to the preallocated @psi.buf, since
+ * memory allocation may be broken during an Oops. Regardless,
+ * @buf must be proccesed or copied before returning. The
+ * backend is also expected to write @id with something that
+ 8 can help identify this record to a future @erase callback.
+ *
+ * Returns 0 on success, and non-zero on error.
+ *
+ * @write_user:
+ * Perform a frontend write to a backend record, using a specified
+ * buffer that is coming directly from userspace, instead of the
+ * @record @buf.
+ *
+ * @record: pointer to record metadata.
+ * @buf: pointer to userspace contents to write to backend
+ *
+ * Returns 0 on success, and non-zero on error.
+ *
+ * @erase:
+ * Delete a record from backend storage. Different backends
+ * identify records differently, so entire original record is
+ * passed back to assist in identification of what the backend
+ * should remove from storage.
+ *
+ * @record: pointer to record metadata.
+ *
+ * Returns 0 on success, and non-zero on error.
+ *
+ */
struct pstore_info {
struct module *owner;
char *name;
- spinlock_t buf_lock; /* serialize access to 'buf' */
+
+ spinlock_t buf_lock;
char *buf;
size_t bufsize;
- struct mutex read_mutex; /* serialize open/read/close */
+
+ struct mutex read_mutex;
+
int flags;
+ void *data;
+
int (*open)(struct pstore_info *psi);
int (*close)(struct pstore_info *psi);
- ssize_t (*read)(u64 *id, enum pstore_type_id *type,
- int *count, struct timespec *time, char **buf,
- bool *compressed, ssize_t *ecc_notice_size,
- struct pstore_info *psi);
- int (*write)(enum pstore_type_id type,
- enum kmsg_dump_reason reason, u64 *id,
- unsigned int part, int count, bool compressed,
- size_t size, struct pstore_info *psi);
- int (*write_buf)(enum pstore_type_id type,
- enum kmsg_dump_reason reason, u64 *id,
- unsigned int part, const char *buf, bool compressed,
- size_t size, struct pstore_info *psi);
- int (*write_buf_user)(enum pstore_type_id type,
- enum kmsg_dump_reason reason, u64 *id,
- unsigned int part, const char __user *buf,
- bool compressed, size_t size, struct pstore_info *psi);
- int (*erase)(enum pstore_type_id type, u64 id,
- int count, struct timespec time,
- struct pstore_info *psi);
- void *data;
+ ssize_t (*read)(struct pstore_record *record);
+ int (*write)(struct pstore_record *record);
+ int (*write_user)(struct pstore_record *record,
+ const char __user *buf);
+ int (*erase)(struct pstore_record *record);
};
+/* Supported frontends */
#define PSTORE_FLAGS_DMESG (1 << 0)
-#define PSTORE_FLAGS_FRAGILE PSTORE_FLAGS_DMESG
#define PSTORE_FLAGS_CONSOLE (1 << 1)
#define PSTORE_FLAGS_FTRACE (1 << 2)
#define PSTORE_FLAGS_PMSG (1 << 3)
diff --git a/include/linux/qed/common_hsi.h b/include/linux/qed/common_hsi.h
index 52966b9..fbab6e0 100644
--- a/include/linux/qed/common_hsi.h
+++ b/include/linux/qed/common_hsi.h
@@ -100,8 +100,8 @@
#define MAX_NUM_LL2_TX_STATS_COUNTERS 32
#define FW_MAJOR_VERSION 8
-#define FW_MINOR_VERSION 10
-#define FW_REVISION_VERSION 10
+#define FW_MINOR_VERSION 15
+#define FW_REVISION_VERSION 3
#define FW_ENGINEERING_VERSION 0
/***********************/
@@ -187,6 +187,9 @@
/* DEMS */
#define DQ_DEMS_LEGACY 0
+#define DQ_DEMS_TOE_MORE_TO_SEND 3
+#define DQ_DEMS_TOE_LOCAL_ADV_WND 4
+#define DQ_DEMS_ROCE_CQ_CONS 7
/* XCM agg val selection */
#define DQ_XCM_AGG_VAL_SEL_WORD2 0
@@ -214,6 +217,9 @@
#define DQ_XCM_ISCSI_MORE_TO_SEND_SEQ_CMD DQ_XCM_AGG_VAL_SEL_REG3
#define DQ_XCM_ISCSI_EXP_STAT_SN_CMD DQ_XCM_AGG_VAL_SEL_REG6
#define DQ_XCM_ROCE_SQ_PROD_CMD DQ_XCM_AGG_VAL_SEL_WORD4
+#define DQ_XCM_TOE_TX_BD_PROD_CMD DQ_XCM_AGG_VAL_SEL_WORD4
+#define DQ_XCM_TOE_MORE_TO_SEND_SEQ_CMD DQ_XCM_AGG_VAL_SEL_REG3
+#define DQ_XCM_TOE_LOCAL_ADV_WND_SEQ_CMD DQ_XCM_AGG_VAL_SEL_REG4
/* UCM agg val selection (HW) */
#define DQ_UCM_AGG_VAL_SEL_WORD0 0
@@ -269,6 +275,8 @@
#define DQ_XCM_ISCSI_DQ_FLUSH_CMD BIT(DQ_XCM_AGG_FLG_SHIFT_CF19)
#define DQ_XCM_ISCSI_SLOW_PATH_CMD BIT(DQ_XCM_AGG_FLG_SHIFT_CF22)
#define DQ_XCM_ISCSI_PROC_ONLY_CLEANUP_CMD BIT(DQ_XCM_AGG_FLG_SHIFT_CF23)
+#define DQ_XCM_TOE_DQ_FLUSH_CMD BIT(DQ_XCM_AGG_FLG_SHIFT_CF19)
+#define DQ_XCM_TOE_SLOW_PATH_CMD BIT(DQ_XCM_AGG_FLG_SHIFT_CF22)
/* UCM agg counter flag selection (HW) */
#define DQ_UCM_AGG_FLG_SHIFT_CF0 0
@@ -285,6 +293,9 @@
#define DQ_UCM_ETH_PMD_RX_ARM_CMD BIT(DQ_UCM_AGG_FLG_SHIFT_CF5)
#define DQ_UCM_ROCE_CQ_ARM_SE_CF_CMD BIT(DQ_UCM_AGG_FLG_SHIFT_CF4)
#define DQ_UCM_ROCE_CQ_ARM_CF_CMD BIT(DQ_UCM_AGG_FLG_SHIFT_CF5)
+#define DQ_UCM_TOE_TIMER_STOP_ALL_CMD BIT(DQ_UCM_AGG_FLG_SHIFT_CF3)
+#define DQ_UCM_TOE_SLOW_PATH_CF_CMD BIT(DQ_UCM_AGG_FLG_SHIFT_CF4)
+#define DQ_UCM_TOE_DQ_CF_CMD BIT(DQ_UCM_AGG_FLG_SHIFT_CF5)
/* TCM agg counter flag selection (HW) */
#define DQ_TCM_AGG_FLG_SHIFT_CF0 0
@@ -301,6 +312,9 @@
#define DQ_TCM_FCOE_TIMER_STOP_ALL_CMD BIT(DQ_TCM_AGG_FLG_SHIFT_CF3)
#define DQ_TCM_ISCSI_FLUSH_Q0_CMD BIT(DQ_TCM_AGG_FLG_SHIFT_CF1)
#define DQ_TCM_ISCSI_TIMER_STOP_ALL_CMD BIT(DQ_TCM_AGG_FLG_SHIFT_CF3)
+#define DQ_TCM_TOE_FLUSH_Q0_CMD BIT(DQ_TCM_AGG_FLG_SHIFT_CF1)
+#define DQ_TCM_TOE_TIMER_STOP_ALL_CMD BIT(DQ_TCM_AGG_FLG_SHIFT_CF3)
+#define DQ_TCM_IWARP_POST_RQ_CF_CMD BIT(DQ_TCM_AGG_FLG_SHIFT_CF1)
/* PWM address mapping */
#define DQ_PWM_OFFSET_DPM_BASE 0x0
@@ -689,6 +703,16 @@
#define ISCSI_EQE_DATA_RESERVED0_SHIFT 7
};
+struct rdma_eqe_destroy_qp {
+ __le32 cid;
+ u8 reserved[4];
+};
+
+union rdma_eqe_data {
+ struct regpair async_handle;
+ struct rdma_eqe_destroy_qp rdma_destroy_qp_data;
+};
+
struct malicious_vf_eqe_data {
u8 vf_id;
u8 err_id;
@@ -705,9 +729,9 @@
u8 bytes[8];
struct vf_pf_channel_eqe_data vf_pf_channel;
struct iscsi_eqe_data iscsi_info;
+ union rdma_eqe_data rdma_data;
struct malicious_vf_eqe_data malicious_vf;
struct initial_cleanup_eqe_data vf_init_cleanup;
- struct regpair roce_handle;
};
/* Event Ring Entry */
diff --git a/include/linux/qed/eth_common.h b/include/linux/qed/eth_common.h
index 4b402fb..34d93eb 100644
--- a/include/linux/qed/eth_common.h
+++ b/include/linux/qed/eth_common.h
@@ -49,6 +49,9 @@
#define ETH_RX_CQE_PAGE_SIZE_BYTES 4096
#define ETH_RX_NUM_NEXT_PAGE_BDS 2
+#define ETH_MAX_TUNN_LSO_INNER_IPV4_OFFSET 253
+#define ETH_MAX_TUNN_LSO_INNER_IPV6_OFFSET 251
+
#define ETH_TX_MIN_BDS_PER_NON_LSO_PKT 1
#define ETH_TX_MAX_BDS_PER_NON_LSO_PACKET 18
#define ETH_TX_MAX_BDS_PER_LSO_PACKET 255
diff --git a/include/linux/qed/fcoe_common.h b/include/linux/qed/fcoe_common.h
index 2e417a4..947a635 100644
--- a/include/linux/qed/fcoe_common.h
+++ b/include/linux/qed/fcoe_common.h
@@ -109,13 +109,6 @@
struct regpair terminate_params_addr;
};
-struct fcoe_fast_sgl_ctx {
- struct regpair sgl_start_addr;
- __le32 sgl_byte_offset;
- __le16 task_reuse_cnt;
- __le16 init_offset_in_first_sge;
-};
-
struct fcoe_slow_sgl_ctx {
struct regpair base_sgl_addr;
__le16 curr_sge_off;
@@ -124,23 +117,16 @@
__le16 reserved;
};
-struct fcoe_sge {
- struct regpair sge_addr;
- __le16 size;
- __le16 reserved0;
- u8 reserved1[3];
- u8 is_valid_sge;
-};
-
-union fcoe_data_desc_ctx {
- struct fcoe_fast_sgl_ctx fast;
- struct fcoe_slow_sgl_ctx slow;
- struct fcoe_sge single_sge;
-};
-
union fcoe_dix_desc_ctx {
struct fcoe_slow_sgl_ctx dix_sgl;
- struct fcoe_sge cached_dix_sge;
+ struct scsi_sge cached_dix_sge;
+};
+
+struct fcoe_fast_sgl_ctx {
+ struct regpair sgl_start_addr;
+ __le32 sgl_byte_offset;
+ __le16 task_reuse_cnt;
+ __le16 init_offset_in_first_sge;
};
struct fcoe_fcp_cmd_payload {
@@ -172,57 +158,6 @@
MAX_FCOE_MODE_TYPE
};
-struct fcoe_mstorm_fcoe_task_st_ctx_fp {
- __le16 flags;
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_FP_RSRV0_MASK 0x7FFF
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_FP_RSRV0_SHIFT 0
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_FP_MP_INCLUDE_FC_HEADER_MASK 0x1
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_FP_MP_INCLUDE_FC_HEADER_SHIFT 15
- __le16 difDataResidue;
- __le16 parent_id;
- __le16 single_sge_saved_offset;
- __le32 data_2_trns_rem;
- __le32 offset_in_io;
- union fcoe_dix_desc_ctx dix_desc;
- union fcoe_data_desc_ctx data_desc;
-};
-
-struct fcoe_mstorm_fcoe_task_st_ctx_non_fp {
- __le16 flags;
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_HOST_INTERFACE_MASK 0x3
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_HOST_INTERFACE_SHIFT 0
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_DIF_TO_PEER_MASK 0x1
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_DIF_TO_PEER_SHIFT 2
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_VALIDATE_DIX_APP_TAG_MASK 0x1
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_VALIDATE_DIX_APP_TAG_SHIFT 3
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_INTERVAL_SIZE_LOG_MASK 0xF
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_INTERVAL_SIZE_LOG_SHIFT 4
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_DIX_BLOCK_SIZE_MASK 0x3
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_DIX_BLOCK_SIZE_SHIFT 8
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_RESERVED_MASK 0x1
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_RESERVED_SHIFT 10
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_HAS_FIRST_PACKET_ARRIVED_MASK 0x1
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_HAS_FIRST_PACKET_ARRIVED_SHIFT 11
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_VALIDATE_DIX_REF_TAG_MASK 0x1
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_VALIDATE_DIX_REF_TAG_SHIFT 12
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_DIX_CACHED_SGE_FLG_MASK 0x1
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_DIX_CACHED_SGE_FLG_SHIFT 13
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_OFFSET_IN_IO_VALID_MASK 0x1
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_OFFSET_IN_IO_VALID_SHIFT 14
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_DIF_SUPPORTED_MASK 0x1
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_DIF_SUPPORTED_SHIFT 15
- u8 tx_rx_sgl_mode;
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_TX_SGL_MODE_MASK 0x7
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_TX_SGL_MODE_SHIFT 0
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_RX_SGL_MODE_MASK 0x7
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_RX_SGL_MODE_SHIFT 3
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_RSRV1_MASK 0x3
-#define FCOE_MSTORM_FCOE_TASK_ST_CTX_NON_FP_RSRV1_SHIFT 6
- u8 rsrv2;
- __le32 num_prm_zero_read;
- struct regpair rsp_buf_addr;
-};
-
struct fcoe_rx_stat {
struct regpair fcoe_rx_byte_cnt;
struct regpair fcoe_rx_data_pkt_cnt;
@@ -236,16 +171,6 @@
__le32 rsrv;
};
-enum fcoe_sgl_mode {
- FCOE_SLOW_SGL,
- FCOE_SINGLE_FAST_SGE,
- FCOE_2_FAST_SGE,
- FCOE_3_FAST_SGE,
- FCOE_4_FAST_SGE,
- FCOE_MUL_FAST_SGES,
- MAX_FCOE_SGL_MODE
-};
-
struct fcoe_stat_ramrod_data {
struct regpair stat_params_addr;
};
@@ -328,22 +253,24 @@
struct ystorm_fcoe_task_st_ctx {
u8 task_type;
u8 sgl_mode;
-#define YSTORM_FCOE_TASK_ST_CTX_TX_SGL_MODE_MASK 0x7
+#define YSTORM_FCOE_TASK_ST_CTX_TX_SGL_MODE_MASK 0x1
#define YSTORM_FCOE_TASK_ST_CTX_TX_SGL_MODE_SHIFT 0
-#define YSTORM_FCOE_TASK_ST_CTX_RSRV_MASK 0x1F
-#define YSTORM_FCOE_TASK_ST_CTX_RSRV_SHIFT 3
+#define YSTORM_FCOE_TASK_ST_CTX_RSRV_MASK 0x7F
+#define YSTORM_FCOE_TASK_ST_CTX_RSRV_SHIFT 1
u8 cached_dix_sge;
u8 expect_first_xfer;
__le32 num_pbf_zero_write;
union protection_info_union_ctx protection_info_union;
__le32 data_2_trns_rem;
+ struct scsi_sgl_params sgl_params;
+ u8 reserved1[12];
union fcoe_tx_info_union_ctx tx_info_union;
union fcoe_dix_desc_ctx dix_desc;
- union fcoe_data_desc_ctx data_desc;
+ struct scsi_cached_sges data_desc;
__le16 ox_id;
__le16 rx_id;
__le32 task_rety_identifier;
- __le32 reserved1[2];
+ u8 reserved2[8];
};
struct ystorm_fcoe_task_ag_ctx {
@@ -484,22 +411,22 @@
struct fcoe_tstorm_fcoe_task_st_ctx_read_write {
union fcoe_cleanup_addr_exp_ro_union cleanup_addr_exp_ro_union;
__le16 flags;
-#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_RX_SGL_MODE_MASK 0x7
+#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_RX_SGL_MODE_MASK 0x1
#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_RX_SGL_MODE_SHIFT 0
#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_EXP_FIRST_FRAME_MASK 0x1
-#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_EXP_FIRST_FRAME_SHIFT 3
+#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_EXP_FIRST_FRAME_SHIFT 1
#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_SEQ_ACTIVE_MASK 0x1
-#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_SEQ_ACTIVE_SHIFT 4
+#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_SEQ_ACTIVE_SHIFT 2
#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_SEQ_TIMEOUT_MASK 0x1
-#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_SEQ_TIMEOUT_SHIFT 5
+#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_SEQ_TIMEOUT_SHIFT 3
#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_SINGLE_PKT_IN_EX_MASK 0x1
-#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_SINGLE_PKT_IN_EX_SHIFT 6
+#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_SINGLE_PKT_IN_EX_SHIFT 4
#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_OOO_RX_SEQ_STAT_MASK 0x1
-#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_OOO_RX_SEQ_STAT_SHIFT 7
+#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_OOO_RX_SEQ_STAT_SHIFT 5
#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_CQ_ADD_ADV_MASK 0x3
-#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_CQ_ADD_ADV_SHIFT 8
-#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_RSRV1_MASK 0x3F
-#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_RSRV1_SHIFT 10
+#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_CQ_ADD_ADV_SHIFT 6
+#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_RSRV1_MASK 0xFF
+#define FCOE_TSTORM_FCOE_TASK_ST_CTX_READ_WRITE_RSRV1_SHIFT 8
__le16 seq_cnt;
u8 seq_id;
u8 ooo_rx_seq_id;
@@ -582,8 +509,34 @@
};
struct mstorm_fcoe_task_st_ctx {
- struct fcoe_mstorm_fcoe_task_st_ctx_non_fp non_fp;
- struct fcoe_mstorm_fcoe_task_st_ctx_fp fp;
+ struct regpair rsp_buf_addr;
+ __le32 rsrv[2];
+ struct scsi_sgl_params sgl_params;
+ __le32 data_2_trns_rem;
+ __le32 data_buffer_offset;
+ __le16 parent_id;
+ __le16 flags;
+#define MSTORM_FCOE_TASK_ST_CTX_INTERVAL_SIZE_LOG_MASK 0xF
+#define MSTORM_FCOE_TASK_ST_CTX_INTERVAL_SIZE_LOG_SHIFT 0
+#define MSTORM_FCOE_TASK_ST_CTX_HOST_INTERFACE_MASK 0x3
+#define MSTORM_FCOE_TASK_ST_CTX_HOST_INTERFACE_SHIFT 4
+#define MSTORM_FCOE_TASK_ST_CTX_DIF_TO_PEER_MASK 0x1
+#define MSTORM_FCOE_TASK_ST_CTX_DIF_TO_PEER_SHIFT 6
+#define MSTORM_FCOE_TASK_ST_CTX_MP_INCLUDE_FC_HEADER_MASK 0x1
+#define MSTORM_FCOE_TASK_ST_CTX_MP_INCLUDE_FC_HEADER_SHIFT 7
+#define MSTORM_FCOE_TASK_ST_CTX_DIX_BLOCK_SIZE_MASK 0x3
+#define MSTORM_FCOE_TASK_ST_CTX_DIX_BLOCK_SIZE_SHIFT 8
+#define MSTORM_FCOE_TASK_ST_CTX_VALIDATE_DIX_REF_TAG_MASK 0x1
+#define MSTORM_FCOE_TASK_ST_CTX_VALIDATE_DIX_REF_TAG_SHIFT 10
+#define MSTORM_FCOE_TASK_ST_CTX_DIX_CACHED_SGE_FLG_MASK 0x1
+#define MSTORM_FCOE_TASK_ST_CTX_DIX_CACHED_SGE_FLG_SHIFT 11
+#define MSTORM_FCOE_TASK_ST_CTX_DIF_SUPPORTED_MASK 0x1
+#define MSTORM_FCOE_TASK_ST_CTX_DIF_SUPPORTED_SHIFT 12
+#define MSTORM_FCOE_TASK_ST_CTX_TX_SGL_MODE_MASK 0x1
+#define MSTORM_FCOE_TASK_ST_CTX_TX_SGL_MODE_SHIFT 13
+#define MSTORM_FCOE_TASK_ST_CTX_RESERVED_MASK 0x3
+#define MSTORM_FCOE_TASK_ST_CTX_RESERVED_SHIFT 14
+ struct scsi_cached_sges data_desc;
};
struct ustorm_fcoe_task_ag_ctx {
@@ -646,6 +599,7 @@
struct fcoe_task_context {
struct ystorm_fcoe_task_st_ctx ystorm_st_context;
+ struct regpair ystorm_st_padding[2];
struct tdif_task_context tdif_context;
struct ystorm_fcoe_task_ag_ctx ystorm_ag_context;
struct tstorm_fcoe_task_ag_ctx tstorm_ag_context;
@@ -668,20 +622,20 @@
struct fcoe_wqe {
__le16 task_id;
__le16 flags;
-#define FCOE_WQE_REQ_TYPE_MASK 0xF
-#define FCOE_WQE_REQ_TYPE_SHIFT 0
-#define FCOE_WQE_SGL_MODE_MASK 0x7
-#define FCOE_WQE_SGL_MODE_SHIFT 4
-#define FCOE_WQE_CONTINUATION_MASK 0x1
-#define FCOE_WQE_CONTINUATION_SHIFT 7
-#define FCOE_WQE_INVALIDATE_PTU_MASK 0x1
-#define FCOE_WQE_INVALIDATE_PTU_SHIFT 8
-#define FCOE_WQE_SUPER_IO_MASK 0x1
-#define FCOE_WQE_SUPER_IO_SHIFT 9
-#define FCOE_WQE_SEND_AUTO_RSP_MASK 0x1
-#define FCOE_WQE_SEND_AUTO_RSP_SHIFT 10
-#define FCOE_WQE_RESERVED0_MASK 0x1F
-#define FCOE_WQE_RESERVED0_SHIFT 11
+#define FCOE_WQE_REQ_TYPE_MASK 0xF
+#define FCOE_WQE_REQ_TYPE_SHIFT 0
+#define FCOE_WQE_SGL_MODE_MASK 0x1
+#define FCOE_WQE_SGL_MODE_SHIFT 4
+#define FCOE_WQE_CONTINUATION_MASK 0x1
+#define FCOE_WQE_CONTINUATION_SHIFT 5
+#define FCOE_WQE_SEND_AUTO_RSP_MASK 0x1
+#define FCOE_WQE_SEND_AUTO_RSP_SHIFT 6
+#define FCOE_WQE_RESERVED_MASK 0x1
+#define FCOE_WQE_RESERVED_SHIFT 7
+#define FCOE_WQE_NUM_SGES_MASK 0xF
+#define FCOE_WQE_NUM_SGES_SHIFT 8
+#define FCOE_WQE_RESERVED1_MASK 0xF
+#define FCOE_WQE_RESERVED1_SHIFT 12
union fcoe_additional_info_union additional_info_union;
};
diff --git a/include/linux/qed/iscsi_common.h b/include/linux/qed/iscsi_common.h
index 4c5747b..69949f8 100644
--- a/include/linux/qed/iscsi_common.h
+++ b/include/linux/qed/iscsi_common.h
@@ -39,17 +39,9 @@
/* iSCSI HSI constants */
#define ISCSI_DEFAULT_MTU (1500)
-/* Current iSCSI HSI version number composed of two fields (16 bit) */
-#define ISCSI_HSI_MAJOR_VERSION (0)
-#define ISCSI_HSI_MINOR_VERSION (0)
-
/* KWQ (kernel work queue) layer codes */
#define ISCSI_SLOW_PATH_LAYER_CODE (6)
-/* CQE completion status */
-#define ISCSI_EQE_COMPLETION_SUCCESS (0x0)
-#define ISCSI_EQE_RST_CONN_RCVD (0x1)
-
/* iSCSI parameter defaults */
#define ISCSI_DEFAULT_HEADER_DIGEST (0)
#define ISCSI_DEFAULT_DATA_DIGEST (0)
@@ -68,6 +60,10 @@
#define ISCSI_MIN_VAL_MAX_OUTSTANDING_R2T (1)
#define ISCSI_MAX_VAL_MAX_OUTSTANDING_R2T (0xff)
+#define ISCSI_AHS_CNTL_SIZE 4
+
+#define ISCSI_WQE_NUM_SGES_SLOWIO (0xf)
+
/* iSCSI reserved params */
#define ISCSI_ITT_ALL_ONES (0xffffffff)
#define ISCSI_TTT_ALL_ONES (0xffffffff)
@@ -173,19 +169,6 @@
__le32 reserved7;
};
-struct iscsi_sge {
- struct regpair sge_addr;
- __le16 sge_len;
- __le16 reserved0;
- __le32 reserved1;
-};
-
-struct iscsi_cached_sge_ctx {
- struct iscsi_sge sge;
- struct regpair reserved;
- __le32 dsgl_curr_offset[2];
-};
-
struct iscsi_cmd_hdr {
__le16 reserved1;
u8 flags_attr;
@@ -229,8 +212,13 @@
#define ISCSI_COMMON_HDR_DATA_SEG_LEN_SHIFT 0
#define ISCSI_COMMON_HDR_TOTAL_AHS_LEN_MASK 0xFF
#define ISCSI_COMMON_HDR_TOTAL_AHS_LEN_SHIFT 24
- __le32 lun_reserved[4];
- __le32 data[6];
+ struct regpair lun_reserved;
+ __le32 itt;
+ __le32 ttt;
+ __le32 cmdstat_sn;
+ __le32 exp_statcmd_sn;
+ __le32 max_cmd_sn;
+ __le32 data[3];
};
struct iscsi_conn_offload_params {
@@ -246,8 +234,10 @@
#define ISCSI_CONN_OFFLOAD_PARAMS_TCP_ON_CHIP_1B_SHIFT 0
#define ISCSI_CONN_OFFLOAD_PARAMS_TARGET_MODE_MASK 0x1
#define ISCSI_CONN_OFFLOAD_PARAMS_TARGET_MODE_SHIFT 1
-#define ISCSI_CONN_OFFLOAD_PARAMS_RESERVED1_MASK 0x3F
-#define ISCSI_CONN_OFFLOAD_PARAMS_RESERVED1_SHIFT 2
+#define ISCSI_CONN_OFFLOAD_PARAMS_RESTRICTED_MODE_MASK 0x1
+#define ISCSI_CONN_OFFLOAD_PARAMS_RESTRICTED_MODE_SHIFT 2
+#define ISCSI_CONN_OFFLOAD_PARAMS_RESERVED1_MASK 0x1F
+#define ISCSI_CONN_OFFLOAD_PARAMS_RESERVED1_SHIFT 3
u8 pbl_page_size_log;
u8 pbe_page_size_log;
u8 default_cq;
@@ -278,8 +268,12 @@
#define ISCSI_CONN_UPDATE_RAMROD_PARAMS_INITIAL_R2T_SHIFT 2
#define ISCSI_CONN_UPDATE_RAMROD_PARAMS_IMMEDIATE_DATA_MASK 0x1
#define ISCSI_CONN_UPDATE_RAMROD_PARAMS_IMMEDIATE_DATA_SHIFT 3
-#define ISCSI_CONN_UPDATE_RAMROD_PARAMS_RESERVED1_MASK 0xF
-#define ISCSI_CONN_UPDATE_RAMROD_PARAMS_RESERVED1_SHIFT 4
+#define ISCSI_CONN_UPDATE_RAMROD_PARAMS_DIF_BLOCK_SIZE_MASK 0x1
+#define ISCSI_CONN_UPDATE_RAMROD_PARAMS_DIF_BLOCK_SIZE_SHIFT 4
+#define ISCSI_CONN_UPDATE_RAMROD_PARAMS_DIF_ON_HOST_EN_MASK 0x1
+#define ISCSI_CONN_UPDATE_RAMROD_PARAMS_DIF_ON_HOST_EN_SHIFT 5
+#define ISCSI_CONN_UPDATE_RAMROD_PARAMS_RESERVED1_MASK 0x3
+#define ISCSI_CONN_UPDATE_RAMROD_PARAMS_RESERVED1_SHIFT 6
u8 reserved0[3];
__le32 max_seq_size;
__le32 max_send_pdu_length;
@@ -312,7 +306,7 @@
__le32 expected_transfer_length;
__le32 cmd_sn;
__le32 exp_stat_sn;
- struct iscsi_sge cdb_sge;
+ struct scsi_sge cdb_sge;
};
struct iscsi_login_req_hdr {
@@ -519,8 +513,8 @@
__le32 exp_cmd_sn;
__le32 max_cmd_sn;
__le32 reserved4;
- __le16 time2retain;
- __le16 time2wait;
+ __le16 time_2_retain;
+ __le16 time_2_wait;
__le32 reserved5[1];
};
@@ -602,7 +596,7 @@
#define ISCSI_TMF_RESPONSE_HDR_TOTAL_AHS_LEN_SHIFT 24
struct regpair reserved0;
__le32 itt;
- __le32 rtt;
+ __le32 reserved1;
__le32 stat_sn;
__le32 exp_cmd_sn;
__le32 max_cmd_sn;
@@ -641,7 +635,7 @@
#define ISCSI_REJECT_HDR_TOTAL_AHS_LEN_MASK 0xFF
#define ISCSI_REJECT_HDR_TOTAL_AHS_LEN_SHIFT 24
struct regpair reserved0;
- __le32 reserved1;
+ __le32 all_ones;
__le32 reserved2;
__le32 stat_sn;
__le32 exp_cmd_sn;
@@ -688,7 +682,9 @@
__le16 itid;
u8 task_type;
u8 fw_dbg_field;
- __le32 reserved1[2];
+ u8 caused_conn_err;
+ u8 reserved0[3];
+ __le32 reserved1[1];
union iscsi_task_hdr iscsi_hdr;
};
@@ -727,35 +723,6 @@
MAX_ISCSI_CQE_UNSOLICITED_TYPE
};
-struct iscsi_virt_sgl_ctx {
- struct regpair sgl_base;
- struct regpair dsgl_base;
- __le32 sgl_initial_offset;
- __le32 dsgl_initial_offset;
- __le32 dsgl_curr_offset[2];
-};
-
-struct iscsi_sgl_var_params {
- u8 sgl_ptr;
- u8 dsgl_ptr;
- __le16 sge_offset;
- __le16 dsge_offset;
-};
-
-struct iscsi_phys_sgl_ctx {
- struct regpair sgl_base;
- struct regpair dsgl_base;
- u8 sgl_size;
- u8 dsgl_size;
- __le16 reserved;
- struct iscsi_sgl_var_params var_params[2];
-};
-
-union iscsi_data_desc_ctx {
- struct iscsi_virt_sgl_ctx virt_sgl;
- struct iscsi_phys_sgl_ctx phys_sgl;
- struct iscsi_cached_sge_ctx cached_sge;
-};
struct iscsi_debug_modes {
u8 flags;
@@ -771,8 +738,10 @@
#define ISCSI_DEBUG_MODES_ASSERT_IF_RECV_REJECT_OR_ASYNC_SHIFT 4
#define ISCSI_DEBUG_MODES_ASSERT_IF_RECV_NOP_MASK 0x1
#define ISCSI_DEBUG_MODES_ASSERT_IF_RECV_NOP_SHIFT 5
-#define ISCSI_DEBUG_MODES_RESERVED0_MASK 0x3
-#define ISCSI_DEBUG_MODES_RESERVED0_SHIFT 6
+#define ISCSI_DEBUG_MODES_ASSERT_IF_DATA_DIGEST_ERROR_MASK 0x1
+#define ISCSI_DEBUG_MODES_ASSERT_IF_DATA_DIGEST_ERROR_SHIFT 6
+#define ISCSI_DEBUG_MODES_ASSERT_IF_DIF_ERROR_MASK 0x1
+#define ISCSI_DEBUG_MODES_ASSERT_IF_DIF_ERROR_SHIFT 7
};
struct iscsi_dif_flags {
@@ -806,7 +775,6 @@
ISCSI_EVENT_TYPE_ASYN_FIN_WAIT2,
ISCSI_EVENT_TYPE_ISCSI_CONN_ERROR,
ISCSI_EVENT_TYPE_TCP_CONN_ERROR,
- ISCSI_EVENT_TYPE_ASYN_DELETE_OOO_ISLES,
MAX_ISCSI_EQE_OPCODE
};
@@ -856,31 +824,11 @@
ISCSI_CONN_ERROR_PROTOCOL_ERR_DIF_TX,
ISCSI_CONN_ERROR_SENSE_DATA_LENGTH,
ISCSI_CONN_ERROR_DATA_PLACEMENT_ERROR,
+ ISCSI_CONN_ERROR_INVALID_ITT,
ISCSI_ERROR_UNKNOWN,
MAX_ISCSI_ERROR_TYPES
};
-struct iscsi_mflags {
- u8 mflags;
-#define ISCSI_MFLAGS_SLOW_IO_MASK 0x1
-#define ISCSI_MFLAGS_SLOW_IO_SHIFT 0
-#define ISCSI_MFLAGS_SINGLE_SGE_MASK 0x1
-#define ISCSI_MFLAGS_SINGLE_SGE_SHIFT 1
-#define ISCSI_MFLAGS_RESERVED_MASK 0x3F
-#define ISCSI_MFLAGS_RESERVED_SHIFT 2
-};
-
-struct iscsi_sgl {
- struct regpair sgl_addr;
- __le16 updated_sge_size;
- __le16 updated_sge_offset;
- __le32 byte_offset;
-};
-
-union iscsi_mstorm_sgl {
- struct iscsi_sgl sgl_struct;
- struct iscsi_sge single_sge;
-};
enum iscsi_ramrod_cmd_id {
ISCSI_RAMROD_CMD_ID_UNUSED = 0,
@@ -896,10 +844,10 @@
struct iscsi_reg1 {
__le32 reg1_map;
-#define ISCSI_REG1_NUM_FAST_SGES_MASK 0x7
-#define ISCSI_REG1_NUM_FAST_SGES_SHIFT 0
-#define ISCSI_REG1_RESERVED1_MASK 0x1FFFFFFF
-#define ISCSI_REG1_RESERVED1_SHIFT 3
+#define ISCSI_REG1_NUM_SGES_MASK 0xF
+#define ISCSI_REG1_NUM_SGES_SHIFT 0
+#define ISCSI_REG1_RESERVED1_MASK 0xFFFFFFF
+#define ISCSI_REG1_RESERVED1_SHIFT 4
};
union iscsi_seq_num {
@@ -967,22 +915,33 @@
};
struct ystorm_iscsi_task_state {
- union iscsi_data_desc_ctx sgl_ctx_union;
- __le32 buffer_offset[2];
- __le16 bytes_nxt_dif;
- __le16 rxmit_bytes_nxt_dif;
- union iscsi_seq_num seq_num_union;
- u8 dif_bytes_leftover;
- u8 rxmit_dif_bytes_leftover;
- __le16 reuse_count;
- struct iscsi_dif_flags dif_flags;
- u8 local_comp;
+ struct scsi_cached_sges data_desc;
+ struct scsi_sgl_params sgl_params;
__le32 exp_r2t_sn;
- __le32 sgl_offset[2];
+ __le32 buffer_offset;
+ union iscsi_seq_num seq_num;
+ struct iscsi_dif_flags dif_flags;
+ u8 flags;
+#define YSTORM_ISCSI_TASK_STATE_LOCAL_COMP_MASK 0x1
+#define YSTORM_ISCSI_TASK_STATE_LOCAL_COMP_SHIFT 0
+#define YSTORM_ISCSI_TASK_STATE_SLOW_IO_MASK 0x1
+#define YSTORM_ISCSI_TASK_STATE_SLOW_IO_SHIFT 1
+#define YSTORM_ISCSI_TASK_STATE_RESERVED0_MASK 0x3F
+#define YSTORM_ISCSI_TASK_STATE_RESERVED0_SHIFT 2
+};
+
+struct ystorm_iscsi_task_rxmit_opt {
+ __le32 fast_rxmit_sge_offset;
+ __le32 scan_start_buffer_offset;
+ __le32 fast_rxmit_buffer_offset;
+ u8 scan_start_sgl_index;
+ u8 fast_rxmit_sgl_index;
+ __le16 reserved;
};
struct ystorm_iscsi_task_st_ctx {
struct ystorm_iscsi_task_state state;
+ struct ystorm_iscsi_task_rxmit_opt rxmit_opt;
union iscsi_task_hdr pdu_hdr;
};
@@ -1152,25 +1111,16 @@
};
struct mstorm_iscsi_task_st_ctx {
- union iscsi_mstorm_sgl sgl_union;
- struct iscsi_dif_flags dif_flags;
- struct iscsi_mflags flags;
- u8 sgl_size;
- u8 host_sge_index;
- __le16 dix_cur_sge_offset;
- __le16 dix_cur_sge_size;
- __le32 data_offset_rtid;
- u8 dif_offset;
- u8 dix_sgl_size;
- u8 dix_sge_index;
- u8 task_type;
- struct regpair sense_db;
- struct regpair dix_sgl_cur_sge;
+ struct scsi_cached_sges data_desc;
+ struct scsi_sgl_params sgl_params;
__le32 rem_task_size;
- __le16 reuse_count;
- __le16 dif_data_residue;
- u8 reserved0[4];
- __le32 reserved1[1];
+ __le32 data_buffer_offset;
+ u8 task_type;
+ struct iscsi_dif_flags dif_flags;
+ u8 reserved0[2];
+ struct regpair sense_db;
+ __le32 expected_itt;
+ __le32 reserved1;
};
struct ustorm_iscsi_task_st_ctx {
@@ -1184,7 +1134,7 @@
#define USTORM_ISCSI_TASK_ST_CTX_AHS_EXIST_SHIFT 0
#define USTORM_ISCSI_TASK_ST_CTX_RESERVED1_MASK 0x7F
#define USTORM_ISCSI_TASK_ST_CTX_RESERVED1_SHIFT 1
- u8 reserved2;
+ struct iscsi_dif_flags dif_flags;
__le16 reserved3;
__le32 reserved4;
__le32 reserved5;
@@ -1207,10 +1157,10 @@
#define USTORM_ISCSI_TASK_ST_CTX_LOCAL_COMP_SHIFT 2
#define USTORM_ISCSI_TASK_ST_CTX_Q0_R2TQE_WRITE_MASK 0x1
#define USTORM_ISCSI_TASK_ST_CTX_Q0_R2TQE_WRITE_SHIFT 3
-#define USTORM_ISCSI_TASK_ST_CTX_TOTALDATAACKED_DONE_MASK 0x1
-#define USTORM_ISCSI_TASK_ST_CTX_TOTALDATAACKED_DONE_SHIFT 4
-#define USTORM_ISCSI_TASK_ST_CTX_HQSCANNED_DONE_MASK 0x1
-#define USTORM_ISCSI_TASK_ST_CTX_HQSCANNED_DONE_SHIFT 5
+#define USTORM_ISCSI_TASK_ST_CTX_TOTAL_DATA_ACKED_DONE_MASK 0x1
+#define USTORM_ISCSI_TASK_ST_CTX_TOTAL_DATA_ACKED_DONE_SHIFT 4
+#define USTORM_ISCSI_TASK_ST_CTX_HQ_SCANNED_DONE_MASK 0x1
+#define USTORM_ISCSI_TASK_ST_CTX_HQ_SCANNED_DONE_SHIFT 5
#define USTORM_ISCSI_TASK_ST_CTX_R2T2RECV_DONE_MASK 0x1
#define USTORM_ISCSI_TASK_ST_CTX_R2T2RECV_DONE_SHIFT 6
#define USTORM_ISCSI_TASK_ST_CTX_RESERVED0_MASK 0x1
@@ -1220,7 +1170,6 @@
struct iscsi_task_context {
struct ystorm_iscsi_task_st_ctx ystorm_st_context;
- struct regpair ystorm_st_padding[2];
struct ystorm_iscsi_task_ag_ctx ystorm_ag_context;
struct regpair ystorm_ag_padding[2];
struct tdif_task_context tdif_context;
@@ -1272,32 +1221,22 @@
#define ISCSI_UHQE_TASK_ID_LO_SHIFT 24
};
-struct iscsi_wqe_field {
- __le32 contlen_cdbsize_field;
-#define ISCSI_WQE_FIELD_CONT_LEN_MASK 0xFFFFFF
-#define ISCSI_WQE_FIELD_CONT_LEN_SHIFT 0
-#define ISCSI_WQE_FIELD_CDB_SIZE_MASK 0xFF
-#define ISCSI_WQE_FIELD_CDB_SIZE_SHIFT 24
-};
-
-union iscsi_wqe_field_union {
- struct iscsi_wqe_field cont_field;
- __le32 prev_tid;
-};
struct iscsi_wqe {
__le16 task_id;
u8 flags;
#define ISCSI_WQE_WQE_TYPE_MASK 0x7
#define ISCSI_WQE_WQE_TYPE_SHIFT 0
-#define ISCSI_WQE_NUM_FAST_SGES_MASK 0x7
-#define ISCSI_WQE_NUM_FAST_SGES_SHIFT 3
-#define ISCSI_WQE_PTU_INVALIDATE_MASK 0x1
-#define ISCSI_WQE_PTU_INVALIDATE_SHIFT 6
+#define ISCSI_WQE_NUM_SGES_MASK 0xF
+#define ISCSI_WQE_NUM_SGES_SHIFT 3
#define ISCSI_WQE_RESPONSE_MASK 0x1
#define ISCSI_WQE_RESPONSE_SHIFT 7
struct iscsi_dif_flags prot_flags;
- union iscsi_wqe_field_union cont_prevtid_union;
+ __le32 contlen_cdbsize;
+#define ISCSI_WQE_CONT_LEN_MASK 0xFFFFFF
+#define ISCSI_WQE_CONT_LEN_SHIFT 0
+#define ISCSI_WQE_CDB_SIZE_MASK 0xFF
+#define ISCSI_WQE_CDB_SIZE_SHIFT 24
};
enum iscsi_wqe_type {
@@ -1318,17 +1257,15 @@
u8 total_ahs_length;
u8 opcode;
u8 flags;
-#define ISCSI_XHQE_NUM_FAST_SGES_MASK 0x7
-#define ISCSI_XHQE_NUM_FAST_SGES_SHIFT 0
-#define ISCSI_XHQE_FINAL_MASK 0x1
-#define ISCSI_XHQE_FINAL_SHIFT 3
-#define ISCSI_XHQE_SUPER_IO_MASK 0x1
-#define ISCSI_XHQE_SUPER_IO_SHIFT 4
-#define ISCSI_XHQE_STATUS_BIT_MASK 0x1
-#define ISCSI_XHQE_STATUS_BIT_SHIFT 5
-#define ISCSI_XHQE_RESERVED_MASK 0x3
-#define ISCSI_XHQE_RESERVED_SHIFT 6
- union iscsi_seq_num seq_num_union;
+#define ISCSI_XHQE_FINAL_MASK 0x1
+#define ISCSI_XHQE_FINAL_SHIFT 0
+#define ISCSI_XHQE_STATUS_BIT_MASK 0x1
+#define ISCSI_XHQE_STATUS_BIT_SHIFT 1
+#define ISCSI_XHQE_NUM_SGES_MASK 0xF
+#define ISCSI_XHQE_NUM_SGES_SHIFT 2
+#define ISCSI_XHQE_RESERVED0_MASK 0x3
+#define ISCSI_XHQE_RESERVED0_SHIFT 6
+ union iscsi_seq_num seq_num;
__le16 reserved1;
};
diff --git a/include/linux/qed/qed_eth_if.h b/include/linux/qed/qed_eth_if.h
index 4cd1f0c..d66d16a 100644
--- a/include/linux/qed/qed_eth_if.h
+++ b/include/linux/qed/qed_eth_if.h
@@ -158,15 +158,27 @@
struct qed_eth_cb_ops {
struct qed_common_cb_ops common;
void (*force_mac) (void *dev, u8 *mac, bool forced);
+ void (*ports_update)(void *dev, u16 vxlan_port, u16 geneve_port);
};
#define QED_MAX_PHC_DRIFT_PPB 291666666
enum qed_ptp_filter_type {
- QED_PTP_FILTER_L2,
- QED_PTP_FILTER_IPV4,
- QED_PTP_FILTER_IPV4_IPV6,
- QED_PTP_FILTER_L2_IPV4_IPV6
+ QED_PTP_FILTER_NONE,
+ QED_PTP_FILTER_ALL,
+ QED_PTP_FILTER_V1_L4_EVENT,
+ QED_PTP_FILTER_V1_L4_GEN,
+ QED_PTP_FILTER_V2_L4_EVENT,
+ QED_PTP_FILTER_V2_L4_GEN,
+ QED_PTP_FILTER_V2_L2_EVENT,
+ QED_PTP_FILTER_V2_L2_GEN,
+ QED_PTP_FILTER_V2_EVENT,
+ QED_PTP_FILTER_V2_GEN
+};
+
+enum qed_ptp_hwtstamp_tx_type {
+ QED_PTP_HWTSTAMP_TX_OFF,
+ QED_PTP_HWTSTAMP_TX_ON,
};
#ifdef CONFIG_DCB
@@ -229,8 +241,8 @@
#endif
struct qed_eth_ptp_ops {
- int (*hwtstamp_tx_on)(struct qed_dev *);
- int (*cfg_rx_filters)(struct qed_dev *, enum qed_ptp_filter_type);
+ int (*cfg_filters)(struct qed_dev *, enum qed_ptp_filter_type,
+ enum qed_ptp_hwtstamp_tx_type);
int (*read_rx_ts)(struct qed_dev *, u64 *);
int (*read_tx_ts)(struct qed_dev *, u64 *);
int (*read_cc)(struct qed_dev *, u64 *);
@@ -301,6 +313,14 @@
int (*tunn_config)(struct qed_dev *cdev,
struct qed_tunn_params *params);
+
+ int (*ntuple_filter_config)(struct qed_dev *cdev, void *cookie,
+ dma_addr_t mapping, u16 length,
+ u16 vport_id, u16 rx_queue_id,
+ bool add_filter);
+
+ int (*configure_arfs_searcher)(struct qed_dev *cdev,
+ bool en_searcher);
};
const struct qed_eth_ops *qed_get_eth_ops(void);
diff --git a/include/linux/qed/qed_if.h b/include/linux/qed/qed_if.h
index fde56c4..5544d7b 100644
--- a/include/linux/qed/qed_if.h
+++ b/include/linux/qed/qed_if.h
@@ -144,6 +144,7 @@
bool enabled;
bool ieee;
bool cee;
+ bool local;
u32 err;
};
@@ -178,6 +179,12 @@
* to update_pf_params routine invoked before slowpath start
*/
u16 num_cons;
+
+ /* To enable arfs, previous to HW-init a positive number needs to be
+ * set [as filters require allocated searcher ILT memory].
+ * This will set the maximal number of configured steering-filters.
+ */
+ u32 num_arfs_filters;
};
struct qed_fcoe_pf_params {
@@ -263,7 +270,6 @@
* the doorbell BAR).
*/
u32 min_dpis; /* number of requested DPIs */
- u32 num_mrs; /* number of requested memory regions */
u32 num_qps; /* number of requested Queue Pairs */
u32 num_srqs; /* number of requested SRQ */
u8 roce_edpm_mode; /* see QED_ROCE_EDPM_MODE_ENABLE */
@@ -300,6 +306,11 @@
struct qed_dev *cdev;
};
+enum qed_dev_type {
+ QED_DEV_TYPE_BB,
+ QED_DEV_TYPE_AH,
+};
+
struct qed_dev_info {
unsigned long pci_mem_start;
unsigned long pci_mem_end;
@@ -325,6 +336,13 @@
u16 mtu;
bool wol_support;
+
+ enum qed_dev_type dev_type;
+
+ /* Output parameters for qede */
+ bool vxlan_enable;
+ bool gre_enable;
+ bool geneve_enable;
};
enum qed_sb_type {
@@ -421,6 +439,7 @@
};
struct qed_common_cb_ops {
+ void (*arfs_filter_op)(void *dev, void *fltr, u8 fw_rc);
void (*link_update)(void *dev,
struct qed_link_output *link);
void (*dcbx_aen)(void *dev, struct qed_dcbx_get *get, u32 mib_type);
@@ -752,7 +771,7 @@
QED_MF_NPAR,
};
-struct qed_eth_stats {
+struct qed_eth_stats_common {
u64 no_buff_discards;
u64 packet_too_big_discard;
u64 ttl0_discard;
@@ -784,11 +803,6 @@
u64 rx_256_to_511_byte_packets;
u64 rx_512_to_1023_byte_packets;
u64 rx_1024_to_1518_byte_packets;
- u64 rx_1519_to_1522_byte_packets;
- u64 rx_1519_to_2047_byte_packets;
- u64 rx_2048_to_4095_byte_packets;
- u64 rx_4096_to_9216_byte_packets;
- u64 rx_9217_to_16383_byte_packets;
u64 rx_crc_errors;
u64 rx_mac_crtl_frames;
u64 rx_pause_frames;
@@ -805,14 +819,8 @@
u64 tx_256_to_511_byte_packets;
u64 tx_512_to_1023_byte_packets;
u64 tx_1024_to_1518_byte_packets;
- u64 tx_1519_to_2047_byte_packets;
- u64 tx_2048_to_4095_byte_packets;
- u64 tx_4096_to_9216_byte_packets;
- u64 tx_9217_to_16383_byte_packets;
u64 tx_pause_frames;
u64 tx_pfc_frames;
- u64 tx_lpi_entry_count;
- u64 tx_total_collisions;
u64 brb_truncates;
u64 brb_discards;
u64 rx_mac_bytes;
@@ -827,6 +835,34 @@
u64 tx_mac_ctrl_frames;
};
+struct qed_eth_stats_bb {
+ u64 rx_1519_to_1522_byte_packets;
+ u64 rx_1519_to_2047_byte_packets;
+ u64 rx_2048_to_4095_byte_packets;
+ u64 rx_4096_to_9216_byte_packets;
+ u64 rx_9217_to_16383_byte_packets;
+ u64 tx_1519_to_2047_byte_packets;
+ u64 tx_2048_to_4095_byte_packets;
+ u64 tx_4096_to_9216_byte_packets;
+ u64 tx_9217_to_16383_byte_packets;
+ u64 tx_lpi_entry_count;
+ u64 tx_total_collisions;
+};
+
+struct qed_eth_stats_ah {
+ u64 rx_1519_to_max_byte_packets;
+ u64 tx_1519_to_max_byte_packets;
+};
+
+struct qed_eth_stats {
+ struct qed_eth_stats_common common;
+
+ union {
+ struct qed_eth_stats_bb bb;
+ struct qed_eth_stats_ah ah;
+ };
+};
+
#define QED_SB_IDX 0x0002
#define RX_PI 0
diff --git a/include/linux/qed/qed_iscsi_if.h b/include/linux/qed/qed_iscsi_if.h
index f70bb81..3414649 100644
--- a/include/linux/qed/qed_iscsi_if.h
+++ b/include/linux/qed/qed_iscsi_if.h
@@ -67,6 +67,8 @@
void __iomem *primary_dbq_rq_addr;
void __iomem *secondary_bdq_rq_addr;
+
+ u8 num_cqs;
};
struct qed_iscsi_id_params {
diff --git a/include/linux/qed/qed_roce_if.h b/include/linux/qed/qed_roce_if.h
index f742d43..cbb2ff0 100644
--- a/include/linux/qed/qed_roce_if.h
+++ b/include/linux/qed/qed_roce_if.h
@@ -240,6 +240,7 @@
u64 dpi_addr;
u64 dpi_phys_addr;
u32 dpi_size;
+ u16 wid_count;
};
enum roce_mode {
@@ -533,6 +534,7 @@
struct qed_dev_rdma_info {
struct qed_dev_info common;
enum qed_rdma_type rdma_type;
+ u8 user_dpm_enabled;
};
struct qed_rdma_ops {
diff --git a/include/linux/qed/rdma_common.h b/include/linux/qed/rdma_common.h
index f773aa5..72c770f 100644
--- a/include/linux/qed/rdma_common.h
+++ b/include/linux/qed/rdma_common.h
@@ -52,7 +52,8 @@
#define RDMA_MAX_PDS (64 * 1024)
#define RDMA_NUM_STATISTIC_COUNTERS MAX_NUM_VPORTS
-#define RDMA_NUM_STATISTIC_COUNTERS_BB MAX_NUM_VPORTS_BB
+#define RDMA_NUM_STATISTIC_COUNTERS_K2 MAX_NUM_VPORTS_K2
+#define RDMA_NUM_STATISTIC_COUNTERS_BB MAX_NUM_VPORTS_BB
#define RDMA_TASK_TYPE (PROTOCOLID_ROCE)
diff --git a/include/linux/qed/roce_common.h b/include/linux/qed/roce_common.h
index bad02df..866f063 100644
--- a/include/linux/qed/roce_common.h
+++ b/include/linux/qed/roce_common.h
@@ -38,4 +38,21 @@
#define ROCE_MAX_QPS (32 * 1024)
+enum roce_async_events_type {
+ ROCE_ASYNC_EVENT_NONE = 0,
+ ROCE_ASYNC_EVENT_COMM_EST = 1,
+ ROCE_ASYNC_EVENT_SQ_DRAINED,
+ ROCE_ASYNC_EVENT_SRQ_LIMIT,
+ ROCE_ASYNC_EVENT_LAST_WQE_REACHED,
+ ROCE_ASYNC_EVENT_CQ_ERR,
+ ROCE_ASYNC_EVENT_LOCAL_INVALID_REQUEST_ERR,
+ ROCE_ASYNC_EVENT_LOCAL_CATASTROPHIC_ERR,
+ ROCE_ASYNC_EVENT_LOCAL_ACCESS_ERR,
+ ROCE_ASYNC_EVENT_QP_CATASTROPHIC_ERR,
+ ROCE_ASYNC_EVENT_CQ_OVERFLOW_ERR,
+ ROCE_ASYNC_EVENT_SRQ_EMPTY,
+ ROCE_ASYNC_EVENT_DESTROY_QP_DONE,
+ MAX_ROCE_ASYNC_EVENTS_TYPE
+};
+
#endif /* __ROCE_COMMON__ */
diff --git a/include/linux/qed/storage_common.h b/include/linux/qed/storage_common.h
index 03f3e37..08df82a 100644
--- a/include/linux/qed/storage_common.h
+++ b/include/linux/qed/storage_common.h
@@ -40,6 +40,8 @@
#define BDQ_ID_IMM_DATA (1)
#define BDQ_NUM_IDS (2)
+#define SCSI_NUM_SGES_SLOW_SGL_THR 8
+
#define BDQ_MAX_EXTERNAL_RING_SIZE (1 << 15)
struct scsi_bd {
@@ -52,6 +54,16 @@
__le16 reserved0[3];
};
+struct scsi_sge {
+ struct regpair sge_addr;
+ __le32 sge_len;
+ __le32 reserved;
+};
+
+struct scsi_cached_sges {
+ struct scsi_sge sge[4];
+};
+
struct scsi_drv_cmdq {
__le16 cmdq_cons;
__le16 reserved0;
@@ -99,11 +111,19 @@
struct scsi_bdq_ram_drv_data drv_data_per_bdq_id[BDQ_NUM_IDS];
};
-struct scsi_sge {
- struct regpair sge_addr;
- __le16 sge_len;
- __le16 reserved0;
- __le32 reserved1;
+enum scsi_sgl_mode {
+ SCSI_TX_SLOW_SGL,
+ SCSI_FAST_SGL,
+ MAX_SCSI_SGL_MODE
+};
+
+struct scsi_sgl_params {
+ struct regpair sgl_addr;
+ __le32 sgl_total_length;
+ __le32 sge_offset;
+ __le16 sgl_num_sges;
+ u8 sgl_index;
+ u8 reserved;
};
struct scsi_terminate_extra_params {
diff --git a/include/linux/qed/tcp_common.h b/include/linux/qed/tcp_common.h
index 46fe785..a5e8432 100644
--- a/include/linux/qed/tcp_common.h
+++ b/include/linux/qed/tcp_common.h
@@ -173,6 +173,7 @@
TCP_EVENT_ADD_ISLE_RIGHT,
TCP_EVENT_ADD_ISLE_LEFT,
TCP_EVENT_JOIN,
+ TCP_EVENT_DELETE_ISLES,
TCP_EVENT_NOP,
MAX_TCP_SEG_PLACEMENT_EVENT
};
diff --git a/include/linux/ras.h b/include/linux/ras.h
index 2aceeaf..ffb1471 100644
--- a/include/linux/ras.h
+++ b/include/linux/ras.h
@@ -1,14 +1,25 @@
#ifndef __RAS_H__
#define __RAS_H__
+#include <asm/errno.h>
+
#ifdef CONFIG_DEBUG_FS
int ras_userspace_consumers(void);
void ras_debugfs_init(void);
int ras_add_daemon_trace(void);
#else
static inline int ras_userspace_consumers(void) { return 0; }
-static inline void ras_debugfs_init(void) { return; }
+static inline void ras_debugfs_init(void) { }
static inline int ras_add_daemon_trace(void) { return 0; }
#endif
+#ifdef CONFIG_RAS_CEC
+void __init cec_init(void);
+int __init parse_cec_param(char *str);
+int cec_add_elem(u64 pfn);
+#else
+static inline void __init cec_init(void) { }
+static inline int cec_add_elem(u64 pfn) { return -ENODEV; }
#endif
+
+#endif /* __RAS_H__ */
diff --git a/include/linux/refcount.h b/include/linux/refcount.h
index 0023fee..b34aa64 100644
--- a/include/linux/refcount.h
+++ b/include/linux/refcount.h
@@ -6,17 +6,36 @@
#include <linux/spinlock.h>
#include <linux/kernel.h>
+/**
+ * refcount_t - variant of atomic_t specialized for reference counts
+ * @refs: atomic_t counter field
+ *
+ * The counter saturates at UINT_MAX and will not move once
+ * there. This avoids wrapping the counter and causing 'spurious'
+ * use-after-free bugs.
+ */
typedef struct refcount_struct {
atomic_t refs;
} refcount_t;
#define REFCOUNT_INIT(n) { .refs = ATOMIC_INIT(n), }
+/**
+ * refcount_set - set a refcount's value
+ * @r: the refcount
+ * @n: value to which the refcount will be set
+ */
static inline void refcount_set(refcount_t *r, unsigned int n)
{
atomic_set(&r->refs, n);
}
+/**
+ * refcount_read - get a refcount's value
+ * @r: the refcount
+ *
+ * Return: the refcount's value
+ */
static inline unsigned int refcount_read(const refcount_t *r)
{
return atomic_read(&r->refs);
diff --git a/include/linux/rhashtable.h b/include/linux/rhashtable.h
index 092292b..7d56a7e 100644
--- a/include/linux/rhashtable.h
+++ b/include/linux/rhashtable.h
@@ -49,6 +49,21 @@
/* Base bits plus 1 bit for nulls marker */
#define RHT_HASH_RESERVED_SPACE (RHT_BASE_BITS + 1)
+/* Maximum chain length before rehash
+ *
+ * The maximum (not average) chain length grows with the size of the hash
+ * table, at a rate of (log N)/(log log N).
+ *
+ * The value of 16 is selected so that even if the hash table grew to
+ * 2^32 you would not expect the maximum chain length to exceed it
+ * unless we are under attack (or extremely unlucky).
+ *
+ * As this limit is only to detect attacks, we don't need to set it to a
+ * lower value as you'd need the chain length to vastly exceed 16 to have
+ * any real effect on the system.
+ */
+#define RHT_ELASTICITY 16u
+
struct rhash_head {
struct rhash_head __rcu *next;
};
@@ -110,29 +125,25 @@
* @key_len: Length of key
* @key_offset: Offset of key in struct to be hashed
* @head_offset: Offset of rhash_head in struct to be hashed
- * @insecure_max_entries: Maximum number of entries (may be exceeded)
* @max_size: Maximum size while expanding
* @min_size: Minimum size while shrinking
- * @nulls_base: Base value to generate nulls marker
- * @insecure_elasticity: Set to true to disable chain length checks
- * @automatic_shrinking: Enable automatic shrinking of tables
* @locks_mul: Number of bucket locks to allocate per cpu (default: 128)
+ * @automatic_shrinking: Enable automatic shrinking of tables
+ * @nulls_base: Base value to generate nulls marker
* @hashfn: Hash function (default: jhash2 if !(key_len % 4), or jhash)
* @obj_hashfn: Function to hash object
* @obj_cmpfn: Function to compare key with object
*/
struct rhashtable_params {
- size_t nelem_hint;
- size_t key_len;
- size_t key_offset;
- size_t head_offset;
- unsigned int insecure_max_entries;
+ u16 nelem_hint;
+ u16 key_len;
+ u16 key_offset;
+ u16 head_offset;
unsigned int max_size;
- unsigned int min_size;
- u32 nulls_base;
- bool insecure_elasticity;
+ u16 min_size;
bool automatic_shrinking;
- size_t locks_mul;
+ u8 locks_mul;
+ u32 nulls_base;
rht_hashfn_t hashfn;
rht_obj_hashfn_t obj_hashfn;
rht_obj_cmpfn_t obj_cmpfn;
@@ -143,8 +154,8 @@
* @tbl: Bucket table
* @nelems: Number of elements in table
* @key_len: Key length for hashfn
- * @elasticity: Maximum chain length before rehash
* @p: Configuration parameters
+ * @max_elems: Maximum number of elements in table
* @rhlist: True if this is an rhltable
* @run_work: Deferred worker to expand/shrink asynchronously
* @mutex: Mutex to protect current/future table swapping
@@ -154,8 +165,8 @@
struct bucket_table __rcu *tbl;
atomic_t nelems;
unsigned int key_len;
- unsigned int elasticity;
struct rhashtable_params p;
+ unsigned int max_elems;
bool rhlist;
struct work_struct run_work;
struct mutex mutex;
@@ -318,8 +329,7 @@
static inline bool rht_grow_above_max(const struct rhashtable *ht,
const struct bucket_table *tbl)
{
- return ht->p.insecure_max_entries &&
- atomic_read(&ht->nelems) >= ht->p.insecure_max_entries;
+ return atomic_read(&ht->nelems) >= ht->max_elems;
}
/* The bucket lock is selected based on the hash and protects mutations
@@ -726,7 +736,7 @@
return rhashtable_insert_slow(ht, key, obj);
}
- elasticity = ht->elasticity;
+ elasticity = RHT_ELASTICITY;
pprev = rht_bucket_insert(ht, tbl, hash);
data = ERR_PTR(-ENOMEM);
if (!pprev)
@@ -916,6 +926,28 @@
}
/**
+ * rhashtable_lookup_get_insert_fast - lookup and insert object into hash table
+ * @ht: hash table
+ * @obj: pointer to hash head inside object
+ * @params: hash table parameters
+ *
+ * Just like rhashtable_lookup_insert_fast(), but this function returns the
+ * object if it exists, NULL if it did not and the insertion was successful,
+ * and an ERR_PTR otherwise.
+ */
+static inline void *rhashtable_lookup_get_insert_fast(
+ struct rhashtable *ht, struct rhash_head *obj,
+ const struct rhashtable_params params)
+{
+ const char *key = rht_obj(ht, obj);
+
+ BUG_ON(ht->p.obj_hashfn);
+
+ return __rhashtable_insert_fast(ht, key + ht->p.key_offset, obj, params,
+ false);
+}
+
+/**
* rhashtable_lookup_insert_key - search and insert object to hash table
* with explicit key
* @ht: hash table
diff --git a/include/linux/rpmsg/qcom_smd.h b/include/linux/rpmsg/qcom_smd.h
index 8ec8b64..f27917e 100644
--- a/include/linux/rpmsg/qcom_smd.h
+++ b/include/linux/rpmsg/qcom_smd.h
@@ -6,7 +6,7 @@
struct qcom_smd_edge;
-#if IS_ENABLED(CONFIG_RPMSG_QCOM_SMD) || IS_ENABLED(CONFIG_QCOM_SMD)
+#if IS_ENABLED(CONFIG_RPMSG_QCOM_SMD)
struct qcom_smd_edge *qcom_smd_register_edge(struct device *parent,
struct device_node *node);
diff --git a/include/linux/sbitmap.h b/include/linux/sbitmap.h
index d4e0a20..a1904aa 100644
--- a/include/linux/sbitmap.h
+++ b/include/linux/sbitmap.h
@@ -176,6 +176,25 @@
int sbitmap_get(struct sbitmap *sb, unsigned int alloc_hint, bool round_robin);
/**
+ * sbitmap_get_shallow() - Try to allocate a free bit from a &struct sbitmap,
+ * limiting the depth used from each word.
+ * @sb: Bitmap to allocate from.
+ * @alloc_hint: Hint for where to start searching for a free bit.
+ * @shallow_depth: The maximum number of bits to allocate from a single word.
+ *
+ * This rather specific operation allows for having multiple users with
+ * different allocation limits. E.g., there can be a high-priority class that
+ * uses sbitmap_get() and a low-priority class that uses sbitmap_get_shallow()
+ * with a @shallow_depth of (1 << (@sb->shift - 1)). Then, the low-priority
+ * class can only allocate half of the total bits in the bitmap, preventing it
+ * from starving out the high-priority class.
+ *
+ * Return: Non-negative allocated bit number if successful, -1 otherwise.
+ */
+int sbitmap_get_shallow(struct sbitmap *sb, unsigned int alloc_hint,
+ unsigned long shallow_depth);
+
+/**
* sbitmap_any_bit_set() - Check for a set bit in a &struct sbitmap.
* @sb: Bitmap to check.
*
@@ -326,6 +345,19 @@
int __sbitmap_queue_get(struct sbitmap_queue *sbq);
/**
+ * __sbitmap_queue_get_shallow() - Try to allocate a free bit from a &struct
+ * sbitmap_queue, limiting the depth used from each word, with preemption
+ * already disabled.
+ * @sbq: Bitmap queue to allocate from.
+ * @shallow_depth: The maximum number of bits to allocate from a single word.
+ * See sbitmap_get_shallow().
+ *
+ * Return: Non-negative allocated bit number if successful, -1 otherwise.
+ */
+int __sbitmap_queue_get_shallow(struct sbitmap_queue *sbq,
+ unsigned int shallow_depth);
+
+/**
* sbitmap_queue_get() - Try to allocate a free bit from a &struct
* sbitmap_queue.
* @sbq: Bitmap queue to allocate from.
@@ -346,6 +378,29 @@
}
/**
+ * sbitmap_queue_get_shallow() - Try to allocate a free bit from a &struct
+ * sbitmap_queue, limiting the depth used from each word.
+ * @sbq: Bitmap queue to allocate from.
+ * @cpu: Output parameter; will contain the CPU we ran on (e.g., to be passed to
+ * sbitmap_queue_clear()).
+ * @shallow_depth: The maximum number of bits to allocate from a single word.
+ * See sbitmap_get_shallow().
+ *
+ * Return: Non-negative allocated bit number if successful, -1 otherwise.
+ */
+static inline int sbitmap_queue_get_shallow(struct sbitmap_queue *sbq,
+ unsigned int *cpu,
+ unsigned int shallow_depth)
+{
+ int nr;
+
+ *cpu = get_cpu();
+ nr = __sbitmap_queue_get_shallow(sbq, shallow_depth);
+ put_cpu();
+ return nr;
+}
+
+/**
* sbitmap_queue_clear() - Free an allocated bit and wake up waiters on a
* &struct sbitmap_queue.
* @sbq: Bitmap to free from.
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 4cf9a59..1e590c3 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -186,7 +186,7 @@
extern void io_schedule(void);
/**
- * struct prev_cputime - snaphsot of system and user cputime
+ * struct prev_cputime - snapshot of system and user cputime
* @utime: time spent in user mode
* @stime: time spent in system mode
* @lock: protects the above two fields
@@ -779,6 +779,8 @@
/* PI waiters blocked on a rt_mutex held by this task: */
struct rb_root pi_waiters;
struct rb_node *pi_waiters_leftmost;
+ /* Updated under owner's pi_lock and rq lock */
+ struct task_struct *pi_top_task;
/* Deadlock detection and priority inheritance handling: */
struct rt_mutex_waiter *pi_blocked_on;
#endif
@@ -1042,6 +1044,9 @@
/* A live task holds one reference: */
atomic_t stack_refcount;
#endif
+#ifdef CONFIG_LIVEPATCH
+ int patch_state;
+#endif
/* CPU-specific state of this task: */
struct thread_struct thread;
@@ -1290,10 +1295,10 @@
TASK_PFA_SET(LMK_WAITING, lmk_waiting)
static inline void
-tsk_restore_flags(struct task_struct *task, unsigned long orig_flags, unsigned long flags)
+current_restore_flags(unsigned long orig_flags, unsigned long flags)
{
- task->flags &= ~flags;
- task->flags |= orig_flags & flags;
+ current->flags &= ~flags;
+ current->flags |= orig_flags & flags;
}
extern int cpuset_cpumask_can_shrink(const struct cpumask *cur, const struct cpumask *trial);
diff --git a/include/linux/sched/rt.h b/include/linux/sched/rt.h
index 3bd66841..f93329a 100644
--- a/include/linux/sched/rt.h
+++ b/include/linux/sched/rt.h
@@ -18,27 +18,20 @@
}
#ifdef CONFIG_RT_MUTEXES
-extern int rt_mutex_getprio(struct task_struct *p);
-extern void rt_mutex_setprio(struct task_struct *p, int prio);
-extern int rt_mutex_get_effective_prio(struct task_struct *task, int newprio);
-extern struct task_struct *rt_mutex_get_top_task(struct task_struct *task);
+/*
+ * Must hold either p->pi_lock or task_rq(p)->lock.
+ */
+static inline struct task_struct *rt_mutex_get_top_task(struct task_struct *p)
+{
+ return p->pi_top_task;
+}
+extern void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task);
extern void rt_mutex_adjust_pi(struct task_struct *p);
static inline bool tsk_is_pi_blocked(struct task_struct *tsk)
{
return tsk->pi_blocked_on != NULL;
}
#else
-static inline int rt_mutex_getprio(struct task_struct *p)
-{
- return p->normal_prio;
-}
-
-static inline int rt_mutex_get_effective_prio(struct task_struct *task,
- int newprio)
-{
- return newprio;
-}
-
static inline struct task_struct *rt_mutex_get_top_task(struct task_struct *task)
{
return NULL;
diff --git a/include/linux/serdev.h b/include/linux/serdev.h
index 9519da6..37395b8 100644
--- a/include/linux/serdev.h
+++ b/include/linux/serdev.h
@@ -15,6 +15,8 @@
#include <linux/types.h>
#include <linux/device.h>
+#include <linux/termios.h>
+#include <linux/delay.h>
struct serdev_controller;
struct serdev_device;
@@ -81,6 +83,9 @@
void (*close)(struct serdev_controller *);
void (*set_flow_control)(struct serdev_controller *, bool);
unsigned int (*set_baudrate)(struct serdev_controller *, unsigned int);
+ void (*wait_until_sent)(struct serdev_controller *, long);
+ int (*get_tiocm)(struct serdev_controller *);
+ int (*set_tiocm)(struct serdev_controller *, unsigned int, unsigned int);
};
/**
@@ -186,6 +191,9 @@
void serdev_device_close(struct serdev_device *);
unsigned int serdev_device_set_baudrate(struct serdev_device *, unsigned int);
void serdev_device_set_flow_control(struct serdev_device *, bool);
+void serdev_device_wait_until_sent(struct serdev_device *, long);
+int serdev_device_get_tiocm(struct serdev_device *);
+int serdev_device_set_tiocm(struct serdev_device *, int, int);
int serdev_device_write_buf(struct serdev_device *, const unsigned char *, size_t);
void serdev_device_write_flush(struct serdev_device *);
int serdev_device_write_room(struct serdev_device *);
@@ -223,6 +231,15 @@
return 0;
}
static inline void serdev_device_set_flow_control(struct serdev_device *sdev, bool enable) {}
+static inline void serdev_device_wait_until_sent(struct serdev_device *sdev, long timeout) {}
+static inline int serdev_device_get_tiocm(struct serdev_device *serdev)
+{
+ return -ENOTSUPP;
+}
+static inline int serdev_device_set_tiocm(struct serdev_device *serdev, int set, int clear)
+{
+ return -ENOTSUPP;
+}
static inline int serdev_device_write_buf(struct serdev_device *sdev, const unsigned char *buf, size_t count)
{
return -ENODEV;
@@ -238,6 +255,36 @@
#endif /* CONFIG_SERIAL_DEV_BUS */
+static inline bool serdev_device_get_cts(struct serdev_device *serdev)
+{
+ int status = serdev_device_get_tiocm(serdev);
+ return !!(status & TIOCM_CTS);
+}
+
+static inline int serdev_device_wait_for_cts(struct serdev_device *serdev, bool state, int timeout_ms)
+{
+ unsigned long timeout;
+ bool signal;
+
+ timeout = jiffies + msecs_to_jiffies(timeout_ms);
+ while (time_is_after_jiffies(timeout)) {
+ signal = serdev_device_get_cts(serdev);
+ if (signal == state)
+ return 0;
+ usleep_range(1000, 2000);
+ }
+
+ return -ETIMEDOUT;
+}
+
+static inline int serdev_device_set_rts(struct serdev_device *serdev, bool enable)
+{
+ if (enable)
+ return serdev_device_set_tiocm(serdev, TIOCM_RTS, 0);
+ else
+ return serdev_device_set_tiocm(serdev, 0, TIOCM_RTS);
+}
+
/*
* serdev hooks into TTY core
*/
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index c776abd..a098d95 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -413,14 +413,15 @@
* the end of the header data, ie. at skb->end.
*/
struct skb_shared_info {
+ unsigned short _unused;
unsigned char nr_frags;
__u8 tx_flags;
unsigned short gso_size;
/* Warning: this field is not always filled in (UFO)! */
unsigned short gso_segs;
- unsigned short gso_type;
struct sk_buff *frag_list;
struct skb_shared_hwtstamps hwtstamps;
+ unsigned int gso_type;
u32 tskey;
__be32 ip6_frag_id;
@@ -491,6 +492,8 @@
SKB_GSO_TUNNEL_REMCSUM = 1 << 14,
SKB_GSO_SCTP = 1 << 15,
+
+ SKB_GSO_ESP = 1 << 16,
};
#if BITS_PER_LONG > 32
@@ -3113,7 +3116,7 @@
static inline int memcpy_from_msg(void *data, struct msghdr *msg, int len)
{
- return copy_from_iter(data, len, &msg->msg_iter) == len ? 0 : -EFAULT;
+ return copy_from_iter_full(data, len, &msg->msg_iter) ? 0 : -EFAULT;
}
static inline int memcpy_to_msg(struct msghdr *msg, void *data, int len)
diff --git a/include/linux/smp.h b/include/linux/smp.h
index 8e0cb7a..68123c1 100644
--- a/include/linux/smp.h
+++ b/include/linux/smp.h
@@ -120,6 +120,13 @@
extern void __init setup_nr_cpu_ids(void);
extern void __init smp_init(void);
+extern int __boot_cpu_id;
+
+static inline int get_boot_cpu_id(void)
+{
+ return __boot_cpu_id;
+}
+
#else /* !SMP */
static inline void smp_send_stop(void) { }
@@ -158,6 +165,11 @@
static inline void smp_init(void) { }
#endif
+static inline int get_boot_cpu_id(void)
+{
+ return 0;
+}
+
#endif /* !SMP */
/*
diff --git a/include/linux/soc/qcom/smd.h b/include/linux/soc/qcom/smd.h
deleted file mode 100644
index f148e0f..0000000
--- a/include/linux/soc/qcom/smd.h
+++ /dev/null
@@ -1,139 +0,0 @@
-#ifndef __QCOM_SMD_H__
-#define __QCOM_SMD_H__
-
-#include <linux/device.h>
-#include <linux/mod_devicetable.h>
-
-struct qcom_smd;
-struct qcom_smd_channel;
-struct qcom_smd_lookup;
-
-/**
- * struct qcom_smd_id - struct used for matching a smd device
- * @name: name of the channel
- */
-struct qcom_smd_id {
- char name[20];
-};
-
-/**
- * struct qcom_smd_device - smd device struct
- * @dev: the device struct
- * @channel: handle to the smd channel for this device
- */
-struct qcom_smd_device {
- struct device dev;
- struct qcom_smd_channel *channel;
-};
-
-typedef int (*qcom_smd_cb_t)(struct qcom_smd_channel *, const void *, size_t);
-
-/**
- * struct qcom_smd_driver - smd driver struct
- * @driver: underlying device driver
- * @smd_match_table: static channel match table
- * @probe: invoked when the smd channel is found
- * @remove: invoked when the smd channel is closed
- * @callback: invoked when an inbound message is received on the channel,
- * should return 0 on success or -EBUSY if the data cannot be
- * consumed at this time
- */
-struct qcom_smd_driver {
- struct device_driver driver;
- const struct qcom_smd_id *smd_match_table;
-
- int (*probe)(struct qcom_smd_device *dev);
- void (*remove)(struct qcom_smd_device *dev);
- qcom_smd_cb_t callback;
-};
-
-#if IS_ENABLED(CONFIG_QCOM_SMD)
-
-int qcom_smd_driver_register(struct qcom_smd_driver *drv);
-void qcom_smd_driver_unregister(struct qcom_smd_driver *drv);
-
-struct qcom_smd_channel *qcom_smd_open_channel(struct qcom_smd_channel *channel,
- const char *name,
- qcom_smd_cb_t cb);
-void qcom_smd_close_channel(struct qcom_smd_channel *channel);
-void *qcom_smd_get_drvdata(struct qcom_smd_channel *channel);
-void qcom_smd_set_drvdata(struct qcom_smd_channel *channel, void *data);
-int qcom_smd_send(struct qcom_smd_channel *channel, const void *data, int len);
-
-
-struct qcom_smd_edge *qcom_smd_register_edge(struct device *parent,
- struct device_node *node);
-int qcom_smd_unregister_edge(struct qcom_smd_edge *edge);
-
-#else
-
-static inline int qcom_smd_driver_register(struct qcom_smd_driver *drv)
-{
- return -ENXIO;
-}
-
-static inline void qcom_smd_driver_unregister(struct qcom_smd_driver *drv)
-{
- /* This shouldn't be possible */
- WARN_ON(1);
-}
-
-static inline struct qcom_smd_channel *
-qcom_smd_open_channel(struct qcom_smd_channel *channel,
- const char *name,
- qcom_smd_cb_t cb)
-{
- /* This shouldn't be possible */
- WARN_ON(1);
- return NULL;
-}
-
-static inline void qcom_smd_close_channel(struct qcom_smd_channel *channel)
-{
- /* This shouldn't be possible */
- WARN_ON(1);
-}
-
-static inline void *qcom_smd_get_drvdata(struct qcom_smd_channel *channel)
-{
- /* This shouldn't be possible */
- WARN_ON(1);
- return NULL;
-}
-
-static inline void qcom_smd_set_drvdata(struct qcom_smd_channel *channel, void *data)
-{
- /* This shouldn't be possible */
- WARN_ON(1);
-}
-
-static inline int qcom_smd_send(struct qcom_smd_channel *channel,
- const void *data, int len)
-{
- /* This shouldn't be possible */
- WARN_ON(1);
- return -ENXIO;
-}
-
-static inline struct qcom_smd_edge *
-qcom_smd_register_edge(struct device *parent,
- struct device_node *node)
-{
- return ERR_PTR(-ENXIO);
-}
-
-static inline int qcom_smd_unregister_edge(struct qcom_smd_edge *edge)
-{
- /* This shouldn't be possible */
- WARN_ON(1);
- return -ENXIO;
-}
-
-#endif
-
-#define module_qcom_smd_driver(__smd_driver) \
- module_driver(__smd_driver, qcom_smd_driver_register, \
- qcom_smd_driver_unregister)
-
-
-#endif
diff --git a/include/linux/soc/qcom/wcnss_ctrl.h b/include/linux/soc/qcom/wcnss_ctrl.h
index eab6497..a4dd4d7c 100644
--- a/include/linux/soc/qcom/wcnss_ctrl.h
+++ b/include/linux/soc/qcom/wcnss_ctrl.h
@@ -1,16 +1,19 @@
#ifndef __WCNSS_CTRL_H__
#define __WCNSS_CTRL_H__
-#include <linux/soc/qcom/smd.h>
+#include <linux/rpmsg.h>
#if IS_ENABLED(CONFIG_QCOM_WCNSS_CTRL)
-struct qcom_smd_channel *qcom_wcnss_open_channel(void *wcnss, const char *name, qcom_smd_cb_t cb);
+struct rpmsg_endpoint *qcom_wcnss_open_channel(void *wcnss, const char *name,
+ rpmsg_rx_cb_t cb, void *priv);
#else
-static inline struct qcom_smd_channel*
-qcom_wcnss_open_channel(void *wcnss, const char *name, qcom_smd_cb_t cb)
+static struct rpmsg_endpoint *qcom_wcnss_open_channel(void *wcnss,
+ const char *name,
+ rpmsg_rx_cb_t cb,
+ void *priv)
{
WARN_ON(1);
return ERR_PTR(-ENXIO);
diff --git a/include/linux/sock_diag.h b/include/linux/sock_diag.h
index a0596ca0..a2f8109 100644
--- a/include/linux/sock_diag.h
+++ b/include/linux/sock_diag.h
@@ -24,6 +24,7 @@
void sock_diag_register_inet_compat(int (*fn)(struct sk_buff *skb, struct nlmsghdr *nlh));
void sock_diag_unregister_inet_compat(int (*fn)(struct sk_buff *skb, struct nlmsghdr *nlh));
+u64 sock_gen_cookie(struct sock *sk);
int sock_diag_check_cookie(struct sock *sk, const __u32 *cookie);
void sock_diag_save_cookie(struct sock *sk, __u32 *cookie);
diff --git a/include/linux/splice.h b/include/linux/splice.h
index 00a2116..db42746 100644
--- a/include/linux/splice.h
+++ b/include/linux/splice.h
@@ -20,6 +20,8 @@
#define SPLICE_F_MORE (0x04) /* expect more data */
#define SPLICE_F_GIFT (0x08) /* pages passed in are a gift */
+#define SPLICE_F_ALL (SPLICE_F_MOVE|SPLICE_F_NONBLOCK|SPLICE_F_MORE|SPLICE_F_GIFT)
+
/*
* Passed to the actors
*/
@@ -55,7 +57,6 @@
struct partial_page *partial; /* pages[] may not be contig */
int nr_pages; /* number of populated pages in map */
unsigned int nr_pages_max; /* pages[] & partial[] arrays size */
- unsigned int flags; /* splice flags */
const struct pipe_buf_operations *ops;/* ops associated with output pipe */
void (*spd_release)(struct splice_pipe_desc *, unsigned int);
};
@@ -82,7 +83,6 @@
*/
extern int splice_grow_spd(const struct pipe_inode_info *, struct splice_pipe_desc *);
extern void splice_shrink_spd(struct splice_pipe_desc *);
-extern void spd_release_page(struct splice_pipe_desc *, unsigned int);
extern const struct pipe_buf_operations page_cache_pipe_buf_ops;
extern const struct pipe_buf_operations default_pipe_buf_ops;
diff --git a/include/linux/stacktrace.h b/include/linux/stacktrace.h
index 0a34489..4205f71 100644
--- a/include/linux/stacktrace.h
+++ b/include/linux/stacktrace.h
@@ -18,6 +18,8 @@
struct stack_trace *trace);
extern void save_stack_trace_tsk(struct task_struct *tsk,
struct stack_trace *trace);
+extern int save_stack_trace_tsk_reliable(struct task_struct *tsk,
+ struct stack_trace *trace);
extern void print_stack_trace(struct stack_trace *trace, int spaces);
extern int snprint_stack_trace(char *buf, size_t size,
@@ -29,12 +31,13 @@
# define save_stack_trace_user(trace) do { } while (0)
#endif
-#else
+#else /* !CONFIG_STACKTRACE */
# define save_stack_trace(trace) do { } while (0)
# define save_stack_trace_tsk(tsk, trace) do { } while (0)
# define save_stack_trace_user(trace) do { } while (0)
# define print_stack_trace(trace, spaces) do { } while (0)
# define snprint_stack_trace(buf, size, trace, spaces) do { } while (0)
-#endif
+# define save_stack_trace_tsk_reliable(tsk, trace) ({ -ENOSYS; })
+#endif /* CONFIG_STACKTRACE */
-#endif
+#endif /* __LINUX_STACKTRACE_H */
diff --git a/include/linux/stmmac.h b/include/linux/stmmac.h
index fc273e9..3921cb9 100644
--- a/include/linux/stmmac.h
+++ b/include/linux/stmmac.h
@@ -28,6 +28,9 @@
#include <linux/platform_device.h>
+#define MTL_MAX_RX_QUEUES 8
+#define MTL_MAX_TX_QUEUES 8
+
#define STMMAC_RX_COE_NONE 0
#define STMMAC_RX_COE_TYPE1 1
#define STMMAC_RX_COE_TYPE2 2
@@ -44,6 +47,18 @@
#define STMMAC_CSR_150_250M 0x4 /* MDC = clk_scr_i/102 */
#define STMMAC_CSR_250_300M 0x5 /* MDC = clk_scr_i/122 */
+/* MTL algorithms identifiers */
+#define MTL_TX_ALGORITHM_WRR 0x0
+#define MTL_TX_ALGORITHM_WFQ 0x1
+#define MTL_TX_ALGORITHM_DWRR 0x2
+#define MTL_TX_ALGORITHM_SP 0x3
+#define MTL_RX_ALGORITHM_SP 0x4
+#define MTL_RX_ALGORITHM_WSP 0x5
+
+/* RX/TX Queue Mode */
+#define MTL_QUEUE_AVB 0x0
+#define MTL_QUEUE_DCB 0x1
+
/* The MDC clock could be set higher than the IEEE 802.3
* specified frequency limit 0f 2.5 MHz, by programming a clock divider
* of value different than the above defined values. The resultant MDIO
@@ -109,6 +124,26 @@
bool axi_rb;
};
+struct stmmac_rxq_cfg {
+ u8 mode_to_use;
+ u8 chan;
+ u8 pkt_route;
+ bool use_prio;
+ u32 prio;
+};
+
+struct stmmac_txq_cfg {
+ u8 weight;
+ u8 mode_to_use;
+ /* Credit Base Shaper parameters */
+ u32 send_slope;
+ u32 idle_slope;
+ u32 high_credit;
+ u32 low_credit;
+ bool use_prio;
+ u32 prio;
+};
+
struct plat_stmmacenet_data {
int bus_id;
int phy_addr;
@@ -133,6 +168,12 @@
int unicast_filter_entries;
int tx_fifo_size;
int rx_fifo_size;
+ u8 rx_queues_to_use;
+ u8 tx_queues_to_use;
+ u8 rx_sched_algorithm;
+ u8 tx_sched_algorithm;
+ struct stmmac_rxq_cfg rx_queues_cfg[MTL_MAX_RX_QUEUES];
+ struct stmmac_txq_cfg tx_queues_cfg[MTL_MAX_TX_QUEUES];
void (*fix_mac_speed)(void *priv, unsigned int speed);
int (*init)(struct platform_device *pdev, void *priv);
void (*exit)(struct platform_device *pdev, void *priv);
diff --git a/include/linux/t10-pi.h b/include/linux/t10-pi.h
index 9fba9dd..9375d23 100644
--- a/include/linux/t10-pi.h
+++ b/include/linux/t10-pi.h
@@ -34,9 +34,9 @@
};
-extern struct blk_integrity_profile t10_pi_type1_crc;
-extern struct blk_integrity_profile t10_pi_type1_ip;
-extern struct blk_integrity_profile t10_pi_type3_crc;
-extern struct blk_integrity_profile t10_pi_type3_ip;
+extern const struct blk_integrity_profile t10_pi_type1_crc;
+extern const struct blk_integrity_profile t10_pi_type1_ip;
+extern const struct blk_integrity_profile t10_pi_type3_crc;
+extern const struct blk_integrity_profile t10_pi_type3_ip;
#endif
diff --git a/include/linux/tcp.h b/include/linux/tcp.h
index cfc2d95..b6d5adce 100644
--- a/include/linux/tcp.h
+++ b/include/linux/tcp.h
@@ -233,12 +233,14 @@
u8 syn_data:1, /* SYN includes data */
syn_fastopen:1, /* SYN includes Fast Open option */
syn_fastopen_exp:1,/* SYN includes Fast Open exp. option */
+ syn_fastopen_ch:1, /* Active TFO re-enabling probe */
syn_data_acked:1,/* data in SYN is acked by SYN-ACK */
save_syn:1, /* Save headers of SYN packet */
is_cwnd_limited:1;/* forward progress limited by snd_cwnd? */
u32 tlp_high_seq; /* snd_nxt at the time of TLP retransmit. */
/* RTT measurement */
+ struct skb_mstamp tcp_mstamp; /* most recent packet received/sent */
u32 srtt_us; /* smoothed round trip time << 3 in usecs */
u32 mdev_us; /* medium deviation */
u32 mdev_max_us; /* maximal mdev for the last rtt period */
@@ -331,16 +333,16 @@
/* Receiver side RTT estimation */
struct {
- u32 rtt;
- u32 seq;
- u32 time;
+ u32 rtt_us;
+ u32 seq;
+ struct skb_mstamp time;
} rcv_rtt_est;
/* Receiver queue space */
struct {
- int space;
- u32 seq;
- u32 time;
+ int space;
+ u32 seq;
+ struct skb_mstamp time;
} rcvq_space;
/* TCP-specific MTU probe information. */
diff --git a/include/linux/thread_info.h b/include/linux/thread_info.h
index 5837387..d7d3ea6 100644
--- a/include/linux/thread_info.h
+++ b/include/linux/thread_info.h
@@ -22,6 +22,18 @@
#endif
#include <linux/bitops.h>
+
+/*
+ * For per-arch arch_within_stack_frames() implementations, defined in
+ * asm/thread_info.h.
+ */
+enum {
+ BAD_STACK = -1,
+ NOT_STACK = 0,
+ GOOD_FRAME,
+ GOOD_STACK,
+};
+
#include <asm/thread_info.h>
#ifdef __KERNEL__
@@ -101,6 +113,10 @@
{ }
#endif /* CONFIG_HARDENED_USERCOPY */
+#ifndef arch_setup_new_exec
+static inline void arch_setup_new_exec(void) { }
+#endif
+
#endif /* __KERNEL__ */
#endif /* _LINUX_THREAD_INFO_H */
diff --git a/include/linux/tick.h b/include/linux/tick.h
index a04fea1..fe01e68 100644
--- a/include/linux/tick.h
+++ b/include/linux/tick.h
@@ -117,6 +117,7 @@
extern void tick_nohz_idle_exit(void);
extern void tick_nohz_irq_exit(void);
extern ktime_t tick_nohz_get_sleep_length(void);
+extern unsigned long tick_nohz_get_idle_calls(void);
extern u64 get_cpu_idle_time_us(int cpu, u64 *last_update_time);
extern u64 get_cpu_iowait_time_us(int cpu, u64 *last_update_time);
#else /* !CONFIG_NO_HZ_COMMON */
diff --git a/include/linux/timekeeping.h b/include/linux/timekeeping.h
index b598cbc..ddc229f 100644
--- a/include/linux/timekeeping.h
+++ b/include/linux/timekeeping.h
@@ -19,21 +19,6 @@
extern int do_settimeofday64(const struct timespec64 *ts);
extern int do_sys_settimeofday64(const struct timespec64 *tv,
const struct timezone *tz);
-static inline int do_sys_settimeofday(const struct timespec *tv,
- const struct timezone *tz)
-{
- struct timespec64 ts64;
-
- if (!tv)
- return do_sys_settimeofday64(NULL, tz);
-
- if (!timespec_valid(tv))
- return -EINVAL;
-
- ts64 = timespec_to_timespec64(*tv);
- return do_sys_settimeofday64(&ts64, tz);
-}
-
/*
* Kernel time accessors
*/
@@ -273,6 +258,11 @@
*ts = ktime_to_timespec(ktime_get_clocktai());
}
+static inline void timekeeping_clocktai64(struct timespec64 *ts)
+{
+ *ts = ktime_to_timespec64(ktime_get_clocktai());
+}
+
/*
* RTC specific
*/
diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h
index f30c187..e0cbfb09 100644
--- a/include/linux/uaccess.h
+++ b/include/linux/uaccess.h
@@ -2,8 +2,199 @@
#define __LINUX_UACCESS_H__
#include <linux/sched.h>
+#include <linux/thread_info.h>
+#include <linux/kasan-checks.h>
+
+#define VERIFY_READ 0
+#define VERIFY_WRITE 1
+
+#define uaccess_kernel() segment_eq(get_fs(), KERNEL_DS)
+
#include <asm/uaccess.h>
+/*
+ * Architectures should provide two primitives (raw_copy_{to,from}_user())
+ * and get rid of their private instances of copy_{to,from}_user() and
+ * __copy_{to,from}_user{,_inatomic}().
+ *
+ * raw_copy_{to,from}_user(to, from, size) should copy up to size bytes and
+ * return the amount left to copy. They should assume that access_ok() has
+ * already been checked (and succeeded); they should *not* zero-pad anything.
+ * No KASAN or object size checks either - those belong here.
+ *
+ * Both of these functions should attempt to copy size bytes starting at from
+ * into the area starting at to. They must not fetch or store anything
+ * outside of those areas. Return value must be between 0 (everything
+ * copied successfully) and size (nothing copied).
+ *
+ * If raw_copy_{to,from}_user(to, from, size) returns N, size - N bytes starting
+ * at to must become equal to the bytes fetched from the corresponding area
+ * starting at from. All data past to + size - N must be left unmodified.
+ *
+ * If copying succeeds, the return value must be 0. If some data cannot be
+ * fetched, it is permitted to copy less than had been fetched; the only
+ * hard requirement is that not storing anything at all (i.e. returning size)
+ * should happen only when nothing could be copied. In other words, you don't
+ * have to squeeze as much as possible - it is allowed, but not necessary.
+ *
+ * For raw_copy_from_user() to always points to kernel memory and no faults
+ * on store should happen. Interpretation of from is affected by set_fs().
+ * For raw_copy_to_user() it's the other way round.
+ *
+ * Both can be inlined - it's up to architectures whether it wants to bother
+ * with that. They should not be used directly; they are used to implement
+ * the 6 functions (copy_{to,from}_user(), __copy_{to,from}_user_inatomic())
+ * that are used instead. Out of those, __... ones are inlined. Plain
+ * copy_{to,from}_user() might or might not be inlined. If you want them
+ * inlined, have asm/uaccess.h define INLINE_COPY_{TO,FROM}_USER.
+ *
+ * NOTE: only copy_from_user() zero-pads the destination in case of short copy.
+ * Neither __copy_from_user() nor __copy_from_user_inatomic() zero anything
+ * at all; their callers absolutely must check the return value.
+ *
+ * Biarch ones should also provide raw_copy_in_user() - similar to the above,
+ * but both source and destination are __user pointers (affected by set_fs()
+ * as usual) and both source and destination can trigger faults.
+ */
+
+static __always_inline unsigned long
+__copy_from_user_inatomic(void *to, const void __user *from, unsigned long n)
+{
+ kasan_check_write(to, n);
+ check_object_size(to, n, false);
+ return raw_copy_from_user(to, from, n);
+}
+
+static __always_inline unsigned long
+__copy_from_user(void *to, const void __user *from, unsigned long n)
+{
+ might_fault();
+ kasan_check_write(to, n);
+ check_object_size(to, n, false);
+ return raw_copy_from_user(to, from, n);
+}
+
+/**
+ * __copy_to_user_inatomic: - Copy a block of data into user space, with less checking.
+ * @to: Destination address, in user space.
+ * @from: Source address, in kernel space.
+ * @n: Number of bytes to copy.
+ *
+ * Context: User context only.
+ *
+ * Copy data from kernel space to user space. Caller must check
+ * the specified block with access_ok() before calling this function.
+ * The caller should also make sure he pins the user space address
+ * so that we don't result in page fault and sleep.
+ */
+static __always_inline unsigned long
+__copy_to_user_inatomic(void __user *to, const void *from, unsigned long n)
+{
+ kasan_check_read(from, n);
+ check_object_size(from, n, true);
+ return raw_copy_to_user(to, from, n);
+}
+
+static __always_inline unsigned long
+__copy_to_user(void __user *to, const void *from, unsigned long n)
+{
+ might_fault();
+ kasan_check_read(from, n);
+ check_object_size(from, n, true);
+ return raw_copy_to_user(to, from, n);
+}
+
+#ifdef INLINE_COPY_FROM_USER
+static inline unsigned long
+_copy_from_user(void *to, const void __user *from, unsigned long n)
+{
+ unsigned long res = n;
+ if (likely(access_ok(VERIFY_READ, from, n)))
+ res = raw_copy_from_user(to, from, n);
+ if (unlikely(res))
+ memset(to + (n - res), 0, res);
+ return res;
+}
+#else
+extern unsigned long
+_copy_from_user(void *, const void __user *, unsigned long);
+#endif
+
+#ifdef INLINE_COPY_TO_USER
+static inline unsigned long
+_copy_to_user(void __user *to, const void *from, unsigned long n)
+{
+ if (access_ok(VERIFY_WRITE, to, n))
+ n = raw_copy_to_user(to, from, n);
+ return n;
+}
+#else
+extern unsigned long
+_copy_to_user(void __user *, const void *, unsigned long);
+#endif
+
+extern void __compiletime_error("usercopy buffer size is too small")
+__bad_copy_user(void);
+
+static inline void copy_user_overflow(int size, unsigned long count)
+{
+ WARN(1, "Buffer overflow detected (%d < %lu)!\n", size, count);
+}
+
+static __always_inline unsigned long __must_check
+copy_from_user(void *to, const void __user *from, unsigned long n)
+{
+ int sz = __compiletime_object_size(to);
+
+ might_fault();
+ kasan_check_write(to, n);
+
+ if (likely(sz < 0 || sz >= n)) {
+ check_object_size(to, n, false);
+ n = _copy_from_user(to, from, n);
+ } else if (!__builtin_constant_p(n))
+ copy_user_overflow(sz, n);
+ else
+ __bad_copy_user();
+
+ return n;
+}
+
+static __always_inline unsigned long __must_check
+copy_to_user(void __user *to, const void *from, unsigned long n)
+{
+ int sz = __compiletime_object_size(from);
+
+ kasan_check_read(from, n);
+ might_fault();
+
+ if (likely(sz < 0 || sz >= n)) {
+ check_object_size(from, n, true);
+ n = _copy_to_user(to, from, n);
+ } else if (!__builtin_constant_p(n))
+ copy_user_overflow(sz, n);
+ else
+ __bad_copy_user();
+
+ return n;
+}
+#ifdef CONFIG_COMPAT
+static __always_inline unsigned long __must_check
+__copy_in_user(void __user *to, const void *from, unsigned long n)
+{
+ might_fault();
+ return raw_copy_in_user(to, from, n);
+}
+static __always_inline unsigned long __must_check
+copy_in_user(void __user *to, const void *from, unsigned long n)
+{
+ might_fault();
+ if (access_ok(VERIFY_WRITE, to, n) && access_ok(VERIFY_READ, from, n))
+ n = raw_copy_in_user(to, from, n);
+ return n;
+}
+#endif
+
static __always_inline void pagefault_disabled_inc(void)
{
current->pagefault_disabled++;
@@ -67,12 +258,6 @@
return __copy_from_user_inatomic(to, from, n);
}
-static inline unsigned long __copy_from_user_nocache(void *to,
- const void __user *from, unsigned long n)
-{
- return __copy_from_user(to, from, n);
-}
-
#endif /* ARCH_HAS_NOCACHE_UACCESS */
/*
diff --git a/include/linux/udp.h b/include/linux/udp.h
index c0f5308..6cb4061 100644
--- a/include/linux/udp.h
+++ b/include/linux/udp.h
@@ -115,6 +115,6 @@
#define udp_portaddr_for_each_entry_rcu(__sk, list) \
hlist_for_each_entry_rcu(__sk, list, __sk_common.skc_portaddr_node)
-#define IS_UDPLITE(__sk) (udp_sk(__sk)->pcflag)
+#define IS_UDPLITE(__sk) (__sk->sk_protocol == IPPROTO_UDPLITE)
#endif /* _LINUX_UDP_H */
diff --git a/include/linux/usb/composite.h b/include/linux/usb/composite.h
index 4616a49..f665d2c 100644
--- a/include/linux/usb/composite.h
+++ b/include/linux/usb/composite.h
@@ -451,6 +451,7 @@
* sure doing that won't hurt too much.
*
* One notion for how to handle Wireless USB devices involves:
+ *
* (a) a second gadget here, discovery mechanism TBD, but likely
* needing separate "register/unregister WUSB gadget" calls;
* (b) updates to usb_gadget to include flags "is it wireless",
@@ -503,8 +504,9 @@
/* protects deactivations and delayed_status counts*/
spinlock_t lock;
- unsigned setup_pending:1;
- unsigned os_desc_pending:1;
+ /* public: */
+ unsigned int setup_pending:1;
+ unsigned int os_desc_pending:1;
};
extern int usb_string_id(struct usb_composite_dev *c);
diff --git a/include/linux/usb/gadget.h b/include/linux/usb/gadget.h
index e4516e9..fbc22a3 100644
--- a/include/linux/usb/gadget.h
+++ b/include/linux/usb/gadget.h
@@ -188,7 +188,7 @@
* @caps:The structure describing types and directions supported by endoint.
* @maxpacket:The maximum packet size used on this endpoint. The initial
* value can sometimes be reduced (hardware allowing), according to
- * the endpoint descriptor used to configure the endpoint.
+ * the endpoint descriptor used to configure the endpoint.
* @maxpacket_limit:The maximum packet size value which can be handled by this
* endpoint. It's set once by UDC driver when endpoint is initialized, and
* should not be changed. Should not be confused with maxpacket.
diff --git a/include/linux/usb/usbnet.h b/include/linux/usb/usbnet.h
index 6e0ce8c..7dffa56 100644
--- a/include/linux/usb/usbnet.h
+++ b/include/linux/usb/usbnet.h
@@ -64,6 +64,8 @@
struct usb_anchor deferred;
struct tasklet_struct bh;
+ struct pcpu_sw_netstats __percpu *stats64;
+
struct work_struct kevent;
unsigned long flags;
# define EVENT_TX_HALT 0
@@ -261,10 +263,10 @@
extern void usbnet_resume_rx(struct usbnet *);
extern void usbnet_purge_paused_rxq(struct usbnet *);
-extern int usbnet_get_settings(struct net_device *net,
- struct ethtool_cmd *cmd);
-extern int usbnet_set_settings(struct net_device *net,
- struct ethtool_cmd *cmd);
+extern int usbnet_get_link_ksettings(struct net_device *net,
+ struct ethtool_link_ksettings *cmd);
+extern int usbnet_set_link_ksettings(struct net_device *net,
+ const struct ethtool_link_ksettings *cmd);
extern u32 usbnet_get_link(struct net_device *net);
extern u32 usbnet_get_msglevel(struct net_device *);
extern void usbnet_set_msglevel(struct net_device *, u32);
@@ -278,5 +280,7 @@
extern void usbnet_status_stop(struct usbnet *dev);
extern void usbnet_update_max_qlen(struct usbnet *dev);
+extern void usbnet_get_stats64(struct net_device *dev,
+ struct rtnl_link_stats64 *stats);
#endif /* __LINUX_USB_USBNET_H */
diff --git a/include/linux/usb/xhci-dbgp.h b/include/linux/usb/xhci-dbgp.h
new file mode 100644
index 0000000..80c1cca
--- /dev/null
+++ b/include/linux/usb/xhci-dbgp.h
@@ -0,0 +1,29 @@
+/*
+ * Standalone xHCI debug capability driver
+ *
+ * Copyright (C) 2016 Intel Corporation
+ *
+ * Author: Lu Baolu <baolu.lu@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __LINUX_XHCI_DBGP_H
+#define __LINUX_XHCI_DBGP_H
+
+#ifdef CONFIG_EARLY_PRINTK_USB_XDBC
+int __init early_xdbc_parse_parameter(char *s);
+int __init early_xdbc_setup_hardware(void);
+void __init early_xdbc_register_console(void);
+#else
+static inline int __init early_xdbc_setup_hardware(void)
+{
+ return -ENODEV;
+}
+static inline void __init early_xdbc_register_console(void)
+{
+}
+#endif /* CONFIG_EARLY_PRINTK_USB_XDBC */
+#endif /* __LINUX_XHCI_DBGP_H */
diff --git a/include/linux/virtio_vsock.h b/include/linux/virtio_vsock.h
index 584f9a6..ab13f07 100644
--- a/include/linux/virtio_vsock.h
+++ b/include/linux/virtio_vsock.h
@@ -153,5 +153,6 @@
void virtio_transport_inc_tx_pkt(struct virtio_vsock_sock *vvs, struct virtio_vsock_pkt *pkt);
u32 virtio_transport_get_credit(struct virtio_vsock_sock *vvs, u32 wanted);
void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit);
+void virtio_transport_deliver_tap_pkt(struct virtio_vsock_pkt *pkt);
#endif /* _LINUX_VIRTIO_VSOCK_H */
diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h
index bde063c..c102ef6 100644
--- a/include/linux/workqueue.h
+++ b/include/linux/workqueue.h
@@ -608,8 +608,13 @@
{
return fn(arg);
}
+static inline long work_on_cpu_safe(int cpu, long (*fn)(void *), void *arg)
+{
+ return fn(arg);
+}
#else
long work_on_cpu(int cpu, long (*fn)(void *), void *arg);
+long work_on_cpu_safe(int cpu, long (*fn)(void *), void *arg);
#endif /* CONFIG_SMP */
#ifdef CONFIG_FREEZER
diff --git a/include/linux/writeback.h b/include/linux/writeback.h
index a3c0cbd..d581579 100644
--- a/include/linux/writeback.h
+++ b/include/linux/writeback.h
@@ -237,6 +237,7 @@
static inline void inode_detach_wb(struct inode *inode)
{
if (inode->i_wb) {
+ WARN_ON_ONCE(!(inode->i_state & I_CLEAR));
wb_put(inode->i_wb);
inode->i_wb = NULL;
}
diff --git a/include/net/6lowpan.h b/include/net/6lowpan.h
index 5ab4c99..a713780 100644
--- a/include/net/6lowpan.h
+++ b/include/net/6lowpan.h
@@ -198,6 +198,21 @@
ipaddr->s6_addr[8] ^= 0x02;
}
+static inline void lowpan_iphc_uncompress_eui48_lladdr(struct in6_addr *ipaddr,
+ const void *lladdr)
+{
+ /* fe:80::XXXX:XXff:feXX:XXXX
+ * \_________________/
+ * hwaddr
+ */
+ ipaddr->s6_addr[0] = 0xFE;
+ ipaddr->s6_addr[1] = 0x80;
+ memcpy(&ipaddr->s6_addr[8], lladdr, 3);
+ ipaddr->s6_addr[11] = 0xFF;
+ ipaddr->s6_addr[12] = 0xFE;
+ memcpy(&ipaddr->s6_addr[13], lladdr + 3, 3);
+}
+
#ifdef DEBUG
/* print data in line */
static inline void raw_dump_inline(const char *caller, char *msg,
diff --git a/include/net/addrconf.h b/include/net/addrconf.h
index 17c6fd8..1aeb25d 100644
--- a/include/net/addrconf.h
+++ b/include/net/addrconf.h
@@ -262,8 +262,8 @@
int unregister_inet6addr_notifier(struct notifier_block *nb);
int inet6addr_notifier_call_chain(unsigned long val, void *v);
-void inet6_netconf_notify_devconf(struct net *net, int type, int ifindex,
- struct ipv6_devconf *devconf);
+void inet6_netconf_notify_devconf(struct net *net, int event, int type,
+ int ifindex, struct ipv6_devconf *devconf);
/**
* __in6_dev_get - get inet6_dev pointer from netdevice
diff --git a/include/net/af_rxrpc.h b/include/net/af_rxrpc.h
index 1061a47..b5f5187 100644
--- a/include/net/af_rxrpc.h
+++ b/include/net/af_rxrpc.h
@@ -39,7 +39,7 @@
struct msghdr *, size_t);
int rxrpc_kernel_recv_data(struct socket *, struct rxrpc_call *,
void *, size_t, size_t *, bool, u32 *);
-void rxrpc_kernel_abort_call(struct socket *, struct rxrpc_call *,
+bool rxrpc_kernel_abort_call(struct socket *, struct rxrpc_call *,
u32, int, const char *);
void rxrpc_kernel_end_call(struct socket *, struct rxrpc_call *);
void rxrpc_kernel_get_peer(struct socket *, struct rxrpc_call *,
diff --git a/include/net/af_vsock.h b/include/net/af_vsock.h
index f32ed9a..f9fb566 100644
--- a/include/net/af_vsock.h
+++ b/include/net/af_vsock.h
@@ -188,4 +188,17 @@
void vsock_remove_sock(struct vsock_sock *vsk);
void vsock_for_each_connected_socket(void (*fn)(struct sock *sk));
+/**** TAP ****/
+
+struct vsock_tap {
+ struct net_device *dev;
+ struct module *module;
+ struct list_head list;
+};
+
+int vsock_init_tap(void);
+int vsock_add_tap(struct vsock_tap *vt);
+int vsock_remove_tap(struct vsock_tap *vt);
+void vsock_deliver_tap(struct sk_buff *build_skb(void *opaque), void *opaque);
+
#endif /* __AF_VSOCK_H__ */
diff --git a/include/net/bluetooth/l2cap.h b/include/net/bluetooth/l2cap.h
index 5ee3c68..0697fd4 100644
--- a/include/net/bluetooth/l2cap.h
+++ b/include/net/bluetooth/l2cap.h
@@ -282,7 +282,7 @@
#define L2CAP_CR_BAD_KEY_SIZE 0x0007
#define L2CAP_CR_ENCRYPTION 0x0008
#define L2CAP_CR_INVALID_SCID 0x0009
-#define L2CAP_CR_SCID_IN_USE 0x0010
+#define L2CAP_CR_SCID_IN_USE 0x000A
/* connect/create channel status */
#define L2CAP_CS_NO_INFO 0x0000
diff --git a/include/net/bluetooth/rfcomm.h b/include/net/bluetooth/rfcomm.h
index 4190af5..da4acef 100644
--- a/include/net/bluetooth/rfcomm.h
+++ b/include/net/bluetooth/rfcomm.h
@@ -21,6 +21,8 @@
SOFTWARE IS DISCLAIMED.
*/
+#include <linux/refcount.h>
+
#ifndef __RFCOMM_H
#define __RFCOMM_H
@@ -174,7 +176,7 @@
struct mutex lock;
unsigned long state;
unsigned long flags;
- atomic_t refcnt;
+ refcount_t refcnt;
u8 dlci;
u8 addr;
u8 priority;
@@ -247,12 +249,12 @@
static inline void rfcomm_dlc_hold(struct rfcomm_dlc *d)
{
- atomic_inc(&d->refcnt);
+ refcount_inc(&d->refcnt);
}
static inline void rfcomm_dlc_put(struct rfcomm_dlc *d)
{
- if (atomic_dec_and_test(&d->refcnt))
+ if (refcount_dec_and_test(&d->refcnt))
rfcomm_dlc_free(d);
}
diff --git a/include/net/bonding.h b/include/net/bonding.h
index 3c85777..b00508d 100644
--- a/include/net/bonding.h
+++ b/include/net/bonding.h
@@ -153,7 +153,8 @@
unsigned long last_link_up;
unsigned long last_rx;
unsigned long target_last_arp_rx[BOND_MAX_ARP_TARGETS];
- s8 link; /* one of BOND_LINK_XXXX */
+ s8 link; /* one of BOND_LINK_XXXX */
+ s8 link_new_state; /* one of BOND_LINK_XXXX */
s8 new_link;
u8 backup:1, /* indicates backup slave. Value corresponds with
BOND_STATE_ACTIVE and BOND_STATE_BACKUP */
@@ -165,7 +166,7 @@
u32 link_failure_count;
u32 speed;
u16 queue_id;
- u8 perm_hwaddr[ETH_ALEN];
+ u8 perm_hwaddr[MAX_ADDR_LEN];
struct ad_slave_info *ad_info;
struct tlb_slave_info tlb_info;
#ifdef CONFIG_NET_POLL_CONTROLLER
@@ -401,6 +402,16 @@
bond_is_active_slave(slave);
}
+static inline void bond_hw_addr_copy(u8 *dst, const u8 *src, unsigned int len)
+{
+ if (len == ETH_ALEN) {
+ ether_addr_copy(dst, src);
+ return;
+ }
+
+ memcpy(dst, src, len);
+}
+
#define BOND_PRI_RESELECT_ALWAYS 0
#define BOND_PRI_RESELECT_BETTER 1
#define BOND_PRI_RESELECT_FAILURE 2
@@ -504,13 +515,17 @@
return slave->inactive;
}
-static inline void bond_set_slave_link_state(struct slave *slave, int state,
- bool notify)
+static inline void bond_propose_link_state(struct slave *slave, int state)
{
- if (slave->link == state)
+ slave->link_new_state = state;
+}
+
+static inline void bond_commit_link_state(struct slave *slave, bool notify)
+{
+ if (slave->link == slave->link_new_state)
return;
- slave->link = state;
+ slave->link = slave->link_new_state;
if (notify) {
bond_queue_slave_event(slave);
bond_lower_state_changed(slave);
@@ -523,6 +538,13 @@
}
}
+static inline void bond_set_slave_link_state(struct slave *slave, int state,
+ bool notify)
+{
+ bond_propose_link_state(slave, state);
+ bond_commit_link_state(slave, notify);
+}
+
static inline void bond_slave_link_notify(struct bonding *bond)
{
struct list_head *iter;
@@ -592,6 +614,7 @@
int level);
int bond_update_slave_arr(struct bonding *bond, struct slave *skipslave);
void bond_slave_arr_work_rearm(struct bonding *bond, unsigned long delay);
+void bond_work_init_all(struct bonding *bond);
#ifdef CONFIG_PROC_FS
void bond_create_proc_entry(struct bonding *bond);
diff --git a/include/net/busy_poll.h b/include/net/busy_poll.h
index c0452de..8ffd434 100644
--- a/include/net/busy_poll.h
+++ b/include/net/busy_poll.h
@@ -35,49 +35,27 @@
extern unsigned int sysctl_net_busy_read __read_mostly;
extern unsigned int sysctl_net_busy_poll __read_mostly;
+/* 0 - Reserved to indicate value not set
+ * 1..NR_CPUS - Reserved for sender_cpu
+ * NR_CPUS+1..~0 - Region available for NAPI IDs
+ */
+#define MIN_NAPI_ID ((unsigned int)(NR_CPUS + 1))
+
static inline bool net_busy_loop_on(void)
{
return sysctl_net_busy_poll;
}
-static inline u64 busy_loop_us_clock(void)
-{
- return local_clock() >> 10;
-}
-
-static inline unsigned long sk_busy_loop_end_time(struct sock *sk)
-{
- return busy_loop_us_clock() + ACCESS_ONCE(sk->sk_ll_usec);
-}
-
-/* in poll/select we use the global sysctl_net_ll_poll value */
-static inline unsigned long busy_loop_end_time(void)
-{
- return busy_loop_us_clock() + ACCESS_ONCE(sysctl_net_busy_poll);
-}
-
static inline bool sk_can_busy_loop(const struct sock *sk)
{
- return sk->sk_ll_usec && sk->sk_napi_id && !signal_pending(current);
+ return sk->sk_ll_usec && !signal_pending(current);
}
+bool sk_busy_loop_end(void *p, unsigned long start_time);
-static inline bool busy_loop_timeout(unsigned long end_time)
-{
- unsigned long now = busy_loop_us_clock();
-
- return time_after(now, end_time);
-}
-
-bool sk_busy_loop(struct sock *sk, int nonblock);
-
-/* used in the NIC receive handler to mark the skb */
-static inline void skb_mark_napi_id(struct sk_buff *skb,
- struct napi_struct *napi)
-{
- skb->napi_id = napi->napi_id;
-}
-
+void napi_busy_loop(unsigned int napi_id,
+ bool (*loop_end)(void *, unsigned long),
+ void *loop_end_arg);
#else /* CONFIG_NET_RX_BUSY_POLL */
static inline unsigned long net_busy_loop_on(void)
@@ -85,32 +63,72 @@
return 0;
}
-static inline unsigned long busy_loop_end_time(void)
-{
- return 0;
-}
-
static inline bool sk_can_busy_loop(struct sock *sk)
{
return false;
}
-static inline void skb_mark_napi_id(struct sk_buff *skb,
- struct napi_struct *napi)
+#endif /* CONFIG_NET_RX_BUSY_POLL */
+
+static inline unsigned long busy_loop_current_time(void)
{
+#ifdef CONFIG_NET_RX_BUSY_POLL
+ return (unsigned long)(local_clock() >> 10);
+#else
+ return 0;
+#endif
}
-static inline bool busy_loop_timeout(unsigned long end_time)
+/* in poll/select we use the global sysctl_net_ll_poll value */
+static inline bool busy_loop_timeout(unsigned long start_time)
{
+#ifdef CONFIG_NET_RX_BUSY_POLL
+ unsigned long bp_usec = READ_ONCE(sysctl_net_busy_poll);
+
+ if (bp_usec) {
+ unsigned long end_time = start_time + bp_usec;
+ unsigned long now = busy_loop_current_time();
+
+ return time_after(now, end_time);
+ }
+#endif
return true;
}
-static inline bool sk_busy_loop(struct sock *sk, int nonblock)
+static inline bool sk_busy_loop_timeout(struct sock *sk,
+ unsigned long start_time)
{
- return false;
+#ifdef CONFIG_NET_RX_BUSY_POLL
+ unsigned long bp_usec = READ_ONCE(sk->sk_ll_usec);
+
+ if (bp_usec) {
+ unsigned long end_time = start_time + bp_usec;
+ unsigned long now = busy_loop_current_time();
+
+ return time_after(now, end_time);
+ }
+#endif
+ return true;
}
-#endif /* CONFIG_NET_RX_BUSY_POLL */
+static inline void sk_busy_loop(struct sock *sk, int nonblock)
+{
+#ifdef CONFIG_NET_RX_BUSY_POLL
+ unsigned int napi_id = READ_ONCE(sk->sk_napi_id);
+
+ if (napi_id >= MIN_NAPI_ID)
+ napi_busy_loop(napi_id, nonblock ? NULL : sk_busy_loop_end, sk);
+#endif
+}
+
+/* used in the NIC receive handler to mark the skb */
+static inline void skb_mark_napi_id(struct sk_buff *skb,
+ struct napi_struct *napi)
+{
+#ifdef CONFIG_NET_RX_BUSY_POLL
+ skb->napi_id = napi->napi_id;
+#endif
+}
/* used in the protocol hanlder to propagate the napi_id to the socket */
static inline void sk_mark_napi_id(struct sock *sk, const struct sk_buff *skb)
diff --git a/include/net/cfg80211.h b/include/net/cfg80211.h
index ead1aa6..6e90f1a 100644
--- a/include/net/cfg80211.h
+++ b/include/net/cfg80211.h
@@ -363,6 +363,8 @@
/**
* struct vif_params - describes virtual interface parameters
+ * @flags: monitor interface flags, unchanged if 0, otherwise
+ * %MONITOR_FLAG_CHANGED will be set
* @use_4addr: use 4-address frames
* @macaddr: address to use for this virtual interface.
* If this parameter is set to zero address the driver may
@@ -370,13 +372,17 @@
* This feature is only fully supported by drivers that enable the
* %NL80211_FEATURE_MAC_ON_CREATE flag. Others may support creating
** only p2p devices with specified MAC.
- * @vht_mumimo_groups: MU-MIMO groupID. used for monitoring only
- * packets belonging to that MU-MIMO groupID.
+ * @vht_mumimo_groups: MU-MIMO groupID, used for monitoring MU-MIMO packets
+ * belonging to that MU-MIMO groupID; %NULL if not changed
+ * @vht_mumimo_follow_addr: MU-MIMO follow address, used for monitoring
+ * MU-MIMO packets going to the specified station; %NULL if not changed
*/
struct vif_params {
+ u32 flags;
int use_4addr;
u8 macaddr[ETH_ALEN];
- u8 vht_mumimo_groups[VHT_MUMIMO_GROUPS_DATA_LEN];
+ const u8 *vht_mumimo_groups;
+ const u8 *vht_mumimo_follow_addr;
};
/**
@@ -1211,6 +1217,7 @@
* Monitor interface configuration flags. Note that these must be the bits
* according to the nl80211 flags.
*
+ * @MONITOR_FLAG_CHANGED: set if the flags were changed
* @MONITOR_FLAG_FCSFAIL: pass frames with bad FCS
* @MONITOR_FLAG_PLCPFAIL: pass frames with bad PLCP
* @MONITOR_FLAG_CONTROL: pass control frames
@@ -1219,6 +1226,7 @@
* @MONITOR_FLAG_ACTIVE: active monitor, ACKs frames on its MAC address
*/
enum monitor_flags {
+ MONITOR_FLAG_CHANGED = 1<<__NL80211_MNTR_FLAG_INVALID,
MONITOR_FLAG_FCSFAIL = 1<<NL80211_MNTR_FLAG_FCSFAIL,
MONITOR_FLAG_PLCPFAIL = 1<<NL80211_MNTR_FLAG_PLCPFAIL,
MONITOR_FLAG_CONTROL = 1<<NL80211_MNTR_FLAG_CONTROL,
@@ -1605,11 +1613,15 @@
/**
* struct cfg80211_match_set - sets of attributes to match
*
- * @ssid: SSID to be matched; may be zero-length for no match (RSSI only)
+ * @ssid: SSID to be matched; may be zero-length in case of BSSID match
+ * or no match (RSSI only)
+ * @bssid: BSSID to be matched; may be all-zero BSSID in case of SSID match
+ * or no match (RSSI only)
* @rssi_thold: don't report scan results below this threshold (in s32 dBm)
*/
struct cfg80211_match_set {
struct cfg80211_ssid ssid;
+ u8 bssid[ETH_ALEN];
s32 rssi_thold;
};
@@ -1641,6 +1653,7 @@
/**
* struct cfg80211_sched_scan_request - scheduled scan request description
*
+ * @reqid: identifies this request.
* @ssids: SSIDs to scan for (passed in the probe_reqs in active scans)
* @n_ssids: number of SSIDs
* @n_channels: total number of channels to scan
@@ -1653,6 +1666,7 @@
* (others are filtered out).
* If ommited, all results are passed.
* @n_match_sets: number of match sets
+ * @results_wk: worker for processing results notification.
* @wiphy: the wiphy this was for
* @dev: the interface
* @scan_start: start time of the scheduled scan
@@ -1669,6 +1683,8 @@
* @rcu_head: RCU callback used to free the struct
* @owner_nlportid: netlink portid of owner (if this should is a request
* owned by a particular socket)
+ * @nl_owner_dead: netlink owner socket was closed - this request be freed
+ * @list: for keeping list of requests.
* @delay: delay in seconds to use before starting the first scan
* cycle. The driver may ignore this parameter and start
* immediately (or at any other time), if this feature is not
@@ -1685,6 +1701,7 @@
* comparisions.
*/
struct cfg80211_sched_scan_request {
+ u64 reqid;
struct cfg80211_ssid *ssids;
int n_ssids;
u32 n_channels;
@@ -1710,8 +1727,11 @@
struct wiphy *wiphy;
struct net_device *dev;
unsigned long scan_start;
+ bool report_results;
struct rcu_head rcu_head;
u32 owner_nlportid;
+ bool nl_owner_dead;
+ struct list_head list;
/* keep last */
struct ieee80211_channel *channels[0];
@@ -2073,6 +2093,19 @@
* the BSSID of the current association, i.e., to the value that is
* included in the Current AP address field of the Reassociation Request
* frame.
+ * @fils_erp_username: EAP re-authentication protocol (ERP) username part of the
+ * NAI or %NULL if not specified. This is used to construct FILS wrapped
+ * data IE.
+ * @fils_erp_username_len: Length of @fils_erp_username in octets.
+ * @fils_erp_realm: EAP re-authentication protocol (ERP) realm part of NAI or
+ * %NULL if not specified. This specifies the domain name of ER server and
+ * is used to construct FILS wrapped data IE.
+ * @fils_erp_realm_len: Length of @fils_erp_realm in octets.
+ * @fils_erp_next_seq_num: The next sequence number to use in the FILS ERP
+ * messages. This is also used to construct FILS wrapped data IE.
+ * @fils_erp_rrk: ERP re-authentication Root Key (rRK) used to derive additional
+ * keys in FILS or %NULL if not specified.
+ * @fils_erp_rrk_len: Length of @fils_erp_rrk in octets.
*/
struct cfg80211_connect_params {
struct ieee80211_channel *channel;
@@ -2098,6 +2131,13 @@
bool pbss;
struct cfg80211_bss_selection bss_select;
const u8 *prev_bssid;
+ const u8 *fils_erp_username;
+ size_t fils_erp_username_len;
+ const u8 *fils_erp_realm;
+ size_t fils_erp_realm_len;
+ u16 fils_erp_next_seq_num;
+ const u8 *fils_erp_rrk;
+ size_t fils_erp_rrk_len;
};
/**
@@ -2136,12 +2176,27 @@
* This structure is passed to the set/del_pmksa() method for PMKSA
* caching.
*
- * @bssid: The AP's BSSID.
- * @pmkid: The PMK material itself.
+ * @bssid: The AP's BSSID (may be %NULL).
+ * @pmkid: The identifier to refer a PMKSA.
+ * @pmk: The PMK for the PMKSA identified by @pmkid. This is used for key
+ * derivation by a FILS STA. Otherwise, %NULL.
+ * @pmk_len: Length of the @pmk. The length of @pmk can differ depending on
+ * the hash algorithm used to generate this.
+ * @ssid: SSID to specify the ESS within which a PMKSA is valid when using FILS
+ * cache identifier (may be %NULL).
+ * @ssid_len: Length of the @ssid in octets.
+ * @cache_id: 2-octet cache identifier advertized by a FILS AP identifying the
+ * scope of PMKSA. This is valid only if @ssid_len is non-zero (may be
+ * %NULL).
*/
struct cfg80211_pmksa {
const u8 *bssid;
const u8 *pmkid;
+ const u8 *pmk;
+ size_t pmk_len;
+ const u8 *ssid;
+ size_t ssid_len;
+ const u8 *cache_id;
};
/**
@@ -2633,8 +2688,7 @@
* indication of requesting reassociation.
* In both the driver-initiated and new connect() call initiated roaming
* cases, the result of roaming is indicated with a call to
- * cfg80211_roamed() or cfg80211_roamed_bss().
- * (invoked with the wireless_dev mutex held)
+ * cfg80211_roamed(). (invoked with the wireless_dev mutex held)
* @update_connect_params: Update the connect parameters while connected to a
* BSS. The updated parameters can be used by driver/firmware for
* subsequent BSS selection (roaming) decisions and to form the
@@ -2712,15 +2766,20 @@
* the current level is above/below the configured threshold; this may
* need some care when the configuration is changed (without first being
* disabled.)
+ * @set_cqm_rssi_range_config: Configure two RSSI thresholds in the
+ * connection quality monitor. An event is to be sent only when the
+ * signal level is found to be outside the two values. The driver should
+ * set %NL80211_EXT_FEATURE_CQM_RSSI_LIST if this method is implemented.
+ * If it is provided then there's no point providing @set_cqm_rssi_config.
* @set_cqm_txe_config: Configure connection quality monitor TX error
* thresholds.
* @sched_scan_start: Tell the driver to start a scheduled scan.
- * @sched_scan_stop: Tell the driver to stop an ongoing scheduled scan. This
- * call must stop the scheduled scan and be ready for starting a new one
- * before it returns, i.e. @sched_scan_start may be called immediately
- * after that again and should not fail in that case. The driver should
- * not call cfg80211_sched_scan_stopped() for a requested stop (when this
- * method returns 0.)
+ * @sched_scan_stop: Tell the driver to stop an ongoing scheduled scan with
+ * given request id. This call must stop the scheduled scan and be ready
+ * for starting a new one before it returns, i.e. @sched_scan_start may be
+ * called immediately after that again and should not fail in that case.
+ * The driver should not call cfg80211_sched_scan_stopped() for a requested
+ * stop (when this method returns 0).
*
* @mgmt_frame_register: Notify driver that a management frame type was
* registered. The callback is allowed to sleep.
@@ -2826,13 +2885,12 @@
const char *name,
unsigned char name_assign_type,
enum nl80211_iftype type,
- u32 *flags,
struct vif_params *params);
int (*del_virtual_intf)(struct wiphy *wiphy,
struct wireless_dev *wdev);
int (*change_virtual_intf)(struct wiphy *wiphy,
struct net_device *dev,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params);
int (*add_key)(struct wiphy *wiphy, struct net_device *netdev,
@@ -3001,6 +3059,10 @@
struct net_device *dev,
s32 rssi_thold, u32 rssi_hyst);
+ int (*set_cqm_rssi_range_config)(struct wiphy *wiphy,
+ struct net_device *dev,
+ s32 rssi_low, s32 rssi_high);
+
int (*set_cqm_txe_config)(struct wiphy *wiphy,
struct net_device *dev,
u32 rate, u32 pkts, u32 intvl);
@@ -3015,7 +3077,8 @@
int (*sched_scan_start)(struct wiphy *wiphy,
struct net_device *dev,
struct cfg80211_sched_scan_request *request);
- int (*sched_scan_stop)(struct wiphy *wiphy, struct net_device *dev);
+ int (*sched_scan_stop)(struct wiphy *wiphy, struct net_device *dev,
+ u64 reqid);
int (*set_rekey_data)(struct wiphy *wiphy, struct net_device *dev,
struct cfg80211_gtk_rekey_data *data);
@@ -3160,7 +3223,7 @@
WIPHY_FLAG_CONTROL_PORT_PROTOCOL = BIT(7),
WIPHY_FLAG_IBSS_RSN = BIT(8),
WIPHY_FLAG_MESH_AUTH = BIT(10),
- WIPHY_FLAG_SUPPORTS_SCHED_SCAN = BIT(11),
+ /* use hole at 11 */
/* use hole at 12 */
WIPHY_FLAG_SUPPORTS_FW_ROAM = BIT(13),
WIPHY_FLAG_AP_UAPSD = BIT(14),
@@ -3498,6 +3561,8 @@
* this variable determines its size
* @max_scan_ssids: maximum number of SSIDs the device can scan for in
* any given scan
+ * @max_sched_scan_reqs: maximum number of scheduled scan requests that
+ * the device can run concurrently.
* @max_sched_scan_ssids: maximum number of SSIDs the device can scan
* for in any given scheduled scan
* @max_match_sets: maximum number of match sets the device can handle
@@ -3634,6 +3699,7 @@
int bss_priv_size;
u8 max_scan_ssids;
+ u8 max_sched_scan_reqs;
u8 max_sched_scan_ssids;
u8 max_match_sets;
u16 max_scan_ie_len;
@@ -3871,6 +3937,7 @@
struct cfg80211_conn;
struct cfg80211_internal_bss;
struct cfg80211_cached_keys;
+struct cfg80211_cqm_config;
/**
* struct wireless_dev - wireless device state
@@ -3934,6 +4001,8 @@
* @event_list: (private) list for internal event processing
* @event_lock: (private) lock for event list
* @owner_nlportid: (private) owner socket port ID
+ * @nl_owner_dead: (private) owner socket went away
+ * @cqm_config: (private) nl80211 RSSI monitor state
*/
struct wireless_dev {
struct wiphy *wiphy;
@@ -3982,12 +4051,13 @@
u32 ap_unexpected_nlportid;
+ u32 owner_nlportid;
+ bool nl_owner_dead;
+
bool cac_started;
unsigned long cac_start_time;
unsigned int cac_time_ms;
- u32 owner_nlportid;
-
#ifdef CONFIG_CFG80211_WEXT
/* wext data */
struct {
@@ -4002,6 +4072,8 @@
bool prev_bssid_valid;
} wext;
#endif
+
+ struct cfg80211_cqm_config *cqm_config;
};
static inline u8 *wdev_address(struct wireless_dev *wdev)
@@ -4494,31 +4566,34 @@
* cfg80211_sched_scan_results - notify that new scan results are available
*
* @wiphy: the wiphy which got scheduled scan results
+ * @reqid: identifier for the related scheduled scan request
*/
-void cfg80211_sched_scan_results(struct wiphy *wiphy);
+void cfg80211_sched_scan_results(struct wiphy *wiphy, u64 reqid);
/**
* cfg80211_sched_scan_stopped - notify that the scheduled scan has stopped
*
* @wiphy: the wiphy on which the scheduled scan stopped
+ * @reqid: identifier for the related scheduled scan request
*
* The driver can call this function to inform cfg80211 that the
* scheduled scan had to be stopped, for whatever reason. The driver
* is then called back via the sched_scan_stop operation when done.
*/
-void cfg80211_sched_scan_stopped(struct wiphy *wiphy);
+void cfg80211_sched_scan_stopped(struct wiphy *wiphy, u64 reqid);
/**
* cfg80211_sched_scan_stopped_rtnl - notify that the scheduled scan has stopped
*
* @wiphy: the wiphy on which the scheduled scan stopped
+ * @reqid: identifier for the related scheduled scan request
*
* The driver can call this function to inform cfg80211 that the
* scheduled scan had to be stopped, for whatever reason. The driver
* is then called back via the sched_scan_stop operation when done.
* This function should be called with rtnl locked.
*/
-void cfg80211_sched_scan_stopped_rtnl(struct wiphy *wiphy);
+void cfg80211_sched_scan_stopped_rtnl(struct wiphy *wiphy, u64 reqid);
/**
* cfg80211_inform_bss_frame_data - inform cfg80211 of a received BSS frame
@@ -4651,12 +4726,22 @@
gfp);
}
+/**
+ * cfg80211_get_bss - get a BSS reference
+ * @wiphy: the wiphy this BSS struct belongs to
+ * @channel: the channel to search on (or %NULL)
+ * @bssid: the desired BSSID (or %NULL)
+ * @ssid: the desired SSID (or %NULL)
+ * @ssid_len: length of the SSID (or 0)
+ * @bss_type: type of BSS, see &enum ieee80211_bss_type
+ * @privacy: privacy filter, see &enum ieee80211_privacy
+ */
struct cfg80211_bss *cfg80211_get_bss(struct wiphy *wiphy,
struct ieee80211_channel *channel,
const u8 *bssid,
const u8 *ssid, size_t ssid_len,
enum ieee80211_bss_type bss_type,
- enum ieee80211_privacy);
+ enum ieee80211_privacy privacy);
static inline struct cfg80211_bss *
cfg80211_get_ibss(struct wiphy *wiphy,
struct ieee80211_channel *channel,
@@ -5123,6 +5208,78 @@
#endif
/**
+ * struct cfg80211_connect_resp_params - Connection response params
+ * @status: Status code, %WLAN_STATUS_SUCCESS for successful connection, use
+ * %WLAN_STATUS_UNSPECIFIED_FAILURE if your device cannot give you
+ * the real status code for failures. If this call is used to report a
+ * failure due to a timeout (e.g., not receiving an Authentication frame
+ * from the AP) instead of an explicit rejection by the AP, -1 is used to
+ * indicate that this is a failure, but without a status code.
+ * @timeout_reason is used to report the reason for the timeout in that
+ * case.
+ * @bssid: The BSSID of the AP (may be %NULL)
+ * @bss: Entry of bss to which STA got connected to, can be obtained through
+ * cfg80211_get_bss() (may be %NULL). Only one parameter among @bssid and
+ * @bss needs to be specified.
+ * @req_ie: Association request IEs (may be %NULL)
+ * @req_ie_len: Association request IEs length
+ * @resp_ie: Association response IEs (may be %NULL)
+ * @resp_ie_len: Association response IEs length
+ * @fils_kek: KEK derived from a successful FILS connection (may be %NULL)
+ * @fils_kek_len: Length of @fils_kek in octets
+ * @update_erp_next_seq_num: Boolean value to specify whether the value in
+ * @fils_erp_next_seq_num is valid.
+ * @fils_erp_next_seq_num: The next sequence number to use in ERP message in
+ * FILS Authentication. This value should be specified irrespective of the
+ * status for a FILS connection.
+ * @pmk: A new PMK if derived from a successful FILS connection (may be %NULL).
+ * @pmk_len: Length of @pmk in octets
+ * @pmkid: A new PMKID if derived from a successful FILS connection or the PMKID
+ * used for this FILS connection (may be %NULL).
+ * @timeout_reason: Reason for connection timeout. This is used when the
+ * connection fails due to a timeout instead of an explicit rejection from
+ * the AP. %NL80211_TIMEOUT_UNSPECIFIED is used when the timeout reason is
+ * not known. This value is used only if @status < 0 to indicate that the
+ * failure is due to a timeout and not due to explicit rejection by the AP.
+ * This value is ignored in other cases (@status >= 0).
+ */
+struct cfg80211_connect_resp_params {
+ int status;
+ const u8 *bssid;
+ struct cfg80211_bss *bss;
+ const u8 *req_ie;
+ size_t req_ie_len;
+ const u8 *resp_ie;
+ size_t resp_ie_len;
+ const u8 *fils_kek;
+ size_t fils_kek_len;
+ bool update_erp_next_seq_num;
+ u16 fils_erp_next_seq_num;
+ const u8 *pmk;
+ size_t pmk_len;
+ const u8 *pmkid;
+ enum nl80211_timeout_reason timeout_reason;
+};
+
+/**
+ * cfg80211_connect_done - notify cfg80211 of connection result
+ *
+ * @dev: network device
+ * @params: connection response parameters
+ * @gfp: allocation flags
+ *
+ * It should be called by the underlying driver once execution of the connection
+ * request from connect() has been completed. This is similar to
+ * cfg80211_connect_bss(), but takes a structure pointer for connection response
+ * parameters. Only one of the functions among cfg80211_connect_bss(),
+ * cfg80211_connect_result(), cfg80211_connect_timeout(),
+ * and cfg80211_connect_done() should be called.
+ */
+void cfg80211_connect_done(struct net_device *dev,
+ struct cfg80211_connect_resp_params *params,
+ gfp_t gfp);
+
+/**
* cfg80211_connect_bss - notify cfg80211 of connection result
*
* @dev: network device
@@ -5152,13 +5309,31 @@
* It should be called by the underlying driver once execution of the connection
* request from connect() has been completed. This is similar to
* cfg80211_connect_result(), but with the option of identifying the exact bss
- * entry for the connection. Only one of these functions should be called.
+ * entry for the connection. Only one of the functions among
+ * cfg80211_connect_bss(), cfg80211_connect_result(),
+ * cfg80211_connect_timeout(), and cfg80211_connect_done() should be called.
*/
-void cfg80211_connect_bss(struct net_device *dev, const u8 *bssid,
- struct cfg80211_bss *bss, const u8 *req_ie,
- size_t req_ie_len, const u8 *resp_ie,
- size_t resp_ie_len, int status, gfp_t gfp,
- enum nl80211_timeout_reason timeout_reason);
+static inline void
+cfg80211_connect_bss(struct net_device *dev, const u8 *bssid,
+ struct cfg80211_bss *bss, const u8 *req_ie,
+ size_t req_ie_len, const u8 *resp_ie,
+ size_t resp_ie_len, int status, gfp_t gfp,
+ enum nl80211_timeout_reason timeout_reason)
+{
+ struct cfg80211_connect_resp_params params;
+
+ memset(¶ms, 0, sizeof(params));
+ params.status = status;
+ params.bssid = bssid;
+ params.bss = bss;
+ params.req_ie = req_ie;
+ params.req_ie_len = req_ie_len;
+ params.resp_ie = resp_ie;
+ params.resp_ie_len = resp_ie_len;
+ params.timeout_reason = timeout_reason;
+
+ cfg80211_connect_done(dev, ¶ms, gfp);
+}
/**
* cfg80211_connect_result - notify cfg80211 of connection result
@@ -5177,7 +5352,8 @@
* It should be called by the underlying driver once execution of the connection
* request from connect() has been completed. This is similar to
* cfg80211_connect_bss() which allows the exact bss entry to be specified. Only
- * one of these functions should be called.
+ * one of the functions among cfg80211_connect_bss(), cfg80211_connect_result(),
+ * cfg80211_connect_timeout(), and cfg80211_connect_done() should be called.
*/
static inline void
cfg80211_connect_result(struct net_device *dev, const u8 *bssid,
@@ -5204,7 +5380,9 @@
* in a sequence where no explicit authentication/association rejection was
* received from the AP. This could happen, e.g., due to not being able to send
* out the Authentication or Association Request frame or timing out while
- * waiting for the response.
+ * waiting for the response. Only one of the functions among
+ * cfg80211_connect_bss(), cfg80211_connect_result(),
+ * cfg80211_connect_timeout(), and cfg80211_connect_done() should be called.
*/
static inline void
cfg80211_connect_timeout(struct net_device *dev, const u8 *bssid,
@@ -5216,51 +5394,46 @@
}
/**
+ * struct cfg80211_roam_info - driver initiated roaming information
+ *
+ * @channel: the channel of the new AP
+ * @bss: entry of bss to which STA got roamed (may be %NULL if %bssid is set)
+ * @bssid: the BSSID of the new AP (may be %NULL if %bss is set)
+ * @req_ie: association request IEs (maybe be %NULL)
+ * @req_ie_len: association request IEs length
+ * @resp_ie: association response IEs (may be %NULL)
+ * @resp_ie_len: assoc response IEs length
+ */
+struct cfg80211_roam_info {
+ struct ieee80211_channel *channel;
+ struct cfg80211_bss *bss;
+ const u8 *bssid;
+ const u8 *req_ie;
+ size_t req_ie_len;
+ const u8 *resp_ie;
+ size_t resp_ie_len;
+};
+
+/**
* cfg80211_roamed - notify cfg80211 of roaming
*
* @dev: network device
- * @channel: the channel of the new AP
- * @bssid: the BSSID of the new AP
- * @req_ie: association request IEs (maybe be %NULL)
- * @req_ie_len: association request IEs length
- * @resp_ie: association response IEs (may be %NULL)
- * @resp_ie_len: assoc response IEs length
+ * @info: information about the new BSS. struct &cfg80211_roam_info.
* @gfp: allocation flags
*
- * It should be called by the underlying driver whenever it roamed
- * from one AP to another while connected.
- */
-void cfg80211_roamed(struct net_device *dev,
- struct ieee80211_channel *channel,
- const u8 *bssid,
- const u8 *req_ie, size_t req_ie_len,
- const u8 *resp_ie, size_t resp_ie_len, gfp_t gfp);
-
-/**
- * cfg80211_roamed_bss - notify cfg80211 of roaming
- *
- * @dev: network device
- * @bss: entry of bss to which STA got roamed
- * @req_ie: association request IEs (maybe be %NULL)
- * @req_ie_len: association request IEs length
- * @resp_ie: association response IEs (may be %NULL)
- * @resp_ie_len: assoc response IEs length
- * @gfp: allocation flags
- *
- * This is just a wrapper to notify cfg80211 of roaming event with driver
- * passing bss to avoid a race in timeout of the bss entry. It should be
- * called by the underlying driver whenever it roamed from one AP to another
- * while connected. Drivers which have roaming implemented in firmware
- * may use this function to avoid a race in bss entry timeout where the bss
- * entry of the new AP is seen in the driver, but gets timed out by the time
- * it is accessed in __cfg80211_roamed() due to delay in scheduling
+ * This function may be called with the driver passing either the BSSID of the
+ * new AP or passing the bss entry to avoid a race in timeout of the bss entry.
+ * It should be called by the underlying driver whenever it roamed from one AP
+ * to another while connected. Drivers which have roaming implemented in
+ * firmware should pass the bss entry to avoid a race in bss entry timeout where
+ * the bss entry of the new AP is seen in the driver, but gets timed out by the
+ * time it is accessed in __cfg80211_roamed() due to delay in scheduling
* rdev->event_work. In case of any failures, the reference is released
- * either in cfg80211_roamed_bss() or in __cfg80211_romed(), Otherwise,
- * it will be released while diconneting from the current bss.
+ * either in cfg80211_roamed() or in __cfg80211_romed(), Otherwise, it will be
+ * released while diconneting from the current bss.
*/
-void cfg80211_roamed_bss(struct net_device *dev, struct cfg80211_bss *bss,
- const u8 *req_ie, size_t req_ie_len,
- const u8 *resp_ie, size_t resp_ie_len, gfp_t gfp);
+void cfg80211_roamed(struct net_device *dev, struct cfg80211_roam_info *info,
+ gfp_t gfp);
/**
* cfg80211_disconnected - notify cfg80211 that connection was dropped
diff --git a/include/net/devlink.h b/include/net/devlink.h
index d29e5fc..ed7687b 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -25,6 +25,8 @@
struct list_head list;
struct list_head port_list;
struct list_head sb_list;
+ struct list_head dpipe_table_list;
+ struct devlink_dpipe_headers *dpipe_headers;
const struct devlink_ops *ops;
struct device *dev;
possible_net_t _net;
@@ -49,6 +51,178 @@
enum devlink_sb_threshold_type threshold_type;
};
+/**
+ * struct devlink_dpipe_field - dpipe field object
+ * @name: field name
+ * @id: index inside the headers field array
+ * @bitwidth: bitwidth
+ * @mapping_type: mapping type
+ */
+struct devlink_dpipe_field {
+ const char *name;
+ unsigned int id;
+ unsigned int bitwidth;
+ enum devlink_dpipe_field_mapping_type mapping_type;
+};
+
+/**
+ * struct devlink_dpipe_header - dpipe header object
+ * @name: header name
+ * @id: index, global/local detrmined by global bit
+ * @fields: fields
+ * @fields_count: number of fields
+ * @global: indicates if header is shared like most protocol header
+ * or driver specific
+ */
+struct devlink_dpipe_header {
+ const char *name;
+ unsigned int id;
+ struct devlink_dpipe_field *fields;
+ unsigned int fields_count;
+ bool global;
+};
+
+/**
+ * struct devlink_dpipe_match - represents match operation
+ * @type: type of match
+ * @header_index: header index (packets can have several headers of same
+ * type like in case of tunnels)
+ * @header: header
+ * @fieled_id: field index
+ */
+struct devlink_dpipe_match {
+ enum devlink_dpipe_match_type type;
+ unsigned int header_index;
+ struct devlink_dpipe_header *header;
+ unsigned int field_id;
+};
+
+/**
+ * struct devlink_dpipe_action - represents action operation
+ * @type: type of action
+ * @header_index: header index (packets can have several headers of same
+ * type like in case of tunnels)
+ * @header: header
+ * @fieled_id: field index
+ */
+struct devlink_dpipe_action {
+ enum devlink_dpipe_action_type type;
+ unsigned int header_index;
+ struct devlink_dpipe_header *header;
+ unsigned int field_id;
+};
+
+/**
+ * struct devlink_dpipe_value - represents value of match/action
+ * @action: action
+ * @match: match
+ * @mapping_value: in case the field has some mapping this value
+ * specified the mapping value
+ * @mapping_valid: specify if mapping value is valid
+ * @value_size: value size
+ * @value: value
+ * @mask: bit mask
+ */
+struct devlink_dpipe_value {
+ union {
+ struct devlink_dpipe_action *action;
+ struct devlink_dpipe_match *match;
+ };
+ unsigned int mapping_value;
+ bool mapping_valid;
+ unsigned int value_size;
+ void *value;
+ void *mask;
+};
+
+/**
+ * struct devlink_dpipe_entry - table entry object
+ * @index: index of the entry in the table
+ * @match_values: match values
+ * @matche_values_count: count of matches tuples
+ * @action_values: actions values
+ * @action_values_count: count of actions values
+ * @counter: value of counter
+ * @counter_valid: Specify if value is valid from hardware
+ */
+struct devlink_dpipe_entry {
+ u64 index;
+ struct devlink_dpipe_value *match_values;
+ unsigned int match_values_count;
+ struct devlink_dpipe_value *action_values;
+ unsigned int action_values_count;
+ u64 counter;
+ bool counter_valid;
+};
+
+/**
+ * struct devlink_dpipe_dump_ctx - context provided to driver in order
+ * to dump
+ * @info: info
+ * @cmd: devlink command
+ * @skb: skb
+ * @nest: top attribute
+ * @hdr: hdr
+ */
+struct devlink_dpipe_dump_ctx {
+ struct genl_info *info;
+ enum devlink_command cmd;
+ struct sk_buff *skb;
+ struct nlattr *nest;
+ void *hdr;
+};
+
+struct devlink_dpipe_table_ops;
+
+/**
+ * struct devlink_dpipe_table - table object
+ * @priv: private
+ * @name: table name
+ * @size: maximum number of entries
+ * @counters_enabled: indicates if counters are active
+ * @counter_control_extern: indicates if counter control is in dpipe or
+ * external tool
+ * @table_ops: table operations
+ * @rcu: rcu
+ */
+struct devlink_dpipe_table {
+ void *priv;
+ struct list_head list;
+ const char *name;
+ u64 size;
+ bool counters_enabled;
+ bool counter_control_extern;
+ struct devlink_dpipe_table_ops *table_ops;
+ struct rcu_head rcu;
+};
+
+/**
+ * struct devlink_dpipe_table_ops - dpipe_table ops
+ * @actions_dump - dumps all tables actions
+ * @matches_dump - dumps all tables matches
+ * @entries_dump - dumps all active entries in the table
+ * @counters_set_update - when changing the counter status hardware sync
+ * maybe needed to allocate/free counter related
+ * resources
+ */
+struct devlink_dpipe_table_ops {
+ int (*actions_dump)(void *priv, struct sk_buff *skb);
+ int (*matches_dump)(void *priv, struct sk_buff *skb);
+ int (*entries_dump)(void *priv, bool counters_enabled,
+ struct devlink_dpipe_dump_ctx *dump_ctx);
+ int (*counters_set_update)(void *priv, bool enable);
+};
+
+/**
+ * struct devlink_dpipe_headers - dpipe headers
+ * @headers - header array can be shared (global bit) or driver specific
+ * @headers_count - count of headers
+ */
+struct devlink_dpipe_headers {
+ struct devlink_dpipe_header **headers;
+ unsigned int headers_count;
+};
+
struct devlink_ops {
int (*port_type_set)(struct devlink_port *devlink_port,
enum devlink_port_type port_type);
@@ -94,6 +268,8 @@
int (*eswitch_mode_set)(struct devlink *devlink, u16 mode);
int (*eswitch_inline_mode_get)(struct devlink *devlink, u8 *p_inline_mode);
int (*eswitch_inline_mode_set)(struct devlink *devlink, u8 inline_mode);
+ int (*eswitch_encap_mode_get)(struct devlink *devlink, u8 *p_encap_mode);
+ int (*eswitch_encap_mode_set)(struct devlink *devlink, u8 encap_mode);
};
static inline void *devlink_priv(struct devlink *devlink)
@@ -132,6 +308,26 @@
u16 egress_pools_count, u16 ingress_tc_count,
u16 egress_tc_count);
void devlink_sb_unregister(struct devlink *devlink, unsigned int sb_index);
+int devlink_dpipe_table_register(struct devlink *devlink,
+ const char *table_name,
+ struct devlink_dpipe_table_ops *table_ops,
+ void *priv, u64 size,
+ bool counter_control_extern);
+void devlink_dpipe_table_unregister(struct devlink *devlink,
+ const char *table_name);
+int devlink_dpipe_headers_register(struct devlink *devlink,
+ struct devlink_dpipe_headers *dpipe_headers);
+void devlink_dpipe_headers_unregister(struct devlink *devlink);
+bool devlink_dpipe_table_counter_enabled(struct devlink *devlink,
+ const char *table_name);
+int devlink_dpipe_entry_ctx_prepare(struct devlink_dpipe_dump_ctx *dump_ctx);
+int devlink_dpipe_entry_ctx_append(struct devlink_dpipe_dump_ctx *dump_ctx,
+ struct devlink_dpipe_entry *entry);
+int devlink_dpipe_entry_ctx_close(struct devlink_dpipe_dump_ctx *dump_ctx);
+int devlink_dpipe_action_put(struct sk_buff *skb,
+ struct devlink_dpipe_action *action);
+int devlink_dpipe_match_put(struct sk_buff *skb,
+ struct devlink_dpipe_match *match);
#else
@@ -200,6 +396,71 @@
{
}
+static inline int
+devlink_dpipe_table_register(struct devlink *devlink,
+ const char *table_name,
+ struct devlink_dpipe_table_ops *table_ops,
+ void *priv, u64 size,
+ bool counter_control_extern)
+{
+ return 0;
+}
+
+static inline void devlink_dpipe_table_unregister(struct devlink *devlink,
+ const char *table_name)
+{
+}
+
+static inline int devlink_dpipe_headers_register(struct devlink *devlink,
+ struct devlink_dpipe_headers *
+ dpipe_headers)
+{
+ return 0;
+}
+
+static inline void devlink_dpipe_headers_unregister(struct devlink *devlink)
+{
+}
+
+static inline bool devlink_dpipe_table_counter_enabled(struct devlink *devlink,
+ const char *table_name)
+{
+ return false;
+}
+
+static inline int
+devlink_dpipe_entry_ctx_prepare(struct devlink_dpipe_dump_ctx *dump_ctx)
+{
+ return 0;
+}
+
+static inline int
+devlink_dpipe_entry_ctx_append(struct devlink_dpipe_dump_ctx *dump_ctx,
+ struct devlink_dpipe_entry *entry)
+{
+ return 0;
+}
+
+static inline int
+devlink_dpipe_entry_ctx_close(struct devlink_dpipe_dump_ctx *dump_ctx)
+{
+ return 0;
+}
+
+static inline int
+devlink_dpipe_action_put(struct sk_buff *skb,
+ struct devlink_dpipe_action *action)
+{
+ return 0;
+}
+
+static inline int
+devlink_dpipe_match_put(struct sk_buff *skb,
+ struct devlink_dpipe_match *match)
+{
+ return 0;
+}
+
#endif
#endif /* _NET_DEVLINK_H_ */
diff --git a/include/net/dsa.h b/include/net/dsa.h
index 4e13e69..8e24677 100644
--- a/include/net/dsa.h
+++ b/include/net/dsa.h
@@ -19,6 +19,7 @@
#include <linux/workqueue.h>
#include <linux/of.h>
#include <linux/ethtool.h>
+#include <net/devlink.h>
struct tc_action;
struct phy_device;
@@ -31,6 +32,8 @@
DSA_TAG_PROTO_EDSA,
DSA_TAG_PROTO_BRCM,
DSA_TAG_PROTO_QCA,
+ DSA_TAG_PROTO_MTK,
+ DSA_TAG_PROTO_LAN9303,
DSA_TAG_LAST, /* MUST BE LAST */
};
@@ -122,7 +125,7 @@
* protocol to use.
*/
struct net_device *master_netdev;
- int (*rcv)(struct sk_buff *skb,
+ struct sk_buff * (*rcv)(struct sk_buff *skb,
struct net_device *dev,
struct packet_type *pt,
struct net_device *orig_dev);
@@ -182,6 +185,7 @@
unsigned int ageing_time;
u8 stp_state;
struct net_device *bridge_dev;
+ struct devlink_port devlink_port;
};
struct dsa_switch {
@@ -233,6 +237,13 @@
u32 phys_mii_mask;
struct mii_bus *slave_mii_bus;
+ /* Ageing Time limits in msecs */
+ unsigned int ageing_time_min;
+ unsigned int ageing_time_max;
+
+ /* devlink used to represent this switch device */
+ struct devlink *devlink;
+
/* Dynamically allocated ports, keep last */
size_t num_ports;
struct dsa_port ports[];
@@ -248,6 +259,11 @@
return !!((ds->dsa_port_mask) & (1 << p));
}
+static inline bool dsa_is_normal_port(struct dsa_switch *ds, int p)
+{
+ return !dsa_is_cpu_port(ds, p) && !dsa_is_dsa_port(ds, p);
+}
+
static inline bool dsa_is_port_initialized(struct dsa_switch *ds, int p)
{
return ds->enabled_port_mask & (1 << p) && ds->ports[p].netdev;
@@ -287,7 +303,7 @@
struct dsa_switch_ops {
/*
- * Probing and setup.
+ * Legacy probing.
*/
const char *(*probe)(struct device *dsa_dev,
struct device *host_dev, int sw_addr,
@@ -442,6 +458,14 @@
bool ingress);
void (*port_mirror_del)(struct dsa_switch *ds, int port,
struct dsa_mall_mirror_tc_entry *mirror);
+
+ /*
+ * Cross-chip operations
+ */
+ int (*crosschip_bridge_join)(struct dsa_switch *ds, int sw_index,
+ int port, struct net_device *br);
+ void (*crosschip_bridge_leave)(struct dsa_switch *ds, int sw_index,
+ int port, struct net_device *br);
};
struct dsa_switch_driver {
@@ -449,9 +473,11 @@
const struct dsa_switch_ops *ops;
};
+/* Legacy driver registration */
void register_switch_driver(struct dsa_switch_driver *type);
void unregister_switch_driver(struct dsa_switch_driver *type);
struct mii_bus *dsa_host_dev_to_mii_bus(struct device *dev);
+
struct net_device *dsa_dev_to_net_device(struct device *dev);
static inline bool dsa_uses_tagged_protocol(struct dsa_switch_tree *dst)
@@ -459,6 +485,15 @@
return dst->rcv != NULL;
}
+static inline bool netdev_uses_dsa(struct net_device *dev)
+{
+#if IS_ENABLED(CONFIG_NET_DSA)
+ if (dev->dsa_ptr != NULL)
+ return dsa_uses_tagged_protocol(dev->dsa_ptr);
+#endif
+ return false;
+}
+
struct dsa_switch *dsa_switch_alloc(struct device *dev, size_t n);
void dsa_unregister_switch(struct dsa_switch *ds);
int dsa_register_switch(struct dsa_switch *ds, struct device *dev);
diff --git a/include/net/esp.h b/include/net/esp.h
index a43be85..c41994d 100644
--- a/include/net/esp.h
+++ b/include/net/esp.h
@@ -10,4 +10,23 @@
return (struct ip_esp_hdr *)skb_transport_header(skb);
}
+struct esp_info {
+ struct ip_esp_hdr *esph;
+ __be64 seqno;
+ int tfclen;
+ int tailen;
+ int plen;
+ int clen;
+ int len;
+ int nfrags;
+ __u8 proto;
+ bool inplace;
+};
+
+int esp_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *esp);
+int esp_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *esp);
+int esp_input_done2(struct sk_buff *skb, int err);
+int esp6_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *esp);
+int esp6_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *esp);
+int esp6_input_done2(struct sk_buff *skb, int err);
#endif
diff --git a/include/net/fib_rules.h b/include/net/fib_rules.h
index 8dbfdf7..76c7300 100644
--- a/include/net/fib_rules.h
+++ b/include/net/fib_rules.h
@@ -141,7 +141,10 @@
struct fib_lookup_arg *);
int fib_default_rule_add(struct fib_rules_ops *, u32 pref, u32 table,
u32 flags);
+bool fib_rule_matchall(const struct fib_rule *rule);
-int fib_nl_newrule(struct sk_buff *skb, struct nlmsghdr *nlh);
-int fib_nl_delrule(struct sk_buff *skb, struct nlmsghdr *nlh);
+int fib_nl_newrule(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack);
+int fib_nl_delrule(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack);
#endif
diff --git a/include/net/flow.h b/include/net/flow.h
index 6984f19..bae198b 100644
--- a/include/net/flow.h
+++ b/include/net/flow.h
@@ -202,7 +202,7 @@
typedef unsigned long flow_compare_t;
-static inline size_t flow_key_size(u16 family)
+static inline unsigned int flow_key_size(u16 family)
{
switch (family) {
case AF_INET:
diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h
index ac97030..8d21d44 100644
--- a/include/net/flow_dissector.h
+++ b/include/net/flow_dissector.h
@@ -41,6 +41,13 @@
u16 padding;
};
+struct flow_dissector_key_mpls {
+ u32 mpls_ttl:8,
+ mpls_bos:1,
+ mpls_tc:3,
+ mpls_label:20;
+};
+
struct flow_dissector_key_keyid {
__be32 keyid;
};
@@ -169,6 +176,7 @@
FLOW_DISSECTOR_KEY_ENC_IPV6_ADDRS, /* struct flow_dissector_key_ipv6_addrs */
FLOW_DISSECTOR_KEY_ENC_CONTROL, /* struct flow_dissector_key_control */
FLOW_DISSECTOR_KEY_ENC_PORTS, /* struct flow_dissector_key_ports */
+ FLOW_DISSECTOR_KEY_MPLS, /* struct flow_dissector_key_mpls */
FLOW_DISSECTOR_KEY_MAX,
};
diff --git a/include/net/flowcache.h b/include/net/flowcache.h
index 9caf3bf..51eb971 100644
--- a/include/net/flowcache.h
+++ b/include/net/flowcache.h
@@ -8,7 +8,7 @@
struct flow_cache_percpu {
struct hlist_head *hash_table;
- int hash_count;
+ unsigned int hash_count;
u32 hash_rnd;
int hash_rnd_recalc;
struct tasklet_struct flush_tasklet;
@@ -18,8 +18,8 @@
u32 hash_shift;
struct flow_cache_percpu __percpu *percpu;
struct hlist_node node;
- int low_watermark;
- int high_watermark;
+ unsigned int low_watermark;
+ unsigned int high_watermark;
struct timer_list rnd_timer;
};
#endif /* _NET_FLOWCACHE_H */
diff --git a/include/net/genetlink.h b/include/net/genetlink.h
index a34275b..68b8819 100644
--- a/include/net/genetlink.h
+++ b/include/net/genetlink.h
@@ -84,6 +84,7 @@
* @attrs: netlink attributes
* @_net: network namespace
* @user_ptr: user pointers
+ * @extack: extended ACK report struct
*/
struct genl_info {
u32 snd_seq;
@@ -94,6 +95,7 @@
struct nlattr ** attrs;
possible_net_t _net;
void * user_ptr[2];
+ struct netlink_ext_ack *extack;
};
static inline struct net *genl_info_net(struct genl_info *info)
@@ -106,6 +108,16 @@
write_pnet(&info->_net, net);
}
+#define GENL_SET_ERR_MSG(info, msg) NL_SET_ERR_MSG((info)->extack, msg)
+
+static inline int genl_err_attr(struct genl_info *info, int err,
+ struct nlattr *attr)
+{
+ info->extack->bad_attr = attr;
+
+ return err;
+}
+
/**
* struct genl_ops - generic netlink operations
* @cmd: command identifier
@@ -162,14 +174,16 @@
* @tb: destination array with maxtype+1 elements
* @maxtype: maximum attribute type to be expected
* @policy: validation policy
- * */
+ * @extack: extended ACK report struct
+ */
static inline int genlmsg_parse(const struct nlmsghdr *nlh,
const struct genl_family *family,
struct nlattr *tb[], int maxtype,
- const struct nla_policy *policy)
+ const struct nla_policy *policy,
+ struct netlink_ext_ack *extack)
{
return nlmsg_parse(nlh, family->hdrsize + GENL_HDRLEN, tb, maxtype,
- policy);
+ policy, extack);
}
/**
diff --git a/include/net/ip.h b/include/net/ip.h
index bf264a8..821cedc 100644
--- a/include/net/ip.h
+++ b/include/net/ip.h
@@ -33,6 +33,8 @@
#include <net/flow.h>
#include <net/flow_dissector.h>
+#define IPV4_MAX_PMTU 65535U /* RFC 2675, Section 5.1 */
+
struct sock;
struct inet_skb_parm {
diff --git a/include/net/ip6_tunnel.h b/include/net/ip6_tunnel.h
index 1b1cf33..08fbc7f 100644
--- a/include/net/ip6_tunnel.h
+++ b/include/net/ip6_tunnel.h
@@ -33,6 +33,8 @@
__be16 o_flags;
__be32 i_key;
__be32 o_key;
+
+ __u32 fwmark;
};
/* IPv6 tunnel */
diff --git a/include/net/ip_fib.h b/include/net/ip_fib.h
index 368bb40..6692c57 100644
--- a/include/net/ip_fib.h
+++ b/include/net/ip_fib.h
@@ -213,6 +213,11 @@
u32 tb_id;
};
+struct fib_rule_notifier_info {
+ struct fib_notifier_info info; /* must be first */
+ struct fib_rule *rule;
+};
+
struct fib_nh_notifier_info {
struct fib_notifier_info info; /* must be first */
struct fib_nh *fib_nh;
@@ -232,9 +237,21 @@
int register_fib_notifier(struct notifier_block *nb,
void (*cb)(struct notifier_block *nb));
int unregister_fib_notifier(struct notifier_block *nb);
+int call_fib_notifier(struct notifier_block *nb, struct net *net,
+ enum fib_event_type event_type,
+ struct fib_notifier_info *info);
int call_fib_notifiers(struct net *net, enum fib_event_type event_type,
struct fib_notifier_info *info);
+void fib_notify(struct net *net, struct notifier_block *nb);
+#ifdef CONFIG_IP_MULTIPLE_TABLES
+void fib_rules_notify(struct net *net, struct notifier_block *nb);
+#else
+static inline void fib_rules_notify(struct net *net, struct notifier_block *nb)
+{
+}
+#endif
+
struct fib_table {
struct hlist_node tb_hlist;
u32 tb_id;
@@ -299,6 +316,11 @@
return err;
}
+static inline bool fib4_rule_default(const struct fib_rule *rule)
+{
+ return true;
+}
+
#else /* CONFIG_IP_MULTIPLE_TABLES */
int __net_init fib4_rules_init(struct net *net);
void __net_exit fib4_rules_exit(struct net *net);
@@ -343,6 +365,8 @@
return err;
}
+bool fib4_rule_default(const struct fib_rule *rule);
+
#endif /* CONFIG_IP_MULTIPLE_TABLES */
/* Exported by fib_frontend.c */
@@ -371,17 +395,13 @@
int fib_sync_down_addr(struct net_device *dev, __be32 local);
int fib_sync_up(struct net_device *dev, unsigned int nh_flags);
-extern u32 fib_multipath_secret __read_mostly;
-
-static inline int fib_multipath_hash(__be32 saddr, __be32 daddr)
-{
- return jhash_2words((__force u32)saddr, (__force u32)daddr,
- fib_multipath_secret) >> 1;
-}
-
+#ifdef CONFIG_IP_ROUTE_MULTIPATH
+int fib_multipath_hash(const struct fib_info *fi, const struct flowi4 *fl4,
+ const struct sk_buff *skb);
+#endif
void fib_select_multipath(struct fib_result *res, int hash);
void fib_select_path(struct net *net, struct fib_result *res,
- struct flowi4 *fl4, int mp_hash);
+ struct flowi4 *fl4, const struct sk_buff *skb);
/* Exported by fib_trie.c */
void fib_trie_init(void);
diff --git a/include/net/ip_tunnels.h b/include/net/ip_tunnels.h
index 9505679..5208099 100644
--- a/include/net/ip_tunnels.h
+++ b/include/net/ip_tunnels.h
@@ -132,6 +132,7 @@
unsigned int prl_count; /* # of entries in PRL */
unsigned int ip_tnl_net_id;
struct gro_cells gro_cells;
+ __u32 fwmark;
bool collect_md;
bool ignore_df;
};
@@ -273,9 +274,9 @@
const struct tnl_ptk_info *tpi, struct metadata_dst *tun_dst,
bool log_ecn_error);
int ip_tunnel_changelink(struct net_device *dev, struct nlattr *tb[],
- struct ip_tunnel_parm *p);
+ struct ip_tunnel_parm *p, __u32 fwmark);
int ip_tunnel_newlink(struct net_device *dev, struct nlattr *tb[],
- struct ip_tunnel_parm *p);
+ struct ip_tunnel_parm *p, __u32 fwmark);
void ip_tunnel_setup(struct net_device *dev, unsigned int net_id);
struct ip_tunnel_encap_ops {
diff --git a/include/net/ip_vs.h b/include/net/ip_vs.h
index 7bdfa7d..4f4f786 100644
--- a/include/net/ip_vs.h
+++ b/include/net/ip_vs.h
@@ -12,6 +12,8 @@
#include <linux/list.h> /* for struct list_head */
#include <linux/spinlock.h> /* for struct rwlock_t */
#include <linux/atomic.h> /* for struct atomic_t */
+#include <linux/refcount.h> /* for struct refcount_t */
+
#include <linux/compiler.h>
#include <linux/timer.h>
#include <linux/bug.h>
@@ -525,7 +527,7 @@
struct netns_ipvs *ipvs;
/* counter and timer */
- atomic_t refcnt; /* reference count */
+ refcount_t refcnt; /* reference count */
struct timer_list timer; /* Expiration timer */
volatile unsigned long timeout; /* timeout */
@@ -667,7 +669,7 @@
atomic_t conn_flags; /* flags to copy to conn */
atomic_t weight; /* server weight */
- atomic_t refcnt; /* reference counter */
+ refcount_t refcnt; /* reference counter */
struct ip_vs_stats stats; /* statistics */
unsigned long idle_start; /* start time, jiffies */
@@ -1211,14 +1213,14 @@
*/
static inline bool __ip_vs_conn_get(struct ip_vs_conn *cp)
{
- return atomic_inc_not_zero(&cp->refcnt);
+ return refcount_inc_not_zero(&cp->refcnt);
}
/* put back the conn without restarting its timer */
static inline void __ip_vs_conn_put(struct ip_vs_conn *cp)
{
smp_mb__before_atomic();
- atomic_dec(&cp->refcnt);
+ refcount_dec(&cp->refcnt);
}
void ip_vs_conn_put(struct ip_vs_conn *cp);
void ip_vs_conn_fill_cport(struct ip_vs_conn *cp, __be16 cport);
@@ -1347,8 +1349,6 @@
void ip_vs_protocol_cleanup(void);
void ip_vs_protocol_timeout_change(struct netns_ipvs *ipvs, int flags);
int *ip_vs_create_timeout_table(int *table, int size);
-int ip_vs_set_state_timeout(int *table, int num, const char *const *names,
- const char *name, int to);
void ip_vs_tcpudp_debug_packet(int af, struct ip_vs_protocol *pp,
const struct sk_buff *skb, int offset,
const char *msg);
@@ -1410,18 +1410,18 @@
static inline void ip_vs_dest_hold(struct ip_vs_dest *dest)
{
- atomic_inc(&dest->refcnt);
+ refcount_inc(&dest->refcnt);
}
static inline void ip_vs_dest_put(struct ip_vs_dest *dest)
{
smp_mb__before_atomic();
- atomic_dec(&dest->refcnt);
+ refcount_dec(&dest->refcnt);
}
static inline void ip_vs_dest_put_and_free(struct ip_vs_dest *dest)
{
- if (atomic_dec_and_test(&dest->refcnt))
+ if (refcount_dec_and_test(&dest->refcnt))
kfree(dest);
}
@@ -1553,13 +1553,9 @@
enum ip_conntrack_info ctinfo;
struct nf_conn *ct = nf_ct_get(skb, &ctinfo);
- if (!ct || !nf_ct_is_untracked(ct)) {
- struct nf_conn *untracked;
-
+ if (ct) {
nf_conntrack_put(&ct->ct_general);
- untracked = nf_ct_untracked_get();
- nf_conntrack_get(&untracked->ct_general);
- nf_ct_set(skb, untracked, IP_CT_NEW);
+ nf_ct_set(skb, NULL, IP_CT_UNTRACKED);
}
#endif
}
@@ -1618,7 +1614,7 @@
if (!(cp->flags & IP_VS_CONN_F_NFCT))
return false;
ct = nf_ct_get(skb, &ctinfo);
- if (ct && !nf_ct_is_untracked(ct))
+ if (ct)
return true;
#endif
return false;
diff --git a/include/net/mac80211.h b/include/net/mac80211.h
index a3bab3c..4d05a94 100644
--- a/include/net/mac80211.h
+++ b/include/net/mac80211.h
@@ -5,7 +5,7 @@
* Copyright 2006-2007 Jiri Benc <jbenc@suse.cz>
* Copyright 2007-2010 Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
- * Copyright (C) 2015 - 2016 Intel Deutschland GmbH
+ * Copyright (C) 2015 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -299,6 +299,8 @@
* context had been assigned.
* @BSS_CHANGED_OCB: OCB join status changed
* @BSS_CHANGED_MU_GROUPS: VHT MU-MIMO group id or user position changed
+ * @BSS_CHANGED_KEEP_ALIVE: keep alive options (idle period or protected
+ * keep alive) changed.
*/
enum ieee80211_bss_change {
BSS_CHANGED_ASSOC = 1<<0,
@@ -325,6 +327,7 @@
BSS_CHANGED_BANDWIDTH = 1<<21,
BSS_CHANGED_OCB = 1<<22,
BSS_CHANGED_MU_GROUPS = 1<<23,
+ BSS_CHANGED_KEEP_ALIVE = 1<<24,
/* when adding here, make sure to change ieee80211_reconfig */
};
@@ -501,6 +504,10 @@
* implies disabled. As with the cfg80211 callback, a change here should
* cause an event to be sent indicating where the current value is in
* relation to the newly configured threshold.
+ * @cqm_rssi_low: Connection quality monitor RSSI lower threshold, a zero value
+ * implies disabled. This is an alternative mechanism to the single
+ * threshold event and can't be enabled simultaneously with it.
+ * @cqm_rssi_high: Connection quality monitor RSSI upper threshold.
* @cqm_rssi_hyst: Connection quality monitor RSSI hysteresis
* @arp_addr_list: List of IPv4 addresses for hardware ARP filtering. The
* may filter ARP queries targeted for other addresses than listed here.
@@ -529,6 +536,13 @@
* @allow_p2p_go_ps: indication for AP or P2P GO interface, whether it's allowed
* to use P2P PS mechanism or not. AP/P2P GO is not allowed to use P2P PS
* if it has associated clients without P2P PS support.
+ * @max_idle_period: the time period during which the station can refrain from
+ * transmitting frames to its associated AP without being disassociated.
+ * In units of 1000 TUs. Zero value indicates that the AP did not include
+ * a (valid) BSS Max Idle Period Element.
+ * @protected_keep_alive: if set, indicates that the station should send an RSN
+ * protected frame to the AP to reset the idle timer at the AP for the
+ * station.
*/
struct ieee80211_bss_conf {
const u8 *bssid;
@@ -553,6 +567,8 @@
u16 ht_operation_mode;
s32 cqm_rssi_thold;
u32 cqm_rssi_hyst;
+ s32 cqm_rssi_low;
+ s32 cqm_rssi_high;
struct cfg80211_chan_def chandef;
struct ieee80211_mu_group_data mu_group;
__be32 arp_addr_list[IEEE80211_BSS_ARP_ADDR_LIST_LEN];
@@ -567,6 +583,8 @@
enum nl80211_tx_power_setting txpower_type;
struct ieee80211_p2p_noa_attr p2p_noa_attr;
bool allow_p2p_go_ps;
+ u16 max_idle_period;
+ bool protected_keep_alive;
};
/**
@@ -943,6 +961,19 @@
};
/**
+ * struct ieee80211_tx_status - extended tx staus info for rate control
+ *
+ * @sta: Station that the packet was transmitted for
+ * @info: Basic tx status information
+ * @skb: Packet skb (can be NULL if not provided by the driver)
+ */
+struct ieee80211_tx_status {
+ struct ieee80211_sta *sta;
+ struct ieee80211_tx_info *info;
+ struct sk_buff *skb;
+};
+
+/**
* struct ieee80211_scan_ies - descriptors for different blocks of IEs
*
* This structure is used to point to different blocks of IEs in HW scan
@@ -1039,16 +1070,8 @@
* (including FCS) was received.
* @RX_FLAG_MACTIME_PLCP_START: The timestamp passed in the RX status (@mactime
* field) is valid and contains the time the SYNC preamble was received.
- * @RX_FLAG_SHORTPRE: Short preamble was used for this frame
- * @RX_FLAG_HT: HT MCS was used and rate_idx is MCS index
- * @RX_FLAG_VHT: VHT MCS was used and rate_index is MCS index
- * @RX_FLAG_40MHZ: HT40 (40 MHz) was used
- * @RX_FLAG_SHORT_GI: Short guard interval was used
* @RX_FLAG_NO_SIGNAL_VAL: The signal strength value is not present.
* Valid only for data frames (mainly A-MPDU)
- * @RX_FLAG_HT_GF: This frame was received in a HT-greenfield transmission, if
- * the driver fills this value it should add %IEEE80211_RADIOTAP_MCS_HAVE_FMT
- * to hw.radiotap_mcs_details to advertise that fact
* @RX_FLAG_AMPDU_DETAILS: A-MPDU details are known, in particular the reference
* number (@ampdu_reference) must be populated and be a distinct number for
* each A-MPDU
@@ -1061,7 +1084,6 @@
* is stored in the @ampdu_delimiter_crc field)
* @RX_FLAG_MIC_STRIPPED: The mic was stripped of this packet. Decryption was
* done by the hardware
- * @RX_FLAG_LDPC: LDPC was used
* @RX_FLAG_ONLY_MONITOR: Report frame only to monitor interfaces without
* processing it in any regular way.
* This is useful if drivers offload some frames but still want to report
@@ -1070,9 +1092,6 @@
* monitor interfaces.
* This is useful if drivers offload some frames but still want to report
* them for sniffing purposes.
- * @RX_FLAG_STBC_MASK: STBC 2 bit bitmask. 1 - Nss=1, 2 - Nss=2, 3 - Nss=3
- * @RX_FLAG_10MHZ: 10 MHz (half channel) was used
- * @RX_FLAG_5MHZ: 5 MHz (quarter channel) was used
* @RX_FLAG_AMSDU_MORE: Some drivers may prefer to report separate A-MSDU
* subframes instead of a one huge frame for performance reasons.
* All, but the last MSDU from an A-MSDU should have this flag set. E.g.
@@ -1100,50 +1119,54 @@
RX_FLAG_FAILED_FCS_CRC = BIT(5),
RX_FLAG_FAILED_PLCP_CRC = BIT(6),
RX_FLAG_MACTIME_START = BIT(7),
- RX_FLAG_SHORTPRE = BIT(8),
- RX_FLAG_HT = BIT(9),
- RX_FLAG_40MHZ = BIT(10),
- RX_FLAG_SHORT_GI = BIT(11),
- RX_FLAG_NO_SIGNAL_VAL = BIT(12),
- RX_FLAG_HT_GF = BIT(13),
- RX_FLAG_AMPDU_DETAILS = BIT(14),
- RX_FLAG_PN_VALIDATED = BIT(15),
- RX_FLAG_DUP_VALIDATED = BIT(16),
- RX_FLAG_AMPDU_LAST_KNOWN = BIT(17),
- RX_FLAG_AMPDU_IS_LAST = BIT(18),
- RX_FLAG_AMPDU_DELIM_CRC_ERROR = BIT(19),
- RX_FLAG_AMPDU_DELIM_CRC_KNOWN = BIT(20),
- RX_FLAG_MACTIME_END = BIT(21),
- RX_FLAG_VHT = BIT(22),
- RX_FLAG_LDPC = BIT(23),
- RX_FLAG_ONLY_MONITOR = BIT(24),
- RX_FLAG_SKIP_MONITOR = BIT(25),
- RX_FLAG_STBC_MASK = BIT(26) | BIT(27),
- RX_FLAG_10MHZ = BIT(28),
- RX_FLAG_5MHZ = BIT(29),
- RX_FLAG_AMSDU_MORE = BIT(30),
- RX_FLAG_RADIOTAP_VENDOR_DATA = BIT(31),
- RX_FLAG_MIC_STRIPPED = BIT_ULL(32),
- RX_FLAG_ALLOW_SAME_PN = BIT_ULL(33),
- RX_FLAG_ICV_STRIPPED = BIT_ULL(34),
+ RX_FLAG_NO_SIGNAL_VAL = BIT(8),
+ RX_FLAG_AMPDU_DETAILS = BIT(9),
+ RX_FLAG_PN_VALIDATED = BIT(10),
+ RX_FLAG_DUP_VALIDATED = BIT(11),
+ RX_FLAG_AMPDU_LAST_KNOWN = BIT(12),
+ RX_FLAG_AMPDU_IS_LAST = BIT(13),
+ RX_FLAG_AMPDU_DELIM_CRC_ERROR = BIT(14),
+ RX_FLAG_AMPDU_DELIM_CRC_KNOWN = BIT(15),
+ RX_FLAG_MACTIME_END = BIT(16),
+ RX_FLAG_ONLY_MONITOR = BIT(17),
+ RX_FLAG_SKIP_MONITOR = BIT(18),
+ RX_FLAG_AMSDU_MORE = BIT(19),
+ RX_FLAG_RADIOTAP_VENDOR_DATA = BIT(20),
+ RX_FLAG_MIC_STRIPPED = BIT(21),
+ RX_FLAG_ALLOW_SAME_PN = BIT(22),
+ RX_FLAG_ICV_STRIPPED = BIT(23),
};
-#define RX_FLAG_STBC_SHIFT 26
-
/**
- * enum mac80211_rx_vht_flags - receive VHT flags
+ * enum mac80211_rx_encoding_flags - MCS & bandwidth flags
*
- * These flags are used with the @vht_flag member of
- * &struct ieee80211_rx_status.
- * @RX_VHT_FLAG_80MHZ: 80 MHz was used
- * @RX_VHT_FLAG_160MHZ: 160 MHz was used
- * @RX_VHT_FLAG_BF: packet was beamformed
+ * @RX_ENC_FLAG_SHORTPRE: Short preamble was used for this frame
+ * @RX_ENC_FLAG_40MHZ: HT40 (40 MHz) was used
+ * @RX_ENC_FLAG_SHORT_GI: Short guard interval was used
+ * @RX_ENC_FLAG_HT_GF: This frame was received in a HT-greenfield transmission,
+ * if the driver fills this value it should add
+ * %IEEE80211_RADIOTAP_MCS_HAVE_FMT
+ * to hw.radiotap_mcs_details to advertise that fact
+ * @RX_ENC_FLAG_LDPC: LDPC was used
+ * @RX_ENC_FLAG_STBC_MASK: STBC 2 bit bitmask. 1 - Nss=1, 2 - Nss=2, 3 - Nss=3
+ * @RX_ENC_FLAG_BF: packet was beamformed
*/
+enum mac80211_rx_encoding_flags {
+ RX_ENC_FLAG_SHORTPRE = BIT(0),
+ RX_ENC_FLAG_40MHZ = BIT(1),
+ RX_ENC_FLAG_SHORT_GI = BIT(2),
+ RX_ENC_FLAG_HT_GF = BIT(3),
+ RX_ENC_FLAG_STBC_MASK = BIT(4) | BIT(5),
+ RX_ENC_FLAG_LDPC = BIT(6),
+ RX_ENC_FLAG_BF = BIT(7),
+};
-enum mac80211_rx_vht_flags {
- RX_VHT_FLAG_80MHZ = BIT(0),
- RX_VHT_FLAG_160MHZ = BIT(1),
- RX_VHT_FLAG_BF = BIT(2),
+#define RX_ENC_FLAG_STBC_SHIFT 4
+
+enum mac80211_rx_encoding {
+ RX_ENC_LEGACY = 0,
+ RX_ENC_HT,
+ RX_ENC_VHT,
};
/**
@@ -1173,9 +1196,11 @@
* @antenna: antenna used
* @rate_idx: index of data rate into band's supported rates or MCS index if
* HT or VHT is used (%RX_FLAG_HT/%RX_FLAG_VHT)
- * @vht_nss: number of streams (VHT only)
+ * @nss: number of streams (VHT and HE only)
* @flag: %RX_FLAG_\*
- * @vht_flag: %RX_VHT_FLAG_\*
+ * @encoding: &enum mac80211_rx_encoding
+ * @bw: &enum rate_info_bw
+ * @enc_flags: uses bits from &enum mac80211_rx_encoding_flags
* @rx_flags: internal RX flags for mac80211
* @ampdu_reference: A-MPDU reference number, must be a different value for
* each A-MPDU but the same for each subframe within one A-MPDU
@@ -1186,11 +1211,12 @@
u64 boottime_ns;
u32 device_timestamp;
u32 ampdu_reference;
- u64 flag;
+ u32 flag;
u16 freq;
- u8 vht_flag;
+ u8 enc_flags;
+ u8 encoding:2, bw:3;
u8 rate_idx;
- u8 vht_nss;
+ u8 nss;
u8 rx_flags;
u8 band;
u8 antenna;
@@ -4200,6 +4226,23 @@
struct sk_buff *skb);
/**
+ * ieee80211_tx_status_ext - extended transmit status callback
+ *
+ * This function can be used as a replacement for ieee80211_tx_status
+ * in drivers that may want to provide extra information that does not
+ * fit into &struct ieee80211_tx_info.
+ *
+ * Calls to this function for a single hardware must be synchronized
+ * against each other. Calls to this function, ieee80211_tx_status_ni()
+ * and ieee80211_tx_status_irqsafe() may not be mixed for a single hardware.
+ *
+ * @hw: the hardware the frame was transmitted by
+ * @status: tx status information
+ */
+void ieee80211_tx_status_ext(struct ieee80211_hw *hw,
+ struct ieee80211_tx_status *status);
+
+/**
* ieee80211_tx_status_noskb - transmit status callback without skb
*
* This function can be used as a replacement for ieee80211_tx_status
@@ -4215,9 +4258,17 @@
* (NULL for multicast packets)
* @info: tx status information
*/
-void ieee80211_tx_status_noskb(struct ieee80211_hw *hw,
- struct ieee80211_sta *sta,
- struct ieee80211_tx_info *info);
+static inline void ieee80211_tx_status_noskb(struct ieee80211_hw *hw,
+ struct ieee80211_sta *sta,
+ struct ieee80211_tx_info *info)
+{
+ struct ieee80211_tx_status status = {
+ .sta = sta,
+ .info = info,
+ };
+
+ ieee80211_tx_status_ext(hw, &status);
+}
/**
* ieee80211_tx_status_ni - transmit status callback (in process context)
@@ -5438,9 +5489,6 @@
* RTS threshold
* @short_preamble: whether mac80211 will request short-preamble transmission
* if the selected rate supports it
- * @max_rate_idx: user-requested maximum (legacy) rate
- * (deprecated; this will be removed once drivers get updated to use
- * rate_idx_mask)
* @rate_idx_mask: user-requested (legacy) rate mask
* @rate_idx_mcs_mask: user-requested MCS rate mask (NULL if not in use)
* @bss: whether this frame is sent out in AP or IBSS mode
@@ -5452,7 +5500,6 @@
struct sk_buff *skb;
struct ieee80211_tx_rate reported_rate;
bool rts, short_preamble;
- u8 max_rate_idx;
u32 rate_idx_mask;
u8 *rate_idx_mcs_mask;
bool bss;
@@ -5474,10 +5521,9 @@
void (*free_sta)(void *priv, struct ieee80211_sta *sta,
void *priv_sta);
- void (*tx_status_noskb)(void *priv,
- struct ieee80211_supported_band *sband,
- struct ieee80211_sta *sta, void *priv_sta,
- struct ieee80211_tx_info *info);
+ void (*tx_status_ext)(void *priv,
+ struct ieee80211_supported_band *sband,
+ void *priv_sta, struct ieee80211_tx_status *st);
void (*tx_status)(void *priv, struct ieee80211_supported_band *sband,
struct ieee80211_sta *sta, void *priv_sta,
struct sk_buff *skb);
diff --git a/include/net/mpls_iptunnel.h b/include/net/mpls_iptunnel.h
index 179253f..9d22bf6 100644
--- a/include/net/mpls_iptunnel.h
+++ b/include/net/mpls_iptunnel.h
@@ -14,11 +14,12 @@
#ifndef _NET_MPLS_IPTUNNEL_H
#define _NET_MPLS_IPTUNNEL_H 1
-#define MAX_NEW_LABELS 2
-
struct mpls_iptunnel_encap {
- u32 label[MAX_NEW_LABELS];
u8 labels;
+ u8 ttl_propagate;
+ u8 default_ttl;
+ u8 reserved1;
+ u32 label[0];
};
static inline struct mpls_iptunnel_encap *mpls_lwtunnel_encap(struct lwtunnel_state *lwtstate)
diff --git a/include/net/ndisc.h b/include/net/ndisc.h
index 8a02146..1036c90 100644
--- a/include/net/ndisc.h
+++ b/include/net/ndisc.h
@@ -439,8 +439,10 @@
* IGMP
*/
int igmp6_init(void);
+int igmp6_late_init(void);
void igmp6_cleanup(void);
+void igmp6_late_cleanup(void);
int igmp6_event_query(struct sk_buff *skb);
diff --git a/include/net/neighbour.h b/include/net/neighbour.h
index 5ebf694..e4dd3a2 100644
--- a/include/net/neighbour.h
+++ b/include/net/neighbour.h
@@ -314,7 +314,8 @@
}
void neigh_destroy(struct neighbour *neigh);
int __neigh_event_send(struct neighbour *neigh, struct sk_buff *skb);
-int neigh_update(struct neighbour *neigh, const u8 *lladdr, u8 new, u32 flags);
+int neigh_update(struct neighbour *neigh, const u8 *lladdr, u8 new, u32 flags,
+ u32 nlmsg_pid);
void __neigh_set_probe_once(struct neighbour *neigh);
void neigh_changeaddr(struct neigh_table *tbl, struct net_device *dev);
int neigh_ifdown(struct neigh_table *tbl, struct net_device *dev);
@@ -449,7 +450,7 @@
static inline int neigh_hh_output(const struct hh_cache *hh, struct sk_buff *skb)
{
unsigned int seq;
- int hh_len;
+ unsigned int hh_len;
do {
seq = read_seqbegin(&hh->hh_lock);
@@ -458,7 +459,7 @@
/* this is inlined by gcc */
memcpy(skb->data - HH_DATA_MOD, hh->hh_data, HH_DATA_MOD);
} else {
- int hh_alen = HH_DATA_ALIGN(hh_len);
+ unsigned int hh_alen = HH_DATA_ALIGN(hh_len);
memcpy(skb->data - hh_alen, hh->hh_data, hh_alen);
}
diff --git a/include/net/net_namespace.h b/include/net/net_namespace.h
index af8fe8a..fe80bb4 100644
--- a/include/net/net_namespace.h
+++ b/include/net/net_namespace.h
@@ -27,6 +27,7 @@
#include <net/netns/nftables.h>
#include <net/netns/xfrm.h>
#include <net/netns/mpls.h>
+#include <net/netns/can.h>
#include <linux/ns_common.h>
#include <linux/idr.h>
#include <linux/skbuff.h>
@@ -141,6 +142,9 @@
#if IS_ENABLED(CONFIG_MPLS)
struct netns_mpls mpls;
#endif
+#if IS_ENABLED(CONFIG_CAN)
+ struct netns_can can;
+#endif
struct sock *diag_nlsk;
atomic_t fnhe_genid;
};
diff --git a/include/net/netfilter/ipv4/nf_conntrack_ipv4.h b/include/net/netfilter/ipv4/nf_conntrack_ipv4.h
index 6ff3281..919e4e8 100644
--- a/include/net/netfilter/ipv4/nf_conntrack_ipv4.h
+++ b/include/net/netfilter/ipv4/nf_conntrack_ipv4.h
@@ -14,7 +14,6 @@
extern struct nf_conntrack_l4proto nf_conntrack_l4proto_tcp4;
extern struct nf_conntrack_l4proto nf_conntrack_l4proto_udp4;
-extern struct nf_conntrack_l4proto nf_conntrack_l4proto_udplite4;
extern struct nf_conntrack_l4proto nf_conntrack_l4proto_icmp;
#ifdef CONFIG_NF_CT_PROTO_DCCP
extern struct nf_conntrack_l4proto nf_conntrack_l4proto_dccp4;
diff --git a/include/net/netfilter/ipv6/nf_conntrack_ipv6.h b/include/net/netfilter/ipv6/nf_conntrack_ipv6.h
index c59b824..eaea968 100644
--- a/include/net/netfilter/ipv6/nf_conntrack_ipv6.h
+++ b/include/net/netfilter/ipv6/nf_conntrack_ipv6.h
@@ -5,7 +5,6 @@
extern struct nf_conntrack_l4proto nf_conntrack_l4proto_tcp6;
extern struct nf_conntrack_l4proto nf_conntrack_l4proto_udp6;
-extern struct nf_conntrack_l4proto nf_conntrack_l4proto_udplite6;
extern struct nf_conntrack_l4proto nf_conntrack_l4proto_icmpv6;
#ifdef CONFIG_NF_CT_PROTO_DCCP
extern struct nf_conntrack_l4proto nf_conntrack_l4proto_dccp6;
diff --git a/include/net/netfilter/nf_conntrack.h b/include/net/netfilter/nf_conntrack.h
index 1960587..8ece361 100644
--- a/include/net/netfilter/nf_conntrack.h
+++ b/include/net/netfilter/nf_conntrack.h
@@ -50,25 +50,6 @@
#define NF_CT_ASSERT(x)
#endif
-struct nf_conntrack_helper;
-
-/* Must be kept in sync with the classes defined by helpers */
-#define NF_CT_MAX_EXPECT_CLASSES 4
-
-/* nf_conn feature for connections that have a helper */
-struct nf_conn_help {
- /* Helper. if any */
- struct nf_conntrack_helper __rcu *helper;
-
- struct hlist_head expectations;
-
- /* Current number of expected connections */
- u8 expecting[NF_CT_MAX_EXPECT_CLASSES];
-
- /* private helper information. */
- char data[];
-};
-
#include <net/netfilter/ipv4/nf_conntrack_ipv4.h>
#include <net/netfilter/ipv6/nf_conntrack_ipv6.h>
@@ -243,14 +224,6 @@
enum ip_conntrack_dir dir,
u32 seq);
-/* Fake conntrack entry for untracked connections */
-DECLARE_PER_CPU_ALIGNED(struct nf_conn, nf_conntrack_untracked);
-static inline struct nf_conn *nf_ct_untracked_get(void)
-{
- return raw_cpu_ptr(&nf_conntrack_untracked);
-}
-void nf_ct_untracked_status_or(unsigned long bits);
-
/* Iterate over all conntracks: if iter returns true, it's deleted. */
void nf_ct_iterate_cleanup(struct net *net,
int (*iter)(struct nf_conn *i, void *data),
@@ -281,11 +254,6 @@
return test_bit(IPS_DYING_BIT, &ct->status);
}
-static inline int nf_ct_is_untracked(const struct nf_conn *ct)
-{
- return test_bit(IPS_UNTRACKED_BIT, &ct->status);
-}
-
/* Packet is received from loopback */
static inline bool nf_is_loopback_packet(const struct sk_buff *skb)
{
diff --git a/include/net/netfilter/nf_conntrack_core.h b/include/net/netfilter/nf_conntrack_core.h
index 84ec7ca..81d7f8a 100644
--- a/include/net/netfilter/nf_conntrack_core.h
+++ b/include/net/netfilter/nf_conntrack_core.h
@@ -65,7 +65,7 @@
struct nf_conn *ct = (struct nf_conn *)skb_nfct(skb);
int ret = NF_ACCEPT;
- if (ct && !nf_ct_is_untracked(ct)) {
+ if (ct) {
if (!nf_ct_is_confirmed(ct))
ret = __nf_conntrack_confirm(skb);
if (likely(ret == NF_ACCEPT))
diff --git a/include/net/netfilter/nf_conntrack_ecache.h b/include/net/netfilter/nf_conntrack_ecache.h
index 12d967b..2a10c65 100644
--- a/include/net/netfilter/nf_conntrack_ecache.h
+++ b/include/net/netfilter/nf_conntrack_ecache.h
@@ -20,11 +20,11 @@
struct nf_conntrack_ecache {
unsigned long cache; /* bitops want long */
- unsigned long missed; /* missed events */
+ u16 missed; /* missed events */
u16 ctmask; /* bitmask of ct events to be delivered */
u16 expmask; /* bitmask of expect events to be delivered */
+ enum nf_ct_ecache_state state:8;/* ecache state */
u32 portid; /* netlink portid of destroyer */
- enum nf_ct_ecache_state state; /* ecache state */
};
static inline struct nf_conntrack_ecache *
diff --git a/include/net/netfilter/nf_conntrack_expect.h b/include/net/netfilter/nf_conntrack_expect.h
index 5ed33ea..2ba54fe 100644
--- a/include/net/netfilter/nf_conntrack_expect.h
+++ b/include/net/netfilter/nf_conntrack_expect.h
@@ -5,6 +5,8 @@
#ifndef _NF_CONNTRACK_EXPECT_H
#define _NF_CONNTRACK_EXPECT_H
+#include <linux/refcount.h>
+
#include <net/netfilter/nf_conntrack.h>
#include <net/netfilter/nf_conntrack_zones.h>
@@ -37,7 +39,7 @@
struct timer_list timeout;
/* Usage count. */
- atomic_t use;
+ refcount_t use;
/* Flags */
unsigned int flags;
@@ -71,6 +73,7 @@
};
#define NF_CT_EXPECT_CLASS_DEFAULT 0
+#define NF_CT_EXPECT_MAX_CNT 255
int nf_conntrack_expect_pernet_init(struct net *net);
void nf_conntrack_expect_pernet_fini(struct net *net);
@@ -102,6 +105,7 @@
void nf_ct_remove_expectations(struct nf_conn *ct);
void nf_ct_unexpect_related(struct nf_conntrack_expect *exp);
+bool nf_ct_remove_expect(struct nf_conntrack_expect *exp);
/* Allocate space for an expectation: this is mandatory before calling
nf_ct_expect_related. You will have to call put afterwards. */
diff --git a/include/net/netfilter/nf_conntrack_extend.h b/include/net/netfilter/nf_conntrack_extend.h
index 1c3035d..4944bc9 100644
--- a/include/net/netfilter/nf_conntrack_extend.h
+++ b/include/net/netfilter/nf_conntrack_extend.h
@@ -43,8 +43,8 @@
/* Extensions: optional stuff which isn't permanently in struct. */
struct nf_ct_ext {
struct rcu_head rcu;
- u16 offset[NF_CT_EXT_NUM];
- u16 len;
+ u8 offset[NF_CT_EXT_NUM];
+ u8 len;
char data[0];
};
@@ -69,12 +69,7 @@
((id##_TYPE *)__nf_ct_ext_find((ext), (id)))
/* Destroy all relationships */
-void __nf_ct_ext_destroy(struct nf_conn *ct);
-static inline void nf_ct_ext_destroy(struct nf_conn *ct)
-{
- if (ct->ext)
- __nf_ct_ext_destroy(ct);
-}
+void nf_ct_ext_destroy(struct nf_conn *ct);
/* Free operation. If you want to free a object referred from private area,
* please implement __nf_ct_ext_free() and call it.
@@ -86,15 +81,7 @@
}
/* Add this type, returns pointer to data or NULL. */
-void *__nf_ct_ext_add_length(struct nf_conn *ct, enum nf_ct_ext_id id,
- size_t var_alloc_len, gfp_t gfp);
-
-#define nf_ct_ext_add(ct, id, gfp) \
- ((id##_TYPE *)__nf_ct_ext_add_length((ct), (id), 0, (gfp)))
-#define nf_ct_ext_add_length(ct, id, len, gfp) \
- ((id##_TYPE *)__nf_ct_ext_add_length((ct), (id), (len), (gfp)))
-
-#define NF_CT_EXT_F_PREALLOC 0x0001
+void *nf_ct_ext_add(struct nf_conn *ct, enum nf_ct_ext_id id, gfp_t gfp);
struct nf_ct_ext_type {
/* Destroys relationships (can be NULL). */
@@ -102,15 +89,11 @@
enum nf_ct_ext_id id;
- unsigned int flags;
-
/* Length and min alignment. */
u8 len;
u8 align;
- /* initial size of nf_ct_ext. */
- u8 alloc_size;
};
-int nf_ct_extend_register(struct nf_ct_ext_type *type);
-void nf_ct_extend_unregister(struct nf_ct_ext_type *type);
+int nf_ct_extend_register(const struct nf_ct_ext_type *type);
+void nf_ct_extend_unregister(const struct nf_ct_ext_type *type);
#endif /* _NF_CONNTRACK_EXTEND_H */
diff --git a/include/net/netfilter/nf_conntrack_helper.h b/include/net/netfilter/nf_conntrack_helper.h
index 1eaac1f..e04fa769 100644
--- a/include/net/netfilter/nf_conntrack_helper.h
+++ b/include/net/netfilter/nf_conntrack_helper.h
@@ -29,9 +29,6 @@
struct module *me; /* pointer to self */
const struct nf_conntrack_expect_policy *expect_policy;
- /* length of internal data, ie. sizeof(struct nf_ct_*_master) */
- size_t data_len;
-
/* Tuple of things we will help (compared against server response) */
struct nf_conntrack_tuple tuple;
@@ -49,9 +46,33 @@
unsigned int expect_class_max;
unsigned int flags;
- unsigned int queue_num; /* For user-space helpers. */
+
+ /* For user-space helpers: */
+ unsigned int queue_num;
+ /* length of userspace private data stored in nf_conn_help->data */
+ u16 data_len;
};
+/* Must be kept in sync with the classes defined by helpers */
+#define NF_CT_MAX_EXPECT_CLASSES 4
+
+/* nf_conn feature for connections that have a helper */
+struct nf_conn_help {
+ /* Helper. if any */
+ struct nf_conntrack_helper __rcu *helper;
+
+ struct hlist_head expectations;
+
+ /* Current number of expected connections */
+ u8 expecting[NF_CT_MAX_EXPECT_CLASSES];
+
+ /* private helper information. */
+ char data[32] __aligned(8);
+};
+
+#define NF_CT_HELPER_BUILD_BUG_ON(structsize) \
+ BUILD_BUG_ON((structsize) > FIELD_SIZEOF(struct nf_conn_help, data))
+
struct nf_conntrack_helper *__nf_conntrack_helper_find(const char *name,
u16 l3num, u8 protonum);
@@ -62,7 +83,7 @@
u16 l3num, u16 protonum, const char *name,
u16 default_port, u16 spec_port, u32 id,
const struct nf_conntrack_expect_policy *exp_pol,
- u32 expect_class_max, u32 data_len,
+ u32 expect_class_max,
int (*help)(struct sk_buff *skb, unsigned int protoff,
struct nf_conn *ct,
enum ip_conntrack_info ctinfo),
diff --git a/include/net/netfilter/nf_conntrack_l4proto.h b/include/net/netfilter/nf_conntrack_l4proto.h
index 85e993e..7032e04 100644
--- a/include/net/netfilter/nf_conntrack_l4proto.h
+++ b/include/net/netfilter/nf_conntrack_l4proto.h
@@ -58,6 +58,9 @@
unsigned int dataoff,
u_int8_t pf, unsigned int hooknum);
+ /* called by gc worker if table is full */
+ bool (*can_early_drop)(const struct nf_conn *ct);
+
/* Print out the per-protocol part of the tuple. Return like seq_* */
void (*print_tuple)(struct seq_file *s,
const struct nf_conntrack_tuple *);
diff --git a/include/net/netfilter/nf_conntrack_synproxy.h b/include/net/netfilter/nf_conntrack_synproxy.h
index b0ca402..a2fcb52 100644
--- a/include/net/netfilter/nf_conntrack_synproxy.h
+++ b/include/net/netfilter/nf_conntrack_synproxy.h
@@ -52,6 +52,8 @@
struct synproxy_net {
struct nf_conn *tmpl;
struct synproxy_stats __percpu *stats;
+ unsigned int hook_ref4;
+ unsigned int hook_ref6;
};
extern unsigned int synproxy_net_id;
diff --git a/include/net/netfilter/nf_conntrack_timeout.h b/include/net/netfilter/nf_conntrack_timeout.h
index 5cc5e9e..d40b893 100644
--- a/include/net/netfilter/nf_conntrack_timeout.h
+++ b/include/net/netfilter/nf_conntrack_timeout.h
@@ -4,6 +4,7 @@
#include <net/net_namespace.h>
#include <linux/netfilter/nf_conntrack_common.h>
#include <linux/netfilter/nf_conntrack_tuple_common.h>
+#include <linux/refcount.h>
#include <net/netfilter/nf_conntrack.h>
#include <net/netfilter/nf_conntrack_extend.h>
@@ -12,7 +13,7 @@
struct ctnl_timeout {
struct list_head head;
struct rcu_head rcu_head;
- atomic_t refcnt;
+ refcount_t refcnt;
char name[CTNL_TIMEOUT_NAME_MAX];
__u16 l3num;
struct nf_conntrack_l4proto *l4proto;
diff --git a/include/net/netfilter/nf_nat.h b/include/net/netfilter/nf_nat.h
index c327a43..05c82a1 100644
--- a/include/net/netfilter/nf_nat.h
+++ b/include/net/netfilter/nf_nat.h
@@ -67,7 +67,7 @@
{
#if IS_ENABLED(CONFIG_NF_NAT_MASQUERADE_IPV4) || \
IS_ENABLED(CONFIG_NF_NAT_MASQUERADE_IPV6)
- return nat->masq_index && hooknum == NF_INET_POST_ROUTING &&
+ return nat && nat->masq_index && hooknum == NF_INET_POST_ROUTING &&
CTINFO2DIR(ctinfo) == IP_CT_DIR_ORIGINAL &&
nat->masq_index != out->ifindex;
#else
diff --git a/include/net/netfilter/nf_nat_helper.h b/include/net/netfilter/nf_nat_helper.h
index 01bcc6b..fbfa5ac 100644
--- a/include/net/netfilter/nf_nat_helper.h
+++ b/include/net/netfilter/nf_nat_helper.h
@@ -7,31 +7,31 @@
struct sk_buff;
/* These return true or false. */
-int __nf_nat_mangle_tcp_packet(struct sk_buff *skb, struct nf_conn *ct,
- enum ip_conntrack_info ctinfo,
- unsigned int protoff, unsigned int match_offset,
- unsigned int match_len, const char *rep_buffer,
- unsigned int rep_len, bool adjust);
+bool __nf_nat_mangle_tcp_packet(struct sk_buff *skb, struct nf_conn *ct,
+ enum ip_conntrack_info ctinfo,
+ unsigned int protoff, unsigned int match_offset,
+ unsigned int match_len, const char *rep_buffer,
+ unsigned int rep_len, bool adjust);
-static inline int nf_nat_mangle_tcp_packet(struct sk_buff *skb,
- struct nf_conn *ct,
- enum ip_conntrack_info ctinfo,
- unsigned int protoff,
- unsigned int match_offset,
- unsigned int match_len,
- const char *rep_buffer,
- unsigned int rep_len)
+static inline bool nf_nat_mangle_tcp_packet(struct sk_buff *skb,
+ struct nf_conn *ct,
+ enum ip_conntrack_info ctinfo,
+ unsigned int protoff,
+ unsigned int match_offset,
+ unsigned int match_len,
+ const char *rep_buffer,
+ unsigned int rep_len)
{
return __nf_nat_mangle_tcp_packet(skb, ct, ctinfo, protoff,
match_offset, match_len,
rep_buffer, rep_len, true);
}
-int nf_nat_mangle_udp_packet(struct sk_buff *skb, struct nf_conn *ct,
- enum ip_conntrack_info ctinfo,
- unsigned int protoff, unsigned int match_offset,
- unsigned int match_len, const char *rep_buffer,
- unsigned int rep_len);
+bool nf_nat_mangle_udp_packet(struct sk_buff *skb, struct nf_conn *ct,
+ enum ip_conntrack_info ctinfo,
+ unsigned int protoff, unsigned int match_offset,
+ unsigned int match_len, const char *rep_buffer,
+ unsigned int rep_len);
/* Setup NAT on this expected conntrack so it follows master, but goes
* to port ct->master->saved_proto. */
diff --git a/include/net/netfilter/nf_queue.h b/include/net/netfilter/nf_queue.h
index 09948d1..4454719f 100644
--- a/include/net/netfilter/nf_queue.h
+++ b/include/net/netfilter/nf_queue.h
@@ -24,8 +24,7 @@
struct nf_queue_handler {
int (*outfn)(struct nf_queue_entry *entry,
unsigned int queuenum);
- void (*nf_hook_drop)(struct net *net,
- const struct nf_hook_entry *hooks);
+ unsigned int (*nf_hook_drop)(struct net *net);
};
void nf_register_queue_handler(struct net *net, const struct nf_queue_handler *qh);
diff --git a/include/net/netfilter/nf_tables.h b/include/net/netfilter/nf_tables.h
index 01360286..028faec8 100644
--- a/include/net/netfilter/nf_tables.h
+++ b/include/net/netfilter/nf_tables.h
@@ -413,10 +413,11 @@
return (void *)priv - offsetof(struct nft_set, data);
}
-struct nft_set *nf_tables_set_lookup(const struct nft_table *table,
- const struct nlattr *nla, u8 genmask);
-struct nft_set *nf_tables_set_lookup_byid(const struct net *net,
- const struct nlattr *nla, u8 genmask);
+struct nft_set *nft_set_lookup(const struct net *net,
+ const struct nft_table *table,
+ const struct nlattr *nla_set_name,
+ const struct nlattr *nla_set_id,
+ u8 genmask);
static inline unsigned long nft_set_gc_interval(const struct nft_set *set)
{
@@ -910,6 +911,11 @@
return container_of(chain, struct nft_base_chain, chain);
}
+static inline bool nft_is_base_chain(const struct nft_chain *chain)
+{
+ return chain->flags & NFT_BASE_CHAIN;
+}
+
int __nft_release_basechain(struct nft_ctx *ctx);
unsigned int nft_do_chain(struct nft_pktinfo *pkt, void *priv);
@@ -1044,7 +1050,8 @@
unsigned int maxattr;
struct module *owner;
const struct nla_policy *policy;
- int (*init)(const struct nlattr * const tb[],
+ int (*init)(const struct nft_ctx *ctx,
+ const struct nlattr *const tb[],
struct nft_object *obj);
void (*destroy)(struct nft_object *obj);
int (*dump)(struct sk_buff *skb,
diff --git a/include/net/netfilter/nft_fib.h b/include/net/netfilter/nft_fib.h
index 5ceb220..381af94 100644
--- a/include/net/netfilter/nft_fib.h
+++ b/include/net/netfilter/nft_fib.h
@@ -32,6 +32,6 @@
void nft_fib6_eval(const struct nft_expr *expr, struct nft_regs *regs,
const struct nft_pktinfo *pkt);
-void nft_fib_store_result(void *reg, enum nft_fib_result r,
+void nft_fib_store_result(void *reg, const struct nft_fib *priv,
const struct nft_pktinfo *pkt, int index);
#endif
diff --git a/include/net/netlink.h b/include/net/netlink.h
index b239fcd..0170917 100644
--- a/include/net/netlink.h
+++ b/include/net/netlink.h
@@ -233,14 +233,17 @@
};
int netlink_rcv_skb(struct sk_buff *skb,
- int (*cb)(struct sk_buff *, struct nlmsghdr *));
+ int (*cb)(struct sk_buff *, struct nlmsghdr *,
+ struct netlink_ext_ack *));
int nlmsg_notify(struct sock *sk, struct sk_buff *skb, u32 portid,
unsigned int group, int report, gfp_t flags);
int nla_validate(const struct nlattr *head, int len, int maxtype,
- const struct nla_policy *policy);
+ const struct nla_policy *policy,
+ struct netlink_ext_ack *extack);
int nla_parse(struct nlattr **tb, int maxtype, const struct nlattr *head,
- int len, const struct nla_policy *policy);
+ int len, const struct nla_policy *policy,
+ struct netlink_ext_ack *extack);
int nla_policy_len(const struct nla_policy *, int);
struct nlattr *nla_find(const struct nlattr *head, int len, int attrtype);
size_t nla_strlcpy(char *dst, const struct nlattr *nla, size_t dstsize);
@@ -374,18 +377,20 @@
* @tb: destination array with maxtype+1 elements
* @maxtype: maximum attribute type to be expected
* @policy: validation policy
+ * @extack: extended ACK report struct
*
* See nla_parse()
*/
static inline int nlmsg_parse(const struct nlmsghdr *nlh, int hdrlen,
struct nlattr *tb[], int maxtype,
- const struct nla_policy *policy)
+ const struct nla_policy *policy,
+ struct netlink_ext_ack *extack)
{
if (nlh->nlmsg_len < nlmsg_msg_size(hdrlen))
return -EINVAL;
return nla_parse(tb, maxtype, nlmsg_attrdata(nlh, hdrlen),
- nlmsg_attrlen(nlh, hdrlen), policy);
+ nlmsg_attrlen(nlh, hdrlen), policy, extack);
}
/**
@@ -409,16 +414,19 @@
* @hdrlen: length of familiy specific header
* @maxtype: maximum attribute type to be expected
* @policy: validation policy
+ * @extack: extended ACK report struct
*/
static inline int nlmsg_validate(const struct nlmsghdr *nlh,
int hdrlen, int maxtype,
- const struct nla_policy *policy)
+ const struct nla_policy *policy,
+ struct netlink_ext_ack *extack)
{
if (nlh->nlmsg_len < nlmsg_msg_size(hdrlen))
return -EINVAL;
return nla_validate(nlmsg_attrdata(nlh, hdrlen),
- nlmsg_attrlen(nlh, hdrlen), maxtype, policy);
+ nlmsg_attrlen(nlh, hdrlen), maxtype, policy,
+ extack);
}
/**
@@ -739,14 +747,17 @@
* @maxtype: maximum attribute type to be expected
* @nla: attribute containing the nested attributes
* @policy: validation policy
+ * @extack: extended ACK report struct
*
* See nla_parse()
*/
static inline int nla_parse_nested(struct nlattr *tb[], int maxtype,
const struct nlattr *nla,
- const struct nla_policy *policy)
+ const struct nla_policy *policy,
+ struct netlink_ext_ack *extack)
{
- return nla_parse(tb, maxtype, nla_data(nla), nla_len(nla), policy);
+ return nla_parse(tb, maxtype, nla_data(nla), nla_len(nla), policy,
+ extack);
}
/**
@@ -1252,6 +1263,7 @@
* @start: container attribute
* @maxtype: maximum attribute type to be expected
* @policy: validation policy
+ * @extack: extended ACK report struct
*
* Validates all attributes in the nested attribute stream against the
* specified policy. Attributes with a type exceeding maxtype will be
@@ -1260,9 +1272,11 @@
* Returns 0 on success or a negative error code.
*/
static inline int nla_validate_nested(const struct nlattr *start, int maxtype,
- const struct nla_policy *policy)
+ const struct nla_policy *policy,
+ struct netlink_ext_ack *extack)
{
- return nla_validate(nla_data(start), nla_len(start), maxtype, policy);
+ return nla_validate(nla_data(start), nla_len(start), maxtype, policy,
+ extack);
}
/**
diff --git a/include/net/netns/can.h b/include/net/netns/can.h
new file mode 100644
index 0000000..b106e6a
--- /dev/null
+++ b/include/net/netns/can.h
@@ -0,0 +1,40 @@
+/*
+ * can in net namespaces
+ */
+
+#ifndef __NETNS_CAN_H__
+#define __NETNS_CAN_H__
+
+#include <linux/spinlock.h>
+
+struct dev_rcv_lists;
+struct s_stats;
+struct s_pstats;
+
+struct netns_can {
+#if IS_ENABLED(CONFIG_PROC_FS)
+ struct proc_dir_entry *proc_dir;
+ struct proc_dir_entry *pde_version;
+ struct proc_dir_entry *pde_stats;
+ struct proc_dir_entry *pde_reset_stats;
+ struct proc_dir_entry *pde_rcvlist_all;
+ struct proc_dir_entry *pde_rcvlist_fil;
+ struct proc_dir_entry *pde_rcvlist_inv;
+ struct proc_dir_entry *pde_rcvlist_sff;
+ struct proc_dir_entry *pde_rcvlist_eff;
+ struct proc_dir_entry *pde_rcvlist_err;
+ struct proc_dir_entry *bcmproc_dir;
+#endif
+
+ /* receive filters subscribed for 'all' CAN devices */
+ struct dev_rcv_lists *can_rx_alldev_list;
+ spinlock_t can_rcvlists_lock;
+ struct timer_list can_stattimer;/* timer for statistics update */
+ struct s_stats *can_stats; /* packet statistics */
+ struct s_pstats *can_pstats; /* receive list statistics */
+
+ /* CAN GW per-net gateway jobs */
+ struct hlist_head cgw_list;
+};
+
+#endif /* __NETNS_CAN_H__ */
diff --git a/include/net/netns/ipv4.h b/include/net/netns/ipv4.h
index 622d2da..cd686c4 100644
--- a/include/net/netns/ipv4.h
+++ b/include/net/netns/ipv4.h
@@ -33,7 +33,6 @@
atomic_t tw_count;
struct inet_hashinfo *hashinfo ____cacheline_aligned_in_smp;
- int sysctl_tw_recycle;
int sysctl_max_tw_buckets;
};
@@ -96,6 +95,8 @@
/* Shall we try to damage output packets if routing dev changes? */
int sysctl_ip_dynaddr;
int sysctl_ip_early_demux;
+ int sysctl_tcp_early_demux;
+ int sysctl_udp_early_demux;
int sysctl_fwmark_reflect;
int sysctl_tcp_fwmark_accept;
@@ -152,6 +153,7 @@
#endif
#ifdef CONFIG_IP_ROUTE_MULTIPATH
int sysctl_fib_multipath_use_neigh;
+ int sysctl_fib_multipath_hash_policy;
#endif
unsigned int fib_seq; /* protected by rtnl_mutex */
diff --git a/include/net/netns/mpls.h b/include/net/netns/mpls.h
index d292036..6608b36 100644
--- a/include/net/netns/mpls.h
+++ b/include/net/netns/mpls.h
@@ -9,8 +9,11 @@
struct ctl_table_header;
struct netns_mpls {
+ int ip_ttl_propagate;
+ int default_ttl;
size_t platform_labels;
struct mpls_route __rcu * __rcu *platform_label;
+
struct ctl_table_header *ctl;
};
diff --git a/include/net/nfc/nfc.h b/include/net/nfc/nfc.h
index 1a3de8b..bbdc73a 100644
--- a/include/net/nfc/nfc.h
+++ b/include/net/nfc/nfc.h
@@ -27,6 +27,7 @@
#include <linux/device.h>
#include <linux/skbuff.h>
+#define nfc_dbg(dev, fmt, ...) dev_dbg((dev), "NFC: " fmt, ##__VA_ARGS__)
#define nfc_info(dev, fmt, ...) dev_info((dev), "NFC: " fmt, ##__VA_ARGS__)
#define nfc_err(dev, fmt, ...) dev_err((dev), "NFC: " fmt, ##__VA_ARGS__)
diff --git a/include/net/pkt_sched.h b/include/net/pkt_sched.h
index f1b76b8..bec46f6 100644
--- a/include/net/pkt_sched.h
+++ b/include/net/pkt_sched.h
@@ -92,7 +92,7 @@
void qdisc_get_default(char *id, size_t len);
int qdisc_set_default(const char *id);
-void qdisc_hash_add(struct Qdisc *q);
+void qdisc_hash_add(struct Qdisc *q, bool invisible);
void qdisc_hash_del(struct Qdisc *q);
struct Qdisc *qdisc_lookup(struct net_device *dev, u32 handle);
struct Qdisc *qdisc_lookup_class(struct net_device *dev, u32 handle);
diff --git a/include/net/protocol.h b/include/net/protocol.h
index bf36ca3..65ba335 100644
--- a/include/net/protocol.h
+++ b/include/net/protocol.h
@@ -40,6 +40,7 @@
/* This is used to register protocols. */
struct net_protocol {
void (*early_demux)(struct sk_buff *skb);
+ void (*early_demux_handler)(struct sk_buff *skb);
int (*handler)(struct sk_buff *skb);
void (*err_handler)(struct sk_buff *skb, u32 info);
unsigned int no_policy:1,
@@ -54,7 +55,7 @@
#if IS_ENABLED(CONFIG_IPV6)
struct inet6_protocol {
void (*early_demux)(struct sk_buff *skb);
-
+ void (*early_demux_handler)(struct sk_buff *skb);
int (*handler)(struct sk_buff *skb);
void (*err_handler)(struct sk_buff *skb,
@@ -92,12 +93,12 @@
#define INET_PROTOSW_PERMANENT 0x02 /* Permanent protocols are unremovable. */
#define INET_PROTOSW_ICSK 0x04 /* Is this an inet_connection_sock? */
-extern const struct net_protocol __rcu *inet_protos[MAX_INET_PROTOS];
+extern struct net_protocol __rcu *inet_protos[MAX_INET_PROTOS];
extern const struct net_offload __rcu *inet_offloads[MAX_INET_PROTOS];
extern const struct net_offload __rcu *inet6_offloads[MAX_INET_PROTOS];
#if IS_ENABLED(CONFIG_IPV6)
-extern const struct inet6_protocol __rcu *inet6_protos[MAX_INET_PROTOS];
+extern struct inet6_protocol __rcu *inet6_protos[MAX_INET_PROTOS];
#endif
int inet_add_protocol(const struct net_protocol *prot, unsigned char num);
diff --git a/include/net/route.h b/include/net/route.h
index c0874c8..2cc0e14 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -113,13 +113,13 @@
int ip_rt_init(void);
void rt_cache_flush(struct net *net);
void rt_flush_dev(struct net_device *dev);
-struct rtable *__ip_route_output_key_hash(struct net *, struct flowi4 *flp,
- int mp_hash);
+struct rtable *__ip_route_output_key_hash(struct net *net, struct flowi4 *flp,
+ const struct sk_buff *skb);
static inline struct rtable *__ip_route_output_key(struct net *net,
struct flowi4 *flp)
{
- return __ip_route_output_key_hash(net, flp, -1);
+ return __ip_route_output_key_hash(net, flp, NULL);
}
struct rtable *ip_route_output_flow(struct net *, struct flowi4 *flp,
diff --git a/include/net/rtnetlink.h b/include/net/rtnetlink.h
index 106de5f..78fa5fe 100644
--- a/include/net/rtnetlink.h
+++ b/include/net/rtnetlink.h
@@ -4,7 +4,8 @@
#include <linux/rtnetlink.h>
#include <net/netlink.h>
-typedef int (*rtnl_doit_func)(struct sk_buff *, struct nlmsghdr *);
+typedef int (*rtnl_doit_func)(struct sk_buff *, struct nlmsghdr *,
+ struct netlink_ext_ack *);
typedef int (*rtnl_dumpit_func)(struct sk_buff *, struct netlink_callback *);
typedef u16 (*rtnl_calcit_func)(struct sk_buff *, struct nlmsghdr *);
@@ -158,7 +159,8 @@
int rtnl_delete_link(struct net_device *dev);
int rtnl_configure_link(struct net_device *dev, const struct ifinfomsg *ifm);
-int rtnl_nla_parse_ifla(struct nlattr **tb, const struct nlattr *head, int len);
+int rtnl_nla_parse_ifla(struct nlattr **tb, const struct nlattr *head, int len,
+ struct netlink_ext_ack *exterr);
#define MODULE_ALIAS_RTNL_LINK(kind) MODULE_ALIAS("rtnl-link-" kind)
diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
index aeec408..22e5209 100644
--- a/include/net/sch_generic.h
+++ b/include/net/sch_generic.h
@@ -66,6 +66,7 @@
#define TCQ_F_NOPARENT 0x40 /* root of its hierarchy :
* qdisc_tree_decrease_qlen() should stop.
*/
+#define TCQ_F_INVISIBLE 0x80 /* invisible by default in dump */
u32 limit;
const struct Qdisc_ops *ops;
struct qdisc_size_table __rcu *stab;
@@ -203,14 +204,14 @@
const struct tcf_proto *,
struct tcf_result *);
int (*init)(struct tcf_proto*);
- bool (*destroy)(struct tcf_proto*, bool);
+ void (*destroy)(struct tcf_proto*);
unsigned long (*get)(struct tcf_proto*, u32 handle);
int (*change)(struct net *net, struct sk_buff *,
struct tcf_proto*, unsigned long,
u32 handle, struct nlattr **,
unsigned long *, bool);
- int (*delete)(struct tcf_proto*, unsigned long);
+ int (*delete)(struct tcf_proto*, unsigned long, bool*);
void (*walk)(struct tcf_proto*, struct tcf_walker *arg);
/* rtnetlink specific */
diff --git a/include/net/sctp/sm.h b/include/net/sctp/sm.h
index b6f682e..47113f2 100644
--- a/include/net/sctp/sm.h
+++ b/include/net/sctp/sm.h
@@ -293,6 +293,22 @@
struct sctp_association *asoc,
union sctp_params param,
struct sctp_ulpevent **evp);
+struct sctp_chunk *sctp_process_strreset_tsnreq(
+ struct sctp_association *asoc,
+ union sctp_params param,
+ struct sctp_ulpevent **evp);
+struct sctp_chunk *sctp_process_strreset_addstrm_out(
+ struct sctp_association *asoc,
+ union sctp_params param,
+ struct sctp_ulpevent **evp);
+struct sctp_chunk *sctp_process_strreset_addstrm_in(
+ struct sctp_association *asoc,
+ union sctp_params param,
+ struct sctp_ulpevent **evp);
+struct sctp_chunk *sctp_process_strreset_resp(
+ struct sctp_association *asoc,
+ union sctp_params param,
+ struct sctp_ulpevent **evp);
/* Prototypes for statetable processing. */
diff --git a/include/net/sctp/structs.h b/include/net/sctp/structs.h
index 138f861..a8b38e1 100644
--- a/include/net/sctp/structs.h
+++ b/include/net/sctp/structs.h
@@ -1315,6 +1315,8 @@
struct sctp_stream_out {
__u16 ssn;
__u8 state;
+ __u64 abandoned_unsent[SCTP_PR_INDEX(MAX) + 1];
+ __u64 abandoned_sent[SCTP_PR_INDEX(MAX) + 1];
};
struct sctp_stream_in {
@@ -1887,6 +1889,7 @@
__u32 strreset_outseq; /* Update after receiving response */
__u32 strreset_inseq; /* Update after receiving request */
+ __u32 strreset_result[2]; /* save the results of last 2 responses */
struct sctp_chunk *strreset_chunk; /* save request chunk */
diff --git a/include/net/sctp/ulpevent.h b/include/net/sctp/ulpevent.h
index 324b596..1060494 100644
--- a/include/net/sctp/ulpevent.h
+++ b/include/net/sctp/ulpevent.h
@@ -132,6 +132,14 @@
const struct sctp_association *asoc, __u16 flags,
__u16 stream_num, __u16 *stream_list, gfp_t gfp);
+struct sctp_ulpevent *sctp_ulpevent_make_assoc_reset_event(
+ const struct sctp_association *asoc, __u16 flags,
+ __u32 local_tsn, __u32 remote_tsn, gfp_t gfp);
+
+struct sctp_ulpevent *sctp_ulpevent_make_stream_change_event(
+ const struct sctp_association *asoc, __u16 flags,
+ __u32 strchange_instrms, __u32 strchange_outstrms, gfp_t gfp);
+
void sctp_ulpevent_read_sndrcvinfo(const struct sctp_ulpevent *event,
struct msghdr *);
void sctp_ulpevent_read_rcvinfo(const struct sctp_ulpevent *event,
diff --git a/include/net/secure_seq.h b/include/net/secure_seq.h
index 0caee63..fe236b3 100644
--- a/include/net/secure_seq.h
+++ b/include/net/secure_seq.h
@@ -6,10 +6,10 @@
u32 secure_ipv4_port_ephemeral(__be32 saddr, __be32 daddr, __be16 dport);
u32 secure_ipv6_port_ephemeral(const __be32 *saddr, const __be32 *daddr,
__be16 dport);
-u32 secure_tcp_sequence_number(__be32 saddr, __be32 daddr,
+u32 secure_tcp_seq_and_tsoff(__be32 saddr, __be32 daddr,
+ __be16 sport, __be16 dport, u32 *tsoff);
+u32 secure_tcpv6_seq_and_tsoff(const __be32 *saddr, const __be32 *daddr,
__be16 sport, __be16 dport, u32 *tsoff);
-u32 secure_tcpv6_sequence_number(const __be32 *saddr, const __be32 *daddr,
- __be16 sport, __be16 dport, u32 *tsoff);
u64 secure_dccp_sequence_number(__be32 saddr, __be32 daddr,
__be16 sport, __be16 dport);
u64 secure_dccpv6_sequence_number(__be32 *saddr, __be32 *daddr,
diff --git a/include/net/sock.h b/include/net/sock.h
index 03252d5..66349e4 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1783,11 +1783,8 @@
sk_tx_queue_clear(sk);
sk->sk_dst_pending_confirm = 0;
- /*
- * This can be called while sk is owned by the caller only,
- * with no state that can be checked in a rcu_dereference_check() cond
- */
- old_dst = rcu_dereference_raw(sk->sk_dst_cache);
+ old_dst = rcu_dereference_protected(sk->sk_dst_cache,
+ lockdep_sock_is_held(sk));
rcu_assign_pointer(sk->sk_dst_cache, dst);
dst_release(old_dst);
}
@@ -2242,6 +2239,7 @@
void __sock_recv_ts_and_drops(struct msghdr *msg, struct sock *sk,
struct sk_buff *skb);
+#define SK_DEFAULT_STAMP (-1L * NSEC_PER_SEC)
static inline void sock_recv_ts_and_drops(struct msghdr *msg, struct sock *sk,
struct sk_buff *skb)
{
@@ -2252,8 +2250,10 @@
if (sk->sk_flags & FLAGS_TS_OR_DROPS || sk->sk_tsflags & TSFLAGS_ANY)
__sock_recv_ts_and_drops(msg, sk, skb);
- else
+ else if (unlikely(sock_flag(sk, SOCK_TIMESTAMP)))
sk->sk_stamp = skb->tstamp;
+ else if (unlikely(sk->sk_stamp == SK_DEFAULT_STAMP))
+ sk->sk_stamp = 0;
}
void __sock_tx_timestamp(__u16 tsflags, __u8 *tx_flags);
@@ -2365,6 +2365,8 @@
bool sk_capable(const struct sock *sk, int cap);
bool sk_net_capable(const struct sock *sk, int cap);
+void sk_get_meminfo(const struct sock *sk, u32 *meminfo);
+
extern __u32 sysctl_wmem_max;
extern __u32 sysctl_rmem_max;
diff --git a/include/net/tc_act/tc_pedit.h b/include/net/tc_act/tc_pedit.h
index dfbd6ee..a46c3f2 100644
--- a/include/net/tc_act/tc_pedit.h
+++ b/include/net/tc_act/tc_pedit.h
@@ -2,6 +2,7 @@
#define __NET_TC_PED_H
#include <net/act_api.h>
+#include <linux/tc_act/tc_pedit.h>
struct tcf_pedit_key_ex {
enum pedit_header_type htype;
@@ -17,4 +18,48 @@
};
#define to_pedit(a) ((struct tcf_pedit *)a)
+static inline bool is_tcf_pedit(const struct tc_action *a)
+{
+#ifdef CONFIG_NET_CLS_ACT
+ if (a->ops && a->ops->type == TCA_ACT_PEDIT)
+ return true;
+#endif
+ return false;
+}
+
+static inline int tcf_pedit_nkeys(const struct tc_action *a)
+{
+ return to_pedit(a)->tcfp_nkeys;
+}
+
+static inline u32 tcf_pedit_htype(const struct tc_action *a, int index)
+{
+ if (to_pedit(a)->tcfp_keys_ex)
+ return to_pedit(a)->tcfp_keys_ex[index].htype;
+
+ return TCA_PEDIT_KEY_EX_HDR_TYPE_NETWORK;
+}
+
+static inline u32 tcf_pedit_cmd(const struct tc_action *a, int index)
+{
+ if (to_pedit(a)->tcfp_keys_ex)
+ return to_pedit(a)->tcfp_keys_ex[index].cmd;
+
+ return __PEDIT_CMD_MAX;
+}
+
+static inline u32 tcf_pedit_mask(const struct tc_action *a, int index)
+{
+ return to_pedit(a)->tcfp_keys[index].mask;
+}
+
+static inline u32 tcf_pedit_val(const struct tc_action *a, int index)
+{
+ return to_pedit(a)->tcfp_keys[index].val;
+}
+
+static inline u32 tcf_pedit_offset(const struct tc_action *a, int index)
+{
+ return to_pedit(a)->tcfp_keys[index].off;
+}
#endif /* __NET_TC_PED_H */
diff --git a/include/net/tc_act/tc_vlan.h b/include/net/tc_act/tc_vlan.h
index 48cca32..c2090df 100644
--- a/include/net/tc_act/tc_vlan.h
+++ b/include/net/tc_act/tc_vlan.h
@@ -13,9 +13,6 @@
#include <net/act_api.h>
#include <linux/tc_act/tc_vlan.h>
-#define VLAN_F_POP 0x1
-#define VLAN_F_PUSH 0x2
-
struct tcf_vlan {
struct tc_action common;
int tcfv_action;
@@ -49,4 +46,9 @@
return to_vlan(a)->tcfv_push_proto;
}
+static inline u8 tcf_vlan_push_prio(const struct tc_action *a)
+{
+ return to_vlan(a)->tcfv_push_prio;
+}
+
#endif /* __NET_TC_VLAN_H */
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 6ec4ea6..270e5cc 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -78,6 +78,9 @@
/* Maximal number of ACKs sent quickly to accelerate slow-start. */
#define TCP_MAX_QUICKACKS 16U
+/* Maximal number of window scale according to RFC1323 */
+#define TCP_MAX_WSCALE 14U
+
/* urg_data states */
#define TCP_URG_VALID 0x0100
#define TCP_URG_NOTYET 0x0200
@@ -406,11 +409,7 @@
void tcp_update_metrics(struct sock *sk);
void tcp_init_metrics(struct sock *sk);
void tcp_metrics_init(void);
-bool tcp_peer_is_proven(struct request_sock *req, struct dst_entry *dst,
- bool paws_check, bool timestamps);
-bool tcp_remember_stamp(struct sock *sk);
-bool tcp_tw_remember_stamp(struct inet_timewait_sock *tw);
-void tcp_fetch_timewait_stamp(struct sock *sk, struct dst_entry *dst);
+bool tcp_peer_is_proven(struct request_sock *req, struct dst_entry *dst);
void tcp_disable_fack(struct tcp_sock *tp);
void tcp_close(struct sock *sk, long timeout);
void tcp_init_sock(struct sock *sk);
@@ -1005,7 +1004,7 @@
void tcp_rate_skb_delivered(struct sock *sk, struct sk_buff *skb,
struct rate_sample *rs);
void tcp_rate_gen(struct sock *sk, u32 delivered, u32 lost,
- struct skb_mstamp *now, struct rate_sample *rs);
+ struct rate_sample *rs);
void tcp_rate_check_app_limited(struct sock *sk);
/* These functions determine how the current flow behaves in respect of SACK
@@ -1252,9 +1251,11 @@
static inline int tcp_win_from_space(int space)
{
- return sysctl_tcp_adv_win_scale<=0 ?
- (space>>(-sysctl_tcp_adv_win_scale)) :
- space - (space>>sysctl_tcp_adv_win_scale);
+ int tcp_adv_win_scale = sysctl_tcp_adv_win_scale;
+
+ return tcp_adv_win_scale <= 0 ?
+ (space>>(-tcp_adv_win_scale)) :
+ space - (space>>tcp_adv_win_scale);
}
/* Note: caller must be prepared to deal with negative returns */
@@ -1505,6 +1506,12 @@
struct rcu_head rcu;
};
+extern unsigned int sysctl_tcp_fastopen_blackhole_timeout;
+void tcp_fastopen_active_disable(struct sock *sk);
+bool tcp_fastopen_active_should_disable(struct sock *sk);
+void tcp_fastopen_active_disable_ofo_check(struct sock *sk);
+void tcp_fastopen_active_timeout_reset(void);
+
/* Latencies incurred by various limits for a sender. They are
* chronograph-like stats that are mutually exclusive.
*/
@@ -1814,9 +1821,8 @@
__u16 *mss);
#endif
struct dst_entry *(*route_req)(const struct sock *sk, struct flowi *fl,
- const struct request_sock *req,
- bool *strict);
- __u32 (*init_seq)(const struct sk_buff *skb, u32 *tsoff);
+ const struct request_sock *req);
+ __u32 (*init_seq_tsoff)(const struct sk_buff *skb, u32 *tsoff);
int (*send_synack)(const struct sock *sk, struct dst_entry *dst,
struct flowi *fl, struct request_sock *req,
struct tcp_fastopen_cookie *foc,
@@ -1847,10 +1853,9 @@
void tcp_init(void);
/* tcp_recovery.c */
-extern void tcp_rack_mark_lost(struct sock *sk, const struct skb_mstamp *now);
+extern void tcp_rack_mark_lost(struct sock *sk);
extern void tcp_rack_advance(struct tcp_sock *tp, u8 sacked, u32 end_seq,
- const struct skb_mstamp *xmit_time,
- const struct skb_mstamp *ack_time);
+ const struct skb_mstamp *xmit_time);
extern void tcp_rack_reo_timeout(struct sock *sk);
/*
diff --git a/include/net/udp.h b/include/net/udp.h
index c9d8b8e..3391dbd 100644
--- a/include/net/udp.h
+++ b/include/net/udp.h
@@ -372,4 +372,5 @@
#if IS_ENABLED(CONFIG_IPV6)
void udpv6_encap_enable(void);
#endif
+
#endif /* _UDP_H */
diff --git a/include/net/xfrm.h b/include/net/xfrm.h
index 14d82bf..6793a30c 100644
--- a/include/net/xfrm.h
+++ b/include/net/xfrm.h
@@ -120,6 +120,13 @@
struct xfrm_address_filter *filter;
};
+struct xfrm_state_offload {
+ struct net_device *dev;
+ unsigned long offload_handle;
+ unsigned int num_exthdrs;
+ u8 flags;
+};
+
/* Full description of state of transformer. */
struct xfrm_state {
possible_net_t xs_net;
@@ -207,6 +214,8 @@
struct xfrm_lifetime_cur curlft;
struct tasklet_hrtimer mtimer;
+ struct xfrm_state_offload xso;
+
/* used to fix curlft->add_time when changing date */
long saved_tmo;
@@ -222,6 +231,8 @@
struct xfrm_mode *inner_mode_iaf;
struct xfrm_mode *outer_mode;
+ const struct xfrm_type_offload *type_offload;
+
/* Security context */
struct xfrm_sec_ctx *security;
@@ -314,12 +325,14 @@
int __xfrm_state_delete(struct xfrm_state *x);
struct xfrm_state_afinfo {
- unsigned int family;
- unsigned int proto;
- __be16 eth_proto;
- struct module *owner;
- const struct xfrm_type *type_map[IPPROTO_MAX];
- struct xfrm_mode *mode_map[XFRM_MODE_MAX];
+ unsigned int family;
+ unsigned int proto;
+ __be16 eth_proto;
+ struct module *owner;
+ const struct xfrm_type *type_map[IPPROTO_MAX];
+ const struct xfrm_type_offload *type_offload_map[IPPROTO_MAX];
+ struct xfrm_mode *mode_map[XFRM_MODE_MAX];
+
int (*init_flags)(struct xfrm_state *x);
void (*init_tempsel)(struct xfrm_selector *sel,
const struct flowi *fl);
@@ -380,6 +393,18 @@
int xfrm_register_type(const struct xfrm_type *type, unsigned short family);
int xfrm_unregister_type(const struct xfrm_type *type, unsigned short family);
+struct xfrm_type_offload {
+ char *description;
+ struct module *owner;
+ u8 proto;
+ void (*encap)(struct xfrm_state *, struct sk_buff *pskb);
+ int (*input_tail)(struct xfrm_state *x, struct sk_buff *skb);
+ int (*xmit)(struct xfrm_state *, struct sk_buff *pskb, netdev_features_t features);
+};
+
+int xfrm_register_type_offload(const struct xfrm_type_offload *type, unsigned short family);
+int xfrm_unregister_type_offload(const struct xfrm_type_offload *type, unsigned short family);
+
struct xfrm_mode {
/*
* Remove encapsulation header.
@@ -428,6 +453,16 @@
*/
int (*output)(struct xfrm_state *x, struct sk_buff *skb);
+ /*
+ * Adjust pointers into the packet and do GSO segmentation.
+ */
+ struct sk_buff *(*gso_segment)(struct xfrm_state *x, struct sk_buff *skb, netdev_features_t features);
+
+ /*
+ * Adjust pointers into the packet when IPsec is done at layer2.
+ */
+ void (*xmit)(struct xfrm_state *x, struct sk_buff *skb);
+
struct xfrm_state_afinfo *afinfo;
struct module *owner;
unsigned int encap;
@@ -586,7 +621,6 @@
struct xfrm_mgr {
struct list_head list;
- char *id;
int (*notify)(struct xfrm_state *x, const struct km_event *c);
int (*acquire)(struct xfrm_state *x, struct xfrm_tmpl *, struct xfrm_policy *xp);
struct xfrm_policy *(*compile_policy)(struct sock *sk, int opt, u8 *data, int len, int *dir);
@@ -817,12 +851,12 @@
}
static inline bool addr_match(const void *token1, const void *token2,
- int prefixlen)
+ unsigned int prefixlen)
{
const __be32 *a1 = token1;
const __be32 *a2 = token2;
- int pdw;
- int pbi;
+ unsigned int pdw;
+ unsigned int pbi;
pdw = prefixlen >> 5; /* num of whole u32 in prefix */
pbi = prefixlen & 0x1f; /* num of bits in incomplete u32 in prefix */
@@ -846,9 +880,9 @@
static inline bool addr4_match(__be32 a1, __be32 a2, u8 prefixlen)
{
/* C99 6.5.7 (3): u32 << 32 is undefined behaviour */
- if (prefixlen == 0)
+ if (sizeof(long) == 4 && prefixlen == 0)
return true;
- return !((a1 ^ a2) & htonl(0xFFFFFFFFu << (32 - prefixlen)));
+ return !((a1 ^ a2) & htonl(~0UL << (32 - prefixlen)));
}
static __inline__
@@ -1533,6 +1567,7 @@
struct xfrm_state *xfrm_find_acq_byseq(struct net *net, u32 mark, u32 seq);
int xfrm_state_delete(struct xfrm_state *x);
int xfrm_state_flush(struct net *net, u8 proto, bool task_valid);
+int xfrm_dev_state_flush(struct net *net, struct net_device *dev, bool task_valid);
void xfrm_sad_getinfo(struct net *net, struct xfrmk_sadinfo *si);
void xfrm_spd_getinfo(struct net *net, struct xfrmk_spdinfo *si);
u32 xfrm_replay_seqhi(struct xfrm_state *x, __be32 net_seq);
@@ -1615,6 +1650,11 @@
}
#endif
+struct dst_entry *__xfrm_dst_lookup(struct net *net, int tos, int oif,
+ const xfrm_address_t *saddr,
+ const xfrm_address_t *daddr,
+ int family);
+
struct xfrm_policy *xfrm_policy_alloc(struct net *net, gfp_t gfp);
void xfrm_policy_walk_init(struct xfrm_policy_walk *walk, u8 type);
@@ -1820,6 +1860,61 @@
}
#endif
+#ifdef CONFIG_XFRM_OFFLOAD
+void __net_init xfrm_dev_init(void);
+int validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t features);
+int xfrm_dev_state_add(struct net *net, struct xfrm_state *x,
+ struct xfrm_user_offload *xuo);
+bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x);
+
+static inline void xfrm_dev_state_delete(struct xfrm_state *x)
+{
+ struct xfrm_state_offload *xso = &x->xso;
+
+ if (xso->dev)
+ xso->dev->xfrmdev_ops->xdo_dev_state_delete(x);
+}
+
+static inline void xfrm_dev_state_free(struct xfrm_state *x)
+{
+ struct xfrm_state_offload *xso = &x->xso;
+ struct net_device *dev = xso->dev;
+
+ if (dev && dev->xfrmdev_ops) {
+ dev->xfrmdev_ops->xdo_dev_state_free(x);
+ xso->dev = NULL;
+ dev_put(dev);
+ }
+}
+#else
+static inline void __net_init xfrm_dev_init(void)
+{
+}
+
+static inline int validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t features)
+{
+ return 0;
+}
+
+static inline int xfrm_dev_state_add(struct net *net, struct xfrm_state *x, struct xfrm_user_offload *xuo)
+{
+ return 0;
+}
+
+static inline void xfrm_dev_state_delete(struct xfrm_state *x)
+{
+}
+
+static inline void xfrm_dev_state_free(struct xfrm_state *x)
+{
+}
+
+static inline bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x)
+{
+ return false;
+}
+#endif
+
static inline int xfrm_mark_get(struct nlattr **attrs, struct xfrm_mark *m)
{
if (attrs[XFRMA_MARK])
diff --git a/include/rdma/ib.h b/include/rdma/ib.h
index 9b4c22a3..66dbed0 100644
--- a/include/rdma/ib.h
+++ b/include/rdma/ib.h
@@ -100,7 +100,7 @@
*/
static inline bool ib_safe_file_access(struct file *filp)
{
- return filp->f_cred == current_cred() && segment_eq(get_fs(), USER_DS);
+ return filp->f_cred == current_cred() && !uaccess_kernel();
}
#endif /* _RDMA_IB_H */
diff --git a/include/scsi/scsi_request.h b/include/scsi/scsi_request.h
index ba0aeb9..f0c76f9 100644
--- a/include/scsi/scsi_request.h
+++ b/include/scsi/scsi_request.h
@@ -9,8 +9,10 @@
unsigned char __cmd[BLK_MAX_CDB];
unsigned char *cmd;
unsigned short cmd_len;
+ int result;
unsigned int sense_len;
unsigned int resid_len; /* residual count */
+ int retries;
void *sense;
};
diff --git a/include/soc/fsl/qman.h b/include/soc/fsl/qman.h
index 3d4df74..d4dfefd 100644
--- a/include/soc/fsl/qman.h
+++ b/include/soc/fsl/qman.h
@@ -36,8 +36,11 @@
/* Hardware constants */
#define QM_CHANNEL_SWPORTAL0 0
#define QMAN_CHANNEL_POOL1 0x21
+#define QMAN_CHANNEL_CAAM 0x80
#define QMAN_CHANNEL_POOL1_REV3 0x401
+#define QMAN_CHANNEL_CAAM_REV3 0x840
extern u16 qm_channel_pool1;
+extern u16 qm_channel_caam;
/* Portal processing (interrupt) sources */
#define QM_PIRQ_CSCI 0x00100000 /* Congestion State Change */
@@ -165,6 +168,7 @@
#define qm_fd_set_contig_big(fd, len) \
qm_fd_set_param(fd, qm_fd_contig_big, 0, len)
#define qm_fd_set_sg_big(fd, len) qm_fd_set_param(fd, qm_fd_sg_big, 0, len)
+#define qm_fd_set_compound(fd, len) qm_fd_set_param(fd, qm_fd_compound, 0, len)
static inline void qm_fd_clear_fd(struct qm_fd *fd)
{
@@ -639,6 +643,7 @@
#define QM_CGR_WE_MODE 0x0001
#define QMAN_CGR_FLAG_USE_INIT 0x00000001
+#define QMAN_CGR_MODE_FRAME 0x00000001
/* Portal and Frame Queues */
/* Represents a managed portal */
@@ -791,6 +796,84 @@
#define QMAN_INITFQ_FLAG_SCHED 0x00000001 /* schedule rather than park */
#define QMAN_INITFQ_FLAG_LOCAL 0x00000004 /* set dest portal */
+/*
+ * For qman_volatile_dequeue(); Choose one PRECEDENCE. EXACT is optional. Use
+ * NUMFRAMES(n) (6-bit) or NUMFRAMES_TILLEMPTY to fill in the frame-count. Use
+ * FQID(n) to fill in the frame queue ID.
+ */
+#define QM_VDQCR_PRECEDENCE_VDQCR 0x0
+#define QM_VDQCR_PRECEDENCE_SDQCR 0x80000000
+#define QM_VDQCR_EXACT 0x40000000
+#define QM_VDQCR_NUMFRAMES_MASK 0x3f000000
+#define QM_VDQCR_NUMFRAMES_SET(n) (((n) & 0x3f) << 24)
+#define QM_VDQCR_NUMFRAMES_GET(n) (((n) >> 24) & 0x3f)
+#define QM_VDQCR_NUMFRAMES_TILLEMPTY QM_VDQCR_NUMFRAMES_SET(0)
+
+#define QMAN_VOLATILE_FLAG_WAIT 0x00000001 /* wait if VDQCR is in use */
+#define QMAN_VOLATILE_FLAG_WAIT_INT 0x00000002 /* if wait, interruptible? */
+#define QMAN_VOLATILE_FLAG_FINISH 0x00000004 /* wait till VDQCR completes */
+
+/* "Query FQ Non-Programmable Fields" */
+struct qm_mcr_queryfq_np {
+ u8 verb;
+ u8 result;
+ u8 __reserved1;
+ u8 state; /* QM_MCR_NP_STATE_*** */
+ u32 fqd_link; /* 24-bit, _res2[24-31] */
+ u16 odp_seq; /* 14-bit, _res3[14-15] */
+ u16 orp_nesn; /* 14-bit, _res4[14-15] */
+ u16 orp_ea_hseq; /* 15-bit, _res5[15] */
+ u16 orp_ea_tseq; /* 15-bit, _res6[15] */
+ u32 orp_ea_hptr; /* 24-bit, _res7[24-31] */
+ u32 orp_ea_tptr; /* 24-bit, _res8[24-31] */
+ u32 pfdr_hptr; /* 24-bit, _res9[24-31] */
+ u32 pfdr_tptr; /* 24-bit, _res10[24-31] */
+ u8 __reserved2[5];
+ u8 is; /* 1-bit, _res12[1-7] */
+ u16 ics_surp;
+ u32 byte_cnt;
+ u32 frm_cnt; /* 24-bit, _res13[24-31] */
+ u32 __reserved3;
+ u16 ra1_sfdr; /* QM_MCR_NP_RA1_*** */
+ u16 ra2_sfdr; /* QM_MCR_NP_RA2_*** */
+ u16 __reserved4;
+ u16 od1_sfdr; /* QM_MCR_NP_OD1_*** */
+ u16 od2_sfdr; /* QM_MCR_NP_OD2_*** */
+ u16 od3_sfdr; /* QM_MCR_NP_OD3_*** */
+} __packed;
+
+#define QM_MCR_NP_STATE_FE 0x10
+#define QM_MCR_NP_STATE_R 0x08
+#define QM_MCR_NP_STATE_MASK 0x07 /* Reads FQD::STATE; */
+#define QM_MCR_NP_STATE_OOS 0x00
+#define QM_MCR_NP_STATE_RETIRED 0x01
+#define QM_MCR_NP_STATE_TEN_SCHED 0x02
+#define QM_MCR_NP_STATE_TRU_SCHED 0x03
+#define QM_MCR_NP_STATE_PARKED 0x04
+#define QM_MCR_NP_STATE_ACTIVE 0x05
+#define QM_MCR_NP_PTR_MASK 0x07ff /* for RA[12] & OD[123] */
+#define QM_MCR_NP_RA1_NRA(v) (((v) >> 14) & 0x3) /* FQD::NRA */
+#define QM_MCR_NP_RA2_IT(v) (((v) >> 14) & 0x1) /* FQD::IT */
+#define QM_MCR_NP_OD1_NOD(v) (((v) >> 14) & 0x3) /* FQD::NOD */
+#define QM_MCR_NP_OD3_NPC(v) (((v) >> 14) & 0x3) /* FQD::NPC */
+
+enum qm_mcr_queryfq_np_masks {
+ qm_mcr_fqd_link_mask = BIT(24) - 1,
+ qm_mcr_odp_seq_mask = BIT(14) - 1,
+ qm_mcr_orp_nesn_mask = BIT(14) - 1,
+ qm_mcr_orp_ea_hseq_mask = BIT(15) - 1,
+ qm_mcr_orp_ea_tseq_mask = BIT(15) - 1,
+ qm_mcr_orp_ea_hptr_mask = BIT(24) - 1,
+ qm_mcr_orp_ea_tptr_mask = BIT(24) - 1,
+ qm_mcr_pfdr_hptr_mask = BIT(24) - 1,
+ qm_mcr_pfdr_tptr_mask = BIT(24) - 1,
+ qm_mcr_is_mask = BIT(1) - 1,
+ qm_mcr_frm_cnt_mask = BIT(24) - 1,
+};
+
+#define qm_mcr_np_get(np, field) \
+ ((np)->field & (qm_mcr_##field##_mask))
+
/* Portal Management */
/**
* qman_p_irqsource_add - add processing sources to be interrupt-driven
@@ -963,6 +1046,25 @@
*/
int qman_oos_fq(struct qman_fq *fq);
+/*
+ * qman_volatile_dequeue - Issue a volatile dequeue command
+ * @fq: the frame queue object to dequeue from
+ * @flags: a bit-mask of QMAN_VOLATILE_FLAG_*** options
+ * @vdqcr: bit mask of QM_VDQCR_*** options, as per qm_dqrr_vdqcr_set()
+ *
+ * Attempts to lock access to the portal's VDQCR volatile dequeue functionality.
+ * The function will block and sleep if QMAN_VOLATILE_FLAG_WAIT is specified and
+ * the VDQCR is already in use, otherwise returns non-zero for failure. If
+ * QMAN_VOLATILE_FLAG_FINISH is specified, the function will only return once
+ * the VDQCR command has finished executing (ie. once the callback for the last
+ * DQRR entry resulting from the VDQCR command has been called). If not using
+ * the FINISH flag, completion can be determined either by detecting the
+ * presence of the QM_DQRR_STAT_UNSCHEDULED and QM_DQRR_STAT_DQCR_EXPIRED bits
+ * in the "stat" parameter passed to the FQ's dequeue callback, or by waiting
+ * for the QMAN_FQ_STATE_VDQCR bit to disappear.
+ */
+int qman_volatile_dequeue(struct qman_fq *fq, u32 flags, u32 vdqcr);
+
/**
* qman_enqueue - Enqueue a frame to a frame queue
* @fq: the frame queue object to enqueue to
@@ -994,6 +1096,13 @@
*/
int qman_release_fqid(u32 fqid);
+/**
+ * qman_query_fq_np - Queries non-programmable FQD fields
+ * @fq: the frame queue object to be queried
+ * @np: storage for the queried FQD fields
+ */
+int qman_query_fq_np(struct qman_fq *fq, struct qm_mcr_queryfq_np *np);
+
/* Pool-channel management */
/**
* qman_alloc_pool_range - Allocate a contiguous range of pool-channel IDs
diff --git a/include/trace/events/block.h b/include/trace/events/block.h
index a88ed13..d0dbe60 100644
--- a/include/trace/events/block.h
+++ b/include/trace/events/block.h
@@ -61,55 +61,6 @@
TP_ARGS(bh)
);
-DECLARE_EVENT_CLASS(block_rq_with_error,
-
- TP_PROTO(struct request_queue *q, struct request *rq),
-
- TP_ARGS(q, rq),
-
- TP_STRUCT__entry(
- __field( dev_t, dev )
- __field( sector_t, sector )
- __field( unsigned int, nr_sector )
- __field( int, errors )
- __array( char, rwbs, RWBS_LEN )
- __dynamic_array( char, cmd, 1 )
- ),
-
- TP_fast_assign(
- __entry->dev = rq->rq_disk ? disk_devt(rq->rq_disk) : 0;
- __entry->sector = blk_rq_trace_sector(rq);
- __entry->nr_sector = blk_rq_trace_nr_sectors(rq);
- __entry->errors = rq->errors;
-
- blk_fill_rwbs(__entry->rwbs, rq->cmd_flags, blk_rq_bytes(rq));
- __get_str(cmd)[0] = '\0';
- ),
-
- TP_printk("%d,%d %s (%s) %llu + %u [%d]",
- MAJOR(__entry->dev), MINOR(__entry->dev),
- __entry->rwbs, __get_str(cmd),
- (unsigned long long)__entry->sector,
- __entry->nr_sector, __entry->errors)
-);
-
-/**
- * block_rq_abort - abort block operation request
- * @q: queue containing the block operation request
- * @rq: block IO operation request
- *
- * Called immediately after pending block IO operation request @rq in
- * queue @q is aborted. The fields in the operation request @rq
- * can be examined to determine which device and sectors the pending
- * operation would access.
- */
-DEFINE_EVENT(block_rq_with_error, block_rq_abort,
-
- TP_PROTO(struct request_queue *q, struct request *rq),
-
- TP_ARGS(q, rq)
-);
-
/**
* block_rq_requeue - place block IO request back on a queue
* @q: queue holding operation
@@ -119,17 +70,40 @@
* @q. For some reason the request was not completed and needs to be
* put back in the queue.
*/
-DEFINE_EVENT(block_rq_with_error, block_rq_requeue,
+TRACE_EVENT(block_rq_requeue,
TP_PROTO(struct request_queue *q, struct request *rq),
- TP_ARGS(q, rq)
+ TP_ARGS(q, rq),
+
+ TP_STRUCT__entry(
+ __field( dev_t, dev )
+ __field( sector_t, sector )
+ __field( unsigned int, nr_sector )
+ __array( char, rwbs, RWBS_LEN )
+ __dynamic_array( char, cmd, 1 )
+ ),
+
+ TP_fast_assign(
+ __entry->dev = rq->rq_disk ? disk_devt(rq->rq_disk) : 0;
+ __entry->sector = blk_rq_trace_sector(rq);
+ __entry->nr_sector = blk_rq_trace_nr_sectors(rq);
+
+ blk_fill_rwbs(__entry->rwbs, rq->cmd_flags, blk_rq_bytes(rq));
+ __get_str(cmd)[0] = '\0';
+ ),
+
+ TP_printk("%d,%d %s (%s) %llu + %u [%d]",
+ MAJOR(__entry->dev), MINOR(__entry->dev),
+ __entry->rwbs, __get_str(cmd),
+ (unsigned long long)__entry->sector,
+ __entry->nr_sector, 0)
);
/**
* block_rq_complete - block IO operation completed by device driver
- * @q: queue containing the block operation request
* @rq: block operations request
+ * @error: status code
* @nr_bytes: number of completed bytes
*
* The block_rq_complete tracepoint event indicates that some portion
@@ -140,16 +114,15 @@
*/
TRACE_EVENT(block_rq_complete,
- TP_PROTO(struct request_queue *q, struct request *rq,
- unsigned int nr_bytes),
+ TP_PROTO(struct request *rq, int error, unsigned int nr_bytes),
- TP_ARGS(q, rq, nr_bytes),
+ TP_ARGS(rq, error, nr_bytes),
TP_STRUCT__entry(
__field( dev_t, dev )
__field( sector_t, sector )
__field( unsigned int, nr_sector )
- __field( int, errors )
+ __field( int, error )
__array( char, rwbs, RWBS_LEN )
__dynamic_array( char, cmd, 1 )
),
@@ -158,7 +131,7 @@
__entry->dev = rq->rq_disk ? disk_devt(rq->rq_disk) : 0;
__entry->sector = blk_rq_pos(rq);
__entry->nr_sector = nr_bytes >> 9;
- __entry->errors = rq->errors;
+ __entry->error = error;
blk_fill_rwbs(__entry->rwbs, rq->cmd_flags, nr_bytes);
__get_str(cmd)[0] = '\0';
@@ -168,7 +141,7 @@
MAJOR(__entry->dev), MINOR(__entry->dev),
__entry->rwbs, __get_str(cmd),
(unsigned long long)__entry->sector,
- __entry->nr_sector, __entry->errors)
+ __entry->nr_sector, __entry->error)
);
DECLARE_EVENT_CLASS(block_rq,
diff --git a/include/trace/events/bpf.h b/include/trace/events/bpf.h
index c3a53fd..52c8425 100644
--- a/include/trace/events/bpf.h
+++ b/include/trace/events/bpf.h
@@ -321,11 +321,14 @@
__dynamic_array(u8, key, map->key_size)
__dynamic_array(u8, nxt, map->key_size)
__field(bool, key_trunc)
+ __field(bool, key_null)
__field(int, ufd)
),
TP_fast_assign(
- memcpy(__get_dynamic_array(key), key, map->key_size);
+ if (key)
+ memcpy(__get_dynamic_array(key), key, map->key_size);
+ __entry->key_null = !key;
memcpy(__get_dynamic_array(nxt), key_next, map->key_size);
__entry->type = map->map_type;
__entry->key_len = min(map->key_size, 16U);
@@ -336,8 +339,9 @@
TP_printk("map type=%s ufd=%d key=[%s%s] next=[%s%s]",
__print_symbolic(__entry->type, __MAP_TYPE_SYM_TAB),
__entry->ufd,
- __print_hex(__get_dynamic_array(key), __entry->key_len),
- __entry->key_trunc ? " ..." : "",
+ __entry->key_null ? "NULL" : __print_hex(__get_dynamic_array(key),
+ __entry->key_len),
+ __entry->key_trunc && !__entry->key_null ? " ..." : "",
__print_hex(__get_dynamic_array(nxt), __entry->key_len),
__entry->key_trunc ? " ..." : "")
);
diff --git a/include/trace/events/rxrpc.h b/include/trace/events/rxrpc.h
index 39123c0..29a3d53 100644
--- a/include/trace/events/rxrpc.h
+++ b/include/trace/events/rxrpc.h
@@ -683,6 +683,57 @@
__entry->n_acks)
);
+TRACE_EVENT(rxrpc_rx_abort,
+ TP_PROTO(struct rxrpc_call *call, rxrpc_serial_t serial,
+ u32 abort_code),
+
+ TP_ARGS(call, serial, abort_code),
+
+ TP_STRUCT__entry(
+ __field(struct rxrpc_call *, call )
+ __field(rxrpc_serial_t, serial )
+ __field(u32, abort_code )
+ ),
+
+ TP_fast_assign(
+ __entry->call = call;
+ __entry->serial = serial;
+ __entry->abort_code = abort_code;
+ ),
+
+ TP_printk("c=%p ABORT %08x ac=%d",
+ __entry->call,
+ __entry->serial,
+ __entry->abort_code)
+ );
+
+TRACE_EVENT(rxrpc_rx_rwind_change,
+ TP_PROTO(struct rxrpc_call *call, rxrpc_serial_t serial,
+ u32 rwind, bool wake),
+
+ TP_ARGS(call, serial, rwind, wake),
+
+ TP_STRUCT__entry(
+ __field(struct rxrpc_call *, call )
+ __field(rxrpc_serial_t, serial )
+ __field(u32, rwind )
+ __field(bool, wake )
+ ),
+
+ TP_fast_assign(
+ __entry->call = call;
+ __entry->serial = serial;
+ __entry->rwind = rwind;
+ __entry->wake = wake;
+ ),
+
+ TP_printk("c=%p %08x rw=%u%s",
+ __entry->call,
+ __entry->serial,
+ __entry->rwind,
+ __entry->wake ? " wake" : "")
+ );
+
TRACE_EVENT(rxrpc_tx_data,
TP_PROTO(struct rxrpc_call *call, rxrpc_seq_t seq,
rxrpc_serial_t serial, u8 flags, bool retrans, bool lose),
@@ -1087,6 +1138,56 @@
__entry->abort_code)
);
+TRACE_EVENT(rxrpc_rx_eproto,
+ TP_PROTO(struct rxrpc_call *call, rxrpc_serial_t serial,
+ const char *why),
+
+ TP_ARGS(call, serial, why),
+
+ TP_STRUCT__entry(
+ __field(struct rxrpc_call *, call )
+ __field(rxrpc_serial_t, serial )
+ __field(const char *, why )
+ ),
+
+ TP_fast_assign(
+ __entry->call = call;
+ __entry->serial = serial;
+ __entry->why = why;
+ ),
+
+ TP_printk("c=%p EPROTO %08x %s",
+ __entry->call,
+ __entry->serial,
+ __entry->why)
+ );
+
+TRACE_EVENT(rxrpc_connect_call,
+ TP_PROTO(struct rxrpc_call *call),
+
+ TP_ARGS(call),
+
+ TP_STRUCT__entry(
+ __field(struct rxrpc_call *, call )
+ __field(unsigned long, user_call_ID )
+ __field(u32, cid )
+ __field(u32, call_id )
+ ),
+
+ TP_fast_assign(
+ __entry->call = call;
+ __entry->user_call_ID = call->user_call_ID;
+ __entry->cid = call->cid;
+ __entry->call_id = call->call_id;
+ ),
+
+ TP_printk("c=%p u=%p %08x:%08x",
+ __entry->call,
+ (void *)__entry->user_call_ID,
+ __entry->cid,
+ __entry->call_id)
+ );
+
#endif /* _TRACE_RXRPC_H */
/* This part must be outside protection */
diff --git a/include/trace/events/sched.h b/include/trace/events/sched.h
index 9e3ef6c..ae1409f 100644
--- a/include/trace/events/sched.h
+++ b/include/trace/events/sched.h
@@ -70,7 +70,7 @@
TP_fast_assign(
memcpy(__entry->comm, p->comm, TASK_COMM_LEN);
__entry->pid = p->pid;
- __entry->prio = p->prio;
+ __entry->prio = p->prio; /* XXX SCHED_DEADLINE */
__entry->success = 1; /* rudiment, kill when possible */
__entry->target_cpu = task_cpu(p);
),
@@ -147,6 +147,7 @@
memcpy(__entry->prev_comm, prev->comm, TASK_COMM_LEN);
__entry->next_pid = next->pid;
__entry->next_prio = next->prio;
+ /* XXX SCHED_DEADLINE */
),
TP_printk("prev_comm=%s prev_pid=%d prev_prio=%d prev_state=%s%s ==> next_comm=%s next_pid=%d next_prio=%d",
@@ -181,7 +182,7 @@
TP_fast_assign(
memcpy(__entry->comm, p->comm, TASK_COMM_LEN);
__entry->pid = p->pid;
- __entry->prio = p->prio;
+ __entry->prio = p->prio; /* XXX SCHED_DEADLINE */
__entry->orig_cpu = task_cpu(p);
__entry->dest_cpu = dest_cpu;
),
@@ -206,7 +207,7 @@
TP_fast_assign(
memcpy(__entry->comm, p->comm, TASK_COMM_LEN);
__entry->pid = p->pid;
- __entry->prio = p->prio;
+ __entry->prio = p->prio; /* XXX SCHED_DEADLINE */
),
TP_printk("comm=%s pid=%d prio=%d",
@@ -253,7 +254,7 @@
TP_fast_assign(
memcpy(__entry->comm, current->comm, TASK_COMM_LEN);
__entry->pid = pid_nr(pid);
- __entry->prio = current->prio;
+ __entry->prio = current->prio; /* XXX SCHED_DEADLINE */
),
TP_printk("comm=%s pid=%d prio=%d",
@@ -413,9 +414,9 @@
*/
TRACE_EVENT(sched_pi_setprio,
- TP_PROTO(struct task_struct *tsk, int newprio),
+ TP_PROTO(struct task_struct *tsk, struct task_struct *pi_task),
- TP_ARGS(tsk, newprio),
+ TP_ARGS(tsk, pi_task),
TP_STRUCT__entry(
__array( char, comm, TASK_COMM_LEN )
@@ -428,7 +429,8 @@
memcpy(__entry->comm, tsk->comm, TASK_COMM_LEN);
__entry->pid = tsk->pid;
__entry->oldprio = tsk->prio;
- __entry->newprio = newprio;
+ __entry->newprio = pi_task ? pi_task->prio : tsk->prio;
+ /* XXX SCHED_DEADLINE bits missing */
),
TP_printk("comm=%s pid=%d oldprio=%d newprio=%d",
diff --git a/include/trace/events/xen.h b/include/trace/events/xen.h
index bce990f..31acce9 100644
--- a/include/trace/events/xen.h
+++ b/include/trace/events/xen.h
@@ -241,21 +241,21 @@
(int)sizeof(pudval_t) * 2, (unsigned long long)__entry->pudval)
);
-TRACE_EVENT(xen_mmu_set_pgd,
- TP_PROTO(pgd_t *pgdp, pgd_t *user_pgdp, pgd_t pgdval),
- TP_ARGS(pgdp, user_pgdp, pgdval),
+TRACE_EVENT(xen_mmu_set_p4d,
+ TP_PROTO(p4d_t *p4dp, p4d_t *user_p4dp, p4d_t p4dval),
+ TP_ARGS(p4dp, user_p4dp, p4dval),
TP_STRUCT__entry(
- __field(pgd_t *, pgdp)
- __field(pgd_t *, user_pgdp)
- __field(pgdval_t, pgdval)
+ __field(p4d_t *, p4dp)
+ __field(p4d_t *, user_p4dp)
+ __field(p4dval_t, p4dval)
),
- TP_fast_assign(__entry->pgdp = pgdp;
- __entry->user_pgdp = user_pgdp;
- __entry->pgdval = pgdval.pgd),
- TP_printk("pgdp %p user_pgdp %p pgdval %0*llx (raw %0*llx)",
- __entry->pgdp, __entry->user_pgdp,
- (int)sizeof(pgdval_t) * 2, (unsigned long long)pgd_val(native_make_pgd(__entry->pgdval)),
- (int)sizeof(pgdval_t) * 2, (unsigned long long)__entry->pgdval)
+ TP_fast_assign(__entry->p4dp = p4dp;
+ __entry->user_p4dp = user_p4dp;
+ __entry->p4dval = p4d_val(p4dval)),
+ TP_printk("p4dp %p user_p4dp %p p4dval %0*llx (raw %0*llx)",
+ __entry->p4dp, __entry->user_p4dp,
+ (int)sizeof(p4dval_t) * 2, (unsigned long long)pgd_val(native_make_pgd(__entry->p4dval)),
+ (int)sizeof(p4dval_t) * 2, (unsigned long long)__entry->p4dval)
);
TRACE_EVENT(xen_mmu_pud_clear,
diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/socket.h
index 2c748dd..2b48856 100644
--- a/include/uapi/asm-generic/socket.h
+++ b/include/uapi/asm-generic/socket.h
@@ -94,4 +94,10 @@
#define SCM_TIMESTAMPING_OPT_STATS 54
+#define SO_MEMINFO 55
+
+#define SO_INCOMING_NAPI_ID 56
+
+#define SO_COOKIE 57
+
#endif /* __ASM_GENERIC_SOCKET_H */
diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h
index a076cf1..061185a 100644
--- a/include/uapi/asm-generic/unistd.h
+++ b/include/uapi/asm-generic/unistd.h
@@ -194,8 +194,7 @@
/* fs/readdir.c */
#define __NR_getdents64 61
-#define __ARCH_WANT_COMPAT_SYS_GETDENTS64
-__SC_COMP(__NR_getdents64, sys_getdents64, compat_sys_getdents64)
+__SYSCALL(__NR_getdents64, sys_getdents64)
/* fs/read_write.c */
#define __NR3264_lseek 62
diff --git a/include/uapi/linux/Kbuild b/include/uapi/linux/Kbuild
index f8d9fed..6b0e275 100644
--- a/include/uapi/linux/Kbuild
+++ b/include/uapi/linux/Kbuild
@@ -477,6 +477,7 @@
header-y += virtio_vsock.h
header-y += virtio_crypto.h
header-y += vm_sockets.h
+header-y += vsockmon.h
header-y += vt.h
header-y += vtpm_proxy.h
header-y += wait.h
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 0539a0c..945a1f5 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -81,6 +81,7 @@
BPF_OBJ_GET,
BPF_PROG_ATTACH,
BPF_PROG_DETACH,
+ BPF_PROG_TEST_RUN,
};
enum bpf_map_type {
@@ -96,6 +97,8 @@
BPF_MAP_TYPE_LRU_HASH,
BPF_MAP_TYPE_LRU_PERCPU_HASH,
BPF_MAP_TYPE_LPM_TRIE,
+ BPF_MAP_TYPE_ARRAY_OF_MAPS,
+ BPF_MAP_TYPE_HASH_OF_MAPS,
};
enum bpf_prog_type {
@@ -152,6 +155,7 @@
__u32 value_size; /* size of value in bytes */
__u32 max_entries; /* max number of entries in a map */
__u32 map_flags; /* prealloc or not */
+ __u32 inner_map_fd; /* fd pointing to the inner map */
};
struct { /* anonymous struct used by BPF_MAP_*_ELEM commands */
@@ -186,6 +190,17 @@
__u32 attach_type;
__u32 attach_flags;
};
+
+ struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */
+ __u32 prog_fd;
+ __u32 retval;
+ __u32 data_size_in;
+ __u32 data_size_out;
+ __aligned_u64 data_in;
+ __aligned_u64 data_out;
+ __u32 repeat;
+ __u32 duration;
+ } test;
} __attribute__((aligned(8)));
/* BPF helper function descriptions:
@@ -456,6 +471,17 @@
* Return:
* > 0 length of the string including the trailing NUL on success
* < 0 error
+ *
+ * u64 bpf_get_socket_cookie(skb)
+ * Get the cookie for the socket stored inside sk_buff.
+ * @skb: pointer to skb
+ * Return: 8 Bytes non-decreasing number on success or 0 if the socket
+ * field is missing inside sk_buff
+ *
+ * u32 bpf_get_socket_uid(skb)
+ * Get the owner uid of the socket stored inside sk_buff.
+ * @skb: pointer to skb
+ * Return: uid of the socket owner on success or overflowuid if failed.
*/
#define __BPF_FUNC_MAPPER(FN) \
FN(unspec), \
@@ -503,7 +529,9 @@
FN(get_numa_node_id), \
FN(skb_change_head), \
FN(xdp_adjust_head), \
- FN(probe_read_str),
+ FN(probe_read_str), \
+ FN(get_socket_cookie), \
+ FN(get_socket_uid),
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
* function eBPF program intends to call
@@ -574,6 +602,7 @@
__u32 tc_classid;
__u32 data;
__u32 data_end;
+ __u32 napi_id;
};
struct bpf_tunnel_key {
diff --git a/include/uapi/linux/can/vxcan.h b/include/uapi/linux/can/vxcan.h
new file mode 100644
index 0000000..ffb0b71
--- /dev/null
+++ b/include/uapi/linux/can/vxcan.h
@@ -0,0 +1,12 @@
+#ifndef _UAPI_CAN_VXCAN_H
+#define _UAPI_CAN_VXCAN_H
+
+enum {
+ VXCAN_INFO_UNSPEC,
+ VXCAN_INFO_PEER,
+
+ __VXCAN_INFO_MAX
+#define VXCAN_INFO_MAX (__VXCAN_INFO_MAX - 1)
+};
+
+#endif
diff --git a/include/uapi/linux/cryptouser.h b/include/uapi/linux/cryptouser.h
index 11d21fc..b4def5c 100644
--- a/include/uapi/linux/cryptouser.h
+++ b/include/uapi/linux/cryptouser.h
@@ -31,7 +31,7 @@
#define CRYPTO_MSG_MAX (__CRYPTO_MSG_MAX - 1)
#define CRYPTO_NR_MSGTYPES (CRYPTO_MSG_MAX + 1 - CRYPTO_MSG_BASE)
-#define CRYPTO_MAX_NAME CRYPTO_MAX_ALG_NAME
+#define CRYPTO_MAX_NAME 64
/* Netlink message attributes. */
enum crypto_attr_type_t {
@@ -53,9 +53,9 @@
};
struct crypto_user_alg {
- char cru_name[CRYPTO_MAX_ALG_NAME];
- char cru_driver_name[CRYPTO_MAX_ALG_NAME];
- char cru_module_name[CRYPTO_MAX_ALG_NAME];
+ char cru_name[CRYPTO_MAX_NAME];
+ char cru_driver_name[CRYPTO_MAX_NAME];
+ char cru_module_name[CRYPTO_MAX_NAME];
__u32 cru_type;
__u32 cru_mask;
__u32 cru_refcnt;
@@ -73,7 +73,7 @@
};
struct crypto_report_cipher {
- char type[CRYPTO_MAX_ALG_NAME];
+ char type[CRYPTO_MAX_NAME];
unsigned int blocksize;
unsigned int min_keysize;
unsigned int max_keysize;
diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
index 0f1f3a1..b0e807a 100644
--- a/include/uapi/linux/devlink.h
+++ b/include/uapi/linux/devlink.h
@@ -65,8 +65,12 @@
#define DEVLINK_CMD_ESWITCH_MODE_SET /* obsolete, never use this! */ \
DEVLINK_CMD_ESWITCH_SET
- /* add new commands above here */
+ DEVLINK_CMD_DPIPE_TABLE_GET,
+ DEVLINK_CMD_DPIPE_ENTRIES_GET,
+ DEVLINK_CMD_DPIPE_HEADERS_GET,
+ DEVLINK_CMD_DPIPE_TABLE_COUNTERS_SET,
+ /* add new commands above here */
__DEVLINK_CMD_MAX,
DEVLINK_CMD_MAX = __DEVLINK_CMD_MAX - 1
};
@@ -115,6 +119,11 @@
DEVLINK_ESWITCH_INLINE_MODE_TRANSPORT,
};
+enum devlink_eswitch_encap_mode {
+ DEVLINK_ESWITCH_ENCAP_MODE_NONE,
+ DEVLINK_ESWITCH_ENCAP_MODE_BASIC,
+};
+
enum devlink_attr {
/* don't change the order or add anything between, this is ABI! */
DEVLINK_ATTR_UNSPEC,
@@ -148,10 +157,73 @@
DEVLINK_ATTR_ESWITCH_MODE, /* u16 */
DEVLINK_ATTR_ESWITCH_INLINE_MODE, /* u8 */
+ DEVLINK_ATTR_DPIPE_TABLES, /* nested */
+ DEVLINK_ATTR_DPIPE_TABLE, /* nested */
+ DEVLINK_ATTR_DPIPE_TABLE_NAME, /* string */
+ DEVLINK_ATTR_DPIPE_TABLE_SIZE, /* u64 */
+ DEVLINK_ATTR_DPIPE_TABLE_MATCHES, /* nested */
+ DEVLINK_ATTR_DPIPE_TABLE_ACTIONS, /* nested */
+ DEVLINK_ATTR_DPIPE_TABLE_COUNTERS_ENABLED, /* u8 */
+
+ DEVLINK_ATTR_DPIPE_ENTRIES, /* nested */
+ DEVLINK_ATTR_DPIPE_ENTRY, /* nested */
+ DEVLINK_ATTR_DPIPE_ENTRY_INDEX, /* u64 */
+ DEVLINK_ATTR_DPIPE_ENTRY_MATCH_VALUES, /* nested */
+ DEVLINK_ATTR_DPIPE_ENTRY_ACTION_VALUES, /* nested */
+ DEVLINK_ATTR_DPIPE_ENTRY_COUNTER, /* u64 */
+
+ DEVLINK_ATTR_DPIPE_MATCH, /* nested */
+ DEVLINK_ATTR_DPIPE_MATCH_VALUE, /* nested */
+ DEVLINK_ATTR_DPIPE_MATCH_TYPE, /* u32 */
+
+ DEVLINK_ATTR_DPIPE_ACTION, /* nested */
+ DEVLINK_ATTR_DPIPE_ACTION_VALUE, /* nested */
+ DEVLINK_ATTR_DPIPE_ACTION_TYPE, /* u32 */
+
+ DEVLINK_ATTR_DPIPE_VALUE,
+ DEVLINK_ATTR_DPIPE_VALUE_MASK,
+ DEVLINK_ATTR_DPIPE_VALUE_MAPPING, /* u32 */
+
+ DEVLINK_ATTR_DPIPE_HEADERS, /* nested */
+ DEVLINK_ATTR_DPIPE_HEADER, /* nested */
+ DEVLINK_ATTR_DPIPE_HEADER_NAME, /* string */
+ DEVLINK_ATTR_DPIPE_HEADER_ID, /* u32 */
+ DEVLINK_ATTR_DPIPE_HEADER_FIELDS, /* nested */
+ DEVLINK_ATTR_DPIPE_HEADER_GLOBAL, /* u8 */
+ DEVLINK_ATTR_DPIPE_HEADER_INDEX, /* u32 */
+
+ DEVLINK_ATTR_DPIPE_FIELD, /* nested */
+ DEVLINK_ATTR_DPIPE_FIELD_NAME, /* string */
+ DEVLINK_ATTR_DPIPE_FIELD_ID, /* u32 */
+ DEVLINK_ATTR_DPIPE_FIELD_BITWIDTH, /* u32 */
+ DEVLINK_ATTR_DPIPE_FIELD_MAPPING_TYPE, /* u32 */
+
+ DEVLINK_ATTR_PAD,
+
+ DEVLINK_ATTR_ESWITCH_ENCAP_MODE, /* u8 */
+
/* add new attributes above here, update the policy in devlink.c */
__DEVLINK_ATTR_MAX,
DEVLINK_ATTR_MAX = __DEVLINK_ATTR_MAX - 1
};
+/* Mapping between internal resource described by the field and system
+ * structure
+ */
+enum devlink_dpipe_field_mapping_type {
+ DEVLINK_DPIPE_FIELD_MAPPING_TYPE_NONE,
+ DEVLINK_DPIPE_FIELD_MAPPING_TYPE_IFINDEX,
+};
+
+/* Match type - specify the type of the match */
+enum devlink_dpipe_match_type {
+ DEVLINK_DPIPE_MATCH_TYPE_FIELD_EXACT,
+};
+
+/* Action type - specify the action type */
+enum devlink_dpipe_action_type {
+ DEVLINK_DPIPE_ACTION_TYPE_FIELD_MODIFY,
+};
+
#endif /* _UAPI_LINUX_DEVLINK_H_ */
diff --git a/include/uapi/linux/elf-em.h b/include/uapi/linux/elf-em.h
index cb5d1a5..9cd1de9 100644
--- a/include/uapi/linux/elf-em.h
+++ b/include/uapi/linux/elf-em.h
@@ -42,7 +42,6 @@
#define EM_TILEGX 191 /* Tilera TILE-Gx */
#define EM_BPF 247 /* Linux BPF - in-kernel virtual machine */
#define EM_FRV 0x5441 /* Fujitsu FR-V */
-#define EM_AVR32 0x18ad /* Atmel AVR32 */
/*
* This is an interim value that we will use until the committee comes
diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h
index b59ee07..176b6cb 100644
--- a/include/uapi/linux/elf.h
+++ b/include/uapi/linux/elf.h
@@ -409,6 +409,8 @@
#define NT_S390_TDB 0x308 /* s390 transaction diagnostic block */
#define NT_S390_VXRS_LOW 0x309 /* s390 vector registers 0-15 upper half */
#define NT_S390_VXRS_HIGH 0x30a /* s390 vector registers 16-31 */
+#define NT_S390_GS_CB 0x30b /* s390 guarded storage registers */
+#define NT_S390_GS_BC 0x30c /* s390 guarded storage broadcast control block */
#define NT_ARM_VFP 0x400 /* ARM VFP/NEON registers */
#define NT_ARM_TLS 0x401 /* ARM TLS register */
#define NT_ARM_HW_BREAK 0x402 /* ARM hardware breakpoint registers */
diff --git a/include/uapi/linux/ethtool.h b/include/uapi/linux/ethtool.h
index 3dc91a4..5f4ea28 100644
--- a/include/uapi/linux/ethtool.h
+++ b/include/uapi/linux/ethtool.h
@@ -1487,6 +1487,7 @@
*/
/* The forced speed, in units of 1Mb. All values 0 to INT_MAX are legal. */
+/* Update drivers/net/phy/phy.c:phy_speed_to_str() when adding new values */
#define SPEED_10 10
#define SPEED_100 100
#define SPEED_1000 1000
diff --git a/include/uapi/linux/gtp.h b/include/uapi/linux/gtp.h
index 72a04a0..57d1edb 100644
--- a/include/uapi/linux/gtp.h
+++ b/include/uapi/linux/gtp.h
@@ -19,7 +19,8 @@
GTPA_LINK,
GTPA_VERSION,
GTPA_TID, /* for GTPv0 only */
- GTPA_SGSN_ADDRESS,
+ GTPA_PEER_ADDRESS, /* Remote GSN peer, either SGSN or GGSN */
+#define GTPA_SGSN_ADDRESS GTPA_PEER_ADDRESS /* maintain legacy attr name */
GTPA_MS_ADDRESS,
GTPA_FLOW,
GTPA_NET_NS_FD,
diff --git a/include/uapi/linux/if_arp.h b/include/uapi/linux/if_arp.h
index 4d024d7..cf73510 100644
--- a/include/uapi/linux/if_arp.h
+++ b/include/uapi/linux/if_arp.h
@@ -95,6 +95,7 @@
#define ARPHRD_IP6GRE 823 /* GRE over IPv6 */
#define ARPHRD_NETLINK 824 /* Netlink header */
#define ARPHRD_6LOWPAN 825 /* IPv6 over LoWPAN */
+#define ARPHRD_VSOCKMON 826 /* Vsock monitor header */
#define ARPHRD_VOID 0xFFFF /* Void type, nothing is known */
#define ARPHRD_NONE 0xFFFE /* zero header length */
diff --git a/include/uapi/linux/if_link.h b/include/uapi/linux/if_link.h
index 320fc1e..8e56ac7 100644
--- a/include/uapi/linux/if_link.h
+++ b/include/uapi/linux/if_link.h
@@ -323,6 +323,7 @@
IFLA_BRPORT_MCAST_FLOOD,
IFLA_BRPORT_MCAST_TO_UCAST,
IFLA_BRPORT_VLAN_TUNNEL,
+ IFLA_BRPORT_BCAST_FLOOD,
__IFLA_BRPORT_MAX
};
#define IFLA_BRPORT_MAX (__IFLA_BRPORT_MAX - 1)
@@ -538,11 +539,18 @@
#define IFLA_PPP_MAX (__IFLA_PPP_MAX - 1)
/* GTP section */
+
+enum ifla_gtp_role {
+ GTP_ROLE_GGSN = 0,
+ GTP_ROLE_SGSN,
+};
+
enum {
IFLA_GTP_UNSPEC,
IFLA_GTP_FD0,
IFLA_GTP_FD1,
IFLA_GTP_PDP_HASHSIZE,
+ IFLA_GTP_ROLE,
__IFLA_GTP_MAX,
};
#define IFLA_GTP_MAX (__IFLA_GTP_MAX - 1)
@@ -880,7 +888,9 @@
/* XDP section */
#define XDP_FLAGS_UPDATE_IF_NOEXIST (1U << 0)
-#define XDP_FLAGS_MASK (XDP_FLAGS_UPDATE_IF_NOEXIST)
+#define XDP_FLAGS_SKB_MODE (2U << 0)
+#define XDP_FLAGS_MASK (XDP_FLAGS_UPDATE_IF_NOEXIST | \
+ XDP_FLAGS_SKB_MODE)
enum {
IFLA_XDP_UNSPEC,
diff --git a/include/uapi/linux/if_packet.h b/include/uapi/linux/if_packet.h
index 9e7edfd..4df96a7 100644
--- a/include/uapi/linux/if_packet.h
+++ b/include/uapi/linux/if_packet.h
@@ -66,6 +66,7 @@
#define PACKET_FANOUT_CBPF 6
#define PACKET_FANOUT_EBPF 7
#define PACKET_FANOUT_FLAG_ROLLOVER 0x1000
+#define PACKET_FANOUT_FLAG_UNIQUEID 0x2000
#define PACKET_FANOUT_FLAG_DEFRAG 0x8000
struct tpacket_stats {
diff --git a/include/uapi/linux/if_tunnel.h b/include/uapi/linux/if_tunnel.h
index 92f3c86..6792d19 100644
--- a/include/uapi/linux/if_tunnel.h
+++ b/include/uapi/linux/if_tunnel.h
@@ -75,6 +75,7 @@
IFLA_IPTUN_ENCAP_SPORT,
IFLA_IPTUN_ENCAP_DPORT,
IFLA_IPTUN_COLLECT_METADATA,
+ IFLA_IPTUN_FWMARK,
__IFLA_IPTUN_MAX,
};
#define IFLA_IPTUN_MAX (__IFLA_IPTUN_MAX - 1)
@@ -132,6 +133,7 @@
IFLA_GRE_ENCAP_DPORT,
IFLA_GRE_COLLECT_METADATA,
IFLA_GRE_IGNORE_DF,
+ IFLA_GRE_FWMARK,
__IFLA_GRE_MAX,
};
@@ -147,6 +149,7 @@
IFLA_VTI_OKEY,
IFLA_VTI_LOCAL,
IFLA_VTI_REMOTE,
+ IFLA_VTI_FWMARK,
__IFLA_VTI_MAX,
};
diff --git a/include/uapi/linux/input-event-codes.h b/include/uapi/linux/input-event-codes.h
index 3af60ee..f5a8d96 100644
--- a/include/uapi/linux/input-event-codes.h
+++ b/include/uapi/linux/input-event-codes.h
@@ -641,6 +641,7 @@
* e.g. teletext or data broadcast application (MHEG, MHP, HbbTV, etc.)
*/
#define KEY_DATA 0x277
+#define KEY_ONSCREEN_KEYBOARD 0x278
#define BTN_TRIGGER_HAPPY 0x2c0
#define BTN_TRIGGER_HAPPY1 0x2c0
diff --git a/include/uapi/linux/input.h b/include/uapi/linux/input.h
index e794f7b..f561c0e 100644
--- a/include/uapi/linux/input.h
+++ b/include/uapi/linux/input.h
@@ -61,9 +61,14 @@
* Note that input core does not clamp reported values to the
* [minimum, maximum] limits, such task is left to userspace.
*
- * Resolution for main axes (ABS_X, ABS_Y, ABS_Z) is reported in
- * units per millimeter (units/mm), resolution for rotational axes
- * (ABS_RX, ABS_RY, ABS_RZ) is reported in units per radian.
+ * The default resolution for main axes (ABS_X, ABS_Y, ABS_Z)
+ * is reported in units per millimeter (units/mm), resolution
+ * for rotational axes (ABS_RX, ABS_RY, ABS_RZ) is reported
+ * in units per radian.
+ * When INPUT_PROP_ACCELEROMETER is set the resolution changes.
+ * The main axes (ABS_X, ABS_Y, ABS_Z) are then reported in
+ * in units per g (units/g) and in units per degree per second
+ * (units/deg/s) for rotational axes (ABS_RX, ABS_RY, ABS_RZ).
*/
struct input_absinfo {
__s32 value;
diff --git a/include/uapi/linux/ipv6.h b/include/uapi/linux/ipv6.h
index 8ef9e75..2ae5917 100644
--- a/include/uapi/linux/ipv6.h
+++ b/include/uapi/linux/ipv6.h
@@ -183,6 +183,8 @@
DEVCONF_SEG6_REQUIRE_HMAC,
DEVCONF_ENHANCED_DAD,
DEVCONF_ADDR_GEN_MODE,
+ DEVCONF_DISABLE_POLICY,
+ DEVCONF_ACCEPT_RA_RT_INFO_MIN_PLEN,
DEVCONF_MAX
};
diff --git a/include/uapi/linux/lightnvm.h b/include/uapi/linux/lightnvm.h
index fd19f36..c8aec4b 100644
--- a/include/uapi/linux/lightnvm.h
+++ b/include/uapi/linux/lightnvm.h
@@ -85,6 +85,10 @@
};
};
+enum {
+ NVM_TARGET_FACTORY = 1 << 0, /* Init target in factory mode */
+};
+
struct nvm_ioctl_create {
char dev[DISK_NAME_LEN]; /* open-channel SSD device */
char tgttype[NVM_TTYPE_NAME_MAX]; /* target type name */
diff --git a/include/uapi/linux/mpls_iptunnel.h b/include/uapi/linux/mpls_iptunnel.h
index d80a049..f5e4509 100644
--- a/include/uapi/linux/mpls_iptunnel.h
+++ b/include/uapi/linux/mpls_iptunnel.h
@@ -16,11 +16,13 @@
/* MPLS tunnel attributes
* [RTA_ENCAP] = {
* [MPLS_IPTUNNEL_DST]
+ * [MPLS_IPTUNNEL_TTL]
* }
*/
enum {
MPLS_IPTUNNEL_UNSPEC,
MPLS_IPTUNNEL_DST,
+ MPLS_IPTUNNEL_TTL,
__MPLS_IPTUNNEL_MAX,
};
#define MPLS_IPTUNNEL_MAX (__MPLS_IPTUNNEL_MAX - 1)
diff --git a/include/uapi/linux/nbd-netlink.h b/include/uapi/linux/nbd-netlink.h
new file mode 100644
index 0000000..6f7ca3d
--- /dev/null
+++ b/include/uapi/linux/nbd-netlink.h
@@ -0,0 +1,98 @@
+/*
+ * Copyright (C) 2017 Facebook. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+#ifndef _UAPILINUX_NBD_NETLINK_H
+#define _UAPILINUX_NBD_NETLINK_H
+
+#define NBD_GENL_FAMILY_NAME "nbd"
+#define NBD_GENL_VERSION 0x1
+#define NBD_GENL_MCAST_GROUP_NAME "nbd_mc_group"
+
+/* Configuration policy attributes, used for CONNECT */
+enum {
+ NBD_ATTR_UNSPEC,
+ NBD_ATTR_INDEX,
+ NBD_ATTR_SIZE_BYTES,
+ NBD_ATTR_BLOCK_SIZE_BYTES,
+ NBD_ATTR_TIMEOUT,
+ NBD_ATTR_SERVER_FLAGS,
+ NBD_ATTR_CLIENT_FLAGS,
+ NBD_ATTR_SOCKETS,
+ NBD_ATTR_DEAD_CONN_TIMEOUT,
+ NBD_ATTR_DEVICE_LIST,
+ __NBD_ATTR_MAX,
+};
+#define NBD_ATTR_MAX (__NBD_ATTR_MAX - 1)
+
+/*
+ * This is the format for multiple devices with NBD_ATTR_DEVICE_LIST
+ *
+ * [NBD_ATTR_DEVICE_LIST]
+ * [NBD_DEVICE_ITEM]
+ * [NBD_DEVICE_INDEX]
+ * [NBD_DEVICE_CONNECTED]
+ */
+enum {
+ NBD_DEVICE_ITEM_UNSPEC,
+ NBD_DEVICE_ITEM,
+ __NBD_DEVICE_ITEM_MAX,
+};
+#define NBD_DEVICE_ITEM_MAX (__NBD_DEVICE_ITEM_MAX - 1)
+
+enum {
+ NBD_DEVICE_UNSPEC,
+ NBD_DEVICE_INDEX,
+ NBD_DEVICE_CONNECTED,
+ __NBD_DEVICE_MAX,
+};
+#define NBD_DEVICE_ATTR_MAX (__NBD_DEVICE_MAX - 1)
+
+/*
+ * This is the format for multiple sockets with NBD_ATTR_SOCKETS
+ *
+ * [NBD_ATTR_SOCKETS]
+ * [NBD_SOCK_ITEM]
+ * [NBD_SOCK_FD]
+ * [NBD_SOCK_ITEM]
+ * [NBD_SOCK_FD]
+ */
+enum {
+ NBD_SOCK_ITEM_UNSPEC,
+ NBD_SOCK_ITEM,
+ __NBD_SOCK_ITEM_MAX,
+};
+#define NBD_SOCK_ITEM_MAX (__NBD_SOCK_ITEM_MAX - 1)
+
+enum {
+ NBD_SOCK_UNSPEC,
+ NBD_SOCK_FD,
+ __NBD_SOCK_MAX,
+};
+#define NBD_SOCK_MAX (__NBD_SOCK_MAX - 1)
+
+enum {
+ NBD_CMD_UNSPEC,
+ NBD_CMD_CONNECT,
+ NBD_CMD_DISCONNECT,
+ NBD_CMD_RECONFIGURE,
+ NBD_CMD_LINK_DEAD,
+ NBD_CMD_STATUS,
+ __NBD_CMD_MAX,
+};
+#define NBD_CMD_MAX (__NBD_CMD_MAX - 1)
+
+#endif /* _UAPILINUX_NBD_NETLINK_H */
diff --git a/include/uapi/linux/nbd.h b/include/uapi/linux/nbd.h
index c91c642..155e33f 100644
--- a/include/uapi/linux/nbd.h
+++ b/include/uapi/linux/nbd.h
@@ -37,7 +37,7 @@
NBD_CMD_TRIM = 4
};
-/* values for flags field */
+/* values for flags field, these are server interaction specific. */
#define NBD_FLAG_HAS_FLAGS (1 << 0) /* nbd-server supports flags */
#define NBD_FLAG_READ_ONLY (1 << 1) /* device is read-only */
#define NBD_FLAG_SEND_FLUSH (1 << 2) /* can flush writeback cache */
@@ -45,6 +45,10 @@
#define NBD_FLAG_SEND_TRIM (1 << 5) /* send trim/discard */
#define NBD_FLAG_CAN_MULTI_CONN (1 << 8) /* Server supports multiple connections per export. */
+/* These are client behavior specific flags. */
+#define NBD_CFLAG_DESTROY_ON_DISCONNECT (1 << 0) /* delete the nbd device on
+ disconnect. */
+
/* userspace doesn't need the nbd_device structure */
/* These are sent over the network in the request/reply magic fields */
diff --git a/include/uapi/linux/netfilter/nf_conntrack_common.h b/include/uapi/linux/netfilter/nf_conntrack_common.h
index 38fc383..dc947e5 100644
--- a/include/uapi/linux/netfilter/nf_conntrack_common.h
+++ b/include/uapi/linux/netfilter/nf_conntrack_common.h
@@ -28,12 +28,14 @@
/* only for userspace compatibility */
#ifndef __KERNEL__
IP_CT_NEW_REPLY = IP_CT_NUMBER,
+#else
+ IP_CT_UNTRACKED = 7,
#endif
};
#define NF_CT_STATE_INVALID_BIT (1 << 0)
#define NF_CT_STATE_BIT(ctinfo) (1 << ((ctinfo) % IP_CT_IS_REPLY + 1))
-#define NF_CT_STATE_UNTRACKED_BIT (1 << (IP_CT_NUMBER + 1))
+#define NF_CT_STATE_UNTRACKED_BIT (1 << (IP_CT_UNTRACKED + 1))
/* Bitset representing status of connection. */
enum ip_conntrack_status {
@@ -90,7 +92,7 @@
IPS_TEMPLATE_BIT = 11,
IPS_TEMPLATE = (1 << IPS_TEMPLATE_BIT),
- /* Conntrack is a fake untracked entry */
+ /* Conntrack is a fake untracked entry. Obsolete and not used anymore */
IPS_UNTRACKED_BIT = 12,
IPS_UNTRACKED = (1 << IPS_UNTRACKED_BIT),
@@ -122,6 +124,9 @@
IPCT_NATSEQADJ = IPCT_SEQADJ,
IPCT_SECMARK, /* new security mark has been set */
IPCT_LABEL, /* new connlabel has been set */
+#ifdef __KERNEL__
+ __IPCT_MAX
+#endif
};
enum ip_conntrack_expect_events {
diff --git a/include/uapi/linux/netfilter/nf_tables.h b/include/uapi/linux/netfilter/nf_tables.h
index 05215d3..683f6f8 100644
--- a/include/uapi/linux/netfilter/nf_tables.h
+++ b/include/uapi/linux/netfilter/nf_tables.h
@@ -816,6 +816,17 @@
};
/**
+ * enum nft_hash_types - nf_tables hash expression types
+ *
+ * @NFT_HASH_JENKINS: Jenkins Hash
+ * @NFT_HASH_SYM: Symmetric Hash
+ */
+enum nft_hash_types {
+ NFT_HASH_JENKINS,
+ NFT_HASH_SYM,
+};
+
+/**
* enum nft_hash_attributes - nf_tables hash expression netlink attributes
*
* @NFTA_HASH_SREG: source register (NLA_U32)
@@ -824,6 +835,7 @@
* @NFTA_HASH_MODULUS: modulus value (NLA_U32)
* @NFTA_HASH_SEED: seed value (NLA_U32)
* @NFTA_HASH_OFFSET: add this offset value to hash result (NLA_U32)
+ * @NFTA_HASH_TYPE: hash operation (NLA_U32: nft_hash_types)
*/
enum nft_hash_attributes {
NFTA_HASH_UNSPEC,
@@ -833,6 +845,7 @@
NFTA_HASH_MODULUS,
NFTA_HASH_SEED,
NFTA_HASH_OFFSET,
+ NFTA_HASH_TYPE,
__NFTA_HASH_MAX,
};
#define NFTA_HASH_MAX (__NFTA_HASH_MAX - 1)
@@ -888,6 +901,7 @@
* @NFT_CT_BYTES: conntrack bytes
* @NFT_CT_AVGPKT: conntrack average bytes per packet
* @NFT_CT_ZONE: conntrack zone
+ * @NFT_CT_EVENTMASK: ctnetlink events to be generated for this conntrack
*/
enum nft_ct_keys {
NFT_CT_STATE,
@@ -908,6 +922,7 @@
NFT_CT_BYTES,
NFT_CT_AVGPKT,
NFT_CT_ZONE,
+ NFT_CT_EVENTMASK,
};
/**
@@ -1244,12 +1259,23 @@
NFTA_FIB_F_MARK = 1 << 2, /* use skb->mark */
NFTA_FIB_F_IIF = 1 << 3, /* restrict to iif */
NFTA_FIB_F_OIF = 1 << 4, /* restrict to oif */
+ NFTA_FIB_F_PRESENT = 1 << 5, /* check existence only */
};
+enum nft_ct_helper_attributes {
+ NFTA_CT_HELPER_UNSPEC,
+ NFTA_CT_HELPER_NAME,
+ NFTA_CT_HELPER_L3PROTO,
+ NFTA_CT_HELPER_L4PROTO,
+ __NFTA_CT_HELPER_MAX,
+};
+#define NFTA_CT_HELPER_MAX (__NFTA_CT_HELPER_MAX - 1)
+
#define NFT_OBJECT_UNSPEC 0
#define NFT_OBJECT_COUNTER 1
#define NFT_OBJECT_QUOTA 2
-#define __NFT_OBJECT_MAX 3
+#define NFT_OBJECT_CT_HELPER 3
+#define __NFT_OBJECT_MAX 4
#define NFT_OBJECT_MAX (__NFT_OBJECT_MAX - 1)
/**
diff --git a/include/uapi/linux/netlink.h b/include/uapi/linux/netlink.h
index f3946a2..f86127a 100644
--- a/include/uapi/linux/netlink.h
+++ b/include/uapi/linux/netlink.h
@@ -50,12 +50,12 @@
/* Flags values */
-#define NLM_F_REQUEST 1 /* It is request message. */
-#define NLM_F_MULTI 2 /* Multipart message, terminated by NLMSG_DONE */
-#define NLM_F_ACK 4 /* Reply with ack, with zero or error code */
-#define NLM_F_ECHO 8 /* Echo this request */
-#define NLM_F_DUMP_INTR 16 /* Dump was inconsistent due to sequence change */
-#define NLM_F_DUMP_FILTERED 32 /* Dump was filtered as requested */
+#define NLM_F_REQUEST 0x01 /* It is request message. */
+#define NLM_F_MULTI 0x02 /* Multipart message, terminated by NLMSG_DONE */
+#define NLM_F_ACK 0x04 /* Reply with ack, with zero or error code */
+#define NLM_F_ECHO 0x08 /* Echo this request */
+#define NLM_F_DUMP_INTR 0x10 /* Dump was inconsistent due to sequence change */
+#define NLM_F_DUMP_FILTERED 0x20 /* Dump was filtered as requested */
/* Modifiers to GET request */
#define NLM_F_ROOT 0x100 /* specify tree root */
@@ -69,6 +69,10 @@
#define NLM_F_CREATE 0x400 /* Create, if it does not exist */
#define NLM_F_APPEND 0x800 /* Add to end of list */
+/* Flags for ACK message */
+#define NLM_F_CAPPED 0x100 /* request was capped */
+#define NLM_F_ACK_TLVS 0x200 /* extended ACK TVLs were included */
+
/*
4.4BSD ADD NLM_F_CREATE|NLM_F_EXCL
4.4BSD CHANGE NLM_F_REPLACE
@@ -101,6 +105,37 @@
struct nlmsgerr {
int error;
struct nlmsghdr msg;
+ /*
+ * followed by the message contents unless NETLINK_CAP_ACK was set
+ * or the ACK indicates success (error == 0)
+ * message length is aligned with NLMSG_ALIGN()
+ */
+ /*
+ * followed by TLVs defined in enum nlmsgerr_attrs
+ * if NETLINK_EXT_ACK was set
+ */
+};
+
+/**
+ * enum nlmsgerr_attrs - nlmsgerr attributes
+ * @NLMSGERR_ATTR_UNUSED: unused
+ * @NLMSGERR_ATTR_MSG: error message string (string)
+ * @NLMSGERR_ATTR_OFFS: offset of the invalid attribute in the original
+ * message, counting from the beginning of the header (u32)
+ * @NLMSGERR_ATTR_COOKIE: arbitrary subsystem specific cookie to
+ * be used - in the success case - to identify a created
+ * object or operation or similar (binary)
+ * @__NLMSGERR_ATTR_MAX: number of attributes
+ * @NLMSGERR_ATTR_MAX: highest attribute number
+ */
+enum nlmsgerr_attrs {
+ NLMSGERR_ATTR_UNUSED,
+ NLMSGERR_ATTR_MSG,
+ NLMSGERR_ATTR_OFFS,
+ NLMSGERR_ATTR_COOKIE,
+
+ __NLMSGERR_ATTR_MAX,
+ NLMSGERR_ATTR_MAX = __NLMSGERR_ATTR_MAX - 1
};
#define NETLINK_ADD_MEMBERSHIP 1
@@ -115,6 +150,7 @@
#define NETLINK_LISTEN_ALL_NSID 8
#define NETLINK_LIST_MEMBERSHIPS 9
#define NETLINK_CAP_ACK 10
+#define NETLINK_EXT_ACK 11
struct nl_pktinfo {
__u32 group;
diff --git a/include/uapi/linux/netlink_diag.h b/include/uapi/linux/netlink_diag.h
index 76b4d87..6dcd4de 100644
--- a/include/uapi/linux/netlink_diag.h
+++ b/include/uapi/linux/netlink_diag.h
@@ -38,6 +38,7 @@
NETLINK_DIAG_GROUPS,
NETLINK_DIAG_RX_RING,
NETLINK_DIAG_TX_RING,
+ NETLINK_DIAG_FLAGS,
__NETLINK_DIAG_MAX,
};
@@ -52,5 +53,14 @@
/* deprecated since 4.6 */
#define NDIAG_SHOW_RING_CFG 0x00000004 /* show ring configuration */
#endif
+#define NDIAG_SHOW_FLAGS 0x00000008 /* show flags of a netlink socket */
+
+/* flags */
+#define NDIAG_FLAG_CB_RUNNING 0x00000001
+#define NDIAG_FLAG_PKTINFO 0x00000002
+#define NDIAG_FLAG_BROADCAST_ERROR 0x00000004
+#define NDIAG_FLAG_NO_ENOBUFS 0x00000008
+#define NDIAG_FLAG_LISTEN_ALL_NSID 0x00000010
+#define NDIAG_FLAG_CAP_ACK 0x00000020
#endif
diff --git a/include/uapi/linux/nl80211.h b/include/uapi/linux/nl80211.h
index 5ed257c..b8c44b9 100644
--- a/include/uapi/linux/nl80211.h
+++ b/include/uapi/linux/nl80211.h
@@ -173,6 +173,42 @@
*/
/**
+ * DOC: FILS shared key authentication offload
+ *
+ * FILS shared key authentication offload can be advertized by drivers by
+ * setting @NL80211_EXT_FEATURE_FILS_SK_OFFLOAD flag. The drivers that support
+ * FILS shared key authentication offload should be able to construct the
+ * authentication and association frames for FILS shared key authentication and
+ * eventually do a key derivation as per IEEE 802.11ai. The below additional
+ * parameters should be given to driver in %NL80211_CMD_CONNECT.
+ * %NL80211_ATTR_FILS_ERP_USERNAME - used to construct keyname_nai
+ * %NL80211_ATTR_FILS_ERP_REALM - used to construct keyname_nai
+ * %NL80211_ATTR_FILS_ERP_NEXT_SEQ_NUM - used to construct erp message
+ * %NL80211_ATTR_FILS_ERP_RRK - used to generate the rIK and rMSK
+ * rIK should be used to generate an authentication tag on the ERP message and
+ * rMSK should be used to derive a PMKSA.
+ * rIK, rMSK should be generated and keyname_nai, sequence number should be used
+ * as specified in IETF RFC 6696.
+ *
+ * When FILS shared key authentication is completed, driver needs to provide the
+ * below additional parameters to userspace.
+ * %NL80211_ATTR_FILS_KEK - used for key renewal
+ * %NL80211_ATTR_FILS_ERP_NEXT_SEQ_NUM - used in further EAP-RP exchanges
+ * %NL80211_ATTR_PMKID - used to identify the PMKSA used/generated
+ * %Nl80211_ATTR_PMK - used to update PMKSA cache in userspace
+ * The PMKSA can be maintained in userspace persistently so that it can be used
+ * later after reboots or wifi turn off/on also.
+ *
+ * %NL80211_ATTR_FILS_CACHE_ID is the cache identifier advertized by a FILS
+ * capable AP supporting PMK caching. It specifies the scope within which the
+ * PMKSAs are cached in an ESS. %NL80211_CMD_SET_PMKSA and
+ * %NL80211_CMD_DEL_PMKSA are enhanced to allow support for PMKSA caching based
+ * on FILS cache identifier. Additionally %NL80211_ATTR_PMK is used with
+ * %NL80211_SET_PMKSA to specify the PMK corresponding to a PMKSA for driver to
+ * use in a FILS shared key connection with PMKSA caching.
+ */
+
+/**
* enum nl80211_commands - supported nl80211 commands
*
* @NL80211_CMD_UNSPEC: unspecified command to catch errors
@@ -351,7 +387,9 @@
* are used. Extra IEs can also be passed from the userspace by
* using the %NL80211_ATTR_IE attribute. The first cycle of the
* scheduled scan can be delayed by %NL80211_ATTR_SCHED_SCAN_DELAY
- * is supplied.
+ * is supplied. If the device supports multiple concurrent scheduled
+ * scans, it will allow such when the caller provides the flag attribute
+ * %NL80211_ATTR_SCHED_SCAN_MULTI to indicate user-space support for it.
* @NL80211_CMD_STOP_SCHED_SCAN: stop a scheduled scan. Returns -ENOENT if
* scheduled scan is not running. The caller may assume that as soon
* as the call returns, it is safe to start a new scheduled scan again.
@@ -370,10 +408,18 @@
* @NL80211_CMD_NEW_SURVEY_RESULTS: survey data notification (as a reply to
* NL80211_CMD_GET_SURVEY and on the "scan" multicast group)
*
- * @NL80211_CMD_SET_PMKSA: Add a PMKSA cache entry, using %NL80211_ATTR_MAC
- * (for the BSSID) and %NL80211_ATTR_PMKID.
+ * @NL80211_CMD_SET_PMKSA: Add a PMKSA cache entry using %NL80211_ATTR_MAC
+ * (for the BSSID), %NL80211_ATTR_PMKID, and optionally %NL80211_ATTR_PMK
+ * (PMK is used for PTKSA derivation in case of FILS shared key offload) or
+ * using %NL80211_ATTR_SSID, %NL80211_ATTR_FILS_CACHE_ID,
+ * %NL80211_ATTR_PMKID, and %NL80211_ATTR_PMK in case of FILS
+ * authentication where %NL80211_ATTR_FILS_CACHE_ID is the identifier
+ * advertized by a FILS capable AP identifying the scope of PMKSA in an
+ * ESS.
* @NL80211_CMD_DEL_PMKSA: Delete a PMKSA cache entry, using %NL80211_ATTR_MAC
- * (for the BSSID) and %NL80211_ATTR_PMKID.
+ * (for the BSSID) and %NL80211_ATTR_PMKID or using %NL80211_ATTR_SSID,
+ * %NL80211_ATTR_FILS_CACHE_ID, and %NL80211_ATTR_PMKID in case of FILS
+ * authentication.
* @NL80211_CMD_FLUSH_PMKSA: Flush all PMKSA cache entries.
*
* @NL80211_CMD_REG_CHANGE: indicates to userspace the regulatory domain
@@ -2012,6 +2058,36 @@
* u32 attribute with an &enum nl80211_timeout_reason value. This is used,
* e.g., with %NL80211_CMD_CONNECT event.
*
+ * @NL80211_ATTR_FILS_ERP_USERNAME: EAP Re-authentication Protocol (ERP)
+ * username part of NAI used to refer keys rRK and rIK. This is used with
+ * %NL80211_CMD_CONNECT.
+ *
+ * @NL80211_ATTR_FILS_ERP_REALM: EAP Re-authentication Protocol (ERP) realm part
+ * of NAI specifying the domain name of the ER server. This is used with
+ * %NL80211_CMD_CONNECT.
+ *
+ * @NL80211_ATTR_FILS_ERP_NEXT_SEQ_NUM: Unsigned 16-bit ERP next sequence number
+ * to use in ERP messages. This is used in generating the FILS wrapped data
+ * for FILS authentication and is used with %NL80211_CMD_CONNECT.
+ *
+ * @NL80211_ATTR_FILS_ERP_RRK: ERP re-authentication Root Key (rRK) for the
+ * NAI specified by %NL80211_ATTR_FILS_ERP_USERNAME and
+ * %NL80211_ATTR_FILS_ERP_REALM. This is used for generating rIK and rMSK
+ * from successful FILS authentication and is used with
+ * %NL80211_CMD_CONNECT.
+ *
+ * @NL80211_ATTR_FILS_CACHE_ID: A 2-octet identifier advertized by a FILS AP
+ * identifying the scope of PMKSAs. This is used with
+ * @NL80211_CMD_SET_PMKSA and @NL80211_CMD_DEL_PMKSA.
+ *
+ * @NL80211_ATTR_PMK: PMK for the PMKSA identified by %NL80211_ATTR_PMKID.
+ * This is used with @NL80211_CMD_SET_PMKSA.
+ *
+ * @NL80211_ATTR_SCHED_SCAN_MULTI: flag attribute which user-space shall use to
+ * indicate that it supports multiple active scheduled scan requests.
+ * @NL80211_ATTR_SCHED_SCAN_MAX_REQS: indicates maximum number of scheduled
+ * scan request that may be active for the device (u32).
+ *
* @NUM_NL80211_ATTR: total number of nl80211_attrs available
* @NL80211_ATTR_MAX: highest attribute number currently defined
* @__NL80211_ATTR_AFTER_LAST: internal use
@@ -2423,6 +2499,17 @@
NL80211_ATTR_TIMEOUT_REASON,
+ NL80211_ATTR_FILS_ERP_USERNAME,
+ NL80211_ATTR_FILS_ERP_REALM,
+ NL80211_ATTR_FILS_ERP_NEXT_SEQ_NUM,
+ NL80211_ATTR_FILS_ERP_RRK,
+ NL80211_ATTR_FILS_CACHE_ID,
+
+ NL80211_ATTR_PMK,
+
+ NL80211_ATTR_SCHED_SCAN_MULTI,
+ NL80211_ATTR_SCHED_SCAN_MAX_REQS,
+
/* add attributes here, update the policy in nl80211.c */
__NL80211_ATTR_AFTER_LAST,
@@ -3107,6 +3194,7 @@
* @__NL80211_SCHED_SCAN_MATCH_ATTR_INVALID: attribute number 0 is reserved
* @NL80211_SCHED_SCAN_MATCH_ATTR_SSID: SSID to be used for matching,
* only report BSS with matching SSID.
+ * (This cannot be used together with BSSID.)
* @NL80211_SCHED_SCAN_MATCH_ATTR_RSSI: RSSI threshold (in dBm) for reporting a
* BSS in scan results. Filtering is turned off if not specified. Note that
* if this attribute is in a match set of its own, then it is treated as
@@ -3122,6 +3210,8 @@
* BSS-es in the specified band is to be adjusted before doing
* RSSI-based BSS selection. The attribute value is a packed structure
* value as specified by &struct nl80211_bss_select_rssi_adjust.
+ * @NL80211_SCHED_SCAN_MATCH_ATTR_BSSID: BSSID to be used for matching
+ * (this cannot be used together with SSID).
* @NL80211_SCHED_SCAN_MATCH_ATTR_MAX: highest scheduled scan filter
* attribute number currently defined
* @__NL80211_SCHED_SCAN_MATCH_ATTR_AFTER_LAST: internal use
@@ -3133,6 +3223,7 @@
NL80211_SCHED_SCAN_MATCH_ATTR_RSSI,
NL80211_SCHED_SCAN_MATCH_ATTR_RELATIVE_RSSI,
NL80211_SCHED_SCAN_MATCH_ATTR_RSSI_ADJUST,
+ NL80211_SCHED_SCAN_MATCH_ATTR_BSSID,
/* keep last */
__NL80211_SCHED_SCAN_MATCH_ATTR_AFTER_LAST,
@@ -3942,7 +4033,10 @@
* @__NL80211_ATTR_CQM_INVALID: invalid
* @NL80211_ATTR_CQM_RSSI_THOLD: RSSI threshold in dBm. This value specifies
* the threshold for the RSSI level at which an event will be sent. Zero
- * to disable.
+ * to disable. Alternatively, if %NL80211_EXT_FEATURE_CQM_RSSI_LIST is
+ * set, multiple values can be supplied as a low-to-high sorted array of
+ * threshold values in dBm. Events will be sent when the RSSI value
+ * crosses any of the thresholds.
* @NL80211_ATTR_CQM_RSSI_HYST: RSSI hysteresis in dBm. This value specifies
* the minimum amount the RSSI level must change after an event before a
* new event may be issued (to reduce effects of RSSI oscillation).
@@ -4753,6 +4847,11 @@
* @NL80211_EXT_FEATURE_SCHED_SCAN_RELATIVE_RSSI: The driver supports sched_scan
* for reporting BSSs with better RSSI than the current connected BSS
* (%NL80211_ATTR_SCHED_SCAN_RELATIVE_RSSI).
+ * @NL80211_EXT_FEATURE_CQM_RSSI_LIST: With this driver the
+ * %NL80211_ATTR_CQM_RSSI_THOLD attribute accepts a list of zero or more
+ * RSSI threshold values to monitor rather than exactly one threshold.
+ * @NL80211_EXT_FEATURE_FILS_SK_OFFLOAD: Driver SME supports FILS shared key
+ * authentication with %NL80211_CMD_CONNECT.
*
* @NUM_NL80211_EXT_FEATURES: number of extended features.
* @MAX_NL80211_EXT_FEATURES: highest extended feature index.
@@ -4771,6 +4870,8 @@
NL80211_EXT_FEATURE_MGMT_TX_RANDOM_TA,
NL80211_EXT_FEATURE_MGMT_TX_RANDOM_TA_CONNECTED,
NL80211_EXT_FEATURE_SCHED_SCAN_RELATIVE_RSSI,
+ NL80211_EXT_FEATURE_CQM_RSSI_LIST,
+ NL80211_EXT_FEATURE_FILS_SK_OFFLOAD,
/* add new features before the definition below */
NUM_NL80211_EXT_FEATURES,
@@ -4906,12 +5007,17 @@
* change to the channel status.
* @NL80211_RADAR_NOP_FINISHED: The Non-Occupancy Period for this channel is
* over, channel becomes usable.
+ * @NL80211_RADAR_PRE_CAC_EXPIRED: Channel Availability Check done on this
+ * non-operating channel is expired and no longer valid. New CAC must
+ * be done on this channel before starting the operation. This is not
+ * applicable for ETSI dfs domain where pre-CAC is valid for ever.
*/
enum nl80211_radar_event {
NL80211_RADAR_DETECTED,
NL80211_RADAR_CAC_FINISHED,
NL80211_RADAR_CAC_ABORTED,
NL80211_RADAR_NOP_FINISHED,
+ NL80211_RADAR_PRE_CAC_EXPIRED,
};
/**
diff --git a/include/uapi/linux/nubus.h b/include/uapi/linux/nubus.h
index 77513d2..ac51606 100644
--- a/include/uapi/linux/nubus.h
+++ b/include/uapi/linux/nubus.h
@@ -113,13 +113,15 @@
NUBUS_DRHW_SIGMA_CLRMAX = 0x0007, /* Sigma Design ColorMax */
NUBUS_DRHW_APPLE_SE30 = 0x0009, /* Apple SE/30 video */
NUBUS_DRHW_APPLE_HRVC = 0x0013, /* Mac II High-Res Video Card */
+ NUBUS_DRHW_APPLE_MVC = 0x0014, /* Mac II Monochrome Video Card */
NUBUS_DRHW_APPLE_PVC = 0x0017, /* Mac II Portrait Video Card */
NUBUS_DRHW_APPLE_RBV1 = 0x0018, /* IIci RBV video */
NUBUS_DRHW_APPLE_MDC = 0x0019, /* Macintosh Display Card */
+ NUBUS_DRHW_APPLE_VSC = 0x0020, /* Duo MiniDock ViSC framebuffer */
NUBUS_DRHW_APPLE_SONORA = 0x0022, /* Sonora built-in video */
+ NUBUS_DRHW_APPLE_JET = 0x0029, /* Jet framebuffer (DuoDock) */
NUBUS_DRHW_APPLE_24AC = 0x002b, /* Mac 24AC Video Card */
NUBUS_DRHW_APPLE_VALKYRIE = 0x002e,
- NUBUS_DRHW_APPLE_JET = 0x0029, /* Jet framebuffer (DuoDock) */
NUBUS_DRHW_SMAC_GFX = 0x0105, /* SuperMac GFX */
NUBUS_DRHW_RASTER_CB264 = 0x013B, /* RasterOps ColorBoard 264 */
NUBUS_DRHW_MICRON_XCEED = 0x0146, /* Micron Exceed color */
diff --git a/include/uapi/linux/openvswitch.h b/include/uapi/linux/openvswitch.h
index 7f41f7d..61b7d36 100644
--- a/include/uapi/linux/openvswitch.h
+++ b/include/uapi/linux/openvswitch.h
@@ -578,10 +578,25 @@
OVS_SAMPLE_ATTR_PROBABILITY, /* u32 number */
OVS_SAMPLE_ATTR_ACTIONS, /* Nested OVS_ACTION_ATTR_* attributes. */
__OVS_SAMPLE_ATTR_MAX,
+
+#ifdef __KERNEL__
+ OVS_SAMPLE_ATTR_ARG /* struct sample_arg */
+#endif
};
#define OVS_SAMPLE_ATTR_MAX (__OVS_SAMPLE_ATTR_MAX - 1)
+#ifdef __KERNEL__
+struct sample_arg {
+ bool exec; /* When true, actions in sample will not
+ * change flow keys. False otherwise.
+ */
+ u32 probability; /* Same value as
+ * 'OVS_SAMPLE_ATTR_PROBABILITY'.
+ */
+};
+#endif
+
/**
* enum ovs_userspace_attr - Attributes for %OVS_ACTION_ATTR_USERSPACE action.
* @OVS_USERSPACE_ATTR_PID: u32 Netlink PID to which the %OVS_PACKET_CMD_ACTION
@@ -678,6 +693,17 @@
* nothing if the connection is already committed will check that the current
* packet is in conntrack entry's original direction. If directionality does
* not match, will delete the existing conntrack entry and commit a new one.
+ * @OVS_CT_ATTR_EVENTMASK: Mask of bits indicating which conntrack event types
+ * (enum ip_conntrack_events IPCT_*) should be reported. For any bit set to
+ * zero, the corresponding event type is not generated. Default behavior
+ * depends on system configuration, but typically all event types are
+ * generated, hence listening on NFNLGRP_CONNTRACK_UPDATE events may get a lot
+ * of events. Explicitly passing this attribute allows limiting the updates
+ * received to the events of interest. The bit 1 << IPCT_NEW, 1 <<
+ * IPCT_RELATED, and 1 << IPCT_DESTROY must be set to ones for those events to
+ * be received on NFNLGRP_CONNTRACK_NEW and NFNLGRP_CONNTRACK_DESTROY groups,
+ * respectively. Remaining bits control the changes for which an event is
+ * delivered on the NFNLGRP_CONNTRACK_UPDATE group.
*/
enum ovs_ct_attr {
OVS_CT_ATTR_UNSPEC,
@@ -689,6 +715,7 @@
related connections. */
OVS_CT_ATTR_NAT, /* Nested OVS_NAT_ATTR_* */
OVS_CT_ATTR_FORCE_COMMIT, /* No argument */
+ OVS_CT_ATTR_EVENTMASK, /* u32 mask of IPCT_* events. */
__OVS_CT_ATTR_MAX
};
diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h
index c66a485..d09a9cd 100644
--- a/include/uapi/linux/perf_event.h
+++ b/include/uapi/linux/perf_event.h
@@ -344,7 +344,8 @@
use_clockid : 1, /* use @clockid for time fields */
context_switch : 1, /* context switch data */
write_backward : 1, /* Write ring buffer from end to beginning */
- __reserved_1 : 36;
+ namespaces : 1, /* include namespaces data */
+ __reserved_1 : 35;
union {
__u32 wakeup_events; /* wakeup every n events */
@@ -610,6 +611,23 @@
__u16 size;
};
+struct perf_ns_link_info {
+ __u64 dev;
+ __u64 ino;
+};
+
+enum {
+ NET_NS_INDEX = 0,
+ UTS_NS_INDEX = 1,
+ IPC_NS_INDEX = 2,
+ PID_NS_INDEX = 3,
+ USER_NS_INDEX = 4,
+ MNT_NS_INDEX = 5,
+ CGROUP_NS_INDEX = 6,
+
+ NR_NAMESPACES, /* number of available namespaces */
+};
+
enum perf_event_type {
/*
@@ -862,6 +880,18 @@
*/
PERF_RECORD_SWITCH_CPU_WIDE = 15,
+ /*
+ * struct {
+ * struct perf_event_header header;
+ * u32 pid;
+ * u32 tid;
+ * u64 nr_namespaces;
+ * { u64 dev, inode; } [nr_namespaces];
+ * struct sample_id sample_id;
+ * };
+ */
+ PERF_RECORD_NAMESPACES = 16,
+
PERF_RECORD_MAX, /* non-ABI */
};
@@ -885,6 +915,7 @@
*/
#define PERF_AUX_FLAG_TRUNCATED 0x01 /* record was truncated to fit */
#define PERF_AUX_FLAG_OVERWRITE 0x02 /* snapshot from overwrite mode */
+#define PERF_AUX_FLAG_PARTIAL 0x04 /* record contains gaps */
#define PERF_FLAG_FD_NO_GROUP (1UL << 0)
#define PERF_FLAG_FD_OUTPUT (1UL << 1)
diff --git a/include/uapi/linux/pkt_cls.h b/include/uapi/linux/pkt_cls.h
index 7a69f2a..d613be3 100644
--- a/include/uapi/linux/pkt_cls.h
+++ b/include/uapi/linux/pkt_cls.h
@@ -37,7 +37,20 @@
#define TC_ACT_QUEUED 5
#define TC_ACT_REPEAT 6
#define TC_ACT_REDIRECT 7
-#define TC_ACT_JUMP 0x10000000
+
+/* There is a special kind of actions called "extended actions",
+ * which need a value parameter. These have a local opcode located in
+ * the highest nibble, starting from 1. The rest of the bits
+ * are used to carry the value. These two parts together make
+ * a combined opcode.
+ */
+#define __TC_ACT_EXT_SHIFT 28
+#define __TC_ACT_EXT(local) ((local) << __TC_ACT_EXT_SHIFT)
+#define TC_ACT_EXT_VAL_MASK ((1 << __TC_ACT_EXT_SHIFT) - 1)
+#define TC_ACT_EXT_CMP(combined, opcode) \
+ (((combined) & (~TC_ACT_EXT_VAL_MASK)) == opcode)
+
+#define TC_ACT_JUMP __TC_ACT_EXT(1)
/* Action type identifiers*/
enum {
@@ -432,6 +445,11 @@
TCA_FLOWER_KEY_ARP_THA, /* ETH_ALEN */
TCA_FLOWER_KEY_ARP_THA_MASK, /* ETH_ALEN */
+ TCA_FLOWER_KEY_MPLS_TTL, /* u8 - 8 bits */
+ TCA_FLOWER_KEY_MPLS_BOS, /* u8 - 1 bit */
+ TCA_FLOWER_KEY_MPLS_TC, /* u8 - 3 bits */
+ TCA_FLOWER_KEY_MPLS_LABEL, /* be32 - 20 bits */
+
__TCA_FLOWER_MAX,
};
diff --git a/include/uapi/linux/pkt_sched.h b/include/uapi/linux/pkt_sched.h
index df7451d..099bf55 100644
--- a/include/uapi/linux/pkt_sched.h
+++ b/include/uapi/linux/pkt_sched.h
@@ -617,6 +617,14 @@
#define TC_QOPT_BITMASK 15
#define TC_QOPT_MAX_QUEUE 16
+enum {
+ TC_MQPRIO_HW_OFFLOAD_NONE, /* no offload requested */
+ TC_MQPRIO_HW_OFFLOAD_TCS, /* offload TCs, no queue counts */
+ __TC_MQPRIO_HW_OFFLOAD_MAX
+};
+
+#define TC_MQPRIO_HW_OFFLOAD_MAX (__TC_MQPRIO_HW_OFFLOAD_MAX - 1)
+
struct tc_mqprio_qopt {
__u8 num_tc;
__u8 prio_tc_map[TC_QOPT_BITMASK + 1];
diff --git a/include/uapi/linux/rtnetlink.h b/include/uapi/linux/rtnetlink.h
index 6546917..cce0613 100644
--- a/include/uapi/linux/rtnetlink.h
+++ b/include/uapi/linux/rtnetlink.h
@@ -122,6 +122,8 @@
RTM_NEWNETCONF = 80,
#define RTM_NEWNETCONF RTM_NEWNETCONF
+ RTM_DELNETCONF,
+#define RTM_DELNETCONF RTM_DELNETCONF
RTM_GETNETCONF = 82,
#define RTM_GETNETCONF RTM_GETNETCONF
@@ -319,6 +321,7 @@
RTA_EXPIRES,
RTA_PAD,
RTA_UID,
+ RTA_TTL_PROPAGATE,
__RTA_MAX
};
@@ -545,6 +548,7 @@
TCA_STATS2,
TCA_STAB,
TCA_PAD,
+ TCA_DUMP_INVISIBLE,
__TCA_MAX
};
diff --git a/include/uapi/linux/sctp.h b/include/uapi/linux/sctp.h
index d3ae381..ced9d8b 100644
--- a/include/uapi/linux/sctp.h
+++ b/include/uapi/linux/sctp.h
@@ -115,6 +115,8 @@
#define SCTP_PR_SUPPORTED 113
#define SCTP_DEFAULT_PRINFO 114
#define SCTP_PR_ASSOC_STATUS 115
+#define SCTP_PR_STREAM_STATUS 116
+#define SCTP_RECONFIG_SUPPORTED 117
#define SCTP_ENABLE_STREAM_RESET 118
#define SCTP_RESET_STREAMS 119
#define SCTP_RESET_ASSOC 120
@@ -502,6 +504,28 @@
__u16 strreset_stream_list[];
};
+#define SCTP_ASSOC_RESET_DENIED 0x0004
+#define SCTP_ASSOC_RESET_FAILED 0x0008
+struct sctp_assoc_reset_event {
+ __u16 assocreset_type;
+ __u16 assocreset_flags;
+ __u32 assocreset_length;
+ sctp_assoc_t assocreset_assoc_id;
+ __u32 assocreset_local_tsn;
+ __u32 assocreset_remote_tsn;
+};
+
+#define SCTP_ASSOC_CHANGE_DENIED 0x0004
+#define SCTP_ASSOC_CHANGE_FAILED 0x0008
+struct sctp_stream_change_event {
+ __u16 strchange_type;
+ __u16 strchange_flags;
+ __u32 strchange_length;
+ sctp_assoc_t strchange_assoc_id;
+ __u16 strchange_instrms;
+ __u16 strchange_outstrms;
+};
+
/*
* Described in Section 7.3
* Ancillary Data and Notification Interest Options
@@ -518,6 +542,8 @@
__u8 sctp_authentication_event;
__u8 sctp_sender_dry_event;
__u8 sctp_stream_reset_event;
+ __u8 sctp_assoc_reset_event;
+ __u8 sctp_stream_change_event;
};
/*
@@ -543,6 +569,8 @@
struct sctp_authkey_event sn_authkey_event;
struct sctp_sender_dry_event sn_sender_dry_event;
struct sctp_stream_reset_event sn_strreset_event;
+ struct sctp_assoc_reset_event sn_assocreset_event;
+ struct sctp_stream_change_event sn_strchange_event;
};
/* Section 5.3.1
@@ -572,6 +600,10 @@
#define SCTP_SENDER_DRY_EVENT SCTP_SENDER_DRY_EVENT
SCTP_STREAM_RESET_EVENT,
#define SCTP_STREAM_RESET_EVENT SCTP_STREAM_RESET_EVENT
+ SCTP_ASSOC_RESET_EVENT,
+#define SCTP_ASSOC_RESET_EVENT SCTP_ASSOC_RESET_EVENT
+ SCTP_STREAM_CHANGE_EVENT,
+#define SCTP_STREAM_CHANGE_EVENT SCTP_STREAM_CHANGE_EVENT
};
/* Notification error codes used to fill up the error fields in some
diff --git a/include/uapi/linux/snmp.h b/include/uapi/linux/snmp.h
index 3b2bed7..95cffcb 100644
--- a/include/uapi/linux/snmp.h
+++ b/include/uapi/linux/snmp.h
@@ -177,7 +177,6 @@
LINUX_MIB_TIMEWAITED, /* TimeWaited */
LINUX_MIB_TIMEWAITRECYCLED, /* TimeWaitRecycled */
LINUX_MIB_TIMEWAITKILLED, /* TimeWaitKilled */
- LINUX_MIB_PAWSPASSIVEREJECTED, /* PAWSPassiveRejected */
LINUX_MIB_PAWSACTIVEREJECTED, /* PAWSActiveRejected */
LINUX_MIB_PAWSESTABREJECTED, /* PAWSEstabRejected */
LINUX_MIB_DELAYEDACKS, /* DelayedACKs */
@@ -260,6 +259,7 @@
LINUX_MIB_TCPFASTOPENPASSIVEFAIL, /* TCPFastOpenPassiveFail */
LINUX_MIB_TCPFASTOPENLISTENOVERFLOW, /* TCPFastOpenListenOverflow */
LINUX_MIB_TCPFASTOPENCOOKIEREQD, /* TCPFastOpenCookieReqd */
+ LINUX_MIB_TCPFASTOPENBLACKHOLE, /* TCPFastOpenBlackholeDetect */
LINUX_MIB_TCPSPURIOUS_RTX_HOSTQUEUES, /* TCPSpuriousRtxHostQueues */
LINUX_MIB_BUSYPOLLRXPACKETS, /* BusyPollRxPackets */
LINUX_MIB_TCPAUTOCORKING, /* TCPAutoCorking */
diff --git a/include/uapi/linux/stat.h b/include/uapi/linux/stat.h
index d538897..17b1030 100644
--- a/include/uapi/linux/stat.h
+++ b/include/uapi/linux/stat.h
@@ -48,17 +48,13 @@
* tv_sec holds the number of seconds before (negative) or after (positive)
* 00:00:00 1st January 1970 UTC.
*
- * tv_nsec holds a number of nanoseconds before (0..-999,999,999 if tv_sec is
- * negative) or after (0..999,999,999 if tv_sec is positive) the tv_sec time.
- *
- * Note that if both tv_sec and tv_nsec are non-zero, then the two values must
- * either be both positive or both negative.
+ * tv_nsec holds a number of nanoseconds (0..999,999,999) after the tv_sec time.
*
* __reserved is held in case we need a yet finer resolution.
*/
struct statx_timestamp {
__s64 tv_sec;
- __s32 tv_nsec;
+ __u32 tv_nsec;
__s32 __reserved;
};
diff --git a/include/uapi/linux/sysctl.h b/include/uapi/linux/sysctl.h
index d2b1215..e13d480 100644
--- a/include/uapi/linux/sysctl.h
+++ b/include/uapi/linux/sysctl.h
@@ -568,6 +568,7 @@
NET_IPV6_PROXY_NDP=23,
NET_IPV6_ACCEPT_SOURCE_ROUTE=25,
NET_IPV6_ACCEPT_RA_FROM_LOCAL=26,
+ NET_IPV6_ACCEPT_RA_RT_INFO_MIN_PLEN=27,
__NET_IPV6_MAX
};
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 519eff3..ae46105 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -198,6 +198,7 @@
#define VFIO_DEVICE_FLAGS_PCI (1 << 1) /* vfio-pci device */
#define VFIO_DEVICE_FLAGS_PLATFORM (1 << 2) /* vfio-platform device */
#define VFIO_DEVICE_FLAGS_AMBA (1 << 3) /* vfio-amba device */
+#define VFIO_DEVICE_FLAGS_CCW (1 << 4) /* vfio-ccw device */
__u32 num_regions; /* Max region index + 1 */
__u32 num_irqs; /* Max IRQ index + 1 */
};
@@ -212,6 +213,7 @@
#define VFIO_DEVICE_API_PCI_STRING "vfio-pci"
#define VFIO_DEVICE_API_PLATFORM_STRING "vfio-platform"
#define VFIO_DEVICE_API_AMBA_STRING "vfio-amba"
+#define VFIO_DEVICE_API_CCW_STRING "vfio-ccw"
/**
* VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8,
@@ -446,6 +448,22 @@
VFIO_PCI_NUM_IRQS
};
+/*
+ * The vfio-ccw bus driver makes use of the following fixed region and
+ * IRQ index mapping. Unimplemented regions return a size of zero.
+ * Unimplemented IRQ types return a count of zero.
+ */
+
+enum {
+ VFIO_CCW_CONFIG_REGION_INDEX,
+ VFIO_CCW_NUM_REGIONS
+};
+
+enum {
+ VFIO_CCW_IO_IRQ_INDEX,
+ VFIO_CCW_NUM_IRQS
+};
+
/**
* VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IORW(VFIO_TYPE, VFIO_BASE + 12,
* struct vfio_pci_hot_reset_info)
diff --git a/include/uapi/linux/vfio_ccw.h b/include/uapi/linux/vfio_ccw.h
new file mode 100644
index 0000000..34a7f6f
--- /dev/null
+++ b/include/uapi/linux/vfio_ccw.h
@@ -0,0 +1,24 @@
+/*
+ * Interfaces for vfio-ccw
+ *
+ * Copyright IBM Corp. 2017
+ *
+ * Author(s): Dong Jia Shi <bjsdjshi@linux.vnet.ibm.com>
+ */
+
+#ifndef _VFIO_CCW_H_
+#define _VFIO_CCW_H_
+
+#include <linux/types.h>
+
+struct ccw_io_region {
+#define ORB_AREA_SIZE 12
+ __u8 orb_area[ORB_AREA_SIZE];
+#define SCSW_AREA_SIZE 12
+ __u8 scsw_area[SCSW_AREA_SIZE];
+#define IRB_AREA_SIZE 96
+ __u8 irb_area[IRB_AREA_SIZE];
+ __u32 ret_code;
+} __packed;
+
+#endif
diff --git a/include/uapi/linux/vsockmon.h b/include/uapi/linux/vsockmon.h
new file mode 100644
index 0000000..a08b522
--- /dev/null
+++ b/include/uapi/linux/vsockmon.h
@@ -0,0 +1,60 @@
+#ifndef _UAPI_VSOCKMON_H
+#define _UAPI_VSOCKMON_H
+
+#include <linux/virtio_vsock.h>
+
+/*
+ * vsockmon is the AF_VSOCK packet capture device. Packets captured have the
+ * following layout:
+ *
+ * +-----------------------------------+
+ * | vsockmon header |
+ * | (struct af_vsockmon_hdr) |
+ * +-----------------------------------+
+ * | transport header |
+ * | (af_vsockmon_hdr->len bytes long) |
+ * +-----------------------------------+
+ * | payload |
+ * | (until end of packet) |
+ * +-----------------------------------+
+ *
+ * The vsockmon header is a transport-independent description of the packet.
+ * It duplicates some of the information from the transport header so that
+ * no transport-specific knowledge is necessary to process packets.
+ *
+ * The transport header is useful for low-level transport-specific packet
+ * analysis. Transport type is given in af_vsockmon_hdr->transport and
+ * transport header length is given in af_vsockmon_hdr->len.
+ *
+ * If af_vsockmon_hdr->op is AF_VSOCK_OP_PAYLOAD then the payload follows the
+ * transport header. Other ops do not have a payload.
+ */
+
+struct af_vsockmon_hdr {
+ __le64 src_cid;
+ __le64 dst_cid;
+ __le32 src_port;
+ __le32 dst_port;
+ __le16 op; /* enum af_vsockmon_op */
+ __le16 transport; /* enum af_vsockmon_transport */
+ __le16 len; /* Transport header length */
+ __u8 reserved[2];
+};
+
+enum af_vsockmon_op {
+ AF_VSOCK_OP_UNKNOWN = 0,
+ AF_VSOCK_OP_CONNECT = 1,
+ AF_VSOCK_OP_DISCONNECT = 2,
+ AF_VSOCK_OP_CONTROL = 3,
+ AF_VSOCK_OP_PAYLOAD = 4,
+};
+
+enum af_vsockmon_transport {
+ AF_VSOCK_TRANSPORT_UNKNOWN = 0,
+ AF_VSOCK_TRANSPORT_NO_INFO = 1, /* No transport information */
+
+ /* Transport header type: struct virtio_vsock_hdr */
+ AF_VSOCK_TRANSPORT_VIRTIO = 2,
+};
+
+#endif
diff --git a/include/uapi/linux/xfrm.h b/include/uapi/linux/xfrm.h
index 1fc62b2..2b384ff 100644
--- a/include/uapi/linux/xfrm.h
+++ b/include/uapi/linux/xfrm.h
@@ -303,6 +303,7 @@
XFRMA_PROTO, /* __u8 */
XFRMA_ADDRESS_FILTER, /* struct xfrm_address_filter */
XFRMA_PAD,
+ XFRMA_OFFLOAD_DEV, /* struct xfrm_state_offload */
__XFRMA_MAX
#define XFRMA_MAX (__XFRMA_MAX - 1)
@@ -494,6 +495,13 @@
__u8 dplen;
};
+struct xfrm_user_offload {
+ int ifindex;
+ __u8 flags;
+};
+#define XFRM_OFFLOAD_IPV6 1
+#define XFRM_OFFLOAD_INBOUND 2
+
#ifndef __KERNEL__
/* backwards compatibility for userspace */
#define XFRMGRP_ACQUIRE 1
diff --git a/include/video/udlfb.h b/include/video/udlfb.h
index f9466fa..3ea90ae 100644
--- a/include/video/udlfb.h
+++ b/include/video/udlfb.h
@@ -92,6 +92,6 @@
/* remove these once align.h patch is taken into kernel */
#define DL_ALIGN_UP(x, a) ALIGN(x, a)
-#define DL_ALIGN_DOWN(x, a) ALIGN(x-(a-1), a)
+#define DL_ALIGN_DOWN(x, a) ALIGN_DOWN(x, a)
#endif
diff --git a/include/xen/page.h b/include/xen/page.h
index 9dc46cb..064194f 100644
--- a/include/xen/page.h
+++ b/include/xen/page.h
@@ -38,7 +38,7 @@
unsigned long n_pfns;
};
-#define XEN_EXTRA_MEM_MAX_REGIONS 128 /* == E820MAX */
+#define XEN_EXTRA_MEM_MAX_REGIONS 128 /* == E820_MAX_ENTRIES_ZEROPAGE */
extern __initdata
struct xen_memory_region xen_extra_mem[XEN_EXTRA_MEM_MAX_REGIONS];
diff --git a/init/main.c b/init/main.c
index b0c11cb..b1b9dbf 100644
--- a/init/main.c
+++ b/init/main.c
@@ -495,7 +495,7 @@
debug_objects_early_init();
/*
- * Set up the the initial canary ASAP:
+ * Set up the initial canary ASAP:
*/
boot_init_stack_canary();
@@ -504,10 +504,10 @@
local_irq_disable();
early_boot_irqs_disabled = true;
-/*
- * Interrupts are still disabled. Do necessary setups, then
- * enable them
- */
+ /*
+ * Interrupts are still disabled. Do necessary setups, then
+ * enable them.
+ */
boot_cpu_init();
page_address_init();
pr_notice("%s", linux_banner);
diff --git a/ipc/util.c b/ipc/util.c
index 798cad1..3459a16 100644
--- a/ipc/util.c
+++ b/ipc/util.c
@@ -474,7 +474,7 @@
* Check user, group, other permissions for access
* to ipc resources. return 0 if allowed
*
- * @flag will most probably be 0 or S_...UGO from <linux/stat.h>
+ * @flag will most probably be 0 or ``S_...UGO`` from <linux/stat.h>
*/
int ipcperms(struct ipc_namespace *ns, struct kern_ipc_perm *ipcp, short flag)
{
@@ -672,10 +672,12 @@
*
* This function does some common audit and permissions check for some IPC_XXX
* cmd and is called from semctl_down, shmctl_down and msgctl_down.
- * It must be called without any lock held and
- * - retrieves the ipc with the given id in the given table.
- * - performs some audit and permission check, depending on the given cmd
- * - returns a pointer to the ipc object or otherwise, the corresponding error.
+ * It must be called without any lock held and:
+ *
+ * - retrieves the ipc with the given id in the given table.
+ * - performs some audit and permission check, depending on the given cmd
+ * - returns a pointer to the ipc object or otherwise, the corresponding
+ * error.
*
* Call holding the both the rwsem and the rcu read lock.
*/
diff --git a/kernel/audit.c b/kernel/audit.c
index a871bf8..dc202d5 100644
--- a/kernel/audit.c
+++ b/kernel/audit.c
@@ -1399,7 +1399,7 @@
err = audit_receive_msg(skb, nlh);
/* if err or if this message says it wants a response */
if (err || (nlh->nlmsg_flags & NLM_F_ACK))
- netlink_ack(skb, nlh, err);
+ netlink_ack(skb, nlh, err, NULL);
nlh = nlmsg_next(nlh, &len);
}
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index e1ce4f4..e1e5e65 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -1,7 +1,7 @@
obj-y := core.o
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o
-obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o
+obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o
ifeq ($(CONFIG_PERF_EVENTS),y)
obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
endif
diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
index 6b6f41f..5e00b23 100644
--- a/kernel/bpf/arraymap.c
+++ b/kernel/bpf/arraymap.c
@@ -1,4 +1,5 @@
/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
+ * Copyright (c) 2016,2017 Facebook
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of version 2 of the GNU General Public
@@ -16,6 +17,8 @@
#include <linux/filter.h>
#include <linux/perf_event.h>
+#include "map_in_map.h"
+
static void bpf_array_free_percpu(struct bpf_array *array)
{
int i;
@@ -113,6 +116,30 @@
return array->value + array->elem_size * index;
}
+/* emit BPF instructions equivalent to C code of array_map_lookup_elem() */
+static u32 array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
+{
+ struct bpf_insn *insn = insn_buf;
+ u32 elem_size = round_up(map->value_size, 8);
+ const int ret = BPF_REG_0;
+ const int map_ptr = BPF_REG_1;
+ const int index = BPF_REG_2;
+
+ *insn++ = BPF_ALU64_IMM(BPF_ADD, map_ptr, offsetof(struct bpf_array, value));
+ *insn++ = BPF_LDX_MEM(BPF_W, ret, index, 0);
+ *insn++ = BPF_JMP_IMM(BPF_JGE, ret, map->max_entries, 3);
+
+ if (is_power_of_2(elem_size)) {
+ *insn++ = BPF_ALU64_IMM(BPF_LSH, ret, ilog2(elem_size));
+ } else {
+ *insn++ = BPF_ALU64_IMM(BPF_MUL, ret, elem_size);
+ }
+ *insn++ = BPF_ALU64_REG(BPF_ADD, ret, map_ptr);
+ *insn++ = BPF_JMP_IMM(BPF_JA, 0, 0, 1);
+ *insn++ = BPF_MOV64_IMM(ret, 0);
+ return insn - insn_buf;
+}
+
/* Called from eBPF program */
static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)
{
@@ -155,7 +182,7 @@
static int array_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
- u32 index = *(u32 *)key;
+ u32 index = key ? *(u32 *)key : U32_MAX;
u32 *next = (u32 *)next_key;
if (index >= array->map.max_entries) {
@@ -260,21 +287,17 @@
bpf_map_area_free(array);
}
-static const struct bpf_map_ops array_ops = {
+const struct bpf_map_ops array_map_ops = {
.map_alloc = array_map_alloc,
.map_free = array_map_free,
.map_get_next_key = array_map_get_next_key,
.map_lookup_elem = array_map_lookup_elem,
.map_update_elem = array_map_update_elem,
.map_delete_elem = array_map_delete_elem,
+ .map_gen_lookup = array_map_gen_lookup,
};
-static struct bpf_map_type_list array_type __ro_after_init = {
- .ops = &array_ops,
- .type = BPF_MAP_TYPE_ARRAY,
-};
-
-static const struct bpf_map_ops percpu_array_ops = {
+const struct bpf_map_ops percpu_array_map_ops = {
.map_alloc = array_map_alloc,
.map_free = array_map_free,
.map_get_next_key = array_map_get_next_key,
@@ -283,19 +306,6 @@
.map_delete_elem = array_map_delete_elem,
};
-static struct bpf_map_type_list percpu_array_type __ro_after_init = {
- .ops = &percpu_array_ops,
- .type = BPF_MAP_TYPE_PERCPU_ARRAY,
-};
-
-static int __init register_array_map(void)
-{
- bpf_register_map_type(&array_type);
- bpf_register_map_type(&percpu_array_type);
- return 0;
-}
-late_initcall(register_array_map);
-
static struct bpf_map *fd_array_map_alloc(union bpf_attr *attr)
{
/* only file descriptors can be stored in this type of map */
@@ -399,7 +409,7 @@
fd_array_map_delete_elem(map, &i);
}
-static const struct bpf_map_ops prog_array_ops = {
+const struct bpf_map_ops prog_array_map_ops = {
.map_alloc = fd_array_map_alloc,
.map_free = fd_array_map_free,
.map_get_next_key = array_map_get_next_key,
@@ -409,18 +419,6 @@
.map_fd_put_ptr = prog_fd_array_put_ptr,
};
-static struct bpf_map_type_list prog_array_type __ro_after_init = {
- .ops = &prog_array_ops,
- .type = BPF_MAP_TYPE_PROG_ARRAY,
-};
-
-static int __init register_prog_array_map(void)
-{
- bpf_register_map_type(&prog_array_type);
- return 0;
-}
-late_initcall(register_prog_array_map);
-
static struct bpf_event_entry *bpf_event_entry_gen(struct file *perf_file,
struct file *map_file)
{
@@ -511,7 +509,7 @@
rcu_read_unlock();
}
-static const struct bpf_map_ops perf_event_array_ops = {
+const struct bpf_map_ops perf_event_array_map_ops = {
.map_alloc = fd_array_map_alloc,
.map_free = fd_array_map_free,
.map_get_next_key = array_map_get_next_key,
@@ -522,18 +520,6 @@
.map_release = perf_event_fd_array_release,
};
-static struct bpf_map_type_list perf_event_array_type __ro_after_init = {
- .ops = &perf_event_array_ops,
- .type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
-};
-
-static int __init register_perf_event_array_map(void)
-{
- bpf_register_map_type(&perf_event_array_type);
- return 0;
-}
-late_initcall(register_perf_event_array_map);
-
#ifdef CONFIG_CGROUPS
static void *cgroup_fd_array_get_ptr(struct bpf_map *map,
struct file *map_file /* not used */,
@@ -554,7 +540,7 @@
fd_array_map_free(map);
}
-static const struct bpf_map_ops cgroup_array_ops = {
+const struct bpf_map_ops cgroup_array_map_ops = {
.map_alloc = fd_array_map_alloc,
.map_free = cgroup_fd_array_free,
.map_get_next_key = array_map_get_next_key,
@@ -563,16 +549,53 @@
.map_fd_get_ptr = cgroup_fd_array_get_ptr,
.map_fd_put_ptr = cgroup_fd_array_put_ptr,
};
-
-static struct bpf_map_type_list cgroup_array_type __ro_after_init = {
- .ops = &cgroup_array_ops,
- .type = BPF_MAP_TYPE_CGROUP_ARRAY,
-};
-
-static int __init register_cgroup_array_map(void)
-{
- bpf_register_map_type(&cgroup_array_type);
- return 0;
-}
-late_initcall(register_cgroup_array_map);
#endif
+
+static struct bpf_map *array_of_map_alloc(union bpf_attr *attr)
+{
+ struct bpf_map *map, *inner_map_meta;
+
+ inner_map_meta = bpf_map_meta_alloc(attr->inner_map_fd);
+ if (IS_ERR(inner_map_meta))
+ return inner_map_meta;
+
+ map = fd_array_map_alloc(attr);
+ if (IS_ERR(map)) {
+ bpf_map_meta_free(inner_map_meta);
+ return map;
+ }
+
+ map->inner_map_meta = inner_map_meta;
+
+ return map;
+}
+
+static void array_of_map_free(struct bpf_map *map)
+{
+ /* map->inner_map_meta is only accessed by syscall which
+ * is protected by fdget/fdput.
+ */
+ bpf_map_meta_free(map->inner_map_meta);
+ bpf_fd_array_map_clear(map);
+ fd_array_map_free(map);
+}
+
+static void *array_of_map_lookup_elem(struct bpf_map *map, void *key)
+{
+ struct bpf_map **inner_map = array_map_lookup_elem(map, key);
+
+ if (!inner_map)
+ return NULL;
+
+ return READ_ONCE(*inner_map);
+}
+
+const struct bpf_map_ops array_of_maps_map_ops = {
+ .map_alloc = array_of_map_alloc,
+ .map_free = array_of_map_free,
+ .map_get_next_key = array_map_get_next_key,
+ .map_lookup_elem = array_of_map_lookup_elem,
+ .map_delete_elem = fd_array_map_delete_elem,
+ .map_fd_get_ptr = bpf_map_fd_get_ptr,
+ .map_fd_put_ptr = bpf_map_fd_put_ptr,
+};
diff --git a/kernel/bpf/bpf_lru_list.c b/kernel/bpf/bpf_lru_list.c
index f62d1d56..e6ef440 100644
--- a/kernel/bpf/bpf_lru_list.c
+++ b/kernel/bpf/bpf_lru_list.c
@@ -13,7 +13,7 @@
#define LOCAL_FREE_TARGET (128)
#define LOCAL_NR_SCANS LOCAL_FREE_TARGET
-#define PERCPU_FREE_TARGET (16)
+#define PERCPU_FREE_TARGET (4)
#define PERCPU_NR_SCANS PERCPU_FREE_TARGET
/* Helpers to get the local list index */
diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c
index da0f536..ea6033c 100644
--- a/kernel/bpf/cgroup.c
+++ b/kernel/bpf/cgroup.c
@@ -154,7 +154,7 @@
/**
* __cgroup_bpf_run_filter_skb() - Run a program for packet filtering
- * @sk: The socken sending or receiving traffic
+ * @sk: The socket sending or receiving traffic
* @skb: The skb that is being sent or received
* @type: The type of program to be exectuted
*
@@ -189,10 +189,13 @@
prog = rcu_dereference(cgrp->bpf.effective[type]);
if (prog) {
unsigned int offset = skb->data - skb_network_header(skb);
+ struct sock *save_sk = skb->sk;
+ skb->sk = sk;
__skb_push(skb, offset);
ret = bpf_prog_run_save_cb(prog, skb) == 1 ? 0 : -EPERM;
__skb_pull(skb, offset);
+ skb->sk = save_sk;
}
rcu_read_unlock();
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index b4f1cb0..6f81e0f 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -394,27 +394,23 @@
void bpf_prog_kallsyms_add(struct bpf_prog *fp)
{
- unsigned long flags;
-
if (!bpf_prog_kallsyms_candidate(fp) ||
!capable(CAP_SYS_ADMIN))
return;
- spin_lock_irqsave(&bpf_lock, flags);
+ spin_lock_bh(&bpf_lock);
bpf_prog_ksym_node_add(fp->aux);
- spin_unlock_irqrestore(&bpf_lock, flags);
+ spin_unlock_bh(&bpf_lock);
}
void bpf_prog_kallsyms_del(struct bpf_prog *fp)
{
- unsigned long flags;
-
if (!bpf_prog_kallsyms_candidate(fp))
return;
- spin_lock_irqsave(&bpf_lock, flags);
+ spin_lock_bh(&bpf_lock);
bpf_prog_ksym_node_del(fp->aux);
- spin_unlock_irqrestore(&bpf_lock, flags);
+ spin_unlock_bh(&bpf_lock);
}
static struct bpf_prog *bpf_prog_kallsyms_find(unsigned long addr)
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 361a69d..004334e 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -16,6 +16,7 @@
#include <linux/rculist_nulls.h>
#include "percpu_freelist.h"
#include "bpf_lru_list.h"
+#include "map_in_map.h"
struct bucket {
struct hlist_nulls_head head;
@@ -86,6 +87,11 @@
return *(void __percpu **)(l->key + key_size);
}
+static void *fd_htab_map_get_ptr(const struct bpf_map *map, struct htab_elem *l)
+{
+ return *(void **)(l->key + roundup(map->key_size, 8));
+}
+
static struct htab_elem *get_htab_elem(struct bpf_htab *htab, int i)
{
return (struct htab_elem *) (htab->elems + i * htab->elem_size);
@@ -426,7 +432,11 @@
return NULL;
}
-/* Called from syscall or from eBPF program */
+/* Called from syscall or from eBPF program directly, so
+ * arguments have to match bpf_map_lookup_elem() exactly.
+ * The return value is adjusted by BPF instructions
+ * in htab_map_gen_lookup().
+ */
static void *__htab_map_lookup_elem(struct bpf_map *map, void *key)
{
struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
@@ -458,6 +468,30 @@
return NULL;
}
+/* inline bpf_map_lookup_elem() call.
+ * Instead of:
+ * bpf_prog
+ * bpf_map_lookup_elem
+ * map->ops->map_lookup_elem
+ * htab_map_lookup_elem
+ * __htab_map_lookup_elem
+ * do:
+ * bpf_prog
+ * __htab_map_lookup_elem
+ */
+static u32 htab_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
+{
+ struct bpf_insn *insn = insn_buf;
+ const int ret = BPF_REG_0;
+
+ *insn++ = BPF_EMIT_CALL((u64 (*)(u64, u64, u64, u64, u64))__htab_map_lookup_elem);
+ *insn++ = BPF_JMP_IMM(BPF_JEQ, ret, 0, 1);
+ *insn++ = BPF_ALU64_IMM(BPF_ADD, ret,
+ offsetof(struct htab_elem, key) +
+ round_up(map->key_size, 8));
+ return insn - insn_buf;
+}
+
static void *htab_lru_map_lookup_elem(struct bpf_map *map, void *key)
{
struct htab_elem *l = __htab_map_lookup_elem(map, key);
@@ -506,12 +540,15 @@
struct hlist_nulls_head *head;
struct htab_elem *l, *next_l;
u32 hash, key_size;
- int i;
+ int i = 0;
WARN_ON_ONCE(!rcu_read_lock_held());
key_size = map->key_size;
+ if (!key)
+ goto find_first_elem;
+
hash = htab_map_hash(key, key_size);
head = select_bucket(htab, hash);
@@ -519,10 +556,8 @@
/* lookup the key */
l = lookup_nulls_elem_raw(head, hash, key, key_size, htab->n_buckets);
- if (!l) {
- i = 0;
+ if (!l)
goto find_first_elem;
- }
/* key was found, get next key in the same bucket */
next_l = hlist_nulls_entry_safe(rcu_dereference_raw(hlist_nulls_next_rcu(&l->hash_node)),
@@ -582,6 +617,14 @@
static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
{
+ struct bpf_map *map = &htab->map;
+
+ if (map->ops->map_fd_put_ptr) {
+ void *ptr = fd_htab_map_get_ptr(map, l);
+
+ map->ops->map_fd_put_ptr(ptr);
+ }
+
if (htab_is_prealloc(htab)) {
pcpu_freelist_push(&htab->freelist, &l->fnode);
} else {
@@ -1027,6 +1070,7 @@
}
}
}
+
/* Called when map->refcnt goes to zero, either from workqueue or from syscall */
static void htab_map_free(struct bpf_map *map)
{
@@ -1053,21 +1097,17 @@
kfree(htab);
}
-static const struct bpf_map_ops htab_ops = {
+const struct bpf_map_ops htab_map_ops = {
.map_alloc = htab_map_alloc,
.map_free = htab_map_free,
.map_get_next_key = htab_map_get_next_key,
.map_lookup_elem = htab_map_lookup_elem,
.map_update_elem = htab_map_update_elem,
.map_delete_elem = htab_map_delete_elem,
+ .map_gen_lookup = htab_map_gen_lookup,
};
-static struct bpf_map_type_list htab_type __ro_after_init = {
- .ops = &htab_ops,
- .type = BPF_MAP_TYPE_HASH,
-};
-
-static const struct bpf_map_ops htab_lru_ops = {
+const struct bpf_map_ops htab_lru_map_ops = {
.map_alloc = htab_map_alloc,
.map_free = htab_map_free,
.map_get_next_key = htab_map_get_next_key,
@@ -1076,11 +1116,6 @@
.map_delete_elem = htab_lru_map_delete_elem,
};
-static struct bpf_map_type_list htab_lru_type __ro_after_init = {
- .ops = &htab_lru_ops,
- .type = BPF_MAP_TYPE_LRU_HASH,
-};
-
/* Called from eBPF program */
static void *htab_percpu_map_lookup_elem(struct bpf_map *map, void *key)
{
@@ -1154,7 +1189,7 @@
return ret;
}
-static const struct bpf_map_ops htab_percpu_ops = {
+const struct bpf_map_ops htab_percpu_map_ops = {
.map_alloc = htab_map_alloc,
.map_free = htab_map_free,
.map_get_next_key = htab_map_get_next_key,
@@ -1163,12 +1198,7 @@
.map_delete_elem = htab_map_delete_elem,
};
-static struct bpf_map_type_list htab_percpu_type __ro_after_init = {
- .ops = &htab_percpu_ops,
- .type = BPF_MAP_TYPE_PERCPU_HASH,
-};
-
-static const struct bpf_map_ops htab_lru_percpu_ops = {
+const struct bpf_map_ops htab_lru_percpu_map_ops = {
.map_alloc = htab_map_alloc,
.map_free = htab_map_free,
.map_get_next_key = htab_map_get_next_key,
@@ -1177,17 +1207,102 @@
.map_delete_elem = htab_lru_map_delete_elem,
};
-static struct bpf_map_type_list htab_lru_percpu_type __ro_after_init = {
- .ops = &htab_lru_percpu_ops,
- .type = BPF_MAP_TYPE_LRU_PERCPU_HASH,
-};
-
-static int __init register_htab_map(void)
+static struct bpf_map *fd_htab_map_alloc(union bpf_attr *attr)
{
- bpf_register_map_type(&htab_type);
- bpf_register_map_type(&htab_percpu_type);
- bpf_register_map_type(&htab_lru_type);
- bpf_register_map_type(&htab_lru_percpu_type);
- return 0;
+ struct bpf_map *map;
+
+ if (attr->value_size != sizeof(u32))
+ return ERR_PTR(-EINVAL);
+
+ /* pointer is stored internally */
+ attr->value_size = sizeof(void *);
+ map = htab_map_alloc(attr);
+ attr->value_size = sizeof(u32);
+
+ return map;
}
-late_initcall(register_htab_map);
+
+static void fd_htab_map_free(struct bpf_map *map)
+{
+ struct bpf_htab *htab = container_of(map, struct bpf_htab, map);
+ struct hlist_nulls_node *n;
+ struct hlist_nulls_head *head;
+ struct htab_elem *l;
+ int i;
+
+ for (i = 0; i < htab->n_buckets; i++) {
+ head = select_bucket(htab, i);
+
+ hlist_nulls_for_each_entry_safe(l, n, head, hash_node) {
+ void *ptr = fd_htab_map_get_ptr(map, l);
+
+ map->ops->map_fd_put_ptr(ptr);
+ }
+ }
+
+ htab_map_free(map);
+}
+
+/* only called from syscall */
+int bpf_fd_htab_map_update_elem(struct bpf_map *map, struct file *map_file,
+ void *key, void *value, u64 map_flags)
+{
+ void *ptr;
+ int ret;
+ u32 ufd = *(u32 *)value;
+
+ ptr = map->ops->map_fd_get_ptr(map, map_file, ufd);
+ if (IS_ERR(ptr))
+ return PTR_ERR(ptr);
+
+ ret = htab_map_update_elem(map, key, &ptr, map_flags);
+ if (ret)
+ map->ops->map_fd_put_ptr(ptr);
+
+ return ret;
+}
+
+static struct bpf_map *htab_of_map_alloc(union bpf_attr *attr)
+{
+ struct bpf_map *map, *inner_map_meta;
+
+ inner_map_meta = bpf_map_meta_alloc(attr->inner_map_fd);
+ if (IS_ERR(inner_map_meta))
+ return inner_map_meta;
+
+ map = fd_htab_map_alloc(attr);
+ if (IS_ERR(map)) {
+ bpf_map_meta_free(inner_map_meta);
+ return map;
+ }
+
+ map->inner_map_meta = inner_map_meta;
+
+ return map;
+}
+
+static void *htab_of_map_lookup_elem(struct bpf_map *map, void *key)
+{
+ struct bpf_map **inner_map = htab_map_lookup_elem(map, key);
+
+ if (!inner_map)
+ return NULL;
+
+ return READ_ONCE(*inner_map);
+}
+
+static void htab_of_map_free(struct bpf_map *map)
+{
+ bpf_map_meta_free(map->inner_map_meta);
+ fd_htab_map_free(map);
+}
+
+const struct bpf_map_ops htab_of_maps_map_ops = {
+ .map_alloc = htab_of_map_alloc,
+ .map_free = htab_of_map_free,
+ .map_get_next_key = htab_map_get_next_key,
+ .map_lookup_elem = htab_of_map_lookup_elem,
+ .map_delete_elem = htab_map_delete_elem,
+ .map_fd_get_ptr = bpf_map_fd_get_ptr,
+ .map_fd_put_ptr = bpf_map_fd_put_ptr,
+};
diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c
index b37bd9a..39cfafd 100644
--- a/kernel/bpf/lpm_trie.c
+++ b/kernel/bpf/lpm_trie.c
@@ -505,7 +505,7 @@
return -ENOTSUPP;
}
-static const struct bpf_map_ops trie_ops = {
+const struct bpf_map_ops trie_map_ops = {
.map_alloc = trie_alloc,
.map_free = trie_free,
.map_get_next_key = trie_get_next_key,
@@ -513,15 +513,3 @@
.map_update_elem = trie_update_elem,
.map_delete_elem = trie_delete_elem,
};
-
-static struct bpf_map_type_list trie_type __ro_after_init = {
- .ops = &trie_ops,
- .type = BPF_MAP_TYPE_LPM_TRIE,
-};
-
-static int __init register_trie_map(void)
-{
- bpf_register_map_type(&trie_type);
- return 0;
-}
-late_initcall(register_trie_map);
diff --git a/kernel/bpf/map_in_map.c b/kernel/bpf/map_in_map.c
new file mode 100644
index 0000000..59bcdf8
--- /dev/null
+++ b/kernel/bpf/map_in_map.c
@@ -0,0 +1,97 @@
+/* Copyright (c) 2017 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <linux/slab.h>
+#include <linux/bpf.h>
+
+#include "map_in_map.h"
+
+struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)
+{
+ struct bpf_map *inner_map, *inner_map_meta;
+ struct fd f;
+
+ f = fdget(inner_map_ufd);
+ inner_map = __bpf_map_get(f);
+ if (IS_ERR(inner_map))
+ return inner_map;
+
+ /* prog_array->owner_prog_type and owner_jited
+ * is a runtime binding. Doing static check alone
+ * in the verifier is not enough.
+ */
+ if (inner_map->map_type == BPF_MAP_TYPE_PROG_ARRAY) {
+ fdput(f);
+ return ERR_PTR(-ENOTSUPP);
+ }
+
+ /* Does not support >1 level map-in-map */
+ if (inner_map->inner_map_meta) {
+ fdput(f);
+ return ERR_PTR(-EINVAL);
+ }
+
+ inner_map_meta = kzalloc(sizeof(*inner_map_meta), GFP_USER);
+ if (!inner_map_meta) {
+ fdput(f);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ inner_map_meta->map_type = inner_map->map_type;
+ inner_map_meta->key_size = inner_map->key_size;
+ inner_map_meta->value_size = inner_map->value_size;
+ inner_map_meta->map_flags = inner_map->map_flags;
+ inner_map_meta->ops = inner_map->ops;
+ inner_map_meta->max_entries = inner_map->max_entries;
+
+ fdput(f);
+ return inner_map_meta;
+}
+
+void bpf_map_meta_free(struct bpf_map *map_meta)
+{
+ kfree(map_meta);
+}
+
+bool bpf_map_meta_equal(const struct bpf_map *meta0,
+ const struct bpf_map *meta1)
+{
+ /* No need to compare ops because it is covered by map_type */
+ return meta0->map_type == meta1->map_type &&
+ meta0->key_size == meta1->key_size &&
+ meta0->value_size == meta1->value_size &&
+ meta0->map_flags == meta1->map_flags &&
+ meta0->max_entries == meta1->max_entries;
+}
+
+void *bpf_map_fd_get_ptr(struct bpf_map *map,
+ struct file *map_file /* not used */,
+ int ufd)
+{
+ struct bpf_map *inner_map;
+ struct fd f;
+
+ f = fdget(ufd);
+ inner_map = __bpf_map_get(f);
+ if (IS_ERR(inner_map))
+ return inner_map;
+
+ if (bpf_map_meta_equal(map->inner_map_meta, inner_map))
+ inner_map = bpf_map_inc(inner_map, false);
+ else
+ inner_map = ERR_PTR(-EINVAL);
+
+ fdput(f);
+ return inner_map;
+}
+
+void bpf_map_fd_put_ptr(void *ptr)
+{
+ /* ptr->ops->map_free() has to go through one
+ * rcu grace period by itself.
+ */
+ bpf_map_put(ptr);
+}
diff --git a/kernel/bpf/map_in_map.h b/kernel/bpf/map_in_map.h
new file mode 100644
index 0000000..177fadb
--- /dev/null
+++ b/kernel/bpf/map_in_map.h
@@ -0,0 +1,23 @@
+/* Copyright (c) 2017 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#ifndef __MAP_IN_MAP_H__
+#define __MAP_IN_MAP_H__
+
+#include <linux/types.h>
+
+struct file;
+struct bpf_map;
+
+struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd);
+void bpf_map_meta_free(struct bpf_map *map_meta);
+bool bpf_map_meta_equal(const struct bpf_map *meta0,
+ const struct bpf_map *meta1);
+void *bpf_map_fd_get_ptr(struct bpf_map *map, struct file *map_file,
+ int ufd);
+void bpf_map_fd_put_ptr(void *ptr);
+
+#endif
diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c
index 22aa45c..4dfd6f2 100644
--- a/kernel/bpf/stackmap.c
+++ b/kernel/bpf/stackmap.c
@@ -264,7 +264,7 @@
put_callchain_buffers();
}
-static const struct bpf_map_ops stack_map_ops = {
+const struct bpf_map_ops stack_map_ops = {
.map_alloc = stack_map_alloc,
.map_free = stack_map_free,
.map_get_next_key = stack_map_get_next_key,
@@ -272,15 +272,3 @@
.map_update_elem = stack_map_update_elem,
.map_delete_elem = stack_map_delete_elem,
};
-
-static struct bpf_map_type_list stack_map_type __ro_after_init = {
- .ops = &stack_map_ops,
- .type = BPF_MAP_TYPE_STACK_TRACE,
-};
-
-static int __init register_stack_map(void)
-{
- bpf_register_map_type(&stack_map_type);
- return 0;
-}
-late_initcall(register_stack_map);
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 821f9e8..13642c7 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -27,30 +27,29 @@
int sysctl_unprivileged_bpf_disabled __read_mostly;
-static LIST_HEAD(bpf_map_types);
+static const struct bpf_map_ops * const bpf_map_types[] = {
+#define BPF_PROG_TYPE(_id, _ops)
+#define BPF_MAP_TYPE(_id, _ops) \
+ [_id] = &_ops,
+#include <linux/bpf_types.h>
+#undef BPF_PROG_TYPE
+#undef BPF_MAP_TYPE
+};
static struct bpf_map *find_and_alloc_map(union bpf_attr *attr)
{
- struct bpf_map_type_list *tl;
struct bpf_map *map;
- list_for_each_entry(tl, &bpf_map_types, list_node) {
- if (tl->type == attr->map_type) {
- map = tl->ops->map_alloc(attr);
- if (IS_ERR(map))
- return map;
- map->ops = tl->ops;
- map->map_type = attr->map_type;
- return map;
- }
- }
- return ERR_PTR(-EINVAL);
-}
+ if (attr->map_type >= ARRAY_SIZE(bpf_map_types) ||
+ !bpf_map_types[attr->map_type])
+ return ERR_PTR(-EINVAL);
-/* boot time registration of different map implementations */
-void bpf_register_map_type(struct bpf_map_type_list *tl)
-{
- list_add(&tl->list_node, &bpf_map_types);
+ map = bpf_map_types[attr->map_type]->map_alloc(attr);
+ if (IS_ERR(map))
+ return map;
+ map->ops = bpf_map_types[attr->map_type];
+ map->map_type = attr->map_type;
+ return map;
}
void *bpf_map_area_alloc(size_t size)
@@ -215,7 +214,7 @@
offsetof(union bpf_attr, CMD##_LAST_FIELD) - \
sizeof(attr->CMD##_LAST_FIELD)) != NULL
-#define BPF_MAP_CREATE_LAST_FIELD map_flags
+#define BPF_MAP_CREATE_LAST_FIELD inner_map_fd
/* called via syscall */
static int map_create(union bpf_attr *attr)
{
@@ -352,6 +351,9 @@
err = bpf_percpu_array_copy(map, key, value);
} else if (map->map_type == BPF_MAP_TYPE_STACK_TRACE) {
err = bpf_stackmap_copy(map, key, value);
+ } else if (map->map_type == BPF_MAP_TYPE_ARRAY_OF_MAPS ||
+ map->map_type == BPF_MAP_TYPE_HASH_OF_MAPS) {
+ err = -ENOTSUPP;
} else {
rcu_read_lock();
ptr = map->ops->map_lookup_elem(map, key);
@@ -438,11 +440,17 @@
err = bpf_percpu_array_update(map, key, value, attr->flags);
} else if (map->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY ||
map->map_type == BPF_MAP_TYPE_PROG_ARRAY ||
- map->map_type == BPF_MAP_TYPE_CGROUP_ARRAY) {
+ map->map_type == BPF_MAP_TYPE_CGROUP_ARRAY ||
+ map->map_type == BPF_MAP_TYPE_ARRAY_OF_MAPS) {
rcu_read_lock();
err = bpf_fd_array_map_update_elem(map, f.file, key, value,
attr->flags);
rcu_read_unlock();
+ } else if (map->map_type == BPF_MAP_TYPE_HASH_OF_MAPS) {
+ rcu_read_lock();
+ err = bpf_fd_htab_map_update_elem(map, f.file, key, value,
+ attr->flags);
+ rcu_read_unlock();
} else {
rcu_read_lock();
err = map->ops->map_update_elem(map, key, value, attr->flags);
@@ -528,14 +536,18 @@
if (IS_ERR(map))
return PTR_ERR(map);
- err = -ENOMEM;
- key = kmalloc(map->key_size, GFP_USER);
- if (!key)
- goto err_put;
+ if (ukey) {
+ err = -ENOMEM;
+ key = kmalloc(map->key_size, GFP_USER);
+ if (!key)
+ goto err_put;
- err = -EFAULT;
- if (copy_from_user(key, ukey, map->key_size) != 0)
- goto free_key;
+ err = -EFAULT;
+ if (copy_from_user(key, ukey, map->key_size) != 0)
+ goto free_key;
+ } else {
+ key = NULL;
+ }
err = -ENOMEM;
next_key = kmalloc(map->key_size, GFP_USER);
@@ -564,87 +576,23 @@
return err;
}
-static LIST_HEAD(bpf_prog_types);
+static const struct bpf_verifier_ops * const bpf_prog_types[] = {
+#define BPF_PROG_TYPE(_id, _ops) \
+ [_id] = &_ops,
+#define BPF_MAP_TYPE(_id, _ops)
+#include <linux/bpf_types.h>
+#undef BPF_PROG_TYPE
+#undef BPF_MAP_TYPE
+};
static int find_prog_type(enum bpf_prog_type type, struct bpf_prog *prog)
{
- struct bpf_prog_type_list *tl;
+ if (type >= ARRAY_SIZE(bpf_prog_types) || !bpf_prog_types[type])
+ return -EINVAL;
- list_for_each_entry(tl, &bpf_prog_types, list_node) {
- if (tl->type == type) {
- prog->aux->ops = tl->ops;
- prog->type = type;
- return 0;
- }
- }
-
- return -EINVAL;
-}
-
-void bpf_register_prog_type(struct bpf_prog_type_list *tl)
-{
- list_add(&tl->list_node, &bpf_prog_types);
-}
-
-/* fixup insn->imm field of bpf_call instructions:
- * if (insn->imm == BPF_FUNC_map_lookup_elem)
- * insn->imm = bpf_map_lookup_elem - __bpf_call_base;
- * else if (insn->imm == BPF_FUNC_map_update_elem)
- * insn->imm = bpf_map_update_elem - __bpf_call_base;
- * else ...
- *
- * this function is called after eBPF program passed verification
- */
-static void fixup_bpf_calls(struct bpf_prog *prog)
-{
- const struct bpf_func_proto *fn;
- int i;
-
- for (i = 0; i < prog->len; i++) {
- struct bpf_insn *insn = &prog->insnsi[i];
-
- if (insn->code == (BPF_JMP | BPF_CALL)) {
- /* we reach here when program has bpf_call instructions
- * and it passed bpf_check(), means that
- * ops->get_func_proto must have been supplied, check it
- */
- BUG_ON(!prog->aux->ops->get_func_proto);
-
- if (insn->imm == BPF_FUNC_get_route_realm)
- prog->dst_needed = 1;
- if (insn->imm == BPF_FUNC_get_prandom_u32)
- bpf_user_rnd_init_once();
- if (insn->imm == BPF_FUNC_xdp_adjust_head)
- prog->xdp_adjust_head = 1;
- if (insn->imm == BPF_FUNC_tail_call) {
- /* If we tail call into other programs, we
- * cannot make any assumptions since they
- * can be replaced dynamically during runtime
- * in the program array.
- */
- prog->cb_access = 1;
- prog->xdp_adjust_head = 1;
-
- /* mark bpf_tail_call as different opcode
- * to avoid conditional branch in
- * interpeter for every normal call
- * and to prevent accidental JITing by
- * JIT compiler that doesn't support
- * bpf_tail_call yet
- */
- insn->imm = 0;
- insn->code |= BPF_X;
- continue;
- }
-
- fn = prog->aux->ops->get_func_proto(insn->imm);
- /* all functions that have prototype and verifier allowed
- * programs to call them, must be real in-kernel functions
- */
- BUG_ON(!fn->func);
- insn->imm = fn->func - __bpf_call_base;
- }
- }
+ prog->aux->ops = bpf_prog_types[type];
+ prog->type = type;
+ return 0;
}
/* drop refcnt on maps used by eBPF program and free auxilary data */
@@ -900,9 +848,6 @@
if (err < 0)
goto free_used_maps;
- /* fixup BPF_CALL->imm field */
- fixup_bpf_calls(prog);
-
/* eBPF program is ready to be JITed */
prog = bpf_prog_select_runtime(prog, &err);
if (err < 0)
@@ -1028,6 +973,28 @@
}
#endif /* CONFIG_CGROUP_BPF */
+#define BPF_PROG_TEST_RUN_LAST_FIELD test.duration
+
+static int bpf_prog_test_run(const union bpf_attr *attr,
+ union bpf_attr __user *uattr)
+{
+ struct bpf_prog *prog;
+ int ret = -ENOTSUPP;
+
+ if (CHECK_ATTR(BPF_PROG_TEST_RUN))
+ return -EINVAL;
+
+ prog = bpf_prog_get(attr->test.prog_fd);
+ if (IS_ERR(prog))
+ return PTR_ERR(prog);
+
+ if (prog->aux->ops->test_run)
+ ret = prog->aux->ops->test_run(prog, attr, uattr);
+
+ bpf_prog_put(prog);
+ return ret;
+}
+
SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
{
union bpf_attr attr = {};
@@ -1094,7 +1061,6 @@
case BPF_OBJ_GET:
err = bpf_obj_get(&attr);
break;
-
#ifdef CONFIG_CGROUP_BPF
case BPF_PROG_ATTACH:
err = bpf_prog_attach(&attr);
@@ -1103,7 +1069,9 @@
err = bpf_prog_detach(&attr);
break;
#endif
-
+ case BPF_PROG_TEST_RUN:
+ err = bpf_prog_test_run(&attr, uattr);
+ break;
default:
err = -EINVAL;
break;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index a834068..c2ff608 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -143,6 +143,8 @@
#define BPF_COMPLEXITY_LIMIT_INSNS 65536
#define BPF_COMPLEXITY_LIMIT_STACK 1024
+#define BPF_MAP_PTR_POISON ((void *)0xeB9F + POISON_POINTER_DELTA)
+
struct bpf_call_arg_meta {
struct bpf_map *map_ptr;
bool raw_mode;
@@ -1215,6 +1217,10 @@
func_id != BPF_FUNC_current_task_under_cgroup)
goto error;
break;
+ case BPF_MAP_TYPE_ARRAY_OF_MAPS:
+ case BPF_MAP_TYPE_HASH_OF_MAPS:
+ if (func_id != BPF_FUNC_map_lookup_elem)
+ goto error;
default:
break;
}
@@ -1291,7 +1297,7 @@
}
}
-static int check_call(struct bpf_verifier_env *env, int func_id)
+static int check_call(struct bpf_verifier_env *env, int func_id, int insn_idx)
{
struct bpf_verifier_state *state = &env->cur_state;
const struct bpf_func_proto *fn = NULL;
@@ -1375,6 +1381,8 @@
} else if (fn->ret_type == RET_VOID) {
regs[BPF_REG_0].type = NOT_INIT;
} else if (fn->ret_type == RET_PTR_TO_MAP_VALUE_OR_NULL) {
+ struct bpf_insn_aux_data *insn_aux;
+
regs[BPF_REG_0].type = PTR_TO_MAP_VALUE_OR_NULL;
regs[BPF_REG_0].max_value = regs[BPF_REG_0].min_value = 0;
/* remember map_ptr, so that check_map_access()
@@ -1387,6 +1395,11 @@
}
regs[BPF_REG_0].map_ptr = meta.map_ptr;
regs[BPF_REG_0].id = ++env->id_gen;
+ insn_aux = &env->insn_aux_data[insn_idx];
+ if (!insn_aux->map_ptr)
+ insn_aux->map_ptr = meta.map_ptr;
+ else if (insn_aux->map_ptr != meta.map_ptr)
+ insn_aux->map_ptr = BPF_MAP_PTR_POISON;
} else {
verbose("unknown return type %d of func %s#%d\n",
fn->ret_type, func_id_name(func_id), func_id);
@@ -1911,6 +1924,17 @@
return 0;
} else if (opcode == BPF_ADD &&
BPF_CLASS(insn->code) == BPF_ALU64 &&
+ dst_reg->type == PTR_TO_STACK &&
+ ((BPF_SRC(insn->code) == BPF_X &&
+ regs[insn->src_reg].type == CONST_IMM) ||
+ BPF_SRC(insn->code) == BPF_K)) {
+ if (BPF_SRC(insn->code) == BPF_X)
+ dst_reg->imm += regs[insn->src_reg].imm;
+ else
+ dst_reg->imm += insn->imm;
+ return 0;
+ } else if (opcode == BPF_ADD &&
+ BPF_CLASS(insn->code) == BPF_ALU64 &&
(dst_reg->type == PTR_TO_PACKET ||
(BPF_SRC(insn->code) == BPF_X &&
regs[insn->src_reg].type == PTR_TO_PACKET))) {
@@ -2112,14 +2136,19 @@
struct bpf_reg_state *reg = ®s[regno];
if (reg->type == PTR_TO_MAP_VALUE_OR_NULL && reg->id == id) {
- reg->type = type;
+ if (type == UNKNOWN_VALUE) {
+ __mark_reg_unknown_value(regs, regno);
+ } else if (reg->map_ptr->inner_map_meta) {
+ reg->type = CONST_PTR_TO_MAP;
+ reg->map_ptr = reg->map_ptr->inner_map_meta;
+ } else {
+ reg->type = type;
+ }
/* We don't need id from this point onwards anymore, thus we
* should better reset it, so that state pruning has chances
* to take effect.
*/
reg->id = 0;
- if (type == UNKNOWN_VALUE)
- __mark_reg_unknown_value(regs, regno);
}
}
@@ -2960,7 +2989,7 @@
return -EINVAL;
}
- err = check_call(env, insn->imm);
+ err = check_call(env, insn->imm, insn_idx);
if (err)
return err;
@@ -3044,16 +3073,33 @@
return 0;
}
+static int check_map_prealloc(struct bpf_map *map)
+{
+ return (map->map_type != BPF_MAP_TYPE_HASH &&
+ map->map_type != BPF_MAP_TYPE_PERCPU_HASH &&
+ map->map_type != BPF_MAP_TYPE_HASH_OF_MAPS) ||
+ !(map->map_flags & BPF_F_NO_PREALLOC);
+}
+
static int check_map_prog_compatibility(struct bpf_map *map,
struct bpf_prog *prog)
{
- if (prog->type == BPF_PROG_TYPE_PERF_EVENT &&
- (map->map_type == BPF_MAP_TYPE_HASH ||
- map->map_type == BPF_MAP_TYPE_PERCPU_HASH) &&
- (map->map_flags & BPF_F_NO_PREALLOC)) {
- verbose("perf_event programs can only use preallocated hash map\n");
- return -EINVAL;
+ /* Make sure that BPF_PROG_TYPE_PERF_EVENT programs only use
+ * preallocated hash maps, since doing memory allocation
+ * in overflow_handler can crash depending on where nmi got
+ * triggered.
+ */
+ if (prog->type == BPF_PROG_TYPE_PERF_EVENT) {
+ if (!check_map_prealloc(map)) {
+ verbose("perf_event programs can only use preallocated hash map\n");
+ return -EINVAL;
+ }
+ if (map->inner_map_meta &&
+ !check_map_prealloc(map->inner_map_meta)) {
+ verbose("perf_event programs can only use preallocated inner hash map\n");
+ return -EINVAL;
+ }
}
return 0;
}
@@ -3182,6 +3228,41 @@
insn->src_reg = 0;
}
+/* single env->prog->insni[off] instruction was replaced with the range
+ * insni[off, off + cnt). Adjust corresponding insn_aux_data by copying
+ * [0, off) and [off, end) to new locations, so the patched range stays zero
+ */
+static int adjust_insn_aux_data(struct bpf_verifier_env *env, u32 prog_len,
+ u32 off, u32 cnt)
+{
+ struct bpf_insn_aux_data *new_data, *old_data = env->insn_aux_data;
+
+ if (cnt == 1)
+ return 0;
+ new_data = vzalloc(sizeof(struct bpf_insn_aux_data) * prog_len);
+ if (!new_data)
+ return -ENOMEM;
+ memcpy(new_data, old_data, sizeof(struct bpf_insn_aux_data) * off);
+ memcpy(new_data + off + cnt - 1, old_data + off,
+ sizeof(struct bpf_insn_aux_data) * (prog_len - off - cnt + 1));
+ env->insn_aux_data = new_data;
+ vfree(old_data);
+ return 0;
+}
+
+static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
+ const struct bpf_insn *patch, u32 len)
+{
+ struct bpf_prog *new_prog;
+
+ new_prog = bpf_patch_insn_single(env->prog, off, patch, len);
+ if (!new_prog)
+ return NULL;
+ if (adjust_insn_aux_data(env, new_prog->len, off, len))
+ return NULL;
+ return new_prog;
+}
+
/* convert load instructions that access fields of 'struct __sk_buff'
* into sequence of instructions that access fields of 'struct sk_buff'
*/
@@ -3201,10 +3282,10 @@
verbose("bpf verifier is misconfigured\n");
return -EINVAL;
} else if (cnt) {
- new_prog = bpf_patch_insn_single(env->prog, 0,
- insn_buf, cnt);
+ new_prog = bpf_patch_insn_data(env, 0, insn_buf, cnt);
if (!new_prog)
return -ENOMEM;
+
env->prog = new_prog;
delta += cnt - 1;
}
@@ -3229,7 +3310,7 @@
else
continue;
- if (env->insn_aux_data[i].ptr_type != PTR_TO_CTX)
+ if (env->insn_aux_data[i + delta].ptr_type != PTR_TO_CTX)
continue;
cnt = ops->convert_ctx_access(type, insn, insn_buf, env->prog);
@@ -3238,8 +3319,7 @@
return -EINVAL;
}
- new_prog = bpf_patch_insn_single(env->prog, i + delta, insn_buf,
- cnt);
+ new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
if (!new_prog)
return -ENOMEM;
@@ -3253,6 +3333,89 @@
return 0;
}
+/* fixup insn->imm field of bpf_call instructions
+ * and inline eligible helpers as explicit sequence of BPF instructions
+ *
+ * this function is called after eBPF program passed verification
+ */
+static int fixup_bpf_calls(struct bpf_verifier_env *env)
+{
+ struct bpf_prog *prog = env->prog;
+ struct bpf_insn *insn = prog->insnsi;
+ const struct bpf_func_proto *fn;
+ const int insn_cnt = prog->len;
+ struct bpf_insn insn_buf[16];
+ struct bpf_prog *new_prog;
+ struct bpf_map *map_ptr;
+ int i, cnt, delta = 0;
+
+ for (i = 0; i < insn_cnt; i++, insn++) {
+ if (insn->code != (BPF_JMP | BPF_CALL))
+ continue;
+
+ if (insn->imm == BPF_FUNC_get_route_realm)
+ prog->dst_needed = 1;
+ if (insn->imm == BPF_FUNC_get_prandom_u32)
+ bpf_user_rnd_init_once();
+ if (insn->imm == BPF_FUNC_tail_call) {
+ /* If we tail call into other programs, we
+ * cannot make any assumptions since they can
+ * be replaced dynamically during runtime in
+ * the program array.
+ */
+ prog->cb_access = 1;
+
+ /* mark bpf_tail_call as different opcode to avoid
+ * conditional branch in the interpeter for every normal
+ * call and to prevent accidental JITing by JIT compiler
+ * that doesn't support bpf_tail_call yet
+ */
+ insn->imm = 0;
+ insn->code |= BPF_X;
+ continue;
+ }
+
+ if (ebpf_jit_enabled() && insn->imm == BPF_FUNC_map_lookup_elem) {
+ map_ptr = env->insn_aux_data[i + delta].map_ptr;
+ if (map_ptr == BPF_MAP_PTR_POISON ||
+ !map_ptr->ops->map_gen_lookup)
+ goto patch_call_imm;
+
+ cnt = map_ptr->ops->map_gen_lookup(map_ptr, insn_buf);
+ if (cnt == 0 || cnt >= ARRAY_SIZE(insn_buf)) {
+ verbose("bpf verifier is misconfigured\n");
+ return -EINVAL;
+ }
+
+ new_prog = bpf_patch_insn_data(env, i + delta, insn_buf,
+ cnt);
+ if (!new_prog)
+ return -ENOMEM;
+
+ delta += cnt - 1;
+
+ /* keep walking new program and skip insns we just inserted */
+ env->prog = prog = new_prog;
+ insn = new_prog->insnsi + i + delta;
+ continue;
+ }
+
+patch_call_imm:
+ fn = prog->aux->ops->get_func_proto(insn->imm);
+ /* all functions that have prototype and verifier allowed
+ * programs to call them, must be real in-kernel functions
+ */
+ if (!fn->func) {
+ verbose("kernel subsystem misconfigured func %s#%d\n",
+ func_id_name(insn->imm), insn->imm);
+ return -EFAULT;
+ }
+ insn->imm = fn->func - __bpf_call_base;
+ }
+
+ return 0;
+}
+
static void free_states(struct bpf_verifier_env *env)
{
struct bpf_verifier_state_list *sl, *sln;
@@ -3348,6 +3511,9 @@
/* program is valid, convert *(u32*)(ctx + off) accesses */
ret = convert_ctx_accesses(env);
+ if (ret == 0)
+ ret = fixup_bpf_calls(env);
+
if (log_level && log_len >= log_size - 1) {
BUG_ON(log_len >= log_size);
/* verifier log exceeded user supplied buffer */
diff --git a/kernel/cgroup/cgroup-internal.h b/kernel/cgroup/cgroup-internal.h
index 9203bfb..00f4d6b 100644
--- a/kernel/cgroup/cgroup-internal.h
+++ b/kernel/cgroup/cgroup-internal.h
@@ -5,6 +5,7 @@
#include <linux/kernfs.h>
#include <linux/workqueue.h>
#include <linux/list.h>
+#include <linux/refcount.h>
/*
* A cgroup can be associated with multiple css_sets as different tasks may
@@ -134,7 +135,7 @@
* can see it. Similar to atomic_dec_and_lock(), but for an
* rwlock
*/
- if (atomic_add_unless(&cset->refcount, -1, 1))
+ if (refcount_dec_not_one(&cset->refcount))
return;
spin_lock_irqsave(&css_set_lock, flags);
@@ -147,7 +148,7 @@
*/
static inline void get_css_set(struct css_set *cset)
{
- atomic_inc(&cset->refcount);
+ refcount_inc(&cset->refcount);
}
bool cgroup_ssid_enabled(int ssid);
@@ -163,7 +164,7 @@
void cgroup_free_root(struct cgroup_root *root);
void init_cgroup_root(struct cgroup_root *root, struct cgroup_sb_opts *opts);
-int cgroup_setup_root(struct cgroup_root *root, u16 ss_mask);
+int cgroup_setup_root(struct cgroup_root *root, u16 ss_mask, int ref_flags);
int rebind_subsystems(struct cgroup_root *dst_root, u16 ss_mask);
struct dentry *cgroup_do_mount(struct file_system_type *fs_type, int flags,
struct cgroup_root *root, unsigned long magic,
diff --git a/kernel/cgroup/cgroup-v1.c b/kernel/cgroup/cgroup-v1.c
index 1dc22f6..85d7515 100644
--- a/kernel/cgroup/cgroup-v1.c
+++ b/kernel/cgroup/cgroup-v1.c
@@ -346,7 +346,7 @@
spin_lock_irq(&css_set_lock);
list_for_each_entry(link, &cgrp->cset_links, cset_link)
- count += atomic_read(&link->cset->refcount);
+ count += refcount_read(&link->cset->refcount);
spin_unlock_irq(&css_set_lock);
return count;
}
@@ -1072,6 +1072,7 @@
struct cgroup_subsys *ss;
struct dentry *dentry;
int i, ret;
+ bool new_root = false;
cgroup_lock_and_drain_offline(&cgrp_dfl_root.cgrp);
@@ -1181,10 +1182,11 @@
ret = -ENOMEM;
goto out_unlock;
}
+ new_root = true;
init_cgroup_root(root, &opts);
- ret = cgroup_setup_root(root, opts.subsys_mask);
+ ret = cgroup_setup_root(root, opts.subsys_mask, PERCPU_REF_INIT_DEAD);
if (ret)
cgroup_free_root(root);
@@ -1201,6 +1203,18 @@
CGROUP_SUPER_MAGIC, ns);
/*
+ * There's a race window after we release cgroup_mutex and before
+ * allocating a superblock. Make sure a concurrent process won't
+ * be able to re-use the root during this window by delaying the
+ * initialization of root refcnt.
+ */
+ if (new_root) {
+ mutex_lock(&cgroup_mutex);
+ percpu_ref_reinit(&root->cgrp.self.refcnt);
+ mutex_unlock(&cgroup_mutex);
+ }
+
+ /*
* If @pinned_sb, we're reusing an existing root and holding an
* extra ref on its sb. Mount is complete. Put the extra ref.
*/
@@ -1286,7 +1300,7 @@
u64 count;
rcu_read_lock();
- count = atomic_read(&task_css_set(current)->refcount);
+ count = refcount_read(&task_css_set(current)->refcount);
rcu_read_unlock();
return count;
}
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index 687f5e0..c3c9a0e 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -189,7 +189,7 @@
/* cgroup namespace for init task */
struct cgroup_namespace init_cgroup_ns = {
- .count = { .counter = 2, },
+ .count = REFCOUNT_INIT(2),
.user_ns = &init_user_ns,
.ns.ops = &cgroupns_operations,
.ns.inum = PROC_CGROUP_INIT_INO,
@@ -436,7 +436,12 @@
return css;
}
-static void cgroup_get(struct cgroup *cgrp)
+static void __maybe_unused cgroup_get(struct cgroup *cgrp)
+{
+ css_get(&cgrp->self);
+}
+
+static void cgroup_get_live(struct cgroup *cgrp)
{
WARN_ON_ONCE(cgroup_is_dead(cgrp));
css_get(&cgrp->self);
@@ -554,7 +559,7 @@
* haven't been created.
*/
struct css_set init_css_set = {
- .refcount = ATOMIC_INIT(1),
+ .refcount = REFCOUNT_INIT(1),
.tasks = LIST_HEAD_INIT(init_css_set.tasks),
.mg_tasks = LIST_HEAD_INIT(init_css_set.mg_tasks),
.task_iters = LIST_HEAD_INIT(init_css_set.task_iters),
@@ -724,7 +729,7 @@
lockdep_assert_held(&css_set_lock);
- if (!atomic_dec_and_test(&cset->refcount))
+ if (!refcount_dec_and_test(&cset->refcount))
return;
/* This css_set is dead. unlink it and release cgroup and css refs */
@@ -932,7 +937,7 @@
list_add_tail(&link->cgrp_link, &cset->cgrp_links);
if (cgroup_parent(cgrp))
- cgroup_get(cgrp);
+ cgroup_get_live(cgrp);
}
/**
@@ -977,7 +982,7 @@
return NULL;
}
- atomic_set(&cset->refcount, 1);
+ refcount_set(&cset->refcount, 1);
INIT_LIST_HEAD(&cset->tasks);
INIT_LIST_HEAD(&cset->mg_tasks);
INIT_LIST_HEAD(&cset->task_iters);
@@ -1640,7 +1645,7 @@
set_bit(CGRP_CPUSET_CLONE_CHILDREN, &root->cgrp.flags);
}
-int cgroup_setup_root(struct cgroup_root *root, u16 ss_mask)
+int cgroup_setup_root(struct cgroup_root *root, u16 ss_mask, int ref_flags)
{
LIST_HEAD(tmp_links);
struct cgroup *root_cgrp = &root->cgrp;
@@ -1656,8 +1661,8 @@
root_cgrp->id = ret;
root_cgrp->ancestor_ids[0] = ret;
- ret = percpu_ref_init(&root_cgrp->self.refcnt, css_release, 0,
- GFP_KERNEL);
+ ret = percpu_ref_init(&root_cgrp->self.refcnt, css_release,
+ ref_flags, GFP_KERNEL);
if (ret)
goto out;
@@ -1802,7 +1807,7 @@
return ERR_PTR(-EINVAL);
}
cgrp_dfl_visible = true;
- cgroup_get(&cgrp_dfl_root.cgrp);
+ cgroup_get_live(&cgrp_dfl_root.cgrp);
dentry = cgroup_do_mount(&cgroup2_fs_type, flags, &cgrp_dfl_root,
CGROUP2_SUPER_MAGIC, ns);
@@ -2576,7 +2581,7 @@
if (!css || !percpu_ref_is_dying(&css->refcnt))
continue;
- cgroup_get(dsct);
+ cgroup_get_live(dsct);
prepare_to_wait(&dsct->offline_waitq, &wait,
TASK_UNINTERRUPTIBLE);
@@ -3947,7 +3952,7 @@
{
lockdep_assert_held(&cgroup_mutex);
- cgroup_get(cgrp);
+ cgroup_get_live(cgrp);
memset(css, 0, sizeof(*css));
css->cgroup = cgrp;
@@ -4123,7 +4128,7 @@
/* allocation complete, commit to creation */
list_add_tail_rcu(&cgrp->self.sibling, &cgroup_parent(cgrp)->self.children);
atomic_inc(&root->nr_cgrps);
- cgroup_get(parent);
+ cgroup_get_live(parent);
/*
* @cgrp is now fully operational. If something fails after this
@@ -4513,7 +4518,7 @@
hash_add(css_set_table, &init_css_set.hlist,
css_set_hash(init_css_set.subsys));
- BUG_ON(cgroup_setup_root(&cgrp_dfl_root, 0));
+ BUG_ON(cgroup_setup_root(&cgrp_dfl_root, 0, 0));
mutex_unlock(&cgroup_mutex);
@@ -4947,7 +4952,7 @@
if (kn) {
if (kernfs_type(kn) == KERNFS_DIR) {
cgrp = kn->priv;
- cgroup_get(cgrp);
+ cgroup_get_live(cgrp);
} else {
cgrp = ERR_PTR(-ENOTDIR);
}
@@ -5027,6 +5032,11 @@
/* Socket clone path */
if (skcd->val) {
+ /*
+ * We might be cloning a socket which is left in an empty
+ * cgroup and the cgroup might have already been rmdir'd.
+ * Don't use cgroup_get_live().
+ */
cgroup_get(sock_cgroup_ptr(skcd));
return;
}
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 0f41292..f6501f4 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -2121,10 +2121,8 @@
{
int err = 0;
- if (!alloc_cpumask_var(&top_cpuset.cpus_allowed, GFP_KERNEL))
- BUG();
- if (!alloc_cpumask_var(&top_cpuset.effective_cpus, GFP_KERNEL))
- BUG();
+ BUG_ON(!alloc_cpumask_var(&top_cpuset.cpus_allowed, GFP_KERNEL));
+ BUG_ON(!alloc_cpumask_var(&top_cpuset.effective_cpus, GFP_KERNEL));
cpumask_setall(top_cpuset.cpus_allowed);
nodes_setall(top_cpuset.mems_allowed);
@@ -2139,8 +2137,7 @@
if (err < 0)
return err;
- if (!alloc_cpumask_var(&cpus_attach, GFP_KERNEL))
- BUG();
+ BUG_ON(!alloc_cpumask_var(&cpus_attach, GFP_KERNEL));
return 0;
}
@@ -2354,7 +2351,7 @@
rebuild_sched_domains();
}
-void cpuset_update_active_cpus(bool cpu_online)
+void cpuset_update_active_cpus(void)
{
/*
* We're inside cpu hotplug critical region which usually nests
diff --git a/kernel/cgroup/namespace.c b/kernel/cgroup/namespace.c
index 96d38da..66129eb 100644
--- a/kernel/cgroup/namespace.c
+++ b/kernel/cgroup/namespace.c
@@ -31,7 +31,7 @@
kfree(new_ns);
return ERR_PTR(ret);
}
- atomic_set(&new_ns->count, 1);
+ refcount_set(&new_ns->count, 1);
new_ns->ns.ops = &cgroupns_operations;
return new_ns;
}
diff --git a/kernel/compat.c b/kernel/compat.c
index 19aec5d..933bcb3 100644
--- a/kernel/compat.c
+++ b/kernel/compat.c
@@ -108,8 +108,8 @@
COMPAT_SYSCALL_DEFINE2(settimeofday, struct compat_timeval __user *, tv,
struct timezone __user *, tz)
{
+ struct timespec64 new_ts;
struct timeval user_tv;
- struct timespec new_ts;
struct timezone new_tz;
if (tv) {
@@ -123,7 +123,7 @@
return -EFAULT;
}
- return do_sys_settimeofday(tv ? &new_ts : NULL, tz ? &new_tz : NULL);
+ return do_sys_settimeofday64(tv ? &new_ts : NULL, tz ? &new_tz : NULL);
}
static int __compat_get_timeval(struct timeval *tv, const struct compat_timeval __user *ctv)
@@ -240,18 +240,20 @@
struct compat_timespec __user *, rmtp)
{
struct timespec tu, rmt;
+ struct timespec64 tu64;
mm_segment_t oldfs;
long ret;
if (compat_get_timespec(&tu, rqtp))
return -EFAULT;
- if (!timespec_valid(&tu))
+ tu64 = timespec_to_timespec64(tu);
+ if (!timespec64_valid(&tu64))
return -EINVAL;
oldfs = get_fs();
set_fs(KERNEL_DS);
- ret = hrtimer_nanosleep(&tu,
+ ret = hrtimer_nanosleep(&tu64,
rmtp ? (struct timespec __user *)&rmt : NULL,
HRTIMER_MODE_REL, CLOCK_MONOTONIC);
set_fs(oldfs);
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 37b223e..9ae6fbe 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1125,6 +1125,8 @@
#endif /* CONFIG_PM_SLEEP_SMP */
+int __boot_cpu_id;
+
#endif /* CONFIG_SMP */
/* Boot processor state steps */
@@ -1815,6 +1817,10 @@
set_cpu_active(cpu, true);
set_cpu_present(cpu, true);
set_cpu_possible(cpu, true);
+
+#ifdef CONFIG_SMP
+ __boot_cpu_id = cpu;
+#endif
}
/*
diff --git a/kernel/events/core.c b/kernel/events/core.c
index ff01cba..6e75a5c 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -48,6 +48,8 @@
#include <linux/parser.h>
#include <linux/sched/clock.h>
#include <linux/sched/mm.h>
+#include <linux/proc_ns.h>
+#include <linux/mount.h>
#include "internal.h"
@@ -379,6 +381,7 @@
static atomic_t nr_mmap_events __read_mostly;
static atomic_t nr_comm_events __read_mostly;
+static atomic_t nr_namespaces_events __read_mostly;
static atomic_t nr_task_events __read_mostly;
static atomic_t nr_freq_events __read_mostly;
static atomic_t nr_switch_events __read_mostly;
@@ -3991,6 +3994,8 @@
atomic_dec(&nr_mmap_events);
if (event->attr.comm)
atomic_dec(&nr_comm_events);
+ if (event->attr.namespaces)
+ atomic_dec(&nr_namespaces_events);
if (event->attr.task)
atomic_dec(&nr_task_events);
if (event->attr.freq)
@@ -6491,6 +6496,7 @@
void perf_event_fork(struct task_struct *task)
{
perf_event_task(task, NULL, 1);
+ perf_event_namespaces(task);
}
/*
@@ -6593,6 +6599,132 @@
}
/*
+ * namespaces tracking
+ */
+
+struct perf_namespaces_event {
+ struct task_struct *task;
+
+ struct {
+ struct perf_event_header header;
+
+ u32 pid;
+ u32 tid;
+ u64 nr_namespaces;
+ struct perf_ns_link_info link_info[NR_NAMESPACES];
+ } event_id;
+};
+
+static int perf_event_namespaces_match(struct perf_event *event)
+{
+ return event->attr.namespaces;
+}
+
+static void perf_event_namespaces_output(struct perf_event *event,
+ void *data)
+{
+ struct perf_namespaces_event *namespaces_event = data;
+ struct perf_output_handle handle;
+ struct perf_sample_data sample;
+ int ret;
+
+ if (!perf_event_namespaces_match(event))
+ return;
+
+ perf_event_header__init_id(&namespaces_event->event_id.header,
+ &sample, event);
+ ret = perf_output_begin(&handle, event,
+ namespaces_event->event_id.header.size);
+ if (ret)
+ return;
+
+ namespaces_event->event_id.pid = perf_event_pid(event,
+ namespaces_event->task);
+ namespaces_event->event_id.tid = perf_event_tid(event,
+ namespaces_event->task);
+
+ perf_output_put(&handle, namespaces_event->event_id);
+
+ perf_event__output_id_sample(event, &handle, &sample);
+
+ perf_output_end(&handle);
+}
+
+static void perf_fill_ns_link_info(struct perf_ns_link_info *ns_link_info,
+ struct task_struct *task,
+ const struct proc_ns_operations *ns_ops)
+{
+ struct path ns_path;
+ struct inode *ns_inode;
+ void *error;
+
+ error = ns_get_path(&ns_path, task, ns_ops);
+ if (!error) {
+ ns_inode = ns_path.dentry->d_inode;
+ ns_link_info->dev = new_encode_dev(ns_inode->i_sb->s_dev);
+ ns_link_info->ino = ns_inode->i_ino;
+ }
+}
+
+void perf_event_namespaces(struct task_struct *task)
+{
+ struct perf_namespaces_event namespaces_event;
+ struct perf_ns_link_info *ns_link_info;
+
+ if (!atomic_read(&nr_namespaces_events))
+ return;
+
+ namespaces_event = (struct perf_namespaces_event){
+ .task = task,
+ .event_id = {
+ .header = {
+ .type = PERF_RECORD_NAMESPACES,
+ .misc = 0,
+ .size = sizeof(namespaces_event.event_id),
+ },
+ /* .pid */
+ /* .tid */
+ .nr_namespaces = NR_NAMESPACES,
+ /* .link_info[NR_NAMESPACES] */
+ },
+ };
+
+ ns_link_info = namespaces_event.event_id.link_info;
+
+ perf_fill_ns_link_info(&ns_link_info[MNT_NS_INDEX],
+ task, &mntns_operations);
+
+#ifdef CONFIG_USER_NS
+ perf_fill_ns_link_info(&ns_link_info[USER_NS_INDEX],
+ task, &userns_operations);
+#endif
+#ifdef CONFIG_NET_NS
+ perf_fill_ns_link_info(&ns_link_info[NET_NS_INDEX],
+ task, &netns_operations);
+#endif
+#ifdef CONFIG_UTS_NS
+ perf_fill_ns_link_info(&ns_link_info[UTS_NS_INDEX],
+ task, &utsns_operations);
+#endif
+#ifdef CONFIG_IPC_NS
+ perf_fill_ns_link_info(&ns_link_info[IPC_NS_INDEX],
+ task, &ipcns_operations);
+#endif
+#ifdef CONFIG_PID_NS
+ perf_fill_ns_link_info(&ns_link_info[PID_NS_INDEX],
+ task, &pidns_operations);
+#endif
+#ifdef CONFIG_CGROUPS
+ perf_fill_ns_link_info(&ns_link_info[CGROUP_NS_INDEX],
+ task, &cgroupns_operations);
+#endif
+
+ perf_iterate_sb(perf_event_namespaces_output,
+ &namespaces_event,
+ NULL);
+}
+
+/*
* mmap tracking
*/
@@ -9146,6 +9278,8 @@
atomic_inc(&nr_mmap_events);
if (event->attr.comm)
atomic_inc(&nr_comm_events);
+ if (event->attr.namespaces)
+ atomic_inc(&nr_namespaces_events);
if (event->attr.task)
atomic_inc(&nr_task_events);
if (event->attr.freq)
@@ -9691,6 +9825,11 @@
return -EACCES;
}
+ if (attr.namespaces) {
+ if (!capable(CAP_SYS_ADMIN))
+ return -EACCES;
+ }
+
if (attr.freq) {
if (attr.sample_freq > sysctl_perf_event_sample_rate)
return -EINVAL;
diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 257fa46..2831480 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -297,6 +297,19 @@
rb->paused = 1;
}
+void perf_aux_output_flag(struct perf_output_handle *handle, u64 flags)
+{
+ /*
+ * OVERWRITE is determined by perf_aux_output_end() and can't
+ * be passed in directly.
+ */
+ if (WARN_ON_ONCE(flags & PERF_AUX_FLAG_OVERWRITE))
+ return;
+
+ handle->aux_flags |= flags;
+}
+EXPORT_SYMBOL_GPL(perf_aux_output_flag);
+
/*
* This is called before hardware starts writing to the AUX area to
* obtain an output handle and make sure there's room in the buffer.
@@ -360,6 +373,7 @@
handle->event = event;
handle->head = aux_head;
handle->size = 0;
+ handle->aux_flags = 0;
/*
* In overwrite mode, AUX data stores do not depend on aux_tail,
@@ -408,34 +422,32 @@
* of the AUX buffer management code is that after pmu::stop(), the AUX
* transaction must be stopped and therefore drop the AUX reference count.
*/
-void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size,
- bool truncated)
+void perf_aux_output_end(struct perf_output_handle *handle, unsigned long size)
{
+ bool wakeup = !!(handle->aux_flags & PERF_AUX_FLAG_TRUNCATED);
struct ring_buffer *rb = handle->rb;
- bool wakeup = truncated;
unsigned long aux_head;
- u64 flags = 0;
-
- if (truncated)
- flags |= PERF_AUX_FLAG_TRUNCATED;
/* in overwrite mode, driver provides aux_head via handle */
if (rb->aux_overwrite) {
- flags |= PERF_AUX_FLAG_OVERWRITE;
+ handle->aux_flags |= PERF_AUX_FLAG_OVERWRITE;
aux_head = handle->head;
local_set(&rb->aux_head, aux_head);
} else {
+ handle->aux_flags &= ~PERF_AUX_FLAG_OVERWRITE;
+
aux_head = local_read(&rb->aux_head);
local_add(size, &rb->aux_head);
}
- if (size || flags) {
+ if (size || handle->aux_flags) {
/*
* Only send RECORD_AUX if we have something useful to communicate
*/
- perf_event_aux_event(handle->event, aux_head, size, flags);
+ perf_event_aux_event(handle->event, aux_head, size,
+ handle->aux_flags);
}
aux_head = rb->user_page->aux_head = local_read(&rb->aux_head);
@@ -446,7 +458,7 @@
}
if (wakeup) {
- if (truncated)
+ if (handle->aux_flags & PERF_AUX_FLAG_TRUNCATED)
handle->event->pending_disable = 1;
perf_output_wakeup(handle);
}
diff --git a/kernel/fork.c b/kernel/fork.c
index 6c463c80..56d85fd 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -87,6 +87,7 @@
#include <linux/compiler.h>
#include <linux/sysctl.h>
#include <linux/kcov.h>
+#include <linux/livepatch.h>
#include <asm/pgtable.h>
#include <asm/pgalloc.h>
@@ -1438,6 +1439,7 @@
#ifdef CONFIG_RT_MUTEXES
p->pi_waiters = RB_ROOT;
p->pi_waiters_leftmost = NULL;
+ p->pi_top_task = NULL;
p->pi_blocked_on = NULL;
#endif
}
@@ -1797,6 +1799,8 @@
p->parent_exec_id = current->self_exec_id;
}
+ klp_copy_process(p);
+
spin_lock(¤t->sighand->siglock);
/*
@@ -2352,6 +2356,8 @@
}
}
+ perf_event_namespaces(current);
+
bad_unshare_cleanup_cred:
if (new_cred)
put_cred(new_cred);
diff --git a/kernel/futex.c b/kernel/futex.c
index 45858ec..357348a 100644
--- a/kernel/futex.c
+++ b/kernel/futex.c
@@ -802,7 +802,7 @@
return 0;
}
-static struct futex_pi_state * alloc_pi_state(void)
+static struct futex_pi_state *alloc_pi_state(void)
{
struct futex_pi_state *pi_state = current->pi_state_cache;
@@ -812,6 +812,11 @@
return pi_state;
}
+static void get_pi_state(struct futex_pi_state *pi_state)
+{
+ WARN_ON_ONCE(!atomic_inc_not_zero(&pi_state->refcount));
+}
+
/*
* Drops a reference to the pi_state object and frees or caches it
* when the last reference is gone.
@@ -856,7 +861,7 @@
* Look up the task based on what TID userspace gave us.
* We dont trust it.
*/
-static struct task_struct * futex_find_get_task(pid_t pid)
+static struct task_struct *futex_find_get_task(pid_t pid)
{
struct task_struct *p;
@@ -916,10 +921,12 @@
pi_state->owner = NULL;
raw_spin_unlock_irq(&curr->pi_lock);
- rt_mutex_unlock(&pi_state->pi_mutex);
-
+ get_pi_state(pi_state);
spin_unlock(&hb->lock);
+ rt_mutex_futex_unlock(&pi_state->pi_mutex);
+ put_pi_state(pi_state);
+
raw_spin_lock_irq(&curr->pi_lock);
}
raw_spin_unlock_irq(&curr->pi_lock);
@@ -973,6 +980,39 @@
*
* [10] There is no transient state which leaves owner and user space
* TID out of sync.
+ *
+ *
+ * Serialization and lifetime rules:
+ *
+ * hb->lock:
+ *
+ * hb -> futex_q, relation
+ * futex_q -> pi_state, relation
+ *
+ * (cannot be raw because hb can contain arbitrary amount
+ * of futex_q's)
+ *
+ * pi_mutex->wait_lock:
+ *
+ * {uval, pi_state}
+ *
+ * (and pi_mutex 'obviously')
+ *
+ * p->pi_lock:
+ *
+ * p->pi_state_list -> pi_state->list, relation
+ *
+ * pi_state->refcount:
+ *
+ * pi_state lifetime
+ *
+ *
+ * Lock order:
+ *
+ * hb->lock
+ * pi_mutex->wait_lock
+ * p->pi_lock
+ *
*/
/*
@@ -980,10 +1020,13 @@
* the pi_state against the user space value. If correct, attach to
* it.
*/
-static int attach_to_pi_state(u32 uval, struct futex_pi_state *pi_state,
+static int attach_to_pi_state(u32 __user *uaddr, u32 uval,
+ struct futex_pi_state *pi_state,
struct futex_pi_state **ps)
{
pid_t pid = uval & FUTEX_TID_MASK;
+ u32 uval2;
+ int ret;
/*
* Userspace might have messed up non-PI and PI futexes [3]
@@ -991,9 +1034,39 @@
if (unlikely(!pi_state))
return -EINVAL;
+ /*
+ * We get here with hb->lock held, and having found a
+ * futex_top_waiter(). This means that futex_lock_pi() of said futex_q
+ * has dropped the hb->lock in between queue_me() and unqueue_me_pi(),
+ * which in turn means that futex_lock_pi() still has a reference on
+ * our pi_state.
+ *
+ * The waiter holding a reference on @pi_state also protects against
+ * the unlocked put_pi_state() in futex_unlock_pi(), futex_lock_pi()
+ * and futex_wait_requeue_pi() as it cannot go to 0 and consequently
+ * free pi_state before we can take a reference ourselves.
+ */
WARN_ON(!atomic_read(&pi_state->refcount));
/*
+ * Now that we have a pi_state, we can acquire wait_lock
+ * and do the state validation.
+ */
+ raw_spin_lock_irq(&pi_state->pi_mutex.wait_lock);
+
+ /*
+ * Since {uval, pi_state} is serialized by wait_lock, and our current
+ * uval was read without holding it, it can have changed. Verify it
+ * still is what we expect it to be, otherwise retry the entire
+ * operation.
+ */
+ if (get_futex_value_locked(&uval2, uaddr))
+ goto out_efault;
+
+ if (uval != uval2)
+ goto out_eagain;
+
+ /*
* Handle the owner died case:
*/
if (uval & FUTEX_OWNER_DIED) {
@@ -1008,11 +1081,11 @@
* is not 0. Inconsistent state. [5]
*/
if (pid)
- return -EINVAL;
+ goto out_einval;
/*
* Take a ref on the state and return success. [4]
*/
- goto out_state;
+ goto out_attach;
}
/*
@@ -1024,14 +1097,14 @@
* Take a ref on the state and return success. [6]
*/
if (!pid)
- goto out_state;
+ goto out_attach;
} else {
/*
* If the owner died bit is not set, then the pi_state
* must have an owner. [7]
*/
if (!pi_state->owner)
- return -EINVAL;
+ goto out_einval;
}
/*
@@ -1040,11 +1113,29 @@
* user space TID. [9/10]
*/
if (pid != task_pid_vnr(pi_state->owner))
- return -EINVAL;
-out_state:
- atomic_inc(&pi_state->refcount);
+ goto out_einval;
+
+out_attach:
+ get_pi_state(pi_state);
+ raw_spin_unlock_irq(&pi_state->pi_mutex.wait_lock);
*ps = pi_state;
return 0;
+
+out_einval:
+ ret = -EINVAL;
+ goto out_error;
+
+out_eagain:
+ ret = -EAGAIN;
+ goto out_error;
+
+out_efault:
+ ret = -EFAULT;
+ goto out_error;
+
+out_error:
+ raw_spin_unlock_irq(&pi_state->pi_mutex.wait_lock);
+ return ret;
}
/*
@@ -1095,6 +1186,9 @@
/*
* No existing pi state. First waiter. [2]
+ *
+ * This creates pi_state, we have hb->lock held, this means nothing can
+ * observe this state, wait_lock is irrelevant.
*/
pi_state = alloc_pi_state();
@@ -1119,17 +1213,18 @@
return 0;
}
-static int lookup_pi_state(u32 uval, struct futex_hash_bucket *hb,
+static int lookup_pi_state(u32 __user *uaddr, u32 uval,
+ struct futex_hash_bucket *hb,
union futex_key *key, struct futex_pi_state **ps)
{
- struct futex_q *match = futex_top_waiter(hb, key);
+ struct futex_q *top_waiter = futex_top_waiter(hb, key);
/*
* If there is a waiter on that futex, validate it and
* attach to the pi_state when the validation succeeds.
*/
- if (match)
- return attach_to_pi_state(uval, match->pi_state, ps);
+ if (top_waiter)
+ return attach_to_pi_state(uaddr, uval, top_waiter->pi_state, ps);
/*
* We are the first waiter - try to look up the owner based on
@@ -1148,7 +1243,7 @@
if (unlikely(cmpxchg_futex_value_locked(&curval, uaddr, uval, newval)))
return -EFAULT;
- /*If user space value changed, let the caller retry */
+ /* If user space value changed, let the caller retry */
return curval != uval ? -EAGAIN : 0;
}
@@ -1176,7 +1271,7 @@
struct task_struct *task, int set_waiters)
{
u32 uval, newval, vpid = task_pid_vnr(task);
- struct futex_q *match;
+ struct futex_q *top_waiter;
int ret;
/*
@@ -1202,9 +1297,9 @@
* Lookup existing state first. If it exists, try to attach to
* its pi_state.
*/
- match = futex_top_waiter(hb, key);
- if (match)
- return attach_to_pi_state(uval, match->pi_state, ps);
+ top_waiter = futex_top_waiter(hb, key);
+ if (top_waiter)
+ return attach_to_pi_state(uaddr, uval, top_waiter->pi_state, ps);
/*
* No waiter and user TID is 0. We are here because the
@@ -1285,50 +1380,44 @@
wake_q_add(wake_q, p);
__unqueue_futex(q);
/*
- * The waiting task can free the futex_q as soon as
- * q->lock_ptr = NULL is written, without taking any locks. A
- * memory barrier is required here to prevent the following
- * store to lock_ptr from getting ahead of the plist_del.
+ * The waiting task can free the futex_q as soon as q->lock_ptr = NULL
+ * is written, without taking any locks. This is possible in the event
+ * of a spurious wakeup, for example. A memory barrier is required here
+ * to prevent the following store to lock_ptr from getting ahead of the
+ * plist_del in __unqueue_futex().
*/
- smp_wmb();
- q->lock_ptr = NULL;
+ smp_store_release(&q->lock_ptr, NULL);
}
-static int wake_futex_pi(u32 __user *uaddr, u32 uval, struct futex_q *this,
- struct futex_hash_bucket *hb)
+/*
+ * Caller must hold a reference on @pi_state.
+ */
+static int wake_futex_pi(u32 __user *uaddr, u32 uval, struct futex_pi_state *pi_state)
{
- struct task_struct *new_owner;
- struct futex_pi_state *pi_state = this->pi_state;
u32 uninitialized_var(curval), newval;
+ struct task_struct *new_owner;
+ bool postunlock = false;
DEFINE_WAKE_Q(wake_q);
- bool deboost;
int ret = 0;
- if (!pi_state)
- return -EINVAL;
-
- /*
- * If current does not own the pi_state then the futex is
- * inconsistent and user space fiddled with the futex value.
- */
- if (pi_state->owner != current)
- return -EINVAL;
-
- raw_spin_lock_irq(&pi_state->pi_mutex.wait_lock);
new_owner = rt_mutex_next_owner(&pi_state->pi_mutex);
+ if (WARN_ON_ONCE(!new_owner)) {
+ /*
+ * As per the comment in futex_unlock_pi() this should not happen.
+ *
+ * When this happens, give up our locks and try again, giving
+ * the futex_lock_pi() instance time to complete, either by
+ * waiting on the rtmutex or removing itself from the futex
+ * queue.
+ */
+ ret = -EAGAIN;
+ goto out_unlock;
+ }
/*
- * It is possible that the next waiter (the one that brought
- * this owner to the kernel) timed out and is no longer
- * waiting on the lock.
- */
- if (!new_owner)
- new_owner = this->task;
-
- /*
- * We pass it to the next owner. The WAITERS bit is always
- * kept enabled while there is PI state around. We cleanup the
- * owner died bit, because we are the owner.
+ * We pass it to the next owner. The WAITERS bit is always kept
+ * enabled while there is PI state around. We cleanup the owner
+ * died bit, because we are the owner.
*/
newval = FUTEX_WAITERS | task_pid_vnr(new_owner);
@@ -1337,6 +1426,7 @@
if (cmpxchg_futex_value_locked(&curval, uaddr, uval, newval)) {
ret = -EFAULT;
+
} else if (curval != uval) {
/*
* If a unconditional UNLOCK_PI operation (user space did not
@@ -1349,10 +1439,14 @@
else
ret = -EINVAL;
}
- if (ret) {
- raw_spin_unlock_irq(&pi_state->pi_mutex.wait_lock);
- return ret;
- }
+
+ if (ret)
+ goto out_unlock;
+
+ /*
+ * This is a point of no return; once we modify the uval there is no
+ * going back and subsequent operations must not fail.
+ */
raw_spin_lock(&pi_state->owner->pi_lock);
WARN_ON(list_empty(&pi_state->list));
@@ -1365,22 +1459,15 @@
pi_state->owner = new_owner;
raw_spin_unlock(&new_owner->pi_lock);
+ postunlock = __rt_mutex_futex_unlock(&pi_state->pi_mutex, &wake_q);
+
+out_unlock:
raw_spin_unlock_irq(&pi_state->pi_mutex.wait_lock);
- deboost = rt_mutex_futex_unlock(&pi_state->pi_mutex, &wake_q);
+ if (postunlock)
+ rt_mutex_postunlock(&wake_q);
- /*
- * First unlock HB so the waiter does not spin on it once he got woken
- * up. Second wake up the waiter before the priority is adjusted. If we
- * deboost first (and lose our higher priority), then the task might get
- * scheduled away before the wake up can take place.
- */
- spin_unlock(&hb->lock);
- wake_up_q(&wake_q);
- if (deboost)
- rt_mutex_adjust_prio(current);
-
- return 0;
+ return ret;
}
/*
@@ -1826,7 +1913,7 @@
* If that call succeeds then we have pi_state and an
* initial refcount on it.
*/
- ret = lookup_pi_state(ret, hb2, &key2, &pi_state);
+ ret = lookup_pi_state(uaddr2, ret, hb2, &key2, &pi_state);
}
switch (ret) {
@@ -1909,7 +1996,7 @@
* refcount on the pi_state and store the pointer in
* the futex_q object of the waiter.
*/
- atomic_inc(&pi_state->refcount);
+ get_pi_state(pi_state);
this->pi_state = pi_state;
ret = rt_mutex_start_proxy_lock(&pi_state->pi_mutex,
this->rt_waiter,
@@ -2009,20 +2096,7 @@
hb_waiters_dec(hb);
}
-/**
- * queue_me() - Enqueue the futex_q on the futex_hash_bucket
- * @q: The futex_q to enqueue
- * @hb: The destination hash bucket
- *
- * The hb->lock must be held by the caller, and is released here. A call to
- * queue_me() is typically paired with exactly one call to unqueue_me(). The
- * exceptions involve the PI related operations, which may use unqueue_me_pi()
- * or nothing if the unqueue is done as part of the wake process and the unqueue
- * state is implicit in the state of woken task (see futex_wait_requeue_pi() for
- * an example).
- */
-static inline void queue_me(struct futex_q *q, struct futex_hash_bucket *hb)
- __releases(&hb->lock)
+static inline void __queue_me(struct futex_q *q, struct futex_hash_bucket *hb)
{
int prio;
@@ -2039,6 +2113,24 @@
plist_node_init(&q->list, prio);
plist_add(&q->list, &hb->chain);
q->task = current;
+}
+
+/**
+ * queue_me() - Enqueue the futex_q on the futex_hash_bucket
+ * @q: The futex_q to enqueue
+ * @hb: The destination hash bucket
+ *
+ * The hb->lock must be held by the caller, and is released here. A call to
+ * queue_me() is typically paired with exactly one call to unqueue_me(). The
+ * exceptions involve the PI related operations, which may use unqueue_me_pi()
+ * or nothing if the unqueue is done as part of the wake process and the unqueue
+ * state is implicit in the state of woken task (see futex_wait_requeue_pi() for
+ * an example).
+ */
+static inline void queue_me(struct futex_q *q, struct futex_hash_bucket *hb)
+ __releases(&hb->lock)
+{
+ __queue_me(q, hb);
spin_unlock(&hb->lock);
}
@@ -2125,10 +2217,13 @@
{
u32 newtid = task_pid_vnr(newowner) | FUTEX_WAITERS;
struct futex_pi_state *pi_state = q->pi_state;
- struct task_struct *oldowner = pi_state->owner;
u32 uval, uninitialized_var(curval), newval;
+ struct task_struct *oldowner;
int ret;
+ raw_spin_lock_irq(&pi_state->pi_mutex.wait_lock);
+
+ oldowner = pi_state->owner;
/* Owner died? */
if (!pi_state->owner)
newtid |= FUTEX_OWNER_DIED;
@@ -2136,7 +2231,8 @@
/*
* We are here either because we stole the rtmutex from the
* previous highest priority waiter or we are the highest priority
- * waiter but failed to get the rtmutex the first time.
+ * waiter but have failed to get the rtmutex the first time.
+ *
* We have to replace the newowner TID in the user space variable.
* This must be atomic as we have to preserve the owner died bit here.
*
@@ -2144,17 +2240,16 @@
* because we can fault here. Imagine swapped out pages or a fork
* that marked all the anonymous memory readonly for cow.
*
- * Modifying pi_state _before_ the user space value would
- * leave the pi_state in an inconsistent state when we fault
- * here, because we need to drop the hash bucket lock to
- * handle the fault. This might be observed in the PID check
- * in lookup_pi_state.
+ * Modifying pi_state _before_ the user space value would leave the
+ * pi_state in an inconsistent state when we fault here, because we
+ * need to drop the locks to handle the fault. This might be observed
+ * in the PID check in lookup_pi_state.
*/
retry:
if (get_futex_value_locked(&uval, uaddr))
goto handle_fault;
- while (1) {
+ for (;;) {
newval = (uval & FUTEX_OWNER_DIED) | newtid;
if (cmpxchg_futex_value_locked(&curval, uaddr, uval, newval))
@@ -2169,47 +2264,60 @@
* itself.
*/
if (pi_state->owner != NULL) {
- raw_spin_lock_irq(&pi_state->owner->pi_lock);
+ raw_spin_lock(&pi_state->owner->pi_lock);
WARN_ON(list_empty(&pi_state->list));
list_del_init(&pi_state->list);
- raw_spin_unlock_irq(&pi_state->owner->pi_lock);
+ raw_spin_unlock(&pi_state->owner->pi_lock);
}
pi_state->owner = newowner;
- raw_spin_lock_irq(&newowner->pi_lock);
+ raw_spin_lock(&newowner->pi_lock);
WARN_ON(!list_empty(&pi_state->list));
list_add(&pi_state->list, &newowner->pi_state_list);
- raw_spin_unlock_irq(&newowner->pi_lock);
+ raw_spin_unlock(&newowner->pi_lock);
+ raw_spin_unlock_irq(&pi_state->pi_mutex.wait_lock);
+
return 0;
/*
- * To handle the page fault we need to drop the hash bucket
- * lock here. That gives the other task (either the highest priority
- * waiter itself or the task which stole the rtmutex) the
- * chance to try the fixup of the pi_state. So once we are
- * back from handling the fault we need to check the pi_state
- * after reacquiring the hash bucket lock and before trying to
- * do another fixup. When the fixup has been done already we
- * simply return.
+ * To handle the page fault we need to drop the locks here. That gives
+ * the other task (either the highest priority waiter itself or the
+ * task which stole the rtmutex) the chance to try the fixup of the
+ * pi_state. So once we are back from handling the fault we need to
+ * check the pi_state after reacquiring the locks and before trying to
+ * do another fixup. When the fixup has been done already we simply
+ * return.
+ *
+ * Note: we hold both hb->lock and pi_mutex->wait_lock. We can safely
+ * drop hb->lock since the caller owns the hb -> futex_q relation.
+ * Dropping the pi_mutex->wait_lock requires the state revalidate.
*/
handle_fault:
+ raw_spin_unlock_irq(&pi_state->pi_mutex.wait_lock);
spin_unlock(q->lock_ptr);
ret = fault_in_user_writeable(uaddr);
spin_lock(q->lock_ptr);
+ raw_spin_lock_irq(&pi_state->pi_mutex.wait_lock);
/*
* Check if someone else fixed it for us:
*/
- if (pi_state->owner != oldowner)
- return 0;
+ if (pi_state->owner != oldowner) {
+ ret = 0;
+ goto out_unlock;
+ }
if (ret)
- return ret;
+ goto out_unlock;
goto retry;
+
+out_unlock:
+ raw_spin_unlock_irq(&pi_state->pi_mutex.wait_lock);
+ return ret;
}
static long futex_wait_restart(struct restart_block *restart);
@@ -2231,13 +2339,16 @@
*/
static int fixup_owner(u32 __user *uaddr, struct futex_q *q, int locked)
{
- struct task_struct *owner;
int ret = 0;
if (locked) {
/*
* Got the lock. We might not be the anticipated owner if we
* did a lock-steal - fix up the PI-state in that case:
+ *
+ * We can safely read pi_state->owner without holding wait_lock
+ * because we now own the rt_mutex, only the owner will attempt
+ * to change it.
*/
if (q->pi_state->owner != current)
ret = fixup_pi_state_owner(uaddr, q, current);
@@ -2245,43 +2356,15 @@
}
/*
- * Catch the rare case, where the lock was released when we were on the
- * way back before we locked the hash bucket.
- */
- if (q->pi_state->owner == current) {
- /*
- * Try to get the rt_mutex now. This might fail as some other
- * task acquired the rt_mutex after we removed ourself from the
- * rt_mutex waiters list.
- */
- if (rt_mutex_trylock(&q->pi_state->pi_mutex)) {
- locked = 1;
- goto out;
- }
-
- /*
- * pi_state is incorrect, some other task did a lock steal and
- * we returned due to timeout or signal without taking the
- * rt_mutex. Too late.
- */
- raw_spin_lock_irq(&q->pi_state->pi_mutex.wait_lock);
- owner = rt_mutex_owner(&q->pi_state->pi_mutex);
- if (!owner)
- owner = rt_mutex_next_owner(&q->pi_state->pi_mutex);
- raw_spin_unlock_irq(&q->pi_state->pi_mutex.wait_lock);
- ret = fixup_pi_state_owner(uaddr, q, owner);
- goto out;
- }
-
- /*
* Paranoia check. If we did not take the lock, then we should not be
* the owner of the rt_mutex.
*/
- if (rt_mutex_owner(&q->pi_state->pi_mutex) == current)
+ if (rt_mutex_owner(&q->pi_state->pi_mutex) == current) {
printk(KERN_ERR "fixup_owner: ret = %d pi-mutex: %p "
"pi-state %p\n", ret,
q->pi_state->pi_mutex.owner,
q->pi_state->owner);
+ }
out:
return ret ? ret : locked;
@@ -2505,6 +2588,8 @@
ktime_t *time, int trylock)
{
struct hrtimer_sleeper timeout, *to = NULL;
+ struct futex_pi_state *pi_state = NULL;
+ struct rt_mutex_waiter rt_waiter;
struct futex_hash_bucket *hb;
struct futex_q q = futex_q_init;
int res, ret;
@@ -2557,25 +2642,68 @@
}
}
+ WARN_ON(!q.pi_state);
+
/*
* Only actually queue now that the atomic ops are done:
*/
- queue_me(&q, hb);
+ __queue_me(&q, hb);
- WARN_ON(!q.pi_state);
- /*
- * Block on the PI mutex:
- */
- if (!trylock) {
- ret = rt_mutex_timed_futex_lock(&q.pi_state->pi_mutex, to);
- } else {
- ret = rt_mutex_trylock(&q.pi_state->pi_mutex);
+ if (trylock) {
+ ret = rt_mutex_futex_trylock(&q.pi_state->pi_mutex);
/* Fixup the trylock return value: */
ret = ret ? 0 : -EWOULDBLOCK;
+ goto no_block;
}
+ rt_mutex_init_waiter(&rt_waiter);
+
+ /*
+ * On PREEMPT_RT_FULL, when hb->lock becomes an rt_mutex, we must not
+ * hold it while doing rt_mutex_start_proxy(), because then it will
+ * include hb->lock in the blocking chain, even through we'll not in
+ * fact hold it while blocking. This will lead it to report -EDEADLK
+ * and BUG when futex_unlock_pi() interleaves with this.
+ *
+ * Therefore acquire wait_lock while holding hb->lock, but drop the
+ * latter before calling rt_mutex_start_proxy_lock(). This still fully
+ * serializes against futex_unlock_pi() as that does the exact same
+ * lock handoff sequence.
+ */
+ raw_spin_lock_irq(&q.pi_state->pi_mutex.wait_lock);
+ spin_unlock(q.lock_ptr);
+ ret = __rt_mutex_start_proxy_lock(&q.pi_state->pi_mutex, &rt_waiter, current);
+ raw_spin_unlock_irq(&q.pi_state->pi_mutex.wait_lock);
+
+ if (ret) {
+ if (ret == 1)
+ ret = 0;
+
+ spin_lock(q.lock_ptr);
+ goto no_block;
+ }
+
+
+ if (unlikely(to))
+ hrtimer_start_expires(&to->timer, HRTIMER_MODE_ABS);
+
+ ret = rt_mutex_wait_proxy_lock(&q.pi_state->pi_mutex, to, &rt_waiter);
+
spin_lock(q.lock_ptr);
/*
+ * If we failed to acquire the lock (signal/timeout), we must
+ * first acquire the hb->lock before removing the lock from the
+ * rt_mutex waitqueue, such that we can keep the hb and rt_mutex
+ * wait lists consistent.
+ *
+ * In particular; it is important that futex_unlock_pi() can not
+ * observe this inconsistency.
+ */
+ if (ret && !rt_mutex_cleanup_proxy_lock(&q.pi_state->pi_mutex, &rt_waiter))
+ ret = 0;
+
+no_block:
+ /*
* Fixup the pi_state owner and possibly acquire the lock if we
* haven't already.
*/
@@ -2591,12 +2719,19 @@
* If fixup_owner() faulted and was unable to handle the fault, unlock
* it and return the fault to userspace.
*/
- if (ret && (rt_mutex_owner(&q.pi_state->pi_mutex) == current))
- rt_mutex_unlock(&q.pi_state->pi_mutex);
+ if (ret && (rt_mutex_owner(&q.pi_state->pi_mutex) == current)) {
+ pi_state = q.pi_state;
+ get_pi_state(pi_state);
+ }
/* Unqueue and drop the lock */
unqueue_me_pi(&q);
+ if (pi_state) {
+ rt_mutex_futex_unlock(&pi_state->pi_mutex);
+ put_pi_state(pi_state);
+ }
+
goto out_put_key;
out_unlock_put_key:
@@ -2605,8 +2740,10 @@
out_put_key:
put_futex_key(&q.key);
out:
- if (to)
+ if (to) {
+ hrtimer_cancel(&to->timer);
destroy_hrtimer_on_stack(&to->timer);
+ }
return ret != -EINTR ? ret : -ERESTARTNOINTR;
uaddr_faulted:
@@ -2633,7 +2770,7 @@
u32 uninitialized_var(curval), uval, vpid = task_pid_vnr(current);
union futex_key key = FUTEX_KEY_INIT;
struct futex_hash_bucket *hb;
- struct futex_q *match;
+ struct futex_q *top_waiter;
int ret;
retry:
@@ -2657,12 +2794,37 @@
* all and we at least want to know if user space fiddled
* with the futex value instead of blindly unlocking.
*/
- match = futex_top_waiter(hb, &key);
- if (match) {
- ret = wake_futex_pi(uaddr, uval, match, hb);
+ top_waiter = futex_top_waiter(hb, &key);
+ if (top_waiter) {
+ struct futex_pi_state *pi_state = top_waiter->pi_state;
+
+ ret = -EINVAL;
+ if (!pi_state)
+ goto out_unlock;
+
/*
- * In case of success wake_futex_pi dropped the hash
- * bucket lock.
+ * If current does not own the pi_state then the futex is
+ * inconsistent and user space fiddled with the futex value.
+ */
+ if (pi_state->owner != current)
+ goto out_unlock;
+
+ get_pi_state(pi_state);
+ /*
+ * By taking wait_lock while still holding hb->lock, we ensure
+ * there is no point where we hold neither; and therefore
+ * wake_futex_pi() must observe a state consistent with what we
+ * observed.
+ */
+ raw_spin_lock_irq(&pi_state->pi_mutex.wait_lock);
+ spin_unlock(&hb->lock);
+
+ ret = wake_futex_pi(uaddr, uval, pi_state);
+
+ put_pi_state(pi_state);
+
+ /*
+ * Success, we're done! No tricky corner cases.
*/
if (!ret)
goto out_putkey;
@@ -2677,7 +2839,6 @@
* setting the FUTEX_WAITERS bit. Try again.
*/
if (ret == -EAGAIN) {
- spin_unlock(&hb->lock);
put_futex_key(&key);
goto retry;
}
@@ -2685,7 +2846,7 @@
* wake_futex_pi has detected invalid state. Tell user
* space.
*/
- goto out_unlock;
+ goto out_putkey;
}
/*
@@ -2695,8 +2856,10 @@
* preserve the WAITERS bit not the OWNER_DIED one. We are the
* owner.
*/
- if (cmpxchg_futex_value_locked(&curval, uaddr, uval, 0))
+ if (cmpxchg_futex_value_locked(&curval, uaddr, uval, 0)) {
+ spin_unlock(&hb->lock);
goto pi_faulted;
+ }
/*
* If uval has changed, let user space handle it.
@@ -2710,7 +2873,6 @@
return ret;
pi_faulted:
- spin_unlock(&hb->lock);
put_futex_key(&key);
ret = fault_in_user_writeable(uaddr);
@@ -2814,6 +2976,7 @@
u32 __user *uaddr2)
{
struct hrtimer_sleeper timeout, *to = NULL;
+ struct futex_pi_state *pi_state = NULL;
struct rt_mutex_waiter rt_waiter;
struct futex_hash_bucket *hb;
union futex_key key2 = FUTEX_KEY_INIT;
@@ -2840,10 +3003,7 @@
* The waiter is allocated on our stack, manipulated by the requeue
* code while we sleep on uaddr.
*/
- debug_rt_mutex_init_waiter(&rt_waiter);
- RB_CLEAR_NODE(&rt_waiter.pi_tree_entry);
- RB_CLEAR_NODE(&rt_waiter.tree_entry);
- rt_waiter.task = NULL;
+ rt_mutex_init_waiter(&rt_waiter);
ret = get_futex_key(uaddr2, flags & FLAGS_SHARED, &key2, VERIFY_WRITE);
if (unlikely(ret != 0))
@@ -2898,8 +3058,10 @@
if (q.pi_state && (q.pi_state->owner != current)) {
spin_lock(q.lock_ptr);
ret = fixup_pi_state_owner(uaddr2, &q, current);
- if (ret && rt_mutex_owner(&q.pi_state->pi_mutex) == current)
- rt_mutex_unlock(&q.pi_state->pi_mutex);
+ if (ret && rt_mutex_owner(&q.pi_state->pi_mutex) == current) {
+ pi_state = q.pi_state;
+ get_pi_state(pi_state);
+ }
/*
* Drop the reference to the pi state which
* the requeue_pi() code acquired for us.
@@ -2917,10 +3079,13 @@
*/
WARN_ON(!q.pi_state);
pi_mutex = &q.pi_state->pi_mutex;
- ret = rt_mutex_finish_proxy_lock(pi_mutex, to, &rt_waiter);
- debug_rt_mutex_free_waiter(&rt_waiter);
+ ret = rt_mutex_wait_proxy_lock(pi_mutex, to, &rt_waiter);
spin_lock(q.lock_ptr);
+ if (ret && !rt_mutex_cleanup_proxy_lock(pi_mutex, &rt_waiter))
+ ret = 0;
+
+ debug_rt_mutex_free_waiter(&rt_waiter);
/*
* Fixup the pi_state owner and possibly acquire the lock if we
* haven't already.
@@ -2938,13 +3103,20 @@
* the fault, unlock the rt_mutex and return the fault to
* userspace.
*/
- if (ret && rt_mutex_owner(pi_mutex) == current)
- rt_mutex_unlock(pi_mutex);
+ if (ret && rt_mutex_owner(&q.pi_state->pi_mutex) == current) {
+ pi_state = q.pi_state;
+ get_pi_state(pi_state);
+ }
/* Unqueue and drop the lock. */
unqueue_me_pi(&q);
}
+ if (pi_state) {
+ rt_mutex_futex_unlock(&pi_state->pi_mutex);
+ put_pi_state(pi_state);
+ }
+
if (ret == -EINTR) {
/*
* We've already been requeued, but cannot restart by calling
diff --git a/kernel/irq/affinity.c b/kernel/irq/affinity.c
index d052947..e2d356d 100644
--- a/kernel/irq/affinity.c
+++ b/kernel/irq/affinity.c
@@ -98,7 +98,7 @@
int ncpus, v, vecs_to_assign, vecs_per_node;
/* Spread the vectors per node */
- vecs_per_node = (affv - curvec) / nodes;
+ vecs_per_node = (affv - (curvec - affd->pre_vectors)) / nodes;
/* Get the cpus on this node which are in the mask */
cpumask_and(nmsk, cpu_online_mask, cpumask_of_node(n));
diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
index be3c34e..686be4b 100644
--- a/kernel/irq/chip.c
+++ b/kernel/irq/chip.c
@@ -348,7 +348,10 @@
irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS);
raw_spin_unlock_irq(&desc->lock);
- action_ret = action->thread_fn(action->irq, action->dev_id);
+ action_ret = IRQ_NONE;
+ for_each_action_of_desc(desc, action)
+ action_ret |= action->thread_fn(action->irq, action->dev_id);
+
if (!noirqdebug)
note_interrupt(desc, action_ret);
diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index a4afe5c..ae1c90f 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -852,7 +852,7 @@
* This code is triggered unconditionally. Check the affinity
* mask pointer. For CPU_MASK_OFFSTACK=n this is optimized out.
*/
- if (desc->irq_common_data.affinity)
+ if (cpumask_available(desc->irq_common_data.affinity))
cpumask_copy(mask, desc->irq_common_data.affinity);
else
valid = false;
@@ -1212,8 +1212,10 @@
* set the trigger type must match. Also all must
* agree on ONESHOT.
*/
+ unsigned int oldtype = irqd_get_trigger_type(&desc->irq_data);
+
if (!((old->flags & new->flags) & IRQF_SHARED) ||
- ((old->flags ^ new->flags) & IRQF_TRIGGER_MASK) ||
+ (oldtype != (new->flags & IRQF_TRIGGER_MASK)) ||
((old->flags ^ new->flags) & IRQF_ONESHOT))
goto mismatch;
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 699c5bc..d733479 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -1391,21 +1391,19 @@
* This returns encoded errors if it fails to look up symbol or invalid
* combination of parameters.
*/
-static kprobe_opcode_t *kprobe_addr(struct kprobe *p)
+static kprobe_opcode_t *_kprobe_addr(kprobe_opcode_t *addr,
+ const char *symbol_name, unsigned int offset)
{
- kprobe_opcode_t *addr = p->addr;
-
- if ((p->symbol_name && p->addr) ||
- (!p->symbol_name && !p->addr))
+ if ((symbol_name && addr) || (!symbol_name && !addr))
goto invalid;
- if (p->symbol_name) {
- kprobe_lookup_name(p->symbol_name, addr);
+ if (symbol_name) {
+ kprobe_lookup_name(symbol_name, addr);
if (!addr)
return ERR_PTR(-ENOENT);
}
- addr = (kprobe_opcode_t *)(((char *)addr) + p->offset);
+ addr = (kprobe_opcode_t *)(((char *)addr) + offset);
if (addr)
return addr;
@@ -1413,6 +1411,11 @@
return ERR_PTR(-EINVAL);
}
+static kprobe_opcode_t *kprobe_addr(struct kprobe *p)
+{
+ return _kprobe_addr(p->addr, p->symbol_name, p->offset);
+}
+
/* Check passed kprobe is valid and return kprobe in kprobe_table. */
static struct kprobe *__get_valid_kprobe(struct kprobe *p)
{
@@ -1740,11 +1743,12 @@
}
EXPORT_SYMBOL_GPL(unregister_kprobes);
-int __weak __kprobes kprobe_exceptions_notify(struct notifier_block *self,
- unsigned long val, void *data)
+int __weak kprobe_exceptions_notify(struct notifier_block *self,
+ unsigned long val, void *data)
{
return NOTIFY_DONE;
}
+NOKPROBE_SYMBOL(kprobe_exceptions_notify);
static struct notifier_block kprobe_exceptions_nb = {
.notifier_call = kprobe_exceptions_notify,
@@ -1875,6 +1879,25 @@
}
NOKPROBE_SYMBOL(pre_handler_kretprobe);
+bool __weak arch_function_offset_within_entry(unsigned long offset)
+{
+ return !offset;
+}
+
+bool function_offset_within_entry(kprobe_opcode_t *addr, const char *sym, unsigned long offset)
+{
+ kprobe_opcode_t *kp_addr = _kprobe_addr(addr, sym, offset);
+
+ if (IS_ERR(kp_addr))
+ return false;
+
+ if (!kallsyms_lookup_size_offset((unsigned long)kp_addr, NULL, &offset) ||
+ !arch_function_offset_within_entry(offset))
+ return false;
+
+ return true;
+}
+
int register_kretprobe(struct kretprobe *rp)
{
int ret = 0;
@@ -1882,6 +1905,9 @@
int i;
void *addr;
+ if (!function_offset_within_entry(rp->kp.addr, rp->kp.symbol_name, rp->kp.offset))
+ return -EINVAL;
+
if (kretprobe_blacklist_size) {
addr = kprobe_addr(&rp->kp);
if (IS_ERR(addr))
diff --git a/kernel/livepatch/Makefile b/kernel/livepatch/Makefile
index e8780c0..2b8bdb1 100644
--- a/kernel/livepatch/Makefile
+++ b/kernel/livepatch/Makefile
@@ -1,3 +1,3 @@
obj-$(CONFIG_LIVEPATCH) += livepatch.o
-livepatch-objs := core.o
+livepatch-objs := core.o patch.o transition.o
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index af46438..b9628e4 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -24,61 +24,31 @@
#include <linux/kernel.h>
#include <linux/mutex.h>
#include <linux/slab.h>
-#include <linux/ftrace.h>
#include <linux/list.h>
#include <linux/kallsyms.h>
#include <linux/livepatch.h>
#include <linux/elf.h>
#include <linux/moduleloader.h>
+#include <linux/completion.h>
#include <asm/cacheflush.h>
-
-/**
- * struct klp_ops - structure for tracking registered ftrace ops structs
- *
- * A single ftrace_ops is shared between all enabled replacement functions
- * (klp_func structs) which have the same old_addr. This allows the switch
- * between function versions to happen instantaneously by updating the klp_ops
- * struct's func_stack list. The winner is the klp_func at the top of the
- * func_stack (front of the list).
- *
- * @node: node for the global klp_ops list
- * @func_stack: list head for the stack of klp_func's (active func is on top)
- * @fops: registered ftrace ops struct
- */
-struct klp_ops {
- struct list_head node;
- struct list_head func_stack;
- struct ftrace_ops fops;
-};
+#include "core.h"
+#include "patch.h"
+#include "transition.h"
/*
- * The klp_mutex protects the global lists and state transitions of any
- * structure reachable from them. References to any structure must be obtained
- * under mutex protection (except in klp_ftrace_handler(), which uses RCU to
- * ensure it gets consistent data).
+ * klp_mutex is a coarse lock which serializes access to klp data. All
+ * accesses to klp-related variables and structures must have mutex protection,
+ * except within the following functions which carefully avoid the need for it:
+ *
+ * - klp_ftrace_handler()
+ * - klp_update_patch_state()
*/
-static DEFINE_MUTEX(klp_mutex);
+DEFINE_MUTEX(klp_mutex);
static LIST_HEAD(klp_patches);
-static LIST_HEAD(klp_ops);
static struct kobject *klp_root_kobj;
-static struct klp_ops *klp_find_ops(unsigned long old_addr)
-{
- struct klp_ops *ops;
- struct klp_func *func;
-
- list_for_each_entry(ops, &klp_ops, node) {
- func = list_first_entry(&ops->func_stack, struct klp_func,
- stack_node);
- if (func->old_addr == old_addr)
- return ops;
- }
-
- return NULL;
-}
-
static bool klp_is_module(struct klp_object *obj)
{
return obj->name;
@@ -117,7 +87,6 @@
mutex_unlock(&module_mutex);
}
-/* klp_mutex must be held by caller */
static bool klp_is_patch_registered(struct klp_patch *patch)
{
struct klp_patch *mypatch;
@@ -182,7 +151,10 @@
};
mutex_lock(&module_mutex);
- kallsyms_on_each_symbol(klp_find_callback, &args);
+ if (objname)
+ module_kallsyms_on_each_symbol(klp_find_callback, &args);
+ else
+ kallsyms_on_each_symbol(klp_find_callback, &args);
mutex_unlock(&module_mutex);
/*
@@ -233,7 +205,7 @@
for (i = 0; i < relasec->sh_size / sizeof(Elf_Rela); i++) {
sym = pmod->core_kallsyms.symtab + ELF_R_SYM(relas[i].r_info);
if (sym->st_shndx != SHN_LIVEPATCH) {
- pr_err("symbol %s is not marked as a livepatch symbol",
+ pr_err("symbol %s is not marked as a livepatch symbol\n",
strtab + sym->st_name);
return -EINVAL;
}
@@ -243,7 +215,7 @@
".klp.sym.%55[^.].%127[^,],%lu",
objname, symname, &sympos);
if (cnt != 3) {
- pr_err("symbol %s has an incorrectly formatted name",
+ pr_err("symbol %s has an incorrectly formatted name\n",
strtab + sym->st_name);
return -EINVAL;
}
@@ -288,7 +260,7 @@
*/
cnt = sscanf(secname, ".klp.rela.%55[^.]", sec_objname);
if (cnt != 1) {
- pr_err("section %s has an incorrectly formatted name",
+ pr_err("section %s has an incorrectly formatted name\n",
secname);
ret = -EINVAL;
break;
@@ -311,191 +283,30 @@
return ret;
}
-static void notrace klp_ftrace_handler(unsigned long ip,
- unsigned long parent_ip,
- struct ftrace_ops *fops,
- struct pt_regs *regs)
-{
- struct klp_ops *ops;
- struct klp_func *func;
-
- ops = container_of(fops, struct klp_ops, fops);
-
- rcu_read_lock();
- func = list_first_or_null_rcu(&ops->func_stack, struct klp_func,
- stack_node);
- if (WARN_ON_ONCE(!func))
- goto unlock;
-
- klp_arch_set_pc(regs, (unsigned long)func->new_func);
-unlock:
- rcu_read_unlock();
-}
-
-/*
- * Convert a function address into the appropriate ftrace location.
- *
- * Usually this is just the address of the function, but on some architectures
- * it's more complicated so allow them to provide a custom behaviour.
- */
-#ifndef klp_get_ftrace_location
-static unsigned long klp_get_ftrace_location(unsigned long faddr)
-{
- return faddr;
-}
-#endif
-
-static void klp_disable_func(struct klp_func *func)
-{
- struct klp_ops *ops;
-
- if (WARN_ON(func->state != KLP_ENABLED))
- return;
- if (WARN_ON(!func->old_addr))
- return;
-
- ops = klp_find_ops(func->old_addr);
- if (WARN_ON(!ops))
- return;
-
- if (list_is_singular(&ops->func_stack)) {
- unsigned long ftrace_loc;
-
- ftrace_loc = klp_get_ftrace_location(func->old_addr);
- if (WARN_ON(!ftrace_loc))
- return;
-
- WARN_ON(unregister_ftrace_function(&ops->fops));
- WARN_ON(ftrace_set_filter_ip(&ops->fops, ftrace_loc, 1, 0));
-
- list_del_rcu(&func->stack_node);
- list_del(&ops->node);
- kfree(ops);
- } else {
- list_del_rcu(&func->stack_node);
- }
-
- func->state = KLP_DISABLED;
-}
-
-static int klp_enable_func(struct klp_func *func)
-{
- struct klp_ops *ops;
- int ret;
-
- if (WARN_ON(!func->old_addr))
- return -EINVAL;
-
- if (WARN_ON(func->state != KLP_DISABLED))
- return -EINVAL;
-
- ops = klp_find_ops(func->old_addr);
- if (!ops) {
- unsigned long ftrace_loc;
-
- ftrace_loc = klp_get_ftrace_location(func->old_addr);
- if (!ftrace_loc) {
- pr_err("failed to find location for function '%s'\n",
- func->old_name);
- return -EINVAL;
- }
-
- ops = kzalloc(sizeof(*ops), GFP_KERNEL);
- if (!ops)
- return -ENOMEM;
-
- ops->fops.func = klp_ftrace_handler;
- ops->fops.flags = FTRACE_OPS_FL_SAVE_REGS |
- FTRACE_OPS_FL_DYNAMIC |
- FTRACE_OPS_FL_IPMODIFY;
-
- list_add(&ops->node, &klp_ops);
-
- INIT_LIST_HEAD(&ops->func_stack);
- list_add_rcu(&func->stack_node, &ops->func_stack);
-
- ret = ftrace_set_filter_ip(&ops->fops, ftrace_loc, 0, 0);
- if (ret) {
- pr_err("failed to set ftrace filter for function '%s' (%d)\n",
- func->old_name, ret);
- goto err;
- }
-
- ret = register_ftrace_function(&ops->fops);
- if (ret) {
- pr_err("failed to register ftrace handler for function '%s' (%d)\n",
- func->old_name, ret);
- ftrace_set_filter_ip(&ops->fops, ftrace_loc, 1, 0);
- goto err;
- }
-
-
- } else {
- list_add_rcu(&func->stack_node, &ops->func_stack);
- }
-
- func->state = KLP_ENABLED;
-
- return 0;
-
-err:
- list_del_rcu(&func->stack_node);
- list_del(&ops->node);
- kfree(ops);
- return ret;
-}
-
-static void klp_disable_object(struct klp_object *obj)
-{
- struct klp_func *func;
-
- klp_for_each_func(obj, func)
- if (func->state == KLP_ENABLED)
- klp_disable_func(func);
-
- obj->state = KLP_DISABLED;
-}
-
-static int klp_enable_object(struct klp_object *obj)
-{
- struct klp_func *func;
- int ret;
-
- if (WARN_ON(obj->state != KLP_DISABLED))
- return -EINVAL;
-
- if (WARN_ON(!klp_is_object_loaded(obj)))
- return -EINVAL;
-
- klp_for_each_func(obj, func) {
- ret = klp_enable_func(func);
- if (ret) {
- klp_disable_object(obj);
- return ret;
- }
- }
- obj->state = KLP_ENABLED;
-
- return 0;
-}
-
static int __klp_disable_patch(struct klp_patch *patch)
{
- struct klp_object *obj;
+ if (klp_transition_patch)
+ return -EBUSY;
/* enforce stacking: only the last enabled patch can be disabled */
if (!list_is_last(&patch->list, &klp_patches) &&
- list_next_entry(patch, list)->state == KLP_ENABLED)
+ list_next_entry(patch, list)->enabled)
return -EBUSY;
- pr_notice("disabling patch '%s'\n", patch->mod->name);
+ klp_init_transition(patch, KLP_UNPATCHED);
- klp_for_each_object(patch, obj) {
- if (obj->state == KLP_ENABLED)
- klp_disable_object(obj);
- }
+ /*
+ * Enforce the order of the func->transition writes in
+ * klp_init_transition() and the TIF_PATCH_PENDING writes in
+ * klp_start_transition(). In the rare case where klp_ftrace_handler()
+ * is called shortly after klp_update_patch_state() switches the task,
+ * this ensures the handler sees that func->transition is set.
+ */
+ smp_wmb();
- patch->state = KLP_DISABLED;
+ klp_start_transition();
+ klp_try_complete_transition();
+ patch->enabled = false;
return 0;
}
@@ -519,7 +330,7 @@
goto err;
}
- if (patch->state == KLP_DISABLED) {
+ if (!patch->enabled) {
ret = -EINVAL;
goto err;
}
@@ -537,32 +348,61 @@
struct klp_object *obj;
int ret;
- if (WARN_ON(patch->state != KLP_DISABLED))
+ if (klp_transition_patch)
+ return -EBUSY;
+
+ if (WARN_ON(patch->enabled))
return -EINVAL;
/* enforce stacking: only the first disabled patch can be enabled */
if (patch->list.prev != &klp_patches &&
- list_prev_entry(patch, list)->state == KLP_DISABLED)
+ !list_prev_entry(patch, list)->enabled)
return -EBUSY;
+ /*
+ * A reference is taken on the patch module to prevent it from being
+ * unloaded.
+ *
+ * Note: For immediate (no consistency model) patches we don't allow
+ * patch modules to unload since there is no safe/sane method to
+ * determine if a thread is still running in the patched code contained
+ * in the patch module once the ftrace registration is successful.
+ */
+ if (!try_module_get(patch->mod))
+ return -ENODEV;
+
pr_notice("enabling patch '%s'\n", patch->mod->name);
+ klp_init_transition(patch, KLP_PATCHED);
+
+ /*
+ * Enforce the order of the func->transition writes in
+ * klp_init_transition() and the ops->func_stack writes in
+ * klp_patch_object(), so that klp_ftrace_handler() will see the
+ * func->transition updates before the handler is registered and the
+ * new funcs become visible to the handler.
+ */
+ smp_wmb();
+
klp_for_each_object(patch, obj) {
if (!klp_is_object_loaded(obj))
continue;
- ret = klp_enable_object(obj);
- if (ret)
- goto unregister;
+ ret = klp_patch_object(obj);
+ if (ret) {
+ pr_warn("failed to enable patch '%s'\n",
+ patch->mod->name);
+
+ klp_cancel_transition();
+ return ret;
+ }
}
- patch->state = KLP_ENABLED;
+ klp_start_transition();
+ klp_try_complete_transition();
+ patch->enabled = true;
return 0;
-
-unregister:
- WARN_ON(__klp_disable_patch(patch));
- return ret;
}
/**
@@ -599,6 +439,7 @@
* /sys/kernel/livepatch
* /sys/kernel/livepatch/<patch>
* /sys/kernel/livepatch/<patch>/enabled
+ * /sys/kernel/livepatch/<patch>/transition
* /sys/kernel/livepatch/<patch>/<object>
* /sys/kernel/livepatch/<patch>/<object>/<function,sympos>
*/
@@ -608,26 +449,34 @@
{
struct klp_patch *patch;
int ret;
- unsigned long val;
+ bool enabled;
- ret = kstrtoul(buf, 10, &val);
+ ret = kstrtobool(buf, &enabled);
if (ret)
- return -EINVAL;
-
- if (val != KLP_DISABLED && val != KLP_ENABLED)
- return -EINVAL;
+ return ret;
patch = container_of(kobj, struct klp_patch, kobj);
mutex_lock(&klp_mutex);
- if (val == patch->state) {
+ if (!klp_is_patch_registered(patch)) {
+ /*
+ * Module with the patch could either disappear meanwhile or is
+ * not properly initialized yet.
+ */
+ ret = -EINVAL;
+ goto err;
+ }
+
+ if (patch->enabled == enabled) {
/* already in requested state */
ret = -EINVAL;
goto err;
}
- if (val == KLP_ENABLED) {
+ if (patch == klp_transition_patch) {
+ klp_reverse_transition();
+ } else if (enabled) {
ret = __klp_enable_patch(patch);
if (ret)
goto err;
@@ -652,21 +501,33 @@
struct klp_patch *patch;
patch = container_of(kobj, struct klp_patch, kobj);
- return snprintf(buf, PAGE_SIZE-1, "%d\n", patch->state);
+ return snprintf(buf, PAGE_SIZE-1, "%d\n", patch->enabled);
+}
+
+static ssize_t transition_show(struct kobject *kobj,
+ struct kobj_attribute *attr, char *buf)
+{
+ struct klp_patch *patch;
+
+ patch = container_of(kobj, struct klp_patch, kobj);
+ return snprintf(buf, PAGE_SIZE-1, "%d\n",
+ patch == klp_transition_patch);
}
static struct kobj_attribute enabled_kobj_attr = __ATTR_RW(enabled);
+static struct kobj_attribute transition_kobj_attr = __ATTR_RO(transition);
static struct attribute *klp_patch_attrs[] = {
&enabled_kobj_attr.attr,
+ &transition_kobj_attr.attr,
NULL
};
static void klp_kobj_release_patch(struct kobject *kobj)
{
- /*
- * Once we have a consistency model we'll need to module_put() the
- * patch module here. See klp_register_patch() for more details.
- */
+ struct klp_patch *patch;
+
+ patch = container_of(kobj, struct klp_patch, kobj);
+ complete(&patch->finish);
}
static struct kobj_type klp_ktype_patch = {
@@ -737,7 +598,6 @@
klp_free_objects_limited(patch, NULL);
if (!list_empty(&patch->list))
list_del(&patch->list);
- kobject_put(&patch->kobj);
}
static int klp_init_func(struct klp_object *obj, struct klp_func *func)
@@ -746,7 +606,8 @@
return -EINVAL;
INIT_LIST_HEAD(&func->stack_node);
- func->state = KLP_DISABLED;
+ func->patched = false;
+ func->transition = false;
/* The format for the sysfs directory is <function,sympos> where sympos
* is the nth occurrence of this symbol in kallsyms for the patched
@@ -787,6 +648,22 @@
&func->old_addr);
if (ret)
return ret;
+
+ ret = kallsyms_lookup_size_offset(func->old_addr,
+ &func->old_size, NULL);
+ if (!ret) {
+ pr_err("kallsyms size lookup failed for '%s'\n",
+ func->old_name);
+ return -ENOENT;
+ }
+
+ ret = kallsyms_lookup_size_offset((unsigned long)func->new_func,
+ &func->new_size, NULL);
+ if (!ret) {
+ pr_err("kallsyms size lookup failed for '%s' replacement\n",
+ func->old_name);
+ return -ENOENT;
+ }
}
return 0;
@@ -801,7 +678,7 @@
if (!obj->funcs)
return -EINVAL;
- obj->state = KLP_DISABLED;
+ obj->patched = false;
obj->mod = NULL;
klp_find_object_module(obj);
@@ -842,12 +719,15 @@
mutex_lock(&klp_mutex);
- patch->state = KLP_DISABLED;
+ patch->enabled = false;
+ init_completion(&patch->finish);
ret = kobject_init_and_add(&patch->kobj, &klp_ktype_patch,
klp_root_kobj, "%s", patch->mod->name);
- if (ret)
- goto unlock;
+ if (ret) {
+ mutex_unlock(&klp_mutex);
+ return ret;
+ }
klp_for_each_object(patch, obj) {
ret = klp_init_object(patch, obj);
@@ -863,9 +743,12 @@
free:
klp_free_objects_limited(patch, obj);
- kobject_put(&patch->kobj);
-unlock:
+
mutex_unlock(&klp_mutex);
+
+ kobject_put(&patch->kobj);
+ wait_for_completion(&patch->finish);
+
return ret;
}
@@ -879,23 +762,29 @@
*/
int klp_unregister_patch(struct klp_patch *patch)
{
- int ret = 0;
+ int ret;
mutex_lock(&klp_mutex);
if (!klp_is_patch_registered(patch)) {
ret = -EINVAL;
- goto out;
+ goto err;
}
- if (patch->state == KLP_ENABLED) {
+ if (patch->enabled) {
ret = -EBUSY;
- goto out;
+ goto err;
}
klp_free_patch(patch);
-out:
+ mutex_unlock(&klp_mutex);
+
+ kobject_put(&patch->kobj);
+ wait_for_completion(&patch->finish);
+
+ return 0;
+err:
mutex_unlock(&klp_mutex);
return ret;
}
@@ -908,17 +797,18 @@
* Initializes the data structure associated with the patch and
* creates the sysfs interface.
*
+ * There is no need to take the reference on the patch module here. It is done
+ * later when the patch is enabled.
+ *
* Return: 0 on success, otherwise error
*/
int klp_register_patch(struct klp_patch *patch)
{
- int ret;
-
if (!patch || !patch->mod)
return -EINVAL;
if (!is_livepatch_module(patch->mod)) {
- pr_err("module %s is not marked as a livepatch module",
+ pr_err("module %s is not marked as a livepatch module\n",
patch->mod->name);
return -EINVAL;
}
@@ -927,20 +817,16 @@
return -ENODEV;
/*
- * A reference is taken on the patch module to prevent it from being
- * unloaded. Right now, we don't allow patch modules to unload since
- * there is currently no method to determine if a thread is still
- * running in the patched code contained in the patch module once
- * the ftrace registration is successful.
+ * Architectures without reliable stack traces have to set
+ * patch->immediate because there's currently no way to patch kthreads
+ * with the consistency model.
*/
- if (!try_module_get(patch->mod))
- return -ENODEV;
+ if (!klp_have_reliable_stack() && !patch->immediate) {
+ pr_err("This architecture doesn't have support for the livepatch consistency model.\n");
+ return -ENOSYS;
+ }
- ret = klp_init_patch(patch);
- if (ret)
- module_put(patch->mod);
-
- return ret;
+ return klp_init_patch(patch);
}
EXPORT_SYMBOL_GPL(klp_register_patch);
@@ -975,13 +861,17 @@
goto err;
}
- if (patch->state == KLP_DISABLED)
+ /*
+ * Only patch the module if the patch is enabled or is
+ * in transition.
+ */
+ if (!patch->enabled && patch != klp_transition_patch)
break;
pr_notice("applying patch '%s' to loading module '%s'\n",
patch->mod->name, obj->mod->name);
- ret = klp_enable_object(obj);
+ ret = klp_patch_object(obj);
if (ret) {
pr_warn("failed to apply patch '%s' to module '%s' (%d)\n",
patch->mod->name, obj->mod->name, ret);
@@ -1032,10 +922,14 @@
if (!klp_is_module(obj) || strcmp(obj->name, mod->name))
continue;
- if (patch->state != KLP_DISABLED) {
+ /*
+ * Only unpatch the module if the patch is enabled or
+ * is in transition.
+ */
+ if (patch->enabled || patch == klp_transition_patch) {
pr_notice("reverting patch '%s' on unloading module '%s'\n",
patch->mod->name, obj->mod->name);
- klp_disable_object(obj);
+ klp_unpatch_object(obj);
}
klp_free_object_loaded(obj);
diff --git a/kernel/livepatch/core.h b/kernel/livepatch/core.h
new file mode 100644
index 0000000..c74f24c
--- /dev/null
+++ b/kernel/livepatch/core.h
@@ -0,0 +1,6 @@
+#ifndef _LIVEPATCH_CORE_H
+#define _LIVEPATCH_CORE_H
+
+extern struct mutex klp_mutex;
+
+#endif /* _LIVEPATCH_CORE_H */
diff --git a/kernel/livepatch/patch.c b/kernel/livepatch/patch.c
new file mode 100644
index 0000000..f826903
--- /dev/null
+++ b/kernel/livepatch/patch.c
@@ -0,0 +1,272 @@
+/*
+ * patch.c - livepatch patching functions
+ *
+ * Copyright (C) 2014 Seth Jennings <sjenning@redhat.com>
+ * Copyright (C) 2014 SUSE
+ * Copyright (C) 2015 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/livepatch.h>
+#include <linux/list.h>
+#include <linux/ftrace.h>
+#include <linux/rculist.h>
+#include <linux/slab.h>
+#include <linux/bug.h>
+#include <linux/printk.h>
+#include "patch.h"
+#include "transition.h"
+
+static LIST_HEAD(klp_ops);
+
+struct klp_ops *klp_find_ops(unsigned long old_addr)
+{
+ struct klp_ops *ops;
+ struct klp_func *func;
+
+ list_for_each_entry(ops, &klp_ops, node) {
+ func = list_first_entry(&ops->func_stack, struct klp_func,
+ stack_node);
+ if (func->old_addr == old_addr)
+ return ops;
+ }
+
+ return NULL;
+}
+
+static void notrace klp_ftrace_handler(unsigned long ip,
+ unsigned long parent_ip,
+ struct ftrace_ops *fops,
+ struct pt_regs *regs)
+{
+ struct klp_ops *ops;
+ struct klp_func *func;
+ int patch_state;
+
+ ops = container_of(fops, struct klp_ops, fops);
+
+ rcu_read_lock();
+
+ func = list_first_or_null_rcu(&ops->func_stack, struct klp_func,
+ stack_node);
+
+ /*
+ * func should never be NULL because preemption should be disabled here
+ * and unregister_ftrace_function() does the equivalent of a
+ * synchronize_sched() before the func_stack removal.
+ */
+ if (WARN_ON_ONCE(!func))
+ goto unlock;
+
+ /*
+ * In the enable path, enforce the order of the ops->func_stack and
+ * func->transition reads. The corresponding write barrier is in
+ * __klp_enable_patch().
+ *
+ * (Note that this barrier technically isn't needed in the disable
+ * path. In the rare case where klp_update_patch_state() runs before
+ * this handler, its TIF_PATCH_PENDING read and this func->transition
+ * read need to be ordered. But klp_update_patch_state() already
+ * enforces that.)
+ */
+ smp_rmb();
+
+ if (unlikely(func->transition)) {
+
+ /*
+ * Enforce the order of the func->transition and
+ * current->patch_state reads. Otherwise we could read an
+ * out-of-date task state and pick the wrong function. The
+ * corresponding write barrier is in klp_init_transition().
+ */
+ smp_rmb();
+
+ patch_state = current->patch_state;
+
+ WARN_ON_ONCE(patch_state == KLP_UNDEFINED);
+
+ if (patch_state == KLP_UNPATCHED) {
+ /*
+ * Use the previously patched version of the function.
+ * If no previous patches exist, continue with the
+ * original function.
+ */
+ func = list_entry_rcu(func->stack_node.next,
+ struct klp_func, stack_node);
+
+ if (&func->stack_node == &ops->func_stack)
+ goto unlock;
+ }
+ }
+
+ klp_arch_set_pc(regs, (unsigned long)func->new_func);
+unlock:
+ rcu_read_unlock();
+}
+
+/*
+ * Convert a function address into the appropriate ftrace location.
+ *
+ * Usually this is just the address of the function, but on some architectures
+ * it's more complicated so allow them to provide a custom behaviour.
+ */
+#ifndef klp_get_ftrace_location
+static unsigned long klp_get_ftrace_location(unsigned long faddr)
+{
+ return faddr;
+}
+#endif
+
+static void klp_unpatch_func(struct klp_func *func)
+{
+ struct klp_ops *ops;
+
+ if (WARN_ON(!func->patched))
+ return;
+ if (WARN_ON(!func->old_addr))
+ return;
+
+ ops = klp_find_ops(func->old_addr);
+ if (WARN_ON(!ops))
+ return;
+
+ if (list_is_singular(&ops->func_stack)) {
+ unsigned long ftrace_loc;
+
+ ftrace_loc = klp_get_ftrace_location(func->old_addr);
+ if (WARN_ON(!ftrace_loc))
+ return;
+
+ WARN_ON(unregister_ftrace_function(&ops->fops));
+ WARN_ON(ftrace_set_filter_ip(&ops->fops, ftrace_loc, 1, 0));
+
+ list_del_rcu(&func->stack_node);
+ list_del(&ops->node);
+ kfree(ops);
+ } else {
+ list_del_rcu(&func->stack_node);
+ }
+
+ func->patched = false;
+}
+
+static int klp_patch_func(struct klp_func *func)
+{
+ struct klp_ops *ops;
+ int ret;
+
+ if (WARN_ON(!func->old_addr))
+ return -EINVAL;
+
+ if (WARN_ON(func->patched))
+ return -EINVAL;
+
+ ops = klp_find_ops(func->old_addr);
+ if (!ops) {
+ unsigned long ftrace_loc;
+
+ ftrace_loc = klp_get_ftrace_location(func->old_addr);
+ if (!ftrace_loc) {
+ pr_err("failed to find location for function '%s'\n",
+ func->old_name);
+ return -EINVAL;
+ }
+
+ ops = kzalloc(sizeof(*ops), GFP_KERNEL);
+ if (!ops)
+ return -ENOMEM;
+
+ ops->fops.func = klp_ftrace_handler;
+ ops->fops.flags = FTRACE_OPS_FL_SAVE_REGS |
+ FTRACE_OPS_FL_DYNAMIC |
+ FTRACE_OPS_FL_IPMODIFY;
+
+ list_add(&ops->node, &klp_ops);
+
+ INIT_LIST_HEAD(&ops->func_stack);
+ list_add_rcu(&func->stack_node, &ops->func_stack);
+
+ ret = ftrace_set_filter_ip(&ops->fops, ftrace_loc, 0, 0);
+ if (ret) {
+ pr_err("failed to set ftrace filter for function '%s' (%d)\n",
+ func->old_name, ret);
+ goto err;
+ }
+
+ ret = register_ftrace_function(&ops->fops);
+ if (ret) {
+ pr_err("failed to register ftrace handler for function '%s' (%d)\n",
+ func->old_name, ret);
+ ftrace_set_filter_ip(&ops->fops, ftrace_loc, 1, 0);
+ goto err;
+ }
+
+
+ } else {
+ list_add_rcu(&func->stack_node, &ops->func_stack);
+ }
+
+ func->patched = true;
+
+ return 0;
+
+err:
+ list_del_rcu(&func->stack_node);
+ list_del(&ops->node);
+ kfree(ops);
+ return ret;
+}
+
+void klp_unpatch_object(struct klp_object *obj)
+{
+ struct klp_func *func;
+
+ klp_for_each_func(obj, func)
+ if (func->patched)
+ klp_unpatch_func(func);
+
+ obj->patched = false;
+}
+
+int klp_patch_object(struct klp_object *obj)
+{
+ struct klp_func *func;
+ int ret;
+
+ if (WARN_ON(obj->patched))
+ return -EINVAL;
+
+ klp_for_each_func(obj, func) {
+ ret = klp_patch_func(func);
+ if (ret) {
+ klp_unpatch_object(obj);
+ return ret;
+ }
+ }
+ obj->patched = true;
+
+ return 0;
+}
+
+void klp_unpatch_objects(struct klp_patch *patch)
+{
+ struct klp_object *obj;
+
+ klp_for_each_object(patch, obj)
+ if (obj->patched)
+ klp_unpatch_object(obj);
+}
diff --git a/kernel/livepatch/patch.h b/kernel/livepatch/patch.h
new file mode 100644
index 0000000..0db2271
--- /dev/null
+++ b/kernel/livepatch/patch.h
@@ -0,0 +1,33 @@
+#ifndef _LIVEPATCH_PATCH_H
+#define _LIVEPATCH_PATCH_H
+
+#include <linux/livepatch.h>
+#include <linux/list.h>
+#include <linux/ftrace.h>
+
+/**
+ * struct klp_ops - structure for tracking registered ftrace ops structs
+ *
+ * A single ftrace_ops is shared between all enabled replacement functions
+ * (klp_func structs) which have the same old_addr. This allows the switch
+ * between function versions to happen instantaneously by updating the klp_ops
+ * struct's func_stack list. The winner is the klp_func at the top of the
+ * func_stack (front of the list).
+ *
+ * @node: node for the global klp_ops list
+ * @func_stack: list head for the stack of klp_func's (active func is on top)
+ * @fops: registered ftrace ops struct
+ */
+struct klp_ops {
+ struct list_head node;
+ struct list_head func_stack;
+ struct ftrace_ops fops;
+};
+
+struct klp_ops *klp_find_ops(unsigned long old_addr);
+
+int klp_patch_object(struct klp_object *obj);
+void klp_unpatch_object(struct klp_object *obj);
+void klp_unpatch_objects(struct klp_patch *patch);
+
+#endif /* _LIVEPATCH_PATCH_H */
diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
new file mode 100644
index 0000000..adc0cc6
--- /dev/null
+++ b/kernel/livepatch/transition.c
@@ -0,0 +1,553 @@
+/*
+ * transition.c - Kernel Live Patching transition functions
+ *
+ * Copyright (C) 2015-2016 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/cpu.h>
+#include <linux/stacktrace.h>
+#include "core.h"
+#include "patch.h"
+#include "transition.h"
+#include "../sched/sched.h"
+
+#define MAX_STACK_ENTRIES 100
+#define STACK_ERR_BUF_SIZE 128
+
+struct klp_patch *klp_transition_patch;
+
+static int klp_target_state = KLP_UNDEFINED;
+
+/*
+ * This work can be performed periodically to finish patching or unpatching any
+ * "straggler" tasks which failed to transition in the first attempt.
+ */
+static void klp_transition_work_fn(struct work_struct *work)
+{
+ mutex_lock(&klp_mutex);
+
+ if (klp_transition_patch)
+ klp_try_complete_transition();
+
+ mutex_unlock(&klp_mutex);
+}
+static DECLARE_DELAYED_WORK(klp_transition_work, klp_transition_work_fn);
+
+/*
+ * The transition to the target patch state is complete. Clean up the data
+ * structures.
+ */
+static void klp_complete_transition(void)
+{
+ struct klp_object *obj;
+ struct klp_func *func;
+ struct task_struct *g, *task;
+ unsigned int cpu;
+ bool immediate_func = false;
+
+ if (klp_target_state == KLP_UNPATCHED) {
+ /*
+ * All tasks have transitioned to KLP_UNPATCHED so we can now
+ * remove the new functions from the func_stack.
+ */
+ klp_unpatch_objects(klp_transition_patch);
+
+ /*
+ * Make sure klp_ftrace_handler() can no longer see functions
+ * from this patch on the ops->func_stack. Otherwise, after
+ * func->transition gets cleared, the handler may choose a
+ * removed function.
+ */
+ synchronize_rcu();
+ }
+
+ if (klp_transition_patch->immediate)
+ goto done;
+
+ klp_for_each_object(klp_transition_patch, obj) {
+ klp_for_each_func(obj, func) {
+ func->transition = false;
+ if (func->immediate)
+ immediate_func = true;
+ }
+ }
+
+ if (klp_target_state == KLP_UNPATCHED && !immediate_func)
+ module_put(klp_transition_patch->mod);
+
+ /* Prevent klp_ftrace_handler() from seeing KLP_UNDEFINED state */
+ if (klp_target_state == KLP_PATCHED)
+ synchronize_rcu();
+
+ read_lock(&tasklist_lock);
+ for_each_process_thread(g, task) {
+ WARN_ON_ONCE(test_tsk_thread_flag(task, TIF_PATCH_PENDING));
+ task->patch_state = KLP_UNDEFINED;
+ }
+ read_unlock(&tasklist_lock);
+
+ for_each_possible_cpu(cpu) {
+ task = idle_task(cpu);
+ WARN_ON_ONCE(test_tsk_thread_flag(task, TIF_PATCH_PENDING));
+ task->patch_state = KLP_UNDEFINED;
+ }
+
+done:
+ klp_target_state = KLP_UNDEFINED;
+ klp_transition_patch = NULL;
+}
+
+/*
+ * This is called in the error path, to cancel a transition before it has
+ * started, i.e. klp_init_transition() has been called but
+ * klp_start_transition() hasn't. If the transition *has* been started,
+ * klp_reverse_transition() should be used instead.
+ */
+void klp_cancel_transition(void)
+{
+ if (WARN_ON_ONCE(klp_target_state != KLP_PATCHED))
+ return;
+
+ klp_target_state = KLP_UNPATCHED;
+ klp_complete_transition();
+}
+
+/*
+ * Switch the patched state of the task to the set of functions in the target
+ * patch state.
+ *
+ * NOTE: If task is not 'current', the caller must ensure the task is inactive.
+ * Otherwise klp_ftrace_handler() might read the wrong 'patch_state' value.
+ */
+void klp_update_patch_state(struct task_struct *task)
+{
+ rcu_read_lock();
+
+ /*
+ * This test_and_clear_tsk_thread_flag() call also serves as a read
+ * barrier (smp_rmb) for two cases:
+ *
+ * 1) Enforce the order of the TIF_PATCH_PENDING read and the
+ * klp_target_state read. The corresponding write barrier is in
+ * klp_init_transition().
+ *
+ * 2) Enforce the order of the TIF_PATCH_PENDING read and a future read
+ * of func->transition, if klp_ftrace_handler() is called later on
+ * the same CPU. See __klp_disable_patch().
+ */
+ if (test_and_clear_tsk_thread_flag(task, TIF_PATCH_PENDING))
+ task->patch_state = READ_ONCE(klp_target_state);
+
+ rcu_read_unlock();
+}
+
+/*
+ * Determine whether the given stack trace includes any references to a
+ * to-be-patched or to-be-unpatched function.
+ */
+static int klp_check_stack_func(struct klp_func *func,
+ struct stack_trace *trace)
+{
+ unsigned long func_addr, func_size, address;
+ struct klp_ops *ops;
+ int i;
+
+ if (func->immediate)
+ return 0;
+
+ for (i = 0; i < trace->nr_entries; i++) {
+ address = trace->entries[i];
+
+ if (klp_target_state == KLP_UNPATCHED) {
+ /*
+ * Check for the to-be-unpatched function
+ * (the func itself).
+ */
+ func_addr = (unsigned long)func->new_func;
+ func_size = func->new_size;
+ } else {
+ /*
+ * Check for the to-be-patched function
+ * (the previous func).
+ */
+ ops = klp_find_ops(func->old_addr);
+
+ if (list_is_singular(&ops->func_stack)) {
+ /* original function */
+ func_addr = func->old_addr;
+ func_size = func->old_size;
+ } else {
+ /* previously patched function */
+ struct klp_func *prev;
+
+ prev = list_next_entry(func, stack_node);
+ func_addr = (unsigned long)prev->new_func;
+ func_size = prev->new_size;
+ }
+ }
+
+ if (address >= func_addr && address < func_addr + func_size)
+ return -EAGAIN;
+ }
+
+ return 0;
+}
+
+/*
+ * Determine whether it's safe to transition the task to the target patch state
+ * by looking for any to-be-patched or to-be-unpatched functions on its stack.
+ */
+static int klp_check_stack(struct task_struct *task, char *err_buf)
+{
+ static unsigned long entries[MAX_STACK_ENTRIES];
+ struct stack_trace trace;
+ struct klp_object *obj;
+ struct klp_func *func;
+ int ret;
+
+ trace.skip = 0;
+ trace.nr_entries = 0;
+ trace.max_entries = MAX_STACK_ENTRIES;
+ trace.entries = entries;
+ ret = save_stack_trace_tsk_reliable(task, &trace);
+ WARN_ON_ONCE(ret == -ENOSYS);
+ if (ret) {
+ snprintf(err_buf, STACK_ERR_BUF_SIZE,
+ "%s: %s:%d has an unreliable stack\n",
+ __func__, task->comm, task->pid);
+ return ret;
+ }
+
+ klp_for_each_object(klp_transition_patch, obj) {
+ if (!obj->patched)
+ continue;
+ klp_for_each_func(obj, func) {
+ ret = klp_check_stack_func(func, &trace);
+ if (ret) {
+ snprintf(err_buf, STACK_ERR_BUF_SIZE,
+ "%s: %s:%d is sleeping on function %s\n",
+ __func__, task->comm, task->pid,
+ func->old_name);
+ return ret;
+ }
+ }
+ }
+
+ return 0;
+}
+
+/*
+ * Try to safely switch a task to the target patch state. If it's currently
+ * running, or it's sleeping on a to-be-patched or to-be-unpatched function, or
+ * if the stack is unreliable, return false.
+ */
+static bool klp_try_switch_task(struct task_struct *task)
+{
+ struct rq *rq;
+ struct rq_flags flags;
+ int ret;
+ bool success = false;
+ char err_buf[STACK_ERR_BUF_SIZE];
+
+ err_buf[0] = '\0';
+
+ /* check if this task has already switched over */
+ if (task->patch_state == klp_target_state)
+ return true;
+
+ /*
+ * For arches which don't have reliable stack traces, we have to rely
+ * on other methods (e.g., switching tasks at kernel exit).
+ */
+ if (!klp_have_reliable_stack())
+ return false;
+
+ /*
+ * Now try to check the stack for any to-be-patched or to-be-unpatched
+ * functions. If all goes well, switch the task to the target patch
+ * state.
+ */
+ rq = task_rq_lock(task, &flags);
+
+ if (task_running(rq, task) && task != current) {
+ snprintf(err_buf, STACK_ERR_BUF_SIZE,
+ "%s: %s:%d is running\n", __func__, task->comm,
+ task->pid);
+ goto done;
+ }
+
+ ret = klp_check_stack(task, err_buf);
+ if (ret)
+ goto done;
+
+ success = true;
+
+ clear_tsk_thread_flag(task, TIF_PATCH_PENDING);
+ task->patch_state = klp_target_state;
+
+done:
+ task_rq_unlock(rq, task, &flags);
+
+ /*
+ * Due to console deadlock issues, pr_debug() can't be used while
+ * holding the task rq lock. Instead we have to use a temporary buffer
+ * and print the debug message after releasing the lock.
+ */
+ if (err_buf[0] != '\0')
+ pr_debug("%s", err_buf);
+
+ return success;
+
+}
+
+/*
+ * Try to switch all remaining tasks to the target patch state by walking the
+ * stacks of sleeping tasks and looking for any to-be-patched or
+ * to-be-unpatched functions. If such functions are found, the task can't be
+ * switched yet.
+ *
+ * If any tasks are still stuck in the initial patch state, schedule a retry.
+ */
+void klp_try_complete_transition(void)
+{
+ unsigned int cpu;
+ struct task_struct *g, *task;
+ bool complete = true;
+
+ WARN_ON_ONCE(klp_target_state == KLP_UNDEFINED);
+
+ /*
+ * If the patch can be applied or reverted immediately, skip the
+ * per-task transitions.
+ */
+ if (klp_transition_patch->immediate)
+ goto success;
+
+ /*
+ * Try to switch the tasks to the target patch state by walking their
+ * stacks and looking for any to-be-patched or to-be-unpatched
+ * functions. If such functions are found on a stack, or if the stack
+ * is deemed unreliable, the task can't be switched yet.
+ *
+ * Usually this will transition most (or all) of the tasks on a system
+ * unless the patch includes changes to a very common function.
+ */
+ read_lock(&tasklist_lock);
+ for_each_process_thread(g, task)
+ if (!klp_try_switch_task(task))
+ complete = false;
+ read_unlock(&tasklist_lock);
+
+ /*
+ * Ditto for the idle "swapper" tasks.
+ */
+ get_online_cpus();
+ for_each_possible_cpu(cpu) {
+ task = idle_task(cpu);
+ if (cpu_online(cpu)) {
+ if (!klp_try_switch_task(task))
+ complete = false;
+ } else if (task->patch_state != klp_target_state) {
+ /* offline idle tasks can be switched immediately */
+ clear_tsk_thread_flag(task, TIF_PATCH_PENDING);
+ task->patch_state = klp_target_state;
+ }
+ }
+ put_online_cpus();
+
+ if (!complete) {
+ /*
+ * Some tasks weren't able to be switched over. Try again
+ * later and/or wait for other methods like kernel exit
+ * switching.
+ */
+ schedule_delayed_work(&klp_transition_work,
+ round_jiffies_relative(HZ));
+ return;
+ }
+
+success:
+ pr_notice("'%s': %s complete\n", klp_transition_patch->mod->name,
+ klp_target_state == KLP_PATCHED ? "patching" : "unpatching");
+
+ /* we're done, now cleanup the data structures */
+ klp_complete_transition();
+}
+
+/*
+ * Start the transition to the specified target patch state so tasks can begin
+ * switching to it.
+ */
+void klp_start_transition(void)
+{
+ struct task_struct *g, *task;
+ unsigned int cpu;
+
+ WARN_ON_ONCE(klp_target_state == KLP_UNDEFINED);
+
+ pr_notice("'%s': %s...\n", klp_transition_patch->mod->name,
+ klp_target_state == KLP_PATCHED ? "patching" : "unpatching");
+
+ /*
+ * If the patch can be applied or reverted immediately, skip the
+ * per-task transitions.
+ */
+ if (klp_transition_patch->immediate)
+ return;
+
+ /*
+ * Mark all normal tasks as needing a patch state update. They'll
+ * switch either in klp_try_complete_transition() or as they exit the
+ * kernel.
+ */
+ read_lock(&tasklist_lock);
+ for_each_process_thread(g, task)
+ if (task->patch_state != klp_target_state)
+ set_tsk_thread_flag(task, TIF_PATCH_PENDING);
+ read_unlock(&tasklist_lock);
+
+ /*
+ * Mark all idle tasks as needing a patch state update. They'll switch
+ * either in klp_try_complete_transition() or at the idle loop switch
+ * point.
+ */
+ for_each_possible_cpu(cpu) {
+ task = idle_task(cpu);
+ if (task->patch_state != klp_target_state)
+ set_tsk_thread_flag(task, TIF_PATCH_PENDING);
+ }
+}
+
+/*
+ * Initialize the global target patch state and all tasks to the initial patch
+ * state, and initialize all function transition states to true in preparation
+ * for patching or unpatching.
+ */
+void klp_init_transition(struct klp_patch *patch, int state)
+{
+ struct task_struct *g, *task;
+ unsigned int cpu;
+ struct klp_object *obj;
+ struct klp_func *func;
+ int initial_state = !state;
+
+ WARN_ON_ONCE(klp_target_state != KLP_UNDEFINED);
+
+ klp_transition_patch = patch;
+
+ /*
+ * Set the global target patch state which tasks will switch to. This
+ * has no effect until the TIF_PATCH_PENDING flags get set later.
+ */
+ klp_target_state = state;
+
+ /*
+ * If the patch can be applied or reverted immediately, skip the
+ * per-task transitions.
+ */
+ if (patch->immediate)
+ return;
+
+ /*
+ * Initialize all tasks to the initial patch state to prepare them for
+ * switching to the target state.
+ */
+ read_lock(&tasklist_lock);
+ for_each_process_thread(g, task) {
+ WARN_ON_ONCE(task->patch_state != KLP_UNDEFINED);
+ task->patch_state = initial_state;
+ }
+ read_unlock(&tasklist_lock);
+
+ /*
+ * Ditto for the idle "swapper" tasks.
+ */
+ for_each_possible_cpu(cpu) {
+ task = idle_task(cpu);
+ WARN_ON_ONCE(task->patch_state != KLP_UNDEFINED);
+ task->patch_state = initial_state;
+ }
+
+ /*
+ * Enforce the order of the task->patch_state initializations and the
+ * func->transition updates to ensure that klp_ftrace_handler() doesn't
+ * see a func in transition with a task->patch_state of KLP_UNDEFINED.
+ *
+ * Also enforce the order of the klp_target_state write and future
+ * TIF_PATCH_PENDING writes to ensure klp_update_patch_state() doesn't
+ * set a task->patch_state to KLP_UNDEFINED.
+ */
+ smp_wmb();
+
+ /*
+ * Set the func transition states so klp_ftrace_handler() will know to
+ * switch to the transition logic.
+ *
+ * When patching, the funcs aren't yet in the func_stack and will be
+ * made visible to the ftrace handler shortly by the calls to
+ * klp_patch_object().
+ *
+ * When unpatching, the funcs are already in the func_stack and so are
+ * already visible to the ftrace handler.
+ */
+ klp_for_each_object(patch, obj)
+ klp_for_each_func(obj, func)
+ func->transition = true;
+}
+
+/*
+ * This function can be called in the middle of an existing transition to
+ * reverse the direction of the target patch state. This can be done to
+ * effectively cancel an existing enable or disable operation if there are any
+ * tasks which are stuck in the initial patch state.
+ */
+void klp_reverse_transition(void)
+{
+ unsigned int cpu;
+ struct task_struct *g, *task;
+
+ klp_transition_patch->enabled = !klp_transition_patch->enabled;
+
+ klp_target_state = !klp_target_state;
+
+ /*
+ * Clear all TIF_PATCH_PENDING flags to prevent races caused by
+ * klp_update_patch_state() running in parallel with
+ * klp_start_transition().
+ */
+ read_lock(&tasklist_lock);
+ for_each_process_thread(g, task)
+ clear_tsk_thread_flag(task, TIF_PATCH_PENDING);
+ read_unlock(&tasklist_lock);
+
+ for_each_possible_cpu(cpu)
+ clear_tsk_thread_flag(idle_task(cpu), TIF_PATCH_PENDING);
+
+ /* Let any remaining calls to klp_update_patch_state() complete */
+ synchronize_rcu();
+
+ klp_start_transition();
+}
+
+/* Called from copy_process() during fork */
+void klp_copy_process(struct task_struct *child)
+{
+ child->patch_state = current->patch_state;
+
+ /* TIF_PATCH_PENDING gets copied in setup_thread_stack() */
+}
diff --git a/kernel/livepatch/transition.h b/kernel/livepatch/transition.h
new file mode 100644
index 0000000..ce09b32
--- /dev/null
+++ b/kernel/livepatch/transition.h
@@ -0,0 +1,14 @@
+#ifndef _LIVEPATCH_TRANSITION_H
+#define _LIVEPATCH_TRANSITION_H
+
+#include <linux/livepatch.h>
+
+extern struct klp_patch *klp_transition_patch;
+
+void klp_init_transition(struct klp_patch *patch, int state);
+void klp_cancel_transition(void);
+void klp_start_transition(void);
+void klp_try_complete_transition(void);
+void klp_reverse_transition(void);
+
+#endif /* _LIVEPATCH_TRANSITION_H */
diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
index a95e5d1..98dd623 100644
--- a/kernel/locking/lockdep.c
+++ b/kernel/locking/lockdep.c
@@ -660,6 +660,7 @@
struct lockdep_subclass_key *key;
struct hlist_head *hash_head;
struct lock_class *class;
+ bool is_static = false;
if (unlikely(subclass >= MAX_LOCKDEP_SUBCLASSES)) {
debug_locks_off();
@@ -673,10 +674,23 @@
/*
* Static locks do not have their class-keys yet - for them the key
- * is the lock object itself:
+ * is the lock object itself. If the lock is in the per cpu area,
+ * the canonical address of the lock (per cpu offset removed) is
+ * used.
*/
- if (unlikely(!lock->key))
- lock->key = (void *)lock;
+ if (unlikely(!lock->key)) {
+ unsigned long can_addr, addr = (unsigned long)lock;
+
+ if (__is_kernel_percpu_address(addr, &can_addr))
+ lock->key = (void *)can_addr;
+ else if (__is_module_percpu_address(addr, &can_addr))
+ lock->key = (void *)can_addr;
+ else if (static_obj(lock))
+ lock->key = (void *)lock;
+ else
+ return ERR_PTR(-EINVAL);
+ is_static = true;
+ }
/*
* NOTE: the class-key must be unique. For dynamic locks, a static
@@ -708,7 +722,7 @@
}
}
- return NULL;
+ return is_static || static_obj(lock->key) ? NULL : ERR_PTR(-EINVAL);
}
/*
@@ -726,19 +740,18 @@
DEBUG_LOCKS_WARN_ON(!irqs_disabled());
class = look_up_lock_class(lock, subclass);
- if (likely(class))
+ if (likely(!IS_ERR_OR_NULL(class)))
goto out_set_class_cache;
/*
* Debug-check: all keys must be persistent!
- */
- if (!static_obj(lock->key)) {
+ */
+ if (IS_ERR(class)) {
debug_locks_off();
printk("INFO: trying to register non-static key.\n");
printk("the code is fine but needs lockdep annotation.\n");
printk("turning off the locking correctness validator.\n");
dump_stack();
-
return NULL;
}
@@ -3419,7 +3432,7 @@
* Clearly if the lock hasn't been acquired _ever_, we're not
* holding it either, so report failure.
*/
- if (!class)
+ if (IS_ERR_OR_NULL(class))
return 0;
/*
@@ -3437,13 +3450,67 @@
return 0;
}
+/* @depth must not be zero */
+static struct held_lock *find_held_lock(struct task_struct *curr,
+ struct lockdep_map *lock,
+ unsigned int depth, int *idx)
+{
+ struct held_lock *ret, *hlock, *prev_hlock;
+ int i;
+
+ i = depth - 1;
+ hlock = curr->held_locks + i;
+ ret = hlock;
+ if (match_held_lock(hlock, lock))
+ goto out;
+
+ ret = NULL;
+ for (i--, prev_hlock = hlock--;
+ i >= 0;
+ i--, prev_hlock = hlock--) {
+ /*
+ * We must not cross into another context:
+ */
+ if (prev_hlock->irq_context != hlock->irq_context) {
+ ret = NULL;
+ break;
+ }
+ if (match_held_lock(hlock, lock)) {
+ ret = hlock;
+ break;
+ }
+ }
+
+out:
+ *idx = i;
+ return ret;
+}
+
+static int reacquire_held_locks(struct task_struct *curr, unsigned int depth,
+ int idx)
+{
+ struct held_lock *hlock;
+
+ for (hlock = curr->held_locks + idx; idx < depth; idx++, hlock++) {
+ if (!__lock_acquire(hlock->instance,
+ hlock_class(hlock)->subclass,
+ hlock->trylock,
+ hlock->read, hlock->check,
+ hlock->hardirqs_off,
+ hlock->nest_lock, hlock->acquire_ip,
+ hlock->references, hlock->pin_count))
+ return 1;
+ }
+ return 0;
+}
+
static int
__lock_set_class(struct lockdep_map *lock, const char *name,
struct lock_class_key *key, unsigned int subclass,
unsigned long ip)
{
struct task_struct *curr = current;
- struct held_lock *hlock, *prev_hlock;
+ struct held_lock *hlock;
struct lock_class *class;
unsigned int depth;
int i;
@@ -3456,21 +3523,10 @@
if (DEBUG_LOCKS_WARN_ON(!depth))
return 0;
- prev_hlock = NULL;
- for (i = depth-1; i >= 0; i--) {
- hlock = curr->held_locks + i;
- /*
- * We must not cross into another context:
- */
- if (prev_hlock && prev_hlock->irq_context != hlock->irq_context)
- break;
- if (match_held_lock(hlock, lock))
- goto found_it;
- prev_hlock = hlock;
- }
- return print_unlock_imbalance_bug(curr, lock, ip);
+ hlock = find_held_lock(curr, lock, depth, &i);
+ if (!hlock)
+ return print_unlock_imbalance_bug(curr, lock, ip);
-found_it:
lockdep_init_map(lock, name, key, 0);
class = register_lock_class(lock, subclass, 0);
hlock->class_idx = class - lock_classes + 1;
@@ -3478,15 +3534,46 @@
curr->lockdep_depth = i;
curr->curr_chain_key = hlock->prev_chain_key;
- for (; i < depth; i++) {
- hlock = curr->held_locks + i;
- if (!__lock_acquire(hlock->instance,
- hlock_class(hlock)->subclass, hlock->trylock,
- hlock->read, hlock->check, hlock->hardirqs_off,
- hlock->nest_lock, hlock->acquire_ip,
- hlock->references, hlock->pin_count))
- return 0;
- }
+ if (reacquire_held_locks(curr, depth, i))
+ return 0;
+
+ /*
+ * I took it apart and put it back together again, except now I have
+ * these 'spare' parts.. where shall I put them.
+ */
+ if (DEBUG_LOCKS_WARN_ON(curr->lockdep_depth != depth))
+ return 0;
+ return 1;
+}
+
+static int __lock_downgrade(struct lockdep_map *lock, unsigned long ip)
+{
+ struct task_struct *curr = current;
+ struct held_lock *hlock;
+ unsigned int depth;
+ int i;
+
+ depth = curr->lockdep_depth;
+ /*
+ * This function is about (re)setting the class of a held lock,
+ * yet we're not actually holding any locks. Naughty user!
+ */
+ if (DEBUG_LOCKS_WARN_ON(!depth))
+ return 0;
+
+ hlock = find_held_lock(curr, lock, depth, &i);
+ if (!hlock)
+ return print_unlock_imbalance_bug(curr, lock, ip);
+
+ curr->lockdep_depth = i;
+ curr->curr_chain_key = hlock->prev_chain_key;
+
+ WARN(hlock->read, "downgrading a read lock");
+ hlock->read = 1;
+ hlock->acquire_ip = ip;
+
+ if (reacquire_held_locks(curr, depth, i))
+ return 0;
/*
* I took it apart and put it back together again, except now I have
@@ -3508,7 +3595,7 @@
__lock_release(struct lockdep_map *lock, int nested, unsigned long ip)
{
struct task_struct *curr = current;
- struct held_lock *hlock, *prev_hlock;
+ struct held_lock *hlock;
unsigned int depth;
int i;
@@ -3527,21 +3614,10 @@
* Check whether the lock exists in the current stack
* of held locks:
*/
- prev_hlock = NULL;
- for (i = depth-1; i >= 0; i--) {
- hlock = curr->held_locks + i;
- /*
- * We must not cross into another context:
- */
- if (prev_hlock && prev_hlock->irq_context != hlock->irq_context)
- break;
- if (match_held_lock(hlock, lock))
- goto found_it;
- prev_hlock = hlock;
- }
- return print_unlock_imbalance_bug(curr, lock, ip);
+ hlock = find_held_lock(curr, lock, depth, &i);
+ if (!hlock)
+ return print_unlock_imbalance_bug(curr, lock, ip);
-found_it:
if (hlock->instance == lock)
lock_release_holdtime(hlock);
@@ -3568,15 +3644,8 @@
curr->lockdep_depth = i;
curr->curr_chain_key = hlock->prev_chain_key;
- for (i++; i < depth; i++) {
- hlock = curr->held_locks + i;
- if (!__lock_acquire(hlock->instance,
- hlock_class(hlock)->subclass, hlock->trylock,
- hlock->read, hlock->check, hlock->hardirqs_off,
- hlock->nest_lock, hlock->acquire_ip,
- hlock->references, hlock->pin_count))
- return 0;
- }
+ if (reacquire_held_locks(curr, depth, i + 1))
+ return 0;
/*
* We had N bottles of beer on the wall, we drank one, but now
@@ -3741,6 +3810,23 @@
}
EXPORT_SYMBOL_GPL(lock_set_class);
+void lock_downgrade(struct lockdep_map *lock, unsigned long ip)
+{
+ unsigned long flags;
+
+ if (unlikely(current->lockdep_recursion))
+ return;
+
+ raw_local_irq_save(flags);
+ current->lockdep_recursion = 1;
+ check_flags(flags);
+ if (__lock_downgrade(lock, ip))
+ check_chain_key(current);
+ current->lockdep_recursion = 0;
+ raw_local_irq_restore(flags);
+}
+EXPORT_SYMBOL_GPL(lock_downgrade);
+
/*
* We are not always called with irqs disabled - do that here,
* and also avoid lockdep recursion:
@@ -3903,7 +3989,7 @@
__lock_contended(struct lockdep_map *lock, unsigned long ip)
{
struct task_struct *curr = current;
- struct held_lock *hlock, *prev_hlock;
+ struct held_lock *hlock;
struct lock_class_stats *stats;
unsigned int depth;
int i, contention_point, contending_point;
@@ -3916,22 +4002,12 @@
if (DEBUG_LOCKS_WARN_ON(!depth))
return;
- prev_hlock = NULL;
- for (i = depth-1; i >= 0; i--) {
- hlock = curr->held_locks + i;
- /*
- * We must not cross into another context:
- */
- if (prev_hlock && prev_hlock->irq_context != hlock->irq_context)
- break;
- if (match_held_lock(hlock, lock))
- goto found_it;
- prev_hlock = hlock;
+ hlock = find_held_lock(curr, lock, depth, &i);
+ if (!hlock) {
+ print_lock_contention_bug(curr, lock, ip);
+ return;
}
- print_lock_contention_bug(curr, lock, ip);
- return;
-found_it:
if (hlock->instance != lock)
return;
@@ -3955,7 +4031,7 @@
__lock_acquired(struct lockdep_map *lock, unsigned long ip)
{
struct task_struct *curr = current;
- struct held_lock *hlock, *prev_hlock;
+ struct held_lock *hlock;
struct lock_class_stats *stats;
unsigned int depth;
u64 now, waittime = 0;
@@ -3969,22 +4045,12 @@
if (DEBUG_LOCKS_WARN_ON(!depth))
return;
- prev_hlock = NULL;
- for (i = depth-1; i >= 0; i--) {
- hlock = curr->held_locks + i;
- /*
- * We must not cross into another context:
- */
- if (prev_hlock && prev_hlock->irq_context != hlock->irq_context)
- break;
- if (match_held_lock(hlock, lock))
- goto found_it;
- prev_hlock = hlock;
+ hlock = find_held_lock(curr, lock, depth, &i);
+ if (!hlock) {
+ print_lock_contention_bug(curr, lock, _RET_IP_);
+ return;
}
- print_lock_contention_bug(curr, lock, _RET_IP_);
- return;
-found_it:
if (hlock->instance != lock)
return;
@@ -4172,7 +4238,7 @@
* If the class exists we look it up and zap it:
*/
class = look_up_lock_class(lock, j);
- if (class)
+ if (!IS_ERR_OR_NULL(class))
zap_class(class);
}
/*
diff --git a/kernel/locking/rtmutex-debug.c b/kernel/locking/rtmutex-debug.c
index 97ee9df..32fe775 100644
--- a/kernel/locking/rtmutex-debug.c
+++ b/kernel/locking/rtmutex-debug.c
@@ -174,12 +174,3 @@
lock->name = name;
}
-void
-rt_mutex_deadlock_account_lock(struct rt_mutex *lock, struct task_struct *task)
-{
-}
-
-void rt_mutex_deadlock_account_unlock(struct task_struct *task)
-{
-}
-
diff --git a/kernel/locking/rtmutex-debug.h b/kernel/locking/rtmutex-debug.h
index d0519c3..b585af9 100644
--- a/kernel/locking/rtmutex-debug.h
+++ b/kernel/locking/rtmutex-debug.h
@@ -9,9 +9,6 @@
* This file contains macros used solely by rtmutex.c. Debug version.
*/
-extern void
-rt_mutex_deadlock_account_lock(struct rt_mutex *lock, struct task_struct *task);
-extern void rt_mutex_deadlock_account_unlock(struct task_struct *task);
extern void debug_rt_mutex_init_waiter(struct rt_mutex_waiter *waiter);
extern void debug_rt_mutex_free_waiter(struct rt_mutex_waiter *waiter);
extern void debug_rt_mutex_init(struct rt_mutex *lock, const char *name);
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 6edc32e..b955094 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -224,6 +224,12 @@
}
#endif
+/*
+ * Only use with rt_mutex_waiter_{less,equal}()
+ */
+#define task_to_waiter(p) \
+ &(struct rt_mutex_waiter){ .prio = (p)->prio, .deadline = (p)->dl.deadline }
+
static inline int
rt_mutex_waiter_less(struct rt_mutex_waiter *left,
struct rt_mutex_waiter *right)
@@ -238,12 +244,30 @@
* then right waiter has a dl_prio() too.
*/
if (dl_prio(left->prio))
- return dl_time_before(left->task->dl.deadline,
- right->task->dl.deadline);
+ return dl_time_before(left->deadline, right->deadline);
return 0;
}
+static inline int
+rt_mutex_waiter_equal(struct rt_mutex_waiter *left,
+ struct rt_mutex_waiter *right)
+{
+ if (left->prio != right->prio)
+ return 0;
+
+ /*
+ * If both waiters have dl_prio(), we check the deadlines of the
+ * associated tasks.
+ * If left waiter has a dl_prio(), and we didn't return 0 above,
+ * then right waiter has a dl_prio() too.
+ */
+ if (dl_prio(left->prio))
+ return left->deadline == right->deadline;
+
+ return 1;
+}
+
static void
rt_mutex_enqueue(struct rt_mutex *lock, struct rt_mutex_waiter *waiter)
{
@@ -322,72 +346,16 @@
RB_CLEAR_NODE(&waiter->pi_tree_entry);
}
-/*
- * Calculate task priority from the waiter tree priority
- *
- * Return task->normal_prio when the waiter tree is empty or when
- * the waiter is not allowed to do priority boosting
- */
-int rt_mutex_getprio(struct task_struct *task)
+static void rt_mutex_adjust_prio(struct task_struct *p)
{
- if (likely(!task_has_pi_waiters(task)))
- return task->normal_prio;
+ struct task_struct *pi_task = NULL;
- return min(task_top_pi_waiter(task)->prio,
- task->normal_prio);
-}
+ lockdep_assert_held(&p->pi_lock);
-struct task_struct *rt_mutex_get_top_task(struct task_struct *task)
-{
- if (likely(!task_has_pi_waiters(task)))
- return NULL;
+ if (task_has_pi_waiters(p))
+ pi_task = task_top_pi_waiter(p)->task;
- return task_top_pi_waiter(task)->task;
-}
-
-/*
- * Called by sched_setscheduler() to get the priority which will be
- * effective after the change.
- */
-int rt_mutex_get_effective_prio(struct task_struct *task, int newprio)
-{
- if (!task_has_pi_waiters(task))
- return newprio;
-
- if (task_top_pi_waiter(task)->task->prio <= newprio)
- return task_top_pi_waiter(task)->task->prio;
- return newprio;
-}
-
-/*
- * Adjust the priority of a task, after its pi_waiters got modified.
- *
- * This can be both boosting and unboosting. task->pi_lock must be held.
- */
-static void __rt_mutex_adjust_prio(struct task_struct *task)
-{
- int prio = rt_mutex_getprio(task);
-
- if (task->prio != prio || dl_prio(prio))
- rt_mutex_setprio(task, prio);
-}
-
-/*
- * Adjust task priority (undo boosting). Called from the exit path of
- * rt_mutex_slowunlock() and rt_mutex_slowlock().
- *
- * (Note: We do this outside of the protection of lock->wait_lock to
- * allow the lock to be taken while or before we readjust the priority
- * of task. We do not use the spin_xx_mutex() variants here as we are
- * outside of the debug path.)
- */
-void rt_mutex_adjust_prio(struct task_struct *task)
-{
- unsigned long flags;
-
- raw_spin_lock_irqsave(&task->pi_lock, flags);
- __rt_mutex_adjust_prio(task);
- raw_spin_unlock_irqrestore(&task->pi_lock, flags);
+ rt_mutex_setprio(p, pi_task);
}
/*
@@ -610,7 +578,7 @@
* enabled we continue, but stop the requeueing in the chain
* walk.
*/
- if (waiter->prio == task->prio) {
+ if (rt_mutex_waiter_equal(waiter, task_to_waiter(task))) {
if (!detect_deadlock)
goto out_unlock_pi;
else
@@ -706,7 +674,26 @@
/* [7] Requeue the waiter in the lock waiter tree. */
rt_mutex_dequeue(lock, waiter);
+
+ /*
+ * Update the waiter prio fields now that we're dequeued.
+ *
+ * These values can have changed through either:
+ *
+ * sys_sched_set_scheduler() / sys_sched_setattr()
+ *
+ * or
+ *
+ * DL CBS enforcement advancing the effective deadline.
+ *
+ * Even though pi_waiters also uses these fields, and that tree is only
+ * updated in [11], we can do this here, since we hold [L], which
+ * serializes all pi_waiters access and rb_erase() does not care about
+ * the values of the node being removed.
+ */
waiter->prio = task->prio;
+ waiter->deadline = task->dl.deadline;
+
rt_mutex_enqueue(lock, waiter);
/* [8] Release the task */
@@ -747,7 +734,7 @@
*/
rt_mutex_dequeue_pi(task, prerequeue_top_waiter);
rt_mutex_enqueue_pi(task, waiter);
- __rt_mutex_adjust_prio(task);
+ rt_mutex_adjust_prio(task);
} else if (prerequeue_top_waiter == waiter) {
/*
@@ -763,7 +750,7 @@
rt_mutex_dequeue_pi(task, waiter);
waiter = rt_mutex_top_waiter(lock);
rt_mutex_enqueue_pi(task, waiter);
- __rt_mutex_adjust_prio(task);
+ rt_mutex_adjust_prio(task);
} else {
/*
* Nothing changed. No need to do any priority
@@ -833,6 +820,8 @@
static int try_to_take_rt_mutex(struct rt_mutex *lock, struct task_struct *task,
struct rt_mutex_waiter *waiter)
{
+ lockdep_assert_held(&lock->wait_lock);
+
/*
* Before testing whether we can acquire @lock, we set the
* RT_MUTEX_HAS_WAITERS bit in @lock->owner. This forces all
@@ -892,7 +881,8 @@
* the top waiter priority (kernel view),
* @task lost.
*/
- if (task->prio >= rt_mutex_top_waiter(lock)->prio)
+ if (!rt_mutex_waiter_less(task_to_waiter(task),
+ rt_mutex_top_waiter(lock)))
return 0;
/*
@@ -938,8 +928,6 @@
*/
rt_mutex_set_owner(lock, task);
- rt_mutex_deadlock_account_lock(lock, task);
-
return 1;
}
@@ -960,6 +948,8 @@
struct rt_mutex *next_lock;
int chain_walk = 0, res;
+ lockdep_assert_held(&lock->wait_lock);
+
/*
* Early deadlock detection. We really don't want the task to
* enqueue on itself just to untangle the mess later. It's not
@@ -973,10 +963,11 @@
return -EDEADLK;
raw_spin_lock(&task->pi_lock);
- __rt_mutex_adjust_prio(task);
+ rt_mutex_adjust_prio(task);
waiter->task = task;
waiter->lock = lock;
waiter->prio = task->prio;
+ waiter->deadline = task->dl.deadline;
/* Get the top priority waiter on the lock */
if (rt_mutex_has_waiters(lock))
@@ -995,7 +986,7 @@
rt_mutex_dequeue_pi(owner, top_waiter);
rt_mutex_enqueue_pi(owner, waiter);
- __rt_mutex_adjust_prio(owner);
+ rt_mutex_adjust_prio(owner);
if (owner->pi_blocked_on)
chain_walk = 1;
} else if (rt_mutex_cond_detect_deadlock(waiter, chwalk)) {
@@ -1047,12 +1038,14 @@
waiter = rt_mutex_top_waiter(lock);
/*
- * Remove it from current->pi_waiters. We do not adjust a
- * possible priority boost right now. We execute wakeup in the
- * boosted mode and go back to normal after releasing
- * lock->wait_lock.
+ * Remove it from current->pi_waiters and deboost.
+ *
+ * We must in fact deboost here in order to ensure we call
+ * rt_mutex_setprio() to update p->pi_top_task before the
+ * task unblocks.
*/
rt_mutex_dequeue_pi(current, waiter);
+ rt_mutex_adjust_prio(current);
/*
* As we are waking up the top waiter, and the waiter stays
@@ -1064,9 +1057,19 @@
*/
lock->owner = (void *) RT_MUTEX_HAS_WAITERS;
- raw_spin_unlock(¤t->pi_lock);
-
+ /*
+ * We deboosted before waking the top waiter task such that we don't
+ * run two tasks with the 'same' priority (and ensure the
+ * p->pi_top_task pointer points to a blocked task). This however can
+ * lead to priority inversion if we would get preempted after the
+ * deboost but before waking our donor task, hence the preempt_disable()
+ * before unlock.
+ *
+ * Pairs with preempt_enable() in rt_mutex_postunlock();
+ */
+ preempt_disable();
wake_q_add(wake_q, waiter->task);
+ raw_spin_unlock(¤t->pi_lock);
}
/*
@@ -1082,6 +1085,8 @@
struct task_struct *owner = rt_mutex_owner(lock);
struct rt_mutex *next_lock;
+ lockdep_assert_held(&lock->wait_lock);
+
raw_spin_lock(¤t->pi_lock);
rt_mutex_dequeue(lock, waiter);
current->pi_blocked_on = NULL;
@@ -1101,7 +1106,7 @@
if (rt_mutex_has_waiters(lock))
rt_mutex_enqueue_pi(owner, rt_mutex_top_waiter(lock));
- __rt_mutex_adjust_prio(owner);
+ rt_mutex_adjust_prio(owner);
/* Store the lock on which owner is blocked or NULL */
next_lock = task_blocked_on_lock(owner);
@@ -1140,8 +1145,7 @@
raw_spin_lock_irqsave(&task->pi_lock, flags);
waiter = task->pi_blocked_on;
- if (!waiter || (waiter->prio == task->prio &&
- !dl_prio(task->prio))) {
+ if (!waiter || rt_mutex_waiter_equal(waiter, task_to_waiter(task))) {
raw_spin_unlock_irqrestore(&task->pi_lock, flags);
return;
}
@@ -1155,6 +1159,14 @@
next_lock, NULL, task);
}
+void rt_mutex_init_waiter(struct rt_mutex_waiter *waiter)
+{
+ debug_rt_mutex_init_waiter(waiter);
+ RB_CLEAR_NODE(&waiter->pi_tree_entry);
+ RB_CLEAR_NODE(&waiter->tree_entry);
+ waiter->task = NULL;
+}
+
/**
* __rt_mutex_slowlock() - Perform the wait-wake-try-to-take loop
* @lock: the rt_mutex to take
@@ -1237,9 +1249,7 @@
unsigned long flags;
int ret = 0;
- debug_rt_mutex_init_waiter(&waiter);
- RB_CLEAR_NODE(&waiter.pi_tree_entry);
- RB_CLEAR_NODE(&waiter.tree_entry);
+ rt_mutex_init_waiter(&waiter);
/*
* Technically we could use raw_spin_[un]lock_irq() here, but this can
@@ -1330,7 +1340,8 @@
/*
* Slow path to release a rt-mutex.
- * Return whether the current task needs to undo a potential priority boosting.
+ *
+ * Return whether the current task needs to call rt_mutex_postunlock().
*/
static bool __sched rt_mutex_slowunlock(struct rt_mutex *lock,
struct wake_q_head *wake_q)
@@ -1342,8 +1353,6 @@
debug_rt_mutex_unlock(lock);
- rt_mutex_deadlock_account_unlock(current);
-
/*
* We must be careful here if the fast path is enabled. If we
* have no waiters queued we cannot set owner to NULL here
@@ -1390,11 +1399,9 @@
* Queue the next waiter for wakeup once we release the wait_lock.
*/
mark_wakeup_next_waiter(wake_q, lock);
-
raw_spin_unlock_irqrestore(&lock->wait_lock, flags);
- /* check PI boosting */
- return true;
+ return true; /* call rt_mutex_postunlock() */
}
/*
@@ -1409,11 +1416,10 @@
struct hrtimer_sleeper *timeout,
enum rtmutex_chainwalk chwalk))
{
- if (likely(rt_mutex_cmpxchg_acquire(lock, NULL, current))) {
- rt_mutex_deadlock_account_lock(lock, current);
+ if (likely(rt_mutex_cmpxchg_acquire(lock, NULL, current)))
return 0;
- } else
- return slowfn(lock, state, NULL, RT_MUTEX_MIN_CHAINWALK);
+
+ return slowfn(lock, state, NULL, RT_MUTEX_MIN_CHAINWALK);
}
static inline int
@@ -1425,24 +1431,33 @@
enum rtmutex_chainwalk chwalk))
{
if (chwalk == RT_MUTEX_MIN_CHAINWALK &&
- likely(rt_mutex_cmpxchg_acquire(lock, NULL, current))) {
- rt_mutex_deadlock_account_lock(lock, current);
+ likely(rt_mutex_cmpxchg_acquire(lock, NULL, current)))
return 0;
- } else
- return slowfn(lock, state, timeout, chwalk);
+
+ return slowfn(lock, state, timeout, chwalk);
}
static inline int
rt_mutex_fasttrylock(struct rt_mutex *lock,
int (*slowfn)(struct rt_mutex *lock))
{
- if (likely(rt_mutex_cmpxchg_acquire(lock, NULL, current))) {
- rt_mutex_deadlock_account_lock(lock, current);
+ if (likely(rt_mutex_cmpxchg_acquire(lock, NULL, current)))
return 1;
- }
+
return slowfn(lock);
}
+/*
+ * Performs the wakeup of the the top-waiter and re-enables preemption.
+ */
+void rt_mutex_postunlock(struct wake_q_head *wake_q)
+{
+ wake_up_q(wake_q);
+
+ /* Pairs with preempt_disable() in rt_mutex_slowunlock() */
+ preempt_enable();
+}
+
static inline void
rt_mutex_fastunlock(struct rt_mutex *lock,
bool (*slowfn)(struct rt_mutex *lock,
@@ -1450,18 +1465,11 @@
{
DEFINE_WAKE_Q(wake_q);
- if (likely(rt_mutex_cmpxchg_release(lock, current, NULL))) {
- rt_mutex_deadlock_account_unlock(current);
+ if (likely(rt_mutex_cmpxchg_release(lock, current, NULL)))
+ return;
- } else {
- bool deboost = slowfn(lock, &wake_q);
-
- wake_up_q(&wake_q);
-
- /* Undo pi boosting if necessary: */
- if (deboost)
- rt_mutex_adjust_prio(current);
- }
+ if (slowfn(lock, &wake_q))
+ rt_mutex_postunlock(&wake_q);
}
/**
@@ -1495,16 +1503,11 @@
EXPORT_SYMBOL_GPL(rt_mutex_lock_interruptible);
/*
- * Futex variant with full deadlock detection.
+ * Futex variant, must not use fastpath.
*/
-int rt_mutex_timed_futex_lock(struct rt_mutex *lock,
- struct hrtimer_sleeper *timeout)
+int __sched rt_mutex_futex_trylock(struct rt_mutex *lock)
{
- might_sleep();
-
- return rt_mutex_timed_fastlock(lock, TASK_INTERRUPTIBLE, timeout,
- RT_MUTEX_FULL_CHAINWALK,
- rt_mutex_slowlock);
+ return rt_mutex_slowtrylock(lock);
}
/**
@@ -1563,20 +1566,43 @@
EXPORT_SYMBOL_GPL(rt_mutex_unlock);
/**
- * rt_mutex_futex_unlock - Futex variant of rt_mutex_unlock
- * @lock: the rt_mutex to be unlocked
- *
- * Returns: true/false indicating whether priority adjustment is
- * required or not.
+ * Futex variant, that since futex variants do not use the fast-path, can be
+ * simple and will not need to retry.
*/
-bool __sched rt_mutex_futex_unlock(struct rt_mutex *lock,
- struct wake_q_head *wqh)
+bool __sched __rt_mutex_futex_unlock(struct rt_mutex *lock,
+ struct wake_q_head *wake_q)
{
- if (likely(rt_mutex_cmpxchg_release(lock, current, NULL))) {
- rt_mutex_deadlock_account_unlock(current);
- return false;
+ lockdep_assert_held(&lock->wait_lock);
+
+ debug_rt_mutex_unlock(lock);
+
+ if (!rt_mutex_has_waiters(lock)) {
+ lock->owner = NULL;
+ return false; /* done */
}
- return rt_mutex_slowunlock(lock, wqh);
+
+ /*
+ * We've already deboosted, mark_wakeup_next_waiter() will
+ * retain preempt_disabled when we drop the wait_lock, to
+ * avoid inversion prior to the wakeup. preempt_disable()
+ * therein pairs with rt_mutex_postunlock().
+ */
+ mark_wakeup_next_waiter(wake_q, lock);
+
+ return true; /* call postunlock() */
+}
+
+void __sched rt_mutex_futex_unlock(struct rt_mutex *lock)
+{
+ DEFINE_WAKE_Q(wake_q);
+ bool postunlock;
+
+ raw_spin_lock_irq(&lock->wait_lock);
+ postunlock = __rt_mutex_futex_unlock(lock, &wake_q);
+ raw_spin_unlock_irq(&lock->wait_lock);
+
+ if (postunlock)
+ rt_mutex_postunlock(&wake_q);
}
/**
@@ -1637,7 +1663,6 @@
__rt_mutex_init(lock, NULL);
debug_rt_mutex_proxy_lock(lock, proxy_owner);
rt_mutex_set_owner(lock, proxy_owner);
- rt_mutex_deadlock_account_lock(lock, proxy_owner);
}
/**
@@ -1657,7 +1682,37 @@
{
debug_rt_mutex_proxy_unlock(lock);
rt_mutex_set_owner(lock, NULL);
- rt_mutex_deadlock_account_unlock(proxy_owner);
+}
+
+int __rt_mutex_start_proxy_lock(struct rt_mutex *lock,
+ struct rt_mutex_waiter *waiter,
+ struct task_struct *task)
+{
+ int ret;
+
+ if (try_to_take_rt_mutex(lock, task, NULL))
+ return 1;
+
+ /* We enforce deadlock detection for futexes */
+ ret = task_blocks_on_rt_mutex(lock, waiter, task,
+ RT_MUTEX_FULL_CHAINWALK);
+
+ if (ret && !rt_mutex_owner(lock)) {
+ /*
+ * Reset the return value. We might have
+ * returned with -EDEADLK and the owner
+ * released the lock while we were walking the
+ * pi chain. Let the waiter sort it out.
+ */
+ ret = 0;
+ }
+
+ if (unlikely(ret))
+ remove_waiter(lock, waiter);
+
+ debug_rt_mutex_print_deadlock(waiter);
+
+ return ret;
}
/**
@@ -1680,33 +1735,9 @@
int ret;
raw_spin_lock_irq(&lock->wait_lock);
-
- if (try_to_take_rt_mutex(lock, task, NULL)) {
- raw_spin_unlock_irq(&lock->wait_lock);
- return 1;
- }
-
- /* We enforce deadlock detection for futexes */
- ret = task_blocks_on_rt_mutex(lock, waiter, task,
- RT_MUTEX_FULL_CHAINWALK);
-
- if (ret && !rt_mutex_owner(lock)) {
- /*
- * Reset the return value. We might have
- * returned with -EDEADLK and the owner
- * released the lock while we were walking the
- * pi chain. Let the waiter sort it out.
- */
- ret = 0;
- }
-
- if (unlikely(ret))
- remove_waiter(lock, waiter);
-
+ ret = __rt_mutex_start_proxy_lock(lock, waiter, task);
raw_spin_unlock_irq(&lock->wait_lock);
- debug_rt_mutex_print_deadlock(waiter);
-
return ret;
}
@@ -1731,21 +1762,23 @@
}
/**
- * rt_mutex_finish_proxy_lock() - Complete lock acquisition
+ * rt_mutex_wait_proxy_lock() - Wait for lock acquisition
* @lock: the rt_mutex we were woken on
* @to: the timeout, null if none. hrtimer should already have
* been started.
* @waiter: the pre-initialized rt_mutex_waiter
*
- * Complete the lock acquisition started our behalf by another thread.
+ * Wait for the the lock acquisition started on our behalf by
+ * rt_mutex_start_proxy_lock(). Upon failure, the caller must call
+ * rt_mutex_cleanup_proxy_lock().
*
* Returns:
* 0 - success
* <0 - error, one of -EINTR, -ETIMEDOUT
*
- * Special API call for PI-futex requeue support
+ * Special API call for PI-futex support
*/
-int rt_mutex_finish_proxy_lock(struct rt_mutex *lock,
+int rt_mutex_wait_proxy_lock(struct rt_mutex *lock,
struct hrtimer_sleeper *to,
struct rt_mutex_waiter *waiter)
{
@@ -1758,8 +1791,45 @@
/* sleep on the mutex */
ret = __rt_mutex_slowlock(lock, TASK_INTERRUPTIBLE, to, waiter);
- if (unlikely(ret))
+ raw_spin_unlock_irq(&lock->wait_lock);
+
+ return ret;
+}
+
+/**
+ * rt_mutex_cleanup_proxy_lock() - Cleanup failed lock acquisition
+ * @lock: the rt_mutex we were woken on
+ * @waiter: the pre-initialized rt_mutex_waiter
+ *
+ * Attempt to clean up after a failed rt_mutex_wait_proxy_lock().
+ *
+ * Unless we acquired the lock; we're still enqueued on the wait-list and can
+ * in fact still be granted ownership until we're removed. Therefore we can
+ * find we are in fact the owner and must disregard the
+ * rt_mutex_wait_proxy_lock() failure.
+ *
+ * Returns:
+ * true - did the cleanup, we done.
+ * false - we acquired the lock after rt_mutex_wait_proxy_lock() returned,
+ * caller should disregards its return value.
+ *
+ * Special API call for PI-futex support
+ */
+bool rt_mutex_cleanup_proxy_lock(struct rt_mutex *lock,
+ struct rt_mutex_waiter *waiter)
+{
+ bool cleanup = false;
+
+ raw_spin_lock_irq(&lock->wait_lock);
+ /*
+ * Unless we're the owner; we're still enqueued on the wait_list.
+ * So check if we became owner, if not, take us off the wait_list.
+ */
+ if (rt_mutex_owner(lock) != current) {
remove_waiter(lock, waiter);
+ fixup_rt_mutex_waiters(lock);
+ cleanup = true;
+ }
/*
* try_to_take_rt_mutex() sets the waiter bit unconditionally. We might
@@ -1769,5 +1839,5 @@
raw_spin_unlock_irq(&lock->wait_lock);
- return ret;
+ return cleanup;
}
diff --git a/kernel/locking/rtmutex.h b/kernel/locking/rtmutex.h
index c406058..6607802 100644
--- a/kernel/locking/rtmutex.h
+++ b/kernel/locking/rtmutex.h
@@ -11,8 +11,6 @@
*/
#define rt_mutex_deadlock_check(l) (0)
-#define rt_mutex_deadlock_account_lock(m, t) do { } while (0)
-#define rt_mutex_deadlock_account_unlock(l) do { } while (0)
#define debug_rt_mutex_init_waiter(w) do { } while (0)
#define debug_rt_mutex_free_waiter(w) do { } while (0)
#define debug_rt_mutex_lock(l) do { } while (0)
diff --git a/kernel/locking/rtmutex_common.h b/kernel/locking/rtmutex_common.h
index 856dfff..72ad45a 100644
--- a/kernel/locking/rtmutex_common.h
+++ b/kernel/locking/rtmutex_common.h
@@ -34,6 +34,7 @@
struct rt_mutex *deadlock_lock;
#endif
int prio;
+ u64 deadline;
};
/*
@@ -103,16 +104,26 @@
struct task_struct *proxy_owner);
extern void rt_mutex_proxy_unlock(struct rt_mutex *lock,
struct task_struct *proxy_owner);
+extern void rt_mutex_init_waiter(struct rt_mutex_waiter *waiter);
+extern int __rt_mutex_start_proxy_lock(struct rt_mutex *lock,
+ struct rt_mutex_waiter *waiter,
+ struct task_struct *task);
extern int rt_mutex_start_proxy_lock(struct rt_mutex *lock,
struct rt_mutex_waiter *waiter,
struct task_struct *task);
-extern int rt_mutex_finish_proxy_lock(struct rt_mutex *lock,
- struct hrtimer_sleeper *to,
- struct rt_mutex_waiter *waiter);
-extern int rt_mutex_timed_futex_lock(struct rt_mutex *l, struct hrtimer_sleeper *to);
-extern bool rt_mutex_futex_unlock(struct rt_mutex *lock,
- struct wake_q_head *wqh);
-extern void rt_mutex_adjust_prio(struct task_struct *task);
+extern int rt_mutex_wait_proxy_lock(struct rt_mutex *lock,
+ struct hrtimer_sleeper *to,
+ struct rt_mutex_waiter *waiter);
+extern bool rt_mutex_cleanup_proxy_lock(struct rt_mutex *lock,
+ struct rt_mutex_waiter *waiter);
+
+extern int rt_mutex_futex_trylock(struct rt_mutex *l);
+
+extern void rt_mutex_futex_unlock(struct rt_mutex *lock);
+extern bool __rt_mutex_futex_unlock(struct rt_mutex *lock,
+ struct wake_q_head *wqh);
+
+extern void rt_mutex_postunlock(struct wake_q_head *wake_q);
#ifdef CONFIG_DEBUG_RT_MUTEXES
# include "rtmutex-debug.h"
diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c
index 90a74cc..4d48b1c 100644
--- a/kernel/locking/rwsem.c
+++ b/kernel/locking/rwsem.c
@@ -124,10 +124,8 @@
*/
void downgrade_write(struct rw_semaphore *sem)
{
- /*
- * lockdep: a downgraded write will live on as a write
- * dependency.
- */
+ lock_downgrade(&sem->dep_map, _RET_IP_);
+
rwsem_set_reader_owned(sem);
__downgrade_write(sem);
}
diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
index 6b7abb3..39f56c8 100644
--- a/kernel/locking/test-ww_mutex.c
+++ b/kernel/locking/test-ww_mutex.c
@@ -353,8 +353,8 @@
struct stress {
struct work_struct work;
struct ww_mutex *locks;
+ unsigned long timeout;
int nlocks;
- int nloops;
};
static int *get_random_order(int count)
@@ -398,12 +398,11 @@
if (!order)
return;
- ww_acquire_init(&ctx, &ww_class);
-
do {
int contended = -1;
int n, err;
+ ww_acquire_init(&ctx, &ww_class);
retry:
err = 0;
for (n = 0; n < nlocks; n++) {
@@ -433,9 +432,9 @@
__func__, err);
break;
}
- } while (--stress->nloops);
- ww_acquire_fini(&ctx);
+ ww_acquire_fini(&ctx);
+ } while (!time_after(jiffies, stress->timeout));
kfree(order);
kfree(stress);
@@ -470,9 +469,9 @@
kfree(order);
order = NULL;
- ww_acquire_init(&ctx, &ww_class);
-
do {
+ ww_acquire_init(&ctx, &ww_class);
+
list_for_each_entry(ll, &locks, link) {
err = ww_mutex_lock(ll->lock, &ctx);
if (!err)
@@ -495,9 +494,9 @@
dummy_load(stress);
list_for_each_entry(ll, &locks, link)
ww_mutex_unlock(ll->lock);
- } while (--stress->nloops);
- ww_acquire_fini(&ctx);
+ ww_acquire_fini(&ctx);
+ } while (!time_after(jiffies, stress->timeout));
out:
list_for_each_entry_safe(ll, ln, &locks, link)
@@ -523,7 +522,7 @@
__func__, err);
break;
}
- } while (--stress->nloops);
+ } while (!time_after(jiffies, stress->timeout));
kfree(stress);
}
@@ -533,7 +532,7 @@
#define STRESS_ONE BIT(2)
#define STRESS_ALL (STRESS_INORDER | STRESS_REORDER | STRESS_ONE)
-static int stress(int nlocks, int nthreads, int nloops, unsigned int flags)
+static int stress(int nlocks, int nthreads, unsigned int flags)
{
struct ww_mutex *locks;
int n;
@@ -575,7 +574,7 @@
INIT_WORK(&stress->work, fn);
stress->locks = locks;
stress->nlocks = nlocks;
- stress->nloops = nloops;
+ stress->timeout = jiffies + 2*HZ;
queue_work(wq, &stress->work);
nthreads--;
@@ -619,15 +618,15 @@
if (ret)
return ret;
- ret = stress(16, 2*ncpus, 1<<10, STRESS_INORDER);
+ ret = stress(16, 2*ncpus, STRESS_INORDER);
if (ret)
return ret;
- ret = stress(16, 2*ncpus, 1<<10, STRESS_REORDER);
+ ret = stress(16, 2*ncpus, STRESS_REORDER);
if (ret)
return ret;
- ret = stress(4095, hweight32(STRESS_ALL)*ncpus, 1<<12, STRESS_ALL);
+ ret = stress(4095, hweight32(STRESS_ALL)*ncpus, STRESS_ALL);
if (ret)
return ret;
diff --git a/kernel/memremap.c b/kernel/memremap.c
index 07e85e5..23a6483 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -182,18 +182,6 @@
struct vmem_altmap altmap;
};
-void get_zone_device_page(struct page *page)
-{
- percpu_ref_get(page->pgmap->ref);
-}
-EXPORT_SYMBOL(get_zone_device_page);
-
-void put_zone_device_page(struct page *page)
-{
- put_dev_pagemap(page->pgmap);
-}
-EXPORT_SYMBOL(put_zone_device_page);
-
static void pgmap_radix_release(struct resource *res)
{
resource_size_t key, align_start, align_size, align_end;
@@ -237,6 +225,10 @@
struct resource *res = &page_map->res;
resource_size_t align_start, align_size;
struct dev_pagemap *pgmap = &page_map->pgmap;
+ unsigned long pfn;
+
+ for_each_device_pfn(pfn, page_map)
+ put_page(pfn_to_page(pfn));
if (percpu_ref_tryget_live(pgmap->ref)) {
dev_WARN(dev, "%s: page mapping is still live!\n", __func__);
@@ -277,7 +269,10 @@
*
* Notes:
* 1/ @ref must be 'live' on entry and 'dead' before devm_memunmap_pages() time
- * (or devm release event).
+ * (or devm release event). The expected order of events is that @ref has
+ * been through percpu_ref_kill() before devm_memremap_pages_release(). The
+ * wait for the completion of all references being dropped and
+ * percpu_ref_exit() must occur after devm_memremap_pages_release().
*
* 2/ @res is expected to be a host memory range that could feasibly be
* treated as a "System RAM" range, i.e. not a device mmio range, but
@@ -379,6 +374,7 @@
*/
list_del(&page->lru);
page->pgmap = pgmap;
+ percpu_ref_get(ref);
}
devres_add(dev, page_map);
return __va(res->start);
diff --git a/kernel/module.c b/kernel/module.c
index 7eba6de..6d99880 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -665,16 +665,7 @@
memcpy(per_cpu_ptr(mod->percpu, cpu), from, size);
}
-/**
- * is_module_percpu_address - test whether address is from module static percpu
- * @addr: address to test
- *
- * Test whether @addr belongs to module static percpu area.
- *
- * RETURNS:
- * %true if @addr is from module static percpu area
- */
-bool is_module_percpu_address(unsigned long addr)
+bool __is_module_percpu_address(unsigned long addr, unsigned long *can_addr)
{
struct module *mod;
unsigned int cpu;
@@ -688,9 +679,15 @@
continue;
for_each_possible_cpu(cpu) {
void *start = per_cpu_ptr(mod->percpu, cpu);
+ void *va = (void *)addr;
- if ((void *)addr >= start &&
- (void *)addr < start + mod->percpu_size) {
+ if (va >= start && va < start + mod->percpu_size) {
+ if (can_addr) {
+ *can_addr = (unsigned long) (va - start);
+ *can_addr += (unsigned long)
+ per_cpu_ptr(mod->percpu,
+ get_boot_cpu_id());
+ }
preempt_enable();
return true;
}
@@ -701,6 +698,20 @@
return false;
}
+/**
+ * is_module_percpu_address - test whether address is from module static percpu
+ * @addr: address to test
+ *
+ * Test whether @addr belongs to module static percpu area.
+ *
+ * RETURNS:
+ * %true if @addr is from module static percpu area
+ */
+bool is_module_percpu_address(unsigned long addr)
+{
+ return __is_module_percpu_address(addr, NULL);
+}
+
#else /* ... !CONFIG_SMP */
static inline void __percpu *mod_percpu(struct module *mod)
@@ -732,6 +743,11 @@
return false;
}
+bool __is_module_percpu_address(unsigned long addr, unsigned long *can_addr)
+{
+ return false;
+}
+
#endif /* CONFIG_SMP */
#define MODINFO_ATTR(field) \
diff --git a/kernel/nsproxy.c b/kernel/nsproxy.c
index 782102e..f6c5d33 100644
--- a/kernel/nsproxy.c
+++ b/kernel/nsproxy.c
@@ -26,6 +26,7 @@
#include <linux/file.h>
#include <linux/syscalls.h>
#include <linux/cgroup.h>
+#include <linux/perf_event.h>
static struct kmem_cache *nsproxy_cachep;
@@ -262,6 +263,8 @@
goto out;
}
switch_task_namespaces(tsk, new_nsproxy);
+
+ perf_event_namespaces(tsk);
out:
fput(file);
return err;
diff --git a/kernel/padata.c b/kernel/padata.c
index 3202aa1..ac8f1e5 100644
--- a/kernel/padata.c
+++ b/kernel/padata.c
@@ -154,8 +154,6 @@
* A pointer to the control struct of the next object that needs
* serialization, if present in one of the percpu reorder queues.
*
- * NULL, if all percpu reorder queues are empty.
- *
* -EINPROGRESS, if the next object that needs serialization will
* be parallel processed by another cpu and is not yet present in
* the cpu's reorder queue.
@@ -182,8 +180,6 @@
cpu = padata_index_to_cpu(pd, next_index);
next_queue = per_cpu_ptr(pd->pqueue, cpu);
- padata = NULL;
-
reorder = &next_queue->reorder;
spin_lock(&reorder->lock);
@@ -235,12 +231,11 @@
padata = padata_get_next(pd);
/*
- * All reorder queues are empty, or the next object that needs
- * serialization is parallel processed by another cpu and is
- * still on it's way to the cpu's reorder queue, nothing to
- * do for now.
+ * If the next object that needs serialization is parallel
+ * processed by another cpu and is still on it's way to the
+ * cpu's reorder queue, nothing to do for now.
*/
- if (!padata || PTR_ERR(padata) == -EINPROGRESS)
+ if (PTR_ERR(padata) == -EINPROGRESS)
break;
/*
@@ -354,7 +349,7 @@
cpumask_and(pd->cpumask.pcpu, pcpumask, cpu_online_mask);
if (!alloc_cpumask_var(&pd->cpumask.cbcpu, GFP_KERNEL)) {
- free_cpumask_var(pd->cpumask.cbcpu);
+ free_cpumask_var(pd->cpumask.pcpu);
return -ENOMEM;
}
diff --git a/kernel/params.c b/kernel/params.c
index a6d6149..60b2d81 100644
--- a/kernel/params.c
+++ b/kernel/params.c
@@ -160,58 +160,6 @@
return -ENOENT;
}
-/* You can use " around spaces, but can't escape ". */
-/* Hyphens and underscores equivalent in parameter names. */
-static char *next_arg(char *args, char **param, char **val)
-{
- unsigned int i, equals = 0;
- int in_quote = 0, quoted = 0;
- char *next;
-
- if (*args == '"') {
- args++;
- in_quote = 1;
- quoted = 1;
- }
-
- for (i = 0; args[i]; i++) {
- if (isspace(args[i]) && !in_quote)
- break;
- if (equals == 0) {
- if (args[i] == '=')
- equals = i;
- }
- if (args[i] == '"')
- in_quote = !in_quote;
- }
-
- *param = args;
- if (!equals)
- *val = NULL;
- else {
- args[equals] = '\0';
- *val = args + equals + 1;
-
- /* Don't include quotes in value. */
- if (**val == '"') {
- (*val)++;
- if (args[i-1] == '"')
- args[i-1] = '\0';
- }
- }
- if (quoted && args[i-1] == '"')
- args[i-1] = '\0';
-
- if (args[i]) {
- args[i] = '\0';
- next = args + i + 1;
- } else
- next = args + i;
-
- /* Chew up trailing spaces. */
- return skip_spaces(next);
-}
-
/* Args looks like "foo=bar,bar2 baz=fuz wiz". */
char *parse_args(const char *doing,
char *args,
diff --git a/kernel/relay.c b/kernel/relay.c
index 0e413d9..39a9dfc 100644
--- a/kernel/relay.c
+++ b/kernel/relay.c
@@ -1212,7 +1212,6 @@
.nr_pages = 0,
.nr_pages_max = PIPE_DEF_BUFFERS,
.partial = partial,
- .flags = flags,
.ops = &relay_pipe_buf_ops,
.spd_release = relay_page_release,
};
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 3b31fc0..c51147a 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -86,21 +86,6 @@
cpumask_var_t cpu_isolated_map;
/*
- * this_rq_lock - lock this runqueue and disable interrupts.
- */
-static struct rq *this_rq_lock(void)
- __acquires(rq->lock)
-{
- struct rq *rq;
-
- local_irq_disable();
- rq = this_rq();
- raw_spin_lock(&rq->lock);
-
- return rq;
-}
-
-/*
* __task_rq_lock - lock the rq @p resides on.
*/
struct rq *__task_rq_lock(struct task_struct *p, struct rq_flags *rf)
@@ -233,8 +218,11 @@
return;
#ifdef CONFIG_SCHED_DEBUG
+ if (sched_feat(WARN_DOUBLE_CLOCK))
+ SCHED_WARN_ON(rq->clock_update_flags & RQCF_UPDATED);
rq->clock_update_flags |= RQCF_UPDATED;
#endif
+
delta = sched_clock_cpu(cpu_of(rq)) - rq->clock;
if (delta < 0)
return;
@@ -261,13 +249,14 @@
static enum hrtimer_restart hrtick(struct hrtimer *timer)
{
struct rq *rq = container_of(timer, struct rq, hrtick_timer);
+ struct rq_flags rf;
WARN_ON_ONCE(cpu_of(rq) != smp_processor_id());
- raw_spin_lock(&rq->lock);
+ rq_lock(rq, &rf);
update_rq_clock(rq);
rq->curr->sched_class->task_tick(rq, rq->curr, 1);
- raw_spin_unlock(&rq->lock);
+ rq_unlock(rq, &rf);
return HRTIMER_NORESTART;
}
@@ -287,11 +276,12 @@
static void __hrtick_start(void *arg)
{
struct rq *rq = arg;
+ struct rq_flags rf;
- raw_spin_lock(&rq->lock);
+ rq_lock(rq, &rf);
__hrtick_restart(rq);
rq->hrtick_csd_pending = 0;
- raw_spin_unlock(&rq->lock);
+ rq_unlock(rq, &rf);
}
/*
@@ -762,17 +752,23 @@
static inline void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
{
- update_rq_clock(rq);
+ if (!(flags & ENQUEUE_NOCLOCK))
+ update_rq_clock(rq);
+
if (!(flags & ENQUEUE_RESTORE))
sched_info_queued(rq, p);
+
p->sched_class->enqueue_task(rq, p, flags);
}
static inline void dequeue_task(struct rq *rq, struct task_struct *p, int flags)
{
- update_rq_clock(rq);
+ if (!(flags & DEQUEUE_NOCLOCK))
+ update_rq_clock(rq);
+
if (!(flags & DEQUEUE_SAVE))
sched_info_dequeued(rq, p);
+
p->sched_class->dequeue_task(rq, p, flags);
}
@@ -946,18 +942,19 @@
*
* Returns (locked) new rq. Old rq's lock is released.
*/
-static struct rq *move_queued_task(struct rq *rq, struct task_struct *p, int new_cpu)
+static struct rq *move_queued_task(struct rq *rq, struct rq_flags *rf,
+ struct task_struct *p, int new_cpu)
{
lockdep_assert_held(&rq->lock);
p->on_rq = TASK_ON_RQ_MIGRATING;
- dequeue_task(rq, p, 0);
+ dequeue_task(rq, p, DEQUEUE_NOCLOCK);
set_task_cpu(p, new_cpu);
- raw_spin_unlock(&rq->lock);
+ rq_unlock(rq, rf);
rq = cpu_rq(new_cpu);
- raw_spin_lock(&rq->lock);
+ rq_lock(rq, rf);
BUG_ON(task_cpu(p) != new_cpu);
enqueue_task(rq, p, 0);
p->on_rq = TASK_ON_RQ_QUEUED;
@@ -980,7 +977,8 @@
* So we race with normal scheduler movements, but that's OK, as long
* as the task is no longer on this CPU.
*/
-static struct rq *__migrate_task(struct rq *rq, struct task_struct *p, int dest_cpu)
+static struct rq *__migrate_task(struct rq *rq, struct rq_flags *rf,
+ struct task_struct *p, int dest_cpu)
{
if (unlikely(!cpu_active(dest_cpu)))
return rq;
@@ -989,7 +987,8 @@
if (!cpumask_test_cpu(dest_cpu, &p->cpus_allowed))
return rq;
- rq = move_queued_task(rq, p, dest_cpu);
+ update_rq_clock(rq);
+ rq = move_queued_task(rq, rf, p, dest_cpu);
return rq;
}
@@ -1004,6 +1003,7 @@
struct migration_arg *arg = data;
struct task_struct *p = arg->task;
struct rq *rq = this_rq();
+ struct rq_flags rf;
/*
* The original target CPU might have gone down and we might
@@ -1018,7 +1018,7 @@
sched_ttwu_pending();
raw_spin_lock(&p->pi_lock);
- raw_spin_lock(&rq->lock);
+ rq_lock(rq, &rf);
/*
* If task_rq(p) != rq, it cannot be migrated here, because we're
* holding rq->lock, if p->on_rq == 0 it cannot get enqueued because
@@ -1026,11 +1026,11 @@
*/
if (task_rq(p) == rq) {
if (task_on_rq_queued(p))
- rq = __migrate_task(rq, p, arg->dest_cpu);
+ rq = __migrate_task(rq, &rf, p, arg->dest_cpu);
else
p->wake_cpu = arg->dest_cpu;
}
- raw_spin_unlock(&rq->lock);
+ rq_unlock(rq, &rf);
raw_spin_unlock(&p->pi_lock);
local_irq_enable();
@@ -1063,7 +1063,7 @@
* holding rq->lock.
*/
lockdep_assert_held(&rq->lock);
- dequeue_task(rq, p, DEQUEUE_SAVE);
+ dequeue_task(rq, p, DEQUEUE_SAVE | DEQUEUE_NOCLOCK);
}
if (running)
put_prev_task(rq, p);
@@ -1071,7 +1071,7 @@
p->sched_class->set_cpus_allowed(p, new_mask);
if (queued)
- enqueue_task(rq, p, ENQUEUE_RESTORE);
+ enqueue_task(rq, p, ENQUEUE_RESTORE | ENQUEUE_NOCLOCK);
if (running)
set_curr_task(rq, p);
}
@@ -1150,9 +1150,7 @@
* OK, since we're going to drop the lock immediately
* afterwards anyway.
*/
- rq_unpin_lock(rq, &rf);
- rq = move_queued_task(rq, p, dest_cpu);
- rq_repin_lock(rq, &rf);
+ rq = move_queued_task(rq, &rf, p, dest_cpu);
}
out:
task_rq_unlock(rq, p, &rf);
@@ -1217,16 +1215,24 @@
{
if (task_on_rq_queued(p)) {
struct rq *src_rq, *dst_rq;
+ struct rq_flags srf, drf;
src_rq = task_rq(p);
dst_rq = cpu_rq(cpu);
+ rq_pin_lock(src_rq, &srf);
+ rq_pin_lock(dst_rq, &drf);
+
p->on_rq = TASK_ON_RQ_MIGRATING;
deactivate_task(src_rq, p, 0);
set_task_cpu(p, cpu);
activate_task(dst_rq, p, 0);
p->on_rq = TASK_ON_RQ_QUEUED;
check_preempt_curr(dst_rq, p, 0);
+
+ rq_unpin_lock(dst_rq, &drf);
+ rq_unpin_lock(src_rq, &srf);
+
} else {
/*
* Task isn't running anymore; make it appear like we migrated
@@ -1680,7 +1686,7 @@
ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
struct rq_flags *rf)
{
- int en_flags = ENQUEUE_WAKEUP;
+ int en_flags = ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK;
lockdep_assert_held(&rq->lock);
@@ -1726,14 +1732,13 @@
struct rq *rq = this_rq();
struct llist_node *llist = llist_del_all(&rq->wake_list);
struct task_struct *p;
- unsigned long flags;
struct rq_flags rf;
if (!llist)
return;
- raw_spin_lock_irqsave(&rq->lock, flags);
- rq_pin_lock(rq, &rf);
+ rq_lock_irqsave(rq, &rf);
+ update_rq_clock(rq);
while (llist) {
int wake_flags = 0;
@@ -1747,8 +1752,7 @@
ttwu_do_activate(rq, p, wake_flags, &rf);
}
- rq_unpin_lock(rq, &rf);
- raw_spin_unlock_irqrestore(&rq->lock, flags);
+ rq_unlock_irqrestore(rq, &rf);
}
void scheduler_ipi(void)
@@ -1806,7 +1810,7 @@
void wake_up_if_idle(int cpu)
{
struct rq *rq = cpu_rq(cpu);
- unsigned long flags;
+ struct rq_flags rf;
rcu_read_lock();
@@ -1816,11 +1820,11 @@
if (set_nr_if_polling(rq->idle)) {
trace_sched_wake_idle_without_ipi(cpu);
} else {
- raw_spin_lock_irqsave(&rq->lock, flags);
+ rq_lock_irqsave(rq, &rf);
if (is_idle_task(rq->curr))
smp_send_reschedule(cpu);
/* Else CPU is not idle, do nothing here: */
- raw_spin_unlock_irqrestore(&rq->lock, flags);
+ rq_unlock_irqrestore(rq, &rf);
}
out:
@@ -1846,11 +1850,10 @@
}
#endif
- raw_spin_lock(&rq->lock);
- rq_pin_lock(rq, &rf);
+ rq_lock(rq, &rf);
+ update_rq_clock(rq);
ttwu_do_activate(rq, p, wake_flags, &rf);
- rq_unpin_lock(rq, &rf);
- raw_spin_unlock(&rq->lock);
+ rq_unlock(rq, &rf);
}
/*
@@ -2097,11 +2100,9 @@
* disabled avoiding further scheduler activity on it and we've
* not yet picked a replacement task.
*/
- rq_unpin_lock(rq, rf);
- raw_spin_unlock(&rq->lock);
+ rq_unlock(rq, rf);
raw_spin_lock(&p->pi_lock);
- raw_spin_lock(&rq->lock);
- rq_repin_lock(rq, rf);
+ rq_relock(rq, rf);
}
if (!(p->state & TASK_NORMAL))
@@ -2114,7 +2115,7 @@
delayacct_blkio_end();
atomic_dec(&rq->nr_iowait);
}
- ttwu_activate(rq, p, ENQUEUE_WAKEUP);
+ ttwu_activate(rq, p, ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK);
}
ttwu_do_wakeup(rq, p, 0, rf);
@@ -2555,7 +2556,7 @@
update_rq_clock(rq);
post_init_entity_util_avg(&p->se);
- activate_task(rq, p, 0);
+ activate_task(rq, p, ENQUEUE_NOCLOCK);
p->on_rq = TASK_ON_RQ_QUEUED;
trace_sched_wakeup_new(p);
check_preempt_curr(rq, p, WF_FORK);
@@ -3093,15 +3094,18 @@
int cpu = smp_processor_id();
struct rq *rq = cpu_rq(cpu);
struct task_struct *curr = rq->curr;
+ struct rq_flags rf;
sched_clock_tick();
- raw_spin_lock(&rq->lock);
+ rq_lock(rq, &rf);
+
update_rq_clock(rq);
curr->sched_class->task_tick(rq, curr, 0);
cpu_load_update_active(rq);
calc_global_load_tick(rq);
- raw_spin_unlock(&rq->lock);
+
+ rq_unlock(rq, &rf);
perf_event_task_tick();
@@ -3386,18 +3390,18 @@
* done by the caller to avoid the race with signal_wake_up().
*/
smp_mb__before_spinlock();
- raw_spin_lock(&rq->lock);
- rq_pin_lock(rq, &rf);
+ rq_lock(rq, &rf);
/* Promote REQ to ACT */
rq->clock_update_flags <<= 1;
+ update_rq_clock(rq);
switch_count = &prev->nivcsw;
if (!preempt && prev->state) {
if (unlikely(signal_pending_state(prev->state, prev))) {
prev->state = TASK_RUNNING;
} else {
- deactivate_task(rq, prev, DEQUEUE_SLEEP);
+ deactivate_task(rq, prev, DEQUEUE_SLEEP | DEQUEUE_NOCLOCK);
prev->on_rq = 0;
if (prev->in_iowait) {
@@ -3421,9 +3425,6 @@
switch_count = &prev->nvcsw;
}
- if (task_on_rq_queued(prev))
- update_rq_clock(rq);
-
next = pick_next_task(rq, prev, &rf);
clear_tsk_need_resched(prev);
clear_preempt_need_resched();
@@ -3439,8 +3440,7 @@
rq = context_switch(rq, prev, next, &rf);
} else {
rq->clock_update_flags &= ~(RQCF_ACT_SKIP|RQCF_REQ_SKIP);
- rq_unpin_lock(rq, &rf);
- raw_spin_unlock_irq(&rq->lock);
+ rq_unlock_irq(rq, &rf);
}
balance_callback(rq);
@@ -3671,10 +3671,25 @@
#ifdef CONFIG_RT_MUTEXES
+static inline int __rt_effective_prio(struct task_struct *pi_task, int prio)
+{
+ if (pi_task)
+ prio = min(prio, pi_task->prio);
+
+ return prio;
+}
+
+static inline int rt_effective_prio(struct task_struct *p, int prio)
+{
+ struct task_struct *pi_task = rt_mutex_get_top_task(p);
+
+ return __rt_effective_prio(pi_task, prio);
+}
+
/*
* rt_mutex_setprio - set the current priority of a task
- * @p: task
- * @prio: prio value (kernel-internal form)
+ * @p: task to boost
+ * @pi_task: donor task
*
* This function changes the 'effective' priority of a task. It does
* not touch ->normal_prio like __setscheduler().
@@ -3682,17 +3697,42 @@
* Used by the rt_mutex code to implement priority inheritance
* logic. Call site only calls if the priority of the task changed.
*/
-void rt_mutex_setprio(struct task_struct *p, int prio)
+void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
{
- int oldprio, queued, running, queue_flag = DEQUEUE_SAVE | DEQUEUE_MOVE;
+ int prio, oldprio, queued, running, queue_flag =
+ DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK;
const struct sched_class *prev_class;
struct rq_flags rf;
struct rq *rq;
- BUG_ON(prio > MAX_PRIO);
+ /* XXX used to be waiter->prio, not waiter->task->prio */
+ prio = __rt_effective_prio(pi_task, p->normal_prio);
+
+ /*
+ * If nothing changed; bail early.
+ */
+ if (p->pi_top_task == pi_task && prio == p->prio && !dl_prio(prio))
+ return;
rq = __task_rq_lock(p, &rf);
update_rq_clock(rq);
+ /*
+ * Set under pi_lock && rq->lock, such that the value can be used under
+ * either lock.
+ *
+ * Note that there is loads of tricky to make this pointer cache work
+ * right. rt_mutex_slowunlock()+rt_mutex_postunlock() work together to
+ * ensure a task is de-boosted (pi_task is set to NULL) before the
+ * task is allowed to run again (and can exit). This ensures the pointer
+ * points to a blocked task -- which guaratees the task is present.
+ */
+ p->pi_top_task = pi_task;
+
+ /*
+ * For FIFO/RR we only need to set prio, if that matches we're done.
+ */
+ if (prio == p->prio && !dl_prio(prio))
+ goto out_unlock;
/*
* Idle task boosting is a nono in general. There is one
@@ -3712,7 +3752,7 @@
goto out_unlock;
}
- trace_sched_pi_setprio(p, prio);
+ trace_sched_pi_setprio(p, pi_task);
oldprio = p->prio;
if (oldprio == prio)
@@ -3736,7 +3776,6 @@
* running task
*/
if (dl_prio(prio)) {
- struct task_struct *pi_task = rt_mutex_get_top_task(p);
if (!dl_prio(p->normal_prio) ||
(pi_task && dl_entity_preempt(&pi_task->dl, &p->dl))) {
p->dl.dl_boosted = 1;
@@ -3774,6 +3813,11 @@
balance_callback(rq);
preempt_enable();
}
+#else
+static inline int rt_effective_prio(struct task_struct *p, int prio)
+{
+ return prio;
+}
#endif
void set_user_nice(struct task_struct *p, long nice)
@@ -3805,7 +3849,7 @@
queued = task_on_rq_queued(p);
running = task_current(rq, p);
if (queued)
- dequeue_task(rq, p, DEQUEUE_SAVE);
+ dequeue_task(rq, p, DEQUEUE_SAVE | DEQUEUE_NOCLOCK);
if (running)
put_prev_task(rq, p);
@@ -3816,7 +3860,7 @@
delta = p->prio - old_prio;
if (queued) {
- enqueue_task(rq, p, ENQUEUE_RESTORE);
+ enqueue_task(rq, p, ENQUEUE_RESTORE | ENQUEUE_NOCLOCK);
/*
* If the task increased its priority or is running and
* lowered its priority, then reschedule its CPU:
@@ -4020,10 +4064,9 @@
* Keep a potential priority boosting if called from
* sched_setscheduler().
*/
+ p->prio = normal_prio(p);
if (keep_boost)
- p->prio = rt_mutex_get_effective_prio(p, normal_prio(p));
- else
- p->prio = normal_prio(p);
+ p->prio = rt_effective_prio(p, p->prio);
if (dl_prio(p->prio))
p->sched_class = &dl_sched_class;
@@ -4126,7 +4169,7 @@
const struct sched_class *prev_class;
struct rq_flags rf;
int reset_on_fork;
- int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE;
+ int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK;
struct rq *rq;
/* May grab non-irq protected spin_locks: */
@@ -4310,7 +4353,7 @@
* the runqueue. This will be done when the task deboost
* itself.
*/
- new_effective_prio = rt_mutex_get_effective_prio(p, newprio);
+ new_effective_prio = rt_effective_prio(p, newprio);
if (new_effective_prio == oldprio)
queue_flags &= ~DEQUEUE_MOVE;
}
@@ -4923,7 +4966,12 @@
*/
SYSCALL_DEFINE0(sched_yield)
{
- struct rq *rq = this_rq_lock();
+ struct rq_flags rf;
+ struct rq *rq;
+
+ local_irq_disable();
+ rq = this_rq();
+ rq_lock(rq, &rf);
schedstat_inc(rq->yld_count);
current->sched_class->yield_task(rq);
@@ -4932,9 +4980,8 @@
* Since we are going to call schedule() anyway, there's
* no need to preempt or enable interrupts:
*/
- __release(rq->lock);
- spin_release(&rq->lock.dep_map, 1, _THIS_IP_);
- do_raw_spin_unlock(&rq->lock);
+ preempt_disable();
+ rq_unlock(rq, &rf);
sched_preempt_enable_no_resched();
schedule();
@@ -5514,7 +5561,7 @@
p->numa_preferred_nid = nid;
if (queued)
- enqueue_task(rq, p, ENQUEUE_RESTORE);
+ enqueue_task(rq, p, ENQUEUE_RESTORE | ENQUEUE_NOCLOCK);
if (running)
set_curr_task(rq, p);
task_rq_unlock(rq, p, &rf);
@@ -5579,11 +5626,11 @@
* there's no concurrency possible, we hold the required locks anyway
* because of lock validation efforts.
*/
-static void migrate_tasks(struct rq *dead_rq)
+static void migrate_tasks(struct rq *dead_rq, struct rq_flags *rf)
{
struct rq *rq = dead_rq;
struct task_struct *next, *stop = rq->stop;
- struct rq_flags rf;
+ struct rq_flags orf = *rf;
int dest_cpu;
/*
@@ -5602,9 +5649,7 @@
* class method both need to have an up-to-date
* value of rq->clock[_task]
*/
- rq_pin_lock(rq, &rf);
update_rq_clock(rq);
- rq_unpin_lock(rq, &rf);
for (;;) {
/*
@@ -5617,8 +5662,7 @@
/*
* pick_next_task() assumes pinned rq->lock:
*/
- rq_repin_lock(rq, &rf);
- next = pick_next_task(rq, &fake_task, &rf);
+ next = pick_next_task(rq, &fake_task, rf);
BUG_ON(!next);
next->sched_class->put_prev_task(rq, next);
@@ -5631,10 +5675,9 @@
* because !cpu_active at this point, which means load-balance
* will not interfere. Also, stop-machine.
*/
- rq_unpin_lock(rq, &rf);
- raw_spin_unlock(&rq->lock);
+ rq_unlock(rq, rf);
raw_spin_lock(&next->pi_lock);
- raw_spin_lock(&rq->lock);
+ rq_relock(rq, rf);
/*
* Since we're inside stop-machine, _nothing_ should have
@@ -5648,12 +5691,12 @@
/* Find suitable destination for @next, with force if needed. */
dest_cpu = select_fallback_rq(dead_rq->cpu, next);
-
- rq = __migrate_task(rq, next, dest_cpu);
+ rq = __migrate_task(rq, rf, next, dest_cpu);
if (rq != dead_rq) {
- raw_spin_unlock(&rq->lock);
+ rq_unlock(rq, rf);
rq = dead_rq;
- raw_spin_lock(&rq->lock);
+ *rf = orf;
+ rq_relock(rq, rf);
}
raw_spin_unlock(&next->pi_lock);
}
@@ -5732,7 +5775,7 @@
* cpuset configurations.
*/
}
- cpuset_update_active_cpus(true);
+ cpuset_update_active_cpus();
}
static int cpuset_cpu_inactive(unsigned int cpu)
@@ -5755,7 +5798,7 @@
if (overflow)
return -EBUSY;
- cpuset_update_active_cpus(false);
+ cpuset_update_active_cpus();
} else {
num_cpus_frozen++;
partition_sched_domains(1, NULL, NULL);
@@ -5766,7 +5809,7 @@
int sched_cpu_activate(unsigned int cpu)
{
struct rq *rq = cpu_rq(cpu);
- unsigned long flags;
+ struct rq_flags rf;
set_cpu_active(cpu, true);
@@ -5784,12 +5827,12 @@
* 2) At runtime, if cpuset_cpu_active() fails to rebuild the
* domains.
*/
- raw_spin_lock_irqsave(&rq->lock, flags);
+ rq_lock_irqsave(rq, &rf);
if (rq->rd) {
BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span));
set_rq_online(rq);
}
- raw_spin_unlock_irqrestore(&rq->lock, flags);
+ rq_unlock_irqrestore(rq, &rf);
update_max_interval();
@@ -5847,18 +5890,20 @@
int sched_cpu_dying(unsigned int cpu)
{
struct rq *rq = cpu_rq(cpu);
- unsigned long flags;
+ struct rq_flags rf;
/* Handle pending wakeups and then migrate everything off */
sched_ttwu_pending();
- raw_spin_lock_irqsave(&rq->lock, flags);
+
+ rq_lock_irqsave(rq, &rf);
if (rq->rd) {
BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span));
set_rq_offline(rq);
}
- migrate_tasks(rq);
+ migrate_tasks(rq, &rf);
BUG_ON(rq->nr_running != 1);
- raw_spin_unlock_irqrestore(&rq->lock, flags);
+ rq_unlock_irqrestore(rq, &rf);
+
calc_load_migrate(rq);
update_max_interval();
nohz_balance_exit_idle(cpu);
@@ -6412,7 +6457,8 @@
*/
void sched_move_task(struct task_struct *tsk)
{
- int queued, running;
+ int queued, running, queue_flags =
+ DEQUEUE_SAVE | DEQUEUE_MOVE | DEQUEUE_NOCLOCK;
struct rq_flags rf;
struct rq *rq;
@@ -6423,14 +6469,14 @@
queued = task_on_rq_queued(tsk);
if (queued)
- dequeue_task(rq, tsk, DEQUEUE_SAVE | DEQUEUE_MOVE);
+ dequeue_task(rq, tsk, queue_flags);
if (running)
put_prev_task(rq, tsk);
sched_change_group(tsk, TASK_MOVE_GROUP);
if (queued)
- enqueue_task(rq, tsk, ENQUEUE_RESTORE | ENQUEUE_MOVE);
+ enqueue_task(rq, tsk, queue_flags);
if (running)
set_curr_task(rq, tsk);
@@ -7008,14 +7054,15 @@
for_each_online_cpu(i) {
struct cfs_rq *cfs_rq = tg->cfs_rq[i];
struct rq *rq = cfs_rq->rq;
+ struct rq_flags rf;
- raw_spin_lock_irq(&rq->lock);
+ rq_lock_irq(rq, &rf);
cfs_rq->runtime_enabled = runtime_enabled;
cfs_rq->runtime_remaining = 0;
if (cfs_rq->throttled)
unthrottle_cfs_rq(cfs_rq);
- raw_spin_unlock_irq(&rq->lock);
+ rq_unlock_irq(rq, &rf);
}
if (runtime_was_enabled && !runtime_enabled)
cfs_bandwidth_usage_dec();
diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c
index 54c5775..76877a6 100644
--- a/kernel/sched/cpufreq_schedutil.c
+++ b/kernel/sched/cpufreq_schedutil.c
@@ -61,6 +61,11 @@
unsigned long util;
unsigned long max;
unsigned int flags;
+
+ /* The field below is for single-CPU policies only. */
+#ifdef CONFIG_NO_HZ_COMMON
+ unsigned long saved_idle_calls;
+#endif
};
static DEFINE_PER_CPU(struct sugov_cpu, sugov_cpu);
@@ -93,22 +98,23 @@
{
struct cpufreq_policy *policy = sg_policy->policy;
+ if (sg_policy->next_freq == next_freq)
+ return;
+
+ if (sg_policy->next_freq > next_freq)
+ next_freq = (sg_policy->next_freq + next_freq) >> 1;
+
+ sg_policy->next_freq = next_freq;
sg_policy->last_freq_update_time = time;
if (policy->fast_switch_enabled) {
- if (sg_policy->next_freq == next_freq) {
- trace_cpu_frequency(policy->cur, smp_processor_id());
- return;
- }
- sg_policy->next_freq = next_freq;
next_freq = cpufreq_driver_fast_switch(policy, next_freq);
if (next_freq == CPUFREQ_ENTRY_INVALID)
return;
policy->cur = next_freq;
trace_cpu_frequency(next_freq, smp_processor_id());
- } else if (sg_policy->next_freq != next_freq) {
- sg_policy->next_freq = next_freq;
+ } else {
sg_policy->work_in_progress = true;
irq_work_queue(&sg_policy->irq_work);
}
@@ -192,6 +198,19 @@
sg_cpu->iowait_boost >>= 1;
}
+#ifdef CONFIG_NO_HZ_COMMON
+static bool sugov_cpu_is_busy(struct sugov_cpu *sg_cpu)
+{
+ unsigned long idle_calls = tick_nohz_get_idle_calls();
+ bool ret = idle_calls == sg_cpu->saved_idle_calls;
+
+ sg_cpu->saved_idle_calls = idle_calls;
+ return ret;
+}
+#else
+static inline bool sugov_cpu_is_busy(struct sugov_cpu *sg_cpu) { return false; }
+#endif /* CONFIG_NO_HZ_COMMON */
+
static void sugov_update_single(struct update_util_data *hook, u64 time,
unsigned int flags)
{
@@ -200,6 +219,7 @@
struct cpufreq_policy *policy = sg_policy->policy;
unsigned long util, max;
unsigned int next_f;
+ bool busy;
sugov_set_iowait_boost(sg_cpu, time, flags);
sg_cpu->last_update = time;
@@ -207,40 +227,37 @@
if (!sugov_should_update_freq(sg_policy, time))
return;
+ busy = sugov_cpu_is_busy(sg_cpu);
+
if (flags & SCHED_CPUFREQ_RT_DL) {
next_f = policy->cpuinfo.max_freq;
} else {
sugov_get_util(&util, &max);
sugov_iowait_boost(sg_cpu, &util, &max);
next_f = get_next_freq(sg_policy, util, max);
+ /*
+ * Do not reduce the frequency if the CPU has not been idle
+ * recently, as the reduction is likely to be premature then.
+ */
+ if (busy && next_f < sg_policy->next_freq)
+ next_f = sg_policy->next_freq;
}
sugov_update_commit(sg_policy, time, next_f);
}
-static unsigned int sugov_next_freq_shared(struct sugov_cpu *sg_cpu,
- unsigned long util, unsigned long max,
- unsigned int flags)
+static unsigned int sugov_next_freq_shared(struct sugov_cpu *sg_cpu)
{
struct sugov_policy *sg_policy = sg_cpu->sg_policy;
struct cpufreq_policy *policy = sg_policy->policy;
- unsigned int max_f = policy->cpuinfo.max_freq;
u64 last_freq_update_time = sg_policy->last_freq_update_time;
+ unsigned long util = 0, max = 1;
unsigned int j;
- if (flags & SCHED_CPUFREQ_RT_DL)
- return max_f;
-
- sugov_iowait_boost(sg_cpu, &util, &max);
-
for_each_cpu(j, policy->cpus) {
- struct sugov_cpu *j_sg_cpu;
+ struct sugov_cpu *j_sg_cpu = &per_cpu(sugov_cpu, j);
unsigned long j_util, j_max;
s64 delta_ns;
- if (j == smp_processor_id())
- continue;
-
- j_sg_cpu = &per_cpu(sugov_cpu, j);
/*
* If the CPU utilization was last updated before the previous
* frequency update and the time elapsed between the last update
@@ -254,7 +271,7 @@
continue;
}
if (j_sg_cpu->flags & SCHED_CPUFREQ_RT_DL)
- return max_f;
+ return policy->cpuinfo.max_freq;
j_util = j_sg_cpu->util;
j_max = j_sg_cpu->max;
@@ -289,7 +306,11 @@
sg_cpu->last_update = time;
if (sugov_should_update_freq(sg_policy, time)) {
- next_f = sugov_next_freq_shared(sg_cpu, util, max, flags);
+ if (flags & SCHED_CPUFREQ_RT_DL)
+ next_f = sg_policy->policy->cpuinfo.max_freq;
+ else
+ next_f = sugov_next_freq_shared(sg_cpu);
+
sugov_update_commit(sg_policy, time, next_f);
}
@@ -473,7 +494,6 @@
{
struct sugov_policy *sg_policy;
struct sugov_tunables *tunables;
- unsigned int lat;
int ret = 0;
/* State should be equivalent to EXIT */
@@ -512,10 +532,16 @@
goto stop_kthread;
}
- tunables->rate_limit_us = LATENCY_MULTIPLIER;
- lat = policy->cpuinfo.transition_latency / NSEC_PER_USEC;
- if (lat)
- tunables->rate_limit_us *= lat;
+ if (policy->transition_delay_us) {
+ tunables->rate_limit_us = policy->transition_delay_us;
+ } else {
+ unsigned int lat;
+
+ tunables->rate_limit_us = LATENCY_MULTIPLIER;
+ lat = policy->cpuinfo.transition_latency / NSEC_PER_USEC;
+ if (lat)
+ tunables->rate_limit_us *= lat;
+ }
policy->governor_data = sg_policy;
sg_policy->tunables = tunables;
diff --git a/kernel/sched/cputime.c b/kernel/sched/cputime.c
index f3778e2b..aea3135 100644
--- a/kernel/sched/cputime.c
+++ b/kernel/sched/cputime.c
@@ -34,6 +34,18 @@
sched_clock_irqtime = 0;
}
+static void irqtime_account_delta(struct irqtime *irqtime, u64 delta,
+ enum cpu_usage_stat idx)
+{
+ u64 *cpustat = kcpustat_this_cpu->cpustat;
+
+ u64_stats_update_begin(&irqtime->sync);
+ cpustat[idx] += delta;
+ irqtime->total += delta;
+ irqtime->tick_delta += delta;
+ u64_stats_update_end(&irqtime->sync);
+}
+
/*
* Called before incrementing preempt_count on {soft,}irq_enter
* and before decrementing preempt_count on {soft,}irq_exit.
@@ -41,7 +53,6 @@
void irqtime_account_irq(struct task_struct *curr)
{
struct irqtime *irqtime = this_cpu_ptr(&cpu_irqtime);
- u64 *cpustat = kcpustat_this_cpu->cpustat;
s64 delta;
int cpu;
@@ -52,22 +63,16 @@
delta = sched_clock_cpu(cpu) - irqtime->irq_start_time;
irqtime->irq_start_time += delta;
- u64_stats_update_begin(&irqtime->sync);
/*
* We do not account for softirq time from ksoftirqd here.
* We want to continue accounting softirq time to ksoftirqd thread
* in that case, so as not to confuse scheduler with a special task
* that do not consume any time, but still wants to run.
*/
- if (hardirq_count()) {
- cpustat[CPUTIME_IRQ] += delta;
- irqtime->tick_delta += delta;
- } else if (in_serving_softirq() && curr != this_cpu_ksoftirqd()) {
- cpustat[CPUTIME_SOFTIRQ] += delta;
- irqtime->tick_delta += delta;
- }
-
- u64_stats_update_end(&irqtime->sync);
+ if (hardirq_count())
+ irqtime_account_delta(irqtime, delta, CPUTIME_IRQ);
+ else if (in_serving_softirq() && curr != this_cpu_ksoftirqd())
+ irqtime_account_delta(irqtime, delta, CPUTIME_SOFTIRQ);
}
EXPORT_SYMBOL_GPL(irqtime_account_irq);
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index dea1389..d711093 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -717,18 +717,12 @@
}
#ifdef CONFIG_SMP
+
+#include "sched-pelt.h"
+
static int select_idle_sibling(struct task_struct *p, int prev_cpu, int cpu);
static unsigned long task_h_load(struct task_struct *p);
-/*
- * We choose a half-life close to 1 scheduling period.
- * Note: The tables runnable_avg_yN_inv and runnable_avg_yN_sum are
- * dependent on this value.
- */
-#define LOAD_AVG_PERIOD 32
-#define LOAD_AVG_MAX 47742 /* maximum possible load avg */
-#define LOAD_AVG_MAX_N 345 /* number of full periods to produce LOAD_AVG_MAX */
-
/* Give new sched_entity start runnable values to heavy its load in infant time */
void init_entity_runnable_average(struct sched_entity *se)
{
@@ -2733,47 +2727,15 @@
#endif /* CONFIG_FAIR_GROUP_SCHED */
#ifdef CONFIG_SMP
-/* Precomputed fixed inverse multiplies for multiplication by y^n */
-static const u32 runnable_avg_yN_inv[] = {
- 0xffffffff, 0xfa83b2da, 0xf5257d14, 0xefe4b99a, 0xeac0c6e6, 0xe5b906e6,
- 0xe0ccdeeb, 0xdbfbb796, 0xd744fcc9, 0xd2a81d91, 0xce248c14, 0xc9b9bd85,
- 0xc5672a10, 0xc12c4cc9, 0xbd08a39e, 0xb8fbaf46, 0xb504f333, 0xb123f581,
- 0xad583ee9, 0xa9a15ab4, 0xa5fed6a9, 0xa2704302, 0x9ef5325f, 0x9b8d39b9,
- 0x9837f050, 0x94f4efa8, 0x91c3d373, 0x8ea4398a, 0x8b95c1e3, 0x88980e80,
- 0x85aac367, 0x82cd8698,
-};
-
-/*
- * Precomputed \Sum y^k { 1<=k<=n }. These are floor(true_value) to prevent
- * over-estimates when re-combining.
- */
-static const u32 runnable_avg_yN_sum[] = {
- 0, 1002, 1982, 2941, 3880, 4798, 5697, 6576, 7437, 8279, 9103,
- 9909,10698,11470,12226,12966,13690,14398,15091,15769,16433,17082,
- 17718,18340,18949,19545,20128,20698,21256,21802,22336,22859,23371,
-};
-
-/*
- * Precomputed \Sum y^k { 1<=k<=n, where n%32=0). Values are rolled down to
- * lower integers. See Documentation/scheduler/sched-avg.txt how these
- * were generated:
- */
-static const u32 __accumulated_sum_N32[] = {
- 0, 23371, 35056, 40899, 43820, 45281,
- 46011, 46376, 46559, 46650, 46696, 46719,
-};
-
/*
* Approximate:
* val * y^n, where y^32 ~= 0.5 (~1 scheduling period)
*/
-static __always_inline u64 decay_load(u64 val, u64 n)
+static u64 decay_load(u64 val, u64 n)
{
unsigned int local_n;
- if (!n)
- return val;
- else if (unlikely(n > LOAD_AVG_PERIOD * 63))
+ if (unlikely(n > LOAD_AVG_PERIOD * 63))
return 0;
/* after bounds checking we can collapse to 32-bit */
@@ -2795,32 +2757,99 @@
return val;
}
-/*
- * For updates fully spanning n periods, the contribution to runnable
- * average will be: \Sum 1024*y^n
- *
- * We can compute this reasonably efficiently by combining:
- * y^PERIOD = 1/2 with precomputed \Sum 1024*y^n {for n <PERIOD}
- */
-static u32 __compute_runnable_contrib(u64 n)
+static u32 __accumulate_pelt_segments(u64 periods, u32 d1, u32 d3)
{
- u32 contrib = 0;
+ u32 c1, c2, c3 = d3; /* y^0 == 1 */
- if (likely(n <= LOAD_AVG_PERIOD))
- return runnable_avg_yN_sum[n];
- else if (unlikely(n >= LOAD_AVG_MAX_N))
- return LOAD_AVG_MAX;
+ /*
+ * c1 = d1 y^p
+ */
+ c1 = decay_load((u64)d1, periods);
- /* Since n < LOAD_AVG_MAX_N, n/LOAD_AVG_PERIOD < 11 */
- contrib = __accumulated_sum_N32[n/LOAD_AVG_PERIOD];
- n %= LOAD_AVG_PERIOD;
- contrib = decay_load(contrib, n);
- return contrib + runnable_avg_yN_sum[n];
+ /*
+ * p-1
+ * c2 = 1024 \Sum y^n
+ * n=1
+ *
+ * inf inf
+ * = 1024 ( \Sum y^n - \Sum y^n - y^0 )
+ * n=0 n=p
+ */
+ c2 = LOAD_AVG_MAX - decay_load(LOAD_AVG_MAX, periods) - 1024;
+
+ return c1 + c2 + c3;
}
#define cap_scale(v, s) ((v)*(s) >> SCHED_CAPACITY_SHIFT)
/*
+ * Accumulate the three separate parts of the sum; d1 the remainder
+ * of the last (incomplete) period, d2 the span of full periods and d3
+ * the remainder of the (incomplete) current period.
+ *
+ * d1 d2 d3
+ * ^ ^ ^
+ * | | |
+ * |<->|<----------------->|<--->|
+ * ... |---x---|------| ... |------|-----x (now)
+ *
+ * p-1
+ * u' = (u + d1) y^p + 1024 \Sum y^n + d3 y^0
+ * n=1
+ *
+ * = u y^p + (Step 1)
+ *
+ * p-1
+ * d1 y^p + 1024 \Sum y^n + d3 y^0 (Step 2)
+ * n=1
+ */
+static __always_inline u32
+accumulate_sum(u64 delta, int cpu, struct sched_avg *sa,
+ unsigned long weight, int running, struct cfs_rq *cfs_rq)
+{
+ unsigned long scale_freq, scale_cpu;
+ u32 contrib = (u32)delta; /* p == 0 -> delta < 1024 */
+ u64 periods;
+
+ scale_freq = arch_scale_freq_capacity(NULL, cpu);
+ scale_cpu = arch_scale_cpu_capacity(NULL, cpu);
+
+ delta += sa->period_contrib;
+ periods = delta / 1024; /* A period is 1024us (~1ms) */
+
+ /*
+ * Step 1: decay old *_sum if we crossed period boundaries.
+ */
+ if (periods) {
+ sa->load_sum = decay_load(sa->load_sum, periods);
+ if (cfs_rq) {
+ cfs_rq->runnable_load_sum =
+ decay_load(cfs_rq->runnable_load_sum, periods);
+ }
+ sa->util_sum = decay_load((u64)(sa->util_sum), periods);
+
+ /*
+ * Step 2
+ */
+ delta %= 1024;
+ contrib = __accumulate_pelt_segments(periods,
+ 1024 - sa->period_contrib, delta);
+ }
+ sa->period_contrib = delta;
+
+ contrib = cap_scale(contrib, scale_freq);
+ if (weight) {
+ sa->load_sum += weight * contrib;
+ if (cfs_rq)
+ cfs_rq->runnable_load_sum += weight * contrib;
+ }
+ if (running)
+ sa->util_sum += contrib * scale_cpu;
+
+ return periods;
+}
+
+/*
* We can represent the historical contribution to runnable average as the
* coefficients of a geometric series. To do this we sub-divide our runnable
* history into segments of approximately 1ms (1024us); label the segment that
@@ -2849,13 +2878,10 @@
* = u_0 + u_1*y + u_2*y^2 + ... [re-labeling u_i --> u_{i+1}]
*/
static __always_inline int
-__update_load_avg(u64 now, int cpu, struct sched_avg *sa,
+___update_load_avg(u64 now, int cpu, struct sched_avg *sa,
unsigned long weight, int running, struct cfs_rq *cfs_rq)
{
- u64 delta, scaled_delta, periods;
- u32 contrib;
- unsigned int delta_w, scaled_delta_w, decayed = 0;
- unsigned long scale_freq, scale_cpu;
+ u64 delta;
delta = now - sa->last_update_time;
/*
@@ -2874,83 +2900,52 @@
delta >>= 10;
if (!delta)
return 0;
- sa->last_update_time = now;
- scale_freq = arch_scale_freq_capacity(NULL, cpu);
- scale_cpu = arch_scale_cpu_capacity(NULL, cpu);
+ sa->last_update_time += delta << 10;
- /* delta_w is the amount already accumulated against our next period */
- delta_w = sa->period_contrib;
- if (delta + delta_w >= 1024) {
- decayed = 1;
+ /*
+ * Now we know we crossed measurement unit boundaries. The *_avg
+ * accrues by two steps:
+ *
+ * Step 1: accumulate *_sum since last_update_time. If we haven't
+ * crossed period boundaries, finish.
+ */
+ if (!accumulate_sum(delta, cpu, sa, weight, running, cfs_rq))
+ return 0;
- /* how much left for next period will start over, we don't know yet */
- sa->period_contrib = 0;
-
- /*
- * Now that we know we're crossing a period boundary, figure
- * out how much from delta we need to complete the current
- * period and accrue it.
- */
- delta_w = 1024 - delta_w;
- scaled_delta_w = cap_scale(delta_w, scale_freq);
- if (weight) {
- sa->load_sum += weight * scaled_delta_w;
- if (cfs_rq) {
- cfs_rq->runnable_load_sum +=
- weight * scaled_delta_w;
- }
- }
- if (running)
- sa->util_sum += scaled_delta_w * scale_cpu;
-
- delta -= delta_w;
-
- /* Figure out how many additional periods this update spans */
- periods = delta / 1024;
- delta %= 1024;
-
- sa->load_sum = decay_load(sa->load_sum, periods + 1);
- if (cfs_rq) {
- cfs_rq->runnable_load_sum =
- decay_load(cfs_rq->runnable_load_sum, periods + 1);
- }
- sa->util_sum = decay_load((u64)(sa->util_sum), periods + 1);
-
- /* Efficiently calculate \sum (1..n_period) 1024*y^i */
- contrib = __compute_runnable_contrib(periods);
- contrib = cap_scale(contrib, scale_freq);
- if (weight) {
- sa->load_sum += weight * contrib;
- if (cfs_rq)
- cfs_rq->runnable_load_sum += weight * contrib;
- }
- if (running)
- sa->util_sum += contrib * scale_cpu;
+ /*
+ * Step 2: update *_avg.
+ */
+ sa->load_avg = div_u64(sa->load_sum, LOAD_AVG_MAX);
+ if (cfs_rq) {
+ cfs_rq->runnable_load_avg =
+ div_u64(cfs_rq->runnable_load_sum, LOAD_AVG_MAX);
}
+ sa->util_avg = sa->util_sum / LOAD_AVG_MAX;
- /* Remainder of delta accrued against u_0` */
- scaled_delta = cap_scale(delta, scale_freq);
- if (weight) {
- sa->load_sum += weight * scaled_delta;
- if (cfs_rq)
- cfs_rq->runnable_load_sum += weight * scaled_delta;
- }
- if (running)
- sa->util_sum += scaled_delta * scale_cpu;
+ return 1;
+}
- sa->period_contrib += delta;
+static int
+__update_load_avg_blocked_se(u64 now, int cpu, struct sched_entity *se)
+{
+ return ___update_load_avg(now, cpu, &se->avg, 0, 0, NULL);
+}
- if (decayed) {
- sa->load_avg = div_u64(sa->load_sum, LOAD_AVG_MAX);
- if (cfs_rq) {
- cfs_rq->runnable_load_avg =
- div_u64(cfs_rq->runnable_load_sum, LOAD_AVG_MAX);
- }
- sa->util_avg = sa->util_sum / LOAD_AVG_MAX;
- }
+static int
+__update_load_avg_se(u64 now, int cpu, struct cfs_rq *cfs_rq, struct sched_entity *se)
+{
+ return ___update_load_avg(now, cpu, &se->avg,
+ se->on_rq * scale_load_down(se->load.weight),
+ cfs_rq->curr == se, NULL);
+}
- return decayed;
+static int
+__update_load_avg_cfs_rq(u64 now, int cpu, struct cfs_rq *cfs_rq)
+{
+ return ___update_load_avg(now, cpu, &cfs_rq->avg,
+ scale_load_down(cfs_rq->load.weight),
+ cfs_rq->curr != NULL, cfs_rq);
}
/*
@@ -3014,6 +3009,9 @@
void set_task_rq_fair(struct sched_entity *se,
struct cfs_rq *prev, struct cfs_rq *next)
{
+ u64 p_last_update_time;
+ u64 n_last_update_time;
+
if (!sched_feat(ATTACH_AGE_LOAD))
return;
@@ -3024,11 +3022,11 @@
* time. This will result in the wakee task is less decayed, but giving
* the wakee more load sounds not bad.
*/
- if (se->avg.last_update_time && prev) {
- u64 p_last_update_time;
- u64 n_last_update_time;
+ if (!(se->avg.last_update_time && prev))
+ return;
#ifndef CONFIG_64BIT
+ {
u64 p_last_update_time_copy;
u64 n_last_update_time_copy;
@@ -3043,14 +3041,13 @@
} while (p_last_update_time != p_last_update_time_copy ||
n_last_update_time != n_last_update_time_copy);
-#else
- p_last_update_time = prev->avg.last_update_time;
- n_last_update_time = next->avg.last_update_time;
-#endif
- __update_load_avg(p_last_update_time, cpu_of(rq_of(prev)),
- &se->avg, 0, 0, NULL);
- se->avg.last_update_time = n_last_update_time;
}
+#else
+ p_last_update_time = prev->avg.last_update_time;
+ n_last_update_time = next->avg.last_update_time;
+#endif
+ __update_load_avg_blocked_se(p_last_update_time, cpu_of(rq_of(prev)), se);
+ se->avg.last_update_time = n_last_update_time;
}
/* Take into account change of utilization of a child task group */
@@ -3173,6 +3170,36 @@
return 1;
}
+/*
+ * Check if we need to update the load and the utilization of a blocked
+ * group_entity:
+ */
+static inline bool skip_blocked_update(struct sched_entity *se)
+{
+ struct cfs_rq *gcfs_rq = group_cfs_rq(se);
+
+ /*
+ * If sched_entity still have not zero load or utilization, we have to
+ * decay it:
+ */
+ if (se->avg.load_avg || se->avg.util_avg)
+ return false;
+
+ /*
+ * If there is a pending propagation, we have to update the load and
+ * the utilization of the sched_entity:
+ */
+ if (gcfs_rq->propagate_avg)
+ return false;
+
+ /*
+ * Otherwise, the load and the utilization of the sched_entity is
+ * already zero and there is no pending propagation, so it will be a
+ * waste of time to try to decay it:
+ */
+ return true;
+}
+
#else /* CONFIG_FAIR_GROUP_SCHED */
static inline void update_tg_load_avg(struct cfs_rq *cfs_rq, int force) {}
@@ -3265,8 +3292,7 @@
set_tg_cfs_propagate(cfs_rq);
}
- decayed = __update_load_avg(now, cpu_of(rq_of(cfs_rq)), sa,
- scale_load_down(cfs_rq->load.weight), cfs_rq->curr != NULL, cfs_rq);
+ decayed = __update_load_avg_cfs_rq(now, cpu_of(rq_of(cfs_rq)), cfs_rq);
#ifndef CONFIG_64BIT
smp_wmb();
@@ -3298,11 +3324,8 @@
* Track task load average for carrying it to new CPU after migrated, and
* track group sched_entity load average for task_h_load calc in migration
*/
- if (se->avg.last_update_time && !(flags & SKIP_AGE_LOAD)) {
- __update_load_avg(now, cpu, &se->avg,
- se->on_rq * scale_load_down(se->load.weight),
- cfs_rq->curr == se, NULL);
- }
+ if (se->avg.last_update_time && !(flags & SKIP_AGE_LOAD))
+ __update_load_avg_se(now, cpu, cfs_rq, se);
decayed = update_cfs_rq_load_avg(now, cfs_rq, true);
decayed |= propagate_entity_load_avg(se);
@@ -3407,7 +3430,7 @@
u64 last_update_time;
last_update_time = cfs_rq_last_update_time(cfs_rq);
- __update_load_avg(last_update_time, cpu_of(rq_of(cfs_rq)), &se->avg, 0, 0, NULL);
+ __update_load_avg_blocked_se(last_update_time, cpu_of(rq_of(cfs_rq)), se);
}
/*
@@ -4271,8 +4294,9 @@
list_for_each_entry_rcu(cfs_rq, &cfs_b->throttled_cfs_rq,
throttled_list) {
struct rq *rq = rq_of(cfs_rq);
+ struct rq_flags rf;
- raw_spin_lock(&rq->lock);
+ rq_lock(rq, &rf);
if (!cfs_rq_throttled(cfs_rq))
goto next;
@@ -4289,7 +4313,7 @@
unthrottle_cfs_rq(cfs_rq);
next:
- raw_spin_unlock(&rq->lock);
+ rq_unlock(rq, &rf);
if (!remaining)
break;
@@ -5097,15 +5121,16 @@
unsigned long curr_jiffies = READ_ONCE(jiffies);
struct rq *this_rq = this_rq();
unsigned long load;
+ struct rq_flags rf;
if (curr_jiffies == this_rq->last_load_update_tick)
return;
load = weighted_cpuload(cpu_of(this_rq));
- raw_spin_lock(&this_rq->lock);
+ rq_lock(this_rq, &rf);
update_rq_clock(this_rq);
cpu_load_update_nohz(this_rq, curr_jiffies, load);
- raw_spin_unlock(&this_rq->lock);
+ rq_unlock(this_rq, &rf);
}
#else /* !CONFIG_NO_HZ_COMMON */
static inline void cpu_load_update_nohz(struct rq *this_rq,
@@ -6769,7 +6794,7 @@
lockdep_assert_held(&env->src_rq->lock);
p->on_rq = TASK_ON_RQ_MIGRATING;
- deactivate_task(env->src_rq, p, 0);
+ deactivate_task(env->src_rq, p, DEQUEUE_NOCLOCK);
set_task_cpu(p, env->dst_cpu);
}
@@ -6902,7 +6927,7 @@
lockdep_assert_held(&rq->lock);
BUG_ON(task_rq(p) != rq);
- activate_task(rq, p, 0);
+ activate_task(rq, p, ENQUEUE_NOCLOCK);
p->on_rq = TASK_ON_RQ_QUEUED;
check_preempt_curr(rq, p, 0);
}
@@ -6913,9 +6938,12 @@
*/
static void attach_one_task(struct rq *rq, struct task_struct *p)
{
- raw_spin_lock(&rq->lock);
+ struct rq_flags rf;
+
+ rq_lock(rq, &rf);
+ update_rq_clock(rq);
attach_task(rq, p);
- raw_spin_unlock(&rq->lock);
+ rq_unlock(rq, &rf);
}
/*
@@ -6926,8 +6954,10 @@
{
struct list_head *tasks = &env->tasks;
struct task_struct *p;
+ struct rq_flags rf;
- raw_spin_lock(&env->dst_rq->lock);
+ rq_lock(env->dst_rq, &rf);
+ update_rq_clock(env->dst_rq);
while (!list_empty(tasks)) {
p = list_first_entry(tasks, struct task_struct, se.group_node);
@@ -6936,7 +6966,7 @@
attach_task(env->dst_rq, p);
}
- raw_spin_unlock(&env->dst_rq->lock);
+ rq_unlock(env->dst_rq, &rf);
}
#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -6944,9 +6974,9 @@
{
struct rq *rq = cpu_rq(cpu);
struct cfs_rq *cfs_rq;
- unsigned long flags;
+ struct rq_flags rf;
- raw_spin_lock_irqsave(&rq->lock, flags);
+ rq_lock_irqsave(rq, &rf);
update_rq_clock(rq);
/*
@@ -6954,6 +6984,8 @@
* list_add_leaf_cfs_rq() for details.
*/
for_each_leaf_cfs_rq(rq, cfs_rq) {
+ struct sched_entity *se;
+
/* throttled entities do not contribute to load */
if (throttled_hierarchy(cfs_rq))
continue;
@@ -6961,11 +6993,12 @@
if (update_cfs_rq_load_avg(cfs_rq_clock_task(cfs_rq), cfs_rq, true))
update_tg_load_avg(cfs_rq, 0);
- /* Propagate pending load changes to the parent */
- if (cfs_rq->tg->se[cpu])
- update_load_avg(cfs_rq->tg->se[cpu], 0);
+ /* Propagate pending load changes to the parent, if any: */
+ se = cfs_rq->tg->se[cpu];
+ if (se && !skip_blocked_update(se))
+ update_load_avg(se, 0);
}
- raw_spin_unlock_irqrestore(&rq->lock, flags);
+ rq_unlock_irqrestore(rq, &rf);
}
/*
@@ -7019,12 +7052,12 @@
{
struct rq *rq = cpu_rq(cpu);
struct cfs_rq *cfs_rq = &rq->cfs;
- unsigned long flags;
+ struct rq_flags rf;
- raw_spin_lock_irqsave(&rq->lock, flags);
+ rq_lock_irqsave(rq, &rf);
update_rq_clock(rq);
update_cfs_rq_load_avg(cfs_rq_clock_task(cfs_rq), cfs_rq, true);
- raw_spin_unlock_irqrestore(&rq->lock, flags);
+ rq_unlock_irqrestore(rq, &rf);
}
static unsigned long task_h_load(struct task_struct *p)
@@ -7525,6 +7558,7 @@
{
struct sched_domain *child = env->sd->child;
struct sched_group *sg = env->sd->groups;
+ struct sg_lb_stats *local = &sds->local_stat;
struct sg_lb_stats tmp_sgs;
int load_idx, prefer_sibling = 0;
bool overload = false;
@@ -7541,7 +7575,7 @@
local_group = cpumask_test_cpu(env->dst_cpu, sched_group_cpus(sg));
if (local_group) {
sds->local = sg;
- sgs = &sds->local_stat;
+ sgs = local;
if (env->idle != CPU_NEWLY_IDLE ||
time_after_eq(jiffies, sg->sgc->next_update))
@@ -7565,8 +7599,8 @@
* the tasks on the system).
*/
if (prefer_sibling && sds->local &&
- group_has_capacity(env, &sds->local_stat) &&
- (sgs->sum_nr_running > 1)) {
+ group_has_capacity(env, local) &&
+ (sgs->sum_nr_running > local->sum_nr_running + 1)) {
sgs->group_no_capacity = 1;
sgs->group_type = group_classify(sg, sgs);
}
@@ -7597,7 +7631,7 @@
/**
* check_asym_packing - Check to see if the group is packed into the
- * sched doman.
+ * sched domain.
*
* This is primarily intended to used at the sibling level. Some
* cores like POWER7 prefer to use lower numbered SMT threads. In the
@@ -8042,7 +8076,7 @@
struct sched_domain *sd_parent = sd->parent;
struct sched_group *group;
struct rq *busiest;
- unsigned long flags;
+ struct rq_flags rf;
struct cpumask *cpus = this_cpu_cpumask_var_ptr(load_balance_mask);
struct lb_env env = {
@@ -8105,7 +8139,7 @@
env.loop_max = min(sysctl_sched_nr_migrate, busiest->nr_running);
more_balance:
- raw_spin_lock_irqsave(&busiest->lock, flags);
+ rq_lock_irqsave(busiest, &rf);
update_rq_clock(busiest);
/*
@@ -8122,14 +8156,14 @@
* See task_rq_lock() family for the details.
*/
- raw_spin_unlock(&busiest->lock);
+ rq_unlock(busiest, &rf);
if (cur_ld_moved) {
attach_tasks(&env);
ld_moved += cur_ld_moved;
}
- local_irq_restore(flags);
+ local_irq_restore(rf.flags);
if (env.flags & LBF_NEED_BREAK) {
env.flags &= ~LBF_NEED_BREAK;
@@ -8207,6 +8241,8 @@
sd->nr_balance_failed++;
if (need_active_balance(&env)) {
+ unsigned long flags;
+
raw_spin_lock_irqsave(&busiest->lock, flags);
/* don't kick the active_load_balance_cpu_stop,
@@ -8444,8 +8480,9 @@
struct rq *target_rq = cpu_rq(target_cpu);
struct sched_domain *sd;
struct task_struct *p = NULL;
+ struct rq_flags rf;
- raw_spin_lock_irq(&busiest_rq->lock);
+ rq_lock_irq(busiest_rq, &rf);
/* make sure the requested cpu hasn't gone down in the meantime */
if (unlikely(busiest_cpu != smp_processor_id() ||
@@ -8496,7 +8533,7 @@
rcu_read_unlock();
out_unlock:
busiest_rq->active_balance = 0;
- raw_spin_unlock(&busiest_rq->lock);
+ rq_unlock(busiest_rq, &rf);
if (p)
attach_one_task(target_rq, p);
@@ -8794,10 +8831,13 @@
* do the balance.
*/
if (time_after_eq(jiffies, rq->next_balance)) {
- raw_spin_lock_irq(&rq->lock);
+ struct rq_flags rf;
+
+ rq_lock_irq(rq, &rf);
update_rq_clock(rq);
cpu_load_update_idle(rq);
- raw_spin_unlock_irq(&rq->lock);
+ rq_unlock_irq(rq, &rf);
+
rebalance_domains(rq, CPU_IDLE);
}
@@ -8988,8 +9028,9 @@
struct cfs_rq *cfs_rq;
struct sched_entity *se = &p->se, *curr;
struct rq *rq = this_rq();
+ struct rq_flags rf;
- raw_spin_lock(&rq->lock);
+ rq_lock(rq, &rf);
update_rq_clock(rq);
cfs_rq = task_cfs_rq(current);
@@ -9010,7 +9051,7 @@
}
se->vruntime -= cfs_rq->min_vruntime;
- raw_spin_unlock(&rq->lock);
+ rq_unlock(rq, &rf);
}
/*
@@ -9372,7 +9413,6 @@
int sched_group_set_shares(struct task_group *tg, unsigned long shares)
{
int i;
- unsigned long flags;
/*
* We can't change the weight of the root cgroup.
@@ -9389,19 +9429,17 @@
tg->shares = shares;
for_each_possible_cpu(i) {
struct rq *rq = cpu_rq(i);
- struct sched_entity *se;
+ struct sched_entity *se = tg->se[i];
+ struct rq_flags rf;
- se = tg->se[i];
/* Propagate contribution to hierarchy */
- raw_spin_lock_irqsave(&rq->lock, flags);
-
- /* Possible calls to update_curr() need rq clock */
+ rq_lock_irqsave(rq, &rf);
update_rq_clock(rq);
for_each_sched_entity(se) {
update_load_avg(se, UPDATE_TG);
update_cfs_shares(se);
}
- raw_spin_unlock_irqrestore(&rq->lock, flags);
+ rq_unlock_irqrestore(rq, &rf);
}
done:
diff --git a/kernel/sched/features.h b/kernel/sched/features.h
index 1b3c818..11192e0 100644
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -56,6 +56,13 @@
*/
SCHED_FEAT(SIS_AVG_CPU, false)
+/*
+ * Issue a WARN when we do multiple update_rq_clock() calls
+ * in a single rq->lock section. Default disabled because the
+ * annotations are not complete.
+ */
+SCHED_FEAT(WARN_DOUBLE_CLOCK, false)
+
#ifdef HAVE_RT_PUSH_IPI
/*
* In order to avoid a thundering herd attack of CPUs that are
diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index ac6d517..2a25a9e 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -10,6 +10,7 @@
#include <linux/mm.h>
#include <linux/stackprotector.h>
#include <linux/suspend.h>
+#include <linux/livepatch.h>
#include <asm/tlb.h>
@@ -265,6 +266,9 @@
sched_ttwu_pending();
schedule_preempt_disabled();
+
+ if (unlikely(klp_patch_pending(current)))
+ klp_update_patch_state(current);
}
bool cpu_in_idle(unsigned long pc)
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 9f3e402..979b734 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -1927,6 +1927,87 @@
#define RT_PUSH_IPI_EXECUTING 1
#define RT_PUSH_IPI_RESTART 2
+/*
+ * When a high priority task schedules out from a CPU and a lower priority
+ * task is scheduled in, a check is made to see if there's any RT tasks
+ * on other CPUs that are waiting to run because a higher priority RT task
+ * is currently running on its CPU. In this case, the CPU with multiple RT
+ * tasks queued on it (overloaded) needs to be notified that a CPU has opened
+ * up that may be able to run one of its non-running queued RT tasks.
+ *
+ * On large CPU boxes, there's the case that several CPUs could schedule
+ * a lower priority task at the same time, in which case it will look for
+ * any overloaded CPUs that it could pull a task from. To do this, the runqueue
+ * lock must be taken from that overloaded CPU. Having 10s of CPUs all fighting
+ * for a single overloaded CPU's runqueue lock can produce a large latency.
+ * (This has actually been observed on large boxes running cyclictest).
+ * Instead of taking the runqueue lock of the overloaded CPU, each of the
+ * CPUs that scheduled a lower priority task simply sends an IPI to the
+ * overloaded CPU. An IPI is much cheaper than taking an runqueue lock with
+ * lots of contention. The overloaded CPU will look to push its non-running
+ * RT task off, and if it does, it can then ignore the other IPIs coming
+ * in, and just pass those IPIs off to any other overloaded CPU.
+ *
+ * When a CPU schedules a lower priority task, it only sends an IPI to
+ * the "next" CPU that has overloaded RT tasks. This prevents IPI storms,
+ * as having 10 CPUs scheduling lower priority tasks and 10 CPUs with
+ * RT overloaded tasks, would cause 100 IPIs to go out at once.
+ *
+ * The overloaded RT CPU, when receiving an IPI, will try to push off its
+ * overloaded RT tasks and then send an IPI to the next CPU that has
+ * overloaded RT tasks. This stops when all CPUs with overloaded RT tasks
+ * have completed. Just because a CPU may have pushed off its own overloaded
+ * RT task does not mean it should stop sending the IPI around to other
+ * overloaded CPUs. There may be another RT task waiting to run on one of
+ * those CPUs that are of higher priority than the one that was just
+ * pushed.
+ *
+ * An optimization that could possibly be made is to make a CPU array similar
+ * to the cpupri array mask of all running RT tasks, but for the overloaded
+ * case, then the IPI could be sent to only the CPU with the highest priority
+ * RT task waiting, and that CPU could send off further IPIs to the CPU with
+ * the next highest waiting task. Since the overloaded case is much less likely
+ * to happen, the complexity of this implementation may not be worth it.
+ * Instead, just send an IPI around to all overloaded CPUs.
+ *
+ * The rq->rt.push_flags holds the status of the IPI that is going around.
+ * A run queue can only send out a single IPI at a time. The possible flags
+ * for rq->rt.push_flags are:
+ *
+ * (None or zero): No IPI is going around for the current rq
+ * RT_PUSH_IPI_EXECUTING: An IPI for the rq is being passed around
+ * RT_PUSH_IPI_RESTART: The priority of the running task for the rq
+ * has changed, and the IPI should restart
+ * circulating the overloaded CPUs again.
+ *
+ * rq->rt.push_cpu contains the CPU that is being sent the IPI. It is updated
+ * before sending to the next CPU.
+ *
+ * Instead of having all CPUs that schedule a lower priority task send
+ * an IPI to the same "first" CPU in the RT overload mask, they send it
+ * to the next overloaded CPU after their own CPU. This helps distribute
+ * the work when there's more than one overloaded CPU and multiple CPUs
+ * scheduling in lower priority tasks.
+ *
+ * When a rq schedules a lower priority task than what was currently
+ * running, the next CPU with overloaded RT tasks is examined first.
+ * That is, if CPU 1 and 5 are overloaded, and CPU 3 schedules a lower
+ * priority task, it will send an IPI first to CPU 5, then CPU 5 will
+ * send to CPU 1 if it is still overloaded. CPU 1 will clear the
+ * rq->rt.push_flags if RT_PUSH_IPI_RESTART is not set.
+ *
+ * The first CPU to notice IPI_RESTART is set, will clear that flag and then
+ * send an IPI to the next overloaded CPU after the rq->cpu and not the next
+ * CPU after push_cpu. That is, if CPU 1, 4 and 5 are overloaded when CPU 3
+ * schedules a lower priority task, and the IPI_RESTART gets set while the
+ * handling is being done on CPU 5, it will clear the flag and send it back to
+ * CPU 4 instead of CPU 1.
+ *
+ * Note, the above logic can be disabled by turning off the sched_feature
+ * RT_PUSH_IPI. Then the rq lock of the overloaded CPU will simply be
+ * taken by the CPU requesting a pull and the waiting RT task will be pulled
+ * by that CPU. This may be fine for machines with few CPUs.
+ */
static void tell_cpu_to_push(struct rq *rq)
{
int cpu;
diff --git a/kernel/sched/sched-pelt.h b/kernel/sched/sched-pelt.h
new file mode 100644
index 0000000..cd200d1
--- /dev/null
+++ b/kernel/sched/sched-pelt.h
@@ -0,0 +1,13 @@
+/* Generated by Documentation/scheduler/sched-pelt; do not modify. */
+
+static const u32 runnable_avg_yN_inv[] = {
+ 0xffffffff, 0xfa83b2da, 0xf5257d14, 0xefe4b99a, 0xeac0c6e6, 0xe5b906e6,
+ 0xe0ccdeeb, 0xdbfbb796, 0xd744fcc9, 0xd2a81d91, 0xce248c14, 0xc9b9bd85,
+ 0xc5672a10, 0xc12c4cc9, 0xbd08a39e, 0xb8fbaf46, 0xb504f333, 0xb123f581,
+ 0xad583ee9, 0xa9a15ab4, 0xa5fed6a9, 0xa2704302, 0x9ef5325f, 0x9b8d39b9,
+ 0x9837f050, 0x94f4efa8, 0x91c3d373, 0x8ea4398a, 0x8b95c1e3, 0x88980e80,
+ 0x85aac367, 0x82cd8698,
+};
+
+#define LOAD_AVG_PERIOD 32
+#define LOAD_AVG_MAX 47742
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 5cbf922..7808ab0 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1331,15 +1331,17 @@
#define DEQUEUE_SLEEP 0x01
#define DEQUEUE_SAVE 0x02 /* matches ENQUEUE_RESTORE */
#define DEQUEUE_MOVE 0x04 /* matches ENQUEUE_MOVE */
+#define DEQUEUE_NOCLOCK 0x08 /* matches ENQUEUE_NOCLOCK */
#define ENQUEUE_WAKEUP 0x01
#define ENQUEUE_RESTORE 0x02
#define ENQUEUE_MOVE 0x04
+#define ENQUEUE_NOCLOCK 0x08
-#define ENQUEUE_HEAD 0x08
-#define ENQUEUE_REPLENISH 0x10
+#define ENQUEUE_HEAD 0x10
+#define ENQUEUE_REPLENISH 0x20
#ifdef CONFIG_SMP
-#define ENQUEUE_MIGRATED 0x20
+#define ENQUEUE_MIGRATED 0x40
#else
#define ENQUEUE_MIGRATED 0x00
#endif
@@ -1624,6 +1626,7 @@
struct rq *__task_rq_lock(struct task_struct *p, struct rq_flags *rf)
__acquires(rq->lock);
+
struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf)
__acquires(p->pi_lock)
__acquires(rq->lock);
@@ -1645,6 +1648,62 @@
raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
}
+static inline void
+rq_lock_irqsave(struct rq *rq, struct rq_flags *rf)
+ __acquires(rq->lock)
+{
+ raw_spin_lock_irqsave(&rq->lock, rf->flags);
+ rq_pin_lock(rq, rf);
+}
+
+static inline void
+rq_lock_irq(struct rq *rq, struct rq_flags *rf)
+ __acquires(rq->lock)
+{
+ raw_spin_lock_irq(&rq->lock);
+ rq_pin_lock(rq, rf);
+}
+
+static inline void
+rq_lock(struct rq *rq, struct rq_flags *rf)
+ __acquires(rq->lock)
+{
+ raw_spin_lock(&rq->lock);
+ rq_pin_lock(rq, rf);
+}
+
+static inline void
+rq_relock(struct rq *rq, struct rq_flags *rf)
+ __acquires(rq->lock)
+{
+ raw_spin_lock(&rq->lock);
+ rq_repin_lock(rq, rf);
+}
+
+static inline void
+rq_unlock_irqrestore(struct rq *rq, struct rq_flags *rf)
+ __releases(rq->lock)
+{
+ rq_unpin_lock(rq, rf);
+ raw_spin_unlock_irqrestore(&rq->lock, rf->flags);
+}
+
+static inline void
+rq_unlock_irq(struct rq *rq, struct rq_flags *rf)
+ __releases(rq->lock)
+{
+ rq_unpin_lock(rq, rf);
+ raw_spin_unlock_irq(&rq->lock);
+}
+
+static inline void
+rq_unlock(struct rq *rq, struct rq_flags *rf)
+ __releases(rq->lock)
+{
+ rq_unpin_lock(rq, rf);
+ raw_spin_unlock(&rq->lock);
+}
+
#ifdef CONFIG_SMP
#ifdef CONFIG_PREEMPT
@@ -1869,6 +1928,7 @@
#ifdef CONFIG_IRQ_TIME_ACCOUNTING
struct irqtime {
+ u64 total;
u64 tick_delta;
u64 irq_start_time;
struct u64_stats_sync sync;
@@ -1876,16 +1936,20 @@
DECLARE_PER_CPU(struct irqtime, cpu_irqtime);
+/*
+ * Returns the irqtime minus the softirq time computed by ksoftirqd.
+ * Otherwise ksoftirqd's sum_exec_runtime is substracted its own runtime
+ * and never move forward.
+ */
static inline u64 irq_time_read(int cpu)
{
struct irqtime *irqtime = &per_cpu(cpu_irqtime, cpu);
- u64 *cpustat = kcpustat_cpu(cpu).cpustat;
unsigned int seq;
u64 total;
do {
seq = __u64_stats_fetch_begin(&irqtime->sync);
- total = cpustat[CPUTIME_SOFTIRQ] + cpustat[CPUTIME_IRQ];
+ total = irqtime->total;
} while (__u64_stats_fetch_retry(&irqtime->sync, seq));
return total;
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 744fa61..4e09821 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -309,7 +309,7 @@
account_irq_exit_time(current);
__local_bh_enable(SOFTIRQ_OFFSET);
WARN_ON_ONCE(in_interrupt());
- tsk_restore_flags(current, old_flags, PF_MEMALLOC);
+ current_restore_flags(old_flags, PF_MEMALLOC);
}
asmlinkage __visible void do_softirq(void)
diff --git a/kernel/stacktrace.c b/kernel/stacktrace.c
index 9c15a91..f8edee9 100644
--- a/kernel/stacktrace.c
+++ b/kernel/stacktrace.c
@@ -54,8 +54,8 @@
EXPORT_SYMBOL_GPL(snprint_stack_trace);
/*
- * Architectures that do not implement save_stack_trace_tsk or
- * save_stack_trace_regs get this weak alias and a once-per-bootup warning
+ * Architectures that do not implement save_stack_trace_*()
+ * get these weak aliases and once-per-bootup warnings
* (whenever this facility is utilized - for example by procfs):
*/
__weak void
@@ -69,3 +69,11 @@
{
WARN_ONCE(1, KERN_INFO "save_stack_trace_regs() not implemented yet.\n");
}
+
+__weak int
+save_stack_trace_tsk_reliable(struct task_struct *tsk,
+ struct stack_trace *trace)
+{
+ WARN_ONCE(1, KERN_INFO "save_stack_tsk_reliable() not implemented yet.\n");
+ return -ENOSYS;
+}
diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index 8c8714f..21343d1 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -1176,6 +1176,8 @@
.maxlen = sizeof(unsigned int),
.mode = 0644,
.proc_handler = timer_migration_handler,
+ .extra1 = &zero,
+ .extra2 = &one,
},
#endif
#ifdef CONFIG_BPF_SYSCALL
diff --git a/kernel/time/alarmtimer.c b/kernel/time/alarmtimer.c
index ce3a31e..5cb5b00 100644
--- a/kernel/time/alarmtimer.c
+++ b/kernel/time/alarmtimer.c
@@ -541,7 +541,7 @@
*
* Returns the granularity of underlying alarm base clock
*/
-static int alarm_clock_getres(const clockid_t which_clock, struct timespec *tp)
+static int alarm_clock_getres(const clockid_t which_clock, struct timespec64 *tp)
{
if (!alarmtimer_get_rtcdev())
return -EINVAL;
@@ -558,14 +558,14 @@
*
* Provides the underlying alarm base time.
*/
-static int alarm_clock_get(clockid_t which_clock, struct timespec *tp)
+static int alarm_clock_get(clockid_t which_clock, struct timespec64 *tp)
{
struct alarm_base *base = &alarm_bases[clock2alarm(which_clock)];
if (!alarmtimer_get_rtcdev())
return -EINVAL;
- *tp = ktime_to_timespec(base->gettime());
+ *tp = ktime_to_timespec64(base->gettime());
return 0;
}
@@ -598,19 +598,19 @@
* Copies out the current itimerspec data
*/
static void alarm_timer_get(struct k_itimer *timr,
- struct itimerspec *cur_setting)
+ struct itimerspec64 *cur_setting)
{
ktime_t relative_expiry_time =
alarm_expires_remaining(&(timr->it.alarm.alarmtimer));
if (ktime_to_ns(relative_expiry_time) > 0) {
- cur_setting->it_value = ktime_to_timespec(relative_expiry_time);
+ cur_setting->it_value = ktime_to_timespec64(relative_expiry_time);
} else {
cur_setting->it_value.tv_sec = 0;
cur_setting->it_value.tv_nsec = 0;
}
- cur_setting->it_interval = ktime_to_timespec(timr->it.alarm.interval);
+ cur_setting->it_interval = ktime_to_timespec64(timr->it.alarm.interval);
}
/**
@@ -640,8 +640,8 @@
* Sets the timer to new_setting, and starts the timer.
*/
static int alarm_timer_set(struct k_itimer *timr, int flags,
- struct itimerspec *new_setting,
- struct itimerspec *old_setting)
+ struct itimerspec64 *new_setting,
+ struct itimerspec64 *old_setting)
{
ktime_t exp;
@@ -659,8 +659,8 @@
return TIMER_RETRY;
/* start the timer */
- timr->it.alarm.interval = timespec_to_ktime(new_setting->it_interval);
- exp = timespec_to_ktime(new_setting->it_value);
+ timr->it.alarm.interval = timespec64_to_ktime(new_setting->it_interval);
+ exp = timespec64_to_ktime(new_setting->it_value);
/* Convert (if necessary) to absolute time */
if (flags != TIMER_ABSTIME) {
ktime_t now;
@@ -790,13 +790,14 @@
* Handles clock_nanosleep calls against _ALARM clockids
*/
static int alarm_timer_nsleep(const clockid_t which_clock, int flags,
- struct timespec *tsreq, struct timespec __user *rmtp)
+ struct timespec64 *tsreq,
+ struct timespec __user *rmtp)
{
enum alarmtimer_type type = clock2alarm(which_clock);
+ struct restart_block *restart;
struct alarm alarm;
ktime_t exp;
int ret = 0;
- struct restart_block *restart;
if (!alarmtimer_get_rtcdev())
return -ENOTSUPP;
@@ -809,7 +810,7 @@
alarm_init(&alarm, type, alarmtimer_nsleep_wakeup);
- exp = timespec_to_ktime(*tsreq);
+ exp = timespec64_to_ktime(*tsreq);
/* Convert (if necessary) to absolute time */
if (flags != TIMER_ABSTIME) {
ktime_t now = alarm_bases[type].gettime();
diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
index 97ac095..4237e07 100644
--- a/kernel/time/clockevents.c
+++ b/kernel/time/clockevents.c
@@ -468,7 +468,7 @@
}
EXPORT_SYMBOL_GPL(clockevents_register_device);
-void clockevents_config(struct clock_event_device *dev, u32 freq)
+static void clockevents_config(struct clock_event_device *dev, u32 freq)
{
u64 sec;
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index ec08f52..ac053bb 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -987,7 +987,7 @@
* Returns:
* 0 when the timer was not active
* 1 when the timer was active
- * -1 when the timer is currently excuting the callback function and
+ * -1 when the timer is currently executing the callback function and
* cannot be stopped
*/
int hrtimer_try_to_cancel(struct hrtimer *timer)
@@ -1368,10 +1368,7 @@
ktime_to_ns(delta));
}
-/*
- * local version of hrtimer_peek_ahead_timers() called with interrupts
- * disabled.
- */
+/* called with interrupts disabled */
static inline void __hrtimer_peek_ahead_timers(void)
{
struct tick_device *td;
@@ -1506,7 +1503,7 @@
return ret;
}
-long hrtimer_nanosleep(struct timespec *rqtp, struct timespec __user *rmtp,
+long hrtimer_nanosleep(struct timespec64 *rqtp, struct timespec __user *rmtp,
const enum hrtimer_mode mode, const clockid_t clockid)
{
struct restart_block *restart;
@@ -1519,7 +1516,7 @@
slack = 0;
hrtimer_init_on_stack(&t.timer, clockid, mode);
- hrtimer_set_expires_range_ns(&t.timer, timespec_to_ktime(*rqtp), slack);
+ hrtimer_set_expires_range_ns(&t.timer, timespec64_to_ktime(*rqtp), slack);
if (do_nanosleep(&t, mode))
goto out;
@@ -1550,15 +1547,17 @@
SYSCALL_DEFINE2(nanosleep, struct timespec __user *, rqtp,
struct timespec __user *, rmtp)
{
+ struct timespec64 tu64;
struct timespec tu;
if (copy_from_user(&tu, rqtp, sizeof(tu)))
return -EFAULT;
- if (!timespec_valid(&tu))
+ tu64 = timespec_to_timespec64(tu);
+ if (!timespec64_valid(&tu64))
return -EINVAL;
- return hrtimer_nanosleep(&tu, rmtp, HRTIMER_MODE_REL, CLOCK_MONOTONIC);
+ return hrtimer_nanosleep(&tu64, rmtp, HRTIMER_MODE_REL, CLOCK_MONOTONIC);
}
/*
diff --git a/kernel/time/posix-clock.c b/kernel/time/posix-clock.c
index 9cff0ab..31d588d 100644
--- a/kernel/time/posix-clock.c
+++ b/kernel/time/posix-clock.c
@@ -297,7 +297,7 @@
return err;
}
-static int pc_clock_gettime(clockid_t id, struct timespec *ts)
+static int pc_clock_gettime(clockid_t id, struct timespec64 *ts)
{
struct posix_clock_desc cd;
int err;
@@ -316,7 +316,7 @@
return err;
}
-static int pc_clock_getres(clockid_t id, struct timespec *ts)
+static int pc_clock_getres(clockid_t id, struct timespec64 *ts)
{
struct posix_clock_desc cd;
int err;
@@ -335,7 +335,7 @@
return err;
}
-static int pc_clock_settime(clockid_t id, const struct timespec *ts)
+static int pc_clock_settime(clockid_t id, const struct timespec64 *ts)
{
struct posix_clock_desc cd;
int err;
@@ -399,7 +399,7 @@
return err;
}
-static void pc_timer_gettime(struct k_itimer *kit, struct itimerspec *ts)
+static void pc_timer_gettime(struct k_itimer *kit, struct itimerspec64 *ts)
{
clockid_t id = kit->it_clock;
struct posix_clock_desc cd;
@@ -414,7 +414,7 @@
}
static int pc_timer_settime(struct k_itimer *kit, int flags,
- struct itimerspec *ts, struct itimerspec *old)
+ struct itimerspec64 *ts, struct itimerspec64 *old)
{
clockid_t id = kit->it_clock;
struct posix_clock_desc cd;
diff --git a/kernel/time/posix-cpu-timers.c b/kernel/time/posix-cpu-timers.c
index 4513ad1..949e434 100644
--- a/kernel/time/posix-cpu-timers.c
+++ b/kernel/time/posix-cpu-timers.c
@@ -116,7 +116,7 @@
}
static int
-posix_cpu_clock_getres(const clockid_t which_clock, struct timespec *tp)
+posix_cpu_clock_getres(const clockid_t which_clock, struct timespec64 *tp)
{
int error = check_clock(which_clock);
if (!error) {
@@ -135,7 +135,7 @@
}
static int
-posix_cpu_clock_set(const clockid_t which_clock, const struct timespec *tp)
+posix_cpu_clock_set(const clockid_t which_clock, const struct timespec64 *tp)
{
/*
* You can never reset a CPU clock, but we check for other errors
@@ -261,7 +261,7 @@
static int posix_cpu_clock_get_task(struct task_struct *tsk,
const clockid_t which_clock,
- struct timespec *tp)
+ struct timespec64 *tp)
{
int err = -EINVAL;
u64 rtn;
@@ -275,13 +275,13 @@
}
if (!err)
- *tp = ns_to_timespec(rtn);
+ *tp = ns_to_timespec64(rtn);
return err;
}
-static int posix_cpu_clock_get(const clockid_t which_clock, struct timespec *tp)
+static int posix_cpu_clock_get(const clockid_t which_clock, struct timespec64 *tp)
{
const pid_t pid = CPUCLOCK_PID(which_clock);
int err = -EINVAL;
@@ -562,7 +562,7 @@
* and try again. (This happens when the timer is in the middle of firing.)
*/
static int posix_cpu_timer_set(struct k_itimer *timer, int timer_flags,
- struct itimerspec *new, struct itimerspec *old)
+ struct itimerspec64 *new, struct itimerspec64 *old)
{
unsigned long flags;
struct sighand_struct *sighand;
@@ -572,7 +572,7 @@
WARN_ON_ONCE(p == NULL);
- new_expires = timespec_to_ns(&new->it_value);
+ new_expires = timespec64_to_ns(&new->it_value);
/*
* Protect against sighand release/switch in exit/exec and p->cpu_timers
@@ -633,7 +633,7 @@
bump_cpu_timer(timer, val);
if (val < timer->it.cpu.expires) {
old_expires = timer->it.cpu.expires - val;
- old->it_value = ns_to_timespec(old_expires);
+ old->it_value = ns_to_timespec64(old_expires);
} else {
old->it_value.tv_nsec = 1;
old->it_value.tv_sec = 0;
@@ -671,7 +671,7 @@
* Install the new reload setting, and
* set up the signal and overrun bookkeeping.
*/
- timer->it.cpu.incr = timespec_to_ns(&new->it_interval);
+ timer->it.cpu.incr = timespec64_to_ns(&new->it_interval);
/*
* This acts as a modification timestamp for the timer,
@@ -695,12 +695,12 @@
ret = 0;
out:
if (old)
- old->it_interval = ns_to_timespec(old_incr);
+ old->it_interval = ns_to_timespec64(old_incr);
return ret;
}
-static void posix_cpu_timer_get(struct k_itimer *timer, struct itimerspec *itp)
+static void posix_cpu_timer_get(struct k_itimer *timer, struct itimerspec64 *itp)
{
u64 now;
struct task_struct *p = timer->it.cpu.task;
@@ -710,7 +710,7 @@
/*
* Easy part: convert the reload time.
*/
- itp->it_interval = ns_to_timespec(timer->it.cpu.incr);
+ itp->it_interval = ns_to_timespec64(timer->it.cpu.incr);
if (timer->it.cpu.expires == 0) { /* Timer not armed at all. */
itp->it_value.tv_sec = itp->it_value.tv_nsec = 0;
@@ -739,7 +739,7 @@
* Call the timer disarmed, nothing else to do.
*/
timer->it.cpu.expires = 0;
- itp->it_value = ns_to_timespec(timer->it.cpu.expires);
+ itp->it_value = ns_to_timespec64(timer->it.cpu.expires);
return;
} else {
cpu_timer_sample_group(timer->it_clock, p, &now);
@@ -748,7 +748,7 @@
}
if (now < timer->it.cpu.expires) {
- itp->it_value = ns_to_timespec(timer->it.cpu.expires - now);
+ itp->it_value = ns_to_timespec64(timer->it.cpu.expires - now);
} else {
/*
* The timer should have expired already, but the firing
@@ -825,6 +825,8 @@
* At the hard limit, we just die.
* No need to calculate anything else now.
*/
+ pr_info("CPU Watchdog Timeout (hard): %s[%d]\n",
+ tsk->comm, task_pid_nr(tsk));
__group_send_sig_info(SIGKILL, SEND_SIG_PRIV, tsk);
return;
}
@@ -836,8 +838,7 @@
soft += USEC_PER_SEC;
sig->rlim[RLIMIT_RTTIME].rlim_cur = soft;
}
- printk(KERN_INFO
- "RT Watchdog Timeout: %s[%d]\n",
+ pr_info("RT Watchdog Timeout (soft): %s[%d]\n",
tsk->comm, task_pid_nr(tsk));
__group_send_sig_info(SIGXCPU, SEND_SIG_PRIV, tsk);
}
@@ -935,6 +936,8 @@
* At the hard limit, we just die.
* No need to calculate anything else now.
*/
+ pr_info("RT Watchdog Timeout (hard): %s[%d]\n",
+ tsk->comm, task_pid_nr(tsk));
__group_send_sig_info(SIGKILL, SEND_SIG_PRIV, tsk);
return;
}
@@ -942,6 +945,8 @@
/*
* At the soft limit, send a SIGXCPU every second.
*/
+ pr_info("CPU Watchdog Timeout (soft): %s[%d]\n",
+ tsk->comm, task_pid_nr(tsk));
__group_send_sig_info(SIGXCPU, SEND_SIG_PRIV, tsk);
if (soft < hard) {
soft++;
@@ -1214,7 +1219,7 @@
}
static int do_cpu_nanosleep(const clockid_t which_clock, int flags,
- struct timespec *rqtp, struct itimerspec *it)
+ struct timespec64 *rqtp, struct itimerspec64 *it)
{
struct k_itimer timer;
int error;
@@ -1229,7 +1234,7 @@
error = posix_cpu_timer_create(&timer);
timer.it_process = current;
if (!error) {
- static struct itimerspec zero_it;
+ static struct itimerspec64 zero_it;
memset(it, 0, sizeof *it);
it->it_value = *rqtp;
@@ -1264,7 +1269,7 @@
/*
* We were interrupted by a signal.
*/
- *rqtp = ns_to_timespec(timer.it.cpu.expires);
+ *rqtp = ns_to_timespec64(timer.it.cpu.expires);
error = posix_cpu_timer_set(&timer, 0, &zero_it, it);
if (!error) {
/*
@@ -1301,10 +1306,11 @@
static long posix_cpu_nsleep_restart(struct restart_block *restart_block);
static int posix_cpu_nsleep(const clockid_t which_clock, int flags,
- struct timespec *rqtp, struct timespec __user *rmtp)
+ struct timespec64 *rqtp, struct timespec __user *rmtp)
{
struct restart_block *restart_block = ¤t->restart_block;
- struct itimerspec it;
+ struct itimerspec64 it;
+ struct timespec ts;
int error;
/*
@@ -1324,13 +1330,14 @@
/*
* Report back to the user the time still remaining.
*/
- if (rmtp && copy_to_user(rmtp, &it.it_value, sizeof *rmtp))
+ ts = timespec64_to_timespec(it.it_value);
+ if (rmtp && copy_to_user(rmtp, &ts, sizeof(*rmtp)))
return -EFAULT;
restart_block->fn = posix_cpu_nsleep_restart;
restart_block->nanosleep.clockid = which_clock;
restart_block->nanosleep.rmtp = rmtp;
- restart_block->nanosleep.expires = timespec_to_ns(rqtp);
+ restart_block->nanosleep.expires = timespec64_to_ns(rqtp);
}
return error;
}
@@ -1338,11 +1345,12 @@
static long posix_cpu_nsleep_restart(struct restart_block *restart_block)
{
clockid_t which_clock = restart_block->nanosleep.clockid;
- struct timespec t;
- struct itimerspec it;
+ struct itimerspec64 it;
+ struct timespec64 t;
+ struct timespec tmp;
int error;
- t = ns_to_timespec(restart_block->nanosleep.expires);
+ t = ns_to_timespec64(restart_block->nanosleep.expires);
error = do_cpu_nanosleep(which_clock, TIMER_ABSTIME, &t, &it);
@@ -1351,10 +1359,11 @@
/*
* Report back to the user the time still remaining.
*/
- if (rmtp && copy_to_user(rmtp, &it.it_value, sizeof *rmtp))
+ tmp = timespec64_to_timespec(it.it_value);
+ if (rmtp && copy_to_user(rmtp, &tmp, sizeof(*rmtp)))
return -EFAULT;
- restart_block->nanosleep.expires = timespec_to_ns(&t);
+ restart_block->nanosleep.expires = timespec64_to_ns(&t);
}
return error;
@@ -1364,12 +1373,12 @@
#define THREAD_CLOCK MAKE_THREAD_CPUCLOCK(0, CPUCLOCK_SCHED)
static int process_cpu_clock_getres(const clockid_t which_clock,
- struct timespec *tp)
+ struct timespec64 *tp)
{
return posix_cpu_clock_getres(PROCESS_CLOCK, tp);
}
static int process_cpu_clock_get(const clockid_t which_clock,
- struct timespec *tp)
+ struct timespec64 *tp)
{
return posix_cpu_clock_get(PROCESS_CLOCK, tp);
}
@@ -1379,7 +1388,7 @@
return posix_cpu_timer_create(timer);
}
static int process_cpu_nsleep(const clockid_t which_clock, int flags,
- struct timespec *rqtp,
+ struct timespec64 *rqtp,
struct timespec __user *rmtp)
{
return posix_cpu_nsleep(PROCESS_CLOCK, flags, rqtp, rmtp);
@@ -1389,12 +1398,12 @@
return -EINVAL;
}
static int thread_cpu_clock_getres(const clockid_t which_clock,
- struct timespec *tp)
+ struct timespec64 *tp)
{
return posix_cpu_clock_getres(THREAD_CLOCK, tp);
}
static int thread_cpu_clock_get(const clockid_t which_clock,
- struct timespec *tp)
+ struct timespec64 *tp)
{
return posix_cpu_clock_get(THREAD_CLOCK, tp);
}
diff --git a/kernel/time/posix-stubs.c b/kernel/time/posix-stubs.c
index cd6716e..c0cd53e 100644
--- a/kernel/time/posix-stubs.c
+++ b/kernel/time/posix-stubs.c
@@ -49,26 +49,32 @@
SYSCALL_DEFINE2(clock_settime, const clockid_t, which_clock,
const struct timespec __user *, tp)
{
+ struct timespec64 new_tp64;
struct timespec new_tp;
if (which_clock != CLOCK_REALTIME)
return -EINVAL;
if (copy_from_user(&new_tp, tp, sizeof (*tp)))
return -EFAULT;
- return do_sys_settimeofday(&new_tp, NULL);
+
+ new_tp64 = timespec_to_timespec64(new_tp);
+ return do_sys_settimeofday64(&new_tp64, NULL);
}
SYSCALL_DEFINE2(clock_gettime, const clockid_t, which_clock,
struct timespec __user *,tp)
{
+ struct timespec64 kernel_tp64;
struct timespec kernel_tp;
switch (which_clock) {
- case CLOCK_REALTIME: ktime_get_real_ts(&kernel_tp); break;
- case CLOCK_MONOTONIC: ktime_get_ts(&kernel_tp); break;
- case CLOCK_BOOTTIME: get_monotonic_boottime(&kernel_tp); break;
+ case CLOCK_REALTIME: ktime_get_real_ts64(&kernel_tp64); break;
+ case CLOCK_MONOTONIC: ktime_get_ts64(&kernel_tp64); break;
+ case CLOCK_BOOTTIME: get_monotonic_boottime64(&kernel_tp64); break;
default: return -EINVAL;
}
+
+ kernel_tp = timespec64_to_timespec(kernel_tp64);
if (copy_to_user(tp, &kernel_tp, sizeof (kernel_tp)))
return -EFAULT;
return 0;
@@ -97,6 +103,7 @@
const struct timespec __user *, rqtp,
struct timespec __user *, rmtp)
{
+ struct timespec64 t64;
struct timespec t;
switch (which_clock) {
@@ -105,9 +112,10 @@
case CLOCK_BOOTTIME:
if (copy_from_user(&t, rqtp, sizeof (struct timespec)))
return -EFAULT;
- if (!timespec_valid(&t))
+ t64 = timespec_to_timespec64(t);
+ if (!timespec64_valid(&t64))
return -EINVAL;
- return hrtimer_nanosleep(&t, rmtp, flags & TIMER_ABSTIME ?
+ return hrtimer_nanosleep(&t64, rmtp, flags & TIMER_ABSTIME ?
HRTIMER_MODE_ABS : HRTIMER_MODE_REL,
which_clock);
default:
diff --git a/kernel/time/posix-timers.c b/kernel/time/posix-timers.c
index 50a6a47..4d7b2ce 100644
--- a/kernel/time/posix-timers.c
+++ b/kernel/time/posix-timers.c
@@ -130,12 +130,12 @@
/*
* These ones are defined below.
*/
-static int common_nsleep(const clockid_t, int flags, struct timespec *t,
+static int common_nsleep(const clockid_t, int flags, struct timespec64 *t,
struct timespec __user *rmtp);
static int common_timer_create(struct k_itimer *new_timer);
-static void common_timer_get(struct k_itimer *, struct itimerspec *);
+static void common_timer_get(struct k_itimer *, struct itimerspec64 *);
static int common_timer_set(struct k_itimer *, int,
- struct itimerspec *, struct itimerspec *);
+ struct itimerspec64 *, struct itimerspec64 *);
static int common_timer_del(struct k_itimer *timer);
static enum hrtimer_restart posix_timer_fn(struct hrtimer *data);
@@ -204,17 +204,17 @@
}
/* Get clock_realtime */
-static int posix_clock_realtime_get(clockid_t which_clock, struct timespec *tp)
+static int posix_clock_realtime_get(clockid_t which_clock, struct timespec64 *tp)
{
- ktime_get_real_ts(tp);
+ ktime_get_real_ts64(tp);
return 0;
}
/* Set clock_realtime */
static int posix_clock_realtime_set(const clockid_t which_clock,
- const struct timespec *tp)
+ const struct timespec64 *tp)
{
- return do_sys_settimeofday(tp, NULL);
+ return do_sys_settimeofday64(tp, NULL);
}
static int posix_clock_realtime_adj(const clockid_t which_clock,
@@ -226,54 +226,54 @@
/*
* Get monotonic time for posix timers
*/
-static int posix_ktime_get_ts(clockid_t which_clock, struct timespec *tp)
+static int posix_ktime_get_ts(clockid_t which_clock, struct timespec64 *tp)
{
- ktime_get_ts(tp);
+ ktime_get_ts64(tp);
return 0;
}
/*
* Get monotonic-raw time for posix timers
*/
-static int posix_get_monotonic_raw(clockid_t which_clock, struct timespec *tp)
+static int posix_get_monotonic_raw(clockid_t which_clock, struct timespec64 *tp)
{
- getrawmonotonic(tp);
+ getrawmonotonic64(tp);
return 0;
}
-static int posix_get_realtime_coarse(clockid_t which_clock, struct timespec *tp)
+static int posix_get_realtime_coarse(clockid_t which_clock, struct timespec64 *tp)
{
- *tp = current_kernel_time();
+ *tp = current_kernel_time64();
return 0;
}
static int posix_get_monotonic_coarse(clockid_t which_clock,
- struct timespec *tp)
+ struct timespec64 *tp)
{
- *tp = get_monotonic_coarse();
+ *tp = get_monotonic_coarse64();
return 0;
}
-static int posix_get_coarse_res(const clockid_t which_clock, struct timespec *tp)
+static int posix_get_coarse_res(const clockid_t which_clock, struct timespec64 *tp)
{
- *tp = ktime_to_timespec(KTIME_LOW_RES);
+ *tp = ktime_to_timespec64(KTIME_LOW_RES);
return 0;
}
-static int posix_get_boottime(const clockid_t which_clock, struct timespec *tp)
+static int posix_get_boottime(const clockid_t which_clock, struct timespec64 *tp)
{
- get_monotonic_boottime(tp);
+ get_monotonic_boottime64(tp);
return 0;
}
-static int posix_get_tai(clockid_t which_clock, struct timespec *tp)
+static int posix_get_tai(clockid_t which_clock, struct timespec64 *tp)
{
- timekeeping_clocktai(tp);
+ timekeeping_clocktai64(tp);
return 0;
}
-static int posix_get_hrtimer_res(clockid_t which_clock, struct timespec *tp)
+static int posix_get_hrtimer_res(clockid_t which_clock, struct timespec64 *tp)
{
tp->tv_sec = 0;
tp->tv_nsec = hrtimer_resolution;
@@ -734,18 +734,18 @@
* report.
*/
static void
-common_timer_get(struct k_itimer *timr, struct itimerspec *cur_setting)
+common_timer_get(struct k_itimer *timr, struct itimerspec64 *cur_setting)
{
ktime_t now, remaining, iv;
struct hrtimer *timer = &timr->it.real.timer;
- memset(cur_setting, 0, sizeof(struct itimerspec));
+ memset(cur_setting, 0, sizeof(*cur_setting));
iv = timr->it.real.interval;
/* interval timer ? */
if (iv)
- cur_setting->it_interval = ktime_to_timespec(iv);
+ cur_setting->it_interval = ktime_to_timespec64(iv);
else if (!hrtimer_active(timer) &&
(timr->it_sigev_notify & ~SIGEV_THREAD_ID) != SIGEV_NONE)
return;
@@ -771,13 +771,14 @@
if ((timr->it_sigev_notify & ~SIGEV_THREAD_ID) != SIGEV_NONE)
cur_setting->it_value.tv_nsec = 1;
} else
- cur_setting->it_value = ktime_to_timespec(remaining);
+ cur_setting->it_value = ktime_to_timespec64(remaining);
}
/* Get the time remaining on a POSIX.1b interval timer. */
SYSCALL_DEFINE2(timer_gettime, timer_t, timer_id,
struct itimerspec __user *, setting)
{
+ struct itimerspec64 cur_setting64;
struct itimerspec cur_setting;
struct k_itimer *timr;
struct k_clock *kc;
@@ -792,10 +793,11 @@
if (WARN_ON_ONCE(!kc || !kc->timer_get))
ret = -EINVAL;
else
- kc->timer_get(timr, &cur_setting);
+ kc->timer_get(timr, &cur_setting64);
unlock_timer(timr, flags);
+ cur_setting = itimerspec64_to_itimerspec(&cur_setting64);
if (!ret && copy_to_user(setting, &cur_setting, sizeof (cur_setting)))
return -EFAULT;
@@ -831,7 +833,7 @@
/* timr->it_lock is taken. */
static int
common_timer_set(struct k_itimer *timr, int flags,
- struct itimerspec *new_setting, struct itimerspec *old_setting)
+ struct itimerspec64 *new_setting, struct itimerspec64 *old_setting)
{
struct hrtimer *timer = &timr->it.real.timer;
enum hrtimer_mode mode;
@@ -860,10 +862,10 @@
hrtimer_init(&timr->it.real.timer, timr->it_clock, mode);
timr->it.real.timer.function = posix_timer_fn;
- hrtimer_set_expires(timer, timespec_to_ktime(new_setting->it_value));
+ hrtimer_set_expires(timer, timespec64_to_ktime(new_setting->it_value));
/* Convert interval */
- timr->it.real.interval = timespec_to_ktime(new_setting->it_interval);
+ timr->it.real.interval = timespec64_to_ktime(new_setting->it_interval);
/* SIGEV_NONE timers are not queued ! See common_timer_get */
if (((timr->it_sigev_notify & ~SIGEV_THREAD_ID) == SIGEV_NONE)) {
@@ -883,21 +885,23 @@
const struct itimerspec __user *, new_setting,
struct itimerspec __user *, old_setting)
{
- struct k_itimer *timr;
+ struct itimerspec64 new_spec64, old_spec64;
+ struct itimerspec64 *rtn = old_setting ? &old_spec64 : NULL;
struct itimerspec new_spec, old_spec;
- int error = 0;
+ struct k_itimer *timr;
unsigned long flag;
- struct itimerspec *rtn = old_setting ? &old_spec : NULL;
struct k_clock *kc;
+ int error = 0;
if (!new_setting)
return -EINVAL;
if (copy_from_user(&new_spec, new_setting, sizeof (new_spec)))
return -EFAULT;
+ new_spec64 = itimerspec_to_itimerspec64(&new_spec);
- if (!timespec_valid(&new_spec.it_interval) ||
- !timespec_valid(&new_spec.it_value))
+ if (!timespec64_valid(&new_spec64.it_interval) ||
+ !timespec64_valid(&new_spec64.it_value))
return -EINVAL;
retry:
timr = lock_timer(timer_id, &flag);
@@ -908,7 +912,7 @@
if (WARN_ON_ONCE(!kc || !kc->timer_set))
error = -EINVAL;
else
- error = kc->timer_set(timr, flags, &new_spec, rtn);
+ error = kc->timer_set(timr, flags, &new_spec64, rtn);
unlock_timer(timr, flag);
if (error == TIMER_RETRY) {
@@ -916,6 +920,7 @@
goto retry;
}
+ old_spec = itimerspec64_to_itimerspec(&old_spec64);
if (old_setting && !error &&
copy_to_user(old_setting, &old_spec, sizeof (old_spec)))
error = -EFAULT;
@@ -1014,6 +1019,7 @@
const struct timespec __user *, tp)
{
struct k_clock *kc = clockid_to_kclock(which_clock);
+ struct timespec64 new_tp64;
struct timespec new_tp;
if (!kc || !kc->clock_set)
@@ -1021,21 +1027,24 @@
if (copy_from_user(&new_tp, tp, sizeof (*tp)))
return -EFAULT;
+ new_tp64 = timespec_to_timespec64(new_tp);
- return kc->clock_set(which_clock, &new_tp);
+ return kc->clock_set(which_clock, &new_tp64);
}
SYSCALL_DEFINE2(clock_gettime, const clockid_t, which_clock,
struct timespec __user *,tp)
{
struct k_clock *kc = clockid_to_kclock(which_clock);
+ struct timespec64 kernel_tp64;
struct timespec kernel_tp;
int error;
if (!kc)
return -EINVAL;
- error = kc->clock_get(which_clock, &kernel_tp);
+ error = kc->clock_get(which_clock, &kernel_tp64);
+ kernel_tp = timespec64_to_timespec(kernel_tp64);
if (!error && copy_to_user(tp, &kernel_tp, sizeof (kernel_tp)))
error = -EFAULT;
@@ -1070,13 +1079,15 @@
struct timespec __user *, tp)
{
struct k_clock *kc = clockid_to_kclock(which_clock);
+ struct timespec64 rtn_tp64;
struct timespec rtn_tp;
int error;
if (!kc)
return -EINVAL;
- error = kc->clock_getres(which_clock, &rtn_tp);
+ error = kc->clock_getres(which_clock, &rtn_tp64);
+ rtn_tp = timespec64_to_timespec(rtn_tp64);
if (!error && tp && copy_to_user(tp, &rtn_tp, sizeof (rtn_tp)))
error = -EFAULT;
@@ -1088,7 +1099,7 @@
* nanosleep for monotonic and realtime clocks
*/
static int common_nsleep(const clockid_t which_clock, int flags,
- struct timespec *tsave, struct timespec __user *rmtp)
+ struct timespec64 *tsave, struct timespec __user *rmtp)
{
return hrtimer_nanosleep(tsave, rmtp, flags & TIMER_ABSTIME ?
HRTIMER_MODE_ABS : HRTIMER_MODE_REL,
@@ -1100,6 +1111,7 @@
struct timespec __user *, rmtp)
{
struct k_clock *kc = clockid_to_kclock(which_clock);
+ struct timespec64 t64;
struct timespec t;
if (!kc)
@@ -1110,10 +1122,11 @@
if (copy_from_user(&t, rqtp, sizeof (struct timespec)))
return -EFAULT;
- if (!timespec_valid(&t))
+ t64 = timespec_to_timespec64(t);
+ if (!timespec64_valid(&t64))
return -EINVAL;
- return kc->nsleep(which_clock, flags, &t, rmtp);
+ return kc->nsleep(which_clock, flags, &t64, rmtp);
}
/*
diff --git a/kernel/time/sched_clock.c b/kernel/time/sched_clock.c
index ea6b610..2d8f05a 100644
--- a/kernel/time/sched_clock.c
+++ b/kernel/time/sched_clock.c
@@ -206,6 +206,11 @@
update_clock_read_data(&rd);
+ if (sched_clock_timer.function != NULL) {
+ /* update timeout for clock wrap */
+ hrtimer_start(&sched_clock_timer, cd.wrap_kt, HRTIMER_MODE_REL);
+ }
+
r = rate;
if (r >= 4000000) {
r /= 1000000;
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 7fe53be..64c97fc 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -993,6 +993,18 @@
return ts->sleep_length;
}
+/**
+ * tick_nohz_get_idle_calls - return the current idle calls counter value
+ *
+ * Called from the schedutil frequency scaling governor in scheduler context.
+ */
+unsigned long tick_nohz_get_idle_calls(void)
+{
+ struct tick_sched *ts = this_cpu_ptr(&tick_cpu_sched);
+
+ return ts->idle_calls;
+}
+
static void tick_nohz_account_idle_ticks(struct tick_sched *ts)
{
#ifndef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE
diff --git a/kernel/time/time.c b/kernel/time/time.c
index 25bdd25..6574bba 100644
--- a/kernel/time/time.c
+++ b/kernel/time/time.c
@@ -193,8 +193,8 @@
SYSCALL_DEFINE2(settimeofday, struct timeval __user *, tv,
struct timezone __user *, tz)
{
+ struct timespec64 new_ts;
struct timeval user_tv;
- struct timespec new_ts;
struct timezone new_tz;
if (tv) {
@@ -212,7 +212,7 @@
return -EFAULT;
}
- return do_sys_settimeofday(tv ? &new_ts : NULL, tz ? &new_tz : NULL);
+ return do_sys_settimeofday64(tv ? &new_ts : NULL, tz ? &new_tz : NULL);
}
SYSCALL_DEFINE1(adjtimex, struct timex __user *, txc_p)
diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index 5b63a21..9652bc5 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -996,8 +996,7 @@
return 0;
/* Interpolate shortest distance from beginning or end of history */
- interp_forward = partial_history_cycles > total_history_cycles/2 ?
- true : false;
+ interp_forward = partial_history_cycles > total_history_cycles / 2;
partial_history_cycles = interp_forward ?
total_history_cycles - partial_history_cycles :
partial_history_cycles;
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 1dc0256..152a706 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -241,7 +241,7 @@
int ret;
mutex_lock(&mutex);
- ret = proc_dointvec(table, write, buffer, lenp, ppos);
+ ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
if (!ret && write)
timers_update_migration(false);
mutex_unlock(&mutex);
@@ -1120,7 +1120,7 @@
EXPORT_SYMBOL_GPL(add_timer_on);
/**
- * del_timer - deactive a timer.
+ * del_timer - deactivate a timer.
* @timer: the timer to be deactivated
*
* del_timer() deactivates a timer - this works on both active and inactive
diff --git a/kernel/time/timer_list.c b/kernel/time/timer_list.c
index ff8d5c1..0e7f542 100644
--- a/kernel/time/timer_list.c
+++ b/kernel/time/timer_list.c
@@ -16,6 +16,7 @@
#include <linux/sched.h>
#include <linux/seq_file.h>
#include <linux/kallsyms.h>
+#include <linux/nmi.h>
#include <linux/uaccess.h>
@@ -86,6 +87,9 @@
next_one:
i = 0;
+
+ touch_nmi_watchdog();
+
raw_spin_lock_irqsave(&base->cpu_base->lock, flags);
curr = timerqueue_getnext(&base->active);
@@ -197,6 +201,8 @@
{
struct clock_event_device *dev = td->evtdev;
+ touch_nmi_watchdog();
+
SEQ_printf(m, "Tick Device: mode: %d\n", td->mode);
if (cpu < 0)
SEQ_printf(m, "Broadcast device\n");
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index d4a06e7..9619b57 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -455,7 +455,7 @@
select UPROBES
select PROBE_EVENTS
select TRACING
- default n
+ default y
help
This allows the user to add tracing events on top of userspace
dynamic events (similar to tracepoints) on the fly via the trace
diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
index b2058a7..bd8ae8d 100644
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -690,8 +690,8 @@
/**
* blk_add_trace_rq - Add a trace for a request oriented action
- * @q: queue the io is for
* @rq: the source request
+ * @error: return status to log
* @nr_bytes: number of completed bytes
* @what: the action
*
@@ -699,10 +699,10 @@
* Records an action against a request. Will log the bio offset + size.
*
**/
-static void blk_add_trace_rq(struct request_queue *q, struct request *rq,
+static void blk_add_trace_rq(struct request *rq, int error,
unsigned int nr_bytes, u32 what)
{
- struct blk_trace *bt = q->blk_trace;
+ struct blk_trace *bt = rq->q->blk_trace;
if (likely(!bt))
return;
@@ -713,40 +713,32 @@
what |= BLK_TC_ACT(BLK_TC_FS);
__blk_add_trace(bt, blk_rq_trace_sector(rq), nr_bytes, req_op(rq),
- rq->cmd_flags, what, rq->errors, 0, NULL);
-}
-
-static void blk_add_trace_rq_abort(void *ignore,
- struct request_queue *q, struct request *rq)
-{
- blk_add_trace_rq(q, rq, blk_rq_bytes(rq), BLK_TA_ABORT);
+ rq->cmd_flags, what, error, 0, NULL);
}
static void blk_add_trace_rq_insert(void *ignore,
struct request_queue *q, struct request *rq)
{
- blk_add_trace_rq(q, rq, blk_rq_bytes(rq), BLK_TA_INSERT);
+ blk_add_trace_rq(rq, 0, blk_rq_bytes(rq), BLK_TA_INSERT);
}
static void blk_add_trace_rq_issue(void *ignore,
struct request_queue *q, struct request *rq)
{
- blk_add_trace_rq(q, rq, blk_rq_bytes(rq), BLK_TA_ISSUE);
+ blk_add_trace_rq(rq, 0, blk_rq_bytes(rq), BLK_TA_ISSUE);
}
static void blk_add_trace_rq_requeue(void *ignore,
struct request_queue *q,
struct request *rq)
{
- blk_add_trace_rq(q, rq, blk_rq_bytes(rq), BLK_TA_REQUEUE);
+ blk_add_trace_rq(rq, 0, blk_rq_bytes(rq), BLK_TA_REQUEUE);
}
-static void blk_add_trace_rq_complete(void *ignore,
- struct request_queue *q,
- struct request *rq,
- unsigned int nr_bytes)
+static void blk_add_trace_rq_complete(void *ignore, struct request *rq,
+ int error, unsigned int nr_bytes)
{
- blk_add_trace_rq(q, rq, nr_bytes, BLK_TA_COMPLETE);
+ blk_add_trace_rq(rq, error, nr_bytes, BLK_TA_COMPLETE);
}
/**
@@ -941,7 +933,7 @@
r.sector_from = cpu_to_be64(from);
__blk_add_trace(bt, blk_rq_pos(rq), blk_rq_bytes(rq),
- rq_data_dir(rq), 0, BLK_TA_REMAP, !!rq->errors,
+ rq_data_dir(rq), 0, BLK_TA_REMAP, 0,
sizeof(r), &r);
}
@@ -966,7 +958,7 @@
return;
__blk_add_trace(bt, blk_rq_trace_sector(rq), blk_rq_bytes(rq), 0, 0,
- BLK_TA_DRV_DATA, rq->errors, len, data);
+ BLK_TA_DRV_DATA, 0, len, data);
}
EXPORT_SYMBOL_GPL(blk_add_driver_data);
@@ -974,8 +966,6 @@
{
int ret;
- ret = register_trace_block_rq_abort(blk_add_trace_rq_abort, NULL);
- WARN_ON(ret);
ret = register_trace_block_rq_insert(blk_add_trace_rq_insert, NULL);
WARN_ON(ret);
ret = register_trace_block_rq_issue(blk_add_trace_rq_issue, NULL);
@@ -1028,7 +1018,6 @@
unregister_trace_block_rq_requeue(blk_add_trace_rq_requeue, NULL);
unregister_trace_block_rq_issue(blk_add_trace_rq_issue, NULL);
unregister_trace_block_rq_insert(blk_add_trace_rq_insert, NULL);
- unregister_trace_block_rq_abort(blk_add_trace_rq_abort, NULL);
tracepoint_synchronize_unregister();
}
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index cee9802..460a031 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -96,7 +96,7 @@
if (unlikely(in_interrupt() ||
current->flags & (PF_KTHREAD | PF_EXITING)))
return -EPERM;
- if (unlikely(segment_eq(get_fs(), KERNEL_DS)))
+ if (unlikely(uaccess_kernel()))
return -EPERM;
if (!access_ok(VERIFY_WRITE, unsafe_ptr, size))
return -EPERM;
@@ -501,16 +501,11 @@
return true;
}
-static const struct bpf_verifier_ops kprobe_prog_ops = {
+const struct bpf_verifier_ops kprobe_prog_ops = {
.get_func_proto = kprobe_prog_func_proto,
.is_valid_access = kprobe_prog_is_valid_access,
};
-static struct bpf_prog_type_list kprobe_tl __ro_after_init = {
- .ops = &kprobe_prog_ops,
- .type = BPF_PROG_TYPE_KPROBE,
-};
-
BPF_CALL_5(bpf_perf_event_output_tp, void *, tp_buff, struct bpf_map *, map,
u64, flags, void *, data, u64, size)
{
@@ -584,16 +579,11 @@
return true;
}
-static const struct bpf_verifier_ops tracepoint_prog_ops = {
+const struct bpf_verifier_ops tracepoint_prog_ops = {
.get_func_proto = tp_prog_func_proto,
.is_valid_access = tp_prog_is_valid_access,
};
-static struct bpf_prog_type_list tracepoint_tl __ro_after_init = {
- .ops = &tracepoint_prog_ops,
- .type = BPF_PROG_TYPE_TRACEPOINT,
-};
-
static bool pe_prog_is_valid_access(int off, int size, enum bpf_access_type type,
enum bpf_reg_type *reg_type)
{
@@ -642,22 +632,8 @@
return insn - insn_buf;
}
-static const struct bpf_verifier_ops perf_event_prog_ops = {
+const struct bpf_verifier_ops perf_event_prog_ops = {
.get_func_proto = tp_prog_func_proto,
.is_valid_access = pe_prog_is_valid_access,
.convert_ctx_access = pe_prog_convert_ctx_access,
};
-
-static struct bpf_prog_type_list perf_event_tl __ro_after_init = {
- .ops = &perf_event_prog_ops,
- .type = BPF_PROG_TYPE_PERF_EVENT,
-};
-
-static int __init register_kprobe_prog_ops(void)
-{
- bpf_register_prog_type(&kprobe_tl);
- bpf_register_prog_type(&tracepoint_tl);
- bpf_register_prog_type(&perf_event_tl);
- return 0;
-}
-late_initcall(register_kprobe_prog_ops);
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 0ad75e9..0029fe6 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -4355,6 +4355,7 @@
"\t -:[<group>/]<event>\n"
#ifdef CONFIG_KPROBE_EVENTS
"\t place: [<module>:]<symbol>[+<offset>]|<memaddr>\n"
+ "place (kretprobe): [<module>:]<symbol>[+<offset>]|<memaddr>\n"
#endif
#ifdef CONFIG_UPROBE_EVENTS
"\t place: <path>:<offset>\n"
@@ -5529,7 +5530,6 @@
.partial = partial_def,
.nr_pages = 0, /* This gets updated below. */
.nr_pages_max = PIPE_DEF_BUFFERS,
- .flags = flags,
.ops = &tracing_pipe_buf_ops,
.spd_release = tracing_spd_release_pipe,
};
@@ -6427,7 +6427,6 @@
.pages = pages_def,
.partial = partial_def,
.nr_pages_max = PIPE_DEF_BUFFERS,
- .flags = flags,
.ops = &buffer_pipe_buf_ops,
.spd_release = buffer_spd_release,
};
diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index 5f688cc..013f4e7 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -681,10 +681,6 @@
return -EINVAL;
}
if (isdigit(argv[1][0])) {
- if (is_return) {
- pr_info("Return probe point must be a symbol.\n");
- return -EINVAL;
- }
/* an address specified */
ret = kstrtoul(&argv[1][0], 0, (unsigned long *)&addr);
if (ret) {
@@ -700,8 +696,9 @@
pr_info("Failed to parse symbol.\n");
return ret;
}
- if (offset && is_return) {
- pr_info("Return probe must be used without offset.\n");
+ if (offset && is_return &&
+ !function_offset_within_entry(NULL, symbol, offset)) {
+ pr_info("Given offset is not valid for return probe.\n");
return -EINVAL;
}
}
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index c0168b7..c74bf39 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -3209,9 +3209,8 @@
INIT_LIST_HEAD(&pool->idle_list);
hash_init(pool->busy_hash);
- init_timer_deferrable(&pool->idle_timer);
- pool->idle_timer.function = idle_worker_timeout;
- pool->idle_timer.data = (unsigned long)pool;
+ setup_deferrable_timer(&pool->idle_timer, idle_worker_timeout,
+ (unsigned long)pool);
setup_timer(&pool->mayday_timer, pool_mayday_timeout,
(unsigned long)pool);
@@ -4735,6 +4734,29 @@
return wfc.ret;
}
EXPORT_SYMBOL_GPL(work_on_cpu);
+
+/**
+ * work_on_cpu_safe - run a function in thread context on a particular cpu
+ * @cpu: the cpu to run on
+ * @fn: the function to run
+ * @arg: the function argument
+ *
+ * Disables CPU hotplug and calls work_on_cpu(). The caller must not hold
+ * any locks which would prevent @fn from completing.
+ *
+ * Return: The value @fn returns.
+ */
+long work_on_cpu_safe(int cpu, long (*fn)(void *), void *arg)
+{
+ long ret = -ENODEV;
+
+ get_online_cpus();
+ if (cpu_online(cpu))
+ ret = work_on_cpu(cpu, fn, arg);
+ put_online_cpus();
+ return ret;
+}
+EXPORT_SYMBOL_GPL(work_on_cpu_safe);
#endif /* CONFIG_SMP */
#ifdef CONFIG_FREEZER
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index fa16c0f..e2a617e 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -130,7 +130,8 @@
nullarbor:~ # echo -n 'func svc_process -p' >
<debugfs>/dynamic_debug/control
- See Documentation/dynamic-debug-howto.txt for additional information.
+ See Documentation/admin-guide/dynamic-debug-howto.rst for additional
+ information.
endmenu # "printk and dmesg options"
@@ -356,7 +357,7 @@
bool "Compile the kernel with frame pointers"
depends on DEBUG_KERNEL && \
(CRIS || M68K || FRV || UML || \
- AVR32 || SUPERH || BLACKFIN || MN10300 || METAG) || \
+ SUPERH || BLACKFIN || MN10300 || METAG) || \
ARCH_WANT_FRAME_POINTERS
default y if (DEBUG_INFO && UML) || ARCH_WANT_FRAME_POINTERS
help
@@ -404,8 +405,8 @@
by pressing various keys while holding SysRq (Alt+PrintScreen). It
also works on a serial console (on PC hardware at least), if you
send a BREAK and then within 5 seconds a command keypress. The
- keys are documented in <file:Documentation/sysrq.txt>. Don't say Y
- unless you really know what this hack does.
+ keys are documented in <file:Documentation/admin-guide/sysrq.rst>.
+ Don't say Y unless you really know what this hack does.
config MAGIC_SYSRQ_DEFAULT_ENABLE
hex "Enable magic SysRq key functions by default"
@@ -414,7 +415,7 @@
help
Specifies which SysRq key functions are enabled by default.
This may be set to 1 or 0 to enable or disable them all, or
- to a bitmask as described in Documentation/sysrq.txt.
+ to a bitmask as described in Documentation/admin-guide/sysrq.rst.
config MAGIC_SYSRQ_SERIAL
bool "Enable magic SysRq key over serial"
diff --git a/lib/Makefile b/lib/Makefile
index 320ac46a..a155c73 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -19,7 +19,7 @@
lib-y := ctype.o string.o vsprintf.o cmdline.o \
rbtree.o radix-tree.o dump_stack.o timerqueue.o\
idr.o int_sqrt.o extable.o \
- sha1.o chacha20.o md5.o irq_regs.o argv_split.o \
+ sha1.o chacha20.o irq_regs.o argv_split.o \
flex_proportions.o ratelimit.o show_mem.o \
is_single_threaded.o plist.o decompress.o kobject_uevent.o \
earlycpio.o seq_buf.o siphash.o \
@@ -41,7 +41,7 @@
gcd.o lcm.o list_sort.o uuid.o flex_array.o iov_iter.o clz_ctz.o \
bsearch.o find_bit.o llist.o memweight.o kfifo.o \
percpu-refcount.o percpu_ida.o rhashtable.o reciprocal_div.o \
- once.o refcount.o
+ once.o refcount.o usercopy.o
obj-y += string_helpers.o
obj-$(CONFIG_TEST_STRING_HELPERS) += test-string_helpers.o
obj-y += hexdump.o
diff --git a/lib/bitmap.c b/lib/bitmap.c
index 0b66f0e..08c6ef3 100644
--- a/lib/bitmap.c
+++ b/lib/bitmap.c
@@ -502,11 +502,11 @@
* Syntax: range:used_size/group_size
* Example: 0-1023:2/256 ==> 0,1,256,257,512,513,768,769
*
- * Returns 0 on success, -errno on invalid input strings.
- * Error values:
- * %-EINVAL: second number in range smaller than first
- * %-EINVAL: invalid character in string
- * %-ERANGE: bit number specified too large for mask
+ * Returns: 0 on success, -errno on invalid input strings. Error values:
+ *
+ * - ``-EINVAL``: second number in range smaller than first
+ * - ``-EINVAL``: invalid character in string
+ * - ``-ERANGE``: bit number specified too large for mask
*/
static int __bitmap_parselist(const char *buf, unsigned int buflen,
int is_user, unsigned long *maskp,
@@ -864,14 +864,16 @@
* 11 was set in @orig had no affect on @dst.
*
* Example [2] for bitmap_fold() + bitmap_onto():
- * Let's say @relmap has these ten bits set:
+ * Let's say @relmap has these ten bits set::
+ *
* 40 41 42 43 45 48 53 61 74 95
+ *
* (for the curious, that's 40 plus the first ten terms of the
* Fibonacci sequence.)
*
* Further lets say we use the following code, invoking
* bitmap_fold() then bitmap_onto, as suggested above to
- * avoid the possibility of an empty @dst result:
+ * avoid the possibility of an empty @dst result::
*
* unsigned long *tmp; // a temporary bitmap's bits
*
@@ -882,22 +884,26 @@
* various @orig's. I list the zero-based positions of each set bit.
* The tmp column shows the intermediate result, as computed by
* using bitmap_fold() to fold the @orig bitmap modulo ten
- * (the weight of @relmap).
+ * (the weight of @relmap):
*
+ * =============== ============== =================
* @orig tmp @dst
* 0 0 40
* 1 1 41
* 9 9 95
- * 10 0 40 (*)
+ * 10 0 40 [#f1]_
* 1 3 5 7 1 3 5 7 41 43 48 61
* 0 1 2 3 4 0 1 2 3 4 40 41 42 43 45
* 0 9 18 27 0 9 8 7 40 61 74 95
* 0 10 20 30 0 40
* 0 11 22 33 0 1 2 3 40 41 42 43
* 0 12 24 36 0 2 4 6 40 42 45 53
- * 78 102 211 1 2 8 41 42 74 (*)
+ * 78 102 211 1 2 8 41 42 74 [#f1]_
+ * =============== ============== =================
*
- * (*) For these marked lines, if we hadn't first done bitmap_fold()
+ * .. [#f1]
+ *
+ * For these marked lines, if we hadn't first done bitmap_fold()
* into tmp, then the @dst result would have been empty.
*
* If either of @orig or @relmap is empty (no set bits), then @dst
diff --git a/lib/bug.c b/lib/bug.c
index 06edbbe..a6a1137 100644
--- a/lib/bug.c
+++ b/lib/bug.c
@@ -47,7 +47,7 @@
#include <linux/sched.h>
#include <linux/rculist.h>
-extern const struct bug_entry __start___bug_table[], __stop___bug_table[];
+extern struct bug_entry __start___bug_table[], __stop___bug_table[];
static inline unsigned long bug_addr(const struct bug_entry *bug)
{
@@ -62,10 +62,10 @@
/* Updates are protected by module mutex */
static LIST_HEAD(module_bug_list);
-static const struct bug_entry *module_find_bug(unsigned long bugaddr)
+static struct bug_entry *module_find_bug(unsigned long bugaddr)
{
struct module *mod;
- const struct bug_entry *bug = NULL;
+ struct bug_entry *bug = NULL;
rcu_read_lock_sched();
list_for_each_entry_rcu(mod, &module_bug_list, bug_list) {
@@ -122,15 +122,15 @@
#else
-static inline const struct bug_entry *module_find_bug(unsigned long bugaddr)
+static inline struct bug_entry *module_find_bug(unsigned long bugaddr)
{
return NULL;
}
#endif
-const struct bug_entry *find_bug(unsigned long bugaddr)
+struct bug_entry *find_bug(unsigned long bugaddr)
{
- const struct bug_entry *bug;
+ struct bug_entry *bug;
for (bug = __start___bug_table; bug < __stop___bug_table; ++bug)
if (bugaddr == bug_addr(bug))
@@ -141,9 +141,9 @@
enum bug_trap_type report_bug(unsigned long bugaddr, struct pt_regs *regs)
{
- const struct bug_entry *bug;
+ struct bug_entry *bug;
const char *file;
- unsigned line, warning;
+ unsigned line, warning, once, done;
if (!is_valid_bugaddr(bugaddr))
return BUG_TRAP_TYPE_NONE;
@@ -164,6 +164,18 @@
line = bug->line;
#endif
warning = (bug->flags & BUGFLAG_WARNING) != 0;
+ once = (bug->flags & BUGFLAG_ONCE) != 0;
+ done = (bug->flags & BUGFLAG_DONE) != 0;
+
+ if (warning && once) {
+ if (done)
+ return BUG_TRAP_TYPE_WARN;
+
+ /*
+ * Since this is the only store, concurrency is not an issue.
+ */
+ bug->flags |= BUGFLAG_DONE;
+ }
}
if (warning) {
diff --git a/lib/cmdline.c b/lib/cmdline.c
index 8f13cf7..3c6432df 100644
--- a/lib/cmdline.c
+++ b/lib/cmdline.c
@@ -15,6 +15,7 @@
#include <linux/export.h>
#include <linux/kernel.h>
#include <linux/string.h>
+#include <linux/ctype.h>
/*
* If a hyphen was found in get_option, this will handle the
@@ -189,3 +190,59 @@
return false;
}
+
+/*
+ * Parse a string to get a param value pair.
+ * You can use " around spaces, but can't escape ".
+ * Hyphens and underscores equivalent in parameter names.
+ */
+char *next_arg(char *args, char **param, char **val)
+{
+ unsigned int i, equals = 0;
+ int in_quote = 0, quoted = 0;
+ char *next;
+
+ if (*args == '"') {
+ args++;
+ in_quote = 1;
+ quoted = 1;
+ }
+
+ for (i = 0; args[i]; i++) {
+ if (isspace(args[i]) && !in_quote)
+ break;
+ if (equals == 0) {
+ if (args[i] == '=')
+ equals = i;
+ }
+ if (args[i] == '"')
+ in_quote = !in_quote;
+ }
+
+ *param = args;
+ if (!equals)
+ *val = NULL;
+ else {
+ args[equals] = '\0';
+ *val = args + equals + 1;
+
+ /* Don't include quotes in value. */
+ if (**val == '"') {
+ (*val)++;
+ if (args[i-1] == '"')
+ args[i-1] = '\0';
+ }
+ }
+ if (quoted && args[i-1] == '"')
+ args[i-1] = '\0';
+
+ if (args[i]) {
+ args[i] = '\0';
+ next = args + i + 1;
+ } else
+ next = args + i;
+
+ /* Chew up trailing spaces. */
+ return skip_spaces(next);
+ //return next;
+}
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 60abc44..4952311 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -413,7 +413,7 @@
size_t count)
{
/* It will get better. Eventually... */
- if (segment_eq(get_fs(), KERNEL_DS)) {
+ if (uaccess_kernel()) {
direction |= ITER_KVEC;
i->type = direction;
i->kvec = (struct kvec *)iov;
@@ -604,7 +604,7 @@
return 0;
}
iterate_and_advance(i, bytes, v,
- __copy_from_user_nocache((to += v.iov_len) - v.iov_len,
+ __copy_from_user_inatomic_nocache((to += v.iov_len) - v.iov_len,
v.iov_base, v.iov_len),
memcpy_from_page((to += v.bv_len) - v.bv_len, v.bv_page,
v.bv_offset, v.bv_len),
@@ -625,7 +625,7 @@
if (unlikely(i->count < bytes))
return false;
iterate_all_kinds(i, bytes, v, ({
- if (__copy_from_user_nocache((to += v.iov_len) - v.iov_len,
+ if (__copy_from_user_inatomic_nocache((to += v.iov_len) - v.iov_len,
v.iov_base, v.iov_len))
return false;
0;}),
@@ -798,7 +798,7 @@
while (1) {
size_t n = off - pipe->bufs[idx].offset;
if (unroll < n) {
- off -= (n - unroll);
+ off -= unroll;
break;
}
unroll -= n;
diff --git a/lib/kobject.c b/lib/kobject.c
index 445dcae..763d70a 100644
--- a/lib/kobject.c
+++ b/lib/kobject.c
@@ -601,12 +601,15 @@
}
EXPORT_SYMBOL(kobject_get);
-static struct kobject * __must_check kobject_get_unless_zero(struct kobject *kobj)
+struct kobject * __must_check kobject_get_unless_zero(struct kobject *kobj)
{
+ if (!kobj)
+ return NULL;
if (!kref_get_unless_zero(&kobj->kref))
kobj = NULL;
return kobj;
}
+EXPORT_SYMBOL(kobject_get_unless_zero);
/*
* kobject_cleanup - free kobject resources.
diff --git a/lib/md5.c b/lib/md5.c
deleted file mode 100644
index bb0cd01..0000000
--- a/lib/md5.c
+++ /dev/null
@@ -1,95 +0,0 @@
-#include <linux/compiler.h>
-#include <linux/export.h>
-#include <linux/cryptohash.h>
-
-#define F1(x, y, z) (z ^ (x & (y ^ z)))
-#define F2(x, y, z) F1(z, x, y)
-#define F3(x, y, z) (x ^ y ^ z)
-#define F4(x, y, z) (y ^ (x | ~z))
-
-#define MD5STEP(f, w, x, y, z, in, s) \
- (w += f(x, y, z) + in, w = (w<<s | w>>(32-s)) + x)
-
-void md5_transform(__u32 *hash, __u32 const *in)
-{
- u32 a, b, c, d;
-
- a = hash[0];
- b = hash[1];
- c = hash[2];
- d = hash[3];
-
- MD5STEP(F1, a, b, c, d, in[0] + 0xd76aa478, 7);
- MD5STEP(F1, d, a, b, c, in[1] + 0xe8c7b756, 12);
- MD5STEP(F1, c, d, a, b, in[2] + 0x242070db, 17);
- MD5STEP(F1, b, c, d, a, in[3] + 0xc1bdceee, 22);
- MD5STEP(F1, a, b, c, d, in[4] + 0xf57c0faf, 7);
- MD5STEP(F1, d, a, b, c, in[5] + 0x4787c62a, 12);
- MD5STEP(F1, c, d, a, b, in[6] + 0xa8304613, 17);
- MD5STEP(F1, b, c, d, a, in[7] + 0xfd469501, 22);
- MD5STEP(F1, a, b, c, d, in[8] + 0x698098d8, 7);
- MD5STEP(F1, d, a, b, c, in[9] + 0x8b44f7af, 12);
- MD5STEP(F1, c, d, a, b, in[10] + 0xffff5bb1, 17);
- MD5STEP(F1, b, c, d, a, in[11] + 0x895cd7be, 22);
- MD5STEP(F1, a, b, c, d, in[12] + 0x6b901122, 7);
- MD5STEP(F1, d, a, b, c, in[13] + 0xfd987193, 12);
- MD5STEP(F1, c, d, a, b, in[14] + 0xa679438e, 17);
- MD5STEP(F1, b, c, d, a, in[15] + 0x49b40821, 22);
-
- MD5STEP(F2, a, b, c, d, in[1] + 0xf61e2562, 5);
- MD5STEP(F2, d, a, b, c, in[6] + 0xc040b340, 9);
- MD5STEP(F2, c, d, a, b, in[11] + 0x265e5a51, 14);
- MD5STEP(F2, b, c, d, a, in[0] + 0xe9b6c7aa, 20);
- MD5STEP(F2, a, b, c, d, in[5] + 0xd62f105d, 5);
- MD5STEP(F2, d, a, b, c, in[10] + 0x02441453, 9);
- MD5STEP(F2, c, d, a, b, in[15] + 0xd8a1e681, 14);
- MD5STEP(F2, b, c, d, a, in[4] + 0xe7d3fbc8, 20);
- MD5STEP(F2, a, b, c, d, in[9] + 0x21e1cde6, 5);
- MD5STEP(F2, d, a, b, c, in[14] + 0xc33707d6, 9);
- MD5STEP(F2, c, d, a, b, in[3] + 0xf4d50d87, 14);
- MD5STEP(F2, b, c, d, a, in[8] + 0x455a14ed, 20);
- MD5STEP(F2, a, b, c, d, in[13] + 0xa9e3e905, 5);
- MD5STEP(F2, d, a, b, c, in[2] + 0xfcefa3f8, 9);
- MD5STEP(F2, c, d, a, b, in[7] + 0x676f02d9, 14);
- MD5STEP(F2, b, c, d, a, in[12] + 0x8d2a4c8a, 20);
-
- MD5STEP(F3, a, b, c, d, in[5] + 0xfffa3942, 4);
- MD5STEP(F3, d, a, b, c, in[8] + 0x8771f681, 11);
- MD5STEP(F3, c, d, a, b, in[11] + 0x6d9d6122, 16);
- MD5STEP(F3, b, c, d, a, in[14] + 0xfde5380c, 23);
- MD5STEP(F3, a, b, c, d, in[1] + 0xa4beea44, 4);
- MD5STEP(F3, d, a, b, c, in[4] + 0x4bdecfa9, 11);
- MD5STEP(F3, c, d, a, b, in[7] + 0xf6bb4b60, 16);
- MD5STEP(F3, b, c, d, a, in[10] + 0xbebfbc70, 23);
- MD5STEP(F3, a, b, c, d, in[13] + 0x289b7ec6, 4);
- MD5STEP(F3, d, a, b, c, in[0] + 0xeaa127fa, 11);
- MD5STEP(F3, c, d, a, b, in[3] + 0xd4ef3085, 16);
- MD5STEP(F3, b, c, d, a, in[6] + 0x04881d05, 23);
- MD5STEP(F3, a, b, c, d, in[9] + 0xd9d4d039, 4);
- MD5STEP(F3, d, a, b, c, in[12] + 0xe6db99e5, 11);
- MD5STEP(F3, c, d, a, b, in[15] + 0x1fa27cf8, 16);
- MD5STEP(F3, b, c, d, a, in[2] + 0xc4ac5665, 23);
-
- MD5STEP(F4, a, b, c, d, in[0] + 0xf4292244, 6);
- MD5STEP(F4, d, a, b, c, in[7] + 0x432aff97, 10);
- MD5STEP(F4, c, d, a, b, in[14] + 0xab9423a7, 15);
- MD5STEP(F4, b, c, d, a, in[5] + 0xfc93a039, 21);
- MD5STEP(F4, a, b, c, d, in[12] + 0x655b59c3, 6);
- MD5STEP(F4, d, a, b, c, in[3] + 0x8f0ccc92, 10);
- MD5STEP(F4, c, d, a, b, in[10] + 0xffeff47d, 15);
- MD5STEP(F4, b, c, d, a, in[1] + 0x85845dd1, 21);
- MD5STEP(F4, a, b, c, d, in[8] + 0x6fa87e4f, 6);
- MD5STEP(F4, d, a, b, c, in[15] + 0xfe2ce6e0, 10);
- MD5STEP(F4, c, d, a, b, in[6] + 0xa3014314, 15);
- MD5STEP(F4, b, c, d, a, in[13] + 0x4e0811a1, 21);
- MD5STEP(F4, a, b, c, d, in[4] + 0xf7537e82, 6);
- MD5STEP(F4, d, a, b, c, in[11] + 0xbd3af235, 10);
- MD5STEP(F4, c, d, a, b, in[2] + 0x2ad7d2bb, 15);
- MD5STEP(F4, b, c, d, a, in[9] + 0xeb86d391, 21);
-
- hash[0] += a;
- hash[1] += b;
- hash[2] += c;
- hash[3] += d;
-}
-EXPORT_SYMBOL(md5_transform);
diff --git a/lib/nlattr.c b/lib/nlattr.c
index b42b857..a7e0b16 100644
--- a/lib/nlattr.c
+++ b/lib/nlattr.c
@@ -112,6 +112,7 @@
* @len: length of attribute stream
* @maxtype: maximum attribute type to be expected
* @policy: validation policy
+ * @extack: extended ACK report struct
*
* Validates all attributes in the specified attribute stream against the
* specified policy. Attributes with a type exceeding maxtype will be
@@ -120,20 +121,23 @@
* Returns 0 on success or a negative error code.
*/
int nla_validate(const struct nlattr *head, int len, int maxtype,
- const struct nla_policy *policy)
+ const struct nla_policy *policy,
+ struct netlink_ext_ack *extack)
{
const struct nlattr *nla;
- int rem, err;
+ int rem;
nla_for_each_attr(nla, head, len, rem) {
- err = validate_nla(nla, maxtype, policy);
- if (err < 0)
- goto errout;
+ int err = validate_nla(nla, maxtype, policy);
+
+ if (err < 0) {
+ if (extack)
+ extack->bad_attr = nla;
+ return err;
+ }
}
- err = 0;
-errout:
- return err;
+ return 0;
}
EXPORT_SYMBOL(nla_validate);
@@ -180,7 +184,8 @@
* Returns 0 on success or a negative error code.
*/
int nla_parse(struct nlattr **tb, int maxtype, const struct nlattr *head,
- int len, const struct nla_policy *policy)
+ int len, const struct nla_policy *policy,
+ struct netlink_ext_ack *extack)
{
const struct nlattr *nla;
int rem, err;
@@ -193,8 +198,11 @@
if (type > 0 && type <= maxtype) {
if (policy) {
err = validate_nla(nla, maxtype, policy);
- if (err < 0)
+ if (err < 0) {
+ if (extack)
+ extack->bad_attr = nla;
goto errout;
+ }
}
tb[type] = (struct nlattr *)nla;
diff --git a/lib/refcount.c b/lib/refcount.c
index aa09ad3..f42124c 100644
--- a/lib/refcount.c
+++ b/lib/refcount.c
@@ -37,11 +37,29 @@
#include <linux/refcount.h>
#include <linux/bug.h>
+/**
+ * refcount_add_not_zero - add a value to a refcount unless it is 0
+ * @i: the value to add to the refcount
+ * @r: the refcount
+ *
+ * Will saturate at UINT_MAX and WARN.
+ *
+ * Provides no memory ordering, it is assumed the caller has guaranteed the
+ * object memory to be stable (RCU, etc.). It does provide a control dependency
+ * and thereby orders future stores. See the comment on top.
+ *
+ * Use of this function is not recommended for the normal reference counting
+ * use case in which references are taken and released one at a time. In these
+ * cases, refcount_inc(), or one of its variants, should instead be used to
+ * increment a reference count.
+ *
+ * Return: false if the passed refcount is 0, true otherwise
+ */
bool refcount_add_not_zero(unsigned int i, refcount_t *r)
{
- unsigned int old, new, val = atomic_read(&r->refs);
+ unsigned int new, val = atomic_read(&r->refs);
- for (;;) {
+ do {
if (!val)
return false;
@@ -51,12 +69,8 @@
new = val + i;
if (new < val)
new = UINT_MAX;
- old = atomic_cmpxchg_relaxed(&r->refs, val, new);
- if (old == val)
- break;
- val = old;
- }
+ } while (!atomic_try_cmpxchg_relaxed(&r->refs, &val, new));
WARN_ONCE(new == UINT_MAX, "refcount_t: saturated; leaking memory.\n");
@@ -64,24 +78,45 @@
}
EXPORT_SYMBOL_GPL(refcount_add_not_zero);
+/**
+ * refcount_add - add a value to a refcount
+ * @i: the value to add to the refcount
+ * @r: the refcount
+ *
+ * Similar to atomic_add(), but will saturate at UINT_MAX and WARN.
+ *
+ * Provides no memory ordering, it is assumed the caller has guaranteed the
+ * object memory to be stable (RCU, etc.). It does provide a control dependency
+ * and thereby orders future stores. See the comment on top.
+ *
+ * Use of this function is not recommended for the normal reference counting
+ * use case in which references are taken and released one at a time. In these
+ * cases, refcount_inc(), or one of its variants, should instead be used to
+ * increment a reference count.
+ */
void refcount_add(unsigned int i, refcount_t *r)
{
WARN_ONCE(!refcount_add_not_zero(i, r), "refcount_t: addition on 0; use-after-free.\n");
}
EXPORT_SYMBOL_GPL(refcount_add);
-/*
- * Similar to atomic_inc_not_zero(), will saturate at UINT_MAX and WARN.
+/**
+ * refcount_inc_not_zero - increment a refcount unless it is 0
+ * @r: the refcount to increment
+ *
+ * Similar to atomic_inc_not_zero(), but will saturate at UINT_MAX and WARN.
*
* Provides no memory ordering, it is assumed the caller has guaranteed the
* object memory to be stable (RCU, etc.). It does provide a control dependency
* and thereby orders future stores. See the comment on top.
+ *
+ * Return: true if the increment was successful, false otherwise
*/
bool refcount_inc_not_zero(refcount_t *r)
{
- unsigned int old, new, val = atomic_read(&r->refs);
+ unsigned int new, val = atomic_read(&r->refs);
- for (;;) {
+ do {
new = val + 1;
if (!val)
@@ -90,12 +125,7 @@
if (unlikely(!new))
return true;
- old = atomic_cmpxchg_relaxed(&r->refs, val, new);
- if (old == val)
- break;
-
- val = old;
- }
+ } while (!atomic_try_cmpxchg_relaxed(&r->refs, &val, new));
WARN_ONCE(new == UINT_MAX, "refcount_t: saturated; leaking memory.\n");
@@ -103,11 +133,17 @@
}
EXPORT_SYMBOL_GPL(refcount_inc_not_zero);
-/*
- * Similar to atomic_inc(), will saturate at UINT_MAX and WARN.
+/**
+ * refcount_inc - increment a refcount
+ * @r: the refcount to increment
+ *
+ * Similar to atomic_inc(), but will saturate at UINT_MAX and WARN.
*
* Provides no memory ordering, it is assumed the caller already has a
- * reference on the object, will WARN when this is not so.
+ * reference on the object.
+ *
+ * Will WARN if the refcount is 0, as this represents a possible use-after-free
+ * condition.
*/
void refcount_inc(refcount_t *r)
{
@@ -115,11 +151,31 @@
}
EXPORT_SYMBOL_GPL(refcount_inc);
+/**
+ * refcount_sub_and_test - subtract from a refcount and test if it is 0
+ * @i: amount to subtract from the refcount
+ * @r: the refcount
+ *
+ * Similar to atomic_dec_and_test(), but it will WARN, return false and
+ * ultimately leak on underflow and will fail to decrement when saturated
+ * at UINT_MAX.
+ *
+ * Provides release memory ordering, such that prior loads and stores are done
+ * before, and provides a control dependency such that free() must come after.
+ * See the comment on top.
+ *
+ * Use of this function is not recommended for the normal reference counting
+ * use case in which references are taken and released one at a time. In these
+ * cases, refcount_dec(), or one of its variants, should instead be used to
+ * decrement a reference count.
+ *
+ * Return: true if the resulting refcount is 0, false otherwise
+ */
bool refcount_sub_and_test(unsigned int i, refcount_t *r)
{
- unsigned int old, new, val = atomic_read(&r->refs);
+ unsigned int new, val = atomic_read(&r->refs);
- for (;;) {
+ do {
if (unlikely(val == UINT_MAX))
return false;
@@ -129,24 +185,24 @@
return false;
}
- old = atomic_cmpxchg_release(&r->refs, val, new);
- if (old == val)
- break;
-
- val = old;
- }
+ } while (!atomic_try_cmpxchg_release(&r->refs, &val, new));
return !new;
}
EXPORT_SYMBOL_GPL(refcount_sub_and_test);
-/*
+/**
+ * refcount_dec_and_test - decrement a refcount and test if it is 0
+ * @r: the refcount
+ *
* Similar to atomic_dec_and_test(), it will WARN on underflow and fail to
* decrement when saturated at UINT_MAX.
*
* Provides release memory ordering, such that prior loads and stores are done
* before, and provides a control dependency such that free() must come after.
* See the comment on top.
+ *
+ * Return: true if the resulting refcount is 0, false otherwise
*/
bool refcount_dec_and_test(refcount_t *r)
{
@@ -154,21 +210,26 @@
}
EXPORT_SYMBOL_GPL(refcount_dec_and_test);
-/*
+/**
+ * refcount_dec - decrement a refcount
+ * @r: the refcount
+ *
* Similar to atomic_dec(), it will WARN on underflow and fail to decrement
* when saturated at UINT_MAX.
*
* Provides release memory ordering, such that prior loads and stores are done
* before.
*/
-
void refcount_dec(refcount_t *r)
{
WARN_ONCE(refcount_dec_and_test(r), "refcount_t: decrement hit 0; leaking memory.\n");
}
EXPORT_SYMBOL_GPL(refcount_dec);
-/*
+/**
+ * refcount_dec_if_one - decrement a refcount if it is 1
+ * @r: the refcount
+ *
* No atomic_t counterpart, it attempts a 1 -> 0 transition and returns the
* success thereof.
*
@@ -178,24 +239,33 @@
* It can be used like a try-delete operator; this explicit case is provided
* and not cmpxchg in generic, because that would allow implementing unsafe
* operations.
+ *
+ * Return: true if the resulting refcount is 0, false otherwise
*/
bool refcount_dec_if_one(refcount_t *r)
{
- return atomic_cmpxchg_release(&r->refs, 1, 0) == 1;
+ int val = 1;
+
+ return atomic_try_cmpxchg_release(&r->refs, &val, 0);
}
EXPORT_SYMBOL_GPL(refcount_dec_if_one);
-/*
+/**
+ * refcount_dec_not_one - decrement a refcount if it is not 1
+ * @r: the refcount
+ *
* No atomic_t counterpart, it decrements unless the value is 1, in which case
* it will return false.
*
* Was often done like: atomic_add_unless(&var, -1, 1)
+ *
+ * Return: true if the decrement operation was successful, false otherwise
*/
bool refcount_dec_not_one(refcount_t *r)
{
- unsigned int old, new, val = atomic_read(&r->refs);
+ unsigned int new, val = atomic_read(&r->refs);
- for (;;) {
+ do {
if (unlikely(val == UINT_MAX))
return true;
@@ -208,24 +278,27 @@
return true;
}
- old = atomic_cmpxchg_release(&r->refs, val, new);
- if (old == val)
- break;
-
- val = old;
- }
+ } while (!atomic_try_cmpxchg_release(&r->refs, &val, new));
return true;
}
EXPORT_SYMBOL_GPL(refcount_dec_not_one);
-/*
+/**
+ * refcount_dec_and_mutex_lock - return holding mutex if able to decrement
+ * refcount to 0
+ * @r: the refcount
+ * @lock: the mutex to be locked
+ *
* Similar to atomic_dec_and_mutex_lock(), it will WARN on underflow and fail
* to decrement when saturated at UINT_MAX.
*
* Provides release memory ordering, such that prior loads and stores are done
* before, and provides a control dependency such that free() must come after.
* See the comment on top.
+ *
+ * Return: true and hold mutex if able to decrement refcount to 0, false
+ * otherwise
*/
bool refcount_dec_and_mutex_lock(refcount_t *r, struct mutex *lock)
{
@@ -242,13 +315,21 @@
}
EXPORT_SYMBOL_GPL(refcount_dec_and_mutex_lock);
-/*
+/**
+ * refcount_dec_and_lock - return holding spinlock if able to decrement
+ * refcount to 0
+ * @r: the refcount
+ * @lock: the spinlock to be locked
+ *
* Similar to atomic_dec_and_lock(), it will WARN on underflow and fail to
* decrement when saturated at UINT_MAX.
*
* Provides release memory ordering, such that prior loads and stores are done
* before, and provides a control dependency such that free() must come after.
* See the comment on top.
+ *
+ * Return: true and hold spinlock if able to decrement refcount to 0, false
+ * otherwise
*/
bool refcount_dec_and_lock(refcount_t *r, spinlock_t *lock)
{
diff --git a/lib/rhashtable.c b/lib/rhashtable.c
index f8635fd..a930e43 100644
--- a/lib/rhashtable.c
+++ b/lib/rhashtable.c
@@ -535,7 +535,7 @@
struct rhash_head *head;
int elasticity;
- elasticity = ht->elasticity;
+ elasticity = RHT_ELASTICITY;
pprev = rht_bucket_var(tbl, hash);
rht_for_each_continue(head, *pprev, tbl, hash) {
struct rhlist_head *list;
@@ -958,35 +958,20 @@
if (params->min_size)
ht->p.min_size = roundup_pow_of_two(params->min_size);
- if (params->max_size)
+ /* Cap total entries at 2^31 to avoid nelems overflow. */
+ ht->max_elems = 1u << 31;
+
+ if (params->max_size) {
ht->p.max_size = rounddown_pow_of_two(params->max_size);
+ if (ht->p.max_size < ht->max_elems / 2)
+ ht->max_elems = ht->p.max_size * 2;
+ }
- if (params->insecure_max_entries)
- ht->p.insecure_max_entries =
- rounddown_pow_of_two(params->insecure_max_entries);
- else
- ht->p.insecure_max_entries = ht->p.max_size * 2;
-
- ht->p.min_size = max(ht->p.min_size, HASH_MIN_SIZE);
+ ht->p.min_size = max_t(u16, ht->p.min_size, HASH_MIN_SIZE);
if (params->nelem_hint)
size = rounded_hashtable_size(&ht->p);
- /* The maximum (not average) chain length grows with the
- * size of the hash table, at a rate of (log N)/(log log N).
- * The value of 16 is selected so that even if the hash
- * table grew to 2^32 you would not expect the maximum
- * chain length to exceed it unless we are under attack
- * (or extremely unlucky).
- *
- * As this limit is only to detect attacks, we don't need
- * to set it to a lower value as you'd need the chain
- * length to vastly exceed 16 to have any real effect
- * on the system.
- */
- if (!params->insecure_elasticity)
- ht->elasticity = 16;
-
if (params->locks_mul)
ht->p.locks_mul = roundup_pow_of_two(params->locks_mul);
else
diff --git a/lib/sbitmap.c b/lib/sbitmap.c
index 60e800e..80aa8d5 100644
--- a/lib/sbitmap.c
+++ b/lib/sbitmap.c
@@ -79,15 +79,15 @@
}
EXPORT_SYMBOL_GPL(sbitmap_resize);
-static int __sbitmap_get_word(struct sbitmap_word *word, unsigned int hint,
- bool wrap)
+static int __sbitmap_get_word(unsigned long *word, unsigned long depth,
+ unsigned int hint, bool wrap)
{
unsigned int orig_hint = hint;
int nr;
while (1) {
- nr = find_next_zero_bit(&word->word, word->depth, hint);
- if (unlikely(nr >= word->depth)) {
+ nr = find_next_zero_bit(word, depth, hint);
+ if (unlikely(nr >= depth)) {
/*
* We started with an offset, and we didn't reset the
* offset to 0 in a failure case, so start from 0 to
@@ -100,11 +100,11 @@
return -1;
}
- if (!test_and_set_bit(nr, &word->word))
+ if (!test_and_set_bit(nr, word))
break;
hint = nr + 1;
- if (hint >= word->depth - 1)
+ if (hint >= depth - 1)
hint = 0;
}
@@ -119,7 +119,8 @@
index = SB_NR_TO_INDEX(sb, alloc_hint);
for (i = 0; i < sb->map_nr; i++) {
- nr = __sbitmap_get_word(&sb->map[index],
+ nr = __sbitmap_get_word(&sb->map[index].word,
+ sb->map[index].depth,
SB_NR_TO_BIT(sb, alloc_hint),
!round_robin);
if (nr != -1) {
@@ -141,6 +142,37 @@
}
EXPORT_SYMBOL_GPL(sbitmap_get);
+int sbitmap_get_shallow(struct sbitmap *sb, unsigned int alloc_hint,
+ unsigned long shallow_depth)
+{
+ unsigned int i, index;
+ int nr = -1;
+
+ index = SB_NR_TO_INDEX(sb, alloc_hint);
+
+ for (i = 0; i < sb->map_nr; i++) {
+ nr = __sbitmap_get_word(&sb->map[index].word,
+ min(sb->map[index].depth, shallow_depth),
+ SB_NR_TO_BIT(sb, alloc_hint), true);
+ if (nr != -1) {
+ nr += index << sb->shift;
+ break;
+ }
+
+ /* Jump to next index. */
+ index++;
+ alloc_hint = index << sb->shift;
+
+ if (index >= sb->map_nr) {
+ index = 0;
+ alloc_hint = 0;
+ }
+ }
+
+ return nr;
+}
+EXPORT_SYMBOL_GPL(sbitmap_get_shallow);
+
bool sbitmap_any_bit_set(const struct sbitmap *sb)
{
unsigned int i;
@@ -342,6 +374,35 @@
}
EXPORT_SYMBOL_GPL(__sbitmap_queue_get);
+int __sbitmap_queue_get_shallow(struct sbitmap_queue *sbq,
+ unsigned int shallow_depth)
+{
+ unsigned int hint, depth;
+ int nr;
+
+ hint = this_cpu_read(*sbq->alloc_hint);
+ depth = READ_ONCE(sbq->sb.depth);
+ if (unlikely(hint >= depth)) {
+ hint = depth ? prandom_u32() % depth : 0;
+ this_cpu_write(*sbq->alloc_hint, hint);
+ }
+ nr = sbitmap_get_shallow(&sbq->sb, hint, shallow_depth);
+
+ if (nr == -1) {
+ /* If the map is full, a hint won't do us much good. */
+ this_cpu_write(*sbq->alloc_hint, 0);
+ } else if (nr == hint || unlikely(sbq->round_robin)) {
+ /* Only update the hint if we used it. */
+ hint = nr + 1;
+ if (hint >= depth - 1)
+ hint = 0;
+ this_cpu_write(*sbq->alloc_hint, hint);
+ }
+
+ return nr;
+}
+EXPORT_SYMBOL_GPL(__sbitmap_queue_get_shallow);
+
static struct sbq_wait_state *sbq_wake_ptr(struct sbitmap_queue *sbq)
{
int i, wake_index;
diff --git a/lib/string.c b/lib/string.c
index ed83562..b5c9a11 100644
--- a/lib/string.c
+++ b/lib/string.c
@@ -131,7 +131,7 @@
* @src: Where to copy the string from
* @size: size of destination buffer
*
- * Compatible with *BSD: the result is always a valid
+ * Compatible with ``*BSD``: the result is always a valid
* NUL-terminated string that fits in the buffer (unless,
* of course, the buffer size is zero). It does not pad
* out the result like strncpy() does.
diff --git a/lib/test_bpf.c b/lib/test_bpf.c
index 0362da0..889bc31 100644
--- a/lib/test_bpf.c
+++ b/lib/test_bpf.c
@@ -434,6 +434,41 @@
return 0;
}
+static int __bpf_fill_stxdw(struct bpf_test *self, int size)
+{
+ unsigned int len = BPF_MAXINSNS;
+ struct bpf_insn *insn;
+ int i;
+
+ insn = kmalloc_array(len, sizeof(*insn), GFP_KERNEL);
+ if (!insn)
+ return -ENOMEM;
+
+ insn[0] = BPF_ALU32_IMM(BPF_MOV, R0, 1);
+ insn[1] = BPF_ST_MEM(size, R10, -40, 42);
+
+ for (i = 2; i < len - 2; i++)
+ insn[i] = BPF_STX_XADD(size, R10, R0, -40);
+
+ insn[len - 2] = BPF_LDX_MEM(size, R0, R10, -40);
+ insn[len - 1] = BPF_EXIT_INSN();
+
+ self->u.ptr.insns = insn;
+ self->u.ptr.len = len;
+
+ return 0;
+}
+
+static int bpf_fill_stxw(struct bpf_test *self)
+{
+ return __bpf_fill_stxdw(self, BPF_W);
+}
+
+static int bpf_fill_stxdw(struct bpf_test *self)
+{
+ return __bpf_fill_stxdw(self, BPF_DW);
+}
+
static struct bpf_test tests[] = {
{
"TAX",
@@ -4303,6 +4338,41 @@
{ { 0, 0x22 } },
},
{
+ "STX_XADD_W: Test side-effects, r10: 0x12 + 0x10 = 0x22",
+ .u.insns_int = {
+ BPF_ALU64_REG(BPF_MOV, R1, R10),
+ BPF_ALU32_IMM(BPF_MOV, R0, 0x12),
+ BPF_ST_MEM(BPF_W, R10, -40, 0x10),
+ BPF_STX_XADD(BPF_W, R10, R0, -40),
+ BPF_ALU64_REG(BPF_MOV, R0, R10),
+ BPF_ALU64_REG(BPF_SUB, R0, R1),
+ BPF_EXIT_INSN(),
+ },
+ INTERNAL,
+ { },
+ { { 0, 0 } },
+ },
+ {
+ "STX_XADD_W: Test side-effects, r0: 0x12 + 0x10 = 0x22",
+ .u.insns_int = {
+ BPF_ALU32_IMM(BPF_MOV, R0, 0x12),
+ BPF_ST_MEM(BPF_W, R10, -40, 0x10),
+ BPF_STX_XADD(BPF_W, R10, R0, -40),
+ BPF_EXIT_INSN(),
+ },
+ INTERNAL,
+ { },
+ { { 0, 0x12 } },
+ },
+ {
+ "STX_XADD_W: X + 1 + 1 + 1 + ...",
+ { },
+ INTERNAL,
+ { },
+ { { 0, 4134 } },
+ .fill_helper = bpf_fill_stxw,
+ },
+ {
"STX_XADD_DW: Test: 0x12 + 0x10 = 0x22",
.u.insns_int = {
BPF_ALU32_IMM(BPF_MOV, R0, 0x12),
@@ -4315,6 +4385,41 @@
{ },
{ { 0, 0x22 } },
},
+ {
+ "STX_XADD_DW: Test side-effects, r10: 0x12 + 0x10 = 0x22",
+ .u.insns_int = {
+ BPF_ALU64_REG(BPF_MOV, R1, R10),
+ BPF_ALU32_IMM(BPF_MOV, R0, 0x12),
+ BPF_ST_MEM(BPF_DW, R10, -40, 0x10),
+ BPF_STX_XADD(BPF_DW, R10, R0, -40),
+ BPF_ALU64_REG(BPF_MOV, R0, R10),
+ BPF_ALU64_REG(BPF_SUB, R0, R1),
+ BPF_EXIT_INSN(),
+ },
+ INTERNAL,
+ { },
+ { { 0, 0 } },
+ },
+ {
+ "STX_XADD_DW: Test side-effects, r0: 0x12 + 0x10 = 0x22",
+ .u.insns_int = {
+ BPF_ALU32_IMM(BPF_MOV, R0, 0x12),
+ BPF_ST_MEM(BPF_DW, R10, -40, 0x10),
+ BPF_STX_XADD(BPF_DW, R10, R0, -40),
+ BPF_EXIT_INSN(),
+ },
+ INTERNAL,
+ { },
+ { { 0, 0x12 } },
+ },
+ {
+ "STX_XADD_DW: X + 1 + 1 + 1 + ...",
+ { },
+ INTERNAL,
+ { },
+ { { 0, 4134 } },
+ .fill_helper = bpf_fill_stxdw,
+ },
/* BPF_JMP | BPF_EXIT */
{
"JMP_EXIT",
@@ -4656,6 +4761,51 @@
{ },
{ { 0, 1 } },
},
+ {
+ /* Mainly testing JIT + imm64 here. */
+ "JMP_JGE_X: ldimm64 test 1",
+ .u.insns_int = {
+ BPF_ALU32_IMM(BPF_MOV, R0, 0),
+ BPF_LD_IMM64(R1, 3),
+ BPF_LD_IMM64(R2, 2),
+ BPF_JMP_REG(BPF_JGE, R1, R2, 2),
+ BPF_LD_IMM64(R0, 0xffffffffffffffffULL),
+ BPF_LD_IMM64(R0, 0xeeeeeeeeeeeeeeeeULL),
+ BPF_EXIT_INSN(),
+ },
+ INTERNAL,
+ { },
+ { { 0, 0xeeeeeeeeU } },
+ },
+ {
+ "JMP_JGE_X: ldimm64 test 2",
+ .u.insns_int = {
+ BPF_ALU32_IMM(BPF_MOV, R0, 0),
+ BPF_LD_IMM64(R1, 3),
+ BPF_LD_IMM64(R2, 2),
+ BPF_JMP_REG(BPF_JGE, R1, R2, 0),
+ BPF_LD_IMM64(R0, 0xffffffffffffffffULL),
+ BPF_EXIT_INSN(),
+ },
+ INTERNAL,
+ { },
+ { { 0, 0xffffffffU } },
+ },
+ {
+ "JMP_JGE_X: ldimm64 test 3",
+ .u.insns_int = {
+ BPF_ALU32_IMM(BPF_MOV, R0, 1),
+ BPF_LD_IMM64(R1, 3),
+ BPF_LD_IMM64(R2, 2),
+ BPF_JMP_REG(BPF_JGE, R1, R2, 4),
+ BPF_LD_IMM64(R0, 0xffffffffffffffffULL),
+ BPF_LD_IMM64(R0, 0xeeeeeeeeeeeeeeeeULL),
+ BPF_EXIT_INSN(),
+ },
+ INTERNAL,
+ { },
+ { { 0, 1 } },
+ },
/* BPF_JMP | BPF_JNE | BPF_X */
{
"JMP_JNE_X: if (3 != 2) return 1",
diff --git a/lib/test_user_copy.c b/lib/test_user_copy.c
index 1a8d71a..4621db8 100644
--- a/lib/test_user_copy.c
+++ b/lib/test_user_copy.c
@@ -31,7 +31,6 @@
* their capability at compile-time, we just have to opt-out certain archs.
*/
#if BITS_PER_LONG == 64 || (!(defined(CONFIG_ARM) && !defined(MMU)) && \
- !defined(CONFIG_AVR32) && \
!defined(CONFIG_BLACKFIN) && \
!defined(CONFIG_M32R) && \
!defined(CONFIG_M68K) && \
diff --git a/lib/usercopy.c b/lib/usercopy.c
new file mode 100644
index 0000000..1b6010a
--- /dev/null
+++ b/lib/usercopy.c
@@ -0,0 +1,26 @@
+#include <linux/uaccess.h>
+
+/* out-of-line parts */
+
+#ifndef INLINE_COPY_FROM_USER
+unsigned long _copy_from_user(void *to, const void __user *from, unsigned long n)
+{
+ unsigned long res = n;
+ if (likely(access_ok(VERIFY_READ, from, n)))
+ res = raw_copy_from_user(to, from, n);
+ if (unlikely(res))
+ memset(to + (n - res), 0, res);
+ return res;
+}
+EXPORT_SYMBOL(_copy_from_user);
+#endif
+
+#ifndef INLINE_COPY_TO_USER
+unsigned long _copy_to_user(void *to, const void __user *from, unsigned long n)
+{
+ if (likely(access_ok(VERIFY_WRITE, to, n)))
+ n = raw_copy_to_user(to, from, n);
+ return n;
+}
+EXPORT_SYMBOL(_copy_to_user);
+#endif
diff --git a/lib/vsprintf.c b/lib/vsprintf.c
index e3bf4e0..176641c 100644
--- a/lib/vsprintf.c
+++ b/lib/vsprintf.c
@@ -1954,13 +1954,13 @@
* This function generally follows C99 vsnprintf, but has some
* extensions and a few limitations:
*
- * %n is unsupported
- * %p* is handled by pointer()
+ * - ``%n`` is unsupported
+ * - ``%p*`` is handled by pointer()
*
* See pointer() or Documentation/printk-formats.txt for more
* extensive description.
*
- * ** Please update the documentation in both places when making changes **
+ * **Please update the documentation in both places when making changes**
*
* The return value is the number of characters which would
* be generated for the given input, excluding the trailing
diff --git a/mm/Kconfig b/mm/Kconfig
index 9b8fccb..beb7a45 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -312,7 +312,6 @@
config NR_QUICK
int
depends on QUICKLIST
- default "2" if AVR32
default "1"
config VIRT_TO_BUS
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index c6f2a37..f028a9a 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -12,8 +12,6 @@
#include <linux/device.h>
#include <trace/events/writeback.h>
-static atomic_long_t bdi_seq = ATOMIC_LONG_INIT(0);
-
struct backing_dev_info noop_backing_dev_info = {
.name = "noop",
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
@@ -242,6 +240,8 @@
}
postcore_initcall(bdi_class_init);
+static int bdi_init(struct backing_dev_info *bdi);
+
static int __init default_bdi_init(void)
{
int err;
@@ -294,6 +294,8 @@
memset(wb, 0, sizeof(*wb));
+ if (wb != &bdi->wb)
+ bdi_get(bdi);
wb->bdi = bdi;
wb->last_old_flush = jiffies;
INIT_LIST_HEAD(&wb->b_dirty);
@@ -314,8 +316,10 @@
wb->dirty_sleep = jiffies;
wb->congested = wb_congested_get_create(bdi, blkcg_id, gfp);
- if (!wb->congested)
- return -ENOMEM;
+ if (!wb->congested) {
+ err = -ENOMEM;
+ goto out_put_bdi;
+ }
err = fprop_local_init_percpu(&wb->completions, gfp);
if (err)
@@ -335,9 +339,14 @@
fprop_local_destroy_percpu(&wb->completions);
out_put_cong:
wb_congested_put(wb->congested);
+out_put_bdi:
+ if (wb != &bdi->wb)
+ bdi_put(bdi);
return err;
}
+static void cgwb_remove_from_bdi_list(struct bdi_writeback *wb);
+
/*
* Remove bdi from the global list and shutdown any threads we have running
*/
@@ -347,10 +356,18 @@
spin_lock_bh(&wb->work_lock);
if (!test_and_clear_bit(WB_registered, &wb->state)) {
spin_unlock_bh(&wb->work_lock);
+ /*
+ * Wait for wb shutdown to finish if someone else is just
+ * running wb_shutdown(). Otherwise we could proceed to wb /
+ * bdi destruction before wb_shutdown() is finished.
+ */
+ wait_on_bit(&wb->state, WB_shutting_down, TASK_UNINTERRUPTIBLE);
return;
}
+ set_bit(WB_shutting_down, &wb->state);
spin_unlock_bh(&wb->work_lock);
+ cgwb_remove_from_bdi_list(wb);
/*
* Drain work list and shutdown the delayed_work. !WB_registered
* tells wb_workfn() that @wb is dying and its work_list needs to
@@ -359,6 +376,12 @@
mod_delayed_work(bdi_wq, &wb->dwork, 0);
flush_delayed_work(&wb->dwork);
WARN_ON(!list_empty(&wb->work_list));
+ /*
+ * Make sure bit gets cleared after shutdown is finished. Matches with
+ * the barrier provided by test_and_clear_bit() above.
+ */
+ smp_wmb();
+ clear_bit(WB_shutting_down, &wb->state);
}
static void wb_exit(struct bdi_writeback *wb)
@@ -372,6 +395,8 @@
fprop_local_destroy_percpu(&wb->completions);
wb_congested_put(wb->congested);
+ if (wb != &wb->bdi->wb)
+ bdi_put(wb->bdi);
}
#ifdef CONFIG_CGROUP_WRITEBACK
@@ -381,11 +406,9 @@
/*
* cgwb_lock protects bdi->cgwb_tree, bdi->cgwb_congested_tree,
* blkcg->cgwb_list, and memcg->cgwb_list. bdi->cgwb_tree is also RCU
- * protected. cgwb_release_wait is used to wait for the completion of cgwb
- * releases from bdi destruction path.
+ * protected.
*/
static DEFINE_SPINLOCK(cgwb_lock);
-static DECLARE_WAIT_QUEUE_HEAD(cgwb_release_wait);
/**
* wb_congested_get_create - get or create a wb_congested
@@ -438,7 +461,7 @@
return NULL;
atomic_set(&new_congested->refcnt, 0);
- new_congested->bdi = bdi;
+ new_congested->__bdi = bdi;
new_congested->blkcg_id = blkcg_id;
goto retry;
@@ -466,10 +489,10 @@
}
/* bdi might already have been destroyed leaving @congested unlinked */
- if (congested->bdi) {
+ if (congested->__bdi) {
rb_erase(&congested->rb_node,
- &congested->bdi->cgwb_congested_tree);
- congested->bdi = NULL;
+ &congested->__bdi->cgwb_congested_tree);
+ congested->__bdi = NULL;
}
spin_unlock_irqrestore(&cgwb_lock, flags);
@@ -480,11 +503,6 @@
{
struct bdi_writeback *wb = container_of(work, struct bdi_writeback,
release_work);
- struct backing_dev_info *bdi = wb->bdi;
-
- spin_lock_irq(&cgwb_lock);
- list_del_rcu(&wb->bdi_node);
- spin_unlock_irq(&cgwb_lock);
wb_shutdown(wb);
@@ -495,9 +513,6 @@
percpu_ref_exit(&wb->refcnt);
wb_exit(wb);
kfree_rcu(wb, rcu);
-
- if (atomic_dec_and_test(&bdi->usage_cnt))
- wake_up_all(&cgwb_release_wait);
}
static void cgwb_release(struct percpu_ref *refcnt)
@@ -517,6 +532,13 @@
percpu_ref_kill(&wb->refcnt);
}
+static void cgwb_remove_from_bdi_list(struct bdi_writeback *wb)
+{
+ spin_lock_irq(&cgwb_lock);
+ list_del_rcu(&wb->bdi_node);
+ spin_unlock_irq(&cgwb_lock);
+}
+
static int cgwb_create(struct backing_dev_info *bdi,
struct cgroup_subsys_state *memcg_css, gfp_t gfp)
{
@@ -580,7 +602,6 @@
/* we might have raced another instance of this function */
ret = radix_tree_insert(&bdi->cgwb_tree, memcg_css->id, wb);
if (!ret) {
- atomic_inc(&bdi->usage_cnt);
list_add_tail_rcu(&wb->bdi_node, &bdi->wb_list);
list_add(&wb->memcg_node, memcg_cgwb_list);
list_add(&wb->blkcg_node, blkcg_cgwb_list);
@@ -670,7 +691,6 @@
INIT_RADIX_TREE(&bdi->cgwb_tree, GFP_ATOMIC);
bdi->cgwb_congested_tree = RB_ROOT;
- atomic_set(&bdi->usage_cnt, 1);
ret = wb_init(&bdi->wb, bdi, 1, GFP_KERNEL);
if (!ret) {
@@ -680,29 +700,26 @@
return ret;
}
-static void cgwb_bdi_destroy(struct backing_dev_info *bdi)
+static void cgwb_bdi_unregister(struct backing_dev_info *bdi)
{
struct radix_tree_iter iter;
void **slot;
+ struct bdi_writeback *wb;
WARN_ON(test_bit(WB_registered, &bdi->wb.state));
spin_lock_irq(&cgwb_lock);
radix_tree_for_each_slot(slot, &bdi->cgwb_tree, &iter, 0)
cgwb_kill(*slot);
- spin_unlock_irq(&cgwb_lock);
- /*
- * All cgwb's must be shutdown and released before returning. Drain
- * the usage counter to wait for all cgwb's ever created on @bdi.
- */
- atomic_dec(&bdi->usage_cnt);
- wait_event(cgwb_release_wait, !atomic_read(&bdi->usage_cnt));
- /*
- * Grab back our reference so that we hold it when @bdi gets
- * re-registered.
- */
- atomic_inc(&bdi->usage_cnt);
+ while (!list_empty(&bdi->wb_list)) {
+ wb = list_first_entry(&bdi->wb_list, struct bdi_writeback,
+ bdi_node);
+ spin_unlock_irq(&cgwb_lock);
+ wb_shutdown(wb);
+ spin_lock_irq(&cgwb_lock);
+ }
+ spin_unlock_irq(&cgwb_lock);
}
/**
@@ -752,11 +769,18 @@
rb_entry(rbn, struct bdi_writeback_congested, rb_node);
rb_erase(rbn, &bdi->cgwb_congested_tree);
- congested->bdi = NULL; /* mark @congested unlinked */
+ congested->__bdi = NULL; /* mark @congested unlinked */
}
spin_unlock_irq(&cgwb_lock);
}
+static void cgwb_bdi_register(struct backing_dev_info *bdi)
+{
+ spin_lock_irq(&cgwb_lock);
+ list_add_tail_rcu(&bdi->wb.bdi_node, &bdi->wb_list);
+ spin_unlock_irq(&cgwb_lock);
+}
+
#else /* CONFIG_CGROUP_WRITEBACK */
static int cgwb_bdi_init(struct backing_dev_info *bdi)
@@ -777,16 +801,26 @@
return 0;
}
-static void cgwb_bdi_destroy(struct backing_dev_info *bdi) { }
+static void cgwb_bdi_unregister(struct backing_dev_info *bdi) { }
static void cgwb_bdi_exit(struct backing_dev_info *bdi)
{
wb_congested_put(bdi->wb_congested);
}
+static void cgwb_bdi_register(struct backing_dev_info *bdi)
+{
+ list_add_tail_rcu(&bdi->wb.bdi_node, &bdi->wb_list);
+}
+
+static void cgwb_remove_from_bdi_list(struct bdi_writeback *wb)
+{
+ list_del_rcu(&wb->bdi_node);
+}
+
#endif /* CONFIG_CGROUP_WRITEBACK */
-int bdi_init(struct backing_dev_info *bdi)
+static int bdi_init(struct backing_dev_info *bdi)
{
int ret;
@@ -802,11 +836,8 @@
ret = cgwb_bdi_init(bdi);
- list_add_tail_rcu(&bdi->wb.bdi_node, &bdi->wb_list);
-
return ret;
}
-EXPORT_SYMBOL(bdi_init);
struct backing_dev_info *bdi_alloc_node(gfp_t gfp_mask, int node_id)
{
@@ -823,22 +854,20 @@
}
return bdi;
}
+EXPORT_SYMBOL(bdi_alloc_node);
-int bdi_register(struct backing_dev_info *bdi, struct device *parent,
- const char *fmt, ...)
+int bdi_register_va(struct backing_dev_info *bdi, const char *fmt, va_list args)
{
- va_list args;
struct device *dev;
if (bdi->dev) /* The driver needs to use separate queues per device */
return 0;
- va_start(args, fmt);
- dev = device_create_vargs(bdi_class, parent, MKDEV(0, 0), bdi, fmt, args);
- va_end(args);
+ dev = device_create_vargs(bdi_class, NULL, MKDEV(0, 0), bdi, fmt, args);
if (IS_ERR(dev))
return PTR_ERR(dev);
+ cgwb_bdi_register(bdi);
bdi->dev = dev;
bdi_debug_register(bdi, dev_name(dev));
@@ -851,20 +880,25 @@
trace_writeback_bdi_register(bdi);
return 0;
}
-EXPORT_SYMBOL(bdi_register);
+EXPORT_SYMBOL(bdi_register_va);
-int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev)
+int bdi_register(struct backing_dev_info *bdi, const char *fmt, ...)
{
- return bdi_register(bdi, NULL, "%u:%u", MAJOR(dev), MINOR(dev));
+ va_list args;
+ int ret;
+
+ va_start(args, fmt);
+ ret = bdi_register_va(bdi, fmt, args);
+ va_end(args);
+ return ret;
}
-EXPORT_SYMBOL(bdi_register_dev);
+EXPORT_SYMBOL(bdi_register);
int bdi_register_owner(struct backing_dev_info *bdi, struct device *owner)
{
int rc;
- rc = bdi_register(bdi, NULL, "%u:%u", MAJOR(owner->devt),
- MINOR(owner->devt));
+ rc = bdi_register(bdi, "%u:%u", MAJOR(owner->devt), MINOR(owner->devt));
if (rc)
return rc;
/* Leaking owner reference... */
@@ -892,7 +926,7 @@
/* make sure nobody finds us on the bdi_list anymore */
bdi_remove_from_list(bdi);
wb_shutdown(&bdi->wb);
- cgwb_bdi_destroy(bdi);
+ cgwb_bdi_unregister(bdi);
if (bdi->dev) {
bdi_debug_unregister(bdi);
@@ -906,19 +940,16 @@
}
}
-static void bdi_exit(struct backing_dev_info *bdi)
-{
- WARN_ON_ONCE(bdi->dev);
- wb_exit(&bdi->wb);
- cgwb_bdi_exit(bdi);
-}
-
static void release_bdi(struct kref *ref)
{
struct backing_dev_info *bdi =
container_of(ref, struct backing_dev_info, refcnt);
- bdi_exit(bdi);
+ if (test_bit(WB_registered, &bdi->wb.state))
+ bdi_unregister(bdi);
+ WARN_ON_ONCE(bdi->dev);
+ wb_exit(&bdi->wb);
+ cgwb_bdi_exit(bdi);
kfree(bdi);
}
@@ -926,38 +957,7 @@
{
kref_put(&bdi->refcnt, release_bdi);
}
-
-void bdi_destroy(struct backing_dev_info *bdi)
-{
- bdi_unregister(bdi);
- bdi_exit(bdi);
-}
-EXPORT_SYMBOL(bdi_destroy);
-
-/*
- * For use from filesystems to quickly init and register a bdi associated
- * with dirty writeback
- */
-int bdi_setup_and_register(struct backing_dev_info *bdi, char *name)
-{
- int err;
-
- bdi->name = name;
- bdi->capabilities = 0;
- err = bdi_init(bdi);
- if (err)
- return err;
-
- err = bdi_register(bdi, NULL, "%.28s-%ld", name,
- atomic_long_inc_return(&bdi_seq));
- if (err) {
- bdi_destroy(bdi);
- return err;
- }
-
- return 0;
-}
-EXPORT_SYMBOL(bdi_setup_and_register);
+EXPORT_SYMBOL(bdi_put);
static wait_queue_head_t congestion_wqh[2] = {
__WAIT_QUEUE_HEAD_INITIALIZER(congestion_wqh[0]),
diff --git a/mm/filemap.c b/mm/filemap.c
index 1694623..dc59c5f 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -519,7 +519,7 @@
*
* Write out and wait upon file offsets lstart->lend, inclusive.
*
- * Note that `lend' is inclusive (describes the last byte to be written) so
+ * Note that @lend is inclusive (describes the last byte to be written) so
* that this function can be used to write to the very end-of-file (end = -1).
*/
int filemap_write_and_wait_range(struct address_space *mapping,
@@ -1277,12 +1277,14 @@
*
* PCG flags modify how the page is returned.
*
- * FGP_ACCESSED: the page will be marked accessed
- * FGP_LOCK: Page is return locked
- * FGP_CREAT: If page is not present then a new page is allocated using
- * @gfp_mask and added to the page cache and the VM's LRU
- * list. The page is returned locked and with an increased
- * refcount. Otherwise, %NULL is returned.
+ * @fgp_flags can be:
+ *
+ * - FGP_ACCESSED: the page will be marked accessed
+ * - FGP_LOCK: Page is return locked
+ * - FGP_CREAT: If page is not present then a new page is allocated using
+ * @gfp_mask and added to the page cache and the VM's LRU
+ * list. The page is returned locked and with an increased
+ * refcount. Otherwise, NULL is returned.
*
* If FGP_LOCK or FGP_CREAT are specified then the function may sleep even
* if the GFP flags specified for FGP_CREAT are atomic.
@@ -2033,7 +2035,6 @@
if (iocb->ki_flags & IOCB_DIRECT) {
struct address_space *mapping = file->f_mapping;
struct inode *inode = mapping->host;
- struct iov_iter data = *iter;
loff_t size;
size = i_size_read(inode);
@@ -2044,11 +2045,12 @@
file_accessed(file);
- retval = mapping->a_ops->direct_IO(iocb, &data);
+ retval = mapping->a_ops->direct_IO(iocb, iter);
if (retval >= 0) {
iocb->ki_pos += retval;
- iov_iter_advance(iter, retval);
+ count -= retval;
}
+ iov_iter_revert(iter, iov_iter_count(iter) - count);
/*
* Btrfs can have a short DIO read if we encounter
@@ -2059,7 +2061,7 @@
* the rest of the read. Buffered reads will not work for
* DAX files, so don't bother trying.
*/
- if (retval < 0 || !iov_iter_count(iter) || iocb->ki_pos >= size ||
+ if (retval < 0 || !count || iocb->ki_pos >= size ||
IS_DAX(inode))
goto out;
}
@@ -2704,7 +2706,6 @@
ssize_t written;
size_t write_len;
pgoff_t end;
- struct iov_iter data;
write_len = iov_iter_count(from);
end = (pos + write_len - 1) >> PAGE_SHIFT;
@@ -2733,8 +2734,7 @@
}
}
- data = *from;
- written = mapping->a_ops->direct_IO(iocb, &data);
+ written = mapping->a_ops->direct_IO(iocb, from);
/*
* Finally, try again to invalidate clean pages which might have been
@@ -2751,13 +2751,14 @@
if (written > 0) {
pos += written;
- iov_iter_advance(from, written);
+ write_len -= written;
if (pos > i_size_read(inode) && !S_ISBLK(inode->i_mode)) {
i_size_write(inode, pos);
mark_inode_dirty(inode);
}
iocb->ki_pos = pos;
}
+ iov_iter_revert(from, write_len - iov_iter_count(from));
out:
return written;
}
@@ -3001,7 +3002,7 @@
* @gfp_mask: memory allocation flags (and I/O mode)
*
* The address_space is to try to release any data against the page
- * (presumably at page->private). If the release was successful, return `1'.
+ * (presumably at page->private). If the release was successful, return '1'.
* Otherwise return zero.
*
* This may also be called if PG_fscache is set on a page, indicating that the
diff --git a/mm/gup.c b/mm/gup.c
index 04aa405..527ec2c 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -1189,34 +1189,57 @@
*/
#ifdef CONFIG_HAVE_GENERIC_RCU_GUP
+#ifndef gup_get_pte
+/*
+ * We assume that the PTE can be read atomically. If this is not the case for
+ * your architecture, please provide the helper.
+ */
+static inline pte_t gup_get_pte(pte_t *ptep)
+{
+ return READ_ONCE(*ptep);
+}
+#endif
+
+static void undo_dev_pagemap(int *nr, int nr_start, struct page **pages)
+{
+ while ((*nr) - nr_start) {
+ struct page *page = pages[--(*nr)];
+
+ ClearPageReferenced(page);
+ put_page(page);
+ }
+}
+
#ifdef __HAVE_ARCH_PTE_SPECIAL
static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,
int write, struct page **pages, int *nr)
{
+ struct dev_pagemap *pgmap = NULL;
+ int nr_start = *nr, ret = 0;
pte_t *ptep, *ptem;
- int ret = 0;
ptem = ptep = pte_offset_map(&pmd, addr);
do {
- /*
- * In the line below we are assuming that the pte can be read
- * atomically. If this is not the case for your architecture,
- * please wrap this in a helper function!
- *
- * for an example see gup_get_pte in arch/x86/mm/gup.c
- */
- pte_t pte = READ_ONCE(*ptep);
+ pte_t pte = gup_get_pte(ptep);
struct page *head, *page;
/*
* Similar to the PMD case below, NUMA hinting must take slow
* path using the pte_protnone check.
*/
- if (!pte_present(pte) || pte_special(pte) ||
- pte_protnone(pte) || (write && !pte_write(pte)))
+ if (pte_protnone(pte))
goto pte_unmap;
- if (!arch_pte_access_permitted(pte, write))
+ if (!pte_access_permitted(pte, write))
+ goto pte_unmap;
+
+ if (pte_devmap(pte)) {
+ pgmap = get_dev_pagemap(pte_pfn(pte), pgmap);
+ if (unlikely(!pgmap)) {
+ undo_dev_pagemap(nr, nr_start, pages);
+ goto pte_unmap;
+ }
+ } else if (pte_special(pte))
goto pte_unmap;
VM_BUG_ON(!pfn_valid(pte_pfn(pte)));
@@ -1232,6 +1255,9 @@
}
VM_BUG_ON_PAGE(compound_head(page) != head, page);
+
+ put_dev_pagemap(pgmap);
+ SetPageReferenced(page);
pages[*nr] = page;
(*nr)++;
@@ -1261,15 +1287,76 @@
}
#endif /* __HAVE_ARCH_PTE_SPECIAL */
+#ifdef __HAVE_ARCH_PTE_DEVMAP
+static int __gup_device_huge(unsigned long pfn, unsigned long addr,
+ unsigned long end, struct page **pages, int *nr)
+{
+ int nr_start = *nr;
+ struct dev_pagemap *pgmap = NULL;
+
+ do {
+ struct page *page = pfn_to_page(pfn);
+
+ pgmap = get_dev_pagemap(pfn, pgmap);
+ if (unlikely(!pgmap)) {
+ undo_dev_pagemap(nr, nr_start, pages);
+ return 0;
+ }
+ SetPageReferenced(page);
+ pages[*nr] = page;
+ get_page(page);
+ put_dev_pagemap(pgmap);
+ (*nr)++;
+ pfn++;
+ } while (addr += PAGE_SIZE, addr != end);
+ return 1;
+}
+
+static int __gup_device_huge_pmd(pmd_t pmd, unsigned long addr,
+ unsigned long end, struct page **pages, int *nr)
+{
+ unsigned long fault_pfn;
+
+ fault_pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
+ return __gup_device_huge(fault_pfn, addr, end, pages, nr);
+}
+
+static int __gup_device_huge_pud(pud_t pud, unsigned long addr,
+ unsigned long end, struct page **pages, int *nr)
+{
+ unsigned long fault_pfn;
+
+ fault_pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
+ return __gup_device_huge(fault_pfn, addr, end, pages, nr);
+}
+#else
+static int __gup_device_huge_pmd(pmd_t pmd, unsigned long addr,
+ unsigned long end, struct page **pages, int *nr)
+{
+ BUILD_BUG();
+ return 0;
+}
+
+static int __gup_device_huge_pud(pud_t pud, unsigned long addr,
+ unsigned long end, struct page **pages, int *nr)
+{
+ BUILD_BUG();
+ return 0;
+}
+#endif
+
static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,
unsigned long end, int write, struct page **pages, int *nr)
{
struct page *head, *page;
int refs;
- if (write && !pmd_write(orig))
+ if (!pmd_access_permitted(orig, write))
return 0;
+ if (pmd_devmap(orig))
+ return __gup_device_huge_pmd(orig, addr, end, pages, nr);
+
refs = 0;
head = pmd_page(orig);
page = head + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
@@ -1293,6 +1380,7 @@
return 0;
}
+ SetPageReferenced(head);
return 1;
}
@@ -1302,9 +1390,12 @@
struct page *head, *page;
int refs;
- if (write && !pud_write(orig))
+ if (!pud_access_permitted(orig, write))
return 0;
+ if (pud_devmap(orig))
+ return __gup_device_huge_pud(orig, addr, end, pages, nr);
+
refs = 0;
head = pud_page(orig);
page = head + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
@@ -1328,6 +1419,7 @@
return 0;
}
+ SetPageReferenced(head);
return 1;
}
@@ -1338,9 +1430,10 @@
int refs;
struct page *head, *page;
- if (write && !pgd_write(orig))
+ if (!pgd_access_permitted(orig, write))
return 0;
+ BUILD_BUG_ON(pgd_devmap(orig));
refs = 0;
head = pgd_page(orig);
page = head + ((addr & ~PGDIR_MASK) >> PAGE_SHIFT);
@@ -1364,6 +1457,7 @@
return 0;
}
+ SetPageReferenced(head);
return 1;
}
@@ -1520,6 +1614,21 @@
return nr;
}
+#ifndef gup_fast_permitted
+/*
+ * Check if it's allowed to use __get_user_pages_fast() for the range, or
+ * we need to fall back to the slow version:
+ */
+bool gup_fast_permitted(unsigned long start, int nr_pages, int write)
+{
+ unsigned long len, end;
+
+ len = (unsigned long) nr_pages << PAGE_SHIFT;
+ end = start + len;
+ return end >= start;
+}
+#endif
+
/**
* get_user_pages_fast() - pin user pages in memory
* @start: starting user address
@@ -1539,11 +1648,14 @@
int get_user_pages_fast(unsigned long start, int nr_pages, int write,
struct page **pages)
{
- int nr, ret;
+ int nr = 0, ret = 0;
start &= PAGE_MASK;
- nr = __get_user_pages_fast(start, nr_pages, write, pages);
- ret = nr;
+
+ if (gup_fast_permitted(start, nr_pages, write)) {
+ nr = __get_user_pages_fast(start, nr_pages, write, pages);
+ ret = nr;
+ }
if (nr < nr_pages) {
/* Try to get the remaining pages with get_user_pages */
diff --git a/mm/memory.c b/mm/memory.c
index 235ba51..6ff5d72 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4298,7 +4298,7 @@
* get paged out, therefore we'll never actually fault, and the
* below annotations will generate false positives.
*/
- if (segment_eq(get_fs(), KERNEL_DS))
+ if (uaccess_kernel())
return;
if (pagefault_disabled())
return;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 07efbc3..bd01501 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4247,7 +4247,8 @@
* nr_free_zone_pages() counts the number of counts pages which are beyond the
* high watermark within all zones at or below a given zone index. For each
* zone, the number of pages is calculated as:
- * managed_pages - high_pages
+ *
+ * nr_free_zone_pages = managed_pages - high_pages
*/
static unsigned long nr_free_zone_pages(int offset)
{
diff --git a/mm/percpu.c b/mm/percpu.c
index 60a6488..e0aa8ae 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -1284,6 +1284,31 @@
}
EXPORT_SYMBOL_GPL(free_percpu);
+bool __is_kernel_percpu_address(unsigned long addr, unsigned long *can_addr)
+{
+#ifdef CONFIG_SMP
+ const size_t static_size = __per_cpu_end - __per_cpu_start;
+ void __percpu *base = __addr_to_pcpu_ptr(pcpu_base_addr);
+ unsigned int cpu;
+
+ for_each_possible_cpu(cpu) {
+ void *start = per_cpu_ptr(base, cpu);
+ void *va = (void *)addr;
+
+ if (va >= start && va < start + static_size) {
+ if (can_addr) {
+ *can_addr = (unsigned long) (va - start);
+ *can_addr += (unsigned long)
+ per_cpu_ptr(base, get_boot_cpu_id());
+ }
+ return true;
+ }
+ }
+#endif
+ /* on UP, can't distinguish from other static vars, always false */
+ return false;
+}
+
/**
* is_kernel_percpu_address - test whether address is from static percpu area
* @addr: address to test
@@ -1297,20 +1322,7 @@
*/
bool is_kernel_percpu_address(unsigned long addr)
{
-#ifdef CONFIG_SMP
- const size_t static_size = __per_cpu_end - __per_cpu_start;
- void __percpu *base = __addr_to_pcpu_ptr(pcpu_base_addr);
- unsigned int cpu;
-
- for_each_possible_cpu(cpu) {
- void *start = per_cpu_ptr(base, cpu);
-
- if ((void *)addr >= start && (void *)addr < start + static_size)
- return true;
- }
-#endif
- /* on UP, can't distinguish from other static vars, always false */
- return false;
+ return __is_kernel_percpu_address(addr, NULL);
}
/**
diff --git a/mm/swap.c b/mm/swap.c
index 5dabf44..d8d9ee9 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -97,6 +97,16 @@
void __put_page(struct page *page)
{
+ if (is_zone_device_page(page)) {
+ put_dev_pagemap(page->pgmap);
+
+ /*
+ * The page belongs to the device that created pgmap. Do
+ * not return it to page allocator.
+ */
+ return;
+ }
+
if (unlikely(PageCompound(page)))
__put_compound_page(page);
else
diff --git a/mm/usercopy.c b/mm/usercopy.c
index d155e12..a9852b2 100644
--- a/mm/usercopy.c
+++ b/mm/usercopy.c
@@ -19,15 +19,9 @@
#include <linux/sched.h>
#include <linux/sched/task.h>
#include <linux/sched/task_stack.h>
+#include <linux/thread_info.h>
#include <asm/sections.h>
-enum {
- BAD_STACK = -1,
- NOT_STACK = 0,
- GOOD_FRAME,
- GOOD_STACK,
-};
-
/*
* Checks if a given pointer and length is contained by the current
* stack frame (if possible).
@@ -206,17 +200,6 @@
{
struct page *page;
- /*
- * Some architectures (arm64) return true for virt_addr_valid() on
- * vmalloced addresses. Work around this by checking for vmalloc
- * first.
- *
- * We also need to check for module addresses explicitly since we
- * may copy static data from modules to userspace
- */
- if (is_vmalloc_or_module_addr(ptr))
- return NULL;
-
if (!virt_addr_valid(ptr))
return NULL;
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 0b05762..b52aeed 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1579,7 +1579,7 @@
* have CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG, but making the calling
* conventions for vfree() arch-depenedent would be a really bad idea)
*
- * NOTE: assumes that the object at *addr has a size >= sizeof(llist_node)
+ * NOTE: assumes that the object at @addr has a size >= sizeof(llist_node)
*/
void vfree(const void *addr)
{
diff --git a/net/6lowpan/core.c b/net/6lowpan/core.c
index 5945f7e..40d3d72 100644
--- a/net/6lowpan/core.c
+++ b/net/6lowpan/core.c
@@ -23,10 +23,18 @@
{
int i, ret;
- dev->addr_len = EUI64_ADDR_LEN;
+ switch (lltype) {
+ case LOWPAN_LLTYPE_IEEE802154:
+ dev->addr_len = EUI64_ADDR_LEN;
+ break;
+
+ case LOWPAN_LLTYPE_BTLE:
+ dev->addr_len = ETH_ALEN;
+ break;
+ }
+
dev->type = ARPHRD_6LOWPAN;
dev->mtu = IPV6_MIN_MTU;
- dev->priv_flags |= IFF_NO_QUEUE;
lowpan_dev(dev)->lltype = lltype;
diff --git a/net/6lowpan/iphc.c b/net/6lowpan/iphc.c
index 79f1fa2..6b1042e 100644
--- a/net/6lowpan/iphc.c
+++ b/net/6lowpan/iphc.c
@@ -278,6 +278,23 @@
return ret;
}
+static void lowpan_iphc_uncompress_lladdr(const struct net_device *dev,
+ struct in6_addr *ipaddr,
+ const void *lladdr)
+{
+ switch (dev->addr_len) {
+ case ETH_ALEN:
+ lowpan_iphc_uncompress_eui48_lladdr(ipaddr, lladdr);
+ break;
+ case EUI64_ADDR_LEN:
+ lowpan_iphc_uncompress_eui64_lladdr(ipaddr, lladdr);
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ break;
+ }
+}
+
/* Uncompress address function for source and
* destination address(non-multicast).
*
@@ -320,7 +337,7 @@
lowpan_iphc_uncompress_802154_lladdr(ipaddr, lladdr);
break;
default:
- lowpan_iphc_uncompress_eui64_lladdr(ipaddr, lladdr);
+ lowpan_iphc_uncompress_lladdr(dev, ipaddr, lladdr);
break;
}
break;
@@ -381,7 +398,7 @@
lowpan_iphc_uncompress_802154_lladdr(ipaddr, lladdr);
break;
default:
- lowpan_iphc_uncompress_eui64_lladdr(ipaddr, lladdr);
+ lowpan_iphc_uncompress_lladdr(dev, ipaddr, lladdr);
break;
}
ipv6_addr_prefix_copy(ipaddr, &ctx->pfx, ctx->plen);
@@ -666,6 +683,8 @@
switch (iphc1 & (LOWPAN_IPHC_M | LOWPAN_IPHC_DAC)) {
case LOWPAN_IPHC_M | LOWPAN_IPHC_DAC:
+ skb->pkt_type = PACKET_BROADCAST;
+
spin_lock_bh(&lowpan_dev(dev)->ctx.lock);
ci = lowpan_iphc_ctx_get_by_id(dev, LOWPAN_IPHC_CID_DCI(cid));
if (!ci) {
@@ -681,11 +700,15 @@
spin_unlock_bh(&lowpan_dev(dev)->ctx.lock);
break;
case LOWPAN_IPHC_M:
+ skb->pkt_type = PACKET_BROADCAST;
+
/* multicast */
err = lowpan_uncompress_multicast_daddr(skb, &hdr.daddr,
iphc1 & LOWPAN_IPHC_DAM_MASK);
break;
case LOWPAN_IPHC_DAC:
+ skb->pkt_type = PACKET_HOST;
+
spin_lock_bh(&lowpan_dev(dev)->ctx.lock);
ci = lowpan_iphc_ctx_get_by_id(dev, LOWPAN_IPHC_CID_DCI(cid));
if (!ci) {
@@ -701,6 +724,8 @@
spin_unlock_bh(&lowpan_dev(dev)->ctx.lock);
break;
default:
+ skb->pkt_type = PACKET_HOST;
+
err = lowpan_iphc_uncompress_addr(skb, dev, &hdr.daddr,
iphc1 & LOWPAN_IPHC_DAM_MASK,
daddr);
@@ -802,6 +827,21 @@
return lladdr_compress;
}
+static bool lowpan_iphc_addr_equal(const struct net_device *dev,
+ const struct lowpan_iphc_ctx *ctx,
+ const struct in6_addr *ipaddr,
+ const void *lladdr)
+{
+ struct in6_addr tmp = {};
+
+ lowpan_iphc_uncompress_lladdr(dev, &tmp, lladdr);
+
+ if (ctx)
+ ipv6_addr_prefix_copy(&tmp, &ctx->pfx, ctx->plen);
+
+ return ipv6_addr_equal(&tmp, ipaddr);
+}
+
static u8 lowpan_compress_ctx_addr(u8 **hc_ptr, const struct net_device *dev,
const struct in6_addr *ipaddr,
const struct lowpan_iphc_ctx *ctx,
@@ -819,13 +859,7 @@
}
break;
default:
- /* check for SAM/DAM = 11 */
- memcpy(&tmp.s6_addr[8], lladdr, EUI64_ADDR_LEN);
- /* second bit-flip (Universe/Local) is done according RFC2464 */
- tmp.s6_addr[8] ^= 0x02;
- /* context information are always used */
- ipv6_addr_prefix_copy(&tmp, &ctx->pfx, ctx->plen);
- if (ipv6_addr_equal(&tmp, ipaddr)) {
+ if (lowpan_iphc_addr_equal(dev, ctx, ipaddr, lladdr)) {
dam = LOWPAN_IPHC_DAM_11;
goto out;
}
@@ -921,11 +955,12 @@
}
break;
default:
- if (is_addr_mac_addr_based(ipaddr, lladdr)) {
- dam = LOWPAN_IPHC_DAM_11; /* 0-bits */
+ if (lowpan_iphc_addr_equal(dev, NULL, ipaddr, lladdr)) {
+ dam = LOWPAN_IPHC_DAM_11;
pr_debug("address compression 0 bits\n");
goto out;
}
+
break;
}
diff --git a/net/8021q/vlan_dev.c b/net/8021q/vlan_dev.c
index e97ab82..9ee5787 100644
--- a/net/8021q/vlan_dev.c
+++ b/net/8021q/vlan_dev.c
@@ -562,8 +562,7 @@
NETIF_F_HIGHDMA | NETIF_F_SCTP_CRC |
NETIF_F_ALL_FCOE;
- dev->features |= real_dev->vlan_features | NETIF_F_LLTX |
- NETIF_F_GSO_SOFTWARE;
+ dev->features |= dev->hw_features | NETIF_F_LLTX;
dev->gso_max_size = real_dev->gso_max_size;
dev->gso_max_segs = real_dev->gso_max_segs;
if (dev->features & NETIF_F_VLAN_FEATURES)
diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
index 1270207..9c94aad 100644
--- a/net/8021q/vlan_netlink.c
+++ b/net/8021q/vlan_netlink.c
@@ -35,7 +35,8 @@
{
if (!attr)
return 0;
- return nla_validate_nested(attr, IFLA_VLAN_QOS_MAX, vlan_map_policy);
+ return nla_validate_nested(attr, IFLA_VLAN_QOS_MAX, vlan_map_policy,
+ NULL);
}
static int vlan_validate(struct nlattr *tb[], struct nlattr *data[])
diff --git a/net/9p/client.c b/net/9p/client.c
index 3ce672a..1218fb3 100644
--- a/net/9p/client.c
+++ b/net/9p/client.c
@@ -592,9 +592,8 @@
ename = &req->rc->sdata[req->rc->offset];
if (len > inline_len) {
/* We have error in external buffer */
- err = copy_from_iter(ename + inline_len,
- len - inline_len, uidata);
- if (err != len - inline_len) {
+ if (!copy_from_iter_full(ename + inline_len,
+ len - inline_len, uidata)) {
err = -EFAULT;
goto out_err;
}
@@ -2101,6 +2100,10 @@
trace_9p_protocol_dump(clnt, req->rc);
goto free_and_error;
}
+ if (rsize < count) {
+ pr_err("bogus RREADDIR count (%d > %d)\n", count, rsize);
+ count = rsize;
+ }
p9_debug(P9_DEBUG_9P, "<<< RREADDIR count %d\n", count);
diff --git a/net/9p/protocol.c b/net/9p/protocol.c
index 16d2875..16e1068 100644
--- a/net/9p/protocol.c
+++ b/net/9p/protocol.c
@@ -74,7 +74,7 @@
{
size_t len = min(pdu->capacity - pdu->size, size);
struct iov_iter i = *from;
- if (copy_from_iter(&pdu->sdata[pdu->size], len, &i) != len)
+ if (!copy_from_iter_full(&pdu->sdata[pdu->size], len, &i))
len = 0;
pdu->size += len;
diff --git a/net/Makefile b/net/Makefile
index 9b681550..9086ffb 100644
--- a/net/Makefile
+++ b/net/Makefile
@@ -12,7 +12,7 @@
# LLC has to be linked before the files in net/802/
obj-$(CONFIG_LLC) += llc/
-obj-$(CONFIG_NET) += ethernet/ 802/ sched/ netlink/
+obj-$(CONFIG_NET) += ethernet/ 802/ sched/ netlink/ bpf/
obj-$(CONFIG_NETFILTER) += netfilter/
obj-$(CONFIG_INET) += ipv4/
obj-$(CONFIG_XFRM) += xfrm/
diff --git a/net/atm/clip.c b/net/atm/clip.c
index 53b4ac0..ec527b6 100644
--- a/net/atm/clip.c
+++ b/net/atm/clip.c
@@ -106,7 +106,7 @@
entry->expires = jiffies - 1;
/* force resolution or expiration */
error = neigh_update(entry->neigh, NULL, NUD_NONE,
- NEIGH_UPDATE_F_ADMIN);
+ NEIGH_UPDATE_F_ADMIN, 0);
if (error)
pr_crit("neigh_update failed with %d\n", error);
goto out;
@@ -481,7 +481,7 @@
link_vcc(clip_vcc, entry);
}
error = neigh_update(neigh, llc_oui, NUD_PERMANENT,
- NEIGH_UPDATE_F_OVERRIDE | NEIGH_UPDATE_F_ADMIN);
+ NEIGH_UPDATE_F_OVERRIDE | NEIGH_UPDATE_F_ADMIN, 0);
neigh_release(neigh);
return error;
}
diff --git a/net/atm/common.c b/net/atm/common.c
index 9613381..f06422f 100644
--- a/net/atm/common.c
+++ b/net/atm/common.c
@@ -62,21 +62,16 @@
write_unlock_irq(&vcc_sklist_lock);
}
-static struct sk_buff *alloc_tx(struct atm_vcc *vcc, unsigned int size)
+static bool vcc_tx_ready(struct atm_vcc *vcc, unsigned int size)
{
- struct sk_buff *skb;
struct sock *sk = sk_atm(vcc);
if (sk_wmem_alloc_get(sk) && !atm_may_send(vcc, size)) {
pr_debug("Sorry: wmem_alloc = %d, size = %d, sndbuf = %d\n",
sk_wmem_alloc_get(sk), size, sk->sk_sndbuf);
- return NULL;
+ return false;
}
- while (!(skb = alloc_skb(size, GFP_KERNEL)))
- schedule();
- pr_debug("%d += %d\n", sk_wmem_alloc_get(sk), skb->truesize);
- atomic_add(skb->truesize, &sk->sk_wmem_alloc);
- return skb;
+ return true;
}
static void vcc_sock_destruct(struct sock *sk)
@@ -606,7 +601,7 @@
eff = (size+3) & ~3; /* align to word boundary */
prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
error = 0;
- while (!(skb = alloc_tx(vcc, eff))) {
+ while (!vcc_tx_ready(vcc, eff)) {
if (m->msg_flags & MSG_DONTWAIT) {
error = -EAGAIN;
break;
@@ -628,6 +623,15 @@
finish_wait(sk_sleep(sk), &wait);
if (error)
goto out;
+
+ skb = alloc_skb(eff, GFP_KERNEL);
+ if (!skb) {
+ error = -ENOMEM;
+ goto out;
+ }
+ pr_debug("%d += %d\n", sk_wmem_alloc_get(sk), skb->truesize);
+ atomic_add(skb->truesize, &sk->sk_wmem_alloc);
+
skb->dev = NULL; /* for paths shared with net_device interfaces */
ATM_SKB(skb)->atm_options = vcc->atm_options;
if (!copy_from_iter_full(skb_put(skb, size), size, &m->msg_iter)) {
diff --git a/net/batman-adv/bat_iv_ogm.c b/net/batman-adv/bat_iv_ogm.c
index 71343d0..495ba7c 100644
--- a/net/batman-adv/bat_iv_ogm.c
+++ b/net/batman-adv/bat_iv_ogm.c
@@ -679,15 +679,11 @@
{
struct batadv_priv *bat_priv = netdev_priv(if_incoming->soft_iface);
struct batadv_forw_packet *forw_packet_aggr;
+ struct sk_buff *skb;
unsigned char *skb_buff;
unsigned int skb_size;
atomic_t *queue_left = own_packet ? NULL : &bat_priv->batman_queue_left;
- forw_packet_aggr = batadv_forw_packet_alloc(if_incoming, if_outgoing,
- queue_left, bat_priv);
- if (!forw_packet_aggr)
- return;
-
if (atomic_read(&bat_priv->aggregated_ogms) &&
packet_len < BATADV_MAX_AGGREGATION_BYTES)
skb_size = BATADV_MAX_AGGREGATION_BYTES;
@@ -696,9 +692,14 @@
skb_size += ETH_HLEN;
- forw_packet_aggr->skb = netdev_alloc_skb_ip_align(NULL, skb_size);
- if (!forw_packet_aggr->skb) {
- batadv_forw_packet_free(forw_packet_aggr, true);
+ skb = netdev_alloc_skb_ip_align(NULL, skb_size);
+ if (!skb)
+ return;
+
+ forw_packet_aggr = batadv_forw_packet_alloc(if_incoming, if_outgoing,
+ queue_left, bat_priv, skb);
+ if (!forw_packet_aggr) {
+ kfree_skb(skb);
return;
}
diff --git a/net/batman-adv/bridge_loop_avoidance.c b/net/batman-adv/bridge_loop_avoidance.c
index ba8420d..d07e89e 100644
--- a/net/batman-adv/bridge_loop_avoidance.c
+++ b/net/batman-adv/bridge_loop_avoidance.c
@@ -395,7 +395,7 @@
ether_addr_copy(ethhdr->h_source, mac);
batadv_dbg(BATADV_DBG_BLA, bat_priv,
"bla_send_claim(): CLAIM %pM on vid %d\n", mac,
- BATADV_PRINT_VID(vid));
+ batadv_print_vid(vid));
break;
case BATADV_CLAIM_TYPE_UNCLAIM:
/* unclaim frame
@@ -404,7 +404,7 @@
ether_addr_copy(hw_src, mac);
batadv_dbg(BATADV_DBG_BLA, bat_priv,
"bla_send_claim(): UNCLAIM %pM on vid %d\n", mac,
- BATADV_PRINT_VID(vid));
+ batadv_print_vid(vid));
break;
case BATADV_CLAIM_TYPE_ANNOUNCE:
/* announcement frame
@@ -413,7 +413,7 @@
ether_addr_copy(hw_src, mac);
batadv_dbg(BATADV_DBG_BLA, bat_priv,
"bla_send_claim(): ANNOUNCE of %pM on vid %d\n",
- ethhdr->h_source, BATADV_PRINT_VID(vid));
+ ethhdr->h_source, batadv_print_vid(vid));
break;
case BATADV_CLAIM_TYPE_REQUEST:
/* request frame
@@ -425,14 +425,14 @@
batadv_dbg(BATADV_DBG_BLA, bat_priv,
"bla_send_claim(): REQUEST of %pM to %pM on vid %d\n",
ethhdr->h_source, ethhdr->h_dest,
- BATADV_PRINT_VID(vid));
+ batadv_print_vid(vid));
break;
case BATADV_CLAIM_TYPE_LOOPDETECT:
ether_addr_copy(ethhdr->h_source, mac);
batadv_dbg(BATADV_DBG_BLA, bat_priv,
"bla_send_claim(): LOOPDETECT of %pM to %pM on vid %d\n",
ethhdr->h_source, ethhdr->h_dest,
- BATADV_PRINT_VID(vid));
+ batadv_print_vid(vid));
break;
}
@@ -475,9 +475,9 @@
batadv_info(bat_priv->soft_iface,
"Possible loop on VLAN %d detected which can't be handled by BLA - please check your network setup!\n",
- BATADV_PRINT_VID(backbone_gw->vid));
+ batadv_print_vid(backbone_gw->vid));
snprintf(vid_str, sizeof(vid_str), "%d",
- BATADV_PRINT_VID(backbone_gw->vid));
+ batadv_print_vid(backbone_gw->vid));
vid_str[sizeof(vid_str) - 1] = 0;
batadv_throw_uevent(bat_priv, BATADV_UEV_BLA, BATADV_UEV_LOOPDETECT,
@@ -510,7 +510,7 @@
batadv_dbg(BATADV_DBG_BLA, bat_priv,
"bla_get_backbone_gw(): not found (%pM, %d), creating new entry\n",
- orig, BATADV_PRINT_VID(vid));
+ orig, batadv_print_vid(vid));
entry = kzalloc(sizeof(*entry), GFP_ATOMIC);
if (!entry)
@@ -719,7 +719,7 @@
batadv_dbg(BATADV_DBG_BLA, bat_priv,
"bla_add_claim(): adding new entry %pM, vid %d to hash ...\n",
- mac, BATADV_PRINT_VID(vid));
+ mac, batadv_print_vid(vid));
kref_get(&claim->refcount);
hash_added = batadv_hash_add(bat_priv->bla.claim_hash,
@@ -739,8 +739,8 @@
goto claim_free_ref;
batadv_dbg(BATADV_DBG_BLA, bat_priv,
- "bla_add_claim(): changing ownership for %pM, vid %d\n",
- mac, BATADV_PRINT_VID(vid));
+ "bla_add_claim(): changing ownership for %pM, vid %d to gw %pM\n",
+ mac, batadv_print_vid(vid), backbone_gw->orig);
remove_crc = true;
}
@@ -809,7 +809,7 @@
return;
batadv_dbg(BATADV_DBG_BLA, bat_priv, "bla_del_claim(): %pM, vid %d\n",
- mac, BATADV_PRINT_VID(vid));
+ mac, batadv_print_vid(vid));
batadv_hash_remove(bat_priv->bla.claim_hash, batadv_compare_claim,
batadv_choose_claim, claim);
@@ -849,7 +849,7 @@
batadv_dbg(BATADV_DBG_BLA, bat_priv,
"handle_announce(): ANNOUNCE vid %d (sent by %pM)... CRC = %#.4x\n",
- BATADV_PRINT_VID(vid), backbone_gw->orig, crc);
+ batadv_print_vid(vid), backbone_gw->orig, crc);
spin_lock_bh(&backbone_gw->crc_lock);
backbone_crc = backbone_gw->crc;
@@ -859,7 +859,7 @@
batadv_dbg(BATADV_DBG_BLA, backbone_gw->bat_priv,
"handle_announce(): CRC FAILED for %pM/%d (my = %#.4x, sent = %#.4x)\n",
backbone_gw->orig,
- BATADV_PRINT_VID(backbone_gw->vid),
+ batadv_print_vid(backbone_gw->vid),
backbone_crc, crc);
batadv_bla_send_request(backbone_gw);
@@ -904,7 +904,7 @@
batadv_dbg(BATADV_DBG_BLA, bat_priv,
"handle_request(): REQUEST vid %d (sent by %pM)...\n",
- BATADV_PRINT_VID(vid), ethhdr->h_source);
+ batadv_print_vid(vid), ethhdr->h_source);
batadv_bla_answer_request(bat_priv, primary_if, vid);
return true;
@@ -941,7 +941,7 @@
/* this must be an UNCLAIM frame */
batadv_dbg(BATADV_DBG_BLA, bat_priv,
"handle_unclaim(): UNCLAIM %pM on vid %d (sent by %pM)...\n",
- claim_addr, BATADV_PRINT_VID(vid), backbone_gw->orig);
+ claim_addr, batadv_print_vid(vid), backbone_gw->orig);
batadv_bla_del_claim(bat_priv, claim_addr, vid);
batadv_backbone_gw_put(backbone_gw);
@@ -1161,7 +1161,7 @@
if (ret == 1)
batadv_dbg(BATADV_DBG_BLA, bat_priv,
"bla_process_claim(): received a claim frame from another group. From: %pM on vid %d ...(hw_src %pM, hw_dst %pM)\n",
- ethhdr->h_source, BATADV_PRINT_VID(vid), hw_src,
+ ethhdr->h_source, batadv_print_vid(vid), hw_src,
hw_dst);
if (ret < 2)
@@ -1197,7 +1197,7 @@
batadv_dbg(BATADV_DBG_BLA, bat_priv,
"bla_process_claim(): ERROR - this looks like a claim frame, but is useless. eth src %pM on vid %d ...(hw_src %pM, hw_dst %pM)\n",
- ethhdr->h_source, BATADV_PRINT_VID(vid), hw_src, hw_dst);
+ ethhdr->h_source, batadv_print_vid(vid), hw_src, hw_dst);
return true;
}
@@ -1295,10 +1295,13 @@
goto skip;
batadv_dbg(BATADV_DBG_BLA, bat_priv,
- "bla_purge_claims(): %pM, vid %d, time out\n",
- claim->addr, claim->vid);
+ "bla_purge_claims(): timed out.\n");
purge_now:
+ batadv_dbg(BATADV_DBG_BLA, bat_priv,
+ "bla_purge_claims(): %pM, vid %d\n",
+ claim->addr, claim->vid);
+
batadv_handle_unclaim(bat_priv, primary_if,
backbone_gw->orig,
claim->addr, claim->vid);
@@ -1846,6 +1849,13 @@
/* possible optimization: race for a claim */
/* No claim exists yet, claim it for us!
*/
+
+ batadv_dbg(BATADV_DBG_BLA, bat_priv,
+ "bla_rx(): Unclaimed MAC %pM found. Claim it. Local: %s\n",
+ ethhdr->h_source,
+ batadv_is_my_client(bat_priv,
+ ethhdr->h_source, vid) ?
+ "yes" : "no");
batadv_handle_claim(bat_priv, primary_if,
primary_if->net_dev->dev_addr,
ethhdr->h_source, vid);
@@ -1963,10 +1973,22 @@
/* if yes, the client has roamed and we have
* to unclaim it.
*/
- batadv_handle_unclaim(bat_priv, primary_if,
- primary_if->net_dev->dev_addr,
- ethhdr->h_source, vid);
- goto allow;
+ if (batadv_has_timed_out(claim->lasttime, 100)) {
+ /* only unclaim if the last claim entry is
+ * older than 100 ms to make sure we really
+ * have a roaming client here.
+ */
+ batadv_dbg(BATADV_DBG_BLA, bat_priv, "bla_tx(): Roaming client %pM detected. Unclaim it.\n",
+ ethhdr->h_source);
+ batadv_handle_unclaim(bat_priv, primary_if,
+ primary_if->net_dev->dev_addr,
+ ethhdr->h_source, vid);
+ goto allow;
+ } else {
+ batadv_dbg(BATADV_DBG_BLA, bat_priv, "bla_tx(): Race for claim %pM detected. Drop packet.\n",
+ ethhdr->h_source);
+ goto handled;
+ }
}
/* check if it is a multicast/broadcast frame */
@@ -2042,7 +2064,7 @@
backbone_crc = backbone_gw->crc;
spin_unlock_bh(&backbone_gw->crc_lock);
seq_printf(seq, " * %pM on %5d by %pM [%c] (%#.4x)\n",
- claim->addr, BATADV_PRINT_VID(claim->vid),
+ claim->addr, batadv_print_vid(claim->vid),
backbone_gw->orig,
(is_own ? 'x' : ' '),
backbone_crc);
@@ -2274,7 +2296,7 @@
seq_printf(seq, " * %pM on %5d %4i.%03is (%#.4x)\n",
backbone_gw->orig,
- BATADV_PRINT_VID(backbone_gw->vid), secs,
+ batadv_print_vid(backbone_gw->vid), secs,
msecs, backbone_crc);
}
rcu_read_unlock();
@@ -2449,3 +2471,52 @@
return ret;
}
+
+#ifdef CONFIG_BATMAN_ADV_DAT
+/**
+ * batadv_bla_check_claim - check if address is claimed
+ *
+ * @bat_priv: the bat priv with all the soft interface information
+ * @addr: mac address of which the claim status is checked
+ * @vid: the VLAN ID
+ *
+ * addr is checked if this address is claimed by the local device itself.
+ *
+ * Return: true if bla is disabled or the mac is claimed by the device,
+ * false if the device addr is already claimed by another gateway
+ */
+bool batadv_bla_check_claim(struct batadv_priv *bat_priv,
+ u8 *addr, unsigned short vid)
+{
+ struct batadv_bla_claim search_claim;
+ struct batadv_bla_claim *claim = NULL;
+ struct batadv_hard_iface *primary_if = NULL;
+ bool ret = true;
+
+ if (!atomic_read(&bat_priv->bridge_loop_avoidance))
+ return ret;
+
+ primary_if = batadv_primary_if_get_selected(bat_priv);
+ if (!primary_if)
+ return ret;
+
+ /* First look if the mac address is claimed */
+ ether_addr_copy(search_claim.addr, addr);
+ search_claim.vid = vid;
+
+ claim = batadv_claim_hash_find(bat_priv, &search_claim);
+
+ /* If there is a claim and we are not owner of the claim,
+ * return false.
+ */
+ if (claim) {
+ if (!batadv_compare_eth(claim->backbone_gw->orig,
+ primary_if->net_dev->dev_addr))
+ ret = false;
+ batadv_claim_put(claim);
+ }
+
+ batadv_hardif_put(primary_if);
+ return ret;
+}
+#endif
diff --git a/net/batman-adv/bridge_loop_avoidance.h b/net/batman-adv/bridge_loop_avoidance.h
index e157986..2347757 100644
--- a/net/batman-adv/bridge_loop_avoidance.h
+++ b/net/batman-adv/bridge_loop_avoidance.h
@@ -69,6 +69,10 @@
int batadv_bla_init(struct batadv_priv *bat_priv);
void batadv_bla_free(struct batadv_priv *bat_priv);
int batadv_bla_claim_dump(struct sk_buff *msg, struct netlink_callback *cb);
+#ifdef CONFIG_BATMAN_ADV_DAT
+bool batadv_bla_check_claim(struct batadv_priv *bat_priv, u8 *addr,
+ unsigned short vid);
+#endif
#define BATADV_BLA_CRC_INIT 0
#else /* ifdef CONFIG_BATMAN_ADV_BLA */
@@ -145,6 +149,13 @@
return -EOPNOTSUPP;
}
+static inline
+bool batadv_bla_check_claim(struct batadv_priv *bat_priv, u8 *addr,
+ unsigned short vid)
+{
+ return true;
+}
+
#endif /* ifdef CONFIG_BATMAN_ADV_BLA */
#endif /* ifndef _NET_BATMAN_ADV_BLA_H_ */
diff --git a/net/batman-adv/distributed-arp-table.c b/net/batman-adv/distributed-arp-table.c
index 1bfd1db..013e970 100644
--- a/net/batman-adv/distributed-arp-table.c
+++ b/net/batman-adv/distributed-arp-table.c
@@ -43,6 +43,7 @@
#include <linux/workqueue.h>
#include <net/arp.h>
+#include "bridge_loop_avoidance.h"
#include "hard-interface.h"
#include "hash.h"
#include "log.h"
@@ -330,7 +331,7 @@
batadv_dbg(BATADV_DBG_DAT, bat_priv,
"Entry updated: %pI4 %pM (vid: %d)\n",
&dat_entry->ip, dat_entry->mac_addr,
- BATADV_PRINT_VID(vid));
+ batadv_print_vid(vid));
goto out;
}
@@ -356,7 +357,7 @@
}
batadv_dbg(BATADV_DBG_DAT, bat_priv, "New entry added: %pI4 %pM (vid: %d)\n",
- &dat_entry->ip, dat_entry->mac_addr, BATADV_PRINT_VID(vid));
+ &dat_entry->ip, dat_entry->mac_addr, batadv_print_vid(vid));
out:
if (dat_entry)
@@ -835,7 +836,7 @@
seq_printf(seq, " * %15pI4 %14pM %4i %6i:%02i\n",
&dat_entry->ip, dat_entry->mac_addr,
- BATADV_PRINT_VID(dat_entry->vid),
+ batadv_print_vid(dat_entry->vid),
last_seen_mins, last_seen_secs);
}
rcu_read_unlock();
@@ -1002,6 +1003,7 @@
bool ret = false;
struct batadv_dat_entry *dat_entry = NULL;
struct sk_buff *skb_new;
+ struct net_device *soft_iface = bat_priv->soft_iface;
int hdr_size = 0;
unsigned short vid;
@@ -1040,16 +1042,30 @@
goto out;
}
+ /* If BLA is enabled, only send ARP replies if we have claimed
+ * the destination for the ARP request or if no one else of
+ * the backbone gws belonging to our backbone has claimed the
+ * destination.
+ */
+ if (!batadv_bla_check_claim(bat_priv,
+ dat_entry->mac_addr, vid)) {
+ batadv_dbg(BATADV_DBG_DAT, bat_priv,
+ "Device %pM claimed by another backbone gw. Don't send ARP reply!",
+ dat_entry->mac_addr);
+ ret = true;
+ goto out;
+ }
+
skb_new = batadv_dat_arp_create_reply(bat_priv, ip_dst, ip_src,
dat_entry->mac_addr,
hw_src, vid);
if (!skb_new)
goto out;
- skb_new->protocol = eth_type_trans(skb_new,
- bat_priv->soft_iface);
- bat_priv->stats.rx_packets++;
- bat_priv->stats.rx_bytes += skb->len + ETH_HLEN + hdr_size;
+ skb_new->protocol = eth_type_trans(skb_new, soft_iface);
+
+ soft_iface->stats.rx_packets++;
+ soft_iface->stats.rx_bytes += skb->len + ETH_HLEN + hdr_size;
netif_rx(skb_new);
batadv_dbg(BATADV_DBG_DAT, bat_priv, "ARP request replied locally\n");
@@ -1188,6 +1204,7 @@
bool batadv_dat_snoop_incoming_arp_reply(struct batadv_priv *bat_priv,
struct sk_buff *skb, int hdr_size)
{
+ struct batadv_dat_entry *dat_entry = NULL;
u16 type;
__be32 ip_src, ip_dst;
u8 *hw_src, *hw_dst;
@@ -1210,12 +1227,41 @@
hw_dst = batadv_arp_hw_dst(skb, hdr_size);
ip_dst = batadv_arp_ip_dst(skb, hdr_size);
+ /* If ip_dst is already in cache and has the right mac address,
+ * drop this frame if this ARP reply is destined for us because it's
+ * most probably an ARP reply generated by another node of the DHT.
+ * We have most probably received already a reply earlier. Delivering
+ * this frame would lead to doubled receive of an ARP reply.
+ */
+ dat_entry = batadv_dat_entry_hash_find(bat_priv, ip_src, vid);
+ if (dat_entry && batadv_compare_eth(hw_src, dat_entry->mac_addr)) {
+ batadv_dbg(BATADV_DBG_DAT, bat_priv, "Doubled ARP reply removed: ARP MSG = [src: %pM-%pI4 dst: %pM-%pI4]; dat_entry: %pM-%pI4\n",
+ hw_src, &ip_src, hw_dst, &ip_dst,
+ dat_entry->mac_addr, &dat_entry->ip);
+ dropped = true;
+ goto out;
+ }
+
/* Update our internal cache with both the IP addresses the node got
* within the ARP reply
*/
batadv_dat_entry_add(bat_priv, ip_src, hw_src, vid);
batadv_dat_entry_add(bat_priv, ip_dst, hw_dst, vid);
+ /* If BLA is enabled, only forward ARP replies if we have claimed the
+ * source of the ARP reply or if no one else of the same backbone has
+ * already claimed that client. This prevents that different gateways
+ * to the same backbone all forward the ARP reply leading to multiple
+ * replies in the backbone.
+ */
+ if (!batadv_bla_check_claim(bat_priv, hw_src, vid)) {
+ batadv_dbg(BATADV_DBG_DAT, bat_priv,
+ "Device %pM claimed by another backbone gw. Drop ARP reply.\n",
+ hw_src);
+ dropped = true;
+ goto out;
+ }
+
/* if this REPLY is directed to a client of mine, let's deliver the
* packet to the interface
*/
@@ -1228,6 +1274,8 @@
out:
if (dropped)
kfree_skb(skb);
+ if (dat_entry)
+ batadv_dat_entry_put(dat_entry);
/* if dropped == false -> deliver to the interface */
return dropped;
}
@@ -1256,7 +1304,7 @@
/* If this packet is an ARP_REQUEST and the node already has the
* information that it is going to ask, then the packet can be dropped
*/
- if (forw_packet->num_packets)
+ if (batadv_forw_packet_is_rebroadcast(forw_packet))
goto out;
vid = batadv_dat_get_vid(forw_packet->skb, &hdr_size);
diff --git a/net/batman-adv/log.h b/net/batman-adv/log.h
index 7a2b9f4..65ce97e 100644
--- a/net/batman-adv/log.h
+++ b/net/batman-adv/log.h
@@ -73,9 +73,10 @@
/* possibly ratelimited debug output */
#define _batadv_dbg(type, bat_priv, ratelimited, fmt, arg...) \
do { \
- if (atomic_read(&(bat_priv)->log_level) & (type) && \
+ struct batadv_priv *__batpriv = (bat_priv); \
+ if (atomic_read(&__batpriv->log_level) & (type) && \
(!(ratelimited) || net_ratelimit())) \
- batadv_debug_log(bat_priv, fmt, ## arg); \
+ batadv_debug_log(__batpriv, fmt, ## arg); \
} \
while (0)
#else /* !CONFIG_BATMAN_ADV_DEBUG */
diff --git a/net/batman-adv/main.c b/net/batman-adv/main.c
index 5000c54..fb381fb 100644
--- a/net/batman-adv/main.c
+++ b/net/batman-adv/main.c
@@ -516,6 +516,9 @@
BUILD_BUG_ON(sizeof(struct batadv_tvlv_tt_change) != 12);
BUILD_BUG_ON(sizeof(struct batadv_tvlv_roam_adv) != 8);
+ i = FIELD_SIZEOF(struct sk_buff, cb);
+ BUILD_BUG_ON(sizeof(struct batadv_skb_cb) > i);
+
/* broadcast packet */
batadv_rx_handler[BATADV_BCAST] = batadv_recv_bcast_packet;
diff --git a/net/batman-adv/main.h b/net/batman-adv/main.h
index 57a8103..810f7d0 100644
--- a/net/batman-adv/main.h
+++ b/net/batman-adv/main.h
@@ -24,7 +24,7 @@
#define BATADV_DRIVER_DEVICE "batman-adv"
#ifndef BATADV_SOURCE_VERSION
-#define BATADV_SOURCE_VERSION "2017.0"
+#define BATADV_SOURCE_VERSION "2017.1"
#endif
/* B.A.T.M.A.N. parameters */
@@ -193,6 +193,7 @@
#include <linux/percpu.h>
#include <linux/types.h>
+#include "packet.h"
#include "types.h"
struct net_device;
@@ -200,8 +201,19 @@
struct seq_file;
struct sk_buff;
-#define BATADV_PRINT_VID(vid) (((vid) & BATADV_VLAN_HAS_TAG) ? \
- (int)((vid) & VLAN_VID_MASK) : -1)
+/**
+ * batadv_print_vid - return printable version of vid information
+ * @vid: the VLAN identifier
+ *
+ * Return: -1 when no VLAN is used, VLAN id otherwise
+ */
+static inline int batadv_print_vid(unsigned short vid)
+{
+ if (vid & BATADV_VLAN_HAS_TAG)
+ return (int)(vid & VLAN_VID_MASK);
+ else
+ return -1;
+}
extern struct list_head batadv_hardif_list;
diff --git a/net/batman-adv/multicast.c b/net/batman-adv/multicast.c
index 952ba81..d327670 100644
--- a/net/batman-adv/multicast.c
+++ b/net/batman-adv/multicast.c
@@ -494,9 +494,8 @@
if (!bridged)
goto update;
-#if !IS_ENABLED(CONFIG_BRIDGE_IGMP_SNOOPING)
- pr_warn_once("No bridge IGMP snooping compiled - multicast optimizations disabled\n");
-#endif
+ if (!IS_ENABLED(CONFIG_BRIDGE_IGMP_SNOOPING))
+ pr_warn_once("No bridge IGMP snooping compiled - multicast optimizations disabled\n");
querier4.exists = br_multicast_has_querier_anywhere(dev, ETH_P_IP);
querier4.shadowing = br_multicast_has_querier_adjacent(dev, ETH_P_IP);
@@ -671,7 +670,6 @@
return 0;
}
-#if IS_ENABLED(CONFIG_IPV6)
/**
* batadv_mcast_is_report_ipv6 - check for MLD reports
* @skb: the ethernet frame destined for the mesh
@@ -736,7 +734,6 @@
return 0;
}
-#endif
/**
* batadv_mcast_forw_mode_check - check for optimized forwarding potential
@@ -765,11 +762,12 @@
case ETH_P_IP:
return batadv_mcast_forw_mode_check_ipv4(bat_priv, skb,
is_unsnoopable);
-#if IS_ENABLED(CONFIG_IPV6)
case ETH_P_IPV6:
+ if (!IS_ENABLED(CONFIG_IPV6))
+ return -EINVAL;
+
return batadv_mcast_forw_mode_check_ipv6(bat_priv, skb,
is_unsnoopable);
-#endif
default:
return -EINVAL;
}
diff --git a/net/batman-adv/routing.c b/net/batman-adv/routing.c
index 7fd740b..e1ebe14 100644
--- a/net/batman-adv/routing.c
+++ b/net/batman-adv/routing.c
@@ -941,15 +941,17 @@
struct batadv_priv *bat_priv = netdev_priv(recv_if->soft_iface);
struct batadv_unicast_packet *unicast_packet;
struct batadv_unicast_4addr_packet *unicast_4addr_packet;
- u8 *orig_addr;
- struct batadv_orig_node *orig_node = NULL;
+ u8 *orig_addr, *orig_addr_gw;
+ struct batadv_orig_node *orig_node = NULL, *orig_node_gw = NULL;
int check, hdr_size = sizeof(*unicast_packet);
enum batadv_subtype subtype;
- bool is4addr;
+ struct ethhdr *ethhdr;
int ret = NET_RX_DROP;
+ bool is4addr, is_gw;
unicast_packet = (struct batadv_unicast_packet *)skb->data;
unicast_4addr_packet = (struct batadv_unicast_4addr_packet *)skb->data;
+ ethhdr = eth_hdr(skb);
is4addr = unicast_packet->packet_type == BATADV_UNICAST_4ADDR;
/* the caller function should have already pulled 2 bytes */
@@ -972,6 +974,23 @@
/* packet for me */
if (batadv_is_my_mac(bat_priv, unicast_packet->dest)) {
+ /* If this is a unicast packet from another backgone gw,
+ * drop it.
+ */
+ orig_addr_gw = ethhdr->h_source;
+ orig_node_gw = batadv_orig_hash_find(bat_priv, orig_addr_gw);
+ if (orig_node_gw) {
+ is_gw = batadv_bla_is_backbone_gw(skb, orig_node_gw,
+ hdr_size);
+ batadv_orig_node_put(orig_node_gw);
+ if (is_gw) {
+ batadv_dbg(BATADV_DBG_BLA, bat_priv,
+ "recv_unicast_packet(): Dropped unicast pkt received from another backbone gw %pM.\n",
+ orig_addr_gw);
+ return NET_RX_DROP;
+ }
+ }
+
if (is4addr) {
subtype = unicast_4addr_packet->subtype;
batadv_dat_inc_counter(bat_priv, subtype);
diff --git a/net/batman-adv/send.c b/net/batman-adv/send.c
index 1489ec2..403df59 100644
--- a/net/batman-adv/send.c
+++ b/net/batman-adv/send.c
@@ -482,6 +482,7 @@
* @if_outgoing: The (optional) if_outgoing to be grabbed
* @queue_left: The (optional) queue counter to decrease
* @bat_priv: The bat_priv for the mesh of this forw_packet
+ * @skb: The raw packet this forwarding packet shall contain
*
* Allocates a forwarding packet and tries to get a reference to the
* (optional) if_incoming, if_outgoing and queue_left. If queue_left
@@ -493,7 +494,8 @@
batadv_forw_packet_alloc(struct batadv_hard_iface *if_incoming,
struct batadv_hard_iface *if_outgoing,
atomic_t *queue_left,
- struct batadv_priv *bat_priv)
+ struct batadv_priv *bat_priv,
+ struct sk_buff *skb)
{
struct batadv_forw_packet *forw_packet;
const char *qname;
@@ -525,7 +527,7 @@
INIT_HLIST_NODE(&forw_packet->list);
INIT_HLIST_NODE(&forw_packet->cleanup_list);
- forw_packet->skb = NULL;
+ forw_packet->skb = skb;
forw_packet->queue_left = queue_left;
forw_packet->if_incoming = if_incoming;
forw_packet->if_outgoing = if_outgoing;
@@ -756,22 +758,23 @@
if (!primary_if)
goto err;
+ newskb = skb_copy(skb, GFP_ATOMIC);
+ if (!newskb) {
+ batadv_hardif_put(primary_if);
+ goto err;
+ }
+
forw_packet = batadv_forw_packet_alloc(primary_if, NULL,
&bat_priv->bcast_queue_left,
- bat_priv);
+ bat_priv, newskb);
batadv_hardif_put(primary_if);
if (!forw_packet)
- goto err;
-
- newskb = skb_copy(skb, GFP_ATOMIC);
- if (!newskb)
goto err_packet_free;
/* as we have a copy now, it is safe to decrease the TTL */
bcast_packet = (struct batadv_bcast_packet *)newskb->data;
bcast_packet->ttl--;
- forw_packet->skb = newskb;
forw_packet->own = own_packet;
INIT_DELAYED_WORK(&forw_packet->delayed_work,
@@ -781,11 +784,60 @@
return NETDEV_TX_OK;
err_packet_free:
- batadv_forw_packet_free(forw_packet, true);
+ kfree_skb(newskb);
err:
return NETDEV_TX_BUSY;
}
+/**
+ * batadv_forw_packet_bcasts_left - check if a retransmission is necessary
+ * @forw_packet: the forwarding packet to check
+ * @hard_iface: the interface to check on
+ *
+ * Checks whether a given packet has any (re)transmissions left on the provided
+ * interface.
+ *
+ * hard_iface may be NULL: In that case the number of transmissions this skb had
+ * so far is compared with the maximum amount of retransmissions independent of
+ * any interface instead.
+ *
+ * Return: True if (re)transmissions are left, false otherwise.
+ */
+static bool
+batadv_forw_packet_bcasts_left(struct batadv_forw_packet *forw_packet,
+ struct batadv_hard_iface *hard_iface)
+{
+ unsigned int max;
+
+ if (hard_iface)
+ max = hard_iface->num_bcasts;
+ else
+ max = BATADV_NUM_BCASTS_MAX;
+
+ return BATADV_SKB_CB(forw_packet->skb)->num_bcasts < max;
+}
+
+/**
+ * batadv_forw_packet_bcasts_inc - increment retransmission counter of a packet
+ * @forw_packet: the packet to increase the counter for
+ */
+static void
+batadv_forw_packet_bcasts_inc(struct batadv_forw_packet *forw_packet)
+{
+ BATADV_SKB_CB(forw_packet->skb)->num_bcasts++;
+}
+
+/**
+ * batadv_forw_packet_is_rebroadcast - check packet for previous transmissions
+ * @forw_packet: the packet to check
+ *
+ * Return: True if this packet was transmitted before, false otherwise.
+ */
+bool batadv_forw_packet_is_rebroadcast(struct batadv_forw_packet *forw_packet)
+{
+ return BATADV_SKB_CB(forw_packet->skb)->num_bcasts > 0;
+}
+
static void batadv_send_outstanding_bcast_packet(struct work_struct *work)
{
struct batadv_hard_iface *hard_iface;
@@ -826,7 +878,7 @@
if (hard_iface->soft_iface != soft_iface)
continue;
- if (forw_packet->num_packets >= hard_iface->num_bcasts)
+ if (!batadv_forw_packet_bcasts_left(forw_packet, hard_iface))
continue;
if (forw_packet->own) {
@@ -884,10 +936,10 @@
}
rcu_read_unlock();
- forw_packet->num_packets++;
+ batadv_forw_packet_bcasts_inc(forw_packet);
/* if we still have some more bcasts to send */
- if (forw_packet->num_packets < BATADV_NUM_BCASTS_MAX) {
+ if (batadv_forw_packet_bcasts_left(forw_packet, NULL)) {
batadv_forw_packet_bcast_queue(bat_priv, forw_packet,
send_time);
return;
diff --git a/net/batman-adv/send.h b/net/batman-adv/send.h
index f21166d..a16b34f 100644
--- a/net/batman-adv/send.h
+++ b/net/batman-adv/send.h
@@ -34,11 +34,13 @@
batadv_forw_packet_alloc(struct batadv_hard_iface *if_incoming,
struct batadv_hard_iface *if_outgoing,
atomic_t *queue_left,
- struct batadv_priv *bat_priv);
+ struct batadv_priv *bat_priv,
+ struct sk_buff *skb);
bool batadv_forw_packet_steal(struct batadv_forw_packet *packet, spinlock_t *l);
void batadv_forw_packet_ogmv1_queue(struct batadv_priv *bat_priv,
struct batadv_forw_packet *forw_packet,
unsigned long send_time);
+bool batadv_forw_packet_is_rebroadcast(struct batadv_forw_packet *forw_packet);
int batadv_send_skb_to_orig(struct sk_buff *skb,
struct batadv_orig_node *orig_node,
diff --git a/net/batman-adv/soft-interface.c b/net/batman-adv/soft-interface.c
index d042c99..b25789a 100644
--- a/net/batman-adv/soft-interface.c
+++ b/net/batman-adv/soft-interface.c
@@ -64,28 +64,6 @@
#include "sysfs.h"
#include "translation-table.h"
-static int batadv_get_settings(struct net_device *dev, struct ethtool_cmd *cmd);
-static void batadv_get_drvinfo(struct net_device *dev,
- struct ethtool_drvinfo *info);
-static u32 batadv_get_msglevel(struct net_device *dev);
-static void batadv_set_msglevel(struct net_device *dev, u32 value);
-static u32 batadv_get_link(struct net_device *dev);
-static void batadv_get_strings(struct net_device *dev, u32 stringset, u8 *data);
-static void batadv_get_ethtool_stats(struct net_device *dev,
- struct ethtool_stats *stats, u64 *data);
-static int batadv_get_sset_count(struct net_device *dev, int stringset);
-
-static const struct ethtool_ops batadv_ethtool_ops = {
- .get_settings = batadv_get_settings,
- .get_drvinfo = batadv_get_drvinfo,
- .get_msglevel = batadv_get_msglevel,
- .set_msglevel = batadv_set_msglevel,
- .get_link = batadv_get_link,
- .get_strings = batadv_get_strings,
- .get_ethtool_stats = batadv_get_ethtool_stats,
- .get_sset_count = batadv_get_sset_count,
-};
-
int batadv_skb_head_push(struct sk_buff *skb, unsigned int len)
{
int result;
@@ -140,7 +118,7 @@
static struct net_device_stats *batadv_interface_stats(struct net_device *dev)
{
struct batadv_priv *bat_priv = netdev_priv(dev);
- struct net_device_stats *stats = &bat_priv->stats;
+ struct net_device_stats *stats = &dev->stats;
stats->tx_packets = batadv_sum_counter(bat_priv, BATADV_CNT_TX);
stats->tx_bytes = batadv_sum_counter(bat_priv, BATADV_CNT_TX_BYTES);
@@ -230,6 +208,9 @@
if (atomic_read(&bat_priv->mesh_state) != BATADV_MESH_ACTIVE)
goto dropped;
+ /* reset control block to avoid left overs from previous users */
+ memset(skb->cb, 0, sizeof(struct batadv_skb_cb));
+
netif_trans_update(soft_iface);
vid = batadv_get_vid(skb, 0);
ethhdr = eth_hdr(skb);
@@ -947,6 +928,98 @@
.ndo_del_slave = batadv_softif_slave_del,
};
+static void batadv_get_drvinfo(struct net_device *dev,
+ struct ethtool_drvinfo *info)
+{
+ strlcpy(info->driver, "B.A.T.M.A.N. advanced", sizeof(info->driver));
+ strlcpy(info->version, BATADV_SOURCE_VERSION, sizeof(info->version));
+ strlcpy(info->fw_version, "N/A", sizeof(info->fw_version));
+ strlcpy(info->bus_info, "batman", sizeof(info->bus_info));
+}
+
+/* Inspired by drivers/net/ethernet/dlink/sundance.c:1702
+ * Declare each description string in struct.name[] to get fixed sized buffer
+ * and compile time checking for strings longer than ETH_GSTRING_LEN.
+ */
+static const struct {
+ const char name[ETH_GSTRING_LEN];
+} batadv_counters_strings[] = {
+ { "tx" },
+ { "tx_bytes" },
+ { "tx_dropped" },
+ { "rx" },
+ { "rx_bytes" },
+ { "forward" },
+ { "forward_bytes" },
+ { "mgmt_tx" },
+ { "mgmt_tx_bytes" },
+ { "mgmt_rx" },
+ { "mgmt_rx_bytes" },
+ { "frag_tx" },
+ { "frag_tx_bytes" },
+ { "frag_rx" },
+ { "frag_rx_bytes" },
+ { "frag_fwd" },
+ { "frag_fwd_bytes" },
+ { "tt_request_tx" },
+ { "tt_request_rx" },
+ { "tt_response_tx" },
+ { "tt_response_rx" },
+ { "tt_roam_adv_tx" },
+ { "tt_roam_adv_rx" },
+#ifdef CONFIG_BATMAN_ADV_DAT
+ { "dat_get_tx" },
+ { "dat_get_rx" },
+ { "dat_put_tx" },
+ { "dat_put_rx" },
+ { "dat_cached_reply_tx" },
+#endif
+#ifdef CONFIG_BATMAN_ADV_NC
+ { "nc_code" },
+ { "nc_code_bytes" },
+ { "nc_recode" },
+ { "nc_recode_bytes" },
+ { "nc_buffer" },
+ { "nc_decode" },
+ { "nc_decode_bytes" },
+ { "nc_decode_failed" },
+ { "nc_sniffed" },
+#endif
+};
+
+static void batadv_get_strings(struct net_device *dev, u32 stringset, u8 *data)
+{
+ if (stringset == ETH_SS_STATS)
+ memcpy(data, batadv_counters_strings,
+ sizeof(batadv_counters_strings));
+}
+
+static void batadv_get_ethtool_stats(struct net_device *dev,
+ struct ethtool_stats *stats, u64 *data)
+{
+ struct batadv_priv *bat_priv = netdev_priv(dev);
+ int i;
+
+ for (i = 0; i < BATADV_CNT_NUM; i++)
+ data[i] = batadv_sum_counter(bat_priv, i);
+}
+
+static int batadv_get_sset_count(struct net_device *dev, int stringset)
+{
+ if (stringset == ETH_SS_STATS)
+ return BATADV_CNT_NUM;
+
+ return -EOPNOTSUPP;
+}
+
+static const struct ethtool_ops batadv_ethtool_ops = {
+ .get_drvinfo = batadv_get_drvinfo,
+ .get_link = ethtool_op_get_link,
+ .get_strings = batadv_get_strings,
+ .get_ethtool_stats = batadv_get_ethtool_stats,
+ .get_sset_count = batadv_get_sset_count,
+};
+
/**
* batadv_softif_free - Deconstructor of batadv_soft_interface
* @dev: Device to cleanup and remove
@@ -971,8 +1044,6 @@
*/
static void batadv_softif_init_early(struct net_device *dev)
{
- struct batadv_priv *priv = netdev_priv(dev);
-
ether_setup(dev);
dev->netdev_ops = &batadv_netdev_ops;
@@ -989,8 +1060,6 @@
eth_hw_addr_random(dev);
dev->ethtool_ops = &batadv_ethtool_ops;
-
- memset(priv, 0, sizeof(*priv));
}
struct net_device *batadv_softif_create(struct net *net, const char *name)
@@ -1083,118 +1152,3 @@
.setup = batadv_softif_init_early,
.dellink = batadv_softif_destroy_netlink,
};
-
-/* ethtool */
-static int batadv_get_settings(struct net_device *dev, struct ethtool_cmd *cmd)
-{
- cmd->supported = 0;
- cmd->advertising = 0;
- ethtool_cmd_speed_set(cmd, SPEED_10);
- cmd->duplex = DUPLEX_FULL;
- cmd->port = PORT_TP;
- cmd->phy_address = 0;
- cmd->transceiver = XCVR_INTERNAL;
- cmd->autoneg = AUTONEG_DISABLE;
- cmd->maxtxpkt = 0;
- cmd->maxrxpkt = 0;
-
- return 0;
-}
-
-static void batadv_get_drvinfo(struct net_device *dev,
- struct ethtool_drvinfo *info)
-{
- strlcpy(info->driver, "B.A.T.M.A.N. advanced", sizeof(info->driver));
- strlcpy(info->version, BATADV_SOURCE_VERSION, sizeof(info->version));
- strlcpy(info->fw_version, "N/A", sizeof(info->fw_version));
- strlcpy(info->bus_info, "batman", sizeof(info->bus_info));
-}
-
-static u32 batadv_get_msglevel(struct net_device *dev)
-{
- return -EOPNOTSUPP;
-}
-
-static void batadv_set_msglevel(struct net_device *dev, u32 value)
-{
-}
-
-static u32 batadv_get_link(struct net_device *dev)
-{
- return 1;
-}
-
-/* Inspired by drivers/net/ethernet/dlink/sundance.c:1702
- * Declare each description string in struct.name[] to get fixed sized buffer
- * and compile time checking for strings longer than ETH_GSTRING_LEN.
- */
-static const struct {
- const char name[ETH_GSTRING_LEN];
-} batadv_counters_strings[] = {
- { "tx" },
- { "tx_bytes" },
- { "tx_dropped" },
- { "rx" },
- { "rx_bytes" },
- { "forward" },
- { "forward_bytes" },
- { "mgmt_tx" },
- { "mgmt_tx_bytes" },
- { "mgmt_rx" },
- { "mgmt_rx_bytes" },
- { "frag_tx" },
- { "frag_tx_bytes" },
- { "frag_rx" },
- { "frag_rx_bytes" },
- { "frag_fwd" },
- { "frag_fwd_bytes" },
- { "tt_request_tx" },
- { "tt_request_rx" },
- { "tt_response_tx" },
- { "tt_response_rx" },
- { "tt_roam_adv_tx" },
- { "tt_roam_adv_rx" },
-#ifdef CONFIG_BATMAN_ADV_DAT
- { "dat_get_tx" },
- { "dat_get_rx" },
- { "dat_put_tx" },
- { "dat_put_rx" },
- { "dat_cached_reply_tx" },
-#endif
-#ifdef CONFIG_BATMAN_ADV_NC
- { "nc_code" },
- { "nc_code_bytes" },
- { "nc_recode" },
- { "nc_recode_bytes" },
- { "nc_buffer" },
- { "nc_decode" },
- { "nc_decode_bytes" },
- { "nc_decode_failed" },
- { "nc_sniffed" },
-#endif
-};
-
-static void batadv_get_strings(struct net_device *dev, u32 stringset, u8 *data)
-{
- if (stringset == ETH_SS_STATS)
- memcpy(data, batadv_counters_strings,
- sizeof(batadv_counters_strings));
-}
-
-static void batadv_get_ethtool_stats(struct net_device *dev,
- struct ethtool_stats *stats, u64 *data)
-{
- struct batadv_priv *bat_priv = netdev_priv(dev);
- int i;
-
- for (i = 0; i < BATADV_CNT_NUM; i++)
- data[i] = batadv_sum_counter(bat_priv, i);
-}
-
-static int batadv_get_sset_count(struct net_device *dev, int stringset)
-{
- if (stringset == ETH_SS_STATS)
- return BATADV_CNT_NUM;
-
- return -EOPNOTSUPP;
-}
diff --git a/net/batman-adv/tp_meter.c b/net/batman-adv/tp_meter.c
index c94ebde..556f9a8 100644
--- a/net/batman-adv/tp_meter.c
+++ b/net/batman-adv/tp_meter.c
@@ -873,8 +873,8 @@
/* something went wrong during the preparation/transmission */
if (unlikely(err && err != BATADV_TP_REASON_CANT_SEND)) {
batadv_dbg(BATADV_DBG_TP_METER, bat_priv,
- "Meter: batadv_tp_send() cannot send packets (%d)\n",
- err);
+ "Meter: %s() cannot send packets (%d)\n",
+ __func__, err);
/* ensure nobody else tries to stop the thread now */
if (atomic_dec_and_test(&tp_vars->sending))
tp_vars->reason = err;
@@ -979,7 +979,8 @@
if (!tp_vars) {
spin_unlock_bh(&bat_priv->tp_list_lock);
batadv_dbg(BATADV_DBG_TP_METER, bat_priv,
- "Meter: batadv_tp_start cannot allocate list elements\n");
+ "Meter: %s cannot allocate list elements\n",
+ __func__);
batadv_tp_batctl_error_notify(BATADV_TP_REASON_MEMORY_ERROR,
dst, bat_priv, session_cookie);
return;
diff --git a/net/batman-adv/translation-table.c b/net/batman-adv/translation-table.c
index 6077a87..e75b493 100644
--- a/net/batman-adv/translation-table.c
+++ b/net/batman-adv/translation-table.c
@@ -617,7 +617,7 @@
batadv_dbg(BATADV_DBG_TT, bat_priv,
"Deleting global tt entry %pM (vid: %d): %s\n",
tt_global->common.addr,
- BATADV_PRINT_VID(tt_global->common.vid), message);
+ batadv_print_vid(tt_global->common.vid), message);
batadv_hash_remove(bat_priv->tt.global_hash, batadv_compare_tt,
batadv_choose_tt, &tt_global->common);
@@ -671,7 +671,7 @@
if (tt_local->common.flags & BATADV_TT_CLIENT_PENDING) {
batadv_dbg(BATADV_DBG_TT, bat_priv,
"Re-adding pending client %pM (vid: %d)\n",
- addr, BATADV_PRINT_VID(vid));
+ addr, batadv_print_vid(vid));
/* whatever the reason why the PENDING flag was set,
* this is a client which was enqueued to be removed in
* this orig_interval. Since it popped up again, the
@@ -684,7 +684,7 @@
if (tt_local->common.flags & BATADV_TT_CLIENT_ROAM) {
batadv_dbg(BATADV_DBG_TT, bat_priv,
"Roaming client %pM (vid: %d) came back to its original location\n",
- addr, BATADV_PRINT_VID(vid));
+ addr, batadv_print_vid(vid));
/* the ROAM flag is set because this client roamed away
* and the node got a roaming_advertisement message. Now
* that the client popped up again at its original
@@ -716,7 +716,7 @@
if (!vlan) {
net_ratelimited_function(batadv_info, soft_iface,
"adding TT local entry %pM to non-existent VLAN %d\n",
- addr, BATADV_PRINT_VID(vid));
+ addr, batadv_print_vid(vid));
kmem_cache_free(batadv_tl_cache, tt_local);
tt_local = NULL;
goto out;
@@ -724,7 +724,7 @@
batadv_dbg(BATADV_DBG_TT, bat_priv,
"Creating new local tt entry: %pM (vid: %d, ttvn: %d)\n",
- addr, BATADV_PRINT_VID(vid),
+ addr, batadv_print_vid(vid),
(u8)atomic_read(&bat_priv->tt.vn));
ether_addr_copy(tt_local->common.addr, addr);
@@ -1097,7 +1097,7 @@
seq_printf(seq,
" * %pM %4i [%c%c%c%c%c%c] %3u.%03u (%#.8x)\n",
tt_common_entry->addr,
- BATADV_PRINT_VID(tt_common_entry->vid),
+ batadv_print_vid(tt_common_entry->vid),
((tt_common_entry->flags &
BATADV_TT_CLIENT_ROAM) ? 'R' : '.'),
no_purge ? 'P' : '.',
@@ -1296,7 +1296,7 @@
batadv_dbg(BATADV_DBG_TT, bat_priv,
"Local tt entry (%pM, vid: %d) pending to be removed: %s\n",
tt_local_entry->common.addr,
- BATADV_PRINT_VID(tt_local_entry->common.vid), message);
+ batadv_print_vid(tt_local_entry->common.vid), message);
}
/**
@@ -1727,7 +1727,7 @@
batadv_dbg(BATADV_DBG_TT, bat_priv,
"Creating new global tt entry: %pM (vid: %d, via %pM)\n",
- common->addr, BATADV_PRINT_VID(common->vid),
+ common->addr, batadv_print_vid(common->vid),
orig_node->orig);
ret = true;
@@ -1835,7 +1835,7 @@
if (!vlan) {
seq_printf(seq,
" * Cannot retrieve VLAN %d for originator %pM\n",
- BATADV_PRINT_VID(tt_common_entry->vid),
+ batadv_print_vid(tt_common_entry->vid),
best_entry->orig_node->orig);
goto print_list;
}
@@ -1844,7 +1844,7 @@
seq_printf(seq,
" %c %pM %4i (%3u) via %pM (%3u) (%#.8x) [%c%c%c%c]\n",
'*', tt_global_entry->common.addr,
- BATADV_PRINT_VID(tt_global_entry->common.vid),
+ batadv_print_vid(tt_global_entry->common.vid),
best_entry->ttvn, best_entry->orig_node->orig,
last_ttvn, vlan->tt.crc,
((flags & BATADV_TT_CLIENT_ROAM) ? 'R' : '.'),
@@ -1867,7 +1867,7 @@
if (!vlan) {
seq_printf(seq,
" + Cannot retrieve VLAN %d for originator %pM\n",
- BATADV_PRINT_VID(tt_common_entry->vid),
+ batadv_print_vid(tt_common_entry->vid),
orig_entry->orig_node->orig);
continue;
}
@@ -1876,7 +1876,7 @@
seq_printf(seq,
" %c %pM %4d (%3u) via %pM (%3u) (%#.8x) [%c%c%c%c]\n",
'+', tt_global_entry->common.addr,
- BATADV_PRINT_VID(tt_global_entry->common.vid),
+ batadv_print_vid(tt_global_entry->common.vid),
orig_entry->ttvn, orig_entry->orig_node->orig,
last_ttvn, vlan->tt.crc,
((flags & BATADV_TT_CLIENT_ROAM) ? 'R' : '.'),
@@ -2213,7 +2213,7 @@
"Deleting %pM from global tt entry %pM (vid: %d): %s\n",
orig_node->orig,
tt_global_entry->common.addr,
- BATADV_PRINT_VID(vid), message);
+ batadv_print_vid(vid), message);
_batadv_tt_global_del_orig_entry(tt_global_entry,
orig_entry);
}
@@ -2253,12 +2253,13 @@
/* its the last one, mark for roaming. */
tt_global_entry->common.flags |= BATADV_TT_CLIENT_ROAM;
tt_global_entry->roam_at = jiffies;
- } else
+ } else {
/* there is another entry, we can simply delete this
* one and can still use the other one.
*/
batadv_tt_global_del_orig_node(bat_priv, tt_global_entry,
orig_node, message);
+ }
}
/**
@@ -2314,10 +2315,11 @@
/* local entry exists, case 2: client roamed to us. */
batadv_tt_global_del_orig_list(tt_global_entry);
batadv_tt_global_free(bat_priv, tt_global_entry, message);
- } else
+ } else {
/* no local entry exists, case 1: check for roaming */
batadv_tt_global_del_roaming(bat_priv, tt_global_entry,
orig_node, message);
+ }
out:
if (tt_global_entry)
@@ -2375,7 +2377,7 @@
batadv_dbg(BATADV_DBG_TT, bat_priv,
"Deleting global tt entry %pM (vid: %d): %s\n",
tt_global->common.addr,
- BATADV_PRINT_VID(vid), message);
+ batadv_print_vid(vid), message);
hlist_del_rcu(&tt_common_entry->hash_entry);
batadv_tt_global_entry_put(tt_global);
}
@@ -2435,7 +2437,7 @@
batadv_dbg(BATADV_DBG_TT, bat_priv,
"Deleting global tt entry %pM (vid: %d): %s\n",
tt_global->common.addr,
- BATADV_PRINT_VID(tt_global->common.vid),
+ batadv_print_vid(tt_global->common.vid),
msg);
hlist_del_rcu(&tt_common->hash_entry);
@@ -3650,7 +3652,7 @@
batadv_dbg(BATADV_DBG_TT, bat_priv,
"Sending ROAMING_ADV to %pM (client %pM, vid: %d)\n",
- orig_node->orig, client, BATADV_PRINT_VID(vid));
+ orig_node->orig, client, batadv_print_vid(vid));
batadv_inc_counter(bat_priv, BATADV_CNT_TT_ROAM_ADV_TX);
@@ -3773,7 +3775,7 @@
batadv_dbg(BATADV_DBG_TT, bat_priv,
"Deleting local tt entry (%pM, vid: %d): pending\n",
tt_common->addr,
- BATADV_PRINT_VID(tt_common->vid));
+ batadv_print_vid(tt_common->vid));
batadv_tt_local_size_dec(bat_priv, tt_common->vid);
hlist_del_rcu(&tt_common->hash_entry);
@@ -4017,7 +4019,7 @@
batadv_dbg(BATADV_DBG_TT, bat_priv,
"Added temporary global client (addr: %pM, vid: %d, orig: %pM)\n",
- addr, BATADV_PRINT_VID(vid), orig_node->orig);
+ addr, batadv_print_vid(vid), orig_node->orig);
ret = true;
out:
return ret;
diff --git a/net/batman-adv/types.h b/net/batman-adv/types.h
index 246f21b..ea43a64 100644
--- a/net/batman-adv/types.h
+++ b/net/batman-adv/types.h
@@ -1000,7 +1000,6 @@
* struct batadv_priv - per mesh interface data
* @mesh_state: current status of the mesh (inactive/active/deactivating)
* @soft_iface: net device which holds this struct as private data
- * @stats: structure holding the data for the ndo_get_stats() call
* @bat_counters: mesh internal traffic statistic counters (see batadv_counters)
* @aggregated_ogms: bool indicating whether OGM aggregation is enabled
* @bonding: bool indicating whether traffic bonding is enabled
@@ -1055,7 +1054,6 @@
struct batadv_priv {
atomic_t mesh_state;
struct net_device *soft_iface;
- struct net_device_stats stats;
u64 __percpu *bat_counters; /* Per cpu counters */
atomic_t aggregated_ogms;
atomic_t bonding;
@@ -1377,9 +1375,11 @@
* relevant to batman-adv in the skb->cb buffer in skbs.
* @decoded: Marks a skb as decoded, which is checked when searching for coding
* opportunities in network-coding.c
+ * @num_bcasts: Counter for broadcast packet retransmissions
*/
struct batadv_skb_cb {
bool decoded;
+ unsigned int num_bcasts;
};
/**
@@ -1392,7 +1392,7 @@
* @skb: bcast packet's skb buffer
* @packet_len: size of aggregated OGM packet inside the skb buffer
* @direct_link_flags: direct link flags for aggregated OGM packets
- * @num_packets: counter for bcast packet retransmission
+ * @num_packets: counter for aggregated OGMv1 packets
* @delayed_work: work queue callback item for packet sending
* @if_incoming: pointer to incoming hard-iface or primary iface if
* locally generated packet
diff --git a/net/bluetooth/6lowpan.c b/net/bluetooth/6lowpan.c
index d491529..6089599 100644
--- a/net/bluetooth/6lowpan.c
+++ b/net/bluetooth/6lowpan.c
@@ -20,6 +20,7 @@
#include <net/ipv6.h>
#include <net/ip6_route.h>
#include <net/addrconf.h>
+#include <net/pkt_sched.h>
#include <net/bluetooth/bluetooth.h>
#include <net/bluetooth/hci_core.h>
@@ -38,7 +39,6 @@
struct in6_addr addr;
struct in6_addr gw;
struct l2cap_chan *chan;
- int status;
};
#define lowpan_cb(skb) ((struct skb_cb *)((skb)->cb))
@@ -64,7 +64,7 @@
struct l2cap_chan *chan;
/* peer addresses in various formats */
- unsigned char eui64_addr[EUI64_ADDR_LEN];
+ unsigned char lladdr[ETH_ALEN];
struct in6_addr peer_addr;
};
@@ -270,28 +270,20 @@
}
static int iphc_decompress(struct sk_buff *skb, struct net_device *netdev,
- struct l2cap_chan *chan)
+ struct lowpan_peer *peer)
{
- const u8 *saddr, *daddr;
+ const u8 *saddr;
struct lowpan_btle_dev *dev;
- struct lowpan_peer *peer;
dev = lowpan_btle_dev(netdev);
- rcu_read_lock();
- peer = __peer_lookup_chan(dev, chan);
- rcu_read_unlock();
- if (!peer)
- return -EINVAL;
+ saddr = peer->lladdr;
- saddr = peer->eui64_addr;
- daddr = dev->netdev->dev_addr;
-
- return lowpan_header_decompress(skb, netdev, daddr, saddr);
+ return lowpan_header_decompress(skb, netdev, netdev->dev_addr, saddr);
}
static int recv_pkt(struct sk_buff *skb, struct net_device *dev,
- struct l2cap_chan *chan)
+ struct lowpan_peer *peer)
{
struct sk_buff *local_skb;
int ret;
@@ -344,8 +336,9 @@
local_skb->dev = dev;
- ret = iphc_decompress(local_skb, dev, chan);
+ ret = iphc_decompress(local_skb, dev, peer);
if (ret < 0) {
+ BT_DBG("iphc_decompress failed: %d", ret);
kfree_skb(local_skb);
goto drop;
}
@@ -365,6 +358,7 @@
consume_skb(local_skb);
consume_skb(skb);
} else {
+ BT_DBG("unknown packet type");
goto drop;
}
@@ -390,7 +384,7 @@
if (!dev || !dev->netdev)
return -ENOENT;
- err = recv_pkt(skb, dev->netdev, chan);
+ err = recv_pkt(skb, dev->netdev, peer);
if (err) {
BT_DBG("recv pkt %d", err);
err = -EAGAIN;
@@ -399,37 +393,6 @@
return err;
}
-static u8 get_addr_type_from_eui64(u8 byte)
-{
- /* Is universal(0) or local(1) bit */
- return ((byte & 0x02) ? BDADDR_LE_RANDOM : BDADDR_LE_PUBLIC);
-}
-
-static void copy_to_bdaddr(struct in6_addr *ip6_daddr, bdaddr_t *addr)
-{
- u8 *eui64 = ip6_daddr->s6_addr + 8;
-
- addr->b[0] = eui64[7];
- addr->b[1] = eui64[6];
- addr->b[2] = eui64[5];
- addr->b[3] = eui64[2];
- addr->b[4] = eui64[1];
- addr->b[5] = eui64[0];
-}
-
-static void convert_dest_bdaddr(struct in6_addr *ip6_daddr,
- bdaddr_t *addr, u8 *addr_type)
-{
- copy_to_bdaddr(ip6_daddr, addr);
-
- /* We need to toggle the U/L bit that we got from IPv6 address
- * so that we get the proper address and type of the BD address.
- */
- addr->b[5] ^= 0x02;
-
- *addr_type = get_addr_type_from_eui64(addr->b[5]);
-}
-
static int setup_header(struct sk_buff *skb, struct net_device *netdev,
bdaddr_t *peer_addr, u8 *peer_addr_type)
{
@@ -437,8 +400,7 @@
struct ipv6hdr *hdr;
struct lowpan_btle_dev *dev;
struct lowpan_peer *peer;
- bdaddr_t addr, *any = BDADDR_ANY;
- u8 *daddr = any->b;
+ u8 *daddr;
int err, status = 0;
hdr = ipv6_hdr(skb);
@@ -449,34 +411,24 @@
if (ipv6_addr_is_multicast(&ipv6_daddr)) {
lowpan_cb(skb)->chan = NULL;
+ daddr = NULL;
} else {
- u8 addr_type;
+ BT_DBG("dest IP %pI6c", &ipv6_daddr);
- /* Get destination BT device from skb.
- * If there is no such peer then discard the packet.
+ /* The packet might be sent to 6lowpan interface
+ * because of routing (either via default route
+ * or user set route) so get peer according to
+ * the destination address.
*/
- convert_dest_bdaddr(&ipv6_daddr, &addr, &addr_type);
-
- BT_DBG("dest addr %pMR type %d IP %pI6c", &addr,
- addr_type, &ipv6_daddr);
-
- peer = peer_lookup_ba(dev, &addr, addr_type);
+ peer = peer_lookup_dst(dev, &ipv6_daddr, skb);
if (!peer) {
- /* The packet might be sent to 6lowpan interface
- * because of routing (either via default route
- * or user set route) so get peer according to
- * the destination address.
- */
- peer = peer_lookup_dst(dev, &ipv6_daddr, skb);
- if (!peer) {
- BT_DBG("no such peer %pMR found", &addr);
- return -ENOENT;
- }
+ BT_DBG("no such peer");
+ return -ENOENT;
}
- daddr = peer->eui64_addr;
- *peer_addr = addr;
- *peer_addr_type = addr_type;
+ daddr = peer->lladdr;
+ *peer_addr = peer->chan->dst;
+ *peer_addr_type = peer->chan->dst_type;
lowpan_cb(skb)->chan = peer->chan;
status = 1;
@@ -527,15 +479,8 @@
return 0;
}
- if (!err)
- err = lowpan_cb(skb)->status;
-
- if (err < 0) {
- if (err == -EAGAIN)
- netdev->stats.tx_dropped++;
- else
- netdev->stats.tx_errors++;
- }
+ if (err < 0)
+ netdev->stats.tx_errors++;
return err;
}
@@ -647,9 +592,9 @@
{
dev->hard_header_len = 0;
dev->needed_tailroom = 0;
- dev->flags = IFF_RUNNING | IFF_POINTOPOINT |
- IFF_MULTICAST;
+ dev->flags = IFF_RUNNING | IFF_MULTICAST;
dev->watchdog_timeo = 0;
+ dev->tx_queue_len = DEFAULT_TX_QUEUE_LEN;
dev->netdev_ops = &netdev_ops;
dev->header_ops = &header_ops;
@@ -660,34 +605,6 @@
.name = "bluetooth",
};
-static void set_addr(u8 *eui, u8 *addr, u8 addr_type)
-{
- /* addr is the BT address in little-endian format */
- eui[0] = addr[5];
- eui[1] = addr[4];
- eui[2] = addr[3];
- eui[3] = 0xFF;
- eui[4] = 0xFE;
- eui[5] = addr[2];
- eui[6] = addr[1];
- eui[7] = addr[0];
-
- /* Universal/local bit set, BT 6lowpan draft ch. 3.2.1 */
- if (addr_type == BDADDR_LE_PUBLIC)
- eui[0] &= ~0x02;
- else
- eui[0] |= 0x02;
-
- BT_DBG("type %d addr %*phC", addr_type, 8, eui);
-}
-
-static void set_dev_addr(struct net_device *netdev, bdaddr_t *addr,
- u8 addr_type)
-{
- netdev->addr_assign_type = NET_ADDR_PERM;
- set_addr(netdev->dev_addr, addr->b, addr_type);
-}
-
static void ifup(struct net_device *netdev)
{
int err;
@@ -746,16 +663,9 @@
return chan;
}
-static void set_ip_addr_bits(u8 addr_type, u8 *addr)
-{
- if (addr_type == BDADDR_LE_PUBLIC)
- *addr |= 0x02;
- else
- *addr &= ~0x02;
-}
-
static struct l2cap_chan *add_peer_chan(struct l2cap_chan *chan,
- struct lowpan_btle_dev *dev)
+ struct lowpan_btle_dev *dev,
+ bool new_netdev)
{
struct lowpan_peer *peer;
@@ -766,19 +676,9 @@
peer->chan = chan;
memset(&peer->peer_addr, 0, sizeof(struct in6_addr));
- /* RFC 2464 ch. 5 */
- peer->peer_addr.s6_addr[0] = 0xFE;
- peer->peer_addr.s6_addr[1] = 0x80;
- set_addr((u8 *)&peer->peer_addr.s6_addr + 8, chan->dst.b,
- chan->dst_type);
+ baswap((void *)peer->lladdr, &chan->dst);
- memcpy(&peer->eui64_addr, (u8 *)&peer->peer_addr.s6_addr + 8,
- EUI64_ADDR_LEN);
-
- /* IPv6 address needs to have the U/L bit set properly so toggle
- * it back here.
- */
- set_ip_addr_bits(chan->dst_type, (u8 *)&peer->peer_addr.s6_addr + 8);
+ lowpan_iphc_uncompress_eui48_lladdr(&peer->peer_addr, peer->lladdr);
spin_lock(&devices_lock);
INIT_LIST_HEAD(&peer->list);
@@ -786,7 +686,8 @@
spin_unlock(&devices_lock);
/* Notifying peers about us needs to be done without locks held */
- INIT_DELAYED_WORK(&dev->notify_peers, do_notify_peers);
+ if (new_netdev)
+ INIT_DELAYED_WORK(&dev->notify_peers, do_notify_peers);
schedule_delayed_work(&dev->notify_peers, msecs_to_jiffies(100));
return peer->chan;
@@ -803,7 +704,8 @@
if (!netdev)
return -ENOMEM;
- set_dev_addr(netdev, &chan->src, chan->src_type);
+ netdev->addr_assign_type = NET_ADDR_PERM;
+ baswap((void *)netdev->dev_addr, &chan->src);
netdev->netdev_ops = &netdev_ops;
SET_NETDEV_DEV(netdev, &chan->conn->hcon->hdev->dev);
@@ -843,6 +745,7 @@
static inline void chan_ready_cb(struct l2cap_chan *chan)
{
struct lowpan_btle_dev *dev;
+ bool new_netdev = false;
dev = lookup_dev(chan->conn);
@@ -853,12 +756,13 @@
l2cap_chan_del(chan, -ENOENT);
return;
}
+ new_netdev = true;
}
if (!try_module_get(THIS_MODULE))
return;
- add_peer_chan(chan, dev);
+ add_peer_chan(chan, dev, new_netdev);
ifup(dev->netdev);
}
@@ -964,26 +868,28 @@
static void chan_suspend_cb(struct l2cap_chan *chan)
{
- struct sk_buff *skb = chan->data;
+ struct lowpan_btle_dev *dev;
- BT_DBG("chan %p conn %p skb %p", chan, chan->conn, skb);
+ BT_DBG("chan %p suspend", chan);
- if (!skb)
+ dev = lookup_dev(chan->conn);
+ if (!dev || !dev->netdev)
return;
- lowpan_cb(skb)->status = -EAGAIN;
+ netif_stop_queue(dev->netdev);
}
static void chan_resume_cb(struct l2cap_chan *chan)
{
- struct sk_buff *skb = chan->data;
+ struct lowpan_btle_dev *dev;
- BT_DBG("chan %p conn %p skb %p", chan, chan->conn, skb);
+ BT_DBG("chan %p resume", chan);
- if (!skb)
+ dev = lookup_dev(chan->conn);
+ if (!dev || !dev->netdev)
return;
- lowpan_cb(skb)->status = 0;
+ netif_wake_queue(dev->netdev);
}
static long chan_get_sndtimeo_cb(struct l2cap_chan *chan)
diff --git a/net/bluetooth/Kconfig b/net/bluetooth/Kconfig
index 06c31b9..68f951b 100644
--- a/net/bluetooth/Kconfig
+++ b/net/bluetooth/Kconfig
@@ -13,6 +13,7 @@
select CRYPTO_CMAC
select CRYPTO_ECB
select CRYPTO_SHA256
+ select CRYPTO_ECDH
help
Bluetooth is low-cost, low-power, short-range wireless technology.
It was designed as a replacement for cables and other short-range
diff --git a/net/bluetooth/Makefile b/net/bluetooth/Makefile
index 4bfaa19..5d0a113 100644
--- a/net/bluetooth/Makefile
+++ b/net/bluetooth/Makefile
@@ -13,7 +13,7 @@
bluetooth-y := af_bluetooth.o hci_core.o hci_conn.o hci_event.o mgmt.o \
hci_sock.o hci_sysfs.o l2cap_core.o l2cap_sock.o smp.o lib.o \
- ecc.o hci_request.o mgmt_util.o
+ ecdh_helper.o hci_request.o mgmt_util.o
bluetooth-$(CONFIG_BT_BREDR) += sco.o
bluetooth-$(CONFIG_BT_HS) += a2mp.o amp.o
diff --git a/net/bluetooth/af_bluetooth.c b/net/bluetooth/af_bluetooth.c
index 69e1f7d..42d0997 100644
--- a/net/bluetooth/af_bluetooth.c
+++ b/net/bluetooth/af_bluetooth.c
@@ -159,12 +159,17 @@
BT_DBG("parent %p, sk %p", parent, sk);
sock_hold(sk);
+ lock_sock(sk);
list_add_tail(&bt_sk(sk)->accept_q, &bt_sk(parent)->accept_q);
bt_sk(sk)->parent = parent;
+ release_sock(sk);
parent->sk_ack_backlog++;
}
EXPORT_SYMBOL(bt_accept_enqueue);
+/* Calling function must hold the sk lock.
+ * bt_sk(sk)->parent must be non-NULL meaning sk is in the parent list.
+ */
void bt_accept_unlink(struct sock *sk)
{
BT_DBG("sk %p state %d", sk, sk->sk_state);
@@ -183,11 +188,32 @@
BT_DBG("parent %p", parent);
+restart:
list_for_each_entry_safe(s, n, &bt_sk(parent)->accept_q, accept_q) {
sk = (struct sock *)s;
+ /* Prevent early freeing of sk due to unlink and sock_kill */
+ sock_hold(sk);
lock_sock(sk);
+ /* Check sk has not already been unlinked via
+ * bt_accept_unlink() due to serialisation caused by sk locking
+ */
+ if (!bt_sk(sk)->parent) {
+ BT_DBG("sk %p, already unlinked", sk);
+ release_sock(sk);
+ sock_put(sk);
+
+ /* Restart the loop as sk is no longer in the list
+ * and also avoid a potential infinite loop because
+ * list_for_each_entry_safe() is not thread safe.
+ */
+ goto restart;
+ }
+
+ /* sk is safely in the parent list so reduce reference count */
+ sock_put(sk);
+
/* FIXME: Is this check still needed */
if (sk->sk_state == BT_CLOSED) {
bt_accept_unlink(sk);
diff --git a/net/bluetooth/amp.c b/net/bluetooth/amp.c
index 02a4ccc..ebcab5b 100644
--- a/net/bluetooth/amp.c
+++ b/net/bluetooth/amp.c
@@ -263,7 +263,7 @@
struct hci_cp_read_local_amp_assoc cp;
struct amp_assoc *loc_assoc = &hdev->loc_assoc;
struct hci_request req;
- int err = 0;
+ int err;
BT_DBG("%s handle %d", hdev->name, phy_handle);
@@ -282,7 +282,7 @@
{
struct hci_cp_read_local_amp_assoc cp;
struct hci_request req;
- int err = 0;
+ int err;
memset(&hdev->loc_assoc, 0, sizeof(struct amp_assoc));
memset(&cp, 0, sizeof(cp));
@@ -292,7 +292,7 @@
set_bit(READ_LOC_AMP_ASSOC, &mgr->state);
hci_req_init(&req, hdev);
hci_req_add(&req, HCI_OP_READ_LOCAL_AMP_ASSOC, sizeof(cp), &cp);
- hci_req_run_skb(&req, read_local_amp_assoc_complete);
+ err = hci_req_run_skb(&req, read_local_amp_assoc_complete);
if (err < 0)
a2mp_send_getampassoc_rsp(hdev, A2MP_STATUS_INVALID_CTRL_ID);
}
@@ -303,7 +303,7 @@
struct hci_cp_read_local_amp_assoc cp;
struct amp_mgr *mgr = hcon->amp_mgr;
struct hci_request req;
- int err = 0;
+ int err;
cp.phy_handle = hcon->handle;
cp.len_so_far = cpu_to_le16(0);
@@ -314,7 +314,7 @@
/* Read Local AMP Assoc final link information data */
hci_req_init(&req, hdev);
hci_req_add(&req, HCI_OP_READ_LOCAL_AMP_ASSOC, sizeof(cp), &cp);
- hci_req_run_skb(&req, read_local_amp_assoc_complete);
+ err = hci_req_run_skb(&req, read_local_amp_assoc_complete);
if (err < 0)
a2mp_send_getampassoc_rsp(hdev, A2MP_STATUS_INVALID_CTRL_ID);
}
diff --git a/net/bluetooth/ecc.c b/net/bluetooth/ecc.c
deleted file mode 100644
index e1709f8..0000000
--- a/net/bluetooth/ecc.c
+++ /dev/null
@@ -1,816 +0,0 @@
-/*
- * Copyright (c) 2013, Kenneth MacKay
- * All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are
- * met:
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above copyright
- * notice, this list of conditions and the following disclaimer in the
- * documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- * HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include <linux/random.h>
-
-#include "ecc.h"
-
-/* 256-bit curve */
-#define ECC_BYTES 32
-
-#define MAX_TRIES 16
-
-/* Number of u64's needed */
-#define NUM_ECC_DIGITS (ECC_BYTES / 8)
-
-struct ecc_point {
- u64 x[NUM_ECC_DIGITS];
- u64 y[NUM_ECC_DIGITS];
-};
-
-typedef struct {
- u64 m_low;
- u64 m_high;
-} uint128_t;
-
-#define CURVE_P_32 { 0xFFFFFFFFFFFFFFFFull, 0x00000000FFFFFFFFull, \
- 0x0000000000000000ull, 0xFFFFFFFF00000001ull }
-
-#define CURVE_G_32 { \
- { 0xF4A13945D898C296ull, 0x77037D812DEB33A0ull, \
- 0xF8BCE6E563A440F2ull, 0x6B17D1F2E12C4247ull }, \
- { 0xCBB6406837BF51F5ull, 0x2BCE33576B315ECEull, \
- 0x8EE7EB4A7C0F9E16ull, 0x4FE342E2FE1A7F9Bull } \
-}
-
-#define CURVE_N_32 { 0xF3B9CAC2FC632551ull, 0xBCE6FAADA7179E84ull, \
- 0xFFFFFFFFFFFFFFFFull, 0xFFFFFFFF00000000ull }
-
-static u64 curve_p[NUM_ECC_DIGITS] = CURVE_P_32;
-static struct ecc_point curve_g = CURVE_G_32;
-static u64 curve_n[NUM_ECC_DIGITS] = CURVE_N_32;
-
-static void vli_clear(u64 *vli)
-{
- int i;
-
- for (i = 0; i < NUM_ECC_DIGITS; i++)
- vli[i] = 0;
-}
-
-/* Returns true if vli == 0, false otherwise. */
-static bool vli_is_zero(const u64 *vli)
-{
- int i;
-
- for (i = 0; i < NUM_ECC_DIGITS; i++) {
- if (vli[i])
- return false;
- }
-
- return true;
-}
-
-/* Returns nonzero if bit bit of vli is set. */
-static u64 vli_test_bit(const u64 *vli, unsigned int bit)
-{
- return (vli[bit / 64] & ((u64) 1 << (bit % 64)));
-}
-
-/* Counts the number of 64-bit "digits" in vli. */
-static unsigned int vli_num_digits(const u64 *vli)
-{
- int i;
-
- /* Search from the end until we find a non-zero digit.
- * We do it in reverse because we expect that most digits will
- * be nonzero.
- */
- for (i = NUM_ECC_DIGITS - 1; i >= 0 && vli[i] == 0; i--);
-
- return (i + 1);
-}
-
-/* Counts the number of bits required for vli. */
-static unsigned int vli_num_bits(const u64 *vli)
-{
- unsigned int i, num_digits;
- u64 digit;
-
- num_digits = vli_num_digits(vli);
- if (num_digits == 0)
- return 0;
-
- digit = vli[num_digits - 1];
- for (i = 0; digit; i++)
- digit >>= 1;
-
- return ((num_digits - 1) * 64 + i);
-}
-
-/* Sets dest = src. */
-static void vli_set(u64 *dest, const u64 *src)
-{
- int i;
-
- for (i = 0; i < NUM_ECC_DIGITS; i++)
- dest[i] = src[i];
-}
-
-/* Returns sign of left - right. */
-static int vli_cmp(const u64 *left, const u64 *right)
-{
- int i;
-
- for (i = NUM_ECC_DIGITS - 1; i >= 0; i--) {
- if (left[i] > right[i])
- return 1;
- else if (left[i] < right[i])
- return -1;
- }
-
- return 0;
-}
-
-/* Computes result = in << c, returning carry. Can modify in place
- * (if result == in). 0 < shift < 64.
- */
-static u64 vli_lshift(u64 *result, const u64 *in,
- unsigned int shift)
-{
- u64 carry = 0;
- int i;
-
- for (i = 0; i < NUM_ECC_DIGITS; i++) {
- u64 temp = in[i];
-
- result[i] = (temp << shift) | carry;
- carry = temp >> (64 - shift);
- }
-
- return carry;
-}
-
-/* Computes vli = vli >> 1. */
-static void vli_rshift1(u64 *vli)
-{
- u64 *end = vli;
- u64 carry = 0;
-
- vli += NUM_ECC_DIGITS;
-
- while (vli-- > end) {
- u64 temp = *vli;
- *vli = (temp >> 1) | carry;
- carry = temp << 63;
- }
-}
-
-/* Computes result = left + right, returning carry. Can modify in place. */
-static u64 vli_add(u64 *result, const u64 *left,
- const u64 *right)
-{
- u64 carry = 0;
- int i;
-
- for (i = 0; i < NUM_ECC_DIGITS; i++) {
- u64 sum;
-
- sum = left[i] + right[i] + carry;
- if (sum != left[i])
- carry = (sum < left[i]);
-
- result[i] = sum;
- }
-
- return carry;
-}
-
-/* Computes result = left - right, returning borrow. Can modify in place. */
-static u64 vli_sub(u64 *result, const u64 *left, const u64 *right)
-{
- u64 borrow = 0;
- int i;
-
- for (i = 0; i < NUM_ECC_DIGITS; i++) {
- u64 diff;
-
- diff = left[i] - right[i] - borrow;
- if (diff != left[i])
- borrow = (diff > left[i]);
-
- result[i] = diff;
- }
-
- return borrow;
-}
-
-static uint128_t mul_64_64(u64 left, u64 right)
-{
- u64 a0 = left & 0xffffffffull;
- u64 a1 = left >> 32;
- u64 b0 = right & 0xffffffffull;
- u64 b1 = right >> 32;
- u64 m0 = a0 * b0;
- u64 m1 = a0 * b1;
- u64 m2 = a1 * b0;
- u64 m3 = a1 * b1;
- uint128_t result;
-
- m2 += (m0 >> 32);
- m2 += m1;
-
- /* Overflow */
- if (m2 < m1)
- m3 += 0x100000000ull;
-
- result.m_low = (m0 & 0xffffffffull) | (m2 << 32);
- result.m_high = m3 + (m2 >> 32);
-
- return result;
-}
-
-static uint128_t add_128_128(uint128_t a, uint128_t b)
-{
- uint128_t result;
-
- result.m_low = a.m_low + b.m_low;
- result.m_high = a.m_high + b.m_high + (result.m_low < a.m_low);
-
- return result;
-}
-
-static void vli_mult(u64 *result, const u64 *left, const u64 *right)
-{
- uint128_t r01 = { 0, 0 };
- u64 r2 = 0;
- unsigned int i, k;
-
- /* Compute each digit of result in sequence, maintaining the
- * carries.
- */
- for (k = 0; k < NUM_ECC_DIGITS * 2 - 1; k++) {
- unsigned int min;
-
- if (k < NUM_ECC_DIGITS)
- min = 0;
- else
- min = (k + 1) - NUM_ECC_DIGITS;
-
- for (i = min; i <= k && i < NUM_ECC_DIGITS; i++) {
- uint128_t product;
-
- product = mul_64_64(left[i], right[k - i]);
-
- r01 = add_128_128(r01, product);
- r2 += (r01.m_high < product.m_high);
- }
-
- result[k] = r01.m_low;
- r01.m_low = r01.m_high;
- r01.m_high = r2;
- r2 = 0;
- }
-
- result[NUM_ECC_DIGITS * 2 - 1] = r01.m_low;
-}
-
-static void vli_square(u64 *result, const u64 *left)
-{
- uint128_t r01 = { 0, 0 };
- u64 r2 = 0;
- int i, k;
-
- for (k = 0; k < NUM_ECC_DIGITS * 2 - 1; k++) {
- unsigned int min;
-
- if (k < NUM_ECC_DIGITS)
- min = 0;
- else
- min = (k + 1) - NUM_ECC_DIGITS;
-
- for (i = min; i <= k && i <= k - i; i++) {
- uint128_t product;
-
- product = mul_64_64(left[i], left[k - i]);
-
- if (i < k - i) {
- r2 += product.m_high >> 63;
- product.m_high = (product.m_high << 1) |
- (product.m_low >> 63);
- product.m_low <<= 1;
- }
-
- r01 = add_128_128(r01, product);
- r2 += (r01.m_high < product.m_high);
- }
-
- result[k] = r01.m_low;
- r01.m_low = r01.m_high;
- r01.m_high = r2;
- r2 = 0;
- }
-
- result[NUM_ECC_DIGITS * 2 - 1] = r01.m_low;
-}
-
-/* Computes result = (left + right) % mod.
- * Assumes that left < mod and right < mod, result != mod.
- */
-static void vli_mod_add(u64 *result, const u64 *left, const u64 *right,
- const u64 *mod)
-{
- u64 carry;
-
- carry = vli_add(result, left, right);
-
- /* result > mod (result = mod + remainder), so subtract mod to
- * get remainder.
- */
- if (carry || vli_cmp(result, mod) >= 0)
- vli_sub(result, result, mod);
-}
-
-/* Computes result = (left - right) % mod.
- * Assumes that left < mod and right < mod, result != mod.
- */
-static void vli_mod_sub(u64 *result, const u64 *left, const u64 *right,
- const u64 *mod)
-{
- u64 borrow = vli_sub(result, left, right);
-
- /* In this case, p_result == -diff == (max int) - diff.
- * Since -x % d == d - x, we can get the correct result from
- * result + mod (with overflow).
- */
- if (borrow)
- vli_add(result, result, mod);
-}
-
-/* Computes result = product % curve_p
- from http://www.nsa.gov/ia/_files/nist-routines.pdf */
-static void vli_mmod_fast(u64 *result, const u64 *product)
-{
- u64 tmp[NUM_ECC_DIGITS];
- int carry;
-
- /* t */
- vli_set(result, product);
-
- /* s1 */
- tmp[0] = 0;
- tmp[1] = product[5] & 0xffffffff00000000ull;
- tmp[2] = product[6];
- tmp[3] = product[7];
- carry = vli_lshift(tmp, tmp, 1);
- carry += vli_add(result, result, tmp);
-
- /* s2 */
- tmp[1] = product[6] << 32;
- tmp[2] = (product[6] >> 32) | (product[7] << 32);
- tmp[3] = product[7] >> 32;
- carry += vli_lshift(tmp, tmp, 1);
- carry += vli_add(result, result, tmp);
-
- /* s3 */
- tmp[0] = product[4];
- tmp[1] = product[5] & 0xffffffff;
- tmp[2] = 0;
- tmp[3] = product[7];
- carry += vli_add(result, result, tmp);
-
- /* s4 */
- tmp[0] = (product[4] >> 32) | (product[5] << 32);
- tmp[1] = (product[5] >> 32) | (product[6] & 0xffffffff00000000ull);
- tmp[2] = product[7];
- tmp[3] = (product[6] >> 32) | (product[4] << 32);
- carry += vli_add(result, result, tmp);
-
- /* d1 */
- tmp[0] = (product[5] >> 32) | (product[6] << 32);
- tmp[1] = (product[6] >> 32);
- tmp[2] = 0;
- tmp[3] = (product[4] & 0xffffffff) | (product[5] << 32);
- carry -= vli_sub(result, result, tmp);
-
- /* d2 */
- tmp[0] = product[6];
- tmp[1] = product[7];
- tmp[2] = 0;
- tmp[3] = (product[4] >> 32) | (product[5] & 0xffffffff00000000ull);
- carry -= vli_sub(result, result, tmp);
-
- /* d3 */
- tmp[0] = (product[6] >> 32) | (product[7] << 32);
- tmp[1] = (product[7] >> 32) | (product[4] << 32);
- tmp[2] = (product[4] >> 32) | (product[5] << 32);
- tmp[3] = (product[6] << 32);
- carry -= vli_sub(result, result, tmp);
-
- /* d4 */
- tmp[0] = product[7];
- tmp[1] = product[4] & 0xffffffff00000000ull;
- tmp[2] = product[5];
- tmp[3] = product[6] & 0xffffffff00000000ull;
- carry -= vli_sub(result, result, tmp);
-
- if (carry < 0) {
- do {
- carry += vli_add(result, result, curve_p);
- } while (carry < 0);
- } else {
- while (carry || vli_cmp(curve_p, result) != 1)
- carry -= vli_sub(result, result, curve_p);
- }
-}
-
-/* Computes result = (left * right) % curve_p. */
-static void vli_mod_mult_fast(u64 *result, const u64 *left, const u64 *right)
-{
- u64 product[2 * NUM_ECC_DIGITS];
-
- vli_mult(product, left, right);
- vli_mmod_fast(result, product);
-}
-
-/* Computes result = left^2 % curve_p. */
-static void vli_mod_square_fast(u64 *result, const u64 *left)
-{
- u64 product[2 * NUM_ECC_DIGITS];
-
- vli_square(product, left);
- vli_mmod_fast(result, product);
-}
-
-#define EVEN(vli) (!(vli[0] & 1))
-/* Computes result = (1 / p_input) % mod. All VLIs are the same size.
- * See "From Euclid's GCD to Montgomery Multiplication to the Great Divide"
- * https://labs.oracle.com/techrep/2001/smli_tr-2001-95.pdf
- */
-static void vli_mod_inv(u64 *result, const u64 *input, const u64 *mod)
-{
- u64 a[NUM_ECC_DIGITS], b[NUM_ECC_DIGITS];
- u64 u[NUM_ECC_DIGITS], v[NUM_ECC_DIGITS];
- u64 carry;
- int cmp_result;
-
- if (vli_is_zero(input)) {
- vli_clear(result);
- return;
- }
-
- vli_set(a, input);
- vli_set(b, mod);
- vli_clear(u);
- u[0] = 1;
- vli_clear(v);
-
- while ((cmp_result = vli_cmp(a, b)) != 0) {
- carry = 0;
-
- if (EVEN(a)) {
- vli_rshift1(a);
-
- if (!EVEN(u))
- carry = vli_add(u, u, mod);
-
- vli_rshift1(u);
- if (carry)
- u[NUM_ECC_DIGITS - 1] |= 0x8000000000000000ull;
- } else if (EVEN(b)) {
- vli_rshift1(b);
-
- if (!EVEN(v))
- carry = vli_add(v, v, mod);
-
- vli_rshift1(v);
- if (carry)
- v[NUM_ECC_DIGITS - 1] |= 0x8000000000000000ull;
- } else if (cmp_result > 0) {
- vli_sub(a, a, b);
- vli_rshift1(a);
-
- if (vli_cmp(u, v) < 0)
- vli_add(u, u, mod);
-
- vli_sub(u, u, v);
- if (!EVEN(u))
- carry = vli_add(u, u, mod);
-
- vli_rshift1(u);
- if (carry)
- u[NUM_ECC_DIGITS - 1] |= 0x8000000000000000ull;
- } else {
- vli_sub(b, b, a);
- vli_rshift1(b);
-
- if (vli_cmp(v, u) < 0)
- vli_add(v, v, mod);
-
- vli_sub(v, v, u);
- if (!EVEN(v))
- carry = vli_add(v, v, mod);
-
- vli_rshift1(v);
- if (carry)
- v[NUM_ECC_DIGITS - 1] |= 0x8000000000000000ull;
- }
- }
-
- vli_set(result, u);
-}
-
-/* ------ Point operations ------ */
-
-/* Returns true if p_point is the point at infinity, false otherwise. */
-static bool ecc_point_is_zero(const struct ecc_point *point)
-{
- return (vli_is_zero(point->x) && vli_is_zero(point->y));
-}
-
-/* Point multiplication algorithm using Montgomery's ladder with co-Z
- * coordinates. From http://eprint.iacr.org/2011/338.pdf
- */
-
-/* Double in place */
-static void ecc_point_double_jacobian(u64 *x1, u64 *y1, u64 *z1)
-{
- /* t1 = x, t2 = y, t3 = z */
- u64 t4[NUM_ECC_DIGITS];
- u64 t5[NUM_ECC_DIGITS];
-
- if (vli_is_zero(z1))
- return;
-
- vli_mod_square_fast(t4, y1); /* t4 = y1^2 */
- vli_mod_mult_fast(t5, x1, t4); /* t5 = x1*y1^2 = A */
- vli_mod_square_fast(t4, t4); /* t4 = y1^4 */
- vli_mod_mult_fast(y1, y1, z1); /* t2 = y1*z1 = z3 */
- vli_mod_square_fast(z1, z1); /* t3 = z1^2 */
-
- vli_mod_add(x1, x1, z1, curve_p); /* t1 = x1 + z1^2 */
- vli_mod_add(z1, z1, z1, curve_p); /* t3 = 2*z1^2 */
- vli_mod_sub(z1, x1, z1, curve_p); /* t3 = x1 - z1^2 */
- vli_mod_mult_fast(x1, x1, z1); /* t1 = x1^2 - z1^4 */
-
- vli_mod_add(z1, x1, x1, curve_p); /* t3 = 2*(x1^2 - z1^4) */
- vli_mod_add(x1, x1, z1, curve_p); /* t1 = 3*(x1^2 - z1^4) */
- if (vli_test_bit(x1, 0)) {
- u64 carry = vli_add(x1, x1, curve_p);
- vli_rshift1(x1);
- x1[NUM_ECC_DIGITS - 1] |= carry << 63;
- } else {
- vli_rshift1(x1);
- }
- /* t1 = 3/2*(x1^2 - z1^4) = B */
-
- vli_mod_square_fast(z1, x1); /* t3 = B^2 */
- vli_mod_sub(z1, z1, t5, curve_p); /* t3 = B^2 - A */
- vli_mod_sub(z1, z1, t5, curve_p); /* t3 = B^2 - 2A = x3 */
- vli_mod_sub(t5, t5, z1, curve_p); /* t5 = A - x3 */
- vli_mod_mult_fast(x1, x1, t5); /* t1 = B * (A - x3) */
- vli_mod_sub(t4, x1, t4, curve_p); /* t4 = B * (A - x3) - y1^4 = y3 */
-
- vli_set(x1, z1);
- vli_set(z1, y1);
- vli_set(y1, t4);
-}
-
-/* Modify (x1, y1) => (x1 * z^2, y1 * z^3) */
-static void apply_z(u64 *x1, u64 *y1, u64 *z)
-{
- u64 t1[NUM_ECC_DIGITS];
-
- vli_mod_square_fast(t1, z); /* z^2 */
- vli_mod_mult_fast(x1, x1, t1); /* x1 * z^2 */
- vli_mod_mult_fast(t1, t1, z); /* z^3 */
- vli_mod_mult_fast(y1, y1, t1); /* y1 * z^3 */
-}
-
-/* P = (x1, y1) => 2P, (x2, y2) => P' */
-static void xycz_initial_double(u64 *x1, u64 *y1, u64 *x2, u64 *y2,
- u64 *p_initial_z)
-{
- u64 z[NUM_ECC_DIGITS];
-
- vli_set(x2, x1);
- vli_set(y2, y1);
-
- vli_clear(z);
- z[0] = 1;
-
- if (p_initial_z)
- vli_set(z, p_initial_z);
-
- apply_z(x1, y1, z);
-
- ecc_point_double_jacobian(x1, y1, z);
-
- apply_z(x2, y2, z);
-}
-
-/* Input P = (x1, y1, Z), Q = (x2, y2, Z)
- * Output P' = (x1', y1', Z3), P + Q = (x3, y3, Z3)
- * or P => P', Q => P + Q
- */
-static void xycz_add(u64 *x1, u64 *y1, u64 *x2, u64 *y2)
-{
- /* t1 = X1, t2 = Y1, t3 = X2, t4 = Y2 */
- u64 t5[NUM_ECC_DIGITS];
-
- vli_mod_sub(t5, x2, x1, curve_p); /* t5 = x2 - x1 */
- vli_mod_square_fast(t5, t5); /* t5 = (x2 - x1)^2 = A */
- vli_mod_mult_fast(x1, x1, t5); /* t1 = x1*A = B */
- vli_mod_mult_fast(x2, x2, t5); /* t3 = x2*A = C */
- vli_mod_sub(y2, y2, y1, curve_p); /* t4 = y2 - y1 */
- vli_mod_square_fast(t5, y2); /* t5 = (y2 - y1)^2 = D */
-
- vli_mod_sub(t5, t5, x1, curve_p); /* t5 = D - B */
- vli_mod_sub(t5, t5, x2, curve_p); /* t5 = D - B - C = x3 */
- vli_mod_sub(x2, x2, x1, curve_p); /* t3 = C - B */
- vli_mod_mult_fast(y1, y1, x2); /* t2 = y1*(C - B) */
- vli_mod_sub(x2, x1, t5, curve_p); /* t3 = B - x3 */
- vli_mod_mult_fast(y2, y2, x2); /* t4 = (y2 - y1)*(B - x3) */
- vli_mod_sub(y2, y2, y1, curve_p); /* t4 = y3 */
-
- vli_set(x2, t5);
-}
-
-/* Input P = (x1, y1, Z), Q = (x2, y2, Z)
- * Output P + Q = (x3, y3, Z3), P - Q = (x3', y3', Z3)
- * or P => P - Q, Q => P + Q
- */
-static void xycz_add_c(u64 *x1, u64 *y1, u64 *x2, u64 *y2)
-{
- /* t1 = X1, t2 = Y1, t3 = X2, t4 = Y2 */
- u64 t5[NUM_ECC_DIGITS];
- u64 t6[NUM_ECC_DIGITS];
- u64 t7[NUM_ECC_DIGITS];
-
- vli_mod_sub(t5, x2, x1, curve_p); /* t5 = x2 - x1 */
- vli_mod_square_fast(t5, t5); /* t5 = (x2 - x1)^2 = A */
- vli_mod_mult_fast(x1, x1, t5); /* t1 = x1*A = B */
- vli_mod_mult_fast(x2, x2, t5); /* t3 = x2*A = C */
- vli_mod_add(t5, y2, y1, curve_p); /* t4 = y2 + y1 */
- vli_mod_sub(y2, y2, y1, curve_p); /* t4 = y2 - y1 */
-
- vli_mod_sub(t6, x2, x1, curve_p); /* t6 = C - B */
- vli_mod_mult_fast(y1, y1, t6); /* t2 = y1 * (C - B) */
- vli_mod_add(t6, x1, x2, curve_p); /* t6 = B + C */
- vli_mod_square_fast(x2, y2); /* t3 = (y2 - y1)^2 */
- vli_mod_sub(x2, x2, t6, curve_p); /* t3 = x3 */
-
- vli_mod_sub(t7, x1, x2, curve_p); /* t7 = B - x3 */
- vli_mod_mult_fast(y2, y2, t7); /* t4 = (y2 - y1)*(B - x3) */
- vli_mod_sub(y2, y2, y1, curve_p); /* t4 = y3 */
-
- vli_mod_square_fast(t7, t5); /* t7 = (y2 + y1)^2 = F */
- vli_mod_sub(t7, t7, t6, curve_p); /* t7 = x3' */
- vli_mod_sub(t6, t7, x1, curve_p); /* t6 = x3' - B */
- vli_mod_mult_fast(t6, t6, t5); /* t6 = (y2 + y1)*(x3' - B) */
- vli_mod_sub(y1, t6, y1, curve_p); /* t2 = y3' */
-
- vli_set(x1, t7);
-}
-
-static void ecc_point_mult(struct ecc_point *result,
- const struct ecc_point *point, u64 *scalar,
- u64 *initial_z, int num_bits)
-{
- /* R0 and R1 */
- u64 rx[2][NUM_ECC_DIGITS];
- u64 ry[2][NUM_ECC_DIGITS];
- u64 z[NUM_ECC_DIGITS];
- int i, nb;
-
- vli_set(rx[1], point->x);
- vli_set(ry[1], point->y);
-
- xycz_initial_double(rx[1], ry[1], rx[0], ry[0], initial_z);
-
- for (i = num_bits - 2; i > 0; i--) {
- nb = !vli_test_bit(scalar, i);
- xycz_add_c(rx[1 - nb], ry[1 - nb], rx[nb], ry[nb]);
- xycz_add(rx[nb], ry[nb], rx[1 - nb], ry[1 - nb]);
- }
-
- nb = !vli_test_bit(scalar, 0);
- xycz_add_c(rx[1 - nb], ry[1 - nb], rx[nb], ry[nb]);
-
- /* Find final 1/Z value. */
- vli_mod_sub(z, rx[1], rx[0], curve_p); /* X1 - X0 */
- vli_mod_mult_fast(z, z, ry[1 - nb]); /* Yb * (X1 - X0) */
- vli_mod_mult_fast(z, z, point->x); /* xP * Yb * (X1 - X0) */
- vli_mod_inv(z, z, curve_p); /* 1 / (xP * Yb * (X1 - X0)) */
- vli_mod_mult_fast(z, z, point->y); /* yP / (xP * Yb * (X1 - X0)) */
- vli_mod_mult_fast(z, z, rx[1 - nb]); /* Xb * yP / (xP * Yb * (X1 - X0)) */
- /* End 1/Z calculation */
-
- xycz_add(rx[nb], ry[nb], rx[1 - nb], ry[1 - nb]);
-
- apply_z(rx[0], ry[0], z);
-
- vli_set(result->x, rx[0]);
- vli_set(result->y, ry[0]);
-}
-
-static void ecc_bytes2native(const u8 bytes[ECC_BYTES],
- u64 native[NUM_ECC_DIGITS])
-{
- int i;
-
- for (i = 0; i < NUM_ECC_DIGITS; i++) {
- const u8 *digit = bytes + 8 * (NUM_ECC_DIGITS - 1 - i);
-
- native[NUM_ECC_DIGITS - 1 - i] =
- ((u64) digit[0] << 0) |
- ((u64) digit[1] << 8) |
- ((u64) digit[2] << 16) |
- ((u64) digit[3] << 24) |
- ((u64) digit[4] << 32) |
- ((u64) digit[5] << 40) |
- ((u64) digit[6] << 48) |
- ((u64) digit[7] << 56);
- }
-}
-
-static void ecc_native2bytes(const u64 native[NUM_ECC_DIGITS],
- u8 bytes[ECC_BYTES])
-{
- int i;
-
- for (i = 0; i < NUM_ECC_DIGITS; i++) {
- u8 *digit = bytes + 8 * (NUM_ECC_DIGITS - 1 - i);
-
- digit[0] = native[NUM_ECC_DIGITS - 1 - i] >> 0;
- digit[1] = native[NUM_ECC_DIGITS - 1 - i] >> 8;
- digit[2] = native[NUM_ECC_DIGITS - 1 - i] >> 16;
- digit[3] = native[NUM_ECC_DIGITS - 1 - i] >> 24;
- digit[4] = native[NUM_ECC_DIGITS - 1 - i] >> 32;
- digit[5] = native[NUM_ECC_DIGITS - 1 - i] >> 40;
- digit[6] = native[NUM_ECC_DIGITS - 1 - i] >> 48;
- digit[7] = native[NUM_ECC_DIGITS - 1 - i] >> 56;
- }
-}
-
-bool ecc_make_key(u8 public_key[64], u8 private_key[32])
-{
- struct ecc_point pk;
- u64 priv[NUM_ECC_DIGITS];
- unsigned int tries = 0;
-
- do {
- if (tries++ >= MAX_TRIES)
- return false;
-
- get_random_bytes(priv, ECC_BYTES);
-
- if (vli_is_zero(priv))
- continue;
-
- /* Make sure the private key is in the range [1, n-1]. */
- if (vli_cmp(curve_n, priv) != 1)
- continue;
-
- ecc_point_mult(&pk, &curve_g, priv, NULL, vli_num_bits(priv));
- } while (ecc_point_is_zero(&pk));
-
- ecc_native2bytes(priv, private_key);
- ecc_native2bytes(pk.x, public_key);
- ecc_native2bytes(pk.y, &public_key[32]);
-
- return true;
-}
-
-bool ecdh_shared_secret(const u8 public_key[64], const u8 private_key[32],
- u8 secret[32])
-{
- u64 priv[NUM_ECC_DIGITS];
- u64 rand[NUM_ECC_DIGITS];
- struct ecc_point product, pk;
-
- get_random_bytes(rand, ECC_BYTES);
-
- ecc_bytes2native(public_key, pk.x);
- ecc_bytes2native(&public_key[32], pk.y);
- ecc_bytes2native(private_key, priv);
-
- ecc_point_mult(&product, &pk, priv, rand, vli_num_bits(priv));
-
- ecc_native2bytes(product.x, secret);
-
- return !ecc_point_is_zero(&product);
-}
diff --git a/net/bluetooth/ecc.h b/net/bluetooth/ecc.h
deleted file mode 100644
index 8d6a2f4..0000000
--- a/net/bluetooth/ecc.h
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Copyright (c) 2013, Kenneth MacKay
- * All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are
- * met:
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above copyright
- * notice, this list of conditions and the following disclaimer in the
- * documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- * HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-/* Create a public/private key pair.
- * Outputs:
- * public_key - Will be filled in with the public key.
- * private_key - Will be filled in with the private key.
- *
- * Returns true if the key pair was generated successfully, false
- * if an error occurred. The keys are with the LSB first.
- */
-bool ecc_make_key(u8 public_key[64], u8 private_key[32]);
-
-/* Compute a shared secret given your secret key and someone else's
- * public key.
- * Note: It is recommended that you hash the result of ecdh_shared_secret
- * before using it for symmetric encryption or HMAC.
- *
- * Inputs:
- * public_key - The public key of the remote party
- * private_key - Your private key.
- *
- * Outputs:
- * secret - Will be filled in with the shared secret value.
- *
- * Returns true if the shared secret was generated successfully, false
- * if an error occurred. Both input and output parameters are with the
- * LSB first.
- */
-bool ecdh_shared_secret(const u8 public_key[64], const u8 private_key[32],
- u8 secret[32]);
diff --git a/net/bluetooth/ecdh_helper.c b/net/bluetooth/ecdh_helper.c
new file mode 100644
index 0000000..24d4e60
--- /dev/null
+++ b/net/bluetooth/ecdh_helper.c
@@ -0,0 +1,231 @@
+/*
+ * ECDH helper functions - KPP wrappings
+ *
+ * Copyright (C) 2017 Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation;
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+ * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS.
+ * IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) AND AUTHOR(S) BE LIABLE FOR ANY
+ * CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ *
+ * ALL LIABILITY, INCLUDING LIABILITY FOR INFRINGEMENT OF ANY PATENTS,
+ * COPYRIGHTS, TRADEMARKS OR OTHER RIGHTS, RELATING TO USE OF THIS
+ * SOFTWARE IS DISCLAIMED.
+ */
+#include "ecdh_helper.h"
+
+#include <linux/scatterlist.h>
+#include <crypto/kpp.h>
+#include <crypto/ecdh.h>
+
+struct ecdh_completion {
+ struct completion completion;
+ int err;
+};
+
+static void ecdh_complete(struct crypto_async_request *req, int err)
+{
+ struct ecdh_completion *res = req->data;
+
+ if (err == -EINPROGRESS)
+ return;
+
+ res->err = err;
+ complete(&res->completion);
+}
+
+static inline void swap_digits(u64 *in, u64 *out, unsigned int ndigits)
+{
+ int i;
+
+ for (i = 0; i < ndigits; i++)
+ out[i] = __swab64(in[ndigits - 1 - i]);
+}
+
+bool compute_ecdh_secret(const u8 public_key[64], const u8 private_key[32],
+ u8 secret[32])
+{
+ struct crypto_kpp *tfm;
+ struct kpp_request *req;
+ struct ecdh p;
+ struct ecdh_completion result;
+ struct scatterlist src, dst;
+ u8 *tmp, *buf;
+ unsigned int buf_len;
+ int err = -ENOMEM;
+
+ tmp = kmalloc(64, GFP_KERNEL);
+ if (!tmp)
+ return false;
+
+ tfm = crypto_alloc_kpp("ecdh", CRYPTO_ALG_INTERNAL, 0);
+ if (IS_ERR(tfm)) {
+ pr_err("alg: kpp: Failed to load tfm for kpp: %ld\n",
+ PTR_ERR(tfm));
+ goto free_tmp;
+ }
+
+ req = kpp_request_alloc(tfm, GFP_KERNEL);
+ if (!req)
+ goto free_kpp;
+
+ init_completion(&result.completion);
+
+ /* Security Manager Protocol holds digits in litte-endian order
+ * while ECC API expect big-endian data
+ */
+ swap_digits((u64 *)private_key, (u64 *)tmp, 4);
+ p.key = (char *)tmp;
+ p.key_size = 32;
+ /* Set curve_id */
+ p.curve_id = ECC_CURVE_NIST_P256;
+ buf_len = crypto_ecdh_key_len(&p);
+ buf = kmalloc(buf_len, GFP_KERNEL);
+ if (!buf) {
+ pr_err("alg: kpp: Failed to allocate %d bytes for buf\n",
+ buf_len);
+ goto free_req;
+ }
+ crypto_ecdh_encode_key(buf, buf_len, &p);
+
+ /* Set A private Key */
+ err = crypto_kpp_set_secret(tfm, (void *)buf, buf_len);
+ if (err)
+ goto free_all;
+
+ swap_digits((u64 *)public_key, (u64 *)tmp, 4); /* x */
+ swap_digits((u64 *)&public_key[32], (u64 *)&tmp[32], 4); /* y */
+
+ sg_init_one(&src, tmp, 64);
+ sg_init_one(&dst, secret, 32);
+ kpp_request_set_input(req, &src, 64);
+ kpp_request_set_output(req, &dst, 32);
+ kpp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
+ ecdh_complete, &result);
+ err = crypto_kpp_compute_shared_secret(req);
+ if (err == -EINPROGRESS) {
+ wait_for_completion(&result.completion);
+ err = result.err;
+ }
+ if (err < 0) {
+ pr_err("alg: ecdh: compute shared secret failed. err %d\n",
+ err);
+ goto free_all;
+ }
+
+ swap_digits((u64 *)secret, (u64 *)tmp, 4);
+ memcpy(secret, tmp, 32);
+
+free_all:
+ kzfree(buf);
+free_req:
+ kpp_request_free(req);
+free_kpp:
+ crypto_free_kpp(tfm);
+free_tmp:
+ kfree(tmp);
+ return (err == 0);
+}
+
+bool generate_ecdh_keys(u8 public_key[64], u8 private_key[32])
+{
+ struct crypto_kpp *tfm;
+ struct kpp_request *req;
+ struct ecdh p;
+ struct ecdh_completion result;
+ struct scatterlist dst;
+ u8 *tmp, *buf;
+ unsigned int buf_len;
+ int err = -ENOMEM;
+ const unsigned short max_tries = 16;
+ unsigned short tries = 0;
+
+ tmp = kmalloc(64, GFP_KERNEL);
+ if (!tmp)
+ return false;
+
+ tfm = crypto_alloc_kpp("ecdh", CRYPTO_ALG_INTERNAL, 0);
+ if (IS_ERR(tfm)) {
+ pr_err("alg: kpp: Failed to load tfm for kpp: %ld\n",
+ PTR_ERR(tfm));
+ goto free_tmp;
+ }
+
+ req = kpp_request_alloc(tfm, GFP_KERNEL);
+ if (!req)
+ goto free_kpp;
+
+ init_completion(&result.completion);
+
+ /* Set curve_id */
+ p.curve_id = ECC_CURVE_NIST_P256;
+ p.key_size = 32;
+ buf_len = crypto_ecdh_key_len(&p);
+ buf = kmalloc(buf_len, GFP_KERNEL);
+ if (!buf) {
+ pr_err("alg: kpp: Failed to allocate %d bytes for buf\n",
+ buf_len);
+ goto free_req;
+ }
+
+ do {
+ if (tries++ >= max_tries)
+ goto free_all;
+
+ /* Set private Key */
+ p.key = (char *)private_key;
+ crypto_ecdh_encode_key(buf, buf_len, &p);
+ err = crypto_kpp_set_secret(tfm, buf, buf_len);
+ if (err)
+ goto free_all;
+
+ sg_init_one(&dst, tmp, 64);
+ kpp_request_set_input(req, NULL, 0);
+ kpp_request_set_output(req, &dst, 64);
+ kpp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
+ ecdh_complete, &result);
+
+ err = crypto_kpp_generate_public_key(req);
+
+ if (err == -EINPROGRESS) {
+ wait_for_completion(&result.completion);
+ err = result.err;
+ }
+
+ /* Private key is not valid. Regenerate */
+ if (err == -EINVAL)
+ continue;
+
+ if (err < 0)
+ goto free_all;
+ else
+ break;
+
+ } while (true);
+
+ /* Keys are handed back in little endian as expected by Security
+ * Manager Protocol
+ */
+ swap_digits((u64 *)tmp, (u64 *)public_key, 4); /* x */
+ swap_digits((u64 *)&tmp[32], (u64 *)&public_key[32], 4); /* y */
+ swap_digits((u64 *)private_key, (u64 *)tmp, 4);
+ memcpy(private_key, tmp, 32);
+
+free_all:
+ kzfree(buf);
+free_req:
+ kpp_request_free(req);
+free_kpp:
+ crypto_free_kpp(tfm);
+free_tmp:
+ kfree(tmp);
+ return (err == 0);
+}
diff --git a/net/bluetooth/ecdh_helper.h b/net/bluetooth/ecdh_helper.h
new file mode 100644
index 0000000..7a423fa
--- /dev/null
+++ b/net/bluetooth/ecdh_helper.h
@@ -0,0 +1,27 @@
+/*
+ * ECDH helper functions - KPP wrappings
+ *
+ * Copyright (C) 2017 Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation;
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+ * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS.
+ * IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) AND AUTHOR(S) BE LIABLE FOR ANY
+ * CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ *
+ * ALL LIABILITY, INCLUDING LIABILITY FOR INFRINGEMENT OF ANY PATENTS,
+ * COPYRIGHTS, TRADEMARKS OR OTHER RIGHTS, RELATING TO USE OF THIS
+ * SOFTWARE IS DISCLAIMED.
+ */
+#include <linux/types.h>
+
+bool compute_ecdh_secret(const u8 pub_a[64], const u8 priv_b[32],
+ u8 secret[32]);
+bool generate_ecdh_keys(u8 public_key[64], u8 private_key[32]);
diff --git a/net/bluetooth/hci_core.c b/net/bluetooth/hci_core.c
index 3ac89e9..0568677 100644
--- a/net/bluetooth/hci_core.c
+++ b/net/bluetooth/hci_core.c
@@ -2950,8 +2950,8 @@
hdev->le_adv_max_interval = 0x0800;
hdev->le_scan_interval = 0x0060;
hdev->le_scan_window = 0x0030;
- hdev->le_conn_min_interval = 0x0028;
- hdev->le_conn_max_interval = 0x0038;
+ hdev->le_conn_min_interval = 0x0018;
+ hdev->le_conn_max_interval = 0x0028;
hdev->le_conn_latency = 0x0000;
hdev->le_supv_timeout = 0x002a;
hdev->le_def_tx_len = 0x001b;
diff --git a/net/bluetooth/hci_sock.c b/net/bluetooth/hci_sock.c
index f64d656..638bf0e 100644
--- a/net/bluetooth/hci_sock.c
+++ b/net/bluetooth/hci_sock.c
@@ -1680,7 +1680,8 @@
if (msg->msg_flags & MSG_OOB)
return -EOPNOTSUPP;
- if (msg->msg_flags & ~(MSG_DONTWAIT|MSG_NOSIGNAL|MSG_ERRQUEUE))
+ if (msg->msg_flags & ~(MSG_DONTWAIT|MSG_NOSIGNAL|MSG_ERRQUEUE|
+ MSG_CMSG_COMPAT))
return -EINVAL;
if (len < 4 || len > HCI_MAX_FRAME_SIZE)
diff --git a/net/bluetooth/l2cap_core.c b/net/bluetooth/l2cap_core.c
index fc7f321..f88ac995 100644
--- a/net/bluetooth/l2cap_core.c
+++ b/net/bluetooth/l2cap_core.c
@@ -2425,6 +2425,22 @@
return 0;
}
+static void l2cap_le_flowctl_send(struct l2cap_chan *chan)
+{
+ int sent = 0;
+
+ BT_DBG("chan %p", chan);
+
+ while (chan->tx_credits && !skb_queue_empty(&chan->tx_q)) {
+ l2cap_do_send(chan, skb_dequeue(&chan->tx_q));
+ chan->tx_credits--;
+ sent++;
+ }
+
+ BT_DBG("Sent %d credits %u queued %u", sent, chan->tx_credits,
+ skb_queue_len(&chan->tx_q));
+}
+
int l2cap_chan_send(struct l2cap_chan *chan, struct msghdr *msg, size_t len)
{
struct sk_buff *skb;
@@ -2458,9 +2474,6 @@
if (len > chan->omtu)
return -EMSGSIZE;
- if (!chan->tx_credits)
- return -EAGAIN;
-
__skb_queue_head_init(&seg_queue);
err = l2cap_segment_le_sdu(chan, &seg_queue, msg, len);
@@ -2475,10 +2488,7 @@
skb_queue_splice_tail_init(&seg_queue, &chan->tx_q);
- while (chan->tx_credits && !skb_queue_empty(&chan->tx_q)) {
- l2cap_do_send(chan, skb_dequeue(&chan->tx_q));
- chan->tx_credits--;
- }
+ l2cap_le_flowctl_send(chan);
if (!chan->tx_credits)
chan->ops->suspend(chan);
@@ -5570,10 +5580,8 @@
chan->tx_credits += credits;
- while (chan->tx_credits && !skb_queue_empty(&chan->tx_q)) {
- l2cap_do_send(chan, skb_dequeue(&chan->tx_q));
- chan->tx_credits--;
- }
+ /* Resume sending */
+ l2cap_le_flowctl_send(chan);
if (chan->tx_credits)
chan->ops->resume(chan);
diff --git a/net/bluetooth/rfcomm/core.c b/net/bluetooth/rfcomm/core.c
index f7eb02f..8ebca90 100644
--- a/net/bluetooth/rfcomm/core.c
+++ b/net/bluetooth/rfcomm/core.c
@@ -311,7 +311,7 @@
skb_queue_head_init(&d->tx_queue);
mutex_init(&d->lock);
- atomic_set(&d->refcnt, 1);
+ refcount_set(&d->refcnt, 1);
rfcomm_dlc_clear_state(d);
@@ -342,7 +342,7 @@
{
struct rfcomm_session *s = d->session;
- BT_DBG("dlc %p refcnt %d session %p", d, atomic_read(&d->refcnt), s);
+ BT_DBG("dlc %p refcnt %d session %p", d, refcount_read(&d->refcnt), s);
list_del(&d->list);
d->session = NULL;
diff --git a/net/bluetooth/selftest.c b/net/bluetooth/selftest.c
index dc688f1..ee92c92 100644
--- a/net/bluetooth/selftest.c
+++ b/net/bluetooth/selftest.c
@@ -26,7 +26,7 @@
#include <net/bluetooth/bluetooth.h>
#include <net/bluetooth/hci_core.h>
-#include "ecc.h"
+#include "ecdh_helper.h"
#include "smp.h"
#include "selftest.h"
@@ -142,18 +142,30 @@
const u8 pub_a[64], const u8 pub_b[64],
const u8 dhkey[32])
{
- u8 dhkey_a[32], dhkey_b[32];
+ u8 *tmp, *dhkey_a, *dhkey_b;
+ int ret = 0;
- ecdh_shared_secret(pub_b, priv_a, dhkey_a);
- ecdh_shared_secret(pub_a, priv_b, dhkey_b);
-
- if (memcmp(dhkey_a, dhkey, 32))
+ tmp = kmalloc(64, GFP_KERNEL);
+ if (!tmp)
return -EINVAL;
+ dhkey_a = &tmp[0];
+ dhkey_b = &tmp[32];
+
+ compute_ecdh_secret(pub_b, priv_a, dhkey_a);
+ compute_ecdh_secret(pub_a, priv_b, dhkey_b);
+
+ if (memcmp(dhkey_a, dhkey, 32)) {
+ ret = -EINVAL;
+ goto out;
+ }
+
if (memcmp(dhkey_b, dhkey, 32))
- return -EINVAL;
+ ret = -EINVAL;
- return 0;
+out:
+ kfree(dhkey_a);
+ return ret;
}
static char test_ecdh_buffer[32];
diff --git a/net/bluetooth/smp.c b/net/bluetooth/smp.c
index fae391f..14585ed 100644
--- a/net/bluetooth/smp.c
+++ b/net/bluetooth/smp.c
@@ -31,7 +31,7 @@
#include <net/bluetooth/l2cap.h>
#include <net/bluetooth/mgmt.h>
-#include "ecc.h"
+#include "ecdh_helper.h"
#include "smp.h"
#define SMP_DEV(hdev) \
@@ -569,8 +569,11 @@
smp->debug_key = true;
} else {
while (true) {
+ /* Seed private key with random number */
+ get_random_bytes(smp->local_sk, 32);
+
/* Generate local key pair for Secure Connections */
- if (!ecc_make_key(smp->local_pk, smp->local_sk))
+ if (!generate_ecdh_keys(smp->local_pk, smp->local_sk))
return -EIO;
/* This is unlikely, but we need to check that
@@ -1895,8 +1898,11 @@
set_bit(SMP_FLAG_DEBUG_KEY, &smp->flags);
} else {
while (true) {
+ /* Seed private key with random number */
+ get_random_bytes(smp->local_sk, 32);
+
/* Generate local key pair for Secure Connections */
- if (!ecc_make_key(smp->local_pk, smp->local_sk))
+ if (!generate_ecdh_keys(smp->local_pk, smp->local_sk))
return SMP_UNSPECIFIED;
/* This is unlikely, but we need to check that
@@ -2670,7 +2676,7 @@
SMP_DBG("Remote Public Key X: %32phN", smp->remote_pk);
SMP_DBG("Remote Public Key Y: %32phN", smp->remote_pk + 32);
- if (!ecdh_shared_secret(smp->remote_pk, smp->local_sk, smp->dhkey))
+ if (!compute_ecdh_secret(smp->remote_pk, smp->local_sk, smp->dhkey))
return SMP_UNSPECIFIED;
SMP_DBG("DHKey %32phN", smp->dhkey);
@@ -3483,6 +3489,32 @@
#if IS_ENABLED(CONFIG_BT_SELFTEST_SMP)
+static inline void swap_digits(u64 *in, u64 *out, unsigned int ndigits)
+{
+ int i;
+
+ for (i = 0; i < ndigits; i++)
+ out[i] = __swab64(in[ndigits - 1 - i]);
+}
+
+static int __init test_debug_key(void)
+{
+ u8 pk[64], sk[32];
+
+ swap_digits((u64 *)debug_sk, (u64 *)sk, 4);
+
+ if (!generate_ecdh_keys(pk, sk))
+ return -EINVAL;
+
+ if (memcmp(sk, debug_sk, 32))
+ return -EINVAL;
+
+ if (memcmp(pk, debug_pk, 64))
+ return -EINVAL;
+
+ return 0;
+}
+
static int __init test_ah(struct crypto_cipher *tfm_aes)
{
const u8 irk[16] = {
@@ -3738,6 +3770,12 @@
calltime = ktime_get();
+ err = test_debug_key();
+ if (err) {
+ BT_ERR("debug_key test failed");
+ goto done;
+ }
+
err = test_ah(tfm_aes);
if (err) {
BT_ERR("smp_ah test failed");
diff --git a/net/bpf/Makefile b/net/bpf/Makefile
new file mode 100644
index 0000000..27b2992
--- /dev/null
+++ b/net/bpf/Makefile
@@ -0,0 +1 @@
+obj-y := test_run.o
diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c
new file mode 100644
index 0000000..6be41a4
--- /dev/null
+++ b/net/bpf/test_run.c
@@ -0,0 +1,173 @@
+/* Copyright (c) 2017 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <linux/bpf.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+#include <linux/etherdevice.h>
+#include <linux/filter.h>
+#include <linux/sched/signal.h>
+
+static __always_inline u32 bpf_test_run_one(struct bpf_prog *prog, void *ctx)
+{
+ u32 ret;
+
+ preempt_disable();
+ rcu_read_lock();
+ ret = BPF_PROG_RUN(prog, ctx);
+ rcu_read_unlock();
+ preempt_enable();
+
+ return ret;
+}
+
+static u32 bpf_test_run(struct bpf_prog *prog, void *ctx, u32 repeat, u32 *time)
+{
+ u64 time_start, time_spent = 0;
+ u32 ret = 0, i;
+
+ if (!repeat)
+ repeat = 1;
+ time_start = ktime_get_ns();
+ for (i = 0; i < repeat; i++) {
+ ret = bpf_test_run_one(prog, ctx);
+ if (need_resched()) {
+ if (signal_pending(current))
+ break;
+ time_spent += ktime_get_ns() - time_start;
+ cond_resched();
+ time_start = ktime_get_ns();
+ }
+ }
+ time_spent += ktime_get_ns() - time_start;
+ do_div(time_spent, repeat);
+ *time = time_spent > U32_MAX ? U32_MAX : (u32)time_spent;
+
+ return ret;
+}
+
+static int bpf_test_finish(const union bpf_attr *kattr,
+ union bpf_attr __user *uattr, const void *data,
+ u32 size, u32 retval, u32 duration)
+{
+ void __user *data_out = u64_to_user_ptr(kattr->test.data_out);
+ int err = -EFAULT;
+
+ if (data_out && copy_to_user(data_out, data, size))
+ goto out;
+ if (copy_to_user(&uattr->test.data_size_out, &size, sizeof(size)))
+ goto out;
+ if (copy_to_user(&uattr->test.retval, &retval, sizeof(retval)))
+ goto out;
+ if (copy_to_user(&uattr->test.duration, &duration, sizeof(duration)))
+ goto out;
+ err = 0;
+out:
+ return err;
+}
+
+static void *bpf_test_init(const union bpf_attr *kattr, u32 size,
+ u32 headroom, u32 tailroom)
+{
+ void __user *data_in = u64_to_user_ptr(kattr->test.data_in);
+ void *data;
+
+ if (size < ETH_HLEN || size > PAGE_SIZE - headroom - tailroom)
+ return ERR_PTR(-EINVAL);
+
+ data = kzalloc(size + headroom + tailroom, GFP_USER);
+ if (!data)
+ return ERR_PTR(-ENOMEM);
+
+ if (copy_from_user(data + headroom, data_in, size)) {
+ kfree(data);
+ return ERR_PTR(-EFAULT);
+ }
+ return data;
+}
+
+int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr,
+ union bpf_attr __user *uattr)
+{
+ bool is_l2 = false, is_direct_pkt_access = false;
+ u32 size = kattr->test.data_size_in;
+ u32 repeat = kattr->test.repeat;
+ u32 retval, duration;
+ struct sk_buff *skb;
+ void *data;
+ int ret;
+
+ data = bpf_test_init(kattr, size, NET_SKB_PAD + NET_IP_ALIGN,
+ SKB_DATA_ALIGN(sizeof(struct skb_shared_info)));
+ if (IS_ERR(data))
+ return PTR_ERR(data);
+
+ switch (prog->type) {
+ case BPF_PROG_TYPE_SCHED_CLS:
+ case BPF_PROG_TYPE_SCHED_ACT:
+ is_l2 = true;
+ /* fall through */
+ case BPF_PROG_TYPE_LWT_IN:
+ case BPF_PROG_TYPE_LWT_OUT:
+ case BPF_PROG_TYPE_LWT_XMIT:
+ is_direct_pkt_access = true;
+ break;
+ default:
+ break;
+ }
+
+ skb = build_skb(data, 0);
+ if (!skb) {
+ kfree(data);
+ return -ENOMEM;
+ }
+
+ skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
+ __skb_put(skb, size);
+ skb->protocol = eth_type_trans(skb, current->nsproxy->net_ns->loopback_dev);
+ skb_reset_network_header(skb);
+
+ if (is_l2)
+ __skb_push(skb, ETH_HLEN);
+ if (is_direct_pkt_access)
+ bpf_compute_data_end(skb);
+ retval = bpf_test_run(prog, skb, repeat, &duration);
+ if (!is_l2)
+ __skb_push(skb, ETH_HLEN);
+ size = skb->len;
+ /* bpf program can never convert linear skb to non-linear */
+ if (WARN_ON_ONCE(skb_is_nonlinear(skb)))
+ size = skb_headlen(skb);
+ ret = bpf_test_finish(kattr, uattr, skb->data, size, retval, duration);
+ kfree_skb(skb);
+ return ret;
+}
+
+int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,
+ union bpf_attr __user *uattr)
+{
+ u32 size = kattr->test.data_size_in;
+ u32 repeat = kattr->test.repeat;
+ struct xdp_buff xdp = {};
+ u32 retval, duration;
+ void *data;
+ int ret;
+
+ data = bpf_test_init(kattr, size, XDP_PACKET_HEADROOM + NET_IP_ALIGN, 0);
+ if (IS_ERR(data))
+ return PTR_ERR(data);
+
+ xdp.data_hard_start = data;
+ xdp.data = data + XDP_PACKET_HEADROOM + NET_IP_ALIGN;
+ xdp.data_end = xdp.data + size;
+
+ retval = bpf_test_run(prog, &xdp, repeat, &duration);
+ if (xdp.data != data + XDP_PACKET_HEADROOM + NET_IP_ALIGN)
+ size = xdp.data_end - xdp.data;
+ ret = bpf_test_finish(kattr, uattr, xdp.data, size, retval, duration);
+ kfree(data);
+ return ret;
+}
diff --git a/net/bridge/br_device.c b/net/bridge/br_device.c
index 90f49a1..430b53e 100644
--- a/net/bridge/br_device.c
+++ b/net/bridge/br_device.c
@@ -123,6 +123,7 @@
{
struct net_bridge *br = netdev_priv(dev);
+ br_multicast_dev_del(br);
br_multicast_uninit_stats(br);
br_vlan_flush(br);
free_percpu(br->stats);
diff --git a/net/bridge/br_fdb.c b/net/bridge/br_fdb.c
index 6e08b71..ab0c7cc 100644
--- a/net/bridge/br_fdb.c
+++ b/net/bridge/br_fdb.c
@@ -589,6 +589,9 @@
if (unlikely(source != fdb->dst)) {
fdb->dst = source;
fdb_modified = true;
+ /* Take over HW learned entry */
+ if (unlikely(fdb->added_by_external_learn))
+ fdb->added_by_external_learn = 0;
}
if (now != fdb->updated)
fdb->updated = now;
@@ -854,6 +857,8 @@
br_fdb_update(br, p, addr, vid, true);
rcu_read_unlock();
local_bh_enable();
+ } else if (ndm->ndm_flags & NTF_EXT_LEARNED) {
+ err = br_fdb_external_learn_add(br, p, addr, vid);
} else {
spin_lock_bh(&br->hash_lock);
err = fdb_add_entry(br, p, addr, ndm->ndm_state,
diff --git a/net/bridge/br_forward.c b/net/bridge/br_forward.c
index 902af6b..48fb174 100644
--- a/net/bridge/br_forward.c
+++ b/net/bridge/br_forward.c
@@ -183,13 +183,23 @@
struct net_bridge_port *p;
list_for_each_entry_rcu(p, &br->port_list, list) {
- /* Do not flood unicast traffic to ports that turn it off */
- if (pkt_type == BR_PKT_UNICAST && !(p->flags & BR_FLOOD))
- continue;
- /* Do not flood if mc off, except for traffic we originate */
- if (pkt_type == BR_PKT_MULTICAST &&
- !(p->flags & BR_MCAST_FLOOD) && skb->dev != br->dev)
- continue;
+ /* Do not flood unicast traffic to ports that turn it off, nor
+ * other traffic if flood off, except for traffic we originate
+ */
+ switch (pkt_type) {
+ case BR_PKT_UNICAST:
+ if (!(p->flags & BR_FLOOD))
+ continue;
+ break;
+ case BR_PKT_MULTICAST:
+ if (!(p->flags & BR_MCAST_FLOOD) && skb->dev != br->dev)
+ continue;
+ break;
+ case BR_PKT_BROADCAST:
+ if (!(p->flags & BR_BCAST_FLOOD) && skb->dev != br->dev)
+ continue;
+ break;
+ }
/* Do not flood to ports that enable proxy ARP */
if (p->flags & BR_PROXYARP)
diff --git a/net/bridge/br_if.c b/net/bridge/br_if.c
index 56a2a72..7f8d05c 100644
--- a/net/bridge/br_if.c
+++ b/net/bridge/br_if.c
@@ -22,6 +22,7 @@
#include <linux/rtnetlink.h>
#include <linux/if_ether.h>
#include <linux/slab.h>
+#include <net/dsa.h>
#include <net/sock.h>
#include <linux/if_vlan.h>
#include <net/switchdev.h>
@@ -311,7 +312,6 @@
br_fdb_delete_by_port(br, NULL, 0, 1);
- br_multicast_dev_del(br);
cancel_delayed_work_sync(&br->gc_work);
br_sysfs_delbr(br->dev);
@@ -361,7 +361,7 @@
p->path_cost = port_cost(dev);
p->priority = 0x8000 >> BR_PORT_BITS;
p->port_no = index;
- p->flags = BR_LEARNING | BR_FLOOD | BR_MCAST_FLOOD;
+ p->flags = BR_LEARNING | BR_FLOOD | BR_MCAST_FLOOD | BR_BCAST_FLOOD;
br_init_port(p);
br_set_state(p, BR_STATE_DISABLED);
br_stp_port_timer_init(p);
diff --git a/net/bridge/br_mdb.c b/net/bridge/br_mdb.c
index 056e6ac..b084548 100644
--- a/net/bridge/br_mdb.c
+++ b/net/bridge/br_mdb.c
@@ -464,7 +464,8 @@
struct net_device *dev;
int err;
- err = nlmsg_parse(nlh, sizeof(*bpm), tb, MDBA_SET_ENTRY_MAX, NULL);
+ err = nlmsg_parse(nlh, sizeof(*bpm), tb, MDBA_SET_ENTRY_MAX, NULL,
+ NULL);
if (err < 0)
return err;
@@ -568,7 +569,8 @@
return ret;
}
-static int br_mdb_add(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int br_mdb_add(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct net_bridge_vlan_group *vg;
@@ -662,7 +664,8 @@
return err;
}
-static int br_mdb_del(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int br_mdb_del(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct net_bridge_vlan_group *vg;
diff --git a/net/bridge/br_netfilter_hooks.c b/net/bridge/br_netfilter_hooks.c
index 1f1e620..067cf03 100644
--- a/net/bridge/br_netfilter_hooks.c
+++ b/net/bridge/br_netfilter_hooks.c
@@ -997,13 +997,10 @@
if (!elem)
return okfn(net, sk, skb);
- /* We may already have this, but read-locks nest anyway */
- rcu_read_lock();
nf_hook_state_init(&state, hook, NFPROTO_BRIDGE, indev, outdev,
sk, net, okfn);
ret = nf_hook_slow(skb, &state, elem);
- rcu_read_unlock();
if (ret == 1)
ret = okfn(net, sk, skb);
diff --git a/net/bridge/br_netlink.c b/net/bridge/br_netlink.c
index 225ef7d..a572db71 100644
--- a/net/bridge/br_netlink.c
+++ b/net/bridge/br_netlink.c
@@ -189,6 +189,8 @@
!!(p->flags & BR_FLOOD)) ||
nla_put_u8(skb, IFLA_BRPORT_MCAST_FLOOD,
!!(p->flags & BR_MCAST_FLOOD)) ||
+ nla_put_u8(skb, IFLA_BRPORT_BCAST_FLOOD,
+ !!(p->flags & BR_BCAST_FLOOD)) ||
nla_put_u8(skb, IFLA_BRPORT_PROXYARP, !!(p->flags & BR_PROXYARP)) ||
nla_put_u8(skb, IFLA_BRPORT_PROXYARP_WIFI,
!!(p->flags & BR_PROXYARP_WIFI)) ||
@@ -683,6 +685,7 @@
br_set_port_flag(p, tb, IFLA_BRPORT_UNICAST_FLOOD, BR_FLOOD);
br_set_port_flag(p, tb, IFLA_BRPORT_MCAST_FLOOD, BR_MCAST_FLOOD);
br_set_port_flag(p, tb, IFLA_BRPORT_MCAST_TO_UCAST, BR_MULTICAST_TO_UNICAST);
+ br_set_port_flag(p, tb, IFLA_BRPORT_BCAST_FLOOD, BR_BCAST_FLOOD);
br_set_port_flag(p, tb, IFLA_BRPORT_PROXYARP, BR_PROXYARP);
br_set_port_flag(p, tb, IFLA_BRPORT_PROXYARP_WIFI, BR_PROXYARP_WIFI);
@@ -748,8 +751,8 @@
if (p && protinfo) {
if (protinfo->nla_type & NLA_F_NESTED) {
- err = nla_parse_nested(tb, IFLA_BRPORT_MAX,
- protinfo, br_port_policy);
+ err = nla_parse_nested(tb, IFLA_BRPORT_MAX, protinfo,
+ br_port_policy, NULL);
if (err)
return err;
diff --git a/net/bridge/br_netlink_tunnel.c b/net/bridge/br_netlink_tunnel.c
index c913491..3712c7f 100644
--- a/net/bridge/br_netlink_tunnel.c
+++ b/net/bridge/br_netlink_tunnel.c
@@ -227,8 +227,8 @@
memset(tinfo, 0, sizeof(*tinfo));
- err = nla_parse_nested(tb, IFLA_BRIDGE_VLAN_TUNNEL_MAX,
- attr, vlan_tunnel_policy);
+ err = nla_parse_nested(tb, IFLA_BRIDGE_VLAN_TUNNEL_MAX, attr,
+ vlan_tunnel_policy, NULL);
if (err < 0)
return err;
diff --git a/net/bridge/br_sysfs_if.c b/net/bridge/br_sysfs_if.c
index 79aee75..5d5d413a 100644
--- a/net/bridge/br_sysfs_if.c
+++ b/net/bridge/br_sysfs_if.c
@@ -173,6 +173,7 @@
BRPORT_ATTR_FLAG(proxyarp, BR_PROXYARP);
BRPORT_ATTR_FLAG(proxyarp_wifi, BR_PROXYARP_WIFI);
BRPORT_ATTR_FLAG(multicast_flood, BR_MCAST_FLOOD);
+BRPORT_ATTR_FLAG(broadcast_flood, BR_BCAST_FLOOD);
#ifdef CONFIG_BRIDGE_IGMP_SNOOPING
static ssize_t show_multicast_router(struct net_bridge_port *p, char *buf)
@@ -221,6 +222,7 @@
&brport_attr_proxyarp,
&brport_attr_proxyarp_wifi,
&brport_attr_multicast_flood,
+ &brport_attr_broadcast_flood,
NULL
};
diff --git a/net/bridge/netfilter/ebt_log.c b/net/bridge/netfilter/ebt_log.c
index 98b9c8e..707caea 100644
--- a/net/bridge/netfilter/ebt_log.c
+++ b/net/bridge/netfilter/ebt_log.c
@@ -62,10 +62,10 @@
pptr = skb_header_pointer(skb, offset,
sizeof(_ports), &_ports);
if (pptr == NULL) {
- printk(" INCOMPLETE TCP/UDP header");
+ pr_cont(" INCOMPLETE TCP/UDP header");
return;
}
- printk(" SPT=%u DPT=%u", ntohs(pptr->src), ntohs(pptr->dst));
+ pr_cont(" SPT=%u DPT=%u", ntohs(pptr->src), ntohs(pptr->dst));
}
}
@@ -100,11 +100,11 @@
ih = skb_header_pointer(skb, 0, sizeof(_iph), &_iph);
if (ih == NULL) {
- printk(" INCOMPLETE IP header");
+ pr_cont(" INCOMPLETE IP header");
goto out;
}
- printk(" IP SRC=%pI4 IP DST=%pI4, IP tos=0x%02X, IP proto=%d",
- &ih->saddr, &ih->daddr, ih->tos, ih->protocol);
+ pr_cont(" IP SRC=%pI4 IP DST=%pI4, IP tos=0x%02X, IP proto=%d",
+ &ih->saddr, &ih->daddr, ih->tos, ih->protocol);
print_ports(skb, ih->protocol, ih->ihl*4);
goto out;
}
@@ -120,11 +120,11 @@
ih = skb_header_pointer(skb, 0, sizeof(_iph), &_iph);
if (ih == NULL) {
- printk(" INCOMPLETE IPv6 header");
+ pr_cont(" INCOMPLETE IPv6 header");
goto out;
}
- printk(" IPv6 SRC=%pI6 IPv6 DST=%pI6, IPv6 priority=0x%01X, Next Header=%d",
- &ih->saddr, &ih->daddr, ih->priority, ih->nexthdr);
+ pr_cont(" IPv6 SRC=%pI6 IPv6 DST=%pI6, IPv6 priority=0x%01X, Next Header=%d",
+ &ih->saddr, &ih->daddr, ih->priority, ih->nexthdr);
nexthdr = ih->nexthdr;
offset_ph = ipv6_skip_exthdr(skb, sizeof(_iph), &nexthdr, &frag_off);
if (offset_ph == -1)
@@ -142,12 +142,12 @@
ah = skb_header_pointer(skb, 0, sizeof(_arph), &_arph);
if (ah == NULL) {
- printk(" INCOMPLETE ARP header");
+ pr_cont(" INCOMPLETE ARP header");
goto out;
}
- printk(" ARP HTYPE=%d, PTYPE=0x%04x, OPCODE=%d",
- ntohs(ah->ar_hrd), ntohs(ah->ar_pro),
- ntohs(ah->ar_op));
+ pr_cont(" ARP HTYPE=%d, PTYPE=0x%04x, OPCODE=%d",
+ ntohs(ah->ar_hrd), ntohs(ah->ar_pro),
+ ntohs(ah->ar_op));
/* If it's for Ethernet and the lengths are OK,
* then log the ARP payload
@@ -161,17 +161,17 @@
ap = skb_header_pointer(skb, sizeof(_arph),
sizeof(_arpp), &_arpp);
if (ap == NULL) {
- printk(" INCOMPLETE ARP payload");
+ pr_cont(" INCOMPLETE ARP payload");
goto out;
}
- printk(" ARP MAC SRC=%pM ARP IP SRC=%pI4 ARP MAC DST=%pM ARP IP DST=%pI4",
- ap->mac_src, ap->ip_src, ap->mac_dst, ap->ip_dst);
+ pr_cont(" ARP MAC SRC=%pM ARP IP SRC=%pI4 ARP MAC DST=%pM ARP IP DST=%pI4",
+ ap->mac_src, ap->ip_src,
+ ap->mac_dst, ap->ip_dst);
}
}
out:
- printk("\n");
+ pr_cont("\n");
spin_unlock_bh(&ebt_log_lock);
-
}
static unsigned int
diff --git a/net/bridge/netfilter/ebtable_broute.c b/net/bridge/netfilter/ebtable_broute.c
index 8fe36dc..2585b10 100644
--- a/net/bridge/netfilter/ebtable_broute.c
+++ b/net/bridge/netfilter/ebtable_broute.c
@@ -65,13 +65,13 @@
static int __net_init broute_net_init(struct net *net)
{
- net->xt.broute_table = ebt_register_table(net, &broute_table);
+ net->xt.broute_table = ebt_register_table(net, &broute_table, NULL);
return PTR_ERR_OR_ZERO(net->xt.broute_table);
}
static void __net_exit broute_net_exit(struct net *net)
{
- ebt_unregister_table(net, net->xt.broute_table);
+ ebt_unregister_table(net, net->xt.broute_table, NULL);
}
static struct pernet_operations broute_net_ops = {
diff --git a/net/bridge/netfilter/ebtable_filter.c b/net/bridge/netfilter/ebtable_filter.c
index 593a1bdc..f22ef7c 100644
--- a/net/bridge/netfilter/ebtable_filter.c
+++ b/net/bridge/netfilter/ebtable_filter.c
@@ -93,13 +93,13 @@
static int __net_init frame_filter_net_init(struct net *net)
{
- net->xt.frame_filter = ebt_register_table(net, &frame_filter);
+ net->xt.frame_filter = ebt_register_table(net, &frame_filter, ebt_ops_filter);
return PTR_ERR_OR_ZERO(net->xt.frame_filter);
}
static void __net_exit frame_filter_net_exit(struct net *net)
{
- ebt_unregister_table(net, net->xt.frame_filter);
+ ebt_unregister_table(net, net->xt.frame_filter, ebt_ops_filter);
}
static struct pernet_operations frame_filter_net_ops = {
@@ -109,20 +109,11 @@
static int __init ebtable_filter_init(void)
{
- int ret;
-
- ret = register_pernet_subsys(&frame_filter_net_ops);
- if (ret < 0)
- return ret;
- ret = nf_register_hooks(ebt_ops_filter, ARRAY_SIZE(ebt_ops_filter));
- if (ret < 0)
- unregister_pernet_subsys(&frame_filter_net_ops);
- return ret;
+ return register_pernet_subsys(&frame_filter_net_ops);
}
static void __exit ebtable_filter_fini(void)
{
- nf_unregister_hooks(ebt_ops_filter, ARRAY_SIZE(ebt_ops_filter));
unregister_pernet_subsys(&frame_filter_net_ops);
}
diff --git a/net/bridge/netfilter/ebtable_nat.c b/net/bridge/netfilter/ebtable_nat.c
index eb33919..2f7a4f3 100644
--- a/net/bridge/netfilter/ebtable_nat.c
+++ b/net/bridge/netfilter/ebtable_nat.c
@@ -93,13 +93,13 @@
static int __net_init frame_nat_net_init(struct net *net)
{
- net->xt.frame_nat = ebt_register_table(net, &frame_nat);
+ net->xt.frame_nat = ebt_register_table(net, &frame_nat, ebt_ops_nat);
return PTR_ERR_OR_ZERO(net->xt.frame_nat);
}
static void __net_exit frame_nat_net_exit(struct net *net)
{
- ebt_unregister_table(net, net->xt.frame_nat);
+ ebt_unregister_table(net, net->xt.frame_nat, ebt_ops_nat);
}
static struct pernet_operations frame_nat_net_ops = {
@@ -109,20 +109,11 @@
static int __init ebtable_nat_init(void)
{
- int ret;
-
- ret = register_pernet_subsys(&frame_nat_net_ops);
- if (ret < 0)
- return ret;
- ret = nf_register_hooks(ebt_ops_nat, ARRAY_SIZE(ebt_ops_nat));
- if (ret < 0)
- unregister_pernet_subsys(&frame_nat_net_ops);
- return ret;
+ return register_pernet_subsys(&frame_nat_net_ops);
}
static void __exit ebtable_nat_fini(void)
{
- nf_unregister_hooks(ebt_ops_nat, ARRAY_SIZE(ebt_ops_nat));
unregister_pernet_subsys(&frame_nat_net_ops);
}
diff --git a/net/bridge/netfilter/ebtables.c b/net/bridge/netfilter/ebtables.c
index 79b6991..9ec0c9f 100644
--- a/net/bridge/netfilter/ebtables.c
+++ b/net/bridge/netfilter/ebtables.c
@@ -1157,8 +1157,30 @@
return ret;
}
+static void __ebt_unregister_table(struct net *net, struct ebt_table *table)
+{
+ int i;
+
+ mutex_lock(&ebt_mutex);
+ list_del(&table->list);
+ mutex_unlock(&ebt_mutex);
+ EBT_ENTRY_ITERATE(table->private->entries, table->private->entries_size,
+ ebt_cleanup_entry, net, NULL);
+ if (table->private->nentries)
+ module_put(table->me);
+ vfree(table->private->entries);
+ if (table->private->chainstack) {
+ for_each_possible_cpu(i)
+ vfree(table->private->chainstack[i]);
+ vfree(table->private->chainstack);
+ }
+ vfree(table->private);
+ kfree(table);
+}
+
struct ebt_table *
-ebt_register_table(struct net *net, const struct ebt_table *input_table)
+ebt_register_table(struct net *net, const struct ebt_table *input_table,
+ const struct nf_hook_ops *ops)
{
struct ebt_table_info *newinfo;
struct ebt_table *t, *table;
@@ -1238,6 +1260,16 @@
}
list_add(&table->list, &net->xt.tables[NFPROTO_BRIDGE]);
mutex_unlock(&ebt_mutex);
+
+ if (!ops)
+ return table;
+
+ ret = nf_register_net_hooks(net, ops, hweight32(table->valid_hooks));
+ if (ret) {
+ __ebt_unregister_table(net, table);
+ return ERR_PTR(ret);
+ }
+
return table;
free_unlock:
mutex_unlock(&ebt_mutex);
@@ -1256,29 +1288,12 @@
return ERR_PTR(ret);
}
-void ebt_unregister_table(struct net *net, struct ebt_table *table)
+void ebt_unregister_table(struct net *net, struct ebt_table *table,
+ const struct nf_hook_ops *ops)
{
- int i;
-
- if (!table) {
- BUGPRINT("Request to unregister NULL table!!!\n");
- return;
- }
- mutex_lock(&ebt_mutex);
- list_del(&table->list);
- mutex_unlock(&ebt_mutex);
- EBT_ENTRY_ITERATE(table->private->entries, table->private->entries_size,
- ebt_cleanup_entry, net, NULL);
- if (table->private->nentries)
- module_put(table->me);
- vfree(table->private->entries);
- if (table->private->chainstack) {
- for_each_possible_cpu(i)
- vfree(table->private->chainstack[i]);
- vfree(table->private->chainstack);
- }
- vfree(table->private);
- kfree(table);
+ if (ops)
+ nf_unregister_net_hooks(net, ops, hweight32(table->valid_hooks));
+ __ebt_unregister_table(net, table);
}
/* userspace just supplied us with counters */
@@ -1713,7 +1728,7 @@
if (*size < sizeof(*ce))
return -EINVAL;
- ce = (struct ebt_entry __user *)*dstptr;
+ ce = *dstptr;
if (copy_to_user(ce, e, sizeof(*ce)))
return -EFAULT;
diff --git a/net/bridge/netfilter/nft_meta_bridge.c b/net/bridge/netfilter/nft_meta_bridge.c
index 5974dbc..bb63c9a 100644
--- a/net/bridge/netfilter/nft_meta_bridge.c
+++ b/net/bridge/netfilter/nft_meta_bridge.c
@@ -111,7 +111,7 @@
static struct nft_expr_type nft_meta_bridge_type __read_mostly = {
.family = NFPROTO_BRIDGE,
.name = "meta",
- .select_ops = &nft_meta_bridge_select_ops,
+ .select_ops = nft_meta_bridge_select_ops,
.policy = nft_meta_policy,
.maxattr = NFTA_META_MAX,
.owner = THIS_MODULE,
diff --git a/net/bridge/netfilter/nft_reject_bridge.c b/net/bridge/netfilter/nft_reject_bridge.c
index 206dc26..346ef6b 100644
--- a/net/bridge/netfilter/nft_reject_bridge.c
+++ b/net/bridge/netfilter/nft_reject_bridge.c
@@ -375,11 +375,7 @@
const struct nlattr * const tb[])
{
struct nft_reject *priv = nft_expr_priv(expr);
- int icmp_code, err;
-
- err = nft_reject_bridge_validate(ctx, expr, NULL);
- if (err < 0)
- return err;
+ int icmp_code;
if (tb[NFTA_REJECT_TYPE] == NULL)
return -EINVAL;
diff --git a/net/can/af_can.c b/net/can/af_can.c
index 5488e4a..b6406fe 100644
--- a/net/can/af_can.c
+++ b/net/can/af_can.c
@@ -2,7 +2,7 @@
* af_can.c - Protocol family CAN core module
* (used by different CAN protocol modules)
*
- * Copyright (c) 2002-2007 Volkswagen Group Electronic Research
+ * Copyright (c) 2002-2017 Volkswagen Group Electronic Research
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -75,20 +75,12 @@
module_param(stats_timer, int, S_IRUGO);
MODULE_PARM_DESC(stats_timer, "enable timer for statistics (default:on)");
-/* receive filters subscribed for 'all' CAN devices */
-struct dev_rcv_lists can_rx_alldev_list;
-static DEFINE_SPINLOCK(can_rcvlists_lock);
-
static struct kmem_cache *rcv_cache __read_mostly;
/* table of registered CAN protocols */
static const struct can_proto *proto_tab[CAN_NPROTO] __read_mostly;
static DEFINE_MUTEX(proto_tab_lock);
-struct timer_list can_stattimer; /* timer for statistics update */
-struct s_stats can_stats; /* packet statistics */
-struct s_pstats can_pstats; /* receive list statistics */
-
static atomic_t skbcounter = ATOMIC_INIT(0);
/*
@@ -145,9 +137,6 @@
if (protocol < 0 || protocol >= CAN_NPROTO)
return -EINVAL;
- if (!net_eq(net, &init_net))
- return -EAFNOSUPPORT;
-
cp = can_get_proto(protocol);
#ifdef CONFIG_MODULES
@@ -228,6 +217,7 @@
{
struct sk_buff *newskb = NULL;
struct canfd_frame *cfd = (struct canfd_frame *)skb->data;
+ struct s_stats *can_stats = dev_net(skb->dev)->can.can_stats;
int err = -EINVAL;
if (skb->len == CAN_MTU) {
@@ -316,8 +306,8 @@
netif_rx_ni(newskb);
/* update statistics */
- can_stats.tx_frames++;
- can_stats.tx_frames_delta++;
+ can_stats->tx_frames++;
+ can_stats->tx_frames_delta++;
return 0;
@@ -331,10 +321,11 @@
* af_can rx path
*/
-static struct dev_rcv_lists *find_dev_rcv_lists(struct net_device *dev)
+static struct dev_rcv_lists *find_dev_rcv_lists(struct net *net,
+ struct net_device *dev)
{
if (!dev)
- return &can_rx_alldev_list;
+ return net->can.can_rx_alldev_list;
else
return (struct dev_rcv_lists *)dev->ml_priv;
}
@@ -467,13 +458,14 @@
* -ENOMEM on missing cache mem to create subscription entry
* -ENODEV unknown device
*/
-int can_rx_register(struct net_device *dev, canid_t can_id, canid_t mask,
- void (*func)(struct sk_buff *, void *), void *data,
- char *ident, struct sock *sk)
+int can_rx_register(struct net *net, struct net_device *dev, canid_t can_id,
+ canid_t mask, void (*func)(struct sk_buff *, void *),
+ void *data, char *ident, struct sock *sk)
{
struct receiver *r;
struct hlist_head *rl;
struct dev_rcv_lists *d;
+ struct s_pstats *can_pstats = net->can.can_pstats;
int err = 0;
/* insert new receiver (dev,canid,mask) -> (func,data) */
@@ -481,13 +473,16 @@
if (dev && dev->type != ARPHRD_CAN)
return -ENODEV;
+ if (dev && !net_eq(net, dev_net(dev)))
+ return -ENODEV;
+
r = kmem_cache_alloc(rcv_cache, GFP_KERNEL);
if (!r)
return -ENOMEM;
- spin_lock(&can_rcvlists_lock);
+ spin_lock(&net->can.can_rcvlists_lock);
- d = find_dev_rcv_lists(dev);
+ d = find_dev_rcv_lists(net, dev);
if (d) {
rl = find_rcv_list(&can_id, &mask, d);
@@ -502,15 +497,15 @@
hlist_add_head_rcu(&r->list, rl);
d->entries++;
- can_pstats.rcv_entries++;
- if (can_pstats.rcv_entries_max < can_pstats.rcv_entries)
- can_pstats.rcv_entries_max = can_pstats.rcv_entries;
+ can_pstats->rcv_entries++;
+ if (can_pstats->rcv_entries_max < can_pstats->rcv_entries)
+ can_pstats->rcv_entries_max = can_pstats->rcv_entries;
} else {
kmem_cache_free(rcv_cache, r);
err = -ENODEV;
}
- spin_unlock(&can_rcvlists_lock);
+ spin_unlock(&net->can.can_rcvlists_lock);
return err;
}
@@ -540,19 +535,24 @@
* Description:
* Removes subscription entry depending on given (subscription) values.
*/
-void can_rx_unregister(struct net_device *dev, canid_t can_id, canid_t mask,
- void (*func)(struct sk_buff *, void *), void *data)
+void can_rx_unregister(struct net *net, struct net_device *dev, canid_t can_id,
+ canid_t mask, void (*func)(struct sk_buff *, void *),
+ void *data)
{
struct receiver *r = NULL;
struct hlist_head *rl;
+ struct s_pstats *can_pstats = net->can.can_pstats;
struct dev_rcv_lists *d;
if (dev && dev->type != ARPHRD_CAN)
return;
- spin_lock(&can_rcvlists_lock);
+ if (dev && !net_eq(net, dev_net(dev)))
+ return;
- d = find_dev_rcv_lists(dev);
+ spin_lock(&net->can.can_rcvlists_lock);
+
+ d = find_dev_rcv_lists(net, dev);
if (!d) {
pr_err("BUG: receive list not found for "
"dev %s, id %03X, mask %03X\n",
@@ -588,8 +588,8 @@
hlist_del_rcu(&r->list);
d->entries--;
- if (can_pstats.rcv_entries > 0)
- can_pstats.rcv_entries--;
+ if (can_pstats->rcv_entries > 0)
+ can_pstats->rcv_entries--;
/* remove device structure requested by NETDEV_UNREGISTER */
if (d->remove_on_zero_entries && !d->entries) {
@@ -598,7 +598,7 @@
}
out:
- spin_unlock(&can_rcvlists_lock);
+ spin_unlock(&net->can.can_rcvlists_lock);
/* schedule the receiver item for deletion */
if (r) {
@@ -683,11 +683,13 @@
static void can_receive(struct sk_buff *skb, struct net_device *dev)
{
struct dev_rcv_lists *d;
+ struct net *net = dev_net(dev);
+ struct s_stats *can_stats = net->can.can_stats;
int matches;
/* update statistics */
- can_stats.rx_frames++;
- can_stats.rx_frames_delta++;
+ can_stats->rx_frames++;
+ can_stats->rx_frames_delta++;
/* create non-zero unique skb identifier together with *skb */
while (!(can_skb_prv(skb)->skbcnt))
@@ -696,10 +698,10 @@
rcu_read_lock();
/* deliver the packet to sockets listening on all devices */
- matches = can_rcv_filter(&can_rx_alldev_list, skb);
+ matches = can_rcv_filter(net->can.can_rx_alldev_list, skb);
/* find receive list for this device */
- d = find_dev_rcv_lists(dev);
+ d = find_dev_rcv_lists(net, dev);
if (d)
matches += can_rcv_filter(d, skb);
@@ -709,8 +711,8 @@
consume_skb(skb);
if (matches > 0) {
- can_stats.matches++;
- can_stats.matches_delta++;
+ can_stats->matches++;
+ can_stats->matches_delta++;
}
}
@@ -719,9 +721,6 @@
{
struct canfd_frame *cfd = (struct canfd_frame *)skb->data;
- if (unlikely(!net_eq(dev_net(dev), &init_net)))
- goto drop;
-
if (WARN_ONCE(dev->type != ARPHRD_CAN ||
skb->len != CAN_MTU ||
cfd->len > CAN_MAX_DLEN,
@@ -743,9 +742,6 @@
{
struct canfd_frame *cfd = (struct canfd_frame *)skb->data;
- if (unlikely(!net_eq(dev_net(dev), &init_net)))
- goto drop;
-
if (WARN_ONCE(dev->type != ARPHRD_CAN ||
skb->len != CANFD_MTU ||
cfd->len > CANFD_MAX_DLEN,
@@ -835,9 +831,6 @@
struct net_device *dev = netdev_notifier_info_to_dev(ptr);
struct dev_rcv_lists *d;
- if (!net_eq(dev_net(dev), &init_net))
- return NOTIFY_DONE;
-
if (dev->type != ARPHRD_CAN)
return NOTIFY_DONE;
@@ -855,7 +848,7 @@
break;
case NETDEV_UNREGISTER:
- spin_lock(&can_rcvlists_lock);
+ spin_lock(&dev_net(dev)->can.can_rcvlists_lock);
d = dev->ml_priv;
if (d) {
@@ -869,7 +862,7 @@
pr_err("can: notifier: receive list not found for dev "
"%s\n", dev->name);
- spin_unlock(&can_rcvlists_lock);
+ spin_unlock(&dev_net(dev)->can.can_rcvlists_lock);
break;
}
@@ -877,6 +870,59 @@
return NOTIFY_DONE;
}
+static int can_pernet_init(struct net *net)
+{
+ net->can.can_rcvlists_lock =
+ __SPIN_LOCK_UNLOCKED(net->can.can_rcvlists_lock);
+ net->can.can_rx_alldev_list =
+ kzalloc(sizeof(struct dev_rcv_lists), GFP_KERNEL);
+
+ net->can.can_stats = kzalloc(sizeof(struct s_stats), GFP_KERNEL);
+ net->can.can_pstats = kzalloc(sizeof(struct s_pstats), GFP_KERNEL);
+
+ if (IS_ENABLED(CONFIG_PROC_FS)) {
+ /* the statistics are updated every second (timer triggered) */
+ if (stats_timer) {
+ setup_timer(&net->can.can_stattimer, can_stat_update,
+ (unsigned long)net);
+ mod_timer(&net->can.can_stattimer,
+ round_jiffies(jiffies + HZ));
+ }
+ net->can.can_stats->jiffies_init = jiffies;
+ can_init_proc(net);
+ }
+
+ return 0;
+}
+
+static void can_pernet_exit(struct net *net)
+{
+ struct net_device *dev;
+
+ if (IS_ENABLED(CONFIG_PROC_FS)) {
+ can_remove_proc(net);
+ if (stats_timer)
+ del_timer_sync(&net->can.can_stattimer);
+ }
+
+ /* remove created dev_rcv_lists from still registered CAN devices */
+ rcu_read_lock();
+ for_each_netdev_rcu(net, dev) {
+ if (dev->type == ARPHRD_CAN && dev->ml_priv) {
+ struct dev_rcv_lists *d = dev->ml_priv;
+
+ BUG_ON(d->entries);
+ kfree(d);
+ dev->ml_priv = NULL;
+ }
+ }
+ rcu_read_unlock();
+
+ kfree(net->can.can_rx_alldev_list);
+ kfree(net->can.can_stats);
+ kfree(net->can.can_pstats);
+}
+
/*
* af_can module init/exit functions
*/
@@ -902,6 +948,11 @@
.notifier_call = can_notifier,
};
+static struct pernet_operations can_pernet_ops __read_mostly = {
+ .init = can_pernet_init,
+ .exit = can_pernet_exit,
+};
+
static __init int can_init(void)
{
/* check for correct padding to be able to use the structs similarly */
@@ -912,21 +963,12 @@
pr_info("can: controller area network core (" CAN_VERSION_STRING ")\n");
- memset(&can_rx_alldev_list, 0, sizeof(can_rx_alldev_list));
-
rcv_cache = kmem_cache_create("can_receiver", sizeof(struct receiver),
0, 0, NULL);
if (!rcv_cache)
return -ENOMEM;
- if (IS_ENABLED(CONFIG_PROC_FS)) {
- if (stats_timer) {
- /* the statistics are updated every second (timer triggered) */
- setup_timer(&can_stattimer, can_stat_update, 0);
- mod_timer(&can_stattimer, round_jiffies(jiffies + HZ));
- }
- can_init_proc();
- }
+ register_pernet_subsys(&can_pernet_ops);
/* protocol register */
sock_register(&can_family_ops);
@@ -939,34 +981,13 @@
static __exit void can_exit(void)
{
- struct net_device *dev;
-
- if (IS_ENABLED(CONFIG_PROC_FS)) {
- if (stats_timer)
- del_timer_sync(&can_stattimer);
-
- can_remove_proc();
- }
-
/* protocol unregister */
dev_remove_pack(&canfd_packet);
dev_remove_pack(&can_packet);
unregister_netdevice_notifier(&can_netdev_notifier);
sock_unregister(PF_CAN);
- /* remove created dev_rcv_lists from still registered CAN devices */
- rcu_read_lock();
- for_each_netdev_rcu(&init_net, dev) {
- if (dev->type == ARPHRD_CAN && dev->ml_priv) {
-
- struct dev_rcv_lists *d = dev->ml_priv;
-
- BUG_ON(d->entries);
- kfree(d);
- dev->ml_priv = NULL;
- }
- }
- rcu_read_unlock();
+ unregister_pernet_subsys(&can_pernet_ops);
rcu_barrier(); /* Wait for completion of call_rcu()'s */
diff --git a/net/can/af_can.h b/net/can/af_can.h
index b86f512..d0ef45b 100644
--- a/net/can/af_can.h
+++ b/net/can/af_can.h
@@ -110,18 +110,9 @@
unsigned long rcv_entries_max;
};
-/* receive filters subscribed for 'all' CAN devices */
-extern struct dev_rcv_lists can_rx_alldev_list;
-
/* function prototypes for the CAN networklayer procfs (proc.c) */
-void can_init_proc(void);
-void can_remove_proc(void);
+void can_init_proc(struct net *net);
+void can_remove_proc(struct net *net);
void can_stat_update(unsigned long data);
-/* structures and variables from af_can.c needed in proc.c for reading */
-extern struct timer_list can_stattimer; /* timer for statistics update */
-extern struct s_stats can_stats; /* packet statistics */
-extern struct s_pstats can_pstats; /* receive list statistics */
-extern struct hlist_head can_rx_dev_list; /* rx dispatcher structures */
-
#endif /* AF_CAN_H */
diff --git a/net/can/bcm.c b/net/can/bcm.c
index 95d13b2..6543263 100644
--- a/net/can/bcm.c
+++ b/net/can/bcm.c
@@ -1,7 +1,7 @@
/*
* bcm.c - Broadcast Manager to filter/send (cyclic) CAN content
*
- * Copyright (c) 2002-2016 Volkswagen Group Electronic Research
+ * Copyright (c) 2002-2017 Volkswagen Group Electronic Research
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -77,7 +77,7 @@
(CAN_EFF_MASK | CAN_EFF_FLAG | CAN_RTR_FLAG) : \
(CAN_SFF_MASK | CAN_EFF_FLAG | CAN_RTR_FLAG))
-#define CAN_BCM_VERSION "20161123"
+#define CAN_BCM_VERSION "20170425"
MODULE_DESCRIPTION("PF_CAN broadcast manager protocol");
MODULE_LICENSE("Dual BSD/GPL");
@@ -118,8 +118,6 @@
struct net_device *rx_reg_dev;
};
-static struct proc_dir_entry *proc_dir;
-
struct bcm_sock {
struct sock sk;
int bound;
@@ -149,7 +147,8 @@
/*
* procfs functions
*/
-static char *bcm_proc_getifname(char *result, int ifindex)
+#if IS_ENABLED(CONFIG_PROC_FS)
+static char *bcm_proc_getifname(struct net *net, char *result, int ifindex)
{
struct net_device *dev;
@@ -157,7 +156,7 @@
return "any";
rcu_read_lock();
- dev = dev_get_by_index_rcu(&init_net, ifindex);
+ dev = dev_get_by_index_rcu(net, ifindex);
if (dev)
strcpy(result, dev->name);
else
@@ -170,7 +169,8 @@
static int bcm_proc_show(struct seq_file *m, void *v)
{
char ifname[IFNAMSIZ];
- struct sock *sk = (struct sock *)m->private;
+ struct net *net = m->private;
+ struct sock *sk = (struct sock *)PDE_DATA(m->file->f_inode);
struct bcm_sock *bo = bcm_sk(sk);
struct bcm_op *op;
@@ -178,7 +178,7 @@
seq_printf(m, " / sk %pK", sk);
seq_printf(m, " / bo %pK", bo);
seq_printf(m, " / dropped %lu", bo->dropped_usr_msgs);
- seq_printf(m, " / bound %s", bcm_proc_getifname(ifname, bo->ifindex));
+ seq_printf(m, " / bound %s", bcm_proc_getifname(net, ifname, bo->ifindex));
seq_printf(m, " <<<\n");
list_for_each_entry(op, &bo->rx_ops, list) {
@@ -190,7 +190,7 @@
continue;
seq_printf(m, "rx_op: %03X %-5s ", op->can_id,
- bcm_proc_getifname(ifname, op->ifindex));
+ bcm_proc_getifname(net, ifname, op->ifindex));
if (op->flags & CAN_FD_FRAME)
seq_printf(m, "(%u)", op->nframes);
@@ -219,7 +219,7 @@
list_for_each_entry(op, &bo->tx_ops, list) {
seq_printf(m, "tx_op: %03X %s ", op->can_id,
- bcm_proc_getifname(ifname, op->ifindex));
+ bcm_proc_getifname(net, ifname, op->ifindex));
if (op->flags & CAN_FD_FRAME)
seq_printf(m, "(%u) ", op->nframes);
@@ -242,7 +242,7 @@
static int bcm_proc_open(struct inode *inode, struct file *file)
{
- return single_open(file, bcm_proc_show, PDE_DATA(inode));
+ return single_open_net(inode, file, bcm_proc_show);
}
static const struct file_operations bcm_proc_fops = {
@@ -252,6 +252,7 @@
.llseek = seq_lseek,
.release = single_release,
};
+#endif /* CONFIG_PROC_FS */
/*
* bcm_can_tx - send the (next) CAN frame to the appropriate CAN interface
@@ -267,7 +268,7 @@
if (!op->ifindex)
return;
- dev = dev_get_by_index(&init_net, op->ifindex);
+ dev = dev_get_by_index(sock_net(op->sk), op->ifindex);
if (!dev) {
/* RFC: should this bcm_op remove itself here? */
return;
@@ -764,8 +765,8 @@
static void bcm_rx_unreg(struct net_device *dev, struct bcm_op *op)
{
if (op->rx_reg_dev == dev) {
- can_rx_unregister(dev, op->can_id, REGMASK(op->can_id),
- bcm_rx_handler, op);
+ can_rx_unregister(dev_net(dev), dev, op->can_id,
+ REGMASK(op->can_id), bcm_rx_handler, op);
/* mark as removed subscription */
op->rx_reg_dev = NULL;
@@ -800,7 +801,7 @@
if (op->rx_reg_dev) {
struct net_device *dev;
- dev = dev_get_by_index(&init_net,
+ dev = dev_get_by_index(sock_net(op->sk),
op->ifindex);
if (dev) {
bcm_rx_unreg(dev, op);
@@ -808,7 +809,8 @@
}
}
} else
- can_rx_unregister(NULL, op->can_id,
+ can_rx_unregister(sock_net(op->sk), NULL,
+ op->can_id,
REGMASK(op->can_id),
bcm_rx_handler, op);
@@ -1220,9 +1222,10 @@
if (ifindex) {
struct net_device *dev;
- dev = dev_get_by_index(&init_net, ifindex);
+ dev = dev_get_by_index(sock_net(sk), ifindex);
if (dev) {
- err = can_rx_register(dev, op->can_id,
+ err = can_rx_register(sock_net(sk), dev,
+ op->can_id,
REGMASK(op->can_id),
bcm_rx_handler, op,
"bcm", sk);
@@ -1232,7 +1235,7 @@
}
} else
- err = can_rx_register(NULL, op->can_id,
+ err = can_rx_register(sock_net(sk), NULL, op->can_id,
REGMASK(op->can_id),
bcm_rx_handler, op, "bcm", sk);
if (err) {
@@ -1272,7 +1275,7 @@
return err;
}
- dev = dev_get_by_index(&init_net, ifindex);
+ dev = dev_get_by_index(sock_net(sk), ifindex);
if (!dev) {
kfree_skb(skb);
return -ENODEV;
@@ -1337,7 +1340,7 @@
if (ifindex) {
struct net_device *dev;
- dev = dev_get_by_index(&init_net, ifindex);
+ dev = dev_get_by_index(sock_net(sk), ifindex);
if (!dev)
return -ENODEV;
@@ -1418,7 +1421,7 @@
struct bcm_op *op;
int notify_enodev = 0;
- if (!net_eq(dev_net(dev), &init_net))
+ if (!net_eq(dev_net(dev), sock_net(sk)))
return NOTIFY_DONE;
if (dev->type != ARPHRD_CAN)
@@ -1490,6 +1493,7 @@
static int bcm_release(struct socket *sock)
{
struct sock *sk = sock->sk;
+ struct net *net = sock_net(sk);
struct bcm_sock *bo;
struct bcm_op *op, *next;
@@ -1521,23 +1525,25 @@
if (op->rx_reg_dev) {
struct net_device *dev;
- dev = dev_get_by_index(&init_net, op->ifindex);
+ dev = dev_get_by_index(net, op->ifindex);
if (dev) {
bcm_rx_unreg(dev, op);
dev_put(dev);
}
}
} else
- can_rx_unregister(NULL, op->can_id,
+ can_rx_unregister(net, NULL, op->can_id,
REGMASK(op->can_id),
bcm_rx_handler, op);
bcm_remove_op(op);
}
+#if IS_ENABLED(CONFIG_PROC_FS)
/* remove procfs entry */
- if (proc_dir && bo->bcm_proc_read)
- remove_proc_entry(bo->procname, proc_dir);
+ if (net->can.bcmproc_dir && bo->bcm_proc_read)
+ remove_proc_entry(bo->procname, net->can.bcmproc_dir);
+#endif /* CONFIG_PROC_FS */
/* remove device reference */
if (bo->bound) {
@@ -1560,6 +1566,7 @@
struct sockaddr_can *addr = (struct sockaddr_can *)uaddr;
struct sock *sk = sock->sk;
struct bcm_sock *bo = bcm_sk(sk);
+ struct net *net = sock_net(sk);
int ret = 0;
if (len < sizeof(*addr))
@@ -1576,7 +1583,7 @@
if (addr->can_ifindex) {
struct net_device *dev;
- dev = dev_get_by_index(&init_net, addr->can_ifindex);
+ dev = dev_get_by_index(net, addr->can_ifindex);
if (!dev) {
ret = -ENODEV;
goto fail;
@@ -1595,17 +1602,19 @@
bo->ifindex = 0;
}
- if (proc_dir) {
+#if IS_ENABLED(CONFIG_PROC_FS)
+ if (net->can.bcmproc_dir) {
/* unique socket address as filename */
sprintf(bo->procname, "%lu", sock_i_ino(sk));
bo->bcm_proc_read = proc_create_data(bo->procname, 0644,
- proc_dir,
+ net->can.bcmproc_dir,
&bcm_proc_fops, sk);
if (!bo->bcm_proc_read) {
ret = -ENOMEM;
goto fail;
}
}
+#endif /* CONFIG_PROC_FS */
bo->bound = 1;
@@ -1686,6 +1695,30 @@
.prot = &bcm_proto,
};
+static int canbcm_pernet_init(struct net *net)
+{
+#if IS_ENABLED(CONFIG_PROC_FS)
+ /* create /proc/net/can-bcm directory */
+ net->can.bcmproc_dir = proc_net_mkdir(net, "can-bcm", net->proc_net);
+#endif /* CONFIG_PROC_FS */
+
+ return 0;
+}
+
+static void canbcm_pernet_exit(struct net *net)
+{
+#if IS_ENABLED(CONFIG_PROC_FS)
+ /* remove /proc/net/can-bcm directory */
+ if (net->can.bcmproc_dir)
+ remove_proc_entry("can-bcm", net->proc_net);
+#endif /* CONFIG_PROC_FS */
+}
+
+static struct pernet_operations canbcm_pernet_ops __read_mostly = {
+ .init = canbcm_pernet_init,
+ .exit = canbcm_pernet_exit,
+};
+
static int __init bcm_module_init(void)
{
int err;
@@ -1698,17 +1731,14 @@
return err;
}
- /* create /proc/net/can-bcm directory */
- proc_dir = proc_mkdir("can-bcm", init_net.proc_net);
+ register_pernet_subsys(&canbcm_pernet_ops);
return 0;
}
static void __exit bcm_module_exit(void)
{
can_proto_unregister(&bcm_can_proto);
-
- if (proc_dir)
- remove_proc_entry("can-bcm", init_net.proc_net);
+ unregister_pernet_subsys(&canbcm_pernet_ops);
}
module_init(bcm_module_init);
diff --git a/net/can/gw.c b/net/can/gw.c
index 7056a1a..29748d8 100644
--- a/net/can/gw.c
+++ b/net/can/gw.c
@@ -1,7 +1,7 @@
/*
* gw.c - CAN frame Gateway/Router/Bridge with netlink interface
*
- * Copyright (c) 2011 Volkswagen Group Electronic Research
+ * Copyright (c) 2017 Volkswagen Group Electronic Research
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -59,7 +59,7 @@
#include <net/net_namespace.h>
#include <net/sock.h>
-#define CAN_GW_VERSION "20130117"
+#define CAN_GW_VERSION "20170425"
#define CAN_GW_NAME "can-gw"
MODULE_DESCRIPTION("PF_CAN netlink gateway");
@@ -79,9 +79,7 @@
__stringify(CGW_MAX_HOPS) " hops, "
"default: " __stringify(CGW_DEFAULT_HOPS) ")");
-static HLIST_HEAD(cgw_list);
static struct notifier_block notifier;
-
static struct kmem_cache *cgw_cache __read_mostly;
/* structure that contains the (on-the-fly) CAN frame modifications */
@@ -438,16 +436,16 @@
gwj->handled_frames++;
}
-static inline int cgw_register_filter(struct cgw_job *gwj)
+static inline int cgw_register_filter(struct net *net, struct cgw_job *gwj)
{
- return can_rx_register(gwj->src.dev, gwj->ccgw.filter.can_id,
+ return can_rx_register(net, gwj->src.dev, gwj->ccgw.filter.can_id,
gwj->ccgw.filter.can_mask, can_can_gw_rcv,
gwj, "gw", NULL);
}
-static inline void cgw_unregister_filter(struct cgw_job *gwj)
+static inline void cgw_unregister_filter(struct net *net, struct cgw_job *gwj)
{
- can_rx_unregister(gwj->src.dev, gwj->ccgw.filter.can_id,
+ can_rx_unregister(net, gwj->src.dev, gwj->ccgw.filter.can_id,
gwj->ccgw.filter.can_mask, can_can_gw_rcv, gwj);
}
@@ -455,9 +453,8 @@
unsigned long msg, void *ptr)
{
struct net_device *dev = netdev_notifier_info_to_dev(ptr);
+ struct net *net = dev_net(dev);
- if (!net_eq(dev_net(dev), &init_net))
- return NOTIFY_DONE;
if (dev->type != ARPHRD_CAN)
return NOTIFY_DONE;
@@ -468,11 +465,11 @@
ASSERT_RTNL();
- hlist_for_each_entry_safe(gwj, nx, &cgw_list, list) {
+ hlist_for_each_entry_safe(gwj, nx, &net->can.cgw_list, list) {
if (gwj->src.dev == dev || gwj->dst.dev == dev) {
hlist_del(&gwj->list);
- cgw_unregister_filter(gwj);
+ cgw_unregister_filter(net, gwj);
kmem_cache_free(cgw_cache, gwj);
}
}
@@ -592,12 +589,13 @@
/* Dump information about all CAN gateway jobs, in response to RTM_GETROUTE */
static int cgw_dump_jobs(struct sk_buff *skb, struct netlink_callback *cb)
{
+ struct net *net = sock_net(skb->sk);
struct cgw_job *gwj = NULL;
int idx = 0;
int s_idx = cb->args[0];
rcu_read_lock();
- hlist_for_each_entry_rcu(gwj, &cgw_list, list) {
+ hlist_for_each_entry_rcu(gwj, &net->can.cgw_list, list) {
if (idx < s_idx)
goto cont;
@@ -641,7 +639,7 @@
memset(mod, 0, sizeof(*mod));
err = nlmsg_parse(nlh, sizeof(struct rtcanmsg), tb, CGW_MAX,
- cgw_policy);
+ cgw_policy, NULL);
if (err < 0)
return err;
@@ -809,8 +807,10 @@
return 0;
}
-static int cgw_create_job(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int cgw_create_job(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
+ struct net *net = sock_net(skb->sk);
struct rtcanmsg *r;
struct cgw_job *gwj;
struct cf_mod mod;
@@ -841,7 +841,7 @@
ASSERT_RTNL();
/* check for updating an existing job with identical uid */
- hlist_for_each_entry(gwj, &cgw_list, list) {
+ hlist_for_each_entry(gwj, &net->can.cgw_list, list) {
if (gwj->mod.uid != mod.uid)
continue;
@@ -879,7 +879,7 @@
err = -ENODEV;
- gwj->src.dev = __dev_get_by_index(&init_net, gwj->ccgw.src_idx);
+ gwj->src.dev = __dev_get_by_index(net, gwj->ccgw.src_idx);
if (!gwj->src.dev)
goto out;
@@ -887,7 +887,7 @@
if (gwj->src.dev->type != ARPHRD_CAN)
goto out;
- gwj->dst.dev = __dev_get_by_index(&init_net, gwj->ccgw.dst_idx);
+ gwj->dst.dev = __dev_get_by_index(net, gwj->ccgw.dst_idx);
if (!gwj->dst.dev)
goto out;
@@ -897,9 +897,9 @@
ASSERT_RTNL();
- err = cgw_register_filter(gwj);
+ err = cgw_register_filter(net, gwj);
if (!err)
- hlist_add_head_rcu(&gwj->list, &cgw_list);
+ hlist_add_head_rcu(&gwj->list, &net->can.cgw_list);
out:
if (err)
kmem_cache_free(cgw_cache, gwj);
@@ -907,22 +907,24 @@
return err;
}
-static void cgw_remove_all_jobs(void)
+static void cgw_remove_all_jobs(struct net *net)
{
struct cgw_job *gwj = NULL;
struct hlist_node *nx;
ASSERT_RTNL();
- hlist_for_each_entry_safe(gwj, nx, &cgw_list, list) {
+ hlist_for_each_entry_safe(gwj, nx, &net->can.cgw_list, list) {
hlist_del(&gwj->list);
- cgw_unregister_filter(gwj);
+ cgw_unregister_filter(net, gwj);
kmem_cache_free(cgw_cache, gwj);
}
}
-static int cgw_remove_job(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int cgw_remove_job(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
+ struct net *net = sock_net(skb->sk);
struct cgw_job *gwj = NULL;
struct hlist_node *nx;
struct rtcanmsg *r;
@@ -951,7 +953,7 @@
/* two interface indices both set to 0 => remove all entries */
if (!ccgw.src_idx && !ccgw.dst_idx) {
- cgw_remove_all_jobs();
+ cgw_remove_all_jobs(net);
return 0;
}
@@ -960,7 +962,7 @@
ASSERT_RTNL();
/* remove only the first matching entry */
- hlist_for_each_entry_safe(gwj, nx, &cgw_list, list) {
+ hlist_for_each_entry_safe(gwj, nx, &net->can.cgw_list, list) {
if (gwj->flags != r->flags)
continue;
@@ -983,7 +985,7 @@
continue;
hlist_del(&gwj->list);
- cgw_unregister_filter(gwj);
+ cgw_unregister_filter(net, gwj);
kmem_cache_free(cgw_cache, gwj);
err = 0;
break;
@@ -992,6 +994,24 @@
return err;
}
+static int __net_init cangw_pernet_init(struct net *net)
+{
+ INIT_HLIST_HEAD(&net->can.cgw_list);
+ return 0;
+}
+
+static void __net_exit cangw_pernet_exit(struct net *net)
+{
+ rtnl_lock();
+ cgw_remove_all_jobs(net);
+ rtnl_unlock();
+}
+
+static struct pernet_operations cangw_pernet_ops = {
+ .init = cangw_pernet_init,
+ .exit = cangw_pernet_exit,
+};
+
static __init int cgw_module_init(void)
{
/* sanitize given module parameter */
@@ -1000,6 +1020,7 @@
pr_info("can: netlink gateway (rev " CAN_GW_VERSION ") max_hops=%d\n",
max_hops);
+ register_pernet_subsys(&cangw_pernet_ops);
cgw_cache = kmem_cache_create("can_gw", sizeof(struct cgw_job),
0, 0, NULL);
@@ -1029,10 +1050,7 @@
unregister_netdevice_notifier(¬ifier);
- rtnl_lock();
- cgw_remove_all_jobs();
- rtnl_unlock();
-
+ unregister_pernet_subsys(&cangw_pernet_ops);
rcu_barrier(); /* Wait for completion of call_rcu()'s */
kmem_cache_destroy(cgw_cache);
diff --git a/net/can/proc.c b/net/can/proc.c
index 85ef7bb..83045f0 100644
--- a/net/can/proc.c
+++ b/net/can/proc.c
@@ -62,17 +62,6 @@
#define CAN_PROC_RCVLIST_EFF "rcvlist_eff"
#define CAN_PROC_RCVLIST_ERR "rcvlist_err"
-static struct proc_dir_entry *can_dir;
-static struct proc_dir_entry *pde_version;
-static struct proc_dir_entry *pde_stats;
-static struct proc_dir_entry *pde_reset_stats;
-static struct proc_dir_entry *pde_rcvlist_all;
-static struct proc_dir_entry *pde_rcvlist_fil;
-static struct proc_dir_entry *pde_rcvlist_inv;
-static struct proc_dir_entry *pde_rcvlist_sff;
-static struct proc_dir_entry *pde_rcvlist_eff;
-static struct proc_dir_entry *pde_rcvlist_err;
-
static int user_reset;
static const char rx_list_name[][8] = {
@@ -86,21 +75,23 @@
* af_can statistics stuff
*/
-static void can_init_stats(void)
+static void can_init_stats(struct net *net)
{
+ struct s_stats *can_stats = net->can.can_stats;
+ struct s_pstats *can_pstats = net->can.can_pstats;
/*
* This memset function is called from a timer context (when
* can_stattimer is active which is the default) OR in a process
* context (reading the proc_fs when can_stattimer is disabled).
*/
- memset(&can_stats, 0, sizeof(can_stats));
- can_stats.jiffies_init = jiffies;
+ memset(can_stats, 0, sizeof(struct s_stats));
+ can_stats->jiffies_init = jiffies;
- can_pstats.stats_reset++;
+ can_pstats->stats_reset++;
if (user_reset) {
user_reset = 0;
- can_pstats.user_reset++;
+ can_pstats->user_reset++;
}
}
@@ -126,64 +117,66 @@
void can_stat_update(unsigned long data)
{
+ struct net *net = (struct net *)data;
+ struct s_stats *can_stats = net->can.can_stats;
unsigned long j = jiffies; /* snapshot */
/* restart counting in timer context on user request */
if (user_reset)
- can_init_stats();
+ can_init_stats(net);
/* restart counting on jiffies overflow */
- if (j < can_stats.jiffies_init)
- can_init_stats();
+ if (j < can_stats->jiffies_init)
+ can_init_stats(net);
/* prevent overflow in calc_rate() */
- if (can_stats.rx_frames > (ULONG_MAX / HZ))
- can_init_stats();
+ if (can_stats->rx_frames > (ULONG_MAX / HZ))
+ can_init_stats(net);
/* prevent overflow in calc_rate() */
- if (can_stats.tx_frames > (ULONG_MAX / HZ))
- can_init_stats();
+ if (can_stats->tx_frames > (ULONG_MAX / HZ))
+ can_init_stats(net);
/* matches overflow - very improbable */
- if (can_stats.matches > (ULONG_MAX / 100))
- can_init_stats();
+ if (can_stats->matches > (ULONG_MAX / 100))
+ can_init_stats(net);
/* calc total values */
- if (can_stats.rx_frames)
- can_stats.total_rx_match_ratio = (can_stats.matches * 100) /
- can_stats.rx_frames;
+ if (can_stats->rx_frames)
+ can_stats->total_rx_match_ratio = (can_stats->matches * 100) /
+ can_stats->rx_frames;
- can_stats.total_tx_rate = calc_rate(can_stats.jiffies_init, j,
- can_stats.tx_frames);
- can_stats.total_rx_rate = calc_rate(can_stats.jiffies_init, j,
- can_stats.rx_frames);
+ can_stats->total_tx_rate = calc_rate(can_stats->jiffies_init, j,
+ can_stats->tx_frames);
+ can_stats->total_rx_rate = calc_rate(can_stats->jiffies_init, j,
+ can_stats->rx_frames);
/* calc current values */
- if (can_stats.rx_frames_delta)
- can_stats.current_rx_match_ratio =
- (can_stats.matches_delta * 100) /
- can_stats.rx_frames_delta;
+ if (can_stats->rx_frames_delta)
+ can_stats->current_rx_match_ratio =
+ (can_stats->matches_delta * 100) /
+ can_stats->rx_frames_delta;
- can_stats.current_tx_rate = calc_rate(0, HZ, can_stats.tx_frames_delta);
- can_stats.current_rx_rate = calc_rate(0, HZ, can_stats.rx_frames_delta);
+ can_stats->current_tx_rate = calc_rate(0, HZ, can_stats->tx_frames_delta);
+ can_stats->current_rx_rate = calc_rate(0, HZ, can_stats->rx_frames_delta);
/* check / update maximum values */
- if (can_stats.max_tx_rate < can_stats.current_tx_rate)
- can_stats.max_tx_rate = can_stats.current_tx_rate;
+ if (can_stats->max_tx_rate < can_stats->current_tx_rate)
+ can_stats->max_tx_rate = can_stats->current_tx_rate;
- if (can_stats.max_rx_rate < can_stats.current_rx_rate)
- can_stats.max_rx_rate = can_stats.current_rx_rate;
+ if (can_stats->max_rx_rate < can_stats->current_rx_rate)
+ can_stats->max_rx_rate = can_stats->current_rx_rate;
- if (can_stats.max_rx_match_ratio < can_stats.current_rx_match_ratio)
- can_stats.max_rx_match_ratio = can_stats.current_rx_match_ratio;
+ if (can_stats->max_rx_match_ratio < can_stats->current_rx_match_ratio)
+ can_stats->max_rx_match_ratio = can_stats->current_rx_match_ratio;
/* clear values for 'current rate' calculation */
- can_stats.tx_frames_delta = 0;
- can_stats.rx_frames_delta = 0;
- can_stats.matches_delta = 0;
+ can_stats->tx_frames_delta = 0;
+ can_stats->rx_frames_delta = 0;
+ can_stats->matches_delta = 0;
/* restart timer (one second) */
- mod_timer(&can_stattimer, round_jiffies(jiffies + HZ));
+ mod_timer(&net->can.can_stattimer, round_jiffies(jiffies + HZ));
}
/*
@@ -217,57 +210,61 @@
static int can_stats_proc_show(struct seq_file *m, void *v)
{
+ struct net *net = m->private;
+ struct s_stats *can_stats = net->can.can_stats;
+ struct s_pstats *can_pstats = net->can.can_pstats;
+
seq_putc(m, '\n');
- seq_printf(m, " %8ld transmitted frames (TXF)\n", can_stats.tx_frames);
- seq_printf(m, " %8ld received frames (RXF)\n", can_stats.rx_frames);
- seq_printf(m, " %8ld matched frames (RXMF)\n", can_stats.matches);
+ seq_printf(m, " %8ld transmitted frames (TXF)\n", can_stats->tx_frames);
+ seq_printf(m, " %8ld received frames (RXF)\n", can_stats->rx_frames);
+ seq_printf(m, " %8ld matched frames (RXMF)\n", can_stats->matches);
seq_putc(m, '\n');
- if (can_stattimer.function == can_stat_update) {
+ if (net->can.can_stattimer.function == can_stat_update) {
seq_printf(m, " %8ld %% total match ratio (RXMR)\n",
- can_stats.total_rx_match_ratio);
+ can_stats->total_rx_match_ratio);
seq_printf(m, " %8ld frames/s total tx rate (TXR)\n",
- can_stats.total_tx_rate);
+ can_stats->total_tx_rate);
seq_printf(m, " %8ld frames/s total rx rate (RXR)\n",
- can_stats.total_rx_rate);
+ can_stats->total_rx_rate);
seq_putc(m, '\n');
seq_printf(m, " %8ld %% current match ratio (CRXMR)\n",
- can_stats.current_rx_match_ratio);
+ can_stats->current_rx_match_ratio);
seq_printf(m, " %8ld frames/s current tx rate (CTXR)\n",
- can_stats.current_tx_rate);
+ can_stats->current_tx_rate);
seq_printf(m, " %8ld frames/s current rx rate (CRXR)\n",
- can_stats.current_rx_rate);
+ can_stats->current_rx_rate);
seq_putc(m, '\n');
seq_printf(m, " %8ld %% max match ratio (MRXMR)\n",
- can_stats.max_rx_match_ratio);
+ can_stats->max_rx_match_ratio);
seq_printf(m, " %8ld frames/s max tx rate (MTXR)\n",
- can_stats.max_tx_rate);
+ can_stats->max_tx_rate);
seq_printf(m, " %8ld frames/s max rx rate (MRXR)\n",
- can_stats.max_rx_rate);
+ can_stats->max_rx_rate);
seq_putc(m, '\n');
}
seq_printf(m, " %8ld current receive list entries (CRCV)\n",
- can_pstats.rcv_entries);
+ can_pstats->rcv_entries);
seq_printf(m, " %8ld maximum receive list entries (MRCV)\n",
- can_pstats.rcv_entries_max);
+ can_pstats->rcv_entries_max);
- if (can_pstats.stats_reset)
+ if (can_pstats->stats_reset)
seq_printf(m, "\n %8ld statistic resets (STR)\n",
- can_pstats.stats_reset);
+ can_pstats->stats_reset);
- if (can_pstats.user_reset)
+ if (can_pstats->user_reset)
seq_printf(m, " %8ld user statistic resets (USTR)\n",
- can_pstats.user_reset);
+ can_pstats->user_reset);
seq_putc(m, '\n');
return 0;
@@ -275,7 +272,7 @@
static int can_stats_proc_open(struct inode *inode, struct file *file)
{
- return single_open(file, can_stats_proc_show, NULL);
+ return single_open_net(inode, file, can_stats_proc_show);
}
static const struct file_operations can_stats_proc_fops = {
@@ -288,25 +285,28 @@
static int can_reset_stats_proc_show(struct seq_file *m, void *v)
{
+ struct net *net = m->private;
+ struct s_pstats *can_pstats = net->can.can_pstats;
+ struct s_stats *can_stats = net->can.can_stats;
+
user_reset = 1;
- if (can_stattimer.function == can_stat_update) {
+ if (net->can.can_stattimer.function == can_stat_update) {
seq_printf(m, "Scheduled statistic reset #%ld.\n",
- can_pstats.stats_reset + 1);
-
+ can_pstats->stats_reset + 1);
} else {
- if (can_stats.jiffies_init != jiffies)
- can_init_stats();
+ if (can_stats->jiffies_init != jiffies)
+ can_init_stats(net);
seq_printf(m, "Performed statistic reset #%ld.\n",
- can_pstats.stats_reset);
+ can_pstats->stats_reset);
}
return 0;
}
static int can_reset_stats_proc_open(struct inode *inode, struct file *file)
{
- return single_open(file, can_reset_stats_proc_show, NULL);
+ return single_open_net(inode, file, can_reset_stats_proc_show);
}
static const struct file_operations can_reset_stats_proc_fops = {
@@ -325,7 +325,7 @@
static int can_version_proc_open(struct inode *inode, struct file *file)
{
- return single_open(file, can_version_proc_show, NULL);
+ return single_open_net(inode, file, can_version_proc_show);
}
static const struct file_operations can_version_proc_fops = {
@@ -351,20 +351,21 @@
static int can_rcvlist_proc_show(struct seq_file *m, void *v)
{
/* double cast to prevent GCC warning */
- int idx = (int)(long)m->private;
+ int idx = (int)(long)PDE_DATA(m->file->f_inode);
struct net_device *dev;
struct dev_rcv_lists *d;
+ struct net *net = m->private;
seq_printf(m, "\nreceive list '%s':\n", rx_list_name[idx]);
rcu_read_lock();
/* receive list for 'all' CAN devices (dev == NULL) */
- d = &can_rx_alldev_list;
+ d = net->can.can_rx_alldev_list;
can_rcvlist_proc_show_one(m, idx, NULL, d);
/* receive list for registered CAN devices */
- for_each_netdev_rcu(&init_net, dev) {
+ for_each_netdev_rcu(net, dev) {
if (dev->type == ARPHRD_CAN && dev->ml_priv)
can_rcvlist_proc_show_one(m, idx, dev, dev->ml_priv);
}
@@ -377,7 +378,7 @@
static int can_rcvlist_proc_open(struct inode *inode, struct file *file)
{
- return single_open(file, can_rcvlist_proc_show, PDE_DATA(inode));
+ return single_open_net(inode, file, can_rcvlist_proc_show);
}
static const struct file_operations can_rcvlist_proc_fops = {
@@ -417,6 +418,7 @@
{
struct net_device *dev;
struct dev_rcv_lists *d;
+ struct net *net = m->private;
/* RX_SFF */
seq_puts(m, "\nreceive list 'rx_sff':\n");
@@ -424,11 +426,11 @@
rcu_read_lock();
/* sff receive list for 'all' CAN devices (dev == NULL) */
- d = &can_rx_alldev_list;
+ d = net->can.can_rx_alldev_list;
can_rcvlist_proc_show_array(m, NULL, d->rx_sff, ARRAY_SIZE(d->rx_sff));
/* sff receive list for registered CAN devices */
- for_each_netdev_rcu(&init_net, dev) {
+ for_each_netdev_rcu(net, dev) {
if (dev->type == ARPHRD_CAN && dev->ml_priv) {
d = dev->ml_priv;
can_rcvlist_proc_show_array(m, dev, d->rx_sff,
@@ -444,7 +446,7 @@
static int can_rcvlist_sff_proc_open(struct inode *inode, struct file *file)
{
- return single_open(file, can_rcvlist_sff_proc_show, NULL);
+ return single_open_net(inode, file, can_rcvlist_sff_proc_show);
}
static const struct file_operations can_rcvlist_sff_proc_fops = {
@@ -460,6 +462,7 @@
{
struct net_device *dev;
struct dev_rcv_lists *d;
+ struct net *net = m->private;
/* RX_EFF */
seq_puts(m, "\nreceive list 'rx_eff':\n");
@@ -467,11 +470,11 @@
rcu_read_lock();
/* eff receive list for 'all' CAN devices (dev == NULL) */
- d = &can_rx_alldev_list;
+ d = net->can.can_rx_alldev_list;
can_rcvlist_proc_show_array(m, NULL, d->rx_eff, ARRAY_SIZE(d->rx_eff));
/* eff receive list for registered CAN devices */
- for_each_netdev_rcu(&init_net, dev) {
+ for_each_netdev_rcu(net, dev) {
if (dev->type == ARPHRD_CAN && dev->ml_priv) {
d = dev->ml_priv;
can_rcvlist_proc_show_array(m, dev, d->rx_eff,
@@ -487,7 +490,7 @@
static int can_rcvlist_eff_proc_open(struct inode *inode, struct file *file)
{
- return single_open(file, can_rcvlist_eff_proc_show, NULL);
+ return single_open_net(inode, file, can_rcvlist_eff_proc_show);
}
static const struct file_operations can_rcvlist_eff_proc_fops = {
@@ -499,81 +502,85 @@
};
/*
- * proc utility functions
- */
-
-static void can_remove_proc_readentry(const char *name)
-{
- if (can_dir)
- remove_proc_entry(name, can_dir);
-}
-
-/*
* can_init_proc - create main CAN proc directory and procfs entries
*/
-void can_init_proc(void)
+void can_init_proc(struct net *net)
{
/* create /proc/net/can directory */
- can_dir = proc_mkdir("can", init_net.proc_net);
+ net->can.proc_dir = proc_net_mkdir(net, "can", net->proc_net);
- if (!can_dir) {
- pr_info("can: failed to create /proc/net/can.\n");
+ if (!net->can.proc_dir) {
+ printk(KERN_INFO "can: failed to create /proc/net/can . "
+ "CONFIG_PROC_FS missing?\n");
return;
}
/* own procfs entries from the AF_CAN core */
- pde_version = proc_create(CAN_PROC_VERSION, 0644, can_dir,
- &can_version_proc_fops);
- pde_stats = proc_create(CAN_PROC_STATS, 0644, can_dir,
- &can_stats_proc_fops);
- pde_reset_stats = proc_create(CAN_PROC_RESET_STATS, 0644, can_dir,
- &can_reset_stats_proc_fops);
- pde_rcvlist_err = proc_create_data(CAN_PROC_RCVLIST_ERR, 0644, can_dir,
- &can_rcvlist_proc_fops, (void *)RX_ERR);
- pde_rcvlist_all = proc_create_data(CAN_PROC_RCVLIST_ALL, 0644, can_dir,
- &can_rcvlist_proc_fops, (void *)RX_ALL);
- pde_rcvlist_fil = proc_create_data(CAN_PROC_RCVLIST_FIL, 0644, can_dir,
- &can_rcvlist_proc_fops, (void *)RX_FIL);
- pde_rcvlist_inv = proc_create_data(CAN_PROC_RCVLIST_INV, 0644, can_dir,
- &can_rcvlist_proc_fops, (void *)RX_INV);
- pde_rcvlist_eff = proc_create(CAN_PROC_RCVLIST_EFF, 0644, can_dir,
- &can_rcvlist_eff_proc_fops);
- pde_rcvlist_sff = proc_create(CAN_PROC_RCVLIST_SFF, 0644, can_dir,
- &can_rcvlist_sff_proc_fops);
+ net->can.pde_version = proc_create(CAN_PROC_VERSION, 0644,
+ net->can.proc_dir,
+ &can_version_proc_fops);
+ net->can.pde_stats = proc_create(CAN_PROC_STATS, 0644,
+ net->can.proc_dir,
+ &can_stats_proc_fops);
+ net->can.pde_reset_stats = proc_create(CAN_PROC_RESET_STATS, 0644,
+ net->can.proc_dir,
+ &can_reset_stats_proc_fops);
+ net->can.pde_rcvlist_err = proc_create_data(CAN_PROC_RCVLIST_ERR, 0644,
+ net->can.proc_dir,
+ &can_rcvlist_proc_fops,
+ (void *)RX_ERR);
+ net->can.pde_rcvlist_all = proc_create_data(CAN_PROC_RCVLIST_ALL, 0644,
+ net->can.proc_dir,
+ &can_rcvlist_proc_fops,
+ (void *)RX_ALL);
+ net->can.pde_rcvlist_fil = proc_create_data(CAN_PROC_RCVLIST_FIL, 0644,
+ net->can.proc_dir,
+ &can_rcvlist_proc_fops,
+ (void *)RX_FIL);
+ net->can.pde_rcvlist_inv = proc_create_data(CAN_PROC_RCVLIST_INV, 0644,
+ net->can.proc_dir,
+ &can_rcvlist_proc_fops,
+ (void *)RX_INV);
+ net->can.pde_rcvlist_eff = proc_create(CAN_PROC_RCVLIST_EFF, 0644,
+ net->can.proc_dir,
+ &can_rcvlist_eff_proc_fops);
+ net->can.pde_rcvlist_sff = proc_create(CAN_PROC_RCVLIST_SFF, 0644,
+ net->can.proc_dir,
+ &can_rcvlist_sff_proc_fops);
}
/*
* can_remove_proc - remove procfs entries and main CAN proc directory
*/
-void can_remove_proc(void)
+void can_remove_proc(struct net *net)
{
- if (pde_version)
- can_remove_proc_readentry(CAN_PROC_VERSION);
+ if (net->can.pde_version)
+ remove_proc_entry(CAN_PROC_VERSION, net->can.proc_dir);
- if (pde_stats)
- can_remove_proc_readentry(CAN_PROC_STATS);
+ if (net->can.pde_stats)
+ remove_proc_entry(CAN_PROC_STATS, net->can.proc_dir);
- if (pde_reset_stats)
- can_remove_proc_readentry(CAN_PROC_RESET_STATS);
+ if (net->can.pde_reset_stats)
+ remove_proc_entry(CAN_PROC_RESET_STATS, net->can.proc_dir);
- if (pde_rcvlist_err)
- can_remove_proc_readentry(CAN_PROC_RCVLIST_ERR);
+ if (net->can.pde_rcvlist_err)
+ remove_proc_entry(CAN_PROC_RCVLIST_ERR, net->can.proc_dir);
- if (pde_rcvlist_all)
- can_remove_proc_readentry(CAN_PROC_RCVLIST_ALL);
+ if (net->can.pde_rcvlist_all)
+ remove_proc_entry(CAN_PROC_RCVLIST_ALL, net->can.proc_dir);
- if (pde_rcvlist_fil)
- can_remove_proc_readentry(CAN_PROC_RCVLIST_FIL);
+ if (net->can.pde_rcvlist_fil)
+ remove_proc_entry(CAN_PROC_RCVLIST_FIL, net->can.proc_dir);
- if (pde_rcvlist_inv)
- can_remove_proc_readentry(CAN_PROC_RCVLIST_INV);
+ if (net->can.pde_rcvlist_inv)
+ remove_proc_entry(CAN_PROC_RCVLIST_INV, net->can.proc_dir);
- if (pde_rcvlist_eff)
- can_remove_proc_readentry(CAN_PROC_RCVLIST_EFF);
+ if (net->can.pde_rcvlist_eff)
+ remove_proc_entry(CAN_PROC_RCVLIST_EFF, net->can.proc_dir);
- if (pde_rcvlist_sff)
- can_remove_proc_readentry(CAN_PROC_RCVLIST_SFF);
+ if (net->can.pde_rcvlist_sff)
+ remove_proc_entry(CAN_PROC_RCVLIST_SFF, net->can.proc_dir);
- if (can_dir)
- remove_proc_entry("can", init_net.proc_net);
+ if (net->can.proc_dir)
+ remove_proc_entry("can", net->proc_net);
}
diff --git a/net/can/raw.c b/net/can/raw.c
index 6dc546a..864c80d 100644
--- a/net/can/raw.c
+++ b/net/can/raw.c
@@ -181,20 +181,21 @@
kfree_skb(skb);
}
-static int raw_enable_filters(struct net_device *dev, struct sock *sk,
- struct can_filter *filter, int count)
+static int raw_enable_filters(struct net *net, struct net_device *dev,
+ struct sock *sk, struct can_filter *filter,
+ int count)
{
int err = 0;
int i;
for (i = 0; i < count; i++) {
- err = can_rx_register(dev, filter[i].can_id,
+ err = can_rx_register(net, dev, filter[i].can_id,
filter[i].can_mask,
raw_rcv, sk, "raw", sk);
if (err) {
/* clean up successfully registered filters */
while (--i >= 0)
- can_rx_unregister(dev, filter[i].can_id,
+ can_rx_unregister(net, dev, filter[i].can_id,
filter[i].can_mask,
raw_rcv, sk);
break;
@@ -204,57 +205,62 @@
return err;
}
-static int raw_enable_errfilter(struct net_device *dev, struct sock *sk,
- can_err_mask_t err_mask)
+static int raw_enable_errfilter(struct net *net, struct net_device *dev,
+ struct sock *sk, can_err_mask_t err_mask)
{
int err = 0;
if (err_mask)
- err = can_rx_register(dev, 0, err_mask | CAN_ERR_FLAG,
+ err = can_rx_register(net, dev, 0, err_mask | CAN_ERR_FLAG,
raw_rcv, sk, "raw", sk);
return err;
}
-static void raw_disable_filters(struct net_device *dev, struct sock *sk,
- struct can_filter *filter, int count)
+static void raw_disable_filters(struct net *net, struct net_device *dev,
+ struct sock *sk, struct can_filter *filter,
+ int count)
{
int i;
for (i = 0; i < count; i++)
- can_rx_unregister(dev, filter[i].can_id, filter[i].can_mask,
- raw_rcv, sk);
+ can_rx_unregister(net, dev, filter[i].can_id,
+ filter[i].can_mask, raw_rcv, sk);
}
-static inline void raw_disable_errfilter(struct net_device *dev,
+static inline void raw_disable_errfilter(struct net *net,
+ struct net_device *dev,
struct sock *sk,
can_err_mask_t err_mask)
{
if (err_mask)
- can_rx_unregister(dev, 0, err_mask | CAN_ERR_FLAG,
+ can_rx_unregister(net, dev, 0, err_mask | CAN_ERR_FLAG,
raw_rcv, sk);
}
-static inline void raw_disable_allfilters(struct net_device *dev,
+static inline void raw_disable_allfilters(struct net *net,
+ struct net_device *dev,
struct sock *sk)
{
struct raw_sock *ro = raw_sk(sk);
- raw_disable_filters(dev, sk, ro->filter, ro->count);
- raw_disable_errfilter(dev, sk, ro->err_mask);
+ raw_disable_filters(net, dev, sk, ro->filter, ro->count);
+ raw_disable_errfilter(net, dev, sk, ro->err_mask);
}
-static int raw_enable_allfilters(struct net_device *dev, struct sock *sk)
+static int raw_enable_allfilters(struct net *net, struct net_device *dev,
+ struct sock *sk)
{
struct raw_sock *ro = raw_sk(sk);
int err;
- err = raw_enable_filters(dev, sk, ro->filter, ro->count);
+ err = raw_enable_filters(net, dev, sk, ro->filter, ro->count);
if (!err) {
- err = raw_enable_errfilter(dev, sk, ro->err_mask);
+ err = raw_enable_errfilter(net, dev, sk, ro->err_mask);
if (err)
- raw_disable_filters(dev, sk, ro->filter, ro->count);
+ raw_disable_filters(net, dev, sk, ro->filter,
+ ro->count);
}
return err;
@@ -267,7 +273,7 @@
struct raw_sock *ro = container_of(nb, struct raw_sock, notifier);
struct sock *sk = &ro->sk;
- if (!net_eq(dev_net(dev), &init_net))
+ if (!net_eq(dev_net(dev), sock_net(sk)))
return NOTIFY_DONE;
if (dev->type != ARPHRD_CAN)
@@ -282,7 +288,7 @@
lock_sock(sk);
/* remove current filters & unregister */
if (ro->bound)
- raw_disable_allfilters(dev, sk);
+ raw_disable_allfilters(dev_net(dev), dev, sk);
if (ro->count > 1)
kfree(ro->filter);
@@ -358,13 +364,13 @@
if (ro->ifindex) {
struct net_device *dev;
- dev = dev_get_by_index(&init_net, ro->ifindex);
+ dev = dev_get_by_index(sock_net(sk), ro->ifindex);
if (dev) {
- raw_disable_allfilters(dev, sk);
+ raw_disable_allfilters(dev_net(dev), dev, sk);
dev_put(dev);
}
} else
- raw_disable_allfilters(NULL, sk);
+ raw_disable_allfilters(sock_net(sk), NULL, sk);
}
if (ro->count > 1)
@@ -404,7 +410,7 @@
if (addr->can_ifindex) {
struct net_device *dev;
- dev = dev_get_by_index(&init_net, addr->can_ifindex);
+ dev = dev_get_by_index(sock_net(sk), addr->can_ifindex);
if (!dev) {
err = -ENODEV;
goto out;
@@ -420,13 +426,13 @@
ifindex = dev->ifindex;
/* filters set by default/setsockopt */
- err = raw_enable_allfilters(dev, sk);
+ err = raw_enable_allfilters(sock_net(sk), dev, sk);
dev_put(dev);
} else {
ifindex = 0;
/* filters set by default/setsockopt */
- err = raw_enable_allfilters(NULL, sk);
+ err = raw_enable_allfilters(sock_net(sk), NULL, sk);
}
if (!err) {
@@ -435,13 +441,15 @@
if (ro->ifindex) {
struct net_device *dev;
- dev = dev_get_by_index(&init_net, ro->ifindex);
+ dev = dev_get_by_index(sock_net(sk),
+ ro->ifindex);
if (dev) {
- raw_disable_allfilters(dev, sk);
+ raw_disable_allfilters(dev_net(dev),
+ dev, sk);
dev_put(dev);
}
} else
- raw_disable_allfilters(NULL, sk);
+ raw_disable_allfilters(sock_net(sk), NULL, sk);
}
ro->ifindex = ifindex;
ro->bound = 1;
@@ -517,15 +525,16 @@
lock_sock(sk);
if (ro->bound && ro->ifindex)
- dev = dev_get_by_index(&init_net, ro->ifindex);
+ dev = dev_get_by_index(sock_net(sk), ro->ifindex);
if (ro->bound) {
/* (try to) register the new filters */
if (count == 1)
- err = raw_enable_filters(dev, sk, &sfilter, 1);
+ err = raw_enable_filters(sock_net(sk), dev, sk,
+ &sfilter, 1);
else
- err = raw_enable_filters(dev, sk, filter,
- count);
+ err = raw_enable_filters(sock_net(sk), dev, sk,
+ filter, count);
if (err) {
if (count > 1)
kfree(filter);
@@ -533,7 +542,8 @@
}
/* remove old filter registrations */
- raw_disable_filters(dev, sk, ro->filter, ro->count);
+ raw_disable_filters(sock_net(sk), dev, sk, ro->filter,
+ ro->count);
}
/* remove old filter space */
@@ -569,18 +579,20 @@
lock_sock(sk);
if (ro->bound && ro->ifindex)
- dev = dev_get_by_index(&init_net, ro->ifindex);
+ dev = dev_get_by_index(sock_net(sk), ro->ifindex);
/* remove current error mask */
if (ro->bound) {
/* (try to) register the new err_mask */
- err = raw_enable_errfilter(dev, sk, err_mask);
+ err = raw_enable_errfilter(sock_net(sk), dev, sk,
+ err_mask);
if (err)
goto out_err;
/* remove old err_mask registration */
- raw_disable_errfilter(dev, sk, ro->err_mask);
+ raw_disable_errfilter(sock_net(sk), dev, sk,
+ ro->err_mask);
}
/* link new err_mask to the socket */
@@ -741,7 +753,7 @@
return -EINVAL;
}
- dev = dev_get_by_index(&init_net, ifindex);
+ dev = dev_get_by_index(sock_net(sk), ifindex);
if (!dev)
return -ENXIO;
diff --git a/net/core/datagram.c b/net/core/datagram.c
index f4947e7..db1866f2 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -256,8 +256,12 @@
}
spin_unlock_irqrestore(&queue->lock, cpu_flags);
- } while (sk_can_busy_loop(sk) &&
- sk_busy_loop(sk, flags & MSG_DONTWAIT));
+
+ if (!sk_can_busy_loop(sk))
+ break;
+
+ sk_busy_loop(sk, flags & MSG_DONTWAIT);
+ } while (!skb_queue_empty(&sk->sk_receive_queue));
error = -EAGAIN;
@@ -760,7 +764,7 @@
if (msg_data_left(msg) < chunk) {
if (__skb_checksum_complete(skb))
- goto csum_error;
+ return -EINVAL;
if (skb_copy_datagram_msg(skb, hlen, msg, chunk))
goto fault;
} else {
@@ -768,15 +772,16 @@
if (skb_copy_and_csum_datagram(skb, hlen, &msg->msg_iter,
chunk, &csum))
goto fault;
- if (csum_fold(csum))
- goto csum_error;
+
+ if (csum_fold(csum)) {
+ iov_iter_revert(&msg->msg_iter, chunk);
+ return -EINVAL;
+ }
+
if (unlikely(skb->ip_summed == CHECKSUM_COMPLETE))
netdev_rx_csum_fault(skb->dev);
}
return 0;
-csum_error:
- iov_iter_revert(&msg->msg_iter, chunk);
- return -EINVAL;
fault:
return -EFAULT;
}
diff --git a/net/core/dev.c b/net/core/dev.c
index 533a6d6..d07aa5f 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -95,6 +95,7 @@
#include <linux/notifier.h>
#include <linux/skbuff.h>
#include <linux/bpf.h>
+#include <linux/bpf_trace.h>
#include <net/net_namespace.h>
#include <net/sock.h>
#include <net/busy_poll.h>
@@ -2450,6 +2451,9 @@
{
unsigned long flags;
+ if (unlikely(!skb))
+ return;
+
if (likely(atomic_read(&skb->users) == 1)) {
smp_rmb();
atomic_set(&skb->users, 0);
@@ -2972,6 +2976,9 @@
__skb_linearize(skb))
goto out_kfree_skb;
+ if (validate_xmit_xfrm(skb, features))
+ goto out_kfree_skb;
+
/* If packet is not checksummed and device does not
* support checksumming for this protocol, complete
* checksumming here.
@@ -3441,6 +3448,7 @@
int netdev_tstamp_prequeue __read_mostly = 1;
int netdev_budget __read_mostly = 300;
+unsigned int __read_mostly netdev_budget_usecs = 2000;
int weight_p __read_mostly = 64; /* old backlog weight */
int dev_weight_rx_bias __read_mostly = 1; /* bias for backlog weight */
int dev_weight_tx_bias __read_mostly = 1; /* bias for output_queue quota */
@@ -4240,13 +4248,132 @@
*/
current->flags |= PF_MEMALLOC;
ret = __netif_receive_skb_core(skb, true);
- tsk_restore_flags(current, pflags, PF_MEMALLOC);
+ current_restore_flags(pflags, PF_MEMALLOC);
} else
ret = __netif_receive_skb_core(skb, false);
return ret;
}
+static struct static_key generic_xdp_needed __read_mostly;
+
+static int generic_xdp_install(struct net_device *dev, struct netdev_xdp *xdp)
+{
+ struct bpf_prog *new = xdp->prog;
+ int ret = 0;
+
+ switch (xdp->command) {
+ case XDP_SETUP_PROG: {
+ struct bpf_prog *old = rtnl_dereference(dev->xdp_prog);
+
+ rcu_assign_pointer(dev->xdp_prog, new);
+ if (old)
+ bpf_prog_put(old);
+
+ if (old && !new) {
+ static_key_slow_dec(&generic_xdp_needed);
+ } else if (new && !old) {
+ static_key_slow_inc(&generic_xdp_needed);
+ dev_disable_lro(dev);
+ }
+ break;
+ }
+
+ case XDP_QUERY_PROG:
+ xdp->prog_attached = !!rcu_access_pointer(dev->xdp_prog);
+ break;
+
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static u32 netif_receive_generic_xdp(struct sk_buff *skb,
+ struct bpf_prog *xdp_prog)
+{
+ struct xdp_buff xdp;
+ u32 act = XDP_DROP;
+ void *orig_data;
+ int hlen, off;
+ u32 mac_len;
+
+ /* Reinjected packets coming from act_mirred or similar should
+ * not get XDP generic processing.
+ */
+ if (skb_cloned(skb))
+ return XDP_PASS;
+
+ if (skb_linearize(skb))
+ goto do_drop;
+
+ /* The XDP program wants to see the packet starting at the MAC
+ * header.
+ */
+ mac_len = skb->data - skb_mac_header(skb);
+ hlen = skb_headlen(skb) + mac_len;
+ xdp.data = skb->data - mac_len;
+ xdp.data_end = xdp.data + hlen;
+ xdp.data_hard_start = skb->data - skb_headroom(skb);
+ orig_data = xdp.data;
+
+ act = bpf_prog_run_xdp(xdp_prog, &xdp);
+
+ off = xdp.data - orig_data;
+ if (off > 0)
+ __skb_pull(skb, off);
+ else if (off < 0)
+ __skb_push(skb, -off);
+
+ switch (act) {
+ case XDP_TX:
+ __skb_push(skb, mac_len);
+ /* fall through */
+ case XDP_PASS:
+ break;
+
+ default:
+ bpf_warn_invalid_xdp_action(act);
+ /* fall through */
+ case XDP_ABORTED:
+ trace_xdp_exception(skb->dev, xdp_prog, act);
+ /* fall through */
+ case XDP_DROP:
+ do_drop:
+ kfree_skb(skb);
+ break;
+ }
+
+ return act;
+}
+
+/* When doing generic XDP we have to bypass the qdisc layer and the
+ * network taps in order to match in-driver-XDP behavior.
+ */
+static void generic_xdp_tx(struct sk_buff *skb, struct bpf_prog *xdp_prog)
+{
+ struct net_device *dev = skb->dev;
+ struct netdev_queue *txq;
+ bool free_skb = true;
+ int cpu, rc;
+
+ txq = netdev_pick_tx(dev, skb, NULL);
+ cpu = smp_processor_id();
+ HARD_TX_LOCK(dev, txq, cpu);
+ if (!netif_xmit_stopped(txq)) {
+ rc = netdev_start_xmit(skb, dev, txq, 0);
+ if (dev_xmit_complete(rc))
+ free_skb = false;
+ }
+ HARD_TX_UNLOCK(dev, txq);
+ if (free_skb) {
+ trace_xdp_exception(dev, xdp_prog, XDP_TX);
+ kfree_skb(skb);
+ }
+}
+
static int netif_receive_skb_internal(struct sk_buff *skb)
{
int ret;
@@ -4258,6 +4385,21 @@
rcu_read_lock();
+ if (static_key_false(&generic_xdp_needed)) {
+ struct bpf_prog *xdp_prog = rcu_dereference(skb->dev->xdp_prog);
+
+ if (xdp_prog) {
+ u32 act = netif_receive_generic_xdp(skb, xdp_prog);
+
+ if (act != XDP_PASS) {
+ rcu_read_unlock();
+ if (act == XDP_TX)
+ generic_xdp_tx(skb, xdp_prog);
+ return NET_RX_DROP;
+ }
+ }
+ }
+
#ifdef CONFIG_RPS
if (static_key_false(&rps_needed)) {
struct rps_dev_flow voidflow, *rflow = &voidflow;
@@ -4490,7 +4632,7 @@
enum gro_result ret;
int grow;
- if (!(skb->dev->features & NETIF_F_GRO))
+ if (netif_elide_gro(skb->dev))
goto normal;
if (skb->csum_bad)
@@ -5060,27 +5202,28 @@
do_softirq();
}
-bool sk_busy_loop(struct sock *sk, int nonblock)
+void napi_busy_loop(unsigned int napi_id,
+ bool (*loop_end)(void *, unsigned long),
+ void *loop_end_arg)
{
- unsigned long end_time = !nonblock ? sk_busy_loop_end_time(sk) : 0;
+ unsigned long start_time = loop_end ? busy_loop_current_time() : 0;
int (*napi_poll)(struct napi_struct *napi, int budget);
void *have_poll_lock = NULL;
struct napi_struct *napi;
- int rc;
restart:
- rc = false;
napi_poll = NULL;
rcu_read_lock();
- napi = napi_by_id(sk->sk_napi_id);
+ napi = napi_by_id(napi_id);
if (!napi)
goto out;
preempt_disable();
for (;;) {
- rc = 0;
+ int work = 0;
+
local_bh_disable();
if (!napi_poll) {
unsigned long val = READ_ONCE(napi->state);
@@ -5098,16 +5241,15 @@
have_poll_lock = netpoll_poll_lock(napi);
napi_poll = napi->poll;
}
- rc = napi_poll(napi, BUSY_POLL_BUDGET);
- trace_napi_poll(napi, rc, BUSY_POLL_BUDGET);
+ work = napi_poll(napi, BUSY_POLL_BUDGET);
+ trace_napi_poll(napi, work, BUSY_POLL_BUDGET);
count:
- if (rc > 0)
- __NET_ADD_STATS(sock_net(sk),
- LINUX_MIB_BUSYPOLLRXPACKETS, rc);
+ if (work > 0)
+ __NET_ADD_STATS(dev_net(napi->dev),
+ LINUX_MIB_BUSYPOLLRXPACKETS, work);
local_bh_enable();
- if (nonblock || !skb_queue_empty(&sk->sk_receive_queue) ||
- busy_loop_timeout(end_time))
+ if (!loop_end || loop_end(loop_end_arg, start_time))
break;
if (unlikely(need_resched())) {
@@ -5116,9 +5258,8 @@
preempt_enable();
rcu_read_unlock();
cond_resched();
- rc = !skb_queue_empty(&sk->sk_receive_queue);
- if (rc || busy_loop_timeout(end_time))
- return rc;
+ if (loop_end(loop_end_arg, start_time))
+ return;
goto restart;
}
cpu_relax();
@@ -5126,12 +5267,10 @@
if (napi_poll)
busy_poll_stop(napi, have_poll_lock);
preempt_enable();
- rc = !skb_queue_empty(&sk->sk_receive_queue);
out:
rcu_read_unlock();
- return rc;
}
-EXPORT_SYMBOL(sk_busy_loop);
+EXPORT_SYMBOL(napi_busy_loop);
#endif /* CONFIG_NET_RX_BUSY_POLL */
@@ -5143,10 +5282,10 @@
spin_lock(&napi_hash_lock);
- /* 0..NR_CPUS+1 range is reserved for sender_cpu use */
+ /* 0..NR_CPUS range is reserved for sender_cpu use */
do {
- if (unlikely(++napi_gen_id < NR_CPUS + 1))
- napi_gen_id = NR_CPUS + 1;
+ if (unlikely(++napi_gen_id < MIN_NAPI_ID))
+ napi_gen_id = MIN_NAPI_ID;
} while (napi_by_id(napi_gen_id));
napi->napi_id = napi_gen_id;
@@ -5310,7 +5449,8 @@
static __latent_entropy void net_rx_action(struct softirq_action *h)
{
struct softnet_data *sd = this_cpu_ptr(&softnet_data);
- unsigned long time_limit = jiffies + 2;
+ unsigned long time_limit = jiffies +
+ usecs_to_jiffies(netdev_budget_usecs);
int budget = netdev_budget;
LIST_HEAD(list);
LIST_HEAD(repoll);
@@ -6714,13 +6854,16 @@
/**
* dev_change_xdp_fd - set or clear a bpf program for a device rx path
* @dev: device
+ * @extack: netlink extended ack
* @fd: new program fd or negative value to clear
* @flags: xdp-related flags
*
* Set or clear a bpf program for a device
*/
-int dev_change_xdp_fd(struct net_device *dev, int fd, u32 flags)
+int dev_change_xdp_fd(struct net_device *dev, struct netlink_ext_ack *extack,
+ int fd, u32 flags)
{
+ int (*xdp_op)(struct net_device *dev, struct netdev_xdp *xdp);
const struct net_device_ops *ops = dev->netdev_ops;
struct bpf_prog *prog = NULL;
struct netdev_xdp xdp;
@@ -6728,14 +6871,16 @@
ASSERT_RTNL();
- if (!ops->ndo_xdp)
- return -EOPNOTSUPP;
+ xdp_op = ops->ndo_xdp;
+ if (!xdp_op || (flags & XDP_FLAGS_SKB_MODE))
+ xdp_op = generic_xdp_install;
+
if (fd >= 0) {
if (flags & XDP_FLAGS_UPDATE_IF_NOEXIST) {
memset(&xdp, 0, sizeof(xdp));
xdp.command = XDP_QUERY_PROG;
- err = ops->ndo_xdp(dev, &xdp);
+ err = xdp_op(dev, &xdp);
if (err < 0)
return err;
if (xdp.prog_attached)
@@ -6749,9 +6894,10 @@
memset(&xdp, 0, sizeof(xdp));
xdp.command = XDP_SETUP_PROG;
+ xdp.extack = extack;
xdp.prog = prog;
- err = ops->ndo_xdp(dev, &xdp);
+ err = xdp_op(dev, &xdp);
if (err < 0 && prog)
bpf_prog_put(prog);
@@ -7102,13 +7248,10 @@
else
netif_dormant_off(dev);
- if (netif_carrier_ok(rootdev)) {
- if (!netif_carrier_ok(dev))
- netif_carrier_on(dev);
- } else {
- if (netif_carrier_ok(dev))
- netif_carrier_off(dev);
- }
+ if (netif_carrier_ok(rootdev))
+ netif_carrier_on(dev);
+ else
+ netif_carrier_off(dev);
}
EXPORT_SYMBOL(netif_stacked_transfer_operstate);
@@ -7791,6 +7934,7 @@
void free_netdev(struct net_device *dev)
{
struct napi_struct *p, *n;
+ struct bpf_prog *prog;
might_sleep();
netif_free_tx_queues(dev);
@@ -7809,6 +7953,12 @@
free_percpu(dev->pcpu_refcnt);
dev->pcpu_refcnt = NULL;
+ prog = rcu_dereference_protected(dev->xdp_prog, 1);
+ if (prog) {
+ bpf_prog_put(prog);
+ static_key_slow_dec(&generic_xdp_needed);
+ }
+
/* Compatibility with error handling in drivers */
if (dev->reg_state == NETREG_UNINITIALIZED) {
netdev_freemem(dev);
diff --git a/net/core/devlink.c b/net/core/devlink.c
index e9c1e6a..b0b87a2 100644
--- a/net/core/devlink.c
+++ b/net/core/devlink.c
@@ -1397,10 +1397,10 @@
u32 seq, int flags)
{
const struct devlink_ops *ops = devlink->ops;
+ u8 inline_mode, encap_mode;
void *hdr;
int err = 0;
u16 mode;
- u8 inline_mode;
hdr = genlmsg_put(msg, portid, seq, &devlink_nl_family, flags, cmd);
if (!hdr)
@@ -1429,6 +1429,15 @@
goto nla_put_failure;
}
+ if (ops->eswitch_encap_mode_get) {
+ err = ops->eswitch_encap_mode_get(devlink, &encap_mode);
+ if (err)
+ goto nla_put_failure;
+ err = nla_put_u8(msg, DEVLINK_ATTR_ESWITCH_ENCAP_MODE, encap_mode);
+ if (err)
+ goto nla_put_failure;
+ }
+
genlmsg_end(msg, hdr);
return 0;
@@ -1468,9 +1477,9 @@
{
struct devlink *devlink = info->user_ptr[0];
const struct devlink_ops *ops = devlink->ops;
- u16 mode;
- u8 inline_mode;
+ u8 inline_mode, encap_mode;
int err = 0;
+ u16 mode;
if (!ops)
return -EOPNOTSUPP;
@@ -1494,9 +1503,697 @@
return err;
}
+ if (info->attrs[DEVLINK_ATTR_ESWITCH_ENCAP_MODE]) {
+ if (!ops->eswitch_encap_mode_set)
+ return -EOPNOTSUPP;
+ encap_mode = nla_get_u8(info->attrs[DEVLINK_ATTR_ESWITCH_ENCAP_MODE]);
+ err = ops->eswitch_encap_mode_set(devlink, encap_mode);
+ if (err)
+ return err;
+ }
+
return 0;
}
+int devlink_dpipe_match_put(struct sk_buff *skb,
+ struct devlink_dpipe_match *match)
+{
+ struct devlink_dpipe_header *header = match->header;
+ struct devlink_dpipe_field *field = &header->fields[match->field_id];
+ struct nlattr *match_attr;
+
+ match_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_MATCH);
+ if (!match_attr)
+ return -EMSGSIZE;
+
+ if (nla_put_u32(skb, DEVLINK_ATTR_DPIPE_MATCH_TYPE, match->type) ||
+ nla_put_u32(skb, DEVLINK_ATTR_DPIPE_HEADER_INDEX, match->header_index) ||
+ nla_put_u32(skb, DEVLINK_ATTR_DPIPE_HEADER_ID, header->id) ||
+ nla_put_u32(skb, DEVLINK_ATTR_DPIPE_FIELD_ID, field->id) ||
+ nla_put_u8(skb, DEVLINK_ATTR_DPIPE_HEADER_GLOBAL, header->global))
+ goto nla_put_failure;
+
+ nla_nest_end(skb, match_attr);
+ return 0;
+
+nla_put_failure:
+ nla_nest_cancel(skb, match_attr);
+ return -EMSGSIZE;
+}
+EXPORT_SYMBOL_GPL(devlink_dpipe_match_put);
+
+static int devlink_dpipe_matches_put(struct devlink_dpipe_table *table,
+ struct sk_buff *skb)
+{
+ struct nlattr *matches_attr;
+
+ matches_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_TABLE_MATCHES);
+ if (!matches_attr)
+ return -EMSGSIZE;
+
+ if (table->table_ops->matches_dump(table->priv, skb))
+ goto nla_put_failure;
+
+ nla_nest_end(skb, matches_attr);
+ return 0;
+
+nla_put_failure:
+ nla_nest_cancel(skb, matches_attr);
+ return -EMSGSIZE;
+}
+
+int devlink_dpipe_action_put(struct sk_buff *skb,
+ struct devlink_dpipe_action *action)
+{
+ struct devlink_dpipe_header *header = action->header;
+ struct devlink_dpipe_field *field = &header->fields[action->field_id];
+ struct nlattr *action_attr;
+
+ action_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_ACTION);
+ if (!action_attr)
+ return -EMSGSIZE;
+
+ if (nla_put_u32(skb, DEVLINK_ATTR_DPIPE_ACTION_TYPE, action->type) ||
+ nla_put_u32(skb, DEVLINK_ATTR_DPIPE_HEADER_INDEX, action->header_index) ||
+ nla_put_u32(skb, DEVLINK_ATTR_DPIPE_HEADER_ID, header->id) ||
+ nla_put_u32(skb, DEVLINK_ATTR_DPIPE_FIELD_ID, field->id) ||
+ nla_put_u8(skb, DEVLINK_ATTR_DPIPE_HEADER_GLOBAL, header->global))
+ goto nla_put_failure;
+
+ nla_nest_end(skb, action_attr);
+ return 0;
+
+nla_put_failure:
+ nla_nest_cancel(skb, action_attr);
+ return -EMSGSIZE;
+}
+EXPORT_SYMBOL_GPL(devlink_dpipe_action_put);
+
+static int devlink_dpipe_actions_put(struct devlink_dpipe_table *table,
+ struct sk_buff *skb)
+{
+ struct nlattr *actions_attr;
+
+ actions_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_TABLE_ACTIONS);
+ if (!actions_attr)
+ return -EMSGSIZE;
+
+ if (table->table_ops->actions_dump(table->priv, skb))
+ goto nla_put_failure;
+
+ nla_nest_end(skb, actions_attr);
+ return 0;
+
+nla_put_failure:
+ nla_nest_cancel(skb, actions_attr);
+ return -EMSGSIZE;
+}
+
+static int devlink_dpipe_table_put(struct sk_buff *skb,
+ struct devlink_dpipe_table *table)
+{
+ struct nlattr *table_attr;
+
+ table_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_TABLE);
+ if (!table_attr)
+ return -EMSGSIZE;
+
+ if (nla_put_string(skb, DEVLINK_ATTR_DPIPE_TABLE_NAME, table->name) ||
+ nla_put_u64_64bit(skb, DEVLINK_ATTR_DPIPE_TABLE_SIZE, table->size,
+ DEVLINK_ATTR_PAD))
+ goto nla_put_failure;
+ if (nla_put_u8(skb, DEVLINK_ATTR_DPIPE_TABLE_COUNTERS_ENABLED,
+ table->counters_enabled))
+ goto nla_put_failure;
+
+ if (devlink_dpipe_matches_put(table, skb))
+ goto nla_put_failure;
+
+ if (devlink_dpipe_actions_put(table, skb))
+ goto nla_put_failure;
+
+ nla_nest_end(skb, table_attr);
+ return 0;
+
+nla_put_failure:
+ nla_nest_cancel(skb, table_attr);
+ return -EMSGSIZE;
+}
+
+static int devlink_dpipe_send_and_alloc_skb(struct sk_buff **pskb,
+ struct genl_info *info)
+{
+ int err;
+
+ if (*pskb) {
+ err = genlmsg_reply(*pskb, info);
+ if (err)
+ return err;
+ }
+ *pskb = genlmsg_new(GENLMSG_DEFAULT_SIZE, GFP_KERNEL);
+ if (!*pskb)
+ return -ENOMEM;
+ return 0;
+}
+
+static int devlink_dpipe_tables_fill(struct genl_info *info,
+ enum devlink_command cmd, int flags,
+ struct list_head *dpipe_tables,
+ const char *table_name)
+{
+ struct devlink *devlink = info->user_ptr[0];
+ struct devlink_dpipe_table *table;
+ struct nlattr *tables_attr;
+ struct sk_buff *skb = NULL;
+ struct nlmsghdr *nlh;
+ bool incomplete;
+ void *hdr;
+ int i;
+ int err;
+
+ table = list_first_entry(dpipe_tables,
+ struct devlink_dpipe_table, list);
+start_again:
+ err = devlink_dpipe_send_and_alloc_skb(&skb, info);
+ if (err)
+ return err;
+
+ hdr = genlmsg_put(skb, info->snd_portid, info->snd_seq,
+ &devlink_nl_family, NLM_F_MULTI, cmd);
+ if (!hdr)
+ return -EMSGSIZE;
+
+ if (devlink_nl_put_handle(skb, devlink))
+ goto nla_put_failure;
+ tables_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_TABLES);
+ if (!tables_attr)
+ goto nla_put_failure;
+
+ i = 0;
+ incomplete = false;
+ list_for_each_entry_from(table, dpipe_tables, list) {
+ if (!table_name) {
+ err = devlink_dpipe_table_put(skb, table);
+ if (err) {
+ if (!i)
+ goto err_table_put;
+ incomplete = true;
+ break;
+ }
+ } else {
+ if (!strcmp(table->name, table_name)) {
+ err = devlink_dpipe_table_put(skb, table);
+ if (err)
+ break;
+ }
+ }
+ i++;
+ }
+
+ nla_nest_end(skb, tables_attr);
+ genlmsg_end(skb, hdr);
+ if (incomplete)
+ goto start_again;
+
+send_done:
+ nlh = nlmsg_put(skb, info->snd_portid, info->snd_seq,
+ NLMSG_DONE, 0, flags | NLM_F_MULTI);
+ if (!nlh) {
+ err = devlink_dpipe_send_and_alloc_skb(&skb, info);
+ if (err)
+ goto err_skb_send_alloc;
+ goto send_done;
+ }
+
+ return genlmsg_reply(skb, info);
+
+nla_put_failure:
+ err = -EMSGSIZE;
+err_table_put:
+err_skb_send_alloc:
+ genlmsg_cancel(skb, hdr);
+ nlmsg_free(skb);
+ return err;
+}
+
+static int devlink_nl_cmd_dpipe_table_get(struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct devlink *devlink = info->user_ptr[0];
+ const char *table_name = NULL;
+
+ if (info->attrs[DEVLINK_ATTR_DPIPE_TABLE_NAME])
+ table_name = nla_data(info->attrs[DEVLINK_ATTR_DPIPE_TABLE_NAME]);
+
+ return devlink_dpipe_tables_fill(info, DEVLINK_CMD_DPIPE_TABLE_GET, 0,
+ &devlink->dpipe_table_list,
+ table_name);
+}
+
+static int devlink_dpipe_value_put(struct sk_buff *skb,
+ struct devlink_dpipe_value *value)
+{
+ if (nla_put(skb, DEVLINK_ATTR_DPIPE_VALUE,
+ value->value_size, value->value))
+ return -EMSGSIZE;
+ if (value->mask)
+ if (nla_put(skb, DEVLINK_ATTR_DPIPE_VALUE_MASK,
+ value->value_size, value->mask))
+ return -EMSGSIZE;
+ if (value->mapping_valid)
+ if (nla_put_u32(skb, DEVLINK_ATTR_DPIPE_VALUE_MAPPING,
+ value->mapping_value))
+ return -EMSGSIZE;
+ return 0;
+}
+
+static int devlink_dpipe_action_value_put(struct sk_buff *skb,
+ struct devlink_dpipe_value *value)
+{
+ if (!value->action)
+ return -EINVAL;
+ if (devlink_dpipe_action_put(skb, value->action))
+ return -EMSGSIZE;
+ if (devlink_dpipe_value_put(skb, value))
+ return -EMSGSIZE;
+ return 0;
+}
+
+static int devlink_dpipe_action_values_put(struct sk_buff *skb,
+ struct devlink_dpipe_value *values,
+ unsigned int values_count)
+{
+ struct nlattr *action_attr;
+ int i;
+ int err;
+
+ for (i = 0; i < values_count; i++) {
+ action_attr = nla_nest_start(skb,
+ DEVLINK_ATTR_DPIPE_ACTION_VALUE);
+ if (!action_attr)
+ return -EMSGSIZE;
+ err = devlink_dpipe_action_value_put(skb, &values[i]);
+ if (err)
+ goto err_action_value_put;
+ nla_nest_end(skb, action_attr);
+ }
+ return 0;
+
+err_action_value_put:
+ nla_nest_cancel(skb, action_attr);
+ return err;
+}
+
+static int devlink_dpipe_match_value_put(struct sk_buff *skb,
+ struct devlink_dpipe_value *value)
+{
+ if (!value->match)
+ return -EINVAL;
+ if (devlink_dpipe_match_put(skb, value->match))
+ return -EMSGSIZE;
+ if (devlink_dpipe_value_put(skb, value))
+ return -EMSGSIZE;
+ return 0;
+}
+
+static int devlink_dpipe_match_values_put(struct sk_buff *skb,
+ struct devlink_dpipe_value *values,
+ unsigned int values_count)
+{
+ struct nlattr *match_attr;
+ int i;
+ int err;
+
+ for (i = 0; i < values_count; i++) {
+ match_attr = nla_nest_start(skb,
+ DEVLINK_ATTR_DPIPE_MATCH_VALUE);
+ if (!match_attr)
+ return -EMSGSIZE;
+ err = devlink_dpipe_match_value_put(skb, &values[i]);
+ if (err)
+ goto err_match_value_put;
+ nla_nest_end(skb, match_attr);
+ }
+ return 0;
+
+err_match_value_put:
+ nla_nest_cancel(skb, match_attr);
+ return err;
+}
+
+static int devlink_dpipe_entry_put(struct sk_buff *skb,
+ struct devlink_dpipe_entry *entry)
+{
+ struct nlattr *entry_attr, *matches_attr, *actions_attr;
+ int err;
+
+ entry_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_ENTRY);
+ if (!entry_attr)
+ return -EMSGSIZE;
+
+ if (nla_put_u64_64bit(skb, DEVLINK_ATTR_DPIPE_ENTRY_INDEX, entry->index,
+ DEVLINK_ATTR_PAD))
+ goto nla_put_failure;
+ if (entry->counter_valid)
+ if (nla_put_u64_64bit(skb, DEVLINK_ATTR_DPIPE_ENTRY_COUNTER,
+ entry->counter, DEVLINK_ATTR_PAD))
+ goto nla_put_failure;
+
+ matches_attr = nla_nest_start(skb,
+ DEVLINK_ATTR_DPIPE_ENTRY_MATCH_VALUES);
+ if (!matches_attr)
+ goto nla_put_failure;
+
+ err = devlink_dpipe_match_values_put(skb, entry->match_values,
+ entry->match_values_count);
+ if (err) {
+ nla_nest_cancel(skb, matches_attr);
+ goto err_match_values_put;
+ }
+ nla_nest_end(skb, matches_attr);
+
+ actions_attr = nla_nest_start(skb,
+ DEVLINK_ATTR_DPIPE_ENTRY_ACTION_VALUES);
+ if (!actions_attr)
+ goto nla_put_failure;
+
+ err = devlink_dpipe_action_values_put(skb, entry->action_values,
+ entry->action_values_count);
+ if (err) {
+ nla_nest_cancel(skb, actions_attr);
+ goto err_action_values_put;
+ }
+ nla_nest_end(skb, actions_attr);
+
+ nla_nest_end(skb, entry_attr);
+ return 0;
+
+nla_put_failure:
+ err = -EMSGSIZE;
+err_match_values_put:
+err_action_values_put:
+ nla_nest_cancel(skb, entry_attr);
+ return err;
+}
+
+static struct devlink_dpipe_table *
+devlink_dpipe_table_find(struct list_head *dpipe_tables,
+ const char *table_name)
+{
+ struct devlink_dpipe_table *table;
+
+ list_for_each_entry_rcu(table, dpipe_tables, list) {
+ if (!strcmp(table->name, table_name))
+ return table;
+ }
+ return NULL;
+}
+
+int devlink_dpipe_entry_ctx_prepare(struct devlink_dpipe_dump_ctx *dump_ctx)
+{
+ struct devlink *devlink;
+ int err;
+
+ err = devlink_dpipe_send_and_alloc_skb(&dump_ctx->skb,
+ dump_ctx->info);
+ if (err)
+ return err;
+
+ dump_ctx->hdr = genlmsg_put(dump_ctx->skb,
+ dump_ctx->info->snd_portid,
+ dump_ctx->info->snd_seq,
+ &devlink_nl_family, NLM_F_MULTI,
+ dump_ctx->cmd);
+ if (!dump_ctx->hdr)
+ goto nla_put_failure;
+
+ devlink = dump_ctx->info->user_ptr[0];
+ if (devlink_nl_put_handle(dump_ctx->skb, devlink))
+ goto nla_put_failure;
+ dump_ctx->nest = nla_nest_start(dump_ctx->skb,
+ DEVLINK_ATTR_DPIPE_ENTRIES);
+ if (!dump_ctx->nest)
+ goto nla_put_failure;
+ return 0;
+
+nla_put_failure:
+ genlmsg_cancel(dump_ctx->skb, dump_ctx->hdr);
+ nlmsg_free(dump_ctx->skb);
+ return -EMSGSIZE;
+}
+EXPORT_SYMBOL_GPL(devlink_dpipe_entry_ctx_prepare);
+
+int devlink_dpipe_entry_ctx_append(struct devlink_dpipe_dump_ctx *dump_ctx,
+ struct devlink_dpipe_entry *entry)
+{
+ return devlink_dpipe_entry_put(dump_ctx->skb, entry);
+}
+EXPORT_SYMBOL_GPL(devlink_dpipe_entry_ctx_append);
+
+int devlink_dpipe_entry_ctx_close(struct devlink_dpipe_dump_ctx *dump_ctx)
+{
+ nla_nest_end(dump_ctx->skb, dump_ctx->nest);
+ genlmsg_end(dump_ctx->skb, dump_ctx->hdr);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(devlink_dpipe_entry_ctx_close);
+
+static int devlink_dpipe_entries_fill(struct genl_info *info,
+ enum devlink_command cmd, int flags,
+ struct devlink_dpipe_table *table)
+{
+ struct devlink_dpipe_dump_ctx dump_ctx;
+ struct nlmsghdr *nlh;
+ int err;
+
+ dump_ctx.skb = NULL;
+ dump_ctx.cmd = cmd;
+ dump_ctx.info = info;
+
+ err = table->table_ops->entries_dump(table->priv,
+ table->counters_enabled,
+ &dump_ctx);
+ if (err)
+ goto err_entries_dump;
+
+send_done:
+ nlh = nlmsg_put(dump_ctx.skb, info->snd_portid, info->snd_seq,
+ NLMSG_DONE, 0, flags | NLM_F_MULTI);
+ if (!nlh) {
+ err = devlink_dpipe_send_and_alloc_skb(&dump_ctx.skb, info);
+ if (err)
+ goto err_skb_send_alloc;
+ goto send_done;
+ }
+ return genlmsg_reply(dump_ctx.skb, info);
+
+err_entries_dump:
+err_skb_send_alloc:
+ genlmsg_cancel(dump_ctx.skb, dump_ctx.hdr);
+ nlmsg_free(dump_ctx.skb);
+ return err;
+}
+
+static int devlink_nl_cmd_dpipe_entries_get(struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct devlink *devlink = info->user_ptr[0];
+ struct devlink_dpipe_table *table;
+ const char *table_name;
+
+ if (!info->attrs[DEVLINK_ATTR_DPIPE_TABLE_NAME])
+ return -EINVAL;
+
+ table_name = nla_data(info->attrs[DEVLINK_ATTR_DPIPE_TABLE_NAME]);
+ table = devlink_dpipe_table_find(&devlink->dpipe_table_list,
+ table_name);
+ if (!table)
+ return -EINVAL;
+
+ if (!table->table_ops->entries_dump)
+ return -EINVAL;
+
+ return devlink_dpipe_entries_fill(info, DEVLINK_CMD_DPIPE_ENTRIES_GET,
+ 0, table);
+}
+
+static int devlink_dpipe_fields_put(struct sk_buff *skb,
+ const struct devlink_dpipe_header *header)
+{
+ struct devlink_dpipe_field *field;
+ struct nlattr *field_attr;
+ int i;
+
+ for (i = 0; i < header->fields_count; i++) {
+ field = &header->fields[i];
+ field_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_FIELD);
+ if (!field_attr)
+ return -EMSGSIZE;
+ if (nla_put_string(skb, DEVLINK_ATTR_DPIPE_FIELD_NAME, field->name) ||
+ nla_put_u32(skb, DEVLINK_ATTR_DPIPE_FIELD_ID, field->id) ||
+ nla_put_u32(skb, DEVLINK_ATTR_DPIPE_FIELD_BITWIDTH, field->bitwidth) ||
+ nla_put_u32(skb, DEVLINK_ATTR_DPIPE_FIELD_MAPPING_TYPE, field->mapping_type))
+ goto nla_put_failure;
+ nla_nest_end(skb, field_attr);
+ }
+ return 0;
+
+nla_put_failure:
+ nla_nest_cancel(skb, field_attr);
+ return -EMSGSIZE;
+}
+
+static int devlink_dpipe_header_put(struct sk_buff *skb,
+ struct devlink_dpipe_header *header)
+{
+ struct nlattr *fields_attr, *header_attr;
+ int err;
+
+ header_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_HEADER);
+ if (!header_attr)
+ return -EMSGSIZE;
+
+ if (nla_put_string(skb, DEVLINK_ATTR_DPIPE_HEADER_NAME, header->name) ||
+ nla_put_u32(skb, DEVLINK_ATTR_DPIPE_HEADER_ID, header->id) ||
+ nla_put_u8(skb, DEVLINK_ATTR_DPIPE_HEADER_GLOBAL, header->global))
+ goto nla_put_failure;
+
+ fields_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_HEADER_FIELDS);
+ if (!fields_attr)
+ goto nla_put_failure;
+
+ err = devlink_dpipe_fields_put(skb, header);
+ if (err) {
+ nla_nest_cancel(skb, fields_attr);
+ goto nla_put_failure;
+ }
+ nla_nest_end(skb, fields_attr);
+ nla_nest_end(skb, header_attr);
+ return 0;
+
+nla_put_failure:
+ err = -EMSGSIZE;
+ nla_nest_cancel(skb, header_attr);
+ return err;
+}
+
+static int devlink_dpipe_headers_fill(struct genl_info *info,
+ enum devlink_command cmd, int flags,
+ struct devlink_dpipe_headers *
+ dpipe_headers)
+{
+ struct devlink *devlink = info->user_ptr[0];
+ struct nlattr *headers_attr;
+ struct sk_buff *skb = NULL;
+ struct nlmsghdr *nlh;
+ void *hdr;
+ int i, j;
+ int err;
+
+ i = 0;
+start_again:
+ err = devlink_dpipe_send_and_alloc_skb(&skb, info);
+ if (err)
+ return err;
+
+ hdr = genlmsg_put(skb, info->snd_portid, info->snd_seq,
+ &devlink_nl_family, NLM_F_MULTI, cmd);
+ if (!hdr)
+ return -EMSGSIZE;
+
+ if (devlink_nl_put_handle(skb, devlink))
+ goto nla_put_failure;
+ headers_attr = nla_nest_start(skb, DEVLINK_ATTR_DPIPE_HEADERS);
+ if (!headers_attr)
+ goto nla_put_failure;
+
+ j = 0;
+ for (; i < dpipe_headers->headers_count; i++) {
+ err = devlink_dpipe_header_put(skb, dpipe_headers->headers[i]);
+ if (err) {
+ if (!j)
+ goto err_table_put;
+ break;
+ }
+ j++;
+ }
+ nla_nest_end(skb, headers_attr);
+ genlmsg_end(skb, hdr);
+ if (i != dpipe_headers->headers_count)
+ goto start_again;
+
+send_done:
+ nlh = nlmsg_put(skb, info->snd_portid, info->snd_seq,
+ NLMSG_DONE, 0, flags | NLM_F_MULTI);
+ if (!nlh) {
+ err = devlink_dpipe_send_and_alloc_skb(&skb, info);
+ if (err)
+ goto err_skb_send_alloc;
+ goto send_done;
+ }
+ return genlmsg_reply(skb, info);
+
+nla_put_failure:
+ err = -EMSGSIZE;
+err_table_put:
+err_skb_send_alloc:
+ genlmsg_cancel(skb, hdr);
+ nlmsg_free(skb);
+ return err;
+}
+
+static int devlink_nl_cmd_dpipe_headers_get(struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct devlink *devlink = info->user_ptr[0];
+
+ if (!devlink->dpipe_headers)
+ return -EOPNOTSUPP;
+ return devlink_dpipe_headers_fill(info, DEVLINK_CMD_DPIPE_HEADERS_GET,
+ 0, devlink->dpipe_headers);
+}
+
+static int devlink_dpipe_table_counters_set(struct devlink *devlink,
+ const char *table_name,
+ bool enable)
+{
+ struct devlink_dpipe_table *table;
+
+ table = devlink_dpipe_table_find(&devlink->dpipe_table_list,
+ table_name);
+ if (!table)
+ return -EINVAL;
+
+ if (table->counter_control_extern)
+ return -EOPNOTSUPP;
+
+ if (!(table->counters_enabled ^ enable))
+ return 0;
+
+ table->counters_enabled = enable;
+ if (table->table_ops->counters_set_update)
+ table->table_ops->counters_set_update(table->priv, enable);
+ return 0;
+}
+
+static int devlink_nl_cmd_dpipe_table_counters_set(struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct devlink *devlink = info->user_ptr[0];
+ const char *table_name;
+ bool counters_enable;
+
+ if (!info->attrs[DEVLINK_ATTR_DPIPE_TABLE_NAME] ||
+ !info->attrs[DEVLINK_ATTR_DPIPE_TABLE_COUNTERS_ENABLED])
+ return -EINVAL;
+
+ table_name = nla_data(info->attrs[DEVLINK_ATTR_DPIPE_TABLE_NAME]);
+ counters_enable = !!nla_get_u8(info->attrs[DEVLINK_ATTR_DPIPE_TABLE_COUNTERS_ENABLED]);
+
+ return devlink_dpipe_table_counters_set(devlink, table_name,
+ counters_enable);
+}
+
static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = {
[DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING },
[DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING },
@@ -1512,6 +2209,9 @@
[DEVLINK_ATTR_SB_TC_INDEX] = { .type = NLA_U16 },
[DEVLINK_ATTR_ESWITCH_MODE] = { .type = NLA_U16 },
[DEVLINK_ATTR_ESWITCH_INLINE_MODE] = { .type = NLA_U8 },
+ [DEVLINK_ATTR_ESWITCH_ENCAP_MODE] = { .type = NLA_U8 },
+ [DEVLINK_ATTR_DPIPE_TABLE_NAME] = { .type = NLA_NUL_STRING },
+ [DEVLINK_ATTR_DPIPE_TABLE_COUNTERS_ENABLED] = { .type = NLA_U8 },
};
static const struct genl_ops devlink_nl_ops[] = {
@@ -1644,6 +2344,34 @@
.flags = GENL_ADMIN_PERM,
.internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK,
},
+ {
+ .cmd = DEVLINK_CMD_DPIPE_TABLE_GET,
+ .doit = devlink_nl_cmd_dpipe_table_get,
+ .policy = devlink_nl_policy,
+ .flags = GENL_ADMIN_PERM,
+ .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK,
+ },
+ {
+ .cmd = DEVLINK_CMD_DPIPE_ENTRIES_GET,
+ .doit = devlink_nl_cmd_dpipe_entries_get,
+ .policy = devlink_nl_policy,
+ .flags = GENL_ADMIN_PERM,
+ .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK,
+ },
+ {
+ .cmd = DEVLINK_CMD_DPIPE_HEADERS_GET,
+ .doit = devlink_nl_cmd_dpipe_headers_get,
+ .policy = devlink_nl_policy,
+ .flags = GENL_ADMIN_PERM,
+ .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK,
+ },
+ {
+ .cmd = DEVLINK_CMD_DPIPE_TABLE_COUNTERS_SET,
+ .doit = devlink_nl_cmd_dpipe_table_counters_set,
+ .policy = devlink_nl_policy,
+ .flags = GENL_ADMIN_PERM,
+ .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK,
+ },
};
static struct genl_family devlink_nl_family __ro_after_init = {
@@ -1680,6 +2408,7 @@
devlink_net_set(devlink, &init_net);
INIT_LIST_HEAD(&devlink->port_list);
INIT_LIST_HEAD(&devlink->sb_list);
+ INIT_LIST_HEAD_RCU(&devlink->dpipe_table_list);
return devlink;
}
EXPORT_SYMBOL_GPL(devlink_alloc);
@@ -1880,6 +2609,133 @@
}
EXPORT_SYMBOL_GPL(devlink_sb_unregister);
+/**
+ * devlink_dpipe_headers_register - register dpipe headers
+ *
+ * @devlink: devlink
+ * @dpipe_headers: dpipe header array
+ *
+ * Register the headers supported by hardware.
+ */
+int devlink_dpipe_headers_register(struct devlink *devlink,
+ struct devlink_dpipe_headers *dpipe_headers)
+{
+ mutex_lock(&devlink_mutex);
+ devlink->dpipe_headers = dpipe_headers;
+ mutex_unlock(&devlink_mutex);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(devlink_dpipe_headers_register);
+
+/**
+ * devlink_dpipe_headers_unregister - unregister dpipe headers
+ *
+ * @devlink: devlink
+ *
+ * Unregister the headers supported by hardware.
+ */
+void devlink_dpipe_headers_unregister(struct devlink *devlink)
+{
+ mutex_lock(&devlink_mutex);
+ devlink->dpipe_headers = NULL;
+ mutex_unlock(&devlink_mutex);
+}
+EXPORT_SYMBOL_GPL(devlink_dpipe_headers_unregister);
+
+/**
+ * devlink_dpipe_table_counter_enabled - check if counter allocation
+ * required
+ * @devlink: devlink
+ * @table_name: tables name
+ *
+ * Used by driver to check if counter allocation is required.
+ * After counter allocation is turned on the table entries
+ * are updated to include counter statistics.
+ *
+ * After that point on the driver must respect the counter
+ * state so that each entry added to the table is added
+ * with a counter.
+ */
+bool devlink_dpipe_table_counter_enabled(struct devlink *devlink,
+ const char *table_name)
+{
+ struct devlink_dpipe_table *table;
+ bool enabled;
+
+ rcu_read_lock();
+ table = devlink_dpipe_table_find(&devlink->dpipe_table_list,
+ table_name);
+ enabled = false;
+ if (table)
+ enabled = table->counters_enabled;
+ rcu_read_unlock();
+ return enabled;
+}
+EXPORT_SYMBOL_GPL(devlink_dpipe_table_counter_enabled);
+
+/**
+ * devlink_dpipe_table_register - register dpipe table
+ *
+ * @devlink: devlink
+ * @table_name: table name
+ * @table_ops: table ops
+ * @priv: priv
+ * @size: size
+ * @counter_control_extern: external control for counters
+ */
+int devlink_dpipe_table_register(struct devlink *devlink,
+ const char *table_name,
+ struct devlink_dpipe_table_ops *table_ops,
+ void *priv, u64 size,
+ bool counter_control_extern)
+{
+ struct devlink_dpipe_table *table;
+
+ if (devlink_dpipe_table_find(&devlink->dpipe_table_list, table_name))
+ return -EEXIST;
+
+ table = kzalloc(sizeof(*table), GFP_KERNEL);
+ if (!table)
+ return -ENOMEM;
+
+ table->name = table_name;
+ table->table_ops = table_ops;
+ table->priv = priv;
+ table->size = size;
+ table->counter_control_extern = counter_control_extern;
+
+ mutex_lock(&devlink_mutex);
+ list_add_tail_rcu(&table->list, &devlink->dpipe_table_list);
+ mutex_unlock(&devlink_mutex);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(devlink_dpipe_table_register);
+
+/**
+ * devlink_dpipe_table_unregister - unregister dpipe table
+ *
+ * @devlink: devlink
+ * @table_name: table name
+ */
+void devlink_dpipe_table_unregister(struct devlink *devlink,
+ const char *table_name)
+{
+ struct devlink_dpipe_table *table;
+
+ mutex_lock(&devlink_mutex);
+ table = devlink_dpipe_table_find(&devlink->dpipe_table_list,
+ table_name);
+ if (!table)
+ goto unlock;
+ list_del_rcu(&table->list);
+ mutex_unlock(&devlink_mutex);
+ kfree_rcu(table, rcu);
+ return;
+unlock:
+ mutex_unlock(&devlink_mutex);
+}
+EXPORT_SYMBOL_GPL(devlink_dpipe_table_unregister);
+
static int __init devlink_module_init(void)
{
return genl_register_family(&devlink_nl_family);
diff --git a/net/core/drop_monitor.c b/net/core/drop_monitor.c
index fb55327..70ccda2 100644
--- a/net/core/drop_monitor.c
+++ b/net/core/drop_monitor.c
@@ -412,9 +412,8 @@
for_each_possible_cpu(cpu) {
data = &per_cpu(dm_cpu_data, cpu);
INIT_WORK(&data->dm_alert_work, send_dm_alert);
- init_timer(&data->send_timer);
- data->send_timer.data = (unsigned long)data;
- data->send_timer.function = sched_send_work;
+ setup_timer(&data->send_timer, sched_send_work,
+ (unsigned long)data);
spin_lock_init(&data->lock);
reset_per_cpu_data(data);
}
diff --git a/net/core/ethtool.c b/net/core/ethtool.c
index aecb2c7..03111a2 100644
--- a/net/core/ethtool.c
+++ b/net/core/ethtool.c
@@ -90,6 +90,7 @@
[NETIF_F_GSO_UDP_TUNNEL_CSUM_BIT] = "tx-udp_tnl-csum-segmentation",
[NETIF_F_GSO_PARTIAL_BIT] = "tx-gso-partial",
[NETIF_F_GSO_SCTP_BIT] = "tx-sctp-segmentation",
+ [NETIF_F_GSO_ESP_BIT] = "tx-esp-segmentation",
[NETIF_F_FCOE_CRC_BIT] = "tx-checksum-fcoe-crc",
[NETIF_F_SCTP_CRC_BIT] = "tx-checksum-sctp",
@@ -103,12 +104,15 @@
[NETIF_F_RXALL_BIT] = "rx-all",
[NETIF_F_HW_L2FW_DOFFLOAD_BIT] = "l2-fwd-offload",
[NETIF_F_HW_TC_BIT] = "hw-tc-offload",
+ [NETIF_F_HW_ESP_BIT] = "esp-hw-offload",
+ [NETIF_F_HW_ESP_TX_CSUM_BIT] = "esp-tx-csum-hw-offload",
};
static const char
rss_hash_func_strings[ETH_RSS_HASH_FUNCS_COUNT][ETH_GSTRING_LEN] = {
[ETH_RSS_HASH_TOP_BIT] = "toeplitz",
[ETH_RSS_HASH_XOR_BIT] = "xor",
+ [ETH_RSS_HASH_CRC32_BIT] = "crc32",
};
static const char
diff --git a/net/core/fib_rules.c b/net/core/fib_rules.c
index b6791d9..f21c4d3 100644
--- a/net/core/fib_rules.c
+++ b/net/core/fib_rules.c
@@ -23,6 +23,20 @@
KUIDT_INIT(~0),
};
+bool fib_rule_matchall(const struct fib_rule *rule)
+{
+ if (rule->iifindex || rule->oifindex || rule->mark || rule->tun_id ||
+ rule->flags)
+ return false;
+ if (rule->suppress_ifgroup != -1 || rule->suppress_prefixlen != -1)
+ return false;
+ if (!uid_eq(rule->uid_range.start, fib_kuid_range_unset.start) ||
+ !uid_eq(rule->uid_range.end, fib_kuid_range_unset.end))
+ return false;
+ return true;
+}
+EXPORT_SYMBOL_GPL(fib_rule_matchall);
+
int fib_default_rule_add(struct fib_rules_ops *ops,
u32 pref, u32 table, u32 flags)
{
@@ -354,7 +368,8 @@
return 0;
}
-int fib_nl_newrule(struct sk_buff *skb, struct nlmsghdr *nlh)
+int fib_nl_newrule(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct fib_rule_hdr *frh = nlmsg_data(nlh);
@@ -372,7 +387,7 @@
goto errout;
}
- err = nlmsg_parse(nlh, sizeof(*frh), tb, FRA_MAX, ops->policy);
+ err = nlmsg_parse(nlh, sizeof(*frh), tb, FRA_MAX, ops->policy, extack);
if (err < 0)
goto errout;
@@ -425,6 +440,7 @@
if (tb[FRA_TUN_ID])
rule->tun_id = nla_get_be64(tb[FRA_TUN_ID]);
+ err = -EINVAL;
if (tb[FRA_L3MDEV]) {
#ifdef CONFIG_NET_L3_MASTER_DEV
rule->l3mdev = nla_get_u8(tb[FRA_L3MDEV]);
@@ -446,7 +462,6 @@
else
rule->suppress_ifgroup = -1;
- err = -EINVAL;
if (tb[FRA_GOTO]) {
if (rule->action != FR_ACT_GOTO)
goto errout_free;
@@ -547,7 +562,8 @@
}
EXPORT_SYMBOL_GPL(fib_nl_newrule);
-int fib_nl_delrule(struct sk_buff *skb, struct nlmsghdr *nlh)
+int fib_nl_delrule(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct fib_rule_hdr *frh = nlmsg_data(nlh);
@@ -566,7 +582,7 @@
goto errout;
}
- err = nlmsg_parse(nlh, sizeof(*frh), tb, FRA_MAX, ops->policy);
+ err = nlmsg_parse(nlh, sizeof(*frh), tb, FRA_MAX, ops->policy, extack);
if (err < 0)
goto errout;
@@ -576,8 +592,10 @@
if (tb[FRA_UID_RANGE]) {
range = nla_get_kuid_range(tb);
- if (!uid_range_set(&range))
+ if (!uid_range_set(&range)) {
+ err = -EINVAL;
goto errout;
+ }
} else {
range = fib_kuid_range_unset;
}
diff --git a/net/core/filter.c b/net/core/filter.c
index ebaeaf2..a253a61 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -26,6 +26,7 @@
#include <linux/mm.h>
#include <linux/fcntl.h>
#include <linux/socket.h>
+#include <linux/sock_diag.h>
#include <linux/in.h>
#include <linux/inet.h>
#include <linux/netdevice.h>
@@ -52,6 +53,7 @@
#include <net/dst_metadata.h>
#include <net/dst.h>
#include <net/sock_reuseport.h>
+#include <net/busy_poll.h>
/**
* sk_filter_trim_cap - run a packet through a socket filter
@@ -91,7 +93,12 @@
rcu_read_lock();
filter = rcu_dereference(sk->sk_filter);
if (filter) {
- unsigned int pkt_len = bpf_prog_run_save_cb(filter->prog, skb);
+ struct sock *save_sk = skb->sk;
+ unsigned int pkt_len;
+
+ skb->sk = sk;
+ pkt_len = bpf_prog_run_save_cb(filter->prog, skb);
+ skb->sk = save_sk;
err = pkt_len ? pskb_trim(skb, max(cap, pkt_len)) : -EPERM;
}
rcu_read_unlock();
@@ -348,7 +355,8 @@
* @new_prog: buffer where converted program will be stored
* @new_len: pointer to store length of converted program
*
- * Remap 'sock_filter' style BPF instruction set to 'sock_filter_ext' style.
+ * Remap 'sock_filter' style classic BPF (cBPF) instruction set to 'bpf_insn'
+ * style extended BPF (eBPF).
* Conversion workflow:
*
* 1) First pass for calculating the new program length:
@@ -928,7 +936,7 @@
*/
static void sk_filter_release(struct sk_filter *fp)
{
- if (atomic_dec_and_test(&fp->refcnt))
+ if (refcount_dec_and_test(&fp->refcnt))
call_rcu(&fp->rcu, sk_filter_release_rcu);
}
@@ -943,20 +951,27 @@
/* try to charge the socket memory if there is space available
* return true on success
*/
-bool sk_filter_charge(struct sock *sk, struct sk_filter *fp)
+static bool __sk_filter_charge(struct sock *sk, struct sk_filter *fp)
{
u32 filter_size = bpf_prog_size(fp->prog->len);
/* same check as in sock_kmalloc() */
if (filter_size <= sysctl_optmem_max &&
atomic_read(&sk->sk_omem_alloc) + filter_size < sysctl_optmem_max) {
- atomic_inc(&fp->refcnt);
atomic_add(filter_size, &sk->sk_omem_alloc);
return true;
}
return false;
}
+bool sk_filter_charge(struct sock *sk, struct sk_filter *fp)
+{
+ bool ret = __sk_filter_charge(sk, fp);
+ if (ret)
+ refcount_inc(&fp->refcnt);
+ return ret;
+}
+
static struct bpf_prog *bpf_migrate_filter(struct bpf_prog *fp)
{
struct sock_filter *old_prog;
@@ -1179,12 +1194,12 @@
return -ENOMEM;
fp->prog = prog;
- atomic_set(&fp->refcnt, 0);
- if (!sk_filter_charge(sk, fp)) {
+ if (!__sk_filter_charge(sk, fp)) {
kfree(fp);
return -ENOMEM;
}
+ refcount_set(&fp->refcnt, 1);
old_fp = rcu_dereference_protected(sk->sk_filter,
lockdep_sock_is_held(sk));
@@ -2599,6 +2614,36 @@
.arg5_type = ARG_CONST_SIZE,
};
+BPF_CALL_1(bpf_get_socket_cookie, struct sk_buff *, skb)
+{
+ return skb->sk ? sock_gen_cookie(skb->sk) : 0;
+}
+
+static const struct bpf_func_proto bpf_get_socket_cookie_proto = {
+ .func = bpf_get_socket_cookie,
+ .gpl_only = false,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_PTR_TO_CTX,
+};
+
+BPF_CALL_1(bpf_get_socket_uid, struct sk_buff *, skb)
+{
+ struct sock *sk = sk_to_full_sk(skb->sk);
+ kuid_t kuid;
+
+ if (!sk || !sk_fullsock(sk))
+ return overflowuid;
+ kuid = sock_net_uid(sock_net(sk), sk);
+ return from_kuid_munged(sock_net(sk)->user_ns, kuid);
+}
+
+static const struct bpf_func_proto bpf_get_socket_uid_proto = {
+ .func = bpf_get_socket_uid,
+ .gpl_only = false,
+ .ret_type = RET_INTEGER,
+ .arg1_type = ARG_PTR_TO_CTX,
+};
+
static const struct bpf_func_proto *
bpf_base_func_proto(enum bpf_func_id func_id)
{
@@ -2633,6 +2678,10 @@
switch (func_id) {
case BPF_FUNC_skb_load_bytes:
return &bpf_skb_load_bytes_proto;
+ case BPF_FUNC_get_socket_cookie:
+ return &bpf_get_socket_cookie_proto;
+ case BPF_FUNC_get_socket_uid:
+ return &bpf_get_socket_uid_proto;
default:
return bpf_base_func_proto(func_id);
}
@@ -2692,6 +2741,10 @@
return &bpf_get_smp_processor_id_proto;
case BPF_FUNC_skb_under_cgroup:
return &bpf_skb_under_cgroup_proto;
+ case BPF_FUNC_get_socket_cookie:
+ return &bpf_get_socket_cookie_proto;
+ case BPF_FUNC_get_socket_uid:
+ return &bpf_get_socket_uid_proto;
default:
return bpf_base_func_proto(func_id);
}
@@ -2715,12 +2768,7 @@
static const struct bpf_func_proto *
cg_skb_func_proto(enum bpf_func_id func_id)
{
- switch (func_id) {
- case BPF_FUNC_skb_load_bytes:
- return &bpf_skb_load_bytes_proto;
- default:
- return bpf_base_func_proto(func_id);
- }
+ return sk_filter_func_proto(func_id);
}
static const struct bpf_func_proto *
@@ -3156,6 +3204,19 @@
*insn++ = BPF_MOV64_IMM(si->dst_reg, 0);
#endif
break;
+
+ case offsetof(struct __sk_buff, napi_id):
+#if defined(CONFIG_NET_RX_BUSY_POLL)
+ BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, napi_id) != 4);
+
+ *insn++ = BPF_LDX_MEM(BPF_W, si->dst_reg, si->src_reg,
+ offsetof(struct sk_buff, napi_id));
+ *insn++ = BPF_JMP_IMM(BPF_JGE, si->dst_reg, MIN_NAPI_ID, 1);
+ *insn++ = BPF_MOV64_IMM(si->dst_reg, 0);
+#else
+ *insn++ = BPF_MOV64_IMM(si->dst_reg, 0);
+#endif
+ break;
}
return insn - insn_buf;
@@ -3252,111 +3313,55 @@
return insn - insn_buf;
}
-static const struct bpf_verifier_ops sk_filter_ops = {
+const struct bpf_verifier_ops sk_filter_prog_ops = {
.get_func_proto = sk_filter_func_proto,
.is_valid_access = sk_filter_is_valid_access,
.convert_ctx_access = bpf_convert_ctx_access,
};
-static const struct bpf_verifier_ops tc_cls_act_ops = {
+const struct bpf_verifier_ops tc_cls_act_prog_ops = {
.get_func_proto = tc_cls_act_func_proto,
.is_valid_access = tc_cls_act_is_valid_access,
.convert_ctx_access = tc_cls_act_convert_ctx_access,
.gen_prologue = tc_cls_act_prologue,
+ .test_run = bpf_prog_test_run_skb,
};
-static const struct bpf_verifier_ops xdp_ops = {
+const struct bpf_verifier_ops xdp_prog_ops = {
.get_func_proto = xdp_func_proto,
.is_valid_access = xdp_is_valid_access,
.convert_ctx_access = xdp_convert_ctx_access,
+ .test_run = bpf_prog_test_run_xdp,
};
-static const struct bpf_verifier_ops cg_skb_ops = {
+const struct bpf_verifier_ops cg_skb_prog_ops = {
.get_func_proto = cg_skb_func_proto,
.is_valid_access = sk_filter_is_valid_access,
.convert_ctx_access = bpf_convert_ctx_access,
+ .test_run = bpf_prog_test_run_skb,
};
-static const struct bpf_verifier_ops lwt_inout_ops = {
+const struct bpf_verifier_ops lwt_inout_prog_ops = {
.get_func_proto = lwt_inout_func_proto,
.is_valid_access = lwt_is_valid_access,
.convert_ctx_access = bpf_convert_ctx_access,
+ .test_run = bpf_prog_test_run_skb,
};
-static const struct bpf_verifier_ops lwt_xmit_ops = {
+const struct bpf_verifier_ops lwt_xmit_prog_ops = {
.get_func_proto = lwt_xmit_func_proto,
.is_valid_access = lwt_is_valid_access,
.convert_ctx_access = bpf_convert_ctx_access,
.gen_prologue = tc_cls_act_prologue,
+ .test_run = bpf_prog_test_run_skb,
};
-static const struct bpf_verifier_ops cg_sock_ops = {
+const struct bpf_verifier_ops cg_sock_prog_ops = {
.get_func_proto = bpf_base_func_proto,
.is_valid_access = sock_filter_is_valid_access,
.convert_ctx_access = sock_filter_convert_ctx_access,
};
-static struct bpf_prog_type_list sk_filter_type __ro_after_init = {
- .ops = &sk_filter_ops,
- .type = BPF_PROG_TYPE_SOCKET_FILTER,
-};
-
-static struct bpf_prog_type_list sched_cls_type __ro_after_init = {
- .ops = &tc_cls_act_ops,
- .type = BPF_PROG_TYPE_SCHED_CLS,
-};
-
-static struct bpf_prog_type_list sched_act_type __ro_after_init = {
- .ops = &tc_cls_act_ops,
- .type = BPF_PROG_TYPE_SCHED_ACT,
-};
-
-static struct bpf_prog_type_list xdp_type __ro_after_init = {
- .ops = &xdp_ops,
- .type = BPF_PROG_TYPE_XDP,
-};
-
-static struct bpf_prog_type_list cg_skb_type __ro_after_init = {
- .ops = &cg_skb_ops,
- .type = BPF_PROG_TYPE_CGROUP_SKB,
-};
-
-static struct bpf_prog_type_list lwt_in_type __ro_after_init = {
- .ops = &lwt_inout_ops,
- .type = BPF_PROG_TYPE_LWT_IN,
-};
-
-static struct bpf_prog_type_list lwt_out_type __ro_after_init = {
- .ops = &lwt_inout_ops,
- .type = BPF_PROG_TYPE_LWT_OUT,
-};
-
-static struct bpf_prog_type_list lwt_xmit_type __ro_after_init = {
- .ops = &lwt_xmit_ops,
- .type = BPF_PROG_TYPE_LWT_XMIT,
-};
-
-static struct bpf_prog_type_list cg_sock_type __ro_after_init = {
- .ops = &cg_sock_ops,
- .type = BPF_PROG_TYPE_CGROUP_SOCK
-};
-
-static int __init register_sk_filter_ops(void)
-{
- bpf_register_prog_type(&sk_filter_type);
- bpf_register_prog_type(&sched_cls_type);
- bpf_register_prog_type(&sched_act_type);
- bpf_register_prog_type(&xdp_type);
- bpf_register_prog_type(&cg_skb_type);
- bpf_register_prog_type(&cg_sock_type);
- bpf_register_prog_type(&lwt_in_type);
- bpf_register_prog_type(&lwt_out_type);
- bpf_register_prog_type(&lwt_xmit_type);
-
- return 0;
-}
-late_initcall(register_sk_filter_ops);
-
int sk_detach_filter(struct sock *sk)
{
int ret = -ENOENT;
diff --git a/net/core/flow.c b/net/core/flow.c
index f765c11..f7f5d19 100644
--- a/net/core/flow.c
+++ b/net/core/flow.c
@@ -47,7 +47,7 @@
static struct kmem_cache *flow_cachep __read_mostly;
-#define flow_cache_hash_size(cache) (1 << (cache)->hash_shift)
+#define flow_cache_hash_size(cache) (1U << (cache)->hash_shift)
#define FLOW_HASH_RND_PERIOD (10 * 60 * HZ)
static void flow_cache_new_hashrnd(unsigned long arg)
@@ -99,7 +99,8 @@
}
static void flow_cache_queue_garbage(struct flow_cache_percpu *fcp,
- int deleted, struct list_head *gc_list,
+ unsigned int deleted,
+ struct list_head *gc_list,
struct netns_xfrm *xfrm)
{
if (deleted) {
@@ -114,17 +115,18 @@
static void __flow_cache_shrink(struct flow_cache *fc,
struct flow_cache_percpu *fcp,
- int shrink_to)
+ unsigned int shrink_to)
{
struct flow_cache_entry *fle;
struct hlist_node *tmp;
LIST_HEAD(gc_list);
- int i, deleted = 0;
+ unsigned int deleted = 0;
struct netns_xfrm *xfrm = container_of(fc, struct netns_xfrm,
flow_cache_global);
+ unsigned int i;
for (i = 0; i < flow_cache_hash_size(fc); i++) {
- int saved = 0;
+ unsigned int saved = 0;
hlist_for_each_entry_safe(fle, tmp,
&fcp->hash_table[i], u.hlist) {
@@ -145,7 +147,7 @@
static void flow_cache_shrink(struct flow_cache *fc,
struct flow_cache_percpu *fcp)
{
- int shrink_to = fc->low_watermark / flow_cache_hash_size(fc);
+ unsigned int shrink_to = fc->low_watermark / flow_cache_hash_size(fc);
__flow_cache_shrink(fc, fcp, shrink_to);
}
@@ -161,7 +163,7 @@
static u32 flow_hash_code(struct flow_cache *fc,
struct flow_cache_percpu *fcp,
const struct flowi *key,
- size_t keysize)
+ unsigned int keysize)
{
const u32 *k = (const u32 *) key;
const u32 length = keysize * sizeof(flow_compare_t) / sizeof(u32);
@@ -174,7 +176,7 @@
* important assumptions that we can here, such as alignment.
*/
static int flow_key_compare(const struct flowi *key1, const struct flowi *key2,
- size_t keysize)
+ unsigned int keysize)
{
const flow_compare_t *k1, *k1_lim, *k2;
@@ -199,7 +201,7 @@
struct flow_cache_percpu *fcp;
struct flow_cache_entry *fle, *tfle;
struct flow_cache_object *flo;
- size_t keysize;
+ unsigned int keysize;
unsigned int hash;
local_bh_disable();
@@ -295,9 +297,10 @@
struct flow_cache_entry *fle;
struct hlist_node *tmp;
LIST_HEAD(gc_list);
- int i, deleted = 0;
+ unsigned int deleted = 0;
struct netns_xfrm *xfrm = container_of(fc, struct netns_xfrm,
flow_cache_global);
+ unsigned int i;
fcp = this_cpu_ptr(fc->percpu);
for (i = 0; i < flow_cache_hash_size(fc); i++) {
@@ -327,7 +330,7 @@
static int flow_cache_percpu_empty(struct flow_cache *fc, int cpu)
{
struct flow_cache_percpu *fcp;
- int i;
+ unsigned int i;
fcp = per_cpu_ptr(fc->percpu, cpu);
for (i = 0; i < flow_cache_hash_size(fc); i++)
@@ -402,12 +405,12 @@
static int flow_cache_cpu_prepare(struct flow_cache *fc, int cpu)
{
struct flow_cache_percpu *fcp = per_cpu_ptr(fc->percpu, cpu);
- size_t sz = sizeof(struct hlist_head) * flow_cache_hash_size(fc);
+ unsigned int sz = sizeof(struct hlist_head) * flow_cache_hash_size(fc);
if (!fcp->hash_table) {
fcp->hash_table = kzalloc_node(sz, GFP_KERNEL, cpu_to_node(cpu));
if (!fcp->hash_table) {
- pr_err("NET: failed to allocate flow cache sz %zu\n", sz);
+ pr_err("NET: failed to allocate flow cache sz %u\n", sz);
return -ENOMEM;
}
fcp->hash_rnd_recalc = 1;
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index d98d499..28d94bc 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -113,6 +113,235 @@
}
EXPORT_SYMBOL(__skb_flow_get_ports);
+enum flow_dissect_ret {
+ FLOW_DISSECT_RET_OUT_GOOD,
+ FLOW_DISSECT_RET_OUT_BAD,
+ FLOW_DISSECT_RET_OUT_PROTO_AGAIN,
+};
+
+static enum flow_dissect_ret
+__skb_flow_dissect_mpls(const struct sk_buff *skb,
+ struct flow_dissector *flow_dissector,
+ void *target_container, void *data, int nhoff, int hlen)
+{
+ struct flow_dissector_key_keyid *key_keyid;
+ struct mpls_label *hdr, _hdr[2];
+ u32 entry, label;
+
+ if (!dissector_uses_key(flow_dissector,
+ FLOW_DISSECTOR_KEY_MPLS_ENTROPY) &&
+ !dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_MPLS))
+ return FLOW_DISSECT_RET_OUT_GOOD;
+
+ hdr = __skb_header_pointer(skb, nhoff, sizeof(_hdr), data,
+ hlen, &_hdr);
+ if (!hdr)
+ return FLOW_DISSECT_RET_OUT_BAD;
+
+ entry = ntohl(hdr[0].entry);
+ label = (entry & MPLS_LS_LABEL_MASK) >> MPLS_LS_LABEL_SHIFT;
+
+ if (dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_MPLS)) {
+ struct flow_dissector_key_mpls *key_mpls;
+
+ key_mpls = skb_flow_dissector_target(flow_dissector,
+ FLOW_DISSECTOR_KEY_MPLS,
+ target_container);
+ key_mpls->mpls_label = label;
+ key_mpls->mpls_ttl = (entry & MPLS_LS_TTL_MASK)
+ >> MPLS_LS_TTL_SHIFT;
+ key_mpls->mpls_tc = (entry & MPLS_LS_TC_MASK)
+ >> MPLS_LS_TC_SHIFT;
+ key_mpls->mpls_bos = (entry & MPLS_LS_S_MASK)
+ >> MPLS_LS_S_SHIFT;
+ }
+
+ if (label == MPLS_LABEL_ENTROPY) {
+ key_keyid = skb_flow_dissector_target(flow_dissector,
+ FLOW_DISSECTOR_KEY_MPLS_ENTROPY,
+ target_container);
+ key_keyid->keyid = hdr[1].entry & htonl(MPLS_LS_LABEL_MASK);
+ }
+ return FLOW_DISSECT_RET_OUT_GOOD;
+}
+
+static enum flow_dissect_ret
+__skb_flow_dissect_arp(const struct sk_buff *skb,
+ struct flow_dissector *flow_dissector,
+ void *target_container, void *data, int nhoff, int hlen)
+{
+ struct flow_dissector_key_arp *key_arp;
+ struct {
+ unsigned char ar_sha[ETH_ALEN];
+ unsigned char ar_sip[4];
+ unsigned char ar_tha[ETH_ALEN];
+ unsigned char ar_tip[4];
+ } *arp_eth, _arp_eth;
+ const struct arphdr *arp;
+ struct arphdr _arp;
+
+ if (!dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_ARP))
+ return FLOW_DISSECT_RET_OUT_GOOD;
+
+ arp = __skb_header_pointer(skb, nhoff, sizeof(_arp), data,
+ hlen, &_arp);
+ if (!arp)
+ return FLOW_DISSECT_RET_OUT_BAD;
+
+ if (arp->ar_hrd != htons(ARPHRD_ETHER) ||
+ arp->ar_pro != htons(ETH_P_IP) ||
+ arp->ar_hln != ETH_ALEN ||
+ arp->ar_pln != 4 ||
+ (arp->ar_op != htons(ARPOP_REPLY) &&
+ arp->ar_op != htons(ARPOP_REQUEST)))
+ return FLOW_DISSECT_RET_OUT_BAD;
+
+ arp_eth = __skb_header_pointer(skb, nhoff + sizeof(_arp),
+ sizeof(_arp_eth), data,
+ hlen, &_arp_eth);
+ if (!arp_eth)
+ return FLOW_DISSECT_RET_OUT_BAD;
+
+ key_arp = skb_flow_dissector_target(flow_dissector,
+ FLOW_DISSECTOR_KEY_ARP,
+ target_container);
+
+ memcpy(&key_arp->sip, arp_eth->ar_sip, sizeof(key_arp->sip));
+ memcpy(&key_arp->tip, arp_eth->ar_tip, sizeof(key_arp->tip));
+
+ /* Only store the lower byte of the opcode;
+ * this covers ARPOP_REPLY and ARPOP_REQUEST.
+ */
+ key_arp->op = ntohs(arp->ar_op) & 0xff;
+
+ ether_addr_copy(key_arp->sha, arp_eth->ar_sha);
+ ether_addr_copy(key_arp->tha, arp_eth->ar_tha);
+
+ return FLOW_DISSECT_RET_OUT_GOOD;
+}
+
+static enum flow_dissect_ret
+__skb_flow_dissect_gre(const struct sk_buff *skb,
+ struct flow_dissector_key_control *key_control,
+ struct flow_dissector *flow_dissector,
+ void *target_container, void *data,
+ __be16 *p_proto, int *p_nhoff, int *p_hlen,
+ unsigned int flags)
+{
+ struct flow_dissector_key_keyid *key_keyid;
+ struct gre_base_hdr *hdr, _hdr;
+ int offset = 0;
+ u16 gre_ver;
+
+ hdr = __skb_header_pointer(skb, *p_nhoff, sizeof(_hdr),
+ data, *p_hlen, &_hdr);
+ if (!hdr)
+ return FLOW_DISSECT_RET_OUT_BAD;
+
+ /* Only look inside GRE without routing */
+ if (hdr->flags & GRE_ROUTING)
+ return FLOW_DISSECT_RET_OUT_GOOD;
+
+ /* Only look inside GRE for version 0 and 1 */
+ gre_ver = ntohs(hdr->flags & GRE_VERSION);
+ if (gre_ver > 1)
+ return FLOW_DISSECT_RET_OUT_GOOD;
+
+ *p_proto = hdr->protocol;
+ if (gre_ver) {
+ /* Version1 must be PPTP, and check the flags */
+ if (!(*p_proto == GRE_PROTO_PPP && (hdr->flags & GRE_KEY)))
+ return FLOW_DISSECT_RET_OUT_GOOD;
+ }
+
+ offset += sizeof(struct gre_base_hdr);
+
+ if (hdr->flags & GRE_CSUM)
+ offset += sizeof(((struct gre_full_hdr *) 0)->csum) +
+ sizeof(((struct gre_full_hdr *) 0)->reserved1);
+
+ if (hdr->flags & GRE_KEY) {
+ const __be32 *keyid;
+ __be32 _keyid;
+
+ keyid = __skb_header_pointer(skb, *p_nhoff + offset,
+ sizeof(_keyid),
+ data, *p_hlen, &_keyid);
+ if (!keyid)
+ return FLOW_DISSECT_RET_OUT_BAD;
+
+ if (dissector_uses_key(flow_dissector,
+ FLOW_DISSECTOR_KEY_GRE_KEYID)) {
+ key_keyid = skb_flow_dissector_target(flow_dissector,
+ FLOW_DISSECTOR_KEY_GRE_KEYID,
+ target_container);
+ if (gre_ver == 0)
+ key_keyid->keyid = *keyid;
+ else
+ key_keyid->keyid = *keyid & GRE_PPTP_KEY_MASK;
+ }
+ offset += sizeof(((struct gre_full_hdr *) 0)->key);
+ }
+
+ if (hdr->flags & GRE_SEQ)
+ offset += sizeof(((struct pptp_gre_header *) 0)->seq);
+
+ if (gre_ver == 0) {
+ if (*p_proto == htons(ETH_P_TEB)) {
+ const struct ethhdr *eth;
+ struct ethhdr _eth;
+
+ eth = __skb_header_pointer(skb, *p_nhoff + offset,
+ sizeof(_eth),
+ data, *p_hlen, &_eth);
+ if (!eth)
+ return FLOW_DISSECT_RET_OUT_BAD;
+ *p_proto = eth->h_proto;
+ offset += sizeof(*eth);
+
+ /* Cap headers that we access via pointers at the
+ * end of the Ethernet header as our maximum alignment
+ * at that point is only 2 bytes.
+ */
+ if (NET_IP_ALIGN)
+ *p_hlen = *p_nhoff + offset;
+ }
+ } else { /* version 1, must be PPTP */
+ u8 _ppp_hdr[PPP_HDRLEN];
+ u8 *ppp_hdr;
+
+ if (hdr->flags & GRE_ACK)
+ offset += sizeof(((struct pptp_gre_header *) 0)->ack);
+
+ ppp_hdr = __skb_header_pointer(skb, *p_nhoff + offset,
+ sizeof(_ppp_hdr),
+ data, *p_hlen, _ppp_hdr);
+ if (!ppp_hdr)
+ return FLOW_DISSECT_RET_OUT_BAD;
+
+ switch (PPP_PROTOCOL(ppp_hdr)) {
+ case PPP_IP:
+ *p_proto = htons(ETH_P_IP);
+ break;
+ case PPP_IPV6:
+ *p_proto = htons(ETH_P_IPV6);
+ break;
+ default:
+ /* Could probably catch some more like MPLS */
+ break;
+ }
+
+ offset += PPP_HDRLEN;
+ }
+
+ *p_nhoff += offset;
+ key_control->flags |= FLOW_DIS_ENCAPSULATION;
+ if (flags & FLOW_DISSECTOR_F_STOP_AT_ENCAP)
+ return FLOW_DISSECT_RET_OUT_GOOD;
+
+ return FLOW_DISSECT_RET_OUT_PROTO_AGAIN;
+}
+
/**
* __skb_flow_dissect - extract the flow_keys struct and return it
* @skb: sk_buff to extract the flow from, can be NULL if the rest are specified
@@ -138,12 +367,10 @@
struct flow_dissector_key_control *key_control;
struct flow_dissector_key_basic *key_basic;
struct flow_dissector_key_addrs *key_addrs;
- struct flow_dissector_key_arp *key_arp;
struct flow_dissector_key_ports *key_ports;
struct flow_dissector_key_icmp *key_icmp;
struct flow_dissector_key_tags *key_tags;
struct flow_dissector_key_vlan *key_vlan;
- struct flow_dissector_key_keyid *key_keyid;
bool skip_vlan = false;
u8 ip_proto = 0;
bool ret;
@@ -181,7 +408,7 @@
memcpy(key_eth_addrs, ð->h_dest, sizeof(*key_eth_addrs));
}
-again:
+proto_again:
switch (proto) {
case htons(ETH_P_IP): {
const struct iphdr *iph;
@@ -284,7 +511,7 @@
proto = vlan->h_vlan_encapsulated_proto;
nhoff += sizeof(*vlan);
if (skip_vlan)
- goto again;
+ goto proto_again;
}
skip_vlan = true;
@@ -307,7 +534,7 @@
}
}
- goto again;
+ goto proto_again;
}
case htons(ETH_P_PPP_SES): {
struct {
@@ -349,31 +576,17 @@
}
case htons(ETH_P_MPLS_UC):
- case htons(ETH_P_MPLS_MC): {
- struct mpls_label *hdr, _hdr[2];
+ case htons(ETH_P_MPLS_MC):
mpls:
- hdr = __skb_header_pointer(skb, nhoff, sizeof(_hdr), data,
- hlen, &_hdr);
- if (!hdr)
- goto out_bad;
-
- if ((ntohl(hdr[0].entry) & MPLS_LS_LABEL_MASK) >>
- MPLS_LS_LABEL_SHIFT == MPLS_LABEL_ENTROPY) {
- if (dissector_uses_key(flow_dissector,
- FLOW_DISSECTOR_KEY_MPLS_ENTROPY)) {
- key_keyid = skb_flow_dissector_target(flow_dissector,
- FLOW_DISSECTOR_KEY_MPLS_ENTROPY,
- target_container);
- key_keyid->keyid = hdr[1].entry &
- htonl(MPLS_LS_LABEL_MASK);
- }
-
+ switch (__skb_flow_dissect_mpls(skb, flow_dissector,
+ target_container, data,
+ nhoff, hlen)) {
+ case FLOW_DISSECT_RET_OUT_GOOD:
goto out_good;
+ case FLOW_DISSECT_RET_OUT_BAD:
+ default:
+ goto out_bad;
}
-
- goto out_good;
- }
-
case htons(ETH_P_FCOE):
if ((hlen - nhoff) < FCOE_HEADER_LEN)
goto out_bad;
@@ -382,177 +595,33 @@
goto out_good;
case htons(ETH_P_ARP):
- case htons(ETH_P_RARP): {
- struct {
- unsigned char ar_sha[ETH_ALEN];
- unsigned char ar_sip[4];
- unsigned char ar_tha[ETH_ALEN];
- unsigned char ar_tip[4];
- } *arp_eth, _arp_eth;
- const struct arphdr *arp;
- struct arphdr _arp;
-
- arp = __skb_header_pointer(skb, nhoff, sizeof(_arp), data,
- hlen, &_arp);
- if (!arp)
+ case htons(ETH_P_RARP):
+ switch (__skb_flow_dissect_arp(skb, flow_dissector,
+ target_container, data,
+ nhoff, hlen)) {
+ case FLOW_DISSECT_RET_OUT_GOOD:
+ goto out_good;
+ case FLOW_DISSECT_RET_OUT_BAD:
+ default:
goto out_bad;
-
- if (arp->ar_hrd != htons(ARPHRD_ETHER) ||
- arp->ar_pro != htons(ETH_P_IP) ||
- arp->ar_hln != ETH_ALEN ||
- arp->ar_pln != 4 ||
- (arp->ar_op != htons(ARPOP_REPLY) &&
- arp->ar_op != htons(ARPOP_REQUEST)))
- goto out_bad;
-
- arp_eth = __skb_header_pointer(skb, nhoff + sizeof(_arp),
- sizeof(_arp_eth), data,
- hlen,
- &_arp_eth);
- if (!arp_eth)
- goto out_bad;
-
- if (dissector_uses_key(flow_dissector,
- FLOW_DISSECTOR_KEY_ARP)) {
-
- key_arp = skb_flow_dissector_target(flow_dissector,
- FLOW_DISSECTOR_KEY_ARP,
- target_container);
-
- memcpy(&key_arp->sip, arp_eth->ar_sip,
- sizeof(key_arp->sip));
- memcpy(&key_arp->tip, arp_eth->ar_tip,
- sizeof(key_arp->tip));
-
- /* Only store the lower byte of the opcode;
- * this covers ARPOP_REPLY and ARPOP_REQUEST.
- */
- key_arp->op = ntohs(arp->ar_op) & 0xff;
-
- ether_addr_copy(key_arp->sha, arp_eth->ar_sha);
- ether_addr_copy(key_arp->tha, arp_eth->ar_tha);
}
-
- goto out_good;
- }
-
default:
goto out_bad;
}
ip_proto_again:
switch (ip_proto) {
- case IPPROTO_GRE: {
- struct gre_base_hdr *hdr, _hdr;
- u16 gre_ver;
- int offset = 0;
-
- hdr = __skb_header_pointer(skb, nhoff, sizeof(_hdr), data, hlen, &_hdr);
- if (!hdr)
- goto out_bad;
-
- /* Only look inside GRE without routing */
- if (hdr->flags & GRE_ROUTING)
- break;
-
- /* Only look inside GRE for version 0 and 1 */
- gre_ver = ntohs(hdr->flags & GRE_VERSION);
- if (gre_ver > 1)
- break;
-
- proto = hdr->protocol;
- if (gre_ver) {
- /* Version1 must be PPTP, and check the flags */
- if (!(proto == GRE_PROTO_PPP && (hdr->flags & GRE_KEY)))
- break;
- }
-
- offset += sizeof(struct gre_base_hdr);
-
- if (hdr->flags & GRE_CSUM)
- offset += sizeof(((struct gre_full_hdr *)0)->csum) +
- sizeof(((struct gre_full_hdr *)0)->reserved1);
-
- if (hdr->flags & GRE_KEY) {
- const __be32 *keyid;
- __be32 _keyid;
-
- keyid = __skb_header_pointer(skb, nhoff + offset, sizeof(_keyid),
- data, hlen, &_keyid);
- if (!keyid)
- goto out_bad;
-
- if (dissector_uses_key(flow_dissector,
- FLOW_DISSECTOR_KEY_GRE_KEYID)) {
- key_keyid = skb_flow_dissector_target(flow_dissector,
- FLOW_DISSECTOR_KEY_GRE_KEYID,
- target_container);
- if (gre_ver == 0)
- key_keyid->keyid = *keyid;
- else
- key_keyid->keyid = *keyid & GRE_PPTP_KEY_MASK;
- }
- offset += sizeof(((struct gre_full_hdr *)0)->key);
- }
-
- if (hdr->flags & GRE_SEQ)
- offset += sizeof(((struct pptp_gre_header *)0)->seq);
-
- if (gre_ver == 0) {
- if (proto == htons(ETH_P_TEB)) {
- const struct ethhdr *eth;
- struct ethhdr _eth;
-
- eth = __skb_header_pointer(skb, nhoff + offset,
- sizeof(_eth),
- data, hlen, &_eth);
- if (!eth)
- goto out_bad;
- proto = eth->h_proto;
- offset += sizeof(*eth);
-
- /* Cap headers that we access via pointers at the
- * end of the Ethernet header as our maximum alignment
- * at that point is only 2 bytes.
- */
- if (NET_IP_ALIGN)
- hlen = (nhoff + offset);
- }
- } else { /* version 1, must be PPTP */
- u8 _ppp_hdr[PPP_HDRLEN];
- u8 *ppp_hdr;
-
- if (hdr->flags & GRE_ACK)
- offset += sizeof(((struct pptp_gre_header *)0)->ack);
-
- ppp_hdr = __skb_header_pointer(skb, nhoff + offset,
- sizeof(_ppp_hdr),
- data, hlen, _ppp_hdr);
- if (!ppp_hdr)
- goto out_bad;
-
- switch (PPP_PROTOCOL(ppp_hdr)) {
- case PPP_IP:
- proto = htons(ETH_P_IP);
- break;
- case PPP_IPV6:
- proto = htons(ETH_P_IPV6);
- break;
- default:
- /* Could probably catch some more like MPLS */
- break;
- }
-
- offset += PPP_HDRLEN;
- }
-
- nhoff += offset;
- key_control->flags |= FLOW_DIS_ENCAPSULATION;
- if (flags & FLOW_DISSECTOR_F_STOP_AT_ENCAP)
+ case IPPROTO_GRE:
+ switch (__skb_flow_dissect_gre(skb, key_control, flow_dissector,
+ target_container, data,
+ &proto, &nhoff, &hlen, flags)) {
+ case FLOW_DISSECT_RET_OUT_GOOD:
goto out_good;
-
- goto again;
- }
+ case FLOW_DISSECT_RET_OUT_BAD:
+ goto out_bad;
+ case FLOW_DISSECT_RET_OUT_PROTO_AGAIN:
+ goto proto_again;
+ }
case NEXTHDR_HOP:
case NEXTHDR_ROUTING:
case NEXTHDR_DEST: {
diff --git a/net/core/gro_cells.c b/net/core/gro_cells.c
index c98bbfb..814e58a 100644
--- a/net/core/gro_cells.c
+++ b/net/core/gro_cells.c
@@ -13,7 +13,7 @@
struct net_device *dev = skb->dev;
struct gro_cell *cell;
- if (!gcells->cells || skb_cloned(skb) || !(dev->features & NETIF_F_GRO))
+ if (!gcells->cells || skb_cloned(skb) || netif_elide_gro(dev))
return netif_rx(skb);
cell = this_cpu_ptr(gcells->cells);
diff --git a/net/core/lwt_bpf.c b/net/core/lwt_bpf.c
index 0cfe7b0..b3bc0a3 100644
--- a/net/core/lwt_bpf.c
+++ b/net/core/lwt_bpf.c
@@ -209,7 +209,8 @@
int ret;
u32 fd;
- ret = nla_parse_nested(tb, LWT_BPF_PROG_MAX, attr, bpf_prog_policy);
+ ret = nla_parse_nested(tb, LWT_BPF_PROG_MAX, attr, bpf_prog_policy,
+ NULL);
if (ret < 0)
return ret;
@@ -249,7 +250,7 @@
if (family != AF_INET && family != AF_INET6)
return -EAFNOSUPPORT;
- ret = nla_parse_nested(tb, LWT_BPF_MAX, nla, bpf_nl_policy);
+ ret = nla_parse_nested(tb, LWT_BPF_MAX, nla, bpf_nl_policy, NULL);
if (ret < 0)
return ret;
diff --git a/net/core/lwtunnel.c b/net/core/lwtunnel.c
index 6df9f8f..cfae3d5 100644
--- a/net/core/lwtunnel.c
+++ b/net/core/lwtunnel.c
@@ -162,7 +162,6 @@
struct rtnexthop *rtnh = (struct rtnexthop *)attr;
struct nlattr *nla_entype;
struct nlattr *attrs;
- struct nlattr *nla;
u16 encap_type;
int attrlen;
@@ -170,7 +169,6 @@
attrlen = rtnh_attrlen(rtnh);
if (attrlen > 0) {
attrs = rtnh_attrs(rtnh);
- nla = nla_find(attrs, attrlen, RTA_ENCAP);
nla_entype = nla_find(attrs, attrlen, RTA_ENCAP_TYPE);
if (nla_entype) {
@@ -205,7 +203,7 @@
{
const struct lwtunnel_encap_ops *ops;
struct nlattr *nest;
- int ret = -EINVAL;
+ int ret;
if (!lwtstate)
return 0;
@@ -214,8 +212,11 @@
lwtstate->type > LWTUNNEL_ENCAP_MAX)
return 0;
- ret = -EOPNOTSUPP;
nest = nla_nest_start(skb, RTA_ENCAP);
+ if (!nest)
+ return -EMSGSIZE;
+
+ ret = -EOPNOTSUPP;
rcu_read_lock();
ops = rcu_dereference(lwtun_encaps[lwtstate->type]);
if (likely(ops && ops->fill_encap))
diff --git a/net/core/neighbour.c b/net/core/neighbour.c
index 4526cbd..58b0bcc 100644
--- a/net/core/neighbour.c
+++ b/net/core/neighbour.c
@@ -52,8 +52,9 @@
#define PNEIGH_HASHMASK 0xF
static void neigh_timer_handler(unsigned long arg);
-static void __neigh_notify(struct neighbour *n, int type, int flags);
-static void neigh_update_notify(struct neighbour *neigh);
+static void __neigh_notify(struct neighbour *n, int type, int flags,
+ u32 pid);
+static void neigh_update_notify(struct neighbour *neigh, u32 nlmsg_pid);
static int pneigh_ifdown(struct neigh_table *tbl, struct net_device *dev);
#ifdef CONFIG_PROC_FS
@@ -99,7 +100,7 @@
if (neigh->parms->neigh_cleanup)
neigh->parms->neigh_cleanup(neigh);
- __neigh_notify(neigh, RTM_DELNEIGH, 0);
+ __neigh_notify(neigh, RTM_DELNEIGH, 0, 0);
call_netevent_notifiers(NETEVENT_NEIGH_UPDATE, neigh);
neigh_release(neigh);
}
@@ -949,7 +950,7 @@
}
if (notify)
- neigh_update_notify(neigh);
+ neigh_update_notify(neigh, 0);
neigh_release(neigh);
}
@@ -1073,7 +1074,7 @@
*/
int neigh_update(struct neighbour *neigh, const u8 *lladdr, u8 new,
- u32 flags)
+ u32 flags, u32 nlmsg_pid)
{
u8 old;
int err;
@@ -1230,7 +1231,7 @@
write_unlock_bh(&neigh->lock);
if (notify)
- neigh_update_notify(neigh);
+ neigh_update_notify(neigh, nlmsg_pid);
return err;
}
@@ -1261,7 +1262,7 @@
lladdr || !dev->addr_len);
if (neigh)
neigh_update(neigh, lladdr, NUD_STALE,
- NEIGH_UPDATE_F_OVERRIDE);
+ NEIGH_UPDATE_F_OVERRIDE, 0);
return neigh;
}
EXPORT_SYMBOL(neigh_event_ns);
@@ -1589,7 +1590,8 @@
return tbl;
}
-static int neigh_delete(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int neigh_delete(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct ndmsg *ndm;
@@ -1639,14 +1641,16 @@
err = neigh_update(neigh, NULL, NUD_FAILED,
NEIGH_UPDATE_F_OVERRIDE |
- NEIGH_UPDATE_F_ADMIN);
+ NEIGH_UPDATE_F_ADMIN,
+ NETLINK_CB(skb).portid);
neigh_release(neigh);
out:
return err;
}
-static int neigh_add(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int neigh_add(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
int flags = NEIGH_UPDATE_F_ADMIN | NEIGH_UPDATE_F_OVERRIDE;
struct net *net = sock_net(skb->sk);
@@ -1659,7 +1663,7 @@
int err;
ASSERT_RTNL();
- err = nlmsg_parse(nlh, sizeof(*ndm), tb, NDA_MAX, NULL);
+ err = nlmsg_parse(nlh, sizeof(*ndm), tb, NDA_MAX, NULL, extack);
if (err < 0)
goto out;
@@ -1730,7 +1734,8 @@
neigh_event_send(neigh, NULL);
err = 0;
} else
- err = neigh_update(neigh, lladdr, ndm->ndm_state, flags);
+ err = neigh_update(neigh, lladdr, ndm->ndm_state, flags,
+ NETLINK_CB(skb).portid);
neigh_release(neigh);
out:
@@ -1933,7 +1938,8 @@
[NDTPA_LOCKTIME] = { .type = NLA_U64 },
};
-static int neightbl_set(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int neightbl_set(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct neigh_table *tbl;
@@ -1943,7 +1949,7 @@
int err, tidx;
err = nlmsg_parse(nlh, sizeof(*ndtmsg), tb, NDTA_MAX,
- nl_neightbl_policy);
+ nl_neightbl_policy, extack);
if (err < 0)
goto errout;
@@ -1981,7 +1987,7 @@
int i, ifindex = 0;
err = nla_parse_nested(tbp, NDTPA_MAX, tb[NDTA_PARMS],
- nl_ntbl_parm_policy);
+ nl_ntbl_parm_policy, extack);
if (err < 0)
goto errout_tbl_lock;
@@ -2230,10 +2236,10 @@
return -EMSGSIZE;
}
-static void neigh_update_notify(struct neighbour *neigh)
+static void neigh_update_notify(struct neighbour *neigh, u32 nlmsg_pid)
{
call_netevent_notifiers(NETEVENT_NEIGH_UPDATE, neigh);
- __neigh_notify(neigh, RTM_NEWNEIGH, 0);
+ __neigh_notify(neigh, RTM_NEWNEIGH, 0, nlmsg_pid);
}
static bool neigh_master_filtered(struct net_device *dev, int master_idx)
@@ -2272,7 +2278,7 @@
unsigned int flags = NLM_F_MULTI;
int err;
- err = nlmsg_parse(nlh, sizeof(struct ndmsg), tb, NDA_MAX, NULL);
+ err = nlmsg_parse(nlh, sizeof(struct ndmsg), tb, NDA_MAX, NULL, NULL);
if (!err) {
if (tb[NDA_IFINDEX])
filter_idx = nla_get_u32(tb[NDA_IFINDEX]);
@@ -2831,7 +2837,8 @@
+ nla_total_size(4); /* NDA_PROBES */
}
-static void __neigh_notify(struct neighbour *n, int type, int flags)
+static void __neigh_notify(struct neighbour *n, int type, int flags,
+ u32 pid)
{
struct net *net = dev_net(n->dev);
struct sk_buff *skb;
@@ -2841,7 +2848,7 @@
if (skb == NULL)
goto errout;
- err = neigh_fill_info(skb, n, 0, 0, type, flags);
+ err = neigh_fill_info(skb, n, pid, 0, type, flags);
if (err < 0) {
/* -EMSGSIZE implies BUG in neigh_nlmsg_size() */
WARN_ON(err == -EMSGSIZE);
@@ -2857,7 +2864,7 @@
void neigh_app_ns(struct neighbour *n)
{
- __neigh_notify(n, RTM_GETNEIGH, NLM_F_REQUEST);
+ __neigh_notify(n, RTM_GETNEIGH, NLM_F_REQUEST, 0);
}
EXPORT_SYMBOL(neigh_app_ns);
diff --git a/net/core/net_namespace.c b/net/core/net_namespace.c
index 652468f..1934efd 100644
--- a/net/core/net_namespace.c
+++ b/net/core/net_namespace.c
@@ -35,7 +35,8 @@
EXPORT_SYMBOL_GPL(net_namespace_list);
struct net init_net = {
- .dev_base_head = LIST_HEAD_INIT(init_net.dev_base_head),
+ .count = ATOMIC_INIT(1),
+ .dev_base_head = LIST_HEAD_INIT(init_net.dev_base_head),
};
EXPORT_SYMBOL(init_net);
@@ -571,7 +572,8 @@
[NETNSA_FD] = { .type = NLA_U32 },
};
-static int rtnl_net_newid(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int rtnl_net_newid(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct nlattr *tb[NETNSA_MAX + 1];
@@ -579,7 +581,7 @@
int nsid, err;
err = nlmsg_parse(nlh, sizeof(struct rtgenmsg), tb, NETNSA_MAX,
- rtnl_net_policy);
+ rtnl_net_policy, extack);
if (err < 0)
return err;
if (!tb[NETNSA_NSID])
@@ -644,7 +646,8 @@
return -EMSGSIZE;
}
-static int rtnl_net_getid(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int rtnl_net_getid(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct nlattr *tb[NETNSA_MAX + 1];
@@ -653,7 +656,7 @@
int err, id;
err = nlmsg_parse(nlh, sizeof(struct rtgenmsg), tb, NETNSA_MAX,
- rtnl_net_policy);
+ rtnl_net_policy, extack);
if (err < 0)
return err;
if (tb[NETNSA_PID])
diff --git a/net/core/netprio_cgroup.c b/net/core/netprio_cgroup.c
index 0f9275e..1c48109 100644
--- a/net/core/netprio_cgroup.c
+++ b/net/core/netprio_cgroup.c
@@ -11,6 +11,7 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/module.h>
#include <linux/slab.h>
#include <linux/types.h>
#include <linux/module.h>
diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
index c4e84c5..6e67315 100644
--- a/net/core/rtnetlink.c
+++ b/net/core/rtnetlink.c
@@ -896,15 +896,13 @@
return port_self_size;
}
-static size_t rtnl_xdp_size(const struct net_device *dev)
+static size_t rtnl_xdp_size(void)
{
size_t xdp_size = nla_total_size(0) + /* nest IFLA_XDP */
- nla_total_size(1); /* XDP_ATTACHED */
+ nla_total_size(1) + /* XDP_ATTACHED */
+ nla_total_size(4); /* XDP_FLAGS */
- if (!dev->netdev_ops->ndo_xdp)
- return 0;
- else
- return xdp_size;
+ return xdp_size;
}
static noinline size_t if_nlmsg_size(const struct net_device *dev,
@@ -943,7 +941,7 @@
+ nla_total_size(MAX_PHYS_ITEM_ID_LEN) /* IFLA_PHYS_PORT_ID */
+ nla_total_size(MAX_PHYS_ITEM_ID_LEN) /* IFLA_PHYS_SWITCH_ID */
+ nla_total_size(IFNAMSIZ) /* IFLA_PHYS_PORT_NAME */
- + rtnl_xdp_size(dev) /* IFLA_XDP */
+ + rtnl_xdp_size() /* IFLA_XDP */
+ nla_total_size(1); /* IFLA_PROTO_DOWN */
}
@@ -1251,23 +1249,35 @@
static int rtnl_xdp_fill(struct sk_buff *skb, struct net_device *dev)
{
- struct netdev_xdp xdp_op = {};
struct nlattr *xdp;
+ u32 xdp_flags = 0;
+ u8 val = 0;
int err;
- if (!dev->netdev_ops->ndo_xdp)
- return 0;
xdp = nla_nest_start(skb, IFLA_XDP);
if (!xdp)
return -EMSGSIZE;
- xdp_op.command = XDP_QUERY_PROG;
- err = dev->netdev_ops->ndo_xdp(dev, &xdp_op);
- if (err)
- goto err_cancel;
- err = nla_put_u8(skb, IFLA_XDP_ATTACHED, xdp_op.prog_attached);
+ if (rcu_access_pointer(dev->xdp_prog)) {
+ xdp_flags = XDP_FLAGS_SKB_MODE;
+ val = 1;
+ } else if (dev->netdev_ops->ndo_xdp) {
+ struct netdev_xdp xdp_op = {};
+
+ xdp_op.command = XDP_QUERY_PROG;
+ err = dev->netdev_ops->ndo_xdp(dev, &xdp_op);
+ if (err)
+ goto err_cancel;
+ val = xdp_op.prog_attached;
+ }
+ err = nla_put_u8(skb, IFLA_XDP_ATTACHED, val);
if (err)
goto err_cancel;
+ if (xdp_flags) {
+ err = nla_put_u32(skb, IFLA_XDP_FLAGS, xdp_flags);
+ if (err)
+ goto err_cancel;
+ }
nla_nest_end(skb, xdp);
return 0;
@@ -1515,7 +1525,8 @@
const struct rtnl_link_ops *ops = NULL;
struct nlattr *linfo[IFLA_INFO_MAX + 1];
- if (nla_parse_nested(linfo, IFLA_INFO_MAX, nla, ifla_info_policy) < 0)
+ if (nla_parse_nested(linfo, IFLA_INFO_MAX, nla,
+ ifla_info_policy, NULL) < 0)
return NULL;
if (linfo[IFLA_INFO_KIND]) {
@@ -1592,8 +1603,8 @@
hdrlen = nlmsg_len(cb->nlh) < sizeof(struct ifinfomsg) ?
sizeof(struct rtgenmsg) : sizeof(struct ifinfomsg);
- if (nlmsg_parse(cb->nlh, hdrlen, tb, IFLA_MAX, ifla_policy) >= 0) {
-
+ if (nlmsg_parse(cb->nlh, hdrlen, tb, IFLA_MAX,
+ ifla_policy, NULL) >= 0) {
if (tb[IFLA_EXT_MASK])
ext_filter_mask = nla_get_u32(tb[IFLA_EXT_MASK]);
@@ -1640,9 +1651,10 @@
return skb->len;
}
-int rtnl_nla_parse_ifla(struct nlattr **tb, const struct nlattr *head, int len)
+int rtnl_nla_parse_ifla(struct nlattr **tb, const struct nlattr *head, int len,
+ struct netlink_ext_ack *exterr)
{
- return nla_parse(tb, IFLA_MAX, head, len, ifla_policy);
+ return nla_parse(tb, IFLA_MAX, head, len, ifla_policy, exterr);
}
EXPORT_SYMBOL(rtnl_nla_parse_ifla);
@@ -1907,6 +1919,7 @@
#define DO_SETLINK_NOTIFY 0x03
static int do_setlink(const struct sk_buff *skb,
struct net_device *dev, struct ifinfomsg *ifm,
+ struct netlink_ext_ack *extack,
struct nlattr **tb, char *ifname, int status)
{
const struct net_device_ops *ops = dev->netdev_ops;
@@ -2078,7 +2091,7 @@
goto errout;
}
err = nla_parse_nested(vfinfo, IFLA_VF_MAX, attr,
- ifla_vf_policy);
+ ifla_vf_policy, NULL);
if (err < 0)
goto errout;
err = do_setvfinfo(dev, vfinfo);
@@ -2106,7 +2119,7 @@
goto errout;
}
err = nla_parse_nested(port, IFLA_PORT_MAX, attr,
- ifla_port_policy);
+ ifla_port_policy, NULL);
if (err < 0)
goto errout;
if (!port[IFLA_PORT_VF]) {
@@ -2126,7 +2139,8 @@
struct nlattr *port[IFLA_PORT_MAX+1];
err = nla_parse_nested(port, IFLA_PORT_MAX,
- tb[IFLA_PORT_SELF], ifla_port_policy);
+ tb[IFLA_PORT_SELF], ifla_port_policy,
+ NULL);
if (err < 0)
goto errout;
@@ -2170,7 +2184,7 @@
u32 xdp_flags = 0;
err = nla_parse_nested(xdp, IFLA_XDP_MAX, tb[IFLA_XDP],
- ifla_xdp_policy);
+ ifla_xdp_policy, NULL);
if (err < 0)
goto errout;
@@ -2188,7 +2202,7 @@
}
if (xdp[IFLA_XDP_FD]) {
- err = dev_change_xdp_fd(dev,
+ err = dev_change_xdp_fd(dev, extack,
nla_get_s32(xdp[IFLA_XDP_FD]),
xdp_flags);
if (err)
@@ -2210,7 +2224,8 @@
return err;
}
-static int rtnl_setlink(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int rtnl_setlink(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct ifinfomsg *ifm;
@@ -2219,7 +2234,8 @@
struct nlattr *tb[IFLA_MAX+1];
char ifname[IFNAMSIZ];
- err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy,
+ extack);
if (err < 0)
goto errout;
@@ -2246,7 +2262,7 @@
if (err < 0)
goto errout;
- err = do_setlink(skb, dev, ifm, tb, ifname, 0);
+ err = do_setlink(skb, dev, ifm, extack, tb, ifname, 0);
errout:
return err;
}
@@ -2303,7 +2319,8 @@
}
EXPORT_SYMBOL_GPL(rtnl_delete_link);
-static int rtnl_dellink(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int rtnl_dellink(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct net_device *dev;
@@ -2312,7 +2329,7 @@
struct nlattr *tb[IFLA_MAX+1];
int err;
- err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy, extack);
if (err < 0)
return err;
@@ -2407,6 +2424,7 @@
static int rtnl_group_changelink(const struct sk_buff *skb,
struct net *net, int group,
struct ifinfomsg *ifm,
+ struct netlink_ext_ack *extack,
struct nlattr **tb)
{
struct net_device *dev, *aux;
@@ -2414,7 +2432,7 @@
for_each_netdev_safe(net, dev, aux) {
if (dev->group == group) {
- err = do_setlink(skb, dev, ifm, tb, NULL, 0);
+ err = do_setlink(skb, dev, ifm, extack, tb, NULL, 0);
if (err < 0)
return err;
}
@@ -2423,7 +2441,8 @@
return 0;
}
-static int rtnl_newlink(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int rtnl_newlink(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
const struct rtnl_link_ops *ops;
@@ -2441,7 +2460,7 @@
#ifdef CONFIG_MODULES
replay:
#endif
- err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy, extack);
if (err < 0)
return err;
@@ -2472,7 +2491,8 @@
if (tb[IFLA_LINKINFO]) {
err = nla_parse_nested(linkinfo, IFLA_INFO_MAX,
- tb[IFLA_LINKINFO], ifla_info_policy);
+ tb[IFLA_LINKINFO], ifla_info_policy,
+ NULL);
if (err < 0)
return err;
} else
@@ -2497,7 +2517,7 @@
if (ops->maxtype && linkinfo[IFLA_INFO_DATA]) {
err = nla_parse_nested(attr, ops->maxtype,
linkinfo[IFLA_INFO_DATA],
- ops->policy);
+ ops->policy, NULL);
if (err < 0)
return err;
data = attr;
@@ -2515,7 +2535,8 @@
err = nla_parse_nested(slave_attr,
m_ops->slave_maxtype,
linkinfo[IFLA_INFO_SLAVE_DATA],
- m_ops->slave_policy);
+ m_ops->slave_policy,
+ NULL);
if (err < 0)
return err;
slave_data = slave_attr;
@@ -2557,14 +2578,15 @@
status |= DO_SETLINK_NOTIFY;
}
- return do_setlink(skb, dev, ifm, tb, ifname, status);
+ return do_setlink(skb, dev, ifm, extack, tb, ifname,
+ status);
}
if (!(nlh->nlmsg_flags & NLM_F_CREATE)) {
if (ifm->ifi_index == 0 && tb[IFLA_GROUP])
return rtnl_group_changelink(skb, net,
nla_get_u32(tb[IFLA_GROUP]),
- ifm, tb);
+ ifm, extack, tb);
return -ENODEV;
}
@@ -2673,7 +2695,8 @@
}
}
-static int rtnl_getlink(struct sk_buff *skb, struct nlmsghdr* nlh)
+static int rtnl_getlink(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct ifinfomsg *ifm;
@@ -2684,7 +2707,7 @@
int err;
u32 ext_filter_mask = 0;
- err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFLA_MAX, ifla_policy, extack);
if (err < 0)
return err;
@@ -2734,7 +2757,7 @@
hdrlen = nlmsg_len(nlh) < sizeof(struct ifinfomsg) ?
sizeof(struct rtgenmsg) : sizeof(struct ifinfomsg);
- if (nlmsg_parse(nlh, hdrlen, tb, IFLA_MAX, ifla_policy) >= 0) {
+ if (nlmsg_parse(nlh, hdrlen, tb, IFLA_MAX, ifla_policy, NULL) >= 0) {
if (tb[IFLA_EXT_MASK])
ext_filter_mask = nla_get_u32(tb[IFLA_EXT_MASK]);
}
@@ -2955,7 +2978,8 @@
return 0;
}
-static int rtnl_fdb_add(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int rtnl_fdb_add(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct ndmsg *ndm;
@@ -2965,7 +2989,7 @@
u16 vid;
int err;
- err = nlmsg_parse(nlh, sizeof(*ndm), tb, NDA_MAX, NULL);
+ err = nlmsg_parse(nlh, sizeof(*ndm), tb, NDA_MAX, NULL, extack);
if (err < 0)
return err;
@@ -3055,7 +3079,8 @@
}
EXPORT_SYMBOL(ndo_dflt_fdb_del);
-static int rtnl_fdb_del(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int rtnl_fdb_del(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct ndmsg *ndm;
@@ -3068,7 +3093,7 @@
if (!netlink_capable(skb, CAP_NET_ADMIN))
return -EPERM;
- err = nlmsg_parse(nlh, sizeof(*ndm), tb, NDA_MAX, NULL);
+ err = nlmsg_parse(nlh, sizeof(*ndm), tb, NDA_MAX, NULL, extack);
if (err < 0)
return err;
@@ -3203,8 +3228,8 @@
int err = 0;
int fidx = 0;
- if (nlmsg_parse(cb->nlh, sizeof(struct ifinfomsg), tb, IFLA_MAX,
- ifla_policy) == 0) {
+ if (nlmsg_parse(cb->nlh, sizeof(struct ifinfomsg), tb,
+ IFLA_MAX, ifla_policy, NULL) == 0) {
if (tb[IFLA_MASTER])
br_idx = nla_get_u32(tb[IFLA_MASTER]);
}
@@ -3498,7 +3523,8 @@
return err;
}
-static int rtnl_bridge_setlink(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int rtnl_bridge_setlink(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct ifinfomsg *ifm;
@@ -3572,7 +3598,8 @@
return err;
}
-static int rtnl_bridge_dellink(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int rtnl_bridge_dellink(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct ifinfomsg *ifm;
@@ -3940,7 +3967,8 @@
return size;
}
-static int rtnl_stats_get(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int rtnl_stats_get(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct net_device *dev = NULL;
@@ -4046,7 +4074,8 @@
/* Process one rtnetlink message. */
-static int rtnetlink_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int rtnetlink_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
rtnl_doit_func doit;
@@ -4101,7 +4130,7 @@
if (doit == NULL)
return -EOPNOTSUPP;
- return doit(skb, nlh);
+ return doit(skb, nlh, extack);
}
static void rtnetlink_rcv(struct sk_buff *skb)
@@ -4116,22 +4145,16 @@
struct net_device *dev = netdev_notifier_info_to_dev(ptr);
switch (event) {
- case NETDEV_UP:
- case NETDEV_DOWN:
- case NETDEV_PRE_UP:
- case NETDEV_POST_INIT:
- case NETDEV_REGISTER:
- case NETDEV_CHANGE:
- case NETDEV_PRE_TYPE_CHANGE:
- case NETDEV_GOING_DOWN:
- case NETDEV_UNREGISTER:
- case NETDEV_UNREGISTER_FINAL:
- case NETDEV_RELEASE:
- case NETDEV_JOIN:
- case NETDEV_BONDING_INFO:
+ case NETDEV_REBOOT:
+ case NETDEV_CHANGENAME:
+ case NETDEV_FEAT_CHANGE:
+ case NETDEV_BONDING_FAILOVER:
+ case NETDEV_NOTIFY_PEERS:
+ case NETDEV_RESEND_IGMP:
+ case NETDEV_CHANGEINFODATA:
+ rtmsg_ifinfo(RTM_NEWLINK, dev, 0, GFP_KERNEL);
break;
default:
- rtmsg_ifinfo(RTM_NEWLINK, dev, 0, GFP_KERNEL);
break;
}
return NOTIFY_DONE;
@@ -4185,6 +4208,7 @@
rtnl_register(PF_UNSPEC, RTM_GETADDR, NULL, rtnl_dump_all, NULL);
rtnl_register(PF_UNSPEC, RTM_GETROUTE, NULL, rtnl_dump_all, NULL);
+ rtnl_register(PF_UNSPEC, RTM_GETNETCONF, NULL, rtnl_dump_all, NULL);
rtnl_register(PF_BRIDGE, RTM_NEWNEIGH, rtnl_fdb_add, NULL, NULL);
rtnl_register(PF_BRIDGE, RTM_DELNEIGH, rtnl_fdb_del, NULL, NULL);
diff --git a/net/core/secure_seq.c b/net/core/secure_seq.c
index d28da7d..6bd2f8f 100644
--- a/net/core/secure_seq.c
+++ b/net/core/secure_seq.c
@@ -64,8 +64,8 @@
&ts_secret);
}
-u32 secure_tcpv6_sequence_number(const __be32 *saddr, const __be32 *daddr,
- __be16 sport, __be16 dport, u32 *tsoff)
+u32 secure_tcpv6_seq_and_tsoff(const __be32 *saddr, const __be32 *daddr,
+ __be16 sport, __be16 dport, u32 *tsoff)
{
const struct {
struct in6_addr saddr;
@@ -85,7 +85,7 @@
*tsoff = secure_tcpv6_ts_off(saddr, daddr);
return seq_scale(hash);
}
-EXPORT_SYMBOL(secure_tcpv6_sequence_number);
+EXPORT_SYMBOL(secure_tcpv6_seq_and_tsoff);
u32 secure_ipv6_port_ephemeral(const __be32 *saddr, const __be32 *daddr,
__be16 dport)
@@ -116,14 +116,13 @@
&ts_secret);
}
-/* secure_tcp_sequence_number(a, b, 0, d) == secure_ipv4_port_ephemeral(a, b, d),
+/* secure_tcp_seq_and_tsoff(a, b, 0, d) == secure_ipv4_port_ephemeral(a, b, d),
* but fortunately, `sport' cannot be 0 in any circumstances. If this changes,
* it would be easy enough to have the former function use siphash_4u32, passing
* the arguments as separate u32.
*/
-
-u32 secure_tcp_sequence_number(__be32 saddr, __be32 daddr,
- __be16 sport, __be16 dport, u32 *tsoff)
+u32 secure_tcp_seq_and_tsoff(__be32 saddr, __be32 daddr,
+ __be16 sport, __be16 dport, u32 *tsoff)
{
u64 hash;
net_secret_init();
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index f86bf69..346d3e8 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -1576,6 +1576,8 @@
skb_set_tail_pointer(skb, len);
}
+ if (!skb->sk || skb->destructor == sock_edemux)
+ skb_condense(skb);
return 0;
}
EXPORT_SYMBOL(___pskb_trim);
@@ -1980,7 +1982,6 @@
.pages = pages,
.partial = partial,
.nr_pages_max = MAX_SKB_FRAGS,
- .flags = flags,
.ops = &nosteal_pipe_buf_ops,
.spd_release = sock_spd_release,
};
@@ -3100,7 +3101,7 @@
skb_walk_frags(head_skb, iter) {
if (frag_len != iter->len && iter->next)
goto normal;
- if (skb_headlen(iter))
+ if (skb_headlen(iter) && !iter->head_frag)
goto normal;
len -= iter->len;
diff --git a/net/core/sock.c b/net/core/sock.c
index 2c4f574..b5baeb9 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -247,12 +247,66 @@
static const char *const af_family_kern_clock_key_strings[AF_MAX+1] = {
_sock_locks("k-clock-")
};
+static const char *const af_family_rlock_key_strings[AF_MAX+1] = {
+ "rlock-AF_UNSPEC", "rlock-AF_UNIX" , "rlock-AF_INET" ,
+ "rlock-AF_AX25" , "rlock-AF_IPX" , "rlock-AF_APPLETALK",
+ "rlock-AF_NETROM", "rlock-AF_BRIDGE" , "rlock-AF_ATMPVC" ,
+ "rlock-AF_X25" , "rlock-AF_INET6" , "rlock-AF_ROSE" ,
+ "rlock-AF_DECnet", "rlock-AF_NETBEUI" , "rlock-AF_SECURITY" ,
+ "rlock-AF_KEY" , "rlock-AF_NETLINK" , "rlock-AF_PACKET" ,
+ "rlock-AF_ASH" , "rlock-AF_ECONET" , "rlock-AF_ATMSVC" ,
+ "rlock-AF_RDS" , "rlock-AF_SNA" , "rlock-AF_IRDA" ,
+ "rlock-AF_PPPOX" , "rlock-AF_WANPIPE" , "rlock-AF_LLC" ,
+ "rlock-27" , "rlock-28" , "rlock-AF_CAN" ,
+ "rlock-AF_TIPC" , "rlock-AF_BLUETOOTH", "rlock-AF_IUCV" ,
+ "rlock-AF_RXRPC" , "rlock-AF_ISDN" , "rlock-AF_PHONET" ,
+ "rlock-AF_IEEE802154", "rlock-AF_CAIF" , "rlock-AF_ALG" ,
+ "rlock-AF_NFC" , "rlock-AF_VSOCK" , "rlock-AF_KCM" ,
+ "rlock-AF_QIPCRTR", "rlock-AF_SMC" , "rlock-AF_MAX"
+};
+static const char *const af_family_wlock_key_strings[AF_MAX+1] = {
+ "wlock-AF_UNSPEC", "wlock-AF_UNIX" , "wlock-AF_INET" ,
+ "wlock-AF_AX25" , "wlock-AF_IPX" , "wlock-AF_APPLETALK",
+ "wlock-AF_NETROM", "wlock-AF_BRIDGE" , "wlock-AF_ATMPVC" ,
+ "wlock-AF_X25" , "wlock-AF_INET6" , "wlock-AF_ROSE" ,
+ "wlock-AF_DECnet", "wlock-AF_NETBEUI" , "wlock-AF_SECURITY" ,
+ "wlock-AF_KEY" , "wlock-AF_NETLINK" , "wlock-AF_PACKET" ,
+ "wlock-AF_ASH" , "wlock-AF_ECONET" , "wlock-AF_ATMSVC" ,
+ "wlock-AF_RDS" , "wlock-AF_SNA" , "wlock-AF_IRDA" ,
+ "wlock-AF_PPPOX" , "wlock-AF_WANPIPE" , "wlock-AF_LLC" ,
+ "wlock-27" , "wlock-28" , "wlock-AF_CAN" ,
+ "wlock-AF_TIPC" , "wlock-AF_BLUETOOTH", "wlock-AF_IUCV" ,
+ "wlock-AF_RXRPC" , "wlock-AF_ISDN" , "wlock-AF_PHONET" ,
+ "wlock-AF_IEEE802154", "wlock-AF_CAIF" , "wlock-AF_ALG" ,
+ "wlock-AF_NFC" , "wlock-AF_VSOCK" , "wlock-AF_KCM" ,
+ "wlock-AF_QIPCRTR", "wlock-AF_SMC" , "wlock-AF_MAX"
+};
+static const char *const af_family_elock_key_strings[AF_MAX+1] = {
+ "elock-AF_UNSPEC", "elock-AF_UNIX" , "elock-AF_INET" ,
+ "elock-AF_AX25" , "elock-AF_IPX" , "elock-AF_APPLETALK",
+ "elock-AF_NETROM", "elock-AF_BRIDGE" , "elock-AF_ATMPVC" ,
+ "elock-AF_X25" , "elock-AF_INET6" , "elock-AF_ROSE" ,
+ "elock-AF_DECnet", "elock-AF_NETBEUI" , "elock-AF_SECURITY" ,
+ "elock-AF_KEY" , "elock-AF_NETLINK" , "elock-AF_PACKET" ,
+ "elock-AF_ASH" , "elock-AF_ECONET" , "elock-AF_ATMSVC" ,
+ "elock-AF_RDS" , "elock-AF_SNA" , "elock-AF_IRDA" ,
+ "elock-AF_PPPOX" , "elock-AF_WANPIPE" , "elock-AF_LLC" ,
+ "elock-27" , "elock-28" , "elock-AF_CAN" ,
+ "elock-AF_TIPC" , "elock-AF_BLUETOOTH", "elock-AF_IUCV" ,
+ "elock-AF_RXRPC" , "elock-AF_ISDN" , "elock-AF_PHONET" ,
+ "elock-AF_IEEE802154", "elock-AF_CAIF" , "elock-AF_ALG" ,
+ "elock-AF_NFC" , "elock-AF_VSOCK" , "elock-AF_KCM" ,
+ "elock-AF_QIPCRTR", "elock-AF_SMC" , "elock-AF_MAX"
+};
/*
- * sk_callback_lock locking rules are per-address-family,
+ * sk_callback_lock and sk queues locking rules are per-address-family,
* so split the lock classes by using a per-AF key:
*/
static struct lock_class_key af_callback_keys[AF_MAX];
+static struct lock_class_key af_rlock_keys[AF_MAX];
+static struct lock_class_key af_wlock_keys[AF_MAX];
+static struct lock_class_key af_elock_keys[AF_MAX];
static struct lock_class_key af_kern_callback_keys[AF_MAX];
/* Take into consideration the size of the struct sk_buff overhead in the
@@ -325,7 +379,7 @@
current->flags |= PF_MEMALLOC;
ret = sk->sk_backlog_rcv(sk, skb);
- tsk_restore_flags(current, pflags, PF_MEMALLOC);
+ current_restore_flags(pflags, PF_MEMALLOC);
return ret;
}
@@ -1029,6 +1083,7 @@
union {
int val;
+ u64 val64;
struct linger ling;
struct timeval tm;
} v;
@@ -1259,6 +1314,40 @@
v.val = sk->sk_incoming_cpu;
break;
+ case SO_MEMINFO:
+ {
+ u32 meminfo[SK_MEMINFO_VARS];
+
+ if (get_user(len, optlen))
+ return -EFAULT;
+
+ sk_get_meminfo(sk, meminfo);
+
+ len = min_t(unsigned int, len, sizeof(meminfo));
+ if (copy_to_user(optval, &meminfo, len))
+ return -EFAULT;
+
+ goto lenout;
+ }
+
+#ifdef CONFIG_NET_RX_BUSY_POLL
+ case SO_INCOMING_NAPI_ID:
+ v.val = READ_ONCE(sk->sk_napi_id);
+
+ /* aggregate non-NAPI IDs down to 0 */
+ if (v.val < MIN_NAPI_ID)
+ v.val = 0;
+
+ break;
+#endif
+
+ case SO_COOKIE:
+ lv = sizeof(u64);
+ if (len < lv)
+ return -EINVAL;
+ v.val64 = sock_gen_cookie(sk);
+ break;
+
default:
/* We implement the SO_SNDLOWAT etc to not be settable
* (1003.1g 7).
@@ -1483,6 +1572,27 @@
}
EXPORT_SYMBOL(sk_free);
+static void sk_init_common(struct sock *sk)
+{
+ skb_queue_head_init(&sk->sk_receive_queue);
+ skb_queue_head_init(&sk->sk_write_queue);
+ skb_queue_head_init(&sk->sk_error_queue);
+
+ rwlock_init(&sk->sk_callback_lock);
+ lockdep_set_class_and_name(&sk->sk_receive_queue.lock,
+ af_rlock_keys + sk->sk_family,
+ af_family_rlock_key_strings[sk->sk_family]);
+ lockdep_set_class_and_name(&sk->sk_write_queue.lock,
+ af_wlock_keys + sk->sk_family,
+ af_family_wlock_key_strings[sk->sk_family]);
+ lockdep_set_class_and_name(&sk->sk_error_queue.lock,
+ af_elock_keys + sk->sk_family,
+ af_family_elock_key_strings[sk->sk_family]);
+ lockdep_set_class_and_name(&sk->sk_callback_lock,
+ af_callback_keys + sk->sk_family,
+ af_family_clock_key_strings[sk->sk_family]);
+}
+
/**
* sk_clone_lock - clone a socket, and lock its clone
* @sk: the socket to clone
@@ -1516,13 +1626,7 @@
*/
atomic_set(&newsk->sk_wmem_alloc, 1);
atomic_set(&newsk->sk_omem_alloc, 0);
- skb_queue_head_init(&newsk->sk_receive_queue);
- skb_queue_head_init(&newsk->sk_write_queue);
-
- rwlock_init(&newsk->sk_callback_lock);
- lockdep_set_class_and_name(&newsk->sk_callback_lock,
- af_callback_keys + newsk->sk_family,
- af_family_clock_key_strings[newsk->sk_family]);
+ sk_init_common(newsk);
newsk->sk_dst_cache = NULL;
newsk->sk_dst_pending_confirm = 0;
@@ -1533,7 +1637,6 @@
newsk->sk_userlocks = sk->sk_userlocks & ~SOCK_BINDPORT_LOCK;
sock_reset_flag(newsk, SOCK_DONE);
- skb_queue_head_init(&newsk->sk_error_queue);
filter = rcu_dereference_protected(newsk->sk_filter, 1);
if (filter != NULL)
@@ -2466,10 +2569,7 @@
void sock_init_data(struct socket *sock, struct sock *sk)
{
- skb_queue_head_init(&sk->sk_receive_queue);
- skb_queue_head_init(&sk->sk_write_queue);
- skb_queue_head_init(&sk->sk_error_queue);
-
+ sk_init_common(sk);
sk->sk_send_head = NULL;
init_timer(&sk->sk_timer);
@@ -2521,7 +2621,7 @@
sk->sk_rcvtimeo = MAX_SCHEDULE_TIMEOUT;
sk->sk_sndtimeo = MAX_SCHEDULE_TIMEOUT;
- sk->sk_stamp = ktime_set(-1L, 0);
+ sk->sk_stamp = SK_DEFAULT_STAMP;
#ifdef CONFIG_NET_RX_BUSY_POLL
sk->sk_napi_id = 0;
@@ -2802,6 +2902,21 @@
}
EXPORT_SYMBOL(sk_common_release);
+void sk_get_meminfo(const struct sock *sk, u32 *mem)
+{
+ memset(mem, 0, sizeof(*mem) * SK_MEMINFO_VARS);
+
+ mem[SK_MEMINFO_RMEM_ALLOC] = sk_rmem_alloc_get(sk);
+ mem[SK_MEMINFO_RCVBUF] = sk->sk_rcvbuf;
+ mem[SK_MEMINFO_WMEM_ALLOC] = sk_wmem_alloc_get(sk);
+ mem[SK_MEMINFO_SNDBUF] = sk->sk_sndbuf;
+ mem[SK_MEMINFO_FWD_ALLOC] = sk->sk_forward_alloc;
+ mem[SK_MEMINFO_WMEM_QUEUED] = sk->sk_wmem_queued;
+ mem[SK_MEMINFO_OPTMEM] = atomic_read(&sk->sk_omem_alloc);
+ mem[SK_MEMINFO_BACKLOG] = sk->sk_backlog.len;
+ mem[SK_MEMINFO_DROPS] = atomic_read(&sk->sk_drops);
+}
+
#ifdef CONFIG_PROC_FS
#define PROTO_INUSE_NR 64 /* should be enough for the first time */
struct prot_inuse {
@@ -3142,3 +3257,14 @@
subsys_initcall(proto_init);
#endif /* PROC_FS */
+
+#ifdef CONFIG_NET_RX_BUSY_POLL
+bool sk_busy_loop_end(void *p, unsigned long start_time)
+{
+ struct sock *sk = p;
+
+ return !skb_queue_empty(&sk->sk_receive_queue) ||
+ sk_busy_loop_timeout(sk, start_time);
+}
+EXPORT_SYMBOL(sk_busy_loop_end);
+#endif /* CONFIG_NET_RX_BUSY_POLL */
diff --git a/net/core/sock_diag.c b/net/core/sock_diag.c
index 6b10573..217f4e3 100644
--- a/net/core/sock_diag.c
+++ b/net/core/sock_diag.c
@@ -19,7 +19,7 @@
static DEFINE_MUTEX(sock_diag_table_mutex);
static struct workqueue_struct *broadcast_wq;
-static u64 sock_gen_cookie(struct sock *sk)
+u64 sock_gen_cookie(struct sock *sk)
{
while (1) {
u64 res = atomic64_read(&sk->sk_cookie);
@@ -59,15 +59,7 @@
{
u32 mem[SK_MEMINFO_VARS];
- mem[SK_MEMINFO_RMEM_ALLOC] = sk_rmem_alloc_get(sk);
- mem[SK_MEMINFO_RCVBUF] = sk->sk_rcvbuf;
- mem[SK_MEMINFO_WMEM_ALLOC] = sk_wmem_alloc_get(sk);
- mem[SK_MEMINFO_SNDBUF] = sk->sk_sndbuf;
- mem[SK_MEMINFO_FWD_ALLOC] = sk->sk_forward_alloc;
- mem[SK_MEMINFO_WMEM_QUEUED] = sk->sk_wmem_queued;
- mem[SK_MEMINFO_OPTMEM] = atomic_read(&sk->sk_omem_alloc);
- mem[SK_MEMINFO_BACKLOG] = sk->sk_backlog.len;
- mem[SK_MEMINFO_DROPS] = atomic_read(&sk->sk_drops);
+ sk_get_meminfo(sk, mem);
return nla_put(skb, attrtype, sizeof(mem), &mem);
}
@@ -246,7 +238,8 @@
return err;
}
-static int sock_diag_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int sock_diag_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
int ret;
diff --git a/net/core/sock_reuseport.c b/net/core/sock_reuseport.c
index 9a1a352..eed1ebf 100644
--- a/net/core/sock_reuseport.c
+++ b/net/core/sock_reuseport.c
@@ -13,9 +13,9 @@
static DEFINE_SPINLOCK(reuseport_lock);
-static struct sock_reuseport *__reuseport_alloc(u16 max_socks)
+static struct sock_reuseport *__reuseport_alloc(unsigned int max_socks)
{
- size_t size = sizeof(struct sock_reuseport) +
+ unsigned int size = sizeof(struct sock_reuseport) +
sizeof(struct sock *) * max_socks;
struct sock_reuseport *reuse = kzalloc(size, GFP_ATOMIC);
diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c
index 7f9cc40..ea23254 100644
--- a/net/core/sysctl_net_core.c
+++ b/net/core/sysctl_net_core.c
@@ -452,6 +452,14 @@
.extra1 = &one,
.extra2 = &max_skb_frags,
},
+ {
+ .procname = "netdev_budget_usecs",
+ .data = &netdev_budget_usecs,
+ .maxlen = sizeof(unsigned int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = &zero,
+ },
{ }
};
diff --git a/net/core/utils.c b/net/core/utils.c
index 6592d7b..93066bd 100644
--- a/net/core/utils.c
+++ b/net/core/utils.c
@@ -26,9 +26,11 @@
#include <linux/percpu.h>
#include <linux/init.h>
#include <linux/ratelimit.h>
+#include <linux/socket.h>
#include <net/sock.h>
#include <net/net_ratelimit.h>
+#include <net/ipv6.h>
#include <asm/byteorder.h>
#include <linux/uaccess.h>
@@ -51,7 +53,7 @@
__be32 in_aton(const char *str)
{
- unsigned long l;
+ unsigned int l;
unsigned int val;
int i;
@@ -300,6 +302,107 @@
}
EXPORT_SYMBOL(in6_pton);
+static int inet4_pton(const char *src, u16 port_num,
+ struct sockaddr_storage *addr)
+{
+ struct sockaddr_in *addr4 = (struct sockaddr_in *)addr;
+ int srclen = strlen(src);
+
+ if (srclen > INET_ADDRSTRLEN)
+ return -EINVAL;
+
+ if (in4_pton(src, srclen, (u8 *)&addr4->sin_addr.s_addr,
+ '\n', NULL) == 0)
+ return -EINVAL;
+
+ addr4->sin_family = AF_INET;
+ addr4->sin_port = htons(port_num);
+
+ return 0;
+}
+
+static int inet6_pton(struct net *net, const char *src, u16 port_num,
+ struct sockaddr_storage *addr)
+{
+ struct sockaddr_in6 *addr6 = (struct sockaddr_in6 *)addr;
+ const char *scope_delim;
+ int srclen = strlen(src);
+
+ if (srclen > INET6_ADDRSTRLEN)
+ return -EINVAL;
+
+ if (in6_pton(src, srclen, (u8 *)&addr6->sin6_addr.s6_addr,
+ '%', &scope_delim) == 0)
+ return -EINVAL;
+
+ if (ipv6_addr_type(&addr6->sin6_addr) & IPV6_ADDR_LINKLOCAL &&
+ src + srclen != scope_delim && *scope_delim == '%') {
+ struct net_device *dev;
+ char scope_id[16];
+ size_t scope_len = min_t(size_t, sizeof(scope_id) - 1,
+ src + srclen - scope_delim - 1);
+
+ memcpy(scope_id, scope_delim + 1, scope_len);
+ scope_id[scope_len] = '\0';
+
+ dev = dev_get_by_name(net, scope_id);
+ if (dev) {
+ addr6->sin6_scope_id = dev->ifindex;
+ dev_put(dev);
+ } else if (kstrtouint(scope_id, 0, &addr6->sin6_scope_id)) {
+ return -EINVAL;
+ }
+ }
+
+ addr6->sin6_family = AF_INET6;
+ addr6->sin6_port = htons(port_num);
+
+ return 0;
+}
+
+/**
+ * inet_pton_with_scope - convert an IPv4/IPv6 and port to socket address
+ * @net: net namespace (used for scope handling)
+ * @af: address family, AF_INET, AF_INET6 or AF_UNSPEC for either
+ * @src: the start of the address string
+ * @port: the start of the port string (or NULL for none)
+ * @addr: output socket address
+ *
+ * Return zero on success, return errno when any error occurs.
+ */
+int inet_pton_with_scope(struct net *net, __kernel_sa_family_t af,
+ const char *src, const char *port, struct sockaddr_storage *addr)
+{
+ u16 port_num;
+ int ret = -EINVAL;
+
+ if (port) {
+ if (kstrtou16(port, 0, &port_num))
+ return -EINVAL;
+ } else {
+ port_num = 0;
+ }
+
+ switch (af) {
+ case AF_INET:
+ ret = inet4_pton(src, port_num, addr);
+ break;
+ case AF_INET6:
+ ret = inet6_pton(net, src, port_num, addr);
+ break;
+ case AF_UNSPEC:
+ ret = inet4_pton(src, port_num, addr);
+ if (ret)
+ ret = inet6_pton(net, src, port_num, addr);
+ break;
+ default:
+ pr_err("unexpected address family %d\n", af);
+ };
+
+ return ret;
+}
+EXPORT_SYMBOL(inet_pton_with_scope);
+
void inet_proto_csum_replace4(__sum16 *sum, struct sk_buff *skb,
__be32 from, __be32 to, bool pseudohdr)
{
diff --git a/net/dcb/dcbnl.c b/net/dcb/dcbnl.c
index 3202d75..9310612 100644
--- a/net/dcb/dcbnl.c
+++ b/net/dcb/dcbnl.c
@@ -245,8 +245,7 @@
return -EOPNOTSUPP;
ret = nla_parse_nested(data, DCB_PFC_UP_ATTR_MAX,
- tb[DCB_ATTR_PFC_CFG],
- dcbnl_pfc_up_nest);
+ tb[DCB_ATTR_PFC_CFG], dcbnl_pfc_up_nest, NULL);
if (ret)
return ret;
@@ -304,7 +303,7 @@
return -EOPNOTSUPP;
ret = nla_parse_nested(data, DCB_CAP_ATTR_MAX, tb[DCB_ATTR_CAP],
- dcbnl_cap_nest);
+ dcbnl_cap_nest, NULL);
if (ret)
return ret;
@@ -348,7 +347,7 @@
return -EOPNOTSUPP;
ret = nla_parse_nested(data, DCB_NUMTCS_ATTR_MAX, tb[DCB_ATTR_NUMTCS],
- dcbnl_numtcs_nest);
+ dcbnl_numtcs_nest, NULL);
if (ret)
return ret;
@@ -393,7 +392,7 @@
return -EOPNOTSUPP;
ret = nla_parse_nested(data, DCB_NUMTCS_ATTR_MAX, tb[DCB_ATTR_NUMTCS],
- dcbnl_numtcs_nest);
+ dcbnl_numtcs_nest, NULL);
if (ret)
return ret;
@@ -452,7 +451,7 @@
return -EINVAL;
ret = nla_parse_nested(app_tb, DCB_APP_ATTR_MAX, tb[DCB_ATTR_APP],
- dcbnl_app_nest);
+ dcbnl_app_nest, NULL);
if (ret)
return ret;
@@ -520,7 +519,7 @@
return -EINVAL;
ret = nla_parse_nested(app_tb, DCB_APP_ATTR_MAX, tb[DCB_ATTR_APP],
- dcbnl_app_nest);
+ dcbnl_app_nest, NULL);
if (ret)
return ret;
@@ -577,8 +576,8 @@
!netdev->dcbnl_ops->getpgbwgcfgrx)
return -EOPNOTSUPP;
- ret = nla_parse_nested(pg_tb, DCB_PG_ATTR_MAX,
- tb[DCB_ATTR_PG_CFG], dcbnl_pg_nest);
+ ret = nla_parse_nested(pg_tb, DCB_PG_ATTR_MAX, tb[DCB_ATTR_PG_CFG],
+ dcbnl_pg_nest, NULL);
if (ret)
return ret;
@@ -597,8 +596,8 @@
data = pg_tb[DCB_PG_ATTR_TC_ALL];
else
data = pg_tb[i];
- ret = nla_parse_nested(param_tb, DCB_TC_ATTR_PARAM_MAX,
- data, dcbnl_tc_param_nest);
+ ret = nla_parse_nested(param_tb, DCB_TC_ATTR_PARAM_MAX, data,
+ dcbnl_tc_param_nest, NULL);
if (ret)
goto err_pg;
@@ -735,8 +734,7 @@
return -EOPNOTSUPP;
ret = nla_parse_nested(data, DCB_PFC_UP_ATTR_MAX,
- tb[DCB_ATTR_PFC_CFG],
- dcbnl_pfc_up_nest);
+ tb[DCB_ATTR_PFC_CFG], dcbnl_pfc_up_nest, NULL);
if (ret)
return ret;
@@ -791,8 +789,8 @@
!netdev->dcbnl_ops->setpgbwgcfgrx)
return -EOPNOTSUPP;
- ret = nla_parse_nested(pg_tb, DCB_PG_ATTR_MAX,
- tb[DCB_ATTR_PG_CFG], dcbnl_pg_nest);
+ ret = nla_parse_nested(pg_tb, DCB_PG_ATTR_MAX, tb[DCB_ATTR_PG_CFG],
+ dcbnl_pg_nest, NULL);
if (ret)
return ret;
@@ -801,7 +799,7 @@
continue;
ret = nla_parse_nested(param_tb, DCB_TC_ATTR_PARAM_MAX,
- pg_tb[i], dcbnl_tc_param_nest);
+ pg_tb[i], dcbnl_tc_param_nest, NULL);
if (ret)
return ret;
@@ -889,8 +887,8 @@
!netdev->dcbnl_ops->getbcncfg)
return -EOPNOTSUPP;
- ret = nla_parse_nested(bcn_tb, DCB_BCN_ATTR_MAX,
- tb[DCB_ATTR_BCN], dcbnl_bcn_nest);
+ ret = nla_parse_nested(bcn_tb, DCB_BCN_ATTR_MAX, tb[DCB_ATTR_BCN],
+ dcbnl_bcn_nest, NULL);
if (ret)
return ret;
@@ -948,9 +946,8 @@
!netdev->dcbnl_ops->setbcnrp)
return -EOPNOTSUPP;
- ret = nla_parse_nested(data, DCB_BCN_ATTR_MAX,
- tb[DCB_ATTR_BCN],
- dcbnl_pfc_up_nest);
+ ret = nla_parse_nested(data, DCB_BCN_ATTR_MAX, tb[DCB_ATTR_BCN],
+ dcbnl_pfc_up_nest, NULL);
if (ret)
return ret;
@@ -1424,8 +1421,8 @@
if (!tb[DCB_ATTR_IEEE])
return -EINVAL;
- err = nla_parse_nested(ieee, DCB_ATTR_IEEE_MAX,
- tb[DCB_ATTR_IEEE], dcbnl_ieee_policy);
+ err = nla_parse_nested(ieee, DCB_ATTR_IEEE_MAX, tb[DCB_ATTR_IEEE],
+ dcbnl_ieee_policy, NULL);
if (err)
return err;
@@ -1508,8 +1505,8 @@
if (!tb[DCB_ATTR_IEEE])
return -EINVAL;
- err = nla_parse_nested(ieee, DCB_ATTR_IEEE_MAX,
- tb[DCB_ATTR_IEEE], dcbnl_ieee_policy);
+ err = nla_parse_nested(ieee, DCB_ATTR_IEEE_MAX, tb[DCB_ATTR_IEEE],
+ dcbnl_ieee_policy, NULL);
if (err)
return err;
@@ -1581,8 +1578,8 @@
if (!tb[DCB_ATTR_FEATCFG])
return -EINVAL;
- ret = nla_parse_nested(data, DCB_FEATCFG_ATTR_MAX, tb[DCB_ATTR_FEATCFG],
- dcbnl_featcfg_nest);
+ ret = nla_parse_nested(data, DCB_FEATCFG_ATTR_MAX,
+ tb[DCB_ATTR_FEATCFG], dcbnl_featcfg_nest, NULL);
if (ret)
return ret;
@@ -1625,8 +1622,8 @@
if (!tb[DCB_ATTR_FEATCFG])
return -EINVAL;
- ret = nla_parse_nested(data, DCB_FEATCFG_ATTR_MAX, tb[DCB_ATTR_FEATCFG],
- dcbnl_featcfg_nest);
+ ret = nla_parse_nested(data, DCB_FEATCFG_ATTR_MAX,
+ tb[DCB_ATTR_FEATCFG], dcbnl_featcfg_nest, NULL);
if (ret)
goto err;
@@ -1699,7 +1696,8 @@
[DCB_CMD_CEE_GET] = { RTM_GETDCB, dcbnl_cee_get },
};
-static int dcb_doit(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int dcb_doit(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct net_device *netdev;
@@ -1715,7 +1713,7 @@
return -EPERM;
ret = nlmsg_parse(nlh, sizeof(*dcb), tb, DCB_ATTR_MAX,
- dcbnl_rtnl_policy);
+ dcbnl_rtnl_policy, extack);
if (ret < 0)
return ret;
diff --git a/net/decnet/af_decnet.c b/net/decnet/af_decnet.c
index 7de5b40..9afa2a5 100644
--- a/net/decnet/af_decnet.c
+++ b/net/decnet/af_decnet.c
@@ -132,6 +132,7 @@
#include <net/neighbour.h>
#include <net/dst.h>
#include <net/fib_rules.h>
+#include <net/tcp.h>
#include <net/dn.h>
#include <net/dn_nsp.h>
#include <net/dn_dev.h>
@@ -1469,18 +1470,18 @@
case DSO_NODELAY:
if (optlen != sizeof(int))
return -EINVAL;
- if (scp->nonagle == 2)
+ if (scp->nonagle == TCP_NAGLE_CORK)
return -EINVAL;
- scp->nonagle = (u.val == 0) ? 0 : 1;
+ scp->nonagle = (u.val == 0) ? 0 : TCP_NAGLE_OFF;
/* if (scp->nonagle == 1) { Push pending frames } */
break;
case DSO_CORK:
if (optlen != sizeof(int))
return -EINVAL;
- if (scp->nonagle == 1)
+ if (scp->nonagle == TCP_NAGLE_OFF)
return -EINVAL;
- scp->nonagle = (u.val == 0) ? 0 : 2;
+ scp->nonagle = (u.val == 0) ? 0 : TCP_NAGLE_CORK;
/* if (scp->nonagle == 0) { Push pending frames } */
break;
@@ -1608,14 +1609,14 @@
case DSO_NODELAY:
if (r_len > sizeof(int))
r_len = sizeof(int);
- val = (scp->nonagle == 1);
+ val = (scp->nonagle == TCP_NAGLE_OFF);
r_data = &val;
break;
case DSO_CORK:
if (r_len > sizeof(int))
r_len = sizeof(int);
- val = (scp->nonagle == 2);
+ val = (scp->nonagle == TCP_NAGLE_CORK);
r_data = &val;
break;
diff --git a/net/decnet/dn_dev.c b/net/decnet/dn_dev.c
index 8fdd9f4..9017a9a 100644
--- a/net/decnet/dn_dev.c
+++ b/net/decnet/dn_dev.c
@@ -565,7 +565,8 @@
[IFA_FLAGS] = { .type = NLA_U32 },
};
-static int dn_nl_deladdr(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int dn_nl_deladdr(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct nlattr *tb[IFA_MAX+1];
@@ -581,7 +582,8 @@
if (!net_eq(net, &init_net))
goto errout;
- err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, dn_ifa_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, dn_ifa_policy,
+ extack);
if (err < 0)
goto errout;
@@ -609,7 +611,8 @@
return err;
}
-static int dn_nl_newaddr(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int dn_nl_newaddr(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct nlattr *tb[IFA_MAX+1];
@@ -625,7 +628,8 @@
if (!net_eq(net, &init_net))
return -EINVAL;
- err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, dn_ifa_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, dn_ifa_policy,
+ extack);
if (err < 0)
return err;
diff --git a/net/decnet/dn_fib.c b/net/decnet/dn_fib.c
index 7af0ba61..f9058eb 100644
--- a/net/decnet/dn_fib.c
+++ b/net/decnet/dn_fib.c
@@ -501,7 +501,8 @@
return table;
}
-static int dn_fib_rtm_delroute(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int dn_fib_rtm_delroute(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct dn_fib_table *tb;
@@ -515,7 +516,8 @@
if (!net_eq(net, &init_net))
return -EINVAL;
- err = nlmsg_parse(nlh, sizeof(*r), attrs, RTA_MAX, rtm_dn_policy);
+ err = nlmsg_parse(nlh, sizeof(*r), attrs, RTA_MAX, rtm_dn_policy,
+ extack);
if (err < 0)
return err;
@@ -526,7 +528,8 @@
return tb->delete(tb, r, attrs, nlh, &NETLINK_CB(skb));
}
-static int dn_fib_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int dn_fib_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct dn_fib_table *tb;
@@ -540,7 +543,8 @@
if (!net_eq(net, &init_net))
return -EINVAL;
- err = nlmsg_parse(nlh, sizeof(*r), attrs, RTA_MAX, rtm_dn_policy);
+ err = nlmsg_parse(nlh, sizeof(*r), attrs, RTA_MAX, rtm_dn_policy,
+ extack);
if (err < 0)
return err;
diff --git a/net/decnet/dn_route.c b/net/decnet/dn_route.c
index b1dc096..4b9518a 100644
--- a/net/decnet/dn_route.c
+++ b/net/decnet/dn_route.c
@@ -1640,7 +1640,8 @@
/*
* This is called by both endnodes and routers now.
*/
-static int dn_cache_getroute(struct sk_buff *in_skb, struct nlmsghdr *nlh)
+static int dn_cache_getroute(struct sk_buff *in_skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(in_skb->sk);
struct rtmsg *rtm = nlmsg_data(nlh);
@@ -1654,7 +1655,8 @@
if (!net_eq(net, &init_net))
return -EINVAL;
- err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_dn_policy);
+ err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_dn_policy,
+ extack);
if (err < 0)
return err;
diff --git a/net/decnet/netfilter/dn_rtmsg.c b/net/decnet/netfilter/dn_rtmsg.c
index 85f2fdc..1ed81ac 100644
--- a/net/decnet/netfilter/dn_rtmsg.c
+++ b/net/decnet/netfilter/dn_rtmsg.c
@@ -96,7 +96,7 @@
}
-#define RCV_SKB_FAIL(err) do { netlink_ack(skb, nlh, (err)); return; } while (0)
+#define RCV_SKB_FAIL(err) do { netlink_ack(skb, nlh, (err), NULL); return; } while (0)
static inline void dnrmg_receive_user_skb(struct sk_buff *skb)
{
@@ -134,7 +134,7 @@
return -ENOMEM;
}
- rv = nf_register_hook(&dnrmg_ops);
+ rv = nf_register_net_hook(&init_net, &dnrmg_ops);
if (rv) {
netlink_kernel_release(dnrmg);
}
@@ -144,7 +144,7 @@
static void __exit dn_rtmsg_fini(void)
{
- nf_unregister_hook(&dnrmg_ops);
+ nf_unregister_net_hook(&init_net, &dnrmg_ops);
netlink_kernel_release(dnrmg);
}
diff --git a/net/dsa/Kconfig b/net/dsa/Kconfig
index 9649238..81a0868 100644
--- a/net/dsa/Kconfig
+++ b/net/dsa/Kconfig
@@ -6,7 +6,7 @@
config NET_DSA
tristate "Distributed Switch Architecture"
- depends on HAVE_NET_DSA
+ depends on HAVE_NET_DSA && MAY_USE_DEVLINK
select NET_SWITCHDEV
select PHYLIB
---help---
@@ -31,4 +31,10 @@
config NET_DSA_TAG_QCA
bool
+config NET_DSA_TAG_MTK
+ bool
+
+config NET_DSA_TAG_LAN9303
+ bool
+
endif
diff --git a/net/dsa/Makefile b/net/dsa/Makefile
index 31d3437..0b747d7 100644
--- a/net/dsa/Makefile
+++ b/net/dsa/Makefile
@@ -1,6 +1,6 @@
# the core
obj-$(CONFIG_NET_DSA) += dsa_core.o
-dsa_core-y += dsa.o slave.o dsa2.o switch.o
+dsa_core-y += dsa.o slave.o dsa2.o switch.o legacy.o
# tagging formats
dsa_core-$(CONFIG_NET_DSA_TAG_BRCM) += tag_brcm.o
@@ -8,3 +8,5 @@
dsa_core-$(CONFIG_NET_DSA_TAG_EDSA) += tag_edsa.o
dsa_core-$(CONFIG_NET_DSA_TAG_TRAILER) += tag_trailer.o
dsa_core-$(CONFIG_NET_DSA_TAG_QCA) += tag_qca.o
+dsa_core-$(CONFIG_NET_DSA_TAG_MTK) += tag_mtk.o
+dsa_core-$(CONFIG_NET_DSA_TAG_LAN9303) += tag_lan9303.o
diff --git a/net/dsa/dsa.c b/net/dsa/dsa.c
index b6d4f6a..26130ae 100644
--- a/net/dsa/dsa.c
+++ b/net/dsa/dsa.c
@@ -14,15 +14,17 @@
#include <linux/platform_device.h>
#include <linux/slab.h>
#include <linux/module.h>
-#include <net/dsa.h>
#include <linux/of.h>
#include <linux/of_mdio.h>
#include <linux/of_platform.h>
#include <linux/of_net.h>
#include <linux/of_gpio.h>
+#include <linux/netdevice.h>
#include <linux/sysfs.h>
#include <linux/phy_fixed.h>
#include <linux/gpio/consumer.h>
+#include <linux/etherdevice.h>
+#include <net/dsa.h>
#include "dsa_priv.h"
static struct sk_buff *dsa_slave_notag_xmit(struct sk_buff *skb,
@@ -53,62 +55,15 @@
#ifdef CONFIG_NET_DSA_TAG_QCA
[DSA_TAG_PROTO_QCA] = &qca_netdev_ops,
#endif
+#ifdef CONFIG_NET_DSA_TAG_MTK
+ [DSA_TAG_PROTO_MTK] = &mtk_netdev_ops,
+#endif
+#ifdef CONFIG_NET_DSA_TAG_LAN9303
+ [DSA_TAG_PROTO_LAN9303] = &lan9303_netdev_ops,
+#endif
[DSA_TAG_PROTO_NONE] = &none_ops,
};
-/* switch driver registration ***********************************************/
-static DEFINE_MUTEX(dsa_switch_drivers_mutex);
-static LIST_HEAD(dsa_switch_drivers);
-
-void register_switch_driver(struct dsa_switch_driver *drv)
-{
- mutex_lock(&dsa_switch_drivers_mutex);
- list_add_tail(&drv->list, &dsa_switch_drivers);
- mutex_unlock(&dsa_switch_drivers_mutex);
-}
-EXPORT_SYMBOL_GPL(register_switch_driver);
-
-void unregister_switch_driver(struct dsa_switch_driver *drv)
-{
- mutex_lock(&dsa_switch_drivers_mutex);
- list_del_init(&drv->list);
- mutex_unlock(&dsa_switch_drivers_mutex);
-}
-EXPORT_SYMBOL_GPL(unregister_switch_driver);
-
-static const struct dsa_switch_ops *
-dsa_switch_probe(struct device *parent, struct device *host_dev, int sw_addr,
- const char **_name, void **priv)
-{
- const struct dsa_switch_ops *ret;
- struct list_head *list;
- const char *name;
-
- ret = NULL;
- name = NULL;
-
- mutex_lock(&dsa_switch_drivers_mutex);
- list_for_each(list, &dsa_switch_drivers) {
- const struct dsa_switch_ops *ops;
- struct dsa_switch_driver *drv;
-
- drv = list_entry(list, struct dsa_switch_driver, list);
- ops = drv->ops;
-
- name = ops->probe(parent, host_dev, sw_addr, priv);
- if (name != NULL) {
- ret = ops;
- break;
- }
- }
- mutex_unlock(&dsa_switch_drivers_mutex);
-
- *_name = name;
-
- return ret;
-}
-
-/* basic switch operations **************************************************/
int dsa_cpu_dsa_setup(struct dsa_switch *ds, struct device *dev,
struct dsa_port *dport, int port)
{
@@ -140,23 +95,6 @@
return 0;
}
-static int dsa_cpu_dsa_setups(struct dsa_switch *ds, struct device *dev)
-{
- struct dsa_port *dport;
- int ret, port;
-
- for (port = 0; port < ds->num_ports; port++) {
- if (!(dsa_is_cpu_port(ds, port) || dsa_is_dsa_port(ds, port)))
- continue;
-
- dport = &ds->ports[port];
- ret = dsa_cpu_dsa_setup(ds, dev, dport, port);
- if (ret)
- return ret;
- }
- return 0;
-}
-
const struct dsa_device_ops *dsa_resolve_tag_protocol(int tag_protocol)
{
const struct dsa_device_ops *ops;
@@ -206,168 +144,6 @@
master->ethtool_ops = ds->dst->master_orig_ethtool_ops;
}
-static int dsa_switch_setup_one(struct dsa_switch *ds, struct device *parent)
-{
- const struct dsa_switch_ops *ops = ds->ops;
- struct dsa_switch_tree *dst = ds->dst;
- struct dsa_chip_data *cd = ds->cd;
- bool valid_name_found = false;
- int index = ds->index;
- int i, ret;
-
- /*
- * Validate supplied switch configuration.
- */
- for (i = 0; i < ds->num_ports; i++) {
- char *name;
-
- name = cd->port_names[i];
- if (name == NULL)
- continue;
-
- if (!strcmp(name, "cpu")) {
- if (dst->cpu_switch) {
- netdev_err(dst->master_netdev,
- "multiple cpu ports?!\n");
- return -EINVAL;
- }
- dst->cpu_switch = ds;
- dst->cpu_port = i;
- ds->cpu_port_mask |= 1 << i;
- } else if (!strcmp(name, "dsa")) {
- ds->dsa_port_mask |= 1 << i;
- } else {
- ds->enabled_port_mask |= 1 << i;
- }
- valid_name_found = true;
- }
-
- if (!valid_name_found && i == ds->num_ports)
- return -EINVAL;
-
- /* Make the built-in MII bus mask match the number of ports,
- * switch drivers can override this later
- */
- ds->phys_mii_mask = ds->enabled_port_mask;
-
- /*
- * If the CPU connects to this switch, set the switch tree
- * tagging protocol to the preferred tagging format of this
- * switch.
- */
- if (dst->cpu_switch == ds) {
- enum dsa_tag_protocol tag_protocol;
-
- tag_protocol = ops->get_tag_protocol(ds);
- dst->tag_ops = dsa_resolve_tag_protocol(tag_protocol);
- if (IS_ERR(dst->tag_ops))
- return PTR_ERR(dst->tag_ops);
-
- dst->rcv = dst->tag_ops->rcv;
- }
-
- memcpy(ds->rtable, cd->rtable, sizeof(ds->rtable));
-
- /*
- * Do basic register setup.
- */
- ret = ops->setup(ds);
- if (ret < 0)
- return ret;
-
- ret = dsa_switch_register_notifier(ds);
- if (ret)
- return ret;
-
- if (ops->set_addr) {
- ret = ops->set_addr(ds, dst->master_netdev->dev_addr);
- if (ret < 0)
- return ret;
- }
-
- if (!ds->slave_mii_bus && ops->phy_read) {
- ds->slave_mii_bus = devm_mdiobus_alloc(parent);
- if (!ds->slave_mii_bus)
- return -ENOMEM;
- dsa_slave_mii_bus_init(ds);
-
- ret = mdiobus_register(ds->slave_mii_bus);
- if (ret < 0)
- return ret;
- }
-
- /*
- * Create network devices for physical switch ports.
- */
- for (i = 0; i < ds->num_ports; i++) {
- ds->ports[i].dn = cd->port_dn[i];
-
- if (!(ds->enabled_port_mask & (1 << i)))
- continue;
-
- ret = dsa_slave_create(ds, parent, i, cd->port_names[i]);
- if (ret < 0)
- netdev_err(dst->master_netdev, "[%d]: can't create dsa slave device for port %d(%s): %d\n",
- index, i, cd->port_names[i], ret);
- }
-
- /* Perform configuration of the CPU and DSA ports */
- ret = dsa_cpu_dsa_setups(ds, parent);
- if (ret < 0)
- netdev_err(dst->master_netdev, "[%d] : can't configure CPU and DSA ports\n",
- index);
-
- ret = dsa_cpu_port_ethtool_setup(ds);
- if (ret)
- return ret;
-
- return 0;
-}
-
-static struct dsa_switch *
-dsa_switch_setup(struct dsa_switch_tree *dst, int index,
- struct device *parent, struct device *host_dev)
-{
- struct dsa_chip_data *cd = dst->pd->chip + index;
- const struct dsa_switch_ops *ops;
- struct dsa_switch *ds;
- int ret;
- const char *name;
- void *priv;
-
- /*
- * Probe for switch model.
- */
- ops = dsa_switch_probe(parent, host_dev, cd->sw_addr, &name, &priv);
- if (!ops) {
- netdev_err(dst->master_netdev, "[%d]: could not detect attached switch\n",
- index);
- return ERR_PTR(-EINVAL);
- }
- netdev_info(dst->master_netdev, "[%d]: detected a %s switch\n",
- index, name);
-
-
- /*
- * Allocate and initialise switch state.
- */
- ds = dsa_switch_alloc(parent, DSA_MAX_PORTS);
- if (!ds)
- return ERR_PTR(-ENOMEM);
-
- ds->dst = dst;
- ds->index = index;
- ds->cd = cd;
- ds->ops = ops;
- ds->priv = priv;
-
- ret = dsa_switch_setup_one(ds, parent);
- if (ret)
- return ERR_PTR(ret);
-
- return ds;
-}
-
void dsa_cpu_dsa_destroy(struct dsa_port *port)
{
struct device_node *port_dn = port->dn;
@@ -376,86 +152,6 @@
of_phy_deregister_fixed_link(port_dn);
}
-static void dsa_switch_destroy(struct dsa_switch *ds)
-{
- int port;
-
- /* Destroy network devices for physical switch ports. */
- for (port = 0; port < ds->num_ports; port++) {
- if (!(ds->enabled_port_mask & (1 << port)))
- continue;
-
- if (!ds->ports[port].netdev)
- continue;
-
- dsa_slave_destroy(ds->ports[port].netdev);
- }
-
- /* Disable configuration of the CPU and DSA ports */
- for (port = 0; port < ds->num_ports; port++) {
- if (!(dsa_is_cpu_port(ds, port) || dsa_is_dsa_port(ds, port)))
- continue;
- dsa_cpu_dsa_destroy(&ds->ports[port]);
-
- /* Clearing a bit which is not set does no harm */
- ds->cpu_port_mask |= ~(1 << port);
- ds->dsa_port_mask |= ~(1 << port);
- }
-
- if (ds->slave_mii_bus && ds->ops->phy_read)
- mdiobus_unregister(ds->slave_mii_bus);
-
- dsa_switch_unregister_notifier(ds);
-}
-
-#ifdef CONFIG_PM_SLEEP
-int dsa_switch_suspend(struct dsa_switch *ds)
-{
- int i, ret = 0;
-
- /* Suspend slave network devices */
- for (i = 0; i < ds->num_ports; i++) {
- if (!dsa_is_port_initialized(ds, i))
- continue;
-
- ret = dsa_slave_suspend(ds->ports[i].netdev);
- if (ret)
- return ret;
- }
-
- if (ds->ops->suspend)
- ret = ds->ops->suspend(ds);
-
- return ret;
-}
-EXPORT_SYMBOL_GPL(dsa_switch_suspend);
-
-int dsa_switch_resume(struct dsa_switch *ds)
-{
- int i, ret = 0;
-
- if (ds->ops->resume)
- ret = ds->ops->resume(ds);
-
- if (ret)
- return ret;
-
- /* Resume slave network devices */
- for (i = 0; i < ds->num_ports; i++) {
- if (!dsa_is_port_initialized(ds, i))
- continue;
-
- ret = dsa_slave_resume(ds->ports[i].netdev);
- if (ret)
- return ret;
- }
-
- return 0;
-}
-EXPORT_SYMBOL_GPL(dsa_switch_resume);
-#endif
-
-/* platform driver init and cleanup *****************************************/
static int dev_is_class(struct device *dev, void *class)
{
if (dev->class != NULL && !strcmp(dev->class->name, class))
@@ -474,24 +170,6 @@
return device_find_child(parent, class, dev_is_class);
}
-struct mii_bus *dsa_host_dev_to_mii_bus(struct device *dev)
-{
- struct device *d;
-
- d = dev_find_class(dev, "mdio_bus");
- if (d != NULL) {
- struct mii_bus *bus;
-
- bus = to_mii_bus(d);
- put_device(d);
-
- return bus;
- }
-
- return NULL;
-}
-EXPORT_SYMBOL_GPL(dsa_host_dev_to_mii_bus);
-
struct net_device *dsa_dev_to_net_device(struct device *dev)
{
struct device *d;
@@ -511,398 +189,38 @@
}
EXPORT_SYMBOL_GPL(dsa_dev_to_net_device);
-#ifdef CONFIG_OF
-static int dsa_of_setup_routing_table(struct dsa_platform_data *pd,
- struct dsa_chip_data *cd,
- int chip_index, int port_index,
- struct device_node *link)
-{
- const __be32 *reg;
- int link_sw_addr;
- struct device_node *parent_sw;
- int len;
-
- parent_sw = of_get_parent(link);
- if (!parent_sw)
- return -EINVAL;
-
- reg = of_get_property(parent_sw, "reg", &len);
- if (!reg || (len != sizeof(*reg) * 2))
- return -EINVAL;
-
- /*
- * Get the destination switch number from the second field of its 'reg'
- * property, i.e. for "reg = <0x19 1>" sw_addr is '1'.
- */
- link_sw_addr = be32_to_cpup(reg + 1);
-
- if (link_sw_addr >= pd->nr_chips)
- return -EINVAL;
-
- cd->rtable[link_sw_addr] = port_index;
-
- return 0;
-}
-
-static int dsa_of_probe_links(struct dsa_platform_data *pd,
- struct dsa_chip_data *cd,
- int chip_index, int port_index,
- struct device_node *port,
- const char *port_name)
-{
- struct device_node *link;
- int link_index;
- int ret;
-
- for (link_index = 0;; link_index++) {
- link = of_parse_phandle(port, "link", link_index);
- if (!link)
- break;
-
- if (!strcmp(port_name, "dsa") && pd->nr_chips > 1) {
- ret = dsa_of_setup_routing_table(pd, cd, chip_index,
- port_index, link);
- if (ret)
- return ret;
- }
- }
- return 0;
-}
-
-static void dsa_of_free_platform_data(struct dsa_platform_data *pd)
-{
- int i;
- int port_index;
-
- for (i = 0; i < pd->nr_chips; i++) {
- port_index = 0;
- while (port_index < DSA_MAX_PORTS) {
- kfree(pd->chip[i].port_names[port_index]);
- port_index++;
- }
-
- /* Drop our reference to the MDIO bus device */
- if (pd->chip[i].host_dev)
- put_device(pd->chip[i].host_dev);
- }
- kfree(pd->chip);
-}
-
-static int dsa_of_probe(struct device *dev)
-{
- struct device_node *np = dev->of_node;
- struct device_node *child, *mdio, *ethernet, *port;
- struct mii_bus *mdio_bus, *mdio_bus_switch;
- struct net_device *ethernet_dev;
- struct dsa_platform_data *pd;
- struct dsa_chip_data *cd;
- const char *port_name;
- int chip_index, port_index;
- const unsigned int *sw_addr, *port_reg;
- u32 eeprom_len;
- int ret;
-
- mdio = of_parse_phandle(np, "dsa,mii-bus", 0);
- if (!mdio)
- return -EINVAL;
-
- mdio_bus = of_mdio_find_bus(mdio);
- if (!mdio_bus)
- return -EPROBE_DEFER;
-
- ethernet = of_parse_phandle(np, "dsa,ethernet", 0);
- if (!ethernet) {
- ret = -EINVAL;
- goto out_put_mdio;
- }
-
- ethernet_dev = of_find_net_device_by_node(ethernet);
- if (!ethernet_dev) {
- ret = -EPROBE_DEFER;
- goto out_put_mdio;
- }
-
- pd = kzalloc(sizeof(*pd), GFP_KERNEL);
- if (!pd) {
- ret = -ENOMEM;
- goto out_put_ethernet;
- }
-
- dev->platform_data = pd;
- pd->of_netdev = ethernet_dev;
- pd->nr_chips = of_get_available_child_count(np);
- if (pd->nr_chips > DSA_MAX_SWITCHES)
- pd->nr_chips = DSA_MAX_SWITCHES;
-
- pd->chip = kcalloc(pd->nr_chips, sizeof(struct dsa_chip_data),
- GFP_KERNEL);
- if (!pd->chip) {
- ret = -ENOMEM;
- goto out_free;
- }
-
- chip_index = -1;
- for_each_available_child_of_node(np, child) {
- int i;
-
- chip_index++;
- cd = &pd->chip[chip_index];
-
- cd->of_node = child;
-
- /* Initialize the routing table */
- for (i = 0; i < DSA_MAX_SWITCHES; ++i)
- cd->rtable[i] = DSA_RTABLE_NONE;
-
- /* When assigning the host device, increment its refcount */
- cd->host_dev = get_device(&mdio_bus->dev);
-
- sw_addr = of_get_property(child, "reg", NULL);
- if (!sw_addr)
- continue;
-
- cd->sw_addr = be32_to_cpup(sw_addr);
- if (cd->sw_addr >= PHY_MAX_ADDR)
- continue;
-
- if (!of_property_read_u32(child, "eeprom-length", &eeprom_len))
- cd->eeprom_len = eeprom_len;
-
- mdio = of_parse_phandle(child, "mii-bus", 0);
- if (mdio) {
- mdio_bus_switch = of_mdio_find_bus(mdio);
- if (!mdio_bus_switch) {
- ret = -EPROBE_DEFER;
- goto out_free_chip;
- }
-
- /* Drop the mdio_bus device ref, replacing the host
- * device with the mdio_bus_switch device, keeping
- * the refcount from of_mdio_find_bus() above.
- */
- put_device(cd->host_dev);
- cd->host_dev = &mdio_bus_switch->dev;
- }
-
- for_each_available_child_of_node(child, port) {
- port_reg = of_get_property(port, "reg", NULL);
- if (!port_reg)
- continue;
-
- port_index = be32_to_cpup(port_reg);
- if (port_index >= DSA_MAX_PORTS)
- break;
-
- port_name = of_get_property(port, "label", NULL);
- if (!port_name)
- continue;
-
- cd->port_dn[port_index] = port;
-
- cd->port_names[port_index] = kstrdup(port_name,
- GFP_KERNEL);
- if (!cd->port_names[port_index]) {
- ret = -ENOMEM;
- goto out_free_chip;
- }
-
- ret = dsa_of_probe_links(pd, cd, chip_index,
- port_index, port, port_name);
- if (ret)
- goto out_free_chip;
-
- }
- }
-
- /* The individual chips hold their own refcount on the mdio bus,
- * so drop ours */
- put_device(&mdio_bus->dev);
-
- return 0;
-
-out_free_chip:
- dsa_of_free_platform_data(pd);
-out_free:
- kfree(pd);
- dev->platform_data = NULL;
-out_put_ethernet:
- put_device(ðernet_dev->dev);
-out_put_mdio:
- put_device(&mdio_bus->dev);
- return ret;
-}
-
-static void dsa_of_remove(struct device *dev)
-{
- struct dsa_platform_data *pd = dev->platform_data;
-
- if (!dev->of_node)
- return;
-
- dsa_of_free_platform_data(pd);
- put_device(&pd->of_netdev->dev);
- kfree(pd);
-}
-#else
-static inline int dsa_of_probe(struct device *dev)
-{
- return 0;
-}
-
-static inline void dsa_of_remove(struct device *dev)
-{
-}
-#endif
-
-static int dsa_setup_dst(struct dsa_switch_tree *dst, struct net_device *dev,
- struct device *parent, struct dsa_platform_data *pd)
-{
- int i;
- unsigned configured = 0;
-
- dst->pd = pd;
- dst->master_netdev = dev;
- dst->cpu_port = -1;
-
- for (i = 0; i < pd->nr_chips; i++) {
- struct dsa_switch *ds;
-
- ds = dsa_switch_setup(dst, i, parent, pd->chip[i].host_dev);
- if (IS_ERR(ds)) {
- netdev_err(dev, "[%d]: couldn't create dsa switch instance (error %ld)\n",
- i, PTR_ERR(ds));
- continue;
- }
-
- dst->ds[i] = ds;
-
- ++configured;
- }
-
- /*
- * If no switch was found, exit cleanly
- */
- if (!configured)
- return -EPROBE_DEFER;
-
- /*
- * If we use a tagging format that doesn't have an ethertype
- * field, make sure that all packets from this point on get
- * sent to the tag format's receive function.
- */
- wmb();
- dev->dsa_ptr = (void *)dst;
-
- return 0;
-}
-
-static int dsa_probe(struct platform_device *pdev)
-{
- struct dsa_platform_data *pd = pdev->dev.platform_data;
- struct net_device *dev;
- struct dsa_switch_tree *dst;
- int ret;
-
- if (pdev->dev.of_node) {
- ret = dsa_of_probe(&pdev->dev);
- if (ret)
- return ret;
-
- pd = pdev->dev.platform_data;
- }
-
- if (pd == NULL || (pd->netdev == NULL && pd->of_netdev == NULL))
- return -EINVAL;
-
- if (pd->of_netdev) {
- dev = pd->of_netdev;
- dev_hold(dev);
- } else {
- dev = dsa_dev_to_net_device(pd->netdev);
- }
- if (dev == NULL) {
- ret = -EPROBE_DEFER;
- goto out;
- }
-
- if (dev->dsa_ptr != NULL) {
- dev_put(dev);
- ret = -EEXIST;
- goto out;
- }
-
- dst = devm_kzalloc(&pdev->dev, sizeof(*dst), GFP_KERNEL);
- if (dst == NULL) {
- dev_put(dev);
- ret = -ENOMEM;
- goto out;
- }
-
- platform_set_drvdata(pdev, dst);
-
- ret = dsa_setup_dst(dst, dev, &pdev->dev, pd);
- if (ret) {
- dev_put(dev);
- goto out;
- }
-
- return 0;
-
-out:
- dsa_of_remove(&pdev->dev);
-
- return ret;
-}
-
-static void dsa_remove_dst(struct dsa_switch_tree *dst)
-{
- int i;
-
- dst->master_netdev->dsa_ptr = NULL;
-
- /* If we used a tagging format that doesn't have an ethertype
- * field, make sure that all packets from this point get sent
- * without the tag and go through the regular receive path.
- */
- wmb();
-
- for (i = 0; i < dst->pd->nr_chips; i++) {
- struct dsa_switch *ds = dst->ds[i];
-
- if (ds)
- dsa_switch_destroy(ds);
- }
-
- dsa_cpu_port_ethtool_restore(dst->cpu_switch);
-
- dev_put(dst->master_netdev);
-}
-
-static int dsa_remove(struct platform_device *pdev)
-{
- struct dsa_switch_tree *dst = platform_get_drvdata(pdev);
-
- dsa_remove_dst(dst);
- dsa_of_remove(&pdev->dev);
-
- return 0;
-}
-
-static void dsa_shutdown(struct platform_device *pdev)
-{
-}
-
static int dsa_switch_rcv(struct sk_buff *skb, struct net_device *dev,
struct packet_type *pt, struct net_device *orig_dev)
{
struct dsa_switch_tree *dst = dev->dsa_ptr;
+ struct sk_buff *nskb = NULL;
if (unlikely(dst == NULL)) {
kfree_skb(skb);
return 0;
}
- return dst->rcv(skb, dev, pt, orig_dev);
+ skb = skb_unshare(skb, GFP_ATOMIC);
+ if (!skb)
+ return 0;
+
+ nskb = dst->rcv(skb, dev, pt, orig_dev);
+ if (!nskb) {
+ kfree_skb(skb);
+ return 0;
+ }
+
+ skb = nskb;
+ skb_push(skb, ETH_HLEN);
+ skb->pkt_type = PACKET_HOST;
+ skb->protocol = eth_type_trans(skb, skb->dev);
+
+ skb->dev->stats.rx_packets++;
+ skb->dev->stats.rx_bytes += skb->len;
+
+ netif_receive_skb(skb);
+
+ return 0;
}
static struct packet_type dsa_pack_type __read_mostly = {
@@ -910,59 +228,6 @@
.func = dsa_switch_rcv,
};
-#ifdef CONFIG_PM_SLEEP
-static int dsa_suspend(struct device *d)
-{
- struct platform_device *pdev = to_platform_device(d);
- struct dsa_switch_tree *dst = platform_get_drvdata(pdev);
- int i, ret = 0;
-
- for (i = 0; i < dst->pd->nr_chips; i++) {
- struct dsa_switch *ds = dst->ds[i];
-
- if (ds != NULL)
- ret = dsa_switch_suspend(ds);
- }
-
- return ret;
-}
-
-static int dsa_resume(struct device *d)
-{
- struct platform_device *pdev = to_platform_device(d);
- struct dsa_switch_tree *dst = platform_get_drvdata(pdev);
- int i, ret = 0;
-
- for (i = 0; i < dst->pd->nr_chips; i++) {
- struct dsa_switch *ds = dst->ds[i];
-
- if (ds != NULL)
- ret = dsa_switch_resume(ds);
- }
-
- return ret;
-}
-#endif
-
-static SIMPLE_DEV_PM_OPS(dsa_pm_ops, dsa_suspend, dsa_resume);
-
-static const struct of_device_id dsa_of_match_table[] = {
- { .compatible = "marvell,dsa", },
- {}
-};
-MODULE_DEVICE_TABLE(of, dsa_of_match_table);
-
-static struct platform_driver dsa_driver = {
- .probe = dsa_probe,
- .remove = dsa_remove,
- .shutdown = dsa_shutdown,
- .driver = {
- .name = "dsa",
- .of_match_table = dsa_of_match_table,
- .pm = &dsa_pm_ops,
- },
-};
-
static int __init dsa_init_module(void)
{
int rc;
@@ -971,7 +236,7 @@
if (rc)
return rc;
- rc = platform_driver_register(&dsa_driver);
+ rc = dsa_legacy_register();
if (rc)
return rc;
@@ -985,7 +250,7 @@
{
dsa_slave_unregister_notifier();
dev_remove_pack(&dsa_pack_type);
- platform_driver_unregister(&dsa_driver);
+ dsa_legacy_unregister();
}
module_exit(dsa_cleanup_module);
diff --git a/net/dsa/dsa2.c b/net/dsa/dsa2.c
index 737be64..033b3bf 100644
--- a/net/dsa/dsa2.c
+++ b/net/dsa/dsa2.c
@@ -13,16 +13,20 @@
#include <linux/device.h>
#include <linux/err.h>
#include <linux/list.h>
+#include <linux/netdevice.h>
#include <linux/slab.h>
#include <linux/rtnetlink.h>
-#include <net/dsa.h>
#include <linux/of.h>
#include <linux/of_net.h>
+#include <net/dsa.h>
#include "dsa_priv.h"
static LIST_HEAD(dsa_switch_trees);
static DEFINE_MUTEX(dsa2_mutex);
+static const struct devlink_ops dsa_devlink_ops = {
+};
+
static struct dsa_switch_tree *dsa_get_dst(u32 tree)
{
struct dsa_switch_tree *dst;
@@ -222,12 +226,18 @@
return err;
}
- return 0;
+ memset(&ds->ports[index].devlink_port, 0,
+ sizeof(ds->ports[index].devlink_port));
+
+ return devlink_port_register(ds->devlink,
+ &ds->ports[index].devlink_port,
+ index);
}
static void dsa_dsa_port_unapply(struct dsa_port *port, u32 index,
struct dsa_switch *ds)
{
+ devlink_port_unregister(&ds->ports[index].devlink_port);
dsa_cpu_dsa_destroy(port);
}
@@ -245,12 +255,17 @@
ds->cpu_port_mask |= BIT(index);
- return 0;
+ memset(&ds->ports[index].devlink_port, 0,
+ sizeof(ds->ports[index].devlink_port));
+ err = devlink_port_register(ds->devlink, &ds->ports[index].devlink_port,
+ index);
+ return err;
}
static void dsa_cpu_port_unapply(struct dsa_port *port, u32 index,
struct dsa_switch *ds)
{
+ devlink_port_unregister(&ds->ports[index].devlink_port);
dsa_cpu_dsa_destroy(port);
ds->cpu_port_mask &= ~BIT(index);
@@ -275,12 +290,23 @@
return err;
}
+ memset(&ds->ports[index].devlink_port, 0,
+ sizeof(ds->ports[index].devlink_port));
+ err = devlink_port_register(ds->devlink, &ds->ports[index].devlink_port,
+ index);
+ if (err)
+ return err;
+
+ devlink_port_type_eth_set(&ds->ports[index].devlink_port,
+ ds->ports[index].netdev);
+
return 0;
}
static void dsa_user_port_unapply(struct dsa_port *port, u32 index,
struct dsa_switch *ds)
{
+ devlink_port_unregister(&ds->ports[index].devlink_port);
if (ds->ports[index].netdev) {
dsa_slave_destroy(ds->ports[index].netdev);
ds->ports[index].netdev = NULL;
@@ -301,6 +327,17 @@
*/
ds->phys_mii_mask = ds->enabled_port_mask;
+ /* Add the switch to devlink before calling setup, so that setup can
+ * add dpipe tables
+ */
+ ds->devlink = devlink_alloc(&dsa_devlink_ops, 0);
+ if (!ds->devlink)
+ return -ENOMEM;
+
+ err = devlink_register(ds->devlink, ds->dev);
+ if (err)
+ return err;
+
err = ds->ops->setup(ds);
if (err < 0)
return err;
@@ -381,6 +418,13 @@
mdiobus_unregister(ds->slave_mii_bus);
dsa_switch_unregister_notifier(ds);
+
+ if (ds->devlink) {
+ devlink_unregister(ds->devlink);
+ devlink_free(ds->devlink);
+ ds->devlink = NULL;
+ }
+
}
static int dsa_dst_apply(struct dsa_switch_tree *dst)
diff --git a/net/dsa/dsa_priv.h b/net/dsa/dsa_priv.h
index 0706a51..f4a88e4 100644
--- a/net/dsa/dsa_priv.h
+++ b/net/dsa/dsa_priv.h
@@ -17,8 +17,9 @@
struct dsa_device_ops {
struct sk_buff *(*xmit)(struct sk_buff *skb, struct net_device *dev);
- int (*rcv)(struct sk_buff *skb, struct net_device *dev,
- struct packet_type *pt, struct net_device *orig_dev);
+ struct sk_buff *(*rcv)(struct sk_buff *skb, struct net_device *dev,
+ struct packet_type *pt,
+ struct net_device *orig_dev);
};
struct dsa_slave_priv {
@@ -54,6 +55,10 @@
int dsa_cpu_port_ethtool_setup(struct dsa_switch *ds);
void dsa_cpu_port_ethtool_restore(struct dsa_switch *ds);
+/* legacy.c */
+int dsa_legacy_register(void);
+void dsa_legacy_unregister(void);
+
/* slave.c */
extern const struct dsa_device_ops notag_netdev_ops;
void dsa_slave_mii_bus_init(struct dsa_switch *ds);
@@ -85,4 +90,10 @@
/* tag_qca.c */
extern const struct dsa_device_ops qca_netdev_ops;
+/* tag_mtk.c */
+extern const struct dsa_device_ops mtk_netdev_ops;
+
+/* tag_lan9303.c */
+extern const struct dsa_device_ops lan9303_netdev_ops;
+
#endif
diff --git a/net/dsa/legacy.c b/net/dsa/legacy.c
new file mode 100644
index 0000000..ad345c8
--- /dev/null
+++ b/net/dsa/legacy.c
@@ -0,0 +1,818 @@
+/*
+ * net/dsa/legacy.c - Hardware switch handling
+ * Copyright (c) 2008-2009 Marvell Semiconductor
+ * Copyright (c) 2013 Florian Fainelli <florian@openwrt.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/device.h>
+#include <linux/list.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/of_mdio.h>
+#include <linux/of_platform.h>
+#include <linux/of_net.h>
+#include <linux/netdevice.h>
+#include <linux/sysfs.h>
+#include <linux/phy_fixed.h>
+#include <linux/etherdevice.h>
+#include <net/dsa.h>
+#include "dsa_priv.h"
+
+/* switch driver registration ***********************************************/
+static DEFINE_MUTEX(dsa_switch_drivers_mutex);
+static LIST_HEAD(dsa_switch_drivers);
+
+void register_switch_driver(struct dsa_switch_driver *drv)
+{
+ mutex_lock(&dsa_switch_drivers_mutex);
+ list_add_tail(&drv->list, &dsa_switch_drivers);
+ mutex_unlock(&dsa_switch_drivers_mutex);
+}
+EXPORT_SYMBOL_GPL(register_switch_driver);
+
+void unregister_switch_driver(struct dsa_switch_driver *drv)
+{
+ mutex_lock(&dsa_switch_drivers_mutex);
+ list_del_init(&drv->list);
+ mutex_unlock(&dsa_switch_drivers_mutex);
+}
+EXPORT_SYMBOL_GPL(unregister_switch_driver);
+
+static const struct dsa_switch_ops *
+dsa_switch_probe(struct device *parent, struct device *host_dev, int sw_addr,
+ const char **_name, void **priv)
+{
+ const struct dsa_switch_ops *ret;
+ struct list_head *list;
+ const char *name;
+
+ ret = NULL;
+ name = NULL;
+
+ mutex_lock(&dsa_switch_drivers_mutex);
+ list_for_each(list, &dsa_switch_drivers) {
+ const struct dsa_switch_ops *ops;
+ struct dsa_switch_driver *drv;
+
+ drv = list_entry(list, struct dsa_switch_driver, list);
+ ops = drv->ops;
+
+ name = ops->probe(parent, host_dev, sw_addr, priv);
+ if (name != NULL) {
+ ret = ops;
+ break;
+ }
+ }
+ mutex_unlock(&dsa_switch_drivers_mutex);
+
+ *_name = name;
+
+ return ret;
+}
+
+/* basic switch operations **************************************************/
+static int dsa_cpu_dsa_setups(struct dsa_switch *ds, struct device *dev)
+{
+ struct dsa_port *dport;
+ int ret, port;
+
+ for (port = 0; port < ds->num_ports; port++) {
+ if (!(dsa_is_cpu_port(ds, port) || dsa_is_dsa_port(ds, port)))
+ continue;
+
+ dport = &ds->ports[port];
+ ret = dsa_cpu_dsa_setup(ds, dev, dport, port);
+ if (ret)
+ return ret;
+ }
+ return 0;
+}
+
+static int dsa_switch_setup_one(struct dsa_switch *ds, struct device *parent)
+{
+ const struct dsa_switch_ops *ops = ds->ops;
+ struct dsa_switch_tree *dst = ds->dst;
+ struct dsa_chip_data *cd = ds->cd;
+ bool valid_name_found = false;
+ int index = ds->index;
+ int i, ret;
+
+ /*
+ * Validate supplied switch configuration.
+ */
+ for (i = 0; i < ds->num_ports; i++) {
+ char *name;
+
+ name = cd->port_names[i];
+ if (name == NULL)
+ continue;
+
+ if (!strcmp(name, "cpu")) {
+ if (dst->cpu_switch) {
+ netdev_err(dst->master_netdev,
+ "multiple cpu ports?!\n");
+ return -EINVAL;
+ }
+ dst->cpu_switch = ds;
+ dst->cpu_port = i;
+ ds->cpu_port_mask |= 1 << i;
+ } else if (!strcmp(name, "dsa")) {
+ ds->dsa_port_mask |= 1 << i;
+ } else {
+ ds->enabled_port_mask |= 1 << i;
+ }
+ valid_name_found = true;
+ }
+
+ if (!valid_name_found && i == ds->num_ports)
+ return -EINVAL;
+
+ /* Make the built-in MII bus mask match the number of ports,
+ * switch drivers can override this later
+ */
+ ds->phys_mii_mask = ds->enabled_port_mask;
+
+ /*
+ * If the CPU connects to this switch, set the switch tree
+ * tagging protocol to the preferred tagging format of this
+ * switch.
+ */
+ if (dst->cpu_switch == ds) {
+ enum dsa_tag_protocol tag_protocol;
+
+ tag_protocol = ops->get_tag_protocol(ds);
+ dst->tag_ops = dsa_resolve_tag_protocol(tag_protocol);
+ if (IS_ERR(dst->tag_ops))
+ return PTR_ERR(dst->tag_ops);
+
+ dst->rcv = dst->tag_ops->rcv;
+ }
+
+ memcpy(ds->rtable, cd->rtable, sizeof(ds->rtable));
+
+ /*
+ * Do basic register setup.
+ */
+ ret = ops->setup(ds);
+ if (ret < 0)
+ return ret;
+
+ ret = dsa_switch_register_notifier(ds);
+ if (ret)
+ return ret;
+
+ if (ops->set_addr) {
+ ret = ops->set_addr(ds, dst->master_netdev->dev_addr);
+ if (ret < 0)
+ return ret;
+ }
+
+ if (!ds->slave_mii_bus && ops->phy_read) {
+ ds->slave_mii_bus = devm_mdiobus_alloc(parent);
+ if (!ds->slave_mii_bus)
+ return -ENOMEM;
+ dsa_slave_mii_bus_init(ds);
+
+ ret = mdiobus_register(ds->slave_mii_bus);
+ if (ret < 0)
+ return ret;
+ }
+
+ /*
+ * Create network devices for physical switch ports.
+ */
+ for (i = 0; i < ds->num_ports; i++) {
+ ds->ports[i].dn = cd->port_dn[i];
+
+ if (!(ds->enabled_port_mask & (1 << i)))
+ continue;
+
+ ret = dsa_slave_create(ds, parent, i, cd->port_names[i]);
+ if (ret < 0)
+ netdev_err(dst->master_netdev, "[%d]: can't create dsa slave device for port %d(%s): %d\n",
+ index, i, cd->port_names[i], ret);
+ }
+
+ /* Perform configuration of the CPU and DSA ports */
+ ret = dsa_cpu_dsa_setups(ds, parent);
+ if (ret < 0)
+ netdev_err(dst->master_netdev, "[%d] : can't configure CPU and DSA ports\n",
+ index);
+
+ ret = dsa_cpu_port_ethtool_setup(ds);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static struct dsa_switch *
+dsa_switch_setup(struct dsa_switch_tree *dst, int index,
+ struct device *parent, struct device *host_dev)
+{
+ struct dsa_chip_data *cd = dst->pd->chip + index;
+ const struct dsa_switch_ops *ops;
+ struct dsa_switch *ds;
+ int ret;
+ const char *name;
+ void *priv;
+
+ /*
+ * Probe for switch model.
+ */
+ ops = dsa_switch_probe(parent, host_dev, cd->sw_addr, &name, &priv);
+ if (!ops) {
+ netdev_err(dst->master_netdev, "[%d]: could not detect attached switch\n",
+ index);
+ return ERR_PTR(-EINVAL);
+ }
+ netdev_info(dst->master_netdev, "[%d]: detected a %s switch\n",
+ index, name);
+
+
+ /*
+ * Allocate and initialise switch state.
+ */
+ ds = dsa_switch_alloc(parent, DSA_MAX_PORTS);
+ if (!ds)
+ return ERR_PTR(-ENOMEM);
+
+ ds->dst = dst;
+ ds->index = index;
+ ds->cd = cd;
+ ds->ops = ops;
+ ds->priv = priv;
+
+ ret = dsa_switch_setup_one(ds, parent);
+ if (ret)
+ return ERR_PTR(ret);
+
+ return ds;
+}
+
+static void dsa_switch_destroy(struct dsa_switch *ds)
+{
+ int port;
+
+ /* Destroy network devices for physical switch ports. */
+ for (port = 0; port < ds->num_ports; port++) {
+ if (!(ds->enabled_port_mask & (1 << port)))
+ continue;
+
+ if (!ds->ports[port].netdev)
+ continue;
+
+ dsa_slave_destroy(ds->ports[port].netdev);
+ }
+
+ /* Disable configuration of the CPU and DSA ports */
+ for (port = 0; port < ds->num_ports; port++) {
+ if (!(dsa_is_cpu_port(ds, port) || dsa_is_dsa_port(ds, port)))
+ continue;
+ dsa_cpu_dsa_destroy(&ds->ports[port]);
+
+ /* Clearing a bit which is not set does no harm */
+ ds->cpu_port_mask |= ~(1 << port);
+ ds->dsa_port_mask |= ~(1 << port);
+ }
+
+ if (ds->slave_mii_bus && ds->ops->phy_read)
+ mdiobus_unregister(ds->slave_mii_bus);
+
+ dsa_switch_unregister_notifier(ds);
+}
+
+#ifdef CONFIG_PM_SLEEP
+int dsa_switch_suspend(struct dsa_switch *ds)
+{
+ int i, ret = 0;
+
+ /* Suspend slave network devices */
+ for (i = 0; i < ds->num_ports; i++) {
+ if (!dsa_is_port_initialized(ds, i))
+ continue;
+
+ ret = dsa_slave_suspend(ds->ports[i].netdev);
+ if (ret)
+ return ret;
+ }
+
+ if (ds->ops->suspend)
+ ret = ds->ops->suspend(ds);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(dsa_switch_suspend);
+
+int dsa_switch_resume(struct dsa_switch *ds)
+{
+ int i, ret = 0;
+
+ if (ds->ops->resume)
+ ret = ds->ops->resume(ds);
+
+ if (ret)
+ return ret;
+
+ /* Resume slave network devices */
+ for (i = 0; i < ds->num_ports; i++) {
+ if (!dsa_is_port_initialized(ds, i))
+ continue;
+
+ ret = dsa_slave_resume(ds->ports[i].netdev);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(dsa_switch_resume);
+#endif
+
+/* platform driver init and cleanup *****************************************/
+static int dev_is_class(struct device *dev, void *class)
+{
+ if (dev->class != NULL && !strcmp(dev->class->name, class))
+ return 1;
+
+ return 0;
+}
+
+static struct device *dev_find_class(struct device *parent, char *class)
+{
+ if (dev_is_class(parent, class)) {
+ get_device(parent);
+ return parent;
+ }
+
+ return device_find_child(parent, class, dev_is_class);
+}
+
+struct mii_bus *dsa_host_dev_to_mii_bus(struct device *dev)
+{
+ struct device *d;
+
+ d = dev_find_class(dev, "mdio_bus");
+ if (d != NULL) {
+ struct mii_bus *bus;
+
+ bus = to_mii_bus(d);
+ put_device(d);
+
+ return bus;
+ }
+
+ return NULL;
+}
+EXPORT_SYMBOL_GPL(dsa_host_dev_to_mii_bus);
+
+#ifdef CONFIG_OF
+static int dsa_of_setup_routing_table(struct dsa_platform_data *pd,
+ struct dsa_chip_data *cd,
+ int chip_index, int port_index,
+ struct device_node *link)
+{
+ const __be32 *reg;
+ int link_sw_addr;
+ struct device_node *parent_sw;
+ int len;
+
+ parent_sw = of_get_parent(link);
+ if (!parent_sw)
+ return -EINVAL;
+
+ reg = of_get_property(parent_sw, "reg", &len);
+ if (!reg || (len != sizeof(*reg) * 2))
+ return -EINVAL;
+
+ /*
+ * Get the destination switch number from the second field of its 'reg'
+ * property, i.e. for "reg = <0x19 1>" sw_addr is '1'.
+ */
+ link_sw_addr = be32_to_cpup(reg + 1);
+
+ if (link_sw_addr >= pd->nr_chips)
+ return -EINVAL;
+
+ cd->rtable[link_sw_addr] = port_index;
+
+ return 0;
+}
+
+static int dsa_of_probe_links(struct dsa_platform_data *pd,
+ struct dsa_chip_data *cd,
+ int chip_index, int port_index,
+ struct device_node *port,
+ const char *port_name)
+{
+ struct device_node *link;
+ int link_index;
+ int ret;
+
+ for (link_index = 0;; link_index++) {
+ link = of_parse_phandle(port, "link", link_index);
+ if (!link)
+ break;
+
+ if (!strcmp(port_name, "dsa") && pd->nr_chips > 1) {
+ ret = dsa_of_setup_routing_table(pd, cd, chip_index,
+ port_index, link);
+ if (ret)
+ return ret;
+ }
+ }
+ return 0;
+}
+
+static void dsa_of_free_platform_data(struct dsa_platform_data *pd)
+{
+ int i;
+ int port_index;
+
+ for (i = 0; i < pd->nr_chips; i++) {
+ port_index = 0;
+ while (port_index < DSA_MAX_PORTS) {
+ kfree(pd->chip[i].port_names[port_index]);
+ port_index++;
+ }
+
+ /* Drop our reference to the MDIO bus device */
+ if (pd->chip[i].host_dev)
+ put_device(pd->chip[i].host_dev);
+ }
+ kfree(pd->chip);
+}
+
+static int dsa_of_probe(struct device *dev)
+{
+ struct device_node *np = dev->of_node;
+ struct device_node *child, *mdio, *ethernet, *port;
+ struct mii_bus *mdio_bus, *mdio_bus_switch;
+ struct net_device *ethernet_dev;
+ struct dsa_platform_data *pd;
+ struct dsa_chip_data *cd;
+ const char *port_name;
+ int chip_index, port_index;
+ const unsigned int *sw_addr, *port_reg;
+ u32 eeprom_len;
+ int ret;
+
+ mdio = of_parse_phandle(np, "dsa,mii-bus", 0);
+ if (!mdio)
+ return -EINVAL;
+
+ mdio_bus = of_mdio_find_bus(mdio);
+ if (!mdio_bus)
+ return -EPROBE_DEFER;
+
+ ethernet = of_parse_phandle(np, "dsa,ethernet", 0);
+ if (!ethernet) {
+ ret = -EINVAL;
+ goto out_put_mdio;
+ }
+
+ ethernet_dev = of_find_net_device_by_node(ethernet);
+ if (!ethernet_dev) {
+ ret = -EPROBE_DEFER;
+ goto out_put_mdio;
+ }
+
+ pd = kzalloc(sizeof(*pd), GFP_KERNEL);
+ if (!pd) {
+ ret = -ENOMEM;
+ goto out_put_ethernet;
+ }
+
+ dev->platform_data = pd;
+ pd->of_netdev = ethernet_dev;
+ pd->nr_chips = of_get_available_child_count(np);
+ if (pd->nr_chips > DSA_MAX_SWITCHES)
+ pd->nr_chips = DSA_MAX_SWITCHES;
+
+ pd->chip = kcalloc(pd->nr_chips, sizeof(struct dsa_chip_data),
+ GFP_KERNEL);
+ if (!pd->chip) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
+
+ chip_index = -1;
+ for_each_available_child_of_node(np, child) {
+ int i;
+
+ chip_index++;
+ cd = &pd->chip[chip_index];
+
+ cd->of_node = child;
+
+ /* Initialize the routing table */
+ for (i = 0; i < DSA_MAX_SWITCHES; ++i)
+ cd->rtable[i] = DSA_RTABLE_NONE;
+
+ /* When assigning the host device, increment its refcount */
+ cd->host_dev = get_device(&mdio_bus->dev);
+
+ sw_addr = of_get_property(child, "reg", NULL);
+ if (!sw_addr)
+ continue;
+
+ cd->sw_addr = be32_to_cpup(sw_addr);
+ if (cd->sw_addr >= PHY_MAX_ADDR)
+ continue;
+
+ if (!of_property_read_u32(child, "eeprom-length", &eeprom_len))
+ cd->eeprom_len = eeprom_len;
+
+ mdio = of_parse_phandle(child, "mii-bus", 0);
+ if (mdio) {
+ mdio_bus_switch = of_mdio_find_bus(mdio);
+ if (!mdio_bus_switch) {
+ ret = -EPROBE_DEFER;
+ goto out_free_chip;
+ }
+
+ /* Drop the mdio_bus device ref, replacing the host
+ * device with the mdio_bus_switch device, keeping
+ * the refcount from of_mdio_find_bus() above.
+ */
+ put_device(cd->host_dev);
+ cd->host_dev = &mdio_bus_switch->dev;
+ }
+
+ for_each_available_child_of_node(child, port) {
+ port_reg = of_get_property(port, "reg", NULL);
+ if (!port_reg)
+ continue;
+
+ port_index = be32_to_cpup(port_reg);
+ if (port_index >= DSA_MAX_PORTS)
+ break;
+
+ port_name = of_get_property(port, "label", NULL);
+ if (!port_name)
+ continue;
+
+ cd->port_dn[port_index] = port;
+
+ cd->port_names[port_index] = kstrdup(port_name,
+ GFP_KERNEL);
+ if (!cd->port_names[port_index]) {
+ ret = -ENOMEM;
+ goto out_free_chip;
+ }
+
+ ret = dsa_of_probe_links(pd, cd, chip_index,
+ port_index, port, port_name);
+ if (ret)
+ goto out_free_chip;
+
+ }
+ }
+
+ /* The individual chips hold their own refcount on the mdio bus,
+ * so drop ours */
+ put_device(&mdio_bus->dev);
+
+ return 0;
+
+out_free_chip:
+ dsa_of_free_platform_data(pd);
+out_free:
+ kfree(pd);
+ dev->platform_data = NULL;
+out_put_ethernet:
+ put_device(ðernet_dev->dev);
+out_put_mdio:
+ put_device(&mdio_bus->dev);
+ return ret;
+}
+
+static void dsa_of_remove(struct device *dev)
+{
+ struct dsa_platform_data *pd = dev->platform_data;
+
+ if (!dev->of_node)
+ return;
+
+ dsa_of_free_platform_data(pd);
+ put_device(&pd->of_netdev->dev);
+ kfree(pd);
+}
+#else
+static inline int dsa_of_probe(struct device *dev)
+{
+ return 0;
+}
+
+static inline void dsa_of_remove(struct device *dev)
+{
+}
+#endif
+
+static int dsa_setup_dst(struct dsa_switch_tree *dst, struct net_device *dev,
+ struct device *parent, struct dsa_platform_data *pd)
+{
+ int i;
+ unsigned configured = 0;
+
+ dst->pd = pd;
+ dst->master_netdev = dev;
+ dst->cpu_port = -1;
+
+ for (i = 0; i < pd->nr_chips; i++) {
+ struct dsa_switch *ds;
+
+ ds = dsa_switch_setup(dst, i, parent, pd->chip[i].host_dev);
+ if (IS_ERR(ds)) {
+ netdev_err(dev, "[%d]: couldn't create dsa switch instance (error %ld)\n",
+ i, PTR_ERR(ds));
+ continue;
+ }
+
+ dst->ds[i] = ds;
+
+ ++configured;
+ }
+
+ /*
+ * If no switch was found, exit cleanly
+ */
+ if (!configured)
+ return -EPROBE_DEFER;
+
+ /*
+ * If we use a tagging format that doesn't have an ethertype
+ * field, make sure that all packets from this point on get
+ * sent to the tag format's receive function.
+ */
+ wmb();
+ dev->dsa_ptr = (void *)dst;
+
+ return 0;
+}
+
+static int dsa_probe(struct platform_device *pdev)
+{
+ struct dsa_platform_data *pd = pdev->dev.platform_data;
+ struct net_device *dev;
+ struct dsa_switch_tree *dst;
+ int ret;
+
+ if (pdev->dev.of_node) {
+ ret = dsa_of_probe(&pdev->dev);
+ if (ret)
+ return ret;
+
+ pd = pdev->dev.platform_data;
+ }
+
+ if (pd == NULL || (pd->netdev == NULL && pd->of_netdev == NULL))
+ return -EINVAL;
+
+ if (pd->of_netdev) {
+ dev = pd->of_netdev;
+ dev_hold(dev);
+ } else {
+ dev = dsa_dev_to_net_device(pd->netdev);
+ }
+ if (dev == NULL) {
+ ret = -EPROBE_DEFER;
+ goto out;
+ }
+
+ if (dev->dsa_ptr != NULL) {
+ dev_put(dev);
+ ret = -EEXIST;
+ goto out;
+ }
+
+ dst = devm_kzalloc(&pdev->dev, sizeof(*dst), GFP_KERNEL);
+ if (dst == NULL) {
+ dev_put(dev);
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ platform_set_drvdata(pdev, dst);
+
+ ret = dsa_setup_dst(dst, dev, &pdev->dev, pd);
+ if (ret) {
+ dev_put(dev);
+ goto out;
+ }
+
+ return 0;
+
+out:
+ dsa_of_remove(&pdev->dev);
+
+ return ret;
+}
+
+static void dsa_remove_dst(struct dsa_switch_tree *dst)
+{
+ int i;
+
+ dst->master_netdev->dsa_ptr = NULL;
+
+ /* If we used a tagging format that doesn't have an ethertype
+ * field, make sure that all packets from this point get sent
+ * without the tag and go through the regular receive path.
+ */
+ wmb();
+
+ for (i = 0; i < dst->pd->nr_chips; i++) {
+ struct dsa_switch *ds = dst->ds[i];
+
+ if (ds)
+ dsa_switch_destroy(ds);
+ }
+
+ dsa_cpu_port_ethtool_restore(dst->cpu_switch);
+
+ dev_put(dst->master_netdev);
+}
+
+static int dsa_remove(struct platform_device *pdev)
+{
+ struct dsa_switch_tree *dst = platform_get_drvdata(pdev);
+
+ dsa_remove_dst(dst);
+ dsa_of_remove(&pdev->dev);
+
+ return 0;
+}
+
+static void dsa_shutdown(struct platform_device *pdev)
+{
+}
+
+#ifdef CONFIG_PM_SLEEP
+static int dsa_suspend(struct device *d)
+{
+ struct platform_device *pdev = to_platform_device(d);
+ struct dsa_switch_tree *dst = platform_get_drvdata(pdev);
+ int i, ret = 0;
+
+ for (i = 0; i < dst->pd->nr_chips; i++) {
+ struct dsa_switch *ds = dst->ds[i];
+
+ if (ds != NULL)
+ ret = dsa_switch_suspend(ds);
+ }
+
+ return ret;
+}
+
+static int dsa_resume(struct device *d)
+{
+ struct platform_device *pdev = to_platform_device(d);
+ struct dsa_switch_tree *dst = platform_get_drvdata(pdev);
+ int i, ret = 0;
+
+ for (i = 0; i < dst->pd->nr_chips; i++) {
+ struct dsa_switch *ds = dst->ds[i];
+
+ if (ds != NULL)
+ ret = dsa_switch_resume(ds);
+ }
+
+ return ret;
+}
+#endif
+
+static SIMPLE_DEV_PM_OPS(dsa_pm_ops, dsa_suspend, dsa_resume);
+
+static const struct of_device_id dsa_of_match_table[] = {
+ { .compatible = "marvell,dsa", },
+ {}
+};
+MODULE_DEVICE_TABLE(of, dsa_of_match_table);
+
+static struct platform_driver dsa_driver = {
+ .probe = dsa_probe,
+ .remove = dsa_remove,
+ .shutdown = dsa_shutdown,
+ .driver = {
+ .name = "dsa",
+ .of_match_table = dsa_of_match_table,
+ .pm = &dsa_pm_ops,
+ },
+};
+
+int dsa_legacy_register(void)
+{
+ return platform_driver_register(&dsa_driver);
+}
+
+void dsa_legacy_unregister(void)
+{
+ platform_driver_unregister(&dsa_driver);
+}
diff --git a/net/dsa/slave.c b/net/dsa/slave.c
index c34872e..7693182 100644
--- a/net/dsa/slave.c
+++ b/net/dsa/slave.c
@@ -17,6 +17,7 @@
#include <linux/of_mdio.h>
#include <linux/mdio.h>
#include <linux/list.h>
+#include <net/dsa.h>
#include <net/rtnetlink.h>
#include <net/switchdev.h>
#include <net/pkt_cls.h>
@@ -419,8 +420,8 @@
return 0;
}
-static int dsa_fastest_ageing_time(struct dsa_switch *ds,
- unsigned int ageing_time)
+static unsigned int dsa_fastest_ageing_time(struct dsa_switch *ds,
+ unsigned int ageing_time)
{
int i;
@@ -443,9 +444,13 @@
unsigned long ageing_jiffies = clock_t_to_jiffies(attr->u.ageing_time);
unsigned int ageing_time = jiffies_to_msecs(ageing_jiffies);
- /* bridge skips -EOPNOTSUPP, so skip the prepare phase */
- if (switchdev_trans_ph_prepare(trans))
+ if (switchdev_trans_ph_prepare(trans)) {
+ if (ds->ageing_time_min && ageing_time < ds->ageing_time_min)
+ return -ERANGE;
+ if (ds->ageing_time_max && ageing_time > ds->ageing_time_max)
+ return -ERANGE;
return 0;
+ }
/* Keep the fastest ageing time in case of multiple bridges */
p->dp->ageing_time = ageing_time;
diff --git a/net/dsa/switch.c b/net/dsa/switch.c
index 6456dac..ca6e26e 100644
--- a/net/dsa/switch.c
+++ b/net/dsa/switch.c
@@ -1,7 +1,8 @@
/*
* Handling of a single switch chip, part of a switch fabric
*
- * Copyright (c) 2017 Vivien Didelot <vivien.didelot@savoirfairelinux.com>
+ * Copyright (c) 2017 Savoir-faire Linux Inc.
+ * Vivien Didelot <vivien.didelot@savoirfairelinux.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -19,9 +20,9 @@
if (ds->index == info->sw_index && ds->ops->port_bridge_join)
return ds->ops->port_bridge_join(ds, info->port, info->br);
- if (ds->index != info->sw_index)
- dev_dbg(ds->dev, "crosschip DSA port %d.%d bridged to %s\n",
- info->sw_index, info->port, netdev_name(info->br));
+ if (ds->index != info->sw_index && ds->ops->crosschip_bridge_join)
+ return ds->ops->crosschip_bridge_join(ds, info->sw_index,
+ info->port, info->br);
return 0;
}
@@ -32,9 +33,9 @@
if (ds->index == info->sw_index && ds->ops->port_bridge_leave)
ds->ops->port_bridge_leave(ds, info->port, info->br);
- if (ds->index != info->sw_index)
- dev_dbg(ds->dev, "crosschip DSA port %d.%d unbridged from %s\n",
- info->sw_index, info->port, netdev_name(info->br));
+ if (ds->index != info->sw_index && ds->ops->crosschip_bridge_leave)
+ ds->ops->crosschip_bridge_leave(ds, info->sw_index, info->port,
+ info->br);
return 0;
}
diff --git a/net/dsa/tag_brcm.c b/net/dsa/tag_brcm.c
index 5d925b6..2a9b52c 100644
--- a/net/dsa/tag_brcm.c
+++ b/net/dsa/tag_brcm.c
@@ -12,6 +12,7 @@
#include <linux/etherdevice.h>
#include <linux/list.h>
#include <linux/slab.h>
+#include <net/dsa.h>
#include "dsa_priv.h"
/* This tag length is 4 bytes, older ones were 6 bytes, we do not
@@ -91,23 +92,17 @@
return NULL;
}
-static int brcm_tag_rcv(struct sk_buff *skb, struct net_device *dev,
- struct packet_type *pt, struct net_device *orig_dev)
+static struct sk_buff *brcm_tag_rcv(struct sk_buff *skb, struct net_device *dev,
+ struct packet_type *pt,
+ struct net_device *orig_dev)
{
struct dsa_switch_tree *dst = dev->dsa_ptr;
struct dsa_switch *ds;
int source_port;
u8 *brcm_tag;
- if (unlikely(dst == NULL))
- goto out_drop;
-
ds = dst->cpu_switch;
- skb = skb_unshare(skb, GFP_ATOMIC);
- if (skb == NULL)
- goto out;
-
if (unlikely(!pskb_may_pull(skb, BRCM_TAG_LEN)))
goto out_drop;
@@ -139,22 +134,12 @@
skb->data - ETH_HLEN - BRCM_TAG_LEN,
2 * ETH_ALEN);
- skb_push(skb, ETH_HLEN);
- skb->pkt_type = PACKET_HOST;
skb->dev = ds->ports[source_port].netdev;
- skb->protocol = eth_type_trans(skb, skb->dev);
- skb->dev->stats.rx_packets++;
- skb->dev->stats.rx_bytes += skb->len;
-
- netif_receive_skb(skb);
-
- return 0;
+ return skb;
out_drop:
- kfree_skb(skb);
-out:
- return 0;
+ return NULL;
}
const struct dsa_device_ops brcm_netdev_ops = {
diff --git a/net/dsa/tag_dsa.c b/net/dsa/tag_dsa.c
index 72579ce..1c6633f 100644
--- a/net/dsa/tag_dsa.c
+++ b/net/dsa/tag_dsa.c
@@ -11,6 +11,7 @@
#include <linux/etherdevice.h>
#include <linux/list.h>
#include <linux/slab.h>
+#include <net/dsa.h>
#include "dsa_priv.h"
#define DSA_HLEN 4
@@ -67,8 +68,9 @@
return NULL;
}
-static int dsa_rcv(struct sk_buff *skb, struct net_device *dev,
- struct packet_type *pt, struct net_device *orig_dev)
+static struct sk_buff *dsa_rcv(struct sk_buff *skb, struct net_device *dev,
+ struct packet_type *pt,
+ struct net_device *orig_dev)
{
struct dsa_switch_tree *dst = dev->dsa_ptr;
struct dsa_switch *ds;
@@ -76,13 +78,6 @@
int source_device;
int source_port;
- if (unlikely(dst == NULL))
- goto out_drop;
-
- skb = skb_unshare(skb, GFP_ATOMIC);
- if (skb == NULL)
- goto out;
-
if (unlikely(!pskb_may_pull(skb, DSA_HLEN)))
goto out_drop;
@@ -164,21 +159,11 @@
}
skb->dev = ds->ports[source_port].netdev;
- skb_push(skb, ETH_HLEN);
- skb->pkt_type = PACKET_HOST;
- skb->protocol = eth_type_trans(skb, skb->dev);
- skb->dev->stats.rx_packets++;
- skb->dev->stats.rx_bytes += skb->len;
-
- netif_receive_skb(skb);
-
- return 0;
+ return skb;
out_drop:
- kfree_skb(skb);
-out:
- return 0;
+ return NULL;
}
const struct dsa_device_ops dsa_netdev_ops = {
diff --git a/net/dsa/tag_edsa.c b/net/dsa/tag_edsa.c
index 648c051..d9c668a 100644
--- a/net/dsa/tag_edsa.c
+++ b/net/dsa/tag_edsa.c
@@ -11,6 +11,7 @@
#include <linux/etherdevice.h>
#include <linux/list.h>
#include <linux/slab.h>
+#include <net/dsa.h>
#include "dsa_priv.h"
#define DSA_HLEN 4
@@ -80,8 +81,9 @@
return NULL;
}
-static int edsa_rcv(struct sk_buff *skb, struct net_device *dev,
- struct packet_type *pt, struct net_device *orig_dev)
+static struct sk_buff *edsa_rcv(struct sk_buff *skb, struct net_device *dev,
+ struct packet_type *pt,
+ struct net_device *orig_dev)
{
struct dsa_switch_tree *dst = dev->dsa_ptr;
struct dsa_switch *ds;
@@ -89,13 +91,6 @@
int source_device;
int source_port;
- if (unlikely(dst == NULL))
- goto out_drop;
-
- skb = skb_unshare(skb, GFP_ATOMIC);
- if (skb == NULL)
- goto out;
-
if (unlikely(!pskb_may_pull(skb, EDSA_HLEN)))
goto out_drop;
@@ -183,21 +178,11 @@
}
skb->dev = ds->ports[source_port].netdev;
- skb_push(skb, ETH_HLEN);
- skb->pkt_type = PACKET_HOST;
- skb->protocol = eth_type_trans(skb, skb->dev);
- skb->dev->stats.rx_packets++;
- skb->dev->stats.rx_bytes += skb->len;
-
- netif_receive_skb(skb);
-
- return 0;
+ return skb;
out_drop:
- kfree_skb(skb);
-out:
- return 0;
+ return NULL;
}
const struct dsa_device_ops edsa_netdev_ops = {
diff --git a/net/dsa/tag_lan9303.c b/net/dsa/tag_lan9303.c
new file mode 100644
index 0000000..70130ed
--- /dev/null
+++ b/net/dsa/tag_lan9303.c
@@ -0,0 +1,136 @@
+/*
+ * Copyright (C) 2017 Pengutronix, Juergen Borleis <jbe@pengutronix.de>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+#include <linux/etherdevice.h>
+#include <linux/list.h>
+#include <linux/slab.h>
+#include <net/dsa.h>
+#include "dsa_priv.h"
+
+/* To define the outgoing port and to discover the incoming port a regular
+ * VLAN tag is used by the LAN9303. But its VID meaning is 'special':
+ *
+ * Dest MAC Src MAC TAG Type
+ * ...| 1 2 3 4 5 6 | 1 2 3 4 5 6 | 1 2 3 4 | 1 2 |...
+ * |<------->|
+ * TAG:
+ * |<------------->|
+ * | 1 2 | 3 4 |
+ * TPID VID
+ * 0x8100
+ *
+ * VID bit 3 indicates a request for an ALR lookup.
+ *
+ * If VID bit 3 is zero, then bits 0 and 1 specify the destination port
+ * (0, 1, 2) or broadcast (3) or the source port (1, 2).
+ *
+ * VID bit 4 is used to specify if the STP port state should be overridden.
+ * Required when no forwarding between the external ports should happen.
+ */
+
+#define LAN9303_TAG_LEN 4
+#define LAN9303_MAX_PORTS 3
+
+static struct sk_buff *lan9303_xmit(struct sk_buff *skb, struct net_device *dev)
+{
+ struct dsa_slave_priv *p = netdev_priv(dev);
+ u16 *lan9303_tag;
+
+ /* insert a special VLAN tag between the MAC addresses
+ * and the current ethertype field.
+ */
+ if (skb_cow_head(skb, LAN9303_TAG_LEN) < 0) {
+ dev_dbg(&dev->dev,
+ "Cannot make room for the special tag. Dropping packet\n");
+ goto out_free;
+ }
+
+ /* provide 'LAN9303_TAG_LEN' bytes additional space */
+ skb_push(skb, LAN9303_TAG_LEN);
+
+ /* make room between MACs and Ether-Type */
+ memmove(skb->data, skb->data + LAN9303_TAG_LEN, 2 * ETH_ALEN);
+
+ lan9303_tag = (u16 *)(skb->data + 2 * ETH_ALEN);
+ lan9303_tag[0] = htons(ETH_P_8021Q);
+ lan9303_tag[1] = htons(p->dp->index | BIT(4));
+
+ return skb;
+out_free:
+ kfree_skb(skb);
+ return NULL;
+}
+
+static struct sk_buff *lan9303_rcv(struct sk_buff *skb, struct net_device *dev,
+ struct packet_type *pt, struct net_device *orig_dev)
+{
+ u16 *lan9303_tag;
+ struct dsa_switch_tree *dst = dev->dsa_ptr;
+ struct dsa_switch *ds;
+ unsigned int source_port;
+
+ ds = dst->ds[0];
+
+ if (unlikely(!ds)) {
+ dev_warn_ratelimited(&dev->dev, "Dropping packet, due to missing DSA switch device\n");
+ return NULL;
+ }
+
+ if (unlikely(!pskb_may_pull(skb, LAN9303_TAG_LEN))) {
+ dev_warn_ratelimited(&dev->dev,
+ "Dropping packet, cannot pull\n");
+ return NULL;
+ }
+
+ /* '->data' points into the middle of our special VLAN tag information:
+ *
+ * ~ MAC src | 0x81 | 0x00 | 0xyy | 0xzz | ether type
+ * ^
+ * ->data
+ */
+ lan9303_tag = (u16 *)(skb->data - 2);
+
+ if (lan9303_tag[0] != htons(ETH_P_8021Q)) {
+ dev_warn_ratelimited(&dev->dev, "Dropping packet due to invalid VLAN marker\n");
+ return NULL;
+ }
+
+ source_port = ntohs(lan9303_tag[1]) & 0x3;
+
+ if (source_port >= LAN9303_MAX_PORTS) {
+ dev_warn_ratelimited(&dev->dev, "Dropping packet due to invalid source port\n");
+ return NULL;
+ }
+
+ if (!ds->ports[source_port].netdev) {
+ dev_warn_ratelimited(&dev->dev, "Dropping packet due to invalid netdev or device\n");
+ return NULL;
+ }
+
+ /* remove the special VLAN tag between the MAC addresses
+ * and the current ethertype field.
+ */
+ skb_pull_rcsum(skb, 2 + 2);
+ memmove(skb->data - ETH_HLEN, skb->data - (ETH_HLEN + LAN9303_TAG_LEN),
+ 2 * ETH_ALEN);
+
+ /* forward the packet to the dedicated interface */
+ skb->dev = ds->ports[source_port].netdev;
+
+ return skb;
+}
+
+const struct dsa_device_ops lan9303_netdev_ops = {
+ .xmit = lan9303_xmit,
+ .rcv = lan9303_rcv,
+};
diff --git a/net/dsa/tag_mtk.c b/net/dsa/tag_mtk.c
new file mode 100644
index 0000000..837cddd
--- /dev/null
+++ b/net/dsa/tag_mtk.c
@@ -0,0 +1,100 @@
+/*
+ * Mediatek DSA Tag support
+ * Copyright (C) 2017 Landen Chao <landen.chao@mediatek.com>
+ * Sean Wang <sean.wang@mediatek.com>
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/etherdevice.h>
+#include <net/dsa.h>
+#include "dsa_priv.h"
+
+#define MTK_HDR_LEN 4
+#define MTK_HDR_RECV_SOURCE_PORT_MASK GENMASK(2, 0)
+#define MTK_HDR_XMIT_DP_BIT_MASK GENMASK(5, 0)
+
+static struct sk_buff *mtk_tag_xmit(struct sk_buff *skb,
+ struct net_device *dev)
+{
+ struct dsa_slave_priv *p = netdev_priv(dev);
+ u8 *mtk_tag;
+
+ if (skb_cow_head(skb, MTK_HDR_LEN) < 0)
+ goto out_free;
+
+ skb_push(skb, MTK_HDR_LEN);
+
+ memmove(skb->data, skb->data + MTK_HDR_LEN, 2 * ETH_ALEN);
+
+ /* Build the tag after the MAC Source Address */
+ mtk_tag = skb->data + 2 * ETH_ALEN;
+ mtk_tag[0] = 0;
+ mtk_tag[1] = (1 << p->dp->index) & MTK_HDR_XMIT_DP_BIT_MASK;
+ mtk_tag[2] = 0;
+ mtk_tag[3] = 0;
+
+ return skb;
+
+out_free:
+ kfree_skb(skb);
+ return NULL;
+}
+
+static struct sk_buff *mtk_tag_rcv(struct sk_buff *skb, struct net_device *dev,
+ struct packet_type *pt,
+ struct net_device *orig_dev)
+{
+ struct dsa_switch_tree *dst = dev->dsa_ptr;
+ struct dsa_switch *ds;
+ int port;
+ __be16 *phdr, hdr;
+
+ if (unlikely(!pskb_may_pull(skb, MTK_HDR_LEN)))
+ goto out_drop;
+
+ /* The MTK header is added by the switch between src addr
+ * and ethertype at this point, skb->data points to 2 bytes
+ * after src addr so header should be 2 bytes right before.
+ */
+ phdr = (__be16 *)(skb->data - 2);
+ hdr = ntohs(*phdr);
+
+ /* Remove MTK tag and recalculate checksum. */
+ skb_pull_rcsum(skb, MTK_HDR_LEN);
+
+ memmove(skb->data - ETH_HLEN,
+ skb->data - ETH_HLEN - MTK_HDR_LEN,
+ 2 * ETH_ALEN);
+
+ /* This protocol doesn't support cascading multiple
+ * switches so it's safe to assume the switch is first
+ * in the tree.
+ */
+ ds = dst->ds[0];
+ if (!ds)
+ goto out_drop;
+
+ /* Get source port information */
+ port = (hdr & MTK_HDR_RECV_SOURCE_PORT_MASK);
+ if (!ds->ports[port].netdev)
+ goto out_drop;
+
+ skb->dev = ds->ports[port].netdev;
+
+ return skb;
+
+out_drop:
+ return NULL;
+}
+
+const struct dsa_device_ops mtk_netdev_ops = {
+ .xmit = mtk_tag_xmit,
+ .rcv = mtk_tag_rcv,
+};
diff --git a/net/dsa/tag_qca.c b/net/dsa/tag_qca.c
index 30240f3..3ba3f59 100644
--- a/net/dsa/tag_qca.c
+++ b/net/dsa/tag_qca.c
@@ -12,6 +12,7 @@
*/
#include <linux/etherdevice.h>
+#include <net/dsa.h>
#include "dsa_priv.h"
#define QCA_HDR_LEN 2
@@ -65,8 +66,9 @@
return NULL;
}
-static int qca_tag_rcv(struct sk_buff *skb, struct net_device *dev,
- struct packet_type *pt, struct net_device *orig_dev)
+static struct sk_buff *qca_tag_rcv(struct sk_buff *skb, struct net_device *dev,
+ struct packet_type *pt,
+ struct net_device *orig_dev)
{
struct dsa_switch_tree *dst = dev->dsa_ptr;
struct dsa_switch *ds;
@@ -74,13 +76,6 @@
int port;
__be16 *phdr, hdr;
- if (unlikely(!dst))
- goto out_drop;
-
- skb = skb_unshare(skb, GFP_ATOMIC);
- if (!skb)
- goto out;
-
if (unlikely(!pskb_may_pull(skb, QCA_HDR_LEN)))
goto out_drop;
@@ -114,22 +109,12 @@
goto out_drop;
/* Update skb & forward the frame accordingly */
- skb_push(skb, ETH_HLEN);
- skb->pkt_type = PACKET_HOST;
skb->dev = ds->ports[port].netdev;
- skb->protocol = eth_type_trans(skb, skb->dev);
- skb->dev->stats.rx_packets++;
- skb->dev->stats.rx_bytes += skb->len;
-
- netif_receive_skb(skb);
-
- return 0;
+ return skb;
out_drop:
- kfree_skb(skb);
-out:
- return 0;
+ return NULL;
}
const struct dsa_device_ops qca_netdev_ops = {
diff --git a/net/dsa/tag_trailer.c b/net/dsa/tag_trailer.c
index 26f9771..aafc2fc 100644
--- a/net/dsa/tag_trailer.c
+++ b/net/dsa/tag_trailer.c
@@ -11,6 +11,7 @@
#include <linux/etherdevice.h>
#include <linux/list.h>
#include <linux/slab.h>
+#include <net/dsa.h>
#include "dsa_priv.h"
static struct sk_buff *trailer_xmit(struct sk_buff *skb, struct net_device *dev)
@@ -57,22 +58,17 @@
return nskb;
}
-static int trailer_rcv(struct sk_buff *skb, struct net_device *dev,
- struct packet_type *pt, struct net_device *orig_dev)
+static struct sk_buff *trailer_rcv(struct sk_buff *skb, struct net_device *dev,
+ struct packet_type *pt,
+ struct net_device *orig_dev)
{
struct dsa_switch_tree *dst = dev->dsa_ptr;
struct dsa_switch *ds;
u8 *trailer;
int source_port;
- if (unlikely(dst == NULL))
- goto out_drop;
ds = dst->cpu_switch;
- skb = skb_unshare(skb, GFP_ATOMIC);
- if (skb == NULL)
- goto out;
-
if (skb_linearize(skb))
goto out_drop;
@@ -88,21 +84,11 @@
pskb_trim_rcsum(skb, skb->len - 4);
skb->dev = ds->ports[source_port].netdev;
- skb_push(skb, ETH_HLEN);
- skb->pkt_type = PACKET_HOST;
- skb->protocol = eth_type_trans(skb, skb->dev);
- skb->dev->stats.rx_packets++;
- skb->dev->stats.rx_bytes += skb->len;
-
- netif_receive_skb(skb);
-
- return 0;
+ return skb;
out_drop:
- kfree_skb(skb);
-out:
- return 0;
+ return NULL;
}
const struct dsa_device_ops trailer_netdev_ops = {
diff --git a/net/hsr/hsr_netlink.c b/net/hsr/hsr_netlink.c
index 1ab30e7..81dac16 100644
--- a/net/hsr/hsr_netlink.c
+++ b/net/hsr/hsr_netlink.c
@@ -350,7 +350,7 @@
return 0;
invalid:
- netlink_ack(skb_in, nlmsg_hdr(skb_in), -EINVAL);
+ netlink_ack(skb_in, nlmsg_hdr(skb_in), -EINVAL, NULL);
return 0;
nla_put_failure:
@@ -432,7 +432,7 @@
return 0;
invalid:
- netlink_ack(skb_in, nlmsg_hdr(skb_in), -EINVAL);
+ netlink_ack(skb_in, nlmsg_hdr(skb_in), -EINVAL, NULL);
return 0;
nla_put_failure:
diff --git a/net/ieee802154/nl802154.c b/net/ieee802154/nl802154.c
index fc60cd0..99f6c25 100644
--- a/net/ieee802154/nl802154.c
+++ b/net/ieee802154/nl802154.c
@@ -249,8 +249,7 @@
if (!cb->args[0]) {
err = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl802154_fam.hdrsize,
genl_family_attrbuf(&nl802154_fam),
- nl802154_fam.maxattr,
- nl802154_policy);
+ nl802154_fam.maxattr, nl802154_policy, NULL);
if (err)
goto out_unlock;
@@ -562,8 +561,8 @@
struct nl802154_dump_wpan_phy_state *state)
{
struct nlattr **tb = genl_family_attrbuf(&nl802154_fam);
- int ret = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl802154_fam.hdrsize,
- tb, nl802154_fam.maxattr, nl802154_policy);
+ int ret = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl802154_fam.hdrsize, tb,
+ nl802154_fam.maxattr, nl802154_policy, NULL);
/* TODO check if we can handle error here,
* we have no backward compatibility
@@ -1308,7 +1307,7 @@
struct nlattr *attrs[NL802154_DEV_ADDR_ATTR_MAX + 1];
if (!nla || nla_parse_nested(attrs, NL802154_DEV_ADDR_ATTR_MAX, nla,
- nl802154_dev_addr_policy))
+ nl802154_dev_addr_policy, NULL))
return -EINVAL;
if (!attrs[NL802154_DEV_ADDR_ATTR_PAN_ID] ||
@@ -1348,7 +1347,7 @@
struct nlattr *attrs[NL802154_KEY_ID_ATTR_MAX + 1];
if (!nla || nla_parse_nested(attrs, NL802154_KEY_ID_ATTR_MAX, nla,
- nl802154_key_id_policy))
+ nl802154_key_id_policy, NULL))
return -EINVAL;
if (!attrs[NL802154_KEY_ID_ATTR_MODE])
@@ -1565,7 +1564,7 @@
if (nla_parse_nested(attrs, NL802154_KEY_ATTR_MAX,
info->attrs[NL802154_ATTR_SEC_KEY],
- nl802154_key_policy))
+ nl802154_key_policy, info->extack))
return -EINVAL;
if (!attrs[NL802154_KEY_ATTR_USAGE_FRAMES] ||
@@ -1615,7 +1614,7 @@
if (nla_parse_nested(attrs, NL802154_KEY_ATTR_MAX,
info->attrs[NL802154_ATTR_SEC_KEY],
- nl802154_key_policy))
+ nl802154_key_policy, info->extack))
return -EINVAL;
if (ieee802154_llsec_parse_key_id(attrs[NL802154_KEY_ATTR_ID], &id) < 0)
@@ -1729,8 +1728,8 @@
{
struct nlattr *attrs[NL802154_DEV_ATTR_MAX + 1];
- if (!nla || nla_parse_nested(attrs, NL802154_DEV_ATTR_MAX, nla,
- nl802154_dev_policy))
+ if (!nla || nla_parse_nested(attrs, NL802154_DEV_ATTR_MAX,
+ nla, nl802154_dev_policy, NULL))
return -EINVAL;
memset(dev, 0, sizeof(*dev));
@@ -1783,7 +1782,7 @@
if (nla_parse_nested(attrs, NL802154_DEV_ATTR_MAX,
info->attrs[NL802154_ATTR_SEC_DEVICE],
- nl802154_dev_policy))
+ nl802154_dev_policy, info->extack))
return -EINVAL;
if (!attrs[NL802154_DEV_ATTR_EXTENDED_ADDR])
@@ -1911,7 +1910,7 @@
if (!info->attrs[NL802154_ATTR_SEC_DEVKEY] ||
nla_parse_nested(attrs, NL802154_DEVKEY_ATTR_MAX,
info->attrs[NL802154_ATTR_SEC_DEVKEY],
- nl802154_devkey_policy) < 0)
+ nl802154_devkey_policy, info->extack) < 0)
return -EINVAL;
if (!attrs[NL802154_DEVKEY_ATTR_FRAME_COUNTER] ||
@@ -1943,7 +1942,7 @@
if (nla_parse_nested(attrs, NL802154_DEVKEY_ATTR_MAX,
info->attrs[NL802154_ATTR_SEC_DEVKEY],
- nl802154_devkey_policy))
+ nl802154_devkey_policy, info->extack))
return -EINVAL;
if (!attrs[NL802154_DEVKEY_ATTR_EXTENDED_ADDR])
@@ -2063,8 +2062,8 @@
{
struct nlattr *attrs[NL802154_SECLEVEL_ATTR_MAX + 1];
- if (!nla || nla_parse_nested(attrs, NL802154_SECLEVEL_ATTR_MAX, nla,
- nl802154_seclevel_policy))
+ if (!nla || nla_parse_nested(attrs, NL802154_SECLEVEL_ATTR_MAX,
+ nla, nl802154_seclevel_policy, NULL))
return -EINVAL;
memset(sl, 0, sizeof(*sl));
diff --git a/net/ipv4/Makefile b/net/ipv4/Makefile
index c6d4238..f83de23 100644
--- a/net/ipv4/Makefile
+++ b/net/ipv4/Makefile
@@ -11,7 +11,7 @@
tcp_rate.o tcp_recovery.o \
tcp_offload.o datagram.o raw.o udp.o udplite.o \
udp_offload.o arp.o icmp.o devinet.o af_inet.o igmp.o \
- fib_frontend.o fib_semantics.o fib_trie.o \
+ fib_frontend.o fib_semantics.o fib_trie.o fib_notifier.o \
inet_fragment.o ping.o ip_tunnel_core.o gre_offload.o
obj-$(CONFIG_NET_IP_TUNNEL) += ip_tunnel.o
diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index 6b1fc6e..f3dad16 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -1343,6 +1343,9 @@
if (*(u8 *)iph != 0x45)
goto out_unlock;
+ if (ip_is_fragment(iph))
+ goto out_unlock;
+
if (unlikely(ip_fast_csum((u8 *)iph, 5)))
goto out_unlock;
@@ -1599,8 +1602,9 @@
};
#endif
-static const struct net_protocol tcp_protocol = {
+static struct net_protocol tcp_protocol = {
.early_demux = tcp_v4_early_demux,
+ .early_demux_handler = tcp_v4_early_demux,
.handler = tcp_v4_rcv,
.err_handler = tcp_v4_err,
.no_policy = 1,
@@ -1608,8 +1612,9 @@
.icmp_strict_tag_validation = 1,
};
-static const struct net_protocol udp_protocol = {
+static struct net_protocol udp_protocol = {
.early_demux = udp_v4_early_demux,
+ .early_demux_handler = udp_v4_early_demux,
.handler = udp_rcv,
.err_handler = udp_err,
.no_policy = 1,
@@ -1720,6 +1725,8 @@
net->ipv4.sysctl_ip_default_ttl = IPDEFTTL;
net->ipv4.sysctl_ip_dynaddr = 0;
net->ipv4.sysctl_ip_early_demux = 1;
+ net->ipv4.sysctl_udp_early_demux = 1;
+ net->ipv4.sysctl_tcp_early_demux = 1;
#ifdef CONFIG_SYSCTL
net->ipv4.sysctl_ip_prot_sock = PROT_SOCK;
#endif
diff --git a/net/ipv4/arp.c b/net/ipv4/arp.c
index 51b27ae..0937b34 100644
--- a/net/ipv4/arp.c
+++ b/net/ipv4/arp.c
@@ -872,7 +872,7 @@
skb->pkt_type != PACKET_HOST)
state = NUD_STALE;
neigh_update(n, sha, state,
- override ? NEIGH_UPDATE_F_OVERRIDE : 0);
+ override ? NEIGH_UPDATE_F_OVERRIDE : 0, 0);
neigh_release(n);
}
@@ -1033,7 +1033,7 @@
err = neigh_update(neigh, (r->arp_flags & ATF_COM) ?
r->arp_ha.sa_data : NULL, state,
NEIGH_UPDATE_F_OVERRIDE |
- NEIGH_UPDATE_F_ADMIN);
+ NEIGH_UPDATE_F_ADMIN, 0);
neigh_release(neigh);
}
return err;
@@ -1084,7 +1084,7 @@
if (neigh->nud_state & ~NUD_NOARP)
err = neigh_update(neigh, NULL, NUD_FAILED,
NEIGH_UPDATE_F_OVERRIDE|
- NEIGH_UPDATE_F_ADMIN);
+ NEIGH_UPDATE_F_ADMIN, 0);
neigh_release(neigh);
}
diff --git a/net/ipv4/devinet.c b/net/ipv4/devinet.c
index cebedd5..df14815 100644
--- a/net/ipv4/devinet.c
+++ b/net/ipv4/devinet.c
@@ -571,7 +571,8 @@
return ret;
}
-static int inet_rtm_deladdr(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int inet_rtm_deladdr(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct nlattr *tb[IFA_MAX+1];
@@ -582,7 +583,8 @@
ASSERT_RTNL();
- err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv4_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv4_policy,
+ extack);
if (err < 0)
goto errout;
@@ -752,7 +754,8 @@
struct in_device *in_dev;
int err;
- err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv4_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv4_policy,
+ NULL);
if (err < 0)
goto errout;
@@ -843,7 +846,8 @@
return NULL;
}
-static int inet_rtm_newaddr(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int inet_rtm_newaddr(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct in_ifaddr *ifa;
@@ -1192,6 +1196,18 @@
return done;
}
+static __be32 in_dev_select_addr(const struct in_device *in_dev,
+ int scope)
+{
+ for_primary_ifa(in_dev) {
+ if (ifa->ifa_scope != RT_SCOPE_LINK &&
+ ifa->ifa_scope <= scope)
+ return ifa->ifa_local;
+ } endfor_ifa(in_dev);
+
+ return 0;
+}
+
__be32 inet_select_addr(const struct net_device *dev, __be32 dst, int scope)
{
__be32 addr = 0;
@@ -1228,13 +1244,9 @@
if (master_idx &&
(dev = dev_get_by_index_rcu(net, master_idx)) &&
(in_dev = __in_dev_get_rcu(dev))) {
- for_primary_ifa(in_dev) {
- if (ifa->ifa_scope != RT_SCOPE_LINK &&
- ifa->ifa_scope <= scope) {
- addr = ifa->ifa_local;
- goto out_unlock;
- }
- } endfor_ifa(in_dev);
+ addr = in_dev_select_addr(in_dev, scope);
+ if (addr)
+ goto out_unlock;
}
/* Not loopback addresses on loopback should be preferred
@@ -1249,13 +1261,9 @@
if (!in_dev)
continue;
- for_primary_ifa(in_dev) {
- if (ifa->ifa_scope != RT_SCOPE_LINK &&
- ifa->ifa_scope <= scope) {
- addr = ifa->ifa_local;
- goto out_unlock;
- }
- } endfor_ifa(in_dev);
+ addr = in_dev_select_addr(in_dev, scope);
+ if (addr)
+ goto out_unlock;
}
out_unlock:
rcu_read_unlock();
@@ -1713,7 +1721,7 @@
if (dev && !__in_dev_get_rtnl(dev))
return -EAFNOSUPPORT;
- err = nla_parse_nested(tb, IFLA_INET_MAX, nla, inet_af_policy);
+ err = nla_parse_nested(tb, IFLA_INET_MAX, nla, inet_af_policy, NULL);
if (err < 0)
return err;
@@ -1741,7 +1749,7 @@
if (!in_dev)
return -EAFNOSUPPORT;
- if (nla_parse_nested(tb, IFLA_INET_MAX, nla, NULL) < 0)
+ if (nla_parse_nested(tb, IFLA_INET_MAX, nla, NULL, NULL) < 0)
BUG();
if (tb[IFLA_INET_CONF]) {
@@ -1798,6 +1806,9 @@
if (nla_put_s32(skb, NETCONFA_IFINDEX, ifindex) < 0)
goto nla_put_failure;
+ if (!devconf)
+ goto out;
+
if ((all || type == NETCONFA_FORWARDING) &&
nla_put_s32(skb, NETCONFA_FORWARDING,
IPV4_DEVCONF(*devconf, FORWARDING)) < 0)
@@ -1819,6 +1830,7 @@
IPV4_DEVCONF(*devconf, IGNORE_ROUTES_WITH_LINKDOWN)) < 0)
goto nla_put_failure;
+out:
nlmsg_end(skb, nlh);
return 0;
@@ -1827,8 +1839,8 @@
return -EMSGSIZE;
}
-void inet_netconf_notify_devconf(struct net *net, int type, int ifindex,
- struct ipv4_devconf *devconf)
+void inet_netconf_notify_devconf(struct net *net, int event, int type,
+ int ifindex, struct ipv4_devconf *devconf)
{
struct sk_buff *skb;
int err = -ENOBUFS;
@@ -1838,7 +1850,7 @@
goto errout;
err = inet_netconf_fill_devconf(skb, ifindex, devconf, 0, 0,
- RTM_NEWNETCONF, 0, type);
+ event, 0, type);
if (err < 0) {
/* -EMSGSIZE implies BUG in inet_netconf_msgsize_devconf() */
WARN_ON(err == -EMSGSIZE);
@@ -1861,7 +1873,8 @@
};
static int inet_netconf_get_devconf(struct sk_buff *in_skb,
- struct nlmsghdr *nlh)
+ struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(in_skb->sk);
struct nlattr *tb[NETCONFA_MAX+1];
@@ -1874,7 +1887,7 @@
int err;
err = nlmsg_parse(nlh, sizeof(*ncm), tb, NETCONFA_MAX,
- devconf_ipv4_policy);
+ devconf_ipv4_policy, extack);
if (err < 0)
goto errout;
@@ -2017,10 +2030,12 @@
IPV4_DEVCONF_ALL(net, ACCEPT_REDIRECTS) = !on;
IPV4_DEVCONF_DFLT(net, FORWARDING) = on;
- inet_netconf_notify_devconf(net, NETCONFA_FORWARDING,
+ inet_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_FORWARDING,
NETCONFA_IFINDEX_ALL,
net->ipv4.devconf_all);
- inet_netconf_notify_devconf(net, NETCONFA_FORWARDING,
+ inet_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_FORWARDING,
NETCONFA_IFINDEX_DEFAULT,
net->ipv4.devconf_dflt);
@@ -2033,7 +2048,8 @@
in_dev = __in_dev_get_rtnl(dev);
if (in_dev) {
IN_DEV_CONF_SET(in_dev, FORWARDING, on);
- inet_netconf_notify_devconf(net, NETCONFA_FORWARDING,
+ inet_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_FORWARDING,
dev->ifindex, &in_dev->cnf);
}
}
@@ -2078,19 +2094,22 @@
if (i == IPV4_DEVCONF_RP_FILTER - 1 &&
new_value != old_value) {
ifindex = devinet_conf_ifindex(net, cnf);
- inet_netconf_notify_devconf(net, NETCONFA_RP_FILTER,
+ inet_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_RP_FILTER,
ifindex, cnf);
}
if (i == IPV4_DEVCONF_PROXY_ARP - 1 &&
new_value != old_value) {
ifindex = devinet_conf_ifindex(net, cnf);
- inet_netconf_notify_devconf(net, NETCONFA_PROXY_NEIGH,
+ inet_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_PROXY_NEIGH,
ifindex, cnf);
}
if (i == IPV4_DEVCONF_IGNORE_ROUTES_WITH_LINKDOWN - 1 &&
new_value != old_value) {
ifindex = devinet_conf_ifindex(net, cnf);
- inet_netconf_notify_devconf(net, NETCONFA_IGNORE_ROUTES_WITH_LINKDOWN,
+ inet_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_IGNORE_ROUTES_WITH_LINKDOWN,
ifindex, cnf);
}
}
@@ -2125,7 +2144,7 @@
container_of(cnf, struct in_device, cnf);
if (*valp)
dev_disable_lro(idev->dev);
- inet_netconf_notify_devconf(net,
+ inet_netconf_notify_devconf(net, RTM_NEWNETCONF,
NETCONFA_FORWARDING,
idev->dev->ifindex,
cnf);
@@ -2133,7 +2152,8 @@
rtnl_unlock();
rt_cache_flush(net);
} else
- inet_netconf_notify_devconf(net, NETCONFA_FORWARDING,
+ inet_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_FORWARDING,
NETCONFA_IFINDEX_DEFAULT,
net->ipv4.devconf_dflt);
}
@@ -2255,7 +2275,8 @@
p->sysctl = t;
- inet_netconf_notify_devconf(net, NETCONFA_ALL, ifindex, p);
+ inet_netconf_notify_devconf(net, RTM_NEWNETCONF, NETCONFA_ALL,
+ ifindex, p);
return 0;
free:
@@ -2264,16 +2285,18 @@
return -ENOBUFS;
}
-static void __devinet_sysctl_unregister(struct ipv4_devconf *cnf)
+static void __devinet_sysctl_unregister(struct net *net,
+ struct ipv4_devconf *cnf, int ifindex)
{
struct devinet_sysctl_table *t = cnf->sysctl;
- if (!t)
- return;
+ if (t) {
+ cnf->sysctl = NULL;
+ unregister_net_sysctl_table(t->sysctl_header);
+ kfree(t);
+ }
- cnf->sysctl = NULL;
- unregister_net_sysctl_table(t->sysctl_header);
- kfree(t);
+ inet_netconf_notify_devconf(net, RTM_DELNETCONF, 0, ifindex, NULL);
}
static int devinet_sysctl_register(struct in_device *idev)
@@ -2295,7 +2318,9 @@
static void devinet_sysctl_unregister(struct in_device *idev)
{
- __devinet_sysctl_unregister(&idev->cnf);
+ struct net *net = dev_net(idev->dev);
+
+ __devinet_sysctl_unregister(net, &idev->cnf, idev->dev->ifindex);
neigh_sysctl_unregister(idev->arp_parms);
}
@@ -2370,9 +2395,9 @@
#ifdef CONFIG_SYSCTL
err_reg_ctl:
- __devinet_sysctl_unregister(dflt);
+ __devinet_sysctl_unregister(net, dflt, NETCONFA_IFINDEX_DEFAULT);
err_reg_dflt:
- __devinet_sysctl_unregister(all);
+ __devinet_sysctl_unregister(net, all, NETCONFA_IFINDEX_ALL);
err_reg_all:
if (tbl != ctl_forward_entry)
kfree(tbl);
@@ -2394,8 +2419,10 @@
tbl = net->ipv4.forw_hdr->ctl_table_arg;
unregister_net_sysctl_table(net->ipv4.forw_hdr);
- __devinet_sysctl_unregister(net->ipv4.devconf_dflt);
- __devinet_sysctl_unregister(net->ipv4.devconf_all);
+ __devinet_sysctl_unregister(net, net->ipv4.devconf_dflt,
+ NETCONFA_IFINDEX_DEFAULT);
+ __devinet_sysctl_unregister(net, net->ipv4.devconf_all,
+ NETCONFA_IFINDEX_ALL);
kfree(tbl);
#endif
kfree(net->ipv4.devconf_dflt);
diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c
index b1e2444..65cc02b 100644
--- a/net/ipv4/esp4.c
+++ b/net/ipv4/esp4.c
@@ -152,21 +152,28 @@
}
static struct ip_esp_hdr *esp_output_set_extra(struct sk_buff *skb,
+ struct xfrm_state *x,
struct ip_esp_hdr *esph,
struct esp_output_extra *extra)
{
- struct xfrm_state *x = skb_dst(skb)->xfrm;
-
/* For ESN we move the header forward by 4 bytes to
* accomodate the high bits. We will move it back after
* encryption.
*/
if ((x->props.flags & XFRM_STATE_ESN)) {
+ __u32 seqhi;
+ struct xfrm_offload *xo = xfrm_offload(skb);
+
+ if (xo)
+ seqhi = xo->seq.hi;
+ else
+ seqhi = XFRM_SKB_CB(skb)->seq.output.hi;
+
extra->esphoff = (unsigned char *)esph -
skb_transport_header(skb);
esph = (struct ip_esp_hdr *)((unsigned char *)esph - 4);
extra->seqhi = esph->spi;
- esph->seq_no = htonl(XFRM_SKB_CB(skb)->seq.output.hi);
+ esph->seq_no = htonl(seqhi);
}
esph->spi = x->id.spi;
@@ -198,98 +205,56 @@
tail[plen - 1] = proto;
}
-static int esp_output(struct xfrm_state *x, struct sk_buff *skb)
+static void esp_output_udp_encap(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *esp)
{
- struct esp_output_extra *extra;
- int err = -ENOMEM;
- struct ip_esp_hdr *esph;
- struct crypto_aead *aead;
- struct aead_request *req;
- struct scatterlist *sg, *dsg;
- struct sk_buff *trailer;
- struct page *page;
- void *tmp;
- u8 *iv;
+ int encap_type;
+ struct udphdr *uh;
+ __be32 *udpdata32;
+ __be16 sport, dport;
+ struct xfrm_encap_tmpl *encap = x->encap;
+ struct ip_esp_hdr *esph = esp->esph;
+
+ spin_lock_bh(&x->lock);
+ sport = encap->encap_sport;
+ dport = encap->encap_dport;
+ encap_type = encap->encap_type;
+ spin_unlock_bh(&x->lock);
+
+ uh = (struct udphdr *)esph;
+ uh->source = sport;
+ uh->dest = dport;
+ uh->len = htons(skb->len + esp->tailen
+ - skb_transport_offset(skb));
+ uh->check = 0;
+
+ switch (encap_type) {
+ default:
+ case UDP_ENCAP_ESPINUDP:
+ esph = (struct ip_esp_hdr *)(uh + 1);
+ break;
+ case UDP_ENCAP_ESPINUDP_NON_IKE:
+ udpdata32 = (__be32 *)(uh + 1);
+ udpdata32[0] = udpdata32[1] = 0;
+ esph = (struct ip_esp_hdr *)(udpdata32 + 2);
+ break;
+ }
+
+ *skb_mac_header(skb) = IPPROTO_UDP;
+ esp->esph = esph;
+}
+
+int esp_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *esp)
+{
u8 *tail;
u8 *vaddr;
- int blksize;
- int clen;
- int alen;
- int plen;
- int ivlen;
- int tfclen;
int nfrags;
- int assoclen;
- int extralen;
- int tailen;
- __be64 seqno;
- __u8 proto = *skb_mac_header(skb);
-
- /* skb is pure payload to encrypt */
-
- aead = x->data;
- alen = crypto_aead_authsize(aead);
- ivlen = crypto_aead_ivsize(aead);
-
- tfclen = 0;
- if (x->tfcpad) {
- struct xfrm_dst *dst = (struct xfrm_dst *)skb_dst(skb);
- u32 padto;
-
- padto = min(x->tfcpad, esp4_get_mtu(x, dst->child_mtu_cached));
- if (skb->len < padto)
- tfclen = padto - skb->len;
- }
- blksize = ALIGN(crypto_aead_blocksize(aead), 4);
- clen = ALIGN(skb->len + 2 + tfclen, blksize);
- plen = clen - skb->len - tfclen;
- tailen = tfclen + plen + alen;
- assoclen = sizeof(*esph);
- extralen = 0;
-
- if (x->props.flags & XFRM_STATE_ESN) {
- extralen += sizeof(*extra);
- assoclen += sizeof(__be32);
- }
-
- *skb_mac_header(skb) = IPPROTO_ESP;
- esph = ip_esp_hdr(skb);
+ struct page *page;
+ struct sk_buff *trailer;
+ int tailen = esp->tailen;
/* this is non-NULL only with UDP Encapsulation */
- if (x->encap) {
- struct xfrm_encap_tmpl *encap = x->encap;
- struct udphdr *uh;
- __be32 *udpdata32;
- __be16 sport, dport;
- int encap_type;
-
- spin_lock_bh(&x->lock);
- sport = encap->encap_sport;
- dport = encap->encap_dport;
- encap_type = encap->encap_type;
- spin_unlock_bh(&x->lock);
-
- uh = (struct udphdr *)esph;
- uh->source = sport;
- uh->dest = dport;
- uh->len = htons(skb->len + tailen
- - skb_transport_offset(skb));
- uh->check = 0;
-
- switch (encap_type) {
- default:
- case UDP_ENCAP_ESPINUDP:
- esph = (struct ip_esp_hdr *)(uh + 1);
- break;
- case UDP_ENCAP_ESPINUDP_NON_IKE:
- udpdata32 = (__be32 *)(uh + 1);
- udpdata32[0] = udpdata32[1] = 0;
- esph = (struct ip_esp_hdr *)(udpdata32 + 2);
- break;
- }
-
- *skb_mac_header(skb) = IPPROTO_UDP;
- }
+ if (x->encap)
+ esp_output_udp_encap(x, skb, esp);
if (!skb_cloned(skb)) {
if (tailen <= skb_availroom(skb)) {
@@ -304,6 +269,8 @@
struct sock *sk = skb->sk;
struct page_frag *pfrag = &x->xfrag;
+ esp->inplace = false;
+
allocsize = ALIGN(tailen, L1_CACHE_BYTES);
spin_lock_bh(&x->lock);
@@ -320,10 +287,12 @@
tail = vaddr + pfrag->offset;
- esp_output_fill_trailer(tail, tfclen, plen, proto);
+ esp_output_fill_trailer(tail, esp->tfclen, esp->plen, esp->proto);
kunmap_atomic(vaddr);
+ spin_unlock_bh(&x->lock);
+
nfrags = skb_shinfo(skb)->nr_frags;
__skb_fill_page_desc(skb, nfrags, page, pfrag->offset,
@@ -339,107 +308,113 @@
if (sk)
atomic_add(tailen, &sk->sk_wmem_alloc);
- skb_push(skb, -skb_network_offset(skb));
-
- esph->seq_no = htonl(XFRM_SKB_CB(skb)->seq.output.low);
- esph->spi = x->id.spi;
-
- tmp = esp_alloc_tmp(aead, nfrags + 2, extralen);
- if (!tmp) {
- spin_unlock_bh(&x->lock);
- err = -ENOMEM;
- goto error;
- }
-
- extra = esp_tmp_extra(tmp);
- iv = esp_tmp_iv(aead, tmp, extralen);
- req = esp_tmp_req(aead, iv);
- sg = esp_req_sg(aead, req);
- dsg = &sg[nfrags];
-
- esph = esp_output_set_extra(skb, esph, extra);
-
- sg_init_table(sg, nfrags);
- skb_to_sgvec(skb, sg,
- (unsigned char *)esph - skb->data,
- assoclen + ivlen + clen + alen);
-
- allocsize = ALIGN(skb->data_len, L1_CACHE_BYTES);
-
- if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
- spin_unlock_bh(&x->lock);
- err = -ENOMEM;
- goto error;
- }
-
- skb_shinfo(skb)->nr_frags = 1;
-
- page = pfrag->page;
- get_page(page);
- /* replace page frags in skb with new page */
- __skb_fill_page_desc(skb, 0, page, pfrag->offset, skb->data_len);
- pfrag->offset = pfrag->offset + allocsize;
-
- sg_init_table(dsg, skb_shinfo(skb)->nr_frags + 1);
- skb_to_sgvec(skb, dsg,
- (unsigned char *)esph - skb->data,
- assoclen + ivlen + clen + alen);
-
- spin_unlock_bh(&x->lock);
-
- goto skip_cow2;
+ goto out;
}
}
cow:
- err = skb_cow_data(skb, tailen, &trailer);
- if (err < 0)
- goto error;
- nfrags = err;
+ nfrags = skb_cow_data(skb, tailen, &trailer);
+ if (nfrags < 0)
+ goto out;
tail = skb_tail_pointer(trailer);
- esph = ip_esp_hdr(skb);
+ esp->esph = ip_esp_hdr(skb);
skip_cow:
- esp_output_fill_trailer(tail, tfclen, plen, proto);
+ esp_output_fill_trailer(tail, esp->tfclen, esp->plen, esp->proto);
+ pskb_put(skb, trailer, tailen);
- pskb_put(skb, trailer, clen - skb->len + alen);
- skb_push(skb, -skb_network_offset(skb));
- esph->seq_no = htonl(XFRM_SKB_CB(skb)->seq.output.low);
- esph->spi = x->id.spi;
+out:
+ return nfrags;
+}
+EXPORT_SYMBOL_GPL(esp_output_head);
- tmp = esp_alloc_tmp(aead, nfrags, extralen);
- if (!tmp) {
- err = -ENOMEM;
- goto error;
+int esp_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *esp)
+{
+ u8 *iv;
+ int alen;
+ void *tmp;
+ int ivlen;
+ int assoclen;
+ int extralen;
+ struct page *page;
+ struct ip_esp_hdr *esph;
+ struct crypto_aead *aead;
+ struct aead_request *req;
+ struct scatterlist *sg, *dsg;
+ struct esp_output_extra *extra;
+ int err = -ENOMEM;
+
+ assoclen = sizeof(struct ip_esp_hdr);
+ extralen = 0;
+
+ if (x->props.flags & XFRM_STATE_ESN) {
+ extralen += sizeof(*extra);
+ assoclen += sizeof(__be32);
}
+ aead = x->data;
+ alen = crypto_aead_authsize(aead);
+ ivlen = crypto_aead_ivsize(aead);
+
+ tmp = esp_alloc_tmp(aead, esp->nfrags + 2, extralen);
+ if (!tmp)
+ goto error;
+
extra = esp_tmp_extra(tmp);
iv = esp_tmp_iv(aead, tmp, extralen);
req = esp_tmp_req(aead, iv);
sg = esp_req_sg(aead, req);
- dsg = sg;
- esph = esp_output_set_extra(skb, esph, extra);
+ if (esp->inplace)
+ dsg = sg;
+ else
+ dsg = &sg[esp->nfrags];
- sg_init_table(sg, nfrags);
+ esph = esp_output_set_extra(skb, x, esp->esph, extra);
+ esp->esph = esph;
+
+ sg_init_table(sg, esp->nfrags);
skb_to_sgvec(skb, sg,
(unsigned char *)esph - skb->data,
- assoclen + ivlen + clen + alen);
+ assoclen + ivlen + esp->clen + alen);
-skip_cow2:
+ if (!esp->inplace) {
+ int allocsize;
+ struct page_frag *pfrag = &x->xfrag;
+
+ allocsize = ALIGN(skb->data_len, L1_CACHE_BYTES);
+
+ spin_lock_bh(&x->lock);
+ if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+ spin_unlock_bh(&x->lock);
+ goto error;
+ }
+
+ skb_shinfo(skb)->nr_frags = 1;
+
+ page = pfrag->page;
+ get_page(page);
+ /* replace page frags in skb with new page */
+ __skb_fill_page_desc(skb, 0, page, pfrag->offset, skb->data_len);
+ pfrag->offset = pfrag->offset + allocsize;
+ spin_unlock_bh(&x->lock);
+
+ sg_init_table(dsg, skb_shinfo(skb)->nr_frags + 1);
+ skb_to_sgvec(skb, dsg,
+ (unsigned char *)esph - skb->data,
+ assoclen + ivlen + esp->clen + alen);
+ }
+
if ((x->props.flags & XFRM_STATE_ESN))
aead_request_set_callback(req, 0, esp_output_done_esn, skb);
else
aead_request_set_callback(req, 0, esp_output_done, skb);
- aead_request_set_crypt(req, sg, dsg, ivlen + clen, iv);
+ aead_request_set_crypt(req, sg, dsg, ivlen + esp->clen, iv);
aead_request_set_ad(req, assoclen);
- seqno = cpu_to_be64(XFRM_SKB_CB(skb)->seq.output.low +
- ((u64)XFRM_SKB_CB(skb)->seq.output.hi << 32));
-
memset(iv, 0, ivlen);
- memcpy(iv + ivlen - min(ivlen, 8), (u8 *)&seqno + 8 - min(ivlen, 8),
+ memcpy(iv + ivlen - min(ivlen, 8), (u8 *)&esp->seqno + 8 - min(ivlen, 8),
min(ivlen, 8));
ESP_SKB_CB(skb)->tmp = tmp;
@@ -465,11 +440,63 @@
error:
return err;
}
+EXPORT_SYMBOL_GPL(esp_output_tail);
-static int esp_input_done2(struct sk_buff *skb, int err)
+static int esp_output(struct xfrm_state *x, struct sk_buff *skb)
+{
+ int alen;
+ int blksize;
+ struct ip_esp_hdr *esph;
+ struct crypto_aead *aead;
+ struct esp_info esp;
+
+ esp.inplace = true;
+
+ esp.proto = *skb_mac_header(skb);
+ *skb_mac_header(skb) = IPPROTO_ESP;
+
+ /* skb is pure payload to encrypt */
+
+ aead = x->data;
+ alen = crypto_aead_authsize(aead);
+
+ esp.tfclen = 0;
+ if (x->tfcpad) {
+ struct xfrm_dst *dst = (struct xfrm_dst *)skb_dst(skb);
+ u32 padto;
+
+ padto = min(x->tfcpad, esp4_get_mtu(x, dst->child_mtu_cached));
+ if (skb->len < padto)
+ esp.tfclen = padto - skb->len;
+ }
+ blksize = ALIGN(crypto_aead_blocksize(aead), 4);
+ esp.clen = ALIGN(skb->len + 2 + esp.tfclen, blksize);
+ esp.plen = esp.clen - skb->len - esp.tfclen;
+ esp.tailen = esp.tfclen + esp.plen + alen;
+
+ esp.esph = ip_esp_hdr(skb);
+
+ esp.nfrags = esp_output_head(x, skb, &esp);
+ if (esp.nfrags < 0)
+ return esp.nfrags;
+
+ esph = esp.esph;
+ esph->spi = x->id.spi;
+
+ esph->seq_no = htonl(XFRM_SKB_CB(skb)->seq.output.low);
+ esp.seqno = cpu_to_be64(XFRM_SKB_CB(skb)->seq.output.low +
+ ((u64)XFRM_SKB_CB(skb)->seq.output.hi << 32));
+
+ skb_push(skb, -skb_network_offset(skb));
+
+ return esp_output_tail(x, skb, &esp);
+}
+
+int esp_input_done2(struct sk_buff *skb, int err)
{
const struct iphdr *iph;
struct xfrm_state *x = xfrm_input_state(skb);
+ struct xfrm_offload *xo = xfrm_offload(skb);
struct crypto_aead *aead = x->data;
int alen = crypto_aead_authsize(aead);
int hlen = sizeof(struct ip_esp_hdr) + crypto_aead_ivsize(aead);
@@ -478,7 +505,8 @@
u8 nexthdr[2];
int padlen;
- kfree(ESP_SKB_CB(skb)->tmp);
+ if (!xo || (xo && !(xo->flags & CRYPTO_DONE)))
+ kfree(ESP_SKB_CB(skb)->tmp);
if (unlikely(err))
goto out;
@@ -549,6 +577,7 @@
out:
return err;
}
+EXPORT_SYMBOL_GPL(esp_input_done2);
static void esp_input_done(struct crypto_async_request *base, int err)
{
@@ -751,13 +780,17 @@
char aead_name[CRYPTO_MAX_ALG_NAME];
struct crypto_aead *aead;
int err;
+ u32 mask = 0;
err = -ENAMETOOLONG;
if (snprintf(aead_name, CRYPTO_MAX_ALG_NAME, "%s(%s)",
x->geniv, x->aead->alg_name) >= CRYPTO_MAX_ALG_NAME)
goto error;
- aead = crypto_alloc_aead(aead_name, 0, 0);
+ if (x->xso.offload_handle)
+ mask |= CRYPTO_ALG_ASYNC;
+
+ aead = crypto_alloc_aead(aead_name, 0, mask);
err = PTR_ERR(aead);
if (IS_ERR(aead))
goto error;
@@ -787,6 +820,7 @@
char authenc_name[CRYPTO_MAX_ALG_NAME];
unsigned int keylen;
int err;
+ u32 mask = 0;
err = -EINVAL;
if (!x->ealg)
@@ -812,7 +846,10 @@
goto error;
}
- aead = crypto_alloc_aead(authenc_name, 0, 0);
+ if (x->xso.offload_handle)
+ mask |= CRYPTO_ALG_ASYNC;
+
+ aead = crypto_alloc_aead(authenc_name, 0, mask);
err = PTR_ERR(aead);
if (IS_ERR(aead))
goto error;
@@ -931,7 +968,7 @@
.destructor = esp_destroy,
.get_mtu = esp4_get_mtu,
.input = esp_input,
- .output = esp_output
+ .output = esp_output,
};
static struct xfrm4_protocol esp4_protocol = {
diff --git a/net/ipv4/esp4_offload.c b/net/ipv4/esp4_offload.c
index 1de4426..e066601 100644
--- a/net/ipv4/esp4_offload.c
+++ b/net/ipv4/esp4_offload.c
@@ -43,27 +43,31 @@
if ((err = xfrm_parse_spi(skb, IPPROTO_ESP, &spi, &seq)) != 0)
goto out;
- err = secpath_set(skb);
- if (err)
- goto out;
-
- if (skb->sp->len == XFRM_MAX_DEPTH)
- goto out;
-
- x = xfrm_state_lookup(dev_net(skb->dev), skb->mark,
- (xfrm_address_t *)&ip_hdr(skb)->daddr,
- spi, IPPROTO_ESP, AF_INET);
- if (!x)
- goto out;
-
- skb->sp->xvec[skb->sp->len++] = x;
- skb->sp->olen++;
-
xo = xfrm_offload(skb);
- if (!xo) {
- xfrm_state_put(x);
- goto out;
+ if (!xo || !(xo->flags & CRYPTO_DONE)) {
+ err = secpath_set(skb);
+ if (err)
+ goto out;
+
+ if (skb->sp->len == XFRM_MAX_DEPTH)
+ goto out;
+
+ x = xfrm_state_lookup(dev_net(skb->dev), skb->mark,
+ (xfrm_address_t *)&ip_hdr(skb)->daddr,
+ spi, IPPROTO_ESP, AF_INET);
+ if (!x)
+ goto out;
+
+ skb->sp->xvec[skb->sp->len++] = x;
+ skb->sp->olen++;
+
+ xo = xfrm_offload(skb);
+ if (!xo) {
+ xfrm_state_put(x);
+ goto out;
+ }
}
+
xo->flags |= XFRM_GRO;
XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip4 = NULL;
@@ -84,19 +88,214 @@
return NULL;
}
+static void esp4_gso_encap(struct xfrm_state *x, struct sk_buff *skb)
+{
+ struct ip_esp_hdr *esph;
+ struct iphdr *iph = ip_hdr(skb);
+ struct xfrm_offload *xo = xfrm_offload(skb);
+ int proto = iph->protocol;
+
+ skb_push(skb, -skb_network_offset(skb));
+ esph = ip_esp_hdr(skb);
+ *skb_mac_header(skb) = IPPROTO_ESP;
+
+ esph->spi = x->id.spi;
+ esph->seq_no = htonl(XFRM_SKB_CB(skb)->seq.output.low);
+
+ xo->proto = proto;
+}
+
+static struct sk_buff *esp4_gso_segment(struct sk_buff *skb,
+ netdev_features_t features)
+{
+ __u32 seq;
+ int err = 0;
+ struct sk_buff *skb2;
+ struct xfrm_state *x;
+ struct ip_esp_hdr *esph;
+ struct crypto_aead *aead;
+ struct sk_buff *segs = ERR_PTR(-EINVAL);
+ netdev_features_t esp_features = features;
+ struct xfrm_offload *xo = xfrm_offload(skb);
+
+ if (!xo)
+ goto out;
+
+ seq = xo->seq.low;
+
+ x = skb->sp->xvec[skb->sp->len - 1];
+ aead = x->data;
+ esph = ip_esp_hdr(skb);
+
+ if (esph->spi != x->id.spi)
+ goto out;
+
+ if (!pskb_may_pull(skb, sizeof(*esph) + crypto_aead_ivsize(aead)))
+ goto out;
+
+ __skb_pull(skb, sizeof(*esph) + crypto_aead_ivsize(aead));
+
+ skb->encap_hdr_csum = 1;
+
+ if (!(features & NETIF_F_HW_ESP))
+ esp_features = features & ~(NETIF_F_SG | NETIF_F_CSUM_MASK);
+
+ segs = x->outer_mode->gso_segment(x, skb, esp_features);
+ if (IS_ERR_OR_NULL(segs))
+ goto out;
+
+ __skb_pull(skb, skb->data - skb_mac_header(skb));
+
+ skb2 = segs;
+ do {
+ struct sk_buff *nskb = skb2->next;
+
+ xo = xfrm_offload(skb2);
+ xo->flags |= XFRM_GSO_SEGMENT;
+ xo->seq.low = seq;
+ xo->seq.hi = xfrm_replay_seqhi(x, seq);
+
+ if(!(features & NETIF_F_HW_ESP))
+ xo->flags |= CRYPTO_FALLBACK;
+
+ x->outer_mode->xmit(x, skb2);
+
+ err = x->type_offload->xmit(x, skb2, esp_features);
+ if (err) {
+ kfree_skb_list(segs);
+ return ERR_PTR(err);
+ }
+
+ if (!skb_is_gso(skb2))
+ seq++;
+ else
+ seq += skb_shinfo(skb2)->gso_segs;
+
+ skb_push(skb2, skb2->mac_len);
+ skb2 = nskb;
+ } while (skb2);
+
+out:
+ return segs;
+}
+
+static int esp_input_tail(struct xfrm_state *x, struct sk_buff *skb)
+{
+ struct crypto_aead *aead = x->data;
+
+ if (!pskb_may_pull(skb, sizeof(struct ip_esp_hdr) + crypto_aead_ivsize(aead)))
+ return -EINVAL;
+
+ skb->ip_summed = CHECKSUM_NONE;
+
+ return esp_input_done2(skb, 0);
+}
+
+static int esp_xmit(struct xfrm_state *x, struct sk_buff *skb, netdev_features_t features)
+{
+ int err;
+ int alen;
+ int blksize;
+ struct xfrm_offload *xo;
+ struct ip_esp_hdr *esph;
+ struct crypto_aead *aead;
+ struct esp_info esp;
+ bool hw_offload = true;
+
+ esp.inplace = true;
+
+ xo = xfrm_offload(skb);
+
+ if (!xo)
+ return -EINVAL;
+
+ if (!(features & NETIF_F_HW_ESP) || !x->xso.offload_handle ||
+ (x->xso.dev != skb->dev)) {
+ xo->flags |= CRYPTO_FALLBACK;
+ hw_offload = false;
+ }
+
+ esp.proto = xo->proto;
+
+ /* skb is pure payload to encrypt */
+
+ aead = x->data;
+ alen = crypto_aead_authsize(aead);
+
+ esp.tfclen = 0;
+ /* XXX: Add support for tfc padding here. */
+
+ blksize = ALIGN(crypto_aead_blocksize(aead), 4);
+ esp.clen = ALIGN(skb->len + 2 + esp.tfclen, blksize);
+ esp.plen = esp.clen - skb->len - esp.tfclen;
+ esp.tailen = esp.tfclen + esp.plen + alen;
+
+ esp.esph = ip_esp_hdr(skb);
+
+
+ if (!hw_offload || (hw_offload && !skb_is_gso(skb))) {
+ esp.nfrags = esp_output_head(x, skb, &esp);
+ if (esp.nfrags < 0)
+ return esp.nfrags;
+ }
+
+ esph = esp.esph;
+ esph->spi = x->id.spi;
+
+ skb_push(skb, -skb_network_offset(skb));
+
+ if (xo->flags & XFRM_GSO_SEGMENT) {
+ esph->seq_no = htonl(xo->seq.low);
+ } else {
+ ip_hdr(skb)->tot_len = htons(skb->len);
+ ip_send_check(ip_hdr(skb));
+ }
+
+ if (hw_offload)
+ return 0;
+
+ esp.seqno = cpu_to_be64(xo->seq.low + ((u64)xo->seq.hi << 32));
+
+ err = esp_output_tail(x, skb, &esp);
+ if (err < 0)
+ return err;
+
+ secpath_reset(skb);
+
+ return 0;
+}
+
static const struct net_offload esp4_offload = {
.callbacks = {
.gro_receive = esp4_gro_receive,
+ .gso_segment = esp4_gso_segment,
},
};
+static const struct xfrm_type_offload esp_type_offload = {
+ .description = "ESP4 OFFLOAD",
+ .owner = THIS_MODULE,
+ .proto = IPPROTO_ESP,
+ .input_tail = esp_input_tail,
+ .xmit = esp_xmit,
+ .encap = esp4_gso_encap,
+};
+
static int __init esp4_offload_init(void)
{
+ if (xfrm_register_type_offload(&esp_type_offload, AF_INET) < 0) {
+ pr_info("%s: can't add xfrm type offload\n", __func__);
+ return -EAGAIN;
+ }
+
return inet_add_offload(&esp4_offload, IPPROTO_ESP);
}
static void __exit esp4_offload_exit(void)
{
+ if (xfrm_unregister_type_offload(&esp_type_offload, AF_INET) < 0)
+ pr_info("%s: can't remove xfrm type offload\n", __func__);
+
inet_del_offload(&esp4_offload, IPPROTO_ESP);
}
diff --git a/net/ipv4/fib_frontend.c b/net/ipv4/fib_frontend.c
index 8f2133f..39bd1ed 100644
--- a/net/ipv4/fib_frontend.c
+++ b/net/ipv4/fib_frontend.c
@@ -632,7 +632,8 @@
int err, remaining;
struct rtmsg *rtm;
- err = nlmsg_validate(nlh, sizeof(*rtm), RTA_MAX, rtm_ipv4_policy);
+ err = nlmsg_validate(nlh, sizeof(*rtm), RTA_MAX, rtm_ipv4_policy,
+ NULL);
if (err < 0)
goto errout;
@@ -709,7 +710,8 @@
return err;
}
-static int inet_rtm_delroute(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int inet_rtm_delroute(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct fib_config cfg;
@@ -731,7 +733,8 @@
return err;
}
-static int inet_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int inet_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct fib_config cfg;
@@ -1127,7 +1130,8 @@
{
if (fib_sync_down_dev(dev, event, force))
fib_flush(dev_net(dev));
- rt_cache_flush(dev_net(dev));
+ else
+ rt_cache_flush(dev_net(dev));
arp_ifdown(dev);
}
diff --git a/net/ipv4/fib_notifier.c b/net/ipv4/fib_notifier.c
new file mode 100644
index 0000000..e0714d9
--- /dev/null
+++ b/net/ipv4/fib_notifier.c
@@ -0,0 +1,86 @@
+#include <linux/rtnetlink.h>
+#include <linux/notifier.h>
+#include <linux/rcupdate.h>
+#include <linux/kernel.h>
+#include <net/net_namespace.h>
+#include <net/netns/ipv4.h>
+#include <net/ip_fib.h>
+
+static ATOMIC_NOTIFIER_HEAD(fib_chain);
+
+int call_fib_notifier(struct notifier_block *nb, struct net *net,
+ enum fib_event_type event_type,
+ struct fib_notifier_info *info)
+{
+ info->net = net;
+ return nb->notifier_call(nb, event_type, info);
+}
+
+int call_fib_notifiers(struct net *net, enum fib_event_type event_type,
+ struct fib_notifier_info *info)
+{
+ net->ipv4.fib_seq++;
+ info->net = net;
+ return atomic_notifier_call_chain(&fib_chain, event_type, info);
+}
+
+static unsigned int fib_seq_sum(void)
+{
+ unsigned int fib_seq = 0;
+ struct net *net;
+
+ rtnl_lock();
+ for_each_net(net)
+ fib_seq += net->ipv4.fib_seq;
+ rtnl_unlock();
+
+ return fib_seq;
+}
+
+static bool fib_dump_is_consistent(struct notifier_block *nb,
+ void (*cb)(struct notifier_block *nb),
+ unsigned int fib_seq)
+{
+ atomic_notifier_chain_register(&fib_chain, nb);
+ if (fib_seq == fib_seq_sum())
+ return true;
+ atomic_notifier_chain_unregister(&fib_chain, nb);
+ if (cb)
+ cb(nb);
+ return false;
+}
+
+#define FIB_DUMP_MAX_RETRIES 5
+int register_fib_notifier(struct notifier_block *nb,
+ void (*cb)(struct notifier_block *nb))
+{
+ int retries = 0;
+
+ do {
+ unsigned int fib_seq = fib_seq_sum();
+ struct net *net;
+
+ /* Mutex semantics guarantee that every change done to
+ * FIB tries before we read the change sequence counter
+ * is now visible to us.
+ */
+ rcu_read_lock();
+ for_each_net_rcu(net) {
+ fib_rules_notify(net, nb);
+ fib_notify(net, nb);
+ }
+ rcu_read_unlock();
+
+ if (fib_dump_is_consistent(nb, cb, fib_seq))
+ return 0;
+ } while (++retries < FIB_DUMP_MAX_RETRIES);
+
+ return -EBUSY;
+}
+EXPORT_SYMBOL(register_fib_notifier);
+
+int unregister_fib_notifier(struct notifier_block *nb)
+{
+ return atomic_notifier_chain_unregister(&fib_chain, nb);
+}
+EXPORT_SYMBOL(unregister_fib_notifier);
diff --git a/net/ipv4/fib_rules.c b/net/ipv4/fib_rules.c
index 2e50062..778ecf9 100644
--- a/net/ipv4/fib_rules.c
+++ b/net/ipv4/fib_rules.c
@@ -47,6 +47,27 @@
#endif
};
+static bool fib4_rule_matchall(const struct fib_rule *rule)
+{
+ struct fib4_rule *r = container_of(rule, struct fib4_rule, common);
+
+ if (r->dst_len || r->src_len || r->tos)
+ return false;
+ return fib_rule_matchall(rule);
+}
+
+bool fib4_rule_default(const struct fib_rule *rule)
+{
+ if (!fib4_rule_matchall(rule) || rule->action != FR_ACT_TO_TBL ||
+ rule->l3mdev)
+ return false;
+ if (rule->table != RT_TABLE_LOCAL && rule->table != RT_TABLE_MAIN &&
+ rule->table != RT_TABLE_DEFAULT)
+ return false;
+ return true;
+}
+EXPORT_SYMBOL_GPL(fib4_rule_default);
+
int __fib_lookup(struct net *net, struct flowi4 *flp,
struct fib_result *res, unsigned int flags)
{
@@ -164,12 +185,36 @@
return NULL;
}
-static int call_fib_rule_notifiers(struct net *net,
- enum fib_event_type event_type)
+static int call_fib_rule_notifier(struct notifier_block *nb, struct net *net,
+ enum fib_event_type event_type,
+ struct fib_rule *rule)
{
- struct fib_notifier_info info;
+ struct fib_rule_notifier_info info = {
+ .rule = rule,
+ };
- return call_fib_notifiers(net, event_type, &info);
+ return call_fib_notifier(nb, net, event_type, &info.info);
+}
+
+static int call_fib_rule_notifiers(struct net *net,
+ enum fib_event_type event_type,
+ struct fib_rule *rule)
+{
+ struct fib_rule_notifier_info info = {
+ .rule = rule,
+ };
+
+ return call_fib_notifiers(net, event_type, &info.info);
+}
+
+/* Called with rcu_read_lock() */
+void fib_rules_notify(struct net *net, struct notifier_block *nb)
+{
+ struct fib_rules_ops *ops = net->ipv4.rules_ops;
+ struct fib_rule *rule;
+
+ list_for_each_entry_rcu(rule, &ops->rules_list, list)
+ call_fib_rule_notifier(nb, net, FIB_EVENT_RULE_ADD, rule);
}
static const struct nla_policy fib4_rule_policy[FRA_MAX+1] = {
@@ -228,7 +273,7 @@
rule4->tos = frh->tos;
net->ipv4.fib_has_custom_rules = true;
- call_fib_rule_notifiers(net, FIB_EVENT_RULE_ADD);
+ call_fib_rule_notifiers(net, FIB_EVENT_RULE_ADD, rule);
err = 0;
errout:
@@ -250,7 +295,7 @@
net->ipv4.fib_num_tclassid_users--;
#endif
net->ipv4.fib_has_custom_rules = true;
- call_fib_rule_notifiers(net, FIB_EVENT_RULE_DEL);
+ call_fib_rule_notifiers(net, FIB_EVENT_RULE_DEL, rule);
errout:
return err;
}
diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c
index 317026a..da449dd 100644
--- a/net/ipv4/fib_semantics.c
+++ b/net/ipv4/fib_semantics.c
@@ -57,7 +57,6 @@
static struct hlist_head fib_info_devhash[DEVINDEX_HASHSIZE];
#ifdef CONFIG_IP_ROUTE_MULTIPATH
-u32 fib_multipath_secret __read_mostly;
#define for_nexthops(fi) { \
int nhsel; const struct fib_nh *nh; \
@@ -576,9 +575,6 @@
atomic_set(&nexthop_nh->nh_upper_bound, upper_bound);
} endfor_nexthops(fi);
-
- net_get_random_once(&fib_multipath_secret,
- sizeof(fib_multipath_secret));
}
static inline void fib_add_weight(struct fib_info *fi,
@@ -1641,7 +1637,7 @@
#endif
void fib_select_path(struct net *net, struct fib_result *res,
- struct flowi4 *fl4, int mp_hash)
+ struct flowi4 *fl4, const struct sk_buff *skb)
{
bool oif_check;
@@ -1650,10 +1646,9 @@
#ifdef CONFIG_IP_ROUTE_MULTIPATH
if (res->fi->fib_nhs > 1 && oif_check) {
- if (mp_hash < 0)
- mp_hash = get_hash_from_flowi4(fl4) >> 1;
+ int h = fib_multipath_hash(res->fi, fl4, skb);
- fib_select_multipath(res, mp_hash);
+ fib_select_multipath(res, h);
}
else
#endif
diff --git a/net/ipv4/fib_trie.c b/net/ipv4/fib_trie.c
index 2f0d823..1201409 100644
--- a/net/ipv4/fib_trie.c
+++ b/net/ipv4/fib_trie.c
@@ -84,43 +84,6 @@
#include <trace/events/fib.h>
#include "fib_lookup.h"
-static unsigned int fib_seq_sum(void)
-{
- unsigned int fib_seq = 0;
- struct net *net;
-
- rtnl_lock();
- for_each_net(net)
- fib_seq += net->ipv4.fib_seq;
- rtnl_unlock();
-
- return fib_seq;
-}
-
-static ATOMIC_NOTIFIER_HEAD(fib_chain);
-
-static int call_fib_notifier(struct notifier_block *nb, struct net *net,
- enum fib_event_type event_type,
- struct fib_notifier_info *info)
-{
- info->net = net;
- return nb->notifier_call(nb, event_type, info);
-}
-
-static void fib_rules_notify(struct net *net, struct notifier_block *nb,
- enum fib_event_type event_type)
-{
-#ifdef CONFIG_IP_MULTIPLE_TABLES
- struct fib_notifier_info info;
-
- if (net->ipv4.fib_has_custom_rules)
- call_fib_notifier(nb, net, event_type, &info);
-#endif
-}
-
-static void fib_notify(struct net *net, struct notifier_block *nb,
- enum fib_event_type event_type);
-
static int call_fib_entry_notifier(struct notifier_block *nb, struct net *net,
enum fib_event_type event_type, u32 dst,
int dst_len, struct fib_info *fi,
@@ -137,62 +100,6 @@
return call_fib_notifier(nb, net, event_type, &info.info);
}
-static bool fib_dump_is_consistent(struct notifier_block *nb,
- void (*cb)(struct notifier_block *nb),
- unsigned int fib_seq)
-{
- atomic_notifier_chain_register(&fib_chain, nb);
- if (fib_seq == fib_seq_sum())
- return true;
- atomic_notifier_chain_unregister(&fib_chain, nb);
- if (cb)
- cb(nb);
- return false;
-}
-
-#define FIB_DUMP_MAX_RETRIES 5
-int register_fib_notifier(struct notifier_block *nb,
- void (*cb)(struct notifier_block *nb))
-{
- int retries = 0;
-
- do {
- unsigned int fib_seq = fib_seq_sum();
- struct net *net;
-
- /* Mutex semantics guarantee that every change done to
- * FIB tries before we read the change sequence counter
- * is now visible to us.
- */
- rcu_read_lock();
- for_each_net_rcu(net) {
- fib_rules_notify(net, nb, FIB_EVENT_RULE_ADD);
- fib_notify(net, nb, FIB_EVENT_ENTRY_ADD);
- }
- rcu_read_unlock();
-
- if (fib_dump_is_consistent(nb, cb, fib_seq))
- return 0;
- } while (++retries < FIB_DUMP_MAX_RETRIES);
-
- return -EBUSY;
-}
-EXPORT_SYMBOL(register_fib_notifier);
-
-int unregister_fib_notifier(struct notifier_block *nb)
-{
- return atomic_notifier_chain_unregister(&fib_chain, nb);
-}
-EXPORT_SYMBOL(unregister_fib_notifier);
-
-int call_fib_notifiers(struct net *net, enum fib_event_type event_type,
- struct fib_notifier_info *info)
-{
- net->ipv4.fib_seq++;
- info->net = net;
- return atomic_notifier_call_chain(&fib_chain, event_type, info);
-}
-
static int call_fib_entry_notifiers(struct net *net,
enum fib_event_type event_type, u32 dst,
int dst_len, struct fib_info *fi,
@@ -1995,8 +1902,7 @@
}
static void fib_leaf_notify(struct net *net, struct key_vector *l,
- struct fib_table *tb, struct notifier_block *nb,
- enum fib_event_type event_type)
+ struct fib_table *tb, struct notifier_block *nb)
{
struct fib_alias *fa;
@@ -2012,22 +1918,21 @@
if (tb->tb_id != fa->tb_id)
continue;
- call_fib_entry_notifier(nb, net, event_type, l->key,
+ call_fib_entry_notifier(nb, net, FIB_EVENT_ENTRY_ADD, l->key,
KEYLENGTH - fa->fa_slen, fi, fa->fa_tos,
fa->fa_type, fa->tb_id);
}
}
static void fib_table_notify(struct net *net, struct fib_table *tb,
- struct notifier_block *nb,
- enum fib_event_type event_type)
+ struct notifier_block *nb)
{
struct trie *t = (struct trie *)tb->tb_data;
struct key_vector *l, *tp = t->kv;
t_key key = 0;
while ((l = leaf_walk_rcu(&tp, key)) != NULL) {
- fib_leaf_notify(net, l, tb, nb, event_type);
+ fib_leaf_notify(net, l, tb, nb);
key = l->key + 1;
/* stop in case of wrap around */
@@ -2036,8 +1941,7 @@
}
}
-static void fib_notify(struct net *net, struct notifier_block *nb,
- enum fib_event_type event_type)
+void fib_notify(struct net *net, struct notifier_block *nb)
{
unsigned int h;
@@ -2046,7 +1950,7 @@
struct fib_table *tb;
hlist_for_each_entry_rcu(tb, head, tb_hlist)
- fib_table_notify(net, tb, nb, event_type);
+ fib_table_notify(net, tb, nb);
}
}
diff --git a/net/ipv4/icmp.c b/net/ipv4/icmp.c
index fc310db..43318b5 100644
--- a/net/ipv4/icmp.c
+++ b/net/ipv4/icmp.c
@@ -464,22 +464,6 @@
local_bh_enable();
}
-#ifdef CONFIG_IP_ROUTE_MULTIPATH
-
-/* Source and destination is swapped. See ip_multipath_icmp_hash */
-static int icmp_multipath_hash_skb(const struct sk_buff *skb)
-{
- const struct iphdr *iph = ip_hdr(skb);
-
- return fib_multipath_hash(iph->daddr, iph->saddr);
-}
-
-#else
-
-#define icmp_multipath_hash_skb(skb) (-1)
-
-#endif
-
static struct rtable *icmp_route_lookup(struct net *net,
struct flowi4 *fl4,
struct sk_buff *skb_in,
@@ -505,8 +489,7 @@
fl4->flowi4_oif = l3mdev_master_ifindex(skb_dst(skb_in)->dev);
security_skb_classify_flow(skb_in, flowi4_to_flowi(fl4));
- rt = __ip_route_output_key_hash(net, fl4,
- icmp_multipath_hash_skb(skb_in));
+ rt = __ip_route_output_key_hash(net, fl4, skb_in);
if (IS_ERR(rt))
return rt;
diff --git a/net/ipv4/ip_gre.c b/net/ipv4/ip_gre.c
index c9c1cb6..e90c80a 100644
--- a/net/ipv4/ip_gre.c
+++ b/net/ipv4/ip_gre.c
@@ -829,7 +829,8 @@
static int ipgre_netlink_parms(struct net_device *dev,
struct nlattr *data[],
struct nlattr *tb[],
- struct ip_tunnel_parm *parms)
+ struct ip_tunnel_parm *parms,
+ __u32 *fwmark)
{
struct ip_tunnel *t = netdev_priv(dev);
@@ -886,6 +887,9 @@
t->ignore_df = !!nla_get_u8(data[IFLA_GRE_IGNORE_DF]);
}
+ if (data[IFLA_GRE_FWMARK])
+ *fwmark = nla_get_u32(data[IFLA_GRE_FWMARK]);
+
return 0;
}
@@ -957,6 +961,7 @@
{
struct ip_tunnel_parm p;
struct ip_tunnel_encap ipencap;
+ __u32 fwmark = 0;
int err;
if (ipgre_netlink_encap_parms(data, &ipencap)) {
@@ -967,31 +972,32 @@
return err;
}
- err = ipgre_netlink_parms(dev, data, tb, &p);
+ err = ipgre_netlink_parms(dev, data, tb, &p, &fwmark);
if (err < 0)
return err;
- return ip_tunnel_newlink(dev, tb, &p);
+ return ip_tunnel_newlink(dev, tb, &p, fwmark);
}
static int ipgre_changelink(struct net_device *dev, struct nlattr *tb[],
struct nlattr *data[])
{
+ struct ip_tunnel *t = netdev_priv(dev);
struct ip_tunnel_parm p;
struct ip_tunnel_encap ipencap;
+ __u32 fwmark = t->fwmark;
int err;
if (ipgre_netlink_encap_parms(data, &ipencap)) {
- struct ip_tunnel *t = netdev_priv(dev);
err = ip_tunnel_encap_setup(t, &ipencap);
if (err < 0)
return err;
}
- err = ipgre_netlink_parms(dev, data, tb, &p);
+ err = ipgre_netlink_parms(dev, data, tb, &p, &fwmark);
if (err < 0)
return err;
- return ip_tunnel_changelink(dev, tb, &p);
+ return ip_tunnel_changelink(dev, tb, &p, fwmark);
}
static size_t ipgre_get_size(const struct net_device *dev)
@@ -1029,6 +1035,8 @@
nla_total_size(0) +
/* IFLA_GRE_IGNORE_DF */
nla_total_size(1) +
+ /* IFLA_GRE_FWMARK */
+ nla_total_size(4) +
0;
}
@@ -1049,7 +1057,8 @@
nla_put_u8(skb, IFLA_GRE_TTL, p->iph.ttl) ||
nla_put_u8(skb, IFLA_GRE_TOS, p->iph.tos) ||
nla_put_u8(skb, IFLA_GRE_PMTUDISC,
- !!(p->iph.frag_off & htons(IP_DF))))
+ !!(p->iph.frag_off & htons(IP_DF))) ||
+ nla_put_u32(skb, IFLA_GRE_FWMARK, t->fwmark))
goto nla_put_failure;
if (nla_put_u16(skb, IFLA_GRE_ENCAP_TYPE,
@@ -1093,6 +1102,7 @@
[IFLA_GRE_ENCAP_DPORT] = { .type = NLA_U16 },
[IFLA_GRE_COLLECT_METADATA] = { .type = NLA_FLAG },
[IFLA_GRE_IGNORE_DF] = { .type = NLA_U8 },
+ [IFLA_GRE_FWMARK] = { .type = NLA_U32 },
};
static struct rtnl_link_ops ipgre_link_ops __read_mostly = {
diff --git a/net/ipv4/ip_input.c b/net/ipv4/ip_input.c
index d6feabb..fa2dc8f 100644
--- a/net/ipv4/ip_input.c
+++ b/net/ipv4/ip_input.c
@@ -313,6 +313,7 @@
const struct iphdr *iph = ip_hdr(skb);
struct rtable *rt;
struct net_device *dev = skb->dev;
+ void (*edemux)(struct sk_buff *skb);
/* if ingress device is enslaved to an L3 master device pass the
* skb to its handler for processing
@@ -329,8 +330,8 @@
int protocol = iph->protocol;
ipprot = rcu_dereference(inet_protos[protocol]);
- if (ipprot && ipprot->early_demux) {
- ipprot->early_demux(skb);
+ if (ipprot && (edemux = READ_ONCE(ipprot->early_demux))) {
+ edemux(skb);
/* must reload iph, skb->head might have changed */
iph = ip_hdr(skb);
}
diff --git a/net/ipv4/ip_sockglue.c b/net/ipv4/ip_sockglue.c
index 1d46d05..ec4fe3d 100644
--- a/net/ipv4/ip_sockglue.c
+++ b/net/ipv4/ip_sockglue.c
@@ -330,7 +330,6 @@
sent to multicast group to reach destination designated router.
*/
struct ip_ra_chain __rcu *ip_ra_chain;
-static DEFINE_SPINLOCK(ip_ra_lock);
static void ip_ra_destroy_rcu(struct rcu_head *head)
@@ -352,21 +351,17 @@
new_ra = on ? kmalloc(sizeof(*new_ra), GFP_KERNEL) : NULL;
- spin_lock_bh(&ip_ra_lock);
for (rap = &ip_ra_chain;
- (ra = rcu_dereference_protected(*rap,
- lockdep_is_held(&ip_ra_lock))) != NULL;
+ (ra = rtnl_dereference(*rap)) != NULL;
rap = &ra->next) {
if (ra->sk == sk) {
if (on) {
- spin_unlock_bh(&ip_ra_lock);
kfree(new_ra);
return -EADDRINUSE;
}
/* dont let ip_call_ra_chain() use sk again */
ra->sk = NULL;
RCU_INIT_POINTER(*rap, ra->next);
- spin_unlock_bh(&ip_ra_lock);
if (ra->destructor)
ra->destructor(sk);
@@ -380,17 +375,14 @@
return 0;
}
}
- if (!new_ra) {
- spin_unlock_bh(&ip_ra_lock);
+ if (!new_ra)
return -ENOBUFS;
- }
new_ra->sk = sk;
new_ra->destructor = destructor;
RCU_INIT_POINTER(new_ra->next, ra);
rcu_assign_pointer(*rap, new_ra);
sock_hold(sk);
- spin_unlock_bh(&ip_ra_lock);
return 0;
}
diff --git a/net/ipv4/ip_tunnel.c b/net/ipv4/ip_tunnel.c
index 823abae..b878ecb 100644
--- a/net/ipv4/ip_tunnel.c
+++ b/net/ipv4/ip_tunnel.c
@@ -293,7 +293,8 @@
static inline void init_tunnel_flow(struct flowi4 *fl4,
int proto,
__be32 daddr, __be32 saddr,
- __be32 key, __u8 tos, int oif)
+ __be32 key, __u8 tos, int oif,
+ __u32 mark)
{
memset(fl4, 0, sizeof(*fl4));
fl4->flowi4_oif = oif;
@@ -302,6 +303,7 @@
fl4->flowi4_tos = tos;
fl4->flowi4_proto = proto;
fl4->fl4_gre_key = key;
+ fl4->flowi4_mark = mark;
}
static int ip_tunnel_bind_dev(struct net_device *dev)
@@ -322,7 +324,8 @@
init_tunnel_flow(&fl4, iph->protocol, iph->daddr,
iph->saddr, tunnel->parms.o_key,
- RT_TOS(iph->tos), tunnel->parms.link);
+ RT_TOS(iph->tos), tunnel->parms.link,
+ tunnel->fwmark);
rt = ip_route_output_key(tunnel->net, &fl4);
if (!IS_ERR(rt)) {
@@ -578,7 +581,7 @@
tos = ipv6_get_dsfield((const struct ipv6hdr *)inner_iph);
}
init_tunnel_flow(&fl4, proto, key->u.ipv4.dst, key->u.ipv4.src, 0,
- RT_TOS(tos), tunnel->parms.link);
+ RT_TOS(tos), tunnel->parms.link, tunnel->fwmark);
if (tunnel->encap.type != TUNNEL_ENCAP_NONE)
goto tx_error;
rt = ip_route_output_key(tunnel->net, &fl4);
@@ -707,7 +710,8 @@
}
init_tunnel_flow(&fl4, protocol, dst, tnl_params->saddr,
- tunnel->parms.o_key, RT_TOS(tos), tunnel->parms.link);
+ tunnel->parms.o_key, RT_TOS(tos), tunnel->parms.link,
+ tunnel->fwmark);
if (ip_tunnel_encap(skb, tunnel, &protocol, &fl4) < 0)
goto tx_error;
@@ -795,7 +799,8 @@
struct ip_tunnel *t,
struct net_device *dev,
struct ip_tunnel_parm *p,
- bool set_mtu)
+ bool set_mtu,
+ __u32 fwmark)
{
ip_tunnel_del(itn, t);
t->parms.iph.saddr = p->iph.saddr;
@@ -812,10 +817,11 @@
t->parms.iph.tos = p->iph.tos;
t->parms.iph.frag_off = p->iph.frag_off;
- if (t->parms.link != p->link) {
+ if (t->parms.link != p->link || t->fwmark != fwmark) {
int mtu;
t->parms.link = p->link;
+ t->fwmark = fwmark;
mtu = ip_tunnel_bind_dev(dev);
if (set_mtu)
dev->mtu = mtu;
@@ -893,7 +899,7 @@
if (t) {
err = 0;
- ip_tunnel_update(itn, t, dev, p, true);
+ ip_tunnel_update(itn, t, dev, p, true, 0);
} else {
err = -ENOENT;
}
@@ -1066,7 +1072,7 @@
EXPORT_SYMBOL_GPL(ip_tunnel_delete_net);
int ip_tunnel_newlink(struct net_device *dev, struct nlattr *tb[],
- struct ip_tunnel_parm *p)
+ struct ip_tunnel_parm *p, __u32 fwmark)
{
struct ip_tunnel *nt;
struct net *net = dev_net(dev);
@@ -1087,6 +1093,7 @@
nt->net = net;
nt->parms = *p;
+ nt->fwmark = fwmark;
err = register_netdevice(dev);
if (err)
goto out;
@@ -1105,7 +1112,7 @@
EXPORT_SYMBOL_GPL(ip_tunnel_newlink);
int ip_tunnel_changelink(struct net_device *dev, struct nlattr *tb[],
- struct ip_tunnel_parm *p)
+ struct ip_tunnel_parm *p, __u32 fwmark)
{
struct ip_tunnel *t;
struct ip_tunnel *tunnel = netdev_priv(dev);
@@ -1137,7 +1144,7 @@
}
}
- ip_tunnel_update(itn, t, dev, p, !tb[IFLA_MTU]);
+ ip_tunnel_update(itn, t, dev, p, !tb[IFLA_MTU], fwmark);
return 0;
}
EXPORT_SYMBOL_GPL(ip_tunnel_changelink);
diff --git a/net/ipv4/ip_tunnel_core.c b/net/ipv4/ip_tunnel_core.c
index a31f47c..baf196e 100644
--- a/net/ipv4/ip_tunnel_core.c
+++ b/net/ipv4/ip_tunnel_core.c
@@ -235,7 +235,7 @@
struct nlattr *tb[LWTUNNEL_IP_MAX + 1];
int err;
- err = nla_parse_nested(tb, LWTUNNEL_IP_MAX, attr, ip_tun_policy);
+ err = nla_parse_nested(tb, LWTUNNEL_IP_MAX, attr, ip_tun_policy, NULL);
if (err < 0)
return err;
@@ -332,7 +332,8 @@
struct nlattr *tb[LWTUNNEL_IP6_MAX + 1];
int err;
- err = nla_parse_nested(tb, LWTUNNEL_IP6_MAX, attr, ip6_tun_policy);
+ err = nla_parse_nested(tb, LWTUNNEL_IP6_MAX, attr, ip6_tun_policy,
+ NULL);
if (err < 0)
return err;
diff --git a/net/ipv4/ip_vti.c b/net/ipv4/ip_vti.c
index 8b14f14..4097741 100644
--- a/net/ipv4/ip_vti.c
+++ b/net/ipv4/ip_vti.c
@@ -471,7 +471,8 @@
}
static void vti_netlink_parms(struct nlattr *data[],
- struct ip_tunnel_parm *parms)
+ struct ip_tunnel_parm *parms,
+ __u32 *fwmark)
{
memset(parms, 0, sizeof(*parms));
@@ -497,24 +498,29 @@
if (data[IFLA_VTI_REMOTE])
parms->iph.daddr = nla_get_in_addr(data[IFLA_VTI_REMOTE]);
+ if (data[IFLA_VTI_FWMARK])
+ *fwmark = nla_get_u32(data[IFLA_VTI_FWMARK]);
}
static int vti_newlink(struct net *src_net, struct net_device *dev,
struct nlattr *tb[], struct nlattr *data[])
{
struct ip_tunnel_parm parms;
+ __u32 fwmark = 0;
- vti_netlink_parms(data, &parms);
- return ip_tunnel_newlink(dev, tb, &parms);
+ vti_netlink_parms(data, &parms, &fwmark);
+ return ip_tunnel_newlink(dev, tb, &parms, fwmark);
}
static int vti_changelink(struct net_device *dev, struct nlattr *tb[],
struct nlattr *data[])
{
+ struct ip_tunnel *t = netdev_priv(dev);
+ __u32 fwmark = t->fwmark;
struct ip_tunnel_parm p;
- vti_netlink_parms(data, &p);
- return ip_tunnel_changelink(dev, tb, &p);
+ vti_netlink_parms(data, &p, &fwmark);
+ return ip_tunnel_changelink(dev, tb, &p, fwmark);
}
static size_t vti_get_size(const struct net_device *dev)
@@ -530,6 +536,8 @@
nla_total_size(4) +
/* IFLA_VTI_REMOTE */
nla_total_size(4) +
+ /* IFLA_VTI_FWMARK */
+ nla_total_size(4) +
0;
}
@@ -543,6 +551,7 @@
nla_put_be32(skb, IFLA_VTI_OKEY, p->o_key);
nla_put_in_addr(skb, IFLA_VTI_LOCAL, p->iph.saddr);
nla_put_in_addr(skb, IFLA_VTI_REMOTE, p->iph.daddr);
+ nla_put_u32(skb, IFLA_VTI_FWMARK, t->fwmark);
return 0;
}
@@ -553,6 +562,7 @@
[IFLA_VTI_OKEY] = { .type = NLA_U32 },
[IFLA_VTI_LOCAL] = { .len = FIELD_SIZEOF(struct iphdr, saddr) },
[IFLA_VTI_REMOTE] = { .len = FIELD_SIZEOF(struct iphdr, daddr) },
+ [IFLA_VTI_FWMARK] = { .type = NLA_U32 },
};
static struct rtnl_link_ops vti_link_ops __read_mostly = {
diff --git a/net/ipv4/ipconfig.c b/net/ipv4/ipconfig.c
index dfb2ab2..c3b12b1 100644
--- a/net/ipv4/ipconfig.c
+++ b/net/ipv4/ipconfig.c
@@ -57,6 +57,7 @@
#include <linux/export.h>
#include <net/net_namespace.h>
#include <net/arp.h>
+#include <net/dsa.h>
#include <net/ip.h>
#include <net/ipconfig.h>
#include <net/route.h>
diff --git a/net/ipv4/ipip.c b/net/ipv4/ipip.c
index 00d4229..1e441c6 100644
--- a/net/ipv4/ipip.c
+++ b/net/ipv4/ipip.c
@@ -390,7 +390,8 @@
}
static void ipip_netlink_parms(struct nlattr *data[],
- struct ip_tunnel_parm *parms, bool *collect_md)
+ struct ip_tunnel_parm *parms, bool *collect_md,
+ __u32 *fwmark)
{
memset(parms, 0, sizeof(*parms));
@@ -428,6 +429,9 @@
if (data[IFLA_IPTUN_COLLECT_METADATA])
*collect_md = true;
+
+ if (data[IFLA_IPTUN_FWMARK])
+ *fwmark = nla_get_u32(data[IFLA_IPTUN_FWMARK]);
}
/* This function returns true when ENCAP attributes are present in the nl msg */
@@ -470,6 +474,7 @@
struct ip_tunnel *t = netdev_priv(dev);
struct ip_tunnel_parm p;
struct ip_tunnel_encap ipencap;
+ __u32 fwmark = 0;
if (ipip_netlink_encap_parms(data, &ipencap)) {
int err = ip_tunnel_encap_setup(t, &ipencap);
@@ -478,26 +483,27 @@
return err;
}
- ipip_netlink_parms(data, &p, &t->collect_md);
- return ip_tunnel_newlink(dev, tb, &p);
+ ipip_netlink_parms(data, &p, &t->collect_md, &fwmark);
+ return ip_tunnel_newlink(dev, tb, &p, fwmark);
}
static int ipip_changelink(struct net_device *dev, struct nlattr *tb[],
struct nlattr *data[])
{
+ struct ip_tunnel *t = netdev_priv(dev);
struct ip_tunnel_parm p;
struct ip_tunnel_encap ipencap;
bool collect_md;
+ __u32 fwmark = t->fwmark;
if (ipip_netlink_encap_parms(data, &ipencap)) {
- struct ip_tunnel *t = netdev_priv(dev);
int err = ip_tunnel_encap_setup(t, &ipencap);
if (err < 0)
return err;
}
- ipip_netlink_parms(data, &p, &collect_md);
+ ipip_netlink_parms(data, &p, &collect_md, &fwmark);
if (collect_md)
return -EINVAL;
@@ -505,7 +511,7 @@
(!(dev->flags & IFF_POINTOPOINT) && p.iph.daddr))
return -EINVAL;
- return ip_tunnel_changelink(dev, tb, &p);
+ return ip_tunnel_changelink(dev, tb, &p, fwmark);
}
static size_t ipip_get_size(const struct net_device *dev)
@@ -535,6 +541,8 @@
nla_total_size(2) +
/* IFLA_IPTUN_COLLECT_METADATA */
nla_total_size(0) +
+ /* IFLA_IPTUN_FWMARK */
+ nla_total_size(4) +
0;
}
@@ -550,7 +558,8 @@
nla_put_u8(skb, IFLA_IPTUN_TOS, parm->iph.tos) ||
nla_put_u8(skb, IFLA_IPTUN_PROTO, parm->iph.protocol) ||
nla_put_u8(skb, IFLA_IPTUN_PMTUDISC,
- !!(parm->iph.frag_off & htons(IP_DF))))
+ !!(parm->iph.frag_off & htons(IP_DF))) ||
+ nla_put_u32(skb, IFLA_IPTUN_FWMARK, tunnel->fwmark))
goto nla_put_failure;
if (nla_put_u16(skb, IFLA_IPTUN_ENCAP_TYPE,
@@ -585,6 +594,7 @@
[IFLA_IPTUN_ENCAP_SPORT] = { .type = NLA_U16 },
[IFLA_IPTUN_ENCAP_DPORT] = { .type = NLA_U16 },
[IFLA_IPTUN_COLLECT_METADATA] = { .type = NLA_FLAG },
+ [IFLA_IPTUN_FWMARK] = { .type = NLA_U32 },
};
static struct rtnl_link_ops ipip_link_ops __read_mostly = {
diff --git a/net/ipv4/ipmr.c b/net/ipv4/ipmr.c
index b036e85..3a02d52 100644
--- a/net/ipv4/ipmr.c
+++ b/net/ipv4/ipmr.c
@@ -631,7 +631,7 @@
in_dev = __in_dev_get_rtnl(dev);
if (in_dev) {
IPV4_DEVCONF(in_dev->cnf, MC_FORWARDING)--;
- inet_netconf_notify_devconf(dev_net(dev),
+ inet_netconf_notify_devconf(dev_net(dev), RTM_NEWNETCONF,
NETCONFA_MC_FORWARDING,
dev->ifindex, &in_dev->cnf);
ip_rt_multicast_event(in_dev);
@@ -820,8 +820,8 @@
return -EADDRNOTAVAIL;
}
IPV4_DEVCONF(in_dev->cnf, MC_FORWARDING)++;
- inet_netconf_notify_devconf(net, NETCONFA_MC_FORWARDING, dev->ifindex,
- &in_dev->cnf);
+ inet_netconf_notify_devconf(net, RTM_NEWNETCONF, NETCONFA_MC_FORWARDING,
+ dev->ifindex, &in_dev->cnf);
ip_rt_multicast_event(in_dev);
/* Fill in the VIF structures */
@@ -1282,7 +1282,8 @@
ipmr_for_each_table(mrt, net) {
if (sk == rtnl_dereference(mrt->mroute_sk)) {
IPV4_DEVCONF_ALL(net, MC_FORWARDING)--;
- inet_netconf_notify_devconf(net, NETCONFA_MC_FORWARDING,
+ inet_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_MC_FORWARDING,
NETCONFA_IFINDEX_ALL,
net->ipv4.devconf_all);
RCU_INIT_POINTER(mrt->mroute_sk, NULL);
@@ -1343,7 +1344,8 @@
if (ret == 0) {
rcu_assign_pointer(mrt->mroute_sk, sk);
IPV4_DEVCONF_ALL(net, MC_FORWARDING)++;
- inet_netconf_notify_devconf(net, NETCONFA_MC_FORWARDING,
+ inet_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_MC_FORWARDING,
NETCONFA_IFINDEX_ALL,
net->ipv4.devconf_all);
}
@@ -2421,7 +2423,8 @@
/* returns < 0 on error, 0 for ADD_MFC and 1 for ADD_MFC_PROXY */
static int rtm_to_ipmr_mfcc(struct net *net, struct nlmsghdr *nlh,
struct mfcctl *mfcc, int *mrtsock,
- struct mr_table **mrtret)
+ struct mr_table **mrtret,
+ struct netlink_ext_ack *extack)
{
struct net_device *dev = NULL;
u32 tblid = RT_TABLE_DEFAULT;
@@ -2430,7 +2433,8 @@
struct rtmsg *rtm;
int ret, rem;
- ret = nlmsg_validate(nlh, sizeof(*rtm), RTA_MAX, rtm_ipmr_policy);
+ ret = nlmsg_validate(nlh, sizeof(*rtm), RTA_MAX, rtm_ipmr_policy,
+ extack);
if (ret < 0)
goto out;
rtm = nlmsg_data(nlh);
@@ -2489,7 +2493,8 @@
}
/* takes care of both newroute and delroute */
-static int ipmr_rtm_route(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int ipmr_rtm_route(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
int ret, mrtsock, parent;
@@ -2498,7 +2503,7 @@
mrtsock = 0;
tbl = NULL;
- ret = rtm_to_ipmr_mfcc(net, nlh, &mfcc, &mrtsock, &tbl);
+ ret = rtm_to_ipmr_mfcc(net, nlh, &mfcc, &mrtsock, &tbl, extack);
if (ret < 0)
return ret;
diff --git a/net/ipv4/netfilter/arp_tables.c b/net/ipv4/netfilter/arp_tables.c
index 6241a81..0bc3c3d 100644
--- a/net/ipv4/netfilter/arp_tables.c
+++ b/net/ipv4/netfilter/arp_tables.c
@@ -309,8 +309,7 @@
*/
for (hook = 0; hook < NF_ARP_NUMHOOKS; hook++) {
unsigned int pos = newinfo->hook_entry[hook];
- struct arpt_entry *e
- = (struct arpt_entry *)(entry0 + pos);
+ struct arpt_entry *e = entry0 + pos;
if (!(valid_hooks & (1 << hook)))
continue;
@@ -354,14 +353,12 @@
if (pos == oldpos)
goto next;
- e = (struct arpt_entry *)
- (entry0 + pos);
+ e = entry0 + pos;
} while (oldpos == pos + e->next_offset);
/* Move along one */
size = e->next_offset;
- e = (struct arpt_entry *)
- (entry0 + pos + size);
+ e = entry0 + pos + size;
if (pos + size >= newinfo->size)
return 0;
e->counters.pcnt = pos;
@@ -376,16 +373,14 @@
if (!xt_find_jump_offset(offsets, newpos,
newinfo->number))
return 0;
- e = (struct arpt_entry *)
- (entry0 + newpos);
+ e = entry0 + newpos;
} else {
/* ... this is a fallthru */
newpos = pos + e->next_offset;
if (newpos >= newinfo->size)
return 0;
}
- e = (struct arpt_entry *)
- (entry0 + newpos);
+ e = entry0 + newpos;
e->counters.pcnt = pos;
pos = newpos;
}
@@ -562,8 +557,6 @@
XT_ERROR_TARGET) == 0)
++newinfo->stacksize;
}
- if (ret != 0)
- goto out_free;
ret = -EINVAL;
if (i != repl->num_entries)
@@ -683,7 +676,7 @@
for (off = 0, num = 0; off < total_size; off += e->next_offset, num++){
const struct xt_entry_target *t;
- e = (struct arpt_entry *)(loc_cpu_entry + off);
+ e = loc_cpu_entry + off;
if (copy_to_user(userptr + off, e, sizeof(*e))) {
ret = -EFAULT;
goto free_counters;
@@ -1130,7 +1123,7 @@
int h;
origsize = *size;
- de = (struct arpt_entry *)*dstptr;
+ de = *dstptr;
memcpy(de, e, sizeof(struct arpt_entry));
memcpy(&de->counters, &e->counters, sizeof(e->counters));
@@ -1324,7 +1317,7 @@
int ret;
origsize = *size;
- ce = (struct compat_arpt_entry __user *)*dstptr;
+ ce = *dstptr;
if (copy_to_user(ce, e, sizeof(struct arpt_entry)) != 0 ||
copy_to_user(&ce->counters, &counters[i],
sizeof(counters[i])) != 0)
diff --git a/net/ipv4/netfilter/ip_tables.c b/net/ipv4/netfilter/ip_tables.c
index 384b857..2a55a40 100644
--- a/net/ipv4/netfilter/ip_tables.c
+++ b/net/ipv4/netfilter/ip_tables.c
@@ -382,7 +382,7 @@
to 0 as we leave), and comefrom to save source hook bitmask */
for (hook = 0; hook < NF_INET_NUMHOOKS; hook++) {
unsigned int pos = newinfo->hook_entry[hook];
- struct ipt_entry *e = (struct ipt_entry *)(entry0 + pos);
+ struct ipt_entry *e = entry0 + pos;
if (!(valid_hooks & (1 << hook)))
continue;
@@ -424,14 +424,12 @@
if (pos == oldpos)
goto next;
- e = (struct ipt_entry *)
- (entry0 + pos);
+ e = entry0 + pos;
} while (oldpos == pos + e->next_offset);
/* Move along one */
size = e->next_offset;
- e = (struct ipt_entry *)
- (entry0 + pos + size);
+ e = entry0 + pos + size;
if (pos + size >= newinfo->size)
return 0;
e->counters.pcnt = pos;
@@ -446,16 +444,14 @@
if (!xt_find_jump_offset(offsets, newpos,
newinfo->number))
return 0;
- e = (struct ipt_entry *)
- (entry0 + newpos);
+ e = entry0 + newpos;
} else {
/* ... this is a fallthru */
newpos = pos + e->next_offset;
if (newpos >= newinfo->size)
return 0;
}
- e = (struct ipt_entry *)
- (entry0 + newpos);
+ e = entry0 + newpos;
e->counters.pcnt = pos;
pos = newpos;
}
@@ -834,7 +830,7 @@
const struct xt_entry_match *m;
const struct xt_entry_target *t;
- e = (struct ipt_entry *)(loc_cpu_entry + off);
+ e = loc_cpu_entry + off;
if (copy_to_user(userptr + off, e, sizeof(*e))) {
ret = -EFAULT;
goto free_counters;
@@ -1229,7 +1225,7 @@
int ret = 0;
origsize = *size;
- ce = (struct compat_ipt_entry __user *)*dstptr;
+ ce = *dstptr;
if (copy_to_user(ce, e, sizeof(struct ipt_entry)) != 0 ||
copy_to_user(&ce->counters, &counters[i],
sizeof(counters[i])) != 0)
@@ -1366,7 +1362,7 @@
struct xt_entry_match *ematch;
origsize = *size;
- de = (struct ipt_entry *)*dstptr;
+ de = *dstptr;
memcpy(de, e, sizeof(struct ipt_entry));
memcpy(&de->counters, &e->counters, sizeof(e->counters));
diff --git a/net/ipv4/netfilter/ipt_CLUSTERIP.c b/net/ipv4/netfilter/ipt_CLUSTERIP.c
index 9b88413..038f293 100644
--- a/net/ipv4/netfilter/ipt_CLUSTERIP.c
+++ b/net/ipv4/netfilter/ipt_CLUSTERIP.c
@@ -22,6 +22,7 @@
#include <linux/icmp.h>
#include <linux/if_arp.h>
#include <linux/seq_file.h>
+#include <linux/refcount.h>
#include <linux/netfilter_arp.h>
#include <linux/netfilter/x_tables.h>
#include <linux/netfilter_ipv4/ip_tables.h>
@@ -40,8 +41,8 @@
struct clusterip_config {
struct list_head list; /* list of all configs */
- atomic_t refcount; /* reference count */
- atomic_t entries; /* number of entries/rules
+ refcount_t refcount; /* reference count */
+ refcount_t entries; /* number of entries/rules
* referencing us */
__be32 clusterip; /* the IP address */
@@ -77,7 +78,7 @@
static inline void
clusterip_config_get(struct clusterip_config *c)
{
- atomic_inc(&c->refcount);
+ refcount_inc(&c->refcount);
}
@@ -89,7 +90,7 @@
static inline void
clusterip_config_put(struct clusterip_config *c)
{
- if (atomic_dec_and_test(&c->refcount))
+ if (refcount_dec_and_test(&c->refcount))
call_rcu_bh(&c->rcu, clusterip_config_rcu_free);
}
@@ -103,7 +104,7 @@
struct clusterip_net *cn = net_generic(net, clusterip_net_id);
local_bh_disable();
- if (atomic_dec_and_lock(&c->entries, &cn->lock)) {
+ if (refcount_dec_and_lock(&c->entries, &cn->lock)) {
list_del_rcu(&c->list);
spin_unlock(&cn->lock);
local_bh_enable();
@@ -149,10 +150,10 @@
c = NULL;
else
#endif
- if (unlikely(!atomic_inc_not_zero(&c->refcount)))
+ if (unlikely(!refcount_inc_not_zero(&c->refcount)))
c = NULL;
else if (entry)
- atomic_inc(&c->entries);
+ refcount_inc(&c->entries);
}
rcu_read_unlock_bh();
@@ -188,8 +189,8 @@
clusterip_config_init_nodelist(c, i);
c->hash_mode = i->hash_mode;
c->hash_initval = i->hash_initval;
- atomic_set(&c->refcount, 1);
- atomic_set(&c->entries, 1);
+ refcount_set(&c->refcount, 1);
+ refcount_set(&c->entries, 1);
spin_lock_bh(&cn->lock);
if (__clusterip_config_find(net, ip)) {
diff --git a/net/ipv4/netfilter/ipt_SYNPROXY.c b/net/ipv4/netfilter/ipt_SYNPROXY.c
index 3240a26..af2b69b 100644
--- a/net/ipv4/netfilter/ipt_SYNPROXY.c
+++ b/net/ipv4/netfilter/ipt_SYNPROXY.c
@@ -293,12 +293,16 @@
XT_SYNPROXY_OPT_ECN);
synproxy_send_client_synack(net, skb, th, &opts);
- return NF_DROP;
-
+ consume_skb(skb);
+ return NF_STOLEN;
} else if (th->ack && !(th->fin || th->rst || th->syn)) {
/* ACK from client */
- synproxy_recv_client_ack(net, skb, th, &opts, ntohl(th->seq));
- return NF_DROP;
+ if (synproxy_recv_client_ack(net, skb, th, &opts, ntohl(th->seq))) {
+ consume_skb(skb);
+ return NF_STOLEN;
+ } else {
+ return NF_DROP;
+ }
}
return XT_CONTINUE;
@@ -367,10 +371,13 @@
* number match the one of first SYN.
*/
if (synproxy_recv_client_ack(net, skb, th, &opts,
- ntohl(th->seq) + 1))
+ ntohl(th->seq) + 1)) {
this_cpu_inc(snet->stats->cookie_retrans);
-
- return NF_DROP;
+ consume_skb(skb);
+ return NF_STOLEN;
+ } else {
+ return NF_DROP;
+ }
}
synproxy->isn = ntohl(th->ack_seq);
@@ -409,33 +416,6 @@
return NF_ACCEPT;
}
-static int synproxy_tg4_check(const struct xt_tgchk_param *par)
-{
- const struct ipt_entry *e = par->entryinfo;
-
- if (e->ip.proto != IPPROTO_TCP ||
- e->ip.invflags & XT_INV_PROTO)
- return -EINVAL;
-
- return nf_ct_netns_get(par->net, par->family);
-}
-
-static void synproxy_tg4_destroy(const struct xt_tgdtor_param *par)
-{
- nf_ct_netns_put(par->net, par->family);
-}
-
-static struct xt_target synproxy_tg4_reg __read_mostly = {
- .name = "SYNPROXY",
- .family = NFPROTO_IPV4,
- .hooks = (1 << NF_INET_LOCAL_IN) | (1 << NF_INET_FORWARD),
- .target = synproxy_tg4,
- .targetsize = sizeof(struct xt_synproxy_info),
- .checkentry = synproxy_tg4_check,
- .destroy = synproxy_tg4_destroy,
- .me = THIS_MODULE,
-};
-
static struct nf_hook_ops ipv4_synproxy_ops[] __read_mostly = {
{
.hook = ipv4_synproxy_hook,
@@ -451,31 +431,63 @@
},
};
-static int __init synproxy_tg4_init(void)
+static int synproxy_tg4_check(const struct xt_tgchk_param *par)
{
+ struct synproxy_net *snet = synproxy_pernet(par->net);
+ const struct ipt_entry *e = par->entryinfo;
int err;
- err = nf_register_hooks(ipv4_synproxy_ops,
- ARRAY_SIZE(ipv4_synproxy_ops));
- if (err < 0)
- goto err1;
+ if (e->ip.proto != IPPROTO_TCP ||
+ e->ip.invflags & XT_INV_PROTO)
+ return -EINVAL;
- err = xt_register_target(&synproxy_tg4_reg);
- if (err < 0)
- goto err2;
+ err = nf_ct_netns_get(par->net, par->family);
+ if (err)
+ return err;
- return 0;
+ if (snet->hook_ref4 == 0) {
+ err = nf_register_net_hooks(par->net, ipv4_synproxy_ops,
+ ARRAY_SIZE(ipv4_synproxy_ops));
+ if (err) {
+ nf_ct_netns_put(par->net, par->family);
+ return err;
+ }
+ }
-err2:
- nf_unregister_hooks(ipv4_synproxy_ops, ARRAY_SIZE(ipv4_synproxy_ops));
-err1:
+ snet->hook_ref4++;
return err;
}
+static void synproxy_tg4_destroy(const struct xt_tgdtor_param *par)
+{
+ struct synproxy_net *snet = synproxy_pernet(par->net);
+
+ snet->hook_ref4--;
+ if (snet->hook_ref4 == 0)
+ nf_unregister_net_hooks(par->net, ipv4_synproxy_ops,
+ ARRAY_SIZE(ipv4_synproxy_ops));
+ nf_ct_netns_put(par->net, par->family);
+}
+
+static struct xt_target synproxy_tg4_reg __read_mostly = {
+ .name = "SYNPROXY",
+ .family = NFPROTO_IPV4,
+ .hooks = (1 << NF_INET_LOCAL_IN) | (1 << NF_INET_FORWARD),
+ .target = synproxy_tg4,
+ .targetsize = sizeof(struct xt_synproxy_info),
+ .checkentry = synproxy_tg4_check,
+ .destroy = synproxy_tg4_destroy,
+ .me = THIS_MODULE,
+};
+
+static int __init synproxy_tg4_init(void)
+{
+ return xt_register_target(&synproxy_tg4_reg);
+}
+
static void __exit synproxy_tg4_exit(void)
{
xt_unregister_target(&synproxy_tg4_reg);
- nf_unregister_hooks(ipv4_synproxy_ops, ARRAY_SIZE(ipv4_synproxy_ops));
}
module_init(synproxy_tg4_init);
diff --git a/net/ipv4/netfilter/nf_dup_ipv4.c b/net/ipv4/netfilter/nf_dup_ipv4.c
index f0dbff0..39895b9 100644
--- a/net/ipv4/netfilter/nf_dup_ipv4.c
+++ b/net/ipv4/netfilter/nf_dup_ipv4.c
@@ -69,8 +69,7 @@
#if IS_ENABLED(CONFIG_NF_CONNTRACK)
/* Avoid counting cloned packets towards the original connection. */
nf_reset(skb);
- nf_ct_set(skb, nf_ct_untracked_get(), IP_CT_NEW);
- nf_conntrack_get(skb_nfct(skb));
+ nf_ct_set(skb, NULL, IP_CT_UNTRACKED);
#endif
/*
* If we are in PREROUTING/INPUT, decrease the TTL to mitigate potential
diff --git a/net/ipv4/netfilter/nf_nat_l3proto_ipv4.c b/net/ipv4/netfilter/nf_nat_l3proto_ipv4.c
index 6f5e8d0..feedd75 100644
--- a/net/ipv4/netfilter/nf_nat_l3proto_ipv4.c
+++ b/net/ipv4/netfilter/nf_nat_l3proto_ipv4.c
@@ -264,13 +264,7 @@
if (!ct)
return NF_ACCEPT;
- /* Don't try to NAT if this packet is not conntracked */
- if (nf_ct_is_untracked(ct))
- return NF_ACCEPT;
-
- nat = nf_ct_nat_ext_add(ct);
- if (nat == NULL)
- return NF_ACCEPT;
+ nat = nfct_nat(ct);
switch (ctinfo) {
case IP_CT_RELATED:
diff --git a/net/ipv4/netfilter/nf_nat_masquerade_ipv4.c b/net/ipv4/netfilter/nf_nat_masquerade_ipv4.c
index ea91058..dc1dea1 100644
--- a/net/ipv4/netfilter/nf_nat_masquerade_ipv4.c
+++ b/net/ipv4/netfilter/nf_nat_masquerade_ipv4.c
@@ -37,7 +37,6 @@
NF_CT_ASSERT(hooknum == NF_INET_POST_ROUTING);
ct = nf_ct_get(skb, &ctinfo);
- nat = nfct_nat(ct);
NF_CT_ASSERT(ct && (ctinfo == IP_CT_NEW || ctinfo == IP_CT_RELATED ||
ctinfo == IP_CT_RELATED_REPLY));
@@ -56,7 +55,9 @@
return NF_DROP;
}
- nat->masq_index = out->ifindex;
+ nat = nf_ct_nat_ext_add(ct);
+ if (nat)
+ nat->masq_index = out->ifindex;
/* Transfer from original range. */
memset(&newrange.min_addr, 0, sizeof(newrange.min_addr));
diff --git a/net/ipv4/netfilter/nf_nat_pptp.c b/net/ipv4/netfilter/nf_nat_pptp.c
index b3ca21b..8a69363 100644
--- a/net/ipv4/netfilter/nf_nat_pptp.c
+++ b/net/ipv4/netfilter/nf_nat_pptp.c
@@ -49,9 +49,14 @@
const struct nf_ct_pptp_master *ct_pptp_info;
const struct nf_nat_pptp *nat_pptp_info;
struct nf_nat_range range;
+ struct nf_conn_nat *nat;
+ nat = nf_ct_nat_ext_add(ct);
+ if (WARN_ON_ONCE(!nat))
+ return;
+
+ nat_pptp_info = &nat->help.nat_pptp_info;
ct_pptp_info = nfct_help_data(master);
- nat_pptp_info = &nfct_nat(master)->help.nat_pptp_info;
/* And here goes the grand finale of corrosion... */
if (exp->dir == IP_CT_DIR_ORIGINAL) {
@@ -120,13 +125,17 @@
{
struct nf_ct_pptp_master *ct_pptp_info;
+ struct nf_conn_nat *nat = nfct_nat(ct);
struct nf_nat_pptp *nat_pptp_info;
u_int16_t msg;
__be16 new_callid;
unsigned int cid_off;
+ if (WARN_ON_ONCE(!nat))
+ return NF_DROP;
+
+ nat_pptp_info = &nat->help.nat_pptp_info;
ct_pptp_info = nfct_help_data(ct);
- nat_pptp_info = &nfct_nat(ct)->help.nat_pptp_info;
new_callid = ct_pptp_info->pns_call_id;
@@ -177,11 +186,11 @@
ntohs(REQ_CID(pptpReq, cid_off)), ntohs(new_callid));
/* mangle packet */
- if (nf_nat_mangle_tcp_packet(skb, ct, ctinfo, protoff,
- cid_off + sizeof(struct pptp_pkt_hdr) +
- sizeof(struct PptpControlHeader),
- sizeof(new_callid), (char *)&new_callid,
- sizeof(new_callid)) == 0)
+ if (!nf_nat_mangle_tcp_packet(skb, ct, ctinfo, protoff,
+ cid_off + sizeof(struct pptp_pkt_hdr) +
+ sizeof(struct PptpControlHeader),
+ sizeof(new_callid), (char *)&new_callid,
+ sizeof(new_callid)))
return NF_DROP;
return NF_ACCEPT;
}
@@ -191,11 +200,15 @@
struct nf_conntrack_expect *expect_reply)
{
const struct nf_conn *ct = expect_orig->master;
+ struct nf_conn_nat *nat = nfct_nat(ct);
struct nf_ct_pptp_master *ct_pptp_info;
struct nf_nat_pptp *nat_pptp_info;
+ if (WARN_ON_ONCE(!nat))
+ return;
+
+ nat_pptp_info = &nat->help.nat_pptp_info;
ct_pptp_info = nfct_help_data(ct);
- nat_pptp_info = &nfct_nat(ct)->help.nat_pptp_info;
/* save original PAC call ID in nat_info */
nat_pptp_info->pac_call_id = ct_pptp_info->pac_call_id;
@@ -223,11 +236,15 @@
union pptp_ctrl_union *pptpReq)
{
const struct nf_nat_pptp *nat_pptp_info;
+ struct nf_conn_nat *nat = nfct_nat(ct);
u_int16_t msg;
__be16 new_pcid;
unsigned int pcid_off;
- nat_pptp_info = &nfct_nat(ct)->help.nat_pptp_info;
+ if (WARN_ON_ONCE(!nat))
+ return NF_DROP;
+
+ nat_pptp_info = &nat->help.nat_pptp_info;
new_pcid = nat_pptp_info->pns_call_id;
switch (msg = ntohs(ctlh->messageType)) {
@@ -271,11 +288,11 @@
pr_debug("altering peer call id from 0x%04x to 0x%04x\n",
ntohs(REQ_CID(pptpReq, pcid_off)), ntohs(new_pcid));
- if (nf_nat_mangle_tcp_packet(skb, ct, ctinfo, protoff,
- pcid_off + sizeof(struct pptp_pkt_hdr) +
- sizeof(struct PptpControlHeader),
- sizeof(new_pcid), (char *)&new_pcid,
- sizeof(new_pcid)) == 0)
+ if (!nf_nat_mangle_tcp_packet(skb, ct, ctinfo, protoff,
+ pcid_off + sizeof(struct pptp_pkt_hdr) +
+ sizeof(struct PptpControlHeader),
+ sizeof(new_pcid), (char *)&new_pcid,
+ sizeof(new_pcid)))
return NF_DROP;
return NF_ACCEPT;
}
diff --git a/net/ipv4/netfilter/nf_nat_snmp_basic.c b/net/ipv4/netfilter/nf_nat_snmp_basic.c
index 53e49f5..d5b1e0b 100644
--- a/net/ipv4/netfilter/nf_nat_snmp_basic.c
+++ b/net/ipv4/netfilter/nf_nat_snmp_basic.c
@@ -827,8 +827,8 @@
return 1;
}
-static unsigned char snmp_request_decode(struct asn1_ctx *ctx,
- struct snmp_request *request)
+static unsigned char noinline_for_stack
+snmp_request_decode(struct asn1_ctx *ctx, struct snmp_request *request)
{
unsigned int cls, con, tag;
unsigned char *end;
@@ -920,10 +920,10 @@
}
}
-static unsigned char snmp_trap_decode(struct asn1_ctx *ctx,
- struct snmp_v1_trap *trap,
- const struct oct1_map *map,
- __sum16 *check)
+static unsigned char noinline_for_stack
+snmp_trap_decode(struct asn1_ctx *ctx, struct snmp_v1_trap *trap,
+ const struct oct1_map *map,
+ __sum16 *check)
{
unsigned int cls, con, tag, len;
unsigned char *end;
@@ -998,18 +998,6 @@
*
*****************************************************************************/
-static void hex_dump(const unsigned char *buf, size_t len)
-{
- size_t i;
-
- for (i = 0; i < len; i++) {
- if (i && !(i % 16))
- printk("\n");
- printk("%02x ", *(buf + i));
- }
- printk("\n");
-}
-
/*
* Parse and mangle SNMP message according to mapping.
* (And this is the fucking 'basic' method).
@@ -1026,7 +1014,8 @@
struct snmp_object *obj;
if (debug > 1)
- hex_dump(msg, len);
+ print_hex_dump(KERN_DEBUG, "", DUMP_PREFIX_NONE, 16, 1,
+ msg, len, 0);
asn1_open(&ctx, msg, len);
diff --git a/net/ipv4/netfilter/nf_reject_ipv4.c b/net/ipv4/netfilter/nf_reject_ipv4.c
index 146d861..7cd8d0d 100644
--- a/net/ipv4/netfilter/nf_reject_ipv4.c
+++ b/net/ipv4/netfilter/nf_reject_ipv4.c
@@ -104,7 +104,6 @@
void nf_send_reset(struct net *net, struct sk_buff *oldskb, int hook)
{
struct sk_buff *nskb;
- const struct iphdr *oiph;
struct iphdr *niph;
const struct tcphdr *oth;
struct tcphdr _oth;
@@ -116,8 +115,6 @@
if (skb_rtable(oldskb)->rt_flags & (RTCF_BROADCAST | RTCF_MULTICAST))
return;
- oiph = ip_hdr(oldskb);
-
nskb = alloc_skb(sizeof(struct iphdr) + sizeof(struct tcphdr) +
LL_MAX_HEADER, GFP_ATOMIC);
if (!nskb)
diff --git a/net/ipv4/netfilter/nf_socket_ipv4.c b/net/ipv4/netfilter/nf_socket_ipv4.c
index a83d558..e9293bd 100644
--- a/net/ipv4/netfilter/nf_socket_ipv4.c
+++ b/net/ipv4/netfilter/nf_socket_ipv4.c
@@ -139,7 +139,7 @@
* SNAT-ted connection.
*/
ct = nf_ct_get(skb, &ctinfo);
- if (ct && !nf_ct_is_untracked(ct) &&
+ if (ct &&
((iph->protocol != IPPROTO_ICMP &&
ctinfo == IP_CT_ESTABLISHED_REPLY) ||
(iph->protocol == IPPROTO_ICMP &&
diff --git a/net/ipv4/netfilter/nft_fib_ipv4.c b/net/ipv4/netfilter/nft_fib_ipv4.c
index 29812919..de3681d 100644
--- a/net/ipv4/netfilter/nft_fib_ipv4.c
+++ b/net/ipv4/netfilter/nft_fib_ipv4.c
@@ -90,7 +90,7 @@
if (nft_hook(pkt) == NF_INET_PRE_ROUTING &&
nft_fib_is_loopback(pkt->skb, nft_in(pkt))) {
- nft_fib_store_result(dest, priv->result, pkt,
+ nft_fib_store_result(dest, priv, pkt,
nft_in(pkt)->ifindex);
return;
}
@@ -99,7 +99,7 @@
if (ipv4_is_zeronet(iph->saddr)) {
if (ipv4_is_lbcast(iph->daddr) ||
ipv4_is_local_multicast(iph->daddr)) {
- nft_fib_store_result(dest, priv->result, pkt,
+ nft_fib_store_result(dest, priv, pkt,
get_ifindex(pkt->skb->dev));
return;
}
@@ -212,7 +212,7 @@
static struct nft_expr_type nft_fib4_type __read_mostly = {
.name = "fib",
- .select_ops = &nft_fib4_select_ops,
+ .select_ops = nft_fib4_select_ops,
.policy = nft_fib_policy,
.maxattr = NFTA_FIB_MAX,
.family = NFPROTO_IPV4,
diff --git a/net/ipv4/proc.c b/net/ipv4/proc.c
index 69cf49e..fa44e75 100644
--- a/net/ipv4/proc.c
+++ b/net/ipv4/proc.c
@@ -199,7 +199,6 @@
SNMP_MIB_ITEM("TW", LINUX_MIB_TIMEWAITED),
SNMP_MIB_ITEM("TWRecycled", LINUX_MIB_TIMEWAITRECYCLED),
SNMP_MIB_ITEM("TWKilled", LINUX_MIB_TIMEWAITKILLED),
- SNMP_MIB_ITEM("PAWSPassive", LINUX_MIB_PAWSPASSIVEREJECTED),
SNMP_MIB_ITEM("PAWSActive", LINUX_MIB_PAWSACTIVEREJECTED),
SNMP_MIB_ITEM("PAWSEstab", LINUX_MIB_PAWSESTABREJECTED),
SNMP_MIB_ITEM("DelayedACKs", LINUX_MIB_DELAYEDACKS),
@@ -282,6 +281,7 @@
SNMP_MIB_ITEM("TCPFastOpenPassiveFail", LINUX_MIB_TCPFASTOPENPASSIVEFAIL),
SNMP_MIB_ITEM("TCPFastOpenListenOverflow", LINUX_MIB_TCPFASTOPENLISTENOVERFLOW),
SNMP_MIB_ITEM("TCPFastOpenCookieReqd", LINUX_MIB_TCPFASTOPENCOOKIEREQD),
+ SNMP_MIB_ITEM("TCPFastOpenBlackhole", LINUX_MIB_TCPFASTOPENBLACKHOLE),
SNMP_MIB_ITEM("TCPSpuriousRtxHostQueues", LINUX_MIB_TCPSPURIOUS_RTX_HOSTQUEUES),
SNMP_MIB_ITEM("BusyPollRxPackets", LINUX_MIB_BUSYPOLLRXPACKETS),
SNMP_MIB_ITEM("TCPAutoCorking", LINUX_MIB_TCPAUTOCORKING),
diff --git a/net/ipv4/protocol.c b/net/ipv4/protocol.c
index 4b7c0ec..32a691b 100644
--- a/net/ipv4/protocol.c
+++ b/net/ipv4/protocol.c
@@ -28,7 +28,7 @@
#include <linux/spinlock.h>
#include <net/protocol.h>
-const struct net_protocol __rcu *inet_protos[MAX_INET_PROTOS] __read_mostly;
+struct net_protocol __rcu *inet_protos[MAX_INET_PROTOS] __read_mostly;
const struct net_offload __rcu *inet_offloads[MAX_INET_PROTOS] __read_mostly;
EXPORT_SYMBOL(inet_offloads);
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index acd69cf..655d9ee 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -1250,15 +1250,11 @@
static unsigned int ipv4_default_advmss(const struct dst_entry *dst)
{
- unsigned int advmss = dst_metric_raw(dst, RTAX_ADVMSS);
+ unsigned int header_size = sizeof(struct tcphdr) + sizeof(struct iphdr);
+ unsigned int advmss = max_t(unsigned int, dst->dev->mtu - header_size,
+ ip_rt_min_advmss);
- if (advmss == 0) {
- advmss = max_t(unsigned int, dst->dev->mtu - 40,
- ip_rt_min_advmss);
- if (advmss > 65535 - 40)
- advmss = 65535 - 40;
- }
- return advmss;
+ return min(advmss, IPV4_MAX_PMTU - header_size);
}
static unsigned int ipv4_mtu(const struct dst_entry *dst)
@@ -1734,45 +1730,97 @@
}
#ifdef CONFIG_IP_ROUTE_MULTIPATH
-
/* To make ICMP packets follow the right flow, the multipath hash is
- * calculated from the inner IP addresses in reverse order.
+ * calculated from the inner IP addresses.
*/
-static int ip_multipath_icmp_hash(struct sk_buff *skb)
+static void ip_multipath_l3_keys(const struct sk_buff *skb,
+ struct flow_keys *hash_keys)
{
const struct iphdr *outer_iph = ip_hdr(skb);
- struct icmphdr _icmph;
+ const struct iphdr *inner_iph;
const struct icmphdr *icmph;
struct iphdr _inner_iph;
- const struct iphdr *inner_iph;
+ struct icmphdr _icmph;
+
+ hash_keys->addrs.v4addrs.src = outer_iph->saddr;
+ hash_keys->addrs.v4addrs.dst = outer_iph->daddr;
+ if (likely(outer_iph->protocol != IPPROTO_ICMP))
+ return;
if (unlikely((outer_iph->frag_off & htons(IP_OFFSET)) != 0))
- goto standard_hash;
+ return;
icmph = skb_header_pointer(skb, outer_iph->ihl * 4, sizeof(_icmph),
&_icmph);
if (!icmph)
- goto standard_hash;
+ return;
if (icmph->type != ICMP_DEST_UNREACH &&
icmph->type != ICMP_REDIRECT &&
icmph->type != ICMP_TIME_EXCEEDED &&
- icmph->type != ICMP_PARAMETERPROB) {
- goto standard_hash;
- }
+ icmph->type != ICMP_PARAMETERPROB)
+ return;
inner_iph = skb_header_pointer(skb,
outer_iph->ihl * 4 + sizeof(_icmph),
sizeof(_inner_iph), &_inner_iph);
if (!inner_iph)
- goto standard_hash;
-
- return fib_multipath_hash(inner_iph->daddr, inner_iph->saddr);
-
-standard_hash:
- return fib_multipath_hash(outer_iph->saddr, outer_iph->daddr);
+ return;
+ hash_keys->addrs.v4addrs.src = inner_iph->saddr;
+ hash_keys->addrs.v4addrs.dst = inner_iph->daddr;
}
+/* if skb is set it will be used and fl4 can be NULL */
+int fib_multipath_hash(const struct fib_info *fi, const struct flowi4 *fl4,
+ const struct sk_buff *skb)
+{
+ struct net *net = fi->fib_net;
+ struct flow_keys hash_keys;
+ u32 mhash;
+
+ switch (net->ipv4.sysctl_fib_multipath_hash_policy) {
+ case 0:
+ memset(&hash_keys, 0, sizeof(hash_keys));
+ hash_keys.control.addr_type = FLOW_DISSECTOR_KEY_IPV4_ADDRS;
+ if (skb) {
+ ip_multipath_l3_keys(skb, &hash_keys);
+ } else {
+ hash_keys.addrs.v4addrs.src = fl4->saddr;
+ hash_keys.addrs.v4addrs.dst = fl4->daddr;
+ }
+ break;
+ case 1:
+ /* skb is currently provided only when forwarding */
+ if (skb) {
+ unsigned int flag = FLOW_DISSECTOR_F_STOP_AT_ENCAP;
+ struct flow_keys keys;
+
+ /* short-circuit if we already have L4 hash present */
+ if (skb->l4_hash)
+ return skb_get_hash_raw(skb) >> 1;
+ memset(&hash_keys, 0, sizeof(hash_keys));
+ skb_flow_dissect_flow_keys(skb, &keys, flag);
+ hash_keys.addrs.v4addrs.src = keys.addrs.v4addrs.src;
+ hash_keys.addrs.v4addrs.dst = keys.addrs.v4addrs.dst;
+ hash_keys.ports.src = keys.ports.src;
+ hash_keys.ports.dst = keys.ports.dst;
+ hash_keys.basic.ip_proto = keys.basic.ip_proto;
+ } else {
+ memset(&hash_keys, 0, sizeof(hash_keys));
+ hash_keys.control.addr_type = FLOW_DISSECTOR_KEY_IPV4_ADDRS;
+ hash_keys.addrs.v4addrs.src = fl4->saddr;
+ hash_keys.addrs.v4addrs.dst = fl4->daddr;
+ hash_keys.ports.src = fl4->fl4_sport;
+ hash_keys.ports.dst = fl4->fl4_dport;
+ hash_keys.basic.ip_proto = fl4->flowi4_proto;
+ }
+ break;
+ }
+ mhash = flow_hash_from_keys(&hash_keys);
+
+ return mhash >> 1;
+}
+EXPORT_SYMBOL_GPL(fib_multipath_hash);
#endif /* CONFIG_IP_ROUTE_MULTIPATH */
static int ip_mkroute_input(struct sk_buff *skb,
@@ -1782,12 +1830,8 @@
{
#ifdef CONFIG_IP_ROUTE_MULTIPATH
if (res->fi && res->fi->fib_nhs > 1) {
- int h;
+ int h = fib_multipath_hash(res->fi, NULL, skb);
- if (unlikely(ip_hdr(skb)->protocol == IPPROTO_ICMP))
- h = ip_multipath_icmp_hash(skb);
- else
- h = fib_multipath_hash(saddr, daddr);
fib_select_multipath(res, h);
}
#endif
@@ -2203,7 +2247,7 @@
*/
struct rtable *__ip_route_output_key_hash(struct net *net, struct flowi4 *fl4,
- int mp_hash)
+ const struct sk_buff *skb)
{
struct net_device *dev_out = NULL;
__u8 tos = RT_FL_TOS(fl4);
@@ -2359,13 +2403,14 @@
}
/* L3 master device is the loopback for that domain */
- dev_out = l3mdev_master_dev_rcu(dev_out) ? : net->loopback_dev;
+ dev_out = l3mdev_master_dev_rcu(FIB_RES_DEV(res)) ? :
+ net->loopback_dev;
fl4->flowi4_oif = dev_out->ifindex;
flags |= RTCF_LOCAL;
goto make_route;
}
- fib_select_path(net, &res, fl4, mp_hash);
+ fib_select_path(net, &res, fl4, skb);
dev_out = FIB_RES_DEV(res);
fl4->flowi4_oif = dev_out->ifindex;
@@ -2585,7 +2630,8 @@
return -EMSGSIZE;
}
-static int inet_rtm_getroute(struct sk_buff *in_skb, struct nlmsghdr *nlh)
+static int inet_rtm_getroute(struct sk_buff *in_skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(in_skb->sk);
struct rtmsg *rtm;
@@ -2601,7 +2647,8 @@
u32 table_id = RT_TABLE_MAIN;
kuid_t uid;
- err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_ipv4_policy);
+ err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_ipv4_policy,
+ extack);
if (err < 0)
goto errout;
@@ -2619,10 +2666,6 @@
skb_reset_mac_header(skb);
skb_reset_network_header(skb);
- /* Bugfix: need to give ip_route_input enough of an IP header to not gag. */
- ip_hdr(skb)->protocol = IPPROTO_UDP;
- skb_reserve(skb, MAX_HEADER + sizeof(struct iphdr));
-
src = tb[RTA_SRC] ? nla_get_in_addr(tb[RTA_SRC]) : 0;
dst = tb[RTA_DST] ? nla_get_in_addr(tb[RTA_DST]) : 0;
iif = tb[RTA_IIF] ? nla_get_u32(tb[RTA_IIF]) : 0;
@@ -2632,6 +2675,15 @@
else
uid = (iif ? INVALID_UID : current_uid());
+ /* Bugfix: need to give ip_route_input enough of an IP header to
+ * not gag.
+ */
+ ip_hdr(skb)->protocol = IPPROTO_UDP;
+ ip_hdr(skb)->saddr = src;
+ ip_hdr(skb)->daddr = dst;
+
+ skb_reserve(skb, MAX_HEADER + sizeof(struct iphdr));
+
memset(&fl4, 0, sizeof(fl4));
fl4.daddr = dst;
fl4.saddr = src;
diff --git a/net/ipv4/sysctl_net_ipv4.c b/net/ipv4/sysctl_net_ipv4.c
index d6880a6..86957e9 100644
--- a/net/ipv4/sysctl_net_ipv4.c
+++ b/net/ipv4/sysctl_net_ipv4.c
@@ -24,6 +24,7 @@
#include <net/cipso_ipv4.h>
#include <net/inet_frag.h>
#include <net/ping.h>
+#include <net/protocol.h>
static int zero;
static int one = 1;
@@ -294,6 +295,74 @@
return ret;
}
+static void proc_configure_early_demux(int enabled, int protocol)
+{
+ struct net_protocol *ipprot;
+#if IS_ENABLED(CONFIG_IPV6)
+ struct inet6_protocol *ip6prot;
+#endif
+
+ rcu_read_lock();
+
+ ipprot = rcu_dereference(inet_protos[protocol]);
+ if (ipprot)
+ ipprot->early_demux = enabled ? ipprot->early_demux_handler :
+ NULL;
+
+#if IS_ENABLED(CONFIG_IPV6)
+ ip6prot = rcu_dereference(inet6_protos[protocol]);
+ if (ip6prot)
+ ip6prot->early_demux = enabled ? ip6prot->early_demux_handler :
+ NULL;
+#endif
+ rcu_read_unlock();
+}
+
+static int proc_tcp_early_demux(struct ctl_table *table, int write,
+ void __user *buffer, size_t *lenp, loff_t *ppos)
+{
+ int ret = 0;
+
+ ret = proc_dointvec(table, write, buffer, lenp, ppos);
+
+ if (write && !ret) {
+ int enabled = init_net.ipv4.sysctl_tcp_early_demux;
+
+ proc_configure_early_demux(enabled, IPPROTO_TCP);
+ }
+
+ return ret;
+}
+
+static int proc_udp_early_demux(struct ctl_table *table, int write,
+ void __user *buffer, size_t *lenp, loff_t *ppos)
+{
+ int ret = 0;
+
+ ret = proc_dointvec(table, write, buffer, lenp, ppos);
+
+ if (write && !ret) {
+ int enabled = init_net.ipv4.sysctl_udp_early_demux;
+
+ proc_configure_early_demux(enabled, IPPROTO_UDP);
+ }
+
+ return ret;
+}
+
+static int proc_tfo_blackhole_detect_timeout(struct ctl_table *table,
+ int write,
+ void __user *buffer,
+ size_t *lenp, loff_t *ppos)
+{
+ int ret;
+
+ ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
+ if (write && ret == 0)
+ tcp_fastopen_active_timeout_reset();
+ return ret;
+}
+
static struct ctl_table ipv4_table[] = {
{
.procname = "tcp_timestamps",
@@ -344,6 +413,14 @@
.proc_handler = proc_tcp_fastopen_key,
},
{
+ .procname = "tcp_fastopen_blackhole_timeout_sec",
+ .data = &sysctl_tcp_fastopen_blackhole_timeout,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_tfo_blackhole_detect_timeout,
+ .extra1 = &zero,
+ },
+ {
.procname = "tcp_abort_on_overflow",
.data = &sysctl_tcp_abort_on_overflow,
.maxlen = sizeof(int),
@@ -750,6 +827,20 @@
.proc_handler = proc_dointvec
},
{
+ .procname = "udp_early_demux",
+ .data = &init_net.ipv4.sysctl_udp_early_demux,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_udp_early_demux
+ },
+ {
+ .procname = "tcp_early_demux",
+ .data = &init_net.ipv4.sysctl_tcp_early_demux,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_tcp_early_demux
+ },
+ {
.procname = "ip_default_ttl",
.data = &init_net.ipv4.sysctl_ip_default_ttl,
.maxlen = sizeof(int),
@@ -981,13 +1072,6 @@
.proc_handler = proc_dointvec
},
{
- .procname = "tcp_tw_recycle",
- .data = &init_net.ipv4.tcp_death_row.sysctl_tw_recycle,
- .maxlen = sizeof(int),
- .mode = 0644,
- .proc_handler = proc_dointvec
- },
- {
.procname = "tcp_max_syn_backlog",
.data = &init_net.ipv4.sysctl_max_syn_backlog,
.maxlen = sizeof(int),
@@ -1004,6 +1088,15 @@
.extra1 = &zero,
.extra2 = &one,
},
+ {
+ .procname = "fib_multipath_hash_policy",
+ .data = &init_net.ipv4.sysctl_fib_multipath_hash_policy,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = &zero,
+ .extra2 = &one,
+ },
#endif
{
.procname = "ip_unprivileged_port_start",
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 40ba424..1e4c76d 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -533,7 +533,7 @@
if (tp->urg_data & TCP_URG_VALID)
mask |= POLLPRI;
- } else if (sk->sk_state == TCP_SYN_SENT && inet_sk(sk)->defer_connect) {
+ } else if (state == TCP_SYN_SENT && inet_sk(sk)->defer_connect) {
/* Active TCP fastopen socket with defer_connect
* Return POLLOUT so application can call write()
* in order for kernel to generate SYN+data
@@ -2296,6 +2296,7 @@
tcp_clear_xmit_timers(sk);
__skb_queue_purge(&sk->sk_receive_queue);
tcp_write_queue_purge(sk);
+ tcp_fastopen_active_disable_ofo_check(sk);
skb_rbtree_purge(&tp->out_of_order_queue);
inet->inet_dport = 0;
@@ -2394,7 +2395,7 @@
u16 snd_wscale = opt.opt_val & 0xFFFF;
u16 rcv_wscale = opt.opt_val >> 16;
- if (snd_wscale > 14 || rcv_wscale > 14)
+ if (snd_wscale > TCP_MAX_WSCALE || rcv_wscale > TCP_MAX_WSCALE)
return -EFBIG;
tp->rx_opt.snd_wscale = snd_wscale;
@@ -2471,7 +2472,7 @@
/* Values greater than interface MTU won't take effect. However
* at the point when this call is done we typically don't yet
* know which interface is going to be used */
- if (val < TCP_MIN_MSS || val > MAX_TCP_WINDOW) {
+ if (val && (val < TCP_MIN_MSS || val > MAX_TCP_WINDOW)) {
err = -EINVAL;
break;
}
@@ -2852,7 +2853,7 @@
info->tcpi_snd_ssthresh = tp->snd_ssthresh;
info->tcpi_advmss = tp->advmss;
- info->tcpi_rcv_rtt = jiffies_to_usecs(tp->rcv_rtt_est.rtt)>>3;
+ info->tcpi_rcv_rtt = tp->rcv_rtt_est.rtt_us >> 3;
info->tcpi_rcv_space = tp->rcvq_space.space;
info->tcpi_total_retrans = tp->total_retrans;
diff --git a/net/ipv4/tcp_cong.c b/net/ipv4/tcp_cong.c
index 79c4817..6e3c512 100644
--- a/net/ipv4/tcp_cong.c
+++ b/net/ipv4/tcp_cong.c
@@ -168,12 +168,8 @@
}
out:
rcu_read_unlock();
+ memset(icsk->icsk_ca_priv, 0, sizeof(icsk->icsk_ca_priv));
- /* Clear out private data before diag gets it and
- * the ca has not been initialized.
- */
- if (ca->get_info)
- memset(icsk->icsk_ca_priv, 0, sizeof(icsk->icsk_ca_priv));
if (ca->flags & TCP_CONG_NEEDS_ECN)
INET_ECN_xmit(sk);
else
@@ -200,11 +196,10 @@
tcp_cleanup_congestion_control(sk);
icsk->icsk_ca_ops = ca;
icsk->icsk_ca_setsockopt = 1;
+ memset(icsk->icsk_ca_priv, 0, sizeof(icsk->icsk_ca_priv));
- if (sk->sk_state != TCP_CLOSE) {
- memset(icsk->icsk_ca_priv, 0, sizeof(icsk->icsk_ca_priv));
+ if (sk->sk_state != TCP_CLOSE)
tcp_init_congestion_control(sk);
- }
}
/* Manage refcounts on socket close. */
diff --git a/net/ipv4/tcp_cubic.c b/net/ipv4/tcp_cubic.c
index c99230e..0683ba4 100644
--- a/net/ipv4/tcp_cubic.c
+++ b/net/ipv4/tcp_cubic.c
@@ -72,7 +72,7 @@
module_param(hystart, int, 0644);
MODULE_PARM_DESC(hystart, "turn on/off hybrid slow start algorithm");
module_param(hystart_detect, int, 0644);
-MODULE_PARM_DESC(hystart_detect, "hyrbrid slow start detection mechanisms"
+MODULE_PARM_DESC(hystart_detect, "hybrid slow start detection mechanisms"
" 1: packet-train 2: delay 3: both packet-train and delay");
module_param(hystart_low_window, int, 0644);
MODULE_PARM_DESC(hystart_low_window, "lower bound cwnd for hybrid slow start");
diff --git a/net/ipv4/tcp_fastopen.c b/net/ipv4/tcp_fastopen.c
index 8ea4e97..4af82b9 100644
--- a/net/ipv4/tcp_fastopen.c
+++ b/net/ipv4/tcp_fastopen.c
@@ -341,6 +341,13 @@
cookie->len = -1;
return false;
}
+
+ /* Firewall blackhole issue check */
+ if (tcp_fastopen_active_should_disable(sk)) {
+ cookie->len = -1;
+ return false;
+ }
+
if (sysctl_tcp_fastopen & TFO_CLIENT_NO_COOKIE) {
cookie->len = -1;
return true;
@@ -380,3 +387,98 @@
return false;
}
EXPORT_SYMBOL(tcp_fastopen_defer_connect);
+
+/*
+ * The following code block is to deal with middle box issues with TFO:
+ * Middlebox firewall issues can potentially cause server's data being
+ * blackholed after a successful 3WHS using TFO.
+ * The proposed solution is to disable active TFO globally under the
+ * following circumstances:
+ * 1. client side TFO socket receives out of order FIN
+ * 2. client side TFO socket receives out of order RST
+ * We disable active side TFO globally for 1hr at first. Then if it
+ * happens again, we disable it for 2h, then 4h, 8h, ...
+ * And we reset the timeout back to 1hr when we see a successful active
+ * TFO connection with data exchanges.
+ */
+
+/* Default to 1hr */
+unsigned int sysctl_tcp_fastopen_blackhole_timeout __read_mostly = 60 * 60;
+static atomic_t tfo_active_disable_times __read_mostly = ATOMIC_INIT(0);
+static unsigned long tfo_active_disable_stamp __read_mostly;
+
+/* Disable active TFO and record current jiffies and
+ * tfo_active_disable_times
+ */
+void tcp_fastopen_active_disable(struct sock *sk)
+{
+ atomic_inc(&tfo_active_disable_times);
+ tfo_active_disable_stamp = jiffies;
+ NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPFASTOPENBLACKHOLE);
+}
+
+/* Reset tfo_active_disable_times to 0 */
+void tcp_fastopen_active_timeout_reset(void)
+{
+ atomic_set(&tfo_active_disable_times, 0);
+}
+
+/* Calculate timeout for tfo active disable
+ * Return true if we are still in the active TFO disable period
+ * Return false if timeout already expired and we should use active TFO
+ */
+bool tcp_fastopen_active_should_disable(struct sock *sk)
+{
+ int tfo_da_times = atomic_read(&tfo_active_disable_times);
+ int multiplier;
+ unsigned long timeout;
+
+ if (!tfo_da_times)
+ return false;
+
+ /* Limit timout to max: 2^6 * initial timeout */
+ multiplier = 1 << min(tfo_da_times - 1, 6);
+ timeout = multiplier * sysctl_tcp_fastopen_blackhole_timeout * HZ;
+ if (time_before(jiffies, tfo_active_disable_stamp + timeout))
+ return true;
+
+ /* Mark check bit so we can check for successful active TFO
+ * condition and reset tfo_active_disable_times
+ */
+ tcp_sk(sk)->syn_fastopen_ch = 1;
+ return false;
+}
+
+/* Disable active TFO if FIN is the only packet in the ofo queue
+ * and no data is received.
+ * Also check if we can reset tfo_active_disable_times if data is
+ * received successfully on a marked active TFO sockets opened on
+ * a non-loopback interface
+ */
+void tcp_fastopen_active_disable_ofo_check(struct sock *sk)
+{
+ struct tcp_sock *tp = tcp_sk(sk);
+ struct rb_node *p;
+ struct sk_buff *skb;
+ struct dst_entry *dst;
+
+ if (!tp->syn_fastopen)
+ return;
+
+ if (!tp->data_segs_in) {
+ p = rb_first(&tp->out_of_order_queue);
+ if (p && !rb_next(p)) {
+ skb = rb_entry(p, struct sk_buff, rbnode);
+ if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN) {
+ tcp_fastopen_active_disable(sk);
+ return;
+ }
+ }
+ } else if (tp->syn_fastopen_ch &&
+ atomic_read(&tfo_active_disable_times)) {
+ dst = sk_dst_get(sk);
+ if (!(dst && dst->dev && (dst->dev->flags & IFF_LOOPBACK)))
+ tcp_fastopen_active_timeout_reset();
+ dst_release(dst);
+ }
+}
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 659d1ba..9739962 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -442,7 +442,8 @@
tcp_sndbuf_expand(sk);
tp->rcvq_space.space = tp->rcv_wnd;
- tp->rcvq_space.time = tcp_time_stamp;
+ skb_mstamp_get(&tp->tcp_mstamp);
+ tp->rcvq_space.time = tp->tcp_mstamp;
tp->rcvq_space.seq = tp->copied_seq;
maxwin = tcp_full_space(sk);
@@ -518,7 +519,7 @@
*/
static void tcp_rcv_rtt_update(struct tcp_sock *tp, u32 sample, int win_dep)
{
- u32 new_sample = tp->rcv_rtt_est.rtt;
+ u32 new_sample = tp->rcv_rtt_est.rtt_us;
long m = sample;
if (m == 0)
@@ -548,21 +549,23 @@
new_sample = m << 3;
}
- if (tp->rcv_rtt_est.rtt != new_sample)
- tp->rcv_rtt_est.rtt = new_sample;
+ tp->rcv_rtt_est.rtt_us = new_sample;
}
static inline void tcp_rcv_rtt_measure(struct tcp_sock *tp)
{
- if (tp->rcv_rtt_est.time == 0)
+ u32 delta_us;
+
+ if (tp->rcv_rtt_est.time.v64 == 0)
goto new_measure;
if (before(tp->rcv_nxt, tp->rcv_rtt_est.seq))
return;
- tcp_rcv_rtt_update(tp, tcp_time_stamp - tp->rcv_rtt_est.time, 1);
+ delta_us = skb_mstamp_us_delta(&tp->tcp_mstamp, &tp->rcv_rtt_est.time);
+ tcp_rcv_rtt_update(tp, delta_us, 1);
new_measure:
tp->rcv_rtt_est.seq = tp->rcv_nxt + tp->rcv_wnd;
- tp->rcv_rtt_est.time = tcp_time_stamp;
+ tp->rcv_rtt_est.time = tp->tcp_mstamp;
}
static inline void tcp_rcv_rtt_measure_ts(struct sock *sk,
@@ -572,7 +575,10 @@
if (tp->rx_opt.rcv_tsecr &&
(TCP_SKB_CB(skb)->end_seq -
TCP_SKB_CB(skb)->seq >= inet_csk(sk)->icsk_ack.rcv_mss))
- tcp_rcv_rtt_update(tp, tcp_time_stamp - tp->rx_opt.rcv_tsecr, 0);
+ tcp_rcv_rtt_update(tp,
+ jiffies_to_usecs(tcp_time_stamp -
+ tp->rx_opt.rcv_tsecr),
+ 0);
}
/*
@@ -585,8 +591,8 @@
int time;
int copied;
- time = tcp_time_stamp - tp->rcvq_space.time;
- if (time < (tp->rcv_rtt_est.rtt >> 3) || tp->rcv_rtt_est.rtt == 0)
+ time = skb_mstamp_us_delta(&tp->tcp_mstamp, &tp->rcvq_space.time);
+ if (time < (tp->rcv_rtt_est.rtt_us >> 3) || tp->rcv_rtt_est.rtt_us == 0)
return;
/* Number of bytes copied to user in last RTT */
@@ -642,7 +648,7 @@
new_measure:
tp->rcvq_space.seq = tp->copied_seq;
- tp->rcvq_space.time = tcp_time_stamp;
+ tp->rcvq_space.time = tp->tcp_mstamp;
}
/* There is something which you must keep in mind when you analyze the
@@ -1131,7 +1137,6 @@
*/
struct skb_mstamp first_sackt;
struct skb_mstamp last_sackt;
- struct skb_mstamp ack_time; /* Timestamp when the S/ACK was received */
struct rate_sample *rate;
int flag;
};
@@ -1214,8 +1219,7 @@
return sacked;
if (!(sacked & TCPCB_SACKED_ACKED)) {
- tcp_rack_advance(tp, sacked, end_seq,
- xmit_time, &state->ack_time);
+ tcp_rack_advance(tp, sacked, end_seq, xmit_time);
if (sacked & TCPCB_SACKED_RETRANS) {
/* If the segment is not tagged as lost,
@@ -2760,8 +2764,7 @@
return false;
}
-static void tcp_rack_identify_loss(struct sock *sk, int *ack_flag,
- const struct skb_mstamp *ack_time)
+static void tcp_rack_identify_loss(struct sock *sk, int *ack_flag)
{
struct tcp_sock *tp = tcp_sk(sk);
@@ -2769,7 +2772,7 @@
if (sysctl_tcp_recovery & TCP_RACK_LOSS_DETECTION) {
u32 prior_retrans = tp->retrans_out;
- tcp_rack_mark_lost(sk, ack_time);
+ tcp_rack_mark_lost(sk);
if (prior_retrans > tp->retrans_out)
*ack_flag |= FLAG_LOST_RETRANS;
}
@@ -2788,8 +2791,7 @@
* tcp_xmit_retransmit_queue().
*/
static void tcp_fastretrans_alert(struct sock *sk, const int acked,
- bool is_dupack, int *ack_flag, int *rexmit,
- const struct skb_mstamp *ack_time)
+ bool is_dupack, int *ack_flag, int *rexmit)
{
struct inet_connection_sock *icsk = inet_csk(sk);
struct tcp_sock *tp = tcp_sk(sk);
@@ -2857,11 +2859,11 @@
tcp_try_keep_open(sk);
return;
}
- tcp_rack_identify_loss(sk, ack_flag, ack_time);
+ tcp_rack_identify_loss(sk, ack_flag);
break;
case TCP_CA_Loss:
tcp_process_loss(sk, flag, is_dupack, rexmit);
- tcp_rack_identify_loss(sk, ack_flag, ack_time);
+ tcp_rack_identify_loss(sk, ack_flag);
if (!(icsk->icsk_ca_state == TCP_CA_Open ||
(*ack_flag & FLAG_LOST_RETRANS)))
return;
@@ -2877,7 +2879,7 @@
if (icsk->icsk_ca_state <= TCP_CA_Disorder)
tcp_try_undo_dsack(sk);
- tcp_rack_identify_loss(sk, ack_flag, ack_time);
+ tcp_rack_identify_loss(sk, ack_flag);
if (!tcp_time_to_recover(sk, flag)) {
tcp_try_to_open(sk, flag);
return;
@@ -3059,8 +3061,8 @@
{
const struct inet_connection_sock *icsk = inet_csk(sk);
struct skb_mstamp first_ackt, last_ackt;
- struct skb_mstamp *now = &sack->ack_time;
struct tcp_sock *tp = tcp_sk(sk);
+ struct skb_mstamp *now = &tp->tcp_mstamp;
u32 prior_sacked = tp->sacked_out;
u32 reord = tp->packets_out;
bool fully_acked = true;
@@ -3120,8 +3122,7 @@
tp->delivered += acked_pcount;
if (!tcp_skb_spurious_retrans(tp, skb))
tcp_rack_advance(tp, sacked, scb->end_seq,
- &skb->skb_mstamp,
- &sack->ack_time);
+ &skb->skb_mstamp);
}
if (sacked & TCPCB_LOST)
tp->lost_out -= acked_pcount;
@@ -3576,8 +3577,6 @@
if (after(ack, tp->snd_nxt))
goto invalid_ack;
- skb_mstamp_get(&sack_state.ack_time);
-
if (icsk->icsk_pending == ICSK_TIME_LOSS_PROBE)
tcp_rearm_rto(sk);
@@ -3647,8 +3646,7 @@
if (tcp_ack_is_dubious(sk, flag)) {
is_dupack = !(flag & (FLAG_SND_UNA_ADVANCED | FLAG_NOT_DUP));
- tcp_fastretrans_alert(sk, acked, is_dupack, &flag, &rexmit,
- &sack_state.ack_time);
+ tcp_fastretrans_alert(sk, acked, is_dupack, &flag, &rexmit);
}
if (tp->tlp_high_seq)
tcp_process_tlp_ack(sk, ack, flag);
@@ -3660,8 +3658,7 @@
tcp_schedule_loss_probe(sk);
delivered = tp->delivered - delivered; /* freshly ACKed or SACKed */
lost = tp->lost - lost; /* freshly marked lost */
- tcp_rate_gen(sk, delivered, lost, &sack_state.ack_time,
- sack_state.rate);
+ tcp_rate_gen(sk, delivered, lost, sack_state.rate);
tcp_cong_control(sk, ack, delivered, flag, sack_state.rate);
tcp_xmit_recovery(sk, rexmit);
return 1;
@@ -3669,8 +3666,7 @@
no_queue:
/* If data was DSACKed, see if we can undo a cwnd reduction. */
if (flag & FLAG_DSACKING_ACK)
- tcp_fastretrans_alert(sk, acked, is_dupack, &flag, &rexmit,
- &sack_state.ack_time);
+ tcp_fastretrans_alert(sk, acked, is_dupack, &flag, &rexmit);
/* If this ack opens up a zero window, clear backoff. It was
* being used to time the probes, and is probably far higher than
* it needs to be for normal retransmission.
@@ -3691,11 +3687,9 @@
* If data was DSACKed, see if we can undo a cwnd reduction.
*/
if (TCP_SKB_CB(skb)->sacked) {
- skb_mstamp_get(&sack_state.ack_time);
flag |= tcp_sacktag_write_queue(sk, skb, prior_snd_una,
&sack_state);
- tcp_fastretrans_alert(sk, acked, is_dupack, &flag, &rexmit,
- &sack_state.ack_time);
+ tcp_fastretrans_alert(sk, acked, is_dupack, &flag, &rexmit);
tcp_xmit_recovery(sk, rexmit);
}
@@ -3768,11 +3762,12 @@
!estab && sysctl_tcp_window_scaling) {
__u8 snd_wscale = *(__u8 *)ptr;
opt_rx->wscale_ok = 1;
- if (snd_wscale > 14) {
- net_info_ratelimited("%s: Illegal window scaling value %d >14 received\n",
+ if (snd_wscale > TCP_MAX_WSCALE) {
+ net_info_ratelimited("%s: Illegal window scaling value %d > %u received\n",
__func__,
- snd_wscale);
- snd_wscale = 14;
+ snd_wscale,
+ TCP_MAX_WSCALE);
+ snd_wscale = TCP_MAX_WSCALE;
}
opt_rx->snd_wscale = snd_wscale;
}
@@ -4007,10 +4002,10 @@
/* This barrier is coupled with smp_rmb() in tcp_poll() */
smp_wmb();
+ tcp_done(sk);
+
if (!sock_flag(sk, SOCK_DEAD))
sk->sk_error_report(sk);
-
- tcp_done(sk);
}
/*
@@ -5299,8 +5294,16 @@
if (rst_seq_match)
tcp_reset(sk);
- else
+ else {
+ /* Disable TFO if RST is out-of-order
+ * and no data has been received
+ * for current active TFO socket
+ */
+ if (tp->syn_fastopen && !tp->data_segs_in &&
+ sk->sk_state == TCP_ESTABLISHED)
+ tcp_fastopen_active_disable(sk);
tcp_send_challenge_ack(sk, skb);
+ }
goto discard;
}
@@ -5353,6 +5356,7 @@
{
struct tcp_sock *tp = tcp_sk(sk);
+ skb_mstamp_get(&tp->tcp_mstamp);
if (unlikely(!sk->sk_rx_dst))
inet_csk(sk)->icsk_af_ops->sk_rx_dst_set(sk, skb);
/*
@@ -5579,10 +5583,6 @@
else
tp->pred_flags = 0;
- if (!sock_flag(sk, SOCK_DEAD)) {
- sk->sk_state_change(sk);
- sk_wake_async(sk, SOCK_WAKE_IO, POLL_OUT);
- }
}
static bool tcp_rcv_fastopen_synack(struct sock *sk, struct sk_buff *synack,
@@ -5651,6 +5651,7 @@
struct tcp_sock *tp = tcp_sk(sk);
struct tcp_fastopen_cookie foc = { .len = -1 };
int saved_clamp = tp->rx_opt.mss_clamp;
+ bool fastopen_fail;
tcp_parse_options(skb, &tp->rx_opt, 0, &foc);
if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
@@ -5754,10 +5755,15 @@
tcp_finish_connect(sk, skb);
- if ((tp->syn_fastopen || tp->syn_data) &&
- tcp_rcv_fastopen_synack(sk, skb, &foc))
- return -1;
+ fastopen_fail = (tp->syn_fastopen || tp->syn_data) &&
+ tcp_rcv_fastopen_synack(sk, skb, &foc);
+ if (!sock_flag(sk, SOCK_DEAD)) {
+ sk->sk_state_change(sk);
+ sk_wake_async(sk, SOCK_WAKE_IO, POLL_OUT);
+ }
+ if (fastopen_fail)
+ return -1;
if (sk->sk_write_pending ||
icsk->icsk_accept_queue.rskq_defer_accept ||
icsk->icsk_ack.pingpong) {
@@ -5911,6 +5917,7 @@
case TCP_SYN_SENT:
tp->rx_opt.saw_tstamp = 0;
+ skb_mstamp_get(&tp->tcp_mstamp);
queued = tcp_rcv_synsent_state_process(sk, skb, th);
if (queued >= 0)
return queued;
@@ -5922,6 +5929,7 @@
return 0;
}
+ skb_mstamp_get(&tp->tcp_mstamp);
tp->rx_opt.saw_tstamp = 0;
req = tp->fastopen_rsk;
if (req) {
@@ -6041,9 +6049,16 @@
break;
}
- if (tp->linger2 < 0 ||
- (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
- after(TCP_SKB_CB(skb)->end_seq - th->fin, tp->rcv_nxt))) {
+ if (tp->linger2 < 0) {
+ tcp_done(sk);
+ NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
+ return 1;
+ }
+ if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
+ after(TCP_SKB_CB(skb)->end_seq - th->fin, tp->rcv_nxt)) {
+ /* Receive out of order FIN after close() */
+ if (tp->syn_fastopen && th->fin)
+ tcp_fastopen_active_disable(sk);
tcp_done(sk);
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
return 1;
@@ -6333,36 +6348,14 @@
goto drop_and_free;
if (isn && tmp_opt.tstamp_ok)
- af_ops->init_seq(skb, &tcp_rsk(req)->ts_off);
+ af_ops->init_seq_tsoff(skb, &tcp_rsk(req)->ts_off);
if (!want_cookie && !isn) {
- /* VJ's idea. We save last timestamp seen
- * from the destination in peer table, when entering
- * state TIME-WAIT, and check against it before
- * accepting new connection request.
- *
- * If "isn" is not zero, this request hit alive
- * timewait bucket, so that all the necessary checks
- * are made in the function processing timewait state.
- */
- if (net->ipv4.tcp_death_row.sysctl_tw_recycle) {
- bool strict;
-
- dst = af_ops->route_req(sk, &fl, req, &strict);
-
- if (dst && strict &&
- !tcp_peer_is_proven(req, dst, true,
- tmp_opt.saw_tstamp)) {
- NET_INC_STATS(sock_net(sk), LINUX_MIB_PAWSPASSIVEREJECTED);
- goto drop_and_release;
- }
- }
/* Kill the following clause, if you dislike this way. */
- else if (!net->ipv4.sysctl_tcp_syncookies &&
- (net->ipv4.sysctl_max_syn_backlog - inet_csk_reqsk_queue_len(sk) <
- (net->ipv4.sysctl_max_syn_backlog >> 2)) &&
- !tcp_peer_is_proven(req, dst, false,
- tmp_opt.saw_tstamp)) {
+ if (!net->ipv4.sysctl_tcp_syncookies &&
+ (net->ipv4.sysctl_max_syn_backlog - inet_csk_reqsk_queue_len(sk) <
+ (net->ipv4.sysctl_max_syn_backlog >> 2)) &&
+ !tcp_peer_is_proven(req, dst)) {
/* Without syncookies last quarter of
* backlog is filled with destinations,
* proven to be alive.
@@ -6375,10 +6368,10 @@
goto drop_and_release;
}
- isn = af_ops->init_seq(skb, &tcp_rsk(req)->ts_off);
+ isn = af_ops->init_seq_tsoff(skb, &tcp_rsk(req)->ts_off);
}
if (!dst) {
- dst = af_ops->route_req(sk, &fl, req, NULL);
+ dst = af_ops->route_req(sk, &fl, req);
if (!dst)
goto drop_and_free;
}
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 575e19d..cbbafe5 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -94,12 +94,12 @@
struct inet_hashinfo tcp_hashinfo;
EXPORT_SYMBOL(tcp_hashinfo);
-static u32 tcp_v4_init_sequence(const struct sk_buff *skb, u32 *tsoff)
+static u32 tcp_v4_init_seq_and_tsoff(const struct sk_buff *skb, u32 *tsoff)
{
- return secure_tcp_sequence_number(ip_hdr(skb)->daddr,
- ip_hdr(skb)->saddr,
- tcp_hdr(skb)->dest,
- tcp_hdr(skb)->source, tsoff);
+ return secure_tcp_seq_and_tsoff(ip_hdr(skb)->daddr,
+ ip_hdr(skb)->saddr,
+ tcp_hdr(skb)->dest,
+ tcp_hdr(skb)->source, tsoff);
}
int tcp_twsk_unique(struct sock *sk, struct sock *sktw, void *twp)
@@ -198,10 +198,6 @@
tp->write_seq = 0;
}
- if (tcp_death_row->sysctl_tw_recycle &&
- !tp->rx_opt.ts_recent_stamp && fl4->daddr == daddr)
- tcp_fetch_timewait_stamp(sk, &rt->dst);
-
inet->inet_dport = usin->sin_port;
sk_daddr_set(sk, daddr);
@@ -236,11 +232,11 @@
rt = NULL;
if (likely(!tp->repair)) {
- seq = secure_tcp_sequence_number(inet->inet_saddr,
- inet->inet_daddr,
- inet->inet_sport,
- usin->sin_port,
- &tp->tsoffset);
+ seq = secure_tcp_seq_and_tsoff(inet->inet_saddr,
+ inet->inet_daddr,
+ inet->inet_sport,
+ usin->sin_port,
+ &tp->tsoffset);
if (!tp->write_seq)
tp->write_seq = seq;
}
@@ -1217,19 +1213,9 @@
static struct dst_entry *tcp_v4_route_req(const struct sock *sk,
struct flowi *fl,
- const struct request_sock *req,
- bool *strict)
+ const struct request_sock *req)
{
- struct dst_entry *dst = inet_csk_route_req(sk, &fl->u.ip4, req);
-
- if (strict) {
- if (fl->u.ip4.daddr == inet_rsk(req)->ir_rmt_addr)
- *strict = true;
- else
- *strict = false;
- }
-
- return dst;
+ return inet_csk_route_req(sk, &fl->u.ip4, req);
}
struct request_sock_ops tcp_request_sock_ops __read_mostly = {
@@ -1253,7 +1239,7 @@
.cookie_init_seq = cookie_v4_init_sequence,
#endif
.route_req = tcp_v4_route_req,
- .init_seq = tcp_v4_init_sequence,
+ .init_seq_tsoff = tcp_v4_init_seq_and_tsoff,
.send_synack = tcp_v4_send_synack,
};
@@ -1423,8 +1409,6 @@
if (!nsk)
goto discard;
if (nsk != sk) {
- sock_rps_save_rxhash(nsk, skb);
- sk_mark_napi_id(nsk, skb);
if (tcp_child_process(sk, nsk, skb)) {
rsk = nsk;
goto reset;
@@ -1871,6 +1855,9 @@
/* Cleanup up the write buffer. */
tcp_write_queue_purge(sk);
+ /* Check if we want to disable active TFO */
+ tcp_fastopen_active_disable_ofo_check(sk);
+
/* Cleans up our, hopefully empty, out_of_order_queue. */
skb_rbtree_purge(&tp->out_of_order_queue);
@@ -2466,7 +2453,6 @@
net->ipv4.sysctl_tcp_tw_reuse = 0;
cnt = tcp_hashinfo.ehash_mask + 1;
- net->ipv4.tcp_death_row.sysctl_tw_recycle = 0;
net->ipv4.tcp_death_row.sysctl_max_tw_buckets = (cnt + 1) / 2;
net->ipv4.tcp_death_row.hashinfo = &tcp_hashinfo;
diff --git a/net/ipv4/tcp_lp.c b/net/ipv4/tcp_lp.c
index 046fd39..d6fb6c0 100644
--- a/net/ipv4/tcp_lp.c
+++ b/net/ipv4/tcp_lp.c
@@ -264,13 +264,15 @@
{
struct tcp_sock *tp = tcp_sk(sk);
struct lp *lp = inet_csk_ca(sk);
+ u32 delta;
if (sample->rtt_us > 0)
tcp_lp_rtt_sample(sk, sample->rtt_us);
/* calc inference */
- if (tcp_time_stamp > tp->rx_opt.rcv_tsecr)
- lp->inference = 3 * (tcp_time_stamp - tp->rx_opt.rcv_tsecr);
+ delta = tcp_time_stamp - tp->rx_opt.rcv_tsecr;
+ if ((s32)delta > 0)
+ lp->inference = 3 * delta;
/* test if within inference */
if (lp->last_drop && (tcp_time_stamp - lp->last_drop < lp->inference))
diff --git a/net/ipv4/tcp_metrics.c b/net/ipv4/tcp_metrics.c
index 0f46e5f..9d0d4f3 100644
--- a/net/ipv4/tcp_metrics.c
+++ b/net/ipv4/tcp_metrics.c
@@ -45,8 +45,6 @@
struct inetpeer_addr tcpm_saddr;
struct inetpeer_addr tcpm_daddr;
unsigned long tcpm_stamp;
- u32 tcpm_ts;
- u32 tcpm_ts_stamp;
u32 tcpm_lock;
u32 tcpm_vals[TCP_METRIC_MAX_KERNEL + 1];
struct tcp_fastopen_metrics tcpm_fastopen;
@@ -123,8 +121,6 @@
tm->tcpm_vals[TCP_METRIC_SSTHRESH] = dst_metric_raw(dst, RTAX_SSTHRESH);
tm->tcpm_vals[TCP_METRIC_CWND] = dst_metric_raw(dst, RTAX_CWND);
tm->tcpm_vals[TCP_METRIC_REORDERING] = dst_metric_raw(dst, RTAX_REORDERING);
- tm->tcpm_ts = 0;
- tm->tcpm_ts_stamp = 0;
if (fastopen_clear) {
tm->tcpm_fastopen.mss = 0;
tm->tcpm_fastopen.syn_loss = 0;
@@ -273,48 +269,6 @@
return tm;
}
-static struct tcp_metrics_block *__tcp_get_metrics_tw(struct inet_timewait_sock *tw)
-{
- struct tcp_metrics_block *tm;
- struct inetpeer_addr saddr, daddr;
- unsigned int hash;
- struct net *net;
-
- if (tw->tw_family == AF_INET) {
- inetpeer_set_addr_v4(&saddr, tw->tw_rcv_saddr);
- inetpeer_set_addr_v4(&daddr, tw->tw_daddr);
- hash = ipv4_addr_hash(tw->tw_daddr);
- }
-#if IS_ENABLED(CONFIG_IPV6)
- else if (tw->tw_family == AF_INET6) {
- if (ipv6_addr_v4mapped(&tw->tw_v6_daddr)) {
- inetpeer_set_addr_v4(&saddr, tw->tw_rcv_saddr);
- inetpeer_set_addr_v4(&daddr, tw->tw_daddr);
- hash = ipv4_addr_hash(tw->tw_daddr);
- } else {
- inetpeer_set_addr_v6(&saddr, &tw->tw_v6_rcv_saddr);
- inetpeer_set_addr_v6(&daddr, &tw->tw_v6_daddr);
- hash = ipv6_addr_hash(&tw->tw_v6_daddr);
- }
- }
-#endif
- else
- return NULL;
-
- net = twsk_net(tw);
- hash ^= net_hash_mix(net);
- hash = hash_32(hash, tcp_metrics_hash_log);
-
- for (tm = rcu_dereference(tcp_metrics_hash[hash].chain); tm;
- tm = rcu_dereference(tm->tcpm_next)) {
- if (addr_same(&tm->tcpm_saddr, &saddr) &&
- addr_same(&tm->tcpm_daddr, &daddr) &&
- net_eq(tm_net(tm), net))
- break;
- }
- return tm;
-}
-
static struct tcp_metrics_block *tcp_get_metrics(struct sock *sk,
struct dst_entry *dst,
bool create)
@@ -573,8 +527,7 @@
tp->snd_cwnd_stamp = tcp_time_stamp;
}
-bool tcp_peer_is_proven(struct request_sock *req, struct dst_entry *dst,
- bool paws_check, bool timestamps)
+bool tcp_peer_is_proven(struct request_sock *req, struct dst_entry *dst)
{
struct tcp_metrics_block *tm;
bool ret;
@@ -584,94 +537,10 @@
rcu_read_lock();
tm = __tcp_get_metrics_req(req, dst);
- if (paws_check) {
- if (tm &&
- (u32)get_seconds() - tm->tcpm_ts_stamp < TCP_PAWS_MSL &&
- ((s32)(tm->tcpm_ts - req->ts_recent) > TCP_PAWS_WINDOW ||
- !timestamps))
- ret = false;
- else
- ret = true;
- } else {
- if (tm && tcp_metric_get(tm, TCP_METRIC_RTT) && tm->tcpm_ts_stamp)
- ret = true;
- else
- ret = false;
- }
- rcu_read_unlock();
-
- return ret;
-}
-
-void tcp_fetch_timewait_stamp(struct sock *sk, struct dst_entry *dst)
-{
- struct tcp_metrics_block *tm;
-
- rcu_read_lock();
- tm = tcp_get_metrics(sk, dst, true);
- if (tm) {
- struct tcp_sock *tp = tcp_sk(sk);
-
- if ((u32)get_seconds() - tm->tcpm_ts_stamp <= TCP_PAWS_MSL) {
- tp->rx_opt.ts_recent_stamp = tm->tcpm_ts_stamp;
- tp->rx_opt.ts_recent = tm->tcpm_ts;
- }
- }
- rcu_read_unlock();
-}
-EXPORT_SYMBOL_GPL(tcp_fetch_timewait_stamp);
-
-/* VJ's idea. Save last timestamp seen from this destination and hold
- * it at least for normal timewait interval to use for duplicate
- * segment detection in subsequent connections, before they enter
- * synchronized state.
- */
-bool tcp_remember_stamp(struct sock *sk)
-{
- struct dst_entry *dst = __sk_dst_get(sk);
- bool ret = false;
-
- if (dst) {
- struct tcp_metrics_block *tm;
-
- rcu_read_lock();
- tm = tcp_get_metrics(sk, dst, true);
- if (tm) {
- struct tcp_sock *tp = tcp_sk(sk);
-
- if ((s32)(tm->tcpm_ts - tp->rx_opt.ts_recent) <= 0 ||
- ((u32)get_seconds() - tm->tcpm_ts_stamp > TCP_PAWS_MSL &&
- tm->tcpm_ts_stamp <= (u32)tp->rx_opt.ts_recent_stamp)) {
- tm->tcpm_ts_stamp = (u32)tp->rx_opt.ts_recent_stamp;
- tm->tcpm_ts = tp->rx_opt.ts_recent;
- }
- ret = true;
- }
- rcu_read_unlock();
- }
- return ret;
-}
-
-bool tcp_tw_remember_stamp(struct inet_timewait_sock *tw)
-{
- struct tcp_metrics_block *tm;
- bool ret = false;
-
- rcu_read_lock();
- tm = __tcp_get_metrics_tw(tw);
- if (tm) {
- const struct tcp_timewait_sock *tcptw;
- struct sock *sk = (struct sock *) tw;
-
- tcptw = tcp_twsk(sk);
- if ((s32)(tm->tcpm_ts - tcptw->tw_ts_recent) <= 0 ||
- ((u32)get_seconds() - tm->tcpm_ts_stamp > TCP_PAWS_MSL &&
- tm->tcpm_ts_stamp <= (u32)tcptw->tw_ts_recent_stamp)) {
- tm->tcpm_ts_stamp = (u32)tcptw->tw_ts_recent_stamp;
- tm->tcpm_ts = tcptw->tw_ts_recent;
- }
+ if (tm && tcp_metric_get(tm, TCP_METRIC_RTT))
ret = true;
- }
+ else
+ ret = false;
rcu_read_unlock();
return ret;
@@ -791,14 +660,6 @@
jiffies - tm->tcpm_stamp,
TCP_METRICS_ATTR_PAD) < 0)
goto nla_put_failure;
- if (tm->tcpm_ts_stamp) {
- if (nla_put_s32(msg, TCP_METRICS_ATTR_TW_TS_STAMP,
- (s32) (get_seconds() - tm->tcpm_ts_stamp)) < 0)
- goto nla_put_failure;
- if (nla_put_u32(msg, TCP_METRICS_ATTR_TW_TSVAL,
- tm->tcpm_ts) < 0)
- goto nla_put_failure;
- }
{
int n = 0;
diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c
index 65c0f3d..8f6373b 100644
--- a/net/ipv4/tcp_minisocks.c
+++ b/net/ipv4/tcp_minisocks.c
@@ -26,6 +26,7 @@
#include <net/tcp.h>
#include <net/inet_common.h>
#include <net/xfrm.h>
+#include <net/busy_poll.h>
int sysctl_tcp_abort_on_overflow __read_mostly;
@@ -94,7 +95,6 @@
struct tcp_options_received tmp_opt;
struct tcp_timewait_sock *tcptw = tcp_twsk((struct sock *)tw);
bool paws_reject = false;
- struct inet_timewait_death_row *tcp_death_row = &sock_net((struct sock*)tw)->ipv4.tcp_death_row;
tmp_opt.saw_tstamp = 0;
if (th->doff > (sizeof(*th) >> 2) && tcptw->tw_ts_recent_stamp) {
@@ -149,12 +149,7 @@
tcptw->tw_ts_recent = tmp_opt.rcv_tsval;
}
- if (tcp_death_row->sysctl_tw_recycle &&
- tcptw->tw_ts_recent_stamp &&
- tcp_tw_remember_stamp(tw))
- inet_twsk_reschedule(tw, tw->tw_timeout);
- else
- inet_twsk_reschedule(tw, TCP_TIMEWAIT_LEN);
+ inet_twsk_reschedule(tw, TCP_TIMEWAIT_LEN);
return TCP_TW_ACK;
}
@@ -259,12 +254,8 @@
const struct inet_connection_sock *icsk = inet_csk(sk);
const struct tcp_sock *tp = tcp_sk(sk);
struct inet_timewait_sock *tw;
- bool recycle_ok = false;
struct inet_timewait_death_row *tcp_death_row = &sock_net(sk)->ipv4.tcp_death_row;
- if (tcp_death_row->sysctl_tw_recycle && tp->rx_opt.ts_recent_stamp)
- recycle_ok = tcp_remember_stamp(sk);
-
tw = inet_twsk_alloc(sk, tcp_death_row, state);
if (tw) {
@@ -317,13 +308,9 @@
if (timeo < rto)
timeo = rto;
- if (recycle_ok) {
- tw->tw_timeout = rto;
- } else {
- tw->tw_timeout = TCP_TIMEWAIT_LEN;
- if (state == TCP_TIME_WAIT)
- timeo = TCP_TIMEWAIT_LEN;
- }
+ tw->tw_timeout = TCP_TIMEWAIT_LEN;
+ if (state == TCP_TIME_WAIT)
+ timeo = TCP_TIMEWAIT_LEN;
inet_twsk_schedule(tw, timeo);
/* Linkage updates. */
@@ -813,6 +800,9 @@
int ret = 0;
int state = child->sk_state;
+ /* record NAPI ID of child */
+ sk_mark_napi_id(child, skb);
+
tcp_segs_in(tcp_sk(child), skb);
if (!sock_owned_by_user(child)) {
ret = tcp_rcv_state_process(child, skb);
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index c3c082e..60111a0 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -212,12 +212,12 @@
/* If no clamp set the clamp to the max possible scaled window */
if (*window_clamp == 0)
- (*window_clamp) = (65535 << 14);
+ (*window_clamp) = (U16_MAX << TCP_MAX_WSCALE);
space = min(*window_clamp, space);
/* Quantize space offering to a multiple of mss if possible. */
if (space > mss)
- space = (space / mss) * mss;
+ space = rounddown(space, mss);
/* NOTE: offering an initial window larger than 32767
* will break some buggy TCP stacks. If the admin tells us
@@ -234,13 +234,11 @@
(*rcv_wscale) = 0;
if (wscale_ok) {
- /* Set window scaling on max possible window
- * See RFC1323 for an explanation of the limit to 14
- */
+ /* Set window scaling on max possible window */
space = max_t(u32, space, sysctl_tcp_rmem[2]);
space = max_t(u32, space, sysctl_rmem_max);
space = min_t(u32, space, *window_clamp);
- while (space > 65535 && (*rcv_wscale) < 14) {
+ while (space > U16_MAX && (*rcv_wscale) < TCP_MAX_WSCALE) {
space >>= 1;
(*rcv_wscale)++;
}
@@ -253,7 +251,7 @@
}
/* Set the clamp no higher than max representable value */
- (*window_clamp) = min(65535U << (*rcv_wscale), *window_clamp);
+ (*window_clamp) = min_t(__u32, U16_MAX << (*rcv_wscale), *window_clamp);
}
EXPORT_SYMBOL(tcp_select_initial_window);
@@ -1267,7 +1265,7 @@
* eventually). The difference is that pulled data not copied, but
* immediately discarded.
*/
-static void __pskb_trim_head(struct sk_buff *skb, int len)
+static int __pskb_trim_head(struct sk_buff *skb, int len)
{
struct skb_shared_info *shinfo;
int i, k, eat;
@@ -1277,7 +1275,7 @@
__skb_pull(skb, eat);
len -= eat;
if (!len)
- return;
+ return 0;
}
eat = len;
k = 0;
@@ -1303,23 +1301,28 @@
skb_reset_tail_pointer(skb);
skb->data_len -= len;
skb->len = skb->data_len;
+ return len;
}
/* Remove acked data from a packet in the transmit queue. */
int tcp_trim_head(struct sock *sk, struct sk_buff *skb, u32 len)
{
+ u32 delta_truesize;
+
if (skb_unclone(skb, GFP_ATOMIC))
return -ENOMEM;
- __pskb_trim_head(skb, len);
+ delta_truesize = __pskb_trim_head(skb, len);
TCP_SKB_CB(skb)->seq += len;
skb->ip_summed = CHECKSUM_PARTIAL;
- skb->truesize -= len;
- sk->sk_wmem_queued -= len;
- sk_mem_uncharge(sk, len);
- sock_set_flag(sk, SOCK_QUEUE_SHRUNK);
+ if (delta_truesize) {
+ skb->truesize -= delta_truesize;
+ sk->sk_wmem_queued -= delta_truesize;
+ sk_mem_uncharge(sk, delta_truesize);
+ sock_set_flag(sk, SOCK_QUEUE_SHRUNK);
+ }
/* Any change of skb->len requires recalculation of tso factor. */
if (tcp_skb_pcount(skb) > 1)
@@ -2561,7 +2564,6 @@
/* Don't do rounding if we are using window scaling, since the
* scaled window will not line up with the MSS boundary anyway.
*/
- window = tp->rcv_wnd;
if (tp->rx_opt.rcv_wscale) {
window = free_space;
@@ -2569,10 +2571,9 @@
* Import case: prevent zero window announcement if
* 1<<rcv_wscale > mss.
*/
- if (((window >> tp->rx_opt.rcv_wscale) << tp->rx_opt.rcv_wscale) != window)
- window = (((window >> tp->rx_opt.rcv_wscale) + 1)
- << tp->rx_opt.rcv_wscale);
+ window = ALIGN(window, (1 << tp->rx_opt.rcv_wscale));
} else {
+ window = tp->rcv_wnd;
/* Get the largest window that is a nice multiple of mss.
* Window clamp already applied above.
* If our current window offering is within 1 mss of the
@@ -2582,7 +2583,7 @@
* is too small.
*/
if (window <= free_space - mss || window > free_space)
- window = (free_space / mss) * mss;
+ window = rounddown(free_space, mss);
else if (mss == full_space &&
free_space > window + (full_space >> 1))
window = free_space;
diff --git a/net/ipv4/tcp_rate.c b/net/ipv4/tcp_rate.c
index 9be1581..c6a9fa8 100644
--- a/net/ipv4/tcp_rate.c
+++ b/net/ipv4/tcp_rate.c
@@ -106,7 +106,7 @@
/* Update the connection delivery information and generate a rate sample. */
void tcp_rate_gen(struct sock *sk, u32 delivered, u32 lost,
- struct skb_mstamp *now, struct rate_sample *rs)
+ struct rate_sample *rs)
{
struct tcp_sock *tp = tcp_sk(sk);
u32 snd_us, ack_us;
@@ -120,7 +120,7 @@
* to carry current time, flags, stats like "tcp_sacktag_state".
*/
if (delivered)
- tp->delivered_mstamp = *now;
+ tp->delivered_mstamp = tp->tcp_mstamp;
rs->acked_sacked = delivered; /* freshly ACKed or SACKed */
rs->losses = lost; /* freshly marked lost */
@@ -138,7 +138,8 @@
* longer phase.
*/
snd_us = rs->interval_us; /* send phase */
- ack_us = skb_mstamp_us_delta(now, &rs->prior_mstamp); /* ack phase */
+ ack_us = skb_mstamp_us_delta(&tp->tcp_mstamp,
+ &rs->prior_mstamp); /* ack phase */
rs->interval_us = max(snd_us, ack_us);
/* Normally we expect interval_us >= min-rtt.
diff --git a/net/ipv4/tcp_recovery.c b/net/ipv4/tcp_recovery.c
index d8acbd9..362b8c7 100644
--- a/net/ipv4/tcp_recovery.c
+++ b/net/ipv4/tcp_recovery.c
@@ -45,8 +45,7 @@
* or tcp_time_to_recover()'s "Trick#1: the loss is proven" code path will
* make us enter the CA_Recovery state.
*/
-static void tcp_rack_detect_loss(struct sock *sk, const struct skb_mstamp *now,
- u32 *reo_timeout)
+static void tcp_rack_detect_loss(struct sock *sk, u32 *reo_timeout)
{
struct tcp_sock *tp = tcp_sk(sk);
struct sk_buff *skb;
@@ -79,7 +78,7 @@
* A packet is lost if its elapsed time is beyond
* the recent RTT plus the reordering window.
*/
- u32 elapsed = skb_mstamp_us_delta(now,
+ u32 elapsed = skb_mstamp_us_delta(&tp->tcp_mstamp,
&skb->skb_mstamp);
s32 remaining = tp->rack.rtt_us + reo_wnd - elapsed;
@@ -105,7 +104,7 @@
}
}
-void tcp_rack_mark_lost(struct sock *sk, const struct skb_mstamp *now)
+void tcp_rack_mark_lost(struct sock *sk)
{
struct tcp_sock *tp = tcp_sk(sk);
u32 timeout;
@@ -115,7 +114,7 @@
/* Reset the advanced flag to avoid unnecessary queue scanning */
tp->rack.advanced = 0;
- tcp_rack_detect_loss(sk, now, &timeout);
+ tcp_rack_detect_loss(sk, &timeout);
if (timeout) {
timeout = usecs_to_jiffies(timeout + TCP_REO_TIMEOUT_MIN);
inet_csk_reset_xmit_timer(sk, ICSK_TIME_REO_TIMEOUT,
@@ -128,8 +127,7 @@
* draft-cheng-tcpm-rack-00.txt
*/
void tcp_rack_advance(struct tcp_sock *tp, u8 sacked, u32 end_seq,
- const struct skb_mstamp *xmit_time,
- const struct skb_mstamp *ack_time)
+ const struct skb_mstamp *xmit_time)
{
u32 rtt_us;
@@ -138,7 +136,7 @@
end_seq, tp->rack.end_seq))
return;
- rtt_us = skb_mstamp_us_delta(ack_time, xmit_time);
+ rtt_us = skb_mstamp_us_delta(&tp->tcp_mstamp, xmit_time);
if (sacked & TCPCB_RETRANS) {
/* If the sacked packet was retransmitted, it's ambiguous
* whether the retransmission or the original (or the prior
@@ -165,12 +163,11 @@
void tcp_rack_reo_timeout(struct sock *sk)
{
struct tcp_sock *tp = tcp_sk(sk);
- struct skb_mstamp now;
u32 timeout, prior_inflight;
- skb_mstamp_get(&now);
prior_inflight = tcp_packets_in_flight(tp);
- tcp_rack_detect_loss(sk, &now, &timeout);
+ skb_mstamp_get(&tp->tcp_mstamp);
+ tcp_rack_detect_loss(sk, &timeout);
if (prior_inflight != tcp_packets_in_flight(tp)) {
if (inet_csk(sk)->icsk_ca_state != TCP_CA_Recovery) {
tcp_enter_recovery(sk, false);
diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
index b2ab411..1467254 100644
--- a/net/ipv4/tcp_timer.c
+++ b/net/ipv4/tcp_timer.c
@@ -201,11 +201,10 @@
if (retransmits_timed_out(sk, net->ipv4.sysctl_tcp_retries1, 0, 0)) {
/* Some middle-boxes may black-hole Fast Open _after_
* the handshake. Therefore we conservatively disable
- * Fast Open on this path on recurring timeouts with
- * few or zero bytes acked after Fast Open.
+ * Fast Open on this path on recurring timeouts after
+ * successful Fast Open.
*/
- if (tp->syn_data_acked &&
- tp->bytes_acked <= tp->rx_opt.mss_clamp) {
+ if (tp->syn_data_acked) {
tcp_fastopen_cache_set(sk, 0, NULL, true, 0);
if (icsk->icsk_retransmits == net->ipv4.sysctl_tcp_retries1)
NET_INC_STATS(sock_net(sk),
diff --git a/net/ipv4/tcp_westwood.c b/net/ipv4/tcp_westwood.c
index fed66dc..9775453 100644
--- a/net/ipv4/tcp_westwood.c
+++ b/net/ipv4/tcp_westwood.c
@@ -265,8 +265,8 @@
if (ext & (1 << (INET_DIAG_VEGASINFO - 1))) {
info->vegas.tcpv_enabled = 1;
info->vegas.tcpv_rttcnt = 0;
- info->vegas.tcpv_rtt = jiffies_to_usecs(ca->rtt),
- info->vegas.tcpv_minrtt = jiffies_to_usecs(ca->rtt_min),
+ info->vegas.tcpv_rtt = jiffies_to_usecs(ca->rtt);
+ info->vegas.tcpv_minrtt = jiffies_to_usecs(ca->rtt_min);
*attr = INET_DIAG_VEGASINFO;
return sizeof(struct tcpvegas_info);
diff --git a/net/ipv4/udp_offload.c b/net/ipv4/udp_offload.c
index b2be1d9..7812501 100644
--- a/net/ipv4/udp_offload.c
+++ b/net/ipv4/udp_offload.c
@@ -29,6 +29,7 @@
u16 mac_len = skb->mac_len;
int udp_offset, outer_hlen;
__wsum partial;
+ bool need_ipsec;
if (unlikely(!pskb_may_pull(skb, tnl_hlen)))
goto out;
@@ -62,8 +63,10 @@
ufo = !!(skb_shinfo(skb)->gso_type & SKB_GSO_UDP);
+ need_ipsec = skb_dst(skb) && dst_xfrm(skb_dst(skb));
/* Try to offload checksum if possible */
offload_csum = !!(need_csum &&
+ !need_ipsec &&
(skb->dev->features &
(is_ipv6 ? (NETIF_F_HW_CSUM | NETIF_F_IPV6_CSUM) :
(NETIF_F_HW_CSUM | NETIF_F_IP_CSUM))));
diff --git a/net/ipv4/xfrm4_mode_transport.c b/net/ipv4/xfrm4_mode_transport.c
index 4acc050..3d36644 100644
--- a/net/ipv4/xfrm4_mode_transport.c
+++ b/net/ipv4/xfrm4_mode_transport.c
@@ -12,6 +12,7 @@
#include <net/dst.h>
#include <net/ip.h>
#include <net/xfrm.h>
+#include <net/protocol.h>
/* Add encapsulation header.
*
@@ -23,6 +24,8 @@
struct iphdr *iph = ip_hdr(skb);
int ihl = iph->ihl * 4;
+ skb_set_inner_transport_header(skb, skb_transport_offset(skb));
+
skb_set_network_header(skb, -x->props.header_len);
skb->mac_header = skb->network_header +
offsetof(struct iphdr, protocol);
@@ -56,9 +59,40 @@
return 0;
}
+static struct sk_buff *xfrm4_transport_gso_segment(struct xfrm_state *x,
+ struct sk_buff *skb,
+ netdev_features_t features)
+{
+ const struct net_offload *ops;
+ struct sk_buff *segs = ERR_PTR(-EINVAL);
+ struct xfrm_offload *xo = xfrm_offload(skb);
+
+ skb->transport_header += x->props.header_len;
+ ops = rcu_dereference(inet_offloads[xo->proto]);
+ if (likely(ops && ops->callbacks.gso_segment))
+ segs = ops->callbacks.gso_segment(skb, features);
+
+ return segs;
+}
+
+static void xfrm4_transport_xmit(struct xfrm_state *x, struct sk_buff *skb)
+{
+ struct xfrm_offload *xo = xfrm_offload(skb);
+
+ skb_reset_mac_len(skb);
+ pskb_pull(skb, skb->mac_len + sizeof(struct iphdr) + x->props.header_len);
+
+ if (xo->flags & XFRM_GSO_SEGMENT) {
+ skb_reset_transport_header(skb);
+ skb->transport_header -= x->props.header_len;
+ }
+}
+
static struct xfrm_mode xfrm4_transport_mode = {
.input = xfrm4_transport_input,
.output = xfrm4_transport_output,
+ .gso_segment = xfrm4_transport_gso_segment,
+ .xmit = xfrm4_transport_xmit,
.owner = THIS_MODULE,
.encap = XFRM_MODE_TRANSPORT,
};
diff --git a/net/ipv4/xfrm4_mode_tunnel.c b/net/ipv4/xfrm4_mode_tunnel.c
index 35feda6..e6265e2 100644
--- a/net/ipv4/xfrm4_mode_tunnel.c
+++ b/net/ipv4/xfrm4_mode_tunnel.c
@@ -33,6 +33,9 @@
struct iphdr *top_iph;
int flags;
+ skb_set_inner_network_header(skb, skb_network_offset(skb));
+ skb_set_inner_transport_header(skb, skb_transport_offset(skb));
+
skb_set_network_header(skb, -x->props.header_len);
skb->mac_header = skb->network_header +
offsetof(struct iphdr, protocol);
@@ -96,11 +99,36 @@
return err;
}
+static struct sk_buff *xfrm4_mode_tunnel_gso_segment(struct xfrm_state *x,
+ struct sk_buff *skb,
+ netdev_features_t features)
+{
+ __skb_push(skb, skb->mac_len);
+ return skb_mac_gso_segment(skb, features);
+
+}
+
+static void xfrm4_mode_tunnel_xmit(struct xfrm_state *x, struct sk_buff *skb)
+{
+ struct xfrm_offload *xo = xfrm_offload(skb);
+
+ if (xo->flags & XFRM_GSO_SEGMENT) {
+ skb->network_header = skb->network_header - x->props.header_len;
+ skb->transport_header = skb->network_header +
+ sizeof(struct iphdr);
+ }
+
+ skb_reset_mac_len(skb);
+ pskb_pull(skb, skb->mac_len + x->props.header_len);
+}
+
static struct xfrm_mode xfrm4_tunnel_mode = {
.input2 = xfrm4_mode_tunnel_input,
.input = xfrm_prepare_input,
.output2 = xfrm4_mode_tunnel_output,
.output = xfrm4_prepare_output,
+ .gso_segment = xfrm4_mode_tunnel_gso_segment,
+ .xmit = xfrm4_mode_tunnel_xmit,
.owner = THIS_MODULE,
.encap = XFRM_MODE_TUNNEL,
.flags = XFRM_MODE_FLAG_TUNNEL,
diff --git a/net/ipv4/xfrm4_output.c b/net/ipv4/xfrm4_output.c
index 7ee6518..94b8702 100644
--- a/net/ipv4/xfrm4_output.c
+++ b/net/ipv4/xfrm4_output.c
@@ -29,7 +29,8 @@
goto out;
mtu = dst_mtu(skb_dst(skb));
- if (skb->len > mtu) {
+ if ((!skb_is_gso(skb) && skb->len > mtu) ||
+ (skb_is_gso(skb) && skb_gso_network_seglen(skb) > ip_skb_dst_mtu(skb->sk, skb))) {
skb->protocol = htons(ETH_P_IP);
if (skb->sk)
diff --git a/net/ipv6/Kconfig b/net/ipv6/Kconfig
index e2afe677..48c4529 100644
--- a/net/ipv6/Kconfig
+++ b/net/ipv6/Kconfig
@@ -307,6 +307,7 @@
bool "IPv6: Segment Routing Header encapsulation support"
depends on IPV6
select LWTUNNEL
+ select DST_CACHE
---help---
Support for encapsulation of packets within an outer IPv6
header and a Segment Routing Header using the lightweight
diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
index 80ce478..a2a370b 100644
--- a/net/ipv6/addrconf.c
+++ b/net/ipv6/addrconf.c
@@ -224,6 +224,7 @@
.accept_ra_rtr_pref = 1,
.rtr_probe_interval = 60 * HZ,
#ifdef CONFIG_IPV6_ROUTE_INFO
+ .accept_ra_rt_info_min_plen = 0,
.accept_ra_rt_info_max_plen = 0,
#endif
#endif
@@ -245,6 +246,7 @@
#endif
.enhanced_dad = 1,
.addr_gen_mode = IN6_ADDR_GEN_MODE_EUI64,
+ .disable_policy = 0,
};
static struct ipv6_devconf ipv6_devconf_dflt __read_mostly = {
@@ -276,6 +278,7 @@
.accept_ra_rtr_pref = 1,
.rtr_probe_interval = 60 * HZ,
#ifdef CONFIG_IPV6_ROUTE_INFO
+ .accept_ra_rt_info_min_plen = 0,
.accept_ra_rt_info_max_plen = 0,
#endif
#endif
@@ -297,6 +300,7 @@
#endif
.enhanced_dad = 1,
.addr_gen_mode = IN6_ADDR_GEN_MODE_EUI64,
+ .disable_policy = 0,
};
/* Check if a valid qdisc is available */
@@ -545,6 +549,9 @@
if (nla_put_s32(skb, NETCONFA_IFINDEX, ifindex) < 0)
goto nla_put_failure;
+ if (!devconf)
+ goto out;
+
if ((all || type == NETCONFA_FORWARDING) &&
nla_put_s32(skb, NETCONFA_FORWARDING, devconf->forwarding) < 0)
goto nla_put_failure;
@@ -563,6 +570,7 @@
devconf->ignore_routes_with_linkdown) < 0)
goto nla_put_failure;
+out:
nlmsg_end(skb, nlh);
return 0;
@@ -571,8 +579,8 @@
return -EMSGSIZE;
}
-void inet6_netconf_notify_devconf(struct net *net, int type, int ifindex,
- struct ipv6_devconf *devconf)
+void inet6_netconf_notify_devconf(struct net *net, int event, int type,
+ int ifindex, struct ipv6_devconf *devconf)
{
struct sk_buff *skb;
int err = -ENOBUFS;
@@ -582,7 +590,7 @@
goto errout;
err = inet6_netconf_fill_devconf(skb, ifindex, devconf, 0, 0,
- RTM_NEWNETCONF, 0, type);
+ event, 0, type);
if (err < 0) {
/* -EMSGSIZE implies BUG in inet6_netconf_msgsize_devconf() */
WARN_ON(err == -EMSGSIZE);
@@ -603,7 +611,8 @@
};
static int inet6_netconf_get_devconf(struct sk_buff *in_skb,
- struct nlmsghdr *nlh)
+ struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(in_skb->sk);
struct nlattr *tb[NETCONFA_MAX+1];
@@ -616,7 +625,7 @@
int err;
err = nlmsg_parse(nlh, sizeof(*ncm), tb, NETCONFA_MAX,
- devconf_ipv6_policy);
+ devconf_ipv6_policy, extack);
if (err < 0)
goto errout;
@@ -765,7 +774,8 @@
else
addrconf_leave_anycast(ifa);
}
- inet6_netconf_notify_devconf(dev_net(dev), NETCONFA_FORWARDING,
+ inet6_netconf_notify_devconf(dev_net(dev), RTM_NEWNETCONF,
+ NETCONFA_FORWARDING,
dev->ifindex, &idev->cnf);
}
@@ -800,7 +810,8 @@
if (p == &net->ipv6.devconf_dflt->forwarding) {
if ((!newf) ^ (!old))
- inet6_netconf_notify_devconf(net, NETCONFA_FORWARDING,
+ inet6_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_FORWARDING,
NETCONFA_IFINDEX_DEFAULT,
net->ipv6.devconf_dflt);
rtnl_unlock();
@@ -812,13 +823,15 @@
net->ipv6.devconf_dflt->forwarding = newf;
if ((!newf) ^ (!old_dflt))
- inet6_netconf_notify_devconf(net, NETCONFA_FORWARDING,
+ inet6_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_FORWARDING,
NETCONFA_IFINDEX_DEFAULT,
net->ipv6.devconf_dflt);
addrconf_forward_change(net, newf);
if ((!newf) ^ (!old))
- inet6_netconf_notify_devconf(net, NETCONFA_FORWARDING,
+ inet6_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_FORWARDING,
NETCONFA_IFINDEX_ALL,
net->ipv6.devconf_all);
} else if ((!newf) ^ (!old))
@@ -843,6 +856,7 @@
idev->cnf.ignore_routes_with_linkdown = newf;
if (changed)
inet6_netconf_notify_devconf(dev_net(dev),
+ RTM_NEWNETCONF,
NETCONFA_IGNORE_ROUTES_WITH_LINKDOWN,
dev->ifindex,
&idev->cnf);
@@ -865,6 +879,7 @@
if (p == &net->ipv6.devconf_dflt->ignore_routes_with_linkdown) {
if ((!newf) ^ (!old))
inet6_netconf_notify_devconf(net,
+ RTM_NEWNETCONF,
NETCONFA_IGNORE_ROUTES_WITH_LINKDOWN,
NETCONFA_IFINDEX_DEFAULT,
net->ipv6.devconf_dflt);
@@ -877,6 +892,7 @@
addrconf_linkdown_change(net, newf);
if ((!newf) ^ (!old))
inet6_netconf_notify_devconf(net,
+ RTM_NEWNETCONF,
NETCONFA_IGNORE_ROUTES_WITH_LINKDOWN,
NETCONFA_IFINDEX_ALL,
net->ipv6.devconf_all);
@@ -944,6 +960,7 @@
const struct in6_addr *peer_addr, int pfxlen,
int scope, u32 flags, u32 valid_lft, u32 prefered_lft)
{
+ struct net *net = dev_net(idev->dev);
struct inet6_ifaddr *ifa = NULL;
struct rt6_info *rt;
unsigned int hash;
@@ -990,6 +1007,10 @@
goto out;
}
+ if (net->ipv6.devconf_all->disable_policy ||
+ idev->cnf.disable_policy)
+ rt->dst.flags |= DST_NOPOLICY;
+
neigh_parms_data_state_setall(idev->nd_parms);
ifa->addr = *addr;
@@ -2053,12 +2074,23 @@
__ipv6_dev_ac_dec(ifp->idev, &addr);
}
-static int addrconf_ifid_eui64(u8 *eui, struct net_device *dev)
+static int addrconf_ifid_6lowpan(u8 *eui, struct net_device *dev)
{
- if (dev->addr_len != EUI64_ADDR_LEN)
+ switch (dev->addr_len) {
+ case ETH_ALEN:
+ memcpy(eui, dev->dev_addr, 3);
+ eui[3] = 0xFF;
+ eui[4] = 0xFE;
+ memcpy(eui + 5, dev->dev_addr + 3, 3);
+ break;
+ case EUI64_ADDR_LEN:
+ memcpy(eui, dev->dev_addr, EUI64_ADDR_LEN);
+ eui[0] ^= 2;
+ break;
+ default:
return -1;
- memcpy(eui, dev->dev_addr, EUI64_ADDR_LEN);
- eui[0] ^= 2;
+ }
+
return 0;
}
@@ -2150,7 +2182,7 @@
case ARPHRD_TUNNEL:
return addrconf_ifid_gre(eui, dev);
case ARPHRD_6LOWPAN:
- return addrconf_ifid_eui64(eui, dev);
+ return addrconf_ifid_6lowpan(eui, dev);
case ARPHRD_IEEE1394:
return addrconf_ifid_ieee1394(eui, dev);
case ARPHRD_TUNNEL6:
@@ -3271,14 +3303,24 @@
static int fixup_permanent_addr(struct inet6_dev *idev,
struct inet6_ifaddr *ifp)
{
- if (!ifp->rt) {
- struct rt6_info *rt;
+ /* rt6i_ref == 0 means the host route was removed from the
+ * FIB, for example, if 'lo' device is taken down. In that
+ * case regenerate the host route.
+ */
+ if (!ifp->rt || !atomic_read(&ifp->rt->rt6i_ref)) {
+ struct rt6_info *rt, *prev;
rt = addrconf_dst_alloc(idev, &ifp->addr, false);
if (unlikely(IS_ERR(rt)))
return PTR_ERR(rt);
+ /* ifp->rt can be accessed outside of rtnl */
+ spin_lock(&ifp->lock);
+ prev = ifp->rt;
ifp->rt = rt;
+ spin_unlock(&ifp->lock);
+
+ ip6_rt_put(prev);
}
if (!(ifp->flags & IFA_F_NOPREFIXROUTE)) {
@@ -3286,7 +3328,8 @@
idev->dev, 0, 0);
}
- addrconf_dad_start(ifp);
+ if (ifp->state == INET6_IFADDR_STATE_PREDAD)
+ addrconf_dad_start(ifp);
return 0;
}
@@ -3641,7 +3684,7 @@
if (keep) {
/* set state to skip the notifier below */
state = INET6_IFADDR_STATE_DEAD;
- ifa->state = 0;
+ ifa->state = INET6_IFADDR_STATE_PREDAD;
if (!(ifa->flags & IFA_F_NODAD))
ifa->flags |= IFA_F_TENTATIVE;
@@ -4382,7 +4425,8 @@
};
static int
-inet6_rtm_deladdr(struct sk_buff *skb, struct nlmsghdr *nlh)
+inet6_rtm_deladdr(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct ifaddrmsg *ifm;
@@ -4391,7 +4435,8 @@
u32 ifa_flags;
int err;
- err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv6_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv6_policy,
+ extack);
if (err < 0)
return err;
@@ -4491,7 +4536,8 @@
}
static int
-inet6_rtm_newaddr(struct sk_buff *skb, struct nlmsghdr *nlh)
+inet6_rtm_newaddr(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct ifaddrmsg *ifm;
@@ -4503,7 +4549,8 @@
u32 ifa_flags;
int err;
- err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv6_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv6_policy,
+ extack);
if (err < 0)
return err;
@@ -4853,7 +4900,8 @@
return inet6_dump_addr(skb, cb, type);
}
-static int inet6_rtm_getaddr(struct sk_buff *in_skb, struct nlmsghdr *nlh)
+static int inet6_rtm_getaddr(struct sk_buff *in_skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(in_skb->sk);
struct ifaddrmsg *ifm;
@@ -4864,7 +4912,8 @@
struct sk_buff *skb;
int err;
- err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv6_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_ipv6_policy,
+ extack);
if (err < 0)
goto errout;
@@ -4975,6 +5024,7 @@
array[DEVCONF_RTR_PROBE_INTERVAL] =
jiffies_to_msecs(cnf->rtr_probe_interval);
#ifdef CONFIG_IPV6_ROUTE_INFO
+ array[DEVCONF_ACCEPT_RA_RT_INFO_MIN_PLEN] = cnf->accept_ra_rt_info_min_plen;
array[DEVCONF_ACCEPT_RA_RT_INFO_MAX_PLEN] = cnf->accept_ra_rt_info_max_plen;
#endif
#endif
@@ -5006,6 +5056,7 @@
#endif
array[DEVCONF_ENHANCED_DAD] = cnf->enhanced_dad;
array[DEVCONF_ADDR_GEN_MODE] = cnf->addr_gen_mode;
+ array[DEVCONF_DISABLE_POLICY] = cnf->disable_policy;
}
static inline size_t inet6_ifla6_size(void)
@@ -5232,7 +5283,8 @@
if (dev && !__in6_dev_get(dev))
return -EAFNOSUPPORT;
- return nla_parse_nested(tb, IFLA_INET6_MAX, nla, inet6_af_policy);
+ return nla_parse_nested(tb, IFLA_INET6_MAX, nla, inet6_af_policy,
+ NULL);
}
static int check_addr_gen_mode(int mode)
@@ -5264,7 +5316,7 @@
if (!idev)
return -EAFNOSUPPORT;
- if (nla_parse_nested(tb, IFLA_INET6_MAX, nla, NULL) < 0)
+ if (nla_parse_nested(tb, IFLA_INET6_MAX, nla, NULL, NULL) < 0)
BUG();
if (tb[IFLA_INET6_TOKEN]) {
@@ -5667,17 +5719,20 @@
return restart_syscall();
if (valp == &net->ipv6.devconf_dflt->proxy_ndp)
- inet6_netconf_notify_devconf(net, NETCONFA_PROXY_NEIGH,
+ inet6_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_PROXY_NEIGH,
NETCONFA_IFINDEX_DEFAULT,
net->ipv6.devconf_dflt);
else if (valp == &net->ipv6.devconf_all->proxy_ndp)
- inet6_netconf_notify_devconf(net, NETCONFA_PROXY_NEIGH,
+ inet6_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_PROXY_NEIGH,
NETCONFA_IFINDEX_ALL,
net->ipv6.devconf_all);
else {
struct inet6_dev *idev = ctl->extra1;
- inet6_netconf_notify_devconf(net, NETCONFA_PROXY_NEIGH,
+ inet6_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_PROXY_NEIGH,
idev->dev->ifindex,
&idev->cnf);
}
@@ -5830,6 +5885,105 @@
return ret;
}
+static
+void addrconf_set_nopolicy(struct rt6_info *rt, int action)
+{
+ if (rt) {
+ if (action)
+ rt->dst.flags |= DST_NOPOLICY;
+ else
+ rt->dst.flags &= ~DST_NOPOLICY;
+ }
+}
+
+static
+void addrconf_disable_policy_idev(struct inet6_dev *idev, int val)
+{
+ struct inet6_ifaddr *ifa;
+
+ read_lock_bh(&idev->lock);
+ list_for_each_entry(ifa, &idev->addr_list, if_list) {
+ spin_lock(&ifa->lock);
+ if (ifa->rt) {
+ struct rt6_info *rt = ifa->rt;
+ struct fib6_table *table = rt->rt6i_table;
+ int cpu;
+
+ read_lock(&table->tb6_lock);
+ addrconf_set_nopolicy(ifa->rt, val);
+ if (rt->rt6i_pcpu) {
+ for_each_possible_cpu(cpu) {
+ struct rt6_info **rtp;
+
+ rtp = per_cpu_ptr(rt->rt6i_pcpu, cpu);
+ addrconf_set_nopolicy(*rtp, val);
+ }
+ }
+ read_unlock(&table->tb6_lock);
+ }
+ spin_unlock(&ifa->lock);
+ }
+ read_unlock_bh(&idev->lock);
+}
+
+static
+int addrconf_disable_policy(struct ctl_table *ctl, int *valp, int val)
+{
+ struct inet6_dev *idev;
+ struct net *net;
+
+ if (!rtnl_trylock())
+ return restart_syscall();
+
+ *valp = val;
+
+ net = (struct net *)ctl->extra2;
+ if (valp == &net->ipv6.devconf_dflt->disable_policy) {
+ rtnl_unlock();
+ return 0;
+ }
+
+ if (valp == &net->ipv6.devconf_all->disable_policy) {
+ struct net_device *dev;
+
+ for_each_netdev(net, dev) {
+ idev = __in6_dev_get(dev);
+ if (idev)
+ addrconf_disable_policy_idev(idev, val);
+ }
+ } else {
+ idev = (struct inet6_dev *)ctl->extra1;
+ addrconf_disable_policy_idev(idev, val);
+ }
+
+ rtnl_unlock();
+ return 0;
+}
+
+static
+int addrconf_sysctl_disable_policy(struct ctl_table *ctl, int write,
+ void __user *buffer, size_t *lenp,
+ loff_t *ppos)
+{
+ int *valp = ctl->data;
+ int val = *valp;
+ loff_t pos = *ppos;
+ struct ctl_table lctl;
+ int ret;
+
+ lctl = *ctl;
+ lctl.data = &val;
+ ret = proc_dointvec(&lctl, write, buffer, lenp, ppos);
+
+ if (write && (*valp != val))
+ ret = addrconf_disable_policy(ctl, valp, val);
+
+ if (ret)
+ *ppos = pos;
+
+ return ret;
+}
+
static int minus_one = -1;
static const int one = 1;
static const int two_five_five = 255;
@@ -6018,6 +6172,13 @@
},
#ifdef CONFIG_IPV6_ROUTE_INFO
{
+ .procname = "accept_ra_rt_info_min_plen",
+ .data = &ipv6_devconf.accept_ra_rt_info_min_plen,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec,
+ },
+ {
.procname = "accept_ra_rt_info_max_plen",
.data = &ipv6_devconf.accept_ra_rt_info_max_plen,
.maxlen = sizeof(int),
@@ -6188,6 +6349,13 @@
.proc_handler = addrconf_sysctl_addr_gen_mode,
},
{
+ .procname = "disable_policy",
+ .data = &ipv6_devconf.disable_policy,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = addrconf_sysctl_disable_policy,
+ },
+ {
/* sentinel */
}
};
@@ -6227,7 +6395,8 @@
ifindex = NETCONFA_IFINDEX_DEFAULT;
else
ifindex = idev->dev->ifindex;
- inet6_netconf_notify_devconf(net, NETCONFA_ALL, ifindex, p);
+ inet6_netconf_notify_devconf(net, RTM_NEWNETCONF, NETCONFA_ALL,
+ ifindex, p);
return 0;
free:
@@ -6236,7 +6405,8 @@
return -ENOBUFS;
}
-static void __addrconf_sysctl_unregister(struct ipv6_devconf *p)
+static void __addrconf_sysctl_unregister(struct net *net,
+ struct ipv6_devconf *p, int ifindex)
{
struct ctl_table *table;
@@ -6247,6 +6417,8 @@
unregister_net_sysctl_table(p->sysctl_header);
p->sysctl_header = NULL;
kfree(table);
+
+ inet6_netconf_notify_devconf(net, RTM_DELNETCONF, 0, ifindex, NULL);
}
static int addrconf_sysctl_register(struct inet6_dev *idev)
@@ -6270,7 +6442,8 @@
static void addrconf_sysctl_unregister(struct inet6_dev *idev)
{
- __addrconf_sysctl_unregister(&idev->cnf);
+ __addrconf_sysctl_unregister(dev_net(idev->dev), &idev->cnf,
+ idev->dev->ifindex);
neigh_sysctl_unregister(idev->nd_parms);
}
@@ -6313,7 +6486,7 @@
#ifdef CONFIG_SYSCTL
err_reg_dflt:
- __addrconf_sysctl_unregister(all);
+ __addrconf_sysctl_unregister(net, all, NETCONFA_IFINDEX_ALL);
err_reg_all:
kfree(dflt);
#endif
@@ -6326,8 +6499,10 @@
static void __net_exit addrconf_exit_net(struct net *net)
{
#ifdef CONFIG_SYSCTL
- __addrconf_sysctl_unregister(net->ipv6.devconf_dflt);
- __addrconf_sysctl_unregister(net->ipv6.devconf_all);
+ __addrconf_sysctl_unregister(net, net->ipv6.devconf_dflt,
+ NETCONFA_IFINDEX_DEFAULT);
+ __addrconf_sysctl_unregister(net, net->ipv6.devconf_all,
+ NETCONFA_IFINDEX_ALL);
#endif
kfree(net->ipv6.devconf_dflt);
kfree(net->ipv6.devconf_all);
diff --git a/net/ipv6/addrlabel.c b/net/ipv6/addrlabel.c
index a8f6986..07cd7d2 100644
--- a/net/ipv6/addrlabel.c
+++ b/net/ipv6/addrlabel.c
@@ -404,7 +404,8 @@
[IFAL_LABEL] = { .len = sizeof(u32), },
};
-static int ip6addrlbl_newdel(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int ip6addrlbl_newdel(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct ifaddrlblmsg *ifal;
@@ -413,7 +414,8 @@
u32 label;
int err = 0;
- err = nlmsg_parse(nlh, sizeof(*ifal), tb, IFAL_MAX, ifal_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifal), tb, IFAL_MAX, ifal_policy,
+ extack);
if (err < 0)
return err;
@@ -521,7 +523,8 @@
+ nla_total_size(4); /* IFAL_LABEL */
}
-static int ip6addrlbl_get(struct sk_buff *in_skb, struct nlmsghdr *nlh)
+static int ip6addrlbl_get(struct sk_buff *in_skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(in_skb->sk);
struct ifaddrlblmsg *ifal;
@@ -532,7 +535,8 @@
struct ip6addrlbl_entry *p;
struct sk_buff *skb;
- err = nlmsg_parse(nlh, sizeof(*ifal), tb, IFAL_MAX, ifal_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifal), tb, IFAL_MAX, ifal_policy,
+ extack);
if (err < 0)
return err;
diff --git a/net/ipv6/af_inet6.c b/net/ipv6/af_inet6.c
index a9a9553..a88b5b5 100644
--- a/net/ipv6/af_inet6.c
+++ b/net/ipv6/af_inet6.c
@@ -933,8 +933,6 @@
if (err)
goto igmp_fail;
- ipv6_stub = &ipv6_stub_impl;
-
err = ipv6_netfilter_init();
if (err)
goto netfilter_fail;
@@ -1005,18 +1003,28 @@
if (err)
goto seg6_fail;
+ err = igmp6_late_init();
+ if (err)
+ goto igmp6_late_err;
+
#ifdef CONFIG_SYSCTL
err = ipv6_sysctl_register();
if (err)
goto sysctl_fail;
#endif
+
+ /* ensure that ipv6 stubs are visible only after ipv6 is ready */
+ wmb();
+ ipv6_stub = &ipv6_stub_impl;
out:
return err;
#ifdef CONFIG_SYSCTL
sysctl_fail:
- seg6_exit();
+ igmp6_late_cleanup();
#endif
+igmp6_late_err:
+ seg6_exit();
seg6_fail:
calipso_exit();
calipso_fail:
diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c
index ff54faa..1fe99ba 100644
--- a/net/ipv6/esp6.c
+++ b/net/ipv6/esp6.c
@@ -170,19 +170,23 @@
}
static struct ip_esp_hdr *esp_output_set_esn(struct sk_buff *skb,
+ struct xfrm_state *x,
struct ip_esp_hdr *esph,
__be32 *seqhi)
{
- struct xfrm_state *x = skb_dst(skb)->xfrm;
-
/* For ESN we move the header forward by 4 bytes to
* accomodate the high bits. We will move it back after
* encryption.
*/
if ((x->props.flags & XFRM_STATE_ESN)) {
+ struct xfrm_offload *xo = xfrm_offload(skb);
+
esph = (void *)(skb_transport_header(skb) - sizeof(__be32));
*seqhi = esph->spi;
- esph->seq_no = htonl(XFRM_SKB_CB(skb)->seq.output.hi);
+ if (xo)
+ esph->seq_no = htonl(xo->seq.hi);
+ else
+ esph->seq_no = htonl(XFRM_SKB_CB(skb)->seq.output.hi);
}
esph->spi = x->id.spi;
@@ -214,61 +218,16 @@
tail[plen - 1] = proto;
}
-static int esp6_output(struct xfrm_state *x, struct sk_buff *skb)
+int esp6_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *esp)
{
- int err;
- struct ip_esp_hdr *esph;
- struct crypto_aead *aead;
- struct aead_request *req;
- struct scatterlist *sg, *dsg;
- struct sk_buff *trailer;
- struct page *page;
- void *tmp;
- int blksize;
- int clen;
- int alen;
- int plen;
- int ivlen;
- int tfclen;
- int nfrags;
- int assoclen;
- int seqhilen;
- int tailen;
- u8 *iv;
u8 *tail;
u8 *vaddr;
- __be32 *seqhi;
- __be64 seqno;
- __u8 proto = *skb_mac_header(skb);
+ int nfrags;
+ struct page *page;
+ struct ip_esp_hdr *esph;
+ struct sk_buff *trailer;
+ int tailen = esp->tailen;
- /* skb is pure payload to encrypt */
- aead = x->data;
- alen = crypto_aead_authsize(aead);
- ivlen = crypto_aead_ivsize(aead);
-
- tfclen = 0;
- if (x->tfcpad) {
- struct xfrm_dst *dst = (struct xfrm_dst *)skb_dst(skb);
- u32 padto;
-
- padto = min(x->tfcpad, esp6_get_mtu(x, dst->child_mtu_cached));
- if (skb->len < padto)
- tfclen = padto - skb->len;
- }
- blksize = ALIGN(crypto_aead_blocksize(aead), 4);
- clen = ALIGN(skb->len + 2 + tfclen, blksize);
- plen = clen - skb->len - tfclen;
- tailen = tfclen + plen + alen;
-
- assoclen = sizeof(*esph);
- seqhilen = 0;
-
- if (x->props.flags & XFRM_STATE_ESN) {
- seqhilen += sizeof(__be32);
- assoclen += seqhilen;
- }
-
- *skb_mac_header(skb) = IPPROTO_ESP;
esph = ip_esp_hdr(skb);
if (!skb_cloned(skb)) {
@@ -284,6 +243,8 @@
struct sock *sk = skb->sk;
struct page_frag *pfrag = &x->xfrag;
+ esp->inplace = false;
+
allocsize = ALIGN(tailen, L1_CACHE_BYTES);
spin_lock_bh(&x->lock);
@@ -300,10 +261,12 @@
tail = vaddr + pfrag->offset;
- esp_output_fill_trailer(tail, tfclen, plen, proto);
+ esp_output_fill_trailer(tail, esp->tfclen, esp->plen, esp->proto);
kunmap_atomic(vaddr);
+ spin_unlock_bh(&x->lock);
+
nfrags = skb_shinfo(skb)->nr_frags;
__skb_fill_page_desc(skb, nfrags, page, pfrag->offset,
@@ -319,108 +282,111 @@
if (sk)
atomic_add(tailen, &sk->sk_wmem_alloc);
- skb_push(skb, -skb_network_offset(skb));
-
- esph->seq_no = htonl(XFRM_SKB_CB(skb)->seq.output.low);
- esph->spi = x->id.spi;
-
- tmp = esp_alloc_tmp(aead, nfrags + 2, seqhilen);
- if (!tmp) {
- spin_unlock_bh(&x->lock);
- err = -ENOMEM;
- goto error;
- }
- seqhi = esp_tmp_seqhi(tmp);
- iv = esp_tmp_iv(aead, tmp, seqhilen);
- req = esp_tmp_req(aead, iv);
- sg = esp_req_sg(aead, req);
- dsg = &sg[nfrags];
-
- esph = esp_output_set_esn(skb, esph, seqhi);
-
- sg_init_table(sg, nfrags);
- skb_to_sgvec(skb, sg,
- (unsigned char *)esph - skb->data,
- assoclen + ivlen + clen + alen);
-
- allocsize = ALIGN(skb->data_len, L1_CACHE_BYTES);
-
- if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
- spin_unlock_bh(&x->lock);
- err = -ENOMEM;
- goto error;
- }
-
- skb_shinfo(skb)->nr_frags = 1;
-
- page = pfrag->page;
- get_page(page);
- /* replace page frags in skb with new page */
- __skb_fill_page_desc(skb, 0, page, pfrag->offset, skb->data_len);
- pfrag->offset = pfrag->offset + allocsize;
-
- sg_init_table(dsg, skb_shinfo(skb)->nr_frags + 1);
- skb_to_sgvec(skb, dsg,
- (unsigned char *)esph - skb->data,
- assoclen + ivlen + clen + alen);
-
- spin_unlock_bh(&x->lock);
-
- goto skip_cow2;
+ goto out;
}
}
cow:
- err = skb_cow_data(skb, tailen, &trailer);
- if (err < 0)
- goto error;
- nfrags = err;
-
+ nfrags = skb_cow_data(skb, tailen, &trailer);
+ if (nfrags < 0)
+ goto out;
tail = skb_tail_pointer(trailer);
- esph = ip_esp_hdr(skb);
skip_cow:
- esp_output_fill_trailer(tail, tfclen, plen, proto);
+ esp_output_fill_trailer(tail, esp->tfclen, esp->plen, esp->proto);
+ pskb_put(skb, trailer, tailen);
- pskb_put(skb, trailer, clen - skb->len + alen);
- skb_push(skb, -skb_network_offset(skb));
+out:
+ return nfrags;
+}
+EXPORT_SYMBOL_GPL(esp6_output_head);
- esph->seq_no = htonl(XFRM_SKB_CB(skb)->seq.output.low);
- esph->spi = x->id.spi;
+int esp6_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *esp)
+{
+ u8 *iv;
+ int alen;
+ void *tmp;
+ int ivlen;
+ int assoclen;
+ int seqhilen;
+ __be32 *seqhi;
+ struct page *page;
+ struct ip_esp_hdr *esph;
+ struct aead_request *req;
+ struct crypto_aead *aead;
+ struct scatterlist *sg, *dsg;
+ int err = -ENOMEM;
- tmp = esp_alloc_tmp(aead, nfrags, seqhilen);
- if (!tmp) {
- err = -ENOMEM;
- goto error;
+ assoclen = sizeof(struct ip_esp_hdr);
+ seqhilen = 0;
+
+ if (x->props.flags & XFRM_STATE_ESN) {
+ seqhilen += sizeof(__be32);
+ assoclen += sizeof(__be32);
}
+ aead = x->data;
+ alen = crypto_aead_authsize(aead);
+ ivlen = crypto_aead_ivsize(aead);
+
+ tmp = esp_alloc_tmp(aead, esp->nfrags + 2, seqhilen);
+ if (!tmp)
+ goto error;
+
seqhi = esp_tmp_seqhi(tmp);
iv = esp_tmp_iv(aead, tmp, seqhilen);
req = esp_tmp_req(aead, iv);
sg = esp_req_sg(aead, req);
- dsg = sg;
- esph = esp_output_set_esn(skb, esph, seqhi);
+ if (esp->inplace)
+ dsg = sg;
+ else
+ dsg = &sg[esp->nfrags];
- sg_init_table(sg, nfrags);
+ esph = esp_output_set_esn(skb, x, ip_esp_hdr(skb), seqhi);
+
+ sg_init_table(sg, esp->nfrags);
skb_to_sgvec(skb, sg,
(unsigned char *)esph - skb->data,
- assoclen + ivlen + clen + alen);
+ assoclen + ivlen + esp->clen + alen);
-skip_cow2:
+ if (!esp->inplace) {
+ int allocsize;
+ struct page_frag *pfrag = &x->xfrag;
+
+ allocsize = ALIGN(skb->data_len, L1_CACHE_BYTES);
+
+ spin_lock_bh(&x->lock);
+ if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+ spin_unlock_bh(&x->lock);
+ goto error;
+ }
+
+ skb_shinfo(skb)->nr_frags = 1;
+
+ page = pfrag->page;
+ get_page(page);
+ /* replace page frags in skb with new page */
+ __skb_fill_page_desc(skb, 0, page, pfrag->offset, skb->data_len);
+ pfrag->offset = pfrag->offset + allocsize;
+ spin_unlock_bh(&x->lock);
+
+ sg_init_table(dsg, skb_shinfo(skb)->nr_frags + 1);
+ skb_to_sgvec(skb, dsg,
+ (unsigned char *)esph - skb->data,
+ assoclen + ivlen + esp->clen + alen);
+ }
+
if ((x->props.flags & XFRM_STATE_ESN))
aead_request_set_callback(req, 0, esp_output_done_esn, skb);
else
aead_request_set_callback(req, 0, esp_output_done, skb);
- aead_request_set_crypt(req, sg, dsg, ivlen + clen, iv);
+ aead_request_set_crypt(req, sg, dsg, ivlen + esp->clen, iv);
aead_request_set_ad(req, assoclen);
- seqno = cpu_to_be64(XFRM_SKB_CB(skb)->seq.output.low +
- ((u64)XFRM_SKB_CB(skb)->seq.output.hi << 32));
-
memset(iv, 0, ivlen);
- memcpy(iv + ivlen - min(ivlen, 8), (u8 *)&seqno + 8 - min(ivlen, 8),
+ memcpy(iv + ivlen - min(ivlen, 8), (u8 *)&esp->seqno + 8 - min(ivlen, 8),
min(ivlen, 8));
ESP_SKB_CB(skb)->tmp = tmp;
@@ -446,10 +412,60 @@
error:
return err;
}
+EXPORT_SYMBOL_GPL(esp6_output_tail);
-static int esp_input_done2(struct sk_buff *skb, int err)
+static int esp6_output(struct xfrm_state *x, struct sk_buff *skb)
+{
+ int alen;
+ int blksize;
+ struct ip_esp_hdr *esph;
+ struct crypto_aead *aead;
+ struct esp_info esp;
+
+ esp.inplace = true;
+
+ esp.proto = *skb_mac_header(skb);
+ *skb_mac_header(skb) = IPPROTO_ESP;
+
+ /* skb is pure payload to encrypt */
+
+ aead = x->data;
+ alen = crypto_aead_authsize(aead);
+
+ esp.tfclen = 0;
+ if (x->tfcpad) {
+ struct xfrm_dst *dst = (struct xfrm_dst *)skb_dst(skb);
+ u32 padto;
+
+ padto = min(x->tfcpad, esp6_get_mtu(x, dst->child_mtu_cached));
+ if (skb->len < padto)
+ esp.tfclen = padto - skb->len;
+ }
+ blksize = ALIGN(crypto_aead_blocksize(aead), 4);
+ esp.clen = ALIGN(skb->len + 2 + esp.tfclen, blksize);
+ esp.plen = esp.clen - skb->len - esp.tfclen;
+ esp.tailen = esp.tfclen + esp.plen + alen;
+
+ esp.nfrags = esp6_output_head(x, skb, &esp);
+ if (esp.nfrags < 0)
+ return esp.nfrags;
+
+ esph = ip_esp_hdr(skb);
+ esph->spi = x->id.spi;
+
+ esph->seq_no = htonl(XFRM_SKB_CB(skb)->seq.output.low);
+ esp.seqno = cpu_to_be64(XFRM_SKB_CB(skb)->seq.output.low +
+ ((u64)XFRM_SKB_CB(skb)->seq.output.hi << 32));
+
+ skb_push(skb, -skb_network_offset(skb));
+
+ return esp6_output_tail(x, skb, &esp);
+}
+
+int esp6_input_done2(struct sk_buff *skb, int err)
{
struct xfrm_state *x = xfrm_input_state(skb);
+ struct xfrm_offload *xo = xfrm_offload(skb);
struct crypto_aead *aead = x->data;
int alen = crypto_aead_authsize(aead);
int hlen = sizeof(struct ip_esp_hdr) + crypto_aead_ivsize(aead);
@@ -458,7 +474,8 @@
int padlen;
u8 nexthdr[2];
- kfree(ESP_SKB_CB(skb)->tmp);
+ if (!xo || (xo && !(xo->flags & CRYPTO_DONE)))
+ kfree(ESP_SKB_CB(skb)->tmp);
if (unlikely(err))
goto out;
@@ -492,12 +509,13 @@
out:
return err;
}
+EXPORT_SYMBOL_GPL(esp6_input_done2);
static void esp_input_done(struct crypto_async_request *base, int err)
{
struct sk_buff *skb = base->data;
- xfrm_input_resume(skb, esp_input_done2(skb, err));
+ xfrm_input_resume(skb, esp6_input_done2(skb, err));
}
static void esp_input_restore_header(struct sk_buff *skb)
@@ -619,7 +637,7 @@
if ((x->props.flags & XFRM_STATE_ESN))
esp_input_restore_header(skb);
- ret = esp_input_done2(skb, ret);
+ ret = esp6_input_done2(skb, ret);
out:
return ret;
@@ -682,13 +700,17 @@
char aead_name[CRYPTO_MAX_ALG_NAME];
struct crypto_aead *aead;
int err;
+ u32 mask = 0;
err = -ENAMETOOLONG;
if (snprintf(aead_name, CRYPTO_MAX_ALG_NAME, "%s(%s)",
x->geniv, x->aead->alg_name) >= CRYPTO_MAX_ALG_NAME)
goto error;
- aead = crypto_alloc_aead(aead_name, 0, 0);
+ if (x->xso.offload_handle)
+ mask |= CRYPTO_ALG_ASYNC;
+
+ aead = crypto_alloc_aead(aead_name, 0, mask);
err = PTR_ERR(aead);
if (IS_ERR(aead))
goto error;
@@ -718,6 +740,7 @@
char authenc_name[CRYPTO_MAX_ALG_NAME];
unsigned int keylen;
int err;
+ u32 mask = 0;
err = -EINVAL;
if (!x->ealg)
@@ -743,7 +766,10 @@
goto error;
}
- aead = crypto_alloc_aead(authenc_name, 0, 0);
+ if (x->xso.offload_handle)
+ mask |= CRYPTO_ALG_ASYNC;
+
+ aead = crypto_alloc_aead(authenc_name, 0, mask);
err = PTR_ERR(aead);
if (IS_ERR(aead))
goto error;
diff --git a/net/ipv6/esp6_offload.c b/net/ipv6/esp6_offload.c
index d914eb9..d950d43 100644
--- a/net/ipv6/esp6_offload.c
+++ b/net/ipv6/esp6_offload.c
@@ -45,27 +45,31 @@
if ((err = xfrm_parse_spi(skb, IPPROTO_ESP, &spi, &seq)) != 0)
goto out;
- err = secpath_set(skb);
- if (err)
- goto out;
-
- if (skb->sp->len == XFRM_MAX_DEPTH)
- goto out;
-
- x = xfrm_state_lookup(dev_net(skb->dev), skb->mark,
- (xfrm_address_t *)&ipv6_hdr(skb)->daddr,
- spi, IPPROTO_ESP, AF_INET6);
- if (!x)
- goto out;
-
- skb->sp->xvec[skb->sp->len++] = x;
- skb->sp->olen++;
-
xo = xfrm_offload(skb);
- if (!xo) {
- xfrm_state_put(x);
- goto out;
+ if (!xo || !(xo->flags & CRYPTO_DONE)) {
+ err = secpath_set(skb);
+ if (err)
+ goto out;
+
+ if (skb->sp->len == XFRM_MAX_DEPTH)
+ goto out;
+
+ x = xfrm_state_lookup(dev_net(skb->dev), skb->mark,
+ (xfrm_address_t *)&ipv6_hdr(skb)->daddr,
+ spi, IPPROTO_ESP, AF_INET6);
+ if (!x)
+ goto out;
+
+ skb->sp->xvec[skb->sp->len++] = x;
+ skb->sp->olen++;
+
+ xo = xfrm_offload(skb);
+ if (!xo) {
+ xfrm_state_put(x);
+ goto out;
+ }
}
+
xo->flags |= XFRM_GRO;
XFRM_TUNNEL_SKB_CB(skb)->tunnel.ip6 = NULL;
@@ -86,19 +90,216 @@
return NULL;
}
+static void esp6_gso_encap(struct xfrm_state *x, struct sk_buff *skb)
+{
+ struct ip_esp_hdr *esph;
+ struct ipv6hdr *iph = ipv6_hdr(skb);
+ struct xfrm_offload *xo = xfrm_offload(skb);
+ int proto = iph->nexthdr;
+
+ skb_push(skb, -skb_network_offset(skb));
+ esph = ip_esp_hdr(skb);
+ *skb_mac_header(skb) = IPPROTO_ESP;
+
+ esph->spi = x->id.spi;
+ esph->seq_no = htonl(XFRM_SKB_CB(skb)->seq.output.low);
+
+ xo->proto = proto;
+}
+
+static struct sk_buff *esp6_gso_segment(struct sk_buff *skb,
+ netdev_features_t features)
+{
+ __u32 seq;
+ int err = 0;
+ struct sk_buff *skb2;
+ struct xfrm_state *x;
+ struct ip_esp_hdr *esph;
+ struct crypto_aead *aead;
+ struct sk_buff *segs = ERR_PTR(-EINVAL);
+ netdev_features_t esp_features = features;
+ struct xfrm_offload *xo = xfrm_offload(skb);
+
+ if (!xo)
+ goto out;
+
+ seq = xo->seq.low;
+
+ x = skb->sp->xvec[skb->sp->len - 1];
+ aead = x->data;
+ esph = ip_esp_hdr(skb);
+
+ if (esph->spi != x->id.spi)
+ goto out;
+
+ if (!pskb_may_pull(skb, sizeof(*esph) + crypto_aead_ivsize(aead)))
+ goto out;
+
+ __skb_pull(skb, sizeof(*esph) + crypto_aead_ivsize(aead));
+
+ skb->encap_hdr_csum = 1;
+
+ if (!(features & NETIF_F_HW_ESP))
+ esp_features = features & ~(NETIF_F_SG | NETIF_F_CSUM_MASK);
+
+ segs = x->outer_mode->gso_segment(x, skb, esp_features);
+ if (IS_ERR_OR_NULL(segs))
+ goto out;
+
+ __skb_pull(skb, skb->data - skb_mac_header(skb));
+
+ skb2 = segs;
+ do {
+ struct sk_buff *nskb = skb2->next;
+
+ xo = xfrm_offload(skb2);
+ xo->flags |= XFRM_GSO_SEGMENT;
+ xo->seq.low = seq;
+ xo->seq.hi = xfrm_replay_seqhi(x, seq);
+
+ if(!(features & NETIF_F_HW_ESP))
+ xo->flags |= CRYPTO_FALLBACK;
+
+ x->outer_mode->xmit(x, skb2);
+
+ err = x->type_offload->xmit(x, skb2, esp_features);
+ if (err) {
+ kfree_skb_list(segs);
+ return ERR_PTR(err);
+ }
+
+ if (!skb_is_gso(skb2))
+ seq++;
+ else
+ seq += skb_shinfo(skb2)->gso_segs;
+
+ skb_push(skb2, skb2->mac_len);
+ skb2 = nskb;
+ } while (skb2);
+
+out:
+ return segs;
+}
+
+static int esp6_input_tail(struct xfrm_state *x, struct sk_buff *skb)
+{
+ struct crypto_aead *aead = x->data;
+
+ if (!pskb_may_pull(skb, sizeof(struct ip_esp_hdr) + crypto_aead_ivsize(aead)))
+ return -EINVAL;
+
+ skb->ip_summed = CHECKSUM_NONE;
+
+ return esp6_input_done2(skb, 0);
+}
+
+static int esp6_xmit(struct xfrm_state *x, struct sk_buff *skb, netdev_features_t features)
+{
+ int err;
+ int alen;
+ int blksize;
+ struct xfrm_offload *xo;
+ struct ip_esp_hdr *esph;
+ struct crypto_aead *aead;
+ struct esp_info esp;
+ bool hw_offload = true;
+
+ esp.inplace = true;
+
+ xo = xfrm_offload(skb);
+
+ if (!xo)
+ return -EINVAL;
+
+ if (!(features & NETIF_F_HW_ESP) || !x->xso.offload_handle ||
+ (x->xso.dev != skb->dev)) {
+ xo->flags |= CRYPTO_FALLBACK;
+ hw_offload = false;
+ }
+
+ esp.proto = xo->proto;
+
+ /* skb is pure payload to encrypt */
+
+ aead = x->data;
+ alen = crypto_aead_authsize(aead);
+
+ esp.tfclen = 0;
+ /* XXX: Add support for tfc padding here. */
+
+ blksize = ALIGN(crypto_aead_blocksize(aead), 4);
+ esp.clen = ALIGN(skb->len + 2 + esp.tfclen, blksize);
+ esp.plen = esp.clen - skb->len - esp.tfclen;
+ esp.tailen = esp.tfclen + esp.plen + alen;
+
+ if (!hw_offload || (hw_offload && !skb_is_gso(skb))) {
+ esp.nfrags = esp6_output_head(x, skb, &esp);
+ if (esp.nfrags < 0)
+ return esp.nfrags;
+ }
+
+ esph = ip_esp_hdr(skb);
+ esph->spi = x->id.spi;
+
+ skb_push(skb, -skb_network_offset(skb));
+
+ if (xo->flags & XFRM_GSO_SEGMENT) {
+ esph->seq_no = htonl(xo->seq.low);
+ } else {
+ int len;
+
+ len = skb->len - sizeof(struct ipv6hdr);
+ if (len > IPV6_MAXPLEN)
+ len = 0;
+
+ ipv6_hdr(skb)->payload_len = htons(len);
+ }
+
+ if (hw_offload)
+ return 0;
+
+ esp.seqno = cpu_to_be64(xo->seq.low + ((u64)xo->seq.hi << 32));
+
+ err = esp6_output_tail(x, skb, &esp);
+ if (err < 0)
+ return err;
+
+ secpath_reset(skb);
+
+ return 0;
+}
+
static const struct net_offload esp6_offload = {
.callbacks = {
.gro_receive = esp6_gro_receive,
+ .gso_segment = esp6_gso_segment,
},
};
+static const struct xfrm_type_offload esp6_type_offload = {
+ .description = "ESP6 OFFLOAD",
+ .owner = THIS_MODULE,
+ .proto = IPPROTO_ESP,
+ .input_tail = esp6_input_tail,
+ .xmit = esp6_xmit,
+ .encap = esp6_gso_encap,
+};
+
static int __init esp6_offload_init(void)
{
+ if (xfrm_register_type_offload(&esp6_type_offload, AF_INET6) < 0) {
+ pr_info("%s: can't add xfrm type offload\n", __func__);
+ return -EAGAIN;
+ }
+
return inet6_add_offload(&esp6_offload, IPPROTO_ESP);
}
static void __exit esp6_offload_exit(void)
{
+ if (xfrm_unregister_type_offload(&esp6_type_offload, AF_INET6) < 0)
+ pr_info("%s: can't remove xfrm type offload\n", __func__);
+
inet6_del_offload(&esp6_offload, IPPROTO_ESP);
}
diff --git a/net/ipv6/exthdrs.c b/net/ipv6/exthdrs.c
index 25192a3..b636f1da 100644
--- a/net/ipv6/exthdrs.c
+++ b/net/ipv6/exthdrs.c
@@ -909,6 +909,8 @@
{
switch (opt->type) {
case IPV6_SRCRT_TYPE_0:
+ case IPV6_SRCRT_STRICT:
+ case IPV6_SRCRT_TYPE_2:
ipv6_push_rthdr0(skb, proto, opt, addr_p, saddr);
break;
case IPV6_SRCRT_TYPE_4:
@@ -944,13 +946,13 @@
if (opt->hopopt)
ipv6_push_exthdr(skb, proto, NEXTHDR_HOP, opt->hopopt);
}
-EXPORT_SYMBOL(ipv6_push_nfrag_opts);
void ipv6_push_frag_opts(struct sk_buff *skb, struct ipv6_txoptions *opt, u8 *proto)
{
if (opt->dst1opt)
ipv6_push_exthdr(skb, proto, NEXTHDR_DEST, opt->dst1opt);
}
+EXPORT_SYMBOL(ipv6_push_frag_opts);
struct ipv6_txoptions *
ipv6_dup_options(struct sock *sk, struct ipv6_txoptions *opt)
@@ -1163,6 +1165,8 @@
switch (opt->srcrt->type) {
case IPV6_SRCRT_TYPE_0:
+ case IPV6_SRCRT_STRICT:
+ case IPV6_SRCRT_TYPE_2:
fl6->daddr = *((struct rt0_hdr *)opt->srcrt)->addr;
break;
case IPV6_SRCRT_TYPE_4:
diff --git a/net/ipv6/ila/ila_lwt.c b/net/ipv6/ila/ila_lwt.c
index ce1aae4..b3df03e 100644
--- a/net/ipv6/ila/ila_lwt.c
+++ b/net/ipv6/ila/ila_lwt.c
@@ -146,8 +146,7 @@
return -EINVAL;
}
- ret = nla_parse_nested(tb, ILA_ATTR_MAX, nla,
- ila_nl_policy);
+ ret = nla_parse_nested(tb, ILA_ATTR_MAX, nla, ila_nl_policy, NULL);
if (ret < 0)
return ret;
diff --git a/net/ipv6/ip6_gre.c b/net/ipv6/ip6_gre.c
index 6fcb7cb..8d128ba 100644
--- a/net/ipv6/ip6_gre.c
+++ b/net/ipv6/ip6_gre.c
@@ -544,6 +544,8 @@
& IPV6_TCLASS_MASK;
if (t->parms.flags & IP6_TNL_F_USE_ORIG_FWMARK)
fl6.flowi6_mark = skb->mark;
+ else
+ fl6.flowi6_mark = t->parms.fwmark;
fl6.flowi6_uid = sock_net_uid(dev_net(dev), NULL);
@@ -603,6 +605,8 @@
fl6.flowlabel |= ip6_flowlabel(ipv6h);
if (t->parms.flags & IP6_TNL_F_USE_ORIG_FWMARK)
fl6.flowi6_mark = skb->mark;
+ else
+ fl6.flowi6_mark = t->parms.fwmark;
fl6.flowi6_uid = sock_net_uid(dev_net(dev), NULL);
@@ -780,6 +784,7 @@
t->parms.o_key = p->o_key;
t->parms.i_flags = p->i_flags;
t->parms.o_flags = p->o_flags;
+ t->parms.fwmark = p->fwmark;
dst_cache_reset(&t->dst_cache);
ip6gre_tnl_link_config(t, set_mtu);
return 0;
@@ -1249,6 +1254,9 @@
if (data[IFLA_GRE_FLAGS])
parms->flags = nla_get_u32(data[IFLA_GRE_FLAGS]);
+
+ if (data[IFLA_GRE_FWMARK])
+ parms->fwmark = nla_get_u32(data[IFLA_GRE_FWMARK]);
}
static int ip6gre_tap_init(struct net_device *dev)
@@ -1470,6 +1478,8 @@
nla_total_size(2) +
/* IFLA_GRE_ENCAP_DPORT */
nla_total_size(2) +
+ /* IFLA_GRE_FWMARK */
+ nla_total_size(4) +
0;
}
@@ -1490,7 +1500,8 @@
nla_put_u8(skb, IFLA_GRE_TTL, p->hop_limit) ||
nla_put_u8(skb, IFLA_GRE_ENCAP_LIMIT, p->encap_limit) ||
nla_put_be32(skb, IFLA_GRE_FLOWINFO, p->flowinfo) ||
- nla_put_u32(skb, IFLA_GRE_FLAGS, p->flags))
+ nla_put_u32(skb, IFLA_GRE_FLAGS, p->flags) ||
+ nla_put_u32(skb, IFLA_GRE_FWMARK, p->fwmark))
goto nla_put_failure;
if (nla_put_u16(skb, IFLA_GRE_ENCAP_TYPE,
@@ -1525,6 +1536,7 @@
[IFLA_GRE_ENCAP_FLAGS] = { .type = NLA_U16 },
[IFLA_GRE_ENCAP_SPORT] = { .type = NLA_U16 },
[IFLA_GRE_ENCAP_DPORT] = { .type = NLA_U16 },
+ [IFLA_GRE_FWMARK] = { .type = NLA_U32 },
};
static struct rtnl_link_ops ip6gre_link_ops __read_mostly = {
diff --git a/net/ipv6/ip6_input.c b/net/ipv6/ip6_input.c
index c45b12b..9ee208a 100644
--- a/net/ipv6/ip6_input.c
+++ b/net/ipv6/ip6_input.c
@@ -49,6 +49,8 @@
int ip6_rcv_finish(struct net *net, struct sock *sk, struct sk_buff *skb)
{
+ void (*edemux)(struct sk_buff *skb);
+
/* if ingress device is enslaved to an L3 master device pass the
* skb to its handler for processing
*/
@@ -60,8 +62,8 @@
const struct inet6_protocol *ipprot;
ipprot = rcu_dereference(inet6_protos[ipv6_hdr(skb)->nexthdr]);
- if (ipprot && ipprot->early_demux)
- ipprot->early_demux(skb);
+ if (ipprot && (edemux = READ_ONCE(ipprot->early_demux)))
+ edemux(skb);
}
if (!skb_valid_dst(skb))
ip6_route_input(skb);
diff --git a/net/ipv6/ip6_tunnel.c b/net/ipv6/ip6_tunnel.c
index 75fac93..6eb2ae5 100644
--- a/net/ipv6/ip6_tunnel.c
+++ b/net/ipv6/ip6_tunnel.c
@@ -954,7 +954,7 @@
opt->dst_opt[5] = IPV6_TLV_PADN;
opt->dst_opt[6] = 1;
- opt->ops.dst0opt = (struct ipv6_opt_hdr *) opt->dst_opt;
+ opt->ops.dst1opt = (struct ipv6_opt_hdr *) opt->dst_opt;
opt->ops.opt_nflen = 8;
}
@@ -1037,7 +1037,7 @@
struct ip6_tnl *t = netdev_priv(dev);
struct net *net = t->net;
struct net_device_stats *stats = &t->dev->stats;
- struct ipv6hdr *ipv6h = ipv6_hdr(skb);
+ struct ipv6hdr *ipv6h;
struct ipv6_tel_txoption opt;
struct dst_entry *dst = NULL, *ndst = NULL;
struct net_device *tdev;
@@ -1057,26 +1057,28 @@
/* NBMA tunnel */
if (ipv6_addr_any(&t->parms.raddr)) {
- struct in6_addr *addr6;
- struct neighbour *neigh;
- int addr_type;
+ if (skb->protocol == htons(ETH_P_IPV6)) {
+ struct in6_addr *addr6;
+ struct neighbour *neigh;
+ int addr_type;
- if (!skb_dst(skb))
- goto tx_err_link_failure;
+ if (!skb_dst(skb))
+ goto tx_err_link_failure;
- neigh = dst_neigh_lookup(skb_dst(skb),
- &ipv6_hdr(skb)->daddr);
- if (!neigh)
- goto tx_err_link_failure;
+ neigh = dst_neigh_lookup(skb_dst(skb),
+ &ipv6_hdr(skb)->daddr);
+ if (!neigh)
+ goto tx_err_link_failure;
- addr6 = (struct in6_addr *)&neigh->primary_key;
- addr_type = ipv6_addr_type(addr6);
+ addr6 = (struct in6_addr *)&neigh->primary_key;
+ addr_type = ipv6_addr_type(addr6);
- if (addr_type == IPV6_ADDR_ANY)
- addr6 = &ipv6_hdr(skb)->daddr;
+ if (addr_type == IPV6_ADDR_ANY)
+ addr6 = &ipv6_hdr(skb)->daddr;
- memcpy(&fl6->daddr, addr6, sizeof(fl6->daddr));
- neigh_release(neigh);
+ memcpy(&fl6->daddr, addr6, sizeof(fl6->daddr));
+ neigh_release(neigh);
+ }
} else if (!(t->parms.flags &
(IP6_TNL_F_USE_ORIG_TCLASS | IP6_TNL_F_USE_ORIG_FWMARK))) {
/* enable the cache only only if the routing decision does
@@ -1176,7 +1178,7 @@
if (encap_limit >= 0) {
init_tel_txopt(&opt, encap_limit);
- ipv6_push_nfrag_opts(skb, &opt.ops, &proto, NULL, NULL);
+ ipv6_push_frag_opts(skb, &opt.ops, &proto);
}
/* Calculate max headroom for all the headers and adjust
@@ -1256,6 +1258,8 @@
& IPV6_TCLASS_MASK;
if (t->parms.flags & IP6_TNL_F_USE_ORIG_FWMARK)
fl6.flowi6_mark = skb->mark;
+ else
+ fl6.flowi6_mark = t->parms.fwmark;
}
fl6.flowi6_uid = sock_net_uid(dev_net(dev), NULL);
@@ -1338,6 +1342,8 @@
fl6.flowlabel |= ip6_flowlabel(ipv6h);
if (t->parms.flags & IP6_TNL_F_USE_ORIG_FWMARK)
fl6.flowi6_mark = skb->mark;
+ else
+ fl6.flowi6_mark = t->parms.fwmark;
}
fl6.flowi6_uid = sock_net_uid(dev_net(dev), NULL);
@@ -1467,6 +1473,7 @@
t->parms.flowinfo = p->flowinfo;
t->parms.link = p->link;
t->parms.proto = p->proto;
+ t->parms.fwmark = p->fwmark;
dst_cache_reset(&t->dst_cache);
ip6_tnl_link_config(t);
return 0;
@@ -1918,6 +1925,9 @@
if (data[IFLA_IPTUN_COLLECT_METADATA])
parms->collect_md = true;
+
+ if (data[IFLA_IPTUN_FWMARK])
+ parms->fwmark = nla_get_u32(data[IFLA_IPTUN_FWMARK]);
}
static bool ip6_tnl_netlink_encap_parms(struct nlattr *data[],
@@ -2054,6 +2064,8 @@
nla_total_size(2) +
/* IFLA_IPTUN_COLLECT_METADATA */
nla_total_size(0) +
+ /* IFLA_IPTUN_FWMARK */
+ nla_total_size(4) +
0;
}
@@ -2069,7 +2081,8 @@
nla_put_u8(skb, IFLA_IPTUN_ENCAP_LIMIT, parm->encap_limit) ||
nla_put_be32(skb, IFLA_IPTUN_FLOWINFO, parm->flowinfo) ||
nla_put_u32(skb, IFLA_IPTUN_FLAGS, parm->flags) ||
- nla_put_u8(skb, IFLA_IPTUN_PROTO, parm->proto))
+ nla_put_u8(skb, IFLA_IPTUN_PROTO, parm->proto) ||
+ nla_put_u32(skb, IFLA_IPTUN_FWMARK, parm->fwmark))
goto nla_put_failure;
if (nla_put_u16(skb, IFLA_IPTUN_ENCAP_TYPE, tunnel->encap.type) ||
@@ -2081,6 +2094,7 @@
if (parm->collect_md)
if (nla_put_flag(skb, IFLA_IPTUN_COLLECT_METADATA))
goto nla_put_failure;
+
return 0;
nla_put_failure:
@@ -2109,6 +2123,7 @@
[IFLA_IPTUN_ENCAP_SPORT] = { .type = NLA_U16 },
[IFLA_IPTUN_ENCAP_DPORT] = { .type = NLA_U16 },
[IFLA_IPTUN_COLLECT_METADATA] = { .type = NLA_FLAG },
+ [IFLA_IPTUN_FWMARK] = { .type = NLA_U32 },
};
static struct rtnl_link_ops ip6_link_ops __read_mostly = {
diff --git a/net/ipv6/ip6_vti.c b/net/ipv6/ip6_vti.c
index 3d8a3b6..d67ef56 100644
--- a/net/ipv6/ip6_vti.c
+++ b/net/ipv6/ip6_vti.c
@@ -657,6 +657,7 @@
t->parms.i_key = p->i_key;
t->parms.o_key = p->o_key;
t->parms.proto = p->proto;
+ t->parms.fwmark = p->fwmark;
dst_cache_reset(&t->dst_cache);
vti6_link_config(t);
return 0;
@@ -933,6 +934,9 @@
if (data[IFLA_VTI_OKEY])
parms->o_key = nla_get_be32(data[IFLA_VTI_OKEY]);
+
+ if (data[IFLA_VTI_FWMARK])
+ parms->fwmark = nla_get_u32(data[IFLA_VTI_FWMARK]);
}
static int vti6_newlink(struct net *src_net, struct net_device *dev,
@@ -998,6 +1002,8 @@
nla_total_size(4) +
/* IFLA_VTI_OKEY */
nla_total_size(4) +
+ /* IFLA_VTI_FWMARK */
+ nla_total_size(4) +
0;
}
@@ -1010,7 +1016,8 @@
nla_put_in6_addr(skb, IFLA_VTI_LOCAL, &parm->laddr) ||
nla_put_in6_addr(skb, IFLA_VTI_REMOTE, &parm->raddr) ||
nla_put_be32(skb, IFLA_VTI_IKEY, parm->i_key) ||
- nla_put_be32(skb, IFLA_VTI_OKEY, parm->o_key))
+ nla_put_be32(skb, IFLA_VTI_OKEY, parm->o_key) ||
+ nla_put_u32(skb, IFLA_VTI_FWMARK, parm->fwmark))
goto nla_put_failure;
return 0;
@@ -1024,6 +1031,7 @@
[IFLA_VTI_REMOTE] = { .len = sizeof(struct in6_addr) },
[IFLA_VTI_IKEY] = { .type = NLA_U32 },
[IFLA_VTI_OKEY] = { .type = NLA_U32 },
+ [IFLA_VTI_FWMARK] = { .type = NLA_U32 },
};
static struct rtnl_link_ops vti6_link_ops __read_mostly = {
diff --git a/net/ipv6/ip6mr.c b/net/ipv6/ip6mr.c
index bf34d09..374997d 100644
--- a/net/ipv6/ip6mr.c
+++ b/net/ipv6/ip6mr.c
@@ -816,7 +816,7 @@
in6_dev = __in6_dev_get(dev);
if (in6_dev) {
in6_dev->cnf.mc_forwarding--;
- inet6_netconf_notify_devconf(dev_net(dev),
+ inet6_netconf_notify_devconf(dev_net(dev), RTM_NEWNETCONF,
NETCONFA_MC_FORWARDING,
dev->ifindex, &in6_dev->cnf);
}
@@ -975,7 +975,7 @@
in6_dev = __in6_dev_get(dev);
if (in6_dev) {
in6_dev->cnf.mc_forwarding++;
- inet6_netconf_notify_devconf(dev_net(dev),
+ inet6_netconf_notify_devconf(dev_net(dev), RTM_NEWNETCONF,
NETCONFA_MC_FORWARDING,
dev->ifindex, &in6_dev->cnf);
}
@@ -1598,7 +1598,8 @@
write_unlock_bh(&mrt_lock);
if (!err)
- inet6_netconf_notify_devconf(net, NETCONFA_MC_FORWARDING,
+ inet6_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_MC_FORWARDING,
NETCONFA_IFINDEX_ALL,
net->ipv6.devconf_all);
rtnl_unlock();
@@ -1619,7 +1620,7 @@
mrt->mroute6_sk = NULL;
net->ipv6.devconf_all->mc_forwarding--;
write_unlock_bh(&mrt_lock);
- inet6_netconf_notify_devconf(net,
+ inet6_netconf_notify_devconf(net, RTM_NEWNETCONF,
NETCONFA_MC_FORWARDING,
NETCONFA_IFINDEX_ALL,
net->ipv6.devconf_all);
diff --git a/net/ipv6/mcast.c b/net/ipv6/mcast.c
index 1bdc703..07403fa 100644
--- a/net/ipv6/mcast.c
+++ b/net/ipv6/mcast.c
@@ -2463,7 +2463,6 @@
mld_ifc_start_timer(idev, 1);
}
-
static void igmp6_timer_handler(unsigned long data)
{
struct ifmcaddr6 *ma = (struct ifmcaddr6 *) data;
@@ -2599,6 +2598,44 @@
write_unlock_bh(&idev->lock);
}
+static void ipv6_mc_rejoin_groups(struct inet6_dev *idev)
+{
+ struct ifmcaddr6 *pmc;
+
+ ASSERT_RTNL();
+
+ if (mld_in_v1_mode(idev)) {
+ read_lock_bh(&idev->lock);
+ for (pmc = idev->mc_list; pmc; pmc = pmc->next)
+ igmp6_join_group(pmc);
+ read_unlock_bh(&idev->lock);
+ } else
+ mld_send_report(idev, NULL);
+}
+
+static int ipv6_mc_netdev_event(struct notifier_block *this,
+ unsigned long event,
+ void *ptr)
+{
+ struct net_device *dev = netdev_notifier_info_to_dev(ptr);
+ struct inet6_dev *idev = __in6_dev_get(dev);
+
+ switch (event) {
+ case NETDEV_RESEND_IGMP:
+ if (idev)
+ ipv6_mc_rejoin_groups(idev);
+ break;
+ default:
+ break;
+ }
+
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block igmp6_netdev_notifier = {
+ .notifier_call = ipv6_mc_netdev_event,
+};
+
#ifdef CONFIG_PROC_FS
struct igmp6_mc_iter_state {
struct seq_net_private p;
@@ -2970,7 +3007,17 @@
return register_pernet_subsys(&igmp6_net_ops);
}
+int __init igmp6_late_init(void)
+{
+ return register_netdevice_notifier(&igmp6_netdev_notifier);
+}
+
void igmp6_cleanup(void)
{
unregister_pernet_subsys(&igmp6_net_ops);
}
+
+void igmp6_late_cleanup(void)
+{
+ unregister_netdevice_notifier(&igmp6_netdev_notifier);
+}
diff --git a/net/ipv6/ndisc.c b/net/ipv6/ndisc.c
index 7ebac63..d310dc4 100644
--- a/net/ipv6/ndisc.c
+++ b/net/ipv6/ndisc.c
@@ -732,7 +732,7 @@
const u8 *lladdr, u8 new, u32 flags, u8 icmp6_type,
struct ndisc_options *ndopts)
{
- neigh_update(neigh, lladdr, new, flags);
+ neigh_update(neigh, lladdr, new, flags, 0);
/* report ndisc ops about neighbour update */
ndisc_ops_update(dev, neigh, flags, icmp6_type, ndopts);
}
@@ -1418,6 +1418,8 @@
if (ri->prefix_len == 0 &&
!in6_dev->cnf.accept_ra_defrtr)
continue;
+ if (ri->prefix_len < in6_dev->cnf.accept_ra_rt_info_min_plen)
+ continue;
if (ri->prefix_len > in6_dev->cnf.accept_ra_rt_info_max_plen)
continue;
rt6_route_rcv(skb->dev, (u8 *)p, (p->nd_opt_len) << 3,
@@ -1746,10 +1748,13 @@
case NETDEV_CHANGEADDR:
neigh_changeaddr(&nd_tbl, dev);
fib6_run_gc(0, net, false);
+ /* fallthrough */
+ case NETDEV_UP:
idev = in6_dev_get(dev);
if (!idev)
break;
- if (idev->cnf.ndisc_notify)
+ if (idev->cnf.ndisc_notify ||
+ net->ipv6.devconf_all->ndisc_notify)
ndisc_send_unsol_na(dev);
in6_dev_put(idev);
break;
diff --git a/net/ipv6/netfilter/ip6_tables.c b/net/ipv6/netfilter/ip6_tables.c
index 1e15c54..1f90644 100644
--- a/net/ipv6/netfilter/ip6_tables.c
+++ b/net/ipv6/netfilter/ip6_tables.c
@@ -51,15 +51,6 @@
}
EXPORT_SYMBOL_GPL(ip6t_alloc_initial_table);
-/*
- We keep a set of rules for each CPU, so we can avoid write-locking
- them in the softirq when updating the counters and therefore
- only need to read-lock in the softirq; doing a write_lock_bh() in user
- context stops packets coming through and allows user context to read
- the counters or update the rules.
-
- Hence the start of any table is given by get_table() below. */
-
/* Returns whether matches rule or not. */
/* Performance critical - called for every packet */
static inline bool
@@ -411,7 +402,7 @@
to 0 as we leave), and comefrom to save source hook bitmask */
for (hook = 0; hook < NF_INET_NUMHOOKS; hook++) {
unsigned int pos = newinfo->hook_entry[hook];
- struct ip6t_entry *e = (struct ip6t_entry *)(entry0 + pos);
+ struct ip6t_entry *e = entry0 + pos;
if (!(valid_hooks & (1 << hook)))
continue;
@@ -453,14 +444,12 @@
if (pos == oldpos)
goto next;
- e = (struct ip6t_entry *)
- (entry0 + pos);
+ e = entry0 + pos;
} while (oldpos == pos + e->next_offset);
/* Move along one */
size = e->next_offset;
- e = (struct ip6t_entry *)
- (entry0 + pos + size);
+ e = entry0 + pos + size;
if (pos + size >= newinfo->size)
return 0;
e->counters.pcnt = pos;
@@ -475,16 +464,14 @@
if (!xt_find_jump_offset(offsets, newpos,
newinfo->number))
return 0;
- e = (struct ip6t_entry *)
- (entry0 + newpos);
+ e = entry0 + newpos;
} else {
/* ... this is a fallthru */
newpos = pos + e->next_offset;
if (newpos >= newinfo->size)
return 0;
}
- e = (struct ip6t_entry *)
- (entry0 + newpos);
+ e = entry0 + newpos;
e->counters.pcnt = pos;
pos = newpos;
}
@@ -863,7 +850,7 @@
const struct xt_entry_match *m;
const struct xt_entry_target *t;
- e = (struct ip6t_entry *)(loc_cpu_entry + off);
+ e = loc_cpu_entry + off;
if (copy_to_user(userptr + off, e, sizeof(*e))) {
ret = -EFAULT;
goto free_counters;
@@ -1258,7 +1245,7 @@
int ret = 0;
origsize = *size;
- ce = (struct compat_ip6t_entry __user *)*dstptr;
+ ce = *dstptr;
if (copy_to_user(ce, e, sizeof(struct ip6t_entry)) != 0 ||
copy_to_user(&ce->counters, &counters[i],
sizeof(counters[i])) != 0)
@@ -1394,7 +1381,7 @@
struct xt_entry_match *ematch;
origsize = *size;
- de = (struct ip6t_entry *)*dstptr;
+ de = *dstptr;
memcpy(de, e, sizeof(struct ip6t_entry));
memcpy(&de->counters, &e->counters, sizeof(e->counters));
diff --git a/net/ipv6/netfilter/ip6t_SYNPROXY.c b/net/ipv6/netfilter/ip6t_SYNPROXY.c
index 4ef1ddd..d3c4daa 100644
--- a/net/ipv6/netfilter/ip6t_SYNPROXY.c
+++ b/net/ipv6/netfilter/ip6t_SYNPROXY.c
@@ -307,12 +307,17 @@
XT_SYNPROXY_OPT_ECN);
synproxy_send_client_synack(net, skb, th, &opts);
- return NF_DROP;
+ consume_skb(skb);
+ return NF_STOLEN;
} else if (th->ack && !(th->fin || th->rst || th->syn)) {
/* ACK from client */
- synproxy_recv_client_ack(net, skb, th, &opts, ntohl(th->seq));
- return NF_DROP;
+ if (synproxy_recv_client_ack(net, skb, th, &opts, ntohl(th->seq))) {
+ consume_skb(skb);
+ return NF_STOLEN;
+ } else {
+ return NF_DROP;
+ }
}
return XT_CONTINUE;
@@ -388,10 +393,13 @@
* number match the one of first SYN.
*/
if (synproxy_recv_client_ack(net, skb, th, &opts,
- ntohl(th->seq) + 1))
+ ntohl(th->seq) + 1)) {
this_cpu_inc(snet->stats->cookie_retrans);
-
- return NF_DROP;
+ consume_skb(skb);
+ return NF_STOLEN;
+ } else {
+ return NF_DROP;
+ }
}
synproxy->isn = ntohl(th->ack_seq);
@@ -430,34 +438,6 @@
return NF_ACCEPT;
}
-static int synproxy_tg6_check(const struct xt_tgchk_param *par)
-{
- const struct ip6t_entry *e = par->entryinfo;
-
- if (!(e->ipv6.flags & IP6T_F_PROTO) ||
- e->ipv6.proto != IPPROTO_TCP ||
- e->ipv6.invflags & XT_INV_PROTO)
- return -EINVAL;
-
- return nf_ct_netns_get(par->net, par->family);
-}
-
-static void synproxy_tg6_destroy(const struct xt_tgdtor_param *par)
-{
- nf_ct_netns_put(par->net, par->family);
-}
-
-static struct xt_target synproxy_tg6_reg __read_mostly = {
- .name = "SYNPROXY",
- .family = NFPROTO_IPV6,
- .hooks = (1 << NF_INET_LOCAL_IN) | (1 << NF_INET_FORWARD),
- .target = synproxy_tg6,
- .targetsize = sizeof(struct xt_synproxy_info),
- .checkentry = synproxy_tg6_check,
- .destroy = synproxy_tg6_destroy,
- .me = THIS_MODULE,
-};
-
static struct nf_hook_ops ipv6_synproxy_ops[] __read_mostly = {
{
.hook = ipv6_synproxy_hook,
@@ -473,31 +453,64 @@
},
};
-static int __init synproxy_tg6_init(void)
+static int synproxy_tg6_check(const struct xt_tgchk_param *par)
{
+ struct synproxy_net *snet = synproxy_pernet(par->net);
+ const struct ip6t_entry *e = par->entryinfo;
int err;
- err = nf_register_hooks(ipv6_synproxy_ops,
- ARRAY_SIZE(ipv6_synproxy_ops));
- if (err < 0)
- goto err1;
+ if (!(e->ipv6.flags & IP6T_F_PROTO) ||
+ e->ipv6.proto != IPPROTO_TCP ||
+ e->ipv6.invflags & XT_INV_PROTO)
+ return -EINVAL;
- err = xt_register_target(&synproxy_tg6_reg);
- if (err < 0)
- goto err2;
+ err = nf_ct_netns_get(par->net, par->family);
+ if (err)
+ return err;
- return 0;
+ if (snet->hook_ref6 == 0) {
+ err = nf_register_net_hooks(par->net, ipv6_synproxy_ops,
+ ARRAY_SIZE(ipv6_synproxy_ops));
+ if (err) {
+ nf_ct_netns_put(par->net, par->family);
+ return err;
+ }
+ }
-err2:
- nf_unregister_hooks(ipv6_synproxy_ops, ARRAY_SIZE(ipv6_synproxy_ops));
-err1:
+ snet->hook_ref6++;
return err;
}
+static void synproxy_tg6_destroy(const struct xt_tgdtor_param *par)
+{
+ struct synproxy_net *snet = synproxy_pernet(par->net);
+
+ snet->hook_ref6--;
+ if (snet->hook_ref6 == 0)
+ nf_unregister_net_hooks(par->net, ipv6_synproxy_ops,
+ ARRAY_SIZE(ipv6_synproxy_ops));
+ nf_ct_netns_put(par->net, par->family);
+}
+
+static struct xt_target synproxy_tg6_reg __read_mostly = {
+ .name = "SYNPROXY",
+ .family = NFPROTO_IPV6,
+ .hooks = (1 << NF_INET_LOCAL_IN) | (1 << NF_INET_FORWARD),
+ .target = synproxy_tg6,
+ .targetsize = sizeof(struct xt_synproxy_info),
+ .checkentry = synproxy_tg6_check,
+ .destroy = synproxy_tg6_destroy,
+ .me = THIS_MODULE,
+};
+
+static int __init synproxy_tg6_init(void)
+{
+ return xt_register_target(&synproxy_tg6_reg);
+}
+
static void __exit synproxy_tg6_exit(void)
{
xt_unregister_target(&synproxy_tg6_reg);
- nf_unregister_hooks(ipv6_synproxy_ops, ARRAY_SIZE(ipv6_synproxy_ops));
}
module_init(synproxy_tg6_init);
diff --git a/net/ipv6/netfilter/nf_conntrack_proto_icmpv6.c b/net/ipv6/netfilter/nf_conntrack_proto_icmpv6.c
index d2c2ccb..d5f028e 100644
--- a/net/ipv6/netfilter/nf_conntrack_proto_icmpv6.c
+++ b/net/ipv6/netfilter/nf_conntrack_proto_icmpv6.c
@@ -221,8 +221,7 @@
type = icmp6h->icmp6_type - 130;
if (type >= 0 && type < sizeof(noct_valid_new) &&
noct_valid_new[type]) {
- nf_ct_set(skb, nf_ct_untracked_get(), IP_CT_NEW);
- nf_conntrack_get(skb_nfct(skb));
+ nf_ct_set(skb, NULL, IP_CT_UNTRACKED);
return NF_ACCEPT;
}
diff --git a/net/ipv6/netfilter/nf_dup_ipv6.c b/net/ipv6/netfilter/nf_dup_ipv6.c
index 888ecd1..4a7dded 100644
--- a/net/ipv6/netfilter/nf_dup_ipv6.c
+++ b/net/ipv6/netfilter/nf_dup_ipv6.c
@@ -58,8 +58,7 @@
#if IS_ENABLED(CONFIG_NF_CONNTRACK)
nf_reset(skb);
- nf_ct_set(skb, nf_ct_untracked_get(), IP_CT_NEW);
- nf_conntrack_get(skb_nfct(skb));
+ nf_ct_set(skb, NULL, IP_CT_UNTRACKED);
#endif
if (hooknum == NF_INET_PRE_ROUTING ||
hooknum == NF_INET_LOCAL_IN) {
diff --git a/net/ipv6/netfilter/nf_nat_l3proto_ipv6.c b/net/ipv6/netfilter/nf_nat_l3proto_ipv6.c
index 69937b6..b2b4f03 100644
--- a/net/ipv6/netfilter/nf_nat_l3proto_ipv6.c
+++ b/net/ipv6/netfilter/nf_nat_l3proto_ipv6.c
@@ -273,13 +273,7 @@
if (!ct)
return NF_ACCEPT;
- /* Don't try to NAT if this packet is not conntracked */
- if (nf_ct_is_untracked(ct))
- return NF_ACCEPT;
-
- nat = nf_ct_nat_ext_add(ct);
- if (nat == NULL)
- return NF_ACCEPT;
+ nat = nfct_nat(ct);
switch (ctinfo) {
case IP_CT_RELATED:
diff --git a/net/ipv6/netfilter/nf_nat_masquerade_ipv6.c b/net/ipv6/netfilter/nf_nat_masquerade_ipv6.c
index 051b6a6..2297c9f 100644
--- a/net/ipv6/netfilter/nf_nat_masquerade_ipv6.c
+++ b/net/ipv6/netfilter/nf_nat_masquerade_ipv6.c
@@ -30,6 +30,7 @@
const struct net_device *out)
{
enum ip_conntrack_info ctinfo;
+ struct nf_conn_nat *nat;
struct in6_addr src;
struct nf_conn *ct;
struct nf_nat_range newrange;
@@ -42,7 +43,9 @@
&ipv6_hdr(skb)->daddr, 0, &src) < 0)
return NF_DROP;
- nfct_nat(ct)->masq_index = out->ifindex;
+ nat = nf_ct_nat_ext_add(ct);
+ if (nat)
+ nat->masq_index = out->ifindex;
newrange.flags = range->flags | NF_NAT_RANGE_MAP_IPS;
newrange.min_addr.in6 = src;
diff --git a/net/ipv6/netfilter/nft_fib_ipv6.c b/net/ipv6/netfilter/nft_fib_ipv6.c
index 765facf..43f91d9 100644
--- a/net/ipv6/netfilter/nft_fib_ipv6.c
+++ b/net/ipv6/netfilter/nft_fib_ipv6.c
@@ -159,7 +159,7 @@
if (nft_hook(pkt) == NF_INET_PRE_ROUTING &&
nft_fib_is_loopback(pkt->skb, nft_in(pkt))) {
- nft_fib_store_result(dest, priv->result, pkt,
+ nft_fib_store_result(dest, priv, pkt,
nft_in(pkt)->ifindex);
return;
}
@@ -246,7 +246,7 @@
static struct nft_expr_type nft_fib6_type __read_mostly = {
.name = "fib",
- .select_ops = &nft_fib6_select_ops,
+ .select_ops = nft_fib6_select_ops,
.policy = nft_fib_policy,
.maxattr = NFTA_FIB_MAX,
.family = NFPROTO_IPV6,
diff --git a/net/ipv6/protocol.c b/net/ipv6/protocol.c
index e3770ab..b5d54d4 100644
--- a/net/ipv6/protocol.c
+++ b/net/ipv6/protocol.c
@@ -26,7 +26,7 @@
#include <net/protocol.h>
#if IS_ENABLED(CONFIG_IPV6)
-const struct inet6_protocol __rcu *inet6_protos[MAX_INET_PROTOS] __read_mostly;
+struct inet6_protocol __rcu *inet6_protos[MAX_INET_PROTOS] __read_mostly;
EXPORT_SYMBOL(inet6_protos);
int inet6_add_protocol(const struct inet6_protocol *prot, unsigned char protocol)
diff --git a/net/ipv6/raw.c b/net/ipv6/raw.c
index f174e76..0da6a12 100644
--- a/net/ipv6/raw.c
+++ b/net/ipv6/raw.c
@@ -1178,8 +1178,7 @@
spin_lock_bh(&sk->sk_receive_queue.lock);
skb = skb_peek(&sk->sk_receive_queue);
if (skb)
- amount = skb_tail_pointer(skb) -
- skb_transport_header(skb);
+ amount = skb->len;
spin_unlock_bh(&sk->sk_receive_queue.lock);
return put_user(amount, (int __user *)arg);
}
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index fb174b5..a1bf426 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -2910,7 +2910,8 @@
unsigned int pref;
int err;
- err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_ipv6_policy);
+ err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_ipv6_policy,
+ NULL);
if (err < 0)
goto errout;
@@ -3263,7 +3264,8 @@
return last_err;
}
-static int inet6_rtm_delroute(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int inet6_rtm_delroute(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct fib6_config cfg;
int err;
@@ -3280,7 +3282,8 @@
}
}
-static int inet6_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int inet6_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct fib6_config cfg;
int err;
@@ -3568,7 +3571,8 @@
NLM_F_MULTI);
}
-static int inet6_rtm_getroute(struct sk_buff *in_skb, struct nlmsghdr *nlh)
+static int inet6_rtm_getroute(struct sk_buff *in_skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(in_skb->sk);
struct nlattr *tb[RTA_MAX+1];
@@ -3578,7 +3582,8 @@
struct flowi6 fl6;
int err, iif = 0, oif = 0;
- err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_ipv6_policy);
+ err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_ipv6_policy,
+ extack);
if (err < 0)
goto errout;
diff --git a/net/ipv6/seg6_iptunnel.c b/net/ipv6/seg6_iptunnel.c
index 8558225..6a49549 100644
--- a/net/ipv6/seg6_iptunnel.c
+++ b/net/ipv6/seg6_iptunnel.c
@@ -26,17 +26,13 @@
#include <linux/seg6_iptunnel.h>
#include <net/addrconf.h>
#include <net/ip6_route.h>
-#ifdef CONFIG_DST_CACHE
#include <net/dst_cache.h>
-#endif
#ifdef CONFIG_IPV6_SEG6_HMAC
#include <net/seg6_hmac.h>
#endif
struct seg6_lwt {
-#ifdef CONFIG_DST_CACHE
struct dst_cache cache;
-#endif
struct seg6_iptunnel_encap tuninfo[0];
};
@@ -105,7 +101,7 @@
hdrlen = (osrh->hdrlen + 1) << 3;
tot_len = hdrlen + sizeof(*hdr);
- err = pskb_expand_head(skb, tot_len, 0, GFP_ATOMIC);
+ err = skb_cow_head(skb, tot_len);
if (unlikely(err))
return err;
@@ -156,7 +152,7 @@
hdrlen = (osrh->hdrlen + 1) << 3;
- err = pskb_expand_head(skb, hdrlen, 0, GFP_ATOMIC);
+ err = skb_cow_head(skb, hdrlen);
if (unlikely(err))
return err;
@@ -237,6 +233,9 @@
static int seg6_input(struct sk_buff *skb)
{
+ struct dst_entry *orig_dst = skb_dst(skb);
+ struct dst_entry *dst = NULL;
+ struct seg6_lwt *slwt;
int err;
err = seg6_do_srh(skb);
@@ -245,8 +244,30 @@
return err;
}
+ slwt = seg6_lwt_lwtunnel(orig_dst->lwtstate);
+
+ preempt_disable();
+ dst = dst_cache_get(&slwt->cache);
+ preempt_enable();
+
skb_dst_drop(skb);
- ip6_route_input(skb);
+
+ if (!dst) {
+ ip6_route_input(skb);
+ dst = skb_dst(skb);
+ if (!dst->error) {
+ preempt_disable();
+ dst_cache_set_ip6(&slwt->cache, dst,
+ &ipv6_hdr(skb)->saddr);
+ preempt_enable();
+ }
+ } else {
+ skb_dst_set(skb, dst);
+ }
+
+ err = skb_cow_head(skb, LL_RESERVED_SPACE(dst->dev));
+ if (unlikely(err))
+ return err;
return dst_input(skb);
}
@@ -264,11 +285,9 @@
slwt = seg6_lwt_lwtunnel(orig_dst->lwtstate);
-#ifdef CONFIG_DST_CACHE
preempt_disable();
dst = dst_cache_get(&slwt->cache);
preempt_enable();
-#endif
if (unlikely(!dst)) {
struct ipv6hdr *hdr = ipv6_hdr(skb);
@@ -287,16 +306,18 @@
goto drop;
}
-#ifdef CONFIG_DST_CACHE
preempt_disable();
dst_cache_set_ip6(&slwt->cache, dst, &fl6.saddr);
preempt_enable();
-#endif
}
skb_dst_drop(skb);
skb_dst_set(skb, dst);
+ err = skb_cow_head(skb, LL_RESERVED_SPACE(dst->dev));
+ if (unlikely(err))
+ goto drop;
+
return dst_output(net, sk, skb);
drop:
kfree_skb(skb);
@@ -315,7 +336,7 @@
int err;
err = nla_parse_nested(tb, SEG6_IPTUNNEL_MAX, nla,
- seg6_iptunnel_policy);
+ seg6_iptunnel_policy, NULL);
if (err < 0)
return err;
@@ -355,13 +376,11 @@
slwt = seg6_lwt_lwtunnel(newts);
-#ifdef CONFIG_DST_CACHE
err = dst_cache_init(&slwt->cache, GFP_KERNEL);
if (err) {
kfree(newts);
return err;
}
-#endif
memcpy(&slwt->tuninfo, tuninfo, tuninfo_len);
@@ -375,12 +394,10 @@
return 0;
}
-#ifdef CONFIG_DST_CACHE
static void seg6_destroy_state(struct lwtunnel_state *lwt)
{
dst_cache_destroy(&seg6_lwt_lwtunnel(lwt)->cache);
}
-#endif
static int seg6_fill_encap_info(struct sk_buff *skb,
struct lwtunnel_state *lwtstate)
@@ -414,9 +431,7 @@
static const struct lwtunnel_encap_ops seg6_iptun_ops = {
.build_state = seg6_build_state,
-#ifdef CONFIG_DST_CACHE
.destroy_state = seg6_destroy_state,
-#endif
.output = seg6_output,
.input = seg6_input,
.fill_encap = seg6_fill_encap_info,
diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
index 99853c6..61e5902 100644
--- a/net/ipv6/sit.c
+++ b/net/ipv6/sit.c
@@ -881,11 +881,12 @@
goto tx_error;
}
- rt = ip_route_output_ports(tunnel->net, &fl4, NULL,
- dst, tiph->saddr,
- 0, 0,
- IPPROTO_IPV6, RT_TOS(tos),
- tunnel->parms.link);
+ flowi4_init_output(&fl4, tunnel->parms.link, tunnel->fwmark,
+ RT_TOS(tos), RT_SCOPE_UNIVERSE, IPPROTO_IPV6,
+ 0, dst, tiph->saddr, 0, 0,
+ sock_net_uid(tunnel->net, NULL));
+ rt = ip_route_output_flow(tunnel->net, &fl4, NULL);
+
if (IS_ERR(rt)) {
dev->stats.tx_carrier_errors++;
goto tx_error_icmp;
@@ -1071,7 +1072,8 @@
}
}
-static void ipip6_tunnel_update(struct ip_tunnel *t, struct ip_tunnel_parm *p)
+static void ipip6_tunnel_update(struct ip_tunnel *t, struct ip_tunnel_parm *p,
+ __u32 fwmark)
{
struct net *net = t->net;
struct sit_net *sitn = net_generic(net, sit_net_id);
@@ -1085,8 +1087,9 @@
ipip6_tunnel_link(sitn, t);
t->parms.iph.ttl = p->iph.ttl;
t->parms.iph.tos = p->iph.tos;
- if (t->parms.link != p->link) {
+ if (t->parms.link != p->link || t->fwmark != fwmark) {
t->parms.link = p->link;
+ t->fwmark = fwmark;
ipip6_tunnel_bind_dev(t->dev);
}
dst_cache_reset(&t->dst_cache);
@@ -1220,7 +1223,7 @@
t = netdev_priv(dev);
}
- ipip6_tunnel_update(t, &p);
+ ipip6_tunnel_update(t, &p, t->fwmark);
}
if (t) {
@@ -1418,7 +1421,8 @@
}
static void ipip6_netlink_parms(struct nlattr *data[],
- struct ip_tunnel_parm *parms)
+ struct ip_tunnel_parm *parms,
+ __u32 *fwmark)
{
memset(parms, 0, sizeof(*parms));
@@ -1457,6 +1461,8 @@
if (data[IFLA_IPTUN_PROTO])
parms->iph.protocol = nla_get_u8(data[IFLA_IPTUN_PROTO]);
+ if (data[IFLA_IPTUN_FWMARK])
+ *fwmark = nla_get_u32(data[IFLA_IPTUN_FWMARK]);
}
/* This function returns true when ENCAP attributes are present in the nl msg */
@@ -1549,7 +1555,7 @@
return err;
}
- ipip6_netlink_parms(data, &nt->parms);
+ ipip6_netlink_parms(data, &nt->parms, &nt->fwmark);
if (ipip6_tunnel_locate(net, &nt->parms, 0))
return -EEXIST;
@@ -1577,6 +1583,7 @@
#ifdef CONFIG_IPV6_SIT_6RD
struct ip_tunnel_6rd ip6rd;
#endif
+ __u32 fwmark = t->fwmark;
int err;
if (dev == sitn->fb_tunnel_dev)
@@ -1588,7 +1595,7 @@
return err;
}
- ipip6_netlink_parms(data, &p);
+ ipip6_netlink_parms(data, &p, &fwmark);
if (((dev->flags & IFF_POINTOPOINT) && !p.iph.daddr) ||
(!(dev->flags & IFF_POINTOPOINT) && p.iph.daddr))
@@ -1602,7 +1609,7 @@
} else
t = netdev_priv(dev);
- ipip6_tunnel_update(t, &p);
+ ipip6_tunnel_update(t, &p, fwmark);
#ifdef CONFIG_IPV6_SIT_6RD
if (ipip6_netlink_6rd_parms(data, &ip6rd))
@@ -1649,6 +1656,8 @@
nla_total_size(2) +
/* IFLA_IPTUN_ENCAP_DPORT */
nla_total_size(2) +
+ /* IFLA_IPTUN_FWMARK */
+ nla_total_size(4) +
0;
}
@@ -1665,7 +1674,8 @@
nla_put_u8(skb, IFLA_IPTUN_PMTUDISC,
!!(parm->iph.frag_off & htons(IP_DF))) ||
nla_put_u8(skb, IFLA_IPTUN_PROTO, parm->iph.protocol) ||
- nla_put_be16(skb, IFLA_IPTUN_FLAGS, parm->i_flags))
+ nla_put_be16(skb, IFLA_IPTUN_FLAGS, parm->i_flags) ||
+ nla_put_u32(skb, IFLA_IPTUN_FWMARK, tunnel->fwmark))
goto nla_put_failure;
#ifdef CONFIG_IPV6_SIT_6RD
@@ -1715,6 +1725,7 @@
[IFLA_IPTUN_ENCAP_FLAGS] = { .type = NLA_U16 },
[IFLA_IPTUN_ENCAP_SPORT] = { .type = NLA_U16 },
[IFLA_IPTUN_ENCAP_DPORT] = { .type = NLA_U16 },
+ [IFLA_IPTUN_FWMARK] = { .type = NLA_U32 },
};
static void ipip6_dellink(struct net_device *dev, struct list_head *head)
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 49fa2e8..8e42e8f 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -101,12 +101,12 @@
}
}
-static u32 tcp_v6_init_sequence(const struct sk_buff *skb, u32 *tsoff)
+static u32 tcp_v6_init_seq_and_tsoff(const struct sk_buff *skb, u32 *tsoff)
{
- return secure_tcpv6_sequence_number(ipv6_hdr(skb)->daddr.s6_addr32,
- ipv6_hdr(skb)->saddr.s6_addr32,
- tcp_hdr(skb)->dest,
- tcp_hdr(skb)->source, tsoff);
+ return secure_tcpv6_seq_and_tsoff(ipv6_hdr(skb)->daddr.s6_addr32,
+ ipv6_hdr(skb)->saddr.s6_addr32,
+ tcp_hdr(skb)->dest,
+ tcp_hdr(skb)->source, tsoff);
}
static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr,
@@ -265,11 +265,6 @@
sk->sk_gso_type = SKB_GSO_TCPV6;
ip6_dst_store(sk, dst, NULL, NULL);
- if (tcp_death_row->sysctl_tw_recycle &&
- !tp->rx_opt.ts_recent_stamp &&
- ipv6_addr_equal(&fl6.daddr, &sk->sk_v6_daddr))
- tcp_fetch_timewait_stamp(sk, dst);
-
icsk->icsk_ext_hdr_len = 0;
if (opt)
icsk->icsk_ext_hdr_len = opt->opt_flen +
@@ -287,11 +282,11 @@
sk_set_txhash(sk);
if (likely(!tp->repair)) {
- seq = secure_tcpv6_sequence_number(np->saddr.s6_addr32,
- sk->sk_v6_daddr.s6_addr32,
- inet->inet_sport,
- inet->inet_dport,
- &tp->tsoffset);
+ seq = secure_tcpv6_seq_and_tsoff(np->saddr.s6_addr32,
+ sk->sk_v6_daddr.s6_addr32,
+ inet->inet_sport,
+ inet->inet_dport,
+ &tp->tsoffset);
if (!tp->write_seq)
tp->write_seq = seq;
}
@@ -727,11 +722,8 @@
static struct dst_entry *tcp_v6_route_req(const struct sock *sk,
struct flowi *fl,
- const struct request_sock *req,
- bool *strict)
+ const struct request_sock *req)
{
- if (strict)
- *strict = true;
return inet6_csk_route_req(sk, &fl->u.ip6, req, IPPROTO_TCP);
}
@@ -757,7 +749,7 @@
.cookie_init_seq = cookie_v6_init_sequence,
#endif
.route_req = tcp_v6_route_req,
- .init_seq = tcp_v6_init_sequence,
+ .init_seq_tsoff = tcp_v6_init_seq_and_tsoff,
.send_synack = tcp_v6_send_synack,
};
@@ -1301,8 +1293,6 @@
goto discard;
if (nsk != sk) {
- sock_rps_save_rxhash(nsk, skb);
- sk_mark_napi_id(nsk, skb);
if (tcp_child_process(sk, nsk, skb))
goto reset;
if (opt_skb)
@@ -1933,8 +1923,9 @@
.diag_destroy = tcp_abort,
};
-static const struct inet6_protocol tcpv6_protocol = {
+static struct inet6_protocol tcpv6_protocol = {
.early_demux = tcp_v6_early_demux,
+ .early_demux_handler = tcp_v6_early_demux,
.handler = tcp_v6_rcv,
.err_handler = tcp_v6_err,
.flags = INET6_PROTO_NOPOLICY|INET6_PROTO_FINAL,
diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
index e28082f..04862ab 100644
--- a/net/ipv6/udp.c
+++ b/net/ipv6/udp.c
@@ -46,6 +46,7 @@
#include <net/tcp_states.h>
#include <net/ip6_checksum.h>
#include <net/xfrm.h>
+#include <net/inet_hashtables.h>
#include <net/inet6_hashtables.h>
#include <net/busy_poll.h>
#include <net/sock_reuseport.h>
@@ -864,6 +865,69 @@
return 0;
}
+
+static struct sock *__udp6_lib_demux_lookup(struct net *net,
+ __be16 loc_port, const struct in6_addr *loc_addr,
+ __be16 rmt_port, const struct in6_addr *rmt_addr,
+ int dif)
+{
+ unsigned short hnum = ntohs(loc_port);
+ unsigned int hash2 = udp6_portaddr_hash(net, loc_addr, hnum);
+ unsigned int slot2 = hash2 & udp_table.mask;
+ struct udp_hslot *hslot2 = &udp_table.hash2[slot2];
+ const __portpair ports = INET_COMBINED_PORTS(rmt_port, hnum);
+ struct sock *sk;
+
+ udp_portaddr_for_each_entry_rcu(sk, &hslot2->head) {
+ if (INET6_MATCH(sk, net, rmt_addr, loc_addr, ports, dif))
+ return sk;
+ /* Only check first socket in chain */
+ break;
+ }
+ return NULL;
+}
+
+static void udp_v6_early_demux(struct sk_buff *skb)
+{
+ struct net *net = dev_net(skb->dev);
+ const struct udphdr *uh;
+ struct sock *sk;
+ struct dst_entry *dst;
+ int dif = skb->dev->ifindex;
+
+ if (!pskb_may_pull(skb, skb_transport_offset(skb) +
+ sizeof(struct udphdr)))
+ return;
+
+ uh = udp_hdr(skb);
+
+ if (skb->pkt_type == PACKET_HOST)
+ sk = __udp6_lib_demux_lookup(net, uh->dest,
+ &ipv6_hdr(skb)->daddr,
+ uh->source, &ipv6_hdr(skb)->saddr,
+ dif);
+ else
+ return;
+
+ if (!sk || !atomic_inc_not_zero_hint(&sk->sk_refcnt, 2))
+ return;
+
+ skb->sk = sk;
+ skb->destructor = sock_efree;
+ dst = READ_ONCE(sk->sk_rx_dst);
+
+ if (dst)
+ dst = dst_check(dst, inet6_sk(sk)->rx_dst_cookie);
+ if (dst) {
+ if (dst->flags & DST_NOCACHE) {
+ if (likely(atomic_inc_not_zero(&dst->__refcnt)))
+ skb_dst_set(skb, dst);
+ } else {
+ skb_dst_set_noref(skb, dst);
+ }
+ }
+}
+
static __inline__ int udpv6_rcv(struct sk_buff *skb)
{
return __udp6_lib_rcv(skb, &udp_table, IPPROTO_UDP);
@@ -1378,7 +1442,9 @@
}
#endif
-static const struct inet6_protocol udpv6_protocol = {
+static struct inet6_protocol udpv6_protocol = {
+ .early_demux = udp_v6_early_demux,
+ .early_demux_handler = udp_v6_early_demux,
.handler = udpv6_rcv,
.err_handler = udpv6_err,
.flags = INET6_PROTO_NOPOLICY|INET6_PROTO_FINAL,
diff --git a/net/ipv6/xfrm6_mode_transport.c b/net/ipv6/xfrm6_mode_transport.c
index 4439ee4..7a92c0f 100644
--- a/net/ipv6/xfrm6_mode_transport.c
+++ b/net/ipv6/xfrm6_mode_transport.c
@@ -13,6 +13,7 @@
#include <net/dst.h>
#include <net/ipv6.h>
#include <net/xfrm.h>
+#include <net/protocol.h>
/* Add encapsulation header.
*
@@ -26,6 +27,7 @@
int hdr_len;
iph = ipv6_hdr(skb);
+ skb_set_inner_transport_header(skb, skb_transport_offset(skb));
hdr_len = x->type->hdr_offset(x, skb, &prevhdr);
skb_set_mac_header(skb, (prevhdr - x->props.header_len) - skb->data);
@@ -61,9 +63,41 @@
return 0;
}
+static struct sk_buff *xfrm4_transport_gso_segment(struct xfrm_state *x,
+ struct sk_buff *skb,
+ netdev_features_t features)
+{
+ const struct net_offload *ops;
+ struct sk_buff *segs = ERR_PTR(-EINVAL);
+ struct xfrm_offload *xo = xfrm_offload(skb);
+
+ skb->transport_header += x->props.header_len;
+ ops = rcu_dereference(inet6_offloads[xo->proto]);
+ if (likely(ops && ops->callbacks.gso_segment))
+ segs = ops->callbacks.gso_segment(skb, features);
+
+ return segs;
+}
+
+static void xfrm6_transport_xmit(struct xfrm_state *x, struct sk_buff *skb)
+{
+ struct xfrm_offload *xo = xfrm_offload(skb);
+
+ skb_reset_mac_len(skb);
+ pskb_pull(skb, skb->mac_len + sizeof(struct ipv6hdr) + x->props.header_len);
+
+ if (xo->flags & XFRM_GSO_SEGMENT) {
+ skb_reset_transport_header(skb);
+ skb->transport_header -= x->props.header_len;
+ }
+}
+
+
static struct xfrm_mode xfrm6_transport_mode = {
.input = xfrm6_transport_input,
.output = xfrm6_transport_output,
+ .gso_segment = xfrm4_transport_gso_segment,
+ .xmit = xfrm6_transport_xmit,
.owner = THIS_MODULE,
.encap = XFRM_MODE_TRANSPORT,
};
diff --git a/net/ipv6/xfrm6_mode_tunnel.c b/net/ipv6/xfrm6_mode_tunnel.c
index 372855e..02556e3 100644
--- a/net/ipv6/xfrm6_mode_tunnel.c
+++ b/net/ipv6/xfrm6_mode_tunnel.c
@@ -36,6 +36,9 @@
struct ipv6hdr *top_iph;
int dsfield;
+ skb_set_inner_network_header(skb, skb_network_offset(skb));
+ skb_set_inner_transport_header(skb, skb_transport_offset(skb));
+
skb_set_network_header(skb, -x->props.header_len);
skb->mac_header = skb->network_header +
offsetof(struct ipv6hdr, nexthdr);
@@ -96,11 +99,35 @@
return err;
}
+static struct sk_buff *xfrm6_mode_tunnel_gso_segment(struct xfrm_state *x,
+ struct sk_buff *skb,
+ netdev_features_t features)
+{
+ __skb_push(skb, skb->mac_len);
+ return skb_mac_gso_segment(skb, features);
+
+}
+
+static void xfrm6_mode_tunnel_xmit(struct xfrm_state *x, struct sk_buff *skb)
+{
+ struct xfrm_offload *xo = xfrm_offload(skb);
+
+ if (xo->flags & XFRM_GSO_SEGMENT) {
+ skb->network_header = skb->network_header - x->props.header_len;
+ skb->transport_header = skb->network_header + sizeof(struct ipv6hdr);
+ }
+
+ skb_reset_mac_len(skb);
+ pskb_pull(skb, skb->mac_len + x->props.header_len);
+}
+
static struct xfrm_mode xfrm6_tunnel_mode = {
.input2 = xfrm6_mode_tunnel_input,
.input = xfrm_prepare_input,
.output2 = xfrm6_mode_tunnel_output,
.output = xfrm6_prepare_output,
+ .gso_segment = xfrm6_mode_tunnel_gso_segment,
+ .xmit = xfrm6_mode_tunnel_xmit,
.owner = THIS_MODULE,
.encap = XFRM_MODE_TUNNEL,
.flags = XFRM_MODE_FLAG_TUNNEL,
diff --git a/net/ipv6/xfrm6_output.c b/net/ipv6/xfrm6_output.c
index 4d09ce6..8ae87d4 100644
--- a/net/ipv6/xfrm6_output.c
+++ b/net/ipv6/xfrm6_output.c
@@ -73,11 +73,16 @@
int mtu, ret = 0;
struct dst_entry *dst = skb_dst(skb);
+ if (skb->ignore_df)
+ goto out;
+
mtu = dst_mtu(dst);
if (mtu < IPV6_MIN_MTU)
mtu = IPV6_MIN_MTU;
- if (!skb->ignore_df && skb->len > mtu) {
+ if ((!skb_is_gso(skb) && skb->len > mtu) ||
+ (skb_is_gso(skb) &&
+ skb_gso_network_seglen(skb) > ip6_skb_dst_mtu(skb))) {
skb->dev = dst->dev;
skb->protocol = htons(ETH_P_IPV6);
@@ -89,7 +94,7 @@
icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
ret = -EMSGSIZE;
}
-
+out:
return ret;
}
diff --git a/net/ipx/af_ipx.c b/net/ipx/af_ipx.c
index 8a9219f..fa31ef2 100644
--- a/net/ipx/af_ipx.c
+++ b/net/ipx/af_ipx.c
@@ -1168,11 +1168,10 @@
sipx->sipx_network = ipxif->if_netnum;
memcpy(sipx->sipx_node, ipxif->if_node,
sizeof(sipx->sipx_node));
- rc = -EFAULT;
- if (copy_to_user(arg, &ifr, sizeof(ifr)))
- break;
- ipxitf_put(ipxif);
rc = 0;
+ if (copy_to_user(arg, &ifr, sizeof(ifr)))
+ rc = -EFAULT;
+ ipxitf_put(ipxif);
break;
}
case SIOCAIPXITFCRT:
diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c
index 31762f7..deca20f 100644
--- a/net/kcm/kcmsock.c
+++ b/net/kcm/kcmsock.c
@@ -1707,11 +1707,7 @@
struct kcm_clone info;
struct socket *newsock = NULL;
- if (copy_from_user(&info, (void __user *)arg, sizeof(info)))
- return -EFAULT;
-
err = kcm_clone(sock, &info, &newsock);
-
if (!err) {
if (copy_to_user((void __user *)arg, &info,
sizeof(info))) {
diff --git a/net/key/af_key.c b/net/key/af_key.c
index be8cecc..c1950bb 100644
--- a/net/key/af_key.c
+++ b/net/key/af_key.c
@@ -3827,7 +3827,6 @@
static struct xfrm_mgr pfkeyv2_mgr =
{
- .id = "pfkeyv2",
.notify = pfkey_send_notify,
.acquire = pfkey_send_acquire,
.compile_policy = pfkey_compile_policy,
diff --git a/net/l2tp/l2tp_core.c b/net/l2tp/l2tp_core.c
index e37d955..fa03425 100644
--- a/net/l2tp/l2tp_core.c
+++ b/net/l2tp/l2tp_core.c
@@ -120,7 +120,7 @@
return sk->sk_user_data;
}
-static inline struct l2tp_net *l2tp_pernet(struct net *net)
+static inline struct l2tp_net *l2tp_pernet(const struct net *net)
{
BUG_ON(!net);
@@ -217,27 +217,6 @@
sock_put(sk);
}
-/* Lookup a session by id in the global session list
- */
-static struct l2tp_session *l2tp_session_find_2(struct net *net, u32 session_id)
-{
- struct l2tp_net *pn = l2tp_pernet(net);
- struct hlist_head *session_list =
- l2tp_session_id_hash_2(pn, session_id);
- struct l2tp_session *session;
-
- rcu_read_lock_bh();
- hlist_for_each_entry_rcu(session, session_list, global_hlist) {
- if (session->session_id == session_id) {
- rcu_read_unlock_bh();
- return session;
- }
- }
- rcu_read_unlock_bh();
-
- return NULL;
-}
-
/* Session hash list.
* The session_id SHOULD be random according to RFC2661, but several
* L2TP implementations (Cisco and Microsoft) use incrementing
@@ -250,38 +229,10 @@
return &tunnel->session_hlist[hash_32(session_id, L2TP_HASH_BITS)];
}
-/* Lookup a session by id
- */
-struct l2tp_session *l2tp_session_find(struct net *net, struct l2tp_tunnel *tunnel, u32 session_id)
-{
- struct hlist_head *session_list;
- struct l2tp_session *session;
-
- /* In L2TPv3, session_ids are unique over all tunnels and we
- * sometimes need to look them up before we know the
- * tunnel.
- */
- if (tunnel == NULL)
- return l2tp_session_find_2(net, session_id);
-
- session_list = l2tp_session_id_hash(tunnel, session_id);
- read_lock_bh(&tunnel->hlist_lock);
- hlist_for_each_entry(session, session_list, hlist) {
- if (session->session_id == session_id) {
- read_unlock_bh(&tunnel->hlist_lock);
- return session;
- }
- }
- read_unlock_bh(&tunnel->hlist_lock);
-
- return NULL;
-}
-EXPORT_SYMBOL_GPL(l2tp_session_find);
-
-/* Like l2tp_session_find() but takes a reference on the returned session.
+/* Lookup a session. A new reference is held on the returned session.
* Optionally calls session->ref() too if do_ref is true.
*/
-struct l2tp_session *l2tp_session_get(struct net *net,
+struct l2tp_session *l2tp_session_get(const struct net *net,
struct l2tp_tunnel *tunnel,
u32 session_id, bool do_ref)
{
@@ -356,7 +307,8 @@
/* Lookup a session by interface name.
* This is very inefficient but is only used by management interfaces.
*/
-struct l2tp_session *l2tp_session_get_by_ifname(struct net *net, char *ifname,
+struct l2tp_session *l2tp_session_get_by_ifname(const struct net *net,
+ const char *ifname,
bool do_ref)
{
struct l2tp_net *pn = l2tp_pernet(net);
@@ -427,7 +379,7 @@
/* Lookup a tunnel by id
*/
-struct l2tp_tunnel *l2tp_tunnel_find(struct net *net, u32 tunnel_id)
+struct l2tp_tunnel *l2tp_tunnel_find(const struct net *net, u32 tunnel_id)
{
struct l2tp_tunnel *tunnel;
struct l2tp_net *pn = l2tp_pernet(net);
@@ -445,7 +397,7 @@
}
EXPORT_SYMBOL_GPL(l2tp_tunnel_find);
-struct l2tp_tunnel *l2tp_tunnel_find_nth(struct net *net, int nth)
+struct l2tp_tunnel *l2tp_tunnel_find_nth(const struct net *net, int nth)
{
struct l2tp_net *pn = l2tp_pernet(net);
struct l2tp_tunnel *tunnel;
diff --git a/net/l2tp/l2tp_core.h b/net/l2tp/l2tp_core.h
index 8ce7818..eec5ad2 100644
--- a/net/l2tp/l2tp_core.h
+++ b/net/l2tp/l2tp_core.h
@@ -230,18 +230,16 @@
return tunnel;
}
-struct l2tp_session *l2tp_session_get(struct net *net,
+struct l2tp_session *l2tp_session_get(const struct net *net,
struct l2tp_tunnel *tunnel,
u32 session_id, bool do_ref);
-struct l2tp_session *l2tp_session_find(struct net *net,
- struct l2tp_tunnel *tunnel,
- u32 session_id);
struct l2tp_session *l2tp_session_get_nth(struct l2tp_tunnel *tunnel, int nth,
bool do_ref);
-struct l2tp_session *l2tp_session_get_by_ifname(struct net *net, char *ifname,
+struct l2tp_session *l2tp_session_get_by_ifname(const struct net *net,
+ const char *ifname,
bool do_ref);
-struct l2tp_tunnel *l2tp_tunnel_find(struct net *net, u32 tunnel_id);
-struct l2tp_tunnel *l2tp_tunnel_find_nth(struct net *net, int nth);
+struct l2tp_tunnel *l2tp_tunnel_find(const struct net *net, u32 tunnel_id);
+struct l2tp_tunnel *l2tp_tunnel_find_nth(const struct net *net, int nth);
int l2tp_tunnel_create(struct net *net, int fd, int version, u32 tunnel_id,
u32 peer_tunnel_id, struct l2tp_tunnel_cfg *cfg,
diff --git a/net/l2tp/l2tp_eth.c b/net/l2tp/l2tp_eth.c
index 6fd41d7..8b21af7 100644
--- a/net/l2tp/l2tp_eth.c
+++ b/net/l2tp/l2tp_eth.c
@@ -30,6 +30,9 @@
#include <net/xfrm.h>
#include <net/net_namespace.h>
#include <net/netns/generic.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <linux/udp.h>
#include "l2tp_core.h"
@@ -127,8 +130,13 @@
.ndo_set_mac_address = eth_mac_addr,
};
+static struct device_type l2tpeth_type = {
+ .name = "l2tpeth",
+};
+
static void l2tp_eth_dev_setup(struct net_device *dev)
{
+ SET_NETDEV_DEVTYPE(dev, &l2tpeth_type);
ether_setup(dev);
dev->priv_flags &= ~IFF_TX_SKB_SHARING;
dev->features |= NETIF_F_LLTX;
@@ -204,8 +212,58 @@
}
#endif
+static void l2tp_eth_adjust_mtu(struct l2tp_tunnel *tunnel,
+ struct l2tp_session *session,
+ struct net_device *dev)
+{
+ unsigned int overhead = 0;
+ struct dst_entry *dst;
+ u32 l3_overhead = 0;
+
+ /* if the encap is UDP, account for UDP header size */
+ if (tunnel->encap == L2TP_ENCAPTYPE_UDP) {
+ overhead += sizeof(struct udphdr);
+ dev->needed_headroom += sizeof(struct udphdr);
+ }
+ if (session->mtu != 0) {
+ dev->mtu = session->mtu;
+ dev->needed_headroom += session->hdr_len;
+ return;
+ }
+ lock_sock(tunnel->sock);
+ l3_overhead = kernel_sock_ip_overhead(tunnel->sock);
+ release_sock(tunnel->sock);
+ if (l3_overhead == 0) {
+ /* L3 Overhead couldn't be identified, this could be
+ * because tunnel->sock was NULL or the socket's
+ * address family was not IPv4 or IPv6,
+ * dev mtu stays at 1500.
+ */
+ return;
+ }
+ /* Adjust MTU, factor overhead - underlay L3, overlay L2 hdr
+ * UDP overhead, if any, was already factored in above.
+ */
+ overhead += session->hdr_len + ETH_HLEN + l3_overhead;
+
+ /* If PMTU discovery was enabled, use discovered MTU on L2TP device */
+ dst = sk_dst_get(tunnel->sock);
+ if (dst) {
+ /* dst_mtu will use PMTU if found, else fallback to intf MTU */
+ u32 pmtu = dst_mtu(dst);
+
+ if (pmtu != 0)
+ dev->mtu = pmtu;
+ dst_release(dst);
+ }
+ session->mtu = dev->mtu - overhead;
+ dev->mtu = session->mtu;
+ dev->needed_headroom += session->hdr_len;
+}
+
static int l2tp_eth_create(struct net *net, u32 tunnel_id, u32 session_id, u32 peer_session_id, struct l2tp_session_cfg *cfg)
{
+ unsigned char name_assign_type;
struct net_device *dev;
char name[IFNAMSIZ];
struct l2tp_tunnel *tunnel;
@@ -222,15 +280,12 @@
}
if (cfg->ifname) {
- dev = dev_get_by_name(net, cfg->ifname);
- if (dev) {
- dev_put(dev);
- rc = -EEXIST;
- goto out;
- }
strlcpy(name, cfg->ifname, IFNAMSIZ);
- } else
+ name_assign_type = NET_NAME_USER;
+ } else {
strcpy(name, L2TP_ETH_DEV_NAME);
+ name_assign_type = NET_NAME_ENUM;
+ }
session = l2tp_session_create(sizeof(*spriv), tunnel, session_id,
peer_session_id, cfg);
@@ -239,7 +294,7 @@
goto out;
}
- dev = alloc_netdev(sizeof(*priv), name, NET_NAME_UNKNOWN,
+ dev = alloc_netdev(sizeof(*priv), name, name_assign_type,
l2tp_eth_dev_setup);
if (!dev) {
rc = -ENOMEM;
@@ -247,12 +302,9 @@
}
dev_net_set(dev, net);
- if (session->mtu == 0)
- session->mtu = dev->mtu - session->hdr_len;
- dev->mtu = session->mtu;
- dev->needed_headroom += session->hdr_len;
dev->min_mtu = 0;
dev->max_mtu = ETH_MAX_MTU;
+ l2tp_eth_adjust_mtu(tunnel, session, dev);
priv = netdev_priv(dev);
priv->dev = dev;
diff --git a/net/l2tp/l2tp_netlink.c b/net/l2tp/l2tp_netlink.c
index 7e3e669..12cfcd0 100644
--- a/net/l2tp/l2tp_netlink.c
+++ b/net/l2tp/l2tp_netlink.c
@@ -521,11 +521,6 @@
goto out;
}
session_id = nla_get_u32(info->attrs[L2TP_ATTR_SESSION_ID]);
- session = l2tp_session_find(net, tunnel, session_id);
- if (session) {
- ret = -EEXIST;
- goto out;
- }
if (!info->attrs[L2TP_ATTR_PEER_SESSION_ID]) {
ret = -EINVAL;
diff --git a/net/mac80211/agg-rx.c b/net/mac80211/agg-rx.c
index 4456559..1b7a4da 100644
--- a/net/mac80211/agg-rx.c
+++ b/net/mac80211/agg-rx.c
@@ -357,14 +357,14 @@
spin_lock_init(&tid_agg_rx->reorder_lock);
/* rx timer */
- tid_agg_rx->session_timer.function = sta_rx_agg_session_timer_expired;
- tid_agg_rx->session_timer.data = (unsigned long)&sta->timer_to_tid[tid];
- init_timer_deferrable(&tid_agg_rx->session_timer);
+ setup_deferrable_timer(&tid_agg_rx->session_timer,
+ sta_rx_agg_session_timer_expired,
+ (unsigned long)&sta->timer_to_tid[tid]);
/* rx reorder timer */
- tid_agg_rx->reorder_timer.function = sta_rx_agg_reorder_timer_expired;
- tid_agg_rx->reorder_timer.data = (unsigned long)&sta->timer_to_tid[tid];
- init_timer(&tid_agg_rx->reorder_timer);
+ setup_timer(&tid_agg_rx->reorder_timer,
+ sta_rx_agg_reorder_timer_expired,
+ (unsigned long)&sta->timer_to_tid[tid]);
/* prepare reordering buffer */
tid_agg_rx->reorder_buf =
diff --git a/net/mac80211/agg-tx.c b/net/mac80211/agg-tx.c
index 45319cc..60e2a62 100644
--- a/net/mac80211/agg-tx.c
+++ b/net/mac80211/agg-tx.c
@@ -670,14 +670,14 @@
tid_tx->timeout = timeout;
/* response timer */
- tid_tx->addba_resp_timer.function = sta_addba_resp_timer_expired;
- tid_tx->addba_resp_timer.data = (unsigned long)&sta->timer_to_tid[tid];
- init_timer(&tid_tx->addba_resp_timer);
+ setup_timer(&tid_tx->addba_resp_timer,
+ sta_addba_resp_timer_expired,
+ (unsigned long)&sta->timer_to_tid[tid]);
/* tx timer */
- tid_tx->session_timer.function = sta_tx_agg_session_timer_expired;
- tid_tx->session_timer.data = (unsigned long)&sta->timer_to_tid[tid];
- init_timer_deferrable(&tid_tx->session_timer);
+ setup_deferrable_timer(&tid_tx->session_timer,
+ sta_tx_agg_session_timer_expired,
+ (unsigned long)&sta->timer_to_tid[tid]);
/* assign a dialog token */
sta->ampdu_mlme.dialog_token_allocator++;
diff --git a/net/mac80211/cfg.c b/net/mac80211/cfg.c
index ac879bb..6c2e606 100644
--- a/net/mac80211/cfg.c
+++ b/net/mac80211/cfg.c
@@ -3,7 +3,7 @@
*
* Copyright 2006-2010 Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2015 Intel Mobile Communications GmbH
- * Copyright (C) 2015-2016 Intel Deutschland GmbH
+ * Copyright (C) 2015-2017 Intel Deutschland GmbH
*
* This file is GPLv2 as found in COPYING.
*/
@@ -22,11 +22,98 @@
#include "mesh.h"
#include "wme.h"
+static void ieee80211_set_mu_mimo_follow(struct ieee80211_sub_if_data *sdata,
+ struct vif_params *params)
+{
+ bool mu_mimo_groups = false;
+ bool mu_mimo_follow = false;
+
+ if (params->vht_mumimo_groups) {
+ u64 membership;
+
+ BUILD_BUG_ON(sizeof(membership) != WLAN_MEMBERSHIP_LEN);
+
+ memcpy(sdata->vif.bss_conf.mu_group.membership,
+ params->vht_mumimo_groups, WLAN_MEMBERSHIP_LEN);
+ memcpy(sdata->vif.bss_conf.mu_group.position,
+ params->vht_mumimo_groups + WLAN_MEMBERSHIP_LEN,
+ WLAN_USER_POSITION_LEN);
+ ieee80211_bss_info_change_notify(sdata, BSS_CHANGED_MU_GROUPS);
+ /* don't care about endianness - just check for 0 */
+ memcpy(&membership, params->vht_mumimo_groups,
+ WLAN_MEMBERSHIP_LEN);
+ mu_mimo_groups = membership != 0;
+ }
+
+ if (params->vht_mumimo_follow_addr) {
+ mu_mimo_follow =
+ is_valid_ether_addr(params->vht_mumimo_follow_addr);
+ ether_addr_copy(sdata->u.mntr.mu_follow_addr,
+ params->vht_mumimo_follow_addr);
+ }
+
+ sdata->vif.mu_mimo_owner = mu_mimo_groups || mu_mimo_follow;
+}
+
+static int ieee80211_set_mon_options(struct ieee80211_sub_if_data *sdata,
+ struct vif_params *params)
+{
+ struct ieee80211_local *local = sdata->local;
+ struct ieee80211_sub_if_data *monitor_sdata;
+
+ /* check flags first */
+ if (params->flags && ieee80211_sdata_running(sdata)) {
+ u32 mask = MONITOR_FLAG_COOK_FRAMES | MONITOR_FLAG_ACTIVE;
+
+ /*
+ * Prohibit MONITOR_FLAG_COOK_FRAMES and
+ * MONITOR_FLAG_ACTIVE to be changed while the
+ * interface is up.
+ * Else we would need to add a lot of cruft
+ * to update everything:
+ * cooked_mntrs, monitor and all fif_* counters
+ * reconfigure hardware
+ */
+ if ((params->flags & mask) != (sdata->u.mntr.flags & mask))
+ return -EBUSY;
+ }
+
+ /* also validate MU-MIMO change */
+ monitor_sdata = rtnl_dereference(local->monitor_sdata);
+
+ if (!monitor_sdata &&
+ (params->vht_mumimo_groups || params->vht_mumimo_follow_addr))
+ return -EOPNOTSUPP;
+
+ /* apply all changes now - no failures allowed */
+
+ if (monitor_sdata)
+ ieee80211_set_mu_mimo_follow(monitor_sdata, params);
+
+ if (params->flags) {
+ if (ieee80211_sdata_running(sdata)) {
+ ieee80211_adjust_monitor_flags(sdata, -1);
+ sdata->u.mntr.flags = params->flags;
+ ieee80211_adjust_monitor_flags(sdata, 1);
+
+ ieee80211_configure_filter(local);
+ } else {
+ /*
+ * Because the interface is down, ieee80211_do_stop
+ * and ieee80211_do_open take care of "everything"
+ * mentioned in the comment above.
+ */
+ sdata->u.mntr.flags = params->flags;
+ }
+ }
+
+ return 0;
+}
+
static struct wireless_dev *ieee80211_add_iface(struct wiphy *wiphy,
const char *name,
unsigned char name_assign_type,
enum nl80211_iftype type,
- u32 *flags,
struct vif_params *params)
{
struct ieee80211_local *local = wiphy_priv(wiphy);
@@ -38,9 +125,14 @@
if (err)
return ERR_PTR(err);
- if (type == NL80211_IFTYPE_MONITOR && flags) {
- sdata = IEEE80211_WDEV_TO_SUB_IF(wdev);
- sdata->u.mntr.flags = *flags;
+ sdata = IEEE80211_WDEV_TO_SUB_IF(wdev);
+
+ if (type == NL80211_IFTYPE_MONITOR) {
+ err = ieee80211_set_mon_options(sdata, params);
+ if (err) {
+ ieee80211_if_remove(sdata);
+ return NULL;
+ }
}
return wdev;
@@ -55,7 +147,7 @@
static int ieee80211_change_iface(struct wiphy *wiphy,
struct net_device *dev,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct ieee80211_sub_if_data *sdata = IEEE80211_DEV_TO_SUB_IF(dev);
@@ -75,58 +167,9 @@
}
if (sdata->vif.type == NL80211_IFTYPE_MONITOR) {
- struct ieee80211_local *local = sdata->local;
- struct ieee80211_sub_if_data *monitor_sdata;
- u32 mu_mntr_cap_flag = NL80211_EXT_FEATURE_MU_MIMO_AIR_SNIFFER;
-
- monitor_sdata = rtnl_dereference(local->monitor_sdata);
- if (monitor_sdata &&
- wiphy_ext_feature_isset(wiphy, mu_mntr_cap_flag)) {
- memcpy(monitor_sdata->vif.bss_conf.mu_group.membership,
- params->vht_mumimo_groups, WLAN_MEMBERSHIP_LEN);
- memcpy(monitor_sdata->vif.bss_conf.mu_group.position,
- params->vht_mumimo_groups + WLAN_MEMBERSHIP_LEN,
- WLAN_USER_POSITION_LEN);
- monitor_sdata->vif.mu_mimo_owner = true;
- ieee80211_bss_info_change_notify(monitor_sdata,
- BSS_CHANGED_MU_GROUPS);
-
- ether_addr_copy(monitor_sdata->u.mntr.mu_follow_addr,
- params->macaddr);
- }
-
- if (!flags)
- return 0;
-
- if (ieee80211_sdata_running(sdata)) {
- u32 mask = MONITOR_FLAG_COOK_FRAMES |
- MONITOR_FLAG_ACTIVE;
-
- /*
- * Prohibit MONITOR_FLAG_COOK_FRAMES and
- * MONITOR_FLAG_ACTIVE to be changed while the
- * interface is up.
- * Else we would need to add a lot of cruft
- * to update everything:
- * cooked_mntrs, monitor and all fif_* counters
- * reconfigure hardware
- */
- if ((*flags & mask) != (sdata->u.mntr.flags & mask))
- return -EBUSY;
-
- ieee80211_adjust_monitor_flags(sdata, -1);
- sdata->u.mntr.flags = *flags;
- ieee80211_adjust_monitor_flags(sdata, 1);
-
- ieee80211_configure_filter(local);
- } else {
- /*
- * Because the interface is down, ieee80211_do_stop
- * and ieee80211_do_open take care of "everything"
- * mentioned in the comment above.
- */
- sdata->u.mntr.flags = *flags;
- }
+ ret = ieee80211_set_mon_options(sdata, params);
+ if (ret)
+ return ret;
}
return 0;
@@ -617,10 +660,11 @@
int shift = ieee80211_vif_get_shift(&sta->sdata->vif);
u16 brate;
- sband = sta->local->hw.wiphy->bands[
- ieee80211_get_sdata_band(sta->sdata)];
- brate = sband->bitrates[rate->idx].bitrate;
- rinfo->legacy = DIV_ROUND_UP(brate, 1 << shift);
+ sband = ieee80211_get_sband(sta->sdata);
+ if (sband) {
+ brate = sband->bitrates[rate->idx].bitrate;
+ rinfo->legacy = DIV_ROUND_UP(brate, 1 << shift);
+ }
}
if (rate->flags & IEEE80211_TX_RC_40_MHZ_WIDTH)
rinfo->bw = RATE_INFO_BW_40;
@@ -696,11 +740,8 @@
return 0;
mutex_lock(&local->mtx);
- mutex_lock(&local->iflist_mtx);
if (local->use_chanctx) {
- sdata = rcu_dereference_protected(
- local->monitor_sdata,
- lockdep_is_held(&local->iflist_mtx));
+ sdata = rtnl_dereference(local->monitor_sdata);
if (sdata) {
ieee80211_vif_release_channel(sdata);
ret = ieee80211_vif_use_channel(sdata, chandef,
@@ -713,7 +754,6 @@
if (ret == 0)
local->monitor_chandef = *chandef;
- mutex_unlock(&local->iflist_mtx);
mutex_unlock(&local->mtx);
return ret;
@@ -1214,10 +1254,11 @@
int ret = 0;
struct ieee80211_supported_band *sband;
struct ieee80211_sub_if_data *sdata = sta->sdata;
- enum nl80211_band band = ieee80211_get_sdata_band(sdata);
u32 mask, set;
- sband = local->hw.wiphy->bands[band];
+ sband = ieee80211_get_sband(sdata);
+ if (!sband)
+ return -EINVAL;
mask = params->sta_flags_mask;
set = params->sta_flags_set;
@@ -1350,7 +1391,7 @@
ieee80211_parse_bitrates(&sdata->vif.bss_conf.chandef,
sband, params->supported_rates,
params->supported_rates_len,
- &sta->sta.supp_rates[band]);
+ &sta->sta.supp_rates[sband->band]);
}
if (params->ht_capa)
@@ -1366,8 +1407,8 @@
/* returned value is only needed for rc update, but the
* rc isn't initialized here yet, so ignore it
*/
- __ieee80211_vht_handle_opmode(sdata, sta,
- params->opmode_notif, band);
+ __ieee80211_vht_handle_opmode(sdata, sta, params->opmode_notif,
+ sband->band);
}
if (params->support_p2p_ps >= 0)
@@ -2005,13 +2046,15 @@
struct bss_parameters *params)
{
struct ieee80211_sub_if_data *sdata = IEEE80211_DEV_TO_SUB_IF(dev);
- enum nl80211_band band;
+ struct ieee80211_supported_band *sband;
u32 changed = 0;
if (!sdata_dereference(sdata->u.ap.beacon, sdata))
return -ENOENT;
- band = ieee80211_get_sdata_band(sdata);
+ sband = ieee80211_get_sband(sdata);
+ if (!sband)
+ return -EINVAL;
if (params->use_cts_prot >= 0) {
sdata->vif.bss_conf.use_cts_prot = params->use_cts_prot;
@@ -2024,7 +2067,7 @@
}
if (!sdata->vif.bss_conf.use_short_slot &&
- band == NL80211_BAND_5GHZ) {
+ sband->band == NL80211_BAND_5GHZ) {
sdata->vif.bss_conf.use_short_slot = true;
changed |= BSS_CHANGED_ERP_SLOT;
}
@@ -2037,11 +2080,12 @@
if (params->basic_rates) {
ieee80211_parse_bitrates(&sdata->vif.bss_conf.chandef,
- wiphy->bands[band],
+ wiphy->bands[sband->band],
params->basic_rates,
params->basic_rates_len,
&sdata->vif.bss_conf.basic_rates);
changed |= BSS_CHANGED_BASIC_RATES;
+ ieee80211_check_rate_mask(sdata);
}
if (params->ap_isolate >= 0) {
@@ -2198,7 +2242,8 @@
}
static int
-ieee80211_sched_scan_stop(struct wiphy *wiphy, struct net_device *dev)
+ieee80211_sched_scan_stop(struct wiphy *wiphy, struct net_device *dev,
+ u64 reqid)
{
struct ieee80211_local *local = wiphy_priv(wiphy);
@@ -2630,6 +2675,33 @@
bss_conf->cqm_rssi_thold = rssi_thold;
bss_conf->cqm_rssi_hyst = rssi_hyst;
+ bss_conf->cqm_rssi_low = 0;
+ bss_conf->cqm_rssi_high = 0;
+ sdata->u.mgd.last_cqm_event_signal = 0;
+
+ /* tell the driver upon association, unless already associated */
+ if (sdata->u.mgd.associated &&
+ sdata->vif.driver_flags & IEEE80211_VIF_SUPPORTS_CQM_RSSI)
+ ieee80211_bss_info_change_notify(sdata, BSS_CHANGED_CQM);
+
+ return 0;
+}
+
+static int ieee80211_set_cqm_rssi_range_config(struct wiphy *wiphy,
+ struct net_device *dev,
+ s32 rssi_low, s32 rssi_high)
+{
+ struct ieee80211_sub_if_data *sdata = IEEE80211_DEV_TO_SUB_IF(dev);
+ struct ieee80211_vif *vif = &sdata->vif;
+ struct ieee80211_bss_conf *bss_conf = &vif->bss_conf;
+
+ if (sdata->vif.driver_flags & IEEE80211_VIF_BEACON_FILTER)
+ return -EOPNOTSUPP;
+
+ bss_conf->cqm_rssi_low = rssi_low;
+ bss_conf->cqm_rssi_high = rssi_high;
+ bss_conf->cqm_rssi_thold = 0;
+ bss_conf->cqm_rssi_hyst = 0;
sdata->u.mgd.last_cqm_event_signal = 0;
/* tell the driver upon association, unless already associated */
@@ -2658,6 +2730,21 @@
return ret;
}
+ /*
+ * If active validate the setting and reject it if it doesn't leave
+ * at least one basic rate usable, since we really have to be able
+ * to send something, and if we're an AP we have to be able to do
+ * so at a basic rate so that all clients can receive it.
+ */
+ if (rcu_access_pointer(sdata->vif.chanctx_conf) &&
+ sdata->vif.bss_conf.chandef.chan) {
+ u32 basic_rates = sdata->vif.bss_conf.basic_rates;
+ enum nl80211_band band = sdata->vif.bss_conf.chandef.chan->band;
+
+ if (!(mask->control[band].legacy & basic_rates))
+ return -EINVAL;
+ }
+
for (i = 0; i < NUM_NL80211_BANDS; i++) {
struct ieee80211_supported_band *sband = wiphy->bands[i];
int j;
@@ -3639,6 +3726,7 @@
.mgmt_tx = ieee80211_mgmt_tx,
.mgmt_tx_cancel_wait = ieee80211_mgmt_tx_cancel_wait,
.set_cqm_rssi_config = ieee80211_set_cqm_rssi_config,
+ .set_cqm_rssi_range_config = ieee80211_set_cqm_rssi_range_config,
.mgmt_frame_register = ieee80211_mgmt_frame_register,
.set_antenna = ieee80211_set_antenna,
.get_antenna = ieee80211_get_antenna,
diff --git a/net/mac80211/ibss.c b/net/mac80211/ibss.c
index 98999d3..6db09fa 100644
--- a/net/mac80211/ibss.c
+++ b/net/mac80211/ibss.c
@@ -425,7 +425,7 @@
case NL80211_CHAN_WIDTH_5:
case NL80211_CHAN_WIDTH_10:
cfg80211_chandef_create(&chandef, cbss->channel,
- NL80211_CHAN_WIDTH_20_NOHT);
+ NL80211_CHAN_NO_HT);
chandef.width = sdata->u.ibss.chandef.width;
break;
case NL80211_CHAN_WIDTH_80:
@@ -437,7 +437,7 @@
default:
/* fall back to 20 MHz for unsupported modes */
cfg80211_chandef_create(&chandef, cbss->channel,
- NL80211_CHAN_WIDTH_20_NOHT);
+ NL80211_CHAN_NO_HT);
break;
}
@@ -992,7 +992,7 @@
enum nl80211_band band = rx_status->band;
enum nl80211_bss_scan_width scan_width;
struct ieee80211_local *local = sdata->local;
- struct ieee80211_supported_band *sband = local->hw.wiphy->bands[band];
+ struct ieee80211_supported_band *sband;
bool rates_updated = false;
u32 supp_rates = 0;
@@ -1002,6 +1002,10 @@
if (!ether_addr_equal(mgmt->bssid, sdata->u.ibss.bssid))
return;
+ sband = local->hw.wiphy->bands[band];
+ if (WARN_ON(!sband))
+ return;
+
rcu_read_lock();
sta = sta_info_get(sdata, mgmt->sa);
@@ -1014,9 +1018,9 @@
prev_rates = sta->sta.supp_rates[band];
/* make sure mandatory rates are always added */
scan_width = NL80211_BSS_CHAN_WIDTH_20;
- if (rx_status->flag & RX_FLAG_5MHZ)
+ if (rx_status->bw == RATE_INFO_BW_5)
scan_width = NL80211_BSS_CHAN_WIDTH_5;
- if (rx_status->flag & RX_FLAG_10MHZ)
+ else if (rx_status->bw == RATE_INFO_BW_10)
scan_width = NL80211_BSS_CHAN_WIDTH_10;
sta->sta.supp_rates[band] = supp_rates |
diff --git a/net/mac80211/ieee80211_i.h b/net/mac80211/ieee80211_i.h
index 0e71843..f8f6c14 100644
--- a/net/mac80211/ieee80211_i.h
+++ b/net/mac80211/ieee80211_i.h
@@ -839,6 +839,8 @@
struct ieee80211_if_mntr {
u32 flags;
u8 mu_follow_addr[ETH_ALEN] __aligned(2);
+
+ struct list_head list;
};
/**
@@ -999,21 +1001,6 @@
lockdep_assert_held(&sdata->wdev.mtx);
}
-static inline enum nl80211_band
-ieee80211_get_sdata_band(struct ieee80211_sub_if_data *sdata)
-{
- enum nl80211_band band = NL80211_BAND_2GHZ;
- struct ieee80211_chanctx_conf *chanctx_conf;
-
- rcu_read_lock();
- chanctx_conf = rcu_dereference(sdata->vif.chanctx_conf);
- if (!WARN_ON(!chanctx_conf))
- band = chanctx_conf->def.chan->band;
- rcu_read_unlock();
-
- return band;
-}
-
static inline int
ieee80211_chandef_get_shift(struct cfg80211_chan_def *chandef)
{
@@ -1259,6 +1246,7 @@
/* see iface.c */
struct list_head interfaces;
+ struct list_head mon_list; /* only that are IFF_UP && !cooked */
struct mutex iflist_mtx;
/*
@@ -1418,6 +1406,27 @@
return container_of(wdev, struct ieee80211_sub_if_data, wdev);
}
+static inline struct ieee80211_supported_band *
+ieee80211_get_sband(struct ieee80211_sub_if_data *sdata)
+{
+ struct ieee80211_local *local = sdata->local;
+ struct ieee80211_chanctx_conf *chanctx_conf;
+ enum nl80211_band band;
+
+ rcu_read_lock();
+ chanctx_conf = rcu_dereference(sdata->vif.chanctx_conf);
+
+ if (WARN_ON(!chanctx_conf)) {
+ rcu_read_unlock();
+ return NULL;
+ }
+
+ band = chanctx_conf->def.chan->band;
+ rcu_read_unlock();
+
+ return local->hw.wiphy->bands[band];
+}
+
/* this struct represents 802.11n's RA/TID combination */
struct ieee80211_ra_tid {
u8 ra[ETH_ALEN];
@@ -1474,6 +1483,7 @@
const u8 *opmode_notif;
const struct ieee80211_sec_chan_offs_ie *sec_chan_offs;
const struct ieee80211_mesh_chansw_params_ie *mesh_chansw_params_ie;
+ const struct ieee80211_bss_max_idle_period_ie *max_idle_period_ie;
/* length of them, respectively */
u8 ext_capab_len;
@@ -1527,9 +1537,9 @@
status->flag & RX_FLAG_MACTIME_END);
if (status->flag & (RX_FLAG_MACTIME_START | RX_FLAG_MACTIME_END))
return true;
- /* can't handle HT/VHT preamble yet */
+ /* can't handle non-legacy preamble yet */
if (status->flag & RX_FLAG_MACTIME_PLCP_START &&
- !(status->flag & (RX_FLAG_HT | RX_FLAG_VHT)))
+ status->encoding != RX_ENC_LEGACY)
return true;
return false;
}
diff --git a/net/mac80211/iface.c b/net/mac80211/iface.c
index 5bb0c50..3bd5b81 100644
--- a/net/mac80211/iface.c
+++ b/net/mac80211/iface.c
@@ -676,7 +676,8 @@
set_bit(SDATA_STATE_RUNNING, &sdata->state);
- if (sdata->vif.type == NL80211_IFTYPE_WDS) {
+ switch (sdata->vif.type) {
+ case NL80211_IFTYPE_WDS:
/* Create STA entry for the WDS peer */
sta = sta_info_alloc(sdata, sdata->u.wds.remote_addr,
GFP_KERNEL);
@@ -697,8 +698,17 @@
rate_control_rate_init(sta);
netif_carrier_on(dev);
- } else if (sdata->vif.type == NL80211_IFTYPE_P2P_DEVICE) {
+ break;
+ case NL80211_IFTYPE_P2P_DEVICE:
rcu_assign_pointer(local->p2p_sdata, sdata);
+ break;
+ case NL80211_IFTYPE_MONITOR:
+ if (sdata->u.mntr.flags & MONITOR_FLAG_COOK_FRAMES)
+ break;
+ list_add_tail_rcu(&sdata->u.mntr.list, &local->mon_list);
+ break;
+ default:
+ break;
}
/*
@@ -817,6 +827,11 @@
case NL80211_IFTYPE_AP:
cancel_work_sync(&sdata->u.ap.request_smps_work);
break;
+ case NL80211_IFTYPE_MONITOR:
+ if (sdata->u.mntr.flags & MONITOR_FLAG_COOK_FRAMES)
+ break;
+ list_del_rcu(&sdata->u.mntr.list);
+ break;
default:
break;
}
diff --git a/net/mac80211/main.c b/net/mac80211/main.c
index 56fb479..8aa1f5b 100644
--- a/net/mac80211/main.c
+++ b/net/mac80211/main.c
@@ -253,6 +253,7 @@
WARN(test_bit(SCAN_HW_SCANNING, &local->scanning),
"%s called with hardware scan in progress\n", __func__);
+ flush_work(&local->radar_detected_work);
rtnl_lock();
list_for_each_entry(sdata, &local->interfaces, list)
flush_delayed_work(&sdata->dec_tailroom_needed_wk);
@@ -603,6 +604,7 @@
ARRAY_SIZE(local->ext_capa);
INIT_LIST_HEAD(&local->interfaces);
+ INIT_LIST_HEAD(&local->mon_list);
__hw_addr_init(&local->mc_list);
@@ -1186,6 +1188,7 @@
cancel_work_sync(&local->reconfig_filter);
cancel_work_sync(&local->tdls_chsw_work);
flush_work(&local->sched_scan_stopped_work);
+ flush_work(&local->radar_detected_work);
ieee80211_clear_tx_pending(local);
rate_control_deinitialize(local);
diff --git a/net/mac80211/mesh.c b/net/mac80211/mesh.c
index 6e7b6a0..737e1f0 100644
--- a/net/mac80211/mesh.c
+++ b/net/mac80211/mesh.c
@@ -63,6 +63,7 @@
struct ieee80211_if_mesh *ifmsh = &sdata->u.mesh;
u32 basic_rates = 0;
struct cfg80211_chan_def sta_chan_def;
+ struct ieee80211_supported_band *sband;
/*
* As support for each feature is added, check for matching
@@ -83,7 +84,11 @@
(ifmsh->mesh_auth_id == ie->mesh_config->meshconf_auth)))
return false;
- ieee80211_sta_get_rates(sdata, ie, ieee80211_get_sdata_band(sdata),
+ sband = ieee80211_get_sband(sdata);
+ if (!sband)
+ return false;
+
+ ieee80211_sta_get_rates(sdata, ie, sband->band,
&basic_rates);
if (sdata->vif.bss_conf.basic_rates != basic_rates)
@@ -399,12 +404,13 @@
int mesh_add_ht_cap_ie(struct ieee80211_sub_if_data *sdata,
struct sk_buff *skb)
{
- struct ieee80211_local *local = sdata->local;
- enum nl80211_band band = ieee80211_get_sdata_band(sdata);
struct ieee80211_supported_band *sband;
u8 *pos;
- sband = local->hw.wiphy->bands[band];
+ sband = ieee80211_get_sband(sdata);
+ if (!sband)
+ return -EINVAL;
+
if (!sband->ht_cap.ht_supported ||
sdata->vif.bss_conf.chandef.width == NL80211_CHAN_WIDTH_20_NOHT ||
sdata->vif.bss_conf.chandef.width == NL80211_CHAN_WIDTH_5 ||
@@ -462,12 +468,13 @@
int mesh_add_vht_cap_ie(struct ieee80211_sub_if_data *sdata,
struct sk_buff *skb)
{
- struct ieee80211_local *local = sdata->local;
- enum nl80211_band band = ieee80211_get_sdata_band(sdata);
struct ieee80211_supported_band *sband;
u8 *pos;
- sband = local->hw.wiphy->bands[band];
+ sband = ieee80211_get_sband(sdata);
+ if (!sband)
+ return -EINVAL;
+
if (!sband->vht_cap.vht_supported ||
sdata->vif.bss_conf.chandef.width == NL80211_CHAN_WIDTH_20_NOHT ||
sdata->vif.bss_conf.chandef.width == NL80211_CHAN_WIDTH_5 ||
@@ -916,12 +923,16 @@
struct cfg80211_csa_settings params;
struct ieee80211_csa_ie csa_ie;
struct ieee80211_if_mesh *ifmsh = &sdata->u.mesh;
- enum nl80211_band band = ieee80211_get_sdata_band(sdata);
+ struct ieee80211_supported_band *sband;
int err;
u32 sta_flags;
sdata_assert_lock(sdata);
+ sband = ieee80211_get_sband(sdata);
+ if (!sband)
+ return false;
+
sta_flags = IEEE80211_STA_DISABLE_VHT;
switch (sdata->vif.bss_conf.chandef.width) {
case NL80211_CHAN_WIDTH_20_NOHT:
@@ -935,7 +946,7 @@
memset(¶ms, 0, sizeof(params));
memset(&csa_ie, 0, sizeof(csa_ie));
- err = ieee80211_parse_ch_switch_ie(sdata, elems, band,
+ err = ieee80211_parse_ch_switch_ie(sdata, elems, sband->band,
sta_flags, sdata->vif.addr,
&csa_ie);
if (err < 0)
@@ -1100,8 +1111,14 @@
if (!channel || channel->flags & IEEE80211_CHAN_DISABLED)
return;
- if (mesh_matches_local(sdata, &elems))
- mesh_neighbour_update(sdata, mgmt->sa, &elems);
+ if (mesh_matches_local(sdata, &elems)) {
+ mpl_dbg(sdata, "rssi_threshold=%d,rx_status->signal=%d\n",
+ sdata->u.mesh.mshcfg.rssi_threshold, rx_status->signal);
+ if (!sdata->u.mesh.user_mpm ||
+ sdata->u.mesh.mshcfg.rssi_threshold == 0 ||
+ sdata->u.mesh.mshcfg.rssi_threshold < rx_status->signal)
+ mesh_neighbour_update(sdata, mgmt->sa, &elems);
+ }
if (ifmsh->sync_ops)
ifmsh->sync_ops->rx_bcn_presp(sdata,
diff --git a/net/mac80211/mesh_hwmp.c b/net/mac80211/mesh_hwmp.c
index b747c96..4005edd 100644
--- a/net/mac80211/mesh_hwmp.c
+++ b/net/mac80211/mesh_hwmp.c
@@ -16,6 +16,7 @@
#define TEST_FRAME_LEN 8192
#define MAX_METRIC 0xffffffff
#define ARITH_SHIFT 8
+#define LINK_FAIL_THRESH 95
#define MAX_PREQ_QUEUE_LEN 64
@@ -307,10 +308,12 @@
failed = !(txinfo->flags & IEEE80211_TX_STAT_ACK);
- /* moving average, scaled to 100 */
- sta->mesh->fail_avg =
- ((80 * sta->mesh->fail_avg + 5) / 100 + 20 * failed);
- if (sta->mesh->fail_avg > 95)
+ /* moving average, scaled to 100.
+ * feed failure as 100 and success as 0
+ */
+ ewma_mesh_fail_avg_add(&sta->mesh->fail_avg, failed * 100);
+ if (ewma_mesh_fail_avg_read(&sta->mesh->fail_avg) >
+ LINK_FAIL_THRESH)
mesh_plink_broken(sta);
}
@@ -325,6 +328,8 @@
int rate, err;
u32 tx_time, estimated_retx;
u64 result;
+ unsigned long fail_avg =
+ ewma_mesh_fail_avg_read(&sta->mesh->fail_avg);
/* Try to get rate based on HW/SW RC algorithm.
* Rate is returned in units of Kbps, correct this
@@ -336,7 +341,7 @@
if (rate) {
err = 0;
} else {
- if (sta->mesh->fail_avg >= 100)
+ if (fail_avg > LINK_FAIL_THRESH)
return MAX_METRIC;
sta_set_rate_info_tx(sta, &sta->tx_stats.last_rate, &rinfo);
@@ -344,7 +349,7 @@
if (WARN_ON(!rate))
return MAX_METRIC;
- err = (sta->mesh->fail_avg << ARITH_SHIFT) / 100;
+ err = (fail_avg << ARITH_SHIFT) / 100;
}
/* bitrate is in units of 100 Kbps, while we need rate in units of
@@ -484,6 +489,9 @@
? mpath->exp_time : exp_time;
mesh_path_activate(mpath);
spin_unlock_bh(&mpath->state_lock);
+ ewma_mesh_fail_avg_init(&sta->mesh->fail_avg);
+ /* init it at a low value - 0 start is tricky */
+ ewma_mesh_fail_avg_add(&sta->mesh->fail_avg, 1);
mesh_path_tx_pending(mpath);
/* draft says preq_id should be saved to, but there does
* not seem to be any use for it, skipping by now
@@ -522,6 +530,9 @@
? mpath->exp_time : exp_time;
mesh_path_activate(mpath);
spin_unlock_bh(&mpath->state_lock);
+ ewma_mesh_fail_avg_init(&sta->mesh->fail_avg);
+ /* init it at a low value - 0 start is tricky */
+ ewma_mesh_fail_avg_add(&sta->mesh->fail_avg, 1);
mesh_path_tx_pending(mpath);
} else
spin_unlock_bh(&mpath->state_lock);
diff --git a/net/mac80211/mesh_pathtbl.c b/net/mac80211/mesh_pathtbl.c
index f0e6175..97269ca 100644
--- a/net/mac80211/mesh_pathtbl.c
+++ b/net/mac80211/mesh_pathtbl.c
@@ -397,11 +397,10 @@
new_mpath->sdata = sdata;
new_mpath->flags = 0;
skb_queue_head_init(&new_mpath->frame_queue);
- new_mpath->timer.data = (unsigned long) new_mpath;
- new_mpath->timer.function = mesh_path_timer;
new_mpath->exp_time = jiffies;
spin_lock_init(&new_mpath->state_lock);
- init_timer(&new_mpath->timer);
+ setup_timer(&new_mpath->timer, mesh_path_timer,
+ (unsigned long) new_mpath);
return new_mpath;
}
@@ -829,6 +828,9 @@
mpath->flags = MESH_PATH_FIXED | MESH_PATH_SN_VALID;
mesh_path_activate(mpath);
spin_unlock_bh(&mpath->state_lock);
+ ewma_mesh_fail_avg_init(&next_hop->mesh->fail_avg);
+ /* init it at a low value - 0 start is tricky */
+ ewma_mesh_fail_avg_add(&next_hop->mesh->fail_avg, 1);
mesh_path_tx_pending(mpath);
}
diff --git a/net/mac80211/mesh_plink.c b/net/mac80211/mesh_plink.c
index 953d71e..1131cd5 100644
--- a/net/mac80211/mesh_plink.c
+++ b/net/mac80211/mesh_plink.c
@@ -95,19 +95,23 @@
static u32 mesh_set_short_slot_time(struct ieee80211_sub_if_data *sdata)
{
struct ieee80211_local *local = sdata->local;
- enum nl80211_band band = ieee80211_get_sdata_band(sdata);
- struct ieee80211_supported_band *sband = local->hw.wiphy->bands[band];
+ struct ieee80211_supported_band *sband;
struct sta_info *sta;
u32 erp_rates = 0, changed = 0;
int i;
bool short_slot = false;
- if (band == NL80211_BAND_5GHZ) {
+ sband = ieee80211_get_sband(sdata);
+ if (!sband)
+ return changed;
+
+ if (sband->band == NL80211_BAND_5GHZ) {
/* (IEEE 802.11-2012 19.4.5) */
short_slot = true;
goto out;
- } else if (band != NL80211_BAND_2GHZ)
+ } else if (sband->band != NL80211_BAND_2GHZ) {
goto out;
+ }
for (i = 0; i < sband->n_bitrates; i++)
if (sband->bitrates[i].flags & IEEE80211_RATE_ERP_G)
@@ -123,7 +127,7 @@
continue;
short_slot = false;
- if (erp_rates & sta->sta.supp_rates[band])
+ if (erp_rates & sta->sta.supp_rates[sband->band])
short_slot = true;
else
break;
@@ -249,7 +253,15 @@
mgmt->u.action.u.self_prot.action_code = action;
if (action != WLAN_SP_MESH_PEERING_CLOSE) {
- enum nl80211_band band = ieee80211_get_sdata_band(sdata);
+ struct ieee80211_supported_band *sband;
+ enum nl80211_band band;
+
+ sband = ieee80211_get_sband(sdata);
+ if (!sband) {
+ err = -EINVAL;
+ goto free;
+ }
+ band = sband->band;
/* capability info */
pos = skb_put(skb, 2);
@@ -395,13 +407,16 @@
struct ieee802_11_elems *elems, bool insert)
{
struct ieee80211_local *local = sdata->local;
- enum nl80211_band band = ieee80211_get_sdata_band(sdata);
struct ieee80211_supported_band *sband;
u32 rates, basic_rates = 0, changed = 0;
enum ieee80211_sta_rx_bandwidth bw = sta->sta.bandwidth;
- sband = local->hw.wiphy->bands[band];
- rates = ieee80211_sta_get_rates(sdata, elems, band, &basic_rates);
+ sband = ieee80211_get_sband(sdata);
+ if (!sband)
+ return;
+
+ rates = ieee80211_sta_get_rates(sdata, elems, sband->band,
+ &basic_rates);
spin_lock_bh(&sta->mesh->plink_lock);
sta->rx_stats.last_rx = jiffies;
@@ -412,9 +427,9 @@
goto out;
sta->mesh->processed_beacon = true;
- if (sta->sta.supp_rates[band] != rates)
+ if (sta->sta.supp_rates[sband->band] != rates)
changed |= IEEE80211_RC_SUPP_RATES_CHANGED;
- sta->sta.supp_rates[band] = rates;
+ sta->sta.supp_rates[sband->band] = rates;
if (ieee80211_ht_cap_ie_to_sta_ht_cap(sdata, sband,
elems->ht_cap_elem, sta))
diff --git a/net/mac80211/mlme.c b/net/mac80211/mlme.c
index 6e90301..89dff56 100644
--- a/net/mac80211/mlme.c
+++ b/net/mac80211/mlme.c
@@ -6,7 +6,7 @@
* Copyright 2006-2007 Jiri Benc <jbenc@suse.cz>
* Copyright 2007, Michael Wu <flamingice@sourmilk.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
- * Copyright (C) 2015 - 2016 Intel Deutschland GmbH
+ * Copyright (C) 2015 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -1855,11 +1855,16 @@
u16 capab, bool erp_valid, u8 erp)
{
struct ieee80211_bss_conf *bss_conf = &sdata->vif.bss_conf;
+ struct ieee80211_supported_band *sband;
u32 changed = 0;
bool use_protection;
bool use_short_preamble;
bool use_short_slot;
+ sband = ieee80211_get_sband(sdata);
+ if (!sband)
+ return changed;
+
if (erp_valid) {
use_protection = (erp & WLAN_ERP_USE_PROTECTION) != 0;
use_short_preamble = (erp & WLAN_ERP_BARKER_PREAMBLE) == 0;
@@ -1869,7 +1874,7 @@
}
use_short_slot = !!(capab & WLAN_CAPABILITY_SHORT_SLOT_TIME);
- if (ieee80211_get_sdata_band(sdata) == NL80211_BAND_5GHZ)
+ if (sband->band == NL80211_BAND_5GHZ)
use_short_slot = true;
if (use_protection != bss_conf->use_cts_prot) {
@@ -1908,6 +1913,8 @@
sdata->u.mgd.associated = cbss;
memcpy(sdata->u.mgd.bssid, cbss->bssid, ETH_ALEN);
+ ieee80211_check_rate_mask(sdata);
+
sdata->u.mgd.flags |= IEEE80211_STA_RESET_SIGNAL_AVE;
if (sdata->vif.p2p ||
@@ -2797,8 +2804,9 @@
reason_code = le16_to_cpu(mgmt->u.disassoc.reason_code);
- sdata_info(sdata, "disassociated from %pM (Reason: %u)\n",
- mgmt->sa, reason_code);
+ sdata_info(sdata, "disassociated from %pM (Reason: %u=%s)\n",
+ mgmt->sa, reason_code,
+ ieee80211_get_reason_code_string(reason_code));
ieee80211_set_disassoc(sdata, 0, 0, false, NULL);
@@ -2822,15 +2830,15 @@
*have_higher_than_11mbit = true;
/*
- * BSS_MEMBERSHIP_SELECTOR_HT_PHY is defined in 802.11n-2009
- * 7.3.2.2 as a magic value instead of a rate. Hence, skip it.
+ * Skip HT and VHT BSS membership selectors since they're not
+ * rates.
*
- * Note: Even through the membership selector and the basic
+ * Note: Even though the membership selector and the basic
* rate flag share the same bit, they are not exactly
* the same.
*/
- if (!!(supp_rates[i] & 0x80) &&
- (supp_rates[i] & 0x7f) == BSS_MEMBERSHIP_SELECTOR_HT_PHY)
+ if (supp_rates[i] == (0x80 | BSS_MEMBERSHIP_SELECTOR_HT_PHY) ||
+ supp_rates[i] == (0x80 | BSS_MEMBERSHIP_SELECTOR_VHT_PHY))
continue;
for (j = 0; j < sband->n_bitrates; j++) {
@@ -3001,7 +3009,12 @@
goto out;
}
- sband = local->hw.wiphy->bands[ieee80211_get_sdata_band(sdata)];
+ sband = ieee80211_get_sband(sdata);
+ if (!sband) {
+ mutex_unlock(&sdata->local->sta_mtx);
+ ret = false;
+ goto out;
+ }
/* Set up internal HT/VHT capabilities */
if (elems.ht_cap_elem && !(ifmgd->flags & IEEE80211_STA_DISABLE_HT))
@@ -3085,6 +3098,18 @@
}
changed |= BSS_CHANGED_QOS;
+ if (elems.max_idle_period_ie) {
+ bss_conf->max_idle_period =
+ le16_to_cpu(elems.max_idle_period_ie->max_idle_period);
+ bss_conf->protected_keep_alive =
+ !!(elems.max_idle_period_ie->idle_options &
+ WLAN_IDLE_OPTIONS_PROTECTED_KEEP_ALIVE);
+ changed |= BSS_CHANGED_KEEP_ALIVE;
+ } else {
+ bss_conf->max_idle_period = 0;
+ bss_conf->protected_keep_alive = false;
+ }
+
/* set AID and assoc capability,
* ieee80211_set_associated() will tell the driver */
bss_conf->aid = aid;
@@ -3430,6 +3455,30 @@
}
}
+ if (bss_conf->cqm_rssi_low &&
+ ifmgd->count_beacon_signal >= IEEE80211_SIGNAL_AVE_MIN_COUNT) {
+ int sig = -ewma_beacon_signal_read(&ifmgd->ave_beacon_signal);
+ int last_event = ifmgd->last_cqm_event_signal;
+ int low = bss_conf->cqm_rssi_low;
+ int high = bss_conf->cqm_rssi_high;
+
+ if (sig < low &&
+ (last_event == 0 || last_event >= low)) {
+ ifmgd->last_cqm_event_signal = sig;
+ ieee80211_cqm_rssi_notify(
+ &sdata->vif,
+ NL80211_CQM_RSSI_THRESHOLD_EVENT_LOW,
+ sig, GFP_KERNEL);
+ } else if (sig > high &&
+ (last_event == 0 || last_event <= high)) {
+ ifmgd->last_cqm_event_signal = sig;
+ ieee80211_cqm_rssi_notify(
+ &sdata->vif,
+ NL80211_CQM_RSSI_THRESHOLD_EVENT_HIGH,
+ sig, GFP_KERNEL);
+ }
+ }
+
if (ifmgd->flags & IEEE80211_STA_CONNECTION_POLL) {
mlme_dbg_ratelimited(sdata,
"cancelling AP probe due to a received beacon\n");
diff --git a/net/mac80211/pm.c b/net/mac80211/pm.c
index 76a8bcd..a87d195 100644
--- a/net/mac80211/pm.c
+++ b/net/mac80211/pm.c
@@ -10,7 +10,7 @@
{
if (ieee80211_request_sched_scan_stop(local))
return;
- cfg80211_sched_scan_stopped_rtnl(local->hw.wiphy);
+ cfg80211_sched_scan_stopped_rtnl(local->hw.wiphy, 0);
}
int __ieee80211_suspend(struct ieee80211_hw *hw, struct cfg80211_wowlan *wowlan)
diff --git a/net/mac80211/rate.c b/net/mac80211/rate.c
index 206698b..ea1f431 100644
--- a/net/mac80211/rate.c
+++ b/net/mac80211/rate.c
@@ -2,6 +2,7 @@
* Copyright 2002-2005, Instant802 Networks, Inc.
* Copyright 2005-2006, Devicescape Software, Inc.
* Copyright (c) 2006 Jiri Benc <jbenc@suse.cz>
+ * Copyright 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -61,6 +62,28 @@
set_sta_flag(sta, WLAN_STA_RATE_CONTROL);
}
+void rate_control_tx_status(struct ieee80211_local *local,
+ struct ieee80211_supported_band *sband,
+ struct ieee80211_tx_status *st)
+{
+ struct rate_control_ref *ref = local->rate_ctrl;
+ struct sta_info *sta = container_of(st->sta, struct sta_info, sta);
+ void *priv_sta = sta->rate_ctrl_priv;
+
+ if (!ref || !test_sta_flag(sta, WLAN_STA_RATE_CONTROL))
+ return;
+
+ spin_lock_bh(&sta->rate_ctrl_lock);
+ if (ref->ops->tx_status_ext)
+ ref->ops->tx_status_ext(ref->priv, sband, priv_sta, st);
+ else if (st->skb)
+ ref->ops->tx_status(ref->priv, sband, st->sta, priv_sta, st->skb);
+ else
+ WARN_ON_ONCE(1);
+
+ spin_unlock_bh(&sta->rate_ctrl_lock);
+}
+
void rate_control_rate_update(struct ieee80211_local *local,
struct ieee80211_supported_band *sband,
struct sta_info *sta, u32 changed)
@@ -173,9 +196,11 @@
/* try default if specific alg requested but not found */
ops = ieee80211_try_rate_control_ops_get(ieee80211_default_rc_algo);
- /* try built-in one if specific alg requested but not found */
- if (!ops && strlen(CONFIG_MAC80211_RC_DEFAULT))
+ /* Note: check for > 0 is intentional to avoid clang warning */
+ if (!ops && (strlen(CONFIG_MAC80211_RC_DEFAULT) > 0))
+ /* try built-in one if specific alg requested but not found */
ops = ieee80211_try_rate_control_ops_get(CONFIG_MAC80211_RC_DEFAULT);
+
kernel_param_unlock(THIS_MODULE);
return ops;
@@ -208,7 +233,6 @@
ref = kmalloc(sizeof(struct rate_control_ref), GFP_KERNEL);
if (!ref)
return NULL;
- ref->local = local;
ref->ops = ieee80211_rate_control_ops_get(name);
if (!ref->ops)
goto free;
@@ -229,18 +253,45 @@
return NULL;
}
-static void rate_control_free(struct rate_control_ref *ctrl_ref)
+static void rate_control_free(struct ieee80211_local *local,
+ struct rate_control_ref *ctrl_ref)
{
ctrl_ref->ops->free(ctrl_ref->priv);
#ifdef CONFIG_MAC80211_DEBUGFS
- debugfs_remove_recursive(ctrl_ref->local->debugfs.rcdir);
- ctrl_ref->local->debugfs.rcdir = NULL;
+ debugfs_remove_recursive(local->debugfs.rcdir);
+ local->debugfs.rcdir = NULL;
#endif
kfree(ctrl_ref);
}
+void ieee80211_check_rate_mask(struct ieee80211_sub_if_data *sdata)
+{
+ struct ieee80211_local *local = sdata->local;
+ struct ieee80211_supported_band *sband;
+ u32 user_mask, basic_rates = sdata->vif.bss_conf.basic_rates;
+ enum nl80211_band band;
+
+ if (WARN_ON(!sdata->vif.bss_conf.chandef.chan))
+ return;
+
+ if (WARN_ON_ONCE(!basic_rates))
+ return;
+
+ band = sdata->vif.bss_conf.chandef.chan->band;
+ user_mask = sdata->rc_rateidx_mask[band];
+ sband = local->hw.wiphy->bands[band];
+
+ if (user_mask & basic_rates)
+ return;
+
+ sdata_dbg(sdata,
+ "no overlap between basic rates (0x%x) and user mask (0x%x on band %d) - clearing the latter",
+ basic_rates, user_mask, band);
+ sdata->rc_rateidx_mask[band] = (1 << sband->n_bitrates) - 1;
+}
+
static bool rc_no_data_or_no_ack_use_min(struct ieee80211_tx_rate_control *txrc)
{
struct sk_buff *skb = txrc->skb;
@@ -875,7 +926,9 @@
struct ieee80211_sta_rates *old;
struct ieee80211_supported_band *sband;
- sband = hw->wiphy->bands[ieee80211_get_sdata_band(sta->sdata)];
+ sband = ieee80211_get_sband(sta->sdata);
+ if (!sband)
+ return -EINVAL;
rate_control_apply_mask_ratetbl(sta, sband, rates);
/*
* mac80211 guarantees that this function will not be called
@@ -936,6 +989,6 @@
return;
local->rate_ctrl = NULL;
- rate_control_free(ref);
+ rate_control_free(local, ref);
}
diff --git a/net/mac80211/rate.h b/net/mac80211/rate.h
index 8d32607..8212bfe 100644
--- a/net/mac80211/rate.h
+++ b/net/mac80211/rate.h
@@ -20,7 +20,6 @@
#include "driver-ops.h"
struct rate_control_ref {
- struct ieee80211_local *local;
const struct rate_control_ops *ops;
void *priv;
};
@@ -29,47 +28,9 @@
struct sta_info *sta,
struct ieee80211_tx_rate_control *txrc);
-static inline void rate_control_tx_status(struct ieee80211_local *local,
- struct ieee80211_supported_band *sband,
- struct sta_info *sta,
- struct sk_buff *skb)
-{
- struct rate_control_ref *ref = local->rate_ctrl;
- struct ieee80211_sta *ista = &sta->sta;
- void *priv_sta = sta->rate_ctrl_priv;
- struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
-
- if (!ref || !test_sta_flag(sta, WLAN_STA_RATE_CONTROL))
- return;
-
- spin_lock_bh(&sta->rate_ctrl_lock);
- if (ref->ops->tx_status)
- ref->ops->tx_status(ref->priv, sband, ista, priv_sta, skb);
- else
- ref->ops->tx_status_noskb(ref->priv, sband, ista, priv_sta, info);
- spin_unlock_bh(&sta->rate_ctrl_lock);
-}
-
-static inline void
-rate_control_tx_status_noskb(struct ieee80211_local *local,
- struct ieee80211_supported_band *sband,
- struct sta_info *sta,
- struct ieee80211_tx_info *info)
-{
- struct rate_control_ref *ref = local->rate_ctrl;
- struct ieee80211_sta *ista = &sta->sta;
- void *priv_sta = sta->rate_ctrl_priv;
-
- if (!ref || !test_sta_flag(sta, WLAN_STA_RATE_CONTROL))
- return;
-
- if (WARN_ON_ONCE(!ref->ops->tx_status_noskb))
- return;
-
- spin_lock_bh(&sta->rate_ctrl_lock);
- ref->ops->tx_status_noskb(ref->priv, sband, ista, priv_sta, info);
- spin_unlock_bh(&sta->rate_ctrl_lock);
-}
+void rate_control_tx_status(struct ieee80211_local *local,
+ struct ieee80211_supported_band *sband,
+ struct ieee80211_tx_status *st);
void rate_control_rate_init(struct sta_info *sta);
void rate_control_rate_update(struct ieee80211_local *local,
@@ -111,6 +72,8 @@
#endif
}
+void ieee80211_check_rate_mask(struct ieee80211_sub_if_data *sdata);
+
/* Get a reference to the rate control algorithm. If `name' is NULL, get the
* first available algorithm. */
int ieee80211_init_rate_ctrl_alg(struct ieee80211_local *local,
diff --git a/net/mac80211/rc80211_minstrel.c b/net/mac80211/rc80211_minstrel.c
index 3ebe440..9766c1c 100644
--- a/net/mac80211/rc80211_minstrel.c
+++ b/net/mac80211/rc80211_minstrel.c
@@ -264,9 +264,9 @@
static void
minstrel_tx_status(void *priv, struct ieee80211_supported_band *sband,
- struct ieee80211_sta *sta, void *priv_sta,
- struct ieee80211_tx_info *info)
+ void *priv_sta, struct ieee80211_tx_status *st)
{
+ struct ieee80211_tx_info *info = st->info;
struct minstrel_priv *mp = priv;
struct minstrel_sta_info *mi = priv_sta;
struct ieee80211_tx_rate *ar = info->status.rates;
@@ -726,7 +726,7 @@
const struct rate_control_ops mac80211_minstrel = {
.name = "minstrel",
- .tx_status_noskb = minstrel_tx_status,
+ .tx_status_ext = minstrel_tx_status,
.get_rate = minstrel_get_rate,
.rate_init = minstrel_rate_init,
.alloc = minstrel_alloc,
diff --git a/net/mac80211/rc80211_minstrel_ht.c b/net/mac80211/rc80211_minstrel_ht.c
index 8e783e1..4a5bdad 100644
--- a/net/mac80211/rc80211_minstrel_ht.c
+++ b/net/mac80211/rc80211_minstrel_ht.c
@@ -678,9 +678,9 @@
static void
minstrel_ht_tx_status(void *priv, struct ieee80211_supported_band *sband,
- struct ieee80211_sta *sta, void *priv_sta,
- struct ieee80211_tx_info *info)
+ void *priv_sta, struct ieee80211_tx_status *st)
{
+ struct ieee80211_tx_info *info = st->info;
struct minstrel_ht_sta_priv *msp = priv_sta;
struct minstrel_ht_sta *mi = &msp->ht;
struct ieee80211_tx_rate *ar = info->status.rates;
@@ -690,8 +690,8 @@
int i;
if (!msp->is_ht)
- return mac80211_minstrel.tx_status_noskb(priv, sband, sta,
- &msp->legacy, info);
+ return mac80211_minstrel.tx_status_ext(priv, sband,
+ &msp->legacy, st);
/* This packet was aggregated but doesn't carry status info */
if ((info->flags & IEEE80211_TX_CTL_AMPDU) &&
@@ -1374,7 +1374,7 @@
static const struct rate_control_ops mac80211_minstrel_ht = {
.name = "minstrel_ht",
- .tx_status_noskb = minstrel_ht_tx_status,
+ .tx_status_ext = minstrel_ht_tx_status,
.get_rate = minstrel_ht_get_rate,
.rate_init = minstrel_ht_rate_init,
.rate_update = minstrel_ht_rate_update,
diff --git a/net/mac80211/rx.c b/net/mac80211/rx.c
index 4d7543d..35f4c7d 100644
--- a/net/mac80211/rx.c
+++ b/net/mac80211/rx.c
@@ -95,24 +95,13 @@
* This function cleans up the SKB, i.e. it removes all the stuff
* only useful for monitoring.
*/
-static struct sk_buff *remove_monitor_info(struct ieee80211_local *local,
- struct sk_buff *skb,
- unsigned int rtap_vendor_space)
+static void remove_monitor_info(struct sk_buff *skb,
+ unsigned int present_fcs_len,
+ unsigned int rtap_vendor_space)
{
- if (ieee80211_hw_check(&local->hw, RX_INCLUDES_FCS)) {
- if (likely(skb->len > FCS_LEN))
- __pskb_trim(skb, skb->len - FCS_LEN);
- else {
- /* driver bug */
- WARN_ON(1);
- dev_kfree_skb(skb);
- return NULL;
- }
- }
-
+ if (present_fcs_len)
+ __pskb_trim(skb, skb->len - present_fcs_len);
__pskb_pull(skb, rtap_vendor_space);
-
- return skb;
}
static inline bool should_drop_frame(struct sk_buff *skb, int present_fcs_len,
@@ -167,7 +156,7 @@
/* padding for RX_FLAGS if necessary */
len = ALIGN(len, 2);
- if (status->flag & RX_FLAG_HT) /* HT info */
+ if (status->encoding == RX_ENC_HT) /* HT info */
len += 3;
if (status->flag & RX_FLAG_AMPDU_DETAILS) {
@@ -175,7 +164,7 @@
len += 8;
}
- if (status->flag & RX_FLAG_VHT) {
+ if (status->encoding == RX_ENC_VHT) {
len = ALIGN(len, 2);
len += 12;
}
@@ -340,12 +329,12 @@
*pos |= IEEE80211_RADIOTAP_F_FCS;
if (status->flag & (RX_FLAG_FAILED_FCS_CRC | RX_FLAG_FAILED_PLCP_CRC))
*pos |= IEEE80211_RADIOTAP_F_BADFCS;
- if (status->flag & RX_FLAG_SHORTPRE)
+ if (status->enc_flags & RX_ENC_FLAG_SHORTPRE)
*pos |= IEEE80211_RADIOTAP_F_SHORTPRE;
pos++;
/* IEEE80211_RADIOTAP_RATE */
- if (!rate || status->flag & (RX_FLAG_HT | RX_FLAG_VHT)) {
+ if (!rate || status->encoding != RX_ENC_LEGACY) {
/*
* Without rate information don't add it. If we have,
* MCS information is a separate field in radiotap,
@@ -356,9 +345,9 @@
} else {
int shift = 0;
rthdr->it_present |= cpu_to_le32(1 << IEEE80211_RADIOTAP_RATE);
- if (status->flag & RX_FLAG_10MHZ)
+ if (status->bw == RATE_INFO_BW_10)
shift = 1;
- else if (status->flag & RX_FLAG_5MHZ)
+ else if (status->bw == RATE_INFO_BW_5)
shift = 2;
*pos = DIV_ROUND_UP(rate->bitrate, 5 * (1 << shift));
}
@@ -367,14 +356,14 @@
/* IEEE80211_RADIOTAP_CHANNEL */
put_unaligned_le16(status->freq, pos);
pos += 2;
- if (status->flag & RX_FLAG_10MHZ)
+ if (status->bw == RATE_INFO_BW_10)
channel_flags |= IEEE80211_CHAN_HALF;
- else if (status->flag & RX_FLAG_5MHZ)
+ else if (status->bw == RATE_INFO_BW_5)
channel_flags |= IEEE80211_CHAN_QUARTER;
if (status->band == NL80211_BAND_5GHZ)
channel_flags |= IEEE80211_CHAN_OFDM | IEEE80211_CHAN_5GHZ;
- else if (status->flag & (RX_FLAG_HT | RX_FLAG_VHT))
+ else if (status->encoding != RX_ENC_LEGACY)
channel_flags |= IEEE80211_CHAN_DYN | IEEE80211_CHAN_2GHZ;
else if (rate && rate->flags & IEEE80211_RATE_ERP_G)
channel_flags |= IEEE80211_CHAN_OFDM | IEEE80211_CHAN_2GHZ;
@@ -413,21 +402,21 @@
put_unaligned_le16(rx_flags, pos);
pos += 2;
- if (status->flag & RX_FLAG_HT) {
+ if (status->encoding == RX_ENC_HT) {
unsigned int stbc;
rthdr->it_present |= cpu_to_le32(1 << IEEE80211_RADIOTAP_MCS);
*pos++ = local->hw.radiotap_mcs_details;
*pos = 0;
- if (status->flag & RX_FLAG_SHORT_GI)
+ if (status->enc_flags & RX_ENC_FLAG_SHORT_GI)
*pos |= IEEE80211_RADIOTAP_MCS_SGI;
- if (status->flag & RX_FLAG_40MHZ)
+ if (status->bw == RATE_INFO_BW_40)
*pos |= IEEE80211_RADIOTAP_MCS_BW_40;
- if (status->flag & RX_FLAG_HT_GF)
+ if (status->enc_flags & RX_ENC_FLAG_HT_GF)
*pos |= IEEE80211_RADIOTAP_MCS_FMT_GF;
- if (status->flag & RX_FLAG_LDPC)
+ if (status->enc_flags & RX_ENC_FLAG_LDPC)
*pos |= IEEE80211_RADIOTAP_MCS_FEC_LDPC;
- stbc = (status->flag & RX_FLAG_STBC_MASK) >> RX_FLAG_STBC_SHIFT;
+ stbc = (status->enc_flags & RX_ENC_FLAG_STBC_MASK) >> RX_ENC_FLAG_STBC_SHIFT;
*pos |= stbc << IEEE80211_RADIOTAP_MCS_STBC_SHIFT;
pos++;
*pos++ = status->rate_idx;
@@ -460,35 +449,40 @@
*pos++ = 0;
}
- if (status->flag & RX_FLAG_VHT) {
+ if (status->encoding == RX_ENC_VHT) {
u16 known = local->hw.radiotap_vht_details;
rthdr->it_present |= cpu_to_le32(1 << IEEE80211_RADIOTAP_VHT);
put_unaligned_le16(known, pos);
pos += 2;
/* flags */
- if (status->flag & RX_FLAG_SHORT_GI)
+ if (status->enc_flags & RX_ENC_FLAG_SHORT_GI)
*pos |= IEEE80211_RADIOTAP_VHT_FLAG_SGI;
/* in VHT, STBC is binary */
- if (status->flag & RX_FLAG_STBC_MASK)
+ if (status->enc_flags & RX_ENC_FLAG_STBC_MASK)
*pos |= IEEE80211_RADIOTAP_VHT_FLAG_STBC;
- if (status->vht_flag & RX_VHT_FLAG_BF)
+ if (status->enc_flags & RX_ENC_FLAG_BF)
*pos |= IEEE80211_RADIOTAP_VHT_FLAG_BEAMFORMED;
pos++;
/* bandwidth */
- if (status->vht_flag & RX_VHT_FLAG_80MHZ)
+ switch (status->bw) {
+ case RATE_INFO_BW_80:
*pos++ = 4;
- else if (status->vht_flag & RX_VHT_FLAG_160MHZ)
+ break;
+ case RATE_INFO_BW_160:
*pos++ = 11;
- else if (status->flag & RX_FLAG_40MHZ)
+ break;
+ case RATE_INFO_BW_40:
*pos++ = 1;
- else /* 20 MHz */
+ break;
+ default:
*pos++ = 0;
+ }
/* MCS/NSS */
- *pos = (status->rate_idx << 4) | status->vht_nss;
+ *pos = (status->rate_idx << 4) | status->nss;
pos += 4;
/* coding field */
- if (status->flag & RX_FLAG_LDPC)
+ if (status->enc_flags & RX_ENC_FLAG_LDPC)
*pos |= IEEE80211_RADIOTAP_CODING_LDPC_USER0;
pos++;
/* group ID */
@@ -544,69 +538,24 @@
}
}
-/*
- * This function copies a received frame to all monitor interfaces and
- * returns a cleaned-up SKB that no longer includes the FCS nor the
- * radiotap header the driver might have added.
- */
static struct sk_buff *
-ieee80211_rx_monitor(struct ieee80211_local *local, struct sk_buff *origskb,
- struct ieee80211_rate *rate)
+ieee80211_make_monitor_skb(struct ieee80211_local *local,
+ struct sk_buff **origskb,
+ struct ieee80211_rate *rate,
+ int rtap_vendor_space, bool use_origskb)
{
- struct ieee80211_rx_status *status = IEEE80211_SKB_RXCB(origskb);
- struct ieee80211_sub_if_data *sdata;
+ struct ieee80211_rx_status *status = IEEE80211_SKB_RXCB(*origskb);
int rt_hdrlen, needed_headroom;
- struct sk_buff *skb, *skb2;
- struct net_device *prev_dev = NULL;
- int present_fcs_len = 0;
- unsigned int rtap_vendor_space = 0;
- struct ieee80211_sub_if_data *monitor_sdata =
- rcu_dereference(local->monitor_sdata);
-
- if (unlikely(status->flag & RX_FLAG_RADIOTAP_VENDOR_DATA)) {
- struct ieee80211_vendor_radiotap *rtap = (void *)origskb->data;
-
- rtap_vendor_space = sizeof(*rtap) + rtap->len + rtap->pad;
- }
-
- /*
- * First, we may need to make a copy of the skb because
- * (1) we need to modify it for radiotap (if not present), and
- * (2) the other RX handlers will modify the skb we got.
- *
- * We don't need to, of course, if we aren't going to return
- * the SKB because it has a bad FCS/PLCP checksum.
- */
-
- if (ieee80211_hw_check(&local->hw, RX_INCLUDES_FCS))
- present_fcs_len = FCS_LEN;
-
- /* ensure hdr->frame_control and vendor radiotap data are in skb head */
- if (!pskb_may_pull(origskb, 2 + rtap_vendor_space)) {
- dev_kfree_skb(origskb);
- return NULL;
- }
-
- if (!local->monitors || (status->flag & RX_FLAG_SKIP_MONITOR)) {
- if (should_drop_frame(origskb, present_fcs_len,
- rtap_vendor_space)) {
- dev_kfree_skb(origskb);
- return NULL;
- }
-
- return remove_monitor_info(local, origskb, rtap_vendor_space);
- }
-
- ieee80211_handle_mu_mimo_mon(monitor_sdata, origskb, rtap_vendor_space);
+ struct sk_buff *skb;
/* room for the radiotap header based on driver features */
- rt_hdrlen = ieee80211_rx_radiotap_hdrlen(local, status, origskb);
+ rt_hdrlen = ieee80211_rx_radiotap_hdrlen(local, status, *origskb);
needed_headroom = rt_hdrlen - rtap_vendor_space;
- if (should_drop_frame(origskb, present_fcs_len, rtap_vendor_space)) {
+ if (use_origskb) {
/* only need to expand headroom if necessary */
- skb = origskb;
- origskb = NULL;
+ skb = *origskb;
+ *origskb = NULL;
/*
* This shouldn't trigger often because most devices have an
@@ -625,13 +574,10 @@
* Need to make a copy and possibly remove radiotap header
* and FCS from the original.
*/
- skb = skb_copy_expand(origskb, needed_headroom, 0, GFP_ATOMIC);
-
- origskb = remove_monitor_info(local, origskb,
- rtap_vendor_space);
+ skb = skb_copy_expand(*origskb, needed_headroom, 0, GFP_ATOMIC);
if (!skb)
- return origskb;
+ return NULL;
}
/* prepend radiotap information */
@@ -642,34 +588,114 @@
skb->pkt_type = PACKET_OTHERHOST;
skb->protocol = htons(ETH_P_802_2);
- list_for_each_entry_rcu(sdata, &local->interfaces, list) {
- if (sdata->vif.type != NL80211_IFTYPE_MONITOR)
- continue;
+ return skb;
+}
- if (sdata->u.mntr.flags & MONITOR_FLAG_COOK_FRAMES)
- continue;
+/*
+ * This function copies a received frame to all monitor interfaces and
+ * returns a cleaned-up SKB that no longer includes the FCS nor the
+ * radiotap header the driver might have added.
+ */
+static struct sk_buff *
+ieee80211_rx_monitor(struct ieee80211_local *local, struct sk_buff *origskb,
+ struct ieee80211_rate *rate)
+{
+ struct ieee80211_rx_status *status = IEEE80211_SKB_RXCB(origskb);
+ struct ieee80211_sub_if_data *sdata;
+ struct sk_buff *monskb = NULL;
+ int present_fcs_len = 0;
+ unsigned int rtap_vendor_space = 0;
+ struct ieee80211_sub_if_data *monitor_sdata =
+ rcu_dereference(local->monitor_sdata);
+ bool only_monitor = false;
- if (!ieee80211_sdata_running(sdata))
- continue;
+ if (unlikely(status->flag & RX_FLAG_RADIOTAP_VENDOR_DATA)) {
+ struct ieee80211_vendor_radiotap *rtap = (void *)origskb->data;
- if (prev_dev) {
- skb2 = skb_clone(skb, GFP_ATOMIC);
- if (skb2) {
- skb2->dev = prev_dev;
- netif_receive_skb(skb2);
+ rtap_vendor_space = sizeof(*rtap) + rtap->len + rtap->pad;
+ }
+
+ /*
+ * First, we may need to make a copy of the skb because
+ * (1) we need to modify it for radiotap (if not present), and
+ * (2) the other RX handlers will modify the skb we got.
+ *
+ * We don't need to, of course, if we aren't going to return
+ * the SKB because it has a bad FCS/PLCP checksum.
+ */
+
+ if (ieee80211_hw_check(&local->hw, RX_INCLUDES_FCS)) {
+ if (unlikely(origskb->len <= FCS_LEN)) {
+ /* driver bug */
+ WARN_ON(1);
+ dev_kfree_skb(origskb);
+ return NULL;
+ }
+ present_fcs_len = FCS_LEN;
+ }
+
+ /* ensure hdr->frame_control and vendor radiotap data are in skb head */
+ if (!pskb_may_pull(origskb, 2 + rtap_vendor_space)) {
+ dev_kfree_skb(origskb);
+ return NULL;
+ }
+
+ only_monitor = should_drop_frame(origskb, present_fcs_len,
+ rtap_vendor_space);
+
+ if (!local->monitors || (status->flag & RX_FLAG_SKIP_MONITOR)) {
+ if (only_monitor) {
+ dev_kfree_skb(origskb);
+ return NULL;
+ }
+
+ remove_monitor_info(origskb, present_fcs_len,
+ rtap_vendor_space);
+ return origskb;
+ }
+
+ ieee80211_handle_mu_mimo_mon(monitor_sdata, origskb, rtap_vendor_space);
+
+ list_for_each_entry_rcu(sdata, &local->mon_list, u.mntr.list) {
+ bool last_monitor = list_is_last(&sdata->u.mntr.list,
+ &local->mon_list);
+
+ if (!monskb)
+ monskb = ieee80211_make_monitor_skb(local, &origskb,
+ rate,
+ rtap_vendor_space,
+ only_monitor &&
+ last_monitor);
+
+ if (monskb) {
+ struct sk_buff *skb;
+
+ if (last_monitor) {
+ skb = monskb;
+ monskb = NULL;
+ } else {
+ skb = skb_clone(monskb, GFP_ATOMIC);
+ }
+
+ if (skb) {
+ skb->dev = sdata->dev;
+ ieee80211_rx_stats(skb->dev, skb->len);
+ netif_receive_skb(skb);
}
}
- prev_dev = sdata->dev;
- ieee80211_rx_stats(sdata->dev, skb->len);
+ if (last_monitor)
+ break;
}
- if (prev_dev) {
- skb->dev = prev_dev;
- netif_receive_skb(skb);
- } else
- dev_kfree_skb(skb);
+ /* this happens if last_monitor was erroneously false */
+ dev_kfree_skb(monskb);
+ /* ditto */
+ if (!origskb)
+ return NULL;
+
+ remove_monitor_info(origskb, present_fcs_len, rtap_vendor_space);
return origskb;
}
@@ -3315,8 +3341,8 @@
status = IEEE80211_SKB_RXCB((rx->skb));
sband = rx->local->hw.wiphy->bands[status->band];
- if (!(status->flag & RX_FLAG_HT) &&
- !(status->flag & RX_FLAG_VHT))
+ if (!(status->encoding == RX_ENC_HT) &&
+ !(status->encoding == RX_ENC_VHT))
rate = &sband->bitrates[status->rate_idx];
ieee80211_rx_cooked_monitor(rx, rate);
@@ -3553,7 +3579,7 @@
struct ieee80211_hdr *hdr = (void *)skb->data;
struct ieee80211_rx_status *status = IEEE80211_SKB_RXCB(skb);
u8 *bssid = ieee80211_get_bssid(hdr, skb->len, sdata->vif.type);
- int multicast = is_multicast_ether_addr(hdr->addr1);
+ bool multicast = is_multicast_ether_addr(hdr->addr1);
switch (sdata->vif.type) {
case NL80211_IFTYPE_STATION:
@@ -3577,7 +3603,7 @@
return false;
if (!rx->sta) {
int rate_idx;
- if (status->flag & (RX_FLAG_HT | RX_FLAG_VHT))
+ if (status->encoding != RX_ENC_LEGACY)
rate_idx = 0; /* TODO: HT/VHT rates */
else
rate_idx = status->rate_idx;
@@ -3597,7 +3623,7 @@
return false;
if (!rx->sta) {
int rate_idx;
- if (status->flag & RX_FLAG_HT)
+ if (status->encoding != RX_ENC_LEGACY)
rate_idx = 0; /* TODO: HT rates */
else
rate_idx = status->rate_idx;
@@ -4260,7 +4286,8 @@
* we probably can't have a valid rate here anyway.
*/
- if (status->flag & RX_FLAG_HT) {
+ switch (status->encoding) {
+ case RX_ENC_HT:
/*
* rate_idx is MCS index, which can be [0-76]
* as documented on:
@@ -4278,14 +4305,19 @@
status->rate_idx,
status->rate_idx))
goto drop;
- } else if (status->flag & RX_FLAG_VHT) {
+ break;
+ case RX_ENC_VHT:
if (WARN_ONCE(status->rate_idx > 9 ||
- !status->vht_nss ||
- status->vht_nss > 8,
+ !status->nss ||
+ status->nss > 8,
"Rate marked as a VHT rate but data is invalid: MCS: %d, NSS: %d\n",
- status->rate_idx, status->vht_nss))
+ status->rate_idx, status->nss))
goto drop;
- } else {
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ /* fall through */
+ case RX_ENC_LEGACY:
if (WARN_ON(status->rate_idx >= sband->n_bitrates))
goto drop;
rate = &sband->bitrates[status->rate_idx];
diff --git a/net/mac80211/scan.c b/net/mac80211/scan.c
index faab3c4..47d2ed5 100644
--- a/net/mac80211/scan.c
+++ b/net/mac80211/scan.c
@@ -79,9 +79,9 @@
bss_meta.signal = (rx_status->signal * 100) / local->hw.max_signal;
bss_meta.scan_width = NL80211_BSS_CHAN_WIDTH_20;
- if (rx_status->flag & RX_FLAG_5MHZ)
+ if (rx_status->bw == RATE_INFO_BW_5)
bss_meta.scan_width = NL80211_BSS_CHAN_WIDTH_5;
- if (rx_status->flag & RX_FLAG_10MHZ)
+ else if (rx_status->bw == RATE_INFO_BW_10)
bss_meta.scan_width = NL80211_BSS_CHAN_WIDTH_10;
bss_meta.chan = channel;
@@ -174,8 +174,8 @@
if (beacon) {
struct ieee80211_supported_band *sband =
local->hw.wiphy->bands[rx_status->band];
- if (!(rx_status->flag & RX_FLAG_HT) &&
- !(rx_status->flag & RX_FLAG_VHT))
+ if (!(rx_status->encoding == RX_ENC_HT) &&
+ !(rx_status->encoding == RX_ENC_VHT))
bss->beacon_rate =
&sband->bitrates[rx_status->rate_idx];
}
@@ -1219,7 +1219,7 @@
trace_api_sched_scan_results(local);
- cfg80211_sched_scan_results(hw->wiphy);
+ cfg80211_sched_scan_results(hw->wiphy, 0);
}
EXPORT_SYMBOL(ieee80211_sched_scan_results);
@@ -1239,7 +1239,7 @@
mutex_unlock(&local->mtx);
- cfg80211_sched_scan_stopped(local->hw.wiphy);
+ cfg80211_sched_scan_stopped(local->hw.wiphy, 0);
}
void ieee80211_sched_scan_stopped_work(struct work_struct *work)
diff --git a/net/mac80211/spectmgmt.c b/net/mac80211/spectmgmt.c
index 97f4c9d..0782e48 100644
--- a/net/mac80211/spectmgmt.c
+++ b/net/mac80211/spectmgmt.c
@@ -132,9 +132,9 @@
struct ieee80211_vht_operation vht_oper = {
.chan_width =
wide_bw_chansw_ie->new_channel_width,
- .center_freq_seg1_idx =
+ .center_freq_seg0_idx =
wide_bw_chansw_ie->new_center_freq_seg0,
- .center_freq_seg2_idx =
+ .center_freq_seg1_idx =
wide_bw_chansw_ie->new_center_freq_seg1,
/* .basic_mcs_set doesn't matter */
};
diff --git a/net/mac80211/sta_info.c b/net/mac80211/sta_info.c
index 3323a2f..7cdf7a8 100644
--- a/net/mac80211/sta_info.c
+++ b/net/mac80211/sta_info.c
@@ -2,7 +2,7 @@
* Copyright 2002-2005, Instant802 Networks, Inc.
* Copyright 2006-2007 Jiri Benc <jbenc@suse.cz>
* Copyright 2013-2014 Intel Mobile Communications GmbH
- * Copyright (C) 2015 - 2016 Intel Deutschland GmbH
+ * Copyright (C) 2015 - 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -395,10 +395,15 @@
sta->sta.smps_mode = IEEE80211_SMPS_OFF;
if (sdata->vif.type == NL80211_IFTYPE_AP ||
sdata->vif.type == NL80211_IFTYPE_AP_VLAN) {
- struct ieee80211_supported_band *sband =
- hw->wiphy->bands[ieee80211_get_sdata_band(sdata)];
- u8 smps = (sband->ht_cap.cap & IEEE80211_HT_CAP_SM_PS) >>
- IEEE80211_HT_CAP_SM_PS_SHIFT;
+ struct ieee80211_supported_band *sband;
+ u8 smps;
+
+ sband = ieee80211_get_sband(sdata);
+ if (!sband)
+ goto free_txq;
+
+ smps = (sband->ht_cap.cap & IEEE80211_HT_CAP_SM_PS) >>
+ IEEE80211_HT_CAP_SM_PS_SHIFT;
/*
* Assume that hostapd advertises our caps in the beacon and
* this is the known_smps_mode for a station that just assciated
@@ -1957,24 +1962,32 @@
static void sta_stats_decode_rate(struct ieee80211_local *local, u16 rate,
struct rate_info *rinfo)
{
- rinfo->bw = (rate & STA_STATS_RATE_BW_MASK) >>
- STA_STATS_RATE_BW_SHIFT;
+ rinfo->bw = STA_STATS_GET(BW, rate);
- if (rate & STA_STATS_RATE_VHT) {
+ switch (STA_STATS_GET(TYPE, rate)) {
+ case STA_STATS_RATE_TYPE_VHT:
rinfo->flags = RATE_INFO_FLAGS_VHT_MCS;
- rinfo->mcs = rate & 0xf;
- rinfo->nss = (rate & 0xf0) >> 4;
- } else if (rate & STA_STATS_RATE_HT) {
+ rinfo->mcs = STA_STATS_GET(VHT_MCS, rate);
+ rinfo->nss = STA_STATS_GET(VHT_NSS, rate);
+ if (STA_STATS_GET(SGI, rate))
+ rinfo->flags |= RATE_INFO_FLAGS_SHORT_GI;
+ break;
+ case STA_STATS_RATE_TYPE_HT:
rinfo->flags = RATE_INFO_FLAGS_MCS;
- rinfo->mcs = rate & 0xff;
- } else if (rate & STA_STATS_RATE_LEGACY) {
+ rinfo->mcs = STA_STATS_GET(HT_MCS, rate);
+ if (STA_STATS_GET(SGI, rate))
+ rinfo->flags |= RATE_INFO_FLAGS_SHORT_GI;
+ break;
+ case STA_STATS_RATE_TYPE_LEGACY: {
struct ieee80211_supported_band *sband;
u16 brate;
unsigned int shift;
+ int band = STA_STATS_GET(LEGACY_BAND, rate);
+ int rate_idx = STA_STATS_GET(LEGACY_IDX, rate);
rinfo->flags = 0;
- sband = local->hw.wiphy->bands[(rate >> 4) & 0xf];
- brate = sband->bitrates[rate & 0xf].bitrate;
+ sband = local->hw.wiphy->bands[band];
+ brate = sband->bitrates[rate_idx].bitrate;
if (rinfo->bw == RATE_INFO_BW_5)
shift = 2;
else if (rinfo->bw == RATE_INFO_BW_10)
@@ -1982,10 +1995,9 @@
else
shift = 0;
rinfo->legacy = DIV_ROUND_UP(brate, 1 << shift);
+ break;
+ }
}
-
- if (rate & STA_STATS_RATE_SGI)
- rinfo->flags |= RATE_INFO_FLAGS_SHORT_GI;
}
static int sta_set_rate_info_rx(struct sta_info *sta, struct rate_info *rinfo)
diff --git a/net/mac80211/sta_info.h b/net/mac80211/sta_info.h
index e65cda3..5609cac 100644
--- a/net/mac80211/sta_info.h
+++ b/net/mac80211/sta_info.h
@@ -1,7 +1,7 @@
/*
* Copyright 2002-2005, Devicescape Software, Inc.
* Copyright 2013-2014 Intel Mobile Communications GmbH
- * Copyright(c) 2015-2016 Intel Deutschland GmbH
+ * Copyright(c) 2015-2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -16,6 +16,7 @@
#include <linux/if_ether.h>
#include <linux/workqueue.h>
#include <linux/average.h>
+#include <linux/bitfield.h>
#include <linux/etherdevice.h>
#include <linux/rhashtable.h>
#include <linux/u64_stats_sync.h>
@@ -324,6 +325,9 @@
struct rcu_head rcu_head;
};
+/* we use only values in the range 0-100, so pick a large precision */
+DECLARE_EWMA(mesh_fail_avg, 20, 8)
+
/**
* struct mesh_sta - mesh STA information
* @plink_lock: serialize access to plink fields
@@ -369,7 +373,7 @@
enum nl80211_mesh_power_mode nonpeer_pm;
/* moving percentage of failed MSDUs */
- unsigned int fail_avg;
+ struct ewma_mesh_fail_avg fail_avg;
};
DECLARE_EWMA(signal, 10, 8)
@@ -724,40 +728,55 @@
unsigned long ieee80211_sta_last_active(struct sta_info *sta);
+enum sta_stats_type {
+ STA_STATS_RATE_TYPE_INVALID = 0,
+ STA_STATS_RATE_TYPE_LEGACY,
+ STA_STATS_RATE_TYPE_HT,
+ STA_STATS_RATE_TYPE_VHT,
+};
+
+#define STA_STATS_FIELD_HT_MCS GENMASK( 7, 0)
+#define STA_STATS_FIELD_LEGACY_IDX GENMASK( 3, 0)
+#define STA_STATS_FIELD_LEGACY_BAND GENMASK( 7, 4)
+#define STA_STATS_FIELD_VHT_MCS GENMASK( 3, 0)
+#define STA_STATS_FIELD_VHT_NSS GENMASK( 7, 4)
+#define STA_STATS_FIELD_BW GENMASK(11, 8)
+#define STA_STATS_FIELD_SGI GENMASK(12, 12)
+#define STA_STATS_FIELD_TYPE GENMASK(15, 13)
+
+#define STA_STATS_FIELD(_n, _v) FIELD_PREP(STA_STATS_FIELD_ ## _n, _v)
+#define STA_STATS_GET(_n, _v) FIELD_GET(STA_STATS_FIELD_ ## _n, _v)
+
#define STA_STATS_RATE_INVALID 0
-#define STA_STATS_RATE_VHT 0x8000
-#define STA_STATS_RATE_HT 0x4000
-#define STA_STATS_RATE_LEGACY 0x2000
-#define STA_STATS_RATE_SGI 0x1000
-#define STA_STATS_RATE_BW_SHIFT 9
-#define STA_STATS_RATE_BW_MASK (0x7 << STA_STATS_RATE_BW_SHIFT)
-static inline u16 sta_stats_encode_rate(struct ieee80211_rx_status *s)
+static inline u32 sta_stats_encode_rate(struct ieee80211_rx_status *s)
{
- u16 r = s->rate_idx;
+ u16 r;
- if (s->vht_flag & RX_VHT_FLAG_80MHZ)
- r |= RATE_INFO_BW_80 << STA_STATS_RATE_BW_SHIFT;
- else if (s->vht_flag & RX_VHT_FLAG_160MHZ)
- r |= RATE_INFO_BW_160 << STA_STATS_RATE_BW_SHIFT;
- else if (s->flag & RX_FLAG_40MHZ)
- r |= RATE_INFO_BW_40 << STA_STATS_RATE_BW_SHIFT;
- else if (s->flag & RX_FLAG_10MHZ)
- r |= RATE_INFO_BW_10 << STA_STATS_RATE_BW_SHIFT;
- else if (s->flag & RX_FLAG_5MHZ)
- r |= RATE_INFO_BW_5 << STA_STATS_RATE_BW_SHIFT;
- else
- r |= RATE_INFO_BW_20 << STA_STATS_RATE_BW_SHIFT;
+ r = STA_STATS_FIELD(BW, s->bw);
- if (s->flag & RX_FLAG_SHORT_GI)
- r |= STA_STATS_RATE_SGI;
+ if (s->enc_flags & RX_ENC_FLAG_SHORT_GI)
+ r |= STA_STATS_FIELD(SGI, 1);
- if (s->flag & RX_FLAG_VHT)
- r |= STA_STATS_RATE_VHT | (s->vht_nss << 4);
- else if (s->flag & RX_FLAG_HT)
- r |= STA_STATS_RATE_HT;
- else
- r |= STA_STATS_RATE_LEGACY | (s->band << 4);
+ switch (s->encoding) {
+ case RX_ENC_VHT:
+ r |= STA_STATS_FIELD(TYPE, STA_STATS_RATE_TYPE_VHT);
+ r |= STA_STATS_FIELD(VHT_NSS, s->nss);
+ r |= STA_STATS_FIELD(VHT_MCS, s->rate_idx);
+ break;
+ case RX_ENC_HT:
+ r |= STA_STATS_FIELD(TYPE, STA_STATS_RATE_TYPE_HT);
+ r |= STA_STATS_FIELD(HT_MCS, s->rate_idx);
+ break;
+ case RX_ENC_LEGACY:
+ r |= STA_STATS_FIELD(TYPE, STA_STATS_RATE_TYPE_LEGACY);
+ r |= STA_STATS_FIELD(LEGACY_BAND, s->band);
+ r |= STA_STATS_FIELD(LEGACY_IDX, s->rate_idx);
+ break;
+ default:
+ WARN_ON(1);
+ return STA_STATS_RATE_INVALID;
+ }
return r;
}
diff --git a/net/mac80211/status.c b/net/mac80211/status.c
index 83b8b11..be47ac5 100644
--- a/net/mac80211/status.c
+++ b/net/mac80211/status.c
@@ -200,6 +200,7 @@
}
if (ieee80211_is_action(mgmt->frame_control) &&
+ !ieee80211_has_protected(mgmt->frame_control) &&
mgmt->u.action.category == WLAN_CATEGORY_HT &&
mgmt->u.action.u.ht_smps.action == WLAN_HT_ACTION_SMPS &&
ieee80211_sdata_running(sdata)) {
@@ -630,61 +631,6 @@
return rates_idx;
}
-void ieee80211_tx_status_noskb(struct ieee80211_hw *hw,
- struct ieee80211_sta *pubsta,
- struct ieee80211_tx_info *info)
-{
- struct ieee80211_local *local = hw_to_local(hw);
- struct ieee80211_supported_band *sband;
- int retry_count;
- bool acked, noack_success;
-
- ieee80211_tx_get_rates(hw, info, &retry_count);
-
- sband = hw->wiphy->bands[info->band];
-
- acked = !!(info->flags & IEEE80211_TX_STAT_ACK);
- noack_success = !!(info->flags & IEEE80211_TX_STAT_NOACK_TRANSMITTED);
-
- if (pubsta) {
- struct sta_info *sta;
-
- sta = container_of(pubsta, struct sta_info, sta);
-
- if (!acked)
- sta->status_stats.retry_failed++;
- sta->status_stats.retry_count += retry_count;
-
- if (acked) {
- sta->status_stats.last_ack = jiffies;
-
- if (sta->status_stats.lost_packets)
- sta->status_stats.lost_packets = 0;
-
- /* Track when last TDLS packet was ACKed */
- if (test_sta_flag(sta, WLAN_STA_TDLS_PEER_AUTH))
- sta->status_stats.last_tdls_pkt_time = jiffies;
- } else {
- ieee80211_lost_packet(sta, info);
- }
-
- rate_control_tx_status_noskb(local, sband, sta, info);
- }
-
- if (acked || noack_success) {
- I802_DEBUG_INC(local->dot11TransmittedFrameCount);
- if (!pubsta)
- I802_DEBUG_INC(local->dot11MulticastTransmittedFrameCount);
- if (retry_count > 0)
- I802_DEBUG_INC(local->dot11RetryCount);
- if (retry_count > 1)
- I802_DEBUG_INC(local->dot11MultipleRetryCount);
- } else {
- I802_DEBUG_INC(local->dot11FailedCount);
- }
-}
-EXPORT_SYMBOL(ieee80211_tx_status_noskb);
-
void ieee80211_tx_monitor(struct ieee80211_local *local, struct sk_buff *skb,
struct ieee80211_supported_band *sband,
int retry_count, int shift, bool send_to_cooked)
@@ -742,15 +688,16 @@
dev_kfree_skb(skb);
}
-void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
+static void __ieee80211_tx_status(struct ieee80211_hw *hw,
+ struct ieee80211_tx_status *status)
{
+ struct sk_buff *skb = status->skb;
struct ieee80211_hdr *hdr = (struct ieee80211_hdr *) skb->data;
struct ieee80211_local *local = hw_to_local(hw);
- struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
+ struct ieee80211_tx_info *info = status->info;
+ struct sta_info *sta;
__le16 fc;
struct ieee80211_supported_band *sband;
- struct rhlist_head *tmp;
- struct sta_info *sta;
int retry_count;
int rates_idx;
bool send_to_cooked;
@@ -761,16 +708,11 @@
rates_idx = ieee80211_tx_get_rates(hw, info, &retry_count);
- rcu_read_lock();
-
sband = local->hw.wiphy->bands[info->band];
fc = hdr->frame_control;
- for_each_sta_info(local, hdr->addr1, sta, tmp) {
- /* skip wrong virtual interface */
- if (!ether_addr_equal(hdr->addr2, sta->sdata->vif.addr))
- continue;
-
+ if (status->sta) {
+ sta = container_of(status->sta, struct sta_info, sta);
shift = ieee80211_vif_get_shift(&sta->sdata->vif);
if (info->flags & IEEE80211_TX_STATUS_EOSP)
@@ -790,7 +732,6 @@
* that this TX packet failed because of that.
*/
ieee80211_handle_filtered_frame(local, sta, skb);
- rcu_read_unlock();
return;
}
@@ -840,7 +781,6 @@
if (info->flags & IEEE80211_TX_STAT_TX_FILTERED) {
ieee80211_handle_filtered_frame(local, sta, skb);
- rcu_read_unlock();
return;
} else {
if (!acked)
@@ -856,7 +796,7 @@
}
}
- rate_control_tx_status(local, sband, sta, skb);
+ rate_control_tx_status(local, sband, status);
if (ieee80211_vif_is_mesh(&sta->sdata->vif))
ieee80211s_update_metric(local, sta, skb);
@@ -883,8 +823,6 @@
}
}
- rcu_read_unlock();
-
ieee80211_led_tx(local);
/* SNMP counters
@@ -949,8 +887,96 @@
/* send to monitor interfaces */
ieee80211_tx_monitor(local, skb, sband, retry_count, shift, send_to_cooked);
}
+
+void ieee80211_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
+{
+ struct ieee80211_hdr *hdr = (struct ieee80211_hdr *) skb->data;
+ struct ieee80211_local *local = hw_to_local(hw);
+ struct ieee80211_tx_status status = {
+ .skb = skb,
+ .info = IEEE80211_SKB_CB(skb),
+ };
+ struct rhlist_head *tmp;
+ struct sta_info *sta;
+
+ rcu_read_lock();
+
+ for_each_sta_info(local, hdr->addr1, sta, tmp) {
+ /* skip wrong virtual interface */
+ if (!ether_addr_equal(hdr->addr2, sta->sdata->vif.addr))
+ continue;
+
+ status.sta = &sta->sta;
+ break;
+ }
+
+ __ieee80211_tx_status(hw, &status);
+ rcu_read_unlock();
+}
EXPORT_SYMBOL(ieee80211_tx_status);
+void ieee80211_tx_status_ext(struct ieee80211_hw *hw,
+ struct ieee80211_tx_status *status)
+{
+ struct ieee80211_local *local = hw_to_local(hw);
+ struct ieee80211_tx_info *info = status->info;
+ struct ieee80211_sta *pubsta = status->sta;
+ struct ieee80211_supported_band *sband;
+ int retry_count;
+ bool acked, noack_success;
+
+ if (status->skb)
+ return __ieee80211_tx_status(hw, status);
+
+ if (!status->sta)
+ return;
+
+ ieee80211_tx_get_rates(hw, info, &retry_count);
+
+ sband = hw->wiphy->bands[info->band];
+
+ acked = !!(info->flags & IEEE80211_TX_STAT_ACK);
+ noack_success = !!(info->flags & IEEE80211_TX_STAT_NOACK_TRANSMITTED);
+
+ if (pubsta) {
+ struct sta_info *sta;
+
+ sta = container_of(pubsta, struct sta_info, sta);
+
+ if (!acked)
+ sta->status_stats.retry_failed++;
+ sta->status_stats.retry_count += retry_count;
+
+ if (acked) {
+ sta->status_stats.last_ack = jiffies;
+
+ if (sta->status_stats.lost_packets)
+ sta->status_stats.lost_packets = 0;
+
+ /* Track when last TDLS packet was ACKed */
+ if (test_sta_flag(sta, WLAN_STA_TDLS_PEER_AUTH))
+ sta->status_stats.last_tdls_pkt_time = jiffies;
+ } else {
+ ieee80211_lost_packet(sta, info);
+ }
+
+ rate_control_tx_status(local, sband, status);
+ }
+
+ if (acked || noack_success) {
+ I802_DEBUG_INC(local->dot11TransmittedFrameCount);
+ if (!pubsta)
+ I802_DEBUG_INC(local->dot11MulticastTransmittedFrameCount);
+ if (retry_count > 0)
+ I802_DEBUG_INC(local->dot11RetryCount);
+ if (retry_count > 1)
+ I802_DEBUG_INC(local->dot11MultipleRetryCount);
+ } else {
+ I802_DEBUG_INC(local->dot11FailedCount);
+ }
+}
+EXPORT_SYMBOL(ieee80211_tx_status_ext);
+
void ieee80211_report_low_ack(struct ieee80211_sta *pubsta, u32 num_packets)
{
struct sta_info *sta = container_of(pubsta, struct sta_info, sta);
diff --git a/net/mac80211/tdls.c b/net/mac80211/tdls.c
index afca7d1..f20dcf1 100644
--- a/net/mac80211/tdls.c
+++ b/net/mac80211/tdls.c
@@ -47,8 +47,7 @@
NL80211_FEATURE_TDLS_CHANNEL_SWITCH;
bool wider_band = ieee80211_hw_check(&local->hw, TDLS_WIDER_BW) &&
!ifmgd->tdls_wider_bw_prohibited;
- enum nl80211_band band = ieee80211_get_sdata_band(sdata);
- struct ieee80211_supported_band *sband = local->hw.wiphy->bands[band];
+ struct ieee80211_supported_band *sband = ieee80211_get_sband(sdata);
bool vht = sband && sband->vht_cap.vht_supported;
u8 *pos = (void *)skb_put(skb, 10);
@@ -180,11 +179,14 @@
static u16 ieee80211_get_tdls_sta_capab(struct ieee80211_sub_if_data *sdata,
u16 status_code)
{
+ struct ieee80211_supported_band *sband;
+
/* The capability will be 0 when sending a failure code */
if (status_code != 0)
return 0;
- if (ieee80211_get_sdata_band(sdata) == NL80211_BAND_2GHZ) {
+ sband = ieee80211_get_sband(sdata);
+ if (sband && sband->band == NL80211_BAND_2GHZ) {
return WLAN_CAPABILITY_SHORT_SLOT_TIME |
WLAN_CAPABILITY_SHORT_PREAMBLE;
}
@@ -358,17 +360,20 @@
u8 action_code, bool initiator,
const u8 *extra_ies, size_t extra_ies_len)
{
- enum nl80211_band band = ieee80211_get_sdata_band(sdata);
- struct ieee80211_local *local = sdata->local;
struct ieee80211_supported_band *sband;
+ struct ieee80211_local *local = sdata->local;
struct ieee80211_sta_ht_cap ht_cap;
struct ieee80211_sta_vht_cap vht_cap;
struct sta_info *sta = NULL;
size_t offset = 0, noffset;
u8 *pos;
- ieee80211_add_srates_ie(sdata, skb, false, band);
- ieee80211_add_ext_srates_ie(sdata, skb, false, band);
+ sband = ieee80211_get_sband(sdata);
+ if (!sband)
+ return;
+
+ ieee80211_add_srates_ie(sdata, skb, false, sband->band);
+ ieee80211_add_ext_srates_ie(sdata, skb, false, sband->band);
ieee80211_tdls_add_supp_channels(sdata, skb);
/* add any custom IEs that go before Extended Capabilities */
@@ -439,7 +444,6 @@
* the same on all bands. The specification limits the setup to a
* single HT-cap, so use the current band for now.
*/
- sband = local->hw.wiphy->bands[band];
memcpy(&ht_cap, &sband->ht_cap, sizeof(ht_cap));
if ((action_code == WLAN_TDLS_SETUP_REQUEST ||
@@ -545,9 +549,13 @@
struct ieee80211_if_managed *ifmgd = &sdata->u.mgd;
size_t offset = 0, noffset;
struct sta_info *sta, *ap_sta;
- enum nl80211_band band = ieee80211_get_sdata_band(sdata);
+ struct ieee80211_supported_band *sband;
u8 *pos;
+ sband = ieee80211_get_sband(sdata);
+ if (!sband)
+ return;
+
mutex_lock(&local->sta_mtx);
sta = sta_info_get(sdata, peer);
@@ -612,7 +620,8 @@
ieee80211_tdls_add_link_ie(sdata, skb, peer, initiator);
/* only include VHT-operation if not on the 2.4GHz band */
- if (band != NL80211_BAND_2GHZ && sta->sta.vht_cap.vht_supported) {
+ if (sband->band != NL80211_BAND_2GHZ &&
+ sta->sta.vht_cap.vht_supported) {
/*
* if both peers support WIDER_BW, we can expand the chandef to
* a wider compatible one, up to 80MHz
diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c
index ba8d7db..04b22f8 100644
--- a/net/mac80211/tx.c
+++ b/net/mac80211/tx.c
@@ -682,10 +682,6 @@
txrc.skb = tx->skb;
txrc.reported_rate.idx = -1;
txrc.rate_idx_mask = tx->sdata->rc_rateidx_mask[info->band];
- if (txrc.rate_idx_mask == (1 << sband->n_bitrates) - 1)
- txrc.max_rate_idx = -1;
- else
- txrc.max_rate_idx = fls(txrc.rate_idx_mask) - 1;
if (tx->sdata->rc_has_mcs_mask[info->band])
txrc.rate_idx_mcs_mask =
@@ -4249,10 +4245,6 @@
txrc.skb = skb;
txrc.reported_rate.idx = -1;
txrc.rate_idx_mask = sdata->rc_rateidx_mask[band];
- if (txrc.rate_idx_mask == (1 << txrc.sband->n_bitrates) - 1)
- txrc.max_rate_idx = -1;
- else
- txrc.max_rate_idx = fls(txrc.rate_idx_mask) - 1;
txrc.bss = true;
rate_control_get_rate(sdata, NULL, &txrc);
@@ -4305,7 +4297,10 @@
return bcn;
shift = ieee80211_vif_get_shift(vif);
- sband = hw->wiphy->bands[ieee80211_get_sdata_band(vif_to_sdata(vif))];
+ sband = ieee80211_get_sband(vif_to_sdata(vif));
+ if (!sband)
+ return bcn;
+
ieee80211_tx_monitor(hw_to_local(hw), copy, sband, 1, shift, false);
return bcn;
diff --git a/net/mac80211/util.c b/net/mac80211/util.c
index ac59fbd..ac9ac6c 100644
--- a/net/mac80211/util.c
+++ b/net/mac80211/util.c
@@ -4,7 +4,7 @@
* Copyright 2006-2007 Jiri Benc <jbenc@suse.cz>
* Copyright 2007 Johannes Berg <johannes@sipsolutions.net>
* Copyright 2013-2014 Intel Mobile Communications GmbH
- * Copyright (C) 2015-2016 Intel Deutschland GmbH
+ * Copyright (C) 2015-2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -828,6 +828,7 @@
case WLAN_EID_EXT_CAPABILITY:
case WLAN_EID_CHAN_SWITCH_TIMING:
case WLAN_EID_LINK_ID:
+ case WLAN_EID_BSS_MAX_IDLE_PERIOD:
/*
* not listing WLAN_EID_CHANNEL_SWITCH_WRAPPER -- it seems possible
* that if the content gets bigger it might be needed more than once
@@ -1089,6 +1090,10 @@
else
elem_parse_failed = true;
break;
+ case WLAN_EID_BSS_MAX_IDLE_PERIOD:
+ if (elen >= sizeof(*elems->max_idle_period_ie))
+ elems->max_idle_period_ie = (void *)pos;
+ break;
default:
break;
}
@@ -1590,14 +1595,14 @@
size_t num_rates;
u32 supp_rates, rate_flags;
int i, j, shift;
+
sband = sdata->local->hw.wiphy->bands[band];
+ if (WARN_ON(!sband))
+ return 1;
rate_flags = ieee80211_chandef_rate_flags(&sdata->vif.bss_conf.chandef);
shift = ieee80211_vif_get_shift(&sdata->vif);
- if (WARN_ON(!sband))
- return 1;
-
num_rates = sband->n_bitrates;
supp_rates = 0;
for (i = 0; i < elems->supp_rates_len +
@@ -1983,6 +1988,10 @@
if (sdata->u.mgd.have_beacon)
changed |= BSS_CHANGED_BEACON_INFO;
+ if (sdata->vif.bss_conf.max_idle_period ||
+ sdata->vif.bss_conf.protected_keep_alive)
+ changed |= BSS_CHANGED_KEEP_ALIVE;
+
sdata_lock(sdata);
ieee80211_bss_info_change_notify(sdata, changed);
sdata_unlock(sdata);
@@ -2103,7 +2112,7 @@
mutex_unlock(&local->mtx);
if (sched_scan_stopped)
- cfg80211_sched_scan_stopped_rtnl(local->hw.wiphy);
+ cfg80211_sched_scan_stopped_rtnl(local->hw.wiphy, 0);
wake_up:
if (local->in_reconfig) {
@@ -2413,13 +2422,13 @@
*pos++ = WLAN_EID_VHT_OPERATION;
*pos++ = sizeof(struct ieee80211_vht_operation);
vht_oper = (struct ieee80211_vht_operation *)pos;
- vht_oper->center_freq_seg1_idx = ieee80211_frequency_to_channel(
+ vht_oper->center_freq_seg0_idx = ieee80211_frequency_to_channel(
chandef->center_freq1);
if (chandef->center_freq2)
- vht_oper->center_freq_seg2_idx =
+ vht_oper->center_freq_seg1_idx =
ieee80211_frequency_to_channel(chandef->center_freq2);
else
- vht_oper->center_freq_seg2_idx = 0x00;
+ vht_oper->center_freq_seg1_idx = 0x00;
switch (chandef->width) {
case NL80211_CHAN_WIDTH_160:
@@ -2428,11 +2437,11 @@
* workaround.
*/
vht_oper->chan_width = IEEE80211_VHT_CHANWIDTH_80MHZ;
- vht_oper->center_freq_seg2_idx = vht_oper->center_freq_seg1_idx;
+ vht_oper->center_freq_seg1_idx = vht_oper->center_freq_seg0_idx;
if (chandef->chan->center_freq < chandef->center_freq1)
- vht_oper->center_freq_seg1_idx -= 8;
+ vht_oper->center_freq_seg0_idx -= 8;
else
- vht_oper->center_freq_seg1_idx += 8;
+ vht_oper->center_freq_seg0_idx += 8;
break;
case NL80211_CHAN_WIDTH_80P80:
/*
@@ -2491,9 +2500,9 @@
if (!oper)
return false;
- cf1 = ieee80211_channel_to_frequency(oper->center_freq_seg1_idx,
+ cf1 = ieee80211_channel_to_frequency(oper->center_freq_seg0_idx,
chandef->chan->band);
- cf2 = ieee80211_channel_to_frequency(oper->center_freq_seg2_idx,
+ cf2 = ieee80211_channel_to_frequency(oper->center_freq_seg1_idx,
chandef->chan->band);
switch (oper->chan_width) {
@@ -2503,11 +2512,11 @@
new.width = NL80211_CHAN_WIDTH_80;
new.center_freq1 = cf1;
/* If needed, adjust based on the newer interop workaround. */
- if (oper->center_freq_seg2_idx) {
+ if (oper->center_freq_seg1_idx) {
unsigned int diff;
- diff = abs(oper->center_freq_seg2_idx -
- oper->center_freq_seg1_idx);
+ diff = abs(oper->center_freq_seg1_idx -
+ oper->center_freq_seg0_idx);
if (diff == 8) {
new.width = NL80211_CHAN_WIDTH_160;
new.center_freq1 = cf2;
@@ -2715,42 +2724,39 @@
memset(&ri, 0, sizeof(ri));
/* Fill cfg80211 rate info */
- if (status->flag & RX_FLAG_HT) {
+ switch (status->encoding) {
+ case RX_ENC_HT:
ri.mcs = status->rate_idx;
ri.flags |= RATE_INFO_FLAGS_MCS;
- if (status->flag & RX_FLAG_40MHZ)
- ri.bw = RATE_INFO_BW_40;
- else
- ri.bw = RATE_INFO_BW_20;
- if (status->flag & RX_FLAG_SHORT_GI)
+ ri.bw = status->bw;
+ if (status->enc_flags & RX_ENC_FLAG_SHORT_GI)
ri.flags |= RATE_INFO_FLAGS_SHORT_GI;
- } else if (status->flag & RX_FLAG_VHT) {
+ break;
+ case RX_ENC_VHT:
ri.flags |= RATE_INFO_FLAGS_VHT_MCS;
ri.mcs = status->rate_idx;
- ri.nss = status->vht_nss;
- if (status->flag & RX_FLAG_40MHZ)
- ri.bw = RATE_INFO_BW_40;
- else if (status->vht_flag & RX_VHT_FLAG_80MHZ)
- ri.bw = RATE_INFO_BW_80;
- else if (status->vht_flag & RX_VHT_FLAG_160MHZ)
- ri.bw = RATE_INFO_BW_160;
- else
- ri.bw = RATE_INFO_BW_20;
- if (status->flag & RX_FLAG_SHORT_GI)
+ ri.nss = status->nss;
+ ri.bw = status->bw;
+ if (status->enc_flags & RX_ENC_FLAG_SHORT_GI)
ri.flags |= RATE_INFO_FLAGS_SHORT_GI;
- } else {
+ break;
+ default:
+ WARN_ON(1);
+ /* fall through */
+ case RX_ENC_LEGACY: {
struct ieee80211_supported_band *sband;
int shift = 0;
int bitrate;
- if (status->flag & RX_FLAG_10MHZ) {
+ ri.bw = status->bw;
+
+ switch (status->bw) {
+ case RATE_INFO_BW_10:
shift = 1;
- ri.bw = RATE_INFO_BW_10;
- } else if (status->flag & RX_FLAG_5MHZ) {
+ break;
+ case RATE_INFO_BW_5:
shift = 2;
- ri.bw = RATE_INFO_BW_5;
- } else {
- ri.bw = RATE_INFO_BW_20;
+ break;
}
sband = local->hw.wiphy->bands[status->band];
@@ -2762,19 +2768,21 @@
if (status->band == NL80211_BAND_5GHZ) {
ts += 20 << shift;
mpdu_offset += 2;
- } else if (status->flag & RX_FLAG_SHORTPRE) {
+ } else if (status->enc_flags & RX_ENC_FLAG_SHORTPRE) {
ts += 96;
} else {
ts += 192;
}
}
+ break;
+ }
}
rate = cfg80211_calculate_bitrate(&ri);
if (WARN_ONCE(!rate,
"Invalid bitrate: flags=0x%llx, idx=%d, vht_nss=%d\n",
(unsigned long long)status->flag, status->rate_idx,
- status->vht_nss))
+ status->nss))
return 0;
/* rewind from end of MPDU */
@@ -2791,8 +2799,10 @@
struct ieee80211_sub_if_data *sdata;
struct cfg80211_chan_def chandef;
+ /* for interface list, to avoid linking iflist_mtx and chanctx_mtx */
+ ASSERT_RTNL();
+
mutex_lock(&local->mtx);
- mutex_lock(&local->iflist_mtx);
list_for_each_entry(sdata, &local->interfaces, list) {
/* it might be waiting for the local->mtx, but then
* by the time it gets it, sdata->wdev.cac_started
@@ -2809,7 +2819,6 @@
GFP_KERNEL);
}
}
- mutex_unlock(&local->iflist_mtx);
mutex_unlock(&local->mtx);
}
@@ -2831,7 +2840,9 @@
}
mutex_unlock(&local->chanctx_mtx);
+ rtnl_lock();
ieee80211_dfs_cac_cancel(local);
+ rtnl_unlock();
if (num_chanctx > 1)
/* XXX: multi-channel is not supported yet */
@@ -2846,7 +2857,7 @@
trace_api_radar_detected(local);
- ieee80211_queue_work(hw, &local->radar_detected_work);
+ schedule_work(&local->radar_detected_work);
}
EXPORT_SYMBOL(ieee80211_radar_detected);
diff --git a/net/mac802154/ieee802154_i.h b/net/mac802154/ieee802154_i.h
index 56ccffa..62141dc 100644
--- a/net/mac802154/ieee802154_i.h
+++ b/net/mac802154/ieee802154_i.h
@@ -19,6 +19,7 @@
#ifndef __IEEE802154_I_H
#define __IEEE802154_I_H
+#include <linux/interrupt.h>
#include <linux/mutex.h>
#include <linux/hrtimer.h>
#include <net/cfg802154.h>
diff --git a/net/mpls/af_mpls.c b/net/mpls/af_mpls.c
index 6414079..088e2b4 100644
--- a/net/mpls/af_mpls.c
+++ b/net/mpls/af_mpls.c
@@ -24,6 +24,9 @@
#include <net/nexthop.h>
#include "internal.h"
+/* max memory we will use for mpls_route */
+#define MAX_MPLS_ROUTE_MEM 4096
+
/* Maximum number of labels to look ahead at when selecting a path of
* a multipath route
*/
@@ -32,7 +35,9 @@
#define MPLS_NEIGH_TABLE_UNSPEC (NEIGH_LINK_TABLE + 1)
static int zero = 0;
+static int one = 1;
static int label_limit = (1 << 20) - 1;
+static int ttl_max = 255;
static void rtmsg_lfib(int event, u32 label, struct mpls_route *rt,
struct nlmsghdr *nlh, struct net *net, u32 portid,
@@ -58,10 +63,7 @@
static u8 *__mpls_nh_via(struct mpls_route *rt, struct mpls_nh *nh)
{
- u8 *nh0_via = PTR_ALIGN((u8 *)&rt->rt_nh[rt->rt_nhn], VIA_ALEN_ALIGN);
- int nh_index = nh - rt->rt_nh;
-
- return nh0_via + rt->rt_max_alen * nh_index;
+ return (u8 *)nh + rt->rt_via_offset;
}
static const u8 *mpls_nh_via(const struct mpls_route *rt,
@@ -187,21 +189,32 @@
return hash;
}
+static struct mpls_nh *mpls_get_nexthop(struct mpls_route *rt, u8 index)
+{
+ return (struct mpls_nh *)((u8 *)rt->rt_nh + index * rt->rt_nh_size);
+}
+
+/* number of alive nexthops (rt->rt_nhn_alive) and the flags for
+ * a next hop (nh->nh_flags) are modified by netdev event handlers.
+ * Since those fields can change at any moment, use READ_ONCE to
+ * access both.
+ */
static struct mpls_nh *mpls_select_multipath(struct mpls_route *rt,
struct sk_buff *skb)
{
- int alive = ACCESS_ONCE(rt->rt_nhn_alive);
u32 hash = 0;
int nh_index = 0;
int n = 0;
+ u8 alive;
/* No need to look further into packet if there's only
* one path
*/
if (rt->rt_nhn == 1)
- goto out;
+ return rt->rt_nh;
- if (alive <= 0)
+ alive = READ_ONCE(rt->rt_nhn_alive);
+ if (alive == 0)
return NULL;
hash = mpls_multipath_hash(rt, skb);
@@ -209,7 +222,9 @@
if (alive == rt->rt_nhn)
goto out;
for_nexthops(rt) {
- if (nh->nh_flags & (RTNH_F_DEAD | RTNH_F_LINKDOWN))
+ unsigned int nh_flags = READ_ONCE(nh->nh_flags);
+
+ if (nh_flags & (RTNH_F_DEAD | RTNH_F_LINKDOWN))
continue;
if (n == nh_index)
return nh;
@@ -217,11 +232,11 @@
} endfor_nexthops(rt);
out:
- return &rt->rt_nh[nh_index];
+ return mpls_get_nexthop(rt, nh_index);
}
-static bool mpls_egress(struct mpls_route *rt, struct sk_buff *skb,
- struct mpls_entry_decoded dec)
+static bool mpls_egress(struct net *net, struct mpls_route *rt,
+ struct sk_buff *skb, struct mpls_entry_decoded dec)
{
enum mpls_payload_type payload_type;
bool success = false;
@@ -246,22 +261,46 @@
switch (payload_type) {
case MPT_IPV4: {
struct iphdr *hdr4 = ip_hdr(skb);
+ u8 new_ttl;
skb->protocol = htons(ETH_P_IP);
+
+ /* If propagating TTL, take the decremented TTL from
+ * the incoming MPLS header, otherwise decrement the
+ * TTL, but only if not 0 to avoid underflow.
+ */
+ if (rt->rt_ttl_propagate == MPLS_TTL_PROP_ENABLED ||
+ (rt->rt_ttl_propagate == MPLS_TTL_PROP_DEFAULT &&
+ net->mpls.ip_ttl_propagate))
+ new_ttl = dec.ttl;
+ else
+ new_ttl = hdr4->ttl ? hdr4->ttl - 1 : 0;
+
csum_replace2(&hdr4->check,
htons(hdr4->ttl << 8),
- htons(dec.ttl << 8));
- hdr4->ttl = dec.ttl;
+ htons(new_ttl << 8));
+ hdr4->ttl = new_ttl;
success = true;
break;
}
case MPT_IPV6: {
struct ipv6hdr *hdr6 = ipv6_hdr(skb);
skb->protocol = htons(ETH_P_IPV6);
- hdr6->hop_limit = dec.ttl;
+
+ /* If propagating TTL, take the decremented TTL from
+ * the incoming MPLS header, otherwise decrement the
+ * hop limit, but only if not 0 to avoid underflow.
+ */
+ if (rt->rt_ttl_propagate == MPLS_TTL_PROP_ENABLED ||
+ (rt->rt_ttl_propagate == MPLS_TTL_PROP_DEFAULT &&
+ net->mpls.ip_ttl_propagate))
+ hdr6->hop_limit = dec.ttl;
+ else if (hdr6->hop_limit)
+ hdr6->hop_limit = hdr6->hop_limit - 1;
success = true;
break;
}
case MPT_UNSPEC:
+ /* Should have decided which protocol it is by now */
break;
}
@@ -361,7 +400,7 @@
if (unlikely(!new_header_size && dec.bos)) {
/* Penultimate hop popping */
- if (!mpls_egress(rt, skb, dec))
+ if (!mpls_egress(dev_net(out_dev), rt, skb, dec))
goto err;
} else {
bool bos;
@@ -412,6 +451,7 @@
static const struct nla_policy rtm_mpls_policy[RTA_MAX+1] = {
[RTA_DST] = { .type = NLA_U32 },
[RTA_OIF] = { .type = NLA_U32 },
+ [RTA_TTL_PROPAGATE] = { .type = NLA_U8 },
};
struct mpls_route_config {
@@ -421,6 +461,7 @@
u8 rc_via_alen;
u8 rc_via[MAX_VIA_ALEN];
u32 rc_label;
+ u8 rc_ttl_propagate;
u8 rc_output_labels;
u32 rc_output_label[MAX_NEW_LABELS];
u32 rc_nlflags;
@@ -430,20 +471,27 @@
int rc_mp_len;
};
-static struct mpls_route *mpls_rt_alloc(int num_nh, u8 max_alen)
+/* all nexthops within a route have the same size based on max
+ * number of labels and max via length for a hop
+ */
+static struct mpls_route *mpls_rt_alloc(u8 num_nh, u8 max_alen, u8 max_labels)
{
- u8 max_alen_aligned = ALIGN(max_alen, VIA_ALEN_ALIGN);
+ u8 nh_size = MPLS_NH_SIZE(max_labels, max_alen);
struct mpls_route *rt;
+ size_t size;
- rt = kzalloc(ALIGN(sizeof(*rt) + num_nh * sizeof(*rt->rt_nh),
- VIA_ALEN_ALIGN) +
- num_nh * max_alen_aligned,
- GFP_KERNEL);
- if (rt) {
- rt->rt_nhn = num_nh;
- rt->rt_nhn_alive = num_nh;
- rt->rt_max_alen = max_alen_aligned;
- }
+ size = sizeof(*rt) + num_nh * nh_size;
+ if (size > MAX_MPLS_ROUTE_MEM)
+ return ERR_PTR(-EINVAL);
+
+ rt = kzalloc(size, GFP_KERNEL);
+ if (!rt)
+ return ERR_PTR(-ENOMEM);
+
+ rt->rt_nhn = num_nh;
+ rt->rt_nhn_alive = num_nh;
+ rt->rt_nh_size = nh_size;
+ rt->rt_via_offset = MPLS_NH_VIA_OFF(max_labels);
return rt;
}
@@ -648,9 +696,6 @@
return -ENOMEM;
err = -EINVAL;
- /* Ensure only a supported number of labels are present */
- if (cfg->rc_output_labels > MAX_NEW_LABELS)
- goto errout;
nh->nh_labels = cfg->rc_output_labels;
for (i = 0; i < nh->nh_labels; i++)
@@ -675,7 +720,7 @@
static int mpls_nh_build(struct net *net, struct mpls_route *rt,
struct mpls_nh *nh, int oif, struct nlattr *via,
- struct nlattr *newdst)
+ struct nlattr *newdst, u8 max_labels)
{
int err = -ENOMEM;
@@ -683,7 +728,7 @@
goto errout;
if (newdst) {
- err = nla_get_labels(newdst, MAX_NEW_LABELS,
+ err = nla_get_labels(newdst, max_labels,
&nh->nh_labels, nh->nh_label);
if (err)
goto errout;
@@ -708,22 +753,20 @@
return err;
}
-static int mpls_count_nexthops(struct rtnexthop *rtnh, int len,
- u8 cfg_via_alen, u8 *max_via_alen)
+static u8 mpls_count_nexthops(struct rtnexthop *rtnh, int len,
+ u8 cfg_via_alen, u8 *max_via_alen,
+ u8 *max_labels)
{
- int nhs = 0;
int remaining = len;
-
- if (!rtnh) {
- *max_via_alen = cfg_via_alen;
- return 1;
- }
+ u8 nhs = 0;
*max_via_alen = 0;
+ *max_labels = 0;
while (rtnh_ok(rtnh, remaining)) {
struct nlattr *nla, *attrs = rtnh_attrs(rtnh);
int attrlen;
+ u8 n_labels = 0;
attrlen = rtnh_attrlen(rtnh);
nla = nla_find(attrs, attrlen, RTA_VIA);
@@ -737,7 +780,20 @@
via_alen);
}
+ nla = nla_find(attrs, attrlen, RTA_NEWDST);
+ if (nla &&
+ nla_get_labels(nla, MAX_NEW_LABELS, &n_labels, NULL) != 0)
+ return 0;
+
+ *max_labels = max_t(u8, *max_labels, n_labels);
+
+ /* number of nexthops is tracked by a u8.
+ * Check for overflow.
+ */
+ if (nhs == 255)
+ return 0;
nhs++;
+
rtnh = rtnh_next(rtnh, &remaining);
}
@@ -746,13 +802,13 @@
}
static int mpls_nh_build_multi(struct mpls_route_config *cfg,
- struct mpls_route *rt)
+ struct mpls_route *rt, u8 max_labels)
{
struct rtnexthop *rtnh = cfg->rc_mp;
struct nlattr *nla_via, *nla_newdst;
int remaining = cfg->rc_mp_len;
- int nhs = 0;
int err = 0;
+ u8 nhs = 0;
change_nexthops(rt) {
int attrlen;
@@ -779,7 +835,8 @@
}
err = mpls_nh_build(cfg->rc_nlinfo.nl_net, rt, nh,
- rtnh->rtnh_ifindex, nla_via, nla_newdst);
+ rtnh->rtnh_ifindex, nla_via, nla_newdst,
+ max_labels);
if (err)
goto errout;
@@ -806,7 +863,8 @@
int err = -EINVAL;
u8 max_via_alen;
unsigned index;
- int nhs;
+ u8 max_labels;
+ u8 nhs;
index = cfg->rc_label;
@@ -844,21 +902,32 @@
goto errout;
err = -EINVAL;
- nhs = mpls_count_nexthops(cfg->rc_mp, cfg->rc_mp_len,
- cfg->rc_via_alen, &max_via_alen);
+ if (cfg->rc_mp) {
+ nhs = mpls_count_nexthops(cfg->rc_mp, cfg->rc_mp_len,
+ cfg->rc_via_alen, &max_via_alen,
+ &max_labels);
+ } else {
+ max_via_alen = cfg->rc_via_alen;
+ max_labels = cfg->rc_output_labels;
+ nhs = 1;
+ }
+
if (nhs == 0)
goto errout;
err = -ENOMEM;
- rt = mpls_rt_alloc(nhs, max_via_alen);
- if (!rt)
+ rt = mpls_rt_alloc(nhs, max_via_alen, max_labels);
+ if (IS_ERR(rt)) {
+ err = PTR_ERR(rt);
goto errout;
+ }
rt->rt_protocol = cfg->rc_protocol;
rt->rt_payload_type = cfg->rc_payload_type;
+ rt->rt_ttl_propagate = cfg->rc_ttl_propagate;
if (cfg->rc_mp)
- err = mpls_nh_build_multi(cfg, rt);
+ err = mpls_nh_build_multi(cfg, rt, max_labels);
else
err = mpls_nh_build_from_cfg(cfg, rt);
if (err)
@@ -1011,8 +1080,8 @@
return size;
}
-static void mpls_netconf_notify_devconf(struct net *net, int type,
- struct mpls_dev *mdev)
+static void mpls_netconf_notify_devconf(struct net *net, int event,
+ int type, struct mpls_dev *mdev)
{
struct sk_buff *skb;
int err = -ENOBUFS;
@@ -1021,8 +1090,7 @@
if (!skb)
goto errout;
- err = mpls_netconf_fill_devconf(skb, mdev, 0, 0, RTM_NEWNETCONF,
- 0, type);
+ err = mpls_netconf_fill_devconf(skb, mdev, 0, 0, event, 0, type);
if (err < 0) {
/* -EMSGSIZE implies BUG in mpls_netconf_msgsize_devconf() */
WARN_ON(err == -EMSGSIZE);
@@ -1042,7 +1110,8 @@
};
static int mpls_netconf_get_devconf(struct sk_buff *in_skb,
- struct nlmsghdr *nlh)
+ struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(in_skb->sk);
struct nlattr *tb[NETCONFA_MAX + 1];
@@ -1054,7 +1123,7 @@
int err;
err = nlmsg_parse(nlh, sizeof(*ncm), tb, NETCONFA_MAX,
- devconf_mpls_policy);
+ devconf_mpls_policy, NULL);
if (err < 0)
goto errout;
@@ -1155,9 +1224,8 @@
if (i == offsetof(struct mpls_dev, input_enabled) &&
val != oval) {
- mpls_netconf_notify_devconf(net,
- NETCONFA_INPUT,
- mdev);
+ mpls_netconf_notify_devconf(net, RTM_NEWNETCONF,
+ NETCONFA_INPUT, mdev);
}
}
@@ -1198,10 +1266,11 @@
snprintf(path, sizeof(path), "net/mpls/conf/%s", dev->name);
- mdev->sysctl = register_net_sysctl(dev_net(dev), path, table);
+ mdev->sysctl = register_net_sysctl(net, path, table);
if (!mdev->sysctl)
goto free;
+ mpls_netconf_notify_devconf(net, RTM_NEWNETCONF, NETCONFA_ALL, mdev);
return 0;
free:
@@ -1210,13 +1279,17 @@
return -ENOBUFS;
}
-static void mpls_dev_sysctl_unregister(struct mpls_dev *mdev)
+static void mpls_dev_sysctl_unregister(struct net_device *dev,
+ struct mpls_dev *mdev)
{
+ struct net *net = dev_net(dev);
struct ctl_table *table;
table = mdev->sysctl->ctl_table_arg;
unregister_net_sysctl_table(mdev->sysctl);
kfree(table);
+
+ mpls_netconf_notify_devconf(net, RTM_DELNETCONF, 0, mdev);
}
static struct mpls_dev *mpls_add_dev(struct net_device *dev)
@@ -1242,11 +1315,12 @@
u64_stats_init(&mpls_stats->syncp);
}
+ mdev->dev = dev;
+
err = mpls_dev_sysctl_register(dev, mdev);
if (err)
goto free;
- mdev->dev = dev;
rcu_assign_pointer(dev->mpls_ptr, mdev);
return mdev;
@@ -1269,8 +1343,7 @@
{
struct mpls_route __rcu **platform_label;
struct net *net = dev_net(dev);
- unsigned int nh_flags = RTNH_F_DEAD | RTNH_F_LINKDOWN;
- unsigned int alive;
+ u8 alive, deleted;
unsigned index;
platform_label = rtnl_dereference(net->mpls.platform_label);
@@ -1281,36 +1354,48 @@
continue;
alive = 0;
+ deleted = 0;
change_nexthops(rt) {
+ unsigned int nh_flags = nh->nh_flags;
+
if (rtnl_dereference(nh->nh_dev) != dev)
goto next;
switch (event) {
case NETDEV_DOWN:
case NETDEV_UNREGISTER:
- nh->nh_flags |= RTNH_F_DEAD;
+ nh_flags |= RTNH_F_DEAD;
/* fall through */
case NETDEV_CHANGE:
- nh->nh_flags |= RTNH_F_LINKDOWN;
+ nh_flags |= RTNH_F_LINKDOWN;
break;
}
if (event == NETDEV_UNREGISTER)
RCU_INIT_POINTER(nh->nh_dev, NULL);
+
+ if (nh->nh_flags != nh_flags)
+ WRITE_ONCE(nh->nh_flags, nh_flags);
next:
- if (!(nh->nh_flags & nh_flags))
+ if (!(nh_flags & (RTNH_F_DEAD | RTNH_F_LINKDOWN)))
alive++;
+ if (!rtnl_dereference(nh->nh_dev))
+ deleted++;
} endfor_nexthops(rt);
WRITE_ONCE(rt->rt_nhn_alive, alive);
+
+ /* if there are no more nexthops, delete the route */
+ if (event == NETDEV_UNREGISTER && deleted == rt->rt_nhn)
+ mpls_route_update(net, index, NULL, NULL);
}
}
-static void mpls_ifup(struct net_device *dev, unsigned int nh_flags)
+static void mpls_ifup(struct net_device *dev, unsigned int flags)
{
struct mpls_route __rcu **platform_label;
struct net *net = dev_net(dev);
unsigned index;
- int alive;
+ u8 alive;
platform_label = rtnl_dereference(net->mpls.platform_label);
for (index = 0; index < net->mpls.platform_labels; index++) {
@@ -1321,20 +1406,22 @@
alive = 0;
change_nexthops(rt) {
+ unsigned int nh_flags = nh->nh_flags;
struct net_device *nh_dev =
rtnl_dereference(nh->nh_dev);
- if (!(nh->nh_flags & nh_flags)) {
+ if (!(nh_flags & flags)) {
alive++;
continue;
}
if (nh_dev != dev)
continue;
alive++;
- nh->nh_flags &= ~nh_flags;
+ nh_flags &= ~flags;
+ WRITE_ONCE(nh->nh_flags, flags);
} endfor_nexthops(rt);
- ACCESS_ONCE(rt->rt_nhn_alive) = alive;
+ WRITE_ONCE(rt->rt_nhn_alive, alive);
}
}
@@ -1385,7 +1472,7 @@
mpls_ifdown(dev, event);
mdev = mpls_dev_get(dev);
if (mdev) {
- mpls_dev_sysctl_unregister(mdev);
+ mpls_dev_sysctl_unregister(dev, mdev);
RCU_INIT_POINTER(dev->mpls_ptr, NULL);
call_rcu(&mdev->rcu, mpls_dev_destroy_rcu);
}
@@ -1395,7 +1482,7 @@
if (mdev) {
int err;
- mpls_dev_sysctl_unregister(mdev);
+ mpls_dev_sysctl_unregister(dev, mdev);
err = mpls_dev_sysctl_register(dev, mdev);
if (err)
return notifier_from_errno(err);
@@ -1455,16 +1542,18 @@
EXPORT_SYMBOL_GPL(nla_put_labels);
int nla_get_labels(const struct nlattr *nla,
- u32 max_labels, u8 *labels, u32 label[])
+ u8 max_labels, u8 *labels, u32 label[])
{
unsigned len = nla_len(nla);
- unsigned nla_labels;
struct mpls_shim_hdr *nla_label;
+ u8 nla_labels;
bool bos;
int i;
- /* len needs to be an even multiple of 4 (the label size) */
- if (len & 3)
+ /* len needs to be an even multiple of 4 (the label size). Number
+ * of labels is a u8 so check for overflow.
+ */
+ if (len & 3 || len / 4 > 255)
return -EINVAL;
/* Limit the number of new labels allowed */
@@ -1472,6 +1561,10 @@
if (nla_labels > max_labels)
return -EINVAL;
+ /* when label == NULL, caller wants number of labels */
+ if (!label)
+ goto out;
+
nla_label = nla_data(nla);
bos = true;
for (i = nla_labels - 1; i >= 0; i--, bos = false) {
@@ -1495,6 +1588,7 @@
label[i] = dec.label;
}
+out:
*labels = nla_labels;
return 0;
}
@@ -1550,13 +1644,13 @@
int index;
int err;
- err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_mpls_policy);
+ err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_mpls_policy,
+ NULL);
if (err < 0)
goto errout;
err = -EINVAL;
rtm = nlmsg_data(nlh);
- memset(cfg, 0, sizeof(*cfg));
if (rtm->rtm_family != AF_MPLS)
goto errout;
@@ -1584,6 +1678,7 @@
cfg->rc_label = LABEL_NOT_SPECIFIED;
cfg->rc_protocol = rtm->rtm_protocol;
cfg->rc_via_table = MPLS_NEIGH_TABLE_UNSPEC;
+ cfg->rc_ttl_propagate = MPLS_TTL_PROP_DEFAULT;
cfg->rc_nlflags = nlh->nlmsg_flags;
cfg->rc_nlinfo.portid = NETLINK_CB(skb).portid;
cfg->rc_nlinfo.nlh = nlh;
@@ -1630,6 +1725,17 @@
cfg->rc_mp_len = nla_len(nla);
break;
}
+ case RTA_TTL_PROPAGATE:
+ {
+ u8 ttl_propagate = nla_get_u8(nla);
+
+ if (ttl_propagate > 1)
+ goto errout;
+ cfg->rc_ttl_propagate = ttl_propagate ?
+ MPLS_TTL_PROP_ENABLED :
+ MPLS_TTL_PROP_DISABLED;
+ break;
+ }
default:
/* Unsupported attribute */
goto errout;
@@ -1641,29 +1747,47 @@
return err;
}
-static int mpls_rtm_delroute(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int mpls_rtm_delroute(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
- struct mpls_route_config cfg;
+ struct mpls_route_config *cfg;
int err;
- err = rtm_to_route_config(skb, nlh, &cfg);
- if (err < 0)
- return err;
+ cfg = kzalloc(sizeof(*cfg), GFP_KERNEL);
+ if (!cfg)
+ return -ENOMEM;
- return mpls_route_del(&cfg);
+ err = rtm_to_route_config(skb, nlh, cfg);
+ if (err < 0)
+ goto out;
+
+ err = mpls_route_del(cfg);
+out:
+ kfree(cfg);
+
+ return err;
}
-static int mpls_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int mpls_rtm_newroute(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
- struct mpls_route_config cfg;
+ struct mpls_route_config *cfg;
int err;
- err = rtm_to_route_config(skb, nlh, &cfg);
- if (err < 0)
- return err;
+ cfg = kzalloc(sizeof(*cfg), GFP_KERNEL);
+ if (!cfg)
+ return -ENOMEM;
- return mpls_route_add(&cfg);
+ err = rtm_to_route_config(skb, nlh, cfg);
+ if (err < 0)
+ goto out;
+
+ err = mpls_route_add(cfg);
+out:
+ kfree(cfg);
+
+ return err;
}
static int mpls_dump_route(struct sk_buff *skb, u32 portid, u32 seq, int event,
@@ -1690,6 +1814,15 @@
if (nla_put_labels(skb, RTA_DST, 1, &label))
goto nla_put_failure;
+
+ if (rt->rt_ttl_propagate != MPLS_TTL_PROP_DEFAULT) {
+ bool ttl_propagate =
+ rt->rt_ttl_propagate == MPLS_TTL_PROP_ENABLED;
+
+ if (nla_put_u8(skb, RTA_TTL_PROPAGATE,
+ ttl_propagate))
+ goto nla_put_failure;
+ }
if (rt->rt_nhn == 1) {
const struct mpls_nh *nh = rt->rt_nh;
@@ -1711,21 +1844,23 @@
} else {
struct rtnexthop *rtnh;
struct nlattr *mp;
- int dead = 0;
- int linkdown = 0;
+ u8 linkdown = 0;
+ u8 dead = 0;
mp = nla_nest_start(skb, RTA_MULTIPATH);
if (!mp)
goto nla_put_failure;
for_nexthops(rt) {
+ dev = rtnl_dereference(nh->nh_dev);
+ if (!dev)
+ continue;
+
rtnh = nla_reserve_nohdr(skb, sizeof(*rtnh));
if (!rtnh)
goto nla_put_failure;
- dev = rtnl_dereference(nh->nh_dev);
- if (dev)
- rtnh->rtnh_ifindex = dev->ifindex;
+ rtnh->rtnh_ifindex = dev->ifindex;
if (nh->nh_flags & RTNH_F_LINKDOWN) {
rtnh->rtnh_flags |= RTNH_F_LINKDOWN;
linkdown++;
@@ -1800,7 +1935,8 @@
{
size_t payload =
NLMSG_ALIGN(sizeof(struct rtmsg))
- + nla_total_size(4); /* RTA_DST */
+ + nla_total_size(4) /* RTA_DST */
+ + nla_total_size(1); /* RTA_TTL_PROPAGATE */
if (rt->rt_nhn == 1) {
struct mpls_nh *nh = rt->rt_nh;
@@ -1816,6 +1952,8 @@
size_t nhsize = 0;
for_nexthops(rt) {
+ if (!rtnl_dereference(nh->nh_dev))
+ continue;
nhsize += nla_total_size(sizeof(struct rtnexthop));
/* RTA_VIA */
if (nh->nh_via_table != MPLS_NEIGH_TABLE_UNSPEC)
@@ -1878,12 +2016,13 @@
/* In case the predefined labels need to be populated */
if (limit > MPLS_LABEL_IPV4NULL) {
struct net_device *lo = net->loopback_dev;
- rt0 = mpls_rt_alloc(1, lo->addr_len);
- if (!rt0)
+ rt0 = mpls_rt_alloc(1, lo->addr_len, 0);
+ if (IS_ERR(rt0))
goto nort0;
RCU_INIT_POINTER(rt0->rt_nh->nh_dev, lo);
rt0->rt_protocol = RTPROT_KERNEL;
rt0->rt_payload_type = MPT_IPV4;
+ rt0->rt_ttl_propagate = MPLS_TTL_PROP_DEFAULT;
rt0->rt_nh->nh_via_table = NEIGH_LINK_TABLE;
rt0->rt_nh->nh_via_alen = lo->addr_len;
memcpy(__mpls_nh_via(rt0, rt0->rt_nh), lo->dev_addr,
@@ -1891,12 +2030,13 @@
}
if (limit > MPLS_LABEL_IPV6NULL) {
struct net_device *lo = net->loopback_dev;
- rt2 = mpls_rt_alloc(1, lo->addr_len);
- if (!rt2)
+ rt2 = mpls_rt_alloc(1, lo->addr_len, 0);
+ if (IS_ERR(rt2))
goto nort2;
RCU_INIT_POINTER(rt2->rt_nh->nh_dev, lo);
rt2->rt_protocol = RTPROT_KERNEL;
rt2->rt_payload_type = MPT_IPV6;
+ rt2->rt_ttl_propagate = MPLS_TTL_PROP_DEFAULT;
rt2->rt_nh->nh_via_table = NEIGH_LINK_TABLE;
rt2->rt_nh->nh_via_alen = lo->addr_len;
memcpy(__mpls_nh_via(rt2, rt2->rt_nh), lo->dev_addr,
@@ -1978,6 +2118,9 @@
return ret;
}
+#define MPLS_NS_SYSCTL_OFFSET(field) \
+ (&((struct net *)0)->field)
+
static const struct ctl_table mpls_table[] = {
{
.procname = "platform_labels",
@@ -1986,21 +2129,47 @@
.mode = 0644,
.proc_handler = mpls_platform_labels,
},
+ {
+ .procname = "ip_ttl_propagate",
+ .data = MPLS_NS_SYSCTL_OFFSET(mpls.ip_ttl_propagate),
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = &zero,
+ .extra2 = &one,
+ },
+ {
+ .procname = "default_ttl",
+ .data = MPLS_NS_SYSCTL_OFFSET(mpls.default_ttl),
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = &one,
+ .extra2 = &ttl_max,
+ },
{ }
};
static int mpls_net_init(struct net *net)
{
struct ctl_table *table;
+ int i;
net->mpls.platform_labels = 0;
net->mpls.platform_label = NULL;
+ net->mpls.ip_ttl_propagate = 1;
+ net->mpls.default_ttl = 255;
table = kmemdup(mpls_table, sizeof(mpls_table), GFP_KERNEL);
if (table == NULL)
return -ENOMEM;
- table[0].data = net;
+ /* Table data contains only offsets relative to the base of
+ * the mdev at this point, so make them absolute.
+ */
+ for (i = 0; i < ARRAY_SIZE(mpls_table) - 1; i++)
+ table[i].data = (char *)net + (uintptr_t)table[i].data;
+
net->mpls.ctl = register_net_sysctl(net, "net/mpls", table);
if (net->mpls.ctl == NULL) {
kfree(table);
diff --git a/net/mpls/internal.h b/net/mpls/internal.h
index 76360d8..4db6a59 100644
--- a/net/mpls/internal.h
+++ b/net/mpls/internal.h
@@ -2,6 +2,11 @@
#define MPLS_INTERNAL_H
#include <net/mpls.h>
+/* put a reasonable limit on the number of labels
+ * we will accept from userspace
+ */
+#define MAX_NEW_LABELS 30
+
struct mpls_entry_decoded {
u32 label;
u8 ttl;
@@ -64,7 +69,6 @@
struct sk_buff;
#define LABEL_NOT_SPECIFIED (1 << 20)
-#define MAX_NEW_LABELS 2
/* This maximum ha length copied from the definition of struct neighbour */
#define VIA_ALEN_ALIGN sizeof(unsigned long)
@@ -83,11 +87,35 @@
struct mpls_nh { /* next hop label forwarding entry */
struct net_device __rcu *nh_dev;
+
+ /* nh_flags is accessed under RCU in the packet path; it is
+ * modified handling netdev events with rtnl lock held
+ */
unsigned int nh_flags;
- u32 nh_label[MAX_NEW_LABELS];
u8 nh_labels;
u8 nh_via_alen;
u8 nh_via_table;
+ u8 nh_reserved1;
+
+ u32 nh_label[0];
+};
+
+/* offset of via from beginning of mpls_nh */
+#define MPLS_NH_VIA_OFF(num_labels) \
+ ALIGN(sizeof(struct mpls_nh) + (num_labels) * sizeof(u32), \
+ VIA_ALEN_ALIGN)
+
+/* all nexthops within a route have the same size based on the
+ * max number of labels and max via length across all nexthops
+ */
+#define MPLS_NH_SIZE(num_labels, max_via_alen) \
+ (MPLS_NH_VIA_OFF((num_labels)) + \
+ ALIGN((max_via_alen), VIA_ALEN_ALIGN))
+
+enum mpls_ttl_propagation {
+ MPLS_TTL_PROP_DEFAULT,
+ MPLS_TTL_PROP_ENABLED,
+ MPLS_TTL_PROP_DISABLED,
};
/* The route, nexthops and vias are stored together in the same memory
@@ -98,16 +126,16 @@
* +----------------------+
* | mpls_nh 0 |
* +----------------------+
- * | ... |
- * +----------------------+
- * | mpls_nh n-1 |
- * +----------------------+
- * | alignment padding |
+ * | alignment padding | 4 bytes for odd number of labels
* +----------------------+
* | via[rt_max_alen] 0 |
* +----------------------+
+ * | alignment padding | via's aligned on sizeof(unsigned long)
+ * +----------------------+
* | ... |
* +----------------------+
+ * | mpls_nh n-1 |
+ * +----------------------+
* | via[rt_max_alen] n-1 |
* +----------------------+
*/
@@ -116,22 +144,30 @@
u8 rt_protocol;
u8 rt_payload_type;
u8 rt_max_alen;
- unsigned int rt_nhn;
- unsigned int rt_nhn_alive;
+ u8 rt_ttl_propagate;
+ u8 rt_nhn;
+ /* rt_nhn_alive is accessed under RCU in the packet path; it
+ * is modified handling netdev events with rtnl lock held
+ */
+ u8 rt_nhn_alive;
+ u8 rt_nh_size;
+ u8 rt_via_offset;
+ u8 rt_reserved1;
struct mpls_nh rt_nh[0];
};
#define for_nexthops(rt) { \
- int nhsel; struct mpls_nh *nh; \
- for (nhsel = 0, nh = (rt)->rt_nh; \
+ int nhsel; struct mpls_nh *nh; u8 *__nh; \
+ for (nhsel = 0, nh = (rt)->rt_nh, __nh = (u8 *)((rt)->rt_nh); \
nhsel < (rt)->rt_nhn; \
- nh++, nhsel++)
+ __nh += rt->rt_nh_size, nh = (struct mpls_nh *)__nh, nhsel++)
#define change_nexthops(rt) { \
- int nhsel; struct mpls_nh *nh; \
- for (nhsel = 0, nh = (struct mpls_nh *)((rt)->rt_nh); \
+ int nhsel; struct mpls_nh *nh; u8 *__nh; \
+ for (nhsel = 0, nh = (struct mpls_nh *)((rt)->rt_nh), \
+ __nh = (u8 *)((rt)->rt_nh); \
nhsel < (rt)->rt_nhn; \
- nh++, nhsel++)
+ __nh += rt->rt_nh_size, nh = (struct mpls_nh *)__nh, nhsel++)
#define endfor_nexthops(rt) }
@@ -166,7 +202,7 @@
int nla_put_labels(struct sk_buff *skb, int attrtype, u8 labels,
const u32 label[]);
-int nla_get_labels(const struct nlattr *nla, u32 max_labels, u8 *labels,
+int nla_get_labels(const struct nlattr *nla, u8 max_labels, u8 *labels,
u32 label[]);
int nla_get_via(const struct nlattr *nla, u8 *via_alen, u8 *via_table,
u8 via[]);
diff --git a/net/mpls/mpls_iptunnel.c b/net/mpls/mpls_iptunnel.c
index e4e4424..369c7a2 100644
--- a/net/mpls/mpls_iptunnel.c
+++ b/net/mpls/mpls_iptunnel.c
@@ -29,6 +29,7 @@
static const struct nla_policy mpls_iptunnel_policy[MPLS_IPTUNNEL_MAX + 1] = {
[MPLS_IPTUNNEL_DST] = { .type = NLA_U32 },
+ [MPLS_IPTUNNEL_TTL] = { .type = NLA_U8 },
};
static unsigned int mpls_encap_size(struct mpls_iptunnel_encap *en)
@@ -49,6 +50,7 @@
struct rtable *rt = NULL;
struct rt6_info *rt6 = NULL;
struct mpls_dev *out_mdev;
+ struct net *net;
int err = 0;
bool bos;
int i;
@@ -56,17 +58,7 @@
/* Find the output device */
out_dev = dst->dev;
-
- /* Obtain the ttl */
- if (dst->ops->family == AF_INET) {
- ttl = ip_hdr(skb)->ttl;
- rt = (struct rtable *)dst;
- } else if (dst->ops->family == AF_INET6) {
- ttl = ipv6_hdr(skb)->hop_limit;
- rt6 = (struct rt6_info *)dst;
- } else {
- goto drop;
- }
+ net = dev_net(out_dev);
skb_orphan(skb);
@@ -78,6 +70,38 @@
tun_encap_info = mpls_lwtunnel_encap(dst->lwtstate);
+ /* Obtain the ttl using the following set of rules.
+ *
+ * LWT ttl propagation setting:
+ * - disabled => use default TTL value from LWT
+ * - enabled => use TTL value from IPv4/IPv6 header
+ * - default =>
+ * Global ttl propagation setting:
+ * - disabled => use default TTL value from global setting
+ * - enabled => use TTL value from IPv4/IPv6 header
+ */
+ if (dst->ops->family == AF_INET) {
+ if (tun_encap_info->ttl_propagate == MPLS_TTL_PROP_DISABLED)
+ ttl = tun_encap_info->default_ttl;
+ else if (tun_encap_info->ttl_propagate == MPLS_TTL_PROP_DEFAULT &&
+ !net->mpls.ip_ttl_propagate)
+ ttl = net->mpls.default_ttl;
+ else
+ ttl = ip_hdr(skb)->ttl;
+ rt = (struct rtable *)dst;
+ } else if (dst->ops->family == AF_INET6) {
+ if (tun_encap_info->ttl_propagate == MPLS_TTL_PROP_DISABLED)
+ ttl = tun_encap_info->default_ttl;
+ else if (tun_encap_info->ttl_propagate == MPLS_TTL_PROP_DEFAULT &&
+ !net->mpls.ip_ttl_propagate)
+ ttl = net->mpls.default_ttl;
+ else
+ ttl = ipv6_hdr(skb)->hop_limit;
+ rt6 = (struct rt6_info *)dst;
+ } else {
+ goto drop;
+ }
+
/* Verify the destination can hold the packet */
new_header_size = mpls_encap_size(tun_encap_info);
mtu = mpls_dev_mtu(out_dev);
@@ -140,10 +164,11 @@
struct mpls_iptunnel_encap *tun_encap_info;
struct nlattr *tb[MPLS_IPTUNNEL_MAX + 1];
struct lwtunnel_state *newts;
+ u8 n_labels;
int ret;
ret = nla_parse_nested(tb, MPLS_IPTUNNEL_MAX, nla,
- mpls_iptunnel_policy);
+ mpls_iptunnel_policy, NULL);
if (ret < 0)
return ret;
@@ -151,15 +176,32 @@
return -EINVAL;
- newts = lwtunnel_state_alloc(sizeof(*tun_encap_info));
+ /* determine number of labels */
+ if (nla_get_labels(tb[MPLS_IPTUNNEL_DST],
+ MAX_NEW_LABELS, &n_labels, NULL))
+ return -EINVAL;
+
+ newts = lwtunnel_state_alloc(sizeof(*tun_encap_info) +
+ n_labels * sizeof(u32));
if (!newts)
return -ENOMEM;
tun_encap_info = mpls_lwtunnel_encap(newts);
- ret = nla_get_labels(tb[MPLS_IPTUNNEL_DST], MAX_NEW_LABELS,
+ ret = nla_get_labels(tb[MPLS_IPTUNNEL_DST], n_labels,
&tun_encap_info->labels, tun_encap_info->label);
if (ret)
goto errout;
+
+ tun_encap_info->ttl_propagate = MPLS_TTL_PROP_DEFAULT;
+
+ if (tb[MPLS_IPTUNNEL_TTL]) {
+ tun_encap_info->default_ttl = nla_get_u8(tb[MPLS_IPTUNNEL_TTL]);
+ /* TTL 0 implies propagate from IP header */
+ tun_encap_info->ttl_propagate = tun_encap_info->default_ttl ?
+ MPLS_TTL_PROP_DISABLED :
+ MPLS_TTL_PROP_ENABLED;
+ }
+
newts->type = LWTUNNEL_ENCAP_MPLS;
newts->flags |= LWTUNNEL_STATE_XMIT_REDIRECT;
newts->headroom = mpls_encap_size(tun_encap_info);
@@ -186,6 +228,10 @@
tun_encap_info->label))
goto nla_put_failure;
+ if (tun_encap_info->ttl_propagate != MPLS_TTL_PROP_DEFAULT &&
+ nla_put_u8(skb, MPLS_IPTUNNEL_TTL, tun_encap_info->default_ttl))
+ goto nla_put_failure;
+
return 0;
nla_put_failure:
@@ -195,10 +241,16 @@
static int mpls_encap_nlsize(struct lwtunnel_state *lwtstate)
{
struct mpls_iptunnel_encap *tun_encap_info;
+ int nlsize;
tun_encap_info = mpls_lwtunnel_encap(lwtstate);
- return nla_total_size(tun_encap_info->labels * 4);
+ nlsize = nla_total_size(tun_encap_info->labels * 4);
+
+ if (tun_encap_info->ttl_propagate != MPLS_TTL_PROP_DEFAULT)
+ nlsize += nla_total_size(1);
+
+ return nlsize;
}
static int mpls_encap_cmp(struct lwtunnel_state *a, struct lwtunnel_state *b)
@@ -207,10 +259,12 @@
struct mpls_iptunnel_encap *b_hdr = mpls_lwtunnel_encap(b);
int l;
- if (a_hdr->labels != b_hdr->labels)
+ if (a_hdr->labels != b_hdr->labels ||
+ a_hdr->ttl_propagate != b_hdr->ttl_propagate ||
+ a_hdr->default_ttl != b_hdr->default_ttl)
return 1;
- for (l = 0; l < MAX_NEW_LABELS; l++)
+ for (l = 0; l < a_hdr->labels; l++)
if (a_hdr->label[l] != b_hdr->label[l])
return 1;
return 0;
diff --git a/net/netfilter/core.c b/net/netfilter/core.c
index a87a6f8..552d606 100644
--- a/net/netfilter/core.c
+++ b/net/netfilter/core.c
@@ -126,14 +126,15 @@
}
EXPORT_SYMBOL(nf_register_net_hook);
-void nf_unregister_net_hook(struct net *net, const struct nf_hook_ops *reg)
+static struct nf_hook_entry *
+__nf_unregister_net_hook(struct net *net, const struct nf_hook_ops *reg)
{
struct nf_hook_entry __rcu **pp;
struct nf_hook_entry *p;
pp = nf_hook_entry_head(net, reg);
if (WARN_ON_ONCE(!pp))
- return;
+ return NULL;
mutex_lock(&nf_hook_mutex);
for (; (p = nf_entry_dereference(*pp)) != NULL; pp = &p->next) {
@@ -145,7 +146,7 @@
mutex_unlock(&nf_hook_mutex);
if (!p) {
WARN(1, "nf_unregister_net_hook: hook not found!\n");
- return;
+ return NULL;
}
#ifdef CONFIG_NETFILTER_INGRESS
if (reg->pf == NFPROTO_NETDEV && reg->hooknum == NF_NETDEV_INGRESS)
@@ -154,10 +155,24 @@
#ifdef HAVE_JUMP_LABEL
static_key_slow_dec(&nf_hooks_needed[reg->pf][reg->hooknum]);
#endif
+
+ return p;
+}
+
+void nf_unregister_net_hook(struct net *net, const struct nf_hook_ops *reg)
+{
+ struct nf_hook_entry *p = __nf_unregister_net_hook(net, reg);
+ unsigned int nfq;
+
+ if (!p)
+ return;
+
synchronize_net();
- nf_queue_nf_hook_drop(net, p);
+
/* other cpu might still process nfqueue verdict that used reg */
- synchronize_net();
+ nfq = nf_queue_nf_hook_drop(net);
+ if (nfq)
+ synchronize_net();
kfree(p);
}
EXPORT_SYMBOL(nf_unregister_net_hook);
@@ -183,10 +198,32 @@
EXPORT_SYMBOL(nf_register_net_hooks);
void nf_unregister_net_hooks(struct net *net, const struct nf_hook_ops *reg,
- unsigned int n)
+ unsigned int hookcount)
{
- while (n-- > 0)
- nf_unregister_net_hook(net, ®[n]);
+ struct nf_hook_entry *to_free[16];
+ unsigned int i, n, nfq;
+
+ do {
+ n = min_t(unsigned int, hookcount, ARRAY_SIZE(to_free));
+
+ for (i = 0; i < n; i++)
+ to_free[i] = __nf_unregister_net_hook(net, ®[i]);
+
+ synchronize_net();
+
+ /* need 2nd synchronize_net() if nfqueue is used, skb
+ * can get reinjected right before nf_queue_hook_drop()
+ */
+ nfq = nf_queue_nf_hook_drop(net);
+ if (nfq)
+ synchronize_net();
+
+ for (i = 0; i < n; i++)
+ kfree(to_free[i]);
+
+ reg += n;
+ hookcount -= n;
+ } while (hookcount > 0);
}
EXPORT_SYMBOL(nf_unregister_net_hooks);
diff --git a/net/netfilter/ipset/ip_set_bitmap_gen.h b/net/netfilter/ipset/ip_set_bitmap_gen.h
index 6f09a99..8ad2b52 100644
--- a/net/netfilter/ipset/ip_set_bitmap_gen.h
+++ b/net/netfilter/ipset/ip_set_bitmap_gen.h
@@ -232,7 +232,7 @@
if (!test_bit(id, map->members) ||
(SET_WITH_TIMEOUT(set) &&
#ifdef IP_SET_BITMAP_STORED_TIMEOUT
- mtype_is_filled((const struct mtype_elem *)x) &&
+ mtype_is_filled(x) &&
#endif
ip_set_timeout_expired(ext_timeout(x, set))))
continue;
@@ -248,8 +248,7 @@
}
if (mtype_do_list(skb, map, id, set->dsize))
goto nla_put_failure;
- if (ip_set_put_extensions(skb, set, x,
- mtype_is_filled((const struct mtype_elem *)x)))
+ if (ip_set_put_extensions(skb, set, x, mtype_is_filled(x)))
goto nla_put_failure;
ipset_nest_end(skb, nested);
}
diff --git a/net/netfilter/ipset/ip_set_core.c b/net/netfilter/ipset/ip_set_core.c
index c296f9b..ba6a551 100644
--- a/net/netfilter/ipset/ip_set_core.c
+++ b/net/netfilter/ipset/ip_set_core.c
@@ -295,7 +295,8 @@
if (unlikely(!flag_nested(nla)))
return -IPSET_ERR_PROTOCOL;
- if (nla_parse_nested(tb, IPSET_ATTR_IPADDR_MAX, nla, ipaddr_policy))
+ if (nla_parse_nested(tb, IPSET_ATTR_IPADDR_MAX, nla,
+ ipaddr_policy, NULL))
return -IPSET_ERR_PROTOCOL;
if (unlikely(!ip_set_attr_netorder(tb, IPSET_ATTR_IPADDR_IPV4)))
return -IPSET_ERR_PROTOCOL;
@@ -313,7 +314,8 @@
if (unlikely(!flag_nested(nla)))
return -IPSET_ERR_PROTOCOL;
- if (nla_parse_nested(tb, IPSET_ATTR_IPADDR_MAX, nla, ipaddr_policy))
+ if (nla_parse_nested(tb, IPSET_ATTR_IPADDR_MAX, nla,
+ ipaddr_policy, NULL))
return -IPSET_ERR_PROTOCOL;
if (unlikely(!ip_set_attr_netorder(tb, IPSET_ATTR_IPADDR_IPV6)))
return -IPSET_ERR_PROTOCOL;
@@ -501,14 +503,6 @@
* a separate reference counter
*/
static inline void
-__ip_set_get_netlink(struct ip_set *set)
-{
- write_lock_bh(&ip_set_ref_lock);
- set->ref_netlink++;
- write_unlock_bh(&ip_set_ref_lock);
-}
-
-static inline void
__ip_set_put_netlink(struct ip_set *set)
{
write_lock_bh(&ip_set_ref_lock);
@@ -769,7 +763,7 @@
struct nlmsghdr *nlh;
struct nfgenmsg *nfmsg;
- nlh = nlmsg_put(skb, portid, seq, cmd | (NFNL_SUBSYS_IPSET << 8),
+ nlh = nlmsg_put(skb, portid, seq, nfnl_msg_type(NFNL_SUBSYS_IPSET, cmd),
sizeof(*nfmsg), flags);
if (!nlh)
return NULL;
@@ -906,7 +900,7 @@
/* Without holding any locks, create private part. */
if (attr[IPSET_ATTR_DATA] &&
nla_parse_nested(tb, IPSET_ATTR_CREATE_MAX, attr[IPSET_ATTR_DATA],
- set->type->create_policy)) {
+ set->type->create_policy, NULL)) {
ret = -IPSET_ERR_PROTOCOL;
goto put_out;
}
@@ -1257,8 +1251,8 @@
ip_set_id_t index;
/* Second pass, so parser can't fail */
- nla_parse(cda, IPSET_ATTR_CMD_MAX,
- attr, nlh->nlmsg_len - min_len, ip_set_setname_policy);
+ nla_parse(cda, IPSET_ATTR_CMD_MAX, attr, nlh->nlmsg_len - min_len,
+ ip_set_setname_policy, NULL);
if (cda[IPSET_ATTR_SETNAME]) {
struct ip_set *set;
@@ -1305,7 +1299,7 @@
* manually :-(
*/
if (nlh->nlmsg_flags & NLM_F_ACK)
- netlink_ack(cb->skb, nlh, ret);
+ netlink_ack(cb->skb, nlh, ret, NULL);
return ret;
}
}
@@ -1501,9 +1495,8 @@
memcpy(&errmsg->msg, nlh, nlh->nlmsg_len);
cmdattr = (void *)&errmsg->msg + min_len;
- nla_parse(cda, IPSET_ATTR_CMD_MAX,
- cmdattr, nlh->nlmsg_len - min_len,
- ip_set_adt_policy);
+ nla_parse(cda, IPSET_ATTR_CMD_MAX, cmdattr,
+ nlh->nlmsg_len - min_len, ip_set_adt_policy, NULL);
errline = nla_data(cda[IPSET_ATTR_LINENO]);
@@ -1549,7 +1542,7 @@
if (attr[IPSET_ATTR_DATA]) {
if (nla_parse_nested(tb, IPSET_ATTR_ADT_MAX,
attr[IPSET_ATTR_DATA],
- set->type->adt_policy))
+ set->type->adt_policy, NULL))
return -IPSET_ERR_PROTOCOL;
ret = call_ad(ctnl, skb, set, tb, IPSET_ADD, flags,
use_lineno);
@@ -1561,7 +1554,7 @@
if (nla_type(nla) != IPSET_ATTR_DATA ||
!flag_nested(nla) ||
nla_parse_nested(tb, IPSET_ATTR_ADT_MAX, nla,
- set->type->adt_policy))
+ set->type->adt_policy, NULL))
return -IPSET_ERR_PROTOCOL;
ret = call_ad(ctnl, skb, set, tb, IPSET_ADD,
flags, use_lineno);
@@ -1603,7 +1596,7 @@
if (attr[IPSET_ATTR_DATA]) {
if (nla_parse_nested(tb, IPSET_ATTR_ADT_MAX,
attr[IPSET_ATTR_DATA],
- set->type->adt_policy))
+ set->type->adt_policy, NULL))
return -IPSET_ERR_PROTOCOL;
ret = call_ad(ctnl, skb, set, tb, IPSET_DEL, flags,
use_lineno);
@@ -1615,7 +1608,7 @@
if (nla_type(nla) != IPSET_ATTR_DATA ||
!flag_nested(nla) ||
nla_parse_nested(tb, IPSET_ATTR_ADT_MAX, nla,
- set->type->adt_policy))
+ set->type->adt_policy, NULL))
return -IPSET_ERR_PROTOCOL;
ret = call_ad(ctnl, skb, set, tb, IPSET_DEL,
flags, use_lineno);
@@ -1646,7 +1639,7 @@
return -ENOENT;
if (nla_parse_nested(tb, IPSET_ATTR_ADT_MAX, attr[IPSET_ATTR_DATA],
- set->type->adt_policy))
+ set->type->adt_policy, NULL))
return -IPSET_ERR_PROTOCOL;
rcu_read_lock_bh();
@@ -1915,7 +1908,7 @@
ret = -EFAULT;
goto done;
}
- op = (unsigned int *)data;
+ op = data;
if (*op < IP_SET_OP_VERSION) {
/* Check the version at the beginning of operations */
@@ -2013,7 +2006,7 @@
.pf = PF_INET,
.get_optmin = SO_IP_SET,
.get_optmax = SO_IP_SET + 1,
- .get = &ip_set_sockfn_get,
+ .get = ip_set_sockfn_get,
.owner = THIS_MODULE,
};
diff --git a/net/netfilter/ipvs/ip_vs_conn.c b/net/netfilter/ipvs/ip_vs_conn.c
index e6a2753..3d2ac71a 100644
--- a/net/netfilter/ipvs/ip_vs_conn.c
+++ b/net/netfilter/ipvs/ip_vs_conn.c
@@ -181,7 +181,7 @@
if (!(cp->flags & IP_VS_CONN_F_HASHED)) {
cp->flags |= IP_VS_CONN_F_HASHED;
- atomic_inc(&cp->refcnt);
+ refcount_inc(&cp->refcnt);
hlist_add_head_rcu(&cp->c_list, &ip_vs_conn_tab[hash]);
ret = 1;
} else {
@@ -215,7 +215,7 @@
if (cp->flags & IP_VS_CONN_F_HASHED) {
hlist_del_rcu(&cp->c_list);
cp->flags &= ~IP_VS_CONN_F_HASHED;
- atomic_dec(&cp->refcnt);
+ refcount_dec(&cp->refcnt);
ret = 1;
} else
ret = 0;
@@ -242,13 +242,13 @@
if (cp->flags & IP_VS_CONN_F_HASHED) {
ret = false;
/* Decrease refcnt and unlink conn only if we are last user */
- if (atomic_cmpxchg(&cp->refcnt, 1, 0) == 1) {
+ if (refcount_dec_if_one(&cp->refcnt)) {
hlist_del_rcu(&cp->c_list);
cp->flags &= ~IP_VS_CONN_F_HASHED;
ret = true;
}
} else
- ret = atomic_read(&cp->refcnt) ? false : true;
+ ret = refcount_read(&cp->refcnt) ? false : true;
spin_unlock(&cp->lock);
ct_write_unlock_bh(hash);
@@ -475,7 +475,7 @@
void ip_vs_conn_put(struct ip_vs_conn *cp)
{
if ((cp->flags & IP_VS_CONN_F_ONE_PACKET) &&
- (atomic_read(&cp->refcnt) == 1) &&
+ (refcount_read(&cp->refcnt) == 1) &&
!timer_pending(&cp->timer))
/* expire connection immediately */
__ip_vs_conn_put_notimer(cp);
@@ -617,8 +617,8 @@
IP_VS_DBG_ADDR(cp->af, &cp->vaddr), ntohs(cp->vport),
IP_VS_DBG_ADDR(cp->daf, &cp->daddr), ntohs(cp->dport),
ip_vs_fwd_tag(cp), cp->state,
- cp->flags, atomic_read(&cp->refcnt),
- atomic_read(&dest->refcnt));
+ cp->flags, refcount_read(&cp->refcnt),
+ refcount_read(&dest->refcnt));
/* Update the connection counters */
if (!(flags & IP_VS_CONN_F_TEMPLATE)) {
@@ -714,8 +714,8 @@
IP_VS_DBG_ADDR(cp->af, &cp->vaddr), ntohs(cp->vport),
IP_VS_DBG_ADDR(cp->daf, &cp->daddr), ntohs(cp->dport),
ip_vs_fwd_tag(cp), cp->state,
- cp->flags, atomic_read(&cp->refcnt),
- atomic_read(&dest->refcnt));
+ cp->flags, refcount_read(&cp->refcnt),
+ refcount_read(&dest->refcnt));
/* Update the connection counters */
if (!(cp->flags & IP_VS_CONN_F_TEMPLATE)) {
@@ -863,10 +863,10 @@
expire_later:
IP_VS_DBG(7, "delayed: conn->refcnt=%d conn->n_control=%d\n",
- atomic_read(&cp->refcnt),
+ refcount_read(&cp->refcnt),
atomic_read(&cp->n_control));
- atomic_inc(&cp->refcnt);
+ refcount_inc(&cp->refcnt);
cp->timeout = 60*HZ;
if (ipvs->sync_state & IP_VS_STATE_MASTER)
@@ -941,7 +941,7 @@
* it in the table, so that other thread run ip_vs_random_dropentry
* but cannot drop this entry.
*/
- atomic_set(&cp->refcnt, 1);
+ refcount_set(&cp->refcnt, 1);
cp->control = NULL;
atomic_set(&cp->n_control, 0);
diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
index db40050..d2d7bdf 100644
--- a/net/netfilter/ipvs/ip_vs_core.c
+++ b/net/netfilter/ipvs/ip_vs_core.c
@@ -542,7 +542,7 @@
IP_VS_DBG_ADDR(cp->af, &cp->caddr), ntohs(cp->cport),
IP_VS_DBG_ADDR(cp->af, &cp->vaddr), ntohs(cp->vport),
IP_VS_DBG_ADDR(cp->daf, &cp->daddr), ntohs(cp->dport),
- cp->flags, atomic_read(&cp->refcnt));
+ cp->flags, refcount_read(&cp->refcnt));
ip_vs_conn_stats(cp, svc);
return cp;
@@ -1193,7 +1193,7 @@
IP_VS_DBG_ADDR(cp->af, &cp->caddr), ntohs(cp->cport),
IP_VS_DBG_ADDR(cp->af, &cp->vaddr), ntohs(cp->vport),
IP_VS_DBG_ADDR(cp->af, &cp->daddr), ntohs(cp->dport),
- cp->flags, atomic_read(&cp->refcnt));
+ cp->flags, refcount_read(&cp->refcnt));
LeaveFunction(12);
return cp;
}
@@ -2200,6 +2200,7 @@
static int __net_init __ip_vs_init(struct net *net)
{
struct netns_ipvs *ipvs;
+ int ret;
ipvs = net_generic(net, ip_vs_net_id);
if (ipvs == NULL)
@@ -2231,13 +2232,17 @@
if (ip_vs_sync_net_init(ipvs) < 0)
goto sync_fail;
- printk(KERN_INFO "IPVS: Creating netns size=%zu id=%d\n",
- sizeof(struct netns_ipvs), ipvs->gen);
+ ret = nf_register_net_hooks(net, ip_vs_ops, ARRAY_SIZE(ip_vs_ops));
+ if (ret < 0)
+ goto hook_fail;
+
return 0;
/*
* Error handling
*/
+hook_fail:
+ ip_vs_sync_net_cleanup(ipvs);
sync_fail:
ip_vs_conn_net_cleanup(ipvs);
conn_fail:
@@ -2257,6 +2262,7 @@
{
struct netns_ipvs *ipvs = net_ipvs(net);
+ nf_unregister_net_hooks(net, ip_vs_ops, ARRAY_SIZE(ip_vs_ops));
ip_vs_service_net_cleanup(ipvs); /* ip_vs_flush() with locks */
ip_vs_conn_net_cleanup(ipvs);
ip_vs_app_net_cleanup(ipvs);
@@ -2317,24 +2323,16 @@
if (ret < 0)
goto cleanup_sub;
- ret = nf_register_hooks(ip_vs_ops, ARRAY_SIZE(ip_vs_ops));
- if (ret < 0) {
- pr_err("can't register hooks.\n");
- goto cleanup_dev;
- }
-
ret = ip_vs_register_nl_ioctl();
if (ret < 0) {
pr_err("can't register netlink/ioctl.\n");
- goto cleanup_hooks;
+ goto cleanup_dev;
}
pr_info("ipvs loaded.\n");
return ret;
-cleanup_hooks:
- nf_unregister_hooks(ip_vs_ops, ARRAY_SIZE(ip_vs_ops));
cleanup_dev:
unregister_pernet_device(&ipvs_core_dev_ops);
cleanup_sub:
@@ -2351,7 +2349,6 @@
static void __exit ip_vs_cleanup(void)
{
ip_vs_unregister_nl_ioctl();
- nf_unregister_hooks(ip_vs_ops, ARRAY_SIZE(ip_vs_ops));
unregister_pernet_device(&ipvs_core_dev_ops);
unregister_pernet_subsys(&ipvs_core_ops); /* free ip_vs struct */
ip_vs_conn_cleanup();
diff --git a/net/netfilter/ipvs/ip_vs_ctl.c b/net/netfilter/ipvs/ip_vs_ctl.c
index 4d753be..1fa3c23 100644
--- a/net/netfilter/ipvs/ip_vs_ctl.c
+++ b/net/netfilter/ipvs/ip_vs_ctl.c
@@ -699,7 +699,7 @@
dest->vfwmark,
IP_VS_DBG_ADDR(dest->af, &dest->addr),
ntohs(dest->port),
- atomic_read(&dest->refcnt));
+ refcount_read(&dest->refcnt));
if (dest->af == dest_af &&
ip_vs_addr_equal(dest_af, &dest->addr, daddr) &&
dest->port == dport &&
@@ -934,7 +934,7 @@
atomic_set(&dest->activeconns, 0);
atomic_set(&dest->inactconns, 0);
atomic_set(&dest->persistconns, 0);
- atomic_set(&dest->refcnt, 1);
+ refcount_set(&dest->refcnt, 1);
INIT_HLIST_NODE(&dest->d_list);
spin_lock_init(&dest->dst_lock);
@@ -998,7 +998,7 @@
IP_VS_DBG_BUF(3, "Get destination %s:%u from trash, "
"dest->refcnt=%d, service %u/%s:%u\n",
IP_VS_DBG_ADDR(udest->af, &daddr), ntohs(dport),
- atomic_read(&dest->refcnt),
+ refcount_read(&dest->refcnt),
dest->vfwmark,
IP_VS_DBG_ADDR(svc->af, &dest->vaddr),
ntohs(dest->vport));
@@ -1074,7 +1074,7 @@
spin_lock_bh(&ipvs->dest_trash_lock);
IP_VS_DBG_BUF(3, "Moving dest %s:%u into trash, dest->refcnt=%d\n",
IP_VS_DBG_ADDR(dest->af, &dest->addr), ntohs(dest->port),
- atomic_read(&dest->refcnt));
+ refcount_read(&dest->refcnt));
if (list_empty(&ipvs->dest_trash) && !cleanup)
mod_timer(&ipvs->dest_trash_timer,
jiffies + (IP_VS_DEST_TRASH_PERIOD >> 1));
@@ -1157,7 +1157,7 @@
spin_lock(&ipvs->dest_trash_lock);
list_for_each_entry_safe(dest, next, &ipvs->dest_trash, t_list) {
- if (atomic_read(&dest->refcnt) > 1)
+ if (refcount_read(&dest->refcnt) > 1)
continue;
if (dest->idle_start) {
if (time_before(now, dest->idle_start +
@@ -1545,7 +1545,7 @@
dev->name,
IP_VS_DBG_ADDR(dest->af, &dest->addr),
ntohs(dest->port),
- atomic_read(&dest->refcnt));
+ refcount_read(&dest->refcnt));
__ip_vs_dst_cache_reset(dest);
}
spin_unlock_bh(&dest->dst_lock);
@@ -1774,13 +1774,13 @@
.procname = "sync_version",
.maxlen = sizeof(int),
.mode = 0644,
- .proc_handler = &proc_do_sync_mode,
+ .proc_handler = proc_do_sync_mode,
},
{
.procname = "sync_ports",
.maxlen = sizeof(int),
.mode = 0644,
- .proc_handler = &proc_do_sync_ports,
+ .proc_handler = proc_do_sync_ports,
},
{
.procname = "sync_persist_mode",
@@ -2130,8 +2130,8 @@
/* 01234567 01234567 01234567 0123456701234567 0123456701234567 */
seq_puts(seq,
" Total Incoming Outgoing Incoming Outgoing\n");
- seq_printf(seq,
- " Conns Packets Packets Bytes Bytes\n");
+ seq_puts(seq,
+ " Conns Packets Packets Bytes Bytes\n");
ip_vs_copy_stats(&show, &net_ipvs(net)->tot_stats);
seq_printf(seq, "%8LX %8LX %8LX %16LX %16LX\n\n",
@@ -2178,8 +2178,8 @@
/* 01234567 01234567 01234567 0123456701234567 0123456701234567 */
seq_puts(seq,
" Total Incoming Outgoing Incoming Outgoing\n");
- seq_printf(seq,
- "CPU Conns Packets Packets Bytes Bytes\n");
+ seq_puts(seq,
+ "CPU Conns Packets Packets Bytes Bytes\n");
for_each_possible_cpu(i) {
struct ip_vs_cpu_stats *u = per_cpu_ptr(cpustats, i);
@@ -3100,7 +3100,8 @@
/* Parse mandatory identifying service fields first */
if (nla == NULL ||
- nla_parse_nested(attrs, IPVS_SVC_ATTR_MAX, nla, ip_vs_svc_policy))
+ nla_parse_nested(attrs, IPVS_SVC_ATTR_MAX, nla,
+ ip_vs_svc_policy, NULL))
return -EINVAL;
nla_af = attrs[IPVS_SVC_ATTR_AF];
@@ -3258,8 +3259,8 @@
mutex_lock(&__ip_vs_mutex);
/* Try to find the service for which to dump destinations */
- if (nlmsg_parse(cb->nlh, GENL_HDRLEN, attrs,
- IPVS_CMD_ATTR_MAX, ip_vs_cmd_policy))
+ if (nlmsg_parse(cb->nlh, GENL_HDRLEN, attrs, IPVS_CMD_ATTR_MAX,
+ ip_vs_cmd_policy, NULL))
goto out_err;
@@ -3295,7 +3296,8 @@
/* Parse mandatory identifying destination fields first */
if (nla == NULL ||
- nla_parse_nested(attrs, IPVS_DEST_ATTR_MAX, nla, ip_vs_dest_policy))
+ nla_parse_nested(attrs, IPVS_DEST_ATTR_MAX, nla,
+ ip_vs_dest_policy, NULL))
return -EINVAL;
nla_addr = attrs[IPVS_DEST_ATTR_ADDR];
@@ -3537,7 +3539,7 @@
if (!info->attrs[IPVS_CMD_ATTR_DAEMON] ||
nla_parse_nested(daemon_attrs, IPVS_DAEMON_ATTR_MAX,
info->attrs[IPVS_CMD_ATTR_DAEMON],
- ip_vs_daemon_policy))
+ ip_vs_daemon_policy, info->extack))
goto out;
if (cmd == IPVS_CMD_NEW_DAEMON)
diff --git a/net/netfilter/ipvs/ip_vs_ftp.c b/net/netfilter/ipvs/ip_vs_ftp.c
index d30c327..fb780be 100644
--- a/net/netfilter/ipvs/ip_vs_ftp.c
+++ b/net/netfilter/ipvs/ip_vs_ftp.c
@@ -260,7 +260,9 @@
buf_len = strlen(buf);
ct = nf_ct_get(skb, &ctinfo);
- if (ct && !nf_ct_is_untracked(ct) && nfct_nat(ct)) {
+ if (ct && (ct->status & IPS_NAT_MASK)) {
+ bool mangled;
+
/* If mangling fails this function will return 0
* which will cause the packet to be dropped.
* Mangling can only fail under memory pressure,
@@ -268,12 +270,13 @@
* packet.
*/
rcu_read_lock();
- ret = nf_nat_mangle_tcp_packet(skb, ct, ctinfo,
- iph->ihl * 4,
- start-data, end-start,
- buf, buf_len);
+ mangled = nf_nat_mangle_tcp_packet(skb, ct, ctinfo,
+ iph->ihl * 4,
+ start - data,
+ end - start,
+ buf, buf_len);
rcu_read_unlock();
- if (ret) {
+ if (mangled) {
ip_vs_nfct_expect_related(skb, ct, n_cp,
IPPROTO_TCP, 0, 0);
if (skb->ip_summed == CHECKSUM_COMPLETE)
@@ -482,11 +485,8 @@
static int __init ip_vs_ftp_init(void)
{
- int rv;
-
- rv = register_pernet_subsys(&ip_vs_ftp_ops);
/* rcu_barrier() is called by netns on error */
- return rv;
+ return register_pernet_subsys(&ip_vs_ftp_ops);
}
/*
diff --git a/net/netfilter/ipvs/ip_vs_lblc.c b/net/netfilter/ipvs/ip_vs_lblc.c
index 5824927..b6aa4a9 100644
--- a/net/netfilter/ipvs/ip_vs_lblc.c
+++ b/net/netfilter/ipvs/ip_vs_lblc.c
@@ -448,7 +448,7 @@
IP_VS_DBG_ADDR(least->af, &least->addr),
ntohs(least->port),
atomic_read(&least->activeconns),
- atomic_read(&least->refcnt),
+ refcount_read(&least->refcnt),
atomic_read(&least->weight), loh);
return least;
diff --git a/net/netfilter/ipvs/ip_vs_lblcr.c b/net/netfilter/ipvs/ip_vs_lblcr.c
index 703f118..c13ff57 100644
--- a/net/netfilter/ipvs/ip_vs_lblcr.c
+++ b/net/netfilter/ipvs/ip_vs_lblcr.c
@@ -204,7 +204,7 @@
IP_VS_DBG_ADDR(least->af, &least->addr),
ntohs(least->port),
atomic_read(&least->activeconns),
- atomic_read(&least->refcnt),
+ refcount_read(&least->refcnt),
atomic_read(&least->weight), loh);
return least;
}
@@ -249,7 +249,7 @@
__func__,
IP_VS_DBG_ADDR(most->af, &most->addr), ntohs(most->port),
atomic_read(&most->activeconns),
- atomic_read(&most->refcnt),
+ refcount_read(&most->refcnt),
atomic_read(&most->weight), moh);
return most;
}
@@ -612,7 +612,7 @@
IP_VS_DBG_ADDR(least->af, &least->addr),
ntohs(least->port),
atomic_read(&least->activeconns),
- atomic_read(&least->refcnt),
+ refcount_read(&least->refcnt),
atomic_read(&least->weight), loh);
return least;
diff --git a/net/netfilter/ipvs/ip_vs_nfct.c b/net/netfilter/ipvs/ip_vs_nfct.c
index fc230d9..6cf3fd8 100644
--- a/net/netfilter/ipvs/ip_vs_nfct.c
+++ b/net/netfilter/ipvs/ip_vs_nfct.c
@@ -85,7 +85,7 @@
struct nf_conn *ct = nf_ct_get(skb, &ctinfo);
struct nf_conntrack_tuple new_tuple;
- if (ct == NULL || nf_ct_is_confirmed(ct) || nf_ct_is_untracked(ct) ||
+ if (ct == NULL || nf_ct_is_confirmed(ct) ||
nf_ct_is_dying(ct))
return;
@@ -232,7 +232,7 @@
{
struct nf_conntrack_expect *exp;
- if (ct == NULL || nf_ct_is_untracked(ct))
+ if (ct == NULL)
return;
exp = nf_ct_expect_alloc(ct);
diff --git a/net/netfilter/ipvs/ip_vs_nq.c b/net/netfilter/ipvs/ip_vs_nq.c
index a8b6340..7d9d4ac 100644
--- a/net/netfilter/ipvs/ip_vs_nq.c
+++ b/net/netfilter/ipvs/ip_vs_nq.c
@@ -110,7 +110,7 @@
IP_VS_DBG_ADDR(least->af, &least->addr),
ntohs(least->port),
atomic_read(&least->activeconns),
- atomic_read(&least->refcnt),
+ refcount_read(&least->refcnt),
atomic_read(&least->weight), loh);
return least;
diff --git a/net/netfilter/ipvs/ip_vs_proto.c b/net/netfilter/ipvs/ip_vs_proto.c
index 8ae4807..ca880a3 100644
--- a/net/netfilter/ipvs/ip_vs_proto.c
+++ b/net/netfilter/ipvs/ip_vs_proto.c
@@ -193,28 +193,6 @@
}
-/*
- * Set timeout value for state specified by name
- */
-int
-ip_vs_set_state_timeout(int *table, int num, const char *const *names,
- const char *name, int to)
-{
- int i;
-
- if (!table || !name || !to)
- return -EINVAL;
-
- for (i = 0; i < num; i++) {
- if (strcmp(names[i], name))
- continue;
- table[i] = to * HZ;
- return 0;
- }
- return -ENOENT;
-}
-
-
const char * ip_vs_state_name(__u16 proto, int state)
{
struct ip_vs_protocol *pp = ip_vs_proto_get(proto);
diff --git a/net/netfilter/ipvs/ip_vs_proto_sctp.c b/net/netfilter/ipvs/ip_vs_proto_sctp.c
index d952d67..56f8e4b 100644
--- a/net/netfilter/ipvs/ip_vs_proto_sctp.c
+++ b/net/netfilter/ipvs/ip_vs_proto_sctp.c
@@ -447,7 +447,7 @@
ntohs(cp->cport),
sctp_state_name(cp->state),
sctp_state_name(next_state),
- atomic_read(&cp->refcnt));
+ refcount_read(&cp->refcnt));
if (dest) {
if (!(cp->flags & IP_VS_CONN_F_INACTIVE) &&
(next_state != IP_VS_SCTP_S_ESTABLISHED)) {
diff --git a/net/netfilter/ipvs/ip_vs_proto_tcp.c b/net/netfilter/ipvs/ip_vs_proto_tcp.c
index 5117bcb..12dc8d5 100644
--- a/net/netfilter/ipvs/ip_vs_proto_tcp.c
+++ b/net/netfilter/ipvs/ip_vs_proto_tcp.c
@@ -557,7 +557,7 @@
ntohs(cp->cport),
tcp_state_name(cp->state),
tcp_state_name(new_state),
- atomic_read(&cp->refcnt));
+ refcount_read(&cp->refcnt));
if (dest) {
if (!(cp->flags & IP_VS_CONN_F_INACTIVE) &&
diff --git a/net/netfilter/ipvs/ip_vs_rr.c b/net/netfilter/ipvs/ip_vs_rr.c
index 58bacfc..ee0530d1 100644
--- a/net/netfilter/ipvs/ip_vs_rr.c
+++ b/net/netfilter/ipvs/ip_vs_rr.c
@@ -97,7 +97,7 @@
"activeconns %d refcnt %d weight %d\n",
IP_VS_DBG_ADDR(dest->af, &dest->addr), ntohs(dest->port),
atomic_read(&dest->activeconns),
- atomic_read(&dest->refcnt), atomic_read(&dest->weight));
+ refcount_read(&dest->refcnt), atomic_read(&dest->weight));
return dest;
}
diff --git a/net/netfilter/ipvs/ip_vs_sed.c b/net/netfilter/ipvs/ip_vs_sed.c
index f8e2d00..ab23cf2 100644
--- a/net/netfilter/ipvs/ip_vs_sed.c
+++ b/net/netfilter/ipvs/ip_vs_sed.c
@@ -111,7 +111,7 @@
IP_VS_DBG_ADDR(least->af, &least->addr),
ntohs(least->port),
atomic_read(&least->activeconns),
- atomic_read(&least->refcnt),
+ refcount_read(&least->refcnt),
atomic_read(&least->weight), loh);
return least;
diff --git a/net/netfilter/ipvs/ip_vs_sync.c b/net/netfilter/ipvs/ip_vs_sync.c
index b03c280..0e5b64a 100644
--- a/net/netfilter/ipvs/ip_vs_sync.c
+++ b/net/netfilter/ipvs/ip_vs_sync.c
@@ -520,7 +520,7 @@
if (!(cp->flags & IP_VS_CONN_F_TEMPLATE) &&
pkts % sync_period != sysctl_sync_threshold(ipvs))
return 0;
- } else if (sync_refresh_period <= 0 &&
+ } else if (!sync_refresh_period &&
pkts != sysctl_sync_threshold(ipvs))
return 0;
@@ -1849,7 +1849,7 @@
if (state == IP_VS_STATE_MASTER) {
struct ipvs_master_sync_state *ms;
- ipvs->ms = kzalloc(count * sizeof(ipvs->ms[0]), GFP_KERNEL);
+ ipvs->ms = kcalloc(count, sizeof(ipvs->ms[0]), GFP_KERNEL);
if (!ipvs->ms)
goto out;
ms = ipvs->ms;
@@ -1862,7 +1862,7 @@
ms->ipvs = ipvs;
}
} else {
- array = kzalloc(count * sizeof(struct task_struct *),
+ array = kcalloc(count, sizeof(struct task_struct *),
GFP_KERNEL);
if (!array)
goto out;
diff --git a/net/netfilter/ipvs/ip_vs_wlc.c b/net/netfilter/ipvs/ip_vs_wlc.c
index 6b366fd..6add39e 100644
--- a/net/netfilter/ipvs/ip_vs_wlc.c
+++ b/net/netfilter/ipvs/ip_vs_wlc.c
@@ -83,7 +83,7 @@
IP_VS_DBG_ADDR(least->af, &least->addr),
ntohs(least->port),
atomic_read(&least->activeconns),
- atomic_read(&least->refcnt),
+ refcount_read(&least->refcnt),
atomic_read(&least->weight), loh);
return least;
diff --git a/net/netfilter/ipvs/ip_vs_wrr.c b/net/netfilter/ipvs/ip_vs_wrr.c
index 17e6d44..62258dd 100644
--- a/net/netfilter/ipvs/ip_vs_wrr.c
+++ b/net/netfilter/ipvs/ip_vs_wrr.c
@@ -218,7 +218,7 @@
"activeconns %d refcnt %d weight %d\n",
IP_VS_DBG_ADDR(dest->af, &dest->addr), ntohs(dest->port),
atomic_read(&dest->activeconns),
- atomic_read(&dest->refcnt),
+ refcount_read(&dest->refcnt),
atomic_read(&dest->weight));
mark->cl = dest;
diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c
index 4e1a98f..2eab1e0 100644
--- a/net/netfilter/ipvs/ip_vs_xmit.c
+++ b/net/netfilter/ipvs/ip_vs_xmit.c
@@ -775,7 +775,7 @@
enum ip_conntrack_info ctinfo;
struct nf_conn *ct = nf_ct_get(skb, &ctinfo);
- if (ct && !nf_ct_is_untracked(ct)) {
+ if (ct) {
IP_VS_DBG_RL_PKT(10, AF_INET, pp, skb, ipvsh->off,
"ip_vs_nat_xmit(): "
"stopping DNAT to local address");
@@ -866,7 +866,7 @@
enum ip_conntrack_info ctinfo;
struct nf_conn *ct = nf_ct_get(skb, &ctinfo);
- if (ct && !nf_ct_is_untracked(ct)) {
+ if (ct) {
IP_VS_DBG_RL_PKT(10, AF_INET6, pp, skb, ipvsh->off,
"ip_vs_nat_xmit_v6(): "
"stopping DNAT to local address");
@@ -1338,7 +1338,7 @@
enum ip_conntrack_info ctinfo;
struct nf_conn *ct = nf_ct_get(skb, &ctinfo);
- if (ct && !nf_ct_is_untracked(ct)) {
+ if (ct) {
IP_VS_DBG(10, "%s(): "
"stopping DNAT to local address %pI4\n",
__func__, &cp->daddr.ip);
@@ -1429,7 +1429,7 @@
enum ip_conntrack_info ctinfo;
struct nf_conn *ct = nf_ct_get(skb, &ctinfo);
- if (ct && !nf_ct_is_untracked(ct)) {
+ if (ct) {
IP_VS_DBG(10, "%s(): "
"stopping DNAT to local address %pI6\n",
__func__, &cp->daddr.in6);
diff --git a/net/netfilter/nf_conntrack_acct.c b/net/netfilter/nf_conntrack_acct.c
index 45da11a..8669167 100644
--- a/net/netfilter/nf_conntrack_acct.c
+++ b/net/netfilter/nf_conntrack_acct.c
@@ -55,7 +55,7 @@
};
EXPORT_SYMBOL_GPL(seq_print_acct);
-static struct nf_ct_ext_type acct_extend __read_mostly = {
+static const struct nf_ct_ext_type acct_extend = {
.len = sizeof(struct nf_conn_acct),
.align = __alignof__(struct nf_conn_acct),
.id = NF_CT_EXT_ACCT,
diff --git a/net/netfilter/nf_conntrack_amanda.c b/net/netfilter/nf_conntrack_amanda.c
index 57a26cc..03d2ccf 100644
--- a/net/netfilter/nf_conntrack_amanda.c
+++ b/net/netfilter/nf_conntrack_amanda.c
@@ -207,6 +207,8 @@
{
int ret, i;
+ NF_CT_HELPER_BUILD_BUG_ON(0);
+
for (i = 0; i < ARRAY_SIZE(search); i++) {
search[i].ts = textsearch_prepare(ts_algo, search[i].string,
search[i].len,
diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c
index ffb78e5..3c8f1ed 100644
--- a/net/netfilter/nf_conntrack_core.c
+++ b/net/netfilter/nf_conntrack_core.c
@@ -76,6 +76,7 @@
struct delayed_work dwork;
u32 last_bucket;
bool exiting;
+ bool early_drop;
long next_gc_run;
};
@@ -180,14 +181,6 @@
unsigned int nf_conntrack_max __read_mostly;
seqcount_t nf_conntrack_generation __read_mostly;
-
-/* nf_conn must be 8 bytes aligned, as the 3 LSB bits are used
- * for the nfctinfo. We cheat by (ab)using the PER CPU cache line
- * alignment to enforce this.
- */
-DEFINE_PER_CPU_ALIGNED(struct nf_conn, nf_conntrack_untracked);
-EXPORT_PER_CPU_SYMBOL(nf_conntrack_untracked);
-
static unsigned int nf_conntrack_hash_rnd __read_mostly;
static u32 hash_conntrack_raw(const struct nf_conntrack_tuple *tuple,
@@ -706,7 +699,7 @@
l4proto = __nf_ct_l4proto_find(nf_ct_l3num(ct), nf_ct_protonum(ct));
if (l4proto->allow_clash &&
- !nfct_nat(ct) &&
+ ((ct->status & IPS_NAT_DONE_MASK) == 0) &&
!nf_ct_is_dying(ct) &&
atomic_inc_not_zero(&ct->ct_general.use)) {
enum ip_conntrack_info oldinfo;
@@ -959,10 +952,30 @@
return false;
}
+static bool gc_worker_skip_ct(const struct nf_conn *ct)
+{
+ return !nf_ct_is_confirmed(ct) || nf_ct_is_dying(ct);
+}
+
+static bool gc_worker_can_early_drop(const struct nf_conn *ct)
+{
+ const struct nf_conntrack_l4proto *l4proto;
+
+ if (!test_bit(IPS_ASSURED_BIT, &ct->status))
+ return true;
+
+ l4proto = __nf_ct_l4proto_find(nf_ct_l3num(ct), nf_ct_protonum(ct));
+ if (l4proto->can_early_drop && l4proto->can_early_drop(ct))
+ return true;
+
+ return false;
+}
+
static void gc_worker(struct work_struct *work)
{
unsigned int min_interval = max(HZ / GC_MAX_BUCKETS_DIV, 1u);
unsigned int i, goal, buckets = 0, expired_count = 0;
+ unsigned int nf_conntrack_max95 = 0;
struct conntrack_gc_work *gc_work;
unsigned int ratio, scanned = 0;
unsigned long next_run;
@@ -971,6 +984,8 @@
goal = nf_conntrack_htable_size / GC_MAX_BUCKETS_DIV;
i = gc_work->last_bucket;
+ if (gc_work->early_drop)
+ nf_conntrack_max95 = nf_conntrack_max / 100u * 95u;
do {
struct nf_conntrack_tuple_hash *h;
@@ -987,6 +1002,8 @@
i = 0;
hlist_nulls_for_each_entry_rcu(h, n, &ct_hash[i], hnnode) {
+ struct net *net;
+
tmp = nf_ct_tuplehash_to_ctrack(h);
scanned++;
@@ -995,6 +1012,27 @@
expired_count++;
continue;
}
+
+ if (nf_conntrack_max95 == 0 || gc_worker_skip_ct(tmp))
+ continue;
+
+ net = nf_ct_net(tmp);
+ if (atomic_read(&net->ct.count) < nf_conntrack_max95)
+ continue;
+
+ /* need to take reference to avoid possible races */
+ if (!atomic_inc_not_zero(&tmp->ct_general.use))
+ continue;
+
+ if (gc_worker_skip_ct(tmp)) {
+ nf_ct_put(tmp);
+ continue;
+ }
+
+ if (gc_worker_can_early_drop(tmp))
+ nf_ct_kill(tmp);
+
+ nf_ct_put(tmp);
}
/* could check get_nulls_value() here and restart if ct
@@ -1040,6 +1078,7 @@
next_run = gc_work->next_gc_run;
gc_work->last_bucket = i;
+ gc_work->early_drop = false;
queue_delayed_work(system_long_wq, &gc_work->dwork, next_run);
}
@@ -1065,6 +1104,8 @@
if (nf_conntrack_max &&
unlikely(atomic_read(&net->ct.count) > nf_conntrack_max)) {
if (!early_drop(net, hash)) {
+ if (!conntrack_gc_work.early_drop)
+ conntrack_gc_work.early_drop = true;
atomic_dec(&net->ct.count);
net_warn_ratelimited("nf_conntrack: table full, dropping packet\n");
return ERR_PTR(-ENOMEM);
@@ -1133,7 +1174,7 @@
/* Allocate a new conntrack: we return -ENOMEM if classification
failed due to stress. Otherwise it really is unclassifiable. */
-static struct nf_conntrack_tuple_hash *
+static noinline struct nf_conntrack_tuple_hash *
init_conntrack(struct net *net, struct nf_conn *tmpl,
const struct nf_conntrack_tuple *tuple,
struct nf_conntrack_l3proto *l3proto,
@@ -1241,21 +1282,20 @@
return &ct->tuplehash[IP_CT_DIR_ORIGINAL];
}
-/* On success, returns conntrack ptr, sets skb->_nfct | ctinfo */
-static inline struct nf_conn *
+/* On success, returns 0, sets skb->_nfct | ctinfo */
+static int
resolve_normal_ct(struct net *net, struct nf_conn *tmpl,
struct sk_buff *skb,
unsigned int dataoff,
u_int16_t l3num,
u_int8_t protonum,
struct nf_conntrack_l3proto *l3proto,
- struct nf_conntrack_l4proto *l4proto,
- int *set_reply,
- enum ip_conntrack_info *ctinfo)
+ struct nf_conntrack_l4proto *l4proto)
{
const struct nf_conntrack_zone *zone;
struct nf_conntrack_tuple tuple;
struct nf_conntrack_tuple_hash *h;
+ enum ip_conntrack_info ctinfo;
struct nf_conntrack_zone tmp;
struct nf_conn *ct;
u32 hash;
@@ -1264,7 +1304,7 @@
dataoff, l3num, protonum, net, &tuple, l3proto,
l4proto)) {
pr_debug("Can't get tuple\n");
- return NULL;
+ return 0;
}
/* look for tuple match */
@@ -1275,33 +1315,30 @@
h = init_conntrack(net, tmpl, &tuple, l3proto, l4proto,
skb, dataoff, hash);
if (!h)
- return NULL;
+ return 0;
if (IS_ERR(h))
- return (void *)h;
+ return PTR_ERR(h);
}
ct = nf_ct_tuplehash_to_ctrack(h);
/* It exists; we have (non-exclusive) reference. */
if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY) {
- *ctinfo = IP_CT_ESTABLISHED_REPLY;
- /* Please set reply bit if this packet OK */
- *set_reply = 1;
+ ctinfo = IP_CT_ESTABLISHED_REPLY;
} else {
/* Once we've had two way comms, always ESTABLISHED. */
if (test_bit(IPS_SEEN_REPLY_BIT, &ct->status)) {
pr_debug("normal packet for %p\n", ct);
- *ctinfo = IP_CT_ESTABLISHED;
+ ctinfo = IP_CT_ESTABLISHED;
} else if (test_bit(IPS_EXPECTED_BIT, &ct->status)) {
pr_debug("related packet for %p\n", ct);
- *ctinfo = IP_CT_RELATED;
+ ctinfo = IP_CT_RELATED;
} else {
pr_debug("new packet for %p\n", ct);
- *ctinfo = IP_CT_NEW;
+ ctinfo = IP_CT_NEW;
}
- *set_reply = 0;
}
- nf_ct_set(skb, ct, *ctinfo);
- return ct;
+ nf_ct_set(skb, ct, ctinfo);
+ return 0;
}
unsigned int
@@ -1315,13 +1352,13 @@
unsigned int *timeouts;
unsigned int dataoff;
u_int8_t protonum;
- int set_reply = 0;
int ret;
tmpl = nf_ct_get(skb, &ctinfo);
- if (tmpl) {
+ if (tmpl || ctinfo == IP_CT_UNTRACKED) {
/* Previously seen (loopback or untracked)? Ignore. */
- if (!nf_ct_is_template(tmpl)) {
+ if ((tmpl && !nf_ct_is_template(tmpl)) ||
+ ctinfo == IP_CT_UNTRACKED) {
NF_CT_STAT_INC_ATOMIC(net, ignore);
return NF_ACCEPT;
}
@@ -1358,23 +1395,22 @@
goto out;
}
repeat:
- ct = resolve_normal_ct(net, tmpl, skb, dataoff, pf, protonum,
- l3proto, l4proto, &set_reply, &ctinfo);
- if (!ct) {
- /* Not valid part of a connection */
- NF_CT_STAT_INC_ATOMIC(net, invalid);
- ret = NF_ACCEPT;
- goto out;
- }
-
- if (IS_ERR(ct)) {
+ ret = resolve_normal_ct(net, tmpl, skb, dataoff, pf, protonum,
+ l3proto, l4proto);
+ if (ret < 0) {
/* Too stressed to deal. */
NF_CT_STAT_INC_ATOMIC(net, drop);
ret = NF_DROP;
goto out;
}
- NF_CT_ASSERT(skb_nfct(skb));
+ ct = nf_ct_get(skb, &ctinfo);
+ if (!ct) {
+ /* Not valid part of a connection */
+ NF_CT_STAT_INC_ATOMIC(net, invalid);
+ ret = NF_ACCEPT;
+ goto out;
+ }
/* Decide what timeout policy we want to apply to this flow. */
timeouts = nf_ct_timeout_lookup(net, ct, l4proto);
@@ -1399,7 +1435,8 @@
goto out;
}
- if (set_reply && !test_and_set_bit(IPS_SEEN_REPLY_BIT, &ct->status))
+ if (ctinfo == IP_CT_ESTABLISHED_REPLY &&
+ !test_and_set_bit(IPS_SEEN_REPLY_BIT, &ct->status))
nf_conntrack_event_cache(IPCT_REPLY, ct);
out:
if (tmpl)
@@ -1634,18 +1671,6 @@
}
EXPORT_SYMBOL_GPL(nf_ct_free_hashtable);
-static int untrack_refs(void)
-{
- int cnt = 0, cpu;
-
- for_each_possible_cpu(cpu) {
- struct nf_conn *ct = &per_cpu(nf_conntrack_untracked, cpu);
-
- cnt += atomic_read(&ct->ct_general.use) - 1;
- }
- return cnt;
-}
-
void nf_conntrack_cleanup_start(void)
{
conntrack_gc_work.exiting = true;
@@ -1655,8 +1680,6 @@
void nf_conntrack_cleanup_end(void)
{
RCU_INIT_POINTER(nf_ct_destroy, NULL);
- while (untrack_refs() > 0)
- schedule();
cancel_delayed_work_sync(&conntrack_gc_work.dwork);
nf_ct_free_hashtable(nf_conntrack_hash, nf_conntrack_htable_size);
@@ -1830,20 +1853,44 @@
module_param_call(hashsize, nf_conntrack_set_hashsize, param_get_uint,
&nf_conntrack_htable_size, 0600);
-void nf_ct_untracked_status_or(unsigned long bits)
+static __always_inline unsigned int total_extension_size(void)
{
- int cpu;
+ /* remember to add new extensions below */
+ BUILD_BUG_ON(NF_CT_EXT_NUM > 9);
- for_each_possible_cpu(cpu)
- per_cpu(nf_conntrack_untracked, cpu).status |= bits;
-}
-EXPORT_SYMBOL_GPL(nf_ct_untracked_status_or);
+ return sizeof(struct nf_ct_ext) +
+ sizeof(struct nf_conn_help)
+#if IS_ENABLED(CONFIG_NF_NAT)
+ + sizeof(struct nf_conn_nat)
+#endif
+ + sizeof(struct nf_conn_seqadj)
+ + sizeof(struct nf_conn_acct)
+#ifdef CONFIG_NF_CONNTRACK_EVENTS
+ + sizeof(struct nf_conntrack_ecache)
+#endif
+#ifdef CONFIG_NF_CONNTRACK_TIMESTAMP
+ + sizeof(struct nf_conn_tstamp)
+#endif
+#ifdef CONFIG_NF_CONNTRACK_TIMEOUT
+ + sizeof(struct nf_conn_timeout)
+#endif
+#ifdef CONFIG_NF_CONNTRACK_LABELS
+ + sizeof(struct nf_conn_labels)
+#endif
+#if IS_ENABLED(CONFIG_NETFILTER_SYNPROXY)
+ + sizeof(struct nf_conn_synproxy)
+#endif
+ ;
+};
int nf_conntrack_init_start(void)
{
int max_factor = 8;
int ret = -ENOMEM;
- int i, cpu;
+ int i;
+
+ /* struct nf_ct_ext uses u8 to store offsets/size */
+ BUILD_BUG_ON(total_extension_size() > 255u);
seqcount_init(&nf_conntrack_generation);
@@ -1926,15 +1973,6 @@
if (ret < 0)
goto err_proto;
- /* Set up fake conntrack: to never be deleted, not in any hashes */
- for_each_possible_cpu(cpu) {
- struct nf_conn *ct = &per_cpu(nf_conntrack_untracked, cpu);
- write_pnet(&ct->ct_net, &init_net);
- atomic_set(&ct->ct_general.use, 1);
- }
- /* - and look it like as a confirmed connection */
- nf_ct_untracked_status_or(IPS_CONFIRMED | IPS_UNTRACKED);
-
conntrack_gc_work_init(&conntrack_gc_work);
queue_delayed_work(system_long_wq, &conntrack_gc_work.dwork, HZ);
@@ -1982,6 +2020,7 @@
int ret = -ENOMEM;
int cpu;
+ BUILD_BUG_ON(IP_CT_UNTRACKED == IP_CT_NUMBER);
atomic_set(&net->ct.count, 0);
net->ct.pcpu_lists = alloc_percpu(struct ct_pcpu);
diff --git a/net/netfilter/nf_conntrack_ecache.c b/net/netfilter/nf_conntrack_ecache.c
index 22fc321..caac41a 100644
--- a/net/netfilter/nf_conntrack_ecache.c
+++ b/net/netfilter/nf_conntrack_ecache.c
@@ -195,7 +195,7 @@
events = xchg(&e->cache, 0);
- if (!nf_ct_is_confirmed(ct) || nf_ct_is_dying(ct) || !events)
+ if (!nf_ct_is_confirmed(ct) || nf_ct_is_dying(ct))
goto out_unlock;
/* We make a copy of the missed event cache without taking
@@ -212,7 +212,7 @@
ret = notify->fcn(events | missed, &item);
- if (likely(ret >= 0 && !missed))
+ if (likely(ret == 0 && !missed))
goto out_unlock;
spin_lock_bh(&ct->lock);
@@ -347,7 +347,7 @@
};
#endif /* CONFIG_SYSCTL */
-static struct nf_ct_ext_type event_extend __read_mostly = {
+static const struct nf_ct_ext_type event_extend = {
.len = sizeof(struct nf_conntrack_ecache),
.align = __alignof__(struct nf_conntrack_ecache),
.id = NF_CT_EXT_ECACHE,
@@ -420,6 +420,9 @@
int ret = nf_ct_extend_register(&event_extend);
if (ret < 0)
pr_err("nf_ct_event: Unable to register event extension.\n");
+
+ BUILD_BUG_ON(__IPCT_MAX >= 16); /* ctmask, missed use u16 */
+
return ret;
}
diff --git a/net/netfilter/nf_conntrack_expect.c b/net/netfilter/nf_conntrack_expect.c
index d800730..e03d16e 100644
--- a/net/netfilter/nf_conntrack_expect.c
+++ b/net/netfilter/nf_conntrack_expect.c
@@ -103,6 +103,17 @@
nf_ct_zone_equal_any(i->master, zone);
}
+bool nf_ct_remove_expect(struct nf_conntrack_expect *exp)
+{
+ if (del_timer(&exp->timeout)) {
+ nf_ct_unlink_expect(exp);
+ nf_ct_expect_put(exp);
+ return true;
+ }
+ return false;
+}
+EXPORT_SYMBOL_GPL(nf_ct_remove_expect);
+
struct nf_conntrack_expect *
__nf_ct_expect_find(struct net *net,
const struct nf_conntrack_zone *zone,
@@ -133,7 +144,7 @@
rcu_read_lock();
i = __nf_ct_expect_find(net, zone, tuple);
- if (i && !atomic_inc_not_zero(&i->use))
+ if (i && !refcount_inc_not_zero(&i->use))
i = NULL;
rcu_read_unlock();
@@ -186,7 +197,7 @@
return NULL;
if (exp->flags & NF_CT_EXPECT_PERMANENT) {
- atomic_inc(&exp->use);
+ refcount_inc(&exp->use);
return exp;
} else if (del_timer(&exp->timeout)) {
nf_ct_unlink_expect(exp);
@@ -211,10 +222,7 @@
spin_lock_bh(&nf_conntrack_expect_lock);
hlist_for_each_entry_safe(exp, next, &help->expectations, lnode) {
- if (del_timer(&exp->timeout)) {
- nf_ct_unlink_expect(exp);
- nf_ct_expect_put(exp);
- }
+ nf_ct_remove_expect(exp);
}
spin_unlock_bh(&nf_conntrack_expect_lock);
}
@@ -255,10 +263,7 @@
void nf_ct_unexpect_related(struct nf_conntrack_expect *exp)
{
spin_lock_bh(&nf_conntrack_expect_lock);
- if (del_timer(&exp->timeout)) {
- nf_ct_unlink_expect(exp);
- nf_ct_expect_put(exp);
- }
+ nf_ct_remove_expect(exp);
spin_unlock_bh(&nf_conntrack_expect_lock);
}
EXPORT_SYMBOL_GPL(nf_ct_unexpect_related);
@@ -275,7 +280,7 @@
return NULL;
new->master = me;
- atomic_set(&new->use, 1);
+ refcount_set(&new->use, 1);
return new;
}
EXPORT_SYMBOL_GPL(nf_ct_expect_alloc);
@@ -348,7 +353,7 @@
void nf_ct_expect_put(struct nf_conntrack_expect *exp)
{
- if (atomic_dec_and_test(&exp->use))
+ if (refcount_dec_and_test(&exp->use))
call_rcu(&exp->rcu, nf_ct_expect_free_rcu);
}
EXPORT_SYMBOL_GPL(nf_ct_expect_put);
@@ -361,7 +366,7 @@
unsigned int h = nf_ct_expect_dst_hash(net, &exp->tuple);
/* two references : one for hash insert, one for the timer */
- atomic_add(2, &exp->use);
+ refcount_add(2, &exp->use);
hlist_add_head_rcu(&exp->lnode, &master_help->expectations);
master_help->expecting[exp->class]++;
@@ -394,10 +399,8 @@
last = exp;
}
- if (last && del_timer(&last->timeout)) {
- nf_ct_unlink_expect(last);
- nf_ct_expect_put(last);
- }
+ if (last)
+ nf_ct_remove_expect(last);
}
static inline int __nf_ct_expect_check(struct nf_conntrack_expect *expect)
@@ -419,11 +422,8 @@
h = nf_ct_expect_dst_hash(net, &expect->tuple);
hlist_for_each_entry_safe(i, next, &nf_ct_expect_hash[h], hnode) {
if (expect_matches(i, expect)) {
- if (del_timer(&i->timeout)) {
- nf_ct_unlink_expect(i);
- nf_ct_expect_put(i);
+ if (nf_ct_remove_expect(expect))
break;
- }
} else if (expect_clash(i, expect)) {
ret = -EBUSY;
goto out;
@@ -549,7 +549,7 @@
seq_printf(s, "%ld ", timer_pending(&expect->timeout)
? (long)(expect->timeout.expires - jiffies)/HZ : 0);
else
- seq_printf(s, "- ");
+ seq_puts(s, "- ");
seq_printf(s, "l3proto = %u proto=%u ",
expect->tuple.src.l3num,
expect->tuple.dst.protonum);
@@ -559,7 +559,7 @@
expect->tuple.dst.protonum));
if (expect->flags & NF_CT_EXPECT_PERMANENT) {
- seq_printf(s, "PERMANENT");
+ seq_puts(s, "PERMANENT");
delim = ",";
}
if (expect->flags & NF_CT_EXPECT_INACTIVE) {
diff --git a/net/netfilter/nf_conntrack_extend.c b/net/netfilter/nf_conntrack_extend.c
index 008299b..6c605e8 100644
--- a/net/netfilter/nf_conntrack_extend.c
+++ b/net/netfilter/nf_conntrack_extend.c
@@ -18,17 +18,14 @@
static struct nf_ct_ext_type __rcu *nf_ct_ext_types[NF_CT_EXT_NUM];
static DEFINE_MUTEX(nf_ct_ext_type_mutex);
+#define NF_CT_EXT_PREALLOC 128u /* conntrack events are on by default */
-void __nf_ct_ext_destroy(struct nf_conn *ct)
+void nf_ct_ext_destroy(struct nf_conn *ct)
{
unsigned int i;
struct nf_ct_ext_type *t;
- struct nf_ct_ext *ext = ct->ext;
for (i = 0; i < NF_CT_EXT_NUM; i++) {
- if (!__nf_ct_ext_exist(ext, i))
- continue;
-
rcu_read_lock();
t = rcu_dereference(nf_ct_ext_types[i]);
@@ -41,54 +38,26 @@
rcu_read_unlock();
}
}
-EXPORT_SYMBOL(__nf_ct_ext_destroy);
+EXPORT_SYMBOL(nf_ct_ext_destroy);
-static void *
-nf_ct_ext_create(struct nf_ct_ext **ext, enum nf_ct_ext_id id,
- size_t var_alloc_len, gfp_t gfp)
+void *nf_ct_ext_add(struct nf_conn *ct, enum nf_ct_ext_id id, gfp_t gfp)
{
- unsigned int off, len;
- struct nf_ct_ext_type *t;
- size_t alloc_size;
-
- rcu_read_lock();
- t = rcu_dereference(nf_ct_ext_types[id]);
- if (!t) {
- rcu_read_unlock();
- return NULL;
- }
-
- off = ALIGN(sizeof(struct nf_ct_ext), t->align);
- len = off + t->len + var_alloc_len;
- alloc_size = t->alloc_size + var_alloc_len;
- rcu_read_unlock();
-
- *ext = kzalloc(alloc_size, gfp);
- if (!*ext)
- return NULL;
-
- (*ext)->offset[id] = off;
- (*ext)->len = len;
-
- return (void *)(*ext) + off;
-}
-
-void *__nf_ct_ext_add_length(struct nf_conn *ct, enum nf_ct_ext_id id,
- size_t var_alloc_len, gfp_t gfp)
-{
+ unsigned int newlen, newoff, oldlen, alloc;
struct nf_ct_ext *old, *new;
- int newlen, newoff;
struct nf_ct_ext_type *t;
/* Conntrack must not be confirmed to avoid races on reallocation. */
NF_CT_ASSERT(!nf_ct_is_confirmed(ct));
old = ct->ext;
- if (!old)
- return nf_ct_ext_create(&ct->ext, id, var_alloc_len, gfp);
- if (__nf_ct_ext_exist(old, id))
- return NULL;
+ if (old) {
+ if (__nf_ct_ext_exist(old, id))
+ return NULL;
+ oldlen = old->len;
+ } else {
+ oldlen = sizeof(*new);
+ }
rcu_read_lock();
t = rcu_dereference(nf_ct_ext_types[id]);
@@ -97,15 +66,19 @@
return NULL;
}
- newoff = ALIGN(old->len, t->align);
- newlen = newoff + t->len + var_alloc_len;
+ newoff = ALIGN(oldlen, t->align);
+ newlen = newoff + t->len;
rcu_read_unlock();
- new = __krealloc(old, newlen, gfp);
+ alloc = max(newlen, NF_CT_EXT_PREALLOC);
+ new = __krealloc(old, alloc, gfp);
if (!new)
return NULL;
- if (new != old) {
+ if (!old) {
+ memset(new->offset, 0, sizeof(new->offset));
+ ct->ext = new;
+ } else if (new != old) {
kfree_rcu(old, rcu);
rcu_assign_pointer(ct->ext, new);
}
@@ -115,45 +88,10 @@
memset((void *)new + newoff, 0, newlen - newoff);
return (void *)new + newoff;
}
-EXPORT_SYMBOL(__nf_ct_ext_add_length);
-
-static void update_alloc_size(struct nf_ct_ext_type *type)
-{
- int i, j;
- struct nf_ct_ext_type *t1, *t2;
- enum nf_ct_ext_id min = 0, max = NF_CT_EXT_NUM - 1;
-
- /* unnecessary to update all types */
- if ((type->flags & NF_CT_EXT_F_PREALLOC) == 0) {
- min = type->id;
- max = type->id;
- }
-
- /* This assumes that extended areas in conntrack for the types
- whose NF_CT_EXT_F_PREALLOC bit set are allocated in order */
- for (i = min; i <= max; i++) {
- t1 = rcu_dereference_protected(nf_ct_ext_types[i],
- lockdep_is_held(&nf_ct_ext_type_mutex));
- if (!t1)
- continue;
-
- t1->alloc_size = ALIGN(sizeof(struct nf_ct_ext), t1->align) +
- t1->len;
- for (j = 0; j < NF_CT_EXT_NUM; j++) {
- t2 = rcu_dereference_protected(nf_ct_ext_types[j],
- lockdep_is_held(&nf_ct_ext_type_mutex));
- if (t2 == NULL || t2 == t1 ||
- (t2->flags & NF_CT_EXT_F_PREALLOC) == 0)
- continue;
-
- t1->alloc_size = ALIGN(t1->alloc_size, t2->align)
- + t2->len;
- }
- }
-}
+EXPORT_SYMBOL(nf_ct_ext_add);
/* This MUST be called in process context. */
-int nf_ct_extend_register(struct nf_ct_ext_type *type)
+int nf_ct_extend_register(const struct nf_ct_ext_type *type)
{
int ret = 0;
@@ -163,12 +101,7 @@
goto out;
}
- /* This ensures that nf_ct_ext_create() can allocate enough area
- before updating alloc_size */
- type->alloc_size = ALIGN(sizeof(struct nf_ct_ext), type->align)
- + type->len;
rcu_assign_pointer(nf_ct_ext_types[type->id], type);
- update_alloc_size(type);
out:
mutex_unlock(&nf_ct_ext_type_mutex);
return ret;
@@ -176,11 +109,10 @@
EXPORT_SYMBOL_GPL(nf_ct_extend_register);
/* This MUST be called in process context. */
-void nf_ct_extend_unregister(struct nf_ct_ext_type *type)
+void nf_ct_extend_unregister(const struct nf_ct_ext_type *type)
{
mutex_lock(&nf_ct_ext_type_mutex);
RCU_INIT_POINTER(nf_ct_ext_types[type->id], NULL);
- update_alloc_size(type);
mutex_unlock(&nf_ct_ext_type_mutex);
synchronize_rcu();
}
diff --git a/net/netfilter/nf_conntrack_ftp.c b/net/netfilter/nf_conntrack_ftp.c
index 4aecef4..f0e9a75 100644
--- a/net/netfilter/nf_conntrack_ftp.c
+++ b/net/netfilter/nf_conntrack_ftp.c
@@ -577,6 +577,8 @@
{
int i, ret = 0;
+ NF_CT_HELPER_BUILD_BUG_ON(sizeof(struct nf_ct_ftp_master));
+
ftp_buffer = kmalloc(65536, GFP_KERNEL);
if (!ftp_buffer)
return -ENOMEM;
@@ -589,12 +591,10 @@
for (i = 0; i < ports_c; i++) {
nf_ct_helper_init(&ftp[2 * i], AF_INET, IPPROTO_TCP, "ftp",
FTP_PORT, ports[i], ports[i], &ftp_exp_policy,
- 0, sizeof(struct nf_ct_ftp_master), help,
- nf_ct_ftp_from_nlattr, THIS_MODULE);
+ 0, help, nf_ct_ftp_from_nlattr, THIS_MODULE);
nf_ct_helper_init(&ftp[2 * i + 1], AF_INET6, IPPROTO_TCP, "ftp",
FTP_PORT, ports[i], ports[i], &ftp_exp_policy,
- 0, sizeof(struct nf_ct_ftp_master), help,
- nf_ct_ftp_from_nlattr, THIS_MODULE);
+ 0, help, nf_ct_ftp_from_nlattr, THIS_MODULE);
}
ret = nf_conntrack_helpers_register(ftp, ports_c * 2);
diff --git a/net/netfilter/nf_conntrack_h323_main.c b/net/netfilter/nf_conntrack_h323_main.c
index f65d9363..3bcdc71 100644
--- a/net/netfilter/nf_conntrack_h323_main.c
+++ b/net/netfilter/nf_conntrack_h323_main.c
@@ -637,7 +637,6 @@
static struct nf_conntrack_helper nf_conntrack_helper_h245 __read_mostly = {
.name = "H.245",
.me = THIS_MODULE,
- .data_len = sizeof(struct nf_ct_h323_master),
.tuple.src.l3num = AF_UNSPEC,
.tuple.dst.protonum = IPPROTO_UDP,
.help = h245_help,
@@ -1215,7 +1214,6 @@
{
.name = "Q.931",
.me = THIS_MODULE,
- .data_len = sizeof(struct nf_ct_h323_master),
.tuple.src.l3num = AF_INET,
.tuple.src.u.tcp.port = cpu_to_be16(Q931_PORT),
.tuple.dst.protonum = IPPROTO_TCP,
@@ -1800,7 +1798,6 @@
{
.name = "RAS",
.me = THIS_MODULE,
- .data_len = sizeof(struct nf_ct_h323_master),
.tuple.src.l3num = AF_INET,
.tuple.src.u.udp.port = cpu_to_be16(RAS_PORT),
.tuple.dst.protonum = IPPROTO_UDP,
@@ -1810,7 +1807,6 @@
{
.name = "RAS",
.me = THIS_MODULE,
- .data_len = sizeof(struct nf_ct_h323_master),
.tuple.src.l3num = AF_INET6,
.tuple.src.u.udp.port = cpu_to_be16(RAS_PORT),
.tuple.dst.protonum = IPPROTO_UDP,
@@ -1836,6 +1832,8 @@
{
int ret;
+ NF_CT_HELPER_BUILD_BUG_ON(sizeof(struct nf_ct_h323_master));
+
h323_buffer = kmalloc(65536, GFP_KERNEL);
if (!h323_buffer)
return -ENOMEM;
diff --git a/net/netfilter/nf_conntrack_helper.c b/net/netfilter/nf_conntrack_helper.c
index 99bcd44..3a60efa 100644
--- a/net/netfilter/nf_conntrack_helper.c
+++ b/net/netfilter/nf_conntrack_helper.c
@@ -187,8 +187,7 @@
{
struct nf_conn_help *help;
- help = nf_ct_ext_add_length(ct, NF_CT_EXT_HELPER,
- helper->data_len, gfp);
+ help = nf_ct_ext_add(ct, NF_CT_EXT_HELPER, gfp);
if (help)
INIT_HLIST_HEAD(&help->expectations);
else
@@ -392,6 +391,9 @@
BUG_ON(me->expect_class_max >= NF_CT_MAX_EXPECT_CLASSES);
BUG_ON(strlen(me->name) > NF_CT_HELPER_NAME_LEN - 1);
+ if (me->expect_policy->max_expected > NF_CT_EXPECT_MAX_CNT)
+ return -EINVAL;
+
mutex_lock(&nf_ct_helper_mutex);
for (i = 0; i < nf_ct_helper_hsize; i++) {
hlist_for_each_entry(cur, &nf_ct_helper_hash[i], hnode) {
@@ -471,11 +473,8 @@
if ((rcu_dereference_protected(
help->helper,
lockdep_is_held(&nf_conntrack_expect_lock)
- ) == me || exp->helper == me) &&
- del_timer(&exp->timeout)) {
- nf_ct_unlink_expect(exp);
- nf_ct_expect_put(exp);
- }
+ ) == me || exp->helper == me))
+ nf_ct_remove_expect(exp);
}
}
spin_unlock_bh(&nf_conntrack_expect_lock);
@@ -507,7 +506,7 @@
u16 l3num, u16 protonum, const char *name,
u16 default_port, u16 spec_port, u32 id,
const struct nf_conntrack_expect_policy *exp_pol,
- u32 expect_class_max, u32 data_len,
+ u32 expect_class_max,
int (*help)(struct sk_buff *skb, unsigned int protoff,
struct nf_conn *ct,
enum ip_conntrack_info ctinfo),
@@ -520,7 +519,6 @@
helper->tuple.src.u.all = htons(spec_port);
helper->expect_policy = exp_pol;
helper->expect_class_max = expect_class_max;
- helper->data_len = data_len;
helper->help = help;
helper->from_nlattr = from_nlattr;
helper->me = module;
@@ -560,7 +558,7 @@
}
EXPORT_SYMBOL_GPL(nf_conntrack_helpers_unregister);
-static struct nf_ct_ext_type helper_extend __read_mostly = {
+static const struct nf_ct_ext_type helper_extend = {
.len = sizeof(struct nf_conn_help),
.align = __alignof__(struct nf_conn_help),
.id = NF_CT_EXT_HELPER,
diff --git a/net/netfilter/nf_conntrack_irc.c b/net/netfilter/nf_conntrack_irc.c
index 1972a14..5523acc 100644
--- a/net/netfilter/nf_conntrack_irc.c
+++ b/net/netfilter/nf_conntrack_irc.c
@@ -243,6 +243,12 @@
return -EINVAL;
}
+ if (max_dcc_channels > NF_CT_EXPECT_MAX_CNT) {
+ pr_err("max_dcc_channels must not be more than %u\n",
+ NF_CT_EXPECT_MAX_CNT);
+ return -EINVAL;
+ }
+
irc_exp_policy.max_expected = max_dcc_channels;
irc_exp_policy.timeout = dcc_timeout;
@@ -257,7 +263,7 @@
for (i = 0; i < ports_c; i++) {
nf_ct_helper_init(&irc[i], AF_INET, IPPROTO_TCP, "irc",
IRC_PORT, ports[i], i, &irc_exp_policy,
- 0, 0, help, NULL, THIS_MODULE);
+ 0, help, NULL, THIS_MODULE);
}
ret = nf_conntrack_helpers_register(&irc[0], ports_c);
diff --git a/net/netfilter/nf_conntrack_labels.c b/net/netfilter/nf_conntrack_labels.c
index bcab8bd..adf2198 100644
--- a/net/netfilter/nf_conntrack_labels.c
+++ b/net/netfilter/nf_conntrack_labels.c
@@ -82,7 +82,7 @@
}
EXPORT_SYMBOL_GPL(nf_connlabels_put);
-static struct nf_ct_ext_type labels_extend __read_mostly = {
+static const struct nf_ct_ext_type labels_extend = {
.len = sizeof(struct nf_conn_labels),
.align = __alignof__(struct nf_conn_labels),
.id = NF_CT_EXT_LABELS,
diff --git a/net/netfilter/nf_conntrack_netbios_ns.c b/net/netfilter/nf_conntrack_netbios_ns.c
index 4c8f30a..496ce17 100644
--- a/net/netfilter/nf_conntrack_netbios_ns.c
+++ b/net/netfilter/nf_conntrack_netbios_ns.c
@@ -58,6 +58,8 @@
static int __init nf_conntrack_netbios_ns_init(void)
{
+ NF_CT_HELPER_BUILD_BUG_ON(0);
+
exp_policy.timeout = timeout;
return nf_conntrack_helper_register(&helper);
}
diff --git a/net/netfilter/nf_conntrack_netlink.c b/net/netfilter/nf_conntrack_netlink.c
index 78f8c9a..dcf561b 100644
--- a/net/netfilter/nf_conntrack_netlink.c
+++ b/net/netfilter/nf_conntrack_netlink.c
@@ -471,7 +471,7 @@
struct nlattr *nest_parms;
unsigned int flags = portid ? NLM_F_MULTI : 0, event;
- event = (NFNL_SUBSYS_CTNETLINK << 8 | IPCTNL_MSG_CT_NEW);
+ event = nfnl_msg_type(NFNL_SUBSYS_CTNETLINK, IPCTNL_MSG_CT_NEW);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(*nfmsg), flags);
if (nlh == NULL)
goto nlmsg_failure;
@@ -631,10 +631,6 @@
unsigned int flags = 0, group;
int err;
- /* ignore our fake conntrack entry */
- if (nf_ct_is_untracked(ct))
- return 0;
-
if (events & (1 << IPCT_DESTROY)) {
type = IPCTNL_MSG_CT_DELETE;
group = NFNLGRP_CONNTRACK_DESTROY;
@@ -656,7 +652,7 @@
if (skb == NULL)
goto errout;
- type |= NFNL_SUBSYS_CTNETLINK << 8;
+ type = nfnl_msg_type(NFNL_SUBSYS_CTNETLINK, type);
nlh = nlmsg_put(skb, item->portid, 0, type, sizeof(*nfmsg), flags);
if (nlh == NULL)
goto nlmsg_failure;
@@ -912,7 +908,7 @@
struct nf_conntrack_l3proto *l3proto;
int ret = 0;
- ret = nla_parse_nested(tb, CTA_IP_MAX, attr, NULL);
+ ret = nla_parse_nested(tb, CTA_IP_MAX, attr, NULL, NULL);
if (ret < 0)
return ret;
@@ -921,7 +917,7 @@
if (likely(l3proto->nlattr_to_tuple)) {
ret = nla_validate_nested(attr, CTA_IP_MAX,
- l3proto->nla_policy);
+ l3proto->nla_policy, NULL);
if (ret == 0)
ret = l3proto->nlattr_to_tuple(tb, tuple);
}
@@ -942,7 +938,8 @@
struct nf_conntrack_l4proto *l4proto;
int ret = 0;
- ret = nla_parse_nested(tb, CTA_PROTO_MAX, attr, proto_nla_policy);
+ ret = nla_parse_nested(tb, CTA_PROTO_MAX, attr, proto_nla_policy,
+ NULL);
if (ret < 0)
return ret;
@@ -955,7 +952,7 @@
if (likely(l4proto->nlattr_to_tuple)) {
ret = nla_validate_nested(attr, CTA_PROTO_MAX,
- l4proto->nla_policy);
+ l4proto->nla_policy, NULL);
if (ret == 0)
ret = l4proto->nlattr_to_tuple(tb, tuple);
}
@@ -1019,7 +1016,8 @@
memset(tuple, 0, sizeof(*tuple));
- err = nla_parse_nested(tb, CTA_TUPLE_MAX, cda[type], tuple_nla_policy);
+ err = nla_parse_nested(tb, CTA_TUPLE_MAX, cda[type], tuple_nla_policy,
+ NULL);
if (err < 0)
return err;
@@ -1069,7 +1067,7 @@
int err;
struct nlattr *tb[CTA_HELP_MAX+1];
- err = nla_parse_nested(tb, CTA_HELP_MAX, attr, help_nla_policy);
+ err = nla_parse_nested(tb, CTA_HELP_MAX, attr, help_nla_policy, NULL);
if (err < 0)
return err;
@@ -1581,7 +1579,8 @@
struct nf_conntrack_l4proto *l4proto;
int err = 0;
- err = nla_parse_nested(tb, CTA_PROTOINFO_MAX, attr, protoinfo_policy);
+ err = nla_parse_nested(tb, CTA_PROTOINFO_MAX, attr, protoinfo_policy,
+ NULL);
if (err < 0)
return err;
@@ -1606,7 +1605,7 @@
int err;
struct nlattr *cda[CTA_SEQADJ_MAX+1];
- err = nla_parse_nested(cda, CTA_SEQADJ_MAX, attr, seqadj_policy);
+ err = nla_parse_nested(cda, CTA_SEQADJ_MAX, attr, seqadj_policy, NULL);
if (err < 0)
return err;
@@ -2001,7 +2000,8 @@
struct nfgenmsg *nfmsg;
unsigned int flags = portid ? NLM_F_MULTI : 0, event;
- event = (NFNL_SUBSYS_CTNETLINK << 8 | IPCTNL_MSG_CT_GET_STATS_CPU);
+ event = nfnl_msg_type(NFNL_SUBSYS_CTNETLINK,
+ IPCTNL_MSG_CT_GET_STATS_CPU);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(*nfmsg), flags);
if (nlh == NULL)
goto nlmsg_failure;
@@ -2084,7 +2084,7 @@
unsigned int flags = portid ? NLM_F_MULTI : 0, event;
unsigned int nr_conntracks = atomic_read(&net->ct.count);
- event = (NFNL_SUBSYS_CTNETLINK << 8 | IPCTNL_MSG_CT_GET_STATS);
+ event = nfnl_msg_type(NFNL_SUBSYS_CTNETLINK, IPCTNL_MSG_CT_GET_STATS);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(*nfmsg), flags);
if (nlh == NULL)
goto nlmsg_failure;
@@ -2190,13 +2190,7 @@
static struct nf_conn *ctnetlink_glue_get_ct(const struct sk_buff *skb,
enum ip_conntrack_info *ctinfo)
{
- struct nf_conn *ct;
-
- ct = nf_ct_get(skb, ctinfo);
- if (ct && nf_ct_is_untracked(ct))
- ct = NULL;
-
- return ct;
+ return nf_ct_get(skb, ctinfo);
}
static int __ctnetlink_glue_build(struct sk_buff *skb, struct nf_conn *ct)
@@ -2366,7 +2360,7 @@
struct nlattr *cda[CTA_MAX+1];
int ret;
- ret = nla_parse_nested(cda, CTA_MAX, attr, ct_nla_policy);
+ ret = nla_parse_nested(cda, CTA_MAX, attr, ct_nla_policy, NULL);
if (ret < 0)
return ret;
@@ -2399,7 +2393,8 @@
struct nf_conntrack_expect *exp;
int err;
- err = nla_parse_nested(cda, CTA_EXPECT_MAX, attr, exp_nla_policy);
+ err = nla_parse_nested(cda, CTA_EXPECT_MAX, attr, exp_nla_policy,
+ NULL);
if (err < 0)
return err;
@@ -2590,7 +2585,7 @@
struct nfgenmsg *nfmsg;
unsigned int flags = portid ? NLM_F_MULTI : 0;
- event |= NFNL_SUBSYS_CTNETLINK_EXP << 8;
+ event = nfnl_msg_type(NFNL_SUBSYS_CTNETLINK_EXP, event);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(*nfmsg), flags);
if (nlh == NULL)
goto nlmsg_failure;
@@ -2641,7 +2636,7 @@
if (skb == NULL)
goto errout;
- type |= NFNL_SUBSYS_CTNETLINK_EXP << 8;
+ type = nfnl_msg_type(NFNL_SUBSYS_CTNETLINK_EXP, type);
nlh = nlmsg_put(skb, item->portid, 0, type, sizeof(*nfmsg), flags);
if (nlh == NULL)
goto nlmsg_failure;
@@ -2707,7 +2702,7 @@
cb->nlh->nlmsg_seq,
IPCTNL_MSG_EXP_NEW,
exp) < 0) {
- if (!atomic_inc_not_zero(&exp->use))
+ if (!refcount_inc_not_zero(&exp->use))
continue;
cb->args[1] = (unsigned long)exp;
goto out;
@@ -2753,7 +2748,7 @@
cb->nlh->nlmsg_seq,
IPCTNL_MSG_EXP_NEW,
exp) < 0) {
- if (!atomic_inc_not_zero(&exp->use))
+ if (!refcount_inc_not_zero(&exp->use))
continue;
cb->args[1] = (unsigned long)exp;
goto out;
@@ -3024,7 +3019,8 @@
struct nf_conntrack_tuple nat_tuple = {};
int err;
- err = nla_parse_nested(tb, CTA_EXPECT_NAT_MAX, attr, exp_nat_nla_policy);
+ err = nla_parse_nested(tb, CTA_EXPECT_NAT_MAX, attr,
+ exp_nat_nla_policy, NULL);
if (err < 0)
return err;
@@ -3058,6 +3054,10 @@
struct nf_conn_help *help;
int err;
+ help = nfct_help(ct);
+ if (!help)
+ return ERR_PTR(-EOPNOTSUPP);
+
if (cda[CTA_EXPECT_CLASS] && helper) {
class = ntohl(nla_get_be32(cda[CTA_EXPECT_CLASS]));
if (class > helper->expect_class_max)
@@ -3067,26 +3067,11 @@
if (!exp)
return ERR_PTR(-ENOMEM);
- help = nfct_help(ct);
- if (!help) {
- if (!cda[CTA_EXPECT_TIMEOUT]) {
- err = -EINVAL;
- goto err_out;
- }
- exp->timeout.expires =
- jiffies + ntohl(nla_get_be32(cda[CTA_EXPECT_TIMEOUT])) * HZ;
-
- exp->flags = NF_CT_EXPECT_USERSPACE;
- if (cda[CTA_EXPECT_FLAGS]) {
- exp->flags |=
- ntohl(nla_get_be32(cda[CTA_EXPECT_FLAGS]));
- }
+ if (cda[CTA_EXPECT_FLAGS]) {
+ exp->flags = ntohl(nla_get_be32(cda[CTA_EXPECT_FLAGS]));
+ exp->flags &= ~NF_CT_EXPECT_USERSPACE;
} else {
- if (cda[CTA_EXPECT_FLAGS]) {
- exp->flags = ntohl(nla_get_be32(cda[CTA_EXPECT_FLAGS]));
- exp->flags &= ~NF_CT_EXPECT_USERSPACE;
- } else
- exp->flags = 0;
+ exp->flags = 0;
}
if (cda[CTA_EXPECT_FN]) {
const char *name = nla_data(cda[CTA_EXPECT_FN]);
@@ -3249,7 +3234,8 @@
struct nfgenmsg *nfmsg;
unsigned int flags = portid ? NLM_F_MULTI : 0, event;
- event = (NFNL_SUBSYS_CTNETLINK << 8 | IPCTNL_MSG_EXP_GET_STATS_CPU);
+ event = nfnl_msg_type(NFNL_SUBSYS_CTNETLINK,
+ IPCTNL_MSG_EXP_GET_STATS_CPU);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(*nfmsg), flags);
if (nlh == NULL)
goto nlmsg_failure;
diff --git a/net/netfilter/nf_conntrack_pptp.c b/net/netfilter/nf_conntrack_pptp.c
index f60a475..6959e93 100644
--- a/net/netfilter/nf_conntrack_pptp.c
+++ b/net/netfilter/nf_conntrack_pptp.c
@@ -263,7 +263,7 @@
goto out_put_both;
}
-static inline int
+static int
pptp_inbound_pkt(struct sk_buff *skb, unsigned int protoff,
struct PptpControlHeader *ctlh,
union pptp_ctrl_union *pptpReq,
@@ -391,7 +391,7 @@
return NF_ACCEPT;
}
-static inline int
+static int
pptp_outbound_pkt(struct sk_buff *skb, unsigned int protoff,
struct PptpControlHeader *ctlh,
union pptp_ctrl_union *pptpReq,
@@ -523,6 +523,14 @@
int ret;
u_int16_t msg;
+#if IS_ENABLED(CONFIG_NF_NAT)
+ if (!nf_ct_is_confirmed(ct) && (ct->status & IPS_NAT_MASK)) {
+ struct nf_conn_nat *nat = nf_ct_ext_find(ct, NF_CT_EXT_NAT);
+
+ if (!nat && !nf_ct_ext_add(ct, NF_CT_EXT_NAT, GFP_ATOMIC))
+ return NF_DROP;
+ }
+#endif
/* don't do any tracking before tcp handshake complete */
if (ctinfo != IP_CT_ESTABLISHED && ctinfo != IP_CT_ESTABLISHED_REPLY)
return NF_ACCEPT;
@@ -596,7 +604,6 @@
static struct nf_conntrack_helper pptp __read_mostly = {
.name = "pptp",
.me = THIS_MODULE,
- .data_len = sizeof(struct nf_ct_pptp_master),
.tuple.src.l3num = AF_INET,
.tuple.src.u.tcp.port = cpu_to_be16(PPTP_CONTROL_PORT),
.tuple.dst.protonum = IPPROTO_TCP,
@@ -607,6 +614,8 @@
static int __init nf_conntrack_pptp_init(void)
{
+ NF_CT_HELPER_BUILD_BUG_ON(sizeof(struct nf_ct_pptp_master));
+
return nf_conntrack_helper_register(&pptp);
}
diff --git a/net/netfilter/nf_conntrack_proto.c b/net/netfilter/nf_conntrack_proto.c
index 2d6ee18..2de6c1f 100644
--- a/net/netfilter/nf_conntrack_proto.c
+++ b/net/netfilter/nf_conntrack_proto.c
@@ -202,7 +202,7 @@
static int kill_l4proto(struct nf_conn *i, void *data)
{
struct nf_conntrack_l4proto *l4proto;
- l4proto = (struct nf_conntrack_l4proto *)data;
+ l4proto = data;
return nf_ct_protonum(i) == l4proto->l4proto &&
nf_ct_l3num(i) == l4proto->l3proto;
}
@@ -441,9 +441,8 @@
void nf_ct_l4proto_pernet_unregister_one(struct net *net,
struct nf_conntrack_l4proto *l4proto)
{
- struct nf_proto_net *pn = NULL;
+ struct nf_proto_net *pn = nf_ct_l4proto_net(net, l4proto);
- pn = nf_ct_l4proto_net(net, l4proto);
if (pn == NULL)
return;
diff --git a/net/netfilter/nf_conntrack_proto_dccp.c b/net/netfilter/nf_conntrack_proto_dccp.c
index 93dd1c5..b553fdd 100644
--- a/net/netfilter/nf_conntrack_proto_dccp.c
+++ b/net/netfilter/nf_conntrack_proto_dccp.c
@@ -609,6 +609,20 @@
return -NF_ACCEPT;
}
+static bool dccp_can_early_drop(const struct nf_conn *ct)
+{
+ switch (ct->proto.dccp.state) {
+ case CT_DCCP_CLOSEREQ:
+ case CT_DCCP_CLOSING:
+ case CT_DCCP_TIMEWAIT:
+ return true;
+ default:
+ break;
+ }
+
+ return false;
+}
+
static void dccp_print_tuple(struct seq_file *s,
const struct nf_conntrack_tuple *tuple)
{
@@ -665,7 +679,7 @@
return 0;
err = nla_parse_nested(tb, CTA_PROTOINFO_DCCP_MAX, attr,
- dccp_nla_policy);
+ dccp_nla_policy, NULL);
if (err < 0)
return err;
@@ -868,6 +882,7 @@
.packet = dccp_packet,
.get_timeouts = dccp_get_timeouts,
.error = dccp_error,
+ .can_early_drop = dccp_can_early_drop,
.print_tuple = dccp_print_tuple,
.print_conntrack = dccp_print_conntrack,
#if IS_ENABLED(CONFIG_NF_CT_NETLINK)
@@ -902,6 +917,7 @@
.packet = dccp_packet,
.get_timeouts = dccp_get_timeouts,
.error = dccp_error,
+ .can_early_drop = dccp_can_early_drop,
.print_tuple = dccp_print_tuple,
.print_conntrack = dccp_print_conntrack,
#if IS_ENABLED(CONFIG_NF_CT_NETLINK)
diff --git a/net/netfilter/nf_conntrack_proto_sctp.c b/net/netfilter/nf_conntrack_proto_sctp.c
index 33279aa..13875d5 100644
--- a/net/netfilter/nf_conntrack_proto_sctp.c
+++ b/net/netfilter/nf_conntrack_proto_sctp.c
@@ -535,6 +535,20 @@
return -NF_ACCEPT;
}
+static bool sctp_can_early_drop(const struct nf_conn *ct)
+{
+ switch (ct->proto.sctp.state) {
+ case SCTP_CONNTRACK_SHUTDOWN_SENT:
+ case SCTP_CONNTRACK_SHUTDOWN_RECD:
+ case SCTP_CONNTRACK_SHUTDOWN_ACK_SENT:
+ return true;
+ default:
+ break;
+ }
+
+ return false;
+}
+
#if IS_ENABLED(CONFIG_NF_CT_NETLINK)
#include <linux/netfilter/nfnetlink.h>
@@ -584,10 +598,8 @@
if (!attr)
return 0;
- err = nla_parse_nested(tb,
- CTA_PROTOINFO_SCTP_MAX,
- attr,
- sctp_nla_policy);
+ err = nla_parse_nested(tb, CTA_PROTOINFO_SCTP_MAX, attr,
+ sctp_nla_policy, NULL);
if (err < 0)
return err;
@@ -783,6 +795,7 @@
.get_timeouts = sctp_get_timeouts,
.new = sctp_new,
.error = sctp_error,
+ .can_early_drop = sctp_can_early_drop,
.me = THIS_MODULE,
#if IS_ENABLED(CONFIG_NF_CT_NETLINK)
.to_nlattr = sctp_to_nlattr,
@@ -818,6 +831,7 @@
.get_timeouts = sctp_get_timeouts,
.new = sctp_new,
.error = sctp_error,
+ .can_early_drop = sctp_can_early_drop,
.me = THIS_MODULE,
#if IS_ENABLED(CONFIG_NF_CT_NETLINK)
.to_nlattr = sctp_to_nlattr,
diff --git a/net/netfilter/nf_conntrack_proto_tcp.c b/net/netfilter/nf_conntrack_proto_tcp.c
index b122e9d..9758a7d 100644
--- a/net/netfilter/nf_conntrack_proto_tcp.c
+++ b/net/netfilter/nf_conntrack_proto_tcp.c
@@ -419,10 +419,9 @@
&& opsize == TCPOLEN_WINDOW) {
state->td_scale = *(u_int8_t *)ptr;
- if (state->td_scale > 14) {
- /* See RFC1323 */
- state->td_scale = 14;
- }
+ if (state->td_scale > TCP_MAX_WSCALE)
+ state->td_scale = TCP_MAX_WSCALE;
+
state->flags |=
IP_CT_TCP_FLAG_WINDOW_SCALE;
}
@@ -1172,6 +1171,22 @@
return true;
}
+static bool tcp_can_early_drop(const struct nf_conn *ct)
+{
+ switch (ct->proto.tcp.state) {
+ case TCP_CONNTRACK_FIN_WAIT:
+ case TCP_CONNTRACK_LAST_ACK:
+ case TCP_CONNTRACK_TIME_WAIT:
+ case TCP_CONNTRACK_CLOSE:
+ case TCP_CONNTRACK_CLOSE_WAIT:
+ return true;
+ default:
+ break;
+ }
+
+ return false;
+}
+
#if IS_ENABLED(CONFIG_NF_CT_NETLINK)
#include <linux/netfilter/nfnetlink.h>
@@ -1234,7 +1249,8 @@
if (!pattr)
return 0;
- err = nla_parse_nested(tb, CTA_PROTOINFO_TCP_MAX, pattr, tcp_nla_policy);
+ err = nla_parse_nested(tb, CTA_PROTOINFO_TCP_MAX, pattr,
+ tcp_nla_policy, NULL);
if (err < 0)
return err;
@@ -1549,6 +1565,7 @@
.get_timeouts = tcp_get_timeouts,
.new = tcp_new,
.error = tcp_error,
+ .can_early_drop = tcp_can_early_drop,
#if IS_ENABLED(CONFIG_NF_CT_NETLINK)
.to_nlattr = tcp_to_nlattr,
.nlattr_size = tcp_nlattr_size,
@@ -1586,6 +1603,7 @@
.get_timeouts = tcp_get_timeouts,
.new = tcp_new,
.error = tcp_error,
+ .can_early_drop = tcp_can_early_drop,
#if IS_ENABLED(CONFIG_NF_CT_NETLINK)
.to_nlattr = tcp_to_nlattr,
.nlattr_size = tcp_nlattr_size,
diff --git a/net/netfilter/nf_conntrack_sane.c b/net/netfilter/nf_conntrack_sane.c
index 9dcb9ee..ae457f3 100644
--- a/net/netfilter/nf_conntrack_sane.c
+++ b/net/netfilter/nf_conntrack_sane.c
@@ -184,6 +184,8 @@
{
int i, ret = 0;
+ NF_CT_HELPER_BUILD_BUG_ON(sizeof(struct nf_ct_sane_master));
+
sane_buffer = kmalloc(65536, GFP_KERNEL);
if (!sane_buffer)
return -ENOMEM;
@@ -196,13 +198,11 @@
for (i = 0; i < ports_c; i++) {
nf_ct_helper_init(&sane[2 * i], AF_INET, IPPROTO_TCP, "sane",
SANE_PORT, ports[i], ports[i],
- &sane_exp_policy, 0,
- sizeof(struct nf_ct_sane_master), help, NULL,
+ &sane_exp_policy, 0, help, NULL,
THIS_MODULE);
nf_ct_helper_init(&sane[2 * i + 1], AF_INET6, IPPROTO_TCP, "sane",
SANE_PORT, ports[i], ports[i],
- &sane_exp_policy, 0,
- sizeof(struct nf_ct_sane_master), help, NULL,
+ &sane_exp_policy, 0, help, NULL,
THIS_MODULE);
}
diff --git a/net/netfilter/nf_conntrack_seqadj.c b/net/netfilter/nf_conntrack_seqadj.c
index ef7063e..a975efd 100644
--- a/net/netfilter/nf_conntrack_seqadj.c
+++ b/net/netfilter/nf_conntrack_seqadj.c
@@ -231,7 +231,7 @@
}
EXPORT_SYMBOL_GPL(nf_ct_seq_offset);
-static struct nf_ct_ext_type nf_ct_seqadj_extend __read_mostly = {
+static const struct nf_ct_ext_type nf_ct_seqadj_extend = {
.len = sizeof(struct nf_conn_seqadj),
.align = __alignof__(struct nf_conn_seqadj),
.id = NF_CT_EXT_SEQADJ,
diff --git a/net/netfilter/nf_conntrack_sip.c b/net/netfilter/nf_conntrack_sip.c
index 0d17894..d38af42 100644
--- a/net/netfilter/nf_conntrack_sip.c
+++ b/net/netfilter/nf_conntrack_sip.c
@@ -829,10 +829,8 @@
hlist_for_each_entry_safe(exp, next, &help->expectations, lnode) {
if ((exp->class != SIP_EXPECT_SIGNALLING) ^ media)
continue;
- if (!del_timer(&exp->timeout))
+ if (!nf_ct_remove_expect(exp))
continue;
- nf_ct_unlink_expect(exp);
- nf_ct_expect_put(exp);
if (!media)
break;
}
@@ -1624,29 +1622,27 @@
{
int i, ret;
+ NF_CT_HELPER_BUILD_BUG_ON(sizeof(struct nf_ct_sip_master));
+
if (ports_c == 0)
ports[ports_c++] = SIP_PORT;
for (i = 0; i < ports_c; i++) {
nf_ct_helper_init(&sip[4 * i], AF_INET, IPPROTO_UDP, "sip",
SIP_PORT, ports[i], i, sip_exp_policy,
- SIP_EXPECT_MAX,
- sizeof(struct nf_ct_sip_master), sip_help_udp,
+ SIP_EXPECT_MAX, sip_help_udp,
NULL, THIS_MODULE);
nf_ct_helper_init(&sip[4 * i + 1], AF_INET, IPPROTO_TCP, "sip",
SIP_PORT, ports[i], i, sip_exp_policy,
- SIP_EXPECT_MAX,
- sizeof(struct nf_ct_sip_master), sip_help_tcp,
+ SIP_EXPECT_MAX, sip_help_tcp,
NULL, THIS_MODULE);
nf_ct_helper_init(&sip[4 * i + 2], AF_INET6, IPPROTO_UDP, "sip",
SIP_PORT, ports[i], i, sip_exp_policy,
- SIP_EXPECT_MAX,
- sizeof(struct nf_ct_sip_master), sip_help_udp,
+ SIP_EXPECT_MAX, sip_help_udp,
NULL, THIS_MODULE);
nf_ct_helper_init(&sip[4 * i + 3], AF_INET6, IPPROTO_TCP, "sip",
SIP_PORT, ports[i], i, sip_exp_policy,
- SIP_EXPECT_MAX,
- sizeof(struct nf_ct_sip_master), sip_help_tcp,
+ SIP_EXPECT_MAX, sip_help_tcp,
NULL, THIS_MODULE);
}
diff --git a/net/netfilter/nf_conntrack_standalone.c b/net/netfilter/nf_conntrack_standalone.c
index 2256147..ccb5cb9 100644
--- a/net/netfilter/nf_conntrack_standalone.c
+++ b/net/netfilter/nf_conntrack_standalone.c
@@ -250,7 +250,7 @@
goto release;
if (!(test_bit(IPS_SEEN_REPLY_BIT, &ct->status)))
- seq_printf(s, "[UNREPLIED] ");
+ seq_puts(s, "[UNREPLIED] ");
print_tuple(s, &ct->tuplehash[IP_CT_DIR_REPLY].tuple,
l3proto, l4proto);
@@ -261,7 +261,7 @@
goto release;
if (test_bit(IPS_ASSURED_BIT, &ct->status))
- seq_printf(s, "[ASSURED] ");
+ seq_puts(s, "[ASSURED] ");
if (seq_has_overflowed(s))
goto release;
@@ -350,7 +350,7 @@
const struct ip_conntrack_stat *st = v;
if (v == SEQ_START_TOKEN) {
- seq_printf(seq, "entries searched found new invalid ignore delete delete_list insert insert_failed drop early_drop icmp_error expect_new expect_create expect_delete search_restart\n");
+ seq_puts(seq, "entries searched found new invalid ignore delete delete_list insert insert_failed drop early_drop icmp_error expect_new expect_create expect_delete search_restart\n");
return 0;
}
diff --git a/net/netfilter/nf_conntrack_tftp.c b/net/netfilter/nf_conntrack_tftp.c
index b1227dc..0ec6779 100644
--- a/net/netfilter/nf_conntrack_tftp.c
+++ b/net/netfilter/nf_conntrack_tftp.c
@@ -113,16 +113,18 @@
{
int i, ret;
+ NF_CT_HELPER_BUILD_BUG_ON(0);
+
if (ports_c == 0)
ports[ports_c++] = TFTP_PORT;
for (i = 0; i < ports_c; i++) {
nf_ct_helper_init(&tftp[2 * i], AF_INET, IPPROTO_UDP, "tftp",
TFTP_PORT, ports[i], i, &tftp_exp_policy,
- 0, 0, tftp_help, NULL, THIS_MODULE);
+ 0, tftp_help, NULL, THIS_MODULE);
nf_ct_helper_init(&tftp[2 * i + 1], AF_INET6, IPPROTO_UDP, "tftp",
TFTP_PORT, ports[i], i, &tftp_exp_policy,
- 0, 0, tftp_help, NULL, THIS_MODULE);
+ 0, tftp_help, NULL, THIS_MODULE);
}
ret = nf_conntrack_helpers_register(tftp, ports_c * 2);
diff --git a/net/netfilter/nf_conntrack_timeout.c b/net/netfilter/nf_conntrack_timeout.c
index 26e7420..46aee65 100644
--- a/net/netfilter/nf_conntrack_timeout.c
+++ b/net/netfilter/nf_conntrack_timeout.c
@@ -31,7 +31,7 @@
void (*nf_ct_timeout_put_hook)(struct ctnl_timeout *timeout) __read_mostly;
EXPORT_SYMBOL_GPL(nf_ct_timeout_put_hook);
-static struct nf_ct_ext_type timeout_extend __read_mostly = {
+static const struct nf_ct_ext_type timeout_extend = {
.len = sizeof(struct nf_conn_timeout),
.align = __alignof__(struct nf_conn_timeout),
.id = NF_CT_EXT_TIMEOUT,
diff --git a/net/netfilter/nf_conntrack_timestamp.c b/net/netfilter/nf_conntrack_timestamp.c
index 7a394df..4c4734b 100644
--- a/net/netfilter/nf_conntrack_timestamp.c
+++ b/net/netfilter/nf_conntrack_timestamp.c
@@ -33,7 +33,7 @@
};
#endif /* CONFIG_SYSCTL */
-static struct nf_ct_ext_type tstamp_extend __read_mostly = {
+static const struct nf_ct_ext_type tstamp_extend = {
.len = sizeof(struct nf_conn_tstamp),
.align = __alignof__(struct nf_conn_tstamp),
.id = NF_CT_EXT_TSTAMP,
diff --git a/net/netfilter/nf_internals.h b/net/netfilter/nf_internals.h
index c46d214..bfa742d 100644
--- a/net/netfilter/nf_internals.h
+++ b/net/netfilter/nf_internals.h
@@ -14,7 +14,7 @@
/* nf_queue.c */
int nf_queue(struct sk_buff *skb, struct nf_hook_state *state,
struct nf_hook_entry **entryp, unsigned int verdict);
-void nf_queue_nf_hook_drop(struct net *net, const struct nf_hook_entry *entry);
+unsigned int nf_queue_nf_hook_drop(struct net *net);
int __init netfilter_queue_init(void);
/* nf_log.c */
diff --git a/net/netfilter/nf_log.c b/net/netfilter/nf_log.c
index 8d85a05..8bb152a 100644
--- a/net/netfilter/nf_log.c
+++ b/net/netfilter/nf_log.c
@@ -71,7 +71,6 @@
RCU_INIT_POINTER(net->nf.nf_loggers[i], NULL);
}
mutex_unlock(&nf_log_mutex);
- synchronize_rcu();
}
EXPORT_SYMBOL(nf_log_unset);
@@ -376,13 +375,13 @@
logger = nft_log_dereference(loggers[*pos][i]);
seq_printf(s, "%s", logger->name);
if (i == 0 && loggers[*pos][i + 1] != NULL)
- seq_printf(s, ",");
+ seq_puts(s, ",");
if (seq_has_overflowed(s))
return -ENOSPC;
}
- seq_printf(s, ")\n");
+ seq_puts(s, ")\n");
if (seq_has_overflowed(s))
return -ENOSPC;
diff --git a/net/netfilter/nf_nat_amanda.c b/net/netfilter/nf_nat_amanda.c
index eb77238..e4d61a7 100644
--- a/net/netfilter/nf_nat_amanda.c
+++ b/net/netfilter/nf_nat_amanda.c
@@ -33,7 +33,6 @@
{
char buffer[sizeof("65535")];
u_int16_t port;
- unsigned int ret;
/* Connection comes from client. */
exp->saved_proto.tcp.port = exp->tuple.dst.u.tcp.port;
@@ -63,14 +62,14 @@
}
sprintf(buffer, "%u", port);
- ret = nf_nat_mangle_udp_packet(skb, exp->master, ctinfo,
- protoff, matchoff, matchlen,
- buffer, strlen(buffer));
- if (ret != NF_ACCEPT) {
+ if (!nf_nat_mangle_udp_packet(skb, exp->master, ctinfo,
+ protoff, matchoff, matchlen,
+ buffer, strlen(buffer))) {
nf_ct_helper_log(skb, exp->master, "cannot mangle packet");
nf_ct_unexpect_related(exp);
+ return NF_DROP;
}
- return ret;
+ return NF_ACCEPT;
}
static void __exit nf_nat_amanda_fini(void)
diff --git a/net/netfilter/nf_nat_core.c b/net/netfilter/nf_nat_core.c
index 82802e4..b48d6b5 100644
--- a/net/netfilter/nf_nat_core.c
+++ b/net/netfilter/nf_nat_core.c
@@ -71,11 +71,10 @@
if (ct == NULL)
return;
- family = ct->tuplehash[IP_CT_DIR_ORIGINAL].tuple.src.l3num;
- rcu_read_lock();
+ family = nf_ct_l3num(ct);
l3proto = __nf_nat_l3proto_find(family);
if (l3proto == NULL)
- goto out;
+ return;
dir = CTINFO2DIR(ctinfo);
if (dir == IP_CT_DIR_ORIGINAL)
@@ -84,8 +83,6 @@
statusbit = IPS_SRC_NAT;
l3proto->decode_session(skb, ct, dir, statusbit, fl);
-out:
- rcu_read_unlock();
}
int nf_xfrm_me_harder(struct net *net, struct sk_buff *skb, unsigned int family)
@@ -411,12 +408,6 @@
enum nf_nat_manip_type maniptype)
{
struct nf_conntrack_tuple curr_tuple, new_tuple;
- struct nf_conn_nat *nat;
-
- /* nat helper or nfctnetlink also setup binding */
- nat = nf_ct_nat_ext_add(ct);
- if (nat == NULL)
- return NF_ACCEPT;
NF_CT_ASSERT(maniptype == NF_NAT_MANIP_SRC ||
maniptype == NF_NAT_MANIP_DST);
@@ -549,10 +540,6 @@
static int nf_nat_proto_remove(struct nf_conn *i, void *data)
{
const struct nf_nat_proto_clean *clean = data;
- struct nf_conn_nat *nat = nfct_nat(i);
-
- if (!nat)
- return 0;
if ((clean->l3proto && nf_ct_l3num(i) != clean->l3proto) ||
(clean->l4proto && nf_ct_protonum(i) != clean->l4proto))
@@ -563,12 +550,10 @@
static int nf_nat_proto_clean(struct nf_conn *ct, void *data)
{
- struct nf_conn_nat *nat = nfct_nat(ct);
-
if (nf_nat_proto_remove(ct, data))
return 1;
- if (!nat)
+ if ((ct->status & IPS_SRC_NAT_DONE) == 0)
return 0;
/* This netns is being destroyed, and conntrack has nat null binding.
@@ -716,13 +701,9 @@
/* No one using conntrack by the time this called. */
static void nf_nat_cleanup_conntrack(struct nf_conn *ct)
{
- struct nf_conn_nat *nat = nf_ct_ext_find(ct, NF_CT_EXT_NAT);
-
- if (!nat)
- return;
-
- rhltable_remove(&nf_nat_bysource_table, &ct->nat_bysource,
- nf_nat_bysource_params);
+ if (ct->status & IPS_SRC_NAT_DONE)
+ rhltable_remove(&nf_nat_bysource_table, &ct->nat_bysource,
+ nf_nat_bysource_params);
}
static struct nf_ct_ext_type nat_extend __read_mostly = {
@@ -730,7 +711,6 @@
.align = __alignof__(struct nf_conn_nat),
.destroy = nf_nat_cleanup_conntrack,
.id = NF_CT_EXT_NAT,
- .flags = NF_CT_EXT_F_PREALLOC,
};
#if IS_ENABLED(CONFIG_NF_CT_NETLINK)
@@ -751,7 +731,8 @@
const struct nf_nat_l4proto *l4proto;
int err;
- err = nla_parse_nested(tb, CTA_PROTONAT_MAX, attr, protonat_nla_policy);
+ err = nla_parse_nested(tb, CTA_PROTONAT_MAX, attr,
+ protonat_nla_policy, NULL);
if (err < 0)
return err;
@@ -780,7 +761,7 @@
memset(range, 0, sizeof(*range));
- err = nla_parse_nested(tb, CTA_NAT_MAX, nat, nat_nla_policy);
+ err = nla_parse_nested(tb, CTA_NAT_MAX, nat, nat_nla_policy, NULL);
if (err < 0)
return err;
@@ -819,7 +800,7 @@
/* No NAT information has been passed, allocate the null-binding */
if (attr == NULL)
- return __nf_nat_alloc_null_binding(ct, manip);
+ return __nf_nat_alloc_null_binding(ct, manip) == NF_DROP ? -ENOMEM : 0;
err = nfnetlink_parse_nat(attr, ct, &range, l3proto);
if (err < 0)
@@ -874,9 +855,6 @@
nf_ct_helper_expectfn_register(&follow_master_nat);
- /* Initialize fake conntrack so that NAT will skip it */
- nf_ct_untracked_status_or(IPS_NAT_DONE_MASK);
-
BUG_ON(nfnetlink_parse_nat_setup_hook != NULL);
RCU_INIT_POINTER(nfnetlink_parse_nat_setup_hook,
nfnetlink_parse_nat_setup);
diff --git a/net/netfilter/nf_nat_helper.c b/net/netfilter/nf_nat_helper.c
index 211661c..607a373 100644
--- a/net/netfilter/nf_nat_helper.c
+++ b/net/netfilter/nf_nat_helper.c
@@ -70,15 +70,15 @@
}
/* Unusual, but possible case. */
-static int enlarge_skb(struct sk_buff *skb, unsigned int extra)
+static bool enlarge_skb(struct sk_buff *skb, unsigned int extra)
{
if (skb->len + extra > 65535)
- return 0;
+ return false;
if (pskb_expand_head(skb, 0, extra - skb_tailroom(skb), GFP_ATOMIC))
- return 0;
+ return false;
- return 1;
+ return true;
}
/* Generic function for mangling variable-length address changes inside
@@ -89,26 +89,26 @@
* skb enlargement, ...
*
* */
-int __nf_nat_mangle_tcp_packet(struct sk_buff *skb,
- struct nf_conn *ct,
- enum ip_conntrack_info ctinfo,
- unsigned int protoff,
- unsigned int match_offset,
- unsigned int match_len,
- const char *rep_buffer,
- unsigned int rep_len, bool adjust)
+bool __nf_nat_mangle_tcp_packet(struct sk_buff *skb,
+ struct nf_conn *ct,
+ enum ip_conntrack_info ctinfo,
+ unsigned int protoff,
+ unsigned int match_offset,
+ unsigned int match_len,
+ const char *rep_buffer,
+ unsigned int rep_len, bool adjust)
{
const struct nf_nat_l3proto *l3proto;
struct tcphdr *tcph;
int oldlen, datalen;
if (!skb_make_writable(skb, skb->len))
- return 0;
+ return false;
if (rep_len > match_len &&
rep_len - match_len > skb_tailroom(skb) &&
!enlarge_skb(skb, rep_len - match_len))
- return 0;
+ return false;
SKB_LINEAR_ASSERT(skb);
@@ -128,7 +128,7 @@
nf_ct_seqadj_set(ct, ctinfo, tcph->seq,
(int)rep_len - (int)match_len);
- return 1;
+ return true;
}
EXPORT_SYMBOL(__nf_nat_mangle_tcp_packet);
@@ -142,7 +142,7 @@
* XXX - This function could be merged with nf_nat_mangle_tcp_packet which
* should be fairly easy to do.
*/
-int
+bool
nf_nat_mangle_udp_packet(struct sk_buff *skb,
struct nf_conn *ct,
enum ip_conntrack_info ctinfo,
@@ -157,12 +157,12 @@
int datalen, oldlen;
if (!skb_make_writable(skb, skb->len))
- return 0;
+ return false;
if (rep_len > match_len &&
rep_len - match_len > skb_tailroom(skb) &&
!enlarge_skb(skb, rep_len - match_len))
- return 0;
+ return false;
udph = (void *)skb->data + protoff;
@@ -176,13 +176,13 @@
/* fix udp checksum if udp checksum was previously calculated */
if (!udph->check && skb->ip_summed != CHECKSUM_PARTIAL)
- return 1;
+ return true;
l3proto = __nf_nat_l3proto_find(nf_ct_l3num(ct));
l3proto->csum_recalc(skb, IPPROTO_UDP, udph, &udph->check,
datalen, oldlen);
- return 1;
+ return true;
}
EXPORT_SYMBOL(nf_nat_mangle_udp_packet);
diff --git a/net/netfilter/nf_nat_irc.c b/net/netfilter/nf_nat_irc.c
index 1fb2258..0648cb0 100644
--- a/net/netfilter/nf_nat_irc.c
+++ b/net/netfilter/nf_nat_irc.c
@@ -37,7 +37,6 @@
struct nf_conn *ct = exp->master;
union nf_inet_addr newaddr;
u_int16_t port;
- unsigned int ret;
/* Reply comes from server. */
newaddr = ct->tuplehash[IP_CT_DIR_REPLY].tuple.dst.u3;
@@ -83,14 +82,14 @@
pr_debug("nf_nat_irc: inserting '%s' == %pI4, port %u\n",
buffer, &newaddr.ip, port);
- ret = nf_nat_mangle_tcp_packet(skb, ct, ctinfo, protoff, matchoff,
- matchlen, buffer, strlen(buffer));
- if (ret != NF_ACCEPT) {
+ if (!nf_nat_mangle_tcp_packet(skb, ct, ctinfo, protoff, matchoff,
+ matchlen, buffer, strlen(buffer))) {
nf_ct_helper_log(skb, ct, "cannot mangle packet");
nf_ct_unexpect_related(exp);
+ return NF_DROP;
}
- return ret;
+ return NF_ACCEPT;
}
static void __exit nf_nat_irc_fini(void)
diff --git a/net/netfilter/nf_queue.c b/net/netfilter/nf_queue.c
index 4a76624..043850c 100644
--- a/net/netfilter/nf_queue.c
+++ b/net/netfilter/nf_queue.c
@@ -96,15 +96,18 @@
}
EXPORT_SYMBOL_GPL(nf_queue_entry_get_refs);
-void nf_queue_nf_hook_drop(struct net *net, const struct nf_hook_entry *entry)
+unsigned int nf_queue_nf_hook_drop(struct net *net)
{
const struct nf_queue_handler *qh;
+ unsigned int count = 0;
rcu_read_lock();
qh = rcu_dereference(net->nf.queue_handler);
if (qh)
- qh->nf_hook_drop(net, entry);
+ count = qh->nf_hook_drop(net);
rcu_read_unlock();
+
+ return count;
}
static int __nf_queue(struct sk_buff *skb, const struct nf_hook_state *state,
diff --git a/net/netfilter/nf_synproxy_core.c b/net/netfilter/nf_synproxy_core.c
index 7c6d1fb..a504e87 100644
--- a/net/netfilter/nf_synproxy_core.c
+++ b/net/netfilter/nf_synproxy_core.c
@@ -66,8 +66,8 @@
case TCPOPT_WINDOW:
if (opsize == TCPOLEN_WINDOW) {
opts->wscale = *ptr;
- if (opts->wscale > 14)
- opts->wscale = 14;
+ if (opts->wscale > TCP_MAX_WSCALE)
+ opts->wscale = TCP_MAX_WSCALE;
opts->options |= XT_SYNPROXY_OPT_WSCALE;
}
break;
@@ -287,9 +287,9 @@
struct synproxy_stats *stats = v;
if (v == SEQ_START_TOKEN) {
- seq_printf(seq, "entries\t\tsyn_received\t"
- "cookie_invalid\tcookie_valid\t"
- "cookie_retrans\tconn_reopened\n");
+ seq_puts(seq, "entries\t\tsyn_received\t"
+ "cookie_invalid\tcookie_valid\t"
+ "cookie_retrans\tconn_reopened\n");
return 0;
}
diff --git a/net/netfilter/nf_tables_api.c b/net/netfilter/nf_tables_api.c
index 11a96e8..5592250 100644
--- a/net/netfilter/nf_tables_api.c
+++ b/net/netfilter/nf_tables_api.c
@@ -144,7 +144,7 @@
unsigned int hook_nops)
{
if (table->flags & NFT_TABLE_F_DORMANT ||
- !(chain->flags & NFT_BASE_CHAIN))
+ !nft_is_base_chain(chain))
return 0;
return nf_register_net_hooks(net, nft_base_chain(chain)->ops,
@@ -157,7 +157,7 @@
unsigned int hook_nops)
{
if (table->flags & NFT_TABLE_F_DORMANT ||
- !(chain->flags & NFT_BASE_CHAIN))
+ !nft_is_base_chain(chain))
return;
nf_unregister_net_hooks(net, nft_base_chain(chain)->ops, hook_nops);
@@ -438,7 +438,7 @@
struct nlmsghdr *nlh;
struct nfgenmsg *nfmsg;
- event |= NFNL_SUBSYS_NFTABLES << 8;
+ event = nfnl_msg_type(NFNL_SUBSYS_NFTABLES, event);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(struct nfgenmsg), flags);
if (nlh == NULL)
goto nla_put_failure;
@@ -587,7 +587,7 @@
list_for_each_entry(chain, &table->chains, list) {
if (!nft_is_active_next(net, chain))
continue;
- if (!(chain->flags & NFT_BASE_CHAIN))
+ if (!nft_is_base_chain(chain))
continue;
if (cnt && i++ == cnt)
@@ -608,7 +608,7 @@
list_for_each_entry(chain, &table->chains, list) {
if (!nft_is_active_next(net, chain))
continue;
- if (!(chain->flags & NFT_BASE_CHAIN))
+ if (!nft_is_base_chain(chain))
continue;
err = nf_register_net_hooks(net, nft_base_chain(chain)->ops,
@@ -989,7 +989,7 @@
struct nlmsghdr *nlh;
struct nfgenmsg *nfmsg;
- event |= NFNL_SUBSYS_NFTABLES << 8;
+ event = nfnl_msg_type(NFNL_SUBSYS_NFTABLES, event);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(struct nfgenmsg), flags);
if (nlh == NULL)
goto nla_put_failure;
@@ -1007,7 +1007,7 @@
if (nla_put_string(skb, NFTA_CHAIN_NAME, chain->name))
goto nla_put_failure;
- if (chain->flags & NFT_BASE_CHAIN) {
+ if (nft_is_base_chain(chain)) {
const struct nft_base_chain *basechain = nft_base_chain(chain);
const struct nf_hook_ops *ops = &basechain->ops[0];
struct nlattr *nest;
@@ -1182,7 +1182,8 @@
struct nft_stats *stats;
int err;
- err = nla_parse_nested(tb, NFTA_COUNTER_MAX, attr, nft_counter_policy);
+ err = nla_parse_nested(tb, NFTA_COUNTER_MAX, attr, nft_counter_policy,
+ NULL);
if (err < 0)
return ERR_PTR(err);
@@ -1226,7 +1227,7 @@
{
BUG_ON(chain->use > 0);
- if (chain->flags & NFT_BASE_CHAIN) {
+ if (nft_is_base_chain(chain)) {
struct nft_base_chain *basechain = nft_base_chain(chain);
module_put(basechain->type->owner);
@@ -1257,7 +1258,7 @@
int err;
err = nla_parse_nested(ha, NFTA_HOOK_MAX, nla[NFTA_CHAIN_HOOK],
- nft_hook_policy);
+ nft_hook_policy, NULL);
if (err < 0)
return err;
@@ -1364,8 +1365,8 @@
}
if (nla[NFTA_CHAIN_POLICY]) {
- if ((chain != NULL &&
- !(chain->flags & NFT_BASE_CHAIN)))
+ if (chain != NULL &&
+ !nft_is_base_chain(chain))
return -EOPNOTSUPP;
if (chain == NULL &&
@@ -1396,7 +1397,7 @@
struct nft_chain_hook hook;
struct nf_hook_ops *ops;
- if (!(chain->flags & NFT_BASE_CHAIN))
+ if (!nft_is_base_chain(chain))
return -EBUSY;
err = nft_chain_parse_hook(net, nla, afi, &hook,
@@ -1433,7 +1434,7 @@
}
if (nla[NFTA_CHAIN_COUNTERS]) {
- if (!(chain->flags & NFT_BASE_CHAIN))
+ if (!nft_is_base_chain(chain))
return -EOPNOTSUPP;
stats = nft_stats_alloc(nla[NFTA_CHAIN_COUNTERS]);
@@ -1724,7 +1725,7 @@
struct nlattr *tb[NFTA_EXPR_MAX + 1];
int err;
- err = nla_parse_nested(tb, NFTA_EXPR_MAX, nla, nft_expr_policy);
+ err = nla_parse_nested(tb, NFTA_EXPR_MAX, nla, nft_expr_policy, NULL);
if (err < 0)
return err;
@@ -1734,7 +1735,7 @@
if (tb[NFTA_EXPR_DATA]) {
err = nla_parse_nested(info->tb, type->maxattr,
- tb[NFTA_EXPR_DATA], type->policy);
+ tb[NFTA_EXPR_DATA], type->policy, NULL);
if (err < 0)
goto err1;
} else
@@ -1772,8 +1773,19 @@
goto err1;
}
+ if (ops->validate) {
+ const struct nft_data *data = NULL;
+
+ err = ops->validate(ctx, expr, &data);
+ if (err < 0)
+ goto err2;
+ }
+
return 0;
+err2:
+ if (ops->destroy)
+ ops->destroy(ctx, expr);
err1:
expr->ops = NULL;
return err;
@@ -1874,10 +1886,9 @@
const struct nft_expr *expr, *next;
struct nlattr *list;
const struct nft_rule *prule;
- int type = event | NFNL_SUBSYS_NFTABLES << 8;
+ u16 type = nfnl_msg_type(NFNL_SUBSYS_NFTABLES, event);
- nlh = nlmsg_put(skb, portid, seq, type, sizeof(struct nfgenmsg),
- flags);
+ nlh = nlmsg_put(skb, portid, seq, type, sizeof(struct nfgenmsg), flags);
if (nlh == NULL)
goto nla_put_failure;
@@ -1895,7 +1906,7 @@
goto nla_put_failure;
if ((event != NFT_MSG_DELRULE) && (rule->list.prev != &chain->rules)) {
- prule = list_entry(rule->list.prev, struct nft_rule, list);
+ prule = list_prev_entry(rule, list);
if (nla_put_be64(skb, NFTA_RULE_POSITION,
cpu_to_be64(prule->handle),
NFTA_RULE_PAD))
@@ -2523,8 +2534,8 @@
return 0;
}
-struct nft_set *nf_tables_set_lookup(const struct nft_table *table,
- const struct nlattr *nla, u8 genmask)
+static struct nft_set *nf_tables_set_lookup(const struct nft_table *table,
+ const struct nlattr *nla, u8 genmask)
{
struct nft_set *set;
@@ -2538,11 +2549,10 @@
}
return ERR_PTR(-ENOENT);
}
-EXPORT_SYMBOL_GPL(nf_tables_set_lookup);
-struct nft_set *nf_tables_set_lookup_byid(const struct net *net,
- const struct nlattr *nla,
- u8 genmask)
+static struct nft_set *nf_tables_set_lookup_byid(const struct net *net,
+ const struct nlattr *nla,
+ u8 genmask)
{
struct nft_trans *trans;
u32 id = ntohl(nla_get_be32(nla));
@@ -2557,7 +2567,25 @@
}
return ERR_PTR(-ENOENT);
}
-EXPORT_SYMBOL_GPL(nf_tables_set_lookup_byid);
+
+struct nft_set *nft_set_lookup(const struct net *net,
+ const struct nft_table *table,
+ const struct nlattr *nla_set_name,
+ const struct nlattr *nla_set_id,
+ u8 genmask)
+{
+ struct nft_set *set;
+
+ set = nf_tables_set_lookup(table, nla_set_name, genmask);
+ if (IS_ERR(set)) {
+ if (!nla_set_id)
+ return set;
+
+ set = nf_tables_set_lookup_byid(net, nla_set_id, genmask);
+ }
+ return set;
+}
+EXPORT_SYMBOL_GPL(nft_set_lookup);
static int nf_tables_set_alloc_name(struct nft_ctx *ctx, struct nft_set *set,
const char *name)
@@ -2617,7 +2645,7 @@
u32 portid = ctx->portid;
u32 seq = ctx->seq;
- event |= NFNL_SUBSYS_NFTABLES << 8;
+ event = nfnl_msg_type(NFNL_SUBSYS_NFTABLES, event);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(struct nfgenmsg),
flags);
if (nlh == NULL)
@@ -2851,7 +2879,8 @@
struct nlattr *da[NFTA_SET_DESC_MAX + 1];
int err;
- err = nla_parse_nested(da, NFTA_SET_DESC_MAX, nla, nft_set_desc_policy);
+ err = nla_parse_nested(da, NFTA_SET_DESC_MAX, nla,
+ nft_set_desc_policy, NULL);
if (err < 0)
return err;
@@ -3353,7 +3382,8 @@
int event, err;
err = nlmsg_parse(cb->nlh, sizeof(struct nfgenmsg), nla,
- NFTA_SET_ELEM_LIST_MAX, nft_set_elem_list_policy);
+ NFTA_SET_ELEM_LIST_MAX, nft_set_elem_list_policy,
+ NULL);
if (err < 0)
return err;
@@ -3367,8 +3397,7 @@
if (IS_ERR(set))
return PTR_ERR(set);
- event = NFT_MSG_NEWSETELEM;
- event |= NFNL_SUBSYS_NFTABLES << 8;
+ event = nfnl_msg_type(NFNL_SUBSYS_NFTABLES, NFT_MSG_NEWSETELEM);
portid = NETLINK_CB(cb->skb).portid;
seq = cb->nlh->nlmsg_seq;
@@ -3453,7 +3482,7 @@
struct nlattr *nest;
int err;
- event |= NFNL_SUBSYS_NFTABLES << 8;
+ event = nfnl_msg_type(NFNL_SUBSYS_NFTABLES, event);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(struct nfgenmsg),
flags);
if (nlh == NULL)
@@ -3612,7 +3641,7 @@
int err;
err = nla_parse_nested(nla, NFTA_SET_ELEM_MAX, attr,
- nft_set_elem_policy);
+ nft_set_elem_policy, NULL);
if (err < 0)
return err;
@@ -3847,7 +3876,7 @@
int err;
err = nla_parse_nested(nla, NFTA_SET_ELEM_MAX, attr,
- nft_set_elem_policy);
+ nft_set_elem_policy, NULL);
if (err < 0)
goto err1;
@@ -4069,7 +4098,8 @@
[NFTA_OBJ_DATA] = { .type = NLA_NESTED },
};
-static struct nft_object *nft_obj_init(const struct nft_object_type *type,
+static struct nft_object *nft_obj_init(const struct nft_ctx *ctx,
+ const struct nft_object_type *type,
const struct nlattr *attr)
{
struct nlattr *tb[type->maxattr + 1];
@@ -4077,7 +4107,8 @@
int err;
if (attr) {
- err = nla_parse_nested(tb, type->maxattr, attr, type->policy);
+ err = nla_parse_nested(tb, type->maxattr, attr, type->policy,
+ NULL);
if (err < 0)
goto err1;
} else {
@@ -4089,7 +4120,7 @@
if (obj == NULL)
goto err1;
- err = type->init((const struct nlattr * const *)tb, obj);
+ err = type->init(ctx, (const struct nlattr * const *)tb, obj);
if (err < 0)
goto err2;
@@ -4197,7 +4228,7 @@
if (IS_ERR(type))
return PTR_ERR(type);
- obj = nft_obj_init(type, nla[NFTA_OBJ_DATA]);
+ obj = nft_obj_init(&ctx, type, nla[NFTA_OBJ_DATA]);
if (IS_ERR(obj)) {
err = PTR_ERR(obj);
goto err1;
@@ -4229,7 +4260,7 @@
struct nfgenmsg *nfmsg;
struct nlmsghdr *nlh;
- event |= NFNL_SUBSYS_NFTABLES << 8;
+ event = nfnl_msg_type(NFNL_SUBSYS_NFTABLES, event);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(struct nfgenmsg), flags);
if (nlh == NULL)
goto nla_put_failure;
@@ -4411,8 +4442,6 @@
err:
kfree_skb(skb2);
return err;
-
- return 0;
}
static void nft_obj_destroy(struct nft_object *obj)
@@ -4502,7 +4531,7 @@
{
struct nlmsghdr *nlh;
struct nfgenmsg *nfmsg;
- int event = (NFNL_SUBSYS_NFTABLES << 8) | NFT_MSG_NEWGEN;
+ int event = nfnl_msg_type(NFNL_SUBSYS_NFTABLES, NFT_MSG_NEWGEN);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(struct nfgenmsg), 0);
if (nlh == NULL)
@@ -4684,7 +4713,7 @@
if (nft_trans_chain_name(trans)[0])
strcpy(trans->ctx.chain->name, nft_trans_chain_name(trans));
- if (!(trans->ctx.chain->flags & NFT_BASE_CHAIN))
+ if (!nft_is_base_chain(trans->ctx.chain))
return;
basechain = nft_base_chain(trans->ctx.chain);
@@ -4998,7 +5027,7 @@
{
const struct nft_base_chain *basechain;
- if (chain->flags & NFT_BASE_CHAIN) {
+ if (nft_is_base_chain(chain)) {
basechain = nft_base_chain(chain);
if (basechain->type->type != type)
return -EOPNOTSUPP;
@@ -5012,7 +5041,7 @@
{
struct nft_base_chain *basechain;
- if (chain->flags & NFT_BASE_CHAIN) {
+ if (nft_is_base_chain(chain)) {
basechain = nft_base_chain(chain);
if ((1 << basechain->ops[0].hooknum) & hook_flags)
@@ -5290,7 +5319,8 @@
struct nft_chain *chain;
int err;
- err = nla_parse_nested(tb, NFTA_VERDICT_MAX, nla, nft_verdict_policy);
+ err = nla_parse_nested(tb, NFTA_VERDICT_MAX, nla, nft_verdict_policy,
+ NULL);
if (err < 0)
return err;
@@ -5321,7 +5351,7 @@
tb[NFTA_VERDICT_CHAIN], genmask);
if (IS_ERR(chain))
return PTR_ERR(chain);
- if (chain->flags & NFT_BASE_CHAIN)
+ if (nft_is_base_chain(chain))
return -EOPNOTSUPP;
chain->use++;
@@ -5420,7 +5450,7 @@
struct nlattr *tb[NFTA_DATA_MAX + 1];
int err;
- err = nla_parse_nested(tb, NFTA_DATA_MAX, nla, nft_data_policy);
+ err = nla_parse_nested(tb, NFTA_DATA_MAX, nla, nft_data_policy, NULL);
if (err < 0)
return err;
@@ -5494,7 +5524,7 @@
{
struct nft_rule *rule, *nr;
- BUG_ON(!(ctx->chain->flags & NFT_BASE_CHAIN));
+ BUG_ON(!nft_is_base_chain(ctx->chain));
nf_tables_unregister_hooks(ctx->net, ctx->chain->table, ctx->chain,
ctx->afi->nops);
diff --git a/net/netfilter/nf_tables_netdev.c b/net/netfilter/nf_tables_netdev.c
index 9e2ae42..4034329 100644
--- a/net/netfilter/nf_tables_netdev.c
+++ b/net/netfilter/nf_tables_netdev.c
@@ -128,7 +128,7 @@
list_for_each_entry(table, &afi->tables, list) {
ctx.table = table;
list_for_each_entry_safe(chain, nr, &table->chains, list) {
- if (!(chain->flags & NFT_BASE_CHAIN))
+ if (!nft_is_base_chain(chain))
continue;
ctx.chain = chain;
diff --git a/net/netfilter/nf_tables_trace.c b/net/netfilter/nf_tables_trace.c
index 12eb904..e1b15e7 100644
--- a/net/netfilter/nf_tables_trace.c
+++ b/net/netfilter/nf_tables_trace.c
@@ -169,7 +169,7 @@
struct nlmsghdr *nlh;
struct sk_buff *skb;
unsigned int size;
- int event = (NFNL_SUBSYS_NFTABLES << 8) | NFT_MSG_TRACE;
+ u16 event;
if (!nfnetlink_has_listeners(nft_net(pkt), NFNLGRP_NFTRACE))
return;
@@ -198,6 +198,7 @@
if (!skb)
return;
+ event = nfnl_msg_type(NFNL_SUBSYS_NFTABLES, NFT_MSG_TRACE);
nlh = nlmsg_put(skb, 0, 0, event, sizeof(struct nfgenmsg), 0);
if (!nlh)
goto nla_put_failure;
diff --git a/net/netfilter/nfnetlink.c b/net/netfilter/nfnetlink.c
index 68eda920..80f5ecf 100644
--- a/net/netfilter/nfnetlink.c
+++ b/net/netfilter/nfnetlink.c
@@ -148,7 +148,8 @@
EXPORT_SYMBOL_GPL(nfnetlink_unicast);
/* Process one complete nfnetlink message. */
-static int nfnetlink_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int nfnetlink_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
const struct nfnl_callback *nc;
@@ -191,8 +192,8 @@
int attrlen = nlh->nlmsg_len - min_len;
__u8 subsys_id = NFNL_SUBSYS_ID(type);
- err = nla_parse(cda, ss->cb[cb_id].attr_count,
- attr, attrlen, ss->cb[cb_id].policy);
+ err = nla_parse(cda, ss->cb[cb_id].attr_count, attr, attrlen,
+ ss->cb[cb_id].policy, extack);
if (err < 0) {
rcu_read_unlock();
return err;
@@ -261,7 +262,7 @@
struct nfnl_err *nfnl_err, *next;
list_for_each_entry_safe(nfnl_err, next, err_list, head) {
- netlink_ack(skb, nfnl_err->nlh, nfnl_err->err);
+ netlink_ack(skb, nfnl_err->nlh, nfnl_err->err, NULL);
nfnl_err_del(nfnl_err);
}
}
@@ -284,13 +285,13 @@
int err;
if (subsys_id >= NFNL_SUBSYS_COUNT)
- return netlink_ack(skb, nlh, -EINVAL);
+ return netlink_ack(skb, nlh, -EINVAL, NULL);
replay:
status = 0;
skb = netlink_skb_clone(oskb, GFP_KERNEL);
if (!skb)
- return netlink_ack(oskb, nlh, -ENOMEM);
+ return netlink_ack(oskb, nlh, -ENOMEM, NULL);
nfnl_lock(subsys_id);
ss = nfnl_dereference_protected(subsys_id);
@@ -304,20 +305,20 @@
#endif
{
nfnl_unlock(subsys_id);
- netlink_ack(oskb, nlh, -EOPNOTSUPP);
+ netlink_ack(oskb, nlh, -EOPNOTSUPP, NULL);
return kfree_skb(skb);
}
}
if (!ss->commit || !ss->abort) {
nfnl_unlock(subsys_id);
- netlink_ack(oskb, nlh, -EOPNOTSUPP);
+ netlink_ack(oskb, nlh, -EOPNOTSUPP, NULL);
return kfree_skb(skb);
}
if (genid && ss->valid_genid && !ss->valid_genid(net, genid)) {
nfnl_unlock(subsys_id);
- netlink_ack(oskb, nlh, -ERESTART);
+ netlink_ack(oskb, nlh, -ERESTART, NULL);
return kfree_skb(skb);
}
@@ -376,8 +377,8 @@
struct nlattr *attr = (void *)nlh + min_len;
int attrlen = nlh->nlmsg_len - min_len;
- err = nla_parse(cda, ss->cb[cb_id].attr_count,
- attr, attrlen, ss->cb[cb_id].policy);
+ err = nla_parse(cda, ss->cb[cb_id].attr_count, attr,
+ attrlen, ss->cb[cb_id].policy, NULL);
if (err < 0)
goto ack;
@@ -407,7 +408,8 @@
* pointing to the batch header.
*/
nfnl_err_reset(&err_list);
- netlink_ack(oskb, nlmsg_hdr(oskb), -ENOMEM);
+ netlink_ack(oskb, nlmsg_hdr(oskb), -ENOMEM,
+ NULL);
status |= NFNL_BATCH_FAILURE;
goto done;
}
@@ -465,9 +467,10 @@
skb->len < NLMSG_HDRLEN + sizeof(struct nfgenmsg))
return;
- err = nla_parse(cda, NFNL_BATCH_MAX, attr, attrlen, nfnl_batch_policy);
+ err = nla_parse(cda, NFNL_BATCH_MAX, attr, attrlen, nfnl_batch_policy,
+ NULL);
if (err < 0) {
- netlink_ack(skb, nlh, err);
+ netlink_ack(skb, nlh, err, NULL);
return;
}
if (cda[NFNL_BATCH_GENID])
@@ -493,14 +496,14 @@
return;
if (!netlink_net_capable(skb, CAP_NET_ADMIN)) {
- netlink_ack(skb, nlh, -EPERM);
+ netlink_ack(skb, nlh, -EPERM, NULL);
return;
}
if (nlh->nlmsg_type == NFNL_MSG_BATCH_BEGIN)
nfnetlink_rcv_skb_batch(skb, nlh);
else
- netlink_rcv_skb(skb, &nfnetlink_rcv_msg);
+ netlink_rcv_skb(skb, nfnetlink_rcv_msg);
}
#ifdef CONFIG_MODULES
diff --git a/net/netfilter/nfnetlink_acct.c b/net/netfilter/nfnetlink_acct.c
index d44d89b..9898fb4 100644
--- a/net/netfilter/nfnetlink_acct.c
+++ b/net/netfilter/nfnetlink_acct.c
@@ -11,6 +11,7 @@
#include <linux/kernel.h>
#include <linux/skbuff.h>
#include <linux/atomic.h>
+#include <linux/refcount.h>
#include <linux/netlink.h>
#include <linux/rculist.h>
#include <linux/slab.h>
@@ -32,7 +33,7 @@
atomic64_t bytes;
unsigned long flags;
struct list_head head;
- atomic_t refcnt;
+ refcount_t refcnt;
char name[NFACCT_NAME_MAX];
struct rcu_head rcu_head;
char data[0];
@@ -123,7 +124,7 @@
atomic64_set(&nfacct->pkts,
be64_to_cpu(nla_get_be64(tb[NFACCT_PKTS])));
}
- atomic_set(&nfacct->refcnt, 1);
+ refcount_set(&nfacct->refcnt, 1);
list_add_tail_rcu(&nfacct->head, &net->nfnl_acct_list);
return 0;
}
@@ -138,7 +139,7 @@
u64 pkts, bytes;
u32 old_flags;
- event |= NFNL_SUBSYS_ACCT << 8;
+ event = nfnl_msg_type(NFNL_SUBSYS_ACCT, event);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(*nfmsg), flags);
if (nlh == NULL)
goto nlmsg_failure;
@@ -166,7 +167,7 @@
NFACCT_PAD) ||
nla_put_be64(skb, NFACCT_BYTES, cpu_to_be64(bytes),
NFACCT_PAD) ||
- nla_put_be32(skb, NFACCT_USE, htonl(atomic_read(&acct->refcnt))))
+ nla_put_be32(skb, NFACCT_USE, htonl(refcount_read(&acct->refcnt))))
goto nla_put_failure;
if (acct->flags & NFACCT_F_QUOTA) {
u64 *quota = (u64 *)acct->data;
@@ -243,7 +244,8 @@
struct nlattr *tb[NFACCT_FILTER_MAX + 1];
int err;
- err = nla_parse_nested(tb, NFACCT_FILTER_MAX, attr, filter_policy);
+ err = nla_parse_nested(tb, NFACCT_FILTER_MAX, attr, filter_policy,
+ NULL);
if (err < 0)
return ERR_PTR(err);
@@ -329,7 +331,7 @@
/* We want to avoid races with nfnl_acct_put. So only when the current
* refcnt is 1, we decrease it to 0.
*/
- if (atomic_cmpxchg(&cur->refcnt, 1, 0) == 1) {
+ if (refcount_dec_if_one(&cur->refcnt)) {
/* We are protected by nfnl mutex. */
list_del_rcu(&cur->head);
kfree_rcu(cur, rcu_head);
@@ -413,7 +415,7 @@
if (!try_module_get(THIS_MODULE))
goto err;
- if (!atomic_inc_not_zero(&cur->refcnt)) {
+ if (!refcount_inc_not_zero(&cur->refcnt)) {
module_put(THIS_MODULE);
goto err;
}
@@ -429,7 +431,7 @@
void nfnl_acct_put(struct nf_acct *acct)
{
- if (atomic_dec_and_test(&acct->refcnt))
+ if (refcount_dec_and_test(&acct->refcnt))
kfree_rcu(acct, rcu_head);
module_put(THIS_MODULE);
@@ -502,7 +504,7 @@
list_for_each_entry_safe(cur, tmp, &net->nfnl_acct_list, head) {
list_del_rcu(&cur->head);
- if (atomic_dec_and_test(&cur->refcnt))
+ if (refcount_dec_and_test(&cur->refcnt))
kfree_rcu(cur, rcu_head);
}
}
diff --git a/net/netfilter/nfnetlink_cthelper.c b/net/netfilter/nfnetlink_cthelper.c
index d455581..950bf6e 100644
--- a/net/netfilter/nfnetlink_cthelper.c
+++ b/net/netfilter/nfnetlink_cthelper.c
@@ -77,7 +77,8 @@
int err;
struct nlattr *tb[NFCTH_TUPLE_MAX+1];
- err = nla_parse_nested(tb, NFCTH_TUPLE_MAX, attr, nfnl_cthelper_tuple_pol);
+ err = nla_parse_nested(tb, NFCTH_TUPLE_MAX, attr,
+ nfnl_cthelper_tuple_pol, NULL);
if (err < 0)
return err;
@@ -104,7 +105,7 @@
if (help->helper->data_len == 0)
return -EINVAL;
- memcpy(help->data, nla_data(attr), help->helper->data_len);
+ nla_memcpy(help->data, nla_data(attr), sizeof(help->data));
return 0;
}
@@ -137,7 +138,8 @@
int err;
struct nlattr *tb[NFCTH_POLICY_MAX+1];
- err = nla_parse_nested(tb, NFCTH_POLICY_MAX, attr, nfnl_cthelper_expect_pol);
+ err = nla_parse_nested(tb, NFCTH_POLICY_MAX, attr,
+ nfnl_cthelper_expect_pol, NULL);
if (err < 0)
return err;
@@ -150,6 +152,9 @@
nla_data(tb[NFCTH_POLICY_NAME]), NF_CT_HELPER_NAME_LEN);
expect_policy->max_expected =
ntohl(nla_get_be32(tb[NFCTH_POLICY_EXPECT_MAX]));
+ if (expect_policy->max_expected > NF_CT_EXPECT_MAX_CNT)
+ return -EINVAL;
+
expect_policy->timeout =
ntohl(nla_get_be32(tb[NFCTH_POLICY_EXPECT_TIMEOUT]));
@@ -171,7 +176,7 @@
unsigned int class_max;
ret = nla_parse_nested(tb, NFCTH_POLICY_SET_MAX, attr,
- nfnl_cthelper_expect_policy_set);
+ nfnl_cthelper_expect_policy_set, NULL);
if (ret < 0)
return ret;
@@ -213,6 +218,7 @@
{
struct nf_conntrack_helper *helper;
struct nfnl_cthelper *nfcth;
+ unsigned int size;
int ret;
if (!tb[NFCTH_TUPLE] || !tb[NFCTH_POLICY] || !tb[NFCTH_PRIV_DATA_LEN])
@@ -228,7 +234,12 @@
goto err1;
strncpy(helper->name, nla_data(tb[NFCTH_NAME]), NF_CT_HELPER_NAME_LEN);
- helper->data_len = ntohl(nla_get_be32(tb[NFCTH_PRIV_DATA_LEN]));
+ size = ntohl(nla_get_be32(tb[NFCTH_PRIV_DATA_LEN]));
+ if (size > FIELD_SIZEOF(struct nf_conn_help, data)) {
+ ret = -ENOMEM;
+ goto err2;
+ }
+
helper->flags |= NF_CT_HELPER_F_USERSPACE;
memcpy(&helper->tuple, tuple, sizeof(struct nf_conntrack_tuple));
@@ -276,7 +287,7 @@
int err;
err = nla_parse_nested(tb, NFCTH_POLICY_MAX, attr,
- nfnl_cthelper_expect_pol);
+ nfnl_cthelper_expect_pol, NULL);
if (err < 0)
return err;
@@ -290,6 +301,9 @@
new_policy->max_expected =
ntohl(nla_get_be32(tb[NFCTH_POLICY_EXPECT_MAX]));
+ if (new_policy->max_expected > NF_CT_EXPECT_MAX_CNT)
+ return -EINVAL;
+
new_policy->timeout =
ntohl(nla_get_be32(tb[NFCTH_POLICY_EXPECT_TIMEOUT]));
@@ -336,7 +350,7 @@
int err;
err = nla_parse_nested(tb, NFCTH_POLICY_SET_MAX, attr,
- nfnl_cthelper_expect_policy_set);
+ nfnl_cthelper_expect_policy_set, NULL);
if (err < 0)
return err;
@@ -501,7 +515,7 @@
unsigned int flags = portid ? NLM_F_MULTI : 0;
int status;
- event |= NFNL_SUBSYS_CTHELPER << 8;
+ event = nfnl_msg_type(NFNL_SUBSYS_CTHELPER, event);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(*nfmsg), flags);
if (nlh == NULL)
goto nlmsg_failure;
diff --git a/net/netfilter/nfnetlink_cttimeout.c b/net/netfilter/nfnetlink_cttimeout.c
index 47d6656..a3e7bb5 100644
--- a/net/netfilter/nfnetlink_cttimeout.c
+++ b/net/netfilter/nfnetlink_cttimeout.c
@@ -56,7 +56,8 @@
struct nlattr *tb[l4proto->ctnl_timeout.nlattr_max+1];
ret = nla_parse_nested(tb, l4proto->ctnl_timeout.nlattr_max,
- attr, l4proto->ctnl_timeout.nla_policy);
+ attr, l4proto->ctnl_timeout.nla_policy,
+ NULL);
if (ret < 0)
return ret;
@@ -138,7 +139,7 @@
strcpy(timeout->name, nla_data(cda[CTA_TIMEOUT_NAME]));
timeout->l3num = l3num;
timeout->l4proto = l4proto;
- atomic_set(&timeout->refcnt, 1);
+ refcount_set(&timeout->refcnt, 1);
list_add_tail_rcu(&timeout->head, &net->nfct_timeout_list);
return 0;
@@ -158,7 +159,7 @@
unsigned int flags = portid ? NLM_F_MULTI : 0;
struct nf_conntrack_l4proto *l4proto = timeout->l4proto;
- event |= NFNL_SUBSYS_CTNETLINK_TIMEOUT << 8;
+ event = nfnl_msg_type(NFNL_SUBSYS_CTNETLINK_TIMEOUT, event);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(*nfmsg), flags);
if (nlh == NULL)
goto nlmsg_failure;
@@ -172,7 +173,7 @@
nla_put_be16(skb, CTA_TIMEOUT_L3PROTO, htons(timeout->l3num)) ||
nla_put_u8(skb, CTA_TIMEOUT_L4PROTO, timeout->l4proto->l4proto) ||
nla_put_be32(skb, CTA_TIMEOUT_USE,
- htonl(atomic_read(&timeout->refcnt))))
+ htonl(refcount_read(&timeout->refcnt))))
goto nla_put_failure;
if (likely(l4proto->ctnl_timeout.obj_to_nlattr)) {
@@ -339,7 +340,7 @@
/* We want to avoid races with ctnl_timeout_put. So only when the
* current refcnt is 1, we decrease it to 0.
*/
- if (atomic_cmpxchg(&timeout->refcnt, 1, 0) == 1) {
+ if (refcount_dec_if_one(&timeout->refcnt)) {
/* We are protected by nfnl mutex. */
list_del_rcu(&timeout->head);
nf_ct_l4proto_put(timeout->l4proto);
@@ -431,7 +432,7 @@
struct nfgenmsg *nfmsg;
unsigned int flags = portid ? NLM_F_MULTI : 0;
- event |= NFNL_SUBSYS_CTNETLINK_TIMEOUT << 8;
+ event = nfnl_msg_type(NFNL_SUBSYS_CTNETLINK_TIMEOUT, event);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(*nfmsg), flags);
if (nlh == NULL)
goto nlmsg_failure;
@@ -536,7 +537,7 @@
if (!try_module_get(THIS_MODULE))
goto err;
- if (!atomic_inc_not_zero(&timeout->refcnt)) {
+ if (!refcount_inc_not_zero(&timeout->refcnt)) {
module_put(THIS_MODULE);
goto err;
}
@@ -550,7 +551,7 @@
static void ctnl_timeout_put(struct ctnl_timeout *timeout)
{
- if (atomic_dec_and_test(&timeout->refcnt))
+ if (refcount_dec_and_test(&timeout->refcnt))
kfree_rcu(timeout, rcu_head);
module_put(THIS_MODULE);
@@ -601,7 +602,7 @@
list_del_rcu(&cur->head);
nf_ct_l4proto_put(cur->l4proto);
- if (atomic_dec_and_test(&cur->refcnt))
+ if (refcount_dec_and_test(&cur->refcnt))
kfree_rcu(cur, rcu_head);
}
}
diff --git a/net/netfilter/nfnetlink_log.c b/net/netfilter/nfnetlink_log.c
index 08247bf..da97049 100644
--- a/net/netfilter/nfnetlink_log.c
+++ b/net/netfilter/nfnetlink_log.c
@@ -40,6 +40,8 @@
#include <net/netfilter/nfnetlink_log.h>
#include <linux/atomic.h>
+#include <linux/refcount.h>
+
#if IS_ENABLED(CONFIG_BRIDGE_NETFILTER)
#include "../bridge/br_private.h"
@@ -57,7 +59,7 @@
struct nfulnl_instance {
struct hlist_node hlist; /* global list of instances */
spinlock_t lock;
- atomic_t use; /* use count */
+ refcount_t use; /* use count */
unsigned int qlen; /* number of nlmsgs in skb */
struct sk_buff *skb; /* pre-allocatd skb */
@@ -115,7 +117,7 @@
static inline void
instance_get(struct nfulnl_instance *inst)
{
- atomic_inc(&inst->use);
+ refcount_inc(&inst->use);
}
static struct nfulnl_instance *
@@ -125,7 +127,7 @@
rcu_read_lock_bh();
inst = __instance_lookup(log, group_num);
- if (inst && !atomic_inc_not_zero(&inst->use))
+ if (inst && !refcount_inc_not_zero(&inst->use))
inst = NULL;
rcu_read_unlock_bh();
@@ -145,7 +147,7 @@
static void
instance_put(struct nfulnl_instance *inst)
{
- if (inst && atomic_dec_and_test(&inst->use))
+ if (inst && refcount_dec_and_test(&inst->use))
call_rcu_bh(&inst->rcu, nfulnl_instance_free_rcu);
}
@@ -180,7 +182,7 @@
INIT_HLIST_NODE(&inst->hlist);
spin_lock_init(&inst->lock);
/* needs to be two, since we _put() after creation */
- atomic_set(&inst->use, 2);
+ refcount_set(&inst->use, 2);
setup_timer(&inst->timer, nfulnl_timer, (unsigned long)inst);
@@ -409,7 +411,7 @@
const unsigned char *hwhdrp;
nlh = nlmsg_put(inst->skb, 0, 0,
- NFNL_SUBSYS_ULOG << 8 | NFULNL_MSG_PACKET,
+ nfnl_msg_type(NFNL_SUBSYS_ULOG, NFULNL_MSG_PACKET),
sizeof(struct nfgenmsg), 0);
if (!nlh)
return -1;
@@ -801,7 +803,7 @@
static struct nf_logger nfulnl_logger __read_mostly = {
.name = "nfnetlink_log",
.type = NF_LOG_TYPE_ULOG,
- .logfn = &nfulnl_log_packet,
+ .logfn = nfulnl_log_packet,
.me = THIS_MODULE,
};
@@ -1031,7 +1033,7 @@
inst->group_num,
inst->peer_portid, inst->qlen,
inst->copy_mode, inst->copy_range,
- inst->flushtimeout, atomic_read(&inst->use));
+ inst->flushtimeout, refcount_read(&inst->use));
return 0;
}
@@ -1138,10 +1140,10 @@
static void __exit nfnetlink_log_fini(void)
{
- nf_log_unregister(&nfulnl_logger);
nfnetlink_subsys_unregister(&nfulnl_subsys);
netlink_unregister_notifier(&nfulnl_rtnl_notifier);
unregister_pernet_subsys(&nfnl_log_net_ops);
+ nf_log_unregister(&nfulnl_logger);
}
MODULE_DESCRIPTION("netfilter userspace logging");
diff --git a/net/netfilter/nfnetlink_queue.c b/net/netfilter/nfnetlink_queue.c
index 933509e..8a0f218 100644
--- a/net/netfilter/nfnetlink_queue.c
+++ b/net/netfilter/nfnetlink_queue.c
@@ -447,7 +447,7 @@
}
nlh = nlmsg_put(skb, 0, 0,
- NFNL_SUBSYS_QUEUE << 8 | NFQNL_MSG_PACKET,
+ nfnl_msg_type(NFNL_SUBSYS_QUEUE, NFQNL_MSG_PACKET),
sizeof(struct nfgenmsg), 0);
if (!nlh) {
skb_tx_error(entskb);
@@ -922,16 +922,10 @@
.notifier_call = nfqnl_rcv_dev_event,
};
-static int nf_hook_cmp(struct nf_queue_entry *entry, unsigned long entry_ptr)
-{
- return rcu_access_pointer(entry->hook) ==
- (struct nf_hook_entry *)entry_ptr;
-}
-
-static void nfqnl_nf_hook_drop(struct net *net,
- const struct nf_hook_entry *hook)
+static unsigned int nfqnl_nf_hook_drop(struct net *net)
{
struct nfnl_queue_net *q = nfnl_queue_pernet(net);
+ unsigned int instances = 0;
int i;
rcu_read_lock();
@@ -939,10 +933,14 @@
struct nfqnl_instance *inst;
struct hlist_head *head = &q->instance_table[i];
- hlist_for_each_entry_rcu(inst, head, hlist)
- nfqnl_flush(inst, nf_hook_cmp, (unsigned long)hook);
+ hlist_for_each_entry_rcu(inst, head, hlist) {
+ nfqnl_flush(inst, NULL, 0);
+ instances++;
+ }
}
rcu_read_unlock();
+
+ return instances;
}
static int
@@ -1109,7 +1107,7 @@
int err;
err = nla_parse_nested(tb, NFQA_VLAN_MAX, nfqa[NFQA_VLAN],
- nfqa_vlan_policy);
+ nfqa_vlan_policy, NULL);
if (err < 0)
return err;
@@ -1213,8 +1211,8 @@
};
static const struct nf_queue_handler nfqh = {
- .outfn = &nfqnl_enqueue_packet,
- .nf_hook_drop = &nfqnl_nf_hook_drop,
+ .outfn = nfqnl_enqueue_packet,
+ .nf_hook_drop = nfqnl_nf_hook_drop,
};
static int nfqnl_recv_config(struct net *net, struct sock *ctnl,
diff --git a/net/netfilter/nft_compat.c b/net/netfilter/nft_compat.c
index c21e7eb..f753ec6 100644
--- a/net/netfilter/nft_compat.c
+++ b/net/netfilter/nft_compat.c
@@ -42,7 +42,8 @@
{
const struct nft_base_chain *basechain;
- if (!tablename || !(chain->flags & NFT_BASE_CHAIN))
+ if (!tablename ||
+ !nft_is_base_chain(chain))
return 0;
basechain = nft_base_chain(chain);
@@ -165,7 +166,7 @@
par->entryinfo = entry;
par->target = target;
par->targinfo = info;
- if (ctx->chain->flags & NFT_BASE_CHAIN) {
+ if (nft_is_base_chain(ctx->chain)) {
const struct nft_base_chain *basechain =
nft_base_chain(ctx->chain);
const struct nf_hook_ops *ops = &basechain->ops[0];
@@ -200,7 +201,7 @@
int err;
err = nla_parse_nested(tb, NFTA_RULE_COMPAT_MAX, attr,
- nft_rule_compat_policy);
+ nft_rule_compat_policy, NULL);
if (err < 0)
return err;
@@ -230,10 +231,6 @@
union nft_entry e = {};
int ret;
- ret = nft_compat_chain_validate_dependency(target->table, ctx->chain);
- if (ret < 0)
- goto err;
-
target_compat_from_user(target, nla_data(tb[NFTA_TARGET_INFO]), info);
if (ctx->nla[NFTA_RULE_COMPAT]) {
@@ -302,7 +299,7 @@
unsigned int hook_mask = 0;
int ret;
- if (ctx->chain->flags & NFT_BASE_CHAIN) {
+ if (nft_is_base_chain(ctx->chain)) {
const struct nft_base_chain *basechain =
nft_base_chain(ctx->chain);
const struct nf_hook_ops *ops = &basechain->ops[0];
@@ -383,7 +380,7 @@
par->entryinfo = entry;
par->match = match;
par->matchinfo = info;
- if (ctx->chain->flags & NFT_BASE_CHAIN) {
+ if (nft_is_base_chain(ctx->chain)) {
const struct nft_base_chain *basechain =
nft_base_chain(ctx->chain);
const struct nf_hook_ops *ops = &basechain->ops[0];
@@ -419,10 +416,6 @@
union nft_entry e = {};
int ret;
- ret = nft_compat_chain_validate_dependency(match->table, ctx->chain);
- if (ret < 0)
- goto err;
-
match_compat_from_user(match, nla_data(tb[NFTA_MATCH_INFO]), info);
if (ctx->nla[NFTA_RULE_COMPAT]) {
@@ -485,7 +478,7 @@
unsigned int hook_mask = 0;
int ret;
- if (ctx->chain->flags & NFT_BASE_CHAIN) {
+ if (nft_is_base_chain(ctx->chain)) {
const struct nft_base_chain *basechain =
nft_base_chain(ctx->chain);
const struct nf_hook_ops *ops = &basechain->ops[0];
@@ -511,7 +504,7 @@
struct nfgenmsg *nfmsg;
unsigned int flags = portid ? NLM_F_MULTI : 0;
- event |= NFNL_SUBSYS_NFT_COMPAT << 8;
+ event = nfnl_msg_type(NFNL_SUBSYS_NFT_COMPAT, event);
nlh = nlmsg_put(skb, portid, seq, event, sizeof(*nfmsg), flags);
if (nlh == NULL)
goto nlmsg_failure;
diff --git a/net/netfilter/nft_counter.c b/net/netfilter/nft_counter.c
index 7f84222..67a710e 100644
--- a/net/netfilter/nft_counter.c
+++ b/net/netfilter/nft_counter.c
@@ -82,7 +82,8 @@
return 0;
}
-static int nft_counter_obj_init(const struct nlattr * const tb[],
+static int nft_counter_obj_init(const struct nft_ctx *ctx,
+ const struct nlattr * const tb[],
struct nft_object *obj)
{
struct nft_counter_percpu_priv *priv = nft_obj_data(obj);
diff --git a/net/netfilter/nft_ct.c b/net/netfilter/nft_ct.c
index 0264258..a34ceb3 100644
--- a/net/netfilter/nft_ct.c
+++ b/net/netfilter/nft_ct.c
@@ -32,6 +32,12 @@
};
};
+struct nft_ct_helper_obj {
+ struct nf_conntrack_helper *helper4;
+ struct nf_conntrack_helper *helper6;
+ u8 l4proto;
+};
+
#ifdef CONFIG_NF_CONNTRACK_ZONES
static DEFINE_PER_CPU(struct nf_conn *, nft_ct_pcpu_template);
static unsigned int nft_ct_pcpu_template_refcnt __read_mostly;
@@ -66,12 +72,12 @@
switch (priv->key) {
case NFT_CT_STATE:
- if (ct == NULL)
- state = NF_CT_STATE_INVALID_BIT;
- else if (nf_ct_is_untracked(ct))
+ if (ct)
+ state = NF_CT_STATE_BIT(ctinfo);
+ else if (ctinfo == IP_CT_UNTRACKED)
state = NF_CT_STATE_UNTRACKED_BIT;
else
- state = NF_CT_STATE_BIT(ctinfo);
+ state = NF_CT_STATE_INVALID_BIT;
*dest = state;
return;
default:
@@ -258,7 +264,7 @@
struct nf_conn *ct;
ct = nf_ct_get(skb, &ctinfo);
- if (ct == NULL)
+ if (ct == NULL || nf_ct_is_template(ct))
return;
switch (priv->key) {
@@ -278,6 +284,22 @@
NF_CT_LABELS_MAX_SIZE / sizeof(u32));
break;
#endif
+#ifdef CONFIG_NF_CONNTRACK_EVENTS
+ case NFT_CT_EVENTMASK: {
+ struct nf_conntrack_ecache *e = nf_ct_ecache_find(ct);
+ u32 ctmask = regs->data[priv->sreg];
+
+ if (e) {
+ if (e->ctmask != ctmask)
+ e->ctmask = ctmask;
+ break;
+ }
+
+ if (ctmask && !nf_ct_is_confirmed(ct))
+ nf_ct_ecache_ext_add(ct, ctmask, 0, GFP_ATOMIC);
+ break;
+ }
+#endif
default:
break;
}
@@ -533,6 +555,13 @@
len = sizeof(u16);
break;
#endif
+#ifdef CONFIG_NF_CONNTRACK_EVENTS
+ case NFT_CT_EVENTMASK:
+ if (tb[NFTA_CT_DIRECTION])
+ return -EINVAL;
+ len = sizeof(u32);
+ break;
+#endif
default:
return -EOPNOTSUPP;
}
@@ -696,7 +725,7 @@
static struct nft_expr_type nft_ct_type __read_mostly = {
.name = "ct",
- .select_ops = &nft_ct_select_ops,
+ .select_ops = nft_ct_select_ops,
.policy = nft_ct_policy,
.maxattr = NFTA_CT_MAX,
.owner = THIS_MODULE,
@@ -712,12 +741,10 @@
ct = nf_ct_get(pkt->skb, &ctinfo);
/* Previously seen (loopback or untracked)? Ignore. */
- if (ct)
+ if (ct || ctinfo == IP_CT_UNTRACKED)
return;
- ct = nf_ct_untracked_get();
- atomic_inc(&ct->ct_general.use);
- nf_ct_set(skb, ct, IP_CT_NEW);
+ nf_ct_set(skb, ct, IP_CT_UNTRACKED);
}
static struct nft_expr_type nft_notrack_type;
@@ -733,6 +760,162 @@
.owner = THIS_MODULE,
};
+static int nft_ct_helper_obj_init(const struct nft_ctx *ctx,
+ const struct nlattr * const tb[],
+ struct nft_object *obj)
+{
+ struct nft_ct_helper_obj *priv = nft_obj_data(obj);
+ struct nf_conntrack_helper *help4, *help6;
+ char name[NF_CT_HELPER_NAME_LEN];
+ int family = ctx->afi->family;
+
+ if (!tb[NFTA_CT_HELPER_NAME] || !tb[NFTA_CT_HELPER_L4PROTO])
+ return -EINVAL;
+
+ priv->l4proto = nla_get_u8(tb[NFTA_CT_HELPER_L4PROTO]);
+ if (!priv->l4proto)
+ return -ENOENT;
+
+ nla_strlcpy(name, tb[NFTA_CT_HELPER_NAME], sizeof(name));
+
+ if (tb[NFTA_CT_HELPER_L3PROTO])
+ family = ntohs(nla_get_be16(tb[NFTA_CT_HELPER_L3PROTO]));
+
+ help4 = NULL;
+ help6 = NULL;
+
+ switch (family) {
+ case NFPROTO_IPV4:
+ if (ctx->afi->family == NFPROTO_IPV6)
+ return -EINVAL;
+
+ help4 = nf_conntrack_helper_try_module_get(name, family,
+ priv->l4proto);
+ break;
+ case NFPROTO_IPV6:
+ if (ctx->afi->family == NFPROTO_IPV4)
+ return -EINVAL;
+
+ help6 = nf_conntrack_helper_try_module_get(name, family,
+ priv->l4proto);
+ break;
+ case NFPROTO_NETDEV: /* fallthrough */
+ case NFPROTO_BRIDGE: /* same */
+ case NFPROTO_INET:
+ help4 = nf_conntrack_helper_try_module_get(name, NFPROTO_IPV4,
+ priv->l4proto);
+ help6 = nf_conntrack_helper_try_module_get(name, NFPROTO_IPV6,
+ priv->l4proto);
+ break;
+ default:
+ return -EAFNOSUPPORT;
+ }
+
+ /* && is intentional; only error if INET found neither ipv4 or ipv6 */
+ if (!help4 && !help6)
+ return -ENOENT;
+
+ priv->helper4 = help4;
+ priv->helper6 = help6;
+
+ return 0;
+}
+
+static void nft_ct_helper_obj_destroy(struct nft_object *obj)
+{
+ struct nft_ct_helper_obj *priv = nft_obj_data(obj);
+
+ if (priv->helper4)
+ module_put(priv->helper4->me);
+ if (priv->helper6)
+ module_put(priv->helper6->me);
+}
+
+static void nft_ct_helper_obj_eval(struct nft_object *obj,
+ struct nft_regs *regs,
+ const struct nft_pktinfo *pkt)
+{
+ const struct nft_ct_helper_obj *priv = nft_obj_data(obj);
+ struct nf_conn *ct = (struct nf_conn *)skb_nfct(pkt->skb);
+ struct nf_conntrack_helper *to_assign = NULL;
+ struct nf_conn_help *help;
+
+ if (!ct ||
+ nf_ct_is_confirmed(ct) ||
+ nf_ct_is_template(ct) ||
+ priv->l4proto != nf_ct_protonum(ct))
+ return;
+
+ switch (nf_ct_l3num(ct)) {
+ case NFPROTO_IPV4:
+ to_assign = priv->helper4;
+ break;
+ case NFPROTO_IPV6:
+ to_assign = priv->helper6;
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ return;
+ }
+
+ if (!to_assign)
+ return;
+
+ if (test_bit(IPS_HELPER_BIT, &ct->status))
+ return;
+
+ help = nf_ct_helper_ext_add(ct, to_assign, GFP_ATOMIC);
+ if (help) {
+ rcu_assign_pointer(help->helper, to_assign);
+ set_bit(IPS_HELPER_BIT, &ct->status);
+ }
+}
+
+static int nft_ct_helper_obj_dump(struct sk_buff *skb,
+ struct nft_object *obj, bool reset)
+{
+ const struct nft_ct_helper_obj *priv = nft_obj_data(obj);
+ const struct nf_conntrack_helper *helper = priv->helper4;
+ u16 family;
+
+ if (nla_put_string(skb, NFTA_CT_HELPER_NAME, helper->name))
+ return -1;
+
+ if (nla_put_u8(skb, NFTA_CT_HELPER_L4PROTO, priv->l4proto))
+ return -1;
+
+ if (priv->helper4 && priv->helper6)
+ family = NFPROTO_INET;
+ else if (priv->helper6)
+ family = NFPROTO_IPV6;
+ else
+ family = NFPROTO_IPV4;
+
+ if (nla_put_be16(skb, NFTA_CT_HELPER_L3PROTO, htons(family)))
+ return -1;
+
+ return 0;
+}
+
+static const struct nla_policy nft_ct_helper_policy[NFTA_CT_HELPER_MAX + 1] = {
+ [NFTA_CT_HELPER_NAME] = { .type = NLA_STRING,
+ .len = NF_CT_HELPER_NAME_LEN - 1 },
+ [NFTA_CT_HELPER_L3PROTO] = { .type = NLA_U16 },
+ [NFTA_CT_HELPER_L4PROTO] = { .type = NLA_U8 },
+};
+
+static struct nft_object_type nft_ct_helper_obj __read_mostly = {
+ .type = NFT_OBJECT_CT_HELPER,
+ .size = sizeof(struct nft_ct_helper_obj),
+ .maxattr = NFTA_CT_HELPER_MAX,
+ .policy = nft_ct_helper_policy,
+ .eval = nft_ct_helper_obj_eval,
+ .init = nft_ct_helper_obj_init,
+ .destroy = nft_ct_helper_obj_destroy,
+ .dump = nft_ct_helper_obj_dump,
+ .owner = THIS_MODULE,
+};
+
static int __init nft_ct_module_init(void)
{
int err;
@@ -747,7 +930,14 @@
if (err < 0)
goto err1;
+ err = nft_register_obj(&nft_ct_helper_obj);
+ if (err < 0)
+ goto err2;
+
return 0;
+
+err2:
+ nft_unregister_expr(&nft_notrack_type);
err1:
nft_unregister_expr(&nft_ct_type);
return err;
@@ -755,6 +945,7 @@
static void __exit nft_ct_module_exit(void)
{
+ nft_unregister_obj(&nft_ct_helper_obj);
nft_unregister_expr(&nft_notrack_type);
nft_unregister_expr(&nft_ct_type);
}
@@ -766,3 +957,4 @@
MODULE_AUTHOR("Patrick McHardy <kaber@trash.net>");
MODULE_ALIAS_NFT_EXPR("ct");
MODULE_ALIAS_NFT_EXPR("notrack");
+MODULE_ALIAS_NFT_OBJ(NFT_OBJECT_CT_HELPER);
diff --git a/net/netfilter/nft_dynset.c b/net/netfilter/nft_dynset.c
index fafbeea..66221ad 100644
--- a/net/netfilter/nft_dynset.c
+++ b/net/netfilter/nft_dynset.c
@@ -132,16 +132,10 @@
priv->invert = true;
}
- set = nf_tables_set_lookup(ctx->table, tb[NFTA_DYNSET_SET_NAME],
- genmask);
- if (IS_ERR(set)) {
- if (tb[NFTA_DYNSET_SET_ID])
- set = nf_tables_set_lookup_byid(ctx->net,
- tb[NFTA_DYNSET_SET_ID],
- genmask);
- if (IS_ERR(set))
- return PTR_ERR(set);
- }
+ set = nft_set_lookup(ctx->net, ctx->table, tb[NFTA_DYNSET_SET_NAME],
+ tb[NFTA_DYNSET_SET_ID], genmask);
+ if (IS_ERR(set))
+ return PTR_ERR(set);
if (set->ops->update == NULL)
return -EOPNOTSUPP;
diff --git a/net/netfilter/nft_exthdr.c b/net/netfilter/nft_exthdr.c
index c308920..1ec49fe 100644
--- a/net/netfilter/nft_exthdr.c
+++ b/net/netfilter/nft_exthdr.c
@@ -98,14 +98,21 @@
goto err;
offset = i + priv->offset;
- dest[priv->len / NFT_REG32_SIZE] = 0;
- memcpy(dest, opt + offset, priv->len);
+ if (priv->flags & NFT_EXTHDR_F_PRESENT) {
+ *dest = 1;
+ } else {
+ dest[priv->len / NFT_REG32_SIZE] = 0;
+ memcpy(dest, opt + offset, priv->len);
+ }
return;
}
err:
- regs->verdict.code = NFT_BREAK;
+ if (priv->flags & NFT_EXTHDR_F_PRESENT)
+ *dest = 0;
+ else
+ regs->verdict.code = NFT_BREAK;
}
static const struct nla_policy nft_exthdr_policy[NFTA_EXTHDR_MAX + 1] = {
@@ -225,7 +232,7 @@
static struct nft_expr_type nft_exthdr_type __read_mostly = {
.name = "exthdr",
- .select_ops = &nft_exthdr_select_ops,
+ .select_ops = nft_exthdr_select_ops,
.policy = nft_exthdr_policy,
.maxattr = NFTA_EXTHDR_MAX,
.owner = THIS_MODULE,
diff --git a/net/netfilter/nft_fib.c b/net/netfilter/nft_fib.c
index 29a4906..21df8cc 100644
--- a/net/netfilter/nft_fib.c
+++ b/net/netfilter/nft_fib.c
@@ -24,7 +24,8 @@
EXPORT_SYMBOL(nft_fib_policy);
#define NFTA_FIB_F_ALL (NFTA_FIB_F_SADDR | NFTA_FIB_F_DADDR | \
- NFTA_FIB_F_MARK | NFTA_FIB_F_IIF | NFTA_FIB_F_OIF)
+ NFTA_FIB_F_MARK | NFTA_FIB_F_IIF | NFTA_FIB_F_OIF | \
+ NFTA_FIB_F_PRESENT)
int nft_fib_validate(const struct nft_ctx *ctx, const struct nft_expr *expr,
const struct nft_data **data)
@@ -112,7 +113,7 @@
if (err < 0)
return err;
- return nft_fib_validate(ctx, expr, NULL);
+ return 0;
}
EXPORT_SYMBOL_GPL(nft_fib_init);
@@ -133,19 +134,22 @@
}
EXPORT_SYMBOL_GPL(nft_fib_dump);
-void nft_fib_store_result(void *reg, enum nft_fib_result r,
+void nft_fib_store_result(void *reg, const struct nft_fib *priv,
const struct nft_pktinfo *pkt, int index)
{
struct net_device *dev;
u32 *dreg = reg;
- switch (r) {
+ switch (priv->result) {
case NFT_FIB_RESULT_OIF:
- *dreg = index;
+ *dreg = (priv->flags & NFTA_FIB_F_PRESENT) ? !!index : index;
break;
case NFT_FIB_RESULT_OIFNAME:
dev = dev_get_by_index_rcu(nft_net(pkt), index);
- strncpy(reg, dev ? dev->name : "", IFNAMSIZ);
+ if (priv->flags & NFTA_FIB_F_PRESENT)
+ *dreg = !!dev;
+ else
+ strncpy(reg, dev ? dev->name : "", IFNAMSIZ);
break;
default:
WARN_ON_ONCE(1);
diff --git a/net/netfilter/nft_hash.c b/net/netfilter/nft_hash.c
index c4dad12..24f2f75 100644
--- a/net/netfilter/nft_hash.c
+++ b/net/netfilter/nft_hash.c
@@ -17,7 +17,7 @@
#include <net/netfilter/nf_tables_core.h>
#include <linux/jhash.h>
-struct nft_hash {
+struct nft_jhash {
enum nft_registers sreg:8;
enum nft_registers dreg:8;
u8 len;
@@ -27,11 +27,11 @@
u32 offset;
};
-static void nft_hash_eval(const struct nft_expr *expr,
- struct nft_regs *regs,
- const struct nft_pktinfo *pkt)
+static void nft_jhash_eval(const struct nft_expr *expr,
+ struct nft_regs *regs,
+ const struct nft_pktinfo *pkt)
{
- struct nft_hash *priv = nft_expr_priv(expr);
+ struct nft_jhash *priv = nft_expr_priv(expr);
const void *data = ®s->data[priv->sreg];
u32 h;
@@ -39,6 +39,25 @@
regs->data[priv->dreg] = h + priv->offset;
}
+struct nft_symhash {
+ enum nft_registers dreg:8;
+ u32 modulus;
+ u32 offset;
+};
+
+static void nft_symhash_eval(const struct nft_expr *expr,
+ struct nft_regs *regs,
+ const struct nft_pktinfo *pkt)
+{
+ struct nft_symhash *priv = nft_expr_priv(expr);
+ struct sk_buff *skb = pkt->skb;
+ u32 h;
+
+ h = reciprocal_scale(__skb_get_hash_symmetric(skb), priv->modulus);
+
+ regs->data[priv->dreg] = h + priv->offset;
+}
+
static const struct nla_policy nft_hash_policy[NFTA_HASH_MAX + 1] = {
[NFTA_HASH_SREG] = { .type = NLA_U32 },
[NFTA_HASH_DREG] = { .type = NLA_U32 },
@@ -46,13 +65,14 @@
[NFTA_HASH_MODULUS] = { .type = NLA_U32 },
[NFTA_HASH_SEED] = { .type = NLA_U32 },
[NFTA_HASH_OFFSET] = { .type = NLA_U32 },
+ [NFTA_HASH_TYPE] = { .type = NLA_U32 },
};
-static int nft_hash_init(const struct nft_ctx *ctx,
- const struct nft_expr *expr,
- const struct nlattr * const tb[])
+static int nft_jhash_init(const struct nft_ctx *ctx,
+ const struct nft_expr *expr,
+ const struct nlattr * const tb[])
{
- struct nft_hash *priv = nft_expr_priv(expr);
+ struct nft_jhash *priv = nft_expr_priv(expr);
u32 len;
int err;
@@ -95,10 +115,36 @@
NFT_DATA_VALUE, sizeof(u32));
}
-static int nft_hash_dump(struct sk_buff *skb,
- const struct nft_expr *expr)
+static int nft_symhash_init(const struct nft_ctx *ctx,
+ const struct nft_expr *expr,
+ const struct nlattr * const tb[])
{
- const struct nft_hash *priv = nft_expr_priv(expr);
+ struct nft_symhash *priv = nft_expr_priv(expr);
+
+ if (!tb[NFTA_HASH_DREG] ||
+ !tb[NFTA_HASH_MODULUS])
+ return -EINVAL;
+
+ if (tb[NFTA_HASH_OFFSET])
+ priv->offset = ntohl(nla_get_be32(tb[NFTA_HASH_OFFSET]));
+
+ priv->dreg = nft_parse_register(tb[NFTA_HASH_DREG]);
+
+ priv->modulus = ntohl(nla_get_be32(tb[NFTA_HASH_MODULUS]));
+ if (priv->modulus <= 1)
+ return -ERANGE;
+
+ if (priv->offset + priv->modulus - 1 < priv->offset)
+ return -EOVERFLOW;
+
+ return nft_validate_register_store(ctx, priv->dreg, NULL,
+ NFT_DATA_VALUE, sizeof(u32));
+}
+
+static int nft_jhash_dump(struct sk_buff *skb,
+ const struct nft_expr *expr)
+{
+ const struct nft_jhash *priv = nft_expr_priv(expr);
if (nft_dump_register(skb, NFTA_HASH_SREG, priv->sreg))
goto nla_put_failure;
@@ -114,6 +160,28 @@
if (priv->offset != 0)
if (nla_put_be32(skb, NFTA_HASH_OFFSET, htonl(priv->offset)))
goto nla_put_failure;
+ if (nla_put_be32(skb, NFTA_HASH_TYPE, htonl(NFT_HASH_JENKINS)))
+ goto nla_put_failure;
+ return 0;
+
+nla_put_failure:
+ return -1;
+}
+
+static int nft_symhash_dump(struct sk_buff *skb,
+ const struct nft_expr *expr)
+{
+ const struct nft_symhash *priv = nft_expr_priv(expr);
+
+ if (nft_dump_register(skb, NFTA_HASH_DREG, priv->dreg))
+ goto nla_put_failure;
+ if (nla_put_be32(skb, NFTA_HASH_MODULUS, htonl(priv->modulus)))
+ goto nla_put_failure;
+ if (priv->offset != 0)
+ if (nla_put_be32(skb, NFTA_HASH_OFFSET, htonl(priv->offset)))
+ goto nla_put_failure;
+ if (nla_put_be32(skb, NFTA_HASH_TYPE, htonl(NFT_HASH_SYM)))
+ goto nla_put_failure;
return 0;
nla_put_failure:
@@ -121,17 +189,46 @@
}
static struct nft_expr_type nft_hash_type;
-static const struct nft_expr_ops nft_hash_ops = {
+static const struct nft_expr_ops nft_jhash_ops = {
.type = &nft_hash_type,
- .size = NFT_EXPR_SIZE(sizeof(struct nft_hash)),
- .eval = nft_hash_eval,
- .init = nft_hash_init,
- .dump = nft_hash_dump,
+ .size = NFT_EXPR_SIZE(sizeof(struct nft_jhash)),
+ .eval = nft_jhash_eval,
+ .init = nft_jhash_init,
+ .dump = nft_jhash_dump,
};
+static const struct nft_expr_ops nft_symhash_ops = {
+ .type = &nft_hash_type,
+ .size = NFT_EXPR_SIZE(sizeof(struct nft_symhash)),
+ .eval = nft_symhash_eval,
+ .init = nft_symhash_init,
+ .dump = nft_symhash_dump,
+};
+
+static const struct nft_expr_ops *
+nft_hash_select_ops(const struct nft_ctx *ctx,
+ const struct nlattr * const tb[])
+{
+ u32 type;
+
+ if (!tb[NFTA_HASH_TYPE])
+ return &nft_jhash_ops;
+
+ type = ntohl(nla_get_be32(tb[NFTA_HASH_TYPE]));
+ switch (type) {
+ case NFT_HASH_SYM:
+ return &nft_symhash_ops;
+ case NFT_HASH_JENKINS:
+ return &nft_jhash_ops;
+ default:
+ break;
+ }
+ return ERR_PTR(-EOPNOTSUPP);
+}
+
static struct nft_expr_type nft_hash_type __read_mostly = {
.name = "hash",
- .ops = &nft_hash_ops,
+ .select_ops = nft_hash_select_ops,
.policy = nft_hash_policy,
.maxattr = NFTA_HASH_MAX,
.owner = THIS_MODULE,
diff --git a/net/netfilter/nft_limit.c b/net/netfilter/nft_limit.c
index c6baf41..18dd57a 100644
--- a/net/netfilter/nft_limit.c
+++ b/net/netfilter/nft_limit.c
@@ -17,9 +17,8 @@
#include <linux/netfilter/nf_tables.h>
#include <net/netfilter/nf_tables.h>
-static DEFINE_SPINLOCK(limit_lock);
-
struct nft_limit {
+ spinlock_t lock;
u64 last;
u64 tokens;
u64 tokens_max;
@@ -34,7 +33,7 @@
u64 now, tokens;
s64 delta;
- spin_lock_bh(&limit_lock);
+ spin_lock_bh(&limit->lock);
now = ktime_get_ns();
tokens = limit->tokens + now - limit->last;
if (tokens > limit->tokens_max)
@@ -44,11 +43,11 @@
delta = tokens - cost;
if (delta >= 0) {
limit->tokens = delta;
- spin_unlock_bh(&limit_lock);
+ spin_unlock_bh(&limit->lock);
return limit->invert;
}
limit->tokens = tokens;
- spin_unlock_bh(&limit_lock);
+ spin_unlock_bh(&limit->lock);
return !limit->invert;
}
@@ -86,6 +85,7 @@
limit->invert = true;
}
limit->last = ktime_get_ns();
+ spin_lock_init(&limit->lock);
return 0;
}
diff --git a/net/netfilter/nft_lookup.c b/net/netfilter/nft_lookup.c
index e21aea7..475570e 100644
--- a/net/netfilter/nft_lookup.c
+++ b/net/netfilter/nft_lookup.c
@@ -71,16 +71,10 @@
tb[NFTA_LOOKUP_SREG] == NULL)
return -EINVAL;
- set = nf_tables_set_lookup(ctx->table, tb[NFTA_LOOKUP_SET], genmask);
- if (IS_ERR(set)) {
- if (tb[NFTA_LOOKUP_SET_ID]) {
- set = nf_tables_set_lookup_byid(ctx->net,
- tb[NFTA_LOOKUP_SET_ID],
- genmask);
- }
- if (IS_ERR(set))
- return PTR_ERR(set);
- }
+ set = nft_set_lookup(ctx->net, ctx->table, tb[NFTA_LOOKUP_SET],
+ tb[NFTA_LOOKUP_SET_ID], genmask);
+ if (IS_ERR(set))
+ return PTR_ERR(set);
if (set->flags & NFT_SET_EVAL)
return -EOPNOTSUPP;
diff --git a/net/netfilter/nft_masq.c b/net/netfilter/nft_masq.c
index 11ce016..6ac03d4 100644
--- a/net/netfilter/nft_masq.c
+++ b/net/netfilter/nft_masq.c
@@ -46,10 +46,6 @@
struct nft_masq *priv = nft_expr_priv(expr);
int err;
- err = nft_masq_validate(ctx, expr, NULL);
- if (err)
- return err;
-
if (tb[NFTA_MASQ_FLAGS]) {
priv->flags = ntohl(nla_get_be32(tb[NFTA_MASQ_FLAGS]));
if (priv->flags & ~NF_NAT_RANGE_MASK)
diff --git a/net/netfilter/nft_meta.c b/net/netfilter/nft_meta.c
index 7b60e01..5a60eb2 100644
--- a/net/netfilter/nft_meta.c
+++ b/net/netfilter/nft_meta.c
@@ -372,10 +372,6 @@
return -EOPNOTSUPP;
}
- err = nft_meta_set_validate(ctx, expr, NULL);
- if (err < 0)
- return err;
-
priv->sreg = nft_parse_register(tb[NFTA_META_SREG]);
err = nft_validate_register_load(priv->sreg, len);
if (err < 0)
@@ -471,7 +467,7 @@
static struct nft_expr_type nft_meta_type __read_mostly = {
.name = "meta",
- .select_ops = &nft_meta_select_ops,
+ .select_ops = nft_meta_select_ops,
.policy = nft_meta_policy,
.maxattr = NFTA_META_MAX,
.owner = THIS_MODULE,
diff --git a/net/netfilter/nft_nat.c b/net/netfilter/nft_nat.c
index 439e0bd..ed548d0 100644
--- a/net/netfilter/nft_nat.c
+++ b/net/netfilter/nft_nat.c
@@ -138,10 +138,6 @@
return -EINVAL;
}
- err = nft_nat_validate(ctx, expr, NULL);
- if (err < 0)
- return err;
-
if (tb[NFTA_NAT_FAMILY] == NULL)
return -EINVAL;
diff --git a/net/netfilter/nft_numgen.c b/net/netfilter/nft_numgen.c
index a66b360..5a3a52c 100644
--- a/net/netfilter/nft_numgen.c
+++ b/net/netfilter/nft_numgen.c
@@ -188,7 +188,7 @@
static struct nft_expr_type nft_ng_type __read_mostly = {
.name = "numgen",
- .select_ops = &nft_ng_select_ops,
+ .select_ops = nft_ng_select_ops,
.policy = nft_ng_policy,
.maxattr = NFTA_NG_MAX,
.owner = THIS_MODULE,
diff --git a/net/netfilter/nft_objref.c b/net/netfilter/nft_objref.c
index 1ae8c49..1dd428f 100644
--- a/net/netfilter/nft_objref.c
+++ b/net/netfilter/nft_objref.c
@@ -116,16 +116,10 @@
struct nft_set *set;
int err;
- set = nf_tables_set_lookup(ctx->table, tb[NFTA_OBJREF_SET_NAME], genmask);
- if (IS_ERR(set)) {
- if (tb[NFTA_OBJREF_SET_ID]) {
- set = nf_tables_set_lookup_byid(ctx->net,
- tb[NFTA_OBJREF_SET_ID],
- genmask);
- }
- if (IS_ERR(set))
- return PTR_ERR(set);
- }
+ set = nft_set_lookup(ctx->net, ctx->table, tb[NFTA_OBJREF_SET_NAME],
+ tb[NFTA_OBJREF_SET_ID], genmask);
+ if (IS_ERR(set))
+ return PTR_ERR(set);
if (!(set->flags & NFT_SET_OBJECT))
return -EINVAL;
diff --git a/net/netfilter/nft_queue.c b/net/netfilter/nft_queue.c
index dbb6aaf..9861365 100644
--- a/net/netfilter/nft_queue.c
+++ b/net/netfilter/nft_queue.c
@@ -197,7 +197,7 @@
static struct nft_expr_type nft_queue_type __read_mostly = {
.name = "queue",
- .select_ops = &nft_queue_select_ops,
+ .select_ops = nft_queue_select_ops,
.policy = nft_queue_policy,
.maxattr = NFTA_QUEUE_MAX,
.owner = THIS_MODULE,
diff --git a/net/netfilter/nft_quota.c b/net/netfilter/nft_quota.c
index 2d6fe35..25e3315 100644
--- a/net/netfilter/nft_quota.c
+++ b/net/netfilter/nft_quota.c
@@ -99,7 +99,8 @@
return 0;
}
-static int nft_quota_obj_init(const struct nlattr * const tb[],
+static int nft_quota_obj_init(const struct nft_ctx *ctx,
+ const struct nlattr * const tb[],
struct nft_object *obj)
{
struct nft_quota *priv = nft_obj_data(obj);
diff --git a/net/netfilter/nft_redir.c b/net/netfilter/nft_redir.c
index 40dcd05..1e66538 100644
--- a/net/netfilter/nft_redir.c
+++ b/net/netfilter/nft_redir.c
@@ -47,10 +47,6 @@
unsigned int plen;
int err;
- err = nft_redir_validate(ctx, expr, NULL);
- if (err < 0)
- return err;
-
plen = FIELD_SIZEOF(struct nf_nat_range, min_addr.all);
if (tb[NFTA_REDIR_REG_PROTO_MIN]) {
priv->sreg_proto_min =
diff --git a/net/netfilter/nft_reject.c b/net/netfilter/nft_reject.c
index c64de3f7..29f5bd2 100644
--- a/net/netfilter/nft_reject.c
+++ b/net/netfilter/nft_reject.c
@@ -42,11 +42,6 @@
const struct nlattr * const tb[])
{
struct nft_reject *priv = nft_expr_priv(expr);
- int err;
-
- err = nft_reject_validate(ctx, expr, NULL);
- if (err < 0)
- return err;
if (tb[NFTA_REJECT_TYPE] == NULL)
return -EINVAL;
diff --git a/net/netfilter/nft_reject_inet.c b/net/netfilter/nft_reject_inet.c
index 9e90a02..5a7fb5f 100644
--- a/net/netfilter/nft_reject_inet.c
+++ b/net/netfilter/nft_reject_inet.c
@@ -66,11 +66,7 @@
const struct nlattr * const tb[])
{
struct nft_reject *priv = nft_expr_priv(expr);
- int icmp_code, err;
-
- err = nft_reject_validate(ctx, expr, NULL);
- if (err < 0)
- return err;
+ int icmp_code;
if (tb[NFTA_REJECT_TYPE] == NULL)
return -EINVAL;
diff --git a/net/netfilter/nft_set_hash.c b/net/netfilter/nft_set_hash.c
index 5f65272..8ec086b 100644
--- a/net/netfilter/nft_set_hash.c
+++ b/net/netfilter/nft_set_hash.c
@@ -352,7 +352,7 @@
static void nft_hash_elem_destroy(void *ptr, void *arg)
{
- nft_set_elem_destroy((const struct nft_set *)arg, ptr, true);
+ nft_set_elem_destroy(arg, ptr, true);
}
static void nft_hash_destroy(const struct nft_set *set)
diff --git a/net/netfilter/nft_set_rbtree.c b/net/netfilter/nft_set_rbtree.c
index 78dfbf9..e97e2fb 100644
--- a/net/netfilter/nft_set_rbtree.c
+++ b/net/netfilter/nft_set_rbtree.c
@@ -18,9 +18,8 @@
#include <linux/netfilter/nf_tables.h>
#include <net/netfilter/nf_tables.h>
-static DEFINE_SPINLOCK(nft_rbtree_lock);
-
struct nft_rbtree {
+ rwlock_t lock;
struct rb_root root;
};
@@ -44,14 +43,14 @@
static bool nft_rbtree_lookup(const struct net *net, const struct nft_set *set,
const u32 *key, const struct nft_set_ext **ext)
{
- const struct nft_rbtree *priv = nft_set_priv(set);
+ struct nft_rbtree *priv = nft_set_priv(set);
const struct nft_rbtree_elem *rbe, *interval = NULL;
u8 genmask = nft_genmask_cur(net);
const struct rb_node *parent;
const void *this;
int d;
- spin_lock_bh(&nft_rbtree_lock);
+ read_lock_bh(&priv->lock);
parent = priv->root.rb_node;
while (parent != NULL) {
rbe = rb_entry(parent, struct nft_rbtree_elem, node);
@@ -75,7 +74,7 @@
}
if (nft_rbtree_interval_end(rbe))
goto out;
- spin_unlock_bh(&nft_rbtree_lock);
+ read_unlock_bh(&priv->lock);
*ext = &rbe->ext;
return true;
@@ -85,12 +84,12 @@
if (set->flags & NFT_SET_INTERVAL && interval != NULL &&
nft_set_elem_active(&interval->ext, genmask) &&
!nft_rbtree_interval_end(interval)) {
- spin_unlock_bh(&nft_rbtree_lock);
+ read_unlock_bh(&priv->lock);
*ext = &interval->ext;
return true;
}
out:
- spin_unlock_bh(&nft_rbtree_lock);
+ read_unlock_bh(&priv->lock);
return false;
}
@@ -140,12 +139,13 @@
const struct nft_set_elem *elem,
struct nft_set_ext **ext)
{
+ struct nft_rbtree *priv = nft_set_priv(set);
struct nft_rbtree_elem *rbe = elem->priv;
int err;
- spin_lock_bh(&nft_rbtree_lock);
+ write_lock_bh(&priv->lock);
err = __nft_rbtree_insert(net, set, rbe, ext);
- spin_unlock_bh(&nft_rbtree_lock);
+ write_unlock_bh(&priv->lock);
return err;
}
@@ -157,9 +157,9 @@
struct nft_rbtree *priv = nft_set_priv(set);
struct nft_rbtree_elem *rbe = elem->priv;
- spin_lock_bh(&nft_rbtree_lock);
+ write_lock_bh(&priv->lock);
rb_erase(&rbe->node, &priv->root);
- spin_unlock_bh(&nft_rbtree_lock);
+ write_unlock_bh(&priv->lock);
}
static void nft_rbtree_activate(const struct net *net,
@@ -224,12 +224,12 @@
struct nft_set *set,
struct nft_set_iter *iter)
{
- const struct nft_rbtree *priv = nft_set_priv(set);
+ struct nft_rbtree *priv = nft_set_priv(set);
struct nft_rbtree_elem *rbe;
struct nft_set_elem elem;
struct rb_node *node;
- spin_lock_bh(&nft_rbtree_lock);
+ read_lock_bh(&priv->lock);
for (node = rb_first(&priv->root); node != NULL; node = rb_next(node)) {
rbe = rb_entry(node, struct nft_rbtree_elem, node);
@@ -242,13 +242,13 @@
iter->err = iter->fn(ctx, set, iter, &elem);
if (iter->err < 0) {
- spin_unlock_bh(&nft_rbtree_lock);
+ read_unlock_bh(&priv->lock);
return;
}
cont:
iter->count++;
}
- spin_unlock_bh(&nft_rbtree_lock);
+ read_unlock_bh(&priv->lock);
}
static unsigned int nft_rbtree_privsize(const struct nlattr * const nla[])
@@ -262,6 +262,7 @@
{
struct nft_rbtree *priv = nft_set_priv(set);
+ rwlock_init(&priv->lock);
priv->root = RB_ROOT;
return 0;
}
diff --git a/net/netfilter/xt_CT.c b/net/netfilter/xt_CT.c
index 81fdcdc..bb7ad82 100644
--- a/net/netfilter/xt_CT.c
+++ b/net/netfilter/xt_CT.c
@@ -26,11 +26,12 @@
if (skb->_nfct != 0)
return XT_CONTINUE;
- /* special case the untracked ct : we want the percpu object */
- if (!ct)
- ct = nf_ct_untracked_get();
- atomic_inc(&ct->ct_general.use);
- nf_ct_set(skb, ct, IP_CT_NEW);
+ if (ct) {
+ atomic_inc(&ct->ct_general.use);
+ nf_ct_set(skb, ct, IP_CT_NEW);
+ } else {
+ nf_ct_set(skb, ct, IP_CT_UNTRACKED);
+ }
return XT_CONTINUE;
}
@@ -342,7 +343,7 @@
struct nf_conn *ct = info->ct;
struct nf_conn_help *help;
- if (ct && !nf_ct_is_untracked(ct)) {
+ if (ct) {
help = nfct_help(ct);
if (help)
module_put(help->helper->me);
@@ -419,8 +420,7 @@
if (skb->_nfct != 0)
return XT_CONTINUE;
- nf_ct_set(skb, nf_ct_untracked_get(), IP_CT_NEW);
- nf_conntrack_get(skb_nfct(skb));
+ nf_ct_set(skb, NULL, IP_CT_UNTRACKED);
return XT_CONTINUE;
}
diff --git a/net/netfilter/xt_HMARK.c b/net/netfilter/xt_HMARK.c
index 02afaf4..60e6dbe 100644
--- a/net/netfilter/xt_HMARK.c
+++ b/net/netfilter/xt_HMARK.c
@@ -84,7 +84,7 @@
struct nf_conntrack_tuple *otuple;
struct nf_conntrack_tuple *rtuple;
- if (ct == NULL || nf_ct_is_untracked(ct))
+ if (ct == NULL)
return -1;
otuple = &ct->tuplehash[IP_CT_DIR_ORIGINAL].tuple;
diff --git a/net/netfilter/xt_cluster.c b/net/netfilter/xt_cluster.c
index 9a9884a..57ef175 100644
--- a/net/netfilter/xt_cluster.c
+++ b/net/netfilter/xt_cluster.c
@@ -121,9 +121,6 @@
if (ct == NULL)
return false;
- if (nf_ct_is_untracked(ct))
- return false;
-
if (ct->master)
hash = xt_cluster_hash(ct->master, info);
else
diff --git a/net/netfilter/xt_connlabel.c b/net/netfilter/xt_connlabel.c
index 7827128..2337287 100644
--- a/net/netfilter/xt_connlabel.c
+++ b/net/netfilter/xt_connlabel.c
@@ -29,7 +29,7 @@
bool invert = info->options & XT_CONNLABEL_OP_INVERT;
ct = nf_ct_get(skb, &ctinfo);
- if (ct == NULL || nf_ct_is_untracked(ct))
+ if (ct == NULL)
return invert;
labels = nf_ct_labels_find(ct);
diff --git a/net/netfilter/xt_connmark.c b/net/netfilter/xt_connmark.c
index 9935d50..ec377cc 100644
--- a/net/netfilter/xt_connmark.c
+++ b/net/netfilter/xt_connmark.c
@@ -44,7 +44,7 @@
u_int32_t newmark;
ct = nf_ct_get(skb, &ctinfo);
- if (ct == NULL || nf_ct_is_untracked(ct))
+ if (ct == NULL)
return XT_CONTINUE;
switch (info->mode) {
@@ -97,7 +97,7 @@
const struct nf_conn *ct;
ct = nf_ct_get(skb, &ctinfo);
- if (ct == NULL || nf_ct_is_untracked(ct))
+ if (ct == NULL)
return false;
return ((ct->mark & info->mask) == info->mark) ^ info->invert;
diff --git a/net/netfilter/xt_conntrack.c b/net/netfilter/xt_conntrack.c
index c0fb217..39cf1d0 100644
--- a/net/netfilter/xt_conntrack.c
+++ b/net/netfilter/xt_conntrack.c
@@ -172,12 +172,11 @@
ct = nf_ct_get(skb, &ctinfo);
- if (ct) {
- if (nf_ct_is_untracked(ct))
- statebit = XT_CONNTRACK_STATE_UNTRACKED;
- else
- statebit = XT_CONNTRACK_STATE_BIT(ctinfo);
- } else
+ if (ct)
+ statebit = XT_CONNTRACK_STATE_BIT(ctinfo);
+ else if (ctinfo == IP_CT_UNTRACKED)
+ statebit = XT_CONNTRACK_STATE_UNTRACKED;
+ else
statebit = XT_CONNTRACK_STATE_INVALID;
if (info->match_flags & XT_CONNTRACK_STATE) {
diff --git a/net/netfilter/xt_hashlimit.c b/net/netfilter/xt_hashlimit.c
index 2a6dfe8..762e187 100644
--- a/net/netfilter/xt_hashlimit.c
+++ b/net/netfilter/xt_hashlimit.c
@@ -119,7 +119,7 @@
cfg_copy(struct hashlimit_cfg2 *to, void *from, int revision)
{
if (revision == 1) {
- struct hashlimit_cfg1 *cfg = (struct hashlimit_cfg1 *)from;
+ struct hashlimit_cfg1 *cfg = from;
to->mode = cfg->mode;
to->avg = cfg->avg;
@@ -895,7 +895,7 @@
static void *dl_seq_next(struct seq_file *s, void *v, loff_t *pos)
{
struct xt_hashlimit_htable *htable = s->private;
- unsigned int *bucket = (unsigned int *)v;
+ unsigned int *bucket = v;
*pos = ++(*bucket);
if (*pos >= htable->cfg.size) {
@@ -909,7 +909,7 @@
__releases(htable->lock)
{
struct xt_hashlimit_htable *htable = s->private;
- unsigned int *bucket = (unsigned int *)v;
+ unsigned int *bucket = v;
if (!IS_ERR(bucket))
kfree(bucket);
@@ -980,7 +980,7 @@
static int dl_seq_show_v1(struct seq_file *s, void *v)
{
struct xt_hashlimit_htable *htable = s->private;
- unsigned int *bucket = (unsigned int *)v;
+ unsigned int *bucket = v;
struct dsthash_ent *ent;
if (!hlist_empty(&htable->hash[*bucket])) {
@@ -994,7 +994,7 @@
static int dl_seq_show(struct seq_file *s, void *v)
{
struct xt_hashlimit_htable *htable = s->private;
- unsigned int *bucket = (unsigned int *)v;
+ unsigned int *bucket = v;
struct dsthash_ent *ent;
if (!hlist_empty(&htable->hash[*bucket])) {
diff --git a/net/netfilter/xt_ipvs.c b/net/netfilter/xt_ipvs.c
index 0fdc890..42540d2 100644
--- a/net/netfilter/xt_ipvs.c
+++ b/net/netfilter/xt_ipvs.c
@@ -116,7 +116,7 @@
enum ip_conntrack_info ctinfo;
struct nf_conn *ct = nf_ct_get(skb, &ctinfo);
- if (ct == NULL || nf_ct_is_untracked(ct)) {
+ if (ct == NULL) {
match = false;
goto out_put_cp;
}
diff --git a/net/netfilter/xt_limit.c b/net/netfilter/xt_limit.c
index dab962d..d27b5f1 100644
--- a/net/netfilter/xt_limit.c
+++ b/net/netfilter/xt_limit.c
@@ -18,6 +18,7 @@
#include <linux/netfilter/xt_limit.h>
struct xt_limit_priv {
+ spinlock_t lock;
unsigned long prev;
uint32_t credit;
};
@@ -32,8 +33,6 @@
* see net/sched/sch_tbf.c in the linux source tree
*/
-static DEFINE_SPINLOCK(limit_lock);
-
/* Rusty: This is my (non-mathematically-inclined) understanding of
this algorithm. The `average rate' in jiffies becomes your initial
amount of credit `credit' and the most credit you can ever have
@@ -72,7 +71,7 @@
struct xt_limit_priv *priv = r->master;
unsigned long now = jiffies;
- spin_lock_bh(&limit_lock);
+ spin_lock_bh(&priv->lock);
priv->credit += (now - xchg(&priv->prev, now)) * CREDITS_PER_JIFFY;
if (priv->credit > r->credit_cap)
priv->credit = r->credit_cap;
@@ -80,11 +79,11 @@
if (priv->credit >= r->cost) {
/* We're not limited. */
priv->credit -= r->cost;
- spin_unlock_bh(&limit_lock);
+ spin_unlock_bh(&priv->lock);
return true;
}
- spin_unlock_bh(&limit_lock);
+ spin_unlock_bh(&priv->lock);
return false;
}
@@ -126,6 +125,8 @@
r->credit_cap = priv->credit; /* Credits full. */
r->cost = user2credits(r->avg);
}
+ spin_lock_init(&priv->lock);
+
return 0;
}
diff --git a/net/netfilter/xt_recent.c b/net/netfilter/xt_recent.c
index 1d89a4e..37d581a 100644
--- a/net/netfilter/xt_recent.c
+++ b/net/netfilter/xt_recent.c
@@ -532,7 +532,7 @@
&e->addr.in6, e->ttl, e->stamps[i], e->index);
for (i = 0; i < e->nstamps; i++)
seq_printf(seq, "%s %lu", i ? "," : "", e->stamps[i]);
- seq_printf(seq, "\n");
+ seq_putc(seq, '\n');
return 0;
}
diff --git a/net/netfilter/xt_state.c b/net/netfilter/xt_state.c
index 5746a33..5fbd791 100644
--- a/net/netfilter/xt_state.c
+++ b/net/netfilter/xt_state.c
@@ -28,14 +28,13 @@
unsigned int statebit;
struct nf_conn *ct = nf_ct_get(skb, &ctinfo);
- if (!ct)
+ if (ct)
+ statebit = XT_STATE_BIT(ctinfo);
+ else if (ctinfo == IP_CT_UNTRACKED)
+ statebit = XT_STATE_UNTRACKED;
+ else
statebit = XT_STATE_INVALID;
- else {
- if (nf_ct_is_untracked(ct))
- statebit = XT_STATE_UNTRACKED;
- else
- statebit = XT_STATE_BIT(ctinfo);
- }
+
return (sinfo->statemask & statebit);
}
diff --git a/net/netlabel/netlabel_cipso_v4.c b/net/netlabel/netlabel_cipso_v4.c
index 4149d3e..9aacf2d 100644
--- a/net/netlabel/netlabel_cipso_v4.c
+++ b/net/netlabel/netlabel_cipso_v4.c
@@ -101,7 +101,7 @@
if (nla_validate_nested(info->attrs[NLBL_CIPSOV4_A_TAGLST],
NLBL_CIPSOV4_A_MAX,
- netlbl_cipsov4_genl_policy) != 0)
+ netlbl_cipsov4_genl_policy, NULL) != 0)
return -EINVAL;
nla_for_each_nested(nla, info->attrs[NLBL_CIPSOV4_A_TAGLST], nla_rem)
@@ -148,7 +148,7 @@
if (nla_validate_nested(info->attrs[NLBL_CIPSOV4_A_MLSLVLLST],
NLBL_CIPSOV4_A_MAX,
- netlbl_cipsov4_genl_policy) != 0)
+ netlbl_cipsov4_genl_policy, NULL) != 0)
return -EINVAL;
doi_def = kmalloc(sizeof(*doi_def), GFP_KERNEL);
@@ -170,10 +170,10 @@
info->attrs[NLBL_CIPSOV4_A_MLSLVLLST],
nla_a_rem)
if (nla_type(nla_a) == NLBL_CIPSOV4_A_MLSLVL) {
- if (nla_validate_nested(nla_a,
- NLBL_CIPSOV4_A_MAX,
- netlbl_cipsov4_genl_policy) != 0)
- goto add_std_failure;
+ if (nla_validate_nested(nla_a, NLBL_CIPSOV4_A_MAX,
+ netlbl_cipsov4_genl_policy,
+ NULL) != 0)
+ goto add_std_failure;
nla_for_each_nested(nla_b, nla_a, nla_b_rem)
switch (nla_type(nla_b)) {
case NLBL_CIPSOV4_A_MLSLVLLOC:
@@ -236,7 +236,7 @@
if (info->attrs[NLBL_CIPSOV4_A_MLSCATLST]) {
if (nla_validate_nested(info->attrs[NLBL_CIPSOV4_A_MLSCATLST],
NLBL_CIPSOV4_A_MAX,
- netlbl_cipsov4_genl_policy) != 0)
+ netlbl_cipsov4_genl_policy, NULL) != 0)
goto add_std_failure;
nla_for_each_nested(nla_a,
@@ -244,8 +244,9 @@
nla_a_rem)
if (nla_type(nla_a) == NLBL_CIPSOV4_A_MLSCAT) {
if (nla_validate_nested(nla_a,
- NLBL_CIPSOV4_A_MAX,
- netlbl_cipsov4_genl_policy) != 0)
+ NLBL_CIPSOV4_A_MAX,
+ netlbl_cipsov4_genl_policy,
+ NULL) != 0)
goto add_std_failure;
nla_for_each_nested(nla_b, nla_a, nla_b_rem)
switch (nla_type(nla_b)) {
diff --git a/net/netlink/af_netlink.c b/net/netlink/af_netlink.c
index 596eaff..ee841f0 100644
--- a/net/netlink/af_netlink.c
+++ b/net/netlink/af_netlink.c
@@ -78,14 +78,6 @@
/* state bits */
#define NETLINK_S_CONGESTED 0x0
-/* flags */
-#define NETLINK_F_KERNEL_SOCKET 0x1
-#define NETLINK_F_RECV_PKTINFO 0x2
-#define NETLINK_F_BROADCAST_SEND_ERROR 0x4
-#define NETLINK_F_RECV_NO_ENOBUFS 0x8
-#define NETLINK_F_LISTEN_ALL_NSID 0x10
-#define NETLINK_F_CAP_ACK 0x20
-
static inline int netlink_is_kernel(struct sock *sk)
{
return nlk_sk(sk)->flags & NETLINK_F_KERNEL_SOCKET;
@@ -1660,6 +1652,13 @@
nlk->flags &= ~NETLINK_F_CAP_ACK;
err = 0;
break;
+ case NETLINK_EXT_ACK:
+ if (val)
+ nlk->flags |= NETLINK_F_EXT_ACK;
+ else
+ nlk->flags &= ~NETLINK_F_EXT_ACK;
+ err = 0;
+ break;
default:
err = -ENOPROTOOPT;
}
@@ -1744,6 +1743,15 @@
return -EFAULT;
err = 0;
break;
+ case NETLINK_EXT_ACK:
+ if (len < sizeof(int))
+ return -EINVAL;
+ len = sizeof(int);
+ val = nlk->flags & NETLINK_F_EXT_ACK ? 1 : 0;
+ if (put_user(len, optlen) || put_user(val, optval))
+ return -EFAULT;
+ err = 0;
+ break;
default:
err = -ENOPROTOOPT;
}
@@ -2275,21 +2283,44 @@
}
EXPORT_SYMBOL(__netlink_dump_start);
-void netlink_ack(struct sk_buff *in_skb, struct nlmsghdr *nlh, int err)
+void netlink_ack(struct sk_buff *in_skb, struct nlmsghdr *nlh, int err,
+ const struct netlink_ext_ack *extack)
{
struct sk_buff *skb;
struct nlmsghdr *rep;
struct nlmsgerr *errmsg;
size_t payload = sizeof(*errmsg);
+ size_t tlvlen = 0;
struct netlink_sock *nlk = nlk_sk(NETLINK_CB(in_skb).sk);
+ unsigned int flags = 0;
/* Error messages get the original request appened, unless the user
- * requests to cap the error message.
+ * requests to cap the error message, and get extra error data if
+ * requested.
*/
- if (!(nlk->flags & NETLINK_F_CAP_ACK) && err)
- payload += nlmsg_len(nlh);
+ if (err) {
+ if (!(nlk->flags & NETLINK_F_CAP_ACK))
+ payload += nlmsg_len(nlh);
+ else
+ flags |= NLM_F_CAPPED;
+ if (nlk->flags & NETLINK_F_EXT_ACK && extack) {
+ if (extack->_msg)
+ tlvlen += nla_total_size(strlen(extack->_msg) + 1);
+ if (extack->bad_attr)
+ tlvlen += nla_total_size(sizeof(u32));
+ }
+ } else {
+ flags |= NLM_F_CAPPED;
- skb = nlmsg_new(payload, GFP_KERNEL);
+ if (nlk->flags & NETLINK_F_EXT_ACK &&
+ extack && extack->cookie_len)
+ tlvlen += nla_total_size(extack->cookie_len);
+ }
+
+ if (tlvlen)
+ flags |= NLM_F_ACK_TLVS;
+
+ skb = nlmsg_new(payload + tlvlen, GFP_KERNEL);
if (!skb) {
struct sock *sk;
@@ -2305,17 +2336,42 @@
}
rep = __nlmsg_put(skb, NETLINK_CB(in_skb).portid, nlh->nlmsg_seq,
- NLMSG_ERROR, payload, 0);
+ NLMSG_ERROR, payload, flags);
errmsg = nlmsg_data(rep);
errmsg->error = err;
memcpy(&errmsg->msg, nlh, payload > sizeof(*errmsg) ? nlh->nlmsg_len : sizeof(*nlh));
+
+ if (nlk->flags & NETLINK_F_EXT_ACK && extack) {
+ if (err) {
+ if (extack->_msg)
+ WARN_ON(nla_put_string(skb, NLMSGERR_ATTR_MSG,
+ extack->_msg));
+ if (extack->bad_attr &&
+ !WARN_ON((u8 *)extack->bad_attr < in_skb->data ||
+ (u8 *)extack->bad_attr >= in_skb->data +
+ in_skb->len))
+ WARN_ON(nla_put_u32(skb, NLMSGERR_ATTR_OFFS,
+ (u8 *)extack->bad_attr -
+ in_skb->data));
+ } else {
+ if (extack->cookie_len)
+ WARN_ON(nla_put(skb, NLMSGERR_ATTR_COOKIE,
+ extack->cookie_len,
+ extack->cookie));
+ }
+ }
+
+ nlmsg_end(skb, rep);
+
netlink_unicast(in_skb->sk, skb, NETLINK_CB(in_skb).portid, MSG_DONTWAIT);
}
EXPORT_SYMBOL(netlink_ack);
int netlink_rcv_skb(struct sk_buff *skb, int (*cb)(struct sk_buff *,
- struct nlmsghdr *))
+ struct nlmsghdr *,
+ struct netlink_ext_ack *))
{
+ struct netlink_ext_ack extack = {};
struct nlmsghdr *nlh;
int err;
@@ -2336,13 +2392,13 @@
if (nlh->nlmsg_type < NLMSG_MIN_TYPE)
goto ack;
- err = cb(skb, nlh);
+ err = cb(skb, nlh, &extack);
if (err == -EINTR)
goto skip;
ack:
if (nlh->nlmsg_flags & NLM_F_ACK || err)
- netlink_ack(skb, nlh, err);
+ netlink_ack(skb, nlh, err, &extack);
skip:
msglen = NLMSG_ALIGN(nlh->nlmsg_len);
diff --git a/net/netlink/af_netlink.h b/net/netlink/af_netlink.h
index 4fdb383..3490f24 100644
--- a/net/netlink/af_netlink.h
+++ b/net/netlink/af_netlink.h
@@ -6,6 +6,15 @@
#include <linux/workqueue.h>
#include <net/sock.h>
+/* flags */
+#define NETLINK_F_KERNEL_SOCKET 0x1
+#define NETLINK_F_RECV_PKTINFO 0x2
+#define NETLINK_F_BROADCAST_SEND_ERROR 0x4
+#define NETLINK_F_RECV_NO_ENOBUFS 0x8
+#define NETLINK_F_LISTEN_ALL_NSID 0x10
+#define NETLINK_F_CAP_ACK 0x20
+#define NETLINK_F_EXT_ACK 0x40
+
#define NLGRPSZ(x) (ALIGN(x, sizeof(unsigned long) * 8) / 8)
#define NLGRPLONGS(x) (NLGRPSZ(x)/sizeof(unsigned long))
diff --git a/net/netlink/diag.c b/net/netlink/diag.c
index a554624..8faa20b 100644
--- a/net/netlink/diag.c
+++ b/net/netlink/diag.c
@@ -19,6 +19,27 @@
nlk->groups);
}
+static int sk_diag_put_flags(struct sock *sk, struct sk_buff *skb)
+{
+ struct netlink_sock *nlk = nlk_sk(sk);
+ u32 flags = 0;
+
+ if (nlk->cb_running)
+ flags |= NDIAG_FLAG_CB_RUNNING;
+ if (nlk->flags & NETLINK_F_RECV_PKTINFO)
+ flags |= NDIAG_FLAG_PKTINFO;
+ if (nlk->flags & NETLINK_F_BROADCAST_SEND_ERROR)
+ flags |= NDIAG_FLAG_BROADCAST_ERROR;
+ if (nlk->flags & NETLINK_F_RECV_NO_ENOBUFS)
+ flags |= NDIAG_FLAG_NO_ENOBUFS;
+ if (nlk->flags & NETLINK_F_LISTEN_ALL_NSID)
+ flags |= NDIAG_FLAG_LISTEN_ALL_NSID;
+ if (nlk->flags & NETLINK_F_CAP_ACK)
+ flags |= NDIAG_FLAG_CAP_ACK;
+
+ return nla_put_u32(skb, NETLINK_DIAG_FLAGS, flags);
+}
+
static int sk_diag_fill(struct sock *sk, struct sk_buff *skb,
struct netlink_diag_req *req,
u32 portid, u32 seq, u32 flags, int sk_ino)
@@ -52,6 +73,10 @@
sock_diag_put_meminfo(sk, skb, NETLINK_DIAG_MEMINFO))
goto out_nlmsg_trim;
+ if ((req->ndiag_show & NDIAG_SHOW_FLAGS) &&
+ sk_diag_put_flags(sk, skb))
+ goto out_nlmsg_trim;
+
nlmsg_end(skb, nlh);
return 0;
diff --git a/net/netlink/genetlink.c b/net/netlink/genetlink.c
index 92e0981..10f8b4c 100644
--- a/net/netlink/genetlink.c
+++ b/net/netlink/genetlink.c
@@ -497,7 +497,8 @@
static int genl_family_rcv_msg(const struct genl_family *family,
struct sk_buff *skb,
- struct nlmsghdr *nlh)
+ struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
const struct genl_ops *ops;
struct net *net = sock_net(skb->sk);
@@ -573,7 +574,7 @@
if (attrbuf) {
err = nlmsg_parse(nlh, hdrlen, attrbuf, family->maxattr,
- ops->policy);
+ ops->policy, extack);
if (err < 0)
goto out;
}
@@ -584,6 +585,7 @@
info.genlhdr = nlmsg_data(nlh);
info.userhdr = nlmsg_data(nlh) + GENL_HDRLEN;
info.attrs = attrbuf;
+ info.extack = extack;
genl_info_net_set(&info, net);
memset(&info.user_ptr, 0, sizeof(info.user_ptr));
@@ -605,7 +607,8 @@
return err;
}
-static int genl_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int genl_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
const struct genl_family *family;
int err;
@@ -617,7 +620,7 @@
if (!family->parallel_ops)
genl_lock();
- err = genl_family_rcv_msg(family, skb, nlh);
+ err = genl_family_rcv_msg(family, skb, nlh, extack);
if (!family->parallel_ops)
genl_unlock();
diff --git a/net/nfc/netlink.c b/net/nfc/netlink.c
index 03f3d5c..6b0850e 100644
--- a/net/nfc/netlink.c
+++ b/net/nfc/netlink.c
@@ -119,7 +119,8 @@
u32 idx;
rc = nlmsg_parse(cb->nlh, GENL_HDRLEN + nfc_genl_family.hdrsize,
- attrbuf, nfc_genl_family.maxattr, nfc_genl_policy);
+ attrbuf, nfc_genl_family.maxattr, nfc_genl_policy,
+ NULL);
if (rc < 0)
return ERR_PTR(rc);
@@ -303,6 +304,17 @@
return -EMSGSIZE;
}
+static int nfc_genl_setup_device_added(struct nfc_dev *dev, struct sk_buff *msg)
+{
+ if (nla_put_string(msg, NFC_ATTR_DEVICE_NAME, nfc_device_name(dev)) ||
+ nla_put_u32(msg, NFC_ATTR_DEVICE_INDEX, dev->idx) ||
+ nla_put_u32(msg, NFC_ATTR_PROTOCOLS, dev->supported_protocols) ||
+ nla_put_u8(msg, NFC_ATTR_DEVICE_POWERED, dev->dev_up) ||
+ nla_put_u8(msg, NFC_ATTR_RF_MODE, dev->rf_mode))
+ return -1;
+ return 0;
+}
+
int nfc_genl_device_added(struct nfc_dev *dev)
{
struct sk_buff *msg;
@@ -317,10 +329,7 @@
if (!hdr)
goto free_msg;
- if (nla_put_string(msg, NFC_ATTR_DEVICE_NAME, nfc_device_name(dev)) ||
- nla_put_u32(msg, NFC_ATTR_DEVICE_INDEX, dev->idx) ||
- nla_put_u32(msg, NFC_ATTR_PROTOCOLS, dev->supported_protocols) ||
- nla_put_u8(msg, NFC_ATTR_DEVICE_POWERED, dev->dev_up))
+ if (nfc_genl_setup_device_added(dev, msg))
goto nla_put_failure;
genlmsg_end(msg, hdr);
@@ -596,11 +605,7 @@
if (cb)
genl_dump_check_consistent(cb, hdr, &nfc_genl_family);
- if (nla_put_string(msg, NFC_ATTR_DEVICE_NAME, nfc_device_name(dev)) ||
- nla_put_u32(msg, NFC_ATTR_DEVICE_INDEX, dev->idx) ||
- nla_put_u32(msg, NFC_ATTR_PROTOCOLS, dev->supported_protocols) ||
- nla_put_u8(msg, NFC_ATTR_DEVICE_POWERED, dev->dev_up) ||
- nla_put_u8(msg, NFC_ATTR_RF_MODE, dev->rf_mode))
+ if (nfc_genl_setup_device_added(dev, msg))
goto nla_put_failure;
genlmsg_end(msg, hdr);
@@ -918,7 +923,7 @@
rc = nfc_activate_target(dev, target_idx, protocol);
nfc_put_device(dev);
- return 0;
+ return rc;
}
static int nfc_genl_dep_link_up(struct sk_buff *skb, struct genl_info *info)
@@ -1161,7 +1166,7 @@
nla_for_each_nested(attr, info->attrs[NFC_ATTR_LLC_SDP], rem) {
rc = nla_parse_nested(sdp_attrs, NFC_SDP_ATTR_MAX, attr,
- nfc_sdp_genl_policy);
+ nfc_sdp_genl_policy, info->extack);
if (rc != 0) {
rc = -EINVAL;
diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c
index c82301c..e461067 100644
--- a/net/openvswitch/actions.c
+++ b/net/openvswitch/actions.c
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2007-2014 Nicira, Inc.
+ * Copyright (c) 2007-2017 Nicira, Inc.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of version 2 of the GNU General Public
@@ -44,13 +44,10 @@
#include "conntrack.h"
#include "vport.h"
-static int do_execute_actions(struct datapath *dp, struct sk_buff *skb,
- struct sw_flow_key *key,
- const struct nlattr *attr, int len);
-
struct deferred_action {
struct sk_buff *skb;
const struct nlattr *actions;
+ int actions_len;
/* Store pkt_key clone when creating deferred action. */
struct sw_flow_key pkt_key;
@@ -82,14 +79,31 @@
struct deferred_action fifo[DEFERRED_ACTION_FIFO_SIZE];
};
-struct recirc_keys {
+struct action_flow_keys {
struct sw_flow_key key[OVS_DEFERRED_ACTION_THRESHOLD];
};
static struct action_fifo __percpu *action_fifos;
-static struct recirc_keys __percpu *recirc_keys;
+static struct action_flow_keys __percpu *flow_keys;
static DEFINE_PER_CPU(int, exec_actions_level);
+/* Make a clone of the 'key', using the pre-allocated percpu 'flow_keys'
+ * space. Return NULL if out of key spaces.
+ */
+static struct sw_flow_key *clone_key(const struct sw_flow_key *key_)
+{
+ struct action_flow_keys *keys = this_cpu_ptr(flow_keys);
+ int level = this_cpu_read(exec_actions_level);
+ struct sw_flow_key *key = NULL;
+
+ if (level <= OVS_DEFERRED_ACTION_THRESHOLD) {
+ key = &keys->key[level - 1];
+ *key = *key_;
+ }
+
+ return key;
+}
+
static void action_fifo_init(struct action_fifo *fifo)
{
fifo->head = 0;
@@ -119,8 +133,9 @@
/* Return true if fifo is not full */
static struct deferred_action *add_deferred_actions(struct sk_buff *skb,
- const struct sw_flow_key *key,
- const struct nlattr *attr)
+ const struct sw_flow_key *key,
+ const struct nlattr *actions,
+ const int actions_len)
{
struct action_fifo *fifo;
struct deferred_action *da;
@@ -129,7 +144,8 @@
da = action_fifo_put(fifo);
if (da) {
da->skb = skb;
- da->actions = attr;
+ da->actions = actions;
+ da->actions_len = actions_len;
da->pkt_key = *key;
}
@@ -146,6 +162,12 @@
return !(key->mac_proto & SW_FLOW_KEY_INVALID);
}
+static int clone_execute(struct datapath *dp, struct sk_buff *skb,
+ struct sw_flow_key *key,
+ u32 recirc_id,
+ const struct nlattr *actions, int len,
+ bool last, bool clone_flow_key);
+
static void update_ethertype(struct sk_buff *skb, struct ethhdr *hdr,
__be16 ethertype)
{
@@ -908,72 +930,35 @@
return ovs_dp_upcall(dp, skb, key, &upcall, cutlen);
}
+/* When 'last' is true, sample() should always consume the 'skb'.
+ * Otherwise, sample() should keep 'skb' intact regardless what
+ * actions are executed within sample().
+ */
static int sample(struct datapath *dp, struct sk_buff *skb,
struct sw_flow_key *key, const struct nlattr *attr,
- const struct nlattr *actions, int actions_len)
+ bool last)
{
- const struct nlattr *acts_list = NULL;
- const struct nlattr *a;
- int rem;
- u32 cutlen = 0;
+ struct nlattr *actions;
+ struct nlattr *sample_arg;
+ int rem = nla_len(attr);
+ const struct sample_arg *arg;
+ bool clone_flow_key;
- for (a = nla_data(attr), rem = nla_len(attr); rem > 0;
- a = nla_next(a, &rem)) {
- u32 probability;
+ /* The first action is always 'OVS_SAMPLE_ATTR_ARG'. */
+ sample_arg = nla_data(attr);
+ arg = nla_data(sample_arg);
+ actions = nla_next(sample_arg, &rem);
- switch (nla_type(a)) {
- case OVS_SAMPLE_ATTR_PROBABILITY:
- probability = nla_get_u32(a);
- if (!probability || prandom_u32() > probability)
- return 0;
- break;
-
- case OVS_SAMPLE_ATTR_ACTIONS:
- acts_list = a;
- break;
- }
- }
-
- rem = nla_len(acts_list);
- a = nla_data(acts_list);
-
- /* Actions list is empty, do nothing */
- if (unlikely(!rem))
+ if ((arg->probability != U32_MAX) &&
+ (!arg->probability || prandom_u32() > arg->probability)) {
+ if (last)
+ consume_skb(skb);
return 0;
-
- /* The only known usage of sample action is having a single user-space
- * action, or having a truncate action followed by a single user-space
- * action. Treat this usage as a special case.
- * The output_userspace() should clone the skb to be sent to the
- * user space. This skb will be consumed by its caller.
- */
- if (unlikely(nla_type(a) == OVS_ACTION_ATTR_TRUNC)) {
- struct ovs_action_trunc *trunc = nla_data(a);
-
- if (skb->len > trunc->max_len)
- cutlen = skb->len - trunc->max_len;
-
- a = nla_next(a, &rem);
}
- if (likely(nla_type(a) == OVS_ACTION_ATTR_USERSPACE &&
- nla_is_last(a, rem)))
- return output_userspace(dp, skb, key, a, actions,
- actions_len, cutlen);
-
- skb = skb_clone(skb, GFP_ATOMIC);
- if (!skb)
- /* Skip the sample action when out of memory. */
- return 0;
-
- if (!add_deferred_actions(skb, key, a)) {
- if (net_ratelimit())
- pr_warn("%s: deferred actions limit reached, dropping sample action\n",
- ovs_dp_name(dp));
-
- kfree_skb(skb);
- }
- return 0;
+ clone_flow_key = !arg->exec;
+ return clone_execute(dp, skb, key, 0, actions, rem, last,
+ clone_flow_key);
}
static void execute_hash(struct sk_buff *skb, struct sw_flow_key *key,
@@ -1084,10 +1069,9 @@
static int execute_recirc(struct datapath *dp, struct sk_buff *skb,
struct sw_flow_key *key,
- const struct nlattr *a, int rem)
+ const struct nlattr *a, bool last)
{
- struct deferred_action *da;
- int level;
+ u32 recirc_id;
if (!is_flow_key_valid(key)) {
int err;
@@ -1098,43 +1082,8 @@
}
BUG_ON(!is_flow_key_valid(key));
- if (!nla_is_last(a, rem)) {
- /* Recirc action is the not the last action
- * of the action list, need to clone the skb.
- */
- skb = skb_clone(skb, GFP_ATOMIC);
-
- /* Skip the recirc action when out of memory, but
- * continue on with the rest of the action list.
- */
- if (!skb)
- return 0;
- }
-
- level = this_cpu_read(exec_actions_level);
- if (level <= OVS_DEFERRED_ACTION_THRESHOLD) {
- struct recirc_keys *rks = this_cpu_ptr(recirc_keys);
- struct sw_flow_key *recirc_key = &rks->key[level - 1];
-
- *recirc_key = *key;
- recirc_key->recirc_id = nla_get_u32(a);
- ovs_dp_process_packet(skb, recirc_key);
-
- return 0;
- }
-
- da = add_deferred_actions(skb, key, NULL);
- if (da) {
- da->pkt_key.recirc_id = nla_get_u32(a);
- } else {
- kfree_skb(skb);
-
- if (net_ratelimit())
- pr_warn("%s: deferred action limit reached, drop recirc action\n",
- ovs_dp_name(dp));
- }
-
- return 0;
+ recirc_id = nla_get_u32(a);
+ return clone_execute(dp, skb, key, recirc_id, NULL, 0, last, true);
}
/* Execute a list of actions against 'skb'. */
@@ -1206,9 +1155,11 @@
err = pop_vlan(skb, key);
break;
- case OVS_ACTION_ATTR_RECIRC:
- err = execute_recirc(dp, skb, key, a, rem);
- if (nla_is_last(a, rem)) {
+ case OVS_ACTION_ATTR_RECIRC: {
+ bool last = nla_is_last(a, rem);
+
+ err = execute_recirc(dp, skb, key, a, last);
+ if (last) {
/* If this is the last action, the skb has
* been consumed or freed.
* Return immediately.
@@ -1216,6 +1167,7 @@
return err;
}
break;
+ }
case OVS_ACTION_ATTR_SET:
err = execute_set_action(skb, key, nla_data(a));
@@ -1226,9 +1178,15 @@
err = execute_masked_set_action(skb, key, nla_data(a));
break;
- case OVS_ACTION_ATTR_SAMPLE:
- err = sample(dp, skb, key, a, attr, len);
+ case OVS_ACTION_ATTR_SAMPLE: {
+ bool last = nla_is_last(a, rem);
+
+ err = sample(dp, skb, key, a, last);
+ if (last)
+ return err;
+
break;
+ }
case OVS_ACTION_ATTR_CT:
if (!is_flow_key_valid(key)) {
@@ -1264,6 +1222,79 @@
return 0;
}
+/* Execute the actions on the clone of the packet. The effect of the
+ * execution does not affect the original 'skb' nor the original 'key'.
+ *
+ * The execution may be deferred in case the actions can not be executed
+ * immediately.
+ */
+static int clone_execute(struct datapath *dp, struct sk_buff *skb,
+ struct sw_flow_key *key, u32 recirc_id,
+ const struct nlattr *actions, int len,
+ bool last, bool clone_flow_key)
+{
+ struct deferred_action *da;
+ struct sw_flow_key *clone;
+
+ skb = last ? skb : skb_clone(skb, GFP_ATOMIC);
+ if (!skb) {
+ /* Out of memory, skip this action.
+ */
+ return 0;
+ }
+
+ /* When clone_flow_key is false, the 'key' will not be change
+ * by the actions, then the 'key' can be used directly.
+ * Otherwise, try to clone key from the next recursion level of
+ * 'flow_keys'. If clone is successful, execute the actions
+ * without deferring.
+ */
+ clone = clone_flow_key ? clone_key(key) : key;
+ if (clone) {
+ int err = 0;
+
+ if (actions) { /* Sample action */
+ if (clone_flow_key)
+ __this_cpu_inc(exec_actions_level);
+
+ err = do_execute_actions(dp, skb, clone,
+ actions, len);
+
+ if (clone_flow_key)
+ __this_cpu_dec(exec_actions_level);
+ } else { /* Recirc action */
+ clone->recirc_id = recirc_id;
+ ovs_dp_process_packet(skb, clone);
+ }
+ return err;
+ }
+
+ /* Out of 'flow_keys' space. Defer actions */
+ da = add_deferred_actions(skb, key, actions, len);
+ if (da) {
+ if (!actions) { /* Recirc action */
+ key = &da->pkt_key;
+ key->recirc_id = recirc_id;
+ }
+ } else {
+ /* Out of per CPU action FIFO space. Drop the 'skb' and
+ * log an error.
+ */
+ kfree_skb(skb);
+
+ if (net_ratelimit()) {
+ if (actions) { /* Sample action */
+ pr_warn("%s: deferred action limit reached, drop sample action\n",
+ ovs_dp_name(dp));
+ } else { /* Recirc action */
+ pr_warn("%s: deferred action limit reached, drop recirc action\n",
+ ovs_dp_name(dp));
+ }
+ }
+ }
+ return 0;
+}
+
static void process_deferred_actions(struct datapath *dp)
{
struct action_fifo *fifo = this_cpu_ptr(action_fifos);
@@ -1278,10 +1309,10 @@
struct sk_buff *skb = da->skb;
struct sw_flow_key *key = &da->pkt_key;
const struct nlattr *actions = da->actions;
+ int actions_len = da->actions_len;
if (actions)
- do_execute_actions(dp, skb, key, actions,
- nla_len(actions));
+ do_execute_actions(dp, skb, key, actions, actions_len);
else
ovs_dp_process_packet(skb, key);
} while (!action_fifo_is_empty(fifo));
@@ -1323,8 +1354,8 @@
if (!action_fifos)
return -ENOMEM;
- recirc_keys = alloc_percpu(struct recirc_keys);
- if (!recirc_keys) {
+ flow_keys = alloc_percpu(struct action_flow_keys);
+ if (!flow_keys) {
free_percpu(action_fifos);
return -ENOMEM;
}
@@ -1335,5 +1366,5 @@
void action_fifos_exit(void)
{
free_percpu(action_fifos);
- free_percpu(recirc_keys);
+ free_percpu(flow_keys);
}
diff --git a/net/openvswitch/conntrack.c b/net/openvswitch/conntrack.c
index c92a579..bf602e3 100644
--- a/net/openvswitch/conntrack.c
+++ b/net/openvswitch/conntrack.c
@@ -66,7 +66,9 @@
u8 commit : 1;
u8 nat : 3; /* enum ovs_ct_nat */
u8 force : 1;
+ u8 have_eventmask : 1;
u16 family;
+ u32 eventmask; /* Mask of 1 << IPCT_*. */
struct md_mark mark;
struct md_labels labels;
#ifdef CONFIG_NF_NAT_NEEDED
@@ -373,7 +375,7 @@
}
/* Labels are included in the IPCTNL_MSG_CT_NEW event only if the
- * IPCT_LABEL bit it set in the event cache.
+ * IPCT_LABEL bit is set in the event cache.
*/
nf_conntrack_event_cache(IPCT_LABEL, ct);
@@ -823,11 +825,6 @@
enum nf_nat_manip_type maniptype;
int err;
- if (nf_ct_is_untracked(ct)) {
- /* A NAT action may only be performed on tracked packets. */
- return NF_ACCEPT;
- }
-
/* Add NAT extension if not confirmed yet. */
if (!nf_ct_is_confirmed(ct) && !nf_ct_nat_ext_add(ct))
return NF_ACCEPT; /* Can't NAT. */
@@ -1035,6 +1032,20 @@
if (!ct)
return 0;
+ /* Set the conntrack event mask if given. NEW and DELETE events have
+ * their own groups, but the NFNLGRP_CONNTRACK_UPDATE group listener
+ * typically would receive many kinds of updates. Setting the event
+ * mask allows those events to be filtered. The set event mask will
+ * remain in effect for the lifetime of the connection unless changed
+ * by a further CT action with both the commit flag and the eventmask
+ * option. */
+ if (info->have_eventmask) {
+ struct nf_conntrack_ecache *cache = nf_ct_ecache_find(ct);
+
+ if (cache)
+ cache->ctmask = info->eventmask;
+ }
+
/* Apply changes before confirming the connection so that the initial
* conntrack NEW netlink event carries the values given in the CT
* action.
@@ -1266,6 +1277,8 @@
/* NAT length is checked when parsing the nested attributes. */
[OVS_CT_ATTR_NAT] = { .minlen = 0, .maxlen = INT_MAX },
#endif
+ [OVS_CT_ATTR_EVENTMASK] = { .minlen = sizeof(u32),
+ .maxlen = sizeof(u32) },
};
static int parse_ct(const struct nlattr *attr, struct ovs_conntrack_info *info,
@@ -1344,6 +1357,11 @@
break;
}
#endif
+ case OVS_CT_ATTR_EVENTMASK:
+ info->have_eventmask = true;
+ info->eventmask = nla_get_u32(a);
+ break;
+
default:
OVS_NLERR(log, "Unknown conntrack attr (%d)",
type);
@@ -1543,6 +1561,10 @@
ct_info->helper->name))
return -EMSGSIZE;
}
+ if (ct_info->have_eventmask &&
+ nla_put_u32(skb, OVS_CT_ATTR_EVENTMASK, ct_info->eventmask))
+ return -EMSGSIZE;
+
#ifdef CONFIG_NF_NAT_NEEDED
if (ct_info->nat && !ovs_ct_nat_to_attr(ct_info, skb))
return -EMSGSIZE;
diff --git a/net/openvswitch/datapath.c b/net/openvswitch/datapath.c
index 9c62b63..7b17da9 100644
--- a/net/openvswitch/datapath.c
+++ b/net/openvswitch/datapath.c
@@ -1353,7 +1353,7 @@
int err;
err = genlmsg_parse(cb->nlh, &dp_flow_genl_family, a,
- OVS_FLOW_ATTR_MAX, flow_policy);
+ OVS_FLOW_ATTR_MAX, flow_policy, NULL);
if (err)
return err;
ufid_flags = ovs_nla_get_ufid_flags(a[OVS_FLOW_ATTR_UFID_FLAGS]);
diff --git a/net/openvswitch/datapath.h b/net/openvswitch/datapath.h
index 1c6e937..da931bd 100644
--- a/net/openvswitch/datapath.h
+++ b/net/openvswitch/datapath.h
@@ -34,8 +34,6 @@
#define DP_MAX_PORTS USHRT_MAX
#define DP_VPORT_HASH_BUCKETS 1024
-#define SAMPLE_ACTION_DEPTH 3
-
/**
* struct dp_stats_percpu - per-cpu packet processing statistics for a given
* datapath.
diff --git a/net/openvswitch/flow_netlink.c b/net/openvswitch/flow_netlink.c
index 1105a83..7e1d8a2 100644
--- a/net/openvswitch/flow_netlink.c
+++ b/net/openvswitch/flow_netlink.c
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2007-2014 Nicira, Inc.
+ * Copyright (c) 2007-2017 Nicira, Inc.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of version 2 of the GNU General Public
@@ -59,6 +59,39 @@
#define OVS_ATTR_NESTED -1
#define OVS_ATTR_VARIABLE -2
+static bool actions_may_change_flow(const struct nlattr *actions)
+{
+ struct nlattr *nla;
+ int rem;
+
+ nla_for_each_nested(nla, actions, rem) {
+ u16 action = nla_type(nla);
+
+ switch (action) {
+ case OVS_ACTION_ATTR_OUTPUT:
+ case OVS_ACTION_ATTR_RECIRC:
+ case OVS_ACTION_ATTR_TRUNC:
+ case OVS_ACTION_ATTR_USERSPACE:
+ break;
+
+ case OVS_ACTION_ATTR_CT:
+ case OVS_ACTION_ATTR_HASH:
+ case OVS_ACTION_ATTR_POP_ETH:
+ case OVS_ACTION_ATTR_POP_MPLS:
+ case OVS_ACTION_ATTR_POP_VLAN:
+ case OVS_ACTION_ATTR_PUSH_ETH:
+ case OVS_ACTION_ATTR_PUSH_MPLS:
+ case OVS_ACTION_ATTR_PUSH_VLAN:
+ case OVS_ACTION_ATTR_SAMPLE:
+ case OVS_ACTION_ATTR_SET:
+ case OVS_ACTION_ATTR_SET_MASKED:
+ default:
+ return true;
+ }
+ }
+ return false;
+}
+
static void update_range(struct sw_flow_match *match,
size_t offset, size_t size, bool is_mask)
{
@@ -2023,18 +2056,20 @@
static int __ovs_nla_copy_actions(struct net *net, const struct nlattr *attr,
const struct sw_flow_key *key,
- int depth, struct sw_flow_actions **sfa,
+ struct sw_flow_actions **sfa,
__be16 eth_type, __be16 vlan_tci, bool log);
static int validate_and_copy_sample(struct net *net, const struct nlattr *attr,
- const struct sw_flow_key *key, int depth,
+ const struct sw_flow_key *key,
struct sw_flow_actions **sfa,
- __be16 eth_type, __be16 vlan_tci, bool log)
+ __be16 eth_type, __be16 vlan_tci,
+ bool log, bool last)
{
const struct nlattr *attrs[OVS_SAMPLE_ATTR_MAX + 1];
const struct nlattr *probability, *actions;
const struct nlattr *a;
- int rem, start, err, st_acts;
+ int rem, start, err;
+ struct sample_arg arg;
memset(attrs, 0, sizeof(attrs));
nla_for_each_nested(a, attr, rem) {
@@ -2058,20 +2093,32 @@
start = add_nested_action_start(sfa, OVS_ACTION_ATTR_SAMPLE, log);
if (start < 0)
return start;
- err = ovs_nla_add_action(sfa, OVS_SAMPLE_ATTR_PROBABILITY,
- nla_data(probability), sizeof(u32), log);
+
+ /* When both skb and flow may be changed, put the sample
+ * into a deferred fifo. On the other hand, if only skb
+ * may be modified, the actions can be executed in place.
+ *
+ * Do this analysis at the flow installation time.
+ * Set 'clone_action->exec' to true if the actions can be
+ * executed without being deferred.
+ *
+ * If the sample is the last action, it can always be excuted
+ * rather than deferred.
+ */
+ arg.exec = last || !actions_may_change_flow(actions);
+ arg.probability = nla_get_u32(probability);
+
+ err = ovs_nla_add_action(sfa, OVS_SAMPLE_ATTR_ARG, &arg, sizeof(arg),
+ log);
if (err)
return err;
- st_acts = add_nested_action_start(sfa, OVS_SAMPLE_ATTR_ACTIONS, log);
- if (st_acts < 0)
- return st_acts;
- err = __ovs_nla_copy_actions(net, actions, key, depth + 1, sfa,
+ err = __ovs_nla_copy_actions(net, actions, key, sfa,
eth_type, vlan_tci, log);
+
if (err)
return err;
- add_nested_action_end(*sfa, st_acts);
add_nested_action_end(*sfa, start);
return 0;
@@ -2380,8 +2427,8 @@
struct nlattr *a[OVS_USERSPACE_ATTR_MAX + 1];
int error;
- error = nla_parse_nested(a, OVS_USERSPACE_ATTR_MAX,
- attr, userspace_policy);
+ error = nla_parse_nested(a, OVS_USERSPACE_ATTR_MAX, attr,
+ userspace_policy, NULL);
if (error)
return error;
@@ -2408,16 +2455,13 @@
static int __ovs_nla_copy_actions(struct net *net, const struct nlattr *attr,
const struct sw_flow_key *key,
- int depth, struct sw_flow_actions **sfa,
+ struct sw_flow_actions **sfa,
__be16 eth_type, __be16 vlan_tci, bool log)
{
u8 mac_proto = ovs_key_mac_proto(key);
const struct nlattr *a;
int rem, err;
- if (depth >= SAMPLE_ACTION_DEPTH)
- return -EOVERFLOW;
-
nla_for_each_nested(a, attr, rem) {
/* Expected argument lengths, (u32)-1 for variable length. */
static const u32 action_lens[OVS_ACTION_ATTR_MAX + 1] = {
@@ -2555,13 +2599,17 @@
return err;
break;
- case OVS_ACTION_ATTR_SAMPLE:
- err = validate_and_copy_sample(net, a, key, depth, sfa,
- eth_type, vlan_tci, log);
+ case OVS_ACTION_ATTR_SAMPLE: {
+ bool last = nla_is_last(a, rem);
+
+ err = validate_and_copy_sample(net, a, key, sfa,
+ eth_type, vlan_tci,
+ log, last);
if (err)
return err;
skip_copy = true;
break;
+ }
case OVS_ACTION_ATTR_CT:
err = ovs_ct_copy_action(net, a, key, sfa, log);
@@ -2615,7 +2663,7 @@
return PTR_ERR(*sfa);
(*sfa)->orig_len = nla_len(attr);
- err = __ovs_nla_copy_actions(net, attr, key, 0, sfa, key->eth.type,
+ err = __ovs_nla_copy_actions(net, attr, key, sfa, key->eth.type,
key->eth.vlan.tci, log);
if (err)
ovs_nla_free_flow_actions(*sfa);
@@ -2623,39 +2671,44 @@
return err;
}
-static int sample_action_to_attr(const struct nlattr *attr, struct sk_buff *skb)
+static int sample_action_to_attr(const struct nlattr *attr,
+ struct sk_buff *skb)
{
- const struct nlattr *a;
- struct nlattr *start;
- int err = 0, rem;
+ struct nlattr *start, *ac_start = NULL, *sample_arg;
+ int err = 0, rem = nla_len(attr);
+ const struct sample_arg *arg;
+ struct nlattr *actions;
start = nla_nest_start(skb, OVS_ACTION_ATTR_SAMPLE);
if (!start)
return -EMSGSIZE;
- nla_for_each_nested(a, attr, rem) {
- int type = nla_type(a);
- struct nlattr *st_sample;
+ sample_arg = nla_data(attr);
+ arg = nla_data(sample_arg);
+ actions = nla_next(sample_arg, &rem);
- switch (type) {
- case OVS_SAMPLE_ATTR_PROBABILITY:
- if (nla_put(skb, OVS_SAMPLE_ATTR_PROBABILITY,
- sizeof(u32), nla_data(a)))
- return -EMSGSIZE;
- break;
- case OVS_SAMPLE_ATTR_ACTIONS:
- st_sample = nla_nest_start(skb, OVS_SAMPLE_ATTR_ACTIONS);
- if (!st_sample)
- return -EMSGSIZE;
- err = ovs_nla_put_actions(nla_data(a), nla_len(a), skb);
- if (err)
- return err;
- nla_nest_end(skb, st_sample);
- break;
- }
+ if (nla_put_u32(skb, OVS_SAMPLE_ATTR_PROBABILITY, arg->probability)) {
+ err = -EMSGSIZE;
+ goto out;
}
- nla_nest_end(skb, start);
+ ac_start = nla_nest_start(skb, OVS_SAMPLE_ATTR_ACTIONS);
+ if (!ac_start) {
+ err = -EMSGSIZE;
+ goto out;
+ }
+
+ err = ovs_nla_put_actions(actions, rem, skb);
+
+out:
+ if (err) {
+ nla_nest_cancel(skb, ac_start);
+ nla_nest_cancel(skb, start);
+ } else {
+ nla_nest_end(skb, ac_start);
+ nla_nest_end(skb, start);
+ }
+
return err;
}
diff --git a/net/openvswitch/vport-vxlan.c b/net/openvswitch/vport-vxlan.c
index 7eb955e..869acb3 100644
--- a/net/openvswitch/vport-vxlan.c
+++ b/net/openvswitch/vport-vxlan.c
@@ -70,7 +70,8 @@
if (nla_len(attr) < sizeof(struct nlattr))
return -EINVAL;
- err = nla_parse_nested(exts, OVS_VXLAN_EXT_MAX, attr, exts_policy);
+ err = nla_parse_nested(exts, OVS_VXLAN_EXT_MAX, attr, exts_policy,
+ NULL);
if (err < 0)
return err;
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 8489bef..f400176 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -1496,6 +1496,7 @@
DEFINE_MUTEX(fanout_mutex);
EXPORT_SYMBOL_GPL(fanout_mutex);
static LIST_HEAD(fanout_list);
+static u16 fanout_next_id;
static void __fanout_link(struct sock *sk, struct packet_sock *po)
{
@@ -1629,6 +1630,36 @@
};
}
+static bool __fanout_id_is_free(struct sock *sk, u16 candidate_id)
+{
+ struct packet_fanout *f;
+
+ list_for_each_entry(f, &fanout_list, list) {
+ if (f->id == candidate_id &&
+ read_pnet(&f->net) == sock_net(sk)) {
+ return false;
+ }
+ }
+ return true;
+}
+
+static bool fanout_find_new_id(struct sock *sk, u16 *new_id)
+{
+ u16 id = fanout_next_id;
+
+ do {
+ if (__fanout_id_is_free(sk, id)) {
+ *new_id = id;
+ fanout_next_id = id + 1;
+ return true;
+ }
+
+ id++;
+ } while (id != fanout_next_id);
+
+ return false;
+}
+
static int fanout_add(struct sock *sk, u16 id, u16 type_flags)
{
struct packet_rollover *rollover = NULL;
@@ -1676,6 +1707,19 @@
po->rollover = rollover;
}
+ if (type_flags & PACKET_FANOUT_FLAG_UNIQUEID) {
+ if (id != 0) {
+ err = -EINVAL;
+ goto out;
+ }
+ if (!fanout_find_new_id(sk, &id)) {
+ err = -ENOMEM;
+ goto out;
+ }
+ /* ephemeral flag for the first socket in the group: drop it */
+ flags &= ~(PACKET_FANOUT_FLAG_UNIQUEID >> 8);
+ }
+
match = NULL;
list_for_each_entry(f, &fanout_list, list) {
if (f->id == id &&
@@ -3836,6 +3880,8 @@
case PACKET_HDRLEN:
if (len > sizeof(int))
len = sizeof(int);
+ if (len < sizeof(int))
+ return -EINVAL;
if (copy_from_user(&val, optval, len))
return -EFAULT;
switch (val) {
diff --git a/net/phonet/pn_netlink.c b/net/phonet/pn_netlink.c
index bc5ee5f..45b3af3 100644
--- a/net/phonet/pn_netlink.c
+++ b/net/phonet/pn_netlink.c
@@ -61,7 +61,8 @@
[IFA_LOCAL] = { .type = NLA_U8 },
};
-static int addr_doit(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int addr_doit(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct nlattr *tb[IFA_MAX+1];
@@ -78,7 +79,8 @@
ASSERT_RTNL();
- err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_phonet_policy);
+ err = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, ifa_phonet_policy,
+ extack);
if (err < 0)
return err;
@@ -226,7 +228,8 @@
[RTA_OIF] = { .type = NLA_U32 },
};
-static int route_doit(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int route_doit(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct nlattr *tb[RTA_MAX+1];
@@ -243,7 +246,8 @@
ASSERT_RTNL();
- err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_phonet_policy);
+ err = nlmsg_parse(nlh, sizeof(*rtm), tb, RTA_MAX, rtm_phonet_policy,
+ extack);
if (err < 0)
return err;
diff --git a/net/qrtr/Kconfig b/net/qrtr/Kconfig
index b83c680..326fd97 100644
--- a/net/qrtr/Kconfig
+++ b/net/qrtr/Kconfig
@@ -16,7 +16,7 @@
config QRTR_SMD
tristate "SMD IPC Router channels"
- depends on QCOM_SMD || (COMPILE_TEST && QCOM_SMD=n)
+ depends on RPMSG || (COMPILE_TEST && RPMSG=n)
---help---
Say Y here to support SMD based ipcrouter channels. SMD is the
most common transport for IPC Router.
diff --git a/net/qrtr/qrtr.c b/net/qrtr/qrtr.c
index 9da7368..a9a8c7d 100644
--- a/net/qrtr/qrtr.c
+++ b/net/qrtr/qrtr.c
@@ -945,7 +945,8 @@
[IFA_LOCAL] = { .type = NLA_U32 },
};
-static int qrtr_addr_doit(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int qrtr_addr_doit(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct nlattr *tb[IFA_MAX + 1];
struct ifaddrmsg *ifm;
@@ -959,7 +960,7 @@
ASSERT_RTNL();
- rc = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, qrtr_policy);
+ rc = nlmsg_parse(nlh, sizeof(*ifm), tb, IFA_MAX, qrtr_policy, extack);
if (rc < 0)
return rc;
diff --git a/net/qrtr/smd.c b/net/qrtr/smd.c
index 0d11132..50615d5 100644
--- a/net/qrtr/smd.c
+++ b/net/qrtr/smd.c
@@ -14,21 +14,21 @@
#include <linux/module.h>
#include <linux/skbuff.h>
-#include <linux/soc/qcom/smd.h>
+#include <linux/rpmsg.h>
#include "qrtr.h"
struct qrtr_smd_dev {
struct qrtr_endpoint ep;
- struct qcom_smd_channel *channel;
+ struct rpmsg_endpoint *channel;
struct device *dev;
};
/* from smd to qrtr */
-static int qcom_smd_qrtr_callback(struct qcom_smd_channel *channel,
- const void *data, size_t len)
+static int qcom_smd_qrtr_callback(struct rpmsg_device *rpdev,
+ void *data, int len, void *priv, u32 addr)
{
- struct qrtr_smd_dev *qdev = qcom_smd_get_drvdata(channel);
+ struct qrtr_smd_dev *qdev = dev_get_drvdata(&rpdev->dev);
int rc;
if (!qdev)
@@ -54,7 +54,7 @@
if (rc)
goto out;
- rc = qcom_smd_send(qdev->channel, skb->data, skb->len);
+ rc = rpmsg_send(qdev->channel, skb->data, skb->len);
out:
if (rc)
@@ -64,57 +64,55 @@
return rc;
}
-static int qcom_smd_qrtr_probe(struct qcom_smd_device *sdev)
+static int qcom_smd_qrtr_probe(struct rpmsg_device *rpdev)
{
struct qrtr_smd_dev *qdev;
int rc;
- qdev = devm_kzalloc(&sdev->dev, sizeof(*qdev), GFP_KERNEL);
+ qdev = devm_kzalloc(&rpdev->dev, sizeof(*qdev), GFP_KERNEL);
if (!qdev)
return -ENOMEM;
- qdev->channel = sdev->channel;
- qdev->dev = &sdev->dev;
+ qdev->channel = rpdev->ept;
+ qdev->dev = &rpdev->dev;
qdev->ep.xmit = qcom_smd_qrtr_send;
rc = qrtr_endpoint_register(&qdev->ep, QRTR_EP_NID_AUTO);
if (rc)
return rc;
- qcom_smd_set_drvdata(sdev->channel, qdev);
- dev_set_drvdata(&sdev->dev, qdev);
+ dev_set_drvdata(&rpdev->dev, qdev);
- dev_dbg(&sdev->dev, "Qualcomm SMD QRTR driver probed\n");
+ dev_dbg(&rpdev->dev, "Qualcomm SMD QRTR driver probed\n");
return 0;
}
-static void qcom_smd_qrtr_remove(struct qcom_smd_device *sdev)
+static void qcom_smd_qrtr_remove(struct rpmsg_device *rpdev)
{
- struct qrtr_smd_dev *qdev = dev_get_drvdata(&sdev->dev);
+ struct qrtr_smd_dev *qdev = dev_get_drvdata(&rpdev->dev);
qrtr_endpoint_unregister(&qdev->ep);
- dev_set_drvdata(&sdev->dev, NULL);
+ dev_set_drvdata(&rpdev->dev, NULL);
}
-static const struct qcom_smd_id qcom_smd_qrtr_smd_match[] = {
+static const struct rpmsg_device_id qcom_smd_qrtr_smd_match[] = {
{ "IPCRTR" },
{}
};
-static struct qcom_smd_driver qcom_smd_qrtr_driver = {
+static struct rpmsg_driver qcom_smd_qrtr_driver = {
.probe = qcom_smd_qrtr_probe,
.remove = qcom_smd_qrtr_remove,
.callback = qcom_smd_qrtr_callback,
- .smd_match_table = qcom_smd_qrtr_smd_match,
- .driver = {
+ .id_table = qcom_smd_qrtr_smd_match,
+ .drv = {
.name = "qcom_smd_qrtr",
- .owner = THIS_MODULE,
},
};
-module_qcom_smd_driver(qcom_smd_qrtr_driver);
+module_rpmsg_driver(qcom_smd_qrtr_driver);
MODULE_DESCRIPTION("Qualcomm IPC-Router SMD interface driver");
MODULE_LICENSE("GPL v2");
diff --git a/net/rds/connection.c b/net/rds/connection.c
index 1fa75ab..6a5ebde 100644
--- a/net/rds/connection.c
+++ b/net/rds/connection.c
@@ -333,11 +333,19 @@
rds_conn_path_reset(cp);
if (!rds_conn_path_transition(cp, RDS_CONN_DISCONNECTING,
+ RDS_CONN_DOWN) &&
+ !rds_conn_path_transition(cp, RDS_CONN_ERROR,
RDS_CONN_DOWN)) {
/* This can happen - eg when we're in the middle of tearing
* down the connection, and someone unloads the rds module.
- * Quite reproduceable with loopback connections.
+ * Quite reproducible with loopback connections.
* Mostly harmless.
+ *
+ * Note that this also happens with rds-tcp because
+ * we could have triggered rds_conn_path_drop in irq
+ * mode from rds_tcp_state change on the receipt of
+ * a FIN, thus we need to recheck for RDS_CONN_ERROR
+ * here.
*/
rds_conn_path_error(cp, "%s: failed to transition "
"to state DOWN, current state "
diff --git a/net/rds/ib_cm.c b/net/rds/ib_cm.c
index 1c38d2c..80fb6f6 100644
--- a/net/rds/ib_cm.c
+++ b/net/rds/ib_cm.c
@@ -702,9 +702,8 @@
event->param.conn.initiator_depth);
/* rdma_accept() calls rdma_reject() internally if it fails */
- err = rdma_accept(cm_id, &conn_param);
- if (err)
- rds_ib_conn_error(conn, "rdma_accept failed (%d)\n", err);
+ if (rdma_accept(cm_id, &conn_param))
+ rds_ib_conn_error(conn, "rdma_accept failed\n");
out:
if (conn)
diff --git a/net/rds/ib_fmr.c b/net/rds/ib_fmr.c
index 4fe8f4f..86ef907 100644
--- a/net/rds/ib_fmr.c
+++ b/net/rds/ib_fmr.c
@@ -78,17 +78,15 @@
return ibmr;
out_no_cigar:
- if (ibmr) {
- if (fmr->fmr)
- ib_dealloc_fmr(fmr->fmr);
- kfree(ibmr);
- }
+ kfree(ibmr);
atomic_dec(&pool->item_count);
+
return ERR_PTR(err);
}
-int rds_ib_map_fmr(struct rds_ib_device *rds_ibdev, struct rds_ib_mr *ibmr,
- struct scatterlist *sg, unsigned int nents)
+static int rds_ib_map_fmr(struct rds_ib_device *rds_ibdev,
+ struct rds_ib_mr *ibmr, struct scatterlist *sg,
+ unsigned int nents)
{
struct ib_device *dev = rds_ibdev->dev;
struct rds_ib_fmr *fmr = &ibmr->u.fmr;
@@ -114,29 +112,39 @@
u64 dma_addr = ib_sg_dma_address(dev, &scat[i]);
if (dma_addr & ~PAGE_MASK) {
- if (i > 0)
+ if (i > 0) {
+ ib_dma_unmap_sg(dev, sg, nents,
+ DMA_BIDIRECTIONAL);
return -EINVAL;
- else
+ } else {
++page_cnt;
+ }
}
if ((dma_addr + dma_len) & ~PAGE_MASK) {
- if (i < sg_dma_len - 1)
+ if (i < sg_dma_len - 1) {
+ ib_dma_unmap_sg(dev, sg, nents,
+ DMA_BIDIRECTIONAL);
return -EINVAL;
- else
+ } else {
++page_cnt;
+ }
}
len += dma_len;
}
page_cnt += len >> PAGE_SHIFT;
- if (page_cnt > ibmr->pool->fmr_attr.max_pages)
+ if (page_cnt > ibmr->pool->fmr_attr.max_pages) {
+ ib_dma_unmap_sg(dev, sg, nents, DMA_BIDIRECTIONAL);
return -EINVAL;
+ }
dma_pages = kmalloc_node(sizeof(u64) * page_cnt, GFP_ATOMIC,
rdsibdev_to_node(rds_ibdev));
- if (!dma_pages)
+ if (!dma_pages) {
+ ib_dma_unmap_sg(dev, sg, nents, DMA_BIDIRECTIONAL);
return -ENOMEM;
+ }
page_cnt = 0;
for (i = 0; i < sg_dma_len; ++i) {
@@ -149,8 +157,10 @@
}
ret = ib_map_phys_fmr(fmr->fmr, dma_pages, page_cnt, io_addr);
- if (ret)
+ if (ret) {
+ ib_dma_unmap_sg(dev, sg, nents, DMA_BIDIRECTIONAL);
goto out;
+ }
/* Success - we successfully remapped the MR, so we can
* safely tear down the old mapping.
diff --git a/net/rds/ib_mr.h b/net/rds/ib_mr.h
index 5d6e98a..0ea4ab0 100644
--- a/net/rds/ib_mr.h
+++ b/net/rds/ib_mr.h
@@ -125,8 +125,6 @@
void __rds_ib_teardown_mr(struct rds_ib_mr *);
void rds_ib_teardown_mr(struct rds_ib_mr *);
struct rds_ib_mr *rds_ib_alloc_fmr(struct rds_ib_device *, int);
-int rds_ib_map_fmr(struct rds_ib_device *, struct rds_ib_mr *,
- struct scatterlist *, unsigned int);
struct rds_ib_mr *rds_ib_reuse_mr(struct rds_ib_mr_pool *);
int rds_ib_flush_mr_pool(struct rds_ib_mr_pool *, int, struct rds_ib_mr **);
struct rds_ib_mr *rds_ib_reg_fmr(struct rds_ib_device *, struct scatterlist *,
diff --git a/net/rds/recv.c b/net/rds/recv.c
index 8b7e7b7..c70c32c 100644
--- a/net/rds/recv.c
+++ b/net/rds/recv.c
@@ -594,7 +594,6 @@
goto out;
while (1) {
- struct iov_iter save;
/* If there are pending notifications, do those - and nothing else */
if (!list_empty(&rs->rs_notify_queue)) {
ret = rds_notify_queue_get(rs, msg);
@@ -630,7 +629,6 @@
rdsdebug("copying inc %p from %pI4:%u to user\n", inc,
&inc->i_conn->c_faddr,
ntohs(inc->i_hdr.h_sport));
- save = msg->msg_iter;
ret = inc->i_conn->c_trans->inc_copy_to_user(inc, &msg->msg_iter);
if (ret < 0)
break;
@@ -644,7 +642,7 @@
rds_inc_put(inc);
inc = NULL;
rds_stats_inc(s_recv_deliver_raced);
- msg->msg_iter = save;
+ iov_iter_revert(&msg->msg_iter, ret);
continue;
}
diff --git a/net/rds/tcp.c b/net/rds/tcp.c
index 2256900..431404d 100644
--- a/net/rds/tcp.c
+++ b/net/rds/tcp.c
@@ -84,13 +84,10 @@
/* doing it this way avoids calling tcp_sk() */
void rds_tcp_nonagle(struct socket *sock)
{
- mm_segment_t oldfs = get_fs();
int val = 1;
- set_fs(KERNEL_DS);
- sock->ops->setsockopt(sock, SOL_TCP, TCP_NODELAY, (char __user *)&val,
+ kernel_setsockopt(sock, SOL_TCP, TCP_NODELAY, (void *)&val,
sizeof(val));
- set_fs(oldfs);
}
u32 rds_tcp_snd_nxt(struct rds_tcp_connection *tc)
diff --git a/net/rds/tcp_send.c b/net/rds/tcp_send.c
index dcf4742..52d11d7 100644
--- a/net/rds/tcp_send.c
+++ b/net/rds/tcp_send.c
@@ -40,13 +40,7 @@
static void rds_tcp_cork(struct socket *sock, int val)
{
- mm_segment_t oldfs;
-
- oldfs = get_fs();
- set_fs(KERNEL_DS);
- sock->ops->setsockopt(sock, SOL_TCP, TCP_CORK, (char __user *)&val,
- sizeof(val));
- set_fs(oldfs);
+ kernel_setsockopt(sock, SOL_TCP, TCP_CORK, (void *)&val, sizeof(val));
}
void rds_tcp_xmit_path_prepare(struct rds_conn_path *cp)
diff --git a/net/rds/threads.c b/net/rds/threads.c
index e36e333..3e447d05 100644
--- a/net/rds/threads.c
+++ b/net/rds/threads.c
@@ -156,7 +156,7 @@
struct rds_connection *conn = cp->cp_conn;
int ret;
- if (cp->cp_index > 1 && cp->cp_conn->c_laddr > cp->cp_conn->c_faddr)
+ if (cp->cp_index > 0 && cp->cp_conn->c_laddr > cp->cp_conn->c_faddr)
return;
clear_bit(RDS_RECONNECT_PENDING, &cp->cp_flags);
ret = rds_conn_path_transition(cp, RDS_CONN_DOWN, RDS_CONN_CONNECTING);
diff --git a/net/rxrpc/ar-internal.h b/net/rxrpc/ar-internal.h
index 26a7b1d..7486926 100644
--- a/net/rxrpc/ar-internal.h
+++ b/net/rxrpc/ar-internal.h
@@ -740,6 +740,25 @@
}
/*
+ * Abort a call due to a protocol error.
+ */
+static inline bool __rxrpc_abort_eproto(struct rxrpc_call *call,
+ struct sk_buff *skb,
+ const char *eproto_why,
+ const char *why,
+ u32 abort_code)
+{
+ struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
+
+ trace_rxrpc_rx_eproto(call, sp->hdr.serial, eproto_why);
+ return rxrpc_abort_call(why, call, sp->hdr.seq, abort_code, -EPROTO);
+}
+
+#define rxrpc_abort_eproto(call, skb, eproto_why, abort_why, abort_code) \
+ __rxrpc_abort_eproto((call), (skb), tracepoint_string(eproto_why), \
+ (abort_why), (abort_code))
+
+/*
* conn_client.c
*/
extern unsigned int rxrpc_max_client_connections;
diff --git a/net/rxrpc/call_accept.c b/net/rxrpc/call_accept.c
index 0ed181f..1752fcf 100644
--- a/net/rxrpc/call_accept.c
+++ b/net/rxrpc/call_accept.c
@@ -413,11 +413,11 @@
case RXRPC_CONN_REMOTELY_ABORTED:
rxrpc_set_call_completion(call, RXRPC_CALL_REMOTELY_ABORTED,
- conn->remote_abort, ECONNABORTED);
+ conn->remote_abort, -ECONNABORTED);
break;
case RXRPC_CONN_LOCALLY_ABORTED:
rxrpc_abort_call("CON", call, sp->hdr.seq,
- conn->local_abort, ECONNABORTED);
+ conn->local_abort, -ECONNABORTED);
break;
default:
BUG();
@@ -600,7 +600,7 @@
write_lock_bh(&call->state_lock);
switch (call->state) {
case RXRPC_CALL_SERVER_ACCEPTING:
- __rxrpc_abort_call("REJ", call, 1, RX_USER_ABORT, ECONNABORTED);
+ __rxrpc_abort_call("REJ", call, 1, RX_USER_ABORT, -ECONNABORTED);
abort = true;
/* fall through */
case RXRPC_CALL_COMPLETE:
diff --git a/net/rxrpc/call_event.c b/net/rxrpc/call_event.c
index 97a17ad..7a77844 100644
--- a/net/rxrpc/call_event.c
+++ b/net/rxrpc/call_event.c
@@ -386,7 +386,7 @@
now = ktime_get_real();
if (ktime_before(call->expire_at, now)) {
- rxrpc_abort_call("EXP", call, 0, RX_CALL_TIMEOUT, ETIME);
+ rxrpc_abort_call("EXP", call, 0, RX_CALL_TIMEOUT, -ETIME);
set_bit(RXRPC_CALL_EV_ABORT, &call->events);
goto recheck_state;
}
diff --git a/net/rxrpc/call_object.c b/net/rxrpc/call_object.c
index d79cd36..47f7f42 100644
--- a/net/rxrpc/call_object.c
+++ b/net/rxrpc/call_object.c
@@ -486,7 +486,7 @@
call = list_entry(rx->to_be_accepted.next,
struct rxrpc_call, accept_link);
list_del(&call->accept_link);
- rxrpc_abort_call("SKR", call, 0, RX_CALL_DEAD, ECONNRESET);
+ rxrpc_abort_call("SKR", call, 0, RX_CALL_DEAD, -ECONNRESET);
rxrpc_put_call(call, rxrpc_call_put);
}
@@ -494,7 +494,7 @@
call = list_entry(rx->sock_calls.next,
struct rxrpc_call, sock_link);
rxrpc_get_call(call, rxrpc_call_got);
- rxrpc_abort_call("SKT", call, 0, RX_CALL_DEAD, ECONNRESET);
+ rxrpc_abort_call("SKT", call, 0, RX_CALL_DEAD, -ECONNRESET);
rxrpc_send_abort_packet(call);
rxrpc_release_call(rx, call);
rxrpc_put_call(call, rxrpc_call_put);
diff --git a/net/rxrpc/conn_client.c b/net/rxrpc/conn_client.c
index c3be03e8..e8dea0d 100644
--- a/net/rxrpc/conn_client.c
+++ b/net/rxrpc/conn_client.c
@@ -550,6 +550,7 @@
call->cid = conn->proto.cid | channel;
call->call_id = call_id;
+ trace_rxrpc_connect_call(call);
_net("CONNECT call %08x:%08x as call %d on conn %d",
call->cid, call->call_id, call->debug_id, conn->debug_id);
diff --git a/net/rxrpc/conn_event.c b/net/rxrpc/conn_event.c
index b099b64..46babcf 100644
--- a/net/rxrpc/conn_event.c
+++ b/net/rxrpc/conn_event.c
@@ -168,7 +168,7 @@
* generate a connection-level abort
*/
static int rxrpc_abort_connection(struct rxrpc_connection *conn,
- u32 error, u32 abort_code)
+ int error, u32 abort_code)
{
struct rxrpc_wire_header whdr;
struct msghdr msg;
@@ -281,14 +281,17 @@
case RXRPC_PACKET_TYPE_ABORT:
if (skb_copy_bits(skb, sizeof(struct rxrpc_wire_header),
- &wtmp, sizeof(wtmp)) < 0)
+ &wtmp, sizeof(wtmp)) < 0) {
+ trace_rxrpc_rx_eproto(NULL, sp->hdr.serial,
+ tracepoint_string("bad_abort"));
return -EPROTO;
+ }
abort_code = ntohl(wtmp);
_proto("Rx ABORT %%%u { ac=%d }", sp->hdr.serial, abort_code);
conn->state = RXRPC_CONN_REMOTELY_ABORTED;
rxrpc_abort_calls(conn, RXRPC_CALL_REMOTELY_ABORTED,
- abort_code, ECONNABORTED);
+ abort_code, -ECONNABORTED);
return -ECONNABORTED;
case RXRPC_PACKET_TYPE_CHALLENGE:
@@ -327,7 +330,8 @@
return 0;
default:
- _leave(" = -EPROTO [%u]", sp->hdr.type);
+ trace_rxrpc_rx_eproto(NULL, sp->hdr.serial,
+ tracepoint_string("bad_conn_pkt"));
return -EPROTO;
}
}
@@ -370,7 +374,7 @@
abort:
_debug("abort %d, %d", ret, abort_code);
- rxrpc_abort_connection(conn, -ret, abort_code);
+ rxrpc_abort_connection(conn, ret, abort_code);
_leave(" [aborted]");
}
@@ -419,9 +423,8 @@
goto out;
protocol_error:
- if (rxrpc_abort_connection(conn, -ret, abort_code) < 0)
+ if (rxrpc_abort_connection(conn, ret, abort_code) < 0)
goto requeue_and_leave;
rxrpc_free_skb(skb, rxrpc_skb_rx_freed);
- _leave(" [EPROTO]");
goto out;
}
diff --git a/net/rxrpc/input.c b/net/rxrpc/input.c
index 18b2ad8..45dba73 100644
--- a/net/rxrpc/input.c
+++ b/net/rxrpc/input.c
@@ -30,7 +30,7 @@
static void rxrpc_proto_abort(const char *why,
struct rxrpc_call *call, rxrpc_seq_t seq)
{
- if (rxrpc_abort_call(why, call, seq, RX_PROTOCOL_ERROR, EBADMSG)) {
+ if (rxrpc_abort_call(why, call, seq, RX_PROTOCOL_ERROR, -EBADMSG)) {
set_bit(RXRPC_CALL_EV_ABORT, &call->events);
rxrpc_queue_call(call);
}
@@ -665,6 +665,8 @@
rwind = RXRPC_RXTX_BUFF_SIZE - 1;
if (rwind > call->tx_winsize)
wake = true;
+ trace_rxrpc_rx_rwind_change(call, sp->hdr.serial,
+ ntohl(ackinfo->rwind), wake);
call->tx_winsize = rwind;
}
@@ -877,7 +879,7 @@
}
/*
- * Process an ABORT packet.
+ * Process an ABORT packet directed at a call.
*/
static void rxrpc_input_abort(struct rxrpc_call *call, struct sk_buff *skb)
{
@@ -892,10 +894,12 @@
&wtmp, sizeof(wtmp)) >= 0)
abort_code = ntohl(wtmp);
+ trace_rxrpc_rx_abort(call, sp->hdr.serial, abort_code);
+
_proto("Rx ABORT %%%u { %x }", sp->hdr.serial, abort_code);
if (rxrpc_set_call_completion(call, RXRPC_CALL_REMOTELY_ABORTED,
- abort_code, ECONNABORTED))
+ abort_code, -ECONNABORTED))
rxrpc_notify_socket(call);
}
@@ -958,7 +962,7 @@
case RXRPC_CALL_COMPLETE:
break;
default:
- if (rxrpc_abort_call("IMP", call, 0, RX_CALL_DEAD, ESHUTDOWN)) {
+ if (rxrpc_abort_call("IMP", call, 0, RX_CALL_DEAD, -ESHUTDOWN)) {
set_bit(RXRPC_CALL_EV_ABORT, &call->events);
rxrpc_queue_call(call);
}
@@ -1017,8 +1021,11 @@
struct rxrpc_wire_header whdr;
/* dig out the RxRPC connection details */
- if (skb_copy_bits(skb, 0, &whdr, sizeof(whdr)) < 0)
+ if (skb_copy_bits(skb, 0, &whdr, sizeof(whdr)) < 0) {
+ trace_rxrpc_rx_eproto(NULL, sp->hdr.serial,
+ tracepoint_string("bad_hdr"));
return -EBADMSG;
+ }
memset(sp, 0, sizeof(*sp));
sp->hdr.epoch = ntohl(whdr.epoch);
diff --git a/net/rxrpc/insecure.c b/net/rxrpc/insecure.c
index 7d4375e..af276f1 100644
--- a/net/rxrpc/insecure.c
+++ b/net/rxrpc/insecure.c
@@ -46,7 +46,10 @@
struct sk_buff *skb,
u32 *_abort_code)
{
- *_abort_code = RX_PROTOCOL_ERROR;
+ struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
+
+ trace_rxrpc_rx_eproto(NULL, sp->hdr.serial,
+ tracepoint_string("chall_none"));
return -EPROTO;
}
@@ -54,7 +57,10 @@
struct sk_buff *skb,
u32 *_abort_code)
{
- *_abort_code = RX_PROTOCOL_ERROR;
+ struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
+
+ trace_rxrpc_rx_eproto(NULL, sp->hdr.serial,
+ tracepoint_string("resp_none"));
return -EPROTO;
}
diff --git a/net/rxrpc/peer_event.c b/net/rxrpc/peer_event.c
index bf13b84..1ed9c0c 100644
--- a/net/rxrpc/peer_event.c
+++ b/net/rxrpc/peer_event.c
@@ -296,7 +296,7 @@
hlist_del_init(&call->error_link);
rxrpc_see_call(call);
- if (rxrpc_set_call_completion(call, compl, 0, error))
+ if (rxrpc_set_call_completion(call, compl, 0, -error))
rxrpc_notify_socket(call);
}
diff --git a/net/rxrpc/recvmsg.c b/net/rxrpc/recvmsg.c
index 3e2f1a8..f9caf3b 100644
--- a/net/rxrpc/recvmsg.c
+++ b/net/rxrpc/recvmsg.c
@@ -83,11 +83,11 @@
ret = put_cmsg(msg, SOL_RXRPC, RXRPC_ABORT, 4, &tmp);
break;
case RXRPC_CALL_NETWORK_ERROR:
- tmp = call->error;
+ tmp = -call->error;
ret = put_cmsg(msg, SOL_RXRPC, RXRPC_NET_ERROR, 4, &tmp);
break;
case RXRPC_CALL_LOCAL_ERROR:
- tmp = call->error;
+ tmp = -call->error;
ret = put_cmsg(msg, SOL_RXRPC, RXRPC_LOCAL_ERROR, 4, &tmp);
break;
default:
@@ -682,14 +682,16 @@
return ret;
short_data:
+ trace_rxrpc_rx_eproto(call, 0, tracepoint_string("short_data"));
ret = -EBADMSG;
goto out;
excess_data:
+ trace_rxrpc_rx_eproto(call, 0, tracepoint_string("excess_data"));
ret = -EMSGSIZE;
goto out;
call_complete:
*_abort = call->abort_code;
- ret = -call->error;
+ ret = call->error;
if (call->completion == RXRPC_CALL_SUCCEEDED) {
ret = 1;
if (size > 0)
diff --git a/net/rxrpc/rxkad.c b/net/rxrpc/rxkad.c
index 4374e7b..1bb9b2c 100644
--- a/net/rxrpc/rxkad.c
+++ b/net/rxrpc/rxkad.c
@@ -148,15 +148,13 @@
u32 data_size,
void *sechdr)
{
- struct rxrpc_skb_priv *sp;
+ struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
SKCIPHER_REQUEST_ON_STACK(req, call->conn->cipher);
struct rxkad_level1_hdr hdr;
struct rxrpc_crypt iv;
struct scatterlist sg;
u16 check;
- sp = rxrpc_skb(skb);
-
_enter("");
check = sp->hdr.seq ^ call->call_id;
@@ -323,6 +321,7 @@
struct rxrpc_crypt iv;
struct scatterlist sg[16];
struct sk_buff *trailer;
+ bool aborted;
u32 data_size, buf;
u16 check;
int nsg;
@@ -330,7 +329,8 @@
_enter("");
if (len < 8) {
- rxrpc_abort_call("V1H", call, seq, RXKADSEALEDINCON, EPROTO);
+ aborted = rxrpc_abort_eproto(call, skb, "rxkad_1_hdr", "V1H",
+ RXKADSEALEDINCON);
goto protocol_error;
}
@@ -355,7 +355,8 @@
/* Extract the decrypted packet length */
if (skb_copy_bits(skb, offset, &sechdr, sizeof(sechdr)) < 0) {
- rxrpc_abort_call("XV1", call, seq, RXKADDATALEN, EPROTO);
+ aborted = rxrpc_abort_eproto(call, skb, "rxkad_1_len", "XV1",
+ RXKADDATALEN);
goto protocol_error;
}
offset += sizeof(sechdr);
@@ -368,12 +369,14 @@
check ^= seq ^ call->call_id;
check &= 0xffff;
if (check != 0) {
- rxrpc_abort_call("V1C", call, seq, RXKADSEALEDINCON, EPROTO);
+ aborted = rxrpc_abort_eproto(call, skb, "rxkad_1_check", "V1C",
+ RXKADSEALEDINCON);
goto protocol_error;
}
if (data_size > len) {
- rxrpc_abort_call("V1L", call, seq, RXKADDATALEN, EPROTO);
+ aborted = rxrpc_abort_eproto(call, skb, "rxkad_1_datalen", "V1L",
+ RXKADDATALEN);
goto protocol_error;
}
@@ -381,8 +384,8 @@
return 0;
protocol_error:
- rxrpc_send_abort_packet(call);
- _leave(" = -EPROTO");
+ if (aborted)
+ rxrpc_send_abort_packet(call);
return -EPROTO;
nomem:
@@ -403,6 +406,7 @@
struct rxrpc_crypt iv;
struct scatterlist _sg[4], *sg;
struct sk_buff *trailer;
+ bool aborted;
u32 data_size, buf;
u16 check;
int nsg;
@@ -410,7 +414,8 @@
_enter(",{%d}", skb->len);
if (len < 8) {
- rxrpc_abort_call("V2H", call, seq, RXKADSEALEDINCON, EPROTO);
+ aborted = rxrpc_abort_eproto(call, skb, "rxkad_2_hdr", "V2H",
+ RXKADSEALEDINCON);
goto protocol_error;
}
@@ -445,7 +450,8 @@
/* Extract the decrypted packet length */
if (skb_copy_bits(skb, offset, &sechdr, sizeof(sechdr)) < 0) {
- rxrpc_abort_call("XV2", call, seq, RXKADDATALEN, EPROTO);
+ aborted = rxrpc_abort_eproto(call, skb, "rxkad_2_len", "XV2",
+ RXKADDATALEN);
goto protocol_error;
}
offset += sizeof(sechdr);
@@ -458,12 +464,14 @@
check ^= seq ^ call->call_id;
check &= 0xffff;
if (check != 0) {
- rxrpc_abort_call("V2C", call, seq, RXKADSEALEDINCON, EPROTO);
+ aborted = rxrpc_abort_eproto(call, skb, "rxkad_2_check", "V2C",
+ RXKADSEALEDINCON);
goto protocol_error;
}
if (data_size > len) {
- rxrpc_abort_call("V2L", call, seq, RXKADDATALEN, EPROTO);
+ aborted = rxrpc_abort_eproto(call, skb, "rxkad_2_datalen", "V2L",
+ RXKADDATALEN);
goto protocol_error;
}
@@ -471,8 +479,8 @@
return 0;
protocol_error:
- rxrpc_send_abort_packet(call);
- _leave(" = -EPROTO");
+ if (aborted)
+ rxrpc_send_abort_packet(call);
return -EPROTO;
nomem:
@@ -491,6 +499,7 @@
SKCIPHER_REQUEST_ON_STACK(req, call->conn->cipher);
struct rxrpc_crypt iv;
struct scatterlist sg;
+ bool aborted;
u16 cksum;
u32 x, y;
@@ -522,10 +531,9 @@
cksum = 1; /* zero checksums are not permitted */
if (cksum != expected_cksum) {
- rxrpc_abort_call("VCK", call, seq, RXKADSEALEDINCON, EPROTO);
- rxrpc_send_abort_packet(call);
- _leave(" = -EPROTO [csum failed]");
- return -EPROTO;
+ aborted = rxrpc_abort_eproto(call, skb, "rxkad_csum", "VCK",
+ RXKADSEALEDINCON);
+ goto protocol_error;
}
switch (call->conn->params.security_level) {
@@ -538,6 +546,11 @@
default:
return -ENOANO;
}
+
+protocol_error:
+ if (aborted)
+ rxrpc_send_abort_packet(call);
+ return -EPROTO;
}
/*
@@ -754,22 +767,23 @@
struct rxkad_response resp
__attribute__((aligned(8))); /* must be aligned for crypto */
struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
+ const char *eproto;
u32 version, nonce, min_level, abort_code;
int ret;
_enter("{%d,%x}", conn->debug_id, key_serial(conn->params.key));
- if (!conn->params.key) {
- _leave(" = -EPROTO [no key]");
- return -EPROTO;
- }
+ eproto = tracepoint_string("chall_no_key");
+ abort_code = RX_PROTOCOL_ERROR;
+ if (!conn->params.key)
+ goto protocol_error;
+ abort_code = RXKADEXPIRED;
ret = key_validate(conn->params.key);
- if (ret < 0) {
- *_abort_code = RXKADEXPIRED;
- return ret;
- }
+ if (ret < 0)
+ goto other_error;
+ eproto = tracepoint_string("chall_short");
abort_code = RXKADPACKETSHORT;
if (skb_copy_bits(skb, sizeof(struct rxrpc_wire_header),
&challenge, sizeof(challenge)) < 0)
@@ -782,13 +796,15 @@
_proto("Rx CHALLENGE %%%u { v=%u n=%u ml=%u }",
sp->hdr.serial, version, nonce, min_level);
+ eproto = tracepoint_string("chall_ver");
abort_code = RXKADINCONSISTENCY;
if (version != RXKAD_VERSION)
goto protocol_error;
abort_code = RXKADLEVELFAIL;
+ ret = -EACCES;
if (conn->params.security_level < min_level)
- goto protocol_error;
+ goto other_error;
token = conn->params.key->payload.data[0];
@@ -815,28 +831,34 @@
return rxkad_send_response(conn, &sp->hdr, &resp, token->kad);
protocol_error:
+ trace_rxrpc_rx_eproto(NULL, sp->hdr.serial, eproto);
+ ret = -EPROTO;
+other_error:
*_abort_code = abort_code;
- _leave(" = -EPROTO [%d]", abort_code);
- return -EPROTO;
+ return ret;
}
/*
* decrypt the kerberos IV ticket in the response
*/
static int rxkad_decrypt_ticket(struct rxrpc_connection *conn,
+ struct sk_buff *skb,
void *ticket, size_t ticket_len,
struct rxrpc_crypt *_session_key,
time_t *_expiry,
u32 *_abort_code)
{
struct skcipher_request *req;
+ struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
struct rxrpc_crypt iv, key;
struct scatterlist sg[1];
struct in_addr addr;
unsigned int life;
+ const char *eproto;
time_t issue, now;
bool little_endian;
int ret;
+ u32 abort_code;
u8 *p, *q, *name, *end;
_enter("{%d},{%x}", conn->debug_id, key_serial(conn->server_key));
@@ -847,11 +869,11 @@
if (ret < 0) {
switch (ret) {
case -EKEYEXPIRED:
- *_abort_code = RXKADEXPIRED;
- goto error;
+ abort_code = RXKADEXPIRED;
+ goto other_error;
default:
- *_abort_code = RXKADNOAUTH;
- goto error;
+ abort_code = RXKADNOAUTH;
+ goto other_error;
}
}
@@ -860,13 +882,11 @@
memcpy(&iv, &conn->server_key->payload.data[2], sizeof(iv));
+ ret = -ENOMEM;
req = skcipher_request_alloc(conn->server_key->payload.data[0],
GFP_NOFS);
- if (!req) {
- *_abort_code = RXKADNOAUTH;
- ret = -ENOMEM;
- goto error;
- }
+ if (!req)
+ goto temporary_error;
sg_init_one(&sg[0], ticket, ticket_len);
skcipher_request_set_callback(req, 0, NULL, NULL);
@@ -877,11 +897,12 @@
p = ticket;
end = p + ticket_len;
-#define Z(size) \
+#define Z(field) \
({ \
u8 *__str = p; \
+ eproto = tracepoint_string("rxkad_bad_"#field); \
q = memchr(p, 0, end - p); \
- if (!q || q - p > (size)) \
+ if (!q || q - p > (field##_SZ)) \
goto bad_ticket; \
for (; p < q; p++) \
if (!isprint(*p)) \
@@ -896,17 +917,18 @@
p++;
/* extract the authentication name */
- name = Z(ANAME_SZ);
+ name = Z(ANAME);
_debug("KIV ANAME: %s", name);
/* extract the principal's instance */
- name = Z(INST_SZ);
+ name = Z(INST);
_debug("KIV INST : %s", name);
/* extract the principal's authentication domain */
- name = Z(REALM_SZ);
+ name = Z(REALM);
_debug("KIV REALM: %s", name);
+ eproto = tracepoint_string("rxkad_bad_len");
if (end - p < 4 + 8 + 4 + 2)
goto bad_ticket;
@@ -941,36 +963,37 @@
/* check the ticket is in date */
if (issue > now) {
- *_abort_code = RXKADNOAUTH;
+ abort_code = RXKADNOAUTH;
ret = -EKEYREJECTED;
- goto error;
+ goto other_error;
}
if (issue < now - life) {
- *_abort_code = RXKADEXPIRED;
+ abort_code = RXKADEXPIRED;
ret = -EKEYEXPIRED;
- goto error;
+ goto other_error;
}
*_expiry = issue + life;
/* get the service name */
- name = Z(SNAME_SZ);
+ name = Z(SNAME);
_debug("KIV SNAME: %s", name);
/* get the service instance name */
- name = Z(INST_SZ);
+ name = Z(INST);
_debug("KIV SINST: %s", name);
-
- ret = 0;
-error:
- _leave(" = %d", ret);
- return ret;
+ return 0;
bad_ticket:
- *_abort_code = RXKADBADTICKET;
- ret = -EBADMSG;
- goto error;
+ trace_rxrpc_rx_eproto(NULL, sp->hdr.serial, eproto);
+ abort_code = RXKADBADTICKET;
+ ret = -EPROTO;
+other_error:
+ *_abort_code = abort_code;
+ return ret;
+temporary_error:
+ return ret;
}
/*
@@ -1020,6 +1043,7 @@
__attribute__((aligned(8))); /* must be aligned for crypto */
struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
struct rxrpc_crypt session_key;
+ const char *eproto;
time_t expiry;
void *ticket;
u32 abort_code, version, kvno, ticket_len, level;
@@ -1028,6 +1052,7 @@
_enter("{%d,%x}", conn->debug_id, key_serial(conn->server_key));
+ eproto = tracepoint_string("rxkad_rsp_short");
abort_code = RXKADPACKETSHORT;
if (skb_copy_bits(skb, sizeof(struct rxrpc_wire_header),
&response, sizeof(response)) < 0)
@@ -1041,40 +1066,43 @@
_proto("Rx RESPONSE %%%u { v=%u kv=%u tl=%u }",
sp->hdr.serial, version, kvno, ticket_len);
+ eproto = tracepoint_string("rxkad_rsp_ver");
abort_code = RXKADINCONSISTENCY;
if (version != RXKAD_VERSION)
goto protocol_error;
+ eproto = tracepoint_string("rxkad_rsp_tktlen");
abort_code = RXKADTICKETLEN;
if (ticket_len < 4 || ticket_len > MAXKRB5TICKETLEN)
goto protocol_error;
+ eproto = tracepoint_string("rxkad_rsp_unkkey");
abort_code = RXKADUNKNOWNKEY;
if (kvno >= RXKAD_TKT_TYPE_KERBEROS_V5)
goto protocol_error;
/* extract the kerberos ticket and decrypt and decode it */
+ ret = -ENOMEM;
ticket = kmalloc(ticket_len, GFP_NOFS);
if (!ticket)
- return -ENOMEM;
+ goto temporary_error;
+ eproto = tracepoint_string("rxkad_tkt_short");
abort_code = RXKADPACKETSHORT;
if (skb_copy_bits(skb, sizeof(struct rxrpc_wire_header),
ticket, ticket_len) < 0)
goto protocol_error_free;
- ret = rxkad_decrypt_ticket(conn, ticket, ticket_len, &session_key,
- &expiry, &abort_code);
- if (ret < 0) {
- *_abort_code = abort_code;
- kfree(ticket);
- return ret;
- }
+ ret = rxkad_decrypt_ticket(conn, skb, ticket, ticket_len, &session_key,
+ &expiry, _abort_code);
+ if (ret < 0)
+ goto temporary_error_free;
/* use the session key from inside the ticket to decrypt the
* response */
rxkad_decrypt_response(conn, &response, &session_key);
+ eproto = tracepoint_string("rxkad_rsp_param");
abort_code = RXKADSEALEDINCON;
if (ntohl(response.encrypted.epoch) != conn->proto.epoch)
goto protocol_error_free;
@@ -1085,6 +1113,7 @@
csum = response.encrypted.checksum;
response.encrypted.checksum = 0;
rxkad_calc_response_checksum(&response);
+ eproto = tracepoint_string("rxkad_rsp_csum");
if (response.encrypted.checksum != csum)
goto protocol_error_free;
@@ -1093,11 +1122,15 @@
struct rxrpc_call *call;
u32 call_id = ntohl(response.encrypted.call_id[i]);
+ eproto = tracepoint_string("rxkad_rsp_callid");
if (call_id > INT_MAX)
goto protocol_error_unlock;
+ eproto = tracepoint_string("rxkad_rsp_callctr");
if (call_id < conn->channels[i].call_counter)
goto protocol_error_unlock;
+
+ eproto = tracepoint_string("rxkad_rsp_callst");
if (call_id > conn->channels[i].call_counter) {
call = rcu_dereference_protected(
conn->channels[i].call,
@@ -1109,10 +1142,12 @@
}
spin_unlock(&conn->channel_lock);
+ eproto = tracepoint_string("rxkad_rsp_seq");
abort_code = RXKADOUTOFSEQUENCE;
if (ntohl(response.encrypted.inc_nonce) != conn->security_nonce + 1)
goto protocol_error_free;
+ eproto = tracepoint_string("rxkad_rsp_level");
abort_code = RXKADLEVELFAIL;
level = ntohl(response.encrypted.level);
if (level > RXRPC_SECURITY_ENCRYPT)
@@ -1123,10 +1158,8 @@
* this the connection security can be handled in exactly the same way
* as for a client connection */
ret = rxrpc_get_server_data_key(conn, &session_key, expiry, kvno);
- if (ret < 0) {
- kfree(ticket);
- return ret;
- }
+ if (ret < 0)
+ goto temporary_error_free;
kfree(ticket);
_leave(" = 0");
@@ -1137,9 +1170,18 @@
protocol_error_free:
kfree(ticket);
protocol_error:
+ trace_rxrpc_rx_eproto(NULL, sp->hdr.serial, eproto);
*_abort_code = abort_code;
- _leave(" = -EPROTO [%d]", abort_code);
return -EPROTO;
+
+temporary_error_free:
+ kfree(ticket);
+temporary_error:
+ /* Ignore the response packet if we got a temporary error such as
+ * ENOMEM. We just want to send the challenge again. Note that we
+ * also come out this way if the ticket decryption fails.
+ */
+ return ret;
}
/*
diff --git a/net/rxrpc/sendmsg.c b/net/rxrpc/sendmsg.c
index 97ab214..96ffa5d 100644
--- a/net/rxrpc/sendmsg.c
+++ b/net/rxrpc/sendmsg.c
@@ -556,7 +556,7 @@
ret = -ESHUTDOWN;
} else if (cmd == RXRPC_CMD_SEND_ABORT) {
ret = 0;
- if (rxrpc_abort_call("CMD", call, 0, abort_code, ECONNABORTED))
+ if (rxrpc_abort_call("CMD", call, 0, abort_code, -ECONNABORTED))
ret = rxrpc_send_abort_packet(call);
} else if (cmd != RXRPC_CMD_SEND_DATA) {
ret = -EINVAL;
@@ -623,7 +623,8 @@
read_unlock_bh(&call->state_lock);
break;
default:
- /* Request phase complete for this client call */
+ /* Request phase complete for this client call */
+ trace_rxrpc_rx_eproto(call, 0, tracepoint_string("late_send"));
ret = -EPROTO;
break;
}
@@ -642,20 +643,24 @@
* @error: Local error value
* @why: 3-char string indicating why.
*
- * Allow a kernel service to abort a call, if it's still in an abortable state.
+ * Allow a kernel service to abort a call, if it's still in an abortable state
+ * and return true if the call was aborted, false if it was already complete.
*/
-void rxrpc_kernel_abort_call(struct socket *sock, struct rxrpc_call *call,
+bool rxrpc_kernel_abort_call(struct socket *sock, struct rxrpc_call *call,
u32 abort_code, int error, const char *why)
{
+ bool aborted;
+
_enter("{%d},%d,%d,%s", call->debug_id, abort_code, error, why);
mutex_lock(&call->user_mutex);
- if (rxrpc_abort_call(why, call, 0, abort_code, error))
+ aborted = rxrpc_abort_call(why, call, 0, abort_code, error);
+ if (aborted)
rxrpc_send_abort_packet(call);
mutex_unlock(&call->user_mutex);
- _leave("");
+ return aborted;
}
EXPORT_SYMBOL(rxrpc_kernel_abort_call);
diff --git a/net/sched/Kconfig b/net/sched/Kconfig
index 403790c..9fb84f0 100644
--- a/net/sched/Kconfig
+++ b/net/sched/Kconfig
@@ -352,6 +352,51 @@
To compile this code as a module, choose M here: the
module will be called sch_plug.
+menuconfig NET_SCH_DEFAULT
+ bool "Allow override default queue discipline"
+ ---help---
+ Support for selection of default queuing discipline.
+
+ Nearly all users can safely say no here, and the default
+ of pfifo_fast will be used. Many distributions already set
+ the default value via /proc/sys/net/core/default_qdisc.
+
+ If unsure, say N.
+
+if NET_SCH_DEFAULT
+
+choice
+ prompt "Default queuing discipline"
+ default DEFAULT_PFIFO_FAST
+ help
+ Select the queueing discipline that will be used by default
+ for all network devices.
+
+ config DEFAULT_FQ
+ bool "Fair Queue" if NET_SCH_FQ
+
+ config DEFAULT_CODEL
+ bool "Controlled Delay" if NET_SCH_CODEL
+
+ config DEFAULT_FQ_CODEL
+ bool "Fair Queue Controlled Delay" if NET_SCH_FQ_CODEL
+
+ config DEFAULT_SFQ
+ bool "Stochastic Fair Queue" if NET_SCH_SFQ
+
+ config DEFAULT_PFIFO_FAST
+ bool "Priority FIFO Fast"
+endchoice
+
+config DEFAULT_NET_SCH
+ string
+ default "pfifo_fast" if DEFAULT_PFIFO_FAST
+ default "fq" if DEFAULT_FQ
+ default "fq_codel" if DEFAULT_FQ_CODEL
+ default "sfq" if DEFAULT_SFQ
+ default "pfifo_fast"
+endif
+
comment "Classification"
config NET_CLS
diff --git a/net/sched/act_api.c b/net/sched/act_api.c
index e05b924..a90e8f3 100644
--- a/net/sched/act_api.c
+++ b/net/sched/act_api.c
@@ -428,24 +428,49 @@
return res;
}
+/*TCA_ACT_MAX_PRIO is 32, there count upto 32 */
+#define TCA_ACT_MAX_PRIO_MASK 0x1FF
int tcf_action_exec(struct sk_buff *skb, struct tc_action **actions,
int nr_actions, struct tcf_result *res)
{
int ret = -1, i;
+ u32 jmp_prgcnt = 0;
+ u32 jmp_ttl = TCA_ACT_MAX_PRIO; /*matches actions per filter */
if (skb_skip_tc_classify(skb))
return TC_ACT_OK;
+restart_act_graph:
for (i = 0; i < nr_actions; i++) {
const struct tc_action *a = actions[i];
+ if (jmp_prgcnt > 0) {
+ jmp_prgcnt -= 1;
+ continue;
+ }
repeat:
ret = a->ops->act(skb, a, res);
if (ret == TC_ACT_REPEAT)
goto repeat; /* we need a ttl - JHS */
+
+ if (TC_ACT_EXT_CMP(ret, TC_ACT_JUMP)) {
+ jmp_prgcnt = ret & TCA_ACT_MAX_PRIO_MASK;
+ if (!jmp_prgcnt || (jmp_prgcnt > nr_actions)) {
+ /* faulty opcode, stop pipeline */
+ return TC_ACT_OK;
+ } else {
+ jmp_ttl -= 1;
+ if (jmp_ttl > 0)
+ goto restart_act_graph;
+ else /* faulty graph, stop pipeline */
+ return TC_ACT_OK;
+ }
+ }
+
if (ret != TC_ACT_PIPE)
break;
}
+
return ret;
}
EXPORT_SYMBOL(tcf_action_exec);
@@ -558,7 +583,7 @@
int err;
if (name == NULL) {
- err = nla_parse_nested(tb, TCA_ACT_MAX, nla, NULL);
+ err = nla_parse_nested(tb, TCA_ACT_MAX, nla, NULL, NULL);
if (err < 0)
goto err_out;
err = -EINVAL;
@@ -663,7 +688,7 @@
int err;
int i;
- err = nla_parse_nested(tb, TCA_ACT_MAX_PRIO, nla, NULL);
+ err = nla_parse_nested(tb, TCA_ACT_MAX_PRIO, nla, NULL, NULL);
if (err < 0)
return err;
@@ -795,7 +820,7 @@
int index;
int err;
- err = nla_parse_nested(tb, TCA_ACT_MAX, nla, NULL);
+ err = nla_parse_nested(tb, TCA_ACT_MAX, nla, NULL, NULL);
if (err < 0)
goto err_out;
@@ -844,7 +869,7 @@
b = skb_tail_pointer(skb);
- err = nla_parse_nested(tb, TCA_ACT_MAX, nla, NULL);
+ err = nla_parse_nested(tb, TCA_ACT_MAX, nla, NULL, NULL);
if (err < 0)
goto err_out;
@@ -930,7 +955,7 @@
struct tc_action *act;
LIST_HEAD(actions);
- ret = nla_parse_nested(tb, TCA_ACT_MAX_PRIO, nla, NULL);
+ ret = nla_parse_nested(tb, TCA_ACT_MAX_PRIO, nla, NULL, NULL);
if (ret < 0)
return ret;
@@ -1002,7 +1027,8 @@
return tcf_add_notify(net, n, &actions, portid);
}
-static int tc_ctl_action(struct sk_buff *skb, struct nlmsghdr *n)
+static int tc_ctl_action(struct sk_buff *skb, struct nlmsghdr *n,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct nlattr *tca[TCA_ACT_MAX + 1];
@@ -1013,7 +1039,8 @@
!netlink_capable(skb, CAP_NET_ADMIN))
return -EPERM;
- ret = nlmsg_parse(n, sizeof(struct tcamsg), tca, TCA_ACT_MAX, NULL);
+ ret = nlmsg_parse(n, sizeof(struct tcamsg), tca, TCA_ACT_MAX, NULL,
+ extack);
if (ret < 0)
return ret;
@@ -1060,19 +1087,20 @@
struct nlattr *nla[TCAA_MAX + 1];
struct nlattr *kind;
- if (nlmsg_parse(n, sizeof(struct tcamsg), nla, TCAA_MAX, NULL) < 0)
+ if (nlmsg_parse(n, sizeof(struct tcamsg), nla, TCAA_MAX,
+ NULL, NULL) < 0)
return NULL;
tb1 = nla[TCA_ACT_TAB];
if (tb1 == NULL)
return NULL;
if (nla_parse(tb, TCA_ACT_MAX_PRIO, nla_data(tb1),
- NLMSG_ALIGN(nla_len(tb1)), NULL) < 0)
+ NLMSG_ALIGN(nla_len(tb1)), NULL, NULL) < 0)
return NULL;
if (tb[1] == NULL)
return NULL;
- if (nla_parse_nested(tb2, TCA_ACT_MAX, tb[1], NULL) < 0)
+ if (nla_parse_nested(tb2, TCA_ACT_MAX, tb[1], NULL, NULL) < 0)
return NULL;
kind = tb2[TCA_ACT_KIND];
diff --git a/net/sched/act_bpf.c b/net/sched/act_bpf.c
index 520baa41..d33947d 100644
--- a/net/sched/act_bpf.c
+++ b/net/sched/act_bpf.c
@@ -283,7 +283,7 @@
if (!nla)
return -EINVAL;
- ret = nla_parse_nested(tb, TCA_ACT_BPF_MAX, nla, act_bpf_policy);
+ ret = nla_parse_nested(tb, TCA_ACT_BPF_MAX, nla, act_bpf_policy, NULL);
if (ret < 0)
return ret;
diff --git a/net/sched/act_connmark.c b/net/sched/act_connmark.c
index f9bb43c..2155bc6 100644
--- a/net/sched/act_connmark.c
+++ b/net/sched/act_connmark.c
@@ -109,7 +109,8 @@
if (!nla)
return -EINVAL;
- ret = nla_parse_nested(tb, TCA_CONNMARK_MAX, nla, connmark_policy);
+ ret = nla_parse_nested(tb, TCA_CONNMARK_MAX, nla, connmark_policy,
+ NULL);
if (ret < 0)
return ret;
diff --git a/net/sched/act_csum.c b/net/sched/act_csum.c
index e978ccd4..ab6fdbd 100644
--- a/net/sched/act_csum.c
+++ b/net/sched/act_csum.c
@@ -59,7 +59,7 @@
if (nla == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_CSUM_MAX, nla, csum_policy);
+ err = nla_parse_nested(tb, TCA_CSUM_MAX, nla, csum_policy, NULL);
if (err < 0)
return err;
@@ -181,6 +181,9 @@
struct tcphdr *tcph;
const struct iphdr *iph;
+ if (skb_is_gso(skb) && skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4)
+ return 1;
+
tcph = tcf_csum_skb_nextlayer(skb, ihl, ipl, sizeof(*tcph));
if (tcph == NULL)
return 0;
@@ -202,6 +205,9 @@
struct tcphdr *tcph;
const struct ipv6hdr *ip6h;
+ if (skb_is_gso(skb) && skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6)
+ return 1;
+
tcph = tcf_csum_skb_nextlayer(skb, ihl, ipl, sizeof(*tcph));
if (tcph == NULL)
return 0;
@@ -225,6 +231,9 @@
const struct iphdr *iph;
u16 ul;
+ if (skb_is_gso(skb) && skb_shinfo(skb)->gso_type & SKB_GSO_UDP)
+ return 1;
+
/*
* Support both UDP and UDPLITE checksum algorithms, Don't use
* udph->len to get the real length without any protocol check,
@@ -278,6 +287,9 @@
const struct ipv6hdr *ip6h;
u16 ul;
+ if (skb_is_gso(skb) && skb_shinfo(skb)->gso_type & SKB_GSO_UDP)
+ return 1;
+
/*
* Support both UDP and UDPLITE checksum algorithms, Don't use
* udph->len to get the real length without any protocol check,
diff --git a/net/sched/act_gact.c b/net/sched/act_gact.c
index e6c874a..99afe8b 100644
--- a/net/sched/act_gact.c
+++ b/net/sched/act_gact.c
@@ -73,7 +73,7 @@
if (nla == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_GACT_MAX, nla, gact_policy);
+ err = nla_parse_nested(tb, TCA_GACT_MAX, nla, gact_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c
index 71e7ff22..c5dec30 100644
--- a/net/sched/act_ife.c
+++ b/net/sched/act_ife.c
@@ -443,7 +443,7 @@
int ret = 0;
int err;
- err = nla_parse_nested(tb, TCA_IFE_MAX, nla, ife_policy);
+ err = nla_parse_nested(tb, TCA_IFE_MAX, nla, ife_policy, NULL);
if (err < 0)
return err;
@@ -514,7 +514,7 @@
if (tb[TCA_IFE_METALST]) {
err = nla_parse_nested(tb2, IFE_META_MAX, tb[TCA_IFE_METALST],
- NULL);
+ NULL, NULL);
if (err) {
metadata_parse_err:
if (exists)
@@ -603,8 +603,8 @@
return -1;
}
-int find_decode_metaid(struct sk_buff *skb, struct tcf_ife_info *ife,
- u16 metaid, u16 mlen, void *mdata)
+static int find_decode_metaid(struct sk_buff *skb, struct tcf_ife_info *ife,
+ u16 metaid, u16 mlen, void *mdata)
{
struct tcf_meta_info *e;
diff --git a/net/sched/act_ipt.c b/net/sched/act_ipt.c
index 992ef8d..36f0ced 100644
--- a/net/sched/act_ipt.c
+++ b/net/sched/act_ipt.c
@@ -107,7 +107,7 @@
if (nla == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_IPT_MAX, nla, ipt_policy);
+ err = nla_parse_nested(tb, TCA_IPT_MAX, nla, ipt_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/act_mirred.c b/net/sched/act_mirred.c
index af49c7d..1b5549a 100644
--- a/net/sched/act_mirred.c
+++ b/net/sched/act_mirred.c
@@ -87,7 +87,7 @@
if (nla == NULL)
return -EINVAL;
- ret = nla_parse_nested(tb, TCA_MIRRED_MAX, nla, mirred_policy);
+ ret = nla_parse_nested(tb, TCA_MIRRED_MAX, nla, mirred_policy, NULL);
if (ret < 0)
return ret;
if (tb[TCA_MIRRED_PARMS] == NULL)
diff --git a/net/sched/act_nat.c b/net/sched/act_nat.c
index 9b6aec6..9016ab8 100644
--- a/net/sched/act_nat.c
+++ b/net/sched/act_nat.c
@@ -50,7 +50,7 @@
if (nla == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_NAT_MAX, nla, nat_policy);
+ err = nla_parse_nested(tb, TCA_NAT_MAX, nla, nat_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/act_pedit.c b/net/sched/act_pedit.c
index c131047..164b5ac 100644
--- a/net/sched/act_pedit.c
+++ b/net/sched/act_pedit.c
@@ -72,7 +72,7 @@
}
err = nla_parse_nested(tb, TCA_PEDIT_KEY_EX_MAX, ka,
- pedit_key_ex_policy);
+ pedit_key_ex_policy, NULL);
if (err)
goto err_out;
@@ -147,7 +147,7 @@
if (nla == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_PEDIT_MAX, nla, pedit_policy);
+ err = nla_parse_nested(tb, TCA_PEDIT_MAX, nla, pedit_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/act_police.c b/net/sched/act_police.c
index 0ba91d1..f42008b 100644
--- a/net/sched/act_police.c
+++ b/net/sched/act_police.c
@@ -90,7 +90,7 @@
if (nla == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_POLICE_MAX, nla, police_policy);
+ err = nla_parse_nested(tb, TCA_POLICE_MAX, nla, police_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/act_sample.c b/net/sched/act_sample.c
index 0b8217b..59d6645 100644
--- a/net/sched/act_sample.c
+++ b/net/sched/act_sample.c
@@ -50,7 +50,7 @@
if (!nla)
return -EINVAL;
- ret = nla_parse_nested(tb, TCA_SAMPLE_MAX, nla, sample_policy);
+ ret = nla_parse_nested(tb, TCA_SAMPLE_MAX, nla, sample_policy, NULL);
if (ret < 0)
return ret;
if (!tb[TCA_SAMPLE_PARMS] || !tb[TCA_SAMPLE_RATE] ||
diff --git a/net/sched/act_simple.c b/net/sched/act_simple.c
index 823a73a..43605e7 100644
--- a/net/sched/act_simple.c
+++ b/net/sched/act_simple.c
@@ -94,7 +94,7 @@
if (nla == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_DEF_MAX, nla, simple_policy);
+ err = nla_parse_nested(tb, TCA_DEF_MAX, nla, simple_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/act_skbedit.c b/net/sched/act_skbedit.c
index 06ccae3..6b3e65d 100644
--- a/net/sched/act_skbedit.c
+++ b/net/sched/act_skbedit.c
@@ -82,7 +82,7 @@
if (nla == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_SKBEDIT_MAX, nla, skbedit_policy);
+ err = nla_parse_nested(tb, TCA_SKBEDIT_MAX, nla, skbedit_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/act_skbmod.c b/net/sched/act_skbmod.c
index c736627..a73c4bb 100644
--- a/net/sched/act_skbmod.c
+++ b/net/sched/act_skbmod.c
@@ -103,7 +103,7 @@
if (!nla)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_SKBMOD_MAX, nla, skbmod_policy);
+ err = nla_parse_nested(tb, TCA_SKBMOD_MAX, nla, skbmod_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/act_tunnel_key.c b/net/sched/act_tunnel_key.c
index e3a58e0..b9a2f24 100644
--- a/net/sched/act_tunnel_key.c
+++ b/net/sched/act_tunnel_key.c
@@ -89,7 +89,8 @@
if (!nla)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_TUNNEL_KEY_MAX, nla, tunnel_key_policy);
+ err = nla_parse_nested(tb, TCA_TUNNEL_KEY_MAX, nla, tunnel_key_policy,
+ NULL);
if (err < 0)
return err;
diff --git a/net/sched/act_vlan.c b/net/sched/act_vlan.c
index 19e0dba..13ba3a8 100644
--- a/net/sched/act_vlan.c
+++ b/net/sched/act_vlan.c
@@ -121,7 +121,7 @@
if (!nla)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_VLAN_MAX, nla, vlan_policy);
+ err = nla_parse_nested(tb, TCA_VLAN_MAX, nla, vlan_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c
index 732f7ca..22f88b3 100644
--- a/net/sched/cls_api.c
+++ b/net/sched/cls_api.c
@@ -178,14 +178,11 @@
return ERR_PTR(err);
}
-static bool tcf_proto_destroy(struct tcf_proto *tp, bool force)
+static void tcf_proto_destroy(struct tcf_proto *tp)
{
- if (tp->ops->destroy(tp, force)) {
- module_put(tp->ops->owner);
- kfree_rcu(tp, rcu);
- return true;
- }
- return false;
+ tp->ops->destroy(tp);
+ module_put(tp->ops->owner);
+ kfree_rcu(tp, rcu);
}
void tcf_destroy_chain(struct tcf_proto __rcu **fl)
@@ -194,14 +191,15 @@
while ((tp = rtnl_dereference(*fl)) != NULL) {
RCU_INIT_POINTER(*fl, tp->next);
- tcf_proto_destroy(tp, true);
+ tcf_proto_destroy(tp);
}
}
EXPORT_SYMBOL(tcf_destroy_chain);
/* Add/change/delete/get a filter node */
-static int tc_ctl_tfilter(struct sk_buff *skb, struct nlmsghdr *n)
+static int tc_ctl_tfilter(struct sk_buff *skb, struct nlmsghdr *n,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct nlattr *tca[TCA_MAX + 1];
@@ -229,7 +227,7 @@
replay:
tp_created = 0;
- err = nlmsg_parse(n, sizeof(*t), tca, TCA_MAX, NULL);
+ err = nlmsg_parse(n, sizeof(*t), tca, TCA_MAX, NULL, extack);
if (err < 0)
return err;
@@ -360,7 +358,7 @@
RCU_INIT_POINTER(*back, next);
tfilter_notify(net, skb, n, tp, fh,
RTM_DELTFILTER, false);
- tcf_proto_destroy(tp, true);
+ tcf_proto_destroy(tp);
err = 0;
goto errout;
}
@@ -371,24 +369,28 @@
goto errout;
}
} else {
+ bool last;
+
switch (n->nlmsg_type) {
case RTM_NEWTFILTER:
if (n->nlmsg_flags & NLM_F_EXCL) {
if (tp_created)
- tcf_proto_destroy(tp, true);
+ tcf_proto_destroy(tp);
err = -EEXIST;
goto errout;
}
break;
case RTM_DELTFILTER:
- err = tp->ops->delete(tp, fh);
+ err = tp->ops->delete(tp, fh, &last);
if (err)
goto errout;
next = rtnl_dereference(tp->next);
tfilter_notify(net, skb, n, tp, t->tcm_handle,
RTM_DELTFILTER, false);
- if (tcf_proto_destroy(tp, false))
+ if (last) {
RCU_INIT_POINTER(*back, next);
+ tcf_proto_destroy(tp);
+ }
goto errout;
case RTM_GETTFILTER:
err = tfilter_notify(net, skb, n, tp, fh,
@@ -410,7 +412,7 @@
tfilter_notify(net, skb, n, tp, fh, RTM_NEWTFILTER, false);
} else {
if (tp_created)
- tcf_proto_destroy(tp, true);
+ tcf_proto_destroy(tp);
}
errout:
diff --git a/net/sched/cls_basic.c b/net/sched/cls_basic.c
index 5877f60..c4fd63a0 100644
--- a/net/sched/cls_basic.c
+++ b/net/sched/cls_basic.c
@@ -93,30 +93,28 @@
kfree(f);
}
-static bool basic_destroy(struct tcf_proto *tp, bool force)
+static void basic_destroy(struct tcf_proto *tp)
{
struct basic_head *head = rtnl_dereference(tp->root);
struct basic_filter *f, *n;
- if (!force && !list_empty(&head->flist))
- return false;
-
list_for_each_entry_safe(f, n, &head->flist, link) {
list_del_rcu(&f->link);
tcf_unbind_filter(tp, &f->res);
call_rcu(&f->rcu, basic_delete_filter);
}
kfree_rcu(head, rcu);
- return true;
}
-static int basic_delete(struct tcf_proto *tp, unsigned long arg)
+static int basic_delete(struct tcf_proto *tp, unsigned long arg, bool *last)
{
+ struct basic_head *head = rtnl_dereference(tp->root);
struct basic_filter *f = (struct basic_filter *) arg;
list_del_rcu(&f->link);
tcf_unbind_filter(tp, &f->res);
call_rcu(&f->rcu, basic_delete_filter);
+ *last = list_empty(&head->flist);
return 0;
}
@@ -174,7 +172,7 @@
return -EINVAL;
err = nla_parse_nested(tb, TCA_BASIC_MAX, tca[TCA_OPTIONS],
- basic_policy);
+ basic_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/cls_bpf.c b/net/sched/cls_bpf.c
index 80f6884..5ebeae9 100644
--- a/net/sched/cls_bpf.c
+++ b/net/sched/cls_bpf.c
@@ -274,25 +274,24 @@
call_rcu(&prog->rcu, cls_bpf_delete_prog_rcu);
}
-static int cls_bpf_delete(struct tcf_proto *tp, unsigned long arg)
+static int cls_bpf_delete(struct tcf_proto *tp, unsigned long arg, bool *last)
{
+ struct cls_bpf_head *head = rtnl_dereference(tp->root);
+
__cls_bpf_delete(tp, (struct cls_bpf_prog *) arg);
+ *last = list_empty(&head->plist);
return 0;
}
-static bool cls_bpf_destroy(struct tcf_proto *tp, bool force)
+static void cls_bpf_destroy(struct tcf_proto *tp)
{
struct cls_bpf_head *head = rtnl_dereference(tp->root);
struct cls_bpf_prog *prog, *tmp;
- if (!force && !list_empty(&head->plist))
- return false;
-
list_for_each_entry_safe(prog, tmp, &head->plist, link)
__cls_bpf_delete(tp, prog);
kfree_rcu(head, rcu);
- return true;
}
static unsigned long cls_bpf_get(struct tcf_proto *tp, u32 handle)
@@ -478,7 +477,8 @@
if (tca[TCA_OPTIONS] == NULL)
return -EINVAL;
- ret = nla_parse_nested(tb, TCA_BPF_MAX, tca[TCA_OPTIONS], bpf_policy);
+ ret = nla_parse_nested(tb, TCA_BPF_MAX, tca[TCA_OPTIONS], bpf_policy,
+ NULL);
if (ret < 0)
return ret;
diff --git a/net/sched/cls_cgroup.c b/net/sched/cls_cgroup.c
index c1f2007..12ce547 100644
--- a/net/sched/cls_cgroup.c
+++ b/net/sched/cls_cgroup.c
@@ -99,7 +99,7 @@
new->handle = handle;
new->tp = tp;
err = nla_parse_nested(tb, TCA_CGROUP_MAX, tca[TCA_OPTIONS],
- cgroup_policy);
+ cgroup_policy, NULL);
if (err < 0)
goto errout;
@@ -131,20 +131,16 @@
return err;
}
-static bool cls_cgroup_destroy(struct tcf_proto *tp, bool force)
+static void cls_cgroup_destroy(struct tcf_proto *tp)
{
struct cls_cgroup_head *head = rtnl_dereference(tp->root);
- if (!force)
- return false;
/* Head can still be NULL due to cls_cgroup_init(). */
if (head)
call_rcu(&head->rcu, cls_cgroup_destroy_rcu);
-
- return true;
}
-static int cls_cgroup_delete(struct tcf_proto *tp, unsigned long arg)
+static int cls_cgroup_delete(struct tcf_proto *tp, unsigned long arg, bool *last)
{
return -EOPNOTSUPP;
}
diff --git a/net/sched/cls_flow.c b/net/sched/cls_flow.c
index 3d6b928..3065752 100644
--- a/net/sched/cls_flow.c
+++ b/net/sched/cls_flow.c
@@ -400,7 +400,7 @@
if (opt == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_FLOW_MAX, opt, flow_policy);
+ err = nla_parse_nested(tb, TCA_FLOW_MAX, opt, flow_policy, NULL);
if (err < 0)
return err;
@@ -508,9 +508,8 @@
get_random_bytes(&fnew->hashrnd, 4);
}
- fnew->perturb_timer.function = flow_perturbation;
- fnew->perturb_timer.data = (unsigned long)fnew;
- init_timer_deferrable(&fnew->perturb_timer);
+ setup_deferrable_timer(&fnew->perturb_timer, flow_perturbation,
+ (unsigned long)fnew);
tcf_exts_change(tp, &fnew->exts, &e);
tcf_em_tree_change(tp, &fnew->ematches, &t);
@@ -563,12 +562,14 @@
return err;
}
-static int flow_delete(struct tcf_proto *tp, unsigned long arg)
+static int flow_delete(struct tcf_proto *tp, unsigned long arg, bool *last)
{
+ struct flow_head *head = rtnl_dereference(tp->root);
struct flow_filter *f = (struct flow_filter *)arg;
list_del_rcu(&f->list);
call_rcu(&f->rcu, flow_destroy_filter);
+ *last = list_empty(&head->filters);
return 0;
}
@@ -584,20 +585,16 @@
return 0;
}
-static bool flow_destroy(struct tcf_proto *tp, bool force)
+static void flow_destroy(struct tcf_proto *tp)
{
struct flow_head *head = rtnl_dereference(tp->root);
struct flow_filter *f, *next;
- if (!force && !list_empty(&head->filters))
- return false;
-
list_for_each_entry_safe(f, next, &head->filters, list) {
list_del_rcu(&f->list);
call_rcu(&f->rcu, flow_destroy_filter);
}
kfree_rcu(head, rcu);
- return true;
}
static unsigned long flow_get(struct tcf_proto *tp, u32 handle)
diff --git a/net/sched/cls_flower.c b/net/sched/cls_flower.c
index 9d0c99d..ca526c0 100644
--- a/net/sched/cls_flower.c
+++ b/net/sched/cls_flower.c
@@ -18,6 +18,7 @@
#include <linux/if_ether.h>
#include <linux/in6.h>
#include <linux/ip.h>
+#include <linux/mpls.h>
#include <net/sch_generic.h>
#include <net/pkt_cls.h>
@@ -47,6 +48,7 @@
struct flow_dissector_key_ipv6_addrs enc_ipv6;
};
struct flow_dissector_key_ports enc_tp;
+ struct flow_dissector_key_mpls mpls;
} __aligned(BITS_PER_LONG / 8); /* Ensure that we can do comparisons as longs. */
struct fl_flow_mask_range {
@@ -328,21 +330,16 @@
schedule_work(&head->work);
}
-static bool fl_destroy(struct tcf_proto *tp, bool force)
+static void fl_destroy(struct tcf_proto *tp)
{
struct cls_fl_head *head = rtnl_dereference(tp->root);
struct cls_fl_filter *f, *next;
- if (!force && !list_empty(&head->filters))
- return false;
-
list_for_each_entry_safe(f, next, &head->filters, list)
__fl_delete(tp, f);
__module_get(THIS_MODULE);
call_rcu(&head->rcu, fl_destroy_rcu);
-
- return true;
}
static unsigned long fl_get(struct tcf_proto *tp, u32 handle)
@@ -423,6 +420,10 @@
[TCA_FLOWER_KEY_ARP_SHA_MASK] = { .len = ETH_ALEN },
[TCA_FLOWER_KEY_ARP_THA] = { .len = ETH_ALEN },
[TCA_FLOWER_KEY_ARP_THA_MASK] = { .len = ETH_ALEN },
+ [TCA_FLOWER_KEY_MPLS_TTL] = { .type = NLA_U8 },
+ [TCA_FLOWER_KEY_MPLS_BOS] = { .type = NLA_U8 },
+ [TCA_FLOWER_KEY_MPLS_TC] = { .type = NLA_U8 },
+ [TCA_FLOWER_KEY_MPLS_LABEL] = { .type = NLA_U32 },
};
static void fl_set_key_val(struct nlattr **tb,
@@ -438,6 +439,41 @@
memcpy(mask, nla_data(tb[mask_type]), len);
}
+static int fl_set_key_mpls(struct nlattr **tb,
+ struct flow_dissector_key_mpls *key_val,
+ struct flow_dissector_key_mpls *key_mask)
+{
+ if (tb[TCA_FLOWER_KEY_MPLS_TTL]) {
+ key_val->mpls_ttl = nla_get_u8(tb[TCA_FLOWER_KEY_MPLS_TTL]);
+ key_mask->mpls_ttl = MPLS_TTL_MASK;
+ }
+ if (tb[TCA_FLOWER_KEY_MPLS_BOS]) {
+ u8 bos = nla_get_u8(tb[TCA_FLOWER_KEY_MPLS_BOS]);
+
+ if (bos & ~MPLS_BOS_MASK)
+ return -EINVAL;
+ key_val->mpls_bos = bos;
+ key_mask->mpls_bos = MPLS_BOS_MASK;
+ }
+ if (tb[TCA_FLOWER_KEY_MPLS_TC]) {
+ u8 tc = nla_get_u8(tb[TCA_FLOWER_KEY_MPLS_TC]);
+
+ if (tc & ~MPLS_TC_MASK)
+ return -EINVAL;
+ key_val->mpls_tc = tc;
+ key_mask->mpls_tc = MPLS_TC_MASK;
+ }
+ if (tb[TCA_FLOWER_KEY_MPLS_LABEL]) {
+ u32 label = nla_get_u32(tb[TCA_FLOWER_KEY_MPLS_LABEL]);
+
+ if (label & ~MPLS_LABEL_MASK)
+ return -EINVAL;
+ key_val->mpls_label = label;
+ key_mask->mpls_label = MPLS_LABEL_MASK;
+ }
+ return 0;
+}
+
static void fl_set_key_vlan(struct nlattr **tb,
struct flow_dissector_key_vlan *key_val,
struct flow_dissector_key_vlan *key_mask)
@@ -594,6 +630,11 @@
&mask->icmp.code,
TCA_FLOWER_KEY_ICMPV6_CODE_MASK,
sizeof(key->icmp.code));
+ } else if (key->basic.n_proto == htons(ETH_P_MPLS_UC) ||
+ key->basic.n_proto == htons(ETH_P_MPLS_MC)) {
+ ret = fl_set_key_mpls(tb, &key->mpls, &mask->mpls);
+ if (ret)
+ return ret;
} else if (key->basic.n_proto == htons(ETH_P_ARP) ||
key->basic.n_proto == htons(ETH_P_RARP)) {
fl_set_key_val(tb, &key->arp.sip, TCA_FLOWER_KEY_ARP_SIP,
@@ -730,6 +771,8 @@
FL_KEY_SET_IF_MASKED(&mask->key, keys, cnt,
FLOW_DISSECTOR_KEY_ARP, arp);
FL_KEY_SET_IF_MASKED(&mask->key, keys, cnt,
+ FLOW_DISSECTOR_KEY_MPLS, mpls);
+ FL_KEY_SET_IF_MASKED(&mask->key, keys, cnt,
FLOW_DISSECTOR_KEY_VLAN, vlan);
FL_KEY_SET_IF_MASKED(&mask->key, keys, cnt,
FLOW_DISSECTOR_KEY_ENC_KEYID, enc_key_id);
@@ -848,7 +891,8 @@
if (!tb)
return -ENOBUFS;
- err = nla_parse_nested(tb, TCA_FLOWER_MAX, tca[TCA_OPTIONS], fl_policy);
+ err = nla_parse_nested(tb, TCA_FLOWER_MAX, tca[TCA_OPTIONS],
+ fl_policy, NULL);
if (err < 0)
goto errout_tb;
@@ -946,7 +990,7 @@
return err;
}
-static int fl_delete(struct tcf_proto *tp, unsigned long arg)
+static int fl_delete(struct tcf_proto *tp, unsigned long arg, bool *last)
{
struct cls_fl_head *head = rtnl_dereference(tp->root);
struct cls_fl_filter *f = (struct cls_fl_filter *) arg;
@@ -955,6 +999,7 @@
rhashtable_remove_fast(&head->ht, &f->ht_node,
head->ht_params);
__fl_delete(tp, f);
+ *last = list_empty(&head->filters);
return 0;
}
@@ -994,6 +1039,41 @@
return 0;
}
+static int fl_dump_key_mpls(struct sk_buff *skb,
+ struct flow_dissector_key_mpls *mpls_key,
+ struct flow_dissector_key_mpls *mpls_mask)
+{
+ int err;
+
+ if (!memchr_inv(mpls_mask, 0, sizeof(*mpls_mask)))
+ return 0;
+ if (mpls_mask->mpls_ttl) {
+ err = nla_put_u8(skb, TCA_FLOWER_KEY_MPLS_TTL,
+ mpls_key->mpls_ttl);
+ if (err)
+ return err;
+ }
+ if (mpls_mask->mpls_tc) {
+ err = nla_put_u8(skb, TCA_FLOWER_KEY_MPLS_TC,
+ mpls_key->mpls_tc);
+ if (err)
+ return err;
+ }
+ if (mpls_mask->mpls_label) {
+ err = nla_put_u32(skb, TCA_FLOWER_KEY_MPLS_LABEL,
+ mpls_key->mpls_label);
+ if (err)
+ return err;
+ }
+ if (mpls_mask->mpls_bos) {
+ err = nla_put_u8(skb, TCA_FLOWER_KEY_MPLS_BOS,
+ mpls_key->mpls_bos);
+ if (err)
+ return err;
+ }
+ return 0;
+}
+
static int fl_dump_key_vlan(struct sk_buff *skb,
struct flow_dissector_key_vlan *vlan_key,
struct flow_dissector_key_vlan *vlan_mask)
@@ -1099,6 +1179,9 @@
sizeof(key->basic.n_proto)))
goto nla_put_failure;
+ if (fl_dump_key_mpls(skb, &key->mpls, &mask->mpls))
+ goto nla_put_failure;
+
if (fl_dump_key_vlan(skb, &key->vlan, &mask->vlan))
goto nla_put_failure;
diff --git a/net/sched/cls_fw.c b/net/sched/cls_fw.c
index 9dc63d5..d388536 100644
--- a/net/sched/cls_fw.c
+++ b/net/sched/cls_fw.c
@@ -127,20 +127,14 @@
kfree(f);
}
-static bool fw_destroy(struct tcf_proto *tp, bool force)
+static void fw_destroy(struct tcf_proto *tp)
{
struct fw_head *head = rtnl_dereference(tp->root);
struct fw_filter *f;
int h;
if (head == NULL)
- return true;
-
- if (!force) {
- for (h = 0; h < HTSIZE; h++)
- if (rcu_access_pointer(head->ht[h]))
- return false;
- }
+ return;
for (h = 0; h < HTSIZE; h++) {
while ((f = rtnl_dereference(head->ht[h])) != NULL) {
@@ -150,17 +144,17 @@
call_rcu(&f->rcu, fw_delete_filter);
}
}
- RCU_INIT_POINTER(tp->root, NULL);
kfree_rcu(head, rcu);
- return true;
}
-static int fw_delete(struct tcf_proto *tp, unsigned long arg)
+static int fw_delete(struct tcf_proto *tp, unsigned long arg, bool *last)
{
struct fw_head *head = rtnl_dereference(tp->root);
struct fw_filter *f = (struct fw_filter *)arg;
struct fw_filter __rcu **fp;
struct fw_filter *pfp;
+ int ret = -EINVAL;
+ int h;
if (head == NULL || f == NULL)
goto out;
@@ -173,11 +167,21 @@
RCU_INIT_POINTER(*fp, rtnl_dereference(f->next));
tcf_unbind_filter(tp, &f->res);
call_rcu(&f->rcu, fw_delete_filter);
- return 0;
+ ret = 0;
+ break;
}
}
+
+ *last = true;
+ for (h = 0; h < HTSIZE; h++) {
+ if (rcu_access_pointer(head->ht[h])) {
+ *last = false;
+ break;
+ }
+ }
+
out:
- return -EINVAL;
+ return ret;
}
static const struct nla_policy fw_policy[TCA_FW_MAX + 1] = {
@@ -250,7 +254,7 @@
if (!opt)
return handle ? -EINVAL : 0; /* Succeed if it is old method. */
- err = nla_parse_nested(tb, TCA_FW_MAX, opt, fw_policy);
+ err = nla_parse_nested(tb, TCA_FW_MAX, opt, fw_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/cls_matchall.c b/net/sched/cls_matchall.c
index 224eb2c..2efb36c 100644
--- a/net/sched/cls_matchall.c
+++ b/net/sched/cls_matchall.c
@@ -90,19 +90,18 @@
&offload);
}
-static bool mall_destroy(struct tcf_proto *tp, bool force)
+static void mall_destroy(struct tcf_proto *tp)
{
struct cls_mall_head *head = rtnl_dereference(tp->root);
struct net_device *dev = tp->q->dev_queue->dev;
if (!head)
- return true;
+ return;
if (tc_should_offload(dev, tp, head->flags))
mall_destroy_hw_filter(tp, head, (unsigned long) head);
call_rcu(&head->rcu, mall_destroy_rcu);
- return true;
}
static unsigned long mall_get(struct tcf_proto *tp, u32 handle)
@@ -161,8 +160,8 @@
if (head)
return -EEXIST;
- err = nla_parse_nested(tb, TCA_MATCHALL_MAX,
- tca[TCA_OPTIONS], mall_policy);
+ err = nla_parse_nested(tb, TCA_MATCHALL_MAX, tca[TCA_OPTIONS],
+ mall_policy, NULL);
if (err < 0)
return err;
@@ -216,7 +215,7 @@
return err;
}
-static int mall_delete(struct tcf_proto *tp, unsigned long arg)
+static int mall_delete(struct tcf_proto *tp, unsigned long arg, bool *last)
{
return -EOPNOTSUPP;
}
diff --git a/net/sched/cls_route.c b/net/sched/cls_route.c
index 455fc8f..d63d550 100644
--- a/net/sched/cls_route.c
+++ b/net/sched/cls_route.c
@@ -140,8 +140,6 @@
goto failure;
id = dst->tclassid;
- if (head == NULL)
- goto old_method;
iif = inet_iif(skb);
@@ -194,15 +192,6 @@
route4_set_fastmap(head, id, iif, ROUTE4_FAILURE);
failure:
return -1;
-
-old_method:
- if (id && (TC_H_MAJ(id) == 0 ||
- !(TC_H_MAJ(id^tp->q->handle)))) {
- res->classid = id;
- res->class = 0;
- return 0;
- }
- return -1;
}
static inline u32 to_hash(u32 id)
@@ -234,9 +223,6 @@
struct route4_filter *f;
unsigned int h1, h2;
- if (!head)
- return 0;
-
h1 = to_hash(handle);
if (h1 > 256)
return 0;
@@ -276,20 +262,13 @@
kfree(f);
}
-static bool route4_destroy(struct tcf_proto *tp, bool force)
+static void route4_destroy(struct tcf_proto *tp)
{
struct route4_head *head = rtnl_dereference(tp->root);
int h1, h2;
if (head == NULL)
- return true;
-
- if (!force) {
- for (h1 = 0; h1 <= 256; h1++) {
- if (rcu_access_pointer(head->table[h1]))
- return false;
- }
- }
+ return;
for (h1 = 0; h1 <= 256; h1++) {
struct route4_bucket *b;
@@ -312,12 +291,10 @@
kfree_rcu(b, rcu);
}
}
- RCU_INIT_POINTER(tp->root, NULL);
kfree_rcu(head, rcu);
- return true;
}
-static int route4_delete(struct tcf_proto *tp, unsigned long arg)
+static int route4_delete(struct tcf_proto *tp, unsigned long arg, bool *last)
{
struct route4_head *head = rtnl_dereference(tp->root);
struct route4_filter *f = (struct route4_filter *)arg;
@@ -325,7 +302,7 @@
struct route4_filter *nf;
struct route4_bucket *b;
unsigned int h = 0;
- int i;
+ int i, h1;
if (!head || !f)
return -EINVAL;
@@ -356,16 +333,25 @@
rt = rtnl_dereference(b->ht[i]);
if (rt)
- return 0;
+ goto out;
}
/* OK, session has no flows */
RCU_INIT_POINTER(head->table[to_hash(h)], NULL);
kfree_rcu(b, rcu);
-
- return 0;
+ break;
}
}
+
+out:
+ *last = true;
+ for (h1 = 0; h1 <= 256; h1++) {
+ if (rcu_access_pointer(head->table[h1])) {
+ *last = false;
+ break;
+ }
+ }
+
return 0;
}
@@ -489,7 +475,7 @@
if (opt == NULL)
return handle ? -EINVAL : 0;
- err = nla_parse_nested(tb, TCA_ROUTE4_MAX, opt, route4_policy);
+ err = nla_parse_nested(tb, TCA_ROUTE4_MAX, opt, route4_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/cls_rsvp.h b/net/sched/cls_rsvp.h
index 322438f..0d9d077 100644
--- a/net/sched/cls_rsvp.h
+++ b/net/sched/cls_rsvp.h
@@ -152,8 +152,6 @@
return -1;
nhptr = ip_hdr(skb);
#endif
- if (unlikely(!head))
- return -1;
restart:
#if RSVP_DST_LEN == 4
@@ -302,22 +300,13 @@
call_rcu(&f->rcu, rsvp_delete_filter_rcu);
}
-static bool rsvp_destroy(struct tcf_proto *tp, bool force)
+static void rsvp_destroy(struct tcf_proto *tp)
{
struct rsvp_head *data = rtnl_dereference(tp->root);
int h1, h2;
if (data == NULL)
- return true;
-
- if (!force) {
- for (h1 = 0; h1 < 256; h1++) {
- if (rcu_access_pointer(data->ht[h1]))
- return false;
- }
- }
-
- RCU_INIT_POINTER(tp->root, NULL);
+ return;
for (h1 = 0; h1 < 256; h1++) {
struct rsvp_session *s;
@@ -337,10 +326,9 @@
}
}
kfree_rcu(data, rcu);
- return true;
}
-static int rsvp_delete(struct tcf_proto *tp, unsigned long arg)
+static int rsvp_delete(struct tcf_proto *tp, unsigned long arg, bool *last)
{
struct rsvp_head *head = rtnl_dereference(tp->root);
struct rsvp_filter *nfp, *f = (struct rsvp_filter *)arg;
@@ -348,7 +336,7 @@
unsigned int h = f->handle;
struct rsvp_session __rcu **sp;
struct rsvp_session *nsp, *s = f->sess;
- int i;
+ int i, h1;
fp = &s->ht[(h >> 8) & 0xFF];
for (nfp = rtnl_dereference(*fp); nfp;
@@ -361,7 +349,7 @@
for (i = 0; i <= 16; i++)
if (s->ht[i])
- return 0;
+ goto out;
/* OK, session has no flows */
sp = &head->ht[h & 0xFF];
@@ -370,13 +358,23 @@
if (nsp == s) {
RCU_INIT_POINTER(*sp, s->next);
kfree_rcu(s, rcu);
- return 0;
+ goto out;
}
}
- return 0;
+ break;
}
}
+
+out:
+ *last = true;
+ for (h1 = 0; h1 < 256; h1++) {
+ if (rcu_access_pointer(head->ht[h1])) {
+ *last = false;
+ break;
+ }
+ }
+
return 0;
}
@@ -484,7 +482,7 @@
if (opt == NULL)
return handle ? -EINVAL : 0;
- err = nla_parse_nested(tb, TCA_RSVP_MAX, opt, rsvp_policy);
+ err = nla_parse_nested(tb, TCA_RSVP_MAX, opt, rsvp_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/cls_tcindex.c b/net/sched/cls_tcindex.c
index 0751245..8a8a583 100644
--- a/net/sched/cls_tcindex.c
+++ b/net/sched/cls_tcindex.c
@@ -150,7 +150,7 @@
kfree(f);
}
-static int tcindex_delete(struct tcf_proto *tp, unsigned long arg)
+static int tcindex_delete(struct tcf_proto *tp, unsigned long arg, bool *last)
{
struct tcindex_data *p = rtnl_dereference(tp->root);
struct tcindex_filter_result *r = (struct tcindex_filter_result *) arg;
@@ -186,6 +186,8 @@
call_rcu(&f->rcu, tcindex_destroy_fexts);
else
call_rcu(&r->rcu, tcindex_destroy_rexts);
+
+ *last = false;
return 0;
}
@@ -193,7 +195,9 @@
unsigned long arg,
struct tcf_walker *walker)
{
- return tcindex_delete(tp, arg);
+ bool last;
+
+ return tcindex_delete(tp, arg, &last);
}
static void __tcindex_destroy(struct rcu_head *head)
@@ -482,7 +486,7 @@
if (!opt)
return 0;
- err = nla_parse_nested(tb, TCA_TCINDEX_MAX, opt, tcindex_policy);
+ err = nla_parse_nested(tb, TCA_TCINDEX_MAX, opt, tcindex_policy, NULL);
if (err < 0)
return err;
@@ -529,14 +533,11 @@
}
}
-static bool tcindex_destroy(struct tcf_proto *tp, bool force)
+static void tcindex_destroy(struct tcf_proto *tp)
{
struct tcindex_data *p = rtnl_dereference(tp->root);
struct tcf_walker walker;
- if (!force)
- return false;
-
pr_debug("tcindex_destroy(tp %p),p %p\n", tp, p);
walker.count = 0;
walker.skip = 0;
@@ -544,7 +545,6 @@
tcindex_walk(tp, &walker);
call_rcu(&p->rcu, __tcindex_destroy);
- return true;
}
diff --git a/net/sched/cls_u32.c b/net/sched/cls_u32.c
index 4dbe0c6..d20e72a 100644
--- a/net/sched/cls_u32.c
+++ b/net/sched/cls_u32.c
@@ -585,37 +585,13 @@
return true;
}
-static bool u32_destroy(struct tcf_proto *tp, bool force)
+static void u32_destroy(struct tcf_proto *tp)
{
struct tc_u_common *tp_c = tp->data;
struct tc_u_hnode *root_ht = rtnl_dereference(tp->root);
WARN_ON(root_ht == NULL);
- if (!force) {
- if (root_ht) {
- if (root_ht->refcnt > 1)
- return false;
- if (root_ht->refcnt == 1) {
- if (!ht_empty(root_ht))
- return false;
- }
- }
-
- if (tp_c->refcnt > 1)
- return false;
-
- if (tp_c->refcnt == 1) {
- struct tc_u_hnode *ht;
-
- for (ht = rtnl_dereference(tp_c->hlist);
- ht;
- ht = rtnl_dereference(ht->next))
- if (!ht_empty(ht))
- return false;
- }
- }
-
if (root_ht && --root_ht->refcnt == 0)
u32_destroy_hnode(tp, root_ht);
@@ -640,20 +616,22 @@
}
tp->data = NULL;
- return true;
}
-static int u32_delete(struct tcf_proto *tp, unsigned long arg)
+static int u32_delete(struct tcf_proto *tp, unsigned long arg, bool *last)
{
struct tc_u_hnode *ht = (struct tc_u_hnode *)arg;
struct tc_u_hnode *root_ht = rtnl_dereference(tp->root);
+ struct tc_u_common *tp_c = tp->data;
+ int ret = 0;
if (ht == NULL)
- return 0;
+ goto out;
if (TC_U32_KEY(ht->handle)) {
u32_remove_hw_knode(tp, ht->handle);
- return u32_delete_key(tp, (struct tc_u_knode *)ht);
+ ret = u32_delete_key(tp, (struct tc_u_knode *)ht);
+ goto out;
}
if (root_ht == ht)
@@ -666,7 +644,40 @@
return -EBUSY;
}
- return 0;
+out:
+ *last = true;
+ if (root_ht) {
+ if (root_ht->refcnt > 1) {
+ *last = false;
+ goto ret;
+ }
+ if (root_ht->refcnt == 1) {
+ if (!ht_empty(root_ht)) {
+ *last = false;
+ goto ret;
+ }
+ }
+ }
+
+ if (tp_c->refcnt > 1) {
+ *last = false;
+ goto ret;
+ }
+
+ if (tp_c->refcnt == 1) {
+ struct tc_u_hnode *ht;
+
+ for (ht = rtnl_dereference(tp_c->hlist);
+ ht;
+ ht = rtnl_dereference(ht->next))
+ if (!ht_empty(ht)) {
+ *last = false;
+ break;
+ }
+ }
+
+ret:
+ return ret;
}
#define NR_U32_NODE (1<<12)
@@ -860,7 +871,7 @@
if (opt == NULL)
return handle ? -EINVAL : 0;
- err = nla_parse_nested(tb, TCA_U32_MAX, opt, u32_policy);
+ err = nla_parse_nested(tb, TCA_U32_MAX, opt, u32_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/em_meta.c b/net/sched/em_meta.c
index ae7e4f5..eb0e9ba 100644
--- a/net/sched/em_meta.c
+++ b/net/sched/em_meta.c
@@ -912,7 +912,7 @@
struct tcf_meta_hdr *hdr;
struct meta_match *meta = NULL;
- err = nla_parse(tb, TCA_EM_META_MAX, data, len, meta_policy);
+ err = nla_parse(tb, TCA_EM_META_MAX, data, len, meta_policy, NULL);
if (err < 0)
goto errout;
diff --git a/net/sched/ematch.c b/net/sched/ematch.c
index fbb7ebf..03b677b 100644
--- a/net/sched/ematch.c
+++ b/net/sched/ematch.c
@@ -314,7 +314,7 @@
if (!nla)
return 0;
- err = nla_parse_nested(tb, TCA_EMATCH_TREE_MAX, nla, em_policy);
+ err = nla_parse_nested(tb, TCA_EMATCH_TREE_MAX, nla, em_policy, NULL);
if (err < 0)
goto errout;
diff --git a/net/sched/sch_api.c b/net/sched/sch_api.c
index bcf49cd..bbe57d5 100644
--- a/net/sched/sch_api.c
+++ b/net/sched/sch_api.c
@@ -251,6 +251,15 @@
return ops ? 0 : -ENOENT;
}
+#ifdef CONFIG_NET_SCH_DEFAULT
+/* Set default value from kernel config */
+static int __init sch_default_qdisc(void)
+{
+ return qdisc_set_default(CONFIG_DEFAULT_NET_SCH);
+}
+late_initcall(sch_default_qdisc);
+#endif
+
/* We know handle. Find qdisc among all qdisc's attached to device
* (root qdisc, all its children, children of children etc.)
* Note: caller either uses rtnl or rcu_read_lock()
@@ -274,7 +283,7 @@
return NULL;
}
-void qdisc_hash_add(struct Qdisc *q)
+void qdisc_hash_add(struct Qdisc *q, bool invisible)
{
if ((q->parent != TC_H_ROOT) && !(q->flags & TCQ_F_INGRESS)) {
struct Qdisc *root = qdisc_dev(q)->qdisc;
@@ -282,6 +291,8 @@
WARN_ON_ONCE(root == &noop_qdisc);
ASSERT_RTNL();
hash_add_rcu(qdisc_dev(q)->qdisc_hash, &q->hash, q->handle);
+ if (invisible)
+ q->flags |= TCQ_F_INVISIBLE;
}
}
EXPORT_SYMBOL(qdisc_hash_add);
@@ -455,7 +466,7 @@
u16 *tab = NULL;
int err;
- err = nla_parse_nested(tb, TCA_STAB_MAX, opt, stab_policy);
+ err = nla_parse_nested(tb, TCA_STAB_MAX, opt, stab_policy, NULL);
if (err < 0)
return ERR_PTR(err);
if (!tb[TCA_STAB_BASE])
@@ -1003,7 +1014,7 @@
goto err_out4;
}
- qdisc_hash_add(sch);
+ qdisc_hash_add(sch, false);
return sch;
}
@@ -1114,7 +1125,8 @@
* Delete/get qdisc.
*/
-static int tc_get_qdisc(struct sk_buff *skb, struct nlmsghdr *n)
+static int tc_get_qdisc(struct sk_buff *skb, struct nlmsghdr *n,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct tcmsg *tcm = nlmsg_data(n);
@@ -1129,7 +1141,7 @@
!netlink_ns_capable(skb, net->user_ns, CAP_NET_ADMIN))
return -EPERM;
- err = nlmsg_parse(n, sizeof(*tcm), tca, TCA_MAX, NULL);
+ err = nlmsg_parse(n, sizeof(*tcm), tca, TCA_MAX, NULL, extack);
if (err < 0)
return err;
@@ -1183,7 +1195,8 @@
* Create/change qdisc.
*/
-static int tc_modify_qdisc(struct sk_buff *skb, struct nlmsghdr *n)
+static int tc_modify_qdisc(struct sk_buff *skb, struct nlmsghdr *n,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct tcmsg *tcm;
@@ -1198,7 +1211,7 @@
replay:
/* Reinit, just in case something touches this. */
- err = nlmsg_parse(n, sizeof(*tcm), tca, TCA_MAX, NULL);
+ err = nlmsg_parse(n, sizeof(*tcm), tca, TCA_MAX, NULL, extack);
if (err < 0)
return err;
@@ -1401,9 +1414,14 @@
return -1;
}
-static bool tc_qdisc_dump_ignore(struct Qdisc *q)
+static bool tc_qdisc_dump_ignore(struct Qdisc *q, bool dump_invisible)
{
- return (q->flags & TCQ_F_BUILTIN) ? true : false;
+ if (q->flags & TCQ_F_BUILTIN)
+ return true;
+ if ((q->flags & TCQ_F_INVISIBLE) && !dump_invisible)
+ return true;
+
+ return false;
}
static int qdisc_notify(struct net *net, struct sk_buff *oskb,
@@ -1417,12 +1435,12 @@
if (!skb)
return -ENOBUFS;
- if (old && !tc_qdisc_dump_ignore(old)) {
+ if (old && !tc_qdisc_dump_ignore(old, false)) {
if (tc_fill_qdisc(skb, old, clid, portid, n->nlmsg_seq,
0, RTM_DELQDISC) < 0)
goto err_out;
}
- if (new && !tc_qdisc_dump_ignore(new)) {
+ if (new && !tc_qdisc_dump_ignore(new, false)) {
if (tc_fill_qdisc(skb, new, clid, portid, n->nlmsg_seq,
old ? NLM_F_REPLACE : 0, RTM_NEWQDISC) < 0)
goto err_out;
@@ -1439,7 +1457,8 @@
static int tc_dump_qdisc_root(struct Qdisc *root, struct sk_buff *skb,
struct netlink_callback *cb,
- int *q_idx_p, int s_q_idx, bool recur)
+ int *q_idx_p, int s_q_idx, bool recur,
+ bool dump_invisible)
{
int ret = 0, q_idx = *q_idx_p;
struct Qdisc *q;
@@ -1452,7 +1471,7 @@
if (q_idx < s_q_idx) {
q_idx++;
} else {
- if (!tc_qdisc_dump_ignore(q) &&
+ if (!tc_qdisc_dump_ignore(q, dump_invisible) &&
tc_fill_qdisc(skb, q, q->parent, NETLINK_CB(cb->skb).portid,
cb->nlh->nlmsg_seq, NLM_F_MULTI,
RTM_NEWQDISC) <= 0)
@@ -1474,7 +1493,7 @@
q_idx++;
continue;
}
- if (!tc_qdisc_dump_ignore(q) &&
+ if (!tc_qdisc_dump_ignore(q, dump_invisible) &&
tc_fill_qdisc(skb, q, q->parent, NETLINK_CB(cb->skb).portid,
cb->nlh->nlmsg_seq, NLM_F_MULTI,
RTM_NEWQDISC) <= 0)
@@ -1496,12 +1515,21 @@
int idx, q_idx;
int s_idx, s_q_idx;
struct net_device *dev;
+ const struct nlmsghdr *nlh = cb->nlh;
+ struct tcmsg *tcm = nlmsg_data(nlh);
+ struct nlattr *tca[TCA_MAX + 1];
+ int err;
s_idx = cb->args[0];
s_q_idx = q_idx = cb->args[1];
idx = 0;
ASSERT_RTNL();
+
+ err = nlmsg_parse(nlh, sizeof(*tcm), tca, TCA_MAX, NULL, NULL);
+ if (err < 0)
+ return err;
+
for_each_netdev(net, dev) {
struct netdev_queue *dev_queue;
@@ -1512,13 +1540,14 @@
q_idx = 0;
if (tc_dump_qdisc_root(dev->qdisc, skb, cb, &q_idx, s_q_idx,
- true) < 0)
+ true, tca[TCA_DUMP_INVISIBLE]) < 0)
goto done;
dev_queue = dev_ingress_queue(dev);
if (dev_queue &&
tc_dump_qdisc_root(dev_queue->qdisc_sleeping, skb, cb,
- &q_idx, s_q_idx, false) < 0)
+ &q_idx, s_q_idx, false,
+ tca[TCA_DUMP_INVISIBLE]) < 0)
goto done;
cont:
@@ -1540,7 +1569,8 @@
-static int tc_ctl_tclass(struct sk_buff *skb, struct nlmsghdr *n)
+static int tc_ctl_tclass(struct sk_buff *skb, struct nlmsghdr *n,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct tcmsg *tcm = nlmsg_data(n);
@@ -1559,7 +1589,7 @@
!netlink_ns_capable(skb, net->user_ns, CAP_NET_ADMIN))
return -EPERM;
- err = nlmsg_parse(n, sizeof(*tcm), tca, TCA_MAX, NULL);
+ err = nlmsg_parse(n, sizeof(*tcm), tca, TCA_MAX, NULL, extack);
if (err < 0)
return err;
@@ -1762,7 +1792,7 @@
{
struct qdisc_dump_args arg;
- if (tc_qdisc_dump_ignore(q) ||
+ if (tc_qdisc_dump_ignore(q, false) ||
*t_p < s_t || !q->ops->cl_ops ||
(tcm->tcm_parent &&
TC_H_MAJ(tcm->tcm_parent) != q->handle)) {
diff --git a/net/sched/sch_atm.c b/net/sched/sch_atm.c
index 2209c2d..40cbcee 100644
--- a/net/sched/sch_atm.c
+++ b/net/sched/sch_atm.c
@@ -214,7 +214,7 @@
if (opt == NULL)
return -EINVAL;
- error = nla_parse_nested(tb, TCA_ATM_MAX, opt, atm_policy);
+ error = nla_parse_nested(tb, TCA_ATM_MAX, opt, atm_policy, NULL);
if (error < 0)
return error;
diff --git a/net/sched/sch_cbq.c b/net/sched/sch_cbq.c
index d6ca18dc..7415859 100644
--- a/net/sched/sch_cbq.c
+++ b/net/sched/sch_cbq.c
@@ -1137,7 +1137,7 @@
struct tc_ratespec *r;
int err;
- err = nla_parse_nested(tb, TCA_CBQ_MAX, opt, cbq_policy);
+ err = nla_parse_nested(tb, TCA_CBQ_MAX, opt, cbq_policy, NULL);
if (err < 0)
return err;
@@ -1161,6 +1161,8 @@
sch->handle);
if (!q->link.q)
q->link.q = &noop_qdisc;
+ else
+ qdisc_hash_add(q->link.q, true);
q->link.priority = TC_CBQ_MAXPRIO - 1;
q->link.priority2 = TC_CBQ_MAXPRIO - 1;
@@ -1472,7 +1474,7 @@
if (opt == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_CBQ_MAX, opt, cbq_policy);
+ err = nla_parse_nested(tb, TCA_CBQ_MAX, opt, cbq_policy, NULL);
if (err < 0)
return err;
@@ -1600,6 +1602,9 @@
cl->q = qdisc_create_dflt(sch->dev_queue, &pfifo_qdisc_ops, classid);
if (!cl->q)
cl->q = &noop_qdisc;
+ else
+ qdisc_hash_add(cl->q, true);
+
cl->common.classid = classid;
cl->tparent = parent;
cl->qdisc = sch;
diff --git a/net/sched/sch_choke.c b/net/sched/sch_choke.c
index 3b86a97..d00f4c7c2 100644
--- a/net/sched/sch_choke.c
+++ b/net/sched/sch_choke.c
@@ -58,7 +58,6 @@
/* Variables */
struct red_vars vars;
- struct tcf_proto __rcu *filter_list;
struct {
u32 prob_drop; /* Early probability drops */
u32 prob_mark; /* Early probability marks */
@@ -152,11 +151,6 @@
choke_skb_cb(skb)->classid = classid;
}
-static u16 choke_get_classid(const struct sk_buff *skb)
-{
- return choke_skb_cb(skb)->classid;
-}
-
/*
* Compare flow of two packets
* Returns true only if source and destination address and port match.
@@ -188,40 +182,6 @@
}
/*
- * Classify flow using either:
- * 1. pre-existing classification result in skb
- * 2. fast internal classification
- * 3. use TC filter based classification
- */
-static bool choke_classify(struct sk_buff *skb,
- struct Qdisc *sch, int *qerr)
-
-{
- struct choke_sched_data *q = qdisc_priv(sch);
- struct tcf_result res;
- struct tcf_proto *fl;
- int result;
-
- fl = rcu_dereference_bh(q->filter_list);
- result = tc_classify(skb, fl, &res, false);
- if (result >= 0) {
-#ifdef CONFIG_NET_CLS_ACT
- switch (result) {
- case TC_ACT_STOLEN:
- case TC_ACT_QUEUED:
- *qerr = NET_XMIT_SUCCESS | __NET_XMIT_STOLEN;
- case TC_ACT_SHOT:
- return false;
- }
-#endif
- choke_set_classid(skb, TC_H_MIN(res.classid));
- return true;
- }
-
- return false;
-}
-
-/*
* Select a packet at random from queue
* HACK: since queue can have holes from previous deletion; retry several
* times to find a random skb but then just give up and return the head
@@ -257,25 +217,15 @@
return false;
oskb = choke_peek_random(q, pidx);
- if (rcu_access_pointer(q->filter_list))
- return choke_get_classid(nskb) == choke_get_classid(oskb);
-
return choke_match_flow(oskb, nskb);
}
static int choke_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct sk_buff **to_free)
{
- int ret = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
struct choke_sched_data *q = qdisc_priv(sch);
const struct red_parms *p = &q->parms;
- if (rcu_access_pointer(q->filter_list)) {
- /* If using external classifiers, get result and record it. */
- if (!choke_classify(skb, sch, &ret))
- goto other_drop; /* Packet was eaten by filter */
- }
-
choke_skb_cb(skb)->keys_valid = 0;
/* Compute average queue usage (see RED) */
q->vars.qavg = red_calc_qavg(p, &q->vars, sch->q.qlen);
@@ -339,12 +289,6 @@
congestion_drop:
qdisc_drop(skb, sch, to_free);
return NET_XMIT_CN;
-
-other_drop:
- if (ret & __NET_XMIT_BYPASS)
- qdisc_qstats_drop(sch);
- __qdisc_drop(skb, to_free);
- return ret;
}
static struct sk_buff *choke_dequeue(struct Qdisc *sch)
@@ -413,7 +357,7 @@
if (opt == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_CHOKE_MAX, opt, choke_policy);
+ err = nla_parse_nested(tb, TCA_CHOKE_MAX, opt, choke_policy, NULL);
if (err < 0)
return err;
@@ -538,7 +482,6 @@
{
struct choke_sched_data *q = qdisc_priv(sch);
- tcf_destroy_chain(&q->filter_list);
choke_free(q->tab);
}
diff --git a/net/sched/sch_codel.c b/net/sched/sch_codel.c
index 5bfa79e..c518a1e 100644
--- a/net/sched/sch_codel.c
+++ b/net/sched/sch_codel.c
@@ -140,7 +140,7 @@
if (!opt)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_CODEL_MAX, opt, codel_policy);
+ err = nla_parse_nested(tb, TCA_CODEL_MAX, opt, codel_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/sch_drr.c b/net/sched/sch_drr.c
index bb4cbdf..58a8c32 100644
--- a/net/sched/sch_drr.c
+++ b/net/sched/sch_drr.c
@@ -76,7 +76,7 @@
if (!opt)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_DRR_MAX, opt, drr_policy);
+ err = nla_parse_nested(tb, TCA_DRR_MAX, opt, drr_policy, NULL);
if (err < 0)
return err;
@@ -117,6 +117,8 @@
&pfifo_qdisc_ops, classid);
if (cl->qdisc == NULL)
cl->qdisc = &noop_qdisc;
+ else
+ qdisc_hash_add(cl->qdisc, true);
if (tca[TCA_RATE]) {
err = gen_replace_estimator(&cl->bstats, NULL, &cl->rate_est,
diff --git a/net/sched/sch_dsmark.c b/net/sched/sch_dsmark.c
index 5334e30..1c0f877 100644
--- a/net/sched/sch_dsmark.c
+++ b/net/sched/sch_dsmark.c
@@ -129,7 +129,7 @@
if (!opt)
goto errout;
- err = nla_parse_nested(tb, TCA_DSMARK_MAX, opt, dsmark_policy);
+ err = nla_parse_nested(tb, TCA_DSMARK_MAX, opt, dsmark_policy, NULL);
if (err < 0)
goto errout;
@@ -342,7 +342,7 @@
if (!opt)
goto errout;
- err = nla_parse_nested(tb, TCA_DSMARK_MAX, opt, dsmark_policy);
+ err = nla_parse_nested(tb, TCA_DSMARK_MAX, opt, dsmark_policy, NULL);
if (err < 0)
goto errout;
@@ -374,6 +374,8 @@
p->q = qdisc_create_dflt(sch->dev_queue, &pfifo_qdisc_ops, sch->handle);
if (p->q == NULL)
p->q = &noop_qdisc;
+ else
+ qdisc_hash_add(p->q, true);
pr_debug("%s: qdisc %p\n", __func__, p->q);
diff --git a/net/sched/sch_fq.c b/net/sched/sch_fq.c
index a4f738a..da4f67b 100644
--- a/net/sched/sch_fq.c
+++ b/net/sched/sch_fq.c
@@ -698,7 +698,7 @@
if (!opt)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_FQ_MAX, opt, fq_policy);
+ err = nla_parse_nested(tb, TCA_FQ_MAX, opt, fq_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c
index 9f3a884..18bbb54 100644
--- a/net/sched/sch_fq_codel.c
+++ b/net/sched/sch_fq_codel.c
@@ -288,7 +288,6 @@
struct fq_codel_flow *flow;
struct list_head *head;
u32 prev_drop_count, prev_ecn_mark;
- unsigned int prev_backlog;
begin:
head = &q->new_flows;
@@ -307,7 +306,6 @@
prev_drop_count = q->cstats.drop_count;
prev_ecn_mark = q->cstats.ecn_mark;
- prev_backlog = sch->qstats.backlog;
skb = codel_dequeue(sch, &sch->qstats.backlog, &q->cparams,
&flow->cvars, &q->cstats, qdisc_pkt_len,
@@ -385,7 +383,8 @@
if (!opt)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_FQ_CODEL_MAX, opt, fq_codel_policy);
+ err = nla_parse_nested(tb, TCA_FQ_CODEL_MAX, opt, fq_codel_policy,
+ NULL);
if (err < 0)
return err;
if (tb[TCA_FQ_CODEL_FLOWS]) {
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index 1a2f9e9..52a2c55 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -795,7 +795,7 @@
}
#ifdef CONFIG_NET_SCHED
if (dev->qdisc != &noop_qdisc)
- qdisc_hash_add(dev->qdisc);
+ qdisc_hash_add(dev->qdisc, false);
#endif
}
diff --git a/net/sched/sch_gred.c b/net/sched/sch_gred.c
index c78a093..17c7130 100644
--- a/net/sched/sch_gred.c
+++ b/net/sched/sch_gred.c
@@ -401,7 +401,7 @@
if (opt == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_GRED_MAX, opt, gred_policy);
+ err = nla_parse_nested(tb, TCA_GRED_MAX, opt, gred_policy, NULL);
if (err < 0)
return err;
@@ -470,7 +470,7 @@
if (opt == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_GRED_MAX, opt, gred_policy);
+ err = nla_parse_nested(tb, TCA_GRED_MAX, opt, gred_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/sch_hfsc.c b/net/sched/sch_hfsc.c
index 3ffaa6f..5cb82f6 100644
--- a/net/sched/sch_hfsc.c
+++ b/net/sched/sch_hfsc.c
@@ -957,7 +957,7 @@
if (opt == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_HFSC_MAX, opt, hfsc_policy);
+ err = nla_parse_nested(tb, TCA_HFSC_MAX, opt, hfsc_policy, NULL);
if (err < 0)
return err;
@@ -1066,6 +1066,8 @@
&pfifo_qdisc_ops, classid);
if (cl->qdisc == NULL)
cl->qdisc = &noop_qdisc;
+ else
+ qdisc_hash_add(cl->qdisc, true);
INIT_LIST_HEAD(&cl->children);
cl->vt_tree = RB_ROOT;
cl->cf_tree = RB_ROOT;
@@ -1425,6 +1427,8 @@
sch->handle);
if (q->root.qdisc == NULL)
q->root.qdisc = &noop_qdisc;
+ else
+ qdisc_hash_add(q->root.qdisc, true);
INIT_LIST_HEAD(&q->root.children);
q->root.vt_tree = RB_ROOT;
q->root.cf_tree = RB_ROOT;
diff --git a/net/sched/sch_hhf.c b/net/sched/sch_hhf.c
index 2fae8b5..c19d346 100644
--- a/net/sched/sch_hhf.c
+++ b/net/sched/sch_hhf.c
@@ -529,7 +529,7 @@
if (!opt)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_HHF_MAX, opt, hhf_policy);
+ err = nla_parse_nested(tb, TCA_HHF_MAX, opt, hhf_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
index 4cd5fb1..570ef3b 100644
--- a/net/sched/sch_htb.c
+++ b/net/sched/sch_htb.c
@@ -1017,7 +1017,7 @@
if (!opt)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_HTB_MAX, opt, htb_policy);
+ err = nla_parse_nested(tb, TCA_HTB_MAX, opt, htb_policy, NULL);
if (err < 0)
return err;
@@ -1342,7 +1342,7 @@
if (!opt)
goto failure;
- err = nla_parse_nested(tb, TCA_HTB_MAX, opt, htb_policy);
+ err = nla_parse_nested(tb, TCA_HTB_MAX, opt, htb_policy, NULL);
if (err < 0)
goto failure;
@@ -1460,6 +1460,8 @@
qdisc_class_hash_insert(&q->clhash, &cl->common);
if (parent)
parent->children++;
+ if (cl->un.leaf.q != &noop_qdisc)
+ qdisc_hash_add(cl->un.leaf.q, true);
} else {
if (tca[TCA_RATE]) {
err = gen_replace_estimator(&cl->bstats, NULL,
diff --git a/net/sched/sch_mq.c b/net/sched/sch_mq.c
index 20b7f16..cadfdd4 100644
--- a/net/sched/sch_mq.c
+++ b/net/sched/sch_mq.c
@@ -84,7 +84,7 @@
qdisc_destroy(old);
#ifdef CONFIG_NET_SCHED
if (ntx < dev->real_num_tx_queues)
- qdisc_hash_add(qdisc);
+ qdisc_hash_add(qdisc, false);
#endif
}
diff --git a/net/sched/sch_mqprio.c b/net/sched/sch_mqprio.c
index 9226834..0a4cf27 100644
--- a/net/sched/sch_mqprio.c
+++ b/net/sched/sch_mqprio.c
@@ -21,14 +21,13 @@
struct mqprio_sched {
struct Qdisc **qdiscs;
- int hw_owned;
+ int hw_offload;
};
static void mqprio_destroy(struct Qdisc *sch)
{
struct net_device *dev = qdisc_dev(sch);
struct mqprio_sched *priv = qdisc_priv(sch);
- struct tc_to_netdev tc = {.type = TC_SETUP_MQPRIO};
unsigned int ntx;
if (priv->qdiscs) {
@@ -39,10 +38,15 @@
kfree(priv->qdiscs);
}
- if (priv->hw_owned && dev->netdev_ops->ndo_setup_tc)
+ if (priv->hw_offload && dev->netdev_ops->ndo_setup_tc) {
+ struct tc_mqprio_qopt offload = { 0 };
+ struct tc_to_netdev tc = { .type = TC_SETUP_MQPRIO,
+ { .mqprio = &offload } };
+
dev->netdev_ops->ndo_setup_tc(dev, sch->handle, 0, &tc);
- else
+ } else {
netdev_set_num_tc(dev, 0);
+ }
}
static int mqprio_parse_opt(struct net_device *dev, struct tc_mqprio_qopt *qopt)
@@ -59,15 +63,20 @@
return -EINVAL;
}
- /* net_device does not support requested operation */
- if (qopt->hw && !dev->netdev_ops->ndo_setup_tc)
- return -EINVAL;
+ /* Limit qopt->hw to maximum supported offload value. Drivers have
+ * the option of overriding this later if they don't support the a
+ * given offload type.
+ */
+ if (qopt->hw > TC_MQPRIO_HW_OFFLOAD_MAX)
+ qopt->hw = TC_MQPRIO_HW_OFFLOAD_MAX;
- /* if hw owned qcount and qoffset are taken from LLD so
- * no reason to verify them here
+ /* If hardware offload is requested we will leave it to the device
+ * to either populate the queue counts itself or to validate the
+ * provided queue counts. If ndo_setup_tc is not present then
+ * hardware doesn't support offload and we should return an error.
*/
if (qopt->hw)
- return 0;
+ return dev->netdev_ops->ndo_setup_tc ? 0 : -EINVAL;
for (i = 0; i < qopt->num_tc; i++) {
unsigned int last = qopt->offset[i] + qopt->count[i];
@@ -139,13 +148,15 @@
* supplied and verified mapping
*/
if (qopt->hw) {
- struct tc_to_netdev tc = {.type = TC_SETUP_MQPRIO,
- { .tc = qopt->num_tc }};
+ struct tc_mqprio_qopt offload = *qopt;
+ struct tc_to_netdev tc = { .type = TC_SETUP_MQPRIO,
+ { .mqprio = &offload } };
- priv->hw_owned = 1;
err = dev->netdev_ops->ndo_setup_tc(dev, sch->handle, 0, &tc);
if (err)
return err;
+
+ priv->hw_offload = offload.hw;
} else {
netdev_set_num_tc(dev, qopt->num_tc);
for (i = 0; i < qopt->num_tc; i++)
@@ -175,7 +186,7 @@
if (old)
qdisc_destroy(old);
if (ntx < dev->real_num_tx_queues)
- qdisc_hash_add(qdisc);
+ qdisc_hash_add(qdisc, false);
}
kfree(priv->qdiscs);
priv->qdiscs = NULL;
@@ -243,7 +254,7 @@
opt.num_tc = netdev_get_num_tc(dev);
memcpy(opt.prio_tc_map, dev->prio_tc_map, sizeof(opt.prio_tc_map));
- opt.hw = priv->hw_owned;
+ opt.hw = priv->hw_offload;
for (i = 0; i < netdev_get_num_tc(dev); i++) {
opt.count[i] = dev->tc_to_txq[i].count;
diff --git a/net/sched/sch_multiq.c b/net/sched/sch_multiq.c
index e7839a0..43a3a10 100644
--- a/net/sched/sch_multiq.c
+++ b/net/sched/sch_multiq.c
@@ -217,6 +217,8 @@
sch_tree_lock(sch);
old = q->queues[i];
q->queues[i] = child;
+ if (child != &noop_qdisc)
+ qdisc_hash_add(child, true);
if (old != &noop_qdisc) {
qdisc_tree_reduce_backlog(old,
diff --git a/net/sched/sch_netem.c b/net/sched/sch_netem.c
index c8bb62a..f0ce478 100644
--- a/net/sched/sch_netem.c
+++ b/net/sched/sch_netem.c
@@ -462,7 +462,7 @@
/* If a delay is expected, orphan the skb. (orphaning usually takes
* place at TX completion time, so _before_ the link transit delay)
*/
- if (q->latency || q->jitter)
+ if (q->latency || q->jitter || q->rate)
skb_orphan_partial(skb);
/*
@@ -530,21 +530,31 @@
now = psched_get_time();
if (q->rate) {
- struct sk_buff *last;
+ struct netem_skb_cb *last = NULL;
- if (sch->q.qlen)
- last = sch->q.tail;
- else
- last = netem_rb_to_skb(rb_last(&q->t_root));
+ if (sch->q.tail)
+ last = netem_skb_cb(sch->q.tail);
+ if (q->t_root.rb_node) {
+ struct sk_buff *t_skb;
+ struct netem_skb_cb *t_last;
+
+ t_skb = netem_rb_to_skb(rb_last(&q->t_root));
+ t_last = netem_skb_cb(t_skb);
+ if (!last ||
+ t_last->time_to_send > last->time_to_send) {
+ last = t_last;
+ }
+ }
+
if (last) {
/*
* Last packet in queue is reference point (now),
* calculate this time bonus and subtract
* from delay.
*/
- delay -= netem_skb_cb(last)->time_to_send - now;
+ delay -= last->time_to_send - now;
delay = max_t(psched_tdiff_t, 0, delay);
- now = netem_skb_cb(last)->time_to_send;
+ now = last->time_to_send;
}
delay += packet_len_2_sched_time(qdisc_pkt_len(skb), q);
@@ -833,7 +843,7 @@
if (nested_len >= nla_attr_size(0))
return nla_parse(tb, maxtype, nla_data(nla) + NLA_ALIGN(len),
- nested_len, policy);
+ nested_len, policy, NULL);
memset(tb, 0, sizeof(struct nlattr *) * (maxtype + 1));
return 0;
diff --git a/net/sched/sch_pie.c b/net/sched/sch_pie.c
index 5c3a99d..6c2791d 100644
--- a/net/sched/sch_pie.c
+++ b/net/sched/sch_pie.c
@@ -190,7 +190,7 @@
if (!opt)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_PIE_MAX, opt, pie_policy);
+ err = nla_parse_nested(tb, TCA_PIE_MAX, opt, pie_policy, NULL);
if (err < 0)
return err;
diff --git a/net/sched/sch_prio.c b/net/sched/sch_prio.c
index d4d7db2..92c2e6d 100644
--- a/net/sched/sch_prio.c
+++ b/net/sched/sch_prio.c
@@ -192,8 +192,11 @@
qdisc_destroy(child);
}
- for (i = oldbands; i < q->bands; i++)
+ for (i = oldbands; i < q->bands; i++) {
q->queues[i] = queues[i];
+ if (q->queues[i] != &noop_qdisc)
+ qdisc_hash_add(q->queues[i], true);
+ }
sch_tree_unlock(sch);
return 0;
diff --git a/net/sched/sch_qfq.c b/net/sched/sch_qfq.c
index f9e712c..041eba3 100644
--- a/net/sched/sch_qfq.c
+++ b/net/sched/sch_qfq.c
@@ -418,7 +418,8 @@
return -EINVAL;
}
- err = nla_parse_nested(tb, TCA_QFQ_MAX, tca[TCA_OPTIONS], qfq_policy);
+ err = nla_parse_nested(tb, TCA_QFQ_MAX, tca[TCA_OPTIONS], qfq_policy,
+ NULL);
if (err < 0)
return err;
@@ -494,6 +495,8 @@
goto destroy_class;
}
+ if (cl->qdisc != &noop_qdisc)
+ qdisc_hash_add(cl->qdisc, true);
sch_tree_lock(sch);
qdisc_class_hash_insert(&q->clhash, &cl->common);
sch_tree_unlock(sch);
diff --git a/net/sched/sch_red.c b/net/sched/sch_red.c
index 249b2a1..11292ad 100644
--- a/net/sched/sch_red.c
+++ b/net/sched/sch_red.c
@@ -173,7 +173,7 @@
if (opt == NULL)
return -EINVAL;
- err = nla_parse_nested(tb, TCA_RED_MAX, opt, red_policy);
+ err = nla_parse_nested(tb, TCA_RED_MAX, opt, red_policy, NULL);
if (err < 0)
return err;
@@ -191,6 +191,8 @@
return PTR_ERR(child);
}
+ if (child != &noop_qdisc)
+ qdisc_hash_add(child, true);
sch_tree_lock(sch);
q->flags = ctl->flags;
q->limit = ctl->limit;
diff --git a/net/sched/sch_sfb.c b/net/sched/sch_sfb.c
index fe6963d..0f77727 100644
--- a/net/sched/sch_sfb.c
+++ b/net/sched/sch_sfb.c
@@ -495,7 +495,7 @@
int err;
if (opt) {
- err = nla_parse_nested(tb, TCA_SFB_MAX, opt, sfb_policy);
+ err = nla_parse_nested(tb, TCA_SFB_MAX, opt, sfb_policy, NULL);
if (err < 0)
return -EINVAL;
@@ -513,6 +513,8 @@
if (IS_ERR(child))
return PTR_ERR(child);
+ if (child != &noop_qdisc)
+ qdisc_hash_add(child, true);
sch_tree_lock(sch);
qdisc_tree_reduce_backlog(q->qdisc, q->qdisc->q.qlen,
diff --git a/net/sched/sch_sfq.c b/net/sched/sch_sfq.c
index 42e8c86..b00e02c 100644
--- a/net/sched/sch_sfq.c
+++ b/net/sched/sch_sfq.c
@@ -714,9 +714,8 @@
struct sfq_sched_data *q = qdisc_priv(sch);
int i;
- q->perturb_timer.function = sfq_perturbation;
- q->perturb_timer.data = (unsigned long)sch;
- init_timer_deferrable(&q->perturb_timer);
+ setup_deferrable_timer(&q->perturb_timer, sfq_perturbation,
+ (unsigned long)sch);
for (i = 0; i < SFQ_MAX_DEPTH + 1; i++) {
q->dep[i].next = i + SFQ_MAX_FLOWS;
diff --git a/net/sched/sch_tbf.c b/net/sched/sch_tbf.c
index 303355c..b2e4b6a 100644
--- a/net/sched/sch_tbf.c
+++ b/net/sched/sch_tbf.c
@@ -315,7 +315,7 @@
s64 buffer, mtu;
u64 rate64 = 0, prate64 = 0;
- err = nla_parse_nested(tb, TCA_TBF_MAX, opt, tbf_policy);
+ err = nla_parse_nested(tb, TCA_TBF_MAX, opt, tbf_policy, NULL);
if (err < 0)
return err;
@@ -396,6 +396,8 @@
q->qdisc->qstats.backlog);
qdisc_destroy(q->qdisc);
q->qdisc = child;
+ if (child != &noop_qdisc)
+ qdisc_hash_add(child, true);
}
q->limit = qopt->limit;
if (tb[TCA_TBF_PBURST])
diff --git a/net/sctp/chunk.c b/net/sctp/chunk.c
index e3621cb..697721a 100644
--- a/net/sctp/chunk.c
+++ b/net/sctp/chunk.c
@@ -306,14 +306,24 @@
if (SCTP_PR_TTL_ENABLED(chunk->sinfo.sinfo_flags) &&
time_after(jiffies, chunk->msg->expires_at)) {
- if (chunk->sent_count)
+ struct sctp_stream_out *streamout =
+ &chunk->asoc->stream->out[chunk->sinfo.sinfo_stream];
+
+ if (chunk->sent_count) {
chunk->asoc->abandoned_sent[SCTP_PR_INDEX(TTL)]++;
- else
+ streamout->abandoned_sent[SCTP_PR_INDEX(TTL)]++;
+ } else {
chunk->asoc->abandoned_unsent[SCTP_PR_INDEX(TTL)]++;
+ streamout->abandoned_unsent[SCTP_PR_INDEX(TTL)]++;
+ }
return 1;
} else if (SCTP_PR_RTX_ENABLED(chunk->sinfo.sinfo_flags) &&
chunk->sent_count > chunk->sinfo.sinfo_timetolive) {
+ struct sctp_stream_out *streamout =
+ &chunk->asoc->stream->out[chunk->sinfo.sinfo_stream];
+
chunk->asoc->abandoned_sent[SCTP_PR_INDEX(RTX)]++;
+ streamout->abandoned_sent[SCTP_PR_INDEX(RTX)]++;
return 1;
} else if (!SCTP_PR_POLICY(chunk->sinfo.sinfo_flags) &&
chunk->msg->expires_at &&
diff --git a/net/sctp/outqueue.c b/net/sctp/outqueue.c
index 8081476..fe4c3d46 100644
--- a/net/sctp/outqueue.c
+++ b/net/sctp/outqueue.c
@@ -353,6 +353,8 @@
struct sctp_chunk *chk, *temp;
list_for_each_entry_safe(chk, temp, queue, transmitted_list) {
+ struct sctp_stream_out *streamout;
+
if (!SCTP_PR_PRIO_ENABLED(chk->sinfo.sinfo_flags) ||
chk->sinfo.sinfo_timetolive <= sinfo->sinfo_timetolive)
continue;
@@ -361,8 +363,10 @@
sctp_insert_list(&asoc->outqueue.abandoned,
&chk->transmitted_list);
+ streamout = &asoc->stream->out[chk->sinfo.sinfo_stream];
asoc->sent_cnt_removable--;
asoc->abandoned_sent[SCTP_PR_INDEX(PRIO)]++;
+ streamout->abandoned_sent[SCTP_PR_INDEX(PRIO)]++;
if (!chk->tsn_gap_acked) {
if (chk->transport)
@@ -396,6 +400,12 @@
q->out_qlen -= chk->skb->len;
asoc->sent_cnt_removable--;
asoc->abandoned_unsent[SCTP_PR_INDEX(PRIO)]++;
+ if (chk->sinfo.sinfo_stream < asoc->stream->outcnt) {
+ struct sctp_stream_out *streamout =
+ &asoc->stream->out[chk->sinfo.sinfo_stream];
+
+ streamout->abandoned_unsent[SCTP_PR_INDEX(PRIO)]++;
+ }
msg_len -= SCTP_DATA_SNDSIZE(chk) +
sizeof(struct sk_buff) +
diff --git a/net/sctp/sm_make_chunk.c b/net/sctp/sm_make_chunk.c
index 118faff..8a08f13 100644
--- a/net/sctp/sm_make_chunk.c
+++ b/net/sctp/sm_make_chunk.c
@@ -1512,14 +1512,12 @@
struct iov_iter *from)
{
void *target;
- ssize_t copied;
/* Make room in chunk for data. */
target = skb_put(chunk->skb, len);
/* Copy data (whole iovec) into chunk */
- copied = copy_from_iter(target, len, from);
- if (copied != len)
+ if (!copy_from_iter_full(target, len, from))
return -EFAULT;
/* Adjust the chunk length field. */
diff --git a/net/sctp/sm_statefuns.c b/net/sctp/sm_statefuns.c
index 24c6ccc..4f5e6cf 100644
--- a/net/sctp/sm_statefuns.c
+++ b/net/sctp/sm_statefuns.c
@@ -3872,9 +3872,18 @@
else if (param.p->type == SCTP_PARAM_RESET_IN_REQUEST)
reply = sctp_process_strreset_inreq(
(struct sctp_association *)asoc, param, &ev);
- /* More handles for other types will be added here, by now it
- * just ignores other types.
- */
+ else if (param.p->type == SCTP_PARAM_RESET_TSN_REQUEST)
+ reply = sctp_process_strreset_tsnreq(
+ (struct sctp_association *)asoc, param, &ev);
+ else if (param.p->type == SCTP_PARAM_RESET_ADD_OUT_STREAMS)
+ reply = sctp_process_strreset_addstrm_out(
+ (struct sctp_association *)asoc, param, &ev);
+ else if (param.p->type == SCTP_PARAM_RESET_ADD_IN_STREAMS)
+ reply = sctp_process_strreset_addstrm_in(
+ (struct sctp_association *)asoc, param, &ev);
+ else if (param.p->type == SCTP_PARAM_RESET_RESPONSE)
+ reply = sctp_process_strreset_resp(
+ (struct sctp_association *)asoc, param, &ev);
if (ev)
sctp_add_cmd_sf(commands, SCTP_CMD_EVENT_ULP,
diff --git a/net/sctp/socket.c b/net/sctp/socket.c
index d9d4c92..f16c8d9 100644
--- a/net/sctp/socket.c
+++ b/net/sctp/socket.c
@@ -3758,6 +3758,39 @@
return retval;
}
+static int sctp_setsockopt_reconfig_supported(struct sock *sk,
+ char __user *optval,
+ unsigned int optlen)
+{
+ struct sctp_assoc_value params;
+ struct sctp_association *asoc;
+ int retval = -EINVAL;
+
+ if (optlen != sizeof(params))
+ goto out;
+
+ if (copy_from_user(¶ms, optval, optlen)) {
+ retval = -EFAULT;
+ goto out;
+ }
+
+ asoc = sctp_id2assoc(sk, params.assoc_id);
+ if (asoc) {
+ asoc->reconf_enable = !!params.assoc_value;
+ } else if (!params.assoc_id) {
+ struct sctp_sock *sp = sctp_sk(sk);
+
+ sp->ep->reconf_enable = !!params.assoc_value;
+ } else {
+ goto out;
+ }
+
+ retval = 0;
+
+out:
+ return retval;
+}
+
static int sctp_setsockopt_enable_strreset(struct sock *sk,
char __user *optval,
unsigned int optlen)
@@ -4038,6 +4071,9 @@
case SCTP_DEFAULT_PRINFO:
retval = sctp_setsockopt_default_prinfo(sk, optval, optlen);
break;
+ case SCTP_RECONFIG_SUPPORTED:
+ retval = sctp_setsockopt_reconfig_supported(sk, optval, optlen);
+ break;
case SCTP_ENABLE_STREAM_RESET:
retval = sctp_setsockopt_enable_strreset(sk, optval, optlen);
break;
@@ -6540,6 +6576,102 @@
return retval;
}
+static int sctp_getsockopt_pr_streamstatus(struct sock *sk, int len,
+ char __user *optval,
+ int __user *optlen)
+{
+ struct sctp_stream_out *streamout;
+ struct sctp_association *asoc;
+ struct sctp_prstatus params;
+ int retval = -EINVAL;
+ int policy;
+
+ if (len < sizeof(params))
+ goto out;
+
+ len = sizeof(params);
+ if (copy_from_user(¶ms, optval, len)) {
+ retval = -EFAULT;
+ goto out;
+ }
+
+ policy = params.sprstat_policy;
+ if (policy & ~SCTP_PR_SCTP_MASK)
+ goto out;
+
+ asoc = sctp_id2assoc(sk, params.sprstat_assoc_id);
+ if (!asoc || params.sprstat_sid >= asoc->stream->outcnt)
+ goto out;
+
+ streamout = &asoc->stream->out[params.sprstat_sid];
+ if (policy == SCTP_PR_SCTP_NONE) {
+ params.sprstat_abandoned_unsent = 0;
+ params.sprstat_abandoned_sent = 0;
+ for (policy = 0; policy <= SCTP_PR_INDEX(MAX); policy++) {
+ params.sprstat_abandoned_unsent +=
+ streamout->abandoned_unsent[policy];
+ params.sprstat_abandoned_sent +=
+ streamout->abandoned_sent[policy];
+ }
+ } else {
+ params.sprstat_abandoned_unsent =
+ streamout->abandoned_unsent[__SCTP_PR_INDEX(policy)];
+ params.sprstat_abandoned_sent =
+ streamout->abandoned_sent[__SCTP_PR_INDEX(policy)];
+ }
+
+ if (put_user(len, optlen) || copy_to_user(optval, ¶ms, len)) {
+ retval = -EFAULT;
+ goto out;
+ }
+
+ retval = 0;
+
+out:
+ return retval;
+}
+
+static int sctp_getsockopt_reconfig_supported(struct sock *sk, int len,
+ char __user *optval,
+ int __user *optlen)
+{
+ struct sctp_assoc_value params;
+ struct sctp_association *asoc;
+ int retval = -EFAULT;
+
+ if (len < sizeof(params)) {
+ retval = -EINVAL;
+ goto out;
+ }
+
+ len = sizeof(params);
+ if (copy_from_user(¶ms, optval, len))
+ goto out;
+
+ asoc = sctp_id2assoc(sk, params.assoc_id);
+ if (asoc) {
+ params.assoc_value = asoc->reconf_enable;
+ } else if (!params.assoc_id) {
+ struct sctp_sock *sp = sctp_sk(sk);
+
+ params.assoc_value = sp->ep->reconf_enable;
+ } else {
+ retval = -EINVAL;
+ goto out;
+ }
+
+ if (put_user(len, optlen))
+ goto out;
+
+ if (copy_to_user(optval, ¶ms, len))
+ goto out;
+
+ retval = 0;
+
+out:
+ return retval;
+}
+
static int sctp_getsockopt_enable_strreset(struct sock *sk, int len,
char __user *optval,
int __user *optlen)
@@ -6748,6 +6880,14 @@
retval = sctp_getsockopt_pr_assocstatus(sk, len, optval,
optlen);
break;
+ case SCTP_PR_STREAM_STATUS:
+ retval = sctp_getsockopt_pr_streamstatus(sk, len, optval,
+ optlen);
+ break;
+ case SCTP_RECONFIG_SUPPORTED:
+ retval = sctp_getsockopt_reconfig_supported(sk, len, optval,
+ optlen);
+ break;
case SCTP_ENABLE_STREAM_RESET:
retval = sctp_getsockopt_enable_strreset(sk, len, optval,
optlen);
@@ -7440,9 +7580,12 @@
if (sk->sk_shutdown & RCV_SHUTDOWN)
break;
- if (sk_can_busy_loop(sk) &&
- sk_busy_loop(sk, noblock))
- continue;
+ if (sk_can_busy_loop(sk)) {
+ sk_busy_loop(sk, noblock);
+
+ if (!skb_queue_empty(&sk->sk_receive_queue))
+ continue;
+ }
/* User doesn't want to wait. */
error = -EAGAIN;
diff --git a/net/sctp/stream.c b/net/sctp/stream.c
index bbed997..dda53a2 100644
--- a/net/sctp/stream.c
+++ b/net/sctp/stream.c
@@ -294,18 +294,6 @@
stream->out = streamout;
}
- if (in) {
- struct sctp_stream_in *streamin;
-
- streamin = krealloc(stream->in, incnt * sizeof(*streamin),
- GFP_KERNEL);
- if (!streamin)
- goto out;
-
- memset(streamin + stream->incnt, 0, in * sizeof(*streamin));
- stream->in = streamin;
- }
-
chunk = sctp_make_strreset_addstrm(asoc, out, in);
if (!chunk)
goto out;
@@ -330,13 +318,14 @@
}
static sctp_paramhdr_t *sctp_chunk_lookup_strreset_param(
- struct sctp_association *asoc, __u32 resp_seq)
+ struct sctp_association *asoc, __u32 resp_seq,
+ __be16 type)
{
struct sctp_chunk *chunk = asoc->strreset_chunk;
struct sctp_reconf_chunk *hdr;
union sctp_params param;
- if (ntohl(resp_seq) != asoc->strreset_outseq || !chunk)
+ if (!chunk)
return NULL;
hdr = (struct sctp_reconf_chunk *)chunk->chunk_hdr;
@@ -347,13 +336,21 @@
*/
struct sctp_strreset_tsnreq *req = param.v;
- if (req->request_seq == resp_seq)
+ if ((!resp_seq || req->request_seq == resp_seq) &&
+ (!type || type == req->param_hdr.type))
return param.v;
}
return NULL;
}
+static void sctp_update_strreset_result(struct sctp_association *asoc,
+ __u32 result)
+{
+ asoc->strreset_result[1] = asoc->strreset_result[0];
+ asoc->strreset_result[0] = result;
+}
+
struct sctp_chunk *sctp_process_strreset_outreq(
struct sctp_association *asoc,
union sctp_params param,
@@ -370,15 +367,19 @@
if (ntohl(outreq->send_reset_at_tsn) >
sctp_tsnmap_get_ctsn(&asoc->peer.tsn_map)) {
result = SCTP_STRRESET_IN_PROGRESS;
- goto out;
+ goto err;
}
- if (request_seq > asoc->strreset_inseq) {
+ if (TSN_lt(asoc->strreset_inseq, request_seq) ||
+ TSN_lt(request_seq, asoc->strreset_inseq - 2)) {
result = SCTP_STRRESET_ERR_BAD_SEQNO;
- goto out;
- } else if (request_seq == asoc->strreset_inseq) {
- asoc->strreset_inseq++;
+ goto err;
+ } else if (TSN_lt(request_seq, asoc->strreset_inseq)) {
+ i = asoc->strreset_inseq - request_seq - 1;
+ result = asoc->strreset_result[i];
+ goto err;
}
+ asoc->strreset_inseq++;
/* Check strreset_enable after inseq inc, as sender cannot tell
* the peer doesn't enable strreset after receiving response with
@@ -388,13 +389,9 @@
goto out;
if (asoc->strreset_chunk) {
- sctp_paramhdr_t *param_hdr;
- struct sctp_transport *t;
-
- param_hdr = sctp_chunk_lookup_strreset_param(
- asoc, outreq->response_seq);
- if (!param_hdr || param_hdr->type !=
- SCTP_PARAM_RESET_IN_REQUEST) {
+ if (!sctp_chunk_lookup_strreset_param(
+ asoc, outreq->response_seq,
+ SCTP_PARAM_RESET_IN_REQUEST)) {
/* same process with outstanding isn't 0 */
result = SCTP_STRRESET_ERR_IN_PROGRESS;
goto out;
@@ -404,6 +401,8 @@
asoc->strreset_outseq++;
if (!asoc->strreset_outstanding) {
+ struct sctp_transport *t;
+
t = asoc->strreset_chunk->transport;
if (del_timer(&t->reconf_timer))
sctp_transport_put(t);
@@ -439,6 +438,8 @@
GFP_ATOMIC);
out:
+ sctp_update_strreset_result(asoc, result);
+err:
return sctp_make_strreset_resp(asoc, result, request_seq);
}
@@ -455,12 +456,18 @@
__u32 request_seq;
request_seq = ntohl(inreq->request_seq);
- if (request_seq > asoc->strreset_inseq) {
+ if (TSN_lt(asoc->strreset_inseq, request_seq) ||
+ TSN_lt(request_seq, asoc->strreset_inseq - 2)) {
result = SCTP_STRRESET_ERR_BAD_SEQNO;
- goto out;
- } else if (request_seq == asoc->strreset_inseq) {
- asoc->strreset_inseq++;
+ goto err;
+ } else if (TSN_lt(request_seq, asoc->strreset_inseq)) {
+ i = asoc->strreset_inseq - request_seq - 1;
+ result = asoc->strreset_result[i];
+ if (result == SCTP_STRRESET_PERFORMED)
+ return NULL;
+ goto err;
}
+ asoc->strreset_inseq++;
if (!(asoc->strreset_enable & SCTP_ENABLE_RESET_STREAM_REQ))
goto out;
@@ -495,12 +502,407 @@
asoc->strreset_outstanding = 1;
sctp_chunk_hold(asoc->strreset_chunk);
+ result = SCTP_STRRESET_PERFORMED;
+
*evp = sctp_ulpevent_make_stream_reset_event(asoc,
SCTP_STREAM_RESET_INCOMING_SSN, nums, str_p, GFP_ATOMIC);
out:
+ sctp_update_strreset_result(asoc, result);
+err:
if (!chunk)
chunk = sctp_make_strreset_resp(asoc, result, request_seq);
return chunk;
}
+
+struct sctp_chunk *sctp_process_strreset_tsnreq(
+ struct sctp_association *asoc,
+ union sctp_params param,
+ struct sctp_ulpevent **evp)
+{
+ __u32 init_tsn = 0, next_tsn = 0, max_tsn_seen;
+ struct sctp_strreset_tsnreq *tsnreq = param.v;
+ struct sctp_stream *stream = asoc->stream;
+ __u32 result = SCTP_STRRESET_DENIED;
+ __u32 request_seq;
+ __u16 i;
+
+ request_seq = ntohl(tsnreq->request_seq);
+ if (TSN_lt(asoc->strreset_inseq, request_seq) ||
+ TSN_lt(request_seq, asoc->strreset_inseq - 2)) {
+ result = SCTP_STRRESET_ERR_BAD_SEQNO;
+ goto err;
+ } else if (TSN_lt(request_seq, asoc->strreset_inseq)) {
+ i = asoc->strreset_inseq - request_seq - 1;
+ result = asoc->strreset_result[i];
+ if (result == SCTP_STRRESET_PERFORMED) {
+ next_tsn = asoc->next_tsn;
+ init_tsn =
+ sctp_tsnmap_get_ctsn(&asoc->peer.tsn_map) + 1;
+ }
+ goto err;
+ }
+ asoc->strreset_inseq++;
+
+ if (!(asoc->strreset_enable & SCTP_ENABLE_RESET_ASSOC_REQ))
+ goto out;
+
+ if (asoc->strreset_outstanding) {
+ result = SCTP_STRRESET_ERR_IN_PROGRESS;
+ goto out;
+ }
+
+ /* G3: The same processing as though a SACK chunk with no gap report
+ * and a cumulative TSN ACK of the Sender's Next TSN minus 1 were
+ * received MUST be performed.
+ */
+ max_tsn_seen = sctp_tsnmap_get_max_tsn_seen(&asoc->peer.tsn_map);
+ sctp_ulpq_reasm_flushtsn(&asoc->ulpq, max_tsn_seen);
+ sctp_ulpq_abort_pd(&asoc->ulpq, GFP_ATOMIC);
+
+ /* G1: Compute an appropriate value for the Receiver's Next TSN -- the
+ * TSN that the peer should use to send the next DATA chunk. The
+ * value SHOULD be the smallest TSN not acknowledged by the
+ * receiver of the request plus 2^31.
+ */
+ init_tsn = sctp_tsnmap_get_ctsn(&asoc->peer.tsn_map) + (1 << 31);
+ sctp_tsnmap_init(&asoc->peer.tsn_map, SCTP_TSN_MAP_INITIAL,
+ init_tsn, GFP_ATOMIC);
+
+ /* G4: The same processing as though a FWD-TSN chunk (as defined in
+ * [RFC3758]) with all streams affected and a new cumulative TSN
+ * ACK of the Receiver's Next TSN minus 1 were received MUST be
+ * performed.
+ */
+ sctp_outq_free(&asoc->outqueue);
+
+ /* G2: Compute an appropriate value for the local endpoint's next TSN,
+ * i.e., the next TSN assigned by the receiver of the SSN/TSN reset
+ * chunk. The value SHOULD be the highest TSN sent by the receiver
+ * of the request plus 1.
+ */
+ next_tsn = asoc->next_tsn;
+ asoc->ctsn_ack_point = next_tsn - 1;
+ asoc->adv_peer_ack_point = asoc->ctsn_ack_point;
+
+ /* G5: The next expected and outgoing SSNs MUST be reset to 0 for all
+ * incoming and outgoing streams.
+ */
+ for (i = 0; i < stream->outcnt; i++)
+ stream->out[i].ssn = 0;
+ for (i = 0; i < stream->incnt; i++)
+ stream->in[i].ssn = 0;
+
+ result = SCTP_STRRESET_PERFORMED;
+
+ *evp = sctp_ulpevent_make_assoc_reset_event(asoc, 0, init_tsn,
+ next_tsn, GFP_ATOMIC);
+
+out:
+ sctp_update_strreset_result(asoc, result);
+err:
+ return sctp_make_strreset_tsnresp(asoc, result, request_seq,
+ next_tsn, init_tsn);
+}
+
+struct sctp_chunk *sctp_process_strreset_addstrm_out(
+ struct sctp_association *asoc,
+ union sctp_params param,
+ struct sctp_ulpevent **evp)
+{
+ struct sctp_strreset_addstrm *addstrm = param.v;
+ struct sctp_stream *stream = asoc->stream;
+ __u32 result = SCTP_STRRESET_DENIED;
+ struct sctp_stream_in *streamin;
+ __u32 request_seq, incnt;
+ __u16 in, i;
+
+ request_seq = ntohl(addstrm->request_seq);
+ if (TSN_lt(asoc->strreset_inseq, request_seq) ||
+ TSN_lt(request_seq, asoc->strreset_inseq - 2)) {
+ result = SCTP_STRRESET_ERR_BAD_SEQNO;
+ goto err;
+ } else if (TSN_lt(request_seq, asoc->strreset_inseq)) {
+ i = asoc->strreset_inseq - request_seq - 1;
+ result = asoc->strreset_result[i];
+ goto err;
+ }
+ asoc->strreset_inseq++;
+
+ if (!(asoc->strreset_enable & SCTP_ENABLE_CHANGE_ASSOC_REQ))
+ goto out;
+
+ if (asoc->strreset_chunk) {
+ if (!sctp_chunk_lookup_strreset_param(
+ asoc, 0, SCTP_PARAM_RESET_ADD_IN_STREAMS)) {
+ /* same process with outstanding isn't 0 */
+ result = SCTP_STRRESET_ERR_IN_PROGRESS;
+ goto out;
+ }
+
+ asoc->strreset_outstanding--;
+ asoc->strreset_outseq++;
+
+ if (!asoc->strreset_outstanding) {
+ struct sctp_transport *t;
+
+ t = asoc->strreset_chunk->transport;
+ if (del_timer(&t->reconf_timer))
+ sctp_transport_put(t);
+
+ sctp_chunk_put(asoc->strreset_chunk);
+ asoc->strreset_chunk = NULL;
+ }
+ }
+
+ in = ntohs(addstrm->number_of_streams);
+ incnt = stream->incnt + in;
+ if (!in || incnt > SCTP_MAX_STREAM)
+ goto out;
+
+ streamin = krealloc(stream->in, incnt * sizeof(*streamin),
+ GFP_ATOMIC);
+ if (!streamin)
+ goto out;
+
+ memset(streamin + stream->incnt, 0, in * sizeof(*streamin));
+ stream->in = streamin;
+ stream->incnt = incnt;
+
+ result = SCTP_STRRESET_PERFORMED;
+
+ *evp = sctp_ulpevent_make_stream_change_event(asoc,
+ 0, ntohs(addstrm->number_of_streams), 0, GFP_ATOMIC);
+
+out:
+ sctp_update_strreset_result(asoc, result);
+err:
+ return sctp_make_strreset_resp(asoc, result, request_seq);
+}
+
+struct sctp_chunk *sctp_process_strreset_addstrm_in(
+ struct sctp_association *asoc,
+ union sctp_params param,
+ struct sctp_ulpevent **evp)
+{
+ struct sctp_strreset_addstrm *addstrm = param.v;
+ struct sctp_stream *stream = asoc->stream;
+ __u32 result = SCTP_STRRESET_DENIED;
+ struct sctp_stream_out *streamout;
+ struct sctp_chunk *chunk = NULL;
+ __u32 request_seq, outcnt;
+ __u16 out, i;
+
+ request_seq = ntohl(addstrm->request_seq);
+ if (TSN_lt(asoc->strreset_inseq, request_seq) ||
+ TSN_lt(request_seq, asoc->strreset_inseq - 2)) {
+ result = SCTP_STRRESET_ERR_BAD_SEQNO;
+ goto err;
+ } else if (TSN_lt(request_seq, asoc->strreset_inseq)) {
+ i = asoc->strreset_inseq - request_seq - 1;
+ result = asoc->strreset_result[i];
+ if (result == SCTP_STRRESET_PERFORMED)
+ return NULL;
+ goto err;
+ }
+ asoc->strreset_inseq++;
+
+ if (!(asoc->strreset_enable & SCTP_ENABLE_CHANGE_ASSOC_REQ))
+ goto out;
+
+ if (asoc->strreset_outstanding) {
+ result = SCTP_STRRESET_ERR_IN_PROGRESS;
+ goto out;
+ }
+
+ out = ntohs(addstrm->number_of_streams);
+ outcnt = stream->outcnt + out;
+ if (!out || outcnt > SCTP_MAX_STREAM)
+ goto out;
+
+ streamout = krealloc(stream->out, outcnt * sizeof(*streamout),
+ GFP_ATOMIC);
+ if (!streamout)
+ goto out;
+
+ memset(streamout + stream->outcnt, 0, out * sizeof(*streamout));
+ stream->out = streamout;
+
+ chunk = sctp_make_strreset_addstrm(asoc, out, 0);
+ if (!chunk)
+ goto out;
+
+ asoc->strreset_chunk = chunk;
+ asoc->strreset_outstanding = 1;
+ sctp_chunk_hold(asoc->strreset_chunk);
+
+ stream->outcnt = outcnt;
+
+ result = SCTP_STRRESET_PERFORMED;
+
+ *evp = sctp_ulpevent_make_stream_change_event(asoc,
+ 0, 0, ntohs(addstrm->number_of_streams), GFP_ATOMIC);
+
+out:
+ sctp_update_strreset_result(asoc, result);
+err:
+ if (!chunk)
+ chunk = sctp_make_strreset_resp(asoc, result, request_seq);
+
+ return chunk;
+}
+
+struct sctp_chunk *sctp_process_strreset_resp(
+ struct sctp_association *asoc,
+ union sctp_params param,
+ struct sctp_ulpevent **evp)
+{
+ struct sctp_strreset_resp *resp = param.v;
+ struct sctp_stream *stream = asoc->stream;
+ struct sctp_transport *t;
+ __u16 i, nums, flags = 0;
+ sctp_paramhdr_t *req;
+ __u32 result;
+
+ req = sctp_chunk_lookup_strreset_param(asoc, resp->response_seq, 0);
+ if (!req)
+ return NULL;
+
+ result = ntohl(resp->result);
+ if (result != SCTP_STRRESET_PERFORMED) {
+ /* if in progress, do nothing but retransmit */
+ if (result == SCTP_STRRESET_IN_PROGRESS)
+ return NULL;
+ else if (result == SCTP_STRRESET_DENIED)
+ flags = SCTP_STREAM_RESET_DENIED;
+ else
+ flags = SCTP_STREAM_RESET_FAILED;
+ }
+
+ if (req->type == SCTP_PARAM_RESET_OUT_REQUEST) {
+ struct sctp_strreset_outreq *outreq;
+ __u16 *str_p;
+
+ outreq = (struct sctp_strreset_outreq *)req;
+ str_p = outreq->list_of_streams;
+ nums = (ntohs(outreq->param_hdr.length) - sizeof(*outreq)) / 2;
+
+ if (result == SCTP_STRRESET_PERFORMED) {
+ if (nums) {
+ for (i = 0; i < nums; i++)
+ stream->out[ntohs(str_p[i])].ssn = 0;
+ } else {
+ for (i = 0; i < stream->outcnt; i++)
+ stream->out[i].ssn = 0;
+ }
+
+ flags = SCTP_STREAM_RESET_OUTGOING_SSN;
+ }
+
+ for (i = 0; i < stream->outcnt; i++)
+ stream->out[i].state = SCTP_STREAM_OPEN;
+
+ *evp = sctp_ulpevent_make_stream_reset_event(asoc, flags,
+ nums, str_p, GFP_ATOMIC);
+ } else if (req->type == SCTP_PARAM_RESET_IN_REQUEST) {
+ struct sctp_strreset_inreq *inreq;
+ __u16 *str_p;
+
+ /* if the result is performed, it's impossible for inreq */
+ if (result == SCTP_STRRESET_PERFORMED)
+ return NULL;
+
+ inreq = (struct sctp_strreset_inreq *)req;
+ str_p = inreq->list_of_streams;
+ nums = (ntohs(inreq->param_hdr.length) - sizeof(*inreq)) / 2;
+
+ *evp = sctp_ulpevent_make_stream_reset_event(asoc, flags,
+ nums, str_p, GFP_ATOMIC);
+ } else if (req->type == SCTP_PARAM_RESET_TSN_REQUEST) {
+ struct sctp_strreset_resptsn *resptsn;
+ __u32 stsn, rtsn;
+
+ /* check for resptsn, as sctp_verify_reconf didn't do it*/
+ if (ntohs(param.p->length) != sizeof(*resptsn))
+ return NULL;
+
+ resptsn = (struct sctp_strreset_resptsn *)resp;
+ stsn = ntohl(resptsn->senders_next_tsn);
+ rtsn = ntohl(resptsn->receivers_next_tsn);
+
+ if (result == SCTP_STRRESET_PERFORMED) {
+ __u32 mtsn = sctp_tsnmap_get_max_tsn_seen(
+ &asoc->peer.tsn_map);
+
+ sctp_ulpq_reasm_flushtsn(&asoc->ulpq, mtsn);
+ sctp_ulpq_abort_pd(&asoc->ulpq, GFP_ATOMIC);
+
+ sctp_tsnmap_init(&asoc->peer.tsn_map,
+ SCTP_TSN_MAP_INITIAL,
+ stsn, GFP_ATOMIC);
+
+ sctp_outq_free(&asoc->outqueue);
+
+ asoc->next_tsn = rtsn;
+ asoc->ctsn_ack_point = asoc->next_tsn - 1;
+ asoc->adv_peer_ack_point = asoc->ctsn_ack_point;
+
+ for (i = 0; i < stream->outcnt; i++)
+ stream->out[i].ssn = 0;
+ for (i = 0; i < stream->incnt; i++)
+ stream->in[i].ssn = 0;
+ }
+
+ for (i = 0; i < stream->outcnt; i++)
+ stream->out[i].state = SCTP_STREAM_OPEN;
+
+ *evp = sctp_ulpevent_make_assoc_reset_event(asoc, flags,
+ stsn, rtsn, GFP_ATOMIC);
+ } else if (req->type == SCTP_PARAM_RESET_ADD_OUT_STREAMS) {
+ struct sctp_strreset_addstrm *addstrm;
+ __u16 number;
+
+ addstrm = (struct sctp_strreset_addstrm *)req;
+ nums = ntohs(addstrm->number_of_streams);
+ number = stream->outcnt - nums;
+
+ if (result == SCTP_STRRESET_PERFORMED)
+ for (i = number; i < stream->outcnt; i++)
+ stream->out[i].state = SCTP_STREAM_OPEN;
+ else
+ stream->outcnt = number;
+
+ *evp = sctp_ulpevent_make_stream_change_event(asoc, flags,
+ 0, nums, GFP_ATOMIC);
+ } else if (req->type == SCTP_PARAM_RESET_ADD_IN_STREAMS) {
+ struct sctp_strreset_addstrm *addstrm;
+
+ /* if the result is performed, it's impossible for addstrm in
+ * request.
+ */
+ if (result == SCTP_STRRESET_PERFORMED)
+ return NULL;
+
+ addstrm = (struct sctp_strreset_addstrm *)req;
+ nums = ntohs(addstrm->number_of_streams);
+
+ *evp = sctp_ulpevent_make_stream_change_event(asoc, flags,
+ nums, 0, GFP_ATOMIC);
+ }
+
+ asoc->strreset_outstanding--;
+ asoc->strreset_outseq++;
+
+ /* remove everything for this reconf request */
+ if (!asoc->strreset_outstanding) {
+ t = asoc->strreset_chunk->transport;
+ if (del_timer(&t->reconf_timer))
+ sctp_transport_put(t);
+
+ sctp_chunk_put(asoc->strreset_chunk);
+ asoc->strreset_chunk = NULL;
+ }
+
+ return NULL;
+}
diff --git a/net/sctp/sysctl.c b/net/sctp/sysctl.c
index daf8554..0e732f6 100644
--- a/net/sctp/sysctl.c
+++ b/net/sctp/sysctl.c
@@ -275,6 +275,13 @@
.proc_handler = proc_dointvec,
},
{
+ .procname = "reconf_enable",
+ .data = &init_net.sctp.reconf_enable,
+ .maxlen = sizeof(int),
+ .mode = 0644,
+ .proc_handler = proc_dointvec,
+ },
+ {
.procname = "auth_enable",
.data = &init_net.sctp.auth_enable,
.maxlen = sizeof(int),
diff --git a/net/sctp/ulpevent.c b/net/sctp/ulpevent.c
index c8881bc5..ec2b3e0 100644
--- a/net/sctp/ulpevent.c
+++ b/net/sctp/ulpevent.c
@@ -883,6 +883,62 @@
return event;
}
+struct sctp_ulpevent *sctp_ulpevent_make_assoc_reset_event(
+ const struct sctp_association *asoc, __u16 flags, __u32 local_tsn,
+ __u32 remote_tsn, gfp_t gfp)
+{
+ struct sctp_assoc_reset_event *areset;
+ struct sctp_ulpevent *event;
+ struct sk_buff *skb;
+
+ event = sctp_ulpevent_new(sizeof(struct sctp_assoc_reset_event),
+ MSG_NOTIFICATION, gfp);
+ if (!event)
+ return NULL;
+
+ skb = sctp_event2skb(event);
+ areset = (struct sctp_assoc_reset_event *)
+ skb_put(skb, sizeof(struct sctp_assoc_reset_event));
+
+ areset->assocreset_type = SCTP_ASSOC_RESET_EVENT;
+ areset->assocreset_flags = flags;
+ areset->assocreset_length = sizeof(struct sctp_assoc_reset_event);
+ sctp_ulpevent_set_owner(event, asoc);
+ areset->assocreset_assoc_id = sctp_assoc2id(asoc);
+ areset->assocreset_local_tsn = local_tsn;
+ areset->assocreset_remote_tsn = remote_tsn;
+
+ return event;
+}
+
+struct sctp_ulpevent *sctp_ulpevent_make_stream_change_event(
+ const struct sctp_association *asoc, __u16 flags,
+ __u32 strchange_instrms, __u32 strchange_outstrms, gfp_t gfp)
+{
+ struct sctp_stream_change_event *schange;
+ struct sctp_ulpevent *event;
+ struct sk_buff *skb;
+
+ event = sctp_ulpevent_new(sizeof(struct sctp_stream_change_event),
+ MSG_NOTIFICATION, gfp);
+ if (!event)
+ return NULL;
+
+ skb = sctp_event2skb(event);
+ schange = (struct sctp_stream_change_event *)
+ skb_put(skb, sizeof(struct sctp_stream_change_event));
+
+ schange->strchange_type = SCTP_STREAM_CHANGE_EVENT;
+ schange->strchange_flags = flags;
+ schange->strchange_length = sizeof(struct sctp_stream_change_event);
+ sctp_ulpevent_set_owner(event, asoc);
+ schange->strchange_assoc_id = sctp_assoc2id(asoc);
+ schange->strchange_instrms = strchange_instrms;
+ schange->strchange_outstrms = strchange_outstrms;
+
+ return event;
+}
+
/* Return the notification type, assuming this is a notification
* event.
*/
diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
index 0938037..5b6ee21 100644
--- a/net/smc/af_smc.c
+++ b/net/smc/af_smc.c
@@ -147,7 +147,6 @@
schedule_delayed_work(&smc->sock_put_work,
SMC_CLOSE_SOCK_PUT_DELAY);
}
- sk->sk_prot->unhash(sk);
release_sock(sk);
sock_put(sk);
@@ -451,6 +450,9 @@
goto decline_rdma_unlock;
}
+ smc_close_init(smc);
+ smc_rx_init(smc);
+
if (local_contact == SMC_FIRST_CONTACT) {
rc = smc_ib_ready_link(link);
if (rc) {
@@ -477,7 +479,6 @@
mutex_unlock(&smc_create_lgr_pending);
smc_tx_init(smc);
- smc_rx_init(smc);
out_connected:
smc_copy_sock_settings_to_clc(smc);
@@ -637,7 +638,8 @@
smc_accept_unlink(new_sk);
if (new_sk->sk_state == SMC_CLOSED) {
- /* tbd in follow-on patch: close this sock */
+ new_sk->sk_prot->unhash(new_sk);
+ sock_put(new_sk);
continue;
}
if (new_sock)
@@ -657,8 +659,13 @@
if (!sk->sk_lingertime)
/* wait for peer closing */
sk->sk_lingertime = SMC_MAX_STREAM_WAIT_TIMEOUT;
- if (!smc->use_fallback)
+ if (smc->use_fallback) {
+ sk->sk_state = SMC_CLOSED;
+ } else {
smc_close_active(smc);
+ sock_set_flag(sk, SOCK_DEAD);
+ sk->sk_shutdown |= SHUTDOWN_MASK;
+ }
if (smc->clcsock) {
struct socket *tcp;
@@ -666,11 +673,9 @@
smc->clcsock = NULL;
sock_release(tcp);
}
- sock_set_flag(sk, SOCK_DEAD);
- sk->sk_shutdown |= SHUTDOWN_MASK;
if (smc->use_fallback) {
schedule_delayed_work(&smc->sock_put_work, TCP_TIMEWAIT_LEN);
- } else {
+ } else if (sk->sk_state == SMC_CLOSED) {
smc_conn_free(&smc->conn);
schedule_delayed_work(&smc->sock_put_work,
SMC_CLOSE_SOCK_PUT_DELAY);
@@ -800,6 +805,9 @@
goto decline_rdma;
}
+ smc_close_init(new_smc);
+ smc_rx_init(new_smc);
+
rc = smc_clc_send_accept(new_smc, local_contact);
if (rc)
goto out_err;
@@ -839,7 +847,6 @@
}
smc_tx_init(new_smc);
- smc_rx_init(new_smc);
out_connected:
sk_refcnt_debug_inc(newsmcsk);
diff --git a/net/smc/smc.h b/net/smc/smc.h
index ee5fbea..6e44313e 100644
--- a/net/smc/smc.h
+++ b/net/smc/smc.h
@@ -164,6 +164,7 @@
#ifndef KERNEL_HAS_ATOMIC64
spinlock_t acurs_lock; /* protect cursors */
#endif
+ struct work_struct close_work; /* peer sent some closing */
};
struct smc_sock { /* smc sock container */
diff --git a/net/smc/smc_cdc.c b/net/smc/smc_cdc.c
index 5a33949..a7294ed 100644
--- a/net/smc/smc_cdc.c
+++ b/net/smc/smc_cdc.c
@@ -217,8 +217,13 @@
smc->sk.sk_err = ECONNRESET;
conn->local_tx_ctrl.conn_state_flags.peer_conn_abort = 1;
}
- if (smc_cdc_rxed_any_close_or_senddone(conn))
- smc_close_passive_received(smc);
+ if (smc_cdc_rxed_any_close_or_senddone(conn)) {
+ smc->sk.sk_shutdown |= RCV_SHUTDOWN;
+ if (smc->clcsock && smc->clcsock->sk)
+ smc->clcsock->sk->sk_shutdown |= RCV_SHUTDOWN;
+ sock_set_flag(&smc->sk, SOCK_DONE);
+ schedule_work(&conn->close_work);
+ }
/* piggy backed tx info */
/* trigger sndbuf consumer: RDMA write into peer RMBE and CDC */
@@ -228,8 +233,6 @@
smc_close_wake_tx_prepared(smc);
}
- /* subsequent patch: trigger socket release if connection closed */
-
/* socket connected but not accepted */
if (!smc->sk.sk_socket)
return;
diff --git a/net/smc/smc_close.c b/net/smc/smc_close.c
index 67a71d1..3c2e166 100644
--- a/net/smc/smc_close.c
+++ b/net/smc/smc_close.c
@@ -117,7 +117,6 @@
struct smc_cdc_conn_state_flags *txflags =
&smc->conn.local_tx_ctrl.conn_state_flags;
- bh_lock_sock(&smc->sk);
smc->sk.sk_err = ECONNABORTED;
if (smc->clcsock && smc->clcsock->sk) {
smc->clcsock->sk->sk_err = ECONNABORTED;
@@ -125,6 +124,7 @@
}
switch (smc->sk.sk_state) {
case SMC_INIT:
+ case SMC_ACTIVE:
smc->sk.sk_state = SMC_PEERABORTWAIT;
break;
case SMC_APPCLOSEWAIT1:
@@ -161,10 +161,15 @@
}
sock_set_flag(&smc->sk, SOCK_DEAD);
- bh_unlock_sock(&smc->sk);
smc->sk.sk_state_change(&smc->sk);
}
+static inline bool smc_close_sent_any_close(struct smc_connection *conn)
+{
+ return conn->local_tx_ctrl.conn_state_flags.peer_conn_abort ||
+ conn->local_tx_ctrl.conn_state_flags.peer_conn_closed;
+}
+
int smc_close_active(struct smc_sock *smc)
{
struct smc_cdc_conn_state_flags *txflags =
@@ -185,8 +190,7 @@
case SMC_INIT:
sk->sk_state = SMC_CLOSED;
if (smc->smc_listen_work.func)
- flush_work(&smc->smc_listen_work);
- sock_put(sk);
+ cancel_work_sync(&smc->smc_listen_work);
break;
case SMC_LISTEN:
sk->sk_state = SMC_CLOSED;
@@ -198,7 +202,7 @@
}
release_sock(sk);
smc_close_cleanup_listen(sk);
- flush_work(&smc->tcp_listen_work);
+ cancel_work_sync(&smc->smc_listen_work);
lock_sock(sk);
break;
case SMC_ACTIVE:
@@ -218,7 +222,7 @@
case SMC_APPFINCLOSEWAIT:
/* socket already shutdown wr or both (active close) */
if (txflags->peer_done_writing &&
- !txflags->peer_conn_closed) {
+ !smc_close_sent_any_close(conn)) {
/* just shutdown wr done, send close request */
rc = smc_close_final(conn);
}
@@ -248,6 +252,13 @@
break;
case SMC_PEERCLOSEWAIT1:
case SMC_PEERCLOSEWAIT2:
+ if (txflags->peer_done_writing &&
+ !smc_close_sent_any_close(conn)) {
+ /* just shutdown wr done, send close request */
+ rc = smc_close_final(conn);
+ }
+ /* peer sending PeerConnectionClosed will cause transition */
+ break;
case SMC_PEERFINCLOSEWAIT:
/* peer sending PeerConnectionClosed will cause transition */
break;
@@ -285,7 +296,7 @@
case SMC_PEERCLOSEWAIT1:
case SMC_PEERCLOSEWAIT2:
if (txflags->peer_done_writing &&
- !txflags->peer_conn_closed) {
+ !smc_close_sent_any_close(&smc->conn)) {
/* just shutdown, but not yet closed locally */
smc_close_abort(&smc->conn);
sk->sk_state = SMC_PROCESSABORT;
@@ -306,22 +317,27 @@
/* Some kind of closing has been received: peer_conn_closed, peer_conn_abort,
* or peer_done_writing.
- * Called under tasklet context.
*/
-void smc_close_passive_received(struct smc_sock *smc)
+static void smc_close_passive_work(struct work_struct *work)
{
- struct smc_cdc_conn_state_flags *rxflags =
- &smc->conn.local_rx_ctrl.conn_state_flags;
+ struct smc_connection *conn = container_of(work,
+ struct smc_connection,
+ close_work);
+ struct smc_sock *smc = container_of(conn, struct smc_sock, conn);
+ struct smc_cdc_conn_state_flags *rxflags;
struct sock *sk = &smc->sk;
int old_state;
- sk->sk_shutdown |= RCV_SHUTDOWN;
- if (smc->clcsock && smc->clcsock->sk)
- smc->clcsock->sk->sk_shutdown |= RCV_SHUTDOWN;
- sock_set_flag(&smc->sk, SOCK_DONE);
-
+ lock_sock(&smc->sk);
old_state = sk->sk_state;
+ if (!conn->alert_token_local) {
+ /* abnormal termination */
+ smc_close_active_abort(smc);
+ goto wakeup;
+ }
+
+ rxflags = &smc->conn.local_rx_ctrl.conn_state_flags;
if (rxflags->peer_conn_abort) {
smc_close_passive_abort_received(smc);
goto wakeup;
@@ -331,7 +347,7 @@
case SMC_INIT:
if (atomic_read(&smc->conn.bytes_to_rcv) ||
(rxflags->peer_done_writing &&
- !rxflags->peer_conn_closed))
+ !smc_cdc_rxed_any_close(conn)))
sk->sk_state = SMC_APPCLOSEWAIT1;
else
sk->sk_state = SMC_CLOSED;
@@ -348,7 +364,7 @@
if (!smc_cdc_rxed_any_close(&smc->conn))
break;
if (sock_flag(sk, SOCK_DEAD) &&
- (sk->sk_shutdown == SHUTDOWN_MASK)) {
+ smc_close_sent_any_close(conn)) {
/* smc_release has already been called locally */
sk->sk_state = SMC_CLOSED;
} else {
@@ -367,17 +383,19 @@
}
wakeup:
- if (old_state != sk->sk_state)
- sk->sk_state_change(sk);
sk->sk_data_ready(sk); /* wakeup blocked rcvbuf consumers */
sk->sk_write_space(sk); /* wakeup blocked sndbuf producers */
- if ((sk->sk_state == SMC_CLOSED) &&
- (sock_flag(sk, SOCK_DEAD) || (old_state == SMC_INIT))) {
- smc_conn_free(&smc->conn);
- schedule_delayed_work(&smc->sock_put_work,
- SMC_CLOSE_SOCK_PUT_DELAY);
+ if (old_state != sk->sk_state) {
+ sk->sk_state_change(sk);
+ if ((sk->sk_state == SMC_CLOSED) &&
+ (sock_flag(sk, SOCK_DEAD) || !sk->sk_socket)) {
+ smc_conn_free(&smc->conn);
+ schedule_delayed_work(&smc->sock_put_work,
+ SMC_CLOSE_SOCK_PUT_DELAY);
+ }
}
+ release_sock(&smc->sk);
}
void smc_close_sock_put_work(struct work_struct *work)
@@ -442,3 +460,9 @@
sk->sk_state_change(&smc->sk);
return rc;
}
+
+/* Initialize close properties on connection establishment. */
+void smc_close_init(struct smc_sock *smc)
+{
+ INIT_WORK(&smc->conn.close_work, smc_close_passive_work);
+}
diff --git a/net/smc/smc_close.h b/net/smc/smc_close.h
index bc9a2df..4a3d99a 100644
--- a/net/smc/smc_close.h
+++ b/net/smc/smc_close.h
@@ -21,8 +21,8 @@
void smc_close_wake_tx_prepared(struct smc_sock *smc);
void smc_close_active_abort(struct smc_sock *smc);
int smc_close_active(struct smc_sock *smc);
-void smc_close_passive_received(struct smc_sock *smc);
void smc_close_sock_put_work(struct work_struct *work);
int smc_close_shutdown_write(struct smc_sock *smc);
+void smc_close_init(struct smc_sock *smc);
#endif /* SMC_CLOSE_H */
diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c
index 0eac633..65020e9 100644
--- a/net/smc/smc_core.c
+++ b/net/smc/smc_core.c
@@ -316,7 +316,7 @@
smc = container_of(conn, struct smc_sock, conn);
sock_hold(&smc->sk);
__smc_lgr_unregister_conn(conn);
- smc_close_active_abort(smc);
+ schedule_work(&conn->close_work);
sock_put(&smc->sk);
node = rb_first(&lgr->conns_all);
}
diff --git a/net/smc/smc_ib.c b/net/smc/smc_ib.c
index e6743c0..16b7c80 100644
--- a/net/smc/smc_ib.c
+++ b/net/smc/smc_ib.c
@@ -179,8 +179,6 @@
u8 port_idx;
smcibdev = container_of(handler, struct smc_ib_device, event_handler);
- if (!smc_pnet_find_ib(smcibdev->ibdev->name))
- return;
switch (ibevent->event) {
case IB_EVENT_PORT_ERR:
@@ -259,7 +257,6 @@
.max_recv_wr = SMC_WR_BUF_CNT * 3,
.max_send_sge = SMC_IB_MAX_SEND_SGE,
.max_recv_sge = 1,
- .max_inline_data = SMC_WR_TX_SIZE,
},
.sq_sig_type = IB_SIGNAL_REQ_WR,
.qp_type = IB_QPT_RC,
diff --git a/net/smc/smc_ib.h b/net/smc/smc_ib.h
index a95f74b..7e1f0e2 100644
--- a/net/smc/smc_ib.h
+++ b/net/smc/smc_ib.h
@@ -11,6 +11,7 @@
#ifndef _SMC_IB_H
#define _SMC_IB_H
+#include <linux/interrupt.h>
#include <linux/if_ether.h>
#include <rdma/ib_verbs.h>
diff --git a/net/smc/smc_pnet.c b/net/smc/smc_pnet.c
index 9d3e7fb..78f7af2 100644
--- a/net/smc/smc_pnet.c
+++ b/net/smc/smc_pnet.c
@@ -219,7 +219,7 @@
}
/* Find an infiniband device by a given name. The device might not exist. */
-struct smc_ib_device *smc_pnet_find_ib(char *ib_name)
+static struct smc_ib_device *smc_pnet_find_ib(char *ib_name)
{
struct smc_ib_device *ibdev;
@@ -523,8 +523,11 @@
read_lock(&smc_pnettable.lock);
list_for_each_entry(pnetelem, &smc_pnettable.pnetlist, list) {
if (dst->dev == pnetelem->ndev) {
- *smcibdev = pnetelem->smcibdev;
- *ibport = pnetelem->ib_port;
+ if (smc_ib_port_active(pnetelem->smcibdev,
+ pnetelem->ib_port)) {
+ *smcibdev = pnetelem->smcibdev;
+ *ibport = pnetelem->ib_port;
+ }
break;
}
}
diff --git a/net/smc/smc_pnet.h b/net/smc/smc_pnet.h
index 32ab3df..c4f1bcc 100644
--- a/net/smc/smc_pnet.h
+++ b/net/smc/smc_pnet.h
@@ -16,7 +16,6 @@
int smc_pnet_init(void) __init;
void smc_pnet_exit(void);
int smc_pnet_remove_by_ibdev(struct smc_ib_device *ibdev);
-struct smc_ib_device *smc_pnet_find_ib(char *ib_name);
void smc_pnet_find_roce_resource(struct sock *sk,
struct smc_ib_device **smcibdev, u8 *ibport);
diff --git a/net/smc/smc_rx.c b/net/smc/smc_rx.c
index c4ef9a4..f0c8b08 100644
--- a/net/smc/smc_rx.c
+++ b/net/smc/smc_rx.c
@@ -36,11 +36,10 @@
if (skwq_has_sleeper(wq))
wake_up_interruptible_sync_poll(&wq->wait, POLLIN | POLLPRI |
POLLRDNORM | POLLRDBAND);
+ sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
if ((sk->sk_shutdown == SHUTDOWN_MASK) ||
(sk->sk_state == SMC_CLOSED))
sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_HUP);
- else
- sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
rcu_read_unlock();
}
diff --git a/net/smc/smc_tx.c b/net/smc/smc_tx.c
index 69a0013..21ec183 100644
--- a/net/smc/smc_tx.c
+++ b/net/smc/smc_tx.c
@@ -431,9 +431,13 @@
struct smc_connection,
tx_work);
struct smc_sock *smc = container_of(conn, struct smc_sock, conn);
+ int rc;
lock_sock(&smc->sk);
- smc_tx_sndbuf_nonempty(conn);
+ rc = smc_tx_sndbuf_nonempty(conn);
+ if (!rc && conn->local_rx_ctrl.prod_flags.write_blocked &&
+ !atomic_read(&conn->bytes_to_rcv))
+ conn->local_rx_ctrl.prod_flags.write_blocked = 0;
release_sock(&smc->sk);
}
diff --git a/net/smc/smc_wr.c b/net/smc/smc_wr.c
index eadf157..874ee9f 100644
--- a/net/smc/smc_wr.c
+++ b/net/smc/smc_wr.c
@@ -447,7 +447,7 @@
lnk->wr_tx_ibs[i].num_sge = 1;
lnk->wr_tx_ibs[i].opcode = IB_WR_SEND;
lnk->wr_tx_ibs[i].send_flags =
- IB_SEND_SIGNALED | IB_SEND_SOLICITED | IB_SEND_INLINE;
+ IB_SEND_SIGNALED | IB_SEND_SOLICITED;
}
for (i = 0; i < lnk->wr_rx_cnt; i++) {
lnk->wr_rx_sges[i].addr =
diff --git a/net/socket.c b/net/socket.c
index 985ef06..c2564eb 100644
--- a/net/socket.c
+++ b/net/socket.c
@@ -3356,3 +3356,49 @@
return sock->ops->shutdown(sock, how);
}
EXPORT_SYMBOL(kernel_sock_shutdown);
+
+/* This routine returns the IP overhead imposed by a socket i.e.
+ * the length of the underlying IP header, depending on whether
+ * this is an IPv4 or IPv6 socket and the length from IP options turned
+ * on at the socket. Assumes that the caller has a lock on the socket.
+ */
+u32 kernel_sock_ip_overhead(struct sock *sk)
+{
+ struct inet_sock *inet;
+ struct ip_options_rcu *opt;
+ u32 overhead = 0;
+ bool owned_by_user;
+#if IS_ENABLED(CONFIG_IPV6)
+ struct ipv6_pinfo *np;
+ struct ipv6_txoptions *optv6 = NULL;
+#endif /* IS_ENABLED(CONFIG_IPV6) */
+
+ if (!sk)
+ return overhead;
+
+ owned_by_user = sock_owned_by_user(sk);
+ switch (sk->sk_family) {
+ case AF_INET:
+ inet = inet_sk(sk);
+ overhead += sizeof(struct iphdr);
+ opt = rcu_dereference_protected(inet->inet_opt,
+ owned_by_user);
+ if (opt)
+ overhead += opt->opt.optlen;
+ return overhead;
+#if IS_ENABLED(CONFIG_IPV6)
+ case AF_INET6:
+ np = inet6_sk(sk);
+ overhead += sizeof(struct ipv6hdr);
+ if (np)
+ optv6 = rcu_dereference_protected(np->opt,
+ owned_by_user);
+ if (optv6)
+ overhead += (optv6->opt_flen + optv6->opt_nflen);
+ return overhead;
+#endif /* IS_ENABLED(CONFIG_IPV6) */
+ default: /* Returns 0 overhead if the socket is not ipv4 or ipv6 */
+ return overhead;
+ }
+}
+EXPORT_SYMBOL(kernel_sock_ip_overhead);
diff --git a/net/switchdev/switchdev.c b/net/switchdev/switchdev.c
index 017801f..8d40a7d 100644
--- a/net/switchdev/switchdev.c
+++ b/net/switchdev/switchdev.c
@@ -826,7 +826,7 @@
int err;
err = nla_validate_nested(protinfo, IFLA_BRPORT_MAX,
- switchdev_port_bridge_policy);
+ switchdev_port_bridge_policy, NULL);
if (err)
return err;
diff --git a/net/tipc/bearer.c b/net/tipc/bearer.c
index 33a5bdf..d174ee3 100644
--- a/net/tipc/bearer.c
+++ b/net/tipc/bearer.c
@@ -802,7 +802,7 @@
err = nla_parse_nested(attrs, TIPC_NLA_BEARER_MAX,
info->attrs[TIPC_NLA_BEARER],
- tipc_nl_bearer_policy);
+ tipc_nl_bearer_policy, info->extack);
if (err)
return err;
@@ -851,7 +851,7 @@
err = nla_parse_nested(attrs, TIPC_NLA_BEARER_MAX,
info->attrs[TIPC_NLA_BEARER],
- tipc_nl_bearer_policy);
+ tipc_nl_bearer_policy, info->extack);
if (err)
return err;
@@ -891,7 +891,7 @@
err = nla_parse_nested(attrs, TIPC_NLA_BEARER_MAX,
info->attrs[TIPC_NLA_BEARER],
- tipc_nl_bearer_policy);
+ tipc_nl_bearer_policy, info->extack);
if (err)
return err;
@@ -939,7 +939,7 @@
err = nla_parse_nested(attrs, TIPC_NLA_BEARER_MAX,
info->attrs[TIPC_NLA_BEARER],
- tipc_nl_bearer_policy);
+ tipc_nl_bearer_policy, info->extack);
if (err)
return err;
@@ -982,7 +982,7 @@
err = nla_parse_nested(attrs, TIPC_NLA_BEARER_MAX,
info->attrs[TIPC_NLA_BEARER],
- tipc_nl_bearer_policy);
+ tipc_nl_bearer_policy, info->extack);
if (err)
return err;
@@ -1104,7 +1104,7 @@
err = nla_parse_nested(attrs, TIPC_NLA_MEDIA_MAX,
info->attrs[TIPC_NLA_MEDIA],
- tipc_nl_media_policy);
+ tipc_nl_media_policy, info->extack);
if (err)
return err;
@@ -1152,7 +1152,7 @@
err = nla_parse_nested(attrs, TIPC_NLA_MEDIA_MAX,
info->attrs[TIPC_NLA_MEDIA],
- tipc_nl_media_policy);
+ tipc_nl_media_policy, info->extack);
if (!attrs[TIPC_NLA_MEDIA_NAME])
return -EINVAL;
diff --git a/net/tipc/link.c b/net/tipc/link.c
index ddd2dd6f..60820dc 100644
--- a/net/tipc/link.c
+++ b/net/tipc/link.c
@@ -1827,7 +1827,7 @@
int err;
err = nla_parse_nested(props, TIPC_NLA_PROP_MAX, prop,
- tipc_nl_prop_policy);
+ tipc_nl_prop_policy, NULL);
if (err)
return err;
diff --git a/net/tipc/name_table.c b/net/tipc/name_table.c
index 9be6592..bd0aac8 100644
--- a/net/tipc/name_table.c
+++ b/net/tipc/name_table.c
@@ -416,6 +416,7 @@
tipc_subscrp_convert_seq(&s->evt.s.seq, s->swap, &ns);
+ tipc_subscrp_get(s);
list_add(&s->nameseq_list, &nseq->subscriptions);
if (!sseq)
@@ -787,6 +788,7 @@
if (seq != NULL) {
spin_lock_bh(&seq->lock);
list_del_init(&s->nameseq_list);
+ tipc_subscrp_put(s);
if (!seq->first_free && list_empty(&seq->subscriptions)) {
hlist_del_init_rcu(&seq->ns_list);
kfree(seq->sseqs);
diff --git a/net/tipc/net.c b/net/tipc/net.c
index ab8a2d5..719c592 100644
--- a/net/tipc/net.c
+++ b/net/tipc/net.c
@@ -211,8 +211,8 @@
return -EINVAL;
err = nla_parse_nested(attrs, TIPC_NLA_NET_MAX,
- info->attrs[TIPC_NLA_NET],
- tipc_nl_net_policy);
+ info->attrs[TIPC_NLA_NET], tipc_nl_net_policy,
+ info->extack);
if (err)
return err;
diff --git a/net/tipc/netlink.c b/net/tipc/netlink.c
index 26ca8dd..b76f13f 100644
--- a/net/tipc/netlink.c
+++ b/net/tipc/netlink.c
@@ -268,7 +268,8 @@
if (!*attr)
return -EOPNOTSUPP;
- return nlmsg_parse(nlh, GENL_HDRLEN, *attr, maxattr, tipc_nl_policy);
+ return nlmsg_parse(nlh, GENL_HDRLEN, *attr, maxattr, tipc_nl_policy,
+ NULL);
}
int __init tipc_netlink_start(void)
diff --git a/net/tipc/netlink_compat.c b/net/tipc/netlink_compat.c
index e1ae8a8..9bfe886 100644
--- a/net/tipc/netlink_compat.c
+++ b/net/tipc/netlink_compat.c
@@ -296,7 +296,7 @@
err = nla_parse(attrbuf, tipc_genl_family.maxattr,
(const struct nlattr *)trans_buf->data,
- trans_buf->len, NULL);
+ trans_buf->len, NULL, NULL);
if (err)
goto parse_out;
@@ -352,7 +352,7 @@
return -EINVAL;
err = nla_parse_nested(bearer, TIPC_NLA_BEARER_MAX,
- attrs[TIPC_NLA_BEARER], NULL);
+ attrs[TIPC_NLA_BEARER], NULL, NULL);
if (err)
return err;
@@ -472,7 +472,7 @@
return -EINVAL;
err = nla_parse_nested(link, TIPC_NLA_LINK_MAX, attrs[TIPC_NLA_LINK],
- NULL);
+ NULL, NULL);
if (err)
return err;
@@ -480,7 +480,7 @@
return -EINVAL;
err = nla_parse_nested(prop, TIPC_NLA_PROP_MAX,
- link[TIPC_NLA_LINK_PROP], NULL);
+ link[TIPC_NLA_LINK_PROP], NULL, NULL);
if (err)
return err;
@@ -488,7 +488,7 @@
return -EINVAL;
err = nla_parse_nested(stats, TIPC_NLA_STATS_MAX,
- link[TIPC_NLA_LINK_STATS], NULL);
+ link[TIPC_NLA_LINK_STATS], NULL, NULL);
if (err)
return err;
@@ -598,7 +598,7 @@
return -EINVAL;
err = nla_parse_nested(link, TIPC_NLA_LINK_MAX, attrs[TIPC_NLA_LINK],
- NULL);
+ NULL, NULL);
if (err)
return err;
@@ -795,7 +795,7 @@
return -EINVAL;
err = nla_parse_nested(nt, TIPC_NLA_NAME_TABLE_MAX,
- attrs[TIPC_NLA_NAME_TABLE], NULL);
+ attrs[TIPC_NLA_NAME_TABLE], NULL, NULL);
if (err)
return err;
@@ -803,7 +803,7 @@
return -EINVAL;
err = nla_parse_nested(publ, TIPC_NLA_PUBL_MAX,
- nt[TIPC_NLA_NAME_TABLE_PUBL], NULL);
+ nt[TIPC_NLA_NAME_TABLE_PUBL], NULL, NULL);
if (err)
return err;
@@ -863,7 +863,7 @@
return -EINVAL;
err = nla_parse_nested(publ, TIPC_NLA_PUBL_MAX, attrs[TIPC_NLA_PUBL],
- NULL);
+ NULL, NULL);
if (err)
return err;
@@ -929,7 +929,7 @@
return -EINVAL;
err = nla_parse_nested(sock, TIPC_NLA_SOCK_MAX, attrs[TIPC_NLA_SOCK],
- NULL);
+ NULL, NULL);
if (err)
return err;
@@ -940,8 +940,8 @@
u32 node;
struct nlattr *con[TIPC_NLA_CON_MAX + 1];
- nla_parse_nested(con, TIPC_NLA_CON_MAX, sock[TIPC_NLA_SOCK_CON],
- NULL);
+ nla_parse_nested(con, TIPC_NLA_CON_MAX,
+ sock[TIPC_NLA_SOCK_CON], NULL, NULL);
node = nla_get_u32(con[TIPC_NLA_CON_NODE]);
tipc_tlv_sprintf(msg->rep, " connected to <%u.%u.%u:%u>",
@@ -977,8 +977,8 @@
if (!attrs[TIPC_NLA_MEDIA])
return -EINVAL;
- err = nla_parse_nested(media, TIPC_NLA_MEDIA_MAX, attrs[TIPC_NLA_MEDIA],
- NULL);
+ err = nla_parse_nested(media, TIPC_NLA_MEDIA_MAX,
+ attrs[TIPC_NLA_MEDIA], NULL, NULL);
if (err)
return err;
@@ -998,7 +998,7 @@
return -EINVAL;
err = nla_parse_nested(node, TIPC_NLA_NODE_MAX, attrs[TIPC_NLA_NODE],
- NULL);
+ NULL, NULL);
if (err)
return err;
@@ -1045,7 +1045,7 @@
return -EINVAL;
err = nla_parse_nested(net, TIPC_NLA_NET_MAX, attrs[TIPC_NLA_NET],
- NULL);
+ NULL, NULL);
if (err)
return err;
diff --git a/net/tipc/node.c b/net/tipc/node.c
index 4512e83..aeef801 100644
--- a/net/tipc/node.c
+++ b/net/tipc/node.c
@@ -1607,8 +1607,8 @@
return -EINVAL;
err = nla_parse_nested(attrs, TIPC_NLA_NET_MAX,
- info->attrs[TIPC_NLA_NET],
- tipc_nl_net_policy);
+ info->attrs[TIPC_NLA_NET], tipc_nl_net_policy,
+ info->extack);
if (err)
return err;
@@ -1774,7 +1774,7 @@
err = nla_parse_nested(attrs, TIPC_NLA_LINK_MAX,
info->attrs[TIPC_NLA_LINK],
- tipc_nl_link_policy);
+ tipc_nl_link_policy, info->extack);
if (err)
return err;
@@ -1902,7 +1902,7 @@
err = nla_parse_nested(attrs, TIPC_NLA_LINK_MAX,
info->attrs[TIPC_NLA_LINK],
- tipc_nl_link_policy);
+ tipc_nl_link_policy, info->extack);
if (err)
return err;
@@ -2042,7 +2042,7 @@
err = nla_parse_nested(attrs, TIPC_NLA_MON_MAX,
info->attrs[TIPC_NLA_MON],
- tipc_nl_monitor_policy);
+ tipc_nl_monitor_policy, info->extack);
if (err)
return err;
@@ -2098,6 +2098,8 @@
int err;
msg.skb = nlmsg_new(NLMSG_GOODSIZE, GFP_KERNEL);
+ if (!msg.skb)
+ return -ENOMEM;
msg.portid = info->snd_portid;
msg.seq = info->snd_seq;
@@ -2163,7 +2165,7 @@
err = nla_parse_nested(mon, TIPC_NLA_MON_MAX,
attrs[TIPC_NLA_MON],
- tipc_nl_monitor_policy);
+ tipc_nl_monitor_policy, NULL);
if (err)
return err;
diff --git a/net/tipc/socket.c b/net/tipc/socket.c
index 7130e73..0d4f2f4 100644
--- a/net/tipc/socket.c
+++ b/net/tipc/socket.c
@@ -51,6 +51,7 @@
#define TIPC_FWD_MSG 1
#define TIPC_MAX_PORT 0xffffffff
#define TIPC_MIN_PORT 1
+#define TIPC_ACK_RATE 4 /* ACK at 1/4 of of rcv window size */
enum {
TIPC_LISTEN = TCP_LISTEN,
@@ -866,6 +867,14 @@
if (!tsk_peer_msg(tsk, hdr))
goto exit;
+ if (unlikely(msg_errcode(hdr))) {
+ tipc_set_sk_state(sk, TIPC_DISCONNECTING);
+ tipc_node_remove_conn(sock_net(sk), tsk_peer_node(tsk),
+ tsk_peer_port(tsk));
+ sk->sk_state_change(sk);
+ goto exit;
+ }
+
tsk->probe_unacked = false;
if (mtyp == CONN_PROBE) {
@@ -1083,7 +1092,7 @@
}
} while (sent < dlen && !rc);
- return rc ? rc : sent;
+ return sent ? sent : rc;
}
/**
@@ -1259,7 +1268,10 @@
struct sock *sk = sock->sk;
DEFINE_WAIT(wait);
long timeo = *timeop;
- int err;
+ int err = sock_error(sk);
+
+ if (err)
+ return err;
for (;;) {
prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
@@ -1281,6 +1293,10 @@
err = sock_intr_errno(timeo);
if (signal_pending(current))
break;
+
+ err = sock_error(sk);
+ if (err)
+ break;
}
finish_wait(sk_sleep(sk), &wait);
*timeop = timeo;
@@ -1290,7 +1306,7 @@
/**
* tipc_recvmsg - receive packet-oriented message
* @m: descriptor for message info
- * @buf_len: total size of user buffer area
+ * @buflen: length of user buffer area
* @flags: receive flags
*
* Used for SOCK_DGRAM, SOCK_RDM, and SOCK_SEQPACKET messages.
@@ -1298,95 +1314,85 @@
*
* Returns size of returned message data, errno otherwise
*/
-static int tipc_recvmsg(struct socket *sock, struct msghdr *m, size_t buf_len,
- int flags)
+static int tipc_recvmsg(struct socket *sock, struct msghdr *m,
+ size_t buflen, int flags)
{
struct sock *sk = sock->sk;
struct tipc_sock *tsk = tipc_sk(sk);
- struct sk_buff *buf;
- struct tipc_msg *msg;
- bool is_connectionless = tipc_sk_type_connectionless(sk);
- long timeo;
- unsigned int sz;
- u32 err;
- int res, hlen;
+ struct sk_buff *skb;
+ struct tipc_msg *hdr;
+ bool connected = !tipc_sk_type_connectionless(sk);
+ int rc, err, hlen, dlen, copy;
+ long timeout;
/* Catch invalid receive requests */
- if (unlikely(!buf_len))
+ if (unlikely(!buflen))
return -EINVAL;
lock_sock(sk);
-
- if (!is_connectionless && unlikely(sk->sk_state == TIPC_OPEN)) {
- res = -ENOTCONN;
+ if (unlikely(connected && sk->sk_state == TIPC_OPEN)) {
+ rc = -ENOTCONN;
goto exit;
}
+ timeout = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
- timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
-restart:
-
- /* Look for a message in receive queue; wait if necessary */
- res = tipc_wait_for_rcvmsg(sock, &timeo);
- if (res)
- goto exit;
-
- /* Look at first message in receive queue */
- buf = skb_peek(&sk->sk_receive_queue);
- msg = buf_msg(buf);
- sz = msg_data_sz(msg);
- hlen = msg_hdr_sz(msg);
- err = msg_errcode(msg);
-
- /* Discard an empty non-errored message & try again */
- if ((!sz) && (!err)) {
- tsk_advance_rx_queue(sk);
- goto restart;
- }
-
- /* Capture sender's address (optional) */
- set_orig_addr(m, msg);
-
- /* Capture ancillary data (optional) */
- res = tipc_sk_anc_data_recv(m, msg, tsk);
- if (res)
- goto exit;
-
- /* Capture message data (if valid) & compute return value (always) */
- if (!err) {
- if (unlikely(buf_len < sz)) {
- sz = buf_len;
- m->msg_flags |= MSG_TRUNC;
- }
- res = skb_copy_datagram_msg(buf, hlen, m, sz);
- if (res)
+ do {
+ /* Look at first msg in receive queue; wait if necessary */
+ rc = tipc_wait_for_rcvmsg(sock, &timeout);
+ if (unlikely(rc))
goto exit;
- res = sz;
- } else {
- if (is_connectionless || err == TIPC_CONN_SHUTDOWN ||
- m->msg_control)
- res = 0;
- else
- res = -ECONNRESET;
- }
+ skb = skb_peek(&sk->sk_receive_queue);
+ hdr = buf_msg(skb);
+ dlen = msg_data_sz(hdr);
+ hlen = msg_hdr_sz(hdr);
+ err = msg_errcode(hdr);
+ if (likely(dlen || err))
+ break;
+ tsk_advance_rx_queue(sk);
+ } while (1);
+ /* Collect msg meta data, including error code and rejected data */
+ set_orig_addr(m, hdr);
+ rc = tipc_sk_anc_data_recv(m, hdr, tsk);
+ if (unlikely(rc))
+ goto exit;
+
+ /* Capture data if non-error msg, otherwise just set return value */
+ if (likely(!err)) {
+ copy = min_t(int, dlen, buflen);
+ if (unlikely(copy != dlen))
+ m->msg_flags |= MSG_TRUNC;
+ rc = skb_copy_datagram_msg(skb, hlen, m, copy);
+ } else {
+ copy = 0;
+ rc = 0;
+ if (err != TIPC_CONN_SHUTDOWN && connected && !m->msg_control)
+ rc = -ECONNRESET;
+ }
+ if (unlikely(rc))
+ goto exit;
+
+ /* Caption of data or error code/rejected data was successful */
if (unlikely(flags & MSG_PEEK))
goto exit;
- if (likely(!is_connectionless)) {
- tsk->rcv_unacked += tsk_inc(tsk, hlen + sz);
- if (unlikely(tsk->rcv_unacked >= (tsk->rcv_win / 4)))
- tipc_sk_send_ack(tsk);
- }
tsk_advance_rx_queue(sk);
+ if (likely(!connected))
+ goto exit;
+
+ /* Send connection flow control ack when applicable */
+ tsk->rcv_unacked += tsk_inc(tsk, hlen + dlen);
+ if (tsk->rcv_unacked >= tsk->rcv_win / TIPC_ACK_RATE)
+ tipc_sk_send_ack(tsk);
exit:
release_sock(sk);
- return res;
+ return rc ? rc : copy;
}
/**
- * tipc_recv_stream - receive stream-oriented data
+ * tipc_recvstream - receive stream-oriented data
* @m: descriptor for message info
- * @buf_len: total size of user buffer area
+ * @buflen: total size of user buffer area
* @flags: receive flags
*
* Used for SOCK_STREAM messages only. If not enough data is available
@@ -1394,111 +1400,98 @@
*
* Returns size of returned message data, errno otherwise
*/
-static int tipc_recv_stream(struct socket *sock, struct msghdr *m,
- size_t buf_len, int flags)
+static int tipc_recvstream(struct socket *sock, struct msghdr *m,
+ size_t buflen, int flags)
{
struct sock *sk = sock->sk;
struct tipc_sock *tsk = tipc_sk(sk);
- struct sk_buff *buf;
- struct tipc_msg *msg;
- long timeo;
- unsigned int sz;
- int target;
- int sz_copied = 0;
- u32 err;
- int res = 0, hlen;
+ struct sk_buff *skb;
+ struct tipc_msg *hdr;
+ struct tipc_skb_cb *skb_cb;
+ bool peek = flags & MSG_PEEK;
+ int offset, required, copy, copied = 0;
+ int hlen, dlen, err, rc;
+ long timeout;
/* Catch invalid receive attempts */
- if (unlikely(!buf_len))
+ if (unlikely(!buflen))
return -EINVAL;
lock_sock(sk);
if (unlikely(sk->sk_state == TIPC_OPEN)) {
- res = -ENOTCONN;
+ rc = -ENOTCONN;
goto exit;
}
+ required = sock_rcvlowat(sk, flags & MSG_WAITALL, buflen);
+ timeout = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
- target = sock_rcvlowat(sk, flags & MSG_WAITALL, buf_len);
- timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);
+ do {
+ /* Look at first msg in receive queue; wait if necessary */
+ rc = tipc_wait_for_rcvmsg(sock, &timeout);
+ if (unlikely(rc))
+ break;
+ skb = skb_peek(&sk->sk_receive_queue);
+ skb_cb = TIPC_SKB_CB(skb);
+ hdr = buf_msg(skb);
+ dlen = msg_data_sz(hdr);
+ hlen = msg_hdr_sz(hdr);
+ err = msg_errcode(hdr);
-restart:
- /* Look for a message in receive queue; wait if necessary */
- res = tipc_wait_for_rcvmsg(sock, &timeo);
- if (res)
- goto exit;
-
- /* Look at first message in receive queue */
- buf = skb_peek(&sk->sk_receive_queue);
- msg = buf_msg(buf);
- sz = msg_data_sz(msg);
- hlen = msg_hdr_sz(msg);
- err = msg_errcode(msg);
-
- /* Discard an empty non-errored message & try again */
- if ((!sz) && (!err)) {
- tsk_advance_rx_queue(sk);
- goto restart;
- }
-
- /* Optionally capture sender's address & ancillary data of first msg */
- if (sz_copied == 0) {
- set_orig_addr(m, msg);
- res = tipc_sk_anc_data_recv(m, msg, tsk);
- if (res)
- goto exit;
- }
-
- /* Capture message data (if valid) & compute return value (always) */
- if (!err) {
- u32 offset = TIPC_SKB_CB(buf)->bytes_read;
- u32 needed;
- int sz_to_copy;
-
- sz -= offset;
- needed = (buf_len - sz_copied);
- sz_to_copy = min(sz, needed);
-
- res = skb_copy_datagram_msg(buf, hlen + offset, m, sz_to_copy);
- if (res)
- goto exit;
-
- sz_copied += sz_to_copy;
-
- if (sz_to_copy < sz) {
- if (!(flags & MSG_PEEK))
- TIPC_SKB_CB(buf)->bytes_read =
- offset + sz_to_copy;
- goto exit;
+ /* Discard any empty non-errored (SYN-) message */
+ if (unlikely(!dlen && !err)) {
+ tsk_advance_rx_queue(sk);
+ continue;
}
- } else {
- if (sz_copied != 0)
- goto exit; /* can't add error msg to valid data */
- if ((err == TIPC_CONN_SHUTDOWN) || m->msg_control)
- res = 0;
- else
- res = -ECONNRESET;
- }
+ /* Collect msg meta data, incl. error code and rejected data */
+ if (!copied) {
+ set_orig_addr(m, hdr);
+ rc = tipc_sk_anc_data_recv(m, hdr, tsk);
+ if (rc)
+ break;
+ }
- if (unlikely(flags & MSG_PEEK))
- goto exit;
+ /* Copy data if msg ok, otherwise return error/partial data */
+ if (likely(!err)) {
+ offset = skb_cb->bytes_read;
+ copy = min_t(int, dlen - offset, buflen - copied);
+ rc = skb_copy_datagram_msg(skb, hlen + offset, m, copy);
+ if (unlikely(rc))
+ break;
+ copied += copy;
+ offset += copy;
+ if (unlikely(offset < dlen)) {
+ if (!peek)
+ skb_cb->bytes_read = offset;
+ break;
+ }
+ } else {
+ rc = 0;
+ if ((err != TIPC_CONN_SHUTDOWN) && !m->msg_control)
+ rc = -ECONNRESET;
+ if (copied || rc)
+ break;
+ }
- tsk->rcv_unacked += tsk_inc(tsk, hlen + sz);
- if (unlikely(tsk->rcv_unacked >= (tsk->rcv_win / 4)))
- tipc_sk_send_ack(tsk);
- tsk_advance_rx_queue(sk);
+ if (unlikely(peek))
+ break;
- /* Loop around if more data is required */
- if ((sz_copied < buf_len) && /* didn't get all requested data */
- (!skb_queue_empty(&sk->sk_receive_queue) ||
- (sz_copied < target)) && /* and more is ready or required */
- (!err)) /* and haven't reached a FIN */
- goto restart;
+ tsk_advance_rx_queue(sk);
+ /* Send connection flow control advertisement when applicable */
+ tsk->rcv_unacked += tsk_inc(tsk, hlen + dlen);
+ if (unlikely(tsk->rcv_unacked >= tsk->rcv_win / TIPC_ACK_RATE))
+ tipc_sk_send_ack(tsk);
+
+ /* Exit if all requested data or FIN/error received */
+ if (copied == buflen || err)
+ break;
+
+ } while (!skb_queue_empty(&sk->sk_receive_queue) || copied < required);
exit:
release_sock(sk);
- return sz_copied ? sz_copied : res;
+ return copied ? copied : rc;
}
/**
@@ -1551,6 +1544,8 @@
struct sock *sk = &tsk->sk;
struct net *net = sock_net(sk);
struct tipc_msg *hdr = buf_msg(skb);
+ u32 pport = msg_origport(hdr);
+ u32 pnode = msg_orignode(hdr);
if (unlikely(msg_mcast(hdr)))
return false;
@@ -1558,18 +1553,28 @@
switch (sk->sk_state) {
case TIPC_CONNECTING:
/* Accept only ACK or NACK message */
- if (unlikely(!msg_connected(hdr)))
- return false;
+ if (unlikely(!msg_connected(hdr))) {
+ if (pport != tsk_peer_port(tsk) ||
+ pnode != tsk_peer_node(tsk))
+ return false;
+
+ tipc_set_sk_state(sk, TIPC_DISCONNECTING);
+ sk->sk_err = ECONNREFUSED;
+ sk->sk_state_change(sk);
+ return true;
+ }
if (unlikely(msg_errcode(hdr))) {
tipc_set_sk_state(sk, TIPC_DISCONNECTING);
sk->sk_err = ECONNREFUSED;
+ sk->sk_state_change(sk);
return true;
}
if (unlikely(!msg_isdata(hdr))) {
tipc_set_sk_state(sk, TIPC_DISCONNECTING);
sk->sk_err = EINVAL;
+ sk->sk_state_change(sk);
return true;
}
@@ -1581,8 +1586,7 @@
return true;
/* If empty 'ACK-' message, wake up sleeping connect() */
- if (waitqueue_active(sk_sleep(sk)))
- wake_up_interruptible(sk_sleep(sk));
+ sk->sk_data_ready(sk);
/* 'ACK-' message is neither accepted nor rejected: */
msg_set_dest_droppable(hdr, 1);
@@ -2511,6 +2515,28 @@
}
}
+static int tipc_socketpair(struct socket *sock1, struct socket *sock2)
+{
+ struct tipc_sock *tsk2 = tipc_sk(sock2->sk);
+ struct tipc_sock *tsk1 = tipc_sk(sock1->sk);
+ u32 onode = tipc_own_addr(sock_net(sock1->sk));
+
+ tsk1->peer.family = AF_TIPC;
+ tsk1->peer.addrtype = TIPC_ADDR_ID;
+ tsk1->peer.scope = TIPC_NODE_SCOPE;
+ tsk1->peer.addr.id.ref = tsk2->portid;
+ tsk1->peer.addr.id.node = onode;
+ tsk2->peer.family = AF_TIPC;
+ tsk2->peer.addrtype = TIPC_ADDR_ID;
+ tsk2->peer.scope = TIPC_NODE_SCOPE;
+ tsk2->peer.addr.id.ref = tsk1->portid;
+ tsk2->peer.addr.id.node = onode;
+
+ tipc_sk_finish_conn(tsk1, tsk2->portid, onode);
+ tipc_sk_finish_conn(tsk2, tsk1->portid, onode);
+ return 0;
+}
+
/* Protocol switches for the various types of TIPC sockets */
static const struct proto_ops msg_ops = {
@@ -2519,7 +2545,7 @@
.release = tipc_release,
.bind = tipc_bind,
.connect = tipc_connect,
- .socketpair = sock_no_socketpair,
+ .socketpair = tipc_socketpair,
.accept = sock_no_accept,
.getname = tipc_getname,
.poll = tipc_poll,
@@ -2540,7 +2566,7 @@
.release = tipc_release,
.bind = tipc_bind,
.connect = tipc_connect,
- .socketpair = sock_no_socketpair,
+ .socketpair = tipc_socketpair,
.accept = tipc_accept,
.getname = tipc_getname,
.poll = tipc_poll,
@@ -2561,7 +2587,7 @@
.release = tipc_release,
.bind = tipc_bind,
.connect = tipc_connect,
- .socketpair = sock_no_socketpair,
+ .socketpair = tipc_socketpair,
.accept = tipc_accept,
.getname = tipc_getname,
.poll = tipc_poll,
@@ -2571,7 +2597,7 @@
.setsockopt = tipc_setsockopt,
.getsockopt = tipc_getsockopt,
.sendmsg = tipc_sendstream,
- .recvmsg = tipc_recv_stream,
+ .recvmsg = tipc_recvstream,
.mmap = sock_no_mmap,
.sendpage = sock_no_sendpage
};
@@ -2844,7 +2870,7 @@
err = nla_parse_nested(sock, TIPC_NLA_SOCK_MAX,
attrs[TIPC_NLA_SOCK],
- tipc_nl_sock_policy);
+ tipc_nl_sock_policy, NULL);
if (err)
return err;
diff --git a/net/tipc/subscr.c b/net/tipc/subscr.c
index 271cd66..0bf91cd 100644
--- a/net/tipc/subscr.c
+++ b/net/tipc/subscr.c
@@ -54,8 +54,6 @@
static void tipc_subscrp_delete(struct tipc_subscription *sub);
static void tipc_subscrb_put(struct tipc_subscriber *subscriber);
-static void tipc_subscrp_put(struct tipc_subscription *subscription);
-static void tipc_subscrp_get(struct tipc_subscription *subscription);
/**
* htohl - convert value to endianness used by destination
@@ -125,7 +123,6 @@
{
struct tipc_name_seq seq;
- tipc_subscrp_get(sub);
tipc_subscrp_convert_seq(&sub->evt.s.seq, sub->swap, &seq);
if (!tipc_subscrp_check_overlap(&seq, found_lower, found_upper))
return;
@@ -135,7 +132,6 @@
tipc_subscrp_send_event(sub, found_lower, found_upper, event, port_ref,
node);
- tipc_subscrp_put(sub);
}
static void tipc_subscrp_timeout(unsigned long data)
@@ -145,6 +141,7 @@
spin_lock_bh(&subscriber->lock);
tipc_nametbl_unsubscribe(sub);
+ list_del(&sub->subscrp_list);
spin_unlock_bh(&subscriber->lock);
/* Notify subscriber of timeout */
@@ -177,20 +174,17 @@
struct tipc_net *tn = net_generic(sub->net, tipc_net_id);
struct tipc_subscriber *subscriber = sub->subscriber;
- spin_lock_bh(&subscriber->lock);
- list_del(&sub->subscrp_list);
atomic_dec(&tn->subscription_count);
- spin_unlock_bh(&subscriber->lock);
kfree(sub);
tipc_subscrb_put(subscriber);
}
-static void tipc_subscrp_put(struct tipc_subscription *subscription)
+void tipc_subscrp_put(struct tipc_subscription *subscription)
{
kref_put(&subscription->kref, tipc_subscrp_kref_release);
}
-static void tipc_subscrp_get(struct tipc_subscription *subscription)
+void tipc_subscrp_get(struct tipc_subscription *subscription)
{
kref_get(&subscription->kref);
}
@@ -210,11 +204,8 @@
continue;
tipc_nametbl_unsubscribe(sub);
- tipc_subscrp_get(sub);
- spin_unlock_bh(&subscriber->lock);
+ list_del(&sub->subscrp_list);
tipc_subscrp_delete(sub);
- tipc_subscrp_put(sub);
- spin_lock_bh(&subscriber->lock);
if (s)
break;
diff --git a/net/tipc/subscr.h b/net/tipc/subscr.h
index ffdc214..ee52957 100644
--- a/net/tipc/subscr.h
+++ b/net/tipc/subscr.h
@@ -78,4 +78,7 @@
int tipc_topsrv_start(struct net *net);
void tipc_topsrv_stop(struct net *net);
+void tipc_subscrp_put(struct tipc_subscription *subscription);
+void tipc_subscrp_get(struct tipc_subscription *subscription);
+
#endif
diff --git a/net/tipc/udp_media.c b/net/tipc/udp_media.c
index 46061cf..ecca64f 100644
--- a/net/tipc/udp_media.c
+++ b/net/tipc/udp_media.c
@@ -457,7 +457,7 @@
err = nla_parse_nested(battrs, TIPC_NLA_BEARER_MAX,
attrs[TIPC_NLA_BEARER],
- tipc_nl_bearer_policy);
+ tipc_nl_bearer_policy, NULL);
if (err)
return err;
@@ -609,7 +609,8 @@
struct nlattr *opts[TIPC_NLA_UDP_MAX + 1];
struct udp_media_addr *dst;
- if (nla_parse_nested(opts, TIPC_NLA_UDP_MAX, attr, tipc_nl_udp_policy))
+ if (nla_parse_nested(opts, TIPC_NLA_UDP_MAX, attr,
+ tipc_nl_udp_policy, NULL))
return -EINVAL;
if (!opts[TIPC_NLA_UDP_REMOTE])
@@ -662,7 +663,7 @@
if (nla_parse_nested(opts, TIPC_NLA_UDP_MAX,
attrs[TIPC_NLA_BEARER_UDP_OPTS],
- tipc_nl_udp_policy))
+ tipc_nl_udp_policy, NULL))
goto err;
if (!opts[TIPC_NLA_UDP_LOCAL] || !opts[TIPC_NLA_UDP_REMOTE]) {
diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c
index 928691c..6a7fe76 100644
--- a/net/unix/af_unix.c
+++ b/net/unix/af_unix.c
@@ -996,7 +996,7 @@
unsigned int hash;
struct unix_address *addr;
struct hlist_head *list;
- struct path path = { NULL, NULL };
+ struct path path = { };
err = -EINVAL;
if (sunaddr->sun_family != AF_UNIX)
diff --git a/net/vmw_vsock/Makefile b/net/vmw_vsock/Makefile
index bc27c70..09fc2eb 100644
--- a/net/vmw_vsock/Makefile
+++ b/net/vmw_vsock/Makefile
@@ -3,7 +3,7 @@
obj-$(CONFIG_VIRTIO_VSOCKETS) += vmw_vsock_virtio_transport.o
obj-$(CONFIG_VIRTIO_VSOCKETS_COMMON) += vmw_vsock_virtio_transport_common.o
-vsock-y += af_vsock.o vsock_addr.o
+vsock-y += af_vsock.o af_vsock_tap.o vsock_addr.o
vmw_vsock_vmci_transport-y += vmci_transport.o vmci_transport_notify.o \
vmci_transport_notify_qstate.o
diff --git a/net/vmw_vsock/af_vsock_tap.c b/net/vmw_vsock/af_vsock_tap.c
new file mode 100644
index 0000000..98f09b5
--- /dev/null
+++ b/net/vmw_vsock/af_vsock_tap.c
@@ -0,0 +1,114 @@
+/*
+ * Tap functions for AF_VSOCK sockets.
+ *
+ * Code based on net/netlink/af_netlink.c tap functions.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#include <linux/module.h>
+#include <net/sock.h>
+#include <net/af_vsock.h>
+#include <linux/if_arp.h>
+
+static DEFINE_SPINLOCK(vsock_tap_lock);
+static struct list_head vsock_tap_all __read_mostly =
+ LIST_HEAD_INIT(vsock_tap_all);
+
+int vsock_add_tap(struct vsock_tap *vt)
+{
+ if (unlikely(vt->dev->type != ARPHRD_VSOCKMON))
+ return -EINVAL;
+
+ __module_get(vt->module);
+
+ spin_lock(&vsock_tap_lock);
+ list_add_rcu(&vt->list, &vsock_tap_all);
+ spin_unlock(&vsock_tap_lock);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(vsock_add_tap);
+
+int vsock_remove_tap(struct vsock_tap *vt)
+{
+ struct vsock_tap *tmp;
+ bool found = false;
+
+ spin_lock(&vsock_tap_lock);
+
+ list_for_each_entry(tmp, &vsock_tap_all, list) {
+ if (vt == tmp) {
+ list_del_rcu(&vt->list);
+ found = true;
+ goto out;
+ }
+ }
+
+ pr_warn("vsock_remove_tap: %p not found\n", vt);
+out:
+ spin_unlock(&vsock_tap_lock);
+
+ synchronize_net();
+
+ if (found)
+ module_put(vt->module);
+
+ return found ? 0 : -ENODEV;
+}
+EXPORT_SYMBOL_GPL(vsock_remove_tap);
+
+static int __vsock_deliver_tap_skb(struct sk_buff *skb,
+ struct net_device *dev)
+{
+ int ret = 0;
+ struct sk_buff *nskb = skb_clone(skb, GFP_ATOMIC);
+
+ if (nskb) {
+ dev_hold(dev);
+
+ nskb->dev = dev;
+ ret = dev_queue_xmit(nskb);
+ if (unlikely(ret > 0))
+ ret = net_xmit_errno(ret);
+
+ dev_put(dev);
+ }
+
+ return ret;
+}
+
+static void __vsock_deliver_tap(struct sk_buff *skb)
+{
+ int ret;
+ struct vsock_tap *tmp;
+
+ list_for_each_entry_rcu(tmp, &vsock_tap_all, list) {
+ ret = __vsock_deliver_tap_skb(skb, tmp->dev);
+ if (unlikely(ret))
+ break;
+ }
+}
+
+void vsock_deliver_tap(struct sk_buff *build_skb(void *opaque), void *opaque)
+{
+ struct sk_buff *skb;
+
+ rcu_read_lock();
+
+ if (likely(list_empty(&vsock_tap_all)))
+ goto out;
+
+ skb = build_skb(opaque);
+ if (skb) {
+ __vsock_deliver_tap(skb);
+ consume_skb(skb);
+ }
+
+out:
+ rcu_read_unlock();
+}
+EXPORT_SYMBOL_GPL(vsock_deliver_tap);
diff --git a/net/vmw_vsock/virtio_transport.c b/net/vmw_vsock/virtio_transport.c
index 68675a1..9dffe02 100644
--- a/net/vmw_vsock/virtio_transport.c
+++ b/net/vmw_vsock/virtio_transport.c
@@ -144,6 +144,8 @@
list_del_init(&pkt->list);
spin_unlock_bh(&vsock->send_pkt_list_lock);
+ virtio_transport_deliver_tap_pkt(pkt);
+
reply = pkt->reply;
sg_init_one(&hdr, &pkt->hdr, sizeof(pkt->hdr));
@@ -370,6 +372,7 @@
}
pkt->len = len - sizeof(pkt->hdr);
+ virtio_transport_deliver_tap_pkt(pkt);
virtio_transport_recv_pkt(pkt);
}
} while (!virtqueue_enable_cb(vq));
diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
index af087b4..18e2479 100644
--- a/net/vmw_vsock/virtio_transport_common.c
+++ b/net/vmw_vsock/virtio_transport_common.c
@@ -16,6 +16,7 @@
#include <linux/virtio_ids.h>
#include <linux/virtio_config.h>
#include <linux/virtio_vsock.h>
+#include <uapi/linux/vsockmon.h>
#include <net/sock.h>
#include <net/af_vsock.h>
@@ -85,6 +86,69 @@
return NULL;
}
+/* Packet capture */
+static struct sk_buff *virtio_transport_build_skb(void *opaque)
+{
+ struct virtio_vsock_pkt *pkt = opaque;
+ unsigned char *t_hdr, *payload;
+ struct af_vsockmon_hdr *hdr;
+ struct sk_buff *skb;
+
+ skb = alloc_skb(sizeof(*hdr) + sizeof(pkt->hdr) + pkt->len,
+ GFP_ATOMIC);
+ if (!skb)
+ return NULL;
+
+ hdr = (struct af_vsockmon_hdr *)skb_put(skb, sizeof(*hdr));
+
+ /* pkt->hdr is little-endian so no need to byteswap here */
+ hdr->src_cid = pkt->hdr.src_cid;
+ hdr->src_port = pkt->hdr.src_port;
+ hdr->dst_cid = pkt->hdr.dst_cid;
+ hdr->dst_port = pkt->hdr.dst_port;
+
+ hdr->transport = cpu_to_le16(AF_VSOCK_TRANSPORT_VIRTIO);
+ hdr->len = cpu_to_le16(sizeof(pkt->hdr));
+ memset(hdr->reserved, 0, sizeof(hdr->reserved));
+
+ switch (le16_to_cpu(pkt->hdr.op)) {
+ case VIRTIO_VSOCK_OP_REQUEST:
+ case VIRTIO_VSOCK_OP_RESPONSE:
+ hdr->op = cpu_to_le16(AF_VSOCK_OP_CONNECT);
+ break;
+ case VIRTIO_VSOCK_OP_RST:
+ case VIRTIO_VSOCK_OP_SHUTDOWN:
+ hdr->op = cpu_to_le16(AF_VSOCK_OP_DISCONNECT);
+ break;
+ case VIRTIO_VSOCK_OP_RW:
+ hdr->op = cpu_to_le16(AF_VSOCK_OP_PAYLOAD);
+ break;
+ case VIRTIO_VSOCK_OP_CREDIT_UPDATE:
+ case VIRTIO_VSOCK_OP_CREDIT_REQUEST:
+ hdr->op = cpu_to_le16(AF_VSOCK_OP_CONTROL);
+ break;
+ default:
+ hdr->op = cpu_to_le16(AF_VSOCK_OP_UNKNOWN);
+ break;
+ }
+
+ t_hdr = skb_put(skb, sizeof(pkt->hdr));
+ memcpy(t_hdr, &pkt->hdr, sizeof(pkt->hdr));
+
+ if (pkt->len) {
+ payload = skb_put(skb, pkt->len);
+ memcpy(payload, pkt->buf, pkt->len);
+ }
+
+ return skb;
+}
+
+void virtio_transport_deliver_tap_pkt(struct virtio_vsock_pkt *pkt)
+{
+ vsock_deliver_tap(virtio_transport_build_skb, pkt);
+}
+EXPORT_SYMBOL_GPL(virtio_transport_deliver_tap_pkt);
+
static int virtio_transport_send_pkt_info(struct vsock_sock *vsk,
struct virtio_vsock_pkt_info *info)
{
diff --git a/net/vmw_vsock/vmci_transport.c b/net/vmw_vsock/vmci_transport.c
index 4be4fbb..10ae782 100644
--- a/net/vmw_vsock/vmci_transport.c
+++ b/net/vmw_vsock/vmci_transport.c
@@ -96,31 +96,23 @@
static s32 vmci_transport_error_to_vsock_error(s32 vmci_error)
{
- int err;
-
switch (vmci_error) {
case VMCI_ERROR_NO_MEM:
- err = ENOMEM;
- break;
+ return -ENOMEM;
case VMCI_ERROR_DUPLICATE_ENTRY:
case VMCI_ERROR_ALREADY_EXISTS:
- err = EADDRINUSE;
- break;
+ return -EADDRINUSE;
case VMCI_ERROR_NO_ACCESS:
- err = EPERM;
- break;
+ return -EPERM;
case VMCI_ERROR_NO_RESOURCES:
- err = ENOBUFS;
- break;
+ return -ENOBUFS;
case VMCI_ERROR_INVALID_RESOURCE:
- err = EHOSTUNREACH;
- break;
+ return -EHOSTUNREACH;
case VMCI_ERROR_INVALID_ARGS:
default:
- err = EINVAL;
+ break;
}
-
- return err > 0 ? -err : err;
+ return -EINVAL;
}
static u32 vmci_transport_peer_rid(u32 peer_cid)
diff --git a/net/wireless/ap.c b/net/wireless/ap.c
index bdad1f9..25666d3 100644
--- a/net/wireless/ap.c
+++ b/net/wireless/ap.c
@@ -32,6 +32,11 @@
rdev_set_qos_map(rdev, dev, NULL);
if (notify)
nl80211_send_ap_stopped(wdev);
+
+ /* Should we apply the grace period during beaconing interface
+ * shutdown also?
+ */
+ cfg80211_sched_dfs_chan_update(rdev);
}
return err;
diff --git a/net/wireless/chan.c b/net/wireless/chan.c
index 5497d022..b8aa5a7 100644
--- a/net/wireless/chan.c
+++ b/net/wireless/chan.c
@@ -456,6 +456,123 @@
return (r1 + r2 > 0);
}
+/*
+ * Checks if center frequency of chan falls with in the bandwidth
+ * range of chandef.
+ */
+bool cfg80211_is_sub_chan(struct cfg80211_chan_def *chandef,
+ struct ieee80211_channel *chan)
+{
+ int width;
+ u32 cf_offset, freq;
+
+ if (chandef->chan->center_freq == chan->center_freq)
+ return true;
+
+ width = cfg80211_chandef_get_width(chandef);
+ if (width <= 20)
+ return false;
+
+ cf_offset = width / 2 - 10;
+
+ for (freq = chandef->center_freq1 - width / 2 + 10;
+ freq <= chandef->center_freq1 + width / 2 - 10; freq += 20) {
+ if (chan->center_freq == freq)
+ return true;
+ }
+
+ if (!chandef->center_freq2)
+ return false;
+
+ for (freq = chandef->center_freq2 - width / 2 + 10;
+ freq <= chandef->center_freq2 + width / 2 - 10; freq += 20) {
+ if (chan->center_freq == freq)
+ return true;
+ }
+
+ return false;
+}
+
+bool cfg80211_beaconing_iface_active(struct wireless_dev *wdev)
+{
+ bool active = false;
+
+ ASSERT_WDEV_LOCK(wdev);
+
+ if (!wdev->chandef.chan)
+ return false;
+
+ switch (wdev->iftype) {
+ case NL80211_IFTYPE_AP:
+ case NL80211_IFTYPE_P2P_GO:
+ active = wdev->beacon_interval != 0;
+ break;
+ case NL80211_IFTYPE_ADHOC:
+ active = wdev->ssid_len != 0;
+ break;
+ case NL80211_IFTYPE_MESH_POINT:
+ active = wdev->mesh_id_len != 0;
+ break;
+ case NL80211_IFTYPE_STATION:
+ case NL80211_IFTYPE_OCB:
+ case NL80211_IFTYPE_P2P_CLIENT:
+ case NL80211_IFTYPE_MONITOR:
+ case NL80211_IFTYPE_AP_VLAN:
+ case NL80211_IFTYPE_WDS:
+ case NL80211_IFTYPE_P2P_DEVICE:
+ /* Can NAN type be considered as beaconing interface? */
+ case NL80211_IFTYPE_NAN:
+ break;
+ case NL80211_IFTYPE_UNSPECIFIED:
+ case NUM_NL80211_IFTYPES:
+ WARN_ON(1);
+ }
+
+ return active;
+}
+
+static bool cfg80211_is_wiphy_oper_chan(struct wiphy *wiphy,
+ struct ieee80211_channel *chan)
+{
+ struct wireless_dev *wdev;
+
+ list_for_each_entry(wdev, &wiphy->wdev_list, list) {
+ wdev_lock(wdev);
+ if (!cfg80211_beaconing_iface_active(wdev)) {
+ wdev_unlock(wdev);
+ continue;
+ }
+
+ if (cfg80211_is_sub_chan(&wdev->chandef, chan)) {
+ wdev_unlock(wdev);
+ return true;
+ }
+ wdev_unlock(wdev);
+ }
+
+ return false;
+}
+
+bool cfg80211_any_wiphy_oper_chan(struct wiphy *wiphy,
+ struct ieee80211_channel *chan)
+{
+ struct cfg80211_registered_device *rdev;
+
+ ASSERT_RTNL();
+
+ if (!(chan->flags & IEEE80211_CHAN_RADAR))
+ return false;
+
+ list_for_each_entry(rdev, &cfg80211_rdev_list, list) {
+ if (!reg_dfs_domain_same(wiphy, &rdev->wiphy))
+ continue;
+
+ if (cfg80211_is_wiphy_oper_chan(&rdev->wiphy, chan))
+ return true;
+ }
+
+ return false;
+}
static bool cfg80211_get_chans_dfs_available(struct wiphy *wiphy,
u32 center_freq,
diff --git a/net/wireless/core.c b/net/wireless/core.c
index e55e05b..83ea164 100644
--- a/net/wireless/core.c
+++ b/net/wireless/core.c
@@ -305,30 +305,14 @@
void cfg80211_destroy_ifaces(struct cfg80211_registered_device *rdev)
{
- struct cfg80211_iface_destroy *item;
+ struct wireless_dev *wdev, *tmp;
ASSERT_RTNL();
- spin_lock_irq(&rdev->destroy_list_lock);
- while ((item = list_first_entry_or_null(&rdev->destroy_list,
- struct cfg80211_iface_destroy,
- list))) {
- struct wireless_dev *wdev, *tmp;
- u32 nlportid = item->nlportid;
-
- list_del(&item->list);
- kfree(item);
- spin_unlock_irq(&rdev->destroy_list_lock);
-
- list_for_each_entry_safe(wdev, tmp,
- &rdev->wiphy.wdev_list, list) {
- if (nlportid == wdev->owner_nlportid)
- rdev_del_virtual_intf(rdev, wdev);
- }
-
- spin_lock_irq(&rdev->destroy_list_lock);
+ list_for_each_entry_safe(wdev, tmp, &rdev->wiphy.wdev_list, list) {
+ if (wdev->nl_owner_dead)
+ rdev_del_virtual_intf(rdev, wdev);
}
- spin_unlock_irq(&rdev->destroy_list_lock);
}
static void cfg80211_destroy_iface_wk(struct work_struct *work)
@@ -346,13 +330,47 @@
static void cfg80211_sched_scan_stop_wk(struct work_struct *work)
{
struct cfg80211_registered_device *rdev;
+ struct cfg80211_sched_scan_request *req, *tmp;
rdev = container_of(work, struct cfg80211_registered_device,
sched_scan_stop_wk);
rtnl_lock();
+ list_for_each_entry_safe(req, tmp, &rdev->sched_scan_req_list, list) {
+ if (req->nl_owner_dead)
+ cfg80211_stop_sched_scan_req(rdev, req, false);
+ }
+ rtnl_unlock();
+}
- __cfg80211_stop_sched_scan(rdev, false);
+static void cfg80211_propagate_radar_detect_wk(struct work_struct *work)
+{
+ struct cfg80211_registered_device *rdev;
+
+ rdev = container_of(work, struct cfg80211_registered_device,
+ propagate_radar_detect_wk);
+
+ rtnl_lock();
+
+ regulatory_propagate_dfs_state(&rdev->wiphy, &rdev->radar_chandef,
+ NL80211_DFS_UNAVAILABLE,
+ NL80211_RADAR_DETECTED);
+
+ rtnl_unlock();
+}
+
+static void cfg80211_propagate_cac_done_wk(struct work_struct *work)
+{
+ struct cfg80211_registered_device *rdev;
+
+ rdev = container_of(work, struct cfg80211_registered_device,
+ propagate_cac_done_wk);
+
+ rtnl_lock();
+
+ regulatory_propagate_dfs_state(&rdev->wiphy, &rdev->cac_done_chandef,
+ NL80211_DFS_AVAILABLE,
+ NL80211_RADAR_CAC_FINISHED);
rtnl_unlock();
}
@@ -436,8 +454,8 @@
spin_lock_init(&rdev->beacon_registrations_lock);
spin_lock_init(&rdev->bss_lock);
INIT_LIST_HEAD(&rdev->bss_list);
+ INIT_LIST_HEAD(&rdev->sched_scan_req_list);
INIT_WORK(&rdev->scan_done_wk, __cfg80211_scan_done);
- INIT_WORK(&rdev->sched_scan_results_wk, __cfg80211_sched_scan_results);
INIT_LIST_HEAD(&rdev->mlme_unreg);
spin_lock_init(&rdev->mlme_unreg_lock);
INIT_WORK(&rdev->mlme_unreg_wk, cfg80211_mlme_unreg_wk);
@@ -452,10 +470,12 @@
rdev->wiphy.dev.platform_data = rdev;
device_enable_async_suspend(&rdev->wiphy.dev);
- INIT_LIST_HEAD(&rdev->destroy_list);
- spin_lock_init(&rdev->destroy_list_lock);
INIT_WORK(&rdev->destroy_work, cfg80211_destroy_iface_wk);
INIT_WORK(&rdev->sched_scan_stop_wk, cfg80211_sched_scan_stop_wk);
+ INIT_WORK(&rdev->sched_scan_res_wk, cfg80211_sched_scan_results_wk);
+ INIT_WORK(&rdev->propagate_radar_detect_wk,
+ cfg80211_propagate_radar_detect_wk);
+ INIT_WORK(&rdev->propagate_cac_done_wk, cfg80211_propagate_cac_done_wk);
#ifdef CONFIG_CFG80211_DEFAULT_PS
rdev->wiphy.flags |= WIPHY_FLAG_PS_ON_BY_DEFAULT;
@@ -915,6 +935,8 @@
flush_work(&rdev->destroy_work);
flush_work(&rdev->sched_scan_stop_wk);
flush_work(&rdev->mlme_unreg_wk);
+ flush_work(&rdev->propagate_radar_detect_wk);
+ flush_work(&rdev->propagate_cac_done_wk);
#ifdef CONFIG_PM
if (rdev->wiphy.wowlan_config && rdev->ops->set_wakeup)
@@ -954,6 +976,12 @@
}
EXPORT_SYMBOL(wiphy_rfkill_set_hw_state);
+void cfg80211_cqm_config_free(struct wireless_dev *wdev)
+{
+ kfree(wdev->cqm_config);
+ wdev->cqm_config = NULL;
+}
+
void cfg80211_unregister_wdev(struct wireless_dev *wdev)
{
struct cfg80211_registered_device *rdev = wiphy_to_rdev(wdev->wiphy);
@@ -980,6 +1008,8 @@
WARN_ON_ONCE(1);
break;
}
+
+ cfg80211_cqm_config_free(wdev);
}
EXPORT_SYMBOL(cfg80211_unregister_wdev);
@@ -1001,7 +1031,7 @@
struct wireless_dev *wdev)
{
struct net_device *dev = wdev->netdev;
- struct cfg80211_sched_scan_request *sched_scan_req;
+ struct cfg80211_sched_scan_request *pos, *tmp;
ASSERT_RTNL();
ASSERT_WDEV_LOCK(wdev);
@@ -1012,9 +1042,11 @@
break;
case NL80211_IFTYPE_P2P_CLIENT:
case NL80211_IFTYPE_STATION:
- sched_scan_req = rtnl_dereference(rdev->sched_scan_req);
- if (sched_scan_req && dev == sched_scan_req->dev)
- __cfg80211_stop_sched_scan(rdev, false);
+ list_for_each_entry_safe(pos, tmp, &rdev->sched_scan_req_list,
+ list) {
+ if (dev == pos->dev)
+ cfg80211_stop_sched_scan_req(rdev, pos, false);
+ }
#ifdef CONFIG_CFG80211_WEXT
kfree(wdev->wext.ie);
@@ -1089,7 +1121,7 @@
struct net_device *dev = netdev_notifier_info_to_dev(ptr);
struct wireless_dev *wdev = dev->ieee80211_ptr;
struct cfg80211_registered_device *rdev;
- struct cfg80211_sched_scan_request *sched_scan_req;
+ struct cfg80211_sched_scan_request *pos, *tmp;
if (!wdev)
return NOTIFY_DONE;
@@ -1114,7 +1146,15 @@
INIT_LIST_HEAD(&wdev->mgmt_registrations);
spin_lock_init(&wdev->mgmt_registrations_lock);
- wdev->identifier = ++rdev->wdev_id;
+ /*
+ * We get here also when the interface changes network namespaces,
+ * as it's registered into the new one, but we don't want it to
+ * change ID in that case. Checking if the ID is already assigned
+ * works, because 0 isn't considered a valid ID and the memory is
+ * 0-initialized.
+ */
+ if (!wdev->identifier)
+ wdev->identifier = ++rdev->wdev_id;
list_add_rcu(&wdev->list, &rdev->wiphy.wdev_list);
rdev->devlist_generation++;
/* can only change netns with wiphy */
@@ -1158,10 +1198,10 @@
___cfg80211_scan_done(rdev, false);
}
- sched_scan_req = rtnl_dereference(rdev->sched_scan_req);
- if (WARN_ON(sched_scan_req &&
- sched_scan_req->dev == wdev->netdev)) {
- __cfg80211_stop_sched_scan(rdev, false);
+ list_for_each_entry_safe(pos, tmp,
+ &rdev->sched_scan_req_list, list) {
+ if (WARN_ON(pos && pos->dev == wdev->netdev))
+ cfg80211_stop_sched_scan_req(rdev, pos, false);
}
rdev->opencount--;
@@ -1208,12 +1248,12 @@
*/
if ((wdev->iftype == NL80211_IFTYPE_STATION ||
wdev->iftype == NL80211_IFTYPE_P2P_CLIENT) &&
- rdev->ops->set_power_mgmt)
- if (rdev_set_power_mgmt(rdev, dev, wdev->ps,
- wdev->ps_timeout)) {
- /* assume this means it's off */
- wdev->ps = false;
- }
+ rdev->ops->set_power_mgmt &&
+ rdev_set_power_mgmt(rdev, dev, wdev->ps,
+ wdev->ps_timeout)) {
+ /* assume this means it's off */
+ wdev->ps = false;
+ }
break;
case NETDEV_UNREGISTER:
/*
@@ -1234,6 +1274,7 @@
kzfree(wdev->wext.keys);
#endif
flush_work(&wdev->disconnect_wk);
+ cfg80211_cqm_config_free(wdev);
}
/*
* synchronise (so that we won't find this netdev
diff --git a/net/wireless/core.h b/net/wireless/core.h
index 58ca206..6e809325 100644
--- a/net/wireless/core.h
+++ b/net/wireless/core.h
@@ -74,10 +74,9 @@
u32 bss_entries;
struct cfg80211_scan_request *scan_req; /* protected by RTNL */
struct sk_buff *scan_msg;
- struct cfg80211_sched_scan_request __rcu *sched_scan_req;
+ struct list_head sched_scan_req_list;
unsigned long suspend_at;
struct work_struct scan_done_wk;
- struct work_struct sched_scan_results_wk;
struct genl_info *cur_cmd_info;
@@ -91,11 +90,15 @@
struct cfg80211_coalesce *coalesce;
- spinlock_t destroy_list_lock;
- struct list_head destroy_list;
struct work_struct destroy_work;
-
struct work_struct sched_scan_stop_wk;
+ struct work_struct sched_scan_res_wk;
+
+ struct cfg80211_chan_def radar_chandef;
+ struct work_struct propagate_radar_detect_wk;
+
+ struct cfg80211_chan_def cac_done_chandef;
+ struct work_struct propagate_cac_done_wk;
/* must be last because of the way we do wiphy_priv(),
* and it should at least be aligned to NETDEV_ALIGN */
@@ -220,23 +223,8 @@
enum cfg80211_event_type type;
union {
- struct {
- u8 bssid[ETH_ALEN];
- const u8 *req_ie;
- const u8 *resp_ie;
- size_t req_ie_len;
- size_t resp_ie_len;
- struct cfg80211_bss *bss;
- int status; /* -1 = failed; 0..65535 = status code */
- enum nl80211_timeout_reason timeout_reason;
- } cr;
- struct {
- const u8 *req_ie;
- const u8 *resp_ie;
- size_t req_ie_len;
- size_t resp_ie_len;
- struct cfg80211_bss *bss;
- } rm;
+ struct cfg80211_connect_resp_params cr;
+ struct cfg80211_roam_info rm;
struct {
const u8 *ie;
size_t ie_len;
@@ -267,9 +255,11 @@
u32 nlportid;
};
-struct cfg80211_iface_destroy {
- struct list_head list;
- u32 nlportid;
+struct cfg80211_cqm_config {
+ u32 rssi_hyst;
+ s32 last_rssi_event_value;
+ int n_rssi_thresholds;
+ s32 rssi_thresholds[0];
};
void cfg80211_destroy_ifaces(struct cfg80211_registered_device *rdev);
@@ -385,21 +375,16 @@
struct cfg80211_connect_params *connect,
struct cfg80211_cached_keys *connkeys,
const u8 *prev_bssid);
-void __cfg80211_connect_result(struct net_device *dev, const u8 *bssid,
- const u8 *req_ie, size_t req_ie_len,
- const u8 *resp_ie, size_t resp_ie_len,
- int status, bool wextev,
- struct cfg80211_bss *bss,
- enum nl80211_timeout_reason timeout_reason);
+void __cfg80211_connect_result(struct net_device *dev,
+ struct cfg80211_connect_resp_params *params,
+ bool wextev);
void __cfg80211_disconnected(struct net_device *dev, const u8 *ie,
size_t ie_len, u16 reason, bool from_ap);
int cfg80211_disconnect(struct cfg80211_registered_device *rdev,
struct net_device *dev, u16 reason,
bool wextev);
void __cfg80211_roamed(struct wireless_dev *wdev,
- struct cfg80211_bss *bss,
- const u8 *req_ie, size_t req_ie_len,
- const u8 *resp_ie, size_t resp_ie_len);
+ struct cfg80211_roam_info *info);
int cfg80211_mgd_wext_connect(struct cfg80211_registered_device *rdev,
struct wireless_dev *wdev);
void cfg80211_autodisconnect_wk(struct work_struct *work);
@@ -423,13 +408,20 @@
void __cfg80211_scan_done(struct work_struct *wk);
void ___cfg80211_scan_done(struct cfg80211_registered_device *rdev,
bool send_message);
-void __cfg80211_sched_scan_results(struct work_struct *wk);
+void cfg80211_add_sched_scan_req(struct cfg80211_registered_device *rdev,
+ struct cfg80211_sched_scan_request *req);
+int cfg80211_sched_scan_req_possible(struct cfg80211_registered_device *rdev,
+ bool want_multi);
+void cfg80211_sched_scan_results_wk(struct work_struct *work);
+int cfg80211_stop_sched_scan_req(struct cfg80211_registered_device *rdev,
+ struct cfg80211_sched_scan_request *req,
+ bool driver_initiated);
int __cfg80211_stop_sched_scan(struct cfg80211_registered_device *rdev,
- bool driver_initiated);
+ u64 reqid, bool driver_initiated);
void cfg80211_upload_connect_keys(struct wireless_dev *wdev);
int cfg80211_change_iface(struct cfg80211_registered_device *rdev,
struct net_device *dev, enum nl80211_iftype ntype,
- u32 *flags, struct vif_params *params);
+ struct vif_params *params);
void cfg80211_process_rdev_events(struct cfg80211_registered_device *rdev);
void cfg80211_process_wdev_events(struct wireless_dev *wdev);
@@ -459,6 +451,16 @@
cfg80211_chandef_dfs_cac_time(struct wiphy *wiphy,
const struct cfg80211_chan_def *chandef);
+void cfg80211_sched_dfs_chan_update(struct cfg80211_registered_device *rdev);
+
+bool cfg80211_any_wiphy_oper_chan(struct wiphy *wiphy,
+ struct ieee80211_channel *chan);
+
+bool cfg80211_beaconing_iface_active(struct wireless_dev *wdev);
+
+bool cfg80211_is_sub_chan(struct cfg80211_chan_def *chandef,
+ struct ieee80211_channel *chan);
+
static inline unsigned int elapsed_jiffies_msecs(unsigned long start)
{
unsigned long end = jiffies;
@@ -512,4 +514,6 @@
#define CFG80211_DEV_WARN_ON(cond) ({bool __r = (cond); __r; })
#endif
+void cfg80211_cqm_config_free(struct wireless_dev *wdev);
+
#endif /* __NET_WIRELESS_CORE_H */
diff --git a/net/wireless/ibss.c b/net/wireless/ibss.c
index 364f900..10bf040 100644
--- a/net/wireless/ibss.c
+++ b/net/wireless/ibss.c
@@ -190,6 +190,7 @@
if (!nowext)
wdev->wext.ibss.ssid_len = 0;
#endif
+ cfg80211_sched_dfs_chan_update(rdev);
}
void cfg80211_clear_ibss(struct net_device *dev, bool nowext)
diff --git a/net/wireless/mesh.c b/net/wireless/mesh.c
index 2d8518a..ec0b1c2 100644
--- a/net/wireless/mesh.c
+++ b/net/wireless/mesh.c
@@ -262,6 +262,7 @@
wdev->beacon_interval = 0;
memset(&wdev->chandef, 0, sizeof(wdev->chandef));
rdev_set_qos_map(rdev, dev, NULL);
+ cfg80211_sched_dfs_chan_update(rdev);
}
return err;
diff --git a/net/wireless/mlme.c b/net/wireless/mlme.c
index 22b3d99..d8df7a5 100644
--- a/net/wireless/mlme.c
+++ b/net/wireless/mlme.c
@@ -26,9 +26,16 @@
struct wiphy *wiphy = wdev->wiphy;
struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy);
struct ieee80211_mgmt *mgmt = (struct ieee80211_mgmt *)buf;
- u8 *ie = mgmt->u.assoc_resp.variable;
- int ieoffs = offsetof(struct ieee80211_mgmt, u.assoc_resp.variable);
- u16 status_code = le16_to_cpu(mgmt->u.assoc_resp.status_code);
+ struct cfg80211_connect_resp_params cr;
+
+ memset(&cr, 0, sizeof(cr));
+ cr.status = (int)le16_to_cpu(mgmt->u.assoc_resp.status_code);
+ cr.bssid = mgmt->bssid;
+ cr.bss = bss;
+ cr.resp_ie = mgmt->u.assoc_resp.variable;
+ cr.resp_ie_len =
+ len - offsetof(struct ieee80211_mgmt, u.assoc_resp.variable);
+ cr.timeout_reason = NL80211_TIMEOUT_UNSPECIFIED;
trace_cfg80211_send_rx_assoc(dev, bss);
@@ -38,7 +45,7 @@
* and got a reject -- we only try again with an assoc
* frame instead of reassoc.
*/
- if (cfg80211_sme_rx_assoc_resp(wdev, status_code)) {
+ if (cfg80211_sme_rx_assoc_resp(wdev, cr.status)) {
cfg80211_unhold_bss(bss_from_pub(bss));
cfg80211_put_bss(wiphy, bss);
return;
@@ -46,10 +53,7 @@
nl80211_send_rx_assoc(rdev, dev, buf, len, GFP_KERNEL, uapsd_queues);
/* update current_bss etc., consumes the bss reference */
- __cfg80211_connect_result(dev, mgmt->bssid, NULL, 0, ie, len - ieoffs,
- status_code,
- status_code == WLAN_STATUS_SUCCESS, bss,
- NL80211_TIMEOUT_UNSPECIFIED);
+ __cfg80211_connect_result(dev, &cr, cr.status == WLAN_STATUS_SUCCESS);
}
EXPORT_SYMBOL(cfg80211_rx_assoc_resp);
@@ -745,6 +749,12 @@
}
EXPORT_SYMBOL(cfg80211_rx_mgmt);
+void cfg80211_sched_dfs_chan_update(struct cfg80211_registered_device *rdev)
+{
+ cancel_delayed_work(&rdev->dfs_update_channels_wk);
+ queue_delayed_work(cfg80211_wq, &rdev->dfs_update_channels_wk, 0);
+}
+
void cfg80211_dfs_channels_update_work(struct work_struct *work)
{
struct delayed_work *delayed_work = to_delayed_work(work);
@@ -755,6 +765,8 @@
struct wiphy *wiphy;
bool check_again = false;
unsigned long timeout, next_time = 0;
+ unsigned long time_dfs_update;
+ enum nl80211_radar_event radar_event;
int bandid, i;
rdev = container_of(delayed_work, struct cfg80211_registered_device,
@@ -770,11 +782,27 @@
for (i = 0; i < sband->n_channels; i++) {
c = &sband->channels[i];
- if (c->dfs_state != NL80211_DFS_UNAVAILABLE)
+ if (!(c->flags & IEEE80211_CHAN_RADAR))
continue;
- timeout = c->dfs_state_entered + msecs_to_jiffies(
- IEEE80211_DFS_MIN_NOP_TIME_MS);
+ if (c->dfs_state != NL80211_DFS_UNAVAILABLE &&
+ c->dfs_state != NL80211_DFS_AVAILABLE)
+ continue;
+
+ if (c->dfs_state == NL80211_DFS_UNAVAILABLE) {
+ time_dfs_update = IEEE80211_DFS_MIN_NOP_TIME_MS;
+ radar_event = NL80211_RADAR_NOP_FINISHED;
+ } else {
+ if (regulatory_pre_cac_allowed(wiphy) ||
+ cfg80211_any_wiphy_oper_chan(wiphy, c))
+ continue;
+
+ time_dfs_update = REG_PRE_CAC_EXPIRY_GRACE_MS;
+ radar_event = NL80211_RADAR_PRE_CAC_EXPIRED;
+ }
+
+ timeout = c->dfs_state_entered +
+ msecs_to_jiffies(time_dfs_update);
if (time_after_eq(jiffies, timeout)) {
c->dfs_state = NL80211_DFS_USABLE;
@@ -784,8 +812,12 @@
NL80211_CHAN_NO_HT);
nl80211_radar_notify(rdev, &chandef,
- NL80211_RADAR_NOP_FINISHED,
- NULL, GFP_ATOMIC);
+ radar_event, NULL,
+ GFP_ATOMIC);
+
+ regulatory_propagate_dfs_state(wiphy, &chandef,
+ c->dfs_state,
+ radar_event);
continue;
}
@@ -810,7 +842,6 @@
gfp_t gfp)
{
struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy);
- unsigned long timeout;
trace_cfg80211_radar_event(wiphy, chandef);
@@ -820,11 +851,12 @@
*/
cfg80211_set_dfs_state(wiphy, chandef, NL80211_DFS_UNAVAILABLE);
- timeout = msecs_to_jiffies(IEEE80211_DFS_MIN_NOP_TIME_MS);
- queue_delayed_work(cfg80211_wq, &rdev->dfs_update_channels_wk,
- timeout);
+ cfg80211_sched_dfs_chan_update(rdev);
nl80211_radar_notify(rdev, chandef, NL80211_RADAR_DETECTED, NULL, gfp);
+
+ memcpy(&rdev->radar_chandef, chandef, sizeof(struct cfg80211_chan_def));
+ queue_work(cfg80211_wq, &rdev->propagate_radar_detect_wk);
}
EXPORT_SYMBOL(cfg80211_radar_event);
@@ -851,6 +883,10 @@
msecs_to_jiffies(wdev->cac_time_ms);
WARN_ON(!time_after_eq(jiffies, timeout));
cfg80211_set_dfs_state(wiphy, chandef, NL80211_DFS_AVAILABLE);
+ memcpy(&rdev->cac_done_chandef, chandef,
+ sizeof(struct cfg80211_chan_def));
+ queue_work(cfg80211_wq, &rdev->propagate_cac_done_wk);
+ cfg80211_sched_dfs_chan_update(rdev);
break;
case NL80211_RADAR_CAC_ABORTED:
break;
diff --git a/net/wireless/nl80211.c b/net/wireless/nl80211.c
index 2312dc2..570fc95 100644
--- a/net/wireless/nl80211.c
+++ b/net/wireless/nl80211.c
@@ -410,6 +410,16 @@
.len = sizeof(struct nl80211_bss_select_rssi_adjust)
},
[NL80211_ATTR_TIMEOUT_REASON] = { .type = NLA_U32 },
+ [NL80211_ATTR_FILS_ERP_USERNAME] = { .type = NLA_BINARY,
+ .len = FILS_ERP_MAX_USERNAME_LEN },
+ [NL80211_ATTR_FILS_ERP_REALM] = { .type = NLA_BINARY,
+ .len = FILS_ERP_MAX_REALM_LEN },
+ [NL80211_ATTR_FILS_ERP_NEXT_SEQ_NUM] = { .type = NLA_U16 },
+ [NL80211_ATTR_FILS_ERP_RRK] = { .type = NLA_BINARY,
+ .len = FILS_ERP_MAX_RRK_LEN },
+ [NL80211_ATTR_FILS_CACHE_ID] = { .len = 2 },
+ [NL80211_ATTR_PMK] = { .type = NLA_BINARY, .len = PMK_MAX_LEN },
+ [NL80211_ATTR_SCHED_SCAN_MULTI] = { .type = NLA_FLAG },
};
/* policy for the key attributes */
@@ -487,6 +497,7 @@
nl80211_match_policy[NL80211_SCHED_SCAN_MATCH_ATTR_MAX + 1] = {
[NL80211_SCHED_SCAN_MATCH_ATTR_SSID] = { .type = NLA_BINARY,
.len = IEEE80211_MAX_SSID_LEN },
+ [NL80211_SCHED_SCAN_MATCH_ATTR_BSSID] = { .len = ETH_ALEN },
[NL80211_SCHED_SCAN_MATCH_ATTR_RSSI] = { .type = NLA_U32 },
};
@@ -548,7 +559,7 @@
if (!cb->args[0]) {
err = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl80211_fam.hdrsize,
genl_family_attrbuf(&nl80211_fam),
- nl80211_fam.maxattr, nl80211_policy);
+ nl80211_fam.maxattr, nl80211_policy, NULL);
if (err)
return err;
@@ -719,7 +730,7 @@
{
struct nlattr *tb[NL80211_KEY_MAX + 1];
int err = nla_parse_nested(tb, NL80211_KEY_MAX, key,
- nl80211_key_policy);
+ nl80211_key_policy, NULL);
if (err)
return err;
@@ -760,7 +771,7 @@
err = nla_parse_nested(kdt, NUM_NL80211_KEY_DEFAULT_TYPES - 1,
tb[NL80211_KEY_DEFAULT_TYPES],
- nl80211_key_default_policy);
+ nl80211_key_default_policy, NULL);
if (err)
return err;
@@ -807,10 +818,11 @@
if (info->attrs[NL80211_ATTR_KEY_DEFAULT_TYPES]) {
struct nlattr *kdt[NUM_NL80211_KEY_DEFAULT_TYPES];
- int err = nla_parse_nested(
- kdt, NUM_NL80211_KEY_DEFAULT_TYPES - 1,
- info->attrs[NL80211_ATTR_KEY_DEFAULT_TYPES],
- nl80211_key_default_policy);
+ int err = nla_parse_nested(kdt,
+ NUM_NL80211_KEY_DEFAULT_TYPES - 1,
+ info->attrs[NL80211_ATTR_KEY_DEFAULT_TYPES],
+ nl80211_key_default_policy,
+ info->extack);
if (err)
return err;
@@ -1366,7 +1378,7 @@
CMD(tdls_mgmt, TDLS_MGMT);
CMD(tdls_oper, TDLS_OPER);
}
- if (rdev->wiphy.flags & WIPHY_FLAG_SUPPORTS_SCHED_SCAN)
+ if (rdev->wiphy.max_sched_scan_reqs)
CMD(sched_scan_start, START_SCHED_SCAN);
CMD(probe_client, PROBE_CLIENT);
CMD(set_noack_map, SET_NOACK_MAP);
@@ -1805,6 +1817,11 @@
nla_put_flag(msg, NL80211_ATTR_WIPHY_SELF_MANAGED_REG))
goto nla_put_failure;
+ if (rdev->wiphy.max_sched_scan_reqs &&
+ nla_put_u32(msg, NL80211_ATTR_SCHED_SCAN_MAX_REQS,
+ rdev->wiphy.max_sched_scan_reqs))
+ goto nla_put_failure;
+
if (nla_put(msg, NL80211_ATTR_EXT_FEATURES,
sizeof(rdev->wiphy.ext_features),
rdev->wiphy.ext_features))
@@ -1892,8 +1909,8 @@
struct nl80211_dump_wiphy_state *state)
{
struct nlattr **tb = genl_family_attrbuf(&nl80211_fam);
- int ret = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl80211_fam.hdrsize,
- tb, nl80211_fam.maxattr, nl80211_policy);
+ int ret = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl80211_fam.hdrsize, tb,
+ nl80211_fam.maxattr, nl80211_policy, NULL);
/* ignore parse errors for backward compatibility */
if (ret)
return 0;
@@ -2308,7 +2325,8 @@
rem_txq_params) {
result = nla_parse_nested(tb, NL80211_TXQ_ATTR_MAX,
nl_txq_params,
- txq_params_policy);
+ txq_params_policy,
+ info->extack);
if (result)
return result;
result = parse_txq_params(tb, &txq_params);
@@ -2695,17 +2713,82 @@
if (!nla)
return -EINVAL;
- if (nla_parse_nested(flags, NL80211_MNTR_FLAG_MAX,
- nla, mntr_flags_policy))
+ if (nla_parse_nested(flags, NL80211_MNTR_FLAG_MAX, nla,
+ mntr_flags_policy, NULL))
return -EINVAL;
for (flag = 1; flag <= NL80211_MNTR_FLAG_MAX; flag++)
if (flags[flag])
*mntrflags |= (1<<flag);
+ *mntrflags |= MONITOR_FLAG_CHANGED;
+
return 0;
}
+static int nl80211_parse_mon_options(struct cfg80211_registered_device *rdev,
+ enum nl80211_iftype type,
+ struct genl_info *info,
+ struct vif_params *params)
+{
+ bool change = false;
+ int err;
+
+ if (info->attrs[NL80211_ATTR_MNTR_FLAGS]) {
+ if (type != NL80211_IFTYPE_MONITOR)
+ return -EINVAL;
+
+ err = parse_monitor_flags(info->attrs[NL80211_ATTR_MNTR_FLAGS],
+ ¶ms->flags);
+ if (err)
+ return err;
+
+ change = true;
+ }
+
+ if (params->flags & MONITOR_FLAG_ACTIVE &&
+ !(rdev->wiphy.features & NL80211_FEATURE_ACTIVE_MONITOR))
+ return -EOPNOTSUPP;
+
+ if (info->attrs[NL80211_ATTR_MU_MIMO_GROUP_DATA]) {
+ const u8 *mumimo_groups;
+ u32 cap_flag = NL80211_EXT_FEATURE_MU_MIMO_AIR_SNIFFER;
+
+ if (type != NL80211_IFTYPE_MONITOR)
+ return -EINVAL;
+
+ if (!wiphy_ext_feature_isset(&rdev->wiphy, cap_flag))
+ return -EOPNOTSUPP;
+
+ mumimo_groups =
+ nla_data(info->attrs[NL80211_ATTR_MU_MIMO_GROUP_DATA]);
+
+ /* bits 0 and 63 are reserved and must be zero */
+ if ((mumimo_groups[0] & BIT(7)) ||
+ (mumimo_groups[VHT_MUMIMO_GROUPS_DATA_LEN - 1] & BIT(0)))
+ return -EINVAL;
+
+ params->vht_mumimo_groups = mumimo_groups;
+ change = true;
+ }
+
+ if (info->attrs[NL80211_ATTR_MU_MIMO_FOLLOW_MAC_ADDR]) {
+ u32 cap_flag = NL80211_EXT_FEATURE_MU_MIMO_AIR_SNIFFER;
+
+ if (type != NL80211_IFTYPE_MONITOR)
+ return -EINVAL;
+
+ if (!wiphy_ext_feature_isset(&rdev->wiphy, cap_flag))
+ return -EOPNOTSUPP;
+
+ params->vht_mumimo_follow_addr =
+ nla_data(info->attrs[NL80211_ATTR_MU_MIMO_FOLLOW_MAC_ADDR]);
+ change = true;
+ }
+
+ return change ? 1 : 0;
+}
+
static int nl80211_valid_4addr(struct cfg80211_registered_device *rdev,
struct net_device *netdev, u8 use_4addr,
enum nl80211_iftype iftype)
@@ -2739,7 +2822,6 @@
int err;
enum nl80211_iftype otype, ntype;
struct net_device *dev = info->user_ptr[1];
- u32 _flags, *flags = NULL;
bool change = false;
memset(¶ms, 0, sizeof(params));
@@ -2782,56 +2864,14 @@
params.use_4addr = -1;
}
- if (info->attrs[NL80211_ATTR_MNTR_FLAGS]) {
- if (ntype != NL80211_IFTYPE_MONITOR)
- return -EINVAL;
- err = parse_monitor_flags(info->attrs[NL80211_ATTR_MNTR_FLAGS],
- &_flags);
- if (err)
- return err;
-
- flags = &_flags;
+ err = nl80211_parse_mon_options(rdev, ntype, info, ¶ms);
+ if (err < 0)
+ return err;
+ if (err > 0)
change = true;
- }
-
- if (info->attrs[NL80211_ATTR_MU_MIMO_GROUP_DATA]) {
- const u8 *mumimo_groups;
- u32 cap_flag = NL80211_EXT_FEATURE_MU_MIMO_AIR_SNIFFER;
-
- if (!wiphy_ext_feature_isset(&rdev->wiphy, cap_flag))
- return -EOPNOTSUPP;
-
- mumimo_groups =
- nla_data(info->attrs[NL80211_ATTR_MU_MIMO_GROUP_DATA]);
-
- /* bits 0 and 63 are reserved and must be zero */
- if ((mumimo_groups[0] & BIT(7)) ||
- (mumimo_groups[VHT_MUMIMO_GROUPS_DATA_LEN - 1] & BIT(0)))
- return -EINVAL;
-
- memcpy(params.vht_mumimo_groups, mumimo_groups,
- VHT_MUMIMO_GROUPS_DATA_LEN);
- change = true;
- }
-
- if (info->attrs[NL80211_ATTR_MU_MIMO_FOLLOW_MAC_ADDR]) {
- u32 cap_flag = NL80211_EXT_FEATURE_MU_MIMO_AIR_SNIFFER;
-
- if (!wiphy_ext_feature_isset(&rdev->wiphy, cap_flag))
- return -EOPNOTSUPP;
-
- nla_memcpy(params.macaddr,
- info->attrs[NL80211_ATTR_MU_MIMO_FOLLOW_MAC_ADDR],
- ETH_ALEN);
- change = true;
- }
-
- if (flags && (*flags & MONITOR_FLAG_ACTIVE) &&
- !(rdev->wiphy.features & NL80211_FEATURE_ACTIVE_MONITOR))
- return -EOPNOTSUPP;
if (change)
- err = cfg80211_change_iface(rdev, dev, ntype, flags, ¶ms);
+ err = cfg80211_change_iface(rdev, dev, ntype, ¶ms);
else
err = 0;
@@ -2849,7 +2889,6 @@
struct sk_buff *msg;
int err;
enum nl80211_iftype type = NL80211_IFTYPE_UNSPECIFIED;
- u32 flags;
/* to avoid failing a new interface creation due to pending removal */
cfg80211_destroy_ifaces(rdev);
@@ -2885,13 +2924,9 @@
return err;
}
- err = parse_monitor_flags(type == NL80211_IFTYPE_MONITOR ?
- info->attrs[NL80211_ATTR_MNTR_FLAGS] : NULL,
- &flags);
-
- if (!err && (flags & MONITOR_FLAG_ACTIVE) &&
- !(rdev->wiphy.features & NL80211_FEATURE_ACTIVE_MONITOR))
- return -EOPNOTSUPP;
+ err = nl80211_parse_mon_options(rdev, type, info, ¶ms);
+ if (err < 0)
+ return err;
msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
if (!msg)
@@ -2899,8 +2934,7 @@
wdev = rdev_add_virtual_intf(rdev,
nla_data(info->attrs[NL80211_ATTR_IFNAME]),
- NET_NAME_USER, type, err ? NULL : &flags,
- ¶ms);
+ NET_NAME_USER, type, ¶ms);
if (WARN_ON(!wdev)) {
nlmsg_free(msg);
return -EPROTO;
@@ -3561,7 +3595,7 @@
if (sband == NULL)
return -EINVAL;
err = nla_parse_nested(tb, NL80211_TXRATE_MAX, tx_rates,
- nl80211_txattr_policy);
+ nl80211_txattr_policy, info->extack);
if (err)
return err;
if (tb[NL80211_TXRATE_LEGACY]) {
@@ -3818,6 +3852,19 @@
return false;
return true;
case NL80211_CMD_CONNECT:
+ /* SAE not supported yet */
+ if (auth_type == NL80211_AUTHTYPE_SAE)
+ return false;
+ /* FILS with SK PFS or PK not supported yet */
+ if (auth_type == NL80211_AUTHTYPE_FILS_SK_PFS ||
+ auth_type == NL80211_AUTHTYPE_FILS_PK)
+ return false;
+ if (!wiphy_ext_feature_isset(
+ &rdev->wiphy,
+ NL80211_EXT_FEATURE_FILS_SK_OFFLOAD) &&
+ auth_type == NL80211_AUTHTYPE_FILS_SK)
+ return false;
+ return true;
case NL80211_CMD_START_AP:
/* SAE not supported yet */
if (auth_type == NL80211_AUTHTYPE_SAE)
@@ -4100,8 +4147,8 @@
if (!nla)
return 0;
- if (nla_parse_nested(flags, NL80211_STA_FLAG_MAX,
- nla, sta_flags_policy))
+ if (nla_parse_nested(flags, NL80211_STA_FLAG_MAX, nla,
+ sta_flags_policy, info->extack))
return -EINVAL;
/*
@@ -4151,7 +4198,7 @@
struct nlattr *rate;
u32 bitrate;
u16 bitrate_compat;
- enum nl80211_attrs rate_flg;
+ enum nl80211_rate_info rate_flg;
rate = nla_nest_start(msg, attr);
if (!rate)
@@ -4728,7 +4775,7 @@
nla = info->attrs[NL80211_ATTR_STA_WME];
err = nla_parse_nested(tb, NL80211_STA_WME_MAX, nla,
- nl80211_sta_wme_policy);
+ nl80211_sta_wme_policy, info->extack);
if (err)
return err;
@@ -5703,7 +5750,7 @@
cur_params.dot11MeshGateAnnouncementProtocol) ||
nla_put_u8(msg, NL80211_MESHCONF_FORWARDING,
cur_params.dot11MeshForwarding) ||
- nla_put_u32(msg, NL80211_MESHCONF_RSSI_THRESHOLD,
+ nla_put_s32(msg, NL80211_MESHCONF_RSSI_THRESHOLD,
cur_params.rssi_threshold) ||
nla_put_u32(msg, NL80211_MESHCONF_HT_OPMODE,
cur_params.ht_opmode) ||
@@ -5853,7 +5900,7 @@
return -EINVAL;
if (nla_parse_nested(tb, NL80211_MESHCONF_ATTR_MAX,
info->attrs[NL80211_ATTR_MESH_CONFIG],
- nl80211_meshconf_params_policy))
+ nl80211_meshconf_params_policy, info->extack))
return -EINVAL;
/* This makes sure that there aren't more than 32 mesh config
@@ -6002,7 +6049,7 @@
return -EINVAL;
if (nla_parse_nested(tb, NL80211_MESH_SETUP_ATTR_MAX,
info->attrs[NL80211_ATTR_MESH_SETUP],
- nl80211_mesh_setup_params_policy))
+ nl80211_mesh_setup_params_policy, info->extack))
return -EINVAL;
if (tb[NL80211_MESH_SETUP_ENABLE_VENDOR_SYNC])
@@ -6393,7 +6440,8 @@
nla_for_each_nested(nl_reg_rule, info->attrs[NL80211_ATTR_REG_RULES],
rem_reg_rules) {
r = nla_parse_nested(tb, NL80211_REG_RULE_ATTR_MAX,
- nl_reg_rule, reg_rule_policy);
+ nl_reg_rule, reg_rule_policy,
+ info->extack);
if (r)
goto bad_reg;
r = parse_reg_rule(tb, &rd->reg_rules[rule_idx]);
@@ -6461,7 +6509,7 @@
return -EINVAL;
err = nla_parse_nested(attr, NL80211_BSS_SELECT_ATTR_MAX, nest,
- nl80211_bss_select_policy);
+ nl80211_bss_select_policy, NULL);
if (err)
return err;
@@ -6545,6 +6593,19 @@
return 0;
}
+static bool cfg80211_off_channel_oper_allowed(struct wireless_dev *wdev)
+{
+ ASSERT_WDEV_LOCK(wdev);
+
+ if (!cfg80211_beaconing_iface_active(wdev))
+ return true;
+
+ if (!(wdev->chandef.chan->flags & IEEE80211_CHAN_RADAR))
+ return true;
+
+ return regulatory_pre_cac_allowed(wdev->wiphy);
+}
+
static int nl80211_trigger_scan(struct sk_buff *skb, struct genl_info *info)
{
struct cfg80211_registered_device *rdev = info->user_ptr[0];
@@ -6670,6 +6731,25 @@
request->n_channels = i;
+ wdev_lock(wdev);
+ if (!cfg80211_off_channel_oper_allowed(wdev)) {
+ struct ieee80211_channel *chan;
+
+ if (request->n_channels != 1) {
+ wdev_unlock(wdev);
+ err = -EBUSY;
+ goto out_free;
+ }
+
+ chan = request->channels[0];
+ if (chan->center_freq != wdev->chandef.chan->center_freq) {
+ wdev_unlock(wdev);
+ err = -EBUSY;
+ goto out_free;
+ }
+ }
+ wdev_unlock(wdev);
+
i = 0;
if (n_ssids) {
nla_for_each_nested(attr, info->attrs[NL80211_ATTR_SCAN_SSIDS], tmp) {
@@ -6862,7 +6942,7 @@
return -EINVAL;
err = nla_parse_nested(plan, NL80211_SCHED_SCAN_PLAN_MAX,
- attr, nl80211_plan_policy);
+ attr, nl80211_plan_policy, NULL);
if (err)
return err;
@@ -6953,11 +7033,19 @@
err = nla_parse_nested(tb,
NL80211_SCHED_SCAN_MATCH_ATTR_MAX,
- attr, nl80211_match_policy);
+ attr, nl80211_match_policy,
+ NULL);
if (err)
return ERR_PTR(err);
+
+ /* SSID and BSSID are mutually exclusive */
+ if (tb[NL80211_SCHED_SCAN_MATCH_ATTR_SSID] &&
+ tb[NL80211_SCHED_SCAN_MATCH_ATTR_BSSID])
+ return ERR_PTR(-EINVAL);
+
/* add other standalone attributes here */
- if (tb[NL80211_SCHED_SCAN_MATCH_ATTR_SSID]) {
+ if (tb[NL80211_SCHED_SCAN_MATCH_ATTR_SSID] ||
+ tb[NL80211_SCHED_SCAN_MATCH_ATTR_BSSID]) {
n_match_sets++;
continue;
}
@@ -7128,15 +7216,17 @@
nla_for_each_nested(attr,
attrs[NL80211_ATTR_SCHED_SCAN_MATCH],
tmp) {
- struct nlattr *ssid, *rssi;
+ struct nlattr *ssid, *bssid, *rssi;
err = nla_parse_nested(tb,
NL80211_SCHED_SCAN_MATCH_ATTR_MAX,
- attr, nl80211_match_policy);
+ attr, nl80211_match_policy,
+ NULL);
if (err)
goto out_free;
ssid = tb[NL80211_SCHED_SCAN_MATCH_ATTR_SSID];
- if (ssid) {
+ bssid = tb[NL80211_SCHED_SCAN_MATCH_ATTR_BSSID];
+ if (ssid || bssid) {
if (WARN_ON(i >= n_match_sets)) {
/* this indicates a programming error,
* the loop above should have verified
@@ -7146,14 +7236,25 @@
goto out_free;
}
- if (nla_len(ssid) > IEEE80211_MAX_SSID_LEN) {
- err = -EINVAL;
- goto out_free;
+ if (ssid) {
+ if (nla_len(ssid) > IEEE80211_MAX_SSID_LEN) {
+ err = -EINVAL;
+ goto out_free;
+ }
+ memcpy(request->match_sets[i].ssid.ssid,
+ nla_data(ssid), nla_len(ssid));
+ request->match_sets[i].ssid.ssid_len =
+ nla_len(ssid);
}
- memcpy(request->match_sets[i].ssid.ssid,
- nla_data(ssid), nla_len(ssid));
- request->match_sets[i].ssid.ssid_len =
- nla_len(ssid);
+ if (bssid) {
+ if (nla_len(bssid) != ETH_ALEN) {
+ err = -EINVAL;
+ goto out_free;
+ }
+ memcpy(request->match_sets[i].bssid,
+ nla_data(bssid), ETH_ALEN);
+ }
+
/* special attribute - old implementation w/a */
request->match_sets[i].rssi_thold =
default_match_rssi;
@@ -7261,14 +7362,16 @@
struct net_device *dev = info->user_ptr[1];
struct wireless_dev *wdev = dev->ieee80211_ptr;
struct cfg80211_sched_scan_request *sched_scan_req;
+ bool want_multi;
int err;
- if (!(rdev->wiphy.flags & WIPHY_FLAG_SUPPORTS_SCHED_SCAN) ||
- !rdev->ops->sched_scan_start)
+ if (!rdev->wiphy.max_sched_scan_reqs || !rdev->ops->sched_scan_start)
return -EOPNOTSUPP;
- if (rdev->sched_scan_req)
- return -EINPROGRESS;
+ want_multi = info->attrs[NL80211_ATTR_SCHED_SCAN_MULTI];
+ err = cfg80211_sched_scan_req_possible(rdev, want_multi);
+ if (err)
+ return err;
sched_scan_req = nl80211_parse_sched_scan(&rdev->wiphy, wdev,
info->attrs,
@@ -7278,6 +7381,14 @@
if (err)
goto out_err;
+ /* leave request id zero for legacy request
+ * or if driver does not support multi-scheduled scan
+ */
+ if (want_multi && rdev->wiphy.max_sched_scan_reqs > 1) {
+ while (!sched_scan_req->reqid)
+ sched_scan_req->reqid = rdev->wiphy.cookie_counter++;
+ }
+
err = rdev_sched_scan_start(rdev, dev, sched_scan_req);
if (err)
goto out_free;
@@ -7288,10 +7399,9 @@
if (info->attrs[NL80211_ATTR_SOCKET_OWNER])
sched_scan_req->owner_nlportid = info->snd_portid;
- rcu_assign_pointer(rdev->sched_scan_req, sched_scan_req);
+ cfg80211_add_sched_scan_req(rdev, sched_scan_req);
- nl80211_send_sched_scan(rdev, dev,
- NL80211_CMD_START_SCHED_SCAN);
+ nl80211_send_sched_scan(sched_scan_req, NL80211_CMD_START_SCHED_SCAN);
return 0;
out_free:
@@ -7303,13 +7413,27 @@
static int nl80211_stop_sched_scan(struct sk_buff *skb,
struct genl_info *info)
{
+ struct cfg80211_sched_scan_request *req;
struct cfg80211_registered_device *rdev = info->user_ptr[0];
+ u64 cookie;
- if (!(rdev->wiphy.flags & WIPHY_FLAG_SUPPORTS_SCHED_SCAN) ||
- !rdev->ops->sched_scan_stop)
+ if (!rdev->wiphy.max_sched_scan_reqs || !rdev->ops->sched_scan_stop)
return -EOPNOTSUPP;
- return __cfg80211_stop_sched_scan(rdev, false);
+ if (info->attrs[NL80211_ATTR_COOKIE]) {
+ cookie = nla_get_u64(info->attrs[NL80211_ATTR_COOKIE]);
+ return __cfg80211_stop_sched_scan(rdev, cookie, false);
+ }
+
+ req = list_first_or_null_rcu(&rdev->sched_scan_req_list,
+ struct cfg80211_sched_scan_request,
+ list);
+ if (!req || req->reqid ||
+ (req->owner_nlportid &&
+ req->owner_nlportid != info->snd_portid))
+ return -ENOENT;
+
+ return cfg80211_stop_sched_scan_req(rdev, req, false);
}
static int nl80211_start_radar_detection(struct sk_buff *skb,
@@ -7433,7 +7557,7 @@
err = nla_parse_nested(csa_attrs, NL80211_ATTR_MAX,
info->attrs[NL80211_ATTR_CSA_IES],
- nl80211_policy);
+ nl80211_policy, info->extack);
if (err)
return err;
@@ -8639,7 +8763,8 @@
struct nlattr **attrbuf = genl_family_attrbuf(&nl80211_fam);
err = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl80211_fam.hdrsize,
- attrbuf, nl80211_fam.maxattr, nl80211_policy);
+ attrbuf, nl80211_fam.maxattr,
+ nl80211_policy, NULL);
if (err)
goto out_err;
@@ -8867,6 +8992,35 @@
}
}
+ if (wiphy_ext_feature_isset(&rdev->wiphy,
+ NL80211_EXT_FEATURE_FILS_SK_OFFLOAD) &&
+ info->attrs[NL80211_ATTR_FILS_ERP_USERNAME] &&
+ info->attrs[NL80211_ATTR_FILS_ERP_REALM] &&
+ info->attrs[NL80211_ATTR_FILS_ERP_NEXT_SEQ_NUM] &&
+ info->attrs[NL80211_ATTR_FILS_ERP_RRK]) {
+ connect.fils_erp_username =
+ nla_data(info->attrs[NL80211_ATTR_FILS_ERP_USERNAME]);
+ connect.fils_erp_username_len =
+ nla_len(info->attrs[NL80211_ATTR_FILS_ERP_USERNAME]);
+ connect.fils_erp_realm =
+ nla_data(info->attrs[NL80211_ATTR_FILS_ERP_REALM]);
+ connect.fils_erp_realm_len =
+ nla_len(info->attrs[NL80211_ATTR_FILS_ERP_REALM]);
+ connect.fils_erp_next_seq_num =
+ nla_get_u16(
+ info->attrs[NL80211_ATTR_FILS_ERP_NEXT_SEQ_NUM]);
+ connect.fils_erp_rrk =
+ nla_data(info->attrs[NL80211_ATTR_FILS_ERP_RRK]);
+ connect.fils_erp_rrk_len =
+ nla_len(info->attrs[NL80211_ATTR_FILS_ERP_RRK]);
+ } else if (info->attrs[NL80211_ATTR_FILS_ERP_USERNAME] ||
+ info->attrs[NL80211_ATTR_FILS_ERP_REALM] ||
+ info->attrs[NL80211_ATTR_FILS_ERP_NEXT_SEQ_NUM] ||
+ info->attrs[NL80211_ATTR_FILS_ERP_RRK]) {
+ kzfree(connkeys);
+ return -EINVAL;
+ }
+
wdev_lock(dev->ieee80211_ptr);
err = cfg80211_connect(rdev, dev, &connect, connkeys,
@@ -8986,14 +9140,28 @@
memset(&pmksa, 0, sizeof(struct cfg80211_pmksa));
- if (!info->attrs[NL80211_ATTR_MAC])
- return -EINVAL;
-
if (!info->attrs[NL80211_ATTR_PMKID])
return -EINVAL;
pmksa.pmkid = nla_data(info->attrs[NL80211_ATTR_PMKID]);
- pmksa.bssid = nla_data(info->attrs[NL80211_ATTR_MAC]);
+
+ if (info->attrs[NL80211_ATTR_MAC]) {
+ pmksa.bssid = nla_data(info->attrs[NL80211_ATTR_MAC]);
+ } else if (info->attrs[NL80211_ATTR_SSID] &&
+ info->attrs[NL80211_ATTR_FILS_CACHE_ID] &&
+ (info->genlhdr->cmd == NL80211_CMD_DEL_PMKSA ||
+ info->attrs[NL80211_ATTR_PMK])) {
+ pmksa.ssid = nla_data(info->attrs[NL80211_ATTR_SSID]);
+ pmksa.ssid_len = nla_len(info->attrs[NL80211_ATTR_SSID]);
+ pmksa.cache_id =
+ nla_data(info->attrs[NL80211_ATTR_FILS_CACHE_ID]);
+ } else {
+ return -EINVAL;
+ }
+ if (info->attrs[NL80211_ATTR_PMK]) {
+ pmksa.pmk = nla_data(info->attrs[NL80211_ATTR_PMK]);
+ pmksa.pmk_len = nla_len(info->attrs[NL80211_ATTR_PMK]);
+ }
if (dev->ieee80211_ptr->iftype != NL80211_IFTYPE_STATION &&
dev->ieee80211_ptr->iftype != NL80211_IFTYPE_P2P_CLIENT)
@@ -9096,6 +9264,7 @@
struct cfg80211_registered_device *rdev = info->user_ptr[0];
struct wireless_dev *wdev = info->user_ptr[1];
struct cfg80211_chan_def chandef;
+ const struct cfg80211_chan_def *compat_chandef;
struct sk_buff *msg;
void *hdr;
u64 cookie;
@@ -9124,6 +9293,18 @@
if (err)
return err;
+ wdev_lock(wdev);
+ if (!cfg80211_off_channel_oper_allowed(wdev) &&
+ !cfg80211_chandef_identical(&wdev->chandef, &chandef)) {
+ compat_chandef = cfg80211_chandef_compatible(&wdev->chandef,
+ &chandef);
+ if (compat_chandef != &chandef) {
+ wdev_unlock(wdev);
+ return -EBUSY;
+ }
+ }
+ wdev_unlock(wdev);
+
msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
if (!msg)
return -ENOMEM;
@@ -9299,6 +9480,13 @@
if (!chandef.chan && params.offchan)
return -EINVAL;
+ wdev_lock(wdev);
+ if (params.offchan && !cfg80211_off_channel_oper_allowed(wdev)) {
+ wdev_unlock(wdev);
+ return -EBUSY;
+ }
+ wdev_unlock(wdev);
+
params.buf = nla_data(info->attrs[NL80211_ATTR_FRAME]);
params.len = nla_len(info->attrs[NL80211_ATTR_FRAME]);
@@ -9466,7 +9654,7 @@
static const struct nla_policy
nl80211_attr_cqm_policy[NL80211_ATTR_CQM_MAX + 1] = {
- [NL80211_ATTR_CQM_RSSI_THOLD] = { .type = NLA_U32 },
+ [NL80211_ATTR_CQM_RSSI_THOLD] = { .type = NLA_BINARY },
[NL80211_ATTR_CQM_RSSI_HYST] = { .type = NLA_U32 },
[NL80211_ATTR_CQM_RSSI_THRESHOLD_EVENT] = { .type = NLA_U32 },
[NL80211_ATTR_CQM_TXE_RATE] = { .type = NLA_U32 },
@@ -9495,28 +9683,123 @@
return rdev_set_cqm_txe_config(rdev, dev, rate, pkts, intvl);
}
+static int cfg80211_cqm_rssi_update(struct cfg80211_registered_device *rdev,
+ struct net_device *dev)
+{
+ struct wireless_dev *wdev = dev->ieee80211_ptr;
+ s32 last, low, high;
+ u32 hyst;
+ int i, n;
+ int err;
+
+ /* RSSI reporting disabled? */
+ if (!wdev->cqm_config)
+ return rdev_set_cqm_rssi_range_config(rdev, dev, 0, 0);
+
+ /*
+ * Obtain current RSSI value if possible, if not and no RSSI threshold
+ * event has been received yet, we should receive an event after a
+ * connection is established and enough beacons received to calculate
+ * the average.
+ */
+ if (!wdev->cqm_config->last_rssi_event_value && wdev->current_bss &&
+ rdev->ops->get_station) {
+ struct station_info sinfo;
+ u8 *mac_addr;
+
+ mac_addr = wdev->current_bss->pub.bssid;
+
+ err = rdev_get_station(rdev, dev, mac_addr, &sinfo);
+ if (err)
+ return err;
+
+ if (sinfo.filled & BIT(NL80211_STA_INFO_BEACON_SIGNAL_AVG))
+ wdev->cqm_config->last_rssi_event_value =
+ (s8) sinfo.rx_beacon_signal_avg;
+ }
+
+ last = wdev->cqm_config->last_rssi_event_value;
+ hyst = wdev->cqm_config->rssi_hyst;
+ n = wdev->cqm_config->n_rssi_thresholds;
+
+ for (i = 0; i < n; i++)
+ if (last < wdev->cqm_config->rssi_thresholds[i])
+ break;
+
+ low = i > 0 ?
+ (wdev->cqm_config->rssi_thresholds[i - 1] - hyst) : S32_MIN;
+ high = i < n ?
+ (wdev->cqm_config->rssi_thresholds[i] + hyst - 1) : S32_MAX;
+
+ return rdev_set_cqm_rssi_range_config(rdev, dev, low, high);
+}
+
static int nl80211_set_cqm_rssi(struct genl_info *info,
- s32 threshold, u32 hysteresis)
+ const s32 *thresholds, int n_thresholds,
+ u32 hysteresis)
{
struct cfg80211_registered_device *rdev = info->user_ptr[0];
struct net_device *dev = info->user_ptr[1];
struct wireless_dev *wdev = dev->ieee80211_ptr;
+ int i, err;
+ s32 prev = S32_MIN;
- if (threshold > 0)
- return -EINVAL;
+ /* Check all values negative and sorted */
+ for (i = 0; i < n_thresholds; i++) {
+ if (thresholds[i] > 0 || thresholds[i] <= prev)
+ return -EINVAL;
- /* disabling - hysteresis should also be zero then */
- if (threshold == 0)
- hysteresis = 0;
-
- if (!rdev->ops->set_cqm_rssi_config)
- return -EOPNOTSUPP;
+ prev = thresholds[i];
+ }
if (wdev->iftype != NL80211_IFTYPE_STATION &&
wdev->iftype != NL80211_IFTYPE_P2P_CLIENT)
return -EOPNOTSUPP;
- return rdev_set_cqm_rssi_config(rdev, dev, threshold, hysteresis);
+ wdev_lock(wdev);
+ cfg80211_cqm_config_free(wdev);
+ wdev_unlock(wdev);
+
+ if (n_thresholds <= 1 && rdev->ops->set_cqm_rssi_config) {
+ if (n_thresholds == 0 || thresholds[0] == 0) /* Disabling */
+ return rdev_set_cqm_rssi_config(rdev, dev, 0, 0);
+
+ return rdev_set_cqm_rssi_config(rdev, dev,
+ thresholds[0], hysteresis);
+ }
+
+ if (!wiphy_ext_feature_isset(&rdev->wiphy,
+ NL80211_EXT_FEATURE_CQM_RSSI_LIST))
+ return -EOPNOTSUPP;
+
+ if (n_thresholds == 1 && thresholds[0] == 0) /* Disabling */
+ n_thresholds = 0;
+
+ wdev_lock(wdev);
+ if (n_thresholds) {
+ struct cfg80211_cqm_config *cqm_config;
+
+ cqm_config = kzalloc(sizeof(struct cfg80211_cqm_config) +
+ n_thresholds * sizeof(s32), GFP_KERNEL);
+ if (!cqm_config) {
+ err = -ENOMEM;
+ goto unlock;
+ }
+
+ cqm_config->rssi_hyst = hysteresis;
+ cqm_config->n_rssi_thresholds = n_thresholds;
+ memcpy(cqm_config->rssi_thresholds, thresholds,
+ n_thresholds * sizeof(s32));
+
+ wdev->cqm_config = cqm_config;
+ }
+
+ err = cfg80211_cqm_rssi_update(rdev, dev);
+
+unlock:
+ wdev_unlock(wdev);
+
+ return err;
}
static int nl80211_set_cqm(struct sk_buff *skb, struct genl_info *info)
@@ -9530,16 +9813,22 @@
return -EINVAL;
err = nla_parse_nested(attrs, NL80211_ATTR_CQM_MAX, cqm,
- nl80211_attr_cqm_policy);
+ nl80211_attr_cqm_policy, info->extack);
if (err)
return err;
if (attrs[NL80211_ATTR_CQM_RSSI_THOLD] &&
attrs[NL80211_ATTR_CQM_RSSI_HYST]) {
- s32 threshold = nla_get_s32(attrs[NL80211_ATTR_CQM_RSSI_THOLD]);
+ const s32 *thresholds =
+ nla_data(attrs[NL80211_ATTR_CQM_RSSI_THOLD]);
+ int len = nla_len(attrs[NL80211_ATTR_CQM_RSSI_THOLD]);
u32 hysteresis = nla_get_u32(attrs[NL80211_ATTR_CQM_RSSI_HYST]);
- return nl80211_set_cqm_rssi(info, threshold, hysteresis);
+ if (len % 4)
+ return -EINVAL;
+
+ return nl80211_set_cqm_rssi(info, thresholds, len / 4,
+ hysteresis);
}
if (attrs[NL80211_ATTR_CQM_TXE_RATE] &&
@@ -9940,7 +10229,7 @@
return -EINVAL;
err = nla_parse_nested(tb, MAX_NL80211_WOWLAN_TCP, attr,
- nl80211_wowlan_tcp_policy);
+ nl80211_wowlan_tcp_policy, NULL);
if (err)
return err;
@@ -10085,7 +10374,8 @@
goto out;
}
- err = nla_parse_nested(tb, NL80211_ATTR_MAX, attr, nl80211_policy);
+ err = nla_parse_nested(tb, NL80211_ATTR_MAX, attr, nl80211_policy,
+ NULL);
if (err)
goto out;
@@ -10122,7 +10412,7 @@
err = nla_parse_nested(tb, MAX_NL80211_WOWLAN_TRIG,
info->attrs[NL80211_ATTR_WOWLAN_TRIGGERS],
- nl80211_wowlan_policy);
+ nl80211_wowlan_policy, info->extack);
if (err)
return err;
@@ -10205,7 +10495,7 @@
u8 *mask_pat;
nla_parse_nested(pat_tb, MAX_NL80211_PKTPAT, pat,
- NULL);
+ NULL, info->extack);
err = -EINVAL;
if (!pat_tb[NL80211_PKTPAT_MASK] ||
!pat_tb[NL80211_PKTPAT_PATTERN])
@@ -10416,7 +10706,7 @@
struct nlattr *pat_tb[NUM_NL80211_PKTPAT];
err = nla_parse_nested(tb, NL80211_ATTR_COALESCE_RULE_MAX, rule,
- nl80211_coalesce_policy);
+ nl80211_coalesce_policy, NULL);
if (err)
return err;
@@ -10454,7 +10744,7 @@
rem) {
u8 *mask_pat;
- nla_parse_nested(pat_tb, MAX_NL80211_PKTPAT, pat, NULL);
+ nla_parse_nested(pat_tb, MAX_NL80211_PKTPAT, pat, NULL, NULL);
if (!pat_tb[NL80211_PKTPAT_MASK] ||
!pat_tb[NL80211_PKTPAT_PATTERN])
return -EINVAL;
@@ -10575,7 +10865,7 @@
err = nla_parse_nested(tb, MAX_NL80211_REKEY_DATA,
info->attrs[NL80211_ATTR_REKEY_DATA],
- nl80211_rekey_policy);
+ nl80211_rekey_policy, info->extack);
if (err)
return err;
@@ -10892,7 +11182,7 @@
err = nla_parse_nested(tb, NL80211_NAN_FUNC_ATTR_MAX,
info->attrs[NL80211_ATTR_NAN_FUNC],
- nl80211_nan_func_policy);
+ nl80211_nan_func_policy, info->extack);
if (err)
return err;
@@ -10989,7 +11279,7 @@
err = nla_parse_nested(srf_tb, NL80211_NAN_SRF_ATTR_MAX,
tb[NL80211_NAN_FUNC_SRF],
- nl80211_nan_srf_policy);
+ nl80211_nan_srf_policy, info->extack);
if (err)
goto out;
@@ -11524,8 +11814,8 @@
return 0;
}
- err = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl80211_fam.hdrsize,
- attrbuf, nl80211_fam.maxattr, nl80211_policy);
+ err = nlmsg_parse(cb->nlh, GENL_HDRLEN + nl80211_fam.hdrsize, attrbuf,
+ nl80211_fam.maxattr, nl80211_policy, NULL);
if (err)
return err;
@@ -12970,18 +13260,19 @@
static int
nl80211_prep_sched_scan_msg(struct sk_buff *msg,
- struct cfg80211_registered_device *rdev,
- struct net_device *netdev,
- u32 portid, u32 seq, int flags, u32 cmd)
+ struct cfg80211_sched_scan_request *req, u32 cmd)
{
void *hdr;
- hdr = nl80211hdr_put(msg, portid, seq, flags, cmd);
+ hdr = nl80211hdr_put(msg, 0, 0, 0, cmd);
if (!hdr)
return -1;
- if (nla_put_u32(msg, NL80211_ATTR_WIPHY, rdev->wiphy_idx) ||
- nla_put_u32(msg, NL80211_ATTR_IFINDEX, netdev->ifindex))
+ if (nla_put_u32(msg, NL80211_ATTR_WIPHY,
+ wiphy_to_rdev(req->wiphy)->wiphy_idx) ||
+ nla_put_u32(msg, NL80211_ATTR_IFINDEX, req->dev->ifindex) ||
+ nla_put_u64_64bit(msg, NL80211_ATTR_COOKIE, req->reqid,
+ NL80211_ATTR_PAD))
goto nla_put_failure;
genlmsg_end(msg, hdr);
@@ -13041,8 +13332,7 @@
NL80211_MCGRP_SCAN, GFP_KERNEL);
}
-void nl80211_send_sched_scan(struct cfg80211_registered_device *rdev,
- struct net_device *netdev, u32 cmd)
+void nl80211_send_sched_scan(struct cfg80211_sched_scan_request *req, u32 cmd)
{
struct sk_buff *msg;
@@ -13050,12 +13340,12 @@
if (!msg)
return;
- if (nl80211_prep_sched_scan_msg(msg, rdev, netdev, 0, 0, 0, cmd) < 0) {
+ if (nl80211_prep_sched_scan_msg(msg, req, cmd) < 0) {
nlmsg_free(msg);
return;
}
- genlmsg_multicast_netns(&nl80211_fam, wiphy_net(&rdev->wiphy), msg, 0,
+ genlmsg_multicast_netns(&nl80211_fam, wiphy_net(req->wiphy), msg, 0,
NL80211_MCGRP_SCAN, GFP_KERNEL);
}
@@ -13296,17 +13586,16 @@
}
void nl80211_send_connect_result(struct cfg80211_registered_device *rdev,
- struct net_device *netdev, const u8 *bssid,
- const u8 *req_ie, size_t req_ie_len,
- const u8 *resp_ie, size_t resp_ie_len,
- int status,
- enum nl80211_timeout_reason timeout_reason,
+ struct net_device *netdev,
+ struct cfg80211_connect_resp_params *cr,
gfp_t gfp)
{
struct sk_buff *msg;
void *hdr;
- msg = nlmsg_new(100 + req_ie_len + resp_ie_len, gfp);
+ msg = nlmsg_new(100 + cr->req_ie_len + cr->resp_ie_len +
+ cr->fils_kek_len + cr->pmk_len +
+ (cr->pmkid ? WLAN_PMKID_LEN : 0), gfp);
if (!msg)
return;
@@ -13318,17 +13607,31 @@
if (nla_put_u32(msg, NL80211_ATTR_WIPHY, rdev->wiphy_idx) ||
nla_put_u32(msg, NL80211_ATTR_IFINDEX, netdev->ifindex) ||
- (bssid && nla_put(msg, NL80211_ATTR_MAC, ETH_ALEN, bssid)) ||
+ (cr->bssid &&
+ nla_put(msg, NL80211_ATTR_MAC, ETH_ALEN, cr->bssid)) ||
nla_put_u16(msg, NL80211_ATTR_STATUS_CODE,
- status < 0 ? WLAN_STATUS_UNSPECIFIED_FAILURE :
- status) ||
- (status < 0 &&
+ cr->status < 0 ? WLAN_STATUS_UNSPECIFIED_FAILURE :
+ cr->status) ||
+ (cr->status < 0 &&
(nla_put_flag(msg, NL80211_ATTR_TIMED_OUT) ||
- nla_put_u32(msg, NL80211_ATTR_TIMEOUT_REASON, timeout_reason))) ||
- (req_ie &&
- nla_put(msg, NL80211_ATTR_REQ_IE, req_ie_len, req_ie)) ||
- (resp_ie &&
- nla_put(msg, NL80211_ATTR_RESP_IE, resp_ie_len, resp_ie)))
+ nla_put_u32(msg, NL80211_ATTR_TIMEOUT_REASON,
+ cr->timeout_reason))) ||
+ (cr->req_ie &&
+ nla_put(msg, NL80211_ATTR_REQ_IE, cr->req_ie_len, cr->req_ie)) ||
+ (cr->resp_ie &&
+ nla_put(msg, NL80211_ATTR_RESP_IE, cr->resp_ie_len,
+ cr->resp_ie)) ||
+ (cr->update_erp_next_seq_num &&
+ nla_put_u16(msg, NL80211_ATTR_FILS_ERP_NEXT_SEQ_NUM,
+ cr->fils_erp_next_seq_num)) ||
+ (cr->status == WLAN_STATUS_SUCCESS &&
+ ((cr->fils_kek &&
+ nla_put(msg, NL80211_ATTR_FILS_KEK, cr->fils_kek_len,
+ cr->fils_kek)) ||
+ (cr->pmk &&
+ nla_put(msg, NL80211_ATTR_PMK, cr->pmk_len, cr->pmk)) ||
+ (cr->pmkid &&
+ nla_put(msg, NL80211_ATTR_PMKID, WLAN_PMKID_LEN, cr->pmkid)))))
goto nla_put_failure;
genlmsg_end(msg, hdr);
@@ -13343,14 +13646,14 @@
}
void nl80211_send_roamed(struct cfg80211_registered_device *rdev,
- struct net_device *netdev, const u8 *bssid,
- const u8 *req_ie, size_t req_ie_len,
- const u8 *resp_ie, size_t resp_ie_len, gfp_t gfp)
+ struct net_device *netdev,
+ struct cfg80211_roam_info *info, gfp_t gfp)
{
struct sk_buff *msg;
void *hdr;
+ const u8 *bssid = info->bss ? info->bss->bssid : info->bssid;
- msg = nlmsg_new(100 + req_ie_len + resp_ie_len, gfp);
+ msg = nlmsg_new(100 + info->req_ie_len + info->resp_ie_len, gfp);
if (!msg)
return;
@@ -13363,10 +13666,12 @@
if (nla_put_u32(msg, NL80211_ATTR_WIPHY, rdev->wiphy_idx) ||
nla_put_u32(msg, NL80211_ATTR_IFINDEX, netdev->ifindex) ||
nla_put(msg, NL80211_ATTR_MAC, ETH_ALEN, bssid) ||
- (req_ie &&
- nla_put(msg, NL80211_ATTR_REQ_IE, req_ie_len, req_ie)) ||
- (resp_ie &&
- nla_put(msg, NL80211_ATTR_RESP_IE, resp_ie_len, resp_ie)))
+ (info->req_ie &&
+ nla_put(msg, NL80211_ATTR_REQ_IE, info->req_ie_len,
+ info->req_ie)) ||
+ (info->resp_ie &&
+ nla_put(msg, NL80211_ATTR_RESP_IE, info->resp_ie_len,
+ info->resp_ie)))
goto nla_put_failure;
genlmsg_end(msg, hdr);
@@ -13968,6 +14273,8 @@
s32 rssi_level, gfp_t gfp)
{
struct sk_buff *msg;
+ struct wireless_dev *wdev = dev->ieee80211_ptr;
+ struct cfg80211_registered_device *rdev = wiphy_to_rdev(wdev->wiphy);
trace_cfg80211_cqm_rssi_notify(dev, rssi_event, rssi_level);
@@ -13975,6 +14282,15 @@
rssi_event != NL80211_CQM_RSSI_THRESHOLD_EVENT_HIGH))
return;
+ if (wdev->cqm_config) {
+ wdev->cqm_config->last_rssi_event_value = rssi_level;
+
+ cfg80211_cqm_rssi_update(rdev, dev);
+
+ if (rssi_level == 0)
+ rssi_level = wdev->cqm_config->last_rssi_event_value;
+ }
+
msg = cfg80211_prepare_cqm(dev, NULL, gfp);
if (!msg)
return;
@@ -14619,26 +14935,26 @@
rcu_read_lock();
list_for_each_entry_rcu(rdev, &cfg80211_rdev_list, list) {
- bool schedule_destroy_work = false;
- struct cfg80211_sched_scan_request *sched_scan_req =
- rcu_dereference(rdev->sched_scan_req);
+ struct cfg80211_sched_scan_request *sched_scan_req;
- if (sched_scan_req && notify->portid &&
- sched_scan_req->owner_nlportid == notify->portid) {
- sched_scan_req->owner_nlportid = 0;
-
- if (rdev->ops->sched_scan_stop &&
- rdev->wiphy.flags & WIPHY_FLAG_SUPPORTS_SCHED_SCAN)
+ list_for_each_entry_rcu(sched_scan_req,
+ &rdev->sched_scan_req_list,
+ list) {
+ if (sched_scan_req->owner_nlportid == notify->portid) {
+ sched_scan_req->nl_owner_dead = true;
schedule_work(&rdev->sched_scan_stop_wk);
+ }
}
list_for_each_entry_rcu(wdev, &rdev->wiphy.wdev_list, list) {
cfg80211_mlme_unregister_socket(wdev, notify->portid);
- if (wdev->owner_nlportid == notify->portid)
- schedule_destroy_work = true;
- else if (wdev->conn_owner_nlportid == notify->portid)
+ if (wdev->owner_nlportid == notify->portid) {
+ wdev->nl_owner_dead = true;
+ schedule_work(&rdev->destroy_work);
+ } else if (wdev->conn_owner_nlportid == notify->portid) {
schedule_work(&wdev->disconnect_wk);
+ }
}
spin_lock_bh(&rdev->beacon_registrations_lock);
@@ -14651,19 +14967,6 @@
}
}
spin_unlock_bh(&rdev->beacon_registrations_lock);
-
- if (schedule_destroy_work) {
- struct cfg80211_iface_destroy *destroy;
-
- destroy = kzalloc(sizeof(*destroy), GFP_ATOMIC);
- if (destroy) {
- destroy->nlportid = notify->portid;
- spin_lock(&rdev->destroy_list_lock);
- list_add(&destroy->list, &rdev->destroy_list);
- spin_unlock(&rdev->destroy_list_lock);
- schedule_work(&rdev->destroy_work);
- }
- }
}
rcu_read_unlock();
diff --git a/net/wireless/nl80211.h b/net/wireless/nl80211.h
index e488dca..b969333 100644
--- a/net/wireless/nl80211.h
+++ b/net/wireless/nl80211.h
@@ -16,8 +16,7 @@
struct wireless_dev *wdev, bool aborted);
void nl80211_send_scan_msg(struct cfg80211_registered_device *rdev,
struct sk_buff *msg);
-void nl80211_send_sched_scan(struct cfg80211_registered_device *rdev,
- struct net_device *netdev, u32 cmd);
+void nl80211_send_sched_scan(struct cfg80211_sched_scan_request *req, u32 cmd);
void nl80211_common_reg_change_event(enum nl80211_commands cmd_id,
struct regulatory_request *request);
@@ -53,16 +52,12 @@
struct net_device *netdev,
const u8 *addr, gfp_t gfp);
void nl80211_send_connect_result(struct cfg80211_registered_device *rdev,
- struct net_device *netdev, const u8 *bssid,
- const u8 *req_ie, size_t req_ie_len,
- const u8 *resp_ie, size_t resp_ie_len,
- int status,
- enum nl80211_timeout_reason timeout_reason,
+ struct net_device *netdev,
+ struct cfg80211_connect_resp_params *params,
gfp_t gfp);
void nl80211_send_roamed(struct cfg80211_registered_device *rdev,
- struct net_device *netdev, const u8 *bssid,
- const u8 *req_ie, size_t req_ie_len,
- const u8 *resp_ie, size_t resp_ie_len, gfp_t gfp);
+ struct net_device *netdev,
+ struct cfg80211_roam_info *info, gfp_t gfp);
void nl80211_send_disconnected(struct cfg80211_registered_device *rdev,
struct net_device *netdev, u16 reason,
const u8 *ie, size_t ie_len, bool from_ap);
diff --git a/net/wireless/rdev-ops.h b/net/wireless/rdev-ops.h
index 2f42507..0598c1e 100644
--- a/net/wireless/rdev-ops.h
+++ b/net/wireless/rdev-ops.h
@@ -36,13 +36,13 @@
static inline struct wireless_dev
*rdev_add_virtual_intf(struct cfg80211_registered_device *rdev, char *name,
unsigned char name_assign_type,
- enum nl80211_iftype type, u32 *flags,
+ enum nl80211_iftype type,
struct vif_params *params)
{
struct wireless_dev *ret;
trace_rdev_add_virtual_intf(&rdev->wiphy, name, type);
ret = rdev->ops->add_virtual_intf(&rdev->wiphy, name, name_assign_type,
- type, flags, params);
+ type, params);
trace_rdev_return_wdev(&rdev->wiphy, ret);
return ret;
}
@@ -61,12 +61,11 @@
static inline int
rdev_change_virtual_intf(struct cfg80211_registered_device *rdev,
struct net_device *dev, enum nl80211_iftype type,
- u32 *flags, struct vif_params *params)
+ struct vif_params *params)
{
int ret;
trace_rdev_change_virtual_intf(&rdev->wiphy, dev, type);
- ret = rdev->ops->change_virtual_intf(&rdev->wiphy, dev, type, flags,
- params);
+ ret = rdev->ops->change_virtual_intf(&rdev->wiphy, dev, type, params);
trace_rdev_return_int(&rdev->wiphy, ret);
return ret;
}
@@ -750,6 +749,18 @@
}
static inline int
+rdev_set_cqm_rssi_range_config(struct cfg80211_registered_device *rdev,
+ struct net_device *dev, s32 low, s32 high)
+{
+ int ret;
+ trace_rdev_set_cqm_rssi_range_config(&rdev->wiphy, dev, low, high);
+ ret = rdev->ops->set_cqm_rssi_range_config(&rdev->wiphy, dev,
+ low, high);
+ trace_rdev_return_int(&rdev->wiphy, ret);
+ return ret;
+}
+
+static inline int
rdev_set_cqm_txe_config(struct cfg80211_registered_device *rdev,
struct net_device *dev, u32 rate, u32 pkts, u32 intvl)
{
@@ -802,18 +813,18 @@
struct cfg80211_sched_scan_request *request)
{
int ret;
- trace_rdev_sched_scan_start(&rdev->wiphy, dev, request);
+ trace_rdev_sched_scan_start(&rdev->wiphy, dev, request->reqid);
ret = rdev->ops->sched_scan_start(&rdev->wiphy, dev, request);
trace_rdev_return_int(&rdev->wiphy, ret);
return ret;
}
static inline int rdev_sched_scan_stop(struct cfg80211_registered_device *rdev,
- struct net_device *dev)
+ struct net_device *dev, u64 reqid)
{
int ret;
- trace_rdev_sched_scan_stop(&rdev->wiphy, dev);
- ret = rdev->ops->sched_scan_stop(&rdev->wiphy, dev);
+ trace_rdev_sched_scan_stop(&rdev->wiphy, dev, reqid);
+ ret = rdev->ops->sched_scan_stop(&rdev->wiphy, dev, reqid);
trace_rdev_return_int(&rdev->wiphy, ret);
return ret;
}
diff --git a/net/wireless/reg.c b/net/wireless/reg.c
index 753efcd..5fae296 100644
--- a/net/wireless/reg.c
+++ b/net/wireless/reg.c
@@ -2067,6 +2067,88 @@
return REG_REQ_IGNORE;
}
+bool reg_dfs_domain_same(struct wiphy *wiphy1, struct wiphy *wiphy2)
+{
+ const struct ieee80211_regdomain *wiphy1_regd = NULL;
+ const struct ieee80211_regdomain *wiphy2_regd = NULL;
+ const struct ieee80211_regdomain *cfg80211_regd = NULL;
+ bool dfs_domain_same;
+
+ rcu_read_lock();
+
+ cfg80211_regd = rcu_dereference(cfg80211_regdomain);
+ wiphy1_regd = rcu_dereference(wiphy1->regd);
+ if (!wiphy1_regd)
+ wiphy1_regd = cfg80211_regd;
+
+ wiphy2_regd = rcu_dereference(wiphy2->regd);
+ if (!wiphy2_regd)
+ wiphy2_regd = cfg80211_regd;
+
+ dfs_domain_same = wiphy1_regd->dfs_region == wiphy2_regd->dfs_region;
+
+ rcu_read_unlock();
+
+ return dfs_domain_same;
+}
+
+static void reg_copy_dfs_chan_state(struct ieee80211_channel *dst_chan,
+ struct ieee80211_channel *src_chan)
+{
+ if (!(dst_chan->flags & IEEE80211_CHAN_RADAR) ||
+ !(src_chan->flags & IEEE80211_CHAN_RADAR))
+ return;
+
+ if (dst_chan->flags & IEEE80211_CHAN_DISABLED ||
+ src_chan->flags & IEEE80211_CHAN_DISABLED)
+ return;
+
+ if (src_chan->center_freq == dst_chan->center_freq &&
+ dst_chan->dfs_state == NL80211_DFS_USABLE) {
+ dst_chan->dfs_state = src_chan->dfs_state;
+ dst_chan->dfs_state_entered = src_chan->dfs_state_entered;
+ }
+}
+
+static void wiphy_share_dfs_chan_state(struct wiphy *dst_wiphy,
+ struct wiphy *src_wiphy)
+{
+ struct ieee80211_supported_band *src_sband, *dst_sband;
+ struct ieee80211_channel *src_chan, *dst_chan;
+ int i, j, band;
+
+ if (!reg_dfs_domain_same(dst_wiphy, src_wiphy))
+ return;
+
+ for (band = 0; band < NUM_NL80211_BANDS; band++) {
+ dst_sband = dst_wiphy->bands[band];
+ src_sband = src_wiphy->bands[band];
+ if (!dst_sband || !src_sband)
+ continue;
+
+ for (i = 0; i < dst_sband->n_channels; i++) {
+ dst_chan = &dst_sband->channels[i];
+ for (j = 0; j < src_sband->n_channels; j++) {
+ src_chan = &src_sband->channels[j];
+ reg_copy_dfs_chan_state(dst_chan, src_chan);
+ }
+ }
+ }
+}
+
+static void wiphy_all_share_dfs_chan_state(struct wiphy *wiphy)
+{
+ struct cfg80211_registered_device *rdev;
+
+ ASSERT_RTNL();
+
+ list_for_each_entry(rdev, &cfg80211_rdev_list, list) {
+ if (wiphy == &rdev->wiphy)
+ continue;
+ wiphy_share_dfs_chan_state(wiphy, &rdev->wiphy);
+ }
+}
+
/* This processes *all* regulatory hints */
static void reg_process_hint(struct regulatory_request *reg_request)
{
@@ -2110,6 +2192,7 @@
if (treatment == REG_REQ_ALREADY_SET && wiphy &&
wiphy->regulatory_flags & REGULATORY_STRICT_REG) {
wiphy_update_regulatory(wiphy, reg_request->initiator);
+ wiphy_all_share_dfs_chan_state(wiphy);
reg_check_channels();
}
@@ -3061,6 +3144,7 @@
lr = get_last_request();
wiphy_update_regulatory(wiphy, lr->initiator);
+ wiphy_all_share_dfs_chan_state(wiphy);
}
void wiphy_regulatory_deregister(struct wiphy *wiphy)
@@ -3120,6 +3204,67 @@
return reg_is_indoor;
}
+bool regulatory_pre_cac_allowed(struct wiphy *wiphy)
+{
+ const struct ieee80211_regdomain *regd = NULL;
+ const struct ieee80211_regdomain *wiphy_regd = NULL;
+ bool pre_cac_allowed = false;
+
+ rcu_read_lock();
+
+ regd = rcu_dereference(cfg80211_regdomain);
+ wiphy_regd = rcu_dereference(wiphy->regd);
+ if (!wiphy_regd) {
+ if (regd->dfs_region == NL80211_DFS_ETSI)
+ pre_cac_allowed = true;
+
+ rcu_read_unlock();
+
+ return pre_cac_allowed;
+ }
+
+ if (regd->dfs_region == wiphy_regd->dfs_region &&
+ wiphy_regd->dfs_region == NL80211_DFS_ETSI)
+ pre_cac_allowed = true;
+
+ rcu_read_unlock();
+
+ return pre_cac_allowed;
+}
+
+void regulatory_propagate_dfs_state(struct wiphy *wiphy,
+ struct cfg80211_chan_def *chandef,
+ enum nl80211_dfs_state dfs_state,
+ enum nl80211_radar_event event)
+{
+ struct cfg80211_registered_device *rdev;
+
+ ASSERT_RTNL();
+
+ if (WARN_ON(!cfg80211_chandef_valid(chandef)))
+ return;
+
+ list_for_each_entry(rdev, &cfg80211_rdev_list, list) {
+ if (wiphy == &rdev->wiphy)
+ continue;
+
+ if (!reg_dfs_domain_same(wiphy, &rdev->wiphy))
+ continue;
+
+ if (!ieee80211_get_channel(&rdev->wiphy,
+ chandef->chan->center_freq))
+ continue;
+
+ cfg80211_set_dfs_state(&rdev->wiphy, chandef, dfs_state);
+
+ if (event == NL80211_RADAR_DETECTED ||
+ event == NL80211_RADAR_CAC_FINISHED)
+ cfg80211_sched_dfs_chan_update(rdev);
+
+ nl80211_radar_notify(rdev, chandef, event, NULL, GFP_KERNEL);
+ }
+}
+
int __init regulatory_init(void)
{
int err = 0;
diff --git a/net/wireless/reg.h b/net/wireless/reg.h
index f6ced31..ca7fedf 100644
--- a/net/wireless/reg.h
+++ b/net/wireless/reg.h
@@ -143,4 +143,40 @@
*/
bool regulatory_indoor_allowed(void);
+/*
+ * Grace period to timeout pre-CAC results on the dfs channels. This timeout
+ * value is used for Non-ETSI domain.
+ * TODO: May be make this timeout available through regdb?
+ */
+#define REG_PRE_CAC_EXPIRY_GRACE_MS 2000
+
+/**
+ * regulatory_pre_cac_allowed - if pre-CAC allowed in the current dfs domain
+ * @wiphy: wiphy for which pre-CAC capability is checked.
+
+ * Pre-CAC is allowed only in ETSI domain.
+ */
+bool regulatory_pre_cac_allowed(struct wiphy *wiphy);
+
+/**
+ * regulatory_propagate_dfs_state - Propagate DFS channel state to other wiphys
+ * @wiphy - wiphy on which radar is detected and the event will be propagated
+ * to other available wiphys having the same DFS domain
+ * @chandef - Channel definition of radar detected channel
+ * @dfs_state - DFS channel state to be set
+ * @event - Type of radar event which triggered this DFS state change
+ *
+ * This function should be called with rtnl lock held.
+ */
+void regulatory_propagate_dfs_state(struct wiphy *wiphy,
+ struct cfg80211_chan_def *chandef,
+ enum nl80211_dfs_state dfs_state,
+ enum nl80211_radar_event event);
+
+/**
+ * reg_dfs_domain_same - Checks if both wiphy have same DFS domain configured
+ * @wiphy1 - wiphy it's dfs_region to be checked against that of wiphy2
+ * @wiphy2 - wiphy it's dfs_region to be checked against that of wiphy1
+ */
+bool reg_dfs_domain_same(struct wiphy *wiphy1, struct wiphy *wiphy2);
#endif /* __NET_WIRELESS_REG_H */
diff --git a/net/wireless/scan.c b/net/wireless/scan.c
index 21be56b..14d5f0c 100644
--- a/net/wireless/scan.c
+++ b/net/wireless/scan.c
@@ -300,93 +300,168 @@
}
EXPORT_SYMBOL(cfg80211_scan_done);
-void __cfg80211_sched_scan_results(struct work_struct *wk)
+void cfg80211_add_sched_scan_req(struct cfg80211_registered_device *rdev,
+ struct cfg80211_sched_scan_request *req)
{
- struct cfg80211_registered_device *rdev;
- struct cfg80211_sched_scan_request *request;
+ ASSERT_RTNL();
- rdev = container_of(wk, struct cfg80211_registered_device,
- sched_scan_results_wk);
+ list_add_rcu(&req->list, &rdev->sched_scan_req_list);
+}
- rtnl_lock();
+static void cfg80211_del_sched_scan_req(struct cfg80211_registered_device *rdev,
+ struct cfg80211_sched_scan_request *req)
+{
+ ASSERT_RTNL();
- request = rtnl_dereference(rdev->sched_scan_req);
+ list_del_rcu(&req->list);
+ kfree_rcu(req, rcu_head);
+}
- /* we don't have sched_scan_req anymore if the scan is stopping */
- if (request) {
- if (request->flags & NL80211_SCAN_FLAG_FLUSH) {
- /* flush entries from previous scans */
- spin_lock_bh(&rdev->bss_lock);
- __cfg80211_bss_expire(rdev, request->scan_start);
- spin_unlock_bh(&rdev->bss_lock);
- request->scan_start = jiffies;
- }
- nl80211_send_sched_scan(rdev, request->dev,
- NL80211_CMD_SCHED_SCAN_RESULTS);
+static struct cfg80211_sched_scan_request *
+cfg80211_find_sched_scan_req(struct cfg80211_registered_device *rdev, u64 reqid)
+{
+ struct cfg80211_sched_scan_request *pos;
+
+ ASSERT_RTNL();
+
+ list_for_each_entry(pos, &rdev->sched_scan_req_list, list) {
+ if (pos->reqid == reqid)
+ return pos;
+ }
+ return NULL;
+}
+
+/*
+ * Determines if a scheduled scan request can be handled. When a legacy
+ * scheduled scan is running no other scheduled scan is allowed regardless
+ * whether the request is for legacy or multi-support scan. When a multi-support
+ * scheduled scan is running a request for legacy scan is not allowed. In this
+ * case a request for multi-support scan can be handled if resources are
+ * available, ie. struct wiphy::max_sched_scan_reqs limit is not yet reached.
+ */
+int cfg80211_sched_scan_req_possible(struct cfg80211_registered_device *rdev,
+ bool want_multi)
+{
+ struct cfg80211_sched_scan_request *pos;
+ int i = 0;
+
+ list_for_each_entry(pos, &rdev->sched_scan_req_list, list) {
+ /* request id zero means legacy in progress */
+ if (!i && !pos->reqid)
+ return -EINPROGRESS;
+ i++;
}
+ if (i) {
+ /* no legacy allowed when multi request(s) are active */
+ if (!want_multi)
+ return -EINPROGRESS;
+
+ /* resource limit reached */
+ if (i == rdev->wiphy.max_sched_scan_reqs)
+ return -ENOSPC;
+ }
+ return 0;
+}
+
+void cfg80211_sched_scan_results_wk(struct work_struct *work)
+{
+ struct cfg80211_registered_device *rdev;
+ struct cfg80211_sched_scan_request *req, *tmp;
+
+ rdev = container_of(work, struct cfg80211_registered_device,
+ sched_scan_res_wk);
+
+ rtnl_lock();
+ list_for_each_entry_safe(req, tmp, &rdev->sched_scan_req_list, list) {
+ if (req->report_results) {
+ req->report_results = false;
+ if (req->flags & NL80211_SCAN_FLAG_FLUSH) {
+ /* flush entries from previous scans */
+ spin_lock_bh(&rdev->bss_lock);
+ __cfg80211_bss_expire(rdev, req->scan_start);
+ spin_unlock_bh(&rdev->bss_lock);
+ req->scan_start = jiffies;
+ }
+ nl80211_send_sched_scan(req,
+ NL80211_CMD_SCHED_SCAN_RESULTS);
+ }
+ }
rtnl_unlock();
}
-void cfg80211_sched_scan_results(struct wiphy *wiphy)
+void cfg80211_sched_scan_results(struct wiphy *wiphy, u64 reqid)
{
- trace_cfg80211_sched_scan_results(wiphy);
+ struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy);
+ struct cfg80211_sched_scan_request *request;
+
+ trace_cfg80211_sched_scan_results(wiphy, reqid);
/* ignore if we're not scanning */
- if (rcu_access_pointer(wiphy_to_rdev(wiphy)->sched_scan_req))
- queue_work(cfg80211_wq,
- &wiphy_to_rdev(wiphy)->sched_scan_results_wk);
+ rtnl_lock();
+ request = cfg80211_find_sched_scan_req(rdev, reqid);
+ if (request) {
+ request->report_results = true;
+ queue_work(cfg80211_wq, &rdev->sched_scan_res_wk);
+ }
+ rtnl_unlock();
}
EXPORT_SYMBOL(cfg80211_sched_scan_results);
-void cfg80211_sched_scan_stopped_rtnl(struct wiphy *wiphy)
+void cfg80211_sched_scan_stopped_rtnl(struct wiphy *wiphy, u64 reqid)
{
struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy);
ASSERT_RTNL();
- trace_cfg80211_sched_scan_stopped(wiphy);
+ trace_cfg80211_sched_scan_stopped(wiphy, reqid);
- __cfg80211_stop_sched_scan(rdev, true);
+ __cfg80211_stop_sched_scan(rdev, reqid, true);
}
EXPORT_SYMBOL(cfg80211_sched_scan_stopped_rtnl);
-void cfg80211_sched_scan_stopped(struct wiphy *wiphy)
+void cfg80211_sched_scan_stopped(struct wiphy *wiphy, u64 reqid)
{
rtnl_lock();
- cfg80211_sched_scan_stopped_rtnl(wiphy);
+ cfg80211_sched_scan_stopped_rtnl(wiphy, reqid);
rtnl_unlock();
}
EXPORT_SYMBOL(cfg80211_sched_scan_stopped);
-int __cfg80211_stop_sched_scan(struct cfg80211_registered_device *rdev,
- bool driver_initiated)
+int cfg80211_stop_sched_scan_req(struct cfg80211_registered_device *rdev,
+ struct cfg80211_sched_scan_request *req,
+ bool driver_initiated)
{
- struct cfg80211_sched_scan_request *sched_scan_req;
- struct net_device *dev;
-
ASSERT_RTNL();
- if (!rdev->sched_scan_req)
- return -ENOENT;
-
- sched_scan_req = rtnl_dereference(rdev->sched_scan_req);
- dev = sched_scan_req->dev;
-
if (!driver_initiated) {
- int err = rdev_sched_scan_stop(rdev, dev);
+ int err = rdev_sched_scan_stop(rdev, req->dev, req->reqid);
if (err)
return err;
}
- nl80211_send_sched_scan(rdev, dev, NL80211_CMD_SCHED_SCAN_STOPPED);
+ nl80211_send_sched_scan(req, NL80211_CMD_SCHED_SCAN_STOPPED);
- RCU_INIT_POINTER(rdev->sched_scan_req, NULL);
- kfree_rcu(sched_scan_req, rcu_head);
+ cfg80211_del_sched_scan_req(rdev, req);
return 0;
}
+int __cfg80211_stop_sched_scan(struct cfg80211_registered_device *rdev,
+ u64 reqid, bool driver_initiated)
+{
+ struct cfg80211_sched_scan_request *sched_scan_req;
+
+ ASSERT_RTNL();
+
+ sched_scan_req = cfg80211_find_sched_scan_req(rdev, reqid);
+ if (!sched_scan_req)
+ return -ENOENT;
+
+ return cfg80211_stop_sched_scan_req(rdev, sched_scan_req,
+ driver_initiated);
+}
+
void cfg80211_bss_age(struct cfg80211_registered_device *rdev,
unsigned long age_secs)
{
diff --git a/net/wireless/sme.c b/net/wireless/sme.c
index b347e63..532a000 100644
--- a/net/wireless/sme.c
+++ b/net/wireless/sme.c
@@ -5,6 +5,7 @@
*
* Copyright 2009 Johannes Berg <johannes@sipsolutions.net>
* Copyright (C) 2009 Intel Corporation. All rights reserved.
+ * Copyright 2017 Intel Deutschland GmbH
*/
#include <linux/etherdevice.h>
@@ -253,10 +254,13 @@
}
treason = NL80211_TIMEOUT_UNSPECIFIED;
if (cfg80211_conn_do_work(wdev, &treason)) {
- __cfg80211_connect_result(
- wdev->netdev, bssid,
- NULL, 0, NULL, 0, -1, false, NULL,
- treason);
+ struct cfg80211_connect_resp_params cr;
+
+ memset(&cr, 0, sizeof(cr));
+ cr.status = -1;
+ cr.bssid = bssid;
+ cr.timeout_reason = treason;
+ __cfg80211_connect_result(wdev->netdev, &cr, false);
}
wdev_unlock(wdev);
}
@@ -359,10 +363,13 @@
wdev->conn->state = CFG80211_CONN_AUTHENTICATE_NEXT;
schedule_work(&rdev->conn_work);
} else if (status_code != WLAN_STATUS_SUCCESS) {
- __cfg80211_connect_result(wdev->netdev, mgmt->bssid,
- NULL, 0, NULL, 0,
- status_code, false, NULL,
- NL80211_TIMEOUT_UNSPECIFIED);
+ struct cfg80211_connect_resp_params cr;
+
+ memset(&cr, 0, sizeof(cr));
+ cr.status = status_code;
+ cr.bssid = mgmt->bssid;
+ cr.timeout_reason = NL80211_TIMEOUT_UNSPECIFIED;
+ __cfg80211_connect_result(wdev->netdev, &cr, false);
} else if (wdev->conn->state == CFG80211_CONN_AUTHENTICATING) {
wdev->conn->state = CFG80211_CONN_ASSOCIATE_NEXT;
schedule_work(&rdev->conn_work);
@@ -669,12 +676,9 @@
*/
/* This method must consume bss one way or another */
-void __cfg80211_connect_result(struct net_device *dev, const u8 *bssid,
- const u8 *req_ie, size_t req_ie_len,
- const u8 *resp_ie, size_t resp_ie_len,
- int status, bool wextev,
- struct cfg80211_bss *bss,
- enum nl80211_timeout_reason timeout_reason)
+void __cfg80211_connect_result(struct net_device *dev,
+ struct cfg80211_connect_resp_params *cr,
+ bool wextev)
{
struct wireless_dev *wdev = dev->ieee80211_ptr;
const u8 *country_ie;
@@ -686,48 +690,48 @@
if (WARN_ON(wdev->iftype != NL80211_IFTYPE_STATION &&
wdev->iftype != NL80211_IFTYPE_P2P_CLIENT)) {
- cfg80211_put_bss(wdev->wiphy, bss);
+ cfg80211_put_bss(wdev->wiphy, cr->bss);
return;
}
- nl80211_send_connect_result(wiphy_to_rdev(wdev->wiphy), dev,
- bssid, req_ie, req_ie_len,
- resp_ie, resp_ie_len,
- status, timeout_reason, GFP_KERNEL);
+ nl80211_send_connect_result(wiphy_to_rdev(wdev->wiphy), dev, cr,
+ GFP_KERNEL);
#ifdef CONFIG_CFG80211_WEXT
if (wextev) {
- if (req_ie && status == WLAN_STATUS_SUCCESS) {
+ if (cr->req_ie && cr->status == WLAN_STATUS_SUCCESS) {
memset(&wrqu, 0, sizeof(wrqu));
- wrqu.data.length = req_ie_len;
- wireless_send_event(dev, IWEVASSOCREQIE, &wrqu, req_ie);
+ wrqu.data.length = cr->req_ie_len;
+ wireless_send_event(dev, IWEVASSOCREQIE, &wrqu,
+ cr->req_ie);
}
- if (resp_ie && status == WLAN_STATUS_SUCCESS) {
+ if (cr->resp_ie && cr->status == WLAN_STATUS_SUCCESS) {
memset(&wrqu, 0, sizeof(wrqu));
- wrqu.data.length = resp_ie_len;
- wireless_send_event(dev, IWEVASSOCRESPIE, &wrqu, resp_ie);
+ wrqu.data.length = cr->resp_ie_len;
+ wireless_send_event(dev, IWEVASSOCRESPIE, &wrqu,
+ cr->resp_ie);
}
memset(&wrqu, 0, sizeof(wrqu));
wrqu.ap_addr.sa_family = ARPHRD_ETHER;
- if (bssid && status == WLAN_STATUS_SUCCESS) {
- memcpy(wrqu.ap_addr.sa_data, bssid, ETH_ALEN);
- memcpy(wdev->wext.prev_bssid, bssid, ETH_ALEN);
+ if (cr->bssid && cr->status == WLAN_STATUS_SUCCESS) {
+ memcpy(wrqu.ap_addr.sa_data, cr->bssid, ETH_ALEN);
+ memcpy(wdev->wext.prev_bssid, cr->bssid, ETH_ALEN);
wdev->wext.prev_bssid_valid = true;
}
wireless_send_event(dev, SIOCGIWAP, &wrqu, NULL);
}
#endif
- if (!bss && (status == WLAN_STATUS_SUCCESS)) {
+ if (!cr->bss && (cr->status == WLAN_STATUS_SUCCESS)) {
WARN_ON_ONCE(!wiphy_to_rdev(wdev->wiphy)->ops->connect);
- bss = cfg80211_get_bss(wdev->wiphy, NULL, bssid,
- wdev->ssid, wdev->ssid_len,
- wdev->conn_bss_type,
- IEEE80211_PRIVACY_ANY);
- if (bss)
- cfg80211_hold_bss(bss_from_pub(bss));
+ cr->bss = cfg80211_get_bss(wdev->wiphy, NULL, cr->bssid,
+ wdev->ssid, wdev->ssid_len,
+ wdev->conn_bss_type,
+ IEEE80211_PRIVACY_ANY);
+ if (cr->bss)
+ cfg80211_hold_bss(bss_from_pub(cr->bss));
}
if (wdev->current_bss) {
@@ -736,29 +740,29 @@
wdev->current_bss = NULL;
}
- if (status != WLAN_STATUS_SUCCESS) {
+ if (cr->status != WLAN_STATUS_SUCCESS) {
kzfree(wdev->connect_keys);
wdev->connect_keys = NULL;
wdev->ssid_len = 0;
wdev->conn_owner_nlportid = 0;
- if (bss) {
- cfg80211_unhold_bss(bss_from_pub(bss));
- cfg80211_put_bss(wdev->wiphy, bss);
+ if (cr->bss) {
+ cfg80211_unhold_bss(bss_from_pub(cr->bss));
+ cfg80211_put_bss(wdev->wiphy, cr->bss);
}
cfg80211_sme_free(wdev);
return;
}
- if (WARN_ON(!bss))
+ if (WARN_ON(!cr->bss))
return;
- wdev->current_bss = bss_from_pub(bss);
+ wdev->current_bss = bss_from_pub(cr->bss);
if (!(wdev->wiphy->flags & WIPHY_FLAG_HAS_STATIC_WEP))
cfg80211_upload_connect_keys(wdev);
rcu_read_lock();
- country_ie = ieee80211_bss_get_ie(bss, WLAN_EID_COUNTRY);
+ country_ie = ieee80211_bss_get_ie(cr->bss, WLAN_EID_COUNTRY);
if (!country_ie) {
rcu_read_unlock();
return;
@@ -775,70 +779,99 @@
* - country_ie + 2, the start of the country ie data, and
* - and country_ie[1] which is the IE length
*/
- regulatory_hint_country_ie(wdev->wiphy, bss->channel->band,
+ regulatory_hint_country_ie(wdev->wiphy, cr->bss->channel->band,
country_ie + 2, country_ie[1]);
kfree(country_ie);
}
/* Consumes bss object one way or another */
-void cfg80211_connect_bss(struct net_device *dev, const u8 *bssid,
- struct cfg80211_bss *bss, const u8 *req_ie,
- size_t req_ie_len, const u8 *resp_ie,
- size_t resp_ie_len, int status, gfp_t gfp,
- enum nl80211_timeout_reason timeout_reason)
+void cfg80211_connect_done(struct net_device *dev,
+ struct cfg80211_connect_resp_params *params,
+ gfp_t gfp)
{
struct wireless_dev *wdev = dev->ieee80211_ptr;
struct cfg80211_registered_device *rdev = wiphy_to_rdev(wdev->wiphy);
struct cfg80211_event *ev;
unsigned long flags;
+ u8 *next;
- if (bss) {
+ if (params->bss) {
/* Make sure the bss entry provided by the driver is valid. */
- struct cfg80211_internal_bss *ibss = bss_from_pub(bss);
+ struct cfg80211_internal_bss *ibss = bss_from_pub(params->bss);
if (WARN_ON(list_empty(&ibss->list))) {
- cfg80211_put_bss(wdev->wiphy, bss);
+ cfg80211_put_bss(wdev->wiphy, params->bss);
return;
}
}
- ev = kzalloc(sizeof(*ev) + req_ie_len + resp_ie_len, gfp);
+ ev = kzalloc(sizeof(*ev) + (params->bssid ? ETH_ALEN : 0) +
+ params->req_ie_len + params->resp_ie_len +
+ params->fils_kek_len + params->pmk_len +
+ (params->pmkid ? WLAN_PMKID_LEN : 0), gfp);
if (!ev) {
- cfg80211_put_bss(wdev->wiphy, bss);
+ cfg80211_put_bss(wdev->wiphy, params->bss);
return;
}
ev->type = EVENT_CONNECT_RESULT;
- if (bssid)
- memcpy(ev->cr.bssid, bssid, ETH_ALEN);
- if (req_ie_len) {
- ev->cr.req_ie = ((u8 *)ev) + sizeof(*ev);
- ev->cr.req_ie_len = req_ie_len;
- memcpy((void *)ev->cr.req_ie, req_ie, req_ie_len);
+ next = ((u8 *)ev) + sizeof(*ev);
+ if (params->bssid) {
+ ev->cr.bssid = next;
+ memcpy((void *)ev->cr.bssid, params->bssid, ETH_ALEN);
+ next += ETH_ALEN;
}
- if (resp_ie_len) {
- ev->cr.resp_ie = ((u8 *)ev) + sizeof(*ev) + req_ie_len;
- ev->cr.resp_ie_len = resp_ie_len;
- memcpy((void *)ev->cr.resp_ie, resp_ie, resp_ie_len);
+ if (params->req_ie_len) {
+ ev->cr.req_ie = next;
+ ev->cr.req_ie_len = params->req_ie_len;
+ memcpy((void *)ev->cr.req_ie, params->req_ie,
+ params->req_ie_len);
+ next += params->req_ie_len;
}
- if (bss)
- cfg80211_hold_bss(bss_from_pub(bss));
- ev->cr.bss = bss;
- ev->cr.status = status;
- ev->cr.timeout_reason = timeout_reason;
+ if (params->resp_ie_len) {
+ ev->cr.resp_ie = next;
+ ev->cr.resp_ie_len = params->resp_ie_len;
+ memcpy((void *)ev->cr.resp_ie, params->resp_ie,
+ params->resp_ie_len);
+ next += params->resp_ie_len;
+ }
+ if (params->fils_kek_len) {
+ ev->cr.fils_kek = next;
+ ev->cr.fils_kek_len = params->fils_kek_len;
+ memcpy((void *)ev->cr.fils_kek, params->fils_kek,
+ params->fils_kek_len);
+ next += params->fils_kek_len;
+ }
+ if (params->pmk_len) {
+ ev->cr.pmk = next;
+ ev->cr.pmk_len = params->pmk_len;
+ memcpy((void *)ev->cr.pmk, params->pmk, params->pmk_len);
+ next += params->pmk_len;
+ }
+ if (params->pmkid) {
+ ev->cr.pmkid = next;
+ memcpy((void *)ev->cr.pmkid, params->pmkid, WLAN_PMKID_LEN);
+ next += WLAN_PMKID_LEN;
+ }
+ ev->cr.update_erp_next_seq_num = params->update_erp_next_seq_num;
+ if (params->update_erp_next_seq_num)
+ ev->cr.fils_erp_next_seq_num = params->fils_erp_next_seq_num;
+ if (params->bss)
+ cfg80211_hold_bss(bss_from_pub(params->bss));
+ ev->cr.bss = params->bss;
+ ev->cr.status = params->status;
+ ev->cr.timeout_reason = params->timeout_reason;
spin_lock_irqsave(&wdev->event_lock, flags);
list_add_tail(&ev->list, &wdev->event_list);
spin_unlock_irqrestore(&wdev->event_lock, flags);
queue_work(cfg80211_wq, &rdev->event_work);
}
-EXPORT_SYMBOL(cfg80211_connect_bss);
+EXPORT_SYMBOL(cfg80211_connect_done);
/* Consumes bss object one way or another */
void __cfg80211_roamed(struct wireless_dev *wdev,
- struct cfg80211_bss *bss,
- const u8 *req_ie, size_t req_ie_len,
- const u8 *resp_ie, size_t resp_ie_len)
+ struct cfg80211_roam_info *info)
{
#ifdef CONFIG_CFG80211_WEXT
union iwreq_data wrqu;
@@ -856,97 +889,84 @@
cfg80211_put_bss(wdev->wiphy, &wdev->current_bss->pub);
wdev->current_bss = NULL;
- cfg80211_hold_bss(bss_from_pub(bss));
- wdev->current_bss = bss_from_pub(bss);
+ if (WARN_ON(!info->bss))
+ return;
+
+ cfg80211_hold_bss(bss_from_pub(info->bss));
+ wdev->current_bss = bss_from_pub(info->bss);
nl80211_send_roamed(wiphy_to_rdev(wdev->wiphy),
- wdev->netdev, bss->bssid,
- req_ie, req_ie_len, resp_ie, resp_ie_len,
- GFP_KERNEL);
+ wdev->netdev, info, GFP_KERNEL);
#ifdef CONFIG_CFG80211_WEXT
- if (req_ie) {
+ if (info->req_ie) {
memset(&wrqu, 0, sizeof(wrqu));
- wrqu.data.length = req_ie_len;
+ wrqu.data.length = info->req_ie_len;
wireless_send_event(wdev->netdev, IWEVASSOCREQIE,
- &wrqu, req_ie);
+ &wrqu, info->req_ie);
}
- if (resp_ie) {
+ if (info->resp_ie) {
memset(&wrqu, 0, sizeof(wrqu));
- wrqu.data.length = resp_ie_len;
+ wrqu.data.length = info->resp_ie_len;
wireless_send_event(wdev->netdev, IWEVASSOCRESPIE,
- &wrqu, resp_ie);
+ &wrqu, info->resp_ie);
}
memset(&wrqu, 0, sizeof(wrqu));
wrqu.ap_addr.sa_family = ARPHRD_ETHER;
- memcpy(wrqu.ap_addr.sa_data, bss->bssid, ETH_ALEN);
- memcpy(wdev->wext.prev_bssid, bss->bssid, ETH_ALEN);
+ memcpy(wrqu.ap_addr.sa_data, info->bss->bssid, ETH_ALEN);
+ memcpy(wdev->wext.prev_bssid, info->bss->bssid, ETH_ALEN);
wdev->wext.prev_bssid_valid = true;
wireless_send_event(wdev->netdev, SIOCGIWAP, &wrqu, NULL);
#endif
return;
out:
- cfg80211_put_bss(wdev->wiphy, bss);
+ cfg80211_put_bss(wdev->wiphy, info->bss);
}
-void cfg80211_roamed(struct net_device *dev,
- struct ieee80211_channel *channel,
- const u8 *bssid,
- const u8 *req_ie, size_t req_ie_len,
- const u8 *resp_ie, size_t resp_ie_len, gfp_t gfp)
-{
- struct wireless_dev *wdev = dev->ieee80211_ptr;
- struct cfg80211_bss *bss;
-
- bss = cfg80211_get_bss(wdev->wiphy, channel, bssid, wdev->ssid,
- wdev->ssid_len,
- wdev->conn_bss_type, IEEE80211_PRIVACY_ANY);
- if (WARN_ON(!bss))
- return;
-
- cfg80211_roamed_bss(dev, bss, req_ie, req_ie_len, resp_ie,
- resp_ie_len, gfp);
-}
-EXPORT_SYMBOL(cfg80211_roamed);
-
-/* Consumes bss object one way or another */
-void cfg80211_roamed_bss(struct net_device *dev,
- struct cfg80211_bss *bss, const u8 *req_ie,
- size_t req_ie_len, const u8 *resp_ie,
- size_t resp_ie_len, gfp_t gfp)
+/* Consumes info->bss object one way or another */
+void cfg80211_roamed(struct net_device *dev, struct cfg80211_roam_info *info,
+ gfp_t gfp)
{
struct wireless_dev *wdev = dev->ieee80211_ptr;
struct cfg80211_registered_device *rdev = wiphy_to_rdev(wdev->wiphy);
struct cfg80211_event *ev;
unsigned long flags;
- if (WARN_ON(!bss))
+ if (!info->bss) {
+ info->bss = cfg80211_get_bss(wdev->wiphy, info->channel,
+ info->bssid, wdev->ssid,
+ wdev->ssid_len,
+ wdev->conn_bss_type,
+ IEEE80211_PRIVACY_ANY);
+ }
+
+ if (WARN_ON(!info->bss))
return;
- ev = kzalloc(sizeof(*ev) + req_ie_len + resp_ie_len, gfp);
+ ev = kzalloc(sizeof(*ev) + info->req_ie_len + info->resp_ie_len, gfp);
if (!ev) {
- cfg80211_put_bss(wdev->wiphy, bss);
+ cfg80211_put_bss(wdev->wiphy, info->bss);
return;
}
ev->type = EVENT_ROAMED;
ev->rm.req_ie = ((u8 *)ev) + sizeof(*ev);
- ev->rm.req_ie_len = req_ie_len;
- memcpy((void *)ev->rm.req_ie, req_ie, req_ie_len);
- ev->rm.resp_ie = ((u8 *)ev) + sizeof(*ev) + req_ie_len;
- ev->rm.resp_ie_len = resp_ie_len;
- memcpy((void *)ev->rm.resp_ie, resp_ie, resp_ie_len);
- ev->rm.bss = bss;
+ ev->rm.req_ie_len = info->req_ie_len;
+ memcpy((void *)ev->rm.req_ie, info->req_ie, info->req_ie_len);
+ ev->rm.resp_ie = ((u8 *)ev) + sizeof(*ev) + info->req_ie_len;
+ ev->rm.resp_ie_len = info->resp_ie_len;
+ memcpy((void *)ev->rm.resp_ie, info->resp_ie, info->resp_ie_len);
+ ev->rm.bss = info->bss;
spin_lock_irqsave(&wdev->event_lock, flags);
list_add_tail(&ev->list, &wdev->event_list);
spin_unlock_irqrestore(&wdev->event_lock, flags);
queue_work(cfg80211_wq, &rdev->event_work);
}
-EXPORT_SYMBOL(cfg80211_roamed_bss);
+EXPORT_SYMBOL(cfg80211_roamed);
void __cfg80211_disconnected(struct net_device *dev, const u8 *ie,
size_t ie_len, u16 reason, bool from_ap)
diff --git a/net/wireless/trace.h b/net/wireless/trace.h
index 776e80c..ca8b205 100644
--- a/net/wireless/trace.h
+++ b/net/wireless/trace.h
@@ -576,11 +576,6 @@
TP_ARGS(wiphy, netdev)
);
-DEFINE_EVENT(wiphy_netdev_evt, rdev_sched_scan_stop,
- TP_PROTO(struct wiphy *wiphy, struct net_device *netdev),
- TP_ARGS(wiphy, netdev)
-);
-
DEFINE_EVENT(wiphy_netdev_evt, rdev_set_rekey_data,
TP_PROTO(struct wiphy *wiphy, struct net_device *netdev),
TP_ARGS(wiphy, netdev)
@@ -1322,6 +1317,28 @@
__entry->rssi_thold, __entry->rssi_hyst)
);
+TRACE_EVENT(rdev_set_cqm_rssi_range_config,
+ TP_PROTO(struct wiphy *wiphy,
+ struct net_device *netdev, s32 low, s32 high),
+ TP_ARGS(wiphy, netdev, low, high),
+ TP_STRUCT__entry(
+ WIPHY_ENTRY
+ NETDEV_ENTRY
+ __field(s32, rssi_low)
+ __field(s32, rssi_high)
+ ),
+ TP_fast_assign(
+ WIPHY_ASSIGN;
+ NETDEV_ASSIGN;
+ __entry->rssi_low = low;
+ __entry->rssi_high = high;
+ ),
+ TP_printk(WIPHY_PR_FMT ", " NETDEV_PR_FMT
+ ", range: %d - %d ",
+ WIPHY_PR_ARG, NETDEV_PR_ARG,
+ __entry->rssi_low, __entry->rssi_high)
+);
+
TRACE_EVENT(rdev_set_cqm_txe_config,
TP_PROTO(struct wiphy *wiphy, struct net_device *netdev, u32 rate,
u32 pkts, u32 intvl),
@@ -1588,20 +1605,31 @@
TP_ARGS(wiphy, rx, tx)
);
-TRACE_EVENT(rdev_sched_scan_start,
- TP_PROTO(struct wiphy *wiphy, struct net_device *netdev,
- struct cfg80211_sched_scan_request *request),
- TP_ARGS(wiphy, netdev, request),
+DECLARE_EVENT_CLASS(wiphy_netdev_id_evt,
+ TP_PROTO(struct wiphy *wiphy, struct net_device *netdev, u64 id),
+ TP_ARGS(wiphy, netdev, id),
TP_STRUCT__entry(
WIPHY_ENTRY
NETDEV_ENTRY
+ __field(u64, id)
),
TP_fast_assign(
WIPHY_ASSIGN;
NETDEV_ASSIGN;
+ __entry->id = id;
),
- TP_printk(WIPHY_PR_FMT ", " NETDEV_PR_FMT,
- WIPHY_PR_ARG, NETDEV_PR_ARG)
+ TP_printk(WIPHY_PR_FMT ", " NETDEV_PR_FMT ", id: %llu",
+ WIPHY_PR_ARG, NETDEV_PR_ARG, __entry->id)
+);
+
+DEFINE_EVENT(wiphy_netdev_id_evt, rdev_sched_scan_start,
+ TP_PROTO(struct wiphy *wiphy, struct net_device *netdev, u64 id),
+ TP_ARGS(wiphy, netdev, id)
+);
+
+DEFINE_EVENT(wiphy_netdev_id_evt, rdev_sched_scan_stop,
+ TP_PROTO(struct wiphy *wiphy, struct net_device *netdev, u64 id),
+ TP_ARGS(wiphy, netdev, id)
);
TRACE_EVENT(rdev_tdls_mgmt,
@@ -2792,14 +2820,28 @@
MAC_PR_ARG(tsf_bssid))
);
-DEFINE_EVENT(wiphy_only_evt, cfg80211_sched_scan_results,
- TP_PROTO(struct wiphy *wiphy),
- TP_ARGS(wiphy)
+DECLARE_EVENT_CLASS(wiphy_id_evt,
+ TP_PROTO(struct wiphy *wiphy, u64 id),
+ TP_ARGS(wiphy, id),
+ TP_STRUCT__entry(
+ WIPHY_ENTRY
+ __field(u64, id)
+ ),
+ TP_fast_assign(
+ WIPHY_ASSIGN;
+ __entry->id = id;
+ ),
+ TP_printk(WIPHY_PR_FMT ", id: %llu", WIPHY_PR_ARG, __entry->id)
);
-DEFINE_EVENT(wiphy_only_evt, cfg80211_sched_scan_stopped,
- TP_PROTO(struct wiphy *wiphy),
- TP_ARGS(wiphy)
+DEFINE_EVENT(wiphy_id_evt, cfg80211_sched_scan_stopped,
+ TP_PROTO(struct wiphy *wiphy, u64 id),
+ TP_ARGS(wiphy, id)
+);
+
+DEFINE_EVENT(wiphy_id_evt, cfg80211_sched_scan_results,
+ TP_PROTO(struct wiphy *wiphy, u64 id),
+ TP_ARGS(wiphy, id)
);
TRACE_EVENT(cfg80211_get_bss,
diff --git a/net/wireless/util.c b/net/wireless/util.c
index 68e5f2ec..7198373 100644
--- a/net/wireless/util.c
+++ b/net/wireless/util.c
@@ -659,7 +659,7 @@
int offset, int len)
{
struct skb_shared_info *sh = skb_shinfo(skb);
- const skb_frag_t *frag = &sh->frags[-1];
+ const skb_frag_t *frag = &sh->frags[0];
struct page *frag_page;
void *frag_ptr;
int frag_len, frag_size;
@@ -672,10 +672,10 @@
while (offset >= frag_size) {
offset -= frag_size;
- frag++;
frag_page = skb_frag_page(frag);
frag_ptr = skb_frag_address(frag);
frag_size = skb_frag_size(frag);
+ frag++;
}
frag_ptr += offset;
@@ -687,12 +687,12 @@
len -= cur_len;
while (len > 0) {
- frag++;
frag_len = skb_frag_size(frag);
cur_len = min(len, frag_len);
__frame_add_frag(frame, skb_frag_page(frag),
skb_frag_address(frag), cur_len, frag_len);
len -= cur_len;
+ frag++;
}
}
@@ -914,11 +914,11 @@
netdev_err(dev, "failed to set key %d\n", i);
continue;
}
- if (wdev->connect_keys->def == i)
- if (rdev_set_default_key(rdev, dev, i, true, true)) {
- netdev_err(dev, "failed to set defkey %d\n", i);
- continue;
- }
+ if (wdev->connect_keys->def == i &&
+ rdev_set_default_key(rdev, dev, i, true, true)) {
+ netdev_err(dev, "failed to set defkey %d\n", i);
+ continue;
+ }
}
kzfree(wdev->connect_keys);
@@ -929,7 +929,6 @@
{
struct cfg80211_event *ev;
unsigned long flags;
- const u8 *bssid = NULL;
spin_lock_irqsave(&wdev->event_lock, flags);
while (!list_empty(&wdev->event_list)) {
@@ -941,20 +940,13 @@
wdev_lock(wdev);
switch (ev->type) {
case EVENT_CONNECT_RESULT:
- if (!is_zero_ether_addr(ev->cr.bssid))
- bssid = ev->cr.bssid;
__cfg80211_connect_result(
- wdev->netdev, bssid,
- ev->cr.req_ie, ev->cr.req_ie_len,
- ev->cr.resp_ie, ev->cr.resp_ie_len,
- ev->cr.status,
- ev->cr.status == WLAN_STATUS_SUCCESS,
- ev->cr.bss, ev->cr.timeout_reason);
+ wdev->netdev,
+ &ev->cr,
+ ev->cr.status == WLAN_STATUS_SUCCESS);
break;
case EVENT_ROAMED:
- __cfg80211_roamed(wdev, ev->rm.bss, ev->rm.req_ie,
- ev->rm.req_ie_len, ev->rm.resp_ie,
- ev->rm.resp_ie_len);
+ __cfg80211_roamed(wdev, &ev->rm);
break;
case EVENT_DISCONNECTED:
__cfg80211_disconnected(wdev->netdev,
@@ -991,7 +983,7 @@
int cfg80211_change_iface(struct cfg80211_registered_device *rdev,
struct net_device *dev, enum nl80211_iftype ntype,
- u32 *flags, struct vif_params *params)
+ struct vif_params *params)
{
int err;
enum nl80211_iftype otype = dev->ieee80211_ptr->iftype;
@@ -1049,7 +1041,7 @@
cfg80211_process_rdev_events(rdev);
}
- err = rdev_change_virtual_intf(rdev, dev, ntype, flags, params);
+ err = rdev_change_virtual_intf(rdev, dev, ntype, params);
WARN_ON(!err && dev->ieee80211_ptr->iftype != ntype);
@@ -1097,6 +1089,35 @@
return err;
}
+static u32 cfg80211_calculate_bitrate_ht(struct rate_info *rate)
+{
+ int modulation, streams, bitrate;
+
+ /* the formula below does only work for MCS values smaller than 32 */
+ if (WARN_ON_ONCE(rate->mcs >= 32))
+ return 0;
+
+ modulation = rate->mcs & 7;
+ streams = (rate->mcs >> 3) + 1;
+
+ bitrate = (rate->bw == RATE_INFO_BW_40) ? 13500000 : 6500000;
+
+ if (modulation < 4)
+ bitrate *= (modulation + 1);
+ else if (modulation == 4)
+ bitrate *= (modulation + 2);
+ else
+ bitrate *= (modulation + 3);
+
+ bitrate *= streams;
+
+ if (rate->flags & RATE_INFO_FLAGS_SHORT_GI)
+ bitrate = (bitrate / 9) * 10;
+
+ /* do NOT round down here */
+ return (bitrate + 50000) / 100000;
+}
+
static u32 cfg80211_calculate_bitrate_60g(struct rate_info *rate)
{
static const u32 __mcs2bitrate[] = {
@@ -1230,39 +1251,14 @@
u32 cfg80211_calculate_bitrate(struct rate_info *rate)
{
- int modulation, streams, bitrate;
-
- if (!(rate->flags & RATE_INFO_FLAGS_MCS) &&
- !(rate->flags & RATE_INFO_FLAGS_VHT_MCS))
- return rate->legacy;
+ if (rate->flags & RATE_INFO_FLAGS_MCS)
+ return cfg80211_calculate_bitrate_ht(rate);
if (rate->flags & RATE_INFO_FLAGS_60G)
return cfg80211_calculate_bitrate_60g(rate);
if (rate->flags & RATE_INFO_FLAGS_VHT_MCS)
return cfg80211_calculate_bitrate_vht(rate);
- /* the formula below does only work for MCS values smaller than 32 */
- if (WARN_ON_ONCE(rate->mcs >= 32))
- return 0;
-
- modulation = rate->mcs & 7;
- streams = (rate->mcs >> 3) + 1;
-
- bitrate = (rate->bw == RATE_INFO_BW_40) ? 13500000 : 6500000;
-
- if (modulation < 4)
- bitrate *= (modulation + 1);
- else if (modulation == 4)
- bitrate *= (modulation + 2);
- else
- bitrate *= (modulation + 3);
-
- bitrate *= streams;
-
- if (rate->flags & RATE_INFO_FLAGS_SHORT_GI)
- bitrate = (bitrate / 9) * 10;
-
- /* do NOT round down here */
- return (bitrate + 50000) / 100000;
+ return rate->legacy;
}
EXPORT_SYMBOL(cfg80211_calculate_bitrate);
diff --git a/net/wireless/wext-compat.c b/net/wireless/wext-compat.c
index a220156..5d4a02c7 100644
--- a/net/wireless/wext-compat.c
+++ b/net/wireless/wext-compat.c
@@ -62,7 +62,7 @@
memset(&vifparams, 0, sizeof(vifparams));
- return cfg80211_change_iface(rdev, dev, type, NULL, &vifparams);
+ return cfg80211_change_iface(rdev, dev, type, &vifparams);
}
EXPORT_WEXT_HANDLER(cfg80211_wext_siwmode);
diff --git a/net/xfrm/Makefile b/net/xfrm/Makefile
index c0e9619..abf81b3 100644
--- a/net/xfrm/Makefile
+++ b/net/xfrm/Makefile
@@ -5,6 +5,7 @@
obj-$(CONFIG_XFRM) := xfrm_policy.o xfrm_state.o xfrm_hash.o \
xfrm_input.o xfrm_output.o \
xfrm_sysctl.o xfrm_replay.o
+obj-$(CONFIG_XFRM_OFFLOAD) += xfrm_device.o
obj-$(CONFIG_XFRM_STATISTICS) += xfrm_proc.o
obj-$(CONFIG_XFRM_ALGO) += xfrm_algo.o
obj-$(CONFIG_XFRM_USER) += xfrm_user.o
diff --git a/net/xfrm/xfrm_device.c b/net/xfrm/xfrm_device.c
new file mode 100644
index 0000000..8ec8a3f
--- /dev/null
+++ b/net/xfrm/xfrm_device.c
@@ -0,0 +1,208 @@
+/*
+ * xfrm_device.c - IPsec device offloading code.
+ *
+ * Copyright (c) 2015 secunet Security Networks AG
+ *
+ * Author:
+ * Steffen Klassert <steffen.klassert@secunet.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#include <linux/errno.h>
+#include <linux/module.h>
+#include <linux/netdevice.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <net/dst.h>
+#include <net/xfrm.h>
+#include <linux/notifier.h>
+
+int validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t features)
+{
+ int err;
+ struct xfrm_state *x;
+ struct xfrm_offload *xo = xfrm_offload(skb);
+
+ if (skb_is_gso(skb))
+ return 0;
+
+ if (xo) {
+ x = skb->sp->xvec[skb->sp->len - 1];
+ if (xo->flags & XFRM_GRO || x->xso.flags & XFRM_OFFLOAD_INBOUND)
+ return 0;
+
+ x->outer_mode->xmit(x, skb);
+
+ err = x->type_offload->xmit(x, skb, features);
+ if (err) {
+ XFRM_INC_STATS(xs_net(x), LINUX_MIB_XFRMOUTSTATEPROTOERROR);
+ return err;
+ }
+
+ skb_push(skb, skb->data - skb_mac_header(skb));
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(validate_xmit_xfrm);
+
+int xfrm_dev_state_add(struct net *net, struct xfrm_state *x,
+ struct xfrm_user_offload *xuo)
+{
+ int err;
+ struct dst_entry *dst;
+ struct net_device *dev;
+ struct xfrm_state_offload *xso = &x->xso;
+ xfrm_address_t *saddr;
+ xfrm_address_t *daddr;
+
+ if (!x->type_offload)
+ return 0;
+
+ /* We don't yet support UDP encapsulation, TFC padding and ESN. */
+ if (x->encap || x->tfcpad || (x->props.flags & XFRM_STATE_ESN))
+ return 0;
+
+ dev = dev_get_by_index(net, xuo->ifindex);
+ if (!dev) {
+ if (!(xuo->flags & XFRM_OFFLOAD_INBOUND)) {
+ saddr = &x->props.saddr;
+ daddr = &x->id.daddr;
+ } else {
+ saddr = &x->id.daddr;
+ daddr = &x->props.saddr;
+ }
+
+ dst = __xfrm_dst_lookup(net, 0, 0, saddr, daddr, x->props.family);
+ if (IS_ERR(dst))
+ return 0;
+
+ dev = dst->dev;
+
+ dev_hold(dev);
+ dst_release(dst);
+ }
+
+ if (!dev->xfrmdev_ops || !dev->xfrmdev_ops->xdo_dev_state_add) {
+ dev_put(dev);
+ return 0;
+ }
+
+ xso->dev = dev;
+ xso->num_exthdrs = 1;
+ xso->flags = xuo->flags;
+
+ err = dev->xfrmdev_ops->xdo_dev_state_add(x);
+ if (err) {
+ dev_put(dev);
+ return err;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xfrm_dev_state_add);
+
+bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x)
+{
+ int mtu;
+ struct dst_entry *dst = skb_dst(skb);
+ struct xfrm_dst *xdst = (struct xfrm_dst *)dst;
+ struct net_device *dev = x->xso.dev;
+
+ if (!x->type_offload || x->encap)
+ return false;
+
+ if ((x->xso.offload_handle && (dev == dst->path->dev)) &&
+ !dst->child->xfrm && x->type->get_mtu) {
+ mtu = x->type->get_mtu(x, xdst->child_mtu_cached);
+
+ if (skb->len <= mtu)
+ goto ok;
+
+ if (skb_is_gso(skb) && skb_gso_validate_mtu(skb, mtu))
+ goto ok;
+ }
+
+ return false;
+
+ok:
+ if (dev && dev->xfrmdev_ops && dev->xfrmdev_ops->xdo_dev_offload_ok)
+ return x->xso.dev->xfrmdev_ops->xdo_dev_offload_ok(skb, x);
+
+ return true;
+}
+EXPORT_SYMBOL_GPL(xfrm_dev_offload_ok);
+
+int xfrm_dev_register(struct net_device *dev)
+{
+ if ((dev->features & NETIF_F_HW_ESP) && !dev->xfrmdev_ops)
+ return NOTIFY_BAD;
+ if ((dev->features & NETIF_F_HW_ESP_TX_CSUM) &&
+ !(dev->features & NETIF_F_HW_ESP))
+ return NOTIFY_BAD;
+
+ return NOTIFY_DONE;
+}
+
+static int xfrm_dev_unregister(struct net_device *dev)
+{
+ return NOTIFY_DONE;
+}
+
+static int xfrm_dev_feat_change(struct net_device *dev)
+{
+ if ((dev->features & NETIF_F_HW_ESP) && !dev->xfrmdev_ops)
+ return NOTIFY_BAD;
+ else if (!(dev->features & NETIF_F_HW_ESP))
+ dev->xfrmdev_ops = NULL;
+
+ if ((dev->features & NETIF_F_HW_ESP_TX_CSUM) &&
+ !(dev->features & NETIF_F_HW_ESP))
+ return NOTIFY_BAD;
+
+ return NOTIFY_DONE;
+}
+
+static int xfrm_dev_down(struct net_device *dev)
+{
+ if (dev->hw_features & NETIF_F_HW_ESP)
+ xfrm_dev_state_flush(dev_net(dev), dev, true);
+
+ xfrm_garbage_collect(dev_net(dev));
+
+ return NOTIFY_DONE;
+}
+
+static int xfrm_dev_event(struct notifier_block *this, unsigned long event, void *ptr)
+{
+ struct net_device *dev = netdev_notifier_info_to_dev(ptr);
+
+ switch (event) {
+ case NETDEV_REGISTER:
+ return xfrm_dev_register(dev);
+
+ case NETDEV_UNREGISTER:
+ return xfrm_dev_unregister(dev);
+
+ case NETDEV_FEAT_CHANGE:
+ return xfrm_dev_feat_change(dev);
+
+ case NETDEV_DOWN:
+ return xfrm_dev_down(dev);
+ }
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block xfrm_dev_notifier = {
+ .notifier_call = xfrm_dev_event,
+};
+
+void __net_init xfrm_dev_init(void)
+{
+ register_netdevice_notifier(&xfrm_dev_notifier);
+}
diff --git a/net/xfrm/xfrm_hash.h b/net/xfrm/xfrm_hash.h
index 666c5ff..eaea9c4 100644
--- a/net/xfrm/xfrm_hash.h
+++ b/net/xfrm/xfrm_hash.h
@@ -54,8 +54,8 @@
static inline unsigned int __xfrm6_pref_hash(const xfrm_address_t *addr,
__u8 prefixlen)
{
- int pdw;
- int pbi;
+ unsigned int pdw;
+ unsigned int pbi;
u32 initval = 0;
pdw = prefixlen >> 5; /* num of whole u32 in prefix */
diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c
index 46bdb4f..9de4b1d 100644
--- a/net/xfrm/xfrm_input.c
+++ b/net/xfrm/xfrm_input.c
@@ -107,6 +107,8 @@
sp->len = 0;
sp->olen = 0;
+ memset(sp->ovec, 0, sizeof(sp->ovec[XFRM_MAX_OFFLOAD_DEPTH]));
+
if (src) {
int i;
@@ -207,8 +209,9 @@
unsigned int family;
int decaps = 0;
int async = 0;
- struct xfrm_offload *xo;
bool xfrm_gro = false;
+ bool crypto_done = false;
+ struct xfrm_offload *xo = xfrm_offload(skb);
if (encap_type < 0) {
x = xfrm_input_state(skb);
@@ -220,9 +223,40 @@
seq = XFRM_SKB_CB(skb)->seq.input.low;
goto resume;
}
+
/* encap_type < -1 indicates a GRO call. */
encap_type = 0;
seq = XFRM_SPI_SKB_CB(skb)->seq;
+
+ if (xo && (xo->flags & CRYPTO_DONE)) {
+ crypto_done = true;
+ x = xfrm_input_state(skb);
+ family = XFRM_SPI_SKB_CB(skb)->family;
+
+ if (!(xo->status & CRYPTO_SUCCESS)) {
+ if (xo->status &
+ (CRYPTO_TRANSPORT_AH_AUTH_FAILED |
+ CRYPTO_TRANSPORT_ESP_AUTH_FAILED |
+ CRYPTO_TUNNEL_AH_AUTH_FAILED |
+ CRYPTO_TUNNEL_ESP_AUTH_FAILED)) {
+
+ xfrm_audit_state_icvfail(x, skb,
+ x->type->proto);
+ x->stats.integrity_failed++;
+ XFRM_INC_STATS(net, LINUX_MIB_XFRMINSTATEPROTOERROR);
+ goto drop;
+ }
+
+ XFRM_INC_STATS(net, LINUX_MIB_XFRMINBUFFERERROR);
+ goto drop;
+ }
+
+ if ((err = xfrm_parse_spi(skb, nexthdr, &spi, &seq)) != 0) {
+ XFRM_INC_STATS(net, LINUX_MIB_XFRMINHDRERROR);
+ goto drop;
+ }
+ }
+
goto lock;
}
@@ -311,7 +345,10 @@
skb_dst_force(skb);
dev_hold(skb->dev);
- nexthdr = x->type->input(x, skb);
+ if (crypto_done)
+ nexthdr = x->type_offload->input_tail(x, skb);
+ else
+ nexthdr = x->type->input(x, skb);
if (nexthdr == -EINPROGRESS)
return 0;
@@ -395,7 +432,7 @@
if (xo)
xfrm_gro = xo->flags & XFRM_GRO;
- err = x->inner_mode->afinfo->transport_finish(skb, async);
+ err = x->inner_mode->afinfo->transport_finish(skb, xfrm_gro || async);
if (xfrm_gro) {
skb_dst_drop(skb);
gro_cells_receive(&gro_cells, skb);
diff --git a/net/xfrm/xfrm_output.c b/net/xfrm/xfrm_output.c
index 8ba29fe..8c0b672 100644
--- a/net/xfrm/xfrm_output.c
+++ b/net/xfrm/xfrm_output.c
@@ -99,12 +99,13 @@
skb_dst_force(skb);
- /* Inner headers are invalid now. */
- skb->encapsulation = 0;
-
- err = x->type->output(x, skb);
- if (err == -EINPROGRESS)
- goto out;
+ if (xfrm_offload(skb)) {
+ x->type_offload->encap(x, skb);
+ } else {
+ err = x->type->output(x, skb);
+ if (err == -EINPROGRESS)
+ goto out;
+ }
resume:
if (err) {
@@ -200,8 +201,40 @@
int xfrm_output(struct sock *sk, struct sk_buff *skb)
{
struct net *net = dev_net(skb_dst(skb)->dev);
+ struct xfrm_state *x = skb_dst(skb)->xfrm;
int err;
+ secpath_reset(skb);
+ skb->encapsulation = 0;
+
+ if (xfrm_dev_offload_ok(skb, x)) {
+ struct sec_path *sp;
+
+ sp = secpath_dup(skb->sp);
+ if (!sp) {
+ XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTERROR);
+ kfree_skb(skb);
+ return -ENOMEM;
+ }
+ if (skb->sp)
+ secpath_put(skb->sp);
+ skb->sp = sp;
+ skb->encapsulation = 1;
+
+ sp->olen++;
+ sp->xvec[skb->sp->len++] = x;
+ xfrm_state_hold(x);
+
+ if (skb_is_gso(skb)) {
+ skb_shinfo(skb)->gso_type |= SKB_GSO_ESP;
+
+ return xfrm_output2(net, sk, skb);
+ }
+
+ if (x->xso.dev && x->xso.dev->features & NETIF_F_HW_ESP_TX_CSUM)
+ goto out;
+ }
+
if (skb_is_gso(skb))
return xfrm_output_gso(net, sk, skb);
@@ -214,6 +247,7 @@
}
}
+out:
return xfrm_output2(net, sk, skb);
}
EXPORT_SYMBOL_GPL(xfrm_output);
diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
index 236cbbc..b00a1d5 100644
--- a/net/xfrm/xfrm_policy.c
+++ b/net/xfrm/xfrm_policy.c
@@ -116,11 +116,10 @@
return afinfo;
}
-static inline struct dst_entry *__xfrm_dst_lookup(struct net *net,
- int tos, int oif,
- const xfrm_address_t *saddr,
- const xfrm_address_t *daddr,
- int family)
+struct dst_entry *__xfrm_dst_lookup(struct net *net, int tos, int oif,
+ const xfrm_address_t *saddr,
+ const xfrm_address_t *daddr,
+ int family)
{
const struct xfrm_policy_afinfo *afinfo;
struct dst_entry *dst;
@@ -135,6 +134,7 @@
return dst;
}
+EXPORT_SYMBOL(__xfrm_dst_lookup);
static inline struct dst_entry *xfrm_dst_lookup(struct xfrm_state *x,
int tos, int oif,
@@ -1006,6 +1006,10 @@
err = -ESRCH;
out:
spin_unlock_bh(&net->xfrm.xfrm_policy_lock);
+
+ if (cnt)
+ xfrm_garbage_collect(net);
+
return err;
}
EXPORT_SYMBOL(xfrm_policy_flush);
@@ -2929,21 +2933,6 @@
}
EXPORT_SYMBOL(xfrm_policy_unregister_afinfo);
-static int xfrm_dev_event(struct notifier_block *this, unsigned long event, void *ptr)
-{
- struct net_device *dev = netdev_notifier_info_to_dev(ptr);
-
- switch (event) {
- case NETDEV_DOWN:
- xfrm_garbage_collect(dev_net(dev));
- }
- return NOTIFY_DONE;
-}
-
-static struct notifier_block xfrm_dev_notifier = {
- .notifier_call = xfrm_dev_event,
-};
-
#ifdef CONFIG_XFRM_STATISTICS
static int __net_init xfrm_statistics_init(struct net *net)
{
@@ -3020,7 +3009,7 @@
INIT_WORK(&net->xfrm.policy_hash_work, xfrm_hash_resize);
INIT_WORK(&net->xfrm.policy_hthresh.work, xfrm_hash_rebuild);
if (net_eq(net, &init_net))
- register_netdevice_notifier(&xfrm_dev_notifier);
+ xfrm_dev_init();
return 0;
out_bydst:
diff --git a/net/xfrm/xfrm_replay.c b/net/xfrm/xfrm_replay.c
index cdc2e2e..8b23c5b 100644
--- a/net/xfrm/xfrm_replay.c
+++ b/net/xfrm/xfrm_replay.c
@@ -45,7 +45,8 @@
return seq_hi;
}
-
+EXPORT_SYMBOL(xfrm_replay_seqhi);
+;
static void xfrm_replay_notify(struct xfrm_state *x, int event)
{
struct km_event c;
@@ -558,6 +559,158 @@
x->repl->notify(x, XFRM_REPLAY_UPDATE);
}
+#ifdef CONFIG_XFRM_OFFLOAD
+static int xfrm_replay_overflow_offload(struct xfrm_state *x, struct sk_buff *skb)
+{
+ int err = 0;
+ struct net *net = xs_net(x);
+ struct xfrm_offload *xo = xfrm_offload(skb);
+ __u32 oseq = x->replay.oseq;
+
+ if (!xo)
+ return xfrm_replay_overflow(x, skb);
+
+ if (x->type->flags & XFRM_TYPE_REPLAY_PROT) {
+ if (!skb_is_gso(skb)) {
+ XFRM_SKB_CB(skb)->seq.output.low = ++oseq;
+ xo->seq.low = oseq;
+ } else {
+ XFRM_SKB_CB(skb)->seq.output.low = oseq + 1;
+ xo->seq.low = oseq + 1;
+ oseq += skb_shinfo(skb)->gso_segs;
+ }
+
+ XFRM_SKB_CB(skb)->seq.output.hi = 0;
+ xo->seq.hi = 0;
+ if (unlikely(oseq < x->replay.oseq)) {
+ xfrm_audit_state_replay_overflow(x, skb);
+ err = -EOVERFLOW;
+
+ return err;
+ }
+
+ x->replay.oseq = oseq;
+
+ if (xfrm_aevent_is_on(net))
+ x->repl->notify(x, XFRM_REPLAY_UPDATE);
+ }
+
+ return err;
+}
+
+static int xfrm_replay_overflow_offload_bmp(struct xfrm_state *x, struct sk_buff *skb)
+{
+ int err = 0;
+ struct xfrm_offload *xo = xfrm_offload(skb);
+ struct xfrm_replay_state_esn *replay_esn = x->replay_esn;
+ struct net *net = xs_net(x);
+ __u32 oseq = replay_esn->oseq;
+
+ if (!xo)
+ return xfrm_replay_overflow_bmp(x, skb);
+
+ if (x->type->flags & XFRM_TYPE_REPLAY_PROT) {
+ if (!skb_is_gso(skb)) {
+ XFRM_SKB_CB(skb)->seq.output.low = ++oseq;
+ xo->seq.low = oseq;
+ } else {
+ XFRM_SKB_CB(skb)->seq.output.low = oseq + 1;
+ xo->seq.low = oseq + 1;
+ oseq += skb_shinfo(skb)->gso_segs;
+ }
+
+ XFRM_SKB_CB(skb)->seq.output.hi = 0;
+ xo->seq.hi = 0;
+ if (unlikely(oseq < replay_esn->oseq)) {
+ xfrm_audit_state_replay_overflow(x, skb);
+ err = -EOVERFLOW;
+
+ return err;
+ } else {
+ replay_esn->oseq = oseq;
+ }
+
+ if (xfrm_aevent_is_on(net))
+ x->repl->notify(x, XFRM_REPLAY_UPDATE);
+ }
+
+ return err;
+}
+
+static int xfrm_replay_overflow_offload_esn(struct xfrm_state *x, struct sk_buff *skb)
+{
+ int err = 0;
+ struct xfrm_offload *xo = xfrm_offload(skb);
+ struct xfrm_replay_state_esn *replay_esn = x->replay_esn;
+ struct net *net = xs_net(x);
+ __u32 oseq = replay_esn->oseq;
+ __u32 oseq_hi = replay_esn->oseq_hi;
+
+ if (!xo)
+ return xfrm_replay_overflow_esn(x, skb);
+
+ if (x->type->flags & XFRM_TYPE_REPLAY_PROT) {
+ if (!skb_is_gso(skb)) {
+ XFRM_SKB_CB(skb)->seq.output.low = ++oseq;
+ XFRM_SKB_CB(skb)->seq.output.hi = oseq_hi;
+ xo->seq.low = oseq;
+ xo->seq.hi = oseq_hi;
+ } else {
+ XFRM_SKB_CB(skb)->seq.output.low = oseq + 1;
+ XFRM_SKB_CB(skb)->seq.output.hi = oseq_hi;
+ xo->seq.low = oseq = oseq + 1;
+ xo->seq.hi = oseq_hi;
+ oseq += skb_shinfo(skb)->gso_segs;
+ }
+
+ if (unlikely(oseq < replay_esn->oseq)) {
+ XFRM_SKB_CB(skb)->seq.output.hi = ++oseq_hi;
+ xo->seq.hi = oseq_hi;
+
+ if (replay_esn->oseq_hi == 0) {
+ replay_esn->oseq--;
+ replay_esn->oseq_hi--;
+ xfrm_audit_state_replay_overflow(x, skb);
+ err = -EOVERFLOW;
+
+ return err;
+ }
+ }
+
+ replay_esn->oseq = oseq;
+ replay_esn->oseq_hi = oseq_hi;
+
+ if (xfrm_aevent_is_on(net))
+ x->repl->notify(x, XFRM_REPLAY_UPDATE);
+ }
+
+ return err;
+}
+
+static const struct xfrm_replay xfrm_replay_legacy = {
+ .advance = xfrm_replay_advance,
+ .check = xfrm_replay_check,
+ .recheck = xfrm_replay_check,
+ .notify = xfrm_replay_notify,
+ .overflow = xfrm_replay_overflow_offload,
+};
+
+static const struct xfrm_replay xfrm_replay_bmp = {
+ .advance = xfrm_replay_advance_bmp,
+ .check = xfrm_replay_check_bmp,
+ .recheck = xfrm_replay_check_bmp,
+ .notify = xfrm_replay_notify_bmp,
+ .overflow = xfrm_replay_overflow_offload_bmp,
+};
+
+static const struct xfrm_replay xfrm_replay_esn = {
+ .advance = xfrm_replay_advance_esn,
+ .check = xfrm_replay_check_esn,
+ .recheck = xfrm_replay_recheck_esn,
+ .notify = xfrm_replay_notify_esn,
+ .overflow = xfrm_replay_overflow_offload_esn,
+};
+#else
static const struct xfrm_replay xfrm_replay_legacy = {
.advance = xfrm_replay_advance,
.check = xfrm_replay_check,
@@ -581,6 +734,7 @@
.notify = xfrm_replay_notify_esn,
.overflow = xfrm_replay_overflow_esn,
};
+#endif
int xfrm_init_replay(struct xfrm_state *x)
{
@@ -595,10 +749,12 @@
if (replay_esn->replay_window == 0)
return -EINVAL;
x->repl = &xfrm_replay_esn;
- } else
+ } else {
x->repl = &xfrm_replay_bmp;
- } else
+ }
+ } else {
x->repl = &xfrm_replay_legacy;
+ }
return 0;
}
diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c
index 5a597db..fc3c5aa 100644
--- a/net/xfrm/xfrm_state.c
+++ b/net/xfrm/xfrm_state.c
@@ -251,6 +251,75 @@
module_put(type->owner);
}
+static DEFINE_SPINLOCK(xfrm_type_offload_lock);
+int xfrm_register_type_offload(const struct xfrm_type_offload *type,
+ unsigned short family)
+{
+ struct xfrm_state_afinfo *afinfo = xfrm_state_get_afinfo(family);
+ const struct xfrm_type_offload **typemap;
+ int err = 0;
+
+ if (unlikely(afinfo == NULL))
+ return -EAFNOSUPPORT;
+ typemap = afinfo->type_offload_map;
+ spin_lock_bh(&xfrm_type_offload_lock);
+
+ if (likely(typemap[type->proto] == NULL))
+ typemap[type->proto] = type;
+ else
+ err = -EEXIST;
+ spin_unlock_bh(&xfrm_type_offload_lock);
+ rcu_read_unlock();
+ return err;
+}
+EXPORT_SYMBOL(xfrm_register_type_offload);
+
+int xfrm_unregister_type_offload(const struct xfrm_type_offload *type,
+ unsigned short family)
+{
+ struct xfrm_state_afinfo *afinfo = xfrm_state_get_afinfo(family);
+ const struct xfrm_type_offload **typemap;
+ int err = 0;
+
+ if (unlikely(afinfo == NULL))
+ return -EAFNOSUPPORT;
+ typemap = afinfo->type_offload_map;
+ spin_lock_bh(&xfrm_type_offload_lock);
+
+ if (unlikely(typemap[type->proto] != type))
+ err = -ENOENT;
+ else
+ typemap[type->proto] = NULL;
+ spin_unlock_bh(&xfrm_type_offload_lock);
+ rcu_read_unlock();
+ return err;
+}
+EXPORT_SYMBOL(xfrm_unregister_type_offload);
+
+static const struct xfrm_type_offload *xfrm_get_type_offload(u8 proto, unsigned short family)
+{
+ struct xfrm_state_afinfo *afinfo;
+ const struct xfrm_type_offload **typemap;
+ const struct xfrm_type_offload *type;
+
+ afinfo = xfrm_state_get_afinfo(family);
+ if (unlikely(afinfo == NULL))
+ return NULL;
+ typemap = afinfo->type_offload_map;
+
+ type = typemap[proto];
+ if ((type && !try_module_get(type->owner)))
+ type = NULL;
+
+ rcu_read_unlock();
+ return type;
+}
+
+static void xfrm_put_type_offload(const struct xfrm_type_offload *type)
+{
+ module_put(type->owner);
+}
+
static DEFINE_SPINLOCK(xfrm_mode_lock);
int xfrm_register_mode(struct xfrm_mode *mode, int family)
{
@@ -365,10 +434,13 @@
xfrm_put_mode(x->inner_mode_iaf);
if (x->outer_mode)
xfrm_put_mode(x->outer_mode);
+ if (x->type_offload)
+ xfrm_put_type_offload(x->type_offload);
if (x->type) {
x->type->destructor(x);
xfrm_put_type(x->type);
}
+ xfrm_dev_state_free(x);
security_xfrm_state_free(x);
kfree(x);
}
@@ -538,6 +610,8 @@
net->xfrm.state_num--;
spin_unlock(&net->xfrm.xfrm_state_lock);
+ xfrm_dev_state_delete(x);
+
/* All xfrm_state objects are created by xfrm_state_alloc.
* The xfrm_state_alloc call gives a reference, and that
* is what we are dropping here.
@@ -582,12 +656,41 @@
return err;
}
+
+static inline int
+xfrm_dev_state_flush_secctx_check(struct net *net, struct net_device *dev, bool task_valid)
+{
+ int i, err = 0;
+
+ for (i = 0; i <= net->xfrm.state_hmask; i++) {
+ struct xfrm_state *x;
+ struct xfrm_state_offload *xso;
+
+ hlist_for_each_entry(x, net->xfrm.state_bydst+i, bydst) {
+ xso = &x->xso;
+
+ if (xso->dev == dev &&
+ (err = security_xfrm_state_delete(x)) != 0) {
+ xfrm_audit_state_delete(x, 0, task_valid);
+ return err;
+ }
+ }
+ }
+
+ return err;
+}
#else
static inline int
xfrm_state_flush_secctx_check(struct net *net, u8 proto, bool task_valid)
{
return 0;
}
+
+static inline int
+xfrm_dev_state_flush_secctx_check(struct net *net, struct net_device *dev, bool task_valid)
+{
+ return 0;
+}
#endif
int xfrm_state_flush(struct net *net, u8 proto, bool task_valid)
@@ -630,6 +733,48 @@
}
EXPORT_SYMBOL(xfrm_state_flush);
+int xfrm_dev_state_flush(struct net *net, struct net_device *dev, bool task_valid)
+{
+ int i, err = 0, cnt = 0;
+
+ spin_lock_bh(&net->xfrm.xfrm_state_lock);
+ err = xfrm_dev_state_flush_secctx_check(net, dev, task_valid);
+ if (err)
+ goto out;
+
+ err = -ESRCH;
+ for (i = 0; i <= net->xfrm.state_hmask; i++) {
+ struct xfrm_state *x;
+ struct xfrm_state_offload *xso;
+restart:
+ hlist_for_each_entry(x, net->xfrm.state_bydst+i, bydst) {
+ xso = &x->xso;
+
+ if (!xfrm_state_kern(x) && xso->dev == dev) {
+ xfrm_state_hold(x);
+ spin_unlock_bh(&net->xfrm.xfrm_state_lock);
+
+ err = xfrm_state_delete(x);
+ xfrm_audit_state_delete(x, err ? 0 : 1,
+ task_valid);
+ xfrm_state_put(x);
+ if (!err)
+ cnt++;
+
+ spin_lock_bh(&net->xfrm.xfrm_state_lock);
+ goto restart;
+ }
+ }
+ }
+ if (cnt)
+ err = 0;
+
+out:
+ spin_unlock_bh(&net->xfrm.xfrm_state_lock);
+ return err;
+}
+EXPORT_SYMBOL(xfrm_dev_state_flush);
+
void xfrm_sad_getinfo(struct net *net, struct xfrmk_sadinfo *si)
{
spin_lock_bh(&net->xfrm.xfrm_state_lock);
@@ -2077,6 +2222,8 @@
if (x->type == NULL)
goto error;
+ x->type_offload = xfrm_get_type_offload(x->id.proto, family);
+
err = x->type->init_state(x);
if (err)
goto error;
diff --git a/net/xfrm/xfrm_user.c b/net/xfrm/xfrm_user.c
index 40a8aa3..38614df 100644
--- a/net/xfrm/xfrm_user.c
+++ b/net/xfrm/xfrm_user.c
@@ -55,7 +55,7 @@
return -EINVAL;
}
- algp->alg_name[CRYPTO_MAX_ALG_NAME - 1] = '\0';
+ algp->alg_name[sizeof(algp->alg_name) - 1] = '\0';
return 0;
}
@@ -71,7 +71,7 @@
if (nla_len(rt) < xfrm_alg_auth_len(algp))
return -EINVAL;
- algp->alg_name[CRYPTO_MAX_ALG_NAME - 1] = '\0';
+ algp->alg_name[sizeof(algp->alg_name) - 1] = '\0';
return 0;
}
@@ -87,7 +87,7 @@
if (nla_len(rt) < aead_len(algp))
return -EINVAL;
- algp->alg_name[CRYPTO_MAX_ALG_NAME - 1] = '\0';
+ algp->alg_name[sizeof(algp->alg_name) - 1] = '\0';
return 0;
}
@@ -595,6 +595,13 @@
goto error;
}
+ if (attrs[XFRMA_OFFLOAD_DEV]) {
+ err = xfrm_dev_state_add(net, x,
+ nla_data(attrs[XFRMA_OFFLOAD_DEV]));
+ if (err)
+ goto error;
+ }
+
if ((err = xfrm_alloc_replay_state_esn(&x->replay_esn, &x->preplay_esn,
attrs[XFRMA_REPLAY_ESN_VAL])))
goto error;
@@ -779,6 +786,23 @@
return 0;
}
+static int copy_user_offload(struct xfrm_state_offload *xso, struct sk_buff *skb)
+{
+ struct xfrm_user_offload *xuo;
+ struct nlattr *attr;
+
+ attr = nla_reserve(skb, XFRMA_OFFLOAD_DEV, sizeof(*xuo));
+ if (attr == NULL)
+ return -EMSGSIZE;
+
+ xuo = nla_data(attr);
+
+ xuo->ifindex = xso->dev->ifindex;
+ xuo->flags = xso->flags;
+
+ return 0;
+}
+
static int copy_to_user_auth(struct xfrm_algo_auth *auth, struct sk_buff *skb)
{
struct xfrm_algo *algo;
@@ -869,6 +893,10 @@
&x->replay);
if (ret)
goto out;
+ if(x->xso.dev)
+ ret = copy_user_offload(&x->xso, skb);
+ if (ret)
+ goto out;
if (x->security)
ret = copy_sec_ctx(x->security, skb);
out:
@@ -932,8 +960,8 @@
u8 proto = 0;
int err;
- err = nlmsg_parse(cb->nlh, 0, attrs, XFRMA_MAX,
- xfrma_policy);
+ err = nlmsg_parse(cb->nlh, 0, attrs, XFRMA_MAX, xfrma_policy,
+ NULL);
if (err < 0)
return err;
@@ -2406,6 +2434,7 @@
[XFRMA_SA_EXTRA_FLAGS] = { .type = NLA_U32 },
[XFRMA_PROTO] = { .type = NLA_U8 },
[XFRMA_ADDRESS_FILTER] = { .len = sizeof(struct xfrm_address_filter) },
+ [XFRMA_OFFLOAD_DEV] = { .len = sizeof(struct xfrm_user_offload) },
};
static const struct nla_policy xfrma_spd_policy[XFRMA_SPD_MAX+1] = {
@@ -2448,7 +2477,8 @@
[XFRM_MSG_GETSPDINFO - XFRM_MSG_BASE] = { .doit = xfrm_get_spdinfo },
};
-static int xfrm_user_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh)
+static int xfrm_user_rcv_msg(struct sk_buff *skb, struct nlmsghdr *nlh,
+ struct netlink_ext_ack *extack)
{
struct net *net = sock_net(skb->sk);
struct nlattr *attrs[XFRMA_MAX+1];
@@ -2488,7 +2518,7 @@
err = nlmsg_parse(nlh, xfrm_msg_min[type], attrs,
link->nla_max ? : XFRMA_MAX,
- link->nla_pol ? : xfrma_policy);
+ link->nla_pol ? : xfrma_policy, extack);
if (err < 0)
return err;
@@ -2622,6 +2652,8 @@
l += nla_total_size(sizeof(*x->coaddr));
if (x->props.extra_flags)
l += nla_total_size(sizeof(x->props.extra_flags));
+ if (x->xso.dev)
+ l += nla_total_size(sizeof(x->xso));
/* Must count x->lastused as it may become non-zero behind our back. */
l += nla_total_size_64bit(sizeof(u64));
@@ -3108,7 +3140,6 @@
}
static struct xfrm_mgr netlink_mgr = {
- .id = "netlink",
.notify = xfrm_send_state_notify,
.acquire = xfrm_send_acquire,
.compile_policy = xfrm_compile_policy,
diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile
index 09e9d53..6c7468e 100644
--- a/samples/bpf/Makefile
+++ b/samples/bpf/Makefile
@@ -34,6 +34,8 @@
hostprogs-y += tc_l2_redirect
hostprogs-y += lwt_len_hist
hostprogs-y += xdp_tx_iptunnel
+hostprogs-y += test_map_in_map
+hostprogs-y += per_socket_stats_example
# Libbpf dependencies
LIBBPF := ../../tools/lib/bpf/bpf.o
@@ -72,6 +74,8 @@
tc_l2_redirect-objs := bpf_load.o $(LIBBPF) tc_l2_redirect_user.o
lwt_len_hist-objs := bpf_load.o $(LIBBPF) lwt_len_hist_user.o
xdp_tx_iptunnel-objs := bpf_load.o $(LIBBPF) xdp_tx_iptunnel_user.o
+test_map_in_map-objs := bpf_load.o $(LIBBPF) test_map_in_map_user.o
+per_socket_stats_example-objs := $(LIBBPF) cookie_uid_helper_example.o
# Tell kbuild to always build the programs
always := $(hostprogs-y)
@@ -105,6 +109,8 @@
always += sampleip_kern.o
always += lwt_len_hist_kern.o
always += xdp_tx_iptunnel_kern.o
+always += test_map_in_map_kern.o
+always += cookie_uid_helper_example.o
HOSTCFLAGS += -I$(objtree)/usr/include
HOSTCFLAGS += -I$(srctree)/tools/lib/
@@ -139,6 +145,7 @@
HOSTLOADLIBES_tc_l2_redirect += -l elf
HOSTLOADLIBES_lwt_len_hist += -l elf
HOSTLOADLIBES_xdp_tx_iptunnel += -lelf
+HOSTLOADLIBES_test_map_in_map += -lelf
# Allows pointing LLC/CLANG to a LLVM backend with bpf support, redefine on cmdline:
# make samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang
@@ -182,4 +189,5 @@
-Wno-compare-distinct-pointer-types \
-Wno-gnu-variable-sized-type-not-at-end \
-Wno-address-of-packed-member -Wno-tautological-compare \
+ -Wno-unknown-warning-option \
-O2 -emit-llvm -c $< -o -| $(LLC) -march=bpf -filetype=obj -o $@
diff --git a/samples/bpf/bpf_helpers.h b/samples/bpf/bpf_helpers.h
index faaffe2..9a9c95f 100644
--- a/samples/bpf/bpf_helpers.h
+++ b/samples/bpf/bpf_helpers.h
@@ -80,6 +80,7 @@
unsigned int value_size;
unsigned int max_entries;
unsigned int map_flags;
+ unsigned int inner_map_idx;
};
static int (*bpf_skb_load_bytes)(void *ctx, int off, void *to, int len) =
@@ -145,11 +146,30 @@
#define PT_REGS_SP(x) ((x)->sp)
#define PT_REGS_IP(x) ((x)->nip)
+#elif defined(__sparc__)
+
+#define PT_REGS_PARM1(x) ((x)->u_regs[UREG_I0])
+#define PT_REGS_PARM2(x) ((x)->u_regs[UREG_I1])
+#define PT_REGS_PARM3(x) ((x)->u_regs[UREG_I2])
+#define PT_REGS_PARM4(x) ((x)->u_regs[UREG_I3])
+#define PT_REGS_PARM5(x) ((x)->u_regs[UREG_I4])
+#define PT_REGS_RET(x) ((x)->u_regs[UREG_I7])
+#define PT_REGS_RC(x) ((x)->u_regs[UREG_I0])
+#define PT_REGS_SP(x) ((x)->u_regs[UREG_FP])
+#if defined(__arch64__)
+#define PT_REGS_IP(x) ((x)->tpc)
+#else
+#define PT_REGS_IP(x) ((x)->pc)
+#endif
+
#endif
#ifdef __powerpc__
#define BPF_KPROBE_READ_RET_IP(ip, ctx) ({ (ip) = (ctx)->link; })
#define BPF_KRETPROBE_READ_RET_IP BPF_KPROBE_READ_RET_IP
+#elif defined(__sparc__)
+#define BPF_KPROBE_READ_RET_IP(ip, ctx) ({ (ip) = PT_REGS_RET(ctx); })
+#define BPF_KRETPROBE_READ_RET_IP BPF_KPROBE_READ_RET_IP
#else
#define BPF_KPROBE_READ_RET_IP(ip, ctx) ({ \
bpf_probe_read(&(ip), sizeof(ip), (void *)PT_REGS_RET(ctx)); })
diff --git a/samples/bpf/bpf_load.c b/samples/bpf/bpf_load.c
index b86ee54..74456b3 100644
--- a/samples/bpf/bpf_load.c
+++ b/samples/bpf/bpf_load.c
@@ -14,6 +14,7 @@
#include <linux/perf_event.h>
#include <linux/netlink.h>
#include <linux/rtnetlink.h>
+#include <linux/types.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/syscall.h>
@@ -21,6 +22,7 @@
#include <sys/mman.h>
#include <poll.h>
#include <ctype.h>
+#include <assert.h>
#include "libbpf.h"
#include "bpf_load.h"
#include "perf-sys.h"
@@ -37,13 +39,8 @@
int prog_cnt;
int prog_array_fd = -1;
-struct bpf_map_def {
- unsigned int type;
- unsigned int key_size;
- unsigned int value_size;
- unsigned int max_entries;
- unsigned int map_flags;
-};
+struct bpf_map_data map_data[MAX_MAPS];
+int map_data_count = 0;
static int populate_prog_array(const char *event, int prog_fd)
{
@@ -192,24 +189,45 @@
return 0;
}
-static int load_maps(struct bpf_map_def *maps, int len)
+static int load_maps(struct bpf_map_data *maps, int nr_maps,
+ fixup_map_cb fixup_map)
{
int i;
- for (i = 0; i < len / sizeof(struct bpf_map_def); i++) {
+ for (i = 0; i < nr_maps; i++) {
+ if (fixup_map) {
+ fixup_map(&maps[i], i);
+ /* Allow userspace to assign map FD prior to creation */
+ if (maps[i].fd != -1) {
+ map_fd[i] = maps[i].fd;
+ continue;
+ }
+ }
- map_fd[i] = bpf_create_map(maps[i].type,
- maps[i].key_size,
- maps[i].value_size,
- maps[i].max_entries,
- maps[i].map_flags);
+ if (maps[i].def.type == BPF_MAP_TYPE_ARRAY_OF_MAPS ||
+ maps[i].def.type == BPF_MAP_TYPE_HASH_OF_MAPS) {
+ int inner_map_fd = map_fd[maps[i].def.inner_map_idx];
+
+ map_fd[i] = bpf_create_map_in_map(maps[i].def.type,
+ maps[i].def.key_size,
+ inner_map_fd,
+ maps[i].def.max_entries,
+ maps[i].def.map_flags);
+ } else {
+ map_fd[i] = bpf_create_map(maps[i].def.type,
+ maps[i].def.key_size,
+ maps[i].def.value_size,
+ maps[i].def.max_entries,
+ maps[i].def.map_flags);
+ }
if (map_fd[i] < 0) {
printf("failed to create a map: %d %s\n",
errno, strerror(errno));
return 1;
}
+ maps[i].fd = map_fd[i];
- if (maps[i].type == BPF_MAP_TYPE_PROG_ARRAY)
+ if (maps[i].def.type == BPF_MAP_TYPE_PROG_ARRAY)
prog_array_fd = map_fd[i];
}
return 0;
@@ -239,7 +257,8 @@
}
static int parse_relo_and_apply(Elf_Data *data, Elf_Data *symbols,
- GElf_Shdr *shdr, struct bpf_insn *insn)
+ GElf_Shdr *shdr, struct bpf_insn *insn,
+ struct bpf_map_data *maps, int nr_maps)
{
int i, nrels;
@@ -249,6 +268,8 @@
GElf_Sym sym;
GElf_Rel rel;
unsigned int insn_idx;
+ bool match = false;
+ int j, map_idx;
gelf_getrel(data, i, &rel);
@@ -262,20 +283,157 @@
return 1;
}
insn[insn_idx].src_reg = BPF_PSEUDO_MAP_FD;
- insn[insn_idx].imm = map_fd[sym.st_value / sizeof(struct bpf_map_def)];
+
+ /* Match FD relocation against recorded map_data[] offset */
+ for (map_idx = 0; map_idx < nr_maps; map_idx++) {
+ if (maps[map_idx].elf_offset == sym.st_value) {
+ match = true;
+ break;
+ }
+ }
+ if (match) {
+ insn[insn_idx].imm = maps[map_idx].fd;
+ } else {
+ printf("invalid relo for insn[%d] no map_data match\n",
+ insn_idx);
+ return 1;
+ }
}
return 0;
}
-int load_bpf_file(char *path)
+static int cmp_symbols(const void *l, const void *r)
{
- int fd, i;
+ const GElf_Sym *lsym = (const GElf_Sym *)l;
+ const GElf_Sym *rsym = (const GElf_Sym *)r;
+
+ if (lsym->st_value < rsym->st_value)
+ return -1;
+ else if (lsym->st_value > rsym->st_value)
+ return 1;
+ else
+ return 0;
+}
+
+static int load_elf_maps_section(struct bpf_map_data *maps, int maps_shndx,
+ Elf *elf, Elf_Data *symbols, int strtabidx)
+{
+ int map_sz_elf, map_sz_copy;
+ bool validate_zero = false;
+ Elf_Data *data_maps;
+ int i, nr_maps;
+ GElf_Sym *sym;
+ Elf_Scn *scn;
+ int copy_sz;
+
+ if (maps_shndx < 0)
+ return -EINVAL;
+ if (!symbols)
+ return -EINVAL;
+
+ /* Get data for maps section via elf index */
+ scn = elf_getscn(elf, maps_shndx);
+ if (scn)
+ data_maps = elf_getdata(scn, NULL);
+ if (!scn || !data_maps) {
+ printf("Failed to get Elf_Data from maps section %d\n",
+ maps_shndx);
+ return -EINVAL;
+ }
+
+ /* For each map get corrosponding symbol table entry */
+ sym = calloc(MAX_MAPS+1, sizeof(GElf_Sym));
+ for (i = 0, nr_maps = 0; i < symbols->d_size / sizeof(GElf_Sym); i++) {
+ assert(nr_maps < MAX_MAPS+1);
+ if (!gelf_getsym(symbols, i, &sym[nr_maps]))
+ continue;
+ if (sym[nr_maps].st_shndx != maps_shndx)
+ continue;
+ /* Only increment iif maps section */
+ nr_maps++;
+ }
+
+ /* Align to map_fd[] order, via sort on offset in sym.st_value */
+ qsort(sym, nr_maps, sizeof(GElf_Sym), cmp_symbols);
+
+ /* Keeping compatible with ELF maps section changes
+ * ------------------------------------------------
+ * The program size of struct bpf_map_def is known by loader
+ * code, but struct stored in ELF file can be different.
+ *
+ * Unfortunately sym[i].st_size is zero. To calculate the
+ * struct size stored in the ELF file, assume all struct have
+ * the same size, and simply divide with number of map
+ * symbols.
+ */
+ map_sz_elf = data_maps->d_size / nr_maps;
+ map_sz_copy = sizeof(struct bpf_map_def);
+ if (map_sz_elf < map_sz_copy) {
+ /*
+ * Backward compat, loading older ELF file with
+ * smaller struct, keeping remaining bytes zero.
+ */
+ map_sz_copy = map_sz_elf;
+ } else if (map_sz_elf > map_sz_copy) {
+ /*
+ * Forward compat, loading newer ELF file with larger
+ * struct with unknown features. Assume zero means
+ * feature not used. Thus, validate rest of struct
+ * data is zero.
+ */
+ validate_zero = true;
+ }
+
+ /* Memcpy relevant part of ELF maps data to loader maps */
+ for (i = 0; i < nr_maps; i++) {
+ unsigned char *addr, *end;
+ struct bpf_map_def *def;
+ const char *map_name;
+ size_t offset;
+
+ map_name = elf_strptr(elf, strtabidx, sym[i].st_name);
+ maps[i].name = strdup(map_name);
+ if (!maps[i].name) {
+ printf("strdup(%s): %s(%d)\n", map_name,
+ strerror(errno), errno);
+ free(sym);
+ return -errno;
+ }
+
+ /* Symbol value is offset into ELF maps section data area */
+ offset = sym[i].st_value;
+ def = (struct bpf_map_def *)(data_maps->d_buf + offset);
+ maps[i].elf_offset = offset;
+ memset(&maps[i].def, 0, sizeof(struct bpf_map_def));
+ memcpy(&maps[i].def, def, map_sz_copy);
+
+ /* Verify no newer features were requested */
+ if (validate_zero) {
+ addr = (unsigned char*) def + map_sz_copy;
+ end = (unsigned char*) def + map_sz_elf;
+ for (; addr < end; addr++) {
+ if (*addr != 0) {
+ free(sym);
+ return -EFBIG;
+ }
+ }
+ }
+ }
+
+ free(sym);
+ return nr_maps;
+}
+
+static int do_load_bpf_file(const char *path, fixup_map_cb fixup_map)
+{
+ int fd, i, ret, maps_shndx = -1, strtabidx = -1;
Elf *elf;
GElf_Ehdr ehdr;
GElf_Shdr shdr, shdr_prog;
- Elf_Data *data, *data_prog, *symbols = NULL;
+ Elf_Data *data, *data_prog, *data_maps = NULL, *symbols = NULL;
char *shname, *shname_prog;
+ int nr_maps = 0;
/* reset global variables */
kern_version = 0;
@@ -323,14 +481,41 @@
}
memcpy(&kern_version, data->d_buf, sizeof(int));
} else if (strcmp(shname, "maps") == 0) {
- processed_sec[i] = true;
- if (load_maps(data->d_buf, data->d_size))
- return 1;
+ int j;
+
+ maps_shndx = i;
+ data_maps = data;
+ for (j = 0; j < MAX_MAPS; j++)
+ map_data[j].fd = -1;
} else if (shdr.sh_type == SHT_SYMTAB) {
+ strtabidx = shdr.sh_link;
symbols = data;
}
}
+ ret = 1;
+
+ if (!symbols) {
+ printf("missing SHT_SYMTAB section\n");
+ goto done;
+ }
+
+ if (data_maps) {
+ nr_maps = load_elf_maps_section(map_data, maps_shndx,
+ elf, symbols, strtabidx);
+ if (nr_maps < 0) {
+ printf("Error: Failed loading ELF maps (errno:%d):%s\n",
+ nr_maps, strerror(-nr_maps));
+ ret = 1;
+ goto done;
+ }
+ if (load_maps(map_data, nr_maps, fixup_map))
+ goto done;
+ map_data_count = nr_maps;
+
+ processed_sec[maps_shndx] = true;
+ }
+
/* load programs that need map fixup (relocations) */
for (i = 1; i < ehdr.e_shnum; i++) {
if (processed_sec[i])
@@ -354,7 +539,8 @@
processed_sec[shdr.sh_info] = true;
processed_sec[i] = true;
- if (parse_relo_and_apply(data, symbols, &shdr, insns))
+ if (parse_relo_and_apply(data, symbols, &shdr, insns,
+ map_data, nr_maps))
continue;
if (memcmp(shname_prog, "kprobe/", 7) == 0 ||
@@ -387,8 +573,20 @@
load_and_attach(shname, data->d_buf, data->d_size);
}
+ ret = 0;
+done:
close(fd);
- return 0;
+ return ret;
+}
+
+int load_bpf_file(char *path)
+{
+ return do_load_bpf_file(path, NULL);
+}
+
+int load_bpf_file_fixup_map(const char *path, fixup_map_cb fixup_map)
+{
+ return do_load_bpf_file(path, fixup_map);
}
void read_trace_pipe(void)
@@ -473,7 +671,7 @@
return &syms[0];
}
-int set_link_xdp_fd(int ifindex, int fd)
+int set_link_xdp_fd(int ifindex, int fd, __u32 flags)
{
struct sockaddr_nl sa;
int sock, seq = 0, len, ret = -1;
@@ -509,15 +707,28 @@
req.nh.nlmsg_seq = ++seq;
req.ifinfo.ifi_family = AF_UNSPEC;
req.ifinfo.ifi_index = ifindex;
+
+ /* started nested attribute for XDP */
nla = (struct nlattr *)(((char *)&req)
+ NLMSG_ALIGN(req.nh.nlmsg_len));
nla->nla_type = NLA_F_NESTED | 43/*IFLA_XDP*/;
+ nla->nla_len = NLA_HDRLEN;
- nla_xdp = (struct nlattr *)((char *)nla + NLA_HDRLEN);
+ /* add XDP fd */
+ nla_xdp = (struct nlattr *)((char *)nla + nla->nla_len);
nla_xdp->nla_type = 1/*IFLA_XDP_FD*/;
nla_xdp->nla_len = NLA_HDRLEN + sizeof(int);
memcpy((char *)nla_xdp + NLA_HDRLEN, &fd, sizeof(fd));
- nla->nla_len = NLA_HDRLEN + nla_xdp->nla_len;
+ nla->nla_len += nla_xdp->nla_len;
+
+ /* if user passed in any flags, add those too */
+ if (flags) {
+ nla_xdp = (struct nlattr *)((char *)nla + nla->nla_len);
+ nla_xdp->nla_type = 3/*IFLA_XDP_FLAGS*/;
+ nla_xdp->nla_len = NLA_HDRLEN + sizeof(flags);
+ memcpy((char *)nla_xdp + NLA_HDRLEN, &flags, sizeof(flags));
+ nla->nla_len += nla_xdp->nla_len;
+ }
req.nh.nlmsg_len += NLA_ALIGN(nla->nla_len);
diff --git a/samples/bpf/bpf_load.h b/samples/bpf/bpf_load.h
index c827827..ca0563d 100644
--- a/samples/bpf/bpf_load.h
+++ b/samples/bpf/bpf_load.h
@@ -6,12 +6,36 @@
#define MAX_MAPS 32
#define MAX_PROGS 32
-extern int map_fd[MAX_MAPS];
+struct bpf_map_def {
+ unsigned int type;
+ unsigned int key_size;
+ unsigned int value_size;
+ unsigned int max_entries;
+ unsigned int map_flags;
+ unsigned int inner_map_idx;
+};
+
+struct bpf_map_data {
+ int fd;
+ char *name;
+ size_t elf_offset;
+ struct bpf_map_def def;
+};
+
+typedef void (*fixup_map_cb)(struct bpf_map_data *map, int idx);
+
extern int prog_fd[MAX_PROGS];
extern int event_fd[MAX_PROGS];
extern char bpf_log_buf[BPF_LOG_BUF_SIZE];
extern int prog_cnt;
+/* There is a one-to-one mapping between map_fd[] and map_data[].
+ * The map_data[] just contains more rich info on the given map.
+ */
+extern int map_fd[MAX_MAPS];
+extern struct bpf_map_data map_data[MAX_MAPS];
+extern int map_data_count;
+
/* parses elf file compiled by llvm .c->.o
* . parses 'maps' section and creates maps via BPF syscall
* . parses 'license' section and passes it to syscall
@@ -25,6 +49,7 @@
* returns zero on success
*/
int load_bpf_file(char *path);
+int load_bpf_file_fixup_map(const char *path, fixup_map_cb fixup_map);
void read_trace_pipe(void);
struct ksym {
@@ -34,5 +59,5 @@
int load_kallsyms(void);
struct ksym *ksym_search(long key);
-int set_link_xdp_fd(int ifindex, int fd);
+int set_link_xdp_fd(int ifindex, int fd, __u32 flags);
#endif
diff --git a/samples/bpf/cookie_uid_helper_example.c b/samples/bpf/cookie_uid_helper_example.c
new file mode 100644
index 0000000..b08ab4e
--- /dev/null
+++ b/samples/bpf/cookie_uid_helper_example.c
@@ -0,0 +1,321 @@
+/* This test is a demo of using get_socket_uid and get_socket_cookie
+ * helper function to do per socket based network traffic monitoring.
+ * It requires iptables version higher then 1.6.1. to load pinned eBPF
+ * program into the xt_bpf match.
+ *
+ * TEST:
+ * ./run_cookie_uid_helper_example.sh -option
+ * option:
+ * -t: do traffic monitoring test, the program will continuously
+ * print out network traffic happens after program started A sample
+ * output is shown below:
+ *
+ * cookie: 877, uid: 0x3e8, Pakcet Count: 20, Bytes Count: 11058
+ * cookie: 132, uid: 0x0, Pakcet Count: 2, Bytes Count: 286
+ * cookie: 812, uid: 0x3e8, Pakcet Count: 3, Bytes Count: 1726
+ * cookie: 802, uid: 0x3e8, Pakcet Count: 2, Bytes Count: 104
+ * cookie: 877, uid: 0x3e8, Pakcet Count: 20, Bytes Count: 11058
+ * cookie: 831, uid: 0x3e8, Pakcet Count: 2, Bytes Count: 104
+ * cookie: 0, uid: 0x0, Pakcet Count: 6, Bytes Count: 712
+ * cookie: 880, uid: 0xfffe, Pakcet Count: 1, Bytes Count: 70
+ *
+ * -s: do getsockopt SO_COOKIE test, the program will set up a pair of
+ * UDP sockets and send packets between them. And read out the traffic data
+ * directly from the ebpf map based on the socket cookie.
+ *
+ * Clean up: if using shell script, the script file will delete the iptables
+ * rule and unmount the bpf program when exit. Else the iptables rule need
+ * to be deleted by hand, see run_cookie_uid_helper_example.sh for detail.
+ */
+
+#define _GNU_SOURCE
+
+#define offsetof(type, member) __builtin_offsetof(type, member)
+#define ARRAY_SIZE(x) (sizeof(x) / sizeof(*(x)))
+
+#include <arpa/inet.h>
+#include <errno.h>
+#include <error.h>
+#include <limits.h>
+#include <linux/bpf.h>
+#include <linux/if_ether.h>
+#include <net/if.h>
+#include <signal.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/socket.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+#include <bpf/bpf.h>
+#include "libbpf.h"
+
+#define PORT 8888
+
+struct stats {
+ uint32_t uid;
+ uint64_t packets;
+ uint64_t bytes;
+};
+
+static int map_fd, prog_fd;
+
+static bool test_finish;
+
+static void maps_create(void)
+{
+ map_fd = bpf_create_map(BPF_MAP_TYPE_HASH, sizeof(uint32_t),
+ sizeof(struct stats), 100, 0);
+ if (map_fd < 0)
+ error(1, errno, "map create failed!\n");
+}
+
+static void prog_load(void)
+{
+ static char log_buf[1 << 16];
+
+ struct bpf_insn prog[] = {
+ /*
+ * Save sk_buff for future usage. value stored in R6 to R10 will
+ * not be reset after a bpf helper function call.
+ */
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ /*
+ * pc1: BPF_FUNC_get_socket_cookie takes one parameter,
+ * R1: sk_buff
+ */
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_get_socket_cookie),
+ /* pc2-4: save &socketCookie to r7 for future usage*/
+ BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -8),
+ BPF_MOV64_REG(BPF_REG_7, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -8),
+ /*
+ * pc5-8: set up the registers for BPF_FUNC_map_lookup_elem,
+ * it takes two parameters (R1: map_fd, R2: &socket_cookie)
+ */
+ BPF_LD_MAP_FD(BPF_REG_1, map_fd),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_7),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ /*
+ * pc9. if r0 != 0x0, go to pc+14, since we have the cookie
+ * stored already
+ * Otherwise do pc10-22 to setup a new data entry.
+ */
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 14),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_get_socket_uid),
+ /*
+ * Place a struct stats in the R10 stack and sequentially
+ * place the member value into the memory. Packets value
+ * is set by directly place a IMM value 1 into the stack.
+ */
+ BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0,
+ -32 + (__s16)offsetof(struct stats, uid)),
+ BPF_ST_MEM(BPF_DW, BPF_REG_10,
+ -32 + (__s16)offsetof(struct stats, packets), 1),
+ /*
+ * __sk_buff is a special struct used for eBPF program to
+ * directly access some sk_buff field.
+ */
+ BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_6,
+ offsetof(struct __sk_buff, len)),
+ BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_1,
+ -32 + (__s16)offsetof(struct stats, bytes)),
+ /*
+ * add new map entry using BPF_FUNC_map_update_elem, it takes
+ * 4 parameters (R1: map_fd, R2: &socket_cookie, R3: &stats,
+ * R4: flags)
+ */
+ BPF_LD_MAP_FD(BPF_REG_1, map_fd),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_7),
+ BPF_MOV64_REG(BPF_REG_3, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, -32),
+ BPF_MOV64_IMM(BPF_REG_4, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_update_elem),
+ BPF_JMP_IMM(BPF_JA, 0, 0, 5),
+ /*
+ * pc24-30 update the packet info to a exist data entry, it can
+ * be done by directly write to pointers instead of using
+ * BPF_FUNC_map_update_elem helper function
+ */
+ BPF_MOV64_REG(BPF_REG_9, BPF_REG_0),
+ BPF_MOV64_IMM(BPF_REG_1, 1),
+ BPF_STX_XADD(BPF_DW, BPF_REG_9, BPF_REG_1,
+ offsetof(struct stats, packets)),
+ BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_6,
+ offsetof(struct __sk_buff, len)),
+ BPF_STX_XADD(BPF_DW, BPF_REG_9, BPF_REG_1,
+ offsetof(struct stats, bytes)),
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6,
+ offsetof(struct __sk_buff, len)),
+ BPF_EXIT_INSN(),
+ };
+ prog_fd = bpf_load_program(BPF_PROG_TYPE_SOCKET_FILTER, prog,
+ ARRAY_SIZE(prog), "GPL", 0,
+ log_buf, sizeof(log_buf));
+ if (prog_fd < 0)
+ error(1, errno, "failed to load prog\n%s\n", log_buf);
+}
+
+static void prog_attach_iptables(char *file)
+{
+ int ret;
+ char rules[100];
+
+ if (bpf_obj_pin(prog_fd, file))
+ error(1, errno, "bpf_obj_pin");
+ if (strlen(file) > 50) {
+ printf("file path too long: %s\n", file);
+ exit(1);
+ }
+ sprintf(rules, "iptables -A OUTPUT -m bpf --object-pinned %s -j ACCEPT",
+ file);
+ ret = system(rules);
+ if (ret < 0) {
+ printf("iptables rule update failed: %d/n", WEXITSTATUS(ret));
+ exit(1);
+ }
+}
+
+static void print_table(void)
+{
+ struct stats curEntry;
+ uint32_t curN = UINT32_MAX;
+ uint32_t nextN;
+ int res;
+
+ while (bpf_map_get_next_key(map_fd, &curN, &nextN) > -1) {
+ curN = nextN;
+ res = bpf_map_lookup_elem(map_fd, &curN, &curEntry);
+ if (res < 0) {
+ error(1, errno, "fail to get entry value of Key: %u\n",
+ curN);
+ } else {
+ printf("cookie: %u, uid: 0x%x, Packet Count: %lu,"
+ " Bytes Count: %lu\n", curN, curEntry.uid,
+ curEntry.packets, curEntry.bytes);
+ }
+ }
+}
+
+static void udp_client(void)
+{
+ struct sockaddr_in si_other = {0};
+ struct sockaddr_in si_me = {0};
+ struct stats dataEntry;
+ int s_rcv, s_send, i, recv_len;
+ char message = 'a';
+ char buf;
+ uint64_t cookie;
+ int res;
+ socklen_t cookie_len = sizeof(cookie);
+ socklen_t slen = sizeof(si_other);
+
+ s_rcv = socket(PF_INET, SOCK_DGRAM, 0);
+ if (s_rcv < 0)
+ error(1, errno, "rcv socket creat failed!\n");
+ si_other.sin_family = AF_INET;
+ si_other.sin_port = htons(PORT);
+ if (inet_aton("127.0.0.1", &si_other.sin_addr) == 0)
+ error(1, errno, "inet_aton\n");
+ if (bind(s_rcv, (struct sockaddr *)&si_other, sizeof(si_other)) == -1)
+ error(1, errno, "bind\n");
+ s_send = socket(PF_INET, SOCK_DGRAM, 0);
+ if (s_send < 0)
+ error(1, errno, "send socket creat failed!\n");
+ res = getsockopt(s_send, SOL_SOCKET, SO_COOKIE, &cookie, &cookie_len);
+ if (res < 0)
+ printf("get cookie failed: %s\n", strerror(errno));
+ res = bpf_map_lookup_elem(map_fd, &cookie, &dataEntry);
+ if (res != -1)
+ error(1, errno, "socket stat found while flow not active\n");
+ for (i = 0; i < 10; i++) {
+ res = sendto(s_send, &message, sizeof(message), 0,
+ (struct sockaddr *)&si_other, slen);
+ if (res == -1)
+ error(1, errno, "send\n");
+ if (res != sizeof(message))
+ error(1, 0, "%uB != %luB\n", res, sizeof(message));
+ recv_len = recvfrom(s_rcv, &buf, sizeof(buf), 0,
+ (struct sockaddr *)&si_me, &slen);
+ if (recv_len < 0)
+ error(1, errno, "revieve\n");
+ res = memcmp(&(si_other.sin_addr), &(si_me.sin_addr),
+ sizeof(si_me.sin_addr));
+ if (res != 0)
+ error(1, EFAULT, "sender addr error: %d\n", res);
+ printf("Message received: %c\n", buf);
+ res = bpf_map_lookup_elem(map_fd, &cookie, &dataEntry);
+ if (res < 0)
+ error(1, errno, "lookup sk stat failed, cookie: %lu\n",
+ cookie);
+ printf("cookie: %lu, uid: 0x%x, Packet Count: %lu,"
+ " Bytes Count: %lu\n\n", cookie, dataEntry.uid,
+ dataEntry.packets, dataEntry.bytes);
+ }
+ close(s_send);
+ close(s_rcv);
+}
+
+static int usage(void)
+{
+ printf("Usage: ./run_cookie_uid_helper_example.sh"
+ " bpfObjName -option\n"
+ " -t traffic monitor test\n"
+ " -s getsockopt cookie test\n");
+ return 1;
+}
+
+static void finish(int ret)
+{
+ test_finish = true;
+}
+
+int main(int argc, char *argv[])
+{
+ int opt;
+ bool cfg_test_traffic = false;
+ bool cfg_test_cookie = false;
+
+ if (argc != 3)
+ return usage();
+ while ((opt = getopt(argc, argv, "ts")) != -1) {
+ switch (opt) {
+ case 't':
+ cfg_test_traffic = true;
+ break;
+ case 's':
+ cfg_test_cookie = true;
+ break;
+
+ default:
+ printf("unknown option %c\n", opt);
+ usage();
+ return -1;
+ }
+ }
+ maps_create();
+ prog_load();
+ prog_attach_iptables(argv[2]);
+ if (cfg_test_traffic) {
+ if (signal(SIGINT, finish) == SIG_ERR)
+ error(1, errno, "register handler failed");
+ while (!test_finish) {
+ print_table();
+ printf("\n");
+ sleep(1);
+ };
+ } else if (cfg_test_cookie) {
+ udp_client();
+ }
+ close(prog_fd);
+ close(map_fd);
+ return 0;
+}
diff --git a/samples/bpf/libbpf.h b/samples/bpf/libbpf.h
index 3705fba..8ab36a0 100644
--- a/samples/bpf/libbpf.h
+++ b/samples/bpf/libbpf.h
@@ -135,6 +135,16 @@
.off = OFF, \
.imm = 0 })
+/* Atomic memory add, *(uint *)(dst_reg + off16) += src_reg */
+
+#define BPF_STX_XADD(SIZE, DST, SRC, OFF) \
+ ((struct bpf_insn) { \
+ .code = BPF_STX | BPF_SIZE(SIZE) | BPF_XADD, \
+ .dst_reg = DST, \
+ .src_reg = SRC, \
+ .off = OFF, \
+ .imm = 0 })
+
/* Memory store, *(uint *) (dst_reg + off16) = imm32 */
#define BPF_ST_MEM(SIZE, DST, OFF, IMM) \
diff --git a/samples/bpf/map_perf_test_kern.c b/samples/bpf/map_perf_test_kern.c
index a91872a..2451658 100644
--- a/samples/bpf/map_perf_test_kern.c
+++ b/samples/bpf/map_perf_test_kern.c
@@ -11,6 +11,7 @@
#include "bpf_helpers.h"
#define MAX_ENTRIES 1000
+#define MAX_NR_CPUS 1024
struct bpf_map_def SEC("maps") hash_map = {
.type = BPF_MAP_TYPE_HASH,
@@ -26,7 +27,7 @@
.max_entries = 10000,
};
-struct bpf_map_def SEC("maps") percpu_lru_hash_map = {
+struct bpf_map_def SEC("maps") nocommon_lru_hash_map = {
.type = BPF_MAP_TYPE_LRU_HASH,
.key_size = sizeof(u32),
.value_size = sizeof(long),
@@ -34,6 +35,19 @@
.map_flags = BPF_F_NO_COMMON_LRU,
};
+struct bpf_map_def SEC("maps") inner_lru_hash_map = {
+ .type = BPF_MAP_TYPE_LRU_HASH,
+ .key_size = sizeof(u32),
+ .value_size = sizeof(long),
+ .max_entries = MAX_ENTRIES,
+};
+
+struct bpf_map_def SEC("maps") array_of_lru_hashs = {
+ .type = BPF_MAP_TYPE_ARRAY_OF_MAPS,
+ .key_size = sizeof(u32),
+ .max_entries = MAX_NR_CPUS,
+};
+
struct bpf_map_def SEC("maps") percpu_hash_map = {
.type = BPF_MAP_TYPE_PERCPU_HASH,
.key_size = sizeof(u32),
@@ -65,6 +79,13 @@
.map_flags = BPF_F_NO_PREALLOC,
};
+struct bpf_map_def SEC("maps") array_map = {
+ .type = BPF_MAP_TYPE_ARRAY,
+ .key_size = sizeof(u32),
+ .value_size = sizeof(long),
+ .max_entries = MAX_ENTRIES,
+};
+
SEC("kprobe/sys_getuid")
int stress_hmap(struct pt_regs *ctx)
{
@@ -93,6 +114,7 @@
bpf_map_delete_elem(&percpu_hash_map, &key);
return 0;
}
+
SEC("kprobe/sys_getgid")
int stress_hmap_alloc(struct pt_regs *ctx)
{
@@ -121,24 +143,56 @@
return 0;
}
-SEC("kprobe/sys_getpid")
+SEC("kprobe/sys_connect")
int stress_lru_hmap_alloc(struct pt_regs *ctx)
{
- u32 key = bpf_get_prandom_u32();
+ struct sockaddr_in6 *in6;
+ u16 test_case, dst6[8];
+ int addrlen, ret;
+ char fmt[] = "Failed at stress_lru_hmap_alloc. ret:%d\n";
long val = 1;
-
- bpf_map_update_elem(&lru_hash_map, &key, &val, BPF_ANY);
-
- return 0;
-}
-
-SEC("kprobe/sys_getppid")
-int stress_percpu_lru_hmap_alloc(struct pt_regs *ctx)
-{
u32 key = bpf_get_prandom_u32();
- long val = 1;
- bpf_map_update_elem(&percpu_lru_hash_map, &key, &val, BPF_ANY);
+ in6 = (struct sockaddr_in6 *)PT_REGS_PARM2(ctx);
+ addrlen = (int)PT_REGS_PARM3(ctx);
+
+ if (addrlen != sizeof(*in6))
+ return 0;
+
+ ret = bpf_probe_read(dst6, sizeof(dst6), &in6->sin6_addr);
+ if (ret)
+ goto done;
+
+ if (dst6[0] != 0xdead || dst6[1] != 0xbeef)
+ return 0;
+
+ test_case = dst6[7];
+
+ if (test_case == 0) {
+ ret = bpf_map_update_elem(&lru_hash_map, &key, &val, BPF_ANY);
+ } else if (test_case == 1) {
+ ret = bpf_map_update_elem(&nocommon_lru_hash_map, &key, &val,
+ BPF_ANY);
+ } else if (test_case == 2) {
+ void *nolocal_lru_map;
+ int cpu = bpf_get_smp_processor_id();
+
+ nolocal_lru_map = bpf_map_lookup_elem(&array_of_lru_hashs,
+ &cpu);
+ if (!nolocal_lru_map) {
+ ret = -ENOENT;
+ goto done;
+ }
+
+ ret = bpf_map_update_elem(nolocal_lru_map, &key, &val,
+ BPF_ANY);
+ } else {
+ ret = -EINVAL;
+ }
+
+done:
+ if (ret)
+ bpf_trace_printk(fmt, sizeof(fmt), ret);
return 0;
}
@@ -165,5 +219,31 @@
return 0;
}
+SEC("kprobe/sys_getpgid")
+int stress_hash_map_lookup(struct pt_regs *ctx)
+{
+ u32 key = 1, i;
+ long *value;
+
+#pragma clang loop unroll(full)
+ for (i = 0; i < 64; ++i)
+ value = bpf_map_lookup_elem(&hash_map, &key);
+
+ return 0;
+}
+
+SEC("kprobe/sys_getpgrp")
+int stress_array_map_lookup(struct pt_regs *ctx)
+{
+ u32 key = 1, i;
+ long *value;
+
+#pragma clang loop unroll(full)
+ for (i = 0; i < 64; ++i)
+ value = bpf_map_lookup_elem(&array_map, &key);
+
+ return 0;
+}
+
char _license[] SEC("license") = "GPL";
u32 _version SEC("version") = LINUX_VERSION_CODE;
diff --git a/samples/bpf/map_perf_test_user.c b/samples/bpf/map_perf_test_user.c
index 680260a..1a8894b 100644
--- a/samples/bpf/map_perf_test_user.c
+++ b/samples/bpf/map_perf_test_user.c
@@ -18,10 +18,14 @@
#include <string.h>
#include <time.h>
#include <sys/resource.h>
+#include <arpa/inet.h>
+#include <errno.h>
+
#include "libbpf.h"
#include "bpf_load.h"
-#define MAX_CNT 1000000
+#define TEST_BIT(t) (1U << (t))
+#define MAX_NR_CPUS 1024
static __u64 time_get_ns(void)
{
@@ -31,15 +35,44 @@
return ts.tv_sec * 1000000000ull + ts.tv_nsec;
}
-#define HASH_PREALLOC (1 << 0)
-#define PERCPU_HASH_PREALLOC (1 << 1)
-#define HASH_KMALLOC (1 << 2)
-#define PERCPU_HASH_KMALLOC (1 << 3)
-#define LRU_HASH_PREALLOC (1 << 4)
-#define PERCPU_LRU_HASH_PREALLOC (1 << 5)
-#define LPM_KMALLOC (1 << 6)
+enum test_type {
+ HASH_PREALLOC,
+ PERCPU_HASH_PREALLOC,
+ HASH_KMALLOC,
+ PERCPU_HASH_KMALLOC,
+ LRU_HASH_PREALLOC,
+ NOCOMMON_LRU_HASH_PREALLOC,
+ LPM_KMALLOC,
+ HASH_LOOKUP,
+ ARRAY_LOOKUP,
+ INNER_LRU_HASH_PREALLOC,
+ NR_TESTS,
+};
+
+const char *test_map_names[NR_TESTS] = {
+ [HASH_PREALLOC] = "hash_map",
+ [PERCPU_HASH_PREALLOC] = "percpu_hash_map",
+ [HASH_KMALLOC] = "hash_map_alloc",
+ [PERCPU_HASH_KMALLOC] = "percpu_hash_map_alloc",
+ [LRU_HASH_PREALLOC] = "lru_hash_map",
+ [NOCOMMON_LRU_HASH_PREALLOC] = "nocommon_lru_hash_map",
+ [LPM_KMALLOC] = "lpm_trie_map_alloc",
+ [HASH_LOOKUP] = "hash_map",
+ [ARRAY_LOOKUP] = "array_map",
+ [INNER_LRU_HASH_PREALLOC] = "inner_lru_hash_map",
+};
static int test_flags = ~0;
+static uint32_t num_map_entries;
+static uint32_t inner_lru_hash_size;
+static int inner_lru_hash_idx = -1;
+static int array_of_lru_hashs_idx = -1;
+static uint32_t max_cnt = 1000000;
+
+static int check_test_flags(enum test_type t)
+{
+ return test_flags & TEST_BIT(t);
+}
static void test_hash_prealloc(int cpu)
{
@@ -47,34 +80,89 @@
int i;
start_time = time_get_ns();
- for (i = 0; i < MAX_CNT; i++)
+ for (i = 0; i < max_cnt; i++)
syscall(__NR_getuid);
printf("%d:hash_map_perf pre-alloc %lld events per sec\n",
- cpu, MAX_CNT * 1000000000ll / (time_get_ns() - start_time));
+ cpu, max_cnt * 1000000000ll / (time_get_ns() - start_time));
+}
+
+static void do_test_lru(enum test_type test, int cpu)
+{
+ static int inner_lru_map_fds[MAX_NR_CPUS];
+
+ struct sockaddr_in6 in6 = { .sin6_family = AF_INET6 };
+ const char *test_name;
+ __u64 start_time;
+ int i, ret;
+
+ if (test == INNER_LRU_HASH_PREALLOC) {
+ int outer_fd = map_fd[array_of_lru_hashs_idx];
+
+ assert(cpu < MAX_NR_CPUS);
+
+ if (cpu) {
+ inner_lru_map_fds[cpu] =
+ bpf_create_map(BPF_MAP_TYPE_LRU_HASH,
+ sizeof(uint32_t), sizeof(long),
+ inner_lru_hash_size, 0);
+ if (inner_lru_map_fds[cpu] == -1) {
+ printf("cannot create BPF_MAP_TYPE_LRU_HASH %s(%d)\n",
+ strerror(errno), errno);
+ exit(1);
+ }
+ } else {
+ inner_lru_map_fds[cpu] = map_fd[inner_lru_hash_idx];
+ }
+
+ ret = bpf_map_update_elem(outer_fd, &cpu,
+ &inner_lru_map_fds[cpu],
+ BPF_ANY);
+ if (ret) {
+ printf("cannot update ARRAY_OF_LRU_HASHS with key:%u. %s(%d)\n",
+ cpu, strerror(errno), errno);
+ exit(1);
+ }
+ }
+
+ in6.sin6_addr.s6_addr16[0] = 0xdead;
+ in6.sin6_addr.s6_addr16[1] = 0xbeef;
+
+ if (test == LRU_HASH_PREALLOC) {
+ test_name = "lru_hash_map_perf";
+ in6.sin6_addr.s6_addr16[7] = 0;
+ } else if (test == NOCOMMON_LRU_HASH_PREALLOC) {
+ test_name = "nocommon_lru_hash_map_perf";
+ in6.sin6_addr.s6_addr16[7] = 1;
+ } else if (test == INNER_LRU_HASH_PREALLOC) {
+ test_name = "inner_lru_hash_map_perf";
+ in6.sin6_addr.s6_addr16[7] = 2;
+ } else {
+ assert(0);
+ }
+
+ start_time = time_get_ns();
+ for (i = 0; i < max_cnt; i++) {
+ ret = connect(-1, (const struct sockaddr *)&in6, sizeof(in6));
+ assert(ret == -1 && errno == EBADF);
+ }
+ printf("%d:%s pre-alloc %lld events per sec\n",
+ cpu, test_name,
+ max_cnt * 1000000000ll / (time_get_ns() - start_time));
}
static void test_lru_hash_prealloc(int cpu)
{
- __u64 start_time;
- int i;
-
- start_time = time_get_ns();
- for (i = 0; i < MAX_CNT; i++)
- syscall(__NR_getpid);
- printf("%d:lru_hash_map_perf pre-alloc %lld events per sec\n",
- cpu, MAX_CNT * 1000000000ll / (time_get_ns() - start_time));
+ do_test_lru(LRU_HASH_PREALLOC, cpu);
}
-static void test_percpu_lru_hash_prealloc(int cpu)
+static void test_nocommon_lru_hash_prealloc(int cpu)
{
- __u64 start_time;
- int i;
+ do_test_lru(NOCOMMON_LRU_HASH_PREALLOC, cpu);
+}
- start_time = time_get_ns();
- for (i = 0; i < MAX_CNT; i++)
- syscall(__NR_getppid);
- printf("%d:lru_hash_map_perf pre-alloc %lld events per sec\n",
- cpu, MAX_CNT * 1000000000ll / (time_get_ns() - start_time));
+static void test_inner_lru_hash_prealloc(int cpu)
+{
+ do_test_lru(INNER_LRU_HASH_PREALLOC, cpu);
}
static void test_percpu_hash_prealloc(int cpu)
@@ -83,10 +171,10 @@
int i;
start_time = time_get_ns();
- for (i = 0; i < MAX_CNT; i++)
+ for (i = 0; i < max_cnt; i++)
syscall(__NR_geteuid);
printf("%d:percpu_hash_map_perf pre-alloc %lld events per sec\n",
- cpu, MAX_CNT * 1000000000ll / (time_get_ns() - start_time));
+ cpu, max_cnt * 1000000000ll / (time_get_ns() - start_time));
}
static void test_hash_kmalloc(int cpu)
@@ -95,10 +183,10 @@
int i;
start_time = time_get_ns();
- for (i = 0; i < MAX_CNT; i++)
+ for (i = 0; i < max_cnt; i++)
syscall(__NR_getgid);
printf("%d:hash_map_perf kmalloc %lld events per sec\n",
- cpu, MAX_CNT * 1000000000ll / (time_get_ns() - start_time));
+ cpu, max_cnt * 1000000000ll / (time_get_ns() - start_time));
}
static void test_percpu_hash_kmalloc(int cpu)
@@ -107,10 +195,10 @@
int i;
start_time = time_get_ns();
- for (i = 0; i < MAX_CNT; i++)
+ for (i = 0; i < max_cnt; i++)
syscall(__NR_getegid);
printf("%d:percpu_hash_map_perf kmalloc %lld events per sec\n",
- cpu, MAX_CNT * 1000000000ll / (time_get_ns() - start_time));
+ cpu, max_cnt * 1000000000ll / (time_get_ns() - start_time));
}
static void test_lpm_kmalloc(int cpu)
@@ -119,40 +207,63 @@
int i;
start_time = time_get_ns();
- for (i = 0; i < MAX_CNT; i++)
+ for (i = 0; i < max_cnt; i++)
syscall(__NR_gettid);
printf("%d:lpm_perf kmalloc %lld events per sec\n",
- cpu, MAX_CNT * 1000000000ll / (time_get_ns() - start_time));
+ cpu, max_cnt * 1000000000ll / (time_get_ns() - start_time));
}
+static void test_hash_lookup(int cpu)
+{
+ __u64 start_time;
+ int i;
+
+ start_time = time_get_ns();
+ for (i = 0; i < max_cnt; i++)
+ syscall(__NR_getpgid, 0);
+ printf("%d:hash_lookup %lld lookups per sec\n",
+ cpu, max_cnt * 1000000000ll * 64 / (time_get_ns() - start_time));
+}
+
+static void test_array_lookup(int cpu)
+{
+ __u64 start_time;
+ int i;
+
+ start_time = time_get_ns();
+ for (i = 0; i < max_cnt; i++)
+ syscall(__NR_getpgrp, 0);
+ printf("%d:array_lookup %lld lookups per sec\n",
+ cpu, max_cnt * 1000000000ll * 64 / (time_get_ns() - start_time));
+}
+
+typedef void (*test_func)(int cpu);
+const test_func test_funcs[] = {
+ [HASH_PREALLOC] = test_hash_prealloc,
+ [PERCPU_HASH_PREALLOC] = test_percpu_hash_prealloc,
+ [HASH_KMALLOC] = test_hash_kmalloc,
+ [PERCPU_HASH_KMALLOC] = test_percpu_hash_kmalloc,
+ [LRU_HASH_PREALLOC] = test_lru_hash_prealloc,
+ [NOCOMMON_LRU_HASH_PREALLOC] = test_nocommon_lru_hash_prealloc,
+ [LPM_KMALLOC] = test_lpm_kmalloc,
+ [HASH_LOOKUP] = test_hash_lookup,
+ [ARRAY_LOOKUP] = test_array_lookup,
+ [INNER_LRU_HASH_PREALLOC] = test_inner_lru_hash_prealloc,
+};
+
static void loop(int cpu)
{
cpu_set_t cpuset;
+ int i;
CPU_ZERO(&cpuset);
CPU_SET(cpu, &cpuset);
sched_setaffinity(0, sizeof(cpuset), &cpuset);
- if (test_flags & HASH_PREALLOC)
- test_hash_prealloc(cpu);
-
- if (test_flags & PERCPU_HASH_PREALLOC)
- test_percpu_hash_prealloc(cpu);
-
- if (test_flags & HASH_KMALLOC)
- test_hash_kmalloc(cpu);
-
- if (test_flags & PERCPU_HASH_KMALLOC)
- test_percpu_hash_kmalloc(cpu);
-
- if (test_flags & LRU_HASH_PREALLOC)
- test_lru_hash_prealloc(cpu);
-
- if (test_flags & PERCPU_LRU_HASH_PREALLOC)
- test_percpu_lru_hash_prealloc(cpu);
-
- if (test_flags & LPM_KMALLOC)
- test_lpm_kmalloc(cpu);
+ for (i = 0; i < NR_TESTS; i++) {
+ if (check_test_flags(i))
+ test_funcs[i](cpu);
+ }
}
static void run_perf_test(int tasks)
@@ -209,6 +320,38 @@
assert(!r);
}
+static void fixup_map(struct bpf_map_data *map, int idx)
+{
+ int i;
+
+ if (!strcmp("inner_lru_hash_map", map->name)) {
+ inner_lru_hash_idx = idx;
+ inner_lru_hash_size = map->def.max_entries;
+ }
+
+ if (!strcmp("array_of_lru_hashs", map->name)) {
+ if (inner_lru_hash_idx == -1) {
+ printf("inner_lru_hash_map must be defined before array_of_lru_hashs\n");
+ exit(1);
+ }
+ map->def.inner_map_idx = inner_lru_hash_idx;
+ array_of_lru_hashs_idx = idx;
+ }
+
+ if (num_map_entries <= 0)
+ return;
+
+ inner_lru_hash_size = num_map_entries;
+
+ /* Only change the max_entries for the enabled test(s) */
+ for (i = 0; i < NR_TESTS; i++) {
+ if (!strcmp(test_map_names[i], map->name) &&
+ (check_test_flags(i))) {
+ map->def.max_entries = num_map_entries;
+ }
+ }
+}
+
int main(int argc, char **argv)
{
struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
@@ -224,7 +367,13 @@
if (argc > 2)
num_cpu = atoi(argv[2]) ? : num_cpu;
- if (load_bpf_file(filename)) {
+ if (argc > 3)
+ num_map_entries = atoi(argv[3]);
+
+ if (argc > 4)
+ max_cnt = atoi(argv[4]);
+
+ if (load_bpf_file_fixup_map(filename, fixup_map)) {
printf("%s", bpf_log_buf);
return 1;
}
diff --git a/samples/bpf/run_cookie_uid_helper_example.sh b/samples/bpf/run_cookie_uid_helper_example.sh
new file mode 100755
index 0000000..f898cfa
--- /dev/null
+++ b/samples/bpf/run_cookie_uid_helper_example.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+local_dir="$(pwd)"
+root_dir=$local_dir/../..
+mnt_dir=$(mktemp -d --tmp)
+
+on_exit() {
+ iptables -D OUTPUT -m bpf --object-pinned ${mnt_dir}/bpf_prog -j ACCEPT
+ umount ${mnt_dir}
+ rm -r ${mnt_dir}
+}
+
+trap on_exit EXIT
+mount -t bpf bpf ${mnt_dir}
+./per_socket_stats_example ${mnt_dir}/bpf_prog $1
diff --git a/samples/bpf/test_lru_dist.c b/samples/bpf/test_lru_dist.c
index d96dc88..73c3571 100644
--- a/samples/bpf/test_lru_dist.c
+++ b/samples/bpf/test_lru_dist.c
@@ -25,7 +25,9 @@
#include "bpf_util.h"
#define min(a, b) ((a) < (b) ? (a) : (b))
-#define offsetof(TYPE, MEMBER) ((size_t)&((TYPE *)0)->MEMBER)
+#ifndef offsetof
+# define offsetof(TYPE, MEMBER) ((size_t)&((TYPE *)0)->MEMBER)
+#endif
#define container_of(ptr, type, member) ({ \
const typeof( ((type *)0)->member ) *__mptr = (ptr); \
(type *)( (char *)__mptr - offsetof(type,member) );})
diff --git a/samples/bpf/test_map_in_map_kern.c b/samples/bpf/test_map_in_map_kern.c
new file mode 100644
index 0000000..42c44d0
--- /dev/null
+++ b/samples/bpf/test_map_in_map_kern.c
@@ -0,0 +1,173 @@
+/*
+ * Copyright (c) 2017 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#define KBUILD_MODNAME "foo"
+#include <linux/ptrace.h>
+#include <linux/version.h>
+#include <uapi/linux/bpf.h>
+#include <uapi/linux/in6.h>
+#include "bpf_helpers.h"
+
+#define MAX_NR_PORTS 65536
+
+/* map #0 */
+struct bpf_map_def SEC("maps") port_a = {
+ .type = BPF_MAP_TYPE_ARRAY,
+ .key_size = sizeof(u32),
+ .value_size = sizeof(int),
+ .max_entries = MAX_NR_PORTS,
+};
+
+/* map #1 */
+struct bpf_map_def SEC("maps") port_h = {
+ .type = BPF_MAP_TYPE_HASH,
+ .key_size = sizeof(u32),
+ .value_size = sizeof(int),
+ .max_entries = 1,
+};
+
+/* map #2 */
+struct bpf_map_def SEC("maps") reg_result_h = {
+ .type = BPF_MAP_TYPE_HASH,
+ .key_size = sizeof(u32),
+ .value_size = sizeof(int),
+ .max_entries = 1,
+};
+
+/* map #3 */
+struct bpf_map_def SEC("maps") inline_result_h = {
+ .type = BPF_MAP_TYPE_HASH,
+ .key_size = sizeof(u32),
+ .value_size = sizeof(int),
+ .max_entries = 1,
+};
+
+/* map #4 */ /* Test case #0 */
+struct bpf_map_def SEC("maps") a_of_port_a = {
+ .type = BPF_MAP_TYPE_ARRAY_OF_MAPS,
+ .key_size = sizeof(u32),
+ .inner_map_idx = 0, /* map_fd[0] is port_a */
+ .max_entries = MAX_NR_PORTS,
+};
+
+/* map #5 */ /* Test case #1 */
+struct bpf_map_def SEC("maps") h_of_port_a = {
+ .type = BPF_MAP_TYPE_HASH_OF_MAPS,
+ .key_size = sizeof(u32),
+ .inner_map_idx = 0, /* map_fd[0] is port_a */
+ .max_entries = 1,
+};
+
+/* map #6 */ /* Test case #2 */
+struct bpf_map_def SEC("maps") h_of_port_h = {
+ .type = BPF_MAP_TYPE_HASH_OF_MAPS,
+ .key_size = sizeof(u32),
+ .inner_map_idx = 1, /* map_fd[1] is port_h */
+ .max_entries = 1,
+};
+
+static __always_inline int do_reg_lookup(void *inner_map, u32 port)
+{
+ int *result;
+
+ result = bpf_map_lookup_elem(inner_map, &port);
+ return result ? *result : -ENOENT;
+}
+
+static __always_inline int do_inline_array_lookup(void *inner_map, u32 port)
+{
+ int *result;
+
+ if (inner_map != &port_a)
+ return -EINVAL;
+
+ result = bpf_map_lookup_elem(&port_a, &port);
+ return result ? *result : -ENOENT;
+}
+
+static __always_inline int do_inline_hash_lookup(void *inner_map, u32 port)
+{
+ int *result;
+
+ if (inner_map != &port_h)
+ return -EINVAL;
+
+ result = bpf_map_lookup_elem(&port_h, &port);
+ return result ? *result : -ENOENT;
+}
+
+SEC("kprobe/sys_connect")
+int trace_sys_connect(struct pt_regs *ctx)
+{
+ struct sockaddr_in6 *in6;
+ u16 test_case, port, dst6[8];
+ int addrlen, ret, inline_ret, ret_key = 0;
+ u32 port_key;
+ void *outer_map, *inner_map;
+ bool inline_hash = false;
+
+ in6 = (struct sockaddr_in6 *)PT_REGS_PARM2(ctx);
+ addrlen = (int)PT_REGS_PARM3(ctx);
+
+ if (addrlen != sizeof(*in6))
+ return 0;
+
+ ret = bpf_probe_read(dst6, sizeof(dst6), &in6->sin6_addr);
+ if (ret) {
+ inline_ret = ret;
+ goto done;
+ }
+
+ if (dst6[0] != 0xdead || dst6[1] != 0xbeef)
+ return 0;
+
+ test_case = dst6[7];
+
+ ret = bpf_probe_read(&port, sizeof(port), &in6->sin6_port);
+ if (ret) {
+ inline_ret = ret;
+ goto done;
+ }
+
+ port_key = port;
+
+ ret = -ENOENT;
+ if (test_case == 0) {
+ outer_map = &a_of_port_a;
+ } else if (test_case == 1) {
+ outer_map = &h_of_port_a;
+ } else if (test_case == 2) {
+ outer_map = &h_of_port_h;
+ } else {
+ ret = __LINE__;
+ inline_ret = ret;
+ goto done;
+ }
+
+ inner_map = bpf_map_lookup_elem(outer_map, &port_key);
+ if (!inner_map) {
+ ret = __LINE__;
+ inline_ret = ret;
+ goto done;
+ }
+
+ ret = do_reg_lookup(inner_map, port_key);
+
+ if (test_case == 0 || test_case == 1)
+ inline_ret = do_inline_array_lookup(inner_map, port_key);
+ else
+ inline_ret = do_inline_hash_lookup(inner_map, port_key);
+
+done:
+ bpf_map_update_elem(®_result_h, &ret_key, &ret, BPF_ANY);
+ bpf_map_update_elem(&inline_result_h, &ret_key, &inline_ret, BPF_ANY);
+
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
+u32 _version SEC("version") = LINUX_VERSION_CODE;
diff --git a/samples/bpf/test_map_in_map_user.c b/samples/bpf/test_map_in_map_user.c
new file mode 100644
index 0000000..f62fdc2
--- /dev/null
+++ b/samples/bpf/test_map_in_map_user.c
@@ -0,0 +1,116 @@
+/*
+ * Copyright (c) 2017 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <sys/resource.h>
+#include <sys/socket.h>
+#include <arpa/inet.h>
+#include <stdint.h>
+#include <assert.h>
+#include <errno.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include "libbpf.h"
+#include "bpf_load.h"
+
+#define PORT_A (map_fd[0])
+#define PORT_H (map_fd[1])
+#define REG_RESULT_H (map_fd[2])
+#define INLINE_RESULT_H (map_fd[3])
+#define A_OF_PORT_A (map_fd[4]) /* Test case #0 */
+#define H_OF_PORT_A (map_fd[5]) /* Test case #1 */
+#define H_OF_PORT_H (map_fd[6]) /* Test case #2 */
+
+static const char * const test_names[] = {
+ "Array of Array",
+ "Hash of Array",
+ "Hash of Hash",
+};
+
+#define NR_TESTS (sizeof(test_names) / sizeof(*test_names))
+
+static void populate_map(uint32_t port_key, int magic_result)
+{
+ int ret;
+
+ ret = bpf_map_update_elem(PORT_A, &port_key, &magic_result, BPF_ANY);
+ assert(!ret);
+
+ ret = bpf_map_update_elem(PORT_H, &port_key, &magic_result,
+ BPF_NOEXIST);
+ assert(!ret);
+
+ ret = bpf_map_update_elem(A_OF_PORT_A, &port_key, &PORT_A, BPF_ANY);
+ assert(!ret);
+
+ ret = bpf_map_update_elem(H_OF_PORT_A, &port_key, &PORT_A, BPF_NOEXIST);
+ assert(!ret);
+
+ ret = bpf_map_update_elem(H_OF_PORT_H, &port_key, &PORT_H, BPF_NOEXIST);
+ assert(!ret);
+}
+
+static void test_map_in_map(void)
+{
+ struct sockaddr_in6 in6 = { .sin6_family = AF_INET6 };
+ uint32_t result_key = 0, port_key;
+ int result, inline_result;
+ int magic_result = 0xfaceb00c;
+ int ret;
+ int i;
+
+ port_key = rand() & 0x00FF;
+ populate_map(port_key, magic_result);
+
+ in6.sin6_addr.s6_addr16[0] = 0xdead;
+ in6.sin6_addr.s6_addr16[1] = 0xbeef;
+ in6.sin6_port = port_key;
+
+ for (i = 0; i < NR_TESTS; i++) {
+ printf("%s: ", test_names[i]);
+
+ in6.sin6_addr.s6_addr16[7] = i;
+ ret = connect(-1, (struct sockaddr *)&in6, sizeof(in6));
+ assert(ret == -1 && errno == EBADF);
+
+ ret = bpf_map_lookup_elem(REG_RESULT_H, &result_key, &result);
+ assert(!ret);
+
+ ret = bpf_map_lookup_elem(INLINE_RESULT_H, &result_key,
+ &inline_result);
+ assert(!ret);
+
+ if (result != magic_result || inline_result != magic_result) {
+ printf("Error. result:%d inline_result:%d\n",
+ result, inline_result);
+ exit(1);
+ }
+
+ bpf_map_delete_elem(REG_RESULT_H, &result_key);
+ bpf_map_delete_elem(INLINE_RESULT_H, &result_key);
+
+ printf("Pass\n");
+ }
+}
+
+int main(int argc, char **argv)
+{
+ struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
+ char filename[256];
+
+ assert(!setrlimit(RLIMIT_MEMLOCK, &r));
+
+ snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+
+ if (load_bpf_file(filename)) {
+ printf("%s", bpf_log_buf);
+ return 1;
+ }
+
+ test_map_in_map();
+
+ return 0;
+}
diff --git a/samples/bpf/tracex2_user.c b/samples/bpf/tracex2_user.c
index ded9804..7fee0f1 100644
--- a/samples/bpf/tracex2_user.c
+++ b/samples/bpf/tracex2_user.c
@@ -4,6 +4,7 @@
#include <signal.h>
#include <linux/bpf.h>
#include <string.h>
+#include <sys/resource.h>
#include "libbpf.h"
#include "bpf_load.h"
@@ -112,6 +113,7 @@
int main(int ac, char **argv)
{
+ struct rlimit r = {1024*1024, RLIM_INFINITY};
char filename[256];
long key, next_key, value;
FILE *f;
@@ -119,6 +121,11 @@
snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+ if (setrlimit(RLIMIT_MEMLOCK, &r)) {
+ perror("setrlimit(RLIMIT_MEMLOCK)");
+ return 1;
+ }
+
signal(SIGINT, int_exit);
/* start 'ping' in the background to have some kfree_skb events */
diff --git a/samples/bpf/tracex3_user.c b/samples/bpf/tracex3_user.c
index 8f7d199..fe37223 100644
--- a/samples/bpf/tracex3_user.c
+++ b/samples/bpf/tracex3_user.c
@@ -11,6 +11,7 @@
#include <stdbool.h>
#include <string.h>
#include <linux/bpf.h>
+#include <sys/resource.h>
#include "libbpf.h"
#include "bpf_load.h"
@@ -112,11 +113,17 @@
int main(int ac, char **argv)
{
+ struct rlimit r = {1024*1024, RLIM_INFINITY};
char filename[256];
int i;
snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+ if (setrlimit(RLIMIT_MEMLOCK, &r)) {
+ perror("setrlimit(RLIMIT_MEMLOCK)");
+ return 1;
+ }
+
if (load_bpf_file(filename)) {
printf("%s", bpf_log_buf);
return 1;
diff --git a/samples/bpf/tracex4_user.c b/samples/bpf/tracex4_user.c
index 03449f77..22c644f 100644
--- a/samples/bpf/tracex4_user.c
+++ b/samples/bpf/tracex4_user.c
@@ -12,6 +12,8 @@
#include <string.h>
#include <time.h>
#include <linux/bpf.h>
+#include <sys/resource.h>
+
#include "libbpf.h"
#include "bpf_load.h"
@@ -50,11 +52,17 @@
int main(int ac, char **argv)
{
+ struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
char filename[256];
int i;
snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
+ if (setrlimit(RLIMIT_MEMLOCK, &r)) {
+ perror("setrlimit(RLIMIT_MEMLOCK, RLIM_INFINITY)");
+ return 1;
+ }
+
if (load_bpf_file(filename)) {
printf("%s", bpf_log_buf);
return 1;
diff --git a/samples/bpf/xdp1_user.c b/samples/bpf/xdp1_user.c
index d2be65d..378850c 100644
--- a/samples/bpf/xdp1_user.c
+++ b/samples/bpf/xdp1_user.c
@@ -5,6 +5,7 @@
* License as published by the Free Software Foundation.
*/
#include <linux/bpf.h>
+#include <linux/if_link.h>
#include <assert.h>
#include <errno.h>
#include <signal.h>
@@ -12,16 +13,18 @@
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
+#include <libgen.h>
#include "bpf_load.h"
#include "bpf_util.h"
#include "libbpf.h"
static int ifindex;
+static __u32 xdp_flags;
static void int_exit(int sig)
{
- set_link_xdp_fd(ifindex, -1);
+ set_link_xdp_fd(ifindex, -1, xdp_flags);
exit(0);
}
@@ -54,18 +57,39 @@
}
}
-int main(int ac, char **argv)
+static void usage(const char *prog)
{
+ fprintf(stderr,
+ "usage: %s [OPTS] IFINDEX\n\n"
+ "OPTS:\n"
+ " -S use skb-mode\n",
+ prog);
+}
+
+int main(int argc, char **argv)
+{
+ const char *optstr = "S";
char filename[256];
+ int opt;
- snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
-
- if (ac != 2) {
- printf("usage: %s IFINDEX\n", argv[0]);
- return 1;
+ while ((opt = getopt(argc, argv, optstr)) != -1) {
+ switch (opt) {
+ case 'S':
+ xdp_flags |= XDP_FLAGS_SKB_MODE;
+ break;
+ default:
+ usage(basename(argv[0]));
+ return 1;
+ }
}
- ifindex = strtoul(argv[1], NULL, 0);
+ if (optind == argc) {
+ usage(basename(argv[0]));
+ return 1;
+ }
+ ifindex = strtoul(argv[optind], NULL, 0);
+
+ snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
if (load_bpf_file(filename)) {
printf("%s", bpf_log_buf);
@@ -79,7 +103,7 @@
signal(SIGINT, int_exit);
- if (set_link_xdp_fd(ifindex, prog_fd[0]) < 0) {
+ if (set_link_xdp_fd(ifindex, prog_fd[0], xdp_flags) < 0) {
printf("link set xdp fd failed\n");
return 1;
}
diff --git a/samples/bpf/xdp_tx_iptunnel_user.c b/samples/bpf/xdp_tx_iptunnel_user.c
index 70e192f..92b8bde 100644
--- a/samples/bpf/xdp_tx_iptunnel_user.c
+++ b/samples/bpf/xdp_tx_iptunnel_user.c
@@ -5,6 +5,7 @@
* License as published by the Free Software Foundation.
*/
#include <linux/bpf.h>
+#include <linux/if_link.h>
#include <assert.h>
#include <errno.h>
#include <signal.h>
@@ -24,11 +25,12 @@
#define STATS_INTERVAL_S 2U
static int ifindex = -1;
+static __u32 xdp_flags = 0;
static void int_exit(int sig)
{
if (ifindex > -1)
- set_link_xdp_fd(ifindex, -1);
+ set_link_xdp_fd(ifindex, -1, xdp_flags);
exit(0);
}
@@ -136,7 +138,7 @@
{
unsigned char opt_flags[256] = {};
unsigned int kill_after_s = 0;
- const char *optstr = "i:a:p:s:d:m:T:P:h";
+ const char *optstr = "i:a:p:s:d:m:T:P:Sh";
int min_port = 0, max_port = 0;
struct iptnl_info tnl = {};
struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
@@ -201,6 +203,9 @@
case 'T':
kill_after_s = atoi(optarg);
break;
+ case 'S':
+ xdp_flags |= XDP_FLAGS_SKB_MODE;
+ break;
default:
usage(argv[0]);
return 1;
@@ -243,14 +248,14 @@
}
}
- if (set_link_xdp_fd(ifindex, prog_fd[0]) < 0) {
+ if (set_link_xdp_fd(ifindex, prog_fd[0], xdp_flags) < 0) {
printf("link set xdp fd failed\n");
return 1;
}
poll_stats(kill_after_s);
- set_link_xdp_fd(ifindex, -1);
+ set_link_xdp_fd(ifindex, -1, xdp_flags);
return 0;
}
diff --git a/samples/livepatch/livepatch-sample.c b/samples/livepatch/livepatch-sample.c
index e34f871..8479522 100644
--- a/samples/livepatch/livepatch-sample.c
+++ b/samples/livepatch/livepatch-sample.c
@@ -17,6 +17,8 @@
* along with this program; if not, see <http://www.gnu.org/licenses/>.
*/
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/livepatch.h>
@@ -69,6 +71,21 @@
{
int ret;
+ if (!klp_have_reliable_stack() && !patch.immediate) {
+ /*
+ * WARNING: Be very careful when using 'patch.immediate' in
+ * your patches. It's ok to use it for simple patches like
+ * this, but for more complex patches which change function
+ * semantics, locking semantics, or data structures, it may not
+ * be safe. Use of this option will also prevent removal of
+ * the patch.
+ *
+ * See Documentation/livepatch/livepatch.txt for more details.
+ */
+ patch.immediate = true;
+ pr_notice("The consistency model isn't supported for your architecture. Bypassing safety mechanisms and applying the patch immediately.\n");
+ }
+
ret = klp_register_patch(&patch);
if (ret)
return ret;
@@ -82,7 +99,6 @@
static void livepatch_exit(void)
{
- WARN_ON(klp_disable_patch(&patch));
WARN_ON(klp_unregister_patch(&patch));
}
diff --git a/scripts/checkstack.pl b/scripts/checkstack.pl
index 3033be7..9d37aa4 100755
--- a/scripts/checkstack.pl
+++ b/scripts/checkstack.pl
@@ -12,7 +12,6 @@
# sh64 port by Paul Mundt
# Random bits by Matt Mackall <mpm@selenic.com>
# M68k port by Geert Uytterhoeven and Andreas Schwab
-# AVR32 port by Haavard Skinnemoen (Atmel)
# AArch64, PARISC ports by Kyle McMartin
# sparc port by Martin Habets <errandir_news@mph.eclipse.co.uk>
#
@@ -51,10 +50,6 @@
} elsif ($arch eq 'arm') {
#c0008ffc: e24dd064 sub sp, sp, #100 ; 0x64
$re = qr/.*sub.*sp, sp, #(([0-9]{2}|[3-9])[0-9]{2})/o;
- } elsif ($arch eq 'avr32') {
- #8000008a: 20 1d sub sp,4
- #80000ca8: fa cd 05 b0 sub sp,sp,1456
- $re = qr/^.*sub.*sp.*,([0-9]{1,8})/o;
} elsif ($arch =~ /^x86(_64)?$/ || $arch =~ /^i[3456]86$/) {
#c0105234: 81 ec ac 05 00 00 sub $0x5ac,%esp
# or
diff --git a/scripts/checksyscalls.sh b/scripts/checksyscalls.sh
index 2c9082b..116b773 100755
--- a/scripts/checksyscalls.sh
+++ b/scripts/checksyscalls.sh
@@ -148,6 +148,7 @@
#define __IGNORE_sysfs
#define __IGNORE_uselib
#define __IGNORE__sysctl
+#define __IGNORE_arch_prctl
/* ... including the "new" 32-bit uid syscalls */
#define __IGNORE_lchown32
diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 33c85df..a26a5f2 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -202,6 +202,7 @@
# '&struct_name.member' - name of a structure member
# '@parameter' - name of a parameter
# '%CONST' - name of a constant.
+# '``LITERAL``' - literal string without any spaces on it.
## init lots of data
@@ -210,7 +211,8 @@
my $anon_struct_union = 0;
# match expressions used to find embedded type information
-my $type_constant = '\%([-_\w]+)';
+my $type_constant = '\b``([^\`]+)``\b';
+my $type_constant2 = '\%([-_\w]+)';
my $type_func = '(\w+)\(\)';
my $type_param = '\@(\w+(\.\.\.)?)';
my $type_fp_param = '\@(\w+)\(\)'; # Special RST handling for func ptr params
@@ -235,6 +237,7 @@
# these work fairly well
my @highlights_html = (
[$type_constant, "<i>\$1</i>"],
+ [$type_constant2, "<i>\$1</i>"],
[$type_func, "<b>\$1</b>"],
[$type_enum_xml, "<i>\$1</i>"],
[$type_struct_xml, "<i>\$1</i>"],
@@ -252,6 +255,7 @@
# html version 5
my @highlights_html5 = (
[$type_constant, "<span class=\"const\">\$1</span>"],
+ [$type_constant2, "<span class=\"const\">\$1</span>"],
[$type_func, "<span class=\"func\">\$1</span>"],
[$type_enum_xml, "<span class=\"enum\">\$1</span>"],
[$type_struct_xml, "<span class=\"struct\">\$1</span>"],
@@ -268,6 +272,7 @@
my @highlights_xml = (
["([^=])\\\"([^\\\"<]+)\\\"", "\$1<quote>\$2</quote>"],
[$type_constant, "<constant>\$1</constant>"],
+ [$type_constant2, "<constant>\$1</constant>"],
[$type_enum_xml, "<type>\$1</type>"],
[$type_struct_xml, "<structname>\$1</structname>"],
[$type_typedef_xml, "<type>\$1</type>"],
@@ -283,6 +288,7 @@
# gnome, docbook format
my @highlights_gnome = (
[$type_constant, "<replaceable class=\"option\">\$1</replaceable>"],
+ [$type_constant2, "<replaceable class=\"option\">\$1</replaceable>"],
[$type_func, "<function>\$1</function>"],
[$type_enum, "<type>\$1</type>"],
[$type_struct, "<structname>\$1</structname>"],
@@ -298,6 +304,7 @@
# these are pretty rough
my @highlights_man = (
[$type_constant, "\$1"],
+ [$type_constant2, "\$1"],
[$type_func, "\\\\fB\$1\\\\fP"],
[$type_enum, "\\\\fI\$1\\\\fP"],
[$type_struct, "\\\\fI\$1\\\\fP"],
@@ -312,6 +319,7 @@
# text-mode
my @highlights_text = (
[$type_constant, "\$1"],
+ [$type_constant2, "\$1"],
[$type_func, "\$1"],
[$type_enum, "\$1"],
[$type_struct, "\$1"],
@@ -326,6 +334,7 @@
# rst-mode
my @highlights_rst = (
[$type_constant, "``\$1``"],
+ [$type_constant2, "``\$1``"],
# Note: need to escape () to avoid func matching later
[$type_member_func, "\\:c\\:type\\:`\$1\$2\$3\\\\(\\\\) <\$1>`"],
[$type_member, "\\:c\\:type\\:`\$1\$2\$3 <\$1>`"],
@@ -344,6 +353,7 @@
# list mode
my @highlights_list = (
[$type_constant, "\$1"],
+ [$type_constant2, "\$1"],
[$type_func, "\$1"],
[$type_enum, "\$1"],
[$type_struct, "\$1"],
@@ -2392,8 +2402,7 @@
}
$anon_struct_union = 0;
- my $param_name = $param;
- $param_name =~ s/\[.*//;
+ $param =~ s/[\[\)].*//;
if ($type eq "" && $param =~ /\.\.\.$/)
{
@@ -2424,9 +2433,9 @@
# but inline preprocessor statements);
# also ignore unnamed structs/unions;
if (!$anon_struct_union) {
- if (!defined $parameterdescs{$param_name} && $param_name !~ /^#/) {
+ if (!defined $parameterdescs{$param} && $param !~ /^#/) {
- $parameterdescs{$param_name} = $undescribed;
+ $parameterdescs{$param} = $undescribed;
if (($type eq 'function') || ($type eq 'enum')) {
print STDERR "${file}:$.: warning: Function parameter ".
diff --git a/scripts/ksymoops/README b/scripts/ksymoops/README
index f6cb06e..4130439 100644
--- a/scripts/ksymoops/README
+++ b/scripts/ksymoops/README
@@ -1,8 +1,7 @@
ksymoops has been removed from the kernel. It was always meant to be a
free standing utility, not linked to any particular kernel version.
The latest version can be found in
-ftp://ftp.<country>.kernel.org/pub/linux/utils/kernel/ksymoops together
-with patches to other utilities in order to give more accurate Oops
-debugging.
+https://www.kernel.org/pub/linux/utils/kernel/ksymoops together with patches to
+other utilities in order to give more accurate Oops debugging.
Keith Owens <kaos@ocs.com.au> Sat Jun 19 10:30:34 EST 1999
diff --git a/scripts/package/builddeb b/scripts/package/builddeb
index 3c575cd0..676fc10 100755
--- a/scripts/package/builddeb
+++ b/scripts/package/builddeb
@@ -262,8 +262,8 @@
cat <<EOF > debian/copyright
This is a packacked upstream version of the Linux kernel.
-The sources may be found at most Linux ftp sites, including:
-ftp://ftp.kernel.org/pub/linux/kernel
+The sources may be found at most Linux archive sites, including:
+https://www.kernel.org/pub/linux/kernel
Copyright: 1991 - 2015 Linus Torvalds and others.
diff --git a/security/Kconfig b/security/Kconfig
index d900f47..213df4d 100644
--- a/security/Kconfig
+++ b/security/Kconfig
@@ -125,17 +125,8 @@
validating memory ranges against heap object sizes in
support of CONFIG_HARDENED_USERCOPY.
-config HAVE_ARCH_HARDENED_USERCOPY
- bool
- help
- The architecture supports CONFIG_HARDENED_USERCOPY by
- calling check_object_size() just before performing the
- userspace copies in the low level implementation of
- copy_to_user() and copy_from_user().
-
config HARDENED_USERCOPY
bool "Harden memory copies between kernel and userspace"
- depends on HAVE_ARCH_HARDENED_USERCOPY
depends on HAVE_HARDENED_USERCOPY_ALLOCATOR
select BUG
help
diff --git a/security/security.c b/security/security.c
index d0e07f2..23555c5 100644
--- a/security/security.c
+++ b/security/security.c
@@ -103,10 +103,14 @@
* to avoid security registration races. This method may also be used
* to check if your LSM is currently loaded during kernel initialization.
*
- * Return true if:
- * -The passed LSM is the one chosen by user at boot time,
- * -or the passed LSM is configured as the default and the user did not
- * choose an alternate LSM at boot time.
+ * Returns:
+ *
+ * true if:
+ *
+ * - The passed LSM is the one chosen by user at boot time,
+ * - or the passed LSM is configured as the default and the user did not
+ * choose an alternate LSM at boot time.
+ *
* Otherwise, return false.
*/
int __init security_module_enable(const char *module)
diff --git a/security/selinux/nlmsgtab.c b/security/selinux/nlmsgtab.c
index 2ca9cde..8e67bb4 100644
--- a/security/selinux/nlmsgtab.c
+++ b/security/selinux/nlmsgtab.c
@@ -69,6 +69,7 @@
{ RTM_GETDCB, NETLINK_ROUTE_SOCKET__NLMSG_READ },
{ RTM_SETDCB, NETLINK_ROUTE_SOCKET__NLMSG_WRITE },
{ RTM_NEWNETCONF, NETLINK_ROUTE_SOCKET__NLMSG_WRITE },
+ { RTM_DELNETCONF, NETLINK_ROUTE_SOCKET__NLMSG_WRITE },
{ RTM_GETNETCONF, NETLINK_ROUTE_SOCKET__NLMSG_READ },
{ RTM_NEWMDB, NETLINK_ROUTE_SOCKET__NLMSG_WRITE },
{ RTM_DELMDB, NETLINK_ROUTE_SOCKET__NLMSG_WRITE },
diff --git a/security/tomoyo/network.c b/security/tomoyo/network.c
index 9752771..6c02ac4 100644
--- a/security/tomoyo/network.c
+++ b/security/tomoyo/network.c
@@ -608,7 +608,7 @@
static bool tomoyo_kernel_service(void)
{
/* Nothing to do if I am a kernel service. */
- return segment_eq(get_fs(), KERNEL_DS);
+ return uaccess_kernel();
}
/**
diff --git a/sound/core/seq/seq_lock.c b/sound/core/seq/seq_lock.c
index 3b693e9..12ba833 100644
--- a/sound/core/seq/seq_lock.c
+++ b/sound/core/seq/seq_lock.c
@@ -28,19 +28,16 @@
/* wait until all locks are released */
void snd_use_lock_sync_helper(snd_use_lock_t *lockp, const char *file, int line)
{
- int max_count = 5 * HZ;
+ int warn_count = 5 * HZ;
if (atomic_read(lockp) < 0) {
pr_warn("ALSA: seq_lock: lock trouble [counter = %d] in %s:%d\n", atomic_read(lockp), file, line);
return;
}
while (atomic_read(lockp) > 0) {
- if (max_count == 0) {
- pr_warn("ALSA: seq_lock: timeout [%d left] in %s:%d\n", atomic_read(lockp), file, line);
- break;
- }
+ if (warn_count-- == 0)
+ pr_warn("ALSA: seq_lock: waiting [%d left] in %s:%d\n", atomic_read(lockp), file, line);
schedule_timeout_uninterruptible(1);
- max_count--;
}
}
diff --git a/sound/firewire/lib.h b/sound/firewire/lib.h
index f676931..c3768cd 100644
--- a/sound/firewire/lib.h
+++ b/sound/firewire/lib.h
@@ -45,7 +45,7 @@
struct snd_rawmidi_substream *substream;
snd_fw_async_midi_port_fill fill;
- unsigned int consume_bytes;
+ int consume_bytes;
};
int snd_fw_async_midi_port_init(struct snd_fw_async_midi_port *port,
diff --git a/sound/firewire/oxfw/oxfw.c b/sound/firewire/oxfw/oxfw.c
index 74d7fb6..413ab63 100644
--- a/sound/firewire/oxfw/oxfw.c
+++ b/sound/firewire/oxfw/oxfw.c
@@ -227,11 +227,11 @@
if (err < 0)
goto error;
- err = detect_quirks(oxfw);
+ err = snd_oxfw_stream_discover(oxfw);
if (err < 0)
goto error;
- err = snd_oxfw_stream_discover(oxfw);
+ err = detect_quirks(oxfw);
if (err < 0)
goto error;
diff --git a/sound/pci/hda/dell_wmi_helper.c b/sound/pci/hda/dell_wmi_helper.c
index 19d41da..7efa7bd 100644
--- a/sound/pci/hda/dell_wmi_helper.c
+++ b/sound/pci/hda/dell_wmi_helper.c
@@ -2,11 +2,11 @@
* to be included from codec driver
*/
-#if IS_ENABLED(CONFIG_LEDS_DELL_NETBOOKS)
+#if IS_ENABLED(CONFIG_DELL_LAPTOP)
#include <linux/dell-led.h>
static int dell_led_value;
-static int (*dell_led_set_func)(int, int);
+static int (*dell_micmute_led_set_func)(int);
static void (*dell_old_cap_hook)(struct hda_codec *,
struct snd_kcontrol *,
struct snd_ctl_elem_value *);
@@ -18,7 +18,7 @@
if (dell_old_cap_hook)
dell_old_cap_hook(codec, kcontrol, ucontrol);
- if (!ucontrol || !dell_led_set_func)
+ if (!ucontrol || !dell_micmute_led_set_func)
return;
if (strcmp("Capture Switch", ucontrol->id.name) == 0 && ucontrol->id.index == 0) {
/* TODO: How do I verify if it's a mono or stereo here? */
@@ -26,8 +26,8 @@
if (val == dell_led_value)
return;
dell_led_value = val;
- if (dell_led_set_func)
- dell_led_set_func(DELL_LED_MICMUTE, dell_led_value);
+ if (dell_micmute_led_set_func)
+ dell_micmute_led_set_func(dell_led_value);
}
}
@@ -39,15 +39,15 @@
bool removefunc = false;
if (action == HDA_FIXUP_ACT_PROBE) {
- if (!dell_led_set_func)
- dell_led_set_func = symbol_request(dell_app_wmi_led_set);
- if (!dell_led_set_func) {
- codec_warn(codec, "Failed to find dell wmi symbol dell_app_wmi_led_set\n");
+ if (!dell_micmute_led_set_func)
+ dell_micmute_led_set_func = symbol_request(dell_micmute_led_set);
+ if (!dell_micmute_led_set_func) {
+ codec_warn(codec, "Failed to find dell wmi symbol dell_micmute_led_set\n");
return;
}
removefunc = true;
- if (dell_led_set_func(DELL_LED_MICMUTE, false) >= 0) {
+ if (dell_micmute_led_set_func(false) >= 0) {
dell_led_value = 0;
if (spec->gen.num_adc_nids > 1 && !spec->gen.dyn_adc_switch)
codec_dbg(codec, "Skipping micmute LED control due to several ADCs");
@@ -60,17 +60,17 @@
}
- if (dell_led_set_func && (action == HDA_FIXUP_ACT_FREE || removefunc)) {
- symbol_put(dell_app_wmi_led_set);
- dell_led_set_func = NULL;
+ if (dell_micmute_led_set_func && (action == HDA_FIXUP_ACT_FREE || removefunc)) {
+ symbol_put(dell_micmute_led_set);
+ dell_micmute_led_set_func = NULL;
dell_old_cap_hook = NULL;
}
}
-#else /* CONFIG_LEDS_DELL_NETBOOKS */
+#else /* CONFIG_DELL_LAPTOP */
static void alc_fixup_dell_wmi(struct hda_codec *codec,
const struct hda_fixup *fix, int action)
{
}
-#endif /* CONFIG_LEDS_DELL_NETBOOKS */
+#endif /* CONFIG_DELL_LAPTOP */
diff --git a/sound/soc/intel/boards/bytcr_rt5640.c b/sound/soc/intel/boards/bytcr_rt5640.c
index 5c7219f..9e2a340 100644
--- a/sound/soc/intel/boards/bytcr_rt5640.c
+++ b/sound/soc/intel/boards/bytcr_rt5640.c
@@ -621,7 +621,7 @@
.codec_dai_name = "snd-soc-dummy-dai",
.codec_name = "snd-soc-dummy",
.platform_name = "sst-mfld-platform",
- .ignore_suspend = 1,
+ .nonatomic = true,
.dynamic = 1,
.dpcm_playback = 1,
.dpcm_capture = 1,
@@ -634,7 +634,6 @@
.codec_dai_name = "snd-soc-dummy-dai",
.codec_name = "snd-soc-dummy",
.platform_name = "sst-mfld-platform",
- .ignore_suspend = 1,
.nonatomic = true,
.dynamic = 1,
.dpcm_playback = 1,
@@ -661,6 +660,7 @@
| SND_SOC_DAIFMT_CBS_CFS,
.be_hw_params_fixup = byt_rt5640_codec_fixup,
.ignore_suspend = 1,
+ .nonatomic = true,
.dpcm_playback = 1,
.dpcm_capture = 1,
.init = byt_rt5640_init,
diff --git a/sound/soc/intel/boards/bytcr_rt5651.c b/sound/soc/intel/boards/bytcr_rt5651.c
index 3186f01..8164bec6 100644
--- a/sound/soc/intel/boards/bytcr_rt5651.c
+++ b/sound/soc/intel/boards/bytcr_rt5651.c
@@ -235,7 +235,6 @@
.codec_dai_name = "snd-soc-dummy-dai",
.codec_name = "snd-soc-dummy",
.platform_name = "sst-mfld-platform",
- .ignore_suspend = 1,
.nonatomic = true,
.dynamic = 1,
.dpcm_playback = 1,
@@ -249,7 +248,6 @@
.codec_dai_name = "snd-soc-dummy-dai",
.codec_name = "snd-soc-dummy",
.platform_name = "sst-mfld-platform",
- .ignore_suspend = 1,
.nonatomic = true,
.dynamic = 1,
.dpcm_playback = 1,
diff --git a/sound/soc/soc-topology.c b/sound/soc/soc-topology.c
index 3e9b1c0..058bc99 100644
--- a/sound/soc/soc-topology.c
+++ b/sound/soc/soc-topology.c
@@ -933,6 +933,7 @@
}
}
+ se->texts = (const char * const *)se->dobj.control.dtexts;
return 0;
err:
diff --git a/sound/soc/sti/uniperif.h b/sound/soc/sti/uniperif.h
index d487dd2..cfcb0ea 100644
--- a/sound/soc/sti/uniperif.h
+++ b/sound/soc/sti/uniperif.h
@@ -1299,6 +1299,7 @@
int ver; /* IP version, used by register access macros */
struct regmap_field *clk_sel;
struct regmap_field *valid_sel;
+ spinlock_t irq_lock; /* use to prevent race condition with IRQ */
/* capabilities */
const struct snd_pcm_hardware *hw;
diff --git a/sound/soc/sti/uniperif_player.c b/sound/soc/sti/uniperif_player.c
index 60ae31a..d7e8dd4 100644
--- a/sound/soc/sti/uniperif_player.c
+++ b/sound/soc/sti/uniperif_player.c
@@ -65,10 +65,13 @@
unsigned int status;
unsigned int tmp;
- if (player->state == UNIPERIF_STATE_STOPPED) {
- /* Unexpected IRQ: do nothing */
- return IRQ_NONE;
- }
+ spin_lock(&player->irq_lock);
+ if (!player->substream)
+ goto irq_spin_unlock;
+
+ snd_pcm_stream_lock(player->substream);
+ if (player->state == UNIPERIF_STATE_STOPPED)
+ goto stream_unlock;
/* Get interrupt status & clear them immediately */
status = GET_UNIPERIF_ITS(player);
@@ -88,9 +91,7 @@
SET_UNIPERIF_ITM_BCLR_FIFO_ERROR(player);
/* Stop the player */
- snd_pcm_stream_lock(player->substream);
snd_pcm_stop(player->substream, SNDRV_PCM_STATE_XRUN);
- snd_pcm_stream_unlock(player->substream);
}
ret = IRQ_HANDLED;
@@ -104,9 +105,7 @@
SET_UNIPERIF_ITM_BCLR_DMA_ERROR(player);
/* Stop the player */
- snd_pcm_stream_lock(player->substream);
snd_pcm_stop(player->substream, SNDRV_PCM_STATE_XRUN);
- snd_pcm_stream_unlock(player->substream);
ret = IRQ_HANDLED;
}
@@ -116,7 +115,8 @@
if (!player->underflow_enabled) {
dev_err(player->dev,
"unexpected Underflow recovering\n");
- return -EPERM;
+ ret = -EPERM;
+ goto stream_unlock;
}
/* Read the underflow recovery duration */
tmp = GET_UNIPERIF_STATUS_1_UNDERFLOW_DURATION(player);
@@ -138,13 +138,16 @@
dev_err(player->dev, "Underflow recovery failed\n");
/* Stop the player */
- snd_pcm_stream_lock(player->substream);
snd_pcm_stop(player->substream, SNDRV_PCM_STATE_XRUN);
- snd_pcm_stream_unlock(player->substream);
ret = IRQ_HANDLED;
}
+stream_unlock:
+ snd_pcm_stream_unlock(player->substream);
+irq_spin_unlock:
+ spin_unlock(&player->irq_lock);
+
return ret;
}
@@ -588,6 +591,7 @@
struct sti_uniperiph_data *priv = snd_soc_dai_get_drvdata(dai);
struct uniperif *player = priv->dai_data.uni;
struct snd_aes_iec958 *iec958 = &player->stream_settings.iec958;
+ unsigned long flags;
mutex_lock(&player->ctrl_lock);
iec958->status[0] = ucontrol->value.iec958.status[0];
@@ -596,12 +600,14 @@
iec958->status[3] = ucontrol->value.iec958.status[3];
mutex_unlock(&player->ctrl_lock);
+ spin_lock_irqsave(&player->irq_lock, flags);
if (player->substream && player->substream->runtime)
uni_player_set_channel_status(player,
player->substream->runtime);
else
uni_player_set_channel_status(player, NULL);
+ spin_unlock_irqrestore(&player->irq_lock, flags);
return 0;
}
@@ -686,9 +692,12 @@
{
struct sti_uniperiph_data *priv = snd_soc_dai_get_drvdata(dai);
struct uniperif *player = priv->dai_data.uni;
+ unsigned long flags;
int ret;
+ spin_lock_irqsave(&player->irq_lock, flags);
player->substream = substream;
+ spin_unlock_irqrestore(&player->irq_lock, flags);
player->clk_adj = 0;
@@ -986,12 +995,15 @@
{
struct sti_uniperiph_data *priv = snd_soc_dai_get_drvdata(dai);
struct uniperif *player = priv->dai_data.uni;
+ unsigned long flags;
+ spin_lock_irqsave(&player->irq_lock, flags);
if (player->state != UNIPERIF_STATE_STOPPED)
/* Stop the player */
uni_player_stop(player);
player->substream = NULL;
+ spin_unlock_irqrestore(&player->irq_lock, flags);
}
static int uni_player_parse_dt_audio_glue(struct platform_device *pdev,
@@ -1096,6 +1108,7 @@
}
mutex_init(&player->ctrl_lock);
+ spin_lock_init(&player->irq_lock);
/* Ensure that disabled by default */
SET_UNIPERIF_CONFIG_BACK_STALL_REQ_DISABLE(player);
diff --git a/sound/soc/sti/uniperif_reader.c b/sound/soc/sti/uniperif_reader.c
index 93a8df6..ee0055e 100644
--- a/sound/soc/sti/uniperif_reader.c
+++ b/sound/soc/sti/uniperif_reader.c
@@ -46,10 +46,15 @@
struct uniperif *reader = dev_id;
unsigned int status;
+ spin_lock(&reader->irq_lock);
+ if (!reader->substream)
+ goto irq_spin_unlock;
+
+ snd_pcm_stream_lock(reader->substream);
if (reader->state == UNIPERIF_STATE_STOPPED) {
/* Unexpected IRQ: do nothing */
dev_warn(reader->dev, "unexpected IRQ\n");
- return IRQ_HANDLED;
+ goto stream_unlock;
}
/* Get interrupt status & clear them immediately */
@@ -60,13 +65,16 @@
if (unlikely(status & UNIPERIF_ITS_FIFO_ERROR_MASK(reader))) {
dev_err(reader->dev, "FIFO error detected\n");
- snd_pcm_stream_lock(reader->substream);
snd_pcm_stop(reader->substream, SNDRV_PCM_STATE_XRUN);
- snd_pcm_stream_unlock(reader->substream);
- return IRQ_HANDLED;
+ ret = IRQ_HANDLED;
}
+stream_unlock:
+ snd_pcm_stream_unlock(reader->substream);
+irq_spin_unlock:
+ spin_unlock(&reader->irq_lock);
+
return ret;
}
@@ -347,9 +355,12 @@
{
struct sti_uniperiph_data *priv = snd_soc_dai_get_drvdata(dai);
struct uniperif *reader = priv->dai_data.uni;
+ unsigned long flags;
int ret;
+ spin_lock_irqsave(&reader->irq_lock, flags);
reader->substream = substream;
+ spin_unlock_irqrestore(&reader->irq_lock, flags);
if (!UNIPERIF_TYPE_IS_TDM(reader))
return 0;
@@ -375,12 +386,15 @@
{
struct sti_uniperiph_data *priv = snd_soc_dai_get_drvdata(dai);
struct uniperif *reader = priv->dai_data.uni;
+ unsigned long flags;
+ spin_lock_irqsave(&reader->irq_lock, flags);
if (reader->state != UNIPERIF_STATE_STOPPED) {
/* Stop the reader */
uni_reader_stop(reader);
}
reader->substream = NULL;
+ spin_unlock_irqrestore(&reader->irq_lock, flags);
}
static const struct snd_soc_dai_ops uni_reader_dai_ops = {
@@ -415,6 +429,8 @@
return -EBUSY;
}
+ spin_lock_init(&reader->irq_lock);
+
return 0;
}
EXPORT_SYMBOL_GPL(uni_reader_init);
diff --git a/tools/arch/arm/include/uapi/asm/kvm.h b/tools/arch/arm/include/uapi/asm/kvm.h
index af05f8e..6ebd3e6 100644
--- a/tools/arch/arm/include/uapi/asm/kvm.h
+++ b/tools/arch/arm/include/uapi/asm/kvm.h
@@ -181,10 +181,23 @@
#define KVM_DEV_ARM_VGIC_GRP_CPU_REGS 2
#define KVM_DEV_ARM_VGIC_CPUID_SHIFT 32
#define KVM_DEV_ARM_VGIC_CPUID_MASK (0xffULL << KVM_DEV_ARM_VGIC_CPUID_SHIFT)
+#define KVM_DEV_ARM_VGIC_V3_MPIDR_SHIFT 32
+#define KVM_DEV_ARM_VGIC_V3_MPIDR_MASK \
+ (0xffffffffULL << KVM_DEV_ARM_VGIC_V3_MPIDR_SHIFT)
#define KVM_DEV_ARM_VGIC_OFFSET_SHIFT 0
#define KVM_DEV_ARM_VGIC_OFFSET_MASK (0xffffffffULL << KVM_DEV_ARM_VGIC_OFFSET_SHIFT)
+#define KVM_DEV_ARM_VGIC_SYSREG_INSTR_MASK (0xffff)
#define KVM_DEV_ARM_VGIC_GRP_NR_IRQS 3
#define KVM_DEV_ARM_VGIC_GRP_CTRL 4
+#define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
+#define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
+#define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7
+#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10
+#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
+ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
+#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK 0x3ff
+#define VGIC_LEVEL_INFO_LINE_LEVEL 0
+
#define KVM_DEV_ARM_VGIC_CTRL_INIT 0
/* KVM_IRQ_LINE irq field index values */
diff --git a/tools/arch/arm64/include/uapi/asm/kvm.h b/tools/arch/arm64/include/uapi/asm/kvm.h
index 3051f86..c286035 100644
--- a/tools/arch/arm64/include/uapi/asm/kvm.h
+++ b/tools/arch/arm64/include/uapi/asm/kvm.h
@@ -201,10 +201,23 @@
#define KVM_DEV_ARM_VGIC_GRP_CPU_REGS 2
#define KVM_DEV_ARM_VGIC_CPUID_SHIFT 32
#define KVM_DEV_ARM_VGIC_CPUID_MASK (0xffULL << KVM_DEV_ARM_VGIC_CPUID_SHIFT)
+#define KVM_DEV_ARM_VGIC_V3_MPIDR_SHIFT 32
+#define KVM_DEV_ARM_VGIC_V3_MPIDR_MASK \
+ (0xffffffffULL << KVM_DEV_ARM_VGIC_V3_MPIDR_SHIFT)
#define KVM_DEV_ARM_VGIC_OFFSET_SHIFT 0
#define KVM_DEV_ARM_VGIC_OFFSET_MASK (0xffffffffULL << KVM_DEV_ARM_VGIC_OFFSET_SHIFT)
+#define KVM_DEV_ARM_VGIC_SYSREG_INSTR_MASK (0xffff)
#define KVM_DEV_ARM_VGIC_GRP_NR_IRQS 3
#define KVM_DEV_ARM_VGIC_GRP_CTRL 4
+#define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
+#define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
+#define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7
+#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10
+#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
+ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
+#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK 0x3ff
+#define VGIC_LEVEL_INFO_LINE_LEVEL 0
+
#define KVM_DEV_ARM_VGIC_CTRL_INIT 0
/* Device Control API on vcpu fd */
diff --git a/tools/arch/powerpc/include/uapi/asm/kvm.h b/tools/arch/powerpc/include/uapi/asm/kvm.h
index 3603b6f..4edbe4b 100644
--- a/tools/arch/powerpc/include/uapi/asm/kvm.h
+++ b/tools/arch/powerpc/include/uapi/asm/kvm.h
@@ -413,6 +413,26 @@
__u16 n_invalid;
};
+/* For KVM_PPC_CONFIGURE_V3_MMU */
+struct kvm_ppc_mmuv3_cfg {
+ __u64 flags;
+ __u64 process_table; /* second doubleword of partition table entry */
+};
+
+/* Flag values for KVM_PPC_CONFIGURE_V3_MMU */
+#define KVM_PPC_MMUV3_RADIX 1 /* 1 = radix mode, 0 = HPT */
+#define KVM_PPC_MMUV3_GTSE 2 /* global translation shootdown enb. */
+
+/* For KVM_PPC_GET_RMMU_INFO */
+struct kvm_ppc_rmmu_info {
+ struct kvm_ppc_radix_geom {
+ __u8 page_shift;
+ __u8 level_bits[4];
+ __u8 pad[3];
+ } geometries[8];
+ __u32 ap_encodings[8];
+};
+
/* Per-vcpu XICS interrupt controller state */
#define KVM_REG_PPC_ICP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U64 | 0x8c)
@@ -613,5 +633,7 @@
#define KVM_XICS_LEVEL_SENSITIVE (1ULL << 40)
#define KVM_XICS_MASKED (1ULL << 41)
#define KVM_XICS_PENDING (1ULL << 42)
+#define KVM_XICS_PRESENTED (1ULL << 43)
+#define KVM_XICS_QUEUED (1ULL << 44)
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/tools/arch/x86/include/asm/atomic.h b/tools/arch/x86/include/asm/atomic.h
index 059e33e..328eece 100644
--- a/tools/arch/x86/include/asm/atomic.h
+++ b/tools/arch/x86/include/asm/atomic.h
@@ -7,6 +7,8 @@
#define LOCK_PREFIX "\n\tlock; "
+#include <asm/cmpxchg.h>
+
/*
* Atomic operations that C can't guarantee us. Useful for
* resource counting etc..
@@ -62,4 +64,9 @@
GEN_UNARY_RMWcc(LOCK_PREFIX "decl", v->counter, "%0", "e");
}
+static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new)
+{
+ return cmpxchg(&v->counter, old, new);
+}
+
#endif /* _TOOLS_LINUX_ASM_X86_ATOMIC_H */
diff --git a/tools/arch/x86/include/asm/cmpxchg.h b/tools/arch/x86/include/asm/cmpxchg.h
new file mode 100644
index 0000000..f525326
--- /dev/null
+++ b/tools/arch/x86/include/asm/cmpxchg.h
@@ -0,0 +1,89 @@
+#ifndef TOOLS_ASM_X86_CMPXCHG_H
+#define TOOLS_ASM_X86_CMPXCHG_H
+
+#include <linux/compiler.h>
+
+/*
+ * Non-existant functions to indicate usage errors at link time
+ * (or compile-time if the compiler implements __compiletime_error().
+ */
+extern void __cmpxchg_wrong_size(void)
+ __compiletime_error("Bad argument size for cmpxchg");
+
+/*
+ * Constants for operation sizes. On 32-bit, the 64-bit size it set to
+ * -1 because sizeof will never return -1, thereby making those switch
+ * case statements guaranteeed dead code which the compiler will
+ * eliminate, and allowing the "missing symbol in the default case" to
+ * indicate a usage error.
+ */
+#define __X86_CASE_B 1
+#define __X86_CASE_W 2
+#define __X86_CASE_L 4
+#ifdef __x86_64__
+#define __X86_CASE_Q 8
+#else
+#define __X86_CASE_Q -1 /* sizeof will never return -1 */
+#endif
+
+/*
+ * Atomic compare and exchange. Compare OLD with MEM, if identical,
+ * store NEW in MEM. Return the initial value in MEM. Success is
+ * indicated by comparing RETURN with OLD.
+ */
+#define __raw_cmpxchg(ptr, old, new, size, lock) \
+({ \
+ __typeof__(*(ptr)) __ret; \
+ __typeof__(*(ptr)) __old = (old); \
+ __typeof__(*(ptr)) __new = (new); \
+ switch (size) { \
+ case __X86_CASE_B: \
+ { \
+ volatile u8 *__ptr = (volatile u8 *)(ptr); \
+ asm volatile(lock "cmpxchgb %2,%1" \
+ : "=a" (__ret), "+m" (*__ptr) \
+ : "q" (__new), "0" (__old) \
+ : "memory"); \
+ break; \
+ } \
+ case __X86_CASE_W: \
+ { \
+ volatile u16 *__ptr = (volatile u16 *)(ptr); \
+ asm volatile(lock "cmpxchgw %2,%1" \
+ : "=a" (__ret), "+m" (*__ptr) \
+ : "r" (__new), "0" (__old) \
+ : "memory"); \
+ break; \
+ } \
+ case __X86_CASE_L: \
+ { \
+ volatile u32 *__ptr = (volatile u32 *)(ptr); \
+ asm volatile(lock "cmpxchgl %2,%1" \
+ : "=a" (__ret), "+m" (*__ptr) \
+ : "r" (__new), "0" (__old) \
+ : "memory"); \
+ break; \
+ } \
+ case __X86_CASE_Q: \
+ { \
+ volatile u64 *__ptr = (volatile u64 *)(ptr); \
+ asm volatile(lock "cmpxchgq %2,%1" \
+ : "=a" (__ret), "+m" (*__ptr) \
+ : "r" (__new), "0" (__old) \
+ : "memory"); \
+ break; \
+ } \
+ default: \
+ __cmpxchg_wrong_size(); \
+ } \
+ __ret; \
+})
+
+#define __cmpxchg(ptr, old, new, size) \
+ __raw_cmpxchg((ptr), (old), (new), (size), LOCK_PREFIX)
+
+#define cmpxchg(ptr, old, new) \
+ __cmpxchg(ptr, old, new, sizeof(*(ptr)))
+
+
+#endif /* TOOLS_ASM_X86_CMPXCHG_H */
diff --git a/tools/arch/x86/include/asm/cpufeatures.h b/tools/arch/x86/include/asm/cpufeatures.h
index 293149a..0fe0044 100644
--- a/tools/arch/x86/include/asm/cpufeatures.h
+++ b/tools/arch/x86/include/asm/cpufeatures.h
@@ -100,7 +100,7 @@
#define X86_FEATURE_XTOPOLOGY ( 3*32+22) /* cpu topology enum extensions */
#define X86_FEATURE_TSC_RELIABLE ( 3*32+23) /* TSC is known to be reliable */
#define X86_FEATURE_NONSTOP_TSC ( 3*32+24) /* TSC does not stop in C states */
-/* free, was #define X86_FEATURE_CLFLUSH_MONITOR ( 3*32+25) * "" clflush reqd with monitor */
+#define X86_FEATURE_CPUID ( 3*32+25) /* CPU has CPUID instruction itself */
#define X86_FEATURE_EXTD_APICID ( 3*32+26) /* has extended APICID (8 bits) */
#define X86_FEATURE_AMD_DCM ( 3*32+27) /* multi-node processor */
#define X86_FEATURE_APERFMPERF ( 3*32+28) /* APERFMPERF */
@@ -186,7 +186,8 @@
*
* Reuse free bits when adding new feature flags!
*/
-
+#define X86_FEATURE_RING3MWAIT ( 7*32+ 0) /* Ring 3 MONITOR/MWAIT */
+#define X86_FEATURE_CPUID_FAULT ( 7*32+ 1) /* Intel CPUID faulting */
#define X86_FEATURE_CPB ( 7*32+ 2) /* AMD Core Performance Boost */
#define X86_FEATURE_EPB ( 7*32+ 3) /* IA32_ENERGY_PERF_BIAS support */
#define X86_FEATURE_CAT_L3 ( 7*32+ 4) /* Cache Allocation Technology L3 */
@@ -289,7 +290,8 @@
#define X86_FEATURE_PKU (16*32+ 3) /* Protection Keys for Userspace */
#define X86_FEATURE_OSPKE (16*32+ 4) /* OS Protection Keys Enable */
#define X86_FEATURE_AVX512_VPOPCNTDQ (16*32+14) /* POPCNT for vectors of DW/QW */
-#define X86_FEATURE_RDPID (16*32+ 22) /* RDPID instruction */
+#define X86_FEATURE_LA57 (16*32+16) /* 5-level page tables */
+#define X86_FEATURE_RDPID (16*32+22) /* RDPID instruction */
/* AMD-defined CPU features, CPUID level 0x80000007 (ebx), word 17 */
#define X86_FEATURE_OVERFLOW_RECOV (17*32+0) /* MCA overflow recovery support */
@@ -321,5 +323,4 @@
#define X86_BUG_SWAPGS_FENCE X86_BUG(11) /* SWAPGS without input dep on GS */
#define X86_BUG_MONITOR X86_BUG(12) /* IPI required to wake up remote CPU */
#define X86_BUG_AMD_E400 X86_BUG(13) /* CPU is among the affected by Erratum 400 */
-
#endif /* _ASM_X86_CPUFEATURES_H */
diff --git a/tools/arch/x86/lib/memcpy_64.S b/tools/arch/x86/lib/memcpy_64.S
index 49e6eba..98dcc11 100644
--- a/tools/arch/x86/lib/memcpy_64.S
+++ b/tools/arch/x86/lib/memcpy_64.S
@@ -286,7 +286,7 @@
_ASM_EXTABLE_FAULT(.L_copy_leading_bytes, .L_memcpy_mcsafe_fail)
_ASM_EXTABLE_FAULT(.L_cache_w0, .L_memcpy_mcsafe_fail)
_ASM_EXTABLE_FAULT(.L_cache_w1, .L_memcpy_mcsafe_fail)
- _ASM_EXTABLE_FAULT(.L_cache_w3, .L_memcpy_mcsafe_fail)
+ _ASM_EXTABLE_FAULT(.L_cache_w2, .L_memcpy_mcsafe_fail)
_ASM_EXTABLE_FAULT(.L_cache_w3, .L_memcpy_mcsafe_fail)
_ASM_EXTABLE_FAULT(.L_cache_w4, .L_memcpy_mcsafe_fail)
_ASM_EXTABLE_FAULT(.L_cache_w5, .L_memcpy_mcsafe_fail)
diff --git a/tools/build/Makefile.feature b/tools/build/Makefile.feature
index e3fb5ec..523911f 100644
--- a/tools/build/Makefile.feature
+++ b/tools/build/Makefile.feature
@@ -63,6 +63,7 @@
lzma \
get_cpuid \
bpf \
+ sched_getcpu \
sdt
# FEATURE_TESTS_BASIC + FEATURE_TESTS_EXTRA is the complete list
diff --git a/tools/build/feature/Makefile b/tools/build/feature/Makefile
index b564a2e..e35e4e5 100644
--- a/tools/build/feature/Makefile
+++ b/tools/build/feature/Makefile
@@ -48,21 +48,22 @@
test-get_cpuid.bin \
test-sdt.bin \
test-cxx.bin \
- test-jvmti.bin
+ test-jvmti.bin \
+ test-sched_getcpu.bin
FILES := $(addprefix $(OUTPUT),$(FILES))
-CC := $(CROSS_COMPILE)gcc -MD
-CXX := $(CROSS_COMPILE)g++ -MD
-PKG_CONFIG := $(CROSS_COMPILE)pkg-config
+CC ?= $(CROSS_COMPILE)gcc
+CXX ?= $(CROSS_COMPILE)g++
+PKG_CONFIG ?= $(CROSS_COMPILE)pkg-config
LLVM_CONFIG ?= llvm-config
all: $(FILES)
-__BUILD = $(CC) $(CFLAGS) -Wall -Werror -o $@ $(patsubst %.bin,%.c,$(@F)) $(LDFLAGS)
+__BUILD = $(CC) $(CFLAGS) -MD -Wall -Werror -o $@ $(patsubst %.bin,%.c,$(@F)) $(LDFLAGS)
BUILD = $(__BUILD) > $(@:.bin=.make.output) 2>&1
-__BUILDXX = $(CXX) $(CXXFLAGS) -Wall -Werror -o $@ $(patsubst %.bin,%.cpp,$(@F)) $(LDFLAGS)
+__BUILDXX = $(CXX) $(CXXFLAGS) -MD -Wall -Werror -o $@ $(patsubst %.bin,%.cpp,$(@F)) $(LDFLAGS)
BUILDXX = $(__BUILDXX) > $(@:.bin=.make.output) 2>&1
###############################
@@ -91,6 +92,9 @@
$(OUTPUT)test-glibc.bin:
$(BUILD)
+$(OUTPUT)test-sched_getcpu.bin:
+ $(BUILD)
+
DWARFLIBS := -ldw
ifeq ($(findstring -static,${LDFLAGS}),-static)
DWARFLIBS += -lelf -lebl -lz -llzma -lbz2
@@ -171,7 +175,7 @@
$(BUILD) $(FLAGS_PERL_EMBED)
$(OUTPUT)test-libpython.bin:
- $(BUILD)
+ $(BUILD) $(FLAGS_PYTHON_EMBED)
$(OUTPUT)test-libpython-version.bin:
$(BUILD)
diff --git a/tools/build/feature/test-all.c b/tools/build/feature/test-all.c
index 699e436..cc6c7c0 100644
--- a/tools/build/feature/test-all.c
+++ b/tools/build/feature/test-all.c
@@ -117,6 +117,10 @@
# include "test-pthread-attr-setaffinity-np.c"
#undef main
+#define main main_test_sched_getcpu
+# include "test-sched_getcpu.c"
+#undef main
+
# if 0
/*
* Disable libbabeltrace check for test-all, because the requested
@@ -182,6 +186,7 @@
main_test_get_cpuid();
main_test_bpf();
main_test_libcrypto();
+ main_test_sched_getcpu();
main_test_sdt();
return 0;
diff --git a/tools/build/feature/test-bpf.c b/tools/build/feature/test-bpf.c
index e04ab89..ebc6dce 100644
--- a/tools/build/feature/test-bpf.c
+++ b/tools/build/feature/test-bpf.c
@@ -9,6 +9,9 @@
# define __NR_bpf 321
# elif defined(__aarch64__)
# define __NR_bpf 280
+# elif defined(__sparc__)
+# define __NR_bpf 349
+# else
# error __NR_bpf not defined. libbpf does not support your arch.
# endif
#endif
diff --git a/tools/build/feature/test-sched_getcpu.c b/tools/build/feature/test-sched_getcpu.c
new file mode 100644
index 0000000..c4a148d
--- /dev/null
+++ b/tools/build/feature/test-sched_getcpu.c
@@ -0,0 +1,7 @@
+#define _GNU_SOURCE
+#include <sched.h>
+
+int main(void)
+{
+ return sched_getcpu();
+}
diff --git a/tools/hv/bondvf.sh b/tools/hv/bondvf.sh
index 4aa5369..d85968c 100755
--- a/tools/hv/bondvf.sh
+++ b/tools/hv/bondvf.sh
@@ -101,9 +101,25 @@
echo BONDING_OPTS=\"mode=active-backup miimon=100 primary=$2\" >>$fn
}
+function del_eth_cfg_ubuntu {
+ local fn=$cfgdir/interfaces
+ local tmpfl=$(mktemp)
+
+ local nic_start='^[ \t]*(auto|iface|mapping|allow-.*)[ \t]+'$1
+ local nic_end='^[ \t]*(auto|iface|mapping|allow-.*|source)'
+
+ awk "/$nic_end/{x=0} x{next} /$nic_start/{x=1;next} 1" $fn >$tmpfl
+
+ cp $tmpfl $fn
+
+ rm $tmpfl
+}
+
function create_eth_cfg_ubuntu {
local fn=$cfgdir/interfaces
+ del_eth_cfg_ubuntu $1
+
echo $'\n'auto $1 >>$fn
echo iface $1 inet manual >>$fn
echo bond-master $2 >>$fn
@@ -119,6 +135,8 @@
function create_bond_cfg_ubuntu {
local fn=$cfgdir/interfaces
+ del_eth_cfg_ubuntu $1
+
echo $'\n'auto $1 >>$fn
echo iface $1 inet dhcp >>$fn
echo bond-mode active-backup >>$fn
diff --git a/tools/include/asm-generic/atomic-gcc.h b/tools/include/asm-generic/atomic-gcc.h
index 2ba78c9..5e9738f 100644
--- a/tools/include/asm-generic/atomic-gcc.h
+++ b/tools/include/asm-generic/atomic-gcc.h
@@ -60,4 +60,12 @@
return __sync_sub_and_fetch(&v->counter, 1) == 0;
}
+#define cmpxchg(ptr, oldval, newval) \
+ __sync_val_compare_and_swap(ptr, oldval, newval)
+
+static inline int atomic_cmpxchg(atomic_t *v, int oldval, int newval)
+{
+ return cmpxchg(&(v)->counter, oldval, newval);
+}
+
#endif /* __TOOLS_ASM_GENERIC_ATOMIC_H */
diff --git a/tools/include/linux/atomic.h b/tools/include/linux/atomic.h
index 4e3d3d1..9f21fc2 100644
--- a/tools/include/linux/atomic.h
+++ b/tools/include/linux/atomic.h
@@ -3,4 +3,10 @@
#include <asm/atomic.h>
+/* atomic_cmpxchg_relaxed */
+#ifndef atomic_cmpxchg_relaxed
+#define atomic_cmpxchg_relaxed atomic_cmpxchg
+#define atomic_cmpxchg_release atomic_cmpxchg
+#endif /* atomic_cmpxchg_relaxed */
+
#endif /* __TOOLS_LINUX_ATOMIC_H */
diff --git a/tools/include/linux/bug.h b/tools/include/linux/bug.h
new file mode 100644
index 0000000..8e4a4f4
--- /dev/null
+++ b/tools/include/linux/bug.h
@@ -0,0 +1,10 @@
+#ifndef _TOOLS_PERF_LINUX_BUG_H
+#define _TOOLS_PERF_LINUX_BUG_H
+
+/* Force a compilation error if condition is true, but also produce a
+ result (of value 0 and type size_t), so the expression can be used
+ e.g. in a structure initializer (or where-ever else comma expressions
+ aren't permitted). */
+#define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:-!!(e); }))
+
+#endif /* _TOOLS_PERF_LINUX_BUG_H */
diff --git a/tools/include/linux/compiler-gcc.h b/tools/include/linux/compiler-gcc.h
index 48af2f1..825d44f 100644
--- a/tools/include/linux/compiler-gcc.h
+++ b/tools/include/linux/compiler-gcc.h
@@ -12,3 +12,10 @@
#if GCC_VERSION >= 70000 && !defined(__CHECKER__)
# define __fallthrough __attribute__ ((fallthrough))
#endif
+
+#if GCC_VERSION >= 40300
+# define __compiletime_error(message) __attribute__((error(message)))
+#endif /* GCC_VERSION >= 40300 */
+
+/* &a[0] degrades to a pointer: a different type from an array */
+#define __must_be_array(a) BUILD_BUG_ON_ZERO(__same_type((a), &(a)[0]))
diff --git a/tools/include/linux/compiler.h b/tools/include/linux/compiler.h
index 8de163b..23299d7 100644
--- a/tools/include/linux/compiler.h
+++ b/tools/include/linux/compiler.h
@@ -5,6 +5,10 @@
#include <linux/compiler-gcc.h>
#endif
+#ifndef __compiletime_error
+# define __compiletime_error(message)
+#endif
+
/* Optimization barrier */
/* The "volatile" is due to gcc bugs */
#define barrier() __asm__ __volatile__("": : :"memory")
@@ -13,6 +17,11 @@
# define __always_inline inline __attribute__((always_inline))
#endif
+/* Are two types/vars the same type (ignoring qualifiers)? */
+#ifndef __same_type
+# define __same_type(a, b) __builtin_types_compatible_p(typeof(a), typeof(b))
+#endif
+
#ifdef __ANDROID__
/*
* FIXME: Big hammer to get rid of tons of:
diff --git a/tools/include/linux/hashtable.h b/tools/include/linux/hashtable.h
index c65cc0a..251eabf 100644
--- a/tools/include/linux/hashtable.h
+++ b/tools/include/linux/hashtable.h
@@ -13,10 +13,6 @@
#include <linux/hash.h>
#include <linux/log2.h>
-#ifndef ARRAY_SIZE
-#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
-#endif
-
#define DEFINE_HASHTABLE(name, bits) \
struct hlist_head name[1 << (bits)] = \
{ [0 ... ((1 << (bits)) - 1)] = HLIST_HEAD_INIT }
diff --git a/tools/include/linux/kernel.h b/tools/include/linux/kernel.h
index 28607db..73ccc48 100644
--- a/tools/include/linux/kernel.h
+++ b/tools/include/linux/kernel.h
@@ -4,6 +4,11 @@
#include <stdarg.h>
#include <stddef.h>
#include <assert.h>
+#include <linux/compiler.h>
+
+#ifndef UINT_MAX
+#define UINT_MAX (~0U)
+#endif
#define DIV_ROUND_UP(n,d) (((n) + (d) - 1) / (d))
@@ -72,6 +77,8 @@
int vscnprintf(char *buf, size_t size, const char *fmt, va_list args);
int scnprintf(char * buf, size_t size, const char * fmt, ...);
+#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0]) + __must_be_array(arr))
+
/*
* This looks more complex than it should be. But we need to
* get the type for the ~ right in round_down (it needs to be
diff --git a/tools/include/linux/log2.h b/tools/include/linux/log2.h
index d5677d3..0325cef 100644
--- a/tools/include/linux/log2.h
+++ b/tools/include/linux/log2.h
@@ -12,6 +12,9 @@
#ifndef _TOOLS_LINUX_LOG2_H
#define _TOOLS_LINUX_LOG2_H
+#include <linux/bitops.h>
+#include <linux/types.h>
+
/*
* non-constant log of base 2 calculators
* - the arch may override these in asm/bitops.h if they can be implemented
diff --git a/tools/include/linux/refcount.h b/tools/include/linux/refcount.h
new file mode 100644
index 0000000..a0177c1
--- /dev/null
+++ b/tools/include/linux/refcount.h
@@ -0,0 +1,151 @@
+#ifndef _TOOLS_LINUX_REFCOUNT_H
+#define _TOOLS_LINUX_REFCOUNT_H
+
+/*
+ * Variant of atomic_t specialized for reference counts.
+ *
+ * The interface matches the atomic_t interface (to aid in porting) but only
+ * provides the few functions one should use for reference counting.
+ *
+ * It differs in that the counter saturates at UINT_MAX and will not move once
+ * there. This avoids wrapping the counter and causing 'spurious'
+ * use-after-free issues.
+ *
+ * Memory ordering rules are slightly relaxed wrt regular atomic_t functions
+ * and provide only what is strictly required for refcounts.
+ *
+ * The increments are fully relaxed; these will not provide ordering. The
+ * rationale is that whatever is used to obtain the object we're increasing the
+ * reference count on will provide the ordering. For locked data structures,
+ * its the lock acquire, for RCU/lockless data structures its the dependent
+ * load.
+ *
+ * Do note that inc_not_zero() provides a control dependency which will order
+ * future stores against the inc, this ensures we'll never modify the object
+ * if we did not in fact acquire a reference.
+ *
+ * The decrements will provide release order, such that all the prior loads and
+ * stores will be issued before, it also provides a control dependency, which
+ * will order us against the subsequent free().
+ *
+ * The control dependency is against the load of the cmpxchg (ll/sc) that
+ * succeeded. This means the stores aren't fully ordered, but this is fine
+ * because the 1->0 transition indicates no concurrency.
+ *
+ * Note that the allocator is responsible for ordering things between free()
+ * and alloc().
+ *
+ */
+
+#include <linux/atomic.h>
+#include <linux/kernel.h>
+
+#ifdef NDEBUG
+#define REFCOUNT_WARN(cond, str) (void)(cond)
+#define __refcount_check
+#else
+#define REFCOUNT_WARN(cond, str) BUG_ON(cond)
+#define __refcount_check __must_check
+#endif
+
+typedef struct refcount_struct {
+ atomic_t refs;
+} refcount_t;
+
+#define REFCOUNT_INIT(n) { .refs = ATOMIC_INIT(n), }
+
+static inline void refcount_set(refcount_t *r, unsigned int n)
+{
+ atomic_set(&r->refs, n);
+}
+
+static inline unsigned int refcount_read(const refcount_t *r)
+{
+ return atomic_read(&r->refs);
+}
+
+/*
+ * Similar to atomic_inc_not_zero(), will saturate at UINT_MAX and WARN.
+ *
+ * Provides no memory ordering, it is assumed the caller has guaranteed the
+ * object memory to be stable (RCU, etc.). It does provide a control dependency
+ * and thereby orders future stores. See the comment on top.
+ */
+static inline __refcount_check
+bool refcount_inc_not_zero(refcount_t *r)
+{
+ unsigned int old, new, val = atomic_read(&r->refs);
+
+ for (;;) {
+ new = val + 1;
+
+ if (!val)
+ return false;
+
+ if (unlikely(!new))
+ return true;
+
+ old = atomic_cmpxchg_relaxed(&r->refs, val, new);
+ if (old == val)
+ break;
+
+ val = old;
+ }
+
+ REFCOUNT_WARN(new == UINT_MAX, "refcount_t: saturated; leaking memory.\n");
+
+ return true;
+}
+
+/*
+ * Similar to atomic_inc(), will saturate at UINT_MAX and WARN.
+ *
+ * Provides no memory ordering, it is assumed the caller already has a
+ * reference on the object, will WARN when this is not so.
+ */
+static inline void refcount_inc(refcount_t *r)
+{
+ REFCOUNT_WARN(!refcount_inc_not_zero(r), "refcount_t: increment on 0; use-after-free.\n");
+}
+
+/*
+ * Similar to atomic_dec_and_test(), it will WARN on underflow and fail to
+ * decrement when saturated at UINT_MAX.
+ *
+ * Provides release memory ordering, such that prior loads and stores are done
+ * before, and provides a control dependency such that free() must come after.
+ * See the comment on top.
+ */
+static inline __refcount_check
+bool refcount_sub_and_test(unsigned int i, refcount_t *r)
+{
+ unsigned int old, new, val = atomic_read(&r->refs);
+
+ for (;;) {
+ if (unlikely(val == UINT_MAX))
+ return false;
+
+ new = val - i;
+ if (new > val) {
+ REFCOUNT_WARN(new > val, "refcount_t: underflow; use-after-free.\n");
+ return false;
+ }
+
+ old = atomic_cmpxchg_release(&r->refs, val, new);
+ if (old == val)
+ break;
+
+ val = old;
+ }
+
+ return !new;
+}
+
+static inline __refcount_check
+bool refcount_dec_and_test(refcount_t *r)
+{
+ return refcount_sub_and_test(1, r);
+}
+
+
+#endif /* _ATOMIC_LINUX_REFCOUNT_H */
diff --git a/tools/include/linux/types.h b/tools/include/linux/types.h
index c24b3e3..77a28a2 100644
--- a/tools/include/linux/types.h
+++ b/tools/include/linux/types.h
@@ -7,6 +7,7 @@
#define __SANE_USERSPACE_TYPES__ /* For PPC64, to get LL64 types */
#include <asm/types.h>
+#include <asm/posix_types.h>
struct page;
struct kmem_cache;
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 0539a0c..e553529 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -81,6 +81,7 @@
BPF_OBJ_GET,
BPF_PROG_ATTACH,
BPF_PROG_DETACH,
+ BPF_PROG_TEST_RUN,
};
enum bpf_map_type {
@@ -96,6 +97,8 @@
BPF_MAP_TYPE_LRU_HASH,
BPF_MAP_TYPE_LRU_PERCPU_HASH,
BPF_MAP_TYPE_LPM_TRIE,
+ BPF_MAP_TYPE_ARRAY_OF_MAPS,
+ BPF_MAP_TYPE_HASH_OF_MAPS,
};
enum bpf_prog_type {
@@ -152,6 +155,7 @@
__u32 value_size; /* size of value in bytes */
__u32 max_entries; /* max number of entries in a map */
__u32 map_flags; /* prealloc or not */
+ __u32 inner_map_fd; /* fd pointing to the inner map */
};
struct { /* anonymous struct used by BPF_MAP_*_ELEM commands */
@@ -186,6 +190,17 @@
__u32 attach_type;
__u32 attach_flags;
};
+
+ struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */
+ __u32 prog_fd;
+ __u32 retval;
+ __u32 data_size_in;
+ __u32 data_size_out;
+ __aligned_u64 data_in;
+ __aligned_u64 data_out;
+ __u32 repeat;
+ __u32 duration;
+ } test;
} __attribute__((aligned(8)));
/* BPF helper function descriptions:
@@ -456,6 +471,18 @@
* Return:
* > 0 length of the string including the trailing NUL on success
* < 0 error
+ *
+ * u64 bpf_get_socket_cookie(skb)
+ * Get the cookie for the socket stored inside sk_buff.
+ * @skb: pointer to skb
+ * Return: 8 Bytes non-decreasing number on success or 0 if the socket
+ * field is missing inside sk_buff
+ *
+ * u32 bpf_get_socket_uid(skb)
+ * Get the owner uid of the socket stored inside sk_buff.
+ * @skb: pointer to skb
+ * Return: uid of the socket owner on success or 0 if the socket pointer
+ * inside sk_buff is NULL
*/
#define __BPF_FUNC_MAPPER(FN) \
FN(unspec), \
@@ -503,7 +530,9 @@
FN(get_numa_node_id), \
FN(skb_change_head), \
FN(xdp_adjust_head), \
- FN(probe_read_str),
+ FN(probe_read_str), \
+ FN(get_socket_cookie), \
+ FN(get_socket_uid),
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
* function eBPF program intends to call
@@ -574,6 +603,7 @@
__u32 tc_classid;
__u32 data;
__u32 data_end;
+ __u32 napi_id;
};
struct bpf_tunnel_key {
diff --git a/tools/include/uapi/linux/fcntl.h b/tools/include/uapi/linux/fcntl.h
new file mode 100644
index 0000000..813afd6
--- /dev/null
+++ b/tools/include/uapi/linux/fcntl.h
@@ -0,0 +1,72 @@
+#ifndef _UAPI_LINUX_FCNTL_H
+#define _UAPI_LINUX_FCNTL_H
+
+#include <asm/fcntl.h>
+
+#define F_SETLEASE (F_LINUX_SPECIFIC_BASE + 0)
+#define F_GETLEASE (F_LINUX_SPECIFIC_BASE + 1)
+
+/*
+ * Cancel a blocking posix lock; internal use only until we expose an
+ * asynchronous lock api to userspace:
+ */
+#define F_CANCELLK (F_LINUX_SPECIFIC_BASE + 5)
+
+/* Create a file descriptor with FD_CLOEXEC set. */
+#define F_DUPFD_CLOEXEC (F_LINUX_SPECIFIC_BASE + 6)
+
+/*
+ * Request nofications on a directory.
+ * See below for events that may be notified.
+ */
+#define F_NOTIFY (F_LINUX_SPECIFIC_BASE+2)
+
+/*
+ * Set and get of pipe page size array
+ */
+#define F_SETPIPE_SZ (F_LINUX_SPECIFIC_BASE + 7)
+#define F_GETPIPE_SZ (F_LINUX_SPECIFIC_BASE + 8)
+
+/*
+ * Set/Get seals
+ */
+#define F_ADD_SEALS (F_LINUX_SPECIFIC_BASE + 9)
+#define F_GET_SEALS (F_LINUX_SPECIFIC_BASE + 10)
+
+/*
+ * Types of seals
+ */
+#define F_SEAL_SEAL 0x0001 /* prevent further seals from being set */
+#define F_SEAL_SHRINK 0x0002 /* prevent file from shrinking */
+#define F_SEAL_GROW 0x0004 /* prevent file from growing */
+#define F_SEAL_WRITE 0x0008 /* prevent writes */
+/* (1U << 31) is reserved for signed error codes */
+
+/*
+ * Types of directory notifications that may be requested.
+ */
+#define DN_ACCESS 0x00000001 /* File accessed */
+#define DN_MODIFY 0x00000002 /* File modified */
+#define DN_CREATE 0x00000004 /* File created */
+#define DN_DELETE 0x00000008 /* File removed */
+#define DN_RENAME 0x00000010 /* File renamed */
+#define DN_ATTRIB 0x00000020 /* File changed attibutes */
+#define DN_MULTISHOT 0x80000000 /* Don't remove notifier */
+
+#define AT_FDCWD -100 /* Special value used to indicate
+ openat should use the current
+ working directory. */
+#define AT_SYMLINK_NOFOLLOW 0x100 /* Do not follow symbolic links. */
+#define AT_REMOVEDIR 0x200 /* Remove directory instead of
+ unlinking file. */
+#define AT_SYMLINK_FOLLOW 0x400 /* Follow symbolic links. */
+#define AT_NO_AUTOMOUNT 0x800 /* Suppress terminal automount traversal */
+#define AT_EMPTY_PATH 0x1000 /* Allow empty relative pathname */
+
+#define AT_STATX_SYNC_TYPE 0x6000 /* Type of synchronisation required from statx() */
+#define AT_STATX_SYNC_AS_STAT 0x0000 /* - Do whatever stat() does */
+#define AT_STATX_FORCE_SYNC 0x2000 /* - Force the attributes to be sync'd with the server */
+#define AT_STATX_DONT_SYNC 0x4000 /* - Don't sync attributes with the server */
+
+
+#endif /* _UAPI_LINUX_FCNTL_H */
diff --git a/tools/include/uapi/linux/perf_event.h b/tools/include/uapi/linux/perf_event.h
index c66a485..d09a9cd 100644
--- a/tools/include/uapi/linux/perf_event.h
+++ b/tools/include/uapi/linux/perf_event.h
@@ -344,7 +344,8 @@
use_clockid : 1, /* use @clockid for time fields */
context_switch : 1, /* context switch data */
write_backward : 1, /* Write ring buffer from end to beginning */
- __reserved_1 : 36;
+ namespaces : 1, /* include namespaces data */
+ __reserved_1 : 35;
union {
__u32 wakeup_events; /* wakeup every n events */
@@ -610,6 +611,23 @@
__u16 size;
};
+struct perf_ns_link_info {
+ __u64 dev;
+ __u64 ino;
+};
+
+enum {
+ NET_NS_INDEX = 0,
+ UTS_NS_INDEX = 1,
+ IPC_NS_INDEX = 2,
+ PID_NS_INDEX = 3,
+ USER_NS_INDEX = 4,
+ MNT_NS_INDEX = 5,
+ CGROUP_NS_INDEX = 6,
+
+ NR_NAMESPACES, /* number of available namespaces */
+};
+
enum perf_event_type {
/*
@@ -862,6 +880,18 @@
*/
PERF_RECORD_SWITCH_CPU_WIDE = 15,
+ /*
+ * struct {
+ * struct perf_event_header header;
+ * u32 pid;
+ * u32 tid;
+ * u64 nr_namespaces;
+ * { u64 dev, inode; } [nr_namespaces];
+ * struct sample_id sample_id;
+ * };
+ */
+ PERF_RECORD_NAMESPACES = 16,
+
PERF_RECORD_MAX, /* non-ABI */
};
@@ -885,6 +915,7 @@
*/
#define PERF_AUX_FLAG_TRUNCATED 0x01 /* record was truncated to fit */
#define PERF_AUX_FLAG_OVERWRITE 0x02 /* snapshot from overwrite mode */
+#define PERF_AUX_FLAG_PARTIAL 0x04 /* record contains gaps */
#define PERF_FLAG_FD_NO_GROUP (1UL << 0)
#define PERF_FLAG_FD_OUTPUT (1UL << 1)
diff --git a/tools/include/uapi/linux/stat.h b/tools/include/uapi/linux/stat.h
new file mode 100644
index 0000000..d538897
--- /dev/null
+++ b/tools/include/uapi/linux/stat.h
@@ -0,0 +1,177 @@
+#ifndef _UAPI_LINUX_STAT_H
+#define _UAPI_LINUX_STAT_H
+
+#include <linux/types.h>
+
+#if defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ < 2)
+
+#define S_IFMT 00170000
+#define S_IFSOCK 0140000
+#define S_IFLNK 0120000
+#define S_IFREG 0100000
+#define S_IFBLK 0060000
+#define S_IFDIR 0040000
+#define S_IFCHR 0020000
+#define S_IFIFO 0010000
+#define S_ISUID 0004000
+#define S_ISGID 0002000
+#define S_ISVTX 0001000
+
+#define S_ISLNK(m) (((m) & S_IFMT) == S_IFLNK)
+#define S_ISREG(m) (((m) & S_IFMT) == S_IFREG)
+#define S_ISDIR(m) (((m) & S_IFMT) == S_IFDIR)
+#define S_ISCHR(m) (((m) & S_IFMT) == S_IFCHR)
+#define S_ISBLK(m) (((m) & S_IFMT) == S_IFBLK)
+#define S_ISFIFO(m) (((m) & S_IFMT) == S_IFIFO)
+#define S_ISSOCK(m) (((m) & S_IFMT) == S_IFSOCK)
+
+#define S_IRWXU 00700
+#define S_IRUSR 00400
+#define S_IWUSR 00200
+#define S_IXUSR 00100
+
+#define S_IRWXG 00070
+#define S_IRGRP 00040
+#define S_IWGRP 00020
+#define S_IXGRP 00010
+
+#define S_IRWXO 00007
+#define S_IROTH 00004
+#define S_IWOTH 00002
+#define S_IXOTH 00001
+
+#endif
+
+/*
+ * Timestamp structure for the timestamps in struct statx.
+ *
+ * tv_sec holds the number of seconds before (negative) or after (positive)
+ * 00:00:00 1st January 1970 UTC.
+ *
+ * tv_nsec holds a number of nanoseconds before (0..-999,999,999 if tv_sec is
+ * negative) or after (0..999,999,999 if tv_sec is positive) the tv_sec time.
+ *
+ * Note that if both tv_sec and tv_nsec are non-zero, then the two values must
+ * either be both positive or both negative.
+ *
+ * __reserved is held in case we need a yet finer resolution.
+ */
+struct statx_timestamp {
+ __s64 tv_sec;
+ __s32 tv_nsec;
+ __s32 __reserved;
+};
+
+/*
+ * Structures for the extended file attribute retrieval system call
+ * (statx()).
+ *
+ * The caller passes a mask of what they're specifically interested in as a
+ * parameter to statx(). What statx() actually got will be indicated in
+ * st_mask upon return.
+ *
+ * For each bit in the mask argument:
+ *
+ * - if the datum is not supported:
+ *
+ * - the bit will be cleared, and
+ *
+ * - the datum will be set to an appropriate fabricated value if one is
+ * available (eg. CIFS can take a default uid and gid), otherwise
+ *
+ * - the field will be cleared;
+ *
+ * - otherwise, if explicitly requested:
+ *
+ * - the datum will be synchronised to the server if AT_STATX_FORCE_SYNC is
+ * set or if the datum is considered out of date, and
+ *
+ * - the field will be filled in and the bit will be set;
+ *
+ * - otherwise, if not requested, but available in approximate form without any
+ * effort, it will be filled in anyway, and the bit will be set upon return
+ * (it might not be up to date, however, and no attempt will be made to
+ * synchronise the internal state first);
+ *
+ * - otherwise the field and the bit will be cleared before returning.
+ *
+ * Items in STATX_BASIC_STATS may be marked unavailable on return, but they
+ * will have values installed for compatibility purposes so that stat() and
+ * co. can be emulated in userspace.
+ */
+struct statx {
+ /* 0x00 */
+ __u32 stx_mask; /* What results were written [uncond] */
+ __u32 stx_blksize; /* Preferred general I/O size [uncond] */
+ __u64 stx_attributes; /* Flags conveying information about the file [uncond] */
+ /* 0x10 */
+ __u32 stx_nlink; /* Number of hard links */
+ __u32 stx_uid; /* User ID of owner */
+ __u32 stx_gid; /* Group ID of owner */
+ __u16 stx_mode; /* File mode */
+ __u16 __spare0[1];
+ /* 0x20 */
+ __u64 stx_ino; /* Inode number */
+ __u64 stx_size; /* File size */
+ __u64 stx_blocks; /* Number of 512-byte blocks allocated */
+ __u64 stx_attributes_mask; /* Mask to show what's supported in stx_attributes */
+ /* 0x40 */
+ struct statx_timestamp stx_atime; /* Last access time */
+ struct statx_timestamp stx_btime; /* File creation time */
+ struct statx_timestamp stx_ctime; /* Last attribute change time */
+ struct statx_timestamp stx_mtime; /* Last data modification time */
+ /* 0x80 */
+ __u32 stx_rdev_major; /* Device ID of special file [if bdev/cdev] */
+ __u32 stx_rdev_minor;
+ __u32 stx_dev_major; /* ID of device containing file [uncond] */
+ __u32 stx_dev_minor;
+ /* 0x90 */
+ __u64 __spare2[14]; /* Spare space for future expansion */
+ /* 0x100 */
+};
+
+/*
+ * Flags to be stx_mask
+ *
+ * Query request/result mask for statx() and struct statx::stx_mask.
+ *
+ * These bits should be set in the mask argument of statx() to request
+ * particular items when calling statx().
+ */
+#define STATX_TYPE 0x00000001U /* Want/got stx_mode & S_IFMT */
+#define STATX_MODE 0x00000002U /* Want/got stx_mode & ~S_IFMT */
+#define STATX_NLINK 0x00000004U /* Want/got stx_nlink */
+#define STATX_UID 0x00000008U /* Want/got stx_uid */
+#define STATX_GID 0x00000010U /* Want/got stx_gid */
+#define STATX_ATIME 0x00000020U /* Want/got stx_atime */
+#define STATX_MTIME 0x00000040U /* Want/got stx_mtime */
+#define STATX_CTIME 0x00000080U /* Want/got stx_ctime */
+#define STATX_INO 0x00000100U /* Want/got stx_ino */
+#define STATX_SIZE 0x00000200U /* Want/got stx_size */
+#define STATX_BLOCKS 0x00000400U /* Want/got stx_blocks */
+#define STATX_BASIC_STATS 0x000007ffU /* The stuff in the normal stat struct */
+#define STATX_BTIME 0x00000800U /* Want/got stx_btime */
+#define STATX_ALL 0x00000fffU /* All currently supported flags */
+#define STATX__RESERVED 0x80000000U /* Reserved for future struct statx expansion */
+
+/*
+ * Attributes to be found in stx_attributes and masked in stx_attributes_mask.
+ *
+ * These give information about the features or the state of a file that might
+ * be of use to ordinary userspace programs such as GUIs or ls rather than
+ * specialised tools.
+ *
+ * Note that the flags marked [I] correspond to generic FS_IOC_FLAGS
+ * semantically. Where possible, the numerical value is picked to correspond
+ * also.
+ */
+#define STATX_ATTR_COMPRESSED 0x00000004 /* [I] File is compressed by the fs */
+#define STATX_ATTR_IMMUTABLE 0x00000010 /* [I] File is marked immutable */
+#define STATX_ATTR_APPEND 0x00000020 /* [I] File is append-only */
+#define STATX_ATTR_NODUMP 0x00000040 /* [I] File is not to be dumped */
+#define STATX_ATTR_ENCRYPTED 0x00000800 /* [I] File requires key to decrypt in fs */
+
+#define STATX_ATTR_AUTOMOUNT 0x00001000 /* Dir: Automount trigger */
+
+
+#endif /* _UAPI_LINUX_STAT_H */
diff --git a/tools/lguest/lguest.c b/tools/lguest/lguest.c
index 5d19fdf..897cd6f 100644
--- a/tools/lguest/lguest.c
+++ b/tools/lguest/lguest.c
@@ -3339,7 +3339,7 @@
* simple, single region.
*/
boot->e820_entries = 1;
- boot->e820_map[0] = ((struct e820entry) { 0, mem, E820_RAM });
+ boot->e820_table[0] = ((struct e820_entry) { 0, mem, E820_TYPE_RAM });
/*
* The boot header contains a command line pointer: we put the command
* line after the boot header.
diff --git a/tools/lib/api/fs/fs.c b/tools/lib/api/fs/fs.c
index 4b6bfc4..809c772 100644
--- a/tools/lib/api/fs/fs.c
+++ b/tools/lib/api/fs/fs.c
@@ -439,6 +439,35 @@
return filename__read_str(path, buf, sizep);
}
+int sysfs__read_bool(const char *entry, bool *value)
+{
+ char *buf;
+ size_t size;
+ int ret;
+
+ ret = sysfs__read_str(entry, &buf, &size);
+ if (ret < 0)
+ return ret;
+
+ switch (buf[0]) {
+ case '1':
+ case 'y':
+ case 'Y':
+ *value = true;
+ break;
+ case '0':
+ case 'n':
+ case 'N':
+ *value = false;
+ break;
+ default:
+ ret = -1;
+ }
+
+ free(buf);
+
+ return ret;
+}
int sysctl__read_int(const char *sysctl, int *value)
{
char path[PATH_MAX];
diff --git a/tools/lib/api/fs/fs.h b/tools/lib/api/fs/fs.h
index 6b332dc..956c211 100644
--- a/tools/lib/api/fs/fs.h
+++ b/tools/lib/api/fs/fs.h
@@ -37,4 +37,5 @@
int sysfs__read_int(const char *entry, int *value);
int sysfs__read_ull(const char *entry, unsigned long long *value);
int sysfs__read_str(const char *entry, char **buf, size_t *sizep);
+int sysfs__read_bool(const char *entry, bool *value);
#endif /* __API_FS__ */
diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c
index 207c2ee..4fe444b80 100644
--- a/tools/lib/bpf/bpf.c
+++ b/tools/lib/bpf/bpf.c
@@ -37,6 +37,8 @@
# define __NR_bpf 321
# elif defined(__aarch64__)
# define __NR_bpf 280
+# elif defined(__sparc__)
+# define __NR_bpf 349
# else
# error __NR_bpf not defined. libbpf does not support your arch.
# endif
@@ -69,6 +71,23 @@
return sys_bpf(BPF_MAP_CREATE, &attr, sizeof(attr));
}
+int bpf_create_map_in_map(enum bpf_map_type map_type, int key_size,
+ int inner_map_fd, int max_entries, __u32 map_flags)
+{
+ union bpf_attr attr;
+
+ memset(&attr, '\0', sizeof(attr));
+
+ attr.map_type = map_type;
+ attr.key_size = key_size;
+ attr.value_size = 4;
+ attr.inner_map_fd = inner_map_fd;
+ attr.max_entries = max_entries;
+ attr.map_flags = map_flags;
+
+ return sys_bpf(BPF_MAP_CREATE, &attr, sizeof(attr));
+}
+
int bpf_load_program(enum bpf_prog_type type, const struct bpf_insn *insns,
size_t insns_cnt, const char *license,
__u32 kern_version, char *log_buf, size_t log_buf_sz)
@@ -192,3 +211,27 @@
return sys_bpf(BPF_PROG_DETACH, &attr, sizeof(attr));
}
+
+int bpf_prog_test_run(int prog_fd, int repeat, void *data, __u32 size,
+ void *data_out, __u32 *size_out, __u32 *retval,
+ __u32 *duration)
+{
+ union bpf_attr attr;
+ int ret;
+
+ bzero(&attr, sizeof(attr));
+ attr.test.prog_fd = prog_fd;
+ attr.test.data_in = ptr_to_u64(data);
+ attr.test.data_out = ptr_to_u64(data_out);
+ attr.test.data_size_in = size;
+ attr.test.repeat = repeat;
+
+ ret = sys_bpf(BPF_PROG_TEST_RUN, &attr, sizeof(attr));
+ if (size_out)
+ *size_out = attr.test.data_size_out;
+ if (retval)
+ *retval = attr.test.retval;
+ if (duration)
+ *duration = attr.test.duration;
+ return ret;
+}
diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h
index 09c3dca..edb4dae 100644
--- a/tools/lib/bpf/bpf.h
+++ b/tools/lib/bpf/bpf.h
@@ -26,6 +26,8 @@
int bpf_create_map(enum bpf_map_type map_type, int key_size, int value_size,
int max_entries, __u32 map_flags);
+int bpf_create_map_in_map(enum bpf_map_type map_type, int key_size,
+ int inner_map_fd, int max_entries, __u32 map_flags);
/* Recommend log buffer size */
#define BPF_LOG_BUF_SIZE 65536
@@ -45,6 +47,8 @@
int bpf_prog_attach(int prog_fd, int attachable_fd, enum bpf_attach_type type,
unsigned int flags);
int bpf_prog_detach(int attachable_fd, enum bpf_attach_type type);
-
+int bpf_prog_test_run(int prog_fd, int repeat, void *data, __u32 size,
+ void *data_out, __u32 *size_out, __u32 *retval,
+ __u32 *duration);
#endif
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index ac6eb86..1a2c07e 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -1618,8 +1618,7 @@
return fd;
}
-static void bpf_program__set_type(struct bpf_program *prog,
- enum bpf_prog_type type)
+void bpf_program__set_type(struct bpf_program *prog, enum bpf_prog_type type)
{
prog->type = type;
}
diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
index b30394f..32c7252 100644
--- a/tools/lib/bpf/libbpf.h
+++ b/tools/lib/bpf/libbpf.h
@@ -25,6 +25,7 @@
#include <stdint.h>
#include <stdbool.h>
#include <sys/types.h> // for size_t
+#include <linux/bpf.h>
enum libbpf_errno {
__LIBBPF_ERRNO__START = 4000,
@@ -185,6 +186,7 @@
int bpf_program__set_sched_act(struct bpf_program *prog);
int bpf_program__set_xdp(struct bpf_program *prog);
int bpf_program__set_perf_event(struct bpf_program *prog);
+void bpf_program__set_type(struct bpf_program *prog, enum bpf_prog_type type);
bool bpf_program__is_socket_filter(struct bpf_program *prog);
bool bpf_program__is_tracepoint(struct bpf_program *prog);
diff --git a/tools/lib/subcmd/help.h b/tools/lib/subcmd/help.h
index e145a02..9bd4223 100644
--- a/tools/lib/subcmd/help.h
+++ b/tools/lib/subcmd/help.h
@@ -2,6 +2,7 @@
#define __SUBCMD_HELP_H
#include <sys/types.h>
+#include <stdio.h>
struct cmdnames {
size_t alloc;
diff --git a/tools/lib/symbol/kallsyms.c b/tools/lib/symbol/kallsyms.c
index 5e43107..d270ac0 100644
--- a/tools/lib/symbol/kallsyms.c
+++ b/tools/lib/symbol/kallsyms.c
@@ -1,3 +1,4 @@
+#include <ctype.h>
#include "symbol/kallsyms.h"
#include <stdio.h>
#include <stdlib.h>
diff --git a/tools/net/bpf_jit_disasm.c b/tools/net/bpf_jit_disasm.c
index 544b05a..ad572e6 100644
--- a/tools/net/bpf_jit_disasm.c
+++ b/tools/net/bpf_jit_disasm.c
@@ -229,6 +229,7 @@
{
printf("Usage: bpf_jit_disasm [...]\n");
printf(" -o Also display related opcodes (default: off).\n");
+ printf(" -O <file> Write binary image of code to file, don't disassemble to stdout.\n");
printf(" -f <file> Read last image dump from file or stdin (default: klog).\n");
printf(" -h Display this help.\n");
}
@@ -238,12 +239,19 @@
unsigned int len, klen, opt, opcodes = 0;
static uint8_t image[32768];
char *kbuff, *file = NULL;
+ char *ofile = NULL;
+ int ofd;
+ ssize_t nr;
+ uint8_t *pos;
- while ((opt = getopt(argc, argv, "of:")) != -1) {
+ while ((opt = getopt(argc, argv, "of:O:")) != -1) {
switch (opt) {
case 'o':
opcodes = 1;
break;
+ case 'O':
+ ofile = optarg;
+ break;
case 'f':
file = optarg;
break;
@@ -263,11 +271,35 @@
}
len = get_last_jit_image(kbuff, klen, image, sizeof(image));
- if (len > 0)
- get_asm_insns(image, len, opcodes);
- else
+ if (len <= 0) {
fprintf(stderr, "No JIT image found!\n");
+ goto done;
+ }
+ if (!ofile) {
+ get_asm_insns(image, len, opcodes);
+ goto done;
+ }
+ ofd = open(ofile, O_WRONLY | O_CREAT | O_TRUNC, DEFFILEMODE);
+ if (ofd < 0) {
+ fprintf(stderr, "Could not open file %s for writing: ", ofile);
+ perror(NULL);
+ goto done;
+ }
+ pos = image;
+ do {
+ nr = write(ofd, pos, len);
+ if (nr < 0) {
+ fprintf(stderr, "Could not write data to %s: ", ofile);
+ perror(NULL);
+ goto done;
+ }
+ len -= nr;
+ pos += nr;
+ } while (len);
+ close(ofd);
+
+done:
put_log_buff(kbuff);
return 0;
}
diff --git a/tools/objtool/builtin-check.c b/tools/objtool/builtin-check.c
index 066086d..282a603 100644
--- a/tools/objtool/builtin-check.c
+++ b/tools/objtool/builtin-check.c
@@ -36,8 +36,7 @@
#include "warn.h"
#include <linux/hashtable.h>
-
-#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
+#include <linux/kernel.h>
#define STATE_FP_SAVED 0x1
#define STATE_FP_SETUP 0x2
diff --git a/tools/objtool/objtool.c b/tools/objtool/objtool.c
index 46c326d..ecc5b1b 100644
--- a/tools/objtool/objtool.c
+++ b/tools/objtool/objtool.c
@@ -31,11 +31,10 @@
#include <stdlib.h>
#include <subcmd/exec-cmd.h>
#include <subcmd/pager.h>
+#include <linux/kernel.h>
#include "builtin.h"
-#define ARRAY_SIZE(x) (sizeof(x)/sizeof(x[0]))
-
struct cmd_struct {
const char *name;
int (*fn)(int, const char **);
diff --git a/tools/perf/.gitignore b/tools/perf/.gitignore
index 3db3db9..643cc4ba 100644
--- a/tools/perf/.gitignore
+++ b/tools/perf/.gitignore
@@ -31,3 +31,5 @@
.config-detected
util/intel-pt-decoder/inat-tables.c
arch/*/include/generated/
+pmu-events/pmu-events.c
+pmu-events/jevents
diff --git a/tools/perf/Build b/tools/perf/Build
index 9b79f8d..bd8eeb6 100644
--- a/tools/perf/Build
+++ b/tools/perf/Build
@@ -50,5 +50,6 @@
libperf-y += arch/
libperf-y += ui/
libperf-y += scripts/
+libperf-y += trace/beauty/
gtk-y += ui/gtk/
diff --git a/tools/perf/Documentation/perf-ftrace.txt b/tools/perf/Documentation/perf-ftrace.txt
index 2d96de6..6e6a8b2 100644
--- a/tools/perf/Documentation/perf-ftrace.txt
+++ b/tools/perf/Documentation/perf-ftrace.txt
@@ -30,6 +30,24 @@
--verbose=::
Verbosity level.
+-p::
+--pid=::
+ Trace on existing process id (comma separated list).
+
+-a::
+--all-cpus::
+ Force system-wide collection. Scripts run without a <command>
+ normally use -a by default, while scripts run with a <command>
+ normally don't - this option allows the latter to be run in
+ system-wide mode.
+
+-C::
+--cpu=::
+ Only trace for the list of CPUs provided. Multiple CPUs can
+ be provided as a comma separated list with no space like: 0,1.
+ Ranges of CPUs are specified with -: 0-2.
+ Default is to trace on all online CPUs.
+
SEE ALSO
--------
diff --git a/tools/perf/Documentation/perf-list.txt b/tools/perf/Documentation/perf-list.txt
index 41857cc..f709de5 100644
--- a/tools/perf/Documentation/perf-list.txt
+++ b/tools/perf/Documentation/perf-list.txt
@@ -8,7 +8,7 @@
SYNOPSIS
--------
[verse]
-'perf list' [--no-desc] [--long-desc] [hw|sw|cache|tracepoint|pmu|event_glob]
+'perf list' [--no-desc] [--long-desc] [hw|sw|cache|tracepoint|pmu|sdt|event_glob]
DESCRIPTION
-----------
@@ -24,6 +24,10 @@
--long-desc::
Print longer event descriptions.
+--details::
+Print how named events are resolved internally into perf events, and also
+any extra expressions computed by perf stat.
+
[[EVENT_MODIFIERS]]
EVENT MODIFIERS
@@ -240,6 +244,8 @@
. 'pmu' to print the kernel supplied PMU events.
+. 'sdt' to list all Statically Defined Tracepoint events.
+
. If none of the above is matched, it will apply the supplied glob to all
events, printing the ones that match.
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index b16003e..ea3789d 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -347,6 +347,9 @@
displayed with the weight and local_weight sort keys. This currently works for TSX
abort events and some memory events in precise mode on modern Intel CPUs.
+--namespaces::
+Record events of type PERF_RECORD_NAMESPACES.
+
--transaction::
Record transaction flags for transaction related events.
diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
index c04cc06..37a1759 100644
--- a/tools/perf/Documentation/perf-report.txt
+++ b/tools/perf/Documentation/perf-report.txt
@@ -72,7 +72,8 @@
--sort=::
Sort histogram entries by given key(s) - multiple keys can be specified
in CSV format. Following sort keys are available:
- pid, comm, dso, symbol, parent, cpu, socket, srcline, weight, local_weight.
+ pid, comm, dso, symbol, parent, cpu, socket, srcline, weight,
+ local_weight, cgroup_id.
Each key has following meaning:
@@ -80,6 +81,7 @@
- pid: command and tid of the task
- dso: name of library or module executed at the time of sample
- symbol: name of function executed at the time of sample
+ - symbol_size: size of function executed at the time of sample
- parent: name of function matched to the parent regex filter. Unmatched
entries are displayed as "[other]".
- cpu: cpu number the task ran at the time of sample
@@ -91,6 +93,7 @@
- weight: Event specific weight, e.g. memory latency or transaction
abort cost. This is the global weight.
- local_weight: Local weight version of the weight above.
+ - cgroup_id: ID derived from cgroup namespace device and inode numbers.
- transaction: Transaction abort flags.
- overhead: Overhead percentage of sample
- overhead_sys: Overhead percentage of sample running in system mode
@@ -172,6 +175,9 @@
By default, every sort keys not specified in -F will be appended
automatically.
+ If the keys starts with a prefix '+', then it will append the specified
+ field(s) to the default field order. For example: perf report -F +period,sample.
+
-p::
--parent=<regex>::
A regex filter to identify parent. The parent is a caller of this
@@ -229,6 +235,7 @@
sort_key can be:
- function: compare on functions (default)
- address: compare on individual code addresses
+ - srcline: compare on source filename and line number
branch can be:
- branch: include last branch information in callgraph when available.
@@ -424,6 +431,10 @@
--hierarchy::
Enable hierarchical output.
+--inline::
+ If a callgraph address belongs to an inlined function, the inline stack
+ will be printed. Each entry is function name or file/line.
+
include::callchain-overhead-calculation.txt[]
SEE ALSO
diff --git a/tools/perf/Documentation/perf-sched.txt b/tools/perf/Documentation/perf-sched.txt
index d33dedd..a092a24 100644
--- a/tools/perf/Documentation/perf-sched.txt
+++ b/tools/perf/Documentation/perf-sched.txt
@@ -132,6 +132,10 @@
--migrations::
Show migration events.
+-n::
+--next::
+ Show next task.
+
-I::
--idle-hist::
Show idle-related events only.
diff --git a/tools/perf/Documentation/perf-script.txt b/tools/perf/Documentation/perf-script.txt
index 4ed5f23..cb0eda3 100644
--- a/tools/perf/Documentation/perf-script.txt
+++ b/tools/perf/Documentation/perf-script.txt
@@ -116,7 +116,7 @@
--fields::
Comma separated list of fields to print. Options are:
comm, tid, pid, time, cpu, event, trace, ip, sym, dso, addr, symoff,
- srcline, period, iregs, brstack, brstacksym, flags, bpf-output,
+ srcline, period, iregs, brstack, brstacksym, flags, bpf-output, brstackinsn,
callindent, insn, insnlen. Field list can be prepended with the type, trace, sw or hw,
to indicate to which event type the field list applies.
e.g., -F sw:comm,tid,time,ip,sym and -F trace:time,cpu,trace
@@ -189,15 +189,20 @@
i.e., -F "" is not allowed.
The brstack output includes branch related information with raw addresses using the
- /v/v/v/v/ syntax in the following order:
+ /v/v/v/v/cycles syntax in the following order:
FROM: branch source instruction
TO : branch target instruction
M/P/-: M=branch target mispredicted or branch direction was mispredicted, P=target predicted or direction predicted, -=not supported
X/- : X=branch inside a transactional region, -=not in transaction region or not supported
A/- : A=TSX abort entry, -=not aborted region or not supported
+ cycles
The brstacksym is identical to brstack, except that the FROM and TO addresses are printed in a symbolic form if possible.
+ When brstackinsn is specified the full assembler sequences of branch sequences for each sample
+ is printed. This is the full execution path leading to the sample. This is only supported when the
+ sample was recorded with perf record -b or -j any.
+
-k::
--vmlinux=<file>::
vmlinux pathname
@@ -248,6 +253,9 @@
--show-mmap-events
Display mmap related events (e.g. MMAP, MMAP2).
+--show-namespace-events
+ Display namespace events i.e. events of type PERF_RECORD_NAMESPACES.
+
--show-switch-events
Display context switch events i.e. events of type PERF_RECORD_SWITCH or
PERF_RECORD_SWITCH_CPU_WIDE.
@@ -299,6 +307,10 @@
stop time is not given (i.e, time string is 'x.y,') then analysis goes
to end of file.
+--max-blocks::
+ Set the maximum number of program blocks to print with brstackasm for
+ each sample.
+
SEE ALSO
--------
linkperf:perf-record[1], linkperf:perf-script-perl[1],
diff --git a/tools/perf/Documentation/perf-stat.txt b/tools/perf/Documentation/perf-stat.txt
index aecf2a8..bd0e441 100644
--- a/tools/perf/Documentation/perf-stat.txt
+++ b/tools/perf/Documentation/perf-stat.txt
@@ -94,8 +94,7 @@
-A::
--no-aggr::
-Do not aggregate counts across all monitored CPUs in system-wide mode (-a).
-This option is only valid in system-wide mode.
+Do not aggregate counts across all monitored CPUs.
-n::
--null::
@@ -237,6 +236,9 @@
CPUs the workload runs on. If needed the CPUs can be forced using
taskset.
+--no-merge::
+Do not merge results from same PMUs.
+
EXAMPLES
--------
diff --git a/tools/perf/Documentation/perf-trace.txt b/tools/perf/Documentation/perf-trace.txt
index afd7286..c1e3288 100644
--- a/tools/perf/Documentation/perf-trace.txt
+++ b/tools/perf/Documentation/perf-trace.txt
@@ -123,7 +123,8 @@
major or all pagefaults. Default value is maj.
--syscalls::
- Trace system calls. This options is enabled by default.
+ Trace system calls. This options is enabled by default, disable with
+ --no-syscalls.
--call-graph [mode,type,min[,limit],order[,key][,branch]]::
Setup and enable call-graph (stack chain/backtrace) recording.
diff --git a/tools/perf/Documentation/perf.data-file-format.txt b/tools/perf/Documentation/perf.data-file-format.txt
index b664b18..fa2a913 100644
--- a/tools/perf/Documentation/perf.data-file-format.txt
+++ b/tools/perf/Documentation/perf.data-file-format.txt
@@ -11,8 +11,8 @@
When perf is writing to a pipe it uses a special version of the file
format that does not rely on seeking to adjust data offsets. This
-format is not described here. The pipe version can be converted to
-normal perf.data with perf inject.
+format is described in "Pipe-mode data" section. The pipe data version can be
+augmented with additional events using perf inject.
The file starts with a perf_header:
@@ -411,6 +411,21 @@
ids points to a array of uint64_t defining the ids for event attr attr.
+Pipe-mode data
+
+Pipe-mode avoid seeks in the file by removing the perf_file_section and flags
+from the struct perf_header. The trimmed header is:
+
+struct perf_pipe_file_header {
+ u64 magic;
+ u64 size;
+};
+
+The information about attrs, data, and event_types is instead in the
+synthesized events PERF_RECORD_ATTR, PERF_RECORD_HEADER_TRACING_DATA and
+PERF_RECORD_HEADER_EVENT_TYPE that are generated by perf record in pipe-mode.
+
+
References:
include/uapi/linux/perf_event.h
diff --git a/tools/perf/MANIFEST b/tools/perf/MANIFEST
index 8672f83..a29da46 100644
--- a/tools/perf/MANIFEST
+++ b/tools/perf/MANIFEST
@@ -12,6 +12,7 @@
tools/arch/sparc/include/asm/barrier_64.h
tools/arch/tile/include/asm/barrier.h
tools/arch/x86/include/asm/barrier.h
+tools/arch/x86/include/asm/cmpxchg.h
tools/arch/x86/include/asm/cpufeatures.h
tools/arch/x86/include/asm/disabled-features.h
tools/arch/x86/include/asm/required-features.h
@@ -63,6 +64,7 @@
tools/include/linux/compiler.h
tools/include/linux/compiler-gcc.h
tools/include/linux/coresight-pmu.h
+tools/include/linux/bug.h
tools/include/linux/filter.h
tools/include/linux/hash.h
tools/include/linux/kernel.h
@@ -72,12 +74,15 @@
tools/include/uapi/asm-generic/mman.h
tools/include/uapi/linux/bpf.h
tools/include/uapi/linux/bpf_common.h
+tools/include/uapi/linux/fcntl.h
tools/include/uapi/linux/hw_breakpoint.h
tools/include/uapi/linux/mman.h
tools/include/uapi/linux/perf_event.h
+tools/include/uapi/linux/stat.h
tools/include/linux/poison.h
tools/include/linux/rbtree.h
tools/include/linux/rbtree_augmented.h
+tools/include/linux/refcount.h
tools/include/linux/string.h
tools/include/linux/stringify.h
tools/include/linux/types.h
diff --git a/tools/perf/Makefile.config b/tools/perf/Makefile.config
index 27c9fbc..8354d04 100644
--- a/tools/perf/Makefile.config
+++ b/tools/perf/Makefile.config
@@ -170,13 +170,20 @@
override PYTHON_CONFIG := \
$(call get-executable-or-default,PYTHON_CONFIG,$(PYTHON2_CONFIG))
+grep-libs = $(filter -l%,$(1))
+strip-libs = $(filter-out -l%,$(1))
+
PYTHON_CONFIG_SQ := $(call shell-sq,$(PYTHON_CONFIG))
-PYTHON_EMBED_LDOPTS := $(shell $(PYTHON_CONFIG_SQ) --ldflags 2>/dev/null)
-PYTHON_EMBED_CCOPTS := $(shell $(PYTHON_CONFIG_SQ) --cflags 2>/dev/null)
-
-ifeq ($(CC), clang)
- PYTHON_EMBED_CCOPTS := $(filter-out -specs=%,$(PYTHON_EMBED_CCOPTS))
+ifdef PYTHON_CONFIG
+ PYTHON_EMBED_LDOPTS := $(shell $(PYTHON_CONFIG_SQ) --ldflags 2>/dev/null)
+ PYTHON_EMBED_LDFLAGS := $(call strip-libs,$(PYTHON_EMBED_LDOPTS))
+ PYTHON_EMBED_LIBADD := $(call grep-libs,$(PYTHON_EMBED_LDOPTS)) -lutil
+ PYTHON_EMBED_CCOPTS := $(shell $(PYTHON_CONFIG_SQ) --cflags 2>/dev/null)
+ ifeq ($(CC), clang)
+ PYTHON_EMBED_CCOPTS := $(filter-out -specs=%,$(PYTHON_EMBED_CCOPTS))
+ endif
+ FLAGS_PYTHON_EMBED := $(PYTHON_EMBED_CCOPTS) $(PYTHON_EMBED_LDOPTS)
endif
FEATURE_CHECK_CFLAGS-libpython := $(PYTHON_EMBED_CCOPTS)
@@ -267,6 +274,7 @@
NO_LIBUNWIND := 1
NO_LIBDW_DWARF_UNWIND := 1
NO_LIBBPF := 1
+ NO_JVMTI := 1
else
ifeq ($(feature-libelf), 0)
ifeq ($(feature-glibc), 1)
@@ -276,7 +284,7 @@
LIBC_SUPPORT := 1
endif
ifeq ($(LIBC_SUPPORT),1)
- msg := $(warning No libelf found, disables 'probe' tool and BPF support in 'perf record', please install libelf-dev, libelf-devel or elfutils-libelf-devel);
+ msg := $(warning No libelf found. Disables 'probe' tool, jvmti and BPF support in 'perf record'. Please install libelf-dev, libelf-devel or elfutils-libelf-devel);
NO_LIBELF := 1
NO_DWARF := 1
@@ -284,6 +292,7 @@
NO_LIBUNWIND := 1
NO_LIBDW_DWARF_UNWIND := 1
NO_LIBBPF := 1
+ NO_JVMTI := 1
else
ifneq ($(filter s% -static%,$(LDFLAGS),),)
msg := $(error No static glibc found, please install glibc-static);
@@ -317,6 +326,10 @@
NO_LIBDW_DWARF_UNWIND := 1
endif
+ifeq ($(feature-sched_getcpu), 1)
+ CFLAGS += -DHAVE_SCHED_GETCPU_SUPPORT
+endif
+
ifndef NO_LIBELF
CFLAGS += -DHAVE_LIBELF_SUPPORT
EXTLIBS += -lelf
@@ -550,8 +563,6 @@
endif
endif
-grep-libs = $(filter -l%,$(1))
-strip-libs = $(filter-out -l%,$(1))
ifdef NO_LIBPERL
CFLAGS += -DNO_LIBPERL
@@ -599,21 +610,9 @@
$(call disable-python,No 'python-config' tool was found: disables Python support - please install python-devel/python-dev)
else
- PYTHON_CONFIG_SQ := $(call shell-sq,$(PYTHON_CONFIG))
-
- PYTHON_EMBED_LDOPTS := $(shell $(PYTHON_CONFIG_SQ) --ldflags 2>/dev/null)
- PYTHON_EMBED_LDFLAGS := $(call strip-libs,$(PYTHON_EMBED_LDOPTS))
- PYTHON_EMBED_LIBADD := $(call grep-libs,$(PYTHON_EMBED_LDOPTS)) -lutil
- PYTHON_EMBED_CCOPTS := $(shell $(PYTHON_CONFIG_SQ) --cflags 2>/dev/null)
- ifeq ($(CC), clang)
- PYTHON_EMBED_CCOPTS := $(filter-out -specs=%,$(PYTHON_EMBED_CCOPTS))
- endif
- FLAGS_PYTHON_EMBED := $(PYTHON_EMBED_CCOPTS) $(PYTHON_EMBED_LDOPTS)
-
ifneq ($(feature-libpython), 1)
$(call disable-python,No 'Python.h' (for Python 2.x support) was found: disables Python support - please install python-devel/python-dev)
else
-
ifneq ($(feature-libpython-version), 1)
$(warning Python 3 is not yet supported; please set)
$(warning PYTHON and/or PYTHON_CONFIG appropriately.)
diff --git a/tools/perf/arch/arm/util/cs-etm.c b/tools/perf/arch/arm/util/cs-etm.c
index dfea6b6..29361d9 100644
--- a/tools/perf/arch/arm/util/cs-etm.c
+++ b/tools/perf/arch/arm/util/cs-etm.c
@@ -33,6 +33,7 @@
#include "../../util/cs-etm.h"
#include <stdlib.h>
+#include <sys/stat.h>
#define ENABLE_SINK_MAX 128
#define CS_BUS_DEVICE_PATH "/bus/coresight/devices/"
diff --git a/tools/perf/arch/arm/util/dwarf-regs.c b/tools/perf/arch/arm/util/dwarf-regs.c
index 33ec5b3..8bb176a 100644
--- a/tools/perf/arch/arm/util/dwarf-regs.c
+++ b/tools/perf/arch/arm/util/dwarf-regs.c
@@ -9,6 +9,7 @@
*/
#include <stddef.h>
+#include <linux/stringify.h>
#include <dwarf-regs.h>
struct pt_regs_dwarfnum {
@@ -16,10 +17,9 @@
unsigned int dwarfnum;
};
-#define STR(s) #s
#define REG_DWARFNUM_NAME(r, num) {.name = r, .dwarfnum = num}
#define GPR_DWARFNUM_NAME(num) \
- {.name = STR(%r##num), .dwarfnum = num}
+ {.name = __stringify(%r##num), .dwarfnum = num}
#define REG_DWARFNUM_END {.name = NULL, .dwarfnum = 0}
/*
diff --git a/tools/perf/arch/arm/util/unwind-libdw.c b/tools/perf/arch/arm/util/unwind-libdw.c
index b4176c6..bacfa00 100644
--- a/tools/perf/arch/arm/util/unwind-libdw.c
+++ b/tools/perf/arch/arm/util/unwind-libdw.c
@@ -1,6 +1,7 @@
#include <elfutils/libdwfl.h>
#include "../../util/unwind-libdw.h"
#include "../../util/perf_regs.h"
+#include "../../util/event.h"
bool libdw__arch_set_initial_registers(Dwfl_Thread *thread, void *arg)
{
diff --git a/tools/perf/arch/arm64/util/dwarf-regs.c b/tools/perf/arch/arm64/util/dwarf-regs.c
index 068b618..cd764a9 100644
--- a/tools/perf/arch/arm64/util/dwarf-regs.c
+++ b/tools/perf/arch/arm64/util/dwarf-regs.c
@@ -8,9 +8,12 @@
* published by the Free Software Foundation.
*/
+#include <errno.h>
#include <stddef.h>
+#include <string.h>
#include <dwarf-regs.h>
#include <linux/ptrace.h> /* for struct user_pt_regs */
+#include <linux/stringify.h>
#include "util.h"
struct pt_regs_dwarfnum {
@@ -20,7 +23,7 @@
#define REG_DWARFNUM_NAME(r, num) {.name = r, .dwarfnum = num}
#define GPR_DWARFNUM_NAME(num) \
- {.name = STR(%x##num), .dwarfnum = num}
+ {.name = __stringify(%x##num), .dwarfnum = num}
#define REG_DWARFNUM_END {.name = NULL, .dwarfnum = 0}
#define DWARFNUM2OFFSET(index) \
(index * sizeof((struct user_pt_regs *)0)->regs[0])
diff --git a/tools/perf/arch/arm64/util/unwind-libunwind.c b/tools/perf/arch/arm64/util/unwind-libunwind.c
index c116b71..b415dfd 100644
--- a/tools/perf/arch/arm64/util/unwind-libunwind.c
+++ b/tools/perf/arch/arm64/util/unwind-libunwind.c
@@ -1,6 +1,6 @@
+#include <errno.h>
#ifndef REMOTE_UNWIND_LIBUNWIND
-#include <errno.h>
#include <libunwind.h>
#include "perf_regs.h"
#include "../../util/unwind.h"
diff --git a/tools/perf/arch/common.c b/tools/perf/arch/common.c
index 886dd2a..837067f 100644
--- a/tools/perf/arch/common.c
+++ b/tools/perf/arch/common.c
@@ -4,6 +4,8 @@
#include "../util/util.h"
#include "../util/debug.h"
+#include "sane_ctype.h"
+
const char *const arm_triplets[] = {
"arm-eabi-",
"arm-linux-androideabi-",
diff --git a/tools/perf/arch/powerpc/util/dwarf-regs.c b/tools/perf/arch/powerpc/util/dwarf-regs.c
index 41bdf95..98ac870 100644
--- a/tools/perf/arch/powerpc/util/dwarf-regs.c
+++ b/tools/perf/arch/powerpc/util/dwarf-regs.c
@@ -15,6 +15,7 @@
#include <dwarf-regs.h>
#include <linux/ptrace.h>
#include <linux/kernel.h>
+#include <linux/stringify.h>
#include "util.h"
struct pt_regs_dwarfnum {
@@ -24,10 +25,10 @@
};
#define REG_DWARFNUM_NAME(r, num) \
- {.name = STR(%)STR(r), .dwarfnum = num, \
+ {.name = __stringify(%)__stringify(r), .dwarfnum = num, \
.ptregs_offset = offsetof(struct pt_regs, r)}
#define GPR_DWARFNUM_NAME(num) \
- {.name = STR(%gpr##num), .dwarfnum = num, \
+ {.name = __stringify(%gpr##num), .dwarfnum = num, \
.ptregs_offset = offsetof(struct pt_regs, gpr[num])}
#define REG_DWARFNUM_END {.name = NULL, .dwarfnum = 0, .ptregs_offset = 0}
diff --git a/tools/perf/arch/powerpc/util/kvm-stat.c b/tools/perf/arch/powerpc/util/kvm-stat.c
index 74eee30..249723f 100644
--- a/tools/perf/arch/powerpc/util/kvm-stat.c
+++ b/tools/perf/arch/powerpc/util/kvm-stat.c
@@ -1,3 +1,4 @@
+#include <errno.h>
#include "util/kvm-stat.h"
#include "util/parse-events.h"
#include "util/debug.h"
diff --git a/tools/perf/arch/powerpc/util/perf_regs.c b/tools/perf/arch/powerpc/util/perf_regs.c
index a3c3e1c..f860dc41 100644
--- a/tools/perf/arch/powerpc/util/perf_regs.c
+++ b/tools/perf/arch/powerpc/util/perf_regs.c
@@ -1,5 +1,11 @@
+#include <errno.h>
+#include <string.h>
+#include <regex.h>
+
#include "../../perf.h"
+#include "../../util/util.h"
#include "../../util/perf_regs.h"
+#include "../../util/debug.h"
const struct sample_reg sample_reg_masks[] = {
SMPL_REG(r0, PERF_REG_POWERPC_R0),
@@ -47,3 +53,109 @@
SMPL_REG(dsisr, PERF_REG_POWERPC_DSISR),
SMPL_REG_END
};
+
+/* REG or %rREG */
+#define SDT_OP_REGEX1 "^(%r)?([1-2]?[0-9]|3[0-1])$"
+
+/* -NUM(REG) or NUM(REG) or -NUM(%rREG) or NUM(%rREG) */
+#define SDT_OP_REGEX2 "^(\\-)?([0-9]+)\\((%r)?([1-2]?[0-9]|3[0-1])\\)$"
+
+static regex_t sdt_op_regex1, sdt_op_regex2;
+
+static int sdt_init_op_regex(void)
+{
+ static int initialized;
+ int ret = 0;
+
+ if (initialized)
+ return 0;
+
+ ret = regcomp(&sdt_op_regex1, SDT_OP_REGEX1, REG_EXTENDED);
+ if (ret)
+ goto error;
+
+ ret = regcomp(&sdt_op_regex2, SDT_OP_REGEX2, REG_EXTENDED);
+ if (ret)
+ goto free_regex1;
+
+ initialized = 1;
+ return 0;
+
+free_regex1:
+ regfree(&sdt_op_regex1);
+error:
+ pr_debug4("Regex compilation error.\n");
+ return ret;
+}
+
+/*
+ * Parse OP and convert it into uprobe format, which is, +/-NUM(%gprREG).
+ * Possible variants of OP are:
+ * Format Example
+ * -------------------------
+ * NUM(REG) 48(18)
+ * -NUM(REG) -48(18)
+ * NUM(%rREG) 48(%r18)
+ * -NUM(%rREG) -48(%r18)
+ * REG 18
+ * %rREG %r18
+ * iNUM i0
+ * i-NUM i-1
+ *
+ * SDT marker arguments on Powerpc uses %rREG form with -mregnames flag
+ * and REG form with -mno-regnames. Here REG is general purpose register,
+ * which is in 0 to 31 range.
+ */
+int arch_sdt_arg_parse_op(char *old_op, char **new_op)
+{
+ int ret, new_len;
+ regmatch_t rm[5];
+ char prefix;
+
+ /* Constant argument. Uprobe does not support it */
+ if (old_op[0] == 'i') {
+ pr_debug4("Skipping unsupported SDT argument: %s\n", old_op);
+ return SDT_ARG_SKIP;
+ }
+
+ ret = sdt_init_op_regex();
+ if (ret < 0)
+ return ret;
+
+ if (!regexec(&sdt_op_regex1, old_op, 3, rm, 0)) {
+ /* REG or %rREG --> %gprREG */
+
+ new_len = 5; /* % g p r NULL */
+ new_len += (int)(rm[2].rm_eo - rm[2].rm_so);
+
+ *new_op = zalloc(new_len);
+ if (!*new_op)
+ return -ENOMEM;
+
+ scnprintf(*new_op, new_len, "%%gpr%.*s",
+ (int)(rm[2].rm_eo - rm[2].rm_so), old_op + rm[2].rm_so);
+ } else if (!regexec(&sdt_op_regex2, old_op, 5, rm, 0)) {
+ /*
+ * -NUM(REG) or NUM(REG) or -NUM(%rREG) or NUM(%rREG) -->
+ * +/-NUM(%gprREG)
+ */
+ prefix = (rm[1].rm_so == -1) ? '+' : '-';
+
+ new_len = 8; /* +/- ( % g p r ) NULL */
+ new_len += (int)(rm[2].rm_eo - rm[2].rm_so);
+ new_len += (int)(rm[4].rm_eo - rm[4].rm_so);
+
+ *new_op = zalloc(new_len);
+ if (!*new_op)
+ return -ENOMEM;
+
+ scnprintf(*new_op, new_len, "%c%.*s(%%gpr%.*s)", prefix,
+ (int)(rm[2].rm_eo - rm[2].rm_so), old_op + rm[2].rm_so,
+ (int)(rm[4].rm_eo - rm[4].rm_so), old_op + rm[4].rm_so);
+ } else {
+ pr_debug4("Skipping unsupported SDT argument: %s\n", old_op);
+ return SDT_ARG_SKIP;
+ }
+
+ return SDT_ARG_VALID;
+}
diff --git a/tools/perf/arch/powerpc/util/sym-handling.c b/tools/perf/arch/powerpc/util/sym-handling.c
index 1030a6e..39dbe51 100644
--- a/tools/perf/arch/powerpc/util/sym-handling.c
+++ b/tools/perf/arch/powerpc/util/sym-handling.c
@@ -10,6 +10,7 @@
#include "symbol.h"
#include "map.h"
#include "probe-event.h"
+#include "probe-file.h"
#ifdef HAVE_LIBELF_SUPPORT
bool elf__needs_adjust_symbols(GElf_Ehdr ehdr)
@@ -79,13 +80,18 @@
* However, if the user specifies an offset, we fall back to using the
* GEP since all userspace applications (objdump/readelf) show function
* disassembly with offsets from the GEP.
- *
- * In addition, we shouldn't specify an offset for kretprobes.
*/
- if (pev->point.offset || (!pev->uprobes && pev->point.retprobe) ||
- !map || !sym)
+ if (pev->point.offset || !map || !sym)
return;
+ /* For kretprobes, add an offset only if the kernel supports it */
+ if (!pev->uprobes && pev->point.retprobe) {
+#ifdef HAVE_LIBELF_SUPPORT
+ if (!kretprobe_offset_is_supported())
+#endif
+ return;
+ }
+
lep_offset = PPC64_LOCAL_ENTRY_OFFSET(sym->arch_sym);
if (map->dso->symtab_type == DSO_BINARY_TYPE__KALLSYMS)
diff --git a/tools/perf/arch/s390/annotate/instructions.c b/tools/perf/arch/s390/annotate/instructions.c
new file mode 100644
index 0000000..745b4b1
--- /dev/null
+++ b/tools/perf/arch/s390/annotate/instructions.c
@@ -0,0 +1,30 @@
+static struct ins_ops *s390__associate_ins_ops(struct arch *arch, const char *name)
+{
+ struct ins_ops *ops = NULL;
+
+ /* catch all kind of jumps */
+ if (strchr(name, 'j') ||
+ !strncmp(name, "bct", 3) ||
+ !strncmp(name, "br", 2))
+ ops = &jump_ops;
+ /* override call/returns */
+ if (!strcmp(name, "bras") ||
+ !strcmp(name, "brasl") ||
+ !strcmp(name, "basr"))
+ ops = &call_ops;
+ if (!strcmp(name, "br"))
+ ops = &ret_ops;
+
+ arch__associate_ins_ops(arch, name, ops);
+ return ops;
+}
+
+static int s390__annotate_init(struct arch *arch)
+{
+ if (!arch->initialized) {
+ arch->initialized = true;
+ arch->associate_instruction_ops = s390__associate_ins_ops;
+ }
+
+ return 0;
+}
diff --git a/tools/perf/arch/s390/util/kvm-stat.c b/tools/perf/arch/s390/util/kvm-stat.c
index ed57df2..d233e2eb 100644
--- a/tools/perf/arch/s390/util/kvm-stat.c
+++ b/tools/perf/arch/s390/util/kvm-stat.c
@@ -9,6 +9,7 @@
* as published by the Free Software Foundation.
*/
+#include <errno.h>
#include "../../util/kvm-stat.h"
#include <asm/sie.h>
diff --git a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl b/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl
index e93ef0b..5aef183 100644
--- a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl
@@ -338,6 +338,7 @@
329 common pkey_mprotect sys_pkey_mprotect
330 common pkey_alloc sys_pkey_alloc
331 common pkey_free sys_pkey_free
+332 common statx sys_statx
#
# x32-specific system call numbers start at 512 to avoid cache impact
diff --git a/tools/perf/arch/x86/tests/intel-cqm.c b/tools/perf/arch/x86/tests/intel-cqm.c
index 7f064eb..f9713a7 100644
--- a/tools/perf/arch/x86/tests/intel-cqm.c
+++ b/tools/perf/arch/x86/tests/intel-cqm.c
@@ -6,7 +6,10 @@
#include "evsel.h"
#include "arch-tests.h"
+#include <signal.h>
#include <sys/mman.h>
+#include <sys/wait.h>
+#include <errno.h>
#include <string.h>
static pid_t spawn(void)
diff --git a/tools/perf/arch/x86/tests/perf-time-to-tsc.c b/tools/perf/arch/x86/tests/perf-time-to-tsc.c
index 5c76cc8..e3ae9cf 100644
--- a/tools/perf/arch/x86/tests/perf-time-to-tsc.c
+++ b/tools/perf/arch/x86/tests/perf-time-to-tsc.c
@@ -1,3 +1,5 @@
+#include <errno.h>
+#include <inttypes.h>
#include <stdio.h>
#include <unistd.h>
#include <linux/types.h>
diff --git a/tools/perf/arch/x86/util/auxtrace.c b/tools/perf/arch/x86/util/auxtrace.c
index cc1d865..6aa3f2a 100644
--- a/tools/perf/arch/x86/util/auxtrace.c
+++ b/tools/perf/arch/x86/util/auxtrace.c
@@ -13,6 +13,7 @@
*
*/
+#include <errno.h>
#include <stdbool.h>
#include "../../util/header.h"
diff --git a/tools/perf/arch/x86/util/intel-bts.c b/tools/perf/arch/x86/util/intel-bts.c
index 5132775..af2bce7 100644
--- a/tools/perf/arch/x86/util/intel-bts.c
+++ b/tools/perf/arch/x86/util/intel-bts.c
@@ -13,6 +13,7 @@
*
*/
+#include <errno.h>
#include <linux/kernel.h>
#include <linux/types.h>
#include <linux/bitops.h>
diff --git a/tools/perf/arch/x86/util/intel-pt.c b/tools/perf/arch/x86/util/intel-pt.c
index 90fa228..f630de0 100644
--- a/tools/perf/arch/x86/util/intel-pt.c
+++ b/tools/perf/arch/x86/util/intel-pt.c
@@ -13,6 +13,7 @@
*
*/
+#include <errno.h>
#include <stdbool.h>
#include <linux/kernel.h>
#include <linux/types.h>
diff --git a/tools/perf/arch/x86/util/kvm-stat.c b/tools/perf/arch/x86/util/kvm-stat.c
index b63d4be..bf817be 100644
--- a/tools/perf/arch/x86/util/kvm-stat.c
+++ b/tools/perf/arch/x86/util/kvm-stat.c
@@ -1,3 +1,4 @@
+#include <errno.h>
#include "../../util/kvm-stat.h"
#include <asm/svm.h>
#include <asm/vmx.h>
diff --git a/tools/perf/arch/x86/util/perf_regs.c b/tools/perf/arch/x86/util/perf_regs.c
index c5db14f..f95edeb 100644
--- a/tools/perf/arch/x86/util/perf_regs.c
+++ b/tools/perf/arch/x86/util/perf_regs.c
@@ -1,5 +1,11 @@
+#include <errno.h>
+#include <string.h>
+#include <regex.h>
+
#include "../../perf.h"
+#include "../../util/util.h"
#include "../../util/perf_regs.h"
+#include "../../util/debug.h"
const struct sample_reg sample_reg_masks[] = {
SMPL_REG(AX, PERF_REG_X86_AX),
@@ -26,3 +32,224 @@
#endif
SMPL_REG_END
};
+
+struct sdt_name_reg {
+ const char *sdt_name;
+ const char *uprobe_name;
+};
+#define SDT_NAME_REG(n, m) {.sdt_name = "%" #n, .uprobe_name = "%" #m}
+#define SDT_NAME_REG_END {.sdt_name = NULL, .uprobe_name = NULL}
+
+static const struct sdt_name_reg sdt_reg_tbl[] = {
+ SDT_NAME_REG(eax, ax),
+ SDT_NAME_REG(rax, ax),
+ SDT_NAME_REG(al, ax),
+ SDT_NAME_REG(ah, ax),
+ SDT_NAME_REG(ebx, bx),
+ SDT_NAME_REG(rbx, bx),
+ SDT_NAME_REG(bl, bx),
+ SDT_NAME_REG(bh, bx),
+ SDT_NAME_REG(ecx, cx),
+ SDT_NAME_REG(rcx, cx),
+ SDT_NAME_REG(cl, cx),
+ SDT_NAME_REG(ch, cx),
+ SDT_NAME_REG(edx, dx),
+ SDT_NAME_REG(rdx, dx),
+ SDT_NAME_REG(dl, dx),
+ SDT_NAME_REG(dh, dx),
+ SDT_NAME_REG(esi, si),
+ SDT_NAME_REG(rsi, si),
+ SDT_NAME_REG(sil, si),
+ SDT_NAME_REG(edi, di),
+ SDT_NAME_REG(rdi, di),
+ SDT_NAME_REG(dil, di),
+ SDT_NAME_REG(ebp, bp),
+ SDT_NAME_REG(rbp, bp),
+ SDT_NAME_REG(bpl, bp),
+ SDT_NAME_REG(rsp, sp),
+ SDT_NAME_REG(esp, sp),
+ SDT_NAME_REG(spl, sp),
+
+ /* rNN registers */
+ SDT_NAME_REG(r8b, r8),
+ SDT_NAME_REG(r8w, r8),
+ SDT_NAME_REG(r8d, r8),
+ SDT_NAME_REG(r9b, r9),
+ SDT_NAME_REG(r9w, r9),
+ SDT_NAME_REG(r9d, r9),
+ SDT_NAME_REG(r10b, r10),
+ SDT_NAME_REG(r10w, r10),
+ SDT_NAME_REG(r10d, r10),
+ SDT_NAME_REG(r11b, r11),
+ SDT_NAME_REG(r11w, r11),
+ SDT_NAME_REG(r11d, r11),
+ SDT_NAME_REG(r12b, r12),
+ SDT_NAME_REG(r12w, r12),
+ SDT_NAME_REG(r12d, r12),
+ SDT_NAME_REG(r13b, r13),
+ SDT_NAME_REG(r13w, r13),
+ SDT_NAME_REG(r13d, r13),
+ SDT_NAME_REG(r14b, r14),
+ SDT_NAME_REG(r14w, r14),
+ SDT_NAME_REG(r14d, r14),
+ SDT_NAME_REG(r15b, r15),
+ SDT_NAME_REG(r15w, r15),
+ SDT_NAME_REG(r15d, r15),
+ SDT_NAME_REG_END,
+};
+
+/*
+ * Perf only supports OP which is in +/-NUM(REG) form.
+ * Here plus-minus sign, NUM and parenthesis are optional,
+ * only REG is mandatory.
+ *
+ * SDT events also supports indirect addressing mode with a
+ * symbol as offset, scaled mode and constants in OP. But
+ * perf does not support them yet. Below are few examples.
+ *
+ * OP with scaled mode:
+ * (%rax,%rsi,8)
+ * 10(%ras,%rsi,8)
+ *
+ * OP with indirect addressing mode:
+ * check_action(%rip)
+ * mp_+52(%rip)
+ * 44+mp_(%rip)
+ *
+ * OP with constant values:
+ * $0
+ * $123
+ * $-1
+ */
+#define SDT_OP_REGEX "^([+\\-]?)([0-9]*)(\\(?)(%[a-z][a-z0-9]+)(\\)?)$"
+
+static regex_t sdt_op_regex;
+
+static int sdt_init_op_regex(void)
+{
+ static int initialized;
+ int ret = 0;
+
+ if (initialized)
+ return 0;
+
+ ret = regcomp(&sdt_op_regex, SDT_OP_REGEX, REG_EXTENDED);
+ if (ret < 0) {
+ pr_debug4("Regex compilation error.\n");
+ return ret;
+ }
+
+ initialized = 1;
+ return 0;
+}
+
+/*
+ * Max x86 register name length is 5(ex: %r15d). So, 6th char
+ * should always contain NULL. This helps to find register name
+ * length using strlen, insted of maintaing one more variable.
+ */
+#define SDT_REG_NAME_SIZE 6
+
+/*
+ * The uprobe parser does not support all gas register names;
+ * so, we have to replace them (ex. for x86_64: %rax -> %ax).
+ * Note: If register does not require renaming, just copy
+ * paste as it is, but don't leave it empty.
+ */
+static void sdt_rename_register(char *sdt_reg, int sdt_len, char *uprobe_reg)
+{
+ int i = 0;
+
+ for (i = 0; sdt_reg_tbl[i].sdt_name != NULL; i++) {
+ if (!strncmp(sdt_reg_tbl[i].sdt_name, sdt_reg, sdt_len)) {
+ strcpy(uprobe_reg, sdt_reg_tbl[i].uprobe_name);
+ return;
+ }
+ }
+
+ strncpy(uprobe_reg, sdt_reg, sdt_len);
+}
+
+int arch_sdt_arg_parse_op(char *old_op, char **new_op)
+{
+ char new_reg[SDT_REG_NAME_SIZE] = {0};
+ int new_len = 0, ret;
+ /*
+ * rm[0]: +/-NUM(REG)
+ * rm[1]: +/-
+ * rm[2]: NUM
+ * rm[3]: (
+ * rm[4]: REG
+ * rm[5]: )
+ */
+ regmatch_t rm[6];
+ /*
+ * Max prefix length is 2 as it may contains sign(+/-)
+ * and displacement 0 (Both sign and displacement 0 are
+ * optional so it may be empty). Use one more character
+ * to hold last NULL so that strlen can be used to find
+ * prefix length, instead of maintaing one more variable.
+ */
+ char prefix[3] = {0};
+
+ ret = sdt_init_op_regex();
+ if (ret < 0)
+ return ret;
+
+ /*
+ * If unsupported OR does not match with regex OR
+ * register name too long, skip it.
+ */
+ if (strchr(old_op, ',') || strchr(old_op, '$') ||
+ regexec(&sdt_op_regex, old_op, 6, rm, 0) ||
+ rm[4].rm_eo - rm[4].rm_so > SDT_REG_NAME_SIZE) {
+ pr_debug4("Skipping unsupported SDT argument: %s\n", old_op);
+ return SDT_ARG_SKIP;
+ }
+
+ /*
+ * Prepare prefix.
+ * If SDT OP has parenthesis but does not provide
+ * displacement, add 0 for displacement.
+ * SDT Uprobe Prefix
+ * -----------------------------
+ * +24(%rdi) +24(%di) +
+ * 24(%rdi) +24(%di) +
+ * %rdi %di
+ * (%rdi) +0(%di) +0
+ * -80(%rbx) -80(%bx) -
+ */
+ if (rm[3].rm_so != rm[3].rm_eo) {
+ if (rm[1].rm_so != rm[1].rm_eo)
+ prefix[0] = *(old_op + rm[1].rm_so);
+ else if (rm[2].rm_so != rm[2].rm_eo)
+ prefix[0] = '+';
+ else
+ strncpy(prefix, "+0", 2);
+ }
+
+ /* Rename register */
+ sdt_rename_register(old_op + rm[4].rm_so, rm[4].rm_eo - rm[4].rm_so,
+ new_reg);
+
+ /* Prepare final OP which should be valid for uprobe_events */
+ new_len = strlen(prefix) +
+ (rm[2].rm_eo - rm[2].rm_so) +
+ (rm[3].rm_eo - rm[3].rm_so) +
+ strlen(new_reg) +
+ (rm[5].rm_eo - rm[5].rm_so) +
+ 1; /* NULL */
+
+ *new_op = zalloc(new_len);
+ if (!*new_op)
+ return -ENOMEM;
+
+ scnprintf(*new_op, new_len, "%.*s%.*s%.*s%.*s%.*s",
+ strlen(prefix), prefix,
+ (int)(rm[2].rm_eo - rm[2].rm_so), old_op + rm[2].rm_so,
+ (int)(rm[3].rm_eo - rm[3].rm_so), old_op + rm[3].rm_so,
+ strlen(new_reg), new_reg,
+ (int)(rm[5].rm_eo - rm[5].rm_so), old_op + rm[5].rm_so);
+
+ return SDT_ARG_VALID;
+}
diff --git a/tools/perf/arch/x86/util/unwind-libdw.c b/tools/perf/arch/x86/util/unwind-libdw.c
index c4b7217..38dc9bb 100644
--- a/tools/perf/arch/x86/util/unwind-libdw.c
+++ b/tools/perf/arch/x86/util/unwind-libdw.c
@@ -1,6 +1,7 @@
#include <elfutils/libdwfl.h>
#include "../../util/unwind-libdw.h"
#include "../../util/perf_regs.h"
+#include "../../util/event.h"
bool libdw__arch_set_initial_registers(Dwfl_Thread *thread, void *arg)
{
diff --git a/tools/perf/bench/bench.h b/tools/perf/bench/bench.h
index 579a592..842ab27 100644
--- a/tools/perf/bench/bench.h
+++ b/tools/perf/bench/bench.h
@@ -25,17 +25,17 @@
# endif
#endif
-int bench_numa(int argc, const char **argv, const char *prefix);
-int bench_sched_messaging(int argc, const char **argv, const char *prefix);
-int bench_sched_pipe(int argc, const char **argv, const char *prefix);
-int bench_mem_memcpy(int argc, const char **argv, const char *prefix);
-int bench_mem_memset(int argc, const char **argv, const char *prefix);
-int bench_futex_hash(int argc, const char **argv, const char *prefix);
-int bench_futex_wake(int argc, const char **argv, const char *prefix);
-int bench_futex_wake_parallel(int argc, const char **argv, const char *prefix);
-int bench_futex_requeue(int argc, const char **argv, const char *prefix);
+int bench_numa(int argc, const char **argv);
+int bench_sched_messaging(int argc, const char **argv);
+int bench_sched_pipe(int argc, const char **argv);
+int bench_mem_memcpy(int argc, const char **argv);
+int bench_mem_memset(int argc, const char **argv);
+int bench_futex_hash(int argc, const char **argv);
+int bench_futex_wake(int argc, const char **argv);
+int bench_futex_wake_parallel(int argc, const char **argv);
+int bench_futex_requeue(int argc, const char **argv);
/* pi futexes */
-int bench_futex_lock_pi(int argc, const char **argv, const char *prefix);
+int bench_futex_lock_pi(int argc, const char **argv);
#define BENCH_FORMAT_DEFAULT_STR "default"
#define BENCH_FORMAT_DEFAULT 0
diff --git a/tools/perf/bench/futex-hash.c b/tools/perf/bench/futex-hash.c
index da04b8c..fe16b31 100644
--- a/tools/perf/bench/futex-hash.c
+++ b/tools/perf/bench/futex-hash.c
@@ -9,6 +9,7 @@
*/
/* For the CLR_() macros */
+#include <string.h>
#include <pthread.h>
#include <errno.h>
@@ -113,8 +114,7 @@
(int) runtime.tv_sec);
}
-int bench_futex_hash(int argc, const char **argv,
- const char *prefix __maybe_unused)
+int bench_futex_hash(int argc, const char **argv)
{
int ret = 0;
cpu_set_t cpu;
diff --git a/tools/perf/bench/futex-lock-pi.c b/tools/perf/bench/futex-lock-pi.c
index 9187777..73a1c44 100644
--- a/tools/perf/bench/futex-lock-pi.c
+++ b/tools/perf/bench/futex-lock-pi.c
@@ -3,6 +3,7 @@
*/
/* For the CLR_() macros */
+#include <string.h>
#include <pthread.h>
#include <signal.h>
@@ -139,8 +140,7 @@
}
}
-int bench_futex_lock_pi(int argc, const char **argv,
- const char *prefix __maybe_unused)
+int bench_futex_lock_pi(int argc, const char **argv)
{
int ret = 0;
unsigned int i;
diff --git a/tools/perf/bench/futex-requeue.c b/tools/perf/bench/futex-requeue.c
index 2b9705a..41786cb 100644
--- a/tools/perf/bench/futex-requeue.c
+++ b/tools/perf/bench/futex-requeue.c
@@ -9,6 +9,7 @@
*/
/* For the CLR_() macros */
+#include <string.h>
#include <pthread.h>
#include <signal.h>
@@ -108,8 +109,7 @@
done = true;
}
-int bench_futex_requeue(int argc, const char **argv,
- const char *prefix __maybe_unused)
+int bench_futex_requeue(int argc, const char **argv)
{
int ret = 0;
unsigned int i, j;
diff --git a/tools/perf/bench/futex-wake-parallel.c b/tools/perf/bench/futex-wake-parallel.c
index 2c8fa67..4ab12c8 100644
--- a/tools/perf/bench/futex-wake-parallel.c
+++ b/tools/perf/bench/futex-wake-parallel.c
@@ -8,6 +8,7 @@
*/
/* For the CLR_() macros */
+#include <string.h>
#include <pthread.h>
#include <signal.h>
@@ -196,8 +197,7 @@
done = true;
}
-int bench_futex_wake_parallel(int argc, const char **argv,
- const char *prefix __maybe_unused)
+int bench_futex_wake_parallel(int argc, const char **argv)
{
int ret = 0;
unsigned int i, j;
diff --git a/tools/perf/bench/futex-wake.c b/tools/perf/bench/futex-wake.c
index e246b1b..2fa4922 100644
--- a/tools/perf/bench/futex-wake.c
+++ b/tools/perf/bench/futex-wake.c
@@ -9,6 +9,7 @@
*/
/* For the CLR_() macros */
+#include <string.h>
#include <pthread.h>
#include <signal.h>
@@ -114,8 +115,7 @@
done = true;
}
-int bench_futex_wake(int argc, const char **argv,
- const char *prefix __maybe_unused)
+int bench_futex_wake(int argc, const char **argv)
{
int ret = 0;
unsigned int i, j;
diff --git a/tools/perf/bench/futex.h b/tools/perf/bench/futex.h
index b2e06d1..e44fd32 100644
--- a/tools/perf/bench/futex.h
+++ b/tools/perf/bench/futex.h
@@ -88,13 +88,11 @@
#ifndef HAVE_PTHREAD_ATTR_SETAFFINITY_NP
#include <pthread.h>
-static inline int pthread_attr_setaffinity_np(pthread_attr_t *attr,
- size_t cpusetsize,
- cpu_set_t *cpuset)
+#include <linux/compiler.h>
+static inline int pthread_attr_setaffinity_np(pthread_attr_t *attr __maybe_unused,
+ size_t cpusetsize __maybe_unused,
+ cpu_set_t *cpuset __maybe_unused)
{
- attr = attr;
- cpusetsize = cpusetsize;
- cpuset = cpuset;
return 0;
}
#endif
diff --git a/tools/perf/bench/mem-functions.c b/tools/perf/bench/mem-functions.c
index 52504a8..fbd732b 100644
--- a/tools/perf/bench/mem-functions.c
+++ b/tools/perf/bench/mem-functions.c
@@ -12,6 +12,7 @@
#include <subcmd/parse-options.h>
#include "../util/header.h"
#include "../util/cloexec.h"
+#include "../util/string2.h"
#include "bench.h"
#include "mem-memcpy-arch.h"
#include "mem-memset-arch.h"
@@ -284,7 +285,7 @@
NULL
};
-int bench_mem_memcpy(int argc, const char **argv, const char *prefix __maybe_unused)
+int bench_mem_memcpy(int argc, const char **argv)
{
struct bench_mem_info info = {
.functions = memcpy_functions,
@@ -358,7 +359,7 @@
{ .name = NULL, }
};
-int bench_mem_memset(int argc, const char **argv, const char *prefix __maybe_unused)
+int bench_mem_memset(int argc, const char **argv)
{
struct bench_mem_info info = {
.functions = memset_functions,
diff --git a/tools/perf/bench/numa.c b/tools/perf/bench/numa.c
index 3083fc3..27de0c8 100644
--- a/tools/perf/bench/numa.c
+++ b/tools/perf/bench/numa.c
@@ -4,6 +4,7 @@
* numa: Simulate NUMA-sensitive workload and measure their NUMA performance
*/
+#include <inttypes.h>
/* For the CLR_() macros */
#include <pthread.h>
@@ -30,6 +31,7 @@
#include <sys/wait.h>
#include <sys/prctl.h>
#include <sys/types.h>
+#include <linux/kernel.h>
#include <linux/time64.h>
#include <numa.h>
@@ -187,7 +189,8 @@
OPT_INCR ('d', "show_details" , &p0.show_details, "Show details"),
OPT_INCR ('a', "all" , &p0.run_all, "Run all tests in the suite"),
OPT_INTEGER('H', "thp" , &p0.thp, "MADV_NOHUGEPAGE < 0 < MADV_HUGEPAGE"),
- OPT_BOOLEAN('c', "show_convergence", &p0.show_convergence, "show convergence details"),
+ OPT_BOOLEAN('c', "show_convergence", &p0.show_convergence, "show convergence details, "
+ "convergence is reached when each process (all its threads) is running on a single NUMA node."),
OPT_BOOLEAN('m', "measure_convergence", &p0.measure_convergence, "measure convergence latency"),
OPT_BOOLEAN('q', "quiet" , &p0.show_quiet, "quiet mode"),
OPT_BOOLEAN('S', "serialize-startup", &p0.serialize_startup,"serialize thread startup"),
@@ -1766,7 +1769,7 @@
return 0;
}
-int bench_numa(int argc, const char **argv, const char *prefix __maybe_unused)
+int bench_numa(int argc, const char **argv)
{
init_params(&p0, "main,", argc, argv);
argc = parse_options(argc, argv, options, bench_numa_usage, 0);
diff --git a/tools/perf/bench/sched-messaging.c b/tools/perf/bench/sched-messaging.c
index 6a111e7..4f961e7 100644
--- a/tools/perf/bench/sched-messaging.c
+++ b/tools/perf/bench/sched-messaging.c
@@ -260,8 +260,7 @@
NULL
};
-int bench_sched_messaging(int argc, const char **argv,
- const char *prefix __maybe_unused)
+int bench_sched_messaging(int argc, const char **argv)
{
unsigned int i, total_children;
struct timeval start, stop, diff;
diff --git a/tools/perf/bench/sched-pipe.c b/tools/perf/bench/sched-pipe.c
index 2243f01..a152737 100644
--- a/tools/perf/bench/sched-pipe.c
+++ b/tools/perf/bench/sched-pipe.c
@@ -76,7 +76,7 @@
return NULL;
}
-int bench_sched_pipe(int argc, const char **argv, const char *prefix __maybe_unused)
+int bench_sched_pipe(int argc, const char **argv)
{
struct thread_data threads[2], *td;
int pipe_1[2], pipe_2[2];
diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
index 4f52d85..7a5dc7e 100644
--- a/tools/perf/builtin-annotate.c
+++ b/tools/perf/builtin-annotate.c
@@ -33,6 +33,7 @@
#include "util/block-range.h"
#include <dlfcn.h>
+#include <errno.h>
#include <linux/bitmap.h>
struct perf_annotate {
@@ -383,7 +384,7 @@
NULL
};
-int cmd_annotate(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_annotate(int argc, const char **argv)
{
struct perf_annotate annotate = {
.tool = {
@@ -393,6 +394,9 @@
.comm = perf_event__process_comm,
.exit = perf_event__process_exit,
.fork = perf_event__process_fork,
+ .namespaces = perf_event__process_namespaces,
+ .attr = perf_event__process_attr,
+ .build_id = perf_event__process_build_id,
.ordered_events = true,
.ordering_requires_timestamps = true,
},
diff --git a/tools/perf/builtin-bench.c b/tools/perf/builtin-bench.c
index a1cddc6..445e628 100644
--- a/tools/perf/builtin-bench.c
+++ b/tools/perf/builtin-bench.c
@@ -25,7 +25,7 @@
#include <string.h>
#include <sys/prctl.h>
-typedef int (*bench_fn_t)(int argc, const char **argv, const char *prefix);
+typedef int (*bench_fn_t)(int argc, const char **argv);
struct bench {
const char *name;
@@ -155,7 +155,7 @@
* to something meaningful:
*/
static int run_bench(const char *coll_name, const char *bench_name, bench_fn_t fn,
- int argc, const char **argv, const char *prefix)
+ int argc, const char **argv)
{
int size;
char *name;
@@ -171,7 +171,7 @@
prctl(PR_SET_NAME, name);
argv[0] = name;
- ret = fn(argc, argv, prefix);
+ ret = fn(argc, argv);
free(name);
@@ -198,7 +198,7 @@
fflush(stdout);
argv[1] = bench->name;
- run_bench(coll->name, bench->name, bench->fn, 1, argv, NULL);
+ run_bench(coll->name, bench->name, bench->fn, 1, argv);
printf("\n");
}
}
@@ -211,7 +211,7 @@
run_collection(coll);
}
-int cmd_bench(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_bench(int argc, const char **argv)
{
struct collection *coll;
int ret = 0;
@@ -270,7 +270,7 @@
if (bench_format == BENCH_FORMAT_DEFAULT)
printf("# Running '%s/%s' benchmark:\n", coll->name, bench->name);
fflush(stdout);
- ret = run_bench(coll->name, bench->name, bench->fn, argc-1, argv+1, prefix);
+ ret = run_bench(coll->name, bench->name, bench->fn, argc-1, argv+1);
goto end;
}
diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
index 30e2b2c..64b44e8 100644
--- a/tools/perf/builtin-buildid-cache.c
+++ b/tools/perf/builtin-buildid-cache.c
@@ -10,6 +10,7 @@
#include <sys/time.h>
#include <time.h>
#include <dirent.h>
+#include <errno.h>
#include <unistd.h>
#include "builtin.h"
#include "perf.h"
@@ -21,6 +22,7 @@
#include "util/build-id.h"
#include "util/session.h"
#include "util/symbol.h"
+#include "util/time-utils.h"
static int build_id_cache__kcore_buildid(const char *proc_dir, char *sbuildid)
{
@@ -276,8 +278,7 @@
return err;
}
-int cmd_buildid_cache(int argc, const char **argv,
- const char *prefix __maybe_unused)
+int cmd_buildid_cache(int argc, const char **argv)
{
struct strlist *list;
struct str_node *pos;
diff --git a/tools/perf/builtin-buildid-list.c b/tools/perf/builtin-buildid-list.c
index 5e914ee..fdaca16 100644
--- a/tools/perf/builtin-buildid-list.c
+++ b/tools/perf/builtin-buildid-list.c
@@ -16,6 +16,7 @@
#include "util/session.h"
#include "util/symbol.h"
#include "util/data.h"
+#include <errno.h>
static int sysfs__fprintf_build_id(FILE *fp)
{
@@ -87,8 +88,7 @@
return 0;
}
-int cmd_buildid_list(int argc, const char **argv,
- const char *prefix __maybe_unused)
+int cmd_buildid_list(int argc, const char **argv)
{
bool show_kernel = false;
bool with_hits = false;
diff --git a/tools/perf/builtin-c2c.c b/tools/perf/builtin-c2c.c
index e2b2172..e33b4ac 100644
--- a/tools/perf/builtin-c2c.c
+++ b/tools/perf/builtin-c2c.c
@@ -9,10 +9,13 @@
* Dick Fowles <fowles@inreach.com>
* Joe Mario <jmario@redhat.com>
*/
+#include <errno.h>
+#include <inttypes.h>
#include <linux/compiler.h>
#include <linux/kernel.h>
#include <linux/stringify.h>
#include <asm/bug.h>
+#include <sys/param.h>
#include "util.h"
#include "debug.h"
#include "builtin.h"
@@ -29,6 +32,7 @@
#include <asm/bug.h>
#include "ui/browsers/hists.h"
#include "evlist.h"
+#include "thread.h"
struct c2c_hists {
struct hists hists;
@@ -2334,7 +2338,7 @@
static void perf_c2c_display(struct perf_session *session)
{
- if (c2c.use_stdio)
+ if (use_browser == 0)
perf_c2c__hists_fprintf(stdout, session);
else
perf_c2c__hists_browse(&c2c.hists.hists);
@@ -2536,7 +2540,7 @@
OPT_BOOLEAN(0, "stdio", &c2c.use_stdio, "Use the stdio interface"),
#endif
OPT_BOOLEAN(0, "stats", &c2c.stats_only,
- "Use the stdio interface"),
+ "Display only statistic tables (implies --stdio)"),
OPT_BOOLEAN(0, "full-symbols", &c2c.symbol_full,
"Display full length of symbols"),
OPT_BOOLEAN(0, "no-source", &no_source,
@@ -2755,12 +2759,12 @@
pr_debug("\n");
}
- ret = cmd_record(i, rec_argv, NULL);
+ ret = cmd_record(i, rec_argv);
free(rec_argv);
return ret;
}
-int cmd_c2c(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_c2c(int argc, const char **argv)
{
argc = parse_options(argc, argv, c2c_options, c2c_usage,
PARSE_OPT_STOP_AT_NON_OPTION);
diff --git a/tools/perf/builtin-config.c b/tools/perf/builtin-config.c
index 8c0d93b..55f04f8 100644
--- a/tools/perf/builtin-config.c
+++ b/tools/perf/builtin-config.c
@@ -154,7 +154,7 @@
return 0;
}
-int cmd_config(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_config(int argc, const char **argv)
{
int i, ret = 0;
struct perf_config_set *set;
diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c
index 7ad6e17..0adb5f8 100644
--- a/tools/perf/builtin-data.c
+++ b/tools/perf/builtin-data.c
@@ -6,7 +6,7 @@
#include "data-convert.h"
#include "data-convert-bt.h"
-typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix);
+typedef int (*data_cmd_fn_t)(int argc, const char **argv);
struct data_cmd {
const char *name;
@@ -50,8 +50,7 @@
NULL
};
-static int cmd_data_convert(int argc, const char **argv,
- const char *prefix __maybe_unused)
+static int cmd_data_convert(int argc, const char **argv)
{
const char *to_ctf = NULL;
struct perf_data_convert_opts opts = {
@@ -98,7 +97,7 @@
{ .name = NULL, },
};
-int cmd_data(int argc, const char **argv, const char *prefix)
+int cmd_data(int argc, const char **argv)
{
struct data_cmd *cmd;
const char *cmdstr;
@@ -118,7 +117,7 @@
if (strcmp(cmd->name, cmdstr))
continue;
- return cmd->fn(argc, argv, prefix);
+ return cmd->fn(argc, argv);
}
pr_err("Unknown command: %s\n", cmdstr);
diff --git a/tools/perf/builtin-diff.c b/tools/perf/builtin-diff.c
index 1b96a31..eec5df8 100644
--- a/tools/perf/builtin-diff.c
+++ b/tools/perf/builtin-diff.c
@@ -19,6 +19,8 @@
#include "util/data.h"
#include "util/config.h"
+#include <errno.h>
+#include <inttypes.h>
#include <stdlib.h>
#include <math.h>
@@ -364,6 +366,7 @@
.exit = perf_event__process_exit,
.fork = perf_event__process_fork,
.lost = perf_event__process_lost,
+ .namespaces = perf_event__process_namespaces,
.ordered_events = true,
.ordering_requires_timestamps = true,
};
@@ -1320,7 +1323,7 @@
return 0;
}
-int cmd_diff(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_diff(int argc, const char **argv)
{
int ret = hists__init();
diff --git a/tools/perf/builtin-evlist.c b/tools/perf/builtin-evlist.c
index e09c428..6d210e4 100644
--- a/tools/perf/builtin-evlist.c
+++ b/tools/perf/builtin-evlist.c
@@ -46,7 +46,7 @@
return 0;
}
-int cmd_evlist(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_evlist(int argc, const char **argv)
{
struct perf_attr_details details = { .verbose = false, };
const struct option options[] = {
diff --git a/tools/perf/builtin-ftrace.c b/tools/perf/builtin-ftrace.c
index c3e6436..9e0b35c 100644
--- a/tools/perf/builtin-ftrace.c
+++ b/tools/perf/builtin-ftrace.c
@@ -9,13 +9,18 @@
#include "builtin.h"
#include "perf.h"
+#include <errno.h>
#include <unistd.h>
#include <signal.h>
+#include <fcntl.h>
+#include <poll.h>
#include "debug.h"
#include <subcmd/parse-options.h>
+#include <api/fs/tracing_path.h>
#include "evlist.h"
#include "target.h"
+#include "cpumap.h"
#include "thread_map.h"
#include "util/config.h"
@@ -50,11 +55,12 @@
done = true;
}
-static int write_tracing_file(const char *name, const char *val)
+static int __write_tracing_file(const char *name, const char *val, bool append)
{
char *file;
int fd, ret = -1;
ssize_t size = strlen(val);
+ int flags = O_WRONLY;
file = get_tracing_file(name);
if (!file) {
@@ -62,7 +68,12 @@
return -1;
}
- fd = open(file, O_WRONLY);
+ if (append)
+ flags |= O_APPEND;
+ else
+ flags |= O_TRUNC;
+
+ fd = open(file, flags);
if (fd < 0) {
pr_debug("cannot open tracing file: %s\n", name);
goto out;
@@ -79,6 +90,18 @@
return ret;
}
+static int write_tracing_file(const char *name, const char *val)
+{
+ return __write_tracing_file(name, val, false);
+}
+
+static int append_tracing_file(const char *name, const char *val)
+{
+ return __write_tracing_file(name, val, true);
+}
+
+static int reset_tracing_cpu(void);
+
static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
{
if (write_tracing_file("tracing_on", "0") < 0)
@@ -90,14 +113,78 @@
if (write_tracing_file("set_ftrace_pid", " ") < 0)
return -1;
+ if (reset_tracing_cpu() < 0)
+ return -1;
+
return 0;
}
+static int set_tracing_pid(struct perf_ftrace *ftrace)
+{
+ int i;
+ char buf[16];
+
+ if (target__has_cpu(&ftrace->target))
+ return 0;
+
+ for (i = 0; i < thread_map__nr(ftrace->evlist->threads); i++) {
+ scnprintf(buf, sizeof(buf), "%d",
+ ftrace->evlist->threads->map[i]);
+ if (append_tracing_file("set_ftrace_pid", buf) < 0)
+ return -1;
+ }
+ return 0;
+}
+
+static int set_tracing_cpumask(struct cpu_map *cpumap)
+{
+ char *cpumask;
+ size_t mask_size;
+ int ret;
+ int last_cpu;
+
+ last_cpu = cpu_map__cpu(cpumap, cpumap->nr - 1);
+ mask_size = (last_cpu + 3) / 4 + 1;
+ mask_size += last_cpu / 32; /* ',' is needed for every 32th cpus */
+
+ cpumask = malloc(mask_size);
+ if (cpumask == NULL) {
+ pr_debug("failed to allocate cpu mask\n");
+ return -1;
+ }
+
+ cpu_map__snprint_mask(cpumap, cpumask, mask_size);
+
+ ret = write_tracing_file("tracing_cpumask", cpumask);
+
+ free(cpumask);
+ return ret;
+}
+
+static int set_tracing_cpu(struct perf_ftrace *ftrace)
+{
+ struct cpu_map *cpumap = ftrace->evlist->cpus;
+
+ if (!target__has_cpu(&ftrace->target))
+ return 0;
+
+ return set_tracing_cpumask(cpumap);
+}
+
+static int reset_tracing_cpu(void)
+{
+ struct cpu_map *cpumap = cpu_map__new(NULL);
+ int ret;
+
+ ret = set_tracing_cpumask(cpumap);
+ cpu_map__put(cpumap);
+ return ret;
+}
+
static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
{
char *trace_file;
int trace_fd;
- char *trace_pid;
char buf[4096];
struct pollfd pollfd = {
.events = POLLIN,
@@ -108,42 +195,43 @@
return -1;
}
- if (argc < 1)
- return -1;
-
signal(SIGINT, sig_handler);
signal(SIGUSR1, sig_handler);
signal(SIGCHLD, sig_handler);
+ signal(SIGPIPE, sig_handler);
- reset_tracing_files(ftrace);
+ if (reset_tracing_files(ftrace) < 0)
+ goto out;
/* reset ftrace buffer */
if (write_tracing_file("trace", "0") < 0)
goto out;
- if (perf_evlist__prepare_workload(ftrace->evlist, &ftrace->target,
- argv, false, ftrace__workload_exec_failed_signal) < 0)
+ if (argc && perf_evlist__prepare_workload(ftrace->evlist,
+ &ftrace->target, argv, false,
+ ftrace__workload_exec_failed_signal) < 0) {
goto out;
+ }
+
+ if (set_tracing_pid(ftrace) < 0) {
+ pr_err("failed to set ftrace pid\n");
+ goto out_reset;
+ }
+
+ if (set_tracing_cpu(ftrace) < 0) {
+ pr_err("failed to set tracing cpumask\n");
+ goto out_reset;
+ }
if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
- goto out;
- }
-
- if (asprintf(&trace_pid, "%d", thread_map__pid(ftrace->evlist->threads, 0)) < 0) {
- pr_err("failed to allocate pid string\n");
- goto out;
- }
-
- if (write_tracing_file("set_ftrace_pid", trace_pid) < 0) {
- pr_err("failed to set pid: %s\n", trace_pid);
- goto out_free_pid;
+ goto out_reset;
}
trace_file = get_tracing_file("trace_pipe");
if (!trace_file) {
pr_err("failed to open trace_pipe\n");
- goto out_free_pid;
+ goto out_reset;
}
trace_fd = open(trace_file, O_RDONLY);
@@ -152,7 +240,7 @@
if (trace_fd < 0) {
pr_err("failed to open trace_pipe\n");
- goto out_free_pid;
+ goto out_reset;
}
fcntl(trace_fd, F_SETFL, O_NONBLOCK);
@@ -163,6 +251,8 @@
goto out_close_fd;
}
+ setup_pager();
+
perf_evlist__start_workload(ftrace->evlist);
while (!done) {
@@ -191,11 +281,9 @@
out_close_fd:
close(trace_fd);
-out_free_pid:
- free(trace_pid);
-out:
+out_reset:
reset_tracing_files(ftrace);
-
+out:
return done ? 0 : -1;
}
@@ -219,7 +307,7 @@
return -1;
}
-int cmd_ftrace(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_ftrace(int argc, const char **argv)
{
int ret;
struct perf_ftrace ftrace = {
@@ -227,15 +315,21 @@
.target = { .uid = UINT_MAX, },
};
const char * const ftrace_usage[] = {
- "perf ftrace [<options>] <command>",
+ "perf ftrace [<options>] [<command>]",
"perf ftrace [<options>] -- <command> [<options>]",
NULL
};
const struct option ftrace_options[] = {
OPT_STRING('t', "tracer", &ftrace.tracer, "tracer",
"tracer to use: function_graph(default) or function"),
+ OPT_STRING('p', "pid", &ftrace.target.pid, "pid",
+ "trace on existing process id"),
OPT_INCR('v', "verbose", &verbose,
"be more verbose"),
+ OPT_BOOLEAN('a', "all-cpus", &ftrace.target.system_wide,
+ "system-wide collection from all CPUs"),
+ OPT_STRING('C', "cpu", &ftrace.target.cpu_list, "cpu",
+ "list of cpus to monitor"),
OPT_END()
};
@@ -245,9 +339,18 @@
argc = parse_options(argc, argv, ftrace_options, ftrace_usage,
PARSE_OPT_STOP_AT_NON_OPTION);
- if (!argc)
+ if (!argc && target__none(&ftrace.target))
usage_with_options(ftrace_usage, ftrace_options);
+ ret = target__validate(&ftrace.target);
+ if (ret) {
+ char errbuf[512];
+
+ target__strerror(&ftrace.target, ret, errbuf, 512);
+ pr_err("%s\n", errbuf);
+ return -EINVAL;
+ }
+
ftrace.evlist = perf_evlist__new();
if (ftrace.evlist == NULL)
return -ENOMEM;
diff --git a/tools/perf/builtin-help.c b/tools/perf/builtin-help.c
index aed0d84..492f8e1 100644
--- a/tools/perf/builtin-help.c
+++ b/tools/perf/builtin-help.c
@@ -12,16 +12,22 @@
#include <subcmd/run-command.h>
#include <subcmd/help.h>
#include "util/debug.h"
+#include <linux/kernel.h>
+#include <errno.h>
+#include <stdio.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
static struct man_viewer_list {
struct man_viewer_list *next;
- char name[FLEX_ARRAY];
+ char name[0];
} *man_viewer_list;
static struct man_viewer_info_list {
struct man_viewer_info_list *next;
const char *info;
- char name[FLEX_ARRAY];
+ char name[0];
} *man_viewer_info_list;
enum help_format {
@@ -301,12 +307,6 @@
}
}
-static int is_perf_command(const char *s)
-{
- return is_in_cmdlist(&main_cmds, s) ||
- is_in_cmdlist(&other_cmds, s);
-}
-
static const char *cmd_to_page(const char *perf_cmd)
{
char *s;
@@ -418,7 +418,7 @@
return 0;
}
-int cmd_help(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_help(int argc, const char **argv)
{
bool show_all = false;
enum help_format help_format = HELP_FORMAT_MAN;
@@ -446,7 +446,6 @@
"perf help [--all] [--man|--web|--info] [command]",
NULL
};
- const char *alias;
int rc;
load_command_list("perf-", &main_cmds, &other_cmds);
@@ -472,12 +471,6 @@
return 0;
}
- alias = alias_lookup(argv[0]);
- if (alias && !is_perf_command(argv[0])) {
- printf("`perf %s' is aliased to `%s'\n", argv[0], alias);
- return 0;
- }
-
switch (help_format) {
case HELP_FORMAT_MAN:
rc = show_man_page(argv[0]);
diff --git a/tools/perf/builtin-inject.c b/tools/perf/builtin-inject.c
index b9bc7e3..ea8db38 100644
--- a/tools/perf/builtin-inject.c
+++ b/tools/perf/builtin-inject.c
@@ -18,10 +18,13 @@
#include "util/data.h"
#include "util/auxtrace.h"
#include "util/jit.h"
+#include "util/thread.h"
#include <subcmd/parse-options.h>
#include <linux/list.h>
+#include <errno.h>
+#include <signal.h>
struct perf_inject {
struct perf_tool tool;
@@ -333,6 +336,18 @@
return err;
}
+static int perf_event__repipe_namespaces(struct perf_tool *tool,
+ union perf_event *event,
+ struct perf_sample *sample,
+ struct machine *machine)
+{
+ int err = perf_event__process_namespaces(tool, event, sample, machine);
+
+ perf_event__repipe(tool, event, sample, machine);
+
+ return err;
+}
+
static int perf_event__repipe_exit(struct perf_tool *tool,
union perf_event *event,
struct perf_sample *sample,
@@ -660,6 +675,7 @@
session->itrace_synth_opts = &inject->itrace_synth_opts;
inject->itrace_synth_opts.inject = true;
inject->tool.comm = perf_event__repipe_comm;
+ inject->tool.namespaces = perf_event__repipe_namespaces;
inject->tool.exit = perf_event__repipe_exit;
inject->tool.id_index = perf_event__repipe_id_index;
inject->tool.auxtrace_info = perf_event__process_auxtrace_info;
@@ -681,6 +697,8 @@
lseek(fd, output_data_offset, SEEK_SET);
ret = perf_session__process_events(session);
+ if (ret)
+ return ret;
if (!file_out->is_pipe) {
if (inject->build_ids)
@@ -725,7 +743,7 @@
return ret;
}
-int cmd_inject(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_inject(int argc, const char **argv)
{
struct perf_inject inject = {
.tool = {
diff --git a/tools/perf/builtin-kallsyms.c b/tools/perf/builtin-kallsyms.c
index 224bfc4..bcfb363 100644
--- a/tools/perf/builtin-kallsyms.c
+++ b/tools/perf/builtin-kallsyms.c
@@ -7,6 +7,7 @@
*
* Released under the GPL v2. (and only v2, not any later version)
*/
+#include <inttypes.h>
#include "builtin.h"
#include <linux/compiler.h>
#include <subcmd/parse-options.h>
@@ -43,7 +44,7 @@
return 0;
}
-int cmd_kallsyms(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_kallsyms(int argc, const char **argv)
{
const struct option options[] = {
OPT_INCR('v', "verbose", &verbose, "be more verbose (show counter open errors, etc)"),
diff --git a/tools/perf/builtin-kmem.c b/tools/perf/builtin-kmem.c
index 6da8d08..9409c94 100644
--- a/tools/perf/builtin-kmem.c
+++ b/tools/perf/builtin-kmem.c
@@ -20,11 +20,16 @@
#include "util/debug.h"
+#include <linux/kernel.h>
#include <linux/rbtree.h>
#include <linux/string.h>
+#include <errno.h>
+#include <inttypes.h>
#include <locale.h>
#include <regex.h>
+#include "sane_ctype.h"
+
static int kmem_slab;
static int kmem_page;
@@ -964,6 +969,7 @@
.comm = perf_event__process_comm,
.mmap = perf_event__process_mmap,
.mmap2 = perf_event__process_mmap2,
+ .namespaces = perf_event__process_namespaces,
.ordered_events = true,
};
@@ -1865,7 +1871,7 @@
for (j = 1; j < (unsigned int)argc; j++, i++)
rec_argv[i] = argv[j];
- return cmd_record(i, rec_argv, NULL);
+ return cmd_record(i, rec_argv);
}
static int kmem_config(const char *var, const char *value, void *cb __maybe_unused)
@@ -1884,7 +1890,7 @@
return 0;
}
-int cmd_kmem(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_kmem(int argc, const char **argv)
{
const char * const default_slab_sort = "frag,hit,bytes";
const char * const default_page_sort = "bytes,hit";
diff --git a/tools/perf/builtin-kvm.c b/tools/perf/builtin-kvm.c
index 08fa88f..f309c37 100644
--- a/tools/perf/builtin-kvm.c
+++ b/tools/perf/builtin-kvm.c
@@ -3,6 +3,7 @@
#include "util/evsel.h"
#include "util/evlist.h"
+#include "util/term.h"
#include "util/util.h"
#include "util/cache.h"
#include "util/symbol.h"
@@ -23,13 +24,33 @@
#ifdef HAVE_TIMERFD_SUPPORT
#include <sys/timerfd.h>
#endif
+#include <sys/time.h>
+#include <linux/kernel.h>
#include <linux/time64.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <poll.h>
#include <termios.h>
#include <semaphore.h>
+#include <signal.h>
#include <pthread.h>
#include <math.h>
+static const char *get_filename_for_perf_kvm(void)
+{
+ const char *filename;
+
+ if (perf_host && !perf_guest)
+ filename = strdup("perf.data.host");
+ else if (!perf_host && perf_guest)
+ filename = strdup("perf.data.guest");
+ else
+ filename = strdup("perf.data.kvm");
+
+ return filename;
+}
+
#ifdef HAVE_KVM_STAT_SUPPORT
#include "util/kvm-stat.h"
@@ -1044,6 +1065,7 @@
struct perf_tool eops = {
.sample = process_sample_event,
.comm = perf_event__process_comm,
+ .namespaces = perf_event__process_namespaces,
.ordered_events = true,
};
struct perf_data_file file = {
@@ -1208,7 +1230,7 @@
set_option_flag(record_options, 0, "transaction", PARSE_OPT_DISABLED);
record_usage = kvm_stat_record_usage;
- return cmd_record(i, rec_argv, NULL);
+ return cmd_record(i, rec_argv);
}
static int
@@ -1348,6 +1370,7 @@
kvm->tool.exit = perf_event__process_exit;
kvm->tool.fork = perf_event__process_fork;
kvm->tool.lost = process_lost_event;
+ kvm->tool.namespaces = perf_event__process_namespaces;
kvm->tool.ordered_events = true;
perf_tool__fill_defaults(&kvm->tool);
@@ -1475,7 +1498,7 @@
#endif
perf_stat:
- return cmd_stat(argc, argv, NULL);
+ return cmd_stat(argc, argv);
}
#endif /* HAVE_KVM_STAT_SUPPORT */
@@ -1494,7 +1517,7 @@
BUG_ON(i != rec_argc);
- return cmd_record(i, rec_argv, NULL);
+ return cmd_record(i, rec_argv);
}
static int __cmd_report(const char *file_name, int argc, const char **argv)
@@ -1512,7 +1535,7 @@
BUG_ON(i != rec_argc);
- return cmd_report(i, rec_argv, NULL);
+ return cmd_report(i, rec_argv);
}
static int
@@ -1531,10 +1554,10 @@
BUG_ON(i != rec_argc);
- return cmd_buildid_list(i, rec_argv, NULL);
+ return cmd_buildid_list(i, rec_argv);
}
-int cmd_kvm(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_kvm(int argc, const char **argv)
{
const char *file_name = NULL;
const struct option kvm_options[] = {
@@ -1589,9 +1612,9 @@
else if (!strncmp(argv[0], "rep", 3))
return __cmd_report(file_name, argc, argv);
else if (!strncmp(argv[0], "diff", 4))
- return cmd_diff(argc, argv, NULL);
+ return cmd_diff(argc, argv);
else if (!strncmp(argv[0], "top", 3))
- return cmd_top(argc, argv, NULL);
+ return cmd_top(argc, argv);
else if (!strncmp(argv[0], "buildid-list", 12))
return __cmd_buildid_list(file_name, argc, argv);
#ifdef HAVE_KVM_STAT_SUPPORT
diff --git a/tools/perf/builtin-list.c b/tools/perf/builtin-list.c
index 3b9d98b..4bf2cb4 100644
--- a/tools/perf/builtin-list.c
+++ b/tools/perf/builtin-list.c
@@ -18,8 +18,9 @@
#include <subcmd/parse-options.h>
static bool desc_flag = true;
+static bool details_flag;
-int cmd_list(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_list(int argc, const char **argv)
{
int i;
bool raw_dump = false;
@@ -30,6 +31,8 @@
"Print extra event descriptions. --no-desc to not print."),
OPT_BOOLEAN('v', "long-desc", &long_desc_flag,
"Print longer event descriptions."),
+ OPT_BOOLEAN(0, "details", &details_flag,
+ "Print information on the perf event names and expressions used internally by events."),
OPT_INCR(0, "debug", &verbose,
"Enable debugging output"),
OPT_END()
@@ -50,7 +53,8 @@
printf("\nList of pre-defined events (to be used in -e):\n\n");
if (argc == 0) {
- print_events(NULL, raw_dump, !desc_flag, long_desc_flag);
+ print_events(NULL, raw_dump, !desc_flag, long_desc_flag,
+ details_flag);
return 0;
}
@@ -72,7 +76,7 @@
print_hwcache_events(NULL, raw_dump);
else if (strcmp(argv[i], "pmu") == 0)
print_pmu_events(NULL, raw_dump, !desc_flag,
- long_desc_flag);
+ long_desc_flag, details_flag);
else if (strcmp(argv[i], "sdt") == 0)
print_sdt_events(NULL, NULL, raw_dump);
else if ((sep = strchr(argv[i], ':')) != NULL) {
@@ -80,7 +84,8 @@
if (sep == NULL) {
print_events(argv[i], raw_dump, !desc_flag,
- long_desc_flag);
+ long_desc_flag,
+ details_flag);
continue;
}
sep_idx = sep - argv[i];
@@ -103,7 +108,8 @@
event_symbols_sw, PERF_COUNT_SW_MAX, raw_dump);
print_hwcache_events(s, raw_dump);
print_pmu_events(s, raw_dump, !desc_flag,
- long_desc_flag);
+ long_desc_flag,
+ details_flag);
print_tracepoint_events(NULL, s, raw_dump);
print_sdt_events(NULL, s, raw_dump);
free(s);
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index ce3bfb4..ff98652 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -1,3 +1,5 @@
+#include <errno.h>
+#include <inttypes.h>
#include "builtin.h"
#include "perf.h"
@@ -26,6 +28,7 @@
#include <linux/list.h>
#include <linux/hash.h>
+#include <linux/kernel.h>
static struct perf_session *session;
@@ -858,6 +861,7 @@
struct perf_tool eops = {
.sample = process_sample_event,
.comm = perf_event__process_comm,
+ .namespaces = perf_event__process_namespaces,
.ordered_events = true,
};
struct perf_data_file file = {
@@ -940,34 +944,36 @@
BUG_ON(i != rec_argc);
- ret = cmd_record(i, rec_argv, NULL);
+ ret = cmd_record(i, rec_argv);
free(rec_argv);
return ret;
}
-int cmd_lock(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_lock(int argc, const char **argv)
{
+ const struct option lock_options[] = {
+ OPT_STRING('i', "input", &input_name, "file", "input file name"),
+ OPT_INCR('v', "verbose", &verbose, "be more verbose (show symbol address, etc)"),
+ OPT_BOOLEAN('D', "dump-raw-trace", &dump_trace, "dump raw trace in ASCII"),
+ OPT_BOOLEAN('f', "force", &force, "don't complain, do it"),
+ OPT_END()
+ };
+
const struct option info_options[] = {
OPT_BOOLEAN('t', "threads", &info_threads,
"dump thread list in perf.data"),
OPT_BOOLEAN('m', "map", &info_map,
"map of lock instances (address:name table)"),
- OPT_BOOLEAN('f', "force", &force, "don't complain, do it"),
- OPT_END()
+ OPT_PARENT(lock_options)
};
- const struct option lock_options[] = {
- OPT_STRING('i', "input", &input_name, "file", "input file name"),
- OPT_INCR('v', "verbose", &verbose, "be more verbose (show symbol address, etc)"),
- OPT_BOOLEAN('D', "dump-raw-trace", &dump_trace, "dump raw trace in ASCII"),
- OPT_END()
- };
+
const struct option report_options[] = {
OPT_STRING('k', "key", &sort_key, "acquired",
"key for sorting (acquired / contended / avg_wait / wait_total / wait_max / wait_min)"),
- OPT_BOOLEAN('f', "force", &force, "don't complain, do it"),
/* TODO: type */
- OPT_END()
+ OPT_PARENT(lock_options)
};
+
const char * const info_usage[] = {
"perf lock info [<options>]",
NULL
@@ -1006,7 +1012,7 @@
rc = __cmd_report(false);
} else if (!strcmp(argv[0], "script")) {
/* Aliased to 'perf script' */
- return cmd_script(argc, argv, prefix);
+ return cmd_script(argc, argv);
} else if (!strcmp(argv[0], "info")) {
if (argc) {
argc = parse_options(argc, argv,
diff --git a/tools/perf/builtin-mem.c b/tools/perf/builtin-mem.c
index 6114e07..e001c02 100644
--- a/tools/perf/builtin-mem.c
+++ b/tools/perf/builtin-mem.c
@@ -1,3 +1,7 @@
+#include <inttypes.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
#include "builtin.h"
#include "perf.h"
@@ -8,6 +12,7 @@
#include "util/data.h"
#include "util/mem-events.h"
#include "util/debug.h"
+#include "util/symbol.h"
#define MEM_OPERATION_LOAD 0x1
#define MEM_OPERATION_STORE 0x2
@@ -129,7 +134,7 @@
pr_debug("\n");
}
- ret = cmd_record(i, rec_argv, NULL);
+ ret = cmd_record(i, rec_argv);
free(rec_argv);
return ret;
}
@@ -256,7 +261,7 @@
for (j = 1; j < argc; j++, i++)
rep_argv[i] = argv[j];
- ret = cmd_report(i, rep_argv, NULL);
+ ret = cmd_report(i, rep_argv);
free(rep_argv);
return ret;
}
@@ -330,7 +335,7 @@
return ret;
}
-int cmd_mem(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_mem(int argc, const char **argv)
{
struct stat st;
struct perf_mem mem = {
@@ -342,6 +347,7 @@
.lost = perf_event__process_lost,
.fork = perf_event__process_fork,
.build_id = perf_event__process_build_id,
+ .namespaces = perf_event__process_namespaces,
.ordered_events = true,
},
.input_name = "perf.data",
diff --git a/tools/perf/builtin-probe.c b/tools/perf/builtin-probe.c
index 1fcebc3..cf9f9e9 100644
--- a/tools/perf/builtin-probe.c
+++ b/tools/perf/builtin-probe.c
@@ -442,9 +442,9 @@
}
if (ret == -ENOENT && ret2 == -ENOENT)
- pr_debug("\"%s\" does not hit any event.\n", str);
- /* Note that this is silently ignored */
- ret = 0;
+ pr_warning("\"%s\" does not hit any event.\n", str);
+ else
+ ret = 0;
error:
if (kfd >= 0)
@@ -468,7 +468,7 @@
static int
-__cmd_probe(int argc, const char **argv, const char *prefix __maybe_unused)
+__cmd_probe(int argc, const char **argv)
{
const char * const probe_usage[] = {
"perf probe [<options>] 'PROBEDEF' ['PROBEDEF' ...]",
@@ -486,7 +486,7 @@
OPT_INCR('v', "verbose", &verbose,
"be more verbose (show parsed arguments, etc)"),
OPT_BOOLEAN('q', "quiet", ¶ms.quiet,
- "be quiet (do not show any mesages)"),
+ "be quiet (do not show any messages)"),
OPT_CALLBACK_DEFAULT('l', "list", NULL, "[GROUP:]EVENT",
"list up probe events",
opt_set_filter_with_command, DEFAULT_LIST_FILTER),
@@ -687,13 +687,13 @@
return 0;
}
-int cmd_probe(int argc, const char **argv, const char *prefix)
+int cmd_probe(int argc, const char **argv)
{
int ret;
ret = init_params();
if (!ret) {
- ret = __cmd_probe(argc, argv, prefix);
+ ret = __cmd_probe(argc, argv);
cleanup_params();
}
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index bc84a37..ee7d0a8 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -38,11 +38,18 @@
#include "util/bpf-loader.h"
#include "util/trigger.h"
#include "util/perf-hooks.h"
+#include "util/time-utils.h"
+#include "util/units.h"
#include "asm/bug.h"
+#include <errno.h>
+#include <inttypes.h>
+#include <poll.h>
#include <unistd.h>
#include <sched.h>
+#include <signal.h>
#include <sys/mman.h>
+#include <sys/wait.h>
#include <asm/bug.h>
#include <linux/time64.h>
@@ -876,6 +883,9 @@
signal(SIGTERM, sig_handler);
signal(SIGSEGV, sigsegv_handler);
+ if (rec->opts.record_namespaces)
+ tool->namespace_events = true;
+
if (rec->opts.auxtrace_snapshot_mode || rec->switch_output.enabled) {
signal(SIGUSR2, snapshot_sig_handler);
if (rec->opts.auxtrace_snapshot_mode)
@@ -983,6 +993,7 @@
*/
if (forks) {
union perf_event *event;
+ pid_t tgid;
event = malloc(sizeof(event->comm) + machine->id_hdr_size);
if (event == NULL) {
@@ -996,10 +1007,30 @@
* cannot see a correct process name for those events.
* Synthesize COMM event to prevent it.
*/
- perf_event__synthesize_comm(tool, event,
- rec->evlist->workload.pid,
- process_synthesized_event,
- machine);
+ tgid = perf_event__synthesize_comm(tool, event,
+ rec->evlist->workload.pid,
+ process_synthesized_event,
+ machine);
+ free(event);
+
+ if (tgid == -1)
+ goto out_child;
+
+ event = malloc(sizeof(event->namespaces) +
+ (NR_NAMESPACES * sizeof(struct perf_ns_link_info)) +
+ machine->id_hdr_size);
+ if (event == NULL) {
+ err = -ENOMEM;
+ goto out_child;
+ }
+
+ /*
+ * Synthesize NAMESPACES event for the command specified.
+ */
+ perf_event__synthesize_namespaces(tool, event,
+ rec->evlist->workload.pid,
+ tgid, process_synthesized_event,
+ machine);
free(event);
perf_evlist__start_workload(rec->evlist);
@@ -1497,6 +1528,7 @@
.fork = perf_event__process_fork,
.exit = perf_event__process_exit,
.comm = perf_event__process_comm,
+ .namespaces = perf_event__process_namespaces,
.mmap = perf_event__process_mmap,
.mmap2 = perf_event__process_mmap2,
.ordered_events = true,
@@ -1611,6 +1643,8 @@
"opts", "AUX area tracing Snapshot Mode", ""),
OPT_UINTEGER(0, "proc-map-timeout", &record.opts.proc_map_timeout,
"per thread proc mmap processing timeout in ms"),
+ OPT_BOOLEAN(0, "namespaces", &record.opts.record_namespaces,
+ "Record namespaces events"),
OPT_BOOLEAN(0, "switch-events", &record.opts.record_switch_events,
"Record context switch events"),
OPT_BOOLEAN_FLAG(0, "all-kernel", &record.opts.all_kernel,
@@ -1640,7 +1674,7 @@
struct option *record_options = __record_options;
-int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_record(int argc, const char **argv)
{
int err;
struct record *rec = &record;
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 0a88670..22478ff 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -16,7 +16,6 @@
#include <linux/rbtree.h>
#include "util/symbol.h"
#include "util/callchain.h"
-#include "util/strlist.h"
#include "util/values.h"
#include "perf.h"
@@ -38,10 +37,18 @@
#include "arch/common.h"
#include "util/time-utils.h"
#include "util/auxtrace.h"
+#include "util/units.h"
#include <dlfcn.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <regex.h>
+#include <signal.h>
#include <linux/bitmap.h>
#include <linux/stringify.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
struct report {
struct perf_tool tool;
@@ -394,8 +401,7 @@
fprintf(stdout, "\n\n");
}
- if (sort_order == NULL &&
- parent_pattern == default_parent_pattern)
+ if (!quiet)
fprintf(stdout, "#\n# (%s)\n#\n", help);
if (rep->show_threads) {
@@ -682,7 +688,7 @@
CALLCHAIN_REPORT_HELP
"\n\t\t\t\tDefault: " CALLCHAIN_DEFAULT_OPT;
-int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_report(int argc, const char **argv)
{
struct perf_session *session;
struct itrace_synth_opts itrace_synth_opts = { .set = 0, };
@@ -701,6 +707,7 @@
.mmap = perf_event__process_mmap,
.mmap2 = perf_event__process_mmap2,
.comm = perf_event__process_comm,
+ .namespaces = perf_event__process_namespaces,
.exit = perf_event__process_exit,
.fork = perf_event__process_fork,
.lost = perf_event__process_lost,
@@ -845,6 +852,8 @@
stdio__config_color, "always"),
OPT_STRING(0, "time", &report.time_str, "str",
"Time span of interest (start,stop)"),
+ OPT_BOOLEAN(0, "inline", &symbol_conf.inline_name,
+ "Show inline function"),
OPT_END()
};
struct perf_data_file file = {
diff --git a/tools/perf/builtin-sched.c b/tools/perf/builtin-sched.c
index b94cf0d..39996c5 100644
--- a/tools/perf/builtin-sched.c
+++ b/tools/perf/builtin-sched.c
@@ -22,16 +22,21 @@
#include "util/debug.h"
+#include <linux/kernel.h>
#include <linux/log2.h>
#include <sys/prctl.h>
#include <sys/resource.h>
+#include <inttypes.h>
+#include <errno.h>
#include <semaphore.h>
#include <pthread.h>
#include <math.h>
#include <api/fs/fs.h>
#include <linux/time64.h>
+#include "sane_ctype.h"
+
#define PR_SET_NAME 15 /* Set process name */
#define MAX_CPUS 4096
#define COMM_LEN 20
@@ -221,6 +226,7 @@
unsigned int max_stack;
bool show_cpu_visual;
bool show_wakeups;
+ bool show_next;
bool show_migrations;
bool show_state;
u64 skipped_samples;
@@ -1897,14 +1903,18 @@
}
static void timehist_print_sample(struct perf_sched *sched,
+ struct perf_evsel *evsel,
struct perf_sample *sample,
struct addr_location *al,
struct thread *thread,
u64 t, int state)
{
struct thread_runtime *tr = thread__priv(thread);
+ const char *next_comm = perf_evsel__strval(evsel, sample, "next_comm");
+ const u32 next_pid = perf_evsel__intval(evsel, sample, "next_pid");
u32 max_cpus = sched->max_cpu + 1;
char tstr[64];
+ char nstr[30];
u64 wait_time;
timestamp__scnprintf_usec(t, tstr, sizeof(tstr));
@@ -1937,7 +1947,12 @@
if (sched->show_state)
printf(" %5c ", task_state_char(thread, state));
- if (sched->show_wakeups)
+ if (sched->show_next) {
+ snprintf(nstr, sizeof(nstr), "next: %s[%d]", next_comm, next_pid);
+ printf(" %-*s", comm_width, nstr);
+ }
+
+ if (sched->show_wakeups && !sched->show_next)
printf(" %-*s", comm_width, "");
if (thread->tid == 0)
@@ -2531,7 +2546,7 @@
}
if (!sched->summary_only)
- timehist_print_sample(sched, sample, &al, thread, t, state);
+ timehist_print_sample(sched, evsel, sample, &al, thread, t, state);
out:
if (sched->hist_time.start == 0 && t >= ptime->start)
@@ -3262,16 +3277,17 @@
BUG_ON(i != rec_argc);
- return cmd_record(i, rec_argv, NULL);
+ return cmd_record(i, rec_argv);
}
-int cmd_sched(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_sched(int argc, const char **argv)
{
const char default_sort_order[] = "avg, max, switch, runtime";
struct perf_sched sched = {
.tool = {
.sample = perf_sched__process_tracepoint_sample,
.comm = perf_event__process_comm,
+ .namespaces = perf_event__process_namespaces,
.lost = perf_event__process_lost,
.fork = perf_sched__process_fork_event,
.ordered_events = true,
@@ -3340,6 +3356,7 @@
OPT_BOOLEAN('S', "with-summary", &sched.summary,
"Show all syscalls and summary with statistics"),
OPT_BOOLEAN('w', "wakeups", &sched.show_wakeups, "Show wakeup events"),
+ OPT_BOOLEAN('n', "next", &sched.show_next, "Show next task"),
OPT_BOOLEAN('M', "migrations", &sched.show_migrations, "Show migration events"),
OPT_BOOLEAN('V', "cpu-visual", &sched.show_cpu_visual, "Add CPU visual"),
OPT_BOOLEAN('I', "idle-hist", &sched.idle_hist, "Show idle events only"),
@@ -3400,7 +3417,7 @@
* Aliased to 'perf script' for now:
*/
if (!strcmp(argv[0], "script"))
- return cmd_script(argc, argv, prefix);
+ return cmd_script(argc, argv);
if (!strncmp(argv[0], "rec", 3)) {
return __cmd_record(argc, argv);
@@ -3437,10 +3454,14 @@
if (argc)
usage_with_options(timehist_usage, timehist_options);
}
- if (sched.show_wakeups && sched.summary_only) {
- pr_err(" Error: -s and -w are mutually exclusive.\n");
+ if ((sched.show_wakeups || sched.show_next) &&
+ sched.summary_only) {
+ pr_err(" Error: -s and -[n|w] are mutually exclusive.\n");
parse_options_usage(timehist_usage, timehist_options, "s", true);
- parse_options_usage(NULL, timehist_options, "w", true);
+ if (sched.show_wakeups)
+ parse_options_usage(NULL, timehist_options, "w", true);
+ if (sched.show_next)
+ parse_options_usage(NULL, timehist_options, "n", true);
return -EINVAL;
}
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index c0783b4..d05aec4 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -21,13 +21,27 @@
#include "util/cpumap.h"
#include "util/thread_map.h"
#include "util/stat.h"
+#include "util/string2.h"
#include "util/thread-stack.h"
#include "util/time-utils.h"
+#include "print_binary.h"
#include <linux/bitmap.h>
+#include <linux/kernel.h>
#include <linux/stringify.h>
#include <linux/time64.h>
#include "asm/bug.h"
#include "util/mem-events.h"
+#include "util/dump-insn.h"
+#include <dirent.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <signal.h>
+#include <sys/param.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "sane_ctype.h"
static char const *script_name;
static char const *generate_script_lang;
@@ -42,6 +56,7 @@
static const char *cpu_list;
static DECLARE_BITMAP(cpu_bitmap, MAX_NR_CPUS);
static struct perf_stat_config stat_config;
+static int max_blocks;
unsigned int scripting_max_stack = PERF_MAX_STACK_DEPTH;
@@ -69,6 +84,7 @@
PERF_OUTPUT_CALLINDENT = 1U << 20,
PERF_OUTPUT_INSN = 1U << 21,
PERF_OUTPUT_INSNLEN = 1U << 22,
+ PERF_OUTPUT_BRSTACKINSN = 1U << 23,
};
struct output_option {
@@ -98,6 +114,7 @@
{.str = "callindent", .field = PERF_OUTPUT_CALLINDENT},
{.str = "insn", .field = PERF_OUTPUT_INSN},
{.str = "insnlen", .field = PERF_OUTPUT_INSNLEN},
+ {.str = "brstackinsn", .field = PERF_OUTPUT_BRSTACKINSN},
};
/* default set to maintain compatibility with current format */
@@ -292,7 +309,13 @@
"selected. Hence, no address to lookup the source line number.\n");
return -EINVAL;
}
-
+ if (PRINT_FIELD(BRSTACKINSN) &&
+ !(perf_evlist__combined_branch_type(session->evlist) &
+ PERF_SAMPLE_BRANCH_ANY)) {
+ pr_err("Display of branch stack assembler requested, but non all-branch filter set\n"
+ "Hint: run 'perf record -b ...'\n");
+ return -EINVAL;
+ }
if ((PRINT_FIELD(PID) || PRINT_FIELD(TID)) &&
perf_evsel__check_stype(evsel, PERF_SAMPLE_TID, "TID",
PERF_OUTPUT_TID|PERF_OUTPUT_PID))
@@ -546,6 +569,233 @@
}
}
+#define MAXBB 16384UL
+
+static int grab_bb(u8 *buffer, u64 start, u64 end,
+ struct machine *machine, struct thread *thread,
+ bool *is64bit, u8 *cpumode, bool last)
+{
+ long offset, len;
+ struct addr_location al;
+ bool kernel;
+
+ if (!start || !end)
+ return 0;
+
+ kernel = machine__kernel_ip(machine, start);
+ if (kernel)
+ *cpumode = PERF_RECORD_MISC_KERNEL;
+ else
+ *cpumode = PERF_RECORD_MISC_USER;
+
+ /*
+ * Block overlaps between kernel and user.
+ * This can happen due to ring filtering
+ * On Intel CPUs the entry into the kernel is filtered,
+ * but the exit is not. Let the caller patch it up.
+ */
+ if (kernel != machine__kernel_ip(machine, end)) {
+ printf("\tblock %" PRIx64 "-%" PRIx64 " transfers between kernel and user\n",
+ start, end);
+ return -ENXIO;
+ }
+
+ memset(&al, 0, sizeof(al));
+ if (end - start > MAXBB - MAXINSN) {
+ if (last)
+ printf("\tbrstack does not reach to final jump (%" PRIx64 "-%" PRIx64 ")\n", start, end);
+ else
+ printf("\tblock %" PRIx64 "-%" PRIx64 " (%" PRIu64 ") too long to dump\n", start, end, end - start);
+ return 0;
+ }
+
+ thread__find_addr_map(thread, *cpumode, MAP__FUNCTION, start, &al);
+ if (!al.map || !al.map->dso) {
+ printf("\tcannot resolve %" PRIx64 "-%" PRIx64 "\n", start, end);
+ return 0;
+ }
+ if (al.map->dso->data.status == DSO_DATA_STATUS_ERROR) {
+ printf("\tcannot resolve %" PRIx64 "-%" PRIx64 "\n", start, end);
+ return 0;
+ }
+
+ /* Load maps to ensure dso->is_64_bit has been updated */
+ map__load(al.map);
+
+ offset = al.map->map_ip(al.map, start);
+ len = dso__data_read_offset(al.map->dso, machine, offset, (u8 *)buffer,
+ end - start + MAXINSN);
+
+ *is64bit = al.map->dso->is_64_bit;
+ if (len <= 0)
+ printf("\tcannot fetch code for block at %" PRIx64 "-%" PRIx64 "\n",
+ start, end);
+ return len;
+}
+
+static void print_jump(uint64_t ip, struct branch_entry *en,
+ struct perf_insn *x, u8 *inbuf, int len,
+ int insn)
+{
+ printf("\t%016" PRIx64 "\t%-30s\t#%s%s%s%s",
+ ip,
+ dump_insn(x, ip, inbuf, len, NULL),
+ en->flags.predicted ? " PRED" : "",
+ en->flags.mispred ? " MISPRED" : "",
+ en->flags.in_tx ? " INTX" : "",
+ en->flags.abort ? " ABORT" : "");
+ if (en->flags.cycles) {
+ printf(" %d cycles", en->flags.cycles);
+ if (insn)
+ printf(" %.2f IPC", (float)insn / en->flags.cycles);
+ }
+ putchar('\n');
+}
+
+static void print_ip_sym(struct thread *thread, u8 cpumode, int cpu,
+ uint64_t addr, struct symbol **lastsym,
+ struct perf_event_attr *attr)
+{
+ struct addr_location al;
+ int off;
+
+ memset(&al, 0, sizeof(al));
+
+ thread__find_addr_map(thread, cpumode, MAP__FUNCTION, addr, &al);
+ if (!al.map)
+ thread__find_addr_map(thread, cpumode, MAP__VARIABLE,
+ addr, &al);
+ if ((*lastsym) && al.addr >= (*lastsym)->start && al.addr < (*lastsym)->end)
+ return;
+
+ al.cpu = cpu;
+ al.sym = NULL;
+ if (al.map)
+ al.sym = map__find_symbol(al.map, al.addr);
+
+ if (!al.sym)
+ return;
+
+ if (al.addr < al.sym->end)
+ off = al.addr - al.sym->start;
+ else
+ off = al.addr - al.map->start - al.sym->start;
+ printf("\t%s", al.sym->name);
+ if (off)
+ printf("%+d", off);
+ putchar(':');
+ if (PRINT_FIELD(SRCLINE))
+ map__fprintf_srcline(al.map, al.addr, "\t", stdout);
+ putchar('\n');
+ *lastsym = al.sym;
+}
+
+static void print_sample_brstackinsn(struct perf_sample *sample,
+ struct thread *thread,
+ struct perf_event_attr *attr,
+ struct machine *machine)
+{
+ struct branch_stack *br = sample->branch_stack;
+ u64 start, end;
+ int i, insn, len, nr, ilen;
+ struct perf_insn x;
+ u8 buffer[MAXBB];
+ unsigned off;
+ struct symbol *lastsym = NULL;
+
+ if (!(br && br->nr))
+ return;
+ nr = br->nr;
+ if (max_blocks && nr > max_blocks + 1)
+ nr = max_blocks + 1;
+
+ x.thread = thread;
+ x.cpu = sample->cpu;
+
+ putchar('\n');
+
+ /* Handle first from jump, of which we don't know the entry. */
+ len = grab_bb(buffer, br->entries[nr-1].from,
+ br->entries[nr-1].from,
+ machine, thread, &x.is64bit, &x.cpumode, false);
+ if (len > 0) {
+ print_ip_sym(thread, x.cpumode, x.cpu,
+ br->entries[nr - 1].from, &lastsym, attr);
+ print_jump(br->entries[nr - 1].from, &br->entries[nr - 1],
+ &x, buffer, len, 0);
+ }
+
+ /* Print all blocks */
+ for (i = nr - 2; i >= 0; i--) {
+ if (br->entries[i].from || br->entries[i].to)
+ pr_debug("%d: %" PRIx64 "-%" PRIx64 "\n", i,
+ br->entries[i].from,
+ br->entries[i].to);
+ start = br->entries[i + 1].to;
+ end = br->entries[i].from;
+
+ len = grab_bb(buffer, start, end, machine, thread, &x.is64bit, &x.cpumode, false);
+ /* Patch up missing kernel transfers due to ring filters */
+ if (len == -ENXIO && i > 0) {
+ end = br->entries[--i].from;
+ pr_debug("\tpatching up to %" PRIx64 "-%" PRIx64 "\n", start, end);
+ len = grab_bb(buffer, start, end, machine, thread, &x.is64bit, &x.cpumode, false);
+ }
+ if (len <= 0)
+ continue;
+
+ insn = 0;
+ for (off = 0;; off += ilen) {
+ uint64_t ip = start + off;
+
+ print_ip_sym(thread, x.cpumode, x.cpu, ip, &lastsym, attr);
+ if (ip == end) {
+ print_jump(ip, &br->entries[i], &x, buffer + off, len - off, insn);
+ break;
+ } else {
+ printf("\t%016" PRIx64 "\t%s\n", ip,
+ dump_insn(&x, ip, buffer + off, len - off, &ilen));
+ if (ilen == 0)
+ break;
+ insn++;
+ }
+ }
+ }
+
+ /*
+ * Hit the branch? In this case we are already done, and the target
+ * has not been executed yet.
+ */
+ if (br->entries[0].from == sample->ip)
+ return;
+ if (br->entries[0].flags.abort)
+ return;
+
+ /*
+ * Print final block upto sample
+ */
+ start = br->entries[0].to;
+ end = sample->ip;
+ len = grab_bb(buffer, start, end, machine, thread, &x.is64bit, &x.cpumode, true);
+ print_ip_sym(thread, x.cpumode, x.cpu, start, &lastsym, attr);
+ if (len <= 0) {
+ /* Print at least last IP if basic block did not work */
+ len = grab_bb(buffer, sample->ip, sample->ip,
+ machine, thread, &x.is64bit, &x.cpumode, false);
+ if (len <= 0)
+ return;
+
+ printf("\t%016" PRIx64 "\t%s\n", sample->ip,
+ dump_insn(&x, sample->ip, buffer, len, NULL));
+ return;
+ }
+ for (off = 0; off <= end - start; off += ilen) {
+ printf("\t%016" PRIx64 "\t%s\n", start + off,
+ dump_insn(&x, start + off, buffer + off, len - off, &ilen));
+ if (ilen == 0)
+ break;
+ }
+}
static void print_sample_addr(struct perf_sample *sample,
struct thread *thread,
@@ -632,7 +882,9 @@
}
static void print_insn(struct perf_sample *sample,
- struct perf_event_attr *attr)
+ struct perf_event_attr *attr,
+ struct thread *thread,
+ struct machine *machine)
{
if (PRINT_FIELD(INSNLEN))
printf(" ilen: %d", sample->insn_len);
@@ -643,12 +895,15 @@
for (i = 0; i < sample->insn_len; i++)
printf(" %02x", (unsigned char)sample->insn[i]);
}
+ if (PRINT_FIELD(BRSTACKINSN))
+ print_sample_brstackinsn(sample, thread, attr, machine);
}
static void print_sample_bts(struct perf_sample *sample,
struct perf_evsel *evsel,
struct thread *thread,
- struct addr_location *al)
+ struct addr_location *al,
+ struct machine *machine)
{
struct perf_event_attr *attr = &evsel->attr;
bool print_srcline_last = false;
@@ -689,7 +944,7 @@
if (print_srcline_last)
map__fprintf_srcline(al->map, al->addr, "\n ", stdout);
- print_insn(sample, attr);
+ print_insn(sample, attr, thread, machine);
printf("\n");
}
@@ -830,6 +1085,7 @@
bool show_task_events;
bool show_mmap_events;
bool show_switch_events;
+ bool show_namespace_events;
bool allocated;
struct cpu_map *cpus;
struct thread_map *threads;
@@ -871,7 +1127,8 @@
static void process_event(struct perf_script *script,
struct perf_sample *sample, struct perf_evsel *evsel,
- struct addr_location *al)
+ struct addr_location *al,
+ struct machine *machine)
{
struct thread *thread = al->thread;
struct perf_event_attr *attr = &evsel->attr;
@@ -898,7 +1155,7 @@
print_sample_flags(sample->flags);
if (is_bts_event(attr)) {
- print_sample_bts(sample, evsel, thread, al);
+ print_sample_bts(sample, evsel, thread, al, machine);
return;
}
@@ -936,7 +1193,7 @@
if (perf_evsel__is_bpf_output(evsel) && PRINT_FIELD(BPF_OUTPUT))
print_sample_bpf_output(sample);
- print_insn(sample, attr);
+ print_insn(sample, attr, thread, machine);
printf("\n");
}
@@ -1046,7 +1303,7 @@
if (scripting_ops)
scripting_ops->process_event(event, sample, evsel, &al);
else
- process_event(scr, sample, evsel, &al);
+ process_event(scr, sample, evsel, &al, machine);
out_put:
addr_location__put(&al);
@@ -1118,6 +1375,41 @@
return ret;
}
+static int process_namespaces_event(struct perf_tool *tool,
+ union perf_event *event,
+ struct perf_sample *sample,
+ struct machine *machine)
+{
+ struct thread *thread;
+ struct perf_script *script = container_of(tool, struct perf_script, tool);
+ struct perf_session *session = script->session;
+ struct perf_evsel *evsel = perf_evlist__id2evsel(session->evlist, sample->id);
+ int ret = -1;
+
+ thread = machine__findnew_thread(machine, event->namespaces.pid,
+ event->namespaces.tid);
+ if (thread == NULL) {
+ pr_debug("problem processing NAMESPACES event, skipping it.\n");
+ return -1;
+ }
+
+ if (perf_event__process_namespaces(tool, event, sample, machine) < 0)
+ goto out;
+
+ if (!evsel->attr.sample_id_all) {
+ sample->cpu = 0;
+ sample->time = 0;
+ sample->tid = event->namespaces.tid;
+ sample->pid = event->namespaces.pid;
+ }
+ print_sample_start(sample, thread, evsel);
+ perf_event__fprintf(event, stdout);
+ ret = 0;
+out:
+ thread__put(thread);
+ return ret;
+}
+
static int process_fork_event(struct perf_tool *tool,
union perf_event *event,
struct perf_sample *sample,
@@ -1293,6 +1585,8 @@
}
if (script->show_switch_events)
script->tool.context_switch = process_switch_event;
+ if (script->show_namespace_events)
+ script->tool.namespaces = process_namespaces_event;
ret = perf_session__process_events(script->session);
@@ -1427,7 +1721,7 @@
static int parse_output_fields(const struct option *opt __maybe_unused,
const char *arg, int unset __maybe_unused)
{
- char *tok;
+ char *tok, *strtok_saveptr = NULL;
int i, imax = ARRAY_SIZE(all_output_options);
int j;
int rc = 0;
@@ -1488,7 +1782,7 @@
}
}
- for (tok = strtok(tok, ","); tok; tok = strtok(NULL, ",")) {
+ for (tok = strtok_r(tok, ",", &strtok_saveptr); tok; tok = strtok_r(NULL, ",", &strtok_saveptr)) {
for (i = 0; i < imax; ++i) {
if (strcmp(tok, all_output_options[i].str) == 0)
break;
@@ -2078,7 +2372,7 @@
return set_maps(script);
}
-int cmd_script(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_script(int argc, const char **argv)
{
bool show_full_info = false;
bool header = false;
@@ -2097,6 +2391,7 @@
.mmap = perf_event__process_mmap,
.mmap2 = perf_event__process_mmap2,
.comm = perf_event__process_comm,
+ .namespaces = perf_event__process_namespaces,
.exit = perf_event__process_exit,
.fork = perf_event__process_fork,
.attr = process_attr,
@@ -2152,7 +2447,8 @@
"Valid types: hw,sw,trace,raw. "
"Fields: comm,tid,pid,time,cpu,event,trace,ip,sym,dso,"
"addr,symoff,period,iregs,brstack,brstacksym,flags,"
- "bpf-output,callindent,insn,insnlen", parse_output_fields),
+ "bpf-output,callindent,insn,insnlen,brstackinsn",
+ parse_output_fields),
OPT_BOOLEAN('a', "all-cpus", &system_wide,
"system-wide collection from all CPUs"),
OPT_STRING('S', "symbols", &symbol_conf.sym_list_str, "symbol[,symbol...]",
@@ -2180,7 +2476,11 @@
"Show the mmap events"),
OPT_BOOLEAN('\0', "show-switch-events", &script.show_switch_events,
"Show context switch events (if recorded)"),
+ OPT_BOOLEAN('\0', "show-namespace-events", &script.show_namespace_events,
+ "Show namespace events (if recorded)"),
OPT_BOOLEAN('f', "force", &symbol_conf.force, "don't complain, do it"),
+ OPT_INTEGER(0, "max-blocks", &max_blocks,
+ "Maximum number of code blocks to dump with brstackinsn"),
OPT_BOOLEAN(0, "ns", &nanosecs,
"Use 9 decimal places when displaying time"),
OPT_CALLBACK_OPTARG(0, "itrace", &itrace_synth_opts, NULL, "opts",
@@ -2217,7 +2517,7 @@
if (argc > 1 && !strncmp(argv[0], "rec", strlen("rec"))) {
rec_script_path = get_script_path(argv[1], RECORD_SUFFIX);
if (!rec_script_path)
- return cmd_record(argc, argv, NULL);
+ return cmd_record(argc, argv);
}
if (argc > 1 && !strncmp(argv[0], "rep", strlen("rep"))) {
diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index 13b5499..a935b50 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -64,14 +64,24 @@
#include "util/session.h"
#include "util/tool.h"
#include "util/group.h"
+#include "util/string2.h"
#include "asm/bug.h"
#include <linux/time64.h>
#include <api/fs/fs.h>
+#include <errno.h>
+#include <signal.h>
#include <stdlib.h>
#include <sys/prctl.h>
+#include <inttypes.h>
#include <locale.h>
#include <math.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+#include "sane_ctype.h"
#define DEFAULT_SEPARATOR " "
#define CNTR_NOT_SUPPORTED "<not supported>"
@@ -140,12 +150,14 @@
static bool forever = false;
static bool metric_only = false;
static bool force_metric_only = false;
+static bool no_merge = false;
static struct timespec ref_time;
static struct cpu_map *aggr_map;
static aggr_get_id_t aggr_get_id;
static bool append_file;
static const char *output_name;
static int output_fd;
+static int print_free_counters_hint;
struct perf_stat {
bool record;
@@ -310,8 +322,12 @@
struct perf_counts_values *count;
count = perf_counts(counter->counts, cpu, thread);
- if (perf_evsel__read(counter, cpu, thread, count))
+ if (perf_evsel__read(counter, cpu, thread, count)) {
+ counter->counts->scaled = -1;
+ perf_counts(counter->counts, cpu, thread)->ena = 0;
+ perf_counts(counter->counts, cpu, thread)->run = 0;
return -1;
+ }
if (STAT_RECORD) {
if (perf_evsel__write_stat_event(counter, cpu, thread, count)) {
@@ -336,12 +352,14 @@
static void read_counters(void)
{
struct perf_evsel *counter;
+ int ret;
evlist__for_each_entry(evsel_list, counter) {
- if (read_counter(counter))
+ ret = read_counter(counter);
+ if (ret)
pr_debug("failed to read counter %s\n", counter->name);
- if (perf_stat_process_counter(&stat_config, counter))
+ if (ret == 0 && perf_stat_process_counter(&stat_config, counter))
pr_warning("failed to process counter %s\n", counter->name);
}
}
@@ -873,10 +891,7 @@
return;
}
snprintf(buf, sizeof(buf), fmt, val);
- vals = buf;
- while (isspace(*vals))
- vals++;
- ends = vals;
+ ends = vals = ltrim(buf);
while (isdigit(*ends) || *ends == '.')
ends++;
*ends = 0;
@@ -948,10 +963,7 @@
return;
unit = fixunit(tbuf, os->evsel, unit);
snprintf(buf, sizeof buf, fmt, val);
- vals = buf;
- while (isspace(*vals))
- vals++;
- ends = vals;
+ ends = vals = ltrim(buf);
while (isdigit(*ends) || *ends == '.')
ends++;
*ends = 0;
@@ -1109,6 +1121,9 @@
counter->supported ? CNTR_NOT_COUNTED : CNTR_NOT_SUPPORTED,
csv_sep);
+ if (counter->supported)
+ print_free_counters_hint = 1;
+
fprintf(stat_config.output, "%-*s%s",
csv_output ? 0 : unit_width,
counter->unit, csv_sep);
@@ -1140,6 +1155,7 @@
out.print_metric = pm;
out.new_line = nl;
out.ctx = &os;
+ out.force_header = false;
if (csv_output && !metric_only) {
print_noise(counter, noise);
@@ -1178,11 +1194,81 @@
}
}
+static void collect_all_aliases(struct perf_evsel *counter,
+ void (*cb)(struct perf_evsel *counter, void *data,
+ bool first),
+ void *data)
+{
+ struct perf_evsel *alias;
+
+ alias = list_prepare_entry(counter, &(evsel_list->entries), node);
+ list_for_each_entry_continue (alias, &evsel_list->entries, node) {
+ if (strcmp(perf_evsel__name(alias), perf_evsel__name(counter)) ||
+ alias->scale != counter->scale ||
+ alias->cgrp != counter->cgrp ||
+ strcmp(alias->unit, counter->unit) ||
+ nsec_counter(alias) != nsec_counter(counter))
+ break;
+ alias->merged_stat = true;
+ cb(alias, data, false);
+ }
+}
+
+static bool collect_data(struct perf_evsel *counter,
+ void (*cb)(struct perf_evsel *counter, void *data,
+ bool first),
+ void *data)
+{
+ if (counter->merged_stat)
+ return false;
+ cb(counter, data, true);
+ if (!no_merge)
+ collect_all_aliases(counter, cb, data);
+ return true;
+}
+
+struct aggr_data {
+ u64 ena, run, val;
+ int id;
+ int nr;
+ int cpu;
+};
+
+static void aggr_cb(struct perf_evsel *counter, void *data, bool first)
+{
+ struct aggr_data *ad = data;
+ int cpu, s2;
+
+ for (cpu = 0; cpu < perf_evsel__nr_cpus(counter); cpu++) {
+ struct perf_counts_values *counts;
+
+ s2 = aggr_get_id(perf_evsel__cpus(counter), cpu);
+ if (s2 != ad->id)
+ continue;
+ if (first)
+ ad->nr++;
+ counts = perf_counts(counter->counts, cpu, 0);
+ /*
+ * When any result is bad, make them all to give
+ * consistent output in interval mode.
+ */
+ if (counts->ena == 0 || counts->run == 0 ||
+ counter->counts->scaled == -1) {
+ ad->ena = 0;
+ ad->run = 0;
+ break;
+ }
+ ad->val += counts->val;
+ ad->ena += counts->ena;
+ ad->run += counts->run;
+ }
+}
+
static void print_aggr(char *prefix)
{
FILE *output = stat_config.output;
struct perf_evsel *counter;
- int cpu, s, s2, id, nr;
+ int s, id, nr;
double uval;
u64 ena, run, val;
bool first;
@@ -1197,23 +1283,21 @@
* Without each counter has its own line.
*/
for (s = 0; s < aggr_map->nr; s++) {
+ struct aggr_data ad;
if (prefix && metric_only)
fprintf(output, "%s", prefix);
- id = aggr_map->map[s];
+ ad.id = id = aggr_map->map[s];
first = true;
evlist__for_each_entry(evsel_list, counter) {
- val = ena = run = 0;
- nr = 0;
- for (cpu = 0; cpu < perf_evsel__nr_cpus(counter); cpu++) {
- s2 = aggr_get_id(perf_evsel__cpus(counter), cpu);
- if (s2 != id)
- continue;
- val += perf_counts(counter->counts, cpu, 0)->val;
- ena += perf_counts(counter->counts, cpu, 0)->ena;
- run += perf_counts(counter->counts, cpu, 0)->run;
- nr++;
- }
+ ad.val = ad.ena = ad.run = 0;
+ ad.nr = 0;
+ if (!collect_data(counter, aggr_cb, &ad))
+ continue;
+ nr = ad.nr;
+ ena = ad.ena;
+ run = ad.run;
+ val = ad.val;
if (first && metric_only) {
first = false;
aggr_printout(counter, id, nr);
@@ -1257,6 +1341,21 @@
}
}
+struct caggr_data {
+ double avg, avg_enabled, avg_running;
+};
+
+static void counter_aggr_cb(struct perf_evsel *counter, void *data,
+ bool first __maybe_unused)
+{
+ struct caggr_data *cd = data;
+ struct perf_stat_evsel *ps = counter->priv;
+
+ cd->avg += avg_stats(&ps->res_stats[0]);
+ cd->avg_enabled += avg_stats(&ps->res_stats[1]);
+ cd->avg_running += avg_stats(&ps->res_stats[2]);
+}
+
/*
* Print out the results of a single counter:
* aggregated counts in system-wide mode
@@ -1264,23 +1363,31 @@
static void print_counter_aggr(struct perf_evsel *counter, char *prefix)
{
FILE *output = stat_config.output;
- struct perf_stat_evsel *ps = counter->priv;
- double avg = avg_stats(&ps->res_stats[0]);
double uval;
- double avg_enabled, avg_running;
+ struct caggr_data cd = { .avg = 0.0 };
- avg_enabled = avg_stats(&ps->res_stats[1]);
- avg_running = avg_stats(&ps->res_stats[2]);
+ if (!collect_data(counter, counter_aggr_cb, &cd))
+ return;
if (prefix && !metric_only)
fprintf(output, "%s", prefix);
- uval = avg * counter->scale;
- printout(-1, 0, counter, uval, prefix, avg_running, avg_enabled, avg);
+ uval = cd.avg * counter->scale;
+ printout(-1, 0, counter, uval, prefix, cd.avg_running, cd.avg_enabled, cd.avg);
if (!metric_only)
fprintf(output, "\n");
}
+static void counter_cb(struct perf_evsel *counter, void *data,
+ bool first __maybe_unused)
+{
+ struct aggr_data *ad = data;
+
+ ad->val += perf_counts(counter->counts, ad->cpu, 0)->val;
+ ad->ena += perf_counts(counter->counts, ad->cpu, 0)->ena;
+ ad->run += perf_counts(counter->counts, ad->cpu, 0)->run;
+}
+
/*
* Print out the results of a single counter:
* does not use aggregated count in system-wide
@@ -1293,9 +1400,13 @@
int cpu;
for (cpu = 0; cpu < perf_evsel__nr_cpus(counter); cpu++) {
- val = perf_counts(counter->counts, cpu, 0)->val;
- ena = perf_counts(counter->counts, cpu, 0)->ena;
- run = perf_counts(counter->counts, cpu, 0)->run;
+ struct aggr_data ad = { .cpu = cpu };
+
+ if (!collect_data(counter, counter_cb, &ad))
+ return;
+ val = ad.val;
+ ena = ad.ena;
+ run = ad.run;
if (prefix)
fprintf(output, "%s", prefix);
@@ -1380,6 +1491,7 @@
out.ctx = &os;
out.print_metric = print_metric_header;
out.new_line = new_line_metric;
+ out.force_header = true;
os.evsel = counter;
perf_stat__print_shadow_stats(counter, 0,
0,
@@ -1477,6 +1589,13 @@
avg_stats(&walltime_nsecs_stats));
}
fprintf(output, "\n\n");
+
+ if (print_free_counters_hint)
+ fprintf(output,
+"Some events weren't counted. Try disabling the NMI watchdog:\n"
+" echo 0 > /proc/sys/kernel/nmi_watchdog\n"
+" perf stat ...\n"
+" echo 1 > /proc/sys/kernel/nmi_watchdog\n");
}
static void print_counters(struct timespec *ts, int argc, const char **argv)
@@ -1633,6 +1752,7 @@
"list of cpus to monitor in system-wide"),
OPT_SET_UINT('A', "no-aggr", &stat_config.aggr_mode,
"disable CPU count aggregation", AGGR_NONE),
+ OPT_BOOLEAN(0, "no-merge", &no_merge, "Do not merge identical named events"),
OPT_STRING('x', "field-separator", &csv_sep, "separator",
"print counts with custom separator"),
OPT_CALLBACK('G', "cgroup", &evsel_list, "name",
@@ -2339,7 +2459,36 @@
return 0;
}
-int cmd_stat(int argc, const char **argv, const char *prefix __maybe_unused)
+static void setup_system_wide(int forks)
+{
+ /*
+ * Make system wide (-a) the default target if
+ * no target was specified and one of following
+ * conditions is met:
+ *
+ * - there's no workload specified
+ * - there is workload specified but all requested
+ * events are system wide events
+ */
+ if (!target__none(&target))
+ return;
+
+ if (!forks)
+ target.system_wide = true;
+ else {
+ struct perf_evsel *counter;
+
+ evlist__for_each_entry(evsel_list, counter) {
+ if (!counter->system_wide)
+ return;
+ }
+
+ if (evsel_list->nr_entries)
+ target.system_wide = true;
+ }
+}
+
+int cmd_stat(int argc, const char **argv)
{
const char * const stat_usage[] = {
"perf stat [<options>] [<command>]",
@@ -2361,6 +2510,7 @@
argc = parse_options_subcommand(argc, argv, stat_options, stat_subcommands,
(const char **) stat_usage,
PARSE_OPT_STOP_AT_NON_OPTION);
+ perf_stat__collect_metric_expr(evsel_list);
perf_stat__init_shadow_stats();
if (csv_sep) {
@@ -2445,9 +2595,7 @@
} else if (big_num_opt == 0) /* User passed --no-big-num */
big_num = false;
- /* Make system wide (-a) the default target. */
- if (!argc && target__none(&target))
- target.system_wide = true;
+ setup_system_wide(argc);
if (run_count < 0) {
pr_err("Run count must be a positive number\n");
diff --git a/tools/perf/builtin-timechart.c b/tools/perf/builtin-timechart.c
index e7eaa29..4e2e616 100644
--- a/tools/perf/builtin-timechart.c
+++ b/tools/perf/builtin-timechart.c
@@ -12,6 +12,8 @@
* of the License.
*/
+#include <errno.h>
+#include <inttypes.h>
#include <traceevent/event-parse.h>
#include "builtin.h"
@@ -23,11 +25,12 @@
#include "util/cache.h"
#include "util/evlist.h"
#include "util/evsel.h"
+#include <linux/kernel.h>
#include <linux/rbtree.h>
#include <linux/time64.h>
#include "util/symbol.h"
+#include "util/thread.h"
#include "util/callchain.h"
-#include "util/strlist.h"
#include "perf.h"
#include "util/header.h"
@@ -1773,7 +1776,7 @@
for (i = 0; i < (unsigned int)argc; i++)
*p++ = argv[i];
- return cmd_record(rec_argc, rec_argv, NULL);
+ return cmd_record(rec_argc, rec_argv);
}
@@ -1864,7 +1867,7 @@
for (j = 0; j < (unsigned int)argc; j++)
*p++ = argv[j];
- return cmd_record(rec_argc, rec_argv, NULL);
+ return cmd_record(rec_argc, rec_argv);
}
static int
@@ -1917,8 +1920,7 @@
return 0;
}
-int cmd_timechart(int argc, const char **argv,
- const char *prefix __maybe_unused)
+int cmd_timechart(int argc, const char **argv)
{
struct timechart tchart = {
.tool = {
@@ -1933,6 +1935,11 @@
.merge_dist = 1000,
};
const char *output_name = "output.svg";
+ const struct option timechart_common_options[] = {
+ OPT_BOOLEAN('P', "power-only", &tchart.power_only, "output power data only"),
+ OPT_BOOLEAN('T', "tasks-only", &tchart.tasks_only, "output processes data only"),
+ OPT_END()
+ };
const struct option timechart_options[] = {
OPT_STRING('i', "input", &input_name, "file", "input file name"),
OPT_STRING('o', "output", &output_name, "file", "output file name"),
@@ -1940,9 +1947,6 @@
OPT_CALLBACK(0, "highlight", NULL, "duration or task name",
"highlight tasks. Pass duration in ns or process name.",
parse_highlight),
- OPT_BOOLEAN('P', "power-only", &tchart.power_only, "output power data only"),
- OPT_BOOLEAN('T', "tasks-only", &tchart.tasks_only,
- "output processes data only"),
OPT_CALLBACK('p', "process", NULL, "process",
"process selector. Pass a pid or process name.",
parse_process),
@@ -1962,22 +1966,18 @@
"merge events that are merge-dist us apart",
parse_time),
OPT_BOOLEAN('f', "force", &tchart.force, "don't complain, do it"),
- OPT_END()
+ OPT_PARENT(timechart_common_options),
};
const char * const timechart_subcommands[] = { "record", NULL };
const char *timechart_usage[] = {
"perf timechart [<options>] {record}",
NULL
};
-
const struct option timechart_record_options[] = {
- OPT_BOOLEAN('P', "power-only", &tchart.power_only, "output power data only"),
- OPT_BOOLEAN('T', "tasks-only", &tchart.tasks_only,
- "output processes data only"),
OPT_BOOLEAN('I', "io-only", &tchart.io_only,
"record only IO data"),
OPT_BOOLEAN('g', "callchain", &tchart.with_backtrace, "record callchain"),
- OPT_END()
+ OPT_PARENT(timechart_common_options),
};
const char * const timechart_record_usage[] = {
"perf timechart record [<options>]",
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index ab90779..7ab42b8 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -40,6 +40,7 @@
#include "util/cpumap.h"
#include "util/xyarray.h"
#include "util/sort.h"
+#include "util/term.h"
#include "util/intlist.h"
#include "util/parse-branch-options.h"
#include "arch/common.h"
@@ -58,6 +59,7 @@
#include <errno.h>
#include <time.h>
#include <sched.h>
+#include <signal.h>
#include <sys/syscall.h>
#include <sys/ioctl.h>
@@ -72,6 +74,8 @@
#include <linux/time64.h>
#include <linux/types.h>
+#include "sane_ctype.h"
+
static volatile int done;
#define HEADER_LINE_NR 5
@@ -1075,7 +1079,7 @@
const char top_callchain_help[] = CALLCHAIN_RECORD_HELP CALLCHAIN_REPORT_HELP
"\n\t\t\t\tDefault: fp,graph,0.5,caller,function";
-int cmd_top(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_top(int argc, const char **argv)
{
char errbuf[BUFSIZ];
struct perf_top top = {
diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
index 256f1fa..eaa66fb 100644
--- a/tools/perf/builtin-trace.c
+++ b/tools/perf/builtin-trace.c
@@ -24,6 +24,7 @@
#include "util/evlist.h"
#include <subcmd/exec-cmd.h>
#include "util/machine.h"
+#include "util/path.h"
#include "util/session.h"
#include "util/thread.h"
#include <subcmd/parse-options.h>
@@ -31,23 +32,33 @@
#include "util/intlist.h"
#include "util/thread_map.h"
#include "util/stat.h"
+#include "trace/beauty/beauty.h"
#include "trace-event.h"
#include "util/parse-events.h"
#include "util/bpf-loader.h"
#include "callchain.h"
+#include "print_binary.h"
+#include "string2.h"
#include "syscalltbl.h"
#include "rb_resort.h"
+#include <errno.h>
+#include <inttypes.h>
#include <libaudit.h> /* FIXME: Still needed for audit_errno_to_name */
+#include <poll.h>
+#include <signal.h>
#include <stdlib.h>
#include <string.h>
#include <linux/err.h>
#include <linux/filter.h>
#include <linux/audit.h>
+#include <linux/kernel.h>
#include <linux/random.h>
#include <linux/stringify.h>
#include <linux/time64.h>
+#include "sane_ctype.h"
+
#ifndef O_CLOEXEC
# define O_CLOEXEC 02000000
#endif
@@ -267,15 +278,6 @@
({ struct syscall_tp *fields = evsel->priv; \
fields->name.pointer(&fields->name, sample); })
-struct syscall_arg {
- unsigned long val;
- struct thread *thread;
- struct trace *trace;
- void *parm;
- u8 idx;
- u8 mask;
-};
-
struct strarray {
int offset;
int nr_entries;
@@ -771,6 +773,10 @@
.arg_parm = { [0] = &strarray__socket_families, /* family */ }, },
{ .name = "stat", .errmsg = true, .alias = "newstat", },
{ .name = "statfs", .errmsg = true, },
+ { .name = "statx", .errmsg = true,
+ .arg_scnprintf = { [0] = SCA_FDAT, /* flags */
+ [2] = SCA_STATX_FLAGS, /* flags */
+ [3] = SCA_STATX_MASK, /* mask */ }, },
{ .name = "swapoff", .errmsg = true,
.arg_scnprintf = { [0] = SCA_FILENAME, /* specialfile */ }, },
{ .name = "swapon", .errmsg = true,
@@ -821,12 +827,21 @@
void **arg_parm;
};
-static size_t fprintf_duration(unsigned long t, FILE *fp)
+/*
+ * We need to have this 'calculated' boolean because in some cases we really
+ * don't know what is the duration of a syscall, for instance, when we start
+ * a session and some threads are waiting for a syscall to finish, say 'poll',
+ * in which case all we can do is to print "( ? ) for duration and for the
+ * start timestamp.
+ */
+static size_t fprintf_duration(unsigned long t, bool calculated, FILE *fp)
{
double duration = (double)t / NSEC_PER_MSEC;
size_t printed = fprintf(fp, "(");
- if (duration >= 1.0)
+ if (!calculated)
+ printed += fprintf(fp, " ? ");
+ else if (duration >= 1.0)
printed += color_fprintf(fp, PERF_COLOR_RED, "%6.3f ms", duration);
else if (duration >= 0.01)
printed += color_fprintf(fp, PERF_COLOR_YELLOW, "%6.3f ms", duration);
@@ -1028,13 +1043,27 @@
return t < (trace->duration_filter * NSEC_PER_MSEC);
}
-static size_t trace__fprintf_tstamp(struct trace *trace, u64 tstamp, FILE *fp)
+static size_t __trace__fprintf_tstamp(struct trace *trace, u64 tstamp, FILE *fp)
{
double ts = (double)(tstamp - trace->base_time) / NSEC_PER_MSEC;
return fprintf(fp, "%10.3f ", ts);
}
+/*
+ * We're handling tstamp=0 as an undefined tstamp, i.e. like when we are
+ * using ttrace->entry_time for a thread that receives a sys_exit without
+ * first having received a sys_enter ("poll" issued before tracing session
+ * starts, lost sys_enter exit due to ring buffer overflow).
+ */
+static size_t trace__fprintf_tstamp(struct trace *trace, u64 tstamp, FILE *fp)
+{
+ if (tstamp > 0)
+ return __trace__fprintf_tstamp(trace, tstamp, fp);
+
+ return fprintf(fp, " ? ");
+}
+
static bool done = false;
static bool interrupted = false;
@@ -1045,10 +1074,10 @@
}
static size_t trace__fprintf_entry_head(struct trace *trace, struct thread *thread,
- u64 duration, u64 tstamp, FILE *fp)
+ u64 duration, bool duration_calculated, u64 tstamp, FILE *fp)
{
size_t printed = trace__fprintf_tstamp(trace, tstamp, fp);
- printed += fprintf_duration(duration, fp);
+ printed += fprintf_duration(duration, duration_calculated, fp);
if (trace->multiple_threads) {
if (trace->show_comm)
@@ -1450,7 +1479,7 @@
duration = sample->time - ttrace->entry_time;
- printed = trace__fprintf_entry_head(trace, trace->current, duration, ttrace->entry_time, trace->output);
+ printed = trace__fprintf_entry_head(trace, trace->current, duration, true, ttrace->entry_time, trace->output);
printed += fprintf(trace->output, "%-70s) ...\n", ttrace->entry_str);
ttrace->entry_pending = false;
@@ -1497,7 +1526,7 @@
if (sc->is_exit) {
if (!(trace->duration_filter || trace->summary_only || trace->min_stack)) {
- trace__fprintf_entry_head(trace, thread, 1, ttrace->entry_time, trace->output);
+ trace__fprintf_entry_head(trace, thread, 0, false, ttrace->entry_time, trace->output);
fprintf(trace->output, "%-70s)\n", ttrace->entry_str);
}
} else {
@@ -1545,6 +1574,7 @@
{
long ret;
u64 duration = 0;
+ bool duration_calculated = false;
struct thread *thread;
int id = perf_evsel__sc_tp_uint(evsel, id, sample), err = -1, callchain_ret = 0;
struct syscall *sc = trace__syscall_info(trace, evsel, id);
@@ -1573,6 +1603,7 @@
duration = sample->time - ttrace->entry_time;
if (trace__filter_duration(trace, duration))
goto out;
+ duration_calculated = true;
} else if (trace->duration_filter)
goto out;
@@ -1588,7 +1619,7 @@
if (trace->summary_only)
goto out;
- trace__fprintf_entry_head(trace, thread, duration, ttrace->entry_time, trace->output);
+ trace__fprintf_entry_head(trace, thread, duration, duration_calculated, ttrace->entry_time, trace->output);
if (ttrace->entry_pending) {
fprintf(trace->output, "%-70s", ttrace->entry_str);
@@ -1653,15 +1684,17 @@
ttrace = thread__priv(thread);
if (!ttrace)
- goto out;
+ goto out_put;
filename_len = strlen(filename);
+ if (filename_len == 0)
+ goto out_put;
if (ttrace->filename.namelen < filename_len) {
char *f = realloc(ttrace->filename.name, filename_len + 1);
if (f == NULL)
- goto out;
+ goto out_put;
ttrace->filename.namelen = filename_len;
ttrace->filename.name = f;
@@ -1671,12 +1704,12 @@
ttrace->filename.pending_open = true;
if (!ttrace->filename.ptr)
- goto out;
+ goto out_put;
entry_str_len = strlen(ttrace->entry_str);
remaining_space = trace__entry_str_size - entry_str_len - 1; /* \0 */
if (remaining_space <= 0)
- goto out;
+ goto out_put;
if (filename_len > (size_t)remaining_space) {
filename += filename_len - remaining_space;
@@ -1690,6 +1723,8 @@
ttrace->filename.ptr = 0;
ttrace->filename.entry_str_pos = 0;
+out_put:
+ thread__put(thread);
out:
return 0;
}
@@ -1710,6 +1745,7 @@
ttrace->runtime_ms += runtime_ms;
trace->runtime_ms += runtime_ms;
+out_put:
thread__put(thread);
return 0;
@@ -1720,8 +1756,7 @@
(pid_t)perf_evsel__intval(evsel, sample, "pid"),
runtime,
perf_evsel__intval(evsel, sample, "vruntime"));
- thread__put(thread);
- return 0;
+ goto out_put;
}
static void bpf_output__printer(enum binary_printer_ops op,
@@ -1851,7 +1886,7 @@
thread__find_addr_location(thread, sample->cpumode, MAP__FUNCTION,
sample->ip, &al);
- trace__fprintf_entry_head(trace, thread, 0, sample->time, trace->output);
+ trace__fprintf_entry_head(trace, thread, 0, true, sample->time, trace->output);
fprintf(trace->output, "%sfault [",
evsel->attr.config == PERF_COUNT_SW_PAGE_FAULTS_MAJ ?
@@ -1920,7 +1955,7 @@
thread = machine__findnew_thread(trace->host, sample->pid, sample->tid);
if (thread && thread__is_filtered(thread))
- return 0;
+ goto out;
trace__set_base_time(trace, evsel, sample);
@@ -1928,7 +1963,8 @@
++trace->nr_events;
handler(trace, evsel, event, sample);
}
-
+out:
+ thread__put(thread);
return err;
}
@@ -1988,7 +2024,7 @@
for (i = 0; i < (unsigned int)argc; i++)
rec_argv[j++] = argv[i];
- return cmd_record(j, rec_argv, NULL);
+ return cmd_record(j, rec_argv);
}
static size_t trace__fprintf_thread_summary(struct trace *trace, FILE *fp);
@@ -2415,8 +2451,9 @@
trace->tool.exit = perf_event__process_exit;
trace->tool.fork = perf_event__process_fork;
trace->tool.attr = perf_event__process_attr;
- trace->tool.tracing_data = perf_event__process_tracing_data;
+ trace->tool.tracing_data = perf_event__process_tracing_data;
trace->tool.build_id = perf_event__process_build_id;
+ trace->tool.namespaces = perf_event__process_namespaces;
trace->tool.ordered_events = true;
trace->tool.ordering_requires_timestamps = true;
@@ -2785,7 +2822,7 @@
return err;
}
-int cmd_trace(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_trace(int argc, const char **argv)
{
const char *trace_usage[] = {
"perf trace [<options>] [<command>]",
diff --git a/tools/perf/builtin-version.c b/tools/perf/builtin-version.c
index 9b10cda..d251494 100644
--- a/tools/perf/builtin-version.c
+++ b/tools/perf/builtin-version.c
@@ -1,9 +1,9 @@
-#include "util/util.h"
#include "builtin.h"
#include "perf.h"
+#include <linux/compiler.h>
+#include <stdio.h>
-int cmd_version(int argc __maybe_unused, const char **argv __maybe_unused,
- const char *prefix __maybe_unused)
+int cmd_version(int argc __maybe_unused, const char **argv __maybe_unused)
{
printf("perf version %s\n", perf_version_string);
return 0;
diff --git a/tools/perf/builtin.h b/tools/perf/builtin.h
index 036e1e3..d4d19fe 100644
--- a/tools/perf/builtin.h
+++ b/tools/perf/builtin.h
@@ -2,46 +2,42 @@
#define BUILTIN_H
#include "util/util.h"
-#include "util/strbuf.h"
extern const char perf_usage_string[];
extern const char perf_more_info_string[];
void list_common_cmds_help(void);
const char *help_unknown_cmd(const char *cmd);
-void prune_packed_objects(int);
-int read_line_with_nul(char *buf, int size, FILE *file);
-int check_pager_config(const char *cmd);
-int cmd_annotate(int argc, const char **argv, const char *prefix);
-int cmd_bench(int argc, const char **argv, const char *prefix);
-int cmd_buildid_cache(int argc, const char **argv, const char *prefix);
-int cmd_buildid_list(int argc, const char **argv, const char *prefix);
-int cmd_config(int argc, const char **argv, const char *prefix);
-int cmd_c2c(int argc, const char **argv, const char *prefix);
-int cmd_diff(int argc, const char **argv, const char *prefix);
-int cmd_evlist(int argc, const char **argv, const char *prefix);
-int cmd_help(int argc, const char **argv, const char *prefix);
-int cmd_sched(int argc, const char **argv, const char *prefix);
-int cmd_kallsyms(int argc, const char **argv, const char *prefix);
-int cmd_list(int argc, const char **argv, const char *prefix);
-int cmd_record(int argc, const char **argv, const char *prefix);
-int cmd_report(int argc, const char **argv, const char *prefix);
-int cmd_stat(int argc, const char **argv, const char *prefix);
-int cmd_timechart(int argc, const char **argv, const char *prefix);
-int cmd_top(int argc, const char **argv, const char *prefix);
-int cmd_script(int argc, const char **argv, const char *prefix);
-int cmd_version(int argc, const char **argv, const char *prefix);
-int cmd_probe(int argc, const char **argv, const char *prefix);
-int cmd_kmem(int argc, const char **argv, const char *prefix);
-int cmd_lock(int argc, const char **argv, const char *prefix);
-int cmd_kvm(int argc, const char **argv, const char *prefix);
-int cmd_test(int argc, const char **argv, const char *prefix);
-int cmd_trace(int argc, const char **argv, const char *prefix);
-int cmd_inject(int argc, const char **argv, const char *prefix);
-int cmd_mem(int argc, const char **argv, const char *prefix);
-int cmd_data(int argc, const char **argv, const char *prefix);
-int cmd_ftrace(int argc, const char **argv, const char *prefix);
+int cmd_annotate(int argc, const char **argv);
+int cmd_bench(int argc, const char **argv);
+int cmd_buildid_cache(int argc, const char **argv);
+int cmd_buildid_list(int argc, const char **argv);
+int cmd_config(int argc, const char **argv);
+int cmd_c2c(int argc, const char **argv);
+int cmd_diff(int argc, const char **argv);
+int cmd_evlist(int argc, const char **argv);
+int cmd_help(int argc, const char **argv);
+int cmd_sched(int argc, const char **argv);
+int cmd_kallsyms(int argc, const char **argv);
+int cmd_list(int argc, const char **argv);
+int cmd_record(int argc, const char **argv);
+int cmd_report(int argc, const char **argv);
+int cmd_stat(int argc, const char **argv);
+int cmd_timechart(int argc, const char **argv);
+int cmd_top(int argc, const char **argv);
+int cmd_script(int argc, const char **argv);
+int cmd_version(int argc, const char **argv);
+int cmd_probe(int argc, const char **argv);
+int cmd_kmem(int argc, const char **argv);
+int cmd_lock(int argc, const char **argv);
+int cmd_kvm(int argc, const char **argv);
+int cmd_test(int argc, const char **argv);
+int cmd_trace(int argc, const char **argv);
+int cmd_inject(int argc, const char **argv);
+int cmd_mem(int argc, const char **argv);
+int cmd_data(int argc, const char **argv);
+int cmd_ftrace(int argc, const char **argv);
int find_scripts(char **scripts_array, char **scripts_path_array);
#endif
diff --git a/tools/perf/check-headers.sh b/tools/perf/check-headers.sh
index c747bfd..83fe220 100755
--- a/tools/perf/check-headers.sh
+++ b/tools/perf/check-headers.sh
@@ -1,7 +1,9 @@
#!/bin/sh
HEADERS='
+include/uapi/linux/fcntl.h
include/uapi/linux/perf_event.h
+include/uapi/linux/stat.h
include/linux/hash.h
include/uapi/linux/hw_breakpoint.h
arch/x86/include/asm/disabled-features.h
diff --git a/tools/perf/command-list.txt b/tools/perf/command-list.txt
index ac3efd3..2d0caf2 100644
--- a/tools/perf/command-list.txt
+++ b/tools/perf/command-list.txt
@@ -9,6 +9,7 @@
perf-buildid-list mainporcelain common
perf-data mainporcelain common
perf-diff mainporcelain common
+perf-c2c mainporcelain common
perf-config mainporcelain common
perf-evlist mainporcelain common
perf-ftrace mainporcelain common
diff --git a/tools/perf/perf.c b/tools/perf/perf.c
index 6d5479e..4cc6960 100644
--- a/tools/perf/perf.c
+++ b/tools/perf/perf.c
@@ -19,9 +19,15 @@
#include "util/debug.h"
#include <api/fs/fs.h>
#include <api/fs/tracing_path.h>
+#include <errno.h>
#include <pthread.h>
+#include <signal.h>
#include <stdlib.h>
#include <time.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+#include <linux/kernel.h>
const char perf_usage_string[] =
"perf [--version] [--help] [OPTIONS] COMMAND [ARGS]";
@@ -34,7 +40,7 @@
struct cmd_struct {
const char *cmd;
- int (*fn)(int, const char **, const char *);
+ int (*fn)(int, const char **);
int option;
};
@@ -88,7 +94,7 @@
}
/* returns 0 for "no pager", 1 for "use pager", and -1 for "not specified" */
-int check_pager_config(const char *cmd)
+static int check_pager_config(const char *cmd)
{
int err;
struct pager_config c;
@@ -267,71 +273,6 @@
return handled;
}
-static int handle_alias(int *argcp, const char ***argv)
-{
- int envchanged = 0, ret = 0, saved_errno = errno;
- int count, option_count;
- const char **new_argv;
- const char *alias_command;
- char *alias_string;
-
- alias_command = (*argv)[0];
- alias_string = alias_lookup(alias_command);
- if (alias_string) {
- if (alias_string[0] == '!') {
- if (*argcp > 1) {
- struct strbuf buf;
-
- if (strbuf_init(&buf, PATH_MAX) < 0 ||
- strbuf_addstr(&buf, alias_string) < 0 ||
- sq_quote_argv(&buf, (*argv) + 1,
- PATH_MAX) < 0)
- die("Failed to allocate memory.");
- free(alias_string);
- alias_string = buf.buf;
- }
- ret = system(alias_string + 1);
- if (ret >= 0 && WIFEXITED(ret) &&
- WEXITSTATUS(ret) != 127)
- exit(WEXITSTATUS(ret));
- die("Failed to run '%s' when expanding alias '%s'",
- alias_string + 1, alias_command);
- }
- count = split_cmdline(alias_string, &new_argv);
- if (count < 0)
- die("Bad alias.%s string", alias_command);
- option_count = handle_options(&new_argv, &count, &envchanged);
- if (envchanged)
- die("alias '%s' changes environment variables\n"
- "You can use '!perf' in the alias to do this.",
- alias_command);
- memmove(new_argv - option_count, new_argv,
- count * sizeof(char *));
- new_argv -= option_count;
-
- if (count < 1)
- die("empty alias for %s", alias_command);
-
- if (!strcmp(alias_command, new_argv[0]))
- die("recursive alias: %s", alias_command);
-
- new_argv = realloc(new_argv, sizeof(char *) *
- (count + *argcp + 1));
- /* insert after command name */
- memcpy(new_argv + count, *argv + 1, sizeof(char *) * *argcp);
- new_argv[count + *argcp] = NULL;
-
- *argv = new_argv;
- *argcp += count - 1;
-
- ret = 1;
- }
-
- errno = saved_errno;
-
- return ret;
-}
-
#define RUN_SETUP (1<<0)
#define USE_PAGER (1<<1)
@@ -339,13 +280,8 @@
{
int status;
struct stat st;
- const char *prefix;
char sbuf[STRERR_BUFSIZE];
- prefix = NULL;
- if (p->option & RUN_SETUP)
- prefix = NULL; /* setup_perf_directory(); */
-
if (use_browser == -1)
use_browser = check_browser_config(p->cmd);
@@ -356,7 +292,7 @@
commit_pager_choice();
perf_env__set_cmdline(&perf_env, argc, argv);
- status = p->fn(argc, argv, prefix);
+ status = p->fn(argc, argv);
perf_config__exit();
exit_browser(status);
perf_env__exit(&perf_env);
@@ -397,16 +333,6 @@
{
const char *cmd = argv[0];
unsigned int i;
- static const char ext[] = STRIP_EXTENSION;
-
- if (sizeof(ext) > 1) {
- i = strlen(argv[0]) - strlen(ext);
- if (i > 0 && !strcmp(argv[0] + i, ext)) {
- char *argv0 = strdup(argv[0]);
- argv[0] = cmd = argv0;
- argv0[i] = '\0';
- }
- }
/* Turn "perf cmd --help" into "perf help cmd" */
if (argc > 1 && !strcmp(argv[1], "--help")) {
@@ -448,7 +374,8 @@
if (status != -ERR_RUN_COMMAND_EXEC) {
if (IS_RUN_COMMAND_ERR(status)) {
do_die:
- die("unable to run '%s'", argv[0]);
+ pr_err("FATAL: unable to run '%s'", argv[0]);
+ status = -128;
}
exit(-status);
}
@@ -460,25 +387,12 @@
static int run_argv(int *argcp, const char ***argv)
{
- int done_alias = 0;
+ /* See if it's an internal command */
+ handle_internal_command(*argcp, *argv);
- while (1) {
- /* See if it's an internal command */
- handle_internal_command(*argcp, *argv);
-
- /* .. then try the external ones */
- execv_dashed_external(*argv);
-
- /* It could be an alias -- this works around the insanity
- * of overriding "perf log" with "perf show" by having
- * alias.log = show
- */
- if (done_alias || !handle_alias(argcp, argv))
- break;
- done_alias = 1;
- }
-
- return done_alias;
+ /* .. then try the external ones */
+ execv_dashed_external(*argv);
+ return 0;
}
static void pthread__block_sigwinch(void)
@@ -566,7 +480,7 @@
#ifdef HAVE_LIBAUDIT_SUPPORT
setup_path();
argv[0] = "trace";
- return cmd_trace(argc, argv, NULL);
+ return cmd_trace(argc, argv);
#else
fprintf(stderr,
"trace command not available: missing audit-libs devel package at build time.\n");
@@ -611,17 +525,12 @@
while (1) {
static int done_help;
- int was_alias = run_argv(&argc, &argv);
+
+ run_argv(&argc, &argv);
if (errno != ENOENT)
break;
- if (was_alias) {
- fprintf(stderr, "Expansion of alias '%s' failed; "
- "'%s' is not a perf-command\n",
- cmd, argv[0]);
- goto out;
- }
if (!done_help) {
cmd = argv[0] = help_unknown_cmd(cmd);
done_help = 1;
diff --git a/tools/perf/perf.h b/tools/perf/perf.h
index 1c27d94..806c216 100644
--- a/tools/perf/perf.h
+++ b/tools/perf/perf.h
@@ -50,6 +50,7 @@
bool running_time;
bool full_auxtrace;
bool auxtrace_snapshot_mode;
+ bool record_namespaces;
bool record_switch_events;
bool all_kernel;
bool all_user;
diff --git a/tools/perf/pmu-events/arch/x86/broadwell/uncore.json b/tools/perf/pmu-events/arch/x86/broadwell/uncore.json
new file mode 100644
index 0000000..28e1e15
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/broadwell/uncore.json
@@ -0,0 +1,278 @@
+[
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x41",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.MISS_XCORE",
+ "BriefDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which misses in some processor core.",
+ "PublicDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which misses in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x81",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.MISS_EVICTION",
+ "BriefDescription": "A cross-core snoop resulted from L3 Eviction which misses in some processor core.",
+ "PublicDescription": "A cross-core snoop resulted from L3 Eviction which misses in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x44",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HIT_XCORE",
+ "BriefDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which hits a non-modified line in some processor core.",
+ "PublicDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which hits a non-modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x48",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HITM_XCORE",
+ "BriefDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which hits a modified line in some processor core.",
+ "PublicDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which hits a modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x11",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.READ_M",
+ "BriefDescription": "L3 Lookup read request that access cache and found line in M-state",
+ "PublicDescription": "L3 Lookup read request that access cache and found line in M-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x21",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.WRITE_M",
+ "BriefDescription": "L3 Lookup write request that access cache and found line in M-state",
+ "PublicDescription": "L3 Lookup write request that access cache and found line in M-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x81",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_M",
+ "BriefDescription": "L3 Lookup any request that access cache and found line in M-state",
+ "PublicDescription": "L3 Lookup any request that access cache and found line in M-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x18",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.READ_I",
+ "BriefDescription": "L3 Lookup read request that access cache and found line in I-state",
+ "PublicDescription": "L3 Lookup read request that access cache and found line in I-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x88",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_I",
+ "BriefDescription": "L3 Lookup any request that access cache and found line in I-state",
+ "PublicDescription": "L3 Lookup any request that access cache and found line in I-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x1f",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.READ_MESI",
+ "BriefDescription": "L3 Lookup read request that access cache and found line in any MESI-state",
+ "PublicDescription": "L3 Lookup read request that access cache and found line in any MESI-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x2f",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.WRITE_MESI",
+ "BriefDescription": "L3 Lookup write request that access cache and found line in MESI-state",
+ "PublicDescription": "L3 Lookup write request that access cache and found line in MESI-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x8f",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_MESI",
+ "BriefDescription": "L3 Lookup any request that access cache and found line in MESI-state",
+ "PublicDescription": "L3 Lookup any request that access cache and found line in MESI-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x86",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_ES",
+ "BriefDescription": "L3 Lookup any request that access cache and found line in E or S-state",
+ "PublicDescription": "L3 Lookup any request that access cache and found line in E or S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x16",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.READ_ES",
+ "BriefDescription": "L3 Lookup read request that access cache and found line in E or S-state",
+ "PublicDescription": "L3 Lookup read request that access cache and found line in E or S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x26",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.WRITE_ES",
+ "BriefDescription": "L3 Lookup write request that access cache and found line in E or S-state",
+ "PublicDescription": "L3 Lookup write request that access cache and found line in E or S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x80",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.ALL",
+ "BriefDescription": "Each cycle count number of all Core outgoing valid entries. Such entry is defined as valid from it's allocation till first of IDI0 or DRS0 messages is sent out. Accounts for Coherent and non-coherent traffic.",
+ "PublicDescription": "Each cycle count number of all Core outgoing valid entries. Such entry is defined as valid from it's allocation till first of IDI0 or DRS0 messages is sent out. Accounts for Coherent and non-coherent traffic.",
+ "Counter": "0,",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x80",
+ "UMask": "0x02",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.DRD_DIRECT",
+ "BriefDescription": "Each cycle count number of 'valid' coherent Data Read entries that are in DirectData mode. Such entry is defined as valid when it is allocated till data sent to Core (first chunk, IDI0). Applicable for IA Cores' requests in normal case.",
+ "PublicDescription": "Each cycle count number of 'valid' coherent Data Read entries that are in DirectData mode. Such entry is defined as valid when it is allocated till data sent to Core (first chunk, IDI0). Applicable for IA Cores' requests in normal case.",
+ "Counter": "0,",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x81",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_REQUESTS.ALL",
+ "BriefDescription": "Total number of Core outgoing entries allocated. Accounts for Coherent and non-coherent traffic.",
+ "PublicDescription": "Total number of Core outgoing entries allocated. Accounts for Coherent and non-coherent traffic.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x81",
+ "UMask": "0x02",
+ "EventName": "UNC_ARB_TRK_REQUESTS.DRD_DIRECT",
+ "BriefDescription": "Number of Core coherent Data Read entries allocated in DirectData mode",
+ "PublicDescription": "Number of Core coherent Data Read entries allocated in DirectData mode.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x81",
+ "UMask": "0x20",
+ "EventName": "UNC_ARB_TRK_REQUESTS.WRITES",
+ "BriefDescription": "Number of Writes allocated - any write transactions: full/partials writes and evictions.",
+ "PublicDescription": "Number of Writes allocated - any write transactions: full/partials writes and evictions.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x84",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_COH_TRK_REQUESTS.ALL",
+ "BriefDescription": "Number of entries allocated. Account for Any type: e.g. Snoop, Core aperture, etc.",
+ "PublicDescription": "Number of entries allocated. Account for Any type: e.g. Snoop, Core aperture, etc.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x80",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.CYCLES_WITH_ANY_REQUEST",
+ "BriefDescription": "Cycles with at least one request outstanding is waiting for data return from memory controller. Account for coherent and non-coherent requests initiated by IA Cores, Processor Graphics Unit, or LLC.;",
+ "PublicDescription": "Cycles with at least one request outstanding is waiting for data return from memory controller. Account for coherent and non-coherent requests initiated by IA Cores, Processor Graphics Unit, or LLC.",
+ "Counter": "0,",
+ "CounterMask": "1",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "NCU",
+ "EventCode": "0x0",
+ "UMask": "0x01",
+ "EventName": "UNC_CLOCK.SOCKET",
+ "BriefDescription": "This 48-bit fixed counter counts the UCLK cycles",
+ "PublicDescription": "This 48-bit fixed counter counts the UCLK cycles.",
+ "Counter": "FIXED",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ }
+]
\ No newline at end of file
diff --git a/tools/perf/pmu-events/arch/x86/broadwellde/uncore-cache.json b/tools/perf/pmu-events/arch/x86/broadwellde/uncore-cache.json
index 076459c..58ed6d3 100644
--- a/tools/perf/pmu-events/arch/x86/broadwellde/uncore-cache.json
+++ b/tools/perf/pmu-events/arch/x86/broadwellde/uncore-cache.json
@@ -1,13 +1,13 @@
[
{
- "BriefDescription": "Uncore cache clock ticks. Derived from unc_c_clockticks",
+ "BriefDescription": "Uncore cache clock ticks",
"Counter": "0,1,2,3",
"EventName": "UNC_C_CLOCKTICKS",
"PerPkg": "1",
"Unit": "CBO"
},
{
- "BriefDescription": "All LLC Misses (code+ data rd + data wr - including demand and prefetch). Derived from unc_c_llc_lookup.any",
+ "BriefDescription": "All LLC Misses (code+ data rd + data wr - including demand and prefetch)",
"Counter": "0,1,2,3",
"EventCode": "0x34",
"EventName": "UNC_C_LLC_LOOKUP.ANY",
@@ -18,7 +18,7 @@
"Unit": "CBO"
},
{
- "BriefDescription": "M line evictions from LLC (writebacks to memory). Derived from unc_c_llc_victims.m_state",
+ "BriefDescription": "M line evictions from LLC (writebacks to memory)",
"Counter": "0,1,2,3",
"EventCode": "0x37",
"EventName": "UNC_C_LLC_VICTIMS.M_STATE",
@@ -212,7 +212,7 @@
"Unit": "CBO"
},
{
- "BriefDescription": "read requests to home agent. Derived from unc_h_requests.reads",
+ "BriefDescription": "read requests to home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.READS",
@@ -221,7 +221,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "read requests to local home agent. Derived from unc_h_requests.reads_local",
+ "BriefDescription": "read requests to local home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.READS_LOCAL",
@@ -230,7 +230,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "read requests to remote home agent. Derived from unc_h_requests.reads_remote",
+ "BriefDescription": "read requests to remote home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.READS_REMOTE",
@@ -239,7 +239,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "write requests to home agent. Derived from unc_h_requests.writes",
+ "BriefDescription": "write requests to home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.WRITES",
@@ -248,7 +248,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "write requests to local home agent. Derived from unc_h_requests.writes_local",
+ "BriefDescription": "write requests to local home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.WRITES_LOCAL",
@@ -257,7 +257,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "write requests to remote home agent. Derived from unc_h_requests.writes_remote",
+ "BriefDescription": "write requests to remote home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.WRITES_REMOTE",
@@ -266,7 +266,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "Conflict requests (requests for same address from multiple agents simultaneously). Derived from unc_h_snoop_resp.rspcnflct",
+ "BriefDescription": "Conflict requests (requests for same address from multiple agents simultaneously)",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPCNFLCT",
@@ -275,7 +275,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "M line forwarded from remote cache along with writeback to memory. Derived from unc_h_snoop_resp.rsp_fwd_wb",
+ "BriefDescription": "M line forwarded from remote cache along with writeback to memory",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSP_FWD_WB",
@@ -285,7 +285,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "M line forwarded from remote cache with no writeback to memory. Derived from unc_h_snoop_resp.rspifwd",
+ "BriefDescription": "M line forwarded from remote cache with no writeback to memory",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPIFWD",
@@ -295,7 +295,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "Shared line response from remote cache. Derived from unc_h_snoop_resp.rsps",
+ "BriefDescription": "Shared line response from remote cache",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPS",
@@ -305,7 +305,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "Shared line forwarded from remote cache. Derived from unc_h_snoop_resp.rspsfwd",
+ "BriefDescription": "Shared line forwarded from remote cache",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPSFWD",
diff --git a/tools/perf/pmu-events/arch/x86/broadwellde/uncore-memory.json b/tools/perf/pmu-events/arch/x86/broadwellde/uncore-memory.json
index d17dc23..f4b0745 100644
--- a/tools/perf/pmu-events/arch/x86/broadwellde/uncore-memory.json
+++ b/tools/perf/pmu-events/arch/x86/broadwellde/uncore-memory.json
@@ -3,7 +3,7 @@
"BriefDescription": "read requests to memory controller. Derived from unc_m_cas_count.rd",
"Counter": "0,1,2,3",
"EventCode": "0x4",
- "EventName": "UNC_M_CAS_COUNT.RD",
+ "EventName": "LLC_MISSES.MEM_READ",
"PerPkg": "1",
"ScaleUnit": "64Bytes",
"UMask": "0x3",
@@ -13,48 +13,51 @@
"BriefDescription": "write requests to memory controller. Derived from unc_m_cas_count.wr",
"Counter": "0,1,2,3",
"EventCode": "0x4",
- "EventName": "UNC_M_CAS_COUNT.WR",
+ "EventName": "LLC_MISSES.MEM_WRITE",
"PerPkg": "1",
"ScaleUnit": "64Bytes",
"UMask": "0xC",
"Unit": "iMC"
},
{
- "BriefDescription": "Memory controller clock ticks. Derived from unc_m_clockticks",
+ "BriefDescription": "Memory controller clock ticks",
"Counter": "0,1,2,3",
- "EventName": "UNC_M_CLOCKTICKS",
+ "EventName": "UNC_M_DCLOCKTICKS",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles where DRAM ranks are in power down (CKE) mode. Derived from unc_m_power_channel_ppd",
+ "BriefDescription": "Cycles where DRAM ranks are in power down (CKE) mode",
"Counter": "0,1,2,3",
"EventCode": "0x85",
"EventName": "UNC_M_POWER_CHANNEL_PPD",
- "MetricExpr": "(UNC_M_POWER_CHANNEL_PPD / UNC_M_CLOCKTICKS) * 100.",
+ "MetricExpr": "(UNC_M_POWER_CHANNEL_PPD / UNC_M_DCLOCKTICKS) * 100.",
+ "MetricName": "power_channel_ppd %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles all ranks are in critical thermal throttle. Derived from unc_m_power_critical_throttle_cycles",
+ "BriefDescription": "Cycles all ranks are in critical thermal throttle",
"Counter": "0,1,2,3",
"EventCode": "0x86",
"EventName": "UNC_M_POWER_CRITICAL_THROTTLE_CYCLES",
- "MetricExpr": "(UNC_M_POWER_CRITICAL_THROTTLE_CYCLES / UNC_M_CLOCKTICKS) * 100.",
+ "MetricExpr": "(UNC_M_POWER_CRITICAL_THROTTLE_CYCLES / UNC_M_DCLOCKTICKS) * 100.",
+ "MetricName": "power_critical_throttle_cycles %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles Memory is in self refresh power mode. Derived from unc_m_power_self_refresh",
+ "BriefDescription": "Cycles Memory is in self refresh power mode",
"Counter": "0,1,2,3",
"EventCode": "0x43",
"EventName": "UNC_M_POWER_SELF_REFRESH",
- "MetricExpr": "(UNC_M_POWER_SELF_REFRESH / UNC_M_CLOCKTICKS) * 100.",
+ "MetricExpr": "(UNC_M_POWER_SELF_REFRESH / UNC_M_DCLOCKTICKS) * 100.",
+ "MetricName": "power_self_refresh %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Pre-charges due to page misses. Derived from unc_m_pre_count.page_miss",
+ "BriefDescription": "Pre-charges due to page misses",
"Counter": "0,1,2,3",
"EventCode": "0x2",
"EventName": "UNC_M_PRE_COUNT.PAGE_MISS",
@@ -63,7 +66,7 @@
"Unit": "iMC"
},
{
- "BriefDescription": "Pre-charge for reads. Derived from unc_m_pre_count.rd",
+ "BriefDescription": "Pre-charge for reads",
"Counter": "0,1,2,3",
"EventCode": "0x2",
"EventName": "UNC_M_PRE_COUNT.RD",
@@ -72,7 +75,7 @@
"Unit": "iMC"
},
{
- "BriefDescription": "Pre-charge for writes. Derived from unc_m_pre_count.wr",
+ "BriefDescription": "Pre-charge for writes",
"Counter": "0,1,2,3",
"EventCode": "0x2",
"EventName": "UNC_M_PRE_COUNT.WR",
diff --git a/tools/perf/pmu-events/arch/x86/broadwellde/uncore-power.json b/tools/perf/pmu-events/arch/x86/broadwellde/uncore-power.json
index b44d430..dd1b956 100644
--- a/tools/perf/pmu-events/arch/x86/broadwellde/uncore-power.json
+++ b/tools/perf/pmu-events/arch/x86/broadwellde/uncore-power.json
@@ -1,83 +1,91 @@
[
{
- "BriefDescription": "PCU clock ticks. Use to get percentages of PCU cycles events. Derived from unc_p_clockticks",
+ "BriefDescription": "PCU clock ticks. Use to get percentages of PCU cycles events",
"Counter": "0,1,2,3",
"EventName": "UNC_P_CLOCKTICKS",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "C0 and C1. Derived from unc_p_power_state_occupancy.cores_c0",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C0. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C0",
"Filter": "occ_sel=1",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C0 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c0 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "C3. Derived from unc_p_power_state_occupancy.cores_c3",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C3. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C3",
"Filter": "occ_sel=2",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C3 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c3 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "C6 and C7. Derived from unc_p_power_state_occupancy.cores_c6",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C6. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events ",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C6",
"Filter": "occ_sel=3",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C6 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c6 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "External Prochot. Derived from unc_p_prochot_external_cycles",
+ "BriefDescription": "Counts the number of cycles that we are in external PROCHOT mode. This mode is triggered when a sensor off the die determines that something off-die (like DRAM) is too hot and must throttle to avoid damaging the chip",
"Counter": "0,1,2,3",
"EventCode": "0xA",
"EventName": "UNC_P_PROCHOT_EXTERNAL_CYCLES",
"MetricExpr": "(UNC_P_PROCHOT_EXTERNAL_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "prochot_external_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Thermal Strongest Upper Limit Cycles. Derived from unc_p_freq_max_limit_thermal_cycles",
+ "BriefDescription": "Counts the number of cycles when temperature is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x4",
"EventName": "UNC_P_FREQ_MAX_LIMIT_THERMAL_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_LIMIT_THERMAL_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_limit_thermal_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "OS Strongest Upper Limit Cycles. Derived from unc_p_freq_max_os_cycles",
+ "BriefDescription": "Counts the number of cycles when the OS is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x6",
"EventName": "UNC_P_FREQ_MAX_OS_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_OS_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_os_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Power Strongest Upper Limit Cycles. Derived from unc_p_freq_max_power_cycles",
+ "BriefDescription": "Counts the number of cycles when power is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x5",
"EventName": "UNC_P_FREQ_MAX_POWER_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_POWER_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_power_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Cycles spent changing Frequency. Derived from unc_p_freq_trans_cycles",
+ "BriefDescription": "Counts the number of cycles when current is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x74",
"EventName": "UNC_P_FREQ_TRANS_CYCLES",
"MetricExpr": "(UNC_P_FREQ_TRANS_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_trans_cycles %",
"PerPkg": "1",
"Unit": "PCU"
}
diff --git a/tools/perf/pmu-events/arch/x86/broadwellx/uncore-cache.json b/tools/perf/pmu-events/arch/x86/broadwellx/uncore-cache.json
index 076459c..58ed6d3 100644
--- a/tools/perf/pmu-events/arch/x86/broadwellx/uncore-cache.json
+++ b/tools/perf/pmu-events/arch/x86/broadwellx/uncore-cache.json
@@ -1,13 +1,13 @@
[
{
- "BriefDescription": "Uncore cache clock ticks. Derived from unc_c_clockticks",
+ "BriefDescription": "Uncore cache clock ticks",
"Counter": "0,1,2,3",
"EventName": "UNC_C_CLOCKTICKS",
"PerPkg": "1",
"Unit": "CBO"
},
{
- "BriefDescription": "All LLC Misses (code+ data rd + data wr - including demand and prefetch). Derived from unc_c_llc_lookup.any",
+ "BriefDescription": "All LLC Misses (code+ data rd + data wr - including demand and prefetch)",
"Counter": "0,1,2,3",
"EventCode": "0x34",
"EventName": "UNC_C_LLC_LOOKUP.ANY",
@@ -18,7 +18,7 @@
"Unit": "CBO"
},
{
- "BriefDescription": "M line evictions from LLC (writebacks to memory). Derived from unc_c_llc_victims.m_state",
+ "BriefDescription": "M line evictions from LLC (writebacks to memory)",
"Counter": "0,1,2,3",
"EventCode": "0x37",
"EventName": "UNC_C_LLC_VICTIMS.M_STATE",
@@ -212,7 +212,7 @@
"Unit": "CBO"
},
{
- "BriefDescription": "read requests to home agent. Derived from unc_h_requests.reads",
+ "BriefDescription": "read requests to home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.READS",
@@ -221,7 +221,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "read requests to local home agent. Derived from unc_h_requests.reads_local",
+ "BriefDescription": "read requests to local home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.READS_LOCAL",
@@ -230,7 +230,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "read requests to remote home agent. Derived from unc_h_requests.reads_remote",
+ "BriefDescription": "read requests to remote home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.READS_REMOTE",
@@ -239,7 +239,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "write requests to home agent. Derived from unc_h_requests.writes",
+ "BriefDescription": "write requests to home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.WRITES",
@@ -248,7 +248,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "write requests to local home agent. Derived from unc_h_requests.writes_local",
+ "BriefDescription": "write requests to local home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.WRITES_LOCAL",
@@ -257,7 +257,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "write requests to remote home agent. Derived from unc_h_requests.writes_remote",
+ "BriefDescription": "write requests to remote home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.WRITES_REMOTE",
@@ -266,7 +266,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "Conflict requests (requests for same address from multiple agents simultaneously). Derived from unc_h_snoop_resp.rspcnflct",
+ "BriefDescription": "Conflict requests (requests for same address from multiple agents simultaneously)",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPCNFLCT",
@@ -275,7 +275,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "M line forwarded from remote cache along with writeback to memory. Derived from unc_h_snoop_resp.rsp_fwd_wb",
+ "BriefDescription": "M line forwarded from remote cache along with writeback to memory",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSP_FWD_WB",
@@ -285,7 +285,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "M line forwarded from remote cache with no writeback to memory. Derived from unc_h_snoop_resp.rspifwd",
+ "BriefDescription": "M line forwarded from remote cache with no writeback to memory",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPIFWD",
@@ -295,7 +295,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "Shared line response from remote cache. Derived from unc_h_snoop_resp.rsps",
+ "BriefDescription": "Shared line response from remote cache",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPS",
@@ -305,7 +305,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "Shared line forwarded from remote cache. Derived from unc_h_snoop_resp.rspsfwd",
+ "BriefDescription": "Shared line forwarded from remote cache",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPSFWD",
diff --git a/tools/perf/pmu-events/arch/x86/broadwellx/uncore-interconnect.json b/tools/perf/pmu-events/arch/x86/broadwellx/uncore-interconnect.json
index 39387f7..8249613 100644
--- a/tools/perf/pmu-events/arch/x86/broadwellx/uncore-interconnect.json
+++ b/tools/perf/pmu-events/arch/x86/broadwellx/uncore-interconnect.json
@@ -1,6 +1,6 @@
[
{
- "BriefDescription": "QPI clock ticks. Derived from unc_q_clockticks",
+ "BriefDescription": "QPI clock ticks",
"Counter": "0,1,2,3",
"EventCode": "0x14",
"EventName": "UNC_Q_CLOCKTICKS",
@@ -10,7 +10,7 @@
{
"BriefDescription": "Number of data flits transmitted . Derived from unc_q_txl_flits_g0.data",
"Counter": "0,1,2,3",
- "EventName": "UNC_Q_TxL_FLITS_G0.DATA",
+ "EventName": "QPI_DATA_BANDWIDTH_TX",
"PerPkg": "1",
"ScaleUnit": "8Bytes",
"UMask": "0x2",
@@ -19,7 +19,7 @@
{
"BriefDescription": "Number of non data (control) flits transmitted . Derived from unc_q_txl_flits_g0.non_data",
"Counter": "0,1,2,3",
- "EventName": "UNC_Q_TxL_FLITS_G0.NON_DATA",
+ "EventName": "QPI_CTL_BANDWIDTH_TX",
"PerPkg": "1",
"ScaleUnit": "8Bytes",
"UMask": "0x4",
diff --git a/tools/perf/pmu-events/arch/x86/broadwellx/uncore-memory.json b/tools/perf/pmu-events/arch/x86/broadwellx/uncore-memory.json
index d17dc23..66eed39 100644
--- a/tools/perf/pmu-events/arch/x86/broadwellx/uncore-memory.json
+++ b/tools/perf/pmu-events/arch/x86/broadwellx/uncore-memory.json
@@ -3,7 +3,7 @@
"BriefDescription": "read requests to memory controller. Derived from unc_m_cas_count.rd",
"Counter": "0,1,2,3",
"EventCode": "0x4",
- "EventName": "UNC_M_CAS_COUNT.RD",
+ "EventName": "LLC_MISSES.MEM_READ",
"PerPkg": "1",
"ScaleUnit": "64Bytes",
"UMask": "0x3",
@@ -13,48 +13,51 @@
"BriefDescription": "write requests to memory controller. Derived from unc_m_cas_count.wr",
"Counter": "0,1,2,3",
"EventCode": "0x4",
- "EventName": "UNC_M_CAS_COUNT.WR",
+ "EventName": "LLC_MISSES.MEM_WRITE",
"PerPkg": "1",
"ScaleUnit": "64Bytes",
"UMask": "0xC",
"Unit": "iMC"
},
{
- "BriefDescription": "Memory controller clock ticks. Derived from unc_m_clockticks",
+ "BriefDescription": "Memory controller clock ticks",
"Counter": "0,1,2,3",
"EventName": "UNC_M_CLOCKTICKS",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles where DRAM ranks are in power down (CKE) mode. Derived from unc_m_power_channel_ppd",
+ "BriefDescription": "Cycles where DRAM ranks are in power down (CKE) mode",
"Counter": "0,1,2,3",
"EventCode": "0x85",
"EventName": "UNC_M_POWER_CHANNEL_PPD",
"MetricExpr": "(UNC_M_POWER_CHANNEL_PPD / UNC_M_CLOCKTICKS) * 100.",
+ "MetricName": "power_channel_ppd %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles all ranks are in critical thermal throttle. Derived from unc_m_power_critical_throttle_cycles",
+ "BriefDescription": "Cycles all ranks are in critical thermal throttle",
"Counter": "0,1,2,3",
"EventCode": "0x86",
"EventName": "UNC_M_POWER_CRITICAL_THROTTLE_CYCLES",
"MetricExpr": "(UNC_M_POWER_CRITICAL_THROTTLE_CYCLES / UNC_M_CLOCKTICKS) * 100.",
+ "MetricName": "power_critical_throttle_cycles %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles Memory is in self refresh power mode. Derived from unc_m_power_self_refresh",
+ "BriefDescription": "Cycles Memory is in self refresh power mode",
"Counter": "0,1,2,3",
"EventCode": "0x43",
"EventName": "UNC_M_POWER_SELF_REFRESH",
"MetricExpr": "(UNC_M_POWER_SELF_REFRESH / UNC_M_CLOCKTICKS) * 100.",
+ "MetricName": "power_self_refresh %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Pre-charges due to page misses. Derived from unc_m_pre_count.page_miss",
+ "BriefDescription": "Pre-charges due to page misses",
"Counter": "0,1,2,3",
"EventCode": "0x2",
"EventName": "UNC_M_PRE_COUNT.PAGE_MISS",
@@ -63,7 +66,7 @@
"Unit": "iMC"
},
{
- "BriefDescription": "Pre-charge for reads. Derived from unc_m_pre_count.rd",
+ "BriefDescription": "Pre-charge for reads",
"Counter": "0,1,2,3",
"EventCode": "0x2",
"EventName": "UNC_M_PRE_COUNT.RD",
@@ -72,7 +75,7 @@
"Unit": "iMC"
},
{
- "BriefDescription": "Pre-charge for writes. Derived from unc_m_pre_count.wr",
+ "BriefDescription": "Pre-charge for writes",
"Counter": "0,1,2,3",
"EventCode": "0x2",
"EventName": "UNC_M_PRE_COUNT.WR",
diff --git a/tools/perf/pmu-events/arch/x86/broadwellx/uncore-power.json b/tools/perf/pmu-events/arch/x86/broadwellx/uncore-power.json
index b44d430..dd1b956 100644
--- a/tools/perf/pmu-events/arch/x86/broadwellx/uncore-power.json
+++ b/tools/perf/pmu-events/arch/x86/broadwellx/uncore-power.json
@@ -1,83 +1,91 @@
[
{
- "BriefDescription": "PCU clock ticks. Use to get percentages of PCU cycles events. Derived from unc_p_clockticks",
+ "BriefDescription": "PCU clock ticks. Use to get percentages of PCU cycles events",
"Counter": "0,1,2,3",
"EventName": "UNC_P_CLOCKTICKS",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "C0 and C1. Derived from unc_p_power_state_occupancy.cores_c0",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C0. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C0",
"Filter": "occ_sel=1",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C0 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c0 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "C3. Derived from unc_p_power_state_occupancy.cores_c3",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C3. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C3",
"Filter": "occ_sel=2",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C3 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c3 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "C6 and C7. Derived from unc_p_power_state_occupancy.cores_c6",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C6. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events ",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C6",
"Filter": "occ_sel=3",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C6 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c6 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "External Prochot. Derived from unc_p_prochot_external_cycles",
+ "BriefDescription": "Counts the number of cycles that we are in external PROCHOT mode. This mode is triggered when a sensor off the die determines that something off-die (like DRAM) is too hot and must throttle to avoid damaging the chip",
"Counter": "0,1,2,3",
"EventCode": "0xA",
"EventName": "UNC_P_PROCHOT_EXTERNAL_CYCLES",
"MetricExpr": "(UNC_P_PROCHOT_EXTERNAL_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "prochot_external_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Thermal Strongest Upper Limit Cycles. Derived from unc_p_freq_max_limit_thermal_cycles",
+ "BriefDescription": "Counts the number of cycles when temperature is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x4",
"EventName": "UNC_P_FREQ_MAX_LIMIT_THERMAL_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_LIMIT_THERMAL_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_limit_thermal_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "OS Strongest Upper Limit Cycles. Derived from unc_p_freq_max_os_cycles",
+ "BriefDescription": "Counts the number of cycles when the OS is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x6",
"EventName": "UNC_P_FREQ_MAX_OS_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_OS_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_os_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Power Strongest Upper Limit Cycles. Derived from unc_p_freq_max_power_cycles",
+ "BriefDescription": "Counts the number of cycles when power is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x5",
"EventName": "UNC_P_FREQ_MAX_POWER_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_POWER_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_power_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Cycles spent changing Frequency. Derived from unc_p_freq_trans_cycles",
+ "BriefDescription": "Counts the number of cycles when current is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x74",
"EventName": "UNC_P_FREQ_TRANS_CYCLES",
"MetricExpr": "(UNC_P_FREQ_TRANS_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_trans_cycles %",
"PerPkg": "1",
"Unit": "PCU"
}
diff --git a/tools/perf/pmu-events/arch/x86/haswell/uncore.json b/tools/perf/pmu-events/arch/x86/haswell/uncore.json
new file mode 100644
index 0000000..3ef5c21
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/haswell/uncore.json
@@ -0,0 +1,374 @@
+[
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x21",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.MISS_EXTERNAL",
+ "BriefDescription": "An external snoop misses in some processor core.",
+ "PublicDescription": "An external snoop misses in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x41",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.MISS_XCORE",
+ "BriefDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which misses in some processor core.",
+ "PublicDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which misses in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x81",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.MISS_EVICTION",
+ "BriefDescription": "A cross-core snoop resulted from L3 Eviction which misses in some processor core.",
+ "PublicDescription": "A cross-core snoop resulted from L3 Eviction which misses in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x24",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HIT_EXTERNAL",
+ "BriefDescription": "An external snoop hits a non-modified line in some processor core.",
+ "PublicDescription": "An external snoop hits a non-modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x44",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HIT_XCORE",
+ "BriefDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which hits a non-modified line in some processor core.",
+ "PublicDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which hits a non-modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x84",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HIT_EVICTION",
+ "BriefDescription": "A cross-core snoop resulted from L3 Eviction which hits a non-modified line in some processor core.",
+ "PublicDescription": "A cross-core snoop resulted from L3 Eviction which hits a non-modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x28",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HITM_EXTERNAL",
+ "BriefDescription": "An external snoop hits a modified line in some processor core.",
+ "PublicDescription": "An external snoop hits a modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x48",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HITM_XCORE",
+ "BriefDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which hits a modified line in some processor core.",
+ "PublicDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which hits a modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x88",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HITM_EVICTION",
+ "BriefDescription": "A cross-core snoop resulted from L3 Eviction which hits a modified line in some processor core.",
+ "PublicDescription": "A cross-core snoop resulted from L3 Eviction which hits a modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x11",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.READ_M",
+ "BriefDescription": "L3 Lookup read request that access cache and found line in M-state.",
+ "PublicDescription": "L3 Lookup read request that access cache and found line in M-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x21",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.WRITE_M",
+ "BriefDescription": "L3 Lookup write request that access cache and found line in M-state.",
+ "PublicDescription": "L3 Lookup write request that access cache and found line in M-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x41",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.EXTSNP_M",
+ "BriefDescription": "L3 Lookup external snoop request that access cache and found line in M-state.",
+ "PublicDescription": "L3 Lookup external snoop request that access cache and found line in M-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x81",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_M",
+ "BriefDescription": "L3 Lookup any request that access cache and found line in M-state.",
+ "PublicDescription": "L3 Lookup any request that access cache and found line in M-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x18",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.READ_I",
+ "BriefDescription": "L3 Lookup read request that access cache and found line in I-state.",
+ "PublicDescription": "L3 Lookup read request that access cache and found line in I-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x28",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.WRITE_I",
+ "BriefDescription": "L3 Lookup write request that access cache and found line in I-state.",
+ "PublicDescription": "L3 Lookup write request that access cache and found line in I-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x48",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.EXTSNP_I",
+ "BriefDescription": "L3 Lookup external snoop request that access cache and found line in I-state.",
+ "PublicDescription": "L3 Lookup external snoop request that access cache and found line in I-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x88",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_I",
+ "BriefDescription": "L3 Lookup any request that access cache and found line in I-state.",
+ "PublicDescription": "L3 Lookup any request that access cache and found line in I-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x1f",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.READ_MESI",
+ "BriefDescription": "L3 Lookup read request that access cache and found line in any MESI-state.",
+ "PublicDescription": "L3 Lookup read request that access cache and found line in any MESI-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x2f",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.WRITE_MESI",
+ "BriefDescription": "L3 Lookup write request that access cache and found line in MESI-state.",
+ "PublicDescription": "L3 Lookup write request that access cache and found line in MESI-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x4f",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.EXTSNP_MESI",
+ "BriefDescription": "L3 Lookup external snoop request that access cache and found line in MESI-state.",
+ "PublicDescription": "L3 Lookup external snoop request that access cache and found line in MESI-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x8f",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_MESI",
+ "BriefDescription": "L3 Lookup any request that access cache and found line in MESI-state.",
+ "PublicDescription": "L3 Lookup any request that access cache and found line in MESI-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x86",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_ES",
+ "BriefDescription": "L3 Lookup any request that access cache and found line in E or S-state.",
+ "PublicDescription": "L3 Lookup any request that access cache and found line in E or S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x46",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.EXTSNP_ES",
+ "BriefDescription": "L3 Lookup external snoop request that access cache and found line in E or S-state.",
+ "PublicDescription": "L3 Lookup external snoop request that access cache and found line in E or S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x16",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.READ_ES",
+ "BriefDescription": "L3 Lookup read request that access cache and found line in E or S-state.",
+ "PublicDescription": "L3 Lookup read request that access cache and found line in E or S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x26",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.WRITE_ES",
+ "BriefDescription": "L3 Lookup write request that access cache and found line in E or S-state.",
+ "PublicDescription": "L3 Lookup write request that access cache and found line in E or S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x80",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.ALL",
+ "BriefDescription": "Each cycle count number of all Core outgoing valid entries. Such entry is defined as valid from it's allocation till first of IDI0 or DRS0 messages is sent out. Accounts for Coherent and non-coherent traffic.",
+ "PublicDescription": "Each cycle count number of all Core outgoing valid entries. Such entry is defined as valid from it's allocation till first of IDI0 or DRS0 messages is sent out. Accounts for Coherent and non-coherent traffic.",
+ "Counter": "0",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x81",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_REQUESTS.ALL",
+ "BriefDescription": "Total number of Core outgoing entries allocated. Accounts for Coherent and non-coherent traffic.",
+ "PublicDescription": "Total number of Core outgoing entries allocated. Accounts for Coherent and non-coherent traffic.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x81",
+ "UMask": "0x20",
+ "EventName": "UNC_ARB_TRK_REQUESTS.WRITES",
+ "BriefDescription": "Number of Writes allocated - any write transactions: full/partials writes and evictions.",
+ "PublicDescription": "Number of Writes allocated - any write transactions: full/partials writes and evictions.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x83",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_COH_TRK_OCCUPANCY.All",
+ "BriefDescription": "Each cycle count number of valid entries in Coherency Tracker queue from allocation till deallocation. Aperture requests (snoops) appear as NC decoded internally and become coherent (snoop L3, access memory)",
+ "PublicDescription": "Each cycle count number of valid entries in Coherency Tracker queue from allocation till deallocation. Aperture requests (snoops) appear as NC decoded internally and become coherent (snoop L3, access memory).",
+ "Counter": "0",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x84",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_COH_TRK_REQUESTS.ALL",
+ "BriefDescription": "Number of entries allocated. Account for Any type: e.g. Snoop, Core aperture, etc.",
+ "PublicDescription": "Number of entries allocated. Account for Any type: e.g. Snoop, Core aperture, etc.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "NCU",
+ "EventCode": "0x0",
+ "UMask": "0x01",
+ "EventName": "UNC_CLOCK.SOCKET",
+ "BriefDescription": "This 48-bit fixed counter counts the UCLK cycles.",
+ "PublicDescription": "This 48-bit fixed counter counts the UCLK cycles.",
+ "Counter": "FIXED",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ }
+]
\ No newline at end of file
diff --git a/tools/perf/pmu-events/arch/x86/haswellx/uncore-cache.json b/tools/perf/pmu-events/arch/x86/haswellx/uncore-cache.json
index 076459c..58ed6d3 100644
--- a/tools/perf/pmu-events/arch/x86/haswellx/uncore-cache.json
+++ b/tools/perf/pmu-events/arch/x86/haswellx/uncore-cache.json
@@ -1,13 +1,13 @@
[
{
- "BriefDescription": "Uncore cache clock ticks. Derived from unc_c_clockticks",
+ "BriefDescription": "Uncore cache clock ticks",
"Counter": "0,1,2,3",
"EventName": "UNC_C_CLOCKTICKS",
"PerPkg": "1",
"Unit": "CBO"
},
{
- "BriefDescription": "All LLC Misses (code+ data rd + data wr - including demand and prefetch). Derived from unc_c_llc_lookup.any",
+ "BriefDescription": "All LLC Misses (code+ data rd + data wr - including demand and prefetch)",
"Counter": "0,1,2,3",
"EventCode": "0x34",
"EventName": "UNC_C_LLC_LOOKUP.ANY",
@@ -18,7 +18,7 @@
"Unit": "CBO"
},
{
- "BriefDescription": "M line evictions from LLC (writebacks to memory). Derived from unc_c_llc_victims.m_state",
+ "BriefDescription": "M line evictions from LLC (writebacks to memory)",
"Counter": "0,1,2,3",
"EventCode": "0x37",
"EventName": "UNC_C_LLC_VICTIMS.M_STATE",
@@ -212,7 +212,7 @@
"Unit": "CBO"
},
{
- "BriefDescription": "read requests to home agent. Derived from unc_h_requests.reads",
+ "BriefDescription": "read requests to home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.READS",
@@ -221,7 +221,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "read requests to local home agent. Derived from unc_h_requests.reads_local",
+ "BriefDescription": "read requests to local home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.READS_LOCAL",
@@ -230,7 +230,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "read requests to remote home agent. Derived from unc_h_requests.reads_remote",
+ "BriefDescription": "read requests to remote home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.READS_REMOTE",
@@ -239,7 +239,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "write requests to home agent. Derived from unc_h_requests.writes",
+ "BriefDescription": "write requests to home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.WRITES",
@@ -248,7 +248,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "write requests to local home agent. Derived from unc_h_requests.writes_local",
+ "BriefDescription": "write requests to local home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.WRITES_LOCAL",
@@ -257,7 +257,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "write requests to remote home agent. Derived from unc_h_requests.writes_remote",
+ "BriefDescription": "write requests to remote home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.WRITES_REMOTE",
@@ -266,7 +266,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "Conflict requests (requests for same address from multiple agents simultaneously). Derived from unc_h_snoop_resp.rspcnflct",
+ "BriefDescription": "Conflict requests (requests for same address from multiple agents simultaneously)",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPCNFLCT",
@@ -275,7 +275,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "M line forwarded from remote cache along with writeback to memory. Derived from unc_h_snoop_resp.rsp_fwd_wb",
+ "BriefDescription": "M line forwarded from remote cache along with writeback to memory",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSP_FWD_WB",
@@ -285,7 +285,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "M line forwarded from remote cache with no writeback to memory. Derived from unc_h_snoop_resp.rspifwd",
+ "BriefDescription": "M line forwarded from remote cache with no writeback to memory",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPIFWD",
@@ -295,7 +295,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "Shared line response from remote cache. Derived from unc_h_snoop_resp.rsps",
+ "BriefDescription": "Shared line response from remote cache",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPS",
@@ -305,7 +305,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "Shared line forwarded from remote cache. Derived from unc_h_snoop_resp.rspsfwd",
+ "BriefDescription": "Shared line forwarded from remote cache",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPSFWD",
diff --git a/tools/perf/pmu-events/arch/x86/haswellx/uncore-interconnect.json b/tools/perf/pmu-events/arch/x86/haswellx/uncore-interconnect.json
index 39387f7..8249613 100644
--- a/tools/perf/pmu-events/arch/x86/haswellx/uncore-interconnect.json
+++ b/tools/perf/pmu-events/arch/x86/haswellx/uncore-interconnect.json
@@ -1,6 +1,6 @@
[
{
- "BriefDescription": "QPI clock ticks. Derived from unc_q_clockticks",
+ "BriefDescription": "QPI clock ticks",
"Counter": "0,1,2,3",
"EventCode": "0x14",
"EventName": "UNC_Q_CLOCKTICKS",
@@ -10,7 +10,7 @@
{
"BriefDescription": "Number of data flits transmitted . Derived from unc_q_txl_flits_g0.data",
"Counter": "0,1,2,3",
- "EventName": "UNC_Q_TxL_FLITS_G0.DATA",
+ "EventName": "QPI_DATA_BANDWIDTH_TX",
"PerPkg": "1",
"ScaleUnit": "8Bytes",
"UMask": "0x2",
@@ -19,7 +19,7 @@
{
"BriefDescription": "Number of non data (control) flits transmitted . Derived from unc_q_txl_flits_g0.non_data",
"Counter": "0,1,2,3",
- "EventName": "UNC_Q_TxL_FLITS_G0.NON_DATA",
+ "EventName": "QPI_CTL_BANDWIDTH_TX",
"PerPkg": "1",
"ScaleUnit": "8Bytes",
"UMask": "0x4",
diff --git a/tools/perf/pmu-events/arch/x86/haswellx/uncore-memory.json b/tools/perf/pmu-events/arch/x86/haswellx/uncore-memory.json
index d17dc23..66eed39 100644
--- a/tools/perf/pmu-events/arch/x86/haswellx/uncore-memory.json
+++ b/tools/perf/pmu-events/arch/x86/haswellx/uncore-memory.json
@@ -3,7 +3,7 @@
"BriefDescription": "read requests to memory controller. Derived from unc_m_cas_count.rd",
"Counter": "0,1,2,3",
"EventCode": "0x4",
- "EventName": "UNC_M_CAS_COUNT.RD",
+ "EventName": "LLC_MISSES.MEM_READ",
"PerPkg": "1",
"ScaleUnit": "64Bytes",
"UMask": "0x3",
@@ -13,48 +13,51 @@
"BriefDescription": "write requests to memory controller. Derived from unc_m_cas_count.wr",
"Counter": "0,1,2,3",
"EventCode": "0x4",
- "EventName": "UNC_M_CAS_COUNT.WR",
+ "EventName": "LLC_MISSES.MEM_WRITE",
"PerPkg": "1",
"ScaleUnit": "64Bytes",
"UMask": "0xC",
"Unit": "iMC"
},
{
- "BriefDescription": "Memory controller clock ticks. Derived from unc_m_clockticks",
+ "BriefDescription": "Memory controller clock ticks",
"Counter": "0,1,2,3",
"EventName": "UNC_M_CLOCKTICKS",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles where DRAM ranks are in power down (CKE) mode. Derived from unc_m_power_channel_ppd",
+ "BriefDescription": "Cycles where DRAM ranks are in power down (CKE) mode",
"Counter": "0,1,2,3",
"EventCode": "0x85",
"EventName": "UNC_M_POWER_CHANNEL_PPD",
"MetricExpr": "(UNC_M_POWER_CHANNEL_PPD / UNC_M_CLOCKTICKS) * 100.",
+ "MetricName": "power_channel_ppd %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles all ranks are in critical thermal throttle. Derived from unc_m_power_critical_throttle_cycles",
+ "BriefDescription": "Cycles all ranks are in critical thermal throttle",
"Counter": "0,1,2,3",
"EventCode": "0x86",
"EventName": "UNC_M_POWER_CRITICAL_THROTTLE_CYCLES",
"MetricExpr": "(UNC_M_POWER_CRITICAL_THROTTLE_CYCLES / UNC_M_CLOCKTICKS) * 100.",
+ "MetricName": "power_critical_throttle_cycles %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles Memory is in self refresh power mode. Derived from unc_m_power_self_refresh",
+ "BriefDescription": "Cycles Memory is in self refresh power mode",
"Counter": "0,1,2,3",
"EventCode": "0x43",
"EventName": "UNC_M_POWER_SELF_REFRESH",
"MetricExpr": "(UNC_M_POWER_SELF_REFRESH / UNC_M_CLOCKTICKS) * 100.",
+ "MetricName": "power_self_refresh %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Pre-charges due to page misses. Derived from unc_m_pre_count.page_miss",
+ "BriefDescription": "Pre-charges due to page misses",
"Counter": "0,1,2,3",
"EventCode": "0x2",
"EventName": "UNC_M_PRE_COUNT.PAGE_MISS",
@@ -63,7 +66,7 @@
"Unit": "iMC"
},
{
- "BriefDescription": "Pre-charge for reads. Derived from unc_m_pre_count.rd",
+ "BriefDescription": "Pre-charge for reads",
"Counter": "0,1,2,3",
"EventCode": "0x2",
"EventName": "UNC_M_PRE_COUNT.RD",
@@ -72,7 +75,7 @@
"Unit": "iMC"
},
{
- "BriefDescription": "Pre-charge for writes. Derived from unc_m_pre_count.wr",
+ "BriefDescription": "Pre-charge for writes",
"Counter": "0,1,2,3",
"EventCode": "0x2",
"EventName": "UNC_M_PRE_COUNT.WR",
diff --git a/tools/perf/pmu-events/arch/x86/haswellx/uncore-power.json b/tools/perf/pmu-events/arch/x86/haswellx/uncore-power.json
index b44d430..dd1b956 100644
--- a/tools/perf/pmu-events/arch/x86/haswellx/uncore-power.json
+++ b/tools/perf/pmu-events/arch/x86/haswellx/uncore-power.json
@@ -1,83 +1,91 @@
[
{
- "BriefDescription": "PCU clock ticks. Use to get percentages of PCU cycles events. Derived from unc_p_clockticks",
+ "BriefDescription": "PCU clock ticks. Use to get percentages of PCU cycles events",
"Counter": "0,1,2,3",
"EventName": "UNC_P_CLOCKTICKS",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "C0 and C1. Derived from unc_p_power_state_occupancy.cores_c0",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C0. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C0",
"Filter": "occ_sel=1",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C0 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c0 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "C3. Derived from unc_p_power_state_occupancy.cores_c3",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C3. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C3",
"Filter": "occ_sel=2",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C3 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c3 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "C6 and C7. Derived from unc_p_power_state_occupancy.cores_c6",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C6. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events ",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C6",
"Filter": "occ_sel=3",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C6 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c6 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "External Prochot. Derived from unc_p_prochot_external_cycles",
+ "BriefDescription": "Counts the number of cycles that we are in external PROCHOT mode. This mode is triggered when a sensor off the die determines that something off-die (like DRAM) is too hot and must throttle to avoid damaging the chip",
"Counter": "0,1,2,3",
"EventCode": "0xA",
"EventName": "UNC_P_PROCHOT_EXTERNAL_CYCLES",
"MetricExpr": "(UNC_P_PROCHOT_EXTERNAL_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "prochot_external_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Thermal Strongest Upper Limit Cycles. Derived from unc_p_freq_max_limit_thermal_cycles",
+ "BriefDescription": "Counts the number of cycles when temperature is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x4",
"EventName": "UNC_P_FREQ_MAX_LIMIT_THERMAL_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_LIMIT_THERMAL_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_limit_thermal_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "OS Strongest Upper Limit Cycles. Derived from unc_p_freq_max_os_cycles",
+ "BriefDescription": "Counts the number of cycles when the OS is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x6",
"EventName": "UNC_P_FREQ_MAX_OS_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_OS_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_os_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Power Strongest Upper Limit Cycles. Derived from unc_p_freq_max_power_cycles",
+ "BriefDescription": "Counts the number of cycles when power is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x5",
"EventName": "UNC_P_FREQ_MAX_POWER_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_POWER_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_power_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Cycles spent changing Frequency. Derived from unc_p_freq_trans_cycles",
+ "BriefDescription": "Counts the number of cycles when current is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x74",
"EventName": "UNC_P_FREQ_TRANS_CYCLES",
"MetricExpr": "(UNC_P_FREQ_TRANS_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_trans_cycles %",
"PerPkg": "1",
"Unit": "PCU"
}
diff --git a/tools/perf/pmu-events/arch/x86/ivybridge/uncore.json b/tools/perf/pmu-events/arch/x86/ivybridge/uncore.json
new file mode 100644
index 0000000..42c70ee
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/ivybridge/uncore.json
@@ -0,0 +1,314 @@
+[
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x01",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.MISS",
+ "BriefDescription": "A snoop misses in some processor core.",
+ "PublicDescription": "A snoop misses in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x02",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.INVAL",
+ "BriefDescription": "A snoop invalidates a non-modified line in some processor core.",
+ "PublicDescription": "A snoop invalidates a non-modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x04",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HIT",
+ "BriefDescription": "A snoop hits a non-modified line in some processor core.",
+ "PublicDescription": "A snoop hits a non-modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x08",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HITM",
+ "BriefDescription": "A snoop hits a modified line in some processor core.",
+ "PublicDescription": "A snoop hits a modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x10",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.INVAL_M",
+ "BriefDescription": "A snoop invalidates a modified line in some processor core.",
+ "PublicDescription": "A snoop invalidates a modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x20",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.EXTERNAL_FILTER",
+ "BriefDescription": "Filter on cross-core snoops initiated by this Cbox due to external snoop request.",
+ "PublicDescription": "Filter on cross-core snoops initiated by this Cbox due to external snoop request.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x40",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.XCORE_FILTER",
+ "BriefDescription": "Filter on cross-core snoops initiated by this Cbox due to processor core memory request.",
+ "PublicDescription": "Filter on cross-core snoops initiated by this Cbox due to processor core memory request.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x80",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.EVICTION_FILTER",
+ "BriefDescription": "Filter on cross-core snoops initiated by this Cbox due to LLC eviction.",
+ "PublicDescription": "Filter on cross-core snoops initiated by this Cbox due to LLC eviction.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x01",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.M",
+ "BriefDescription": "LLC lookup request that access cache and found line in M-state.",
+ "PublicDescription": "LLC lookup request that access cache and found line in M-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x02",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.E",
+ "BriefDescription": "LLC lookup request that access cache and found line in E-state.",
+ "PublicDescription": "LLC lookup request that access cache and found line in E-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x04",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.S",
+ "BriefDescription": "LLC lookup request that access cache and found line in S-state.",
+ "PublicDescription": "LLC lookup request that access cache and found line in S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x08",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.I",
+ "BriefDescription": "LLC lookup request that access cache and found line in I-state.",
+ "PublicDescription": "LLC lookup request that access cache and found line in I-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x10",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.READ_FILTER",
+ "BriefDescription": "Filter on processor core initiated cacheable read requests.",
+ "PublicDescription": "Filter on processor core initiated cacheable read requests.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x20",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.WRITE_FILTER",
+ "BriefDescription": "Filter on processor core initiated cacheable write requests.",
+ "PublicDescription": "Filter on processor core initiated cacheable write requests.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x40",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.EXTSNP_FILTER",
+ "BriefDescription": "Filter on external snoop requests.",
+ "PublicDescription": "Filter on external snoop requests.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x80",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_REQUEST_FILTER",
+ "BriefDescription": "Filter on any IRQ or IPQ initiated requests including uncacheable, non-coherent requests.",
+ "PublicDescription": "Filter on any IRQ or IPQ initiated requests including uncacheable, non-coherent requests.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x80",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.ALL",
+ "BriefDescription": "Counts cycles weighted by the number of requests waiting for data returning from the memory controller. Accounts for coherent and non-coherent requests initiated by IA cores, processor graphic units, or LLC.",
+ "PublicDescription": "Counts cycles weighted by the number of requests waiting for data returning from the memory controller. Accounts for coherent and non-coherent requests initiated by IA cores, processor graphic units, or LLC.",
+ "Counter": "0",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x81",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_REQUESTS.ALL",
+ "BriefDescription": "Counts the number of coherent and in-coherent requests initiated by IA cores, processor graphic units, or LLC.",
+ "PublicDescription": "Counts the number of coherent and in-coherent requests initiated by IA cores, processor graphic units, or LLC.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x81",
+ "UMask": "0x20",
+ "EventName": "UNC_ARB_TRK_REQUESTS.WRITES",
+ "BriefDescription": "Counts the number of allocated write entries, include full, partial, and LLC evictions.",
+ "PublicDescription": "Counts the number of allocated write entries, include full, partial, and LLC evictions.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x81",
+ "UMask": "0x80",
+ "EventName": "UNC_ARB_TRK_REQUESTS.EVICTIONS",
+ "BriefDescription": "Counts the number of LLC evictions allocated.",
+ "PublicDescription": "Counts the number of LLC evictions allocated.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x83",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_COH_TRK_OCCUPANCY.ALL",
+ "BriefDescription": "Cycles weighted by number of requests pending in Coherency Tracker.",
+ "PublicDescription": "Cycles weighted by number of requests pending in Coherency Tracker.",
+ "Counter": "0",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x84",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_COH_TRK_REQUESTS.ALL",
+ "BriefDescription": "Number of requests allocated in Coherency Tracker.",
+ "PublicDescription": "Number of requests allocated in Coherency Tracker.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x80",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.CYCLES_WITH_ANY_REQUEST",
+ "BriefDescription": "Cycles with at least one request outstanding is waiting for data return from memory controller. Account for coherent and non-coherent requests initiated by IA Cores, Processor Graphics Unit, or LLC.",
+ "PublicDescription": "Cycles with at least one request outstanding is waiting for data return from memory controller. Account for coherent and non-coherent requests initiated by IA Cores, Processor Graphics Unit, or LLC.",
+ "Counter": "0,1",
+ "CounterMask": "1",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x80",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.CYCLES_OVER_HALF_FULL",
+ "BriefDescription": "Cycles with at least half of the requests outstanding are waiting for data return from memory controller. Account for coherent and non-coherent requests initiated by IA Cores, Processor Graphics Unit, or LLC.",
+ "PublicDescription": "Cycles with at least half of the requests outstanding are waiting for data return from memory controller. Account for coherent and non-coherent requests initiated by IA Cores, Processor Graphics Unit, or LLC.",
+ "Counter": "0,1",
+ "CounterMask": "10",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x0",
+ "UMask": "0x01",
+ "EventName": "UNC_CLOCK.SOCKET",
+ "BriefDescription": "This 48-bit fixed counter counts the UCLK cycles.",
+ "PublicDescription": "This 48-bit fixed counter counts the UCLK cycles.",
+ "Counter": "Fixed",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x06",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ES",
+ "BriefDescription": "LLC lookup request that access cache and found line in E-state or S-state.",
+ "PublicDescription": "LLC lookup request that access cache and found line in E-state or S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ }
+]
\ No newline at end of file
diff --git a/tools/perf/pmu-events/arch/x86/ivytown/uncore-cache.json b/tools/perf/pmu-events/arch/x86/ivytown/uncore-cache.json
index 2efdc67..2674105 100644
--- a/tools/perf/pmu-events/arch/x86/ivytown/uncore-cache.json
+++ b/tools/perf/pmu-events/arch/x86/ivytown/uncore-cache.json
@@ -1,13 +1,13 @@
[
{
- "BriefDescription": "Uncore cache clock ticks. Derived from unc_c_clockticks",
+ "BriefDescription": "Uncore cache clock ticks",
"Counter": "0,1,2,3",
"EventName": "UNC_C_CLOCKTICKS",
"PerPkg": "1",
"Unit": "CBO"
},
{
- "BriefDescription": "All LLC Misses (code+ data rd + data wr - including demand and prefetch). Derived from unc_c_llc_lookup.any",
+ "BriefDescription": "All LLC Misses (code+ data rd + data wr - including demand and prefetch)",
"Counter": "0,1",
"EventCode": "0x34",
"EventName": "UNC_C_LLC_LOOKUP.ANY",
@@ -18,7 +18,7 @@
"Unit": "CBO"
},
{
- "BriefDescription": "M line evictions from LLC (writebacks to memory). Derived from unc_c_llc_victims.m_state",
+ "BriefDescription": "M line evictions from LLC (writebacks to memory)",
"Counter": "0,1",
"EventCode": "0x37",
"EventName": "UNC_C_LLC_VICTIMS.M_STATE",
@@ -237,7 +237,7 @@
"Unit": "CBO"
},
{
- "BriefDescription": "Occupancy for all LLC misses that are addressed to local memory. Derived from unc_c_tor_occupancy.miss_local",
+ "BriefDescription": "Occupancy for all LLC misses that are addressed to local memory",
"EventCode": "0x36",
"EventName": "UNC_C_TOR_OCCUPANCY.MISS_LOCAL",
"PerPkg": "1",
@@ -254,7 +254,7 @@
"Unit": "CBO"
},
{
- "BriefDescription": "Occupancy for all LLC misses that are addressed to remote memory. Derived from unc_c_tor_occupancy.miss_remote",
+ "BriefDescription": "Occupancy for all LLC misses that are addressed to remote memory",
"EventCode": "0x36",
"EventName": "UNC_C_TOR_OCCUPANCY.MISS_REMOTE",
"PerPkg": "1",
@@ -262,7 +262,7 @@
"Unit": "CBO"
},
{
- "BriefDescription": "Read requests to home agent. Derived from unc_h_requests.reads",
+ "BriefDescription": "Read requests to home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.READS",
@@ -271,7 +271,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "Write requests to home agent. Derived from unc_h_requests.writes",
+ "BriefDescription": "Write requests to home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.WRITES",
@@ -280,7 +280,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "M line forwarded from remote cache along with writeback to memory. Derived from unc_h_snoop_resp.rsp_fwd_wb",
+ "BriefDescription": "M line forwarded from remote cache along with writeback to memory",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSP_FWD_WB",
@@ -290,7 +290,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "M line forwarded from remote cache with no writeback to memory. Derived from unc_h_snoop_resp.rspifwd",
+ "BriefDescription": "M line forwarded from remote cache with no writeback to memory",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPIFWD",
@@ -300,7 +300,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "Shared line response from remote cache. Derived from unc_h_snoop_resp.rsps",
+ "BriefDescription": "Shared line response from remote cache",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPS",
@@ -310,7 +310,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "Shared line forwarded from remote cache. Derived from unc_h_snoop_resp.rspsfwd",
+ "BriefDescription": "Shared line forwarded from remote cache",
"Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "UNC_H_SNOOP_RESP.RSPSFWD",
diff --git a/tools/perf/pmu-events/arch/x86/ivytown/uncore-interconnect.json b/tools/perf/pmu-events/arch/x86/ivytown/uncore-interconnect.json
index d7e2fda..b798a86 100644
--- a/tools/perf/pmu-events/arch/x86/ivytown/uncore-interconnect.json
+++ b/tools/perf/pmu-events/arch/x86/ivytown/uncore-interconnect.json
@@ -1,6 +1,6 @@
[
{
- "BriefDescription": "QPI clock ticks. Use to get percentages for QPI cycles events. Derived from unc_q_clockticks",
+ "BriefDescription": "QPI clock ticks. Use to get percentages for QPI cycles events",
"Counter": "0,1,2,3",
"EventCode": "0x14",
"EventName": "UNC_Q_CLOCKTICKS",
@@ -8,25 +8,27 @@
"Unit": "QPI LL"
},
{
- "BriefDescription": "Cycles where receiving QPI link is in half-width mode. Derived from unc_q_rxl0p_power_cycles",
+ "BriefDescription": "Cycles where receiving QPI link is in half-width mode",
"Counter": "0,1,2,3",
"EventCode": "0x10",
"EventName": "UNC_Q_RxL0P_POWER_CYCLES",
"MetricExpr": "(UNC_Q_RxL0P_POWER_CYCLES / UNC_Q_CLOCKTICKS) * 100.",
+ "MetricName": "rxl0p_power_cycles %",
"PerPkg": "1",
"Unit": "QPI LL"
},
{
- "BriefDescription": "Cycles where transmitting QPI link is in half-width mode. Derived from unc_q_txl0p_power_cycles",
+ "BriefDescription": "Cycles where transmitting QPI link is in half-width mode",
"Counter": "0,1,2,3",
"EventCode": "0xd",
"EventName": "UNC_Q_TxL0P_POWER_CYCLES",
"MetricExpr": "(UNC_Q_TxL0P_POWER_CYCLES / UNC_Q_CLOCKTICKS) * 100.",
+ "MetricName": "txl0p_power_cycles %",
"PerPkg": "1",
"Unit": "QPI LL"
},
{
- "BriefDescription": "Number of data flits transmitted . Derived from unc_q_txl_flits_g0.data",
+ "BriefDescription": "Number of data flits transmitted ",
"Counter": "0,1,2,3",
"EventName": "UNC_Q_TxL_FLITS_G0.DATA",
"PerPkg": "1",
@@ -35,7 +37,7 @@
"Unit": "QPI LL"
},
{
- "BriefDescription": "Number of non data (control) flits transmitted . Derived from unc_q_txl_flits_g0.non_data",
+ "BriefDescription": "Number of non data (control) flits transmitted ",
"Counter": "0,1,2,3",
"EventName": "UNC_Q_TxL_FLITS_G0.NON_DATA",
"PerPkg": "1",
diff --git a/tools/perf/pmu-events/arch/x86/ivytown/uncore-memory.json b/tools/perf/pmu-events/arch/x86/ivytown/uncore-memory.json
index ac4ad4d..df4b432 100644
--- a/tools/perf/pmu-events/arch/x86/ivytown/uncore-memory.json
+++ b/tools/perf/pmu-events/arch/x86/ivytown/uncore-memory.json
@@ -1,6 +1,6 @@
[
{
- "BriefDescription": "Memory page activates for reads and writes. Derived from unc_m_act_count.rd",
+ "BriefDescription": "Memory page activates for reads and writes",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_M_ACT_COUNT.RD",
@@ -13,7 +13,7 @@
"BriefDescription": "Read requests to memory controller. Derived from unc_m_cas_count.rd",
"Counter": "0,1,2,3",
"EventCode": "0x4",
- "EventName": "UNC_M_CAS_COUNT.RD",
+ "EventName": "LLC_MISSES.MEM_READ",
"PerPkg": "1",
"ScaleUnit": "64Bytes",
"UMask": "0x3",
@@ -23,48 +23,51 @@
"BriefDescription": "Write requests to memory controller. Derived from unc_m_cas_count.wr",
"Counter": "0,1,2,3",
"EventCode": "0x4",
- "EventName": "UNC_M_CAS_COUNT.WR",
+ "EventName": "LLC_MISSES.MEM_WRITE",
"PerPkg": "1",
"ScaleUnit": "64Bytes",
"UMask": "0xC",
"Unit": "iMC"
},
{
- "BriefDescription": "Memory controller clock ticks. Use to generate percentages for memory controller CYCLES events. Derived from unc_m_clockticks",
+ "BriefDescription": "Memory controller clock ticks. Use to generate percentages for memory controller CYCLES events",
"Counter": "0,1,2,3",
"EventName": "UNC_M_CLOCKTICKS",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles where DRAM ranks are in power down (CKE) mode. Derived from unc_m_power_channel_ppd",
+ "BriefDescription": "Cycles where DRAM ranks are in power down (CKE) mode",
"Counter": "0,1,2,3",
"EventCode": "0x85",
"EventName": "UNC_M_POWER_CHANNEL_PPD",
"MetricExpr": "(UNC_M_POWER_CHANNEL_PPD / UNC_M_CLOCKTICKS) * 100.",
+ "MetricName": "power_channel_ppd %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles all ranks are in critical thermal throttle. Derived from unc_m_power_critical_throttle_cycles",
+ "BriefDescription": "Cycles all ranks are in critical thermal throttle",
"Counter": "0,1,2,3",
"EventCode": "0x86",
"EventName": "UNC_M_POWER_CRITICAL_THROTTLE_CYCLES",
"MetricExpr": "(UNC_M_POWER_CRITICAL_THROTTLE_CYCLES / UNC_M_CLOCKTICKS) * 100.",
+ "MetricName": "power_critical_throttle_cycles %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles Memory is in self refresh power mode. Derived from unc_m_power_self_refresh",
+ "BriefDescription": "Cycles Memory is in self refresh power mode",
"Counter": "0,1,2,3",
"EventCode": "0x43",
"EventName": "UNC_M_POWER_SELF_REFRESH",
"MetricExpr": "(UNC_M_POWER_SELF_REFRESH / UNC_M_CLOCKTICKS) * 100.",
+ "MetricName": "power_self_refresh %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Memory page conflicts. Derived from unc_m_pre_count.page_miss",
+ "BriefDescription": "Memory page conflicts",
"Counter": "0,1,2,3",
"EventCode": "0x2",
"EventName": "UNC_M_PRE_COUNT.PAGE_MISS",
diff --git a/tools/perf/pmu-events/arch/x86/ivytown/uncore-power.json b/tools/perf/pmu-events/arch/x86/ivytown/uncore-power.json
index dc2586d..d40498f 100644
--- a/tools/perf/pmu-events/arch/x86/ivytown/uncore-power.json
+++ b/tools/perf/pmu-events/arch/x86/ivytown/uncore-power.json
@@ -1,44 +1,48 @@
[
{
- "BriefDescription": "PCU clock ticks. Use to get percentages of PCU cycles events. Derived from unc_p_clockticks",
+ "BriefDescription": "PCU clock ticks. Use to get percentages of PCU cycles events",
"Counter": "0,1,2,3",
"EventName": "UNC_P_CLOCKTICKS",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band0=XXX, with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency. Derived from unc_p_freq_band0_cycles",
+ "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band0=XXX, with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency",
"Counter": "0,1,2,3",
"EventCode": "0xb",
"EventName": "UNC_P_FREQ_BAND0_CYCLES",
"MetricExpr": "(UNC_P_FREQ_BAND0_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band0_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band1=XXX, with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency. Derived from unc_p_freq_band1_cycles",
+ "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band1=XXX, with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency",
"Counter": "0,1,2,3",
"EventCode": "0xc",
"EventName": "UNC_P_FREQ_BAND1_CYCLES",
"MetricExpr": "(UNC_P_FREQ_BAND1_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band1_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band2=XXX, with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency. Derived from unc_p_freq_band2_cycles",
+ "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band2=XXX, with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency",
"Counter": "0,1,2,3",
"EventCode": "0xd",
"EventName": "UNC_P_FREQ_BAND2_CYCLES",
"MetricExpr": "(UNC_P_FREQ_BAND2_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band2_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band3=XXX, with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency. Derived from unc_p_freq_band3_cycles",
+ "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band3=XXX, with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency",
"Counter": "0,1,2,3",
"EventCode": "0xe",
"EventName": "UNC_P_FREQ_BAND3_CYCLES",
"MetricExpr": "(UNC_P_FREQ_BAND3_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band3_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -49,6 +53,7 @@
"EventName": "UNC_P_FREQ_BAND0_TRANSITIONS",
"Filter": "edge=1",
"MetricExpr": "(UNC_P_FREQ_BAND0_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band0_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -59,6 +64,7 @@
"EventName": "UNC_P_FREQ_BAND1_TRANSITIONS",
"Filter": "edge=1",
"MetricExpr": "(UNC_P_FREQ_BAND1_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band1_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -69,6 +75,7 @@
"EventName": "UNC_P_FREQ_BAND2_TRANSITIONS",
"Filter": "edge=1",
"MetricExpr": "(UNC_P_FREQ_BAND2_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band2_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -79,90 +86,100 @@
"EventName": "UNC_P_FREQ_BAND3_TRANSITIONS",
"Filter": "edge=1",
"MetricExpr": "(UNC_P_FREQ_BAND3_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band3_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "This is an occupancy event that tracks the number of cores that are in the chosen C-State. It can be used by itself to get the average number of cores in that C-state with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details. Derived from unc_p_power_state_occupancy.cores_c0",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C0. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C0",
"Filter": "occ_sel=1",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C0 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c0 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "This is an occupancy event that tracks the number of cores that are in the chosen C-State. It can be used by itself to get the average number of cores in that C-state with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details. Derived from unc_p_power_state_occupancy.cores_c3",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C3. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C3",
"Filter": "occ_sel=2",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C3 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c3 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "This is an occupancy event that tracks the number of cores that are in the chosen C-State. It can be used by itself to get the average number of cores in that C-state with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details. Derived from unc_p_power_state_occupancy.cores_c6",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C6. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events ",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C6",
"Filter": "occ_sel=3",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C6 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c6 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles that we are in external PROCHOT mode. This mode is triggered when a sensor off the die determines that something off-die (like DRAM) is too hot and must throttle to avoid damaging the chip. Derived from unc_p_prochot_external_cycles",
+ "BriefDescription": "Counts the number of cycles that we are in external PROCHOT mode. This mode is triggered when a sensor off the die determines that something off-die (like DRAM) is too hot and must throttle to avoid damaging the chip",
"Counter": "0,1,2,3",
"EventCode": "0xa",
"EventName": "UNC_P_PROCHOT_EXTERNAL_CYCLES",
"MetricExpr": "(UNC_P_PROCHOT_EXTERNAL_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "prochot_external_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles when thermal conditions are the upper limit on frequency. This is related to the THERMAL_THROTTLE CYCLES_ABOVE_TEMP event, which always counts cycles when we are above the thermal temperature. This event (STRONGEST_UPPER_LIMIT) is sampled at the output of the algorithm that determines the actual frequency, while THERMAL_THROTTLE looks at the input. Derived from unc_p_freq_max_limit_thermal_cycles",
+ "BriefDescription": "Counts the number of cycles when temperature is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x4",
"EventName": "UNC_P_FREQ_MAX_LIMIT_THERMAL_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_LIMIT_THERMAL_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_limit_thermal_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles when the OS is the upper limit on frequency. Derived from unc_p_freq_max_os_cycles",
+ "BriefDescription": "Counts the number of cycles when the OS is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x6",
"EventName": "UNC_P_FREQ_MAX_OS_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_OS_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_os_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles when power is the upper limit on frequency. Derived from unc_p_freq_max_power_cycles",
+ "BriefDescription": "Counts the number of cycles when power is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x5",
"EventName": "UNC_P_FREQ_MAX_POWER_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_POWER_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_power_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles when current is the upper limit on frequency. Derived from unc_p_freq_max_current_cycles",
+ "BriefDescription": "Counts the number of cycles when current is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x7",
"EventName": "UNC_P_FREQ_MAX_CURRENT_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_CURRENT_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_current_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles when the system is changing frequency. This can not be filtered by thread ID. One can also use it with the occupancy counter that monitors number of threads in C0 to estimate the performance impact that frequency transitions had on the system. Derived from unc_p_freq_trans_cycles",
+ "BriefDescription": "Cycles spent changing Frequency",
"Counter": "0,1,2,3",
"EventCode": "0x60",
"EventName": "UNC_P_FREQ_TRANS_CYCLES",
"MetricExpr": "(UNC_P_FREQ_TRANS_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_trans_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -173,6 +190,7 @@
"EventName": "UNC_P_FREQ_GE_1200MHZ_CYCLES",
"Filter": "filter_band0=1200",
"MetricExpr": "(UNC_P_FREQ_GE_1200MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_1200mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -183,6 +201,7 @@
"EventName": "UNC_P_FREQ_GE_2000MHZ_CYCLES",
"Filter": "filter_band1=2000",
"MetricExpr": "(UNC_P_FREQ_GE_2000MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_2000mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -193,6 +212,7 @@
"EventName": "UNC_P_FREQ_GE_3000MHZ_CYCLES",
"Filter": "filter_band2=3000",
"MetricExpr": "(UNC_P_FREQ_GE_3000MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_3000mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -203,6 +223,7 @@
"EventName": "UNC_P_FREQ_GE_4000MHZ_CYCLES",
"Filter": "filter_band3=4000",
"MetricExpr": "(UNC_P_FREQ_GE_4000MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_4000mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -213,6 +234,7 @@
"EventName": "UNC_P_FREQ_GE_1200MHZ_TRANSITIONS",
"Filter": "edge=1,filter_band0=1200",
"MetricExpr": "(UNC_P_FREQ_GE_1200MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_1200mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -223,6 +245,7 @@
"EventName": "UNC_P_FREQ_GE_2000MHZ_TRANSITIONS",
"Filter": "edge=1,filter_band1=2000",
"MetricExpr": "(UNC_P_FREQ_GE_2000MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_2000mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -233,6 +256,7 @@
"EventName": "UNC_P_FREQ_GE_3000MHZ_TRANSITIONS",
"Filter": "edge=1,filter_band2=4000",
"MetricExpr": "(UNC_P_FREQ_GE_3000MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_3000mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -243,6 +267,7 @@
"EventName": "UNC_P_FREQ_GE_4000MHZ_TRANSITIONS",
"Filter": "edge=1,filter_band3=4000",
"MetricExpr": "(UNC_P_FREQ_GE_4000MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_4000mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
}
diff --git a/tools/perf/pmu-events/arch/x86/jaketown/uncore-cache.json b/tools/perf/pmu-events/arch/x86/jaketown/uncore-cache.json
index 2f23cf0..3fa61d9 100644
--- a/tools/perf/pmu-events/arch/x86/jaketown/uncore-cache.json
+++ b/tools/perf/pmu-events/arch/x86/jaketown/uncore-cache.json
@@ -1,13 +1,13 @@
[
{
- "BriefDescription": "Uncore cache clock ticks. Derived from unc_c_clockticks",
+ "BriefDescription": "Uncore cache clock ticks",
"Counter": "0,1,2,3",
"EventName": "UNC_C_CLOCKTICKS",
"PerPkg": "1",
"Unit": "CBO"
},
{
- "BriefDescription": "All LLC Misses (code+ data rd + data wr - including demand and prefetch). Derived from unc_c_llc_lookup.any",
+ "BriefDescription": "All LLC Misses (code+ data rd + data wr - including demand and prefetch)",
"Counter": "0,1",
"EventCode": "0x34",
"EventName": "UNC_C_LLC_LOOKUP.ANY",
@@ -18,7 +18,7 @@
"Unit": "CBO"
},
{
- "BriefDescription": "M line evictions from LLC (writebacks to memory). Derived from unc_c_llc_victims.m_state",
+ "BriefDescription": "M line evictions from LLC (writebacks to memory)",
"Counter": "0,1",
"EventCode": "0x37",
"EventName": "UNC_C_LLC_VICTIMS.M_STATE",
@@ -171,11 +171,12 @@
"Unit": "CBO"
},
{
- "BriefDescription": "Occupancy counter for all LLC misses; we divide this by UNC_C_CLOCKTICKS to get average Q depth. Derived from unc_c_tor_occupancy.miss_all",
+ "BriefDescription": "Occupancy counter for all LLC misses; we divide this by UNC_C_CLOCKTICKS to get average Q depth",
"EventCode": "0x36",
"EventName": "UNC_C_TOR_OCCUPANCY.MISS_ALL",
"Filter": "filter_opc=0x182",
"MetricExpr": "(UNC_C_TOR_OCCUPANCY.MISS_ALL / UNC_C_CLOCKTICKS) * 100.",
+ "MetricName": "tor_occupancy.miss_all %",
"PerPkg": "1",
"UMask": "0xa",
"Unit": "CBO"
@@ -189,7 +190,7 @@
"Unit": "CBO"
},
{
- "BriefDescription": "read requests to home agent. Derived from unc_h_requests.reads",
+ "BriefDescription": "read requests to home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.READS",
@@ -198,7 +199,7 @@
"Unit": "HA"
},
{
- "BriefDescription": "write requests to home agent. Derived from unc_h_requests.writes",
+ "BriefDescription": "write requests to home agent",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_H_REQUESTS.WRITES",
diff --git a/tools/perf/pmu-events/arch/x86/jaketown/uncore-interconnect.json b/tools/perf/pmu-events/arch/x86/jaketown/uncore-interconnect.json
index 6335187..1b53c0e 100644
--- a/tools/perf/pmu-events/arch/x86/jaketown/uncore-interconnect.json
+++ b/tools/perf/pmu-events/arch/x86/jaketown/uncore-interconnect.json
@@ -1,6 +1,6 @@
[
{
- "BriefDescription": "QPI clock ticks. Used to get percentages of QPI cycles events. Derived from unc_q_clockticks",
+ "BriefDescription": "QPI clock ticks. Used to get percentages of QPI cycles events",
"Counter": "0,1,2,3",
"EventCode": "0x14",
"EventName": "UNC_Q_CLOCKTICKS",
@@ -8,25 +8,27 @@
"Unit": "QPI LL"
},
{
- "BriefDescription": "Cycles where receiving QPI link is in half-width mode. Derived from unc_q_rxl0p_power_cycles",
+ "BriefDescription": "Cycles where receiving QPI link is in half-width mode",
"Counter": "0,1,2,3",
"EventCode": "0x10",
"EventName": "UNC_Q_RxL0P_POWER_CYCLES",
"MetricExpr": "(UNC_Q_RxL0P_POWER_CYCLES / UNC_Q_CLOCKTICKS) * 100.",
+ "MetricName": "rxl0p_power_cycles %",
"PerPkg": "1",
"Unit": "QPI LL"
},
{
- "BriefDescription": "Cycles where transmitting QPI link is in half-width mode. Derived from unc_q_txl0p_power_cycles",
+ "BriefDescription": "Cycles where transmitting QPI link is in half-width mode",
"Counter": "0,1,2,3",
"EventCode": "0xd",
"EventName": "UNC_Q_TxL0P_POWER_CYCLES",
"MetricExpr": "(UNC_Q_TxL0P_POWER_CYCLES / UNC_Q_CLOCKTICKS) * 100.",
+ "MetricName": "txl0p_power_cycles %",
"PerPkg": "1",
"Unit": "QPI LL"
},
{
- "BriefDescription": "Number of data flits transmitted . Derived from unc_q_txl_flits_g0.data",
+ "BriefDescription": "Number of data flits transmitted ",
"Counter": "0,1,2,3",
"EventName": "UNC_Q_TxL_FLITS_G0.DATA",
"PerPkg": "1",
@@ -35,7 +37,7 @@
"Unit": "QPI LL"
},
{
- "BriefDescription": "Number of non data (control) flits transmitted . Derived from unc_q_txl_flits_g0.non_data",
+ "BriefDescription": "Number of non data (control) flits transmitted ",
"Counter": "0,1,2,3",
"EventName": "UNC_Q_TxL_FLITS_G0.NON_DATA",
"PerPkg": "1",
diff --git a/tools/perf/pmu-events/arch/x86/jaketown/uncore-memory.json b/tools/perf/pmu-events/arch/x86/jaketown/uncore-memory.json
index e2cf6da..8551ceb 100644
--- a/tools/perf/pmu-events/arch/x86/jaketown/uncore-memory.json
+++ b/tools/perf/pmu-events/arch/x86/jaketown/uncore-memory.json
@@ -1,6 +1,6 @@
[
{
- "BriefDescription": "Memory page activates. Derived from unc_m_act_count",
+ "BriefDescription": "Memory page activates",
"Counter": "0,1,2,3",
"EventCode": "0x1",
"EventName": "UNC_M_ACT_COUNT",
@@ -11,7 +11,7 @@
"BriefDescription": "read requests to memory controller. Derived from unc_m_cas_count.rd",
"Counter": "0,1,2,3",
"EventCode": "0x4",
- "EventName": "UNC_M_CAS_COUNT.RD",
+ "EventName": "LLC_MISSES.MEM_READ",
"PerPkg": "1",
"UMask": "0x3",
"Unit": "iMC"
@@ -20,47 +20,50 @@
"BriefDescription": "write requests to memory controller. Derived from unc_m_cas_count.wr",
"Counter": "0,1,2,3",
"EventCode": "0x4",
- "EventName": "UNC_M_CAS_COUNT.WR",
+ "EventName": "LLC_MISSES.MEM_WRITE",
"PerPkg": "1",
"UMask": "0xc",
"Unit": "iMC"
},
{
- "BriefDescription": "Memory controller clock ticks. Used to get percentages of memory controller cycles events. Derived from unc_m_clockticks",
+ "BriefDescription": "Memory controller clock ticks. Used to get percentages of memory controller cycles events",
"Counter": "0,1,2,3",
"EventName": "UNC_M_CLOCKTICKS",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles where DRAM ranks are in power down (CKE) mode. Derived from unc_m_power_channel_ppd",
+ "BriefDescription": "Cycles where DRAM ranks are in power down (CKE) mode",
"Counter": "0,1,2,3",
"EventCode": "0x85",
"EventName": "UNC_M_POWER_CHANNEL_PPD",
"MetricExpr": "(UNC_M_POWER_CHANNEL_PPD / UNC_M_CLOCKTICKS) * 100.",
+ "MetricName": "power_channel_ppd %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles all ranks are in critical thermal throttle. Derived from unc_m_power_critical_throttle_cycles",
+ "BriefDescription": "Cycles all ranks are in critical thermal throttle",
"Counter": "0,1,2,3",
"EventCode": "0x86",
"EventName": "UNC_M_POWER_CRITICAL_THROTTLE_CYCLES",
"MetricExpr": "(UNC_M_POWER_CRITICAL_THROTTLE_CYCLES / UNC_M_CLOCKTICKS) * 100.",
+ "MetricName": "power_critical_throttle_cycles %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Cycles Memory is in self refresh power mode. Derived from unc_m_power_self_refresh",
+ "BriefDescription": "Cycles Memory is in self refresh power mode",
"Counter": "0,1,2,3",
"EventCode": "0x43",
"EventName": "UNC_M_POWER_SELF_REFRESH",
"MetricExpr": "(UNC_M_POWER_SELF_REFRESH / UNC_M_CLOCKTICKS) * 100.",
+ "MetricName": "power_self_refresh %",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Memory page conflicts. Derived from unc_m_pre_count.page_miss",
+ "BriefDescription": "Memory page conflicts",
"Counter": "0,1,2,3",
"EventCode": "0x2",
"EventName": "UNC_M_PRE_COUNT.PAGE_MISS",
@@ -69,7 +72,7 @@
"Unit": "iMC"
},
{
- "BriefDescription": "Occupancy counter for memory read queue. Derived from unc_m_rpq_occupancy",
+ "BriefDescription": "Occupancy counter for memory read queue",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_M_RPQ_OCCUPANCY",
diff --git a/tools/perf/pmu-events/arch/x86/jaketown/uncore-power.json b/tools/perf/pmu-events/arch/x86/jaketown/uncore-power.json
index bbe36d5..16034bf 100644
--- a/tools/perf/pmu-events/arch/x86/jaketown/uncore-power.json
+++ b/tools/perf/pmu-events/arch/x86/jaketown/uncore-power.json
@@ -1,44 +1,48 @@
[
{
- "BriefDescription": "PCU clock ticks. Use to get percentages of PCU cycles events. Derived from unc_p_clockticks",
+ "BriefDescription": "PCU clock ticks. Use to get percentages of PCU cycles events",
"Counter": "0,1,2,3",
"EventName": "UNC_P_CLOCKTICKS",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band0=XXX with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency. Derived from unc_p_freq_band0_cycles",
+ "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band0=XXX with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency",
"Counter": "0,1,2,3",
"EventCode": "0xb",
"EventName": "UNC_P_FREQ_BAND0_CYCLES",
"MetricExpr": "(UNC_P_FREQ_BAND0_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band0_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band1=XXX with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency. Derived from unc_p_freq_band1_cycles",
+ "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band1=XXX with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency",
"Counter": "0,1,2,3",
"EventCode": "0xc",
"EventName": "UNC_P_FREQ_BAND1_CYCLES",
"MetricExpr": "(UNC_P_FREQ_BAND1_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band1_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band2=XXX with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency. Derived from unc_p_freq_band2_cycles",
+ "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band2=XXX with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency",
"Counter": "0,1,2,3",
"EventCode": "0xd",
"EventName": "UNC_P_FREQ_BAND2_CYCLES",
"MetricExpr": "(UNC_P_FREQ_BAND2_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band2_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band3=XXX, with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency. Derived from unc_p_freq_band3_cycles",
+ "BriefDescription": "Counts the number of cycles that the uncore was running at a frequency greater than or equal to the frequency that is configured in the filter. (filter_band3=XXX, with XXX in 100Mhz units). One can also use inversion (filter_inv=1) to track cycles when we were less than the configured frequency",
"Counter": "0,1,2,3",
"EventCode": "0xe",
"EventName": "UNC_P_FREQ_BAND3_CYCLES",
"MetricExpr": "(UNC_P_FREQ_BAND3_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band3_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -49,6 +53,7 @@
"EventName": "UNC_P_FREQ_BAND0_TRANSITIONS",
"Filter": "edge=1",
"MetricExpr": "(UNC_P_FREQ_BAND0_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band0_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -59,6 +64,7 @@
"EventName": "UNC_P_FREQ_BAND1_TRANSITIONS",
"Filter": "edge=1",
"MetricExpr": "(UNC_P_FREQ_BAND1_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band1_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -69,6 +75,7 @@
"EventName": "UNC_P_FREQ_BAND2_TRANSITIONS",
"Filter": "edge=1",
"MetricExpr": "(UNC_P_FREQ_BAND2_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band2_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -79,89 +86,99 @@
"EventName": "UNC_P_FREQ_BAND3_TRANSITIONS",
"Filter": "edge=1",
"MetricExpr": "(UNC_P_FREQ_BAND3_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_band3_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C0. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details. Derived from unc_p_power_state_occupancy.cores_c0",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C0. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C0",
"Filter": "occ_sel=1",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C0 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c0 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C3. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details. Derived from unc_p_power_state_occupancy.cores_c3",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C3. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events and occupancy triggering to capture other details",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C3",
"Filter": "occ_sel=2",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C3 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c3 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C6. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events . Derived from unc_p_power_state_occupancy.cores_c6",
+ "BriefDescription": "This is an occupancy event that tracks the number of cores that are in C6. It can be used by itself to get the average number of cores in C0, with threshholding to generate histograms, or with other PCU events ",
"Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "UNC_P_POWER_STATE_OCCUPANCY.CORES_C6",
"Filter": "occ_sel=3",
"MetricExpr": "(UNC_P_POWER_STATE_OCCUPANCY.CORES_C6 / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "power_state_occupancy.cores_c6 %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles that we are in external PROCHOT mode. This mode is triggered when a sensor off the die determines that something off-die (like DRAM) is too hot and must throttle to avoid damaging the chip. Derived from unc_p_prochot_external_cycles",
+ "BriefDescription": "Counts the number of cycles that we are in external PROCHOT mode. This mode is triggered when a sensor off the die determines that something off-die (like DRAM) is too hot and must throttle to avoid damaging the chip",
"Counter": "0,1,2,3",
"EventCode": "0xa",
"EventName": "UNC_P_PROCHOT_EXTERNAL_CYCLES",
"MetricExpr": "(UNC_P_PROCHOT_EXTERNAL_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "prochot_external_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles when temperature is the upper limit on frequency. Derived from unc_p_freq_max_limit_thermal_cycles",
+ "BriefDescription": "Counts the number of cycles when temperature is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x4",
"EventName": "UNC_P_FREQ_MAX_LIMIT_THERMAL_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_LIMIT_THERMAL_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_limit_thermal_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles when the OS is the upper limit on frequency. Derived from unc_p_freq_max_os_cycles",
+ "BriefDescription": "Counts the number of cycles when the OS is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x6",
"EventName": "UNC_P_FREQ_MAX_OS_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_OS_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_os_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles when power is the upper limit on frequency. Derived from unc_p_freq_max_power_cycles",
+ "BriefDescription": "Counts the number of cycles when power is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x5",
"EventName": "UNC_P_FREQ_MAX_POWER_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_POWER_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_power_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Counts the number of cycles when current is the upper limit on frequency. Derived from unc_p_freq_max_current_cycles",
+ "BriefDescription": "Counts the number of cycles when current is the upper limit on frequency",
"Counter": "0,1,2,3",
"EventCode": "0x7",
"EventName": "UNC_P_FREQ_MAX_CURRENT_CYCLES",
"MetricExpr": "(UNC_P_FREQ_MAX_CURRENT_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_max_current_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
{
- "BriefDescription": "Cycles spent changing Frequency. Derived from unc_p_freq_trans_cycles",
+ "BriefDescription": "Cycles spent changing Frequency",
"Counter": "0,1,2,3",
"EventName": "UNC_P_FREQ_TRANS_CYCLES",
"MetricExpr": "(UNC_P_FREQ_TRANS_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_trans_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -172,6 +189,7 @@
"EventName": "UNC_P_FREQ_GE_1200MHZ_CYCLES",
"Filter": "filter_band0=1200",
"MetricExpr": "(UNC_P_FREQ_GE_1200MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_1200mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -182,6 +200,7 @@
"EventName": "UNC_P_FREQ_GE_2000MHZ_CYCLES",
"Filter": "filter_band1=2000",
"MetricExpr": "(UNC_P_FREQ_GE_2000MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_2000mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -192,6 +211,7 @@
"EventName": "UNC_P_FREQ_GE_3000MHZ_CYCLES",
"Filter": "filter_band2=3000",
"MetricExpr": "(UNC_P_FREQ_GE_3000MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_3000mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -202,6 +222,7 @@
"EventName": "UNC_P_FREQ_GE_4000MHZ_CYCLES",
"Filter": "filter_band3=4000",
"MetricExpr": "(UNC_P_FREQ_GE_4000MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_4000mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -212,6 +233,7 @@
"EventName": "UNC_P_FREQ_GE_1200MHZ_TRANSITIONS",
"Filter": "edge=1,filter_band0=1200",
"MetricExpr": "(UNC_P_FREQ_GE_1200MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_1200mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -222,6 +244,7 @@
"EventName": "UNC_P_FREQ_GE_2000MHZ_TRANSITIONS",
"Filter": "edge=1,filter_band1=2000",
"MetricExpr": "(UNC_P_FREQ_GE_2000MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_2000mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -232,6 +255,7 @@
"EventName": "UNC_P_FREQ_GE_3000MHZ_TRANSITIONS",
"Filter": "edge=1,filter_band2=4000",
"MetricExpr": "(UNC_P_FREQ_GE_3000MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_3000mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
},
@@ -242,6 +266,7 @@
"EventName": "UNC_P_FREQ_GE_4000MHZ_TRANSITIONS",
"Filter": "edge=1,filter_band3=4000",
"MetricExpr": "(UNC_P_FREQ_GE_4000MHZ_CYCLES / UNC_P_CLOCKTICKS) * 100.",
+ "MetricName": "freq_ge_4000mhz_cycles %",
"PerPkg": "1",
"Unit": "PCU"
}
diff --git a/tools/perf/pmu-events/arch/x86/mapfile.csv b/tools/perf/pmu-events/arch/x86/mapfile.csv
index 12181bb..d1a12e5 100644
--- a/tools/perf/pmu-events/arch/x86/mapfile.csv
+++ b/tools/perf/pmu-events/arch/x86/mapfile.csv
@@ -17,6 +17,7 @@
GenuineIntel-6-3E,v19,ivytown,core
GenuineIntel-6-2D,v20,jaketown,core
GenuineIntel-6-57,v9,knightslanding,core
+GenuineIntel-6-85,v9,knightslanding,core
GenuineIntel-6-1E,v2,nehalemep,core
GenuineIntel-6-1F,v2,nehalemep,core
GenuineIntel-6-1A,v2,nehalemep,core
diff --git a/tools/perf/pmu-events/arch/x86/sandybridge/uncore.json b/tools/perf/pmu-events/arch/x86/sandybridge/uncore.json
new file mode 100644
index 0000000..42c70ee
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/sandybridge/uncore.json
@@ -0,0 +1,314 @@
+[
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x01",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.MISS",
+ "BriefDescription": "A snoop misses in some processor core.",
+ "PublicDescription": "A snoop misses in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x02",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.INVAL",
+ "BriefDescription": "A snoop invalidates a non-modified line in some processor core.",
+ "PublicDescription": "A snoop invalidates a non-modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x04",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HIT",
+ "BriefDescription": "A snoop hits a non-modified line in some processor core.",
+ "PublicDescription": "A snoop hits a non-modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x08",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HITM",
+ "BriefDescription": "A snoop hits a modified line in some processor core.",
+ "PublicDescription": "A snoop hits a modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x10",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.INVAL_M",
+ "BriefDescription": "A snoop invalidates a modified line in some processor core.",
+ "PublicDescription": "A snoop invalidates a modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x20",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.EXTERNAL_FILTER",
+ "BriefDescription": "Filter on cross-core snoops initiated by this Cbox due to external snoop request.",
+ "PublicDescription": "Filter on cross-core snoops initiated by this Cbox due to external snoop request.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x40",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.XCORE_FILTER",
+ "BriefDescription": "Filter on cross-core snoops initiated by this Cbox due to processor core memory request.",
+ "PublicDescription": "Filter on cross-core snoops initiated by this Cbox due to processor core memory request.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x80",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.EVICTION_FILTER",
+ "BriefDescription": "Filter on cross-core snoops initiated by this Cbox due to LLC eviction.",
+ "PublicDescription": "Filter on cross-core snoops initiated by this Cbox due to LLC eviction.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x01",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.M",
+ "BriefDescription": "LLC lookup request that access cache and found line in M-state.",
+ "PublicDescription": "LLC lookup request that access cache and found line in M-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x02",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.E",
+ "BriefDescription": "LLC lookup request that access cache and found line in E-state.",
+ "PublicDescription": "LLC lookup request that access cache and found line in E-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x04",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.S",
+ "BriefDescription": "LLC lookup request that access cache and found line in S-state.",
+ "PublicDescription": "LLC lookup request that access cache and found line in S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x08",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.I",
+ "BriefDescription": "LLC lookup request that access cache and found line in I-state.",
+ "PublicDescription": "LLC lookup request that access cache and found line in I-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x10",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.READ_FILTER",
+ "BriefDescription": "Filter on processor core initiated cacheable read requests.",
+ "PublicDescription": "Filter on processor core initiated cacheable read requests.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x20",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.WRITE_FILTER",
+ "BriefDescription": "Filter on processor core initiated cacheable write requests.",
+ "PublicDescription": "Filter on processor core initiated cacheable write requests.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x40",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.EXTSNP_FILTER",
+ "BriefDescription": "Filter on external snoop requests.",
+ "PublicDescription": "Filter on external snoop requests.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x80",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_REQUEST_FILTER",
+ "BriefDescription": "Filter on any IRQ or IPQ initiated requests including uncacheable, non-coherent requests.",
+ "PublicDescription": "Filter on any IRQ or IPQ initiated requests including uncacheable, non-coherent requests.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x80",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.ALL",
+ "BriefDescription": "Counts cycles weighted by the number of requests waiting for data returning from the memory controller. Accounts for coherent and non-coherent requests initiated by IA cores, processor graphic units, or LLC.",
+ "PublicDescription": "Counts cycles weighted by the number of requests waiting for data returning from the memory controller. Accounts for coherent and non-coherent requests initiated by IA cores, processor graphic units, or LLC.",
+ "Counter": "0",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x81",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_REQUESTS.ALL",
+ "BriefDescription": "Counts the number of coherent and in-coherent requests initiated by IA cores, processor graphic units, or LLC.",
+ "PublicDescription": "Counts the number of coherent and in-coherent requests initiated by IA cores, processor graphic units, or LLC.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x81",
+ "UMask": "0x20",
+ "EventName": "UNC_ARB_TRK_REQUESTS.WRITES",
+ "BriefDescription": "Counts the number of allocated write entries, include full, partial, and LLC evictions.",
+ "PublicDescription": "Counts the number of allocated write entries, include full, partial, and LLC evictions.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x81",
+ "UMask": "0x80",
+ "EventName": "UNC_ARB_TRK_REQUESTS.EVICTIONS",
+ "BriefDescription": "Counts the number of LLC evictions allocated.",
+ "PublicDescription": "Counts the number of LLC evictions allocated.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x83",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_COH_TRK_OCCUPANCY.ALL",
+ "BriefDescription": "Cycles weighted by number of requests pending in Coherency Tracker.",
+ "PublicDescription": "Cycles weighted by number of requests pending in Coherency Tracker.",
+ "Counter": "0",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x84",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_COH_TRK_REQUESTS.ALL",
+ "BriefDescription": "Number of requests allocated in Coherency Tracker.",
+ "PublicDescription": "Number of requests allocated in Coherency Tracker.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x80",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.CYCLES_WITH_ANY_REQUEST",
+ "BriefDescription": "Cycles with at least one request outstanding is waiting for data return from memory controller. Account for coherent and non-coherent requests initiated by IA Cores, Processor Graphics Unit, or LLC.",
+ "PublicDescription": "Cycles with at least one request outstanding is waiting for data return from memory controller. Account for coherent and non-coherent requests initiated by IA Cores, Processor Graphics Unit, or LLC.",
+ "Counter": "0,1",
+ "CounterMask": "1",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x80",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.CYCLES_OVER_HALF_FULL",
+ "BriefDescription": "Cycles with at least half of the requests outstanding are waiting for data return from memory controller. Account for coherent and non-coherent requests initiated by IA Cores, Processor Graphics Unit, or LLC.",
+ "PublicDescription": "Cycles with at least half of the requests outstanding are waiting for data return from memory controller. Account for coherent and non-coherent requests initiated by IA Cores, Processor Graphics Unit, or LLC.",
+ "Counter": "0,1",
+ "CounterMask": "10",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "ARB",
+ "EventCode": "0x0",
+ "UMask": "0x01",
+ "EventName": "UNC_CLOCK.SOCKET",
+ "BriefDescription": "This 48-bit fixed counter counts the UCLK cycles.",
+ "PublicDescription": "This 48-bit fixed counter counts the UCLK cycles.",
+ "Counter": "Fixed",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x06",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ES",
+ "BriefDescription": "LLC lookup request that access cache and found line in E-state or S-state.",
+ "PublicDescription": "LLC lookup request that access cache and found line in E-state or S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ }
+]
\ No newline at end of file
diff --git a/tools/perf/pmu-events/arch/x86/skylake/uncore.json b/tools/perf/pmu-events/arch/x86/skylake/uncore.json
new file mode 100644
index 0000000..dbc1932
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/skylake/uncore.json
@@ -0,0 +1,254 @@
+[
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x41",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.MISS_XCORE",
+ "BriefDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which misses in some processor core.",
+ "PublicDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which misses in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x81",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.MISS_EVICTION",
+ "BriefDescription": "A cross-core snoop resulted from L3 Eviction which misses in some processor core.",
+ "PublicDescription": "A cross-core snoop resulted from L3 Eviction which misses in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x44",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HIT_XCORE",
+ "BriefDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which hits a non-modified line in some processor core.",
+ "PublicDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which hits a non-modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x22",
+ "UMask": "0x48",
+ "EventName": "UNC_CBO_XSNP_RESPONSE.HITM_XCORE",
+ "BriefDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which hits a modified line in some processor core.",
+ "PublicDescription": "A cross-core snoop initiated by this Cbox due to processor core memory request which hits a modified line in some processor core.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x21",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.WRITE_M",
+ "BriefDescription": "L3 Lookup write request that access cache and found line in M-state",
+ "PublicDescription": "L3 Lookup write request that access cache and found line in M-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x81",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_M",
+ "BriefDescription": "L3 Lookup any request that access cache and found line in M-state",
+ "PublicDescription": "L3 Lookup any request that access cache and found line in M-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x18",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.READ_I",
+ "BriefDescription": "L3 Lookup read request that access cache and found line in I-state",
+ "PublicDescription": "L3 Lookup read request that access cache and found line in I-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x88",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_I",
+ "BriefDescription": "L3 Lookup any request that access cache and found line in I-state",
+ "PublicDescription": "L3 Lookup any request that access cache and found line in I-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x1f",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.READ_MESI",
+ "BriefDescription": "L3 Lookup read request that access cache and found line in any MESI-state",
+ "PublicDescription": "L3 Lookup read request that access cache and found line in any MESI-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x2f",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.WRITE_MESI",
+ "BriefDescription": "L3 Lookup write request that access cache and found line in MESI-state",
+ "PublicDescription": "L3 Lookup write request that access cache and found line in MESI-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x8f",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_MESI",
+ "BriefDescription": "L3 Lookup any request that access cache and found line in MESI-state",
+ "PublicDescription": "L3 Lookup any request that access cache and found line in MESI-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x86",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.ANY_ES",
+ "BriefDescription": "L3 Lookup any request that access cache and found line in E or S-state",
+ "PublicDescription": "L3 Lookup any request that access cache and found line in E or S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x16",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.READ_ES",
+ "BriefDescription": "L3 Lookup read request that access cache and found line in E or S-state",
+ "PublicDescription": "L3 Lookup read request that access cache and found line in E or S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "CBO",
+ "EventCode": "0x34",
+ "UMask": "0x26",
+ "EventName": "UNC_CBO_CACHE_LOOKUP.WRITE_ES",
+ "BriefDescription": "L3 Lookup write request that access cache and found line in E or S-state",
+ "PublicDescription": "L3 Lookup write request that access cache and found line in E or S-state.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x80",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.ALL",
+ "BriefDescription": "Each cycle count number of all Core outgoing valid entries. Such entry is defined as valid from its allocation till first of IDI0 or DRS0 messages is sent out. Accounts for Coherent and non-coherent traffic.",
+ "PublicDescription": "Each cycle count number of all Core outgoing valid entries. Such entry is defined as valid from its allocation till first of IDI0 or DRS0 messages is sent out. Accounts for Coherent and non-coherent traffic.",
+ "Counter": "0",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x81",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_REQUESTS.ALL",
+ "BriefDescription": "Total number of Core outgoing entries allocated. Accounts for Coherent and non-coherent traffic.",
+ "PublicDescription": "Total number of Core outgoing entries allocated. Accounts for Coherent and non-coherent traffic.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x81",
+ "UMask": "0x02",
+ "EventName": "UNC_ARB_TRK_REQUESTS.DRD_DIRECT",
+ "BriefDescription": "Number of Core coherent Data Read entries allocated in DirectData mode",
+ "PublicDescription": "Number of Core coherent Data Read entries allocated in DirectData mode.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x81",
+ "UMask": "0x20",
+ "EventName": "UNC_ARB_TRK_REQUESTS.WRITES",
+ "BriefDescription": "Number of Writes allocated - any write transactions: full/partials writes and evictions.",
+ "PublicDescription": "Number of Writes allocated - any write transactions: full/partials writes and evictions.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x84",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_COH_TRK_REQUESTS.ALL",
+ "BriefDescription": "Number of entries allocated. Account for Any type: e.g. Snoop, Core aperture, etc.",
+ "PublicDescription": "Number of entries allocated. Account for Any type: e.g. Snoop, Core aperture, etc.",
+ "Counter": "0,1",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "iMPH-U",
+ "EventCode": "0x80",
+ "UMask": "0x01",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.CYCLES_WITH_ANY_REQUEST",
+ "BriefDescription": "Cycles with at least one request outstanding is waiting for data return from memory controller. Account for coherent and non-coherent requests initiated by IA Cores, Processor Graphics Unit, or LLC.;",
+ "PublicDescription": "Cycles with at least one request outstanding is waiting for data return from memory controller. Account for coherent and non-coherent requests initiated by IA Cores, Processor Graphics Unit, or LLC.",
+ "Counter": "0",
+ "CounterMask": "1",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ },
+ {
+ "Unit": "NCU",
+ "EventCode": "0x0",
+ "UMask": "0x01",
+ "EventName": "UNC_CLOCK.SOCKET",
+ "BriefDescription": "This 48-bit fixed counter counts the UCLK cycles",
+ "PublicDescription": "This 48-bit fixed counter counts the UCLK cycles.",
+ "Counter": "FIXED",
+ "CounterMask": "0",
+ "Invert": "0",
+ "EdgeDetect": "0"
+ }
+]
\ No newline at end of file
diff --git a/tools/perf/pmu-events/jevents.c b/tools/perf/pmu-events/jevents.c
index eed0934..baa073f 100644
--- a/tools/perf/pmu-events/jevents.c
+++ b/tools/perf/pmu-events/jevents.c
@@ -195,6 +195,7 @@
{ "CBO", "uncore_cbox" },
{ "QPI LL", "uncore_qpi" },
{ "SBO", "uncore_sbox" },
+ { "iMPH-U", "uncore_arb" },
{}
};
@@ -291,7 +292,9 @@
static int print_events_table_entry(void *data, char *name, char *event,
char *desc, char *long_desc,
- char *pmu, char *unit, char *perpkg)
+ char *pmu, char *unit, char *perpkg,
+ char *metric_expr,
+ char *metric_name)
{
struct perf_entry_data *pd = data;
FILE *outfp = pd->outfp;
@@ -315,6 +318,10 @@
fprintf(outfp, "\t.unit = \"%s\",\n", unit);
if (perpkg)
fprintf(outfp, "\t.perpkg = \"%s\",\n", perpkg);
+ if (metric_expr)
+ fprintf(outfp, "\t.metric_expr = \"%s\",\n", metric_expr);
+ if (metric_name)
+ fprintf(outfp, "\t.metric_name = \"%s\",\n", metric_name);
fprintf(outfp, "},\n");
return 0;
@@ -362,7 +369,9 @@
int json_events(const char *fn,
int (*func)(void *data, char *name, char *event, char *desc,
char *long_desc,
- char *pmu, char *unit, char *perpkg),
+ char *pmu, char *unit, char *perpkg,
+ char *metric_expr,
+ char *metric_name),
void *data)
{
int err = -EIO;
@@ -388,6 +397,8 @@
char *filter = NULL;
char *perpkg = NULL;
char *unit = NULL;
+ char *metric_expr = NULL;
+ char *metric_name = NULL;
unsigned long long eventcode = 0;
struct msrmap *msr = NULL;
jsmntok_t *msrval = NULL;
@@ -398,6 +409,7 @@
for (j = 0; j < obj->size; j += 2) {
jsmntok_t *field, *val;
int nz;
+ char *s;
field = tok + j;
EXPECT(field->type == JSMN_STRING, tok + j,
@@ -444,7 +456,6 @@
NULL);
} else if (json_streq(map, field, "Unit")) {
const char *ppmu;
- char *s;
ppmu = field_to_perf(unit_to_pmu, map, val);
if (ppmu) {
@@ -458,12 +469,19 @@
}
addfield(map, &desc, ". ", "Unit: ", NULL);
addfield(map, &desc, "", pmu, NULL);
+ addfield(map, &desc, "", " ", NULL);
} else if (json_streq(map, field, "Filter")) {
addfield(map, &filter, "", "", val);
} else if (json_streq(map, field, "ScaleUnit")) {
addfield(map, &unit, "", "", val);
} else if (json_streq(map, field, "PerPkg")) {
addfield(map, &perpkg, "", "", val);
+ } else if (json_streq(map, field, "MetricName")) {
+ addfield(map, &metric_name, "", "", val);
+ } else if (json_streq(map, field, "MetricExpr")) {
+ addfield(map, &metric_expr, "", "", val);
+ for (s = metric_expr; *s; s++)
+ *s = tolower(*s);
}
/* ignore unknown fields */
}
@@ -488,7 +506,7 @@
fixname(name);
err = func(data, name, real_event(name, event), desc, long_desc,
- pmu, unit, perpkg);
+ pmu, unit, perpkg, metric_expr, metric_name);
free(event);
free(desc);
free(name);
@@ -498,6 +516,8 @@
free(filter);
free(perpkg);
free(unit);
+ free(metric_expr);
+ free(metric_name);
if (err)
break;
tok += j;
diff --git a/tools/perf/pmu-events/jevents.h b/tools/perf/pmu-events/jevents.h
index 71e13de..611fac0 100644
--- a/tools/perf/pmu-events/jevents.h
+++ b/tools/perf/pmu-events/jevents.h
@@ -5,7 +5,8 @@
int (*func)(void *data, char *name, char *event, char *desc,
char *long_desc,
char *pmu,
- char *unit, char *perpkg),
+ char *unit, char *perpkg, char *metric_expr,
+ char *metric_name),
void *data);
char *get_cpu_str(void);
diff --git a/tools/perf/pmu-events/pmu-events.h b/tools/perf/pmu-events/pmu-events.h
index c669a3c..569eab3 100644
--- a/tools/perf/pmu-events/pmu-events.h
+++ b/tools/perf/pmu-events/pmu-events.h
@@ -13,6 +13,8 @@
const char *pmu;
const char *unit;
const char *perpkg;
+ const char *metric_expr;
+ const char *metric_name;
};
/*
diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
index 1cb3d9b..af58ebc 100644
--- a/tools/perf/tests/Build
+++ b/tools/perf/tests/Build
@@ -38,6 +38,7 @@
perf-y += stat.o
perf-y += event_update.o
perf-y += event-times.o
+perf-y += expr.o
perf-y += backward-ring-buffer.o
perf-y += sdt.o
perf-y += is_printable_array.o
diff --git a/tools/perf/tests/attr.c b/tools/perf/tests/attr.c
index 88dc51f..0dd77494 100644
--- a/tools/perf/tests/attr.c
+++ b/tools/perf/tests/attr.c
@@ -18,10 +18,16 @@
* permissions. All the event text files are stored there.
*/
+#include <errno.h>
+#include <inttypes.h>
#include <stdlib.h>
#include <stdio.h>
#include <linux/types.h>
#include <linux/kernel.h>
+#include <sys/param.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
#include "../perf.h"
#include "util.h"
#include <subcmd/exec-cmd.h>
diff --git a/tools/perf/tests/backward-ring-buffer.c b/tools/perf/tests/backward-ring-buffer.c
index 42e892b..50f6d7a 100644
--- a/tools/perf/tests/backward-ring-buffer.c
+++ b/tools/perf/tests/backward-ring-buffer.c
@@ -8,6 +8,7 @@
#include <sys/prctl.h>
#include "tests.h"
#include "debug.h"
+#include <errno.h>
#define NR_ITERS 111
diff --git a/tools/perf/tests/bpf.c b/tools/perf/tests/bpf.c
index 1a04fe7..5876da1 100644
--- a/tools/perf/tests/bpf.c
+++ b/tools/perf/tests/bpf.c
@@ -1,10 +1,14 @@
+#include <errno.h>
#include <stdio.h>
#include <sys/epoll.h>
+#include <sys/types.h>
+#include <sys/stat.h>
#include <util/util.h>
#include <util/bpf-loader.h>
#include <util/evlist.h>
#include <linux/bpf.h>
#include <linux/filter.h>
+#include <linux/kernel.h>
#include <api/fs/fs.h>
#include <bpf/bpf.h>
#include "tests.h"
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index 83c4669..9e08d29 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -3,8 +3,10 @@
*
* Builtin regression testing command: ever growing number of sanity tests
*/
+#include <errno.h>
#include <unistd.h>
#include <string.h>
+#include <sys/wait.h>
#include "builtin.h"
#include "hist.h"
#include "intlist.h"
@@ -13,6 +15,7 @@
#include "color.h"
#include <subcmd/parse-options.h>
#include "symbol.h"
+#include <linux/kernel.h>
static bool dont_fork;
@@ -44,6 +47,10 @@
.func = test__parse_events,
},
{
+ .desc = "Simple expression parser",
+ .func = test__expr,
+ },
+ {
.desc = "PERF_RECORD_* events & perf_sample fields",
.func = test__PERF_RECORD,
},
@@ -460,7 +467,7 @@
return 0;
}
-int cmd_test(int argc, const char **argv, const char *prefix __maybe_unused)
+int cmd_test(int argc, const char **argv)
{
const char *test_usage[] = {
"perf test [<options>] [{list <test-name-fragment>|[<test-name-fragments>|<test-numbers>]}]",
diff --git a/tools/perf/tests/clang.c b/tools/perf/tests/clang.c
index f853e24..c5bb220 100644
--- a/tools/perf/tests/clang.c
+++ b/tools/perf/tests/clang.c
@@ -2,6 +2,7 @@
#include "debug.h"
#include "util.h"
#include "c++/clang-c.h"
+#include <linux/kernel.h>
static struct {
int (*func)(void);
diff --git a/tools/perf/tests/code-reading.c b/tools/perf/tests/code-reading.c
index d1f6930..1f14e76 100644
--- a/tools/perf/tests/code-reading.c
+++ b/tools/perf/tests/code-reading.c
@@ -1,9 +1,12 @@
+#include <errno.h>
+#include <linux/kernel.h>
#include <linux/types.h>
+#include <inttypes.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
-#include <ctype.h>
#include <string.h>
+#include <sys/param.h>
#include "parse-events.h"
#include "evlist.h"
@@ -16,6 +19,8 @@
#include "tests.h"
+#include "sane_ctype.h"
+
#define BUFSZ 1024
#define READLEN 128
diff --git a/tools/perf/tests/cpumap.c b/tools/perf/tests/cpumap.c
index f168a85..4478773 100644
--- a/tools/perf/tests/cpumap.c
+++ b/tools/perf/tests/cpumap.c
@@ -66,7 +66,7 @@
TEST_ASSERT_VAL("wrong nr", map->nr == 2);
TEST_ASSERT_VAL("wrong cpu", map->map[0] == 1);
TEST_ASSERT_VAL("wrong cpu", map->map[1] == 256);
- TEST_ASSERT_VAL("wrong refcnt", atomic_read(&map->refcnt) == 1);
+ TEST_ASSERT_VAL("wrong refcnt", refcount_read(&map->refcnt) == 1);
cpu_map__put(map);
return 0;
}
diff --git a/tools/perf/tests/dso-data.c b/tools/perf/tests/dso-data.c
index 13725e0..8f08df5 100644
--- a/tools/perf/tests/dso-data.c
+++ b/tools/perf/tests/dso-data.c
@@ -1,4 +1,6 @@
+#include <dirent.h>
#include <stdlib.h>
+#include <linux/kernel.h>
#include <linux/types.h>
#include <sys/stat.h>
#include <fcntl.h>
diff --git a/tools/perf/tests/dwarf-unwind.c b/tools/perf/tests/dwarf-unwind.c
index 1046491..dfe5c89 100644
--- a/tools/perf/tests/dwarf-unwind.c
+++ b/tools/perf/tests/dwarf-unwind.c
@@ -1,5 +1,6 @@
#include <linux/compiler.h>
#include <linux/types.h>
+#include <inttypes.h>
#include <unistd.h>
#include "tests.h"
#include "debug.h"
diff --git a/tools/perf/tests/event-times.c b/tools/perf/tests/event-times.c
index 19ef77b..634f20c 100644
--- a/tools/perf/tests/event-times.c
+++ b/tools/perf/tests/event-times.c
@@ -1,5 +1,8 @@
#include <linux/compiler.h>
+#include <errno.h>
+#include <inttypes.h>
#include <string.h>
+#include <sys/wait.h>
#include "tests.h"
#include "evlist.h"
#include "evsel.h"
diff --git a/tools/perf/tests/evsel-roundtrip-name.c b/tools/perf/tests/evsel-roundtrip-name.c
index 60926a1..d2bea6f 100644
--- a/tools/perf/tests/evsel-roundtrip-name.c
+++ b/tools/perf/tests/evsel-roundtrip-name.c
@@ -3,6 +3,8 @@
#include "parse-events.h"
#include "tests.h"
#include "debug.h"
+#include <errno.h>
+#include <linux/kernel.h>
static int perf_evsel__roundtrip_cache_name_test(void)
{
diff --git a/tools/perf/tests/expr.c b/tools/perf/tests/expr.c
new file mode 100644
index 0000000..6c6a374
--- /dev/null
+++ b/tools/perf/tests/expr.c
@@ -0,0 +1,56 @@
+#include "util/debug.h"
+#include "util/expr.h"
+#include "tests.h"
+#include <stdlib.h>
+
+static int test(struct parse_ctx *ctx, const char *e, double val2)
+{
+ double val;
+
+ if (expr__parse(&val, ctx, &e))
+ TEST_ASSERT_VAL("parse test failed", 0);
+ TEST_ASSERT_VAL("unexpected value", val == val2);
+ return 0;
+}
+
+int test__expr(int subtest __maybe_unused)
+{
+ const char *p;
+ const char **other;
+ double val;
+ int ret;
+ struct parse_ctx ctx;
+ int num_other;
+
+ expr__ctx_init(&ctx);
+ expr__add_id(&ctx, "FOO", 1);
+ expr__add_id(&ctx, "BAR", 2);
+
+ ret = test(&ctx, "1+1", 2);
+ ret |= test(&ctx, "FOO+BAR", 3);
+ ret |= test(&ctx, "(BAR/2)%2", 1);
+ ret |= test(&ctx, "1 - -4", 5);
+ ret |= test(&ctx, "(FOO-1)*2 + (BAR/2)%2 - -4", 5);
+
+ if (ret)
+ return ret;
+
+ p = "FOO/0";
+ ret = expr__parse(&val, &ctx, &p);
+ TEST_ASSERT_VAL("division by zero", ret == 1);
+
+ p = "BAR/";
+ ret = expr__parse(&val, &ctx, &p);
+ TEST_ASSERT_VAL("missing operand", ret == 1);
+
+ TEST_ASSERT_VAL("find other",
+ expr__find_other("FOO + BAR + BAZ + BOZO", "FOO", &other, &num_other) == 0);
+ TEST_ASSERT_VAL("find other", num_other == 3);
+ TEST_ASSERT_VAL("find other", !strcmp(other[0], "BAR"));
+ TEST_ASSERT_VAL("find other", !strcmp(other[1], "BAZ"));
+ TEST_ASSERT_VAL("find other", !strcmp(other[2], "BOZO"));
+ TEST_ASSERT_VAL("find other", other[3] == NULL);
+ free((void *)other);
+
+ return 0;
+}
diff --git a/tools/perf/tests/hists_common.c b/tools/perf/tests/hists_common.c
index 6b21746..00b8dc5 100644
--- a/tools/perf/tests/hists_common.c
+++ b/tools/perf/tests/hists_common.c
@@ -1,3 +1,4 @@
+#include <inttypes.h>
#include "perf.h"
#include "util/debug.h"
#include "util/symbol.h"
@@ -7,6 +8,7 @@
#include "util/machine.h"
#include "util/thread.h"
#include "tests/hists_common.h"
+#include <linux/kernel.h>
static struct {
u32 pid;
diff --git a/tools/perf/tests/hists_cumulate.c b/tools/perf/tests/hists_cumulate.c
index 9fd54b7..70918b9 100644
--- a/tools/perf/tests/hists_cumulate.c
+++ b/tools/perf/tests/hists_cumulate.c
@@ -9,6 +9,7 @@
#include "util/parse-events.h"
#include "tests/tests.h"
#include "tests/hists_common.h"
+#include <linux/kernel.h>
struct sample {
u32 pid;
diff --git a/tools/perf/tests/hists_filter.c b/tools/perf/tests/hists_filter.c
index 62efb14..f171b2d 100644
--- a/tools/perf/tests/hists_filter.c
+++ b/tools/perf/tests/hists_filter.c
@@ -9,6 +9,7 @@
#include "util/parse-events.h"
#include "tests/tests.h"
#include "tests/hists_common.h"
+#include <linux/kernel.h>
struct sample {
u32 pid;
diff --git a/tools/perf/tests/hists_link.c b/tools/perf/tests/hists_link.c
index eddc740..a26cbb7 100644
--- a/tools/perf/tests/hists_link.c
+++ b/tools/perf/tests/hists_link.c
@@ -9,6 +9,8 @@
#include "thread.h"
#include "parse-events.h"
#include "hists_common.h"
+#include <errno.h>
+#include <linux/kernel.h>
struct sample {
u32 pid;
diff --git a/tools/perf/tests/hists_output.c b/tools/perf/tests/hists_output.c
index 63c5efa..cdf0dde 100644
--- a/tools/perf/tests/hists_output.c
+++ b/tools/perf/tests/hists_output.c
@@ -9,6 +9,7 @@
#include "util/parse-events.h"
#include "tests/tests.h"
#include "tests/hists_common.h"
+#include <linux/kernel.h>
struct sample {
u32 cpu;
diff --git a/tools/perf/tests/is_printable_array.c b/tools/perf/tests/is_printable_array.c
index 42e1339..a5192f6 100644
--- a/tools/perf/tests/is_printable_array.c
+++ b/tools/perf/tests/is_printable_array.c
@@ -1,7 +1,8 @@
#include <linux/compiler.h>
+#include <linux/kernel.h>
#include "tests.h"
#include "debug.h"
-#include "util.h"
+#include "print_binary.h"
int test__is_printable_array(int subtest __maybe_unused)
{
diff --git a/tools/perf/tests/mmap-basic.c b/tools/perf/tests/mmap-basic.c
index 634bce9..15c7708 100644
--- a/tools/perf/tests/mmap-basic.c
+++ b/tools/perf/tests/mmap-basic.c
@@ -1,3 +1,5 @@
+#include <errno.h>
+#include <inttypes.h>
/* For the CLR_() macros */
#include <pthread.h>
@@ -7,6 +9,7 @@
#include "cpumap.h"
#include "tests.h"
#include <linux/err.h>
+#include <linux/kernel.h>
/*
* This test will generate random numbers of calls to some getpid syscalls,
diff --git a/tools/perf/tests/mmap-thread-lookup.c b/tools/perf/tests/mmap-thread-lookup.c
index 0c5ce44..6ea4d8a 100644
--- a/tools/perf/tests/mmap-thread-lookup.c
+++ b/tools/perf/tests/mmap-thread-lookup.c
@@ -1,3 +1,4 @@
+#include <inttypes.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <sys/types.h>
@@ -11,6 +12,7 @@
#include "thread_map.h"
#include "symbol.h"
#include "thread.h"
+#include "util.h"
#define THREADS 4
diff --git a/tools/perf/tests/openat-syscall-all-cpus.c b/tools/perf/tests/openat-syscall-all-cpus.c
index c8d9592..1a74dd9 100644
--- a/tools/perf/tests/openat-syscall-all-cpus.c
+++ b/tools/perf/tests/openat-syscall-all-cpus.c
@@ -1,8 +1,14 @@
+#include <errno.h>
+#include <inttypes.h>
/* For the CPU_* macros */
#include <pthread.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
#include <api/fs/fs.h>
#include <linux/err.h>
+#include <api/fs/tracing_path.h>
#include "evsel.h"
#include "tests.h"
#include "thread_map.h"
diff --git a/tools/perf/tests/openat-syscall-tp-fields.c b/tools/perf/tests/openat-syscall-tp-fields.c
index f52239f..9788fac 100644
--- a/tools/perf/tests/openat-syscall-tp-fields.c
+++ b/tools/perf/tests/openat-syscall-tp-fields.c
@@ -5,6 +5,7 @@
#include "thread_map.h"
#include "tests.h"
#include "debug.h"
+#include <errno.h>
#ifndef O_DIRECTORY
#define O_DIRECTORY 00200000
diff --git a/tools/perf/tests/openat-syscall.c b/tools/perf/tests/openat-syscall.c
index d741412..e44506e 100644
--- a/tools/perf/tests/openat-syscall.c
+++ b/tools/perf/tests/openat-syscall.c
@@ -1,5 +1,10 @@
+#include <errno.h>
+#include <inttypes.h>
#include <api/fs/tracing_path.h>
#include <linux/err.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
#include "thread_map.h"
#include "evsel.h"
#include "debug.h"
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 1dc8380..7fad885 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -1,4 +1,3 @@
-
#include "parse-events.h"
#include "evsel.h"
#include "evlist.h"
@@ -6,8 +5,15 @@
#include "tests.h"
#include "debug.h"
#include "util.h"
+#include <dirent.h>
+#include <errno.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+#include <linux/kernel.h>
#include <linux/hw_breakpoint.h>
#include <api/fs/fs.h>
+#include <api/fs/tracing_path.h>
#define PERF_TP_SAMPLE_TYPE (PERF_SAMPLE_RAW | PERF_SAMPLE_TIME | \
PERF_SAMPLE_CPU | PERF_SAMPLE_PERIOD)
diff --git a/tools/perf/tests/parse-no-sample-id-all.c b/tools/perf/tests/parse-no-sample-id-all.c
index 65dcf48..c6207db 100644
--- a/tools/perf/tests/parse-no-sample-id-all.c
+++ b/tools/perf/tests/parse-no-sample-id-all.c
@@ -1,3 +1,4 @@
+#include <linux/kernel.h>
#include <linux/types.h>
#include <stddef.h>
diff --git a/tools/perf/tests/perf-record.c b/tools/perf/tests/perf-record.c
index 87893f3..d37cd95 100644
--- a/tools/perf/tests/perf-record.c
+++ b/tools/perf/tests/perf-record.c
@@ -1,3 +1,5 @@
+#include <errno.h>
+#include <inttypes.h>
/* For the CLR_() macros */
#include <pthread.h>
diff --git a/tools/perf/tests/pmu.c b/tools/perf/tests/pmu.c
index 1e2ba26..a6d7aef 100644
--- a/tools/perf/tests/pmu.c
+++ b/tools/perf/tests/pmu.c
@@ -2,6 +2,8 @@
#include "pmu.h"
#include "util.h"
#include "tests.h"
+#include <errno.h>
+#include <linux/kernel.h>
/* Simulated format definitions. */
static struct test_format {
diff --git a/tools/perf/tests/sample-parsing.c b/tools/perf/tests/sample-parsing.c
index 5f23710..bac5c38 100644
--- a/tools/perf/tests/sample-parsing.c
+++ b/tools/perf/tests/sample-parsing.c
@@ -1,4 +1,6 @@
#include <stdbool.h>
+#include <inttypes.h>
+#include <linux/kernel.h>
#include <linux/types.h>
#include "util.h"
diff --git a/tools/perf/tests/sdt.c b/tools/perf/tests/sdt.c
index f59d210..f73b3c5 100644
--- a/tools/perf/tests/sdt.c
+++ b/tools/perf/tests/sdt.c
@@ -1,3 +1,4 @@
+#include <errno.h>
#include <stdio.h>
#include <sys/epoll.h>
#include <util/util.h>
@@ -43,7 +44,7 @@
{
char *buf = calloc(PATH_MAX, sizeof(char));
- if (buf && readlink("/proc/self/exe", buf, PATH_MAX) < 0) {
+ if (buf && readlink("/proc/self/exe", buf, PATH_MAX - 1) < 0) {
pr_debug("Failed to get correct path of perf\n");
free(buf);
return NULL;
diff --git a/tools/perf/tests/sw-clock.c b/tools/perf/tests/sw-clock.c
index 4c9fd04..828494d 100644
--- a/tools/perf/tests/sw-clock.c
+++ b/tools/perf/tests/sw-clock.c
@@ -1,3 +1,5 @@
+#include <errno.h>
+#include <inttypes.h>
#include <unistd.h>
#include <stdlib.h>
#include <signal.h>
diff --git a/tools/perf/tests/switch-tracking.c b/tools/perf/tests/switch-tracking.c
index 7ddbe26..65474fd 100644
--- a/tools/perf/tests/switch-tracking.c
+++ b/tools/perf/tests/switch-tracking.c
@@ -1,5 +1,6 @@
#include <sys/time.h>
#include <sys/prctl.h>
+#include <errno.h>
#include <time.h>
#include <stdlib.h>
diff --git a/tools/perf/tests/task-exit.c b/tools/perf/tests/task-exit.c
index 01a5ba2..32873ec 100644
--- a/tools/perf/tests/task-exit.c
+++ b/tools/perf/tests/task-exit.c
@@ -4,6 +4,7 @@
#include "cpumap.h"
#include "tests.h"
+#include <errno.h>
#include <signal.h>
static int exited;
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index 1fa9b9d..6318596 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -62,6 +62,7 @@
int test__keep_tracking(int subtest);
int test__parse_no_sample_id_all(int subtest);
int test__dwarf_unwind(int subtest);
+int test__expr(int subtest);
int test__hists_filter(int subtest);
int test__mmap_thread_lookup(int subtest);
int test__thread_mg_share(int subtest);
diff --git a/tools/perf/tests/thread-map.c b/tools/perf/tests/thread-map.c
index f2d2e54..a63d694 100644
--- a/tools/perf/tests/thread-map.c
+++ b/tools/perf/tests/thread-map.c
@@ -29,7 +29,7 @@
thread_map__comm(map, 0) &&
!strcmp(thread_map__comm(map, 0), NAME));
TEST_ASSERT_VAL("wrong refcnt",
- atomic_read(&map->refcnt) == 1);
+ refcount_read(&map->refcnt) == 1);
thread_map__put(map);
/* test dummy pid */
@@ -44,7 +44,7 @@
thread_map__comm(map, 0) &&
!strcmp(thread_map__comm(map, 0), "dummy"));
TEST_ASSERT_VAL("wrong refcnt",
- atomic_read(&map->refcnt) == 1);
+ refcount_read(&map->refcnt) == 1);
thread_map__put(map);
return 0;
}
@@ -71,7 +71,7 @@
thread_map__comm(threads, 0) &&
!strcmp(thread_map__comm(threads, 0), NAME));
TEST_ASSERT_VAL("wrong refcnt",
- atomic_read(&threads->refcnt) == 1);
+ refcount_read(&threads->refcnt) == 1);
thread_map__put(threads);
return 0;
}
diff --git a/tools/perf/tests/thread-mg-share.c b/tools/perf/tests/thread-mg-share.c
index 188b631..76686dd 100644
--- a/tools/perf/tests/thread-mg-share.c
+++ b/tools/perf/tests/thread-mg-share.c
@@ -43,7 +43,7 @@
leader && t1 && t2 && t3 && other);
mg = leader->mg;
- TEST_ASSERT_EQUAL("wrong refcnt", atomic_read(&mg->refcnt), 4);
+ TEST_ASSERT_EQUAL("wrong refcnt", refcount_read(&mg->refcnt), 4);
/* test the map groups pointer is shared */
TEST_ASSERT_VAL("map groups don't match", mg == t1->mg);
@@ -71,25 +71,25 @@
machine__remove_thread(machine, other_leader);
other_mg = other->mg;
- TEST_ASSERT_EQUAL("wrong refcnt", atomic_read(&other_mg->refcnt), 2);
+ TEST_ASSERT_EQUAL("wrong refcnt", refcount_read(&other_mg->refcnt), 2);
TEST_ASSERT_VAL("map groups don't match", other_mg == other_leader->mg);
/* release thread group */
thread__put(leader);
- TEST_ASSERT_EQUAL("wrong refcnt", atomic_read(&mg->refcnt), 3);
+ TEST_ASSERT_EQUAL("wrong refcnt", refcount_read(&mg->refcnt), 3);
thread__put(t1);
- TEST_ASSERT_EQUAL("wrong refcnt", atomic_read(&mg->refcnt), 2);
+ TEST_ASSERT_EQUAL("wrong refcnt", refcount_read(&mg->refcnt), 2);
thread__put(t2);
- TEST_ASSERT_EQUAL("wrong refcnt", atomic_read(&mg->refcnt), 1);
+ TEST_ASSERT_EQUAL("wrong refcnt", refcount_read(&mg->refcnt), 1);
thread__put(t3);
/* release other group */
thread__put(other_leader);
- TEST_ASSERT_EQUAL("wrong refcnt", atomic_read(&other_mg->refcnt), 1);
+ TEST_ASSERT_EQUAL("wrong refcnt", refcount_read(&other_mg->refcnt), 1);
thread__put(other);
diff --git a/tools/perf/tests/unit_number__scnprintf.c b/tools/perf/tests/unit_number__scnprintf.c
index 623c2aa..44589de 100644
--- a/tools/perf/tests/unit_number__scnprintf.c
+++ b/tools/perf/tests/unit_number__scnprintf.c
@@ -1,7 +1,8 @@
+#include <inttypes.h>
#include <linux/compiler.h>
#include <linux/types.h>
#include "tests.h"
-#include "util.h"
+#include "units.h"
#include "debug.h"
int test__unit_number__scnprint(int subtest __maybe_unused)
diff --git a/tools/perf/tests/vmlinux-kallsyms.c b/tools/perf/tests/vmlinux-kallsyms.c
index 862b043..8456175 100644
--- a/tools/perf/tests/vmlinux-kallsyms.c
+++ b/tools/perf/tests/vmlinux-kallsyms.c
@@ -1,5 +1,6 @@
#include <linux/compiler.h>
#include <linux/rbtree.h>
+#include <inttypes.h>
#include <string.h>
#include "map.h"
#include "symbol.h"
diff --git a/tools/perf/trace/beauty/Build b/tools/perf/trace/beauty/Build
new file mode 100644
index 0000000..be95ac6
--- /dev/null
+++ b/tools/perf/trace/beauty/Build
@@ -0,0 +1 @@
+libperf-y += statx.o
diff --git a/tools/perf/trace/beauty/beauty.h b/tools/perf/trace/beauty/beauty.h
new file mode 100644
index 0000000..cf50be3
--- /dev/null
+++ b/tools/perf/trace/beauty/beauty.h
@@ -0,0 +1,24 @@
+#ifndef _PERF_TRACE_BEAUTY_H
+#define _PERF_TRACE_BEAUTY_H
+
+#include <linux/types.h>
+
+struct trace;
+struct thread;
+
+struct syscall_arg {
+ unsigned long val;
+ struct thread *thread;
+ struct trace *trace;
+ void *parm;
+ u8 idx;
+ u8 mask;
+};
+
+size_t syscall_arg__scnprintf_statx_flags(char *bf, size_t size, struct syscall_arg *arg);
+#define SCA_STATX_FLAGS syscall_arg__scnprintf_statx_flags
+
+size_t syscall_arg__scnprintf_statx_mask(char *bf, size_t size, struct syscall_arg *arg);
+#define SCA_STATX_MASK syscall_arg__scnprintf_statx_mask
+
+#endif /* _PERF_TRACE_BEAUTY_H */
diff --git a/tools/perf/trace/beauty/signum.c b/tools/perf/trace/beauty/signum.c
index d3b0b1f..fde8f2f 100644
--- a/tools/perf/trace/beauty/signum.c
+++ b/tools/perf/trace/beauty/signum.c
@@ -1,3 +1,4 @@
+#include <signal.h>
static size_t syscall_arg__scnprintf_signum(char *bf, size_t size, struct syscall_arg *arg)
{
diff --git a/tools/perf/trace/beauty/statx.c b/tools/perf/trace/beauty/statx.c
new file mode 100644
index 0000000..5643b69
--- /dev/null
+++ b/tools/perf/trace/beauty/statx.c
@@ -0,0 +1,72 @@
+/*
+ * trace/beauty/statx.c
+ *
+ * Copyright (C) 2017, Red Hat Inc, Arnaldo Carvalho de Melo <acme@redhat.com>
+ *
+ * Released under the GPL v2. (and only v2, not any later version)
+ */
+
+#include "trace/beauty/beauty.h"
+#include <linux/kernel.h>
+#include <sys/types.h>
+#include <uapi/linux/fcntl.h>
+#include <uapi/linux/stat.h>
+
+size_t syscall_arg__scnprintf_statx_flags(char *bf, size_t size, struct syscall_arg *arg)
+{
+ int printed = 0, flags = arg->val;
+
+ if (flags == 0)
+ return scnprintf(bf, size, "SYNC_AS_STAT");
+#define P_FLAG(n) \
+ if (flags & AT_##n) { \
+ printed += scnprintf(bf + printed, size - printed, "%s%s", printed ? "|" : "", #n); \
+ flags &= ~AT_##n; \
+ }
+
+ P_FLAG(SYMLINK_NOFOLLOW);
+ P_FLAG(REMOVEDIR);
+ P_FLAG(SYMLINK_FOLLOW);
+ P_FLAG(NO_AUTOMOUNT);
+ P_FLAG(EMPTY_PATH);
+ P_FLAG(STATX_FORCE_SYNC);
+ P_FLAG(STATX_DONT_SYNC);
+
+#undef P_FLAG
+
+ if (flags)
+ printed += scnprintf(bf + printed, size - printed, "%s%#x", printed ? "|" : "", flags);
+
+ return printed;
+}
+
+size_t syscall_arg__scnprintf_statx_mask(char *bf, size_t size, struct syscall_arg *arg)
+{
+ int printed = 0, flags = arg->val;
+
+#define P_FLAG(n) \
+ if (flags & STATX_##n) { \
+ printed += scnprintf(bf + printed, size - printed, "%s%s", printed ? "|" : "", #n); \
+ flags &= ~STATX_##n; \
+ }
+
+ P_FLAG(TYPE);
+ P_FLAG(MODE);
+ P_FLAG(NLINK);
+ P_FLAG(UID);
+ P_FLAG(GID);
+ P_FLAG(ATIME);
+ P_FLAG(MTIME);
+ P_FLAG(CTIME);
+ P_FLAG(INO);
+ P_FLAG(SIZE);
+ P_FLAG(BLOCKS);
+ P_FLAG(BTIME);
+
+#undef P_FLAG
+
+ if (flags)
+ printed += scnprintf(bf + printed, size - printed, "%s%#x", printed ? "|" : "", flags);
+
+ return printed;
+}
diff --git a/tools/perf/ui/browser.c b/tools/perf/ui/browser.c
index 3eb3edb..a4d3762 100644
--- a/tools/perf/ui/browser.c
+++ b/tools/perf/ui/browser.c
@@ -1,4 +1,5 @@
#include "../util.h"
+#include "../string2.h"
#include "../config.h"
#include "../../perf.h"
#include "libslang.h"
@@ -13,6 +14,7 @@
#include "helpline.h"
#include "keysyms.h"
#include "../color.h"
+#include "sane_ctype.h"
static int ui_browser__percent_color(struct ui_browser *browser,
double percent, bool current)
@@ -579,7 +581,7 @@
break;
*bg = '\0';
- while (isspace(*++bg));
+ bg = ltrim(++bg);
ui_browser__colorsets[i].bg = bg;
ui_browser__colorsets[i].fg = fg;
return 0;
diff --git a/tools/perf/ui/browsers/annotate.c b/tools/perf/ui/browsers/annotate.c
index ba36aac..d990ad0 100644
--- a/tools/perf/ui/browsers/annotate.c
+++ b/tools/perf/ui/browsers/annotate.c
@@ -9,7 +9,10 @@
#include "../../util/symbol.h"
#include "../../util/evsel.h"
#include "../../util/config.h"
+#include <inttypes.h>
#include <pthread.h>
+#include <linux/kernel.h>
+#include <sys/ttydefaults.h>
struct disasm_line_samples {
double percent;
diff --git a/tools/perf/ui/browsers/header.c b/tools/perf/ui/browsers/header.c
index edbeaaf..e2c9390 100644
--- a/tools/perf/ui/browsers/header.c
+++ b/tools/perf/ui/browsers/header.c
@@ -8,6 +8,8 @@
#include "util/header.h"
#include "util/session.h"
+#include <sys/ttydefaults.h>
+
static void ui_browser__argv_write(struct ui_browser *browser,
void *entry, int row)
{
diff --git a/tools/perf/ui/browsers/hists.c b/tools/perf/ui/browsers/hists.c
index fc4fb66..69f4570 100644
--- a/tools/perf/ui/browsers/hists.c
+++ b/tools/perf/ui/browsers/hists.c
@@ -1,7 +1,11 @@
+#include <dirent.h>
+#include <errno.h>
+#include <inttypes.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <linux/rbtree.h>
+#include <sys/ttydefaults.h>
#include "../../util/evsel.h"
#include "../../util/evlist.h"
@@ -10,6 +14,7 @@
#include "../../util/sort.h"
#include "../../util/util.h"
#include "../../util/top.h"
+#include "../../util/thread.h"
#include "../../arch/common.h"
#include "../browsers/hists.h"
@@ -18,6 +23,11 @@
#include "../ui.h"
#include "map.h"
#include "annotate.h"
+#include "srcline.h"
+#include "string2.h"
+#include "units.h"
+
+#include "sane_ctype.h"
extern void hist_browser__init_hpp(void);
@@ -144,9 +154,60 @@
cl->unfolded = unfold ? cl->has_children : false;
}
+static struct inline_node *inline_node__create(struct map *map, u64 ip)
+{
+ struct dso *dso;
+ struct inline_node *node;
+
+ if (map == NULL)
+ return NULL;
+
+ dso = map->dso;
+ if (dso == NULL)
+ return NULL;
+
+ if (dso->kernel != DSO_TYPE_USER)
+ return NULL;
+
+ node = dso__parse_addr_inlines(dso,
+ map__rip_2objdump(map, ip));
+
+ return node;
+}
+
+static int inline__count_rows(struct inline_node *node)
+{
+ struct inline_list *ilist;
+ int i = 0;
+
+ if (node == NULL)
+ return 0;
+
+ list_for_each_entry(ilist, &node->val, list) {
+ if ((ilist->filename != NULL) || (ilist->funcname != NULL))
+ i++;
+ }
+
+ return i;
+}
+
+static int callchain_list__inline_rows(struct callchain_list *chain)
+{
+ struct inline_node *node;
+ int rows;
+
+ node = inline_node__create(chain->ms.map, chain->ip);
+ if (node == NULL)
+ return 0;
+
+ rows = inline__count_rows(node);
+ inline_node__delete(node);
+ return rows;
+}
+
static int callchain_node__count_rows_rb_tree(struct callchain_node *node)
{
- int n = 0;
+ int n = 0, inline_rows;
struct rb_node *nd;
for (nd = rb_first(&node->rb_root); nd; nd = rb_next(nd)) {
@@ -156,6 +217,13 @@
list_for_each_entry(chain, &child->val, list) {
++n;
+
+ if (symbol_conf.inline_name) {
+ inline_rows =
+ callchain_list__inline_rows(chain);
+ n += inline_rows;
+ }
+
/* We need this because we may not have children */
folded_sign = callchain_list__folded(chain);
if (folded_sign == '+')
@@ -207,7 +275,7 @@
{
struct callchain_list *chain;
bool unfolded = false;
- int n = 0;
+ int n = 0, inline_rows;
if (callchain_param.mode == CHAIN_FLAT)
return callchain_node__count_flat_rows(node);
@@ -216,6 +284,11 @@
list_for_each_entry(chain, &node->val, list) {
++n;
+ if (symbol_conf.inline_name) {
+ inline_rows = callchain_list__inline_rows(chain);
+ n += inline_rows;
+ }
+
unfolded = chain->unfolded;
}
@@ -362,6 +435,19 @@
he->init_have_children = true;
}
+static void hist_entry_init_inline_node(struct hist_entry *he)
+{
+ if (he->inline_node)
+ return;
+
+ he->inline_node = inline_node__create(he->ms.map, he->ip);
+
+ if (he->inline_node == NULL)
+ return;
+
+ he->has_children = true;
+}
+
static bool hist_browser__toggle_fold(struct hist_browser *browser)
{
struct hist_entry *he = browser->he_selection;
@@ -393,7 +479,12 @@
if (he->unfolded) {
if (he->leaf)
- he->nr_rows = callchain__count_rows(&he->sorted_chain);
+ if (he->inline_node)
+ he->nr_rows = inline__count_rows(
+ he->inline_node);
+ else
+ he->nr_rows = callchain__count_rows(
+ &he->sorted_chain);
else
he->nr_rows = hierarchy_count_rows(browser, he, false);
@@ -753,6 +844,71 @@
#define LEVEL_OFFSET_STEP 3
+static int hist_browser__show_inline(struct hist_browser *browser,
+ struct inline_node *node,
+ unsigned short row,
+ int offset)
+{
+ struct inline_list *ilist;
+ char buf[1024];
+ int color, width, first_row;
+
+ first_row = row;
+ width = browser->b.width - (LEVEL_OFFSET_STEP + 2);
+ list_for_each_entry(ilist, &node->val, list) {
+ if ((ilist->filename != NULL) || (ilist->funcname != NULL)) {
+ color = HE_COLORSET_NORMAL;
+ if (ui_browser__is_current_entry(&browser->b, row))
+ color = HE_COLORSET_SELECTED;
+
+ if (callchain_param.key == CCKEY_ADDRESS ||
+ callchain_param.key == CCKEY_SRCLINE) {
+ if (ilist->filename != NULL)
+ scnprintf(buf, sizeof(buf),
+ "%s:%d (inline)",
+ ilist->filename,
+ ilist->line_nr);
+ else
+ scnprintf(buf, sizeof(buf), "??");
+ } else if (ilist->funcname != NULL)
+ scnprintf(buf, sizeof(buf), "%s (inline)",
+ ilist->funcname);
+ else if (ilist->filename != NULL)
+ scnprintf(buf, sizeof(buf),
+ "%s:%d (inline)",
+ ilist->filename,
+ ilist->line_nr);
+ else
+ scnprintf(buf, sizeof(buf), "??");
+
+ ui_browser__set_color(&browser->b, color);
+ hist_browser__gotorc(browser, row, 0);
+ ui_browser__write_nstring(&browser->b, " ",
+ LEVEL_OFFSET_STEP + offset);
+ ui_browser__write_nstring(&browser->b, buf, width);
+ row++;
+ }
+ }
+
+ return row - first_row;
+}
+
+static size_t show_inline_list(struct hist_browser *browser, struct map *map,
+ u64 ip, int row, int offset)
+{
+ struct inline_node *node;
+ int ret;
+
+ node = inline_node__create(map, ip);
+ if (node == NULL)
+ return 0;
+
+ ret = hist_browser__show_inline(browser, node, row, offset);
+
+ inline_node__delete(node);
+ return ret;
+}
+
static int hist_browser__show_callchain_list(struct hist_browser *browser,
struct callchain_node *node,
struct callchain_list *chain,
@@ -764,6 +920,7 @@
char bf[1024], *alloc_str;
char buf[64], *alloc_str2;
const char *str;
+ int inline_rows = 0, ret = 1;
if (arg->row_offset != 0) {
arg->row_offset--;
@@ -801,10 +958,15 @@
}
print(browser, chain, str, offset, row, arg);
-
free(alloc_str);
free(alloc_str2);
- return 1;
+
+ if (symbol_conf.inline_name) {
+ inline_rows = show_inline_list(browser, chain->ms.map,
+ chain->ip, row + 1, offset);
+ }
+
+ return ret + inline_rows;
}
static bool check_percent_display(struct rb_node *node, u64 parent_total)
@@ -1228,6 +1390,12 @@
folded_sign = hist_entry__folded(entry);
}
+ if (symbol_conf.inline_name &&
+ (!entry->has_children)) {
+ hist_entry_init_inline_node(entry);
+ folded_sign = hist_entry__folded(entry);
+ }
+
if (row_offset == 0) {
struct hpp_arg arg = {
.b = &browser->b,
@@ -1259,7 +1427,8 @@
}
if (first) {
- if (symbol_conf.use_callchain) {
+ if (symbol_conf.use_callchain ||
+ symbol_conf.inline_name) {
ui_browser__printf(&browser->b, "%c ", folded_sign);
width -= 2;
}
@@ -1301,8 +1470,14 @@
.is_current_entry = current_entry,
};
- printed += hist_browser__show_callchain(browser, entry, 1, row,
- hist_browser__show_callchain_entry, &arg,
+ if (entry->inline_node)
+ printed += hist_browser__show_inline(browser,
+ entry->inline_node, row, 0);
+ else
+ printed += hist_browser__show_callchain(browser,
+ entry, 1, row,
+ hist_browser__show_callchain_entry,
+ &arg,
hist_browser__check_output_full);
}
@@ -2308,7 +2483,7 @@
return ret;
memset(options, 0, sizeof(options));
- memset(options, 0, sizeof(abs_path));
+ memset(abs_path, 0, sizeof(abs_path));
while ((dent = readdir(pwd_dir))) {
char path[PATH_MAX];
diff --git a/tools/perf/ui/browsers/map.c b/tools/perf/ui/browsers/map.c
index 9ce142d..ffa5add 100644
--- a/tools/perf/ui/browsers/map.c
+++ b/tools/perf/ui/browsers/map.c
@@ -11,6 +11,8 @@
#include "../keysyms.h"
#include "map.h"
+#include "sane_ctype.h"
+
struct map_browser {
struct ui_browser b;
struct map *map;
diff --git a/tools/perf/ui/gtk/annotate.c b/tools/perf/ui/gtk/annotate.c
index 8c9308a..e99ba86 100644
--- a/tools/perf/ui/gtk/annotate.c
+++ b/tools/perf/ui/gtk/annotate.c
@@ -3,7 +3,8 @@
#include "util/annotate.h"
#include "util/evsel.h"
#include "ui/helpline.h"
-
+#include <inttypes.h>
+#include <signal.h>
enum {
ANN_COL__PERCENT,
diff --git a/tools/perf/ui/gtk/hists.c b/tools/perf/ui/gtk/hists.c
index a4f02de..e24f839 100644
--- a/tools/perf/ui/gtk/hists.c
+++ b/tools/perf/ui/gtk/hists.c
@@ -4,7 +4,9 @@
#include "../sort.h"
#include "../hist.h"
#include "../helpline.h"
+#include "../string2.h"
#include "gtk.h"
+#include <signal.h>
#define MAX_COLUMNS 32
diff --git a/tools/perf/ui/hist.c b/tools/perf/ui/hist.c
index 5d632dc..59addd5 100644
--- a/tools/perf/ui/hist.c
+++ b/tools/perf/ui/hist.c
@@ -1,3 +1,4 @@
+#include <inttypes.h>
#include <math.h>
#include <linux/compiler.h>
diff --git a/tools/perf/ui/setup.c b/tools/perf/ui/setup.c
index 50d13e58..5ea0b40 100644
--- a/tools/perf/ui/setup.c
+++ b/tools/perf/ui/setup.c
@@ -4,6 +4,7 @@
#include "../util/cache.h"
#include "../util/debug.h"
#include "../util/hist.h"
+#include "../util/util.h"
pthread_mutex_t ui__lock = PTHREAD_MUTEX_INITIALIZER;
void *perf_gtk_handle;
diff --git a/tools/perf/ui/stdio/hist.c b/tools/perf/ui/stdio/hist.c
index 668f4ae..42e432b 100644
--- a/tools/perf/ui/stdio/hist.c
+++ b/tools/perf/ui/stdio/hist.c
@@ -4,7 +4,10 @@
#include "../../util/hist.h"
#include "../../util/sort.h"
#include "../../util/evsel.h"
-
+#include "../../util/srcline.h"
+#include "../../util/string2.h"
+#include "../../util/thread.h"
+#include "../../util/sane_ctype.h"
static size_t callchain__fprintf_left_margin(FILE *fp, int left_margin)
{
@@ -17,6 +20,67 @@
return ret;
}
+static size_t inline__fprintf(struct map *map, u64 ip, int left_margin,
+ int depth, int depth_mask, FILE *fp)
+{
+ struct dso *dso;
+ struct inline_node *node;
+ struct inline_list *ilist;
+ int ret = 0, i;
+
+ if (map == NULL)
+ return 0;
+
+ dso = map->dso;
+ if (dso == NULL)
+ return 0;
+
+ if (dso->kernel != DSO_TYPE_USER)
+ return 0;
+
+ node = dso__parse_addr_inlines(dso,
+ map__rip_2objdump(map, ip));
+ if (node == NULL)
+ return 0;
+
+ list_for_each_entry(ilist, &node->val, list) {
+ if ((ilist->filename != NULL) || (ilist->funcname != NULL)) {
+ ret += callchain__fprintf_left_margin(fp, left_margin);
+
+ for (i = 0; i < depth; i++) {
+ if (depth_mask & (1 << i))
+ ret += fprintf(fp, "|");
+ else
+ ret += fprintf(fp, " ");
+ ret += fprintf(fp, " ");
+ }
+
+ if (callchain_param.key == CCKEY_ADDRESS ||
+ callchain_param.key == CCKEY_SRCLINE) {
+ if (ilist->filename != NULL)
+ ret += fprintf(fp, "%s:%d (inline)",
+ ilist->filename,
+ ilist->line_nr);
+ else
+ ret += fprintf(fp, "??");
+ } else if (ilist->funcname != NULL)
+ ret += fprintf(fp, "%s (inline)",
+ ilist->funcname);
+ else if (ilist->filename != NULL)
+ ret += fprintf(fp, "%s:%d (inline)",
+ ilist->filename,
+ ilist->line_nr);
+ else
+ ret += fprintf(fp, "??");
+
+ ret += fprintf(fp, "\n");
+ }
+ }
+
+ inline_node__delete(node);
+ return ret;
+}
+
static size_t ipchain__fprintf_graph_line(FILE *fp, int depth, int depth_mask,
int left_margin)
{
@@ -78,6 +142,10 @@
fputs(str, fp);
fputc('\n', fp);
free(alloc_str);
+
+ if (symbol_conf.inline_name)
+ ret += inline__fprintf(chain->ms.map, chain->ip,
+ left_margin, depth, depth_mask, fp);
return ret;
}
@@ -229,6 +297,7 @@
if (!i++ && field_order == NULL &&
sort_order && !prefixcmp(sort_order, "sym"))
continue;
+
if (!printed) {
ret += callchain__fprintf_left_margin(fp, left_margin);
ret += fprintf(fp, "|\n");
@@ -251,6 +320,13 @@
if (++entries_printed == callchain_param.print_limit)
break;
+
+ if (symbol_conf.inline_name)
+ ret += inline__fprintf(chain->ms.map,
+ chain->ip,
+ left_margin,
+ 0, 0,
+ fp);
}
root = &cnode->rb_root;
}
@@ -529,6 +605,8 @@
bool use_callchain)
{
int ret;
+ int callchain_ret = 0;
+ int inline_ret = 0;
struct perf_hpp hpp = {
.buf = bf,
.size = size,
@@ -547,7 +625,16 @@
ret = fprintf(fp, "%s\n", bf);
if (use_callchain)
- ret += hist_entry_callchain__fprintf(he, total_period, 0, fp);
+ callchain_ret = hist_entry_callchain__fprintf(he, total_period,
+ 0, fp);
+
+ if (callchain_ret == 0 && symbol_conf.inline_name) {
+ inline_ret = inline__fprintf(he->ms.map, he->ip, 0, 0, 0, fp);
+ ret += inline_ret;
+ if (inline_ret > 0)
+ ret += fprintf(fp, "\n");
+ } else
+ ret += callchain_ret;
return ret;
}
diff --git a/tools/perf/ui/tui/setup.c b/tools/perf/ui/tui/setup.c
index 4ea2ba8..d9350a1 100644
--- a/tools/perf/ui/tui/setup.c
+++ b/tools/perf/ui/tui/setup.c
@@ -1,6 +1,7 @@
#include <errno.h>
#include <signal.h>
#include <stdbool.h>
+#include <linux/kernel.h>
#ifdef HAVE_BACKTRACE_SUPPORT
#include <execinfo.h>
#endif
diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index 5da376b..069583b 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -1,4 +1,3 @@
-libperf-y += alias.o
libperf-y += annotate.o
libperf-y += block-range.o
libperf-y += build-id.o
@@ -17,6 +16,7 @@
libperf-y += parse-events.o
libperf-y += perf_regs.o
libperf-y += path.o
+libperf-y += print_binary.o
libperf-y += rbtree.o
libperf-y += libstring.o
libperf-y += bitmap.o
@@ -42,6 +42,7 @@
libperf-y += session.o
libperf-$(CONFIG_AUDIT) += syscalltbl.o
libperf-y += ordered-events.o
+libperf-y += namespaces.o
libperf-y += comm.o
libperf-y += thread.o
libperf-y += thread_map.o
@@ -81,13 +82,16 @@
libperf-$(CONFIG_AUXTRACE) += intel-pt.o
libperf-$(CONFIG_AUXTRACE) += intel-bts.o
libperf-y += parse-branch-options.o
+libperf-y += dump-insn.o
libperf-y += parse-regs-options.o
libperf-y += term.o
libperf-y += help-unknown-cmd.o
libperf-y += mem-events.o
libperf-y += vsprintf.o
libperf-y += drv_configs.o
+libperf-y += units.o
libperf-y += time-utils.o
+libperf-y += expr-bison.o
libperf-$(CONFIG_LIBBPF) += bpf-loader.o
libperf-$(CONFIG_BPF_PROLOGUE) += bpf-prologue.o
@@ -140,6 +144,10 @@
$(call rule_mkdir)
$(Q)$(call echo-cmd,bison)$(BISON) -v util/parse-events.y -d $(PARSER_DEBUG_BISON) -o $@ -p parse_events_
+$(OUTPUT)util/expr-bison.c: util/expr.y
+ $(call rule_mkdir)
+ $(Q)$(call echo-cmd,bison)$(BISON) -v util/expr.y -d $(PARSER_DEBUG_BISON) -o $@ -p expr__
+
$(OUTPUT)util/pmu-flex.c: util/pmu.l $(OUTPUT)util/pmu-bison.c
$(call rule_mkdir)
$(Q)$(call echo-cmd,flex)$(FLEX) -o $@ --header-file=$(OUTPUT)util/pmu-flex.h util/pmu.l
@@ -152,6 +160,7 @@
CFLAGS_pmu-flex.o += -w
CFLAGS_parse-events-bison.o += -DYYENABLE_NLS=0 -w
CFLAGS_pmu-bison.o += -DYYENABLE_NLS=0 -DYYLTYPE_IS_TRIVIAL=0 -w
+CFLAGS_expr-bison.o += -DYYENABLE_NLS=0 -DYYLTYPE_IS_TRIVIAL=0 -w
$(OUTPUT)util/parse-events.o: $(OUTPUT)util/parse-events-flex.c $(OUTPUT)util/parse-events-bison.c
$(OUTPUT)util/pmu.o: $(OUTPUT)util/pmu-flex.c $(OUTPUT)util/pmu-bison.c
diff --git a/tools/perf/util/alias.c b/tools/perf/util/alias.c
deleted file mode 100644
index 6455471..0000000
--- a/tools/perf/util/alias.c
+++ /dev/null
@@ -1,78 +0,0 @@
-#include "cache.h"
-#include "util.h"
-#include "config.h"
-
-static const char *alias_key;
-static char *alias_val;
-
-static int alias_lookup_cb(const char *k, const char *v,
- void *cb __maybe_unused)
-{
- if (!prefixcmp(k, "alias.") && !strcmp(k+6, alias_key)) {
- if (!v)
- return config_error_nonbool(k);
- alias_val = strdup(v);
- return 0;
- }
- return 0;
-}
-
-char *alias_lookup(const char *alias)
-{
- alias_key = alias;
- alias_val = NULL;
- perf_config(alias_lookup_cb, NULL);
- return alias_val;
-}
-
-int split_cmdline(char *cmdline, const char ***argv)
-{
- int src, dst, count = 0, size = 16;
- char quoted = 0;
-
- *argv = malloc(sizeof(char*) * size);
-
- /* split alias_string */
- (*argv)[count++] = cmdline;
- for (src = dst = 0; cmdline[src];) {
- char c = cmdline[src];
- if (!quoted && isspace(c)) {
- cmdline[dst++] = 0;
- while (cmdline[++src]
- && isspace(cmdline[src]))
- ; /* skip */
- if (count >= size) {
- size += 16;
- *argv = realloc(*argv, sizeof(char*) * size);
- }
- (*argv)[count++] = cmdline + dst;
- } else if (!quoted && (c == '\'' || c == '"')) {
- quoted = c;
- src++;
- } else if (c == quoted) {
- quoted = 0;
- src++;
- } else {
- if (c == '\\' && quoted != '\'') {
- src++;
- c = cmdline[src];
- if (!c) {
- zfree(argv);
- return error("cmdline ends with \\");
- }
- }
- cmdline[dst++] = c;
- src++;
- }
- }
-
- cmdline[dst] = 0;
-
- if (quoted) {
- zfree(argv);
- return error("unclosed quote");
- }
-
- return count;
-}
-
diff --git a/tools/perf/util/annotate.c b/tools/perf/util/annotate.c
index 7aa5722..683f834 100644
--- a/tools/perf/util/annotate.c
+++ b/tools/perf/util/annotate.c
@@ -7,6 +7,8 @@
* Released under the GPL v2. (and only v2, not any later version)
*/
+#include <errno.h>
+#include <inttypes.h>
#include "util.h"
#include "ui/ui.h"
#include "sort.h"
@@ -18,12 +20,16 @@
#include "annotate.h"
#include "evsel.h"
#include "block-range.h"
+#include "string2.h"
#include "arch/common.h"
#include <regex.h>
#include <pthread.h>
#include <linux/bitops.h>
+#include <linux/kernel.h>
#include <sys/utsname.h>
+#include "sane_ctype.h"
+
const char *disassembler_style;
const char *objdump_path;
static regex_t file_lineno;
@@ -108,6 +114,7 @@
#include "arch/arm64/annotate/instructions.c"
#include "arch/x86/annotate/instructions.c"
#include "arch/powerpc/annotate/instructions.c"
+#include "arch/s390/annotate/instructions.c"
static struct arch architectures[] = {
{
@@ -132,6 +139,7 @@
},
{
.name = "s390",
+ .init = s390__annotate_init,
.objdump = {
.comment_char = '#',
},
@@ -385,9 +393,7 @@
if (comment == NULL)
return 0;
- while (comment[0] != '\0' && isspace(comment[0]))
- ++comment;
-
+ comment = ltrim(comment);
comment__symbol(ops->source.raw, comment, &ops->source.addr, &ops->source.name);
comment__symbol(ops->target.raw, comment, &ops->target.addr, &ops->target.name);
@@ -432,9 +438,7 @@
if (comment == NULL)
return 0;
- while (comment[0] != '\0' && isspace(comment[0]))
- ++comment;
-
+ comment = ltrim(comment);
comment__symbol(ops->target.raw, comment, &ops->target.addr, &ops->target.name);
return 0;
@@ -783,10 +787,7 @@
static int disasm_line__parse(char *line, const char **namep, char **rawp)
{
- char *name = line, tmp;
-
- while (isspace(name[0]))
- ++name;
+ char tmp, *name = ltrim(line);
if (name[0] == '\0')
return -1;
@@ -804,12 +805,7 @@
goto out_free_name;
(*rawp)[0] = tmp;
-
- if ((*rawp)[0] != '\0') {
- (*rawp)++;
- while (isspace((*rawp)[0]))
- ++(*rawp);
- }
+ *rawp = ltrim(*rawp);
return 0;
@@ -1154,7 +1150,7 @@
{
struct annotation *notes = symbol__annotation(sym);
struct disasm_line *dl;
- char *line = NULL, *parsed_line, *tmp, *tmp2, *c;
+ char *line = NULL, *parsed_line, *tmp, *tmp2;
size_t line_len;
s64 line_ip, offset = -1;
regmatch_t match[2];
@@ -1165,32 +1161,16 @@
if (!line)
return -1;
- while (line_len != 0 && isspace(line[line_len - 1]))
- line[--line_len] = '\0';
-
- c = strchr(line, '\n');
- if (c)
- *c = 0;
-
line_ip = -1;
- parsed_line = line;
+ parsed_line = rtrim(line);
/* /filename:linenr ? Save line number and ignore. */
- if (regexec(&file_lineno, line, 2, match, 0) == 0) {
- *line_nr = atoi(line + match[1].rm_so);
+ if (regexec(&file_lineno, parsed_line, 2, match, 0) == 0) {
+ *line_nr = atoi(parsed_line + match[1].rm_so);
return 0;
}
- /*
- * Strip leading spaces:
- */
- tmp = line;
- while (*tmp) {
- if (*tmp != ' ')
- break;
- tmp++;
- }
-
+ tmp = ltrim(parsed_line);
if (*tmp) {
/*
* Parse hexa addresses followed by ':'
@@ -1313,6 +1293,7 @@
{
char linkname[PATH_MAX];
char *build_id_filename;
+ char *build_id_path = NULL;
if (dso->symtab_type == DSO_BINARY_TYPE__KALLSYMS &&
!dso__is_kcore(dso))
@@ -1328,8 +1309,14 @@
goto fallback;
}
+ build_id_path = strdup(filename);
+ if (!build_id_path)
+ return -1;
+
+ dirname(build_id_path);
+
if (dso__is_kcore(dso) ||
- readlink(filename, linkname, sizeof(linkname)) < 0 ||
+ readlink(build_id_path, linkname, sizeof(linkname)) < 0 ||
strstr(linkname, DSO__NAME_KALLSYMS) ||
access(filename, R_OK)) {
fallback:
@@ -1341,6 +1328,7 @@
__symbol__join_symfs(filename, filename_size, dso->long_name);
}
+ free(build_id_path);
return 0;
}
@@ -1441,7 +1429,7 @@
snprintf(command, sizeof(command),
"%s %s%s --start-address=0x%016" PRIx64
" --stop-address=0x%016" PRIx64
- " -l -d %s %s -C %s 2>/dev/null|grep -v %s|expand",
+ " -l -d %s %s -C %s 2>/dev/null|grep -v %s:|expand",
objdump_path ? objdump_path : "objdump",
disassembler_style ? "-M " : "",
disassembler_style ? disassembler_style : "",
@@ -1488,6 +1476,12 @@
nline = 0;
while (!feof(file)) {
+ /*
+ * The source code line number (lineno) needs to be kept in
+ * accross calls to symbol__parse_objdump_line(), so that it
+ * can associate it with the instructions till the next one.
+ * See disasm_line__new() and struct disasm_line::line_nr.
+ */
if (symbol__parse_objdump_line(sym, map, arch, file, privsize,
&lineno) < 0)
break;
@@ -1657,24 +1651,31 @@
start = map__rip_2objdump(map, sym->start);
for (i = 0; i < len; i++) {
- u64 offset;
+ u64 offset, nr_samples;
double percent_max = 0.0;
src_line->nr_pcnt = nr_pcnt;
for (k = 0; k < nr_pcnt; k++) {
- h = annotation__histogram(notes, evidx + k);
- src_line->samples[k].percent = 100.0 * h->addr[i] / h->sum;
+ double percent = 0.0;
- if (src_line->samples[k].percent > percent_max)
- percent_max = src_line->samples[k].percent;
+ h = annotation__histogram(notes, evidx + k);
+ nr_samples = h->addr[i];
+ if (h->sum)
+ percent = 100.0 * nr_samples / h->sum;
+
+ if (percent > percent_max)
+ percent_max = percent;
+ src_line->samples[k].percent = percent;
+ src_line->samples[k].nr = nr_samples;
}
if (percent_max <= 0.5)
goto next;
offset = start + i;
- src_line->path = get_srcline(map->dso, offset, NULL, false);
+ src_line->path = get_srcline(map->dso, offset, NULL,
+ false, true);
insert_source_line(&tmp_root, src_line);
next:
diff --git a/tools/perf/util/annotate.h b/tools/perf/util/annotate.h
index 09776b5..948aa8e 100644
--- a/tools/perf/util/annotate.h
+++ b/tools/perf/util/annotate.h
@@ -98,7 +98,7 @@
struct source_line_samples {
double percent;
double percent_sum;
- double nr;
+ u64 nr;
};
struct source_line {
diff --git a/tools/perf/util/auxtrace.c b/tools/perf/util/auxtrace.c
index c5a6e0b1..0daf63b 100644
--- a/tools/perf/util/auxtrace.c
+++ b/tools/perf/util/auxtrace.c
@@ -13,10 +13,10 @@
*
*/
+#include <inttypes.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <stdbool.h>
-#include <ctype.h>
#include <string.h>
#include <limits.h>
#include <errno.h>
@@ -46,7 +46,6 @@
#include "cpumap.h"
#include "thread_map.h"
#include "asm/bug.h"
-#include "symbol/kallsyms.h"
#include "auxtrace.h"
#include <linux/hash.h>
@@ -59,6 +58,9 @@
#include "intel-pt.h"
#include "intel-bts.h"
+#include "sane_ctype.h"
+#include "symbol/kallsyms.h"
+
int auxtrace_mmap__mmap(struct auxtrace_mmap *mm,
struct auxtrace_mmap_params *mp,
void *userpg, int fd)
@@ -1826,7 +1828,7 @@
filt->addr = start;
if (filt->range && !filt->size && !filt->sym_to) {
filt->size = size;
- no_size = !!size;
+ no_size = !size;
}
}
@@ -1840,7 +1842,7 @@
if (err)
return err;
filt->size = start + size - filt->addr;
- no_size = !!size;
+ no_size = !size;
}
/* The very last symbol in kallsyms does not imply a particular size */
diff --git a/tools/perf/util/auxtrace.h b/tools/perf/util/auxtrace.h
index 26fb1ee..9f0de72 100644
--- a/tools/perf/util/auxtrace.h
+++ b/tools/perf/util/auxtrace.h
@@ -17,6 +17,7 @@
#define __PERF_AUXTRACE_H
#include <sys/types.h>
+#include <errno.h>
#include <stdbool.h>
#include <stddef.h>
#include <linux/list.h>
diff --git a/tools/perf/util/bpf-loader.c b/tools/perf/util/bpf-loader.c
index bc6bc70..4bd2d1d 100644
--- a/tools/perf/util/bpf-loader.c
+++ b/tools/perf/util/bpf-loader.c
@@ -9,7 +9,9 @@
#include <bpf/libbpf.h>
#include <bpf/bpf.h>
#include <linux/err.h>
+#include <linux/kernel.h>
#include <linux/string.h>
+#include <errno.h>
#include "perf.h"
#include "debug.h"
#include "bpf-loader.h"
@@ -17,6 +19,7 @@
#include "probe-event.h"
#include "probe-finder.h" // for MAX_PROBES
#include "parse-events.h"
+#include "strfilter.h"
#include "llvm-utils.h"
#include "c++/clang-c.h"
diff --git a/tools/perf/util/bpf-loader.h b/tools/perf/util/bpf-loader.h
index f2b737b..4886386 100644
--- a/tools/perf/util/bpf-loader.h
+++ b/tools/perf/util/bpf-loader.h
@@ -85,6 +85,8 @@
char *buf, size_t size);
#else
+#include <errno.h>
+
static inline struct bpf_object *
bpf__prepare_load(const char *filename __maybe_unused,
bool source __maybe_unused)
diff --git a/tools/perf/util/bpf-prologue.c b/tools/perf/util/bpf-prologue.c
index 6cdbee1..1356220 100644
--- a/tools/perf/util/bpf-prologue.c
+++ b/tools/perf/util/bpf-prologue.c
@@ -12,6 +12,7 @@
#include "bpf-loader.h"
#include "bpf-prologue.h"
#include "probe-finder.h"
+#include <errno.h>
#include <dwarf-regs.h>
#include <linux/filter.h>
diff --git a/tools/perf/util/bpf-prologue.h b/tools/perf/util/bpf-prologue.h
index d94cbea..ba56483 100644
--- a/tools/perf/util/bpf-prologue.h
+++ b/tools/perf/util/bpf-prologue.h
@@ -18,6 +18,8 @@
struct bpf_insn *new_prog, size_t *new_cnt,
size_t cnt_space);
#else
+#include <errno.h>
+
static inline int
bpf__gen_prologue(struct probe_trace_arg *args __maybe_unused,
int nargs __maybe_unused,
diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index e528c40..168cc49 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -7,18 +7,26 @@
* Copyright (C) 2009, 2010 Arnaldo Carvalho de Melo <acme@redhat.com>
*/
#include "util.h"
+#include <dirent.h>
+#include <errno.h>
#include <stdio.h>
+#include <sys/stat.h>
+#include <sys/types.h>
#include "build-id.h"
#include "event.h"
#include "symbol.h"
+#include "thread.h"
#include <linux/kernel.h>
#include "debug.h"
#include "session.h"
#include "tool.h"
#include "header.h"
#include "vdso.h"
+#include "path.h"
#include "probe-file.h"
+#include "strlist.h"
+#include "sane_ctype.h"
static bool no_buildid_cache;
@@ -182,13 +190,17 @@
char buf[PATH_MAX];
char *ret = NULL, *p;
size_t offs = 5; /* == strlen("../..") */
+ ssize_t len;
linkname = build_id_cache__linkname(sbuild_id, NULL, 0);
if (!linkname)
return NULL;
- if (readlink(linkname, buf, PATH_MAX) < 0)
+ len = readlink(linkname, buf, sizeof(buf) - 1);
+ if (len <= 0)
goto out;
+ buf[len] = '\0';
+
/* The link should be "../..<origpath>/<sbuild_id>" */
p = strrchr(buf, '/'); /* Cut off the "/<sbuild_id>" */
if (p && (p > buf + offs)) {
@@ -443,14 +455,14 @@
}
static bool lsdir_bid_head_filter(const char *name __maybe_unused,
- struct dirent *d __maybe_unused)
+ struct dirent *d)
{
return (strlen(d->d_name) == 2) &&
isxdigit(d->d_name[0]) && isxdigit(d->d_name[1]);
}
static bool lsdir_bid_tail_filter(const char *name __maybe_unused,
- struct dirent *d __maybe_unused)
+ struct dirent *d)
{
int i = 0;
while (isxdigit(d->d_name[i]) && i < SBUILD_ID_SIZE - 3)
@@ -690,7 +702,7 @@
err = 0;
/* Update SDT cache : error is just warned */
- if (build_id_cache__add_sdt_cache(sbuild_id, realname) < 0)
+ if (realname && build_id_cache__add_sdt_cache(sbuild_id, realname) < 0)
pr_debug4("Failed to update/scan SDT cache for %s\n", realname);
out_free:
diff --git a/tools/perf/util/build-id.h b/tools/perf/util/build-id.h
index d279906..a960811 100644
--- a/tools/perf/util/build-id.h
+++ b/tools/perf/util/build-id.h
@@ -5,7 +5,6 @@
#define SBUILD_ID_SIZE (BUILD_ID_SIZE * 2 + 1)
#include "tool.h"
-#include "strlist.h"
#include <linux/types.h>
extern struct perf_tool build_id__mark_dso_hit_ops;
@@ -34,6 +33,9 @@
char *build_id_cache__linkname(const char *sbuild_id, char *bf, size_t size);
char *build_id_cache__cachedir(const char *sbuild_id, const char *name,
bool is_kallsyms, bool is_vdso);
+
+struct strlist;
+
struct strlist *build_id_cache__list_all(bool validonly);
char *build_id_cache__complement(const char *incomplete_sbuild_id);
int build_id_cache__list_build_ids(const char *pathname,
diff --git a/tools/perf/util/c++/clang-c.h b/tools/perf/util/c++/clang-c.h
index 0eadd79..ccafcf7 100644
--- a/tools/perf/util/c++/clang-c.h
+++ b/tools/perf/util/c++/clang-c.h
@@ -20,6 +20,7 @@
size_t *p_obj_buf_sz);
#else
+#include <errno.h>
static inline void perf_clang__init(void) { }
static inline void perf_clang__cleanup(void) { }
diff --git a/tools/perf/util/cache.h b/tools/perf/util/cache.h
index 512c0c8..0328f29 100644
--- a/tools/perf/util/cache.h
+++ b/tools/perf/util/cache.h
@@ -15,7 +15,6 @@
#define PERF_TRACEFS_ENVIRONMENT "PERF_TRACEFS_DIR"
#define PERF_PAGER_ENVIRONMENT "PERF_PAGER"
-char *alias_lookup(const char *alias);
int split_cmdline(char *cmdline, const char ***argv);
#define alloc_nr(x) (((x)+16)*3/2)
diff --git a/tools/perf/util/callchain.c b/tools/perf/util/callchain.c
index aba9534..81fc29a 100644
--- a/tools/perf/util/callchain.c
+++ b/tools/perf/util/callchain.c
@@ -9,6 +9,7 @@
*
*/
+#include <inttypes.h>
#include <stdlib.h>
#include <stdio.h>
#include <stdbool.h>
@@ -23,6 +24,21 @@
#include "machine.h"
#include "callchain.h"
+#define CALLCHAIN_PARAM_DEFAULT \
+ .mode = CHAIN_GRAPH_ABS, \
+ .min_percent = 0.5, \
+ .order = ORDER_CALLEE, \
+ .key = CCKEY_FUNCTION, \
+ .value = CCVAL_PERCENT, \
+
+struct callchain_param callchain_param = {
+ CALLCHAIN_PARAM_DEFAULT
+};
+
+struct callchain_param callchain_param_default = {
+ CALLCHAIN_PARAM_DEFAULT
+};
+
__thread struct callchain_cursor callchain_cursor;
int parse_callchain_record_opt(const char *arg, struct callchain_param *param)
@@ -80,6 +96,10 @@
callchain_param.key = CCKEY_ADDRESS;
return 0;
}
+ if (!strncmp(value, "srcline", strlen(value))) {
+ callchain_param.key = CCKEY_SRCLINE;
+ return 0;
+ }
if (!strncmp(value, "branch", strlen(value))) {
callchain_param.branch_callstack = 1;
return 0;
@@ -108,11 +128,37 @@
return -1;
}
+static int get_stack_size(const char *str, unsigned long *_size)
+{
+ char *endptr;
+ unsigned long size;
+ unsigned long max_size = round_down(USHRT_MAX, sizeof(u64));
+
+ size = strtoul(str, &endptr, 0);
+
+ do {
+ if (*endptr)
+ break;
+
+ size = round_up(size, sizeof(u64));
+ if (!size || size > max_size)
+ break;
+
+ *_size = size;
+ return 0;
+
+ } while (0);
+
+ pr_err("callchain: Incorrect stack dump size (max %ld): %s\n",
+ max_size, str);
+ return -1;
+}
+
static int
__parse_callchain_report_opt(const char *arg, bool allow_record_opt)
{
char *tok;
- char *endptr;
+ char *endptr, *saveptr = NULL;
bool minpcnt_set = false;
bool record_opt_set = false;
bool try_stack_size = false;
@@ -123,7 +169,7 @@
if (!arg)
return 0;
- while ((tok = strtok((char *)arg, ",")) != NULL) {
+ while ((tok = strtok_r((char *)arg, ",", &saveptr)) != NULL) {
if (!strncmp(tok, "none", strlen(tok))) {
callchain_param.mode = CHAIN_NONE;
callchain_param.enabled = false;
@@ -191,6 +237,68 @@
return __parse_callchain_report_opt(arg, true);
}
+int parse_callchain_record(const char *arg, struct callchain_param *param)
+{
+ char *tok, *name, *saveptr = NULL;
+ char *buf;
+ int ret = -1;
+
+ /* We need buffer that we know we can write to. */
+ buf = malloc(strlen(arg) + 1);
+ if (!buf)
+ return -ENOMEM;
+
+ strcpy(buf, arg);
+
+ tok = strtok_r((char *)buf, ",", &saveptr);
+ name = tok ? : (char *)buf;
+
+ do {
+ /* Framepointer style */
+ if (!strncmp(name, "fp", sizeof("fp"))) {
+ if (!strtok_r(NULL, ",", &saveptr)) {
+ param->record_mode = CALLCHAIN_FP;
+ ret = 0;
+ } else
+ pr_err("callchain: No more arguments "
+ "needed for --call-graph fp\n");
+ break;
+
+ /* Dwarf style */
+ } else if (!strncmp(name, "dwarf", sizeof("dwarf"))) {
+ const unsigned long default_stack_dump_size = 8192;
+
+ ret = 0;
+ param->record_mode = CALLCHAIN_DWARF;
+ param->dump_size = default_stack_dump_size;
+
+ tok = strtok_r(NULL, ",", &saveptr);
+ if (tok) {
+ unsigned long size = 0;
+
+ ret = get_stack_size(tok, &size);
+ param->dump_size = size;
+ }
+ } else if (!strncmp(name, "lbr", sizeof("lbr"))) {
+ if (!strtok_r(NULL, ",", &saveptr)) {
+ param->record_mode = CALLCHAIN_LBR;
+ ret = 0;
+ } else
+ pr_err("callchain: No more arguments "
+ "needed for --call-graph lbr\n");
+ break;
+ } else {
+ pr_err("callchain: Unknown --call-graph option "
+ "value: %s\n", arg);
+ break;
+ }
+
+ } while (0);
+
+ free(buf);
+ return ret;
+}
+
int perf_callchain_config(const char *var, const char *value)
{
char *endptr;
@@ -510,14 +618,51 @@
MATCH_GT,
};
+static enum match_result match_chain_srcline(struct callchain_cursor_node *node,
+ struct callchain_list *cnode)
+{
+ char *left = get_srcline(cnode->ms.map->dso,
+ map__rip_2objdump(cnode->ms.map, cnode->ip),
+ cnode->ms.sym, true, false);
+ char *right = get_srcline(node->map->dso,
+ map__rip_2objdump(node->map, node->ip),
+ node->sym, true, false);
+ enum match_result ret = MATCH_EQ;
+ int cmp;
+
+ if (left && right)
+ cmp = strcmp(left, right);
+ else if (!left && right)
+ cmp = 1;
+ else if (left && !right)
+ cmp = -1;
+ else if (cnode->ip == node->ip)
+ cmp = 0;
+ else
+ cmp = (cnode->ip < node->ip) ? -1 : 1;
+
+ if (cmp != 0)
+ ret = cmp < 0 ? MATCH_LT : MATCH_GT;
+
+ free_srcline(left);
+ free_srcline(right);
+ return ret;
+}
+
static enum match_result match_chain(struct callchain_cursor_node *node,
struct callchain_list *cnode)
{
struct symbol *sym = node->sym;
u64 left, right;
- if (cnode->ms.sym && sym &&
- callchain_param.key == CCKEY_FUNCTION) {
+ if (callchain_param.key == CCKEY_SRCLINE) {
+ enum match_result match = match_chain_srcline(node, cnode);
+
+ if (match != MATCH_ERROR)
+ return match;
+ }
+
+ if (cnode->ms.sym && sym && callchain_param.key == CCKEY_FUNCTION) {
left = cnode->ms.sym->start;
right = sym->start;
} else {
@@ -911,15 +1056,16 @@
char *callchain_list__sym_name(struct callchain_list *cl,
char *bf, size_t bfsize, bool show_dso)
{
+ bool show_addr = callchain_param.key == CCKEY_ADDRESS;
+ bool show_srcline = show_addr || callchain_param.key == CCKEY_SRCLINE;
int printed;
if (cl->ms.sym) {
- if (callchain_param.key == CCKEY_ADDRESS &&
- cl->ms.map && !cl->srcline)
+ if (show_srcline && cl->ms.map && !cl->srcline)
cl->srcline = get_srcline(cl->ms.map->dso,
map__rip_2objdump(cl->ms.map,
cl->ip),
- cl->ms.sym, false);
+ cl->ms.sym, false, show_addr);
if (cl->srcline)
printed = scnprintf(bf, bfsize, "%s %s",
cl->ms.sym->name, cl->srcline);
@@ -1063,63 +1209,100 @@
cycles_count);
}
+static int counts_str_build(char *bf, int bfsize,
+ u64 branch_count, u64 predicted_count,
+ u64 abort_count, u64 cycles_count,
+ u64 iter_count, u64 samples_count)
+{
+ double predicted_percent = 0.0;
+ const char *null_str = "";
+ char iter_str[32];
+ char cycle_str[32];
+ char *istr, *cstr;
+ u64 cycles;
+
+ if (branch_count == 0)
+ return scnprintf(bf, bfsize, " (calltrace)");
+
+ cycles = cycles_count / branch_count;
+
+ if (iter_count && samples_count) {
+ if (cycles > 0)
+ scnprintf(iter_str, sizeof(iter_str),
+ " iterations:%" PRId64 "",
+ iter_count / samples_count);
+ else
+ scnprintf(iter_str, sizeof(iter_str),
+ "iterations:%" PRId64 "",
+ iter_count / samples_count);
+ istr = iter_str;
+ } else
+ istr = (char *)null_str;
+
+ if (cycles > 0) {
+ scnprintf(cycle_str, sizeof(cycle_str),
+ "cycles:%" PRId64 "", cycles);
+ cstr = cycle_str;
+ } else
+ cstr = (char *)null_str;
+
+ predicted_percent = predicted_count * 100.0 / branch_count;
+
+ if ((predicted_count == branch_count) && (abort_count == 0)) {
+ if ((cycles > 0) || (istr != (char *)null_str))
+ return scnprintf(bf, bfsize, " (%s%s)", cstr, istr);
+ else
+ return scnprintf(bf, bfsize, "%s", (char *)null_str);
+ }
+
+ if ((predicted_count < branch_count) && (abort_count == 0)) {
+ if ((cycles > 0) || (istr != (char *)null_str))
+ return scnprintf(bf, bfsize,
+ " (predicted:%.1f%% %s%s)",
+ predicted_percent, cstr, istr);
+ else {
+ return scnprintf(bf, bfsize,
+ " (predicted:%.1f%%)",
+ predicted_percent);
+ }
+ }
+
+ if ((predicted_count == branch_count) && (abort_count > 0)) {
+ if ((cycles > 0) || (istr != (char *)null_str))
+ return scnprintf(bf, bfsize,
+ " (abort:%" PRId64 " %s%s)",
+ abort_count, cstr, istr);
+ else
+ return scnprintf(bf, bfsize,
+ " (abort:%" PRId64 ")",
+ abort_count);
+ }
+
+ if ((cycles > 0) || (istr != (char *)null_str))
+ return scnprintf(bf, bfsize,
+ " (predicted:%.1f%% abort:%" PRId64 " %s%s)",
+ predicted_percent, abort_count, cstr, istr);
+
+ return scnprintf(bf, bfsize,
+ " (predicted:%.1f%% abort:%" PRId64 ")",
+ predicted_percent, abort_count);
+}
+
static int callchain_counts_printf(FILE *fp, char *bf, int bfsize,
u64 branch_count, u64 predicted_count,
u64 abort_count, u64 cycles_count,
u64 iter_count, u64 samples_count)
{
- double predicted_percent = 0.0;
- const char *null_str = "";
- char iter_str[32];
- char *str;
- u64 cycles = 0;
+ char str[128];
- if (branch_count == 0) {
- if (fp)
- return fprintf(fp, " (calltrace)");
-
- return scnprintf(bf, bfsize, " (calltrace)");
- }
-
- if (iter_count && samples_count) {
- scnprintf(iter_str, sizeof(iter_str),
- ", iterations:%" PRId64 "",
- iter_count / samples_count);
- str = iter_str;
- } else
- str = (char *)null_str;
-
- predicted_percent = predicted_count * 100.0 / branch_count;
- cycles = cycles_count / branch_count;
-
- if ((predicted_percent >= 100.0) && (abort_count == 0)) {
- if (fp)
- return fprintf(fp, " (cycles:%" PRId64 "%s)",
- cycles, str);
-
- return scnprintf(bf, bfsize, " (cycles:%" PRId64 "%s)",
- cycles, str);
- }
-
- if ((predicted_percent < 100.0) && (abort_count == 0)) {
- if (fp)
- return fprintf(fp,
- " (predicted:%.1f%%, cycles:%" PRId64 "%s)",
- predicted_percent, cycles, str);
-
- return scnprintf(bf, bfsize,
- " (predicted:%.1f%%, cycles:%" PRId64 "%s)",
- predicted_percent, cycles, str);
- }
+ counts_str_build(str, sizeof(str), branch_count,
+ predicted_count, abort_count, cycles_count,
+ iter_count, samples_count);
if (fp)
- return fprintf(fp,
- " (predicted:%.1f%%, abort:%" PRId64 ", cycles:%" PRId64 "%s)",
- predicted_percent, abort_count, cycles, str);
+ return fprintf(fp, "%s", str);
- return scnprintf(bf, bfsize,
- " (predicted:%.1f%%, abort:%" PRId64 ", cycles:%" PRId64 "%s)",
- predicted_percent, abort_count, cycles, str);
+ return scnprintf(bf, bfsize, "%s", str);
}
int callchain_list_counts__printf_value(struct callchain_node *node,
diff --git a/tools/perf/util/callchain.h b/tools/perf/util/callchain.h
index 4f4b60f..c56c23d 100644
--- a/tools/perf/util/callchain.h
+++ b/tools/perf/util/callchain.h
@@ -77,7 +77,8 @@
enum chain_key {
CCKEY_FUNCTION,
- CCKEY_ADDRESS
+ CCKEY_ADDRESS,
+ CCKEY_SRCLINE
};
enum chain_value {
diff --git a/tools/perf/util/cgroup.c b/tools/perf/util/cgroup.c
index eafbf114..0334774 100644
--- a/tools/perf/util/cgroup.c
+++ b/tools/perf/util/cgroup.c
@@ -4,6 +4,7 @@
#include "evsel.h"
#include "cgroup.h"
#include "evlist.h"
+#include <linux/stringify.h>
int nr_cgroups;
@@ -27,8 +28,8 @@
path_v1[0] = '\0';
path_v2[0] = '\0';
- while (fscanf(fp, "%*s %"STR(PATH_MAX)"s %"STR(PATH_MAX)"s %"
- STR(PATH_MAX)"s %*d %*d\n",
+ while (fscanf(fp, "%*s %"__stringify(PATH_MAX)"s %"__stringify(PATH_MAX)"s %"
+ __stringify(PATH_MAX)"s %*d %*d\n",
mountpoint, type, tokens) == 3) {
if (!path_v1[0] && !strcmp(type, "cgroup")) {
@@ -127,19 +128,19 @@
goto found;
n++;
}
- if (atomic_read(&cgrp->refcnt) == 0)
+ if (refcount_read(&cgrp->refcnt) == 0)
free(cgrp);
return -1;
found:
- atomic_inc(&cgrp->refcnt);
+ refcount_inc(&cgrp->refcnt);
counter->cgrp = cgrp;
return 0;
}
void close_cgroup(struct cgroup_sel *cgrp)
{
- if (cgrp && atomic_dec_and_test(&cgrp->refcnt)) {
+ if (cgrp && refcount_dec_and_test(&cgrp->refcnt)) {
close(cgrp->fd);
zfree(&cgrp->name);
free(cgrp);
diff --git a/tools/perf/util/cgroup.h b/tools/perf/util/cgroup.h
index 31f8dcd..d91966b 100644
--- a/tools/perf/util/cgroup.h
+++ b/tools/perf/util/cgroup.h
@@ -1,14 +1,14 @@
#ifndef __CGROUP_H__
#define __CGROUP_H__
-#include <linux/atomic.h>
+#include <linux/refcount.h>
struct option;
struct cgroup_sel {
char *name;
int fd;
- atomic_t refcnt;
+ refcount_t refcnt;
};
diff --git a/tools/perf/util/cloexec.c b/tools/perf/util/cloexec.c
index f0dcd0e..4b4f00d 100644
--- a/tools/perf/util/cloexec.c
+++ b/tools/perf/util/cloexec.c
@@ -1,3 +1,4 @@
+#include <errno.h>
#include <sched.h>
#include "util.h"
#include "../perf.h"
diff --git a/tools/perf/util/cloexec.h b/tools/perf/util/cloexec.h
index d0d4659..94a5a7d 100644
--- a/tools/perf/util/cloexec.h
+++ b/tools/perf/util/cloexec.h
@@ -3,10 +3,4 @@
unsigned long perf_event_open_cloexec_flag(void);
-#ifdef __GLIBC_PREREQ
-#if !__GLIBC_PREREQ(2, 6) && !defined(__UCLIBC__)
-int sched_getcpu(void) __THROW;
-#endif
-#endif
-
#endif /* __PERF_CLOEXEC_H */
diff --git a/tools/perf/util/color.h b/tools/perf/util/color.h
index a93997f..52122bc 100644
--- a/tools/perf/util/color.h
+++ b/tools/perf/util/color.h
@@ -1,6 +1,8 @@
#ifndef __PERF_COLOR_H
#define __PERF_COLOR_H
+#include <stdio.h>
+
/* "\033[1;38;5;2xx;48;5;2xxm\0" is 23 bytes */
#define COLOR_MAXLEN 24
diff --git a/tools/perf/util/comm.c b/tools/perf/util/comm.c
index 21b7ff3..7bc981b 100644
--- a/tools/perf/util/comm.c
+++ b/tools/perf/util/comm.c
@@ -1,13 +1,15 @@
#include "comm.h"
#include "util.h"
+#include <errno.h>
#include <stdlib.h>
#include <stdio.h>
-#include <linux/atomic.h>
+#include <string.h>
+#include <linux/refcount.h>
struct comm_str {
char *str;
struct rb_node rb_node;
- atomic_t refcnt;
+ refcount_t refcnt;
};
/* Should perhaps be moved to struct machine */
@@ -16,13 +18,13 @@
static struct comm_str *comm_str__get(struct comm_str *cs)
{
if (cs)
- atomic_inc(&cs->refcnt);
+ refcount_inc(&cs->refcnt);
return cs;
}
static void comm_str__put(struct comm_str *cs)
{
- if (cs && atomic_dec_and_test(&cs->refcnt)) {
+ if (cs && refcount_dec_and_test(&cs->refcnt)) {
rb_erase(&cs->rb_node, &comm_str_root);
zfree(&cs->str);
free(cs);
@@ -43,7 +45,7 @@
return NULL;
}
- atomic_set(&cs->refcnt, 0);
+ refcount_set(&cs->refcnt, 1);
return cs;
}
@@ -61,7 +63,7 @@
cmp = strcmp(str, iter->str);
if (!cmp)
- return iter;
+ return comm_str__get(iter);
if (cmp < 0)
p = &(*p)->rb_left;
@@ -95,8 +97,6 @@
return NULL;
}
- comm_str__get(comm->comm_str);
-
return comm;
}
@@ -108,7 +108,6 @@
if (!new)
return -ENOMEM;
- comm_str__get(new);
comm_str__put(old);
comm->comm_str = new;
comm->start = timestamp;
diff --git a/tools/perf/util/compress.h b/tools/perf/util/compress.h
new file mode 100644
index 0000000..67fd1bb
--- /dev/null
+++ b/tools/perf/util/compress.h
@@ -0,0 +1,12 @@
+#ifndef PERF_COMPRESS_H
+#define PERF_COMPRESS_H
+
+#ifdef HAVE_ZLIB_SUPPORT
+int gzip_decompress_to_file(const char *input, int output_fd);
+#endif
+
+#ifdef HAVE_LZMA_SUPPORT
+int lzma_decompress_to_file(const char *input, int output_fd);
+#endif
+
+#endif /* PERF_COMPRESS_H */
diff --git a/tools/perf/util/config.c b/tools/perf/util/config.c
index 0c7d5a4..8d724f0 100644
--- a/tools/perf/util/config.c
+++ b/tools/perf/util/config.c
@@ -8,12 +8,19 @@
* Copyright (C) Johannes Schindelin, 2005
*
*/
+#include <errno.h>
+#include <sys/param.h>
#include "util.h"
#include "cache.h"
#include <subcmd/exec-cmd.h>
#include "util/hist.h" /* perf_hist_config */
#include "util/llvm-utils.h" /* perf_llvm_config */
#include "config.h"
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "sane_ctype.h"
#define MAXNAME (256)
@@ -627,6 +634,8 @@
{
int ret = -1;
const char *home = NULL;
+ char *user_config;
+ struct stat st;
/* Setting $PERF_CONFIG makes perf read _only_ the given config file. */
if (config_exclusive_filename)
@@ -637,35 +646,41 @@
}
home = getenv("HOME");
- if (perf_config_global() && home) {
- char *user_config = strdup(mkpath("%s/.perfconfig", home));
- struct stat st;
- if (user_config == NULL) {
- warning("Not enough memory to process %s/.perfconfig, "
- "ignoring it.", home);
- goto out;
- }
+ /*
+ * Skip reading user config if:
+ * - there is no place to read it from (HOME)
+ * - we are asked not to (PERF_CONFIG_NOGLOBAL=1)
+ */
+ if (!home || !*home || !perf_config_global())
+ return 0;
- if (stat(user_config, &st) < 0) {
- if (errno == ENOENT)
- ret = 0;
- goto out_free;
- }
-
- ret = 0;
-
- if (st.st_uid && (st.st_uid != geteuid())) {
- warning("File %s not owned by current user or root, "
- "ignoring it.", user_config);
- goto out_free;
- }
-
- if (st.st_size)
- ret = perf_config_from_file(collect_config, user_config, set);
-out_free:
- free(user_config);
+ user_config = strdup(mkpath("%s/.perfconfig", home));
+ if (user_config == NULL) {
+ warning("Not enough memory to process %s/.perfconfig, "
+ "ignoring it.", home);
+ goto out;
}
+
+ if (stat(user_config, &st) < 0) {
+ if (errno == ENOENT)
+ ret = 0;
+ goto out_free;
+ }
+
+ ret = 0;
+
+ if (st.st_uid && (st.st_uid != geteuid())) {
+ warning("File %s not owned by current user or root, "
+ "ignoring it.", user_config);
+ goto out_free;
+ }
+
+ if (st.st_size)
+ ret = perf_config_from_file(collect_config, user_config, set);
+
+out_free:
+ free(user_config);
out:
return ret;
}
diff --git a/tools/perf/util/counts.c b/tools/perf/util/counts.c
index e3fde31..c4af82a 100644
--- a/tools/perf/util/counts.c
+++ b/tools/perf/util/counts.c
@@ -1,6 +1,8 @@
+#include <errno.h>
#include <stdlib.h>
#include "evsel.h"
#include "counts.h"
+#include "util.h"
struct perf_counts *perf_counts__new(int ncpus, int nthreads)
{
diff --git a/tools/perf/util/cpumap.c b/tools/perf/util/cpumap.c
index 8c75049..37b3bb7 100644
--- a/tools/perf/util/cpumap.c
+++ b/tools/perf/util/cpumap.c
@@ -3,11 +3,14 @@
#include "../perf.h"
#include "cpumap.h"
#include <assert.h>
+#include <dirent.h>
#include <stdio.h>
#include <stdlib.h>
#include <linux/bitmap.h>
#include "asm/bug.h"
+#include "sane_ctype.h"
+
static int max_cpu_num;
static int max_present_cpu_num;
static int max_node_num;
@@ -29,7 +32,7 @@
cpus->map[i] = i;
cpus->nr = nr_cpus;
- atomic_set(&cpus->refcnt, 1);
+ refcount_set(&cpus->refcnt, 1);
}
return cpus;
@@ -43,7 +46,7 @@
if (cpus != NULL) {
cpus->nr = nr_cpus;
memcpy(cpus->map, tmp_cpus, payload_size);
- atomic_set(&cpus->refcnt, 1);
+ refcount_set(&cpus->refcnt, 1);
}
return cpus;
@@ -252,7 +255,7 @@
if (cpus != NULL) {
cpus->nr = 1;
cpus->map[0] = -1;
- atomic_set(&cpus->refcnt, 1);
+ refcount_set(&cpus->refcnt, 1);
}
return cpus;
@@ -269,7 +272,7 @@
for (i = 0; i < nr; i++)
cpus->map[i] = -1;
- atomic_set(&cpus->refcnt, 1);
+ refcount_set(&cpus->refcnt, 1);
}
return cpus;
@@ -278,7 +281,7 @@
static void cpu_map__delete(struct cpu_map *map)
{
if (map) {
- WARN_ONCE(atomic_read(&map->refcnt) != 0,
+ WARN_ONCE(refcount_read(&map->refcnt) != 0,
"cpu_map refcnt unbalanced\n");
free(map);
}
@@ -287,13 +290,13 @@
struct cpu_map *cpu_map__get(struct cpu_map *map)
{
if (map)
- atomic_inc(&map->refcnt);
+ refcount_inc(&map->refcnt);
return map;
}
void cpu_map__put(struct cpu_map *map)
{
- if (map && atomic_dec_and_test(&map->refcnt))
+ if (map && refcount_dec_and_test(&map->refcnt))
cpu_map__delete(map);
}
@@ -357,7 +360,7 @@
/* ensure we process id in increasing order */
qsort(c->map, c->nr, sizeof(int), cmp_ids);
- atomic_set(&c->refcnt, 1);
+ refcount_set(&c->refcnt, 1);
*res = c;
return 0;
}
@@ -673,3 +676,49 @@
pr_debug("cpumask list: %s\n", buf);
return ret;
}
+
+static char hex_char(unsigned char val)
+{
+ if (val < 10)
+ return val + '0';
+ if (val < 16)
+ return val - 10 + 'a';
+ return '?';
+}
+
+size_t cpu_map__snprint_mask(struct cpu_map *map, char *buf, size_t size)
+{
+ int i, cpu;
+ char *ptr = buf;
+ unsigned char *bitmap;
+ int last_cpu = cpu_map__cpu(map, map->nr - 1);
+
+ bitmap = zalloc((last_cpu + 7) / 8);
+ if (bitmap == NULL) {
+ buf[0] = '\0';
+ return 0;
+ }
+
+ for (i = 0; i < map->nr; i++) {
+ cpu = cpu_map__cpu(map, i);
+ bitmap[cpu / 8] |= 1 << (cpu % 8);
+ }
+
+ for (cpu = last_cpu / 4 * 4; cpu >= 0; cpu -= 4) {
+ unsigned char bits = bitmap[cpu / 8];
+
+ if (cpu % 8)
+ bits >>= 4;
+ else
+ bits &= 0xf;
+
+ *ptr++ = hex_char(bits);
+ if ((cpu % 32) == 0 && cpu > 0)
+ *ptr++ = ',';
+ }
+ *ptr = '\0';
+ free(bitmap);
+
+ buf[size - 1] = '\0';
+ return ptr - buf;
+}
diff --git a/tools/perf/util/cpumap.h b/tools/perf/util/cpumap.h
index 1a0549a..6b8bff8 100644
--- a/tools/perf/util/cpumap.h
+++ b/tools/perf/util/cpumap.h
@@ -3,13 +3,13 @@
#include <stdio.h>
#include <stdbool.h>
-#include <linux/atomic.h>
+#include <linux/refcount.h>
#include "perf.h"
#include "util/debug.h"
struct cpu_map {
- atomic_t refcnt;
+ refcount_t refcnt;
int nr;
int map[];
};
@@ -20,6 +20,7 @@
struct cpu_map *cpu_map__new_data(struct cpu_map_data *data);
struct cpu_map *cpu_map__read(FILE *file);
size_t cpu_map__snprint(struct cpu_map *map, char *buf, size_t size);
+size_t cpu_map__snprint_mask(struct cpu_map *map, char *buf, size_t size);
size_t cpu_map__fprintf(struct cpu_map *map, FILE *fp);
int cpu_map__get_socket_id(int cpu);
int cpu_map__get_socket(struct cpu_map *map, int idx, void *data);
diff --git a/tools/perf/util/ctype.c b/tools/perf/util/ctype.c
index d4a5a21..4b261c2 100644
--- a/tools/perf/util/ctype.c
+++ b/tools/perf/util/ctype.c
@@ -3,7 +3,7 @@
*
* No surprises, and works with signed and unsigned chars.
*/
-#include "util.h"
+#include "sane_ctype.h"
enum {
S = GIT_SPACE,
diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index 4e6cbc9..89d5031 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -7,7 +7,10 @@
* Released under the GPL v2. (and only v2, not any later version)
*/
+#include <errno.h>
+#include <inttypes.h>
#include <linux/compiler.h>
+#include <linux/kernel.h>
#include <babeltrace/ctf-writer/writer.h>
#include <babeltrace/ctf-writer/clock.h>
#include <babeltrace/ctf-writer/stream.h>
@@ -27,6 +30,7 @@
#include "evsel.h"
#include "machine.h"
#include "config.h"
+#include "sane_ctype.h"
#define pr_N(n, fmt, ...) \
eprintf(n, debug_data_convert, fmt, ##__VA_ARGS__)
@@ -1468,6 +1472,7 @@
.lost = perf_event__process_lost,
.tracing_data = perf_event__process_tracing_data,
.build_id = perf_event__process_build_id,
+ .namespaces = perf_event__process_namespaces,
.ordered_events = true,
.ordering_requires_timestamps = true,
},
diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
index 60bfc9c..e84bbc8 100644
--- a/tools/perf/util/data.c
+++ b/tools/perf/util/data.c
@@ -2,6 +2,7 @@
#include <linux/kernel.h>
#include <sys/types.h>
#include <sys/stat.h>
+#include <errno.h>
#include <unistd.h>
#include <string.h>
diff --git a/tools/perf/util/debug.c b/tools/perf/util/debug.c
index 03eb81f..a5b3777 100644
--- a/tools/perf/util/debug.c
+++ b/tools/perf/util/debug.c
@@ -2,19 +2,26 @@
#include "../perf.h"
+#include <inttypes.h>
#include <string.h>
#include <stdarg.h>
#include <stdio.h>
+#include <sys/wait.h>
#include <api/debug.h>
#include <linux/time64.h>
-
+#ifdef HAVE_BACKTRACE_SUPPORT
+#include <execinfo.h>
+#endif
#include "cache.h"
#include "color.h"
#include "event.h"
#include "debug.h"
+#include "print_binary.h"
#include "util.h"
#include "target.h"
+#include "sane_ctype.h"
+
int verbose;
bool dump_trace = false, quiet = false;
int debug_ordered_events;
@@ -244,3 +251,31 @@
{
libapi_set_print(pr_warning_wrapper, pr_warning_wrapper, pr_debug_wrapper);
}
+
+/* Obtain a backtrace and print it to stdout. */
+#ifdef HAVE_BACKTRACE_SUPPORT
+void dump_stack(void)
+{
+ void *array[16];
+ size_t size = backtrace(array, ARRAY_SIZE(array));
+ char **strings = backtrace_symbols(array, size);
+ size_t i;
+
+ printf("Obtained %zd stack frames.\n", size);
+
+ for (i = 0; i < size; i++)
+ printf("%s\n", strings[i]);
+
+ free(strings);
+}
+#else
+void dump_stack(void) {}
+#endif
+
+void sighandler_dump_stack(int sig)
+{
+ psignal(sig, "perf");
+ dump_stack();
+ signal(sig, SIG_DFL);
+ raise(sig);
+}
diff --git a/tools/perf/util/debug.h b/tools/perf/util/debug.h
index 98832f5..8a23ea1 100644
--- a/tools/perf/util/debug.h
+++ b/tools/perf/util/debug.h
@@ -56,4 +56,7 @@
void perf_debug_setup(void);
int perf_quiet_option(void);
+void dump_stack(void);
+void sighandler_dump_stack(int sig);
+
#endif /* __PERF_DEBUG_H */
diff --git a/tools/perf/util/demangle-java.c b/tools/perf/util/demangle-java.c
index 3e6062a..cb66d33 100644
--- a/tools/perf/util/demangle-java.c
+++ b/tools/perf/util/demangle-java.c
@@ -7,6 +7,8 @@
#include "demangle-java.h"
+#include "sane_ctype.h"
+
enum {
MODE_PREFIX = 0,
MODE_CLASS = 1,
diff --git a/tools/perf/util/drv_configs.c b/tools/perf/util/drv_configs.c
index 1647f28..eec7542 100644
--- a/tools/perf/util/drv_configs.c
+++ b/tools/perf/util/drv_configs.c
@@ -17,6 +17,7 @@
#include "evlist.h"
#include "evsel.h"
#include "pmu.h"
+#include <errno.h>
static int
perf_evsel__apply_drv_configs(struct perf_evsel *evsel,
diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index d38b62a..a96a99d 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -1,12 +1,20 @@
#include <asm/bug.h>
+#include <linux/kernel.h>
#include <sys/time.h>
#include <sys/resource.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+#include <errno.h>
+#include "compress.h"
+#include "path.h"
#include "symbol.h"
#include "dso.h"
#include "machine.h"
#include "auxtrace.h"
#include "util.h"
#include "debug.h"
+#include "string2.h"
#include "vdso.h"
static const char * const debuglink_paths[] = {
@@ -1109,7 +1117,7 @@
INIT_LIST_HEAD(&dso->node);
INIT_LIST_HEAD(&dso->data.open_entry);
pthread_mutex_init(&dso->lock, NULL);
- atomic_set(&dso->refcnt, 1);
+ refcount_set(&dso->refcnt, 1);
}
return dso;
@@ -1147,13 +1155,13 @@
struct dso *dso__get(struct dso *dso)
{
if (dso)
- atomic_inc(&dso->refcnt);
+ refcount_inc(&dso->refcnt);
return dso;
}
void dso__put(struct dso *dso)
{
- if (dso && atomic_dec_and_test(&dso->refcnt))
+ if (dso && refcount_dec_and_test(&dso->refcnt))
dso__delete(dso);
}
diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index ecc4bbd..12350b1 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -1,7 +1,7 @@
#ifndef __PERF_DSO
#define __PERF_DSO
-#include <linux/atomic.h>
+#include <linux/refcount.h>
#include <linux/types.h>
#include <linux/rbtree.h>
#include <sys/types.h>
@@ -187,7 +187,7 @@
void *priv;
u64 db_id;
};
- atomic_t refcnt;
+ refcount_t refcnt;
char name[0];
};
diff --git a/tools/perf/util/dump-insn.c b/tools/perf/util/dump-insn.c
new file mode 100644
index 0000000..ffbdb19
--- /dev/null
+++ b/tools/perf/util/dump-insn.c
@@ -0,0 +1,14 @@
+#include <linux/compiler.h>
+#include "dump-insn.h"
+
+/* Fallback code */
+
+__weak
+const char *dump_insn(struct perf_insn *x __maybe_unused,
+ u64 ip __maybe_unused, u8 *inbuf __maybe_unused,
+ int inlen __maybe_unused, int *lenp)
+{
+ if (lenp)
+ *lenp = 0;
+ return "?";
+}
diff --git a/tools/perf/util/dump-insn.h b/tools/perf/util/dump-insn.h
new file mode 100644
index 0000000..90fb115
--- /dev/null
+++ b/tools/perf/util/dump-insn.h
@@ -0,0 +1,22 @@
+#ifndef __PERF_DUMP_INSN_H
+#define __PERF_DUMP_INSN_H 1
+
+#define MAXINSN 15
+
+#include <linux/types.h>
+
+struct thread;
+
+struct perf_insn {
+ /* Initialized by callers: */
+ struct thread *thread;
+ u8 cpumode;
+ bool is64bit;
+ int cpu;
+ /* Temporary */
+ char out[256];
+};
+
+const char *dump_insn(struct perf_insn *x, u64 ip,
+ u8 *inbuf, int inlen, int *lenp);
+#endif
diff --git a/tools/perf/util/dwarf-aux.c b/tools/perf/util/dwarf-aux.c
index 41e068e..f5acda1 100644
--- a/tools/perf/util/dwarf-aux.c
+++ b/tools/perf/util/dwarf-aux.c
@@ -17,10 +17,13 @@
*
*/
+#include <errno.h>
+#include <inttypes.h>
#include <stdbool.h>
#include "util.h"
#include "debug.h"
#include "dwarf-aux.h"
+#include "string2.h"
/**
* cu_find_realpath - Find the realpath of the target file
diff --git a/tools/perf/util/dwarf-regs.c b/tools/perf/util/dwarf-regs.c
index 62bc4a8..c708395 100644
--- a/tools/perf/util/dwarf-regs.c
+++ b/tools/perf/util/dwarf-regs.c
@@ -8,6 +8,7 @@
#include <debug.h>
#include <dwarf-regs.h>
#include <elf.h>
+#include <linux/kernel.h>
#ifndef EM_AARCH64
#define EM_AARCH64 183 /* ARM 64 bit */
diff --git a/tools/perf/util/env.c b/tools/perf/util/env.c
index 075fc77..9e21538 100644
--- a/tools/perf/util/env.c
+++ b/tools/perf/util/env.c
@@ -1,6 +1,7 @@
#include "cpumap.h"
#include "env.h"
#include "util.h"
+#include <errno.h>
struct perf_env perf_env;
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 4ea7ce7..142835c 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -1,15 +1,23 @@
+#include <dirent.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <linux/kernel.h>
#include <linux/types.h>
-#include <linux/mman.h> /* To get things like MAP_HUGETLB even on older libc headers */
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+#include <uapi/linux/mman.h> /* To get things like MAP_HUGETLB even on older libc headers */
#include <api/fs/fs.h>
#include "event.h"
#include "debug.h"
#include "hist.h"
#include "machine.h"
#include "sort.h"
-#include "string.h"
+#include "string2.h"
#include "strlist.h"
#include "thread.h"
#include "thread_map.h"
+#include "sane_ctype.h"
#include "symbol/kallsyms.h"
#include "asm/bug.h"
#include "stat.h"
@@ -31,6 +39,7 @@
[PERF_RECORD_LOST_SAMPLES] = "LOST_SAMPLES",
[PERF_RECORD_SWITCH] = "SWITCH",
[PERF_RECORD_SWITCH_CPU_WIDE] = "SWITCH_CPU_WIDE",
+ [PERF_RECORD_NAMESPACES] = "NAMESPACES",
[PERF_RECORD_HEADER_ATTR] = "ATTR",
[PERF_RECORD_HEADER_EVENT_TYPE] = "EVENT_TYPE",
[PERF_RECORD_HEADER_TRACING_DATA] = "TRACING_DATA",
@@ -49,6 +58,16 @@
[PERF_RECORD_TIME_CONV] = "TIME_CONV",
};
+static const char *perf_ns__names[] = {
+ [NET_NS_INDEX] = "net",
+ [UTS_NS_INDEX] = "uts",
+ [IPC_NS_INDEX] = "ipc",
+ [PID_NS_INDEX] = "pid",
+ [USER_NS_INDEX] = "user",
+ [MNT_NS_INDEX] = "mnt",
+ [CGROUP_NS_INDEX] = "cgroup",
+};
+
const char *perf_event__name(unsigned int id)
{
if (id >= ARRAY_SIZE(perf_event__names))
@@ -58,6 +77,13 @@
return perf_event__names[id];
}
+static const char *perf_ns__name(unsigned int id)
+{
+ if (id >= ARRAY_SIZE(perf_ns__names))
+ return "UNKNOWN";
+ return perf_ns__names[id];
+}
+
static int perf_tool__process_synth_event(struct perf_tool *tool,
union perf_event *event,
struct machine *machine,
@@ -88,7 +114,7 @@
int fd;
size_t size = 0;
ssize_t n;
- char *nl, *name, *tgids, *ppids;
+ char *name, *tgids, *ppids;
*tgid = -1;
*ppid = -1;
@@ -115,10 +141,10 @@
ppids = strstr(bf, "PPid:");
if (name) {
- name += 5; /* strlen("Name:") */
+ char *nl;
- while (*name && isspace(*name))
- ++name;
+ name += 5; /* strlen("Name:") */
+ name = ltrim(name);
nl = strchr(name, '\n');
if (nl)
@@ -203,6 +229,58 @@
return tgid;
}
+static void perf_event__get_ns_link_info(pid_t pid, const char *ns,
+ struct perf_ns_link_info *ns_link_info)
+{
+ struct stat64 st;
+ char proc_ns[128];
+
+ sprintf(proc_ns, "/proc/%u/ns/%s", pid, ns);
+ if (stat64(proc_ns, &st) == 0) {
+ ns_link_info->dev = st.st_dev;
+ ns_link_info->ino = st.st_ino;
+ }
+}
+
+int perf_event__synthesize_namespaces(struct perf_tool *tool,
+ union perf_event *event,
+ pid_t pid, pid_t tgid,
+ perf_event__handler_t process,
+ struct machine *machine)
+{
+ u32 idx;
+ struct perf_ns_link_info *ns_link_info;
+
+ if (!tool || !tool->namespace_events)
+ return 0;
+
+ memset(&event->namespaces, 0, (sizeof(event->namespaces) +
+ (NR_NAMESPACES * sizeof(struct perf_ns_link_info)) +
+ machine->id_hdr_size));
+
+ event->namespaces.pid = tgid;
+ event->namespaces.tid = pid;
+
+ event->namespaces.nr_namespaces = NR_NAMESPACES;
+
+ ns_link_info = event->namespaces.link_info;
+
+ for (idx = 0; idx < event->namespaces.nr_namespaces; idx++)
+ perf_event__get_ns_link_info(pid, perf_ns__name(idx),
+ &ns_link_info[idx]);
+
+ event->namespaces.header.type = PERF_RECORD_NAMESPACES;
+
+ event->namespaces.header.size = (sizeof(event->namespaces) +
+ (NR_NAMESPACES * sizeof(struct perf_ns_link_info)) +
+ machine->id_hdr_size);
+
+ if (perf_tool__process_synth_event(tool, event, machine, process) != 0)
+ return -1;
+
+ return 0;
+}
+
static int perf_event__synthesize_fork(struct perf_tool *tool,
union perf_event *event,
pid_t pid, pid_t tgid, pid_t ppid,
@@ -255,8 +333,8 @@
if (machine__is_default_guest(machine))
return 0;
- snprintf(filename, sizeof(filename), "%s/proc/%d/maps",
- machine->root_dir, pid);
+ snprintf(filename, sizeof(filename), "%s/proc/%d/task/%d/maps",
+ machine->root_dir, pid, pid);
fp = fopen(filename, "r");
if (fp == NULL) {
@@ -434,8 +512,9 @@
static int __event__synthesize_thread(union perf_event *comm_event,
union perf_event *mmap_event,
union perf_event *fork_event,
+ union perf_event *namespaces_event,
pid_t pid, int full,
- perf_event__handler_t process,
+ perf_event__handler_t process,
struct perf_tool *tool,
struct machine *machine,
bool mmap_data,
@@ -455,6 +534,11 @@
if (tgid == -1)
return -1;
+ if (perf_event__synthesize_namespaces(tool, namespaces_event, pid,
+ tgid, process, machine) < 0)
+ return -1;
+
+
return perf_event__synthesize_mmap_events(tool, mmap_event, pid, tgid,
process, machine, mmap_data,
proc_map_timeout);
@@ -488,6 +572,11 @@
if (perf_event__synthesize_fork(tool, fork_event, _pid, tgid,
ppid, process, machine) < 0)
break;
+
+ if (perf_event__synthesize_namespaces(tool, namespaces_event, _pid,
+ tgid, process, machine) < 0)
+ break;
+
/*
* Send the prepared comm event
*/
@@ -516,6 +605,7 @@
unsigned int proc_map_timeout)
{
union perf_event *comm_event, *mmap_event, *fork_event;
+ union perf_event *namespaces_event;
int err = -1, thread, j;
comm_event = malloc(sizeof(comm_event->comm) + machine->id_hdr_size);
@@ -530,10 +620,16 @@
if (fork_event == NULL)
goto out_free_mmap;
+ namespaces_event = malloc(sizeof(namespaces_event->namespaces) +
+ (NR_NAMESPACES * sizeof(struct perf_ns_link_info)) +
+ machine->id_hdr_size);
+ if (namespaces_event == NULL)
+ goto out_free_fork;
+
err = 0;
for (thread = 0; thread < threads->nr; ++thread) {
if (__event__synthesize_thread(comm_event, mmap_event,
- fork_event,
+ fork_event, namespaces_event,
thread_map__pid(threads, thread), 0,
process, tool, machine,
mmap_data, proc_map_timeout)) {
@@ -559,7 +655,7 @@
/* if not, generate events for it */
if (need_leader &&
__event__synthesize_thread(comm_event, mmap_event,
- fork_event,
+ fork_event, namespaces_event,
comm_event->comm.pid, 0,
process, tool, machine,
mmap_data, proc_map_timeout)) {
@@ -568,6 +664,8 @@
}
}
}
+ free(namespaces_event);
+out_free_fork:
free(fork_event);
out_free_mmap:
free(mmap_event);
@@ -587,6 +685,7 @@
char proc_path[PATH_MAX];
struct dirent *dirent;
union perf_event *comm_event, *mmap_event, *fork_event;
+ union perf_event *namespaces_event;
int err = -1;
if (machine__is_default_guest(machine))
@@ -604,11 +703,17 @@
if (fork_event == NULL)
goto out_free_mmap;
+ namespaces_event = malloc(sizeof(namespaces_event->namespaces) +
+ (NR_NAMESPACES * sizeof(struct perf_ns_link_info)) +
+ machine->id_hdr_size);
+ if (namespaces_event == NULL)
+ goto out_free_fork;
+
snprintf(proc_path, sizeof(proc_path), "%s/proc", machine->root_dir);
proc = opendir(proc_path);
if (proc == NULL)
- goto out_free_fork;
+ goto out_free_namespaces;
while ((dirent = readdir(proc)) != NULL) {
char *end;
@@ -620,13 +725,16 @@
* We may race with exiting thread, so don't stop just because
* one thread couldn't be synthesized.
*/
- __event__synthesize_thread(comm_event, mmap_event, fork_event, pid,
- 1, process, tool, machine, mmap_data,
+ __event__synthesize_thread(comm_event, mmap_event, fork_event,
+ namespaces_event, pid, 1, process,
+ tool, machine, mmap_data,
proc_map_timeout);
}
err = 0;
closedir(proc);
+out_free_namespaces:
+ free(namespaces_event);
out_free_fork:
free(fork_event);
out_free_mmap:
@@ -1008,6 +1116,33 @@
return fprintf(fp, "%s: %s:%d/%d\n", s, event->comm.comm, event->comm.pid, event->comm.tid);
}
+size_t perf_event__fprintf_namespaces(union perf_event *event, FILE *fp)
+{
+ size_t ret = 0;
+ struct perf_ns_link_info *ns_link_info;
+ u32 nr_namespaces, idx;
+
+ ns_link_info = event->namespaces.link_info;
+ nr_namespaces = event->namespaces.nr_namespaces;
+
+ ret += fprintf(fp, " %d/%d - nr_namespaces: %u\n\t\t[",
+ event->namespaces.pid,
+ event->namespaces.tid,
+ nr_namespaces);
+
+ for (idx = 0; idx < nr_namespaces; idx++) {
+ if (idx && (idx % 4 == 0))
+ ret += fprintf(fp, "\n\t\t ");
+
+ ret += fprintf(fp, "%u/%s: %" PRIu64 "/%#" PRIx64 "%s", idx,
+ perf_ns__name(idx), (u64)ns_link_info[idx].dev,
+ (u64)ns_link_info[idx].ino,
+ ((idx + 1) != nr_namespaces) ? ", " : "]\n");
+ }
+
+ return ret;
+}
+
int perf_event__process_comm(struct perf_tool *tool __maybe_unused,
union perf_event *event,
struct perf_sample *sample,
@@ -1016,6 +1151,14 @@
return machine__process_comm_event(machine, event, sample);
}
+int perf_event__process_namespaces(struct perf_tool *tool __maybe_unused,
+ union perf_event *event,
+ struct perf_sample *sample,
+ struct machine *machine)
+{
+ return machine__process_namespaces_event(machine, event, sample);
+}
+
int perf_event__process_lost(struct perf_tool *tool __maybe_unused,
union perf_event *event,
struct perf_sample *sample,
@@ -1153,11 +1296,12 @@
size_t perf_event__fprintf_aux(union perf_event *event, FILE *fp)
{
- return fprintf(fp, " offset: %#"PRIx64" size: %#"PRIx64" flags: %#"PRIx64" [%s%s]\n",
+ return fprintf(fp, " offset: %#"PRIx64" size: %#"PRIx64" flags: %#"PRIx64" [%s%s%s]\n",
event->aux.aux_offset, event->aux.aux_size,
event->aux.flags,
event->aux.flags & PERF_AUX_FLAG_TRUNCATED ? "T" : "",
- event->aux.flags & PERF_AUX_FLAG_OVERWRITE ? "O" : "");
+ event->aux.flags & PERF_AUX_FLAG_OVERWRITE ? "O" : "",
+ event->aux.flags & PERF_AUX_FLAG_PARTIAL ? "P" : "");
}
size_t perf_event__fprintf_itrace_start(union perf_event *event, FILE *fp)
@@ -1196,6 +1340,9 @@
case PERF_RECORD_MMAP:
ret += perf_event__fprintf_mmap(event, fp);
break;
+ case PERF_RECORD_NAMESPACES:
+ ret += perf_event__fprintf_namespaces(event, fp);
+ break;
case PERF_RECORD_MMAP2:
ret += perf_event__fprintf_mmap2(event, fp);
break;
diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
index c735c53..db2de64 100644
--- a/tools/perf/util/event.h
+++ b/tools/perf/util/event.h
@@ -3,9 +3,9 @@
#include <limits.h>
#include <stdio.h>
+#include <linux/kernel.h>
#include "../perf.h"
-#include "map.h"
#include "build-id.h"
#include "perf_regs.h"
@@ -39,6 +39,13 @@
char comm[16];
};
+struct namespaces_event {
+ struct perf_event_header header;
+ u32 pid, tid;
+ u64 nr_namespaces;
+ struct perf_ns_link_info link_info[];
+};
+
struct fork_event {
struct perf_event_header header;
u32 pid, ppid;
@@ -269,6 +276,7 @@
u64 total_lost;
u64 total_lost_samples;
u64 total_aux_lost;
+ u64 total_aux_partial;
u64 total_invalid_chains;
u32 nr_events[PERF_RECORD_HEADER_MAX];
u32 nr_non_filtered_samples;
@@ -485,6 +493,7 @@
struct mmap_event mmap;
struct mmap2_event mmap2;
struct comm_event comm;
+ struct namespaces_event namespaces;
struct fork_event fork;
struct lost_event lost;
struct lost_samples_event lost_samples;
@@ -587,6 +596,10 @@
union perf_event *event,
struct perf_sample *sample,
struct machine *machine);
+int perf_event__process_namespaces(struct perf_tool *tool,
+ union perf_event *event,
+ struct perf_sample *sample,
+ struct machine *machine);
int perf_event__process_mmap(struct perf_tool *tool,
union perf_event *event,
struct perf_sample *sample,
@@ -636,6 +649,12 @@
perf_event__handler_t process,
struct machine *machine);
+int perf_event__synthesize_namespaces(struct perf_tool *tool,
+ union perf_event *event,
+ pid_t pid, pid_t tgid,
+ perf_event__handler_t process,
+ struct machine *machine);
+
int perf_event__synthesize_mmap_events(struct perf_tool *tool,
union perf_event *event,
pid_t pid, pid_t tgid,
@@ -653,6 +672,7 @@
size_t perf_event__fprintf_switch(union perf_event *event, FILE *fp);
size_t perf_event__fprintf_thread_map(union perf_event *event, FILE *fp);
size_t perf_event__fprintf_cpu_map(union perf_event *event, FILE *fp);
+size_t perf_event__fprintf_namespaces(union perf_event *event, FILE *fp);
size_t perf_event__fprintf(union perf_event *event, FILE *fp);
u64 kallsyms__get_function_start(const char *kallsyms_filename,
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index b601f28..46c0faf 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -8,6 +8,8 @@
*/
#include "util.h"
#include <api/fs/fs.h>
+#include <errno.h>
+#include <inttypes.h>
#include <poll.h>
#include "cpumap.h"
#include "thread_map.h"
@@ -15,12 +17,15 @@
#include "evlist.h"
#include "evsel.h"
#include "debug.h"
+#include "units.h"
#include "asm/bug.h"
+#include <signal.h>
#include <unistd.h>
#include "parse-events.h"
#include <subcmd/parse-options.h>
+#include <sys/ioctl.h>
#include <sys/mman.h>
#include <linux/bitops.h>
@@ -777,7 +782,7 @@
/*
* Check if event was unmapped due to a POLLHUP/POLLERR.
*/
- if (!atomic_read(&md->refcnt))
+ if (!refcount_read(&md->refcnt))
return NULL;
head = perf_mmap__read_head(md);
@@ -794,7 +799,7 @@
/*
* Check if event was unmapped due to a POLLHUP/POLLERR.
*/
- if (!atomic_read(&md->refcnt))
+ if (!refcount_read(&md->refcnt))
return NULL;
head = perf_mmap__read_head(md);
@@ -856,7 +861,7 @@
{
u64 head;
- if (!atomic_read(&md->refcnt))
+ if (!refcount_read(&md->refcnt))
return;
head = perf_mmap__read_head(md);
@@ -875,14 +880,14 @@
static void perf_mmap__get(struct perf_mmap *map)
{
- atomic_inc(&map->refcnt);
+ refcount_inc(&map->refcnt);
}
static void perf_mmap__put(struct perf_mmap *md)
{
- BUG_ON(md->base && atomic_read(&md->refcnt) == 0);
+ BUG_ON(md->base && refcount_read(&md->refcnt) == 0);
- if (atomic_dec_and_test(&md->refcnt))
+ if (refcount_dec_and_test(&md->refcnt))
perf_mmap__munmap(md);
}
@@ -894,7 +899,7 @@
perf_mmap__write_tail(md, old);
}
- if (atomic_read(&md->refcnt) == 1 && perf_mmap__empty(md))
+ if (refcount_read(&md->refcnt) == 1 && perf_mmap__empty(md))
perf_mmap__put(md);
}
@@ -937,7 +942,7 @@
munmap(map->base, perf_mmap__mmap_len(map));
map->base = NULL;
map->fd = -1;
- atomic_set(&map->refcnt, 0);
+ refcount_set(&map->refcnt, 0);
}
auxtrace_mmap__munmap(&map->auxtrace_mmap);
}
@@ -974,8 +979,19 @@
if (!map)
return NULL;
- for (i = 0; i < evlist->nr_mmaps; i++)
+ for (i = 0; i < evlist->nr_mmaps; i++) {
map[i].fd = -1;
+ /*
+ * When the perf_mmap() call is made we grab one refcount, plus
+ * one extra to let perf_evlist__mmap_consume() get the last
+ * events after all real references (perf_mmap__get()) are
+ * dropped.
+ *
+ * Each PERF_EVENT_IOC_SET_OUTPUT points to this mmap and
+ * thus does perf_mmap__get() on it.
+ */
+ refcount_set(&map[i].refcnt, 0);
+ }
return map;
}
@@ -1001,7 +1017,7 @@
* evlist layer can't just drop it when filtering events in
* perf_evlist__filter_pollfd().
*/
- atomic_set(&map->refcnt, 2);
+ refcount_set(&map->refcnt, 2);
map->prev = 0;
map->mask = mp->mask;
map->base = mmap(NULL, perf_mmap__mmap_len(map), mp->prot,
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index 389b9cc..94cea43 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -1,7 +1,8 @@
#ifndef __PERF_EVLIST_H
#define __PERF_EVLIST_H 1
-#include <linux/atomic.h>
+#include <linux/kernel.h>
+#include <linux/refcount.h>
#include <linux/list.h>
#include <api/fd/array.h>
#include <stdio.h>
@@ -10,6 +11,7 @@
#include "evsel.h"
#include "util.h"
#include "auxtrace.h"
+#include <signal.h>
#include <unistd.h>
struct pollfd;
@@ -29,7 +31,7 @@
void *base;
int mask;
int fd;
- atomic_t refcnt;
+ refcount_t refcnt;
u64 prev;
struct auxtrace_mmap auxtrace_mmap;
char event_copy[PERF_SAMPLE_MAX_SIZE] __attribute__((aligned(8)));
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index ac59710..0e87909 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -8,12 +8,15 @@
*/
#include <byteswap.h>
+#include <errno.h>
+#include <inttypes.h>
#include <linux/bitops.h>
#include <api/fs/tracing_path.h>
#include <traceevent/event-parse.h>
#include <linux/hw_breakpoint.h>
#include <linux/perf_event.h>
#include <linux/err.h>
+#include <sys/ioctl.h>
#include <sys/resource.h>
#include "asm/bug.h"
#include "callchain.h"
@@ -30,6 +33,8 @@
#include "stat.h"
#include "util/parse-branch-options.h"
+#include "sane_ctype.h"
+
static struct {
bool sample_id_all;
bool exclude_guest;
@@ -236,6 +241,10 @@
evsel->sample_size = __perf_evsel__sample_size(attr->sample_type);
perf_evsel__calc_id_pos(evsel);
evsel->cmdline_group_boundary = false;
+ evsel->metric_expr = NULL;
+ evsel->metric_name = NULL;
+ evsel->metric_events = NULL;
+ evsel->collect_stat = false;
}
struct perf_evsel *perf_evsel__new_idx(struct perf_event_attr *attr, int idx)
@@ -932,6 +941,9 @@
attr->mmap2 = track && !perf_missing_features.mmap2;
attr->comm = track;
+ if (opts->record_namespaces)
+ attr->namespaces = track;
+
if (opts->record_switch_events)
attr->context_switch = track;
@@ -1232,7 +1244,7 @@
if (FD(evsel, cpu, thread) < 0)
return -EINVAL;
- if (readn(FD(evsel, cpu, thread), count, sizeof(*count)) < 0)
+ if (readn(FD(evsel, cpu, thread), count, sizeof(*count)) <= 0)
return -errno;
return 0;
@@ -1250,7 +1262,7 @@
if (evsel->counts == NULL && perf_evsel__alloc_counts(evsel, cpu + 1, thread + 1) < 0)
return -ENOMEM;
- if (readn(FD(evsel, cpu, thread), &count, nv * sizeof(u64)) < 0)
+ if (readn(FD(evsel, cpu, thread), &count, nv * sizeof(u64)) <= 0)
return -errno;
perf_evsel__compute_deltas(evsel, cpu, thread, &count);
@@ -2450,11 +2462,17 @@
int err, char *msg, size_t size)
{
char sbuf[STRERR_BUFSIZE];
+ int printed = 0;
switch (err) {
case EPERM:
case EACCES:
- return scnprintf(msg, size,
+ if (err == EPERM)
+ printed = scnprintf(msg, size,
+ "No permission to enable %s event.\n\n",
+ perf_evsel__name(evsel));
+
+ return scnprintf(msg + printed, size - printed,
"You may not have permission to collect %sstats.\n\n"
"Consider tweaking /proc/sys/kernel/perf_event_paranoid,\n"
"which controls use of the performance events system by\n"
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index 06ef6f2..d101695 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -131,6 +131,11 @@
bool cmdline_group_boundary;
struct list_head config_terms;
int bpf_fd;
+ bool merged_stat;
+ const char * metric_expr;
+ const char * metric_name;
+ struct perf_evsel **metric_events;
+ bool collect_stat;
};
union u64_swap {
diff --git a/tools/perf/util/evsel_fprintf.c b/tools/perf/util/evsel_fprintf.c
index 4ef5184..e415aee 100644
--- a/tools/perf/util/evsel_fprintf.c
+++ b/tools/perf/util/evsel_fprintf.c
@@ -1,9 +1,11 @@
+#include <inttypes.h>
#include <stdio.h>
#include <stdbool.h>
#include <traceevent/event-parse.h>
#include "evsel.h"
#include "callchain.h"
#include "map.h"
+#include "strlist.h"
#include "symbol.h"
static int comma_fprintf(FILE *fp, bool *first, const char *fmt, ...)
diff --git a/tools/perf/util/expr.h b/tools/perf/util/expr.h
new file mode 100644
index 0000000..9c2760a
--- /dev/null
+++ b/tools/perf/util/expr.h
@@ -0,0 +1,25 @@
+#ifndef PARSE_CTX_H
+#define PARSE_CTX_H 1
+
+#define EXPR_MAX_OTHER 8
+#define MAX_PARSE_ID EXPR_MAX_OTHER
+
+struct parse_id {
+ const char *name;
+ double val;
+};
+
+struct parse_ctx {
+ int num_ids;
+ struct parse_id ids[MAX_PARSE_ID];
+};
+
+void expr__ctx_init(struct parse_ctx *ctx);
+void expr__add_id(struct parse_ctx *ctx, const char *id, double val);
+#ifndef IN_EXPR_Y
+int expr__parse(double *final_val, struct parse_ctx *ctx, const char **pp);
+#endif
+int expr__find_other(const char *p, const char *one, const char ***other,
+ int *num_other);
+
+#endif
diff --git a/tools/perf/util/expr.y b/tools/perf/util/expr.y
new file mode 100644
index 0000000..954556b
--- /dev/null
+++ b/tools/perf/util/expr.y
@@ -0,0 +1,173 @@
+/* Simple expression parser */
+%{
+#include "util.h"
+#include "util/debug.h"
+#define IN_EXPR_Y 1
+#include "expr.h"
+#include <string.h>
+
+#define MAXIDLEN 256
+%}
+
+%pure-parser
+%parse-param { double *final_val }
+%parse-param { struct parse_ctx *ctx }
+%parse-param { const char **pp }
+%lex-param { const char **pp }
+
+%union {
+ double num;
+ char id[MAXIDLEN+1];
+}
+
+%token <num> NUMBER
+%token <id> ID
+%left '|'
+%left '^'
+%left '&'
+%left '-' '+'
+%left '*' '/' '%'
+%left NEG NOT
+%type <num> expr
+
+%{
+static int expr__lex(YYSTYPE *res, const char **pp);
+
+static void expr__error(double *final_val __maybe_unused,
+ struct parse_ctx *ctx __maybe_unused,
+ const char **pp __maybe_unused,
+ const char *s)
+{
+ pr_debug("%s\n", s);
+}
+
+static int lookup_id(struct parse_ctx *ctx, char *id, double *val)
+{
+ int i;
+
+ for (i = 0; i < ctx->num_ids; i++) {
+ if (!strcasecmp(ctx->ids[i].name, id)) {
+ *val = ctx->ids[i].val;
+ return 0;
+ }
+ }
+ return -1;
+}
+
+%}
+%%
+
+all_expr: expr { *final_val = $1; }
+ ;
+
+expr: NUMBER
+ | ID { if (lookup_id(ctx, $1, &$$) < 0) {
+ pr_debug("%s not found", $1);
+ YYABORT;
+ }
+ }
+ | expr '+' expr { $$ = $1 + $3; }
+ | expr '-' expr { $$ = $1 - $3; }
+ | expr '*' expr { $$ = $1 * $3; }
+ | expr '/' expr { if ($3 == 0) YYABORT; $$ = $1 / $3; }
+ | expr '%' expr { if ((long)$3 == 0) YYABORT; $$ = (long)$1 % (long)$3; }
+ | '-' expr %prec NEG { $$ = -$2; }
+ | '(' expr ')' { $$ = $2; }
+ ;
+
+%%
+
+static int expr__symbol(YYSTYPE *res, const char *p, const char **pp)
+{
+ char *dst = res->id;
+ const char *s = p;
+
+ while (isalnum(*p) || *p == '_' || *p == '.') {
+ if (p - s >= MAXIDLEN)
+ return -1;
+ *dst++ = *p++;
+ }
+ *dst = 0;
+ *pp = p;
+ return ID;
+}
+
+static int expr__lex(YYSTYPE *res, const char **pp)
+{
+ int tok;
+ const char *s;
+ const char *p = *pp;
+
+ while (isspace(*p))
+ p++;
+ s = p;
+ switch (*p++) {
+ case 'a' ... 'z':
+ case 'A' ... 'Z':
+ return expr__symbol(res, p - 1, pp);
+ case '0' ... '9': case '.':
+ res->num = strtod(s, (char **)&p);
+ tok = NUMBER;
+ break;
+ default:
+ tok = *s;
+ break;
+ }
+ *pp = p;
+ return tok;
+}
+
+/* Caller must make sure id is allocated */
+void expr__add_id(struct parse_ctx *ctx, const char *name, double val)
+{
+ int idx;
+ assert(ctx->num_ids < MAX_PARSE_ID);
+ idx = ctx->num_ids++;
+ ctx->ids[idx].name = name;
+ ctx->ids[idx].val = val;
+}
+
+void expr__ctx_init(struct parse_ctx *ctx)
+{
+ ctx->num_ids = 0;
+}
+
+int expr__find_other(const char *p, const char *one, const char ***other,
+ int *num_otherp)
+{
+ const char *orig = p;
+ int err = -1;
+ int num_other;
+
+ *other = malloc((EXPR_MAX_OTHER + 1) * sizeof(char *));
+ if (!*other)
+ return -1;
+
+ num_other = 0;
+ for (;;) {
+ YYSTYPE val;
+ int tok = expr__lex(&val, &p);
+ if (tok == 0) {
+ err = 0;
+ break;
+ }
+ if (tok == ID && strcasecmp(one, val.id)) {
+ if (num_other >= EXPR_MAX_OTHER - 1) {
+ pr_debug("Too many extra events in %s\n", orig);
+ break;
+ }
+ (*other)[num_other] = strdup(val.id);
+ if (!(*other)[num_other])
+ return -1;
+ num_other++;
+ }
+ }
+ (*other)[num_other] = NULL;
+ *num_otherp = num_other;
+ if (err) {
+ *num_otherp = 0;
+ free(*other);
+ *other = NULL;
+ }
+ return err;
+}
diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 05714d5..948b2c5 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -1,4 +1,8 @@
+#include <errno.h>
+#include <inttypes.h>
#include "util.h"
+#include "string2.h"
+#include <sys/param.h>
#include <sys/types.h>
#include <byteswap.h>
#include <unistd.h>
@@ -7,7 +11,10 @@
#include <linux/list.h>
#include <linux/kernel.h>
#include <linux/bitops.h>
+#include <sys/stat.h>
+#include <sys/types.h>
#include <sys/utsname.h>
+#include <unistd.h>
#include "evlist.h"
#include "evsel.h"
@@ -26,6 +33,8 @@
#include <api/fs/fs.h>
#include "asm/bug.h"
+#include "sane_ctype.h"
+
/*
* magic2 = "PERFILE2"
* must be a numerical value to let the endianness
@@ -370,15 +379,11 @@
struct perf_evlist *evlist __maybe_unused)
{
char buf[MAXPATHLEN];
- char proc[32];
u32 n;
int i, ret;
- /*
- * actual atual path to perf binary
- */
- sprintf(proc, "/proc/%d/exe", getpid());
- ret = readlink(proc, buf, sizeof(buf));
+ /* actual path to perf binary */
+ ret = readlink("/proc/self/exe", buf, sizeof(buf) - 1);
if (ret <= 0)
return -1;
@@ -2274,6 +2279,9 @@
perf_header__process_sections(header, fd, &hd,
perf_file_section__fprintf_info);
+ if (session->file->is_pipe)
+ return 0;
+
fprintf(fp, "# missing features: ");
for_each_clear_bit(bit, header->adds_features, HEADER_LAST_FEATURE) {
if (bit)
diff --git a/tools/perf/util/help-unknown-cmd.c b/tools/perf/util/help-unknown-cmd.c
index 2821f8d..1c88ad6 100644
--- a/tools/perf/util/help-unknown-cmd.c
+++ b/tools/perf/util/help-unknown-cmd.c
@@ -1,21 +1,18 @@
#include "cache.h"
#include "config.h"
+#include <poll.h>
#include <stdio.h>
#include <subcmd/help.h>
#include "../builtin.h"
#include "levenshtein.h"
static int autocorrect;
-static struct cmdnames aliases;
static int perf_unknown_cmd_config(const char *var, const char *value,
void *cb __maybe_unused)
{
if (!strcmp(var, "help.autocorrect"))
autocorrect = perf_config_int(var,value);
- /* Also use aliases for command lookup */
- if (!prefixcmp(var, "alias."))
- add_cmdname(&aliases, var + 6, strlen(var + 6));
return 0;
}
@@ -59,14 +56,12 @@
memset(&main_cmds, 0, sizeof(main_cmds));
memset(&other_cmds, 0, sizeof(main_cmds));
- memset(&aliases, 0, sizeof(aliases));
perf_config(perf_unknown_cmd_config, NULL);
load_command_list("perf-", &main_cmds, &other_cmds);
- if (add_cmd_list(&main_cmds, &aliases) < 0 ||
- add_cmd_list(&main_cmds, &other_cmds) < 0) {
+ if (add_cmd_list(&main_cmds, &other_cmds) < 0) {
fprintf(stderr, "ERROR: Failed to allocate command list for unknown command.\n");
goto end;
}
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index eaf72a9..cf0186a 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -3,12 +3,17 @@
#include "hist.h"
#include "map.h"
#include "session.h"
+#include "namespaces.h"
#include "sort.h"
#include "evlist.h"
#include "evsel.h"
#include "annotate.h"
+#include "srcline.h"
+#include "thread.h"
#include "ui/progress.h"
+#include <errno.h>
#include <math.h>
+#include <sys/param.h>
static bool hists__filter_entry_by_dso(struct hists *hists,
struct hist_entry *he);
@@ -169,6 +174,7 @@
hists__set_unres_dso_col_len(hists, HISTC_MEM_DADDR_DSO);
}
+ hists__new_col_len(hists, HISTC_CGROUP_ID, 20);
hists__new_col_len(hists, HISTC_CPU, 3);
hists__new_col_len(hists, HISTC_SOCKET, 6);
hists__new_col_len(hists, HISTC_MEM_LOCKED, 6);
@@ -574,9 +580,14 @@
bool sample_self,
struct hist_entry_ops *ops)
{
+ struct namespaces *ns = thread__namespaces(al->thread);
struct hist_entry entry = {
.thread = al->thread,
.comm = thread__comm(al->thread),
+ .cgroup_id = {
+ .dev = ns ? ns->link_info[CGROUP_NS_INDEX].dev : 0,
+ .ino = ns ? ns->link_info[CGROUP_NS_INDEX].ino : 0,
+ },
.ms = {
.map = al->map,
.sym = al->sym,
@@ -1129,6 +1140,11 @@
zfree(&he->mem_info);
}
+ if (he->inline_node) {
+ inline_node__delete(he->inline_node);
+ he->inline_node = NULL;
+ }
+
zfree(&he->stat_acc);
free_srcline(he->srcline);
if (he->srcfile && he->srcfile[0])
@@ -2447,7 +2463,7 @@
else if (!strcmp(arg, "absolute"))
symbol_conf.filter_relative = false;
else {
- pr_debug("Invalud percentage: %s\n", arg);
+ pr_debug("Invalid percentage: %s\n", arg);
return -1;
}
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 28c216e..ee3670a 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -30,6 +30,7 @@
HISTC_DSO,
HISTC_THREAD,
HISTC_COMM,
+ HISTC_CGROUP_ID,
HISTC_PARENT,
HISTC_CPU,
HISTC_SOCKET,
@@ -57,6 +58,7 @@
HISTC_SRCLINE_FROM,
HISTC_SRCLINE_TO,
HISTC_TRACE,
+ HISTC_SYM_SIZE,
HISTC_NR_COLS, /* Last entry */
};
diff --git a/tools/perf/util/intel-bts.c b/tools/perf/util/intel-bts.c
index 6c2eb5d..b2834ac 100644
--- a/tools/perf/util/intel-bts.c
+++ b/tools/perf/util/intel-bts.c
@@ -14,7 +14,9 @@
*/
#include <endian.h>
+#include <errno.h>
#include <byteswap.h>
+#include <inttypes.h>
#include <linux/kernel.h>
#include <linux/types.h>
#include <linux/bitops.h>
diff --git a/tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.c b/tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.c
index 4f3c758..5481882 100644
--- a/tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.c
+++ b/tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.c
@@ -26,6 +26,7 @@
#include "insn.c"
#include "intel-pt-insn-decoder.h"
+#include "dump-insn.h"
#if INTEL_PT_INSN_BUF_SZ < MAX_INSN_SIZE || INTEL_PT_INSN_BUF_SZ > MAX_INSN
#error Instruction buffer size too small
@@ -39,6 +40,8 @@
enum intel_pt_insn_branch branch = INTEL_PT_BR_NO_BRANCH;
int ext;
+ intel_pt_insn->rel = 0;
+
if (insn_is_avx(insn)) {
intel_pt_insn->op = INTEL_PT_OP_OTHER;
intel_pt_insn->branch = INTEL_PT_BR_NO_BRANCH;
@@ -177,6 +180,29 @@
return 0;
}
+const char *dump_insn(struct perf_insn *x, uint64_t ip __maybe_unused,
+ u8 *inbuf, int inlen, int *lenp)
+{
+ struct insn insn;
+ int n, i;
+ int left;
+
+ insn_init(&insn, inbuf, inlen, x->is64bit);
+ insn_get_length(&insn);
+ if (!insn_complete(&insn) || insn.length > inlen)
+ return "<bad>";
+ if (lenp)
+ *lenp = insn.length;
+ left = sizeof(x->out);
+ n = snprintf(x->out, left, "insn: ");
+ left -= n;
+ for (i = 0; i < insn.length; i++) {
+ n += snprintf(x->out + n, left, "%02x ", inbuf[i]);
+ left -= n;
+ }
+ return x->out;
+}
+
const char *branch_name[] = {
[INTEL_PT_OP_OTHER] = "Other",
[INTEL_PT_OP_CALL] = "Call",
diff --git a/tools/perf/util/intel-pt.c b/tools/perf/util/intel-pt.c
index da20cd5..bdd4a28 100644
--- a/tools/perf/util/intel-pt.c
+++ b/tools/perf/util/intel-pt.c
@@ -13,6 +13,7 @@
*
*/
+#include <inttypes.h>
#include <stdio.h>
#include <stdbool.h>
#include <errno.h>
diff --git a/tools/perf/util/jitdump.c b/tools/perf/util/jitdump.c
index c9a941e..9084930 100644
--- a/tools/perf/util/jitdump.c
+++ b/tools/perf/util/jitdump.c
@@ -1,5 +1,6 @@
#include <sys/sysmacros.h>
#include <sys/types.h>
+#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -9,13 +10,13 @@
#include <byteswap.h>
#include <sys/stat.h>
#include <sys/mman.h>
+#include <linux/stringify.h>
#include "util.h"
#include "event.h"
#include "debug.h"
#include "evlist.h"
#include "symbol.h"
-#include "strlist.h"
#include <elf.h>
#include "tsc.h"
@@ -25,6 +26,8 @@
#include "genelf.h"
#include "../builtin.h"
+#include "sane_ctype.h"
+
struct jit_buf_desc {
struct perf_data_file *output;
struct perf_session *session;
@@ -181,7 +184,7 @@
jd->use_arch_timestamp);
if (header.version > JITHEADER_VERSION) {
- pr_err("wrong jitdump version %u, expected " STR(JITHEADER_VERSION),
+ pr_err("wrong jitdump version %u, expected " __stringify(JITHEADER_VERSION),
header.version);
goto error;
}
diff --git a/tools/perf/util/llvm-utils.c b/tools/perf/util/llvm-utils.c
index 8243564..c6a15f2 100644
--- a/tools/perf/util/llvm-utils.c
+++ b/tools/perf/util/llvm-utils.c
@@ -12,6 +12,7 @@
#include "llvm-utils.h"
#include "config.h"
#include "util.h"
+#include <sys/wait.h>
#define CLANG_BPF_CMD_DEFAULT_TEMPLATE \
"$CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS "\
diff --git a/tools/perf/util/lzma.c b/tools/perf/util/lzma.c
index 9ddea5c..4ca7c5c 100644
--- a/tools/perf/util/lzma.c
+++ b/tools/perf/util/lzma.c
@@ -1,6 +1,8 @@
+#include <errno.h>
#include <lzma.h>
#include <stdio.h>
#include <linux/compiler.h>
+#include "compress.h"
#include "util.h"
#include "debug.h"
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 71c9720..7a47f52 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -1,3 +1,7 @@
+#include <dirent.h>
+#include <errno.h>
+#include <inttypes.h>
+#include <regex.h>
#include "callchain.h"
#include "debug.h"
#include "event.h"
@@ -10,9 +14,15 @@
#include "thread.h"
#include "vdso.h"
#include <stdbool.h>
-#include <symbol/kallsyms.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
#include "unwind.h"
#include "linux/hash.h"
+#include "asm/bug.h"
+
+#include "sane_ctype.h"
+#include <symbol/kallsyms.h>
static void __machine__remove_thread(struct machine *machine, struct thread *th, bool lock);
@@ -501,6 +511,37 @@
return err;
}
+int machine__process_namespaces_event(struct machine *machine __maybe_unused,
+ union perf_event *event,
+ struct perf_sample *sample __maybe_unused)
+{
+ struct thread *thread = machine__findnew_thread(machine,
+ event->namespaces.pid,
+ event->namespaces.tid);
+ int err = 0;
+
+ WARN_ONCE(event->namespaces.nr_namespaces > NR_NAMESPACES,
+ "\nWARNING: kernel seems to support more namespaces than perf"
+ " tool.\nTry updating the perf tool..\n\n");
+
+ WARN_ONCE(event->namespaces.nr_namespaces < NR_NAMESPACES,
+ "\nWARNING: perf tool seems to support more namespaces than"
+ " the kernel.\nTry updating the kernel..\n\n");
+
+ if (dump_trace)
+ perf_event__fprintf_namespaces(event, stdout);
+
+ if (thread == NULL ||
+ thread__set_namespaces(thread, sample->time, &event->namespaces)) {
+ dump_printf("problem processing PERF_RECORD_NAMESPACES, skipping event.\n");
+ err = -1;
+ }
+
+ thread__put(thread);
+
+ return err;
+}
+
int machine__process_lost_event(struct machine *machine __maybe_unused,
union perf_event *event, struct perf_sample *sample __maybe_unused)
{
@@ -1439,7 +1480,7 @@
if (machine->last_match == th)
machine->last_match = NULL;
- BUG_ON(atomic_read(&th->refcnt) == 0);
+ BUG_ON(refcount_read(&th->refcnt) == 0);
if (lock)
pthread_rwlock_wrlock(&machine->threads_lock);
rb_erase_init(&th->rb_node, &machine->threads);
@@ -1538,6 +1579,8 @@
ret = machine__process_comm_event(machine, event, sample); break;
case PERF_RECORD_MMAP:
ret = machine__process_mmap_event(machine, event, sample); break;
+ case PERF_RECORD_NAMESPACES:
+ ret = machine__process_namespaces_event(machine, event, sample); break;
case PERF_RECORD_MMAP2:
ret = machine__process_mmap2_event(machine, event, sample); break;
case PERF_RECORD_FORK:
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index a283050..3cdb134 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -97,6 +97,9 @@
union perf_event *event);
int machine__process_switch_event(struct machine *machine,
union perf_event *event);
+int machine__process_namespaces_event(struct machine *machine,
+ union perf_event *event,
+ struct perf_sample *sample);
int machine__process_mmap_event(struct machine *machine, union perf_event *event,
struct perf_sample *sample);
int machine__process_mmap2_event(struct machine *machine, union perf_event *event,
diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c
index 0a943e7..ebfa5d9 100644
--- a/tools/perf/util/map.c
+++ b/tools/perf/util/map.c
@@ -9,13 +9,13 @@
#include <uapi/linux/mman.h> /* To get things like MAP_HUGETLB even on older libc headers */
#include "map.h"
#include "thread.h"
-#include "strlist.h"
#include "vdso.h"
#include "build-id.h"
#include "util.h"
#include "debug.h"
#include "machine.h"
#include <linux/string.h>
+#include "srcline.h"
#include "unwind.h"
static void __maps__insert(struct maps *maps, struct map *map);
@@ -141,7 +141,7 @@
RB_CLEAR_NODE(&map->rb_node);
map->groups = NULL;
map->erange_warned = false;
- atomic_set(&map->refcnt, 1);
+ refcount_set(&map->refcnt, 1);
}
struct map *map__new(struct machine *machine, u64 start, u64 len,
@@ -255,7 +255,7 @@
void map__put(struct map *map)
{
- if (map && atomic_dec_and_test(&map->refcnt))
+ if (map && refcount_dec_and_test(&map->refcnt))
map__delete(map);
}
@@ -354,7 +354,7 @@
struct map *map = memdup(from, sizeof(*map));
if (map != NULL) {
- atomic_set(&map->refcnt, 1);
+ refcount_set(&map->refcnt, 1);
RB_CLEAR_NODE(&map->rb_node);
dso__get(map->dso);
map->groups = NULL;
@@ -405,7 +405,8 @@
if (map && map->dso) {
srcline = get_srcline(map->dso,
- map__rip_2objdump(map, addr), NULL, true);
+ map__rip_2objdump(map, addr), NULL,
+ true, true);
if (srcline != SRCLINE_UNKNOWN)
ret = fprintf(fp, "%s%s", prefix, srcline);
free_srcline(srcline);
@@ -485,7 +486,7 @@
maps__init(&mg->maps[i]);
}
mg->machine = machine;
- atomic_set(&mg->refcnt, 1);
+ refcount_set(&mg->refcnt, 1);
}
static void __maps__purge(struct maps *maps)
@@ -547,7 +548,7 @@
void map_groups__put(struct map_groups *mg)
{
- if (mg && atomic_dec_and_test(&mg->refcnt))
+ if (mg && refcount_dec_and_test(&mg->refcnt))
map_groups__delete(mg);
}
diff --git a/tools/perf/util/map.h b/tools/perf/util/map.h
index abdacf8..c8a5a64 100644
--- a/tools/perf/util/map.h
+++ b/tools/perf/util/map.h
@@ -1,7 +1,7 @@
#ifndef __PERF_MAP_H
#define __PERF_MAP_H
-#include <linux/atomic.h>
+#include <linux/refcount.h>
#include <linux/compiler.h>
#include <linux/list.h>
#include <linux/rbtree.h>
@@ -51,7 +51,7 @@
struct dso *dso;
struct map_groups *groups;
- atomic_t refcnt;
+ refcount_t refcnt;
};
struct kmap {
@@ -67,7 +67,7 @@
struct map_groups {
struct maps maps[MAP__NR_TYPES];
struct machine *machine;
- atomic_t refcnt;
+ refcount_t refcnt;
};
struct map_groups *map_groups__new(struct machine *machine);
@@ -77,7 +77,7 @@
static inline struct map_groups *map_groups__get(struct map_groups *mg)
{
if (mg)
- atomic_inc(&mg->refcnt);
+ refcount_inc(&mg->refcnt);
return mg;
}
@@ -150,7 +150,7 @@
static inline struct map *map__get(struct map *map)
{
if (map)
- atomic_inc(&map->refcnt);
+ refcount_inc(&map->refcnt);
return map;
}
diff --git a/tools/perf/util/mem-events.c b/tools/perf/util/mem-events.c
index 1d4ab53..06f5a3a 100644
--- a/tools/perf/util/mem-events.c
+++ b/tools/perf/util/mem-events.c
@@ -6,6 +6,7 @@
#include <sys/stat.h>
#include <unistd.h>
#include <api/fs/fs.h>
+#include <linux/kernel.h>
#include "mem-events.h"
#include "debug.h"
#include "symbol.h"
@@ -205,8 +206,8 @@
static const char * const snoop_access[] = {
"N/A",
"None",
- "Miss",
"Hit",
+ "Miss",
"HitM",
};
diff --git a/tools/perf/util/namespaces.c b/tools/perf/util/namespaces.c
new file mode 100644
index 0000000..67dcbcc
--- /dev/null
+++ b/tools/perf/util/namespaces.c
@@ -0,0 +1,37 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * Copyright (C) 2017 Hari Bathini, IBM Corporation
+ */
+
+#include "namespaces.h"
+#include "util.h"
+#include "event.h"
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+
+struct namespaces *namespaces__new(struct namespaces_event *event)
+{
+ struct namespaces *namespaces;
+ u64 link_info_size = ((event ? event->nr_namespaces : NR_NAMESPACES) *
+ sizeof(struct perf_ns_link_info));
+
+ namespaces = zalloc(sizeof(struct namespaces) + link_info_size);
+ if (!namespaces)
+ return NULL;
+
+ namespaces->end_time = -1;
+
+ if (event)
+ memcpy(namespaces->link_info, event->link_info, link_info_size);
+
+ return namespaces;
+}
+
+void namespaces__free(struct namespaces *namespaces)
+{
+ free(namespaces);
+}
diff --git a/tools/perf/util/namespaces.h b/tools/perf/util/namespaces.h
new file mode 100644
index 0000000..468f1e9
--- /dev/null
+++ b/tools/perf/util/namespaces.h
@@ -0,0 +1,26 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * Copyright (C) 2017 Hari Bathini, IBM Corporation
+ */
+
+#ifndef __PERF_NAMESPACES_H
+#define __PERF_NAMESPACES_H
+
+#include "../perf.h"
+#include <linux/list.h>
+
+struct namespaces_event;
+
+struct namespaces {
+ struct list_head list;
+ u64 end_time;
+ struct perf_ns_link_info link_info[];
+};
+
+struct namespaces *namespaces__new(struct namespaces_event *event);
+void namespaces__free(struct namespaces *namespaces);
+
+#endif /* __PERF_NAMESPACES_H */
diff --git a/tools/perf/util/ordered-events.c b/tools/perf/util/ordered-events.c
index fe84df1..4de398c 100644
--- a/tools/perf/util/ordered-events.c
+++ b/tools/perf/util/ordered-events.c
@@ -1,3 +1,5 @@
+#include <errno.h>
+#include <inttypes.h>
#include <linux/list.h>
#include <linux/compiler.h>
#include <linux/string.h>
@@ -79,7 +81,7 @@
static void free_dup_event(struct ordered_events *oe, union perf_event *event)
{
- if (oe->copy_on_queue) {
+ if (event && oe->copy_on_queue) {
oe->cur_alloc_size -= event->header.size;
free(event);
}
@@ -150,6 +152,7 @@
list_move(&event->list, &oe->cache);
oe->nr_events--;
free_dup_event(oe, event->event);
+ event->event = NULL;
}
int ordered_events__queue(struct ordered_events *oe, union perf_event *event,
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index 67a8aeb..01e779b 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -1,13 +1,18 @@
#include <linux/hw_breakpoint.h>
#include <linux/err.h>
-#include "util.h"
+#include <dirent.h>
+#include <errno.h>
+#include <sys/ioctl.h>
+#include <sys/param.h>
+#include "term.h"
#include "../perf.h"
#include "evlist.h"
#include "evsel.h"
#include <subcmd/parse-options.h>
#include "parse-events.h"
#include <subcmd/exec-cmd.h>
-#include "string.h"
+#include "string2.h"
+#include "strlist.h"
#include "symbol.h"
#include "cache.h"
#include "header.h"
@@ -316,8 +321,9 @@
return NULL;
(*idx)++;
- evsel->cpus = cpu_map__get(cpus);
- evsel->own_cpus = cpu_map__get(cpus);
+ evsel->cpus = cpu_map__get(cpus);
+ evsel->own_cpus = cpu_map__get(cpus);
+ evsel->system_wide = !!cpus;
if (name)
evsel->name = strdup(name);
@@ -1254,11 +1260,59 @@
evsel->scale = info.scale;
evsel->per_pkg = info.per_pkg;
evsel->snapshot = info.snapshot;
+ evsel->metric_expr = info.metric_expr;
+ evsel->metric_name = info.metric_name;
}
return evsel ? 0 : -ENOMEM;
}
+int parse_events_multi_pmu_add(struct parse_events_evlist *data,
+ char *str, struct list_head **listp)
+{
+ struct list_head *head;
+ struct parse_events_term *term;
+ struct list_head *list;
+ struct perf_pmu *pmu = NULL;
+ int ok = 0;
+
+ *listp = NULL;
+ /* Add it for all PMUs that support the alias */
+ list = malloc(sizeof(struct list_head));
+ if (!list)
+ return -1;
+ INIT_LIST_HEAD(list);
+ while ((pmu = perf_pmu__scan(pmu)) != NULL) {
+ struct perf_pmu_alias *alias;
+
+ list_for_each_entry(alias, &pmu->aliases, list) {
+ if (!strcasecmp(alias->name, str)) {
+ head = malloc(sizeof(struct list_head));
+ if (!head)
+ return -1;
+ INIT_LIST_HEAD(head);
+ if (parse_events_term__num(&term, PARSE_EVENTS__TERM_TYPE_USER,
+ str, 1, false, &str, NULL) < 0)
+ return -1;
+ list_add_tail(&term->list, head);
+
+ if (!parse_events_add_pmu(data, list,
+ pmu->name, head)) {
+ pr_debug("%s -> %s/%s/\n", str,
+ pmu->name, alias->str);
+ ok++;
+ }
+
+ parse_events_terms__delete(head);
+ }
+ }
+ }
+ if (!ok)
+ return -1;
+ *listp = list;
+ return 0;
+}
+
int parse_events__modifier_group(struct list_head *list,
char *event_mod)
{
@@ -2276,7 +2330,7 @@
* Print the help text for the event symbols:
*/
void print_events(const char *event_glob, bool name_only, bool quiet_flag,
- bool long_desc)
+ bool long_desc, bool details_flag)
{
print_symbol_events(event_glob, PERF_TYPE_HARDWARE,
event_symbols_hw, PERF_COUNT_HW_MAX, name_only);
@@ -2286,7 +2340,8 @@
print_hwcache_events(event_glob, name_only);
- print_pmu_events(event_glob, name_only, quiet_flag, long_desc);
+ print_pmu_events(event_glob, name_only, quiet_flag, long_desc,
+ details_flag);
if (event_glob != NULL)
return;
@@ -2415,6 +2470,31 @@
return new_term(new, &temp, term->val.str, term->val.num);
}
+int parse_events_copy_term_list(struct list_head *old,
+ struct list_head **new)
+{
+ struct parse_events_term *term, *n;
+ int ret;
+
+ if (!old) {
+ *new = NULL;
+ return 0;
+ }
+
+ *new = malloc(sizeof(struct list_head));
+ if (!*new)
+ return -ENOMEM;
+ INIT_LIST_HEAD(*new);
+
+ list_for_each_entry (term, old, list) {
+ ret = parse_events_term__clone(&n, term);
+ if (ret)
+ return ret;
+ list_add_tail(&n->list, *new);
+ }
+ return 0;
+}
+
void parse_events_terms__purge(struct list_head *terms)
{
struct parse_events_term *term, *h;
diff --git a/tools/perf/util/parse-events.h b/tools/perf/util/parse-events.h
index 1af6a26..a235f4d 100644
--- a/tools/perf/util/parse-events.h
+++ b/tools/perf/util/parse-events.h
@@ -8,6 +8,7 @@
#include <stdbool.h>
#include <linux/types.h>
#include <linux/perf_event.h>
+#include <string.h>
struct list_head;
struct perf_evsel;
@@ -166,6 +167,14 @@
int parse_events_add_pmu(struct parse_events_evlist *data,
struct list_head *list, char *name,
struct list_head *head_config);
+
+int parse_events_multi_pmu_add(struct parse_events_evlist *data,
+ char *str,
+ struct list_head **listp);
+
+int parse_events_copy_term_list(struct list_head *old,
+ struct list_head **new);
+
enum perf_pmu_event_symbol_type
perf_pmu__parse_check(const char *name);
void parse_events__set_leader(char *name, struct list_head *list);
@@ -175,7 +184,7 @@
int idx, const char *str);
void print_events(const char *event_glob, bool name_only, bool quiet,
- bool long_desc);
+ bool long_desc, bool details_flag);
struct event_symbol {
const char *symbol;
@@ -196,4 +205,23 @@
int valid_event_mount(const char *eventfs);
char *parse_events_formats_error_string(char *additional_terms);
+#ifdef HAVE_LIBELF_SUPPORT
+/*
+ * If the probe point starts with '%',
+ * or starts with "sdt_" and has a ':' but no '=',
+ * then it should be a SDT/cached probe point.
+ */
+static inline bool is_sdt_event(char *str)
+{
+ return (str[0] == '%' ||
+ (!strncmp(str, "sdt_", 4) &&
+ !!strchr(str, ':') && !strchr(str, '=')));
+}
+#else
+static inline bool is_sdt_event(char *str __maybe_unused)
+{
+ return false;
+}
+#endif /* HAVE_LIBELF_SUPPORT */
+
#endif /* __PERF_PARSE_EVENTS_H */
diff --git a/tools/perf/util/parse-events.y b/tools/perf/util/parse-events.y
index 30f018e..04fd8c9 100644
--- a/tools/perf/util/parse-events.y
+++ b/tools/perf/util/parse-events.y
@@ -226,68 +226,55 @@
PE_NAME opt_event_config
{
struct parse_events_evlist *data = _data;
- struct list_head *list;
+ struct list_head *list, *orig_terms, *terms;
+
+ if (parse_events_copy_term_list($2, &orig_terms))
+ YYABORT;
ALLOC_LIST(list);
- ABORT_ON(parse_events_add_pmu(data, list, $1, $2));
+ if (parse_events_add_pmu(data, list, $1, $2)) {
+ struct perf_pmu *pmu = NULL;
+ int ok = 0;
+
+ while ((pmu = perf_pmu__scan(pmu)) != NULL) {
+ char *name = pmu->name;
+
+ if (!strncmp(name, "uncore_", 7) &&
+ strncmp($1, "uncore_", 7))
+ name += 7;
+ if (!strncmp($1, name, strlen($1))) {
+ if (parse_events_copy_term_list(orig_terms, &terms))
+ YYABORT;
+ if (!parse_events_add_pmu(data, list, pmu->name, terms))
+ ok++;
+ parse_events_terms__delete(terms);
+ }
+ }
+ if (!ok)
+ YYABORT;
+ }
parse_events_terms__delete($2);
+ parse_events_terms__delete(orig_terms);
$$ = list;
}
|
PE_KERNEL_PMU_EVENT sep_dc
{
- struct parse_events_evlist *data = _data;
- struct list_head *head;
- struct parse_events_term *term;
struct list_head *list;
- struct perf_pmu *pmu = NULL;
- int ok = 0;
- /* Add it for all PMUs that support the alias */
- ALLOC_LIST(list);
- while ((pmu = perf_pmu__scan(pmu)) != NULL) {
- struct perf_pmu_alias *alias;
-
- list_for_each_entry(alias, &pmu->aliases, list) {
- if (!strcasecmp(alias->name, $1)) {
- ALLOC_LIST(head);
- ABORT_ON(parse_events_term__num(&term, PARSE_EVENTS__TERM_TYPE_USER,
- $1, 1, false, &@1, NULL));
- list_add_tail(&term->list, head);
-
- if (!parse_events_add_pmu(data, list,
- pmu->name, head)) {
- pr_debug("%s -> %s/%s/\n", $1,
- pmu->name, alias->str);
- ok++;
- }
-
- parse_events_terms__delete(head);
- }
- }
- }
- if (!ok)
+ if (parse_events_multi_pmu_add(_data, $1, &list) < 0)
YYABORT;
$$ = list;
}
|
PE_PMU_EVENT_PRE '-' PE_PMU_EVENT_SUF sep_dc
{
- struct parse_events_evlist *data = _data;
- struct list_head *head;
- struct parse_events_term *term;
struct list_head *list;
char pmu_name[128];
+
snprintf(&pmu_name, 128, "%s-%s", $1, $3);
-
- ALLOC_LIST(head);
- ABORT_ON(parse_events_term__num(&term, PARSE_EVENTS__TERM_TYPE_USER,
- &pmu_name, 1, false, &@1, NULL));
- list_add_tail(&term->list, head);
-
- ALLOC_LIST(list);
- ABORT_ON(parse_events_add_pmu(data, list, "cpu", head));
- parse_events_terms__delete(head);
+ if (parse_events_multi_pmu_add(_data, pmu_name, &list) < 0)
+ YYABORT;
$$ = list;
}
diff --git a/tools/perf/util/path.c b/tools/perf/util/path.c
index 7c7630b..50ec3bc 100644
--- a/tools/perf/util/path.c
+++ b/tools/perf/util/path.c
@@ -11,8 +11,13 @@
* which is what it's designed for.
*/
#include "cache.h"
-#include "util.h"
+#include "path.h"
+#include <linux/kernel.h>
#include <limits.h>
+#include <stdio.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
static char bad_path[] = "/bad-path/";
/*
@@ -50,3 +55,24 @@
return bad_path;
return cleanup_path(pathname);
}
+
+int path__join(char *bf, size_t size, const char *path1, const char *path2)
+{
+ return scnprintf(bf, size, "%s%s%s", path1, path1[0] ? "/" : "", path2);
+}
+
+int path__join3(char *bf, size_t size, const char *path1, const char *path2, const char *path3)
+{
+ return scnprintf(bf, size, "%s%s%s%s%s", path1, path1[0] ? "/" : "",
+ path2, path2[0] ? "/" : "", path3);
+}
+
+bool is_regular_file(const char *file)
+{
+ struct stat st;
+
+ if (stat(file, &st))
+ return false;
+
+ return S_ISREG(st.st_mode);
+}
diff --git a/tools/perf/util/path.h b/tools/perf/util/path.h
new file mode 100644
index 0000000..9a276a58
--- /dev/null
+++ b/tools/perf/util/path.h
@@ -0,0 +1,9 @@
+#ifndef _PERF_PATH_H
+#define _PERF_PATH_H
+
+int path__join(char *bf, size_t size, const char *path1, const char *path2);
+int path__join3(char *bf, size_t size, const char *path1, const char *path2, const char *path3);
+
+bool is_regular_file(const char *file);
+
+#endif /* _PERF_PATH_H */
diff --git a/tools/perf/util/perf-hooks.c b/tools/perf/util/perf-hooks.c
index cb36830..d550929 100644
--- a/tools/perf/util/perf-hooks.c
+++ b/tools/perf/util/perf-hooks.c
@@ -9,6 +9,7 @@
#include <stdlib.h>
#include <setjmp.h>
#include <linux/err.h>
+#include <linux/kernel.h>
#include "util/util.h"
#include "util/debug.h"
#include "util/perf-hooks.h"
diff --git a/tools/perf/util/perf_regs.c b/tools/perf/util/perf_regs.c
index c4023f2..b2ae039 100644
--- a/tools/perf/util/perf_regs.c
+++ b/tools/perf/util/perf_regs.c
@@ -6,6 +6,12 @@
SMPL_REG_END
};
+int __weak arch_sdt_arg_parse_op(char *old_op __maybe_unused,
+ char **new_op __maybe_unused)
+{
+ return SDT_ARG_SKIP;
+}
+
#ifdef HAVE_PERF_REGS_SUPPORT
int perf_reg_value(u64 *valp, struct regs_dump *regs, int id)
{
diff --git a/tools/perf/util/perf_regs.h b/tools/perf/util/perf_regs.h
index 679d6e4..32b37d1 100644
--- a/tools/perf/util/perf_regs.h
+++ b/tools/perf/util/perf_regs.h
@@ -15,6 +15,13 @@
extern const struct sample_reg sample_reg_masks[];
+enum {
+ SDT_ARG_VALID = 0,
+ SDT_ARG_SKIP,
+};
+
+int arch_sdt_arg_parse_op(char *old_op, char **new_op);
+
#ifdef HAVE_PERF_REGS_SUPPORT
#include <perf_regs.h>
diff --git a/tools/perf/util/pmu.c b/tools/perf/util/pmu.c
index 12f84dd..ac16a9d 100644
--- a/tools/perf/util/pmu.c
+++ b/tools/perf/util/pmu.c
@@ -1,6 +1,8 @@
#include <linux/list.h>
#include <linux/compiler.h>
#include <sys/types.h>
+#include <errno.h>
+#include <sys/stat.h>
#include <unistd.h>
#include <stdio.h>
#include <stdbool.h>
@@ -15,6 +17,7 @@
#include "header.h"
#include "pmu-events/pmu-events.h"
#include "cache.h"
+#include "string2.h"
struct perf_pmu_format {
char *name;
@@ -231,7 +234,9 @@
static int __perf_pmu__new_alias(struct list_head *list, char *dir, char *name,
char *desc, char *val,
char *long_desc, char *topic,
- char *unit, char *perpkg)
+ char *unit, char *perpkg,
+ char *metric_expr,
+ char *metric_name)
{
struct perf_pmu_alias *alias;
int ret;
@@ -265,6 +270,8 @@
perf_pmu__parse_snapshot(alias, dir, name);
}
+ alias->metric_expr = metric_expr ? strdup(metric_expr) : NULL;
+ alias->metric_name = metric_name ? strdup(metric_name): NULL;
alias->desc = desc ? strdup(desc) : NULL;
alias->long_desc = long_desc ? strdup(long_desc) :
desc ? strdup(desc) : NULL;
@@ -294,7 +301,7 @@
buf[ret] = 0;
return __perf_pmu__new_alias(list, dir, name, NULL, buf, NULL, NULL, NULL,
- NULL);
+ NULL, NULL, NULL);
}
static inline bool pmu_alias_info_file(char *name)
@@ -564,7 +571,9 @@
__perf_pmu__new_alias(head, NULL, (char *)pe->name,
(char *)pe->desc, (char *)pe->event,
(char *)pe->long_desc, (char *)pe->topic,
- (char *)pe->unit, (char *)pe->perpkg);
+ (char *)pe->unit, (char *)pe->perpkg,
+ (char *)pe->metric_expr,
+ (char *)pe->metric_name);
}
out:
@@ -991,6 +1000,8 @@
info->unit = NULL;
info->scale = 0.0;
info->snapshot = false;
+ info->metric_expr = NULL;
+ info->metric_name = NULL;
list_for_each_entry_safe(term, h, head_terms, list) {
alias = pmu_find_alias(pmu, term);
@@ -1006,6 +1017,8 @@
if (alias->per_pkg)
info->per_pkg = true;
+ info->metric_expr = alias->metric_expr;
+ info->metric_name = alias->metric_name;
list_del(&term->list);
free(term);
@@ -1100,6 +1113,8 @@
char *topic;
char *str;
char *pmu;
+ char *metric_expr;
+ char *metric_name;
};
static int cmp_sevent(const void *a, const void *b)
@@ -1136,13 +1151,12 @@
break;
s += wlen;
column += n;
- while (isspace(*s))
- s++;
+ s = ltrim(s);
}
}
void print_pmu_events(const char *event_glob, bool name_only, bool quiet_flag,
- bool long_desc)
+ bool long_desc, bool details_flag)
{
struct perf_pmu *pmu;
struct perf_pmu_alias *alias;
@@ -1198,6 +1212,8 @@
aliases[j].topic = alias->topic;
aliases[j].str = alias->str;
aliases[j].pmu = pmu->name;
+ aliases[j].metric_expr = alias->metric_expr;
+ aliases[j].metric_name = alias->metric_name;
j++;
}
if (pmu->selectable &&
@@ -1232,8 +1248,14 @@
printf("%*s", 8, "[");
wordwrap(aliases[j].desc, 8, columns, 0);
printf("]\n");
- if (verbose > 0)
- printf("%*s%s/%s/\n", 8, "", aliases[j].pmu, aliases[j].str);
+ if (details_flag) {
+ printf("%*s%s/%s/ ", 8, "", aliases[j].pmu, aliases[j].str);
+ if (aliases[j].metric_name)
+ printf(" MetricName: %s", aliases[j].metric_name);
+ if (aliases[j].metric_expr)
+ printf(" MetricExpr: %s", aliases[j].metric_expr);
+ putchar('\n');
+ }
} else
printf(" %-50s [Kernel PMU event]\n", aliases[j].name);
printed++;
diff --git a/tools/perf/util/pmu.h b/tools/perf/util/pmu.h
index 00852dd..ea7f450 100644
--- a/tools/perf/util/pmu.h
+++ b/tools/perf/util/pmu.h
@@ -31,6 +31,8 @@
struct perf_pmu_info {
const char *unit;
+ const char *metric_expr;
+ const char *metric_name;
double scale;
bool per_pkg;
bool snapshot;
@@ -50,6 +52,8 @@
double scale;
bool per_pkg;
bool snapshot;
+ char *metric_expr;
+ char *metric_name;
};
struct perf_pmu *perf_pmu__find(const char *name);
@@ -76,7 +80,7 @@
struct perf_pmu *perf_pmu__scan(struct perf_pmu *pmu);
void print_pmu_events(const char *event_glob, bool name_only, bool quiet,
- bool long_desc);
+ bool long_desc, bool details_flag);
bool pmu_have_event(const char *pname, const char *name);
int perf_pmu__scan_file(struct perf_pmu *pmu, const char *name, const char *fmt,
diff --git a/tools/perf/util/print_binary.c b/tools/perf/util/print_binary.c
new file mode 100644
index 0000000..e908177
--- /dev/null
+++ b/tools/perf/util/print_binary.c
@@ -0,0 +1,55 @@
+#include "print_binary.h"
+#include <linux/log2.h>
+#include "sane_ctype.h"
+
+void print_binary(unsigned char *data, size_t len,
+ size_t bytes_per_line, print_binary_t printer,
+ void *extra)
+{
+ size_t i, j, mask;
+
+ if (!printer)
+ return;
+
+ bytes_per_line = roundup_pow_of_two(bytes_per_line);
+ mask = bytes_per_line - 1;
+
+ printer(BINARY_PRINT_DATA_BEGIN, 0, extra);
+ for (i = 0; i < len; i++) {
+ if ((i & mask) == 0) {
+ printer(BINARY_PRINT_LINE_BEGIN, -1, extra);
+ printer(BINARY_PRINT_ADDR, i, extra);
+ }
+
+ printer(BINARY_PRINT_NUM_DATA, data[i], extra);
+
+ if (((i & mask) == mask) || i == len - 1) {
+ for (j = 0; j < mask-(i & mask); j++)
+ printer(BINARY_PRINT_NUM_PAD, -1, extra);
+
+ printer(BINARY_PRINT_SEP, i, extra);
+ for (j = i & ~mask; j <= i; j++)
+ printer(BINARY_PRINT_CHAR_DATA, data[j], extra);
+ for (j = 0; j < mask-(i & mask); j++)
+ printer(BINARY_PRINT_CHAR_PAD, i, extra);
+ printer(BINARY_PRINT_LINE_END, -1, extra);
+ }
+ }
+ printer(BINARY_PRINT_DATA_END, -1, extra);
+}
+
+int is_printable_array(char *p, unsigned int len)
+{
+ unsigned int i;
+
+ if (!p || !len || p[len - 1] != 0)
+ return 0;
+
+ len--;
+
+ for (i = 0; i < len; i++) {
+ if (!isprint(p[i]) && !isspace(p[i]))
+ return 0;
+ }
+ return 1;
+}
diff --git a/tools/perf/util/print_binary.h b/tools/perf/util/print_binary.h
new file mode 100644
index 0000000..da04272
--- /dev/null
+++ b/tools/perf/util/print_binary.h
@@ -0,0 +1,28 @@
+#ifndef PERF_PRINT_BINARY_H
+#define PERF_PRINT_BINARY_H
+
+#include <stddef.h>
+
+enum binary_printer_ops {
+ BINARY_PRINT_DATA_BEGIN,
+ BINARY_PRINT_LINE_BEGIN,
+ BINARY_PRINT_ADDR,
+ BINARY_PRINT_NUM_DATA,
+ BINARY_PRINT_NUM_PAD,
+ BINARY_PRINT_SEP,
+ BINARY_PRINT_CHAR_DATA,
+ BINARY_PRINT_CHAR_PAD,
+ BINARY_PRINT_LINE_END,
+ BINARY_PRINT_DATA_END,
+};
+
+typedef void (*print_binary_t)(enum binary_printer_ops op,
+ unsigned int val, void *extra);
+
+void print_binary(unsigned char *data, size_t len,
+ size_t bytes_per_line, print_binary_t printer,
+ void *extra);
+
+int is_printable_array(char *p, unsigned int len);
+
+#endif /* PERF_PRINT_BINARY_H */
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 28fb62c..84e7e69 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -19,6 +19,7 @@
*
*/
+#include <inttypes.h>
#include <sys/utsname.h>
#include <sys/types.h>
#include <sys/stat.h>
@@ -35,6 +36,7 @@
#include "util.h"
#include "event.h"
#include "strlist.h"
+#include "strfilter.h"
#include "debug.h"
#include "cache.h"
#include "color.h"
@@ -46,8 +48,10 @@
#include "probe-finder.h"
#include "probe-file.h"
#include "session.h"
+#include "string2.h"
-#define MAX_CMDLEN 256
+#include "sane_ctype.h"
+
#define PERFPROBE_GROUP "probe"
bool probe_event_dry_run; /* Dry run flag */
@@ -757,7 +761,9 @@
}
for (i = 0; i < ntevs; i++) {
- if (!tevs[i].point.address || tevs[i].point.retprobe)
+ if (!tevs[i].point.address)
+ continue;
+ if (tevs[i].point.retprobe && !kretprobe_offset_is_supported())
continue;
/* If we found a wrong one, mark it by NULL symbol */
if (kprobe_warn_out_range(tevs[i].point.symbol,
@@ -1339,14 +1345,7 @@
if (!arg)
return -EINVAL;
- /*
- * If the probe point starts with '%',
- * or starts with "sdt_" and has a ':' but no '=',
- * then it should be a SDT/cached probe point.
- */
- if (arg[0] == '%' ||
- (!strncmp(arg, "sdt_", 4) &&
- !!strchr(arg, ':') && !strchr(arg, '='))) {
+ if (is_sdt_event(arg)) {
pev->sdt = true;
if (arg[0] == '%')
arg++;
@@ -1528,11 +1527,6 @@
return -EINVAL;
}
- if (pp->retprobe && !pp->function) {
- semantic_error("Return probe requires an entry function.\n");
- return -EINVAL;
- }
-
if ((pp->offset || pp->line || pp->lazy_line) && pp->retprobe) {
semantic_error("Offset/Line/Lazy pattern can't be used with "
"return probe.\n");
@@ -2841,7 +2835,8 @@
}
/* Note that the symbols in the kmodule are not relocated */
- if (!pev->uprobes && !pp->retprobe && !pev->target) {
+ if (!pev->uprobes && !pev->target &&
+ (!pp->retprobe || kretprobe_offset_is_supported())) {
reloc_sym = kernel_get_ref_reloc_sym();
if (!reloc_sym) {
pr_warning("Relocated base symbol is not found!\n");
@@ -3057,7 +3052,7 @@
struct probe_trace_event *new_tevs;
int ret = 0;
- if (ntevs == 0) {
+ if (*ntevs == 0) {
*tevs = *tevs2;
*ntevs = ntevs2;
*tevs2 = NULL;
diff --git a/tools/perf/util/probe-event.h b/tools/perf/util/probe-event.h
index 5d4e940..3738426 100644
--- a/tools/perf/util/probe-event.h
+++ b/tools/perf/util/probe-event.h
@@ -3,8 +3,6 @@
#include <stdbool.h>
#include "intlist.h"
-#include "strlist.h"
-#include "strfilter.h"
/* Probe related configurations */
struct probe_conf {
@@ -107,6 +105,8 @@
struct intlist *line_list; /* Visible lines */
};
+struct strlist;
+
/* List of variables */
struct variable_list {
struct probe_trace_point point; /* Actual probepoint */
@@ -153,6 +153,9 @@
int apply_perf_probe_events(struct perf_probe_event *pevs, int npevs);
int show_probe_trace_events(struct perf_probe_event *pevs, int npevs);
void cleanup_perf_probe_events(struct perf_probe_event *pevs, int npevs);
+
+struct strfilter;
+
int del_perf_probe_events(struct strfilter *filter);
int show_perf_probe_event(const char *group, const char *event,
diff --git a/tools/perf/util/probe-file.c b/tools/perf/util/probe-file.c
index 436b647..d679389 100644
--- a/tools/perf/util/probe-file.c
+++ b/tools/perf/util/probe-file.c
@@ -14,10 +14,15 @@
* GNU General Public License for more details.
*
*/
+#include <errno.h>
+#include <sys/stat.h>
+#include <sys/types.h>
#include <sys/uio.h>
+#include <unistd.h>
#include "util.h"
#include "event.h"
#include "strlist.h"
+#include "strfilter.h"
#include "debug.h"
#include "cache.h"
#include "color.h"
@@ -27,8 +32,11 @@
#include "probe-event.h"
#include "probe-file.h"
#include "session.h"
+#include "perf_regs.h"
+#include "string2.h"
-#define MAX_CMDLEN 256
+/* 4096 - 2 ('\n' + '\0') */
+#define MAX_CMDLEN 4094
static void print_open_warning(int err, bool uprobe)
{
@@ -70,7 +78,7 @@
}
}
-static int open_probe_events(const char *trace_file, bool readwrite)
+int open_trace_file(const char *trace_file, bool readwrite)
{
char buf[PATH_MAX];
int ret;
@@ -92,12 +100,12 @@
static int open_kprobe_events(bool readwrite)
{
- return open_probe_events("kprobe_events", readwrite);
+ return open_trace_file("kprobe_events", readwrite);
}
static int open_uprobe_events(bool readwrite)
{
- return open_probe_events("uprobe_events", readwrite);
+ return open_trace_file("uprobe_events", readwrite);
}
int probe_file__open(int flag)
@@ -687,6 +695,110 @@
: (unsigned long long)note->addr.a64[0];
}
+static const char * const type_to_suffix[] = {
+ ":s64", "", "", "", ":s32", "", ":s16", ":s8",
+ "", ":u8", ":u16", "", ":u32", "", "", "", ":u64"
+};
+
+/*
+ * Isolate the string number and convert it into a decimal value;
+ * this will be an index to get suffix of the uprobe name (defining
+ * the type)
+ */
+static int sdt_arg_parse_size(char *n_ptr, const char **suffix)
+{
+ long type_idx;
+
+ type_idx = strtol(n_ptr, NULL, 10);
+ if (type_idx < -8 || type_idx > 8) {
+ pr_debug4("Failed to get a valid sdt type\n");
+ return -1;
+ }
+
+ *suffix = type_to_suffix[type_idx + 8];
+ return 0;
+}
+
+static int synthesize_sdt_probe_arg(struct strbuf *buf, int i, const char *arg)
+{
+ char *op, *desc = strdup(arg), *new_op = NULL;
+ const char *suffix = "";
+ int ret = -1;
+
+ if (desc == NULL) {
+ pr_debug4("Allocation error\n");
+ return ret;
+ }
+
+ /*
+ * Argument is in N@OP format. N is size of the argument and OP is
+ * the actual assembly operand. N can be omitted; in that case
+ * argument is just OP(without @).
+ */
+ op = strchr(desc, '@');
+ if (op) {
+ op[0] = '\0';
+ op++;
+
+ if (sdt_arg_parse_size(desc, &suffix))
+ goto error;
+ } else {
+ op = desc;
+ }
+
+ ret = arch_sdt_arg_parse_op(op, &new_op);
+
+ if (ret < 0)
+ goto error;
+
+ if (ret == SDT_ARG_VALID) {
+ ret = strbuf_addf(buf, " arg%d=%s%s", i + 1, new_op, suffix);
+ if (ret < 0)
+ goto error;
+ }
+
+ ret = 0;
+error:
+ free(desc);
+ free(new_op);
+ return ret;
+}
+
+static char *synthesize_sdt_probe_command(struct sdt_note *note,
+ const char *pathname,
+ const char *sdtgrp)
+{
+ struct strbuf buf;
+ char *ret = NULL, **args;
+ int i, args_count;
+
+ if (strbuf_init(&buf, 32) < 0)
+ return NULL;
+
+ if (strbuf_addf(&buf, "p:%s/%s %s:0x%llx",
+ sdtgrp, note->name, pathname,
+ sdt_note__get_addr(note)) < 0)
+ goto error;
+
+ if (!note->args)
+ goto out;
+
+ if (note->args) {
+ args = argv_split(note->args, &args_count);
+
+ for (i = 0; i < args_count; ++i) {
+ if (synthesize_sdt_probe_arg(&buf, i, args[i]) < 0)
+ goto error;
+ }
+ }
+
+out:
+ ret = strbuf_detach(&buf, NULL);
+error:
+ strbuf_release(&buf);
+ return ret;
+}
+
int probe_cache__scan_sdt(struct probe_cache *pcache, const char *pathname)
{
struct probe_cache_entry *entry = NULL;
@@ -723,11 +835,12 @@
entry->pev.group = strdup(sdtgrp);
list_add_tail(&entry->node, &pcache->entries);
}
- ret = asprintf(&buf, "p:%s/%s %s:0x%llx",
- sdtgrp, note->name, pathname,
- sdt_note__get_addr(note));
- if (ret < 0)
+ buf = synthesize_sdt_probe_command(note, pathname, sdtgrp);
+ if (!buf) {
+ ret = -ENOMEM;
break;
+ }
+
strlist__add(entry->tevlist, buf);
free(buf);
entry = NULL;
@@ -877,59 +990,72 @@
return 0;
}
-static struct {
- const char *pattern;
- bool avail;
- bool checked;
-} probe_type_table[] = {
-#define DEFINE_TYPE(idx, pat, def_avail) \
- [idx] = {.pattern = pat, .avail = (def_avail)}
- DEFINE_TYPE(PROBE_TYPE_U, "* u8/16/32/64,*", true),
- DEFINE_TYPE(PROBE_TYPE_S, "* s8/16/32/64,*", true),
- DEFINE_TYPE(PROBE_TYPE_X, "* x8/16/32/64,*", false),
- DEFINE_TYPE(PROBE_TYPE_STRING, "* string,*", true),
- DEFINE_TYPE(PROBE_TYPE_BITFIELD,
- "* b<bit-width>@<bit-offset>/<container-size>", true),
+enum ftrace_readme {
+ FTRACE_README_PROBE_TYPE_X = 0,
+ FTRACE_README_KRETPROBE_OFFSET,
+ FTRACE_README_END,
};
-bool probe_type_is_available(enum probe_type type)
+static struct {
+ const char *pattern;
+ bool avail;
+} ftrace_readme_table[] = {
+#define DEFINE_TYPE(idx, pat) \
+ [idx] = {.pattern = pat, .avail = false}
+ DEFINE_TYPE(FTRACE_README_PROBE_TYPE_X, "*type: * x8/16/32/64,*"),
+ DEFINE_TYPE(FTRACE_README_KRETPROBE_OFFSET, "*place (kretprobe): *"),
+};
+
+static bool scan_ftrace_readme(enum ftrace_readme type)
{
+ int fd;
FILE *fp;
char *buf = NULL;
size_t len = 0;
- bool target_line = false;
- bool ret = probe_type_table[type].avail;
+ bool ret = false;
+ static bool scanned = false;
- if (type >= PROBE_TYPE_END)
- return false;
- /* We don't have to check the type which supported by default */
- if (ret || probe_type_table[type].checked)
+ if (scanned)
+ goto result;
+
+ fd = open_trace_file("README", false);
+ if (fd < 0)
return ret;
- if (asprintf(&buf, "%s/README", tracing_path) < 0)
+ fp = fdopen(fd, "r");
+ if (!fp) {
+ close(fd);
return ret;
-
- fp = fopen(buf, "r");
- if (!fp)
- goto end;
-
- zfree(&buf);
- while (getline(&buf, &len, fp) > 0 && !ret) {
- if (!target_line) {
- target_line = !!strstr(buf, " type: ");
- if (!target_line)
- continue;
- } else if (strstr(buf, "\t ") != buf)
- break;
- ret = strglobmatch(buf, probe_type_table[type].pattern);
}
- /* Cache the result */
- probe_type_table[type].checked = true;
- probe_type_table[type].avail = ret;
+
+ while (getline(&buf, &len, fp) > 0)
+ for (enum ftrace_readme i = 0; i < FTRACE_README_END; i++)
+ if (!ftrace_readme_table[i].avail)
+ ftrace_readme_table[i].avail =
+ strglobmatch(buf, ftrace_readme_table[i].pattern);
+ scanned = true;
fclose(fp);
-end:
free(buf);
- return ret;
+result:
+ if (type >= FTRACE_README_END)
+ return false;
+
+ return ftrace_readme_table[type].avail;
+}
+
+bool probe_type_is_available(enum probe_type type)
+{
+ if (type >= PROBE_TYPE_END)
+ return false;
+ else if (type == PROBE_TYPE_X)
+ return scan_ftrace_readme(FTRACE_README_PROBE_TYPE_X);
+
+ return true;
+}
+
+bool kretprobe_offset_is_supported(void)
+{
+ return scan_ftrace_readme(FTRACE_README_KRETPROBE_OFFSET);
}
diff --git a/tools/perf/util/probe-file.h b/tools/perf/util/probe-file.h
index eba44c3..5ecc9d3 100644
--- a/tools/perf/util/probe-file.h
+++ b/tools/perf/util/probe-file.h
@@ -1,10 +1,11 @@
#ifndef __PROBE_FILE_H
#define __PROBE_FILE_H
-#include "strlist.h"
-#include "strfilter.h"
#include "probe-event.h"
+struct strlist;
+struct strfilter;
+
/* Cache of probe definitions */
struct probe_cache_entry {
struct list_head node;
@@ -35,11 +36,13 @@
/* probe-file.c depends on libelf */
#ifdef HAVE_LIBELF_SUPPORT
+int open_trace_file(const char *trace_file, bool readwrite);
int probe_file__open(int flag);
int probe_file__open_both(int *kfd, int *ufd, int flag);
struct strlist *probe_file__get_namelist(int fd);
struct strlist *probe_file__get_rawlist(int fd);
int probe_file__add_event(int fd, struct probe_trace_event *tev);
+
int probe_file__del_events(int fd, struct strfilter *filter);
int probe_file__get_events(int fd, struct strfilter *filter,
struct strlist *plist);
@@ -64,6 +67,7 @@
const char *group, const char *event);
int probe_cache__show_all_caches(struct strfilter *filter);
bool probe_type_is_available(enum probe_type type);
+bool kretprobe_offset_is_supported(void);
#else /* ! HAVE_LIBELF_SUPPORT */
static inline struct probe_cache *probe_cache__new(const char *tgt __maybe_unused)
{
diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
index 57cd268..a5731de 100644
--- a/tools/perf/util/probe-finder.c
+++ b/tools/perf/util/probe-finder.c
@@ -19,6 +19,7 @@
*
*/
+#include <inttypes.h>
#include <sys/utsname.h>
#include <sys/types.h>
#include <sys/stat.h>
@@ -37,9 +38,11 @@
#include "debug.h"
#include "intlist.h"
#include "util.h"
+#include "strlist.h"
#include "symbol.h"
#include "probe-finder.h"
#include "probe-file.h"
+#include "string2.h"
/* Kprobe tracer basic type is up to u64 */
#define MAX_BASIC_TYPE_BITS 64
diff --git a/tools/perf/util/probe-finder.h b/tools/perf/util/probe-finder.h
index 2956c51..27f0615 100644
--- a/tools/perf/util/probe-finder.h
+++ b/tools/perf/util/probe-finder.h
@@ -2,9 +2,9 @@
#define _PROBE_FINDER_H
#include <stdbool.h>
-#include "util.h"
#include "intlist.h"
#include "probe-event.h"
+#include "sane_ctype.h"
#define MAX_PROBE_BUFFER 1024
#define MAX_PROBES 128
diff --git a/tools/perf/util/python-ext-sources b/tools/perf/util/python-ext-sources
index 0546a43..9f3b0d9 100644
--- a/tools/perf/util/python-ext-sources
+++ b/tools/perf/util/python-ext-sources
@@ -21,8 +21,10 @@
util/parse-branch-options.c
util/rblist.c
util/counts.c
+util/print_binary.c
util/strlist.c
util/trace-event.c
../lib/rbtree.c
util/string.c
util/symbol_fprintf.c
+util/units.c
diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index a5fbc01..c129e99 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -4,12 +4,26 @@
#include <poll.h>
#include <linux/err.h>
#include "evlist.h"
+#include "callchain.h"
#include "evsel.h"
#include "event.h"
#include "cpumap.h"
+#include "print_binary.h"
#include "thread_map.h"
/*
+ * Provide these two so that we don't have to link against callchain.c and
+ * start dragging hist.c, etc.
+ */
+struct callchain_param callchain_param;
+
+int parse_callchain_record(const char *arg __maybe_unused,
+ struct callchain_param *param __maybe_unused)
+{
+ return 0;
+}
+
+/*
* Support debug printing even though util/debug.c is not linked. That means
* implementing 'verbose' and 'eprintf'.
*/
diff --git a/tools/perf/util/quote.c b/tools/perf/util/quote.c
index 293534c..1ba8920 100644
--- a/tools/perf/util/quote.c
+++ b/tools/perf/util/quote.c
@@ -1,3 +1,4 @@
+#include <errno.h>
#include <stdlib.h>
#include "strbuf.h"
#include "quote.h"
diff --git a/tools/perf/util/record.c b/tools/perf/util/record.c
index 98bf584..d91bdf5 100644
--- a/tools/perf/util/record.c
+++ b/tools/perf/util/record.c
@@ -2,6 +2,7 @@
#include "evsel.h"
#include "cpumap.h"
#include "parse-events.h"
+#include <errno.h>
#include <api/fs/fs.h>
#include "util.h"
#include "cloexec.h"
diff --git a/tools/perf/util/sane_ctype.h b/tools/perf/util/sane_ctype.h
new file mode 100644
index 0000000..4308c22
--- /dev/null
+++ b/tools/perf/util/sane_ctype.h
@@ -0,0 +1,51 @@
+#ifndef _PERF_SANE_CTYPE_H
+#define _PERF_SANE_CTYPE_H
+
+extern const char *graph_line;
+extern const char *graph_dotted_line;
+extern const char *spaces;
+extern const char *dots;
+
+/* Sane ctype - no locale, and works with signed chars */
+#undef isascii
+#undef isspace
+#undef isdigit
+#undef isxdigit
+#undef isalpha
+#undef isprint
+#undef isalnum
+#undef islower
+#undef isupper
+#undef tolower
+#undef toupper
+
+extern unsigned char sane_ctype[256];
+#define GIT_SPACE 0x01
+#define GIT_DIGIT 0x02
+#define GIT_ALPHA 0x04
+#define GIT_GLOB_SPECIAL 0x08
+#define GIT_REGEX_SPECIAL 0x10
+#define GIT_PRINT_EXTRA 0x20
+#define GIT_PRINT 0x3E
+#define sane_istest(x,mask) ((sane_ctype[(unsigned char)(x)] & (mask)) != 0)
+#define isascii(x) (((x) & ~0x7f) == 0)
+#define isspace(x) sane_istest(x,GIT_SPACE)
+#define isdigit(x) sane_istest(x,GIT_DIGIT)
+#define isxdigit(x) \
+ (sane_istest(toupper(x), GIT_ALPHA | GIT_DIGIT) && toupper(x) < 'G')
+#define isalpha(x) sane_istest(x,GIT_ALPHA)
+#define isalnum(x) sane_istest(x,GIT_ALPHA | GIT_DIGIT)
+#define isprint(x) sane_istest(x,GIT_PRINT)
+#define islower(x) (sane_istest(x,GIT_ALPHA) && (x & 0x20))
+#define isupper(x) (sane_istest(x,GIT_ALPHA) && !(x & 0x20))
+#define tolower(x) sane_case((unsigned char)(x), 0x20)
+#define toupper(x) sane_case((unsigned char)(x), 0)
+
+static inline int sane_case(int x, int high)
+{
+ if (sane_istest(x, GIT_ALPHA))
+ x = (x & ~0x20) | high;
+ return x;
+}
+
+#endif /* _PERF_SANE_CTYPE_H */
diff --git a/tools/perf/util/scripting-engines/trace-event-perl.c b/tools/perf/util/scripting-engines/trace-event-perl.c
index dff043a..2b12bdb 100644
--- a/tools/perf/util/scripting-engines/trace-event-perl.c
+++ b/tools/perf/util/scripting-engines/trace-event-perl.c
@@ -19,6 +19,7 @@
*
*/
+#include <inttypes.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 783326c..9d92af7 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -21,6 +21,7 @@
#include <Python.h>
+#include <inttypes.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -45,6 +46,7 @@
#include "../call-path.h"
#include "thread_map.h"
#include "cpumap.h"
+#include "print_binary.h"
#include "stat.h"
PyMODINIT_FUNC initperf_trace_context(void);
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 1dd617d..3041c6b 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1,5 +1,8 @@
+#include <errno.h>
+#include <inttypes.h>
#include <linux/kernel.h>
#include <traceevent/event-parse.h>
+#include <api/fs/fs.h>
#include <byteswap.h>
#include <unistd.h>
@@ -16,6 +19,7 @@
#include "perf_regs.h"
#include "asm/bug.h"
#include "auxtrace.h"
+#include "thread.h"
#include "thread-stack.h"
#include "stat.h"
@@ -139,8 +143,14 @@
if (perf_session__open(session) < 0)
goto out_close;
- perf_session__set_id_hdr_size(session);
- perf_session__set_comm_exec(session);
+ /*
+ * set session attributes that are present in perf.data
+ * but not in pipe-mode.
+ */
+ if (!file->is_pipe) {
+ perf_session__set_id_hdr_size(session);
+ perf_session__set_comm_exec(session);
+ }
}
} else {
session->machines.host.env = &perf_env;
@@ -155,7 +165,11 @@
pr_warning("Cannot read kernel map\n");
}
- if (tool && tool->ordering_requires_timestamps &&
+ /*
+ * In pipe-mode, evlist is empty until PERF_RECORD_HEADER_ATTR is
+ * processed, so perf_evlist__sample_id_all is not meaningful here.
+ */
+ if ((!file || !file->is_pipe) && tool && tool->ordering_requires_timestamps &&
tool->ordered_events && !perf_evlist__sample_id_all(session->evlist)) {
dump_printf("WARNING: No sample_id_all support, falling back to unordered processing\n");
tool->ordered_events = false;
@@ -1239,6 +1253,8 @@
return tool->mmap2(tool, event, sample, machine);
case PERF_RECORD_COMM:
return tool->comm(tool, event, sample, machine);
+ case PERF_RECORD_NAMESPACES:
+ return tool->namespaces(tool, event, sample, machine);
case PERF_RECORD_FORK:
return tool->fork(tool, event, sample, machine);
case PERF_RECORD_EXIT:
@@ -1258,9 +1274,12 @@
case PERF_RECORD_UNTHROTTLE:
return tool->unthrottle(tool, event, sample, machine);
case PERF_RECORD_AUX:
- if (tool->aux == perf_event__process_aux &&
- (event->aux.flags & PERF_AUX_FLAG_TRUNCATED))
- evlist->stats.total_aux_lost += 1;
+ if (tool->aux == perf_event__process_aux) {
+ if (event->aux.flags & PERF_AUX_FLAG_TRUNCATED)
+ evlist->stats.total_aux_lost += 1;
+ if (event->aux.flags & PERF_AUX_FLAG_PARTIAL)
+ evlist->stats.total_aux_partial += 1;
+ }
return tool->aux(tool, event, sample, machine);
case PERF_RECORD_ITRACE_START:
return tool->itrace_start(tool, event, sample, machine);
@@ -1494,6 +1513,11 @@
err = -1;
}
+ if (thread == NULL || thread__set_namespaces(thread, 0, NULL)) {
+ pr_err("problem inserting idle task.\n");
+ err = -1;
+ }
+
/* machine__findnew_thread() got the thread, so put it */
thread__put(thread);
return err;
@@ -1548,6 +1572,23 @@
stats->nr_events[PERF_RECORD_AUX]);
}
+ if (session->tool->aux == perf_event__process_aux &&
+ stats->total_aux_partial != 0) {
+ bool vmm_exclusive = false;
+
+ (void)sysfs__read_bool("module/kvm_intel/parameters/vmm_exclusive",
+ &vmm_exclusive);
+
+ ui__warning("AUX data had gaps in it %" PRIu64 " times out of %u!\n\n"
+ "Are you running a KVM guest in the background?%s\n\n",
+ stats->total_aux_partial,
+ stats->nr_events[PERF_RECORD_AUX],
+ vmm_exclusive ?
+ "\nReloading kvm_intel module with vmm_exclusive=0\n"
+ "will reduce the gaps to only guest's timeslices." :
+ "");
+ }
+
if (stats->nr_unknown_events != 0) {
ui__warning("Found %u unknown events!\n\n"
"Is this an older tool processing a perf.data "
@@ -1628,6 +1669,7 @@
buf = malloc(cur_size);
if (!buf)
return -errno;
+ ordered_events__set_copy_on_queue(oe, true);
more:
event = buf;
err = readn(fd, event, sizeof(struct perf_event_header));
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index 4bd7585..47b5e7d 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -5,14 +5,14 @@
#include "event.h"
#include "header.h"
#include "machine.h"
-#include "symbol.h"
-#include "thread.h"
#include "data.h"
#include "ordered-events.h"
+#include <linux/kernel.h>
#include <linux/rbtree.h>
#include <linux/perf_event.h>
struct ip_callchain;
+struct symbol;
struct thread;
struct auxtrace;
diff --git a/tools/perf/util/sort.c b/tools/perf/util/sort.c
index 0ff6222..5762ae4 100644
--- a/tools/perf/util/sort.c
+++ b/tools/perf/util/sort.c
@@ -1,12 +1,18 @@
+#include <errno.h>
+#include <inttypes.h>
+#include <regex.h>
#include <sys/mman.h>
#include "sort.h"
#include "hist.h"
#include "comm.h"
#include "symbol.h"
+#include "thread.h"
#include "evsel.h"
#include "evlist.h"
+#include "strlist.h"
#include <traceevent/event-parse.h>
#include "mem-events.h"
+#include <linux/kernel.h>
regex_t parent_regex;
const char default_parent_pattern[] = "^sys_|^do_page_fault";
@@ -323,7 +329,7 @@
return SRCLINE_UNKNOWN;
return get_srcline(map->dso, map__rip_2objdump(map, he->ip),
- he->ms.sym, true);
+ he->ms.sym, true, true);
}
static int64_t
@@ -366,7 +372,8 @@
left->branch_info->srcline_from = get_srcline(map->dso,
map__rip_2objdump(map,
left->branch_info->from.al_addr),
- left->branch_info->from.sym, true);
+ left->branch_info->from.sym,
+ true, true);
}
if (!right->branch_info->srcline_from) {
struct map *map = right->branch_info->from.map;
@@ -376,7 +383,8 @@
right->branch_info->srcline_from = get_srcline(map->dso,
map__rip_2objdump(map,
right->branch_info->from.al_addr),
- right->branch_info->from.sym, true);
+ right->branch_info->from.sym,
+ true, true);
}
return strcmp(right->branch_info->srcline_from, left->branch_info->srcline_from);
}
@@ -407,7 +415,8 @@
left->branch_info->srcline_to = get_srcline(map->dso,
map__rip_2objdump(map,
left->branch_info->to.al_addr),
- left->branch_info->from.sym, true);
+ left->branch_info->from.sym,
+ true, true);
}
if (!right->branch_info->srcline_to) {
struct map *map = right->branch_info->to.map;
@@ -417,7 +426,8 @@
right->branch_info->srcline_to = get_srcline(map->dso,
map__rip_2objdump(map,
right->branch_info->to.al_addr),
- right->branch_info->to.sym, true);
+ right->branch_info->to.sym,
+ true, true);
}
return strcmp(right->branch_info->srcline_to, left->branch_info->srcline_to);
}
@@ -448,7 +458,7 @@
return no_srcfile;
sf = __get_srcline(map->dso, map__rip_2objdump(map, e->ip),
- e->ms.sym, false, true);
+ e->ms.sym, false, true, true);
if (!strcmp(sf, SRCLINE_UNKNOWN))
return no_srcfile;
p = strchr(sf, ':');
@@ -536,6 +546,46 @@
.se_width_idx = HISTC_CPU,
};
+/* --sort cgroup_id */
+
+static int64_t _sort__cgroup_dev_cmp(u64 left_dev, u64 right_dev)
+{
+ return (int64_t)(right_dev - left_dev);
+}
+
+static int64_t _sort__cgroup_inode_cmp(u64 left_ino, u64 right_ino)
+{
+ return (int64_t)(right_ino - left_ino);
+}
+
+static int64_t
+sort__cgroup_id_cmp(struct hist_entry *left, struct hist_entry *right)
+{
+ int64_t ret;
+
+ ret = _sort__cgroup_dev_cmp(right->cgroup_id.dev, left->cgroup_id.dev);
+ if (ret != 0)
+ return ret;
+
+ return _sort__cgroup_inode_cmp(right->cgroup_id.ino,
+ left->cgroup_id.ino);
+}
+
+static int hist_entry__cgroup_id_snprintf(struct hist_entry *he,
+ char *bf, size_t size,
+ unsigned int width __maybe_unused)
+{
+ return repsep_snprintf(bf, size, "%lu/0x%lx", he->cgroup_id.dev,
+ he->cgroup_id.ino);
+}
+
+struct sort_entry sort_cgroup_id = {
+ .se_header = "cgroup id (dev/inode)",
+ .se_cmp = sort__cgroup_id_cmp,
+ .se_snprintf = hist_entry__cgroup_id_snprintf,
+ .se_width_idx = HISTC_CGROUP_ID,
+};
+
/* --sort socket */
static int64_t
@@ -846,6 +896,9 @@
static int64_t
sort__cycles_cmp(struct hist_entry *left, struct hist_entry *right)
{
+ if (!left->branch_info || !right->branch_info)
+ return cmp_null(left->branch_info, right->branch_info);
+
return left->branch_info->flags.cycles -
right->branch_info->flags.cycles;
}
@@ -853,6 +906,8 @@
static int hist_entry__cycles_snprintf(struct hist_entry *he, char *bf,
size_t size, unsigned int width)
{
+ if (!he->branch_info)
+ return scnprintf(bf, size, "%-.*s", width, "N/A");
if (he->branch_info->flags.cycles == 0)
return repsep_snprintf(bf, size, "%-*s", width, "-");
return repsep_snprintf(bf, size, "%-*hd", width,
@@ -1396,6 +1451,46 @@
.se_width_idx = HISTC_TRANSACTION,
};
+/* --sort symbol_size */
+
+static int64_t _sort__sym_size_cmp(struct symbol *sym_l, struct symbol *sym_r)
+{
+ int64_t size_l = sym_l != NULL ? symbol__size(sym_l) : 0;
+ int64_t size_r = sym_r != NULL ? symbol__size(sym_r) : 0;
+
+ return size_l < size_r ? -1 :
+ size_l == size_r ? 0 : 1;
+}
+
+static int64_t
+sort__sym_size_cmp(struct hist_entry *left, struct hist_entry *right)
+{
+ return _sort__sym_size_cmp(right->ms.sym, left->ms.sym);
+}
+
+static int _hist_entry__sym_size_snprintf(struct symbol *sym, char *bf,
+ size_t bf_size, unsigned int width)
+{
+ if (sym)
+ return repsep_snprintf(bf, bf_size, "%*d", width, symbol__size(sym));
+
+ return repsep_snprintf(bf, bf_size, "%*s", width, "unknown");
+}
+
+static int hist_entry__sym_size_snprintf(struct hist_entry *he, char *bf,
+ size_t size, unsigned int width)
+{
+ return _hist_entry__sym_size_snprintf(he->ms.sym, bf, size, width);
+}
+
+struct sort_entry sort_sym_size = {
+ .se_header = "Symbol size",
+ .se_cmp = sort__sym_size_cmp,
+ .se_snprintf = hist_entry__sym_size_snprintf,
+ .se_width_idx = HISTC_SYM_SIZE,
+};
+
+
struct sort_dimension {
const char *name;
struct sort_entry *entry;
@@ -1418,6 +1513,8 @@
DIM(SORT_GLOBAL_WEIGHT, "weight", sort_global_weight),
DIM(SORT_TRANSACTION, "transaction", sort_transaction),
DIM(SORT_TRACE, "trace", sort_trace),
+ DIM(SORT_SYM_SIZE, "symbol_size", sort_sym_size),
+ DIM(SORT_CGROUP_ID, "cgroup_id", sort_cgroup_id),
};
#undef DIM
diff --git a/tools/perf/util/sort.h b/tools/perf/util/sort.h
index 796c847..b7c7559 100644
--- a/tools/perf/util/sort.h
+++ b/tools/perf/util/sort.h
@@ -2,7 +2,7 @@
#define __PERF_SORT_H
#include "../builtin.h"
-#include "util.h"
+#include <regex.h>
#include "color.h"
#include <linux/list.h>
@@ -11,7 +11,6 @@
#include "symbol.h"
#include "string.h"
#include "callchain.h"
-#include "strlist.h"
#include "values.h"
#include "../perf.h"
@@ -21,7 +20,9 @@
#include <subcmd/parse-options.h>
#include "parse-events.h"
#include "hist.h"
-#include "thread.h"
+#include "srcline.h"
+
+struct thread;
extern regex_t parent_regex;
extern const char *sort_order;
@@ -54,6 +55,11 @@
u32 nr_events;
};
+struct namespace_id {
+ u64 dev;
+ u64 ino;
+};
+
struct hist_entry_diff {
bool computed;
union {
@@ -91,6 +97,7 @@
struct map_symbol ms;
struct thread *thread;
struct comm *comm;
+ struct namespace_id cgroup_id;
u64 ip;
u64 transaction;
s32 socket;
@@ -122,6 +129,7 @@
};
char *srcline;
char *srcfile;
+ struct inline_node *inline_node;
struct symbol *parent;
struct branch_info *branch_info;
struct hists *hists;
@@ -211,6 +219,8 @@
SORT_GLOBAL_WEIGHT,
SORT_TRANSACTION,
SORT_TRACE,
+ SORT_SYM_SIZE,
+ SORT_CGROUP_ID,
/* branch stack specific sort keys */
__SORT_BRANCH_STACK,
diff --git a/tools/perf/util/srcline.c b/tools/perf/util/srcline.c
index b4db3f4..df051a5 100644
--- a/tools/perf/util/srcline.c
+++ b/tools/perf/util/srcline.c
@@ -1,3 +1,4 @@
+#include <inttypes.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -7,11 +8,59 @@
#include "util/dso.h"
#include "util/util.h"
#include "util/debug.h"
+#include "util/callchain.h"
+#include "srcline.h"
#include "symbol.h"
bool srcline_full_filename;
+static const char *dso__name(struct dso *dso)
+{
+ const char *dso_name;
+
+ if (dso->symsrc_filename)
+ dso_name = dso->symsrc_filename;
+ else
+ dso_name = dso->long_name;
+
+ if (dso_name[0] == '[')
+ return NULL;
+
+ if (!strncmp(dso_name, "/tmp/perf-", 10))
+ return NULL;
+
+ return dso_name;
+}
+
+static int inline_list__append(char *filename, char *funcname, int line_nr,
+ struct inline_node *node, struct dso *dso)
+{
+ struct inline_list *ilist;
+ char *demangled;
+
+ ilist = zalloc(sizeof(*ilist));
+ if (ilist == NULL)
+ return -1;
+
+ ilist->filename = filename;
+ ilist->line_nr = line_nr;
+
+ if (dso != NULL) {
+ demangled = dso__demangle_sym(dso, 0, funcname);
+ if (demangled == NULL) {
+ ilist->funcname = funcname;
+ } else {
+ ilist->funcname = demangled;
+ free(funcname);
+ }
+ }
+
+ list_add_tail(&ilist->list, &node->val);
+
+ return 0;
+}
+
#ifdef HAVE_LIBBFD_SUPPORT
/*
@@ -151,9 +200,17 @@
#define MAX_INLINE_NEST 1024
+static void inline_list__reverse(struct inline_node *node)
+{
+ struct inline_list *ilist, *n;
+
+ list_for_each_entry_safe_reverse(ilist, n, &node->val, list)
+ list_move_tail(&ilist->list, &node->val);
+}
+
static int addr2line(const char *dso_name, u64 addr,
char **file, unsigned int *line, struct dso *dso,
- bool unwind_inlines)
+ bool unwind_inlines, struct inline_node *node)
{
int ret = 0;
struct a2l_data *a2l = dso->a2l;
@@ -178,8 +235,21 @@
while (bfd_find_inliner_info(a2l->abfd, &a2l->filename,
&a2l->funcname, &a2l->line) &&
- cnt++ < MAX_INLINE_NEST)
- ;
+ cnt++ < MAX_INLINE_NEST) {
+
+ if (node != NULL) {
+ if (inline_list__append(strdup(a2l->filename),
+ strdup(a2l->funcname),
+ a2l->line, node,
+ dso) != 0)
+ return 0;
+ }
+ }
+
+ if ((node != NULL) &&
+ (callchain_param.order != ORDER_CALLEE)) {
+ inline_list__reverse(node);
+ }
}
if (a2l->found && a2l->filename) {
@@ -205,18 +275,68 @@
dso->a2l = NULL;
}
+static struct inline_node *addr2inlines(const char *dso_name, u64 addr,
+ struct dso *dso)
+{
+ char *file = NULL;
+ unsigned int line = 0;
+ struct inline_node *node;
+
+ node = zalloc(sizeof(*node));
+ if (node == NULL) {
+ perror("not enough memory for the inline node");
+ return NULL;
+ }
+
+ INIT_LIST_HEAD(&node->val);
+ node->addr = addr;
+
+ if (!addr2line(dso_name, addr, &file, &line, dso, TRUE, node))
+ goto out_free_inline_node;
+
+ if (list_empty(&node->val))
+ goto out_free_inline_node;
+
+ return node;
+
+out_free_inline_node:
+ inline_node__delete(node);
+ return NULL;
+}
+
#else /* HAVE_LIBBFD_SUPPORT */
+static int filename_split(char *filename, unsigned int *line_nr)
+{
+ char *sep;
+
+ sep = strchr(filename, '\n');
+ if (sep)
+ *sep = '\0';
+
+ if (!strcmp(filename, "??:0"))
+ return 0;
+
+ sep = strchr(filename, ':');
+ if (sep) {
+ *sep++ = '\0';
+ *line_nr = strtoul(sep, NULL, 0);
+ return 1;
+ }
+
+ return 0;
+}
+
static int addr2line(const char *dso_name, u64 addr,
char **file, unsigned int *line_nr,
struct dso *dso __maybe_unused,
- bool unwind_inlines __maybe_unused)
+ bool unwind_inlines __maybe_unused,
+ struct inline_node *node __maybe_unused)
{
FILE *fp;
char cmd[PATH_MAX];
char *filename = NULL;
size_t len;
- char *sep;
int ret = 0;
scnprintf(cmd, sizeof(cmd), "addr2line -e %s %016"PRIx64,
@@ -233,23 +353,14 @@
goto out;
}
- sep = strchr(filename, '\n');
- if (sep)
- *sep = '\0';
-
- if (!strcmp(filename, "??:0")) {
- pr_debug("no debugging info in %s\n", dso_name);
+ ret = filename_split(filename, line_nr);
+ if (ret != 1) {
free(filename);
goto out;
}
- sep = strchr(filename, ':');
- if (sep) {
- *sep++ = '\0';
- *file = filename;
- *line_nr = strtoul(sep, NULL, 0);
- ret = 1;
- }
+ *file = filename;
+
out:
pclose(fp);
return ret;
@@ -259,6 +370,58 @@
{
}
+static struct inline_node *addr2inlines(const char *dso_name, u64 addr,
+ struct dso *dso __maybe_unused)
+{
+ FILE *fp;
+ char cmd[PATH_MAX];
+ struct inline_node *node;
+ char *filename = NULL;
+ size_t len;
+ unsigned int line_nr = 0;
+
+ scnprintf(cmd, sizeof(cmd), "addr2line -e %s -i %016"PRIx64,
+ dso_name, addr);
+
+ fp = popen(cmd, "r");
+ if (fp == NULL) {
+ pr_err("popen failed for %s\n", dso_name);
+ return NULL;
+ }
+
+ node = zalloc(sizeof(*node));
+ if (node == NULL) {
+ perror("not enough memory for the inline node");
+ goto out;
+ }
+
+ INIT_LIST_HEAD(&node->val);
+ node->addr = addr;
+
+ while (getline(&filename, &len, fp) != -1) {
+ if (filename_split(filename, &line_nr) != 1) {
+ free(filename);
+ goto out;
+ }
+
+ if (inline_list__append(filename, NULL, line_nr, node,
+ NULL) != 0)
+ goto out;
+
+ filename = NULL;
+ }
+
+out:
+ pclose(fp);
+
+ if (list_empty(&node->val)) {
+ inline_node__delete(node);
+ return NULL;
+ }
+
+ return node;
+}
+
#endif /* HAVE_LIBBFD_SUPPORT */
/*
@@ -268,7 +431,7 @@
#define A2L_FAIL_LIMIT 123
char *__get_srcline(struct dso *dso, u64 addr, struct symbol *sym,
- bool show_sym, bool unwind_inlines)
+ bool show_sym, bool show_addr, bool unwind_inlines)
{
char *file = NULL;
unsigned line = 0;
@@ -278,18 +441,11 @@
if (!dso->has_srcline)
goto out;
- if (dso->symsrc_filename)
- dso_name = dso->symsrc_filename;
- else
- dso_name = dso->long_name;
-
- if (dso_name[0] == '[')
+ dso_name = dso__name(dso);
+ if (dso_name == NULL)
goto out;
- if (!strncmp(dso_name, "/tmp/perf-", 10))
- goto out;
-
- if (!addr2line(dso_name, addr, &file, &line, dso, unwind_inlines))
+ if (!addr2line(dso_name, addr, &file, &line, dso, unwind_inlines, NULL))
goto out;
if (asprintf(&srcline, "%s:%u",
@@ -309,6 +465,11 @@
dso->has_srcline = 0;
dso__free_a2l(dso);
}
+
+ if (!show_addr)
+ return (show_sym && sym) ?
+ strndup(sym->name, sym->namelen) : NULL;
+
if (sym) {
if (asprintf(&srcline, "%s+%" PRIu64, show_sym ? sym->name : "",
addr - sym->start) < 0)
@@ -325,7 +486,32 @@
}
char *get_srcline(struct dso *dso, u64 addr, struct symbol *sym,
- bool show_sym)
+ bool show_sym, bool show_addr)
{
- return __get_srcline(dso, addr, sym, show_sym, false);
+ return __get_srcline(dso, addr, sym, show_sym, show_addr, false);
+}
+
+struct inline_node *dso__parse_addr_inlines(struct dso *dso, u64 addr)
+{
+ const char *dso_name;
+
+ dso_name = dso__name(dso);
+ if (dso_name == NULL)
+ return NULL;
+
+ return addr2inlines(dso_name, addr, dso);
+}
+
+void inline_node__delete(struct inline_node *node)
+{
+ struct inline_list *ilist, *tmp;
+
+ list_for_each_entry_safe(ilist, tmp, &node->val, list) {
+ list_del_init(&ilist->list);
+ zfree(&ilist->filename);
+ zfree(&ilist->funcname);
+ free(ilist);
+ }
+
+ free(node);
}
diff --git a/tools/perf/util/srcline.h b/tools/perf/util/srcline.h
new file mode 100644
index 0000000..7b52ba8
--- /dev/null
+++ b/tools/perf/util/srcline.h
@@ -0,0 +1,34 @@
+#ifndef PERF_SRCLINE_H
+#define PERF_SRCLINE_H
+
+#include <linux/list.h>
+#include <linux/types.h>
+
+struct dso;
+struct symbol;
+
+extern bool srcline_full_filename;
+char *get_srcline(struct dso *dso, u64 addr, struct symbol *sym,
+ bool show_sym, bool show_addr);
+char *__get_srcline(struct dso *dso, u64 addr, struct symbol *sym,
+ bool show_sym, bool show_addr, bool unwind_inlines);
+void free_srcline(char *srcline);
+
+#define SRCLINE_UNKNOWN ((char *) "??:0")
+
+struct inline_list {
+ char *filename;
+ char *funcname;
+ unsigned int line_nr;
+ struct list_head list;
+};
+
+struct inline_node {
+ u64 addr;
+ struct list_head val;
+};
+
+struct inline_node *dso__parse_addr_inlines(struct dso *dso, u64 addr);
+void inline_node__delete(struct inline_node *node);
+
+#endif /* PERF_SRCLINE_H */
diff --git a/tools/perf/util/stat-shadow.c b/tools/perf/util/stat-shadow.c
index 8a2bbd2..ac10cc6 100644
--- a/tools/perf/util/stat-shadow.c
+++ b/tools/perf/util/stat-shadow.c
@@ -3,6 +3,9 @@
#include "stat.h"
#include "color.h"
#include "pmu.h"
+#include "rblist.h"
+#include "evlist.h"
+#include "expr.h"
enum {
CTX_BIT_USER = 1 << 0,
@@ -41,13 +44,73 @@
static struct stats runtime_topdown_slots_retired[NUM_CTX][MAX_NR_CPUS];
static struct stats runtime_topdown_fetch_bubbles[NUM_CTX][MAX_NR_CPUS];
static struct stats runtime_topdown_recovery_bubbles[NUM_CTX][MAX_NR_CPUS];
+static struct rblist runtime_saved_values;
static bool have_frontend_stalled;
struct stats walltime_nsecs_stats;
+struct saved_value {
+ struct rb_node rb_node;
+ struct perf_evsel *evsel;
+ int cpu;
+ int ctx;
+ struct stats stats;
+};
+
+static int saved_value_cmp(struct rb_node *rb_node, const void *entry)
+{
+ struct saved_value *a = container_of(rb_node,
+ struct saved_value,
+ rb_node);
+ const struct saved_value *b = entry;
+
+ if (a->ctx != b->ctx)
+ return a->ctx - b->ctx;
+ if (a->cpu != b->cpu)
+ return a->cpu - b->cpu;
+ return a->evsel - b->evsel;
+}
+
+static struct rb_node *saved_value_new(struct rblist *rblist __maybe_unused,
+ const void *entry)
+{
+ struct saved_value *nd = malloc(sizeof(struct saved_value));
+
+ if (!nd)
+ return NULL;
+ memcpy(nd, entry, sizeof(struct saved_value));
+ return &nd->rb_node;
+}
+
+static struct saved_value *saved_value_lookup(struct perf_evsel *evsel,
+ int cpu, int ctx,
+ bool create)
+{
+ struct rb_node *nd;
+ struct saved_value dm = {
+ .cpu = cpu,
+ .ctx = ctx,
+ .evsel = evsel,
+ };
+ nd = rblist__find(&runtime_saved_values, &dm);
+ if (nd)
+ return container_of(nd, struct saved_value, rb_node);
+ if (create) {
+ rblist__add_node(&runtime_saved_values, &dm);
+ nd = rblist__find(&runtime_saved_values, &dm);
+ if (nd)
+ return container_of(nd, struct saved_value, rb_node);
+ }
+ return NULL;
+}
+
void perf_stat__init_shadow_stats(void)
{
have_frontend_stalled = pmu_have_event("cpu", "stalled-cycles-frontend");
+ rblist__init(&runtime_saved_values);
+ runtime_saved_values.node_cmp = saved_value_cmp;
+ runtime_saved_values.node_new = saved_value_new;
+ /* No delete for now */
}
static int evsel_context(struct perf_evsel *evsel)
@@ -70,6 +133,8 @@
void perf_stat__reset_shadow_stats(void)
{
+ struct rb_node *pos, *next;
+
memset(runtime_nsecs_stats, 0, sizeof(runtime_nsecs_stats));
memset(runtime_cycles_stats, 0, sizeof(runtime_cycles_stats));
memset(runtime_stalled_cycles_front_stats, 0, sizeof(runtime_stalled_cycles_front_stats));
@@ -92,6 +157,15 @@
memset(runtime_topdown_slots_issued, 0, sizeof(runtime_topdown_slots_issued));
memset(runtime_topdown_fetch_bubbles, 0, sizeof(runtime_topdown_fetch_bubbles));
memset(runtime_topdown_recovery_bubbles, 0, sizeof(runtime_topdown_recovery_bubbles));
+
+ next = rb_first(&runtime_saved_values.entries);
+ while (next) {
+ pos = next;
+ next = rb_next(pos);
+ memset(&container_of(pos, struct saved_value, rb_node)->stats,
+ 0,
+ sizeof(struct stats));
+ }
}
/*
@@ -143,6 +217,12 @@
update_stats(&runtime_dtlb_cache_stats[ctx][cpu], count[0]);
else if (perf_evsel__match(counter, HW_CACHE, HW_CACHE_ITLB))
update_stats(&runtime_itlb_cache_stats[ctx][cpu], count[0]);
+
+ if (counter->collect_stat) {
+ struct saved_value *v = saved_value_lookup(counter, cpu, ctx,
+ true);
+ update_stats(&v->stats, count[0]);
+ }
}
/* used for get_ratio_color() */
@@ -172,6 +252,95 @@
return color;
}
+static struct perf_evsel *perf_stat__find_event(struct perf_evlist *evsel_list,
+ const char *name)
+{
+ struct perf_evsel *c2;
+
+ evlist__for_each_entry (evsel_list, c2) {
+ if (!strcasecmp(c2->name, name))
+ return c2;
+ }
+ return NULL;
+}
+
+/* Mark MetricExpr target events and link events using them to them. */
+void perf_stat__collect_metric_expr(struct perf_evlist *evsel_list)
+{
+ struct perf_evsel *counter, *leader, **metric_events, *oc;
+ bool found;
+ const char **metric_names;
+ int i;
+ int num_metric_names;
+
+ evlist__for_each_entry(evsel_list, counter) {
+ bool invalid = false;
+
+ leader = counter->leader;
+ if (!counter->metric_expr)
+ continue;
+ metric_events = counter->metric_events;
+ if (!metric_events) {
+ if (expr__find_other(counter->metric_expr, counter->name,
+ &metric_names, &num_metric_names) < 0)
+ continue;
+
+ metric_events = calloc(sizeof(struct perf_evsel *),
+ num_metric_names + 1);
+ if (!metric_events)
+ return;
+ counter->metric_events = metric_events;
+ }
+
+ for (i = 0; i < num_metric_names; i++) {
+ found = false;
+ if (leader) {
+ /* Search in group */
+ for_each_group_member (oc, leader) {
+ if (!strcasecmp(oc->name, metric_names[i])) {
+ found = true;
+ break;
+ }
+ }
+ }
+ if (!found) {
+ /* Search ignoring groups */
+ oc = perf_stat__find_event(evsel_list, metric_names[i]);
+ }
+ if (!oc) {
+ /* Deduping one is good enough to handle duplicated PMUs. */
+ static char *printed;
+
+ /*
+ * Adding events automatically would be difficult, because
+ * it would risk creating groups that are not schedulable.
+ * perf stat doesn't understand all the scheduling constraints
+ * of events. So we ask the user instead to add the missing
+ * events.
+ */
+ if (!printed || strcasecmp(printed, metric_names[i])) {
+ fprintf(stderr,
+ "Add %s event to groups to get metric expression for %s\n",
+ metric_names[i],
+ counter->name);
+ printed = strdup(metric_names[i]);
+ }
+ invalid = true;
+ continue;
+ }
+ metric_events[i] = oc;
+ oc->collect_stat = true;
+ }
+ metric_events[i] = NULL;
+ free(metric_names);
+ if (invalid) {
+ free(metric_events);
+ counter->metric_events = NULL;
+ counter->metric_expr = NULL;
+ }
+ }
+}
+
static void print_stalled_cycles_frontend(int cpu,
struct perf_evsel *evsel, double avg,
struct perf_stat_output_ctx *out)
@@ -614,6 +783,34 @@
be_bound * 100.);
else
print_metric(ctxp, NULL, NULL, name, 0);
+ } else if (evsel->metric_expr) {
+ struct parse_ctx pctx;
+ int i;
+
+ expr__ctx_init(&pctx);
+ expr__add_id(&pctx, evsel->name, avg);
+ for (i = 0; evsel->metric_events[i]; i++) {
+ struct saved_value *v;
+
+ v = saved_value_lookup(evsel->metric_events[i], cpu, ctx, false);
+ if (!v)
+ break;
+ expr__add_id(&pctx, evsel->metric_events[i]->name,
+ avg_stats(&v->stats));
+ }
+ if (!evsel->metric_events[i]) {
+ const char *p = evsel->metric_expr;
+
+ if (expr__parse(&ratio, &pctx, &p) == 0)
+ print_metric(ctxp, NULL, "%8.1f",
+ evsel->metric_name ?
+ evsel->metric_name :
+ out->force_header ? evsel->name : "",
+ ratio);
+ else
+ print_metric(ctxp, NULL, NULL, "", 0);
+ } else
+ print_metric(ctxp, NULL, NULL, "", 0);
} else if (runtime_nsecs_stats[cpu].n != 0) {
char unit = 'M';
char unit_buf[10];
diff --git a/tools/perf/util/stat.c b/tools/perf/util/stat.c
index 0d51334..c581744 100644
--- a/tools/perf/util/stat.c
+++ b/tools/perf/util/stat.c
@@ -1,3 +1,5 @@
+#include <errno.h>
+#include <inttypes.h>
#include <math.h>
#include "stat.h"
#include "evlist.h"
diff --git a/tools/perf/util/stat.h b/tools/perf/util/stat.h
index c29bb94..0a65ae2 100644
--- a/tools/perf/util/stat.h
+++ b/tools/perf/util/stat.h
@@ -85,11 +85,13 @@
void *ctx;
print_metric_t print_metric;
new_line_t new_line;
+ bool force_header;
};
void perf_stat__print_shadow_stats(struct perf_evsel *evsel,
double avg, int cpu,
struct perf_stat_output_ctx *out);
+void perf_stat__collect_metric_expr(struct perf_evlist *);
int perf_evlist__alloc_stats(struct perf_evlist *evlist, bool alloc_raw);
void perf_evlist__free_stats(struct perf_evlist *evlist);
diff --git a/tools/perf/util/strbuf.c b/tools/perf/util/strbuf.c
index 81759390..e91b5e8 100644
--- a/tools/perf/util/strbuf.c
+++ b/tools/perf/util/strbuf.c
@@ -1,6 +1,7 @@
#include "debug.h"
#include "util.h"
#include <linux/kernel.h>
+#include <errno.h>
int prefixcmp(const char *str, const char *prefix)
{
diff --git a/tools/perf/util/strfilter.c b/tools/perf/util/strfilter.c
index efb5377..4dc0af6 100644
--- a/tools/perf/util/strfilter.c
+++ b/tools/perf/util/strfilter.c
@@ -1,7 +1,10 @@
#include "util.h"
-#include "string.h"
+#include "string2.h"
#include "strfilter.h"
+#include <errno.h>
+#include "sane_ctype.h"
+
/* Operators */
static const char *OP_and = "&"; /* Logical AND */
static const char *OP_or = "|"; /* Logical OR */
diff --git a/tools/perf/util/string.c b/tools/perf/util/string.c
index bddca51..cca53b6 100644
--- a/tools/perf/util/string.c
+++ b/tools/perf/util/string.c
@@ -1,5 +1,9 @@
-#include "util.h"
-#include "linux/string.h"
+#include "string2.h"
+#include <linux/kernel.h>
+#include <linux/string.h>
+#include <stdlib.h>
+
+#include "sane_ctype.h"
#define K 1024LL
/*
@@ -99,8 +103,10 @@
void argv_free(char **argv)
{
char **p;
- for (p = argv; *p; p++)
- zfree(p);
+ for (p = argv; *p; p++) {
+ free(*p);
+ *p = NULL;
+ }
free(argv);
}
@@ -120,7 +126,7 @@
char **argv_split(const char *str, int *argcp)
{
int argc = count_argc(str);
- char **argv = zalloc(sizeof(*argv) * (argc+1));
+ char **argv = calloc(argc + 1, sizeof(*argv));
char **argvp;
if (argv == NULL)
@@ -322,12 +328,8 @@
*/
char *ltrim(char *s)
{
- int len = strlen(s);
-
- while (len && isspace(*s)) {
- len--;
+ while (isspace(*s))
s++;
- }
return s;
}
@@ -381,7 +383,7 @@
goto out_err_overflow;
if (i > 0)
- printed += snprintf(e + printed, size - printed, " %s ", or_and);
+ printed += scnprintf(e + printed, size - printed, " %s ", or_and);
printed += scnprintf(e + printed, size - printed,
"%s %s %d", var, eq_neq, ints[i]);
}
diff --git a/tools/perf/util/string2.h b/tools/perf/util/string2.h
new file mode 100644
index 0000000..2f61968
--- /dev/null
+++ b/tools/perf/util/string2.h
@@ -0,0 +1,42 @@
+#ifndef PERF_STRING_H
+#define PERF_STRING_H
+
+#include <linux/types.h>
+#include <stddef.h>
+#include <string.h>
+
+s64 perf_atoll(const char *str);
+char **argv_split(const char *str, int *argcp);
+void argv_free(char **argv);
+bool strglobmatch(const char *str, const char *pat);
+bool strglobmatch_nocase(const char *str, const char *pat);
+bool strlazymatch(const char *str, const char *pat);
+static inline bool strisglob(const char *str)
+{
+ return strpbrk(str, "*?[") != NULL;
+}
+int strtailcmp(const char *s1, const char *s2);
+char *strxfrchar(char *s, char from, char to);
+
+char *ltrim(char *s);
+char *rtrim(char *s);
+
+static inline char *trim(char *s)
+{
+ return ltrim(rtrim(s));
+}
+
+char *asprintf_expr_inout_ints(const char *var, bool in, size_t nints, int *ints);
+
+static inline char *asprintf_expr_in_ints(const char *var, size_t nints, int *ints)
+{
+ return asprintf_expr_inout_ints(var, true, nints, ints);
+}
+
+static inline char *asprintf_expr_not_in_ints(const char *var, size_t nints, int *ints)
+{
+ return asprintf_expr_inout_ints(var, false, nints, ints);
+}
+
+
+#endif /* PERF_STRING_H */
diff --git a/tools/perf/util/strlist.c b/tools/perf/util/strlist.c
index 0d3dfcb..9de5434 100644
--- a/tools/perf/util/strlist.c
+++ b/tools/perf/util/strlist.c
@@ -10,6 +10,7 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
+#include <unistd.h>
static
struct rb_node *strlist__node_new(struct rblist *rblist, const void *entry)
diff --git a/tools/perf/util/symbol-elf.c b/tools/perf/util/symbol-elf.c
index 4e59dde..e7ee47f 100644
--- a/tools/perf/util/symbol-elf.c
+++ b/tools/perf/util/symbol-elf.c
@@ -10,8 +10,9 @@
#include "demangle-rust.h"
#include "machine.h"
#include "vdso.h"
-#include <symbol/kallsyms.h>
#include "debug.h"
+#include "sane_ctype.h"
+#include <symbol/kallsyms.h>
#ifndef EM_AARCH64
#define EM_AARCH64 183 /* ARM 64 bit */
@@ -390,6 +391,11 @@
return 0;
}
+char *dso__demangle_sym(struct dso *dso, int kmodule, char *elf_name)
+{
+ return demangle_sym(dso, kmodule, elf_name);
+}
+
/*
* Align offset to 4 bytes as needed for note name and descriptor data.
*/
@@ -1828,7 +1834,7 @@
static int populate_sdt_note(Elf **elf, const char *data, size_t len,
struct list_head *sdt_notes)
{
- const char *provider, *name;
+ const char *provider, *name, *args;
struct sdt_note *tmp = NULL;
GElf_Ehdr ehdr;
GElf_Addr base_off = 0;
@@ -1887,6 +1893,25 @@
goto out_free_prov;
}
+ args = memchr(name, '\0', data + len - name);
+
+ /*
+ * There is no argument if:
+ * - We reached the end of the note;
+ * - There is not enough room to hold a potential string;
+ * - The argument string is empty or just contains ':'.
+ */
+ if (args == NULL || data + len - args < 2 ||
+ args[1] == ':' || args[1] == '\0')
+ tmp->args = NULL;
+ else {
+ tmp->args = strdup(++args);
+ if (!tmp->args) {
+ ret = -ENOMEM;
+ goto out_free_name;
+ }
+ }
+
if (gelf_getclass(*elf) == ELFCLASS32) {
memcpy(&tmp->addr, &buf, 3 * sizeof(Elf32_Addr));
tmp->bit32 = true;
@@ -1898,7 +1923,7 @@
if (!gelf_getehdr(*elf, &ehdr)) {
pr_debug("%s : cannot get elf header.\n", __func__);
ret = -EBADF;
- goto out_free_name;
+ goto out_free_args;
}
/* Adjust the prelink effect :
@@ -1923,6 +1948,8 @@
list_add_tail(&tmp->note_list, sdt_notes);
return 0;
+out_free_args:
+ free(tmp->args);
out_free_name:
free(tmp->name);
out_free_prov:
diff --git a/tools/perf/util/symbol-minimal.c b/tools/perf/util/symbol-minimal.c
index 11cdde9..40bf5d4 100644
--- a/tools/perf/util/symbol-minimal.c
+++ b/tools/perf/util/symbol-minimal.c
@@ -1,6 +1,7 @@
#include "symbol.h"
#include "util.h"
+#include <errno.h>
#include <stdio.h>
#include <fcntl.h>
#include <string.h>
@@ -373,3 +374,10 @@
void symbol__elf_init(void)
{
}
+
+char *dso__demangle_sym(struct dso *dso __maybe_unused,
+ int kmodule __maybe_unused,
+ char *elf_name __maybe_unused)
+{
+ return NULL;
+}
diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index 9b4d8ba..2cb7665 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -3,6 +3,7 @@
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
+#include <linux/kernel.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/param.h>
@@ -18,6 +19,8 @@
#include "strlist.h"
#include "intlist.h"
#include "header.h"
+#include "path.h"
+#include "sane_ctype.h"
#include <elf.h>
#include <limits.h>
diff --git a/tools/perf/util/symbol.h b/tools/perf/util/symbol.h
index 6c358b7..7acd70f 100644
--- a/tools/perf/util/symbol.h
+++ b/tools/perf/util/symbol.h
@@ -13,7 +13,7 @@
#include <libgen.h>
#include "build-id.h"
#include "event.h"
-#include "util.h"
+#include "path.h"
#ifdef HAVE_LIBELF_SUPPORT
#include <libelf.h>
@@ -118,7 +118,8 @@
show_ref_callgraph,
hide_unresolved,
raw_trace,
- report_hierarchy;
+ report_hierarchy,
+ inline_name;
const char *vmlinux_name,
*kallsyms_name,
*source_prefix,
@@ -305,6 +306,8 @@
int dso__synthesize_plt_symbols(struct dso *dso, struct symsrc *ss,
struct map *map);
+char *dso__demangle_sym(struct dso *dso, int kmodule, char *elf_name);
+
void __symbols__insert(struct rb_root *symbols, struct symbol *sym, bool kernel);
void symbols__insert(struct rb_root *symbols, struct symbol *sym);
void symbols__fixup_duplicate(struct rb_root *symbols);
@@ -351,6 +354,7 @@
struct sdt_note {
char *name; /* name of the note*/
char *provider; /* provider name */
+ char *args;
bool bit32; /* whether the location is 32 bits? */
union { /* location, base and semaphore addrs */
Elf64_Addr a64[3];
diff --git a/tools/perf/util/term.c b/tools/perf/util/term.c
index 90b47d8..8f254a7 100644
--- a/tools/perf/util/term.c
+++ b/tools/perf/util/term.c
@@ -1,4 +1,8 @@
-#include "util.h"
+#include "term.h"
+#include <stdlib.h>
+#include <termios.h>
+#include <unistd.h>
+#include <sys/ioctl.h>
void get_term_dimensions(struct winsize *ws)
{
diff --git a/tools/perf/util/thread-stack.c b/tools/perf/util/thread-stack.c
index d330152..dd17d6a 100644
--- a/tools/perf/util/thread-stack.c
+++ b/tools/perf/util/thread-stack.c
@@ -15,6 +15,7 @@
#include <linux/rbtree.h>
#include <linux/list.h>
+#include <errno.h>
#include "thread.h"
#include "event.h"
#include "machine.h"
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index f5af87f..378c418 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -1,12 +1,15 @@
#include "../perf.h"
+#include <errno.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
+#include <linux/kernel.h>
#include "session.h"
#include "thread.h"
#include "thread-stack.h"
#include "util.h"
#include "debug.h"
+#include "namespaces.h"
#include "comm.h"
#include "unwind.h"
@@ -40,6 +43,7 @@
thread->tid = tid;
thread->ppid = -1;
thread->cpu = -1;
+ INIT_LIST_HEAD(&thread->namespaces_list);
INIT_LIST_HEAD(&thread->comm_list);
comm_str = malloc(32);
@@ -53,7 +57,7 @@
goto err_thread;
list_add(&comm->list, &thread->comm_list);
- atomic_set(&thread->refcnt, 1);
+ refcount_set(&thread->refcnt, 1);
RB_CLEAR_NODE(&thread->rb_node);
}
@@ -66,7 +70,8 @@
void thread__delete(struct thread *thread)
{
- struct comm *comm, *tmp;
+ struct namespaces *namespaces, *tmp_namespaces;
+ struct comm *comm, *tmp_comm;
BUG_ON(!RB_EMPTY_NODE(&thread->rb_node));
@@ -76,7 +81,12 @@
map_groups__put(thread->mg);
thread->mg = NULL;
}
- list_for_each_entry_safe(comm, tmp, &thread->comm_list, list) {
+ list_for_each_entry_safe(namespaces, tmp_namespaces,
+ &thread->namespaces_list, list) {
+ list_del(&namespaces->list);
+ namespaces__free(namespaces);
+ }
+ list_for_each_entry_safe(comm, tmp_comm, &thread->comm_list, list) {
list_del(&comm->list);
comm__free(comm);
}
@@ -88,13 +98,13 @@
struct thread *thread__get(struct thread *thread)
{
if (thread)
- atomic_inc(&thread->refcnt);
+ refcount_inc(&thread->refcnt);
return thread;
}
void thread__put(struct thread *thread)
{
- if (thread && atomic_dec_and_test(&thread->refcnt)) {
+ if (thread && refcount_dec_and_test(&thread->refcnt)) {
/*
* Remove it from the dead_threads list, as last reference
* is gone.
@@ -104,6 +114,38 @@
}
}
+struct namespaces *thread__namespaces(const struct thread *thread)
+{
+ if (list_empty(&thread->namespaces_list))
+ return NULL;
+
+ return list_first_entry(&thread->namespaces_list, struct namespaces, list);
+}
+
+int thread__set_namespaces(struct thread *thread, u64 timestamp,
+ struct namespaces_event *event)
+{
+ struct namespaces *new, *curr = thread__namespaces(thread);
+
+ new = namespaces__new(event);
+ if (!new)
+ return -ENOMEM;
+
+ list_add(&new->list, &thread->namespaces_list);
+
+ if (timestamp && curr) {
+ /*
+ * setns syscall must have changed few or all the namespaces
+ * of this thread. Update end time for the namespaces
+ * previously used.
+ */
+ curr = list_next_entry(new, list);
+ curr->end_time = timestamp;
+ }
+
+ return 0;
+}
+
struct comm *thread__comm(const struct thread *thread)
{
if (list_empty(&thread->comm_list))
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index 99263cb..4eb849e 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -1,7 +1,7 @@
#ifndef __PERF_THREAD_H
#define __PERF_THREAD_H
-#include <linux/atomic.h>
+#include <linux/refcount.h>
#include <linux/rbtree.h>
#include <linux/list.h>
#include <unistd.h>
@@ -23,11 +23,12 @@
pid_t tid;
pid_t ppid;
int cpu;
- atomic_t refcnt;
+ refcount_t refcnt;
char shortname[3];
bool comm_set;
int comm_len;
bool dead; /* if set thread has exited */
+ struct list_head namespaces_list;
struct list_head comm_list;
u64 db_id;
@@ -40,6 +41,7 @@
};
struct machine;
+struct namespaces;
struct comm;
struct thread *thread__new(pid_t pid, pid_t tid);
@@ -62,6 +64,10 @@
thread->dead = true;
}
+struct namespaces *thread__namespaces(const struct thread *thread);
+int thread__set_namespaces(struct thread *thread, u64 timestamp,
+ struct namespaces_event *event);
+
int __thread__set_comm(struct thread *thread, const char *comm, u64 timestamp,
bool exec);
static inline int thread__set_comm(struct thread *thread, const char *comm,
diff --git a/tools/perf/util/thread_map.c b/tools/perf/util/thread_map.c
index 7c3fcc5..63ead7b 100644
--- a/tools/perf/util/thread_map.c
+++ b/tools/perf/util/thread_map.c
@@ -1,4 +1,5 @@
#include <dirent.h>
+#include <errno.h>
#include <limits.h>
#include <stdbool.h>
#include <stdlib.h>
@@ -6,6 +7,7 @@
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
+#include "string2.h"
#include "strlist.h"
#include <string.h>
#include <api/fs/fs.h>
@@ -66,7 +68,7 @@
for (i = 0; i < items; i++)
thread_map__set_pid(threads, i, atoi(namelist[i]->d_name));
threads->nr = items;
- atomic_set(&threads->refcnt, 1);
+ refcount_set(&threads->refcnt, 1);
}
for (i=0; i<items; i++)
@@ -83,7 +85,7 @@
if (threads != NULL) {
thread_map__set_pid(threads, 0, tid);
threads->nr = 1;
- atomic_set(&threads->refcnt, 1);
+ refcount_set(&threads->refcnt, 1);
}
return threads;
@@ -105,7 +107,7 @@
goto out_free_threads;
threads->nr = 0;
- atomic_set(&threads->refcnt, 1);
+ refcount_set(&threads->refcnt, 1);
while ((dirent = readdir(proc)) != NULL) {
char *end;
@@ -235,7 +237,7 @@
out:
strlist__delete(slist);
if (threads)
- atomic_set(&threads->refcnt, 1);
+ refcount_set(&threads->refcnt, 1);
return threads;
out_free_namelist:
@@ -255,7 +257,7 @@
if (threads != NULL) {
thread_map__set_pid(threads, 0, -1);
threads->nr = 1;
- atomic_set(&threads->refcnt, 1);
+ refcount_set(&threads->refcnt, 1);
}
return threads;
}
@@ -300,7 +302,7 @@
}
out:
if (threads)
- atomic_set(&threads->refcnt, 1);
+ refcount_set(&threads->refcnt, 1);
return threads;
out_free_threads:
@@ -326,7 +328,7 @@
if (threads) {
int i;
- WARN_ONCE(atomic_read(&threads->refcnt) != 0,
+ WARN_ONCE(refcount_read(&threads->refcnt) != 0,
"thread map refcnt unbalanced\n");
for (i = 0; i < threads->nr; i++)
free(thread_map__comm(threads, i));
@@ -337,13 +339,13 @@
struct thread_map *thread_map__get(struct thread_map *map)
{
if (map)
- atomic_inc(&map->refcnt);
+ refcount_inc(&map->refcnt);
return map;
}
void thread_map__put(struct thread_map *map)
{
- if (map && atomic_dec_and_test(&map->refcnt))
+ if (map && refcount_dec_and_test(&map->refcnt))
thread_map__delete(map);
}
@@ -423,7 +425,7 @@
threads->map[i].comm = strndup(event->entries[i].comm, 16);
}
- atomic_set(&threads->refcnt, 1);
+ refcount_set(&threads->refcnt, 1);
}
struct thread_map *thread_map__new_event(struct thread_map_event *event)
diff --git a/tools/perf/util/thread_map.h b/tools/perf/util/thread_map.h
index ea0ef08..bd34d7a 100644
--- a/tools/perf/util/thread_map.h
+++ b/tools/perf/util/thread_map.h
@@ -3,7 +3,7 @@
#include <sys/types.h>
#include <stdio.h>
-#include <linux/atomic.h>
+#include <linux/refcount.h>
struct thread_map_data {
pid_t pid;
@@ -11,7 +11,7 @@
};
struct thread_map {
- atomic_t refcnt;
+ refcount_t refcnt;
int nr;
struct thread_map_data map[];
};
diff --git a/tools/perf/util/time-utils.c b/tools/perf/util/time-utils.c
index d1b21c7..5b5d021 100644
--- a/tools/perf/util/time-utils.c
+++ b/tools/perf/util/time-utils.c
@@ -117,3 +117,28 @@
return false;
}
+
+int timestamp__scnprintf_usec(u64 timestamp, char *buf, size_t sz)
+{
+ u64 sec = timestamp / NSEC_PER_SEC;
+ u64 usec = (timestamp % NSEC_PER_SEC) / NSEC_PER_USEC;
+
+ return scnprintf(buf, sz, "%"PRIu64".%06"PRIu64, sec, usec);
+}
+
+int fetch_current_timestamp(char *buf, size_t sz)
+{
+ struct timeval tv;
+ struct tm tm;
+ char dt[32];
+
+ if (gettimeofday(&tv, NULL) || !localtime_r(&tv.tv_sec, &tm))
+ return -1;
+
+ if (!strftime(dt, sizeof(dt), "%Y%m%d%H%M%S", &tm))
+ return -1;
+
+ scnprintf(buf, sz, "%s%02u", dt, (unsigned)tv.tv_usec / 10000);
+
+ return 0;
+}
diff --git a/tools/perf/util/time-utils.h b/tools/perf/util/time-utils.h
index c1f197c..8656be0 100644
--- a/tools/perf/util/time-utils.h
+++ b/tools/perf/util/time-utils.h
@@ -1,6 +1,9 @@
#ifndef _TIME_UTILS_H_
#define _TIME_UTILS_H_
+#include <stddef.h>
+#include <linux/types.h>
+
struct perf_time_interval {
u64 start, end;
};
@@ -11,4 +14,8 @@
bool perf_time__skip_sample(struct perf_time_interval *ptime, u64 timestamp);
+int timestamp__scnprintf_usec(u64 timestamp, char *buf, size_t sz);
+
+int fetch_current_timestamp(char *buf, size_t sz);
+
#endif
diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
index ac2590a..829471a 100644
--- a/tools/perf/util/tool.h
+++ b/tools/perf/util/tool.h
@@ -40,6 +40,7 @@
event_op mmap,
mmap2,
comm,
+ namespaces,
fork,
exit,
lost,
@@ -66,6 +67,7 @@
event_op3 auxtrace;
bool ordered_events;
bool ordering_requires_timestamps;
+ bool namespace_events;
};
#endif /* __PERF_TOOL_H */
diff --git a/tools/perf/util/top.h b/tools/perf/util/top.h
index b2940c8..9bdfb78 100644
--- a/tools/perf/util/top.h
+++ b/tools/perf/util/top.h
@@ -5,7 +5,7 @@
#include <linux/types.h>
#include <stddef.h>
#include <stdbool.h>
-#include <termios.h>
+#include <sys/ioctl.h>
struct perf_evlist;
struct perf_evsel;
diff --git a/tools/perf/util/trace-event-parse.c b/tools/perf/util/trace-event-parse.c
index de0078e..746bbee 100644
--- a/tools/perf/util/trace-event-parse.c
+++ b/tools/perf/util/trace-event-parse.c
@@ -21,13 +21,14 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
-#include <ctype.h>
#include <errno.h>
#include "../perf.h"
#include "util.h"
#include "trace-event.h"
+#include "sane_ctype.h"
+
static int get_common_field(struct scripting_context *context,
int *offset, int *size, const char *type)
{
diff --git a/tools/perf/util/trace-event-read.c b/tools/perf/util/trace-event-read.c
index 2742015..8a9a677 100644
--- a/tools/perf/util/trace-event-read.c
+++ b/tools/perf/util/trace-event-read.c
@@ -192,7 +192,7 @@
if (!size)
return 0;
- buf = malloc(size);
+ buf = malloc(size + 1);
if (buf == NULL)
return -1;
@@ -201,6 +201,8 @@
return -1;
}
+ buf[size] = '\0';
+
parse_ftrace_printk(pevent, buf, size);
free(buf);
diff --git a/tools/perf/util/units.c b/tools/perf/util/units.c
new file mode 100644
index 0000000..f6a2a3d
--- /dev/null
+++ b/tools/perf/util/units.c
@@ -0,0 +1,39 @@
+#include "units.h"
+#include <inttypes.h>
+#include <linux/kernel.h>
+#include <linux/time64.h>
+
+unsigned long convert_unit(unsigned long value, char *unit)
+{
+ *unit = ' ';
+
+ if (value > 1000) {
+ value /= 1000;
+ *unit = 'K';
+ }
+
+ if (value > 1000) {
+ value /= 1000;
+ *unit = 'M';
+ }
+
+ if (value > 1000) {
+ value /= 1000;
+ *unit = 'G';
+ }
+
+ return value;
+}
+
+int unit_number__scnprintf(char *buf, size_t size, u64 n)
+{
+ char unit[4] = "BKMG";
+ int i = 0;
+
+ while (((n / 1024) > 1) && (i < 3)) {
+ n /= 1024;
+ i++;
+ }
+
+ return scnprintf(buf, size, "%" PRIu64 "%c", n, unit[i]);
+}
diff --git a/tools/perf/util/units.h b/tools/perf/util/units.h
new file mode 100644
index 0000000..3ed7774
--- /dev/null
+++ b/tools/perf/util/units.h
@@ -0,0 +1,10 @@
+#ifndef PERF_UNIT_H
+#define PERF_UNIT_H
+
+#include <stddef.h>
+#include <linux/types.h>
+
+unsigned long convert_unit(unsigned long value, char *unit);
+int unit_number__scnprintf(char *buf, size_t size, u64 n);
+
+#endif /* PERF_UNIT_H */
diff --git a/tools/perf/util/unwind-libdw.c b/tools/perf/util/unwind-libdw.c
index 783a53f..f90e11a 100644
--- a/tools/perf/util/unwind-libdw.c
+++ b/tools/perf/util/unwind-libdw.c
@@ -12,6 +12,7 @@
#include "event.h"
#include "perf_regs.h"
#include "callchain.h"
+#include "util.h"
static char *debuginfo_path;
diff --git a/tools/perf/util/unwind-libdw.h b/tools/perf/util/unwind-libdw.h
index 5832866..4a2b269 100644
--- a/tools/perf/util/unwind-libdw.h
+++ b/tools/perf/util/unwind-libdw.h
@@ -2,10 +2,12 @@
#define __PERF_UNWIND_LIBDW_H
#include <elfutils/libdwfl.h>
-#include "event.h"
-#include "thread.h"
#include "unwind.h"
+struct machine;
+struct perf_sample;
+struct thread;
+
bool libdw__arch_set_initial_registers(Dwfl_Thread *thread, void *arg);
struct unwind_info {
diff --git a/tools/perf/util/unwind-libunwind-local.c b/tools/perf/util/unwind-libunwind-local.c
index bfb9b79..f8455be 100644
--- a/tools/perf/util/unwind-libunwind-local.c
+++ b/tools/perf/util/unwind-libunwind-local.c
@@ -16,8 +16,10 @@
*/
#include <elf.h>
+#include <errno.h>
#include <gelf.h>
#include <fcntl.h>
+#include <inttypes.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>
diff --git a/tools/perf/util/unwind.h b/tools/perf/util/unwind.h
index 61fb1e9..bfbdcc6 100644
--- a/tools/perf/util/unwind.h
+++ b/tools/perf/util/unwind.h
@@ -1,10 +1,13 @@
#ifndef __UNWIND_H
#define __UNWIND_H
+#include <linux/compiler.h>
#include <linux/types.h>
-#include "event.h"
-#include "symbol.h"
-#include "thread.h"
+
+struct map;
+struct perf_sample;
+struct symbol;
+struct thread;
struct unwind_entry {
struct map *map;
diff --git a/tools/perf/util/util.c b/tools/perf/util/util.c
index d8b45ce..6450c75 100644
--- a/tools/perf/util/util.c
+++ b/tools/perf/util/util.c
@@ -3,10 +3,11 @@
#include "debug.h"
#include <api/fs/fs.h>
#include <sys/mman.h>
+#include <sys/stat.h>
#include <sys/utsname.h>
-#ifdef HAVE_BACKTRACE_SUPPORT
-#include <execinfo.h>
-#endif
+#include <dirent.h>
+#include <inttypes.h>
+#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -17,24 +18,8 @@
#include <linux/log2.h>
#include <linux/time64.h>
#include <unistd.h>
-#include "callchain.h"
#include "strlist.h"
-#define CALLCHAIN_PARAM_DEFAULT \
- .mode = CHAIN_GRAPH_ABS, \
- .min_percent = 0.5, \
- .order = ORDER_CALLEE, \
- .key = CCKEY_FUNCTION, \
- .value = CCVAL_PERCENT, \
-
-struct callchain_param callchain_param = {
- CALLCHAIN_PARAM_DEFAULT
-};
-
-struct callchain_param callchain_param_default = {
- CALLCHAIN_PARAM_DEFAULT
-};
-
/*
* XXX We need to find a better place for these things...
*/
@@ -269,28 +254,6 @@
return copyfile_mode(from, to, 0755);
}
-unsigned long convert_unit(unsigned long value, char *unit)
-{
- *unit = ' ';
-
- if (value > 1000) {
- value /= 1000;
- *unit = 'K';
- }
-
- if (value > 1000) {
- value /= 1000;
- *unit = 'M';
- }
-
- if (value > 1000) {
- value /= 1000;
- *unit = 'G';
- }
-
- return value;
-}
-
static ssize_t ion(bool is_read, int fd, void *buf, size_t n)
{
void *buf_start = buf;
@@ -372,42 +335,6 @@
return p - ptr;
}
-/* Obtain a backtrace and print it to stdout. */
-#ifdef HAVE_BACKTRACE_SUPPORT
-void dump_stack(void)
-{
- void *array[16];
- size_t size = backtrace(array, ARRAY_SIZE(array));
- char **strings = backtrace_symbols(array, size);
- size_t i;
-
- printf("Obtained %zd stack frames.\n", size);
-
- for (i = 0; i < size; i++)
- printf("%s\n", strings[i]);
-
- free(strings);
-}
-#else
-void dump_stack(void) {}
-#endif
-
-void sighandler_dump_stack(int sig)
-{
- psignal(sig, "perf");
- dump_stack();
- signal(sig, SIG_DFL);
- raise(sig);
-}
-
-int timestamp__scnprintf_usec(u64 timestamp, char *buf, size_t sz)
-{
- u64 sec = timestamp / NSEC_PER_SEC;
- u64 usec = (timestamp % NSEC_PER_SEC) / NSEC_PER_USEC;
-
- return scnprintf(buf, sz, "%"PRIu64".%06"PRIu64, sec, usec);
-}
-
unsigned long parse_tag_value(const char *str, struct parse_tag *tags)
{
struct parse_tag *i = tags;
@@ -435,108 +362,6 @@
return (unsigned long) -1;
}
-int get_stack_size(const char *str, unsigned long *_size)
-{
- char *endptr;
- unsigned long size;
- unsigned long max_size = round_down(USHRT_MAX, sizeof(u64));
-
- size = strtoul(str, &endptr, 0);
-
- do {
- if (*endptr)
- break;
-
- size = round_up(size, sizeof(u64));
- if (!size || size > max_size)
- break;
-
- *_size = size;
- return 0;
-
- } while (0);
-
- pr_err("callchain: Incorrect stack dump size (max %ld): %s\n",
- max_size, str);
- return -1;
-}
-
-int parse_callchain_record(const char *arg, struct callchain_param *param)
-{
- char *tok, *name, *saveptr = NULL;
- char *buf;
- int ret = -1;
-
- /* We need buffer that we know we can write to. */
- buf = malloc(strlen(arg) + 1);
- if (!buf)
- return -ENOMEM;
-
- strcpy(buf, arg);
-
- tok = strtok_r((char *)buf, ",", &saveptr);
- name = tok ? : (char *)buf;
-
- do {
- /* Framepointer style */
- if (!strncmp(name, "fp", sizeof("fp"))) {
- if (!strtok_r(NULL, ",", &saveptr)) {
- param->record_mode = CALLCHAIN_FP;
- ret = 0;
- } else
- pr_err("callchain: No more arguments "
- "needed for --call-graph fp\n");
- break;
-
- /* Dwarf style */
- } else if (!strncmp(name, "dwarf", sizeof("dwarf"))) {
- const unsigned long default_stack_dump_size = 8192;
-
- ret = 0;
- param->record_mode = CALLCHAIN_DWARF;
- param->dump_size = default_stack_dump_size;
-
- tok = strtok_r(NULL, ",", &saveptr);
- if (tok) {
- unsigned long size = 0;
-
- ret = get_stack_size(tok, &size);
- param->dump_size = size;
- }
- } else if (!strncmp(name, "lbr", sizeof("lbr"))) {
- if (!strtok_r(NULL, ",", &saveptr)) {
- param->record_mode = CALLCHAIN_LBR;
- ret = 0;
- } else
- pr_err("callchain: No more arguments "
- "needed for --call-graph lbr\n");
- break;
- } else {
- pr_err("callchain: Unknown --call-graph option "
- "value: %s\n", arg);
- break;
- }
-
- } while (0);
-
- free(buf);
- return ret;
-}
-
-const char *get_filename_for_perf_kvm(void)
-{
- const char *filename;
-
- if (perf_host && !perf_guest)
- filename = strdup("perf.data.host");
- else if (!perf_host && perf_guest)
- filename = strdup("perf.data.guest");
- else
- filename = strdup("perf.data.kvm");
-
- return filename;
-}
-
int perf_event_paranoid(void)
{
int value;
@@ -696,7 +521,8 @@
tips = strlist__new("tips.txt", &conf);
if (tips == NULL)
- return errno == ENOENT ? NULL : "Tip: get more memory! ;-p";
+ return errno == ENOENT ? NULL :
+ "Tip: check path of tips.txt or get more memory! ;-p";
if (strlist__nr_entries(tips) == 0)
goto out;
@@ -710,95 +536,3 @@
return tip;
}
-
-bool is_regular_file(const char *file)
-{
- struct stat st;
-
- if (stat(file, &st))
- return false;
-
- return S_ISREG(st.st_mode);
-}
-
-int fetch_current_timestamp(char *buf, size_t sz)
-{
- struct timeval tv;
- struct tm tm;
- char dt[32];
-
- if (gettimeofday(&tv, NULL) || !localtime_r(&tv.tv_sec, &tm))
- return -1;
-
- if (!strftime(dt, sizeof(dt), "%Y%m%d%H%M%S", &tm))
- return -1;
-
- scnprintf(buf, sz, "%s%02u", dt, (unsigned)tv.tv_usec / 10000);
-
- return 0;
-}
-
-void print_binary(unsigned char *data, size_t len,
- size_t bytes_per_line, print_binary_t printer,
- void *extra)
-{
- size_t i, j, mask;
-
- if (!printer)
- return;
-
- bytes_per_line = roundup_pow_of_two(bytes_per_line);
- mask = bytes_per_line - 1;
-
- printer(BINARY_PRINT_DATA_BEGIN, 0, extra);
- for (i = 0; i < len; i++) {
- if ((i & mask) == 0) {
- printer(BINARY_PRINT_LINE_BEGIN, -1, extra);
- printer(BINARY_PRINT_ADDR, i, extra);
- }
-
- printer(BINARY_PRINT_NUM_DATA, data[i], extra);
-
- if (((i & mask) == mask) || i == len - 1) {
- for (j = 0; j < mask-(i & mask); j++)
- printer(BINARY_PRINT_NUM_PAD, -1, extra);
-
- printer(BINARY_PRINT_SEP, i, extra);
- for (j = i & ~mask; j <= i; j++)
- printer(BINARY_PRINT_CHAR_DATA, data[j], extra);
- for (j = 0; j < mask-(i & mask); j++)
- printer(BINARY_PRINT_CHAR_PAD, i, extra);
- printer(BINARY_PRINT_LINE_END, -1, extra);
- }
- }
- printer(BINARY_PRINT_DATA_END, -1, extra);
-}
-
-int is_printable_array(char *p, unsigned int len)
-{
- unsigned int i;
-
- if (!p || !len || p[len - 1] != 0)
- return 0;
-
- len--;
-
- for (i = 0; i < len; i++) {
- if (!isprint(p[i]) && !isspace(p[i]))
- return 0;
- }
- return 1;
-}
-
-int unit_number__scnprintf(char *buf, size_t size, u64 n)
-{
- char unit[4] = "BKMG";
- int i = 0;
-
- while (((n / 1024) > 1) && (i < 3)) {
- n /= 1024;
- i++;
- }
-
- return scnprintf(buf, size, "%" PRIu64 "%c", n, unit[i]);
-}
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index c74708d..3852b6d 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -1,126 +1,21 @@
#ifndef GIT_COMPAT_UTIL_H
#define GIT_COMPAT_UTIL_H
-#ifndef FLEX_ARRAY
-/*
- * See if our compiler is known to support flexible array members.
- */
-#if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L)
-# define FLEX_ARRAY /* empty */
-#elif defined(__GNUC__)
-# if (__GNUC__ >= 3)
-# define FLEX_ARRAY /* empty */
-# else
-# define FLEX_ARRAY 0 /* older GNU extension */
-# endif
-#endif
-
-/*
- * Otherwise, default to safer but a bit wasteful traditional style
- */
-#ifndef FLEX_ARRAY
-# define FLEX_ARRAY 1
-#endif
-#endif
-
-#define ARRAY_SIZE(x) (sizeof(x)/sizeof(x[0]))
-
-#ifdef __GNUC__
-#define TYPEOF(x) (__typeof__(x))
-#else
-#define TYPEOF(x)
-#endif
-
-#define MSB(x, bits) ((x) & TYPEOF(x)(~0ULL << (sizeof(x) * 8 - (bits))))
-#define HAS_MULTI_BITS(i) ((i) & ((i) - 1)) /* checks if an integer has more than 1 bit set */
-
-/* Approximation of the length of the decimal representation of this type. */
-#define decimal_length(x) ((int)(sizeof(x) * 2.56 + 0.5) + 1)
-
#define _ALL_SOURCE 1
#define _BSD_SOURCE 1
/* glibc 2.20 deprecates _BSD_SOURCE in favour of _DEFAULT_SOURCE */
#define _DEFAULT_SOURCE 1
#define HAS_BOOL
-#include <unistd.h>
-#include <stdio.h>
-#include <sys/stat.h>
-#include <sys/statfs.h>
#include <fcntl.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdlib.h>
#include <stdarg.h>
-#include <string.h>
-#include <term.h>
-#include <errno.h>
-#include <limits.h>
-#include <sys/param.h>
-#include <sys/types.h>
-#include <dirent.h>
-#include <sys/time.h>
-#include <time.h>
-#include <signal.h>
-#include <fnmatch.h>
-#include <assert.h>
-#include <regex.h>
-#include <utime.h>
-#include <sys/wait.h>
-#include <poll.h>
-#include <sys/socket.h>
-#include <sys/ioctl.h>
-#include <inttypes.h>
-#include <linux/kernel.h>
#include <linux/types.h>
-#include <sys/ttydefaults.h>
-#include <api/fs/tracing_path.h>
-#include <termios.h>
-#include <linux/bitops.h>
-#include <termios.h>
-#include "strlist.h"
-extern const char *graph_line;
-extern const char *graph_dotted_line;
-extern const char *spaces;
-extern const char *dots;
extern char buildid_dir[];
-/* On most systems <limits.h> would have given us this, but
- * not on some systems (e.g. GNU/Hurd).
- */
-#ifndef PATH_MAX
-#define PATH_MAX 4096
-#endif
-
-#ifndef PRIuMAX
-#define PRIuMAX "llu"
-#endif
-
-#ifndef PRIu32
-#define PRIu32 "u"
-#endif
-
-#ifndef PRIx32
-#define PRIx32 "x"
-#endif
-
-#ifndef PATH_SEP
-#define PATH_SEP ':'
-#endif
-
-#ifndef STRIP_EXTENSION
-#define STRIP_EXTENSION ""
-#endif
-
-#ifndef has_dos_drive_prefix
-#define has_dos_drive_prefix(path) 0
-#endif
-
-#ifndef is_dir_sep
-#define is_dir_sep(c) ((c) == '/')
-#endif
-
#ifdef __GNUC__
#define NORETURN __attribute__((__noreturn__))
#else
@@ -143,22 +38,6 @@
int prefixcmp(const char *str, const char *prefix);
void set_buildid_dir(const char *dir);
-#ifdef __GLIBC_PREREQ
-#if __GLIBC_PREREQ(2, 1)
-#define HAVE_STRCHRNUL
-#endif
-#endif
-
-#ifndef HAVE_STRCHRNUL
-#define strchrnul gitstrchrnul
-static inline char *gitstrchrnul(const char *s, int c)
-{
- while (*s && *s != c)
- s++;
- return (char *)s;
-}
-#endif
-
static inline void *zalloc(size_t size)
{
return calloc(1, size);
@@ -166,47 +45,8 @@
#define zfree(ptr) ({ free(*ptr); *ptr = NULL; })
-/* Sane ctype - no locale, and works with signed chars */
-#undef isascii
-#undef isspace
-#undef isdigit
-#undef isxdigit
-#undef isalpha
-#undef isprint
-#undef isalnum
-#undef islower
-#undef isupper
-#undef tolower
-#undef toupper
-
-extern unsigned char sane_ctype[256];
-#define GIT_SPACE 0x01
-#define GIT_DIGIT 0x02
-#define GIT_ALPHA 0x04
-#define GIT_GLOB_SPECIAL 0x08
-#define GIT_REGEX_SPECIAL 0x10
-#define GIT_PRINT_EXTRA 0x20
-#define GIT_PRINT 0x3E
-#define sane_istest(x,mask) ((sane_ctype[(unsigned char)(x)] & (mask)) != 0)
-#define isascii(x) (((x) & ~0x7f) == 0)
-#define isspace(x) sane_istest(x,GIT_SPACE)
-#define isdigit(x) sane_istest(x,GIT_DIGIT)
-#define isxdigit(x) \
- (sane_istest(toupper(x), GIT_ALPHA | GIT_DIGIT) && toupper(x) < 'G')
-#define isalpha(x) sane_istest(x,GIT_ALPHA)
-#define isalnum(x) sane_istest(x,GIT_ALPHA | GIT_DIGIT)
-#define isprint(x) sane_istest(x,GIT_PRINT)
-#define islower(x) (sane_istest(x,GIT_ALPHA) && (x & 0x20))
-#define isupper(x) (sane_istest(x,GIT_ALPHA) && !(x & 0x20))
-#define tolower(x) sane_case((unsigned char)(x), 0x20)
-#define toupper(x) sane_case((unsigned char)(x), 0)
-
-static inline int sane_case(int x, int high)
-{
- if (sane_istest(x, GIT_ALPHA))
- x = (x & ~0x20) | high;
- return x;
-}
+struct dirent;
+struct strlist;
int mkdir_p(char *path, mode_t mode);
int rm_rf(const char *path);
@@ -216,19 +56,6 @@
int copyfile_mode(const char *from, const char *to, mode_t mode);
int copyfile_offset(int fromfd, loff_t from_ofs, int tofd, loff_t to_ofs, u64 size);
-s64 perf_atoll(const char *str);
-char **argv_split(const char *str, int *argcp);
-void argv_free(char **argv);
-bool strglobmatch(const char *str, const char *pat);
-bool strglobmatch_nocase(const char *str, const char *pat);
-bool strlazymatch(const char *str, const char *pat);
-static inline bool strisglob(const char *str)
-{
- return strpbrk(str, "*?[") != NULL;
-}
-int strtailcmp(const char *s1, const char *s2);
-char *strxfrchar(char *s, char from, char to);
-unsigned long convert_unit(unsigned long value, char *unit);
ssize_t readn(int fd, void *buf, size_t n);
ssize_t writen(int fd, void *buf, size_t n);
@@ -236,23 +63,9 @@
void event_attr_init(struct perf_event_attr *attr);
-#define _STR(x) #x
-#define STR(x) _STR(x)
-
size_t hex_width(u64 v);
int hex2u64(const char *ptr, u64 *val);
-char *ltrim(char *s);
-char *rtrim(char *s);
-
-static inline char *trim(char *s)
-{
- return ltrim(rtrim(s));
-}
-
-void dump_stack(void);
-void sighandler_dump_stack(int sig);
-
extern unsigned int page_size;
extern int cacheline_size;
extern int sysctl_perf_event_max_stack;
@@ -265,63 +78,13 @@
unsigned long parse_tag_value(const char *str, struct parse_tag *tags);
-#define SRCLINE_UNKNOWN ((char *) "??:0")
-
-static inline int path__join(char *bf, size_t size,
- const char *path1, const char *path2)
-{
- return scnprintf(bf, size, "%s%s%s", path1, path1[0] ? "/" : "", path2);
-}
-
-static inline int path__join3(char *bf, size_t size,
- const char *path1, const char *path2,
- const char *path3)
-{
- return scnprintf(bf, size, "%s%s%s%s%s",
- path1, path1[0] ? "/" : "",
- path2, path2[0] ? "/" : "", path3);
-}
-
-struct dso;
-struct symbol;
-
-extern bool srcline_full_filename;
-char *get_srcline(struct dso *dso, u64 addr, struct symbol *sym,
- bool show_sym);
-char *__get_srcline(struct dso *dso, u64 addr, struct symbol *sym,
- bool show_sym, bool unwind_inlines);
-void free_srcline(char *srcline);
-
int perf_event_paranoid(void);
void mem_bswap_64(void *src, int byte_size);
void mem_bswap_32(void *src, int byte_size);
-const char *get_filename_for_perf_kvm(void);
bool find_process(const char *name);
-#ifdef HAVE_ZLIB_SUPPORT
-int gzip_decompress_to_file(const char *input, int output_fd);
-#endif
-
-#ifdef HAVE_LZMA_SUPPORT
-int lzma_decompress_to_file(const char *input, int output_fd);
-#endif
-
-char *asprintf_expr_inout_ints(const char *var, bool in, size_t nints, int *ints);
-
-static inline char *asprintf_expr_in_ints(const char *var, size_t nints, int *ints)
-{
- return asprintf_expr_inout_ints(var, true, nints, ints);
-}
-
-static inline char *asprintf_expr_not_in_ints(const char *var, size_t nints, int *ints)
-{
- return asprintf_expr_inout_ints(var, false, nints, ints);
-}
-
-int get_stack_size(const char *str, unsigned long *_size);
-
int fetch_kernel_version(unsigned int *puint,
char *str, size_t str_sz);
#define KVER_VERSION(x) (((x) >> 16) & 0xff)
@@ -331,37 +94,9 @@
#define KVER_PARAM(x) KVER_VERSION(x), KVER_PATCHLEVEL(x), KVER_SUBLEVEL(x)
const char *perf_tip(const char *dirpath);
-bool is_regular_file(const char *file);
-int fetch_current_timestamp(char *buf, size_t sz);
-enum binary_printer_ops {
- BINARY_PRINT_DATA_BEGIN,
- BINARY_PRINT_LINE_BEGIN,
- BINARY_PRINT_ADDR,
- BINARY_PRINT_NUM_DATA,
- BINARY_PRINT_NUM_PAD,
- BINARY_PRINT_SEP,
- BINARY_PRINT_CHAR_DATA,
- BINARY_PRINT_CHAR_PAD,
- BINARY_PRINT_LINE_END,
- BINARY_PRINT_DATA_END,
-};
-
-typedef void (*print_binary_t)(enum binary_printer_ops,
- unsigned int val,
- void *extra);
-
-void print_binary(unsigned char *data, size_t len,
- size_t bytes_per_line, print_binary_t printer,
- void *extra);
-
-#if !defined(__GLIBC__) && !defined(__ANDROID__)
-extern int sched_getcpu(void);
+#ifndef HAVE_SCHED_GETCPU_SUPPORT
+int sched_getcpu(void);
#endif
-int is_printable_array(char *p, unsigned int len);
-
-int timestamp__scnprintf_usec(u64 timestamp, char *buf, size_t sz);
-
-int unit_number__scnprintf(char *buf, size_t size, u64 n);
#endif /* GIT_COMPAT_UTIL_H */
diff --git a/tools/perf/util/values.c b/tools/perf/util/values.c
index 5074be4..5de2e15 100644
--- a/tools/perf/util/values.c
+++ b/tools/perf/util/values.c
@@ -1,4 +1,7 @@
+#include <inttypes.h>
+#include <stdio.h>
#include <stdlib.h>
+#include <errno.h>
#include "util.h"
#include "values.h"
@@ -108,24 +111,45 @@
return i;
}
-static void perf_read_values__enlarge_counters(struct perf_read_values *values)
+static int perf_read_values__enlarge_counters(struct perf_read_values *values)
{
- int i;
+ char **countername;
+ int i, counters_max = values->counters_max * 2;
+ u64 *counterrawid = realloc(values->counterrawid, counters_max * sizeof(*values->counterrawid));
- values->counters_max *= 2;
- values->counterrawid = realloc(values->counterrawid,
- values->counters_max * sizeof(*values->counterrawid));
- values->countername = realloc(values->countername,
- values->counters_max * sizeof(*values->countername));
- if (!values->counterrawid || !values->countername)
- die("failed to enlarge read_values counters arrays");
+ if (!counterrawid) {
+ pr_debug("failed to enlarge read_values rawid array");
+ goto out_enomem;
+ }
+
+ countername = realloc(values->countername, counters_max * sizeof(*values->countername));
+ if (!countername) {
+ pr_debug("failed to enlarge read_values rawid array");
+ goto out_free_rawid;
+ }
for (i = 0; i < values->threads; i++) {
- values->value[i] = realloc(values->value[i],
- values->counters_max * sizeof(**values->value));
- if (!values->value[i])
- die("failed to enlarge read_values counters arrays");
+ u64 *value = realloc(values->value[i], counters_max * sizeof(**values->value));
+
+ if (value) {
+ pr_debug("failed to enlarge read_values ->values array");
+ goto out_free_name;
+ }
+
+ values->value[i] = value;
}
+
+ values->counters_max = counters_max;
+ values->counterrawid = counterrawid;
+ values->countername = countername;
+
+ return 0;
+out_free_name:
+ free(countername);
+out_free_rawid:
+ free(counterrawid);
+out_enomem:
+ return -ENOMEM;
}
static int perf_read_values__findnew_counter(struct perf_read_values *values,
@@ -137,8 +161,11 @@
if (values->counterrawid[i] == rawid)
return i;
- if (values->counters == values->counters_max)
- perf_read_values__enlarge_counters(values);
+ if (values->counters == values->counters_max) {
+ i = perf_read_values__enlarge_counters(values);
+ if (i)
+ return i;
+ }
i = values->counters++;
values->counterrawid[i] = rawid;
@@ -172,8 +199,10 @@
int *counterwidth;
counterwidth = malloc(values->counters * sizeof(*counterwidth));
- if (!counterwidth)
- die("failed to allocate counterwidth array");
+ if (!counterwidth) {
+ fprintf(fp, "INTERNAL ERROR: Failed to allocate counterwidth array\n");
+ return;
+ }
tidwidth = 3;
pidwidth = 3;
for (j = 0; j < values->counters; j++)
diff --git a/tools/perf/util/vdso.c b/tools/perf/util/vdso.c
index 7bdcad4..d3c39ee 100644
--- a/tools/perf/util/vdso.c
+++ b/tools/perf/util/vdso.c
@@ -1,4 +1,4 @@
-
+#include <errno.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
diff --git a/tools/perf/util/xyarray.c b/tools/perf/util/xyarray.c
index c10ba41..7251fdb 100644
--- a/tools/perf/util/xyarray.c
+++ b/tools/perf/util/xyarray.c
@@ -1,5 +1,7 @@
#include "xyarray.h"
#include "util.h"
+#include <stdlib.h>
+#include <string.h>
struct xyarray *xyarray__new(int xlen, int ylen, size_t entry_size)
{
diff --git a/tools/perf/util/zlib.c b/tools/perf/util/zlib.c
index 495a449..1329d84 100644
--- a/tools/perf/util/zlib.c
+++ b/tools/perf/util/zlib.c
@@ -4,6 +4,7 @@
#include <sys/mman.h>
#include <zlib.h>
+#include "util/compress.h"
#include "util/util.h"
#include "util/debug.h"
diff --git a/tools/power/pm-graph/Makefile b/tools/power/pm-graph/Makefile
new file mode 100644
index 0000000..4d0ccc8
--- /dev/null
+++ b/tools/power/pm-graph/Makefile
@@ -0,0 +1,28 @@
+PREFIX ?= /usr
+DESTDIR ?=
+
+all:
+ @echo "Nothing to build"
+
+install :
+ install -d $(DESTDIR)$(PREFIX)/lib/pm-graph
+ install analyze_suspend.py $(DESTDIR)$(PREFIX)/lib/pm-graph
+ install analyze_boot.py $(DESTDIR)$(PREFIX)/lib/pm-graph
+
+ ln -s $(DESTDIR)$(PREFIX)/lib/pm-graph/analyze_boot.py $(DESTDIR)$(PREFIX)/bin/bootgraph
+ ln -s $(DESTDIR)$(PREFIX)/lib/pm-graph/analyze_suspend.py $(DESTDIR)$(PREFIX)/bin/sleepgraph
+
+ install -d $(DESTDIR)$(PREFIX)/share/man/man8
+ install bootgraph.8 $(DESTDIR)$(PREFIX)/share/man/man8
+ install sleepgraph.8 $(DESTDIR)$(PREFIX)/share/man/man8
+
+uninstall :
+ rm $(DESTDIR)$(PREFIX)/share/man/man8/bootgraph.8
+ rm $(DESTDIR)$(PREFIX)/share/man/man8/sleepgraph.8
+
+ rm $(DESTDIR)$(PREFIX)/bin/bootgraph
+ rm $(DESTDIR)$(PREFIX)/bin/sleepgraph
+
+ rm $(DESTDIR)$(PREFIX)/lib/pm-graph/analyze_boot.py
+ rm $(DESTDIR)$(PREFIX)/lib/pm-graph/analyze_suspend.py
+ rmdir $(DESTDIR)$(PREFIX)/lib/pm-graph
diff --git a/tools/power/pm-graph/analyze_boot.py b/tools/power/pm-graph/analyze_boot.py
new file mode 100755
index 0000000..3e1dcbb
--- /dev/null
+++ b/tools/power/pm-graph/analyze_boot.py
@@ -0,0 +1,824 @@
+#!/usr/bin/python
+#
+# Tool for analyzing boot timing
+# Copyright (c) 2013, Intel Corporation.
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms and conditions of the GNU General Public License,
+# version 2, as published by the Free Software Foundation.
+#
+# This program is distributed in the hope it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+# more details.
+#
+# Authors:
+# Todd Brandt <todd.e.brandt@linux.intel.com>
+#
+# Description:
+# This tool is designed to assist kernel and OS developers in optimizing
+# their linux stack's boot time. It creates an html representation of
+# the kernel boot timeline up to the start of the init process.
+#
+
+# ----------------- LIBRARIES --------------------
+
+import sys
+import time
+import os
+import string
+import re
+import platform
+import shutil
+from datetime import datetime, timedelta
+from subprocess import call, Popen, PIPE
+import analyze_suspend as aslib
+
+# ----------------- CLASSES --------------------
+
+# Class: SystemValues
+# Description:
+# A global, single-instance container used to
+# store system values and test parameters
+class SystemValues(aslib.SystemValues):
+ title = 'BootGraph'
+ version = 2.0
+ hostname = 'localhost'
+ testtime = ''
+ kernel = ''
+ dmesgfile = ''
+ ftracefile = ''
+ htmlfile = 'bootgraph.html'
+ outfile = ''
+ phoronix = False
+ addlogs = False
+ useftrace = False
+ usedevsrc = True
+ suspendmode = 'boot'
+ max_graph_depth = 2
+ graph_filter = 'do_one_initcall'
+ reboot = False
+ manual = False
+ iscronjob = False
+ timeformat = '%.6f'
+ def __init__(self):
+ if('LOG_FILE' in os.environ and 'TEST_RESULTS_IDENTIFIER' in os.environ):
+ self.phoronix = True
+ self.addlogs = True
+ self.outfile = os.environ['LOG_FILE']
+ self.htmlfile = os.environ['LOG_FILE']
+ self.hostname = platform.node()
+ self.testtime = datetime.now().strftime('%Y-%m-%d_%H:%M:%S')
+ if os.path.exists('/proc/version'):
+ fp = open('/proc/version', 'r')
+ val = fp.read().strip()
+ fp.close()
+ self.kernel = self.kernelVersion(val)
+ else:
+ self.kernel = 'unknown'
+ def kernelVersion(self, msg):
+ return msg.split()[2]
+ def kernelParams(self):
+ cmdline = 'initcall_debug log_buf_len=32M'
+ if self.useftrace:
+ cmdline += ' trace_buf_size=128M trace_clock=global '\
+ 'trace_options=nooverwrite,funcgraph-abstime,funcgraph-cpu,'\
+ 'funcgraph-duration,funcgraph-proc,funcgraph-tail,'\
+ 'nofuncgraph-overhead,context-info,graph-time '\
+ 'ftrace=function_graph '\
+ 'ftrace_graph_max_depth=%d '\
+ 'ftrace_graph_filter=%s' % \
+ (self.max_graph_depth, self.graph_filter)
+ return cmdline
+ def setGraphFilter(self, val):
+ fp = open(self.tpath+'available_filter_functions')
+ master = fp.read().split('\n')
+ fp.close()
+ for i in val.split(','):
+ func = i.strip()
+ if func not in master:
+ doError('function "%s" not available for ftrace' % func)
+ self.graph_filter = val
+ def cronjobCmdString(self):
+ cmdline = '%s -cronjob' % os.path.abspath(sys.argv[0])
+ args = iter(sys.argv[1:])
+ for arg in args:
+ if arg in ['-h', '-v', '-cronjob', '-reboot']:
+ continue
+ elif arg in ['-o', '-dmesg', '-ftrace', '-filter']:
+ args.next()
+ continue
+ cmdline += ' '+arg
+ if self.graph_filter != 'do_one_initcall':
+ cmdline += ' -filter "%s"' % self.graph_filter
+ cmdline += ' -o "%s"' % os.path.abspath(self.htmlfile)
+ return cmdline
+ def manualRebootRequired(self):
+ cmdline = self.kernelParams()
+ print 'To generate a new timeline manually, follow these steps:\n'
+ print '1. Add the CMDLINE string to your kernel command line.'
+ print '2. Reboot the system.'
+ print '3. After reboot, re-run this tool with the same arguments but no command (w/o -reboot or -manual).\n'
+ print 'CMDLINE="%s"' % cmdline
+ sys.exit()
+
+sysvals = SystemValues()
+
+# Class: Data
+# Description:
+# The primary container for test data.
+class Data(aslib.Data):
+ dmesg = {} # root data structure
+ start = 0.0 # test start
+ end = 0.0 # test end
+ dmesgtext = [] # dmesg text file in memory
+ testnumber = 0
+ idstr = ''
+ html_device_id = 0
+ valid = False
+ initstart = 0.0
+ boottime = ''
+ phases = ['boot']
+ do_one_initcall = False
+ def __init__(self, num):
+ self.testnumber = num
+ self.idstr = 'a'
+ self.dmesgtext = []
+ self.dmesg = {
+ 'boot': {'list': dict(), 'start': -1.0, 'end': -1.0, 'row': 0, 'color': '#dddddd'}
+ }
+ def deviceTopology(self):
+ return ''
+ def newAction(self, phase, name, start, end, ret, ulen):
+ # new device callback for a specific phase
+ self.html_device_id += 1
+ devid = '%s%d' % (self.idstr, self.html_device_id)
+ list = self.dmesg[phase]['list']
+ length = -1.0
+ if(start >= 0 and end >= 0):
+ length = end - start
+ i = 2
+ origname = name
+ while(name in list):
+ name = '%s[%d]' % (origname, i)
+ i += 1
+ list[name] = {'name': name, 'start': start, 'end': end,
+ 'pid': 0, 'length': length, 'row': 0, 'id': devid,
+ 'ret': ret, 'ulen': ulen }
+ return name
+ def deviceMatch(self, cg):
+ if cg.end - cg.start == 0:
+ return True
+ list = self.dmesg['boot']['list']
+ for devname in list:
+ dev = list[devname]
+ if cg.name == 'do_one_initcall':
+ if(cg.start <= dev['start'] and cg.end >= dev['end'] and dev['length'] > 0):
+ dev['ftrace'] = cg
+ self.do_one_initcall = True
+ return True
+ else:
+ if(cg.start > dev['start'] and cg.end < dev['end']):
+ if 'ftraces' not in dev:
+ dev['ftraces'] = []
+ dev['ftraces'].append(cg)
+ return True
+ return False
+
+# ----------------- FUNCTIONS --------------------
+
+# Function: loadKernelLog
+# Description:
+# Load a raw kernel log from dmesg
+def loadKernelLog():
+ data = Data(0)
+ data.dmesg['boot']['start'] = data.start = ktime = 0.0
+ sysvals.stamp = {
+ 'time': datetime.now().strftime('%B %d %Y, %I:%M:%S %p'),
+ 'host': sysvals.hostname,
+ 'mode': 'boot', 'kernel': ''}
+
+ devtemp = dict()
+ if(sysvals.dmesgfile):
+ lf = open(sysvals.dmesgfile, 'r')
+ else:
+ lf = Popen('dmesg', stdout=PIPE).stdout
+ for line in lf:
+ line = line.replace('\r\n', '')
+ idx = line.find('[')
+ if idx > 1:
+ line = line[idx:]
+ m = re.match('[ \t]*(\[ *)(?P<ktime>[0-9\.]*)(\]) (?P<msg>.*)', line)
+ if(not m):
+ continue
+ ktime = float(m.group('ktime'))
+ if(ktime > 120):
+ break
+ msg = m.group('msg')
+ data.end = data.initstart = ktime
+ data.dmesgtext.append(line)
+ if(ktime == 0.0 and re.match('^Linux version .*', msg)):
+ if(not sysvals.stamp['kernel']):
+ sysvals.stamp['kernel'] = sysvals.kernelVersion(msg)
+ continue
+ m = re.match('.* setting system clock to (?P<t>.*) UTC.*', msg)
+ if(m):
+ bt = datetime.strptime(m.group('t'), '%Y-%m-%d %H:%M:%S')
+ bt = bt - timedelta(seconds=int(ktime))
+ data.boottime = bt.strftime('%Y-%m-%d_%H:%M:%S')
+ sysvals.stamp['time'] = bt.strftime('%B %d %Y, %I:%M:%S %p')
+ continue
+ m = re.match('^calling *(?P<f>.*)\+.*', msg)
+ if(m):
+ devtemp[m.group('f')] = ktime
+ continue
+ m = re.match('^initcall *(?P<f>.*)\+.* returned (?P<r>.*) after (?P<t>.*) usecs', msg)
+ if(m):
+ data.valid = True
+ f, r, t = m.group('f', 'r', 't')
+ if(f in devtemp):
+ data.newAction('boot', f, devtemp[f], ktime, int(r), int(t))
+ data.end = ktime
+ del devtemp[f]
+ continue
+ if(re.match('^Freeing unused kernel memory.*', msg)):
+ break
+
+ data.dmesg['boot']['end'] = data.end
+ lf.close()
+ return data
+
+# Function: loadTraceLog
+# Description:
+# Check if trace is available and copy to a temp file
+def loadTraceLog(data):
+ # load the data to a temp file if none given
+ if not sysvals.ftracefile:
+ lib = aslib.sysvals
+ aslib.rootCheck(True)
+ if not lib.verifyFtrace():
+ doError('ftrace not available')
+ if lib.fgetVal('current_tracer').strip() != 'function_graph':
+ doError('ftrace not configured for a boot callgraph')
+ sysvals.ftracefile = '/tmp/boot_ftrace.%s.txt' % os.getpid()
+ call('cat '+lib.tpath+'trace > '+sysvals.ftracefile, shell=True)
+ if not sysvals.ftracefile:
+ doError('No trace data available')
+
+ # parse the trace log
+ ftemp = dict()
+ tp = aslib.TestProps()
+ tp.setTracerType('function_graph')
+ tf = open(sysvals.ftracefile, 'r')
+ for line in tf:
+ if line[0] == '#':
+ continue
+ m = re.match(tp.ftrace_line_fmt, line.strip())
+ if(not m):
+ continue
+ m_time, m_proc, m_pid, m_msg, m_dur = \
+ m.group('time', 'proc', 'pid', 'msg', 'dur')
+ if float(m_time) > data.end:
+ break
+ if(m_time and m_pid and m_msg):
+ t = aslib.FTraceLine(m_time, m_msg, m_dur)
+ pid = int(m_pid)
+ else:
+ continue
+ if t.fevent or t.fkprobe:
+ continue
+ key = (m_proc, pid)
+ if(key not in ftemp):
+ ftemp[key] = []
+ ftemp[key].append(aslib.FTraceCallGraph(pid))
+ cg = ftemp[key][-1]
+ if(cg.addLine(t)):
+ ftemp[key].append(aslib.FTraceCallGraph(pid))
+ tf.close()
+
+ # add the callgraph data to the device hierarchy
+ for key in ftemp:
+ proc, pid = key
+ for cg in ftemp[key]:
+ if len(cg.list) < 1 or cg.invalid:
+ continue
+ if(not cg.postProcess()):
+ print('Sanity check failed for %s-%d' % (proc, pid))
+ continue
+ # match cg data to devices
+ if not data.deviceMatch(cg):
+ print ' BAD: %s %s-%d [%f - %f]' % (cg.name, proc, pid, cg.start, cg.end)
+
+# Function: colorForName
+# Description:
+# Generate a repeatable color from a list for a given name
+def colorForName(name):
+ list = [
+ ('c1', '#ec9999'),
+ ('c2', '#ffc1a6'),
+ ('c3', '#fff0a6'),
+ ('c4', '#adf199'),
+ ('c5', '#9fadea'),
+ ('c6', '#a699c1'),
+ ('c7', '#ad99b4'),
+ ('c8', '#eaffea'),
+ ('c9', '#dcecfb'),
+ ('c10', '#ffffea')
+ ]
+ i = 0
+ total = 0
+ count = len(list)
+ while i < len(name):
+ total += ord(name[i])
+ i += 1
+ return list[total % count]
+
+def cgOverview(cg, minlen):
+ stats = dict()
+ large = []
+ for l in cg.list:
+ if l.fcall and l.depth == 1:
+ if l.length >= minlen:
+ large.append(l)
+ if l.name not in stats:
+ stats[l.name] = [0, 0.0]
+ stats[l.name][0] += (l.length * 1000.0)
+ stats[l.name][1] += 1
+ return (large, stats)
+
+# Function: createBootGraph
+# Description:
+# Create the output html file from the resident test data
+# Arguments:
+# testruns: array of Data objects from parseKernelLog or parseTraceLog
+# Output:
+# True if the html file was created, false if it failed
+def createBootGraph(data, embedded):
+ # html function templates
+ html_srccall = '<div id={6} title="{5}" class="srccall" style="left:{1}%;top:{2}px;height:{3}px;width:{4}%;line-height:{3}px;">{0}</div>\n'
+ html_timetotal = '<table class="time1">\n<tr>'\
+ '<td class="blue">Time from Kernel Boot to start of User Mode: <b>{0} ms</b></td>'\
+ '</tr>\n</table>\n'
+
+ # device timeline
+ devtl = aslib.Timeline(100, 20)
+
+ # write the test title and general info header
+ devtl.createHeader(sysvals, 'noftrace')
+
+ # Generate the header for this timeline
+ t0 = data.start
+ tMax = data.end
+ tTotal = tMax - t0
+ if(tTotal == 0):
+ print('ERROR: No timeline data')
+ return False
+ boot_time = '%.0f'%(tTotal*1000)
+ devtl.html += html_timetotal.format(boot_time)
+
+ # determine the maximum number of rows we need to draw
+ phase = 'boot'
+ list = data.dmesg[phase]['list']
+ devlist = []
+ for devname in list:
+ d = aslib.DevItem(0, phase, list[devname])
+ devlist.append(d)
+ devtl.getPhaseRows(devlist)
+ devtl.calcTotalRows()
+
+ # draw the timeline background
+ devtl.createZoomBox()
+ boot = data.dmesg[phase]
+ length = boot['end']-boot['start']
+ left = '%.3f' % (((boot['start']-t0)*100.0)/tTotal)
+ width = '%.3f' % ((length*100.0)/tTotal)
+ devtl.html += devtl.html_tblock.format(phase, left, width, devtl.scaleH)
+ devtl.html += devtl.html_phase.format('0', '100', \
+ '%.3f'%devtl.scaleH, '%.3f'%devtl.bodyH, \
+ 'white', '')
+
+ # draw the device timeline
+ num = 0
+ devstats = dict()
+ for devname in sorted(list):
+ cls, color = colorForName(devname)
+ dev = list[devname]
+ info = '@|%.3f|%.3f|%.3f|%d' % (dev['start']*1000.0, dev['end']*1000.0,
+ dev['ulen']/1000.0, dev['ret'])
+ devstats[dev['id']] = {'info':info}
+ dev['color'] = color
+ height = devtl.phaseRowHeight(0, phase, dev['row'])
+ top = '%.6f' % ((dev['row']*height) + devtl.scaleH)
+ left = '%.6f' % (((dev['start']-t0)*100)/tTotal)
+ width = '%.6f' % (((dev['end']-dev['start'])*100)/tTotal)
+ length = ' (%0.3f ms) ' % ((dev['end']-dev['start'])*1000)
+ devtl.html += devtl.html_device.format(dev['id'],
+ devname+length+'kernel_mode', left, top, '%.3f'%height,
+ width, devname, ' '+cls, '')
+ rowtop = devtl.phaseRowTop(0, phase, dev['row'])
+ height = '%.6f' % (devtl.rowH / 2)
+ top = '%.6f' % (rowtop + devtl.scaleH + (devtl.rowH / 2))
+ if data.do_one_initcall:
+ if('ftrace' not in dev):
+ continue
+ cg = dev['ftrace']
+ large, stats = cgOverview(cg, 0.001)
+ devstats[dev['id']]['fstat'] = stats
+ for l in large:
+ left = '%f' % (((l.time-t0)*100)/tTotal)
+ width = '%f' % (l.length*100/tTotal)
+ title = '%s (%0.3fms)' % (l.name, l.length * 1000.0)
+ devtl.html += html_srccall.format(l.name, left,
+ top, height, width, title, 'x%d'%num)
+ num += 1
+ continue
+ if('ftraces' not in dev):
+ continue
+ for cg in dev['ftraces']:
+ left = '%f' % (((cg.start-t0)*100)/tTotal)
+ width = '%f' % ((cg.end-cg.start)*100/tTotal)
+ cglen = (cg.end - cg.start) * 1000.0
+ title = '%s (%0.3fms)' % (cg.name, cglen)
+ cg.id = 'x%d' % num
+ devtl.html += html_srccall.format(cg.name, left,
+ top, height, width, title, dev['id']+cg.id)
+ num += 1
+
+ # draw the time scale, try to make the number of labels readable
+ devtl.createTimeScale(t0, tMax, tTotal, phase)
+ devtl.html += '</div>\n'
+
+ # timeline is finished
+ devtl.html += '</div>\n</div>\n'
+
+ if(sysvals.outfile == sysvals.htmlfile):
+ hf = open(sysvals.htmlfile, 'a')
+ else:
+ hf = open(sysvals.htmlfile, 'w')
+
+ # add the css if this is not an embedded run
+ extra = '\
+ .c1 {background:rgba(209,0,0,0.4);}\n\
+ .c2 {background:rgba(255,102,34,0.4);}\n\
+ .c3 {background:rgba(255,218,33,0.4);}\n\
+ .c4 {background:rgba(51,221,0,0.4);}\n\
+ .c5 {background:rgba(17,51,204,0.4);}\n\
+ .c6 {background:rgba(34,0,102,0.4);}\n\
+ .c7 {background:rgba(51,0,68,0.4);}\n\
+ .c8 {background:rgba(204,255,204,0.4);}\n\
+ .c9 {background:rgba(169,208,245,0.4);}\n\
+ .c10 {background:rgba(255,255,204,0.4);}\n\
+ .vt {transform:rotate(-60deg);transform-origin:0 0;}\n\
+ table.fstat {table-layout:fixed;padding:150px 15px 0 0;font-size:10px;column-width:30px;}\n\
+ .fstat th {width:55px;}\n\
+ .fstat td {text-align:left;width:35px;}\n\
+ .srccall {position:absolute;font-size:10px;z-index:7;overflow:hidden;color:black;text-align:center;white-space:nowrap;border-radius:5px;border:1px solid black;background:linear-gradient(to bottom right,#CCC,#969696);}\n\
+ .srccall:hover {color:white;font-weight:bold;border:1px solid white;}\n'
+ if(not embedded):
+ aslib.addCSS(hf, sysvals, 1, False, extra)
+
+ # write the device timeline
+ hf.write(devtl.html)
+
+ # add boot specific html
+ statinfo = 'var devstats = {\n'
+ for n in sorted(devstats):
+ statinfo += '\t"%s": [\n\t\t"%s",\n' % (n, devstats[n]['info'])
+ if 'fstat' in devstats[n]:
+ funcs = devstats[n]['fstat']
+ for f in sorted(funcs, key=funcs.get, reverse=True):
+ if funcs[f][0] < 0.01 and len(funcs) > 10:
+ break
+ statinfo += '\t\t"%f|%s|%d",\n' % (funcs[f][0], f, funcs[f][1])
+ statinfo += '\t],\n'
+ statinfo += '};\n'
+ html = \
+ '<div id="devicedetailtitle"></div>\n'\
+ '<div id="devicedetail" style="display:none;">\n'\
+ '<div id="devicedetail0">\n'\
+ '<div id="kernel_mode" class="phaselet" style="left:0%;width:100%;background:#DDDDDD"></div>\n'\
+ '</div>\n</div>\n'\
+ '<script type="text/javascript">\n'+statinfo+\
+ '</script>\n'
+ hf.write(html)
+
+ # add the callgraph html
+ if(sysvals.usecallgraph):
+ aslib.addCallgraphs(sysvals, hf, data)
+
+ # add the dmesg log as a hidden div
+ if sysvals.addlogs:
+ hf.write('<div id="dmesglog" style="display:none;">\n')
+ for line in data.dmesgtext:
+ line = line.replace('<', '<').replace('>', '>')
+ hf.write(line)
+ hf.write('</div>\n')
+
+ if(not embedded):
+ # write the footer and close
+ aslib.addScriptCode(hf, [data])
+ hf.write('</body>\n</html>\n')
+ else:
+ # embedded out will be loaded in a page, skip the js
+ hf.write('<div id=bounds style=display:none>%f,%f</div>' % \
+ (data.start*1000, data.initstart*1000))
+ hf.close()
+ return True
+
+# Function: updateCron
+# Description:
+# (restore=False) Set the tool to run automatically on reboot
+# (restore=True) Restore the original crontab
+def updateCron(restore=False):
+ if not restore:
+ sysvals.rootUser(True)
+ crondir = '/var/spool/cron/crontabs/'
+ cronfile = crondir+'root'
+ backfile = crondir+'root-analyze_boot-backup'
+ if not os.path.exists(crondir):
+ doError('%s not found' % crondir)
+ out = Popen(['which', 'crontab'], stdout=PIPE).stdout.read()
+ if not out:
+ doError('crontab not found')
+ # on restore: move the backup cron back into place
+ if restore:
+ if os.path.exists(backfile):
+ shutil.move(backfile, cronfile)
+ return
+ # backup current cron and install new one with reboot
+ if os.path.exists(cronfile):
+ shutil.move(cronfile, backfile)
+ else:
+ fp = open(backfile, 'w')
+ fp.close()
+ res = -1
+ try:
+ fp = open(backfile, 'r')
+ op = open(cronfile, 'w')
+ for line in fp:
+ if '@reboot' not in line:
+ op.write(line)
+ continue
+ fp.close()
+ op.write('@reboot python %s\n' % sysvals.cronjobCmdString())
+ op.close()
+ res = call('crontab %s' % cronfile, shell=True)
+ except Exception, e:
+ print 'Exception: %s' % str(e)
+ shutil.move(backfile, cronfile)
+ res = -1
+ if res != 0:
+ doError('crontab failed')
+
+# Function: updateGrub
+# Description:
+# update grub.cfg for all kernels with our parameters
+def updateGrub(restore=False):
+ # call update-grub on restore
+ if restore:
+ try:
+ call(['update-grub'], stderr=PIPE, stdout=PIPE,
+ env={'PATH': '.:/sbin:/usr/sbin:/usr/bin:/sbin:/bin'})
+ except Exception, e:
+ print 'Exception: %s\n' % str(e)
+ return
+ # verify we can do this
+ sysvals.rootUser(True)
+ grubfile = '/etc/default/grub'
+ if not os.path.exists(grubfile):
+ print 'ERROR: Unable to set the kernel parameters via grub.\n'
+ sysvals.manualRebootRequired()
+ out = Popen(['which', 'update-grub'], stdout=PIPE).stdout.read()
+ if not out:
+ print 'ERROR: Unable to set the kernel parameters via grub.\n'
+ sysvals.manualRebootRequired()
+
+ # extract the option and create a grub config without it
+ tgtopt = 'GRUB_CMDLINE_LINUX_DEFAULT'
+ cmdline = ''
+ tempfile = '/etc/default/grub.analyze_boot'
+ shutil.move(grubfile, tempfile)
+ res = -1
+ try:
+ fp = open(tempfile, 'r')
+ op = open(grubfile, 'w')
+ cont = False
+ for line in fp:
+ line = line.strip()
+ if len(line) == 0 or line[0] == '#':
+ continue
+ opt = line.split('=')[0].strip()
+ if opt == tgtopt:
+ cmdline = line.split('=', 1)[1].strip('\\')
+ if line[-1] == '\\':
+ cont = True
+ elif cont:
+ cmdline += line.strip('\\')
+ if line[-1] != '\\':
+ cont = False
+ else:
+ op.write('%s\n' % line)
+ fp.close()
+ # if the target option value is in quotes, strip them
+ sp = '"'
+ val = cmdline.strip()
+ if val[0] == '\'' or val[0] == '"':
+ sp = val[0]
+ val = val.strip(sp)
+ cmdline = val
+ # append our cmd line options
+ if len(cmdline) > 0:
+ cmdline += ' '
+ cmdline += sysvals.kernelParams()
+ # write out the updated target option
+ op.write('\n%s=%s%s%s\n' % (tgtopt, sp, cmdline, sp))
+ op.close()
+ res = call('update-grub')
+ os.remove(grubfile)
+ except Exception, e:
+ print 'Exception: %s' % str(e)
+ res = -1
+ # cleanup
+ shutil.move(tempfile, grubfile)
+ if res != 0:
+ doError('update-grub failed')
+
+# Function: doError
+# Description:
+# generic error function for catastrphic failures
+# Arguments:
+# msg: the error message to print
+# help: True if printHelp should be called after, False otherwise
+def doError(msg, help=False):
+ if help == True:
+ printHelp()
+ print 'ERROR: %s\n' % msg
+ sys.exit()
+
+# Function: printHelp
+# Description:
+# print out the help text
+def printHelp():
+ print('')
+ print('%s v%.1f' % (sysvals.title, sysvals.version))
+ print('Usage: bootgraph <options> <command>')
+ print('')
+ print('Description:')
+ print(' This tool reads in a dmesg log of linux kernel boot and')
+ print(' creates an html representation of the boot timeline up to')
+ print(' the start of the init process.')
+ print('')
+ print(' If no specific command is given the tool reads the current dmesg')
+ print(' and/or ftrace log and outputs bootgraph.html')
+ print('')
+ print('Options:')
+ print(' -h Print this help text')
+ print(' -v Print the current tool version')
+ print(' -addlogs Add the dmesg log to the html output')
+ print(' -o file Html timeline name (default: bootgraph.html)')
+ print(' [advanced]')
+ print(' -f Use ftrace to add function detail (default: disabled)')
+ print(' -callgraph Add callgraph detail, can be very large (default: disabled)')
+ print(' -maxdepth N limit the callgraph data to N call levels (default: 2)')
+ print(' -mincg ms Discard all callgraphs shorter than ms milliseconds (e.g. 0.001 for us)')
+ print(' -timeprec N Number of significant digits in timestamps (0:S, 3:ms, [6:us])')
+ print(' -expandcg pre-expand the callgraph data in the html output (default: disabled)')
+ print(' -filter list Limit ftrace to comma-delimited list of functions (default: do_one_initcall)')
+ print(' [commands]')
+ print(' -reboot Reboot the machine automatically and generate a new timeline')
+ print(' -manual Show the requirements to generate a new timeline manually')
+ print(' -dmesg file Load a stored dmesg file (used with -ftrace)')
+ print(' -ftrace file Load a stored ftrace file (used with -dmesg)')
+ print(' -flistall Print all functions capable of being captured in ftrace')
+ print('')
+ return True
+
+# ----------------- MAIN --------------------
+# exec start (skipped if script is loaded as library)
+if __name__ == '__main__':
+ # loop through the command line arguments
+ cmd = ''
+ simplecmds = ['-updategrub', '-flistall']
+ args = iter(sys.argv[1:])
+ for arg in args:
+ if(arg == '-h'):
+ printHelp()
+ sys.exit()
+ elif(arg == '-v'):
+ print("Version %.1f" % sysvals.version)
+ sys.exit()
+ elif(arg in simplecmds):
+ cmd = arg[1:]
+ elif(arg == '-f'):
+ sysvals.useftrace = True
+ elif(arg == '-callgraph'):
+ sysvals.useftrace = True
+ sysvals.usecallgraph = True
+ elif(arg == '-mincg'):
+ sysvals.mincglen = aslib.getArgFloat('-mincg', args, 0.0, 10000.0)
+ elif(arg == '-timeprec'):
+ sysvals.setPrecision(aslib.getArgInt('-timeprec', args, 0, 6))
+ elif(arg == '-maxdepth'):
+ sysvals.max_graph_depth = aslib.getArgInt('-maxdepth', args, 0, 1000)
+ elif(arg == '-filter'):
+ try:
+ val = args.next()
+ except:
+ doError('No filter functions supplied', True)
+ aslib.rootCheck(True)
+ sysvals.setGraphFilter(val)
+ elif(arg == '-ftrace'):
+ try:
+ val = args.next()
+ except:
+ doError('No ftrace file supplied', True)
+ if(os.path.exists(val) == False):
+ doError('%s does not exist' % val)
+ sysvals.ftracefile = val
+ elif(arg == '-addlogs'):
+ sysvals.addlogs = True
+ elif(arg == '-expandcg'):
+ sysvals.cgexp = True
+ elif(arg == '-dmesg'):
+ try:
+ val = args.next()
+ except:
+ doError('No dmesg file supplied', True)
+ if(os.path.exists(val) == False):
+ doError('%s does not exist' % val)
+ if(sysvals.htmlfile == val or sysvals.outfile == val):
+ doError('Output filename collision')
+ sysvals.dmesgfile = val
+ elif(arg == '-o'):
+ try:
+ val = args.next()
+ except:
+ doError('No HTML filename supplied', True)
+ if(sysvals.dmesgfile == val or sysvals.ftracefile == val):
+ doError('Output filename collision')
+ sysvals.htmlfile = val
+ elif(arg == '-reboot'):
+ if sysvals.iscronjob:
+ doError('-reboot and -cronjob are incompatible')
+ sysvals.reboot = True
+ elif(arg == '-manual'):
+ sysvals.reboot = True
+ sysvals.manual = True
+ # remaining options are only for cron job use
+ elif(arg == '-cronjob'):
+ sysvals.iscronjob = True
+ if sysvals.reboot:
+ doError('-reboot and -cronjob are incompatible')
+ else:
+ doError('Invalid argument: '+arg, True)
+
+ if cmd != '':
+ if cmd == 'updategrub':
+ updateGrub()
+ elif cmd == 'flistall':
+ sysvals.getFtraceFilterFunctions(False)
+ sys.exit()
+
+ # update grub, setup a cronjob, and reboot
+ if sysvals.reboot:
+ if not sysvals.manual:
+ updateGrub()
+ updateCron()
+ call('reboot')
+ else:
+ sysvals.manualRebootRequired()
+ sys.exit()
+
+ # disable the cronjob
+ if sysvals.iscronjob:
+ updateCron(True)
+ updateGrub(True)
+
+ data = loadKernelLog()
+ if sysvals.useftrace:
+ loadTraceLog(data)
+ if sysvals.iscronjob:
+ try:
+ sysvals.fsetVal('0', 'tracing_on')
+ except:
+ pass
+
+ if(sysvals.outfile and sysvals.phoronix):
+ fp = open(sysvals.outfile, 'w')
+ fp.write('pass %s initstart %.3f end %.3f boot %s\n' %
+ (data.valid, data.initstart*1000, data.end*1000, data.boottime))
+ fp.close()
+ if(not data.valid):
+ if sysvals.dmesgfile:
+ doError('No initcall data found in %s' % sysvals.dmesgfile)
+ else:
+ doError('No initcall data found, is initcall_debug enabled?')
+
+ print(' Host: %s' % sysvals.hostname)
+ print(' Test time: %s' % sysvals.testtime)
+ print(' Boot time: %s' % data.boottime)
+ print('Kernel Version: %s' % sysvals.kernel)
+ print(' Kernel start: %.3f' % (data.start * 1000))
+ print(' init start: %.3f' % (data.initstart * 1000))
+
+ createBootGraph(data, sysvals.phoronix)
diff --git a/scripts/analyze_suspend.py b/tools/power/pm-graph/analyze_suspend.py
similarity index 90%
rename from scripts/analyze_suspend.py
rename to tools/power/pm-graph/analyze_suspend.py
index 20cdb2b..a9206e6 100755
--- a/scripts/analyze_suspend.py
+++ b/tools/power/pm-graph/analyze_suspend.py
@@ -12,10 +12,6 @@
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.
#
-# You should have received a copy of the GNU General Public License along with
-# this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA.
-#
# Authors:
# Todd Brandt <todd.e.brandt@linux.intel.com>
#
@@ -23,7 +19,7 @@
# Home Page
# https://01.org/suspendresume
# Source repo
-# https://github.com/01org/suspendresume
+# https://github.com/01org/pm-graph
#
# Description:
# This tool is designed to assist kernel and OS developers in optimizing
@@ -71,14 +67,16 @@
# A global, single-instance container used to
# store system values and test parameters
class SystemValues:
+ title = 'SleepGraph'
+ version = '4.6'
ansi = False
- version = '4.5'
verbose = False
addlogs = False
mindevlen = 0.0
mincglen = 0.0
cgphase = ''
cgtest = -1
+ max_graph_depth = 0
callloopmaxgap = 0.0001
callloopmaxlen = 0.005
srgap = 0
@@ -106,8 +104,8 @@
ftracefile = ''
htmlfile = ''
embedded = False
- rtcwake = False
- rtcwaketime = 10
+ rtcwake = True
+ rtcwaketime = 15
rtcpath = ''
devicefilter = []
stamp = 0
@@ -235,6 +233,12 @@
self.rtcpath = rtc
if (hasattr(sys.stdout, 'isatty') and sys.stdout.isatty()):
self.ansi = True
+ def rootUser(self, fatal=False):
+ if 'USER' in os.environ and os.environ['USER'] == 'root':
+ return True
+ if fatal:
+ doError('This command must be run as root')
+ return False
def setPrecision(self, num):
if num < 0 or num > 6:
return
@@ -564,7 +568,7 @@
self.fsetVal('global', 'trace_clock')
# set trace buffer to a huge value
self.fsetVal('nop', 'current_tracer')
- self.fsetVal('100000', 'buffer_size_kb')
+ self.fsetVal('131073', 'buffer_size_kb')
# go no further if this is just a status check
if testing:
return
@@ -583,7 +587,7 @@
self.fsetVal('nofuncgraph-overhead', 'trace_options')
self.fsetVal('context-info', 'trace_options')
self.fsetVal('graph-time', 'trace_options')
- self.fsetVal('0', 'max_graph_depth')
+ self.fsetVal('%d' % self.max_graph_depth, 'max_graph_depth')
cf = ['dpm_run_callback']
if(self.usetraceeventsonly):
cf += ['dpm_prepare', 'dpm_complete']
@@ -639,6 +643,12 @@
return '\x1B[%d;40m%s\x1B[m' % (color, str)
sysvals = SystemValues()
+suspendmodename = {
+ 'freeze': 'Freeze (S0)',
+ 'standby': 'Standby (S1)',
+ 'mem': 'Suspend (S3)',
+ 'disk': 'Hibernate (S4)'
+}
# Class: DevProps
# Description:
@@ -1013,6 +1023,8 @@
tmp = dict()
for devname in list:
dev = list[devname]
+ if dev['length'] == 0:
+ continue
tmp[dev['start']] = devname
for t in sorted(tmp):
slist.append(tmp[t])
@@ -1477,12 +1489,14 @@
# Each instance is tied to a single device in a single phase, and is
# comprised of an ordered list of FTraceLine objects
class FTraceCallGraph:
+ id = ''
start = -1.0
end = -1.0
list = []
invalid = False
depth = 0
pid = 0
+ name = ''
def __init__(self, pid):
self.start = -1.0
self.end = -1.0
@@ -1631,9 +1645,17 @@
return True
return False
def postProcess(self, debug=False):
+ if len(self.list) > 0:
+ self.name = self.list[0].name
stack = dict()
cnt = 0
+ last = 0
for l in self.list:
+ # ftrace bug: reported duration is not reliable
+ # check each leaf and clip it at max possible length
+ if(last and last.freturn and last.fcall):
+ if last.length > l.time - last.time:
+ last.length = l.time - last.time
if(l.fcall and not l.freturn):
stack[l.depth] = l
cnt += 1
@@ -1643,11 +1665,12 @@
print 'Post Process Error: Depth missing'
l.debugPrint()
return False
- # transfer total time from return line to call line
- stack[l.depth].length = l.length
+ # calculate call length from call/return lines
+ stack[l.depth].length = l.time - stack[l.depth].time
stack.pop(l.depth)
l.length = 0
cnt -= 1
+ last = l
if(cnt == 0):
# trace caught the whole call tree
return True
@@ -1664,8 +1687,8 @@
'dpm_prepare': 'suspend_prepare',
'dpm_complete': 'resume_complete'
}
- if(self.list[0].name in borderphase):
- p = borderphase[self.list[0].name]
+ if(self.name in borderphase):
+ p = borderphase[self.name]
list = data.dmesg[p]['list']
for devname in list:
dev = list[devname]
@@ -1690,7 +1713,7 @@
break
return found
def newActionFromFunction(self, data):
- name = self.list[0].name
+ name = self.name
if name in ['dpm_run_callback', 'dpm_prepare', 'dpm_complete']:
return
fs = self.start
@@ -1710,7 +1733,7 @@
phase, myname = out
data.dmesg[phase]['list'][myname]['ftrace'] = self
def debugPrint(self):
- print('[%f - %f] %s (%d)') % (self.start, self.end, self.list[0].name, self.pid)
+ print('[%f - %f] %s (%d)') % (self.start, self.end, self.name, self.pid)
for l in self.list:
if(l.freturn and l.fcall):
print('%f (%02d): %s(); (%.3f us)' % (l.time, \
@@ -1738,7 +1761,7 @@
# A container for a device timeline which calculates
# all the html properties to display it correctly
class Timeline:
- html = {}
+ html = ''
height = 0 # total timeline height
scaleH = 20 # timescale (top) row height
rowH = 30 # device row height
@@ -1746,14 +1769,28 @@
rows = 0 # total timeline rows
rowlines = dict()
rowheight = dict()
+ html_tblock = '<div id="block{0}" class="tblock" style="left:{1}%;width:{2}%;"><div class="tback" style="height:{3}px"></div>\n'
+ html_device = '<div id="{0}" title="{1}" class="thread{7}" style="left:{2}%;top:{3}px;height:{4}px;width:{5}%;{8}">{6}</div>\n'
+ html_phase = '<div class="phase" style="left:{0}%;width:{1}%;top:{2}px;height:{3}px;background:{4}">{5}</div>\n'
+ html_phaselet = '<div id="{0}" class="phaselet" style="left:{1}%;width:{2}%;background:{3}"></div>\n'
def __init__(self, rowheight, scaleheight):
self.rowH = rowheight
self.scaleH = scaleheight
- self.html = {
- 'header': '',
- 'timeline': '',
- 'legend': '',
- }
+ self.html = ''
+ def createHeader(self, sv, suppress=''):
+ if(not sv.stamp['time']):
+ return
+ self.html += '<div class="version"><a href="https://01.org/suspendresume">%s v%s</a></div>' \
+ % (sv.title, sv.version)
+ if sv.logmsg and 'log' not in suppress:
+ self.html += '<button id="showtest" class="logbtn">log</button>'
+ if sv.addlogs and 'dmesg' not in suppress:
+ self.html += '<button id="showdmesg" class="logbtn">dmesg</button>'
+ if sv.addlogs and sv.ftracefile and 'ftrace' not in suppress:
+ self.html += '<button id="showftrace" class="logbtn">ftrace</button>'
+ headline_stamp = '<div class="stamp">{0} {1} {2} {3}</div>\n'
+ self.html += headline_stamp.format(sv.stamp['host'], sv.stamp['kernel'],
+ sv.stamp['mode'], sv.stamp['time'])
# Function: getDeviceRows
# Description:
# determine how may rows the device funcs will take
@@ -1880,10 +1917,8 @@
break
top += self.rowheight[test][phase][i]
return top
- # Function: calcTotalRows
- # Description:
- # Calculate the heights and offsets for the header and rows
def calcTotalRows(self):
+ # Calculate the heights and offsets for the header and rows
maxrows = 0
standardphases = []
for t in self.rowlines:
@@ -1901,6 +1936,20 @@
for t, p in standardphases:
for i in sorted(self.rowheight[t][p]):
self.rowheight[t][p][i] = self.bodyH/len(self.rowlines[t][p])
+ def createZoomBox(self, mode='command', testcount=1):
+ # Create bounding box, add buttons
+ html_zoombox = '<center><button id="zoomin">ZOOM IN +</button><button id="zoomout">ZOOM OUT -</button><button id="zoomdef">ZOOM 1:1</button></center>\n'
+ html_timeline = '<div id="dmesgzoombox" class="zoombox">\n<div id="{0}" class="timeline" style="height:{1}px">\n'
+ html_devlist1 = '<button id="devlist1" class="devlist" style="float:left;">Device Detail{0}</button>'
+ html_devlist2 = '<button id="devlist2" class="devlist" style="float:right;">Device Detail2</button>\n'
+ if mode != 'command':
+ if testcount > 1:
+ self.html += html_devlist2
+ self.html += html_devlist1.format('1')
+ else:
+ self.html += html_devlist1.format('')
+ self.html += html_zoombox
+ self.html += html_timeline.format('dmesg', self.height)
# Function: createTimeScale
# Description:
# Create the timescale for a timeline block
@@ -1913,7 +1962,7 @@
# The html code needed to display the time scale
def createTimeScale(self, m0, mMax, tTotal, mode):
timescale = '<div class="t" style="right:{0}%">{1}</div>\n'
- rline = '<div class="t" style="left:0;border-left:1px solid black;border-right:0;">Resume</div>\n'
+ rline = '<div class="t" style="left:0;border-left:1px solid black;border-right:0;">{0}</div>\n'
output = '<div class="timescale">\n'
# set scale for timeline
mTotal = mMax - m0
@@ -1926,21 +1975,20 @@
divEdge = (mTotal - tS*(divTotal-1))*100/mTotal
for i in range(divTotal):
htmlline = ''
- if(mode == 'resume'):
+ if(mode == 'suspend'):
+ pos = '%0.3f' % (100 - ((float(i)*tS*100)/mTotal) - divEdge)
+ val = '%0.fms' % (float(i-divTotal+1)*tS*1000)
+ if(i == divTotal - 1):
+ val = mode
+ htmlline = timescale.format(pos, val)
+ else:
pos = '%0.3f' % (100 - ((float(i)*tS*100)/mTotal))
val = '%0.fms' % (float(i)*tS*1000)
htmlline = timescale.format(pos, val)
if(i == 0):
- htmlline = rline
- else:
- pos = '%0.3f' % (100 - ((float(i)*tS*100)/mTotal) - divEdge)
- val = '%0.fms' % (float(i-divTotal+1)*tS*1000)
- if(i == divTotal - 1):
- val = 'Suspend'
- htmlline = timescale.format(pos, val)
+ htmlline = rline.format(mode)
output += htmlline
- output += '</div>\n'
- return output
+ self.html += output+'</div>\n'
# Class: TestProps
# Description:
@@ -2009,7 +2057,7 @@
val['kern'] = kern
if ujiff > 0 or kjiff > 0:
running[pid] = ujiff + kjiff
- result = process.wait()
+ process.wait()
out = ''
for pid in running:
jiffies = running[pid]
@@ -2071,26 +2119,6 @@
if not sysvals.stamp:
sysvals.stamp = data.stamp
-# Function: diffStamp
-# Description:
-# compare the host, kernel, and mode fields in 3 stamps
-# Arguments:
-# stamp1: string array with mode, kernel, and host
-# stamp2: string array with mode, kernel, and host
-# Return:
-# True if stamps differ, False if they're the same
-def diffStamp(stamp1, stamp2):
- if 'host' in stamp1 and 'host' in stamp2:
- if stamp1['host'] != stamp2['host']:
- return True
- if 'kernel' in stamp1 and 'kernel' in stamp2:
- if stamp1['kernel'] != stamp2['kernel']:
- return True
- if 'mode' in stamp1 and 'mode' in stamp2:
- if stamp1['mode'] != stamp2['mode']:
- return True
- return False
-
# Function: doesTraceLogHaveTraceEvents
# Description:
# Quickly determine if the ftrace log has some or all of the trace events
@@ -2722,7 +2750,7 @@
# create blocks for orphan cg data
for sortkey in sorted(sortlist):
cg = sortlist[sortkey]
- name = cg.list[0].name
+ name = cg.name
if sysvals.isCallgraphFunc(name):
vprint('Callgraph found for task %d: %.3fms, %s' % (cg.pid, (cg.end - cg.start)*1000, name))
cg.newActionFromFunction(data)
@@ -3100,149 +3128,154 @@
data.fixupInitcallsThatDidntReturn()
return True
+def callgraphHTML(sv, hf, num, cg, title, color, devid):
+ html_func_top = '<article id="{0}" class="atop" style="background:{1}">\n<input type="checkbox" class="pf" id="f{2}" checked/><label for="f{2}">{3} {4}</label>\n'
+ html_func_start = '<article>\n<input type="checkbox" class="pf" id="f{0}" checked/><label for="f{0}">{1} {2}</label>\n'
+ html_func_end = '</article>\n'
+ html_func_leaf = '<article>{0} {1}</article>\n'
+
+ cgid = devid
+ if cg.id:
+ cgid += cg.id
+ cglen = (cg.end - cg.start) * 1000
+ if cglen < sv.mincglen:
+ return num
+
+ fmt = '<r>(%.3f ms @ '+sv.timeformat+' to '+sv.timeformat+')</r>'
+ flen = fmt % (cglen, cg.start, cg.end)
+ hf.write(html_func_top.format(cgid, color, num, title, flen))
+ num += 1
+ for line in cg.list:
+ if(line.length < 0.000000001):
+ flen = ''
+ else:
+ fmt = '<n>(%.3f ms @ '+sv.timeformat+')</n>'
+ flen = fmt % (line.length*1000, line.time)
+ if(line.freturn and line.fcall):
+ hf.write(html_func_leaf.format(line.name, flen))
+ elif(line.freturn):
+ hf.write(html_func_end)
+ else:
+ hf.write(html_func_start.format(num, line.name, flen))
+ num += 1
+ hf.write(html_func_end)
+ return num
+
+def addCallgraphs(sv, hf, data):
+ hf.write('<section id="callgraphs" class="callgraph">\n')
+ # write out the ftrace data converted to html
+ num = 0
+ for p in data.phases:
+ if sv.cgphase and p != sv.cgphase:
+ continue
+ list = data.dmesg[p]['list']
+ for devname in data.sortedDevices(p):
+ dev = list[devname]
+ color = 'white'
+ if 'color' in data.dmesg[p]:
+ color = data.dmesg[p]['color']
+ if 'color' in dev:
+ color = dev['color']
+ name = devname
+ if(devname in sv.devprops):
+ name = sv.devprops[devname].altName(devname)
+ if sv.suspendmode in suspendmodename:
+ name += ' '+p
+ if('ftrace' in dev):
+ cg = dev['ftrace']
+ num = callgraphHTML(sv, hf, num, cg,
+ name, color, dev['id'])
+ if('ftraces' in dev):
+ for cg in dev['ftraces']:
+ num = callgraphHTML(sv, hf, num, cg,
+ name+' → '+cg.name, color, dev['id'])
+
+ hf.write('\n\n </section>\n')
+
# Function: createHTMLSummarySimple
# Description:
# Create summary html file for a series of tests
# Arguments:
# testruns: array of Data objects from parseTraceLog
-def createHTMLSummarySimple(testruns, htmlfile):
- # print out the basic summary of all the tests
- hf = open(htmlfile, 'w')
-
+def createHTMLSummarySimple(testruns, htmlfile, folder):
# write the html header first (html head, css code, up to body start)
html = '<!DOCTYPE html>\n<html>\n<head>\n\
<meta http-equiv="content-type" content="text/html; charset=UTF-8">\n\
- <title>AnalyzeSuspend Summary</title>\n\
+ <title>SleepGraph Summary</title>\n\
<style type=\'text/css\'>\n\
- body {overflow-y: scroll;}\n\
- .stamp {width: 100%;text-align:center;background-color:#495E09;line-height:30px;color:white;font: 25px Arial;}\n\
+ .stamp {width: 100%;text-align:center;background:#888;line-height:30px;color:white;font: 25px Arial;}\n\
table {width:100%;border-collapse: collapse;}\n\
- .summary {font: 22px Arial;border:1px solid;}\n\
- th {border: 1px solid black;background-color:#A7C942;color:white;}\n\
- td {text-align: center;}\n\
- tr.alt td {background-color:#EAF2D3;}\n\
- tr.avg td {background-color:#BDE34C;}\n\
- a:link {color: #90B521;}\n\
- a:visited {color: #495E09;}\n\
- a:hover {color: #B1DF28;}\n\
- a:active {color: #FFFFFF;}\n\
+ .summary {border:1px solid;}\n\
+ th {border: 1px solid black;background:#222;color:white;}\n\
+ td {font: 16px "Times New Roman";text-align: center;}\n\
+ tr.alt td {background:#ddd;}\n\
+ tr.avg td {background:#aaa;}\n\
</style>\n</head>\n<body>\n'
# group test header
- count = len(testruns)
- headline_stamp = '<div class="stamp">{0} {1} {2} {3} ({4} tests)</div>\n'
- html += headline_stamp.format(sysvals.stamp['host'],
- sysvals.stamp['kernel'], sysvals.stamp['mode'],
- sysvals.stamp['time'], count)
-
- # check to see if all the tests have the same value
- stampcolumns = False
- for data in testruns:
- if diffStamp(sysvals.stamp, data.stamp):
- stampcolumns = True
- break
-
+ html += '<div class="stamp">%s (%d tests)</div>\n' % (folder, len(testruns))
th = '\t<th>{0}</th>\n'
td = '\t<td>{0}</td>\n'
- tdlink = '\t<td><a href="{0}">Click Here</a></td>\n'
+ tdlink = '\t<td><a href="{0}">html</a></td>\n'
# table header
- html += '<table class="summary">\n<tr>\n'
- html += th.format("Test #")
- if stampcolumns:
- html += th.format("Hostname")
- html += th.format("Kernel Version")
- html += th.format("Suspend Mode")
- html += th.format("Test Time")
- html += th.format("Suspend Time")
- html += th.format("Resume Time")
- html += th.format("Detail")
- html += '</tr>\n'
+ html += '<table class="summary">\n<tr>\n' + th.format('#') +\
+ th.format('Mode') + th.format('Host') + th.format('Kernel') +\
+ th.format('Test Time') + th.format('Suspend') + th.format('Resume') +\
+ th.format('Detail') + '</tr>\n'
# test data, 1 row per test
- sTimeAvg = 0.0
- rTimeAvg = 0.0
- num = 1
- for data in testruns:
- # data.end is the end of post_resume
- resumeEnd = data.dmesg['resume_complete']['end']
+ avg = '<tr class="avg"><td></td><td></td><td></td><td></td>'+\
+ '<td>Average of {0} {1} tests</td><td>{2}</td><td>{3}</td><td></td></tr>\n'
+ sTimeAvg = rTimeAvg = 0.0
+ mode = ''
+ num = 0
+ for data in sorted(testruns, key=lambda v:(v['mode'], v['host'], v['kernel'])):
+ if mode != data['mode']:
+ # test average line
+ if(num > 0):
+ sTimeAvg /= (num - 1)
+ rTimeAvg /= (num - 1)
+ html += avg.format('%d' % (num - 1), mode,
+ '%3.3f ms' % sTimeAvg, '%3.3f ms' % rTimeAvg)
+ sTimeAvg = rTimeAvg = 0.0
+ mode = data['mode']
+ num = 1
+ # alternate row color
if num % 2 == 1:
html += '<tr class="alt">\n'
else:
html += '<tr>\n'
-
- # test num
- html += td.format("test %d" % num)
+ html += td.format("%d" % num)
num += 1
- if stampcolumns:
- # host name
+ # basic info
+ for item in ['mode', 'host', 'kernel', 'time']:
val = "unknown"
- if('host' in data.stamp):
- val = data.stamp['host']
+ if(item in data):
+ val = data[item]
html += td.format(val)
- # host kernel
- val = "unknown"
- if('kernel' in data.stamp):
- val = data.stamp['kernel']
- html += td.format(val)
- # suspend mode
- val = "unknown"
- if('mode' in data.stamp):
- val = data.stamp['mode']
- html += td.format(val)
- # test time
- val = "unknown"
- if('time' in data.stamp):
- val = data.stamp['time']
- html += td.format(val)
# suspend time
- sTime = (data.tSuspended - data.start)*1000
+ sTime = float(data['suspend'])
sTimeAvg += sTime
- html += td.format("%3.3f ms" % sTime)
+ html += td.format('%.3f ms' % sTime)
# resume time
- rTime = (resumeEnd - data.tResumed)*1000
+ rTime = float(data['resume'])
rTimeAvg += rTime
- html += td.format("%3.3f ms" % rTime)
+ html += td.format('%.3f ms' % rTime)
# link to the output html
- html += tdlink.format(data.outfile)
-
- html += '</tr>\n'
-
- # last line: test average
- if(count > 0):
- sTimeAvg /= count
- rTimeAvg /= count
- html += '<tr class="avg">\n'
- html += td.format('Average') # name
- if stampcolumns:
- html += td.format('') # host
- html += td.format('') # kernel
- html += td.format('') # mode
- html += td.format('') # time
- html += td.format("%3.3f ms" % sTimeAvg) # suspend time
- html += td.format("%3.3f ms" % rTimeAvg) # resume time
- html += td.format('') # output link
- html += '</tr>\n'
+ html += tdlink.format(data['url']) + '</tr>\n'
+ # last test average line
+ if(num > 0):
+ sTimeAvg /= (num - 1)
+ rTimeAvg /= (num - 1)
+ html += avg.format('%d' % (num - 1), mode,
+ '%3.3f ms' % sTimeAvg, '%3.3f ms' % rTimeAvg)
# flush the data to file
- hf.write(html+'</table>\n')
- hf.write('</body>\n</html>\n')
+ hf = open(htmlfile, 'w')
+ hf.write(html+'</table>\n</body>\n</html>\n')
hf.close()
-def htmlTitle():
- modename = {
- 'freeze': 'Freeze (S0)',
- 'standby': 'Standby (S1)',
- 'mem': 'Suspend (S3)',
- 'disk': 'Hibernate (S4)'
- }
- kernel = sysvals.stamp['kernel']
- host = sysvals.hostname[0].upper()+sysvals.hostname[1:]
- mode = sysvals.suspendmode
- if sysvals.suspendmode in modename:
- mode = modename[sysvals.suspendmode]
- return host+' '+mode+' '+kernel
-
def ordinal(value):
suffix = 'th'
if value < 10 or value > 19:
@@ -3272,24 +3305,11 @@
kerror = True
data.normalizeTime(testruns[-1].tSuspended)
- x2changes = ['', 'absolute']
- if len(testruns) > 1:
- x2changes = ['1', 'relative']
# html function templates
- headline_version = '<div class="version"><a href="https://01.org/suspendresume">AnalyzeSuspend v%s</a></div>' % sysvals.version
- headline_stamp = '<div class="stamp">{0} {1} {2} {3}</div>\n'
- html_devlist1 = '<button id="devlist1" class="devlist" style="float:left;">Device Detail%s</button>' % x2changes[0]
- html_zoombox = '<center><button id="zoomin">ZOOM IN +</button><button id="zoomout">ZOOM OUT -</button><button id="zoomdef">ZOOM 1:1</button></center>\n'
- html_devlist2 = '<button id="devlist2" class="devlist" style="float:right;">Device Detail2</button>\n'
- html_timeline = '<div id="dmesgzoombox" class="zoombox">\n<div id="{0}" class="timeline" style="height:{1}px">\n'
- html_tblock = '<div id="block{0}" class="tblock" style="left:{1}%;width:{2}%;"><div class="tback" style="height:{3}px"></div>\n'
- html_device = '<div id="{0}" title="{1}" class="thread{7}" style="left:{2}%;top:{3}px;height:{4}px;width:{5}%;{8}">{6}</div>\n'
html_error = '<div id="{1}" title="kernel error/warning" class="err" style="right:{0}%">ERROR→</div>\n'
html_traceevent = '<div title="{0}" class="traceevent{6}" style="left:{1}%;top:{2}px;height:{3}px;width:{4}%;line-height:{3}px;{7}">{5}</div>\n'
html_cpuexec = '<div class="jiffie" style="left:{0}%;top:{1}px;height:{2}px;width:{3}%;background:{4};"></div>\n'
- html_phase = '<div class="phase" style="left:{0}%;width:{1}%;top:{2}px;height:{3}px;background-color:{4}">{5}</div>\n'
- html_phaselet = '<div id="{0}" class="phaselet" style="left:{1}%;width:{2}%;background:{3}"></div>\n'
- html_legend = '<div id="p{3}" class="square" style="left:{0}%;background-color:{1}"> {2}</div>\n'
+ html_legend = '<div id="p{3}" class="square" style="left:{0}%;background:{1}"> {2}</div>\n'
html_timetotal = '<table class="time1">\n<tr>'\
'<td class="green" title="{3}">{2} Suspend Time: <b>{0} ms</b></td>'\
'<td class="yellow" title="{4}">{2} Resume Time: <b>{1} ms</b></td>'\
@@ -3311,20 +3331,18 @@
'</tr>\n</table>\n'
# html format variables
- hoverZ = 'z-index:8;'
- if sysvals.usedevsrc:
- hoverZ = ''
scaleH = 20
- scaleTH = 20
if kerror:
scaleH = 40
- scaleTH = 60
# device timeline
vprint('Creating Device Timeline...')
devtl = Timeline(30, scaleH)
+ # write the test title and general info header
+ devtl.createHeader(sysvals)
+
# Generate the header for this timeline
for data in testruns:
tTotal = data.end - data.start
@@ -3346,7 +3364,7 @@
if(len(testruns) > 1):
testdesc = ordinal(data.testnumber+1)+' '+testdesc
thtml = html_timetotal3.format(run_time, testdesc)
- devtl.html['header'] += thtml
+ devtl.html += thtml
elif data.fwValid:
suspend_time = '%.0f'%(sktime + (data.fwSuspend/1000000.0))
resume_time = '%.0f'%(rktime + (data.fwResume/1000000.0))
@@ -3363,10 +3381,10 @@
else:
thtml = html_timetotal2.format(suspend_time, low_time, \
resume_time, testdesc1, stitle, rtitle)
- devtl.html['header'] += thtml
+ devtl.html += thtml
sftime = '%.3f'%(data.fwSuspend / 1000000.0)
rftime = '%.3f'%(data.fwResume / 1000000.0)
- devtl.html['header'] += html_timegroups.format('%.3f'%sktime, \
+ devtl.html += html_timegroups.format('%.3f'%sktime, \
sftime, rftime, '%.3f'%rktime, testdesc2, sysvals.suspendmode)
else:
suspend_time = '%.3f' % sktime
@@ -3382,7 +3400,7 @@
else:
thtml = html_timetotal2.format(suspend_time, low_time, \
resume_time, testdesc, stitle, rtitle)
- devtl.html['header'] += thtml
+ devtl.html += thtml
# time scale for potentially multiple datasets
t0 = testruns[0].start
@@ -3429,15 +3447,8 @@
devtl.getPhaseRows(threadlist, devtl.rows)
devtl.calcTotalRows()
- # create bounding box, add buttons
- if sysvals.suspendmode != 'command':
- devtl.html['timeline'] += html_devlist1
- if len(testruns) > 1:
- devtl.html['timeline'] += html_devlist2
- devtl.html['timeline'] += html_zoombox
- devtl.html['timeline'] += html_timeline.format('dmesg', devtl.height)
-
# draw the full timeline
+ devtl.createZoomBox(sysvals.suspendmode, len(testruns))
phases = {'suspend':[],'resume':[]}
for phase in data.dmesg:
if 'resume' in phase:
@@ -3452,37 +3463,36 @@
# draw suspend and resume blocks separately
bname = '%s%d' % (dir[0], data.testnumber)
if dir == 'suspend':
- m0 = testruns[data.testnumber].start
- mMax = testruns[data.testnumber].tSuspended
- mTotal = mMax - m0
+ m0 = data.start
+ mMax = data.tSuspended
left = '%f' % (((m0-t0)*100.0)/tTotal)
else:
- m0 = testruns[data.testnumber].tSuspended
- mMax = testruns[data.testnumber].end
+ m0 = data.tSuspended
+ mMax = data.end
# in an x2 run, remove any gap between blocks
if len(testruns) > 1 and data.testnumber == 0:
mMax = testruns[1].start
- mTotal = mMax - m0
left = '%f' % ((((m0-t0)*100.0)+sysvals.srgap/2)/tTotal)
+ mTotal = mMax - m0
# if a timeline block is 0 length, skip altogether
if mTotal == 0:
continue
width = '%f' % (((mTotal*100.0)-sysvals.srgap/2)/tTotal)
- devtl.html['timeline'] += html_tblock.format(bname, left, width, devtl.scaleH)
+ devtl.html += devtl.html_tblock.format(bname, left, width, devtl.scaleH)
for b in sorted(phases[dir]):
# draw the phase color background
phase = data.dmesg[b]
length = phase['end']-phase['start']
left = '%f' % (((phase['start']-m0)*100.0)/mTotal)
width = '%f' % ((length*100.0)/mTotal)
- devtl.html['timeline'] += html_phase.format(left, width, \
+ devtl.html += devtl.html_phase.format(left, width, \
'%.3f'%devtl.scaleH, '%.3f'%devtl.bodyH, \
data.dmesg[b]['color'], '')
for e in data.errorinfo[dir]:
# draw red lines for any kernel errors found
t, err = e
right = '%f' % (((mMax-t)*100.0)/mTotal)
- devtl.html['timeline'] += html_error.format(right, err)
+ devtl.html += html_error.format(right, err)
for b in sorted(phases[dir]):
# draw the devices for this phase
phaselist = data.dmesg[b]['list']
@@ -3496,7 +3506,7 @@
if 'htmlclass' in dev:
xtraclass = dev['htmlclass']
if 'color' in dev:
- xtrastyle = 'background-color:%s;' % dev['color']
+ xtrastyle = 'background:%s;' % dev['color']
if(d in sysvals.devprops):
name = sysvals.devprops[d].altName(d)
xtraclass = sysvals.devprops[d].xtraClass()
@@ -3521,7 +3531,7 @@
title += 'post_resume_process'
else:
title += b
- devtl.html['timeline'] += html_device.format(dev['id'], \
+ devtl.html += devtl.html_device.format(dev['id'], \
title, left, top, '%.3f'%rowheight, width, \
d+drv, xtraclass, xtrastyle)
if('cpuexec' in dev):
@@ -3535,7 +3545,7 @@
left = '%f' % (((start-m0)*100)/mTotal)
width = '%f' % ((end-start)*100/mTotal)
color = 'rgba(255, 0, 0, %f)' % j
- devtl.html['timeline'] += \
+ devtl.html += \
html_cpuexec.format(left, top, height, width, color)
if('src' not in dev):
continue
@@ -3548,20 +3558,20 @@
xtrastyle = ''
if e.color:
xtrastyle = 'background:%s;' % e.color
- devtl.html['timeline'] += \
+ devtl.html += \
html_traceevent.format(e.title(), \
left, top, height, width, e.text(), '', xtrastyle)
# draw the time scale, try to make the number of labels readable
- devtl.html['timeline'] += devtl.createTimeScale(m0, mMax, tTotal, dir)
- devtl.html['timeline'] += '</div>\n'
+ devtl.createTimeScale(m0, mMax, tTotal, dir)
+ devtl.html += '</div>\n'
# timeline is finished
- devtl.html['timeline'] += '</div>\n</div>\n'
+ devtl.html += '</div>\n</div>\n'
# draw a legend which describes the phases by color
if sysvals.suspendmode != 'command':
data = testruns[-1]
- devtl.html['legend'] = '<div class="legend">\n'
+ devtl.html += '<div class="legend">\n'
pdelta = 100.0/len(data.phases)
pmargin = pdelta / 4.0
for phase in data.phases:
@@ -3571,127 +3581,41 @@
id += tmp[1][0]
order = '%.2f' % ((data.dmesg[phase]['order'] * pdelta) + pmargin)
name = string.replace(phase, '_', ' ')
- devtl.html['legend'] += html_legend.format(order, \
+ devtl.html += html_legend.format(order, \
data.dmesg[phase]['color'], name, id)
- devtl.html['legend'] += '</div>\n'
+ devtl.html += '</div>\n'
hf = open(sysvals.htmlfile, 'w')
- if not sysvals.cgexp:
- cgchk = 'checked'
- cgnchk = 'not(:checked)'
- else:
- cgchk = 'not(:checked)'
- cgnchk = 'checked'
-
- # write the html header first (html head, css code, up to body start)
- html_header = '<!DOCTYPE html>\n<html>\n<head>\n\
- <meta http-equiv="content-type" content="text/html; charset=UTF-8">\n\
- <title>'+htmlTitle()+'</title>\n\
- <style type=\'text/css\'>\n\
- body {overflow-y:scroll;}\n\
- .stamp {width:100%;text-align:center;background-color:gray;line-height:30px;color:white;font:25px Arial;}\n\
- .callgraph {margin-top:30px;box-shadow:5px 5px 20px black;}\n\
- .callgraph article * {padding-left:28px;}\n\
- h1 {color:black;font:bold 30px Times;}\n\
- t0 {color:black;font:bold 30px Times;}\n\
- t1 {color:black;font:30px Times;}\n\
- t2 {color:black;font:25px Times;}\n\
- t3 {color:black;font:20px Times;white-space:nowrap;}\n\
- t4 {color:black;font:bold 30px Times;line-height:60px;white-space:nowrap;}\n\
- cS {font:bold 13px Times;}\n\
- table {width:100%;}\n\
- .gray {background-color:rgba(80,80,80,0.1);}\n\
- .green {background-color:rgba(204,255,204,0.4);}\n\
- .purple {background-color:rgba(128,0,128,0.2);}\n\
- .yellow {background-color:rgba(255,255,204,0.4);}\n\
- .time1 {font:22px Arial;border:1px solid;}\n\
- .time2 {font:15px Arial;border-bottom:1px solid;border-left:1px solid;border-right:1px solid;}\n\
- td {text-align:center;}\n\
- r {color:#500000;font:15px Tahoma;}\n\
- n {color:#505050;font:15px Tahoma;}\n\
- .tdhl {color:red;}\n\
- .hide {display:none;}\n\
- .pf {display:none;}\n\
- .pf:'+cgchk+' + label {background:url(\'data:image/svg+xml;utf,<?xml version="1.0" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" height="18" width="18" version="1.1"><circle cx="9" cy="9" r="8" stroke="black" stroke-width="1" fill="white"/><rect x="4" y="8" width="10" height="2" style="fill:black;stroke-width:0"/><rect x="8" y="4" width="2" height="10" style="fill:black;stroke-width:0"/></svg>\') no-repeat left center;}\n\
- .pf:'+cgnchk+' ~ label {background:url(\'data:image/svg+xml;utf,<?xml version="1.0" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" height="18" width="18" version="1.1"><circle cx="9" cy="9" r="8" stroke="black" stroke-width="1" fill="white"/><rect x="4" y="8" width="10" height="2" style="fill:black;stroke-width:0"/></svg>\') no-repeat left center;}\n\
- .pf:'+cgchk+' ~ *:not(:nth-child(2)) {display:none;}\n\
- .zoombox {position:relative;width:100%;overflow-x:scroll;-webkit-user-select:none;-moz-user-select:none;user-select:none;}\n\
- .timeline {position:relative;font-size:14px;cursor:pointer;width:100%; overflow:hidden;background:linear-gradient(#cccccc, white);}\n\
- .thread {position:absolute;height:0%;overflow:hidden;z-index:7;line-height:30px;font-size:14px;border:1px solid;text-align:center;white-space:nowrap;}\n\
- .thread.ps {border-radius:3px;background:linear-gradient(to top, #ccc, #eee);}\n\
- .thread:hover {background-color:white;border:1px solid red;'+hoverZ+'}\n\
- .thread.sec,.thread.sec:hover {background-color:black;border:0;color:white;line-height:15px;font-size:10px;}\n\
- .hover {background-color:white;border:1px solid red;'+hoverZ+'}\n\
- .hover.sync {background-color:white;}\n\
- .hover.bg,.hover.kth,.hover.sync,.hover.ps {background-color:white;}\n\
- .jiffie {position:absolute;pointer-events: none;z-index:8;}\n\
- .traceevent {position:absolute;font-size:10px;z-index:7;overflow:hidden;color:black;text-align:center;white-space:nowrap;border-radius:5px;border:1px solid black;background:linear-gradient(to bottom right,#CCC,#969696);}\n\
- .traceevent:hover {color:white;font-weight:bold;border:1px solid white;}\n\
- .phase {position:absolute;overflow:hidden;border:0px;text-align:center;}\n\
- .phaselet {position:absolute;overflow:hidden;border:0px;text-align:center;height:100px;font-size:24px;}\n\
- .t {position:absolute;line-height:'+('%d'%scaleTH)+'px;pointer-events:none;top:0;height:100%;border-right:1px solid black;z-index:6;}\n\
- .err {position:absolute;top:0%;height:100%;border-right:3px solid red;color:red;font:bold 14px Times;line-height:18px;}\n\
- .legend {position:relative; width:100%; height:40px; text-align:center;margin-bottom:20px}\n\
- .legend .square {position:absolute;cursor:pointer;top:10px; width:0px;height:20px;border:1px solid;padding-left:20px;}\n\
- button {height:40px;width:200px;margin-bottom:20px;margin-top:20px;font-size:24px;}\n\
- .logbtn {position:relative;float:right;height:25px;width:50px;margin-top:3px;margin-bottom:0;font-size:10px;text-align:center;}\n\
- .devlist {position:'+x2changes[1]+';width:190px;}\n\
- a:link {color:white;text-decoration:none;}\n\
- a:visited {color:white;}\n\
- a:hover {color:white;}\n\
- a:active {color:white;}\n\
- .version {position:relative;float:left;color:white;font-size:10px;line-height:30px;margin-left:10px;}\n\
- #devicedetail {height:100px;box-shadow:5px 5px 20px black;}\n\
- .tblock {position:absolute;height:100%;background-color:#ddd;}\n\
- .tback {position:absolute;width:100%;background:linear-gradient(#ccc, #ddd);}\n\
- .bg {z-index:1;}\n\
- </style>\n</head>\n<body>\n'
-
# no header or css if its embedded
if(sysvals.embedded):
hf.write('pass True tSus %.3f tRes %.3f tLow %.3f fwvalid %s tSus %.3f tRes %.3f\n' %
(data.tSuspended-data.start, data.end-data.tSuspended, data.tLow, data.fwValid, \
data.fwSuspend/1000000, data.fwResume/1000000))
else:
- hf.write(html_header)
-
- # write the test title and general info header
- if(sysvals.stamp['time'] != ""):
- hf.write(headline_version)
- if sysvals.logmsg:
- hf.write('<button id="showtest" class="logbtn">log</button>')
- if sysvals.addlogs and sysvals.dmesgfile:
- hf.write('<button id="showdmesg" class="logbtn">dmesg</button>')
- if sysvals.addlogs and sysvals.ftracefile:
- hf.write('<button id="showftrace" class="logbtn">ftrace</button>')
- hf.write(headline_stamp.format(sysvals.stamp['host'],
- sysvals.stamp['kernel'], sysvals.stamp['mode'], \
- sysvals.stamp['time']))
+ addCSS(hf, sysvals, len(testruns), kerror)
# write the device timeline
- hf.write(devtl.html['header'])
- hf.write(devtl.html['timeline'])
- hf.write(devtl.html['legend'])
+ hf.write(devtl.html)
hf.write('<div id="devicedetailtitle"></div>\n')
hf.write('<div id="devicedetail" style="display:none;">\n')
# draw the colored boxes for the device detail section
for data in testruns:
hf.write('<div id="devicedetail%d">\n' % data.testnumber)
pscolor = 'linear-gradient(to top left, #ccc, #eee)'
- hf.write(html_phaselet.format('pre_suspend_process', \
+ hf.write(devtl.html_phaselet.format('pre_suspend_process', \
'0', '0', pscolor))
for b in data.phases:
phase = data.dmesg[b]
length = phase['end']-phase['start']
left = '%.3f' % (((phase['start']-t0)*100.0)/tTotal)
width = '%.3f' % ((length*100.0)/tTotal)
- hf.write(html_phaselet.format(b, left, width, \
+ hf.write(devtl.html_phaselet.format(b, left, width, \
data.dmesg[b]['color']))
- hf.write(html_phaselet.format('post_resume_process', \
+ hf.write(devtl.html_phaselet.format('post_resume_process', \
'0', '0', pscolor))
if sysvals.suspendmode == 'command':
- hf.write(html_phaselet.format('cmdexec', '0', '0', pscolor))
+ hf.write(devtl.html_phaselet.format('cmdexec', '0', '0', pscolor))
hf.write('</div>\n')
hf.write('</div>\n')
@@ -3701,52 +3625,7 @@
else:
data = testruns[-1]
if(sysvals.usecallgraph and not sysvals.embedded):
- hf.write('<section id="callgraphs" class="callgraph">\n')
- # write out the ftrace data converted to html
- html_func_top = '<article id="{0}" class="atop" style="background-color:{1}">\n<input type="checkbox" class="pf" id="f{2}" checked/><label for="f{2}">{3} {4}</label>\n'
- html_func_start = '<article>\n<input type="checkbox" class="pf" id="f{0}" checked/><label for="f{0}">{1} {2}</label>\n'
- html_func_end = '</article>\n'
- html_func_leaf = '<article>{0} {1}</article>\n'
- num = 0
- for p in data.phases:
- if sysvals.cgphase and p != sysvals.cgphase:
- continue
- list = data.dmesg[p]['list']
- for devname in data.sortedDevices(p):
- if('ftrace' not in list[devname]):
- continue
- devid = list[devname]['id']
- cg = list[devname]['ftrace']
- clen = (cg.end - cg.start) * 1000
- if clen < sysvals.mincglen:
- continue
- fmt = '<r>(%.3f ms @ '+sysvals.timeformat+' to '+sysvals.timeformat+')</r>'
- flen = fmt % (clen, cg.start, cg.end)
- name = devname
- if(devname in sysvals.devprops):
- name = sysvals.devprops[devname].altName(devname)
- if sysvals.suspendmode == 'command':
- ftitle = name
- else:
- ftitle = name+' '+p
- hf.write(html_func_top.format(devid, data.dmesg[p]['color'], \
- num, ftitle, flen))
- num += 1
- for line in cg.list:
- if(line.length < 0.000000001):
- flen = ''
- else:
- fmt = '<n>(%.3f ms @ '+sysvals.timeformat+')</n>'
- flen = fmt % (line.length*1000, line.time)
- if(line.freturn and line.fcall):
- hf.write(html_func_leaf.format(line.name, flen))
- elif(line.freturn):
- hf.write(html_func_end)
- else:
- hf.write(html_func_start.format(num, line.name, flen))
- num += 1
- hf.write(html_func_end)
- hf.write('\n\n </section>\n')
+ addCallgraphs(sysvals, hf, data)
# add the test log as a hidden div
if sysvals.logmsg:
@@ -3788,6 +3667,100 @@
hf.close()
return True
+def addCSS(hf, sv, testcount=1, kerror=False, extra=''):
+ kernel = sv.stamp['kernel']
+ host = sv.hostname[0].upper()+sv.hostname[1:]
+ mode = sv.suspendmode
+ if sv.suspendmode in suspendmodename:
+ mode = suspendmodename[sv.suspendmode]
+ title = host+' '+mode+' '+kernel
+
+ # various format changes by flags
+ cgchk = 'checked'
+ cgnchk = 'not(:checked)'
+ if sv.cgexp:
+ cgchk = 'not(:checked)'
+ cgnchk = 'checked'
+
+ hoverZ = 'z-index:8;'
+ if sv.usedevsrc:
+ hoverZ = ''
+
+ devlistpos = 'absolute'
+ if testcount > 1:
+ devlistpos = 'relative'
+
+ scaleTH = 20
+ if kerror:
+ scaleTH = 60
+
+ # write the html header first (html head, css code, up to body start)
+ html_header = '<!DOCTYPE html>\n<html>\n<head>\n\
+ <meta http-equiv="content-type" content="text/html; charset=UTF-8">\n\
+ <title>'+title+'</title>\n\
+ <style type=\'text/css\'>\n\
+ body {overflow-y:scroll;}\n\
+ .stamp {width:100%;text-align:center;background:gray;line-height:30px;color:white;font:25px Arial;}\n\
+ .callgraph {margin-top:30px;box-shadow:5px 5px 20px black;}\n\
+ .callgraph article * {padding-left:28px;}\n\
+ h1 {color:black;font:bold 30px Times;}\n\
+ t0 {color:black;font:bold 30px Times;}\n\
+ t1 {color:black;font:30px Times;}\n\
+ t2 {color:black;font:25px Times;}\n\
+ t3 {color:black;font:20px Times;white-space:nowrap;}\n\
+ t4 {color:black;font:bold 30px Times;line-height:60px;white-space:nowrap;}\n\
+ cS {font:bold 13px Times;}\n\
+ table {width:100%;}\n\
+ .gray {background:rgba(80,80,80,0.1);}\n\
+ .green {background:rgba(204,255,204,0.4);}\n\
+ .purple {background:rgba(128,0,128,0.2);}\n\
+ .yellow {background:rgba(255,255,204,0.4);}\n\
+ .blue {background:rgba(169,208,245,0.4);}\n\
+ .time1 {font:22px Arial;border:1px solid;}\n\
+ .time2 {font:15px Arial;border-bottom:1px solid;border-left:1px solid;border-right:1px solid;}\n\
+ td {text-align:center;}\n\
+ r {color:#500000;font:15px Tahoma;}\n\
+ n {color:#505050;font:15px Tahoma;}\n\
+ .tdhl {color:red;}\n\
+ .hide {display:none;}\n\
+ .pf {display:none;}\n\
+ .pf:'+cgchk+' + label {background:url(\'data:image/svg+xml;utf,<?xml version="1.0" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" height="18" width="18" version="1.1"><circle cx="9" cy="9" r="8" stroke="black" stroke-width="1" fill="white"/><rect x="4" y="8" width="10" height="2" style="fill:black;stroke-width:0"/><rect x="8" y="4" width="2" height="10" style="fill:black;stroke-width:0"/></svg>\') no-repeat left center;}\n\
+ .pf:'+cgnchk+' ~ label {background:url(\'data:image/svg+xml;utf,<?xml version="1.0" standalone="no"?><svg xmlns="http://www.w3.org/2000/svg" height="18" width="18" version="1.1"><circle cx="9" cy="9" r="8" stroke="black" stroke-width="1" fill="white"/><rect x="4" y="8" width="10" height="2" style="fill:black;stroke-width:0"/></svg>\') no-repeat left center;}\n\
+ .pf:'+cgchk+' ~ *:not(:nth-child(2)) {display:none;}\n\
+ .zoombox {position:relative;width:100%;overflow-x:scroll;-webkit-user-select:none;-moz-user-select:none;user-select:none;}\n\
+ .timeline {position:relative;font-size:14px;cursor:pointer;width:100%; overflow:hidden;background:linear-gradient(#cccccc, white);}\n\
+ .thread {position:absolute;height:0%;overflow:hidden;z-index:7;line-height:30px;font-size:14px;border:1px solid;text-align:center;white-space:nowrap;}\n\
+ .thread.ps {border-radius:3px;background:linear-gradient(to top, #ccc, #eee);}\n\
+ .thread:hover {background:white;border:1px solid red;'+hoverZ+'}\n\
+ .thread.sec,.thread.sec:hover {background:black;border:0;color:white;line-height:15px;font-size:10px;}\n\
+ .hover {background:white;border:1px solid red;'+hoverZ+'}\n\
+ .hover.sync {background:white;}\n\
+ .hover.bg,.hover.kth,.hover.sync,.hover.ps {background:white;}\n\
+ .jiffie {position:absolute;pointer-events: none;z-index:8;}\n\
+ .traceevent {position:absolute;font-size:10px;z-index:7;overflow:hidden;color:black;text-align:center;white-space:nowrap;border-radius:5px;border:1px solid black;background:linear-gradient(to bottom right,#CCC,#969696);}\n\
+ .traceevent:hover {color:white;font-weight:bold;border:1px solid white;}\n\
+ .phase {position:absolute;overflow:hidden;border:0px;text-align:center;}\n\
+ .phaselet {float:left;overflow:hidden;border:0px;text-align:center;min-height:100px;font-size:24px;}\n\
+ .t {position:absolute;line-height:'+('%d'%scaleTH)+'px;pointer-events:none;top:0;height:100%;border-right:1px solid black;z-index:6;}\n\
+ .err {position:absolute;top:0%;height:100%;border-right:3px solid red;color:red;font:bold 14px Times;line-height:18px;}\n\
+ .legend {position:relative; width:100%; height:40px; text-align:center;margin-bottom:20px}\n\
+ .legend .square {position:absolute;cursor:pointer;top:10px; width:0px;height:20px;border:1px solid;padding-left:20px;}\n\
+ button {height:40px;width:200px;margin-bottom:20px;margin-top:20px;font-size:24px;}\n\
+ .logbtn {position:relative;float:right;height:25px;width:50px;margin-top:3px;margin-bottom:0;font-size:10px;text-align:center;}\n\
+ .devlist {position:'+devlistpos+';width:190px;}\n\
+ a:link {color:white;text-decoration:none;}\n\
+ a:visited {color:white;}\n\
+ a:hover {color:white;}\n\
+ a:active {color:white;}\n\
+ .version {position:relative;float:left;color:white;font-size:10px;line-height:30px;margin-left:10px;}\n\
+ #devicedetail {min-height:100px;box-shadow:5px 5px 20px black;}\n\
+ .tblock {position:absolute;height:100%;background:#ddd;}\n\
+ .tback {position:absolute;width:100%;background:linear-gradient(#ccc, #ddd);}\n\
+ .bg {z-index:1;}\n\
+'+extra+'\
+ </style>\n</head>\n<body>\n'
+ hf.write(html_header)
+
# Function: addScriptCode
# Description:
# Adds the javascript code to the output html
@@ -3809,7 +3782,7 @@
' var resolution = -1;\n'\
' var dragval = [0, 0];\n'\
' function redrawTimescale(t0, tMax, tS) {\n'\
- ' var rline = \'<div class="t" style="left:0;border-left:1px solid black;border-right:0;"><cS>←R</cS></div>\';\n'\
+ ' var rline = \'<div class="t" style="left:0;border-left:1px solid black;border-right:0;">\';\n'\
' var tTotal = tMax - t0;\n'\
' var list = document.getElementsByClassName("tblock");\n'\
' for (var i = 0; i < list.length; i++) {\n'\
@@ -3824,19 +3797,23 @@
' var pos = 0.0, val = 0.0;\n'\
' for (var j = 0; j < divTotal; j++) {\n'\
' var htmlline = "";\n'\
- ' if(list[i].id[5] == "r") {\n'\
- ' pos = 100 - (((j)*tS*100)/mTotal);\n'\
- ' val = (j)*tS;\n'\
- ' htmlline = \'<div class="t" style="right:\'+pos+\'%">\'+val+\'ms</div>\';\n'\
- ' if(j == 0)\n'\
- ' htmlline = rline;\n'\
- ' } else {\n'\
+ ' var mode = list[i].id[5];\n'\
+ ' if(mode == "s") {\n'\
' pos = 100 - (((j)*tS*100)/mTotal) - divEdge;\n'\
' val = (j-divTotal+1)*tS;\n'\
' if(j == divTotal - 1)\n'\
' htmlline = \'<div class="t" style="right:\'+pos+\'%"><cS>S→</cS></div>\';\n'\
' else\n'\
' htmlline = \'<div class="t" style="right:\'+pos+\'%">\'+val+\'ms</div>\';\n'\
+ ' } else {\n'\
+ ' pos = 100 - (((j)*tS*100)/mTotal);\n'\
+ ' val = (j)*tS;\n'\
+ ' htmlline = \'<div class="t" style="right:\'+pos+\'%">\'+val+\'ms</div>\';\n'\
+ ' if(j == 0)\n'\
+ ' if(mode == "r")\n'\
+ ' htmlline = rline+"<cS>←R</cS></div>";\n'\
+ ' else\n'\
+ ' htmlline = rline+"<cS>0ms</div>";\n'\
' }\n'\
' html += htmlline;\n'\
' }\n'\
@@ -4002,12 +3979,80 @@
' }\n'\
' }\n'\
' }\n'\
+ ' if(typeof devstats !== \'undefined\')\n'\
+ ' callDetail(this.id, this.title);\n'\
' var cglist = document.getElementById("callgraphs");\n'\
' if(!cglist) return;\n'\
' var cg = cglist.getElementsByClassName("atop");\n'\
' if(cg.length < 10) return;\n'\
' for (var i = 0; i < cg.length; i++) {\n'\
- ' if(idlist.indexOf(cg[i].id) >= 0) {\n'\
+ ' cgid = cg[i].id.split("x")[0]\n'\
+ ' if(idlist.indexOf(cgid) >= 0) {\n'\
+ ' cg[i].style.display = "block";\n'\
+ ' } else {\n'\
+ ' cg[i].style.display = "none";\n'\
+ ' }\n'\
+ ' }\n'\
+ ' }\n'\
+ ' function callDetail(devid, devtitle) {\n'\
+ ' if(!(devid in devstats) || devstats[devid].length < 1)\n'\
+ ' return;\n'\
+ ' var list = devstats[devid];\n'\
+ ' var tmp = devtitle.split(" ");\n'\
+ ' var name = tmp[0], phase = tmp[tmp.length-1];\n'\
+ ' var dd = document.getElementById(phase);\n'\
+ ' var total = parseFloat(tmp[1].slice(1));\n'\
+ ' var mlist = [];\n'\
+ ' var maxlen = 0;\n'\
+ ' var info = []\n'\
+ ' for(var i in list) {\n'\
+ ' if(list[i][0] == "@") {\n'\
+ ' info = list[i].split("|");\n'\
+ ' continue;\n'\
+ ' }\n'\
+ ' var tmp = list[i].split("|");\n'\
+ ' var t = parseFloat(tmp[0]), f = tmp[1], c = parseInt(tmp[2]);\n'\
+ ' var p = (t*100.0/total).toFixed(2);\n'\
+ ' mlist[mlist.length] = [f, c, t.toFixed(2), p+"%"];\n'\
+ ' if(f.length > maxlen)\n'\
+ ' maxlen = f.length;\n'\
+ ' }\n'\
+ ' var pad = 5;\n'\
+ ' if(mlist.length == 0) pad = 30;\n'\
+ ' var html = \'<div style="padding-top:\'+pad+\'px"><t3> <b>\'+name+\':</b>\';\n'\
+ ' if(info.length > 2)\n'\
+ ' html += " start=<b>"+info[1]+"</b>, end=<b>"+info[2]+"</b>";\n'\
+ ' if(info.length > 3)\n'\
+ ' html += ", length<i>(w/o overhead)</i>=<b>"+info[3]+" ms</b>";\n'\
+ ' if(info.length > 4)\n'\
+ ' html += ", return=<b>"+info[4]+"</b>";\n'\
+ ' html += "</t3></div>";\n'\
+ ' if(mlist.length > 0) {\n'\
+ ' html += \'<table class=fstat style="padding-top:\'+(maxlen*5)+\'px;"><tr><th>Function</th>\';\n'\
+ ' for(var i in mlist)\n'\
+ ' html += "<td class=vt>"+mlist[i][0]+"</td>";\n'\
+ ' html += "</tr><tr><th>Calls</th>";\n'\
+ ' for(var i in mlist)\n'\
+ ' html += "<td>"+mlist[i][1]+"</td>";\n'\
+ ' html += "</tr><tr><th>Time(ms)</th>";\n'\
+ ' for(var i in mlist)\n'\
+ ' html += "<td>"+mlist[i][2]+"</td>";\n'\
+ ' html += "</tr><tr><th>Percent</th>";\n'\
+ ' for(var i in mlist)\n'\
+ ' html += "<td>"+mlist[i][3]+"</td>";\n'\
+ ' html += "</tr></table>";\n'\
+ ' }\n'\
+ ' dd.innerHTML = html;\n'\
+ ' var height = (maxlen*5)+100;\n'\
+ ' dd.style.height = height+"px";\n'\
+ ' document.getElementById("devicedetail").style.height = height+"px";\n'\
+ ' }\n'\
+ ' function callSelect() {\n'\
+ ' var cglist = document.getElementById("callgraphs");\n'\
+ ' if(!cglist) return;\n'\
+ ' var cg = cglist.getElementsByClassName("atop");\n'\
+ ' for (var i = 0; i < cg.length; i++) {\n'\
+ ' if(this.id == cg[i].id) {\n'\
' cg[i].style.display = "block";\n'\
' } else {\n'\
' cg[i].style.display = "none";\n'\
@@ -4093,6 +4138,9 @@
' dev[i].onmouseover = deviceHover;\n'\
' dev[i].onmouseout = deviceUnhover;\n'\
' }\n'\
+ ' var dev = dmesg.getElementsByClassName("srccall");\n'\
+ ' for (var i = 0; i < dev.length; i++)\n'\
+ ' dev[i].onclick = callSelect;\n'\
' zoomTimeline();\n'\
' });\n'\
'</script>\n'
@@ -4675,7 +4723,7 @@
if(os.access(sysvals.powerfile, os.W_OK)):
return True
if fatal:
- doError('This command must be run as root')
+ doError('This command requires sysfs mount and root access')
return False
# Function: getArgInt
@@ -4767,51 +4815,62 @@
cmd = 'chown -R {0}:{0} {1} > /dev/null 2>&1'
call(cmd.format(os.environ['SUDO_USER'], sysvals.testdir), shell=True)
+def find_in_html(html, strs, div=False):
+ for str in strs:
+ l = len(str)
+ i = html.find(str)
+ if i >= 0:
+ break
+ if i < 0:
+ return ''
+ if not div:
+ return re.search(r'[-+]?\d*\.\d+|\d+', html[i+l:i+l+50]).group()
+ n = html[i+l:].find('</div>')
+ if n < 0:
+ return ''
+ return html[i+l:i+l+n]
+
# Function: runSummary
# Description:
# create a summary of tests in a sub-directory
-def runSummary(subdir, output):
- # get a list of ftrace output files
- files = []
+def runSummary(subdir, local=True):
+ inpath = os.path.abspath(subdir)
+ outpath = inpath
+ if local:
+ outpath = os.path.abspath('.')
+ print('Generating a summary of folder "%s"' % inpath)
+ testruns = []
for dirname, dirnames, filenames in os.walk(subdir):
for filename in filenames:
- if(re.match('.*_ftrace.txt', filename)):
- files.append("%s/%s" % (dirname, filename))
-
- # process the files in order and get an array of data objects
- testruns = []
- for file in sorted(files):
- if output:
- print("Test found in %s" % os.path.dirname(file))
- sysvals.ftracefile = file
- sysvals.dmesgfile = file.replace('_ftrace.txt', '_dmesg.txt')
- doesTraceLogHaveTraceEvents()
- sysvals.usecallgraph = False
- if not sysvals.usetraceeventsonly:
- if(not os.path.exists(sysvals.dmesgfile)):
- print("Skipping %s: not a valid test input" % file)
+ if(not re.match('.*.html', filename)):
continue
- else:
- if output:
- f = os.path.basename(sysvals.ftracefile)
- d = os.path.basename(sysvals.dmesgfile)
- print("\tInput files: %s and %s" % (f, d))
- testdata = loadKernelLog()
- data = testdata[0]
- parseKernelLog(data)
- testdata = [data]
- appendIncompleteTraceLog(testdata)
- else:
- if output:
- print("\tInput file: %s" % os.path.basename(sysvals.ftracefile))
- testdata = parseTraceLog()
- data = testdata[0]
- data.normalizeTime(data.tSuspended)
- link = file.replace(subdir+'/', '').replace('_ftrace.txt', '.html')
- data.outfile = link
- testruns.append(data)
-
- createHTMLSummarySimple(testruns, subdir+'/summary.html')
+ file = os.path.join(dirname, filename)
+ html = open(file, 'r').read(10000)
+ suspend = find_in_html(html,
+ ['Kernel Suspend: ', 'Kernel Suspend Time: '])
+ resume = find_in_html(html,
+ ['Kernel Resume: ', 'Kernel Resume Time: '])
+ line = find_in_html(html, ['<div class="stamp">'], True)
+ stmp = line.split()
+ if not suspend or not resume or len(stmp) < 4:
+ continue
+ data = {
+ 'host': stmp[0],
+ 'kernel': stmp[1],
+ 'mode': stmp[2],
+ 'time': string.join(stmp[3:], ' '),
+ 'suspend': suspend,
+ 'resume': resume,
+ 'url': os.path.relpath(file, outpath),
+ }
+ if len(stmp) == 7:
+ data['kernel'] = 'unknown'
+ data['mode'] = stmp[1]
+ data['time'] = string.join(stmp[2:], ' ')
+ testruns.append(data)
+ outfile = os.path.join(outpath, 'summary.html')
+ print('Summary file: %s' % outfile)
+ createHTMLSummarySimple(testruns, outfile, inpath)
# Function: checkArgBool
# Description:
@@ -4869,9 +4928,14 @@
sysvals.predelay = getArgInt('-predelay', value, 0, 60000, False)
elif(opt.lower() == 'postdelay'):
sysvals.postdelay = getArgInt('-postdelay', value, 0, 60000, False)
+ elif(opt.lower() == 'maxdepth'):
+ sysvals.max_graph_depth = getArgInt('-maxdepth', value, 0, 1000, False)
elif(opt.lower() == 'rtcwake'):
- sysvals.rtcwake = True
- sysvals.rtcwaketime = getArgInt('-rtcwake', value, 0, 3600, False)
+ if value.lower() == 'off':
+ sysvals.rtcwake = False
+ else:
+ sysvals.rtcwake = True
+ sysvals.rtcwaketime = getArgInt('-rtcwake', value, 0, 3600, False)
elif(opt.lower() == 'timeprec'):
sysvals.setPrecision(getArgInt('-timeprec', value, 0, 6, False))
elif(opt.lower() == 'mindev'):
@@ -4969,8 +5033,8 @@
modes = getModes()
print('')
- print('AnalyzeSuspend v%s' % sysvals.version)
- print('Usage: sudo analyze_suspend.py <options>')
+ print('%s v%s' % (sysvals.title, sysvals.version))
+ print('Usage: sudo sleepgraph <options> <commands>')
print('')
print('Description:')
print(' This tool is designed to assist kernel and OS developers in optimizing')
@@ -4981,22 +5045,22 @@
print(' a detailed view of which devices/subsystems are taking the most')
print(' time in suspend/resume.')
print('')
+ print(' If no specific command is given, the default behavior is to initiate')
+ print(' a suspend/resume and capture the dmesg/ftrace output as an html timeline.')
+ print('')
print(' Generates output files in subdirectory: suspend-mmddyy-HHMMSS')
print(' HTML output: <hostname>_<mode>.html')
print(' raw dmesg output: <hostname>_<mode>_dmesg.txt')
print(' raw ftrace output: <hostname>_<mode>_ftrace.txt')
print('')
print('Options:')
- print(' [general]')
print(' -h Print this help text')
print(' -v Print the current tool version')
print(' -config fn Pull arguments and config options from file fn')
print(' -verbose Print extra information during execution and analysis')
- print(' -status Test to see if the system is enabled to run this tool')
- print(' -modes List available suspend modes')
print(' -m mode Mode to initiate for suspend %s (default: %s)') % (modes, sysvals.suspendmode)
print(' -o subdir Override the output subdirectory')
- print(' -rtcwake t Use rtcwake to autoresume after <t> seconds (default: disabled)')
+ print(' -rtcwake t Wakeup t seconds after suspend, set t to "off" to disable (default: 15)')
print(' -addlogs Add the dmesg and ftrace logs to the html output')
print(' -srgap Add a visible gap in the timeline between sus/res (default: disabled)')
print(' [advanced]')
@@ -5012,23 +5076,25 @@
print(' be created in a new subdirectory with a summary page.')
print(' [debug]')
print(' -f Use ftrace to create device callgraphs (default: disabled)')
+ print(' -maxdepth N limit the callgraph data to N call levels (default: 0=all)')
print(' -expandcg pre-expand the callgraph data in the html output (default: disabled)')
- print(' -flist Print the list of functions currently being captured in ftrace')
- print(' -flistall Print all functions capable of being captured in ftrace')
print(' -fadd file Add functions to be graphed in the timeline from a list in a text file')
print(' -filter "d1,d2,..." Filter out all but this comma-delimited list of device names')
print(' -mincg ms Discard all callgraphs shorter than ms milliseconds (e.g. 0.001 for us)')
print(' -cgphase P Only show callgraph data for phase P (e.g. suspend_late)')
print(' -cgtest N Only show callgraph data for test N (e.g. 0 or 1 in an x2 run)')
print(' -timeprec N Number of significant digits in timestamps (0:S, [3:ms], 6:us)')
- print(' [utilities]')
+ print(' [commands]')
+ print(' -ftrace ftracefile Create HTML output using ftrace input (used with -dmesg)')
+ print(' -dmesg dmesgfile Create HTML output using dmesg (used with -ftrace)')
+ print(' -summary directory Create a summary of all test in this dir')
+ print(' -modes List available suspend modes')
+ print(' -status Test to see if the system is enabled to run this tool')
print(' -fpdt Print out the contents of the ACPI Firmware Performance Data Table')
print(' -usbtopo Print out the current USB topology with power info')
print(' -usbauto Enable autosuspend for all connected USB devices')
- print(' [re-analyze data from previous runs]')
- print(' -ftrace ftracefile Create HTML output using ftrace input')
- print(' -dmesg dmesgfile Create HTML output using dmesg (not needed for kernel >= 3.15)')
- print(' -summary directory Create a summary of all test in this dir')
+ print(' -flist Print the list of functions currently being captured in ftrace')
+ print(' -flistall Print all functions capable of being captured in ftrace')
print('')
return True
@@ -5076,9 +5142,18 @@
sysvals.useprocmon = True
elif(arg == '-dev'):
sysvals.usedevsrc = True
+ elif(arg == '-maxdepth'):
+ sysvals.max_graph_depth = getArgInt('-maxdepth', args, 0, 1000)
elif(arg == '-rtcwake'):
- sysvals.rtcwake = True
- sysvals.rtcwaketime = getArgInt('-rtcwake', args, 0, 3600)
+ try:
+ val = args.next()
+ except:
+ doError('No rtcwake time supplied', True)
+ if val.lower() == 'off':
+ sysvals.rtcwake = False
+ else:
+ sysvals.rtcwake = True
+ sysvals.rtcwaketime = getArgInt('-rtcwake', val, 0, 3600, False)
elif(arg == '-timeprec'):
sysvals.setPrecision(getArgInt('-timeprec', args, 0, 6))
elif(arg == '-mindev'):
@@ -5201,7 +5276,6 @@
elif(cmd == 'usbauto'):
setUSBDevicesAuto()
elif(cmd == 'summary'):
- print("Generating a summary of folder \"%s\"" % cmdarg)
runSummary(cmdarg, True)
sys.exit()
diff --git a/tools/power/pm-graph/bootgraph.8 b/tools/power/pm-graph/bootgraph.8
new file mode 100644
index 0000000..55272a6
--- /dev/null
+++ b/tools/power/pm-graph/bootgraph.8
@@ -0,0 +1,132 @@
+.TH BOOTGRAPH 8
+.SH NAME
+bootgraph \- Kernel boot timing analysis
+.SH SYNOPSIS
+.ft B
+.B bootgraph
+.RB [ OPTIONS ]
+.RB [ COMMAND ]
+.SH DESCRIPTION
+\fBbootgraph \fP reads the dmesg log from kernel boot and
+creates an html representation of the initcall timeline up to the start
+of the init process.
+.PP
+If no specific command is given, the tool reads the current dmesg log and
+outputs bootgraph.html.
+.PP
+The tool can also augment the timeline with ftrace data on custom target
+functions as well as full trace callgraphs.
+.SH OPTIONS
+.TP
+\fB-h\fR
+Print this help text
+.TP
+\fB-v\fR
+Print the current tool version
+.TP
+\fB-addlogs\fR
+Add the dmesg log to the html output. It will be viewable by
+clicking a button in the timeline.
+.TP
+\fB-o \fIfile\fR
+Override the HTML output filename (default: bootgraph.html)
+.SS "Ftrace Debug"
+.TP
+\fB-f\fR
+Use ftrace to add function detail (default: disabled)
+.TP
+\fB-callgraph\fR
+Use ftrace to create initcall callgraphs (default: disabled). If -filter
+is not used there will be one callgraph per initcall. This can produce
+very large outputs, i.e. 10MB - 100MB.
+.TP
+\fB-maxdepth \fIlevel\fR
+limit the callgraph trace depth to \fIlevel\fR (default: 2). This is
+the best way to limit the output size when using -callgraph.
+.TP
+\fB-mincg \fIt\fR
+Discard all callgraphs shorter than \fIt\fR milliseconds (default: 0=all).
+This reduces the html file size as there can be many tiny callgraphs
+which are barely visible in the timeline.
+The value is a float: e.g. 0.001 represents 1 us.
+.TP
+\fB-timeprec \fIn\fR
+Number of significant digits in timestamps (0:S, 3:ms, [6:us])
+.TP
+\fB-expandcg\fR
+pre-expand the callgraph data in the html output (default: disabled)
+.TP
+\fB-filter \fI"func1,func2,..."\fR
+Instead of tracing each initcall, trace a custom list of functions (default: do_one_initcall)
+
+.SH COMMANDS
+.TP
+\fB-reboot\fR
+Reboot the machine and generate a new timeline automatically. Works in 4 steps.
+ 1. updates grub with the required kernel parameters
+ 2. installs a cron job which re-runs the tool after reboot
+ 3. reboots the system
+ 4. after startup, extracts the data and generates the timeline
+.TP
+\fB-manual\fR
+Show the requirements to generate a new timeline manually. Requires 3 steps.
+ 1. append the string to the kernel command line via your native boot manager.
+ 2. reboot the system
+ 3. after startup, re-run the tool with the same arguments and no command
+.TP
+\fB-dmesg \fIfile\fR
+Create HTML output from an existing dmesg file.
+.TP
+\fB-ftrace \fIfile\fR
+Create HTML output from an existing ftrace file (used with -dmesg).
+.TP
+\fB-flistall\fR
+Print all ftrace functions capable of being captured. These are all the
+possible values you can add to trace via the -filter argument.
+
+.SH EXAMPLES
+Create a timeline using the current dmesg log.
+.IP
+\f(CW$ bootgraph\fR
+.PP
+Create a timeline using the current dmesg and ftrace log.
+.IP
+\f(CW$ bootgraph -callgraph\fR
+.PP
+Create a timeline using the current dmesg, add the log to the html and change the name.
+.IP
+\f(CW$ bootgraph -addlogs -o myboot.html\fR
+.PP
+Capture a new boot timeline by automatically rebooting the machine.
+.IP
+\f(CW$ sudo bootgraph -reboot -addlogs -o latestboot.html\fR
+.PP
+Capture a new boot timeline with function trace data.
+.IP
+\f(CW$ sudo bootgraph -reboot -f\fR
+.PP
+Capture a new boot timeline with trace & callgraph data. Skip callgraphs smaller than 5ms.
+.IP
+\f(CW$ sudo bootgraph -reboot -callgraph -mincg 5\fR
+.PP
+Capture a new boot timeline with callgraph data over custom functions.
+.IP
+\f(CW$ sudo bootgraph -reboot -callgraph -filter "acpi_ps_parse_aml,msleep"\fR
+.PP
+Capture a brand new boot timeline with manual reboot.
+.IP
+\f(CW$ sudo bootgraph -callgraph -manual\fR
+.IP
+\f(CW$ vi /etc/default/grub # add the CMDLINE string to your kernel params\fR
+.IP
+\f(CW$ sudo reboot # reboot the machine\fR
+.IP
+\f(CW$ sudo bootgraph -callgraph # re-run the tool after restart\fR
+.PP
+
+.SH "SEE ALSO"
+dmesg(1), update-grub(8), crontab(1), reboot(8)
+.PP
+.SH AUTHOR
+.nf
+Written by Todd Brandt <todd.e.brandt@linux.intel.com>
diff --git a/tools/power/pm-graph/sleepgraph.8 b/tools/power/pm-graph/sleepgraph.8
new file mode 100644
index 0000000..610e72e
--- /dev/null
+++ b/tools/power/pm-graph/sleepgraph.8
@@ -0,0 +1,243 @@
+.TH SLEEPGRAPH 8
+.SH NAME
+sleepgraph \- Suspend/Resume timing analysis
+.SH SYNOPSIS
+.ft B
+.B sleepgraph
+.RB [ OPTIONS ]
+.RB [ COMMAND ]
+.SH DESCRIPTION
+\fBsleepgraph \fP is designed to assist kernel and OS developers
+in optimizing their linux stack's suspend/resume time. Using a kernel
+image built with a few extra options enabled, the tool will execute a
+suspend and capture dmesg and ftrace data until resume is complete.
+This data is transformed into a device timeline and an optional
+callgraph to give a detailed view of which devices/subsystems are
+taking the most time in suspend/resume.
+.PP
+If no specific command is given, the default behavior is to initiate
+a suspend/resume.
+.PP
+Generates output files in subdirectory: suspend-yymmdd-HHMMSS
+ html timeline : <hostname>_<mode>.html
+ raw dmesg file : <hostname>_<mode>_dmesg.txt
+ raw ftrace file : <hostname>_<mode>_ftrace.txt
+.SH OPTIONS
+.TP
+\fB-h\fR
+Print the help text.
+.TP
+\fB-v\fR
+Print the current tool version.
+.TP
+\fB-verbose\fR
+Print extra information during execution and analysis.
+.TP
+\fB-config \fIfile\fR
+Pull arguments and config options from a file.
+.TP
+\fB-m \fImode\fR
+Mode to initiate for suspend e.g. standby, freeze, mem (default: mem).
+.TP
+\fB-o \fIsubdir\fR
+Override the output subdirectory. Use {date}, {time}, {hostname} for current values.
+.sp
+e.g. suspend-{hostname}-{date}-{time}
+.TP
+\fB-rtcwake \fIt\fR | off
+Use rtcwake to autoresume after \fIt\fR seconds (default: 15). Set t to "off" to
+disable rtcwake and require a user keypress to resume.
+.TP
+\fB-addlogs\fR
+Add the dmesg and ftrace logs to the html output. They will be viewable by
+clicking buttons in the timeline.
+
+.SS "Advanced"
+.TP
+\fB-cmd \fIstr\fR
+Run the timeline over a custom suspend command, e.g. pm-suspend. By default
+the tool forces suspend via /sys/power/state so this allows testing over
+an OS's official suspend method. The output file will change to
+hostname_command.html and will autodetect which suspend mode was triggered.
+.TP
+\fB-filter \fI"d1,d2,..."\fR
+Filter out all but these device callbacks. These strings can be device names
+or module names. e.g. 0000:00:02.0, ata5, i915, usb, etc.
+.TP
+\fB-mindev \fIt\fR
+Discard all device callbacks shorter than \fIt\fR milliseconds (default: 0.0).
+This reduces the html file size as there can be many tiny callbacks which are barely
+visible. The value is a float: e.g. 0.001 represents 1 us.
+.TP
+\fB-proc\fR
+Add usermode process info into the timeline (default: disabled).
+.TP
+\fB-dev\fR
+Add kernel source calls and threads to the timeline (default: disabled).
+.TP
+\fB-x2\fR
+Run two suspend/resumes back to back (default: disabled).
+.TP
+\fB-x2delay \fIt\fR
+Include \fIt\fR ms delay between multiple test runs (default: 0 ms).
+.TP
+\fB-predelay \fIt\fR
+Include \fIt\fR ms delay before 1st suspend (default: 0 ms).
+.TP
+\fB-postdelay \fIt\fR
+Include \fIt\fR ms delay after last resume (default: 0 ms).
+.TP
+\fB-multi \fIn d\fR
+Execute \fIn\fR consecutive tests at \fId\fR seconds intervals. The outputs will
+be created in a new subdirectory with a summary page: suspend-xN-{date}-{time}.
+
+.SS "Ftrace Debug"
+.TP
+\fB-f\fR
+Use ftrace to create device callgraphs (default: disabled). This can produce
+very large outputs, i.e. 10MB - 100MB.
+.TP
+\fB-maxdepth \fIlevel\fR
+limit the callgraph trace depth to \fIlevel\fR (default: 0=all). This is
+the best way to limit the output size when using callgraphs via -f.
+.TP
+\fB-expandcg\fR
+pre-expand the callgraph data in the html output (default: disabled)
+.TP
+\fB-fadd \fIfile\fR
+Add functions to be graphed in the timeline from a list in a text file
+.TP
+\fB-mincg \fIt\fR
+Discard all callgraphs shorter than \fIt\fR milliseconds (default: 0.0).
+This reduces the html file size as there can be many tiny callgraphs
+which are barely visible in the timeline.
+The value is a float: e.g. 0.001 represents 1 us.
+.TP
+\fB-cgphase \fIp\fR
+Only show callgraph data for phase \fIp\fR (e.g. suspend_late).
+.TP
+\fB-cgtest \fIn\fR
+In an x2 run, only show callgraph data for test \fIn\fR (e.g. 0 or 1).
+.TP
+\fB-timeprec \fIn\fR
+Number of significant digits in timestamps (0:S, [3:ms], 6:us).
+
+.SH COMMANDS
+.TP
+\fB-ftrace \fIfile\fR
+Create HTML output from an existing ftrace file.
+.TP
+\fB-dmesg \fIfile\fR
+Create HTML output from an existing dmesg file.
+.TP
+\fB-summary \fIindir\fR
+Create a summary page of all tests in \fIindir\fR. Creates summary.html
+in the current folder. The output page is a table of tests with
+suspend and resume values sorted by suspend mode, host, and kernel.
+Includes test averages by mode and links to the test html files.
+.TP
+\fB-modes\fR
+List available suspend modes.
+.TP
+\fB-status\fR
+Test to see if the system is able to run this tool. Use this along
+with any options you intend to use to see if they will work.
+.TP
+\fB-fpdt\fR
+Print out the contents of the ACPI Firmware Performance Data Table.
+.TP
+\fB-usbtopo\fR
+Print out the current USB topology with power info.
+.TP
+\fB-usbauto\fR
+Enable autosuspend for all connected USB devices.
+.TP
+\fB-flist\fR
+Print the list of ftrace functions currently being captured. Functions
+that are not available as symbols in the current kernel are shown in red.
+By default, the tool traces a list of important suspend/resume functions
+in order to better fill out the timeline. If the user has added their own
+with -fadd they will also be checked.
+.TP
+\fB-flistall\fR
+Print all ftrace functions capable of being captured. These are all the
+possible values you can add to trace via the -fadd argument.
+
+.SH EXAMPLES
+.SS "Simple Commands"
+Check which suspend modes are currently supported.
+.IP
+\f(CW$ sleepgraph -modes\fR
+.PP
+Read the Firmware Performance Data Table (FPDT)
+.IP
+\f(CW$ sudo sleepgraph -fpdt\fR
+.PP
+Print out the current USB power topology
+.IP
+\f(CW$ sleepgraph -usbtopo
+.PP
+Verify that you can run a command with a set of arguments
+.IP
+\f(CW$ sudo sleepgraph -f -rtcwake 30 -status
+.PP
+Generate a summary of all timelines in a particular folder.
+.IP
+\f(CW$ sleepgraph -summary ~/workspace/myresults/\fR
+.PP
+Re-generate the html output from a previous run's dmesg and ftrace log.
+.IP
+\f(CW$ sleepgraph -dmesg myhost_mem_dmesg.txt -ftrace myhost_mem_ftrace.txt\fR
+.PP
+
+.SS "Capturing Simple Timelines"
+Execute a mem suspend with a 15 second wakeup. Include the logs in the html.
+.IP
+\f(CW$ sudo sleepgraph -rtcwake 15 -addlogs\fR
+.PP
+Execute a standby with a 15 second wakeup. Change the output folder name.
+.IP
+\f(CW$ sudo sleepgraph -m standby -rtcwake 15 -o "standby-{hostname}-{date}-{time}"\fR
+.PP
+Execute a freeze with no wakeup (require keypress). Change output folder name.
+.IP
+\f(CW$ sudo sleepgraph -m freeze -rtcwake off -o "freeze-{hostname}-{date}-{time}"\fR
+.PP
+
+.SS "Capturing Advanced Timelines"
+Execute a suspend & include dev mode source calls, limit callbacks to 5ms or larger.
+.IP
+\f(CW$ sudo sleepgraph -m mem -rtcwake 15 -dev -mindev 5\fR
+.PP
+Run two suspends back to back, include a 500ms delay before, after, and in between runs.
+.IP
+\f(CW$ sudo sleepgraph -m mem -rtcwake 15 -x2 -predelay 500 -x2delay 500 -postdelay 500\fR
+.PP
+Do a batch run of 10 freezes with 30 seconds delay between runs.
+.IP
+\f(CW$ sudo sleepgraph -m freeze -rtcwake 15 -multi 10 30\fR
+.PP
+Execute a suspend using a custom command.
+.IP
+\f(CW$ sudo sleepgraph -cmd "echo mem > /sys/power/state" -rtcwake 15\fR
+.PP
+
+
+.SS "Capturing Timelines with Callgraph Data"
+Add device callgraphs. Limit the trace depth and only show callgraphs 10ms or larger.
+.IP
+\f(CW$ sudo sleepgraph -m mem -rtcwake 15 -f -maxdepth 5 -mincg 10\fR
+.PP
+Capture a full callgraph across all suspend, then filter the html by a single phase.
+.IP
+\f(CW$ sudo sleepgraph -m mem -rtcwake 15 -f\fR
+.IP
+\f(CW$ sleepgraph -dmesg host_mem_dmesg.txt -ftrace host_mem_ftrace.txt -f -cgphase resume
+.PP
+
+.SH "SEE ALSO"
+dmesg(1)
+.PP
+.SH AUTHOR
+.nf
+Written by Todd Brandt <todd.e.brandt@linux.intel.com>
diff --git a/tools/power/x86/intel_pstate_tracer/intel_pstate_tracer.py b/tools/power/x86/intel_pstate_tracer/intel_pstate_tracer.py
index fd706ac..0b24dd9 100755
--- a/tools/power/x86/intel_pstate_tracer/intel_pstate_tracer.py
+++ b/tools/power/x86/intel_pstate_tracer/intel_pstate_tracer.py
@@ -353,6 +353,14 @@
os.system('grep -m 1 common_cpu cpu.csv > cpu{:0>3}.csv'.format(index))
os.system('grep CPU_{:0>3} cpu.csv >> cpu{:0>3}.csv'.format(index, index))
+def fix_ownership(path):
+ """Change the owner of the file to SUDO_UID, if required"""
+
+ uid = os.environ.get('SUDO_UID')
+ gid = os.environ.get('SUDO_GID')
+ if uid is not None:
+ os.chown(path, int(uid), int(gid))
+
def cleanup_data_files():
""" clean up existing data files """
@@ -518,12 +526,16 @@
if not os.path.exists('results'):
os.mkdir('results')
+ # The regular user needs to own the directory, not root.
+ fix_ownership('results')
os.chdir('results')
if os.path.exists(testname):
print('The test name directory already exists. Please provide a unique test name. Test re-run not supported, yet.')
sys.exit()
os.mkdir(testname)
+# The regular user needs to own the directory, not root.
+fix_ownership(testname)
os.chdir(testname)
# Temporary (or perhaps not)
@@ -566,4 +578,9 @@
plot_boost_cpu()
plot_ghz_cpu()
+# It is preferrable, but not necessary, that the regular user owns the files, not root.
+for root, dirs, files in os.walk('.'):
+ for f in files:
+ fix_ownership(f)
+
os.chdir('../../')
diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
index 621578a..fc74db6 100644
--- a/tools/scripts/Makefile.include
+++ b/tools/scripts/Makefile.include
@@ -43,6 +43,15 @@
EXTRA_WARNINGS += -Wstrict-aliasing=3
endif
+# Hack to avoid type-punned warnings on old systems such as RHEL5:
+# We should be changing CFLAGS and checking gcc version, but this
+# will do for now and keep the above -Wstrict-aliasing=3 in place
+# in newer systems.
+# Needed for the __raw_cmpxchg in tools/arch/x86/include/asm/cmpxchg.h
+ifneq ($(filter 3.%,$(MAKE_VERSION)),) # make-3
+EXTRA_WARNINGS += -fno-strict-aliasing
+endif
+
ifneq ($(findstring $(MAKEFLAGS), w),w)
PRINT_DIR = --no-print-directory
else
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 9af09e8..91edd05 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -8,16 +8,18 @@
GENFLAGS := -DHAVE_GENHDR
endif
-CFLAGS += -Wall -O2 -I$(APIDIR) -I$(LIBDIR) -I$(GENDIR) $(GENFLAGS)
-LDLIBS += -lcap
+CFLAGS += -Wall -O2 -I$(APIDIR) -I$(LIBDIR) -I$(GENDIR) $(GENFLAGS) -I../../../include
+LDLIBS += -lcap -lelf
-TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map
+TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test_progs
+
+TEST_GEN_FILES = test_pkt_access.o test_xdp.o test_l4lb.o test_tcp_estats.o
TEST_PROGS := test_kmod.sh
include ../lib.mk
-BPFOBJ := $(OUTPUT)/bpf.o
+BPFOBJ := $(OUTPUT)/libbpf.a
$(TEST_GEN_PROGS): $(BPFOBJ)
@@ -28,3 +30,10 @@
$(BPFOBJ): force
$(MAKE) -C $(BPFDIR) OUTPUT=$(OUTPUT)/
+
+CLANG ?= clang
+
+%.o: %.c
+ $(CLANG) -I. -I../../../include/uapi -I../../../../samples/bpf/ \
+ -Wno-compare-distinct-pointer-types \
+ -O2 -target bpf -c $< -o $@
diff --git a/tools/testing/selftests/bpf/bpf_endian.h b/tools/testing/selftests/bpf/bpf_endian.h
new file mode 100644
index 0000000..19d0604
--- /dev/null
+++ b/tools/testing/selftests/bpf/bpf_endian.h
@@ -0,0 +1,23 @@
+#ifndef __BPF_ENDIAN__
+#define __BPF_ENDIAN__
+
+#include <asm/byteorder.h>
+
+#if __BYTE_ORDER == __LITTLE_ENDIAN
+# define __bpf_ntohs(x) __builtin_bswap16(x)
+# define __bpf_htons(x) __builtin_bswap16(x)
+#elif __BYTE_ORDER == __BIG_ENDIAN
+# define __bpf_ntohs(x) (x)
+# define __bpf_htons(x) (x)
+#else
+# error "Fix your __BYTE_ORDER?!"
+#endif
+
+#define bpf_htons(x) \
+ (__builtin_constant_p(x) ? \
+ __constant_htons(x) : __bpf_htons(x))
+#define bpf_ntohs(x) \
+ (__builtin_constant_p(x) ? \
+ __constant_ntohs(x) : __bpf_ntohs(x))
+
+#endif
diff --git a/tools/testing/selftests/bpf/bpf_util.h b/tools/testing/selftests/bpf/bpf_util.h
index 84a5d18..20ecbaa 100644
--- a/tools/testing/selftests/bpf/bpf_util.h
+++ b/tools/testing/selftests/bpf/bpf_util.h
@@ -35,4 +35,11 @@
return possible_cpus;
}
+#define __bpf_percpu_val_align __attribute__((__aligned__(8)))
+
+#define BPF_DECLARE_PERCPU(type, name) \
+ struct { type v; /* padding */ } __bpf_percpu_val_align \
+ name[bpf_num_possible_cpus()]
+#define bpf_percpu(name, cpu) name[(cpu)].v
+
#endif /* __BPF_UTIL__ */
diff --git a/tools/testing/selftests/bpf/gnu/stubs.h b/tools/testing/selftests/bpf/gnu/stubs.h
new file mode 100644
index 0000000..719225b
--- /dev/null
+++ b/tools/testing/selftests/bpf/gnu/stubs.h
@@ -0,0 +1 @@
+/* dummy .h to trick /usr/include/features.h to work with 'clang -target bpf' */
diff --git a/tools/testing/selftests/bpf/test_iptunnel_common.h b/tools/testing/selftests/bpf/test_iptunnel_common.h
new file mode 100644
index 0000000..e4cd252
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_iptunnel_common.h
@@ -0,0 +1,37 @@
+/* Copyright (c) 2016 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#ifndef _TEST_IPTNL_COMMON_H
+#define _TEST_IPTNL_COMMON_H
+
+#include <linux/types.h>
+
+#define MAX_IPTNL_ENTRIES 256U
+
+struct vip {
+ union {
+ __u32 v6[4];
+ __u32 v4;
+ } daddr;
+ __u16 dport;
+ __u16 family;
+ __u8 protocol;
+};
+
+struct iptnl_info {
+ union {
+ __u32 v6[4];
+ __u32 v4;
+ } saddr;
+ union {
+ __u32 v6[4];
+ __u32 v4;
+ } daddr;
+ __u16 family;
+ __u8 dmac[6];
+};
+
+#endif
diff --git a/tools/testing/selftests/bpf/test_l4lb.c b/tools/testing/selftests/bpf/test_l4lb.c
new file mode 100644
index 0000000..1e10c95
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_l4lb.c
@@ -0,0 +1,473 @@
+/* Copyright (c) 2017 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <stddef.h>
+#include <stdbool.h>
+#include <string.h>
+#include <linux/pkt_cls.h>
+#include <linux/bpf.h>
+#include <linux/in.h>
+#include <linux/if_ether.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <linux/icmp.h>
+#include <linux/icmpv6.h>
+#include <linux/tcp.h>
+#include <linux/udp.h>
+#include "bpf_helpers.h"
+#include "test_iptunnel_common.h"
+#include "bpf_endian.h"
+
+int _version SEC("version") = 1;
+
+static inline __u32 rol32(__u32 word, unsigned int shift)
+{
+ return (word << shift) | (word >> ((-shift) & 31));
+}
+
+/* copy paste of jhash from kernel sources to make sure llvm
+ * can compile it into valid sequence of bpf instructions
+ */
+#define __jhash_mix(a, b, c) \
+{ \
+ a -= c; a ^= rol32(c, 4); c += b; \
+ b -= a; b ^= rol32(a, 6); a += c; \
+ c -= b; c ^= rol32(b, 8); b += a; \
+ a -= c; a ^= rol32(c, 16); c += b; \
+ b -= a; b ^= rol32(a, 19); a += c; \
+ c -= b; c ^= rol32(b, 4); b += a; \
+}
+
+#define __jhash_final(a, b, c) \
+{ \
+ c ^= b; c -= rol32(b, 14); \
+ a ^= c; a -= rol32(c, 11); \
+ b ^= a; b -= rol32(a, 25); \
+ c ^= b; c -= rol32(b, 16); \
+ a ^= c; a -= rol32(c, 4); \
+ b ^= a; b -= rol32(a, 14); \
+ c ^= b; c -= rol32(b, 24); \
+}
+
+#define JHASH_INITVAL 0xdeadbeef
+
+typedef unsigned int u32;
+
+static inline u32 jhash(const void *key, u32 length, u32 initval)
+{
+ u32 a, b, c;
+ const unsigned char *k = key;
+
+ a = b = c = JHASH_INITVAL + length + initval;
+
+ while (length > 12) {
+ a += *(u32 *)(k);
+ b += *(u32 *)(k + 4);
+ c += *(u32 *)(k + 8);
+ __jhash_mix(a, b, c);
+ length -= 12;
+ k += 12;
+ }
+ switch (length) {
+ case 12: c += (u32)k[11]<<24;
+ case 11: c += (u32)k[10]<<16;
+ case 10: c += (u32)k[9]<<8;
+ case 9: c += k[8];
+ case 8: b += (u32)k[7]<<24;
+ case 7: b += (u32)k[6]<<16;
+ case 6: b += (u32)k[5]<<8;
+ case 5: b += k[4];
+ case 4: a += (u32)k[3]<<24;
+ case 3: a += (u32)k[2]<<16;
+ case 2: a += (u32)k[1]<<8;
+ case 1: a += k[0];
+ __jhash_final(a, b, c);
+ case 0: /* Nothing left to add */
+ break;
+ }
+
+ return c;
+}
+
+static inline u32 __jhash_nwords(u32 a, u32 b, u32 c, u32 initval)
+{
+ a += initval;
+ b += initval;
+ c += initval;
+ __jhash_final(a, b, c);
+ return c;
+}
+
+static inline u32 jhash_2words(u32 a, u32 b, u32 initval)
+{
+ return __jhash_nwords(a, b, 0, initval + JHASH_INITVAL + (2 << 2));
+}
+
+#define PCKT_FRAGMENTED 65343
+#define IPV4_HDR_LEN_NO_OPT 20
+#define IPV4_PLUS_ICMP_HDR 28
+#define IPV6_PLUS_ICMP_HDR 48
+#define RING_SIZE 2
+#define MAX_VIPS 12
+#define MAX_REALS 5
+#define CTL_MAP_SIZE 16
+#define CH_RINGS_SIZE (MAX_VIPS * RING_SIZE)
+#define F_IPV6 (1 << 0)
+#define F_HASH_NO_SRC_PORT (1 << 0)
+#define F_ICMP (1 << 0)
+#define F_SYN_SET (1 << 1)
+
+struct packet_description {
+ union {
+ __be32 src;
+ __be32 srcv6[4];
+ };
+ union {
+ __be32 dst;
+ __be32 dstv6[4];
+ };
+ union {
+ __u32 ports;
+ __u16 port16[2];
+ };
+ __u8 proto;
+ __u8 flags;
+};
+
+struct ctl_value {
+ union {
+ __u64 value;
+ __u32 ifindex;
+ __u8 mac[6];
+ };
+};
+
+struct vip_meta {
+ __u32 flags;
+ __u32 vip_num;
+};
+
+struct real_definition {
+ union {
+ __be32 dst;
+ __be32 dstv6[4];
+ };
+ __u8 flags;
+};
+
+struct vip_stats {
+ __u64 bytes;
+ __u64 pkts;
+};
+
+struct eth_hdr {
+ unsigned char eth_dest[ETH_ALEN];
+ unsigned char eth_source[ETH_ALEN];
+ unsigned short eth_proto;
+};
+
+struct bpf_map_def SEC("maps") vip_map = {
+ .type = BPF_MAP_TYPE_HASH,
+ .key_size = sizeof(struct vip),
+ .value_size = sizeof(struct vip_meta),
+ .max_entries = MAX_VIPS,
+};
+
+struct bpf_map_def SEC("maps") ch_rings = {
+ .type = BPF_MAP_TYPE_ARRAY,
+ .key_size = sizeof(__u32),
+ .value_size = sizeof(__u32),
+ .max_entries = CH_RINGS_SIZE,
+};
+
+struct bpf_map_def SEC("maps") reals = {
+ .type = BPF_MAP_TYPE_ARRAY,
+ .key_size = sizeof(__u32),
+ .value_size = sizeof(struct real_definition),
+ .max_entries = MAX_REALS,
+};
+
+struct bpf_map_def SEC("maps") stats = {
+ .type = BPF_MAP_TYPE_PERCPU_ARRAY,
+ .key_size = sizeof(__u32),
+ .value_size = sizeof(struct vip_stats),
+ .max_entries = MAX_VIPS,
+};
+
+struct bpf_map_def SEC("maps") ctl_array = {
+ .type = BPF_MAP_TYPE_ARRAY,
+ .key_size = sizeof(__u32),
+ .value_size = sizeof(struct ctl_value),
+ .max_entries = CTL_MAP_SIZE,
+};
+
+static __always_inline __u32 get_packet_hash(struct packet_description *pckt,
+ bool ipv6)
+{
+ if (ipv6)
+ return jhash_2words(jhash(pckt->srcv6, 16, MAX_VIPS),
+ pckt->ports, CH_RINGS_SIZE);
+ else
+ return jhash_2words(pckt->src, pckt->ports, CH_RINGS_SIZE);
+}
+
+static __always_inline bool get_packet_dst(struct real_definition **real,
+ struct packet_description *pckt,
+ struct vip_meta *vip_info,
+ bool is_ipv6)
+{
+ __u32 hash = get_packet_hash(pckt, is_ipv6) % RING_SIZE;
+ __u32 key = RING_SIZE * vip_info->vip_num + hash;
+ __u32 *real_pos;
+
+ real_pos = bpf_map_lookup_elem(&ch_rings, &key);
+ if (!real_pos)
+ return false;
+ key = *real_pos;
+ *real = bpf_map_lookup_elem(&reals, &key);
+ if (!(*real))
+ return false;
+ return true;
+}
+
+static __always_inline int parse_icmpv6(void *data, void *data_end, __u64 off,
+ struct packet_description *pckt)
+{
+ struct icmp6hdr *icmp_hdr;
+ struct ipv6hdr *ip6h;
+
+ icmp_hdr = data + off;
+ if (icmp_hdr + 1 > data_end)
+ return TC_ACT_SHOT;
+ if (icmp_hdr->icmp6_type != ICMPV6_PKT_TOOBIG)
+ return TC_ACT_OK;
+ off += sizeof(struct icmp6hdr);
+ ip6h = data + off;
+ if (ip6h + 1 > data_end)
+ return TC_ACT_SHOT;
+ pckt->proto = ip6h->nexthdr;
+ pckt->flags |= F_ICMP;
+ memcpy(pckt->srcv6, ip6h->daddr.s6_addr32, 16);
+ memcpy(pckt->dstv6, ip6h->saddr.s6_addr32, 16);
+ return TC_ACT_UNSPEC;
+}
+
+static __always_inline int parse_icmp(void *data, void *data_end, __u64 off,
+ struct packet_description *pckt)
+{
+ struct icmphdr *icmp_hdr;
+ struct iphdr *iph;
+
+ icmp_hdr = data + off;
+ if (icmp_hdr + 1 > data_end)
+ return TC_ACT_SHOT;
+ if (icmp_hdr->type != ICMP_DEST_UNREACH ||
+ icmp_hdr->code != ICMP_FRAG_NEEDED)
+ return TC_ACT_OK;
+ off += sizeof(struct icmphdr);
+ iph = data + off;
+ if (iph + 1 > data_end)
+ return TC_ACT_SHOT;
+ if (iph->ihl != 5)
+ return TC_ACT_SHOT;
+ pckt->proto = iph->protocol;
+ pckt->flags |= F_ICMP;
+ pckt->src = iph->daddr;
+ pckt->dst = iph->saddr;
+ return TC_ACT_UNSPEC;
+}
+
+static __always_inline bool parse_udp(void *data, __u64 off, void *data_end,
+ struct packet_description *pckt)
+{
+ struct udphdr *udp;
+ udp = data + off;
+
+ if (udp + 1 > data_end)
+ return false;
+
+ if (!(pckt->flags & F_ICMP)) {
+ pckt->port16[0] = udp->source;
+ pckt->port16[1] = udp->dest;
+ } else {
+ pckt->port16[0] = udp->dest;
+ pckt->port16[1] = udp->source;
+ }
+ return true;
+}
+
+static __always_inline bool parse_tcp(void *data, __u64 off, void *data_end,
+ struct packet_description *pckt)
+{
+ struct tcphdr *tcp;
+
+ tcp = data + off;
+ if (tcp + 1 > data_end)
+ return false;
+
+ if (tcp->syn)
+ pckt->flags |= F_SYN_SET;
+
+ if (!(pckt->flags & F_ICMP)) {
+ pckt->port16[0] = tcp->source;
+ pckt->port16[1] = tcp->dest;
+ } else {
+ pckt->port16[0] = tcp->dest;
+ pckt->port16[1] = tcp->source;
+ }
+ return true;
+}
+
+static __always_inline int process_packet(void *data, __u64 off, void *data_end,
+ bool is_ipv6, struct __sk_buff *skb)
+{
+ void *pkt_start = (void *)(long)skb->data;
+ struct packet_description pckt = {};
+ struct eth_hdr *eth = pkt_start;
+ struct bpf_tunnel_key tkey = {};
+ struct vip_stats *data_stats;
+ struct real_definition *dst;
+ struct vip_meta *vip_info;
+ struct ctl_value *cval;
+ __u32 v4_intf_pos = 1;
+ __u32 v6_intf_pos = 2;
+ struct ipv6hdr *ip6h;
+ struct vip vip = {};
+ struct iphdr *iph;
+ int tun_flag = 0;
+ __u16 pkt_bytes;
+ __u64 iph_len;
+ __u32 ifindex;
+ __u8 protocol;
+ __u32 vip_num;
+ int action;
+
+ tkey.tunnel_ttl = 64;
+ if (is_ipv6) {
+ ip6h = data + off;
+ if (ip6h + 1 > data_end)
+ return TC_ACT_SHOT;
+
+ iph_len = sizeof(struct ipv6hdr);
+ protocol = ip6h->nexthdr;
+ pckt.proto = protocol;
+ pkt_bytes = bpf_ntohs(ip6h->payload_len);
+ off += iph_len;
+ if (protocol == IPPROTO_FRAGMENT) {
+ return TC_ACT_SHOT;
+ } else if (protocol == IPPROTO_ICMPV6) {
+ action = parse_icmpv6(data, data_end, off, &pckt);
+ if (action >= 0)
+ return action;
+ off += IPV6_PLUS_ICMP_HDR;
+ } else {
+ memcpy(pckt.srcv6, ip6h->saddr.s6_addr32, 16);
+ memcpy(pckt.dstv6, ip6h->daddr.s6_addr32, 16);
+ }
+ } else {
+ iph = data + off;
+ if (iph + 1 > data_end)
+ return TC_ACT_SHOT;
+ if (iph->ihl != 5)
+ return TC_ACT_SHOT;
+
+ protocol = iph->protocol;
+ pckt.proto = protocol;
+ pkt_bytes = bpf_ntohs(iph->tot_len);
+ off += IPV4_HDR_LEN_NO_OPT;
+
+ if (iph->frag_off & PCKT_FRAGMENTED)
+ return TC_ACT_SHOT;
+ if (protocol == IPPROTO_ICMP) {
+ action = parse_icmp(data, data_end, off, &pckt);
+ if (action >= 0)
+ return action;
+ off += IPV4_PLUS_ICMP_HDR;
+ } else {
+ pckt.src = iph->saddr;
+ pckt.dst = iph->daddr;
+ }
+ }
+ protocol = pckt.proto;
+
+ if (protocol == IPPROTO_TCP) {
+ if (!parse_tcp(data, off, data_end, &pckt))
+ return TC_ACT_SHOT;
+ } else if (protocol == IPPROTO_UDP) {
+ if (!parse_udp(data, off, data_end, &pckt))
+ return TC_ACT_SHOT;
+ } else {
+ return TC_ACT_SHOT;
+ }
+
+ if (is_ipv6)
+ memcpy(vip.daddr.v6, pckt.dstv6, 16);
+ else
+ vip.daddr.v4 = pckt.dst;
+
+ vip.dport = pckt.port16[1];
+ vip.protocol = pckt.proto;
+ vip_info = bpf_map_lookup_elem(&vip_map, &vip);
+ if (!vip_info) {
+ vip.dport = 0;
+ vip_info = bpf_map_lookup_elem(&vip_map, &vip);
+ if (!vip_info)
+ return TC_ACT_SHOT;
+ pckt.port16[1] = 0;
+ }
+
+ if (vip_info->flags & F_HASH_NO_SRC_PORT)
+ pckt.port16[0] = 0;
+
+ if (!get_packet_dst(&dst, &pckt, vip_info, is_ipv6))
+ return TC_ACT_SHOT;
+
+ if (dst->flags & F_IPV6) {
+ cval = bpf_map_lookup_elem(&ctl_array, &v6_intf_pos);
+ if (!cval)
+ return TC_ACT_SHOT;
+ ifindex = cval->ifindex;
+ memcpy(tkey.remote_ipv6, dst->dstv6, 16);
+ tun_flag = BPF_F_TUNINFO_IPV6;
+ } else {
+ cval = bpf_map_lookup_elem(&ctl_array, &v4_intf_pos);
+ if (!cval)
+ return TC_ACT_SHOT;
+ ifindex = cval->ifindex;
+ tkey.remote_ipv4 = dst->dst;
+ }
+ vip_num = vip_info->vip_num;
+ data_stats = bpf_map_lookup_elem(&stats, &vip_num);
+ if (!data_stats)
+ return TC_ACT_SHOT;
+ data_stats->pkts++;
+ data_stats->bytes += pkt_bytes;
+ bpf_skb_set_tunnel_key(skb, &tkey, sizeof(tkey), tun_flag);
+ *(u32 *)eth->eth_dest = tkey.remote_ipv4;
+ return bpf_redirect(ifindex, 0);
+}
+
+SEC("l4lb-demo")
+int balancer_ingress(struct __sk_buff *ctx)
+{
+ void *data_end = (void *)(long)ctx->data_end;
+ void *data = (void *)(long)ctx->data;
+ struct eth_hdr *eth = data;
+ __u32 eth_proto;
+ __u32 nh_off;
+
+ nh_off = sizeof(struct eth_hdr);
+ if (data + nh_off > data_end)
+ return TC_ACT_SHOT;
+ eth_proto = eth->eth_proto;
+ if (eth_proto == bpf_htons(ETH_P_IP))
+ return process_packet(data, nh_off, data_end, false, ctx);
+ else if (eth_proto == bpf_htons(ETH_P_IPV6))
+ return process_packet(data, nh_off, data_end, true, ctx);
+ else
+ return TC_ACT_SHOT;
+}
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_lru_map.c b/tools/testing/selftests/bpf/test_lru_map.c
index 00b0aff..8c10c91 100644
--- a/tools/testing/selftests/bpf/test_lru_map.c
+++ b/tools/testing/selftests/bpf/test_lru_map.c
@@ -22,7 +22,7 @@
#include "bpf_util.h"
#define LOCAL_FREE_TARGET (128)
-#define PERCPU_FREE_TARGET (16)
+#define PERCPU_FREE_TARGET (4)
static int nr_cpus;
@@ -191,12 +191,7 @@
int next_cpu = 0;
if (map_flags & BPF_F_NO_COMMON_LRU)
- /* Ther percpu lru list (i.e each cpu has its own LRU
- * list) does not have a local free list. Hence,
- * it will only free old nodes till there is no free
- * from the LRU list. Hence, this test does not apply
- * to BPF_F_NO_COMMON_LRU
- */
+ /* This test is only applicable to common LRU list */
return;
printf("%s (map_type:%d map_flags:0x%X): ", __func__, map_type,
@@ -227,7 +222,7 @@
for (key = 1; key < end_key; key++) {
assert(!bpf_map_lookup_elem(lru_map_fd, &key, value));
assert(!bpf_map_update_elem(expected_map_fd, &key, value,
- BPF_NOEXIST));
+ BPF_NOEXIST));
}
/* Insert 1+tgt_free to 2*tgt_free
@@ -273,12 +268,7 @@
int next_cpu = 0;
if (map_flags & BPF_F_NO_COMMON_LRU)
- /* Ther percpu lru list (i.e each cpu has its own LRU
- * list) does not have a local free list. Hence,
- * it will only free old nodes till there is no free
- * from the LRU list. Hence, this test does not apply
- * to BPF_F_NO_COMMON_LRU
- */
+ /* This test is only applicable to common LRU list */
return;
printf("%s (map_type:%d map_flags:0x%X): ", __func__, map_type,
@@ -290,11 +280,7 @@
assert(batch_size * 2 == tgt_free);
map_size = tgt_free + batch_size;
- if (map_flags & BPF_F_NO_COMMON_LRU)
- lru_map_fd = create_map(map_type, map_flags,
- map_size * nr_cpus);
- else
- lru_map_fd = create_map(map_type, map_flags, map_size);
+ lru_map_fd = create_map(map_type, map_flags, map_size);
assert(lru_map_fd != -1);
expected_map_fd = create_map(BPF_MAP_TYPE_HASH, 0, map_size);
@@ -341,7 +327,7 @@
assert(!bpf_map_lookup_elem(lru_map_fd, &key, value));
assert(value[0] == 4321);
assert(!bpf_map_update_elem(expected_map_fd, &key, value,
- BPF_NOEXIST));
+ BPF_NOEXIST));
}
value[0] = 1234;
@@ -361,7 +347,7 @@
assert(!bpf_map_update_elem(lru_map_fd, &key, value,
BPF_NOEXIST));
assert(!bpf_map_update_elem(expected_map_fd, &key, value,
- BPF_NOEXIST));
+ BPF_NOEXIST));
}
assert(map_equal(lru_map_fd, expected_map_fd));
@@ -387,6 +373,10 @@
unsigned int map_size;
int next_cpu = 0;
+ if (map_flags & BPF_F_NO_COMMON_LRU)
+ /* This test is only applicable to common LRU list */
+ return;
+
printf("%s (map_type:%d map_flags:0x%X): ", __func__, map_type,
map_flags);
@@ -396,11 +386,7 @@
assert(batch_size * 2 == tgt_free);
map_size = tgt_free * 2;
- if (map_flags & BPF_F_NO_COMMON_LRU)
- lru_map_fd = create_map(map_type, map_flags,
- map_size * nr_cpus);
- else
- lru_map_fd = create_map(map_type, map_flags, map_size);
+ lru_map_fd = create_map(map_type, map_flags, map_size);
assert(lru_map_fd != -1);
expected_map_fd = create_map(BPF_MAP_TYPE_HASH, 0, map_size);
@@ -419,7 +405,7 @@
for (key = 1; key < end_key; key++) {
assert(!bpf_map_lookup_elem(lru_map_fd, &key, value));
assert(!bpf_map_update_elem(expected_map_fd, &key, value,
- BPF_NOEXIST));
+ BPF_NOEXIST));
}
/* Add 1+2*tgt_free to tgt_free*5/2
@@ -431,7 +417,7 @@
assert(!bpf_map_update_elem(lru_map_fd, &key, value,
BPF_NOEXIST));
assert(!bpf_map_update_elem(expected_map_fd, &key, value,
- BPF_NOEXIST));
+ BPF_NOEXIST));
}
assert(map_equal(lru_map_fd, expected_map_fd));
@@ -491,7 +477,7 @@
assert(!bpf_map_update_elem(lru_map_fd, &key, value,
BPF_NOEXIST));
assert(!bpf_map_update_elem(expected_map_fd, &key, value,
- BPF_NOEXIST));
+ BPF_NOEXIST));
}
assert(map_equal(lru_map_fd, expected_map_fd));
@@ -566,6 +552,65 @@
printf("Pass\n");
}
+/* Test list rotation for BPF_F_NO_COMMON_LRU map */
+static void test_lru_sanity6(int map_type, int map_flags, int tgt_free)
+{
+ int lru_map_fd, expected_map_fd;
+ unsigned long long key, value[nr_cpus];
+ unsigned int map_size = tgt_free * 2;
+ int next_cpu = 0;
+
+ if (!(map_flags & BPF_F_NO_COMMON_LRU))
+ return;
+
+ printf("%s (map_type:%d map_flags:0x%X): ", __func__, map_type,
+ map_flags);
+
+ assert(sched_next_online(0, &next_cpu) != -1);
+
+ expected_map_fd = create_map(BPF_MAP_TYPE_HASH, 0, map_size);
+ assert(expected_map_fd != -1);
+
+ lru_map_fd = create_map(map_type, map_flags, map_size * nr_cpus);
+ assert(lru_map_fd != -1);
+
+ value[0] = 1234;
+
+ for (key = 1; key <= tgt_free; key++) {
+ assert(!bpf_map_update_elem(lru_map_fd, &key, value,
+ BPF_NOEXIST));
+ assert(!bpf_map_update_elem(expected_map_fd, &key, value,
+ BPF_NOEXIST));
+ }
+
+ for (; key <= tgt_free * 2; key++) {
+ unsigned long long stable_key;
+
+ /* Make ref bit sticky for key: [1, tgt_free] */
+ for (stable_key = 1; stable_key <= tgt_free; stable_key++) {
+ /* Mark the ref bit */
+ assert(!bpf_map_lookup_elem(lru_map_fd, &stable_key,
+ value));
+ }
+ assert(!bpf_map_update_elem(lru_map_fd, &key, value,
+ BPF_NOEXIST));
+ }
+
+ for (; key <= tgt_free * 3; key++) {
+ assert(!bpf_map_update_elem(lru_map_fd, &key, value,
+ BPF_NOEXIST));
+ assert(!bpf_map_update_elem(expected_map_fd, &key, value,
+ BPF_NOEXIST));
+ }
+
+ assert(map_equal(lru_map_fd, expected_map_fd));
+
+ close(expected_map_fd);
+ close(lru_map_fd);
+
+ printf("Pass\n");
+}
+
int main(int argc, char **argv)
{
struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
@@ -593,6 +638,7 @@
test_lru_sanity3(map_types[t], map_flags[f], tgt_free);
test_lru_sanity4(map_types[t], map_flags[f], tgt_free);
test_lru_sanity5(map_types[t], map_flags[f]);
+ test_lru_sanity6(map_types[t], map_flags[f], tgt_free);
printf("\n");
}
diff --git a/tools/testing/selftests/bpf/test_maps.c b/tools/testing/selftests/bpf/test_maps.c
index 20f1871..9331452 100644
--- a/tools/testing/selftests/bpf/test_maps.c
+++ b/tools/testing/selftests/bpf/test_maps.c
@@ -28,7 +28,7 @@
static void test_hashmap(int task, void *data)
{
- long long key, next_key, value;
+ long long key, next_key, first_key, value;
int fd;
fd = bpf_create_map(BPF_MAP_TYPE_HASH, sizeof(key), sizeof(value),
@@ -89,10 +89,13 @@
assert(bpf_map_delete_elem(fd, &key) == -1 && errno == ENOENT);
/* Iterate over two elements. */
+ assert(bpf_map_get_next_key(fd, NULL, &first_key) == 0 &&
+ (first_key == 1 || first_key == 2));
assert(bpf_map_get_next_key(fd, &key, &next_key) == 0 &&
- (next_key == 1 || next_key == 2));
+ (next_key == first_key));
assert(bpf_map_get_next_key(fd, &next_key, &next_key) == 0 &&
- (next_key == 1 || next_key == 2));
+ (next_key == 1 || next_key == 2) &&
+ (next_key != first_key));
assert(bpf_map_get_next_key(fd, &next_key, &next_key) == -1 &&
errno == ENOENT);
@@ -105,6 +108,8 @@
key = 0;
/* Check that map is empty. */
+ assert(bpf_map_get_next_key(fd, NULL, &next_key) == -1 &&
+ errno == ENOENT);
assert(bpf_map_get_next_key(fd, &key, &next_key) == -1 &&
errno == ENOENT);
@@ -132,20 +137,20 @@
static void test_hashmap_percpu(int task, void *data)
{
unsigned int nr_cpus = bpf_num_possible_cpus();
- long long value[nr_cpus];
- long long key, next_key;
+ BPF_DECLARE_PERCPU(long, value);
+ long long key, next_key, first_key;
int expected_key_mask = 0;
int fd, i;
fd = bpf_create_map(BPF_MAP_TYPE_PERCPU_HASH, sizeof(key),
- sizeof(value[0]), 2, map_flags);
+ sizeof(bpf_percpu(value, 0)), 2, map_flags);
if (fd < 0) {
printf("Failed to create hashmap '%s'!\n", strerror(errno));
exit(1);
}
for (i = 0; i < nr_cpus; i++)
- value[i] = i + 100;
+ bpf_percpu(value, i) = i + 100;
key = 1;
/* Insert key=1 element. */
@@ -165,8 +170,9 @@
/* Check that key=1 can be found. Value could be 0 if the lookup
* was run from a different CPU.
*/
- value[0] = 1;
- assert(bpf_map_lookup_elem(fd, &key, value) == 0 && value[0] == 100);
+ bpf_percpu(value, 0) = 1;
+ assert(bpf_map_lookup_elem(fd, &key, value) == 0 &&
+ bpf_percpu(value, 0) == 100);
key = 2;
/* Check that key=2 is not found. */
@@ -193,14 +199,20 @@
assert(bpf_map_delete_elem(fd, &key) == -1 && errno == ENOENT);
/* Iterate over two elements. */
+ assert(bpf_map_get_next_key(fd, NULL, &first_key) == 0 &&
+ ((expected_key_mask & first_key) == first_key));
while (!bpf_map_get_next_key(fd, &key, &next_key)) {
+ if (first_key) {
+ assert(next_key == first_key);
+ first_key = 0;
+ }
assert((expected_key_mask & next_key) == next_key);
expected_key_mask &= ~next_key;
assert(bpf_map_lookup_elem(fd, &next_key, value) == 0);
for (i = 0; i < nr_cpus; i++)
- assert(value[i] == i + 100);
+ assert(bpf_percpu(value, i) == i + 100);
key = next_key;
}
@@ -219,6 +231,8 @@
key = 0;
/* Check that map is empty. */
+ assert(bpf_map_get_next_key(fd, NULL, &next_key) == -1 &&
+ errno == ENOENT);
assert(bpf_map_get_next_key(fd, &key, &next_key) == -1 &&
errno == ENOENT);
@@ -264,6 +278,8 @@
assert(bpf_map_lookup_elem(fd, &key, &value) == -1 && errno == ENOENT);
/* Iterate over two elements. */
+ assert(bpf_map_get_next_key(fd, NULL, &next_key) == 0 &&
+ next_key == 0);
assert(bpf_map_get_next_key(fd, &key, &next_key) == 0 &&
next_key == 0);
assert(bpf_map_get_next_key(fd, &next_key, &next_key) == 0 &&
@@ -281,34 +297,36 @@
static void test_arraymap_percpu(int task, void *data)
{
unsigned int nr_cpus = bpf_num_possible_cpus();
+ BPF_DECLARE_PERCPU(long, values);
int key, next_key, fd, i;
- long long values[nr_cpus];
fd = bpf_create_map(BPF_MAP_TYPE_PERCPU_ARRAY, sizeof(key),
- sizeof(values[0]), 2, 0);
+ sizeof(bpf_percpu(values, 0)), 2, 0);
if (fd < 0) {
printf("Failed to create arraymap '%s'!\n", strerror(errno));
exit(1);
}
for (i = 0; i < nr_cpus; i++)
- values[i] = i + 100;
+ bpf_percpu(values, i) = i + 100;
key = 1;
/* Insert key=1 element. */
assert(bpf_map_update_elem(fd, &key, values, BPF_ANY) == 0);
- values[0] = 0;
+ bpf_percpu(values, 0) = 0;
assert(bpf_map_update_elem(fd, &key, values, BPF_NOEXIST) == -1 &&
errno == EEXIST);
/* Check that key=1 can be found. */
- assert(bpf_map_lookup_elem(fd, &key, values) == 0 && values[0] == 100);
+ assert(bpf_map_lookup_elem(fd, &key, values) == 0 &&
+ bpf_percpu(values, 0) == 100);
key = 0;
/* Check that key=0 is also found and zero initialized. */
assert(bpf_map_lookup_elem(fd, &key, values) == 0 &&
- values[0] == 0 && values[nr_cpus - 1] == 0);
+ bpf_percpu(values, 0) == 0 &&
+ bpf_percpu(values, nr_cpus - 1) == 0);
/* Check that key=2 cannot be inserted due to max_entries limit. */
key = 2;
@@ -319,6 +337,8 @@
assert(bpf_map_lookup_elem(fd, &key, values) == -1 && errno == ENOENT);
/* Iterate over two elements. */
+ assert(bpf_map_get_next_key(fd, NULL, &next_key) == 0 &&
+ next_key == 0);
assert(bpf_map_get_next_key(fd, &key, &next_key) == 0 &&
next_key == 0);
assert(bpf_map_get_next_key(fd, &next_key, &next_key) == 0 &&
@@ -336,15 +356,15 @@
static void test_arraymap_percpu_many_keys(void)
{
unsigned int nr_cpus = bpf_num_possible_cpus();
+ BPF_DECLARE_PERCPU(long, values);
/* nr_keys is not too large otherwise the test stresses percpu
* allocator more than anything else
*/
unsigned int nr_keys = 2000;
- long long values[nr_cpus];
int key, fd, i;
fd = bpf_create_map(BPF_MAP_TYPE_PERCPU_ARRAY, sizeof(key),
- sizeof(values[0]), nr_keys, 0);
+ sizeof(bpf_percpu(values, 0)), nr_keys, 0);
if (fd < 0) {
printf("Failed to create per-cpu arraymap '%s'!\n",
strerror(errno));
@@ -352,19 +372,19 @@
}
for (i = 0; i < nr_cpus; i++)
- values[i] = i + 10;
+ bpf_percpu(values, i) = i + 10;
for (key = 0; key < nr_keys; key++)
assert(bpf_map_update_elem(fd, &key, values, BPF_ANY) == 0);
for (key = 0; key < nr_keys; key++) {
for (i = 0; i < nr_cpus; i++)
- values[i] = 0;
+ bpf_percpu(values, i) = 0;
assert(bpf_map_lookup_elem(fd, &key, values) == 0);
for (i = 0; i < nr_cpus; i++)
- assert(values[i] == i + 10);
+ assert(bpf_percpu(values, i) == i + 10);
}
close(fd);
@@ -400,6 +420,8 @@
errno == E2BIG);
/* Iterate through all elements. */
+ assert(bpf_map_get_next_key(fd, NULL, &key) == 0);
+ key.c = -1;
for (i = 0; i < MAP_SIZE; i++)
assert(bpf_map_get_next_key(fd, &key, &key) == 0);
assert(bpf_map_get_next_key(fd, &key, &key) == -1 && errno == ENOENT);
@@ -499,6 +521,7 @@
errno == EEXIST);
/* Check that all elements were inserted. */
+ assert(bpf_map_get_next_key(fd, NULL, &key) == 0);
key = -1;
for (i = 0; i < MAP_SIZE; i++)
assert(bpf_map_get_next_key(fd, &key, &key) == 0);
@@ -518,6 +541,7 @@
/* Nothing should be left. */
key = -1;
+ assert(bpf_map_get_next_key(fd, NULL, &key) == -1 && errno == ENOENT);
assert(bpf_map_get_next_key(fd, &key, &key) == -1 && errno == ENOENT);
}
diff --git a/tools/testing/selftests/bpf/test_pkt_access.c b/tools/testing/selftests/bpf/test_pkt_access.c
new file mode 100644
index 0000000..39387bb
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_pkt_access.c
@@ -0,0 +1,64 @@
+/* Copyright (c) 2017 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <stddef.h>
+#include <linux/bpf.h>
+#include <linux/if_ether.h>
+#include <linux/if_packet.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <linux/in.h>
+#include <linux/tcp.h>
+#include <linux/pkt_cls.h>
+#include "bpf_helpers.h"
+#include "bpf_endian.h"
+
+#define barrier() __asm__ __volatile__("": : :"memory")
+int _version SEC("version") = 1;
+
+SEC("test1")
+int process(struct __sk_buff *skb)
+{
+ void *data_end = (void *)(long)skb->data_end;
+ void *data = (void *)(long)skb->data;
+ struct ethhdr *eth = (struct ethhdr *)(data);
+ struct tcphdr *tcp = NULL;
+ __u8 proto = 255;
+ __u64 ihl_len;
+
+ if (eth + 1 > data_end)
+ return TC_ACT_SHOT;
+
+ if (eth->h_proto == bpf_htons(ETH_P_IP)) {
+ struct iphdr *iph = (struct iphdr *)(eth + 1);
+
+ if (iph + 1 > data_end)
+ return TC_ACT_SHOT;
+ ihl_len = iph->ihl * 4;
+ proto = iph->protocol;
+ tcp = (struct tcphdr *)((void *)(iph) + ihl_len);
+ } else if (eth->h_proto == bpf_htons(ETH_P_IPV6)) {
+ struct ipv6hdr *ip6h = (struct ipv6hdr *)(eth + 1);
+
+ if (ip6h + 1 > data_end)
+ return TC_ACT_SHOT;
+ ihl_len = sizeof(*ip6h);
+ proto = ip6h->nexthdr;
+ tcp = (struct tcphdr *)((void *)(ip6h) + ihl_len);
+ }
+
+ if (tcp) {
+ if (((void *)(tcp) + 20) > data_end || proto != 6)
+ return TC_ACT_SHOT;
+ barrier(); /* to force ordering of checks */
+ if (((void *)(tcp) + 18) > data_end)
+ return TC_ACT_SHOT;
+ if (tcp->urg_ptr == 123)
+ return TC_ACT_OK;
+ }
+
+ return TC_ACT_UNSPEC;
+}
diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c
new file mode 100644
index 0000000..b59f5ed
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_progs.c
@@ -0,0 +1,299 @@
+/* Copyright (c) 2017 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <stdio.h>
+#include <unistd.h>
+#include <errno.h>
+#include <string.h>
+#include <assert.h>
+#include <stdlib.h>
+
+#include <linux/types.h>
+typedef __u16 __sum16;
+#include <arpa/inet.h>
+#include <linux/if_ether.h>
+#include <linux/if_packet.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <linux/tcp.h>
+
+#include <sys/wait.h>
+#include <sys/resource.h>
+
+#include <linux/bpf.h>
+#include <linux/err.h>
+#include <bpf/bpf.h>
+#include <bpf/libbpf.h>
+#include "test_iptunnel_common.h"
+#include "bpf_util.h"
+#include "bpf_endian.h"
+
+static int error_cnt, pass_cnt;
+
+#define MAGIC_BYTES 123
+
+/* ipv4 test vector */
+static struct {
+ struct ethhdr eth;
+ struct iphdr iph;
+ struct tcphdr tcp;
+} __packed pkt_v4 = {
+ .eth.h_proto = bpf_htons(ETH_P_IP),
+ .iph.ihl = 5,
+ .iph.protocol = 6,
+ .iph.tot_len = bpf_htons(MAGIC_BYTES),
+ .tcp.urg_ptr = 123,
+};
+
+/* ipv6 test vector */
+static struct {
+ struct ethhdr eth;
+ struct ipv6hdr iph;
+ struct tcphdr tcp;
+} __packed pkt_v6 = {
+ .eth.h_proto = bpf_htons(ETH_P_IPV6),
+ .iph.nexthdr = 6,
+ .iph.payload_len = bpf_htons(MAGIC_BYTES),
+ .tcp.urg_ptr = 123,
+};
+
+#define CHECK(condition, tag, format...) ({ \
+ int __ret = !!(condition); \
+ if (__ret) { \
+ error_cnt++; \
+ printf("%s:FAIL:%s ", __func__, tag); \
+ printf(format); \
+ } else { \
+ pass_cnt++; \
+ printf("%s:PASS:%s %d nsec\n", __func__, tag, duration);\
+ } \
+})
+
+static int bpf_prog_load(const char *file, enum bpf_prog_type type,
+ struct bpf_object **pobj, int *prog_fd)
+{
+ struct bpf_program *prog;
+ struct bpf_object *obj;
+ int err;
+
+ obj = bpf_object__open(file);
+ if (IS_ERR(obj)) {
+ error_cnt++;
+ return -ENOENT;
+ }
+
+ prog = bpf_program__next(NULL, obj);
+ if (!prog) {
+ bpf_object__close(obj);
+ error_cnt++;
+ return -ENOENT;
+ }
+
+ bpf_program__set_type(prog, type);
+ err = bpf_object__load(obj);
+ if (err) {
+ bpf_object__close(obj);
+ error_cnt++;
+ return -EINVAL;
+ }
+
+ *pobj = obj;
+ *prog_fd = bpf_program__fd(prog);
+ return 0;
+}
+
+static int bpf_find_map(const char *test, struct bpf_object *obj,
+ const char *name)
+{
+ struct bpf_map *map;
+
+ map = bpf_object__find_map_by_name(obj, name);
+ if (!map) {
+ printf("%s:FAIL:map '%s' not found\n", test, name);
+ error_cnt++;
+ return -1;
+ }
+ return bpf_map__fd(map);
+}
+
+static void test_pkt_access(void)
+{
+ const char *file = "./test_pkt_access.o";
+ struct bpf_object *obj;
+ __u32 duration, retval;
+ int err, prog_fd;
+
+ err = bpf_prog_load(file, BPF_PROG_TYPE_SCHED_CLS, &obj, &prog_fd);
+ if (err)
+ return;
+
+ err = bpf_prog_test_run(prog_fd, 100000, &pkt_v4, sizeof(pkt_v4),
+ NULL, NULL, &retval, &duration);
+ CHECK(err || errno || retval, "ipv4",
+ "err %d errno %d retval %d duration %d\n",
+ err, errno, retval, duration);
+
+ err = bpf_prog_test_run(prog_fd, 100000, &pkt_v6, sizeof(pkt_v6),
+ NULL, NULL, &retval, &duration);
+ CHECK(err || errno || retval, "ipv6",
+ "err %d errno %d retval %d duration %d\n",
+ err, errno, retval, duration);
+ bpf_object__close(obj);
+}
+
+static void test_xdp(void)
+{
+ struct vip key4 = {.protocol = 6, .family = AF_INET};
+ struct vip key6 = {.protocol = 6, .family = AF_INET6};
+ struct iptnl_info value4 = {.family = AF_INET};
+ struct iptnl_info value6 = {.family = AF_INET6};
+ const char *file = "./test_xdp.o";
+ struct bpf_object *obj;
+ char buf[128];
+ struct ipv6hdr *iph6 = (void *)buf + sizeof(struct ethhdr);
+ struct iphdr *iph = (void *)buf + sizeof(struct ethhdr);
+ __u32 duration, retval, size;
+ int err, prog_fd, map_fd;
+
+ err = bpf_prog_load(file, BPF_PROG_TYPE_XDP, &obj, &prog_fd);
+ if (err)
+ return;
+
+ map_fd = bpf_find_map(__func__, obj, "vip2tnl");
+ if (map_fd < 0)
+ goto out;
+ bpf_map_update_elem(map_fd, &key4, &value4, 0);
+ bpf_map_update_elem(map_fd, &key6, &value6, 0);
+
+ err = bpf_prog_test_run(prog_fd, 1, &pkt_v4, sizeof(pkt_v4),
+ buf, &size, &retval, &duration);
+
+ CHECK(err || errno || retval != XDP_TX || size != 74 ||
+ iph->protocol != IPPROTO_IPIP, "ipv4",
+ "err %d errno %d retval %d size %d\n",
+ err, errno, retval, size);
+
+ err = bpf_prog_test_run(prog_fd, 1, &pkt_v6, sizeof(pkt_v6),
+ buf, &size, &retval, &duration);
+ CHECK(err || errno || retval != XDP_TX || size != 114 ||
+ iph6->nexthdr != IPPROTO_IPV6, "ipv6",
+ "err %d errno %d retval %d size %d\n",
+ err, errno, retval, size);
+out:
+ bpf_object__close(obj);
+}
+
+#define MAGIC_VAL 0x1234
+#define NUM_ITER 100000
+#define VIP_NUM 5
+
+static void test_l4lb(void)
+{
+ unsigned int nr_cpus = bpf_num_possible_cpus();
+ const char *file = "./test_l4lb.o";
+ struct vip key = {.protocol = 6};
+ struct vip_meta {
+ __u32 flags;
+ __u32 vip_num;
+ } value = {.vip_num = VIP_NUM};
+ __u32 stats_key = VIP_NUM;
+ struct vip_stats {
+ __u64 bytes;
+ __u64 pkts;
+ } stats[nr_cpus];
+ struct real_definition {
+ union {
+ __be32 dst;
+ __be32 dstv6[4];
+ };
+ __u8 flags;
+ } real_def = {.dst = MAGIC_VAL};
+ __u32 ch_key = 11, real_num = 3;
+ __u32 duration, retval, size;
+ int err, i, prog_fd, map_fd;
+ __u64 bytes = 0, pkts = 0;
+ struct bpf_object *obj;
+ char buf[128];
+ u32 *magic = (u32 *)buf;
+
+ err = bpf_prog_load(file, BPF_PROG_TYPE_SCHED_CLS, &obj, &prog_fd);
+ if (err)
+ return;
+
+ map_fd = bpf_find_map(__func__, obj, "vip_map");
+ if (map_fd < 0)
+ goto out;
+ bpf_map_update_elem(map_fd, &key, &value, 0);
+
+ map_fd = bpf_find_map(__func__, obj, "ch_rings");
+ if (map_fd < 0)
+ goto out;
+ bpf_map_update_elem(map_fd, &ch_key, &real_num, 0);
+
+ map_fd = bpf_find_map(__func__, obj, "reals");
+ if (map_fd < 0)
+ goto out;
+ bpf_map_update_elem(map_fd, &real_num, &real_def, 0);
+
+ err = bpf_prog_test_run(prog_fd, NUM_ITER, &pkt_v4, sizeof(pkt_v4),
+ buf, &size, &retval, &duration);
+ CHECK(err || errno || retval != 7/*TC_ACT_REDIRECT*/ || size != 54 ||
+ *magic != MAGIC_VAL, "ipv4",
+ "err %d errno %d retval %d size %d magic %x\n",
+ err, errno, retval, size, *magic);
+
+ err = bpf_prog_test_run(prog_fd, NUM_ITER, &pkt_v6, sizeof(pkt_v6),
+ buf, &size, &retval, &duration);
+ CHECK(err || errno || retval != 7/*TC_ACT_REDIRECT*/ || size != 74 ||
+ *magic != MAGIC_VAL, "ipv6",
+ "err %d errno %d retval %d size %d magic %x\n",
+ err, errno, retval, size, *magic);
+
+ map_fd = bpf_find_map(__func__, obj, "stats");
+ if (map_fd < 0)
+ goto out;
+ bpf_map_lookup_elem(map_fd, &stats_key, stats);
+ for (i = 0; i < nr_cpus; i++) {
+ bytes += stats[i].bytes;
+ pkts += stats[i].pkts;
+ }
+ if (bytes != MAGIC_BYTES * NUM_ITER * 2 || pkts != NUM_ITER * 2) {
+ error_cnt++;
+ printf("test_l4lb:FAIL:stats %lld %lld\n", bytes, pkts);
+ }
+out:
+ bpf_object__close(obj);
+}
+
+static void test_tcp_estats(void)
+{
+ const char *file = "./test_tcp_estats.o";
+ int err, prog_fd;
+ struct bpf_object *obj;
+ __u32 duration = 0;
+
+ err = bpf_prog_load(file, BPF_PROG_TYPE_TRACEPOINT, &obj, &prog_fd);
+ CHECK(err, "", "err %d errno %d\n", err, errno);
+ if (err)
+ return;
+
+ bpf_object__close(obj);
+}
+
+int main(void)
+{
+ struct rlimit rinf = { RLIM_INFINITY, RLIM_INFINITY };
+
+ setrlimit(RLIMIT_MEMLOCK, &rinf);
+
+ test_pkt_access();
+ test_xdp();
+ test_l4lb();
+ test_tcp_estats();
+
+ printf("Summary: %d PASSED, %d FAILED\n", pass_cnt, error_cnt);
+ return 0;
+}
diff --git a/tools/testing/selftests/bpf/test_tcp_estats.c b/tools/testing/selftests/bpf/test_tcp_estats.c
new file mode 100644
index 0000000..bee3bbe
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_tcp_estats.c
@@ -0,0 +1,258 @@
+/* Copyright (c) 2017 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+
+/* This program shows clang/llvm is able to generate code pattern
+ * like:
+ * _tcp_send_active_reset:
+ * 0: bf 16 00 00 00 00 00 00 r6 = r1
+ * ......
+ * 335: b7 01 00 00 0f 00 00 00 r1 = 15
+ * 336: 05 00 48 00 00 00 00 00 goto 72
+ *
+ * LBB0_3:
+ * 337: b7 01 00 00 01 00 00 00 r1 = 1
+ * 338: 63 1a d0 ff 00 00 00 00 *(u32 *)(r10 - 48) = r1
+ * 408: b7 01 00 00 03 00 00 00 r1 = 3
+ *
+ * LBB0_4:
+ * 409: 71 a2 fe ff 00 00 00 00 r2 = *(u8 *)(r10 - 2)
+ * 410: bf a7 00 00 00 00 00 00 r7 = r10
+ * 411: 07 07 00 00 b8 ff ff ff r7 += -72
+ * 412: bf 73 00 00 00 00 00 00 r3 = r7
+ * 413: 0f 13 00 00 00 00 00 00 r3 += r1
+ * 414: 73 23 2d 00 00 00 00 00 *(u8 *)(r3 + 45) = r2
+ *
+ * From the above code snippet, the code generated by the compiler
+ * is reasonable. The "r1" is assigned to different values in basic
+ * blocks "_tcp_send_active_reset" and "LBB0_3", and used in "LBB0_4".
+ * The verifier should be able to handle such code patterns.
+ */
+#include <string.h>
+#include <linux/bpf.h>
+#include <linux/ipv6.h>
+#include <linux/version.h>
+#include <sys/socket.h>
+#include "bpf_helpers.h"
+
+#define _(P) ({typeof(P) val = 0; bpf_probe_read(&val, sizeof(val), &P); val;})
+#define TCP_ESTATS_MAGIC 0xBAADBEEF
+
+/* This test case needs "sock" and "pt_regs" data structure.
+ * Recursively, "sock" needs "sock_common" and "inet_sock".
+ * However, this is a unit test case only for
+ * verifier purpose without bpf program execution.
+ * We can safely mock much simpler data structures, basically
+ * only taking the necessary fields from kernel headers.
+ */
+typedef __u32 __bitwise __portpair;
+typedef __u64 __bitwise __addrpair;
+
+struct sock_common {
+ unsigned short skc_family;
+ union {
+ __addrpair skc_addrpair;
+ struct {
+ __be32 skc_daddr;
+ __be32 skc_rcv_saddr;
+ };
+ };
+ union {
+ __portpair skc_portpair;
+ struct {
+ __be16 skc_dport;
+ __u16 skc_num;
+ };
+ };
+ struct in6_addr skc_v6_daddr;
+ struct in6_addr skc_v6_rcv_saddr;
+};
+
+struct sock {
+ struct sock_common __sk_common;
+#define sk_family __sk_common.skc_family
+#define sk_v6_daddr __sk_common.skc_v6_daddr
+#define sk_v6_rcv_saddr __sk_common.skc_v6_rcv_saddr
+};
+
+struct inet_sock {
+ struct sock sk;
+#define inet_daddr sk.__sk_common.skc_daddr
+#define inet_dport sk.__sk_common.skc_dport
+ __be32 inet_saddr;
+ __be16 inet_sport;
+};
+
+struct pt_regs {
+ long di;
+};
+
+static inline struct inet_sock *inet_sk(const struct sock *sk)
+{
+ return (struct inet_sock *)sk;
+}
+
+/* Define various data structures for state recording.
+ * Some fields are not used due to test simplification.
+ */
+enum tcp_estats_addrtype {
+ TCP_ESTATS_ADDRTYPE_IPV4 = 1,
+ TCP_ESTATS_ADDRTYPE_IPV6 = 2
+};
+
+enum tcp_estats_event_type {
+ TCP_ESTATS_ESTABLISH,
+ TCP_ESTATS_PERIODIC,
+ TCP_ESTATS_TIMEOUT,
+ TCP_ESTATS_RETRANSMIT_TIMEOUT,
+ TCP_ESTATS_RETRANSMIT_OTHER,
+ TCP_ESTATS_SYN_RETRANSMIT,
+ TCP_ESTATS_SYNACK_RETRANSMIT,
+ TCP_ESTATS_TERM,
+ TCP_ESTATS_TX_RESET,
+ TCP_ESTATS_RX_RESET,
+ TCP_ESTATS_WRITE_TIMEOUT,
+ TCP_ESTATS_CONN_TIMEOUT,
+ TCP_ESTATS_ACK_LATENCY,
+ TCP_ESTATS_NEVENTS,
+};
+
+struct tcp_estats_event {
+ int pid;
+ int cpu;
+ unsigned long ts;
+ unsigned int magic;
+ enum tcp_estats_event_type event_type;
+};
+
+/* The below data structure is packed in order for
+ * llvm compiler to generate expected code.
+ */
+struct tcp_estats_conn_id {
+ unsigned int localaddressType;
+ struct {
+ unsigned char data[16];
+ } localaddress;
+ struct {
+ unsigned char data[16];
+ } remaddress;
+ unsigned short localport;
+ unsigned short remport;
+} __attribute__((__packed__));
+
+struct tcp_estats_basic_event {
+ struct tcp_estats_event event;
+ struct tcp_estats_conn_id conn_id;
+};
+
+struct bpf_map_def SEC("maps") ev_record_map = {
+ .type = BPF_MAP_TYPE_HASH,
+ .key_size = sizeof(__u32),
+ .value_size = sizeof(struct tcp_estats_basic_event),
+ .max_entries = 1024,
+};
+
+struct dummy_tracepoint_args {
+ unsigned long long pad;
+ struct sock *sock;
+};
+
+static __always_inline void tcp_estats_ev_init(struct tcp_estats_event *event,
+ enum tcp_estats_event_type type)
+{
+ event->magic = TCP_ESTATS_MAGIC;
+ event->ts = bpf_ktime_get_ns();
+ event->event_type = type;
+}
+
+static __always_inline void unaligned_u32_set(unsigned char *to, __u8 *from)
+{
+ to[0] = _(from[0]);
+ to[1] = _(from[1]);
+ to[2] = _(from[2]);
+ to[3] = _(from[3]);
+}
+
+static __always_inline void conn_id_ipv4_init(struct tcp_estats_conn_id *conn_id,
+ __be32 *saddr, __be32 *daddr)
+{
+ conn_id->localaddressType = TCP_ESTATS_ADDRTYPE_IPV4;
+
+ unaligned_u32_set(conn_id->localaddress.data, (__u8 *)saddr);
+ unaligned_u32_set(conn_id->remaddress.data, (__u8 *)daddr);
+}
+
+static __always_inline void conn_id_ipv6_init(struct tcp_estats_conn_id *conn_id,
+ __be32 *saddr, __be32 *daddr)
+{
+ conn_id->localaddressType = TCP_ESTATS_ADDRTYPE_IPV6;
+
+ unaligned_u32_set(conn_id->localaddress.data, (__u8 *)saddr);
+ unaligned_u32_set(conn_id->localaddress.data + sizeof(__u32),
+ (__u8 *)(saddr + 1));
+ unaligned_u32_set(conn_id->localaddress.data + sizeof(__u32) * 2,
+ (__u8 *)(saddr + 2));
+ unaligned_u32_set(conn_id->localaddress.data + sizeof(__u32) * 3,
+ (__u8 *)(saddr + 3));
+
+ unaligned_u32_set(conn_id->remaddress.data,
+ (__u8 *)(daddr));
+ unaligned_u32_set(conn_id->remaddress.data + sizeof(__u32),
+ (__u8 *)(daddr + 1));
+ unaligned_u32_set(conn_id->remaddress.data + sizeof(__u32) * 2,
+ (__u8 *)(daddr + 2));
+ unaligned_u32_set(conn_id->remaddress.data + sizeof(__u32) * 3,
+ (__u8 *)(daddr + 3));
+}
+
+static __always_inline void tcp_estats_conn_id_init(struct tcp_estats_conn_id *conn_id,
+ struct sock *sk)
+{
+ conn_id->localport = _(inet_sk(sk)->inet_sport);
+ conn_id->remport = _(inet_sk(sk)->inet_dport);
+
+ if (_(sk->sk_family) == AF_INET6)
+ conn_id_ipv6_init(conn_id,
+ sk->sk_v6_rcv_saddr.s6_addr32,
+ sk->sk_v6_daddr.s6_addr32);
+ else
+ conn_id_ipv4_init(conn_id,
+ &inet_sk(sk)->inet_saddr,
+ &inet_sk(sk)->inet_daddr);
+}
+
+static __always_inline void tcp_estats_init(struct sock *sk,
+ struct tcp_estats_event *event,
+ struct tcp_estats_conn_id *conn_id,
+ enum tcp_estats_event_type type)
+{
+ tcp_estats_ev_init(event, type);
+ tcp_estats_conn_id_init(conn_id, sk);
+}
+
+static __always_inline void send_basic_event(struct sock *sk,
+ enum tcp_estats_event_type type)
+{
+ struct tcp_estats_basic_event ev;
+ __u32 key = bpf_get_prandom_u32();
+
+ memset(&ev, 0, sizeof(ev));
+ tcp_estats_init(sk, &ev.event, &ev.conn_id, type);
+ bpf_map_update_elem(&ev_record_map, &key, &ev, BPF_ANY);
+}
+
+SEC("dummy_tracepoint")
+int _dummy_tracepoint(struct dummy_tracepoint_args *arg)
+{
+ if (!arg->sock)
+ return 0;
+
+ send_basic_event(arg->sock, TCP_ESTATS_TX_RESET);
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
+__u32 _version SEC("version") = 1; /* ignored by tracepoints, required by libbpf.a */
diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c
index c848e90..3773562 100644
--- a/tools/testing/selftests/bpf/test_verifier.c
+++ b/tools/testing/selftests/bpf/test_verifier.c
@@ -46,6 +46,7 @@
#define MAX_INSNS 512
#define MAX_FIXUPS 8
+#define MAX_NR_MAPS 4
#define F_NEEDS_EFFICIENT_UNALIGNED_ACCESS (1 << 0)
@@ -55,6 +56,7 @@
int fixup_map1[MAX_FIXUPS];
int fixup_map2[MAX_FIXUPS];
int fixup_prog[MAX_FIXUPS];
+ int fixup_map_in_map[MAX_FIXUPS];
const char *errstr;
const char *errstr_unpriv;
enum {
@@ -189,6 +191,86 @@
.result = REJECT,
},
{
+ "test6 ld_imm64",
+ .insns = {
+ BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, 0, 0, 0, 0),
+ BPF_RAW_INSN(0, 0, 0, 0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ },
+ {
+ "test7 ld_imm64",
+ .insns = {
+ BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, 0, 0, 0, 1),
+ BPF_RAW_INSN(0, 0, 0, 0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ },
+ {
+ "test8 ld_imm64",
+ .insns = {
+ BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, 0, 0, 1, 1),
+ BPF_RAW_INSN(0, 0, 0, 0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "uses reserved fields",
+ .result = REJECT,
+ },
+ {
+ "test9 ld_imm64",
+ .insns = {
+ BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, 0, 0, 0, 1),
+ BPF_RAW_INSN(0, 0, 0, 1, 1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_ld_imm64 insn",
+ .result = REJECT,
+ },
+ {
+ "test10 ld_imm64",
+ .insns = {
+ BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, 0, 0, 0, 1),
+ BPF_RAW_INSN(0, BPF_REG_1, 0, 0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_ld_imm64 insn",
+ .result = REJECT,
+ },
+ {
+ "test11 ld_imm64",
+ .insns = {
+ BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, 0, 0, 0, 1),
+ BPF_RAW_INSN(0, 0, BPF_REG_1, 0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_ld_imm64 insn",
+ .result = REJECT,
+ },
+ {
+ "test12 ld_imm64",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, 0, BPF_REG_1, 0, 1),
+ BPF_RAW_INSN(0, 0, 0, 0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "not pointing to valid bpf_map",
+ .result = REJECT,
+ },
+ {
+ "test13 ld_imm64",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, 0, BPF_REG_1, 0, 1),
+ BPF_RAW_INSN(0, 0, BPF_REG_1, 0, 1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_ld_imm64 insn",
+ .result = REJECT,
+ },
+ {
"no bpf_exit",
.insns = {
BPF_ALU64_REG(BPF_MOV, BPF_REG_0, BPF_REG_2),
@@ -329,6 +411,30 @@
.result = REJECT,
},
{
+ "invalid fp arithmetic",
+ /* If this gets ever changed, make sure JITs can deal with it. */
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_SUB, BPF_REG_1, 8),
+ BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr_unpriv = "R1 pointer arithmetic",
+ .result_unpriv = REJECT,
+ .errstr = "R1 invalid mem access",
+ .result = REJECT,
+ },
+ {
+ "non-invalid fp arithmetic",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_0, -8),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ },
+ {
"invalid argument register",
.insns = {
BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
@@ -770,6 +876,9 @@
BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1,
offsetof(struct __sk_buff, vlan_tci)),
BPF_JMP_IMM(BPF_JGE, BPF_REG_0, 0, 0),
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1,
+ offsetof(struct __sk_buff, napi_id)),
+ BPF_JMP_IMM(BPF_JGE, BPF_REG_0, 0, 0),
BPF_EXIT_INSN(),
},
.result = ACCEPT,
@@ -1796,6 +1905,20 @@
.result = ACCEPT,
},
{
+ "unpriv: adding of fp",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_MOV64_IMM(BPF_REG_1, 0),
+ BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_10),
+ BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0, -8),
+ BPF_EXIT_INSN(),
+ },
+ .errstr_unpriv = "pointer arithmetic prohibited",
+ .result_unpriv = REJECT,
+ .errstr = "R1 invalid mem access",
+ .result = REJECT,
+ },
+ {
"unpriv: cmp of stack pointer",
.insns = {
BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
@@ -1809,16 +1932,22 @@
.result = ACCEPT,
},
{
- "unpriv: obfuscate stack pointer",
+ "stack pointer arithmetic",
.insns = {
- BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
- BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
- BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_MOV64_IMM(BPF_REG_1, 4),
+ BPF_JMP_IMM(BPF_JA, 0, 0, 0),
+ BPF_MOV64_REG(BPF_REG_7, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_7, -10),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_7),
+ BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1),
+ BPF_ST_MEM(0, BPF_REG_2, 4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_7),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, 8),
+ BPF_ST_MEM(0, BPF_REG_2, 4, 0),
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
- .errstr_unpriv = "R2 pointer arithmetic",
- .result_unpriv = REJECT,
.result = ACCEPT,
},
{
@@ -2467,6 +2596,25 @@
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
},
{
+ "direct packet access: test16 (arith on data_end)",
+ .insns = {
+ BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1,
+ offsetof(struct __sk_buff, data)),
+ BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1,
+ offsetof(struct __sk_buff, data_end)),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_2),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, 16),
+ BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1),
+ BPF_STX_MEM(BPF_B, BPF_REG_2, BPF_REG_2, 0),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid access to packet",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+ },
+ {
"helper access to packet: test1, valid packet_ptr range",
.insns = {
BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1,
@@ -4720,6 +4868,75 @@
.result = REJECT,
.result_unpriv = REJECT,
.flags = F_NEEDS_EFFICIENT_UNALIGNED_ACCESS,
+ },
+ {
+ "map in map access",
+ .insns = {
+ BPF_ST_MEM(0, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 5),
+ BPF_ST_MEM(0, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_MOV64_REG(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_in_map = { 3 },
+ .result = ACCEPT,
+ },
+ {
+ "invalid inner map pointer",
+ .insns = {
+ BPF_ST_MEM(0, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 6),
+ BPF_ST_MEM(0, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 8),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_MOV64_REG(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_in_map = { 3 },
+ .errstr = "R1 type=inv expected=map_ptr",
+ .errstr_unpriv = "R1 pointer arithmetic prohibited",
+ .result = REJECT,
+ },
+ {
+ "forgot null checking on the inner map pointer",
+ .insns = {
+ BPF_ST_MEM(0, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_ST_MEM(0, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_MOV64_REG(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_in_map = { 3 },
+ .errstr = "R1 type=map_value_or_null expected=map_ptr",
+ .result = REJECT,
}
};
@@ -4757,42 +4974,73 @@
return fd;
}
+static int create_map_in_map(void)
+{
+ int inner_map_fd, outer_map_fd;
+
+ inner_map_fd = bpf_create_map(BPF_MAP_TYPE_ARRAY, sizeof(int),
+ sizeof(int), 1, 0);
+ if (inner_map_fd < 0) {
+ printf("Failed to create array '%s'!\n", strerror(errno));
+ return inner_map_fd;
+ }
+
+ outer_map_fd = bpf_create_map_in_map(BPF_MAP_TYPE_ARRAY_OF_MAPS,
+ sizeof(int), inner_map_fd, 1, 0);
+ if (outer_map_fd < 0)
+ printf("Failed to create array of maps '%s'!\n",
+ strerror(errno));
+
+ close(inner_map_fd);
+
+ return outer_map_fd;
+}
+
static char bpf_vlog[32768];
static void do_test_fixup(struct bpf_test *test, struct bpf_insn *prog,
- int *fd_f1, int *fd_f2, int *fd_f3)
+ int *map_fds)
{
int *fixup_map1 = test->fixup_map1;
int *fixup_map2 = test->fixup_map2;
int *fixup_prog = test->fixup_prog;
+ int *fixup_map_in_map = test->fixup_map_in_map;
/* Allocating HTs with 1 elem is fine here, since we only test
* for verifier and not do a runtime lookup, so the only thing
* that really matters is value size in this case.
*/
if (*fixup_map1) {
- *fd_f1 = create_map(sizeof(long long), 1);
+ map_fds[0] = create_map(sizeof(long long), 1);
do {
- prog[*fixup_map1].imm = *fd_f1;
+ prog[*fixup_map1].imm = map_fds[0];
fixup_map1++;
} while (*fixup_map1);
}
if (*fixup_map2) {
- *fd_f2 = create_map(sizeof(struct test_val), 1);
+ map_fds[1] = create_map(sizeof(struct test_val), 1);
do {
- prog[*fixup_map2].imm = *fd_f2;
+ prog[*fixup_map2].imm = map_fds[1];
fixup_map2++;
} while (*fixup_map2);
}
if (*fixup_prog) {
- *fd_f3 = create_prog_array();
+ map_fds[2] = create_prog_array();
do {
- prog[*fixup_prog].imm = *fd_f3;
+ prog[*fixup_prog].imm = map_fds[2];
fixup_prog++;
} while (*fixup_prog);
}
+
+ if (*fixup_map_in_map) {
+ map_fds[3] = create_map_in_map();
+ do {
+ prog[*fixup_map_in_map].imm = map_fds[3];
+ fixup_map_in_map++;
+ } while (*fixup_map_in_map);
+ }
}
static void do_test_single(struct bpf_test *test, bool unpriv,
@@ -4802,10 +5050,14 @@
struct bpf_insn *prog = test->insns;
int prog_len = probe_filter_length(prog);
int prog_type = test->prog_type;
- int fd_f1 = -1, fd_f2 = -1, fd_f3 = -1;
+ int map_fds[MAX_NR_MAPS];
const char *expected_err;
+ int i;
- do_test_fixup(test, prog, &fd_f1, &fd_f2, &fd_f3);
+ for (i = 0; i < MAX_NR_MAPS; i++)
+ map_fds[i] = -1;
+
+ do_test_fixup(test, prog, map_fds);
fd_prog = bpf_load_program(prog_type ? : BPF_PROG_TYPE_SOCKET_FILTER,
prog, prog_len, "GPL", 0, bpf_vlog,
@@ -4848,9 +5100,8 @@
" (NOTE: reject due to unknown alignment)" : "");
close_fds:
close(fd_prog);
- close(fd_f1);
- close(fd_f2);
- close(fd_f3);
+ for (i = 0; i < MAX_NR_MAPS; i++)
+ close(map_fds[i]);
sched_yield();
return;
fail_log:
diff --git a/tools/testing/selftests/bpf/test_xdp.c b/tools/testing/selftests/bpf/test_xdp.c
new file mode 100644
index 0000000..5e7df8b
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_xdp.c
@@ -0,0 +1,235 @@
+/* Copyright (c) 2016,2017 Facebook
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ */
+#include <stddef.h>
+#include <string.h>
+#include <linux/bpf.h>
+#include <linux/if_ether.h>
+#include <linux/if_packet.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <linux/in.h>
+#include <linux/udp.h>
+#include <linux/tcp.h>
+#include <linux/pkt_cls.h>
+#include <sys/socket.h>
+#include "bpf_helpers.h"
+#include "bpf_endian.h"
+#include "test_iptunnel_common.h"
+
+int _version SEC("version") = 1;
+
+struct bpf_map_def SEC("maps") rxcnt = {
+ .type = BPF_MAP_TYPE_PERCPU_ARRAY,
+ .key_size = sizeof(__u32),
+ .value_size = sizeof(__u64),
+ .max_entries = 256,
+};
+
+struct bpf_map_def SEC("maps") vip2tnl = {
+ .type = BPF_MAP_TYPE_HASH,
+ .key_size = sizeof(struct vip),
+ .value_size = sizeof(struct iptnl_info),
+ .max_entries = MAX_IPTNL_ENTRIES,
+};
+
+static __always_inline void count_tx(__u32 protocol)
+{
+ __u64 *rxcnt_count;
+
+ rxcnt_count = bpf_map_lookup_elem(&rxcnt, &protocol);
+ if (rxcnt_count)
+ *rxcnt_count += 1;
+}
+
+static __always_inline int get_dport(void *trans_data, void *data_end,
+ __u8 protocol)
+{
+ struct tcphdr *th;
+ struct udphdr *uh;
+
+ switch (protocol) {
+ case IPPROTO_TCP:
+ th = (struct tcphdr *)trans_data;
+ if (th + 1 > data_end)
+ return -1;
+ return th->dest;
+ case IPPROTO_UDP:
+ uh = (struct udphdr *)trans_data;
+ if (uh + 1 > data_end)
+ return -1;
+ return uh->dest;
+ default:
+ return 0;
+ }
+}
+
+static __always_inline void set_ethhdr(struct ethhdr *new_eth,
+ const struct ethhdr *old_eth,
+ const struct iptnl_info *tnl,
+ __be16 h_proto)
+{
+ memcpy(new_eth->h_source, old_eth->h_dest, sizeof(new_eth->h_source));
+ memcpy(new_eth->h_dest, tnl->dmac, sizeof(new_eth->h_dest));
+ new_eth->h_proto = h_proto;
+}
+
+static __always_inline int handle_ipv4(struct xdp_md *xdp)
+{
+ void *data_end = (void *)(long)xdp->data_end;
+ void *data = (void *)(long)xdp->data;
+ struct iptnl_info *tnl;
+ struct ethhdr *new_eth;
+ struct ethhdr *old_eth;
+ struct iphdr *iph = data + sizeof(struct ethhdr);
+ __u16 *next_iph;
+ __u16 payload_len;
+ struct vip vip = {};
+ int dport;
+ __u32 csum = 0;
+ int i;
+
+ if (iph + 1 > data_end)
+ return XDP_DROP;
+
+ dport = get_dport(iph + 1, data_end, iph->protocol);
+ if (dport == -1)
+ return XDP_DROP;
+
+ vip.protocol = iph->protocol;
+ vip.family = AF_INET;
+ vip.daddr.v4 = iph->daddr;
+ vip.dport = dport;
+ payload_len = bpf_ntohs(iph->tot_len);
+
+ tnl = bpf_map_lookup_elem(&vip2tnl, &vip);
+ /* It only does v4-in-v4 */
+ if (!tnl || tnl->family != AF_INET)
+ return XDP_PASS;
+
+ if (bpf_xdp_adjust_head(xdp, 0 - (int)sizeof(struct iphdr)))
+ return XDP_DROP;
+
+ data = (void *)(long)xdp->data;
+ data_end = (void *)(long)xdp->data_end;
+
+ new_eth = data;
+ iph = data + sizeof(*new_eth);
+ old_eth = data + sizeof(*iph);
+
+ if (new_eth + 1 > data_end ||
+ old_eth + 1 > data_end ||
+ iph + 1 > data_end)
+ return XDP_DROP;
+
+ set_ethhdr(new_eth, old_eth, tnl, bpf_htons(ETH_P_IP));
+
+ iph->version = 4;
+ iph->ihl = sizeof(*iph) >> 2;
+ iph->frag_off = 0;
+ iph->protocol = IPPROTO_IPIP;
+ iph->check = 0;
+ iph->tos = 0;
+ iph->tot_len = bpf_htons(payload_len + sizeof(*iph));
+ iph->daddr = tnl->daddr.v4;
+ iph->saddr = tnl->saddr.v4;
+ iph->ttl = 8;
+
+ next_iph = (__u16 *)iph;
+#pragma clang loop unroll(full)
+ for (i = 0; i < sizeof(*iph) >> 1; i++)
+ csum += *next_iph++;
+
+ iph->check = ~((csum & 0xffff) + (csum >> 16));
+
+ count_tx(vip.protocol);
+
+ return XDP_TX;
+}
+
+static __always_inline int handle_ipv6(struct xdp_md *xdp)
+{
+ void *data_end = (void *)(long)xdp->data_end;
+ void *data = (void *)(long)xdp->data;
+ struct iptnl_info *tnl;
+ struct ethhdr *new_eth;
+ struct ethhdr *old_eth;
+ struct ipv6hdr *ip6h = data + sizeof(struct ethhdr);
+ __u16 payload_len;
+ struct vip vip = {};
+ int dport;
+
+ if (ip6h + 1 > data_end)
+ return XDP_DROP;
+
+ dport = get_dport(ip6h + 1, data_end, ip6h->nexthdr);
+ if (dport == -1)
+ return XDP_DROP;
+
+ vip.protocol = ip6h->nexthdr;
+ vip.family = AF_INET6;
+ memcpy(vip.daddr.v6, ip6h->daddr.s6_addr32, sizeof(vip.daddr));
+ vip.dport = dport;
+ payload_len = ip6h->payload_len;
+
+ tnl = bpf_map_lookup_elem(&vip2tnl, &vip);
+ /* It only does v6-in-v6 */
+ if (!tnl || tnl->family != AF_INET6)
+ return XDP_PASS;
+
+ if (bpf_xdp_adjust_head(xdp, 0 - (int)sizeof(struct ipv6hdr)))
+ return XDP_DROP;
+
+ data = (void *)(long)xdp->data;
+ data_end = (void *)(long)xdp->data_end;
+
+ new_eth = data;
+ ip6h = data + sizeof(*new_eth);
+ old_eth = data + sizeof(*ip6h);
+
+ if (new_eth + 1 > data_end || old_eth + 1 > data_end ||
+ ip6h + 1 > data_end)
+ return XDP_DROP;
+
+ set_ethhdr(new_eth, old_eth, tnl, bpf_htons(ETH_P_IPV6));
+
+ ip6h->version = 6;
+ ip6h->priority = 0;
+ memset(ip6h->flow_lbl, 0, sizeof(ip6h->flow_lbl));
+ ip6h->payload_len = bpf_htons(bpf_ntohs(payload_len) + sizeof(*ip6h));
+ ip6h->nexthdr = IPPROTO_IPV6;
+ ip6h->hop_limit = 8;
+ memcpy(ip6h->saddr.s6_addr32, tnl->saddr.v6, sizeof(tnl->saddr.v6));
+ memcpy(ip6h->daddr.s6_addr32, tnl->daddr.v6, sizeof(tnl->daddr.v6));
+
+ count_tx(vip.protocol);
+
+ return XDP_TX;
+}
+
+SEC("xdp_tx_iptunnel")
+int _xdp_tx_iptunnel(struct xdp_md *xdp)
+{
+ void *data_end = (void *)(long)xdp->data_end;
+ void *data = (void *)(long)xdp->data;
+ struct ethhdr *eth = data;
+ __u16 h_proto;
+
+ if (eth + 1 > data_end)
+ return XDP_DROP;
+
+ h_proto = eth->h_proto;
+
+ if (h_proto == bpf_htons(ETH_P_IP))
+ return handle_ipv4(xdp);
+ else if (h_proto == bpf_htons(ETH_P_IPV6))
+
+ return handle_ipv6(xdp);
+ else
+ return XDP_DROP;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index fbfe5d0..35cbb4c 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -5,7 +5,7 @@
reuseport_bpf_numa: LDFLAGS += -lnuma
-TEST_PROGS := run_netsocktests run_afpackettests test_bpf.sh
+TEST_PROGS := run_netsocktests run_afpackettests test_bpf.sh netdevice.sh
TEST_GEN_FILES = socket
TEST_GEN_FILES += psock_fanout psock_tpacket
TEST_GEN_FILES += reuseport_bpf reuseport_bpf_cpu reuseport_bpf_numa
diff --git a/tools/testing/selftests/net/netdevice.sh b/tools/testing/selftests/net/netdevice.sh
new file mode 100755
index 0000000..4e00568
--- /dev/null
+++ b/tools/testing/selftests/net/netdevice.sh
@@ -0,0 +1,200 @@
+#!/bin/sh
+#
+# This test is for checking network interface
+# For the moment it tests only ethernet interface (but wifi could be easily added)
+#
+# We assume that all network driver are loaded
+# if not they probably have failed earlier in the boot process and their logged error will be catched by another test
+#
+
+# this function will try to up the interface
+# if already up, nothing done
+# arg1: network interface name
+kci_net_start()
+{
+ netdev=$1
+
+ ip link show "$netdev" |grep -q UP
+ if [ $? -eq 0 ];then
+ echo "SKIP: $netdev: interface already up"
+ return 0
+ fi
+
+ ip link set "$netdev" up
+ if [ $? -ne 0 ];then
+ echo "FAIL: $netdev: Fail to up interface"
+ return 1
+ else
+ echo "PASS: $netdev: set interface up"
+ NETDEV_STARTED=1
+ fi
+ return 0
+}
+
+# this function will try to setup an IP and MAC address on a network interface
+# Doing nothing if the interface was already up
+# arg1: network interface name
+kci_net_setup()
+{
+ netdev=$1
+
+ # do nothing if the interface was already up
+ if [ $NETDEV_STARTED -eq 0 ];then
+ return 0
+ fi
+
+ MACADDR='02:03:04:05:06:07'
+ ip link set dev $netdev address "$MACADDR"
+ if [ $? -ne 0 ];then
+ echo "FAIL: $netdev: Cannot set MAC address"
+ else
+ ip link show $netdev |grep -q "$MACADDR"
+ if [ $? -eq 0 ];then
+ echo "PASS: $netdev: set MAC address"
+ else
+ echo "FAIL: $netdev: Cannot set MAC address"
+ fi
+ fi
+
+ #check that the interface did not already have an IP
+ ip address show "$netdev" |grep '^[[:space:]]*inet'
+ if [ $? -eq 0 ];then
+ echo "SKIP: $netdev: already have an IP"
+ return 0
+ fi
+
+ # TODO what ipaddr to set ? DHCP ?
+ echo "SKIP: $netdev: set IP address"
+ return 0
+}
+
+# test an ethtool command
+# arg1: return code for not supported (see ethtool code source)
+# arg2: summary of the command
+# arg3: command to execute
+kci_netdev_ethtool_test()
+{
+ if [ $# -le 2 ];then
+ echo "SKIP: $netdev: ethtool: invalid number of arguments"
+ return 1
+ fi
+ $3 >/dev/null
+ ret=$?
+ if [ $ret -ne 0 ];then
+ if [ $ret -eq "$1" ];then
+ echo "SKIP: $netdev: ethtool $2 not supported"
+ else
+ echo "FAIL: $netdev: ethtool $2"
+ return 1
+ fi
+ else
+ echo "PASS: $netdev: ethtool $2"
+ fi
+ return 0
+}
+
+# test ethtool commands
+# arg1: network interface name
+kci_netdev_ethtool()
+{
+ netdev=$1
+
+ #check presence of ethtool
+ ethtool --version 2>/dev/null >/dev/null
+ if [ $? -ne 0 ];then
+ echo "SKIP: ethtool not present"
+ return 1
+ fi
+
+ TMP_ETHTOOL_FEATURES="$(mktemp)"
+ if [ ! -e "$TMP_ETHTOOL_FEATURES" ];then
+ echo "SKIP: Cannot create a tmp file"
+ return 1
+ fi
+
+ ethtool -k "$netdev" > "$TMP_ETHTOOL_FEATURES"
+ if [ $? -ne 0 ];then
+ echo "FAIL: $netdev: ethtool list features"
+ rm "$TMP_ETHTOOL_FEATURES"
+ return 1
+ fi
+ echo "PASS: $netdev: ethtool list features"
+ #TODO for each non fixed features, try to turn them on/off
+ rm "$TMP_ETHTOOL_FEATURES"
+
+ kci_netdev_ethtool_test 74 'dump' "ethtool -d $netdev"
+ kci_netdev_ethtool_test 94 'stats' "ethtool -S $netdev"
+ return 0
+}
+
+# stop a netdev
+# arg1: network interface name
+kci_netdev_stop()
+{
+ netdev=$1
+
+ if [ $NETDEV_STARTED -eq 0 ];then
+ echo "SKIP: $netdev: interface kept up"
+ return 0
+ fi
+
+ ip link set "$netdev" down
+ if [ $? -ne 0 ];then
+ echo "FAIL: $netdev: stop interface"
+ return 1
+ fi
+ echo "PASS: $netdev: stop interface"
+ return 0
+}
+
+# run all test on a netdev
+# arg1: network interface name
+kci_test_netdev()
+{
+ NETDEV_STARTED=0
+ IFACE_TO_UPDOWN="$1"
+ IFACE_TO_TEST="$1"
+ #check for VLAN interface
+ MASTER_IFACE="$(echo $1 | cut -d@ -f2)"
+ if [ ! -z "$MASTER_IFACE" ];then
+ IFACE_TO_UPDOWN="$MASTER_IFACE"
+ IFACE_TO_TEST="$(echo $1 | cut -d@ -f1)"
+ fi
+
+ NETDEV_STARTED=0
+ kci_net_start "$IFACE_TO_UPDOWN"
+
+ kci_net_setup "$IFACE_TO_TEST"
+
+ kci_netdev_ethtool "$IFACE_TO_TEST"
+
+ kci_netdev_stop "$IFACE_TO_UPDOWN"
+ return 0
+}
+
+#check for needed privileges
+if [ "$(id -u)" -ne 0 ];then
+ echo "SKIP: Need root privileges"
+ exit 0
+fi
+
+ip -Version 2>/dev/null >/dev/null
+if [ $? -ne 0 ];then
+ echo "SKIP: Could not run test without the ip tool"
+ exit 0
+fi
+
+TMP_LIST_NETDEV="$(mktemp)"
+if [ ! -e "$TMP_LIST_NETDEV" ];then
+ echo "FAIL: Cannot create a tmp file"
+ exit 1
+fi
+
+ip link show |grep '^[0-9]' | grep -oE '[[:space:]].*eth[0-9]*:|[[:space:]].*enp[0-9]s[0-9]:' | cut -d\ -f2 | cut -d: -f1> "$TMP_LIST_NETDEV"
+while read netdev
+do
+ kci_test_netdev "$netdev"
+done < "$TMP_LIST_NETDEV"
+
+rm "$TMP_LIST_NETDEV"
+exit 0
diff --git a/tools/testing/selftests/net/psock_fanout.c b/tools/testing/selftests/net/psock_fanout.c
index e62bb35..989f917 100644
--- a/tools/testing/selftests/net/psock_fanout.c
+++ b/tools/testing/selftests/net/psock_fanout.c
@@ -71,7 +71,7 @@
/* Open a socket in a given fanout mode.
* @return -1 if mode is bad, a valid socket otherwise */
-static int sock_fanout_open(uint16_t typeflags, int num_packets)
+static int sock_fanout_open(uint16_t typeflags, uint16_t group_id)
{
int fd, val;
@@ -81,8 +81,7 @@
exit(1);
}
- /* fanout group ID is always 0: tests whether old groups are deleted */
- val = ((int) typeflags) << 16;
+ val = (((int) typeflags) << 16) | group_id;
if (setsockopt(fd, SOL_PACKET, PACKET_FANOUT, &val, sizeof(val))) {
if (close(fd)) {
perror("close packet");
@@ -113,6 +112,20 @@
}
}
+static void sock_fanout_getopts(int fd, uint16_t *typeflags, uint16_t *group_id)
+{
+ int sockopt;
+ socklen_t sockopt_len = sizeof(sockopt);
+
+ if (getsockopt(fd, SOL_PACKET, PACKET_FANOUT,
+ &sockopt, &sockopt_len)) {
+ perror("failed to getsockopt");
+ exit(1);
+ }
+ *typeflags = sockopt >> 16;
+ *group_id = sockopt & 0xfffff;
+}
+
static void sock_fanout_set_ebpf(int fd)
{
const int len_off = __builtin_offsetof(struct __sk_buff, len);
@@ -241,26 +254,26 @@
fprintf(stderr, "test: control multiple sockets\n");
- fds[0] = sock_fanout_open(PACKET_FANOUT_HASH, 20);
+ fds[0] = sock_fanout_open(PACKET_FANOUT_HASH, 0);
if (fds[0] == -1) {
fprintf(stderr, "ERROR: failed to open HASH socket\n");
exit(1);
}
if (sock_fanout_open(PACKET_FANOUT_HASH |
- PACKET_FANOUT_FLAG_DEFRAG, 10) != -1) {
+ PACKET_FANOUT_FLAG_DEFRAG, 0) != -1) {
fprintf(stderr, "ERROR: joined group with wrong flag defrag\n");
exit(1);
}
if (sock_fanout_open(PACKET_FANOUT_HASH |
- PACKET_FANOUT_FLAG_ROLLOVER, 10) != -1) {
+ PACKET_FANOUT_FLAG_ROLLOVER, 0) != -1) {
fprintf(stderr, "ERROR: joined group with wrong flag ro\n");
exit(1);
}
- if (sock_fanout_open(PACKET_FANOUT_CPU, 10) != -1) {
+ if (sock_fanout_open(PACKET_FANOUT_CPU, 0) != -1) {
fprintf(stderr, "ERROR: joined group with wrong mode\n");
exit(1);
}
- fds[1] = sock_fanout_open(PACKET_FANOUT_HASH, 20);
+ fds[1] = sock_fanout_open(PACKET_FANOUT_HASH, 0);
if (fds[1] == -1) {
fprintf(stderr, "ERROR: failed to join group\n");
exit(1);
@@ -271,6 +284,61 @@
}
}
+/* Test creating a unique fanout group ids */
+static void test_unique_fanout_group_ids(void)
+{
+ int fds[3];
+ uint16_t typeflags, first_group_id, second_group_id;
+
+ fprintf(stderr, "test: unique ids\n");
+
+ fds[0] = sock_fanout_open(PACKET_FANOUT_HASH |
+ PACKET_FANOUT_FLAG_UNIQUEID, 0);
+ if (fds[0] == -1) {
+ fprintf(stderr, "ERROR: failed to create a unique id group.\n");
+ exit(1);
+ }
+
+ sock_fanout_getopts(fds[0], &typeflags, &first_group_id);
+ if (typeflags != PACKET_FANOUT_HASH) {
+ fprintf(stderr, "ERROR: unexpected typeflags %x\n", typeflags);
+ exit(1);
+ }
+
+ if (sock_fanout_open(PACKET_FANOUT_CPU, first_group_id) != -1) {
+ fprintf(stderr, "ERROR: joined group with wrong type.\n");
+ exit(1);
+ }
+
+ fds[1] = sock_fanout_open(PACKET_FANOUT_HASH, first_group_id);
+ if (fds[1] == -1) {
+ fprintf(stderr,
+ "ERROR: failed to join previously created group.\n");
+ exit(1);
+ }
+
+ fds[2] = sock_fanout_open(PACKET_FANOUT_HASH |
+ PACKET_FANOUT_FLAG_UNIQUEID, 0);
+ if (fds[2] == -1) {
+ fprintf(stderr,
+ "ERROR: failed to create a second unique id group.\n");
+ exit(1);
+ }
+
+ sock_fanout_getopts(fds[2], &typeflags, &second_group_id);
+ if (sock_fanout_open(PACKET_FANOUT_HASH | PACKET_FANOUT_FLAG_UNIQUEID,
+ second_group_id) != -1) {
+ fprintf(stderr,
+ "ERROR: specified a group id when requesting unique id\n");
+ exit(1);
+ }
+
+ if (close(fds[0]) || close(fds[1]) || close(fds[2])) {
+ fprintf(stderr, "ERROR: closing sockets\n");
+ exit(1);
+ }
+}
+
static int test_datapath(uint16_t typeflags, int port_off,
const int expect1[], const int expect2[])
{
@@ -281,8 +349,8 @@
fprintf(stderr, "test: datapath 0x%hx\n", typeflags);
- fds[0] = sock_fanout_open(typeflags, 20);
- fds[1] = sock_fanout_open(typeflags, 20);
+ fds[0] = sock_fanout_open(typeflags, 0);
+ fds[1] = sock_fanout_open(typeflags, 0);
if (fds[0] == -1 || fds[1] == -1) {
fprintf(stderr, "ERROR: failed open\n");
exit(1);
@@ -349,10 +417,12 @@
const int expect_cpu0[2][2] = { { 20, 0 }, { 20, 0 } };
const int expect_cpu1[2][2] = { { 0, 20 }, { 0, 20 } };
const int expect_bpf[2][2] = { { 15, 5 }, { 15, 20 } };
+ const int expect_uniqueid[2][2] = { { 20, 20}, { 20, 20 } };
int port_off = 2, tries = 5, ret;
test_control_single();
test_control_group();
+ test_unique_fanout_group_ids();
/* find a set of ports that do not collide onto the same socket */
ret = test_datapath(PACKET_FANOUT_HASH, port_off,
@@ -383,6 +453,9 @@
ret |= test_datapath(PACKET_FANOUT_CPU, port_off,
expect_cpu1[0], expect_cpu1[1]);
+ ret |= test_datapath(PACKET_FANOUT_FLAG_UNIQUEID, port_off,
+ expect_uniqueid[0], expect_uniqueid[1]);
+
if (ret)
return 1;
diff --git a/tools/testing/selftests/powerpc/Makefile b/tools/testing/selftests/powerpc/Makefile
index bf13fc2..2132ff8 100644
--- a/tools/testing/selftests/powerpc/Makefile
+++ b/tools/testing/selftests/powerpc/Makefile
@@ -8,7 +8,7 @@
GIT_VERSION = $(shell git describe --always --long --dirty || echo "unknown")
-CFLAGS := -std=gnu99 -Wall -O2 -Wall -Werror -DGIT_VERSION='"$(GIT_VERSION)"' -I$(CURDIR)/include $(CFLAGS)
+CFLAGS := -std=gnu99 -O2 -Wall -Werror -DGIT_VERSION='"$(GIT_VERSION)"' -I$(CURDIR)/include $(CFLAGS)
export CFLAGS
diff --git a/tools/testing/selftests/timers/Makefile b/tools/testing/selftests/timers/Makefile
index b90e50c..5fa1d7e9 100644
--- a/tools/testing/selftests/timers/Makefile
+++ b/tools/testing/selftests/timers/Makefile
@@ -3,7 +3,7 @@
LDFLAGS += -lrt -lpthread
# these are all "safe" tests that don't modify
-# system time or require escalated privledges
+# system time or require escalated privileges
TEST_GEN_PROGS = posix_timers nanosleep nsleep-lat set-timer-lat mqueue-lat \
inconsistency-check raw_skew threadtest rtctest
@@ -14,7 +14,7 @@
include ../lib.mk
-# these tests require escalated privledges
+# these tests require escalated privileges
# and may modify the system time or trigger
# other behavior like suspend
run_destructive_tests: run_tests
diff --git a/tools/testing/selftests/x86/ldt_gdt.c b/tools/testing/selftests/x86/ldt_gdt.c
index f612161..b9a22f1 100644
--- a/tools/testing/selftests/x86/ldt_gdt.c
+++ b/tools/testing/selftests/x86/ldt_gdt.c
@@ -409,6 +409,51 @@
}
}
+#ifdef __i386__
+
+#ifndef SA_RESTORE
+#define SA_RESTORER 0x04000000
+#endif
+
+/*
+ * The UAPI header calls this 'struct sigaction', which conflicts with
+ * glibc. Sigh.
+ */
+struct fake_ksigaction {
+ void *handler; /* the real type is nasty */
+ unsigned long sa_flags;
+ void (*sa_restorer)(void);
+ unsigned char sigset[8];
+};
+
+static void fix_sa_restorer(int sig)
+{
+ struct fake_ksigaction ksa;
+
+ if (syscall(SYS_rt_sigaction, sig, NULL, &ksa, 8) == 0) {
+ /*
+ * glibc has a nasty bug: it sometimes writes garbage to
+ * sa_restorer. This interacts quite badly with anything
+ * that fiddles with SS because it can trigger legacy
+ * stack switching. Patch it up. See:
+ *
+ * https://sourceware.org/bugzilla/show_bug.cgi?id=21269
+ */
+ if (!(ksa.sa_flags & SA_RESTORER) && ksa.sa_restorer) {
+ ksa.sa_restorer = NULL;
+ if (syscall(SYS_rt_sigaction, sig, &ksa, NULL,
+ sizeof(ksa.sigset)) != 0)
+ err(1, "rt_sigaction");
+ }
+ }
+}
+#else
+static void fix_sa_restorer(int sig)
+{
+ /* 64-bit glibc works fine. */
+}
+#endif
+
static void sethandler(int sig, void (*handler)(int, siginfo_t *, void *),
int flags)
{
@@ -420,6 +465,7 @@
if (sigaction(sig, &sa, 0))
err(1, "sigaction");
+ fix_sa_restorer(sig);
}
static jmp_buf jmpbuf;
diff --git a/tools/testing/selftests/x86/mpx-mini-test.c b/tools/testing/selftests/x86/mpx-mini-test.c
index 616ee96..a8df159 100644
--- a/tools/testing/selftests/x86/mpx-mini-test.c
+++ b/tools/testing/selftests/x86/mpx-mini-test.c
@@ -404,8 +404,6 @@
dprintf2("info->si_lower: %p\n", __si_bounds_lower(si));
dprintf2("info->si_upper: %p\n", __si_bounds_upper(si));
- check_siginfo_vs_shadow(si);
-
for (i = 0; i < 8; i++)
dprintf3("[%d]: %p\n", i, si_addr_ptr[i]);
switch (br_reason) {
@@ -416,6 +414,9 @@
exit(5);
case 1: /* #BR MPX bounds exception */
/* these are normal and we expect to see them */
+
+ check_siginfo_vs_shadow(si);
+
dprintf1("bounds exception (normal): status 0x%jx at %p si_addr: %p\n",
status, (void *)ip, si->si_addr);
num_bnd_chk++;
